mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-21 13:28:08 +09:00
Compare commits
1015
Commits
No files matched your search
@@ -0,0 +1,106 @@
|
||||
# Plan: Migrate from MailSlurp to testmail.app
|
||||
|
||||
**Created:** 2026-01-02
|
||||
**Branch:** feat/sess-20260102-2109-migrate-mailslurp-testmail
|
||||
**Status:** Complete
|
||||
|
||||
## Overview
|
||||
|
||||
Replace MailSlurp email testing service with testmail.app for E2E testing of magic link authentication. testmail.app uses a simpler API with namespace-based email addresses that require no inbox creation/deletion - emails just arrive at `{namespace}.{tag}@inbox.testmail.app`.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Remove all code referencing MailSlurp
|
||||
- [x] Remove `mailslurp-client` npm dependency
|
||||
- [x] Implement testmail.app API integration
|
||||
- [x] Update environment variable names (`TESTMAIL_API_KEY`, `TESTMAIL_NAMESPACE`)
|
||||
- [x] Maintain same `EmailHelper` interface for minimal test changes
|
||||
- [x] Update archive docs to note migration (optional - historical accuracy)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### API Differences
|
||||
|
||||
| Feature | MailSlurp | testmail.app |
|
||||
|---------|-----------|--------------|
|
||||
| Email format | Random `xxx@mailslurp.com` | `{namespace}.{tag}@inbox.testmail.app` |
|
||||
| Inbox creation | Required (`createInbox()`) | Not needed - use any tag |
|
||||
| Inbox deletion | Required (`deleteInbox()`) | Not needed |
|
||||
| Get emails | `waitForLatestEmail(inboxId)` | GET `/api/json?apikey=X&namespace=Y&tag=Z&livequery=true` |
|
||||
| Auth | API key in constructor | API key as query param |
|
||||
| Email body | `email.body` | `email.html` or `email.text` |
|
||||
|
||||
### Implementation Strategy
|
||||
|
||||
1. **Keep same `EmailHelper` interface** - `createInbox()`, `waitForEmail()`, `extractMagicLink()`, `deleteInbox()`
|
||||
2. **Use testmail.app's live query feature** - `livequery=true` waits for email to arrive (up to timeout)
|
||||
3. **Generate unique tags** - Use timestamp + random suffix for each test run to avoid collisions
|
||||
4. **Inbox becomes a simple object** - `{ id: tag, emailAddress: '{namespace}.{tag}@inbox.testmail.app' }`
|
||||
5. **Delete becomes no-op** - testmail.app emails auto-expire, no cleanup needed
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Old | New |
|
||||
|-----|-----|
|
||||
| `MAILSLURP_API_KEY` | `TESTMAIL_API_KEY` |
|
||||
| (none) | `TESTMAIL_NAMESPACE` |
|
||||
|
||||
Values: Set via environment variables (see `frontend/.env.local.example`)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. [x] **Update `email-testing.ts`** - Replace MailSlurp implementation with testmail.app
|
||||
- Remove `mailslurp-client` import
|
||||
- Add testmail.app API fetch implementation
|
||||
- Use native `fetch` (no new dependencies)
|
||||
- Generate unique tag for each "inbox"
|
||||
- Implement `livequery=true` for waiting
|
||||
- Adapt email body extraction (`html`/`text` instead of `body`)
|
||||
|
||||
2. [x] **Update `full-user-journey.spec.ts`** - Change references
|
||||
- Update doc comments (env var name)
|
||||
- Update error messages
|
||||
- Update console logs
|
||||
|
||||
3. [x] **Update `playwright.record.config.ts`** - Change env var comment
|
||||
|
||||
4. [x] **Update `package.json`** - Remove `mailslurp-client` dependency
|
||||
|
||||
5. [x] **Run `npm install`** - Update package-lock.json
|
||||
|
||||
6. [x] **Create `.env.local.example`** - Document required env vars for local testing
|
||||
|
||||
7. [x] **Update archive docs** - Skipped intentionally; historical docs preserved for accuracy
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `frontend/e2e/helpers/email-testing.ts` | Modify | Replace MailSlurp with testmail.app |
|
||||
| `frontend/e2e/production/full-user-journey.spec.ts` | Modify | Update env var refs and messages |
|
||||
| `frontend/playwright.record.config.ts` | Modify | Update env var comment |
|
||||
| `frontend/package.json` | Modify | Remove `mailslurp-client` |
|
||||
| `frontend/package-lock.json` | Modify | Auto-updated by npm install |
|
||||
| `frontend/.env.local.example` | Create | Document env vars for local testing |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Manual testing**: Run production E2E test with testmail.app credentials
|
||||
```bash
|
||||
cd frontend
|
||||
TESTMAIL_API_KEY=your-api-key \
|
||||
TESTMAIL_NAMESPACE=your-namespace \
|
||||
npm run test:e2e:prod:headed
|
||||
```
|
||||
- **Verify**: Email received, magic link extracted, auth flow completes
|
||||
- **CI testing**: Env vars already configured in GitHub Actions org secrets
|
||||
|
||||
## Open Questions
|
||||
|
||||
None - requirements are clear.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. Revert the commit
|
||||
2. Run `npm install` to restore mailslurp-client
|
||||
3. Re-add MAILSLURP_API_KEY to secrets if removed
|
||||
@@ -0,0 +1,401 @@
|
||||
# Plan: Video Worker Stage Refactor
|
||||
|
||||
**Created:** 2026-01-03
|
||||
**Branch:** feat/sess-20260103-2250-investigate-youtube-upload
|
||||
**Status:** COMPLETE - Ready for PR
|
||||
**Last Updated:** 2026-01-04
|
||||
|
||||
## Overview
|
||||
|
||||
Refactor `video_worker.py` and `KaraokeFinalise` into separate, composable stages following SOLID principles. This eliminates the GCE vs KaraokeFinalise code path divergence that causes features like YouTube upload to be missing from the GCE encoding path.
|
||||
|
||||
## Three Execution Modes
|
||||
|
||||
Understanding the three ways this system runs is critical to the refactor:
|
||||
|
||||
### 1. Local CLI (`karaoke-gen`)
|
||||
- Runs entirely on user's machine
|
||||
- KaraokeFinalise.process() orchestrates everything
|
||||
- Writes directly to local Dropbox sync folder (e.g., `~/Dropbox/Karaoke/Tracks-Organized/`)
|
||||
- **Produces full output structure** with stems/, lyrics/, visualizations/, etc.
|
||||
- **This is the source of truth for output format**
|
||||
|
||||
### 2. Remote CLI (`karaoke-gen-remote`)
|
||||
- User's CLI talks to cloud backend (api.nomadkaraoke.com)
|
||||
- Cloud does ALL processing (audio, lyrics, video encoding)
|
||||
- CLI just monitors job status and downloads final outputs
|
||||
- With Dropbox configured: cloud uploads full structure to user's Dropbox
|
||||
- Without Dropbox: user downloads individual files
|
||||
|
||||
### 3. Web Frontend (`gen.nomadkaraoke.com`)
|
||||
- Same cloud backend as remote CLI
|
||||
- User sees:
|
||||
- YouTube video link
|
||||
- Dropbox folder link
|
||||
- Direct download: final karaoke video (720p)
|
||||
- Direct download: 4K video
|
||||
- Direct download: "with vocals" video
|
||||
- Direct download: CDG zip (if enabled)
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Currently, `video_worker.py` has two completely divergent code paths:
|
||||
|
||||
```python
|
||||
if use_gce_encoding:
|
||||
# GCE PATH: Only encoding, missing YouTube/Discord/CDG/TXT
|
||||
result = await _encode_via_gce(...)
|
||||
# result DOES NOT include youtube_url!
|
||||
else:
|
||||
# STANDARD PATH: KaraokeFinalise does everything
|
||||
finalise = KaraokeFinalise(...)
|
||||
result = finalise.process()
|
||||
# result includes youtube_url from upload
|
||||
```
|
||||
|
||||
### What GCE Encoding Broke
|
||||
|
||||
| Feature | KaraokeFinalise Path | GCE Path |
|
||||
|---------|---------------------|----------|
|
||||
| Video encoding | ✅ 4 formats | ✅ 4 formats |
|
||||
| YouTube upload | ✅ | ❌ **BROKEN** |
|
||||
| Discord notification | ✅ | ❌ **BROKEN** |
|
||||
| CDG/TXT packaging | ✅ | ❌ **BROKEN** |
|
||||
| Brand code generation | ✅ | ✅ (via Dropbox service) |
|
||||
| Dropbox upload | ✅ (rclone) | ✅ (native SDK) |
|
||||
| Google Drive upload | ❌ | ✅ (native SDK) |
|
||||
|
||||
### Root Cause
|
||||
|
||||
`KaraokeFinalise.process()` is a monolith that handles:
|
||||
1. CDG/TXT packaging
|
||||
2. Video encoding (the CPU-intensive part)
|
||||
3. YouTube upload
|
||||
4. Discord notifications
|
||||
5. Folder organization
|
||||
|
||||
The GCE encoding was added as a bypass around the entire class, rather than replacing just the encoding step.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements
|
||||
- [ ] YouTube upload works regardless of encoding backend (GCE or local)
|
||||
- [ ] Discord notifications work regardless of encoding backend
|
||||
- [ ] CDG/TXT packaging works regardless of encoding backend
|
||||
- [ ] Brand code generation works consistently
|
||||
- [ ] Dropbox/Google Drive upload works (already works in both paths)
|
||||
- [ ] All existing CLI functionality continues to work
|
||||
- [ ] Output structure matches local CLI exactly
|
||||
|
||||
### Output Structure Parity
|
||||
|
||||
Cloud backend must produce identical output to local CLI:
|
||||
|
||||
```
|
||||
NOMAD-XXXX - Artist - Title/
|
||||
├── Artist - Title (Title).mov
|
||||
├── Artist - Title (Title).jpg
|
||||
├── Artist - Title (End).mov
|
||||
├── Artist - Title (End).jpg
|
||||
├── Artist - Title (Final Karaoke Lossless 4k).mp4
|
||||
├── Artist - Title (Final Karaoke Lossless 4k).mkv
|
||||
├── Artist - Title (Final Karaoke Lossy 4k).mp4
|
||||
├── Artist - Title (Final Karaoke Lossy 720p).mp4
|
||||
├── Artist - Title (Final Karaoke CDG).zip
|
||||
├── Artist - Title (Final Karaoke TXT).zip
|
||||
├── Artist - Title (Instrumental {model}).flac
|
||||
├── Artist - Title (Instrumental +BV {model}).flac
|
||||
├── Artist - Title (Karaoke).mp4
|
||||
├── Artist - Title (Karaoke).cdg
|
||||
├── Artist - Title (Karaoke).mp3
|
||||
├── Artist - Title (Karaoke).lrc
|
||||
├── Artist - Title (With Vocals).mp4
|
||||
├── stems/
|
||||
│ ├── Artist - Title (Instrumental {model}).flac
|
||||
│ ├── Artist - Title (Vocals {model}).flac
|
||||
│ ├── Artist - Title (Lead Vocals {model}).flac
|
||||
│ ├── Artist - Title (Backing Vocals {model}).flac
|
||||
│ └── ... (other stems)
|
||||
├── lyrics/
|
||||
│ ├── Artist - Title (Karaoke).lrc
|
||||
│ ├── Artist - Title (Karaoke).ass
|
||||
│ ├── Artist - Title (Karaoke).srt
|
||||
│ └── ... (corrections JSONs)
|
||||
├── previews/
|
||||
├── visualizations/
|
||||
└── styles_assets/
|
||||
```
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [ ] Single Responsibility: Each stage has one job
|
||||
- [ ] Open/Closed: Easy to add new encoding backends without modifying stages
|
||||
- [ ] Testable: Each stage can be unit tested in isolation
|
||||
- [ ] No duplicate code between paths
|
||||
- [ ] Backwards compatible with existing KaraokeFinalise CLI usage
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Architecture: Stage-Based Pipeline
|
||||
|
||||
```
|
||||
video_worker.py (VideoWorkerOrchestrator)
|
||||
│
|
||||
├── 1. EncodingStage (interface)
|
||||
│ ├── LocalEncodingService (wraps KaraokeFinalise encoding methods)
|
||||
│ └── GCEEncodingService (existing encoding_service.py)
|
||||
│
|
||||
├── 2. PackagingStage
|
||||
│ ├── CDGPackager (extracts from KaraokeFinalise)
|
||||
│ └── TXTPackager (extracts from KaraokeFinalise)
|
||||
│
|
||||
├── 3. DistributionStage
|
||||
│ ├── YouTubeUploadService (extracts from KaraokeFinalise)
|
||||
│ ├── DropboxService (existing)
|
||||
│ ├── GoogleDriveService (existing)
|
||||
│ └── DiscordService (extracts from KaraokeFinalise)
|
||||
│
|
||||
└── 4. OrganizationStage
|
||||
└── BrandCodeService (extracts from KaraokeFinalise + DropboxService)
|
||||
```
|
||||
|
||||
### Key Design Decisions
|
||||
|
||||
1. **Extract, Don't Rewrite**: Pull methods out of KaraokeFinalise into services, don't rewrite from scratch
|
||||
2. **Keep KaraokeFinalise for CLI**: Local CLI still uses KaraokeFinalise.process() - we extract services that it can also use
|
||||
3. **Interface-Based Encoding**: EncodingStage interface allows swapping GCE/local without changing orchestration
|
||||
4. **Existing Service Pattern**: Follow singleton + factory pattern already used in codebase
|
||||
5. **Local CLI stays sync**: Local karaoke-gen CLI remains synchronous as it is today
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
JobConfig (from Firestore)
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ VideoWorkerOrchestrator │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 1. Setup: Download files, prepare directories ││
|
||||
│ │ - Download title.mov, end.mov, with_vocals.mkv ││
|
||||
│ │ - Download instrumental audio ││
|
||||
│ │ - Prepare temp directory ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 2. Packaging: CDG/TXT (if enabled) - runs BEFORE encoding ││
|
||||
│ │ Reason: CDG depends on LRC file which is available ││
|
||||
│ │ Input: LRC file, instrumental audio ││
|
||||
│ │ Output: CDG.zip, TXT.zip, .cdg, .mp3 ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 3. Encoding: GCE or Local (via interface) ││
|
||||
│ │ Input: title.mov, karaoke.mov, end.mov ││
|
||||
│ │ Output: 4K lossless, 4K lossy, 720p, MKV ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 4. Organization: Brand code, folder structure ││
|
||||
│ │ - Generate brand code (NOMAD-XXXX) ││
|
||||
│ │ - Prepare full output directory structure ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 5. Distribution: Upload to destinations ││
|
||||
│ │ - YouTube upload (if enabled) ││
|
||||
│ │ - Dropbox upload (if configured) ││
|
||||
│ │ - Google Drive upload (if configured) ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 6. Notification: Discord, Email ││
|
||||
│ │ Input: YouTube URL, brand code, Dropbox link ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
Result (youtube_url, brand_code, dropbox_link, file_paths, etc.)
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Extract Services from KaraokeFinalise (Non-Breaking) ✅ COMPLETE
|
||||
|
||||
These services can be used by both the cloud orchestrator AND KaraokeFinalise.
|
||||
|
||||
1. [x] **Create `backend/services/youtube_upload_service.py`** ✅
|
||||
- Extract YouTube upload logic from KaraokeFinalise
|
||||
- Methods: `upload_video()`, `check_duplicate()`, `delete_video()`, `truncate_title()`
|
||||
- Supports pre-loaded credentials (server-side) or client secrets file (CLI)
|
||||
- 23 unit tests with mocked YouTube API
|
||||
|
||||
2. [x] **Create `backend/services/discord_service.py`** ✅
|
||||
- Extract Discord notification from KaraokeFinalise
|
||||
- Methods: `post_message()`, `post_video_notification()`, `is_enabled()`
|
||||
- Simple HTTP POST to webhook with validation
|
||||
- 20 unit tests with mocked requests
|
||||
|
||||
3. [x] **Create `backend/services/packaging_service.py`** ✅
|
||||
- Extract CDG/TXT generation from KaraokeFinalise
|
||||
- Methods: `create_cdg_package()`, `create_txt_package()`
|
||||
- Uses lyrics_converter and CDGGenerator (existing dependencies)
|
||||
- 18 unit tests with temp files
|
||||
|
||||
4. [x] **Create `backend/services/local_encoding_service.py`** ✅
|
||||
- Extract encoding methods from KaraokeFinalise
|
||||
- Methods: `encode_all_formats()`, individual encoding methods
|
||||
- Hardware acceleration detection with software fallback
|
||||
- 28 unit tests with mocked subprocess
|
||||
|
||||
### Phase 2: Create Encoding Interface ✅ COMPLETE
|
||||
|
||||
5. [x] **Create `backend/services/encoding_interface.py`** ✅
|
||||
- Define ABC `EncodingBackend` with methods: `encode()`, `is_available()`, `get_status()`
|
||||
- `EncodingInput` and `EncodingOutput` dataclasses for config/results
|
||||
- Factory function `get_encoding_backend()` with "local", "gce", "auto" options
|
||||
|
||||
6. [x] **`LocalEncodingBackend`** ✅
|
||||
- Implements `EncodingBackend` interface
|
||||
- Wraps `LocalEncodingService` with `asyncio.to_thread()`
|
||||
- 17 unit tests
|
||||
|
||||
7. [x] **`GCEEncodingBackend`** ✅
|
||||
- Implements `EncodingBackend` interface
|
||||
- Wraps existing `EncodingService` (encoding_service.py)
|
||||
- Requires GCS paths in options for cloud execution
|
||||
- 6 additional unit tests
|
||||
|
||||
### Phase 3: Create Orchestrator ✅ COMPLETE
|
||||
|
||||
8. [x] **Create `backend/workers/video_worker_orchestrator.py`** ✅
|
||||
- New class that coordinates all stages
|
||||
- Replaces the branching logic in `generate_video()`
|
||||
- Methods:
|
||||
- `__init__(config, job_manager, storage, job_logger)`
|
||||
- `async run() -> OrchestratorResult`
|
||||
- `_run_packaging()` - CDG/TXT
|
||||
- `_run_encoding()` - GCE or local via EncodingBackend interface
|
||||
- `_run_organization()` - Brand code, folder structure
|
||||
- `_run_distribution()` - YouTube, Dropbox, GDrive
|
||||
- `_run_notifications()` - Discord
|
||||
- 27 unit tests (test_video_worker_orchestrator.py)
|
||||
|
||||
9. [x] **Update `backend/workers/video_worker.py`** ✅
|
||||
- Added `USE_NEW_ORCHESTRATOR` feature flag (default: true)
|
||||
- `generate_video()` routes to orchestrator or legacy path
|
||||
- `generate_video_orchestrated()` - new unified pipeline
|
||||
- `generate_video_legacy()` - original code for rollback
|
||||
- Legacy GCE and KaraokeFinalise paths preserved for safety
|
||||
|
||||
### Phase 4: Update KaraokeFinalise to Use Services (OPTIONAL)
|
||||
|
||||
10. [ ] **Update `karaoke_gen/karaoke_finalise/karaoke_finalise.py`**
|
||||
- Import and use the new services
|
||||
- Replace inline YouTube/Discord/CDG code with service calls
|
||||
- Keep process() working for CLI backwards compatibility
|
||||
- Services become the single source of truth
|
||||
- **NOTE**: This is optional for MVP - the orchestrator already uses the extracted services directly
|
||||
|
||||
### Phase 5: Testing & Validation ✅ COMPLETE
|
||||
|
||||
11. [x] **Add unit tests for each service** ✅
|
||||
- test_youtube_upload_service.py (23 tests)
|
||||
- test_discord_service.py (20 tests)
|
||||
- test_packaging_service.py (18 tests)
|
||||
- test_local_encoding_service.py (28 tests)
|
||||
- test_encoding_interface.py (23 tests)
|
||||
- test_video_worker_orchestrator.py (27 tests)
|
||||
- **Total: 139 new tests**
|
||||
|
||||
12. [x] **Full test suite passes** ✅
|
||||
- 976 tests passed (86 skipped - emulator/integration tests)
|
||||
- No regressions from changes
|
||||
|
||||
13. [ ] **Manual E2E testing** (After deployment)
|
||||
- [ ] GCE encoding job with YouTube upload enabled
|
||||
- [ ] Local encoding job with YouTube upload enabled
|
||||
- [ ] CLI karaoke-gen with all features
|
||||
- [ ] Job with CDG/TXT enabled
|
||||
- [ ] Verify output structure matches local CLI
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
| File | Action | Description | Status |
|
||||
|------|--------|-------------|--------|
|
||||
| `backend/services/youtube_upload_service.py` | Create | YouTube upload logic (~446 lines) | ✅ |
|
||||
| `backend/services/discord_service.py` | Create | Discord notification service (~167 lines) | ✅ |
|
||||
| `backend/services/packaging_service.py` | Create | CDG/TXT package generation (~266 lines) | ✅ |
|
||||
| `backend/services/local_encoding_service.py` | Create | Local FFmpeg encoding service (~512 lines) | ✅ |
|
||||
| `backend/services/encoding_interface.py` | Create | Abstract interface + backends (~418 lines) | ✅ |
|
||||
| `backend/workers/video_worker_orchestrator.py` | Create | Stage orchestration logic (~550 lines) | ✅ |
|
||||
| `backend/workers/video_worker.py` | Modify | Added orchestrator path + feature flag | ✅ |
|
||||
| `backend/tests/test_youtube_upload_service.py` | Create | 23 unit tests | ✅ |
|
||||
| `backend/tests/test_discord_service.py` | Create | 20 unit tests | ✅ |
|
||||
| `backend/tests/test_packaging_service.py` | Create | 18 unit tests | ✅ |
|
||||
| `backend/tests/test_local_encoding_service.py` | Create | 28 unit tests | ✅ |
|
||||
| `backend/tests/test_encoding_interface.py` | Create | 23 unit tests | ✅ |
|
||||
| `backend/tests/test_video_worker_orchestrator.py` | Create | 27 unit tests | ✅ |
|
||||
| `karaoke_gen/karaoke_finalise/karaoke_finalise.py` | Modify | Use extracted services (optional) | Deferred |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- **youtube_upload_service**: Mock YouTube API, test upload/delete/duplicate check
|
||||
- **discord_service**: Mock requests, test webhook POST
|
||||
- **packaging_service**: Use temp files, test CDG/TXT generation
|
||||
- **local_encoding_service**: Mock subprocess, test FFmpeg command construction
|
||||
|
||||
### Integration Tests
|
||||
- **video_worker_orchestrator**: Mock all services, test stage sequencing
|
||||
- **video_worker**: Test with mocked orchestrator
|
||||
|
||||
### E2E Verification
|
||||
- [ ] GCE encoding job with YouTube upload enabled → YouTube link appears
|
||||
- [ ] Local encoding job with YouTube upload enabled → Same behavior
|
||||
- [ ] CLI `karaoke-gen` with all features → No regression
|
||||
- [ ] Job with CDG/TXT enabled → Packages generated
|
||||
- [ ] Dropbox output structure → Matches local CLI exactly
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] Should we keep KaraokeFinalise.process() or deprecate it?
|
||||
- **Decision**: Keep it for CLI backwards compatibility. It will use the new services internally.
|
||||
|
||||
- [x] Should local_encoding_service run FFmpeg sync or async?
|
||||
- **Decision**: Sync implementation, async wrapper via `asyncio.to_thread()` for interface compatibility.
|
||||
|
||||
- [x] Order of packaging vs encoding?
|
||||
- **Decision**: CDG/TXT runs BEFORE encoding (depends on LRC file, not videos). KaraokeFinalise does it this way too.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. The refactor is additive - new services don't replace old code until orchestrator is wired up
|
||||
2. Feature flag: `USE_NEW_ORCHESTRATOR=true` to enable new code path
|
||||
3. If issues found in production:
|
||||
- Set `USE_NEW_ORCHESTRATOR=false` to revert to old behavior
|
||||
- Old `generate_video()` code preserved behind feature flag
|
||||
4. After 1 week stable, remove feature flag and old code
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. Job `5b78ad46` scenario (GCE + YouTube) would upload to YouTube
|
||||
2. All 803+ existing tests pass
|
||||
3. E2E happy path test passes
|
||||
4. No regression in local CLI functionality
|
||||
5. Output structure matches local CLI exactly
|
||||
6. Code coverage maintained or improved
|
||||
|
||||
## Estimated Scope
|
||||
|
||||
- **Phase 1** (Extract Services): 4 new service files + tests (~400-600 lines each)
|
||||
- **Phase 2** (Interface): 1 new file, 2 modifications (~100 lines)
|
||||
- **Phase 3** (Orchestrator): 2 files (~300-400 lines)
|
||||
- **Phase 4** (KaraokeFinalise): 1 modification (~100 lines of changes)
|
||||
- **Phase 5** (Testing): Integration tests + manual verification
|
||||
|
||||
This is a medium-sized refactor that can be done incrementally. Each phase produces working, tested code.
|
||||
@@ -0,0 +1,222 @@
|
||||
# Plan: Worker Logs Rearchitecture
|
||||
|
||||
**Created:** 2026-01-04
|
||||
**Branch:** `feat/sess-20260104-1931-logs-research`
|
||||
**Status:** Implementation Complete - Pending PR Merge & Deployment
|
||||
|
||||
## Overview
|
||||
|
||||
Move `worker_logs` from embedded array in job documents to a Firestore subcollection to avoid the 1MB document size limit. The current approach caused job 501258e1 to fail when logs reached 1.26 MB (98.6% of document size).
|
||||
|
||||
## Problem Statement
|
||||
|
||||
- Firestore documents have a 1 MB size limit
|
||||
- `worker_logs` is stored as an array field within job documents
|
||||
- Long-running jobs (especially video encoding) can generate thousands of log entries
|
||||
- When the limit is exceeded, job updates fail with: `Document cannot be written because its size exceeds the maximum allowed size`
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Logs must remain associated with their job
|
||||
- [x] Logs should be queryable by job_id, worker type, timestamp
|
||||
- [x] Old logs should auto-cleanup (TTL: 30 days)
|
||||
- [x] Existing API endpoint (`GET /api/jobs/{id}/logs`) must continue to work
|
||||
- [x] Frontend must continue to display logs without changes
|
||||
- [x] Migration path for existing jobs (optional - can just start fresh)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Option A: Subcollection (Recommended)
|
||||
Store logs in `jobs/{job_id}/logs/{log_id}` subcollection.
|
||||
|
||||
**Pros:**
|
||||
- Natural association with job (delete job = delete logs with cascading)
|
||||
- Each log entry is its own document (no size limit concerns)
|
||||
- Can add TTL via Firestore TTL policies
|
||||
- Efficient queries with composite indexes
|
||||
|
||||
**Cons:**
|
||||
- More Firestore read operations for paginated log fetching
|
||||
- Slightly more complex write logic
|
||||
|
||||
### Option B: Separate Collection
|
||||
Store logs in `job_logs` collection with `job_id` field.
|
||||
|
||||
**Pros:**
|
||||
- Can query across all jobs
|
||||
- Simpler TTL management
|
||||
|
||||
**Cons:**
|
||||
- Orphaned logs if job deleted (need cleanup logic)
|
||||
- Extra index needed for job_id queries
|
||||
|
||||
### Decision: Option A (Subcollection)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Backend Changes
|
||||
|
||||
1. [x] **Create log entry model** - `backend/models/worker_log.py`
|
||||
- `WorkerLogEntry` dataclass with: id, job_id, timestamp, level, worker, message, metadata, ttl_expiry
|
||||
- Factory method `create()` for easy instantiation
|
||||
- `to_dict()` for Firestore storage
|
||||
- `to_legacy_dict()` for API compatibility
|
||||
- `from_dict()` for Firestore retrieval
|
||||
|
||||
2. [x] **Update FirestoreService** - `backend/services/firestore_service.py`
|
||||
- Added `append_log_to_subcollection(job_id, log_entry)` method
|
||||
- Added `get_logs_from_subcollection(job_id, limit, since_timestamp, worker, offset)` method
|
||||
- Added `get_logs_count_from_subcollection(job_id)` method
|
||||
- Added `delete_logs_subcollection(job_id, batch_size)` method
|
||||
- Old `append_worker_log` marked as deprecated
|
||||
|
||||
3. [x] **Update JobManager** - `backend/services/job_manager.py`
|
||||
- Modified `append_worker_log()` to use subcollection when `USE_LOG_SUBCOLLECTION=true`
|
||||
- Modified `get_worker_logs()` to read from subcollection with fallback to embedded array
|
||||
- Added `get_worker_logs_count()` for total count
|
||||
- Updated `delete_job()` to clean up logs subcollection
|
||||
|
||||
4. [x] **Update jobs API** - `backend/api/routes/jobs.py`
|
||||
- `GET /api/jobs/{id}/logs` - Updated to use new methods
|
||||
- Response format stays the same (backward compatible)
|
||||
- Fixed `next_index` calculation
|
||||
|
||||
5. [x] **Add feature flag** - `backend/config.py`
|
||||
- Added `USE_LOG_SUBCOLLECTION` environment variable (default: true)
|
||||
|
||||
6. [x] **Add TTL configuration** - `infrastructure/__main__.py`
|
||||
- Added Firestore TTL policy for logs subcollection (30 days)
|
||||
- Added composite index for worker + timestamp queries
|
||||
|
||||
### Phase 2: Testing
|
||||
|
||||
7. [x] **Unit tests** - `backend/tests/test_worker_log_subcollection.py`
|
||||
- 24 tests covering:
|
||||
- WorkerLogEntry model creation and conversion
|
||||
- FirestoreService subcollection methods
|
||||
- JobManager logging with feature flag
|
||||
- Edge cases (unicode, long messages, empty messages)
|
||||
|
||||
8. [x] **Integration tests** - `backend/tests/emulator/test_worker_logs_subcollection.py`
|
||||
- 11 tests using real Firestore emulator:
|
||||
- Single and sequential writes
|
||||
- TTL field verification
|
||||
- Concurrent writes (no race conditions)
|
||||
- Large volume (500 logs)
|
||||
- Worker filtering
|
||||
- Timestamp ordering
|
||||
- Subcollection deletion
|
||||
- Concurrent workers interleaved
|
||||
- Job deletion cleanup
|
||||
- Comparison with embedded array
|
||||
|
||||
### Phase 3: Deployment & Migration
|
||||
|
||||
9. [ ] **Deploy with feature flag**
|
||||
- Add `USE_LOG_SUBCOLLECTION=true` env var (default is true)
|
||||
- Deploy to production
|
||||
|
||||
10. [ ] **Monitor**
|
||||
- Watch for errors in Cloud Logging
|
||||
- Verify logs appear in frontend
|
||||
|
||||
11. [x] **Backward compatibility**
|
||||
- Old jobs with embedded arrays still work (fallback in get_worker_logs)
|
||||
- No migration needed - new jobs use subcollection, old jobs use array
|
||||
|
||||
### Phase 4: Cleanup (Future)
|
||||
|
||||
12. [ ] **Remove old code path** (after confidence period)
|
||||
- Remove `worker_logs` field from job model
|
||||
- Remove old `append_worker_log` method
|
||||
- Remove feature flag
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/worker_log.py` | Created | WorkerLogEntry dataclass |
|
||||
| `backend/services/firestore_service.py` | Modified | Added subcollection methods |
|
||||
| `backend/services/job_manager.py` | Modified | Switch to subcollection with fallback |
|
||||
| `backend/api/routes/jobs.py` | Modified | Updated logs endpoint |
|
||||
| `backend/config.py` | Modified | Added USE_LOG_SUBCOLLECTION setting |
|
||||
| `backend/tests/test_worker_log_subcollection.py` | Created | Unit tests (24 tests) |
|
||||
| `backend/tests/emulator/test_worker_logs_subcollection.py` | Created | Integration tests (11 tests) |
|
||||
| `infrastructure/__main__.py` | Modified | Added TTL policy and index |
|
||||
|
||||
## Log Entry Schema
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class WorkerLogEntry:
|
||||
timestamp: datetime
|
||||
level: str # "DEBUG", "INFO", "WARNING", "ERROR"
|
||||
worker: str # "audio", "lyrics", "video", "render", "screens", "distribution"
|
||||
message: str
|
||||
id: str # Auto-generated UUID
|
||||
job_id: str
|
||||
ttl_expiry: datetime # timestamp + 30 days
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
```
|
||||
|
||||
## Firestore Structure
|
||||
|
||||
```
|
||||
jobs/
|
||||
{job_id}/
|
||||
... (job fields, worker_logs still present for old jobs)
|
||||
logs/ # subcollection for new jobs
|
||||
{log_id}/
|
||||
id: string
|
||||
job_id: string
|
||||
timestamp: datetime
|
||||
level: string
|
||||
worker: string
|
||||
message: string
|
||||
metadata: map (optional)
|
||||
ttl_expiry: datetime
|
||||
```
|
||||
|
||||
## API Response (Unchanged)
|
||||
|
||||
```json
|
||||
{
|
||||
"logs": [
|
||||
{
|
||||
"timestamp": "2026-01-04T12:00:00Z",
|
||||
"level": "INFO",
|
||||
"worker": "video",
|
||||
"message": "Starting video generation"
|
||||
}
|
||||
],
|
||||
"next_index": 42,
|
||||
"total_logs": 150
|
||||
}
|
||||
```
|
||||
|
||||
## Testing Results
|
||||
|
||||
All tests pass:
|
||||
- 24 unit tests in `test_worker_log_subcollection.py`
|
||||
- 11 emulator integration tests in `test_worker_logs_subcollection.py`
|
||||
- 68 total tests in emulator suite pass
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. Set `USE_LOG_SUBCOLLECTION=false` env var
|
||||
2. Redeploy backend
|
||||
3. New logs will go back to embedded array
|
||||
4. Subcollection logs will remain but won't be read
|
||||
|
||||
## Answers to Open Questions
|
||||
|
||||
- [x] **Should we paginate logs in the API?** - Yes, added `offset` parameter support
|
||||
- [x] **What's the optimal TTL period?** - 30 days (configurable via DEFAULT_LOG_TTL_DAYS)
|
||||
- [x] **Should we index by timestamp for efficient range queries?** - Yes, added composite index (worker + timestamp)
|
||||
|
||||
## References
|
||||
|
||||
- [Firestore TTL](https://cloud.google.com/firestore/docs/ttl)
|
||||
- [Firestore Subcollections](https://firebase.google.com/docs/firestore/data-model#subcollections)
|
||||
- [Firestore Limits](https://firebase.google.com/docs/firestore/quotas#limits)
|
||||
- [Pulumi Firestore Field](https://www.pulumi.com/registry/packages/gcp/api-docs/firestore/field/)
|
||||
@@ -1,21 +0,0 @@
|
||||
---
|
||||
description: Git branching and PR workflow
|
||||
alwaysApply: true
|
||||
---
|
||||
|
||||
# Branching Workflow
|
||||
|
||||
**Never commit directly to `main`.** Always work on a feature branch.
|
||||
|
||||
Create descriptive branch names like `feature/add-discord-notifications`, `fix/audio-sync-issue`, or `refactor/cleanup-api-routes`.
|
||||
|
||||
# Creating a Pull Request
|
||||
|
||||
Once all changes are complete, tests pass, and commits are pushed, create a PR with:
|
||||
|
||||
1. **Summary** - Brief overview of the changes
|
||||
2. **Changes Made** - List of specific modifications
|
||||
3. **Testing** - How changes were tested (unit tests, manual testing)
|
||||
4. **Related Issues** - Reference any GitHub issues (e.g., "Fixes #123")
|
||||
|
||||
Only merge after review and CI checks pass.
|
||||
@@ -1,31 +0,0 @@
|
||||
---
|
||||
description: Infrastructure as Code requirements using Pulumi
|
||||
alwaysApply: true
|
||||
---
|
||||
|
||||
# Infrastructure Changes
|
||||
|
||||
All Google Cloud infrastructure (IAM, Cloud Storage, Cloud Run, Secret Manager, Artifact Registry, etc.) must be managed via **Pulumi IaC** in the `infrastructure/` folder.
|
||||
|
||||
## Rules
|
||||
|
||||
- **Reading/querying** with `gcloud` CLI is fine for investigation and debugging
|
||||
- **Temporary changes** via `gcloud` CLI or Console are acceptable for urgent fixes or experimentation, but **must be cleaned up** and properly codified in the IaC afterward
|
||||
- **Permanent changes** must go through Pulumi — never leave `gcloud` or Console changes as the final state
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Make changes in `infrastructure/`
|
||||
2. Run `pulumi preview` and verify the output
|
||||
3. Run `pulumi up` to apply
|
||||
|
||||
## Authentication Issues
|
||||
|
||||
If you encounter any authentication issues with `gcloud` or `pulumi` (e.g., expired tokens, permission errors), **stop and notify the user** so they can run the appropriate auth commands in their external terminal:
|
||||
|
||||
```bash
|
||||
gcloud auth login
|
||||
gcloud auth application-default login
|
||||
```
|
||||
|
||||
Do not attempt to work around auth issues or make assumptions about credentials.
|
||||
@@ -1,27 +0,0 @@
|
||||
---
|
||||
description: Version bumping rule for commits
|
||||
alwaysApply: true
|
||||
---
|
||||
|
||||
# Version Management
|
||||
|
||||
When committing and pushing changes, always bump the patch version in `pyproject.toml`.
|
||||
|
||||
The version is located at `tool.poetry.version` in `pyproject.toml`. For example, if the current version is `0.70.3`, bump it to `0.70.4`.
|
||||
|
||||
This ensures every commit pushed to the repository has a unique version number for tracking and deployment purposes.
|
||||
|
||||
# Testing Requirements
|
||||
|
||||
Before committing and pushing changes, ensure all tests pass:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/ backend/tests/ -v -x --ignore=tests/integration
|
||||
```
|
||||
|
||||
Ensure all code touched is thoroughly covered by unit tests, we have a 70% coverage minimum in CI:
|
||||
```bash
|
||||
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=xml --cov-report=term-missing --cov-fail-under=70
|
||||
```
|
||||
|
||||
Do not commit or push code that fails tests. Fix any test failures before proceeding.
|
||||
@@ -0,0 +1,83 @@
|
||||
# .dockerignore - Reduce Docker build context size
|
||||
# This is critical for fast builds as it reduces upload time and cache invalidation
|
||||
|
||||
# Git
|
||||
.git/
|
||||
.gitignore
|
||||
.gitmodules
|
||||
|
||||
# Node.js / Frontend (not needed for backend image)
|
||||
node_modules/
|
||||
frontend/
|
||||
frontend-react/
|
||||
*.log
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
|
||||
# Python virtual environments (use Docker's Python instead)
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
.venv/
|
||||
venv/
|
||||
backend/venv/
|
||||
ENV/
|
||||
env/
|
||||
*.egg-info/
|
||||
.eggs/
|
||||
dist/
|
||||
build/
|
||||
*.egg
|
||||
|
||||
# IDE
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
.DS_Store
|
||||
|
||||
# Testing (not needed in production image)
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
.tox/
|
||||
.nox/
|
||||
coverage.xml
|
||||
*.cover
|
||||
tests/
|
||||
backend/tests/
|
||||
|
||||
# Documentation (not needed in Docker image)
|
||||
docs/
|
||||
*.md
|
||||
!README.md
|
||||
|
||||
# Infrastructure (not needed for Docker build)
|
||||
infrastructure/
|
||||
|
||||
# CI/CD configs
|
||||
.github/
|
||||
|
||||
# Cloud Build configs (keep for reference but not in image)
|
||||
cloudbuild*.yaml
|
||||
|
||||
# Local development files
|
||||
lyrics_transcriber_local/
|
||||
*.local
|
||||
.env
|
||||
.env.*
|
||||
*.bak
|
||||
*.tmp
|
||||
Thumbs.db
|
||||
|
||||
# Large test/sample files
|
||||
tests/data/
|
||||
*.mp3
|
||||
*.wav
|
||||
*.mp4
|
||||
*.mkv
|
||||
*.flac
|
||||
+659
-174
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,173 @@
|
||||
name: E2E Happy Path Test
|
||||
|
||||
# Comprehensive end-to-end test that validates the full karaoke generation journey
|
||||
# using ONLY browser interactions (no API shortcuts)
|
||||
#
|
||||
# See docs/E2E-HAPPY-PATH-TEST-SPEC.md for full requirements
|
||||
|
||||
on:
|
||||
# Daily run at 6 AM UTC
|
||||
schedule:
|
||||
- cron: '0 6 * * *'
|
||||
|
||||
# Manual trigger with optional parameters
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
debug_mode:
|
||||
description: 'Enable debug mode (slower, more screenshots)'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
browser:
|
||||
description: 'Browser to test with'
|
||||
required: false
|
||||
default: 'chromium'
|
||||
type: choice
|
||||
options:
|
||||
- chromium
|
||||
- firefox
|
||||
- webkit
|
||||
use_test_token:
|
||||
description: 'Skip signup and use E2E_TEST_TOKEN (for iterations)'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
e2e-happy-path:
|
||||
name: "Full Happy Path (Real User)"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 90 # 1.5 hours max (test itself is 60 min)
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Install Playwright browsers
|
||||
working-directory: frontend
|
||||
run: |
|
||||
BROWSER=${{ inputs.browser || 'chromium' }}
|
||||
npx playwright install --with-deps $BROWSER
|
||||
|
||||
- name: Create test-results directory
|
||||
working-directory: frontend
|
||||
run: mkdir -p test-results
|
||||
|
||||
- name: Run E2E Happy Path Test
|
||||
working-directory: frontend
|
||||
env:
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
E2E_TEST_TOKEN: ${{ inputs.use_test_token && secrets.E2E_TEST_TOKEN || '' }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
|
||||
run: |
|
||||
echo "Starting E2E Happy Path Test"
|
||||
echo "Browser: ${{ inputs.browser || 'chromium' }}"
|
||||
echo "Debug mode: ${{ inputs.debug_mode || 'false' }}"
|
||||
echo "Use test token: ${{ inputs.use_test_token || 'false' }}"
|
||||
echo "Time: $(date)"
|
||||
echo ""
|
||||
|
||||
# Run the real user happy path test
|
||||
npx playwright test happy-path-real-user.spec.ts \
|
||||
--config=playwright.production.config.ts \
|
||||
--project=${{ inputs.browser || 'chromium' }} \
|
||||
--reporter=list \
|
||||
--timeout=3600000 \
|
||||
2>&1 | tee test-output.log
|
||||
continue-on-error: true
|
||||
|
||||
- name: Check test result
|
||||
id: test_result
|
||||
working-directory: frontend
|
||||
run: |
|
||||
if grep -q "passed" test-output.log && ! grep -q "failed" test-output.log; then
|
||||
echo "result=success" >> $GITHUB_OUTPUT
|
||||
echo "Test PASSED"
|
||||
else
|
||||
echo "result=failure" >> $GITHUB_OUTPUT
|
||||
echo "Test FAILED - see artifacts for details"
|
||||
fi
|
||||
|
||||
- name: Upload test artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-results-${{ github.run_number }}
|
||||
path: |
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
frontend/test-output.log
|
||||
retention-days: 30
|
||||
|
||||
- name: Upload video recordings
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-videos-${{ github.run_number }}
|
||||
path: frontend/test-results/**/*.webm
|
||||
retention-days: 30
|
||||
if-no-files-found: ignore
|
||||
|
||||
- name: Create summary
|
||||
if: always()
|
||||
run: |
|
||||
echo "## E2E Happy Path Test Results" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Result | ${{ steps.test_result.outputs.result || 'unknown' }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
if [ -f frontend/test-output.log ]; then
|
||||
echo "### Test Output (last 100 lines)" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
tail -100 frontend/test-output.log >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
fi
|
||||
|
||||
- name: Send email notification
|
||||
if: always() && github.event_name == 'schedule'
|
||||
uses: dawidd6/action-send-mail@v3
|
||||
with:
|
||||
server_address: smtp.sendgrid.net
|
||||
server_port: 587
|
||||
username: apikey
|
||||
password: ${{ secrets.SENDGRID_API_KEY }}
|
||||
subject: "E2E Happy Path Test ${{ steps.test_result.outputs.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
|
||||
to: andrew@nomadkaraoke.com
|
||||
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
|
||||
body: |
|
||||
E2E Happy Path Test Results
|
||||
===========================
|
||||
|
||||
Status: ${{ steps.test_result.outputs.result == 'success' && 'PASSED ✅' || 'FAILED ❌' }}
|
||||
Run: #${{ github.run_number }}
|
||||
Trigger: ${{ github.event_name }}
|
||||
Time: ${{ github.event.repository.updated_at }}
|
||||
|
||||
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||
|
||||
This is an automated notification from the daily E2E happy path test.
|
||||
|
||||
- name: Fail if test failed
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
run: exit 1
|
||||
+13
-1
@@ -12,6 +12,9 @@ tests/outputs
|
||||
/input
|
||||
/output/
|
||||
|
||||
# Cloud Infra
|
||||
.emulator-logs
|
||||
|
||||
# Next.js / React Frontend (./frontend)
|
||||
node_modules/
|
||||
frontend/node_modules/
|
||||
@@ -41,6 +44,10 @@ frontend/.pnpm-debug.log*
|
||||
|
||||
# Testing
|
||||
frontend/coverage/
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
frontend/playwright/.cache/
|
||||
frontend/e2e/screenshots/
|
||||
|
||||
# TypeScript
|
||||
frontend/*.tsbuildinfo
|
||||
@@ -76,8 +83,9 @@ eggs/
|
||||
# Python lib directories (not frontend)
|
||||
lib/
|
||||
lib64/
|
||||
# But allow frontend lib
|
||||
# But allow frontend lib directories
|
||||
!frontend/lib/
|
||||
!buy-site/lib/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
@@ -220,3 +228,7 @@ cython_debug/
|
||||
# and can be added to the global gitignore or merged into this file. For a more nuclear
|
||||
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
|
||||
#.idea/
|
||||
flacfetch
|
||||
|
||||
# Test output directories
|
||||
Rick Astley*/
|
||||
@@ -0,0 +1,144 @@
|
||||
# Karaoke-Gen AI Assistant Guidelines
|
||||
|
||||
## Project Overview
|
||||
|
||||
**Karaoke video generation platform** - CLI tool and web service that creates professional karaoke videos with synchronized lyrics.
|
||||
|
||||
- **Production**: <https://gen.nomadkaraoke.com> (frontend), <https://api.nomadkaraoke.com> (backend)
|
||||
- **CLI**: `karaoke-gen` (local), `karaoke-gen-remote` (cloud)
|
||||
- **Repo**: <https://github.com/nomadkaraoke/karaoke-gen>
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| What | Where |
|
||||
|------|-------|
|
||||
| Product vision & goals | `docs/PRODUCT-VISION.md` |
|
||||
| Current status | `docs/README.md` |
|
||||
| Architecture | `docs/ARCHITECTURE.md` |
|
||||
| Dev setup & testing | `docs/DEVELOPMENT.md` |
|
||||
| API reference | `docs/API.md` |
|
||||
| Past learnings | `docs/LESSONS-LEARNED.md` |
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **Backend**: FastAPI on Cloud Run, Firestore, GCS, Secret Manager
|
||||
- **Frontend**: Next.js on Cloudflare Pages
|
||||
- **Processing**: Modal (GPU audio separation), AudioShake/Whisper (transcription)
|
||||
- **Infra**: Pulumi IaC, GitHub Actions CI/CD
|
||||
|
||||
## Essential Rules
|
||||
|
||||
### Git Workflow
|
||||
- **Never commit directly to `main`** - use `/new-worktree <description>` to start
|
||||
- **Follow global workflow** - see `~/.claude/CLAUDE.md` for command sequence
|
||||
- Create PR with summary, changes, testing info
|
||||
- Merge only after CI passes
|
||||
|
||||
### Testing (Required Before Commit)
|
||||
```bash
|
||||
# Backend (10 min timeout, limit output)
|
||||
make test 2>&1 | tail -n 500
|
||||
|
||||
# Frontend
|
||||
cd frontend && npm run test:all 2>&1 | tail -n 200
|
||||
```
|
||||
|
||||
### Version Bumping
|
||||
- Bump `tool.poetry.version` in `pyproject.toml` for code changes
|
||||
- Skip for docs-only changes
|
||||
|
||||
### Infrastructure
|
||||
- All GCP changes via Pulumi in `infrastructure/`
|
||||
- `gcloud` CLI for reading/debugging only
|
||||
- Stop and notify user on auth issues
|
||||
|
||||
## API Authentication (for Agents)
|
||||
|
||||
When you need to call backend APIs programmatically:
|
||||
|
||||
### Internal Worker Endpoints
|
||||
Use the `X-Admin-Token` header with the token from Secret Manager:
|
||||
```bash
|
||||
# Trigger video worker
|
||||
curl -X POST "https://api.nomadkaraoke.com/api/internal/workers/video" \
|
||||
-H "X-Admin-Token: $(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke)" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"job_id": "YOUR_JOB_ID"}'
|
||||
|
||||
# Other internal endpoints follow the same pattern
|
||||
```
|
||||
|
||||
### Firestore Direct Access
|
||||
```bash
|
||||
# Set the correct project
|
||||
export GOOGLE_CLOUD_PROJECT=nomadkaraoke
|
||||
|
||||
# Use Python with google-cloud-firestore
|
||||
python3 << 'EOF'
|
||||
import os
|
||||
os.environ['GOOGLE_CLOUD_PROJECT'] = 'nomadkaraoke'
|
||||
from google.cloud import firestore
|
||||
db = firestore.Client(project='nomadkaraoke')
|
||||
# Query/update documents...
|
||||
EOF
|
||||
```
|
||||
|
||||
### GCE Encoding Worker
|
||||
```bash
|
||||
# SSH to restart service (clears in-memory job queue)
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
|
||||
--command="sudo systemctl restart encoding-worker"
|
||||
|
||||
# Check health
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
|
||||
--command="curl -s http://localhost:8080/health"
|
||||
```
|
||||
|
||||
### What Doesn't Work
|
||||
- `gcloud auth print-identity-token` - Wrong token type for internal endpoints
|
||||
- Cloud Tasks without proper service account - Gets 401 from Cloud Run
|
||||
|
||||
## Documentation Maintenance
|
||||
|
||||
### Structure
|
||||
```text
|
||||
docs/
|
||||
├── README.md # Current status + navigation (UPDATE THIS)
|
||||
├── ARCHITECTURE.md # System design
|
||||
├── DEVELOPMENT.md # Dev setup, testing, deployment
|
||||
├── API.md # Backend API reference
|
||||
├── LESSONS-LEARNED.md # Key insights for future agents
|
||||
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
|
||||
```
|
||||
|
||||
### Rules
|
||||
1. **Before merging PRs**: Run `/docs-review` to check if docs need updating
|
||||
2. **Periodically**: Run `/docs-maintain` to verify docs are organized
|
||||
3. **For significant work**: Create `docs/archive/YYYY-MM-DD-topic.md`
|
||||
4. **Add learnings**: Update `docs/LESSONS-LEARNED.md` with insights
|
||||
5. **Keep status current**: Update `docs/README.md` when project state changes
|
||||
|
||||
### What Goes Where
|
||||
- **README.md**: Current status, known issues, quick links
|
||||
- **ARCHITECTURE.md**: System design, data flow, tech decisions
|
||||
- **DEVELOPMENT.md**: Setup, testing, deployment, CI/CD
|
||||
- **API.md**: Endpoints, request/response formats
|
||||
- **LESSONS-LEARNED.md**: What worked, what didn't, gotchas
|
||||
- **archive/**: Completed work, session summaries, old plans
|
||||
|
||||
## Related Projects
|
||||
|
||||
### flacfetch
|
||||
Location: `/Users/andrew/Projects/flacfetch`
|
||||
|
||||
Workflow: Make changes on `main`, bump version, push, wait for CI, then `poetry update flacfetch` in karaoke-gen.
|
||||
|
||||
## Pre-PR Checklist
|
||||
|
||||
Follow global workflow (`~/.claude/CLAUDE.md`):
|
||||
|
||||
- [ ] Tests pass: `/test` (or `make test` / `npm run test:all`)
|
||||
- [ ] Version bumped (if code changed)
|
||||
- [ ] Docs checked: `/docs-review`
|
||||
- [ ] Code reviewed: `/review` (CodeRabbit CLI)
|
||||
- [ ] PR created: `/pr` (adds @coderabbitai ignore)
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2024 Nomad Karaoke LLC
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,77 @@
|
||||
# Makefile for karaoke-gen project
|
||||
# Run `make help` to see available commands
|
||||
|
||||
.PHONY: help test test-unit test-backend test-integration test-e2e test-all lint clean emulators-start emulators-stop
|
||||
|
||||
# Default target
|
||||
help:
|
||||
@echo "Available commands:"
|
||||
@echo " make test - Run all tests (unit + backend + emulator-based)"
|
||||
@echo " make test-unit - Run unit tests only (karaoke_gen)"
|
||||
@echo " make test-backend - Run backend unit tests only"
|
||||
@echo " make test-e2e - Run E2E tests with emulators (starts/stops automatically)"
|
||||
@echo " make test-all - Alias for 'make test'"
|
||||
@echo " make lint - Run linter checks"
|
||||
@echo " make emulators-start - Start GCP emulators for local development"
|
||||
@echo " make emulators-stop - Stop GCP emulators"
|
||||
@echo ""
|
||||
@echo "Before committing, run: make test"
|
||||
|
||||
# Run unit tests for karaoke_gen package
|
||||
test-unit:
|
||||
@echo "=== Running karaoke_gen unit tests ==="
|
||||
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
|
||||
|
||||
# Run backend unit tests
|
||||
test-backend:
|
||||
@echo "=== Running backend unit tests ==="
|
||||
poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
|
||||
|
||||
# Run E2E integration tests with emulators
|
||||
test-e2e:
|
||||
@echo "=== Running E2E integration tests with emulators ==="
|
||||
@./scripts/start-emulators.sh || (echo "Failed to start emulators" && exit 1)
|
||||
@export FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 && \
|
||||
export STORAGE_EMULATOR_HOST=http://127.0.0.1:4443 && \
|
||||
export GOOGLE_CLOUD_PROJECT=test-project && \
|
||||
export GCS_BUCKET_NAME=test-bucket && \
|
||||
export ADMIN_TOKENS=test-admin-token && \
|
||||
export ENVIRONMENT=test && \
|
||||
poetry run pytest backend/tests/emulator/ -v; \
|
||||
TEST_RESULT=$$?; \
|
||||
./scripts/stop-emulators.sh; \
|
||||
exit $$TEST_RESULT
|
||||
|
||||
# Run all tests (use this before committing!)
|
||||
test: test-unit test-backend test-e2e
|
||||
@echo ""
|
||||
@echo "✅ All tests passed! Ready to commit."
|
||||
|
||||
# Alias for test
|
||||
test-all: test
|
||||
|
||||
# Lint checks
|
||||
lint:
|
||||
@echo "=== Running lint checks ==="
|
||||
poetry run ruff check karaoke_gen/ backend/
|
||||
poetry run ruff format --check karaoke_gen/ backend/
|
||||
|
||||
# Start emulators for local development
|
||||
emulators-start:
|
||||
@echo "=== Starting GCP emulators ==="
|
||||
./scripts/start-emulators.sh
|
||||
|
||||
# Stop emulators
|
||||
emulators-stop:
|
||||
@echo "=== Stopping GCP emulators ==="
|
||||
./scripts/stop-emulators.sh
|
||||
|
||||
# Clean up temporary files
|
||||
clean:
|
||||
find . -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
|
||||
find . -type d -name ".pytest_cache" -exec rm -rf {} + 2>/dev/null || true
|
||||
find . -type d -name ".coverage" -exec rm -rf {} + 2>/dev/null || true
|
||||
find . -type f -name "*.pyc" -delete 2>/dev/null || true
|
||||
rm -rf htmlcov/ .coverage coverage.xml 2>/dev/null || true
|
||||
@echo "Cleaned up temporary files"
|
||||
|
||||
@@ -0,0 +1,693 @@
|
||||
# Karaoke Generator 🎶 🎥 🚀
|
||||
|
||||

|
||||

|
||||

|
||||

|
||||
|
||||
Generate professional karaoke videos with instrumental audio and synchronized lyrics. Available as a **local CLI** (`karaoke-gen`) or **cloud-based CLI** (`karaoke-gen-remote`) that offloads processing to Google Cloud.
|
||||
|
||||
## ✨ Two Ways to Generate Karaoke
|
||||
|
||||
### 1. Local CLI (`karaoke-gen`)
|
||||
Run all processing locally on your machine. Requires GPU for optimal audio separation performance.
|
||||
|
||||
```bash
|
||||
karaoke-gen "ABBA" "Waterloo"
|
||||
```
|
||||
|
||||
### 2. Remote CLI (`karaoke-gen-remote`)
|
||||
Offload all processing to a cloud backend. No GPU required - just authenticate and submit jobs.
|
||||
|
||||
```bash
|
||||
karaoke-gen-remote ./song.flac "ABBA" "Waterloo"
|
||||
```
|
||||
|
||||
Both CLIs produce identical outputs: 4K karaoke videos, CDG+MP3 packages, audio stems, and more.
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Features
|
||||
|
||||
### Core Pipeline
|
||||
- **Audio Separation**: AI-powered vocal/instrumental separation using MDX and Demucs models
|
||||
- **Lyrics Transcription**: Word-level timestamps via AudioShake API
|
||||
- **Lyrics Correction**: Match transcription against online lyrics (Genius, Spotify, Musixmatch)
|
||||
- **Human Review**: Interactive UI for correcting lyrics before final render
|
||||
- **Video Rendering**: High-quality 4K karaoke videos with customizable styles
|
||||
- **Multiple Outputs**: MP4 (4K lossless/lossy, 720p), MKV, CDG+MP3, TXT+MP3
|
||||
|
||||
### Distribution Features
|
||||
- **YouTube Upload**: Automatic upload to your YouTube channel
|
||||
- **Dropbox Integration**: Organize output in brand-coded folders
|
||||
- **Google Drive**: Upload to public share folders
|
||||
- **Discord Notifications**: Webhook notifications on completion
|
||||
|
||||
---
|
||||
|
||||
## 📦 Installation
|
||||
|
||||
```bash
|
||||
pip install karaoke-gen
|
||||
```
|
||||
|
||||
This installs both `karaoke-gen` (local) and `karaoke-gen-remote` (cloud) CLIs.
|
||||
|
||||
### Requirements
|
||||
- Python 3.10-3.13
|
||||
- FFmpeg
|
||||
- For local processing: CUDA-capable GPU or Apple Silicon CPU recommended
|
||||
|
||||
### Transcription Provider Setup
|
||||
|
||||
**Transcription is required** for creating karaoke videos with synchronized lyrics. The system needs word-level timing data to display lyrics in sync with the music.
|
||||
|
||||
#### Option 1: AudioShake (Recommended)
|
||||
Commercial service with high-quality transcription. Best for production use.
|
||||
|
||||
```bash
|
||||
export AUDIOSHAKE_API_TOKEN="your_audioshake_token"
|
||||
```
|
||||
|
||||
Get an API key at [https://www.audioshake.ai/](https://www.audioshake.ai/) - business only, at time of writing this.
|
||||
|
||||
#### Option 2: Local Whisper (No Cloud Required)
|
||||
Run Whisper directly on your local machine using whisper-timestamped. Works on CPU, NVIDIA GPU (CUDA), or Apple Silicon.
|
||||
|
||||
```bash
|
||||
# Install with local Whisper support
|
||||
pip install "karaoke-gen[local-whisper]"
|
||||
|
||||
# Optional: Configure model size (tiny, base, small, medium, large)
|
||||
export WHISPER_MODEL_SIZE="medium"
|
||||
|
||||
# Optional: Force specific device (cpu, cuda, mps)
|
||||
export WHISPER_DEVICE="cpu"
|
||||
```
|
||||
|
||||
**Model Size Guide:**
|
||||
| Model | VRAM | Speed | Quality |
|
||||
|-------|------|-------|---------|
|
||||
| tiny | ~1GB | Fast | Lower |
|
||||
| base | ~1GB | Fast | Basic |
|
||||
| small | ~2GB | Medium | Good |
|
||||
| medium | ~5GB | Slower | Better |
|
||||
| large | ~10GB | Slowest | Best |
|
||||
|
||||
**CPU-Only Installation** (no GPU required):
|
||||
```bash
|
||||
# Pre-install CPU-only PyTorch first
|
||||
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cpu
|
||||
pip install "karaoke-gen[local-whisper]"
|
||||
```
|
||||
|
||||
Local Whisper runs automatically as a fallback when no cloud transcription services are configured.
|
||||
|
||||
#### Option 3: Whisper via RunPod
|
||||
Cloud-based alternative using OpenAI's Whisper model on RunPod infrastructure.
|
||||
|
||||
```bash
|
||||
export RUNPOD_API_KEY="your_runpod_key"
|
||||
export WHISPER_RUNPOD_ID="your_whisper_endpoint_id"
|
||||
```
|
||||
|
||||
Set up a Whisper endpoint at [https://www.runpod.io/](https://www.runpod.io/)
|
||||
|
||||
#### Without Transcription (Instrumental Only)
|
||||
If you don't need synchronized lyrics, use the `--skip-lyrics` flag:
|
||||
|
||||
```bash
|
||||
karaoke-gen --skip-lyrics "Artist" "Title"
|
||||
```
|
||||
|
||||
This creates an instrumental-only karaoke video without lyrics overlay.
|
||||
|
||||
> **Note:** See `lyrics_transcriber_temp/README.md` for detailed transcription provider configuration options.
|
||||
|
||||
---
|
||||
|
||||
## 🖥️ Local CLI (`karaoke-gen`)
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
# Generate from local audio file
|
||||
karaoke-gen ./song.mp3 "Artist Name" "Song Title"
|
||||
|
||||
# Search and download audio automatically
|
||||
karaoke-gen "Rick Astley" "Never Gonna Give You Up"
|
||||
|
||||
# Process from YouTube URL
|
||||
karaoke-gen "https://www.youtube.com/watch?v=dQw4w9WgXcQ" "Rick Astley" "Never Gonna Give You Up"
|
||||
```
|
||||
|
||||
### Remote Audio Separation (Optional)
|
||||
|
||||
Offload just the GPU-intensive audio separation to Modal.com while keeping other processing local:
|
||||
|
||||
```bash
|
||||
export AUDIO_SEPARATOR_API_URL="https://USERNAME--audio-separator-api.modal.run"
|
||||
karaoke-gen "Artist" "Title"
|
||||
```
|
||||
|
||||
### Key Options
|
||||
|
||||
```bash
|
||||
# Custom styling
|
||||
karaoke-gen --style_params_json="./styles.json" "Artist" "Title"
|
||||
|
||||
# Generate CDG and TXT packages
|
||||
karaoke-gen --enable_cdg --enable_txt "Artist" "Title"
|
||||
|
||||
# YouTube upload
|
||||
karaoke-gen --enable_youtube_upload --youtube_description_file="./desc.txt" "Artist" "Title"
|
||||
|
||||
# Full production run
|
||||
karaoke-gen \
|
||||
--style_params_json="./branding.json" \
|
||||
--enable_cdg \
|
||||
--enable_txt \
|
||||
--brand_prefix="BRAND" \
|
||||
--enable_youtube_upload \
|
||||
--youtube_description_file="./description.txt" \
|
||||
"Artist" "Title"
|
||||
```
|
||||
|
||||
### Full Options Reference
|
||||
|
||||
```bash
|
||||
karaoke-gen --help
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ☁️ Remote CLI (`karaoke-gen-remote`)
|
||||
|
||||
The remote CLI submits jobs to a Google Cloud backend that handles all processing. You don't need a GPU or any audio processing libraries installed locally.
|
||||
|
||||
### Setup
|
||||
|
||||
1. **Set the backend URL:**
|
||||
```bash
|
||||
export KARAOKE_GEN_URL="https://api.nomadkaraoke.com" # Or your own backend
|
||||
```
|
||||
|
||||
2. **Authenticate with Google Cloud:**
|
||||
```bash
|
||||
gcloud auth login
|
||||
```
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
# Submit a job
|
||||
karaoke-gen-remote ./song.flac "ABBA" "Waterloo"
|
||||
|
||||
# The CLI will:
|
||||
# 1. Upload your audio file
|
||||
# 2. Monitor processing progress
|
||||
# 3. Open lyrics review UI when ready
|
||||
# 4. Prompt for instrumental selection
|
||||
# 5. Download all outputs when complete
|
||||
```
|
||||
|
||||
### Job Management
|
||||
|
||||
```bash
|
||||
# List all jobs
|
||||
karaoke-gen-remote --list
|
||||
|
||||
# Resume monitoring an existing job
|
||||
karaoke-gen-remote --resume abc12345
|
||||
|
||||
# Cancel a running job
|
||||
karaoke-gen-remote --cancel abc12345
|
||||
|
||||
# Delete a job and its files
|
||||
karaoke-gen-remote --delete abc12345
|
||||
```
|
||||
|
||||
### Full Production Run
|
||||
|
||||
```bash
|
||||
karaoke-gen-remote \
|
||||
--style_params_json="./karaoke-styles.json" \
|
||||
--enable_cdg \
|
||||
--enable_txt \
|
||||
--brand_prefix=NOMAD \
|
||||
--enable_youtube_upload \
|
||||
--youtube_description_file="./youtube-description.txt" \
|
||||
./song.flac "Artist" "Title"
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `KARAOKE_GEN_URL` | Backend service URL | Required |
|
||||
| `KARAOKE_GEN_AUTH_TOKEN` | Admin auth token (for protected endpoints) | Optional |
|
||||
| `REVIEW_UI_URL` | Lyrics review UI URL | `https://gen.nomadkaraoke.com/lyrics/` |
|
||||
| `POLL_INTERVAL` | Seconds between status polls | `5` |
|
||||
|
||||
**Note:** The `REVIEW_UI_URL` defaults to the hosted lyrics review UI. For local development, set it to `http://localhost:5173` if you're running the frontend dev server.
|
||||
|
||||
### Authentication
|
||||
|
||||
The backend uses token-based authentication for admin operations (bulk delete, internal worker triggers). For basic job submission and monitoring, authentication is optional.
|
||||
|
||||
**For admin access:**
|
||||
```bash
|
||||
export KARAOKE_GEN_AUTH_TOKEN="your-admin-token"
|
||||
```
|
||||
|
||||
The token must match one of the tokens configured in the backend's `ADMIN_TOKENS` environment variable.
|
||||
|
||||
### Non-Interactive Mode
|
||||
|
||||
For automated/CI usage:
|
||||
|
||||
```bash
|
||||
karaoke-gen-remote -y ./song.flac "Artist" "Title"
|
||||
```
|
||||
|
||||
The `-y` flag auto-accepts default corrections and selects clean instrumental.
|
||||
|
||||
---
|
||||
|
||||
## 🎨 Style Configuration
|
||||
|
||||
Create a `styles.json` file to customize the karaoke video appearance:
|
||||
|
||||
```json
|
||||
{
|
||||
"intro": {
|
||||
"video_duration": 5,
|
||||
"background_image": "/path/to/title-background.png",
|
||||
"font": "/path/to/Font.ttf",
|
||||
"artist_color": "#ffdf6b",
|
||||
"title_color": "#ffffff"
|
||||
},
|
||||
"karaoke": {
|
||||
"background_image": "/path/to/karaoke-background.png",
|
||||
"font_path": "/path/to/Font.ttf"
|
||||
},
|
||||
"end": {
|
||||
"background_image": "/path/to/end-background.png"
|
||||
},
|
||||
"cdg": {
|
||||
"font_path": "/path/to/Font.ttf",
|
||||
"instrumental_background": "/path/to/cdg-background.png"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
When using `karaoke-gen-remote`, all referenced files are automatically uploaded with your job.
|
||||
|
||||
---
|
||||
|
||||
## 📤 Output Files
|
||||
|
||||
A completed job produces:
|
||||
|
||||
```
|
||||
BRAND-1234 - Artist - Title/
|
||||
├── Artist - Title (Final Karaoke Lossless 4k).mp4 # ProRes 4K
|
||||
├── Artist - Title (Final Karaoke Lossless 4k).mkv # FLAC audio 4K
|
||||
├── Artist - Title (Final Karaoke Lossy 4k).mp4 # H.264 4K
|
||||
├── Artist - Title (Final Karaoke Lossy 720p).mp4 # H.264 720p
|
||||
├── Artist - Title (Final Karaoke CDG).zip # CDG+MP3 package
|
||||
├── Artist - Title (Final Karaoke TXT).zip # TXT+MP3 package
|
||||
├── Artist - Title (Karaoke).cdg # Individual CDG
|
||||
├── Artist - Title (Karaoke).mp3 # Karaoke audio
|
||||
├── Artist - Title (Karaoke).lrc # LRC lyrics
|
||||
├── Artist - Title (Karaoke).ass # ASS subtitles
|
||||
├── Artist - Title (Title).mov # Title screen video
|
||||
├── Artist - Title (End).mov # End screen video
|
||||
├── Artist - Title (Instrumental...).flac # Clean instrumental
|
||||
├── Artist - Title (Instrumental +BV...).flac # With backing vocals
|
||||
└── stems/ # All audio stems
|
||||
├── ...Vocals....flac
|
||||
├── ...Bass....flac
|
||||
├── ...Drums....flac
|
||||
└── ...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Deploy Your Own Backend
|
||||
|
||||
The cloud backend runs on Google Cloud Platform using:
|
||||
- **Cloud Run**: Serverless API hosting
|
||||
- **Firestore**: Job state management
|
||||
- **Cloud Storage**: File uploads and outputs
|
||||
- **Modal.com**: GPU-accelerated audio separation
|
||||
- **AudioShake**: Lyrics transcription API
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Google Cloud account with billing enabled
|
||||
- [Pulumi CLI](https://www.pulumi.com/docs/install/)
|
||||
- Modal.com account (for audio separation)
|
||||
- AudioShake API key
|
||||
|
||||
### Infrastructure Setup
|
||||
|
||||
```bash
|
||||
cd infrastructure
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Login to Pulumi
|
||||
pulumi login
|
||||
|
||||
# Create a stack
|
||||
pulumi stack init prod
|
||||
|
||||
# Configure GCP project
|
||||
pulumi config set gcp:project your-project-id
|
||||
pulumi config set gcp:region us-central1
|
||||
|
||||
# Deploy infrastructure
|
||||
pulumi up
|
||||
```
|
||||
|
||||
This creates:
|
||||
- Firestore database
|
||||
- Cloud Storage bucket
|
||||
- Artifact Registry
|
||||
- Service account with IAM roles
|
||||
- Secret Manager secrets (you add values)
|
||||
|
||||
### Add Secret Values
|
||||
|
||||
```bash
|
||||
# AudioShake API key
|
||||
echo -n "your-audioshake-key" | gcloud secrets versions add audioshake-api-key --data-file=-
|
||||
|
||||
# Genius API key
|
||||
echo -n "your-genius-key" | gcloud secrets versions add genius-api-key --data-file=-
|
||||
|
||||
# Modal API URL
|
||||
echo -n "https://your-modal-url" | gcloud secrets versions add audio-separator-api-url --data-file=-
|
||||
|
||||
# YouTube OAuth credentials (JSON)
|
||||
gcloud secrets versions add youtube-oauth-credentials --data-file=./youtube-creds.json
|
||||
|
||||
# Dropbox OAuth credentials (JSON)
|
||||
gcloud secrets versions add dropbox-oauth-credentials --data-file=./dropbox-creds.json
|
||||
|
||||
# Google Drive service account (JSON)
|
||||
gcloud secrets versions add gdrive-service-account --data-file=./gdrive-sa.json
|
||||
```
|
||||
|
||||
### Deploy Cloud Run
|
||||
|
||||
```bash
|
||||
# Build and deploy
|
||||
gcloud builds submit --config=cloudbuild.yaml
|
||||
|
||||
# Get outputs from Pulumi
|
||||
SA_EMAIL=$(pulumi stack output service_account_email)
|
||||
BUCKET_NAME=$(pulumi stack output bucket_name)
|
||||
|
||||
# Deploy Cloud Run service
|
||||
gcloud run deploy karaoke-backend \
|
||||
--image us-central1-docker.pkg.dev/YOUR-PROJECT/karaoke-repo/karaoke-backend:latest \
|
||||
--platform managed \
|
||||
--region us-central1 \
|
||||
--allow-unauthenticated \
|
||||
--service-account $SA_EMAIL \
|
||||
--memory 2Gi \
|
||||
--cpu 2 \
|
||||
--timeout 600 \
|
||||
--set-env-vars="GOOGLE_CLOUD_PROJECT=YOUR-PROJECT,GCS_BUCKET_NAME=$BUCKET_NAME"
|
||||
```
|
||||
|
||||
### Point CLI to Your Backend
|
||||
|
||||
```bash
|
||||
export KARAOKE_GEN_URL="https://your-backend.run.app"
|
||||
karaoke-gen-remote ./song.flac "Artist" "Title"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔌 Backend API Reference
|
||||
|
||||
The backend exposes a REST API for job management.
|
||||
|
||||
### Job Submission
|
||||
|
||||
**POST** `/api/jobs/upload`
|
||||
|
||||
Submit a new karaoke generation job with audio file and options.
|
||||
|
||||
```bash
|
||||
curl -X POST "https://api.example.com/api/jobs/upload" \
|
||||
-F "file=@song.flac" \
|
||||
-F "artist=ABBA" \
|
||||
-F "title=Waterloo" \
|
||||
-F "enable_cdg=true" \
|
||||
-F "enable_txt=true" \
|
||||
-F "brand_prefix=NOMAD" \
|
||||
-F "style_params=@styles.json" \
|
||||
-F "style_karaoke_background=@background.png"
|
||||
```
|
||||
|
||||
### Job Status
|
||||
|
||||
**GET** `/api/jobs/{job_id}`
|
||||
|
||||
Get job status and details.
|
||||
|
||||
```bash
|
||||
curl "https://api.example.com/api/jobs/abc12345"
|
||||
```
|
||||
|
||||
### List Jobs
|
||||
|
||||
**GET** `/api/jobs`
|
||||
|
||||
List all jobs with optional status filter.
|
||||
|
||||
```bash
|
||||
curl "https://api.example.com/api/jobs?status=complete&limit=10"
|
||||
```
|
||||
|
||||
### Cancel Job
|
||||
|
||||
**POST** `/api/jobs/{job_id}/cancel`
|
||||
|
||||
Cancel a running job.
|
||||
|
||||
```bash
|
||||
curl -X POST "https://api.example.com/api/jobs/abc12345/cancel" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"reason": "User cancelled"}'
|
||||
```
|
||||
|
||||
### Delete Job
|
||||
|
||||
**DELETE** `/api/jobs/{job_id}`
|
||||
|
||||
Delete a job and its files.
|
||||
|
||||
```bash
|
||||
curl -X DELETE "https://api.example.com/api/jobs/abc12345?delete_files=true"
|
||||
```
|
||||
|
||||
### Lyrics Review
|
||||
|
||||
**GET** `/api/review/{job_id}/correction-data`
|
||||
|
||||
Get correction data for lyrics review.
|
||||
|
||||
**POST** `/api/review/{job_id}/complete`
|
||||
|
||||
Submit corrected lyrics and trigger video rendering.
|
||||
|
||||
### Instrumental Selection
|
||||
|
||||
**GET** `/api/jobs/{job_id}/instrumental-options`
|
||||
|
||||
Get available instrumental options.
|
||||
|
||||
**POST** `/api/jobs/{job_id}/select-instrumental`
|
||||
|
||||
Submit instrumental selection (clean or with_backing).
|
||||
|
||||
```bash
|
||||
curl -X POST "https://api.example.com/api/jobs/abc12345/select-instrumental" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"selection": "clean"}'
|
||||
```
|
||||
|
||||
### Download Files
|
||||
|
||||
**GET** `/api/jobs/{job_id}/download-urls`
|
||||
|
||||
Get download URLs for all output files.
|
||||
|
||||
**GET** `/api/jobs/{job_id}/download/{category}/{file_key}`
|
||||
|
||||
Stream download a specific file.
|
||||
|
||||
### Health Check
|
||||
|
||||
**GET** `/api/health`
|
||||
|
||||
Check backend health status.
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Troubleshooting
|
||||
|
||||
### "No suitable files found for processing"
|
||||
|
||||
This error occurs during the finalisation step when the `(With Vocals).mkv` file is missing. This file is created during lyrics transcription.
|
||||
|
||||
**Most common cause:** No transcription provider configured.
|
||||
|
||||
**Quick fix:**
|
||||
1. Check if transcription providers are configured:
|
||||
```bash
|
||||
echo $AUDIOSHAKE_API_TOKEN
|
||||
echo $RUNPOD_API_KEY
|
||||
```
|
||||
|
||||
2. If both are empty, set up a provider (see [Transcription Provider Setup](#transcription-provider-setup))
|
||||
|
||||
3. Or use `--skip-lyrics` for instrumental-only karaoke:
|
||||
```bash
|
||||
karaoke-gen --skip-lyrics "Artist" "Title"
|
||||
```
|
||||
|
||||
**Other causes:**
|
||||
- Invalid API credentials - verify your tokens are correct and active
|
||||
- API service unavailable - check service status pages
|
||||
- Network connectivity issues - ensure you can reach the API endpoints
|
||||
- Transcription timeout - try again or use a different provider
|
||||
|
||||
### Transcription Fails Silently
|
||||
|
||||
If karaoke-gen runs without errors but produces no synchronized lyrics:
|
||||
|
||||
1. **Check logs** - Run with `--log_level debug` for detailed output:
|
||||
```bash
|
||||
karaoke-gen --log_level debug "Artist" "Title"
|
||||
```
|
||||
|
||||
2. **Verify environment variables** - Ensure API tokens are exported in your shell:
|
||||
```bash
|
||||
# Check if set
|
||||
printenv | grep -E "(AUDIOSHAKE|RUNPOD|WHISPER)"
|
||||
|
||||
# Set in current session
|
||||
export AUDIOSHAKE_API_TOKEN="your_token"
|
||||
```
|
||||
|
||||
3. **Test API connectivity** - Verify you can reach the transcription service
|
||||
|
||||
### "No lyrics found from any source"
|
||||
|
||||
This warning means no reference lyrics were fetched from online sources (Genius, Spotify, Musixmatch). The transcription will still work, but auto-correction may be less accurate.
|
||||
|
||||
**To fix:**
|
||||
- Set `GENIUS_API_TOKEN` for Genius lyrics
|
||||
- Set `SPOTIFY_COOKIE_SP_DC` for Spotify lyrics
|
||||
- Set `RAPIDAPI_KEY` for Musixmatch lyrics
|
||||
- Or provide lyrics manually with `--lyrics_file /path/to/lyrics.txt`
|
||||
|
||||
### Video Quality Issues
|
||||
|
||||
If the output video has quality problems:
|
||||
- Ensure FFmpeg is properly installed: `ffmpeg -version`
|
||||
- Check available codecs: `ffmpeg -codecs`
|
||||
- For 4K output, ensure sufficient disk space (10GB+ per track)
|
||||
|
||||
### Local Whisper Issues
|
||||
|
||||
#### GPU Out of Memory
|
||||
If you get CUDA out of memory errors:
|
||||
```bash
|
||||
# Use a smaller model
|
||||
export WHISPER_MODEL_SIZE="small" # or "tiny"
|
||||
|
||||
# Or force CPU mode
|
||||
export WHISPER_DEVICE="cpu"
|
||||
```
|
||||
|
||||
#### Slow Transcription on CPU
|
||||
CPU transcription is significantly slower than GPU. For faster processing:
|
||||
- Use a smaller model (`tiny` or `base`)
|
||||
- Consider using cloud transcription (AudioShake or RunPod)
|
||||
- On Apple Silicon, the `small` model offers good speed/quality balance
|
||||
|
||||
#### Model Download Issues
|
||||
Whisper models are downloaded on first use (~1-3GB depending on size). If downloads fail:
|
||||
- Check your internet connection
|
||||
- Set a custom cache directory: `export WHISPER_CACHE_DIR="/path/with/space"`
|
||||
- Models are cached in `~/.cache/whisper/` by default
|
||||
|
||||
#### whisper-timestamped Not Found
|
||||
If you get "whisper-timestamped is not installed":
|
||||
```bash
|
||||
pip install "karaoke-gen[local-whisper]"
|
||||
# Or install directly:
|
||||
pip install whisper-timestamped
|
||||
```
|
||||
|
||||
#### Disabling Local Whisper
|
||||
If you want to disable local Whisper (e.g., to force cloud transcription):
|
||||
```bash
|
||||
export ENABLE_LOCAL_WHISPER="false"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Development
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
pytest tests/ backend/tests/ -v
|
||||
|
||||
# Run only unit tests
|
||||
pytest tests/unit/ -v
|
||||
|
||||
# Run with coverage
|
||||
pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing
|
||||
```
|
||||
|
||||
### Project Structure
|
||||
|
||||
```
|
||||
karaoke-gen/
|
||||
├── karaoke_gen/ # Core CLI package
|
||||
│ ├── utils/
|
||||
│ │ ├── gen_cli.py # Local CLI (karaoke-gen)
|
||||
│ │ └── remote_cli.py # Remote CLI (karaoke-gen-remote)
|
||||
│ ├── karaoke_finalise/ # Video encoding, packaging, distribution
|
||||
│ └── style_loader.py # Unified style configuration
|
||||
├── backend/ # Cloud backend (FastAPI)
|
||||
│ ├── api/routes/ # API endpoints
|
||||
│ ├── workers/ # Background processing workers
|
||||
│ └── services/ # Business logic services
|
||||
├── infrastructure/ # Pulumi IaC for GCP
|
||||
├── docs/ # Documentation
|
||||
└── tests/ # Test suite
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📄 License
|
||||
|
||||
MIT
|
||||
|
||||
---
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
Contributions are welcome! Please see our contributing guidelines.
|
||||
@@ -0,0 +1,75 @@
|
||||
[
|
||||
{
|
||||
"test_name": "Sound-alike error (wurld -> world)",
|
||||
"success": true,
|
||||
"proposals": [
|
||||
{
|
||||
"action": "ReplaceWord",
|
||||
"word_id": null,
|
||||
"word_ids": [
|
||||
"w1"
|
||||
],
|
||||
"replacement_text": "world",
|
||||
"confidence": 0.75,
|
||||
"reason": "Sound-alike error. Reference suggests: 'world'. The transcription 'wurld' is a phonetic misspelling or sound-alike of the word 'world' which appears in both reference sources (Genius and LRCLIB).",
|
||||
"gap_category": "GapCategory.SOUND_ALIKE",
|
||||
"requires_human_review": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"test_name": "Punctuation difference",
|
||||
"success": true,
|
||||
"proposals": [
|
||||
{
|
||||
"action": "NoAction",
|
||||
"word_id": null,
|
||||
"word_ids": [
|
||||
"w2"
|
||||
],
|
||||
"replacement_text": null,
|
||||
"confidence": 0.99,
|
||||
"reason": "Transcription matches at least one reference source. The gap text 'Hello' followed by the context 'is it me you're looking for' matches the Genius reference lyrics exactly, excluding punctuation. According to the guidelines, if any reference source matches the transcription text (ignoring punctuation), it should be classified as NO_ERROR.",
|
||||
"gap_category": "GapCategory.NO_ERROR",
|
||||
"requires_human_review": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"test_name": "Extra filler word",
|
||||
"success": true,
|
||||
"proposals": [
|
||||
{
|
||||
"action": "DeleteWord",
|
||||
"word_id": "w3",
|
||||
"word_ids": null,
|
||||
"replacement_text": null,
|
||||
"confidence": 0.8,
|
||||
"reason": "Extra filler word 'And' at sentence start not in reference. The transcription includes the filler word 'And' at the beginning of the phrase, which is not present in the reference lyrics. This matches the definition of EXTRA_WORDS.",
|
||||
"gap_category": "GapCategory.EXTRA_WORDS",
|
||||
"requires_human_review": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"test_name": "Complex multi-word error",
|
||||
"success": true,
|
||||
"proposals": [
|
||||
{
|
||||
"action": "NoAction",
|
||||
"word_id": null,
|
||||
"word_ids": [
|
||||
"w7",
|
||||
"w8",
|
||||
"w9",
|
||||
"w10"
|
||||
],
|
||||
"replacement_text": null,
|
||||
"confidence": 0.99,
|
||||
"reason": "Transcription matches at least one reference source. The transcription text 'gonna give you up' matches the Genius reference lyrics exactly when ignoring punctuation and capitalization. In context, the full phrase 'never gonna give you up' is correctly transcribed and aligns perfectly with the known lyrics and title of the song.",
|
||||
"gap_category": "GapCategory.NO_ERROR",
|
||||
"requires_human_review": false
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,43 @@
|
||||
# Fast application image - uses pre-built base with all dependencies
|
||||
#
|
||||
# The base image (karaoke-backend-base) contains:
|
||||
# - Python 3.11, ffmpeg, sox, system deps
|
||||
# - All pip dependencies from requirements-frozen.txt (NO code)
|
||||
#
|
||||
# This Dockerfile adds the actual application code on top.
|
||||
# Build time: ~1-2 minutes (deps already cached in base)
|
||||
#
|
||||
# If the base image doesn't exist yet, build it first:
|
||||
# gcloud builds submit --config=cloudbuild-base.yaml --project=nomadkaraoke
|
||||
|
||||
# Use pre-built base image with all dependencies
|
||||
# Override with --build-arg BASE_IMAGE=karaoke-backend-base:local for local testing
|
||||
ARG BASE_IMAGE=us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest
|
||||
FROM ${BASE_IMAGE}
|
||||
|
||||
# Set working directory (should already exist from base)
|
||||
WORKDIR /app
|
||||
|
||||
# Copy backend code (this is the only thing that changes frequently)
|
||||
COPY backend /app/backend
|
||||
|
||||
# Copy and reinstall karaoke_gen from local source
|
||||
# This ensures new modules (like style_loader.py) are included without rebuilding base image
|
||||
# Note: lyrics_transcriber is vendored in lyrics_transcriber_temp/ and bundled via pyproject.toml
|
||||
COPY karaoke_gen /app/karaoke_gen
|
||||
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
|
||||
COPY pyproject.toml README.md LICENSE /app/
|
||||
RUN pip install --no-cache-dir -e .
|
||||
|
||||
# Set environment variables
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
ENV PORT=8080
|
||||
|
||||
# Expose port
|
||||
EXPOSE 8080
|
||||
|
||||
# Run uvicorn directly
|
||||
# Note: Transmission daemon removed - torrent downloads now handled by
|
||||
# dedicated flacfetch VM service (see docs/00-current-plan/TORRENT-SERVICE-PLAN.md)
|
||||
# Using --loop asyncio (instead of uvloop) to support nest_asyncio for remote flacfetch calls
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8080", "--loop", "asyncio"]
|
||||
@@ -0,0 +1,74 @@
|
||||
# Base image with all Python dependencies pre-installed
|
||||
# This image is rebuilt only when poetry.lock changes
|
||||
#
|
||||
# Build with: gcloud builds submit --config=cloudbuild-base.yaml
|
||||
# Or manually: docker build -f backend/Dockerfile.base -t karaoke-backend-base .
|
||||
|
||||
FROM python:3.11-slim
|
||||
|
||||
# Install system dependencies required by karaoke-gen
|
||||
# Note: Transmission daemon removed - torrent downloads now handled by
|
||||
# dedicated flacfetch VM service (see docs/00-current-plan/TORRENT-SERVICE-PLAN.md)
|
||||
RUN apt-get update && apt-get install -y \
|
||||
libsndfile1 \
|
||||
libsox-dev \
|
||||
sox \
|
||||
build-essential \
|
||||
curl \
|
||||
xz-utils \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install comprehensive font support for international lyrics and symbols
|
||||
# Noto fonts provide full Unicode coverage including:
|
||||
# - Latin, Greek, Cyrillic (core)
|
||||
# - CJK: Chinese, Japanese, Korean (~150MB)
|
||||
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
|
||||
# - Color emoji (limited support in libass, but available)
|
||||
# - Musical symbols (♪ ♫ etc.)
|
||||
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
|
||||
RUN apt-get update && apt-get install -y \
|
||||
fonts-noto-core \
|
||||
fonts-noto-cjk \
|
||||
fonts-noto-extra \
|
||||
fonts-noto-color-emoji \
|
||||
fontconfig \
|
||||
&& fc-cache -f \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install optimized static FFmpeg build (7.0.2) instead of Debian package (5.1.8)
|
||||
# John Van Sickle's builds are compiled with better optimizations for x86_64
|
||||
# This provides ~20-40% performance improvement for encoding tasks
|
||||
# Source: https://johnvansickle.com/ffmpeg/
|
||||
RUN curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz -o /tmp/ffmpeg.tar.xz && \
|
||||
tar -xf /tmp/ffmpeg.tar.xz -C /tmp && \
|
||||
cp /tmp/ffmpeg-*-amd64-static/ffmpeg /usr/local/bin/ && \
|
||||
cp /tmp/ffmpeg-*-amd64-static/ffprobe /usr/local/bin/ && \
|
||||
chmod +x /usr/local/bin/ffmpeg /usr/local/bin/ffprobe && \
|
||||
rm -rf /tmp/ffmpeg* && \
|
||||
ffmpeg -version
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy dependency files and source code needed for pip install
|
||||
COPY pyproject.toml README.md LICENSE /app/
|
||||
COPY karaoke_gen /app/karaoke_gen
|
||||
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
|
||||
|
||||
# Install all Python dependencies (the slow part - cached in base image)
|
||||
RUN pip install --no-cache-dir --upgrade pip && \
|
||||
pip install --no-cache-dir -e .
|
||||
|
||||
# Download spaCy language model for lyrics transcription
|
||||
# This must be done at build time, not runtime (Cloud Run network restrictions)
|
||||
RUN python -m spacy download en_core_web_sm
|
||||
|
||||
# Label with build info
|
||||
ARG BUILD_DATE
|
||||
ARG BUILD_ID
|
||||
ARG BASE_CONTENT_HASH
|
||||
LABEL org.opencontainers.image.created="${BUILD_DATE}"
|
||||
LABEL org.opencontainers.image.revision="${BUILD_ID}"
|
||||
LABEL org.opencontainers.image.title="karaoke-backend-base"
|
||||
LABEL org.opencontainers.image.description="Base image with Python dependencies and system tools for karaoke-backend"
|
||||
LABEL base.content.hash="${BASE_CONTENT_HASH}"
|
||||
@@ -0,0 +1,242 @@
|
||||
# Backend Development Setup
|
||||
|
||||
## One-Time Setup
|
||||
|
||||
### 1. Create Virtual Environment
|
||||
|
||||
```bash
|
||||
cd /Users/andrew/Projects/karaoke-gen
|
||||
python3 -m venv backend/venv
|
||||
```
|
||||
|
||||
### 2. Activate Virtual Environment
|
||||
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
```
|
||||
|
||||
### 3. Install Dependencies
|
||||
|
||||
**IMPORTANT:** Use Python 3.12 (not 3.13/3.14 - pydantic has compatibility issues)
|
||||
|
||||
```bash
|
||||
# Backend dependencies
|
||||
pip install -r backend/requirements.txt
|
||||
|
||||
# Install karaoke_gen in development mode
|
||||
pip install -e .
|
||||
|
||||
# Install lyrics_transcriber submodule
|
||||
pip install -e lyrics_transcriber_local/
|
||||
|
||||
# Install Google API client (for YouTube upload)
|
||||
pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib
|
||||
```
|
||||
|
||||
### 4. Add Shell Alias (Optional)
|
||||
|
||||
Add to `~/.zshrc`:
|
||||
|
||||
```bash
|
||||
alias backend='cd /Users/andrew/Projects/karaoke-gen && source backend/venv/bin/activate'
|
||||
```
|
||||
|
||||
Then you can just run:
|
||||
```bash
|
||||
backend
|
||||
# Now you're in the project directory with venv activated!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Daily Workflow
|
||||
|
||||
### Before Making Changes
|
||||
|
||||
```bash
|
||||
# Activate venv
|
||||
source backend/venv/bin/activate
|
||||
|
||||
# Or if you set up the alias:
|
||||
backend
|
||||
```
|
||||
|
||||
### After Making Changes
|
||||
|
||||
```bash
|
||||
# 1. Run full validation (catches import errors!)
|
||||
python3 backend/validate.py
|
||||
|
||||
# 2. If validation passes, deploy
|
||||
./scripts/deploy.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation Options
|
||||
|
||||
### Full Validation (Recommended - Requires venv)
|
||||
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
python3 backend/validate.py
|
||||
```
|
||||
|
||||
**Catches:**
|
||||
- ✅ Syntax errors
|
||||
- ✅ **Import errors** (like the processing_service issue!)
|
||||
- ✅ Configuration issues
|
||||
- ✅ FastAPI app creation problems
|
||||
- ✅ Missing dependencies
|
||||
|
||||
**This would have caught the ModuleNotFoundError!**
|
||||
|
||||
### Quick Check (No venv needed)
|
||||
|
||||
```bash
|
||||
./backend/quick-check.sh
|
||||
```
|
||||
|
||||
**Catches:**
|
||||
- ✅ Syntax errors
|
||||
- ✅ Missing files
|
||||
- ✅ Some obvious import issues
|
||||
|
||||
**Good for quick checks, but won't catch all import errors**
|
||||
|
||||
---
|
||||
|
||||
## Example: Complete Development Session
|
||||
|
||||
```bash
|
||||
# 1. Activate environment
|
||||
backend # or: source backend/venv/bin/activate
|
||||
|
||||
# 2. Make your changes
|
||||
vim backend/api/routes/jobs.py
|
||||
|
||||
# 3. Validate (catches import errors!)
|
||||
python3 backend/validate.py
|
||||
# ✅ All validations passed!
|
||||
|
||||
# 4. Deploy
|
||||
./scripts/deploy.sh
|
||||
|
||||
# 5. Test
|
||||
export BACKEND_URL="https://karaoke-backend-ipzqd2k4yq-uc.a.run.app"
|
||||
export AUTH_TOKEN=$(gcloud auth print-identity-token)
|
||||
curl -H "Authorization: Bearer $AUTH_TOKEN" $BACKEND_URL/api/health
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What About karaoke_gen Imports?
|
||||
|
||||
The backend imports from `karaoke_gen` (the CLI package):
|
||||
|
||||
```python
|
||||
from karaoke_gen.karaoke_gen import KaraokePrep
|
||||
from karaoke_gen.audio_processor import AudioProcessor
|
||||
```
|
||||
|
||||
**Options:**
|
||||
|
||||
### Option 1: Install in development mode (Recommended)
|
||||
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
This makes `karaoke_gen` available in the venv without copying files.
|
||||
|
||||
### Option 2: Use system Python
|
||||
|
||||
Your system Python already has `karaoke_gen` installed, so validation will work even if the venv doesn't have it (Python will fall back to system packages).
|
||||
|
||||
---
|
||||
|
||||
## Running the Backend Locally
|
||||
|
||||
```bash
|
||||
# Activate venv
|
||||
source backend/venv/bin/activate
|
||||
|
||||
# Set required environment variables
|
||||
export GOOGLE_CLOUD_PROJECT="nomadkaraoke"
|
||||
export GCS_BUCKET_NAME="karaoke-gen-storage-nomadkaraoke"
|
||||
|
||||
# Optional: Point to local credentials
|
||||
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"
|
||||
|
||||
# Run the server
|
||||
cd backend
|
||||
uvicorn main:app --reload --port 8080
|
||||
```
|
||||
|
||||
Visit: http://localhost:8080/api/health
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "No module named 'fastapi'"
|
||||
|
||||
**Fix:**
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
pip install -r backend/requirements.txt
|
||||
```
|
||||
|
||||
### "No module named 'karaoke_gen'"
|
||||
|
||||
**Fix:**
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
cd /Users/andrew/Projects/karaoke-gen
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
### Validation passes locally but fails in Cloud Run
|
||||
|
||||
**Check:**
|
||||
1. Is `requirements.txt` up to date?
|
||||
2. Did you add a new dependency without adding it to `requirements.txt`?
|
||||
3. Run: `pip freeze | grep <package-name>`
|
||||
|
||||
---
|
||||
|
||||
## Updating Dependencies
|
||||
|
||||
When you add new packages:
|
||||
|
||||
```bash
|
||||
# 1. Install in venv
|
||||
source backend/venv/bin/activate
|
||||
pip install new-package
|
||||
|
||||
# 2. Update requirements.txt
|
||||
pip freeze > backend/requirements.txt
|
||||
|
||||
# 3. Or manually add just the new package:
|
||||
echo "new-package==1.2.3" >> backend/requirements.txt
|
||||
|
||||
# 4. Validate
|
||||
python3 backend/validate.py
|
||||
|
||||
# 5. Deploy
|
||||
./scripts/deploy.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
**Best practice workflow:**
|
||||
|
||||
1. **One-time setup:** Create venv and install dependencies
|
||||
2. **Daily:** Activate venv when working on backend
|
||||
3. **Before deploy:** Run `python3 backend/validate.py` (catches import errors!)
|
||||
4. **Deploy:** Use `./scripts/deploy.sh`
|
||||
|
||||
**The import error would have been caught if you ran validation in a venv!** ✅
|
||||
Whitespace-only changes.
Whitespace-only changes.
@@ -0,0 +1,457 @@
|
||||
"""
|
||||
FastAPI dependencies for authentication and authorization.
|
||||
"""
|
||||
import logging
|
||||
import secrets
|
||||
from datetime import datetime, timedelta, UTC
|
||||
from typing import Optional, Tuple, Callable, Union
|
||||
from fastapi import Depends, HTTPException, Header, Query, Request, Path
|
||||
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
|
||||
|
||||
from backend.services.auth_service import get_auth_service, UserType, AuthService, AuthResult
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def generate_review_token() -> str:
|
||||
"""Generate a cryptographically secure review token."""
|
||||
return secrets.token_urlsafe(32)
|
||||
|
||||
|
||||
def get_review_token_expiry(hours: int = 24) -> datetime:
|
||||
"""Get expiry time for a review token."""
|
||||
return datetime.now(UTC) + timedelta(hours=hours)
|
||||
|
||||
|
||||
# HTTP Bearer security scheme (Authorization: Bearer <token>)
|
||||
security = HTTPBearer(auto_error=False)
|
||||
|
||||
|
||||
async def get_token_from_request(
|
||||
request: Request,
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None, description="Access token (alternative to Bearer header)")
|
||||
) -> Optional[str]:
|
||||
"""
|
||||
Extract access token from request.
|
||||
|
||||
Supports three methods (in priority order):
|
||||
1. X-Admin-Token header: Used by Cloud Tasks (since OIDC overwrites Authorization)
|
||||
2. Authorization header: Authorization: Bearer <token>
|
||||
3. Query parameter: ?token=<token> (for download links)
|
||||
|
||||
Returns:
|
||||
Token string or None
|
||||
"""
|
||||
# Check X-Admin-Token header first (used by Cloud Tasks)
|
||||
# Cloud Tasks OIDC token overwrites Authorization header, so we use a custom header
|
||||
admin_token = request.headers.get("X-Admin-Token")
|
||||
if admin_token:
|
||||
return admin_token
|
||||
|
||||
# Try Authorization header
|
||||
if credentials:
|
||||
return credentials.credentials
|
||||
|
||||
# Try query parameter
|
||||
if token:
|
||||
return token
|
||||
|
||||
return None
|
||||
|
||||
|
||||
async def require_auth(
|
||||
request: Request,
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None)
|
||||
) -> AuthResult:
|
||||
"""
|
||||
Require authentication for an endpoint.
|
||||
|
||||
Returns:
|
||||
AuthResult with full authentication context including:
|
||||
- is_valid, user_type, remaining_uses, message (backward compatible via tuple unpacking)
|
||||
- user_email: Email of authenticated user (if session/API key auth)
|
||||
- is_admin: Whether user has admin privileges
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if authentication fails
|
||||
"""
|
||||
# Get token from request
|
||||
token_str = await get_token_from_request(request, credentials, token)
|
||||
|
||||
if not token_str:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide token via Authorization header or ?token= parameter"
|
||||
)
|
||||
|
||||
# Validate token using the full method
|
||||
auth_result = auth_service.validate_token_full(token_str)
|
||||
|
||||
# Get request_id from middleware for correlation
|
||||
request_id = getattr(request.state, "request_id", None)
|
||||
|
||||
if not auth_result.is_valid:
|
||||
# Log auth failure with request_id for correlation
|
||||
auth_header = request.headers.get("Authorization", "")
|
||||
logger.warning(
|
||||
"auth_failed",
|
||||
extra={
|
||||
"request_id": request_id,
|
||||
"audit_type": "auth_event",
|
||||
"auth_message": auth_result.message,
|
||||
"token_provided": bool(token_str),
|
||||
"token_length": len(token_str) if token_str else 0,
|
||||
"auth_header_present": bool(auth_header),
|
||||
}
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail=f"Authentication failed: {auth_result.message}"
|
||||
)
|
||||
|
||||
# Log successful auth with request_id for correlation with request audit
|
||||
logger.info(
|
||||
"auth_success",
|
||||
extra={
|
||||
"request_id": request_id,
|
||||
"user_email": auth_result.user_email,
|
||||
"user_type": auth_result.user_type.value if auth_result.user_type else None,
|
||||
"is_admin": auth_result.is_admin,
|
||||
"remaining_uses": auth_result.remaining_uses,
|
||||
"audit_type": "auth_event",
|
||||
}
|
||||
)
|
||||
|
||||
return auth_result
|
||||
|
||||
|
||||
async def require_auth_legacy(
|
||||
request: Request,
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None)
|
||||
) -> Tuple[str, UserType, int]:
|
||||
"""
|
||||
Legacy authentication dependency that returns tuple.
|
||||
|
||||
DEPRECATED: Use require_auth which returns AuthResult instead.
|
||||
|
||||
Returns:
|
||||
(token, user_type, remaining_uses)
|
||||
"""
|
||||
token_str = await get_token_from_request(request, credentials, token)
|
||||
|
||||
if not token_str:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide token via Authorization header or ?token= parameter"
|
||||
)
|
||||
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(token_str)
|
||||
|
||||
if not is_valid:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail=f"Authentication failed: {message}"
|
||||
)
|
||||
|
||||
return token_str, user_type, remaining_uses
|
||||
|
||||
|
||||
async def require_admin(
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> AuthResult:
|
||||
"""
|
||||
Require admin access for an endpoint.
|
||||
|
||||
Admin access is granted if:
|
||||
- Using an admin token from ADMIN_TOKENS env var
|
||||
- User email is from admin domain (e.g., @nomadkaraoke.com)
|
||||
- User role is set to ADMIN in database
|
||||
|
||||
Returns:
|
||||
AuthResult with admin privileges
|
||||
|
||||
Raises:
|
||||
HTTPException: 403 if user is not admin
|
||||
"""
|
||||
if not auth_result.is_admin:
|
||||
logger.warning(
|
||||
f"Admin access denied for {auth_result.user_type} user "
|
||||
f"(email: {auth_result.user_email or 'unknown'})"
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="Admin access required"
|
||||
)
|
||||
|
||||
return auth_result
|
||||
|
||||
|
||||
def optional_auth(
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None)
|
||||
) -> Optional[Tuple[str, UserType, int]]:
|
||||
"""
|
||||
Optional authentication - doesn't fail if no token provided.
|
||||
|
||||
Useful for endpoints that have different behavior for authenticated users.
|
||||
|
||||
Returns:
|
||||
(token, user_type, remaining_uses) if authenticated, None otherwise
|
||||
"""
|
||||
# Get token
|
||||
token_str = None
|
||||
if credentials:
|
||||
token_str = credentials.credentials
|
||||
elif token:
|
||||
token_str = token
|
||||
|
||||
if not token_str:
|
||||
return None
|
||||
|
||||
# Validate token
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(token_str)
|
||||
|
||||
if not is_valid:
|
||||
logger.debug(f"Optional auth failed: {message}")
|
||||
return None
|
||||
|
||||
return token_str, user_type, remaining_uses
|
||||
|
||||
|
||||
def require_review_auth_factory(job_id_param: str = "job_id"):
|
||||
"""
|
||||
Factory to create a review authentication dependency.
|
||||
|
||||
Accepts either:
|
||||
1. Full user authentication (admin/user token) - also validates job ownership
|
||||
2. Job-specific review token (only valid for the specific job)
|
||||
|
||||
Args:
|
||||
job_id_param: Name of the path parameter containing the job ID
|
||||
|
||||
Returns:
|
||||
Dependency function that validates review access
|
||||
"""
|
||||
async def require_review_auth(
|
||||
request: Request,
|
||||
job_id: str = Path(...),
|
||||
review_token: Optional[str] = Query(None, description="Job-specific review token"),
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None, alias="token", description="Full access token")
|
||||
) -> Tuple[str, str]:
|
||||
"""
|
||||
Validate review access for a job.
|
||||
|
||||
Returns:
|
||||
(job_id, auth_type) where auth_type is "full" or "review_token"
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if authentication fails
|
||||
HTTPException: 403 if user doesn't own the job (for full auth)
|
||||
"""
|
||||
# Import here to avoid circular dependency
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
job_manager = JobManager()
|
||||
|
||||
# Try full authentication first
|
||||
full_token = None
|
||||
if credentials:
|
||||
full_token = credentials.credentials
|
||||
elif token:
|
||||
full_token = token
|
||||
|
||||
if full_token:
|
||||
auth_result = auth_service.validate_token_full(full_token)
|
||||
if auth_result.is_valid:
|
||||
# For full auth, also verify job ownership
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Check ownership: admin can access all, users only their own jobs
|
||||
if not auth_result.is_admin:
|
||||
if auth_result.user_email and job.user_email:
|
||||
if auth_result.user_email.lower() != job.user_email.lower():
|
||||
logger.warning(
|
||||
f"Review access denied: user {auth_result.user_email} "
|
||||
f"tried to access job {job_id} owned by {job.user_email}"
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="You don't have permission to access this job's review"
|
||||
)
|
||||
elif job.user_email:
|
||||
# Token auth without email trying to access a job with owner
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="You don't have permission to access this job's review"
|
||||
)
|
||||
|
||||
logger.info(f"Review access granted via full auth ({auth_result.user_type}) for job {job_id}")
|
||||
return job_id, "full"
|
||||
|
||||
# Try review token
|
||||
if review_token:
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Validate review token matches
|
||||
if job.review_token and secrets.compare_digest(job.review_token, review_token):
|
||||
# Check expiry if set
|
||||
if job.review_token_expires_at:
|
||||
now = datetime.now(UTC)
|
||||
# Handle timezone-naive datetimes from Firestore
|
||||
expiry = job.review_token_expires_at
|
||||
if expiry.tzinfo is None:
|
||||
expiry = expiry.replace(tzinfo=UTC)
|
||||
|
||||
if now > expiry:
|
||||
logger.warning(f"Review token expired for job {job_id}")
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Review token has expired. Please request a new review link."
|
||||
)
|
||||
|
||||
logger.info(f"Review access granted via review_token for job {job_id}")
|
||||
return job_id, "review_token"
|
||||
else:
|
||||
logger.warning(f"Invalid review token for job {job_id}")
|
||||
|
||||
# No valid authentication
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide either a full access token or a valid review_token for this job."
|
||||
)
|
||||
|
||||
return require_review_auth
|
||||
|
||||
|
||||
# Default instance for most review endpoints
|
||||
require_review_auth = require_review_auth_factory()
|
||||
|
||||
|
||||
def require_instrumental_auth_factory(job_id_param: str = "job_id"):
|
||||
"""
|
||||
Factory to create an instrumental review authentication dependency.
|
||||
|
||||
Accepts either:
|
||||
1. Full user authentication (admin/user token) - also validates job ownership
|
||||
2. Job-specific instrumental token (only valid for the specific job)
|
||||
|
||||
Args:
|
||||
job_id_param: Name of the path parameter containing the job ID
|
||||
|
||||
Returns:
|
||||
Dependency function that validates instrumental review access
|
||||
"""
|
||||
async def require_instrumental_auth(
|
||||
request: Request,
|
||||
job_id: str = Path(...),
|
||||
instrumental_token: Optional[str] = Query(None, description="Job-specific instrumental review token"),
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None, alias="token", description="Full access token")
|
||||
) -> Tuple[str, str]:
|
||||
"""
|
||||
Validate instrumental review access for a job.
|
||||
|
||||
Returns:
|
||||
(job_id, auth_type) where auth_type is "full" or "instrumental_token"
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if authentication fails
|
||||
HTTPException: 403 if user doesn't own the job (for full auth)
|
||||
"""
|
||||
# Import here to avoid circular dependency
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
job_manager = JobManager()
|
||||
|
||||
# Try full authentication first
|
||||
full_token = None
|
||||
if credentials:
|
||||
full_token = credentials.credentials
|
||||
elif token:
|
||||
full_token = token
|
||||
|
||||
if full_token:
|
||||
auth_result = auth_service.validate_token_full(full_token)
|
||||
if auth_result.is_valid:
|
||||
# For full auth, also verify job ownership
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Check ownership: admin can access all, users only their own jobs
|
||||
if not auth_result.is_admin:
|
||||
if auth_result.user_email and job.user_email:
|
||||
if auth_result.user_email.lower() != job.user_email.lower():
|
||||
logger.warning(
|
||||
f"Instrumental access denied: user {auth_result.user_email} "
|
||||
f"tried to access job {job_id} owned by {job.user_email}"
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="You don't have permission to access this job's instrumental review"
|
||||
)
|
||||
elif job.user_email:
|
||||
# Token auth without email trying to access a job with owner
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="You don't have permission to access this job's instrumental review"
|
||||
)
|
||||
|
||||
logger.info(f"Instrumental access granted via full auth ({auth_result.user_type}) for job {job_id}")
|
||||
return job_id, "full"
|
||||
|
||||
# Try instrumental token
|
||||
if instrumental_token:
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Validate instrumental token matches
|
||||
if job.instrumental_token and secrets.compare_digest(job.instrumental_token, instrumental_token):
|
||||
# Check expiry if set
|
||||
if job.instrumental_token_expires_at:
|
||||
now = datetime.now(UTC)
|
||||
# Handle timezone-naive datetimes from Firestore
|
||||
expiry = job.instrumental_token_expires_at
|
||||
if expiry.tzinfo is None:
|
||||
expiry = expiry.replace(tzinfo=UTC)
|
||||
|
||||
if now > expiry:
|
||||
logger.warning(f"Instrumental token expired for job {job_id}")
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Instrumental review token has expired. Please request a new review link."
|
||||
)
|
||||
|
||||
logger.info(f"Instrumental access granted via instrumental_token for job {job_id}")
|
||||
return job_id, "instrumental_token"
|
||||
else:
|
||||
logger.warning(f"Invalid instrumental token for job {job_id}")
|
||||
|
||||
# No valid authentication
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide either a full access token or a valid instrumental_token for this job."
|
||||
)
|
||||
|
||||
return require_instrumental_auth
|
||||
|
||||
|
||||
# Default instance for most instrumental review endpoints
|
||||
require_instrumental_auth = require_instrumental_auth_factory()
|
||||
Whitespace-only changes.
@@ -0,0 +1,582 @@
|
||||
"""
|
||||
Admin API routes for dashboard statistics and management.
|
||||
|
||||
Handles:
|
||||
- Dashboard overview statistics
|
||||
- System-wide metrics
|
||||
- Admin-only operations
|
||||
- Audio search cache management
|
||||
"""
|
||||
import logging
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Tuple, List, Optional, Any, Dict
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.user_service import get_user_service, UserService, USERS_COLLECTION
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.flacfetch_client import get_flacfetch_client, FlacfetchServiceError
|
||||
from backend.models.job import JobStatus
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/admin", tags=["admin"])
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Response Models
|
||||
# =============================================================================
|
||||
|
||||
class JobsByStatusResponse(BaseModel):
|
||||
"""Breakdown of jobs by status."""
|
||||
pending: int = 0
|
||||
processing: int = 0
|
||||
awaiting_review: int = 0
|
||||
awaiting_instrumental: int = 0
|
||||
complete: int = 0
|
||||
failed: int = 0
|
||||
cancelled: int = 0
|
||||
|
||||
|
||||
class AdminStatsOverview(BaseModel):
|
||||
"""Overview statistics for admin dashboard."""
|
||||
total_users: int
|
||||
active_users_7d: int
|
||||
active_users_30d: int
|
||||
total_jobs: int
|
||||
jobs_last_7d: int
|
||||
jobs_last_30d: int
|
||||
jobs_by_status: JobsByStatusResponse
|
||||
total_credits_issued_30d: int
|
||||
total_beta_testers: int
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Admin Stats Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/stats/overview", response_model=AdminStatsOverview)
|
||||
async def get_admin_stats_overview(
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
):
|
||||
"""
|
||||
Get overview statistics for admin dashboard.
|
||||
|
||||
Includes:
|
||||
- User counts (total, active in 7d, active in 30d)
|
||||
- Job counts (total, by status, recent)
|
||||
- Credit statistics
|
||||
- Beta program stats
|
||||
"""
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
from google.cloud.firestore_v1 import aggregation
|
||||
|
||||
db = user_service.db
|
||||
now = datetime.utcnow()
|
||||
seven_days_ago = now - timedelta(days=7)
|
||||
thirty_days_ago = now - timedelta(days=30)
|
||||
|
||||
# Helper function to get count using aggregation
|
||||
def get_count(query) -> int:
|
||||
try:
|
||||
agg_query = aggregation.AggregationQuery(query)
|
||||
agg_query.count(alias="count")
|
||||
results = agg_query.get()
|
||||
return results[0][0].value if results else 0
|
||||
except Exception as e:
|
||||
logger.warning(f"Aggregation query failed: {e}")
|
||||
return 0
|
||||
|
||||
# User statistics
|
||||
users_collection = db.collection(USERS_COLLECTION)
|
||||
|
||||
total_users = get_count(users_collection)
|
||||
|
||||
active_users_7d = get_count(
|
||||
users_collection.where(filter=FieldFilter("last_login_at", ">=", seven_days_ago))
|
||||
)
|
||||
|
||||
active_users_30d = get_count(
|
||||
users_collection.where(filter=FieldFilter("last_login_at", ">=", thirty_days_ago))
|
||||
)
|
||||
|
||||
total_beta_testers = get_count(
|
||||
users_collection.where(filter=FieldFilter("is_beta_tester", "==", True))
|
||||
)
|
||||
|
||||
# Job statistics
|
||||
jobs_collection = db.collection("jobs")
|
||||
|
||||
total_jobs = get_count(jobs_collection)
|
||||
|
||||
jobs_last_7d = get_count(
|
||||
jobs_collection.where(filter=FieldFilter("created_at", ">=", seven_days_ago))
|
||||
)
|
||||
|
||||
jobs_last_30d = get_count(
|
||||
jobs_collection.where(filter=FieldFilter("created_at", ">=", thirty_days_ago))
|
||||
)
|
||||
|
||||
# Jobs by status - map multiple statuses to simplified categories
|
||||
processing_statuses = [
|
||||
"downloading", "downloading_audio", "searching_audio", "awaiting_audio_selection",
|
||||
"separating_stage1", "separating_stage2", "transcribing", "correcting",
|
||||
"generating_screens", "applying_padding", "rendering_video",
|
||||
"instrumental_selected", "generating_video", "encoding", "packaging",
|
||||
"uploading", "notifying"
|
||||
]
|
||||
|
||||
jobs_by_status = JobsByStatusResponse(
|
||||
pending=get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "pending"))
|
||||
),
|
||||
processing=sum(
|
||||
get_count(jobs_collection.where(filter=FieldFilter("status", "==", status)))
|
||||
for status in processing_statuses
|
||||
),
|
||||
awaiting_review=get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "awaiting_review"))
|
||||
) + get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "in_review"))
|
||||
),
|
||||
awaiting_instrumental=get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "awaiting_instrumental_selection"))
|
||||
),
|
||||
complete=get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "complete"))
|
||||
) + get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "prep_complete"))
|
||||
),
|
||||
failed=get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "failed"))
|
||||
),
|
||||
cancelled=get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "cancelled"))
|
||||
),
|
||||
)
|
||||
|
||||
# Credit statistics - sum credits added in last 30 days
|
||||
# This is more expensive, so we'll just estimate from users
|
||||
total_credits_issued_30d = 0
|
||||
try:
|
||||
# Get all users and sum recent credit transactions
|
||||
users_docs = users_collection.limit(500).stream()
|
||||
for user_doc in users_docs:
|
||||
user_data = user_doc.to_dict()
|
||||
transactions = user_data.get("credit_transactions", [])
|
||||
for txn in transactions:
|
||||
txn_date = txn.get("created_at")
|
||||
if txn_date:
|
||||
# Handle both datetime and string formats
|
||||
if isinstance(txn_date, str):
|
||||
try:
|
||||
txn_date = datetime.fromisoformat(txn_date.replace("Z", "+00:00"))
|
||||
except Exception:
|
||||
continue
|
||||
if isinstance(txn_date, datetime):
|
||||
txn_date = txn_date.replace(tzinfo=None)
|
||||
else:
|
||||
continue
|
||||
if txn_date >= thirty_days_ago:
|
||||
amount = txn.get("amount", 0)
|
||||
if amount > 0: # Only count additions, not deductions
|
||||
total_credits_issued_30d += amount
|
||||
except Exception as e:
|
||||
logger.warning(f"Error calculating credits: {e}")
|
||||
|
||||
return AdminStatsOverview(
|
||||
total_users=total_users,
|
||||
active_users_7d=active_users_7d,
|
||||
active_users_30d=active_users_30d,
|
||||
total_jobs=total_jobs,
|
||||
jobs_last_7d=jobs_last_7d,
|
||||
jobs_last_30d=jobs_last_30d,
|
||||
jobs_by_status=jobs_by_status,
|
||||
total_credits_issued_30d=total_credits_issued_30d,
|
||||
total_beta_testers=total_beta_testers,
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Audio Search Management Models
|
||||
# =============================================================================
|
||||
|
||||
class AudioSearchResultSummary(BaseModel):
|
||||
"""Summary of a single audio search result."""
|
||||
index: int
|
||||
provider: str
|
||||
artist: str
|
||||
title: str
|
||||
is_lossless: bool
|
||||
quality: Optional[str] = None
|
||||
seeders: Optional[int] = None
|
||||
|
||||
|
||||
class AudioSearchJobSummary(BaseModel):
|
||||
"""Summary of a job with audio search results."""
|
||||
job_id: str
|
||||
status: str
|
||||
user_email: Optional[str] = None
|
||||
audio_search_artist: Optional[str] = None
|
||||
audio_search_title: Optional[str] = None
|
||||
created_at: Optional[datetime] = None
|
||||
results_count: int
|
||||
results_summary: List[AudioSearchResultSummary]
|
||||
has_lossless: bool
|
||||
providers: List[str]
|
||||
|
||||
|
||||
class AudioSearchListResponse(BaseModel):
|
||||
"""Response for listing audio search jobs."""
|
||||
jobs: List[AudioSearchJobSummary]
|
||||
total: int
|
||||
|
||||
|
||||
class ClearSearchCacheResponse(BaseModel):
|
||||
"""Response for clearing search cache."""
|
||||
status: str
|
||||
job_id: str
|
||||
message: str
|
||||
previous_status: str
|
||||
new_status: str
|
||||
results_cleared: int
|
||||
flacfetch_cache_cleared: bool = False
|
||||
flacfetch_error: Optional[str] = None
|
||||
|
||||
|
||||
class ClearAllCacheResponse(BaseModel):
|
||||
"""Response for clearing all flacfetch cache."""
|
||||
status: str
|
||||
message: str
|
||||
deleted_count: int
|
||||
|
||||
|
||||
class CacheStatsResponse(BaseModel):
|
||||
"""Response for cache statistics."""
|
||||
count: int
|
||||
total_size_bytes: int
|
||||
oldest_entry: Optional[str] = None
|
||||
newest_entry: Optional[str] = None
|
||||
configured: bool
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Audio Search Management Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/audio-searches", response_model=AudioSearchListResponse)
|
||||
async def list_audio_searches(
|
||||
limit: int = 50,
|
||||
status_filter: Optional[str] = None,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
):
|
||||
"""
|
||||
List jobs with audio search results.
|
||||
|
||||
Returns jobs that have cached audio search results, useful for:
|
||||
- Monitoring search activity
|
||||
- Identifying stale cached results
|
||||
- Clearing cache for specific jobs
|
||||
|
||||
Args:
|
||||
limit: Maximum number of jobs to return (default 50)
|
||||
status_filter: Optional filter by job status (e.g., 'awaiting_audio_selection')
|
||||
"""
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
|
||||
db = user_service.db
|
||||
jobs_collection = db.collection("jobs")
|
||||
|
||||
# Query jobs - we'll filter for those with audio_search_results in Python
|
||||
# since Firestore can't query for existence of nested fields efficiently
|
||||
query = jobs_collection.order_by("created_at", direction="DESCENDING").limit(500)
|
||||
|
||||
if status_filter:
|
||||
query = jobs_collection.where(
|
||||
filter=FieldFilter("status", "==", status_filter)
|
||||
).order_by("created_at", direction="DESCENDING").limit(500)
|
||||
|
||||
jobs_with_searches = []
|
||||
|
||||
for doc in query.stream():
|
||||
data = doc.to_dict()
|
||||
state_data = data.get("state_data", {})
|
||||
audio_results = state_data.get("audio_search_results", [])
|
||||
|
||||
if not audio_results:
|
||||
continue
|
||||
|
||||
# Compute has_lossless and providers from ALL results (not just first 10)
|
||||
has_lossless = any(r.get("is_lossless", False) for r in audio_results)
|
||||
providers = {r.get("provider", "Unknown") for r in audio_results}
|
||||
|
||||
# Build summary from first 10 results only
|
||||
results_summary = []
|
||||
for r in audio_results[:10]:
|
||||
results_summary.append(AudioSearchResultSummary(
|
||||
index=r.get("index", 0),
|
||||
provider=r.get("provider", "Unknown"),
|
||||
artist=r.get("artist", ""),
|
||||
title=r.get("title", ""),
|
||||
is_lossless=r.get("is_lossless", False),
|
||||
quality=r.get("quality"),
|
||||
seeders=r.get("seeders"),
|
||||
))
|
||||
|
||||
jobs_with_searches.append(AudioSearchJobSummary(
|
||||
job_id=doc.id,
|
||||
status=data.get("status", "unknown"),
|
||||
user_email=data.get("user_email"),
|
||||
audio_search_artist=data.get("audio_search_artist"),
|
||||
audio_search_title=data.get("audio_search_title"),
|
||||
created_at=data.get("created_at"),
|
||||
results_count=len(audio_results),
|
||||
results_summary=results_summary,
|
||||
has_lossless=has_lossless,
|
||||
providers=sorted(providers),
|
||||
))
|
||||
|
||||
if len(jobs_with_searches) >= limit:
|
||||
break
|
||||
|
||||
return AudioSearchListResponse(
|
||||
jobs=jobs_with_searches,
|
||||
total=len(jobs_with_searches),
|
||||
)
|
||||
|
||||
|
||||
@router.post("/audio-searches/{job_id}/clear-cache", response_model=ClearSearchCacheResponse)
|
||||
async def clear_audio_search_cache(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
):
|
||||
"""
|
||||
Clear the cached audio search results for a job.
|
||||
|
||||
This will:
|
||||
1. Remove the cached search results from job.state_data
|
||||
2. Reset the job status to 'pending' so a new search can be performed
|
||||
3. Clear the flacfetch GCS cache for this artist/title (if available)
|
||||
|
||||
Use this when:
|
||||
- Cached results are stale (e.g., flacfetch was updated)
|
||||
- User wants to search again with different terms
|
||||
- Results appear incomplete or incorrect
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Get current state
|
||||
state_data = job.state_data or {}
|
||||
audio_results = state_data.get("audio_search_results", [])
|
||||
results_count = len(audio_results)
|
||||
previous_status = job.status
|
||||
|
||||
if not audio_results:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job {job_id} has no cached audio search results"
|
||||
)
|
||||
|
||||
# Validate job status - only allow cache clear for appropriate states
|
||||
# Don't allow clearing cache for jobs that are actively processing or complete
|
||||
forbidden_statuses = {
|
||||
"downloading", "downloading_audio", "searching_audio",
|
||||
"separating_stage1", "separating_stage2", "transcribing", "correcting",
|
||||
"generating_screens", "applying_padding", "rendering_video",
|
||||
"generating_video", "encoding", "packaging", "uploading",
|
||||
"complete", "prep_complete",
|
||||
}
|
||||
if previous_status in forbidden_statuses:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Cannot clear cache for job in '{previous_status}' state. "
|
||||
f"Only jobs in pending, awaiting_audio_selection, failed, or cancelled states can have cache cleared."
|
||||
)
|
||||
|
||||
# Clear the cache by removing the keys
|
||||
db = user_service.db
|
||||
job_ref = db.collection("jobs").document(job_id)
|
||||
|
||||
from google.cloud.firestore_v1 import DELETE_FIELD
|
||||
|
||||
# Update job: clear cache and reset status
|
||||
job_ref.update({
|
||||
"state_data.audio_search_results": DELETE_FIELD,
|
||||
"state_data.audio_search_count": DELETE_FIELD,
|
||||
"state_data.remote_search_id": DELETE_FIELD,
|
||||
"status": "pending",
|
||||
"progress": 0,
|
||||
"message": "Audio search cache cleared by admin. Ready for new search.",
|
||||
"updated_at": datetime.utcnow(),
|
||||
})
|
||||
|
||||
# Also clear flacfetch's GCS cache if we have artist/title
|
||||
flacfetch_cache_cleared = False
|
||||
flacfetch_error = None
|
||||
artist = job.audio_search_artist
|
||||
title = job.audio_search_title
|
||||
|
||||
if artist and title:
|
||||
flacfetch_client = get_flacfetch_client()
|
||||
if flacfetch_client:
|
||||
try:
|
||||
flacfetch_cache_cleared = await flacfetch_client.clear_search_cache(artist, title)
|
||||
logger.info(
|
||||
f"Cleared flacfetch cache for '{artist}' - '{title}': "
|
||||
f"{'deleted' if flacfetch_cache_cleared else 'no entry found'}"
|
||||
)
|
||||
except FlacfetchServiceError as e:
|
||||
flacfetch_error = str(e)
|
||||
logger.warning(f"Failed to clear flacfetch cache: {e}")
|
||||
else:
|
||||
flacfetch_error = "flacfetch client not configured"
|
||||
logger.debug("Skipping flacfetch cache clear - client not configured")
|
||||
else:
|
||||
flacfetch_error = "missing artist or title"
|
||||
logger.debug(f"Skipping flacfetch cache clear - missing artist ({artist}) or title ({title})")
|
||||
|
||||
logger.info(
|
||||
f"Admin {auth_data[0]} cleared audio search cache for job {job_id}. "
|
||||
f"Cleared {results_count} results. Status changed from {previous_status} to pending. "
|
||||
f"Flacfetch cache cleared: {flacfetch_cache_cleared}"
|
||||
)
|
||||
|
||||
message = f"Cleared {results_count} cached search results. Job reset to pending."
|
||||
if flacfetch_cache_cleared:
|
||||
message += " Flacfetch cache also cleared."
|
||||
elif flacfetch_error:
|
||||
message += f" Note: flacfetch cache not cleared ({flacfetch_error})."
|
||||
|
||||
return ClearSearchCacheResponse(
|
||||
status="success",
|
||||
job_id=job_id,
|
||||
message=message,
|
||||
previous_status=previous_status,
|
||||
new_status="pending",
|
||||
results_cleared=results_count,
|
||||
flacfetch_cache_cleared=flacfetch_cache_cleared,
|
||||
flacfetch_error=flacfetch_error,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/reset-worker-state")
|
||||
async def reset_worker_state(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Reset stale worker progress state for a job.
|
||||
|
||||
This clears the video_progress, render_progress, and screens_progress
|
||||
from state_data, allowing workers to be re-triggered.
|
||||
|
||||
Use this when a job is stuck because worker progress shows 'running'
|
||||
from a previous failed attempt.
|
||||
"""
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Reset worker progress states
|
||||
job_manager.update_state_data(job_id, 'video_progress', {'stage': 'pending'})
|
||||
job_manager.update_state_data(job_id, 'render_progress', {'stage': 'pending'})
|
||||
job_manager.update_state_data(job_id, 'screens_progress', {'stage': 'pending'})
|
||||
|
||||
logger.info(f"Admin {auth_data[0]} reset worker state for job {job_id}")
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"job_id": job_id,
|
||||
"message": "Worker progress states reset to pending"
|
||||
}
|
||||
|
||||
|
||||
@router.delete("/cache", response_model=ClearAllCacheResponse)
|
||||
async def clear_all_flacfetch_cache(
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Clear the entire flacfetch search cache.
|
||||
|
||||
This will delete all cached search results from flacfetch's GCS cache.
|
||||
Use with caution - this will cause all subsequent searches to hit
|
||||
the trackers fresh.
|
||||
|
||||
Note: This does NOT clear Firestore job.state_data caches, only the
|
||||
flacfetch-side GCS cache.
|
||||
"""
|
||||
flacfetch_client = get_flacfetch_client()
|
||||
if not flacfetch_client:
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail="flacfetch client not configured"
|
||||
)
|
||||
|
||||
try:
|
||||
deleted_count = await flacfetch_client.clear_all_cache()
|
||||
logger.info(
|
||||
f"Admin {auth_data[0]} cleared all flacfetch cache. "
|
||||
f"Deleted {deleted_count} entries."
|
||||
)
|
||||
return ClearAllCacheResponse(
|
||||
status="success",
|
||||
message=f"Cleared {deleted_count} cache entries from flacfetch.",
|
||||
deleted_count=deleted_count,
|
||||
)
|
||||
except FlacfetchServiceError as e:
|
||||
logger.error(f"Failed to clear all flacfetch cache: {e}")
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"Failed to clear flacfetch cache: {e}"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/cache/stats", response_model=CacheStatsResponse)
|
||||
async def get_flacfetch_cache_stats(
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Get statistics about the flacfetch search cache.
|
||||
|
||||
Returns information about:
|
||||
- Number of cached entries
|
||||
- Total size in bytes
|
||||
- Oldest and newest cache entries
|
||||
- Whether cache is configured
|
||||
"""
|
||||
flacfetch_client = get_flacfetch_client()
|
||||
if not flacfetch_client:
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail="flacfetch client not configured"
|
||||
)
|
||||
|
||||
try:
|
||||
stats = await flacfetch_client.get_cache_stats()
|
||||
return CacheStatsResponse(
|
||||
count=stats.get("count", 0),
|
||||
total_size_bytes=stats.get("total_size_bytes", 0),
|
||||
oldest_entry=stats.get("oldest_entry"),
|
||||
newest_entry=stats.get("newest_entry"),
|
||||
configured=stats.get("configured", True),
|
||||
)
|
||||
except FlacfetchServiceError as e:
|
||||
logger.error(f"Failed to get flacfetch cache stats: {e}")
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"Failed to get cache stats: {e}"
|
||||
)
|
||||
# Trigger deployment
|
||||
@@ -0,0 +1,884 @@
|
||||
"""
|
||||
Audio search API routes for artist+title search mode.
|
||||
|
||||
This module provides endpoints for:
|
||||
1. Creating a job with audio search (artist+title without file)
|
||||
2. Getting search results for a job
|
||||
3. Selecting an audio source to download
|
||||
|
||||
The flow is:
|
||||
1. POST /api/audio-search/search - Create job and search for audio
|
||||
2. GET /api/audio-search/{job_id}/results - Get search results
|
||||
3. POST /api/audio-search/{job_id}/select - Select audio source
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
from typing import Optional, List, Dict, Any, Tuple
|
||||
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request, Depends
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from backend.models.job import JobCreate, JobStatus
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.worker_service import get_worker_service
|
||||
from backend.services.credential_manager import get_credential_manager, CredentialStatus
|
||||
from backend.services.audio_search_service import (
|
||||
get_audio_search_service,
|
||||
AudioSearchResult,
|
||||
AudioSearchError,
|
||||
NoResultsError,
|
||||
DownloadError,
|
||||
)
|
||||
from backend.config import get_settings
|
||||
from backend.version import VERSION
|
||||
from backend.api.dependencies import require_auth
|
||||
from backend.services.auth_service import UserType, AuthResult
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Valid style file types (from file_upload.py)
|
||||
STYLE_FILE_TYPES = {
|
||||
'style_params': {'.json'},
|
||||
'style_intro_background': {'.jpg', '.jpeg', '.png'},
|
||||
'style_karaoke_background': {'.jpg', '.jpeg', '.png'},
|
||||
'style_end_background': {'.jpg', '.jpeg', '.png'},
|
||||
'style_font': {'.ttf', '.otf'},
|
||||
'style_cdg_instrumental_background': {'.jpg', '.jpeg', '.png'},
|
||||
'style_cdg_title_background': {'.jpg', '.jpeg', '.png'},
|
||||
'style_cdg_outro_background': {'.jpg', '.jpeg', '.png'},
|
||||
}
|
||||
router = APIRouter(tags=["audio-search"])
|
||||
|
||||
# Initialize services
|
||||
job_manager = JobManager()
|
||||
storage_service = StorageService()
|
||||
worker_service = get_worker_service()
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Pydantic models
|
||||
# ============================================================================
|
||||
|
||||
class StyleFileRequest(BaseModel):
|
||||
"""Information about a style file to be uploaded."""
|
||||
filename: str = Field(..., description="Original filename with extension")
|
||||
content_type: str = Field(..., description="MIME type of the file")
|
||||
file_type: str = Field(..., description="Type of file: 'style_params', 'style_intro_background', 'style_karaoke_background', 'style_end_background', 'style_font', 'style_cdg_instrumental_background', 'style_cdg_title_background', 'style_cdg_outro_background'")
|
||||
|
||||
|
||||
class StyleUploadUrl(BaseModel):
|
||||
"""Signed URL for uploading a style file."""
|
||||
file_type: str = Field(..., description="Type of file being uploaded")
|
||||
gcs_path: str = Field(..., description="Destination path in GCS")
|
||||
upload_url: str = Field(..., description="Signed URL to PUT the file to")
|
||||
|
||||
|
||||
class AudioSearchRequest(BaseModel):
|
||||
"""Request to search for audio by artist and title."""
|
||||
artist: str = Field(..., description="Artist name to search for")
|
||||
title: str = Field(..., description="Song title to search for")
|
||||
|
||||
# Auto-download mode
|
||||
auto_download: bool = Field(False, description="Automatically select best result and download")
|
||||
|
||||
# Theme configuration
|
||||
theme_id: Optional[str] = Field(None, description="Theme ID to use (e.g., 'nomad', 'default'). If set, CDG/TXT are enabled by default.")
|
||||
color_overrides: Optional[Dict[str, str]] = Field(None, description="Color overrides: artist_color, title_color, sung_lyrics_color, unsung_lyrics_color (hex #RRGGBB)")
|
||||
|
||||
# Processing options (CDG/TXT require style config or theme, disabled by default unless theme is set)
|
||||
enable_cdg: Optional[bool] = Field(None, description="Generate CDG+MP3 package. Default: True if theme_id set, False otherwise")
|
||||
enable_txt: Optional[bool] = Field(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise")
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
|
||||
# Distribution options
|
||||
dropbox_path: Optional[str] = Field(None, description="Dropbox folder path for organized output")
|
||||
gdrive_folder_id: Optional[str] = Field(None, description="Google Drive folder ID for public share uploads")
|
||||
|
||||
# Lyrics configuration
|
||||
lyrics_artist: Optional[str] = Field(None, description="Override artist name for lyrics search")
|
||||
lyrics_title: Optional[str] = Field(None, description="Override title for lyrics search")
|
||||
subtitle_offset_ms: int = Field(0, description="Subtitle timing offset in milliseconds")
|
||||
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model: Optional[str] = Field(None, description="Model for clean instrumental separation")
|
||||
backing_vocals_models: Optional[List[str]] = Field(None, description="Models for backing vocals separation")
|
||||
other_stems_models: Optional[List[str]] = Field(None, description="Models for other stems")
|
||||
|
||||
# Style file uploads (optional)
|
||||
# Style params JSON and related assets (background images, fonts)
|
||||
style_files: Optional[List[StyleFileRequest]] = Field(None, description="Style files to upload (style_params JSON, backgrounds, fonts)")
|
||||
|
||||
# Non-interactive mode
|
||||
non_interactive: bool = Field(False, description="Skip interactive steps (lyrics review, instrumental selection)")
|
||||
|
||||
|
||||
class AudioSearchResultResponse(BaseModel):
|
||||
"""A single audio search result.
|
||||
|
||||
Contains all fields needed for rich display using flacfetch's
|
||||
shared formatting functions.
|
||||
"""
|
||||
index: int
|
||||
title: str
|
||||
artist: str
|
||||
provider: str # Maps to source_name in Release
|
||||
url: Optional[str] = None # Maps to download_url in Release (may be None for remote)
|
||||
duration: Optional[int] = None # Maps to duration_seconds in Release
|
||||
quality: Optional[str] = None # Stringified quality
|
||||
source_id: Optional[str] = None # Maps to info_hash in Release
|
||||
seeders: Optional[int] = None
|
||||
target_file: Optional[str] = None
|
||||
# Additional fields for rich display (from Release.to_dict())
|
||||
year: Optional[int] = None
|
||||
label: Optional[str] = None
|
||||
edition_info: Optional[str] = None
|
||||
release_type: Optional[str] = None
|
||||
channel: Optional[str] = None # For YouTube
|
||||
view_count: Optional[int] = None # For YouTube
|
||||
size_bytes: Optional[int] = None
|
||||
target_file_size: Optional[int] = None
|
||||
track_pattern: Optional[str] = None
|
||||
match_score: Optional[float] = None
|
||||
# Pre-computed display fields
|
||||
formatted_size: Optional[str] = None
|
||||
formatted_duration: Optional[str] = None
|
||||
formatted_views: Optional[str] = None
|
||||
is_lossless: Optional[bool] = None
|
||||
quality_str: Optional[str] = None
|
||||
# Full quality object for Release.from_dict()
|
||||
quality_data: Optional[Dict[str, Any]] = None
|
||||
|
||||
|
||||
class AudioSearchResponse(BaseModel):
|
||||
"""Response from audio search."""
|
||||
status: str
|
||||
job_id: str
|
||||
message: str
|
||||
results: Optional[List[AudioSearchResultResponse]] = None
|
||||
results_count: int = 0
|
||||
auto_download: bool = False
|
||||
server_version: str
|
||||
# Style file upload URLs (when style_files are specified in request)
|
||||
style_upload_urls: Optional[List[StyleUploadUrl]] = None
|
||||
|
||||
|
||||
class AudioSelectRequest(BaseModel):
|
||||
"""Request to select an audio source."""
|
||||
selection_index: int = Field(..., description="Index of the selected audio source from search results")
|
||||
|
||||
|
||||
class AudioSelectResponse(BaseModel):
|
||||
"""Response from audio source selection."""
|
||||
status: str
|
||||
job_id: str
|
||||
message: str
|
||||
selected_index: int
|
||||
selected_title: str
|
||||
selected_artist: str
|
||||
selected_provider: str
|
||||
|
||||
|
||||
def _resolve_cdg_txt_defaults(
|
||||
theme_id: Optional[str],
|
||||
enable_cdg: Optional[bool],
|
||||
enable_txt: Optional[bool]
|
||||
) -> Tuple[bool, bool]:
|
||||
"""
|
||||
Resolve CDG/TXT settings based on theme and explicit settings.
|
||||
|
||||
When a theme is selected, CDG and TXT are enabled by default.
|
||||
Explicit True/False values always override the default.
|
||||
|
||||
Args:
|
||||
theme_id: Theme identifier (if any)
|
||||
enable_cdg: Explicit CDG setting (None means use default)
|
||||
enable_txt: Explicit TXT setting (None means use default)
|
||||
|
||||
Returns:
|
||||
Tuple of (resolved_enable_cdg, resolved_enable_txt)
|
||||
"""
|
||||
# Default based on whether theme is set
|
||||
default_enabled = theme_id is not None
|
||||
|
||||
# Explicit values override defaults, None uses default
|
||||
resolved_cdg = enable_cdg if enable_cdg is not None else default_enabled
|
||||
resolved_txt = enable_txt if enable_txt is not None else default_enabled
|
||||
|
||||
return resolved_cdg, resolved_txt
|
||||
|
||||
|
||||
def extract_request_metadata(request: Request, created_from: str = "audio_search") -> Dict[str, Any]:
|
||||
"""Extract metadata from request for job tracking."""
|
||||
headers = dict(request.headers)
|
||||
|
||||
client_ip = headers.get('x-forwarded-for', '').split(',')[0].strip()
|
||||
if not client_ip and request.client:
|
||||
client_ip = request.client.host
|
||||
|
||||
user_agent = headers.get('user-agent', '')
|
||||
environment = headers.get('x-environment', '')
|
||||
client_id = headers.get('x-client-id', '')
|
||||
|
||||
custom_headers = {}
|
||||
for key, value in headers.items():
|
||||
if key.lower().startswith('x-') and key.lower() not in ('x-forwarded-for', 'x-forwarded-proto', 'x-forwarded-host'):
|
||||
custom_headers[key] = value
|
||||
|
||||
return {
|
||||
'client_ip': client_ip,
|
||||
'user_agent': user_agent,
|
||||
'environment': environment,
|
||||
'client_id': client_id,
|
||||
'server_version': VERSION,
|
||||
'created_from': created_from,
|
||||
'custom_headers': custom_headers,
|
||||
}
|
||||
|
||||
|
||||
async def _trigger_workers_parallel(job_id: str) -> None:
|
||||
"""Trigger both audio and lyrics workers in parallel."""
|
||||
await asyncio.gather(
|
||||
worker_service.trigger_audio_worker(job_id),
|
||||
worker_service.trigger_lyrics_worker(job_id)
|
||||
)
|
||||
|
||||
|
||||
async def _download_and_start_processing(
|
||||
job_id: str,
|
||||
selection_index: int,
|
||||
audio_search_service,
|
||||
background_tasks: BackgroundTasks,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Download selected audio and start job processing.
|
||||
|
||||
This is called either:
|
||||
- Immediately after search if auto_download=True
|
||||
- When user calls the select endpoint
|
||||
|
||||
For remote flacfetch downloads (torrent sources), the file is downloaded
|
||||
on the flacfetch VM and uploaded directly to GCS. For local downloads
|
||||
(YouTube), the file is downloaded locally and then uploaded to GCS.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Get search results from state_data
|
||||
search_results = job.state_data.get('audio_search_results', [])
|
||||
if not search_results:
|
||||
raise HTTPException(status_code=400, detail="No search results available")
|
||||
|
||||
if selection_index < 0 or selection_index >= len(search_results):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid selection index {selection_index}. Valid range: 0-{len(search_results)-1}"
|
||||
)
|
||||
|
||||
selected = search_results[selection_index]
|
||||
|
||||
# Get remote_search_id from state_data (stored during initial search)
|
||||
remote_search_id = job.state_data.get('remote_search_id')
|
||||
|
||||
# Transition to downloading state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.DOWNLOADING_AUDIO,
|
||||
progress=10,
|
||||
message=f"Downloading from {selected['provider']}: {selected['artist']} - {selected['title']}",
|
||||
state_data_updates={
|
||||
'selected_audio_index': selection_index,
|
||||
'selected_audio_provider': selected['provider'],
|
||||
}
|
||||
)
|
||||
|
||||
try:
|
||||
# Determine if this is a remote torrent download
|
||||
is_torrent_source = selected.get('provider') in ['RED', 'OPS']
|
||||
is_remote_enabled = audio_search_service.is_remote_enabled()
|
||||
|
||||
# Check if we can use download_by_id (preferred - avoids re-searching)
|
||||
source_id = selected.get('source_id')
|
||||
source_name = selected.get('provider')
|
||||
target_file = selected.get('target_file')
|
||||
download_url = selected.get('url')
|
||||
|
||||
# For remote torrent downloads, have flacfetch VM upload directly to GCS
|
||||
if is_torrent_source and is_remote_enabled:
|
||||
# Generate GCS path for remote upload
|
||||
gcs_destination = f"uploads/{job_id}/audio/"
|
||||
|
||||
# Use download_by_id if we have source_id (preferred - no re-search needed)
|
||||
if source_id and source_name:
|
||||
logger.info(f"Using download_by_id for {source_name} ID={source_id} with GCS upload to: {gcs_destination}")
|
||||
|
||||
result = audio_search_service.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir="", # Not used for remote
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_destination,
|
||||
)
|
||||
else:
|
||||
# Fallback to search-based download (requires re-search)
|
||||
logger.info(f"No source_id available, falling back to search-based download to: {gcs_destination}")
|
||||
|
||||
result = audio_search_service.download(
|
||||
result_index=selection_index,
|
||||
output_dir="", # Not used for remote
|
||||
gcs_path=gcs_destination,
|
||||
remote_search_id=remote_search_id,
|
||||
)
|
||||
|
||||
# For remote downloads, filepath is already the GCS path
|
||||
if result.filepath.startswith("gs://"):
|
||||
# Extract the path portion after the bucket name
|
||||
# Format: gs://bucket/uploads/job_id/audio/filename.flac
|
||||
parts = result.filepath.replace("gs://", "").split("/", 1)
|
||||
if len(parts) == 2:
|
||||
audio_gcs_path = parts[1]
|
||||
else:
|
||||
audio_gcs_path = result.filepath
|
||||
filename = os.path.basename(result.filepath)
|
||||
else:
|
||||
# Fallback: treat as local path (shouldn't happen for remote)
|
||||
filename = os.path.basename(result.filepath)
|
||||
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
|
||||
|
||||
logger.warning(f"Remote download returned local path: {result.filepath}, uploading manually")
|
||||
with open(result.filepath, 'rb') as f:
|
||||
storage_service.upload_fileobj(f, audio_gcs_path, content_type='audio/flac')
|
||||
|
||||
logger.info(f"Remote download complete, GCS path: {audio_gcs_path}")
|
||||
else:
|
||||
# Local download (YouTube or fallback)
|
||||
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
|
||||
|
||||
# Use download_by_id if we have source_id and remote is enabled
|
||||
if source_id and source_name and is_remote_enabled:
|
||||
logger.info(f"Using download_by_id for local download: {source_name} ID={source_id}")
|
||||
|
||||
result = audio_search_service.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir=temp_dir,
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
)
|
||||
elif source_name == 'YouTube' and download_url:
|
||||
# For YouTube without remote, download directly using URL from state_data
|
||||
# This avoids relying on in-memory cache which doesn't persist across instances
|
||||
logger.info(f"Downloading YouTube audio directly from URL: {download_url}")
|
||||
from backend.workers.audio_worker import download_from_url
|
||||
|
||||
local_path = await download_from_url(
|
||||
download_url,
|
||||
temp_dir,
|
||||
selected.get('artist'),
|
||||
selected.get('title')
|
||||
)
|
||||
|
||||
if not local_path or not os.path.exists(local_path):
|
||||
raise DownloadError(f"Failed to download from YouTube: {download_url}")
|
||||
|
||||
# Create a result-like object
|
||||
class DownloadResult:
|
||||
def __init__(self, filepath):
|
||||
self.filepath = filepath
|
||||
|
||||
result = DownloadResult(local_path)
|
||||
else:
|
||||
# Fallback to search-based download (may fail if cache is empty)
|
||||
logger.warning(f"Falling back to cache-based download for {source_name} - this may fail on multi-instance deployments")
|
||||
result = audio_search_service.download(
|
||||
result_index=selection_index,
|
||||
output_dir=temp_dir,
|
||||
remote_search_id=remote_search_id, # Pass for potential fallback scenarios
|
||||
)
|
||||
|
||||
# Upload to GCS
|
||||
filename = os.path.basename(result.filepath)
|
||||
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
|
||||
|
||||
with open(result.filepath, 'rb') as f:
|
||||
storage_service.upload_fileobj(
|
||||
f,
|
||||
audio_gcs_path,
|
||||
content_type='audio/flac' # flacfetch typically returns FLAC
|
||||
)
|
||||
|
||||
logger.info(f"Uploaded audio to GCS: {audio_gcs_path}")
|
||||
|
||||
# Clean up temp file
|
||||
try:
|
||||
os.remove(result.filepath)
|
||||
os.rmdir(temp_dir)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to clean up temp files: {e}")
|
||||
|
||||
# Update job with GCS path and transition to DOWNLOADING
|
||||
job_manager.update_job(job_id, {
|
||||
'input_media_gcs_path': audio_gcs_path,
|
||||
'filename': filename,
|
||||
})
|
||||
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.DOWNLOADING,
|
||||
progress=15,
|
||||
message="Audio downloaded, starting processing"
|
||||
)
|
||||
|
||||
# Trigger workers
|
||||
background_tasks.add_task(_trigger_workers_parallel, job_id)
|
||||
|
||||
return {
|
||||
'selected_index': selection_index,
|
||||
'selected_title': selected['title'],
|
||||
'selected_artist': selected['artist'],
|
||||
'selected_provider': selected['provider'],
|
||||
}
|
||||
|
||||
except DownloadError as e:
|
||||
job_manager.fail_job(job_id, f"Audio download failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Download failed: {e}")
|
||||
except Exception as e:
|
||||
job_manager.fail_job(job_id, f"Audio download failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Download failed: {e}")
|
||||
|
||||
|
||||
@router.post("/audio-search/search", response_model=AudioSearchResponse)
|
||||
async def search_audio(
|
||||
request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: AudioSearchRequest,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Search for audio by artist and title, creating a new job.
|
||||
|
||||
This endpoint:
|
||||
1. Creates a job in PENDING state
|
||||
2. Searches for audio using flacfetch
|
||||
3. Either returns search results for user selection, or
|
||||
4. If auto_download=True, automatically selects best and starts processing
|
||||
|
||||
Use cases:
|
||||
- Interactive mode (default): Returns results, user calls /select endpoint
|
||||
- Auto mode (auto_download=True): Automatically selects and downloads best
|
||||
"""
|
||||
try:
|
||||
# Apply default distribution settings
|
||||
settings = get_settings()
|
||||
effective_dropbox_path = body.dropbox_path or settings.default_dropbox_path
|
||||
effective_gdrive_folder_id = body.gdrive_folder_id or settings.default_gdrive_folder_id
|
||||
effective_discord_webhook_url = body.discord_webhook_url or settings.default_discord_webhook_url
|
||||
|
||||
# Apply defaults for YouTube/Dropbox distribution (for web service)
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload or settings.default_enable_youtube_upload
|
||||
effective_brand_prefix = body.brand_prefix or settings.default_brand_prefix
|
||||
effective_youtube_description = body.youtube_description or settings.default_youtube_description
|
||||
|
||||
# Validate credentials if distribution services are requested
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if effective_enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
|
||||
if effective_dropbox_path:
|
||||
result = credential_manager.check_dropbox_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"dropbox ({result.message})")
|
||||
|
||||
if effective_gdrive_folder_id:
|
||||
result = credential_manager.check_gdrive_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"gdrive ({result.message})")
|
||||
|
||||
if invalid_services:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail={
|
||||
"error": "credentials_invalid",
|
||||
"message": f"Distribution services need re-authorization: {', '.join(invalid_services)}",
|
||||
"invalid_services": invalid_services,
|
||||
}
|
||||
)
|
||||
|
||||
# Extract request metadata
|
||||
request_metadata = extract_request_metadata(request, created_from="audio_search")
|
||||
|
||||
# Resolve CDG/TXT defaults based on theme
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
body.theme_id, body.enable_cdg, body.enable_txt
|
||||
)
|
||||
|
||||
# Use authenticated user's email
|
||||
effective_user_email = auth_result.user_email
|
||||
|
||||
# Create job
|
||||
job_create = JobCreate(
|
||||
artist=body.artist,
|
||||
title=body.title,
|
||||
theme_id=body.theme_id,
|
||||
color_overrides=body.color_overrides or {},
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=effective_brand_prefix,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=effective_youtube_description,
|
||||
youtube_description_template=effective_youtube_description, # video_worker reads this field
|
||||
discord_webhook_url=effective_discord_webhook_url,
|
||||
dropbox_path=effective_dropbox_path,
|
||||
gdrive_folder_id=effective_gdrive_folder_id,
|
||||
user_email=effective_user_email,
|
||||
lyrics_artist=body.lyrics_artist,
|
||||
lyrics_title=body.lyrics_title,
|
||||
subtitle_offset_ms=body.subtitle_offset_ms,
|
||||
clean_instrumental_model=body.clean_instrumental_model,
|
||||
backing_vocals_models=body.backing_vocals_models,
|
||||
other_stems_models=body.other_stems_models,
|
||||
audio_search_artist=body.artist,
|
||||
audio_search_title=body.title,
|
||||
auto_download=body.auto_download,
|
||||
request_metadata=request_metadata,
|
||||
non_interactive=body.non_interactive,
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job_id = job.job_id
|
||||
|
||||
logger.info(f"Created job {job_id} for audio search: {body.artist} - {body.title}")
|
||||
|
||||
# Update job with audio search fields
|
||||
job_manager.update_job(job_id, {
|
||||
'audio_search_artist': body.artist,
|
||||
'audio_search_title': body.title,
|
||||
'auto_download': body.auto_download,
|
||||
})
|
||||
|
||||
# If theme is set and no custom style files are being uploaded, prepare theme style now
|
||||
# This copies the theme's style_params.json to the job folder so LyricsTranscriber
|
||||
# can access the style configuration for preview videos
|
||||
if body.theme_id and not body.style_files:
|
||||
from backend.api.routes.file_upload import _prepare_theme_for_job
|
||||
try:
|
||||
style_params_path, theme_style_assets, youtube_desc = _prepare_theme_for_job(
|
||||
job_id, body.theme_id, body.color_overrides
|
||||
)
|
||||
theme_update = {
|
||||
'style_params_gcs_path': style_params_path,
|
||||
'style_assets': theme_style_assets,
|
||||
}
|
||||
if youtube_desc and not effective_youtube_description:
|
||||
theme_update['youtube_description_template'] = youtube_desc
|
||||
job_manager.update_job(job_id, theme_update)
|
||||
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to prepare theme '{body.theme_id}' for job {job_id}: {e}")
|
||||
# Continue without theme - job can still be processed with defaults
|
||||
|
||||
# Handle style file uploads if provided
|
||||
style_upload_urls: List[StyleUploadUrl] = []
|
||||
style_assets = {}
|
||||
|
||||
if body.style_files:
|
||||
logger.info(f"Processing {len(body.style_files)} style file uploads for job {job_id}")
|
||||
|
||||
for file_info in body.style_files:
|
||||
# Validate file type
|
||||
if file_info.file_type not in STYLE_FILE_TYPES:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid file type '{file_info.file_type}'. Must be one of: {', '.join(STYLE_FILE_TYPES.keys())}"
|
||||
)
|
||||
|
||||
# Validate extension
|
||||
ext = Path(file_info.filename).suffix.lower()
|
||||
allowed_exts = STYLE_FILE_TYPES[file_info.file_type]
|
||||
if ext not in allowed_exts:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid extension '{ext}' for {file_info.file_type}. Allowed: {', '.join(allowed_exts)}"
|
||||
)
|
||||
|
||||
# Generate GCS path
|
||||
if file_info.file_type == 'style_params':
|
||||
gcs_path = f"uploads/{job_id}/style/style_params.json"
|
||||
else:
|
||||
# style_intro_background -> intro_background, etc.
|
||||
asset_key = file_info.file_type.replace('style_', '')
|
||||
gcs_path = f"uploads/{job_id}/style/{asset_key}{ext}"
|
||||
|
||||
# Generate signed upload URL
|
||||
signed_url = storage_service.generate_signed_upload_url(
|
||||
gcs_path,
|
||||
content_type=file_info.content_type,
|
||||
expiration_minutes=60
|
||||
)
|
||||
|
||||
style_upload_urls.append(StyleUploadUrl(
|
||||
file_type=file_info.file_type,
|
||||
gcs_path=gcs_path,
|
||||
upload_url=signed_url
|
||||
))
|
||||
|
||||
# Track the expected asset paths
|
||||
if file_info.file_type == 'style_params':
|
||||
style_assets['style_params'] = gcs_path
|
||||
else:
|
||||
asset_key = file_info.file_type.replace('style_', '')
|
||||
style_assets[asset_key] = gcs_path
|
||||
|
||||
# Update job with style asset expectations
|
||||
style_update = {'style_assets': style_assets}
|
||||
if 'style_params' in style_assets:
|
||||
style_update['style_params_gcs_path'] = style_assets['style_params']
|
||||
job_manager.update_job(job_id, style_update)
|
||||
|
||||
logger.info(f"Generated {len(style_upload_urls)} style upload URLs for job {job_id}")
|
||||
|
||||
# Transition to searching state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.SEARCHING_AUDIO,
|
||||
progress=5,
|
||||
message=f"Searching for: {body.artist} - {body.title}"
|
||||
)
|
||||
|
||||
# Perform search
|
||||
audio_search_service = get_audio_search_service()
|
||||
|
||||
try:
|
||||
search_results = audio_search_service.search(body.artist, body.title)
|
||||
except NoResultsError as e:
|
||||
job_manager.fail_job(job_id, f"No audio sources found for: {body.artist} - {body.title}")
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail={
|
||||
"error": "no_results",
|
||||
"message": str(e),
|
||||
"job_id": job_id,
|
||||
}
|
||||
)
|
||||
except AudioSearchError as e:
|
||||
job_manager.fail_job(job_id, f"Audio search failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Search failed: {e}")
|
||||
|
||||
# Store results in job state_data, including remote_search_id if available
|
||||
results_dicts = [r.to_dict() for r in search_results]
|
||||
state_data_update = {
|
||||
'audio_search_results': results_dicts,
|
||||
'audio_search_count': len(results_dicts),
|
||||
}
|
||||
# Store remote_search_id for use during download (important for concurrent requests)
|
||||
if audio_search_service.last_remote_search_id:
|
||||
state_data_update['remote_search_id'] = audio_search_service.last_remote_search_id
|
||||
job_manager.update_job(job_id, {
|
||||
'state_data': state_data_update
|
||||
})
|
||||
|
||||
# If auto_download, select best and start processing
|
||||
if body.auto_download:
|
||||
best_index = audio_search_service.select_best(search_results)
|
||||
|
||||
logger.info(f"Auto-download enabled, selecting result {best_index}")
|
||||
|
||||
selection_info = await _download_and_start_processing(
|
||||
job_id=job_id,
|
||||
selection_index=best_index,
|
||||
audio_search_service=audio_search_service,
|
||||
background_tasks=background_tasks,
|
||||
)
|
||||
|
||||
return AudioSearchResponse(
|
||||
status="success",
|
||||
job_id=job_id,
|
||||
message=f"Audio found and download started: {selection_info['selected_artist']} - {selection_info['selected_title']} ({selection_info['selected_provider']})",
|
||||
results=None, # Don't return results in auto mode
|
||||
results_count=len(search_results),
|
||||
auto_download=True,
|
||||
server_version=VERSION,
|
||||
style_upload_urls=style_upload_urls if style_upload_urls else None,
|
||||
)
|
||||
|
||||
# Interactive mode: return results for user selection
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.AWAITING_AUDIO_SELECTION,
|
||||
progress=10,
|
||||
message=f"Found {len(search_results)} audio sources. Waiting for selection."
|
||||
)
|
||||
|
||||
# Convert to response format with full Release data for rich display
|
||||
result_responses = []
|
||||
for r in search_results:
|
||||
# Get full serialized data from raw_result if available
|
||||
# raw_result can be either:
|
||||
# - A dict (from remote flacfetch API)
|
||||
# - A Release object (from local flacfetch)
|
||||
raw_dict = {}
|
||||
if r.raw_result:
|
||||
if isinstance(r.raw_result, dict):
|
||||
# Remote flacfetch API returns dicts directly
|
||||
raw_dict = r.raw_result
|
||||
else:
|
||||
# Local flacfetch returns Release objects
|
||||
try:
|
||||
raw_dict = r.raw_result.to_dict()
|
||||
except AttributeError:
|
||||
pass # Not a Release object
|
||||
|
||||
result_responses.append(
|
||||
AudioSearchResultResponse(
|
||||
index=r.index,
|
||||
title=r.title,
|
||||
artist=r.artist,
|
||||
provider=r.provider,
|
||||
url=r.url,
|
||||
duration=r.duration,
|
||||
quality=r.quality,
|
||||
source_id=r.source_id,
|
||||
seeders=raw_dict.get('seeders') or r.seeders,
|
||||
target_file=raw_dict.get('target_file') or r.target_file,
|
||||
# Additional fields for rich display
|
||||
year=raw_dict.get('year'),
|
||||
label=raw_dict.get('label'),
|
||||
edition_info=raw_dict.get('edition_info'),
|
||||
release_type=raw_dict.get('release_type'),
|
||||
channel=raw_dict.get('channel'),
|
||||
view_count=raw_dict.get('view_count'),
|
||||
size_bytes=raw_dict.get('size_bytes'),
|
||||
target_file_size=raw_dict.get('target_file_size'),
|
||||
track_pattern=raw_dict.get('track_pattern'),
|
||||
match_score=raw_dict.get('match_score'),
|
||||
# Pre-computed display fields
|
||||
formatted_size=raw_dict.get('formatted_size'),
|
||||
formatted_duration=raw_dict.get('formatted_duration'),
|
||||
formatted_views=raw_dict.get('formatted_views'),
|
||||
is_lossless=raw_dict.get('is_lossless'),
|
||||
quality_str=raw_dict.get('quality_str'),
|
||||
# Full quality object for Release.from_dict()
|
||||
# Remote API uses 'quality_data', local uses 'quality' for the dict
|
||||
quality_data=raw_dict.get('quality_data') or raw_dict.get('quality'),
|
||||
)
|
||||
)
|
||||
|
||||
return AudioSearchResponse(
|
||||
status="awaiting_selection",
|
||||
job_id=job_id,
|
||||
message=f"Found {len(search_results)} audio sources. Call /api/audio-search/{job_id}/select to choose one.",
|
||||
results=result_responses,
|
||||
results_count=len(search_results),
|
||||
auto_download=False,
|
||||
server_version=VERSION,
|
||||
style_upload_urls=style_upload_urls if style_upload_urls else None,
|
||||
)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error in audio search: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.get("/audio-search/{job_id}/results")
|
||||
async def get_audio_search_results(
|
||||
job_id: str,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Get audio search results for a job.
|
||||
|
||||
Returns the cached search results so user can select one.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
search_results = job.state_data.get('audio_search_results', [])
|
||||
|
||||
if not search_results:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="No search results available for this job"
|
||||
)
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"job_id": job_id,
|
||||
"job_status": job.status,
|
||||
"artist": job.audio_search_artist or job.artist,
|
||||
"title": job.audio_search_title or job.title,
|
||||
"results": search_results,
|
||||
"results_count": len(search_results),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/audio-search/{job_id}/select", response_model=AudioSelectResponse)
|
||||
async def select_audio_source(
|
||||
job_id: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: AudioSelectRequest,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Select an audio source and start job processing.
|
||||
|
||||
This endpoint:
|
||||
1. Validates the job is awaiting selection
|
||||
2. Downloads the selected audio
|
||||
3. Uploads to GCS
|
||||
4. Triggers audio and lyrics workers
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Verify job is awaiting selection
|
||||
if job.status != JobStatus.AWAITING_AUDIO_SELECTION:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job is not awaiting audio selection (status: {job.status})"
|
||||
)
|
||||
|
||||
# Get search service instance
|
||||
# Note: With download_by_id, we no longer need to re-search to populate the cache.
|
||||
# The source_id stored in job.state_data['audio_search_results'] is sufficient.
|
||||
audio_search_service = get_audio_search_service()
|
||||
|
||||
# Validate search results exist in job state_data
|
||||
search_results = job.state_data.get('audio_search_results', [])
|
||||
if not search_results:
|
||||
raise HTTPException(status_code=400, detail="No search results cached for this job")
|
||||
|
||||
selection_info = await _download_and_start_processing(
|
||||
job_id=job_id,
|
||||
selection_index=body.selection_index,
|
||||
audio_search_service=audio_search_service,
|
||||
background_tasks=background_tasks,
|
||||
)
|
||||
|
||||
return AudioSelectResponse(
|
||||
status="success",
|
||||
job_id=job_id,
|
||||
message="Audio selected and download started",
|
||||
selected_index=selection_info['selected_index'],
|
||||
selected_title=selection_info['selected_title'],
|
||||
selected_artist=selection_info['selected_artist'],
|
||||
selected_provider=selection_info['selected_provider'],
|
||||
)
|
||||
|
||||
|
||||
@@ -0,0 +1,348 @@
|
||||
"""
|
||||
OAuth credential management API routes.
|
||||
|
||||
Provides endpoints for:
|
||||
1. Checking credential status
|
||||
2. Device authorization flow for re-authentication
|
||||
3. Credential validation before job submission
|
||||
"""
|
||||
import logging
|
||||
from typing import Optional
|
||||
from datetime import datetime
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.services.credential_manager import (
|
||||
get_credential_manager,
|
||||
CredentialStatus,
|
||||
CredentialCheckResult,
|
||||
)
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/auth", tags=["auth"])
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Request/Response Models
|
||||
# ===========================================================================
|
||||
|
||||
class CredentialStatusResponse(BaseModel):
|
||||
"""Response model for credential status."""
|
||||
service: str
|
||||
status: str
|
||||
message: str
|
||||
last_checked: datetime
|
||||
expires_at: Optional[datetime] = None
|
||||
|
||||
class Config:
|
||||
from_attributes = True
|
||||
|
||||
|
||||
class AllCredentialsStatusResponse(BaseModel):
|
||||
"""Response model for all credentials status."""
|
||||
youtube: CredentialStatusResponse
|
||||
gdrive: CredentialStatusResponse
|
||||
dropbox: CredentialStatusResponse
|
||||
all_valid: bool
|
||||
services_needing_auth: list[str]
|
||||
|
||||
|
||||
class DeviceAuthStartRequest(BaseModel):
|
||||
"""Request to start device authorization flow."""
|
||||
client_id: Optional[str] = None # Optional - reads from Secret Manager if not provided
|
||||
client_secret: Optional[str] = None # Optional - reads from Secret Manager if not provided
|
||||
|
||||
|
||||
class DeviceAuthStartResponse(BaseModel):
|
||||
"""Response with device authorization info."""
|
||||
device_code: str
|
||||
user_code: str
|
||||
verification_url: str
|
||||
expires_in: int
|
||||
interval: int
|
||||
instructions: str
|
||||
|
||||
|
||||
class DeviceAuthPollResponse(BaseModel):
|
||||
"""Response from polling device authorization."""
|
||||
status: str # "pending", "complete", "expired", "error"
|
||||
message: Optional[str] = None
|
||||
credentials_saved: bool = False
|
||||
|
||||
|
||||
class CredentialValidationRequest(BaseModel):
|
||||
"""Request to validate credentials for specific services."""
|
||||
youtube: bool = False
|
||||
gdrive: bool = False
|
||||
dropbox: bool = False
|
||||
|
||||
|
||||
class CredentialValidationResponse(BaseModel):
|
||||
"""Response from credential validation."""
|
||||
valid: bool
|
||||
invalid_services: list[str]
|
||||
message: str
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Endpoints
|
||||
# ===========================================================================
|
||||
|
||||
@router.get("/status", response_model=AllCredentialsStatusResponse)
|
||||
async def get_credentials_status():
|
||||
"""
|
||||
Get the status of all OAuth credentials.
|
||||
|
||||
Returns the validation status for YouTube, Google Drive, and Dropbox
|
||||
credentials, including whether they are valid, expired, or need
|
||||
re-authorization.
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
results = manager.check_all_credentials()
|
||||
|
||||
def to_response(result: CredentialCheckResult) -> CredentialStatusResponse:
|
||||
return CredentialStatusResponse(
|
||||
service=result.service,
|
||||
status=result.status.value,
|
||||
message=result.message,
|
||||
last_checked=result.last_checked,
|
||||
expires_at=result.expires_at
|
||||
)
|
||||
|
||||
services_needing_auth = [
|
||||
name for name, result in results.items()
|
||||
if result.status in (CredentialStatus.INVALID, CredentialStatus.EXPIRED, CredentialStatus.NOT_CONFIGURED)
|
||||
]
|
||||
|
||||
return AllCredentialsStatusResponse(
|
||||
youtube=to_response(results["youtube"]),
|
||||
gdrive=to_response(results["gdrive"]),
|
||||
dropbox=to_response(results["dropbox"]),
|
||||
all_valid=len(services_needing_auth) == 0,
|
||||
services_needing_auth=services_needing_auth
|
||||
)
|
||||
|
||||
|
||||
@router.get("/status/{service}", response_model=CredentialStatusResponse)
|
||||
async def get_service_credential_status(service: str):
|
||||
"""
|
||||
Get the status of a specific service's OAuth credentials.
|
||||
|
||||
Args:
|
||||
service: One of "youtube", "gdrive", "dropbox"
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
|
||||
if service == "youtube":
|
||||
result = manager.check_youtube_credentials()
|
||||
elif service == "gdrive":
|
||||
result = manager.check_gdrive_credentials()
|
||||
elif service == "dropbox":
|
||||
result = manager.check_dropbox_credentials()
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail=f"Unknown service: {service}")
|
||||
|
||||
return CredentialStatusResponse(
|
||||
service=result.service,
|
||||
status=result.status.value,
|
||||
message=result.message,
|
||||
last_checked=result.last_checked,
|
||||
expires_at=result.expires_at
|
||||
)
|
||||
|
||||
|
||||
@router.post("/validate", response_model=CredentialValidationResponse)
|
||||
async def validate_credentials(request: CredentialValidationRequest):
|
||||
"""
|
||||
Validate that credentials are available for requested services.
|
||||
|
||||
Use this before submitting a job to ensure all required
|
||||
credentials are valid.
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
invalid_services = []
|
||||
|
||||
if request.youtube:
|
||||
result = manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append("youtube")
|
||||
|
||||
if request.gdrive:
|
||||
result = manager.check_gdrive_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append("gdrive")
|
||||
|
||||
if request.dropbox:
|
||||
result = manager.check_dropbox_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append("dropbox")
|
||||
|
||||
if invalid_services:
|
||||
return CredentialValidationResponse(
|
||||
valid=False,
|
||||
invalid_services=invalid_services,
|
||||
message=f"The following services need re-authorization: {', '.join(invalid_services)}"
|
||||
)
|
||||
|
||||
return CredentialValidationResponse(
|
||||
valid=True,
|
||||
invalid_services=[],
|
||||
message="All requested credentials are valid"
|
||||
)
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Device Authorization Flow Endpoints
|
||||
# ===========================================================================
|
||||
|
||||
@router.post("/youtube/device", response_model=DeviceAuthStartResponse)
|
||||
async def start_youtube_device_auth(request: Optional[DeviceAuthStartRequest] = None):
|
||||
"""
|
||||
Start YouTube device authorization flow.
|
||||
|
||||
This initiates a device auth flow that allows authorization
|
||||
from any device. The user must visit the verification URL
|
||||
and enter the user code.
|
||||
|
||||
Client credentials are loaded from Secret Manager ('youtube-client-credentials')
|
||||
unless explicitly provided in the request body.
|
||||
|
||||
After starting, poll /auth/youtube/device/{device_code} to
|
||||
check for completion.
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
|
||||
try:
|
||||
device_info = manager.start_youtube_device_auth(
|
||||
client_id=request.client_id if request else None,
|
||||
client_secret=request.client_secret if request else None
|
||||
)
|
||||
|
||||
return DeviceAuthStartResponse(
|
||||
device_code=device_info.device_code,
|
||||
user_code=device_info.user_code,
|
||||
verification_url=device_info.verification_url,
|
||||
expires_in=device_info.expires_in,
|
||||
interval=device_info.interval,
|
||||
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to start YouTube device auth: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.get("/youtube/device/{device_code}", response_model=DeviceAuthPollResponse)
|
||||
async def poll_youtube_device_auth(device_code: str):
|
||||
"""
|
||||
Poll for YouTube device authorization completion.
|
||||
|
||||
Call this endpoint at the interval specified in the device auth
|
||||
start response. Keep polling while status is "pending".
|
||||
|
||||
Status values:
|
||||
- pending: User has not yet authorized
|
||||
- complete: Authorization successful, credentials saved
|
||||
- expired: Device code expired, start new flow
|
||||
- error: An error occurred
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
|
||||
status, data = manager.poll_device_auth("youtube", device_code)
|
||||
|
||||
return DeviceAuthPollResponse(
|
||||
status=status,
|
||||
message=data.get("message") if data else None,
|
||||
credentials_saved=(status == "complete")
|
||||
)
|
||||
|
||||
|
||||
@router.post("/gdrive/device", response_model=DeviceAuthStartResponse)
|
||||
async def start_gdrive_device_auth(request: Optional[DeviceAuthStartRequest] = None):
|
||||
"""
|
||||
Start Google Drive device authorization flow.
|
||||
|
||||
Client credentials are loaded from Secret Manager ('gdrive-client-credentials')
|
||||
unless explicitly provided in the request body.
|
||||
|
||||
After starting, poll /auth/gdrive/device/{device_code} to check for completion.
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
|
||||
try:
|
||||
device_info = manager.start_gdrive_device_auth(
|
||||
client_id=request.client_id if request else None,
|
||||
client_secret=request.client_secret if request else None
|
||||
)
|
||||
|
||||
return DeviceAuthStartResponse(
|
||||
device_code=device_info.device_code,
|
||||
user_code=device_info.user_code,
|
||||
verification_url=device_info.verification_url,
|
||||
expires_in=device_info.expires_in,
|
||||
interval=device_info.interval,
|
||||
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to start Google Drive device auth: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.get("/gdrive/device/{device_code}", response_model=DeviceAuthPollResponse)
|
||||
async def poll_gdrive_device_auth(device_code: str):
|
||||
"""
|
||||
Poll for Google Drive device authorization completion.
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
|
||||
status, data = manager.poll_device_auth("gdrive", device_code)
|
||||
|
||||
return DeviceAuthPollResponse(
|
||||
status=status,
|
||||
message=data.get("message") if data else None,
|
||||
credentials_saved=(status == "complete")
|
||||
)
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Alert Testing
|
||||
# ===========================================================================
|
||||
|
||||
@router.post("/test-alert")
|
||||
async def test_credential_alert(background_tasks: BackgroundTasks):
|
||||
"""
|
||||
Test the credential alert mechanism.
|
||||
|
||||
Sends a test alert to Discord if configured.
|
||||
"""
|
||||
settings = get_settings()
|
||||
discord_url = settings.get_secret("discord-alert-webhook")
|
||||
|
||||
if not discord_url:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="Discord alert webhook not configured"
|
||||
)
|
||||
|
||||
manager = get_credential_manager()
|
||||
|
||||
# Create a fake invalid result for testing
|
||||
from backend.services.credential_manager import CredentialCheckResult
|
||||
test_results = [
|
||||
CredentialCheckResult(
|
||||
service="test",
|
||||
status=CredentialStatus.INVALID,
|
||||
message="This is a test alert",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
]
|
||||
|
||||
background_tasks.add_task(
|
||||
manager.send_credential_alert,
|
||||
test_results,
|
||||
discord_url
|
||||
)
|
||||
|
||||
return {"message": "Test alert queued"}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,310 @@
|
||||
"""
|
||||
Health check routes.
|
||||
"""
|
||||
import os
|
||||
import logging
|
||||
import subprocess
|
||||
import platform
|
||||
from fastapi import APIRouter
|
||||
from typing import Dict, Any, Optional
|
||||
|
||||
from backend.version import VERSION
|
||||
from backend.services.flacfetch_client import get_flacfetch_client
|
||||
from backend.services.email_service import get_email_service
|
||||
from backend.services.stripe_service import get_stripe_service
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def get_system_info() -> Dict[str, Any]:
|
||||
"""Get system information for performance analysis."""
|
||||
info = {
|
||||
"platform": platform.system(),
|
||||
"platform_release": platform.release(),
|
||||
"python_version": platform.python_version(),
|
||||
}
|
||||
|
||||
# Get CPU info
|
||||
try:
|
||||
cpu_count = os.cpu_count() or 0
|
||||
info["cpu_count"] = cpu_count
|
||||
|
||||
# Try to get more detailed CPU info on Linux
|
||||
if platform.system() == "Linux":
|
||||
try:
|
||||
with open("/proc/cpuinfo", "r") as f:
|
||||
for line in f:
|
||||
if "model name" in line:
|
||||
info["cpu_model"] = line.split(":")[1].strip()
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as e:
|
||||
info["cpu_error"] = str(e)
|
||||
|
||||
# Get memory info
|
||||
try:
|
||||
if platform.system() == "Linux":
|
||||
with open("/proc/meminfo", "r") as f:
|
||||
for line in f:
|
||||
if line.startswith("MemTotal:"):
|
||||
mem_kb = int(line.split()[1])
|
||||
info["memory_gb"] = round(mem_kb / (1024 * 1024), 1)
|
||||
break
|
||||
except Exception as e:
|
||||
info["memory_error"] = str(e)
|
||||
|
||||
return info
|
||||
|
||||
|
||||
def get_ffmpeg_info() -> Dict[str, Any]:
|
||||
"""Get FFmpeg version and configuration for debugging encoding performance."""
|
||||
try:
|
||||
# Get FFmpeg version
|
||||
result = subprocess.run(
|
||||
["ffmpeg", "-version"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
if result.returncode == 0:
|
||||
lines = result.stdout.split("\n")
|
||||
version_line = lines[0] if lines else "unknown"
|
||||
|
||||
# Check for hardware acceleration support
|
||||
hw_accel = {}
|
||||
config_result = subprocess.run(
|
||||
["ffmpeg", "-hwaccels"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
if config_result.returncode == 0:
|
||||
hw_lines = config_result.stdout.strip().split("\n")
|
||||
hw_accel["available"] = [h.strip() for h in hw_lines[1:] if h.strip()]
|
||||
|
||||
# Check encoder availability
|
||||
encoders = {}
|
||||
encoder_result = subprocess.run(
|
||||
["ffmpeg", "-encoders"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
if encoder_result.returncode == 0:
|
||||
encoder_output = encoder_result.stdout
|
||||
# Check for specific encoders we care about
|
||||
encoders["libx264"] = "libx264" in encoder_output
|
||||
encoders["h264_nvenc"] = "h264_nvenc" in encoder_output
|
||||
encoders["h264_vaapi"] = "h264_vaapi" in encoder_output
|
||||
encoders["hevc_nvenc"] = "hevc_nvenc" in encoder_output
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"version": version_line,
|
||||
"hw_acceleration": hw_accel,
|
||||
"encoders": encoders,
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"available": False,
|
||||
"error": result.stderr[:500] if result.stderr else "Unknown error",
|
||||
}
|
||||
except subprocess.TimeoutExpired:
|
||||
return {"available": False, "error": "Timeout getting FFmpeg info"}
|
||||
except FileNotFoundError:
|
||||
return {"available": False, "error": "FFmpeg not found"}
|
||||
except Exception as e:
|
||||
return {"available": False, "error": str(e)}
|
||||
|
||||
|
||||
def check_transmission_status() -> Dict[str, Any]:
|
||||
"""Check if Transmission daemon is available and responsive."""
|
||||
try:
|
||||
import transmission_rpc
|
||||
host = os.environ.get("TRANSMISSION_HOST", "localhost")
|
||||
port = int(os.environ.get("TRANSMISSION_PORT", "9091"))
|
||||
|
||||
client = transmission_rpc.Client(host=host, port=port, timeout=5)
|
||||
stats = client.session_stats()
|
||||
|
||||
# Get detailed torrent info
|
||||
torrents = client.get_torrents()
|
||||
torrent_details = []
|
||||
for t in torrents:
|
||||
torrent_info = {
|
||||
"name": t.name,
|
||||
"progress": round(t.progress, 1),
|
||||
"status": str(t.status),
|
||||
"peers": t.peers_connected if hasattr(t, 'peers_connected') else 0,
|
||||
"download_speed": round(t.rate_download / 1024, 1) if hasattr(t, 'rate_download') else 0, # KB/s
|
||||
}
|
||||
# Check if stalled (downloading but no progress and no peers)
|
||||
if t.status.downloading and t.progress < 100 and torrent_info['peers'] == 0:
|
||||
torrent_info['stalled'] = True
|
||||
torrent_details.append(torrent_info)
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"host": host,
|
||||
"port": port,
|
||||
"download_dir": stats.download_dir if hasattr(stats, 'download_dir') else None,
|
||||
"active_torrent_count": stats.active_torrent_count if hasattr(stats, 'active_torrent_count') else 0,
|
||||
"torrents": torrent_details,
|
||||
}
|
||||
except ImportError as e:
|
||||
return {
|
||||
"available": False,
|
||||
"error": f"transmission_rpc not installed: {e}",
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"available": False,
|
||||
"host": os.environ.get("TRANSMISSION_HOST", "localhost"),
|
||||
"port": int(os.environ.get("TRANSMISSION_PORT", "9091")),
|
||||
"error": str(e),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def health_check() -> Dict[str, str]:
|
||||
"""Health check endpoint."""
|
||||
return {
|
||||
"status": "healthy",
|
||||
"service": "karaoke-gen-backend"
|
||||
}
|
||||
|
||||
|
||||
async def check_flacfetch_service_status() -> Dict[str, Any]:
|
||||
"""Check if remote flacfetch service is available and healthy."""
|
||||
client = get_flacfetch_client()
|
||||
|
||||
if not client:
|
||||
return {
|
||||
"configured": False,
|
||||
"message": "Remote flacfetch service not configured (FLACFETCH_API_URL not set)",
|
||||
}
|
||||
|
||||
try:
|
||||
health = await client.health_check()
|
||||
return {
|
||||
"configured": True,
|
||||
"available": True,
|
||||
"status": health.get("status"),
|
||||
"version": health.get("version"),
|
||||
"transmission": health.get("transmission", {}),
|
||||
"disk": health.get("disk", {}),
|
||||
"providers": health.get("providers", {}),
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"configured": True,
|
||||
"available": False,
|
||||
"error": str(e),
|
||||
}
|
||||
|
||||
|
||||
async def check_encoding_worker_status() -> Dict[str, Any]:
|
||||
"""Check if GCE encoding worker is available and healthy."""
|
||||
encoding_service = get_encoding_service()
|
||||
|
||||
if not encoding_service.is_configured:
|
||||
return {
|
||||
"configured": False,
|
||||
"enabled": encoding_service.settings.use_gce_encoding,
|
||||
"message": "GCE encoding worker not configured (ENCODING_WORKER_URL or API key not set)",
|
||||
}
|
||||
|
||||
try:
|
||||
health = await encoding_service.health_check()
|
||||
return {
|
||||
"configured": True,
|
||||
"enabled": encoding_service.is_enabled,
|
||||
"available": health.get("status") == "ok",
|
||||
"status": health.get("status"),
|
||||
"active_jobs": health.get("active_jobs", 0),
|
||||
"queue_length": health.get("queue_length", 0),
|
||||
"ffmpeg_version": health.get("ffmpeg_version"),
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"configured": True,
|
||||
"enabled": encoding_service.is_enabled,
|
||||
"available": False,
|
||||
"error": str(e),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/detailed")
|
||||
async def detailed_health_check() -> Dict[str, Any]:
|
||||
"""
|
||||
Detailed health check including dependencies.
|
||||
|
||||
Use this to debug issues with Transmission, flacfetch service, etc.
|
||||
"""
|
||||
transmission_status = check_transmission_status()
|
||||
flacfetch_status = await check_flacfetch_service_status()
|
||||
encoding_status = await check_encoding_worker_status()
|
||||
|
||||
# Check email service
|
||||
email_service = get_email_service()
|
||||
email_status = {
|
||||
"configured": email_service.is_configured(),
|
||||
"provider": type(email_service.provider).__name__,
|
||||
}
|
||||
|
||||
# Check Stripe service
|
||||
stripe_service = get_stripe_service()
|
||||
stripe_status = {
|
||||
"configured": stripe_service.is_configured(),
|
||||
}
|
||||
|
||||
# Log for debugging
|
||||
if not transmission_status.get("available"):
|
||||
logger.warning(f"Local Transmission not available: {transmission_status.get('error')}")
|
||||
else:
|
||||
logger.info(f"Local Transmission available at {transmission_status.get('host')}:{transmission_status.get('port')}")
|
||||
|
||||
if flacfetch_status.get("configured") and not flacfetch_status.get("available"):
|
||||
logger.warning(f"Remote flacfetch service not available: {flacfetch_status.get('error')}")
|
||||
elif flacfetch_status.get("available"):
|
||||
logger.info(f"Remote flacfetch service healthy: {flacfetch_status.get('status')}")
|
||||
|
||||
if not email_status["configured"]:
|
||||
logger.warning("Email service not configured - magic links will not work")
|
||||
|
||||
if not stripe_status["configured"]:
|
||||
logger.warning("Stripe service not configured - payments will not work")
|
||||
|
||||
# Get system and FFmpeg info for performance analysis
|
||||
system_info = get_system_info()
|
||||
ffmpeg_info = get_ffmpeg_info()
|
||||
|
||||
return {
|
||||
"status": "healthy",
|
||||
"service": "karaoke-gen-backend",
|
||||
"version": VERSION,
|
||||
"system": system_info,
|
||||
"ffmpeg": ffmpeg_info,
|
||||
"dependencies": {
|
||||
"transmission_local": transmission_status,
|
||||
"flacfetch_remote": flacfetch_status,
|
||||
"encoding_worker": encoding_status,
|
||||
},
|
||||
"services": {
|
||||
"email": email_status,
|
||||
"stripe": stripe_status,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@router.get("/readiness")
|
||||
async def readiness_check() -> Dict[str, str]:
|
||||
"""Readiness check endpoint for Cloud Run."""
|
||||
return {
|
||||
"status": "ready",
|
||||
"service": "karaoke-gen-backend"
|
||||
}
|
||||
|
||||
@@ -0,0 +1,330 @@
|
||||
"""
|
||||
Internal API routes for worker coordination.
|
||||
|
||||
These endpoints are for internal use only (backend → workers).
|
||||
They are protected by admin authentication.
|
||||
|
||||
With Cloud Tasks integration, these endpoints may be called multiple times
|
||||
(retry on failure). Idempotency checks prevent duplicate processing.
|
||||
|
||||
Observability:
|
||||
- Extracts trace context from incoming requests (propagated via Cloud Tasks)
|
||||
- Creates worker spans linked to the original request trace
|
||||
- All logs include job_id for easy filtering in Cloud Logging
|
||||
"""
|
||||
import logging
|
||||
import asyncio
|
||||
import time
|
||||
from typing import Tuple, Optional
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Request
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.workers.audio_worker import process_audio_separation
|
||||
from backend.workers.lyrics_worker import process_lyrics_transcription
|
||||
from backend.workers.screens_worker import generate_screens
|
||||
from backend.workers.video_worker import generate_video
|
||||
from backend.workers.render_video_worker import process_render_video
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.tracing import (
|
||||
extract_trace_context,
|
||||
start_span_with_context,
|
||||
add_span_attribute,
|
||||
add_span_event,
|
||||
)
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/internal", tags=["internal"])
|
||||
|
||||
|
||||
class WorkerRequest(BaseModel):
|
||||
"""Request to trigger a worker."""
|
||||
job_id: str
|
||||
|
||||
|
||||
class WorkerResponse(BaseModel):
|
||||
"""Response from worker trigger."""
|
||||
status: str
|
||||
job_id: str
|
||||
message: str
|
||||
|
||||
|
||||
def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerResponse]:
|
||||
"""
|
||||
Check if a worker is already running or completed for this job.
|
||||
|
||||
This provides idempotency for Cloud Tasks retries - if a task is retried
|
||||
but the worker is already running or has completed, we skip processing.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to check
|
||||
worker_name: Worker name (audio, lyrics, screens, render, video)
|
||||
|
||||
Returns:
|
||||
WorkerResponse if should skip (already running/complete), None to proceed
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
logger.warning(f"[job:{job_id}] Job not found for {worker_name} worker")
|
||||
return WorkerResponse(
|
||||
status="not_found",
|
||||
job_id=job_id,
|
||||
message=f"Job {job_id} not found"
|
||||
)
|
||||
|
||||
# Check worker-specific progress in state_data
|
||||
progress_key = f"{worker_name}_progress"
|
||||
worker_progress = job.state_data.get(progress_key, {})
|
||||
stage = worker_progress.get('stage')
|
||||
|
||||
if stage == 'running':
|
||||
logger.info(f"[job:{job_id}] {worker_name.capitalize()} worker already running, skipping")
|
||||
return WorkerResponse(
|
||||
status="already_running",
|
||||
job_id=job_id,
|
||||
message=f"{worker_name.capitalize()} worker already in progress"
|
||||
)
|
||||
|
||||
if stage == 'complete':
|
||||
logger.info(f"[job:{job_id}] {worker_name.capitalize()} worker already complete, skipping")
|
||||
return WorkerResponse(
|
||||
status="already_complete",
|
||||
job_id=job_id,
|
||||
message=f"{worker_name.capitalize()} worker already completed"
|
||||
)
|
||||
|
||||
# Mark as running before starting (for idempotency on next retry)
|
||||
job_manager.update_state_data(job_id, progress_key, {'stage': 'running'})
|
||||
return None
|
||||
|
||||
|
||||
@router.post("/workers/audio", response_model=WorkerResponse)
|
||||
async def trigger_audio_worker(
|
||||
request: WorkerRequest,
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Trigger audio separation worker for a job.
|
||||
|
||||
This endpoint is called internally after job creation to start
|
||||
the audio processing track (parallel with lyrics processing).
|
||||
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
|
||||
The worker runs in the background and updates job state as it progresses.
|
||||
"""
|
||||
job_id = request.job_id
|
||||
|
||||
# Extract trace context from incoming request (propagated via Cloud Tasks)
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=audio")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("worker", "audio")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "audio")
|
||||
if skip_response:
|
||||
add_span_event("worker_skipped", {"reason": skip_response.status})
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
# This allows the HTTP response to return immediately
|
||||
# while the worker continues processing
|
||||
background_tasks.add_task(process_audio_separation, job_id)
|
||||
|
||||
add_span_event("worker_started")
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
message="Audio separation worker started"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/workers/lyrics", response_model=WorkerResponse)
|
||||
async def trigger_lyrics_worker(
|
||||
request: WorkerRequest,
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Trigger lyrics transcription worker for a job.
|
||||
|
||||
This endpoint is called internally after job creation to start
|
||||
the lyrics processing track (parallel with audio processing).
|
||||
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
|
||||
The worker runs in the background and updates job state as it progresses.
|
||||
"""
|
||||
job_id = request.job_id
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=lyrics")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("worker", "lyrics")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "lyrics")
|
||||
if skip_response:
|
||||
add_span_event("worker_skipped", {"reason": skip_response.status})
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
background_tasks.add_task(process_lyrics_transcription, job_id)
|
||||
|
||||
add_span_event("worker_started")
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
message="Lyrics transcription worker started"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/workers/screens", response_model=WorkerResponse)
|
||||
async def trigger_screens_worker(
|
||||
request: WorkerRequest,
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Trigger title/end screen generation worker.
|
||||
|
||||
This is called automatically when both audio and lyrics are complete.
|
||||
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
"""
|
||||
job_id = request.job_id
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=screens")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("worker", "screens")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "screens")
|
||||
if skip_response:
|
||||
add_span_event("worker_skipped", {"reason": skip_response.status})
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
background_tasks.add_task(generate_screens, job_id)
|
||||
|
||||
add_span_event("worker_started")
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
message="Screens generation worker started"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/workers/video", response_model=WorkerResponse)
|
||||
async def trigger_video_worker(
|
||||
request: WorkerRequest,
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Trigger final video generation and encoding worker.
|
||||
|
||||
This is called after user selects their preferred instrumental.
|
||||
This is the longest-running stage (15-20 minutes).
|
||||
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
"""
|
||||
job_id = request.job_id
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=video")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("worker", "video")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "video")
|
||||
if skip_response:
|
||||
add_span_event("worker_skipped", {"reason": skip_response.status})
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
background_tasks.add_task(generate_video, job_id)
|
||||
|
||||
add_span_event("worker_started")
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
message="Video generation worker started"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/workers/render-video", response_model=WorkerResponse)
|
||||
async def trigger_render_video_worker(
|
||||
request: WorkerRequest,
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Trigger render video worker (post-review).
|
||||
|
||||
This is called after human review is complete.
|
||||
Uses OutputGenerator from LyricsTranscriber to generate the karaoke video
|
||||
with the corrected lyrics.
|
||||
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
|
||||
Output: with_vocals.mkv in GCS
|
||||
Next state: AWAITING_INSTRUMENTAL_SELECTION
|
||||
"""
|
||||
job_id = request.job_id
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=render-video")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("worker", "render-video")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "render")
|
||||
if skip_response:
|
||||
add_span_event("worker_skipped", {"reason": skip_response.status})
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
background_tasks.add_task(process_render_video, job_id)
|
||||
|
||||
add_span_event("worker_started")
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
message="Render video worker started (post-review)"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def internal_health(
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Internal health check endpoint.
|
||||
|
||||
Used to verify the internal API is responsive.
|
||||
Requires admin authentication.
|
||||
"""
|
||||
return {"status": "healthy", "service": "karaoke-backend-internal"}
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,570 @@
|
||||
"""
|
||||
Review API routes - Compatible with LyricsTranscriber frontend.
|
||||
|
||||
These endpoints match the API that the LyricsTranscriber review frontend expects,
|
||||
allowing us to use the existing React review UI with our cloud backend.
|
||||
|
||||
Usage:
|
||||
Frontend URL: http://localhost:5173/?baseApiUrl=http://localhost:8000/api/review/{job_id}
|
||||
|
||||
The baseApiUrl includes the job_id, and all endpoints are relative to that.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any, Set, Tuple
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request, Depends
|
||||
from fastapi.responses import FileResponse, StreamingResponse
|
||||
from starlette.background import BackgroundTask
|
||||
|
||||
from backend.models.job import JobStatus
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.job_logging import job_log_context, JobLogger
|
||||
from backend.services.tracing import create_span, add_span_attribute, add_span_event
|
||||
from backend.api.dependencies import require_auth, require_review_auth
|
||||
from backend.services.auth_service import UserType
|
||||
|
||||
# LyricsTranscriber imports for preview generation
|
||||
from lyrics_transcriber.types import CorrectionResult
|
||||
from lyrics_transcriber.core.config import OutputConfig
|
||||
from lyrics_transcriber.correction.operations import CorrectionOperations
|
||||
|
||||
# Import from the unified style loader
|
||||
from karaoke_gen.style_loader import load_styles_from_gcs
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/review", tags=["review"])
|
||||
|
||||
# Store job context for the session
|
||||
# In production, this would be handled differently (e.g., session tokens)
|
||||
_job_contexts: Dict[str, Dict[str, Any]] = {}
|
||||
|
||||
# Store preview video paths for serving
|
||||
_preview_videos: Dict[str, Dict[str, str]] = {}
|
||||
|
||||
# Keep references to background tasks to prevent garbage collection
|
||||
_background_tasks: Set[asyncio.Task] = set()
|
||||
|
||||
|
||||
def _get_audio_hash(job_id: str) -> str:
|
||||
"""Generate a consistent audio hash for a job."""
|
||||
return hashlib.md5(job_id.encode()).hexdigest()
|
||||
|
||||
|
||||
@router.get("/{job_id}/ping")
|
||||
async def ping(job_id: str):
|
||||
"""Health check endpoint expected by frontend."""
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@router.get("/{job_id}/correction-data")
|
||||
async def get_correction_data(
|
||||
job_id: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Get correction data for the review interface.
|
||||
|
||||
Returns the CorrectionResult data that the frontend needs to render
|
||||
the lyrics review UI.
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job not ready for review (current status: {job.status})"
|
||||
)
|
||||
|
||||
# Get corrections URL from file_urls
|
||||
corrections_gcs = job.file_urls.get('lyrics', {}).get('corrections')
|
||||
if not corrections_gcs:
|
||||
# Try direct path
|
||||
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
if not storage.file_exists(corrections_gcs):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail="Corrections data not found. Lyrics processing may not be complete."
|
||||
)
|
||||
|
||||
# Download and return corrections data
|
||||
try:
|
||||
corrections_data = storage.download_json(corrections_gcs)
|
||||
|
||||
# Add audio hash for the frontend
|
||||
audio_hash = _get_audio_hash(job_id)
|
||||
if 'metadata' not in corrections_data:
|
||||
corrections_data['metadata'] = {}
|
||||
corrections_data['metadata']['audio_hash'] = audio_hash
|
||||
corrections_data['metadata']['artist'] = job.artist
|
||||
corrections_data['metadata']['title'] = job.title
|
||||
|
||||
# Store context for audio serving
|
||||
_job_contexts[job_id] = {
|
||||
'audio_hash': audio_hash,
|
||||
'audio_gcs_path': job.input_media_gcs_path
|
||||
}
|
||||
|
||||
# Transition to IN_REVIEW if not already
|
||||
if job.status == JobStatus.AWAITING_REVIEW:
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.IN_REVIEW,
|
||||
message="User opened review interface"
|
||||
)
|
||||
|
||||
logger.info(f"Job {job_id}: Serving correction data for review")
|
||||
return corrections_data
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Error loading corrections: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Error loading corrections: {str(e)}")
|
||||
|
||||
|
||||
@router.get("/{job_id}/audio/{audio_hash}")
|
||||
async def get_audio_with_hash(
|
||||
job_id: str,
|
||||
audio_hash: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""Stream the audio file for playback (with hash parameter)."""
|
||||
return await _stream_audio(job_id)
|
||||
|
||||
|
||||
@router.get("/{job_id}/audio/")
|
||||
@router.get("/{job_id}/audio")
|
||||
async def get_audio_no_hash(
|
||||
job_id: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""Stream the audio file for playback (without hash parameter)."""
|
||||
return await _stream_audio(job_id)
|
||||
|
||||
|
||||
async def _stream_audio(job_id: str):
|
||||
"""
|
||||
Stream the audio file for playback in the review interface.
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
audio_gcs_path = job.input_media_gcs_path
|
||||
if not audio_gcs_path:
|
||||
raise HTTPException(status_code=404, detail="Audio file not found")
|
||||
|
||||
# Download to temp file and stream
|
||||
try:
|
||||
# Create temp file
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".flac") as tmp:
|
||||
tmp_path = tmp.name
|
||||
|
||||
storage.download_file(audio_gcs_path, tmp_path)
|
||||
|
||||
# Determine content type
|
||||
if audio_gcs_path.endswith('.flac'):
|
||||
media_type = "audio/flac"
|
||||
elif audio_gcs_path.endswith('.wav'):
|
||||
media_type = "audio/wav"
|
||||
elif audio_gcs_path.endswith('.mp3'):
|
||||
media_type = "audio/mpeg"
|
||||
else:
|
||||
media_type = "audio/mpeg"
|
||||
|
||||
logger.info(f"Job {job_id}: Streaming audio for review")
|
||||
|
||||
return FileResponse(
|
||||
tmp_path,
|
||||
media_type=media_type,
|
||||
filename=os.path.basename(audio_gcs_path),
|
||||
background=BackgroundTask(os.unlink, tmp_path),
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Error streaming audio: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Error streaming audio: {str(e)}")
|
||||
|
||||
|
||||
@router.post("/{job_id}/complete")
|
||||
async def complete_review(
|
||||
job_id: str,
|
||||
updated_data: Dict[str, Any],
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Complete the review and save corrected lyrics.
|
||||
|
||||
This endpoint receives the updated correction data from the frontend
|
||||
and saves it, then triggers the render video worker.
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job not in review state (current status: {job.status})"
|
||||
)
|
||||
|
||||
try:
|
||||
# Save updated corrections to GCS
|
||||
corrections_gcs_path = f"jobs/{job_id}/lyrics/corrections_updated.json"
|
||||
storage.upload_json(corrections_gcs_path, updated_data)
|
||||
job_manager.update_file_url(job_id, 'lyrics', 'corrections_updated', corrections_gcs_path)
|
||||
|
||||
logger.info(f"Job {job_id}: Saved updated corrections")
|
||||
|
||||
# Transition to REVIEW_COMPLETE
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.REVIEW_COMPLETE,
|
||||
progress=70,
|
||||
message="Review complete, rendering video with corrected lyrics"
|
||||
)
|
||||
|
||||
# Trigger render video worker
|
||||
from backend.services.worker_service import get_worker_service
|
||||
worker_service = get_worker_service()
|
||||
|
||||
# Run in background, keep reference to prevent garbage collection
|
||||
task = asyncio.create_task(worker_service.trigger_render_video_worker(job_id))
|
||||
_background_tasks.add(task)
|
||||
task.add_done_callback(_background_tasks.discard)
|
||||
|
||||
logger.info(f"Job {job_id}: Review complete, triggered render video worker")
|
||||
|
||||
return {"status": "success"}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Error completing review: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Error completing review: {str(e)}")
|
||||
|
||||
|
||||
@router.post("/{job_id}/handlers")
|
||||
async def update_handlers(
|
||||
job_id: str,
|
||||
enabled_handlers: list,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Update enabled correction handlers (optional feature).
|
||||
|
||||
For now, just acknowledge the request - full handler support
|
||||
would require re-running correction.
|
||||
"""
|
||||
logger.info(f"Job {job_id}: Handler update requested (not implemented)")
|
||||
return {"status": "success", "message": "Handler updates not yet implemented"}
|
||||
|
||||
|
||||
@router.post("/{job_id}/add-lyrics")
|
||||
async def add_lyrics(
|
||||
job_id: str,
|
||||
data: Dict[str, str],
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Add custom lyrics source and rerun correction.
|
||||
|
||||
Uses the LyricsTranscriber's CorrectionOperations to add a new lyrics source
|
||||
and regenerate corrections with the new source included.
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
# Job must be in review state to add lyrics
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job not in review state (current status: {job.status})"
|
||||
)
|
||||
|
||||
source = data.get("source", "").strip()
|
||||
lyrics_text = data.get("lyrics", "").strip()
|
||||
|
||||
logger.info(f"Job {job_id}: Adding lyrics source '{source}' with {len(lyrics_text)} characters")
|
||||
|
||||
# Use tracing and job_log_context for full observability
|
||||
with create_span("add-lyrics", {"job_id": job_id, "source": source, "lyrics_length": len(lyrics_text)}) as span:
|
||||
with job_log_context(job_id, worker="add-lyrics"):
|
||||
try:
|
||||
# Create temp directory for this operation
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download current corrections.json
|
||||
with create_span("download-corrections") as download_span:
|
||||
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
corrections_path = os.path.join(temp_dir, "corrections.json")
|
||||
storage.download_file(corrections_gcs, corrections_path)
|
||||
download_span.set_attribute("gcs_path", corrections_gcs)
|
||||
|
||||
with open(corrections_path, 'r', encoding='utf-8') as f:
|
||||
original_data = json.load(f)
|
||||
|
||||
# Load as CorrectionResult
|
||||
correction_result = CorrectionResult.from_dict(original_data)
|
||||
add_span_event("corrections_loaded", {
|
||||
"segments": len(correction_result.corrected_segments) if correction_result.corrected_segments else 0,
|
||||
"reference_sources": len(correction_result.reference_lyrics) if correction_result.reference_lyrics else 0,
|
||||
})
|
||||
|
||||
# Set up cache directory
|
||||
cache_dir = os.path.join(temp_dir, "cache")
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
|
||||
# Add lyrics source using CorrectionOperations (this is the heavy operation)
|
||||
with create_span("correction-operations-add-lyrics") as correction_span:
|
||||
correction_span.set_attribute("source_name", source)
|
||||
updated_result = CorrectionOperations.add_lyrics_source(
|
||||
correction_result=correction_result,
|
||||
source=source,
|
||||
lyrics_text=lyrics_text,
|
||||
cache_dir=cache_dir,
|
||||
logger=logger
|
||||
)
|
||||
add_span_event("correction_complete", {
|
||||
"new_segments": len(updated_result.corrected_segments) if updated_result.corrected_segments else 0,
|
||||
})
|
||||
|
||||
# Add audio hash for the frontend
|
||||
audio_hash = _get_audio_hash(job_id)
|
||||
if not updated_result.metadata:
|
||||
updated_result.metadata = {}
|
||||
updated_result.metadata['audio_hash'] = audio_hash
|
||||
updated_result.metadata['artist'] = job.artist
|
||||
updated_result.metadata['title'] = job.title
|
||||
|
||||
# Upload updated corrections back to GCS
|
||||
with create_span("upload-corrections") as upload_span:
|
||||
updated_data = updated_result.to_dict()
|
||||
storage.upload_json(corrections_gcs, updated_data)
|
||||
upload_span.set_attribute("gcs_path", corrections_gcs)
|
||||
|
||||
logger.info(f"Job {job_id}: Successfully added lyrics source '{source}'")
|
||||
span.set_attribute("success", True)
|
||||
|
||||
return {"status": "success", "data": updated_data}
|
||||
|
||||
except ValueError as e:
|
||||
# ValueError from CorrectionOperations (e.g., duplicate source name)
|
||||
logger.warning(f"Job {job_id}: Invalid add lyrics request: {e}")
|
||||
span.set_attribute("error", str(e))
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Failed to add lyrics: {e}", exc_info=True)
|
||||
span.set_attribute("error", str(e))
|
||||
raise HTTPException(status_code=500, detail=f"Failed to add lyrics: {str(e)}")
|
||||
|
||||
|
||||
@router.post("/{job_id}/preview-video")
|
||||
async def generate_preview_video(
|
||||
job_id: str,
|
||||
updated_data: Dict[str, Any],
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Generate a preview video with the current corrections.
|
||||
|
||||
Uses the LyricsTranscriber's CorrectionOperations to generate a 360p preview
|
||||
video with the user's current corrections applied.
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
# Job must be in review state to generate preview
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job not in review state (current status: {job.status})"
|
||||
)
|
||||
|
||||
logger.info(f"Job {job_id}: Generating preview video")
|
||||
|
||||
# Use tracing and job_log_context for full observability
|
||||
with create_span("generate-preview-video", {"job_id": job_id}) as span:
|
||||
with job_log_context(job_id, worker="preview"):
|
||||
try:
|
||||
# Create temp directory for this preview operation
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# 1. Download original corrections.json (has full structure)
|
||||
with create_span("download-corrections-and-audio") as download_span:
|
||||
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
corrections_path = os.path.join(temp_dir, "corrections.json")
|
||||
storage.download_file(corrections_gcs, corrections_path)
|
||||
|
||||
with open(corrections_path, 'r', encoding='utf-8') as f:
|
||||
original_data = json.load(f)
|
||||
|
||||
# 2. Download input audio
|
||||
audio_path = os.path.join(temp_dir, "audio.flac")
|
||||
storage.download_file(job.input_media_gcs_path, audio_path)
|
||||
download_span.set_attribute("audio_gcs_path", job.input_media_gcs_path)
|
||||
|
||||
# 3. Load original as CorrectionResult
|
||||
correction_result = CorrectionResult.from_dict(original_data)
|
||||
add_span_event("corrections_loaded")
|
||||
|
||||
# 4. Get or create styles file for preview using unified style loader
|
||||
with create_span("load-styles") as styles_span:
|
||||
styles_path, _ = load_styles_from_gcs(
|
||||
style_params_gcs_path=job.style_params_gcs_path,
|
||||
style_assets=job.style_assets,
|
||||
temp_dir=temp_dir,
|
||||
download_func=storage.download_file,
|
||||
logger=logger,
|
||||
)
|
||||
styles_span.set_attribute("styles_path", styles_path)
|
||||
|
||||
# 5. Set up output config for preview
|
||||
output_dir = os.path.join(temp_dir, "output")
|
||||
cache_dir = os.path.join(temp_dir, "cache")
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
|
||||
output_config = OutputConfig(
|
||||
output_styles_json=styles_path,
|
||||
output_dir=output_dir,
|
||||
cache_dir=cache_dir,
|
||||
video_resolution="360p",
|
||||
)
|
||||
|
||||
# 6. Generate preview video using CorrectionOperations (heavy operation)
|
||||
with create_span("render-preview-video") as render_span:
|
||||
render_span.set_attribute("resolution", "360p")
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
correction_result=correction_result,
|
||||
updated_data=updated_data,
|
||||
output_config=output_config,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
logger=logger
|
||||
)
|
||||
add_span_event("render_complete")
|
||||
|
||||
# 7. Upload preview video to GCS for serving
|
||||
preview_hash = result["preview_hash"]
|
||||
video_path = result["video_path"]
|
||||
|
||||
with create_span("upload-preview-video") as upload_span:
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
storage.upload_file(video_path, preview_gcs_path)
|
||||
upload_span.set_attribute("gcs_path", preview_gcs_path)
|
||||
|
||||
# Store the GCS path for serving
|
||||
if job_id not in _preview_videos:
|
||||
_preview_videos[job_id] = {}
|
||||
_preview_videos[job_id][preview_hash] = preview_gcs_path
|
||||
|
||||
logger.info(f"Job {job_id}: Preview video generated: {preview_hash}")
|
||||
span.set_attribute("preview_hash", preview_hash)
|
||||
span.set_attribute("success", True)
|
||||
|
||||
return {"status": "success", "preview_hash": preview_hash}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Failed to generate preview video: {e}", exc_info=True)
|
||||
span.set_attribute("error", str(e))
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=f"Failed to generate preview video: {e}"
|
||||
) from e
|
||||
|
||||
|
||||
@router.get("/{job_id}/preview-video/{preview_hash}")
|
||||
async def get_preview_video(
|
||||
job_id: str,
|
||||
preview_hash: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""Stream the generated preview video."""
|
||||
storage = StorageService()
|
||||
|
||||
# Check in-memory cache first
|
||||
preview_gcs_path = None
|
||||
if job_id in _preview_videos and preview_hash in _preview_videos[job_id]:
|
||||
preview_gcs_path = _preview_videos[job_id][preview_hash]
|
||||
else:
|
||||
# Try standard path
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
if not storage.file_exists(preview_gcs_path):
|
||||
raise HTTPException(status_code=404, detail="Preview video not found")
|
||||
|
||||
try:
|
||||
# Download to temp file and stream
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".mp4") as tmp:
|
||||
tmp_path = tmp.name
|
||||
|
||||
storage.download_file(preview_gcs_path, tmp_path)
|
||||
|
||||
logger.info(f"Job {job_id}: Streaming preview video {preview_hash}")
|
||||
|
||||
return FileResponse(
|
||||
tmp_path,
|
||||
media_type="video/mp4",
|
||||
filename=f"preview_{preview_hash}.mp4",
|
||||
headers={
|
||||
"Accept-Ranges": "bytes",
|
||||
"Content-Disposition": "inline",
|
||||
"Cache-Control": "no-cache",
|
||||
},
|
||||
background=BackgroundTask(os.unlink, tmp_path),
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Error streaming preview video: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Error streaming preview video: {str(e)}")
|
||||
|
||||
|
||||
@router.post("/{job_id}/v1/annotations")
|
||||
async def submit_annotation(
|
||||
job_id: str,
|
||||
annotation: Dict[str, Any],
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Submit a correction annotation for ML training data.
|
||||
|
||||
For now, just log and acknowledge - full annotation support
|
||||
would require a database.
|
||||
"""
|
||||
logger.info(f"Job {job_id}: Annotation submitted (logged but not stored)")
|
||||
return {"status": "success", "annotation_id": "stub"}
|
||||
|
||||
|
||||
@router.get("/{job_id}/v1/annotations/stats")
|
||||
async def get_annotation_stats(
|
||||
job_id: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""Get annotation statistics."""
|
||||
return {
|
||||
"total_annotations": 0,
|
||||
"by_type": {},
|
||||
"message": "Annotation stats not yet implemented"
|
||||
}
|
||||
@@ -0,0 +1,162 @@
|
||||
"""
|
||||
Theme API routes for listing and retrieving theme information.
|
||||
|
||||
Themes are pre-created style configurations that users can select when creating jobs.
|
||||
These endpoints are public (no authentication required) to allow theme selection
|
||||
before job creation.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
|
||||
from backend.models.theme import ThemeDetailResponse, ThemesListResponse
|
||||
from backend.services.theme_service import get_theme_service
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/themes", tags=["themes"])
|
||||
|
||||
|
||||
@router.get("", response_model=ThemesListResponse)
|
||||
async def list_themes() -> ThemesListResponse:
|
||||
"""
|
||||
List all available themes.
|
||||
|
||||
Returns a list of themes with their metadata and preview image URLs.
|
||||
Preview URLs are signed and valid for 60 minutes.
|
||||
|
||||
This endpoint is public and does not require authentication.
|
||||
"""
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
themes = theme_service.list_themes()
|
||||
return ThemesListResponse(themes=themes)
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing themes: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load themes"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}", response_model=ThemeDetailResponse)
|
||||
async def get_theme(theme_id: str) -> ThemeDetailResponse:
|
||||
"""
|
||||
Get detailed information about a specific theme.
|
||||
|
||||
Returns the full theme details including style parameters.
|
||||
This can be used to preview theme settings or inspect configuration.
|
||||
|
||||
Args:
|
||||
theme_id: The unique identifier of the theme
|
||||
|
||||
Returns:
|
||||
ThemeDetailResponse with full theme details
|
||||
|
||||
Raises:
|
||||
404: Theme not found
|
||||
"""
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
theme = theme_service.get_theme(theme_id)
|
||||
|
||||
if theme is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
)
|
||||
|
||||
return ThemeDetailResponse(theme=theme)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load theme"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}/preview")
|
||||
async def get_theme_preview(theme_id: str) -> dict:
|
||||
"""
|
||||
Get a signed URL for the theme's preview image.
|
||||
|
||||
This endpoint is useful when you need just the preview URL
|
||||
without loading the full theme details.
|
||||
|
||||
Args:
|
||||
theme_id: The unique identifier of the theme
|
||||
|
||||
Returns:
|
||||
Object with preview_url field
|
||||
|
||||
Raises:
|
||||
404: Theme not found or no preview available
|
||||
"""
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
|
||||
if not theme_service.theme_exists(theme_id):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
)
|
||||
|
||||
theme = theme_service.get_theme(theme_id)
|
||||
if theme is None or theme.preview_url is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"No preview available for theme: {theme_id}"
|
||||
)
|
||||
|
||||
return {"preview_url": theme.preview_url}
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting preview for theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load theme preview"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}/youtube-description")
|
||||
async def get_theme_youtube_description(theme_id: str) -> dict:
|
||||
"""
|
||||
Get the YouTube description template for a theme.
|
||||
|
||||
Some themes include a default YouTube video description template
|
||||
that can be used when uploading videos.
|
||||
|
||||
Args:
|
||||
theme_id: The unique identifier of the theme
|
||||
|
||||
Returns:
|
||||
Object with description field containing the template text,
|
||||
or null if no template is available
|
||||
|
||||
Raises:
|
||||
404: Theme not found
|
||||
"""
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
|
||||
if not theme_service.theme_exists(theme_id):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
)
|
||||
|
||||
description = theme_service.get_youtube_description(theme_id)
|
||||
return {"description": description}
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting YouTube description for theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load YouTube description"
|
||||
)
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,166 @@
|
||||
"""
|
||||
Configuration management for the karaoke generation backend.
|
||||
"""
|
||||
import os
|
||||
import logging
|
||||
from typing import Optional, Dict
|
||||
from pydantic_settings import BaseSettings
|
||||
from google.cloud import secretmanager
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
"""Application settings."""
|
||||
|
||||
# Google Cloud
|
||||
google_cloud_project: str = os.getenv("GOOGLE_CLOUD_PROJECT", "")
|
||||
gcs_bucket_name: str = os.getenv("GCS_BUCKET_NAME", "karaoke-gen-storage")
|
||||
gcs_temp_bucket: str = os.getenv("GCS_TEMP_BUCKET", "karaoke-gen-temp")
|
||||
gcs_output_bucket: str = os.getenv("GCS_OUTPUT_BUCKET", "karaoke-gen-outputs")
|
||||
firestore_collection: str = os.getenv("FIRESTORE_COLLECTION", "jobs")
|
||||
|
||||
# Audio Separator API (for GPU processing)
|
||||
audio_separator_api_url: Optional[str] = os.getenv("AUDIO_SEPARATOR_API_URL")
|
||||
|
||||
# External APIs (can be set via env or Secret Manager)
|
||||
audioshake_api_key: Optional[str] = os.getenv("AUDIOSHAKE_API_KEY")
|
||||
genius_api_key: Optional[str] = os.getenv("GENIUS_API_KEY")
|
||||
spotify_cookie: Optional[str] = os.getenv("SPOTIFY_COOKIE_SP_DC")
|
||||
rapidapi_key: Optional[str] = os.getenv("RAPIDAPI_KEY")
|
||||
|
||||
# Authentication
|
||||
admin_tokens: Optional[str] = os.getenv("ADMIN_TOKENS") # Comma-separated list
|
||||
|
||||
# Application
|
||||
environment: str = os.getenv("ENVIRONMENT", "development")
|
||||
log_level: str = os.getenv("LOG_LEVEL", "INFO")
|
||||
|
||||
# Processing
|
||||
max_concurrent_jobs: int = int(os.getenv("MAX_CONCURRENT_JOBS", "5"))
|
||||
job_timeout_seconds: int = int(os.getenv("JOB_TIMEOUT_SECONDS", "3600"))
|
||||
|
||||
# Agentic AI Correction (for lyrics correction via LLM)
|
||||
# When enabled, uses Gemini via Vertex AI for intelligent lyrics correction
|
||||
use_agentic_ai: bool = os.getenv("USE_AGENTIC_AI", "true").lower() in ("true", "1", "yes")
|
||||
agentic_ai_model: str = os.getenv("AGENTIC_AI_MODEL", "vertexai/gemini-3-flash-preview")
|
||||
# Timeout for agentic correction in seconds. If correction takes longer, abort and
|
||||
# use uncorrected transcription - human review will fix any issues.
|
||||
agentic_correction_timeout_seconds: int = int(os.getenv("AGENTIC_CORRECTION_TIMEOUT_SECONDS", "180"))
|
||||
|
||||
# Cloud Tasks (for scalable worker coordination)
|
||||
# When enabled, workers are triggered via Cloud Tasks for guaranteed delivery
|
||||
# When disabled (default), workers are triggered via direct HTTP (for development)
|
||||
enable_cloud_tasks: bool = os.getenv("ENABLE_CLOUD_TASKS", "false").lower() in ("true", "1", "yes")
|
||||
gcp_region: str = os.getenv("GCP_REGION", "us-central1")
|
||||
|
||||
# Cloud Run Jobs (for long-running video encoding)
|
||||
# When enabled AND enable_cloud_tasks is true, video worker uses Cloud Run Jobs
|
||||
# instead of Cloud Tasks. This supports encoding times >30 minutes (up to 24 hours).
|
||||
# Default is false - Cloud Tasks is sufficient for most videos (15-20 min).
|
||||
use_cloud_run_jobs_for_video: bool = os.getenv("USE_CLOUD_RUN_JOBS_FOR_VIDEO", "false").lower() in ("true", "1", "yes")
|
||||
|
||||
# GCE Encoding Worker (for high-performance video encoding)
|
||||
# When enabled, video encoding is offloaded to a dedicated C4 GCE instance
|
||||
# with faster CPU (Intel Granite Rapids 3.9 GHz) instead of Cloud Run.
|
||||
# This provides 2-3x faster encoding times for CPU-bound FFmpeg libx264 encoding.
|
||||
use_gce_encoding: bool = os.getenv("USE_GCE_ENCODING", "false").lower() in ("true", "1", "yes")
|
||||
encoding_worker_url: Optional[str] = os.getenv("ENCODING_WORKER_URL") # e.g., http://136.119.50.148:8080
|
||||
encoding_worker_api_key: Optional[str] = os.getenv("ENCODING_WORKER_API_KEY")
|
||||
|
||||
# Storage paths
|
||||
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
|
||||
|
||||
# Worker logs storage mode
|
||||
# When enabled, worker logs are stored in a Firestore subcollection (jobs/{job_id}/logs)
|
||||
# instead of an embedded array. This avoids the 1MB document size limit.
|
||||
# Default is true for new deployments.
|
||||
use_log_subcollection: bool = os.getenv("USE_LOG_SUBCOLLECTION", "true").lower() in ("true", "1", "yes")
|
||||
|
||||
# Flacfetch remote service (for torrent downloads)
|
||||
# When configured, audio search uses the remote flacfetch HTTP API instead of local flacfetch.
|
||||
# This is required for torrent downloads since Cloud Run doesn't support BitTorrent.
|
||||
flacfetch_api_url: Optional[str] = os.getenv("FLACFETCH_API_URL") # e.g., http://10.0.0.5:8080
|
||||
flacfetch_api_key: Optional[str] = os.getenv("FLACFETCH_API_KEY")
|
||||
|
||||
# Default distribution settings (can be overridden per-request)
|
||||
default_dropbox_path: Optional[str] = os.getenv("DEFAULT_DROPBOX_PATH")
|
||||
default_gdrive_folder_id: Optional[str] = os.getenv("DEFAULT_GDRIVE_FOLDER_ID")
|
||||
# Strip whitespace/newlines from webhook URL - common issue when env vars are set with trailing newlines
|
||||
default_discord_webhook_url: Optional[str] = (
|
||||
os.getenv("DEFAULT_DISCORD_WEBHOOK_URL", "").strip() or None
|
||||
)
|
||||
|
||||
# Default values for web service jobs (YouTube/Dropbox distribution)
|
||||
default_enable_youtube_upload: bool = os.getenv("DEFAULT_ENABLE_YOUTUBE_UPLOAD", "false").lower() in ("true", "1", "yes")
|
||||
default_brand_prefix: Optional[str] = os.getenv("DEFAULT_BRAND_PREFIX")
|
||||
default_youtube_description: str = os.getenv(
|
||||
"DEFAULT_YOUTUBE_DESCRIPTION",
|
||||
"Karaoke video created with Nomad Karaoke (https://nomadkaraoke.com)\n\n"
|
||||
"AI-powered vocal separation and synchronized lyrics.\n\n"
|
||||
"#karaoke #music #singing #instrumental #lyrics"
|
||||
)
|
||||
|
||||
# Secret Manager cache
|
||||
_secret_cache: Dict[str, str] = {}
|
||||
|
||||
class Config:
|
||||
env_file = ".env"
|
||||
case_sensitive = False
|
||||
|
||||
def get_secret(self, secret_id: str) -> Optional[str]:
|
||||
"""
|
||||
Get a secret from Google Secret Manager.
|
||||
|
||||
Caches secrets in memory to avoid repeated API calls.
|
||||
Falls back to environment variables if Secret Manager unavailable.
|
||||
|
||||
Args:
|
||||
secret_id: Secret name (e.g., "audioshake-api-key")
|
||||
|
||||
Returns:
|
||||
Secret value or None if not found
|
||||
"""
|
||||
# Check cache first
|
||||
if secret_id in self._secret_cache:
|
||||
return self._secret_cache[secret_id]
|
||||
|
||||
# Check environment variable (development mode)
|
||||
env_var = secret_id.upper().replace('-', '_')
|
||||
env_value = os.getenv(env_var)
|
||||
if env_value:
|
||||
logger.debug(f"Using {secret_id} from environment variable")
|
||||
self._secret_cache[secret_id] = env_value
|
||||
return env_value
|
||||
|
||||
# Try Secret Manager (production mode)
|
||||
if not self.google_cloud_project:
|
||||
logger.warning(f"Cannot fetch secret {secret_id}: No GCP project configured")
|
||||
return None
|
||||
|
||||
try:
|
||||
client = secretmanager.SecretManagerServiceClient()
|
||||
name = f"projects/{self.google_cloud_project}/secrets/{secret_id}/versions/latest"
|
||||
response = client.access_secret_version(request={"name": name})
|
||||
# Strip whitespace/newlines - common issue when secrets are created with trailing newlines
|
||||
secret_value = response.payload.data.decode('UTF-8').strip()
|
||||
|
||||
# Cache it
|
||||
self._secret_cache[secret_id] = secret_value
|
||||
logger.info(f"Loaded secret {secret_id} from Secret Manager")
|
||||
return secret_value
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load secret {secret_id} from Secret Manager: {e}")
|
||||
return None
|
||||
|
||||
|
||||
# Global settings instance
|
||||
settings = Settings()
|
||||
|
||||
|
||||
def get_settings() -> Settings:
|
||||
"""Get the global settings instance."""
|
||||
return settings
|
||||
|
||||
+133
@@ -0,0 +1,133 @@
|
||||
"""
|
||||
FastAPI application entry point for karaoke generation backend.
|
||||
"""
|
||||
import logging
|
||||
from contextlib import asynccontextmanager
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from backend.config import settings
|
||||
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin
|
||||
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
|
||||
from backend.services.structured_logging import setup_structured_logging
|
||||
from backend.middleware.audit_logging import AuditLoggingMiddleware
|
||||
|
||||
|
||||
from backend.version import VERSION
|
||||
|
||||
|
||||
# Configure structured logging (JSON in Cloud Run, human-readable locally)
|
||||
# This must happen before any logging calls
|
||||
setup_structured_logging()
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Initialize OpenTelemetry tracing (must happen before app creation)
|
||||
tracing_enabled = setup_tracing(
|
||||
service_name="karaoke-backend",
|
||||
service_version=VERSION,
|
||||
enable_in_dev=False, # Set to True to enable tracing locally
|
||||
)
|
||||
|
||||
|
||||
async def validate_credentials_on_startup():
|
||||
"""Validate OAuth credentials on startup and send alerts if needed."""
|
||||
try:
|
||||
from backend.services.credential_manager import get_credential_manager, CredentialStatus
|
||||
|
||||
manager = get_credential_manager()
|
||||
results = manager.check_all_credentials()
|
||||
|
||||
invalid_services = [
|
||||
result for result in results.values()
|
||||
if result.status in (CredentialStatus.INVALID, CredentialStatus.EXPIRED)
|
||||
]
|
||||
|
||||
if invalid_services:
|
||||
logger.warning(f"Some OAuth credentials need attention:")
|
||||
for result in invalid_services:
|
||||
logger.warning(f" - {result.service}: {result.message}")
|
||||
|
||||
# Try to send Discord alert
|
||||
discord_url = settings.get_secret("discord-alert-webhook") if hasattr(settings, 'get_secret') else None
|
||||
if discord_url:
|
||||
manager.send_credential_alert(invalid_services, discord_url)
|
||||
logger.info("Sent credential alert to Discord")
|
||||
else:
|
||||
logger.info("All OAuth credentials validated successfully")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to validate credentials on startup: {e}")
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
"""Lifespan event handler for startup and shutdown."""
|
||||
# Startup
|
||||
logger.info("Starting karaoke generation backend")
|
||||
logger.info(f"Environment: {settings.environment}")
|
||||
logger.info(f"GCS Bucket: {settings.gcs_bucket_name}")
|
||||
logger.info(f"Tracing enabled: {tracing_enabled}")
|
||||
|
||||
# Validate OAuth credentials (non-blocking)
|
||||
try:
|
||||
await validate_credentials_on_startup()
|
||||
except Exception as e:
|
||||
logger.error(f"Credential validation failed: {e}")
|
||||
|
||||
yield
|
||||
|
||||
# Shutdown
|
||||
logger.info("Shutting down karaoke generation backend")
|
||||
|
||||
|
||||
# Create FastAPI app
|
||||
app = FastAPI(
|
||||
title="Karaoke Generator API",
|
||||
description="Backend API for web-based karaoke video generation",
|
||||
version=VERSION,
|
||||
lifespan=lifespan
|
||||
)
|
||||
|
||||
# Instrument FastAPI with OpenTelemetry (adds automatic spans for all requests)
|
||||
if tracing_enabled:
|
||||
instrument_app(app)
|
||||
|
||||
# Configure CORS
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"], # Configure this properly for production
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
# Add audit logging middleware (captures all requests with request_id for correlation)
|
||||
app.add_middleware(AuditLoggingMiddleware)
|
||||
|
||||
# Include routers
|
||||
app.include_router(health.router, prefix="/api")
|
||||
app.include_router(jobs.router, prefix="/api")
|
||||
app.include_router(file_upload.router, prefix="/api") # File upload endpoint
|
||||
app.include_router(internal.router, prefix="/api") # Internal worker endpoints
|
||||
app.include_router(review.router, prefix="/api") # Review UI compatibility endpoints
|
||||
app.include_router(auth.router, prefix="/api") # OAuth credential management
|
||||
app.include_router(audio_search.router, prefix="/api") # Audio search (artist+title mode)
|
||||
app.include_router(themes.router, prefix="/api") # Theme selection for styles
|
||||
app.include_router(users.router, prefix="/api") # User auth, credits, and Stripe webhooks
|
||||
app.include_router(admin.router, prefix="/api") # Admin dashboard and management
|
||||
|
||||
|
||||
@app.get("/")
|
||||
async def root():
|
||||
"""Root endpoint."""
|
||||
return {
|
||||
"service": "karaoke-gen-backend",
|
||||
"version": VERSION,
|
||||
"status": "running"
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=8080)
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
"""Middleware package for FastAPI application."""
|
||||
|
||||
from backend.middleware.audit_logging import AuditLoggingMiddleware
|
||||
|
||||
__all__ = ["AuditLoggingMiddleware"]
|
||||
@@ -0,0 +1,124 @@
|
||||
"""
|
||||
Audit logging middleware for tracking all HTTP requests.
|
||||
|
||||
This middleware captures request metadata and logs it to Cloud Logging
|
||||
for audit trail and user activity investigation purposes.
|
||||
"""
|
||||
import logging
|
||||
import time
|
||||
import uuid
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.requests import Request
|
||||
from starlette.responses import Response
|
||||
|
||||
|
||||
logger = logging.getLogger("audit")
|
||||
|
||||
# Endpoints to exclude from audit logging (high-frequency health checks)
|
||||
EXCLUDED_PATHS = {
|
||||
"/",
|
||||
"/api/health",
|
||||
"/api/health/detailed",
|
||||
"/api/readiness",
|
||||
"/healthz",
|
||||
"/ready",
|
||||
}
|
||||
|
||||
|
||||
class AuditLoggingMiddleware(BaseHTTPMiddleware):
|
||||
"""
|
||||
Middleware that logs all HTTP requests for audit purposes.
|
||||
|
||||
Captures:
|
||||
- request_id: UUID for correlating with auth logs
|
||||
- method: HTTP method
|
||||
- path: Request path
|
||||
- status_code: Response status
|
||||
- latency_ms: Request duration
|
||||
- client_ip: Client IP (from X-Forwarded-For for proxied requests)
|
||||
- user_agent: Browser/client identifier
|
||||
|
||||
The request_id is stored in request.state and added to response headers,
|
||||
allowing correlation with auth logs that capture user_email.
|
||||
"""
|
||||
|
||||
async def dispatch(self, request: Request, call_next) -> Response:
|
||||
# Skip excluded paths (health checks)
|
||||
if request.url.path in EXCLUDED_PATHS:
|
||||
return await call_next(request)
|
||||
|
||||
# Generate unique request ID for correlation
|
||||
request_id = str(uuid.uuid4())
|
||||
request.state.request_id = request_id
|
||||
|
||||
# Capture start time
|
||||
start_time = time.time()
|
||||
|
||||
# Extract client info
|
||||
client_ip = self._get_client_ip(request)
|
||||
user_agent = request.headers.get("user-agent", "")
|
||||
|
||||
# Process request
|
||||
try:
|
||||
response = await call_next(request)
|
||||
except Exception:
|
||||
# Log failed requests too (exception() auto-includes stack trace)
|
||||
latency_ms = int((time.time() - start_time) * 1000)
|
||||
logger.exception(
|
||||
"request_audit_error",
|
||||
extra={
|
||||
"request_id": request_id,
|
||||
"method": request.method,
|
||||
"path": request.url.path,
|
||||
"query_string": str(request.query_params) if request.query_params else None,
|
||||
"latency_ms": latency_ms,
|
||||
"client_ip": client_ip,
|
||||
"user_agent": user_agent[:200] if user_agent else None,
|
||||
"audit_type": "http_request",
|
||||
}
|
||||
)
|
||||
raise
|
||||
|
||||
# Calculate latency
|
||||
latency_ms = int((time.time() - start_time) * 1000)
|
||||
|
||||
# Log audit entry (INFO level for successful requests)
|
||||
log_level = logging.WARNING if response.status_code >= 400 else logging.INFO
|
||||
logger.log(
|
||||
log_level,
|
||||
"request_audit",
|
||||
extra={
|
||||
"request_id": request_id,
|
||||
"method": request.method,
|
||||
"path": request.url.path,
|
||||
"query_string": str(request.query_params) if request.query_params else None,
|
||||
"status_code": response.status_code,
|
||||
"latency_ms": latency_ms,
|
||||
"client_ip": client_ip,
|
||||
"user_agent": user_agent[:200] if user_agent else None,
|
||||
"audit_type": "http_request",
|
||||
}
|
||||
)
|
||||
|
||||
# Add request_id to response headers for debugging/correlation
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
|
||||
return response
|
||||
|
||||
def _get_client_ip(self, request: Request) -> str:
|
||||
"""
|
||||
Extract client IP address, handling proxy scenarios.
|
||||
|
||||
Cloud Run and other proxies set X-Forwarded-For header.
|
||||
"""
|
||||
# Check X-Forwarded-For for proxy scenarios (Cloud Run, load balancers)
|
||||
forwarded = request.headers.get("x-forwarded-for", "")
|
||||
if forwarded:
|
||||
# First IP is the original client
|
||||
return forwarded.split(",")[0].strip()
|
||||
|
||||
# Fall back to direct connection
|
||||
if request.client:
|
||||
return request.client.host
|
||||
|
||||
return "unknown"
|
||||
Whitespace-only changes.
@@ -0,0 +1,501 @@
|
||||
"""
|
||||
Job data models for karaoke generation.
|
||||
|
||||
This module defines the complete state machine for karaoke generation jobs,
|
||||
mirroring the 8-stage CLI workflow with human-in-the-loop interaction points.
|
||||
"""
|
||||
from typing import Optional, Dict, Any, List
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
from pydantic import BaseModel, Field, validator
|
||||
|
||||
|
||||
class JobStatus(str, Enum):
|
||||
"""
|
||||
Job status enumeration - Complete state machine.
|
||||
|
||||
The workflow has 8 main stages with 3 human interaction points:
|
||||
1. Input & Setup (may include audio source selection)
|
||||
2. Parallel Processing (audio + lyrics)
|
||||
3. Title/End Screen Generation
|
||||
4. Countdown Padding Synchronization
|
||||
5. Human Review (BLOCKING)
|
||||
6. Instrumental Selection (BLOCKING)
|
||||
7. Video Finalization
|
||||
8. Distribution
|
||||
"""
|
||||
# Initial states
|
||||
PENDING = "pending" # Job created, queued for processing
|
||||
|
||||
# Audio search states (for artist+title search mode)
|
||||
SEARCHING_AUDIO = "searching_audio" # Searching for audio sources via flacfetch
|
||||
AWAITING_AUDIO_SELECTION = "awaiting_audio_selection" # ⚠️ WAITING FOR USER - select audio source
|
||||
DOWNLOADING_AUDIO = "downloading_audio" # Downloading selected audio from source
|
||||
|
||||
DOWNLOADING = "downloading" # Downloading from URL or processing upload
|
||||
|
||||
# Stage 2a: Audio separation (parallel track 1)
|
||||
SEPARATING_STAGE1 = "separating_stage1" # Clean instrumental separation (Modal API)
|
||||
SEPARATING_STAGE2 = "separating_stage2" # Backing vocals separation (Modal API)
|
||||
AUDIO_COMPLETE = "audio_complete" # All audio stems ready
|
||||
|
||||
# Stage 2b: Lyrics processing (parallel track 2)
|
||||
TRANSCRIBING = "transcribing" # AudioShake API transcription
|
||||
CORRECTING = "correcting" # Automatic lyrics correction
|
||||
LYRICS_COMPLETE = "lyrics_complete" # Corrections JSON ready
|
||||
|
||||
# Stage 3: Title/End screens
|
||||
GENERATING_SCREENS = "generating_screens" # Creating title and end screen videos
|
||||
|
||||
# Stage 4: Countdown padding (automatic)
|
||||
APPLYING_PADDING = "applying_padding" # Synchronizing countdown padding
|
||||
|
||||
# Stage 5: Human review (BLOCKING)
|
||||
AWAITING_REVIEW = "awaiting_review" # ⚠️ WAITING FOR USER - lyrics review needed
|
||||
IN_REVIEW = "in_review" # User is actively reviewing lyrics
|
||||
REVIEW_COMPLETE = "review_complete" # User submitted corrected lyrics
|
||||
|
||||
# Stage 5.5: Render video with corrected lyrics (post-review)
|
||||
RENDERING_VIDEO = "rendering_video" # Using OutputGenerator to create with_vocals.mkv
|
||||
|
||||
# Stage 6: Instrumental selection (BLOCKING)
|
||||
AWAITING_INSTRUMENTAL_SELECTION = "awaiting_instrumental_selection" # ⚠️ WAITING FOR USER
|
||||
INSTRUMENTAL_SELECTED = "instrumental_selected" # User made selection
|
||||
|
||||
# Stage 7: Video generation and finalization
|
||||
GENERATING_VIDEO = "generating_video" # Creating initial karaoke video
|
||||
ENCODING = "encoding" # Multi-format video encoding (Cloud Build)
|
||||
PACKAGING = "packaging" # CDG/TXT ZIP generation
|
||||
|
||||
# Stage 8: Distribution (optional)
|
||||
UPLOADING = "uploading" # YouTube/Dropbox upload
|
||||
NOTIFYING = "notifying" # Discord/Email notifications
|
||||
|
||||
# Terminal states
|
||||
COMPLETE = "complete" # All processing finished successfully
|
||||
PREP_COMPLETE = "prep_complete" # Prep-only job completed (stops after review)
|
||||
FAILED = "failed" # Unrecoverable error occurred
|
||||
CANCELLED = "cancelled" # User cancelled the job
|
||||
|
||||
# Legacy compatibility (will be removed)
|
||||
QUEUED = "queued" # Deprecated: use PENDING
|
||||
PROCESSING = "processing" # Deprecated: use specific states
|
||||
READY_FOR_FINALIZATION = "ready_for_finalization" # Deprecated
|
||||
FINALIZING = "finalizing" # Deprecated: use ENCODING/PACKAGING
|
||||
ERROR = "error" # Deprecated: use FAILED
|
||||
|
||||
|
||||
# Valid state transitions
|
||||
STATE_TRANSITIONS = {
|
||||
# PENDING can go to DOWNLOADING (file upload), SEARCHING_AUDIO (artist+title search), or AWAITING_INSTRUMENTAL_SELECTION (finalise-only)
|
||||
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
|
||||
# Audio search flow (for artist+title search mode)
|
||||
JobStatus.SEARCHING_AUDIO: [JobStatus.AWAITING_AUDIO_SELECTION, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED],
|
||||
JobStatus.AWAITING_AUDIO_SELECTION: [JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.DOWNLOADING_AUDIO: [JobStatus.DOWNLOADING, JobStatus.FAILED],
|
||||
|
||||
# DOWNLOADING allows parallel processing (audio + lyrics) and then screens when both complete
|
||||
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
|
||||
# Audio separation flow
|
||||
JobStatus.SEPARATING_STAGE1: [JobStatus.SEPARATING_STAGE2, JobStatus.FAILED],
|
||||
JobStatus.SEPARATING_STAGE2: [JobStatus.AUDIO_COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.AUDIO_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
|
||||
# Lyrics flow
|
||||
JobStatus.TRANSCRIBING: [JobStatus.CORRECTING, JobStatus.FAILED],
|
||||
JobStatus.CORRECTING: [JobStatus.LYRICS_COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.LYRICS_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
|
||||
# Post-parallel processing
|
||||
JobStatus.GENERATING_SCREENS: [JobStatus.APPLYING_PADDING, JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED],
|
||||
JobStatus.APPLYING_PADDING: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
|
||||
|
||||
# Human review flow
|
||||
# AWAITING_REVIEW can go directly to REVIEW_COMPLETE (quick review) or to IN_REVIEW (editing)
|
||||
JobStatus.AWAITING_REVIEW: [JobStatus.IN_REVIEW, JobStatus.REVIEW_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.IN_REVIEW: [JobStatus.REVIEW_COMPLETE, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
|
||||
JobStatus.REVIEW_COMPLETE: [JobStatus.RENDERING_VIDEO, JobStatus.PREP_COMPLETE, JobStatus.FAILED], # PREP_COMPLETE for prep-only jobs
|
||||
|
||||
# Video rendering (post-review)
|
||||
JobStatus.RENDERING_VIDEO: [JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
|
||||
|
||||
# Instrumental selection flow
|
||||
JobStatus.AWAITING_INSTRUMENTAL_SELECTION: [JobStatus.INSTRUMENTAL_SELECTED, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.INSTRUMENTAL_SELECTED: [JobStatus.GENERATING_VIDEO, JobStatus.FAILED],
|
||||
|
||||
# Video generation flow
|
||||
JobStatus.GENERATING_VIDEO: [JobStatus.ENCODING, JobStatus.FAILED],
|
||||
JobStatus.ENCODING: [JobStatus.PACKAGING, JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.PACKAGING: [JobStatus.UPLOADING, JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
|
||||
# Distribution flow
|
||||
JobStatus.UPLOADING: [JobStatus.NOTIFYING, JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.NOTIFYING: [JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
|
||||
# Terminal states - COMPLETE, PREP_COMPLETE have no transitions
|
||||
# FAILED and CANCELLED allow retry transitions to resume from checkpoints
|
||||
# PREP_COMPLETE allows finalise-only continuation
|
||||
JobStatus.COMPLETE: [],
|
||||
JobStatus.PREP_COMPLETE: [JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED], # Finalise-only continues from here
|
||||
JobStatus.FAILED: [
|
||||
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
|
||||
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
|
||||
JobStatus.REVIEW_COMPLETE, # Retry from render stage
|
||||
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
|
||||
],
|
||||
JobStatus.CANCELLED: [
|
||||
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
|
||||
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
|
||||
JobStatus.REVIEW_COMPLETE, # Retry from render stage
|
||||
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
|
||||
],
|
||||
|
||||
# Legacy states (for backward compatibility)
|
||||
JobStatus.QUEUED: [JobStatus.PENDING],
|
||||
JobStatus.PROCESSING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING],
|
||||
JobStatus.READY_FOR_FINALIZATION: [JobStatus.GENERATING_VIDEO],
|
||||
JobStatus.FINALIZING: [JobStatus.ENCODING],
|
||||
JobStatus.ERROR: [JobStatus.FAILED],
|
||||
}
|
||||
|
||||
|
||||
class TimelineEvent(BaseModel):
|
||||
"""Timeline event for job progress tracking."""
|
||||
status: str
|
||||
timestamp: str
|
||||
progress: Optional[int] = None
|
||||
message: Optional[str] = None
|
||||
|
||||
|
||||
class LogEntry(BaseModel):
|
||||
"""Worker log entry for debugging and monitoring."""
|
||||
timestamp: str
|
||||
level: str # DEBUG, INFO, WARNING, ERROR
|
||||
worker: str # audio, lyrics, screens, video, render
|
||||
message: str
|
||||
|
||||
|
||||
class Job(BaseModel):
|
||||
"""
|
||||
Complete job data model.
|
||||
|
||||
Tracks the full lifecycle of a karaoke generation job from submission
|
||||
through all 8 stages to completion or failure.
|
||||
"""
|
||||
job_id: str
|
||||
status: JobStatus
|
||||
progress: int = 0 # 0-100 percentage for UI display
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
|
||||
# Input data
|
||||
url: Optional[str] = None # YouTube URL (if provided)
|
||||
artist: Optional[str] = None
|
||||
title: Optional[str] = None
|
||||
filename: Optional[str] = None # Original uploaded filename
|
||||
input_media_gcs_path: Optional[str] = None # GCS path to uploaded file
|
||||
|
||||
# User preferences
|
||||
enable_cdg: bool = False # Generate CDG+MP3 package (requires style config)
|
||||
enable_txt: bool = False # Generate TXT+MP3 package (requires style config)
|
||||
enable_youtube_upload: bool = False # Upload to YouTube
|
||||
youtube_description: Optional[str] = None # YouTube video description
|
||||
webhook_url: Optional[str] = None # Webhook for notifications
|
||||
user_email: Optional[str] = None # Email for notifications
|
||||
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
|
||||
|
||||
# Theme configuration (pre-made themes from GCS)
|
||||
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
|
||||
color_overrides: Dict[str, str] = Field(default_factory=dict)
|
||||
"""
|
||||
User color overrides applied on top of theme. Keys:
|
||||
- artist_color: Hex color for artist name (#RRGGBB)
|
||||
- title_color: Hex color for song title
|
||||
- sung_lyrics_color: Hex color for highlighted lyrics
|
||||
- unsung_lyrics_color: Hex color for unhighlighted lyrics
|
||||
"""
|
||||
|
||||
# Style configuration (uploaded files - used when theme_id is not set)
|
||||
style_params_gcs_path: Optional[str] = None # GCS path to style_params.json
|
||||
style_assets: Dict[str, str] = Field(default_factory=dict)
|
||||
"""
|
||||
GCS paths to style asset files:
|
||||
{
|
||||
"intro_background": "gs://bucket/jobs/{job_id}/style/intro_bg.png",
|
||||
"karaoke_background": "gs://bucket/jobs/{job_id}/style/karaoke_bg.png",
|
||||
"end_background": "gs://bucket/jobs/{job_id}/style/end_bg.png",
|
||||
"font": "gs://bucket/jobs/{job_id}/style/font.ttf",
|
||||
"cdg_instrumental_background": "gs://bucket/jobs/{job_id}/style/cdg_instr.png",
|
||||
"cdg_title_background": "gs://bucket/jobs/{job_id}/style/cdg_title.png",
|
||||
"cdg_outro_background": "gs://bucket/jobs/{job_id}/style/cdg_outro.png"
|
||||
}
|
||||
"""
|
||||
|
||||
# Finalisation configuration
|
||||
brand_prefix: Optional[str] = None # Brand code prefix (e.g., "NOMAD")
|
||||
discord_webhook_url: Optional[str] = None # Discord notification webhook
|
||||
youtube_description_template: Optional[str] = None # YouTube description template text
|
||||
|
||||
# Distribution configuration (native API - for remote CLI)
|
||||
dropbox_path: Optional[str] = None # Dropbox folder path for organized output (e.g., "/Karaoke/Tracks-Organized")
|
||||
gdrive_folder_id: Optional[str] = None # Google Drive folder ID for public share uploads
|
||||
|
||||
# Legacy distribution configuration (rclone - for local CLI backward compat)
|
||||
organised_dir_rclone_root: Optional[str] = None # Deprecated: use dropbox_path instead
|
||||
|
||||
# Lyrics configuration (overrides for search/transcription)
|
||||
lyrics_artist: Optional[str] = None # Override artist name for lyrics search
|
||||
lyrics_title: Optional[str] = None # Override title for lyrics search
|
||||
lyrics_file_gcs_path: Optional[str] = None # GCS path to user-provided lyrics file
|
||||
subtitle_offset_ms: int = 0 # Offset for subtitle timing (positive = delay)
|
||||
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model: Optional[str] = None # Model for clean instrumental separation
|
||||
backing_vocals_models: Optional[List[str]] = None # Models for backing vocals separation
|
||||
other_stems_models: Optional[List[str]] = None # Models for other stems (bass, drums, etc.)
|
||||
|
||||
# Existing instrumental configuration (Batch 3)
|
||||
existing_instrumental_gcs_path: Optional[str] = None # GCS path to user-provided instrumental file
|
||||
|
||||
# Audio search configuration (Batch 5 - artist+title search mode)
|
||||
audio_search_artist: Optional[str] = None # Artist name used for audio search
|
||||
audio_search_title: Optional[str] = None # Title used for audio search
|
||||
auto_download: bool = False # Auto-select best audio source (skip selection)
|
||||
|
||||
# Two-phase workflow configuration (Batch 6)
|
||||
prep_only: bool = False # Stop after review, don't run finalisation
|
||||
finalise_only: bool = False # Skip prep, run only finalisation (requires uploaded prep outputs)
|
||||
keep_brand_code: Optional[str] = None # Preserve existing brand code instead of generating new one
|
||||
|
||||
# Review authentication (Batch 7)
|
||||
review_token: Optional[str] = None # Job-scoped token for lyrics review UI access (generated when entering AWAITING_REVIEW)
|
||||
review_token_expires_at: Optional[datetime] = None # Token expiry time (optional, for extra security)
|
||||
instrumental_token: Optional[str] = None # Job-scoped token for instrumental review UI access (generated when entering AWAITING_INSTRUMENTAL_SELECTION)
|
||||
instrumental_token_expires_at: Optional[datetime] = None # Token expiry time
|
||||
|
||||
# Processing state
|
||||
track_output_dir: Optional[str] = None # Local output directory (temp)
|
||||
audio_hash: Optional[str] = None # Hash for deduplication
|
||||
|
||||
# State-specific data (JSON field for stage-specific metadata)
|
||||
state_data: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
Stage-specific metadata. Examples:
|
||||
- audio_complete: {"stems": {"clean": "gs://...", "backing": "gs://..."}}
|
||||
- lyrics_complete: {"corrections_url": "gs://...", "audio_url": "gs://..."}
|
||||
- review_complete: {"corrected_lyrics": {...}}
|
||||
- instrumental_selected: {"selection": "clean" | "with_backing"}
|
||||
- encoding: {"build_id": "...", "progress": 45}
|
||||
"""
|
||||
|
||||
# Timeline tracking
|
||||
timeline: List[TimelineEvent] = Field(default_factory=list)
|
||||
|
||||
# Worker logs for debugging (limited to last N entries to avoid document size issues)
|
||||
worker_logs: List[LogEntry] = Field(default_factory=list)
|
||||
|
||||
# File URLs (GCS storage)
|
||||
file_urls: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
File storage URLs. Structure:
|
||||
{
|
||||
"input": "gs://bucket/jobs/{job_id}/input.flac",
|
||||
"stems": {
|
||||
"instrumental_clean": "gs://...",
|
||||
"instrumental_with_backing": "gs://...",
|
||||
"vocals": "gs://...",
|
||||
"backing_vocals": "gs://...",
|
||||
"lead_vocals": "gs://...",
|
||||
"bass": "gs://...",
|
||||
"drums": "gs://...",
|
||||
"guitar": "gs://...",
|
||||
"piano": "gs://...",
|
||||
"other": "gs://..."
|
||||
},
|
||||
"lyrics": {
|
||||
"corrections": "gs://bucket/jobs/{job_id}/lyrics/corrections.json",
|
||||
"audio": "gs://bucket/jobs/{job_id}/lyrics/audio.flac",
|
||||
"lrc": "gs://...",
|
||||
"ass": "gs://..."
|
||||
},
|
||||
"screens": {
|
||||
"title": "gs://bucket/jobs/{job_id}/screens/title.mov",
|
||||
"end": "gs://bucket/jobs/{job_id}/screens/end.mov"
|
||||
},
|
||||
"videos": {
|
||||
"with_vocals": "gs://bucket/jobs/{job_id}/videos/with_vocals.mkv"
|
||||
},
|
||||
"finals": {
|
||||
"lossless_4k_mp4": "gs://...",
|
||||
"lossless_4k_mkv": "gs://...",
|
||||
"lossy_4k_mp4": "gs://...",
|
||||
"lossy_720p_mp4": "gs://..."
|
||||
},
|
||||
"packages": {
|
||||
"cdg_zip": "gs://...",
|
||||
"txt_zip": "gs://..."
|
||||
},
|
||||
"youtube": {
|
||||
"url": "https://youtube.com/watch?v=...",
|
||||
"video_id": "..."
|
||||
}
|
||||
}
|
||||
"""
|
||||
|
||||
# Results (for backward compatibility, will be deprecated)
|
||||
output_files: Dict[str, str] = Field(default_factory=dict)
|
||||
download_urls: Dict[str, str] = Field(default_factory=dict)
|
||||
|
||||
# Error handling
|
||||
error_message: Optional[str] = None
|
||||
error_details: Optional[Dict[str, Any]] = None # Structured error information
|
||||
retry_count: int = 0 # Number of retry attempts
|
||||
|
||||
# Worker tracking
|
||||
worker_ids: Dict[str, str] = Field(default_factory=dict)
|
||||
"""
|
||||
IDs of background workers/jobs:
|
||||
{
|
||||
"audio_worker": "cloud-run-request-id",
|
||||
"lyrics_worker": "cloud-run-request-id",
|
||||
"video_encoder": "cloud-build-id",
|
||||
"distribution": "cloud-run-request-id"
|
||||
}
|
||||
"""
|
||||
|
||||
# Request metadata (captured at job creation for tracking and filtering)
|
||||
request_metadata: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
Metadata captured from the original API request.
|
||||
Used for tracking, filtering, and operational management.
|
||||
|
||||
Standard fields:
|
||||
{
|
||||
"client_ip": "192.168.1.1", # IP address of the client
|
||||
"user_agent": "karaoke-gen-remote/0.71.0", # User-Agent header
|
||||
"environment": "test", # From X-Environment header (test/production/development)
|
||||
"client_id": "cli-user-123", # From X-Client-ID header (customer/user identifier)
|
||||
"server_version": "0.71.0", # Server version at job creation
|
||||
"created_from": "upload", # "upload" (file) or "url" (YouTube URL)
|
||||
"custom_headers": { # All X-* headers for extensibility
|
||||
"X-Environment": "test",
|
||||
"X-Client-ID": "cli-user-123",
|
||||
"X-Request-ID": "abc-123"
|
||||
}
|
||||
}
|
||||
|
||||
Use cases:
|
||||
- Filter test vs production jobs
|
||||
- Track jobs by customer/client
|
||||
- Debug issues with specific clients
|
||||
- Bulk cleanup of test jobs
|
||||
"""
|
||||
|
||||
# Note: Status transition validation is handled by JobManager.validate_state_transition()
|
||||
# which is called before status updates. The Job model does not validate transitions
|
||||
# because Firestore updates happen directly without reconstructing the model.
|
||||
|
||||
class Config:
|
||||
use_enum_values = True
|
||||
|
||||
|
||||
class JobCreate(BaseModel):
|
||||
"""
|
||||
Job creation request.
|
||||
|
||||
Either `url` OR file upload is required (file upload handled separately).
|
||||
Artist and title are optional - will be auto-detected from YouTube if not provided.
|
||||
"""
|
||||
url: Optional[str] = None
|
||||
artist: Optional[str] = None
|
||||
title: Optional[str] = None
|
||||
filename: Optional[str] = None # Original uploaded filename
|
||||
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: bool = False
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
|
||||
|
||||
# Theme configuration (pre-made themes from GCS)
|
||||
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
|
||||
color_overrides: Dict[str, str] = Field(default_factory=dict)
|
||||
"""
|
||||
User color overrides applied on top of theme. Keys:
|
||||
- artist_color: Hex color for artist name (#RRGGBB)
|
||||
- title_color: Hex color for song title
|
||||
- sung_lyrics_color: Hex color for highlighted lyrics
|
||||
- unsung_lyrics_color: Hex color for unhighlighted lyrics
|
||||
"""
|
||||
|
||||
# Style configuration (will be populated after file upload, or from theme)
|
||||
style_params_gcs_path: Optional[str] = None
|
||||
style_assets: Dict[str, str] = Field(default_factory=dict)
|
||||
|
||||
# Finalisation configuration
|
||||
brand_prefix: Optional[str] = None
|
||||
discord_webhook_url: Optional[str] = None
|
||||
youtube_description_template: Optional[str] = None
|
||||
|
||||
# Distribution configuration (native API - for remote CLI)
|
||||
dropbox_path: Optional[str] = None # Dropbox folder path for organized output
|
||||
gdrive_folder_id: Optional[str] = None # Google Drive folder ID for public share uploads
|
||||
|
||||
# Legacy (rclone - deprecated, use dropbox_path instead)
|
||||
organised_dir_rclone_root: Optional[str] = None
|
||||
|
||||
# Lyrics configuration (overrides for search/transcription)
|
||||
lyrics_artist: Optional[str] = None # Override artist name for lyrics search
|
||||
lyrics_title: Optional[str] = None # Override title for lyrics search
|
||||
lyrics_file_gcs_path: Optional[str] = None # GCS path to user-provided lyrics file
|
||||
subtitle_offset_ms: int = 0 # Offset for subtitle timing (positive = delay)
|
||||
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model: Optional[str] = None # Model for clean instrumental separation
|
||||
backing_vocals_models: Optional[List[str]] = None # Models for backing vocals separation
|
||||
other_stems_models: Optional[List[str]] = None # Models for other stems (bass, drums, etc.)
|
||||
|
||||
# Existing instrumental configuration (Batch 3)
|
||||
existing_instrumental_gcs_path: Optional[str] = None # GCS path to user-provided instrumental file
|
||||
|
||||
# Audio search configuration (Batch 5 - artist+title search mode)
|
||||
audio_search_artist: Optional[str] = None # Artist name used for audio search
|
||||
audio_search_title: Optional[str] = None # Title used for audio search
|
||||
auto_download: bool = False # Auto-select best audio source (skip selection)
|
||||
|
||||
# Two-phase workflow configuration (Batch 6)
|
||||
prep_only: bool = False # Stop after review, don't run finalisation
|
||||
finalise_only: bool = False # Skip prep, run only finalisation
|
||||
keep_brand_code: Optional[str] = None # Preserve existing brand code instead of generating new one
|
||||
|
||||
# Request metadata (set by API endpoint from request headers)
|
||||
request_metadata: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
Populated by the API endpoint with request context:
|
||||
- client_ip: Client IP address
|
||||
- user_agent: User-Agent header
|
||||
- environment: From X-Environment header (test/production/development)
|
||||
- client_id: From X-Client-ID header
|
||||
- server_version: Current server version
|
||||
- custom_headers: All X-* headers
|
||||
"""
|
||||
|
||||
@validator('url', 'artist', 'title')
|
||||
def validate_inputs(cls, v):
|
||||
"""Validate string inputs are not empty."""
|
||||
if v is not None and isinstance(v, str) and not v.strip():
|
||||
raise ValueError("Field cannot be empty string")
|
||||
return v.strip() if isinstance(v, str) else v
|
||||
|
||||
|
||||
class JobResponse(BaseModel):
|
||||
"""Job response model."""
|
||||
status: str
|
||||
job_id: str
|
||||
message: str
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
"""
|
||||
API request models for karaoke generation endpoints.
|
||||
"""
|
||||
from typing import Optional, Dict, Any, List
|
||||
from pydantic import BaseModel, HttpUrl, validator
|
||||
|
||||
|
||||
class URLSubmissionRequest(BaseModel):
|
||||
"""Request to submit a job from a URL (YouTube, etc.)."""
|
||||
url: HttpUrl
|
||||
artist: Optional[str] = None # Auto-detected if not provided
|
||||
title: Optional[str] = None # Auto-detected if not provided
|
||||
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: bool = False
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
|
||||
|
||||
class UploadSubmissionRequest(BaseModel):
|
||||
"""Request to submit a job from an uploaded file."""
|
||||
artist: str
|
||||
title: str
|
||||
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: bool = False
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
|
||||
|
||||
class CorrectionsSubmission(BaseModel):
|
||||
"""
|
||||
Request to submit corrected lyrics after human review.
|
||||
|
||||
This is the critical human-in-the-loop interaction point.
|
||||
The corrections data comes from the lyrics-transcriber review interface.
|
||||
"""
|
||||
corrections: Dict[str, Any] # Full corrections JSON from frontend
|
||||
user_notes: Optional[str] = None # Optional notes from reviewer
|
||||
|
||||
@validator('corrections')
|
||||
def validate_corrections_format(cls, v):
|
||||
"""Validate corrections has required fields."""
|
||||
required_fields = ['lines', 'metadata']
|
||||
for field in required_fields:
|
||||
if field not in v:
|
||||
raise ValueError(f"Corrections must include '{field}' field")
|
||||
return v
|
||||
|
||||
|
||||
class InstrumentalSelection(BaseModel):
|
||||
"""
|
||||
Request to select instrumental audio option.
|
||||
|
||||
This is the second critical human-in-the-loop interaction point.
|
||||
User chooses between clean instrumental, instrumental with backing vocals,
|
||||
or a custom instrumental (created via create-custom-instrumental endpoint).
|
||||
"""
|
||||
selection: str # "clean", "with_backing", or "custom"
|
||||
|
||||
@validator('selection')
|
||||
def validate_selection(cls, v):
|
||||
"""Validate selection is a valid option."""
|
||||
valid_options = ['clean', 'with_backing', 'custom']
|
||||
if v not in valid_options:
|
||||
raise ValueError(f"Selection must be one of: {valid_options}")
|
||||
return v
|
||||
|
||||
|
||||
class MuteRegionRequest(BaseModel):
|
||||
"""A region to mute in the backing vocals."""
|
||||
start_seconds: float
|
||||
end_seconds: float
|
||||
|
||||
@validator('start_seconds')
|
||||
def validate_start(cls, v):
|
||||
if v < 0:
|
||||
raise ValueError("start_seconds must be non-negative")
|
||||
return v
|
||||
|
||||
@validator('end_seconds')
|
||||
def validate_end(cls, v, values):
|
||||
if 'start_seconds' in values and v <= values['start_seconds']:
|
||||
raise ValueError("end_seconds must be greater than start_seconds")
|
||||
return v
|
||||
|
||||
|
||||
class CreateCustomInstrumentalRequest(BaseModel):
|
||||
"""
|
||||
Request to create a custom instrumental with muted backing vocal regions.
|
||||
|
||||
The mute_regions specify time ranges in the backing vocals track that
|
||||
should be silenced before mixing with the clean instrumental.
|
||||
"""
|
||||
mute_regions: List[MuteRegionRequest]
|
||||
|
||||
@validator('mute_regions')
|
||||
def validate_regions(cls, v):
|
||||
if not v:
|
||||
raise ValueError("At least one mute region is required")
|
||||
return v
|
||||
|
||||
|
||||
class StartReviewRequest(BaseModel):
|
||||
"""Request to mark job as in-review (user opened interface)."""
|
||||
pass # No body needed, just triggers state transition
|
||||
|
||||
|
||||
class CancelJobRequest(BaseModel):
|
||||
"""Request to cancel a job."""
|
||||
reason: Optional[str] = None
|
||||
|
||||
|
||||
class RetryJobRequest(BaseModel):
|
||||
"""Request to retry a failed job."""
|
||||
from_stage: Optional[str] = None # Optional: restart from specific stage
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
"""
|
||||
Theme data models for karaoke-gen.
|
||||
|
||||
Themes are pre-created style configurations stored in GCS that users can select
|
||||
when creating karaoke jobs. Each theme includes:
|
||||
- Style parameters (fonts, colors, backgrounds, layouts)
|
||||
- Asset files (images, fonts)
|
||||
- Preview images
|
||||
- Optional YouTube description template
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class ThemeSummary(BaseModel):
|
||||
"""Summary of a theme for listing purposes."""
|
||||
|
||||
id: str = Field(..., description="Unique theme identifier (e.g., 'nomad', 'default')")
|
||||
name: str = Field(..., description="Human-readable theme name")
|
||||
description: str = Field(..., description="Brief description of the theme style")
|
||||
preview_url: Optional[str] = Field(None, description="Signed URL for preview image")
|
||||
thumbnail_url: Optional[str] = Field(None, description="Signed URL for smaller thumbnail")
|
||||
is_default: bool = Field(False, description="Whether this is the default theme")
|
||||
|
||||
|
||||
class ThemeMetadata(BaseModel):
|
||||
"""Metadata for a single theme in the registry."""
|
||||
|
||||
id: str
|
||||
name: str
|
||||
description: str
|
||||
is_default: bool = False
|
||||
created_at: Optional[datetime] = None
|
||||
|
||||
|
||||
class ThemeRegistry(BaseModel):
|
||||
"""Registry of all available themes, stored as _metadata.json in GCS."""
|
||||
|
||||
version: int = 1
|
||||
themes: List[ThemeMetadata] = Field(default_factory=list)
|
||||
|
||||
|
||||
class ThemeDetail(BaseModel):
|
||||
"""Full theme details including style parameters."""
|
||||
|
||||
id: str
|
||||
name: str
|
||||
description: str
|
||||
preview_url: Optional[str] = None
|
||||
is_default: bool = False
|
||||
style_params: Dict[str, Any] = Field(default_factory=dict)
|
||||
has_youtube_description: bool = False
|
||||
|
||||
|
||||
class ColorOverrides(BaseModel):
|
||||
"""
|
||||
User-customizable color overrides that can be applied on top of a theme.
|
||||
|
||||
All colors are hex format (#RRGGBB). When applied:
|
||||
- artist_color: Applied to intro.artist_color, end.artist_color, cdg.artist_color
|
||||
- title_color: Applied to intro.title_color, end.title_color, cdg.title_color
|
||||
- sung_lyrics_color: Applied to karaoke.primary_color (converted to RGBA), cdg.active_fill
|
||||
- unsung_lyrics_color: Applied to karaoke.secondary_color (converted to RGBA), cdg.inactive_fill
|
||||
"""
|
||||
|
||||
artist_color: Optional[str] = Field(
|
||||
None,
|
||||
pattern=r"^#[0-9A-Fa-f]{6}$",
|
||||
description="Hex color for artist name on intro/end screens",
|
||||
)
|
||||
title_color: Optional[str] = Field(
|
||||
None,
|
||||
pattern=r"^#[0-9A-Fa-f]{6}$",
|
||||
description="Hex color for song title on intro/end screens",
|
||||
)
|
||||
sung_lyrics_color: Optional[str] = Field(
|
||||
None,
|
||||
pattern=r"^#[0-9A-Fa-f]{6}$",
|
||||
description="Hex color for lyrics being sung (highlighted)",
|
||||
)
|
||||
unsung_lyrics_color: Optional[str] = Field(
|
||||
None,
|
||||
pattern=r"^#[0-9A-Fa-f]{6}$",
|
||||
description="Hex color for lyrics not yet sung",
|
||||
)
|
||||
|
||||
def has_overrides(self) -> bool:
|
||||
"""Check if any color overrides are set."""
|
||||
return any(
|
||||
[
|
||||
self.artist_color,
|
||||
self.title_color,
|
||||
self.sung_lyrics_color,
|
||||
self.unsung_lyrics_color,
|
||||
]
|
||||
)
|
||||
|
||||
def to_dict(self) -> Dict[str, str]:
|
||||
"""Convert to dict, excluding None values."""
|
||||
return {k: v for k, v in self.model_dump().items() if v is not None}
|
||||
|
||||
|
||||
class ThemesListResponse(BaseModel):
|
||||
"""Response from GET /api/themes endpoint."""
|
||||
|
||||
themes: List[ThemeSummary]
|
||||
|
||||
|
||||
class ThemeDetailResponse(BaseModel):
|
||||
"""Response from GET /api/themes/{theme_id} endpoint."""
|
||||
|
||||
theme: ThemeDetail
|
||||
|
||||
|
||||
# Utility functions for color conversion
|
||||
|
||||
|
||||
def hex_to_rgba(hex_color: str, alpha: int = 255) -> str:
|
||||
"""
|
||||
Convert hex color (#RRGGBB) to RGBA format for ASS subtitles.
|
||||
|
||||
ASS subtitle format uses "R, G, B, A" string format.
|
||||
|
||||
Args:
|
||||
hex_color: Hex color string like "#7070F7"
|
||||
alpha: Alpha value 0-255 (default 255 = opaque)
|
||||
|
||||
Returns:
|
||||
RGBA string like "112, 112, 247, 255"
|
||||
"""
|
||||
hex_color = hex_color.lstrip("#")
|
||||
r = int(hex_color[0:2], 16)
|
||||
g = int(hex_color[2:4], 16)
|
||||
b = int(hex_color[4:6], 16)
|
||||
return f"{r}, {g}, {b}, {alpha}"
|
||||
|
||||
|
||||
def rgba_to_hex(rgba_str: str) -> str:
|
||||
"""
|
||||
Convert RGBA format to hex color.
|
||||
|
||||
Args:
|
||||
rgba_str: RGBA string like "112, 112, 247, 255"
|
||||
|
||||
Returns:
|
||||
Hex color string like "#7070F7"
|
||||
"""
|
||||
parts = [int(x.strip()) for x in rgba_str.split(",")]
|
||||
r, g, b = parts[0], parts[1], parts[2]
|
||||
return f"#{r:02x}{g:02x}{b:02x}"
|
||||
@@ -0,0 +1,254 @@
|
||||
"""
|
||||
User model for authentication and credits.
|
||||
|
||||
Supports:
|
||||
- Magic link authentication (email-based)
|
||||
- Credit system for karaoke generation
|
||||
- Role-based access control (user/admin)
|
||||
- Stripe integration for payments
|
||||
- Beta tester program with feedback collection
|
||||
"""
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
from typing import Optional, List
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class UserRole(str, Enum):
|
||||
"""User roles for access control."""
|
||||
USER = "user"
|
||||
ADMIN = "admin"
|
||||
|
||||
|
||||
class BetaTesterStatus(str, Enum):
|
||||
"""Status of beta tester participation."""
|
||||
ACTIVE = "active" # Enrolled, free credits available
|
||||
PENDING_FEEDBACK = "pending_feedback" # Job completed, awaiting feedback
|
||||
COMPLETED = "completed" # Feedback submitted
|
||||
EXPIRED = "expired" # 24hr deadline passed without feedback
|
||||
|
||||
|
||||
class CreditTransaction(BaseModel):
|
||||
"""Record of a credit transaction."""
|
||||
id: str
|
||||
amount: int # Positive = add, negative = deduct
|
||||
reason: str # e.g., "purchase", "refund", "admin_grant", "job_creation"
|
||||
job_id: Optional[str] = None
|
||||
stripe_session_id: Optional[str] = None
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
created_by: Optional[str] = None # Admin email if granted by admin
|
||||
|
||||
|
||||
class User(BaseModel):
|
||||
"""
|
||||
User model stored in Firestore.
|
||||
|
||||
Users are identified by email address. Authentication is via magic links
|
||||
(no passwords). Credits are consumed when creating karaoke jobs.
|
||||
"""
|
||||
email: str # Primary identifier
|
||||
role: UserRole = UserRole.USER
|
||||
credits: int = 0
|
||||
|
||||
# Stripe integration
|
||||
stripe_customer_id: Optional[str] = None
|
||||
|
||||
# Account state
|
||||
is_active: bool = True
|
||||
email_verified: bool = False
|
||||
|
||||
# Timestamps
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
updated_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
last_login_at: Optional[datetime] = None
|
||||
|
||||
# Credit history (last 100 transactions)
|
||||
credit_transactions: List[CreditTransaction] = Field(default_factory=list)
|
||||
|
||||
# Job tracking
|
||||
total_jobs_created: int = 0
|
||||
total_jobs_completed: int = 0
|
||||
|
||||
# Optional profile fields for future use
|
||||
display_name: Optional[str] = None
|
||||
|
||||
# Beta tester program
|
||||
is_beta_tester: bool = False
|
||||
beta_tester_status: Optional[BetaTesterStatus] = None
|
||||
beta_enrolled_at: Optional[datetime] = None
|
||||
beta_promise_text: Optional[str] = None # User's promise statement
|
||||
beta_feedback_due_at: Optional[datetime] = None # 24hr after job completion
|
||||
beta_feedback_email_sent: bool = False
|
||||
|
||||
|
||||
class MagicLinkToken(BaseModel):
|
||||
"""
|
||||
Magic link token for passwordless authentication.
|
||||
|
||||
Tokens are short-lived (15 minutes) and single-use.
|
||||
Stored in Firestore with automatic TTL.
|
||||
"""
|
||||
token: str # Secure random token
|
||||
email: str
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
expires_at: datetime
|
||||
used: bool = False
|
||||
used_at: Optional[datetime] = None
|
||||
ip_address: Optional[str] = None
|
||||
user_agent: Optional[str] = None
|
||||
|
||||
|
||||
class Session(BaseModel):
|
||||
"""
|
||||
User session for authenticated requests.
|
||||
|
||||
Sessions are created after successful magic link verification.
|
||||
Sessions expire after 7 days of inactivity or 30 days absolute.
|
||||
"""
|
||||
token: str # Secure random session token
|
||||
user_email: str
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
expires_at: datetime
|
||||
last_activity_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
ip_address: Optional[str] = None
|
||||
user_agent: Optional[str] = None
|
||||
is_active: bool = True
|
||||
|
||||
|
||||
# Pydantic models for API requests/responses
|
||||
|
||||
class SendMagicLinkRequest(BaseModel):
|
||||
"""Request to send a magic link email."""
|
||||
email: str
|
||||
|
||||
|
||||
class SendMagicLinkResponse(BaseModel):
|
||||
"""Response after sending magic link."""
|
||||
status: str
|
||||
message: str
|
||||
|
||||
|
||||
class VerifyMagicLinkRequest(BaseModel):
|
||||
"""Request to verify a magic link token."""
|
||||
token: str
|
||||
|
||||
|
||||
class VerifyMagicLinkResponse(BaseModel):
|
||||
"""Response after verifying magic link."""
|
||||
status: str
|
||||
session_token: str
|
||||
user: "UserPublic"
|
||||
message: str
|
||||
|
||||
|
||||
class UserPublic(BaseModel):
|
||||
"""Public user information (safe to expose to frontend)."""
|
||||
email: str
|
||||
role: UserRole
|
||||
credits: int
|
||||
display_name: Optional[str] = None
|
||||
total_jobs_created: int = 0
|
||||
total_jobs_completed: int = 0
|
||||
|
||||
|
||||
class AddCreditsRequest(BaseModel):
|
||||
"""Admin request to add credits to a user."""
|
||||
email: str
|
||||
amount: int
|
||||
reason: str = "admin_grant"
|
||||
|
||||
|
||||
class AddCreditsResponse(BaseModel):
|
||||
"""Response after adding credits."""
|
||||
status: str
|
||||
email: str
|
||||
credits_added: int
|
||||
new_balance: int
|
||||
message: str
|
||||
|
||||
|
||||
class UserListResponse(BaseModel):
|
||||
"""Response for listing users (admin only)."""
|
||||
users: List[UserPublic]
|
||||
total: int
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Beta Tester Program Models
|
||||
# ============================================================================
|
||||
|
||||
class BetaTesterFeedback(BaseModel):
|
||||
"""
|
||||
Feedback from a beta tester after using the service.
|
||||
|
||||
Stored in Firestore 'beta_feedback' collection.
|
||||
"""
|
||||
id: str # Unique feedback ID
|
||||
user_email: str
|
||||
job_id: Optional[str] = None # The job they're providing feedback on
|
||||
|
||||
# Ratings (1-5 scale)
|
||||
overall_rating: int = Field(ge=1, le=5)
|
||||
ease_of_use_rating: int = Field(ge=1, le=5)
|
||||
lyrics_accuracy_rating: int = Field(ge=1, le=5)
|
||||
correction_experience_rating: int = Field(ge=1, le=5)
|
||||
|
||||
# Open-ended feedback
|
||||
what_went_well: Optional[str] = None
|
||||
what_could_improve: Optional[str] = None
|
||||
additional_comments: Optional[str] = None
|
||||
|
||||
# Would they recommend / use again?
|
||||
would_recommend: bool = True
|
||||
would_use_again: bool = True
|
||||
|
||||
# Metadata
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
submitted_via: str = "web" # 'web' or 'email'
|
||||
|
||||
|
||||
class BetaTesterEnrollRequest(BaseModel):
|
||||
"""Request to enroll as a beta tester."""
|
||||
email: str
|
||||
promise_text: str = Field(
|
||||
min_length=10,
|
||||
description="User's promise to provide feedback (min 10 chars)"
|
||||
)
|
||||
accept_corrections_work: bool = Field(
|
||||
description="User accepts they may need to review/correct lyrics"
|
||||
)
|
||||
|
||||
|
||||
class BetaTesterEnrollResponse(BaseModel):
|
||||
"""Response after enrolling as beta tester."""
|
||||
status: str
|
||||
message: str
|
||||
credits_granted: int
|
||||
session_token: Optional[str] = None # If new user, provide session
|
||||
|
||||
|
||||
class BetaFeedbackRequest(BaseModel):
|
||||
"""Request to submit beta tester feedback."""
|
||||
job_id: Optional[str] = None
|
||||
|
||||
# Ratings (1-5 scale)
|
||||
overall_rating: int = Field(ge=1, le=5)
|
||||
ease_of_use_rating: int = Field(ge=1, le=5)
|
||||
lyrics_accuracy_rating: int = Field(ge=1, le=5)
|
||||
correction_experience_rating: int = Field(ge=1, le=5)
|
||||
|
||||
# Open-ended feedback
|
||||
what_went_well: Optional[str] = None
|
||||
what_could_improve: Optional[str] = None
|
||||
additional_comments: Optional[str] = None
|
||||
|
||||
# Would they recommend / use again?
|
||||
would_recommend: bool = True
|
||||
would_use_again: bool = True
|
||||
|
||||
|
||||
class BetaFeedbackResponse(BaseModel):
|
||||
"""Response after submitting feedback."""
|
||||
status: str
|
||||
message: str
|
||||
bonus_credits: int = 0 # Bonus credits for great feedback
|
||||
@@ -0,0 +1,164 @@
|
||||
"""
|
||||
Worker log entry model for subcollection storage.
|
||||
|
||||
This module defines the LogEntry model for storing worker logs in a Firestore
|
||||
subcollection instead of an embedded array. This avoids the 1MB document size
|
||||
limit that caused job 501258e1 to fail when logs reached 1.26 MB.
|
||||
"""
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from typing import Optional, Dict, Any
|
||||
import uuid
|
||||
|
||||
|
||||
# Default TTL for log entries (30 days)
|
||||
DEFAULT_LOG_TTL_DAYS = 30
|
||||
|
||||
|
||||
@dataclass
|
||||
class WorkerLogEntry:
|
||||
"""
|
||||
Worker log entry for Firestore subcollection storage.
|
||||
|
||||
Stored at: jobs/{job_id}/logs/{log_id}
|
||||
|
||||
This model is separate from the legacy LogEntry (in job.py) which is
|
||||
stored as an embedded array in the job document. This new model supports
|
||||
the subcollection approach with TTL and richer metadata.
|
||||
"""
|
||||
# Core fields (required)
|
||||
timestamp: datetime
|
||||
level: str # "DEBUG", "INFO", "WARNING", "ERROR"
|
||||
worker: str # "audio", "lyrics", "screens", "video", "render", "distribution"
|
||||
message: str
|
||||
|
||||
# Identifiers
|
||||
id: str = field(default_factory=lambda: str(uuid.uuid4()))
|
||||
job_id: str = "" # Set when writing to Firestore
|
||||
|
||||
# TTL for automatic cleanup
|
||||
ttl_expiry: datetime = field(
|
||||
default_factory=lambda: datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
|
||||
)
|
||||
|
||||
# Optional metadata for debugging
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
|
||||
@classmethod
|
||||
def create(
|
||||
cls,
|
||||
job_id: str,
|
||||
worker: str,
|
||||
level: str,
|
||||
message: str,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
ttl_days: int = DEFAULT_LOG_TTL_DAYS
|
||||
) -> "WorkerLogEntry":
|
||||
"""
|
||||
Factory method to create a new log entry.
|
||||
|
||||
Args:
|
||||
job_id: Job ID this log belongs to
|
||||
worker: Worker name (audio, lyrics, screens, video, render, distribution)
|
||||
level: Log level (DEBUG, INFO, WARNING, ERROR)
|
||||
message: Log message (truncated to 1000 chars)
|
||||
metadata: Optional additional metadata
|
||||
ttl_days: Days until log expires (default 30)
|
||||
|
||||
Returns:
|
||||
New WorkerLogEntry instance
|
||||
"""
|
||||
now = datetime.now(timezone.utc)
|
||||
return cls(
|
||||
id=str(uuid.uuid4()),
|
||||
job_id=job_id,
|
||||
timestamp=now,
|
||||
level=level.upper(),
|
||||
worker=worker,
|
||||
message=message[:1000], # Truncate long messages
|
||||
metadata=metadata,
|
||||
ttl_expiry=now + timedelta(days=ttl_days)
|
||||
)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert to dictionary for Firestore storage.
|
||||
|
||||
Returns:
|
||||
Dictionary representation for Firestore
|
||||
"""
|
||||
result = {
|
||||
"id": self.id,
|
||||
"job_id": self.job_id,
|
||||
"timestamp": self.timestamp,
|
||||
"level": self.level,
|
||||
"worker": self.worker,
|
||||
"message": self.message,
|
||||
"ttl_expiry": self.ttl_expiry
|
||||
}
|
||||
if self.metadata:
|
||||
result["metadata"] = self.metadata
|
||||
return result
|
||||
|
||||
def to_legacy_dict(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert to legacy format for backward compatibility with API responses.
|
||||
|
||||
The existing API returns logs in this format:
|
||||
{"timestamp": "...", "level": "INFO", "worker": "audio", "message": "..."}
|
||||
|
||||
Returns:
|
||||
Dictionary in legacy format
|
||||
"""
|
||||
# Format timestamp: use "Z" suffix for UTC, isoformat() for others
|
||||
if self.timestamp.tzinfo is None:
|
||||
# Naive datetime - use as-is
|
||||
timestamp_str = self.timestamp.isoformat()
|
||||
elif self.timestamp.utcoffset() == timedelta(0):
|
||||
# UTC datetime - replace +00:00 with Z for cleaner format
|
||||
timestamp_str = self.timestamp.replace(tzinfo=None).isoformat() + "Z"
|
||||
else:
|
||||
# Non-UTC timezone-aware - use full isoformat with offset
|
||||
timestamp_str = self.timestamp.isoformat()
|
||||
|
||||
return {
|
||||
"timestamp": timestamp_str,
|
||||
"level": self.level,
|
||||
"worker": self.worker,
|
||||
"message": self.message
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, Any]) -> "WorkerLogEntry":
|
||||
"""
|
||||
Create from Firestore document data.
|
||||
|
||||
Args:
|
||||
data: Dictionary from Firestore document
|
||||
|
||||
Returns:
|
||||
WorkerLogEntry instance
|
||||
"""
|
||||
# Handle timestamp conversion (Firestore returns datetime objects)
|
||||
timestamp = data.get("timestamp")
|
||||
if isinstance(timestamp, str):
|
||||
# Parse ISO format string
|
||||
timestamp = datetime.fromisoformat(timestamp.rstrip("Z")).replace(tzinfo=timezone.utc)
|
||||
|
||||
ttl_expiry = data.get("ttl_expiry")
|
||||
if isinstance(ttl_expiry, str):
|
||||
ttl_expiry = datetime.fromisoformat(ttl_expiry.rstrip("Z")).replace(tzinfo=timezone.utc)
|
||||
elif ttl_expiry is None:
|
||||
# Default TTL if not set
|
||||
ttl_expiry = datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
|
||||
|
||||
return cls(
|
||||
id=data.get("id", str(uuid.uuid4())),
|
||||
job_id=data.get("job_id", ""),
|
||||
timestamp=timestamp or datetime.now(timezone.utc),
|
||||
level=data.get("level", "INFO"),
|
||||
worker=data.get("worker", "unknown"),
|
||||
message=data.get("message", ""),
|
||||
metadata=data.get("metadata"),
|
||||
ttl_expiry=ttl_expiry
|
||||
)
|
||||
@@ -0,0 +1,29 @@
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
python_files = ["test_*.py"]
|
||||
python_classes = ["Test*"]
|
||||
python_functions = ["test_*"]
|
||||
addopts = [
|
||||
"-v",
|
||||
"--tb=short",
|
||||
"--strict-markers",
|
||||
"-ra",
|
||||
]
|
||||
|
||||
markers = [
|
||||
"slow: marks tests as slow (deselect with '-m \"not slow\"')",
|
||||
"integration: marks tests as integration tests requiring live service",
|
||||
]
|
||||
|
||||
# Timeout for slow tests (in seconds)
|
||||
timeout = 300
|
||||
|
||||
[tool.coverage.run]
|
||||
source = ["backend"]
|
||||
omit = ["*/tests/*", "*/test_*"]
|
||||
|
||||
[tool.coverage.report]
|
||||
precision = 2
|
||||
show_missing = true
|
||||
skip_covered = false
|
||||
|
||||
Executable
+93
@@ -0,0 +1,93 @@
|
||||
#!/bin/bash
|
||||
# Quick syntax and structure check (no dependencies required)
|
||||
# Run this before every deployment!
|
||||
|
||||
set -e
|
||||
|
||||
echo "🔍 Quick Backend Check (no dependencies required)"
|
||||
echo "=================================================="
|
||||
echo ""
|
||||
|
||||
# Change to backend directory
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
# Colors
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
NC='\033[0m'
|
||||
|
||||
ERRORS=0
|
||||
|
||||
# Check 1: Python syntax
|
||||
echo "1. Checking Python syntax..."
|
||||
for file in $(find . -name "*.py" -not -path "./__pycache__/*" -not -path "./venv/*"); do
|
||||
if ! python3 -m py_compile "$file" 2>/dev/null; then
|
||||
echo -e " ${RED}❌ Syntax error in $file${NC}"
|
||||
python3 -m py_compile "$file"
|
||||
ERRORS=$((ERRORS+1))
|
||||
fi
|
||||
done
|
||||
|
||||
if [ $ERRORS -eq 0 ]; then
|
||||
echo -e " ${GREEN}✅ All Python files have valid syntax${NC}"
|
||||
fi
|
||||
|
||||
# Check 2: No missing imports (basic check)
|
||||
echo ""
|
||||
echo "2. Checking for obvious import issues..."
|
||||
|
||||
# Check that we don't import deleted modules
|
||||
if grep -r "from backend.services.processing_service import" --include="*.py" . 2>/dev/null; then
|
||||
echo -e " ${RED}❌ Found import of deleted processing_service${NC}"
|
||||
ERRORS=$((ERRORS+1))
|
||||
else
|
||||
echo -e " ${GREEN}✅ No imports of deleted modules${NC}"
|
||||
fi
|
||||
|
||||
# Check 3: Required files exist
|
||||
echo ""
|
||||
echo "3. Checking required files exist..."
|
||||
REQUIRED_FILES=(
|
||||
"main.py"
|
||||
"config.py"
|
||||
"requirements.txt"
|
||||
"Dockerfile"
|
||||
"api/routes/health.py"
|
||||
"api/routes/jobs.py"
|
||||
"api/routes/internal.py"
|
||||
"api/routes/file_upload.py"
|
||||
"services/job_manager.py"
|
||||
"services/firestore_service.py"
|
||||
"services/storage_service.py"
|
||||
"services/worker_service.py"
|
||||
"services/auth_service.py"
|
||||
"workers/audio_worker.py"
|
||||
"workers/lyrics_worker.py"
|
||||
"workers/screens_worker.py"
|
||||
"workers/video_worker.py"
|
||||
"models/job.py"
|
||||
"models/requests.py"
|
||||
)
|
||||
|
||||
for file in "${REQUIRED_FILES[@]}"; do
|
||||
if [ ! -f "$file" ]; then
|
||||
echo -e " ${RED}❌ Missing required file: $file${NC}"
|
||||
ERRORS=$((ERRORS+1))
|
||||
fi
|
||||
done
|
||||
|
||||
if [ $ERRORS -eq 0 ]; then
|
||||
echo -e " ${GREEN}✅ All required files exist${NC}"
|
||||
fi
|
||||
|
||||
# Summary
|
||||
echo ""
|
||||
echo "=================================================="
|
||||
if [ $ERRORS -eq 0 ]; then
|
||||
echo -e "${GREEN}✅ All checks passed! Safe to deploy.${NC}"
|
||||
exit 0
|
||||
else
|
||||
echo -e "${RED}❌ $ERRORS error(s) found. Fix before deploying!${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
# DEPRECATED: This file is no longer used for Docker builds or CI.
|
||||
# Docker and CI both use poetry/pyproject.toml directly.
|
||||
# This file is kept for reference only.
|
||||
|
||||
fastapi>=0.104.0,<1.0.0
|
||||
uvicorn[standard]>=0.24.0,<1.0.0
|
||||
pydantic>=2.5.0,<3.0.0
|
||||
pydantic-settings>=2.1.0,<3.0.0
|
||||
python-multipart>=0.0.6,<1.0.0
|
||||
httpx>=0.25.0,<1.0.0
|
||||
google-cloud-firestore>=2.14.0,<3.0.0
|
||||
google-cloud-storage>=2.14.0,<3.0.0
|
||||
google-cloud-secret-manager>=2.18.0,<3.0.0
|
||||
google-cloud-tasks>=2.14.0,<3.0.0
|
||||
google-cloud-run>=0.10.0,<1.0.0
|
||||
python-dotenv>=1.0.0,<2.0.0
|
||||
|
||||
# Audio processing (reuses karaoke_gen library)
|
||||
audio-separator[remote]>=0.18.0 # Remote Modal API client
|
||||
pydub>=0.25.1 # Audio manipulation (used by karaoke_gen)
|
||||
yt-dlp>=2024.0.0 # YouTube downloads (future use)
|
||||
|
||||
# Cloud storage native APIs
|
||||
dropbox>=11.0.0 # Dropbox Python SDK for native API access
|
||||
# google-api-python-client already installed via karaoke_gen for YouTube
|
||||
|
||||
# karaoke-gen dependencies (will be installed from parent dir in Dockerfile)
|
||||
# Using explicit dependencies instead of -e ../
|
||||
|
||||
Executable
+60
@@ -0,0 +1,60 @@
|
||||
#!/bin/bash
|
||||
# Backend Test Runner
|
||||
# Runs integration tests against the deployed Cloud Run service
|
||||
|
||||
set -e
|
||||
|
||||
echo "🧪 Backend Integration Test Suite"
|
||||
echo "=================================="
|
||||
echo ""
|
||||
|
||||
# Check prerequisites
|
||||
echo "Checking prerequisites..."
|
||||
if ! command -v gcloud &> /dev/null; then
|
||||
echo "❌ gcloud CLI not found. Please install Google Cloud SDK."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v pytest &> /dev/null; then
|
||||
echo "❌ pytest not found. Installing test dependencies..."
|
||||
pip install -r tests/requirements.txt
|
||||
fi
|
||||
|
||||
# Verify authentication
|
||||
echo "Verifying authentication..."
|
||||
if ! gcloud auth print-identity-token &> /dev/null; then
|
||||
echo "❌ Not authenticated with gcloud. Run: gcloud auth login"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "✅ Prerequisites met"
|
||||
echo ""
|
||||
|
||||
# Run tests
|
||||
echo "Running integration tests..."
|
||||
echo ""
|
||||
|
||||
# Run fast tests (no slow marker)
|
||||
echo "📝 Running fast tests..."
|
||||
pytest tests/test_api_integration.py -v -m "not slow" --tb=short
|
||||
|
||||
echo ""
|
||||
echo "✅ Fast tests complete!"
|
||||
echo ""
|
||||
|
||||
# Ask if user wants to run slow tests
|
||||
read -p "Run slow/integration tests? These test actual job processing (5-10 min). [y/N] " -n 1 -r
|
||||
echo
|
||||
if [[ $REPLY =~ ^[Yy]$ ]]; then
|
||||
echo "🐌 Running slow integration tests..."
|
||||
pytest tests/test_api_integration.py -v -m "slow" --tb=short
|
||||
echo ""
|
||||
echo "✅ All tests complete!"
|
||||
else
|
||||
echo "⏭️ Skipping slow tests"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "=================================="
|
||||
echo "Test run complete! 🎉"
|
||||
|
||||
Whitespace-only changes.
@@ -0,0 +1,243 @@
|
||||
"""
|
||||
Audio analysis service for analyzing backing vocals.
|
||||
|
||||
This service wraps the shared karaoke_gen.instrumental_review module
|
||||
to provide GCS-integrated audio analysis capabilities.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
from typing import Optional
|
||||
|
||||
from backend.services.storage_service import StorageService
|
||||
from karaoke_gen.instrumental_review import (
|
||||
AudioAnalyzer,
|
||||
AnalysisResult,
|
||||
WaveformGenerator,
|
||||
)
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AudioAnalysisService:
|
||||
"""
|
||||
Service for analyzing backing vocals audio files stored in GCS.
|
||||
|
||||
This service acts as a thin wrapper around the shared AudioAnalyzer
|
||||
and WaveformGenerator classes, handling GCS download/upload operations.
|
||||
|
||||
The actual analysis logic is in the shared karaoke_gen.instrumental_review
|
||||
module, ensuring feature parity between local and remote workflows.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
storage_service: Optional[StorageService] = None,
|
||||
silence_threshold_db: float = -40.0,
|
||||
min_segment_duration_ms: int = 100,
|
||||
):
|
||||
"""
|
||||
Initialize the audio analysis service.
|
||||
|
||||
Args:
|
||||
storage_service: GCS storage service. If not provided, a new
|
||||
instance will be created.
|
||||
silence_threshold_db: Threshold for considering audio as silent.
|
||||
Default is -40.0 dB.
|
||||
min_segment_duration_ms: Minimum duration for audible segments.
|
||||
Default is 100ms.
|
||||
"""
|
||||
self.storage_service = storage_service or StorageService()
|
||||
self.analyzer = AudioAnalyzer(
|
||||
silence_threshold_db=silence_threshold_db,
|
||||
min_segment_duration_ms=min_segment_duration_ms,
|
||||
)
|
||||
self.waveform_generator = WaveformGenerator()
|
||||
|
||||
def analyze_backing_vocals(
|
||||
self,
|
||||
gcs_audio_path: str,
|
||||
job_id: str,
|
||||
) -> AnalysisResult:
|
||||
"""
|
||||
Analyze a backing vocals audio file from GCS.
|
||||
|
||||
This method:
|
||||
1. Downloads the audio file from GCS to a temp file
|
||||
2. Runs the analysis using the shared AudioAnalyzer
|
||||
3. Returns the analysis result
|
||||
|
||||
Args:
|
||||
gcs_audio_path: Path to the audio file in GCS
|
||||
job_id: Job ID for logging
|
||||
|
||||
Returns:
|
||||
AnalysisResult containing analysis data
|
||||
"""
|
||||
logger.info(f"[{job_id}] Analyzing backing vocals: {gcs_audio_path}")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download audio file
|
||||
local_audio_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
self.storage_service.download_file(gcs_audio_path, local_audio_path)
|
||||
|
||||
# Run analysis
|
||||
result = self.analyzer.analyze(local_audio_path)
|
||||
|
||||
logger.info(
|
||||
f"[{job_id}] Analysis complete: "
|
||||
f"has_audible={result.has_audible_content}, "
|
||||
f"segments={result.segment_count}, "
|
||||
f"recommendation={result.recommended_selection.value}"
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
def analyze_and_generate_waveform(
|
||||
self,
|
||||
gcs_audio_path: str,
|
||||
job_id: str,
|
||||
gcs_waveform_destination: str,
|
||||
) -> tuple[AnalysisResult, str]:
|
||||
"""
|
||||
Analyze backing vocals and generate a waveform image.
|
||||
|
||||
This method:
|
||||
1. Downloads the audio file from GCS
|
||||
2. Runs analysis using AudioAnalyzer
|
||||
3. Generates waveform image using WaveformGenerator
|
||||
4. Uploads the waveform image to GCS
|
||||
5. Returns analysis result and waveform GCS path
|
||||
|
||||
Args:
|
||||
gcs_audio_path: Path to the audio file in GCS
|
||||
job_id: Job ID for logging
|
||||
gcs_waveform_destination: Where to upload the waveform image in GCS
|
||||
|
||||
Returns:
|
||||
Tuple of (AnalysisResult, waveform_gcs_path)
|
||||
"""
|
||||
logger.info(f"[{job_id}] Analyzing and generating waveform: {gcs_audio_path}")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download audio file
|
||||
local_audio_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
self.storage_service.download_file(gcs_audio_path, local_audio_path)
|
||||
|
||||
# Run analysis
|
||||
result = self.analyzer.analyze(local_audio_path)
|
||||
|
||||
# Generate waveform
|
||||
local_waveform_path = os.path.join(temp_dir, "waveform.png")
|
||||
self.waveform_generator.generate(
|
||||
audio_path=local_audio_path,
|
||||
output_path=local_waveform_path,
|
||||
segments=result.audible_segments,
|
||||
show_time_axis=True,
|
||||
silence_threshold_db=self.analyzer.silence_threshold_db,
|
||||
)
|
||||
|
||||
# Upload waveform to GCS
|
||||
self.storage_service.upload_file(
|
||||
local_waveform_path,
|
||||
gcs_waveform_destination
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"[{job_id}] Analysis and waveform generation complete. "
|
||||
f"Waveform uploaded to: {gcs_waveform_destination}"
|
||||
)
|
||||
|
||||
return result, gcs_waveform_destination
|
||||
|
||||
def get_waveform_data(
|
||||
self,
|
||||
gcs_audio_path: str,
|
||||
job_id: str,
|
||||
num_points: int = 500,
|
||||
) -> tuple[list[float], float]:
|
||||
"""
|
||||
Get waveform data (amplitude envelope) for client-side rendering.
|
||||
|
||||
This is useful when the frontend wants to render the waveform
|
||||
itself using Canvas or SVG, rather than displaying a pre-generated
|
||||
image.
|
||||
|
||||
Args:
|
||||
gcs_audio_path: Path to the audio file in GCS
|
||||
job_id: Job ID for logging
|
||||
num_points: Number of data points to return
|
||||
|
||||
Returns:
|
||||
Tuple of (amplitude_values, duration_seconds)
|
||||
"""
|
||||
logger.info(f"[{job_id}] Getting waveform data: {gcs_audio_path}")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download audio file
|
||||
local_audio_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
self.storage_service.download_file(gcs_audio_path, local_audio_path)
|
||||
|
||||
# Generate waveform data
|
||||
amplitudes, duration = self.waveform_generator.generate_data_only(
|
||||
audio_path=local_audio_path,
|
||||
num_points=num_points,
|
||||
)
|
||||
|
||||
return amplitudes, duration
|
||||
|
||||
def generate_waveform_with_mutes(
|
||||
self,
|
||||
gcs_audio_path: str,
|
||||
job_id: str,
|
||||
gcs_waveform_destination: str,
|
||||
mute_regions: list,
|
||||
) -> str:
|
||||
"""
|
||||
Generate a waveform image with mute regions highlighted.
|
||||
|
||||
This is useful for showing the user which regions will be muted
|
||||
in the custom instrumental.
|
||||
|
||||
Args:
|
||||
gcs_audio_path: Path to the audio file in GCS
|
||||
job_id: Job ID for logging
|
||||
gcs_waveform_destination: Where to upload the waveform image
|
||||
mute_regions: List of MuteRegion objects to highlight
|
||||
|
||||
Returns:
|
||||
GCS path to the uploaded waveform image
|
||||
"""
|
||||
from karaoke_gen.instrumental_review import MuteRegion
|
||||
|
||||
logger.info(
|
||||
f"[{job_id}] Generating waveform with {len(mute_regions)} mute regions"
|
||||
)
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download audio file
|
||||
local_audio_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
self.storage_service.download_file(gcs_audio_path, local_audio_path)
|
||||
|
||||
# First run analysis to get segments
|
||||
result = self.analyzer.analyze(local_audio_path)
|
||||
|
||||
# Generate waveform with mute regions
|
||||
local_waveform_path = os.path.join(temp_dir, "waveform_with_mutes.png")
|
||||
self.waveform_generator.generate(
|
||||
audio_path=local_audio_path,
|
||||
output_path=local_waveform_path,
|
||||
segments=result.audible_segments,
|
||||
mute_regions=mute_regions,
|
||||
show_time_axis=True,
|
||||
)
|
||||
|
||||
# Upload to GCS
|
||||
self.storage_service.upload_file(
|
||||
local_waveform_path,
|
||||
gcs_waveform_destination
|
||||
)
|
||||
|
||||
return gcs_waveform_destination
|
||||
@@ -0,0 +1,278 @@
|
||||
"""
|
||||
Audio editing service for creating custom instrumentals.
|
||||
|
||||
This service wraps the shared karaoke_gen.instrumental_review module
|
||||
to provide GCS-integrated audio editing capabilities.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
from typing import List, Optional
|
||||
|
||||
from backend.services.storage_service import StorageService
|
||||
from karaoke_gen.instrumental_review import (
|
||||
AudioEditor,
|
||||
MuteRegion,
|
||||
)
|
||||
from karaoke_gen.instrumental_review.models import CustomInstrumentalResult
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AudioEditingService:
|
||||
"""
|
||||
Service for creating custom instrumental tracks with GCS integration.
|
||||
|
||||
This service acts as a thin wrapper around the shared AudioEditor
|
||||
class, handling GCS download/upload operations.
|
||||
|
||||
The actual editing logic is in the shared karaoke_gen.instrumental_review
|
||||
module, ensuring feature parity between local and remote workflows.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
storage_service: Optional[StorageService] = None,
|
||||
output_format: str = "flac",
|
||||
):
|
||||
"""
|
||||
Initialize the audio editing service.
|
||||
|
||||
Args:
|
||||
storage_service: GCS storage service. If not provided, a new
|
||||
instance will be created.
|
||||
output_format: Output audio format. Default is "flac".
|
||||
"""
|
||||
self.storage_service = storage_service or StorageService()
|
||||
self.editor = AudioEditor(output_format=output_format)
|
||||
|
||||
def create_custom_instrumental(
|
||||
self,
|
||||
gcs_clean_instrumental_path: str,
|
||||
gcs_backing_vocals_path: str,
|
||||
mute_regions: List[MuteRegion],
|
||||
gcs_output_path: str,
|
||||
job_id: str,
|
||||
) -> CustomInstrumentalResult:
|
||||
"""
|
||||
Create a custom instrumental by muting regions of backing vocals.
|
||||
|
||||
This method:
|
||||
1. Downloads clean instrumental and backing vocals from GCS
|
||||
2. Applies mute regions to backing vocals
|
||||
3. Combines with clean instrumental
|
||||
4. Uploads result to GCS
|
||||
|
||||
Args:
|
||||
gcs_clean_instrumental_path: Path to clean instrumental in GCS
|
||||
gcs_backing_vocals_path: Path to backing vocals in GCS
|
||||
mute_regions: List of regions to mute in backing vocals
|
||||
gcs_output_path: Where to upload the custom instrumental in GCS
|
||||
job_id: Job ID for logging
|
||||
|
||||
Returns:
|
||||
CustomInstrumentalResult with details about the created file
|
||||
"""
|
||||
logger.info(
|
||||
f"[{job_id}] Creating custom instrumental with "
|
||||
f"{len(mute_regions)} mute regions"
|
||||
)
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download input files
|
||||
local_clean_path = os.path.join(temp_dir, "clean_instrumental.flac")
|
||||
local_backing_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
local_output_path = os.path.join(temp_dir, "custom_instrumental.flac")
|
||||
|
||||
logger.debug(f"[{job_id}] Downloading clean instrumental")
|
||||
self.storage_service.download_file(
|
||||
gcs_clean_instrumental_path,
|
||||
local_clean_path
|
||||
)
|
||||
|
||||
logger.debug(f"[{job_id}] Downloading backing vocals")
|
||||
self.storage_service.download_file(
|
||||
gcs_backing_vocals_path,
|
||||
local_backing_path
|
||||
)
|
||||
|
||||
# Create custom instrumental using shared editor
|
||||
result = self.editor.create_custom_instrumental(
|
||||
clean_instrumental_path=local_clean_path,
|
||||
backing_vocals_path=local_backing_path,
|
||||
mute_regions=mute_regions,
|
||||
output_path=local_output_path,
|
||||
)
|
||||
|
||||
# Upload result to GCS
|
||||
logger.debug(f"[{job_id}] Uploading custom instrumental to GCS")
|
||||
self.storage_service.upload_file(local_output_path, gcs_output_path)
|
||||
|
||||
# Update result with GCS path
|
||||
result.output_path = gcs_output_path
|
||||
|
||||
logger.info(
|
||||
f"[{job_id}] Custom instrumental created: {gcs_output_path}, "
|
||||
f"muted {result.total_muted_duration_seconds:.1f}s across "
|
||||
f"{len(result.mute_regions_applied)} regions"
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
def create_preview(
|
||||
self,
|
||||
gcs_clean_instrumental_path: str,
|
||||
gcs_backing_vocals_path: str,
|
||||
mute_regions: List[MuteRegion],
|
||||
gcs_preview_path: str,
|
||||
job_id: str,
|
||||
preview_duration_seconds: Optional[float] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Create a preview of the custom instrumental.
|
||||
|
||||
This is similar to create_custom_instrumental but optimized for
|
||||
quick preview generation (optionally truncated duration).
|
||||
|
||||
Args:
|
||||
gcs_clean_instrumental_path: Path to clean instrumental in GCS
|
||||
gcs_backing_vocals_path: Path to backing vocals in GCS
|
||||
mute_regions: List of regions to mute
|
||||
gcs_preview_path: Where to upload the preview in GCS
|
||||
job_id: Job ID for logging
|
||||
preview_duration_seconds: Optional max duration for preview
|
||||
|
||||
Returns:
|
||||
GCS path to the uploaded preview
|
||||
"""
|
||||
logger.info(f"[{job_id}] Creating preview with {len(mute_regions)} mute regions")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download input files
|
||||
local_clean_path = os.path.join(temp_dir, "clean_instrumental.flac")
|
||||
local_backing_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
local_preview_path = os.path.join(temp_dir, "preview.flac")
|
||||
|
||||
self.storage_service.download_file(
|
||||
gcs_clean_instrumental_path,
|
||||
local_clean_path
|
||||
)
|
||||
self.storage_service.download_file(
|
||||
gcs_backing_vocals_path,
|
||||
local_backing_path
|
||||
)
|
||||
|
||||
# Generate preview
|
||||
from pydub import AudioSegment
|
||||
|
||||
preview = self.editor.preview_with_mutes(
|
||||
clean_instrumental_path=local_clean_path,
|
||||
backing_vocals_path=local_backing_path,
|
||||
mute_regions=mute_regions,
|
||||
)
|
||||
|
||||
# Optionally truncate
|
||||
if preview_duration_seconds:
|
||||
max_ms = int(preview_duration_seconds * 1000)
|
||||
preview = preview[:max_ms]
|
||||
|
||||
# Export and upload
|
||||
preview.export(local_preview_path, format="flac")
|
||||
self.storage_service.upload_file(local_preview_path, gcs_preview_path)
|
||||
|
||||
logger.info(f"[{job_id}] Preview created: {gcs_preview_path}")
|
||||
|
||||
return gcs_preview_path
|
||||
|
||||
def mute_backing_vocals_only(
|
||||
self,
|
||||
gcs_backing_vocals_path: str,
|
||||
mute_regions: List[MuteRegion],
|
||||
gcs_output_path: str,
|
||||
job_id: str,
|
||||
) -> str:
|
||||
"""
|
||||
Apply mute regions to backing vocals without combining with instrumental.
|
||||
|
||||
This is useful for creating an edited backing vocals track that
|
||||
can be combined with the instrumental later.
|
||||
|
||||
Args:
|
||||
gcs_backing_vocals_path: Path to backing vocals in GCS
|
||||
mute_regions: List of regions to mute
|
||||
gcs_output_path: Where to upload the edited backing vocals
|
||||
job_id: Job ID for logging
|
||||
|
||||
Returns:
|
||||
GCS path to the uploaded edited backing vocals
|
||||
"""
|
||||
logger.info(
|
||||
f"[{job_id}] Muting backing vocals with {len(mute_regions)} regions"
|
||||
)
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
local_backing_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
local_output_path = os.path.join(temp_dir, "muted_backing.flac")
|
||||
|
||||
self.storage_service.download_file(
|
||||
gcs_backing_vocals_path,
|
||||
local_backing_path
|
||||
)
|
||||
|
||||
self.editor.apply_mute_to_single_track(
|
||||
audio_path=local_backing_path,
|
||||
mute_regions=mute_regions,
|
||||
output_path=local_output_path,
|
||||
)
|
||||
|
||||
self.storage_service.upload_file(local_output_path, gcs_output_path)
|
||||
|
||||
logger.info(f"[{job_id}] Muted backing vocals: {gcs_output_path}")
|
||||
|
||||
return gcs_output_path
|
||||
|
||||
def validate_mute_regions(
|
||||
self,
|
||||
mute_regions: List[MuteRegion],
|
||||
total_duration_seconds: float,
|
||||
) -> List[str]:
|
||||
"""
|
||||
Validate mute regions for consistency.
|
||||
|
||||
Args:
|
||||
mute_regions: List of mute regions to validate
|
||||
total_duration_seconds: Total duration of the audio
|
||||
|
||||
Returns:
|
||||
List of validation error messages (empty if valid)
|
||||
"""
|
||||
errors = []
|
||||
|
||||
for i, region in enumerate(mute_regions):
|
||||
if region.start_seconds < 0:
|
||||
errors.append(
|
||||
f"Region {i}: start_seconds ({region.start_seconds}) cannot be negative"
|
||||
)
|
||||
|
||||
if region.end_seconds <= region.start_seconds:
|
||||
errors.append(
|
||||
f"Region {i}: end_seconds ({region.end_seconds}) must be after "
|
||||
f"start_seconds ({region.start_seconds})"
|
||||
)
|
||||
|
||||
if region.start_seconds > total_duration_seconds:
|
||||
errors.append(
|
||||
f"Region {i}: start_seconds ({region.start_seconds}) exceeds "
|
||||
f"audio duration ({total_duration_seconds})"
|
||||
)
|
||||
|
||||
if region.end_seconds > total_duration_seconds:
|
||||
# Not an error, but log a warning - the region will be clamped
|
||||
logger.warning(
|
||||
f"Region {i}: end_seconds ({region.end_seconds}) exceeds "
|
||||
f"audio duration ({total_duration_seconds}), will be clamped"
|
||||
)
|
||||
|
||||
return errors
|
||||
@@ -0,0 +1,702 @@
|
||||
"""
|
||||
Audio search service for finding and downloading audio files.
|
||||
|
||||
This service integrates with karaoke_gen.audio_fetcher (which wraps flacfetch)
|
||||
to search for audio from various sources (YouTube, music trackers, etc.)
|
||||
and download the selected audio file.
|
||||
|
||||
This is a thin wrapper that adds backend-specific functionality:
|
||||
- Caching raw results for API-based selection flow
|
||||
- Firestore-compatible serialization
|
||||
- Singleton pattern for service lifecycle
|
||||
- Remote flacfetch service integration for torrent downloads
|
||||
|
||||
When FLACFETCH_API_URL is configured, this service uses the remote flacfetch
|
||||
HTTP API for search and download operations. This is required for torrent
|
||||
downloads since Cloud Run doesn't support BitTorrent peer connections.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
from typing import List, Optional
|
||||
|
||||
import nest_asyncio
|
||||
|
||||
# Import shared classes from karaoke_gen.audio_fetcher - single source of truth
|
||||
# Note: Import directly from audio_fetcher module, not karaoke_gen.__init__
|
||||
# to avoid pulling in heavier dependencies like lyrics_transcriber
|
||||
from karaoke_gen.audio_fetcher import (
|
||||
FlacFetcher,
|
||||
AudioSearchResult,
|
||||
AudioFetchResult,
|
||||
AudioFetcherError,
|
||||
NoResultsError,
|
||||
DownloadError,
|
||||
)
|
||||
|
||||
from .flacfetch_client import get_flacfetch_client, FlacfetchServiceError
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Re-export exception under backend's naming convention for compatibility
|
||||
AudioSearchError = AudioFetcherError
|
||||
|
||||
|
||||
# Also alias AudioFetchResult as AudioDownloadResult for backwards compatibility
|
||||
AudioDownloadResult = AudioFetchResult
|
||||
|
||||
|
||||
class AudioSearchService:
|
||||
"""
|
||||
Service for searching and downloading audio files.
|
||||
|
||||
This is a thin wrapper around karaoke_gen.audio_fetcher.FlacFetcher that adds
|
||||
backend-specific functionality:
|
||||
- Caches raw results for API-based selection (client picks by index)
|
||||
- Provides Firestore-compatible serialization
|
||||
- Manages service lifecycle as singleton
|
||||
- Remote flacfetch service integration for torrent downloads
|
||||
|
||||
When FLACFETCH_API_URL is configured, this service uses the remote flacfetch
|
||||
HTTP API for search and download operations. This enables torrent downloads
|
||||
which are not possible in Cloud Run due to network restrictions.
|
||||
|
||||
The actual flacfetch integration is in karaoke_gen.audio_fetcher.FlacFetcher,
|
||||
which is shared between local CLI and cloud backend.
|
||||
"""
|
||||
|
||||
# Sentinel value to indicate "use environment variable"
|
||||
_USE_ENV = object()
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
red_api_key: Optional[str] = _USE_ENV,
|
||||
red_api_url: Optional[str] = _USE_ENV,
|
||||
ops_api_key: Optional[str] = _USE_ENV,
|
||||
ops_api_url: Optional[str] = _USE_ENV,
|
||||
):
|
||||
"""
|
||||
Initialize the audio search service.
|
||||
|
||||
Args:
|
||||
red_api_key: API key for RED tracker (optional, uses env if not provided)
|
||||
red_api_url: Base URL for RED tracker API (optional, uses env if not provided)
|
||||
ops_api_key: API key for OPS tracker (optional, uses env if not provided)
|
||||
ops_api_url: Base URL for OPS tracker API (optional, uses env if not provided)
|
||||
"""
|
||||
# Use environment variables if not explicitly provided
|
||||
if red_api_key is self._USE_ENV:
|
||||
red_api_key = os.environ.get("RED_API_KEY")
|
||||
if red_api_url is self._USE_ENV:
|
||||
red_api_url = os.environ.get("RED_API_URL")
|
||||
|
||||
if ops_api_key is self._USE_ENV:
|
||||
ops_api_key = os.environ.get("OPS_API_KEY")
|
||||
if ops_api_url is self._USE_ENV:
|
||||
ops_api_url = os.environ.get("OPS_API_URL")
|
||||
|
||||
# Check for remote flacfetch client
|
||||
self._remote_client = get_flacfetch_client()
|
||||
|
||||
# Log which mode we're using
|
||||
if self._remote_client:
|
||||
logger.info("AudioSearchService using REMOTE flacfetch service (torrent downloads enabled)")
|
||||
else:
|
||||
logger.info("AudioSearchService using LOCAL flacfetch (YouTube only in Cloud Run)")
|
||||
|
||||
# Delegate to shared FlacFetcher implementation (for local mode or fallback)
|
||||
self._fetcher = FlacFetcher(
|
||||
red_api_key=red_api_key,
|
||||
red_api_url=red_api_url,
|
||||
ops_api_key=ops_api_key,
|
||||
ops_api_url=ops_api_url,
|
||||
)
|
||||
|
||||
# Cache search results for API-based selection flow
|
||||
# Key: index, Value: AudioSearchResult (with raw_result)
|
||||
self._cached_results: List[AudioSearchResult] = []
|
||||
|
||||
# Cache for remote search results (search_id -> results)
|
||||
self._remote_search_id: Optional[str] = None
|
||||
|
||||
def search(self, artist: str, title: str) -> List[AudioSearchResult]:
|
||||
"""
|
||||
Search for audio matching the given artist and title.
|
||||
|
||||
Results are cached internally for later download via download().
|
||||
|
||||
If a remote flacfetch service is configured (FLACFETCH_API_URL), uses that
|
||||
for better torrent download support. Otherwise falls back to local flacfetch.
|
||||
|
||||
Args:
|
||||
artist: The artist name to search for
|
||||
title: The track title to search for
|
||||
|
||||
Returns:
|
||||
List of AudioSearchResult objects
|
||||
|
||||
Raises:
|
||||
NoResultsError: If no results are found
|
||||
AudioSearchError: For other errors (e.g., flacfetch not installed)
|
||||
"""
|
||||
try:
|
||||
# Try remote flacfetch service first if configured
|
||||
if self._remote_client:
|
||||
return self._search_remote(artist, title)
|
||||
|
||||
# Fallback to local flacfetch
|
||||
results = self._fetcher.search(artist, title)
|
||||
|
||||
# Cache results for later download
|
||||
self._cached_results = results
|
||||
self._remote_search_id = None
|
||||
|
||||
logger.info(f"Found {len(results)} results for: {artist} - {title}")
|
||||
return results
|
||||
|
||||
except (NoResultsError, AudioFetcherError):
|
||||
# Re-raise these as-is
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Search failed: {e}")
|
||||
raise AudioSearchError(f"Search failed: {e}") from e
|
||||
|
||||
def _search_remote(self, artist: str, title: str) -> List[AudioSearchResult]:
|
||||
"""
|
||||
Search using the remote flacfetch service.
|
||||
|
||||
Runs async code in a sync context for compatibility with existing API.
|
||||
"""
|
||||
try:
|
||||
# Enable nested event loops (needed when called from FastAPI async context)
|
||||
# Apply to the current loop if one exists, otherwise it's a no-op
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
nest_asyncio.apply(loop)
|
||||
except RuntimeError:
|
||||
pass # No running loop, nest_asyncio not needed
|
||||
|
||||
# Run async search in sync context
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
try:
|
||||
response = loop.run_until_complete(
|
||||
self._remote_client.search(artist, title)
|
||||
)
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
# Check for empty results
|
||||
if not response.get("results"):
|
||||
raise NoResultsError(f"No results found for: {artist} - {title}")
|
||||
|
||||
# Store search_id for download
|
||||
self._remote_search_id = response.get("search_id")
|
||||
|
||||
# Convert remote results to AudioSearchResult objects
|
||||
results = []
|
||||
for item in response.get("results", []):
|
||||
result = AudioSearchResult(
|
||||
title=item.get("title", "Unknown"),
|
||||
artist=item.get("artist", "Unknown"),
|
||||
url=item.get("download_url", ""), # May be empty for remote
|
||||
provider=item.get("provider", "Unknown"),
|
||||
duration=item.get("duration_seconds"),
|
||||
quality=item.get("quality", ""),
|
||||
source_id=item.get("info_hash"),
|
||||
index=item.get("index", 0),
|
||||
seeders=item.get("seeders"),
|
||||
target_file=item.get("target_file"),
|
||||
# Store full remote data in raw_result for rich display
|
||||
raw_result=item,
|
||||
)
|
||||
results.append(result)
|
||||
|
||||
# Cache results
|
||||
self._cached_results = results
|
||||
|
||||
logger.info(f"Found {len(results)} results from remote flacfetch for: {artist} - {title}")
|
||||
return results
|
||||
|
||||
except FlacfetchServiceError as e:
|
||||
logger.warning(f"Remote flacfetch search failed, falling back to local: {e}")
|
||||
# Fallback to local search
|
||||
self._remote_search_id = None
|
||||
results = self._fetcher.search(artist, title)
|
||||
self._cached_results = results
|
||||
return results
|
||||
|
||||
def select_best(self, results: List[AudioSearchResult]) -> int:
|
||||
"""
|
||||
Select the best result from a list of search results.
|
||||
|
||||
Uses flacfetch's built-in quality ranking to select the best source.
|
||||
|
||||
Args:
|
||||
results: List of search results
|
||||
|
||||
Returns:
|
||||
Index of the best result
|
||||
"""
|
||||
return self._fetcher.select_best(results)
|
||||
|
||||
def download(
|
||||
self,
|
||||
result_index: int,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
remote_search_id: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Download audio from a cached search result.
|
||||
|
||||
This method uses the cached results from the last search() call.
|
||||
The API flow is:
|
||||
1. Client calls search() -> gets list of results
|
||||
2. Client picks an index
|
||||
3. Client calls download(index) -> gets downloaded file
|
||||
|
||||
If a remote flacfetch service is configured and the search was performed
|
||||
remotely (for torrent sources), uses the remote service for download.
|
||||
|
||||
Args:
|
||||
result_index: Index of the result to download (from search results)
|
||||
output_dir: Directory to save the downloaded file
|
||||
output_filename: Optional filename (without extension)
|
||||
gcs_path: Optional GCS path for remote uploads (e.g., "uploads/job123/audio/")
|
||||
remote_search_id: Optional search_id for remote downloads. If provided,
|
||||
uses this instead of the cached _remote_search_id. This is important
|
||||
for concurrent requests where the service-level cache may be stale.
|
||||
|
||||
Returns:
|
||||
AudioDownloadResult with the downloaded file path
|
||||
|
||||
Raises:
|
||||
DownloadError: If download fails or no cached result for index
|
||||
"""
|
||||
# Use provided search_id or fall back to service-level cache
|
||||
effective_search_id = remote_search_id or self._remote_search_id
|
||||
|
||||
# Check if we have local cached results
|
||||
has_local_cache = result_index >= 0 and result_index < len(self._cached_results)
|
||||
|
||||
# If no local cache but we have remote search_id, use remote download
|
||||
# This handles multi-instance deployments where the cache doesn't persist
|
||||
if not has_local_cache:
|
||||
if effective_search_id and self._remote_client:
|
||||
logger.info(f"No local cache, using remote download with search_id={effective_search_id}, index={result_index}")
|
||||
return self._download_remote(result_index, output_dir, output_filename, gcs_path, effective_search_id)
|
||||
else:
|
||||
# Provide clear error message based on whether cache is empty or index out of bounds
|
||||
if len(self._cached_results) == 0:
|
||||
raise DownloadError(
|
||||
f"No cached result for index {result_index}. "
|
||||
"No cached results available. Run search() first."
|
||||
)
|
||||
raise DownloadError(
|
||||
f"No cached result for index {result_index}. "
|
||||
f"Available indices: 0-{len(self._cached_results) - 1}. "
|
||||
"Run search() first."
|
||||
)
|
||||
|
||||
result = self._cached_results[result_index]
|
||||
|
||||
logger.info(f"Downloading: {result.artist} - {result.title} from {result.provider}")
|
||||
|
||||
# Check if this was a remote search (torrent sources need remote download)
|
||||
if effective_search_id and self._remote_client:
|
||||
# Check if this is a torrent source that needs remote handling
|
||||
is_torrent = result.provider in ["RED", "OPS"]
|
||||
|
||||
if is_torrent:
|
||||
return self._download_remote(result_index, output_dir, output_filename, gcs_path, effective_search_id)
|
||||
|
||||
# Delegate to shared FlacFetcher (local download)
|
||||
fetch_result = self._fetcher.download(result, output_dir, output_filename)
|
||||
|
||||
logger.info(f"Downloaded to: {fetch_result.filepath}")
|
||||
|
||||
return fetch_result
|
||||
|
||||
def download_by_id(
|
||||
self,
|
||||
source_name: str,
|
||||
source_id: str,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
target_file: Optional[str] = None,
|
||||
download_url: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Download audio directly by source ID (no prior search required).
|
||||
|
||||
This is the preferred method when you have stored the source_id from a
|
||||
previous search and want to download later without re-searching. This
|
||||
avoids unnecessary API calls to private trackers.
|
||||
|
||||
Args:
|
||||
source_name: Provider name (RED, OPS, YouTube, Spotify)
|
||||
source_id: Source-specific ID (torrent ID, video ID, track ID)
|
||||
output_dir: Directory to save the downloaded file
|
||||
output_filename: Optional filename (without extension)
|
||||
target_file: For torrents, specific file to extract from the torrent
|
||||
download_url: For YouTube/Spotify, direct URL (optional)
|
||||
gcs_path: Optional GCS path for remote uploads
|
||||
|
||||
Returns:
|
||||
AudioDownloadResult with the downloaded file path
|
||||
|
||||
Raises:
|
||||
DownloadError: If download fails
|
||||
"""
|
||||
logger.info(f"Download by ID: {source_name} ID={source_id}")
|
||||
|
||||
# For torrent sources, must use remote client
|
||||
if source_name in ["RED", "OPS"]:
|
||||
if not self._remote_client:
|
||||
raise DownloadError(
|
||||
f"Cannot download from {source_name} without remote flacfetch service"
|
||||
)
|
||||
return self._download_by_id_remote(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir=output_dir,
|
||||
output_filename=output_filename,
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_path,
|
||||
)
|
||||
|
||||
# For YouTube/Spotify, can use local flacfetch via FetchManager.download_by_id
|
||||
# But we currently only have remote support, so use remote if available
|
||||
if self._remote_client:
|
||||
return self._download_by_id_remote(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir=output_dir,
|
||||
output_filename=output_filename,
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_path,
|
||||
)
|
||||
|
||||
# Local download for YouTube/Spotify (fallback)
|
||||
# Use the local fetcher's download_by_id if available
|
||||
raise DownloadError(
|
||||
f"Local download_by_id not yet implemented for {source_name}. "
|
||||
"Configure FLACFETCH_API_URL for remote downloads."
|
||||
)
|
||||
|
||||
def _download_by_id_remote(
|
||||
self,
|
||||
source_name: str,
|
||||
source_id: str,
|
||||
output_dir: str, # Not used for remote downloads (files go to GCS or remote server)
|
||||
output_filename: Optional[str] = None,
|
||||
target_file: Optional[str] = None,
|
||||
download_url: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Download by ID using the remote flacfetch service.
|
||||
|
||||
Note: output_dir is accepted for API compatibility but ignored for remote
|
||||
downloads. Remote downloads either go to GCS (if gcs_path is set) or to
|
||||
the remote server's download directory.
|
||||
"""
|
||||
# output_dir is intentionally unused - remote downloads don't use local paths
|
||||
_ = output_dir
|
||||
try:
|
||||
# Enable nested event loops
|
||||
try:
|
||||
running_loop = asyncio.get_running_loop()
|
||||
nest_asyncio.apply(running_loop)
|
||||
except RuntimeError:
|
||||
pass
|
||||
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
try:
|
||||
# Start download
|
||||
download_id = loop.run_until_complete(
|
||||
self._remote_client.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_filename=output_filename,
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_path,
|
||||
)
|
||||
)
|
||||
|
||||
logger.info(f"Remote download by ID started: {download_id}")
|
||||
|
||||
# Wait for completion
|
||||
def log_progress(status):
|
||||
progress = status.get("progress", 0)
|
||||
speed = status.get("download_speed_kbps", 0)
|
||||
logger.debug(f"Download progress: {progress:.1f}% ({speed:.1f} KB/s)")
|
||||
|
||||
final_status = loop.run_until_complete(
|
||||
self._remote_client.wait_for_download(
|
||||
download_id,
|
||||
timeout=600,
|
||||
progress_callback=log_progress,
|
||||
)
|
||||
)
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
# Determine file path
|
||||
filepath = final_status.get("gcs_path") or final_status.get("output_path")
|
||||
|
||||
if not filepath:
|
||||
raise DownloadError("Remote download completed but no file path returned")
|
||||
|
||||
logger.info(f"Remote download by ID complete: {filepath}")
|
||||
|
||||
return AudioDownloadResult(
|
||||
filepath=filepath,
|
||||
artist="", # Not available without search
|
||||
title="", # Not available without search
|
||||
provider=source_name,
|
||||
quality="", # Not available without search
|
||||
)
|
||||
|
||||
except FlacfetchServiceError as e:
|
||||
raise DownloadError(f"Remote download by ID failed: {e}") from e
|
||||
except Exception as e:
|
||||
logger.error(f"Remote download by ID error: {e}")
|
||||
raise DownloadError(f"Remote download by ID failed: {e}") from e
|
||||
|
||||
def _download_remote(
|
||||
self,
|
||||
result_index: int,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
search_id: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Download using the remote flacfetch service.
|
||||
|
||||
The remote service downloads via torrent and optionally uploads to GCS.
|
||||
|
||||
Args:
|
||||
result_index: Index of the result to download
|
||||
output_dir: Directory to save downloaded file
|
||||
output_filename: Optional filename
|
||||
gcs_path: Optional GCS path for remote uploads
|
||||
search_id: Remote search ID to use (overrides self._remote_search_id)
|
||||
"""
|
||||
effective_search_id = search_id or self._remote_search_id
|
||||
if not effective_search_id:
|
||||
raise DownloadError("No remote search ID - run search() first")
|
||||
|
||||
# Try to get local result for metadata, but it's optional
|
||||
# The remote service maintains its own cache by search_id
|
||||
result = None
|
||||
if result_index >= 0 and result_index < len(self._cached_results):
|
||||
result = self._cached_results[result_index]
|
||||
|
||||
try:
|
||||
# Enable nested event loops (needed when called from FastAPI async context)
|
||||
try:
|
||||
running_loop = asyncio.get_running_loop()
|
||||
nest_asyncio.apply(running_loop)
|
||||
except RuntimeError:
|
||||
pass # No running loop, nest_asyncio not needed
|
||||
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
try:
|
||||
# Start download
|
||||
download_id = loop.run_until_complete(
|
||||
self._remote_client.download(
|
||||
search_id=effective_search_id,
|
||||
result_index=result_index,
|
||||
output_filename=output_filename,
|
||||
gcs_path=gcs_path,
|
||||
)
|
||||
)
|
||||
|
||||
logger.info(f"Remote download started: {download_id}")
|
||||
|
||||
# Wait for completion with progress logging
|
||||
def log_progress(status):
|
||||
progress = status.get("progress", 0)
|
||||
speed = status.get("download_speed_kbps", 0)
|
||||
logger.debug(f"Download progress: {progress:.1f}% ({speed:.1f} KB/s)")
|
||||
|
||||
final_status = loop.run_until_complete(
|
||||
self._remote_client.wait_for_download(
|
||||
download_id,
|
||||
timeout=600, # 10 minute timeout
|
||||
progress_callback=log_progress,
|
||||
)
|
||||
)
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
# Determine file path
|
||||
filepath = final_status.get("gcs_path") or final_status.get("output_path")
|
||||
|
||||
if not filepath:
|
||||
raise DownloadError("Remote download completed but no file path returned")
|
||||
|
||||
logger.info(f"Remote download complete: {filepath}")
|
||||
|
||||
# Use local result metadata if available, otherwise use empty strings
|
||||
# (the actual download is handled by the remote service using search_id)
|
||||
return AudioDownloadResult(
|
||||
filepath=filepath,
|
||||
artist=result.artist if result else "",
|
||||
title=result.title if result else "",
|
||||
provider=result.provider if result else "remote",
|
||||
quality=result.quality if result else "",
|
||||
)
|
||||
|
||||
except FlacfetchServiceError as e:
|
||||
raise DownloadError(f"Remote download failed: {e}") from e
|
||||
except Exception as e:
|
||||
logger.error(f"Remote download error: {e}")
|
||||
raise DownloadError(f"Remote download failed: {e}") from e
|
||||
|
||||
def search_and_download_auto(
|
||||
self,
|
||||
artist: str,
|
||||
title: str,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Search for audio and automatically download the best result.
|
||||
|
||||
This is a convenience method that combines search(), select_best(),
|
||||
and download() for automated/non-interactive usage.
|
||||
|
||||
Args:
|
||||
artist: The artist name to search for
|
||||
title: The track title to search for
|
||||
output_dir: Directory to save the downloaded file
|
||||
output_filename: Optional filename (without extension)
|
||||
gcs_path: Optional GCS path for remote uploads
|
||||
|
||||
Returns:
|
||||
AudioDownloadResult with the downloaded file path
|
||||
|
||||
Raises:
|
||||
NoResultsError: If no results are found
|
||||
DownloadError: If download fails
|
||||
"""
|
||||
# Search
|
||||
results = self.search(artist, title)
|
||||
|
||||
# Select best
|
||||
best_index = self.select_best(results)
|
||||
best_result = results[best_index]
|
||||
logger.info(
|
||||
f"Auto-selected result {best_index}: "
|
||||
f"{best_result.artist} - {best_result.title} ({best_result.provider})"
|
||||
)
|
||||
|
||||
# Download
|
||||
return self.download(best_index, output_dir, output_filename, gcs_path)
|
||||
|
||||
async def search_async(self, artist: str, title: str) -> List[AudioSearchResult]:
|
||||
"""
|
||||
Async version of search for use in async routes.
|
||||
|
||||
Note: Currently wraps sync search in executor. Future optimization
|
||||
could make this fully async.
|
||||
"""
|
||||
loop = asyncio.get_event_loop()
|
||||
return await loop.run_in_executor(None, self.search, artist, title)
|
||||
|
||||
async def download_async(
|
||||
self,
|
||||
result_index: int,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
remote_search_id: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Async version of download for use in async routes.
|
||||
|
||||
Note: Currently wraps sync download in executor. Future optimization
|
||||
could make this fully async.
|
||||
"""
|
||||
loop = asyncio.get_event_loop()
|
||||
return await loop.run_in_executor(
|
||||
None,
|
||||
lambda: self.download(result_index, output_dir, output_filename, gcs_path, remote_search_id)
|
||||
)
|
||||
|
||||
async def download_by_id_async(
|
||||
self,
|
||||
source_name: str,
|
||||
source_id: str,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
target_file: Optional[str] = None,
|
||||
download_url: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Async version of download_by_id for use in async routes.
|
||||
"""
|
||||
loop = asyncio.get_event_loop()
|
||||
return await loop.run_in_executor(
|
||||
None,
|
||||
lambda: self.download_by_id(
|
||||
source_name, source_id, output_dir, output_filename,
|
||||
target_file, download_url, gcs_path
|
||||
)
|
||||
)
|
||||
|
||||
def is_remote_enabled(self) -> bool:
|
||||
"""Check if remote flacfetch service is configured."""
|
||||
return self._remote_client is not None
|
||||
|
||||
@property
|
||||
def last_remote_search_id(self) -> Optional[str]:
|
||||
"""Get the search_id from the last remote search.
|
||||
|
||||
This should be stored in job state_data after search and passed
|
||||
back to download() to ensure correct remote download handling.
|
||||
"""
|
||||
return self._remote_search_id
|
||||
|
||||
async def check_remote_health(self) -> Optional[dict]:
|
||||
"""
|
||||
Check health of remote flacfetch service.
|
||||
|
||||
Returns:
|
||||
Health status dict if remote service is configured and healthy,
|
||||
None if not configured or unhealthy.
|
||||
"""
|
||||
if not self._remote_client:
|
||||
return None
|
||||
|
||||
try:
|
||||
return await self._remote_client.health_check()
|
||||
except Exception as e:
|
||||
logger.warning(f"Remote flacfetch health check failed: {e}")
|
||||
return None
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_audio_search_service: Optional[AudioSearchService] = None
|
||||
|
||||
|
||||
def get_audio_search_service() -> AudioSearchService:
|
||||
"""Get the singleton AudioSearchService instance."""
|
||||
global _audio_search_service
|
||||
if _audio_search_service is None:
|
||||
_audio_search_service = AudioSearchService()
|
||||
return _audio_search_service
|
||||
@@ -0,0 +1,630 @@
|
||||
"""
|
||||
Authentication service for karaoke backend.
|
||||
|
||||
Provides token-based authentication with support for:
|
||||
- Admin tokens (hardcoded in environment)
|
||||
- Admin by email domain (@nomadkaraoke.com)
|
||||
- User tokens (stored in Firestore auth_tokens collection)
|
||||
- Session tokens (stored in Firestore sessions collection, from magic link auth)
|
||||
- API keys for business users
|
||||
- Usage tracking and limits
|
||||
- Token management API
|
||||
"""
|
||||
import logging
|
||||
import hashlib
|
||||
import time
|
||||
import secrets
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from typing import Optional, Tuple, Dict, Any, List
|
||||
from datetime import datetime
|
||||
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
from backend.config import get_settings
|
||||
from backend.models.user import UserRole
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Admin email domains - users with these domains are automatically admin
|
||||
ADMIN_EMAIL_DOMAINS = ["nomadkaraoke.com"]
|
||||
|
||||
|
||||
class UserType(str, Enum):
|
||||
"""User access levels."""
|
||||
ADMIN = "admin" # Unlimited access, can manage tokens
|
||||
UNLIMITED = "unlimited" # Unlimited karaoke generation
|
||||
LIMITED = "limited" # Limited number of uses
|
||||
STRIPE = "stripe" # Paid access via Stripe
|
||||
API_KEY = "api_key" # Business API key access
|
||||
|
||||
|
||||
@dataclass
|
||||
class AuthResult:
|
||||
"""Result of authentication validation."""
|
||||
is_valid: bool
|
||||
user_type: UserType
|
||||
remaining_uses: int # -1 = unlimited, 0 = exhausted, >0 = remaining
|
||||
message: str
|
||||
user_email: Optional[str] = None # Email if authenticated via session/API key
|
||||
is_admin: bool = False # True if admin token or admin email domain
|
||||
api_key_id: Optional[str] = None # API key ID if authenticated via API key
|
||||
|
||||
def __iter__(self):
|
||||
"""Allow unpacking as tuple for backward compatibility."""
|
||||
return iter((self.is_valid, self.user_type, self.remaining_uses, self.message))
|
||||
|
||||
|
||||
def is_admin_email(email: str) -> bool:
|
||||
"""Check if an email belongs to an admin domain."""
|
||||
if not email:
|
||||
return False
|
||||
email_lower = email.lower()
|
||||
return any(email_lower.endswith(f"@{domain}") for domain in ADMIN_EMAIL_DOMAINS)
|
||||
|
||||
|
||||
class AuthService:
|
||||
"""Service for authentication and token management."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize auth service."""
|
||||
self.firestore = FirestoreService()
|
||||
self.settings = get_settings()
|
||||
self._init_admin_tokens()
|
||||
|
||||
def _init_admin_tokens(self):
|
||||
"""Load admin tokens from environment variables."""
|
||||
# Get admin tokens from secret manager or env vars
|
||||
admin_tokens_str = self.settings.admin_tokens or ""
|
||||
self.admin_tokens = [t.strip() for t in admin_tokens_str.split(",") if t.strip()]
|
||||
|
||||
if self.admin_tokens:
|
||||
logger.info(f"Loaded {len(self.admin_tokens)} admin token(s)")
|
||||
else:
|
||||
logger.warning("No admin tokens configured! Set ADMIN_TOKENS environment variable.")
|
||||
|
||||
def validate_token(self, token: str) -> Tuple[bool, UserType, int, str]:
|
||||
"""
|
||||
Validate an access token.
|
||||
|
||||
Returns:
|
||||
(is_valid, user_type, remaining_uses, message)
|
||||
remaining_uses: -1 = unlimited, 0 = exhausted, >0 = remaining count
|
||||
"""
|
||||
if not token:
|
||||
return False, UserType.LIMITED, 0, "No token provided"
|
||||
|
||||
# Check for admin tokens (highest priority)
|
||||
if token in self.admin_tokens:
|
||||
logger.info("Admin token validated")
|
||||
return True, UserType.ADMIN, -1, "Admin access granted"
|
||||
|
||||
# Debug: Log token comparison info (only first/last few chars for security)
|
||||
if self.admin_tokens:
|
||||
expected_prefix = self.admin_tokens[0][:8] if len(self.admin_tokens[0]) >= 8 else self.admin_tokens[0]
|
||||
provided_prefix = token[:8] if len(token) >= 8 else token
|
||||
logger.debug(
|
||||
f"Token mismatch: expected prefix '{expected_prefix}...', "
|
||||
f"got '{provided_prefix}...', "
|
||||
f"expected len={len(self.admin_tokens[0])}, got len={len(token)}"
|
||||
)
|
||||
|
||||
# Check stored tokens in Firestore (auth_tokens collection)
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
if not token_data:
|
||||
# Fall back to checking session tokens (from magic link auth)
|
||||
# Import here to avoid circular dependency
|
||||
from backend.services.user_service import get_user_service
|
||||
|
||||
user_service = get_user_service()
|
||||
is_valid, user, message = user_service.validate_session(token)
|
||||
|
||||
if is_valid and user:
|
||||
# Session is valid - user is authenticated via magic link
|
||||
# Return their credits as remaining uses
|
||||
credits = user.credits
|
||||
logger.info(f"Session token validated for user {user.email} (credits: {credits})")
|
||||
|
||||
if credits <= 0:
|
||||
# User has no credits but is authenticated
|
||||
return True, UserType.STRIPE, 0, f"Authenticated but no credits remaining"
|
||||
|
||||
return True, UserType.STRIPE, credits, f"Session valid: {credits} credits"
|
||||
|
||||
# Neither auth_token nor session found
|
||||
return False, UserType.LIMITED, 0, "Invalid token"
|
||||
|
||||
# Check if token is active
|
||||
if not token_data.get("active", True):
|
||||
return False, UserType(token_data["type"]), 0, "Token has been revoked"
|
||||
|
||||
token_type = UserType(token_data["type"])
|
||||
max_uses = token_data.get("max_uses", -1)
|
||||
|
||||
# UNLIMITED tokens: no usage limits
|
||||
if token_type == UserType.UNLIMITED:
|
||||
return True, token_type, -1, "Unlimited access granted"
|
||||
|
||||
# LIMITED tokens: check usage count
|
||||
if token_type == UserType.LIMITED:
|
||||
if max_uses <= 0: # -1 means unlimited
|
||||
return True, token_type, -1, "Limited token with unlimited uses"
|
||||
|
||||
current_uses = token_data.get("usage_count", 0)
|
||||
remaining = max_uses - current_uses
|
||||
|
||||
if remaining <= 0:
|
||||
return False, token_type, 0, "Token usage limit exceeded"
|
||||
|
||||
return True, token_type, remaining, f"Limited token: {remaining} uses remaining"
|
||||
|
||||
# STRIPE tokens: check expiration and usage
|
||||
if token_type == UserType.STRIPE:
|
||||
expires_at = token_data.get("expires_at")
|
||||
|
||||
if expires_at:
|
||||
# expires_at is stored as timestamp
|
||||
if time.time() > expires_at:
|
||||
return False, token_type, 0, "Token has expired"
|
||||
|
||||
if max_uses > 0:
|
||||
current_uses = token_data.get("usage_count", 0)
|
||||
remaining = max_uses - current_uses
|
||||
|
||||
if remaining <= 0:
|
||||
return False, token_type, 0, "Token usage limit exceeded"
|
||||
|
||||
return True, token_type, remaining, f"Stripe token: {remaining} uses remaining"
|
||||
|
||||
return True, token_type, -1, "Stripe access granted"
|
||||
|
||||
return False, UserType.LIMITED, 0, "Unknown token type"
|
||||
|
||||
def validate_token_full(self, token: str) -> AuthResult:
|
||||
"""
|
||||
Validate an access token and return full authentication details.
|
||||
|
||||
This is the preferred method for new code as it returns complete
|
||||
authentication context including user email and admin status.
|
||||
|
||||
Returns:
|
||||
AuthResult with full authentication details
|
||||
"""
|
||||
if not token:
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=UserType.LIMITED,
|
||||
remaining_uses=0,
|
||||
message="No token provided"
|
||||
)
|
||||
|
||||
# Check for admin tokens (highest priority)
|
||||
# Admin tokens from env var are associated with the system admin email
|
||||
if token in self.admin_tokens:
|
||||
logger.info("Admin token validated")
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=-1,
|
||||
message="Admin access granted",
|
||||
user_email="admin@nomadkaraoke.com",
|
||||
is_admin=True
|
||||
)
|
||||
|
||||
# Check stored tokens in Firestore (auth_tokens collection)
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
if not token_data:
|
||||
# Fall back to checking session tokens (from magic link auth)
|
||||
from backend.services.user_service import get_user_service
|
||||
|
||||
user_service = get_user_service()
|
||||
is_valid, user, _message = user_service.validate_session(token)
|
||||
|
||||
if is_valid and user:
|
||||
# Session is valid - user is authenticated via magic link
|
||||
credits = user.credits
|
||||
user_email = user.email
|
||||
|
||||
# Check if user is admin by email domain or role
|
||||
user_is_admin = is_admin_email(user_email) or user.role == UserRole.ADMIN
|
||||
|
||||
if user_is_admin:
|
||||
logger.info(f"Admin session validated for {user_email}")
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=-1,
|
||||
message="Admin access granted (domain)",
|
||||
user_email=user_email,
|
||||
is_admin=True
|
||||
)
|
||||
|
||||
logger.info(f"Session token validated for user {user_email} (credits: {credits})")
|
||||
|
||||
if credits <= 0:
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.STRIPE,
|
||||
remaining_uses=0,
|
||||
message="Authenticated but no credits remaining",
|
||||
user_email=user_email,
|
||||
is_admin=False
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.STRIPE,
|
||||
remaining_uses=credits,
|
||||
message=f"Session valid: {credits} credits",
|
||||
user_email=user_email,
|
||||
is_admin=False
|
||||
)
|
||||
|
||||
# Neither auth_token nor session found
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=UserType.LIMITED,
|
||||
remaining_uses=0,
|
||||
message="Invalid token"
|
||||
)
|
||||
|
||||
# Check if token is active
|
||||
if not token_data.get("active", True):
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=UserType(token_data["type"]),
|
||||
remaining_uses=0,
|
||||
message="Token has been revoked"
|
||||
)
|
||||
|
||||
token_type = UserType(token_data["type"])
|
||||
max_uses = token_data.get("max_uses", -1)
|
||||
# All auth_tokens must have an associated user_email for job ownership
|
||||
token_user_email = token_data.get("user_email")
|
||||
api_key_id = token_data.get("api_key_id")
|
||||
|
||||
# Require user_email on all auth_tokens (no anonymous token auth)
|
||||
if not token_user_email:
|
||||
logger.warning("Auth token missing required user_email field")
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=token_type,
|
||||
remaining_uses=0,
|
||||
message="Token configuration error: missing user_email. Please contact support."
|
||||
)
|
||||
|
||||
# Check if token's user is an admin (by email domain)
|
||||
token_is_admin = is_admin_email(token_user_email)
|
||||
|
||||
# UNLIMITED tokens: no usage limits
|
||||
if token_type == UserType.UNLIMITED:
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=-1,
|
||||
message="Unlimited access granted",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
# LIMITED tokens: check usage count
|
||||
if token_type == UserType.LIMITED:
|
||||
if max_uses <= 0: # -1 means unlimited
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=-1,
|
||||
message="Limited token with unlimited uses",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
current_uses = token_data.get("usage_count", 0)
|
||||
remaining = max_uses - current_uses
|
||||
|
||||
if remaining <= 0:
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=token_type,
|
||||
remaining_uses=0,
|
||||
message="Token usage limit exceeded",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=remaining,
|
||||
message=f"Limited token: {remaining} uses remaining",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
# STRIPE tokens: check expiration and usage
|
||||
if token_type == UserType.STRIPE:
|
||||
expires_at = token_data.get("expires_at")
|
||||
|
||||
if expires_at and time.time() > expires_at:
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=token_type,
|
||||
remaining_uses=0,
|
||||
message="Token has expired",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
if max_uses > 0:
|
||||
current_uses = token_data.get("usage_count", 0)
|
||||
remaining = max_uses - current_uses
|
||||
|
||||
if remaining <= 0:
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=token_type,
|
||||
remaining_uses=0,
|
||||
message="Token usage limit exceeded",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=remaining,
|
||||
message=f"Stripe token: {remaining} uses remaining",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=-1,
|
||||
message="Stripe access granted",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
# API_KEY tokens
|
||||
if token_type == UserType.API_KEY:
|
||||
if max_uses > 0:
|
||||
current_uses = token_data.get("usage_count", 0)
|
||||
remaining = max_uses - current_uses
|
||||
if remaining <= 0:
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=token_type,
|
||||
remaining_uses=0,
|
||||
message="API key usage limit exceeded",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=remaining,
|
||||
message=f"API key valid: {remaining} uses remaining",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=-1,
|
||||
message="API key access granted",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=UserType.LIMITED,
|
||||
remaining_uses=0,
|
||||
message="Unknown token type"
|
||||
)
|
||||
|
||||
def increment_token_usage(self, token: str, job_id: str) -> bool:
|
||||
"""
|
||||
Increment usage count for a token and track the job.
|
||||
|
||||
For session tokens (magic link auth), this deducts a credit from the user.
|
||||
For auth_tokens, this increments the usage count.
|
||||
|
||||
Args:
|
||||
token: The access token
|
||||
job_id: The job ID being created
|
||||
|
||||
Returns:
|
||||
True if usage was tracked, False otherwise
|
||||
"""
|
||||
# Validate token first
|
||||
is_valid, user_type, remaining_uses, message = self.validate_token(token)
|
||||
|
||||
if not is_valid:
|
||||
logger.warning(f"Cannot increment usage for invalid token: {message}")
|
||||
return False
|
||||
|
||||
# Don't track usage for admin or unlimited tokens
|
||||
if user_type in [UserType.ADMIN, UserType.UNLIMITED]:
|
||||
logger.debug(f"Skipping usage tracking for {user_type} token")
|
||||
return True
|
||||
|
||||
# For admin tokens (not in Firestore), no tracking needed
|
||||
if token in self.admin_tokens:
|
||||
return True
|
||||
|
||||
# Check if this is a session token (STRIPE type from validate_token means it's a session)
|
||||
# Session tokens are not in auth_tokens collection
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
if not token_data:
|
||||
# This is a session token - deduct credit from user
|
||||
from backend.services.user_service import get_user_service
|
||||
user_service = get_user_service()
|
||||
|
||||
# Get the user email from the session
|
||||
is_valid, user, _ = user_service.validate_session(token)
|
||||
if not is_valid or not user:
|
||||
logger.error(f"Session token validation failed during usage increment")
|
||||
return False
|
||||
|
||||
# Deduct one credit
|
||||
success, new_balance, deduct_message = user_service.deduct_credit(
|
||||
user.email, job_id, reason="job_creation"
|
||||
)
|
||||
|
||||
if success:
|
||||
logger.info(f"Deducted credit for user {user.email} (remaining: {new_balance})")
|
||||
return True
|
||||
else:
|
||||
logger.error(f"Failed to deduct credit for user {user.email}: {deduct_message}")
|
||||
return False
|
||||
|
||||
# Regular auth_token - increment usage in Firestore
|
||||
try:
|
||||
self.firestore.increment_token_usage(token, job_id)
|
||||
logger.info(f"Incremented usage for token (remaining: {remaining_uses - 1})")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to increment token usage: {e}")
|
||||
return False
|
||||
|
||||
def create_token(
|
||||
self,
|
||||
token_value: str,
|
||||
token_type: UserType,
|
||||
max_uses: int = -1,
|
||||
expires_at: Optional[float] = None,
|
||||
created_by: Optional[str] = None
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Create a new access token.
|
||||
|
||||
Args:
|
||||
token_value: The actual token string (should be secure random string)
|
||||
token_type: Type of access (admin, unlimited, limited, stripe)
|
||||
max_uses: Maximum number of uses (-1 = unlimited)
|
||||
expires_at: Expiration timestamp (None = no expiration)
|
||||
created_by: Admin token that created this token
|
||||
|
||||
Returns:
|
||||
Token data dictionary
|
||||
"""
|
||||
token_data = {
|
||||
"token": token_value,
|
||||
"type": token_type.value,
|
||||
"max_uses": max_uses,
|
||||
"usage_count": 0,
|
||||
"active": True,
|
||||
"created_at": time.time(),
|
||||
"created_by": created_by,
|
||||
"expires_at": expires_at,
|
||||
"jobs": []
|
||||
}
|
||||
|
||||
self.firestore.create_token(token_value, token_data)
|
||||
logger.info(f"Created new {token_type} token with max_uses={max_uses}")
|
||||
|
||||
return token_data
|
||||
|
||||
def revoke_token(self, token: str, revoked_by: Optional[str] = None) -> bool:
|
||||
"""
|
||||
Revoke an access token.
|
||||
|
||||
Args:
|
||||
token: The token to revoke
|
||||
revoked_by: Admin token that revoked this token
|
||||
|
||||
Returns:
|
||||
True if revoked, False if token not found
|
||||
"""
|
||||
try:
|
||||
self.firestore.update_token(token, {
|
||||
"active": False,
|
||||
"revoked_at": time.time(),
|
||||
"revoked_by": revoked_by
|
||||
})
|
||||
logger.info(f"Revoked token")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to revoke token: {e}")
|
||||
return False
|
||||
|
||||
def list_tokens(self, include_inactive: bool = False) -> list:
|
||||
"""
|
||||
List all tokens (admin only).
|
||||
|
||||
Args:
|
||||
include_inactive: Whether to include revoked tokens
|
||||
|
||||
Returns:
|
||||
List of token data dictionaries (with sensitive data masked)
|
||||
"""
|
||||
try:
|
||||
tokens = self.firestore.list_tokens()
|
||||
|
||||
# Filter inactive if requested
|
||||
if not include_inactive:
|
||||
tokens = [t for t in tokens if t.get("active", True)]
|
||||
|
||||
# Mask token values for security (show only first 8 chars)
|
||||
for token in tokens:
|
||||
if "token" in token and len(token["token"]) > 8:
|
||||
token["token"] = token["token"][:8] + "..."
|
||||
|
||||
return tokens
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to list tokens: {e}")
|
||||
return []
|
||||
|
||||
def get_token_info(self, token: str) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Get information about a token (for admin dashboard).
|
||||
|
||||
Returns:
|
||||
Token data or None if not found
|
||||
"""
|
||||
# Check if it's an admin token (not stored in Firestore)
|
||||
if token in self.admin_tokens:
|
||||
return {
|
||||
"token": token[:8] + "...",
|
||||
"type": UserType.ADMIN.value,
|
||||
"max_uses": -1,
|
||||
"usage_count": 0,
|
||||
"active": True,
|
||||
"source": "environment"
|
||||
}
|
||||
|
||||
# Get from Firestore
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
if token_data and "token" in token_data:
|
||||
# Mask the full token value
|
||||
token_data["token"] = token_data["token"][:8] + "..."
|
||||
|
||||
return token_data
|
||||
|
||||
|
||||
# Global instance
|
||||
_auth_service = None
|
||||
|
||||
|
||||
def get_auth_service() -> AuthService:
|
||||
"""Get the global auth service instance."""
|
||||
global _auth_service
|
||||
if _auth_service is None:
|
||||
_auth_service = AuthService()
|
||||
return _auth_service
|
||||
|
||||
@@ -0,0 +1,792 @@
|
||||
"""
|
||||
OAuth Credential Manager for validating and refreshing credentials.
|
||||
|
||||
This service provides:
|
||||
1. Credential validation for YouTube, Dropbox, and Google Drive
|
||||
2. Device Authorization Flow for re-authentication
|
||||
3. Proactive monitoring with alerts when credentials expire
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional, Dict, Any, Tuple
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from datetime import datetime
|
||||
|
||||
from google.oauth2.credentials import Credentials as GoogleCredentials
|
||||
from google.auth.transport.requests import Request as GoogleAuthRequest
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class CredentialStatus(str, Enum):
|
||||
"""Status of OAuth credentials."""
|
||||
VALID = "valid"
|
||||
EXPIRED = "expired" # Token expired but refresh may work
|
||||
INVALID = "invalid" # Refresh failed, re-auth needed
|
||||
NOT_CONFIGURED = "not_configured" # No credentials stored
|
||||
ERROR = "error" # Unknown error during validation
|
||||
|
||||
|
||||
@dataclass
|
||||
class CredentialCheckResult:
|
||||
"""Result of a credential validation check."""
|
||||
service: str
|
||||
status: CredentialStatus
|
||||
message: str
|
||||
last_checked: datetime
|
||||
expires_at: Optional[datetime] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class DeviceAuthInfo:
|
||||
"""Information for device authorization flow."""
|
||||
device_code: str
|
||||
user_code: str
|
||||
verification_url: str
|
||||
expires_in: int
|
||||
interval: int
|
||||
started_at: datetime
|
||||
|
||||
|
||||
class CredentialManager:
|
||||
"""
|
||||
Manages OAuth credentials for all external services.
|
||||
|
||||
Provides validation, refresh, and device authorization flow for:
|
||||
- YouTube (Google OAuth)
|
||||
- Google Drive (Google OAuth)
|
||||
- Dropbox
|
||||
"""
|
||||
|
||||
# Secret names in Secret Manager
|
||||
YOUTUBE_SECRET = "youtube-oauth-credentials"
|
||||
GDRIVE_SECRET = "gdrive-oauth-credentials"
|
||||
DROPBOX_SECRET = "dropbox-oauth-credentials"
|
||||
|
||||
# OAuth client credentials (for device auth flow)
|
||||
YOUTUBE_CLIENT_SECRET = "youtube-client-credentials"
|
||||
GDRIVE_CLIENT_SECRET = "gdrive-client-credentials"
|
||||
|
||||
# Google OAuth endpoints
|
||||
GOOGLE_TOKEN_URI = "https://oauth2.googleapis.com/token"
|
||||
GOOGLE_DEVICE_AUTH_URI = "https://oauth2.googleapis.com/device/code"
|
||||
|
||||
# Dropbox OAuth endpoints
|
||||
DROPBOX_TOKEN_URI = "https://api.dropboxapi.com/oauth2/token"
|
||||
|
||||
# Scopes for each service
|
||||
YOUTUBE_SCOPES = ["https://www.googleapis.com/auth/youtube"]
|
||||
GDRIVE_SCOPES = ["https://www.googleapis.com/auth/drive.file"]
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
self._pending_device_auths: Dict[str, DeviceAuthInfo] = {}
|
||||
|
||||
def _get_client_credentials(self, secret_name: str) -> Optional[Dict[str, str]]:
|
||||
"""
|
||||
Load OAuth client credentials from Secret Manager.
|
||||
|
||||
The secret should contain:
|
||||
{
|
||||
"client_id": "...",
|
||||
"client_secret": "..."
|
||||
}
|
||||
"""
|
||||
try:
|
||||
creds_json = self.settings.get_secret(secret_name)
|
||||
if not creds_json:
|
||||
return None
|
||||
|
||||
creds = json.loads(creds_json)
|
||||
if creds.get("client_id") and creds.get("client_secret"):
|
||||
return creds
|
||||
|
||||
logger.warning(f"{secret_name} missing client_id or client_secret")
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load {secret_name}: {e}")
|
||||
return None
|
||||
|
||||
def get_youtube_client_credentials(self) -> Optional[Dict[str, str]]:
|
||||
"""Get YouTube OAuth client credentials from Secret Manager."""
|
||||
return self._get_client_credentials(self.YOUTUBE_CLIENT_SECRET)
|
||||
|
||||
def get_gdrive_client_credentials(self) -> Optional[Dict[str, str]]:
|
||||
"""Get Google Drive OAuth client credentials from Secret Manager."""
|
||||
return self._get_client_credentials(self.GDRIVE_CLIENT_SECRET)
|
||||
|
||||
# =========================================================================
|
||||
# Credential Validation
|
||||
# =========================================================================
|
||||
|
||||
def check_all_credentials(self) -> Dict[str, CredentialCheckResult]:
|
||||
"""
|
||||
Check validity of all OAuth credentials.
|
||||
|
||||
Returns:
|
||||
Dictionary mapping service name to check result
|
||||
"""
|
||||
results = {}
|
||||
|
||||
results["youtube"] = self.check_youtube_credentials()
|
||||
results["gdrive"] = self.check_gdrive_credentials()
|
||||
results["dropbox"] = self.check_dropbox_credentials()
|
||||
|
||||
return results
|
||||
|
||||
def check_youtube_credentials(self) -> CredentialCheckResult:
|
||||
"""Check if YouTube credentials are valid and can be refreshed."""
|
||||
return self._check_google_credentials(
|
||||
secret_name=self.YOUTUBE_SECRET,
|
||||
service_name="youtube",
|
||||
scopes=self.YOUTUBE_SCOPES,
|
||||
test_api_call=self._test_youtube_api
|
||||
)
|
||||
|
||||
def check_gdrive_credentials(self) -> CredentialCheckResult:
|
||||
"""Check if Google Drive credentials are valid and can be refreshed."""
|
||||
return self._check_google_credentials(
|
||||
secret_name=self.GDRIVE_SECRET,
|
||||
service_name="gdrive",
|
||||
scopes=self.GDRIVE_SCOPES,
|
||||
test_api_call=self._test_gdrive_api
|
||||
)
|
||||
|
||||
def check_dropbox_credentials(self) -> CredentialCheckResult:
|
||||
"""Check if Dropbox credentials are valid."""
|
||||
try:
|
||||
creds_json = self.settings.get_secret(self.DROPBOX_SECRET)
|
||||
|
||||
if not creds_json:
|
||||
return CredentialCheckResult(
|
||||
service="dropbox",
|
||||
status=CredentialStatus.NOT_CONFIGURED,
|
||||
message="Dropbox credentials not configured",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
creds = json.loads(creds_json)
|
||||
|
||||
if not creds.get("access_token"):
|
||||
return CredentialCheckResult(
|
||||
service="dropbox",
|
||||
status=CredentialStatus.INVALID,
|
||||
message="Dropbox credentials missing access_token",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
# Test the credentials with a simple API call
|
||||
if self._test_dropbox_api(creds):
|
||||
return CredentialCheckResult(
|
||||
service="dropbox",
|
||||
status=CredentialStatus.VALID,
|
||||
message="Dropbox credentials are valid",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
else:
|
||||
# Try to refresh if we have refresh token
|
||||
if creds.get("refresh_token") and creds.get("app_key") and creds.get("app_secret"):
|
||||
if self._refresh_dropbox_token(creds):
|
||||
return CredentialCheckResult(
|
||||
service="dropbox",
|
||||
status=CredentialStatus.VALID,
|
||||
message="Dropbox credentials refreshed successfully",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
return CredentialCheckResult(
|
||||
service="dropbox",
|
||||
status=CredentialStatus.INVALID,
|
||||
message="Dropbox credentials invalid and refresh failed",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error checking Dropbox credentials: {e}")
|
||||
return CredentialCheckResult(
|
||||
service="dropbox",
|
||||
status=CredentialStatus.ERROR,
|
||||
message=f"Error checking credentials: {str(e)}",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
def _check_google_credentials(
|
||||
self,
|
||||
secret_name: str,
|
||||
service_name: str,
|
||||
scopes: list,
|
||||
test_api_call: callable
|
||||
) -> CredentialCheckResult:
|
||||
"""Generic Google OAuth credential check."""
|
||||
try:
|
||||
creds_json = self.settings.get_secret(secret_name)
|
||||
|
||||
if not creds_json:
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.NOT_CONFIGURED,
|
||||
message=f"{service_name} credentials not configured",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
creds_data = json.loads(creds_json)
|
||||
|
||||
# Check for required fields
|
||||
required = ["refresh_token", "client_id", "client_secret"]
|
||||
missing = [f for f in required if not creds_data.get(f)]
|
||||
if missing:
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.INVALID,
|
||||
message=f"Missing required fields: {missing}",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
# Create credentials object
|
||||
credentials = GoogleCredentials(
|
||||
token=creds_data.get("token"),
|
||||
refresh_token=creds_data.get("refresh_token"),
|
||||
token_uri=creds_data.get("token_uri", self.GOOGLE_TOKEN_URI),
|
||||
client_id=creds_data.get("client_id"),
|
||||
client_secret=creds_data.get("client_secret"),
|
||||
scopes=creds_data.get("scopes", scopes)
|
||||
)
|
||||
|
||||
# Try to refresh if expired
|
||||
if credentials.expired or not credentials.token:
|
||||
try:
|
||||
credentials.refresh(GoogleAuthRequest())
|
||||
# Update stored credentials with new token
|
||||
self._update_google_credentials(secret_name, creds_data, credentials)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to refresh {service_name} credentials: {e}")
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.INVALID,
|
||||
message=f"Token refresh failed: {str(e)}",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
# Test with API call
|
||||
if test_api_call(credentials):
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.VALID,
|
||||
message=f"{service_name} credentials are valid",
|
||||
last_checked=datetime.utcnow(),
|
||||
expires_at=credentials.expiry
|
||||
)
|
||||
else:
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.INVALID,
|
||||
message=f"{service_name} API test failed",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
except json.JSONDecodeError as e:
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.INVALID,
|
||||
message=f"Invalid JSON in credentials: {str(e)}",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Error checking {service_name} credentials: {e}")
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.ERROR,
|
||||
message=f"Error: {str(e)}",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
def _update_google_credentials(
|
||||
self,
|
||||
secret_name: str,
|
||||
original_data: dict,
|
||||
credentials: GoogleCredentials
|
||||
) -> bool:
|
||||
"""Update stored Google credentials with refreshed token."""
|
||||
try:
|
||||
from google.cloud import secretmanager
|
||||
|
||||
updated_data = original_data.copy()
|
||||
updated_data["token"] = credentials.token
|
||||
if credentials.expiry:
|
||||
updated_data["expiry"] = credentials.expiry.isoformat()
|
||||
|
||||
# Add new secret version
|
||||
client = secretmanager.SecretManagerServiceClient()
|
||||
project_id = self.settings.google_cloud_project
|
||||
secret_path = f"projects/{project_id}/secrets/{secret_name}"
|
||||
|
||||
client.add_secret_version(
|
||||
parent=secret_path,
|
||||
payload={"data": json.dumps(updated_data).encode("utf-8")}
|
||||
)
|
||||
|
||||
logger.info(f"Updated {secret_name} with refreshed token")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to update {secret_name}: {e}")
|
||||
return False
|
||||
|
||||
# =========================================================================
|
||||
# API Test Methods
|
||||
# =========================================================================
|
||||
|
||||
def _test_youtube_api(self, credentials: GoogleCredentials) -> bool:
|
||||
"""Test YouTube credentials with a simple API call."""
|
||||
try:
|
||||
from googleapiclient.discovery import build
|
||||
|
||||
youtube = build("youtube", "v3", credentials=credentials)
|
||||
# Just get channel info - minimal API call
|
||||
request = youtube.channels().list(part="id", mine=True)
|
||||
response = request.execute()
|
||||
|
||||
return "items" in response
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"YouTube API test failed: {e}")
|
||||
return False
|
||||
|
||||
def _test_gdrive_api(self, credentials: GoogleCredentials) -> bool:
|
||||
"""Test Google Drive credentials with a simple API call."""
|
||||
try:
|
||||
from googleapiclient.discovery import build
|
||||
|
||||
drive = build("drive", "v3", credentials=credentials)
|
||||
# Just get about info - minimal API call
|
||||
request = drive.about().get(fields="user")
|
||||
response = request.execute()
|
||||
|
||||
return "user" in response
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Google Drive API test failed: {e}")
|
||||
return False
|
||||
|
||||
def _test_dropbox_api(self, creds: dict) -> bool:
|
||||
"""Test Dropbox credentials with a simple API call."""
|
||||
try:
|
||||
import dropbox
|
||||
|
||||
dbx = dropbox.Dropbox(creds["access_token"])
|
||||
# Just get account info - minimal API call
|
||||
dbx.users_get_current_account()
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Dropbox API test failed: {e}")
|
||||
return False
|
||||
|
||||
def _refresh_dropbox_token(self, creds: dict) -> bool:
|
||||
"""Try to refresh Dropbox token."""
|
||||
try:
|
||||
import requests
|
||||
|
||||
response = requests.post(
|
||||
self.DROPBOX_TOKEN_URI,
|
||||
data={
|
||||
"grant_type": "refresh_token",
|
||||
"refresh_token": creds["refresh_token"],
|
||||
"client_id": creds["app_key"],
|
||||
"client_secret": creds["app_secret"],
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
token_data = response.json()
|
||||
creds["access_token"] = token_data["access_token"]
|
||||
|
||||
# Update in Secret Manager
|
||||
self._update_dropbox_credentials(creds)
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Dropbox token refresh failed: {e}")
|
||||
return False
|
||||
|
||||
def _update_dropbox_credentials(self, creds: dict) -> bool:
|
||||
"""Update stored Dropbox credentials."""
|
||||
try:
|
||||
from google.cloud import secretmanager
|
||||
|
||||
client = secretmanager.SecretManagerServiceClient()
|
||||
project_id = self.settings.google_cloud_project
|
||||
secret_path = f"projects/{project_id}/secrets/{self.DROPBOX_SECRET}"
|
||||
|
||||
client.add_secret_version(
|
||||
parent=secret_path,
|
||||
payload={"data": json.dumps(creds).encode("utf-8")}
|
||||
)
|
||||
|
||||
logger.info("Updated Dropbox credentials with refreshed token")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to update Dropbox credentials: {e}")
|
||||
return False
|
||||
|
||||
# =========================================================================
|
||||
# Device Authorization Flow
|
||||
# =========================================================================
|
||||
|
||||
def start_youtube_device_auth(
|
||||
self,
|
||||
client_id: Optional[str] = None,
|
||||
client_secret: Optional[str] = None
|
||||
) -> DeviceAuthInfo:
|
||||
"""
|
||||
Start YouTube device authorization flow.
|
||||
|
||||
Args:
|
||||
client_id: Google OAuth client ID (optional, reads from Secret Manager if not provided)
|
||||
client_secret: Google OAuth client secret (optional, reads from Secret Manager if not provided)
|
||||
|
||||
Returns:
|
||||
DeviceAuthInfo with user code and verification URL
|
||||
"""
|
||||
# Load from Secret Manager if not provided
|
||||
if not client_id or not client_secret:
|
||||
stored_creds = self.get_youtube_client_credentials()
|
||||
if not stored_creds:
|
||||
raise Exception(
|
||||
"YouTube client credentials not found. Either pass client_id/client_secret "
|
||||
"or create the 'youtube-client-credentials' secret in Secret Manager."
|
||||
)
|
||||
client_id = stored_creds["client_id"]
|
||||
client_secret = stored_creds["client_secret"]
|
||||
|
||||
return self._start_google_device_auth(
|
||||
client_id=client_id,
|
||||
client_secret=client_secret,
|
||||
scopes=self.YOUTUBE_SCOPES,
|
||||
service_name="youtube"
|
||||
)
|
||||
|
||||
def start_gdrive_device_auth(
|
||||
self,
|
||||
client_id: Optional[str] = None,
|
||||
client_secret: Optional[str] = None
|
||||
) -> DeviceAuthInfo:
|
||||
"""
|
||||
Start Google Drive device authorization flow.
|
||||
|
||||
Args:
|
||||
client_id: Google OAuth client ID (optional, reads from Secret Manager if not provided)
|
||||
client_secret: Google OAuth client secret (optional, reads from Secret Manager if not provided)
|
||||
|
||||
Returns:
|
||||
DeviceAuthInfo with user code and verification URL
|
||||
"""
|
||||
# Load from Secret Manager if not provided
|
||||
if not client_id or not client_secret:
|
||||
stored_creds = self.get_gdrive_client_credentials()
|
||||
if not stored_creds:
|
||||
raise Exception(
|
||||
"Google Drive client credentials not found. Either pass client_id/client_secret "
|
||||
"or create the 'gdrive-client-credentials' secret in Secret Manager."
|
||||
)
|
||||
client_id = stored_creds["client_id"]
|
||||
client_secret = stored_creds["client_secret"]
|
||||
|
||||
return self._start_google_device_auth(
|
||||
client_id=client_id,
|
||||
client_secret=client_secret,
|
||||
scopes=self.GDRIVE_SCOPES,
|
||||
service_name="gdrive"
|
||||
)
|
||||
|
||||
def _start_google_device_auth(
|
||||
self,
|
||||
client_id: str,
|
||||
client_secret: str,
|
||||
scopes: list,
|
||||
service_name: str
|
||||
) -> DeviceAuthInfo:
|
||||
"""Start Google device authorization flow."""
|
||||
import requests
|
||||
|
||||
logger.info(f"[{service_name}] Starting device auth flow with scopes: {scopes}")
|
||||
|
||||
response = requests.post(
|
||||
self.GOOGLE_DEVICE_AUTH_URI,
|
||||
data={
|
||||
"client_id": client_id,
|
||||
"scope": " ".join(scopes),
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
if response.status_code != 200:
|
||||
logger.error(f"[{service_name}] Device auth request failed: {response.status_code} - {response.text}")
|
||||
raise Exception(f"Device auth request failed: {response.text}")
|
||||
|
||||
logger.info(f"[{service_name}] Device auth initiated successfully")
|
||||
|
||||
data = response.json()
|
||||
|
||||
# Google uses 'verification_uri' but some docs show 'verification_url'
|
||||
verification_url = data.get("verification_uri") or data.get("verification_url")
|
||||
|
||||
device_info = DeviceAuthInfo(
|
||||
device_code=data["device_code"],
|
||||
user_code=data["user_code"],
|
||||
verification_url=verification_url,
|
||||
expires_in=data["expires_in"],
|
||||
interval=data.get("interval", 5),
|
||||
started_at=datetime.utcnow()
|
||||
)
|
||||
|
||||
# Store for polling, include client secret for token exchange
|
||||
self._pending_device_auths[f"{service_name}:{data['device_code']}"] = {
|
||||
"info": device_info,
|
||||
"client_id": client_id,
|
||||
"client_secret": client_secret,
|
||||
"scopes": scopes,
|
||||
"service_name": service_name
|
||||
}
|
||||
|
||||
return device_info
|
||||
|
||||
def poll_device_auth(self, service_name: str, device_code: str) -> Tuple[str, Optional[dict]]:
|
||||
"""
|
||||
Poll for device authorization completion.
|
||||
|
||||
This method is STATELESS - it fetches client credentials from Secret Manager
|
||||
and polls Google directly. This works correctly in serverless environments
|
||||
like Cloud Run where in-memory state is not preserved between requests.
|
||||
|
||||
Args:
|
||||
service_name: "youtube" or "gdrive"
|
||||
device_code: The device code from start_*_device_auth
|
||||
|
||||
Returns:
|
||||
Tuple of (status, token_data)
|
||||
status: "pending", "complete", "expired", "error"
|
||||
token_data: Token data if complete, None otherwise
|
||||
"""
|
||||
import requests
|
||||
|
||||
logger.info(f"[{service_name}] Polling device auth for code: {device_code[:20]}...")
|
||||
|
||||
# Get client credentials from Secret Manager (stateless approach)
|
||||
if service_name == "youtube":
|
||||
client_creds = self.get_youtube_client_credentials()
|
||||
scopes = self.YOUTUBE_SCOPES
|
||||
secret_name = self.YOUTUBE_SECRET
|
||||
elif service_name == "gdrive":
|
||||
client_creds = self.get_gdrive_client_credentials()
|
||||
scopes = self.GDRIVE_SCOPES
|
||||
secret_name = self.GDRIVE_SECRET
|
||||
else:
|
||||
logger.error(f"[{service_name}] Unknown service")
|
||||
return ("error", {"message": f"Unknown service: {service_name}"})
|
||||
|
||||
if not client_creds:
|
||||
logger.error(f"[{service_name}] Client credentials not found in Secret Manager")
|
||||
return ("error", {"message": f"Client credentials not found in Secret Manager for {service_name}"})
|
||||
|
||||
client_id = client_creds.get("client_id")
|
||||
client_secret = client_creds.get("client_secret")
|
||||
|
||||
if not client_id or not client_secret:
|
||||
logger.error(f"[{service_name}] Invalid client credentials (missing client_id or client_secret)")
|
||||
return ("error", {"message": f"Invalid client credentials for {service_name}"})
|
||||
|
||||
logger.info(f"[{service_name}] Got client credentials, polling Google token endpoint...")
|
||||
|
||||
# Poll Google token endpoint
|
||||
response = requests.post(
|
||||
self.GOOGLE_TOKEN_URI,
|
||||
data={
|
||||
"client_id": client_id,
|
||||
"client_secret": client_secret,
|
||||
"device_code": device_code,
|
||||
"grant_type": "urn:ietf:params:oauth:grant-type:device_code",
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
data = response.json()
|
||||
logger.info(f"[{service_name}] Token endpoint response: status={response.status_code}")
|
||||
|
||||
if response.status_code == 200:
|
||||
logger.info(f"[{service_name}] Token exchange successful! Got access token and refresh_token={bool(data.get('refresh_token'))}")
|
||||
|
||||
# Success! Store the credentials
|
||||
token_data = {
|
||||
"token": data["access_token"],
|
||||
"refresh_token": data.get("refresh_token"),
|
||||
"token_uri": self.GOOGLE_TOKEN_URI,
|
||||
"client_id": client_id,
|
||||
"client_secret": client_secret,
|
||||
"scopes": scopes,
|
||||
}
|
||||
|
||||
# Save to Secret Manager
|
||||
logger.info(f"[{service_name}] Saving credentials to secret: {secret_name}")
|
||||
saved = self._save_credentials_to_secret(secret_name, token_data)
|
||||
if not saved:
|
||||
logger.error(f"[{service_name}] Failed to save credentials to Secret Manager!")
|
||||
return ("error", {"message": "Token exchange succeeded but failed to save credentials to Secret Manager"})
|
||||
|
||||
logger.info(f"[{service_name}] Device auth flow COMPLETE - credentials saved successfully")
|
||||
return ("complete", token_data)
|
||||
|
||||
elif "error" in data:
|
||||
error = data["error"]
|
||||
logger.info(f"[{service_name}] Token endpoint returned error: {error}")
|
||||
|
||||
if error == "authorization_pending":
|
||||
return ("pending", {"message": "Waiting for user to authorize. Please visit the verification URL and enter the code."})
|
||||
elif error == "slow_down":
|
||||
return ("pending", {"message": "Please wait a few more seconds before polling again."})
|
||||
elif error == "expired_token":
|
||||
logger.warning(f"[{service_name}] Device code expired")
|
||||
return ("expired", {"message": "Device code expired. Please start a new device auth flow."})
|
||||
elif error == "access_denied":
|
||||
logger.warning(f"[{service_name}] User denied access")
|
||||
return ("error", {"message": "User denied access"})
|
||||
else:
|
||||
logger.error(f"[{service_name}] Unexpected error: {data}")
|
||||
return ("error", {"message": data.get("error_description", error)})
|
||||
|
||||
logger.error(f"[{service_name}] Unknown response from token endpoint: {data}")
|
||||
return ("error", {"message": "Unknown response from token endpoint"})
|
||||
|
||||
def _save_credentials_to_secret(self, secret_name: str, token_data: dict) -> bool:
|
||||
"""Save credentials to Secret Manager, creating the secret if needed."""
|
||||
logger.info(f"[SecretManager] Attempting to save credentials to: {secret_name}")
|
||||
|
||||
try:
|
||||
from google.cloud import secretmanager
|
||||
from google.api_core import exceptions as gcp_exceptions
|
||||
|
||||
client = secretmanager.SecretManagerServiceClient()
|
||||
project_id = self.settings.google_cloud_project
|
||||
|
||||
if not project_id:
|
||||
logger.error(f"[SecretManager] No GCP project configured!")
|
||||
return False
|
||||
|
||||
parent = f"projects/{project_id}"
|
||||
secret_path = f"{parent}/secrets/{secret_name}"
|
||||
|
||||
logger.info(f"[SecretManager] Using project: {project_id}, secret path: {secret_path}")
|
||||
|
||||
# Try to add a version to existing secret
|
||||
try:
|
||||
client.add_secret_version(
|
||||
parent=secret_path,
|
||||
payload={"data": json.dumps(token_data).encode("utf-8")}
|
||||
)
|
||||
logger.info(f"[SecretManager] SUCCESS - Added new version to existing secret: {secret_name}")
|
||||
return True
|
||||
|
||||
except gcp_exceptions.NotFound:
|
||||
# Secret doesn't exist, create it first
|
||||
logger.info(f"[SecretManager] Secret {secret_name} not found, creating new secret...")
|
||||
client.create_secret(
|
||||
parent=parent,
|
||||
secret_id=secret_name,
|
||||
secret={"replication": {"automatic": {}}}
|
||||
)
|
||||
logger.info(f"[SecretManager] Created secret: {secret_name}")
|
||||
|
||||
# Now add the version
|
||||
client.add_secret_version(
|
||||
parent=secret_path,
|
||||
payload={"data": json.dumps(token_data).encode("utf-8")}
|
||||
)
|
||||
logger.info(f"[SecretManager] SUCCESS - Created secret and saved credentials to: {secret_name}")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"[SecretManager] FAILED to save credentials to {secret_name}: {type(e).__name__}: {e}")
|
||||
import traceback
|
||||
logger.error(f"[SecretManager] Traceback: {traceback.format_exc()}")
|
||||
return False
|
||||
|
||||
# =========================================================================
|
||||
# Alerts
|
||||
# =========================================================================
|
||||
|
||||
def send_credential_alert(
|
||||
self,
|
||||
invalid_services: list[CredentialCheckResult],
|
||||
discord_webhook_url: Optional[str] = None
|
||||
) -> bool:
|
||||
"""
|
||||
Send alert about invalid credentials.
|
||||
|
||||
Args:
|
||||
invalid_services: List of services with invalid credentials
|
||||
discord_webhook_url: Discord webhook URL for notifications
|
||||
|
||||
Returns:
|
||||
True if alert was sent successfully
|
||||
"""
|
||||
if not discord_webhook_url:
|
||||
logger.warning("No Discord webhook URL configured for alerts")
|
||||
return False
|
||||
|
||||
try:
|
||||
import requests
|
||||
|
||||
services_list = "\n".join([
|
||||
f"• **{r.service}**: {r.message}"
|
||||
for r in invalid_services
|
||||
])
|
||||
|
||||
# Get the API base URL for re-auth links
|
||||
api_url = self.settings.api_base_url if hasattr(self.settings, 'api_base_url') else "https://your-api-url"
|
||||
|
||||
message = {
|
||||
"embeds": [{
|
||||
"title": "⚠️ OAuth Credentials Need Attention",
|
||||
"description": f"The following service credentials need re-authorization:\n\n{services_list}",
|
||||
"color": 16744256, # Orange
|
||||
"fields": [
|
||||
{
|
||||
"name": "Re-authorize",
|
||||
"value": f"Visit `{api_url}/api/auth/status` to start re-authorization flow",
|
||||
"inline": False
|
||||
}
|
||||
],
|
||||
"footer": {
|
||||
"text": "karaoke-gen backend"
|
||||
},
|
||||
"timestamp": datetime.utcnow().isoformat()
|
||||
}]
|
||||
}
|
||||
|
||||
response = requests.post(discord_webhook_url, json=message, timeout=30)
|
||||
return response.status_code in (200, 204)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to send credential alert: {e}")
|
||||
return False
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_credential_manager: Optional[CredentialManager] = None
|
||||
|
||||
|
||||
def get_credential_manager() -> CredentialManager:
|
||||
"""Get the singleton credential manager instance."""
|
||||
global _credential_manager
|
||||
if _credential_manager is None:
|
||||
_credential_manager = CredentialManager()
|
||||
return _credential_manager
|
||||
@@ -0,0 +1,172 @@
|
||||
"""
|
||||
Discord Notification Service.
|
||||
|
||||
Provides Discord webhook notification functionality, extracted from KaraokeFinalise
|
||||
for use by both the cloud backend (video_worker) and local CLI.
|
||||
|
||||
This service handles:
|
||||
- Posting messages to Discord webhooks
|
||||
- Video upload notifications
|
||||
- Validation of webhook URLs
|
||||
"""
|
||||
|
||||
import logging
|
||||
import re
|
||||
from typing import Optional
|
||||
|
||||
import requests
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class DiscordNotificationService:
|
||||
"""
|
||||
Service for sending Discord webhook notifications.
|
||||
|
||||
Supports posting messages to Discord channels via webhooks,
|
||||
commonly used to notify when new karaoke videos are uploaded.
|
||||
"""
|
||||
|
||||
# Discord webhook URL pattern
|
||||
WEBHOOK_URL_PATTERN = re.compile(
|
||||
r"^https://discord\.com/api/webhooks/\d+/[a-zA-Z0-9_-]+$"
|
||||
)
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
webhook_url: Optional[str] = None,
|
||||
dry_run: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the Discord notification service.
|
||||
|
||||
Args:
|
||||
webhook_url: Discord webhook URL for posting notifications
|
||||
dry_run: If True, log actions without performing them
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.webhook_url = webhook_url
|
||||
self.dry_run = dry_run
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
# Validate webhook URL if provided
|
||||
if webhook_url:
|
||||
self._validate_webhook_url(webhook_url)
|
||||
|
||||
def _validate_webhook_url(self, url: str) -> None:
|
||||
"""
|
||||
Validate that the webhook URL is a valid Discord webhook URL.
|
||||
|
||||
Args:
|
||||
url: URL to validate
|
||||
|
||||
Raises:
|
||||
ValueError: If URL is not a valid Discord webhook URL
|
||||
"""
|
||||
url = url.strip()
|
||||
if not url.startswith("https://discord.com/api/webhooks/"):
|
||||
raise ValueError(
|
||||
f"Invalid Discord webhook URL: {url}. "
|
||||
"URL must start with 'https://discord.com/api/webhooks/'"
|
||||
)
|
||||
|
||||
def post_message(
|
||||
self,
|
||||
message: str,
|
||||
webhook_url: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Post a message to a Discord channel via webhook.
|
||||
|
||||
Args:
|
||||
message: The message content to post
|
||||
webhook_url: Optional webhook URL (uses instance URL if not provided)
|
||||
|
||||
Returns:
|
||||
True if message was posted successfully, False otherwise
|
||||
|
||||
Raises:
|
||||
ValueError: If no webhook URL is available
|
||||
requests.HTTPError: If the webhook request fails
|
||||
"""
|
||||
url = webhook_url or self.webhook_url
|
||||
if not url:
|
||||
raise ValueError("No Discord webhook URL provided")
|
||||
|
||||
url = url.strip()
|
||||
self._validate_webhook_url(url)
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(
|
||||
f"DRY RUN: Would post Discord message: '{message}' to {url}"
|
||||
)
|
||||
return True
|
||||
|
||||
self.logger.info(f"Posting message to Discord webhook...")
|
||||
data = {"content": message}
|
||||
response = requests.post(url, json=data, timeout=30)
|
||||
response.raise_for_status()
|
||||
self.logger.info("Message posted to Discord successfully")
|
||||
return True
|
||||
|
||||
def post_video_notification(
|
||||
self,
|
||||
youtube_url: str,
|
||||
webhook_url: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Post a notification about a new video upload.
|
||||
|
||||
Args:
|
||||
youtube_url: The YouTube URL of the uploaded video
|
||||
webhook_url: Optional webhook URL (uses instance URL if not provided)
|
||||
|
||||
Returns:
|
||||
True if notification was posted successfully, False otherwise
|
||||
"""
|
||||
if not youtube_url:
|
||||
self.logger.info("Skipping Discord notification - no YouTube URL available")
|
||||
return False
|
||||
|
||||
message = f"New upload: {youtube_url}"
|
||||
return self.post_message(message, webhook_url)
|
||||
|
||||
def is_enabled(self) -> bool:
|
||||
"""
|
||||
Check if Discord notifications are enabled.
|
||||
|
||||
Returns:
|
||||
True if a valid webhook URL is configured
|
||||
"""
|
||||
return bool(self.webhook_url)
|
||||
|
||||
|
||||
# Singleton instance and factory function (following existing service pattern)
|
||||
_discord_notification_service: Optional[DiscordNotificationService] = None
|
||||
|
||||
|
||||
def get_discord_notification_service(
|
||||
webhook_url: Optional[str] = None,
|
||||
**kwargs
|
||||
) -> DiscordNotificationService:
|
||||
"""
|
||||
Get a Discord notification service instance.
|
||||
|
||||
Args:
|
||||
webhook_url: Discord webhook URL for notifications
|
||||
**kwargs: Additional arguments passed to DiscordNotificationService
|
||||
|
||||
Returns:
|
||||
DiscordNotificationService instance
|
||||
"""
|
||||
global _discord_notification_service
|
||||
|
||||
# Create new instance if webhook URL changed or not yet created
|
||||
if _discord_notification_service is None or webhook_url:
|
||||
_discord_notification_service = DiscordNotificationService(
|
||||
webhook_url=webhook_url,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
return _discord_notification_service
|
||||
@@ -0,0 +1,301 @@
|
||||
"""Native Dropbox API service for cloud backend.
|
||||
|
||||
This service provides Dropbox operations using the native Python SDK,
|
||||
replacing rclone for server-side operations. It handles:
|
||||
- Folder listing for brand code calculation
|
||||
- File and folder uploads
|
||||
- Shared link generation
|
||||
|
||||
Credentials are loaded from Google Cloud Secret Manager.
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
from typing import Optional
|
||||
|
||||
from google.cloud import secretmanager
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class DropboxService:
|
||||
"""Dropbox operations using native Python SDK."""
|
||||
|
||||
def __init__(self):
|
||||
self._client = None
|
||||
self._is_configured = False
|
||||
|
||||
def _load_credentials(self) -> Optional[dict]:
|
||||
"""Load OAuth credentials from Secret Manager."""
|
||||
try:
|
||||
client = secretmanager.SecretManagerServiceClient()
|
||||
# Try to get the project ID from environment or use default
|
||||
project_id = os.environ.get("GOOGLE_CLOUD_PROJECT", "karaoke-gen")
|
||||
name = f"projects/{project_id}/secrets/dropbox-oauth-credentials/versions/latest"
|
||||
response = client.access_secret_version(request={"name": name})
|
||||
return json.loads(response.payload.data.decode("UTF-8"))
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to load Dropbox credentials from Secret Manager: {e}")
|
||||
return None
|
||||
|
||||
@property
|
||||
def is_configured(self) -> bool:
|
||||
"""Check if Dropbox credentials are available."""
|
||||
if self._is_configured:
|
||||
return True
|
||||
creds = self._load_credentials()
|
||||
self._is_configured = creds is not None and "access_token" in creds
|
||||
return self._is_configured
|
||||
|
||||
@property
|
||||
def client(self):
|
||||
"""Get or create Dropbox client."""
|
||||
if self._client is None:
|
||||
# Import here to avoid import errors if dropbox is not installed
|
||||
try:
|
||||
from dropbox import Dropbox
|
||||
except ImportError:
|
||||
raise ImportError(
|
||||
"dropbox package is not installed. "
|
||||
"Install it with: pip install dropbox"
|
||||
)
|
||||
|
||||
creds = self._load_credentials()
|
||||
if not creds:
|
||||
raise RuntimeError("Dropbox credentials not configured in Secret Manager")
|
||||
|
||||
self._client = Dropbox(
|
||||
oauth2_access_token=creds["access_token"],
|
||||
oauth2_refresh_token=creds.get("refresh_token"),
|
||||
app_key=creds.get("app_key"),
|
||||
app_secret=creds.get("app_secret"),
|
||||
)
|
||||
return self._client
|
||||
|
||||
def list_folders(self, path: str) -> list[str]:
|
||||
"""
|
||||
List folder names at path for brand code calculation.
|
||||
|
||||
Args:
|
||||
path: Dropbox path to list (e.g., "/Karaoke/Tracks-Organized")
|
||||
|
||||
Returns:
|
||||
List of folder names in the path
|
||||
"""
|
||||
from dropbox.files import FolderMetadata
|
||||
|
||||
# Ensure path starts with /
|
||||
if not path.startswith("/"):
|
||||
path = f"/{path}"
|
||||
|
||||
logger.info(f"Listing folders at Dropbox path: {path}")
|
||||
|
||||
result = self.client.files_list_folder(path)
|
||||
folders = []
|
||||
|
||||
# Get all entries (handling pagination)
|
||||
while True:
|
||||
for entry in result.entries:
|
||||
if isinstance(entry, FolderMetadata):
|
||||
folders.append(entry.name)
|
||||
|
||||
if not result.has_more:
|
||||
break
|
||||
result = self.client.files_list_folder_continue(result.cursor)
|
||||
|
||||
logger.info(f"Found {len(folders)} folders in {path}")
|
||||
return folders
|
||||
|
||||
def get_next_brand_code(self, path: str, brand_prefix: str) -> str:
|
||||
"""
|
||||
Calculate next brand code from existing folders.
|
||||
|
||||
Scans the folder for existing brand codes like "NOMAD-0001" and
|
||||
returns the next sequential code.
|
||||
|
||||
Args:
|
||||
path: Dropbox path containing organized folders
|
||||
brand_prefix: Brand prefix (e.g., "NOMAD")
|
||||
|
||||
Returns:
|
||||
Next brand code (e.g., "NOMAD-1164")
|
||||
"""
|
||||
folders = self.list_folders(path)
|
||||
pattern = re.compile(rf"^{re.escape(brand_prefix)}-(\d{{4}})")
|
||||
|
||||
max_num = 0
|
||||
for folder in folders:
|
||||
match = pattern.match(folder)
|
||||
if match:
|
||||
num = int(match.group(1))
|
||||
max_num = max(max_num, num)
|
||||
|
||||
next_code = f"{brand_prefix}-{max_num + 1:04d}"
|
||||
logger.info(f"Next brand code for {brand_prefix}: {next_code} (max existing: {max_num})")
|
||||
return next_code
|
||||
|
||||
def upload_file(self, local_path: str, remote_path: str) -> None:
|
||||
"""
|
||||
Upload a single file to Dropbox.
|
||||
|
||||
Args:
|
||||
local_path: Local file path
|
||||
remote_path: Dropbox destination path (must start with /)
|
||||
"""
|
||||
from dropbox.files import WriteMode
|
||||
|
||||
# Ensure remote path starts with /
|
||||
if not remote_path.startswith("/"):
|
||||
remote_path = f"/{remote_path}"
|
||||
|
||||
file_size = os.path.getsize(local_path)
|
||||
logger.info(f"Uploading {local_path} ({file_size / 1024 / 1024:.1f} MB) to {remote_path}")
|
||||
|
||||
# For large files (>150MB), use upload sessions
|
||||
CHUNK_SIZE = 150 * 1024 * 1024 # 150 MB
|
||||
|
||||
with open(local_path, "rb") as f:
|
||||
if file_size <= CHUNK_SIZE:
|
||||
# Simple upload for smaller files
|
||||
self.client.files_upload(f.read(), remote_path, mode=WriteMode.overwrite)
|
||||
else:
|
||||
# Chunked upload for large files
|
||||
self._upload_large_file(f, remote_path, file_size)
|
||||
|
||||
logger.info(f"Successfully uploaded to {remote_path}")
|
||||
|
||||
def _upload_large_file(self, file_obj, remote_path: str, file_size: int) -> None:
|
||||
"""Upload a large file using upload sessions."""
|
||||
from dropbox.files import CommitInfo, UploadSessionCursor, WriteMode
|
||||
|
||||
CHUNK_SIZE = 150 * 1024 * 1024 # 150 MB
|
||||
|
||||
# Start upload session
|
||||
session_start = self.client.files_upload_session_start(file_obj.read(CHUNK_SIZE))
|
||||
cursor = UploadSessionCursor(
|
||||
session_id=session_start.session_id,
|
||||
offset=file_obj.tell(),
|
||||
)
|
||||
commit = CommitInfo(path=remote_path, mode=WriteMode.overwrite)
|
||||
|
||||
# Upload remaining chunks
|
||||
while file_obj.tell() < file_size:
|
||||
remaining = file_size - file_obj.tell()
|
||||
if remaining <= CHUNK_SIZE:
|
||||
# Final chunk
|
||||
self.client.files_upload_session_finish(
|
||||
file_obj.read(CHUNK_SIZE),
|
||||
cursor,
|
||||
commit,
|
||||
)
|
||||
else:
|
||||
# Intermediate chunk
|
||||
self.client.files_upload_session_append_v2(
|
||||
file_obj.read(CHUNK_SIZE),
|
||||
cursor,
|
||||
)
|
||||
cursor.offset = file_obj.tell()
|
||||
|
||||
def upload_folder(self, local_dir: str, remote_path: str) -> None:
|
||||
"""
|
||||
Recursively upload all files and subdirectories to Dropbox folder.
|
||||
|
||||
Args:
|
||||
local_dir: Local directory to upload
|
||||
remote_path: Dropbox destination folder path
|
||||
"""
|
||||
# Ensure remote path starts with /
|
||||
if not remote_path.startswith("/"):
|
||||
remote_path = f"/{remote_path}"
|
||||
|
||||
logger.info(f"Uploading folder {local_dir} to {remote_path}")
|
||||
|
||||
uploaded_count = 0
|
||||
for root, _dirs, files in os.walk(local_dir):
|
||||
# Calculate the relative path from local_dir to current root
|
||||
rel_root = os.path.relpath(root, local_dir)
|
||||
if rel_root == ".":
|
||||
current_remote = remote_path
|
||||
else:
|
||||
current_remote = f"{remote_path}/{rel_root}"
|
||||
|
||||
for filename in files:
|
||||
local_file = os.path.join(root, filename)
|
||||
remote_file = f"{current_remote}/{filename}"
|
||||
self.upload_file(local_file, remote_file)
|
||||
uploaded_count += 1
|
||||
|
||||
logger.info(f"Uploaded {uploaded_count} files to {remote_path}")
|
||||
|
||||
def create_shared_link(self, path: str) -> str:
|
||||
"""
|
||||
Create and return a shared link for the path.
|
||||
|
||||
Args:
|
||||
path: Dropbox path to share
|
||||
|
||||
Returns:
|
||||
Shared link URL
|
||||
"""
|
||||
from dropbox.exceptions import ApiError
|
||||
from dropbox.sharing import SharedLinkSettings
|
||||
|
||||
# Ensure path starts with /
|
||||
if not path.startswith("/"):
|
||||
path = f"/{path}"
|
||||
|
||||
logger.info(f"Creating shared link for: {path}")
|
||||
|
||||
try:
|
||||
# Try to create a new shared link
|
||||
settings = SharedLinkSettings(requested_visibility=None)
|
||||
link = self.client.sharing_create_shared_link_with_settings(path, settings)
|
||||
logger.info(f"Created new shared link: {link.url}")
|
||||
return link.url
|
||||
except ApiError as e:
|
||||
# Link may already exist - try to get existing link
|
||||
if e.error.is_shared_link_already_exists():
|
||||
logger.info("Shared link already exists, retrieving existing link")
|
||||
links = self.client.sharing_list_shared_links(path=path, direct_only=True)
|
||||
if links.links:
|
||||
logger.info(f"Found existing shared link: {links.links[0].url}")
|
||||
return links.links[0].url
|
||||
raise
|
||||
|
||||
def delete_folder(self, path: str) -> bool:
|
||||
"""
|
||||
Delete a folder and all its contents from Dropbox.
|
||||
|
||||
Args:
|
||||
path: Dropbox path to delete (e.g., "/Karaoke/Tracks-Organized/NOMAD-1234 - Artist - Title")
|
||||
|
||||
Returns:
|
||||
True if deleted successfully, False otherwise
|
||||
"""
|
||||
from dropbox.exceptions import ApiError
|
||||
|
||||
# Ensure path starts with /
|
||||
if not path.startswith("/"):
|
||||
path = f"/{path}"
|
||||
|
||||
logger.info(f"Deleting Dropbox folder: {path}")
|
||||
|
||||
try:
|
||||
self.client.files_delete_v2(path)
|
||||
logger.info(f"Successfully deleted: {path}")
|
||||
return True
|
||||
except ApiError as e:
|
||||
if e.error.is_path_lookup() and e.error.get_path_lookup().is_not_found():
|
||||
logger.warning(f"Folder not found (already deleted?): {path}")
|
||||
return True # Consider it success if already gone
|
||||
logger.error(f"Failed to delete Dropbox folder: {e}")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting Dropbox folder: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def get_dropbox_service() -> DropboxService:
|
||||
"""Factory function to get a DropboxService instance."""
|
||||
return DropboxService()
|
||||
@@ -0,0 +1,829 @@
|
||||
"""
|
||||
Email service for sending transactional emails.
|
||||
|
||||
Supports multiple providers:
|
||||
- SendGrid (recommended for production)
|
||||
- Console logging (for development/testing)
|
||||
|
||||
Future providers can be added:
|
||||
- Mailgun
|
||||
- AWS SES
|
||||
- Postmark
|
||||
"""
|
||||
import html
|
||||
import logging
|
||||
import os
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Optional
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class EmailProvider(ABC):
|
||||
"""Abstract base class for email providers."""
|
||||
|
||||
@abstractmethod
|
||||
def send_email(
|
||||
self,
|
||||
to_email: str,
|
||||
subject: str,
|
||||
html_content: str,
|
||||
text_content: Optional[str] = None
|
||||
) -> bool:
|
||||
"""Send an email. Returns True if successful."""
|
||||
pass
|
||||
|
||||
|
||||
class ConsoleEmailProvider(EmailProvider):
|
||||
"""
|
||||
Development email provider that logs to console.
|
||||
|
||||
Useful for local development and testing.
|
||||
"""
|
||||
|
||||
def send_email(
|
||||
self,
|
||||
to_email: str,
|
||||
subject: str,
|
||||
html_content: str,
|
||||
text_content: Optional[str] = None
|
||||
) -> bool:
|
||||
logger.info("=" * 60)
|
||||
logger.info(f"EMAIL TO: {to_email}")
|
||||
logger.info(f"SUBJECT: {subject}")
|
||||
logger.info("-" * 60)
|
||||
logger.info(text_content or html_content)
|
||||
logger.info("=" * 60)
|
||||
return True
|
||||
|
||||
|
||||
class SendGridEmailProvider(EmailProvider):
|
||||
"""
|
||||
SendGrid email provider for production.
|
||||
|
||||
Requires SENDGRID_API_KEY environment variable.
|
||||
"""
|
||||
|
||||
def __init__(self, api_key: str, from_email: str, from_name: str = "Nomad Karaoke"):
|
||||
self.api_key = api_key
|
||||
self.from_email = from_email
|
||||
self.from_name = from_name
|
||||
|
||||
def send_email(
|
||||
self,
|
||||
to_email: str,
|
||||
subject: str,
|
||||
html_content: str,
|
||||
text_content: Optional[str] = None
|
||||
) -> bool:
|
||||
try:
|
||||
# Import here to avoid requiring sendgrid in all environments
|
||||
from sendgrid import SendGridAPIClient
|
||||
from sendgrid.helpers.mail import Mail, Email, To, Content
|
||||
|
||||
sg = SendGridAPIClient(api_key=self.api_key)
|
||||
|
||||
message = Mail(
|
||||
from_email=Email(self.from_email, self.from_name),
|
||||
to_emails=To(to_email),
|
||||
subject=subject,
|
||||
html_content=Content("text/html", html_content)
|
||||
)
|
||||
|
||||
if text_content:
|
||||
message.add_content(Content("text/plain", text_content))
|
||||
|
||||
response = sg.send(message)
|
||||
|
||||
if response.status_code >= 200 and response.status_code < 300:
|
||||
logger.info(f"Email sent to {to_email} via SendGrid")
|
||||
return True
|
||||
else:
|
||||
logger.error(f"SendGrid returned status {response.status_code}")
|
||||
return False
|
||||
|
||||
except Exception:
|
||||
logger.exception("Failed to send email via SendGrid")
|
||||
return False
|
||||
|
||||
|
||||
class EmailService:
|
||||
"""
|
||||
High-level email service for sending transactional emails.
|
||||
|
||||
Automatically selects the appropriate provider based on configuration.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
self.provider = self._get_provider()
|
||||
self.frontend_url = os.getenv("FRONTEND_URL", "https://gen.nomadkaraoke.com")
|
||||
# After consolidation, buy URL is the same as frontend URL
|
||||
self.buy_url = os.getenv("BUY_URL", self.frontend_url)
|
||||
|
||||
def _get_provider(self) -> EmailProvider:
|
||||
"""Get the configured email provider."""
|
||||
sendgrid_api_key = os.getenv("SENDGRID_API_KEY")
|
||||
from_email = os.getenv("EMAIL_FROM", "gen@nomadkaraoke.com")
|
||||
from_name = os.getenv("EMAIL_FROM_NAME", "Nomad Karaoke")
|
||||
|
||||
if sendgrid_api_key:
|
||||
logger.info("Using SendGrid email provider")
|
||||
return SendGridEmailProvider(sendgrid_api_key, from_email, from_name)
|
||||
else:
|
||||
logger.warning("No email provider configured, using console logging")
|
||||
return ConsoleEmailProvider()
|
||||
|
||||
def is_configured(self) -> bool:
|
||||
"""Check if a real email provider is configured (not just console logging)."""
|
||||
return isinstance(self.provider, SendGridEmailProvider)
|
||||
|
||||
def send_magic_link(self, email: str, token: str) -> bool:
|
||||
"""
|
||||
Send a magic link email for authentication.
|
||||
|
||||
Args:
|
||||
email: User's email address
|
||||
token: Magic link token
|
||||
|
||||
Returns:
|
||||
True if email was sent successfully
|
||||
"""
|
||||
magic_link_url = f"{self.frontend_url}/auth/verify?token={token}"
|
||||
|
||||
subject = "Sign in to Nomad Karaoke"
|
||||
|
||||
html_content = f"""
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<style>
|
||||
body {{
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
line-height: 1.6;
|
||||
color: #333;
|
||||
max-width: 600px;
|
||||
margin: 0 auto;
|
||||
padding: 20px;
|
||||
}}
|
||||
.header {{
|
||||
text-align: center;
|
||||
padding: 20px 0;
|
||||
}}
|
||||
.logo {{
|
||||
font-size: 24px;
|
||||
font-weight: bold;
|
||||
color: #3b82f6;
|
||||
}}
|
||||
.button {{
|
||||
display: inline-block;
|
||||
background-color: #3b82f6;
|
||||
color: white;
|
||||
padding: 14px 28px;
|
||||
text-decoration: none;
|
||||
border-radius: 8px;
|
||||
font-weight: 600;
|
||||
margin: 20px 0;
|
||||
}}
|
||||
.button:hover {{
|
||||
background-color: #2563eb;
|
||||
}}
|
||||
.footer {{
|
||||
margin-top: 30px;
|
||||
padding-top: 20px;
|
||||
border-top: 1px solid #eee;
|
||||
font-size: 12px;
|
||||
color: #666;
|
||||
}}
|
||||
.warning {{
|
||||
background-color: #fef3c7;
|
||||
border: 1px solid #fcd34d;
|
||||
border-radius: 4px;
|
||||
padding: 12px;
|
||||
margin: 20px 0;
|
||||
font-size: 14px;
|
||||
}}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="header">
|
||||
<div class="logo">🎤 Nomad Karaoke</div>
|
||||
</div>
|
||||
|
||||
<p>Hi there,</p>
|
||||
|
||||
<p>Click the button below to sign in to Nomad Karaoke:</p>
|
||||
|
||||
<p style="text-align: center;">
|
||||
<a href="{magic_link_url}" class="button">Sign In</a>
|
||||
</p>
|
||||
|
||||
<div class="warning">
|
||||
⏰ This link expires in 15 minutes and can only be used once.
|
||||
</div>
|
||||
|
||||
<p>If the button doesn't work, copy and paste this link into your browser:</p>
|
||||
<p style="word-break: break-all; font-size: 14px; color: #666;">
|
||||
{magic_link_url}
|
||||
</p>
|
||||
|
||||
<p>If you didn't request this email, you can safely ignore it.</p>
|
||||
|
||||
<div class="footer">
|
||||
<p>© {self._get_year()} Nomad Karaoke. All rights reserved.</p>
|
||||
<p>This is an automated message, please do not reply.</p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
text_content = f"""
|
||||
Sign in to Nomad Karaoke
|
||||
========================
|
||||
|
||||
Click this link to sign in:
|
||||
{magic_link_url}
|
||||
|
||||
This link expires in 15 minutes and can only be used once.
|
||||
|
||||
If you didn't request this email, you can safely ignore it.
|
||||
|
||||
---
|
||||
© {self._get_year()} Nomad Karaoke
|
||||
"""
|
||||
|
||||
return self.provider.send_email(email, subject, html_content, text_content)
|
||||
|
||||
def send_credits_added(self, email: str, credits: int, total_credits: int) -> bool:
|
||||
"""
|
||||
Send notification when credits are added to account.
|
||||
|
||||
Args:
|
||||
email: User's email address
|
||||
credits: Number of credits added
|
||||
total_credits: New total credit balance
|
||||
|
||||
Returns:
|
||||
True if email was sent successfully
|
||||
"""
|
||||
subject = f"🎉 {credits} credits added to your Nomad Karaoke account"
|
||||
|
||||
html_content = f"""
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<style>
|
||||
body {{
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
line-height: 1.6;
|
||||
color: #333;
|
||||
max-width: 600px;
|
||||
margin: 0 auto;
|
||||
padding: 20px;
|
||||
}}
|
||||
.header {{
|
||||
text-align: center;
|
||||
padding: 20px 0;
|
||||
}}
|
||||
.logo {{
|
||||
font-size: 24px;
|
||||
font-weight: bold;
|
||||
color: #3b82f6;
|
||||
}}
|
||||
.credits-box {{
|
||||
background-color: #ecfdf5;
|
||||
border: 2px solid #10b981;
|
||||
border-radius: 12px;
|
||||
padding: 24px;
|
||||
text-align: center;
|
||||
margin: 20px 0;
|
||||
}}
|
||||
.credits-number {{
|
||||
font-size: 48px;
|
||||
font-weight: bold;
|
||||
color: #10b981;
|
||||
}}
|
||||
.button {{
|
||||
display: inline-block;
|
||||
background-color: #3b82f6;
|
||||
color: white;
|
||||
padding: 14px 28px;
|
||||
text-decoration: none;
|
||||
border-radius: 8px;
|
||||
font-weight: 600;
|
||||
margin: 20px 0;
|
||||
}}
|
||||
.footer {{
|
||||
margin-top: 30px;
|
||||
padding-top: 20px;
|
||||
border-top: 1px solid #eee;
|
||||
font-size: 12px;
|
||||
color: #666;
|
||||
}}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="header">
|
||||
<div class="logo">🎤 Nomad Karaoke</div>
|
||||
</div>
|
||||
|
||||
<p>Great news! 🎉</p>
|
||||
|
||||
<p><strong>{credits} credits</strong> have been added to your account.</p>
|
||||
|
||||
<div class="credits-box">
|
||||
<div>Your balance:</div>
|
||||
<div class="credits-number">{total_credits}</div>
|
||||
<div>credits</div>
|
||||
</div>
|
||||
|
||||
<p>Each credit lets you create one professional karaoke video with:</p>
|
||||
<ul>
|
||||
<li>AI-powered vocal/instrumental separation</li>
|
||||
<li>Synchronized lyrics with word-level timing</li>
|
||||
<li>4K video output</li>
|
||||
<li>YouTube upload</li>
|
||||
</ul>
|
||||
|
||||
<p style="text-align: center;">
|
||||
<a href="{self.frontend_url}" class="button">Create Karaoke Now</a>
|
||||
</p>
|
||||
|
||||
<div class="footer">
|
||||
<p>© {self._get_year()} Nomad Karaoke. All rights reserved.</p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
text_content = f"""
|
||||
{credits} credits added to your Nomad Karaoke account!
|
||||
|
||||
Your new balance: {total_credits} credits
|
||||
|
||||
Each credit lets you create one professional karaoke video.
|
||||
|
||||
Start creating: {self.frontend_url}
|
||||
|
||||
---
|
||||
© {self._get_year()} Nomad Karaoke
|
||||
"""
|
||||
|
||||
return self.provider.send_email(email, subject, html_content, text_content)
|
||||
|
||||
def send_welcome_email(self, email: str, credits: int = 0) -> bool:
|
||||
"""
|
||||
Send welcome email to new users.
|
||||
|
||||
Args:
|
||||
email: User's email address
|
||||
credits: Initial credit balance (if any)
|
||||
|
||||
Returns:
|
||||
True if email was sent successfully
|
||||
"""
|
||||
subject = "Welcome to Nomad Karaoke! 🎤"
|
||||
|
||||
credits_text = f"You have <strong>{credits} credits</strong> to get started!" if credits > 0 else ""
|
||||
|
||||
html_content = f"""
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<style>
|
||||
body {{
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
line-height: 1.6;
|
||||
color: #333;
|
||||
max-width: 600px;
|
||||
margin: 0 auto;
|
||||
padding: 20px;
|
||||
}}
|
||||
.header {{
|
||||
text-align: center;
|
||||
padding: 20px 0;
|
||||
}}
|
||||
.logo {{
|
||||
font-size: 24px;
|
||||
font-weight: bold;
|
||||
color: #3b82f6;
|
||||
}}
|
||||
.button {{
|
||||
display: inline-block;
|
||||
background-color: #3b82f6;
|
||||
color: white;
|
||||
padding: 14px 28px;
|
||||
text-decoration: none;
|
||||
border-radius: 8px;
|
||||
font-weight: 600;
|
||||
margin: 20px 0;
|
||||
}}
|
||||
.feature {{
|
||||
display: flex;
|
||||
align-items: flex-start;
|
||||
margin: 16px 0;
|
||||
}}
|
||||
.feature-icon {{
|
||||
font-size: 24px;
|
||||
margin-right: 12px;
|
||||
}}
|
||||
.footer {{
|
||||
margin-top: 30px;
|
||||
padding-top: 20px;
|
||||
border-top: 1px solid #eee;
|
||||
font-size: 12px;
|
||||
color: #666;
|
||||
}}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="header">
|
||||
<div class="logo">🎤 Nomad Karaoke</div>
|
||||
</div>
|
||||
|
||||
<p>Welcome to Nomad Karaoke!</p>
|
||||
|
||||
<p>Turn any song into a professional karaoke video in minutes. {credits_text}</p>
|
||||
|
||||
<p><strong>Here's how it works:</strong></p>
|
||||
|
||||
<div class="feature">
|
||||
<span class="feature-icon">🎵</span>
|
||||
<div>
|
||||
<strong>1. Search for a song</strong><br>
|
||||
Enter the artist and title, and we'll find high-quality audio.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="feature">
|
||||
<span class="feature-icon">✨</span>
|
||||
<div>
|
||||
<strong>2. AI does the magic</strong><br>
|
||||
We separate vocals, transcribe lyrics, and sync everything perfectly.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="feature">
|
||||
<span class="feature-icon">✏️</span>
|
||||
<div>
|
||||
<strong>3. Review & customize</strong><br>
|
||||
Fine-tune the lyrics if needed, choose your instrumental.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="feature">
|
||||
<span class="feature-icon">🎬</span>
|
||||
<div>
|
||||
<strong>4. Get your video</strong><br>
|
||||
Download your 4K karaoke video or upload directly to YouTube.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<p style="text-align: center;">
|
||||
<a href="{self.frontend_url}" class="button">Get Started</a>
|
||||
</p>
|
||||
|
||||
<div class="footer">
|
||||
<p>Questions? Reply to this email and we'll help you out.</p>
|
||||
<p>© {self._get_year()} Nomad Karaoke. All rights reserved.</p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
text_content = f"""
|
||||
Welcome to Nomad Karaoke!
|
||||
|
||||
Turn any song into a professional karaoke video in minutes.
|
||||
|
||||
Here's how it works:
|
||||
|
||||
1. Search for a song
|
||||
Enter the artist and title, and we'll find high-quality audio.
|
||||
|
||||
2. AI does the magic
|
||||
We separate vocals, transcribe lyrics, and sync everything perfectly.
|
||||
|
||||
3. Review & customize
|
||||
Fine-tune the lyrics if needed, choose your instrumental.
|
||||
|
||||
4. Get your video
|
||||
Download your 4K karaoke video or upload directly to YouTube.
|
||||
|
||||
Get started: {self.frontend_url}
|
||||
|
||||
---
|
||||
© {self._get_year()} Nomad Karaoke
|
||||
"""
|
||||
|
||||
return self.provider.send_email(email, subject, html_content, text_content)
|
||||
|
||||
def send_beta_welcome_email(self, email: str, credits: int = 1) -> bool:
|
||||
"""
|
||||
Send welcome email to new beta testers.
|
||||
|
||||
Args:
|
||||
email: User's email address
|
||||
credits: Initial free credits granted
|
||||
|
||||
Returns:
|
||||
True if email was sent successfully
|
||||
"""
|
||||
subject = "Welcome Beta Tester! Free Karaoke Credits Inside 🎤"
|
||||
|
||||
html_content = f"""
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<style>
|
||||
body {{
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
line-height: 1.6;
|
||||
color: #333;
|
||||
max-width: 600px;
|
||||
margin: 0 auto;
|
||||
padding: 20px;
|
||||
}}
|
||||
.header {{
|
||||
text-align: center;
|
||||
padding: 20px 0;
|
||||
}}
|
||||
.logo {{
|
||||
font-size: 24px;
|
||||
font-weight: bold;
|
||||
color: #3b82f6;
|
||||
}}
|
||||
.beta-badge {{
|
||||
display: inline-block;
|
||||
background: linear-gradient(135deg, #8b5cf6, #ec4899);
|
||||
color: white;
|
||||
padding: 4px 12px;
|
||||
border-radius: 20px;
|
||||
font-size: 12px;
|
||||
font-weight: bold;
|
||||
margin-left: 8px;
|
||||
}}
|
||||
.credits-box {{
|
||||
background-color: #ecfdf5;
|
||||
border: 2px solid #10b981;
|
||||
border-radius: 12px;
|
||||
padding: 24px;
|
||||
text-align: center;
|
||||
margin: 20px 0;
|
||||
}}
|
||||
.credits-number {{
|
||||
font-size: 48px;
|
||||
font-weight: bold;
|
||||
color: #10b981;
|
||||
}}
|
||||
.reminder {{
|
||||
background-color: #fef3c7;
|
||||
border: 1px solid #fcd34d;
|
||||
border-radius: 8px;
|
||||
padding: 16px;
|
||||
margin: 20px 0;
|
||||
}}
|
||||
.button {{
|
||||
display: inline-block;
|
||||
background-color: #3b82f6;
|
||||
color: white;
|
||||
padding: 14px 28px;
|
||||
text-decoration: none;
|
||||
border-radius: 8px;
|
||||
font-weight: 600;
|
||||
margin: 20px 0;
|
||||
}}
|
||||
.footer {{
|
||||
margin-top: 30px;
|
||||
padding-top: 20px;
|
||||
border-top: 1px solid #eee;
|
||||
font-size: 12px;
|
||||
color: #666;
|
||||
}}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="header">
|
||||
<span class="logo">🎤 Nomad Karaoke</span>
|
||||
<span class="beta-badge">BETA TESTER</span>
|
||||
</div>
|
||||
|
||||
<p>Thank you for joining our beta program! 🎉</p>
|
||||
|
||||
<p>As promised, here's your free credit to create a karaoke video:</p>
|
||||
|
||||
<div class="credits-box">
|
||||
<div>Your balance:</div>
|
||||
<div class="credits-number">{credits}</div>
|
||||
<div>free credit{'' if credits == 1 else 's'}</div>
|
||||
</div>
|
||||
|
||||
<div class="reminder">
|
||||
<strong>📝 Your Promise:</strong><br>
|
||||
Remember, as a beta tester you agreed to:
|
||||
<ul style="margin: 10px 0; padding-left: 20px;">
|
||||
<li>Review and correct any lyrics timing issues</li>
|
||||
<li>Share your honest feedback after trying the tool</li>
|
||||
</ul>
|
||||
We'll send you a quick feedback form after your job completes!
|
||||
</div>
|
||||
|
||||
<p style="text-align: center;">
|
||||
<a href="{self.frontend_url}" class="button">Create Your Karaoke Video</a>
|
||||
</p>
|
||||
|
||||
<p>Thanks for helping us make Nomad Karaoke better!</p>
|
||||
|
||||
<div class="footer">
|
||||
<p>© {self._get_year()} Nomad Karaoke. All rights reserved.</p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
text_content = f"""
|
||||
Welcome Beta Tester!
|
||||
|
||||
Thank you for joining our beta program!
|
||||
|
||||
You've been granted {credits} free credit{'' if credits == 1 else 's'} to create karaoke videos.
|
||||
|
||||
YOUR PROMISE:
|
||||
As a beta tester, you agreed to:
|
||||
- Review and correct any lyrics timing issues
|
||||
- Share your honest feedback after trying the tool
|
||||
|
||||
We'll send you a quick feedback form after your job completes!
|
||||
|
||||
Create your karaoke video: {self.frontend_url}
|
||||
|
||||
Thanks for helping us make Nomad Karaoke better!
|
||||
|
||||
---
|
||||
© {self._get_year()} Nomad Karaoke
|
||||
"""
|
||||
|
||||
return self.provider.send_email(email, subject, html_content, text_content)
|
||||
|
||||
def send_feedback_request_email(self, email: str, feedback_url: str, job_title: Optional[str] = None) -> bool:
|
||||
"""
|
||||
Send feedback request email to beta testers.
|
||||
|
||||
Args:
|
||||
email: User's email address
|
||||
feedback_url: URL to the feedback form
|
||||
job_title: Optional title of the completed job
|
||||
|
||||
Returns:
|
||||
True if email was sent successfully
|
||||
"""
|
||||
subject = "Quick feedback on your karaoke experience? 🎤"
|
||||
|
||||
# Escape job_title to prevent XSS in email clients that render HTML
|
||||
safe_job_title = html.escape(job_title) if job_title else ""
|
||||
job_context = f" for <strong>{safe_job_title}</strong>" if job_title else ""
|
||||
|
||||
html_content = f"""
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<style>
|
||||
body {{
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
line-height: 1.6;
|
||||
color: #333;
|
||||
max-width: 600px;
|
||||
margin: 0 auto;
|
||||
padding: 20px;
|
||||
}}
|
||||
.header {{
|
||||
text-align: center;
|
||||
padding: 20px 0;
|
||||
}}
|
||||
.logo {{
|
||||
font-size: 24px;
|
||||
font-weight: bold;
|
||||
color: #3b82f6;
|
||||
}}
|
||||
.feedback-box {{
|
||||
background: linear-gradient(135deg, #f0f9ff, #e0f2fe);
|
||||
border: 2px solid #3b82f6;
|
||||
border-radius: 12px;
|
||||
padding: 24px;
|
||||
text-align: center;
|
||||
margin: 20px 0;
|
||||
}}
|
||||
.stars {{
|
||||
font-size: 36px;
|
||||
margin: 10px 0;
|
||||
}}
|
||||
.button {{
|
||||
display: inline-block;
|
||||
background-color: #3b82f6;
|
||||
color: white;
|
||||
padding: 14px 28px;
|
||||
text-decoration: none;
|
||||
border-radius: 8px;
|
||||
font-weight: 600;
|
||||
margin: 20px 0;
|
||||
}}
|
||||
.time-note {{
|
||||
background-color: #fef3c7;
|
||||
border-radius: 8px;
|
||||
padding: 12px;
|
||||
margin: 20px 0;
|
||||
font-size: 14px;
|
||||
}}
|
||||
.footer {{
|
||||
margin-top: 30px;
|
||||
padding-top: 20px;
|
||||
border-top: 1px solid #eee;
|
||||
font-size: 12px;
|
||||
color: #666;
|
||||
}}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="header">
|
||||
<div class="logo">🎤 Nomad Karaoke</div>
|
||||
</div>
|
||||
|
||||
<p>Hi there!</p>
|
||||
|
||||
<p>Hope you enjoyed creating your karaoke video{job_context}! As a beta tester, your feedback is super valuable to us.</p>
|
||||
|
||||
<div class="feedback-box">
|
||||
<div class="stars">⭐⭐⭐⭐⭐</div>
|
||||
<p><strong>How was your experience?</strong></p>
|
||||
<p>Quick 2-minute survey - we'd love to hear your thoughts!</p>
|
||||
</div>
|
||||
|
||||
<p style="text-align: center;">
|
||||
<a href="{feedback_url}" class="button">Share Your Feedback</a>
|
||||
</p>
|
||||
|
||||
<div class="time-note">
|
||||
⏱️ This takes less than 2 minutes and helps us improve the tool for everyone!
|
||||
</div>
|
||||
|
||||
<p>Specifically, we'd love to know:</p>
|
||||
<ul>
|
||||
<li>How easy was it to use?</li>
|
||||
<li>Were the lyrics accurate?</li>
|
||||
<li>How was the correction experience?</li>
|
||||
<li>What could we improve?</li>
|
||||
</ul>
|
||||
|
||||
<p>Thanks for being part of making Nomad Karaoke better! 🙏</p>
|
||||
|
||||
<div class="footer">
|
||||
<p>© {self._get_year()} Nomad Karaoke. All rights reserved.</p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
text_content = f"""
|
||||
Quick feedback on your karaoke experience?
|
||||
|
||||
Hi there!
|
||||
|
||||
Hope you enjoyed creating your karaoke video{' for ' + safe_job_title if safe_job_title else ''}!
|
||||
|
||||
As a beta tester, your feedback is super valuable to us.
|
||||
|
||||
Share your feedback (2-minute survey): {feedback_url}
|
||||
|
||||
We'd love to know:
|
||||
- How easy was it to use?
|
||||
- Were the lyrics accurate?
|
||||
- How was the correction experience?
|
||||
- What could we improve?
|
||||
|
||||
Thanks for being part of making Nomad Karaoke better!
|
||||
|
||||
---
|
||||
© {self._get_year()} Nomad Karaoke
|
||||
"""
|
||||
|
||||
return self.provider.send_email(email, subject, html_content, text_content)
|
||||
|
||||
def _get_year(self) -> int:
|
||||
"""Get current year for copyright notices."""
|
||||
from datetime import datetime
|
||||
return datetime.now().year
|
||||
|
||||
|
||||
# Global instance
|
||||
_email_service = None
|
||||
|
||||
|
||||
def get_email_service() -> EmailService:
|
||||
"""Get the global email service instance."""
|
||||
global _email_service
|
||||
if _email_service is None:
|
||||
_email_service = EmailService()
|
||||
return _email_service
|
||||
@@ -0,0 +1,451 @@
|
||||
"""
|
||||
Encoding Interface.
|
||||
|
||||
Defines the abstract interface for video encoding backends, allowing
|
||||
the video worker orchestrator to use either GCE or local encoding
|
||||
interchangeably.
|
||||
|
||||
This follows the Strategy pattern - different encoding implementations
|
||||
can be swapped without changing the orchestration logic.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from abc import ABC, abstractmethod
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Dict, List, Optional, Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingInput:
|
||||
"""
|
||||
Input configuration for video encoding.
|
||||
|
||||
Contains all the paths and metadata needed to encode a karaoke video.
|
||||
"""
|
||||
# Required input files
|
||||
title_video_path: str # Title card video (MOV)
|
||||
karaoke_video_path: str # Main karaoke video with vocals (MOV/MKV)
|
||||
instrumental_audio_path: str # Instrumental audio track (FLAC)
|
||||
|
||||
# Optional input files
|
||||
end_video_path: Optional[str] = None # End credits video (MOV)
|
||||
|
||||
# Metadata for output naming
|
||||
artist: str = ""
|
||||
title: str = ""
|
||||
brand_code: Optional[str] = None
|
||||
|
||||
# Output directory
|
||||
output_dir: str = ""
|
||||
|
||||
# Additional options
|
||||
options: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingOutput:
|
||||
"""
|
||||
Output from video encoding.
|
||||
|
||||
Contains paths to all generated video files and status information.
|
||||
"""
|
||||
success: bool
|
||||
error_message: Optional[str] = None
|
||||
|
||||
# Output file paths (relative to output_dir or absolute)
|
||||
karaoke_mp4_path: Optional[str] = None # Karaoke video with instrumental audio
|
||||
with_vocals_mp4_path: Optional[str] = None # With vocals MP4 version
|
||||
lossless_4k_mp4_path: Optional[str] = None # Final lossless 4K MP4
|
||||
lossy_4k_mp4_path: Optional[str] = None # Final lossy 4K MP4
|
||||
lossless_mkv_path: Optional[str] = None # MKV with FLAC (for YouTube)
|
||||
lossy_720p_mp4_path: Optional[str] = None # 720p web version
|
||||
|
||||
# All output files as a dict for convenience
|
||||
output_files: Dict[str, str] = field(default_factory=dict)
|
||||
|
||||
# Encoding metadata
|
||||
encoding_time_seconds: Optional[float] = None
|
||||
encoding_backend: Optional[str] = None # "gce" or "local"
|
||||
|
||||
|
||||
class EncodingBackend(ABC):
|
||||
"""
|
||||
Abstract base class for video encoding backends.
|
||||
|
||||
Implementations:
|
||||
- GCEEncodingBackend: Cloud-based encoding using Google Compute Engine
|
||||
- LocalEncodingBackend: Local FFmpeg-based encoding
|
||||
"""
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def name(self) -> str:
|
||||
"""Return the name of this encoding backend."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
|
||||
"""
|
||||
Encode video files according to the input configuration.
|
||||
|
||||
This method should:
|
||||
1. Create karaoke video with instrumental audio
|
||||
2. Convert with-vocals video to MP4 if needed
|
||||
3. Encode lossless 4K MP4 (concatenating title + karaoke + end)
|
||||
4. Encode lossy 4K MP4 with AAC audio
|
||||
5. Create MKV with FLAC audio for YouTube
|
||||
6. Encode 720p version for web
|
||||
|
||||
Args:
|
||||
input_config: EncodingInput with all required paths and options
|
||||
|
||||
Returns:
|
||||
EncodingOutput with paths to generated files and status
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def is_available(self) -> bool:
|
||||
"""
|
||||
Check if this encoding backend is available and configured.
|
||||
|
||||
Returns:
|
||||
True if the backend can be used, False otherwise
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def get_status(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Get the current status of the encoding backend.
|
||||
|
||||
Returns:
|
||||
Dict with status information (availability, queue length, etc.)
|
||||
"""
|
||||
pass
|
||||
|
||||
|
||||
class LocalEncodingBackend(EncodingBackend):
|
||||
"""
|
||||
Local FFmpeg-based encoding backend.
|
||||
|
||||
Wraps the LocalEncodingService to implement the EncodingBackend interface.
|
||||
Uses asyncio.to_thread() to run synchronous FFmpeg operations asynchronously.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
dry_run: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the local encoding backend.
|
||||
|
||||
Args:
|
||||
dry_run: If True, log operations without executing
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.dry_run = dry_run
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
self._service = None
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return "local"
|
||||
|
||||
def _get_service(self):
|
||||
"""Lazy-load the local encoding service."""
|
||||
if self._service is None:
|
||||
from backend.services.local_encoding_service import LocalEncodingService
|
||||
self._service = LocalEncodingService(
|
||||
dry_run=self.dry_run,
|
||||
logger=self.logger
|
||||
)
|
||||
return self._service
|
||||
|
||||
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
|
||||
"""
|
||||
Encode video using local FFmpeg.
|
||||
|
||||
Args:
|
||||
input_config: Encoding input configuration
|
||||
|
||||
Returns:
|
||||
EncodingOutput with results
|
||||
"""
|
||||
import asyncio
|
||||
import time
|
||||
from backend.services.local_encoding_service import EncodingConfig
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
# Build output file paths
|
||||
base_name = f"{input_config.artist} - {input_config.title}"
|
||||
output_dir = input_config.output_dir or "."
|
||||
|
||||
import os
|
||||
config = EncodingConfig(
|
||||
title_video=input_config.title_video_path,
|
||||
karaoke_video=input_config.karaoke_video_path,
|
||||
instrumental_audio=input_config.instrumental_audio_path,
|
||||
end_video=input_config.end_video_path,
|
||||
output_karaoke_mp4=os.path.join(output_dir, f"{base_name} (Karaoke).mp4"),
|
||||
output_with_vocals_mp4=os.path.join(output_dir, f"{base_name} (With Vocals).mp4"),
|
||||
output_lossless_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mp4"),
|
||||
output_lossy_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 4k).mp4"),
|
||||
output_lossless_mkv=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mkv"),
|
||||
output_720p_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 720p).mp4"),
|
||||
)
|
||||
|
||||
# Run encoding in thread pool to avoid blocking
|
||||
service = self._get_service()
|
||||
result = await asyncio.to_thread(service.encode_all_formats, config)
|
||||
|
||||
encoding_time = time.time() - start_time
|
||||
|
||||
if result.success:
|
||||
return EncodingOutput(
|
||||
success=True,
|
||||
karaoke_mp4_path=config.output_karaoke_mp4,
|
||||
with_vocals_mp4_path=config.output_with_vocals_mp4,
|
||||
lossless_4k_mp4_path=config.output_lossless_4k_mp4,
|
||||
lossy_4k_mp4_path=config.output_lossy_4k_mp4,
|
||||
lossless_mkv_path=config.output_lossless_mkv,
|
||||
lossy_720p_mp4_path=config.output_720p_mp4,
|
||||
output_files=result.output_files,
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
)
|
||||
else:
|
||||
return EncodingOutput(
|
||||
success=False,
|
||||
error_message=result.error,
|
||||
output_files=result.output_files,
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
)
|
||||
|
||||
async def is_available(self) -> bool:
|
||||
"""Check if local encoding is available (FFmpeg installed)."""
|
||||
import subprocess
|
||||
import asyncio
|
||||
|
||||
try:
|
||||
result = await asyncio.to_thread(
|
||||
subprocess.run,
|
||||
["ffmpeg", "-version"],
|
||||
capture_output=True,
|
||||
timeout=10
|
||||
)
|
||||
return result.returncode == 0
|
||||
except (FileNotFoundError, subprocess.TimeoutExpired):
|
||||
return False
|
||||
|
||||
async def get_status(self) -> Dict[str, Any]:
|
||||
"""Get local encoding status."""
|
||||
available = await self.is_available()
|
||||
service = self._get_service()
|
||||
|
||||
return {
|
||||
"backend": self.name,
|
||||
"available": available,
|
||||
"hwaccel_available": service.hwaccel_available if available else False,
|
||||
"video_encoder": service.video_encoder if available else None,
|
||||
}
|
||||
|
||||
|
||||
class GCEEncodingBackend(EncodingBackend):
|
||||
"""
|
||||
GCE-based encoding backend.
|
||||
|
||||
Wraps the existing EncodingService to implement the EncodingBackend interface.
|
||||
Submits jobs to a remote GCE worker for high-performance encoding.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
dry_run: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the GCE encoding backend.
|
||||
|
||||
Args:
|
||||
dry_run: Ignored for GCE backend (remote service doesn't support dry run)
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.dry_run = dry_run # Stored but not used (GCE is remote)
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
self._service = None
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return "gce"
|
||||
|
||||
def _get_service(self):
|
||||
"""Lazy-load the GCE encoding service."""
|
||||
if self._service is None:
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
self._service = get_encoding_service()
|
||||
return self._service
|
||||
|
||||
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
|
||||
"""
|
||||
Encode video using GCE worker.
|
||||
|
||||
Note: GCE encoding requires files to be in GCS, so this method
|
||||
expects input_config.options to contain 'input_gcs_path' and
|
||||
'output_gcs_path'.
|
||||
|
||||
Args:
|
||||
input_config: Encoding input configuration
|
||||
|
||||
Returns:
|
||||
EncodingOutput with results
|
||||
"""
|
||||
import time
|
||||
|
||||
start_time = time.time()
|
||||
service = self._get_service()
|
||||
|
||||
# GCE requires GCS paths
|
||||
input_gcs_path = input_config.options.get("input_gcs_path")
|
||||
output_gcs_path = input_config.options.get("output_gcs_path")
|
||||
job_id = input_config.options.get("job_id", input_config.brand_code or "unknown")
|
||||
|
||||
if not input_gcs_path or not output_gcs_path:
|
||||
return EncodingOutput(
|
||||
success=False,
|
||||
error_message="GCE encoding requires input_gcs_path and output_gcs_path in options",
|
||||
encoding_backend=self.name
|
||||
)
|
||||
|
||||
try:
|
||||
# Build encoding config
|
||||
encoding_config = {
|
||||
"formats": ["mp4_4k_lossless", "mp4_4k_lossy", "mkv_4k", "mp4_720p"],
|
||||
"artist": input_config.artist,
|
||||
"title": input_config.title,
|
||||
}
|
||||
|
||||
# Submit and wait for completion
|
||||
result = await service.encode_videos(
|
||||
job_id=job_id,
|
||||
input_gcs_path=input_gcs_path,
|
||||
output_gcs_path=output_gcs_path,
|
||||
encoding_config=encoding_config,
|
||||
)
|
||||
|
||||
encoding_time = time.time() - start_time
|
||||
|
||||
# Extract output file paths from result
|
||||
# Handle case where GCE worker returns a list or unexpected format
|
||||
if isinstance(result, list):
|
||||
# If result is a list, try to find the output_files in the first dict
|
||||
self.logger.warning(f"GCE returned list instead of dict: {result}")
|
||||
result = result[0] if result and isinstance(result[0], dict) else {}
|
||||
if not isinstance(result, dict):
|
||||
self.logger.error(f"Unexpected GCE result type: {type(result)}")
|
||||
result = {}
|
||||
raw_output_files = result.get("output_files", {})
|
||||
|
||||
# Convert output_files from list of paths to dict if needed
|
||||
# GCE worker returns list like: ["path/output_4k_lossless.mp4", "path/output_720p.mp4"]
|
||||
# We need dict like: {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
|
||||
if isinstance(raw_output_files, list):
|
||||
output_files = {}
|
||||
for path in raw_output_files:
|
||||
if not isinstance(path, str):
|
||||
continue
|
||||
filename = path.split("/")[-1] if "/" in path else path
|
||||
# Map filename patterns to output format keys
|
||||
if "4k_lossless" in filename:
|
||||
output_files["mp4_4k_lossless"] = path
|
||||
elif "4k_lossy" in filename:
|
||||
output_files["mp4_4k_lossy"] = path
|
||||
elif "720p" in filename:
|
||||
output_files["mp4_720p"] = path
|
||||
elif filename.endswith(".mkv"):
|
||||
output_files["mkv_4k"] = path
|
||||
self.logger.info(f"Converted output_files list to dict: {output_files}")
|
||||
else:
|
||||
output_files = raw_output_files if isinstance(raw_output_files, dict) else {}
|
||||
|
||||
return EncodingOutput(
|
||||
success=True,
|
||||
lossless_4k_mp4_path=output_files.get("mp4_4k_lossless"),
|
||||
lossy_4k_mp4_path=output_files.get("mp4_4k_lossy"),
|
||||
lossless_mkv_path=output_files.get("mkv_4k"),
|
||||
lossy_720p_mp4_path=output_files.get("mp4_720p"),
|
||||
output_files=output_files,
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
encoding_time = time.time() - start_time
|
||||
self.logger.error(f"GCE encoding failed: {e}")
|
||||
return EncodingOutput(
|
||||
success=False,
|
||||
error_message=str(e),
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
)
|
||||
|
||||
async def is_available(self) -> bool:
|
||||
"""Check if GCE encoding is available and configured."""
|
||||
service = self._get_service()
|
||||
return service.is_enabled
|
||||
|
||||
async def get_status(self) -> Dict[str, Any]:
|
||||
"""Get GCE encoding status."""
|
||||
service = self._get_service()
|
||||
|
||||
status = {
|
||||
"backend": self.name,
|
||||
"available": service.is_enabled,
|
||||
"configured": service.is_configured,
|
||||
}
|
||||
|
||||
if service.is_configured:
|
||||
try:
|
||||
health = await service.health_check()
|
||||
status["health"] = health
|
||||
except Exception as e:
|
||||
status["health_error"] = str(e)
|
||||
|
||||
return status
|
||||
|
||||
|
||||
# Factory function to get an encoding backend
|
||||
def get_encoding_backend(
|
||||
backend_type: str = "auto",
|
||||
**kwargs
|
||||
) -> EncodingBackend:
|
||||
"""
|
||||
Get an encoding backend instance.
|
||||
|
||||
Args:
|
||||
backend_type: Type of backend - "local", "gce", or "auto"
|
||||
**kwargs: Additional arguments for the backend
|
||||
|
||||
Returns:
|
||||
EncodingBackend instance
|
||||
|
||||
Raises:
|
||||
ValueError: If backend_type is unknown
|
||||
"""
|
||||
if backend_type == "local":
|
||||
return LocalEncodingBackend(**kwargs)
|
||||
elif backend_type == "gce":
|
||||
return GCEEncodingBackend(**kwargs)
|
||||
elif backend_type == "auto":
|
||||
# Check if GCE is available, otherwise use local
|
||||
gce_backend = GCEEncodingBackend(**kwargs)
|
||||
if gce_backend._get_service().is_enabled:
|
||||
return gce_backend
|
||||
return LocalEncodingBackend(**kwargs)
|
||||
else:
|
||||
raise ValueError(f"Unknown encoding backend type: {backend_type}")
|
||||
@@ -0,0 +1,279 @@
|
||||
"""
|
||||
GCE Encoding Worker Service.
|
||||
|
||||
This service dispatches video encoding jobs to a dedicated high-performance
|
||||
GCE instance (C4-standard with Intel Granite Rapids CPU) for faster encoding.
|
||||
|
||||
The GCE worker provides:
|
||||
- 3.9 GHz all-core frequency (vs 3.7 GHz on Cloud Run)
|
||||
- Dedicated vCPUs (no contention)
|
||||
- 2-3x faster FFmpeg libx264 encoding
|
||||
|
||||
Usage:
|
||||
encoding_service = get_encoding_service()
|
||||
if encoding_service.is_configured:
|
||||
result = await encoding_service.encode_videos(job_id, input_gcs_path, config)
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
import aiohttp
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class EncodingService:
|
||||
"""Service for dispatching encoding jobs to GCE worker."""
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
self._url = None
|
||||
self._api_key = None
|
||||
self._initialized = False
|
||||
|
||||
def _load_credentials(self):
|
||||
"""Load encoding worker URL and API key from config/secrets."""
|
||||
if self._initialized:
|
||||
return
|
||||
|
||||
# Try environment variables first, then Secret Manager
|
||||
self._url = self.settings.encoding_worker_url
|
||||
self._api_key = self.settings.encoding_worker_api_key
|
||||
|
||||
# Fall back to Secret Manager
|
||||
if not self._url:
|
||||
self._url = self.settings.get_secret("encoding-worker-url")
|
||||
if not self._api_key:
|
||||
self._api_key = self.settings.get_secret("encoding-worker-api-key")
|
||||
|
||||
self._initialized = True
|
||||
|
||||
@property
|
||||
def is_configured(self) -> bool:
|
||||
"""Check if encoding service is configured with URL and API key."""
|
||||
self._load_credentials()
|
||||
return bool(self._url and self._api_key)
|
||||
|
||||
@property
|
||||
def is_enabled(self) -> bool:
|
||||
"""Check if GCE encoding is enabled and configured."""
|
||||
return self.settings.use_gce_encoding and self.is_configured
|
||||
|
||||
async def submit_encoding_job(
|
||||
self,
|
||||
job_id: str,
|
||||
input_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
encoding_config: Dict[str, Any],
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit an encoding job to the GCE worker.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
input_gcs_path: GCS path to input files (gs://bucket/path/)
|
||||
output_gcs_path: GCS path for output files (gs://bucket/path/)
|
||||
encoding_config: Configuration for encoding (formats, quality, etc.)
|
||||
|
||||
Returns:
|
||||
Response from the encoding worker
|
||||
|
||||
Raises:
|
||||
Exception: If submission fails
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._url}/encode"
|
||||
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
|
||||
payload = {
|
||||
"job_id": job_id,
|
||||
"input_gcs_path": input_gcs_path,
|
||||
"output_gcs_path": output_gcs_path,
|
||||
"encoding_config": encoding_config,
|
||||
}
|
||||
|
||||
logger.info(f"[job:{job_id}] Submitting encoding job to GCE worker: {url}")
|
||||
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.post(url, json=payload, headers=headers, timeout=30) as resp:
|
||||
if resp.status == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp.status == 409:
|
||||
raise RuntimeError(f"Encoding job {job_id} already exists")
|
||||
if resp.status != 200:
|
||||
text = await resp.text()
|
||||
raise RuntimeError(f"Failed to submit encoding job: {resp.status} - {text}")
|
||||
|
||||
return await resp.json()
|
||||
|
||||
async def get_job_status(self, job_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Get the status of an encoding job.
|
||||
|
||||
Args:
|
||||
job_id: Job identifier
|
||||
|
||||
Returns:
|
||||
Job status including: status, progress, error, output_files
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._url}/status/{job_id}"
|
||||
headers = {"X-API-Key": self._api_key}
|
||||
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(url, headers=headers, timeout=30) as resp:
|
||||
if resp.status == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp.status == 404:
|
||||
raise RuntimeError(f"Encoding job {job_id} not found")
|
||||
if resp.status != 200:
|
||||
text = await resp.text()
|
||||
raise RuntimeError(f"Failed to get job status: {resp.status} - {text}")
|
||||
|
||||
return await resp.json()
|
||||
|
||||
async def wait_for_completion(
|
||||
self,
|
||||
job_id: str,
|
||||
poll_interval: float = 10.0,
|
||||
timeout: float = 3600.0,
|
||||
progress_callback=None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Poll for encoding job completion.
|
||||
|
||||
Args:
|
||||
job_id: Job identifier
|
||||
poll_interval: Seconds between status checks
|
||||
timeout: Maximum time to wait (default 1 hour)
|
||||
progress_callback: Optional callback(progress: int) for progress updates
|
||||
|
||||
Returns:
|
||||
Final job status with output files
|
||||
|
||||
Raises:
|
||||
TimeoutError: If job doesn't complete within timeout
|
||||
RuntimeError: If job fails
|
||||
"""
|
||||
logger.info(f"[job:{job_id}] Waiting for GCE encoding to complete...")
|
||||
|
||||
start_time = asyncio.get_event_loop().time()
|
||||
last_progress = 0
|
||||
|
||||
while True:
|
||||
elapsed = asyncio.get_event_loop().time() - start_time
|
||||
if elapsed > timeout:
|
||||
raise TimeoutError(f"Encoding job {job_id} timed out after {timeout}s")
|
||||
|
||||
status = await self.get_job_status(job_id)
|
||||
|
||||
# Handle case where GCE worker returns a list instead of dict
|
||||
if isinstance(status, list):
|
||||
logger.warning(f"[job:{job_id}] GCE returned list instead of dict: {status}")
|
||||
status = status[0] if status and isinstance(status[0], dict) else {}
|
||||
if not isinstance(status, dict):
|
||||
logger.error(f"[job:{job_id}] Unexpected status type: {type(status)}")
|
||||
status = {}
|
||||
|
||||
job_status = status.get("status", "unknown")
|
||||
progress = status.get("progress", 0)
|
||||
|
||||
# Report progress
|
||||
if progress != last_progress:
|
||||
logger.info(f"[job:{job_id}] Encoding progress: {progress}%")
|
||||
last_progress = progress
|
||||
if progress_callback:
|
||||
try:
|
||||
progress_callback(progress)
|
||||
except Exception as e:
|
||||
logger.warning(f"Progress callback failed: {e}")
|
||||
|
||||
if job_status == "complete":
|
||||
logger.info(f"[job:{job_id}] GCE encoding complete in {elapsed:.1f}s")
|
||||
return status
|
||||
|
||||
if job_status == "failed":
|
||||
error = status.get("error", "Unknown error")
|
||||
raise RuntimeError(f"Encoding job {job_id} failed: {error}")
|
||||
|
||||
await asyncio.sleep(poll_interval)
|
||||
|
||||
async def encode_videos(
|
||||
self,
|
||||
job_id: str,
|
||||
input_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
encoding_config: Optional[Dict[str, Any]] = None,
|
||||
progress_callback=None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit encoding job and wait for completion.
|
||||
|
||||
This is a convenience method that combines submit + wait.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
input_gcs_path: GCS path to input files
|
||||
output_gcs_path: GCS path for output files
|
||||
encoding_config: Optional encoding configuration
|
||||
progress_callback: Optional callback for progress updates
|
||||
|
||||
Returns:
|
||||
Final job status with output files
|
||||
"""
|
||||
config = encoding_config or {"formats": ["mp4_4k", "mp4_720p"]}
|
||||
|
||||
# Submit the job
|
||||
await self.submit_encoding_job(job_id, input_gcs_path, output_gcs_path, config)
|
||||
|
||||
# Wait for completion
|
||||
return await self.wait_for_completion(
|
||||
job_id, progress_callback=progress_callback
|
||||
)
|
||||
|
||||
async def health_check(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Check the health of the encoding worker.
|
||||
|
||||
Returns:
|
||||
Health status including active jobs and FFmpeg version
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
return {"status": "not_configured"}
|
||||
|
||||
url = f"{self._url}/health"
|
||||
headers = {"X-API-Key": self._api_key}
|
||||
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(url, headers=headers, timeout=10) as resp:
|
||||
if resp.status == 200:
|
||||
return await resp.json()
|
||||
return {"status": "error", "code": resp.status}
|
||||
except Exception as e:
|
||||
return {"status": "error", "error": str(e)}
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_encoding_service: Optional[EncodingService] = None
|
||||
|
||||
|
||||
def get_encoding_service() -> EncodingService:
|
||||
"""Get the singleton encoding service instance."""
|
||||
global _encoding_service
|
||||
if _encoding_service is None:
|
||||
_encoding_service = EncodingService()
|
||||
return _encoding_service
|
||||
@@ -0,0 +1,512 @@
|
||||
"""
|
||||
Firestore database operations for job management.
|
||||
"""
|
||||
import logging
|
||||
from typing import Optional, Dict, Any, List
|
||||
from datetime import datetime
|
||||
from google.cloud import firestore
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
|
||||
from backend.config import settings
|
||||
from backend.models.job import Job, JobStatus, TimelineEvent
|
||||
from backend.models.worker_log import WorkerLogEntry
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class FirestoreService:
|
||||
"""Service for Firestore database operations."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize Firestore client."""
|
||||
self.db = firestore.Client(project=settings.google_cloud_project)
|
||||
self.collection = settings.firestore_collection
|
||||
self.tokens_collection = "auth_tokens" # Collection for access tokens
|
||||
|
||||
def create_job(self, job: Job) -> None:
|
||||
"""Create a new job in Firestore."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.collection).document(job.job_id)
|
||||
doc_ref.set(job.model_dump(mode='json'))
|
||||
logger.info(f"Created job {job.job_id} in Firestore")
|
||||
except Exception as e:
|
||||
logger.error(f"Error creating job {job.job_id}: {e}")
|
||||
raise
|
||||
|
||||
def get_job(self, job_id: str) -> Optional[Job]:
|
||||
"""Get a job by ID."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc = doc_ref.get()
|
||||
|
||||
logger.debug(f"Fetching job {job_id} from collection {self.collection}")
|
||||
logger.debug(f"Document exists: {doc.exists}")
|
||||
|
||||
if not doc.exists:
|
||||
return None
|
||||
|
||||
data = doc.to_dict()
|
||||
logger.debug(f"Document data keys: {list(data.keys()) if data else 'None'}")
|
||||
logger.debug(f"Document data: {data}")
|
||||
|
||||
return Job(**data)
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def update_job(self, job_id: str, updates: Dict[str, Any]) -> None:
|
||||
"""Update a job with partial data."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
|
||||
# Add updated_at timestamp
|
||||
updates['updated_at'] = datetime.utcnow()
|
||||
|
||||
doc_ref.update(updates)
|
||||
logger.info(f"Updated job {job_id} in Firestore")
|
||||
except Exception as e:
|
||||
logger.error(f"Error updating job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def update_job_status(
|
||||
self,
|
||||
job_id: str,
|
||||
status: JobStatus,
|
||||
progress: Optional[int] = None,
|
||||
message: Optional[str] = None,
|
||||
**additional_fields
|
||||
) -> None:
|
||||
"""Update job status and add timeline event."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
|
||||
# Create timeline event
|
||||
timeline_event = TimelineEvent(
|
||||
status=status.value,
|
||||
timestamp=datetime.utcnow().isoformat(),
|
||||
progress=progress,
|
||||
message=message
|
||||
)
|
||||
|
||||
# Prepare updates
|
||||
updates = {
|
||||
'status': status.value,
|
||||
'updated_at': datetime.utcnow(),
|
||||
'timeline': firestore.ArrayUnion([timeline_event.model_dump(mode='json')])
|
||||
}
|
||||
|
||||
if progress is not None:
|
||||
updates['progress'] = progress
|
||||
|
||||
# Add any additional fields
|
||||
updates.update(additional_fields)
|
||||
|
||||
doc_ref.update(updates)
|
||||
logger.info(f"Updated job {job_id} status to {status.value}")
|
||||
except Exception as e:
|
||||
logger.error(f"Error updating job status {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def list_jobs(
|
||||
self,
|
||||
status: Optional[JobStatus] = None,
|
||||
environment: Optional[str] = None,
|
||||
client_id: Optional[str] = None,
|
||||
created_after: Optional[datetime] = None,
|
||||
created_before: Optional[datetime] = None,
|
||||
user_email: Optional[str] = None,
|
||||
limit: int = 100
|
||||
) -> List[Job]:
|
||||
"""
|
||||
List jobs with optional filters.
|
||||
|
||||
Args:
|
||||
status: Filter by job status
|
||||
environment: Filter by request_metadata.environment (test/production/development)
|
||||
client_id: Filter by request_metadata.client_id
|
||||
created_after: Filter jobs created after this datetime
|
||||
created_before: Filter jobs created before this datetime
|
||||
user_email: Filter by user_email (owner of the job)
|
||||
limit: Maximum number of jobs to return
|
||||
|
||||
Returns:
|
||||
List of Job objects matching filters, ordered by created_at descending
|
||||
"""
|
||||
try:
|
||||
query = self.db.collection(self.collection)
|
||||
|
||||
if status:
|
||||
query = query.where(filter=FieldFilter('status', '==', status.value))
|
||||
|
||||
# Filter by request_metadata fields using dot notation
|
||||
if environment:
|
||||
query = query.where(filter=FieldFilter('request_metadata.environment', '==', environment))
|
||||
|
||||
if client_id:
|
||||
query = query.where(filter=FieldFilter('request_metadata.client_id', '==', client_id))
|
||||
|
||||
# Filter by user_email (job owner)
|
||||
if user_email:
|
||||
query = query.where(filter=FieldFilter('user_email', '==', user_email.lower()))
|
||||
|
||||
# Date range filters
|
||||
if created_after:
|
||||
query = query.where(filter=FieldFilter('created_at', '>=', created_after))
|
||||
|
||||
if created_before:
|
||||
query = query.where(filter=FieldFilter('created_at', '<=', created_before))
|
||||
|
||||
query = query.order_by('created_at', direction=firestore.Query.DESCENDING).limit(limit)
|
||||
|
||||
docs = query.stream()
|
||||
jobs = [Job(**doc.to_dict()) for doc in docs]
|
||||
|
||||
return jobs
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing jobs: {e}")
|
||||
raise
|
||||
|
||||
def delete_jobs_by_filter(
|
||||
self,
|
||||
environment: Optional[str] = None,
|
||||
client_id: Optional[str] = None,
|
||||
status: Optional[JobStatus] = None,
|
||||
created_before: Optional[datetime] = None,
|
||||
) -> int:
|
||||
"""
|
||||
Delete multiple jobs matching filter criteria.
|
||||
|
||||
CAUTION: This is a destructive operation. Use carefully.
|
||||
|
||||
Args:
|
||||
environment: Delete jobs with this environment (e.g., "test")
|
||||
client_id: Delete jobs from this client
|
||||
status: Delete jobs with this status
|
||||
created_before: Delete jobs created before this datetime
|
||||
|
||||
Returns:
|
||||
Number of jobs deleted
|
||||
"""
|
||||
try:
|
||||
query = self.db.collection(self.collection)
|
||||
|
||||
if environment:
|
||||
query = query.where(filter=FieldFilter('request_metadata.environment', '==', environment))
|
||||
|
||||
if client_id:
|
||||
query = query.where(filter=FieldFilter('request_metadata.client_id', '==', client_id))
|
||||
|
||||
if status:
|
||||
query = query.where(filter=FieldFilter('status', '==', status.value))
|
||||
|
||||
if created_before:
|
||||
query = query.where(filter=FieldFilter('created_at', '<=', created_before))
|
||||
|
||||
# Get matching documents
|
||||
docs = list(query.stream())
|
||||
deleted_count = 0
|
||||
|
||||
# Delete in batches
|
||||
batch = self.db.batch()
|
||||
batch_count = 0
|
||||
|
||||
for doc in docs:
|
||||
batch.delete(doc.reference)
|
||||
batch_count += 1
|
||||
deleted_count += 1
|
||||
|
||||
# Firestore batch limit is 500
|
||||
if batch_count >= 500:
|
||||
batch.commit()
|
||||
batch = self.db.batch()
|
||||
batch_count = 0
|
||||
|
||||
# Commit any remaining deletes
|
||||
if batch_count > 0:
|
||||
batch.commit()
|
||||
|
||||
logger.info(f"Deleted {deleted_count} jobs matching filter criteria")
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting jobs by filter: {e}")
|
||||
raise
|
||||
|
||||
def delete_job(self, job_id: str) -> None:
|
||||
"""Delete a job from Firestore."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.delete()
|
||||
logger.info(f"Deleted job {job_id} from Firestore")
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def append_worker_log(self, job_id: str, log_entry: Dict[str, Any]) -> None:
|
||||
"""
|
||||
Atomically append a log entry to worker_logs using ArrayUnion.
|
||||
|
||||
This avoids the race condition of read-modify-write when multiple
|
||||
workers are logging concurrently.
|
||||
|
||||
DEPRECATED: Use append_log_to_subcollection() instead to avoid
|
||||
the 1MB document size limit.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
log_entry: Log entry dict with timestamp, level, worker, message
|
||||
"""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.update({
|
||||
'worker_logs': firestore.ArrayUnion([log_entry]),
|
||||
'updated_at': datetime.utcnow()
|
||||
})
|
||||
# Don't log every append - too spammy
|
||||
except Exception as e:
|
||||
# Log but don't raise - logging shouldn't break workers
|
||||
logger.debug(f"Error appending worker log for job {job_id}: {e}")
|
||||
|
||||
# ============================================
|
||||
# Worker Log Subcollection Methods
|
||||
# ============================================
|
||||
# These methods store logs in a subcollection (jobs/{job_id}/logs)
|
||||
# instead of an embedded array to avoid the 1MB document size limit.
|
||||
|
||||
def append_log_to_subcollection(self, job_id: str, log_entry: WorkerLogEntry) -> None:
|
||||
"""
|
||||
Append a log entry to the logs subcollection.
|
||||
|
||||
Stores logs at: jobs/{job_id}/logs/{log_id}
|
||||
|
||||
This approach avoids the 1MB document size limit by storing each
|
||||
log entry as a separate document in a subcollection.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
log_entry: WorkerLogEntry instance
|
||||
"""
|
||||
try:
|
||||
# Ensure job_id is set on the log entry
|
||||
log_entry.job_id = job_id
|
||||
|
||||
# Get subcollection reference
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
|
||||
# Add document with auto-generated ID or use log_entry.id
|
||||
doc_ref = logs_ref.document(log_entry.id)
|
||||
doc_ref.set(log_entry.to_dict())
|
||||
|
||||
# Don't log every append - too spammy
|
||||
except Exception as e:
|
||||
# Log but don't raise - logging shouldn't break workers
|
||||
logger.debug(f"Error appending log to subcollection for job {job_id}: {e}")
|
||||
|
||||
def get_logs_from_subcollection(
|
||||
self,
|
||||
job_id: str,
|
||||
limit: int = 500,
|
||||
since_timestamp: Optional[datetime] = None,
|
||||
worker: Optional[str] = None,
|
||||
offset: int = 0
|
||||
) -> List[WorkerLogEntry]:
|
||||
"""
|
||||
Get log entries from the logs subcollection.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
limit: Maximum number of logs to return (default 500)
|
||||
since_timestamp: Return only logs after this timestamp
|
||||
worker: Filter by worker name (optional)
|
||||
offset: Number of logs to skip (for pagination)
|
||||
|
||||
Returns:
|
||||
List of WorkerLogEntry instances, ordered by timestamp ascending
|
||||
"""
|
||||
try:
|
||||
# Get subcollection reference
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
|
||||
# Build query
|
||||
query = logs_ref.order_by("timestamp", direction=firestore.Query.ASCENDING)
|
||||
|
||||
if since_timestamp:
|
||||
query = query.where(filter=FieldFilter("timestamp", ">", since_timestamp))
|
||||
|
||||
if worker:
|
||||
query = query.where(filter=FieldFilter("worker", "==", worker))
|
||||
|
||||
# Apply offset and limit
|
||||
if offset > 0:
|
||||
query = query.offset(offset)
|
||||
|
||||
query = query.limit(limit)
|
||||
|
||||
# Execute query
|
||||
docs = query.stream()
|
||||
logs = [WorkerLogEntry.from_dict(doc.to_dict()) for doc in docs]
|
||||
|
||||
return logs
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting logs from subcollection for job {job_id}: {e}")
|
||||
return []
|
||||
|
||||
def get_logs_count_from_subcollection(self, job_id: str) -> int:
|
||||
"""
|
||||
Get the total count of log entries in the subcollection.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
|
||||
Returns:
|
||||
Total count of log entries
|
||||
"""
|
||||
try:
|
||||
# Get subcollection reference
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
|
||||
# Use aggregation query for efficient counting
|
||||
count_query = logs_ref.count()
|
||||
result = count_query.get()
|
||||
|
||||
# Result is a list of AggregationResult, we want the first one's count
|
||||
if result and len(result) > 0:
|
||||
return result[0][0].value
|
||||
return 0
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error counting logs for job {job_id}: {e}")
|
||||
return 0
|
||||
|
||||
def delete_logs_subcollection(self, job_id: str, batch_size: int = 500) -> int:
|
||||
"""
|
||||
Delete all log entries in the logs subcollection.
|
||||
|
||||
This is used when deleting a job to clean up its logs.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
batch_size: Number of documents to delete per batch
|
||||
|
||||
Returns:
|
||||
Number of logs deleted
|
||||
"""
|
||||
try:
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
deleted_count = 0
|
||||
|
||||
while True:
|
||||
# Get a batch of documents
|
||||
docs = logs_ref.limit(batch_size).stream()
|
||||
deleted_in_batch = 0
|
||||
|
||||
# Delete in a batch
|
||||
batch = self.db.batch()
|
||||
for doc in docs:
|
||||
batch.delete(doc.reference)
|
||||
deleted_in_batch += 1
|
||||
|
||||
if deleted_in_batch == 0:
|
||||
break
|
||||
|
||||
batch.commit()
|
||||
deleted_count += deleted_in_batch
|
||||
|
||||
# If we deleted less than batch_size, we're done
|
||||
if deleted_in_batch < batch_size:
|
||||
break
|
||||
|
||||
if deleted_count > 0:
|
||||
logger.info(f"Deleted {deleted_count} logs for job {job_id}")
|
||||
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting logs subcollection for job {job_id}: {e}")
|
||||
return 0
|
||||
|
||||
# ============================================
|
||||
# Token Management Methods
|
||||
# ============================================
|
||||
|
||||
def create_token(self, token: str, token_data: Dict[str, Any]) -> None:
|
||||
"""Create a new access token in Firestore."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.tokens_collection).document(token)
|
||||
doc_ref.set(token_data)
|
||||
logger.info(f"Created token in Firestore")
|
||||
except Exception as e:
|
||||
logger.error(f"Error creating token: {e}")
|
||||
raise
|
||||
|
||||
def get_token(self, token: str) -> Optional[Dict[str, Any]]:
|
||||
"""Get token data by token string."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.tokens_collection).document(token)
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
return None
|
||||
|
||||
return doc.to_dict()
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting token: {e}")
|
||||
return None
|
||||
|
||||
def update_token(self, token: str, updates: Dict[str, Any]) -> None:
|
||||
"""Update token data."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.tokens_collection).document(token)
|
||||
doc_ref.update(updates)
|
||||
logger.info(f"Updated token in Firestore")
|
||||
except Exception as e:
|
||||
logger.error(f"Error updating token: {e}")
|
||||
raise
|
||||
|
||||
def increment_token_usage(self, token: str, job_id: str) -> None:
|
||||
"""Increment token usage count and add job to history."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.tokens_collection).document(token)
|
||||
|
||||
# Use Firestore transaction to ensure atomic increment
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
snapshot = doc_ref.get(transaction=transaction)
|
||||
if not snapshot.exists:
|
||||
raise ValueError("Token not found")
|
||||
|
||||
data = snapshot.to_dict()
|
||||
current_usage = data.get("usage_count", 0)
|
||||
jobs = data.get("jobs", [])
|
||||
|
||||
# Increment usage
|
||||
transaction.update(doc_ref, {
|
||||
"usage_count": current_usage + 1,
|
||||
"last_used": datetime.utcnow(),
|
||||
"jobs": firestore.ArrayUnion([{
|
||||
"job_id": job_id,
|
||||
"created_at": datetime.utcnow()
|
||||
}])
|
||||
})
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
|
||||
logger.info(f"Incremented token usage for job {job_id}")
|
||||
except Exception as e:
|
||||
logger.error(f"Error incrementing token usage: {e}")
|
||||
raise
|
||||
|
||||
def list_tokens(self) -> List[Dict[str, Any]]:
|
||||
"""List all tokens (admin only)."""
|
||||
try:
|
||||
docs = self.db.collection(self.tokens_collection).stream()
|
||||
tokens = [doc.to_dict() for doc in docs]
|
||||
logger.info(f"Retrieved {len(tokens)} tokens from Firestore")
|
||||
return tokens
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing tokens: {e}")
|
||||
return []
|
||||
|
||||
@@ -0,0 +1,573 @@
|
||||
"""
|
||||
Client for the remote flacfetch HTTP API service.
|
||||
|
||||
This client communicates with a dedicated flacfetch VM that handles:
|
||||
- BitTorrent downloads from private trackers (RED, OPS)
|
||||
- YouTube downloads
|
||||
- GCS uploads of downloaded files
|
||||
|
||||
The flacfetch service provides:
|
||||
- Full peer connectivity for torrents (not possible in Cloud Run)
|
||||
- Indefinite seeding of completed torrents
|
||||
- Automatic disk cleanup
|
||||
|
||||
Usage:
|
||||
client = get_flacfetch_client()
|
||||
if client:
|
||||
# Search for audio
|
||||
search_result = await client.search("ABBA", "Waterloo")
|
||||
|
||||
# Download the best result
|
||||
download_id = await client.download(
|
||||
search_id=search_result["search_id"],
|
||||
result_index=0,
|
||||
gcs_path="uploads/job123/audio/",
|
||||
)
|
||||
|
||||
# Wait for completion
|
||||
result = await client.wait_for_download(download_id)
|
||||
print(f"Downloaded to: {result['gcs_path']}")
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class FlacfetchServiceError(Exception):
|
||||
"""Error communicating with or returned from flacfetch service."""
|
||||
pass
|
||||
|
||||
|
||||
class FlacfetchClient:
|
||||
"""
|
||||
Client for remote flacfetch HTTP API.
|
||||
|
||||
All endpoints require authentication via X-API-Key header.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
base_url: str,
|
||||
api_key: str,
|
||||
timeout: int = 60,
|
||||
):
|
||||
"""
|
||||
Initialize flacfetch client.
|
||||
|
||||
Args:
|
||||
base_url: Base URL of flacfetch service (e.g., http://10.0.0.5:8080)
|
||||
api_key: API key for authentication
|
||||
timeout: Default request timeout in seconds
|
||||
"""
|
||||
self.base_url = base_url.rstrip('/')
|
||||
self.api_key = api_key
|
||||
self.timeout = timeout
|
||||
logger.info(f"FlacfetchClient initialized with base_url={self.base_url}")
|
||||
|
||||
def _headers(self) -> Dict[str, str]:
|
||||
"""Get request headers with authentication."""
|
||||
return {
|
||||
"X-API-Key": self.api_key,
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
async def health_check(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Check if flacfetch service is healthy.
|
||||
|
||||
Returns:
|
||||
Health status dict with transmission, disk, and provider info
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: If service is unhealthy or unreachable
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.get(
|
||||
f"{self.base_url}/health",
|
||||
headers=self._headers(),
|
||||
timeout=10,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
|
||||
if data.get("status") not in ["healthy", "degraded"]:
|
||||
raise FlacfetchServiceError(f"Service unhealthy: {data}")
|
||||
|
||||
return data
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Cannot reach flacfetch service: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Health check failed: {e.response.status_code}")
|
||||
|
||||
async def search(
|
||||
self,
|
||||
artist: str,
|
||||
title: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Search for audio matching artist and title.
|
||||
|
||||
Args:
|
||||
artist: Artist name
|
||||
title: Track title
|
||||
|
||||
Returns:
|
||||
Search response dict with search_id and results list
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On search failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/search",
|
||||
headers=self._headers(),
|
||||
json={"artist": artist, "title": title},
|
||||
timeout=self.timeout,
|
||||
)
|
||||
|
||||
if resp.status_code == 404:
|
||||
# No results found - return empty results
|
||||
return {
|
||||
"search_id": None,
|
||||
"artist": artist,
|
||||
"title": title,
|
||||
"results": [],
|
||||
"results_count": 0,
|
||||
}
|
||||
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Search request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Search failed: {e.response.status_code} - {e.response.text}")
|
||||
|
||||
async def download(
|
||||
self,
|
||||
search_id: str,
|
||||
result_index: int,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Start downloading an audio file.
|
||||
|
||||
Args:
|
||||
search_id: Search ID from previous search
|
||||
result_index: Index of result to download
|
||||
output_filename: Optional custom filename (without extension)
|
||||
gcs_path: GCS path for upload (e.g., "uploads/job123/audio/")
|
||||
|
||||
Returns:
|
||||
Download ID for tracking progress
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On download start failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
payload = {
|
||||
"search_id": search_id,
|
||||
"result_index": result_index,
|
||||
}
|
||||
if output_filename:
|
||||
payload["output_filename"] = output_filename
|
||||
if gcs_path:
|
||||
payload["upload_to_gcs"] = True
|
||||
payload["gcs_path"] = gcs_path
|
||||
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/download",
|
||||
headers=self._headers(),
|
||||
json=payload,
|
||||
timeout=self.timeout,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
data = resp.json()
|
||||
return data["download_id"]
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Download request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Download start failed: {e.response.status_code} - {e.response.text}")
|
||||
|
||||
async def download_by_id(
|
||||
self,
|
||||
source_name: str,
|
||||
source_id: str,
|
||||
output_filename: Optional[str] = None,
|
||||
target_file: Optional[str] = None,
|
||||
download_url: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Start downloading directly by source ID (no prior search required).
|
||||
|
||||
This is useful when you have stored the source_id from a previous search
|
||||
and want to download later without re-searching.
|
||||
|
||||
Args:
|
||||
source_name: Provider name (RED, OPS, YouTube, Spotify)
|
||||
source_id: Source-specific ID (torrent ID, video ID, track ID)
|
||||
output_filename: Optional custom filename (without extension)
|
||||
target_file: For torrents, specific file to extract
|
||||
download_url: For YouTube/Spotify, direct URL (optional)
|
||||
gcs_path: GCS path for upload (e.g., "uploads/job123/audio/")
|
||||
|
||||
Returns:
|
||||
Download ID for tracking progress
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On download start failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
payload = {
|
||||
"source_name": source_name,
|
||||
"source_id": source_id,
|
||||
}
|
||||
if output_filename:
|
||||
payload["output_filename"] = output_filename
|
||||
if target_file:
|
||||
payload["target_file"] = target_file
|
||||
if download_url:
|
||||
payload["download_url"] = download_url
|
||||
if gcs_path:
|
||||
payload["upload_to_gcs"] = True
|
||||
payload["gcs_path"] = gcs_path
|
||||
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/download-by-id",
|
||||
headers=self._headers(),
|
||||
json=payload,
|
||||
timeout=self.timeout,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
data = resp.json()
|
||||
return data["download_id"]
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Download by ID request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Download by ID failed: {e.response.status_code} - {e.response.text}")
|
||||
|
||||
async def get_download_status(self, download_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Get the current status of a download.
|
||||
|
||||
Args:
|
||||
download_id: Download ID from start_download
|
||||
|
||||
Returns:
|
||||
Status dict with progress, speed, path info
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On status check failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.get(
|
||||
f"{self.base_url}/download/{download_id}/status",
|
||||
headers=self._headers(),
|
||||
timeout=10,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Status request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
if e.response.status_code == 404:
|
||||
raise FlacfetchServiceError(f"Download not found: {download_id}")
|
||||
raise FlacfetchServiceError(f"Status check failed: {e.response.status_code}")
|
||||
|
||||
async def wait_for_download(
|
||||
self,
|
||||
download_id: str,
|
||||
timeout: int = 600,
|
||||
poll_interval: float = 2.0,
|
||||
progress_callback: Optional[callable] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Wait for a download to complete.
|
||||
|
||||
Args:
|
||||
download_id: Download ID to wait for
|
||||
timeout: Maximum wait time in seconds
|
||||
poll_interval: Time between status checks
|
||||
progress_callback: Optional callback(status_dict) for progress updates
|
||||
|
||||
Returns:
|
||||
Final status dict with output_path or gcs_path
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On download failure or timeout
|
||||
"""
|
||||
elapsed = 0.0
|
||||
|
||||
while elapsed < timeout:
|
||||
status = await self.get_download_status(download_id)
|
||||
|
||||
if progress_callback:
|
||||
try:
|
||||
progress_callback(status)
|
||||
except Exception as e:
|
||||
logger.warning(f"Progress callback error: {e}")
|
||||
|
||||
download_status = status.get("status")
|
||||
|
||||
if download_status in ["complete", "seeding"]:
|
||||
logger.info(f"Download {download_id} complete: {status.get('gcs_path') or status.get('output_path')}")
|
||||
return status
|
||||
elif download_status == "failed":
|
||||
error = status.get("error", "Unknown error")
|
||||
raise FlacfetchServiceError(f"Download failed: {error}")
|
||||
elif download_status == "cancelled":
|
||||
raise FlacfetchServiceError("Download was cancelled")
|
||||
|
||||
# Log progress
|
||||
progress = status.get("progress", 0)
|
||||
speed = status.get("download_speed_kbps", 0)
|
||||
peers = status.get("peers", 0)
|
||||
logger.debug(
|
||||
f"Download {download_id}: {progress:.1f}% "
|
||||
f"({speed:.1f} KB/s, {peers} peers)"
|
||||
)
|
||||
|
||||
await asyncio.sleep(poll_interval)
|
||||
elapsed += poll_interval
|
||||
|
||||
raise FlacfetchServiceError(f"Download timed out after {timeout}s")
|
||||
|
||||
async def list_torrents(self) -> Dict[str, Any]:
|
||||
"""
|
||||
List all torrents in Transmission.
|
||||
|
||||
Returns:
|
||||
Torrent list response with torrent info and totals
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.get(
|
||||
f"{self.base_url}/torrents",
|
||||
headers=self._headers(),
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"List torrents failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"List torrents failed: {e.response.status_code}")
|
||||
|
||||
async def delete_torrent(
|
||||
self,
|
||||
torrent_id: int,
|
||||
delete_data: bool = True,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Delete a torrent from Transmission.
|
||||
|
||||
Args:
|
||||
torrent_id: Transmission torrent ID
|
||||
delete_data: Also delete downloaded files
|
||||
|
||||
Returns:
|
||||
Delete response
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.delete(
|
||||
f"{self.base_url}/torrents/{torrent_id}",
|
||||
headers=self._headers(),
|
||||
params={"delete_data": str(delete_data).lower()},
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Delete torrent failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Delete torrent failed: {e.response.status_code}")
|
||||
|
||||
async def cleanup_torrents(
|
||||
self,
|
||||
strategy: str = "oldest",
|
||||
target_free_gb: float = 10.0,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Trigger disk cleanup.
|
||||
|
||||
Args:
|
||||
strategy: Cleanup strategy (oldest, largest, lowest_ratio)
|
||||
target_free_gb: Target free space to achieve
|
||||
|
||||
Returns:
|
||||
Cleanup response with removed count and freed space
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/torrents/cleanup",
|
||||
headers=self._headers(),
|
||||
json={
|
||||
"strategy": strategy,
|
||||
"target_free_gb": target_free_gb,
|
||||
},
|
||||
timeout=60,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Cleanup failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Cleanup failed: {e.response.status_code}")
|
||||
|
||||
# =========================================================================
|
||||
# Cache Management
|
||||
# =========================================================================
|
||||
|
||||
async def clear_search_cache(self, artist: str, title: str) -> bool:
|
||||
"""
|
||||
Clear a specific cached search result by artist and title.
|
||||
|
||||
Args:
|
||||
artist: Artist name
|
||||
title: Track title
|
||||
|
||||
Returns:
|
||||
True if a cache entry was deleted, False if no entry existed
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On request failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.request(
|
||||
"DELETE",
|
||||
f"{self.base_url}/cache/search",
|
||||
headers=self._headers(),
|
||||
json={"artist": artist, "title": title},
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
deleted = data.get("deleted", False)
|
||||
logger.info(
|
||||
f"Cleared flacfetch cache for '{artist}' - '{title}': "
|
||||
f"{'deleted' if deleted else 'no entry found'}"
|
||||
)
|
||||
return deleted
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Clear search cache request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Clear search cache failed: {e.response.status_code} - {e.response.text}"
|
||||
)
|
||||
|
||||
async def clear_all_cache(self) -> int:
|
||||
"""
|
||||
Clear all cached search results.
|
||||
|
||||
Returns:
|
||||
Number of cache entries deleted
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On request failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.delete(
|
||||
f"{self.base_url}/cache",
|
||||
headers=self._headers(),
|
||||
timeout=60,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
deleted_count = data.get("deleted_count", 0)
|
||||
logger.info(f"Cleared all flacfetch cache: {deleted_count} entries deleted")
|
||||
return deleted_count
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Clear all cache request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Clear all cache failed: {e.response.status_code} - {e.response.text}"
|
||||
)
|
||||
|
||||
async def get_cache_stats(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Get statistics about the cache.
|
||||
|
||||
Returns:
|
||||
Dict with count, total_size_bytes, oldest_entry, newest_entry, configured
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On request failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.get(
|
||||
f"{self.base_url}/cache/stats",
|
||||
headers=self._headers(),
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Get cache stats request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Get cache stats failed: {e.response.status_code} - {e.response.text}"
|
||||
)
|
||||
|
||||
|
||||
# Singleton client instance
|
||||
_client: Optional[FlacfetchClient] = None
|
||||
|
||||
|
||||
def get_flacfetch_client() -> Optional[FlacfetchClient]:
|
||||
"""
|
||||
Get the flacfetch client if configured.
|
||||
|
||||
Returns:
|
||||
FlacfetchClient instance if FLACFETCH_API_URL and FLACFETCH_API_KEY are set,
|
||||
None otherwise (indicating local-only mode).
|
||||
"""
|
||||
global _client
|
||||
|
||||
if _client is None:
|
||||
from backend.config import get_settings
|
||||
settings = get_settings()
|
||||
|
||||
if settings.flacfetch_api_url and settings.flacfetch_api_key:
|
||||
_client = FlacfetchClient(
|
||||
base_url=settings.flacfetch_api_url,
|
||||
api_key=settings.flacfetch_api_key,
|
||||
)
|
||||
logger.info("FlacfetchClient initialized for remote service")
|
||||
else:
|
||||
logger.debug("FlacfetchClient not configured (missing URL or API key)")
|
||||
|
||||
return _client
|
||||
|
||||
|
||||
def reset_flacfetch_client():
|
||||
"""Reset the singleton client (for testing)."""
|
||||
global _client
|
||||
_client = None
|
||||
|
||||
@@ -0,0 +1,352 @@
|
||||
"""Native Google Drive API service for cloud backend.
|
||||
|
||||
This service provides Google Drive operations using the native API,
|
||||
for uploading files to public share folders. It handles:
|
||||
- Folder creation and lookup
|
||||
- File uploads with resumable upload support
|
||||
- Uploading to organized folder structure (MP4/, MP4-720p/, CDG/)
|
||||
|
||||
Credentials are loaded from Google Cloud Secret Manager and can be
|
||||
shared with YouTube credentials if scopes include drive.file.
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class GoogleDriveService:
|
||||
"""Google Drive operations using native API."""
|
||||
|
||||
# Secret Manager secret name for Google Drive credentials
|
||||
# Can be same as YouTube if scopes include drive.file
|
||||
GDRIVE_CREDENTIALS_SECRET = "gdrive-oauth-credentials"
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
self._service = None
|
||||
self._credentials_data: Optional[Dict[str, Any]] = None
|
||||
self._loaded = False
|
||||
|
||||
def _load_credentials(self) -> Optional[Dict[str, Any]]:
|
||||
"""Load OAuth credentials from Secret Manager."""
|
||||
if self._loaded:
|
||||
return self._credentials_data
|
||||
|
||||
try:
|
||||
creds_json = self.settings.get_secret(self.GDRIVE_CREDENTIALS_SECRET)
|
||||
|
||||
if not creds_json:
|
||||
# Try falling back to YouTube credentials (may have drive scope)
|
||||
logger.info(
|
||||
"Google Drive credentials not found, trying YouTube credentials"
|
||||
)
|
||||
creds_json = self.settings.get_secret("youtube-oauth-credentials")
|
||||
|
||||
if not creds_json:
|
||||
logger.warning("Google Drive credentials not found in Secret Manager")
|
||||
self._loaded = True
|
||||
return None
|
||||
|
||||
self._credentials_data = json.loads(creds_json)
|
||||
|
||||
# Validate required fields
|
||||
required_fields = ["refresh_token", "client_id", "client_secret"]
|
||||
missing = [f for f in required_fields if not self._credentials_data.get(f)]
|
||||
|
||||
if missing:
|
||||
logger.error(f"Google Drive credentials missing required fields: {missing}")
|
||||
self._credentials_data = None
|
||||
self._loaded = True
|
||||
return None
|
||||
|
||||
logger.info("Google Drive credentials loaded successfully from Secret Manager")
|
||||
self._loaded = True
|
||||
return self._credentials_data
|
||||
|
||||
except json.JSONDecodeError as e:
|
||||
logger.error(f"Failed to parse Google Drive credentials JSON: {e}")
|
||||
self._loaded = True
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load Google Drive credentials: {e}")
|
||||
self._loaded = True
|
||||
return None
|
||||
|
||||
@property
|
||||
def is_configured(self) -> bool:
|
||||
"""Check if Google Drive credentials are available."""
|
||||
creds = self._load_credentials()
|
||||
return creds is not None
|
||||
|
||||
@property
|
||||
def service(self):
|
||||
"""Get or create Google Drive service."""
|
||||
if self._service is None:
|
||||
# Import here to avoid import errors if google packages not installed
|
||||
try:
|
||||
from google.oauth2.credentials import Credentials
|
||||
from googleapiclient.discovery import build
|
||||
except ImportError:
|
||||
raise ImportError(
|
||||
"google-api-python-client and google-auth packages are required. "
|
||||
"Install them with: pip install google-api-python-client google-auth"
|
||||
)
|
||||
|
||||
creds_data = self._load_credentials()
|
||||
if not creds_data:
|
||||
raise RuntimeError(
|
||||
"Google Drive credentials not configured in Secret Manager"
|
||||
)
|
||||
|
||||
# Create credentials object
|
||||
credentials = Credentials(
|
||||
token=creds_data.get("token"),
|
||||
refresh_token=creds_data.get("refresh_token"),
|
||||
token_uri=creds_data.get(
|
||||
"token_uri", "https://oauth2.googleapis.com/token"
|
||||
),
|
||||
client_id=creds_data.get("client_id"),
|
||||
client_secret=creds_data.get("client_secret"),
|
||||
scopes=creds_data.get(
|
||||
"scopes", ["https://www.googleapis.com/auth/drive.file"]
|
||||
),
|
||||
)
|
||||
|
||||
self._service = build("drive", "v3", credentials=credentials)
|
||||
logger.info("Google Drive service initialized successfully")
|
||||
|
||||
return self._service
|
||||
|
||||
def get_or_create_folder(self, parent_id: str, folder_name: str) -> str:
|
||||
"""
|
||||
Get existing folder or create new one, return folder ID.
|
||||
|
||||
Args:
|
||||
parent_id: Parent folder ID
|
||||
folder_name: Name of folder to find or create
|
||||
|
||||
Returns:
|
||||
Folder ID
|
||||
"""
|
||||
logger.info(f"Looking for folder '{folder_name}' in parent {parent_id}")
|
||||
|
||||
# Search for existing folder
|
||||
# Escape single quotes in folder name for Google Drive API query syntax
|
||||
escaped_folder_name = folder_name.replace("'", "\\'")
|
||||
query = (
|
||||
f"name='{escaped_folder_name}' and '{parent_id}' in parents "
|
||||
f"and mimeType='application/vnd.google-apps.folder' and trashed=false"
|
||||
)
|
||||
results = self.service.files().list(q=query, fields="files(id, name)").execute()
|
||||
|
||||
if results.get("files"):
|
||||
folder_id = results["files"][0]["id"]
|
||||
logger.info(f"Found existing folder '{folder_name}': {folder_id}")
|
||||
return folder_id
|
||||
|
||||
# Create folder
|
||||
logger.info(f"Creating new folder '{folder_name}'")
|
||||
metadata = {
|
||||
"name": folder_name,
|
||||
"mimeType": "application/vnd.google-apps.folder",
|
||||
"parents": [parent_id],
|
||||
}
|
||||
folder = self.service.files().create(body=metadata, fields="id").execute()
|
||||
folder_id = folder["id"]
|
||||
logger.info(f"Created folder '{folder_name}': {folder_id}")
|
||||
return folder_id
|
||||
|
||||
def upload_file(
|
||||
self,
|
||||
local_path: str,
|
||||
parent_id: str,
|
||||
filename: str,
|
||||
replace_existing: bool = True,
|
||||
) -> str:
|
||||
"""
|
||||
Upload a file to a specific Drive folder.
|
||||
|
||||
Args:
|
||||
local_path: Local file path
|
||||
parent_id: Parent folder ID in Google Drive
|
||||
filename: Name for the file in Drive
|
||||
replace_existing: If True, delete existing file with same name first
|
||||
|
||||
Returns:
|
||||
File ID of uploaded file
|
||||
"""
|
||||
from googleapiclient.http import MediaFileUpload
|
||||
|
||||
file_size = os.path.getsize(local_path)
|
||||
logger.info(
|
||||
f"Uploading {local_path} ({file_size / 1024 / 1024:.1f} MB) "
|
||||
f"as '{filename}' to folder {parent_id}"
|
||||
)
|
||||
|
||||
# Determine MIME type
|
||||
ext = os.path.splitext(local_path)[1].lower()
|
||||
mime_types = {
|
||||
".mp4": "video/mp4",
|
||||
".mkv": "video/x-matroska",
|
||||
".zip": "application/zip",
|
||||
".flac": "audio/flac",
|
||||
".mp3": "audio/mpeg",
|
||||
".wav": "audio/wav",
|
||||
".png": "image/png",
|
||||
".jpg": "image/jpeg",
|
||||
".jpeg": "image/jpeg",
|
||||
}
|
||||
mime_type = mime_types.get(ext, "application/octet-stream")
|
||||
|
||||
# Check for existing file with same name
|
||||
if replace_existing:
|
||||
# Escape single quotes in filename for Google Drive API query syntax
|
||||
escaped_filename = filename.replace("'", "\\'")
|
||||
query = (
|
||||
f"name='{escaped_filename}' and '{parent_id}' in parents and trashed=false"
|
||||
)
|
||||
results = self.service.files().list(q=query, fields="files(id)").execute()
|
||||
for existing_file in results.get("files", []):
|
||||
logger.info(f"Deleting existing file: {existing_file['id']}")
|
||||
self.service.files().delete(fileId=existing_file["id"]).execute()
|
||||
|
||||
# Upload file
|
||||
metadata = {"name": filename, "parents": [parent_id]}
|
||||
media = MediaFileUpload(local_path, mimetype=mime_type, resumable=True)
|
||||
|
||||
file_result = (
|
||||
self.service.files().create(body=metadata, media_body=media, fields="id").execute()
|
||||
)
|
||||
file_id = file_result["id"]
|
||||
logger.info(f"Successfully uploaded '{filename}': {file_id}")
|
||||
return file_id
|
||||
|
||||
def upload_to_public_share(
|
||||
self,
|
||||
root_folder_id: str,
|
||||
brand_code: str,
|
||||
base_name: str,
|
||||
output_files: dict,
|
||||
) -> Dict[str, str]:
|
||||
"""
|
||||
Upload final files to public share folder structure.
|
||||
|
||||
Creates/uses subfolders:
|
||||
- MP4/{brand_code} - {base_name}.mp4 (lossy 4k)
|
||||
- MP4-720p/{brand_code} - {base_name}.mp4
|
||||
- CDG/{brand_code} - {base_name}.zip
|
||||
|
||||
Args:
|
||||
root_folder_id: Google Drive folder ID for public share root
|
||||
brand_code: Brand code (e.g., "NOMAD-1163")
|
||||
base_name: Base filename (e.g., "Artist - Title")
|
||||
output_files: Dictionary with output file paths:
|
||||
- final_karaoke_lossy_mp4: 4K lossy MP4
|
||||
- final_karaoke_lossy_720p_mp4: 720p lossy MP4
|
||||
- final_karaoke_cdg_zip: CDG package ZIP
|
||||
|
||||
Returns:
|
||||
Dictionary mapping category to uploaded file ID
|
||||
"""
|
||||
filename_base = f"{brand_code} - {base_name}"
|
||||
uploaded_files = {}
|
||||
|
||||
logger.info(
|
||||
f"Uploading public share files to Google Drive folder {root_folder_id}"
|
||||
)
|
||||
logger.info(f"Filename base: {filename_base}")
|
||||
|
||||
# Upload lossy 4k to MP4/
|
||||
lossy_mp4_path = output_files.get("final_karaoke_lossy_mp4")
|
||||
if lossy_mp4_path and os.path.exists(lossy_mp4_path):
|
||||
mp4_folder_id = self.get_or_create_folder(root_folder_id, "MP4")
|
||||
file_id = self.upload_file(
|
||||
lossy_mp4_path,
|
||||
mp4_folder_id,
|
||||
f"{filename_base}.mp4",
|
||||
)
|
||||
uploaded_files["mp4"] = file_id
|
||||
logger.info(f"Uploaded 4K MP4 to MP4/ folder")
|
||||
|
||||
# Upload 720p to MP4-720p/
|
||||
mp4_720p_path = output_files.get("final_karaoke_lossy_720p_mp4")
|
||||
if mp4_720p_path and os.path.exists(mp4_720p_path):
|
||||
mp4_720_folder_id = self.get_or_create_folder(root_folder_id, "MP4-720p")
|
||||
file_id = self.upload_file(
|
||||
mp4_720p_path,
|
||||
mp4_720_folder_id,
|
||||
f"{filename_base}.mp4",
|
||||
)
|
||||
uploaded_files["mp4_720p"] = file_id
|
||||
logger.info(f"Uploaded 720p MP4 to MP4-720p/ folder")
|
||||
|
||||
# Upload CDG ZIP to CDG/
|
||||
cdg_zip_path = output_files.get("final_karaoke_cdg_zip")
|
||||
if cdg_zip_path and os.path.exists(cdg_zip_path):
|
||||
cdg_folder_id = self.get_or_create_folder(root_folder_id, "CDG")
|
||||
file_id = self.upload_file(
|
||||
cdg_zip_path,
|
||||
cdg_folder_id,
|
||||
f"{filename_base}.zip",
|
||||
)
|
||||
uploaded_files["cdg"] = file_id
|
||||
logger.info(f"Uploaded CDG ZIP to CDG/ folder")
|
||||
|
||||
logger.info(f"Public share upload complete: {len(uploaded_files)} files uploaded")
|
||||
return uploaded_files
|
||||
|
||||
def delete_file(self, file_id: str) -> bool:
|
||||
"""
|
||||
Delete a file from Google Drive.
|
||||
|
||||
Args:
|
||||
file_id: Google Drive file ID to delete
|
||||
|
||||
Returns:
|
||||
True if deleted successfully, False otherwise
|
||||
"""
|
||||
logger.info(f"Deleting Google Drive file: {file_id}")
|
||||
|
||||
try:
|
||||
self.service.files().delete(fileId=file_id).execute()
|
||||
logger.info(f"Successfully deleted file: {file_id}")
|
||||
return True
|
||||
except Exception as e:
|
||||
# Check if it's a 404 (already deleted)
|
||||
if hasattr(e, 'resp') and e.resp.status == 404:
|
||||
logger.warning(f"File not found (already deleted?): {file_id}")
|
||||
return True
|
||||
logger.error(f"Failed to delete Google Drive file: {e}")
|
||||
return False
|
||||
|
||||
def delete_files(self, file_ids: list[str]) -> dict[str, bool]:
|
||||
"""
|
||||
Delete multiple files from Google Drive.
|
||||
|
||||
Args:
|
||||
file_ids: List of Google Drive file IDs to delete
|
||||
|
||||
Returns:
|
||||
Dictionary mapping file_id to success status
|
||||
"""
|
||||
results = {}
|
||||
for file_id in file_ids:
|
||||
results[file_id] = self.delete_file(file_id)
|
||||
return results
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_gdrive_service: Optional[GoogleDriveService] = None
|
||||
|
||||
|
||||
def get_gdrive_service() -> GoogleDriveService:
|
||||
"""Get the singleton Google Drive service instance."""
|
||||
global _gdrive_service
|
||||
if _gdrive_service is None:
|
||||
_gdrive_service = GoogleDriveService()
|
||||
return _gdrive_service
|
||||
@@ -0,0 +1,258 @@
|
||||
"""
|
||||
Job-aware logging utilities.
|
||||
|
||||
This module provides a context manager and logging handler that captures
|
||||
Python logging output and forwards it to Firestore for real-time streaming
|
||||
to the CLI.
|
||||
|
||||
Usage:
|
||||
with JobLogContext(job_id, worker="review") as log_context:
|
||||
# Any logging within this block will be captured and sent to Firestore
|
||||
logger.info("Processing...") # This will be stored and streamed to CLI
|
||||
"""
|
||||
import logging
|
||||
import threading
|
||||
from typing import Optional
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime
|
||||
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
|
||||
|
||||
class JobLogHandler(logging.Handler):
|
||||
"""
|
||||
A logging handler that forwards log records to Firestore for job tracking.
|
||||
|
||||
This enables real-time log streaming to the CLI during long-running operations
|
||||
like lyrics correction, preview generation, etc.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
job_id: str,
|
||||
worker: str = "review",
|
||||
firestore: Optional[FirestoreService] = None,
|
||||
min_level: int = logging.INFO,
|
||||
batch_size: int = 1, # Send immediately for real-time experience
|
||||
):
|
||||
"""
|
||||
Initialize the job log handler.
|
||||
|
||||
Args:
|
||||
job_id: The job ID to associate logs with
|
||||
worker: Worker name (e.g., "review", "add-lyrics", "preview")
|
||||
firestore: FirestoreService instance (creates one if not provided)
|
||||
min_level: Minimum log level to capture (default INFO)
|
||||
batch_size: Number of logs to batch before sending (1 = immediate)
|
||||
"""
|
||||
super().__init__()
|
||||
self.job_id = job_id
|
||||
self.worker = worker
|
||||
self.firestore = firestore or FirestoreService()
|
||||
self.min_level = min_level
|
||||
self.batch_size = batch_size
|
||||
self._log_buffer = []
|
||||
self._lock = threading.Lock()
|
||||
|
||||
# Set the handler's level
|
||||
self.setLevel(min_level)
|
||||
|
||||
# Create a formatter that extracts the useful parts
|
||||
self.setFormatter(logging.Formatter('%(message)s'))
|
||||
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
"""
|
||||
Process a log record and send it to Firestore.
|
||||
|
||||
This method is called by the logging framework for each log message.
|
||||
"""
|
||||
try:
|
||||
# Skip if below minimum level
|
||||
if record.levelno < self.min_level:
|
||||
return
|
||||
|
||||
# Format the message
|
||||
message = self.format(record)
|
||||
|
||||
# Truncate very long messages
|
||||
if len(message) > 1000:
|
||||
message = message[:997] + "..."
|
||||
|
||||
# Create log entry
|
||||
log_entry = {
|
||||
'timestamp': datetime.utcnow().isoformat() + 'Z',
|
||||
'level': record.levelname,
|
||||
'worker': self.worker,
|
||||
'message': message,
|
||||
}
|
||||
|
||||
# Add to buffer
|
||||
with self._lock:
|
||||
self._log_buffer.append(log_entry)
|
||||
|
||||
# Flush if batch size reached
|
||||
if len(self._log_buffer) >= self.batch_size:
|
||||
self._flush_buffer()
|
||||
|
||||
except Exception:
|
||||
# Never let logging errors propagate
|
||||
self.handleError(record)
|
||||
|
||||
def _flush_buffer(self) -> None:
|
||||
"""Flush buffered logs to Firestore."""
|
||||
if not self._log_buffer:
|
||||
return
|
||||
|
||||
try:
|
||||
for log_entry in self._log_buffer:
|
||||
self.firestore.append_worker_log(self.job_id, log_entry)
|
||||
self._log_buffer = []
|
||||
except Exception:
|
||||
# If Firestore fails, just clear buffer and continue
|
||||
self._log_buffer = []
|
||||
|
||||
def flush(self) -> None:
|
||||
"""Flush any remaining buffered logs."""
|
||||
with self._lock:
|
||||
self._flush_buffer()
|
||||
|
||||
def close(self) -> None:
|
||||
"""Clean up the handler."""
|
||||
self.flush()
|
||||
super().close()
|
||||
|
||||
|
||||
@contextmanager
|
||||
def job_log_context(
|
||||
job_id: str,
|
||||
worker: str = "review",
|
||||
logger_names: Optional[list] = None,
|
||||
min_level: int = logging.INFO,
|
||||
):
|
||||
"""
|
||||
Context manager that captures logs and forwards them to Firestore.
|
||||
|
||||
This allows the CLI to see real-time logs from operations like
|
||||
add-lyrics, preview generation, etc.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to associate logs with
|
||||
worker: Worker name for log categorization
|
||||
logger_names: List of logger names to capture (None = capture common loggers)
|
||||
min_level: Minimum log level to capture
|
||||
|
||||
Usage:
|
||||
with job_log_context(job_id, worker="add-lyrics"):
|
||||
# All logging in this block will be captured
|
||||
correction_operations.add_lyrics_source(...)
|
||||
"""
|
||||
# Default loggers to capture - these cover most of our processing
|
||||
# Note: We intentionally DON'T include the root logger ('') to avoid
|
||||
# duplicate log entries since logs propagate up the logger hierarchy.
|
||||
# Only capture at the top-level package loggers.
|
||||
if logger_names is None:
|
||||
logger_names = [
|
||||
'backend.api.routes.review',
|
||||
'lyrics_transcriber',
|
||||
'karaoke_gen',
|
||||
]
|
||||
|
||||
# Create the handler
|
||||
handler = JobLogHandler(
|
||||
job_id=job_id,
|
||||
worker=worker,
|
||||
min_level=min_level,
|
||||
)
|
||||
|
||||
# Add handler to all specified loggers
|
||||
loggers = []
|
||||
for name in logger_names:
|
||||
log = logging.getLogger(name) if name else logging.getLogger()
|
||||
log.addHandler(handler)
|
||||
loggers.append(log)
|
||||
|
||||
try:
|
||||
# Log start
|
||||
handler.emit(logging.LogRecord(
|
||||
name='job_log_context',
|
||||
level=logging.INFO,
|
||||
pathname='',
|
||||
lineno=0,
|
||||
msg=f"🚀 Starting {worker} operation for job {job_id}",
|
||||
args=(),
|
||||
exc_info=None,
|
||||
))
|
||||
|
||||
yield handler
|
||||
|
||||
finally:
|
||||
# Log completion
|
||||
handler.emit(logging.LogRecord(
|
||||
name='job_log_context',
|
||||
level=logging.INFO,
|
||||
pathname='',
|
||||
lineno=0,
|
||||
msg=f"✅ Completed {worker} operation for job {job_id}",
|
||||
args=(),
|
||||
exc_info=None,
|
||||
))
|
||||
|
||||
# Flush remaining logs
|
||||
handler.flush()
|
||||
|
||||
# Remove handler from all loggers
|
||||
for log in loggers:
|
||||
log.removeHandler(handler)
|
||||
|
||||
# Close handler
|
||||
handler.close()
|
||||
|
||||
|
||||
class JobLogger:
|
||||
"""
|
||||
A convenience class for direct logging to a job's log stream.
|
||||
|
||||
Use this when you want to log specific messages without capturing
|
||||
all logging output.
|
||||
"""
|
||||
|
||||
def __init__(self, job_id: str, worker: str = "review"):
|
||||
"""
|
||||
Initialize job logger.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to log to
|
||||
worker: Worker name for log categorization
|
||||
"""
|
||||
self.job_id = job_id
|
||||
self.worker = worker
|
||||
self.firestore = FirestoreService()
|
||||
|
||||
def _log(self, level: str, message: str) -> None:
|
||||
"""Internal logging method."""
|
||||
log_entry = {
|
||||
'timestamp': datetime.utcnow().isoformat() + 'Z',
|
||||
'level': level,
|
||||
'worker': self.worker,
|
||||
'message': message[:1000],
|
||||
}
|
||||
try:
|
||||
self.firestore.append_worker_log(self.job_id, log_entry)
|
||||
except Exception:
|
||||
pass # Don't let logging errors propagate
|
||||
|
||||
def info(self, message: str) -> None:
|
||||
"""Log an INFO message."""
|
||||
self._log('INFO', message)
|
||||
|
||||
def warning(self, message: str) -> None:
|
||||
"""Log a WARNING message."""
|
||||
self._log('WARNING', message)
|
||||
|
||||
def error(self, message: str) -> None:
|
||||
"""Log an ERROR message."""
|
||||
self._log('ERROR', message)
|
||||
|
||||
def debug(self, message: str) -> None:
|
||||
"""Log a DEBUG message."""
|
||||
self._log('DEBUG', message)
|
||||
@@ -0,0 +1,704 @@
|
||||
"""
|
||||
Job management and queue operations.
|
||||
|
||||
This module handles the complete job lifecycle including:
|
||||
- Job creation and initialization
|
||||
- State transitions and validation
|
||||
- Worker coordination (parallel audio + lyrics processing)
|
||||
- Progress tracking and timeline events
|
||||
- Error handling and retries
|
||||
"""
|
||||
import logging
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from typing import Optional, Dict, Any, List
|
||||
|
||||
from backend.config import settings
|
||||
from backend.models.job import Job, JobStatus, JobCreate, STATE_TRANSITIONS
|
||||
from backend.models.worker_log import WorkerLogEntry
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class JobManager:
|
||||
"""Manager for job lifecycle and state."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize job manager with required services."""
|
||||
self.firestore = FirestoreService()
|
||||
self.storage = StorageService()
|
||||
|
||||
def create_job(self, job_create: JobCreate) -> Job:
|
||||
"""
|
||||
Create a new job with initial state PENDING.
|
||||
|
||||
Jobs start in PENDING state and transition to DOWNLOADING
|
||||
when a worker picks them up.
|
||||
"""
|
||||
job_id = str(uuid.uuid4())[:8]
|
||||
|
||||
now = datetime.utcnow()
|
||||
job = Job(
|
||||
job_id=job_id,
|
||||
status=JobStatus.PENDING, # New state machine starts with PENDING
|
||||
progress=0,
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
url=str(job_create.url) if job_create.url else None,
|
||||
artist=job_create.artist,
|
||||
title=job_create.title,
|
||||
# User preferences
|
||||
enable_cdg=job_create.enable_cdg,
|
||||
enable_txt=job_create.enable_txt,
|
||||
enable_youtube_upload=job_create.enable_youtube_upload,
|
||||
youtube_description=job_create.youtube_description,
|
||||
youtube_description_template=job_create.youtube_description_template, # video_worker reads this
|
||||
webhook_url=job_create.webhook_url,
|
||||
user_email=job_create.user_email,
|
||||
# Distribution settings
|
||||
brand_prefix=job_create.brand_prefix,
|
||||
discord_webhook_url=job_create.discord_webhook_url,
|
||||
dropbox_path=job_create.dropbox_path,
|
||||
gdrive_folder_id=job_create.gdrive_folder_id,
|
||||
# Theme configuration
|
||||
theme_id=job_create.theme_id,
|
||||
color_overrides=job_create.color_overrides,
|
||||
style_params_gcs_path=job_create.style_params_gcs_path,
|
||||
style_assets=job_create.style_assets,
|
||||
# Two-phase workflow (Batch 6)
|
||||
prep_only=job_create.prep_only,
|
||||
finalise_only=job_create.finalise_only,
|
||||
keep_brand_code=job_create.keep_brand_code,
|
||||
# Request metadata (for tracking and filtering)
|
||||
request_metadata=job_create.request_metadata,
|
||||
)
|
||||
|
||||
self.firestore.create_job(job)
|
||||
logger.info(f"Created new job {job_id} with status PENDING")
|
||||
|
||||
return job
|
||||
|
||||
def get_job(self, job_id: str) -> Optional[Job]:
|
||||
"""Get a job by ID."""
|
||||
return self.firestore.get_job(job_id)
|
||||
|
||||
def update_job_status(
|
||||
self,
|
||||
job_id: str,
|
||||
status: JobStatus,
|
||||
progress: Optional[int] = None,
|
||||
message: Optional[str] = None,
|
||||
**kwargs
|
||||
) -> None:
|
||||
"""Update job status with timeline tracking."""
|
||||
self.firestore.update_job_status(
|
||||
job_id=job_id,
|
||||
status=status,
|
||||
progress=progress,
|
||||
message=message,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
def update_job(self, job_id: str, updates: Dict[str, Any]) -> None:
|
||||
"""Update job with arbitrary fields."""
|
||||
self.firestore.update_job(job_id, updates)
|
||||
|
||||
def list_jobs(
|
||||
self,
|
||||
status: Optional[JobStatus] = None,
|
||||
environment: Optional[str] = None,
|
||||
client_id: Optional[str] = None,
|
||||
created_after: Optional[datetime] = None,
|
||||
created_before: Optional[datetime] = None,
|
||||
user_email: Optional[str] = None,
|
||||
limit: int = 100
|
||||
) -> List[Job]:
|
||||
"""
|
||||
List jobs with optional filtering.
|
||||
|
||||
Args:
|
||||
status: Filter by job status
|
||||
environment: Filter by request_metadata.environment (test/production/development)
|
||||
client_id: Filter by request_metadata.client_id (customer identifier)
|
||||
created_after: Filter jobs created after this datetime
|
||||
created_before: Filter jobs created before this datetime
|
||||
user_email: Filter by user_email (job owner)
|
||||
limit: Maximum number of jobs to return
|
||||
|
||||
Returns:
|
||||
List of Job objects matching filters
|
||||
"""
|
||||
return self.firestore.list_jobs(
|
||||
status=status,
|
||||
environment=environment,
|
||||
client_id=client_id,
|
||||
created_after=created_after,
|
||||
created_before=created_before,
|
||||
user_email=user_email,
|
||||
limit=limit
|
||||
)
|
||||
|
||||
def delete_jobs_by_filter(
|
||||
self,
|
||||
environment: Optional[str] = None,
|
||||
client_id: Optional[str] = None,
|
||||
status: Optional[JobStatus] = None,
|
||||
created_before: Optional[datetime] = None,
|
||||
delete_files: bool = True
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Delete multiple jobs matching filter criteria.
|
||||
|
||||
CAUTION: This is a destructive operation. Use carefully.
|
||||
|
||||
Args:
|
||||
environment: Delete jobs with this environment (e.g., "test")
|
||||
client_id: Delete jobs from this client
|
||||
status: Delete jobs with this status
|
||||
created_before: Delete jobs created before this datetime
|
||||
delete_files: Also delete GCS files (default True)
|
||||
|
||||
Returns:
|
||||
Dict with deletion statistics
|
||||
"""
|
||||
# First get matching jobs to delete their files
|
||||
if delete_files:
|
||||
jobs = self.firestore.list_jobs(
|
||||
status=status,
|
||||
environment=environment,
|
||||
client_id=client_id,
|
||||
created_before=created_before,
|
||||
limit=10000 # High limit for deletion
|
||||
)
|
||||
|
||||
files_deleted = 0
|
||||
for job in jobs:
|
||||
# Delete files from various locations
|
||||
try:
|
||||
# Delete uploads folder
|
||||
self.storage.delete_folder(f"uploads/{job.job_id}/")
|
||||
# Delete jobs folder
|
||||
self.storage.delete_folder(f"jobs/{job.job_id}/")
|
||||
files_deleted += 1
|
||||
except Exception as e:
|
||||
logger.warning(f"Error deleting files for job {job.job_id}: {e}")
|
||||
|
||||
# Delete the jobs from Firestore
|
||||
deleted_count = self.firestore.delete_jobs_by_filter(
|
||||
environment=environment,
|
||||
client_id=client_id,
|
||||
status=status,
|
||||
created_before=created_before
|
||||
)
|
||||
|
||||
return {
|
||||
'jobs_deleted': deleted_count,
|
||||
'files_deleted': files_deleted if delete_files else 0
|
||||
}
|
||||
|
||||
def mark_job_error(self, job_id: str, error_message: str) -> None:
|
||||
"""Mark a job as errored."""
|
||||
self.firestore.update_job_status(
|
||||
job_id=job_id,
|
||||
status=JobStatus.ERROR,
|
||||
progress=0,
|
||||
message=error_message,
|
||||
error_message=error_message
|
||||
)
|
||||
logger.error(f"Job {job_id} marked as error: {error_message}")
|
||||
|
||||
def get_output_urls(self, job_id: str) -> Dict[str, str]:
|
||||
"""Generate signed URLs for job output files."""
|
||||
job = self.get_job(job_id)
|
||||
if not job or not job.output_files:
|
||||
return {}
|
||||
|
||||
urls = {}
|
||||
for file_type, gcs_path in job.output_files.items():
|
||||
try:
|
||||
urls[file_type] = self.storage.generate_signed_url(gcs_path, expiration_minutes=120)
|
||||
except Exception as e:
|
||||
logger.error(f"Error generating URL for {file_type}: {e}")
|
||||
|
||||
return urls
|
||||
|
||||
def delete_job(self, job_id: str, delete_files: bool = True) -> None:
|
||||
"""Delete a job, its files, and its logs subcollection."""
|
||||
if delete_files:
|
||||
job = self.get_job(job_id)
|
||||
if job and job.output_files:
|
||||
for gcs_path in job.output_files.values():
|
||||
try:
|
||||
self.storage.delete_file(gcs_path)
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting file {gcs_path}: {e}")
|
||||
|
||||
# Delete logs subcollection first (must be done before deleting parent doc)
|
||||
try:
|
||||
deleted_logs = self.firestore.delete_logs_subcollection(job_id)
|
||||
if deleted_logs > 0:
|
||||
logger.info(f"Deleted {deleted_logs} log entries for job {job_id}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Error deleting logs subcollection for job {job_id}: {e}")
|
||||
|
||||
self.firestore.delete_job(job_id)
|
||||
logger.info(f"Deleted job {job_id}")
|
||||
|
||||
def validate_state_transition(self, job_id: str, new_status: JobStatus) -> bool:
|
||||
"""
|
||||
Validate that a state transition is legal.
|
||||
|
||||
Returns:
|
||||
True if transition is valid, False otherwise
|
||||
"""
|
||||
job = self.get_job(job_id)
|
||||
if not job:
|
||||
logger.error(f"Job {job_id} not found")
|
||||
return False
|
||||
|
||||
current_status = job.status
|
||||
valid_transitions = STATE_TRANSITIONS.get(current_status, [])
|
||||
|
||||
if new_status not in valid_transitions:
|
||||
logger.error(
|
||||
f"Invalid state transition for job {job_id}: "
|
||||
f"{current_status} -> {new_status}. "
|
||||
f"Valid transitions: {valid_transitions}"
|
||||
)
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
def transition_to_state(
|
||||
self,
|
||||
job_id: str,
|
||||
new_status: JobStatus,
|
||||
progress: Optional[int] = None,
|
||||
message: Optional[str] = None,
|
||||
state_data_updates: Optional[Dict[str, Any]] = None
|
||||
) -> bool:
|
||||
"""
|
||||
Transition job to new state with validation.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
new_status: Target state
|
||||
progress: Progress percentage (0-100)
|
||||
message: Timeline message
|
||||
state_data_updates: Updates to state_data field
|
||||
|
||||
Returns:
|
||||
True if transition succeeded, False otherwise
|
||||
"""
|
||||
if not self.validate_state_transition(job_id, new_status):
|
||||
return False
|
||||
|
||||
updates = {
|
||||
'status': new_status,
|
||||
'updated_at': datetime.utcnow()
|
||||
}
|
||||
|
||||
if progress is not None:
|
||||
updates['progress'] = progress
|
||||
|
||||
# Generate review token when entering AWAITING_REVIEW state
|
||||
if new_status == JobStatus.AWAITING_REVIEW:
|
||||
from backend.api.dependencies import generate_review_token, get_review_token_expiry
|
||||
review_token = generate_review_token()
|
||||
review_token_expires = get_review_token_expiry(hours=48) # 48 hour expiry
|
||||
updates['review_token'] = review_token
|
||||
updates['review_token_expires_at'] = review_token_expires
|
||||
logger.info(f"Generated review token for job {job_id}, expires in 48 hours")
|
||||
|
||||
# Generate instrumental token when entering AWAITING_INSTRUMENTAL_SELECTION state
|
||||
if new_status == JobStatus.AWAITING_INSTRUMENTAL_SELECTION:
|
||||
from backend.api.dependencies import generate_review_token, get_review_token_expiry
|
||||
instrumental_token = generate_review_token() # Reuse same token generator
|
||||
instrumental_token_expires = get_review_token_expiry(hours=48) # 48 hour expiry
|
||||
updates['instrumental_token'] = instrumental_token
|
||||
updates['instrumental_token_expires_at'] = instrumental_token_expires
|
||||
logger.info(f"Generated instrumental token for job {job_id}, expires in 48 hours")
|
||||
|
||||
# If we have state_data_updates, merge them with existing state_data
|
||||
merged_state_data = None
|
||||
if state_data_updates:
|
||||
job = self.get_job(job_id)
|
||||
if job:
|
||||
merged_state_data = {**job.state_data, **state_data_updates}
|
||||
|
||||
# Update job status (includes timeline event), passing state_data if present
|
||||
if merged_state_data is not None:
|
||||
self.update_job_status(
|
||||
job_id=job_id,
|
||||
status=new_status,
|
||||
progress=progress,
|
||||
message=message,
|
||||
state_data=merged_state_data
|
||||
)
|
||||
else:
|
||||
self.update_job_status(
|
||||
job_id=job_id,
|
||||
status=new_status,
|
||||
progress=progress,
|
||||
message=message
|
||||
)
|
||||
|
||||
# Apply review token update separately if generated
|
||||
if new_status == JobStatus.AWAITING_REVIEW and 'review_token' in updates:
|
||||
self.firestore.update_job(job_id, {
|
||||
'review_token': updates['review_token'],
|
||||
'review_token_expires_at': updates['review_token_expires_at']
|
||||
})
|
||||
|
||||
# Apply instrumental token update separately if generated
|
||||
if new_status == JobStatus.AWAITING_INSTRUMENTAL_SELECTION and 'instrumental_token' in updates:
|
||||
self.firestore.update_job(job_id, {
|
||||
'instrumental_token': updates['instrumental_token'],
|
||||
'instrumental_token_expires_at': updates['instrumental_token_expires_at']
|
||||
})
|
||||
|
||||
logger.info(f"Job {job_id} transitioned to {new_status}")
|
||||
return True
|
||||
|
||||
def update_state_data(self, job_id: str, key: str, value: Any) -> None:
|
||||
"""
|
||||
Update a specific key in the job's state_data field.
|
||||
|
||||
This is used by workers to store stage-specific metadata.
|
||||
"""
|
||||
job = self.get_job(job_id)
|
||||
if not job:
|
||||
logger.error(f"Job {job_id} not found")
|
||||
return
|
||||
|
||||
state_data = job.state_data.copy()
|
||||
state_data[key] = value
|
||||
|
||||
self.update_job(job_id, {'state_data': state_data})
|
||||
logger.debug(f"Job {job_id} state_data updated: {key} = {value}")
|
||||
|
||||
def fail_job(self, job_id: str, error_message: str, error_details: Optional[Dict[str, Any]] = None) -> bool:
|
||||
"""
|
||||
Mark a job as failed with error information.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
error_message: Human-readable error message
|
||||
error_details: Optional structured error details
|
||||
|
||||
Returns:
|
||||
True if successful
|
||||
"""
|
||||
try:
|
||||
# Update error fields
|
||||
self.update_job(job_id, {
|
||||
'error_message': error_message,
|
||||
'error_details': error_details or {}
|
||||
})
|
||||
|
||||
# Use update_job_status which handles timeline
|
||||
self.update_job_status(
|
||||
job_id=job_id,
|
||||
status=JobStatus.FAILED,
|
||||
message=error_message
|
||||
)
|
||||
|
||||
logger.error(f"Job {job_id} failed: {error_message}")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Error marking job {job_id} as failed: {e}")
|
||||
return False
|
||||
|
||||
def update_file_url(self, job_id: str, category: str, file_type: str, url: str) -> None:
|
||||
"""
|
||||
Update a file URL in the job's file_urls structure.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
category: Category (e.g., "stems", "lyrics", "finals")
|
||||
file_type: File type within category (e.g., "clean", "lrc", "lossless_4k_mp4")
|
||||
url: GCS URL
|
||||
"""
|
||||
job = self.get_job(job_id)
|
||||
if not job:
|
||||
logger.error(f"Job {job_id} not found")
|
||||
return
|
||||
|
||||
file_urls = job.file_urls.copy()
|
||||
if category not in file_urls:
|
||||
file_urls[category] = {}
|
||||
|
||||
file_urls[category][file_type] = url
|
||||
|
||||
self.update_job(job_id, {'file_urls': file_urls})
|
||||
logger.debug(f"Job {job_id} file URL updated: {category}.{file_type}")
|
||||
|
||||
def check_parallel_processing_complete(self, job_id: str) -> bool:
|
||||
"""
|
||||
Check if both parallel tracks (audio + lyrics) are complete.
|
||||
|
||||
This is called after audio_complete or lyrics_complete to determine
|
||||
if we can proceed to screen generation.
|
||||
|
||||
Returns:
|
||||
True if both tracks complete, False otherwise
|
||||
"""
|
||||
job = self.get_job(job_id)
|
||||
if not job:
|
||||
return False
|
||||
|
||||
audio_complete = job.state_data.get('audio_complete', False)
|
||||
lyrics_complete = job.state_data.get('lyrics_complete', False)
|
||||
|
||||
return audio_complete and lyrics_complete
|
||||
|
||||
def mark_audio_complete(self, job_id: str) -> None:
|
||||
"""
|
||||
Mark audio processing as complete and check if can proceed.
|
||||
|
||||
If lyrics are also complete, automatically triggers screens worker.
|
||||
"""
|
||||
self.update_state_data(job_id, 'audio_complete', True)
|
||||
|
||||
if self.check_parallel_processing_complete(job_id):
|
||||
logger.info(f"Job {job_id}: Both audio and lyrics complete, triggering screens worker")
|
||||
# Transition happens in screens worker
|
||||
# We just trigger it here
|
||||
self._trigger_screens_worker(job_id)
|
||||
|
||||
def mark_lyrics_complete(self, job_id: str) -> None:
|
||||
"""
|
||||
Mark lyrics processing as complete and check if can proceed.
|
||||
|
||||
If audio is also complete, automatically triggers screens worker.
|
||||
"""
|
||||
self.update_state_data(job_id, 'lyrics_complete', True)
|
||||
|
||||
if self.check_parallel_processing_complete(job_id):
|
||||
logger.info(f"Job {job_id}: Both audio and lyrics complete, triggering screens worker")
|
||||
# Transition happens in screens worker
|
||||
# We just trigger it here
|
||||
self._trigger_screens_worker(job_id)
|
||||
|
||||
def _trigger_screens_worker(self, job_id: str) -> None:
|
||||
"""
|
||||
Trigger screens generation worker.
|
||||
|
||||
Uses WorkerService to make HTTP call to internal API.
|
||||
This must be async, so we use asyncio to create a task.
|
||||
"""
|
||||
import asyncio
|
||||
from backend.services.worker_service import get_worker_service
|
||||
|
||||
logger.info(f"Job {job_id}: Triggering screens worker")
|
||||
|
||||
# Create async task to trigger worker
|
||||
# This allows us to call async code from sync context
|
||||
async def _trigger():
|
||||
worker_service = get_worker_service()
|
||||
await worker_service.trigger_screens_worker(job_id)
|
||||
|
||||
# Create task in event loop
|
||||
try:
|
||||
loop = asyncio.get_event_loop()
|
||||
if loop.is_running():
|
||||
# If loop is already running, create task
|
||||
asyncio.create_task(_trigger())
|
||||
else:
|
||||
# If no loop, run directly
|
||||
asyncio.run(_trigger())
|
||||
except RuntimeError:
|
||||
# Fallback: just log
|
||||
logger.warning(f"Job {job_id}: Could not trigger screens worker (no event loop)")
|
||||
# TODO: In production, use message queue instead
|
||||
|
||||
def cancel_job(self, job_id: str, reason: Optional[str] = None) -> bool:
|
||||
"""
|
||||
Cancel a job.
|
||||
|
||||
Only jobs in non-terminal states can be cancelled.
|
||||
"""
|
||||
job = self.get_job(job_id)
|
||||
if not job:
|
||||
return False
|
||||
|
||||
terminal_states = [JobStatus.COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED]
|
||||
if job.status in terminal_states:
|
||||
logger.warning(f"Cannot cancel job {job_id} in terminal state {job.status}")
|
||||
return False
|
||||
|
||||
message = f"Job cancelled{f': {reason}' if reason else ''}"
|
||||
|
||||
self.update_job_status(
|
||||
job_id=job_id,
|
||||
status=JobStatus.CANCELLED,
|
||||
message=message
|
||||
)
|
||||
|
||||
logger.info(f"Job {job_id} cancelled")
|
||||
return True
|
||||
|
||||
def mark_job_failed(
|
||||
self,
|
||||
job_id: str,
|
||||
error_message: str,
|
||||
error_details: Optional[Dict[str, Any]] = None
|
||||
) -> None:
|
||||
"""
|
||||
Mark a job as failed with error details.
|
||||
|
||||
This replaces mark_job_error() with better error tracking.
|
||||
"""
|
||||
updates = {
|
||||
'error_message': error_message
|
||||
}
|
||||
|
||||
if error_details:
|
||||
updates['error_details'] = error_details
|
||||
|
||||
self.firestore.update_job_status(
|
||||
job_id=job_id,
|
||||
status=JobStatus.FAILED,
|
||||
progress=0,
|
||||
message=f"Failed: {error_message}",
|
||||
**updates
|
||||
)
|
||||
logger.error(f"Job {job_id} failed: {error_message}")
|
||||
|
||||
def increment_retry_count(self, job_id: str) -> int:
|
||||
"""
|
||||
Increment retry count for a job.
|
||||
|
||||
Returns:
|
||||
New retry count
|
||||
"""
|
||||
job = self.get_job(job_id)
|
||||
if not job:
|
||||
return 0
|
||||
|
||||
new_count = job.retry_count + 1
|
||||
self.update_job(job_id, {'retry_count': new_count})
|
||||
return new_count
|
||||
|
||||
def append_worker_log(
|
||||
self,
|
||||
job_id: str,
|
||||
worker: str,
|
||||
level: str,
|
||||
message: str,
|
||||
max_logs: int = 500,
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
) -> None:
|
||||
"""
|
||||
Append a log entry to the job's logs.
|
||||
|
||||
By default (USE_LOG_SUBCOLLECTION=true), logs are stored in a Firestore
|
||||
subcollection (jobs/{job_id}/logs) to avoid the 1MB document size limit.
|
||||
|
||||
For backward compatibility (USE_LOG_SUBCOLLECTION=false), logs are stored
|
||||
in the embedded worker_logs array using atomic ArrayUnion.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
worker: Worker name (audio, lyrics, screens, video, render, distribution)
|
||||
level: Log level (DEBUG, INFO, WARNING, ERROR)
|
||||
message: Log message
|
||||
max_logs: Not used (kept for API compatibility)
|
||||
metadata: Optional additional metadata dict
|
||||
"""
|
||||
if settings.use_log_subcollection:
|
||||
# New subcollection approach - avoids 1MB limit
|
||||
log_entry = WorkerLogEntry.create(
|
||||
job_id=job_id,
|
||||
worker=worker,
|
||||
level=level,
|
||||
message=message,
|
||||
metadata=metadata
|
||||
)
|
||||
self.firestore.append_log_to_subcollection(job_id, log_entry)
|
||||
else:
|
||||
# Legacy embedded array approach
|
||||
log_entry = {
|
||||
'timestamp': datetime.utcnow().isoformat() + 'Z',
|
||||
'level': level,
|
||||
'worker': worker,
|
||||
'message': message[:1000] # Truncate long messages
|
||||
}
|
||||
self.firestore.append_worker_log(job_id, log_entry)
|
||||
|
||||
def get_worker_logs(
|
||||
self,
|
||||
job_id: str,
|
||||
since_index: int = 0,
|
||||
worker: Optional[str] = None
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Get worker logs for a job, optionally filtered by worker and index.
|
||||
|
||||
By default (USE_LOG_SUBCOLLECTION=true), logs are read from the
|
||||
subcollection. Falls back to embedded array for older jobs.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
since_index: Return only logs after this index (for pagination)
|
||||
worker: Filter by worker name (optional)
|
||||
|
||||
Returns:
|
||||
List of log entries as dicts (in legacy format for API compatibility)
|
||||
"""
|
||||
if settings.use_log_subcollection:
|
||||
# Try subcollection first
|
||||
subcollection_logs = self.firestore.get_logs_from_subcollection(
|
||||
job_id=job_id,
|
||||
offset=since_index,
|
||||
worker=worker,
|
||||
limit=500
|
||||
)
|
||||
if subcollection_logs:
|
||||
# Convert to legacy format for API compatibility
|
||||
return [log.to_legacy_dict() for log in subcollection_logs]
|
||||
# Fall through to check embedded array for older jobs
|
||||
|
||||
# Embedded array approach (legacy jobs or fallback)
|
||||
job = self.get_job(job_id)
|
||||
if not job or not job.worker_logs:
|
||||
return []
|
||||
|
||||
logs = [log.dict() if hasattr(log, 'dict') else log for log in job.worker_logs]
|
||||
|
||||
# Filter by index
|
||||
if since_index > 0:
|
||||
logs = logs[since_index:]
|
||||
|
||||
# Filter by worker
|
||||
if worker:
|
||||
logs = [log for log in logs if log.get('worker') == worker]
|
||||
|
||||
return logs
|
||||
|
||||
def get_worker_logs_count(self, job_id: str) -> int:
|
||||
"""
|
||||
Get the total count of worker logs for a job.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
|
||||
Returns:
|
||||
Total count of logs
|
||||
"""
|
||||
if settings.use_log_subcollection:
|
||||
# Try subcollection first
|
||||
count = self.firestore.get_logs_count_from_subcollection(job_id)
|
||||
if count > 0:
|
||||
return count
|
||||
# Fall through to check embedded array
|
||||
|
||||
# Embedded array (legacy jobs or fallback)
|
||||
job = self.get_job(job_id)
|
||||
if not job or not job.worker_logs:
|
||||
return 0
|
||||
return len(job.worker_logs)
|
||||
|
||||
@@ -0,0 +1,590 @@
|
||||
"""
|
||||
Local Encoding Service.
|
||||
|
||||
Provides local FFmpeg-based video encoding functionality, extracted from KaraokeFinalise
|
||||
for use by both the cloud backend (video_worker as a fallback) and local CLI.
|
||||
|
||||
This service handles:
|
||||
- Concatenating title, karaoke, and end videos
|
||||
- Encoding to multiple output formats (4K lossless, 4K lossy, 720p, MKV)
|
||||
- Hardware acceleration detection and fallback to software encoding
|
||||
- Remuxing video with instrumental audio
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional, List, Dict, Any, Tuple
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingConfig:
|
||||
"""Configuration for video encoding."""
|
||||
title_video: str # Path to title video
|
||||
karaoke_video: str # Path to karaoke video (with vocals)
|
||||
instrumental_audio: str # Path to instrumental audio
|
||||
end_video: Optional[str] = None # Optional path to end credits video
|
||||
|
||||
# Output paths
|
||||
output_karaoke_mp4: Optional[str] = None
|
||||
output_with_vocals_mp4: Optional[str] = None
|
||||
output_lossless_4k_mp4: Optional[str] = None
|
||||
output_lossy_4k_mp4: Optional[str] = None
|
||||
output_lossless_mkv: Optional[str] = None
|
||||
output_720p_mp4: Optional[str] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingResult:
|
||||
"""Result of video encoding operation."""
|
||||
success: bool
|
||||
output_files: Dict[str, str]
|
||||
error: Optional[str] = None
|
||||
|
||||
|
||||
class LocalEncodingService:
|
||||
"""
|
||||
Service for local FFmpeg-based video encoding.
|
||||
|
||||
Supports hardware acceleration (NVENC) with automatic fallback
|
||||
to software encoding (libx264) when unavailable.
|
||||
"""
|
||||
|
||||
# MP4 flags for better compatibility
|
||||
MP4_FLAGS = "-movflags +faststart"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
dry_run: bool = False,
|
||||
log_level: int = logging.INFO,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the local encoding service.
|
||||
|
||||
Args:
|
||||
dry_run: If True, log commands without executing them
|
||||
log_level: Logging level (affects FFmpeg verbosity)
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.dry_run = dry_run
|
||||
self.log_level = log_level
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
# Hardware acceleration settings (detected on first use)
|
||||
self._hwaccel_available: Optional[bool] = None
|
||||
self._video_encoder: Optional[str] = None
|
||||
self._scale_filter: Optional[str] = None
|
||||
self._hwaccel_decode_flags: Optional[str] = None
|
||||
self._aac_codec: Optional[str] = None
|
||||
|
||||
# Build FFmpeg base command
|
||||
self._ffmpeg_base_command = self._build_ffmpeg_base_command()
|
||||
|
||||
def _build_ffmpeg_base_command(self) -> str:
|
||||
"""Build the FFmpeg base command with appropriate flags."""
|
||||
# Use bundled FFmpeg for frozen builds
|
||||
ffmpeg_path = os.path.join(sys._MEIPASS, "ffmpeg.exe") if getattr(sys, "frozen", False) else "ffmpeg"
|
||||
|
||||
base_cmd = f"{ffmpeg_path} -hide_banner -nostats -y"
|
||||
|
||||
if self.log_level == logging.DEBUG:
|
||||
base_cmd += " -loglevel verbose"
|
||||
else:
|
||||
base_cmd += " -loglevel fatal"
|
||||
|
||||
return base_cmd
|
||||
|
||||
def _detect_hardware_acceleration(self) -> Tuple[bool, str, str, str, str]:
|
||||
"""
|
||||
Detect available hardware acceleration.
|
||||
|
||||
Returns:
|
||||
Tuple of (hwaccel_available, video_encoder, scale_filter,
|
||||
hwaccel_decode_flags, aac_codec)
|
||||
"""
|
||||
self.logger.info("Detecting hardware acceleration capabilities...")
|
||||
|
||||
# Try NVENC (NVIDIA)
|
||||
try:
|
||||
test_cmd = f"{self._ffmpeg_base_command} -hide_banner -loglevel error " \
|
||||
f"-f lavfi -i testsrc=duration=1:size=320x240:rate=1 " \
|
||||
f"-c:v h264_nvenc -f null -"
|
||||
subprocess.run(
|
||||
test_cmd, shell=True, check=True,
|
||||
capture_output=True, timeout=30
|
||||
)
|
||||
self.logger.info("NVIDIA NVENC hardware acceleration available")
|
||||
return (
|
||||
True,
|
||||
"h264_nvenc",
|
||||
"scale_cuda",
|
||||
"-hwaccel cuda -hwaccel_output_format cuda",
|
||||
"aac"
|
||||
)
|
||||
except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError):
|
||||
pass
|
||||
|
||||
# No hardware acceleration available
|
||||
self.logger.info("No hardware acceleration available, using software encoding")
|
||||
return (False, "libx264", "scale", "", "aac")
|
||||
|
||||
@property
|
||||
def hwaccel_available(self) -> bool:
|
||||
"""Check if hardware acceleration is available."""
|
||||
if self._hwaccel_available is None:
|
||||
self._detect_and_set_hwaccel()
|
||||
return self._hwaccel_available
|
||||
|
||||
@property
|
||||
def video_encoder(self) -> str:
|
||||
"""Get the video encoder to use."""
|
||||
if self._video_encoder is None:
|
||||
self._detect_and_set_hwaccel()
|
||||
return self._video_encoder
|
||||
|
||||
@property
|
||||
def scale_filter(self) -> str:
|
||||
"""Get the scale filter to use."""
|
||||
if self._scale_filter is None:
|
||||
self._detect_and_set_hwaccel()
|
||||
return self._scale_filter
|
||||
|
||||
@property
|
||||
def hwaccel_decode_flags(self) -> str:
|
||||
"""Get hardware acceleration decode flags."""
|
||||
if self._hwaccel_decode_flags is None:
|
||||
self._detect_and_set_hwaccel()
|
||||
return self._hwaccel_decode_flags
|
||||
|
||||
@property
|
||||
def aac_codec(self) -> str:
|
||||
"""Get the AAC codec to use."""
|
||||
if self._aac_codec is None:
|
||||
self._detect_and_set_hwaccel()
|
||||
return self._aac_codec
|
||||
|
||||
def _detect_and_set_hwaccel(self) -> None:
|
||||
"""Detect and set hardware acceleration settings."""
|
||||
(
|
||||
self._hwaccel_available,
|
||||
self._video_encoder,
|
||||
self._scale_filter,
|
||||
self._hwaccel_decode_flags,
|
||||
self._aac_codec
|
||||
) = self._detect_hardware_acceleration()
|
||||
|
||||
def _get_nvenc_quality_settings(self, preset: str = "medium") -> str:
|
||||
"""Get NVENC quality settings for different presets."""
|
||||
if not self.hwaccel_available:
|
||||
return ""
|
||||
|
||||
settings = {
|
||||
"lossless": "-preset p7 -tune hq -rc vbr -cq 0 -qmin 0 -qmax 0",
|
||||
"high": "-preset p7 -tune hq -rc vbr -cq 19 -b:v 0",
|
||||
"medium": "-preset p4 -tune hq -rc vbr -cq 23 -b:v 0",
|
||||
"fast": "-preset p2 -tune ll -rc vbr -cq 28 -b:v 0",
|
||||
}
|
||||
return settings.get(preset, settings["medium"])
|
||||
|
||||
def _execute_command(
|
||||
self,
|
||||
command: str,
|
||||
description: str,
|
||||
timeout: int = 3600, # 1 hour default
|
||||
) -> bool:
|
||||
"""
|
||||
Execute an FFmpeg command.
|
||||
|
||||
Args:
|
||||
command: The FFmpeg command to execute
|
||||
description: Human-readable description of the operation
|
||||
timeout: Command timeout in seconds
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
self.logger.info(f"Executing: {description}")
|
||||
self.logger.debug(f"Command: {command}")
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(f"DRY RUN: Would execute: {command}")
|
||||
return True
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
command,
|
||||
shell=True,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout
|
||||
)
|
||||
self.logger.info(f"Completed: {description}")
|
||||
return True
|
||||
except subprocess.CalledProcessError as e:
|
||||
self.logger.error(f"Failed: {description}")
|
||||
self.logger.error(f"Error: {e.stderr}")
|
||||
return False
|
||||
except subprocess.TimeoutExpired:
|
||||
self.logger.error(f"Timeout: {description}")
|
||||
return False
|
||||
|
||||
def _execute_command_with_fallback(
|
||||
self,
|
||||
gpu_command: str,
|
||||
cpu_command: str,
|
||||
description: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Execute a command with GPU, falling back to CPU if it fails.
|
||||
|
||||
Args:
|
||||
gpu_command: Hardware-accelerated command
|
||||
cpu_command: Software fallback command
|
||||
description: Human-readable description
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
if self.hwaccel_available:
|
||||
self.logger.info(f"Trying hardware-accelerated encoding for: {description}")
|
||||
if self._execute_command(gpu_command, description):
|
||||
return True
|
||||
self.logger.warning(f"Hardware encoding failed, falling back to software")
|
||||
|
||||
return self._execute_command(cpu_command, description)
|
||||
|
||||
def remux_with_instrumental(
|
||||
self,
|
||||
input_video: str,
|
||||
instrumental_audio: str,
|
||||
output_file: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Remux video with instrumental audio track.
|
||||
|
||||
Args:
|
||||
input_video: Path to input video file
|
||||
instrumental_audio: Path to instrumental audio file
|
||||
output_file: Path for output file
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
command = (
|
||||
f'{self._ffmpeg_base_command} -i "{input_video}" '
|
||||
f'-i "{instrumental_audio}" -map 0:v -map 1:a -c copy '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
return self._execute_command(command, "Remuxing with instrumental audio")
|
||||
|
||||
def convert_mov_to_mp4(
|
||||
self,
|
||||
input_file: str,
|
||||
output_file: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Convert MOV to MP4 format.
|
||||
|
||||
Args:
|
||||
input_file: Path to input MOV file
|
||||
output_file: Path for output MP4 file
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
gpu_command = (
|
||||
f'{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i "{input_file}" '
|
||||
f'-c:v {self.video_encoder} -c:a copy {self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
cpu_command = (
|
||||
f'{self._ffmpeg_base_command} -i "{input_file}" '
|
||||
f'-c:v libx264 -c:a copy {self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
return self._execute_command_with_fallback(
|
||||
gpu_command, cpu_command, "Converting MOV to MP4"
|
||||
)
|
||||
|
||||
def encode_lossless_mp4(
|
||||
self,
|
||||
title_video: str,
|
||||
karaoke_video: str,
|
||||
output_file: str,
|
||||
end_video: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Create lossless 4K MP4 by concatenating title, karaoke, and optionally end videos.
|
||||
|
||||
Args:
|
||||
title_video: Path to title video
|
||||
karaoke_video: Path to karaoke video
|
||||
output_file: Path for output file
|
||||
end_video: Optional path to end credits video
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
# Quote file paths
|
||||
title_quoted = shlex.quote(os.path.abspath(title_video))
|
||||
karaoke_quoted = shlex.quote(os.path.abspath(karaoke_video))
|
||||
|
||||
# Build filter and inputs for concatenation
|
||||
if end_video and os.path.isfile(end_video):
|
||||
end_quoted = shlex.quote(os.path.abspath(end_video))
|
||||
extra_input = f"-i {end_quoted}"
|
||||
concat_filter = (
|
||||
'-filter_complex "[0:v:0][0:a:0][1:v:0][1:a:0][2:v:0][2:a:0]'
|
||||
'concat=n=3:v=1:a=1[outv][outa]"'
|
||||
)
|
||||
else:
|
||||
extra_input = ""
|
||||
concat_filter = (
|
||||
'-filter_complex "[0:v:0][0:a:0][1:v:0][1:a:0]'
|
||||
'concat=n=2:v=1:a=1[outv][outa]"'
|
||||
)
|
||||
|
||||
gpu_command = (
|
||||
f"{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i {title_quoted} "
|
||||
f"{self.hwaccel_decode_flags} -i {karaoke_quoted} {extra_input} "
|
||||
f'{concat_filter} -map "[outv]" -map "[outa]" -c:v {self.video_encoder} '
|
||||
f'{self._get_nvenc_quality_settings("lossless")} -c:a pcm_s16le '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
cpu_command = (
|
||||
f"{self._ffmpeg_base_command} -i {title_quoted} -i {karaoke_quoted} {extra_input} "
|
||||
f'{concat_filter} -map "[outv]" -map "[outa]" -c:v libx264 -c:a pcm_s16le '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
|
||||
return self._execute_command_with_fallback(
|
||||
gpu_command, cpu_command, "Encoding lossless 4K MP4"
|
||||
)
|
||||
|
||||
def encode_lossy_mp4(
|
||||
self,
|
||||
input_file: str,
|
||||
output_file: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Create lossy 4K MP4 with AAC audio.
|
||||
|
||||
Args:
|
||||
input_file: Path to input file (typically lossless MP4)
|
||||
output_file: Path for output file
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
command = (
|
||||
f'{self._ffmpeg_base_command} -i "{input_file}" '
|
||||
f'-c:v copy -c:a {self.aac_codec} -ar 48000 -b:a 320k '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
return self._execute_command(command, "Encoding lossy 4K MP4 with AAC")
|
||||
|
||||
def encode_lossless_mkv(
|
||||
self,
|
||||
input_file: str,
|
||||
output_file: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Create MKV with FLAC audio (for YouTube upload).
|
||||
|
||||
Args:
|
||||
input_file: Path to input file
|
||||
output_file: Path for output file
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
command = (
|
||||
f'{self._ffmpeg_base_command} -i "{input_file}" '
|
||||
f'-c:v copy -c:a flac "{output_file}"'
|
||||
)
|
||||
return self._execute_command(command, "Creating MKV with FLAC for YouTube")
|
||||
|
||||
def encode_720p(
|
||||
self,
|
||||
input_file: str,
|
||||
output_file: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Create 720p MP4 with AAC audio.
|
||||
|
||||
Args:
|
||||
input_file: Path to input file
|
||||
output_file: Path for output file
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
gpu_command = (
|
||||
f'{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i "{input_file}" '
|
||||
f'-c:v {self.video_encoder} -vf "{self.scale_filter}=1280:720" '
|
||||
f'{self._get_nvenc_quality_settings("medium")} -b:v 2000k '
|
||||
f'-c:a {self.aac_codec} -ar 48000 -b:a 128k '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
cpu_command = (
|
||||
f'{self._ffmpeg_base_command} -i "{input_file}" '
|
||||
f'-c:v libx264 -vf "scale=1280:720" -b:v 2000k -preset medium -tune animation '
|
||||
f'-c:a {self.aac_codec} -ar 48000 -b:a 128k '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
return self._execute_command_with_fallback(
|
||||
gpu_command, cpu_command, "Encoding 720p MP4"
|
||||
)
|
||||
|
||||
def encode_all_formats(
|
||||
self,
|
||||
config: EncodingConfig,
|
||||
) -> EncodingResult:
|
||||
"""
|
||||
Encode video to all output formats.
|
||||
|
||||
This performs the full encoding pipeline:
|
||||
1. Remux with instrumental audio
|
||||
2. Convert to MP4 if needed
|
||||
3. Encode lossless 4K MP4 (concatenated)
|
||||
4. Encode lossy 4K MP4
|
||||
5. Encode lossless MKV (for YouTube)
|
||||
6. Encode 720p MP4
|
||||
|
||||
Args:
|
||||
config: Encoding configuration with input/output paths
|
||||
|
||||
Returns:
|
||||
EncodingResult with success status and output file paths
|
||||
"""
|
||||
output_files = {}
|
||||
|
||||
try:
|
||||
# Step 1: Remux with instrumental audio
|
||||
if config.output_karaoke_mp4:
|
||||
self.logger.info("[Step 1/6] Remuxing video with instrumental audio...")
|
||||
if not self.remux_with_instrumental(
|
||||
config.karaoke_video,
|
||||
config.instrumental_audio,
|
||||
config.output_karaoke_mp4
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to remux with instrumental audio"
|
||||
)
|
||||
output_files["karaoke_mp4"] = config.output_karaoke_mp4
|
||||
|
||||
# Step 2: Convert to MP4 if needed
|
||||
if config.output_with_vocals_mp4:
|
||||
if not config.karaoke_video.endswith(".mp4"):
|
||||
self.logger.info("[Step 2/6] Converting karaoke video to MP4...")
|
||||
if not self.convert_mov_to_mp4(
|
||||
config.karaoke_video,
|
||||
config.output_with_vocals_mp4
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to convert to MP4"
|
||||
)
|
||||
output_files["with_vocals_mp4"] = config.output_with_vocals_mp4
|
||||
else:
|
||||
self.logger.info("[Step 2/6] Skipped - video already MP4")
|
||||
|
||||
# Step 3: Encode lossless 4K MP4
|
||||
if config.output_lossless_4k_mp4:
|
||||
self.logger.info("[Step 3/6] Encoding lossless 4K MP4...")
|
||||
karaoke_for_concat = config.output_karaoke_mp4 or config.karaoke_video
|
||||
if not self.encode_lossless_mp4(
|
||||
config.title_video,
|
||||
karaoke_for_concat,
|
||||
config.output_lossless_4k_mp4,
|
||||
config.end_video
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to encode lossless 4K MP4"
|
||||
)
|
||||
output_files["lossless_4k_mp4"] = config.output_lossless_4k_mp4
|
||||
|
||||
# Step 4: Encode lossy 4K MP4
|
||||
if config.output_lossy_4k_mp4 and config.output_lossless_4k_mp4:
|
||||
self.logger.info("[Step 4/6] Encoding lossy 4K MP4...")
|
||||
if not self.encode_lossy_mp4(
|
||||
config.output_lossless_4k_mp4,
|
||||
config.output_lossy_4k_mp4
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to encode lossy 4K MP4"
|
||||
)
|
||||
output_files["lossy_4k_mp4"] = config.output_lossy_4k_mp4
|
||||
|
||||
# Step 5: Create MKV with FLAC audio
|
||||
if config.output_lossless_mkv and config.output_lossless_4k_mp4:
|
||||
self.logger.info("[Step 5/6] Creating MKV with FLAC audio...")
|
||||
if not self.encode_lossless_mkv(
|
||||
config.output_lossless_4k_mp4,
|
||||
config.output_lossless_mkv
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to create MKV"
|
||||
)
|
||||
output_files["lossless_mkv"] = config.output_lossless_mkv
|
||||
|
||||
# Step 6: Encode 720p version
|
||||
if config.output_720p_mp4 and config.output_lossless_4k_mp4:
|
||||
self.logger.info("[Step 6/6] Encoding 720p MP4...")
|
||||
if not self.encode_720p(
|
||||
config.output_lossless_4k_mp4,
|
||||
config.output_720p_mp4
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to encode 720p"
|
||||
)
|
||||
output_files["720p_mp4"] = config.output_720p_mp4
|
||||
|
||||
self.logger.info("All encoding steps completed successfully")
|
||||
return EncodingResult(success=True, output_files=output_files)
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f"Encoding failed with exception: {e}")
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
|
||||
# Singleton instance and factory function (following existing service pattern)
|
||||
_local_encoding_service: Optional[LocalEncodingService] = None
|
||||
|
||||
|
||||
def get_local_encoding_service(**kwargs) -> LocalEncodingService:
|
||||
"""
|
||||
Get a local encoding service instance.
|
||||
|
||||
Args:
|
||||
**kwargs: Arguments passed to LocalEncodingService
|
||||
|
||||
Returns:
|
||||
LocalEncodingService instance
|
||||
"""
|
||||
global _local_encoding_service
|
||||
|
||||
if _local_encoding_service is None:
|
||||
_local_encoding_service = LocalEncodingService(**kwargs)
|
||||
|
||||
return _local_encoding_service
|
||||
@@ -0,0 +1,216 @@
|
||||
"""
|
||||
LyricsTranscriber cache synchronization with GCS.
|
||||
|
||||
This service persists LyricsTranscriber's cache files to GCS so that
|
||||
cloud workers (Cloud Run instances) can share cache across containers.
|
||||
|
||||
Cache files are stored flat in GCS under lyrics-transcriber-cache/ prefix:
|
||||
- Transcription: {provider}_{audio_hash}_raw.json, {provider}_{audio_hash}_converted.json
|
||||
- Lyrics: {provider}_{artist_title_hash}_raw.json, {provider}_{artist_title_hash}_converted.json
|
||||
|
||||
Hash computation matches LyricsTranscriber's implementation exactly:
|
||||
- Audio hash: MD5 of audio file bytes
|
||||
- Lyrics hash: MD5 of "{artist.lower()}_{title.lower()}"
|
||||
"""
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Providers that use audio file hash as cache key
|
||||
TRANSCRIPTION_PROVIDERS = ["audioshake", "whisper", "localwhisper"]
|
||||
|
||||
# Providers that use artist+title hash as cache key
|
||||
LYRICS_PROVIDERS = ["genius", "spotify", "lrclib", "musixmatch"]
|
||||
|
||||
# Cache file suffixes
|
||||
CACHE_SUFFIXES = ["raw", "converted"]
|
||||
|
||||
|
||||
class LyricsCacheService:
|
||||
"""Service to sync LyricsTranscriber cache with GCS."""
|
||||
|
||||
GCS_CACHE_PREFIX = "lyrics-transcriber-cache/"
|
||||
|
||||
def __init__(self, storage: Optional[StorageService] = None):
|
||||
"""Initialize the cache service.
|
||||
|
||||
Args:
|
||||
storage: StorageService instance. If None, creates a new one.
|
||||
"""
|
||||
self.storage = storage or StorageService()
|
||||
|
||||
def compute_audio_hash(self, audio_path: str) -> str:
|
||||
"""Compute MD5 hash of audio file bytes.
|
||||
|
||||
This matches LyricsTranscriber's _get_file_hash() method exactly.
|
||||
|
||||
Args:
|
||||
audio_path: Path to the audio file.
|
||||
|
||||
Returns:
|
||||
MD5 hex digest of the file contents.
|
||||
"""
|
||||
md5_hash = hashlib.md5()
|
||||
with open(audio_path, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(4096), b""):
|
||||
md5_hash.update(chunk)
|
||||
return md5_hash.hexdigest()
|
||||
|
||||
def compute_lyrics_hash(self, artist: str, title: str) -> str:
|
||||
"""Compute MD5 hash of artist and title.
|
||||
|
||||
This matches LyricsTranscriber's _get_artist_title_hash() method exactly.
|
||||
|
||||
Args:
|
||||
artist: Artist name.
|
||||
title: Track title.
|
||||
|
||||
Returns:
|
||||
MD5 hex digest of "{artist.lower()}_{title.lower()}".
|
||||
"""
|
||||
combined = f"{artist.lower()}_{title.lower()}"
|
||||
return hashlib.md5(combined.encode()).hexdigest()
|
||||
|
||||
def _get_cache_filenames(
|
||||
self, providers: List[str], hash_value: str
|
||||
) -> List[str]:
|
||||
"""Generate list of possible cache filenames for given providers and hash.
|
||||
|
||||
Args:
|
||||
providers: List of provider names (e.g., ["audioshake", "whisper"]).
|
||||
hash_value: The hash to use in filenames.
|
||||
|
||||
Returns:
|
||||
List of filenames like ["audioshake_abc123_raw.json", ...].
|
||||
"""
|
||||
filenames = []
|
||||
for provider in providers:
|
||||
for suffix in CACHE_SUFFIXES:
|
||||
filenames.append(f"{provider}_{hash_value}_{suffix}.json")
|
||||
return filenames
|
||||
|
||||
def sync_cache_from_gcs(
|
||||
self,
|
||||
local_cache_dir: str,
|
||||
audio_hash: str,
|
||||
lyrics_hash: str,
|
||||
) -> Dict[str, int]:
|
||||
"""Download relevant cache files from GCS to local directory.
|
||||
|
||||
Downloads cache files for both transcription (audio hash) and
|
||||
lyrics (artist+title hash) providers.
|
||||
|
||||
Args:
|
||||
local_cache_dir: Local directory to download cache files to.
|
||||
audio_hash: MD5 hash of audio file.
|
||||
lyrics_hash: MD5 hash of artist+title.
|
||||
|
||||
Returns:
|
||||
Dict with counts: {"downloaded": N, "not_found": M, "errors": E}
|
||||
"""
|
||||
os.makedirs(local_cache_dir, exist_ok=True)
|
||||
|
||||
stats = {"downloaded": 0, "not_found": 0, "errors": 0}
|
||||
|
||||
# Get all possible cache filenames
|
||||
transcription_files = self._get_cache_filenames(
|
||||
TRANSCRIPTION_PROVIDERS, audio_hash
|
||||
)
|
||||
lyrics_files = self._get_cache_filenames(LYRICS_PROVIDERS, lyrics_hash)
|
||||
all_files = transcription_files + lyrics_files
|
||||
|
||||
for filename in all_files:
|
||||
gcs_path = f"{self.GCS_CACHE_PREFIX}{filename}"
|
||||
local_path = os.path.join(local_cache_dir, filename)
|
||||
|
||||
try:
|
||||
if self.storage.file_exists(gcs_path):
|
||||
self.storage.download_file(gcs_path, local_path)
|
||||
logger.info(f"Cache hit: downloaded {filename}")
|
||||
stats["downloaded"] += 1
|
||||
else:
|
||||
logger.debug(f"Cache miss: {filename} not in GCS")
|
||||
stats["not_found"] += 1
|
||||
except Exception as e:
|
||||
logger.warning(f"Error downloading cache file {filename}: {e}")
|
||||
stats["errors"] += 1
|
||||
|
||||
logger.info(
|
||||
f"Cache sync from GCS complete: "
|
||||
f"{stats['downloaded']} downloaded, "
|
||||
f"{stats['not_found']} not found, "
|
||||
f"{stats['errors']} errors"
|
||||
)
|
||||
return stats
|
||||
|
||||
def sync_cache_to_gcs(
|
||||
self,
|
||||
local_cache_dir: str,
|
||||
audio_hash: str,
|
||||
lyrics_hash: str,
|
||||
) -> Dict[str, int]:
|
||||
"""Upload new cache files from local directory to GCS.
|
||||
|
||||
Only uploads files that match expected cache patterns for the given
|
||||
hashes and don't already exist in GCS.
|
||||
|
||||
Args:
|
||||
local_cache_dir: Local directory with cache files.
|
||||
audio_hash: MD5 hash of audio file.
|
||||
lyrics_hash: MD5 hash of artist+title.
|
||||
|
||||
Returns:
|
||||
Dict with counts: {"uploaded": N, "skipped": M, "errors": E}
|
||||
"""
|
||||
stats = {"uploaded": 0, "skipped": 0, "errors": 0}
|
||||
|
||||
if not os.path.exists(local_cache_dir):
|
||||
logger.warning(f"Local cache dir does not exist: {local_cache_dir}")
|
||||
return stats
|
||||
|
||||
# Get all possible cache filenames we're interested in
|
||||
transcription_files = self._get_cache_filenames(
|
||||
TRANSCRIPTION_PROVIDERS, audio_hash
|
||||
)
|
||||
lyrics_files = self._get_cache_filenames(LYRICS_PROVIDERS, lyrics_hash)
|
||||
expected_files = set(transcription_files + lyrics_files)
|
||||
|
||||
# Check each file in local cache dir
|
||||
for filename in os.listdir(local_cache_dir):
|
||||
# Only process files we expect (matching our hash patterns)
|
||||
if filename not in expected_files:
|
||||
continue
|
||||
|
||||
local_path = os.path.join(local_cache_dir, filename)
|
||||
if not os.path.isfile(local_path):
|
||||
continue
|
||||
|
||||
gcs_path = f"{self.GCS_CACHE_PREFIX}{filename}"
|
||||
|
||||
try:
|
||||
# Skip if already exists in GCS (same hash = same content)
|
||||
if self.storage.file_exists(gcs_path):
|
||||
logger.debug(f"Cache file already in GCS: {filename}")
|
||||
stats["skipped"] += 1
|
||||
continue
|
||||
|
||||
self.storage.upload_file(local_path, gcs_path)
|
||||
logger.info(f"Uploaded cache file: {filename}")
|
||||
stats["uploaded"] += 1
|
||||
except Exception as e:
|
||||
logger.warning(f"Error uploading cache file {filename}: {e}")
|
||||
stats["errors"] += 1
|
||||
|
||||
logger.info(
|
||||
f"Cache sync to GCS complete: "
|
||||
f"{stats['uploaded']} uploaded, "
|
||||
f"{stats['skipped']} skipped (already exist), "
|
||||
f"{stats['errors']} errors"
|
||||
)
|
||||
return stats
|
||||
@@ -0,0 +1,413 @@
|
||||
"""
|
||||
Custom metrics for Cloud Monitoring.
|
||||
|
||||
This module provides application-level metrics for tracking job processing,
|
||||
worker performance, and external API usage. Metrics can be viewed in:
|
||||
1. Cloud Logging (via structured log entries)
|
||||
2. Cloud Monitoring (via log-based metrics or OpenTelemetry)
|
||||
|
||||
The metrics service uses a pragmatic approach:
|
||||
- Always emits metrics as structured log entries (works immediately)
|
||||
- Uses the same JSON format as Cloud Logging
|
||||
- Can be enhanced with OpenTelemetry metrics exporters when available
|
||||
|
||||
Usage:
|
||||
from backend.services.metrics import metrics
|
||||
|
||||
# Record a job completion
|
||||
metrics.record_job_completed("abc123", source="upload")
|
||||
|
||||
# Record worker duration
|
||||
metrics.record_worker_duration("audio", 45.2, success=True)
|
||||
|
||||
# Record external API call
|
||||
with metrics.time_external_api("modal"):
|
||||
response = await modal_client.separate_audio(...)
|
||||
"""
|
||||
import logging
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from backend.services.tracing import get_current_trace_id, get_current_span_id
|
||||
|
||||
|
||||
logger = logging.getLogger("metrics")
|
||||
|
||||
|
||||
@dataclass
|
||||
class MetricLabels:
|
||||
"""Common metric labels."""
|
||||
job_id: Optional[str] = None
|
||||
worker: Optional[str] = None
|
||||
status: Optional[str] = None
|
||||
source: Optional[str] = None
|
||||
api: Optional[str] = None
|
||||
operation: Optional[str] = None
|
||||
bucket: Optional[str] = None
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
"""Convert to dict, excluding None values."""
|
||||
return {k: v for k, v in self.__dict__.items() if v is not None}
|
||||
|
||||
|
||||
class MetricsService:
|
||||
"""
|
||||
Application metrics service.
|
||||
|
||||
Emits metrics as structured log entries that can be:
|
||||
1. Queried directly in Cloud Logging
|
||||
2. Converted to Cloud Monitoring metrics via log-based metrics
|
||||
3. Exported via OpenTelemetry (when configured)
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize the metrics service."""
|
||||
self._logger = logging.getLogger("metrics")
|
||||
# Ensure metrics logger outputs at INFO level
|
||||
self._logger.setLevel(logging.INFO)
|
||||
|
||||
def _emit_metric(self, metric_name: str, metric_type: str, value: float, labels: Dict[str, Any]) -> None:
|
||||
"""
|
||||
Emit a metric as a structured log entry.
|
||||
|
||||
The log format is designed to be easily parsed by Cloud Logging
|
||||
and converted to log-based metrics.
|
||||
|
||||
Args:
|
||||
metric_name: Name of the metric (e.g., "jobs_total")
|
||||
metric_type: Type of metric (counter, histogram, gauge)
|
||||
value: Metric value
|
||||
labels: Metric labels/dimensions
|
||||
"""
|
||||
# Add trace context if available
|
||||
trace_id = get_current_trace_id()
|
||||
span_id = get_current_span_id()
|
||||
|
||||
# Build metric entry
|
||||
metric_entry = {
|
||||
"metric_name": metric_name,
|
||||
"metric_type": metric_type,
|
||||
"metric_value": value,
|
||||
**labels,
|
||||
}
|
||||
|
||||
if trace_id:
|
||||
metric_entry["trace_id"] = trace_id
|
||||
if span_id:
|
||||
metric_entry["span_id"] = span_id
|
||||
|
||||
# Emit as structured log entry
|
||||
# Use INFO level so metrics always show up
|
||||
self._logger.info(
|
||||
f"METRIC {metric_name}={value}",
|
||||
extra=metric_entry
|
||||
)
|
||||
|
||||
# =========================================
|
||||
# Job Metrics
|
||||
# =========================================
|
||||
|
||||
def record_job_created(self, job_id: str, source: str = "unknown") -> None:
|
||||
"""
|
||||
Record a new job creation.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
source: Job source (upload, url, search)
|
||||
"""
|
||||
self._emit_metric(
|
||||
metric_name="jobs_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels={"job_id": job_id, "status": "created", "source": source}
|
||||
)
|
||||
|
||||
def record_job_completed(self, job_id: str, source: str = "unknown") -> None:
|
||||
"""
|
||||
Record a job completion.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
source: Job source (upload, url, search)
|
||||
"""
|
||||
self._emit_metric(
|
||||
metric_name="jobs_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels={"job_id": job_id, "status": "completed", "source": source}
|
||||
)
|
||||
|
||||
def record_job_failed(self, job_id: str, source: str = "unknown", error: Optional[str] = None) -> None:
|
||||
"""
|
||||
Record a job failure.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
source: Job source (upload, url, search)
|
||||
error: Optional error message
|
||||
"""
|
||||
labels = {"job_id": job_id, "status": "failed", "source": source}
|
||||
if error:
|
||||
labels["error"] = error[:200] # Truncate long errors
|
||||
self._emit_metric(
|
||||
metric_name="jobs_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels=labels
|
||||
)
|
||||
|
||||
def record_job_duration(self, job_id: str, duration_seconds: float, source: str = "unknown") -> None:
|
||||
"""
|
||||
Record total job processing duration.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
duration_seconds: Total processing time in seconds
|
||||
source: Job source (upload, url, search)
|
||||
"""
|
||||
self._emit_metric(
|
||||
metric_name="job_duration_seconds",
|
||||
metric_type="histogram",
|
||||
value=duration_seconds,
|
||||
labels={"job_id": job_id, "source": source}
|
||||
)
|
||||
|
||||
# =========================================
|
||||
# Worker Metrics
|
||||
# =========================================
|
||||
|
||||
def record_worker_started(self, worker: str, job_id: str) -> None:
|
||||
"""
|
||||
Record a worker invocation start.
|
||||
|
||||
Args:
|
||||
worker: Worker name (audio, lyrics, screens, video, render_video)
|
||||
job_id: Job ID
|
||||
"""
|
||||
self._emit_metric(
|
||||
metric_name="worker_invocations_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels={"worker": worker, "job_id": job_id, "status": "started"}
|
||||
)
|
||||
|
||||
def record_worker_duration(self, worker: str, duration_seconds: float, success: bool, job_id: Optional[str] = None) -> None:
|
||||
"""
|
||||
Record worker processing duration.
|
||||
|
||||
Args:
|
||||
worker: Worker name
|
||||
duration_seconds: Processing time in seconds
|
||||
success: Whether worker completed successfully
|
||||
job_id: Optional job ID
|
||||
"""
|
||||
labels = {
|
||||
"worker": worker,
|
||||
"success": str(success).lower(),
|
||||
}
|
||||
if job_id:
|
||||
labels["job_id"] = job_id
|
||||
|
||||
self._emit_metric(
|
||||
metric_name="job_stage_duration_seconds",
|
||||
metric_type="histogram",
|
||||
value=duration_seconds,
|
||||
labels=labels
|
||||
)
|
||||
|
||||
# Also emit a counter for success/failure tracking
|
||||
self._emit_metric(
|
||||
metric_name="worker_invocations_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels={"worker": worker, "success": str(success).lower(), "job_id": job_id or "unknown"}
|
||||
)
|
||||
|
||||
# =========================================
|
||||
# GCS Metrics
|
||||
# =========================================
|
||||
|
||||
def record_gcs_operation(
|
||||
self,
|
||||
operation: str,
|
||||
bucket: str,
|
||||
success: bool,
|
||||
size_bytes: Optional[int] = None,
|
||||
duration_seconds: Optional[float] = None,
|
||||
job_id: Optional[str] = None,
|
||||
) -> None:
|
||||
"""
|
||||
Record a GCS operation.
|
||||
|
||||
Args:
|
||||
operation: Operation type (upload, download, delete)
|
||||
bucket: GCS bucket name
|
||||
success: Whether operation succeeded
|
||||
size_bytes: Optional file size in bytes
|
||||
duration_seconds: Optional operation duration
|
||||
job_id: Optional job ID
|
||||
"""
|
||||
labels = {
|
||||
"operation": operation,
|
||||
"bucket": bucket,
|
||||
"success": str(success).lower(),
|
||||
}
|
||||
if job_id:
|
||||
labels["job_id"] = job_id
|
||||
|
||||
self._emit_metric(
|
||||
metric_name="gcs_operations_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels=labels
|
||||
)
|
||||
|
||||
if size_bytes is not None:
|
||||
self._emit_metric(
|
||||
metric_name="gcs_operation_bytes",
|
||||
metric_type="histogram",
|
||||
value=size_bytes,
|
||||
labels={**labels, "operation": operation}
|
||||
)
|
||||
|
||||
if duration_seconds is not None:
|
||||
self._emit_metric(
|
||||
metric_name="gcs_operation_duration_seconds",
|
||||
metric_type="histogram",
|
||||
value=duration_seconds,
|
||||
labels={**labels, "operation": operation}
|
||||
)
|
||||
|
||||
# =========================================
|
||||
# External API Metrics
|
||||
# =========================================
|
||||
|
||||
def record_external_api_call(
|
||||
self,
|
||||
api: str,
|
||||
success: bool,
|
||||
duration_seconds: float,
|
||||
job_id: Optional[str] = None,
|
||||
error: Optional[str] = None,
|
||||
) -> None:
|
||||
"""
|
||||
Record an external API call.
|
||||
|
||||
Args:
|
||||
api: API name (modal, audioshake, genius, spotify)
|
||||
success: Whether call succeeded
|
||||
duration_seconds: API call duration
|
||||
job_id: Optional job ID
|
||||
error: Optional error message
|
||||
"""
|
||||
labels = {
|
||||
"api": api,
|
||||
"success": str(success).lower(),
|
||||
}
|
||||
if job_id:
|
||||
labels["job_id"] = job_id
|
||||
if error:
|
||||
labels["error"] = error[:100] # Truncate
|
||||
|
||||
self._emit_metric(
|
||||
metric_name="external_api_calls_total",
|
||||
metric_type="counter",
|
||||
value=1,
|
||||
labels=labels
|
||||
)
|
||||
|
||||
self._emit_metric(
|
||||
metric_name="external_api_duration_seconds",
|
||||
metric_type="histogram",
|
||||
value=duration_seconds,
|
||||
labels={"api": api, "success": str(success).lower()}
|
||||
)
|
||||
|
||||
@contextmanager
|
||||
def time_external_api(self, api: str, job_id: Optional[str] = None):
|
||||
"""
|
||||
Context manager to time an external API call.
|
||||
|
||||
Usage:
|
||||
with metrics.time_external_api("modal", job_id) as timer:
|
||||
response = await client.call_api()
|
||||
timer.set_success(True)
|
||||
|
||||
Args:
|
||||
api: API name
|
||||
job_id: Optional job ID
|
||||
|
||||
Yields:
|
||||
Timer object with set_success() method
|
||||
"""
|
||||
timer = _ApiTimer()
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
yield timer
|
||||
except Exception as e:
|
||||
timer.set_success(False)
|
||||
timer.error = str(e)
|
||||
raise
|
||||
finally:
|
||||
duration = time.time() - start_time
|
||||
self.record_external_api_call(
|
||||
api=api,
|
||||
success=timer.success,
|
||||
duration_seconds=duration,
|
||||
job_id=job_id,
|
||||
error=timer.error,
|
||||
)
|
||||
|
||||
@contextmanager
|
||||
def time_worker(self, worker: str, job_id: str):
|
||||
"""
|
||||
Context manager to time a worker execution.
|
||||
|
||||
Usage:
|
||||
with metrics.time_worker("audio", job_id) as timer:
|
||||
await process_audio()
|
||||
timer.set_success(True)
|
||||
|
||||
Args:
|
||||
worker: Worker name
|
||||
job_id: Job ID
|
||||
|
||||
Yields:
|
||||
Timer object with set_success() method
|
||||
"""
|
||||
timer = _ApiTimer()
|
||||
self.record_worker_started(worker, job_id)
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
yield timer
|
||||
except Exception as e:
|
||||
timer.set_success(False)
|
||||
raise
|
||||
finally:
|
||||
duration = time.time() - start_time
|
||||
self.record_worker_duration(
|
||||
worker=worker,
|
||||
duration_seconds=duration,
|
||||
success=timer.success,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
|
||||
class _ApiTimer:
|
||||
"""Helper class for tracking API call success state."""
|
||||
|
||||
def __init__(self):
|
||||
self.success = True # Assume success unless set otherwise
|
||||
self.error: Optional[str] = None
|
||||
|
||||
def set_success(self, success: bool) -> None:
|
||||
self.success = success
|
||||
|
||||
|
||||
# Global metrics instance
|
||||
metrics = MetricsService()
|
||||
|
||||
@@ -0,0 +1,287 @@
|
||||
"""
|
||||
Packaging Service.
|
||||
|
||||
Provides CDG and TXT package generation functionality, extracted from KaraokeFinalise
|
||||
for use by both the cloud backend (video_worker) and local CLI.
|
||||
|
||||
This service handles:
|
||||
- CDG (CD+G) file generation from LRC files
|
||||
- TXT lyric file generation from LRC files
|
||||
- ZIP packaging of CDG/MP3 and TXT/MP3 pairs
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import zipfile
|
||||
from typing import Optional, Dict, Any, Tuple
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class PackagingService:
|
||||
"""
|
||||
Service for creating CDG and TXT karaoke packages.
|
||||
|
||||
CDG (CD+Graphics) is a format used by karaoke machines.
|
||||
TXT packages are used by software karaoke players.
|
||||
Both formats are packaged as ZIP files with an MP3 audio track.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
cdg_styles: Optional[Dict[str, Any]] = None,
|
||||
dry_run: bool = False,
|
||||
non_interactive: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the packaging service.
|
||||
|
||||
Args:
|
||||
cdg_styles: CDG style configuration for CDG generation
|
||||
dry_run: If True, log actions without performing them
|
||||
non_interactive: If True, skip interactive prompts
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.cdg_styles = cdg_styles
|
||||
self.dry_run = dry_run
|
||||
self.non_interactive = non_interactive
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
def create_cdg_package(
|
||||
self,
|
||||
lrc_file: str,
|
||||
audio_file: str,
|
||||
output_zip_path: str,
|
||||
artist: str,
|
||||
title: str,
|
||||
output_mp3_path: Optional[str] = None,
|
||||
output_cdg_path: Optional[str] = None,
|
||||
) -> Tuple[str, Optional[str], Optional[str]]:
|
||||
"""
|
||||
Create a CDG package (ZIP containing CDG and MP3 files).
|
||||
|
||||
Args:
|
||||
lrc_file: Path to the LRC lyrics file
|
||||
audio_file: Path to the instrumental audio file
|
||||
output_zip_path: Path for the output ZIP file
|
||||
artist: Artist name for metadata
|
||||
title: Song title for metadata
|
||||
output_mp3_path: Optional path for the extracted MP3 file
|
||||
output_cdg_path: Optional path for the extracted CDG file
|
||||
|
||||
Returns:
|
||||
Tuple of (zip_path, mp3_path, cdg_path)
|
||||
|
||||
Raises:
|
||||
ValueError: If CDG styles are not configured
|
||||
FileNotFoundError: If input files are missing
|
||||
Exception: If CDG generation fails
|
||||
"""
|
||||
self.logger.info(f"Creating CDG package for {artist} - {title}")
|
||||
|
||||
# Validate inputs
|
||||
if not os.path.isfile(lrc_file):
|
||||
raise FileNotFoundError(f"LRC file not found: {lrc_file}")
|
||||
if not os.path.isfile(audio_file):
|
||||
raise FileNotFoundError(f"Audio file not found: {audio_file}")
|
||||
|
||||
# Check if ZIP already exists
|
||||
if os.path.isfile(output_zip_path):
|
||||
if self.non_interactive:
|
||||
self.logger.info(
|
||||
f"CDG ZIP exists, will be overwritten: {output_zip_path}"
|
||||
)
|
||||
else:
|
||||
self.logger.info(f"CDG ZIP already exists: {output_zip_path}")
|
||||
|
||||
# Check if individual files exist (allows skipping generation)
|
||||
if output_mp3_path and output_cdg_path:
|
||||
if os.path.isfile(output_mp3_path) and os.path.isfile(output_cdg_path):
|
||||
self.logger.info("Found existing MP3 and CDG files, creating ZIP directly")
|
||||
if not self.dry_run:
|
||||
self._create_zip_from_files(
|
||||
output_zip_path,
|
||||
[(output_mp3_path, os.path.basename(output_mp3_path)),
|
||||
(output_cdg_path, os.path.basename(output_cdg_path))]
|
||||
)
|
||||
return output_zip_path, output_mp3_path, output_cdg_path
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(
|
||||
f"DRY RUN: Would generate CDG package: {output_zip_path}"
|
||||
)
|
||||
return output_zip_path, output_mp3_path, output_cdg_path
|
||||
|
||||
# Generate CDG files
|
||||
if self.cdg_styles is None:
|
||||
raise ValueError(
|
||||
"CDG styles configuration is required for CDG generation"
|
||||
)
|
||||
|
||||
self.logger.info("Generating CDG and MP3 files")
|
||||
from lyrics_transcriber.output.cdg import CDGGenerator
|
||||
|
||||
output_dir = os.path.dirname(output_zip_path) or os.getcwd()
|
||||
generator = CDGGenerator(output_dir=output_dir, logger=self.logger)
|
||||
|
||||
cdg_file, mp3_file, zip_file = generator.generate_cdg_from_lrc(
|
||||
lrc_file=lrc_file,
|
||||
audio_file=audio_file,
|
||||
title=title,
|
||||
artist=artist,
|
||||
cdg_styles=self.cdg_styles,
|
||||
)
|
||||
|
||||
# Rename ZIP to expected output path if different
|
||||
if os.path.isfile(zip_file) and zip_file != output_zip_path:
|
||||
os.rename(zip_file, output_zip_path)
|
||||
self.logger.info(f"Renamed CDG ZIP: {zip_file} -> {output_zip_path}")
|
||||
|
||||
if not os.path.isfile(output_zip_path):
|
||||
raise Exception(f"Failed to create CDG ZIP file: {output_zip_path}")
|
||||
|
||||
# Extract the ZIP to get individual files if paths provided
|
||||
extracted_mp3 = None
|
||||
extracted_cdg = None
|
||||
if output_mp3_path or output_cdg_path:
|
||||
self.logger.info(f"Extracting CDG ZIP file: {output_zip_path}")
|
||||
with zipfile.ZipFile(output_zip_path, "r") as zip_ref:
|
||||
zip_ref.extractall(output_dir)
|
||||
|
||||
# Find extracted files
|
||||
if output_mp3_path and os.path.isfile(output_mp3_path):
|
||||
extracted_mp3 = output_mp3_path
|
||||
self.logger.info(f"Extracted MP3: {extracted_mp3}")
|
||||
if output_cdg_path and os.path.isfile(output_cdg_path):
|
||||
extracted_cdg = output_cdg_path
|
||||
self.logger.info(f"Extracted CDG: {extracted_cdg}")
|
||||
|
||||
self.logger.info(f"CDG package created: {output_zip_path}")
|
||||
return output_zip_path, extracted_mp3, extracted_cdg
|
||||
|
||||
def create_txt_package(
|
||||
self,
|
||||
lrc_file: str,
|
||||
mp3_file: str,
|
||||
output_zip_path: str,
|
||||
output_txt_path: Optional[str] = None,
|
||||
) -> Tuple[str, Optional[str]]:
|
||||
"""
|
||||
Create a TXT package (ZIP containing TXT lyrics and MP3 files).
|
||||
|
||||
Args:
|
||||
lrc_file: Path to the LRC lyrics file
|
||||
mp3_file: Path to the MP3 audio file
|
||||
output_zip_path: Path for the output ZIP file
|
||||
output_txt_path: Optional path for the generated TXT file
|
||||
|
||||
Returns:
|
||||
Tuple of (zip_path, txt_path)
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: If input files are missing
|
||||
Exception: If TXT generation fails
|
||||
"""
|
||||
self.logger.info(f"Creating TXT package from {lrc_file}")
|
||||
|
||||
# Validate inputs
|
||||
if not os.path.isfile(lrc_file):
|
||||
raise FileNotFoundError(f"LRC file not found: {lrc_file}")
|
||||
if not os.path.isfile(mp3_file):
|
||||
raise FileNotFoundError(f"MP3 file not found: {mp3_file}")
|
||||
|
||||
# Check if ZIP already exists
|
||||
if os.path.isfile(output_zip_path):
|
||||
if self.non_interactive:
|
||||
self.logger.info(
|
||||
f"TXT ZIP exists, will be overwritten: {output_zip_path}"
|
||||
)
|
||||
else:
|
||||
self.logger.info(f"TXT ZIP already exists: {output_zip_path}")
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(
|
||||
f"DRY RUN: Would create TXT package: {output_zip_path}"
|
||||
)
|
||||
return output_zip_path, output_txt_path
|
||||
|
||||
# Generate TXT from LRC
|
||||
self.logger.info(f"Converting LRC to TXT format: {lrc_file}")
|
||||
from lyrics_converter import LyricsConverter
|
||||
|
||||
txt_converter = LyricsConverter(output_format="txt", filepath=lrc_file)
|
||||
converted_txt = txt_converter.convert_file()
|
||||
|
||||
# Write TXT file
|
||||
if output_txt_path is None:
|
||||
# Default to same name as ZIP but with .txt extension
|
||||
output_txt_path = output_zip_path.replace(".zip", ".txt")
|
||||
|
||||
with open(output_txt_path, "w") as txt_file:
|
||||
txt_file.write(converted_txt)
|
||||
self.logger.info(f"TXT file written: {output_txt_path}")
|
||||
|
||||
# Create ZIP containing MP3 and TXT
|
||||
self.logger.info(f"Creating TXT ZIP: {output_zip_path}")
|
||||
self._create_zip_from_files(
|
||||
output_zip_path,
|
||||
[(mp3_file, os.path.basename(mp3_file)),
|
||||
(output_txt_path, os.path.basename(output_txt_path))]
|
||||
)
|
||||
|
||||
if not os.path.isfile(output_zip_path):
|
||||
raise Exception(f"Failed to create TXT ZIP file: {output_zip_path}")
|
||||
|
||||
self.logger.info(f"TXT package created: {output_zip_path}")
|
||||
return output_zip_path, output_txt_path
|
||||
|
||||
def _create_zip_from_files(
|
||||
self,
|
||||
zip_path: str,
|
||||
files: list,
|
||||
) -> None:
|
||||
"""
|
||||
Create a ZIP file from a list of files.
|
||||
|
||||
Args:
|
||||
zip_path: Path for the output ZIP file
|
||||
files: List of (file_path, archive_name) tuples
|
||||
"""
|
||||
with zipfile.ZipFile(zip_path, "w") as zipf:
|
||||
for file_path, archive_name in files:
|
||||
if os.path.isfile(file_path):
|
||||
zipf.write(file_path, archive_name)
|
||||
self.logger.debug(f"Added to ZIP: {archive_name}")
|
||||
else:
|
||||
self.logger.warning(f"File not found for ZIP: {file_path}")
|
||||
|
||||
|
||||
# Singleton instance and factory function (following existing service pattern)
|
||||
_packaging_service: Optional[PackagingService] = None
|
||||
|
||||
|
||||
def get_packaging_service(
|
||||
cdg_styles: Optional[Dict[str, Any]] = None,
|
||||
**kwargs
|
||||
) -> PackagingService:
|
||||
"""
|
||||
Get a packaging service instance.
|
||||
|
||||
Args:
|
||||
cdg_styles: CDG style configuration
|
||||
**kwargs: Additional arguments passed to PackagingService
|
||||
|
||||
Returns:
|
||||
PackagingService instance
|
||||
"""
|
||||
global _packaging_service
|
||||
|
||||
# Create new instance if settings changed
|
||||
if _packaging_service is None or cdg_styles:
|
||||
_packaging_service = PackagingService(
|
||||
cdg_styles=cdg_styles,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
return _packaging_service
|
||||
@@ -0,0 +1,106 @@
|
||||
"""
|
||||
Rclone configuration service for cloud storage integration.
|
||||
|
||||
This service manages the rclone configuration needed for Dropbox
|
||||
and other cloud storage uploads from the backend workers.
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
from typing import Optional
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class RcloneService:
|
||||
"""Service for managing rclone configuration."""
|
||||
|
||||
# Secret Manager secret name for rclone config
|
||||
RCLONE_CONFIG_SECRET = "rclone-config"
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
self._config_file: Optional[str] = None
|
||||
self._config_loaded = False
|
||||
|
||||
def setup_rclone_config(self) -> bool:
|
||||
"""
|
||||
Load rclone config from Secret Manager and set up environment.
|
||||
|
||||
Writes the config to a temp file and sets RCLONE_CONFIG env var.
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
if self._config_loaded:
|
||||
logger.debug("Rclone config already loaded")
|
||||
return True
|
||||
|
||||
try:
|
||||
# Get rclone config from Secret Manager
|
||||
config_content = self.settings.get_secret(self.RCLONE_CONFIG_SECRET)
|
||||
|
||||
if not config_content:
|
||||
logger.warning("Rclone config not found in Secret Manager")
|
||||
return False
|
||||
|
||||
# Write to a temp file
|
||||
fd, config_path = tempfile.mkstemp(prefix="rclone_", suffix=".conf")
|
||||
try:
|
||||
with os.fdopen(fd, 'w') as f:
|
||||
f.write(config_content)
|
||||
|
||||
self._config_file = config_path
|
||||
|
||||
# Set environment variable for rclone to find the config
|
||||
os.environ["RCLONE_CONFIG"] = config_path
|
||||
|
||||
logger.info(f"Rclone config loaded and written to {config_path}")
|
||||
self._config_loaded = True
|
||||
return True
|
||||
|
||||
except Exception:
|
||||
# Clean up the temp file on error
|
||||
# Note: os.fdopen() takes ownership of fd, so it's already closed
|
||||
# We only need to remove the temp file if it exists
|
||||
if os.path.exists(config_path):
|
||||
os.unlink(config_path)
|
||||
raise
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to setup rclone config: {e}")
|
||||
return False
|
||||
|
||||
def cleanup(self) -> None:
|
||||
"""Remove the temporary config file."""
|
||||
if self._config_file and os.path.exists(self._config_file):
|
||||
try:
|
||||
os.unlink(self._config_file)
|
||||
logger.debug(f"Cleaned up rclone config file: {self._config_file}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to cleanup rclone config: {e}")
|
||||
|
||||
# Always reset internal state and environment, even if the file was missing
|
||||
if self._config_file is not None:
|
||||
os.environ.pop("RCLONE_CONFIG", None)
|
||||
self._config_file = None
|
||||
self._config_loaded = False
|
||||
|
||||
@property
|
||||
def is_configured(self) -> bool:
|
||||
"""Check if rclone is configured and ready to use."""
|
||||
return self._config_loaded and self._config_file is not None
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_rclone_service: Optional[RcloneService] = None
|
||||
|
||||
|
||||
def get_rclone_service() -> RcloneService:
|
||||
"""Get the singleton rclone service instance."""
|
||||
global _rclone_service
|
||||
if _rclone_service is None:
|
||||
_rclone_service = RcloneService()
|
||||
return _rclone_service
|
||||
@@ -0,0 +1,209 @@
|
||||
"""
|
||||
Google Cloud Storage operations for file management.
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
import json
|
||||
from typing import Optional, BinaryIO, Any, Dict
|
||||
from pathlib import Path
|
||||
from google.cloud import storage
|
||||
from datetime import timedelta
|
||||
|
||||
from backend.config import settings
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class StorageService:
|
||||
"""Service for Google Cloud Storage operations."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize GCS client."""
|
||||
self.client = storage.Client(project=settings.google_cloud_project)
|
||||
self.bucket = self.client.bucket(settings.gcs_bucket_name)
|
||||
|
||||
def upload_file(self, local_path: str, destination_path: str) -> str:
|
||||
"""Upload a file to GCS."""
|
||||
try:
|
||||
blob = self.bucket.blob(destination_path)
|
||||
blob.upload_from_filename(local_path)
|
||||
logger.info(f"Uploaded {local_path} to gs://{settings.gcs_bucket_name}/{destination_path}")
|
||||
return destination_path
|
||||
except Exception as e:
|
||||
logger.error(f"Error uploading file {local_path}: {e}")
|
||||
raise
|
||||
|
||||
def upload_fileobj(self, file_obj: BinaryIO, destination_path: str, content_type: Optional[str] = None) -> str:
|
||||
"""Upload a file object to GCS."""
|
||||
try:
|
||||
blob = self.bucket.blob(destination_path)
|
||||
if content_type:
|
||||
blob.content_type = content_type
|
||||
blob.upload_from_file(file_obj, rewind=True)
|
||||
logger.info(f"Uploaded file object to gs://{settings.gcs_bucket_name}/{destination_path}")
|
||||
return destination_path
|
||||
except Exception as e:
|
||||
logger.error(f"Error uploading file object: {e}")
|
||||
raise
|
||||
|
||||
def download_file(self, source_path: str, destination_path: str) -> str:
|
||||
"""Download a file from GCS."""
|
||||
try:
|
||||
blob = self.bucket.blob(source_path)
|
||||
blob.download_to_filename(destination_path)
|
||||
logger.info(f"Downloaded gs://{settings.gcs_bucket_name}/{source_path} to {destination_path}")
|
||||
return destination_path
|
||||
except Exception as e:
|
||||
logger.error(f"Error downloading file {source_path}: {e}")
|
||||
raise
|
||||
|
||||
def generate_signed_url(self, blob_path: str, expiration_minutes: int = 60) -> str:
|
||||
"""Generate a signed URL for downloading a file.
|
||||
|
||||
In Cloud Run, this uses the IAM signBlob API since we don't have
|
||||
a private key available. Requires the service account to have
|
||||
roles/iam.serviceAccountTokenCreator on itself.
|
||||
"""
|
||||
return self._generate_signed_url_internal(blob_path, "GET", expiration_minutes)
|
||||
|
||||
def generate_signed_upload_url(self, blob_path: str, content_type: str = "application/octet-stream", expiration_minutes: int = 60) -> str:
|
||||
"""Generate a signed URL for uploading a file directly to GCS.
|
||||
|
||||
This allows clients to upload files directly to GCS without going through
|
||||
the backend, bypassing any request body size limits.
|
||||
|
||||
Args:
|
||||
blob_path: The destination path in GCS
|
||||
content_type: The expected content type of the upload
|
||||
expiration_minutes: How long the URL is valid for
|
||||
|
||||
Returns:
|
||||
A signed URL that accepts PUT requests with the file content
|
||||
"""
|
||||
return self._generate_signed_url_internal(blob_path, "PUT", expiration_minutes, content_type)
|
||||
|
||||
def _generate_signed_url_internal(self, blob_path: str, method: str, expiration_minutes: int = 60, content_type: Optional[str] = None) -> str:
|
||||
"""Internal method to generate signed URLs for GET or PUT operations."""
|
||||
import google.auth
|
||||
from google.auth.transport import requests
|
||||
|
||||
try:
|
||||
blob = self.bucket.blob(blob_path)
|
||||
|
||||
# Get default credentials and refresh to ensure we have a valid token
|
||||
credentials, project = google.auth.default()
|
||||
|
||||
# Common kwargs for signed URL generation
|
||||
kwargs = {
|
||||
"version": "v4",
|
||||
"expiration": timedelta(minutes=expiration_minutes),
|
||||
"method": method,
|
||||
}
|
||||
|
||||
# For PUT requests, we need to specify the content type in headers
|
||||
if method == "PUT" and content_type:
|
||||
kwargs["headers"] = {"Content-Type": content_type}
|
||||
|
||||
# Check if we're using compute credentials (Cloud Run/GCE)
|
||||
# These need to use IAM signBlob via service_account_email + access_token
|
||||
if hasattr(credentials, 'service_account_email'):
|
||||
# Refresh credentials to get a valid access token
|
||||
auth_request = requests.Request()
|
||||
credentials.refresh(auth_request)
|
||||
|
||||
kwargs["service_account_email"] = credentials.service_account_email
|
||||
kwargs["access_token"] = credentials.token
|
||||
|
||||
url = blob.generate_signed_url(**kwargs)
|
||||
|
||||
logger.info(f"Generated signed {method} URL for {blob_path}")
|
||||
return url
|
||||
except Exception as e:
|
||||
logger.error(f"Error generating signed {method} URL for {blob_path}: {e}")
|
||||
raise
|
||||
|
||||
def delete_file(self, blob_path: str) -> None:
|
||||
"""Delete a file from GCS."""
|
||||
try:
|
||||
blob = self.bucket.blob(blob_path)
|
||||
blob.delete()
|
||||
logger.info(f"Deleted gs://{settings.gcs_bucket_name}/{blob_path}")
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting file {blob_path}: {e}")
|
||||
raise
|
||||
|
||||
def delete_folder(self, prefix: str) -> int:
|
||||
"""
|
||||
Delete all files in GCS with a given prefix (folder).
|
||||
|
||||
Args:
|
||||
prefix: The folder prefix to delete (e.g., "uploads/abc123/")
|
||||
|
||||
Returns:
|
||||
Number of files deleted
|
||||
"""
|
||||
try:
|
||||
blobs = list(self.bucket.list_blobs(prefix=prefix))
|
||||
deleted_count = 0
|
||||
|
||||
for blob in blobs:
|
||||
try:
|
||||
blob.delete()
|
||||
deleted_count += 1
|
||||
except Exception as e:
|
||||
logger.warning(f"Error deleting blob {blob.name}: {e}")
|
||||
|
||||
if deleted_count > 0:
|
||||
logger.info(f"Deleted {deleted_count} files from gs://{settings.gcs_bucket_name}/{prefix}")
|
||||
|
||||
return deleted_count
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting folder {prefix}: {e}")
|
||||
return 0 # Don't raise - folder deletion shouldn't break operations
|
||||
|
||||
def list_files(self, prefix: str) -> list:
|
||||
"""List files in GCS with a given prefix."""
|
||||
try:
|
||||
blobs = self.bucket.list_blobs(prefix=prefix)
|
||||
return [blob.name for blob in blobs]
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing files with prefix {prefix}: {e}")
|
||||
raise
|
||||
|
||||
def file_exists(self, blob_path: str) -> bool:
|
||||
"""Check if a file exists in GCS."""
|
||||
try:
|
||||
blob = self.bucket.blob(blob_path)
|
||||
return blob.exists()
|
||||
except Exception as e:
|
||||
logger.error(f"Error checking file existence {blob_path}: {e}")
|
||||
raise
|
||||
|
||||
def upload_json(self, destination_path: str, data: Dict[str, Any]) -> str:
|
||||
"""Upload a JSON object to GCS."""
|
||||
try:
|
||||
blob = self.bucket.blob(destination_path)
|
||||
blob.content_type = "application/json"
|
||||
blob.upload_from_string(
|
||||
json.dumps(data, indent=2, ensure_ascii=False),
|
||||
content_type="application/json"
|
||||
)
|
||||
logger.info(f"Uploaded JSON to gs://{settings.gcs_bucket_name}/{destination_path}")
|
||||
return destination_path
|
||||
except Exception as e:
|
||||
logger.error(f"Error uploading JSON to {destination_path}: {e}")
|
||||
raise
|
||||
|
||||
def download_json(self, source_path: str) -> Dict[str, Any]:
|
||||
"""Download and parse a JSON file from GCS."""
|
||||
try:
|
||||
blob = self.bucket.blob(source_path)
|
||||
content = blob.download_as_text()
|
||||
data = json.loads(content)
|
||||
logger.info(f"Downloaded JSON from gs://{settings.gcs_bucket_name}/{source_path}")
|
||||
return data
|
||||
except Exception as e:
|
||||
logger.error(f"Error downloading JSON from {source_path}: {e}")
|
||||
raise
|
||||
|
||||
@@ -0,0 +1,275 @@
|
||||
"""
|
||||
Stripe service for payment processing.
|
||||
|
||||
Handles:
|
||||
- Creating checkout sessions for credit purchases
|
||||
- Processing webhook events
|
||||
- Managing customer records
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
from typing import Optional, Dict, Any, Tuple
|
||||
|
||||
import stripe
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Credit packages available for purchase
|
||||
CREDIT_PACKAGES = {
|
||||
"1_credit": {
|
||||
"credits": 1,
|
||||
"price_cents": 500, # $5.00
|
||||
"name": "1 Karaoke Credit",
|
||||
"description": "Create 1 professional karaoke video",
|
||||
},
|
||||
"3_credits": {
|
||||
"credits": 3,
|
||||
"price_cents": 1200, # $12.00 (20% discount)
|
||||
"name": "3 Karaoke Credits",
|
||||
"description": "Create 3 professional karaoke videos (Save 20%)",
|
||||
},
|
||||
"5_credits": {
|
||||
"credits": 5,
|
||||
"price_cents": 1750, # $17.50 (30% discount)
|
||||
"name": "5 Karaoke Credits",
|
||||
"description": "Create 5 professional karaoke videos (Save 30%)",
|
||||
},
|
||||
"10_credits": {
|
||||
"credits": 10,
|
||||
"price_cents": 3000, # $30.00 (40% discount)
|
||||
"name": "10 Karaoke Credits",
|
||||
"description": "Create 10 professional karaoke videos (Save 40%)",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class StripeService:
|
||||
"""Service for Stripe payment processing."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize Stripe with API key."""
|
||||
self.settings = get_settings()
|
||||
self.secret_key = os.getenv("STRIPE_SECRET_KEY")
|
||||
self.webhook_secret = os.getenv("STRIPE_WEBHOOK_SECRET")
|
||||
self.frontend_url = os.getenv("FRONTEND_URL", "https://gen.nomadkaraoke.com")
|
||||
# After consolidation, buy URL is the same as frontend URL
|
||||
self.buy_url = os.getenv("BUY_URL", self.frontend_url)
|
||||
|
||||
if self.secret_key:
|
||||
stripe.api_key = self.secret_key
|
||||
logger.info("Stripe initialized with API key")
|
||||
else:
|
||||
logger.warning("STRIPE_SECRET_KEY not set - payments disabled")
|
||||
|
||||
def is_configured(self) -> bool:
|
||||
"""Check if Stripe is properly configured."""
|
||||
return bool(self.secret_key)
|
||||
|
||||
def get_credit_packages(self) -> Dict[str, Dict[str, Any]]:
|
||||
"""Get available credit packages."""
|
||||
return CREDIT_PACKAGES
|
||||
|
||||
def create_checkout_session(
|
||||
self,
|
||||
package_id: str,
|
||||
user_email: str,
|
||||
success_url: Optional[str] = None,
|
||||
cancel_url: Optional[str] = None,
|
||||
) -> Tuple[bool, Optional[str], str]:
|
||||
"""
|
||||
Create a Stripe Checkout session for purchasing credits.
|
||||
|
||||
Args:
|
||||
package_id: ID of the credit package to purchase
|
||||
user_email: Email of the purchasing user
|
||||
success_url: URL to redirect to on success (optional)
|
||||
cancel_url: URL to redirect to on cancel (optional)
|
||||
|
||||
Returns:
|
||||
(success, checkout_url, message)
|
||||
"""
|
||||
if not self.is_configured():
|
||||
return False, None, "Payment processing is not configured"
|
||||
|
||||
package = CREDIT_PACKAGES.get(package_id)
|
||||
if not package:
|
||||
return False, None, f"Invalid package: {package_id}"
|
||||
|
||||
try:
|
||||
# Default URLs
|
||||
if not success_url:
|
||||
success_url = f"{self.frontend_url}/payment/success?session_id={{CHECKOUT_SESSION_ID}}"
|
||||
if not cancel_url:
|
||||
cancel_url = f"{self.buy_url}?cancelled=true"
|
||||
|
||||
# Create checkout session
|
||||
session = stripe.checkout.Session.create(
|
||||
payment_method_types=['card'],
|
||||
line_items=[{
|
||||
'price_data': {
|
||||
'currency': 'usd',
|
||||
'product_data': {
|
||||
'name': package['name'],
|
||||
'description': package['description'],
|
||||
},
|
||||
'unit_amount': package['price_cents'],
|
||||
},
|
||||
'quantity': 1,
|
||||
}],
|
||||
mode='payment',
|
||||
success_url=success_url,
|
||||
cancel_url=cancel_url,
|
||||
customer_email=user_email,
|
||||
metadata={
|
||||
'package_id': package_id,
|
||||
'credits': str(package['credits']),
|
||||
'user_email': user_email,
|
||||
},
|
||||
# Allow promotion codes
|
||||
allow_promotion_codes=True,
|
||||
)
|
||||
|
||||
logger.info(f"Created checkout session {session.id} for {user_email}, package {package_id}")
|
||||
return True, session.url, "Checkout session created"
|
||||
|
||||
except stripe.error.StripeError as e:
|
||||
logger.error(f"Stripe error creating checkout session: {e}")
|
||||
return False, None, f"Payment error: {str(e)}"
|
||||
except Exception as e:
|
||||
logger.error(f"Error creating checkout session: {e}")
|
||||
return False, None, "Failed to create checkout session"
|
||||
|
||||
def verify_webhook_signature(self, payload: bytes, signature: str) -> Tuple[bool, Optional[Dict], str]:
|
||||
"""
|
||||
Verify a Stripe webhook signature.
|
||||
|
||||
Args:
|
||||
payload: Raw request body
|
||||
signature: Stripe-Signature header value
|
||||
|
||||
Returns:
|
||||
(valid, event_data, message)
|
||||
"""
|
||||
if not self.webhook_secret:
|
||||
logger.error("STRIPE_WEBHOOK_SECRET not configured")
|
||||
return False, None, "Webhook secret not configured"
|
||||
|
||||
try:
|
||||
event = stripe.Webhook.construct_event(
|
||||
payload, signature, self.webhook_secret
|
||||
)
|
||||
return True, event, "Webhook verified"
|
||||
except stripe.error.SignatureVerificationError as e:
|
||||
logger.error(f"Invalid webhook signature: {e}")
|
||||
return False, None, "Invalid signature"
|
||||
except Exception as e:
|
||||
logger.error(f"Error verifying webhook: {e}")
|
||||
return False, None, str(e)
|
||||
|
||||
def handle_checkout_completed(self, session: Dict) -> Tuple[bool, Optional[str], int, str]:
|
||||
"""
|
||||
Handle a completed checkout session.
|
||||
|
||||
Args:
|
||||
session: Stripe checkout session object
|
||||
|
||||
Returns:
|
||||
(success, user_email, credits_to_add, message)
|
||||
"""
|
||||
try:
|
||||
metadata = session.get('metadata', {})
|
||||
user_email = metadata.get('user_email') or session.get('customer_email')
|
||||
try:
|
||||
credits = int(metadata.get('credits', 0))
|
||||
except (ValueError, TypeError):
|
||||
logger.error(f"Invalid credits metadata in session {session.get('id')}: {metadata.get('credits')}")
|
||||
return False, user_email, 0, "Invalid credit amount in session metadata"
|
||||
package_id = metadata.get('package_id')
|
||||
|
||||
if not user_email:
|
||||
logger.error(f"No user email in checkout session {session.get('id')}")
|
||||
return False, None, 0, "No user email found"
|
||||
|
||||
if credits <= 0:
|
||||
logger.error(f"Invalid credits in checkout session {session.get('id')}")
|
||||
return False, user_email, 0, "Invalid credit amount"
|
||||
|
||||
logger.info(
|
||||
f"Checkout completed: {user_email} purchased {credits} credits "
|
||||
f"(package: {package_id}, session: {session.get('id')})"
|
||||
)
|
||||
|
||||
return True, user_email, credits, f"Successfully purchased {credits} credits"
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error handling checkout completed: {e}")
|
||||
return False, None, 0, str(e)
|
||||
|
||||
def get_session(self, session_id: str) -> Optional[Dict]:
|
||||
"""
|
||||
Get a checkout session by ID.
|
||||
|
||||
Args:
|
||||
session_id: Stripe checkout session ID
|
||||
|
||||
Returns:
|
||||
Session data or None
|
||||
"""
|
||||
if not self.is_configured():
|
||||
return None
|
||||
|
||||
try:
|
||||
session = stripe.checkout.Session.retrieve(session_id)
|
||||
return dict(session)
|
||||
except Exception as e:
|
||||
logger.error(f"Error retrieving session {session_id}: {e}")
|
||||
return None
|
||||
|
||||
def create_customer(self, email: str, name: Optional[str] = None) -> Optional[str]:
|
||||
"""
|
||||
Create or get a Stripe customer for a user.
|
||||
|
||||
Args:
|
||||
email: User's email
|
||||
name: User's display name (optional)
|
||||
|
||||
Returns:
|
||||
Stripe customer ID or None
|
||||
"""
|
||||
if not self.is_configured():
|
||||
return None
|
||||
|
||||
try:
|
||||
# Check if customer already exists
|
||||
customers = stripe.Customer.list(email=email, limit=1)
|
||||
if customers.data:
|
||||
return customers.data[0].id
|
||||
|
||||
# Create new customer
|
||||
customer = stripe.Customer.create(
|
||||
email=email,
|
||||
name=name,
|
||||
metadata={'source': 'nomad_karaoke'},
|
||||
)
|
||||
logger.info(f"Created Stripe customer {customer.id} for {email}")
|
||||
return customer.id
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error creating Stripe customer: {e}")
|
||||
return None
|
||||
|
||||
|
||||
# Global instance
|
||||
_stripe_service = None
|
||||
|
||||
|
||||
def get_stripe_service() -> StripeService:
|
||||
"""Get the global Stripe service instance."""
|
||||
global _stripe_service
|
||||
if _stripe_service is None:
|
||||
_stripe_service = StripeService()
|
||||
return _stripe_service
|
||||
@@ -0,0 +1,254 @@
|
||||
"""
|
||||
Structured logging with trace correlation for Cloud Logging.
|
||||
|
||||
This module provides JSON-formatted logging that integrates with Google Cloud Logging
|
||||
and correlates logs with OpenTelemetry traces in Cloud Trace.
|
||||
|
||||
When running in Cloud Run:
|
||||
- Logs are output as JSON for Cloud Logging to parse
|
||||
- Each log entry includes trace ID and span ID for correlation
|
||||
- Custom fields (job_id, worker) are preserved in the log structure
|
||||
|
||||
Usage:
|
||||
# In main.py, before any logging:
|
||||
from backend.services.structured_logging import setup_structured_logging
|
||||
setup_structured_logging()
|
||||
|
||||
# Then use standard logging:
|
||||
logger = logging.getLogger(__name__)
|
||||
logger.info("Processing started", extra={"job_id": "abc123"})
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
|
||||
# Cloud Logging severity mapping (compatible with GCP)
|
||||
# https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogSeverity
|
||||
SEVERITY_MAP = {
|
||||
"DEBUG": "DEBUG",
|
||||
"INFO": "INFO",
|
||||
"WARNING": "WARNING",
|
||||
"ERROR": "ERROR",
|
||||
"CRITICAL": "CRITICAL",
|
||||
}
|
||||
|
||||
|
||||
class StructuredFormatter(logging.Formatter):
|
||||
"""
|
||||
JSON formatter with trace correlation for Google Cloud Logging.
|
||||
|
||||
Produces log entries in the format expected by Cloud Logging, including:
|
||||
- logging.googleapis.com/trace: Links to Cloud Trace
|
||||
- logging.googleapis.com/spanId: Current span ID
|
||||
- severity: Cloud Logging severity level
|
||||
- Custom fields passed via 'extra' dict
|
||||
"""
|
||||
|
||||
def __init__(self, project_id: Optional[str] = None):
|
||||
"""
|
||||
Initialize the structured formatter.
|
||||
|
||||
Args:
|
||||
project_id: GCP project ID for trace URLs (auto-detected if not provided)
|
||||
"""
|
||||
super().__init__()
|
||||
self.project_id = project_id or os.environ.get("GOOGLE_CLOUD_PROJECT") or os.environ.get("GCP_PROJECT")
|
||||
|
||||
def format(self, record: logging.LogRecord) -> str:
|
||||
"""
|
||||
Format a log record as JSON.
|
||||
|
||||
Args:
|
||||
record: The log record to format
|
||||
|
||||
Returns:
|
||||
JSON string for Cloud Logging
|
||||
"""
|
||||
from backend.services.tracing import get_current_trace_id, get_current_span_id
|
||||
|
||||
# Build base log entry
|
||||
log_entry: Dict[str, Any] = {
|
||||
"timestamp": datetime.utcfromtimestamp(record.created).isoformat() + "Z",
|
||||
"severity": SEVERITY_MAP.get(record.levelname, record.levelname),
|
||||
"message": record.getMessage(),
|
||||
"logger": record.name,
|
||||
}
|
||||
|
||||
# Add trace correlation if available
|
||||
trace_id = get_current_trace_id()
|
||||
span_id = get_current_span_id()
|
||||
|
||||
if trace_id and self.project_id:
|
||||
# Format for Cloud Logging trace correlation
|
||||
log_entry["logging.googleapis.com/trace"] = f"projects/{self.project_id}/traces/{trace_id}"
|
||||
|
||||
if span_id:
|
||||
log_entry["logging.googleapis.com/spanId"] = span_id
|
||||
|
||||
# Add custom fields from 'extra' dict
|
||||
# Common fields we want to extract from log records
|
||||
custom_fields = [
|
||||
# Job-related fields
|
||||
"job_id", "worker", "operation", "duration", "status", "error",
|
||||
# Audit logging fields (from middleware and auth)
|
||||
"request_id", "user_email", "client_ip", "latency_ms",
|
||||
"audit_type", "method", "path", "status_code", "query_string",
|
||||
"user_agent", "user_type", "is_admin", "remaining_uses",
|
||||
"auth_message", "token_provided", "token_length", "auth_header_present",
|
||||
]
|
||||
for field in custom_fields:
|
||||
value = getattr(record, field, None)
|
||||
if value is not None:
|
||||
log_entry[field] = value
|
||||
|
||||
# Add source location for debugging
|
||||
if record.pathname and record.lineno:
|
||||
log_entry["logging.googleapis.com/sourceLocation"] = {
|
||||
"file": record.pathname,
|
||||
"line": record.lineno,
|
||||
"function": record.funcName,
|
||||
}
|
||||
|
||||
# Add exception info if present
|
||||
if record.exc_info:
|
||||
log_entry["exception"] = self.formatException(record.exc_info)
|
||||
|
||||
# Remove None values for cleaner output
|
||||
return json.dumps({k: v for k, v in log_entry.items() if v is not None})
|
||||
|
||||
|
||||
class HumanReadableFormatter(logging.Formatter):
|
||||
"""
|
||||
Human-readable formatter for local development.
|
||||
|
||||
Includes trace context when available but outputs in traditional format.
|
||||
"""
|
||||
|
||||
def format(self, record: logging.LogRecord) -> str:
|
||||
"""Format a log record for human readability."""
|
||||
from backend.services.tracing import get_current_trace_id
|
||||
|
||||
# Build base message
|
||||
timestamp = datetime.utcfromtimestamp(record.created).strftime("%Y-%m-%d %H:%M:%S")
|
||||
|
||||
# Add job_id prefix if available
|
||||
job_id = getattr(record, "job_id", None)
|
||||
job_prefix = f"[job:{job_id[:8]}] " if job_id else ""
|
||||
|
||||
# Add trace ID suffix if available
|
||||
trace_id = get_current_trace_id()
|
||||
trace_suffix = f" [trace:{trace_id[:8]}]" if trace_id else ""
|
||||
|
||||
message = f"{timestamp} - {record.name} - {record.levelname} - {job_prefix}{record.getMessage()}{trace_suffix}"
|
||||
|
||||
# Add exception info if present
|
||||
if record.exc_info:
|
||||
message += "\n" + self.formatException(record.exc_info)
|
||||
|
||||
return message
|
||||
|
||||
|
||||
def is_cloud_run() -> bool:
|
||||
"""Check if we're running in Cloud Run."""
|
||||
return os.environ.get("K_SERVICE") is not None
|
||||
|
||||
|
||||
def setup_structured_logging(force_json: bool = False, log_level: Optional[str] = None) -> None:
|
||||
"""
|
||||
Configure structured logging for the application.
|
||||
|
||||
In Cloud Run: Uses JSON format with trace correlation for Cloud Logging
|
||||
In development: Uses human-readable format with optional trace context
|
||||
|
||||
Args:
|
||||
force_json: Force JSON output even in development
|
||||
log_level: Override log level (default from settings or INFO)
|
||||
"""
|
||||
from backend.config import settings
|
||||
|
||||
# Determine log level
|
||||
level = log_level or getattr(settings, "log_level", "INFO")
|
||||
level = getattr(logging, level.upper(), logging.INFO)
|
||||
|
||||
# Choose formatter based on environment
|
||||
if is_cloud_run() or force_json:
|
||||
formatter = StructuredFormatter()
|
||||
else:
|
||||
formatter = HumanReadableFormatter()
|
||||
|
||||
# Create handler
|
||||
handler = logging.StreamHandler(sys.stdout)
|
||||
handler.setFormatter(formatter)
|
||||
handler.setLevel(level)
|
||||
|
||||
# Configure root logger
|
||||
root_logger = logging.getLogger()
|
||||
root_logger.setLevel(level)
|
||||
|
||||
# Remove existing handlers and add our handler
|
||||
root_logger.handlers = [handler]
|
||||
|
||||
# Also configure uvicorn loggers to use our format
|
||||
for logger_name in ["uvicorn", "uvicorn.access", "uvicorn.error"]:
|
||||
logger = logging.getLogger(logger_name)
|
||||
logger.handlers = [handler]
|
||||
logger.propagate = False
|
||||
|
||||
|
||||
class JobLogAdapter(logging.LoggerAdapter):
|
||||
"""
|
||||
Logger adapter that automatically adds job context to log records.
|
||||
|
||||
Usage:
|
||||
logger = logging.getLogger(__name__)
|
||||
job_logger = JobLogAdapter(logger, job_id="abc123", worker="audio")
|
||||
job_logger.info("Processing started") # Automatically includes job_id and worker
|
||||
"""
|
||||
|
||||
def __init__(self, logger: logging.Logger, job_id: str, worker: Optional[str] = None, **extra):
|
||||
"""
|
||||
Initialize the adapter.
|
||||
|
||||
Args:
|
||||
logger: Base logger instance
|
||||
job_id: Job ID to include in all log records
|
||||
worker: Optional worker name
|
||||
**extra: Additional fields to include in all log records
|
||||
"""
|
||||
super().__init__(logger, {"job_id": job_id, "worker": worker, **extra})
|
||||
|
||||
def process(self, msg: str, kwargs: Dict[str, Any]) -> tuple:
|
||||
"""Process a logging call to add extra context."""
|
||||
# Merge extra dict from adapter with any extra passed to the log call
|
||||
extra = kwargs.get("extra", {})
|
||||
extra.update(self.extra)
|
||||
kwargs["extra"] = extra
|
||||
return msg, kwargs
|
||||
|
||||
|
||||
def get_job_logger(job_id: str, worker: Optional[str] = None, name: Optional[str] = None) -> JobLogAdapter:
|
||||
"""
|
||||
Get a logger adapter configured for a specific job.
|
||||
|
||||
This is a convenience function for creating job-specific loggers.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to include in all log records
|
||||
worker: Optional worker name
|
||||
name: Logger name (defaults to "job")
|
||||
|
||||
Returns:
|
||||
JobLogAdapter configured for the job
|
||||
|
||||
Usage:
|
||||
logger = get_job_logger("abc123", "audio")
|
||||
logger.info("Starting audio separation")
|
||||
logger.error("Failed to process", extra={"error": str(e)})
|
||||
"""
|
||||
base_logger = logging.getLogger(name or "job")
|
||||
return JobLogAdapter(base_logger, job_id=job_id, worker=worker)
|
||||
|
||||
@@ -0,0 +1,469 @@
|
||||
"""
|
||||
Theme service for managing pre-created style themes.
|
||||
|
||||
Themes are stored in GCS at themes/{theme_id}/ with:
|
||||
- style_params.json: Complete style configuration
|
||||
- preview.png: Preview image for theme selection UI
|
||||
- youtube_description.txt: Optional YouTube description template
|
||||
- assets/: Fonts, backgrounds, and other assets
|
||||
|
||||
The service provides:
|
||||
- Theme listing with signed preview URLs
|
||||
- Theme detail retrieval
|
||||
- Color override application
|
||||
- Job style preparation (copying theme to job folder)
|
||||
"""
|
||||
|
||||
import copy
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
import threading
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from backend.models.theme import (
|
||||
ColorOverrides,
|
||||
ThemeDetail,
|
||||
ThemeMetadata,
|
||||
ThemeRegistry,
|
||||
ThemeSummary,
|
||||
hex_to_rgba,
|
||||
)
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# GCS paths for themes
|
||||
THEMES_PREFIX = "themes"
|
||||
METADATA_FILE = f"{THEMES_PREFIX}/_metadata.json"
|
||||
|
||||
|
||||
class ThemeService:
|
||||
"""Service for managing themes from GCS."""
|
||||
|
||||
def __init__(self, storage: Optional[StorageService] = None):
|
||||
"""
|
||||
Initialize the theme service.
|
||||
|
||||
Args:
|
||||
storage: StorageService instance (creates new one if not provided)
|
||||
"""
|
||||
self.storage = storage or StorageService()
|
||||
self._metadata_cache: Optional[ThemeRegistry] = None
|
||||
self._cache_time: Optional[datetime] = None
|
||||
self.CACHE_TTL_SECONDS = 300 # 5 minute cache for theme metadata
|
||||
|
||||
def _get_theme_path(self, theme_id: str, filename: str = "") -> str:
|
||||
"""Get the GCS path for a theme file."""
|
||||
if filename:
|
||||
return f"{THEMES_PREFIX}/{theme_id}/{filename}"
|
||||
return f"{THEMES_PREFIX}/{theme_id}"
|
||||
|
||||
def _is_cache_valid(self) -> bool:
|
||||
"""Check if the metadata cache is still valid."""
|
||||
if self._metadata_cache is None or self._cache_time is None:
|
||||
return False
|
||||
age = datetime.now() - self._cache_time
|
||||
return age.total_seconds() < self.CACHE_TTL_SECONDS
|
||||
|
||||
def _load_metadata(self, force_refresh: bool = False) -> ThemeRegistry:
|
||||
"""
|
||||
Load theme metadata from GCS with caching.
|
||||
|
||||
Args:
|
||||
force_refresh: Force reload from GCS even if cache is valid
|
||||
|
||||
Returns:
|
||||
ThemeRegistry containing all theme metadata
|
||||
"""
|
||||
if not force_refresh and self._is_cache_valid():
|
||||
return self._metadata_cache # type: ignore
|
||||
|
||||
try:
|
||||
data = self.storage.download_json(METADATA_FILE)
|
||||
self._metadata_cache = ThemeRegistry(**data)
|
||||
self._cache_time = datetime.now()
|
||||
logger.info(f"Loaded theme metadata: {len(self._metadata_cache.themes)} themes")
|
||||
return self._metadata_cache
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load theme metadata: {e}")
|
||||
# Return empty registry on error
|
||||
return ThemeRegistry(version=1, themes=[])
|
||||
|
||||
def list_themes(self) -> List[ThemeSummary]:
|
||||
"""
|
||||
List all available themes with signed preview URLs.
|
||||
|
||||
Returns:
|
||||
List of ThemeSummary objects with preview URLs
|
||||
"""
|
||||
metadata = self._load_metadata()
|
||||
summaries = []
|
||||
|
||||
for theme in metadata.themes:
|
||||
try:
|
||||
# Generate signed URLs for preview images
|
||||
preview_path = self._get_theme_path(theme.id, "preview.png")
|
||||
thumbnail_path = self._get_theme_path(theme.id, "preview_thumbnail.png")
|
||||
|
||||
preview_url = None
|
||||
thumbnail_url = None
|
||||
|
||||
if self.storage.file_exists(preview_path):
|
||||
preview_url = self.storage.generate_signed_url(preview_path, expiration_minutes=60)
|
||||
|
||||
# Try thumbnail, fall back to main preview
|
||||
if self.storage.file_exists(thumbnail_path):
|
||||
thumbnail_url = self.storage.generate_signed_url(thumbnail_path, expiration_minutes=60)
|
||||
elif preview_url:
|
||||
thumbnail_url = preview_url
|
||||
|
||||
summaries.append(
|
||||
ThemeSummary(
|
||||
id=theme.id,
|
||||
name=theme.name,
|
||||
description=theme.description,
|
||||
preview_url=preview_url,
|
||||
thumbnail_url=thumbnail_url,
|
||||
is_default=theme.is_default,
|
||||
)
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Error processing theme {theme.id}: {e}")
|
||||
# Still include theme but without preview URLs
|
||||
summaries.append(
|
||||
ThemeSummary(
|
||||
id=theme.id,
|
||||
name=theme.name,
|
||||
description=theme.description,
|
||||
is_default=theme.is_default,
|
||||
)
|
||||
)
|
||||
|
||||
return summaries
|
||||
|
||||
def get_theme(self, theme_id: str) -> Optional[ThemeDetail]:
|
||||
"""
|
||||
Get full theme details including style parameters.
|
||||
|
||||
Args:
|
||||
theme_id: The theme identifier
|
||||
|
||||
Returns:
|
||||
ThemeDetail if found, None otherwise
|
||||
"""
|
||||
metadata = self._load_metadata()
|
||||
|
||||
# Find theme in metadata
|
||||
theme_meta = next((t for t in metadata.themes if t.id == theme_id), None)
|
||||
if not theme_meta:
|
||||
logger.warning(f"Theme not found: {theme_id}")
|
||||
return None
|
||||
|
||||
try:
|
||||
# Load style_params.json
|
||||
style_params_path = self._get_theme_path(theme_id, "style_params.json")
|
||||
style_params = self.storage.download_json(style_params_path)
|
||||
|
||||
# Check for YouTube description
|
||||
youtube_desc_path = self._get_theme_path(theme_id, "youtube_description.txt")
|
||||
has_youtube_desc = self.storage.file_exists(youtube_desc_path)
|
||||
|
||||
# Generate preview URL
|
||||
preview_path = self._get_theme_path(theme_id, "preview.png")
|
||||
preview_url = None
|
||||
if self.storage.file_exists(preview_path):
|
||||
preview_url = self.storage.generate_signed_url(preview_path, expiration_minutes=60)
|
||||
|
||||
return ThemeDetail(
|
||||
id=theme_id,
|
||||
name=theme_meta.name,
|
||||
description=theme_meta.description,
|
||||
preview_url=preview_url,
|
||||
is_default=theme_meta.is_default,
|
||||
style_params=style_params,
|
||||
has_youtube_description=has_youtube_desc,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Error loading theme {theme_id}: {e}")
|
||||
return None
|
||||
|
||||
def get_theme_style_params(self, theme_id: str) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Get just the style_params.json for a theme.
|
||||
|
||||
Args:
|
||||
theme_id: The theme identifier
|
||||
|
||||
Returns:
|
||||
Style parameters dict if found, None otherwise
|
||||
"""
|
||||
try:
|
||||
style_params_path = self._get_theme_path(theme_id, "style_params.json")
|
||||
return self.storage.download_json(style_params_path)
|
||||
except Exception as e:
|
||||
logger.error(f"Error loading style params for theme {theme_id}: {e}")
|
||||
return None
|
||||
|
||||
def get_youtube_description(self, theme_id: str) -> Optional[str]:
|
||||
"""
|
||||
Get the YouTube description template for a theme.
|
||||
|
||||
Args:
|
||||
theme_id: The theme identifier
|
||||
|
||||
Returns:
|
||||
YouTube description text if found, None otherwise
|
||||
"""
|
||||
try:
|
||||
youtube_desc_path = self._get_theme_path(theme_id, "youtube_description.txt")
|
||||
if not self.storage.file_exists(youtube_desc_path):
|
||||
return None
|
||||
|
||||
# Download to temp file and read
|
||||
with tempfile.NamedTemporaryFile(mode="w", delete=False, suffix=".txt") as tmp:
|
||||
tmp_path = tmp.name
|
||||
|
||||
try:
|
||||
self.storage.download_file(youtube_desc_path, tmp_path)
|
||||
with open(tmp_path, "r") as f:
|
||||
return f.read()
|
||||
finally:
|
||||
if os.path.exists(tmp_path):
|
||||
os.remove(tmp_path)
|
||||
except Exception as e:
|
||||
logger.error(f"Error loading YouTube description for theme {theme_id}: {e}")
|
||||
return None
|
||||
|
||||
def apply_color_overrides(
|
||||
self, style_params: Dict[str, Any], overrides: ColorOverrides
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Apply user color overrides to theme style parameters.
|
||||
|
||||
Args:
|
||||
style_params: Base style parameters from theme
|
||||
overrides: User color overrides
|
||||
|
||||
Returns:
|
||||
Modified style parameters with overrides applied
|
||||
"""
|
||||
if not overrides.has_overrides():
|
||||
return style_params
|
||||
|
||||
# Deep copy to avoid modifying original
|
||||
result = copy.deepcopy(style_params)
|
||||
|
||||
# Apply artist_color to intro, end, and CDG sections
|
||||
if overrides.artist_color:
|
||||
if "intro" in result:
|
||||
result["intro"]["artist_color"] = overrides.artist_color
|
||||
if "end" in result:
|
||||
result["end"]["artist_color"] = overrides.artist_color
|
||||
if "cdg" in result:
|
||||
result["cdg"]["artist_color"] = overrides.artist_color
|
||||
|
||||
# Apply title_color to intro, end, and CDG sections
|
||||
if overrides.title_color:
|
||||
if "intro" in result:
|
||||
result["intro"]["title_color"] = overrides.title_color
|
||||
if "end" in result:
|
||||
result["end"]["title_color"] = overrides.title_color
|
||||
if "cdg" in result:
|
||||
result["cdg"]["title_color"] = overrides.title_color
|
||||
|
||||
# Apply sung_lyrics_color (karaoke uses RGBA, CDG uses hex)
|
||||
if overrides.sung_lyrics_color:
|
||||
if "karaoke" in result:
|
||||
result["karaoke"]["primary_color"] = hex_to_rgba(overrides.sung_lyrics_color)
|
||||
if "cdg" in result:
|
||||
result["cdg"]["active_fill"] = overrides.sung_lyrics_color
|
||||
|
||||
# Apply unsung_lyrics_color (karaoke uses RGBA, CDG uses hex)
|
||||
if overrides.unsung_lyrics_color:
|
||||
if "karaoke" in result:
|
||||
result["karaoke"]["secondary_color"] = hex_to_rgba(overrides.unsung_lyrics_color)
|
||||
if "cdg" in result:
|
||||
result["cdg"]["inactive_fill"] = overrides.unsung_lyrics_color
|
||||
|
||||
return result
|
||||
|
||||
def prepare_job_style(
|
||||
self,
|
||||
job_id: str,
|
||||
theme_id: str,
|
||||
color_overrides: Optional[ColorOverrides] = None,
|
||||
) -> Tuple[str, Dict[str, str]]:
|
||||
"""
|
||||
Prepare style files for a job by copying theme to job folder.
|
||||
|
||||
This method:
|
||||
1. Loads the theme's style_params.json
|
||||
2. Applies any color overrides
|
||||
3. Updates asset paths to point to theme assets (shared, not copied)
|
||||
4. Uploads modified style_params.json to job folder
|
||||
5. Returns the GCS path and style_assets mapping
|
||||
|
||||
Args:
|
||||
job_id: The job ID
|
||||
theme_id: The theme to use
|
||||
color_overrides: Optional color overrides
|
||||
|
||||
Returns:
|
||||
Tuple of (style_params_gcs_path, style_assets dict)
|
||||
|
||||
Raises:
|
||||
ValueError: If theme not found
|
||||
"""
|
||||
# Load theme style params
|
||||
style_params = self.get_theme_style_params(theme_id)
|
||||
if style_params is None:
|
||||
raise ValueError(f"Theme not found: {theme_id}")
|
||||
|
||||
# Apply color overrides if provided
|
||||
if color_overrides:
|
||||
style_params = self.apply_color_overrides(style_params, color_overrides)
|
||||
|
||||
# Update asset paths to point to theme's shared assets
|
||||
# Theme assets stay in themes/{theme_id}/assets/ - they're shared
|
||||
style_assets = self._build_style_assets_mapping(theme_id, style_params)
|
||||
|
||||
# Update style_params to use theme asset paths (not job-specific)
|
||||
style_params = self._update_asset_paths_in_style(theme_id, style_params)
|
||||
|
||||
# Upload modified style_params.json to job's style folder
|
||||
job_style_path = f"uploads/{job_id}/style/style_params.json"
|
||||
self.storage.upload_json(job_style_path, style_params)
|
||||
|
||||
logger.info(f"Prepared job {job_id} style from theme {theme_id}")
|
||||
|
||||
return job_style_path, style_assets
|
||||
|
||||
def _build_style_assets_mapping(
|
||||
self, theme_id: str, style_params: Dict[str, Any]
|
||||
) -> Dict[str, str]:
|
||||
"""
|
||||
Build the style_assets mapping for a theme.
|
||||
|
||||
Maps asset keys to their GCS paths in the theme folder.
|
||||
"""
|
||||
style_assets = {}
|
||||
theme_assets_prefix = f"{THEMES_PREFIX}/{theme_id}/assets"
|
||||
|
||||
# Check each potential asset type
|
||||
asset_checks = [
|
||||
("intro_background", "intro", "background_image"),
|
||||
("karaoke_background", "karaoke", "background_image"),
|
||||
("end_background", "end", "background_image"),
|
||||
("font", "intro", "font"), # Font is typically shared across sections
|
||||
("cdg_instrumental_background", "cdg", "instrumental_background"),
|
||||
("cdg_title_background", "cdg", "title_screen_background"),
|
||||
("cdg_outro_background", "cdg", "outro_background"),
|
||||
]
|
||||
|
||||
for asset_key, section, field in asset_checks:
|
||||
if section in style_params and field in style_params[section]:
|
||||
value = style_params[section][field]
|
||||
if value and isinstance(value, str):
|
||||
# Check if it's a theme asset path or already has the full path
|
||||
if value.startswith(f"{THEMES_PREFIX}/"):
|
||||
# Already a full theme path
|
||||
style_assets[asset_key] = value
|
||||
elif not value.startswith("/") and not value.startswith("gs://"):
|
||||
# Relative path - prepend theme assets prefix
|
||||
asset_path = f"{theme_assets_prefix}/{os.path.basename(value)}"
|
||||
if self.storage.file_exists(asset_path):
|
||||
style_assets[asset_key] = asset_path
|
||||
|
||||
return style_assets
|
||||
|
||||
def _update_asset_paths_in_style(
|
||||
self, theme_id: str, style_params: Dict[str, Any]
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Update asset paths in style_params to use full GCS theme paths.
|
||||
|
||||
This ensures the style_loader can find assets when processing.
|
||||
"""
|
||||
result = copy.deepcopy(style_params)
|
||||
theme_assets_prefix = f"{THEMES_PREFIX}/{theme_id}/assets"
|
||||
|
||||
# Fields that contain asset paths
|
||||
path_fields = {
|
||||
"intro": ["background_image", "font"],
|
||||
"end": ["background_image", "font"],
|
||||
"karaoke": ["background_image", "font_path"],
|
||||
"cdg": [
|
||||
"font_path",
|
||||
"instrumental_background",
|
||||
"title_screen_background",
|
||||
"outro_background",
|
||||
],
|
||||
}
|
||||
|
||||
for section, fields in path_fields.items():
|
||||
if section not in result:
|
||||
continue
|
||||
for field in fields:
|
||||
if field not in result[section]:
|
||||
continue
|
||||
value = result[section][field]
|
||||
if value and isinstance(value, str):
|
||||
# Skip if already a full path or URL
|
||||
if value.startswith(f"{THEMES_PREFIX}/") or value.startswith("gs://"):
|
||||
continue
|
||||
# Skip absolute local paths (shouldn't happen but be safe)
|
||||
if value.startswith("/"):
|
||||
continue
|
||||
# Update to full theme asset path
|
||||
result[section][field] = f"{theme_assets_prefix}/{os.path.basename(value)}"
|
||||
|
||||
return result
|
||||
|
||||
def theme_exists(self, theme_id: str) -> bool:
|
||||
"""
|
||||
Check if a theme exists.
|
||||
|
||||
Args:
|
||||
theme_id: The theme identifier
|
||||
|
||||
Returns:
|
||||
True if theme exists, False otherwise
|
||||
"""
|
||||
metadata = self._load_metadata()
|
||||
return any(t.id == theme_id for t in metadata.themes)
|
||||
|
||||
def get_default_theme_id(self) -> Optional[str]:
|
||||
"""
|
||||
Get the ID of the default theme.
|
||||
|
||||
Returns:
|
||||
Default theme ID if one exists, None otherwise
|
||||
"""
|
||||
metadata = self._load_metadata()
|
||||
default_theme = next((t for t in metadata.themes if t.is_default), None)
|
||||
return default_theme.id if default_theme else None
|
||||
|
||||
def invalidate_cache(self) -> None:
|
||||
"""Force invalidation of the metadata cache."""
|
||||
self._metadata_cache = None
|
||||
self._cache_time = None
|
||||
logger.info("Theme metadata cache invalidated")
|
||||
|
||||
|
||||
# Singleton instance with thread-safe initialization
|
||||
_theme_service: Optional[ThemeService] = None
|
||||
_theme_service_lock = threading.Lock()
|
||||
|
||||
|
||||
def get_theme_service() -> ThemeService:
|
||||
"""Get or create the singleton ThemeService instance (thread-safe)."""
|
||||
global _theme_service
|
||||
if _theme_service is None:
|
||||
with _theme_service_lock:
|
||||
# Double-check after acquiring lock
|
||||
if _theme_service is None:
|
||||
_theme_service = ThemeService()
|
||||
return _theme_service
|
||||
@@ -0,0 +1,543 @@
|
||||
"""
|
||||
OpenTelemetry tracing configuration for Google Cloud Trace.
|
||||
|
||||
This module sets up distributed tracing that exports to Google Cloud Trace,
|
||||
providing visibility into request flows, latency, and errors across the
|
||||
entire backend.
|
||||
|
||||
Usage:
|
||||
# In main.py or app initialization
|
||||
from backend.services.tracing import setup_tracing, instrument_app
|
||||
|
||||
setup_tracing(service_name="karaoke-backend")
|
||||
app = FastAPI()
|
||||
instrument_app(app)
|
||||
|
||||
# For custom spans in your code
|
||||
from backend.services.tracing import tracer, create_span
|
||||
|
||||
with tracer.start_as_current_span("my-operation") as span:
|
||||
span.set_attribute("job_id", job_id)
|
||||
# ... do work ...
|
||||
|
||||
# For job-specific spans with automatic job_id attribute
|
||||
from backend.services.tracing import job_span
|
||||
|
||||
with job_span("audio-worker", job_id) as span:
|
||||
# job_id is automatically set as an attribute
|
||||
span.set_attribute("stage", "separation")
|
||||
# ... do work ...
|
||||
|
||||
# For propagating trace context through Cloud Tasks
|
||||
from backend.services.tracing import inject_trace_context, extract_trace_context
|
||||
|
||||
# In worker_service.py (when creating task):
|
||||
headers = inject_trace_context({})
|
||||
|
||||
# In internal.py (when receiving task):
|
||||
context = extract_trace_context(request.headers)
|
||||
with tracer.start_as_current_span("worker", context=context):
|
||||
...
|
||||
"""
|
||||
import os
|
||||
import logging
|
||||
from typing import Optional, Any, Dict
|
||||
from contextlib import contextmanager
|
||||
from functools import wraps
|
||||
|
||||
# OpenTelemetry imports
|
||||
from opentelemetry import trace
|
||||
from opentelemetry.sdk.trace import TracerProvider
|
||||
from opentelemetry.sdk.trace.export import BatchSpanProcessor, SimpleSpanProcessor
|
||||
from opentelemetry.sdk.resources import Resource, SERVICE_NAME, SERVICE_VERSION
|
||||
from opentelemetry.trace import Status, StatusCode, Span
|
||||
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
|
||||
|
||||
# Google Cloud Trace exporter
|
||||
try:
|
||||
from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
|
||||
from opentelemetry.resourcedetector.gcp_resource_detector import GoogleCloudResourceDetector
|
||||
GCP_AVAILABLE = True
|
||||
except ImportError:
|
||||
GCP_AVAILABLE = False
|
||||
CloudTraceSpanExporter = None
|
||||
GoogleCloudResourceDetector = None
|
||||
|
||||
# FastAPI instrumentation
|
||||
try:
|
||||
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
|
||||
FASTAPI_INSTRUMENTATION_AVAILABLE = True
|
||||
except ImportError:
|
||||
FASTAPI_INSTRUMENTATION_AVAILABLE = False
|
||||
FastAPIInstrumentor = None
|
||||
|
||||
# HTTPX instrumentation (for outgoing HTTP calls)
|
||||
try:
|
||||
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentation
|
||||
HTTPX_INSTRUMENTATION_AVAILABLE = True
|
||||
except ImportError:
|
||||
HTTPX_INSTRUMENTATION_AVAILABLE = False
|
||||
HTTPXClientInstrumentation = None
|
||||
|
||||
# Logging instrumentation
|
||||
try:
|
||||
from opentelemetry.instrumentation.logging import LoggingInstrumentor
|
||||
LOGGING_INSTRUMENTATION_AVAILABLE = True
|
||||
except ImportError:
|
||||
LOGGING_INSTRUMENTATION_AVAILABLE = False
|
||||
LoggingInstrumentor = None
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Global tracer instance
|
||||
_tracer: Optional[trace.Tracer] = None
|
||||
_initialized = False
|
||||
|
||||
|
||||
def setup_tracing(
|
||||
service_name: str = "karaoke-backend",
|
||||
service_version: str = "0.71.7",
|
||||
enable_in_dev: bool = False,
|
||||
) -> bool:
|
||||
"""
|
||||
Initialize OpenTelemetry tracing with Google Cloud Trace exporter.
|
||||
|
||||
Args:
|
||||
service_name: Name of the service (appears in Cloud Trace)
|
||||
service_version: Version of the service
|
||||
enable_in_dev: Whether to enable tracing in development (default False)
|
||||
|
||||
Returns:
|
||||
True if tracing was initialized, False if skipped
|
||||
"""
|
||||
global _tracer, _initialized
|
||||
|
||||
if _initialized:
|
||||
logger.debug("Tracing already initialized")
|
||||
return True
|
||||
|
||||
# Check if we should enable tracing
|
||||
# In Cloud Run, GOOGLE_CLOUD_PROJECT is always set
|
||||
is_cloud_run = os.environ.get("K_SERVICE") is not None
|
||||
gcp_project = os.environ.get("GOOGLE_CLOUD_PROJECT") or os.environ.get("GCP_PROJECT")
|
||||
|
||||
if not is_cloud_run and not enable_in_dev:
|
||||
logger.info("Tracing disabled in development (set enable_in_dev=True to enable)")
|
||||
_tracer = trace.get_tracer(service_name)
|
||||
return False
|
||||
|
||||
if not GCP_AVAILABLE:
|
||||
logger.warning("Google Cloud Trace exporter not available, tracing disabled")
|
||||
_tracer = trace.get_tracer(service_name)
|
||||
return False
|
||||
|
||||
logger.info(f"Initializing OpenTelemetry tracing for {service_name} v{service_version}")
|
||||
|
||||
try:
|
||||
# Create resource with service info and GCP resource detection
|
||||
resource_attributes = {
|
||||
SERVICE_NAME: service_name,
|
||||
SERVICE_VERSION: service_version,
|
||||
}
|
||||
|
||||
# Detect GCP resources (project, region, instance, etc.)
|
||||
if GoogleCloudResourceDetector and is_cloud_run:
|
||||
try:
|
||||
gcp_resource = GoogleCloudResourceDetector().detect()
|
||||
resource = Resource.create(resource_attributes).merge(gcp_resource)
|
||||
logger.info("Detected GCP resources for tracing")
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not detect GCP resources: {e}")
|
||||
resource = Resource.create(resource_attributes)
|
||||
else:
|
||||
resource = Resource.create(resource_attributes)
|
||||
|
||||
# Create tracer provider
|
||||
provider = TracerProvider(resource=resource)
|
||||
|
||||
# Add Cloud Trace exporter
|
||||
if CloudTraceSpanExporter and gcp_project:
|
||||
try:
|
||||
exporter = CloudTraceSpanExporter(project_id=gcp_project)
|
||||
# Use BatchSpanProcessor for production (better performance)
|
||||
processor = BatchSpanProcessor(exporter)
|
||||
provider.add_span_processor(processor)
|
||||
logger.info(f"Cloud Trace exporter configured for project: {gcp_project}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not configure Cloud Trace exporter: {e}")
|
||||
|
||||
# Set the global tracer provider
|
||||
trace.set_tracer_provider(provider)
|
||||
|
||||
# Get tracer instance
|
||||
_tracer = trace.get_tracer(service_name, service_version)
|
||||
|
||||
# Instrument logging to include trace context
|
||||
if LOGGING_INSTRUMENTATION_AVAILABLE and LoggingInstrumentor:
|
||||
try:
|
||||
LoggingInstrumentor().instrument(set_logging_format=True)
|
||||
logger.info("Logging instrumentation enabled")
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not instrument logging: {e}")
|
||||
|
||||
# Instrument HTTPX for outgoing requests
|
||||
if HTTPX_INSTRUMENTATION_AVAILABLE and HTTPXClientInstrumentation:
|
||||
try:
|
||||
HTTPXClientInstrumentation().instrument()
|
||||
logger.info("HTTPX instrumentation enabled")
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not instrument HTTPX: {e}")
|
||||
|
||||
_initialized = True
|
||||
logger.info("OpenTelemetry tracing initialized successfully")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to initialize tracing: {e}")
|
||||
_tracer = trace.get_tracer(service_name)
|
||||
return False
|
||||
|
||||
|
||||
def instrument_app(app) -> None:
|
||||
"""
|
||||
Instrument a FastAPI application with automatic tracing.
|
||||
|
||||
This adds spans for all incoming requests, including:
|
||||
- Request method and path
|
||||
- Response status code
|
||||
- Request duration
|
||||
- Error information
|
||||
|
||||
Args:
|
||||
app: FastAPI application instance
|
||||
"""
|
||||
if not FASTAPI_INSTRUMENTATION_AVAILABLE or not FastAPIInstrumentor:
|
||||
logger.warning("FastAPI instrumentation not available")
|
||||
return
|
||||
|
||||
try:
|
||||
FastAPIInstrumentor.instrument_app(
|
||||
app,
|
||||
excluded_urls="health,healthz,ready,readiness,ping",
|
||||
)
|
||||
logger.info("FastAPI instrumentation enabled")
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not instrument FastAPI: {e}")
|
||||
|
||||
|
||||
def get_tracer() -> trace.Tracer:
|
||||
"""
|
||||
Get the global tracer instance.
|
||||
|
||||
Returns:
|
||||
Tracer instance (creates a no-op tracer if not initialized)
|
||||
"""
|
||||
global _tracer
|
||||
if _tracer is None:
|
||||
_tracer = trace.get_tracer("karaoke-backend")
|
||||
return _tracer
|
||||
|
||||
|
||||
# Convenience alias - lazily forwards attribute access to get_tracer()
|
||||
class _TracerProxy:
|
||||
"""Proxy that lazily returns the global tracer."""
|
||||
def __getattr__(self, name):
|
||||
return getattr(get_tracer(), name)
|
||||
|
||||
tracer = _TracerProxy()
|
||||
|
||||
|
||||
@contextmanager
|
||||
def create_span(
|
||||
name: str,
|
||||
attributes: Optional[Dict[str, Any]] = None,
|
||||
kind: trace.SpanKind = trace.SpanKind.INTERNAL,
|
||||
):
|
||||
"""
|
||||
Create a traced span as a context manager.
|
||||
|
||||
Usage:
|
||||
with create_span("process-lyrics", {"job_id": job_id}) as span:
|
||||
# ... do work ...
|
||||
span.set_attribute("words_processed", 500)
|
||||
|
||||
Args:
|
||||
name: Span name (appears in Cloud Trace)
|
||||
attributes: Initial span attributes
|
||||
kind: Span kind (INTERNAL, SERVER, CLIENT, PRODUCER, CONSUMER)
|
||||
|
||||
Yields:
|
||||
The active span
|
||||
"""
|
||||
tracer = get_tracer()
|
||||
with tracer.start_as_current_span(name, kind=kind) as span:
|
||||
if attributes:
|
||||
for key, value in attributes.items():
|
||||
span.set_attribute(key, value)
|
||||
try:
|
||||
yield span
|
||||
except Exception as e:
|
||||
span.set_status(Status(StatusCode.ERROR, str(e)))
|
||||
span.record_exception(e)
|
||||
raise
|
||||
|
||||
|
||||
def traced(
|
||||
name: Optional[str] = None,
|
||||
attributes: Optional[Dict[str, Any]] = None,
|
||||
):
|
||||
"""
|
||||
Decorator to trace a function.
|
||||
|
||||
Usage:
|
||||
@traced("process-correction")
|
||||
def process_correction(job_id: str):
|
||||
...
|
||||
|
||||
@traced(attributes={"operation": "add-lyrics"})
|
||||
async def add_lyrics(job_id: str, source: str):
|
||||
...
|
||||
|
||||
Args:
|
||||
name: Span name (defaults to function name)
|
||||
attributes: Static attributes to add to span
|
||||
"""
|
||||
def decorator(func):
|
||||
span_name = name or func.__name__
|
||||
|
||||
@wraps(func)
|
||||
async def async_wrapper(*args, **kwargs):
|
||||
with create_span(span_name, attributes) as span:
|
||||
# Add function arguments as attributes
|
||||
span.set_attribute("function", func.__name__)
|
||||
return await func(*args, **kwargs)
|
||||
|
||||
@wraps(func)
|
||||
def sync_wrapper(*args, **kwargs):
|
||||
with create_span(span_name, attributes) as span:
|
||||
span.set_attribute("function", func.__name__)
|
||||
return func(*args, **kwargs)
|
||||
|
||||
# Return appropriate wrapper based on function type
|
||||
import asyncio
|
||||
if asyncio.iscoroutinefunction(func):
|
||||
return async_wrapper
|
||||
return sync_wrapper
|
||||
|
||||
return decorator
|
||||
|
||||
|
||||
def add_span_attribute(key: str, value: Any) -> None:
|
||||
"""
|
||||
Add an attribute to the current span.
|
||||
|
||||
Args:
|
||||
key: Attribute name
|
||||
value: Attribute value (must be string, int, float, bool, or list thereof)
|
||||
"""
|
||||
span = trace.get_current_span()
|
||||
if span and span.is_recording():
|
||||
span.set_attribute(key, value)
|
||||
|
||||
|
||||
def add_span_event(name: str, attributes: Optional[Dict[str, Any]] = None) -> None:
|
||||
"""
|
||||
Add an event to the current span.
|
||||
|
||||
Events are timestamped annotations that appear in the trace timeline.
|
||||
|
||||
Args:
|
||||
name: Event name
|
||||
attributes: Event attributes
|
||||
"""
|
||||
span = trace.get_current_span()
|
||||
if span and span.is_recording():
|
||||
span.add_event(name, attributes=attributes or {})
|
||||
|
||||
|
||||
def set_span_error(error: Exception) -> None:
|
||||
"""
|
||||
Mark the current span as errored.
|
||||
|
||||
Args:
|
||||
error: The exception that occurred
|
||||
"""
|
||||
span = trace.get_current_span()
|
||||
if span and span.is_recording():
|
||||
span.set_status(Status(StatusCode.ERROR, str(error)))
|
||||
span.record_exception(error)
|
||||
|
||||
|
||||
def get_current_trace_id() -> Optional[str]:
|
||||
"""
|
||||
Get the current trace ID for correlation.
|
||||
|
||||
This can be used to link logs to traces.
|
||||
|
||||
Returns:
|
||||
Trace ID as hex string, or None if no active trace
|
||||
"""
|
||||
span = trace.get_current_span()
|
||||
if span:
|
||||
ctx = span.get_span_context()
|
||||
if ctx.is_valid:
|
||||
return format(ctx.trace_id, '032x')
|
||||
return None
|
||||
|
||||
|
||||
def get_current_span_id() -> Optional[str]:
|
||||
"""
|
||||
Get the current span ID.
|
||||
|
||||
Returns:
|
||||
Span ID as hex string, or None if no active span
|
||||
"""
|
||||
span = trace.get_current_span()
|
||||
if span:
|
||||
ctx = span.get_span_context()
|
||||
if ctx.is_valid:
|
||||
return format(ctx.span_id, '016x')
|
||||
return None
|
||||
|
||||
|
||||
# =====================================================
|
||||
# Job-Aware Tracing Utilities
|
||||
# =====================================================
|
||||
|
||||
@contextmanager
|
||||
def job_span(
|
||||
name: str,
|
||||
job_id: str,
|
||||
attributes: Optional[Dict[str, Any]] = None,
|
||||
kind: trace.SpanKind = trace.SpanKind.INTERNAL,
|
||||
):
|
||||
"""
|
||||
Create a traced span with job_id automatically set as an attribute.
|
||||
|
||||
This is the preferred way to create spans in job processing workers.
|
||||
The job_id attribute enables filtering spans by job in Cloud Trace.
|
||||
|
||||
Usage:
|
||||
with job_span("audio-separation", job_id) as span:
|
||||
span.set_attribute("stage", "clean_instrumental")
|
||||
# ... do work ...
|
||||
|
||||
Args:
|
||||
name: Span name (appears in Cloud Trace)
|
||||
job_id: Job ID to attach to the span
|
||||
attributes: Additional span attributes
|
||||
kind: Span kind (INTERNAL, SERVER, CLIENT, PRODUCER, CONSUMER)
|
||||
|
||||
Yields:
|
||||
The active span
|
||||
"""
|
||||
all_attributes = {
|
||||
"job_id": job_id,
|
||||
"service.operation": name,
|
||||
}
|
||||
if attributes:
|
||||
all_attributes.update(attributes)
|
||||
|
||||
with create_span(name, all_attributes, kind) as span:
|
||||
yield span
|
||||
|
||||
|
||||
def inject_trace_context(headers: Dict[str, str]) -> Dict[str, str]:
|
||||
"""
|
||||
Inject current trace context into HTTP headers.
|
||||
|
||||
Use this when creating Cloud Tasks or making HTTP calls to propagate
|
||||
the trace context so child spans link to the parent trace.
|
||||
|
||||
Usage:
|
||||
headers = inject_trace_context({
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": "Bearer token",
|
||||
})
|
||||
# Use headers in HTTP request or Cloud Task
|
||||
|
||||
Args:
|
||||
headers: Existing headers dict to add trace context to
|
||||
|
||||
Returns:
|
||||
Headers dict with traceparent/tracestate headers added
|
||||
"""
|
||||
propagator = TraceContextTextMapPropagator()
|
||||
carrier = dict(headers) # Make a copy
|
||||
propagator.inject(carrier)
|
||||
return carrier
|
||||
|
||||
|
||||
def extract_trace_context(headers: Dict[str, str]) -> Optional[trace.Context]:
|
||||
"""
|
||||
Extract trace context from HTTP headers.
|
||||
|
||||
Use this when receiving Cloud Tasks or HTTP requests to continue
|
||||
the trace from the parent span.
|
||||
|
||||
Usage:
|
||||
context = extract_trace_context(request.headers)
|
||||
with tracer.start_as_current_span("worker", context=context):
|
||||
# ... processing ...
|
||||
|
||||
Args:
|
||||
headers: HTTP headers containing traceparent/tracestate
|
||||
|
||||
Returns:
|
||||
Context object for use with start_as_current_span, or None
|
||||
"""
|
||||
propagator = TraceContextTextMapPropagator()
|
||||
return propagator.extract(headers)
|
||||
|
||||
|
||||
def start_span_with_context(
|
||||
name: str,
|
||||
context: Optional[trace.Context],
|
||||
job_id: Optional[str] = None,
|
||||
attributes: Optional[Dict[str, Any]] = None,
|
||||
):
|
||||
"""
|
||||
Start a span as child of the given context (or current context if None).
|
||||
|
||||
This is useful for starting worker spans that should link to the
|
||||
trace from the original API request.
|
||||
|
||||
Usage:
|
||||
context = extract_trace_context(request.headers)
|
||||
with start_span_with_context("audio-worker", context, job_id) as span:
|
||||
# ... processing ...
|
||||
|
||||
Args:
|
||||
name: Span name
|
||||
context: Parent context (from extract_trace_context) or None
|
||||
job_id: Optional job ID to set as attribute
|
||||
attributes: Additional span attributes
|
||||
|
||||
Returns:
|
||||
Context manager yielding the span
|
||||
"""
|
||||
tracer = get_tracer()
|
||||
|
||||
all_attributes = attributes or {}
|
||||
if job_id:
|
||||
all_attributes["job_id"] = job_id
|
||||
all_attributes["service.operation"] = name
|
||||
|
||||
@contextmanager
|
||||
def _span():
|
||||
kwargs = {"kind": trace.SpanKind.INTERNAL}
|
||||
if context is not None:
|
||||
kwargs["context"] = context
|
||||
|
||||
with tracer.start_as_current_span(name, **kwargs) as span:
|
||||
for key, value in all_attributes.items():
|
||||
span.set_attribute(key, value)
|
||||
try:
|
||||
yield span
|
||||
except Exception as e:
|
||||
span.set_status(Status(StatusCode.ERROR, str(e)))
|
||||
span.record_exception(e)
|
||||
raise
|
||||
|
||||
return _span()
|
||||
@@ -0,0 +1,721 @@
|
||||
"""
|
||||
User service for authentication, credits, and user management.
|
||||
|
||||
Handles:
|
||||
- Magic link authentication (send, verify)
|
||||
- Session management
|
||||
- Credit operations (add, deduct, refund)
|
||||
- User CRUD operations
|
||||
"""
|
||||
import logging
|
||||
import secrets
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Optional, List, Tuple
|
||||
|
||||
from google.cloud import firestore
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
from google.cloud.firestore_v1 import Increment
|
||||
|
||||
from backend.config import get_settings
|
||||
from backend.models.user import (
|
||||
User,
|
||||
UserRole,
|
||||
UserPublic,
|
||||
MagicLinkToken,
|
||||
Session,
|
||||
CreditTransaction,
|
||||
)
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Collection names
|
||||
USERS_COLLECTION = "gen_users"
|
||||
MAGIC_LINKS_COLLECTION = "magic_links"
|
||||
SESSIONS_COLLECTION = "sessions"
|
||||
PROCESSED_STRIPE_SESSIONS_COLLECTION = "processed_stripe_sessions"
|
||||
|
||||
# Token/session configuration
|
||||
MAGIC_LINK_EXPIRY_MINUTES = 15
|
||||
SESSION_EXPIRY_DAYS = 7
|
||||
SESSION_ABSOLUTE_EXPIRY_DAYS = 30
|
||||
MAX_CREDIT_TRANSACTIONS = 100 # Keep last N transactions
|
||||
|
||||
|
||||
class UserService:
|
||||
"""Service for user management and authentication."""
|
||||
|
||||
# Number of free credits granted to new users
|
||||
NEW_USER_FREE_CREDITS = 1
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize user service with Firestore client."""
|
||||
self.settings = get_settings()
|
||||
self.db = firestore.Client(project=self.settings.google_cloud_project)
|
||||
|
||||
# =========================================================================
|
||||
# User CRUD Operations
|
||||
# =========================================================================
|
||||
|
||||
def get_user(self, email: str) -> Optional[User]:
|
||||
"""Get a user by email."""
|
||||
try:
|
||||
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
return None
|
||||
|
||||
return User(**doc.to_dict())
|
||||
except Exception:
|
||||
logger.exception(f"Error getting user {email}")
|
||||
return None
|
||||
|
||||
def get_or_create_user(self, email: str) -> User:
|
||||
"""
|
||||
Get existing user or create a new one.
|
||||
|
||||
New users receive a welcome credit to try the service.
|
||||
"""
|
||||
email = email.lower()
|
||||
user = self.get_user(email)
|
||||
|
||||
if user:
|
||||
return user
|
||||
|
||||
# Create new user with welcome credit
|
||||
welcome_credit = self.NEW_USER_FREE_CREDITS
|
||||
welcome_transaction = CreditTransaction(
|
||||
id=str(uuid.uuid4()),
|
||||
amount=welcome_credit,
|
||||
reason="welcome_credit",
|
||||
)
|
||||
|
||||
user = User(
|
||||
email=email,
|
||||
credits=welcome_credit,
|
||||
credit_transactions=[welcome_transaction],
|
||||
)
|
||||
self._save_user(user)
|
||||
logger.info(f"Created new user: {email} with {welcome_credit} welcome credit(s)")
|
||||
return user
|
||||
|
||||
def _save_user(self, user: User) -> None:
|
||||
"""Save user to Firestore."""
|
||||
try:
|
||||
user.updated_at = datetime.utcnow()
|
||||
doc_ref = self.db.collection(USERS_COLLECTION).document(user.email.lower())
|
||||
doc_ref.set(user.model_dump(mode='json'))
|
||||
except Exception:
|
||||
logger.exception(f"Error saving user {user.email}")
|
||||
raise
|
||||
|
||||
def update_user(self, email: str, **updates) -> Optional[User]:
|
||||
"""Update user fields."""
|
||||
try:
|
||||
updates['updated_at'] = datetime.utcnow()
|
||||
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
|
||||
doc_ref.update(updates)
|
||||
return self.get_user(email)
|
||||
except Exception:
|
||||
logger.exception(f"Error updating user {email}")
|
||||
return None
|
||||
|
||||
def list_users(self, limit: int = 100, include_inactive: bool = False) -> List[User]:
|
||||
"""List all users (admin only)."""
|
||||
try:
|
||||
query = self.db.collection(USERS_COLLECTION)
|
||||
|
||||
if not include_inactive:
|
||||
query = query.where(filter=FieldFilter('is_active', '==', True))
|
||||
|
||||
query = query.order_by('created_at', direction=firestore.Query.DESCENDING)
|
||||
query = query.limit(limit)
|
||||
|
||||
docs = query.stream()
|
||||
return [User(**doc.to_dict()) for doc in docs]
|
||||
except Exception:
|
||||
logger.exception("Error listing users")
|
||||
return []
|
||||
|
||||
def get_user_public(self, email: str) -> Optional[UserPublic]:
|
||||
"""Get public user info."""
|
||||
user = self.get_user(email)
|
||||
if not user:
|
||||
return None
|
||||
|
||||
return UserPublic(
|
||||
email=user.email,
|
||||
role=user.role,
|
||||
credits=user.credits,
|
||||
display_name=user.display_name,
|
||||
total_jobs_created=user.total_jobs_created,
|
||||
total_jobs_completed=user.total_jobs_completed,
|
||||
)
|
||||
|
||||
# =========================================================================
|
||||
# Magic Link Authentication
|
||||
# =========================================================================
|
||||
|
||||
def create_magic_link(
|
||||
self,
|
||||
email: str,
|
||||
ip_address: Optional[str] = None,
|
||||
user_agent: Optional[str] = None
|
||||
) -> MagicLinkToken:
|
||||
"""
|
||||
Create a magic link token for email authentication.
|
||||
|
||||
Returns the token object. The actual sending of the email
|
||||
should be handled by the caller (or an email service).
|
||||
"""
|
||||
email = email.lower()
|
||||
|
||||
# Ensure user exists
|
||||
self.get_or_create_user(email)
|
||||
|
||||
# Generate secure token
|
||||
token = secrets.token_urlsafe(32)
|
||||
|
||||
magic_link = MagicLinkToken(
|
||||
token=token,
|
||||
email=email,
|
||||
expires_at=datetime.utcnow() + timedelta(minutes=MAGIC_LINK_EXPIRY_MINUTES),
|
||||
ip_address=ip_address,
|
||||
user_agent=user_agent,
|
||||
)
|
||||
|
||||
# Save to Firestore
|
||||
doc_ref = self.db.collection(MAGIC_LINKS_COLLECTION).document(token)
|
||||
doc_ref.set(magic_link.model_dump(mode='json'))
|
||||
|
||||
logger.info(f"Created magic link for {email}")
|
||||
return magic_link
|
||||
|
||||
def verify_magic_link(self, token: str) -> Tuple[bool, Optional[User], str]:
|
||||
"""
|
||||
Verify a magic link token using a Firestore transaction to prevent race conditions.
|
||||
|
||||
Returns:
|
||||
(success, user, message)
|
||||
"""
|
||||
try:
|
||||
doc_ref = self.db.collection(MAGIC_LINKS_COLLECTION).document(token)
|
||||
|
||||
@firestore.transactional
|
||||
def verify_in_transaction(transaction):
|
||||
"""Atomically verify and mark magic link as used."""
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
|
||||
if not doc.exists:
|
||||
return False, None, "Invalid or expired link"
|
||||
|
||||
magic_link = MagicLinkToken(**doc.to_dict())
|
||||
|
||||
# Check if already used
|
||||
if magic_link.used:
|
||||
return False, None, "This link has already been used"
|
||||
|
||||
# Check expiry
|
||||
if datetime.utcnow() > magic_link.expires_at:
|
||||
return False, None, "This link has expired"
|
||||
|
||||
# Mark as used atomically within transaction
|
||||
transaction.update(doc_ref, {
|
||||
'used': True,
|
||||
'used_at': datetime.utcnow()
|
||||
})
|
||||
|
||||
return True, magic_link.email, "Success"
|
||||
|
||||
# Execute the transaction
|
||||
transaction = self.db.transaction()
|
||||
success, email_or_error, message = verify_in_transaction(transaction)
|
||||
|
||||
if not success:
|
||||
return False, None, message
|
||||
|
||||
# Get user and mark email as verified (outside transaction)
|
||||
user = self.get_user(email_or_error)
|
||||
if user:
|
||||
self.update_user(
|
||||
email_or_error,
|
||||
email_verified=True,
|
||||
last_login_at=datetime.utcnow()
|
||||
)
|
||||
user = self.get_user(email_or_error) # Refresh
|
||||
|
||||
logger.info(f"Magic link verified for {email_or_error}")
|
||||
return True, user, "Successfully authenticated"
|
||||
|
||||
except Exception:
|
||||
logger.exception("Error verifying magic link")
|
||||
return False, None, "An error occurred during verification"
|
||||
|
||||
# =========================================================================
|
||||
# Session Management
|
||||
# =========================================================================
|
||||
|
||||
def create_session(
|
||||
self,
|
||||
user_email: str,
|
||||
ip_address: Optional[str] = None,
|
||||
user_agent: Optional[str] = None
|
||||
) -> Session:
|
||||
"""Create a new session for an authenticated user."""
|
||||
token = secrets.token_urlsafe(32)
|
||||
token_prefix = token[:12]
|
||||
|
||||
session = Session(
|
||||
token=token,
|
||||
user_email=user_email.lower(),
|
||||
expires_at=datetime.utcnow() + timedelta(days=SESSION_ABSOLUTE_EXPIRY_DAYS),
|
||||
ip_address=ip_address,
|
||||
user_agent=user_agent,
|
||||
)
|
||||
|
||||
# Serialize and write to Firestore
|
||||
session_data = session.model_dump(mode='json')
|
||||
doc_ref = self.db.collection(SESSIONS_COLLECTION).document(token)
|
||||
doc_ref.set(session_data)
|
||||
|
||||
# Verify the write succeeded by reading back
|
||||
verify_doc = doc_ref.get()
|
||||
if not verify_doc.exists:
|
||||
logger.error(f"Session write verification FAILED for {user_email}: {token_prefix}...")
|
||||
raise RuntimeError(f"Failed to persist session for {user_email}")
|
||||
|
||||
logger.info(f"Created and verified session for {user_email}: {token_prefix}... (expires: {session.expires_at})")
|
||||
|
||||
return session
|
||||
|
||||
def validate_session(self, token: str) -> Tuple[bool, Optional[User], str]:
|
||||
"""
|
||||
Validate a session token.
|
||||
|
||||
Returns:
|
||||
(valid, user, message)
|
||||
"""
|
||||
# Log token info for debugging (only prefix for security)
|
||||
token_prefix = token[:12] if token and len(token) >= 12 else token
|
||||
logger.debug(f"Validating session token: {token_prefix}... (len={len(token) if token else 0})")
|
||||
|
||||
try:
|
||||
doc_ref = self.db.collection(SESSIONS_COLLECTION).document(token)
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
logger.warning(f"Session not found in Firestore: {token_prefix}...")
|
||||
return False, None, "Invalid session"
|
||||
|
||||
raw_data = doc.to_dict()
|
||||
logger.debug(f"Session data found for {token_prefix}...: user_email={raw_data.get('user_email')}, is_active={raw_data.get('is_active')}")
|
||||
|
||||
session = Session(**raw_data)
|
||||
|
||||
# Check if active
|
||||
if not session.is_active:
|
||||
logger.warning(f"Session revoked for {token_prefix}... (user: {session.user_email})")
|
||||
return False, None, "Session has been revoked"
|
||||
|
||||
# Use timezone-aware datetime for all comparisons
|
||||
# Firestore returns timezone-aware datetimes, so we must use aware datetimes
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
# Normalize session datetimes to be timezone-aware for comparison
|
||||
# (handles legacy naive datetimes that may exist in Firestore)
|
||||
expires_at = session.expires_at
|
||||
if expires_at.tzinfo is None:
|
||||
expires_at = expires_at.replace(tzinfo=timezone.utc)
|
||||
|
||||
last_activity = session.last_activity_at
|
||||
if last_activity.tzinfo is None:
|
||||
last_activity = last_activity.replace(tzinfo=timezone.utc)
|
||||
|
||||
# Check expiry
|
||||
if now > expires_at:
|
||||
logger.warning(f"Session expired for {token_prefix}... (user: {session.user_email}, expired_at: {expires_at}, now: {now})")
|
||||
return False, None, "Session has expired"
|
||||
|
||||
# Check inactivity (7 days)
|
||||
inactivity_limit = now - timedelta(days=SESSION_EXPIRY_DAYS)
|
||||
if last_activity < inactivity_limit:
|
||||
logger.warning(f"Session inactive for {token_prefix}... (user: {session.user_email}, last_activity: {last_activity}, limit: {inactivity_limit})")
|
||||
return False, None, "Session expired due to inactivity"
|
||||
|
||||
# Update last activity (use timezone-aware datetime)
|
||||
doc_ref.update({'last_activity_at': now})
|
||||
|
||||
# Get user
|
||||
user = self.get_user(session.user_email)
|
||||
if not user:
|
||||
logger.warning(f"User not found for session {token_prefix}...: {session.user_email}")
|
||||
return False, None, "User not found"
|
||||
|
||||
if not user.is_active:
|
||||
logger.warning(f"User account disabled for session {token_prefix}...: {session.user_email}")
|
||||
return False, None, "User account is disabled"
|
||||
|
||||
logger.debug(f"Session valid for {token_prefix}... (user: {user.email}, credits: {user.credits})")
|
||||
return True, user, "Valid session"
|
||||
|
||||
except Exception:
|
||||
logger.exception(f"Error validating session {token_prefix}...")
|
||||
return False, None, "An error occurred during validation"
|
||||
|
||||
def revoke_session(self, token: str) -> bool:
|
||||
"""Revoke a session (logout)."""
|
||||
try:
|
||||
doc_ref = self.db.collection(SESSIONS_COLLECTION).document(token)
|
||||
doc_ref.update({'is_active': False})
|
||||
logger.info("Session revoked")
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("Error revoking session")
|
||||
return False
|
||||
|
||||
def revoke_all_sessions(self, user_email: str) -> int:
|
||||
"""Revoke all sessions for a user."""
|
||||
try:
|
||||
query = self.db.collection(SESSIONS_COLLECTION).where(
|
||||
filter=FieldFilter('user_email', '==', user_email.lower())
|
||||
).where(
|
||||
filter=FieldFilter('is_active', '==', True)
|
||||
)
|
||||
|
||||
count = 0
|
||||
for doc in query.stream():
|
||||
doc.reference.update({'is_active': False})
|
||||
count += 1
|
||||
|
||||
logger.info(f"Revoked {count} sessions for {user_email}")
|
||||
return count
|
||||
except Exception:
|
||||
logger.exception("Error revoking sessions")
|
||||
return 0
|
||||
|
||||
def list_sessions_for_user(
|
||||
self,
|
||||
user_email: str,
|
||||
include_revoked: bool = False,
|
||||
limit: int = 50
|
||||
) -> List[Session]:
|
||||
"""
|
||||
List all sessions for a user.
|
||||
|
||||
Args:
|
||||
user_email: User's email address
|
||||
include_revoked: If True, include revoked/inactive sessions
|
||||
limit: Maximum number of sessions to return
|
||||
|
||||
Returns:
|
||||
List of Session objects, ordered by created_at descending
|
||||
"""
|
||||
try:
|
||||
query = self.db.collection(SESSIONS_COLLECTION).where(
|
||||
filter=FieldFilter('user_email', '==', user_email.lower())
|
||||
)
|
||||
|
||||
if not include_revoked:
|
||||
query = query.where(filter=FieldFilter('is_active', '==', True))
|
||||
|
||||
query = query.order_by('created_at', direction=firestore.Query.DESCENDING)
|
||||
query = query.limit(limit)
|
||||
|
||||
sessions = []
|
||||
for doc in query.stream():
|
||||
try:
|
||||
sessions.append(Session(**doc.to_dict()))
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to parse session document: {e}")
|
||||
|
||||
logger.debug(f"Found {len(sessions)} sessions for {user_email}")
|
||||
return sessions
|
||||
except Exception:
|
||||
logger.exception(f"Error listing sessions for {user_email}")
|
||||
return []
|
||||
|
||||
# =========================================================================
|
||||
# Credit Operations
|
||||
# =========================================================================
|
||||
|
||||
def add_credits(
|
||||
self,
|
||||
email: str,
|
||||
amount: int,
|
||||
reason: str,
|
||||
job_id: Optional[str] = None,
|
||||
stripe_session_id: Optional[str] = None,
|
||||
admin_email: Optional[str] = None
|
||||
) -> Tuple[bool, int, str]:
|
||||
"""
|
||||
Add credits to a user account.
|
||||
|
||||
Returns:
|
||||
(success, new_balance, message)
|
||||
"""
|
||||
try:
|
||||
# For Stripe sessions, atomically check and mark as processed
|
||||
# Uses create() which fails if document exists - ensures idempotency
|
||||
if stripe_session_id:
|
||||
doc_ref = self.db.collection(PROCESSED_STRIPE_SESSIONS_COLLECTION).document(stripe_session_id)
|
||||
try:
|
||||
doc_ref.create({
|
||||
'stripe_session_id': stripe_session_id,
|
||||
'email': email.lower(),
|
||||
'amount': amount,
|
||||
'processed_at': datetime.utcnow()
|
||||
})
|
||||
except Exception:
|
||||
# Document already exists - this session was already processed
|
||||
logger.info(f"Stripe session {stripe_session_id} already processed (idempotent skip)")
|
||||
return False, 0, "Session already processed"
|
||||
|
||||
email = email.lower()
|
||||
user = self.get_or_create_user(email)
|
||||
|
||||
# Create transaction record
|
||||
transaction = CreditTransaction(
|
||||
id=str(uuid.uuid4()),
|
||||
amount=amount,
|
||||
reason=reason,
|
||||
job_id=job_id,
|
||||
stripe_session_id=stripe_session_id,
|
||||
created_by=admin_email,
|
||||
)
|
||||
|
||||
# Add to transaction history (keep last N)
|
||||
transactions = user.credit_transactions[-MAX_CREDIT_TRANSACTIONS + 1:]
|
||||
transactions.append(transaction)
|
||||
|
||||
# Update user
|
||||
new_balance = user.credits + amount
|
||||
self.update_user(
|
||||
email,
|
||||
credits=new_balance,
|
||||
credit_transactions=[t.model_dump(mode='json') for t in transactions]
|
||||
)
|
||||
|
||||
logger.info(f"Added {amount} credits to {email} ({reason}). New balance: {new_balance}")
|
||||
return True, new_balance, f"Added {amount} credits"
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"Error adding credits to {email}")
|
||||
return False, 0, f"Failed to add credits: {e}"
|
||||
|
||||
def deduct_credit(
|
||||
self,
|
||||
email: str,
|
||||
job_id: str,
|
||||
reason: str = "job_creation"
|
||||
) -> Tuple[bool, int, str]:
|
||||
"""
|
||||
Deduct one credit from a user account.
|
||||
|
||||
Uses Firestore transaction to prevent race conditions.
|
||||
|
||||
Returns:
|
||||
(success, remaining_credits, message)
|
||||
"""
|
||||
try:
|
||||
email = email.lower()
|
||||
doc_ref = self.db.collection(USERS_COLLECTION).document(email)
|
||||
|
||||
@firestore.transactional
|
||||
def deduct_in_transaction(transaction):
|
||||
"""Atomically check and deduct credit."""
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
|
||||
if not doc.exists:
|
||||
return False, 0, "User not found"
|
||||
|
||||
user_data = doc.to_dict()
|
||||
current_credits = user_data.get('credits', 0)
|
||||
|
||||
if current_credits <= 0:
|
||||
return False, 0, "Insufficient credits"
|
||||
|
||||
# Create transaction record
|
||||
credit_txn = CreditTransaction(
|
||||
id=str(uuid.uuid4()),
|
||||
amount=-1,
|
||||
reason=reason,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
# Get existing transactions and add new one
|
||||
existing_transactions = user_data.get('credit_transactions', [])
|
||||
# Keep last N-1 transactions to make room for new one
|
||||
transactions = existing_transactions[-(MAX_CREDIT_TRANSACTIONS - 1):]
|
||||
transactions.append(credit_txn.model_dump(mode='json'))
|
||||
|
||||
# Calculate new values
|
||||
new_balance = current_credits - 1
|
||||
total_jobs = user_data.get('total_jobs_created', 0) + 1
|
||||
|
||||
# Update atomically within transaction
|
||||
transaction.update(doc_ref, {
|
||||
'credits': new_balance,
|
||||
'credit_transactions': transactions,
|
||||
'total_jobs_created': total_jobs,
|
||||
'updated_at': datetime.utcnow()
|
||||
})
|
||||
|
||||
return True, new_balance, f"Credit deducted. {new_balance} remaining"
|
||||
|
||||
# Execute the transaction
|
||||
fs_transaction = self.db.transaction()
|
||||
success, new_balance, message = deduct_in_transaction(fs_transaction)
|
||||
|
||||
if success:
|
||||
logger.info(f"Deducted 1 credit from {email} for job {job_id}. Remaining: {new_balance}")
|
||||
|
||||
return success, new_balance, message
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"Error deducting credit from {email}")
|
||||
return False, 0, f"Failed to deduct credit: {e}"
|
||||
|
||||
def refund_credit(
|
||||
self,
|
||||
email: str,
|
||||
job_id: str,
|
||||
reason: str = "job_failed"
|
||||
) -> Tuple[bool, int, str]:
|
||||
"""
|
||||
Refund one credit to a user account (e.g., after job failure).
|
||||
|
||||
Returns:
|
||||
(success, new_balance, message)
|
||||
"""
|
||||
return self.add_credits(email, 1, reason, job_id=job_id)
|
||||
|
||||
def check_credits(self, email: str) -> int:
|
||||
"""Check user's credit balance."""
|
||||
user = self.get_user(email)
|
||||
return user.credits if user else 0
|
||||
|
||||
def has_credits(self, email: str) -> bool:
|
||||
"""Check if user has at least one credit."""
|
||||
return self.check_credits(email) > 0
|
||||
|
||||
def is_stripe_session_processed(self, stripe_session_id: str) -> bool:
|
||||
"""
|
||||
Check if a Stripe session ID has already been processed.
|
||||
|
||||
Used to ensure idempotency of webhook processing - prevents
|
||||
duplicate credit additions if Stripe sends the same webhook twice.
|
||||
|
||||
Uses a dedicated collection for O(1) lookup instead of scanning all users.
|
||||
|
||||
Args:
|
||||
stripe_session_id: The Stripe checkout session ID
|
||||
|
||||
Returns:
|
||||
True if this session was already processed
|
||||
"""
|
||||
try:
|
||||
doc_ref = self.db.collection(PROCESSED_STRIPE_SESSIONS_COLLECTION).document(stripe_session_id)
|
||||
doc = doc_ref.get()
|
||||
if doc.exists:
|
||||
logger.info(f"Stripe session {stripe_session_id} already processed")
|
||||
return True
|
||||
return False
|
||||
except Exception:
|
||||
logger.exception(f"Error checking if stripe session {stripe_session_id} was processed")
|
||||
# On error, return False to allow processing (better to risk duplicate than block)
|
||||
return False
|
||||
|
||||
def _mark_stripe_session_processed(self, stripe_session_id: str, email: str, amount: int) -> None:
|
||||
"""
|
||||
Mark a Stripe session as processed.
|
||||
|
||||
Called after successfully adding credits from a Stripe webhook.
|
||||
|
||||
Args:
|
||||
stripe_session_id: The Stripe checkout session ID
|
||||
email: User email who received the credits
|
||||
amount: Number of credits added
|
||||
"""
|
||||
try:
|
||||
doc_ref = self.db.collection(PROCESSED_STRIPE_SESSIONS_COLLECTION).document(stripe_session_id)
|
||||
doc_ref.set({
|
||||
'stripe_session_id': stripe_session_id,
|
||||
'email': email,
|
||||
'amount': amount,
|
||||
'processed_at': datetime.utcnow()
|
||||
})
|
||||
except Exception:
|
||||
# Log but don't fail - the credits were already added
|
||||
logger.exception(f"Error marking stripe session {stripe_session_id} as processed")
|
||||
|
||||
# =========================================================================
|
||||
# Admin Operations
|
||||
# =========================================================================
|
||||
|
||||
def set_user_role(self, email: str, role: UserRole, admin_email: str) -> bool:
|
||||
"""Set a user's role (admin only)."""
|
||||
try:
|
||||
user = self.get_user(email)
|
||||
if not user:
|
||||
return False
|
||||
|
||||
self.update_user(email, role=role.value)
|
||||
logger.info(f"Admin {admin_email} set role for {email} to {role.value}")
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("Error setting user role")
|
||||
return False
|
||||
|
||||
def disable_user(self, email: str, admin_email: str) -> bool:
|
||||
"""Disable a user account (admin only)."""
|
||||
try:
|
||||
self.update_user(email, is_active=False)
|
||||
self.revoke_all_sessions(email)
|
||||
logger.info(f"Admin {admin_email} disabled user {email}")
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("Error disabling user")
|
||||
return False
|
||||
|
||||
def enable_user(self, email: str, admin_email: str) -> bool:
|
||||
"""Enable a user account (admin only)."""
|
||||
try:
|
||||
self.update_user(email, is_active=True)
|
||||
logger.info(f"Admin {admin_email} enabled user {email}")
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("Error enabling user")
|
||||
return False
|
||||
|
||||
def increment_jobs_completed(self, email: str) -> bool:
|
||||
"""Increment the completed jobs counter for a user using atomic increment."""
|
||||
try:
|
||||
# Check user exists first
|
||||
if not self.get_user(email):
|
||||
return False
|
||||
|
||||
# Use atomic increment to prevent race conditions
|
||||
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
|
||||
doc_ref.update({
|
||||
'total_jobs_completed': Increment(1),
|
||||
'updated_at': datetime.utcnow()
|
||||
})
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception(f"Error incrementing jobs completed for {email}")
|
||||
return False
|
||||
|
||||
|
||||
# Global instance
|
||||
_user_service = None
|
||||
|
||||
|
||||
def get_user_service() -> UserService:
|
||||
"""Get the global user service instance."""
|
||||
global _user_service
|
||||
if _user_service is None:
|
||||
_user_service = UserService()
|
||||
return _user_service
|
||||
@@ -0,0 +1,460 @@
|
||||
"""
|
||||
Worker coordination service.
|
||||
|
||||
Handles triggering and coordinating background workers.
|
||||
Supports two modes:
|
||||
- Cloud Tasks (production): Guaranteed delivery, automatic retries, horizontal scaling
|
||||
- Direct HTTP (development): Faster iteration, simpler debugging
|
||||
|
||||
SOLID Principles:
|
||||
- Single Responsibility: Only handles worker coordination
|
||||
- Dependency Inversion: Depends on HTTP abstraction, not implementation
|
||||
- Open/Closed: Can add new workers without modifying existing code
|
||||
|
||||
Observability:
|
||||
- Propagates trace context through Cloud Tasks for distributed tracing
|
||||
- All worker invocations create spans linked to original request trace
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
import json
|
||||
from typing import Optional
|
||||
import httpx
|
||||
from google.protobuf import duration_pb2
|
||||
|
||||
from backend.config import get_settings
|
||||
from backend.services.tracing import inject_trace_context
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Mapping of worker types to their Cloud Tasks queue names
|
||||
# These queues are created by Pulumi in infrastructure/__main__.py
|
||||
WORKER_QUEUES = {
|
||||
"audio": "audio-worker-queue",
|
||||
"lyrics": "lyrics-worker-queue",
|
||||
"screens": "screens-worker-queue",
|
||||
"render-video": "render-worker-queue",
|
||||
"video": "video-worker-queue",
|
||||
}
|
||||
|
||||
# Dispatch deadlines for each worker type (in seconds)
|
||||
# Cloud Tasks max is 1800s (30 min). These set how long Cloud Tasks waits
|
||||
# for the HTTP handler to respond before considering it failed.
|
||||
# Must be >= actual worker timeout + buffer for startup/upload.
|
||||
WORKER_DISPATCH_DEADLINES = {
|
||||
"audio": 1800, # 30 min - Modal separation can take 15-20 min
|
||||
"lyrics": 1500, # 25 min - TRANSCRIPTION_TIMEOUT_SECONDS is 1200s (20 min)
|
||||
"screens": 600, # 10 min - Screen generation is fast
|
||||
"render-video": 1800, # 30 min - Video encoding can be slow
|
||||
"video": 1800, # 30 min - Video encoding can be slow
|
||||
}
|
||||
|
||||
|
||||
class WorkerService:
|
||||
"""
|
||||
Service for coordinating background workers.
|
||||
|
||||
Supports two execution modes controlled by ENABLE_CLOUD_TASKS env var:
|
||||
|
||||
1. Cloud Tasks mode (production, ENABLE_CLOUD_TASKS=true):
|
||||
- Tasks enqueued to Cloud Tasks for guaranteed delivery
|
||||
- Automatic retries on failure
|
||||
- Each task runs in dedicated Cloud Run instance
|
||||
- Rate limiting to protect external APIs (Modal, AudioShake)
|
||||
|
||||
2. Direct HTTP mode (development, ENABLE_CLOUD_TASKS=false):
|
||||
- Direct HTTP POST to internal worker endpoints
|
||||
- Worker runs as FastAPI BackgroundTask
|
||||
- Faster iteration for local development
|
||||
- No retry guarantees
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize worker service."""
|
||||
self.settings = get_settings()
|
||||
self._base_url = self._get_base_url()
|
||||
self._admin_token = self._get_admin_token()
|
||||
self._use_cloud_tasks = self._should_use_cloud_tasks()
|
||||
self._tasks_client = None
|
||||
|
||||
if self._use_cloud_tasks:
|
||||
logger.info("WorkerService initialized with Cloud Tasks mode")
|
||||
else:
|
||||
logger.info("WorkerService initialized with direct HTTP mode")
|
||||
|
||||
def _should_use_cloud_tasks(self) -> bool:
|
||||
"""
|
||||
Check if Cloud Tasks should be used for worker coordination.
|
||||
|
||||
Controlled by ENABLE_CLOUD_TASKS setting (from environment variable).
|
||||
Default is false (direct HTTP mode) for backward compatibility.
|
||||
|
||||
Returns:
|
||||
True if Cloud Tasks should be used, False for direct HTTP
|
||||
"""
|
||||
return self.settings.enable_cloud_tasks
|
||||
|
||||
@property
|
||||
def tasks_client(self):
|
||||
"""
|
||||
Lazy-initialize Cloud Tasks client.
|
||||
|
||||
Only created when needed to avoid import overhead in development mode.
|
||||
"""
|
||||
if self._tasks_client is None and self._use_cloud_tasks:
|
||||
try:
|
||||
from google.cloud import tasks_v2
|
||||
self._tasks_client = tasks_v2.CloudTasksClient()
|
||||
except ImportError:
|
||||
logger.error(
|
||||
"google-cloud-tasks not installed. "
|
||||
"Install with: pip install google-cloud-tasks"
|
||||
)
|
||||
raise
|
||||
return self._tasks_client
|
||||
|
||||
def _get_admin_token(self) -> Optional[str]:
|
||||
"""
|
||||
Get admin token for internal API authentication.
|
||||
|
||||
Returns the first admin token from ADMIN_TOKENS env var.
|
||||
"""
|
||||
admin_tokens_str = self.settings.admin_tokens or ""
|
||||
tokens = [t.strip() for t in admin_tokens_str.split(",") if t.strip()]
|
||||
if tokens:
|
||||
return tokens[0]
|
||||
return None
|
||||
|
||||
def _get_base_url(self) -> str:
|
||||
"""
|
||||
Get base URL for internal API calls.
|
||||
|
||||
Priority:
|
||||
1. TEST_SERVER_URL env var (for tests)
|
||||
2. CLOUD_RUN_SERVICE_URL env var (for Cloud Tasks to call back)
|
||||
3. localhost with PORT env var (for development)
|
||||
|
||||
Returns:
|
||||
Base URL for API calls
|
||||
"""
|
||||
# Check for test environment override
|
||||
test_url = os.getenv("TEST_SERVER_URL")
|
||||
if test_url:
|
||||
return test_url
|
||||
|
||||
# Production: Cloud Run service URL
|
||||
# This must be the publicly accessible URL for Cloud Tasks to call
|
||||
service_url = os.getenv("CLOUD_RUN_SERVICE_URL")
|
||||
if service_url:
|
||||
return service_url
|
||||
|
||||
# Development mode: use localhost with PORT env var
|
||||
port = os.getenv("PORT", "8000")
|
||||
return f"http://localhost:{port}"
|
||||
|
||||
async def trigger_worker(
|
||||
self,
|
||||
worker_type: str,
|
||||
job_id: str,
|
||||
timeout_seconds: int = 30
|
||||
) -> bool:
|
||||
"""
|
||||
Trigger a background worker for a job.
|
||||
|
||||
In production (ENABLE_CLOUD_TASKS=true):
|
||||
- Enqueues task to Cloud Tasks queue
|
||||
- Cloud Tasks delivers HTTP request to internal endpoint
|
||||
- Automatic retries on failure with exponential backoff
|
||||
- Rate limiting protects external APIs
|
||||
|
||||
In development (ENABLE_CLOUD_TASKS=false):
|
||||
- Direct HTTP POST to internal endpoint
|
||||
- Faster iteration, but no retry guarantees
|
||||
- Worker runs in same container as API
|
||||
|
||||
Args:
|
||||
worker_type: Worker type ("audio", "lyrics", "screens", "render-video", "video")
|
||||
job_id: Job ID to process
|
||||
timeout_seconds: Request timeout (for direct HTTP mode)
|
||||
|
||||
Returns:
|
||||
True if trigger successful (or task enqueued), False otherwise
|
||||
"""
|
||||
if self._use_cloud_tasks:
|
||||
return await self._enqueue_cloud_task(worker_type, job_id)
|
||||
else:
|
||||
return await self._trigger_http(worker_type, job_id, timeout_seconds)
|
||||
|
||||
async def _enqueue_cloud_task(self, worker_type: str, job_id: str) -> bool:
|
||||
"""
|
||||
Enqueue task to Cloud Tasks for guaranteed delivery.
|
||||
|
||||
The task will be delivered as an HTTP POST to the internal worker endpoint.
|
||||
Cloud Tasks handles:
|
||||
- Retry on failure (with exponential backoff)
|
||||
- Rate limiting (max dispatches per second)
|
||||
- Deduplication (via task name if needed)
|
||||
- OIDC authentication for Cloud Run
|
||||
|
||||
Observability:
|
||||
- Injects trace context headers so worker spans link to parent trace
|
||||
|
||||
Args:
|
||||
worker_type: Worker type
|
||||
job_id: Job ID to process
|
||||
|
||||
Returns:
|
||||
True if task enqueued successfully, False otherwise
|
||||
"""
|
||||
try:
|
||||
from google.cloud import tasks_v2
|
||||
|
||||
queue_name = WORKER_QUEUES.get(worker_type)
|
||||
if not queue_name:
|
||||
logger.error(f"Unknown worker type: {worker_type}")
|
||||
return False
|
||||
|
||||
project = self.settings.google_cloud_project
|
||||
if not project:
|
||||
logger.error("GOOGLE_CLOUD_PROJECT not set, cannot enqueue Cloud Task")
|
||||
return False
|
||||
|
||||
location = self.settings.gcp_region # Must match queue location
|
||||
|
||||
# Build queue path
|
||||
parent = self.tasks_client.queue_path(project, location, queue_name)
|
||||
|
||||
# Build base headers
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
# Add admin auth via custom header
|
||||
# NOTE: We use X-Admin-Token instead of Authorization because Cloud Tasks
|
||||
# OIDC token overwrites the Authorization header when oidc_token is specified.
|
||||
# The OIDC token handles Cloud Run authentication (allows Cloud Tasks to invoke
|
||||
# the service), while X-Admin-Token handles application-level authentication.
|
||||
if self._admin_token:
|
||||
headers["X-Admin-Token"] = self._admin_token
|
||||
logger.debug(
|
||||
f"[job:{job_id}] Using admin token for Cloud Task auth via X-Admin-Token, "
|
||||
f"token prefix: {self._admin_token[:8]}..., len={len(self._admin_token)}"
|
||||
)
|
||||
|
||||
# Inject trace context for distributed tracing
|
||||
# This allows worker spans to link back to the original request trace
|
||||
headers = inject_trace_context(headers)
|
||||
|
||||
# Get dispatch deadline for this worker type (how long Cloud Tasks waits for response)
|
||||
dispatch_deadline_seconds = WORKER_DISPATCH_DEADLINES.get(worker_type, 600)
|
||||
|
||||
# Build task payload
|
||||
task = {
|
||||
"http_request": {
|
||||
"http_method": tasks_v2.HttpMethod.POST,
|
||||
"url": f"{self._base_url}/api/internal/workers/{worker_type}",
|
||||
"headers": headers,
|
||||
"body": json.dumps({"job_id": job_id}).encode(),
|
||||
# Use OIDC token for Cloud Run authentication
|
||||
# This allows Cloud Tasks to invoke the Cloud Run service
|
||||
"oidc_token": {
|
||||
"service_account_email": f"karaoke-backend@{project}.iam.gserviceaccount.com",
|
||||
},
|
||||
},
|
||||
# Set dispatch_deadline - how long Cloud Tasks waits for the handler to respond
|
||||
# Default is 10 min, but audio separation can take 30+ min
|
||||
# Max is 1800s (30 min) for Cloud Tasks
|
||||
"dispatch_deadline": duration_pb2.Duration(seconds=dispatch_deadline_seconds),
|
||||
}
|
||||
|
||||
# Create task
|
||||
# Note: We don't set a task name, allowing Cloud Tasks to generate one
|
||||
# This prevents duplicate task errors on retries
|
||||
response = self.tasks_client.create_task(parent=parent, task=task)
|
||||
logger.info(
|
||||
f"[job:{job_id}] Created Cloud Task for {worker_type} worker: {response.name} "
|
||||
f"(dispatch_deadline={dispatch_deadline_seconds}s)"
|
||||
)
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"[job:{job_id}] Failed to enqueue Cloud Task for {worker_type}: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
return False
|
||||
|
||||
async def _trigger_http(
|
||||
self,
|
||||
worker_type: str,
|
||||
job_id: str,
|
||||
timeout_seconds: int = 30
|
||||
) -> bool:
|
||||
"""
|
||||
Trigger worker via direct HTTP call (development mode).
|
||||
|
||||
This is the original implementation - direct HTTP POST to internal endpoint.
|
||||
The endpoint adds the worker function to FastAPI BackgroundTasks.
|
||||
|
||||
Observability:
|
||||
- Injects trace context headers so worker spans link to parent trace
|
||||
|
||||
Args:
|
||||
worker_type: Worker type
|
||||
job_id: Job ID to process
|
||||
timeout_seconds: Request timeout
|
||||
|
||||
Returns:
|
||||
True if trigger successful, False otherwise
|
||||
"""
|
||||
try:
|
||||
# Build headers with admin auth token
|
||||
headers = {}
|
||||
if self._admin_token:
|
||||
headers["Authorization"] = f"Bearer {self._admin_token}"
|
||||
|
||||
# Inject trace context for distributed tracing
|
||||
headers = inject_trace_context(headers)
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout_seconds) as client:
|
||||
url = f"{self._base_url}/api/internal/workers/{worker_type}"
|
||||
|
||||
response = await client.post(
|
||||
url,
|
||||
json={"job_id": job_id},
|
||||
headers=headers
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
logger.info(f"[job:{job_id}] Successfully triggered {worker_type} worker")
|
||||
return True
|
||||
else:
|
||||
logger.error(
|
||||
f"[job:{job_id}] Failed to trigger {worker_type} worker: "
|
||||
f"HTTP {response.status_code} - {response.text}"
|
||||
)
|
||||
return False
|
||||
|
||||
except httpx.TimeoutException:
|
||||
logger.error(f"[job:{job_id}] Timeout triggering {worker_type} worker")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"[job:{job_id}] Error triggering {worker_type} worker: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
return False
|
||||
|
||||
# Convenience methods for specific workers
|
||||
# These provide a cleaner API and better IDE autocomplete
|
||||
|
||||
async def trigger_audio_worker(self, job_id: str) -> bool:
|
||||
"""Trigger audio separation worker."""
|
||||
return await self.trigger_worker("audio", job_id)
|
||||
|
||||
async def trigger_lyrics_worker(self, job_id: str) -> bool:
|
||||
"""Trigger lyrics transcription worker."""
|
||||
return await self.trigger_worker("lyrics", job_id)
|
||||
|
||||
async def trigger_screens_worker(self, job_id: str) -> bool:
|
||||
"""Trigger screen generation worker."""
|
||||
return await self.trigger_worker("screens", job_id)
|
||||
|
||||
async def trigger_video_worker(self, job_id: str) -> bool:
|
||||
"""
|
||||
Trigger video generation worker.
|
||||
|
||||
When USE_CLOUD_RUN_JOBS_FOR_VIDEO=true and ENABLE_CLOUD_TASKS=true,
|
||||
uses Cloud Run Jobs for execution (supports >30 min encoding).
|
||||
Otherwise, uses Cloud Tasks or direct HTTP.
|
||||
"""
|
||||
if self._use_cloud_tasks and self.settings.use_cloud_run_jobs_for_video:
|
||||
return await self._trigger_cloud_run_job(job_id)
|
||||
return await self.trigger_worker("video", job_id)
|
||||
|
||||
async def _trigger_cloud_run_job(self, job_id: str) -> bool:
|
||||
"""
|
||||
Trigger video encoding as a Cloud Run Job.
|
||||
|
||||
Cloud Run Jobs support up to 24 hours of execution time,
|
||||
making them suitable for very long video encoding tasks.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to process
|
||||
|
||||
Returns:
|
||||
True if job was triggered successfully, False otherwise
|
||||
"""
|
||||
try:
|
||||
from google.cloud import run_v2
|
||||
|
||||
project = self.settings.google_cloud_project
|
||||
if not project:
|
||||
logger.error("GOOGLE_CLOUD_PROJECT not set, cannot trigger Cloud Run Job")
|
||||
return False
|
||||
|
||||
location = self.settings.gcp_region
|
||||
job_name = f"projects/{project}/locations/{location}/jobs/video-encoding-job"
|
||||
|
||||
# Create Cloud Run Jobs client
|
||||
client = run_v2.JobsClient()
|
||||
|
||||
# Run the job with overrides for the specific job_id
|
||||
request = run_v2.RunJobRequest(
|
||||
name=job_name,
|
||||
overrides=run_v2.RunJobRequest.Overrides(
|
||||
container_overrides=[
|
||||
run_v2.RunJobRequest.Overrides.ContainerOverride(
|
||||
args=["python", "-m", "backend.workers.video_worker", "--job-id", job_id],
|
||||
)
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
# Run the job (async operation)
|
||||
operation = client.run_job(request=request)
|
||||
logger.info(f"Started Cloud Run Job for video encoding, job {job_id}: {operation.metadata}")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Failed to trigger Cloud Run Job for video/{job_id}: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
return False
|
||||
|
||||
async def trigger_render_video_worker(self, job_id: str) -> bool:
|
||||
"""Trigger render video worker (post-review)."""
|
||||
return await self.trigger_worker("render-video", job_id)
|
||||
|
||||
|
||||
# Global worker service instance
|
||||
_worker_service: Optional[WorkerService] = None
|
||||
|
||||
|
||||
def get_worker_service() -> WorkerService:
|
||||
"""
|
||||
Get global worker service instance.
|
||||
|
||||
Singleton pattern to reuse HTTP client pool.
|
||||
|
||||
Returns:
|
||||
WorkerService instance
|
||||
"""
|
||||
global _worker_service
|
||||
if _worker_service is None:
|
||||
_worker_service = WorkerService()
|
||||
return _worker_service
|
||||
|
||||
|
||||
def reset_worker_service() -> None:
|
||||
"""
|
||||
Reset the global worker service instance.
|
||||
|
||||
Used in tests to ensure clean state between test cases.
|
||||
"""
|
||||
global _worker_service
|
||||
_worker_service = None
|
||||
@@ -0,0 +1,112 @@
|
||||
"""
|
||||
YouTube credential service for server-side video uploads.
|
||||
|
||||
This service manages YouTube OAuth credentials stored in Secret Manager
|
||||
for non-interactive uploads from backend workers.
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class YouTubeService:
|
||||
"""Service for managing YouTube OAuth credentials."""
|
||||
|
||||
# Secret Manager secret name for YouTube credentials
|
||||
YOUTUBE_CREDENTIALS_SECRET = "youtube-oauth-credentials"
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
self._credentials: Optional[Dict[str, Any]] = None
|
||||
self._loaded = False
|
||||
|
||||
def load_credentials(self) -> bool:
|
||||
"""
|
||||
Load YouTube OAuth credentials from Secret Manager.
|
||||
|
||||
The secret should contain a JSON object with:
|
||||
- token: Current access token (may be expired)
|
||||
- refresh_token: Refresh token for getting new access tokens
|
||||
- token_uri: Token endpoint URL
|
||||
- client_id: OAuth client ID
|
||||
- client_secret: OAuth client secret
|
||||
- scopes: List of OAuth scopes
|
||||
|
||||
Returns:
|
||||
True if credentials were loaded successfully, False otherwise
|
||||
"""
|
||||
if self._loaded:
|
||||
return self._credentials is not None
|
||||
|
||||
try:
|
||||
# Get credentials from Secret Manager
|
||||
creds_json = self.settings.get_secret(self.YOUTUBE_CREDENTIALS_SECRET)
|
||||
|
||||
if not creds_json:
|
||||
logger.warning("YouTube credentials not found in Secret Manager")
|
||||
self._loaded = True
|
||||
return False
|
||||
|
||||
# Parse the credentials JSON
|
||||
self._credentials = json.loads(creds_json)
|
||||
|
||||
# Validate required fields
|
||||
required_fields = ['refresh_token', 'token_uri', 'client_id', 'client_secret']
|
||||
missing = [f for f in required_fields if not self._credentials.get(f)]
|
||||
|
||||
if missing:
|
||||
logger.error(f"YouTube credentials missing required fields: {missing}")
|
||||
self._credentials = None
|
||||
self._loaded = True
|
||||
return False
|
||||
|
||||
logger.info("YouTube credentials loaded successfully from Secret Manager")
|
||||
self._loaded = True
|
||||
return True
|
||||
|
||||
except json.JSONDecodeError as e:
|
||||
logger.error(f"Failed to parse YouTube credentials JSON: {e}")
|
||||
self._loaded = True
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load YouTube credentials: {e}")
|
||||
self._loaded = True
|
||||
return False
|
||||
|
||||
def get_credentials_dict(self) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Get the YouTube credentials as a dictionary.
|
||||
|
||||
This format is compatible with KaraokeFinalise's user_youtube_credentials
|
||||
parameter, which creates a google.oauth2.credentials.Credentials object.
|
||||
|
||||
Returns:
|
||||
Dictionary with credential data, or None if not available
|
||||
"""
|
||||
if not self._loaded:
|
||||
self.load_credentials()
|
||||
|
||||
return self._credentials
|
||||
|
||||
@property
|
||||
def is_configured(self) -> bool:
|
||||
"""Check if YouTube credentials are configured and ready to use."""
|
||||
if not self._loaded:
|
||||
self.load_credentials()
|
||||
return self._credentials is not None
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_youtube_service: Optional[YouTubeService] = None
|
||||
|
||||
|
||||
def get_youtube_service() -> YouTubeService:
|
||||
"""Get the singleton YouTube service instance."""
|
||||
global _youtube_service
|
||||
if _youtube_service is None:
|
||||
_youtube_service = YouTubeService()
|
||||
return _youtube_service
|
||||
@@ -0,0 +1,445 @@
|
||||
"""
|
||||
YouTube Upload Service.
|
||||
|
||||
Provides video upload functionality for YouTube, extracted from KaraokeFinalise
|
||||
for use by both the cloud backend (video_worker) and local CLI.
|
||||
|
||||
This service handles:
|
||||
- Video uploads with metadata (title, description, tags)
|
||||
- Thumbnail uploads
|
||||
- Duplicate video detection and replacement
|
||||
- Authentication via pre-stored credentials or client secrets file
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import Optional, Dict, Any, Tuple
|
||||
|
||||
from thefuzz import fuzz
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class YouTubeUploadService:
|
||||
"""
|
||||
Service for uploading videos to YouTube.
|
||||
|
||||
Supports two authentication modes:
|
||||
1. Pre-stored credentials (for server-side/non-interactive use)
|
||||
2. Client secrets file (for interactive CLI use with browser OAuth)
|
||||
"""
|
||||
|
||||
YOUTUBE_URL_PREFIX = "https://www.youtube.com/watch?v="
|
||||
SCOPES = ["https://www.googleapis.com/auth/youtube"]
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
credentials: Optional[Dict[str, Any]] = None,
|
||||
client_secrets_file: Optional[str] = None,
|
||||
non_interactive: bool = False,
|
||||
server_side_mode: bool = False,
|
||||
dry_run: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the YouTube upload service.
|
||||
|
||||
Args:
|
||||
credentials: Pre-stored OAuth credentials dict with keys:
|
||||
- token: Access token
|
||||
- refresh_token: Refresh token
|
||||
- token_uri: Token endpoint URL
|
||||
- client_id: OAuth client ID
|
||||
- client_secret: OAuth client secret
|
||||
- scopes: List of OAuth scopes
|
||||
client_secrets_file: Path to OAuth client secrets JSON file
|
||||
(for interactive authentication)
|
||||
non_interactive: If True, skip interactive prompts
|
||||
server_side_mode: If True, use exact matching for duplicate detection
|
||||
dry_run: If True, log actions without performing them
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.credentials = credentials
|
||||
self.client_secrets_file = client_secrets_file
|
||||
self.non_interactive = non_interactive
|
||||
self.server_side_mode = server_side_mode
|
||||
self.dry_run = dry_run
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
self._youtube_service = None
|
||||
self._channel_id = None
|
||||
|
||||
def _authenticate(self):
|
||||
"""
|
||||
Authenticate with YouTube API and return service object.
|
||||
|
||||
Returns:
|
||||
YouTube API service object
|
||||
|
||||
Raises:
|
||||
Exception: If authentication fails or is required in non-interactive mode
|
||||
"""
|
||||
from google.auth.transport.requests import Request
|
||||
from google.oauth2.credentials import Credentials
|
||||
from googleapiclient.discovery import build
|
||||
from google_auth_oauthlib.flow import InstalledAppFlow
|
||||
import pickle
|
||||
|
||||
# Check if we have pre-stored credentials (for non-interactive mode)
|
||||
if self.credentials and self.non_interactive:
|
||||
try:
|
||||
creds = Credentials(
|
||||
token=self.credentials.get('token'),
|
||||
refresh_token=self.credentials.get('refresh_token'),
|
||||
token_uri=self.credentials.get('token_uri'),
|
||||
client_id=self.credentials.get('client_id'),
|
||||
client_secret=self.credentials.get('client_secret'),
|
||||
scopes=self.credentials.get('scopes', self.SCOPES)
|
||||
)
|
||||
|
||||
# Refresh token if needed
|
||||
if creds.expired and creds.refresh_token:
|
||||
creds.refresh(Request())
|
||||
|
||||
youtube = build('youtube', 'v3', credentials=creds)
|
||||
self.logger.info("Successfully authenticated with YouTube using pre-stored credentials")
|
||||
return youtube
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to authenticate with pre-stored credentials: {str(e)}")
|
||||
# Fall through to original authentication if pre-stored credentials fail
|
||||
|
||||
# For non-interactive mode without pre-stored credentials, raise error
|
||||
if self.non_interactive:
|
||||
raise Exception(
|
||||
"YouTube authentication required but running in non-interactive mode. "
|
||||
"Please pre-authenticate or disable YouTube upload."
|
||||
)
|
||||
|
||||
# Interactive authentication using client secrets file
|
||||
if not self.client_secrets_file:
|
||||
raise Exception("No YouTube credentials or client secrets file provided")
|
||||
|
||||
# Token file stores the user's access and refresh tokens
|
||||
youtube_token_file = "/tmp/karaoke-finalise-youtube-token.pickle"
|
||||
|
||||
creds = None
|
||||
|
||||
# Check if we have saved credentials
|
||||
if os.path.exists(youtube_token_file):
|
||||
with open(youtube_token_file, "rb") as token:
|
||||
creds = pickle.load(token)
|
||||
|
||||
# If there are no valid credentials, let the user log in
|
||||
if not creds or not creds.valid:
|
||||
if creds and creds.expired and creds.refresh_token:
|
||||
creds.refresh(Request())
|
||||
else:
|
||||
flow = InstalledAppFlow.from_client_secrets_file(
|
||||
self.client_secrets_file, scopes=self.SCOPES
|
||||
)
|
||||
creds = flow.run_local_server(port=0)
|
||||
|
||||
# Save the credentials for the next run
|
||||
with open(youtube_token_file, "wb") as token:
|
||||
pickle.dump(creds, token)
|
||||
|
||||
return build("youtube", "v3", credentials=creds)
|
||||
|
||||
@property
|
||||
def youtube_service(self):
|
||||
"""Lazy-load YouTube service on first access."""
|
||||
if self._youtube_service is None:
|
||||
self._youtube_service = self._authenticate()
|
||||
return self._youtube_service
|
||||
|
||||
def get_channel_id(self) -> Optional[str]:
|
||||
"""
|
||||
Get the authenticated user's YouTube channel ID.
|
||||
|
||||
Returns:
|
||||
Channel ID string, or None if not found
|
||||
"""
|
||||
if self._channel_id:
|
||||
return self._channel_id
|
||||
|
||||
request = self.youtube_service.channels().list(part="snippet", mine=True)
|
||||
response = request.execute()
|
||||
|
||||
if "items" in response and len(response["items"]) > 0:
|
||||
self._channel_id = response["items"][0]["id"]
|
||||
return self._channel_id
|
||||
|
||||
return None
|
||||
|
||||
def check_duplicate(
|
||||
self,
|
||||
title: str,
|
||||
exact_match: Optional[bool] = None
|
||||
) -> Tuple[bool, Optional[str], Optional[str]]:
|
||||
"""
|
||||
Check if a video with the given title already exists on the channel.
|
||||
|
||||
Args:
|
||||
title: Video title to search for
|
||||
exact_match: If True, require exact title match. If None, uses
|
||||
server_side_mode setting (exact in server mode, fuzzy in CLI)
|
||||
|
||||
Returns:
|
||||
Tuple of (exists: bool, video_id: Optional[str], video_url: Optional[str])
|
||||
"""
|
||||
channel_id = self.get_channel_id()
|
||||
if not channel_id:
|
||||
self.logger.warning("Could not get channel ID, skipping duplicate check")
|
||||
return False, None, None
|
||||
|
||||
use_exact_match = exact_match if exact_match is not None else self.server_side_mode
|
||||
|
||||
self.logger.info(f"Searching YouTube channel {channel_id} for title: {title}")
|
||||
request = self.youtube_service.search().list(
|
||||
part="snippet",
|
||||
channelId=channel_id,
|
||||
q=title,
|
||||
type="video",
|
||||
maxResults=10
|
||||
)
|
||||
response = request.execute()
|
||||
|
||||
if "items" not in response or len(response["items"]) == 0:
|
||||
self.logger.info(f"No matching video found with title: {title}")
|
||||
return False, None, None
|
||||
|
||||
for item in response["items"]:
|
||||
# Verify the video actually belongs to our channel
|
||||
result_channel_id = item["snippet"]["channelId"]
|
||||
if result_channel_id != channel_id:
|
||||
self.logger.debug(
|
||||
f"Skipping video from different channel: {item['snippet']['title']} "
|
||||
f"(channel: {result_channel_id})"
|
||||
)
|
||||
continue
|
||||
|
||||
found_title = item["snippet"]["title"]
|
||||
|
||||
# Determine if this is a match
|
||||
if use_exact_match:
|
||||
is_match = title.lower() == found_title.lower()
|
||||
similarity_score = 100 if is_match else 0
|
||||
else:
|
||||
similarity_score = fuzz.ratio(title.lower(), found_title.lower())
|
||||
is_match = similarity_score >= 70
|
||||
|
||||
if is_match:
|
||||
video_id = item["id"]["videoId"]
|
||||
video_url = f"{self.YOUTUBE_URL_PREFIX}{video_id}"
|
||||
self.logger.info(
|
||||
f"Potential match found on YouTube channel with ID: {video_id} "
|
||||
f"and title: {found_title} (similarity: {similarity_score}%)"
|
||||
)
|
||||
|
||||
# In non-interactive mode, return the match directly
|
||||
if self.non_interactive:
|
||||
self.logger.info("Non-interactive mode, found a match.")
|
||||
return True, video_id, video_url
|
||||
|
||||
# Interactive confirmation
|
||||
confirmation = input(
|
||||
f"Is '{found_title}' the video you are finalising? (y/n): "
|
||||
).strip().lower()
|
||||
if confirmation == "y":
|
||||
return True, video_id, video_url
|
||||
|
||||
self.logger.info(f"No matching video found with title: {title}")
|
||||
return False, None, None
|
||||
|
||||
def delete_video(self, video_id: str) -> bool:
|
||||
"""
|
||||
Delete a YouTube video by its ID.
|
||||
|
||||
Args:
|
||||
video_id: The YouTube video ID to delete
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
self.logger.info(f"Deleting YouTube video with ID: {video_id}")
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(f"DRY RUN: Would delete YouTube video with ID: {video_id}")
|
||||
return True
|
||||
|
||||
try:
|
||||
self.youtube_service.videos().delete(id=video_id).execute()
|
||||
self.logger.info(f"Successfully deleted YouTube video with ID: {video_id}")
|
||||
return True
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to delete YouTube video with ID {video_id}: {e}")
|
||||
return False
|
||||
|
||||
@staticmethod
|
||||
def truncate_title(title: str, max_length: int = 95) -> str:
|
||||
"""
|
||||
Truncate title to the nearest whole word within max_length.
|
||||
|
||||
Args:
|
||||
title: Title to truncate
|
||||
max_length: Maximum length (default 95 for YouTube)
|
||||
|
||||
Returns:
|
||||
Truncated title with "..." if needed
|
||||
"""
|
||||
if len(title) <= max_length:
|
||||
return title
|
||||
truncated_title = title[:max_length].rsplit(" ", 1)[0]
|
||||
if len(truncated_title) < max_length:
|
||||
truncated_title += " ..."
|
||||
return truncated_title
|
||||
|
||||
def upload_video(
|
||||
self,
|
||||
video_path: str,
|
||||
title: str,
|
||||
description: str,
|
||||
thumbnail_path: Optional[str] = None,
|
||||
tags: Optional[list] = None,
|
||||
category_id: str = "10", # Music category
|
||||
privacy_status: str = "public",
|
||||
replace_existing: bool = False,
|
||||
) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""
|
||||
Upload a video to YouTube with metadata and optional thumbnail.
|
||||
|
||||
Args:
|
||||
video_path: Path to the video file to upload
|
||||
title: Video title (will be truncated to 95 chars if needed)
|
||||
description: Video description
|
||||
thumbnail_path: Optional path to thumbnail image
|
||||
tags: Optional list of tags/keywords
|
||||
category_id: YouTube category ID (default "10" for Music)
|
||||
privacy_status: "public", "private", or "unlisted"
|
||||
replace_existing: If True, delete existing video with same title
|
||||
|
||||
Returns:
|
||||
Tuple of (video_id, video_url) or (None, None) if upload failed/skipped
|
||||
"""
|
||||
from googleapiclient.http import MediaFileUpload
|
||||
|
||||
# Truncate title if needed
|
||||
youtube_title = self.truncate_title(title)
|
||||
|
||||
self.logger.info(f"Uploading video to YouTube: {youtube_title}")
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(
|
||||
f"DRY RUN: Would upload {video_path} to YouTube with title: {youtube_title}"
|
||||
)
|
||||
return "dry_run_video_id", f"{self.YOUTUBE_URL_PREFIX}dry_run_video_id"
|
||||
|
||||
# Check for existing video
|
||||
should_replace = True if self.server_side_mode else replace_existing
|
||||
exists, existing_id, existing_url = self.check_duplicate(youtube_title)
|
||||
|
||||
if exists:
|
||||
if should_replace:
|
||||
self.logger.info(f"Video already exists on YouTube, deleting before re-upload: {existing_url}")
|
||||
if self.delete_video(existing_id):
|
||||
self.logger.info("Successfully deleted existing video, proceeding with upload")
|
||||
else:
|
||||
self.logger.error("Failed to delete existing video, aborting upload")
|
||||
return None, None
|
||||
else:
|
||||
self.logger.warning(f"Video already exists on YouTube, skipping upload: {existing_url}")
|
||||
return existing_id, existing_url
|
||||
|
||||
# Prepare video metadata
|
||||
body = {
|
||||
"snippet": {
|
||||
"title": youtube_title,
|
||||
"description": description,
|
||||
"tags": tags or [],
|
||||
"categoryId": category_id,
|
||||
},
|
||||
"status": {"privacyStatus": privacy_status},
|
||||
}
|
||||
|
||||
# Determine MIME type based on file extension
|
||||
ext = os.path.splitext(video_path)[1].lower()
|
||||
mime_type = {
|
||||
".mkv": "video/x-matroska",
|
||||
".mp4": "video/mp4",
|
||||
".mov": "video/quicktime",
|
||||
".avi": "video/x-msvideo",
|
||||
}.get(ext, "video/*")
|
||||
|
||||
# Upload video
|
||||
self.logger.info(f"Authenticating with YouTube...")
|
||||
media_file = MediaFileUpload(video_path, mimetype=mime_type, resumable=True)
|
||||
|
||||
self.logger.info(f"Uploading video to YouTube...")
|
||||
request = self.youtube_service.videos().insert(
|
||||
part="snippet,status",
|
||||
body=body,
|
||||
media_body=media_file
|
||||
)
|
||||
response = request.execute()
|
||||
|
||||
video_id = response.get("id")
|
||||
video_url = f"{self.YOUTUBE_URL_PREFIX}{video_id}"
|
||||
self.logger.info(f"Uploaded video to YouTube: {video_url}")
|
||||
|
||||
# Upload thumbnail if provided
|
||||
if thumbnail_path and os.path.isfile(thumbnail_path):
|
||||
try:
|
||||
self.logger.info(f"Uploading thumbnail from: {thumbnail_path}")
|
||||
media_thumbnail = MediaFileUpload(thumbnail_path, mimetype="image/jpeg")
|
||||
self.youtube_service.thumbnails().set(
|
||||
videoId=video_id,
|
||||
media_body=media_thumbnail
|
||||
).execute()
|
||||
self.logger.info(f"Uploaded thumbnail for video ID {video_id}")
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to upload thumbnail: {e}")
|
||||
self.logger.warning(
|
||||
"Video uploaded but thumbnail not set. "
|
||||
"You may need to set it manually on YouTube."
|
||||
)
|
||||
elif thumbnail_path:
|
||||
self.logger.warning(f"Thumbnail file not found, skipping: {thumbnail_path}")
|
||||
|
||||
return video_id, video_url
|
||||
|
||||
|
||||
# Singleton instance and factory function (following existing service pattern)
|
||||
_youtube_upload_service: Optional[YouTubeUploadService] = None
|
||||
|
||||
|
||||
def get_youtube_upload_service(
|
||||
credentials: Optional[Dict[str, Any]] = None,
|
||||
client_secrets_file: Optional[str] = None,
|
||||
**kwargs
|
||||
) -> YouTubeUploadService:
|
||||
"""
|
||||
Get a YouTube upload service instance.
|
||||
|
||||
For server-side use, pass credentials from YouTubeService.
|
||||
For CLI use, pass client_secrets_file.
|
||||
|
||||
Args:
|
||||
credentials: Pre-stored OAuth credentials dict
|
||||
client_secrets_file: Path to OAuth client secrets JSON file
|
||||
**kwargs: Additional arguments passed to YouTubeUploadService
|
||||
|
||||
Returns:
|
||||
YouTubeUploadService instance
|
||||
"""
|
||||
global _youtube_upload_service
|
||||
|
||||
# Create new instance if credentials/settings changed
|
||||
if _youtube_upload_service is None or credentials or client_secrets_file:
|
||||
_youtube_upload_service = YouTubeUploadService(
|
||||
credentials=credentials,
|
||||
client_secrets_file=client_secrets_file,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
return _youtube_upload_service
|
||||
@@ -0,0 +1,4 @@
|
||||
"""
|
||||
Backend integration tests package.
|
||||
"""
|
||||
|
||||
@@ -0,0 +1,224 @@
|
||||
"""
|
||||
Shared pytest fixtures for backend tests.
|
||||
|
||||
Provides common mocks and test utilities across all test modules.
|
||||
|
||||
NOTE: Module-level mocks are NOT applied here to allow emulator tests to work.
|
||||
Individual tests must mock dependencies as needed.
|
||||
"""
|
||||
# IMPORTANT: Set environment variables FIRST, before ANY imports that might
|
||||
# trigger backend module loading and Settings singleton creation.
|
||||
import os
|
||||
|
||||
# Set up test environment variables BEFORE importing any backend modules
|
||||
# This must happen before Settings is instantiated, which occurs on first import
|
||||
if 'FIRESTORE_EMULATOR_HOST' not in os.environ:
|
||||
os.environ.setdefault('ADMIN_TOKENS', 'test-admin-token')
|
||||
os.environ.setdefault('GOOGLE_CLOUD_PROJECT', 'test-project')
|
||||
os.environ.setdefault('GCS_BUCKET_NAME', 'test-bucket')
|
||||
os.environ.setdefault('FIRESTORE_COLLECTION', 'jobs')
|
||||
os.environ.setdefault('ENVIRONMENT', 'test')
|
||||
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, AsyncMock, patch
|
||||
from datetime import datetime, UTC
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
|
||||
# Mock google.auth.default AND firestore for unit tests if not using emulator
|
||||
# This prevents DefaultCredentialsError and FirestoreClient initialization during imports
|
||||
if 'FIRESTORE_EMULATOR_HOST' not in os.environ:
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
# Mock google.auth.default - prevents credential errors
|
||||
try:
|
||||
import google.auth
|
||||
mock_credentials = MagicMock()
|
||||
mock_credentials.token = 'fake-token'
|
||||
mock_credentials.valid = True
|
||||
mock_credentials.universe_domain = 'googleapis.com' # Required by google-cloud-storage
|
||||
mock_credentials.project_id = 'test-project'
|
||||
google.auth.default = MagicMock(return_value=(mock_credentials, 'test-project'))
|
||||
except ImportError:
|
||||
# If google.auth not installed, that's ok for some tests
|
||||
pass
|
||||
|
||||
# Mock google.cloud.firestore.Client - prevents Firestore initialization
|
||||
# This is CRITICAL: when AuthService is instantiated, it creates FirestoreService
|
||||
# which tries to create a real Firestore client. The client returns MagicMocks
|
||||
# which fail when used as enum values (e.g., UserType).
|
||||
try:
|
||||
import google.cloud.firestore as firestore_module
|
||||
|
||||
# Create a mock client that returns proper values for get_token
|
||||
mock_firestore_client = MagicMock()
|
||||
mock_collection = MagicMock()
|
||||
mock_doc = MagicMock()
|
||||
mock_snapshot = MagicMock()
|
||||
|
||||
# For auth token lookups, return None (token not found in Firestore)
|
||||
# This forces validation to use admin tokens from environment
|
||||
mock_snapshot.exists = False
|
||||
mock_snapshot.to_dict.return_value = None
|
||||
mock_doc.get.return_value = mock_snapshot
|
||||
mock_collection.document.return_value = mock_doc
|
||||
mock_firestore_client.collection.return_value = mock_collection
|
||||
|
||||
# Replace the Client class
|
||||
original_client = firestore_module.Client
|
||||
firestore_module.Client = MagicMock(return_value=mock_firestore_client)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
# Mock google.cloud.storage.Client - prevents GCS initialization
|
||||
try:
|
||||
import google.cloud.storage as storage_module
|
||||
mock_storage_client = MagicMock()
|
||||
mock_bucket = MagicMock()
|
||||
mock_storage_client.bucket.return_value = mock_bucket
|
||||
storage_module.Client = MagicMock(return_value=mock_storage_client)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
from backend.models.job import Job, JobStatus, JobCreate
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def mock_auth_dependency(request):
|
||||
"""Mock the require_auth dependency for all tests using FastAPI's dependency override system."""
|
||||
# Skip for emulator tests and integration tests which use real auth
|
||||
test_path = str(request.fspath)
|
||||
if 'emulator' in test_path or 'integration' in test_path:
|
||||
yield
|
||||
return
|
||||
|
||||
# Skip if FIRESTORE_EMULATOR_HOST is set (running in emulator environment)
|
||||
import os
|
||||
if os.environ.get('FIRESTORE_EMULATOR_HOST'):
|
||||
yield
|
||||
return
|
||||
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.api.dependencies import require_auth, require_admin, require_review_auth, require_instrumental_auth
|
||||
from backend.main import app
|
||||
|
||||
# Create mock auth functions
|
||||
async def mock_require_auth():
|
||||
"""Mock require_auth to always return valid admin credentials."""
|
||||
return ("test-admin-token", UserType.ADMIN, 999)
|
||||
|
||||
async def mock_require_admin():
|
||||
"""Mock require_admin to always return valid admin credentials."""
|
||||
return ("test-admin-token", UserType.ADMIN, 999)
|
||||
|
||||
async def mock_require_review_auth(job_id: str = "test123"):
|
||||
"""Mock require_review_auth to always return valid review access."""
|
||||
return (job_id, "full")
|
||||
|
||||
async def mock_require_instrumental_auth(job_id: str = "test123"):
|
||||
"""Mock require_instrumental_auth to always return valid instrumental review access."""
|
||||
return (job_id, "full")
|
||||
|
||||
# Use FastAPI's dependency override system
|
||||
app.dependency_overrides[require_auth] = mock_require_auth
|
||||
app.dependency_overrides[require_admin] = mock_require_admin
|
||||
app.dependency_overrides[require_review_auth] = mock_require_review_auth
|
||||
app.dependency_overrides[require_instrumental_auth] = mock_require_instrumental_auth
|
||||
|
||||
yield
|
||||
|
||||
# Clean up after test
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def auth_headers():
|
||||
"""Get authentication headers for test requests."""
|
||||
return {
|
||||
"Authorization": "Bearer test-admin-token",
|
||||
"Content-Type": "application/json"
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_firestore():
|
||||
"""Mock Firestore client for unit tests."""
|
||||
# Mock at module level for unit tests
|
||||
with patch('google.cloud.firestore.Client') as mock_cls:
|
||||
client = MagicMock()
|
||||
mock_cls.return_value = client
|
||||
yield client
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_storage_client():
|
||||
"""Mock GCS Storage client for unit tests."""
|
||||
# Mock at module level for unit tests
|
||||
with patch('google.cloud.storage.Client') as mock_cls:
|
||||
client = MagicMock()
|
||||
mock_cls.return_value = client
|
||||
yield client
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_httpx_client():
|
||||
"""Mock httpx AsyncClient for worker service."""
|
||||
with patch('backend.services.worker_service.httpx.AsyncClient') as mock:
|
||||
client = AsyncMock()
|
||||
mock.return_value.__aenter__.return_value = client
|
||||
yield client
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_job():
|
||||
"""Create a sample Job instance for testing."""
|
||||
return Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
progress=0,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
url="https://youtube.com/watch?v=test"
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_job_create():
|
||||
"""Create a sample JobCreate instance for testing."""
|
||||
return JobCreate(
|
||||
url="https://youtube.com/watch?v=test",
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def test_client():
|
||||
"""Create a FastAPI TestClient."""
|
||||
# Import here to avoid circular dependency
|
||||
from backend.main import app
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def create_mock_job(**kwargs):
|
||||
"""
|
||||
Factory function to create mock Job instances with custom fields.
|
||||
|
||||
Args:
|
||||
**kwargs: Job fields to override defaults
|
||||
|
||||
Returns:
|
||||
Job instance with specified fields
|
||||
"""
|
||||
defaults = {
|
||||
"job_id": "test123",
|
||||
"status": JobStatus.PENDING,
|
||||
"progress": 0,
|
||||
"created_at": datetime.now(UTC),
|
||||
"updated_at": datetime.now(UTC)
|
||||
}
|
||||
defaults.update(kwargs)
|
||||
return Job(**defaults)
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
"""
|
||||
Emulator integration tests.
|
||||
|
||||
These tests run against local GCP emulators (Firestore, GCS)
|
||||
to provide true integration testing without cloud costs.
|
||||
"""
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
"""
|
||||
Fixtures for emulator integration tests.
|
||||
|
||||
These tests use REAL Firestore and GCS emulators, so we DON'T mock google.cloud modules.
|
||||
Run with: scripts/run-emulator-tests.sh
|
||||
"""
|
||||
import pytest
|
||||
import os
|
||||
import requests
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
|
||||
def emulators_running() -> bool:
|
||||
"""Check if GCP emulators are running."""
|
||||
try:
|
||||
# Check Firestore emulator
|
||||
requests.get("http://127.0.0.1:8080", timeout=1)
|
||||
# Check GCS emulator
|
||||
requests.get("http://127.0.0.1:4443", timeout=1)
|
||||
return True
|
||||
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
|
||||
return False
|
||||
|
||||
|
||||
# Skip all tests in this module if emulators aren't running
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not emulators_running(),
|
||||
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
|
||||
)
|
||||
|
||||
# Set emulator environment variables (use 127.0.0.1 to force IPv4)
|
||||
os.environ["FIRESTORE_EMULATOR_HOST"] = "127.0.0.1:8080"
|
||||
os.environ["STORAGE_EMULATOR_HOST"] = "http://127.0.0.1:4443"
|
||||
os.environ["GOOGLE_CLOUD_PROJECT"] = "test-project"
|
||||
os.environ["GCS_BUCKET_NAME"] = "test-bucket"
|
||||
os.environ["FIRESTORE_COLLECTION"] = "test-jobs"
|
||||
os.environ["ENVIRONMENT"] = "test"
|
||||
os.environ["ADMIN_TOKENS"] = "test-admin-token"
|
||||
|
||||
# Only import app if emulators are running
|
||||
if emulators_running():
|
||||
from fastapi.testclient import TestClient
|
||||
from backend.main import app
|
||||
else:
|
||||
TestClient = None
|
||||
app = None
|
||||
|
||||
|
||||
@pytest.fixture(scope="session", autouse=True)
|
||||
def setup_gcs_bucket():
|
||||
"""Create GCS bucket in emulator before all tests."""
|
||||
try:
|
||||
response = requests.post(
|
||||
"http://127.0.0.1:4443/storage/v1/b",
|
||||
json={"name": "test-bucket"},
|
||||
params={"project": "test-project"},
|
||||
timeout=5
|
||||
)
|
||||
if response.status_code in [200, 409]:
|
||||
print("✅ GCS bucket 'test-bucket' ready in emulator")
|
||||
except Exception as e:
|
||||
print(f"⚠️ GCS bucket setup failed: {e}")
|
||||
yield
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def mock_worker_service():
|
||||
"""Mock worker service to prevent background tasks."""
|
||||
with patch("backend.api.routes.jobs.worker_service") as mock:
|
||||
mock.trigger_audio_worker = AsyncMock(return_value=True)
|
||||
mock.trigger_lyrics_worker = AsyncMock(return_value=True)
|
||||
mock.trigger_screens_worker = AsyncMock(return_value=True)
|
||||
mock.trigger_video_worker = AsyncMock(return_value=True)
|
||||
yield mock
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def client(mock_worker_service):
|
||||
"""Create FastAPI test client with mocked workers."""
|
||||
with patch("backend.api.routes.file_upload.worker_service", mock_worker_service):
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def auth_headers():
|
||||
"""Auth headers for testing."""
|
||||
return {"Authorization": "Bearer test-admin-token"}
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,356 @@
|
||||
"""
|
||||
Integration tests using local GCP emulators.
|
||||
|
||||
These tests use real Firestore and GCS emulators running locally,
|
||||
providing true integration testing without cloud resources or costs.
|
||||
|
||||
Run with: ./scripts/run-emulator-tests.sh
|
||||
|
||||
NOTE: These tests mock background workers to avoid race conditions.
|
||||
The fixtures are defined in conftest.py in this directory.
|
||||
"""
|
||||
import pytest
|
||||
import time
|
||||
|
||||
from .conftest import emulators_running
|
||||
|
||||
|
||||
# Skip all tests in this module if emulators aren't running
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not emulators_running(),
|
||||
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
|
||||
)
|
||||
|
||||
# Fixtures are loaded from conftest.py in this directory
|
||||
|
||||
|
||||
class TestEmulatorBasics:
|
||||
"""Basic emulator connectivity tests."""
|
||||
|
||||
def test_health_endpoint(self, client, auth_headers):
|
||||
"""Test health endpoint works."""
|
||||
response = client.get("/api/health", )
|
||||
assert response.status_code == 200
|
||||
assert response.json()["status"] == "healthy"
|
||||
|
||||
def test_root_endpoint(self, client, auth_headers):
|
||||
"""Test root endpoint works."""
|
||||
response = client.get("/", )
|
||||
assert response.status_code == 200
|
||||
assert response.json()["service"] == "karaoke-gen-backend"
|
||||
|
||||
|
||||
class TestJobCreation:
|
||||
"""Test job creation with Firestore emulator."""
|
||||
|
||||
def test_create_job_simple(self, client, auth_headers):
|
||||
"""Test creating a simple job."""
|
||||
response = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=test123"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
assert "job_id" in data
|
||||
|
||||
def test_create_job_with_metadata(self, client, auth_headers):
|
||||
"""Test creating a job with artist/title."""
|
||||
response = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={
|
||||
"url": "https://youtube.com/watch?v=test",
|
||||
"artist": "Test Artist",
|
||||
"title": "Test Song"
|
||||
}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "job_id" in data
|
||||
|
||||
|
||||
class TestJobRetrieval:
|
||||
"""Test job retrieval from Firestore emulator."""
|
||||
|
||||
def test_create_and_get_job(self, client, auth_headers):
|
||||
"""Test creating and then retrieving a job."""
|
||||
# Create
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={
|
||||
"url": "https://youtube.com/watch?v=abc123",
|
||||
"artist": "Test Artist",
|
||||
"title": "Test Song"
|
||||
}
|
||||
)
|
||||
assert create_resp.status_code == 200
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
# Small delay for emulator consistency
|
||||
time.sleep(0.2)
|
||||
|
||||
# Retrieve
|
||||
get_resp = client.get(f"/api/jobs/{job_id}", headers=auth_headers)
|
||||
|
||||
if get_resp.status_code != 200:
|
||||
print(f"GET failed: {get_resp.status_code} - {get_resp.text}")
|
||||
|
||||
assert get_resp.status_code == 200
|
||||
|
||||
job = get_resp.json()
|
||||
assert job["job_id"] == job_id
|
||||
assert job["status"] == "pending"
|
||||
assert job["artist"] == "Test Artist"
|
||||
assert job["title"] == "Test Song"
|
||||
|
||||
def test_get_nonexistent_job(self, client, auth_headers):
|
||||
"""Test fetching a job that doesn't exist."""
|
||||
response = client.get("/api/jobs/nonexistent-id", headers=auth_headers)
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
class TestJobList:
|
||||
"""Test listing jobs from Firestore."""
|
||||
|
||||
def test_list_jobs(self, client, auth_headers):
|
||||
"""Test listing jobs."""
|
||||
# Create a few jobs
|
||||
for i in range(3):
|
||||
client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": f"https://youtube.com/watch?v=list{i}"}
|
||||
)
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
# List
|
||||
response = client.get("/api/jobs", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
|
||||
jobs = response.json()
|
||||
assert isinstance(jobs, list)
|
||||
# We should have at least the 3 we just created
|
||||
assert len(jobs) >= 3
|
||||
|
||||
|
||||
class TestJobDeletion:
|
||||
"""Test job deletion."""
|
||||
|
||||
def test_delete_job(self, client, auth_headers):
|
||||
"""Test deleting a job."""
|
||||
# Create
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=delete-me"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
# Delete
|
||||
del_resp = client.delete(f"/api/jobs/{job_id}", headers=auth_headers)
|
||||
assert del_resp.status_code == 200
|
||||
|
||||
# Verify deleted
|
||||
get_resp = client.get(f"/api/jobs/{job_id}", headers=auth_headers)
|
||||
assert get_resp.status_code == 404
|
||||
|
||||
|
||||
class TestJobUpdates:
|
||||
"""Test job status updates."""
|
||||
|
||||
def test_cancel_job(self, client, auth_headers):
|
||||
"""Test cancelling a job."""
|
||||
# Create
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=cancel-me"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
# Cancel
|
||||
cancel_resp = client.post(
|
||||
f"/api/jobs/{job_id}/cancel",
|
||||
headers=auth_headers,
|
||||
json={"reason": "test cancellation"}
|
||||
)
|
||||
assert cancel_resp.status_code == 200
|
||||
|
||||
# Verify cancelled
|
||||
get_resp = client.get(f"/api/jobs/{job_id}", headers=auth_headers)
|
||||
assert get_resp.status_code == 200
|
||||
job = get_resp.json()
|
||||
assert job["status"] == "cancelled"
|
||||
|
||||
|
||||
class TestFileUpload:
|
||||
"""Test file upload with GCS emulator."""
|
||||
|
||||
def test_upload_file(self, client, auth_headers):
|
||||
"""Test uploading a file."""
|
||||
test_file_content = b"fake audio data for testing"
|
||||
|
||||
response = client.post(
|
||||
"/api/jobs/upload",
|
||||
headers={"Authorization": auth_headers["Authorization"]},
|
||||
files={"file": ("test.flac", test_file_content, "audio/flac")},
|
||||
data={"artist": "Upload Artist", "title": "Upload Song"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
assert "job_id" in data
|
||||
|
||||
job_id = data["job_id"]
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
# Verify job created with upload data
|
||||
get_resp = client.get(f"/api/jobs/{job_id}", headers=auth_headers)
|
||||
assert get_resp.status_code == 200
|
||||
job = get_resp.json()
|
||||
assert job["artist"] == "Upload Artist"
|
||||
assert job["title"] == "Upload Song"
|
||||
assert "input_media_gcs_path" in job
|
||||
|
||||
|
||||
class TestInternalEndpoints:
|
||||
"""Test internal worker endpoints."""
|
||||
|
||||
def test_internal_workers_exist(self, client, auth_headers):
|
||||
"""Test that internal worker endpoints exist and respond."""
|
||||
# Create a job first
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=worker-test"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
# Test audio worker endpoint exists
|
||||
response = client.post(
|
||||
"/api/internal/workers/audio",
|
||||
headers=auth_headers,
|
||||
json={"job_id": job_id}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert response.json()["status"] == "started"
|
||||
|
||||
|
||||
class TestReviewEndpoints:
|
||||
"""Test lyrics review API endpoints."""
|
||||
|
||||
def test_review_ping(self, client, auth_headers):
|
||||
"""Test review ping endpoint."""
|
||||
# Create a job first
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=review-test"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
# Ping should work for any job
|
||||
response = client.get(f"/api/review/{job_id}/ping")
|
||||
assert response.status_code == 200
|
||||
assert response.json()["status"] == "ok"
|
||||
|
||||
def test_review_correction_data_wrong_status(self, client, auth_headers):
|
||||
"""Test that correction-data returns error for non-review jobs."""
|
||||
# Create a job (starts in pending status)
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=review-status-test"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
# Should fail - job not in AWAITING_REVIEW status
|
||||
response = client.get(f"/api/review/{job_id}/correction-data", headers=auth_headers)
|
||||
assert response.status_code == 400
|
||||
assert "not ready for review" in response.json()["detail"].lower()
|
||||
|
||||
def test_review_audio_no_job(self, client, auth_headers):
|
||||
"""Test audio endpoint returns 404 for nonexistent job."""
|
||||
response = client.get("/api/review/nonexistent-job/audio/", headers=auth_headers)
|
||||
assert response.status_code == 404
|
||||
|
||||
def test_review_preview_video_stub(self, client, auth_headers):
|
||||
"""Test preview video endpoint exists."""
|
||||
# Create a job
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=preview-test"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
# Preview video should return error since job not ready
|
||||
response = client.post(
|
||||
f"/api/review/{job_id}/preview-video",
|
||||
headers=auth_headers,
|
||||
json={"corrections": [], "corrected_segments": []}
|
||||
)
|
||||
# Should return 400 (job not in AWAITING_REVIEW state)
|
||||
assert response.status_code == 400
|
||||
|
||||
def test_review_annotations_stub(self, client, auth_headers):
|
||||
"""Test annotations endpoint (stub)."""
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=annotation-test"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
# Annotations endpoint should accept data (even if it just logs it)
|
||||
response = client.post(
|
||||
f"/api/review/{job_id}/v1/annotations",
|
||||
headers=auth_headers,
|
||||
json={"type": "test", "data": "test annotation"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert response.json()["status"] == "success"
|
||||
|
||||
|
||||
class TestRenderVideoWorker:
|
||||
"""Test render video worker endpoint."""
|
||||
|
||||
def test_render_video_worker_endpoint_exists(self, client, auth_headers):
|
||||
"""Test that render-video worker endpoint exists."""
|
||||
# Create a job
|
||||
create_resp = client.post(
|
||||
"/api/jobs",
|
||||
headers=auth_headers,
|
||||
json={"url": "https://youtube.com/watch?v=render-test"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
# Test render-video worker endpoint exists
|
||||
response = client.post(
|
||||
"/api/internal/workers/render-video",
|
||||
headers=auth_headers,
|
||||
json={"job_id": job_id}
|
||||
)
|
||||
# Should return 200 (endpoint exists and starts)
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
print("✅ Emulator integration tests ready to run")
|
||||
@@ -0,0 +1,436 @@
|
||||
"""
|
||||
Direct tests for render_video_worker style loading.
|
||||
|
||||
Tests the fix for styles being loaded from job.style_params_gcs_path
|
||||
instead of the incorrect job.state_data['styles_gcs_path'].
|
||||
|
||||
Run with:
|
||||
./scripts/start-emulators.sh
|
||||
pytest backend/tests/emulator/test_style_loading_direct.py -v
|
||||
"""
|
||||
import pytest
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
import requests
|
||||
from dataclasses import dataclass
|
||||
from typing import Dict, Optional
|
||||
|
||||
|
||||
def emulators_running() -> bool:
|
||||
"""Check if GCP emulators are running."""
|
||||
try:
|
||||
requests.get("http://127.0.0.1:8080", timeout=1)
|
||||
requests.get("http://127.0.0.1:4443", timeout=1)
|
||||
return True
|
||||
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
|
||||
return False
|
||||
|
||||
|
||||
# Skip all tests in this module if emulators aren't running
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not emulators_running(),
|
||||
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
|
||||
)
|
||||
|
||||
# Set up environment for emulators
|
||||
os.environ["FIRESTORE_EMULATOR_HOST"] = "127.0.0.1:8080"
|
||||
os.environ["STORAGE_EMULATOR_HOST"] = "http://127.0.0.1:4443"
|
||||
os.environ["GOOGLE_CLOUD_PROJECT"] = "test-project"
|
||||
os.environ["GCS_BUCKET_NAME"] = "test-bucket"
|
||||
|
||||
|
||||
@dataclass
|
||||
class MockJob:
|
||||
"""Mock job object for testing style loading."""
|
||||
job_id: str
|
||||
style_params_gcs_path: Optional[str] = None
|
||||
style_assets: Dict[str, str] = None
|
||||
state_data: Dict = None
|
||||
|
||||
def __post_init__(self):
|
||||
if self.style_assets is None:
|
||||
self.style_assets = {}
|
||||
if self.state_data is None:
|
||||
self.state_data = {}
|
||||
|
||||
|
||||
class MockStorageService:
|
||||
"""Mock storage service that uses GCS emulator."""
|
||||
|
||||
def __init__(self):
|
||||
from google.cloud import storage
|
||||
self.client = storage.Client()
|
||||
self.bucket_name = "test-bucket"
|
||||
self._ensure_bucket_exists()
|
||||
|
||||
def _ensure_bucket_exists(self):
|
||||
"""Create bucket in emulator if it doesn't exist."""
|
||||
try:
|
||||
self.client.create_bucket(self.bucket_name)
|
||||
except Exception:
|
||||
pass # Bucket already exists
|
||||
|
||||
def upload_string(self, content: str, gcs_path: str, content_type: str = "application/json"):
|
||||
"""Upload string content to GCS."""
|
||||
bucket = self.client.bucket(self.bucket_name)
|
||||
blob = bucket.blob(gcs_path)
|
||||
blob.upload_from_string(content, content_type=content_type)
|
||||
|
||||
def upload_bytes(self, content: bytes, gcs_path: str, content_type: str = "application/octet-stream"):
|
||||
"""Upload bytes to GCS."""
|
||||
bucket = self.client.bucket(self.bucket_name)
|
||||
blob = bucket.blob(gcs_path)
|
||||
blob.upload_from_string(content, content_type=content_type)
|
||||
|
||||
def download_file(self, gcs_path: str, local_path: str):
|
||||
"""Download file from GCS to local path."""
|
||||
bucket = self.client.bucket(self.bucket_name)
|
||||
blob = bucket.blob(gcs_path)
|
||||
blob.download_to_filename(local_path)
|
||||
|
||||
def file_exists(self, gcs_path: str) -> bool:
|
||||
"""Check if file exists in GCS."""
|
||||
bucket = self.client.bucket(self.bucket_name)
|
||||
blob = bucket.blob(gcs_path)
|
||||
return blob.exists()
|
||||
|
||||
|
||||
class TestStyleLoadingFix:
|
||||
"""
|
||||
Tests for the render_video_worker style loading fix.
|
||||
|
||||
The bug was: render_video_worker looked for styles at
|
||||
job.state_data.get('styles_gcs_path') but styles are actually stored at
|
||||
job.style_params_gcs_path and job.style_assets.
|
||||
"""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup(self):
|
||||
"""Set up storage service and temp directory for each test."""
|
||||
self.storage = MockStorageService()
|
||||
self.temp_dir = tempfile.mkdtemp()
|
||||
yield
|
||||
# Cleanup
|
||||
import shutil
|
||||
shutil.rmtree(self.temp_dir, ignore_errors=True)
|
||||
|
||||
def _import_get_or_create_styles(self):
|
||||
"""
|
||||
Import _get_or_create_styles without importing the full worker module.
|
||||
We copy the function here to avoid import issues with lyrics_transcriber.
|
||||
"""
|
||||
# We'll test using a copy of the function logic
|
||||
# This avoids importing the full worker which has lyrics_transcriber dependency
|
||||
pass
|
||||
|
||||
def test_style_loading_from_correct_location(self):
|
||||
"""
|
||||
Test that styles are loaded from job.style_params_gcs_path,
|
||||
NOT from job.state_data['styles_gcs_path'].
|
||||
"""
|
||||
import time
|
||||
job_id = f"test-style-{int(time.time() * 1000)}"
|
||||
|
||||
# Create style JSON with placeholder paths
|
||||
style_json = {
|
||||
"intro": {
|
||||
"background_image": "/original/path/intro_bg.png",
|
||||
"font": "/original/path/font.ttf"
|
||||
},
|
||||
"karaoke": {
|
||||
"background_image": "/original/path/karaoke_bg.png",
|
||||
"font_path": "/original/path/font.ttf",
|
||||
"font_size": 100
|
||||
},
|
||||
"end": {
|
||||
"background_image": "/original/path/end_bg.png",
|
||||
"font": "/original/path/font.ttf"
|
||||
}
|
||||
}
|
||||
|
||||
# Upload style JSON to GCS (the CORRECT location)
|
||||
style_gcs_path = f"uploads/{job_id}/style/style_params.json"
|
||||
self.storage.upload_string(json.dumps(style_json), style_gcs_path)
|
||||
|
||||
# Upload mock assets
|
||||
assets = {
|
||||
"intro_background": f"uploads/{job_id}/style/intro_background.png",
|
||||
"karaoke_background": f"uploads/{job_id}/style/karaoke_background.png",
|
||||
"end_background": f"uploads/{job_id}/style/end_background.png",
|
||||
"font": f"uploads/{job_id}/style/font.ttf",
|
||||
}
|
||||
|
||||
for asset_key, gcs_path in assets.items():
|
||||
# Upload fake file content
|
||||
self.storage.upload_bytes(b"fake image/font data", gcs_path)
|
||||
|
||||
# Create mock job with styles in the CORRECT location
|
||||
job = MockJob(
|
||||
job_id=job_id,
|
||||
style_params_gcs_path=style_gcs_path, # CORRECT
|
||||
style_assets=assets, # CORRECT
|
||||
state_data={} # state_data is EMPTY (no styles_gcs_path)
|
||||
)
|
||||
|
||||
# Verify files exist in GCS
|
||||
assert self.storage.file_exists(style_gcs_path), "Style JSON should exist in GCS"
|
||||
for gcs_path in assets.values():
|
||||
assert self.storage.file_exists(gcs_path), f"Asset {gcs_path} should exist in GCS"
|
||||
|
||||
# Now test the style loading logic (inline version of _get_or_create_styles)
|
||||
style_dir = os.path.join(self.temp_dir, "style")
|
||||
os.makedirs(style_dir, exist_ok=True)
|
||||
styles_path = os.path.join(style_dir, "styles.json")
|
||||
|
||||
# This is the FIX: check job.style_params_gcs_path, not state_data
|
||||
if job.style_params_gcs_path:
|
||||
# Download style JSON
|
||||
self.storage.download_file(job.style_params_gcs_path, styles_path)
|
||||
|
||||
# Load and update paths
|
||||
with open(styles_path, 'r') as f:
|
||||
style_data = json.load(f)
|
||||
|
||||
# Download assets and update paths
|
||||
local_assets = {}
|
||||
for asset_key, gcs_path in job.style_assets.items():
|
||||
ext = os.path.splitext(gcs_path)[1] or '.png'
|
||||
local_path = os.path.join(style_dir, f"{asset_key}{ext}")
|
||||
self.storage.download_file(gcs_path, local_path)
|
||||
local_assets[asset_key] = local_path
|
||||
|
||||
# Update style_data with local paths
|
||||
asset_mapping = {
|
||||
'intro_background': ('intro', 'background_image'),
|
||||
'karaoke_background': ('karaoke', 'background_image'),
|
||||
'end_background': ('end', 'background_image'),
|
||||
'font': [('intro', 'font'), ('karaoke', 'font_path'), ('end', 'font')],
|
||||
}
|
||||
|
||||
for asset_key, local_path in local_assets.items():
|
||||
if asset_key in asset_mapping:
|
||||
mappings = asset_mapping[asset_key]
|
||||
if isinstance(mappings[0], str):
|
||||
mappings = [mappings]
|
||||
for section, field in mappings:
|
||||
if section in style_data:
|
||||
style_data[section][field] = local_path
|
||||
|
||||
# Save updated styles
|
||||
with open(styles_path, 'w') as f:
|
||||
json.dump(style_data, f, indent=2)
|
||||
|
||||
# Verify the result
|
||||
with open(styles_path, 'r') as f:
|
||||
result = json.load(f)
|
||||
|
||||
# Check that paths were updated to local paths
|
||||
assert style_dir in result['karaoke']['background_image'], \
|
||||
f"karaoke.background_image should be local path, got: {result['karaoke']['background_image']}"
|
||||
assert style_dir in result['karaoke']['font_path'], \
|
||||
f"karaoke.font_path should be local path, got: {result['karaoke']['font_path']}"
|
||||
assert style_dir in result['intro']['background_image'], \
|
||||
f"intro.background_image should be local path"
|
||||
|
||||
# Verify files actually exist locally
|
||||
assert os.path.exists(result['karaoke']['background_image']), \
|
||||
"Downloaded karaoke background should exist"
|
||||
assert os.path.exists(result['karaoke']['font_path']), \
|
||||
"Downloaded font should exist"
|
||||
|
||||
print(f"\n✅ Style loading from job.style_params_gcs_path works!")
|
||||
print(f" karaoke.background_image: {result['karaoke']['background_image']}")
|
||||
print(f" karaoke.font_path: {result['karaoke']['font_path']}")
|
||||
|
||||
def test_old_bug_state_data_lookup_fails(self):
|
||||
"""
|
||||
Demonstrate that the OLD bug would fail to find styles.
|
||||
|
||||
If we look at job.state_data['styles_gcs_path'] (the bug),
|
||||
we won't find anything because styles are stored at
|
||||
job.style_params_gcs_path.
|
||||
"""
|
||||
import time
|
||||
job_id = f"test-bug-{int(time.time() * 1000)}"
|
||||
|
||||
# Upload style to CORRECT location
|
||||
style_gcs_path = f"uploads/{job_id}/style/style_params.json"
|
||||
self.storage.upload_string('{"karaoke": {"font_size": 100}}', style_gcs_path)
|
||||
|
||||
# Create job with styles in CORRECT location
|
||||
job = MockJob(
|
||||
job_id=job_id,
|
||||
style_params_gcs_path=style_gcs_path, # CORRECT location
|
||||
style_assets={},
|
||||
state_data={} # No styles_gcs_path here!
|
||||
)
|
||||
|
||||
# THE BUG: looking in state_data instead of style_params_gcs_path
|
||||
wrong_path = job.state_data.get('styles_gcs_path')
|
||||
correct_path = job.style_params_gcs_path
|
||||
|
||||
assert wrong_path is None, "state_data['styles_gcs_path'] should be None (the bug location)"
|
||||
assert correct_path is not None, "style_params_gcs_path should have the correct path"
|
||||
assert self.storage.file_exists(correct_path), "Style should exist at correct path"
|
||||
|
||||
print(f"\n✅ Demonstrated the bug: state_data lookup returns None")
|
||||
print(f" state_data['styles_gcs_path'] = {wrong_path}")
|
||||
print(f" job.style_params_gcs_path = {correct_path}")
|
||||
|
||||
def test_default_styles_when_no_custom_styles(self):
|
||||
"""Test that default styles are used when no custom styles provided."""
|
||||
import time
|
||||
job_id = f"test-default-{int(time.time() * 1000)}"
|
||||
|
||||
# Job with NO custom styles
|
||||
job = MockJob(
|
||||
job_id=job_id,
|
||||
style_params_gcs_path=None,
|
||||
style_assets={},
|
||||
state_data={}
|
||||
)
|
||||
|
||||
# Simulate the logic
|
||||
style_dir = os.path.join(self.temp_dir, "style")
|
||||
os.makedirs(style_dir, exist_ok=True)
|
||||
styles_path = os.path.join(style_dir, "styles.json")
|
||||
|
||||
if job.style_params_gcs_path:
|
||||
# Would load custom styles
|
||||
pass
|
||||
else:
|
||||
# Use default styles
|
||||
default_styles = {
|
||||
"karaoke": {
|
||||
"background_color": "#000000",
|
||||
"font": "Arial",
|
||||
"font_path": "",
|
||||
"font_size": 100
|
||||
}
|
||||
}
|
||||
with open(styles_path, 'w') as f:
|
||||
json.dump(default_styles, f, indent=2)
|
||||
|
||||
# Verify defaults were used
|
||||
with open(styles_path, 'r') as f:
|
||||
result = json.load(f)
|
||||
|
||||
assert result['karaoke']['background_color'] == "#000000"
|
||||
assert result['karaoke']['font'] == "Arial"
|
||||
|
||||
print(f"\n✅ Default styles used when no custom styles provided")
|
||||
|
||||
|
||||
class TestParallelWorkerExecution:
|
||||
"""
|
||||
Tests for the parallel worker execution fix.
|
||||
|
||||
The bug was: FastAPI's BackgroundTasks runs async tasks sequentially,
|
||||
causing audio worker to complete before lyrics worker starts.
|
||||
|
||||
The fix uses asyncio.gather() to run both workers in parallel.
|
||||
"""
|
||||
|
||||
def test_asyncio_gather_runs_parallel(self):
|
||||
"""Test that asyncio.gather actually runs tasks in parallel."""
|
||||
import asyncio
|
||||
import time
|
||||
|
||||
start_times = {}
|
||||
end_times = {}
|
||||
|
||||
async def task1():
|
||||
start_times['task1'] = time.time()
|
||||
await asyncio.sleep(0.1)
|
||||
end_times['task1'] = time.time()
|
||||
return "task1 done"
|
||||
|
||||
async def task2():
|
||||
start_times['task2'] = time.time()
|
||||
await asyncio.sleep(0.1)
|
||||
end_times['task2'] = time.time()
|
||||
return "task2 done"
|
||||
|
||||
async def run_parallel():
|
||||
return await asyncio.gather(task1(), task2())
|
||||
|
||||
overall_start = time.time()
|
||||
results = asyncio.run(run_parallel())
|
||||
overall_duration = time.time() - overall_start
|
||||
|
||||
# Both tasks should have started at nearly the same time
|
||||
start_diff = abs(start_times['task1'] - start_times['task2'])
|
||||
|
||||
# If running in parallel:
|
||||
# - Both start at ~same time (diff < 0.05s)
|
||||
# - Total duration is ~0.1s (not 0.2s for sequential)
|
||||
|
||||
assert start_diff < 0.05, \
|
||||
f"Tasks should start together, diff was {start_diff:.3f}s"
|
||||
assert overall_duration < 0.15, \
|
||||
f"Parallel execution should take ~0.1s, took {overall_duration:.3f}s"
|
||||
|
||||
print(f"\n✅ asyncio.gather runs tasks in parallel!")
|
||||
print(f" Task start time difference: {start_diff:.3f}s")
|
||||
print(f" Total duration: {overall_duration:.3f}s (sequential would be ~0.2s)")
|
||||
|
||||
def test_sequential_background_tasks_is_slow(self):
|
||||
"""
|
||||
Demonstrate that sequential execution (the bug) is slower.
|
||||
"""
|
||||
import asyncio
|
||||
import time
|
||||
|
||||
execution_order = []
|
||||
|
||||
async def task1():
|
||||
execution_order.append(('task1', 'start'))
|
||||
await asyncio.sleep(0.05)
|
||||
execution_order.append(('task1', 'end'))
|
||||
|
||||
async def task2():
|
||||
execution_order.append(('task2', 'start'))
|
||||
await asyncio.sleep(0.05)
|
||||
execution_order.append(('task2', 'end'))
|
||||
|
||||
# Sequential (the bug)
|
||||
async def run_sequential():
|
||||
await task1()
|
||||
await task2()
|
||||
|
||||
execution_order.clear()
|
||||
start = time.time()
|
||||
asyncio.run(run_sequential())
|
||||
sequential_duration = time.time() - start
|
||||
sequential_order = execution_order.copy()
|
||||
|
||||
# Parallel (the fix)
|
||||
async def run_parallel():
|
||||
await asyncio.gather(task1(), task2())
|
||||
|
||||
execution_order.clear()
|
||||
start = time.time()
|
||||
asyncio.run(run_parallel())
|
||||
parallel_duration = time.time() - start
|
||||
parallel_order = execution_order.copy()
|
||||
|
||||
# Sequential: task1 starts, task1 ends, task2 starts, task2 ends
|
||||
assert sequential_order[0] == ('task1', 'start')
|
||||
assert sequential_order[1] == ('task1', 'end')
|
||||
assert sequential_order[2] == ('task2', 'start')
|
||||
|
||||
# Parallel: task1 starts, task2 starts (interleaved)
|
||||
assert parallel_order[0][1] == 'start'
|
||||
assert parallel_order[1][1] == 'start'
|
||||
|
||||
# Parallel should be ~2x faster
|
||||
assert parallel_duration < sequential_duration * 0.8, \
|
||||
f"Parallel ({parallel_duration:.3f}s) should be faster than sequential ({sequential_duration:.3f}s)"
|
||||
|
||||
print(f"\n✅ Demonstrated sequential vs parallel execution")
|
||||
print(f" Sequential: {sequential_duration:.3f}s - {sequential_order}")
|
||||
print(f" Parallel: {parallel_duration:.3f}s - {parallel_order}")
|
||||
|
||||
|
||||
print("✅ Style loading and parallel execution tests ready")
|
||||
@@ -0,0 +1,229 @@
|
||||
"""
|
||||
Direct tests for worker logging Firestore operations.
|
||||
|
||||
These tests bypass the full app/worker imports and test the Firestore
|
||||
operations directly, avoiding dependency issues.
|
||||
|
||||
Run with: ./scripts/run-emulator-tests.sh
|
||||
"""
|
||||
import pytest
|
||||
import time
|
||||
import requests
|
||||
import os
|
||||
import threading
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from datetime import datetime
|
||||
|
||||
|
||||
def emulators_running() -> bool:
|
||||
"""Check if GCP emulators are running."""
|
||||
try:
|
||||
requests.get("http://127.0.0.1:8080", timeout=1)
|
||||
return True
|
||||
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
|
||||
return False
|
||||
|
||||
|
||||
# Skip all tests in this module if emulators aren't running
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not emulators_running(),
|
||||
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
|
||||
)
|
||||
|
||||
# Set up environment for emulator
|
||||
os.environ["FIRESTORE_EMULATOR_HOST"] = "127.0.0.1:8080"
|
||||
os.environ["GOOGLE_CLOUD_PROJECT"] = "test-project"
|
||||
|
||||
|
||||
class TestWorkerLogsFirestoreDirect:
|
||||
"""Direct Firestore tests for worker logs - no app imports needed."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup_firestore(self):
|
||||
"""Set up Firestore client for each test."""
|
||||
from google.cloud import firestore
|
||||
self.db = firestore.Client(project="test-project")
|
||||
self.collection = "test-worker-logs"
|
||||
yield
|
||||
|
||||
def _create_test_job(self):
|
||||
"""Create a test job document."""
|
||||
job_id = f"test-{int(time.time() * 1000)}"
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.set({
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"created_at": datetime.utcnow(),
|
||||
"worker_logs": []
|
||||
})
|
||||
return job_id
|
||||
|
||||
def _append_log_read_modify_write(self, job_id: str, worker: str, message: str):
|
||||
"""
|
||||
OLD METHOD: Read-modify-write (has race condition).
|
||||
This is what we were doing before.
|
||||
"""
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc = doc_ref.get()
|
||||
if not doc.exists:
|
||||
return
|
||||
|
||||
data = doc.to_dict()
|
||||
logs = data.get("worker_logs", [])
|
||||
logs.append({
|
||||
"timestamp": datetime.utcnow().isoformat() + "Z",
|
||||
"level": "INFO",
|
||||
"worker": worker,
|
||||
"message": message
|
||||
})
|
||||
doc_ref.update({"worker_logs": logs})
|
||||
|
||||
def _append_log_array_union(self, job_id: str, worker: str, message: str):
|
||||
"""
|
||||
NEW METHOD: ArrayUnion (atomic, no race condition).
|
||||
This is what we're doing now.
|
||||
"""
|
||||
from google.cloud import firestore
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.update({
|
||||
"worker_logs": firestore.ArrayUnion([{
|
||||
"timestamp": datetime.utcnow().isoformat() + "Z",
|
||||
"level": "INFO",
|
||||
"worker": worker,
|
||||
"message": message
|
||||
}])
|
||||
})
|
||||
|
||||
def _get_logs(self, job_id: str):
|
||||
"""Get logs from job document."""
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc = doc_ref.get()
|
||||
if not doc.exists:
|
||||
return []
|
||||
return doc.to_dict().get("worker_logs", [])
|
||||
|
||||
def test_array_union_single_write(self):
|
||||
"""Test ArrayUnion works for single write."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
self._append_log_array_union(job_id, "test", "Single log message")
|
||||
|
||||
time.sleep(0.1)
|
||||
logs = self._get_logs(job_id)
|
||||
|
||||
assert len(logs) == 1
|
||||
assert logs[0]["message"] == "Single log message"
|
||||
assert logs[0]["worker"] == "test"
|
||||
|
||||
def test_array_union_sequential_writes(self):
|
||||
"""Test ArrayUnion preserves all sequential writes."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
for i in range(10):
|
||||
self._append_log_array_union(job_id, "test", f"Log {i}")
|
||||
|
||||
time.sleep(0.2)
|
||||
logs = self._get_logs(job_id)
|
||||
|
||||
assert len(logs) == 10, f"Expected 10 logs, got {len(logs)}"
|
||||
|
||||
def test_read_modify_write_race_condition(self):
|
||||
"""
|
||||
Demonstrate the race condition with read-modify-write.
|
||||
This test shows WHY we needed ArrayUnion.
|
||||
"""
|
||||
job_id = self._create_test_job()
|
||||
num_writes = 20
|
||||
|
||||
def write_log(index):
|
||||
self._append_log_read_modify_write(job_id, "worker", f"RMW Log {index}")
|
||||
|
||||
# Write concurrently - should lose some logs due to race condition
|
||||
with ThreadPoolExecutor(max_workers=5) as executor:
|
||||
futures = [executor.submit(write_log, i) for i in range(num_writes)]
|
||||
for future in as_completed(futures):
|
||||
try:
|
||||
future.result()
|
||||
except Exception:
|
||||
pass # Ignore errors for this test
|
||||
|
||||
time.sleep(0.5)
|
||||
logs = self._get_logs(job_id)
|
||||
|
||||
# With read-modify-write, we likely lose some logs
|
||||
# This is expected - the test documents the problem
|
||||
rmw_count = len([l for l in logs if "RMW Log" in l.get("message", "")])
|
||||
print(f"\nRead-modify-write: {rmw_count}/{num_writes} logs preserved")
|
||||
|
||||
# We expect to lose some logs (this is the bug we're fixing)
|
||||
# If all 20 are there, the race condition didn't trigger (which is fine)
|
||||
# The important thing is that ArrayUnion test below ALWAYS preserves all
|
||||
|
||||
def test_array_union_no_race_condition(self):
|
||||
"""
|
||||
Verify ArrayUnion preserves ALL concurrent writes.
|
||||
This is the critical test.
|
||||
"""
|
||||
job_id = self._create_test_job()
|
||||
num_writes = 20
|
||||
|
||||
def write_log(index):
|
||||
self._append_log_array_union(job_id, "worker", f"ArrayUnion Log {index}")
|
||||
|
||||
# Write concurrently
|
||||
with ThreadPoolExecutor(max_workers=5) as executor:
|
||||
futures = [executor.submit(write_log, i) for i in range(num_writes)]
|
||||
for future in as_completed(futures):
|
||||
future.result() # Raise any exceptions
|
||||
|
||||
time.sleep(0.5)
|
||||
logs = self._get_logs(job_id)
|
||||
|
||||
# With ArrayUnion, ALL logs should be preserved
|
||||
au_count = len([l for l in logs if "ArrayUnion Log" in l.get("message", "")])
|
||||
print(f"\nArrayUnion: {au_count}/{num_writes} logs preserved")
|
||||
|
||||
assert au_count == num_writes, \
|
||||
f"ArrayUnion should preserve all {num_writes} logs, got {au_count}"
|
||||
|
||||
def test_concurrent_workers_array_union(self):
|
||||
"""
|
||||
Test simulating audio and lyrics workers writing concurrently.
|
||||
"""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
def audio_worker():
|
||||
for i in range(10):
|
||||
self._append_log_array_union(job_id, "audio", f"Audio log {i}")
|
||||
time.sleep(0.01)
|
||||
|
||||
def lyrics_worker():
|
||||
for i in range(10):
|
||||
self._append_log_array_union(job_id, "lyrics", f"Lyrics log {i}")
|
||||
time.sleep(0.01)
|
||||
|
||||
# Start both workers
|
||||
audio_thread = threading.Thread(target=audio_worker)
|
||||
lyrics_thread = threading.Thread(target=lyrics_worker)
|
||||
|
||||
audio_thread.start()
|
||||
lyrics_thread.start()
|
||||
|
||||
audio_thread.join()
|
||||
lyrics_thread.join()
|
||||
|
||||
time.sleep(0.5)
|
||||
logs = self._get_logs(job_id)
|
||||
|
||||
audio_logs = [l for l in logs if l.get("worker") == "audio"]
|
||||
lyrics_logs = [l for l in logs if l.get("worker") == "lyrics"]
|
||||
|
||||
print(f"\nConcurrent workers: {len(audio_logs)} audio + {len(lyrics_logs)} lyrics")
|
||||
|
||||
assert len(audio_logs) == 10, f"Expected 10 audio logs, got {len(audio_logs)}"
|
||||
assert len(lyrics_logs) == 10, f"Expected 10 lyrics logs, got {len(lyrics_logs)}"
|
||||
|
||||
print("✅ All logs from both workers preserved!")
|
||||
|
||||
|
||||
print("✅ Direct Firestore worker logs tests ready")
|
||||
@@ -0,0 +1,443 @@
|
||||
"""
|
||||
Integration tests for worker log subcollection storage.
|
||||
|
||||
Tests the subcollection approach using real Firestore emulator to verify:
|
||||
- Logs are stored in subcollection (jobs/{job_id}/logs)
|
||||
- TTL expiry field is set correctly
|
||||
- Large log volumes work without 1MB limit issues
|
||||
- Concurrent writes are handled correctly
|
||||
- Logs can be queried efficiently
|
||||
|
||||
Run with: ./scripts/run-emulator-tests.sh
|
||||
"""
|
||||
import pytest
|
||||
import time
|
||||
import requests
|
||||
import os
|
||||
import threading
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from datetime import datetime, timezone, timedelta
|
||||
|
||||
|
||||
def emulators_running() -> bool:
|
||||
"""Check if GCP emulators are running."""
|
||||
try:
|
||||
requests.get("http://127.0.0.1:8080", timeout=1)
|
||||
return True
|
||||
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
|
||||
return False
|
||||
|
||||
|
||||
# Skip all tests in this module if emulators aren't running
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not emulators_running(),
|
||||
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
|
||||
)
|
||||
|
||||
# Set up environment for emulator
|
||||
os.environ["FIRESTORE_EMULATOR_HOST"] = "127.0.0.1:8080"
|
||||
os.environ["GOOGLE_CLOUD_PROJECT"] = "test-project"
|
||||
|
||||
|
||||
class TestWorkerLogsSubcollectionDirect:
|
||||
"""Direct Firestore tests for worker logs subcollection."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup_firestore(self):
|
||||
"""Set up Firestore client for each test."""
|
||||
from google.cloud import firestore
|
||||
self.db = firestore.Client(project="test-project")
|
||||
self.collection = "test-subcollection-jobs"
|
||||
yield
|
||||
|
||||
def _create_test_job(self):
|
||||
"""Create a test job document."""
|
||||
job_id = f"test-sub-{int(time.time() * 1000)}"
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.set({
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"created_at": datetime.now(timezone.utc),
|
||||
"worker_logs": [] # Legacy field - kept empty for new jobs
|
||||
})
|
||||
return job_id
|
||||
|
||||
def _append_log_to_subcollection(self, job_id: str, worker: str, message: str, ttl_days: int = 30):
|
||||
"""Add log to subcollection at jobs/{job_id}/logs."""
|
||||
import uuid
|
||||
log_id = str(uuid.uuid4())
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
logs_ref.document(log_id).set({
|
||||
"id": log_id,
|
||||
"job_id": job_id,
|
||||
"timestamp": now,
|
||||
"level": "INFO",
|
||||
"worker": worker,
|
||||
"message": message,
|
||||
"ttl_expiry": now + timedelta(days=ttl_days)
|
||||
})
|
||||
return log_id
|
||||
|
||||
def _get_logs_from_subcollection(self, job_id: str, worker: str = None, limit: int = 500):
|
||||
"""Get logs from subcollection."""
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
query = logs_ref.order_by("timestamp")
|
||||
|
||||
if worker:
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
query = query.where(filter=FieldFilter("worker", "==", worker))
|
||||
|
||||
query = query.limit(limit)
|
||||
|
||||
return [doc.to_dict() for doc in query.stream()]
|
||||
|
||||
def _count_logs_in_subcollection(self, job_id: str) -> int:
|
||||
"""Count logs in subcollection."""
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
count_query = logs_ref.count()
|
||||
result = count_query.get()
|
||||
if result and len(result) > 0:
|
||||
return result[0][0].value
|
||||
return 0
|
||||
|
||||
def _delete_logs_subcollection(self, job_id: str, batch_size: int = 100) -> int:
|
||||
"""Delete all logs in subcollection."""
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
deleted_count = 0
|
||||
|
||||
while True:
|
||||
docs = list(logs_ref.limit(batch_size).stream())
|
||||
if not docs:
|
||||
break
|
||||
|
||||
batch = self.db.batch()
|
||||
for doc in docs:
|
||||
batch.delete(doc.reference)
|
||||
deleted_count += 1
|
||||
|
||||
batch.commit()
|
||||
|
||||
return deleted_count
|
||||
|
||||
def test_subcollection_single_write(self):
|
||||
"""Test writing single log to subcollection."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
log_id = self._append_log_to_subcollection(job_id, "test", "Single log message")
|
||||
|
||||
time.sleep(0.1)
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(logs) == 1
|
||||
assert logs[0]["message"] == "Single log message"
|
||||
assert logs[0]["worker"] == "test"
|
||||
assert logs[0]["id"] == log_id
|
||||
print("Single write to subcollection works")
|
||||
|
||||
def test_subcollection_ttl_field_is_set(self):
|
||||
"""Test TTL expiry field is set correctly."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
self._append_log_to_subcollection(job_id, "test", "Log with TTL", ttl_days=7)
|
||||
|
||||
time.sleep(0.1)
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(logs) == 1
|
||||
ttl_expiry = logs[0]["ttl_expiry"]
|
||||
assert ttl_expiry is not None
|
||||
|
||||
# TTL should be ~7 days from now
|
||||
expected_ttl = datetime.now(timezone.utc) + timedelta(days=7)
|
||||
if hasattr(ttl_expiry, 'timestamp'):
|
||||
# Firestore datetime object
|
||||
diff = abs((ttl_expiry.replace(tzinfo=timezone.utc) - expected_ttl).total_seconds())
|
||||
else:
|
||||
diff = 0 # If already datetime
|
||||
assert diff < 60, f"TTL expiry should be ~7 days from now, diff was {diff}s"
|
||||
print("TTL field is set correctly")
|
||||
|
||||
def test_subcollection_sequential_writes(self):
|
||||
"""Test multiple sequential writes to subcollection."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
for i in range(20):
|
||||
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
|
||||
|
||||
time.sleep(0.2)
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(logs) == 20, f"Expected 20 logs, got {len(logs)}"
|
||||
print(f"Sequential writes: {len(logs)} logs created")
|
||||
|
||||
def test_subcollection_concurrent_writes(self):
|
||||
"""Test concurrent writes to subcollection - no race conditions."""
|
||||
job_id = self._create_test_job()
|
||||
num_writes = 50
|
||||
|
||||
def write_log(index):
|
||||
self._append_log_to_subcollection(job_id, "worker", f"Concurrent Log {index}")
|
||||
|
||||
with ThreadPoolExecutor(max_workers=10) as executor:
|
||||
futures = [executor.submit(write_log, i) for i in range(num_writes)]
|
||||
for future in as_completed(futures):
|
||||
future.result()
|
||||
|
||||
time.sleep(0.5)
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(logs) == num_writes, f"Expected {num_writes} logs, got {len(logs)}"
|
||||
print(f"Concurrent writes: All {num_writes} logs preserved")
|
||||
|
||||
def test_subcollection_large_volume(self):
|
||||
"""Test large volume of logs - demonstrates no 1MB limit."""
|
||||
job_id = self._create_test_job()
|
||||
num_logs = 500 # Would exceed 1MB in embedded array
|
||||
|
||||
# Write logs in batches
|
||||
batch_size = 50
|
||||
for batch_start in range(0, num_logs, batch_size):
|
||||
with ThreadPoolExecutor(max_workers=10) as executor:
|
||||
futures = []
|
||||
for i in range(batch_start, min(batch_start + batch_size, num_logs)):
|
||||
futures.append(executor.submit(
|
||||
self._append_log_to_subcollection,
|
||||
job_id,
|
||||
"stress",
|
||||
f"Large volume test log {i} - " + "x" * 500 # ~500 bytes per log
|
||||
))
|
||||
for future in as_completed(futures):
|
||||
future.result()
|
||||
|
||||
time.sleep(1)
|
||||
count = self._count_logs_in_subcollection(job_id)
|
||||
|
||||
assert count == num_logs, f"Expected {num_logs} logs, got {count}"
|
||||
print(f"Large volume: {count} logs created (~{count * 500 / 1024:.1f}KB would exceed 1MB if embedded)")
|
||||
|
||||
def test_subcollection_filter_by_worker(self):
|
||||
"""Test filtering logs by worker type."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
# Add logs from different workers
|
||||
for i in range(10):
|
||||
self._append_log_to_subcollection(job_id, "audio", f"Audio log {i}")
|
||||
self._append_log_to_subcollection(job_id, "lyrics", f"Lyrics log {i}")
|
||||
self._append_log_to_subcollection(job_id, "video", f"Video log {i}")
|
||||
|
||||
time.sleep(0.3)
|
||||
|
||||
# Query by worker
|
||||
audio_logs = self._get_logs_from_subcollection(job_id, worker="audio")
|
||||
lyrics_logs = self._get_logs_from_subcollection(job_id, worker="lyrics")
|
||||
all_logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(audio_logs) == 10, f"Expected 10 audio logs, got {len(audio_logs)}"
|
||||
assert len(lyrics_logs) == 10, f"Expected 10 lyrics logs, got {len(lyrics_logs)}"
|
||||
assert len(all_logs) == 30, f"Expected 30 total logs, got {len(all_logs)}"
|
||||
print("Worker filtering works correctly")
|
||||
|
||||
def test_subcollection_ordered_by_timestamp(self):
|
||||
"""Test logs are returned in timestamp order."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
# Add logs with slight delays to ensure different timestamps
|
||||
for i in range(5):
|
||||
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
|
||||
time.sleep(0.02)
|
||||
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
# Verify order (timestamps should be ascending)
|
||||
for i in range(len(logs) - 1):
|
||||
assert logs[i]["timestamp"] <= logs[i + 1]["timestamp"], \
|
||||
f"Logs not in order at index {i}"
|
||||
print("Logs are ordered by timestamp")
|
||||
|
||||
def test_subcollection_delete_all_logs(self):
|
||||
"""Test deleting all logs in subcollection."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
# Add logs
|
||||
for i in range(25):
|
||||
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
|
||||
|
||||
time.sleep(0.2)
|
||||
count_before = self._count_logs_in_subcollection(job_id)
|
||||
assert count_before == 25
|
||||
|
||||
# Delete all logs
|
||||
deleted = self._delete_logs_subcollection(job_id)
|
||||
|
||||
time.sleep(0.2)
|
||||
count_after = self._count_logs_in_subcollection(job_id)
|
||||
|
||||
assert deleted == 25, f"Expected to delete 25 logs, deleted {deleted}"
|
||||
assert count_after == 0, f"Expected 0 logs after delete, got {count_after}"
|
||||
print(f"Deleted {deleted} logs from subcollection")
|
||||
|
||||
def test_subcollection_concurrent_workers_interleaved(self):
|
||||
"""Test simulating audio and lyrics workers writing concurrently."""
|
||||
job_id = self._create_test_job()
|
||||
logs_per_worker = 20
|
||||
|
||||
def audio_worker():
|
||||
for i in range(logs_per_worker):
|
||||
self._append_log_to_subcollection(job_id, "audio", f"Audio processing step {i}")
|
||||
time.sleep(0.005)
|
||||
|
||||
def lyrics_worker():
|
||||
for i in range(logs_per_worker):
|
||||
self._append_log_to_subcollection(job_id, "lyrics", f"Lyrics processing step {i}")
|
||||
time.sleep(0.005)
|
||||
|
||||
def video_worker():
|
||||
for i in range(logs_per_worker):
|
||||
self._append_log_to_subcollection(job_id, "video", f"Video encoding step {i}")
|
||||
time.sleep(0.005)
|
||||
|
||||
# Start all workers
|
||||
threads = [
|
||||
threading.Thread(target=audio_worker),
|
||||
threading.Thread(target=lyrics_worker),
|
||||
threading.Thread(target=video_worker)
|
||||
]
|
||||
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
|
||||
time.sleep(0.5)
|
||||
|
||||
audio_logs = self._get_logs_from_subcollection(job_id, worker="audio")
|
||||
lyrics_logs = self._get_logs_from_subcollection(job_id, worker="lyrics")
|
||||
video_logs = self._get_logs_from_subcollection(job_id, worker="video")
|
||||
total = self._count_logs_in_subcollection(job_id)
|
||||
|
||||
assert len(audio_logs) == logs_per_worker
|
||||
assert len(lyrics_logs) == logs_per_worker
|
||||
assert len(video_logs) == logs_per_worker
|
||||
assert total == logs_per_worker * 3
|
||||
|
||||
print(f"Concurrent workers: {len(audio_logs)} audio + {len(lyrics_logs)} lyrics + {len(video_logs)} video = {total}")
|
||||
|
||||
def test_subcollection_job_deletion_cleans_up_logs(self):
|
||||
"""Test that deleting job document doesn't orphan logs."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
# Add logs
|
||||
for i in range(10):
|
||||
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
|
||||
|
||||
time.sleep(0.2)
|
||||
assert self._count_logs_in_subcollection(job_id) == 10
|
||||
|
||||
# Delete logs first (must happen before parent doc deletion in real code)
|
||||
deleted = self._delete_logs_subcollection(job_id)
|
||||
assert deleted == 10
|
||||
|
||||
# Now delete job document
|
||||
self.db.collection(self.collection).document(job_id).delete()
|
||||
|
||||
# Verify subcollection is empty (Firestore doesn't cascade delete subcollections)
|
||||
time.sleep(0.2)
|
||||
assert self._count_logs_in_subcollection(job_id) == 0
|
||||
print("Job deletion with log cleanup works correctly")
|
||||
|
||||
|
||||
class TestSubcollectionVsEmbeddedArray:
|
||||
"""Compare subcollection approach with embedded array."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup_firestore(self):
|
||||
"""Set up Firestore client for each test."""
|
||||
from google.cloud import firestore
|
||||
self.db = firestore.Client(project="test-project")
|
||||
self.collection = "test-comparison-jobs"
|
||||
yield
|
||||
|
||||
def _create_test_job(self, use_array: bool):
|
||||
"""Create test job."""
|
||||
job_id = f"test-cmp-{int(time.time() * 1000)}-{'arr' if use_array else 'sub'}"
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.set({
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"created_at": datetime.now(timezone.utc),
|
||||
"worker_logs": []
|
||||
})
|
||||
return job_id
|
||||
|
||||
def _append_log_array(self, job_id: str, message: str):
|
||||
"""Add log to embedded array."""
|
||||
from google.cloud import firestore
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.update({
|
||||
"worker_logs": firestore.ArrayUnion([{
|
||||
"timestamp": datetime.now(timezone.utc).isoformat() + "Z",
|
||||
"level": "INFO",
|
||||
"worker": "test",
|
||||
"message": message
|
||||
}])
|
||||
})
|
||||
|
||||
def _append_log_subcollection(self, job_id: str, message: str):
|
||||
"""Add log to subcollection."""
|
||||
import uuid
|
||||
now = datetime.now(timezone.utc)
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
logs_ref.document(str(uuid.uuid4())).set({
|
||||
"timestamp": now,
|
||||
"level": "INFO",
|
||||
"worker": "test",
|
||||
"message": message,
|
||||
"ttl_expiry": now + timedelta(days=30)
|
||||
})
|
||||
|
||||
def test_document_size_comparison(self):
|
||||
"""Compare document sizes between approaches."""
|
||||
array_job_id = self._create_test_job(use_array=True)
|
||||
sub_job_id = self._create_test_job(use_array=False)
|
||||
|
||||
num_logs = 100
|
||||
large_message = "x" * 5000 # 5KB per log
|
||||
|
||||
# Add logs to both
|
||||
print("\nAdding logs to both approaches...")
|
||||
for i in range(num_logs):
|
||||
self._append_log_array(array_job_id, f"Array log {i}: {large_message}")
|
||||
self._append_log_subcollection(sub_job_id, f"Sub log {i}: {large_message}")
|
||||
|
||||
time.sleep(0.5)
|
||||
|
||||
# Get document sizes (approximate via raw data)
|
||||
array_doc = self.db.collection(self.collection).document(array_job_id).get()
|
||||
sub_doc = self.db.collection(self.collection).document(sub_job_id).get()
|
||||
|
||||
array_data = array_doc.to_dict()
|
||||
sub_data = sub_doc.to_dict()
|
||||
|
||||
array_logs = array_data.get("worker_logs", [])
|
||||
sub_logs = sub_data.get("worker_logs", [])
|
||||
|
||||
print(f"\nEmbedded array: {len(array_logs)} logs in document")
|
||||
print(f"Subcollection: {len(sub_logs)} logs in document (0 expected)")
|
||||
|
||||
# Embedded array should have all logs in document
|
||||
assert len(array_logs) == num_logs
|
||||
|
||||
# Subcollection should have empty array in main document
|
||||
assert len(sub_logs) == 0
|
||||
|
||||
# Subcollection logs are in subcollection
|
||||
sub_count = self.db.collection(self.collection).document(sub_job_id).collection("logs").count().get()[0][0].value
|
||||
assert sub_count == num_logs
|
||||
|
||||
print(f"Subcollection logs stored separately: {sub_count}")
|
||||
print("Subcollection approach keeps main document small")
|
||||
|
||||
|
||||
print("Worker logs subcollection integration tests ready")
|
||||
@@ -0,0 +1,10 @@
|
||||
# Testing dependencies
|
||||
pytest>=7.4.0
|
||||
pytest-asyncio>=0.21.0
|
||||
pytest-cov>=4.1.0
|
||||
pytest-mock>=3.11.1
|
||||
httpx>=0.25.0 # For async testing
|
||||
|
||||
# Parent requirements (for import)
|
||||
-r ../requirements.txt
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
pytest>=8.0.0
|
||||
pytest-asyncio>=0.23.0
|
||||
pytest-timeout>=2.2.0
|
||||
pytest-cov>=4.1.0
|
||||
requests>=2.31.0
|
||||
|
||||
@@ -0,0 +1,460 @@
|
||||
"""
|
||||
Integration tests for the karaoke generation backend API.
|
||||
|
||||
These tests verify the backend works end-to-end with real Cloud Run deployment.
|
||||
These tests require a deployed backend and are marked to skip unless explicitly enabled.
|
||||
|
||||
Run with: pytest backend/tests/test_api_integration.py -m integration
|
||||
"""
|
||||
import pytest
|
||||
import requests
|
||||
import subprocess
|
||||
import time
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Optional, Dict, Any
|
||||
import os
|
||||
|
||||
|
||||
# Skip all tests unless explicitly running integration tests
|
||||
pytestmark = pytest.mark.skipif(
|
||||
os.environ.get("RUN_INTEGRATION_TESTS") != "true",
|
||||
reason="Integration tests require deployed backend. Set RUN_INTEGRATION_TESTS=true to run."
|
||||
)
|
||||
|
||||
# Configuration
|
||||
SERVICE_URL = "https://karaoke-backend-718638054799.us-central1.run.app"
|
||||
TEST_YOUTUBE_URL = "https://www.youtube.com/watch?v=dQw4w9WgXcQ" # Rick Astley - Never Gonna Give You Up
|
||||
DEFAULT_TIMEOUT = 30 # seconds
|
||||
|
||||
|
||||
def api_get(url: str, headers: Optional[Dict[str, str]] = None, **kwargs) -> requests.Response:
|
||||
"""Make a GET request with default timeout."""
|
||||
return requests.get(url, headers=headers, timeout=DEFAULT_TIMEOUT, **kwargs)
|
||||
|
||||
|
||||
def api_post(url: str, headers: Optional[Dict[str, str]] = None, **kwargs) -> requests.Response:
|
||||
"""Make a POST request with default timeout."""
|
||||
return requests.post(url, headers=headers, timeout=DEFAULT_TIMEOUT, **kwargs)
|
||||
|
||||
|
||||
def api_delete(url: str, headers: Optional[Dict[str, str]] = None, **kwargs) -> requests.Response:
|
||||
"""Make a DELETE request with default timeout."""
|
||||
return requests.delete(url, headers=headers, timeout=DEFAULT_TIMEOUT, **kwargs)
|
||||
|
||||
|
||||
def get_auth_token() -> str:
|
||||
"""Get authentication token for Cloud Run."""
|
||||
result = subprocess.run(
|
||||
["gcloud", "auth", "print-identity-token"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True
|
||||
)
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def auth_headers():
|
||||
"""Get authentication headers for requests."""
|
||||
token = get_auth_token()
|
||||
return {
|
||||
"Authorization": f"Bearer {token}",
|
||||
"Content-Type": "application/json"
|
||||
}
|
||||
|
||||
|
||||
class TestHealthEndpoint:
|
||||
"""Test health check endpoint."""
|
||||
|
||||
def test_health_check(self, auth_headers):
|
||||
"""Test that health endpoint returns 200 OK."""
|
||||
response = api_get(f"{SERVICE_URL}/api/health", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
|
||||
data = response.json()
|
||||
assert data["status"] == "healthy"
|
||||
assert data["service"] == "karaoke-gen-backend"
|
||||
|
||||
def test_health_check_without_auth(self):
|
||||
"""Test that health endpoint requires authentication."""
|
||||
response = api_get(f"{SERVICE_URL}/api/health")
|
||||
assert response.status_code == 403
|
||||
|
||||
|
||||
class TestRootEndpoint:
|
||||
"""Test root endpoint."""
|
||||
|
||||
def test_root_endpoint(self, auth_headers):
|
||||
"""Test root endpoint returns service info."""
|
||||
response = api_get(SERVICE_URL, headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
|
||||
data = response.json()
|
||||
assert data["service"] == "karaoke-gen-backend"
|
||||
assert data["status"] == "running"
|
||||
assert "version" in data
|
||||
|
||||
|
||||
class TestJobSubmission:
|
||||
"""Test job submission workflows."""
|
||||
|
||||
def test_submit_job_with_youtube_url(self, auth_headers):
|
||||
"""Test submitting a job with a YouTube URL."""
|
||||
payload = {"url": TEST_YOUTUBE_URL}
|
||||
response = api_post(
|
||||
f"{SERVICE_URL}/api/jobs",
|
||||
headers=auth_headers,
|
||||
json=payload
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
|
||||
assert data["status"] == "success"
|
||||
assert "job_id" in data
|
||||
assert len(data["job_id"]) > 0
|
||||
assert "message" in data
|
||||
|
||||
# Store job_id for cleanup
|
||||
return data["job_id"]
|
||||
|
||||
def test_submit_job_with_invalid_url(self, auth_headers):
|
||||
"""Test that invalid URLs are rejected."""
|
||||
payload = {"url": "not-a-url"}
|
||||
response = api_post(
|
||||
f"{SERVICE_URL}/api/jobs",
|
||||
headers=auth_headers,
|
||||
json=payload
|
||||
)
|
||||
|
||||
assert response.status_code == 422 # Validation error
|
||||
|
||||
def test_submit_job_without_url(self, auth_headers):
|
||||
"""Test that missing URL is rejected."""
|
||||
payload = {}
|
||||
response = api_post(
|
||||
f"{SERVICE_URL}/api/jobs",
|
||||
headers=auth_headers,
|
||||
json=payload
|
||||
)
|
||||
|
||||
assert response.status_code == 422 # Validation error
|
||||
|
||||
|
||||
class TestJobRetrieval:
|
||||
"""Test job retrieval and status checking."""
|
||||
|
||||
@pytest.fixture
|
||||
def test_job_id(self, auth_headers):
|
||||
"""Create a test job for retrieval tests."""
|
||||
payload = {"url": TEST_YOUTUBE_URL}
|
||||
response = api_post(
|
||||
f"{SERVICE_URL}/api/jobs",
|
||||
headers=auth_headers,
|
||||
json=payload
|
||||
)
|
||||
assert response.status_code == 200
|
||||
job_id = response.json()["job_id"]
|
||||
|
||||
yield job_id
|
||||
|
||||
# Cleanup
|
||||
api_delete(
|
||||
f"{SERVICE_URL}/api/jobs/{job_id}",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
def test_get_job_status(self, auth_headers, test_job_id):
|
||||
"""Test retrieving job status."""
|
||||
response = api_get(
|
||||
f"{SERVICE_URL}/api/jobs/{test_job_id}",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
|
||||
assert data["job_id"] == test_job_id
|
||||
assert "status" in data
|
||||
# Updated to use current JobStatus enum values
|
||||
assert data["status"] in ["pending", "downloading", "separating_stage1",
|
||||
"separating_stage2", "audio_complete",
|
||||
"transcribing", "correcting", "lyrics_complete",
|
||||
"generating_screens", "applying_padding",
|
||||
"awaiting_review", "in_review", "review_complete",
|
||||
"rendering_video", "awaiting_instrumental_selection",
|
||||
"instrumental_selected", "generating_video",
|
||||
"encoding", "packaging", "uploading", "notifying",
|
||||
"complete", "failed", "cancelled"]
|
||||
assert "progress" in data
|
||||
assert "created_at" in data
|
||||
assert "updated_at" in data
|
||||
assert "timeline" in data
|
||||
|
||||
def test_get_nonexistent_job(self, auth_headers):
|
||||
"""Test that requesting nonexistent job returns 404."""
|
||||
fake_job_id = "nonexistent-job-id"
|
||||
response = api_get(
|
||||
f"{SERVICE_URL}/api/jobs/{fake_job_id}",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
def test_list_jobs(self, auth_headers, test_job_id):
|
||||
"""Test listing all jobs."""
|
||||
response = api_get(
|
||||
f"{SERVICE_URL}/api/jobs",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
|
||||
assert isinstance(data, list)
|
||||
# Our test job should be in the list
|
||||
job_ids = [job["job_id"] for job in data]
|
||||
assert test_job_id in job_ids
|
||||
|
||||
def test_list_jobs_with_status_filter(self, auth_headers):
|
||||
"""Test filtering jobs by status."""
|
||||
response = api_get(
|
||||
f"{SERVICE_URL}/api/jobs?status=pending",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
|
||||
assert isinstance(data, list)
|
||||
# All returned jobs should be pending
|
||||
for job in data:
|
||||
assert job["status"] == "pending"
|
||||
|
||||
def test_list_jobs_with_limit(self, auth_headers):
|
||||
"""Test limiting number of returned jobs."""
|
||||
response = api_get(
|
||||
f"{SERVICE_URL}/api/jobs?limit=5",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
|
||||
assert isinstance(data, list)
|
||||
assert len(data) <= 5
|
||||
|
||||
|
||||
class TestJobDeletion:
|
||||
"""Test job deletion."""
|
||||
|
||||
def test_delete_job(self, auth_headers):
|
||||
"""Test deleting a job."""
|
||||
# Create a job
|
||||
payload = {"url": TEST_YOUTUBE_URL}
|
||||
response = api_post(
|
||||
f"{SERVICE_URL}/api/jobs",
|
||||
headers=auth_headers,
|
||||
json=payload
|
||||
)
|
||||
assert response.status_code == 200
|
||||
job_id = response.json()["job_id"]
|
||||
|
||||
# Delete the job
|
||||
response = api_delete(
|
||||
f"{SERVICE_URL}/api/jobs/{job_id}",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
|
||||
# Verify job is deleted
|
||||
response = api_get(
|
||||
f"{SERVICE_URL}/api/jobs/{job_id}",
|
||||
headers=auth_headers
|
||||
)
|
||||
assert response.status_code == 404
|
||||
|
||||
def test_delete_job_without_files(self, auth_headers):
|
||||
"""Test deleting a job without deleting files."""
|
||||
# Create a job
|
||||
payload = {"url": TEST_YOUTUBE_URL}
|
||||
response = api_post(
|
||||
f"{SERVICE_URL}/api/jobs",
|
||||
headers=auth_headers,
|
||||
json=payload
|
||||
)
|
||||
assert response.status_code == 200
|
||||
job_id = response.json()["job_id"]
|
||||
|
||||
# Delete job but keep files
|
||||
response = api_delete(
|
||||
f"{SERVICE_URL}/api/jobs/{job_id}?delete_files=false",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
|
||||
def test_delete_nonexistent_job(self, auth_headers):
|
||||
"""Test deleting nonexistent job returns 404."""
|
||||
fake_job_id = "nonexistent-job-id"
|
||||
response = api_delete(
|
||||
f"{SERVICE_URL}/api/jobs/{fake_job_id}",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
class TestFileUpload:
|
||||
"""Test file upload endpoint."""
|
||||
|
||||
def test_upload_audio_file(self, auth_headers, tmp_path):
|
||||
"""Test uploading an audio file."""
|
||||
# Create a small test file
|
||||
test_file = tmp_path / "test_audio.mp3"
|
||||
test_file.write_bytes(b"fake audio content for testing")
|
||||
|
||||
# Upload file
|
||||
with open(test_file, "rb") as f:
|
||||
files = {"file": ("test_audio.mp3", f, "audio/mpeg")}
|
||||
data = {
|
||||
"artist": "Test Artist",
|
||||
"title": "Test Song"
|
||||
}
|
||||
|
||||
# Remove Content-Type header for multipart/form-data
|
||||
headers = {
|
||||
"Authorization": auth_headers["Authorization"]
|
||||
}
|
||||
|
||||
response = api_post(
|
||||
f"{SERVICE_URL}/api/jobs/upload",
|
||||
headers=headers,
|
||||
files=files,
|
||||
data=data
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
result = response.json()
|
||||
|
||||
assert result["status"] == "success"
|
||||
assert "job_id" in result
|
||||
|
||||
# Cleanup
|
||||
api_delete(
|
||||
f"{SERVICE_URL}/api/jobs/{result['job_id']}",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
def test_upload_invalid_file_type(self, auth_headers, tmp_path):
|
||||
"""Test that invalid file types are rejected."""
|
||||
# Create a test file with invalid extension
|
||||
test_file = tmp_path / "test.txt"
|
||||
test_file.write_bytes(b"not an audio file")
|
||||
|
||||
with open(test_file, "rb") as f:
|
||||
files = {"file": ("test.txt", f, "text/plain")}
|
||||
data = {
|
||||
"artist": "Test Artist",
|
||||
"title": "Test Song"
|
||||
}
|
||||
|
||||
headers = {
|
||||
"Authorization": auth_headers["Authorization"]
|
||||
}
|
||||
|
||||
response = api_post(
|
||||
f"{SERVICE_URL}/api/jobs/upload",
|
||||
headers=headers,
|
||||
files=files,
|
||||
data=data
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
|
||||
def test_upload_without_metadata(self, auth_headers, tmp_path):
|
||||
"""Test that uploads without artist/title are rejected."""
|
||||
test_file = tmp_path / "test_audio.mp3"
|
||||
test_file.write_bytes(b"fake audio content")
|
||||
|
||||
with open(test_file, "rb") as f:
|
||||
files = {"file": ("test_audio.mp3", f, "audio/mpeg")}
|
||||
|
||||
headers = {
|
||||
"Authorization": auth_headers["Authorization"]
|
||||
}
|
||||
|
||||
response = api_post(
|
||||
f"{SERVICE_URL}/api/jobs/upload",
|
||||
headers=headers,
|
||||
files=files
|
||||
)
|
||||
|
||||
assert response.status_code == 422 # Validation error
|
||||
|
||||
|
||||
class TestJobProcessing:
|
||||
"""Test end-to-end job processing (long-running)."""
|
||||
|
||||
@pytest.mark.slow
|
||||
@pytest.mark.integration
|
||||
def test_complete_job_workflow(self, auth_headers):
|
||||
"""
|
||||
Test complete job workflow from submission to completion.
|
||||
|
||||
This test is slow and requires actual processing.
|
||||
Run with: pytest -m slow
|
||||
"""
|
||||
# Submit job
|
||||
payload = {"url": TEST_YOUTUBE_URL}
|
||||
response = api_post(
|
||||
f"{SERVICE_URL}/api/jobs",
|
||||
headers=auth_headers,
|
||||
json=payload
|
||||
)
|
||||
assert response.status_code == 200
|
||||
job_id = response.json()["job_id"]
|
||||
|
||||
# Poll for completion (with timeout)
|
||||
timeout = 600 # 10 minutes
|
||||
start_time = time.time()
|
||||
|
||||
while time.time() - start_time < timeout:
|
||||
response = api_get(
|
||||
f"{SERVICE_URL}/api/jobs/{job_id}",
|
||||
headers=auth_headers
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
job = response.json()
|
||||
status = job["status"]
|
||||
|
||||
if status == "complete":
|
||||
# Verify outputs exist
|
||||
assert "download_urls" in job
|
||||
assert len(job["download_urls"]) > 0
|
||||
break
|
||||
elif status == "failed":
|
||||
pytest.fail(f"Job failed: {job.get('error_message')}")
|
||||
|
||||
# Wait before next poll
|
||||
time.sleep(10)
|
||||
else:
|
||||
pytest.fail(f"Job did not complete within {timeout} seconds")
|
||||
|
||||
# Cleanup
|
||||
api_delete(
|
||||
f"{SERVICE_URL}/api/jobs/{job_id}",
|
||||
headers=auth_headers
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v", "--tb=short"])
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
"""
|
||||
Unit tests for API routes.
|
||||
|
||||
These tests use FastAPI TestClient with mocked services to test
|
||||
route logic without hitting real cloud services.
|
||||
"""
|
||||
import pytest
|
||||
import json
|
||||
from datetime import datetime, UTC
|
||||
from unittest.mock import MagicMock, AsyncMock, patch
|
||||
from fastapi.testclient import TestClient
|
||||
from io import BytesIO
|
||||
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
class TestHealthRoutes:
|
||||
"""Tests for health.py routes."""
|
||||
|
||||
@pytest.fixture
|
||||
def client(self):
|
||||
"""Create test client with mocked dependencies."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.universe_domain = 'googleapis.com'
|
||||
with patch('backend.services.firestore_service.firestore'), \
|
||||
patch('backend.services.storage_service.storage'), \
|
||||
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
||||
from backend.main import app
|
||||
return TestClient(app)
|
||||
|
||||
def test_health_endpoint_returns_200(self, client, auth_headers):
|
||||
"""Test /api/health returns 200 OK."""
|
||||
response = client.get("/api/health", )
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_health_endpoint_returns_healthy_status(self, client, auth_headers):
|
||||
"""Test health endpoint returns healthy status."""
|
||||
response = client.get("/api/health", )
|
||||
data = response.json()
|
||||
assert data["status"] == "healthy"
|
||||
|
||||
def test_root_endpoint_returns_200(self, client, auth_headers):
|
||||
"""Test root endpoint returns 200."""
|
||||
response = client.get("/", )
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
class TestJobRoutes:
|
||||
"""Tests for jobs.py routes.
|
||||
|
||||
Note: These tests verify the route module structure.
|
||||
Full integration tests are in test_api_integration.py.
|
||||
"""
|
||||
|
||||
def test_jobs_router_exists(self):
|
||||
"""Test jobs router can be imported."""
|
||||
from backend.api.routes import jobs
|
||||
assert hasattr(jobs, 'router')
|
||||
|
||||
def test_jobs_router_has_expected_endpoints(self):
|
||||
"""Test jobs router defines expected endpoints."""
|
||||
from backend.api.routes.jobs import router
|
||||
routes = [route.path for route in router.routes]
|
||||
assert '/jobs' in routes or any('/jobs' in r for r in routes)
|
||||
|
||||
|
||||
class TestInternalRoutes:
|
||||
"""Tests for internal.py routes.
|
||||
|
||||
Note: These tests are minimal because the internal routes
|
||||
trigger actual worker processing which requires complex mocking.
|
||||
The actual worker logic is tested in test_workers.py.
|
||||
"""
|
||||
|
||||
def test_internal_endpoint_structure(self):
|
||||
"""Test internal route module has expected endpoints."""
|
||||
from backend.api.routes import internal
|
||||
# Just verify the module can be imported
|
||||
assert hasattr(internal, 'router')
|
||||
|
||||
|
||||
class TestFileUploadRoutes:
|
||||
"""Tests for file_upload.py routes.
|
||||
|
||||
Note: These tests verify the route module structure.
|
||||
Full integration tests are in test_api_integration.py.
|
||||
"""
|
||||
|
||||
def test_file_upload_router_exists(self):
|
||||
"""Test file upload router can be imported."""
|
||||
from backend.api.routes import file_upload
|
||||
assert hasattr(file_upload, 'router')
|
||||
|
||||
@@ -0,0 +1,294 @@
|
||||
"""
|
||||
Tests for audio_analysis_service.py - Audio analysis for backing vocals.
|
||||
|
||||
These tests mock the storage service and shared karaoke_gen modules to verify:
|
||||
- GCS file download/upload operations
|
||||
- Audio analysis delegation to shared AudioAnalyzer
|
||||
- Waveform generation delegation to shared WaveformGenerator
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, patch, ANY
|
||||
import tempfile
|
||||
import os
|
||||
|
||||
|
||||
class TestAudioAnalysisServiceInit:
|
||||
"""Test AudioAnalysisService initialization."""
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_init_creates_dependencies(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test initialization creates storage service and analyzers."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
service = AudioAnalysisService()
|
||||
|
||||
mock_storage_class.assert_called_once()
|
||||
mock_analyzer_class.assert_called_once_with(
|
||||
silence_threshold_db=-40.0,
|
||||
min_segment_duration_ms=100,
|
||||
)
|
||||
mock_waveform_class.assert_called_once()
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_init_with_custom_params(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test initialization with custom threshold and duration."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
service = AudioAnalysisService(
|
||||
silence_threshold_db=-50.0,
|
||||
min_segment_duration_ms=200,
|
||||
)
|
||||
|
||||
mock_analyzer_class.assert_called_once_with(
|
||||
silence_threshold_db=-50.0,
|
||||
min_segment_duration_ms=200,
|
||||
)
|
||||
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_init_with_provided_storage_service(
|
||||
self, mock_waveform_class, mock_analyzer_class
|
||||
):
|
||||
"""Test initialization with externally provided storage service."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
mock_storage = Mock()
|
||||
service = AudioAnalysisService(storage_service=mock_storage)
|
||||
|
||||
assert service.storage_service is mock_storage
|
||||
|
||||
|
||||
class TestAnalyzeBackingVocals:
|
||||
"""Test analyze_backing_vocals method."""
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_analyze_downloads_and_analyzes(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test analyze_backing_vocals downloads file and runs analysis."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
# Setup mocks
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.has_audible_content = True
|
||||
mock_result.segment_count = 5
|
||||
mock_result.recommended_selection = Mock(value="with_backing")
|
||||
|
||||
mock_analyzer = Mock()
|
||||
mock_analyzer.analyze.return_value = mock_result
|
||||
mock_analyzer_class.return_value = mock_analyzer
|
||||
|
||||
service = AudioAnalysisService()
|
||||
|
||||
# Call method
|
||||
result = service.analyze_backing_vocals(
|
||||
gcs_audio_path="uploads/job123/backing_vocals.flac",
|
||||
job_id="job123",
|
||||
)
|
||||
|
||||
# Verify storage download was called
|
||||
mock_storage.download_file.assert_called_once()
|
||||
call_args = mock_storage.download_file.call_args
|
||||
assert call_args[0][0] == "uploads/job123/backing_vocals.flac"
|
||||
|
||||
# Verify analyzer was called
|
||||
mock_analyzer.analyze.assert_called_once()
|
||||
|
||||
# Verify result
|
||||
assert result is mock_result
|
||||
assert result.has_audible_content is True
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_analyze_cleans_up_temp_files(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test that temp files are cleaned up after analysis."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.has_audible_content = False
|
||||
mock_result.segment_count = 0
|
||||
mock_result.recommended_selection = Mock(value="clean")
|
||||
|
||||
mock_analyzer = Mock()
|
||||
mock_analyzer.analyze.return_value = mock_result
|
||||
mock_analyzer_class.return_value = mock_analyzer
|
||||
|
||||
service = AudioAnalysisService()
|
||||
|
||||
# Call method
|
||||
service.analyze_backing_vocals(
|
||||
gcs_audio_path="uploads/job123/backing.flac",
|
||||
job_id="job123",
|
||||
)
|
||||
|
||||
# Temp directory should be cleaned up (no assertion needed,
|
||||
# TemporaryDirectory context manager handles this)
|
||||
# Just verify the method completes without error
|
||||
|
||||
|
||||
class TestAnalyzeAndGenerateWaveform:
|
||||
"""Test analyze_and_generate_waveform method."""
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_analyze_and_generate_waveform(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test analyze_and_generate_waveform does both operations."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.has_audible_content = True
|
||||
mock_result.audible_segments = [Mock(), Mock()]
|
||||
|
||||
mock_analyzer = Mock()
|
||||
mock_analyzer.analyze.return_value = mock_result
|
||||
mock_analyzer.silence_threshold_db = -40.0
|
||||
mock_analyzer_class.return_value = mock_analyzer
|
||||
|
||||
mock_waveform = Mock()
|
||||
mock_waveform_class.return_value = mock_waveform
|
||||
|
||||
service = AudioAnalysisService()
|
||||
|
||||
# Call method
|
||||
result, waveform_path = service.analyze_and_generate_waveform(
|
||||
gcs_audio_path="uploads/job123/backing.flac",
|
||||
job_id="job123",
|
||||
gcs_waveform_destination="uploads/job123/waveform.png",
|
||||
)
|
||||
|
||||
# Verify analysis was performed
|
||||
mock_analyzer.analyze.assert_called_once()
|
||||
|
||||
# Verify waveform was generated
|
||||
mock_waveform.generate.assert_called_once()
|
||||
call_kwargs = mock_waveform.generate.call_args.kwargs
|
||||
assert call_kwargs["segments"] == mock_result.audible_segments
|
||||
assert call_kwargs["show_time_axis"] is True
|
||||
|
||||
# Verify waveform was uploaded
|
||||
assert mock_storage.upload_file.call_count == 1
|
||||
|
||||
# Verify return values
|
||||
assert result is mock_result
|
||||
assert waveform_path == "uploads/job123/waveform.png"
|
||||
|
||||
|
||||
class TestGetWaveformData:
|
||||
"""Test get_waveform_data method."""
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_get_waveform_data(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test get_waveform_data returns amplitude data."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_waveform = Mock()
|
||||
mock_waveform.generate_data_only.return_value = (
|
||||
[0.1, 0.3, 0.5, 0.3, 0.1], # amplitudes
|
||||
180.5, # duration
|
||||
)
|
||||
mock_waveform_class.return_value = mock_waveform
|
||||
|
||||
service = AudioAnalysisService()
|
||||
|
||||
amplitudes, duration = service.get_waveform_data(
|
||||
gcs_audio_path="uploads/job123/backing.flac",
|
||||
job_id="job123",
|
||||
num_points=500,
|
||||
)
|
||||
|
||||
# Verify storage download
|
||||
mock_storage.download_file.assert_called_once()
|
||||
|
||||
# Verify waveform generator called
|
||||
mock_waveform.generate_data_only.assert_called_once()
|
||||
call_kwargs = mock_waveform.generate_data_only.call_args.kwargs
|
||||
assert call_kwargs["num_points"] == 500
|
||||
|
||||
# Verify return values
|
||||
assert amplitudes == [0.1, 0.3, 0.5, 0.3, 0.1]
|
||||
assert duration == 180.5
|
||||
|
||||
|
||||
class TestGenerateWaveformWithMutes:
|
||||
"""Test generate_waveform_with_mutes method."""
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_generate_waveform_with_mutes(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
"""Test generate_waveform_with_mutes highlights mute regions."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.audible_segments = [Mock(), Mock()]
|
||||
|
||||
mock_analyzer = Mock()
|
||||
mock_analyzer.analyze.return_value = mock_result
|
||||
mock_analyzer_class.return_value = mock_analyzer
|
||||
|
||||
mock_waveform = Mock()
|
||||
mock_waveform_class.return_value = mock_waveform
|
||||
|
||||
# Create mock mute regions
|
||||
mock_mute1 = Mock()
|
||||
mock_mute2 = Mock()
|
||||
mute_regions = [mock_mute1, mock_mute2]
|
||||
|
||||
service = AudioAnalysisService()
|
||||
|
||||
result_path = service.generate_waveform_with_mutes(
|
||||
gcs_audio_path="uploads/job123/backing.flac",
|
||||
job_id="job123",
|
||||
gcs_waveform_destination="uploads/job123/waveform_muted.png",
|
||||
mute_regions=mute_regions,
|
||||
)
|
||||
|
||||
# Verify waveform generator was called with mute_regions
|
||||
mock_waveform.generate.assert_called_once()
|
||||
call_kwargs = mock_waveform.generate.call_args.kwargs
|
||||
assert call_kwargs["mute_regions"] == mute_regions
|
||||
assert call_kwargs["segments"] == mock_result.audible_segments
|
||||
|
||||
# Verify upload
|
||||
mock_storage.upload_file.assert_called_once()
|
||||
|
||||
# Verify return
|
||||
assert result_path == "uploads/job123/waveform_muted.png"
|
||||
|
||||
@@ -0,0 +1,386 @@
|
||||
"""
|
||||
Tests for audio_editing_service.py - Custom instrumental creation.
|
||||
|
||||
These tests mock the storage service and shared karaoke_gen modules to verify:
|
||||
- GCS file download/upload operations
|
||||
- Audio editing delegation to shared AudioEditor
|
||||
- Mute region validation
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, patch, ANY
|
||||
|
||||
|
||||
class TestAudioEditingServiceInit:
|
||||
"""Test AudioEditingService initialization."""
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_init_creates_dependencies(self, mock_editor_class, mock_storage_class):
|
||||
"""Test initialization creates storage service and editor."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
mock_storage_class.assert_called_once()
|
||||
mock_editor_class.assert_called_once_with(output_format="flac")
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_init_with_custom_format(self, mock_editor_class, mock_storage_class):
|
||||
"""Test initialization with custom output format."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
service = AudioEditingService(output_format="wav")
|
||||
|
||||
mock_editor_class.assert_called_once_with(output_format="wav")
|
||||
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_init_with_provided_storage_service(self, mock_editor_class):
|
||||
"""Test initialization with externally provided storage service."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
mock_storage = Mock()
|
||||
service = AudioEditingService(storage_service=mock_storage)
|
||||
|
||||
assert service.storage_service is mock_storage
|
||||
|
||||
|
||||
class TestCreateCustomInstrumental:
|
||||
"""Test create_custom_instrumental method."""
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_create_custom_instrumental_success(
|
||||
self, mock_editor_class, mock_storage_class
|
||||
):
|
||||
"""Test creating a custom instrumental successfully."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.total_muted_duration_seconds = 15.5
|
||||
mock_result.mute_regions_applied = [Mock(), Mock(), Mock()]
|
||||
mock_result.output_path = None # Will be set by service
|
||||
|
||||
mock_editor = Mock()
|
||||
mock_editor.create_custom_instrumental.return_value = mock_result
|
||||
mock_editor_class.return_value = mock_editor
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
# Create mock mute regions
|
||||
mock_mute1 = Mock()
|
||||
mock_mute2 = Mock()
|
||||
mute_regions = [mock_mute1, mock_mute2]
|
||||
|
||||
result = service.create_custom_instrumental(
|
||||
gcs_clean_instrumental_path="uploads/job123/clean.flac",
|
||||
gcs_backing_vocals_path="uploads/job123/backing.flac",
|
||||
mute_regions=mute_regions,
|
||||
gcs_output_path="uploads/job123/custom.flac",
|
||||
job_id="job123",
|
||||
)
|
||||
|
||||
# Verify downloads
|
||||
assert mock_storage.download_file.call_count == 2
|
||||
|
||||
# Verify editor was called
|
||||
mock_editor.create_custom_instrumental.assert_called_once()
|
||||
call_kwargs = mock_editor.create_custom_instrumental.call_args.kwargs
|
||||
assert call_kwargs["mute_regions"] == mute_regions
|
||||
|
||||
# Verify upload
|
||||
mock_storage.upload_file.assert_called_once()
|
||||
|
||||
# Verify result path was updated
|
||||
assert result.output_path == "uploads/job123/custom.flac"
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_create_custom_instrumental_downloads_both_files(
|
||||
self, mock_editor_class, mock_storage_class
|
||||
):
|
||||
"""Test that both input files are downloaded."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.total_muted_duration_seconds = 0
|
||||
mock_result.mute_regions_applied = []
|
||||
|
||||
mock_editor = Mock()
|
||||
mock_editor.create_custom_instrumental.return_value = mock_result
|
||||
mock_editor_class.return_value = mock_editor
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
service.create_custom_instrumental(
|
||||
gcs_clean_instrumental_path="uploads/job123/clean.flac",
|
||||
gcs_backing_vocals_path="uploads/job123/backing.flac",
|
||||
mute_regions=[],
|
||||
gcs_output_path="uploads/job123/output.flac",
|
||||
job_id="job123",
|
||||
)
|
||||
|
||||
# Verify both files were downloaded
|
||||
download_calls = mock_storage.download_file.call_args_list
|
||||
assert len(download_calls) == 2
|
||||
|
||||
# First call should be clean instrumental
|
||||
assert download_calls[0][0][0] == "uploads/job123/clean.flac"
|
||||
# Second call should be backing vocals
|
||||
assert download_calls[1][0][0] == "uploads/job123/backing.flac"
|
||||
|
||||
|
||||
class TestCreatePreview:
|
||||
"""Test create_preview method."""
|
||||
|
||||
@patch("pydub.AudioSegment")
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_create_preview_success(
|
||||
self, mock_editor_class, mock_storage_class, mock_audio_segment
|
||||
):
|
||||
"""Test creating a preview successfully."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_preview = Mock()
|
||||
mock_preview.export = Mock()
|
||||
mock_preview.__getitem__ = Mock(return_value=mock_preview)
|
||||
|
||||
mock_editor = Mock()
|
||||
mock_editor.preview_with_mutes.return_value = mock_preview
|
||||
mock_editor_class.return_value = mock_editor
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
result = service.create_preview(
|
||||
gcs_clean_instrumental_path="uploads/job123/clean.flac",
|
||||
gcs_backing_vocals_path="uploads/job123/backing.flac",
|
||||
mute_regions=[Mock(), Mock()],
|
||||
gcs_preview_path="uploads/job123/preview.flac",
|
||||
job_id="job123",
|
||||
)
|
||||
|
||||
# Verify editor was called
|
||||
mock_editor.preview_with_mutes.assert_called_once()
|
||||
|
||||
# Verify result
|
||||
assert result == "uploads/job123/preview.flac"
|
||||
|
||||
@patch("pydub.AudioSegment")
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_create_preview_with_duration_limit(
|
||||
self, mock_editor_class, mock_storage_class, mock_audio_segment
|
||||
):
|
||||
"""Test creating a preview with duration limit."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_preview = Mock()
|
||||
mock_truncated = Mock()
|
||||
mock_preview.__getitem__ = Mock(return_value=mock_truncated)
|
||||
mock_truncated.export = Mock()
|
||||
|
||||
mock_editor = Mock()
|
||||
mock_editor.preview_with_mutes.return_value = mock_preview
|
||||
mock_editor_class.return_value = mock_editor
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
service.create_preview(
|
||||
gcs_clean_instrumental_path="uploads/job123/clean.flac",
|
||||
gcs_backing_vocals_path="uploads/job123/backing.flac",
|
||||
mute_regions=[],
|
||||
gcs_preview_path="uploads/job123/preview.flac",
|
||||
job_id="job123",
|
||||
preview_duration_seconds=30.0,
|
||||
)
|
||||
|
||||
# Verify preview was truncated (30 seconds = 30000ms)
|
||||
mock_preview.__getitem__.assert_called_once_with(slice(None, 30000))
|
||||
|
||||
|
||||
class TestMuteBackingVocalsOnly:
|
||||
"""Test mute_backing_vocals_only method."""
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_mute_backing_vocals_only(self, mock_editor_class, mock_storage_class):
|
||||
"""Test muting backing vocals without combining."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
mock_editor = Mock()
|
||||
mock_editor_class.return_value = mock_editor
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
mute_regions = [Mock(), Mock()]
|
||||
|
||||
result = service.mute_backing_vocals_only(
|
||||
gcs_backing_vocals_path="uploads/job123/backing.flac",
|
||||
mute_regions=mute_regions,
|
||||
gcs_output_path="uploads/job123/muted_backing.flac",
|
||||
job_id="job123",
|
||||
)
|
||||
|
||||
# Verify only one file was downloaded
|
||||
mock_storage.download_file.assert_called_once()
|
||||
assert mock_storage.download_file.call_args[0][0] == "uploads/job123/backing.flac"
|
||||
|
||||
# Verify editor method was called
|
||||
mock_editor.apply_mute_to_single_track.assert_called_once()
|
||||
call_kwargs = mock_editor.apply_mute_to_single_track.call_args.kwargs
|
||||
assert call_kwargs["mute_regions"] == mute_regions
|
||||
|
||||
# Verify upload
|
||||
mock_storage.upload_file.assert_called_once()
|
||||
|
||||
# Verify result
|
||||
assert result == "uploads/job123/muted_backing.flac"
|
||||
|
||||
|
||||
class TestValidateMuteRegions:
|
||||
"""Test validate_mute_regions method.
|
||||
|
||||
Note: MuteRegion uses pydantic validation that rejects invalid values at
|
||||
creation time. These tests use Mock objects to test the validation logic
|
||||
in the service, simulating edge cases that would normally be rejected.
|
||||
"""
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_valid_mute_regions(self, mock_editor_class, mock_storage_class):
|
||||
"""Test validation passes for valid mute regions."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
from karaoke_gen.instrumental_review import MuteRegion
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
mute_regions = [
|
||||
MuteRegion(start_seconds=10.0, end_seconds=15.0),
|
||||
MuteRegion(start_seconds=30.0, end_seconds=45.0),
|
||||
]
|
||||
|
||||
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=180.0)
|
||||
|
||||
assert errors == []
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_negative_start_seconds(self, mock_editor_class, mock_storage_class):
|
||||
"""Test validation catches negative start time.
|
||||
|
||||
Uses Mock since MuteRegion pydantic model rejects negative values.
|
||||
"""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
# Use mock to simulate invalid region (pydantic would reject this)
|
||||
mock_region = Mock()
|
||||
mock_region.start_seconds = -5.0
|
||||
mock_region.end_seconds = 10.0
|
||||
|
||||
errors = service.validate_mute_regions([mock_region], total_duration_seconds=180.0)
|
||||
|
||||
assert len(errors) == 1
|
||||
assert "cannot be negative" in errors[0]
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_end_before_start(self, mock_editor_class, mock_storage_class):
|
||||
"""Test validation catches end time before start time."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
# Use mock to simulate invalid region
|
||||
mock_region = Mock()
|
||||
mock_region.start_seconds = 30.0
|
||||
mock_region.end_seconds = 20.0
|
||||
|
||||
errors = service.validate_mute_regions([mock_region], total_duration_seconds=180.0)
|
||||
|
||||
assert len(errors) == 1
|
||||
assert "must be after" in errors[0]
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_start_exceeds_duration(self, mock_editor_class, mock_storage_class):
|
||||
"""Test validation catches start time exceeding audio duration."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
from karaoke_gen.instrumental_review import MuteRegion
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
mute_regions = [
|
||||
MuteRegion(start_seconds=200.0, end_seconds=210.0),
|
||||
]
|
||||
|
||||
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=180.0)
|
||||
|
||||
assert len(errors) == 1
|
||||
assert "exceeds audio duration" in errors[0]
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_end_exceeds_duration_is_warning_not_error(
|
||||
self, mock_editor_class, mock_storage_class
|
||||
):
|
||||
"""Test that end time exceeding duration is logged as warning, not error."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
from karaoke_gen.instrumental_review import MuteRegion
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
mute_regions = [
|
||||
MuteRegion(start_seconds=170.0, end_seconds=200.0), # End exceeds 180s
|
||||
]
|
||||
|
||||
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=180.0)
|
||||
|
||||
# Should not be an error (will be clamped)
|
||||
assert len(errors) == 0
|
||||
|
||||
@patch("backend.services.audio_editing_service.StorageService")
|
||||
@patch("backend.services.audio_editing_service.AudioEditor")
|
||||
def test_multiple_errors(self, mock_editor_class, mock_storage_class):
|
||||
"""Test validation reports multiple errors."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
from karaoke_gen.instrumental_review import MuteRegion
|
||||
|
||||
service = AudioEditingService()
|
||||
|
||||
# Use mocks for invalid regions
|
||||
mock_region1 = Mock()
|
||||
mock_region1.start_seconds = -5.0
|
||||
mock_region1.end_seconds = 10.0
|
||||
|
||||
mock_region2 = Mock()
|
||||
mock_region2.start_seconds = 30.0
|
||||
mock_region2.end_seconds = 20.0 # End before start
|
||||
|
||||
mock_region3 = Mock()
|
||||
mock_region3.start_seconds = 200.0 # Exceeds duration
|
||||
mock_region3.end_seconds = 210.0
|
||||
|
||||
mute_regions = [mock_region1, mock_region2, mock_region3]
|
||||
|
||||
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=180.0)
|
||||
|
||||
assert len(errors) == 3
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,378 @@
|
||||
"""
|
||||
Unit tests for audio analysis and editing services.
|
||||
|
||||
These tests verify the backend service wrappers that use the shared
|
||||
karaoke_gen.instrumental_review module with GCS integration.
|
||||
|
||||
NOTE: These tests require ffmpeg to be installed. They will be skipped
|
||||
if ffmpeg is not available (e.g., in CI environments without ffmpeg).
|
||||
"""
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch, ANY
|
||||
|
||||
# Check if ffmpeg is available
|
||||
def _ffmpeg_available():
|
||||
"""Check if ffmpeg is available on the system."""
|
||||
return shutil.which('ffmpeg') is not None
|
||||
|
||||
# Skip all tests in this module if ffmpeg is not available
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not _ffmpeg_available(),
|
||||
reason="ffmpeg not available - required for audio processing tests"
|
||||
)
|
||||
|
||||
from pydub import AudioSegment
|
||||
from pydub.generators import Sine
|
||||
|
||||
from karaoke_gen.instrumental_review import MuteRegion
|
||||
|
||||
|
||||
class TestAudioAnalysisService:
|
||||
"""Tests for AudioAnalysisService."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_storage_service(self):
|
||||
"""Create a mock storage service."""
|
||||
mock = MagicMock()
|
||||
return mock
|
||||
|
||||
@pytest.fixture
|
||||
def temp_audio_file(self, tmp_path):
|
||||
"""Create a temporary audio file for testing."""
|
||||
audio_path = tmp_path / "test_audio.flac"
|
||||
|
||||
# Create a simple sine wave audio
|
||||
tone = Sine(440).to_audio_segment(duration=5000) - 15
|
||||
tone.export(str(audio_path), format="flac")
|
||||
|
||||
return str(audio_path)
|
||||
|
||||
@pytest.fixture
|
||||
def silent_audio_file(self, tmp_path):
|
||||
"""Create a silent audio file for testing."""
|
||||
audio_path = tmp_path / "silent_audio.flac"
|
||||
|
||||
audio = AudioSegment.silent(duration=5000, frame_rate=44100)
|
||||
audio.export(str(audio_path), format="flac")
|
||||
|
||||
return str(audio_path)
|
||||
|
||||
def test_service_initialization(self, mock_storage_service):
|
||||
"""Test service initializes with default parameters."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
service = AudioAnalysisService(storage_service=mock_storage_service)
|
||||
|
||||
assert service.storage_service == mock_storage_service
|
||||
assert service.analyzer.silence_threshold_db == -40.0
|
||||
|
||||
def test_service_custom_threshold(self, mock_storage_service):
|
||||
"""Test service accepts custom silence threshold."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
service = AudioAnalysisService(
|
||||
storage_service=mock_storage_service,
|
||||
silence_threshold_db=-35.0,
|
||||
)
|
||||
|
||||
assert service.analyzer.silence_threshold_db == -35.0
|
||||
|
||||
def test_analyze_backing_vocals_downloads_file(
|
||||
self, mock_storage_service, temp_audio_file
|
||||
):
|
||||
"""Test that analyze_backing_vocals downloads file from GCS."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
# Set up mock to copy the temp file to the download location
|
||||
def mock_download(gcs_path, local_path):
|
||||
import shutil
|
||||
shutil.copy(temp_audio_file, local_path)
|
||||
return local_path
|
||||
|
||||
mock_storage_service.download_file.side_effect = mock_download
|
||||
|
||||
service = AudioAnalysisService(storage_service=mock_storage_service)
|
||||
result = service.analyze_backing_vocals(
|
||||
gcs_audio_path="jobs/test/stems/backing_vocals.flac",
|
||||
job_id="test-job",
|
||||
)
|
||||
|
||||
# Verify download was called
|
||||
mock_storage_service.download_file.assert_called_once()
|
||||
|
||||
# Verify result is valid
|
||||
assert result is not None
|
||||
assert hasattr(result, 'has_audible_content')
|
||||
|
||||
def test_analyze_silent_audio_returns_no_audible(
|
||||
self, mock_storage_service, silent_audio_file
|
||||
):
|
||||
"""Test that silent audio returns has_audible_content=False."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
def mock_download(gcs_path, local_path):
|
||||
import shutil
|
||||
shutil.copy(silent_audio_file, local_path)
|
||||
return local_path
|
||||
|
||||
mock_storage_service.download_file.side_effect = mock_download
|
||||
|
||||
service = AudioAnalysisService(storage_service=mock_storage_service)
|
||||
result = service.analyze_backing_vocals(
|
||||
gcs_audio_path="jobs/test/stems/backing_vocals.flac",
|
||||
job_id="test-job",
|
||||
)
|
||||
|
||||
assert result.has_audible_content is False
|
||||
assert len(result.audible_segments) == 0
|
||||
|
||||
def test_analyze_and_generate_waveform_uploads_image(
|
||||
self, mock_storage_service, temp_audio_file
|
||||
):
|
||||
"""Test that analyze_and_generate_waveform uploads waveform to GCS."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
def mock_download(gcs_path, local_path):
|
||||
import shutil
|
||||
shutil.copy(temp_audio_file, local_path)
|
||||
return local_path
|
||||
|
||||
mock_storage_service.download_file.side_effect = mock_download
|
||||
mock_storage_service.upload_file.return_value = "jobs/test/analysis/waveform.png"
|
||||
|
||||
service = AudioAnalysisService(storage_service=mock_storage_service)
|
||||
result, waveform_path = service.analyze_and_generate_waveform(
|
||||
gcs_audio_path="jobs/test/stems/backing_vocals.flac",
|
||||
job_id="test-job",
|
||||
gcs_waveform_destination="jobs/test/analysis/waveform.png",
|
||||
)
|
||||
|
||||
# Verify upload was called
|
||||
mock_storage_service.upload_file.assert_called_once()
|
||||
|
||||
# Verify waveform path is returned
|
||||
assert waveform_path == "jobs/test/analysis/waveform.png"
|
||||
|
||||
def test_get_waveform_data_returns_amplitudes(
|
||||
self, mock_storage_service, temp_audio_file
|
||||
):
|
||||
"""Test that get_waveform_data returns amplitude data."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
def mock_download(gcs_path, local_path):
|
||||
import shutil
|
||||
shutil.copy(temp_audio_file, local_path)
|
||||
return local_path
|
||||
|
||||
mock_storage_service.download_file.side_effect = mock_download
|
||||
|
||||
service = AudioAnalysisService(storage_service=mock_storage_service)
|
||||
amplitudes, duration = service.get_waveform_data(
|
||||
gcs_audio_path="jobs/test/stems/backing_vocals.flac",
|
||||
job_id="test-job",
|
||||
num_points=100,
|
||||
)
|
||||
|
||||
# Verify we get amplitude data
|
||||
assert isinstance(amplitudes, list)
|
||||
assert len(amplitudes) > 0
|
||||
assert all(isinstance(a, float) for a in amplitudes)
|
||||
|
||||
# Verify duration is reasonable
|
||||
assert duration > 0
|
||||
assert duration <= 10 # Our test file is 5 seconds
|
||||
|
||||
|
||||
class TestAudioEditingService:
|
||||
"""Tests for AudioEditingService."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_storage_service(self):
|
||||
"""Create a mock storage service."""
|
||||
mock = MagicMock()
|
||||
return mock
|
||||
|
||||
@pytest.fixture
|
||||
def temp_audio_files(self, tmp_path):
|
||||
"""Create temporary audio files for testing."""
|
||||
clean_path = tmp_path / "clean_instrumental.flac"
|
||||
backing_path = tmp_path / "backing_vocals.flac"
|
||||
|
||||
# Create clean instrumental (chord)
|
||||
tone1 = Sine(220).to_audio_segment(duration=5000) - 15
|
||||
tone2 = Sine(330).to_audio_segment(duration=5000) - 15
|
||||
clean = tone1.overlay(tone2)
|
||||
clean.export(str(clean_path), format="flac")
|
||||
|
||||
# Create backing vocals
|
||||
backing = Sine(440).to_audio_segment(duration=5000) - 20
|
||||
backing.export(str(backing_path), format="flac")
|
||||
|
||||
return str(clean_path), str(backing_path)
|
||||
|
||||
def test_service_initialization(self, mock_storage_service):
|
||||
"""Test service initializes correctly."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
service = AudioEditingService(storage_service=mock_storage_service)
|
||||
|
||||
assert service.storage_service == mock_storage_service
|
||||
assert service.editor.output_format == "flac"
|
||||
|
||||
def test_create_custom_instrumental_downloads_files(
|
||||
self, mock_storage_service, temp_audio_files
|
||||
):
|
||||
"""Test that create_custom_instrumental downloads input files."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
clean_path, backing_path = temp_audio_files
|
||||
|
||||
# Track which files were downloaded
|
||||
downloaded_files = []
|
||||
|
||||
def mock_download(gcs_path, local_path):
|
||||
downloaded_files.append(gcs_path)
|
||||
# Copy the appropriate test file
|
||||
import shutil
|
||||
if 'clean' in gcs_path:
|
||||
shutil.copy(clean_path, local_path)
|
||||
else:
|
||||
shutil.copy(backing_path, local_path)
|
||||
return local_path
|
||||
|
||||
mock_storage_service.download_file.side_effect = mock_download
|
||||
mock_storage_service.upload_file.return_value = "jobs/test/stems/custom.flac"
|
||||
|
||||
service = AudioEditingService(storage_service=mock_storage_service)
|
||||
|
||||
mute_regions = [MuteRegion(start_seconds=1.0, end_seconds=2.0)]
|
||||
|
||||
result = service.create_custom_instrumental(
|
||||
gcs_clean_instrumental_path="jobs/test/stems/clean.flac",
|
||||
gcs_backing_vocals_path="jobs/test/stems/backing.flac",
|
||||
mute_regions=mute_regions,
|
||||
gcs_output_path="jobs/test/stems/custom.flac",
|
||||
job_id="test-job",
|
||||
)
|
||||
|
||||
# Verify both files were downloaded
|
||||
assert len(downloaded_files) == 2
|
||||
assert mock_storage_service.upload_file.called
|
||||
|
||||
def test_create_custom_instrumental_uploads_result(
|
||||
self, mock_storage_service, temp_audio_files
|
||||
):
|
||||
"""Test that create_custom_instrumental uploads the result."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
clean_path, backing_path = temp_audio_files
|
||||
|
||||
def mock_download(gcs_path, local_path):
|
||||
import shutil
|
||||
if 'clean' in gcs_path:
|
||||
shutil.copy(clean_path, local_path)
|
||||
else:
|
||||
shutil.copy(backing_path, local_path)
|
||||
return local_path
|
||||
|
||||
mock_storage_service.download_file.side_effect = mock_download
|
||||
mock_storage_service.upload_file.return_value = "jobs/test/stems/custom.flac"
|
||||
|
||||
service = AudioEditingService(storage_service=mock_storage_service)
|
||||
|
||||
mute_regions = [MuteRegion(start_seconds=1.0, end_seconds=2.0)]
|
||||
|
||||
result = service.create_custom_instrumental(
|
||||
gcs_clean_instrumental_path="jobs/test/stems/clean.flac",
|
||||
gcs_backing_vocals_path="jobs/test/stems/backing.flac",
|
||||
mute_regions=mute_regions,
|
||||
gcs_output_path="jobs/test/stems/custom.flac",
|
||||
job_id="test-job",
|
||||
)
|
||||
|
||||
# Verify upload was called with correct destination
|
||||
mock_storage_service.upload_file.assert_called_once()
|
||||
call_args = mock_storage_service.upload_file.call_args
|
||||
assert call_args[0][1] == "jobs/test/stems/custom.flac"
|
||||
|
||||
# Verify result has correct path
|
||||
assert result.output_path == "jobs/test/stems/custom.flac"
|
||||
|
||||
def test_create_custom_instrumental_returns_statistics(
|
||||
self, mock_storage_service, temp_audio_files
|
||||
):
|
||||
"""Test that create_custom_instrumental returns correct statistics."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
clean_path, backing_path = temp_audio_files
|
||||
|
||||
def mock_download(gcs_path, local_path):
|
||||
import shutil
|
||||
if 'clean' in gcs_path:
|
||||
shutil.copy(clean_path, local_path)
|
||||
else:
|
||||
shutil.copy(backing_path, local_path)
|
||||
return local_path
|
||||
|
||||
mock_storage_service.download_file.side_effect = mock_download
|
||||
mock_storage_service.upload_file.return_value = "jobs/test/stems/custom.flac"
|
||||
|
||||
service = AudioEditingService(storage_service=mock_storage_service)
|
||||
|
||||
mute_regions = [
|
||||
MuteRegion(start_seconds=1.0, end_seconds=2.0),
|
||||
MuteRegion(start_seconds=3.0, end_seconds=4.0),
|
||||
]
|
||||
|
||||
result = service.create_custom_instrumental(
|
||||
gcs_clean_instrumental_path="jobs/test/stems/clean.flac",
|
||||
gcs_backing_vocals_path="jobs/test/stems/backing.flac",
|
||||
mute_regions=mute_regions,
|
||||
gcs_output_path="jobs/test/stems/custom.flac",
|
||||
job_id="test-job",
|
||||
)
|
||||
|
||||
# Verify statistics
|
||||
assert len(result.mute_regions_applied) == 2
|
||||
assert result.total_muted_duration_seconds == 2.0 # 1s + 1s
|
||||
assert result.output_duration_seconds > 0
|
||||
|
||||
def test_validate_mute_regions_returns_errors_for_invalid(
|
||||
self, mock_storage_service
|
||||
):
|
||||
"""Test that validate_mute_regions catches invalid regions."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
service = AudioEditingService(storage_service=mock_storage_service)
|
||||
|
||||
mute_regions = [
|
||||
MuteRegion(start_seconds=0.0, end_seconds=1.0), # Valid
|
||||
MuteRegion(start_seconds=100.0, end_seconds=110.0), # Beyond duration
|
||||
]
|
||||
|
||||
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=60.0)
|
||||
|
||||
# Should have error for region exceeding duration
|
||||
assert len(errors) == 1
|
||||
assert "exceeds" in errors[0].lower()
|
||||
|
||||
def test_validate_mute_regions_returns_empty_for_valid(
|
||||
self, mock_storage_service
|
||||
):
|
||||
"""Test that validate_mute_regions returns empty for valid regions."""
|
||||
from backend.services.audio_editing_service import AudioEditingService
|
||||
|
||||
service = AudioEditingService(storage_service=mock_storage_service)
|
||||
|
||||
mute_regions = [
|
||||
MuteRegion(start_seconds=0.0, end_seconds=1.0),
|
||||
MuteRegion(start_seconds=10.0, end_seconds=15.0),
|
||||
]
|
||||
|
||||
errors = service.validate_mute_regions(mute_regions, total_duration_seconds=60.0)
|
||||
|
||||
assert len(errors) == 0
|
||||
@@ -0,0 +1,231 @@
|
||||
"""
|
||||
Tests for AuthService and FirestoreService.
|
||||
|
||||
These tests focus on the service initialization and basic operations.
|
||||
"""
|
||||
import pytest
|
||||
from datetime import datetime, UTC
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
|
||||
class TestAuthServiceBasics:
|
||||
"""Basic tests for AuthService."""
|
||||
|
||||
def test_auth_service_can_import(self):
|
||||
"""Test AuthService can be imported."""
|
||||
from backend.services.auth_service import AuthService
|
||||
assert AuthService is not None
|
||||
|
||||
def test_auth_service_initialization(self):
|
||||
"""Test AuthService initializes with mocked dependencies."""
|
||||
mock_firestore = MagicMock()
|
||||
|
||||
with patch('backend.services.auth_service.FirestoreService', return_value=mock_firestore), \
|
||||
patch.dict('os.environ', {'ADMIN_TOKENS': 'test-token'}):
|
||||
from backend.services.auth_service import AuthService
|
||||
service = AuthService()
|
||||
assert service is not None
|
||||
|
||||
def test_auth_service_has_validate_method(self):
|
||||
"""Test AuthService has token validation method."""
|
||||
from backend.services.auth_service import AuthService
|
||||
assert hasattr(AuthService, 'validate_token') or hasattr(AuthService, 'verify_token')
|
||||
|
||||
|
||||
class TestFirestoreServiceBasics:
|
||||
"""Basic tests for FirestoreService."""
|
||||
|
||||
def test_firestore_service_can_import(self):
|
||||
"""Test FirestoreService can be imported."""
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
assert FirestoreService is not None
|
||||
|
||||
def test_firestore_service_initialization(self):
|
||||
"""Test FirestoreService initializes with mocked client."""
|
||||
mock_client = MagicMock()
|
||||
|
||||
with patch('backend.services.firestore_service.firestore.Client', return_value=mock_client):
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
service = FirestoreService()
|
||||
assert service is not None
|
||||
|
||||
def test_firestore_service_has_crud_methods(self):
|
||||
"""Test FirestoreService has CRUD methods."""
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
assert hasattr(FirestoreService, 'create_job')
|
||||
assert hasattr(FirestoreService, 'get_job')
|
||||
assert hasattr(FirestoreService, 'update_job')
|
||||
assert hasattr(FirestoreService, 'delete_job')
|
||||
|
||||
|
||||
class TestStorageServiceBasics:
|
||||
"""Basic tests for StorageService."""
|
||||
|
||||
def test_storage_service_can_import(self):
|
||||
"""Test StorageService can be imported."""
|
||||
from backend.services.storage_service import StorageService
|
||||
assert StorageService is not None
|
||||
|
||||
def test_storage_service_initialization(self):
|
||||
"""Test StorageService initializes with mocked client."""
|
||||
mock_client = MagicMock()
|
||||
mock_bucket = MagicMock()
|
||||
mock_client.bucket.return_value = mock_bucket
|
||||
|
||||
with patch('backend.services.storage_service.storage.Client', return_value=mock_client):
|
||||
from backend.services.storage_service import StorageService
|
||||
service = StorageService()
|
||||
assert service is not None
|
||||
|
||||
def test_storage_service_has_file_methods(self):
|
||||
"""Test StorageService has file operation methods."""
|
||||
from backend.services.storage_service import StorageService
|
||||
assert hasattr(StorageService, 'upload_file')
|
||||
assert hasattr(StorageService, 'download_file')
|
||||
|
||||
|
||||
class TestWorkerServiceBasics:
|
||||
"""Basic tests for WorkerService."""
|
||||
|
||||
def test_worker_service_can_import(self):
|
||||
"""Test WorkerService can be imported."""
|
||||
from backend.services.worker_service import WorkerService
|
||||
assert WorkerService is not None
|
||||
|
||||
def test_worker_service_initialization(self):
|
||||
"""Test WorkerService initializes correctly."""
|
||||
from backend.services.worker_service import WorkerService
|
||||
service = WorkerService()
|
||||
assert service is not None
|
||||
|
||||
def test_worker_service_has_trigger_methods(self):
|
||||
"""Test WorkerService has trigger methods."""
|
||||
from backend.services.worker_service import WorkerService
|
||||
assert hasattr(WorkerService, 'trigger_audio_worker')
|
||||
assert hasattr(WorkerService, 'trigger_lyrics_worker')
|
||||
assert hasattr(WorkerService, 'trigger_screens_worker')
|
||||
assert hasattr(WorkerService, 'trigger_video_worker')
|
||||
|
||||
def test_worker_service_get_base_url(self):
|
||||
"""Test WorkerService constructs base URL."""
|
||||
from backend.services.worker_service import WorkerService
|
||||
service = WorkerService()
|
||||
url = service._get_base_url()
|
||||
assert 'http' in url
|
||||
|
||||
|
||||
class TestSessionTokenValidation:
|
||||
"""Tests for session token validation in AuthService.
|
||||
|
||||
PR #120: Fixed authentication to recognize session tokens from magic link auth.
|
||||
Previously, validate_token() only checked admin tokens and auth_tokens collection.
|
||||
Session tokens (created via magic link) are stored in the sessions collection,
|
||||
so they were incorrectly rejected as "Invalid token".
|
||||
"""
|
||||
|
||||
def test_validate_token_checks_session_on_fallback(self):
|
||||
"""Test that validate_token falls back to checking session tokens."""
|
||||
mock_firestore = MagicMock()
|
||||
mock_firestore.get_token.return_value = None # Token not in auth_tokens
|
||||
|
||||
mock_user = MagicMock()
|
||||
mock_user.email = "test@example.com"
|
||||
mock_user.credits = 5
|
||||
|
||||
mock_user_service = MagicMock()
|
||||
mock_user_service.validate_session.return_value = (True, mock_user, "Valid session")
|
||||
|
||||
with patch('backend.services.auth_service.FirestoreService', return_value=mock_firestore), \
|
||||
patch('backend.services.user_service.get_user_service', return_value=mock_user_service), \
|
||||
patch.dict('os.environ', {'ADMIN_TOKENS': 'admin-token-123'}):
|
||||
# Clear the cached instance to get fresh state
|
||||
import backend.services.auth_service
|
||||
backend.services.auth_service._auth_service = None
|
||||
from backend.services.auth_service import AuthService, UserType
|
||||
|
||||
service = AuthService()
|
||||
is_valid, user_type, remaining, msg = service.validate_token("session-token-xyz")
|
||||
|
||||
assert is_valid is True
|
||||
assert user_type == UserType.STRIPE
|
||||
assert remaining == 5 # User's credits
|
||||
mock_user_service.validate_session.assert_called_once_with("session-token-xyz")
|
||||
|
||||
def test_validate_token_returns_invalid_when_session_invalid(self):
|
||||
"""Test that validate_token returns invalid when session validation fails."""
|
||||
mock_firestore = MagicMock()
|
||||
mock_firestore.get_token.return_value = None # Token not in auth_tokens
|
||||
|
||||
mock_user_service = MagicMock()
|
||||
mock_user_service.validate_session.return_value = (False, None, "Invalid session")
|
||||
|
||||
with patch('backend.services.auth_service.FirestoreService', return_value=mock_firestore), \
|
||||
patch('backend.services.user_service.get_user_service', return_value=mock_user_service), \
|
||||
patch.dict('os.environ', {'ADMIN_TOKENS': 'admin-token-123'}):
|
||||
import backend.services.auth_service
|
||||
backend.services.auth_service._auth_service = None
|
||||
from backend.services.auth_service import AuthService, UserType
|
||||
|
||||
service = AuthService()
|
||||
is_valid, user_type, remaining, msg = service.validate_token("invalid-session")
|
||||
|
||||
assert is_valid is False
|
||||
assert remaining == 0
|
||||
assert "Invalid token" in msg
|
||||
|
||||
def test_validate_token_returns_zero_credits_when_user_has_none(self):
|
||||
"""Test that validate_token handles users with 0 credits correctly."""
|
||||
mock_firestore = MagicMock()
|
||||
mock_firestore.get_token.return_value = None
|
||||
|
||||
mock_user = MagicMock()
|
||||
mock_user.email = "test@example.com"
|
||||
mock_user.credits = 0
|
||||
|
||||
mock_user_service = MagicMock()
|
||||
mock_user_service.validate_session.return_value = (True, mock_user, "Valid session")
|
||||
|
||||
with patch('backend.services.auth_service.FirestoreService', return_value=mock_firestore), \
|
||||
patch('backend.services.user_service.get_user_service', return_value=mock_user_service), \
|
||||
patch.dict('os.environ', {'ADMIN_TOKENS': 'admin-token-123'}):
|
||||
import backend.services.auth_service
|
||||
backend.services.auth_service._auth_service = None
|
||||
from backend.services.auth_service import AuthService, UserType
|
||||
|
||||
service = AuthService()
|
||||
is_valid, user_type, remaining, msg = service.validate_token("session-token")
|
||||
|
||||
# User is authenticated but has no credits
|
||||
assert is_valid is True
|
||||
assert user_type == UserType.STRIPE
|
||||
assert remaining == 0
|
||||
assert "no credits" in msg.lower()
|
||||
|
||||
|
||||
class TestJobManagerBasics:
|
||||
"""Basic tests for JobManager."""
|
||||
|
||||
def test_job_manager_can_import(self):
|
||||
"""Test JobManager can be imported."""
|
||||
from backend.services.job_manager import JobManager
|
||||
assert JobManager is not None
|
||||
|
||||
def test_job_manager_initialization(self):
|
||||
"""Test JobManager initializes with mocked services."""
|
||||
mock_firestore = MagicMock()
|
||||
mock_storage = MagicMock()
|
||||
|
||||
with patch('backend.services.job_manager.FirestoreService', return_value=mock_firestore), \
|
||||
patch('backend.services.job_manager.StorageService', return_value=mock_storage):
|
||||
from backend.services.job_manager import JobManager
|
||||
manager = JobManager()
|
||||
assert manager is not None
|
||||
|
||||
def test_job_manager_has_crud_methods(self):
|
||||
"""Test JobManager has CRUD methods."""
|
||||
from backend.services.job_manager import JobManager
|
||||
assert hasattr(JobManager, 'create_job')
|
||||
assert hasattr(JobManager, 'get_job')
|
||||
assert hasattr(JobManager, 'delete_job')
|
||||
assert hasattr(JobManager, 'list_jobs')
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
"""
|
||||
Extended tests for config.py to improve coverage.
|
||||
"""
|
||||
import pytest
|
||||
import os
|
||||
from unittest.mock import patch
|
||||
|
||||
|
||||
class TestSettingsConfiguration:
|
||||
"""Tests for Settings configuration."""
|
||||
|
||||
def test_settings_class_exists(self):
|
||||
"""Test Settings class exists."""
|
||||
from backend.config import Settings
|
||||
assert Settings is not None
|
||||
|
||||
def test_settings_loads_defaults(self):
|
||||
"""Test Settings loads default values."""
|
||||
from backend.config import Settings
|
||||
settings = Settings()
|
||||
assert settings is not None
|
||||
|
||||
def test_gcs_bucket_name_from_env(self):
|
||||
"""Test GCS bucket name from environment."""
|
||||
with patch.dict(os.environ, {'GCS_BUCKET_NAME': 'test-bucket'}):
|
||||
from backend.config import Settings
|
||||
settings = Settings()
|
||||
assert settings.gcs_bucket_name == 'test-bucket'
|
||||
|
||||
def test_firestore_collection_from_env(self):
|
||||
"""Test Firestore collection from environment."""
|
||||
with patch.dict(os.environ, {'FIRESTORE_COLLECTION': 'test-jobs'}):
|
||||
from backend.config import Settings
|
||||
settings = Settings()
|
||||
assert settings.firestore_collection == 'test-jobs'
|
||||
|
||||
def test_environment_setting(self):
|
||||
"""Test environment setting."""
|
||||
from backend.config import Settings
|
||||
settings = Settings()
|
||||
# Accept 'test' as well since pytest may set ENVIRONMENT=test
|
||||
assert settings.environment in ['development', 'production', 'testing', 'test']
|
||||
|
||||
def test_google_cloud_project(self):
|
||||
"""Test Google Cloud project setting."""
|
||||
with patch.dict(os.environ, {'GOOGLE_CLOUD_PROJECT': 'test-project'}):
|
||||
from backend.config import Settings
|
||||
settings = Settings()
|
||||
# Should either use the env var or have a default
|
||||
|
||||
|
||||
class TestGetSettings:
|
||||
"""Tests for get_settings function."""
|
||||
|
||||
def test_get_settings_returns_settings(self):
|
||||
"""Test get_settings returns Settings instance."""
|
||||
from backend.config import get_settings
|
||||
settings = get_settings()
|
||||
assert settings is not None
|
||||
|
||||
def test_get_settings_returns_same_instance(self):
|
||||
"""Test get_settings returns cached instance."""
|
||||
from backend.config import get_settings
|
||||
settings1 = get_settings()
|
||||
settings2 = get_settings()
|
||||
# Should be same cached instance
|
||||
assert settings1 is settings2
|
||||
|
||||
@@ -0,0 +1,377 @@
|
||||
"""
|
||||
Tests for credential manager and auth endpoints.
|
||||
"""
|
||||
import pytest
|
||||
import json
|
||||
from unittest.mock import MagicMock, patch
|
||||
from datetime import datetime
|
||||
|
||||
from backend.services.credential_manager import (
|
||||
CredentialManager,
|
||||
CredentialStatus,
|
||||
CredentialCheckResult,
|
||||
DeviceAuthInfo,
|
||||
get_credential_manager,
|
||||
)
|
||||
|
||||
|
||||
class TestCredentialStatus:
|
||||
"""Tests for CredentialStatus enum."""
|
||||
|
||||
def test_status_values(self):
|
||||
"""Test all status values exist."""
|
||||
assert CredentialStatus.VALID == "valid"
|
||||
assert CredentialStatus.EXPIRED == "expired"
|
||||
assert CredentialStatus.INVALID == "invalid"
|
||||
assert CredentialStatus.NOT_CONFIGURED == "not_configured"
|
||||
assert CredentialStatus.ERROR == "error"
|
||||
|
||||
|
||||
class TestCredentialCheckResult:
|
||||
"""Tests for CredentialCheckResult dataclass."""
|
||||
|
||||
def test_create_result(self):
|
||||
"""Test creating a check result."""
|
||||
result = CredentialCheckResult(
|
||||
service="youtube",
|
||||
status=CredentialStatus.VALID,
|
||||
message="Credentials are valid",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
assert result.service == "youtube"
|
||||
assert result.status == CredentialStatus.VALID
|
||||
assert result.message == "Credentials are valid"
|
||||
assert result.expires_at is None
|
||||
|
||||
def test_create_result_with_expiry(self):
|
||||
"""Test creating a check result with expiry."""
|
||||
expiry = datetime.utcnow()
|
||||
result = CredentialCheckResult(
|
||||
service="gdrive",
|
||||
status=CredentialStatus.VALID,
|
||||
message="Valid",
|
||||
last_checked=datetime.utcnow(),
|
||||
expires_at=expiry
|
||||
)
|
||||
|
||||
assert result.expires_at == expiry
|
||||
|
||||
|
||||
class TestCredentialManager:
|
||||
"""Tests for CredentialManager class."""
|
||||
|
||||
def test_init(self):
|
||||
"""Test manager initialization."""
|
||||
manager = CredentialManager()
|
||||
assert manager._pending_device_auths == {}
|
||||
|
||||
def test_check_youtube_not_configured(self):
|
||||
"""Test YouTube check when not configured."""
|
||||
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
manager = CredentialManager()
|
||||
result = manager.check_youtube_credentials()
|
||||
|
||||
assert result.status == CredentialStatus.NOT_CONFIGURED
|
||||
assert "not configured" in result.message.lower()
|
||||
|
||||
def test_check_youtube_invalid_json(self):
|
||||
"""Test YouTube check with invalid JSON."""
|
||||
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.get_secret.return_value = "not valid json"
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
manager = CredentialManager()
|
||||
result = manager.check_youtube_credentials()
|
||||
|
||||
assert result.status == CredentialStatus.INVALID
|
||||
assert "json" in result.message.lower()
|
||||
|
||||
def test_check_youtube_missing_fields(self):
|
||||
"""Test YouTube check with missing required fields."""
|
||||
# Missing refresh_token and client_secret
|
||||
creds = json.dumps({"token": "test", "client_id": "test"})
|
||||
|
||||
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.get_secret.return_value = creds
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
manager = CredentialManager()
|
||||
result = manager.check_youtube_credentials()
|
||||
|
||||
assert result.status == CredentialStatus.INVALID
|
||||
assert "missing" in result.message.lower()
|
||||
|
||||
def test_check_gdrive_not_configured(self):
|
||||
"""Test Google Drive check when not configured."""
|
||||
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
manager = CredentialManager()
|
||||
result = manager.check_gdrive_credentials()
|
||||
|
||||
assert result.status == CredentialStatus.NOT_CONFIGURED
|
||||
|
||||
def test_check_dropbox_not_configured(self):
|
||||
"""Test Dropbox check when not configured."""
|
||||
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
manager = CredentialManager()
|
||||
result = manager.check_dropbox_credentials()
|
||||
|
||||
assert result.status == CredentialStatus.NOT_CONFIGURED
|
||||
|
||||
def test_check_dropbox_missing_access_token(self):
|
||||
"""Test Dropbox check with missing access token."""
|
||||
creds = json.dumps({"refresh_token": "test"}) # No access_token
|
||||
|
||||
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.get_secret.return_value = creds
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
manager = CredentialManager()
|
||||
result = manager.check_dropbox_credentials()
|
||||
|
||||
assert result.status == CredentialStatus.INVALID
|
||||
assert "access_token" in result.message.lower()
|
||||
|
||||
def test_check_all_credentials(self):
|
||||
"""Test checking all credentials."""
|
||||
with patch('backend.services.credential_manager.get_settings') as mock_get_settings:
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
mock_get_settings.return_value = mock_settings
|
||||
|
||||
manager = CredentialManager()
|
||||
results = manager.check_all_credentials()
|
||||
|
||||
assert "youtube" in results
|
||||
assert "gdrive" in results
|
||||
assert "dropbox" in results
|
||||
assert all(r.status == CredentialStatus.NOT_CONFIGURED for r in results.values())
|
||||
|
||||
def test_test_youtube_api_failure(self):
|
||||
"""Test YouTube API test failure."""
|
||||
manager = CredentialManager()
|
||||
|
||||
mock_creds = MagicMock()
|
||||
|
||||
with patch('googleapiclient.discovery.build', side_effect=Exception("API Error")):
|
||||
result = manager._test_youtube_api(mock_creds)
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_test_gdrive_api_failure(self):
|
||||
"""Test Google Drive API test failure."""
|
||||
manager = CredentialManager()
|
||||
|
||||
mock_creds = MagicMock()
|
||||
|
||||
with patch('googleapiclient.discovery.build', side_effect=Exception("API Error")):
|
||||
result = manager._test_gdrive_api(mock_creds)
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_test_dropbox_api_failure(self):
|
||||
"""Test Dropbox API test failure."""
|
||||
manager = CredentialManager()
|
||||
|
||||
with patch('dropbox.Dropbox', side_effect=Exception("API Error")):
|
||||
result = manager._test_dropbox_api({"access_token": "test"})
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_send_credential_alert_no_webhook(self):
|
||||
"""Test alert without webhook URL."""
|
||||
manager = CredentialManager()
|
||||
|
||||
result = manager.send_credential_alert([], discord_webhook_url=None)
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_send_credential_alert_success(self):
|
||||
"""Test successful alert sending."""
|
||||
manager = CredentialManager()
|
||||
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 200
|
||||
|
||||
invalid_services = [
|
||||
CredentialCheckResult(
|
||||
service="youtube",
|
||||
status=CredentialStatus.INVALID,
|
||||
message="Test message",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
]
|
||||
|
||||
with patch('requests.post', return_value=mock_response) as mock_post:
|
||||
result = manager.send_credential_alert(
|
||||
invalid_services,
|
||||
discord_webhook_url="https://discord.com/webhook/test"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
mock_post.assert_called_once()
|
||||
|
||||
def test_start_youtube_device_auth(self):
|
||||
"""Test starting YouTube device auth flow."""
|
||||
manager = CredentialManager()
|
||||
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 200
|
||||
mock_response.json.return_value = {
|
||||
"device_code": "test_device_code",
|
||||
"user_code": "TEST-CODE",
|
||||
"verification_uri": "https://google.com/device",
|
||||
"expires_in": 1800,
|
||||
"interval": 5
|
||||
}
|
||||
|
||||
with patch('requests.post', return_value=mock_response):
|
||||
device_info = manager.start_youtube_device_auth(
|
||||
client_id="test_client",
|
||||
client_secret="test_secret"
|
||||
)
|
||||
|
||||
assert device_info.device_code == "test_device_code"
|
||||
assert device_info.user_code == "TEST-CODE"
|
||||
assert device_info.verification_url == "https://google.com/device"
|
||||
|
||||
# Should be stored for polling
|
||||
key = f"youtube:{device_info.device_code}"
|
||||
assert key in manager._pending_device_auths
|
||||
|
||||
def test_poll_device_auth_no_client_creds(self):
|
||||
"""Test polling when client credentials are not in Secret Manager."""
|
||||
manager = CredentialManager()
|
||||
|
||||
# Mock get_youtube_client_credentials to return None (no creds)
|
||||
with patch.object(manager, 'get_youtube_client_credentials', return_value=None):
|
||||
status, data = manager.poll_device_auth("youtube", "some_code")
|
||||
|
||||
assert status == "error"
|
||||
assert "client credentials not found" in data["message"].lower()
|
||||
|
||||
def test_poll_device_auth_expired(self):
|
||||
"""Test polling with expired device code."""
|
||||
manager = CredentialManager()
|
||||
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 400
|
||||
mock_response.json.return_value = {"error": "expired_token"}
|
||||
|
||||
with patch.object(manager, 'get_youtube_client_credentials',
|
||||
return_value={"client_id": "test", "client_secret": "test"}):
|
||||
with patch('requests.post', return_value=mock_response):
|
||||
status, data = manager.poll_device_auth("youtube", "expired_code")
|
||||
|
||||
assert status == "expired"
|
||||
assert "expired" in data["message"].lower()
|
||||
|
||||
def test_poll_device_auth_pending(self):
|
||||
"""Test polling when authorization is pending."""
|
||||
manager = CredentialManager()
|
||||
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 400
|
||||
mock_response.json.return_value = {"error": "authorization_pending"}
|
||||
|
||||
with patch.object(manager, 'get_youtube_client_credentials',
|
||||
return_value={"client_id": "test", "client_secret": "test"}):
|
||||
with patch('requests.post', return_value=mock_response):
|
||||
status, data = manager.poll_device_auth("youtube", "pending_code")
|
||||
|
||||
assert status == "pending"
|
||||
assert "waiting" in data["message"].lower()
|
||||
|
||||
def test_poll_device_auth_success(self):
|
||||
"""Test polling when authorization completes successfully."""
|
||||
manager = CredentialManager()
|
||||
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 200
|
||||
mock_response.json.return_value = {
|
||||
"access_token": "new_access_token",
|
||||
"refresh_token": "new_refresh_token"
|
||||
}
|
||||
|
||||
with patch.object(manager, 'get_youtube_client_credentials',
|
||||
return_value={"client_id": "test_id", "client_secret": "test_secret"}):
|
||||
with patch('requests.post', return_value=mock_response):
|
||||
with patch.object(manager, '_save_credentials_to_secret', return_value=True):
|
||||
status, data = manager.poll_device_auth("youtube", "completed_code")
|
||||
|
||||
assert status == "complete"
|
||||
assert data["token"] == "new_access_token"
|
||||
assert data["refresh_token"] == "new_refresh_token"
|
||||
assert data["client_id"] == "test_id"
|
||||
|
||||
|
||||
class TestGetCredentialManager:
|
||||
"""Tests for singleton factory function."""
|
||||
|
||||
def test_returns_singleton(self):
|
||||
"""Test that factory returns same instance."""
|
||||
# Reset singleton
|
||||
import backend.services.credential_manager as module
|
||||
module._credential_manager = None
|
||||
|
||||
manager1 = get_credential_manager()
|
||||
manager2 = get_credential_manager()
|
||||
|
||||
assert manager1 is manager2
|
||||
|
||||
|
||||
class TestAuthRoutes:
|
||||
"""Tests for auth API routes."""
|
||||
|
||||
def test_routes_import(self):
|
||||
"""Test that routes can be imported."""
|
||||
from backend.api.routes.auth import router
|
||||
assert router is not None
|
||||
|
||||
def test_status_endpoint_exists(self):
|
||||
"""Test that status endpoint is defined."""
|
||||
from backend.api.routes.auth import get_credentials_status
|
||||
assert get_credentials_status is not None
|
||||
|
||||
def test_validate_endpoint_exists(self):
|
||||
"""Test that validate endpoint is defined."""
|
||||
from backend.api.routes.auth import validate_credentials
|
||||
assert validate_credentials is not None
|
||||
|
||||
def test_device_auth_endpoint_exists(self):
|
||||
"""Test that device auth endpoints are defined."""
|
||||
from backend.api.routes.auth import (
|
||||
start_youtube_device_auth,
|
||||
poll_youtube_device_auth,
|
||||
start_gdrive_device_auth,
|
||||
poll_gdrive_device_auth,
|
||||
)
|
||||
assert start_youtube_device_auth is not None
|
||||
assert poll_youtube_device_auth is not None
|
||||
assert start_gdrive_device_auth is not None
|
||||
assert poll_gdrive_device_auth is not None
|
||||
|
||||
|
||||
class TestFileUploadCredentialValidation:
|
||||
"""Tests for credential validation in file upload."""
|
||||
|
||||
def test_credential_manager_imported(self):
|
||||
"""Test that credential manager is imported in file upload."""
|
||||
from backend.api.routes.file_upload import get_credential_manager, CredentialStatus
|
||||
assert get_credential_manager is not None
|
||||
assert CredentialStatus is not None
|
||||
@@ -0,0 +1,54 @@
|
||||
"""
|
||||
Tests for API dependencies (authentication, authorization).
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
|
||||
class TestAuthDependencies:
|
||||
"""Tests for authentication dependencies module structure."""
|
||||
|
||||
def test_get_token_from_request_function_exists(self):
|
||||
"""Test get_token_from_request function exists."""
|
||||
from backend.api.dependencies import get_token_from_request
|
||||
assert get_token_from_request is not None
|
||||
|
||||
def test_require_auth_function_exists(self):
|
||||
"""Test require_auth function exists."""
|
||||
from backend.api.dependencies import require_auth
|
||||
assert require_auth is not None
|
||||
|
||||
def test_require_admin_function_exists(self):
|
||||
"""Test require_admin function exists."""
|
||||
from backend.api.dependencies import require_admin
|
||||
assert require_admin is not None
|
||||
|
||||
def test_optional_auth_function_exists(self):
|
||||
"""Test optional_auth function exists."""
|
||||
from backend.api.dependencies import optional_auth
|
||||
assert optional_auth is not None
|
||||
|
||||
def test_security_scheme_defined(self):
|
||||
"""Test HTTP Bearer security scheme is defined."""
|
||||
from backend.api.dependencies import security
|
||||
assert security is not None
|
||||
|
||||
def test_logger_configured(self):
|
||||
"""Test logger is configured."""
|
||||
from backend.api.dependencies import logger
|
||||
assert logger is not None
|
||||
|
||||
|
||||
class TestAuthServiceAccess:
|
||||
"""Tests for auth service access."""
|
||||
|
||||
def test_get_auth_service_function_exists(self):
|
||||
"""Test get_auth_service function exists."""
|
||||
from backend.services.auth_service import get_auth_service
|
||||
assert get_auth_service is not None
|
||||
|
||||
def test_user_type_enum_defined(self):
|
||||
"""Test UserType enum is defined."""
|
||||
from backend.services.auth_service import UserType
|
||||
assert UserType is not None
|
||||
|
||||
@@ -0,0 +1,244 @@
|
||||
"""
|
||||
Tests for DiscordNotificationService.
|
||||
|
||||
Tests cover:
|
||||
- Service initialization
|
||||
- Webhook URL validation
|
||||
- Message posting
|
||||
- Video notification posting
|
||||
- Dry run mode
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from backend.services.discord_service import (
|
||||
DiscordNotificationService,
|
||||
get_discord_notification_service,
|
||||
)
|
||||
|
||||
|
||||
class TestDiscordNotificationServiceInit:
|
||||
"""Test service initialization."""
|
||||
|
||||
def test_init_with_webhook_url(self):
|
||||
"""Test initialization with valid webhook URL."""
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef123456"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
assert service.webhook_url == url
|
||||
assert service.is_enabled() is True
|
||||
|
||||
def test_init_without_webhook_url(self):
|
||||
"""Test initialization without webhook URL."""
|
||||
service = DiscordNotificationService()
|
||||
assert service.webhook_url is None
|
||||
assert service.is_enabled() is False
|
||||
|
||||
def test_init_with_dry_run(self):
|
||||
"""Test initialization with dry run mode."""
|
||||
service = DiscordNotificationService(dry_run=True)
|
||||
assert service.dry_run is True
|
||||
|
||||
def test_init_with_invalid_webhook_url_raises(self):
|
||||
"""Test that invalid webhook URL raises ValueError."""
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
DiscordNotificationService(webhook_url="https://example.com/webhook")
|
||||
assert "Invalid Discord webhook URL" in str(exc_info.value)
|
||||
|
||||
def test_init_with_non_discord_url_raises(self):
|
||||
"""Test that non-Discord URL raises ValueError."""
|
||||
with pytest.raises(ValueError):
|
||||
DiscordNotificationService(
|
||||
webhook_url="https://slack.com/api/webhooks/123"
|
||||
)
|
||||
|
||||
|
||||
class TestDiscordNotificationServiceValidation:
|
||||
"""Test webhook URL validation."""
|
||||
|
||||
def test_validate_valid_webhook_url(self):
|
||||
"""Test that valid webhook URL passes validation."""
|
||||
service = DiscordNotificationService()
|
||||
# Should not raise
|
||||
service._validate_webhook_url(
|
||||
"https://discord.com/api/webhooks/123456789/abcdef"
|
||||
)
|
||||
|
||||
def test_validate_invalid_webhook_url(self):
|
||||
"""Test that invalid webhook URL raises ValueError."""
|
||||
service = DiscordNotificationService()
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
service._validate_webhook_url("https://example.com/webhook")
|
||||
assert "Invalid Discord webhook URL" in str(exc_info.value)
|
||||
|
||||
def test_validate_strips_whitespace(self):
|
||||
"""Test that whitespace is stripped from URL."""
|
||||
service = DiscordNotificationService()
|
||||
# Should not raise - whitespace is stripped
|
||||
service._validate_webhook_url(
|
||||
" https://discord.com/api/webhooks/123456789/abcdef "
|
||||
)
|
||||
|
||||
|
||||
class TestDiscordNotificationServicePostMessage:
|
||||
"""Test message posting."""
|
||||
|
||||
@patch("backend.services.discord_service.requests.post")
|
||||
def test_post_message_success(self, mock_post):
|
||||
"""Test successful message posting."""
|
||||
mock_response = MagicMock()
|
||||
mock_response.raise_for_status = MagicMock()
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
|
||||
result = service.post_message("Test message")
|
||||
|
||||
assert result is True
|
||||
mock_post.assert_called_once_with(
|
||||
url,
|
||||
json={"content": "Test message"},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
@patch("backend.services.discord_service.requests.post")
|
||||
def test_post_message_with_custom_webhook(self, mock_post):
|
||||
"""Test message posting with custom webhook URL."""
|
||||
mock_response = MagicMock()
|
||||
mock_response.raise_for_status = MagicMock()
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
service = DiscordNotificationService()
|
||||
custom_url = "https://discord.com/api/webhooks/987654321/fedcba"
|
||||
|
||||
result = service.post_message("Test message", webhook_url=custom_url)
|
||||
|
||||
assert result is True
|
||||
mock_post.assert_called_once_with(
|
||||
custom_url,
|
||||
json={"content": "Test message"},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
def test_post_message_no_webhook_raises(self):
|
||||
"""Test that posting without webhook URL raises ValueError."""
|
||||
service = DiscordNotificationService()
|
||||
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
service.post_message("Test message")
|
||||
assert "No Discord webhook URL provided" in str(exc_info.value)
|
||||
|
||||
def test_post_message_dry_run(self):
|
||||
"""Test message posting in dry run mode."""
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url, dry_run=True)
|
||||
|
||||
result = service.post_message("Test message")
|
||||
|
||||
assert result is True
|
||||
# In dry run mode, no actual request should be made
|
||||
|
||||
@patch("backend.services.discord_service.requests.post")
|
||||
def test_post_message_http_error(self, mock_post):
|
||||
"""Test handling of HTTP errors."""
|
||||
import requests as req
|
||||
mock_response = MagicMock()
|
||||
mock_response.raise_for_status.side_effect = req.HTTPError("404 Not Found")
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
|
||||
with pytest.raises(req.HTTPError):
|
||||
service.post_message("Test message")
|
||||
|
||||
|
||||
class TestDiscordNotificationServiceVideoNotification:
|
||||
"""Test video notification posting."""
|
||||
|
||||
@patch("backend.services.discord_service.requests.post")
|
||||
def test_post_video_notification_success(self, mock_post):
|
||||
"""Test successful video notification."""
|
||||
mock_response = MagicMock()
|
||||
mock_response.raise_for_status = MagicMock()
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
|
||||
result = service.post_video_notification(
|
||||
"https://www.youtube.com/watch?v=abc123"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
mock_post.assert_called_once()
|
||||
call_args = mock_post.call_args
|
||||
assert "New upload:" in call_args[1]["json"]["content"]
|
||||
assert "abc123" in call_args[1]["json"]["content"]
|
||||
|
||||
def test_post_video_notification_no_youtube_url(self):
|
||||
"""Test video notification with empty YouTube URL."""
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
|
||||
result = service.post_video_notification("")
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_post_video_notification_none_youtube_url(self):
|
||||
"""Test video notification with None YouTube URL."""
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
|
||||
result = service.post_video_notification(None)
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_post_video_notification_dry_run(self):
|
||||
"""Test video notification in dry run mode."""
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url, dry_run=True)
|
||||
|
||||
result = service.post_video_notification(
|
||||
"https://www.youtube.com/watch?v=abc123"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
|
||||
|
||||
class TestGetDiscordNotificationService:
|
||||
"""Test factory function."""
|
||||
|
||||
def test_get_service_creates_instance(self):
|
||||
"""Test that factory function creates a new instance."""
|
||||
import backend.services.discord_service as module
|
||||
module._discord_notification_service = None
|
||||
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = get_discord_notification_service(webhook_url=url)
|
||||
|
||||
assert service is not None
|
||||
assert isinstance(service, DiscordNotificationService)
|
||||
assert service.webhook_url == url
|
||||
|
||||
def test_get_service_without_webhook(self):
|
||||
"""Test factory function without webhook URL."""
|
||||
import backend.services.discord_service as module
|
||||
module._discord_notification_service = None
|
||||
|
||||
service = get_discord_notification_service()
|
||||
|
||||
assert service is not None
|
||||
assert service.webhook_url is None
|
||||
assert service.is_enabled() is False
|
||||
|
||||
def test_get_service_with_dry_run(self):
|
||||
"""Test factory function with dry run mode."""
|
||||
import backend.services.discord_service as module
|
||||
module._discord_notification_service = None
|
||||
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = get_discord_notification_service(webhook_url=url, dry_run=True)
|
||||
|
||||
assert service.dry_run is True
|
||||
@@ -0,0 +1,820 @@
|
||||
"""
|
||||
Tests for native distribution services (Dropbox and Google Drive).
|
||||
|
||||
These tests verify the service interfaces without requiring actual
|
||||
cloud credentials, using mocks to simulate API responses.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch, mock_open
|
||||
|
||||
|
||||
class TestDropboxService:
|
||||
"""Tests for the DropboxService class."""
|
||||
|
||||
def test_module_imports(self):
|
||||
"""Test that the module imports correctly."""
|
||||
from backend.services.dropbox_service import DropboxService, get_dropbox_service
|
||||
assert DropboxService is not None
|
||||
assert get_dropbox_service is not None
|
||||
|
||||
def test_init_creates_instance(self):
|
||||
"""Test that we can create a DropboxService instance."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
service = DropboxService()
|
||||
assert service is not None
|
||||
assert service._client is None # Client not initialized until first use
|
||||
assert service._is_configured is False
|
||||
|
||||
def test_is_configured_returns_false_without_credentials(self):
|
||||
"""Test that is_configured returns False when credentials are missing."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
with patch.object(DropboxService, '_load_credentials', return_value=None):
|
||||
service = DropboxService()
|
||||
assert service.is_configured is False
|
||||
|
||||
def test_is_configured_returns_true_with_credentials(self):
|
||||
"""Test that is_configured returns True when credentials are present."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_creds = {"access_token": "test_token"}
|
||||
with patch.object(DropboxService, '_load_credentials', return_value=mock_creds):
|
||||
service = DropboxService()
|
||||
assert service.is_configured is True
|
||||
|
||||
def test_is_configured_caches_result(self):
|
||||
"""Test that is_configured caches the result."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
service = DropboxService()
|
||||
service._is_configured = True
|
||||
# Should return True without calling _load_credentials
|
||||
assert service.is_configured is True
|
||||
|
||||
def test_is_configured_returns_false_without_access_token(self):
|
||||
"""Test that is_configured returns False when access_token is missing."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_creds = {"refresh_token": "test"} # No access_token
|
||||
with patch.object(DropboxService, '_load_credentials', return_value=mock_creds):
|
||||
service = DropboxService()
|
||||
assert service.is_configured is False
|
||||
|
||||
def test_get_next_brand_code_first_track(self):
|
||||
"""Test brand code calculation when no existing tracks."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
service = DropboxService()
|
||||
with patch.object(service, 'list_folders', return_value=[]):
|
||||
brand_code = service.get_next_brand_code("/test/path", "NOMAD")
|
||||
assert brand_code == "NOMAD-0001"
|
||||
|
||||
def test_get_next_brand_code_sequential(self):
|
||||
"""Test brand code calculation with existing tracks."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
service = DropboxService()
|
||||
existing_folders = [
|
||||
"NOMAD-0001 - Artist1 - Song1",
|
||||
"NOMAD-0002 - Artist2 - Song2",
|
||||
"NOMAD-0005 - Artist3 - Song3", # Gap in sequence
|
||||
"OTHER-0001 - Different Brand", # Different brand
|
||||
]
|
||||
with patch.object(service, 'list_folders', return_value=existing_folders):
|
||||
brand_code = service.get_next_brand_code("/test/path", "NOMAD")
|
||||
assert brand_code == "NOMAD-0006"
|
||||
|
||||
def test_get_next_brand_code_different_prefix(self):
|
||||
"""Test brand code calculation with different brand prefix."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
service = DropboxService()
|
||||
existing_folders = [
|
||||
"NOMAD-0001 - Artist1 - Song1",
|
||||
"TEST-0010 - Artist2 - Song2",
|
||||
]
|
||||
with patch.object(service, 'list_folders', return_value=existing_folders):
|
||||
brand_code = service.get_next_brand_code("/test/path", "TEST")
|
||||
assert brand_code == "TEST-0011"
|
||||
|
||||
def test_factory_function_returns_instance(self):
|
||||
"""Test that get_dropbox_service returns a service instance."""
|
||||
from backend.services.dropbox_service import get_dropbox_service
|
||||
|
||||
service = get_dropbox_service()
|
||||
assert service is not None
|
||||
|
||||
def test_load_credentials_handles_exception(self):
|
||||
"""Test that _load_credentials handles exceptions gracefully."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
with patch('backend.services.dropbox_service.secretmanager.SecretManagerServiceClient') as mock_client:
|
||||
mock_client.return_value.access_secret_version.side_effect = Exception("API Error")
|
||||
service = DropboxService()
|
||||
result = service._load_credentials()
|
||||
assert result is None
|
||||
|
||||
def test_load_credentials_parses_json(self):
|
||||
"""Test that _load_credentials correctly parses JSON credentials."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_creds = {"access_token": "test_token", "refresh_token": "test_refresh"}
|
||||
mock_response = MagicMock()
|
||||
mock_response.payload.data.decode.return_value = json.dumps(mock_creds)
|
||||
|
||||
with patch('backend.services.dropbox_service.secretmanager.SecretManagerServiceClient') as mock_client:
|
||||
mock_client.return_value.access_secret_version.return_value = mock_response
|
||||
service = DropboxService()
|
||||
result = service._load_credentials()
|
||||
assert result == mock_creds
|
||||
|
||||
def test_client_property_raises_without_credentials(self):
|
||||
"""Test that client property raises error without credentials."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
with patch.object(DropboxService, '_load_credentials', return_value=None):
|
||||
service = DropboxService()
|
||||
with pytest.raises(RuntimeError, match="credentials not configured"):
|
||||
_ = service.client
|
||||
|
||||
def test_client_property_raises_import_error(self):
|
||||
"""Test that client property raises helpful error if dropbox not installed."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
import sys
|
||||
|
||||
mock_creds = {"access_token": "test_token"}
|
||||
with patch.object(DropboxService, '_load_credentials', return_value=mock_creds):
|
||||
with patch.dict(sys.modules, {'dropbox': None}):
|
||||
service = DropboxService()
|
||||
# Force reimport to fail
|
||||
with patch('builtins.__import__', side_effect=ImportError("No module named 'dropbox'")):
|
||||
with pytest.raises(ImportError, match="dropbox package"):
|
||||
_ = service.client
|
||||
|
||||
def test_list_folders_adds_leading_slash(self):
|
||||
"""Test that list_folders adds leading slash if missing."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
service = DropboxService()
|
||||
mock_client = MagicMock()
|
||||
mock_result = MagicMock()
|
||||
mock_result.entries = []
|
||||
mock_result.has_more = False
|
||||
mock_client.files_list_folder.return_value = mock_result
|
||||
service._client = mock_client
|
||||
|
||||
service.list_folders("test/path") # No leading slash
|
||||
|
||||
# Verify it was called with leading slash
|
||||
mock_client.files_list_folder.assert_called_with("/test/path")
|
||||
|
||||
def test_upload_file_adds_leading_slash(self):
|
||||
"""Test that upload_file adds leading slash to remote path."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
import tempfile
|
||||
|
||||
service = DropboxService()
|
||||
mock_client = MagicMock()
|
||||
service._client = mock_client
|
||||
|
||||
# Create a small temp file
|
||||
with tempfile.NamedTemporaryFile(delete=False, mode='wb') as f:
|
||||
f.write(b"test content")
|
||||
temp_path = f.name
|
||||
|
||||
try:
|
||||
service.upload_file(temp_path, "remote/path.txt") # No leading slash
|
||||
# Verify upload was called with leading slash
|
||||
mock_client.files_upload.assert_called_once()
|
||||
call_args = mock_client.files_upload.call_args
|
||||
assert call_args[0][1] == "/remote/path.txt"
|
||||
finally:
|
||||
os.unlink(temp_path)
|
||||
|
||||
def test_upload_folder_uploads_all_files(self):
|
||||
"""Test that upload_folder uploads all files in directory."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
import tempfile
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
# Create some test files
|
||||
(open(os.path.join(tmpdir, "file1.txt"), "w")).write("content1")
|
||||
(open(os.path.join(tmpdir, "file2.txt"), "w")).write("content2")
|
||||
|
||||
with patch.object(service, 'upload_file') as mock_upload:
|
||||
service.upload_folder(tmpdir, "/remote/folder")
|
||||
|
||||
assert mock_upload.call_count == 2
|
||||
|
||||
def test_create_shared_link_success(self):
|
||||
"""Test successful shared link creation."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
service = DropboxService()
|
||||
mock_client = MagicMock()
|
||||
mock_link = MagicMock()
|
||||
mock_link.url = "https://dropbox.com/shared/test"
|
||||
mock_client.sharing_create_shared_link_with_settings.return_value = mock_link
|
||||
service._client = mock_client
|
||||
|
||||
result = service.create_shared_link("/test/path")
|
||||
assert result == "https://dropbox.com/shared/test"
|
||||
|
||||
def test_sharing_list_shared_links_mock_setup(self):
|
||||
"""Test that mock setup for sharing_list_shared_links works correctly.
|
||||
|
||||
Note: Properly mocking Dropbox's ApiError is complex because it
|
||||
requires specific exception class structure. This test verifies
|
||||
the mock configuration is correct for the success path, which is
|
||||
a prerequisite for more complex error-handling tests.
|
||||
"""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
service = DropboxService()
|
||||
mock_client = MagicMock()
|
||||
|
||||
# Mock the existing link retrieval (success case)
|
||||
mock_existing_link = MagicMock()
|
||||
mock_existing_link.url = "https://dropbox.com/existing/link"
|
||||
mock_links_result = MagicMock()
|
||||
mock_links_result.links = [mock_existing_link]
|
||||
mock_client.sharing_list_shared_links.return_value = mock_links_result
|
||||
|
||||
# Assign the mock client to the service
|
||||
service._client = mock_client
|
||||
|
||||
# Verify the mock returns the expected link structure
|
||||
result = mock_client.sharing_list_shared_links(path="/test/path")
|
||||
assert result.links[0].url == "https://dropbox.com/existing/link"
|
||||
mock_client.sharing_list_shared_links.assert_called_once_with(path="/test/path")
|
||||
|
||||
|
||||
class TestGoogleDriveService:
|
||||
"""Tests for the GoogleDriveService class."""
|
||||
|
||||
def test_module_imports(self):
|
||||
"""Test that the module imports correctly."""
|
||||
from backend.services.gdrive_service import GoogleDriveService, get_gdrive_service
|
||||
assert GoogleDriveService is not None
|
||||
assert get_gdrive_service is not None
|
||||
|
||||
def test_init_creates_instance(self):
|
||||
"""Test that we can create a GoogleDriveService instance."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
service = GoogleDriveService()
|
||||
assert service is not None
|
||||
assert service._service is None # Service not initialized until first use
|
||||
assert service._loaded is False
|
||||
|
||||
def test_is_configured_returns_false_without_credentials(self):
|
||||
"""Test that is_configured returns False when credentials are missing."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
with patch.object(GoogleDriveService, '_load_credentials', return_value=None):
|
||||
service = GoogleDriveService()
|
||||
assert service.is_configured is False
|
||||
|
||||
def test_is_configured_returns_true_with_credentials(self):
|
||||
"""Test that is_configured returns True when credentials are present."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_creds = {
|
||||
"token": "test_token",
|
||||
"refresh_token": "test_refresh",
|
||||
"client_id": "test_client_id",
|
||||
"client_secret": "test_client_secret",
|
||||
}
|
||||
with patch.object(GoogleDriveService, '_load_credentials', return_value=mock_creds):
|
||||
service = GoogleDriveService()
|
||||
assert service.is_configured is True
|
||||
|
||||
def test_factory_function_returns_singleton(self):
|
||||
"""Test that get_gdrive_service returns a singleton."""
|
||||
from backend.services.gdrive_service import get_gdrive_service, _gdrive_service
|
||||
|
||||
# Reset singleton
|
||||
import backend.services.gdrive_service as gdrive_module
|
||||
gdrive_module._gdrive_service = None
|
||||
|
||||
service1 = get_gdrive_service()
|
||||
service2 = get_gdrive_service()
|
||||
assert service1 is service2
|
||||
|
||||
def test_load_credentials_caches_result(self):
|
||||
"""Test that _load_credentials caches the result."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
service = GoogleDriveService()
|
||||
service._loaded = True
|
||||
service._credentials_data = {"cached": "data"}
|
||||
|
||||
# Should return cached data without calling settings
|
||||
result = service._load_credentials()
|
||||
assert result == {"cached": "data"}
|
||||
|
||||
def test_load_credentials_falls_back_to_youtube(self):
|
||||
"""Test that _load_credentials falls back to YouTube credentials."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.get_secret.side_effect = [
|
||||
None, # gdrive-oauth-credentials not found
|
||||
'{"refresh_token": "yt_refresh", "client_id": "yt_client", "client_secret": "yt_secret"}', # youtube-oauth-credentials
|
||||
]
|
||||
|
||||
service = GoogleDriveService()
|
||||
service.settings = mock_settings
|
||||
|
||||
result = service._load_credentials()
|
||||
assert result is not None
|
||||
assert result["refresh_token"] == "yt_refresh"
|
||||
|
||||
def test_load_credentials_returns_none_when_both_fail(self):
|
||||
"""Test that _load_credentials returns None when no credentials available."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.get_secret.return_value = None
|
||||
|
||||
service = GoogleDriveService()
|
||||
service.settings = mock_settings
|
||||
|
||||
result = service._load_credentials()
|
||||
assert result is None
|
||||
assert service._loaded is True
|
||||
|
||||
def test_load_credentials_validates_required_fields(self):
|
||||
"""Test that _load_credentials validates required fields."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = MagicMock()
|
||||
# Missing client_secret
|
||||
mock_settings.get_secret.return_value = '{"refresh_token": "test", "client_id": "test"}'
|
||||
|
||||
service = GoogleDriveService()
|
||||
service.settings = mock_settings
|
||||
|
||||
result = service._load_credentials()
|
||||
assert result is None # Should fail validation
|
||||
|
||||
def test_load_credentials_handles_json_error(self):
|
||||
"""Test that _load_credentials handles JSON parse errors."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
mock_settings = MagicMock()
|
||||
mock_settings.get_secret.return_value = "not valid json"
|
||||
|
||||
service = GoogleDriveService()
|
||||
service.settings = mock_settings
|
||||
|
||||
result = service._load_credentials()
|
||||
assert result is None
|
||||
|
||||
def test_service_property_raises_without_credentials(self):
|
||||
"""Test that service property raises error without credentials."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
with patch.object(GoogleDriveService, '_load_credentials', return_value=None):
|
||||
service = GoogleDriveService()
|
||||
with pytest.raises(RuntimeError, match="credentials not configured"):
|
||||
_ = service.service
|
||||
|
||||
def test_get_or_create_folder_finds_existing(self):
|
||||
"""Test that get_or_create_folder finds existing folders."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
service = GoogleDriveService()
|
||||
mock_service = MagicMock()
|
||||
mock_files = MagicMock()
|
||||
mock_list = MagicMock()
|
||||
mock_list.execute.return_value = {"files": [{"id": "existing_folder_id", "name": "TestFolder"}]}
|
||||
mock_files.list.return_value = mock_list
|
||||
mock_service.files.return_value = mock_files
|
||||
service._service = mock_service
|
||||
|
||||
result = service.get_or_create_folder("parent_id", "TestFolder")
|
||||
assert result == "existing_folder_id"
|
||||
|
||||
def test_get_or_create_folder_creates_new(self):
|
||||
"""Test that get_or_create_folder creates new folders when not found."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
service = GoogleDriveService()
|
||||
mock_service = MagicMock()
|
||||
mock_files = MagicMock()
|
||||
|
||||
# First call - folder doesn't exist
|
||||
mock_list = MagicMock()
|
||||
mock_list.execute.return_value = {"files": []}
|
||||
mock_files.list.return_value = mock_list
|
||||
|
||||
# Second call - create folder
|
||||
mock_create = MagicMock()
|
||||
mock_create.execute.return_value = {"id": "new_folder_id"}
|
||||
mock_files.create.return_value = mock_create
|
||||
|
||||
mock_service.files.return_value = mock_files
|
||||
service._service = mock_service
|
||||
|
||||
result = service.get_or_create_folder("parent_id", "NewFolder")
|
||||
assert result == "new_folder_id"
|
||||
|
||||
def test_upload_file_determines_mime_type(self):
|
||||
"""Test that upload_file determines correct MIME type."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
import tempfile
|
||||
|
||||
service = GoogleDriveService()
|
||||
mock_service = MagicMock()
|
||||
mock_files = MagicMock()
|
||||
|
||||
# Mock list (for replace_existing check)
|
||||
mock_list = MagicMock()
|
||||
mock_list.execute.return_value = {"files": []}
|
||||
mock_files.list.return_value = mock_list
|
||||
|
||||
# Mock create
|
||||
mock_create = MagicMock()
|
||||
mock_create.execute.return_value = {"id": "uploaded_file_id"}
|
||||
mock_files.create.return_value = mock_create
|
||||
|
||||
mock_service.files.return_value = mock_files
|
||||
service._service = mock_service
|
||||
|
||||
# Create a temp file with .mp4 extension
|
||||
with tempfile.NamedTemporaryFile(suffix='.mp4', delete=False) as f:
|
||||
f.write(b"fake video content")
|
||||
temp_path = f.name
|
||||
|
||||
try:
|
||||
result = service.upload_file(temp_path, "parent_id", "test.mp4")
|
||||
assert result == "uploaded_file_id"
|
||||
finally:
|
||||
os.unlink(temp_path)
|
||||
|
||||
def test_upload_file_replaces_existing(self):
|
||||
"""Test that upload_file can replace existing files."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
import tempfile
|
||||
|
||||
service = GoogleDriveService()
|
||||
mock_service = MagicMock()
|
||||
mock_files = MagicMock()
|
||||
|
||||
# Mock list - file exists
|
||||
mock_list = MagicMock()
|
||||
mock_list.execute.return_value = {"files": [{"id": "existing_id"}]}
|
||||
mock_files.list.return_value = mock_list
|
||||
|
||||
# Mock delete
|
||||
mock_delete = MagicMock()
|
||||
mock_files.delete.return_value = mock_delete
|
||||
|
||||
# Mock create
|
||||
mock_create = MagicMock()
|
||||
mock_create.execute.return_value = {"id": "new_file_id"}
|
||||
mock_files.create.return_value = mock_create
|
||||
|
||||
mock_service.files.return_value = mock_files
|
||||
service._service = mock_service
|
||||
|
||||
with tempfile.NamedTemporaryFile(suffix='.txt', delete=False) as f:
|
||||
f.write(b"content")
|
||||
temp_path = f.name
|
||||
|
||||
try:
|
||||
result = service.upload_file(temp_path, "parent_id", "test.txt", replace_existing=True)
|
||||
assert result == "new_file_id"
|
||||
mock_files.delete.assert_called_once()
|
||||
finally:
|
||||
os.unlink(temp_path)
|
||||
|
||||
def test_upload_to_public_share_creates_structure(self):
|
||||
"""Test that upload_to_public_share creates folder structure."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
import tempfile
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
# Create test files
|
||||
mp4_path = os.path.join(tmpdir, "test.mp4")
|
||||
mp4_720_path = os.path.join(tmpdir, "test_720p.mp4")
|
||||
cdg_path = os.path.join(tmpdir, "test.zip")
|
||||
|
||||
open(mp4_path, 'wb').write(b"mp4 content")
|
||||
open(mp4_720_path, 'wb').write(b"720p content")
|
||||
open(cdg_path, 'wb').write(b"cdg content")
|
||||
|
||||
with patch.object(service, 'get_or_create_folder', return_value="folder_id"):
|
||||
with patch.object(service, 'upload_file', return_value="file_id"):
|
||||
result = service.upload_to_public_share(
|
||||
root_folder_id="root_id",
|
||||
brand_code="TEST-0001",
|
||||
base_name="Artist - Title",
|
||||
output_files={
|
||||
"final_karaoke_lossy_mp4": mp4_path,
|
||||
"final_karaoke_lossy_720p_mp4": mp4_720_path,
|
||||
"final_karaoke_cdg_zip": cdg_path,
|
||||
}
|
||||
)
|
||||
|
||||
assert "mp4" in result
|
||||
assert "mp4_720p" in result
|
||||
assert "cdg" in result
|
||||
|
||||
def test_upload_to_public_share_handles_missing_files(self):
|
||||
"""Test that upload_to_public_share handles missing files gracefully."""
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
|
||||
service = GoogleDriveService()
|
||||
|
||||
with patch.object(service, 'get_or_create_folder', return_value="folder_id"):
|
||||
with patch.object(service, 'upload_file', return_value="file_id"):
|
||||
result = service.upload_to_public_share(
|
||||
root_folder_id="root_id",
|
||||
brand_code="TEST-0001",
|
||||
base_name="Artist - Title",
|
||||
output_files={
|
||||
"final_karaoke_lossy_mp4": "/nonexistent/path.mp4",
|
||||
}
|
||||
)
|
||||
|
||||
# Should return empty since file doesn't exist
|
||||
assert result == {}
|
||||
|
||||
|
||||
class TestJobModelDistributionFields:
|
||||
"""Tests for distribution fields in Job model."""
|
||||
|
||||
def test_job_model_has_dropbox_path_field(self):
|
||||
"""Test that Job model has dropbox_path field."""
|
||||
from backend.models.job import Job, JobStatus
|
||||
from datetime import datetime
|
||||
|
||||
job = Job(
|
||||
job_id="test-123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(),
|
||||
updated_at=datetime.now(),
|
||||
dropbox_path="/Karaoke/Tracks-Organized",
|
||||
)
|
||||
assert job.dropbox_path == "/Karaoke/Tracks-Organized"
|
||||
|
||||
def test_job_model_has_gdrive_folder_id_field(self):
|
||||
"""Test that Job model has gdrive_folder_id field."""
|
||||
from backend.models.job import Job, JobStatus
|
||||
from datetime import datetime
|
||||
|
||||
job = Job(
|
||||
job_id="test-123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(),
|
||||
updated_at=datetime.now(),
|
||||
gdrive_folder_id="1abc123xyz",
|
||||
)
|
||||
assert job.gdrive_folder_id == "1abc123xyz"
|
||||
|
||||
def test_job_model_distribution_fields_optional(self):
|
||||
"""Test that distribution fields are optional."""
|
||||
from backend.models.job import Job, JobStatus
|
||||
from datetime import datetime
|
||||
|
||||
job = Job(
|
||||
job_id="test-123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(),
|
||||
updated_at=datetime.now(),
|
||||
)
|
||||
assert job.dropbox_path is None
|
||||
assert job.gdrive_folder_id is None
|
||||
|
||||
|
||||
class TestJobCreateDistributionFields:
|
||||
"""Tests for distribution fields in JobCreate model."""
|
||||
|
||||
def test_job_create_has_dropbox_path_field(self):
|
||||
"""Test that JobCreate model has dropbox_path field."""
|
||||
from backend.models.job import JobCreate
|
||||
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
dropbox_path="/Karaoke/Tracks-Organized",
|
||||
)
|
||||
assert job_create.dropbox_path == "/Karaoke/Tracks-Organized"
|
||||
|
||||
def test_job_create_has_gdrive_folder_id_field(self):
|
||||
"""Test that JobCreate model has gdrive_folder_id field."""
|
||||
from backend.models.job import JobCreate
|
||||
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
gdrive_folder_id="1abc123xyz",
|
||||
)
|
||||
assert job_create.gdrive_folder_id == "1abc123xyz"
|
||||
|
||||
def test_job_create_distribution_fields_optional(self):
|
||||
"""Test that distribution fields are optional in JobCreate."""
|
||||
from backend.models.job import JobCreate
|
||||
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
)
|
||||
assert job_create.dropbox_path is None
|
||||
assert job_create.gdrive_folder_id is None
|
||||
|
||||
|
||||
class TestFileUploadDistributionParams:
|
||||
"""Tests for distribution parameters in file upload endpoint."""
|
||||
|
||||
def test_endpoint_has_dropbox_path_parameter(self):
|
||||
"""Test that upload endpoint signature includes dropbox_path parameter."""
|
||||
import inspect
|
||||
from backend.api.routes.file_upload import upload_and_create_job
|
||||
|
||||
sig = inspect.signature(upload_and_create_job)
|
||||
param_names = list(sig.parameters.keys())
|
||||
|
||||
assert "dropbox_path" in param_names
|
||||
|
||||
def test_endpoint_has_gdrive_folder_id_parameter(self):
|
||||
"""Test that upload endpoint signature includes gdrive_folder_id parameter."""
|
||||
import inspect
|
||||
from backend.api.routes.file_upload import upload_and_create_job
|
||||
|
||||
sig = inspect.signature(upload_and_create_job)
|
||||
param_names = list(sig.parameters.keys())
|
||||
|
||||
assert "gdrive_folder_id" in param_names
|
||||
|
||||
def test_endpoint_dropbox_path_is_optional(self):
|
||||
"""Test that dropbox_path parameter has a default value (Form(None))."""
|
||||
import inspect
|
||||
from backend.api.routes.file_upload import upload_and_create_job
|
||||
|
||||
sig = inspect.signature(upload_and_create_job)
|
||||
param = sig.parameters.get("dropbox_path")
|
||||
|
||||
assert param is not None
|
||||
# Default is Form(None), so check that it's not required
|
||||
assert param.default is not inspect.Parameter.empty
|
||||
|
||||
def test_endpoint_gdrive_folder_id_is_optional(self):
|
||||
"""Test that gdrive_folder_id parameter has a default value (Form(None))."""
|
||||
import inspect
|
||||
from backend.api.routes.file_upload import upload_and_create_job
|
||||
|
||||
sig = inspect.signature(upload_and_create_job)
|
||||
param = sig.parameters.get("gdrive_folder_id")
|
||||
|
||||
assert param is not None
|
||||
# Default is Form(None), so check that it's not required
|
||||
assert param.default is not inspect.Parameter.empty
|
||||
|
||||
|
||||
class TestVideoWorkerDistribution:
|
||||
"""Tests for distribution handling in video worker."""
|
||||
|
||||
def test_handle_native_distribution_function_exists(self):
|
||||
"""Test that _handle_native_distribution function exists."""
|
||||
from backend.workers import video_worker
|
||||
assert hasattr(video_worker, '_handle_native_distribution')
|
||||
|
||||
def test_video_worker_imports_distribution_services(self):
|
||||
"""Test that video worker can import distribution services."""
|
||||
# The services should be importable (even if credentials aren't available)
|
||||
try:
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
from backend.services.gdrive_service import GoogleDriveService
|
||||
assert True
|
||||
except ImportError as e:
|
||||
pytest.fail(f"Failed to import distribution services: {e}")
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_handle_native_distribution_skips_without_config(self):
|
||||
"""Test that _handle_native_distribution skips when not configured."""
|
||||
from backend.workers.video_worker import _handle_native_distribution
|
||||
|
||||
mock_job = MagicMock()
|
||||
mock_job.dropbox_path = None
|
||||
mock_job.gdrive_folder_id = None
|
||||
mock_job.brand_prefix = None
|
||||
mock_job.artist = "Test"
|
||||
mock_job.title = "Song"
|
||||
|
||||
mock_job_log = MagicMock()
|
||||
mock_job_manager = MagicMock()
|
||||
|
||||
# Should complete without error
|
||||
await _handle_native_distribution(
|
||||
job_id="test-123",
|
||||
job=mock_job,
|
||||
job_log=mock_job_log,
|
||||
job_manager=mock_job_manager,
|
||||
temp_dir="/tmp/test",
|
||||
result={},
|
||||
)
|
||||
|
||||
# No errors should have been logged
|
||||
mock_job_log.error.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_handle_native_distribution_dropbox_not_configured(self):
|
||||
"""Test Dropbox upload skipped when service not configured."""
|
||||
from backend.workers.video_worker import _handle_native_distribution
|
||||
|
||||
mock_job = MagicMock()
|
||||
mock_job.dropbox_path = "/test/path"
|
||||
mock_job.brand_prefix = "TEST"
|
||||
mock_job.gdrive_folder_id = None
|
||||
mock_job.artist = "Test"
|
||||
mock_job.title = "Song"
|
||||
mock_job.state_data = {}
|
||||
|
||||
mock_job_log = MagicMock()
|
||||
mock_job_manager = MagicMock()
|
||||
|
||||
mock_dropbox = MagicMock()
|
||||
mock_dropbox.is_configured = False
|
||||
|
||||
with patch('backend.services.dropbox_service.get_dropbox_service', return_value=mock_dropbox):
|
||||
await _handle_native_distribution(
|
||||
job_id="test-123",
|
||||
job=mock_job,
|
||||
job_log=mock_job_log,
|
||||
job_manager=mock_job_manager,
|
||||
temp_dir="/tmp/test",
|
||||
result={},
|
||||
)
|
||||
|
||||
# Should log warning about not configured
|
||||
mock_job_log.warning.assert_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_handle_native_distribution_gdrive_not_configured(self):
|
||||
"""Test Google Drive upload skipped when service not configured."""
|
||||
from backend.workers.video_worker import _handle_native_distribution
|
||||
|
||||
mock_job = MagicMock()
|
||||
mock_job.dropbox_path = None
|
||||
mock_job.brand_prefix = None
|
||||
mock_job.gdrive_folder_id = "test_folder_id"
|
||||
mock_job.artist = "Test"
|
||||
mock_job.title = "Song"
|
||||
mock_job.state_data = {}
|
||||
|
||||
mock_job_log = MagicMock()
|
||||
mock_job_manager = MagicMock()
|
||||
|
||||
mock_gdrive = MagicMock()
|
||||
mock_gdrive.is_configured = False
|
||||
|
||||
with patch('backend.services.gdrive_service.get_gdrive_service', return_value=mock_gdrive):
|
||||
await _handle_native_distribution(
|
||||
job_id="test-123",
|
||||
job=mock_job,
|
||||
job_log=mock_job_log,
|
||||
job_manager=mock_job_manager,
|
||||
temp_dir="/tmp/test",
|
||||
result={},
|
||||
)
|
||||
|
||||
# Should log warning about not configured
|
||||
mock_job_log.warning.assert_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_handle_native_distribution_handles_import_error(self):
|
||||
"""Test that import errors for services are handled gracefully."""
|
||||
from backend.workers.video_worker import _handle_native_distribution
|
||||
|
||||
mock_job = MagicMock()
|
||||
mock_job.dropbox_path = "/test/path"
|
||||
mock_job.brand_prefix = "TEST"
|
||||
mock_job.gdrive_folder_id = None
|
||||
mock_job.artist = "Test"
|
||||
mock_job.title = "Song"
|
||||
mock_job.state_data = {}
|
||||
|
||||
mock_job_log = MagicMock()
|
||||
mock_job_manager = MagicMock()
|
||||
|
||||
# Simulate import error
|
||||
with patch('backend.services.dropbox_service.get_dropbox_service', side_effect=ImportError("No module")):
|
||||
await _handle_native_distribution(
|
||||
job_id="test-123",
|
||||
job=mock_job,
|
||||
job_log=mock_job_log,
|
||||
job_manager=mock_job_manager,
|
||||
temp_dir="/tmp/test",
|
||||
result={},
|
||||
)
|
||||
|
||||
# Should log warning about import error
|
||||
mock_job_log.warning.assert_called()
|
||||
@@ -0,0 +1,472 @@
|
||||
"""
|
||||
Tests for dropbox_service.py - Dropbox file operations.
|
||||
|
||||
These tests mock the Dropbox SDK and Secret Manager to verify:
|
||||
- Credential loading from Secret Manager
|
||||
- Folder listing and brand code calculation
|
||||
- File and folder uploads
|
||||
- Shared link creation
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, patch
|
||||
|
||||
|
||||
class TestDropboxServiceInit:
|
||||
"""Test DropboxService initialization."""
|
||||
|
||||
def test_init_creates_service(self):
|
||||
"""Test initialization creates service with no client."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
assert service._client is None
|
||||
assert service._is_configured is False
|
||||
|
||||
|
||||
class TestLoadCredentials:
|
||||
"""Test _load_credentials method."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_load_credentials_success(self, mock_sm_client_class):
|
||||
"""Test successful credential loading from Secret Manager."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({
|
||||
"access_token": "access-token-123",
|
||||
"refresh_token": "refresh-token-456",
|
||||
"app_key": "app-key",
|
||||
"app_secret": "app-secret",
|
||||
}).encode("UTF-8")
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
creds = service._load_credentials()
|
||||
|
||||
assert creds is not None
|
||||
assert creds["access_token"] == "access-token-123"
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_load_credentials_failure(self, mock_sm_client_class):
|
||||
"""Test handling when Secret Manager fails."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_sm_client.access_secret_version.side_effect = Exception("Access denied")
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
creds = service._load_credentials()
|
||||
|
||||
assert creds is None
|
||||
|
||||
|
||||
class TestIsConfigured:
|
||||
"""Test is_configured property."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_is_configured_true(self, mock_sm_client_class):
|
||||
"""Test is_configured returns True when credentials available."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({
|
||||
"access_token": "token"
|
||||
}).encode("UTF-8")
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
assert service.is_configured is True
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_is_configured_false_no_token(self, mock_sm_client_class):
|
||||
"""Test is_configured returns False when no access_token."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({
|
||||
"refresh_token": "refresh" # Missing access_token
|
||||
}).encode("UTF-8")
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
assert service.is_configured is False
|
||||
|
||||
|
||||
class TestDropboxClient:
|
||||
"""Test client property."""
|
||||
|
||||
@patch("dropbox.Dropbox")
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_client_creates_dropbox_instance(self, mock_sm_client_class, mock_dropbox_class):
|
||||
"""Test client property creates Dropbox SDK client."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({
|
||||
"access_token": "token",
|
||||
"refresh_token": "refresh",
|
||||
"app_key": "key",
|
||||
"app_secret": "secret",
|
||||
}).encode("UTF-8")
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
mock_dropbox = Mock()
|
||||
mock_dropbox_class.return_value = mock_dropbox
|
||||
|
||||
service = DropboxService()
|
||||
client = service.client
|
||||
|
||||
mock_dropbox_class.assert_called_once_with(
|
||||
oauth2_access_token="token",
|
||||
oauth2_refresh_token="refresh",
|
||||
app_key="key",
|
||||
app_secret="secret",
|
||||
)
|
||||
assert client == mock_dropbox
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_client_raises_on_missing_credentials(self, mock_sm_client_class):
|
||||
"""Test client raises RuntimeError when credentials missing."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_sm_client.access_secret_version.side_effect = Exception("Not found")
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
with pytest.raises(RuntimeError) as exc_info:
|
||||
_ = service.client
|
||||
|
||||
assert "not configured" in str(exc_info.value)
|
||||
|
||||
|
||||
class TestListFolders:
|
||||
"""Test list_folders method."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_list_folders(self, mock_sm_client_class):
|
||||
"""Test listing folders at a path."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
from dropbox.files import FolderMetadata
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
# Create mock folder entries
|
||||
mock_folder1 = Mock(spec=FolderMetadata)
|
||||
mock_folder1.name = "NOMAD-0001"
|
||||
mock_folder2 = Mock(spec=FolderMetadata)
|
||||
mock_folder2.name = "NOMAD-0002"
|
||||
mock_file = Mock() # Not a FolderMetadata
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.entries = [mock_folder1, mock_folder2, mock_file]
|
||||
mock_result.has_more = False
|
||||
|
||||
service = DropboxService()
|
||||
# Directly set the client to avoid needing to mock the whole init chain
|
||||
mock_dropbox = Mock()
|
||||
mock_dropbox.files_list_folder.return_value = mock_result
|
||||
service._client = mock_dropbox
|
||||
|
||||
folders = service.list_folders("/Karaoke/Tracks")
|
||||
|
||||
mock_dropbox.files_list_folder.assert_called_once_with("/Karaoke/Tracks")
|
||||
assert folders == ["NOMAD-0001", "NOMAD-0002"]
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_list_folders_adds_leading_slash(self, mock_sm_client_class):
|
||||
"""Test that path without leading slash gets one added."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
mock_result = Mock()
|
||||
mock_result.entries = []
|
||||
mock_result.has_more = False
|
||||
|
||||
service = DropboxService()
|
||||
mock_dropbox = Mock()
|
||||
mock_dropbox.files_list_folder.return_value = mock_result
|
||||
service._client = mock_dropbox
|
||||
|
||||
service.list_folders("path/without/slash")
|
||||
|
||||
mock_dropbox.files_list_folder.assert_called_once_with("/path/without/slash")
|
||||
|
||||
|
||||
class TestGetNextBrandCode:
|
||||
"""Test get_next_brand_code method."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_get_next_brand_code(self, mock_sm_client_class):
|
||||
"""Test calculating next brand code from existing folders."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
# Mock list_folders to return existing codes
|
||||
with patch.object(service, "list_folders") as mock_list:
|
||||
mock_list.return_value = [
|
||||
"NOMAD-1161",
|
||||
"NOMAD-1162",
|
||||
"NOMAD-1163",
|
||||
"Other Folder",
|
||||
"NOMAD-0001",
|
||||
]
|
||||
|
||||
next_code = service.get_next_brand_code("/path", "NOMAD")
|
||||
|
||||
assert next_code == "NOMAD-1164"
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_get_next_brand_code_empty_folder(self, mock_sm_client_class):
|
||||
"""Test brand code calculation with no existing codes."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
with patch.object(service, "list_folders") as mock_list:
|
||||
mock_list.return_value = []
|
||||
|
||||
next_code = service.get_next_brand_code("/path", "BRAND")
|
||||
|
||||
assert next_code == "BRAND-0001"
|
||||
|
||||
|
||||
class TestUploadFile:
|
||||
"""Test upload_file method."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_upload_small_file(self, mock_sm_client_class, tmp_path):
|
||||
"""Test uploading a small file directly."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
# Create test file
|
||||
test_file = tmp_path / "test.txt"
|
||||
test_file.write_text("Small file content")
|
||||
|
||||
service = DropboxService()
|
||||
mock_dropbox = Mock()
|
||||
service._client = mock_dropbox
|
||||
|
||||
service.upload_file(str(test_file), "/Uploads/test.txt")
|
||||
|
||||
mock_dropbox.files_upload.assert_called_once()
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_upload_file_adds_leading_slash(self, mock_sm_client_class, tmp_path):
|
||||
"""Test upload adds leading slash to remote path."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
test_file = tmp_path / "test.txt"
|
||||
test_file.write_text("content")
|
||||
|
||||
service = DropboxService()
|
||||
mock_dropbox = Mock()
|
||||
service._client = mock_dropbox
|
||||
|
||||
service.upload_file(str(test_file), "uploads/test.txt")
|
||||
|
||||
# Check that the path has leading slash
|
||||
call_args = mock_dropbox.files_upload.call_args
|
||||
assert call_args[0][1] == "/uploads/test.txt"
|
||||
|
||||
|
||||
class TestUploadFolder:
|
||||
"""Test upload_folder method."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_upload_folder(self, mock_sm_client_class, tmp_path):
|
||||
"""Test uploading a folder with multiple files."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
# Create test folder with files (no subdirs for this test)
|
||||
(tmp_path / "file1.txt").write_text("content1")
|
||||
(tmp_path / "file2.txt").write_text("content2")
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
with patch.object(service, "upload_file") as mock_upload:
|
||||
service.upload_folder(str(tmp_path), "/Uploads/folder")
|
||||
|
||||
# Should upload 2 files
|
||||
assert mock_upload.call_count == 2
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_upload_folder_recursive(self, mock_sm_client_class, tmp_path):
|
||||
"""Test uploading a folder recursively includes subdirectories."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
# Create test folder with files and subdirectories
|
||||
(tmp_path / "root_file.txt").write_text("root content")
|
||||
(tmp_path / "stems").mkdir()
|
||||
(tmp_path / "stems" / "vocals.flac").write_text("vocals")
|
||||
(tmp_path / "stems" / "instrumental.flac").write_text("instrumental")
|
||||
(tmp_path / "lyrics").mkdir()
|
||||
(tmp_path / "lyrics" / "song.lrc").write_text("lyrics")
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
uploaded_files = []
|
||||
def capture_upload(local_path, remote_path):
|
||||
uploaded_files.append(remote_path)
|
||||
|
||||
with patch.object(service, "upload_file", side_effect=capture_upload) as mock_upload:
|
||||
service.upload_folder(str(tmp_path), "/Uploads/folder")
|
||||
|
||||
# Should upload 4 files (1 root + 2 stems + 1 lyrics)
|
||||
assert mock_upload.call_count == 4
|
||||
|
||||
# Check that subdirectory structure is preserved
|
||||
assert "/Uploads/folder/root_file.txt" in uploaded_files
|
||||
assert "/Uploads/folder/stems/vocals.flac" in uploaded_files
|
||||
assert "/Uploads/folder/stems/instrumental.flac" in uploaded_files
|
||||
assert "/Uploads/folder/lyrics/song.lrc" in uploaded_files
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_upload_folder_deeply_nested(self, mock_sm_client_class, tmp_path):
|
||||
"""Test uploading deeply nested folder structure."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
# Create deeply nested structure
|
||||
(tmp_path / "level1").mkdir()
|
||||
(tmp_path / "level1" / "level2").mkdir()
|
||||
(tmp_path / "level1" / "level2" / "deep_file.txt").write_text("deep")
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
uploaded_files = []
|
||||
def capture_upload(local_path, remote_path):
|
||||
uploaded_files.append(remote_path)
|
||||
|
||||
with patch.object(service, "upload_file", side_effect=capture_upload):
|
||||
service.upload_folder(str(tmp_path), "/Uploads/folder")
|
||||
|
||||
# Check deeply nested file is uploaded with correct path
|
||||
assert "/Uploads/folder/level1/level2/deep_file.txt" in uploaded_files
|
||||
|
||||
|
||||
class TestCreateSharedLink:
|
||||
"""Test create_shared_link method."""
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_create_shared_link_new(self, mock_sm_client_class):
|
||||
"""Test creating a new shared link."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
mock_link = Mock()
|
||||
mock_link.url = "https://dropbox.com/s/abc123/file.mp4"
|
||||
|
||||
service = DropboxService()
|
||||
mock_dropbox = Mock()
|
||||
mock_dropbox.sharing_create_shared_link_with_settings.return_value = mock_link
|
||||
service._client = mock_dropbox
|
||||
|
||||
url = service.create_shared_link("/Uploads/file.mp4")
|
||||
|
||||
assert url == "https://dropbox.com/s/abc123/file.mp4"
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_create_shared_link_existing(self, mock_sm_client_class):
|
||||
"""Test getting existing shared link when one already exists."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
from dropbox.exceptions import ApiError
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
# Simulate "link already exists" error
|
||||
mock_error = Mock()
|
||||
mock_error.is_shared_link_already_exists.return_value = True
|
||||
|
||||
mock_existing_link = Mock()
|
||||
mock_existing_link.url = "https://dropbox.com/s/existing/file.mp4"
|
||||
|
||||
mock_links_result = Mock()
|
||||
mock_links_result.links = [mock_existing_link]
|
||||
|
||||
service = DropboxService()
|
||||
mock_dropbox = Mock()
|
||||
mock_dropbox.sharing_create_shared_link_with_settings.side_effect = \
|
||||
ApiError("req_id", mock_error, "message", "headers")
|
||||
mock_dropbox.sharing_list_shared_links.return_value = mock_links_result
|
||||
service._client = mock_dropbox
|
||||
|
||||
url = service.create_shared_link("/Uploads/file.mp4")
|
||||
|
||||
assert url == "https://dropbox.com/s/existing/file.mp4"
|
||||
|
||||
Loaded 100 of 961 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user