mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 21:17:56 +09:00
Compare commits
275
Commits
No files matched your search
@@ -1,74 +0,0 @@
|
||||
---
|
||||
description: Periodic documentation maintenance and organization check
|
||||
allowed-tools: Read, Glob, Grep, Edit, Write, Bash
|
||||
---
|
||||
|
||||
# Documentation Maintenance Check
|
||||
|
||||
You are performing periodic maintenance on the project documentation.
|
||||
|
||||
## Checks to Perform
|
||||
|
||||
### 1. Structure Compliance
|
||||
Verify docs/ follows the correct structure:
|
||||
```
|
||||
docs/
|
||||
├── README.md # Current status + navigation
|
||||
├── ARCHITECTURE.md # System design
|
||||
├── DEVELOPMENT.md # Dev setup, testing, deployment
|
||||
├── API.md # Backend API reference
|
||||
├── LESSONS-LEARNED.md # Accumulated wisdom
|
||||
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
|
||||
```
|
||||
|
||||
Report any files that don't belong or are misplaced.
|
||||
|
||||
### 2. Freshness Check
|
||||
- Read `docs/README.md` - is the status section current?
|
||||
- Check recent git commits - do they suggest docs should be updated?
|
||||
- Look for stale information that contradicts current code
|
||||
|
||||
### 3. Archive Organization
|
||||
- Verify all archive files have YYYY-MM-DD prefix
|
||||
- Check for any docs that should be archived (completed work, old plans)
|
||||
- Remove truly obsolete content (empty files, duplicate info)
|
||||
|
||||
### 4. Cross-Reference Check
|
||||
- Verify CLAUDE.md points to current doc locations
|
||||
- Check that docs reference each other correctly
|
||||
- Ensure no broken internal links
|
||||
|
||||
### 5. Content Quality
|
||||
- Flag any docs over 500 lines (may need splitting)
|
||||
- Identify duplicate information across docs
|
||||
- Note sections that seem outdated
|
||||
|
||||
## Actions to Take
|
||||
|
||||
1. **Fix structural issues** - Move/rename misplaced files
|
||||
2. **Update stale content** - Refresh outdated information
|
||||
3. **Archive completed work** - Move old docs to archive/
|
||||
4. **Report findings** - Summarize what was found and fixed
|
||||
|
||||
## Output Format
|
||||
|
||||
Provide a maintenance report:
|
||||
```
|
||||
## Documentation Maintenance Report - YYYY-MM-DD
|
||||
|
||||
### Issues Found
|
||||
- [list of issues]
|
||||
|
||||
### Actions Taken
|
||||
- [list of fixes made]
|
||||
|
||||
### Recommendations
|
||||
- [suggestions for future attention]
|
||||
|
||||
### Status
|
||||
- [ ] Structure: OK/Fixed
|
||||
- [ ] Freshness: OK/Updated
|
||||
- [ ] Archive: OK/Organized
|
||||
- [ ] Cross-refs: OK/Fixed
|
||||
- [ ] Quality: OK/Flagged
|
||||
```
|
||||
@@ -1,43 +0,0 @@
|
||||
---
|
||||
description: Review current session for documentation updates before merging PR
|
||||
allowed-tools: Read, Glob, Grep, Edit, Write
|
||||
---
|
||||
|
||||
# Pre-Merge Documentation Review
|
||||
|
||||
You are reviewing this chat session to identify documentation updates needed before merging the current PR.
|
||||
|
||||
## Your Task
|
||||
|
||||
1. **Analyze the current session** - What work was done? What was learned? What changed?
|
||||
|
||||
2. **Check if docs need updates:**
|
||||
- Does `docs/README.md` status section need updating?
|
||||
- Should anything be added to `docs/LESSONS-LEARNED.md`?
|
||||
- Does `docs/ARCHITECTURE.md` need changes (if architecture changed)?
|
||||
- Does `docs/DEVELOPMENT.md` need changes (if dev workflow changed)?
|
||||
- Does `docs/API.md` need changes (if API changed)?
|
||||
|
||||
3. **For significant completed work:**
|
||||
- Create `docs/archive/YYYY-MM-DD-topic.md` summarizing the work
|
||||
- Use today's date and a descriptive topic name
|
||||
|
||||
4. **Update docs/README.md** with:
|
||||
- Current status if it changed
|
||||
- Any new known issues discovered
|
||||
- Link to new archive doc if created
|
||||
|
||||
## Guidelines
|
||||
|
||||
- Keep updates concise and factual
|
||||
- Focus on information useful for future AI agents
|
||||
- Don't duplicate information that's in the code
|
||||
- Archive docs should capture "what was done and why" not implementation details
|
||||
|
||||
## Output
|
||||
|
||||
After reviewing, either:
|
||||
- Make the necessary documentation updates
|
||||
- Report "No documentation updates needed" with brief explanation why
|
||||
|
||||
Remember: Good documentation helps the next agent continue where you left off.
|
||||
@@ -0,0 +1,218 @@
|
||||
# Plan: Job Email Notifications & Admin Message Tools
|
||||
|
||||
**Created:** 2026-01-06
|
||||
**Branch:** feat/sess-20260106-0116-add-email-button
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Add comprehensive email notification system for karaoke-gen that:
|
||||
1. **Auto-sends emails** when jobs complete or need user input (after 5-minute idle threshold)
|
||||
2. **Uses cloud-stored templates** for the job completion message
|
||||
3. **Provides admin tools** to copy/send completion messages manually (for Fiverr, etc.)
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements
|
||||
- [ ] Auto-send job completion email to user (CC: gen@nomadkaraoke.com)
|
||||
- [ ] Auto-send "action needed" reminder emails when user is idle for 5+ minutes at blocking states (lyrics review, instrumental selection)
|
||||
- [ ] Store email templates in GCS (editable without code deploy)
|
||||
- [ ] Admin button: "Copy Message" - copies templated completion message to clipboard
|
||||
- [ ] Admin button: "Send Email" - prompts for email address, then sends completion message
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [ ] Email sending must not block job completion (async/background)
|
||||
- [ ] Idle detection via Cloud Tasks scheduled task (not polling)
|
||||
- [ ] Template changes should take effect without restart (no long-term caching)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### 1. Template Storage (GCS)
|
||||
Store templates in `gs://karaoke-gen-storage/templates/`:
|
||||
- `job-completion.txt` - Plain text template for job completion
|
||||
- `job-completion.html` - HTML version (optional, for rich emails)
|
||||
- `action-needed-lyrics.txt` - Reminder for lyrics review
|
||||
- `action-needed-instrumental.txt` - Reminder for instrumental selection
|
||||
|
||||
**Template Variables:**
|
||||
- `{name}` - User's display name or "there" if unknown
|
||||
- `{youtube_url}` - YouTube video URL
|
||||
- `{dropbox_url}` - Dropbox folder URL
|
||||
- `{artist}` - Artist name
|
||||
- `{title}` - Song title
|
||||
- `{job_id}` - Job ID
|
||||
- `{review_url}` - Lyrics review URL (for action-needed emails)
|
||||
- `{instrumental_url}` - Instrumental selection URL
|
||||
- `{feedback_url}` - Feedback form URL (for completion emails)
|
||||
|
||||
### 2. Email Service Extensions
|
||||
|
||||
Extend `backend/services/email_service.py`:
|
||||
- Add `send_job_completion()` method with CC support
|
||||
- Add `send_action_reminder()` method
|
||||
- Add `render_template()` helper to fetch & render GCS templates
|
||||
|
||||
### 3. Auto-Email Triggers
|
||||
|
||||
**Job Completion:**
|
||||
- Trigger point: When job enters `COMPLETE` status in `job_manager.transition_to_state()`
|
||||
- Implementation: Call email service async in background task
|
||||
|
||||
**Action Reminders (5-min idle detection):**
|
||||
- When job enters `AWAITING_REVIEW` or `AWAITING_INSTRUMENTAL_SELECTION`:
|
||||
- Record `blocking_state_entered_at` timestamp in `state_data`
|
||||
- Schedule a Cloud Tasks task for 5 minutes later
|
||||
- The scheduled task checks if job is still in blocking state and user hasn't started review
|
||||
- If idle, send reminder email and set `reminder_sent` flag to prevent duplicates
|
||||
- **Only one reminder per job** - if `reminder_sent` is true, skip sending
|
||||
|
||||
### 4. Admin UI (Frontend)
|
||||
|
||||
Add admin-only buttons to completed jobs in `OutputLinks.tsx`:
|
||||
- "Copy Message" button - fetches rendered template, copies to clipboard
|
||||
- "Send Email" button - opens dialog for email input, calls API to send
|
||||
|
||||
### 5. Backend API Endpoints
|
||||
|
||||
New endpoints in `backend/api/routes/admin.py`:
|
||||
- `GET /api/admin/jobs/{job_id}/completion-message` - Returns rendered completion message
|
||||
- `POST /api/admin/jobs/{job_id}/send-completion-email` - Sends email to specified address
|
||||
|
||||
New internal endpoint for scheduled reminder:
|
||||
- `POST /api/internal/jobs/{job_id}/check-idle-reminder` - Called by Cloud Tasks
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Template Infrastructure
|
||||
1. [ ] Upload initial template to GCS from user's existing template file
|
||||
2. [ ] Create `template_service.py` to fetch/render templates from GCS
|
||||
3. [ ] Add unit tests for template rendering
|
||||
|
||||
### Phase 2: Email Extensions
|
||||
4. [ ] Extend `email_service.py` with CC support in `send_email()`
|
||||
5. [ ] Add `send_job_completion()` method
|
||||
6. [ ] Add `send_action_reminder()` method
|
||||
7. [ ] Add unit tests for new email methods
|
||||
|
||||
### Phase 3: Auto-Email on Completion
|
||||
8. [ ] Modify `job_manager.py` to trigger completion email when entering COMPLETE
|
||||
9. [ ] Add integration test for completion email trigger
|
||||
|
||||
### Phase 4: Idle Reminder System
|
||||
10. [ ] Add `blocking_state_entered_at` field tracking to job state transitions
|
||||
11. [ ] Create Cloud Tasks queue for idle reminders (`idle-reminder-queue`)
|
||||
12. [ ] Schedule 5-minute delayed task when entering blocking states
|
||||
13. [ ] Implement `/api/internal/jobs/{job_id}/check-idle-reminder` endpoint
|
||||
14. [ ] Add tests for idle reminder flow
|
||||
|
||||
### Phase 5: Admin API
|
||||
15. [ ] Add `GET /api/admin/jobs/{job_id}/completion-message` endpoint
|
||||
16. [ ] Add `POST /api/admin/jobs/{job_id}/send-completion-email` endpoint
|
||||
17. [ ] Add tests for admin endpoints
|
||||
|
||||
### Phase 6: Admin UI
|
||||
18. [ ] Add `isAdmin` check to `OutputLinks.tsx` component
|
||||
19. [ ] Add "Copy Message" button (admin-only) with loading state
|
||||
20. [ ] Add "Send Email" button (admin-only) with email input dialog
|
||||
21. [ ] Add E2E test for admin buttons
|
||||
|
||||
### Phase 7: Documentation
|
||||
22. [ ] Document template management in docs/DEVELOPMENT.md
|
||||
23. [ ] Update docs/API.md with new endpoints
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/services/template_service.py` | Create | GCS template fetching and rendering |
|
||||
| `backend/services/email_service.py` | Modify | Add CC support, completion/reminder methods |
|
||||
| `backend/services/job_manager.py` | Modify | Trigger emails on state transitions |
|
||||
| `backend/services/worker_service.py` | Modify | Add idle reminder task scheduling |
|
||||
| `backend/api/routes/admin.py` | Modify | Add completion message/email endpoints |
|
||||
| `backend/api/routes/internal.py` | Modify | Add idle reminder check endpoint |
|
||||
| `backend/models/job.py` | Modify | Add `blocking_state_entered_at`, `reminder_sent_at` to state_data docs |
|
||||
| `infrastructure/__main__.py` | Modify | Add idle-reminder-queue Cloud Tasks queue |
|
||||
| `frontend/components/job/OutputLinks.tsx` | Modify | Add admin-only Copy/Send buttons |
|
||||
| `frontend/lib/api.ts` | Modify | Add API methods for admin endpoints |
|
||||
| `tests/unit/test_template_service.py` | Create | Template rendering tests |
|
||||
| `tests/unit/test_email_service_completion.py` | Create | Completion email tests |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
**Unit Tests:**
|
||||
- Template rendering with all variable combinations
|
||||
- Email service CC support
|
||||
- Completion/reminder email generation
|
||||
- Idle detection logic
|
||||
|
||||
**Integration Tests:**
|
||||
- Full flow: job completion → email sent
|
||||
- Idle reminder: job enters blocking state → 5 min → reminder sent
|
||||
- Admin endpoint authentication
|
||||
|
||||
**Manual Testing:**
|
||||
- Verify emails render correctly in Gmail/Outlook
|
||||
- Test admin buttons on completed job
|
||||
- Test clipboard copy works across browsers
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] Where to store templates? → **GCS bucket: `gs://karaoke-gen-storage/templates/`**
|
||||
- [x] Should we rate-limit reminder emails? → **Max 1 reminder per job** (send once after 5 min idle, then stop)
|
||||
- [x] Should completion emails also include a feedback request link? → **Yes, for all users**
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. **Email sending failures** - All email sends are async/fire-and-forget; job completion unaffected
|
||||
2. **Template fetch failures** - Fall back to hardcoded template in code
|
||||
3. **Idle reminder failures** - Jobs continue normally; users can still access review UI manually
|
||||
4. **Feature flag** - Add `ENABLE_AUTO_EMAILS=true/false` env var for quick disable
|
||||
|
||||
## Template Content
|
||||
|
||||
Initial template (from user's file, with feedback link added):
|
||||
|
||||
```
|
||||
Hi {name},
|
||||
|
||||
Thanks for your order!
|
||||
|
||||
Here's the link for the karaoke video published to YouTube:
|
||||
{youtube_url}
|
||||
|
||||
Here's the dropbox folder with all the finished files and source files, including:
|
||||
- "(Final Karaoke Lossless).mkv": combined karaoke video in 4k H264 with lossless FLAC audio
|
||||
- "(Final Karaoke).mp4": combined karaoke video with title/end screen in 4k H264/AAC
|
||||
- "(Final Karaoke 720p).mp4": combined karaoke video in 720p H264/AAC (smaller file for older systems)
|
||||
- "(With Vocals).mp4": sing along video in 4k H264/AAC with original vocals
|
||||
- "(Karaoke).mov": karaoke video output from MidiCo (no title/end screen)
|
||||
- "(Title).mov"/"(End).mov": title card and end screen videos
|
||||
- "(Final Karaoke CDG).zip": CDG+MP3 format for older/commercial karaoke systems
|
||||
- "(Final Karaoke TXT).zip": TXT+MP3 format for Power Karaoke
|
||||
- stems/*.flac: various separated instrumental and vocal audio stems in lossless format
|
||||
- lyrics/*.txt song lyrics from various sources in plain text format
|
||||
|
||||
{dropbox_url}
|
||||
|
||||
Let me know if anything isn't perfect and I'll happily tweak / fix, or if you need it in any other format I can probably convert it for you!
|
||||
|
||||
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
|
||||
{feedback_url}
|
||||
|
||||
Thanks again and have a great day!
|
||||
-Andrew
|
||||
```
|
||||
|
||||
**Note:** The `{feedback_url}` placeholder will be populated with the feedback form URL. If no feedback URL is configured, this section can be omitted from the rendered output.
|
||||
|
||||
## Updating Templates
|
||||
|
||||
To update email templates after deployment:
|
||||
|
||||
```bash
|
||||
# Upload new template
|
||||
gsutil cp my-template.txt gs://karaoke-gen-storage/templates/job-completion.txt
|
||||
|
||||
# Templates are fetched fresh on each use (no caching in code)
|
||||
# Changes take effect immediately
|
||||
```
|
||||
@@ -0,0 +1,106 @@
|
||||
# Plan: Migrate from MailSlurp to testmail.app
|
||||
|
||||
**Created:** 2026-01-02
|
||||
**Branch:** feat/sess-20260102-2109-migrate-mailslurp-testmail
|
||||
**Status:** Complete
|
||||
|
||||
## Overview
|
||||
|
||||
Replace MailSlurp email testing service with testmail.app for E2E testing of magic link authentication. testmail.app uses a simpler API with namespace-based email addresses that require no inbox creation/deletion - emails just arrive at `{namespace}.{tag}@inbox.testmail.app`.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Remove all code referencing MailSlurp
|
||||
- [x] Remove `mailslurp-client` npm dependency
|
||||
- [x] Implement testmail.app API integration
|
||||
- [x] Update environment variable names (`TESTMAIL_API_KEY`, `TESTMAIL_NAMESPACE`)
|
||||
- [x] Maintain same `EmailHelper` interface for minimal test changes
|
||||
- [x] Update archive docs to note migration (optional - historical accuracy)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### API Differences
|
||||
|
||||
| Feature | MailSlurp | testmail.app |
|
||||
|---------|-----------|--------------|
|
||||
| Email format | Random `xxx@mailslurp.com` | `{namespace}.{tag}@inbox.testmail.app` |
|
||||
| Inbox creation | Required (`createInbox()`) | Not needed - use any tag |
|
||||
| Inbox deletion | Required (`deleteInbox()`) | Not needed |
|
||||
| Get emails | `waitForLatestEmail(inboxId)` | GET `/api/json?apikey=X&namespace=Y&tag=Z&livequery=true` |
|
||||
| Auth | API key in constructor | API key as query param |
|
||||
| Email body | `email.body` | `email.html` or `email.text` |
|
||||
|
||||
### Implementation Strategy
|
||||
|
||||
1. **Keep same `EmailHelper` interface** - `createInbox()`, `waitForEmail()`, `extractMagicLink()`, `deleteInbox()`
|
||||
2. **Use testmail.app's live query feature** - `livequery=true` waits for email to arrive (up to timeout)
|
||||
3. **Generate unique tags** - Use timestamp + random suffix for each test run to avoid collisions
|
||||
4. **Inbox becomes a simple object** - `{ id: tag, emailAddress: '{namespace}.{tag}@inbox.testmail.app' }`
|
||||
5. **Delete becomes no-op** - testmail.app emails auto-expire, no cleanup needed
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Old | New |
|
||||
|-----|-----|
|
||||
| `MAILSLURP_API_KEY` | `TESTMAIL_API_KEY` |
|
||||
| (none) | `TESTMAIL_NAMESPACE` |
|
||||
|
||||
Values: Set via environment variables (see `frontend/.env.local.example`)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. [x] **Update `email-testing.ts`** - Replace MailSlurp implementation with testmail.app
|
||||
- Remove `mailslurp-client` import
|
||||
- Add testmail.app API fetch implementation
|
||||
- Use native `fetch` (no new dependencies)
|
||||
- Generate unique tag for each "inbox"
|
||||
- Implement `livequery=true` for waiting
|
||||
- Adapt email body extraction (`html`/`text` instead of `body`)
|
||||
|
||||
2. [x] **Update `full-user-journey.spec.ts`** - Change references
|
||||
- Update doc comments (env var name)
|
||||
- Update error messages
|
||||
- Update console logs
|
||||
|
||||
3. [x] **Update `playwright.record.config.ts`** - Change env var comment
|
||||
|
||||
4. [x] **Update `package.json`** - Remove `mailslurp-client` dependency
|
||||
|
||||
5. [x] **Run `npm install`** - Update package-lock.json
|
||||
|
||||
6. [x] **Create `.env.local.example`** - Document required env vars for local testing
|
||||
|
||||
7. [x] **Update archive docs** - Skipped intentionally; historical docs preserved for accuracy
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `frontend/e2e/helpers/email-testing.ts` | Modify | Replace MailSlurp with testmail.app |
|
||||
| `frontend/e2e/production/full-user-journey.spec.ts` | Modify | Update env var refs and messages |
|
||||
| `frontend/playwright.record.config.ts` | Modify | Update env var comment |
|
||||
| `frontend/package.json` | Modify | Remove `mailslurp-client` |
|
||||
| `frontend/package-lock.json` | Modify | Auto-updated by npm install |
|
||||
| `frontend/.env.local.example` | Create | Document env vars for local testing |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Manual testing**: Run production E2E test with testmail.app credentials
|
||||
```bash
|
||||
cd frontend
|
||||
TESTMAIL_API_KEY=your-api-key \
|
||||
TESTMAIL_NAMESPACE=your-namespace \
|
||||
npm run test:e2e:prod:headed
|
||||
```
|
||||
- **Verify**: Email received, magic link extracted, auth flow completes
|
||||
- **CI testing**: Env vars already configured in GitHub Actions org secrets
|
||||
|
||||
## Open Questions
|
||||
|
||||
None - requirements are clear.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. Revert the commit
|
||||
2. Run `npm install` to restore mailslurp-client
|
||||
3. Re-add MAILSLURP_API_KEY to secrets if removed
|
||||
@@ -0,0 +1,401 @@
|
||||
# Plan: Video Worker Stage Refactor
|
||||
|
||||
**Created:** 2026-01-03
|
||||
**Branch:** feat/sess-20260103-2250-investigate-youtube-upload
|
||||
**Status:** COMPLETE - Ready for PR
|
||||
**Last Updated:** 2026-01-04
|
||||
|
||||
## Overview
|
||||
|
||||
Refactor `video_worker.py` and `KaraokeFinalise` into separate, composable stages following SOLID principles. This eliminates the GCE vs KaraokeFinalise code path divergence that causes features like YouTube upload to be missing from the GCE encoding path.
|
||||
|
||||
## Three Execution Modes
|
||||
|
||||
Understanding the three ways this system runs is critical to the refactor:
|
||||
|
||||
### 1. Local CLI (`karaoke-gen`)
|
||||
- Runs entirely on user's machine
|
||||
- KaraokeFinalise.process() orchestrates everything
|
||||
- Writes directly to local Dropbox sync folder (e.g., `~/Dropbox/Karaoke/Tracks-Organized/`)
|
||||
- **Produces full output structure** with stems/, lyrics/, visualizations/, etc.
|
||||
- **This is the source of truth for output format**
|
||||
|
||||
### 2. Remote CLI (`karaoke-gen-remote`)
|
||||
- User's CLI talks to cloud backend (api.nomadkaraoke.com)
|
||||
- Cloud does ALL processing (audio, lyrics, video encoding)
|
||||
- CLI just monitors job status and downloads final outputs
|
||||
- With Dropbox configured: cloud uploads full structure to user's Dropbox
|
||||
- Without Dropbox: user downloads individual files
|
||||
|
||||
### 3. Web Frontend (`gen.nomadkaraoke.com`)
|
||||
- Same cloud backend as remote CLI
|
||||
- User sees:
|
||||
- YouTube video link
|
||||
- Dropbox folder link
|
||||
- Direct download: final karaoke video (720p)
|
||||
- Direct download: 4K video
|
||||
- Direct download: "with vocals" video
|
||||
- Direct download: CDG zip (if enabled)
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Currently, `video_worker.py` has two completely divergent code paths:
|
||||
|
||||
```python
|
||||
if use_gce_encoding:
|
||||
# GCE PATH: Only encoding, missing YouTube/Discord/CDG/TXT
|
||||
result = await _encode_via_gce(...)
|
||||
# result DOES NOT include youtube_url!
|
||||
else:
|
||||
# STANDARD PATH: KaraokeFinalise does everything
|
||||
finalise = KaraokeFinalise(...)
|
||||
result = finalise.process()
|
||||
# result includes youtube_url from upload
|
||||
```
|
||||
|
||||
### What GCE Encoding Broke
|
||||
|
||||
| Feature | KaraokeFinalise Path | GCE Path |
|
||||
|---------|---------------------|----------|
|
||||
| Video encoding | ✅ 4 formats | ✅ 4 formats |
|
||||
| YouTube upload | ✅ | ❌ **BROKEN** |
|
||||
| Discord notification | ✅ | ❌ **BROKEN** |
|
||||
| CDG/TXT packaging | ✅ | ❌ **BROKEN** |
|
||||
| Brand code generation | ✅ | ✅ (via Dropbox service) |
|
||||
| Dropbox upload | ✅ (rclone) | ✅ (native SDK) |
|
||||
| Google Drive upload | ❌ | ✅ (native SDK) |
|
||||
|
||||
### Root Cause
|
||||
|
||||
`KaraokeFinalise.process()` is a monolith that handles:
|
||||
1. CDG/TXT packaging
|
||||
2. Video encoding (the CPU-intensive part)
|
||||
3. YouTube upload
|
||||
4. Discord notifications
|
||||
5. Folder organization
|
||||
|
||||
The GCE encoding was added as a bypass around the entire class, rather than replacing just the encoding step.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements
|
||||
- [ ] YouTube upload works regardless of encoding backend (GCE or local)
|
||||
- [ ] Discord notifications work regardless of encoding backend
|
||||
- [ ] CDG/TXT packaging works regardless of encoding backend
|
||||
- [ ] Brand code generation works consistently
|
||||
- [ ] Dropbox/Google Drive upload works (already works in both paths)
|
||||
- [ ] All existing CLI functionality continues to work
|
||||
- [ ] Output structure matches local CLI exactly
|
||||
|
||||
### Output Structure Parity
|
||||
|
||||
Cloud backend must produce identical output to local CLI:
|
||||
|
||||
```
|
||||
NOMAD-XXXX - Artist - Title/
|
||||
├── Artist - Title (Title).mov
|
||||
├── Artist - Title (Title).jpg
|
||||
├── Artist - Title (End).mov
|
||||
├── Artist - Title (End).jpg
|
||||
├── Artist - Title (Final Karaoke Lossless 4k).mp4
|
||||
├── Artist - Title (Final Karaoke Lossless 4k).mkv
|
||||
├── Artist - Title (Final Karaoke Lossy 4k).mp4
|
||||
├── Artist - Title (Final Karaoke Lossy 720p).mp4
|
||||
├── Artist - Title (Final Karaoke CDG).zip
|
||||
├── Artist - Title (Final Karaoke TXT).zip
|
||||
├── Artist - Title (Instrumental {model}).flac
|
||||
├── Artist - Title (Instrumental +BV {model}).flac
|
||||
├── Artist - Title (Karaoke).mp4
|
||||
├── Artist - Title (Karaoke).cdg
|
||||
├── Artist - Title (Karaoke).mp3
|
||||
├── Artist - Title (Karaoke).lrc
|
||||
├── Artist - Title (With Vocals).mp4
|
||||
├── stems/
|
||||
│ ├── Artist - Title (Instrumental {model}).flac
|
||||
│ ├── Artist - Title (Vocals {model}).flac
|
||||
│ ├── Artist - Title (Lead Vocals {model}).flac
|
||||
│ ├── Artist - Title (Backing Vocals {model}).flac
|
||||
│ └── ... (other stems)
|
||||
├── lyrics/
|
||||
│ ├── Artist - Title (Karaoke).lrc
|
||||
│ ├── Artist - Title (Karaoke).ass
|
||||
│ ├── Artist - Title (Karaoke).srt
|
||||
│ └── ... (corrections JSONs)
|
||||
├── previews/
|
||||
├── visualizations/
|
||||
└── styles_assets/
|
||||
```
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [ ] Single Responsibility: Each stage has one job
|
||||
- [ ] Open/Closed: Easy to add new encoding backends without modifying stages
|
||||
- [ ] Testable: Each stage can be unit tested in isolation
|
||||
- [ ] No duplicate code between paths
|
||||
- [ ] Backwards compatible with existing KaraokeFinalise CLI usage
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Architecture: Stage-Based Pipeline
|
||||
|
||||
```
|
||||
video_worker.py (VideoWorkerOrchestrator)
|
||||
│
|
||||
├── 1. EncodingStage (interface)
|
||||
│ ├── LocalEncodingService (wraps KaraokeFinalise encoding methods)
|
||||
│ └── GCEEncodingService (existing encoding_service.py)
|
||||
│
|
||||
├── 2. PackagingStage
|
||||
│ ├── CDGPackager (extracts from KaraokeFinalise)
|
||||
│ └── TXTPackager (extracts from KaraokeFinalise)
|
||||
│
|
||||
├── 3. DistributionStage
|
||||
│ ├── YouTubeUploadService (extracts from KaraokeFinalise)
|
||||
│ ├── DropboxService (existing)
|
||||
│ ├── GoogleDriveService (existing)
|
||||
│ └── DiscordService (extracts from KaraokeFinalise)
|
||||
│
|
||||
└── 4. OrganizationStage
|
||||
└── BrandCodeService (extracts from KaraokeFinalise + DropboxService)
|
||||
```
|
||||
|
||||
### Key Design Decisions
|
||||
|
||||
1. **Extract, Don't Rewrite**: Pull methods out of KaraokeFinalise into services, don't rewrite from scratch
|
||||
2. **Keep KaraokeFinalise for CLI**: Local CLI still uses KaraokeFinalise.process() - we extract services that it can also use
|
||||
3. **Interface-Based Encoding**: EncodingStage interface allows swapping GCE/local without changing orchestration
|
||||
4. **Existing Service Pattern**: Follow singleton + factory pattern already used in codebase
|
||||
5. **Local CLI stays sync**: Local karaoke-gen CLI remains synchronous as it is today
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
JobConfig (from Firestore)
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ VideoWorkerOrchestrator │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 1. Setup: Download files, prepare directories ││
|
||||
│ │ - Download title.mov, end.mov, with_vocals.mkv ││
|
||||
│ │ - Download instrumental audio ││
|
||||
│ │ - Prepare temp directory ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 2. Packaging: CDG/TXT (if enabled) - runs BEFORE encoding ││
|
||||
│ │ Reason: CDG depends on LRC file which is available ││
|
||||
│ │ Input: LRC file, instrumental audio ││
|
||||
│ │ Output: CDG.zip, TXT.zip, .cdg, .mp3 ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 3. Encoding: GCE or Local (via interface) ││
|
||||
│ │ Input: title.mov, karaoke.mov, end.mov ││
|
||||
│ │ Output: 4K lossless, 4K lossy, 720p, MKV ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 4. Organization: Brand code, folder structure ││
|
||||
│ │ - Generate brand code (NOMAD-XXXX) ││
|
||||
│ │ - Prepare full output directory structure ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 5. Distribution: Upload to destinations ││
|
||||
│ │ - YouTube upload (if enabled) ││
|
||||
│ │ - Dropbox upload (if configured) ││
|
||||
│ │ - Google Drive upload (if configured) ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 6. Notification: Discord, Email ││
|
||||
│ │ Input: YouTube URL, brand code, Dropbox link ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
Result (youtube_url, brand_code, dropbox_link, file_paths, etc.)
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Extract Services from KaraokeFinalise (Non-Breaking) ✅ COMPLETE
|
||||
|
||||
These services can be used by both the cloud orchestrator AND KaraokeFinalise.
|
||||
|
||||
1. [x] **Create `backend/services/youtube_upload_service.py`** ✅
|
||||
- Extract YouTube upload logic from KaraokeFinalise
|
||||
- Methods: `upload_video()`, `check_duplicate()`, `delete_video()`, `truncate_title()`
|
||||
- Supports pre-loaded credentials (server-side) or client secrets file (CLI)
|
||||
- 23 unit tests with mocked YouTube API
|
||||
|
||||
2. [x] **Create `backend/services/discord_service.py`** ✅
|
||||
- Extract Discord notification from KaraokeFinalise
|
||||
- Methods: `post_message()`, `post_video_notification()`, `is_enabled()`
|
||||
- Simple HTTP POST to webhook with validation
|
||||
- 20 unit tests with mocked requests
|
||||
|
||||
3. [x] **Create `backend/services/packaging_service.py`** ✅
|
||||
- Extract CDG/TXT generation from KaraokeFinalise
|
||||
- Methods: `create_cdg_package()`, `create_txt_package()`
|
||||
- Uses lyrics_converter and CDGGenerator (existing dependencies)
|
||||
- 18 unit tests with temp files
|
||||
|
||||
4. [x] **Create `backend/services/local_encoding_service.py`** ✅
|
||||
- Extract encoding methods from KaraokeFinalise
|
||||
- Methods: `encode_all_formats()`, individual encoding methods
|
||||
- Hardware acceleration detection with software fallback
|
||||
- 28 unit tests with mocked subprocess
|
||||
|
||||
### Phase 2: Create Encoding Interface ✅ COMPLETE
|
||||
|
||||
5. [x] **Create `backend/services/encoding_interface.py`** ✅
|
||||
- Define ABC `EncodingBackend` with methods: `encode()`, `is_available()`, `get_status()`
|
||||
- `EncodingInput` and `EncodingOutput` dataclasses for config/results
|
||||
- Factory function `get_encoding_backend()` with "local", "gce", "auto" options
|
||||
|
||||
6. [x] **`LocalEncodingBackend`** ✅
|
||||
- Implements `EncodingBackend` interface
|
||||
- Wraps `LocalEncodingService` with `asyncio.to_thread()`
|
||||
- 17 unit tests
|
||||
|
||||
7. [x] **`GCEEncodingBackend`** ✅
|
||||
- Implements `EncodingBackend` interface
|
||||
- Wraps existing `EncodingService` (encoding_service.py)
|
||||
- Requires GCS paths in options for cloud execution
|
||||
- 6 additional unit tests
|
||||
|
||||
### Phase 3: Create Orchestrator ✅ COMPLETE
|
||||
|
||||
8. [x] **Create `backend/workers/video_worker_orchestrator.py`** ✅
|
||||
- New class that coordinates all stages
|
||||
- Replaces the branching logic in `generate_video()`
|
||||
- Methods:
|
||||
- `__init__(config, job_manager, storage, job_logger)`
|
||||
- `async run() -> OrchestratorResult`
|
||||
- `_run_packaging()` - CDG/TXT
|
||||
- `_run_encoding()` - GCE or local via EncodingBackend interface
|
||||
- `_run_organization()` - Brand code, folder structure
|
||||
- `_run_distribution()` - YouTube, Dropbox, GDrive
|
||||
- `_run_notifications()` - Discord
|
||||
- 27 unit tests (test_video_worker_orchestrator.py)
|
||||
|
||||
9. [x] **Update `backend/workers/video_worker.py`** ✅
|
||||
- Added `USE_NEW_ORCHESTRATOR` feature flag (default: true)
|
||||
- `generate_video()` routes to orchestrator or legacy path
|
||||
- `generate_video_orchestrated()` - new unified pipeline
|
||||
- `generate_video_legacy()` - original code for rollback
|
||||
- Legacy GCE and KaraokeFinalise paths preserved for safety
|
||||
|
||||
### Phase 4: Update KaraokeFinalise to Use Services (OPTIONAL)
|
||||
|
||||
10. [ ] **Update `karaoke_gen/karaoke_finalise/karaoke_finalise.py`**
|
||||
- Import and use the new services
|
||||
- Replace inline YouTube/Discord/CDG code with service calls
|
||||
- Keep process() working for CLI backwards compatibility
|
||||
- Services become the single source of truth
|
||||
- **NOTE**: This is optional for MVP - the orchestrator already uses the extracted services directly
|
||||
|
||||
### Phase 5: Testing & Validation ✅ COMPLETE
|
||||
|
||||
11. [x] **Add unit tests for each service** ✅
|
||||
- test_youtube_upload_service.py (23 tests)
|
||||
- test_discord_service.py (20 tests)
|
||||
- test_packaging_service.py (18 tests)
|
||||
- test_local_encoding_service.py (28 tests)
|
||||
- test_encoding_interface.py (23 tests)
|
||||
- test_video_worker_orchestrator.py (27 tests)
|
||||
- **Total: 139 new tests**
|
||||
|
||||
12. [x] **Full test suite passes** ✅
|
||||
- 976 tests passed (86 skipped - emulator/integration tests)
|
||||
- No regressions from changes
|
||||
|
||||
13. [ ] **Manual E2E testing** (After deployment)
|
||||
- [ ] GCE encoding job with YouTube upload enabled
|
||||
- [ ] Local encoding job with YouTube upload enabled
|
||||
- [ ] CLI karaoke-gen with all features
|
||||
- [ ] Job with CDG/TXT enabled
|
||||
- [ ] Verify output structure matches local CLI
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
| File | Action | Description | Status |
|
||||
|------|--------|-------------|--------|
|
||||
| `backend/services/youtube_upload_service.py` | Create | YouTube upload logic (~446 lines) | ✅ |
|
||||
| `backend/services/discord_service.py` | Create | Discord notification service (~167 lines) | ✅ |
|
||||
| `backend/services/packaging_service.py` | Create | CDG/TXT package generation (~266 lines) | ✅ |
|
||||
| `backend/services/local_encoding_service.py` | Create | Local FFmpeg encoding service (~512 lines) | ✅ |
|
||||
| `backend/services/encoding_interface.py` | Create | Abstract interface + backends (~418 lines) | ✅ |
|
||||
| `backend/workers/video_worker_orchestrator.py` | Create | Stage orchestration logic (~550 lines) | ✅ |
|
||||
| `backend/workers/video_worker.py` | Modify | Added orchestrator path + feature flag | ✅ |
|
||||
| `backend/tests/test_youtube_upload_service.py` | Create | 23 unit tests | ✅ |
|
||||
| `backend/tests/test_discord_service.py` | Create | 20 unit tests | ✅ |
|
||||
| `backend/tests/test_packaging_service.py` | Create | 18 unit tests | ✅ |
|
||||
| `backend/tests/test_local_encoding_service.py` | Create | 28 unit tests | ✅ |
|
||||
| `backend/tests/test_encoding_interface.py` | Create | 23 unit tests | ✅ |
|
||||
| `backend/tests/test_video_worker_orchestrator.py` | Create | 27 unit tests | ✅ |
|
||||
| `karaoke_gen/karaoke_finalise/karaoke_finalise.py` | Modify | Use extracted services (optional) | Deferred |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- **youtube_upload_service**: Mock YouTube API, test upload/delete/duplicate check
|
||||
- **discord_service**: Mock requests, test webhook POST
|
||||
- **packaging_service**: Use temp files, test CDG/TXT generation
|
||||
- **local_encoding_service**: Mock subprocess, test FFmpeg command construction
|
||||
|
||||
### Integration Tests
|
||||
- **video_worker_orchestrator**: Mock all services, test stage sequencing
|
||||
- **video_worker**: Test with mocked orchestrator
|
||||
|
||||
### E2E Verification
|
||||
- [ ] GCE encoding job with YouTube upload enabled → YouTube link appears
|
||||
- [ ] Local encoding job with YouTube upload enabled → Same behavior
|
||||
- [ ] CLI `karaoke-gen` with all features → No regression
|
||||
- [ ] Job with CDG/TXT enabled → Packages generated
|
||||
- [ ] Dropbox output structure → Matches local CLI exactly
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] Should we keep KaraokeFinalise.process() or deprecate it?
|
||||
- **Decision**: Keep it for CLI backwards compatibility. It will use the new services internally.
|
||||
|
||||
- [x] Should local_encoding_service run FFmpeg sync or async?
|
||||
- **Decision**: Sync implementation, async wrapper via `asyncio.to_thread()` for interface compatibility.
|
||||
|
||||
- [x] Order of packaging vs encoding?
|
||||
- **Decision**: CDG/TXT runs BEFORE encoding (depends on LRC file, not videos). KaraokeFinalise does it this way too.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. The refactor is additive - new services don't replace old code until orchestrator is wired up
|
||||
2. Feature flag: `USE_NEW_ORCHESTRATOR=true` to enable new code path
|
||||
3. If issues found in production:
|
||||
- Set `USE_NEW_ORCHESTRATOR=false` to revert to old behavior
|
||||
- Old `generate_video()` code preserved behind feature flag
|
||||
4. After 1 week stable, remove feature flag and old code
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. Job `5b78ad46` scenario (GCE + YouTube) would upload to YouTube
|
||||
2. All 803+ existing tests pass
|
||||
3. E2E happy path test passes
|
||||
4. No regression in local CLI functionality
|
||||
5. Output structure matches local CLI exactly
|
||||
6. Code coverage maintained or improved
|
||||
|
||||
## Estimated Scope
|
||||
|
||||
- **Phase 1** (Extract Services): 4 new service files + tests (~400-600 lines each)
|
||||
- **Phase 2** (Interface): 1 new file, 2 modifications (~100 lines)
|
||||
- **Phase 3** (Orchestrator): 2 files (~300-400 lines)
|
||||
- **Phase 4** (KaraokeFinalise): 1 modification (~100 lines of changes)
|
||||
- **Phase 5** (Testing): Integration tests + manual verification
|
||||
|
||||
This is a medium-sized refactor that can be done incrementally. Each phase produces working, tested code.
|
||||
@@ -0,0 +1,222 @@
|
||||
# Plan: Worker Logs Rearchitecture
|
||||
|
||||
**Created:** 2026-01-04
|
||||
**Branch:** `feat/sess-20260104-1931-logs-research`
|
||||
**Status:** Implementation Complete - Pending PR Merge & Deployment
|
||||
|
||||
## Overview
|
||||
|
||||
Move `worker_logs` from embedded array in job documents to a Firestore subcollection to avoid the 1MB document size limit. The current approach caused job 501258e1 to fail when logs reached 1.26 MB (98.6% of document size).
|
||||
|
||||
## Problem Statement
|
||||
|
||||
- Firestore documents have a 1 MB size limit
|
||||
- `worker_logs` is stored as an array field within job documents
|
||||
- Long-running jobs (especially video encoding) can generate thousands of log entries
|
||||
- When the limit is exceeded, job updates fail with: `Document cannot be written because its size exceeds the maximum allowed size`
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Logs must remain associated with their job
|
||||
- [x] Logs should be queryable by job_id, worker type, timestamp
|
||||
- [x] Old logs should auto-cleanup (TTL: 30 days)
|
||||
- [x] Existing API endpoint (`GET /api/jobs/{id}/logs`) must continue to work
|
||||
- [x] Frontend must continue to display logs without changes
|
||||
- [x] Migration path for existing jobs (optional - can just start fresh)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Option A: Subcollection (Recommended)
|
||||
Store logs in `jobs/{job_id}/logs/{log_id}` subcollection.
|
||||
|
||||
**Pros:**
|
||||
- Natural association with job (delete job = delete logs with cascading)
|
||||
- Each log entry is its own document (no size limit concerns)
|
||||
- Can add TTL via Firestore TTL policies
|
||||
- Efficient queries with composite indexes
|
||||
|
||||
**Cons:**
|
||||
- More Firestore read operations for paginated log fetching
|
||||
- Slightly more complex write logic
|
||||
|
||||
### Option B: Separate Collection
|
||||
Store logs in `job_logs` collection with `job_id` field.
|
||||
|
||||
**Pros:**
|
||||
- Can query across all jobs
|
||||
- Simpler TTL management
|
||||
|
||||
**Cons:**
|
||||
- Orphaned logs if job deleted (need cleanup logic)
|
||||
- Extra index needed for job_id queries
|
||||
|
||||
### Decision: Option A (Subcollection)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Backend Changes
|
||||
|
||||
1. [x] **Create log entry model** - `backend/models/worker_log.py`
|
||||
- `WorkerLogEntry` dataclass with: id, job_id, timestamp, level, worker, message, metadata, ttl_expiry
|
||||
- Factory method `create()` for easy instantiation
|
||||
- `to_dict()` for Firestore storage
|
||||
- `to_legacy_dict()` for API compatibility
|
||||
- `from_dict()` for Firestore retrieval
|
||||
|
||||
2. [x] **Update FirestoreService** - `backend/services/firestore_service.py`
|
||||
- Added `append_log_to_subcollection(job_id, log_entry)` method
|
||||
- Added `get_logs_from_subcollection(job_id, limit, since_timestamp, worker, offset)` method
|
||||
- Added `get_logs_count_from_subcollection(job_id)` method
|
||||
- Added `delete_logs_subcollection(job_id, batch_size)` method
|
||||
- Old `append_worker_log` marked as deprecated
|
||||
|
||||
3. [x] **Update JobManager** - `backend/services/job_manager.py`
|
||||
- Modified `append_worker_log()` to use subcollection when `USE_LOG_SUBCOLLECTION=true`
|
||||
- Modified `get_worker_logs()` to read from subcollection with fallback to embedded array
|
||||
- Added `get_worker_logs_count()` for total count
|
||||
- Updated `delete_job()` to clean up logs subcollection
|
||||
|
||||
4. [x] **Update jobs API** - `backend/api/routes/jobs.py`
|
||||
- `GET /api/jobs/{id}/logs` - Updated to use new methods
|
||||
- Response format stays the same (backward compatible)
|
||||
- Fixed `next_index` calculation
|
||||
|
||||
5. [x] **Add feature flag** - `backend/config.py`
|
||||
- Added `USE_LOG_SUBCOLLECTION` environment variable (default: true)
|
||||
|
||||
6. [x] **Add TTL configuration** - `infrastructure/__main__.py`
|
||||
- Added Firestore TTL policy for logs subcollection (30 days)
|
||||
- Added composite index for worker + timestamp queries
|
||||
|
||||
### Phase 2: Testing
|
||||
|
||||
7. [x] **Unit tests** - `backend/tests/test_worker_log_subcollection.py`
|
||||
- 24 tests covering:
|
||||
- WorkerLogEntry model creation and conversion
|
||||
- FirestoreService subcollection methods
|
||||
- JobManager logging with feature flag
|
||||
- Edge cases (unicode, long messages, empty messages)
|
||||
|
||||
8. [x] **Integration tests** - `backend/tests/emulator/test_worker_logs_subcollection.py`
|
||||
- 11 tests using real Firestore emulator:
|
||||
- Single and sequential writes
|
||||
- TTL field verification
|
||||
- Concurrent writes (no race conditions)
|
||||
- Large volume (500 logs)
|
||||
- Worker filtering
|
||||
- Timestamp ordering
|
||||
- Subcollection deletion
|
||||
- Concurrent workers interleaved
|
||||
- Job deletion cleanup
|
||||
- Comparison with embedded array
|
||||
|
||||
### Phase 3: Deployment & Migration
|
||||
|
||||
9. [ ] **Deploy with feature flag**
|
||||
- Add `USE_LOG_SUBCOLLECTION=true` env var (default is true)
|
||||
- Deploy to production
|
||||
|
||||
10. [ ] **Monitor**
|
||||
- Watch for errors in Cloud Logging
|
||||
- Verify logs appear in frontend
|
||||
|
||||
11. [x] **Backward compatibility**
|
||||
- Old jobs with embedded arrays still work (fallback in get_worker_logs)
|
||||
- No migration needed - new jobs use subcollection, old jobs use array
|
||||
|
||||
### Phase 4: Cleanup (Future)
|
||||
|
||||
12. [ ] **Remove old code path** (after confidence period)
|
||||
- Remove `worker_logs` field from job model
|
||||
- Remove old `append_worker_log` method
|
||||
- Remove feature flag
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/worker_log.py` | Created | WorkerLogEntry dataclass |
|
||||
| `backend/services/firestore_service.py` | Modified | Added subcollection methods |
|
||||
| `backend/services/job_manager.py` | Modified | Switch to subcollection with fallback |
|
||||
| `backend/api/routes/jobs.py` | Modified | Updated logs endpoint |
|
||||
| `backend/config.py` | Modified | Added USE_LOG_SUBCOLLECTION setting |
|
||||
| `backend/tests/test_worker_log_subcollection.py` | Created | Unit tests (24 tests) |
|
||||
| `backend/tests/emulator/test_worker_logs_subcollection.py` | Created | Integration tests (11 tests) |
|
||||
| `infrastructure/__main__.py` | Modified | Added TTL policy and index |
|
||||
|
||||
## Log Entry Schema
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class WorkerLogEntry:
|
||||
timestamp: datetime
|
||||
level: str # "DEBUG", "INFO", "WARNING", "ERROR"
|
||||
worker: str # "audio", "lyrics", "video", "render", "screens", "distribution"
|
||||
message: str
|
||||
id: str # Auto-generated UUID
|
||||
job_id: str
|
||||
ttl_expiry: datetime # timestamp + 30 days
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
```
|
||||
|
||||
## Firestore Structure
|
||||
|
||||
```
|
||||
jobs/
|
||||
{job_id}/
|
||||
... (job fields, worker_logs still present for old jobs)
|
||||
logs/ # subcollection for new jobs
|
||||
{log_id}/
|
||||
id: string
|
||||
job_id: string
|
||||
timestamp: datetime
|
||||
level: string
|
||||
worker: string
|
||||
message: string
|
||||
metadata: map (optional)
|
||||
ttl_expiry: datetime
|
||||
```
|
||||
|
||||
## API Response (Unchanged)
|
||||
|
||||
```json
|
||||
{
|
||||
"logs": [
|
||||
{
|
||||
"timestamp": "2026-01-04T12:00:00Z",
|
||||
"level": "INFO",
|
||||
"worker": "video",
|
||||
"message": "Starting video generation"
|
||||
}
|
||||
],
|
||||
"next_index": 42,
|
||||
"total_logs": 150
|
||||
}
|
||||
```
|
||||
|
||||
## Testing Results
|
||||
|
||||
All tests pass:
|
||||
- 24 unit tests in `test_worker_log_subcollection.py`
|
||||
- 11 emulator integration tests in `test_worker_logs_subcollection.py`
|
||||
- 68 total tests in emulator suite pass
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. Set `USE_LOG_SUBCOLLECTION=false` env var
|
||||
2. Redeploy backend
|
||||
3. New logs will go back to embedded array
|
||||
4. Subcollection logs will remain but won't be read
|
||||
|
||||
## Answers to Open Questions
|
||||
|
||||
- [x] **Should we paginate logs in the API?** - Yes, added `offset` parameter support
|
||||
- [x] **What's the optimal TTL period?** - 30 days (configurable via DEFAULT_LOG_TTL_DAYS)
|
||||
- [x] **Should we index by timestamp for efficient range queries?** - Yes, added composite index (worker + timestamp)
|
||||
|
||||
## References
|
||||
|
||||
- [Firestore TTL](https://cloud.google.com/firestore/docs/ttl)
|
||||
- [Firestore Subcollections](https://firebase.google.com/docs/firestore/data-model#subcollections)
|
||||
- [Firestore Limits](https://firebase.google.com/docs/firestore/quotas#limits)
|
||||
- [Pulumi Firestore Field](https://www.pulumi.com/registry/packages/gcp/api-docs/firestore/field/)
|
||||
@@ -0,0 +1,115 @@
|
||||
# Build Encoding Worker GCE Image
|
||||
#
|
||||
# Rebuilds the custom Packer image when provisioning changes.
|
||||
# The image includes Python 3.13, FFmpeg, fonts, and dependencies
|
||||
# pre-installed to reduce VM startup time from ~10 min to ~30 sec.
|
||||
#
|
||||
# Triggers:
|
||||
# - Push to main with changes to infrastructure/packer/**
|
||||
# - Manual dispatch (workflow_dispatch)
|
||||
#
|
||||
# After image build, the encoding-worker VM will use the new image
|
||||
# on next recreation (e.g., via pulumi up or manual restart).
|
||||
|
||||
name: Build Encoding Worker Image
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'infrastructure/packer/**'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
python_version:
|
||||
description: 'Python version to install'
|
||||
required: false
|
||||
default: '3.13.1'
|
||||
|
||||
concurrency:
|
||||
group: build-encoding-worker-image
|
||||
cancel-in-progress: false # Don't cancel in-progress builds
|
||||
|
||||
jobs:
|
||||
build-image:
|
||||
name: Build Packer Image
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write # Required for Workload Identity Federation
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Install Packer
|
||||
run: |
|
||||
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
|
||||
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
|
||||
sudo apt-get update && sudo apt-get install -y packer
|
||||
|
||||
- name: Initialize Packer
|
||||
working-directory: infrastructure/packer
|
||||
run: packer init encoding-worker.pkr.hcl
|
||||
|
||||
- name: Validate Packer Template
|
||||
working-directory: infrastructure/packer
|
||||
run: packer validate encoding-worker.pkr.hcl
|
||||
|
||||
- name: Build Image
|
||||
working-directory: infrastructure/packer
|
||||
env:
|
||||
PYTHON_VERSION: ${{ github.event.inputs.python_version || '3.13.1' }}
|
||||
run: |
|
||||
echo "Building encoding worker image with Python ${PYTHON_VERSION}..."
|
||||
packer build \
|
||||
-var "python_version=${PYTHON_VERSION}" \
|
||||
encoding-worker.pkr.hcl
|
||||
|
||||
- name: Get Latest Image Info
|
||||
id: image-info
|
||||
run: |
|
||||
IMAGE_NAME=$(gcloud compute images list \
|
||||
--project=nomadkaraoke \
|
||||
--filter="family:encoding-worker" \
|
||||
--sort-by="~creationTimestamp" \
|
||||
--limit=1 \
|
||||
--format="value(name)")
|
||||
echo "image_name=${IMAGE_NAME}" >> $GITHUB_OUTPUT
|
||||
echo "Built image: ${IMAGE_NAME}"
|
||||
|
||||
- name: Summary
|
||||
run: |
|
||||
echo "## Encoding Worker Image Built" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Image:** \`${{ steps.image-info.outputs.image_name }}\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Family:** \`encoding-worker\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Project:** \`nomadkaraoke\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "The encoding-worker VM will use this image on next recreation." >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "To force VM recreation now:" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`bash" >> $GITHUB_STEP_SUMMARY
|
||||
echo "gcloud compute instances delete encoding-worker --zone=us-central1-a --project=nomadkaraoke" >> $GITHUB_STEP_SUMMARY
|
||||
echo "# Then run pulumi up or wait for next infrastructure deploy" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
notify-on-failure:
|
||||
name: Notify on Failure
|
||||
runs-on: ubuntu-latest
|
||||
needs: build-image
|
||||
if: failure()
|
||||
steps:
|
||||
- name: Log Failure
|
||||
run: |
|
||||
echo "::error::Encoding worker image build failed. Check the build-image job logs."
|
||||
+500
-51
@@ -4,6 +4,7 @@ on:
|
||||
push:
|
||||
pull_request:
|
||||
branches: [main]
|
||||
types: [opened, synchronize, reopened, labeled]
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
@@ -97,6 +98,156 @@ jobs:
|
||||
path: frontend/coverage/
|
||||
retention-days: 7
|
||||
|
||||
frontend-e2e-smoke:
|
||||
name: "Frontend - E2E Smoke (~2 min)"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
# Quick smoke tests run on every PR with frontend changes
|
||||
if: github.ref != 'refs/heads/main' && needs.changes.outputs.frontend == 'true'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js with caching
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Install Playwright browsers
|
||||
working-directory: frontend
|
||||
run: npx playwright install --with-deps chromium
|
||||
|
||||
- name: Run smoke tests only
|
||||
working-directory: frontend
|
||||
run: npx playwright test smoke.spec.ts --project=chromium --reporter=line
|
||||
|
||||
- name: Upload Playwright report
|
||||
if: failure()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: playwright-smoke-report
|
||||
path: frontend/playwright-report/
|
||||
retention-days: 7
|
||||
|
||||
frontend-e2e-full:
|
||||
name: "Frontend - E2E Full"
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
needs: changes
|
||||
# Full E2E suite only runs when PR has 'e2e' label (on-demand)
|
||||
if: >-
|
||||
github.ref != 'refs/heads/main' &&
|
||||
needs.changes.outputs.frontend == 'true' &&
|
||||
contains(github.event.pull_request.labels.*.name, 'e2e')
|
||||
steps:
|
||||
- name: Check disk space
|
||||
run: |
|
||||
echo "Checking disk space..."
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "Current disk usage: ${USAGE}%"
|
||||
if [ "$USAGE" -gt 70 ]; then
|
||||
echo "Disk usage high, running cleanup..."
|
||||
docker system prune -af || true
|
||||
docker builder prune -af || true
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "After cleanup: ${USAGE}%"
|
||||
fi
|
||||
if [ "$USAGE" -gt 85 ]; then
|
||||
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js with caching
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Install Playwright browsers
|
||||
working-directory: frontend
|
||||
run: npx playwright install --with-deps chromium
|
||||
|
||||
- name: Run full E2E tests
|
||||
working-directory: frontend
|
||||
run: npx playwright test --reporter=list -x
|
||||
|
||||
- name: Upload Playwright report
|
||||
if: failure()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: playwright-report
|
||||
path: frontend/playwright-report/
|
||||
retention-days: 7
|
||||
|
||||
# Happy Path Production Test - runs the comprehensive real-user test against prod
|
||||
frontend-e2e-happy-path:
|
||||
name: "Frontend - E2E Happy Path (Production)"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
# Only runs when PR has 'e2e-happy-path' label
|
||||
if: >-
|
||||
github.ref != 'refs/heads/main' &&
|
||||
contains(github.event.pull_request.labels.*.name, 'e2e-happy-path')
|
||||
timeout-minutes: 90
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Install Playwright browsers
|
||||
working-directory: frontend
|
||||
run: npx playwright install --with-deps chromium
|
||||
|
||||
- name: Create test-results directory
|
||||
working-directory: frontend
|
||||
run: mkdir -p test-results
|
||||
|
||||
- name: Run Happy Path Production Test
|
||||
working-directory: frontend
|
||||
env:
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
E2E_TEST_TOKEN: ${{ secrets.E2E_TEST_TOKEN }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
run: |
|
||||
echo "Starting E2E Happy Path Test against Production"
|
||||
echo "Time: $(date)"
|
||||
npx playwright test happy-path-real-user.spec.ts \
|
||||
--config=playwright.production.config.ts \
|
||||
--project=chromium \
|
||||
--reporter=list \
|
||||
--timeout=3600000
|
||||
|
||||
- name: Upload test artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-results
|
||||
path: |
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
retention-days: 30
|
||||
|
||||
frontend-build:
|
||||
name: "Frontend - Build"
|
||||
runs-on: ubuntu-latest
|
||||
@@ -128,10 +279,27 @@ jobs:
|
||||
|
||||
package-unit-tests:
|
||||
name: "Package - Unit Tests (70% coverage)"
|
||||
runs-on: ubuntu-latest
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- name: Check disk space
|
||||
run: |
|
||||
echo "Checking disk space..."
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "Current disk usage: ${USAGE}%"
|
||||
if [ "$USAGE" -gt 70 ]; then
|
||||
echo "Disk usage high, running cleanup..."
|
||||
docker system prune -af || true
|
||||
docker builder prune -af || true
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "After cleanup: ${USAGE}%"
|
||||
fi
|
||||
if [ "$USAGE" -gt 85 ]; then
|
||||
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
@@ -186,10 +354,27 @@ jobs:
|
||||
|
||||
package-integration-tests:
|
||||
name: "Package - Integration Tests"
|
||||
runs-on: ubuntu-latest
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- name: Check disk space
|
||||
run: |
|
||||
echo "Checking disk space..."
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "Current disk usage: ${USAGE}%"
|
||||
if [ "$USAGE" -gt 70 ]; then
|
||||
echo "Disk usage high, running cleanup..."
|
||||
docker system prune -af || true
|
||||
docker builder prune -af || true
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "After cleanup: ${USAGE}%"
|
||||
fi
|
||||
if [ "$USAGE" -gt 85 ]; then
|
||||
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
@@ -243,6 +428,23 @@ jobs:
|
||||
(needs.package-integration-tests.result == 'success' || needs.package-integration-tests.result == 'skipped') &&
|
||||
(needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- name: Check disk space
|
||||
run: |
|
||||
echo "Checking disk space..."
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "Current disk usage: ${USAGE}%"
|
||||
if [ "$USAGE" -gt 70 ]; then
|
||||
echo "Disk usage high, running cleanup..."
|
||||
docker system prune -af || true
|
||||
docker builder prune -af || true
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "After cleanup: ${USAGE}%"
|
||||
fi
|
||||
if [ "$USAGE" -gt 85 ]; then
|
||||
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
@@ -276,6 +478,23 @@ jobs:
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- name: Check disk space
|
||||
run: |
|
||||
echo "Checking disk space..."
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "Current disk usage: ${USAGE}%"
|
||||
if [ "$USAGE" -gt 70 ]; then
|
||||
echo "Disk usage high, running cleanup..."
|
||||
docker system prune -af || true
|
||||
docker builder prune -af || true
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "After cleanup: ${USAGE}%"
|
||||
fi
|
||||
if [ "$USAGE" -gt 85 ]; then
|
||||
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -313,6 +532,23 @@ jobs:
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true' || needs.changes.outputs.deps == 'true')
|
||||
steps:
|
||||
- name: Check disk space
|
||||
run: |
|
||||
echo "Checking disk space..."
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "Current disk usage: ${USAGE}%"
|
||||
if [ "$USAGE" -gt 70 ]; then
|
||||
echo "Disk usage high, running cleanup..."
|
||||
docker system prune -af || true
|
||||
docker builder prune -af || true
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "After cleanup: ${USAGE}%"
|
||||
fi
|
||||
if [ "$USAGE" -gt 85 ]; then
|
||||
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -356,10 +592,27 @@ jobs:
|
||||
|
||||
backend-lint:
|
||||
name: "Backend - Code Quality"
|
||||
runs-on: ubuntu-latest
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && (needs.changes.outputs.backend == 'true' || needs.changes.outputs.package == 'true')
|
||||
steps:
|
||||
- name: Check disk space
|
||||
run: |
|
||||
echo "Checking disk space..."
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "Current disk usage: ${USAGE}%"
|
||||
if [ "$USAGE" -gt 70 ]; then
|
||||
echo "Disk usage high, running cleanup..."
|
||||
docker system prune -af || true
|
||||
docker builder prune -af || true
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "After cleanup: ${USAGE}%"
|
||||
fi
|
||||
if [ "$USAGE" -gt 85 ]; then
|
||||
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -400,13 +653,16 @@ jobs:
|
||||
ci-gate:
|
||||
name: "CI Gate"
|
||||
runs-on: ubuntu-latest
|
||||
needs: [frontend-unit-tests, frontend-build, package-unit-tests, package-integration-tests, backend-unit-tests, backend-emulator-tests, backend-lint]
|
||||
# Note: frontend-e2e-full is optional (requires 'e2e' label) so not included here
|
||||
# frontend-e2e-smoke IS required - fast smoke tests catch critical regressions
|
||||
needs: [frontend-unit-tests, frontend-e2e-smoke, frontend-build, package-unit-tests, package-integration-tests, backend-unit-tests, backend-emulator-tests, backend-lint]
|
||||
if: github.ref != 'refs/heads/main' && always()
|
||||
steps:
|
||||
- name: Check results
|
||||
run: |
|
||||
results=(
|
||||
"${{ needs.frontend-unit-tests.result }}"
|
||||
"${{ needs.frontend-e2e-smoke.result }}"
|
||||
"${{ needs.frontend-build.result }}"
|
||||
"${{ needs.package-unit-tests.result }}"
|
||||
"${{ needs.package-integration-tests.result }}"
|
||||
@@ -429,9 +685,26 @@ jobs:
|
||||
|
||||
package-build:
|
||||
name: "Package - Build"
|
||||
runs-on: ubuntu-latest
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
steps:
|
||||
- name: Check disk space
|
||||
run: |
|
||||
echo "Checking disk space..."
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "Current disk usage: ${USAGE}%"
|
||||
if [ "$USAGE" -gt 70 ]; then
|
||||
echo "Disk usage high, running cleanup..."
|
||||
docker system prune -af || true
|
||||
docker builder prune -af || true
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "After cleanup: ${USAGE}%"
|
||||
fi
|
||||
if [ "$USAGE" -gt 85 ]; then
|
||||
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
@@ -624,13 +897,20 @@ jobs:
|
||||
id: deployment
|
||||
uses: actions/deploy-pages@v4
|
||||
|
||||
# NOTE: buy-site was consolidated into the main frontend (PR #122, Dec 2024)
|
||||
# Landing page and payment flow are now at gen.nomadkaraoke.com (root)
|
||||
# buy.nomadkaraoke.com subdomain has been deleted
|
||||
|
||||
deploy-backend:
|
||||
name: "Deploy - Backend (Cloud Run)"
|
||||
needs: [package-build]
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
outputs:
|
||||
base_rebuilt: ${{ steps.check-base.outputs.needs_rebuild }}
|
||||
env:
|
||||
REGISTRY: us-central1-docker.pkg.dev
|
||||
PROJECT_ID: nomadkaraoke
|
||||
@@ -638,14 +918,27 @@ jobs:
|
||||
BASE_IMAGE_NAME: karaoke-backend-base
|
||||
APP_IMAGE_NAME: karaoke-backend
|
||||
steps:
|
||||
# ============================================
|
||||
# Check if deployment is needed FIRST
|
||||
# ============================================
|
||||
- name: Check disk space
|
||||
run: |
|
||||
echo "Checking disk space..."
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "Current disk usage: ${USAGE}%"
|
||||
if [ "$USAGE" -gt 70 ]; then
|
||||
echo "Disk usage high, running cleanup..."
|
||||
docker system prune -af || true
|
||||
docker builder prune -af || true
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "After cleanup: ${USAGE}%"
|
||||
fi
|
||||
if [ "$USAGE" -gt 85 ]; then
|
||||
echo "::error::Disk usage ${USAGE}% exceeds 85% even after cleanup. Runner needs maintenance."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: false
|
||||
fetch-depth: 0 # Need full history for tag check
|
||||
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
@@ -655,26 +948,15 @@ jobs:
|
||||
- name: Install Poetry
|
||||
uses: snok/install-poetry@v1
|
||||
|
||||
- name: Extract version and check if deployment needed
|
||||
- name: Extract version
|
||||
id: version_check
|
||||
run: |
|
||||
VERSION=$(poetry version --short)
|
||||
echo "version=$VERSION" >> $GITHUB_OUTPUT
|
||||
echo "📋 Current version: $VERSION"
|
||||
|
||||
if git ls-remote --tags origin | grep -q "refs/tags/v$VERSION$"; then
|
||||
echo "needs_deploy=false" >> $GITHUB_OUTPUT
|
||||
echo "⏭️ Tag v$VERSION already exists - skipping backend deployment"
|
||||
echo " (Bump version in pyproject.toml to trigger a new deployment)"
|
||||
else
|
||||
echo "needs_deploy=true" >> $GITHUB_OUTPUT
|
||||
echo "✨ Tag v$VERSION does not exist - proceeding with deployment"
|
||||
fi
|
||||
|
||||
# No disk cleanup needed - self-hosted runner has 200GB SSD
|
||||
echo "🚀 Deploying on every push to main"
|
||||
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
id: auth
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
@@ -682,13 +964,37 @@ jobs:
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
# ============================================
|
||||
# Upload wheel to GCS for GCE encoding worker
|
||||
# The GCE worker installs this wheel at job start for hot code updates
|
||||
# ============================================
|
||||
|
||||
- name: Download wheel artifact
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
name: package-dist
|
||||
path: dist/
|
||||
|
||||
- name: Upload wheel to GCS for GCE worker
|
||||
run: |
|
||||
echo "📦 Uploading wheel to GCS for GCE encoding worker..."
|
||||
WHEEL_FILE=$(ls dist/*.whl | head -1)
|
||||
if [ -z "$WHEEL_FILE" ]; then
|
||||
echo "::error::No wheel file found in dist/"
|
||||
exit 1
|
||||
fi
|
||||
echo "Found wheel: $WHEEL_FILE"
|
||||
|
||||
# Upload to GCS (GCE worker downloads latest at job start)
|
||||
gsutil cp "$WHEEL_FILE" gs://karaoke-gen-storage-${{ env.PROJECT_ID }}/wheels/
|
||||
|
||||
echo "✅ Wheel uploaded - GCE worker will pick up changes on next job"
|
||||
|
||||
- name: Configure Docker for Artifact Registry
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
|
||||
|
||||
# ============================================
|
||||
@@ -698,11 +1004,9 @@ jobs:
|
||||
# ============================================
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Calculate base image hash
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
id: base-hash
|
||||
run: |
|
||||
# Hash both poetry.lock AND Dockerfile.base - rebuild if either changes
|
||||
@@ -711,26 +1015,36 @@ jobs:
|
||||
echo "📋 Base content hash: $CURRENT_HASH"
|
||||
|
||||
- name: Check if base image exists and is current
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
id: check-base
|
||||
run: |
|
||||
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
|
||||
|
||||
# Try to get the existing base image's hash label
|
||||
# Fetch hash label directly from registry (10KB config blob, not full image)
|
||||
# This avoids issues with stale local Docker cache on GitHub runners
|
||||
EXISTING_HASH=""
|
||||
if docker pull $BASE_IMAGE 2>/dev/null; then
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
|
||||
# Fallback to old label name for backwards compatibility
|
||||
if [ -z "$EXISTING_HASH" ]; then
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "poetry.lock.hash"}}' 2>/dev/null || echo "")
|
||||
fi
|
||||
echo "Existing base image hash: $EXISTING_HASH"
|
||||
|
||||
# Get the manifest (small JSON, ~1KB) - includes config digest
|
||||
MANIFEST=$(docker manifest inspect $BASE_IMAGE 2>/dev/null || echo "")
|
||||
CONFIG_DIGEST=$(echo "$MANIFEST" | jq -r '.config.digest // ""' 2>/dev/null || echo "")
|
||||
|
||||
if [ -n "$CONFIG_DIGEST" ] && [ "$CONFIG_DIGEST" != "null" ] && [ "$CONFIG_DIGEST" != "" ]; then
|
||||
echo "📋 Found manifest, config digest: $CONFIG_DIGEST"
|
||||
|
||||
# Fetch just the config blob (~10KB) to get labels - much faster than pulling full image
|
||||
TOKEN=$(gcloud auth print-access-token)
|
||||
CONFIG_JSON=$(curl -sL -H "Authorization: Bearer $TOKEN" \
|
||||
"https://${{ env.REGISTRY }}/v2/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}/blobs/$CONFIG_DIGEST" 2>/dev/null)
|
||||
|
||||
EXISTING_HASH=$(echo "$CONFIG_JSON" | jq -r '.config.Labels["base.content.hash"] // .config.Labels["poetry.lock.hash"] // ""' 2>/dev/null || echo "")
|
||||
echo "📋 Registry base image hash: $EXISTING_HASH"
|
||||
else
|
||||
echo "⚠️ No existing base image found - will build"
|
||||
echo "⚠️ No existing base image found in registry - will build"
|
||||
fi
|
||||
|
||||
if [ "${{ steps.base-hash.outputs.hash }}" != "$EXISTING_HASH" ]; then
|
||||
echo "🔄 Base image needs rebuild (poetry.lock or Dockerfile.base changed)"
|
||||
echo " Current: ${{ steps.base-hash.outputs.hash }}"
|
||||
echo " Registry: $EXISTING_HASH"
|
||||
echo "needs_rebuild=true" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "✅ Base image is up to date"
|
||||
@@ -738,7 +1052,7 @@ jobs:
|
||||
fi
|
||||
|
||||
- name: Build and push base image (if needed)
|
||||
if: steps.version_check.outputs.needs_deploy == 'true' && steps.check-base.outputs.needs_rebuild == 'true'
|
||||
if: steps.check-base.outputs.needs_rebuild == 'true'
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
@@ -758,8 +1072,32 @@ jobs:
|
||||
# Provenance disabled for smaller image and faster push
|
||||
provenance: false
|
||||
|
||||
- name: Ensure local base image is current (if not rebuilt)
|
||||
if: steps.check-base.outputs.needs_rebuild == 'false'
|
||||
run: |
|
||||
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
|
||||
|
||||
# Get the config digest from registry manifest (identifies the image content)
|
||||
# This is the same value as Docker's Image ID for the same image
|
||||
REGISTRY_CONFIG_DIGEST=$(docker manifest inspect $BASE_IMAGE 2>/dev/null | jq -r '.config.digest' || echo "")
|
||||
|
||||
# Get local image's ID (which is the config digest)
|
||||
LOCAL_IMAGE_ID=$(docker image inspect $BASE_IMAGE --format='{{.Id}}' 2>/dev/null || echo "")
|
||||
|
||||
echo "Registry config digest: $REGISTRY_CONFIG_DIGEST"
|
||||
echo "Local image ID: $LOCAL_IMAGE_ID"
|
||||
|
||||
if [ -z "$LOCAL_IMAGE_ID" ]; then
|
||||
echo "📥 Base image not in local cache - pulling from registry"
|
||||
docker pull $BASE_IMAGE
|
||||
elif [ "$LOCAL_IMAGE_ID" != "$REGISTRY_CONFIG_DIGEST" ]; then
|
||||
echo "🔄 Local cache has different image - pulling latest"
|
||||
docker pull $BASE_IMAGE
|
||||
else
|
||||
echo "✅ Local cache matches registry - skipping pull"
|
||||
fi
|
||||
|
||||
- name: Build and push app image
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
@@ -780,7 +1118,6 @@ jobs:
|
||||
# ============================================
|
||||
|
||||
- name: Deploy to Cloud Run
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
run: |
|
||||
VERSION="${{ steps.version_check.outputs.version }}"
|
||||
IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.APP_IMAGE_NAME }}:${{ github.sha }}"
|
||||
@@ -801,12 +1138,11 @@ jobs:
|
||||
--timeout 1800 \
|
||||
--concurrency 1 \
|
||||
--max-instances 50 \
|
||||
--min-instances 0 \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD" \
|
||||
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest"
|
||||
--min-instances 4 \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true" \
|
||||
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,SENDGRID_API_KEY=sendgrid-api-key:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest"
|
||||
|
||||
- name: Verify deployment and version
|
||||
if: steps.version_check.outputs.needs_deploy == 'true'
|
||||
run: |
|
||||
EXPECTED_VERSION="${{ steps.version_check.outputs.version }}"
|
||||
SERVICE_URL=$(gcloud run services describe karaoke-backend \
|
||||
@@ -847,7 +1183,7 @@ jobs:
|
||||
done
|
||||
|
||||
- name: Summary
|
||||
if: steps.version_check.outputs.needs_deploy == 'true' && success()
|
||||
if: success()
|
||||
run: |
|
||||
echo "🎉 Deployment successful!"
|
||||
echo "Version: ${{ steps.version_check.outputs.version }}"
|
||||
@@ -858,11 +1194,124 @@ jobs:
|
||||
echo "📊 Built on self-hosted GCP runner (200GB SSD, no disk space issues)"
|
||||
echo " Runner: gcp-runner with 4 vCPU, 16GB RAM"
|
||||
|
||||
- name: Skipped deployment summary
|
||||
if: steps.version_check.outputs.needs_deploy == 'false'
|
||||
# ============================================
|
||||
# PROPAGATE BASE IMAGE TO ALL RUNNERS
|
||||
# When base image is rebuilt, pull it to all 20 runners
|
||||
# so subsequent app builds don't wait 8 min for the pull
|
||||
#
|
||||
# NOTE: This is best-effort distribution. GitHub Actions doesn't
|
||||
# support targeting specific runners by name, so we spawn 20 jobs
|
||||
# and rely on the scheduler to distribute them across idle runners.
|
||||
# In practice, with all runners idle after deploy-backend, distribution
|
||||
# is good. Runners that miss this pull will get the image on their
|
||||
# next app build (with the 8 min delay).
|
||||
# ============================================
|
||||
|
||||
propagate-base-image:
|
||||
name: "Propagate Base Image (${{ matrix.runner_index }})"
|
||||
needs: deploy-backend
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push' && needs.deploy-backend.outputs.base_rebuilt == 'true'
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
strategy:
|
||||
matrix:
|
||||
# 20 parallel jobs - GitHub distributes across available runners
|
||||
runner_index: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
|
||||
max-parallel: 20
|
||||
fail-fast: false # Don't cancel other runners if one fails
|
||||
env:
|
||||
REGISTRY: us-central1-docker.pkg.dev
|
||||
PROJECT_ID: nomadkaraoke
|
||||
REPOSITORY: karaoke-repo
|
||||
BASE_IMAGE_NAME: karaoke-backend-base
|
||||
steps:
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Configure Docker for Artifact Registry
|
||||
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
|
||||
|
||||
- name: Pull latest base image
|
||||
run: |
|
||||
echo "⏭️ Backend deployment skipped - version ${{ steps.version_check.outputs.version }} already deployed"
|
||||
echo ""
|
||||
echo "To deploy a new version:"
|
||||
echo " 1. Update version in pyproject.toml"
|
||||
echo " 2. Commit and push to main"
|
||||
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
|
||||
RUNNER_NAME=$(hostname)
|
||||
echo "📥 Pulling latest base image on runner $RUNNER_NAME..."
|
||||
docker pull $BASE_IMAGE
|
||||
echo "✅ Base image cached on $RUNNER_NAME"
|
||||
|
||||
deploy-infrastructure:
|
||||
name: "Deploy - Infrastructure (Pulumi)"
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.13'
|
||||
|
||||
- name: Install Pulumi
|
||||
run: |
|
||||
curl -fsSL https://get.pulumi.com | sh
|
||||
echo "$HOME/.pulumi/bin" >> $GITHUB_PATH
|
||||
|
||||
- name: Install Python dependencies
|
||||
working-directory: infrastructure
|
||||
run: |
|
||||
pip install pulumi pulumi-gcp pulumi-docker
|
||||
|
||||
- name: Configure Pulumi
|
||||
env:
|
||||
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
|
||||
run: |
|
||||
pulumi login
|
||||
|
||||
- name: Deploy Infrastructure
|
||||
working-directory: infrastructure
|
||||
env:
|
||||
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
|
||||
run: |
|
||||
echo "🚀 Deploying infrastructure changes..."
|
||||
# Skip preview to avoid needing broad read permissions on all GCP resources
|
||||
# Pulumi state is authoritative since infrastructure is only managed via Pulumi
|
||||
pulumi up --yes --skip-preview --stack nomadkaraoke/karaoke-gen-infrastructure/prod
|
||||
|
||||
- name: Restart GCE Encoding Worker
|
||||
run: |
|
||||
echo "🔄 Restarting encoding worker to pick up new code..."
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
|
||||
--command="sudo systemctl restart encoding-worker" || true
|
||||
echo "✅ Encoding worker restart triggered"
|
||||
|
||||
- name: Summary
|
||||
if: success()
|
||||
run: |
|
||||
echo "🎉 Infrastructure deployment successful!"
|
||||
echo "Commit: ${{ github.sha }}"
|
||||
echo "Branch: ${{ github.ref_name }}"
|
||||
@@ -0,0 +1,173 @@
|
||||
name: E2E Happy Path Test
|
||||
|
||||
# Comprehensive end-to-end test that validates the full karaoke generation journey
|
||||
# using ONLY browser interactions (no API shortcuts)
|
||||
#
|
||||
# See docs/E2E-HAPPY-PATH-TEST-SPEC.md for full requirements
|
||||
|
||||
on:
|
||||
# Daily run at 6 AM UTC
|
||||
schedule:
|
||||
- cron: '0 6 * * *'
|
||||
|
||||
# Manual trigger with optional parameters
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
debug_mode:
|
||||
description: 'Enable debug mode (slower, more screenshots)'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
browser:
|
||||
description: 'Browser to test with'
|
||||
required: false
|
||||
default: 'chromium'
|
||||
type: choice
|
||||
options:
|
||||
- chromium
|
||||
- firefox
|
||||
- webkit
|
||||
use_test_token:
|
||||
description: 'Skip signup and use E2E_TEST_TOKEN (for iterations)'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
e2e-happy-path:
|
||||
name: "Full Happy Path (Real User)"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 90 # 1.5 hours max (test itself is 60 min)
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Install Playwright browsers
|
||||
working-directory: frontend
|
||||
run: |
|
||||
BROWSER=${{ inputs.browser || 'chromium' }}
|
||||
npx playwright install --with-deps $BROWSER
|
||||
|
||||
- name: Create test-results directory
|
||||
working-directory: frontend
|
||||
run: mkdir -p test-results
|
||||
|
||||
- name: Run E2E Happy Path Test
|
||||
working-directory: frontend
|
||||
env:
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
E2E_TEST_TOKEN: ${{ inputs.use_test_token && secrets.E2E_TEST_TOKEN || '' }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
|
||||
run: |
|
||||
echo "Starting E2E Happy Path Test"
|
||||
echo "Browser: ${{ inputs.browser || 'chromium' }}"
|
||||
echo "Debug mode: ${{ inputs.debug_mode || 'false' }}"
|
||||
echo "Use test token: ${{ inputs.use_test_token || 'false' }}"
|
||||
echo "Time: $(date)"
|
||||
echo ""
|
||||
|
||||
# Run the real user happy path test
|
||||
npx playwright test happy-path-real-user.spec.ts \
|
||||
--config=playwright.production.config.ts \
|
||||
--project=${{ inputs.browser || 'chromium' }} \
|
||||
--reporter=list \
|
||||
--timeout=3600000 \
|
||||
2>&1 | tee test-output.log
|
||||
continue-on-error: true
|
||||
|
||||
- name: Check test result
|
||||
id: test_result
|
||||
working-directory: frontend
|
||||
run: |
|
||||
if grep -q "passed" test-output.log && ! grep -q "failed" test-output.log; then
|
||||
echo "result=success" >> $GITHUB_OUTPUT
|
||||
echo "Test PASSED"
|
||||
else
|
||||
echo "result=failure" >> $GITHUB_OUTPUT
|
||||
echo "Test FAILED - see artifacts for details"
|
||||
fi
|
||||
|
||||
- name: Upload test artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-results-${{ github.run_number }}
|
||||
path: |
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
frontend/test-output.log
|
||||
retention-days: 30
|
||||
|
||||
- name: Upload video recordings
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-videos-${{ github.run_number }}
|
||||
path: frontend/test-results/**/*.webm
|
||||
retention-days: 30
|
||||
if-no-files-found: ignore
|
||||
|
||||
- name: Create summary
|
||||
if: always()
|
||||
run: |
|
||||
echo "## E2E Happy Path Test Results" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Result | ${{ steps.test_result.outputs.result || 'unknown' }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
if [ -f frontend/test-output.log ]; then
|
||||
echo "### Test Output (last 100 lines)" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
tail -100 frontend/test-output.log >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
fi
|
||||
|
||||
- name: Send email notification
|
||||
if: always() && github.event_name == 'schedule'
|
||||
uses: dawidd6/action-send-mail@v3
|
||||
with:
|
||||
server_address: smtp.sendgrid.net
|
||||
server_port: 587
|
||||
username: apikey
|
||||
password: ${{ secrets.SENDGRID_API_KEY }}
|
||||
subject: "E2E Happy Path Test ${{ steps.test_result.outputs.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
|
||||
to: andrew@nomadkaraoke.com
|
||||
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
|
||||
body: |
|
||||
E2E Happy Path Test Results
|
||||
===========================
|
||||
|
||||
Status: ${{ steps.test_result.outputs.result == 'success' && 'PASSED ✅' || 'FAILED ❌' }}
|
||||
Run: #${{ github.run_number }}
|
||||
Trigger: ${{ github.event_name }}
|
||||
Time: ${{ github.event.repository.updated_at }}
|
||||
|
||||
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||
|
||||
This is an automated notification from the daily E2E happy path test.
|
||||
|
||||
- name: Fail if test failed
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
run: exit 1
|
||||
+3
-1
@@ -47,6 +47,7 @@ frontend/coverage/
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
frontend/playwright/.cache/
|
||||
frontend/e2e/screenshots/
|
||||
|
||||
# TypeScript
|
||||
frontend/*.tsbuildinfo
|
||||
@@ -82,8 +83,9 @@ eggs/
|
||||
# Python lib directories (not frontend)
|
||||
lib/
|
||||
lib64/
|
||||
# But allow frontend lib
|
||||
# But allow frontend lib directories
|
||||
!frontend/lib/
|
||||
!buy-site/lib/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
|
||||
@@ -12,11 +12,14 @@
|
||||
|
||||
| What | Where |
|
||||
|------|-------|
|
||||
| Product vision & goals | `docs/PRODUCT-VISION.md` |
|
||||
| Current status | `docs/README.md` |
|
||||
| Architecture | `docs/ARCHITECTURE.md` |
|
||||
| Dev setup & testing | `docs/DEVELOPMENT.md` |
|
||||
| API reference | `docs/API.md` |
|
||||
| Past learnings | `docs/LESSONS-LEARNED.md` |
|
||||
| **Product communication** | `docs/PRODUCT-COMMUNICATION-GUIDE.md` |
|
||||
| Brand style guide | `docs/BRAND-STYLE-GUIDE.md` |
|
||||
|
||||
## Tech Stack
|
||||
|
||||
@@ -28,11 +31,8 @@
|
||||
## Essential Rules
|
||||
|
||||
### Git Workflow
|
||||
- **Never commit directly to `main`** - always use feature branches
|
||||
- **Use git worktrees** for all changes:
|
||||
```bash
|
||||
git worktree add -b feature/xyz ../karaoke-gen-feat-xyz main
|
||||
```
|
||||
- **Never commit directly to `main`** - use `/new-worktree <description>` to start
|
||||
- **Follow global workflow** - see `~/.claude/CLAUDE.md` for command sequence
|
||||
- Create PR with summary, changes, testing info
|
||||
- Merge only after CI passes
|
||||
|
||||
@@ -50,10 +50,58 @@ cd frontend && npm run test:all 2>&1 | tail -n 200
|
||||
- Skip for docs-only changes
|
||||
|
||||
### Infrastructure
|
||||
- All GCP changes via Pulumi in `infrastructure/`
|
||||
- `gcloud` CLI for reading/debugging only
|
||||
- **All GCP changes via Pulumi PRs** - changes in `infrastructure/` deploy automatically on merge to main
|
||||
- Never modify GCP resources directly via console or `gcloud` CLI
|
||||
- `gcloud` CLI for reading/debugging only (e.g., checking logs, SSH to VMs)
|
||||
- Local `pulumi preview` works but `pulumi up` requires broader permissions than CI
|
||||
- Stop and notify user on auth issues
|
||||
|
||||
## API Authentication (for Agents)
|
||||
|
||||
When you need to call backend APIs programmatically:
|
||||
|
||||
### Internal Worker Endpoints
|
||||
Use the `X-Admin-Token` header with the token from Secret Manager:
|
||||
```bash
|
||||
# Trigger video worker
|
||||
curl -X POST "https://api.nomadkaraoke.com/api/internal/workers/video" \
|
||||
-H "X-Admin-Token: $(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke)" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"job_id": "YOUR_JOB_ID"}'
|
||||
|
||||
# Other internal endpoints follow the same pattern
|
||||
```
|
||||
|
||||
### Firestore Direct Access
|
||||
```bash
|
||||
# Set the correct project
|
||||
export GOOGLE_CLOUD_PROJECT=nomadkaraoke
|
||||
|
||||
# Use Python with google-cloud-firestore
|
||||
python3 << 'EOF'
|
||||
import os
|
||||
os.environ['GOOGLE_CLOUD_PROJECT'] = 'nomadkaraoke'
|
||||
from google.cloud import firestore
|
||||
db = firestore.Client(project='nomadkaraoke')
|
||||
# Query/update documents...
|
||||
EOF
|
||||
```
|
||||
|
||||
### GCE Encoding Worker
|
||||
```bash
|
||||
# SSH to restart service (clears in-memory job queue)
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
|
||||
--command="sudo systemctl restart encoding-worker"
|
||||
|
||||
# Check health
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
|
||||
--command="curl -s http://localhost:8080/health"
|
||||
```
|
||||
|
||||
### What Doesn't Work
|
||||
- `gcloud auth print-identity-token` - Wrong token type for internal endpoints
|
||||
- Cloud Tasks without proper service account - Gets 401 from Cloud Run
|
||||
|
||||
## Documentation Maintenance
|
||||
|
||||
### Structure
|
||||
@@ -89,9 +137,12 @@ Location: `/Users/andrew/Projects/flacfetch`
|
||||
|
||||
Workflow: Make changes on `main`, bump version, push, wait for CI, then `poetry update flacfetch` in karaoke-gen.
|
||||
|
||||
## Pre-Commit Checklist
|
||||
## Pre-PR Checklist
|
||||
|
||||
- [ ] Tests pass (`make test` / `npm run test:all`)
|
||||
Follow global workflow (`~/.claude/CLAUDE.md`):
|
||||
|
||||
- [ ] Tests pass: `/test` (or `make test` / `npm run test:all`)
|
||||
- [ ] Version bumped (if code changed)
|
||||
- [ ] `/docs-review` run (check for doc updates)
|
||||
- [ ] PR created with clear description
|
||||
- [ ] Docs checked: `/docs-review`
|
||||
- [ ] Code reviewed: `/review` (CodeRabbit CLI)
|
||||
- [ ] PR created: `/pr` (adds @coderabbitai ignore)
|
||||
@@ -0,0 +1,75 @@
|
||||
[
|
||||
{
|
||||
"test_name": "Sound-alike error (wurld -> world)",
|
||||
"success": true,
|
||||
"proposals": [
|
||||
{
|
||||
"action": "ReplaceWord",
|
||||
"word_id": null,
|
||||
"word_ids": [
|
||||
"w1"
|
||||
],
|
||||
"replacement_text": "world",
|
||||
"confidence": 0.75,
|
||||
"reason": "Sound-alike error. Reference suggests: 'world'. The transcription 'wurld' is a phonetic misspelling or sound-alike of the word 'world' which appears in both reference sources (Genius and LRCLIB).",
|
||||
"gap_category": "GapCategory.SOUND_ALIKE",
|
||||
"requires_human_review": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"test_name": "Punctuation difference",
|
||||
"success": true,
|
||||
"proposals": [
|
||||
{
|
||||
"action": "NoAction",
|
||||
"word_id": null,
|
||||
"word_ids": [
|
||||
"w2"
|
||||
],
|
||||
"replacement_text": null,
|
||||
"confidence": 0.99,
|
||||
"reason": "Transcription matches at least one reference source. The gap text 'Hello' followed by the context 'is it me you're looking for' matches the Genius reference lyrics exactly, excluding punctuation. According to the guidelines, if any reference source matches the transcription text (ignoring punctuation), it should be classified as NO_ERROR.",
|
||||
"gap_category": "GapCategory.NO_ERROR",
|
||||
"requires_human_review": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"test_name": "Extra filler word",
|
||||
"success": true,
|
||||
"proposals": [
|
||||
{
|
||||
"action": "DeleteWord",
|
||||
"word_id": "w3",
|
||||
"word_ids": null,
|
||||
"replacement_text": null,
|
||||
"confidence": 0.8,
|
||||
"reason": "Extra filler word 'And' at sentence start not in reference. The transcription includes the filler word 'And' at the beginning of the phrase, which is not present in the reference lyrics. This matches the definition of EXTRA_WORDS.",
|
||||
"gap_category": "GapCategory.EXTRA_WORDS",
|
||||
"requires_human_review": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"test_name": "Complex multi-word error",
|
||||
"success": true,
|
||||
"proposals": [
|
||||
{
|
||||
"action": "NoAction",
|
||||
"word_id": null,
|
||||
"word_ids": [
|
||||
"w7",
|
||||
"w8",
|
||||
"w9",
|
||||
"w10"
|
||||
],
|
||||
"replacement_text": null,
|
||||
"confidence": 0.99,
|
||||
"reason": "Transcription matches at least one reference source. The transcription text 'gonna give you up' matches the Genius reference lyrics exactly when ignoring punctuation and capitalization. In context, the full phrase 'never gonna give you up' is correctly transcribed and aligns perfectly with the known lyrics and title of the song.",
|
||||
"gap_category": "GapCategory.NO_ERROR",
|
||||
"requires_human_review": false
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
+30
-1
@@ -10,14 +10,43 @@ FROM python:3.11-slim
|
||||
# Note: Transmission daemon removed - torrent downloads now handled by
|
||||
# dedicated flacfetch VM service (see docs/00-current-plan/TORRENT-SERVICE-PLAN.md)
|
||||
RUN apt-get update && apt-get install -y \
|
||||
ffmpeg \
|
||||
libsndfile1 \
|
||||
libsox-dev \
|
||||
sox \
|
||||
build-essential \
|
||||
curl \
|
||||
xz-utils \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install comprehensive font support for international lyrics and symbols
|
||||
# Noto fonts provide full Unicode coverage including:
|
||||
# - Latin, Greek, Cyrillic (core)
|
||||
# - CJK: Chinese, Japanese, Korean (~150MB)
|
||||
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
|
||||
# - Color emoji (limited support in libass, but available)
|
||||
# - Musical symbols (♪ ♫ etc.)
|
||||
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
|
||||
RUN apt-get update && apt-get install -y \
|
||||
fonts-noto-core \
|
||||
fonts-noto-cjk \
|
||||
fonts-noto-extra \
|
||||
fonts-noto-color-emoji \
|
||||
fontconfig \
|
||||
&& fc-cache -f \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install optimized static FFmpeg build (7.0.2) instead of Debian package (5.1.8)
|
||||
# John Van Sickle's builds are compiled with better optimizations for x86_64
|
||||
# This provides ~20-40% performance improvement for encoding tasks
|
||||
# Source: https://johnvansickle.com/ffmpeg/
|
||||
RUN curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz -o /tmp/ffmpeg.tar.xz && \
|
||||
tar -xf /tmp/ffmpeg.tar.xz -C /tmp && \
|
||||
cp /tmp/ffmpeg-*-amd64-static/ffmpeg /usr/local/bin/ && \
|
||||
cp /tmp/ffmpeg-*-amd64-static/ffprobe /usr/local/bin/ && \
|
||||
chmod +x /usr/local/bin/ffmpeg /usr/local/bin/ffprobe && \
|
||||
rm -rf /tmp/ffmpeg* && \
|
||||
ffmpeg -version
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
|
||||
+182
-70
@@ -4,11 +4,11 @@ FastAPI dependencies for authentication and authorization.
|
||||
import logging
|
||||
import secrets
|
||||
from datetime import datetime, timedelta, UTC
|
||||
from typing import Optional, Tuple, Callable
|
||||
from typing import Optional, Tuple, Callable, Union
|
||||
from fastapi import Depends, HTTPException, Header, Query, Request, Path
|
||||
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
|
||||
|
||||
from backend.services.auth_service import get_auth_service, UserType, AuthService
|
||||
from backend.services.auth_service import get_auth_service, UserType, AuthService, AuthResult
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -66,70 +66,130 @@ async def require_auth(
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None)
|
||||
) -> Tuple[str, UserType, int]:
|
||||
) -> AuthResult:
|
||||
"""
|
||||
Require authentication for an endpoint.
|
||||
|
||||
|
||||
Returns:
|
||||
(token, user_type, remaining_uses)
|
||||
|
||||
AuthResult with full authentication context including:
|
||||
- is_valid, user_type, remaining_uses, message (backward compatible via tuple unpacking)
|
||||
- user_email: Email of authenticated user (if session/API key auth)
|
||||
- is_admin: Whether user has admin privileges
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if authentication fails
|
||||
"""
|
||||
# Get token from request
|
||||
token_str = await get_token_from_request(request, credentials, token)
|
||||
|
||||
|
||||
if not token_str:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide token via Authorization header or ?token= parameter"
|
||||
)
|
||||
|
||||
# Validate token
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(token_str)
|
||||
|
||||
if not is_valid:
|
||||
# Log more details for debugging token issues
|
||||
|
||||
# Validate token using the full method
|
||||
auth_result = auth_service.validate_token_full(token_str)
|
||||
|
||||
# Get request_id from middleware for correlation
|
||||
request_id = getattr(request.state, "request_id", None)
|
||||
|
||||
if not auth_result.is_valid:
|
||||
# Log auth failure with request_id for correlation
|
||||
auth_header = request.headers.get("Authorization", "")
|
||||
logger.warning(
|
||||
f"Authentication failed: {message}. "
|
||||
f"Token provided: {bool(token_str)}, "
|
||||
f"Token length: {len(token_str) if token_str else 0}, "
|
||||
f"Auth header present: {bool(auth_header)}, "
|
||||
f"Header prefix: {auth_header[:20] if auth_header else 'none'}..."
|
||||
"auth_failed",
|
||||
extra={
|
||||
"request_id": request_id,
|
||||
"audit_type": "auth_event",
|
||||
"auth_message": auth_result.message,
|
||||
"token_provided": bool(token_str),
|
||||
"token_length": len(token_str) if token_str else 0,
|
||||
"auth_header_present": bool(auth_header),
|
||||
}
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail=f"Authentication failed: {auth_result.message}"
|
||||
)
|
||||
|
||||
# Log successful auth with request_id for correlation with request audit
|
||||
logger.info(
|
||||
"auth_success",
|
||||
extra={
|
||||
"request_id": request_id,
|
||||
"user_email": auth_result.user_email,
|
||||
"user_type": auth_result.user_type.value if auth_result.user_type else None,
|
||||
"is_admin": auth_result.is_admin,
|
||||
"remaining_uses": auth_result.remaining_uses,
|
||||
"audit_type": "auth_event",
|
||||
}
|
||||
)
|
||||
|
||||
return auth_result
|
||||
|
||||
|
||||
async def require_auth_legacy(
|
||||
request: Request,
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None)
|
||||
) -> Tuple[str, UserType, int]:
|
||||
"""
|
||||
Legacy authentication dependency that returns tuple.
|
||||
|
||||
DEPRECATED: Use require_auth which returns AuthResult instead.
|
||||
|
||||
Returns:
|
||||
(token, user_type, remaining_uses)
|
||||
"""
|
||||
token_str = await get_token_from_request(request, credentials, token)
|
||||
|
||||
if not token_str:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide token via Authorization header or ?token= parameter"
|
||||
)
|
||||
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(token_str)
|
||||
|
||||
if not is_valid:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail=f"Authentication failed: {message}"
|
||||
)
|
||||
|
||||
logger.info(f"Authenticated as {user_type} (remaining: {remaining_uses})")
|
||||
|
||||
|
||||
return token_str, user_type, remaining_uses
|
||||
|
||||
|
||||
async def require_admin(
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
) -> Tuple[str, UserType, int]:
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> AuthResult:
|
||||
"""
|
||||
Require admin access for an endpoint.
|
||||
|
||||
|
||||
Admin access is granted if:
|
||||
- Using an admin token from ADMIN_TOKENS env var
|
||||
- User email is from admin domain (e.g., @nomadkaraoke.com)
|
||||
- User role is set to ADMIN in database
|
||||
|
||||
Returns:
|
||||
(token, user_type, remaining_uses)
|
||||
|
||||
AuthResult with admin privileges
|
||||
|
||||
Raises:
|
||||
HTTPException: 403 if user is not admin
|
||||
"""
|
||||
token, user_type, remaining_uses = auth_data
|
||||
|
||||
if user_type != UserType.ADMIN:
|
||||
logger.warning(f"Admin access denied for {user_type} user")
|
||||
if not auth_result.is_admin:
|
||||
logger.warning(
|
||||
f"Admin access denied for {auth_result.user_type} user "
|
||||
f"(email: {auth_result.user_email or 'unknown'})"
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="Admin access required"
|
||||
)
|
||||
|
||||
return token, user_type, remaining_uses
|
||||
|
||||
return auth_result
|
||||
|
||||
|
||||
def optional_auth(
|
||||
@@ -168,14 +228,14 @@ def optional_auth(
|
||||
def require_review_auth_factory(job_id_param: str = "job_id"):
|
||||
"""
|
||||
Factory to create a review authentication dependency.
|
||||
|
||||
|
||||
Accepts either:
|
||||
1. Full user authentication (admin/user token)
|
||||
1. Full user authentication (admin/user token) - also validates job ownership
|
||||
2. Job-specific review token (only valid for the specific job)
|
||||
|
||||
|
||||
Args:
|
||||
job_id_param: Name of the path parameter containing the job ID
|
||||
|
||||
|
||||
Returns:
|
||||
Dependency function that validates review access
|
||||
"""
|
||||
@@ -189,37 +249,63 @@ def require_review_auth_factory(job_id_param: str = "job_id"):
|
||||
) -> Tuple[str, str]:
|
||||
"""
|
||||
Validate review access for a job.
|
||||
|
||||
|
||||
Returns:
|
||||
(job_id, auth_type) where auth_type is "full" or "review_token"
|
||||
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if authentication fails
|
||||
HTTPException: 403 if user doesn't own the job (for full auth)
|
||||
"""
|
||||
# Import here to avoid circular dependency
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
|
||||
job_manager = JobManager()
|
||||
|
||||
# Try full authentication first
|
||||
full_token = None
|
||||
if credentials:
|
||||
full_token = credentials.credentials
|
||||
elif token:
|
||||
full_token = token
|
||||
|
||||
|
||||
if full_token:
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(full_token)
|
||||
if is_valid:
|
||||
logger.info(f"Review access granted via full auth ({user_type}) for job {job_id}")
|
||||
auth_result = auth_service.validate_token_full(full_token)
|
||||
if auth_result.is_valid:
|
||||
# For full auth, also verify job ownership
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Check ownership: admin can access all, users only their own jobs
|
||||
if not auth_result.is_admin:
|
||||
if auth_result.user_email and job.user_email:
|
||||
if auth_result.user_email.lower() != job.user_email.lower():
|
||||
logger.warning(
|
||||
f"Review access denied: user {auth_result.user_email} "
|
||||
f"tried to access job {job_id} owned by {job.user_email}"
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="You don't have permission to access this job's review"
|
||||
)
|
||||
elif job.user_email:
|
||||
# Token auth without email trying to access a job with owner
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="You don't have permission to access this job's review"
|
||||
)
|
||||
|
||||
logger.info(f"Review access granted via full auth ({auth_result.user_type}) for job {job_id}")
|
||||
return job_id, "full"
|
||||
|
||||
|
||||
# Try review token
|
||||
if review_token:
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
|
||||
# Validate review token matches
|
||||
if job.review_token and secrets.compare_digest(job.review_token, review_token):
|
||||
# Check expiry if set
|
||||
@@ -229,25 +315,25 @@ def require_review_auth_factory(job_id_param: str = "job_id"):
|
||||
expiry = job.review_token_expires_at
|
||||
if expiry.tzinfo is None:
|
||||
expiry = expiry.replace(tzinfo=UTC)
|
||||
|
||||
|
||||
if now > expiry:
|
||||
logger.warning(f"Review token expired for job {job_id}")
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Review token has expired. Please request a new review link."
|
||||
)
|
||||
|
||||
|
||||
logger.info(f"Review access granted via review_token for job {job_id}")
|
||||
return job_id, "review_token"
|
||||
else:
|
||||
logger.warning(f"Invalid review token for job {job_id}")
|
||||
|
||||
|
||||
# No valid authentication
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide either a full access token or a valid review_token for this job."
|
||||
)
|
||||
|
||||
|
||||
return require_review_auth
|
||||
|
||||
|
||||
@@ -258,14 +344,14 @@ require_review_auth = require_review_auth_factory()
|
||||
def require_instrumental_auth_factory(job_id_param: str = "job_id"):
|
||||
"""
|
||||
Factory to create an instrumental review authentication dependency.
|
||||
|
||||
|
||||
Accepts either:
|
||||
1. Full user authentication (admin/user token)
|
||||
1. Full user authentication (admin/user token) - also validates job ownership
|
||||
2. Job-specific instrumental token (only valid for the specific job)
|
||||
|
||||
|
||||
Args:
|
||||
job_id_param: Name of the path parameter containing the job ID
|
||||
|
||||
|
||||
Returns:
|
||||
Dependency function that validates instrumental review access
|
||||
"""
|
||||
@@ -279,37 +365,63 @@ def require_instrumental_auth_factory(job_id_param: str = "job_id"):
|
||||
) -> Tuple[str, str]:
|
||||
"""
|
||||
Validate instrumental review access for a job.
|
||||
|
||||
|
||||
Returns:
|
||||
(job_id, auth_type) where auth_type is "full" or "instrumental_token"
|
||||
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if authentication fails
|
||||
HTTPException: 403 if user doesn't own the job (for full auth)
|
||||
"""
|
||||
# Import here to avoid circular dependency
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
|
||||
job_manager = JobManager()
|
||||
|
||||
# Try full authentication first
|
||||
full_token = None
|
||||
if credentials:
|
||||
full_token = credentials.credentials
|
||||
elif token:
|
||||
full_token = token
|
||||
|
||||
|
||||
if full_token:
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(full_token)
|
||||
if is_valid:
|
||||
logger.info(f"Instrumental access granted via full auth ({user_type}) for job {job_id}")
|
||||
auth_result = auth_service.validate_token_full(full_token)
|
||||
if auth_result.is_valid:
|
||||
# For full auth, also verify job ownership
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Check ownership: admin can access all, users only their own jobs
|
||||
if not auth_result.is_admin:
|
||||
if auth_result.user_email and job.user_email:
|
||||
if auth_result.user_email.lower() != job.user_email.lower():
|
||||
logger.warning(
|
||||
f"Instrumental access denied: user {auth_result.user_email} "
|
||||
f"tried to access job {job_id} owned by {job.user_email}"
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="You don't have permission to access this job's instrumental review"
|
||||
)
|
||||
elif job.user_email:
|
||||
# Token auth without email trying to access a job with owner
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="You don't have permission to access this job's instrumental review"
|
||||
)
|
||||
|
||||
logger.info(f"Instrumental access granted via full auth ({auth_result.user_type}) for job {job_id}")
|
||||
return job_id, "full"
|
||||
|
||||
|
||||
# Try instrumental token
|
||||
if instrumental_token:
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
|
||||
# Validate instrumental token matches
|
||||
if job.instrumental_token and secrets.compare_digest(job.instrumental_token, instrumental_token):
|
||||
# Check expiry if set
|
||||
@@ -319,25 +431,25 @@ def require_instrumental_auth_factory(job_id_param: str = "job_id"):
|
||||
expiry = job.instrumental_token_expires_at
|
||||
if expiry.tzinfo is None:
|
||||
expiry = expiry.replace(tzinfo=UTC)
|
||||
|
||||
|
||||
if now > expiry:
|
||||
logger.warning(f"Instrumental token expired for job {job_id}")
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Instrumental review token has expired. Please request a new review link."
|
||||
)
|
||||
|
||||
|
||||
logger.info(f"Instrumental access granted via instrumental_token for job {job_id}")
|
||||
return job_id, "instrumental_token"
|
||||
else:
|
||||
logger.warning(f"Invalid instrumental token for job {job_id}")
|
||||
|
||||
|
||||
# No valid authentication
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide either a full access token or a valid instrumental_token for this job."
|
||||
)
|
||||
|
||||
|
||||
return require_instrumental_auth
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,835 @@
|
||||
"""
|
||||
Admin API routes for dashboard statistics and management.
|
||||
|
||||
Handles:
|
||||
- Dashboard overview statistics
|
||||
- System-wide metrics
|
||||
- Admin-only operations
|
||||
- Audio search cache management
|
||||
"""
|
||||
import logging
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Tuple, List, Optional, Any, Dict
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.user_service import get_user_service, UserService, USERS_COLLECTION
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.flacfetch_client import get_flacfetch_client, FlacfetchServiceError
|
||||
from backend.models.job import JobStatus
|
||||
from backend.utils.test_data import is_test_email
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/admin", tags=["admin"])
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Response Models
|
||||
# =============================================================================
|
||||
|
||||
class JobsByStatusResponse(BaseModel):
|
||||
"""Breakdown of jobs by status."""
|
||||
pending: int = 0
|
||||
processing: int = 0
|
||||
awaiting_review: int = 0
|
||||
awaiting_instrumental: int = 0
|
||||
complete: int = 0
|
||||
failed: int = 0
|
||||
cancelled: int = 0
|
||||
|
||||
|
||||
class AdminStatsOverview(BaseModel):
|
||||
"""Overview statistics for admin dashboard."""
|
||||
total_users: int
|
||||
active_users_7d: int
|
||||
active_users_30d: int
|
||||
total_jobs: int
|
||||
jobs_last_7d: int
|
||||
jobs_last_30d: int
|
||||
jobs_by_status: JobsByStatusResponse
|
||||
total_credits_issued_30d: int
|
||||
total_beta_testers: int
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Admin Stats Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/stats/overview", response_model=AdminStatsOverview)
|
||||
async def get_admin_stats_overview(
|
||||
exclude_test: bool = True,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
):
|
||||
"""
|
||||
Get overview statistics for admin dashboard.
|
||||
|
||||
Args:
|
||||
exclude_test: If True (default), exclude test data (users with @inbox.testmail.app emails
|
||||
and jobs created by test users) from all counts.
|
||||
|
||||
Includes:
|
||||
- User counts (total, active in 7d, active in 30d)
|
||||
- Job counts (total, by status, recent)
|
||||
- Credit statistics
|
||||
- Beta program stats
|
||||
"""
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
from google.cloud.firestore_v1 import aggregation
|
||||
|
||||
db = user_service.db
|
||||
now = datetime.utcnow()
|
||||
seven_days_ago = now - timedelta(days=7)
|
||||
thirty_days_ago = now - timedelta(days=30)
|
||||
|
||||
users_collection = db.collection(USERS_COLLECTION)
|
||||
jobs_collection = db.collection("jobs")
|
||||
|
||||
# Jobs by status category mapping
|
||||
processing_statuses = [
|
||||
"downloading", "downloading_audio", "searching_audio", "awaiting_audio_selection",
|
||||
"separating_stage1", "separating_stage2", "transcribing", "correcting",
|
||||
"generating_screens", "applying_padding", "rendering_video",
|
||||
"instrumental_selected", "generating_video", "encoding", "packaging",
|
||||
"uploading", "notifying"
|
||||
]
|
||||
|
||||
# Limits for streaming queries - these are safety limits to prevent memory issues
|
||||
# If hit, stats may be incomplete so we log a warning
|
||||
USERS_STREAM_LIMIT = 2000
|
||||
JOBS_STREAM_LIMIT = 10000
|
||||
|
||||
if exclude_test:
|
||||
# When excluding test data, we must stream and filter in Python
|
||||
# because Firestore doesn't support "not ends with" queries
|
||||
|
||||
# Stream all users and filter
|
||||
all_users = []
|
||||
users_fetched = 0
|
||||
for doc in users_collection.limit(USERS_STREAM_LIMIT).stream():
|
||||
users_fetched += 1
|
||||
user_data = doc.to_dict()
|
||||
email = user_data.get("email", "")
|
||||
if not is_test_email(email):
|
||||
all_users.append(user_data)
|
||||
|
||||
if users_fetched >= USERS_STREAM_LIMIT:
|
||||
logger.warning(f"Users stream hit limit ({USERS_STREAM_LIMIT}), stats may be incomplete")
|
||||
|
||||
# Calculate user stats from filtered list
|
||||
total_users = len(all_users)
|
||||
active_users_7d = sum(
|
||||
1 for u in all_users
|
||||
if u.get("last_login_at") and _normalize_datetime(u["last_login_at"]) >= seven_days_ago
|
||||
)
|
||||
active_users_30d = sum(
|
||||
1 for u in all_users
|
||||
if u.get("last_login_at") and _normalize_datetime(u["last_login_at"]) >= thirty_days_ago
|
||||
)
|
||||
total_beta_testers = sum(1 for u in all_users if u.get("is_beta_tester"))
|
||||
|
||||
# Calculate credits from filtered users
|
||||
total_credits_issued_30d = 0
|
||||
for user_data in all_users:
|
||||
transactions = user_data.get("credit_transactions", [])
|
||||
for txn in transactions:
|
||||
txn_date = _normalize_datetime(txn.get("created_at"))
|
||||
if txn_date and txn_date >= thirty_days_ago:
|
||||
amount = txn.get("amount", 0)
|
||||
if amount > 0:
|
||||
total_credits_issued_30d += amount
|
||||
|
||||
# Stream all jobs and filter by user_email
|
||||
all_jobs = []
|
||||
jobs_fetched = 0
|
||||
for doc in jobs_collection.limit(JOBS_STREAM_LIMIT).stream():
|
||||
jobs_fetched += 1
|
||||
job_data = doc.to_dict()
|
||||
user_email = job_data.get("user_email", "")
|
||||
if not is_test_email(user_email):
|
||||
all_jobs.append(job_data)
|
||||
|
||||
if jobs_fetched >= JOBS_STREAM_LIMIT:
|
||||
logger.warning(f"Jobs stream hit limit ({JOBS_STREAM_LIMIT}), stats may be incomplete")
|
||||
|
||||
# Calculate job stats from filtered list
|
||||
total_jobs = len(all_jobs)
|
||||
jobs_last_7d = sum(
|
||||
1 for j in all_jobs
|
||||
if j.get("created_at") and _normalize_datetime(j["created_at"]) >= seven_days_ago
|
||||
)
|
||||
jobs_last_30d = sum(
|
||||
1 for j in all_jobs
|
||||
if j.get("created_at") and _normalize_datetime(j["created_at"]) >= thirty_days_ago
|
||||
)
|
||||
|
||||
# Jobs by status
|
||||
jobs_by_status = JobsByStatusResponse(
|
||||
pending=sum(1 for j in all_jobs if j.get("status") == "pending"),
|
||||
processing=sum(1 for j in all_jobs if j.get("status") in processing_statuses),
|
||||
awaiting_review=sum(1 for j in all_jobs if j.get("status") in ["awaiting_review", "in_review"]),
|
||||
awaiting_instrumental=sum(1 for j in all_jobs if j.get("status") == "awaiting_instrumental_selection"),
|
||||
complete=sum(1 for j in all_jobs if j.get("status") in ["complete", "prep_complete"]),
|
||||
failed=sum(1 for j in all_jobs if j.get("status") == "failed"),
|
||||
cancelled=sum(1 for j in all_jobs if j.get("status") == "cancelled"),
|
||||
)
|
||||
else:
|
||||
# When including test data, use efficient aggregation queries
|
||||
def get_count(query) -> int:
|
||||
try:
|
||||
agg_query = aggregation.AggregationQuery(query)
|
||||
agg_query.count(alias="count")
|
||||
results = agg_query.get()
|
||||
return results[0][0].value if results else 0
|
||||
except Exception as e:
|
||||
logger.warning(f"Aggregation query failed: {e}")
|
||||
return 0
|
||||
|
||||
# User statistics
|
||||
total_users = get_count(users_collection)
|
||||
active_users_7d = get_count(
|
||||
users_collection.where(filter=FieldFilter("last_login_at", ">=", seven_days_ago))
|
||||
)
|
||||
active_users_30d = get_count(
|
||||
users_collection.where(filter=FieldFilter("last_login_at", ">=", thirty_days_ago))
|
||||
)
|
||||
total_beta_testers = get_count(
|
||||
users_collection.where(filter=FieldFilter("is_beta_tester", "==", True))
|
||||
)
|
||||
|
||||
# Job statistics
|
||||
total_jobs = get_count(jobs_collection)
|
||||
jobs_last_7d = get_count(
|
||||
jobs_collection.where(filter=FieldFilter("created_at", ">=", seven_days_ago))
|
||||
)
|
||||
jobs_last_30d = get_count(
|
||||
jobs_collection.where(filter=FieldFilter("created_at", ">=", thirty_days_ago))
|
||||
)
|
||||
|
||||
# Jobs by status
|
||||
jobs_by_status = JobsByStatusResponse(
|
||||
pending=get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "pending"))
|
||||
),
|
||||
processing=sum(
|
||||
get_count(jobs_collection.where(filter=FieldFilter("status", "==", status)))
|
||||
for status in processing_statuses
|
||||
),
|
||||
awaiting_review=get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "awaiting_review"))
|
||||
) + get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "in_review"))
|
||||
),
|
||||
awaiting_instrumental=get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "awaiting_instrumental_selection"))
|
||||
),
|
||||
complete=get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "complete"))
|
||||
) + get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "prep_complete"))
|
||||
),
|
||||
failed=get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "failed"))
|
||||
),
|
||||
cancelled=get_count(
|
||||
jobs_collection.where(filter=FieldFilter("status", "==", "cancelled"))
|
||||
),
|
||||
)
|
||||
|
||||
# Credit statistics - sum credits added in last 30 days
|
||||
total_credits_issued_30d = 0
|
||||
try:
|
||||
users_fetched = 0
|
||||
for user_doc in users_collection.limit(USERS_STREAM_LIMIT).stream():
|
||||
users_fetched += 1
|
||||
user_data = user_doc.to_dict()
|
||||
transactions = user_data.get("credit_transactions", [])
|
||||
for txn in transactions:
|
||||
txn_date = _normalize_datetime(txn.get("created_at"))
|
||||
if txn_date and txn_date >= thirty_days_ago:
|
||||
amount = txn.get("amount", 0)
|
||||
if amount > 0:
|
||||
total_credits_issued_30d += amount
|
||||
if users_fetched >= USERS_STREAM_LIMIT:
|
||||
logger.warning(f"Credit calculation hit user limit ({USERS_STREAM_LIMIT}), total may be incomplete")
|
||||
except Exception as e:
|
||||
logger.warning(f"Error calculating credits: {e}")
|
||||
|
||||
return AdminStatsOverview(
|
||||
total_users=total_users,
|
||||
active_users_7d=active_users_7d,
|
||||
active_users_30d=active_users_30d,
|
||||
total_jobs=total_jobs,
|
||||
jobs_last_7d=jobs_last_7d,
|
||||
jobs_last_30d=jobs_last_30d,
|
||||
jobs_by_status=jobs_by_status,
|
||||
total_credits_issued_30d=total_credits_issued_30d,
|
||||
total_beta_testers=total_beta_testers,
|
||||
)
|
||||
|
||||
|
||||
def _normalize_datetime(dt_value) -> Optional[datetime]:
|
||||
"""Normalize datetime values from Firestore (can be datetime or ISO string)."""
|
||||
if dt_value is None:
|
||||
return None
|
||||
if isinstance(dt_value, datetime):
|
||||
return dt_value.replace(tzinfo=None)
|
||||
if isinstance(dt_value, str):
|
||||
try:
|
||||
parsed = datetime.fromisoformat(dt_value.replace("Z", "+00:00"))
|
||||
return parsed.replace(tzinfo=None)
|
||||
except Exception:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Audio Search Management Models
|
||||
# =============================================================================
|
||||
|
||||
class AudioSearchResultSummary(BaseModel):
|
||||
"""Summary of a single audio search result."""
|
||||
index: int
|
||||
provider: str
|
||||
artist: str
|
||||
title: str
|
||||
is_lossless: bool
|
||||
quality: Optional[str] = None
|
||||
seeders: Optional[int] = None
|
||||
|
||||
|
||||
class AudioSearchJobSummary(BaseModel):
|
||||
"""Summary of a job with audio search results."""
|
||||
job_id: str
|
||||
status: str
|
||||
user_email: Optional[str] = None
|
||||
audio_search_artist: Optional[str] = None
|
||||
audio_search_title: Optional[str] = None
|
||||
created_at: Optional[datetime] = None
|
||||
results_count: int
|
||||
results_summary: List[AudioSearchResultSummary]
|
||||
has_lossless: bool
|
||||
providers: List[str]
|
||||
|
||||
|
||||
class AudioSearchListResponse(BaseModel):
|
||||
"""Response for listing audio search jobs."""
|
||||
jobs: List[AudioSearchJobSummary]
|
||||
total: int
|
||||
|
||||
|
||||
class ClearSearchCacheResponse(BaseModel):
|
||||
"""Response for clearing search cache."""
|
||||
status: str
|
||||
job_id: str
|
||||
message: str
|
||||
previous_status: str
|
||||
new_status: str
|
||||
results_cleared: int
|
||||
flacfetch_cache_cleared: bool = False
|
||||
flacfetch_error: Optional[str] = None
|
||||
|
||||
|
||||
class ClearAllCacheResponse(BaseModel):
|
||||
"""Response for clearing all flacfetch cache."""
|
||||
status: str
|
||||
message: str
|
||||
deleted_count: int
|
||||
|
||||
|
||||
class CacheStatsResponse(BaseModel):
|
||||
"""Response for cache statistics."""
|
||||
count: int
|
||||
total_size_bytes: int
|
||||
oldest_entry: Optional[str] = None
|
||||
newest_entry: Optional[str] = None
|
||||
configured: bool
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Audio Search Management Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/audio-searches", response_model=AudioSearchListResponse)
|
||||
async def list_audio_searches(
|
||||
limit: int = 50,
|
||||
status_filter: Optional[str] = None,
|
||||
exclude_test: bool = True,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
):
|
||||
"""
|
||||
List jobs with audio search results.
|
||||
|
||||
Returns jobs that have cached audio search results, useful for:
|
||||
- Monitoring search activity
|
||||
- Identifying stale cached results
|
||||
- Clearing cache for specific jobs
|
||||
|
||||
Args:
|
||||
limit: Maximum number of jobs to return (default 50)
|
||||
status_filter: Optional filter by job status (e.g., 'awaiting_audio_selection')
|
||||
exclude_test: If True (default), exclude jobs from test users
|
||||
"""
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
|
||||
db = user_service.db
|
||||
jobs_collection = db.collection("jobs")
|
||||
|
||||
# Query jobs - we'll filter for those with audio_search_results in Python
|
||||
# since Firestore can't query for existence of nested fields efficiently
|
||||
query = jobs_collection.order_by("created_at", direction="DESCENDING").limit(500)
|
||||
|
||||
if status_filter:
|
||||
query = jobs_collection.where(
|
||||
filter=FieldFilter("status", "==", status_filter)
|
||||
).order_by("created_at", direction="DESCENDING").limit(500)
|
||||
|
||||
jobs_with_searches = []
|
||||
|
||||
for doc in query.stream():
|
||||
data = doc.to_dict()
|
||||
|
||||
# Filter out test users if exclude_test is True
|
||||
if exclude_test and is_test_email(data.get("user_email", "")):
|
||||
continue
|
||||
|
||||
state_data = data.get("state_data", {})
|
||||
audio_results = state_data.get("audio_search_results", [])
|
||||
|
||||
if not audio_results:
|
||||
continue
|
||||
|
||||
# Compute has_lossless and providers from ALL results (not just first 10)
|
||||
has_lossless = any(r.get("is_lossless", False) for r in audio_results)
|
||||
providers = {r.get("provider", "Unknown") for r in audio_results}
|
||||
|
||||
# Build summary from first 10 results only
|
||||
results_summary = []
|
||||
for r in audio_results[:10]:
|
||||
results_summary.append(AudioSearchResultSummary(
|
||||
index=r.get("index", 0),
|
||||
provider=r.get("provider", "Unknown"),
|
||||
artist=r.get("artist", ""),
|
||||
title=r.get("title", ""),
|
||||
is_lossless=r.get("is_lossless", False),
|
||||
quality=r.get("quality"),
|
||||
seeders=r.get("seeders"),
|
||||
))
|
||||
|
||||
jobs_with_searches.append(AudioSearchJobSummary(
|
||||
job_id=doc.id,
|
||||
status=data.get("status", "unknown"),
|
||||
user_email=data.get("user_email"),
|
||||
audio_search_artist=data.get("audio_search_artist"),
|
||||
audio_search_title=data.get("audio_search_title"),
|
||||
created_at=data.get("created_at"),
|
||||
results_count=len(audio_results),
|
||||
results_summary=results_summary,
|
||||
has_lossless=has_lossless,
|
||||
providers=sorted(providers),
|
||||
))
|
||||
|
||||
if len(jobs_with_searches) >= limit:
|
||||
break
|
||||
|
||||
return AudioSearchListResponse(
|
||||
jobs=jobs_with_searches,
|
||||
total=len(jobs_with_searches),
|
||||
)
|
||||
|
||||
|
||||
@router.post("/audio-searches/{job_id}/clear-cache", response_model=ClearSearchCacheResponse)
|
||||
async def clear_audio_search_cache(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
):
|
||||
"""
|
||||
Clear the cached audio search results for a job.
|
||||
|
||||
This will:
|
||||
1. Remove the cached search results from job.state_data
|
||||
2. Reset the job status to 'pending' so a new search can be performed
|
||||
3. Clear the flacfetch GCS cache for this artist/title (if available)
|
||||
|
||||
Use this when:
|
||||
- Cached results are stale (e.g., flacfetch was updated)
|
||||
- User wants to search again with different terms
|
||||
- Results appear incomplete or incorrect
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Get current state
|
||||
state_data = job.state_data or {}
|
||||
audio_results = state_data.get("audio_search_results", [])
|
||||
results_count = len(audio_results)
|
||||
previous_status = job.status
|
||||
|
||||
if not audio_results:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job {job_id} has no cached audio search results"
|
||||
)
|
||||
|
||||
# Validate job status - only allow cache clear for appropriate states
|
||||
# Don't allow clearing cache for jobs that are actively processing or complete
|
||||
forbidden_statuses = {
|
||||
"downloading", "downloading_audio", "searching_audio",
|
||||
"separating_stage1", "separating_stage2", "transcribing", "correcting",
|
||||
"generating_screens", "applying_padding", "rendering_video",
|
||||
"generating_video", "encoding", "packaging", "uploading",
|
||||
"complete", "prep_complete",
|
||||
}
|
||||
if previous_status in forbidden_statuses:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Cannot clear cache for job in '{previous_status}' state. "
|
||||
f"Only jobs in pending, awaiting_audio_selection, failed, or cancelled states can have cache cleared."
|
||||
)
|
||||
|
||||
# Clear the cache by removing the keys
|
||||
db = user_service.db
|
||||
job_ref = db.collection("jobs").document(job_id)
|
||||
|
||||
from google.cloud.firestore_v1 import DELETE_FIELD
|
||||
|
||||
# Update job: clear cache and reset status
|
||||
job_ref.update({
|
||||
"state_data.audio_search_results": DELETE_FIELD,
|
||||
"state_data.audio_search_count": DELETE_FIELD,
|
||||
"state_data.remote_search_id": DELETE_FIELD,
|
||||
"status": "pending",
|
||||
"progress": 0,
|
||||
"message": "Audio search cache cleared by admin. Ready for new search.",
|
||||
"updated_at": datetime.utcnow(),
|
||||
})
|
||||
|
||||
# Also clear flacfetch's GCS cache if we have artist/title
|
||||
flacfetch_cache_cleared = False
|
||||
flacfetch_error = None
|
||||
artist = job.audio_search_artist
|
||||
title = job.audio_search_title
|
||||
|
||||
if artist and title:
|
||||
flacfetch_client = get_flacfetch_client()
|
||||
if flacfetch_client:
|
||||
try:
|
||||
flacfetch_cache_cleared = await flacfetch_client.clear_search_cache(artist, title)
|
||||
logger.info(
|
||||
f"Cleared flacfetch cache for '{artist}' - '{title}': "
|
||||
f"{'deleted' if flacfetch_cache_cleared else 'no entry found'}"
|
||||
)
|
||||
except FlacfetchServiceError as e:
|
||||
flacfetch_error = str(e)
|
||||
logger.warning(f"Failed to clear flacfetch cache: {e}")
|
||||
else:
|
||||
flacfetch_error = "flacfetch client not configured"
|
||||
logger.debug("Skipping flacfetch cache clear - client not configured")
|
||||
else:
|
||||
flacfetch_error = "missing artist or title"
|
||||
logger.debug(f"Skipping flacfetch cache clear - missing artist ({artist}) or title ({title})")
|
||||
|
||||
logger.info(
|
||||
f"Admin {auth_data[0]} cleared audio search cache for job {job_id}. "
|
||||
f"Cleared {results_count} results. Status changed from {previous_status} to pending. "
|
||||
f"Flacfetch cache cleared: {flacfetch_cache_cleared}"
|
||||
)
|
||||
|
||||
message = f"Cleared {results_count} cached search results. Job reset to pending."
|
||||
if flacfetch_cache_cleared:
|
||||
message += " Flacfetch cache also cleared."
|
||||
elif flacfetch_error:
|
||||
message += f" Note: flacfetch cache not cleared ({flacfetch_error})."
|
||||
|
||||
return ClearSearchCacheResponse(
|
||||
status="success",
|
||||
job_id=job_id,
|
||||
message=message,
|
||||
previous_status=previous_status,
|
||||
new_status="pending",
|
||||
results_cleared=results_count,
|
||||
flacfetch_cache_cleared=flacfetch_cache_cleared,
|
||||
flacfetch_error=flacfetch_error,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/reset-worker-state")
|
||||
async def reset_worker_state(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Reset stale worker progress state for a job.
|
||||
|
||||
This clears the video_progress, render_progress, and screens_progress
|
||||
from state_data, allowing workers to be re-triggered.
|
||||
|
||||
Use this when a job is stuck because worker progress shows 'running'
|
||||
from a previous failed attempt.
|
||||
"""
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Reset worker progress states
|
||||
job_manager.update_state_data(job_id, 'video_progress', {'stage': 'pending'})
|
||||
job_manager.update_state_data(job_id, 'render_progress', {'stage': 'pending'})
|
||||
job_manager.update_state_data(job_id, 'screens_progress', {'stage': 'pending'})
|
||||
|
||||
logger.info(f"Admin {auth_data[0]} reset worker state for job {job_id}")
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"job_id": job_id,
|
||||
"message": "Worker progress states reset to pending"
|
||||
}
|
||||
|
||||
|
||||
@router.delete("/cache", response_model=ClearAllCacheResponse)
|
||||
async def clear_all_flacfetch_cache(
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Clear the entire flacfetch search cache.
|
||||
|
||||
This will delete all cached search results from flacfetch's GCS cache.
|
||||
Use with caution - this will cause all subsequent searches to hit
|
||||
the trackers fresh.
|
||||
|
||||
Note: This does NOT clear Firestore job.state_data caches, only the
|
||||
flacfetch-side GCS cache.
|
||||
"""
|
||||
flacfetch_client = get_flacfetch_client()
|
||||
if not flacfetch_client:
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail="flacfetch client not configured"
|
||||
)
|
||||
|
||||
try:
|
||||
deleted_count = await flacfetch_client.clear_all_cache()
|
||||
logger.info(
|
||||
f"Admin {auth_data[0]} cleared all flacfetch cache. "
|
||||
f"Deleted {deleted_count} entries."
|
||||
)
|
||||
return ClearAllCacheResponse(
|
||||
status="success",
|
||||
message=f"Cleared {deleted_count} cache entries from flacfetch.",
|
||||
deleted_count=deleted_count,
|
||||
)
|
||||
except FlacfetchServiceError as e:
|
||||
logger.error(f"Failed to clear all flacfetch cache: {e}")
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"Failed to clear flacfetch cache: {e}"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/cache/stats", response_model=CacheStatsResponse)
|
||||
async def get_flacfetch_cache_stats(
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Get statistics about the flacfetch search cache.
|
||||
|
||||
Returns information about:
|
||||
- Number of cached entries
|
||||
- Total size in bytes
|
||||
- Oldest and newest cache entries
|
||||
- Whether cache is configured
|
||||
"""
|
||||
flacfetch_client = get_flacfetch_client()
|
||||
if not flacfetch_client:
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail="flacfetch client not configured"
|
||||
)
|
||||
|
||||
try:
|
||||
stats = await flacfetch_client.get_cache_stats()
|
||||
return CacheStatsResponse(
|
||||
count=stats.get("count", 0),
|
||||
total_size_bytes=stats.get("total_size_bytes", 0),
|
||||
oldest_entry=stats.get("oldest_entry"),
|
||||
newest_entry=stats.get("newest_entry"),
|
||||
configured=stats.get("configured", True),
|
||||
)
|
||||
except FlacfetchServiceError as e:
|
||||
logger.error(f"Failed to get flacfetch cache stats: {e}")
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"Failed to get cache stats: {e}"
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Job Completion Message Endpoints (for admin copy/send functionality)
|
||||
# =============================================================================
|
||||
|
||||
class CompletionMessageResponse(BaseModel):
|
||||
"""Response containing the rendered completion message."""
|
||||
job_id: str
|
||||
message: str
|
||||
subject: str
|
||||
youtube_url: Optional[str] = None
|
||||
dropbox_url: Optional[str] = None
|
||||
|
||||
|
||||
class SendCompletionEmailRequest(BaseModel):
|
||||
"""Request to send a completion email."""
|
||||
to_email: str
|
||||
cc_admin: bool = True
|
||||
|
||||
|
||||
class SendCompletionEmailResponse(BaseModel):
|
||||
"""Response from sending a completion email."""
|
||||
success: bool
|
||||
job_id: str
|
||||
to_email: str
|
||||
message: str
|
||||
|
||||
|
||||
@router.get("/jobs/{job_id}/completion-message", response_model=CompletionMessageResponse)
|
||||
async def get_job_completion_message(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Get the rendered completion message for a job.
|
||||
|
||||
Returns the plain text message that would be sent to the user,
|
||||
rendered using the job completion template with the job's details.
|
||||
|
||||
This is useful for:
|
||||
- Copying the message to clipboard (e.g., for Fiverr)
|
||||
- Previewing the email before sending
|
||||
|
||||
Requires admin authentication.
|
||||
"""
|
||||
from backend.services.job_notification_service import get_job_notification_service
|
||||
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Get youtube, dropbox URLs, and brand_code from state_data (may be None)
|
||||
state_data = job.state_data or {}
|
||||
youtube_url = state_data.get('youtube_url')
|
||||
dropbox_url = state_data.get('dropbox_link')
|
||||
brand_code = state_data.get('brand_code')
|
||||
|
||||
# Render the completion message
|
||||
notification_service = get_job_notification_service()
|
||||
message = notification_service.get_completion_message(
|
||||
job_id=job.job_id,
|
||||
user_name=None, # Use default "there"
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
youtube_url=youtube_url,
|
||||
dropbox_url=dropbox_url,
|
||||
)
|
||||
|
||||
# Build subject: "NOMAD-1178: Artist - Title (Your karaoke video is ready!)"
|
||||
# Sanitize artist/title to handle Unicode characters (curly quotes, em dashes, etc.)
|
||||
# that cause email header encoding issues (MIME headers use latin-1)
|
||||
safe_artist = sanitize_filename(job.artist) if job.artist else None
|
||||
safe_title = sanitize_filename(job.title) if job.title else None
|
||||
if brand_code and safe_artist and safe_title:
|
||||
subject = f"{brand_code}: {safe_artist} - {safe_title} (Your karaoke video is ready!)"
|
||||
elif safe_artist and safe_title:
|
||||
subject = f"{safe_artist} - {safe_title} (Your karaoke video is ready!)"
|
||||
else:
|
||||
subject = "Your karaoke video is ready!"
|
||||
|
||||
return CompletionMessageResponse(
|
||||
job_id=job_id,
|
||||
message=message,
|
||||
subject=subject,
|
||||
youtube_url=youtube_url,
|
||||
dropbox_url=dropbox_url,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/send-completion-email", response_model=SendCompletionEmailResponse)
|
||||
async def send_job_completion_email(
|
||||
job_id: str,
|
||||
request: SendCompletionEmailRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Send a completion email for a job to a specified email address.
|
||||
|
||||
This allows admins to manually send (or re-send) completion emails,
|
||||
useful for:
|
||||
- Sending to customers who didn't have an email on file
|
||||
- Re-sending if the original email was lost
|
||||
- Sending to alternate email addresses
|
||||
|
||||
Requires admin authentication.
|
||||
"""
|
||||
from backend.services.job_notification_service import get_job_notification_service
|
||||
from backend.services.email_service import get_email_service
|
||||
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Get youtube, dropbox URLs, and brand_code from state_data (may be None)
|
||||
state_data = job.state_data or {}
|
||||
youtube_url = state_data.get('youtube_url')
|
||||
dropbox_url = state_data.get('dropbox_link')
|
||||
brand_code = state_data.get('brand_code')
|
||||
|
||||
# Render the completion message
|
||||
notification_service = get_job_notification_service()
|
||||
message = notification_service.get_completion_message(
|
||||
job_id=job.job_id,
|
||||
user_name=None, # Use default "there"
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
youtube_url=youtube_url,
|
||||
dropbox_url=dropbox_url,
|
||||
)
|
||||
|
||||
# Send the email
|
||||
email_service = get_email_service()
|
||||
success = email_service.send_job_completion(
|
||||
to_email=request.to_email,
|
||||
message_content=message,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
brand_code=brand_code,
|
||||
cc_admin=request.cc_admin,
|
||||
)
|
||||
|
||||
if success:
|
||||
logger.info(f"Admin sent completion email for job {job_id} to {request.to_email}")
|
||||
return SendCompletionEmailResponse(
|
||||
success=True,
|
||||
job_id=job_id,
|
||||
to_email=request.to_email,
|
||||
message=f"Completion email sent to {request.to_email}",
|
||||
)
|
||||
else:
|
||||
logger.error(f"Failed to send completion email for job {job_id}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to send email. Check email service configuration."
|
||||
)
|
||||
@@ -18,9 +18,10 @@ import tempfile
|
||||
from typing import Optional, List, Dict, Any, Tuple
|
||||
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request, Depends
|
||||
from pydantic import BaseModel, Field
|
||||
from pydantic import BaseModel, Field, validator
|
||||
|
||||
from backend.models.job import JobCreate, JobStatus
|
||||
from karaoke_gen.utils import normalize_text
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.worker_service import get_worker_service
|
||||
@@ -32,10 +33,11 @@ from backend.services.audio_search_service import (
|
||||
NoResultsError,
|
||||
DownloadError,
|
||||
)
|
||||
from backend.services.theme_service import get_theme_service
|
||||
from backend.config import get_settings
|
||||
from backend.version import VERSION
|
||||
from backend.api.dependencies import require_auth
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.auth_service import UserType, AuthResult
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -95,7 +97,7 @@ class AudioSearchRequest(BaseModel):
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
|
||||
@@ -107,6 +109,10 @@ class AudioSearchRequest(BaseModel):
|
||||
lyrics_artist: Optional[str] = Field(None, description="Override artist name for lyrics search")
|
||||
lyrics_title: Optional[str] = Field(None, description="Override title for lyrics search")
|
||||
subtitle_offset_ms: int = Field(0, description="Subtitle timing offset in milliseconds")
|
||||
|
||||
# Display overrides (optional - if empty, search values are used for display)
|
||||
display_artist: Optional[str] = Field(None, description="Artist name for title screens/filenames. If empty, uses search artist.")
|
||||
display_title: Optional[str] = Field(None, description="Title for title screens/filenames. If empty, uses search title.")
|
||||
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model: Optional[str] = Field(None, description="Model for clean instrumental separation")
|
||||
@@ -117,6 +123,16 @@ class AudioSearchRequest(BaseModel):
|
||||
# Style params JSON and related assets (background images, fonts)
|
||||
style_files: Optional[List[StyleFileRequest]] = Field(None, description="Style files to upload (style_params JSON, backgrounds, fonts)")
|
||||
|
||||
# Non-interactive mode
|
||||
non_interactive: bool = Field(False, description="Skip interactive steps (lyrics review, instrumental selection)")
|
||||
|
||||
@validator('artist', 'title', 'lyrics_artist', 'lyrics_title', 'display_artist', 'display_title')
|
||||
def normalize_text_fields(cls, v):
|
||||
"""Normalize text fields to standardize Unicode characters."""
|
||||
if v is not None and isinstance(v, str):
|
||||
return normalize_text(v)
|
||||
return v
|
||||
|
||||
|
||||
class AudioSearchResultResponse(BaseModel):
|
||||
"""A single audio search result.
|
||||
@@ -372,8 +388,31 @@ async def _download_and_start_processing(
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
)
|
||||
elif source_name == 'YouTube' and download_url:
|
||||
# For YouTube without remote, download directly using URL from state_data
|
||||
# This avoids relying on in-memory cache which doesn't persist across instances
|
||||
logger.info(f"Downloading YouTube audio directly from URL: {download_url}")
|
||||
from backend.workers.audio_worker import download_from_url
|
||||
|
||||
local_path = await download_from_url(
|
||||
download_url,
|
||||
temp_dir,
|
||||
selected.get('artist'),
|
||||
selected.get('title')
|
||||
)
|
||||
|
||||
if not local_path or not os.path.exists(local_path):
|
||||
raise DownloadError(f"Failed to download from YouTube: {download_url}")
|
||||
|
||||
# Create a result-like object
|
||||
class DownloadResult:
|
||||
def __init__(self, filepath):
|
||||
self.filepath = filepath
|
||||
|
||||
result = DownloadResult(local_path)
|
||||
else:
|
||||
# Fallback to search-based download
|
||||
# Fallback to search-based download (may fail if cache is empty)
|
||||
logger.warning(f"Falling back to cache-based download for {source_name} - this may fail on multi-instance deployments")
|
||||
result = audio_search_service.download(
|
||||
result_index=selection_index,
|
||||
output_dir=temp_dir,
|
||||
@@ -436,7 +475,7 @@ async def search_audio(
|
||||
request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: AudioSearchRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Search for audio by artist and title, creating a new job.
|
||||
@@ -459,7 +498,8 @@ async def search_audio(
|
||||
effective_discord_webhook_url = body.discord_webhook_url or settings.default_discord_webhook_url
|
||||
|
||||
# Apply defaults for YouTube/Dropbox distribution (for web service)
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload or settings.default_enable_youtube_upload
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
effective_brand_prefix = body.brand_prefix or settings.default_brand_prefix
|
||||
effective_youtube_description = body.youtube_description or settings.default_youtube_description
|
||||
|
||||
@@ -495,16 +535,34 @@ async def search_audio(
|
||||
# Extract request metadata
|
||||
request_metadata = extract_request_metadata(request, created_from="audio_search")
|
||||
|
||||
# Apply default theme if none specified
|
||||
# This ensures all karaoke videos use the Nomad theme by default
|
||||
effective_theme_id = body.theme_id
|
||||
if effective_theme_id is None:
|
||||
theme_service = get_theme_service()
|
||||
effective_theme_id = theme_service.get_default_theme_id()
|
||||
if effective_theme_id:
|
||||
logger.info(f"Applying default theme: {effective_theme_id}")
|
||||
|
||||
# Resolve CDG/TXT defaults based on theme
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
body.theme_id, body.enable_cdg, body.enable_txt
|
||||
effective_theme_id, body.enable_cdg, body.enable_txt
|
||||
)
|
||||
|
||||
# Use authenticated user's email
|
||||
effective_user_email = auth_result.user_email
|
||||
|
||||
# Determine display values - use display_* if provided, otherwise fall back to search values
|
||||
# Display values are used for title screens, filenames, YouTube, etc.
|
||||
# Search values (body.artist, body.title) are used for audio search
|
||||
effective_display_artist = body.display_artist.strip() if body.display_artist else body.artist
|
||||
effective_display_title = body.display_title.strip() if body.display_title else body.title
|
||||
|
||||
# Create job
|
||||
job_create = JobCreate(
|
||||
artist=body.artist,
|
||||
title=body.title,
|
||||
theme_id=body.theme_id,
|
||||
artist=effective_display_artist, # Display value for title screens, filenames
|
||||
title=effective_display_title, # Display value for title screens, filenames
|
||||
theme_id=effective_theme_id,
|
||||
color_overrides=body.color_overrides or {},
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
@@ -515,6 +573,7 @@ async def search_audio(
|
||||
discord_webhook_url=effective_discord_webhook_url,
|
||||
dropbox_path=effective_dropbox_path,
|
||||
gdrive_folder_id=effective_gdrive_folder_id,
|
||||
user_email=effective_user_email,
|
||||
lyrics_artist=body.lyrics_artist,
|
||||
lyrics_title=body.lyrics_title,
|
||||
subtitle_offset_ms=body.subtitle_offset_ms,
|
||||
@@ -525,10 +584,11 @@ async def search_audio(
|
||||
audio_search_title=body.title,
|
||||
auto_download=body.auto_download,
|
||||
request_metadata=request_metadata,
|
||||
non_interactive=body.non_interactive,
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job_id = job.job_id
|
||||
|
||||
|
||||
logger.info(f"Created job {job_id} for audio search: {body.artist} - {body.title}")
|
||||
|
||||
# Update job with audio search fields
|
||||
@@ -541,11 +601,11 @@ async def search_audio(
|
||||
# If theme is set and no custom style files are being uploaded, prepare theme style now
|
||||
# This copies the theme's style_params.json to the job folder so LyricsTranscriber
|
||||
# can access the style configuration for preview videos
|
||||
if body.theme_id and not body.style_files:
|
||||
if effective_theme_id and not body.style_files:
|
||||
from backend.api.routes.file_upload import _prepare_theme_for_job
|
||||
try:
|
||||
style_params_path, theme_style_assets, youtube_desc = _prepare_theme_for_job(
|
||||
job_id, body.theme_id, body.color_overrides
|
||||
job_id, effective_theme_id, body.color_overrides
|
||||
)
|
||||
theme_update = {
|
||||
'style_params_gcs_path': style_params_path,
|
||||
@@ -554,9 +614,9 @@ async def search_audio(
|
||||
if youtube_desc and not effective_youtube_description:
|
||||
theme_update['youtube_description_template'] = youtube_desc
|
||||
job_manager.update_job(job_id, theme_update)
|
||||
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
|
||||
logger.info(f"Applied theme '{effective_theme_id}' to job {job_id}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to prepare theme '{body.theme_id}' for job {job_id}: {e}")
|
||||
logger.warning(f"Failed to prepare theme '{effective_theme_id}' for job {job_id}: {e}")
|
||||
# Continue without theme - job can still be processed with defaults
|
||||
|
||||
# Handle style file uploads if provided
|
||||
@@ -766,7 +826,7 @@ async def search_audio(
|
||||
@router.get("/audio-search/{job_id}/results")
|
||||
async def get_audio_search_results(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Get audio search results for a job.
|
||||
@@ -801,7 +861,7 @@ async def select_audio_source(
|
||||
job_id: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: AudioSelectRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Select an audio source and start job processing.
|
||||
|
||||
+347
-140
@@ -15,6 +15,7 @@ import json
|
||||
import logging
|
||||
import tempfile
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from fastapi import APIRouter, UploadFile, File, Form, HTTPException, BackgroundTasks, Request, Body, Depends
|
||||
from pathlib import Path
|
||||
from typing import Optional, List, Dict, Any, Tuple
|
||||
@@ -32,7 +33,7 @@ from backend.config import get_settings
|
||||
from backend.version import VERSION
|
||||
from backend.services.metrics import metrics
|
||||
from backend.api.dependencies import require_auth
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.auth_service import UserType, AuthResult
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(tags=["jobs"])
|
||||
@@ -68,7 +69,7 @@ class CreateJobFromUrlRequest(BaseModel):
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
webhook_url: Optional[str] = Field(None, description="Generic webhook URL")
|
||||
@@ -91,6 +92,9 @@ class CreateJobFromUrlRequest(BaseModel):
|
||||
backing_vocals_models: Optional[List[str]] = Field(None, description="Models for backing vocals separation")
|
||||
other_stems_models: Optional[List[str]] = Field(None, description="Models for other stems")
|
||||
|
||||
# Non-interactive mode
|
||||
non_interactive: bool = Field(False, description="Skip interactive steps (lyrics review, instrumental selection)")
|
||||
|
||||
|
||||
class CreateJobFromUrlResponse(BaseModel):
|
||||
"""Response from creating a job from URL."""
|
||||
@@ -119,7 +123,7 @@ class CreateJobWithUploadUrlsRequest(BaseModel):
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
webhook_url: Optional[str] = Field(None, description="Generic webhook URL")
|
||||
@@ -149,6 +153,9 @@ class CreateJobWithUploadUrlsRequest(BaseModel):
|
||||
prep_only: bool = Field(False, description="Stop after review phase, don't run finalisation")
|
||||
keep_brand_code: Optional[str] = Field(None, description="Preserve existing brand code instead of generating new one")
|
||||
|
||||
# Non-interactive mode
|
||||
non_interactive: bool = Field(False, description="Skip interactive steps (lyrics review, instrumental selection)")
|
||||
|
||||
|
||||
class SignedUploadUrl(BaseModel):
|
||||
"""Signed URL for uploading a file."""
|
||||
@@ -246,6 +253,16 @@ async def _trigger_workers_parallel(job_id: str) -> None:
|
||||
)
|
||||
|
||||
|
||||
async def _trigger_audio_worker_only(job_id: str) -> None:
|
||||
"""
|
||||
Trigger only the audio worker.
|
||||
|
||||
Used for URL jobs where audio needs to be downloaded first.
|
||||
The audio worker will trigger the lyrics worker after download completes.
|
||||
"""
|
||||
await worker_service.trigger_audio_worker(job_id)
|
||||
|
||||
|
||||
def _resolve_cdg_txt_defaults(
|
||||
theme_id: Optional[str],
|
||||
enable_cdg: Optional[bool],
|
||||
@@ -275,6 +292,46 @@ def _resolve_cdg_txt_defaults(
|
||||
return resolved_cdg, resolved_txt
|
||||
|
||||
|
||||
@dataclass
|
||||
class EffectiveDistributionSettings:
|
||||
"""Settings with defaults applied from environment variables."""
|
||||
dropbox_path: Optional[str]
|
||||
gdrive_folder_id: Optional[str]
|
||||
discord_webhook_url: Optional[str]
|
||||
brand_prefix: Optional[str]
|
||||
|
||||
|
||||
def _get_effective_distribution_settings(
|
||||
dropbox_path: Optional[str] = None,
|
||||
gdrive_folder_id: Optional[str] = None,
|
||||
discord_webhook_url: Optional[str] = None,
|
||||
brand_prefix: Optional[str] = None,
|
||||
) -> EffectiveDistributionSettings:
|
||||
"""
|
||||
Get distribution settings with defaults applied from environment variables.
|
||||
|
||||
This ensures consistent handling of defaults across all job creation endpoints.
|
||||
Each parameter, if not provided (None), falls back to the corresponding
|
||||
environment variable configured in settings.
|
||||
|
||||
Args:
|
||||
dropbox_path: Explicit Dropbox path or None for default
|
||||
gdrive_folder_id: Explicit Google Drive folder ID or None for default
|
||||
discord_webhook_url: Explicit Discord webhook URL or None for default
|
||||
brand_prefix: Explicit brand prefix or None for default
|
||||
|
||||
Returns:
|
||||
EffectiveDistributionSettings with defaults applied
|
||||
"""
|
||||
settings = get_settings()
|
||||
return EffectiveDistributionSettings(
|
||||
dropbox_path=dropbox_path or settings.default_dropbox_path,
|
||||
gdrive_folder_id=gdrive_folder_id or settings.default_gdrive_folder_id,
|
||||
discord_webhook_url=discord_webhook_url or settings.default_discord_webhook_url,
|
||||
brand_prefix=brand_prefix or settings.default_brand_prefix,
|
||||
)
|
||||
|
||||
|
||||
def _prepare_theme_for_job(
|
||||
job_id: str,
|
||||
theme_id: str,
|
||||
@@ -327,7 +384,7 @@ def _prepare_theme_for_job(
|
||||
async def upload_and_create_job(
|
||||
request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth),
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
# Required fields
|
||||
file: UploadFile = File(..., description="Audio file to process"),
|
||||
artist: str = Form(..., description="Artist name"),
|
||||
@@ -341,12 +398,15 @@ async def upload_and_create_job(
|
||||
style_cdg_instrumental_background: Optional[UploadFile] = File(None, description="CDG instrumental background"),
|
||||
style_cdg_title_background: Optional[UploadFile] = File(None, description="CDG title screen background"),
|
||||
style_cdg_outro_background: Optional[UploadFile] = File(None, description="CDG outro screen background"),
|
||||
# Processing options (CDG/TXT require style config, disabled by default)
|
||||
enable_cdg: bool = Form(False, description="Generate CDG+MP3 package (requires style config)"),
|
||||
enable_txt: bool = Form(False, description="Generate TXT+MP3 package (requires style config)"),
|
||||
# Theme configuration (use pre-made theme from GCS)
|
||||
theme_id: Optional[str] = Form(None, description="Theme ID to use (e.g., 'nomad', 'default'). If set, CDG/TXT are enabled by default."),
|
||||
color_overrides: Optional[str] = Form(None, description="JSON-encoded color overrides: artist_color, title_color, sung_lyrics_color, unsung_lyrics_color (hex #RRGGBB)"),
|
||||
# Processing options (CDG/TXT require style config or theme, disabled by default unless theme is set)
|
||||
enable_cdg: Optional[bool] = Form(None, description="Generate CDG+MP3 package. Default: True if theme_id set, False otherwise"),
|
||||
enable_txt: Optional[bool] = Form(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise"),
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Form(None, description="Brand code prefix (e.g., NOMAD)"),
|
||||
enable_youtube_upload: bool = Form(False, description="Upload to YouTube"),
|
||||
enable_youtube_upload: Optional[bool] = Form(None, description="Upload to YouTube. None = use server default"),
|
||||
youtube_description: Optional[str] = Form(None, description="YouTube video description text"),
|
||||
discord_webhook_url: Optional[str] = Form(None, description="Discord webhook URL for notifications"),
|
||||
webhook_url: Optional[str] = Form(None, description="Generic webhook URL"),
|
||||
@@ -365,6 +425,8 @@ async def upload_and_create_job(
|
||||
clean_instrumental_model: Optional[str] = Form(None, description="Model for clean instrumental separation (e.g., model_bs_roformer_ep_317_sdr_12.9755.ckpt)"),
|
||||
backing_vocals_models: Optional[str] = Form(None, description="Comma-separated list of models for backing vocals separation"),
|
||||
other_stems_models: Optional[str] = Form(None, description="Comma-separated list of models for other stems (bass, drums, guitar, etc.)"),
|
||||
# Non-interactive mode
|
||||
non_interactive: bool = Form(False, description="Skip interactive steps (lyrics review, instrumental selection)"),
|
||||
):
|
||||
"""
|
||||
Upload an audio file and create a karaoke generation job with full style configuration.
|
||||
@@ -432,38 +494,49 @@ async def upload_and_create_job(
|
||||
)
|
||||
|
||||
# Apply default distribution settings from environment if not provided
|
||||
settings = get_settings()
|
||||
effective_dropbox_path = dropbox_path or settings.default_dropbox_path
|
||||
effective_gdrive_folder_id = gdrive_folder_id or settings.default_gdrive_folder_id
|
||||
effective_discord_webhook_url = discord_webhook_url or settings.default_discord_webhook_url
|
||||
|
||||
if effective_dropbox_path and not dropbox_path:
|
||||
logger.info(f"Using default dropbox_path: {effective_dropbox_path}")
|
||||
if effective_gdrive_folder_id and not gdrive_folder_id:
|
||||
logger.info(f"Using default gdrive_folder_id: {effective_gdrive_folder_id}")
|
||||
if effective_discord_webhook_url and not discord_webhook_url:
|
||||
dist = _get_effective_distribution_settings(
|
||||
dropbox_path=dropbox_path,
|
||||
gdrive_folder_id=gdrive_folder_id,
|
||||
discord_webhook_url=discord_webhook_url,
|
||||
brand_prefix=brand_prefix,
|
||||
)
|
||||
|
||||
if dist.dropbox_path and not dropbox_path:
|
||||
logger.info(f"Using default dropbox_path: {dist.dropbox_path}")
|
||||
if dist.gdrive_folder_id and not gdrive_folder_id:
|
||||
logger.info(f"Using default gdrive_folder_id: {dist.gdrive_folder_id}")
|
||||
if dist.discord_webhook_url and not discord_webhook_url:
|
||||
logger.info("Using default discord_webhook_url (from env)")
|
||||
|
||||
if dist.brand_prefix and not brand_prefix:
|
||||
logger.info(f"Using default brand_prefix: {dist.brand_prefix}")
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_enable_youtube_upload = enable_youtube_upload if enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
if effective_enable_youtube_upload and enable_youtube_upload is None:
|
||||
logger.info("Using default enable_youtube_upload: True (from env)")
|
||||
|
||||
# Validate credentials for requested distribution services (including defaults)
|
||||
# This prevents accepting jobs that will fail later due to missing credentials
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if enable_youtube_upload:
|
||||
|
||||
if effective_enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
|
||||
if effective_dropbox_path:
|
||||
|
||||
if dist.dropbox_path:
|
||||
result = credential_manager.check_dropbox_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"dropbox ({result.message})")
|
||||
|
||||
if effective_gdrive_folder_id:
|
||||
|
||||
if dist.gdrive_folder_id:
|
||||
result = credential_manager.check_gdrive_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"gdrive ({result.message})")
|
||||
|
||||
|
||||
if invalid_services:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
@@ -477,7 +550,44 @@ async def upload_and_create_job(
|
||||
|
||||
# Extract request metadata for tracking and filtering
|
||||
request_metadata = extract_request_metadata(request, created_from="upload")
|
||||
|
||||
|
||||
# Parse color_overrides from JSON if provided
|
||||
parsed_color_overrides: Dict[str, str] = {}
|
||||
if color_overrides:
|
||||
try:
|
||||
parsed_color_overrides = json.loads(color_overrides)
|
||||
except json.JSONDecodeError as e:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid color_overrides JSON: {e}"
|
||||
)
|
||||
|
||||
# Apply default theme if none specified
|
||||
# This ensures all karaoke videos use the Nomad theme by default
|
||||
effective_theme_id = theme_id
|
||||
if effective_theme_id is None:
|
||||
theme_service = get_theme_service()
|
||||
effective_theme_id = theme_service.get_default_theme_id()
|
||||
if effective_theme_id:
|
||||
logger.info(f"Applying default theme: {effective_theme_id}")
|
||||
|
||||
# Resolve CDG/TXT defaults based on theme
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
effective_theme_id, enable_cdg, enable_txt
|
||||
)
|
||||
|
||||
# Check if any custom style files are being uploaded (overrides theme)
|
||||
has_custom_style_files = any([
|
||||
style_params,
|
||||
style_intro_background,
|
||||
style_karaoke_background,
|
||||
style_end_background,
|
||||
style_font,
|
||||
style_cdg_instrumental_background,
|
||||
style_cdg_title_background,
|
||||
style_cdg_outro_background,
|
||||
])
|
||||
|
||||
# Parse comma-separated model lists into arrays
|
||||
parsed_backing_vocals_models = None
|
||||
if backing_vocals_models:
|
||||
@@ -487,22 +597,27 @@ async def upload_and_create_job(
|
||||
if other_stems_models:
|
||||
parsed_other_stems_models = [m.strip() for m in other_stems_models.split(',') if m.strip()]
|
||||
|
||||
# Prefer authenticated user's email over form parameter
|
||||
effective_user_email = auth_result.user_email or user_email
|
||||
|
||||
# Create job first to get job_id
|
||||
job_create = JobCreate(
|
||||
artist=artist,
|
||||
title=title,
|
||||
filename=file.filename,
|
||||
enable_cdg=enable_cdg,
|
||||
enable_txt=enable_txt,
|
||||
brand_prefix=brand_prefix,
|
||||
enable_youtube_upload=enable_youtube_upload,
|
||||
theme_id=effective_theme_id,
|
||||
color_overrides=parsed_color_overrides,
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=dist.brand_prefix,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=youtube_description,
|
||||
discord_webhook_url=effective_discord_webhook_url,
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
webhook_url=webhook_url,
|
||||
user_email=user_email,
|
||||
user_email=effective_user_email,
|
||||
# Native API distribution (preferred for remote CLI)
|
||||
dropbox_path=effective_dropbox_path,
|
||||
gdrive_folder_id=effective_gdrive_folder_id,
|
||||
dropbox_path=dist.dropbox_path,
|
||||
gdrive_folder_id=dist.gdrive_folder_id,
|
||||
# Legacy rclone distribution (deprecated)
|
||||
organised_dir_rclone_root=organised_dir_rclone_root,
|
||||
# Lyrics configuration (overrides for search/transcription)
|
||||
@@ -515,15 +630,35 @@ async def upload_and_create_job(
|
||||
other_stems_models=parsed_other_stems_models,
|
||||
# Request metadata for tracking and filtering
|
||||
request_metadata=request_metadata,
|
||||
# Non-interactive mode
|
||||
non_interactive=non_interactive,
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job_id = job.job_id
|
||||
|
||||
# Record job creation metric
|
||||
metrics.record_job_created(job_id, source="upload")
|
||||
|
||||
|
||||
logger.info(f"Created job {job_id} for {artist} - {title}")
|
||||
|
||||
|
||||
# If theme is set and no custom style files are being uploaded, prepare theme style now
|
||||
# This copies the theme's style_params.json to the job folder so LyricsTranscriber
|
||||
# can access the style configuration for preview videos
|
||||
theme_style_params_path = None
|
||||
theme_style_assets = {}
|
||||
theme_youtube_desc = None
|
||||
if effective_theme_id and not has_custom_style_files:
|
||||
try:
|
||||
theme_style_params_path, theme_style_assets, theme_youtube_desc = _prepare_theme_for_job(
|
||||
job_id, effective_theme_id, parsed_color_overrides or None
|
||||
)
|
||||
logger.info(f"Applied theme '{effective_theme_id}' to job {job_id}")
|
||||
except HTTPException:
|
||||
raise # Re-raise validation errors (e.g., theme not found)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to prepare theme '{effective_theme_id}' for job {job_id}: {e}")
|
||||
# Continue without theme - job can still be processed with defaults
|
||||
|
||||
# Upload main audio file to GCS
|
||||
audio_gcs_path = f"uploads/{job_id}/audio/{file.filename}"
|
||||
logger.info(f"Uploading audio to GCS: {audio_gcs_path}")
|
||||
@@ -592,27 +727,36 @@ async def upload_and_create_job(
|
||||
update_data = {
|
||||
'input_media_gcs_path': audio_gcs_path,
|
||||
'filename': file.filename,
|
||||
'enable_cdg': enable_cdg,
|
||||
'enable_txt': enable_txt,
|
||||
'enable_cdg': resolved_cdg,
|
||||
'enable_txt': resolved_txt,
|
||||
}
|
||||
|
||||
|
||||
# Handle style assets - either from custom uploads or from theme
|
||||
if style_assets:
|
||||
# Custom style files uploaded
|
||||
update_data['style_assets'] = style_assets
|
||||
if 'style_params' in style_assets:
|
||||
update_data['style_params_gcs_path'] = style_assets['style_params']
|
||||
|
||||
if brand_prefix:
|
||||
update_data['brand_prefix'] = brand_prefix
|
||||
if effective_discord_webhook_url:
|
||||
update_data['discord_webhook_url'] = effective_discord_webhook_url
|
||||
if youtube_description:
|
||||
update_data['youtube_description_template'] = youtube_description
|
||||
|
||||
elif theme_style_assets:
|
||||
# Theme style assets (no custom uploads)
|
||||
update_data['style_assets'] = theme_style_assets
|
||||
if theme_style_params_path:
|
||||
update_data['style_params_gcs_path'] = theme_style_params_path
|
||||
|
||||
if dist.brand_prefix:
|
||||
update_data['brand_prefix'] = dist.brand_prefix
|
||||
if dist.discord_webhook_url:
|
||||
update_data['discord_webhook_url'] = dist.discord_webhook_url
|
||||
# Use theme YouTube description if no custom one provided
|
||||
effective_youtube_description = youtube_description or theme_youtube_desc
|
||||
if effective_youtube_description:
|
||||
update_data['youtube_description_template'] = effective_youtube_description
|
||||
|
||||
# Native API distribution (use effective values which include defaults)
|
||||
if effective_dropbox_path:
|
||||
update_data['dropbox_path'] = effective_dropbox_path
|
||||
if effective_gdrive_folder_id:
|
||||
update_data['gdrive_folder_id'] = effective_gdrive_folder_id
|
||||
if dist.dropbox_path:
|
||||
update_data['dropbox_path'] = dist.dropbox_path
|
||||
if dist.gdrive_folder_id:
|
||||
update_data['gdrive_folder_id'] = dist.gdrive_folder_id
|
||||
|
||||
# Legacy rclone distribution (deprecated)
|
||||
if organised_dir_rclone_root:
|
||||
@@ -668,33 +812,34 @@ async def upload_and_create_job(
|
||||
|
||||
# Build distribution services info for response
|
||||
distribution_services: Dict[str, Any] = {}
|
||||
|
||||
if effective_dropbox_path:
|
||||
|
||||
if dist.dropbox_path:
|
||||
dropbox_result = credential_manager.check_dropbox_credentials()
|
||||
distribution_services["dropbox"] = {
|
||||
"enabled": True,
|
||||
"path": effective_dropbox_path,
|
||||
"path": dist.dropbox_path,
|
||||
"credentials_valid": dropbox_result.status == CredentialStatus.VALID,
|
||||
"using_default": dropbox_path is None,
|
||||
}
|
||||
|
||||
if effective_gdrive_folder_id:
|
||||
|
||||
if dist.gdrive_folder_id:
|
||||
gdrive_result = credential_manager.check_gdrive_credentials()
|
||||
distribution_services["gdrive"] = {
|
||||
"enabled": True,
|
||||
"folder_id": effective_gdrive_folder_id,
|
||||
"folder_id": dist.gdrive_folder_id,
|
||||
"credentials_valid": gdrive_result.status == CredentialStatus.VALID,
|
||||
"using_default": gdrive_folder_id is None,
|
||||
}
|
||||
|
||||
if enable_youtube_upload:
|
||||
if effective_enable_youtube_upload:
|
||||
youtube_result = credential_manager.check_youtube_credentials()
|
||||
distribution_services["youtube"] = {
|
||||
"enabled": True,
|
||||
"credentials_valid": youtube_result.status == CredentialStatus.VALID,
|
||||
"using_default": enable_youtube_upload is None,
|
||||
}
|
||||
|
||||
if effective_discord_webhook_url:
|
||||
if dist.discord_webhook_url:
|
||||
distribution_services["discord"] = {
|
||||
"enabled": True,
|
||||
"using_default": discord_webhook_url is None,
|
||||
@@ -780,7 +925,7 @@ class CreateFinaliseOnlyJobRequest(BaseModel):
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
keep_brand_code: Optional[str] = Field(None, description="Preserve existing brand code from folder name")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
|
||||
@@ -868,7 +1013,7 @@ def _get_gcs_path_for_file(job_id: str, file_type: str, filename: str) -> str:
|
||||
async def create_job_with_upload_urls(
|
||||
request: Request,
|
||||
body: CreateJobWithUploadUrlsRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Create a karaoke generation job and return signed URLs for direct file upload to GCS.
|
||||
@@ -911,32 +1056,39 @@ async def create_job_with_upload_urls(
|
||||
status_code=400,
|
||||
detail=f"Invalid file extension '{ext}' for file_type '{file_info.file_type}'. Allowed: {allowed_extensions}"
|
||||
)
|
||||
|
||||
|
||||
# Apply default distribution settings from environment if not provided
|
||||
dist = _get_effective_distribution_settings(
|
||||
dropbox_path=body.dropbox_path,
|
||||
gdrive_folder_id=body.gdrive_folder_id,
|
||||
discord_webhook_url=body.discord_webhook_url,
|
||||
brand_prefix=body.brand_prefix,
|
||||
)
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_dropbox_path = body.dropbox_path or settings.default_dropbox_path
|
||||
effective_gdrive_folder_id = body.gdrive_folder_id or settings.default_gdrive_folder_id
|
||||
effective_discord_webhook_url = body.discord_webhook_url or settings.default_discord_webhook_url
|
||||
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
|
||||
# Validate credentials for requested distribution services
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if body.enable_youtube_upload:
|
||||
|
||||
if effective_enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
|
||||
if effective_dropbox_path:
|
||||
|
||||
if dist.dropbox_path:
|
||||
result = credential_manager.check_dropbox_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"dropbox ({result.message})")
|
||||
|
||||
if effective_gdrive_folder_id:
|
||||
|
||||
if dist.gdrive_folder_id:
|
||||
result = credential_manager.check_gdrive_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"gdrive ({result.message})")
|
||||
|
||||
|
||||
if invalid_services:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
@@ -947,38 +1099,51 @@ async def create_job_with_upload_urls(
|
||||
"auth_url": "/api/auth/status"
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# Extract request metadata for tracking
|
||||
request_metadata = extract_request_metadata(request, created_from="signed_url_upload")
|
||||
|
||||
# Get original audio filename
|
||||
audio_file = audio_files[0]
|
||||
|
||||
# Apply default theme if none specified
|
||||
# This ensures all karaoke videos use the Nomad theme by default
|
||||
effective_theme_id = body.theme_id
|
||||
if effective_theme_id is None:
|
||||
theme_service = get_theme_service()
|
||||
effective_theme_id = theme_service.get_default_theme_id()
|
||||
if effective_theme_id:
|
||||
logger.info(f"Applying default theme: {effective_theme_id}")
|
||||
|
||||
# Resolve CDG/TXT defaults based on theme
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
body.theme_id, body.enable_cdg, body.enable_txt
|
||||
effective_theme_id, body.enable_cdg, body.enable_txt
|
||||
)
|
||||
|
||||
# Check if style_params is being uploaded (overrides theme)
|
||||
has_style_params_upload = any(f.file_type == 'style_params' for f in body.files)
|
||||
|
||||
# Prefer authenticated user's email over request body
|
||||
effective_user_email = auth_result.user_email or body.user_email
|
||||
|
||||
# Create job
|
||||
job_create = JobCreate(
|
||||
artist=body.artist,
|
||||
title=body.title,
|
||||
filename=audio_file.filename,
|
||||
theme_id=body.theme_id,
|
||||
theme_id=effective_theme_id,
|
||||
color_overrides=body.color_overrides or {},
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=body.brand_prefix,
|
||||
enable_youtube_upload=body.enable_youtube_upload,
|
||||
brand_prefix=dist.brand_prefix,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=body.youtube_description,
|
||||
discord_webhook_url=effective_discord_webhook_url,
|
||||
youtube_description_template=body.youtube_description, # video_worker reads this field
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
webhook_url=body.webhook_url,
|
||||
user_email=body.user_email,
|
||||
dropbox_path=effective_dropbox_path,
|
||||
gdrive_folder_id=effective_gdrive_folder_id,
|
||||
user_email=effective_user_email,
|
||||
dropbox_path=dist.dropbox_path,
|
||||
gdrive_folder_id=dist.gdrive_folder_id,
|
||||
organised_dir_rclone_root=body.organised_dir_rclone_root,
|
||||
lyrics_artist=body.lyrics_artist,
|
||||
lyrics_title=body.lyrics_title,
|
||||
@@ -987,6 +1152,7 @@ async def create_job_with_upload_urls(
|
||||
backing_vocals_models=body.backing_vocals_models,
|
||||
other_stems_models=body.other_stems_models,
|
||||
request_metadata=request_metadata,
|
||||
non_interactive=body.non_interactive,
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job_id = job.job_id
|
||||
@@ -997,9 +1163,9 @@ async def create_job_with_upload_urls(
|
||||
logger.info(f"Created job {job_id} for {body.artist} - {body.title} (signed URL upload flow)")
|
||||
|
||||
# If theme is set and no style_params uploaded, prepare theme style now
|
||||
if body.theme_id and not has_style_params_upload:
|
||||
if effective_theme_id and not has_style_params_upload:
|
||||
style_params_path, style_assets, youtube_desc = _prepare_theme_for_job(
|
||||
job_id, body.theme_id, body.color_overrides
|
||||
job_id, effective_theme_id, body.color_overrides
|
||||
)
|
||||
# Update job with theme style data
|
||||
update_data = {
|
||||
@@ -1009,7 +1175,7 @@ async def create_job_with_upload_urls(
|
||||
if youtube_desc and not body.youtube_description:
|
||||
update_data['youtube_description_template'] = youtube_desc
|
||||
job_manager.update_job(job_id, update_data)
|
||||
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
|
||||
logger.info(f"Applied theme '{effective_theme_id}' to job {job_id}")
|
||||
|
||||
# Generate signed upload URLs for each file
|
||||
upload_urls = []
|
||||
@@ -1052,7 +1218,7 @@ async def mark_uploads_complete(
|
||||
job_id: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: UploadsCompleteRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Mark file uploads as complete and start job processing.
|
||||
@@ -1298,7 +1464,7 @@ async def create_job_from_url(
|
||||
request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: CreateJobFromUrlRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Create a karaoke generation job from a YouTube or other online video URL.
|
||||
@@ -1322,32 +1488,39 @@ async def create_job_from_url(
|
||||
status_code=400,
|
||||
detail="Invalid URL. Please provide a valid YouTube, Vimeo, SoundCloud, or other supported video URL."
|
||||
)
|
||||
|
||||
|
||||
# Apply default distribution settings from environment if not provided
|
||||
dist = _get_effective_distribution_settings(
|
||||
dropbox_path=body.dropbox_path,
|
||||
gdrive_folder_id=body.gdrive_folder_id,
|
||||
discord_webhook_url=body.discord_webhook_url,
|
||||
brand_prefix=body.brand_prefix,
|
||||
)
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_dropbox_path = body.dropbox_path or settings.default_dropbox_path
|
||||
effective_gdrive_folder_id = body.gdrive_folder_id or settings.default_gdrive_folder_id
|
||||
effective_discord_webhook_url = body.discord_webhook_url or settings.default_discord_webhook_url
|
||||
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
|
||||
# Validate credentials for requested distribution services
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if body.enable_youtube_upload:
|
||||
|
||||
if effective_enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
|
||||
if effective_dropbox_path:
|
||||
|
||||
if dist.dropbox_path:
|
||||
result = credential_manager.check_dropbox_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"dropbox ({result.message})")
|
||||
|
||||
if effective_gdrive_folder_id:
|
||||
|
||||
if dist.gdrive_folder_id:
|
||||
result = credential_manager.check_gdrive_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"gdrive ({result.message})")
|
||||
|
||||
|
||||
if invalid_services:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
@@ -1358,7 +1531,7 @@ async def create_job_from_url(
|
||||
"auth_url": "/api/auth/status"
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# Extract request metadata for tracking
|
||||
request_metadata = extract_request_metadata(request, created_from="url")
|
||||
|
||||
@@ -1366,29 +1539,41 @@ async def create_job_from_url(
|
||||
artist = body.artist
|
||||
title = body.title
|
||||
|
||||
# Apply default theme if none specified
|
||||
# This ensures all karaoke videos use the Nomad theme by default
|
||||
effective_theme_id = body.theme_id
|
||||
if effective_theme_id is None:
|
||||
theme_service = get_theme_service()
|
||||
effective_theme_id = theme_service.get_default_theme_id()
|
||||
if effective_theme_id:
|
||||
logger.info(f"Applying default theme: {effective_theme_id}")
|
||||
|
||||
# Resolve CDG/TXT defaults based on theme
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
body.theme_id, body.enable_cdg, body.enable_txt
|
||||
effective_theme_id, body.enable_cdg, body.enable_txt
|
||||
)
|
||||
|
||||
# Prefer authenticated user's email over request body
|
||||
effective_user_email = auth_result.user_email or body.user_email
|
||||
|
||||
# Create job with URL
|
||||
job_create = JobCreate(
|
||||
url=body.url,
|
||||
artist=artist,
|
||||
title=title,
|
||||
filename=None, # No file uploaded
|
||||
theme_id=body.theme_id,
|
||||
theme_id=effective_theme_id,
|
||||
color_overrides=body.color_overrides or {},
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=body.brand_prefix,
|
||||
enable_youtube_upload=body.enable_youtube_upload,
|
||||
brand_prefix=dist.brand_prefix,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=body.youtube_description,
|
||||
discord_webhook_url=effective_discord_webhook_url,
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
webhook_url=body.webhook_url,
|
||||
user_email=body.user_email,
|
||||
dropbox_path=effective_dropbox_path,
|
||||
gdrive_folder_id=effective_gdrive_folder_id,
|
||||
user_email=effective_user_email,
|
||||
dropbox_path=dist.dropbox_path,
|
||||
gdrive_folder_id=dist.gdrive_folder_id,
|
||||
organised_dir_rclone_root=body.organised_dir_rclone_root,
|
||||
lyrics_artist=body.lyrics_artist,
|
||||
lyrics_title=body.lyrics_title,
|
||||
@@ -1397,6 +1582,7 @@ async def create_job_from_url(
|
||||
backing_vocals_models=body.backing_vocals_models,
|
||||
other_stems_models=body.other_stems_models,
|
||||
request_metadata=request_metadata,
|
||||
non_interactive=body.non_interactive,
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job_id = job.job_id
|
||||
@@ -1405,9 +1591,9 @@ async def create_job_from_url(
|
||||
metrics.record_job_created(job_id, source="url")
|
||||
|
||||
# If theme is set, prepare theme style now
|
||||
if body.theme_id:
|
||||
if effective_theme_id:
|
||||
style_params_path, style_assets, youtube_desc = _prepare_theme_for_job(
|
||||
job_id, body.theme_id, body.color_overrides
|
||||
job_id, effective_theme_id, body.color_overrides
|
||||
)
|
||||
# Update job with theme style data
|
||||
update_data = {
|
||||
@@ -1417,7 +1603,7 @@ async def create_job_from_url(
|
||||
if youtube_desc and not body.youtube_description:
|
||||
update_data['youtube_description_template'] = youtube_desc
|
||||
job_manager.update_job(job_id, update_data)
|
||||
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
|
||||
logger.info(f"Applied theme '{effective_theme_id}' to job {job_id}")
|
||||
|
||||
logger.info(f"Created URL-based job {job_id} for URL: {body.url}")
|
||||
if artist:
|
||||
@@ -1432,10 +1618,11 @@ async def create_job_from_url(
|
||||
progress=5,
|
||||
message="Starting audio download from URL"
|
||||
)
|
||||
|
||||
# Trigger workers in parallel
|
||||
# The audio worker will handle downloading from URL
|
||||
background_tasks.add_task(_trigger_workers_parallel, job_id)
|
||||
|
||||
# For URL jobs, trigger ONLY audio worker first
|
||||
# The audio worker will trigger lyrics worker after download completes
|
||||
# This prevents race condition where lyrics worker times out waiting for audio
|
||||
background_tasks.add_task(_trigger_audio_worker_only, job_id)
|
||||
|
||||
return CreateJobFromUrlResponse(
|
||||
status="success",
|
||||
@@ -1498,7 +1685,7 @@ def _get_gcs_path_for_finalise_file(job_id: str, file_type: str, filename: str)
|
||||
async def create_finalise_only_job(
|
||||
request: Request,
|
||||
body: CreateFinaliseOnlyJobRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Create a finalise-only job for continuing from a local prep phase.
|
||||
@@ -1563,30 +1750,37 @@ async def create_finalise_only_job(
|
||||
)
|
||||
|
||||
# Apply default distribution settings
|
||||
dist = _get_effective_distribution_settings(
|
||||
dropbox_path=body.dropbox_path,
|
||||
gdrive_folder_id=body.gdrive_folder_id,
|
||||
discord_webhook_url=body.discord_webhook_url,
|
||||
brand_prefix=body.brand_prefix,
|
||||
)
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_dropbox_path = body.dropbox_path or settings.default_dropbox_path
|
||||
effective_gdrive_folder_id = body.gdrive_folder_id or settings.default_gdrive_folder_id
|
||||
effective_discord_webhook_url = body.discord_webhook_url or settings.default_discord_webhook_url
|
||||
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
|
||||
# Validate distribution credentials if services are requested
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if body.enable_youtube_upload:
|
||||
|
||||
if effective_enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
|
||||
if effective_dropbox_path:
|
||||
|
||||
if dist.dropbox_path:
|
||||
result = credential_manager.check_dropbox_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"dropbox ({result.message})")
|
||||
|
||||
if effective_gdrive_folder_id:
|
||||
|
||||
if dist.gdrive_folder_id:
|
||||
result = credential_manager.check_gdrive_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"gdrive ({result.message})")
|
||||
|
||||
|
||||
if invalid_services:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
@@ -1597,33 +1791,46 @@ async def create_finalise_only_job(
|
||||
"auth_url": "/api/auth/status"
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# Extract request metadata
|
||||
request_metadata = extract_request_metadata(request, created_from="finalise_only_upload")
|
||||
|
||||
# Apply default theme if none specified
|
||||
# This ensures all karaoke videos use the Nomad theme by default
|
||||
effective_theme_id = body.theme_id
|
||||
if effective_theme_id is None:
|
||||
theme_service = get_theme_service()
|
||||
effective_theme_id = theme_service.get_default_theme_id()
|
||||
if effective_theme_id:
|
||||
logger.info(f"Applying default theme: {effective_theme_id}")
|
||||
|
||||
# Resolve CDG/TXT defaults based on theme
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
body.theme_id, body.enable_cdg, body.enable_txt
|
||||
effective_theme_id, body.enable_cdg, body.enable_txt
|
||||
)
|
||||
|
||||
# Check if style_params is being uploaded (overrides theme)
|
||||
has_style_params_upload = any(f.file_type == 'style_params' for f in body.files)
|
||||
|
||||
# Use authenticated user's email
|
||||
effective_user_email = auth_result.user_email
|
||||
|
||||
# Create job with finalise_only=True
|
||||
job_create = JobCreate(
|
||||
artist=body.artist,
|
||||
title=body.title,
|
||||
filename="finalise_only", # No single audio file - using prep outputs
|
||||
theme_id=body.theme_id,
|
||||
theme_id=effective_theme_id,
|
||||
color_overrides=body.color_overrides or {},
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=body.brand_prefix,
|
||||
enable_youtube_upload=body.enable_youtube_upload,
|
||||
brand_prefix=dist.brand_prefix,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=body.youtube_description,
|
||||
discord_webhook_url=effective_discord_webhook_url,
|
||||
dropbox_path=effective_dropbox_path,
|
||||
gdrive_folder_id=effective_gdrive_folder_id,
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
dropbox_path=dist.dropbox_path,
|
||||
gdrive_folder_id=dist.gdrive_folder_id,
|
||||
user_email=effective_user_email,
|
||||
finalise_only=True,
|
||||
keep_brand_code=body.keep_brand_code,
|
||||
request_metadata=request_metadata,
|
||||
@@ -1637,9 +1844,9 @@ async def create_finalise_only_job(
|
||||
logger.info(f"Created finalise-only job {job_id} for {body.artist} - {body.title}")
|
||||
|
||||
# If theme is set and no style_params uploaded, prepare theme style now
|
||||
if body.theme_id and not has_style_params_upload:
|
||||
if effective_theme_id and not has_style_params_upload:
|
||||
style_params_path, style_assets, youtube_desc = _prepare_theme_for_job(
|
||||
job_id, body.theme_id, body.color_overrides
|
||||
job_id, effective_theme_id, body.color_overrides
|
||||
)
|
||||
# Update job with theme style data
|
||||
update_data = {
|
||||
@@ -1649,7 +1856,7 @@ async def create_finalise_only_job(
|
||||
if youtube_desc and not body.youtube_description:
|
||||
update_data['youtube_description_template'] = youtube_desc
|
||||
job_manager.update_job(job_id, update_data)
|
||||
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
|
||||
logger.info(f"Applied theme '{effective_theme_id}' to job {job_id}")
|
||||
|
||||
# Generate signed upload URLs for each file
|
||||
upload_urls = []
|
||||
@@ -1692,7 +1899,7 @@ async def mark_finalise_uploads_complete(
|
||||
job_id: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: UploadsCompleteRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Mark finalise-only file uploads as complete and start video generation.
|
||||
|
||||
@@ -3,16 +3,126 @@ Health check routes.
|
||||
"""
|
||||
import os
|
||||
import logging
|
||||
import subprocess
|
||||
import platform
|
||||
from fastapi import APIRouter
|
||||
from typing import Dict, Any, Optional
|
||||
|
||||
from backend.version import VERSION
|
||||
from backend.services.flacfetch_client import get_flacfetch_client
|
||||
from backend.services.email_service import get_email_service
|
||||
from backend.services.stripe_service import get_stripe_service
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
from backend.services.spacy_preloader import get_preloaded_model, is_model_preloaded
|
||||
from backend.services.nltk_preloader import get_preloaded_cmudict, is_cmudict_preloaded
|
||||
from backend.services.langfuse_preloader import get_preloaded_langfuse_handler, is_langfuse_preloaded, is_langfuse_configured
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def get_system_info() -> Dict[str, Any]:
|
||||
"""Get system information for performance analysis."""
|
||||
info = {
|
||||
"platform": platform.system(),
|
||||
"platform_release": platform.release(),
|
||||
"python_version": platform.python_version(),
|
||||
}
|
||||
|
||||
# Get CPU info
|
||||
try:
|
||||
cpu_count = os.cpu_count() or 0
|
||||
info["cpu_count"] = cpu_count
|
||||
|
||||
# Try to get more detailed CPU info on Linux
|
||||
if platform.system() == "Linux":
|
||||
try:
|
||||
with open("/proc/cpuinfo", "r") as f:
|
||||
for line in f:
|
||||
if "model name" in line:
|
||||
info["cpu_model"] = line.split(":")[1].strip()
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as e:
|
||||
info["cpu_error"] = str(e)
|
||||
|
||||
# Get memory info
|
||||
try:
|
||||
if platform.system() == "Linux":
|
||||
with open("/proc/meminfo", "r") as f:
|
||||
for line in f:
|
||||
if line.startswith("MemTotal:"):
|
||||
mem_kb = int(line.split()[1])
|
||||
info["memory_gb"] = round(mem_kb / (1024 * 1024), 1)
|
||||
break
|
||||
except Exception as e:
|
||||
info["memory_error"] = str(e)
|
||||
|
||||
return info
|
||||
|
||||
|
||||
def get_ffmpeg_info() -> Dict[str, Any]:
|
||||
"""Get FFmpeg version and configuration for debugging encoding performance."""
|
||||
try:
|
||||
# Get FFmpeg version
|
||||
result = subprocess.run(
|
||||
["ffmpeg", "-version"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
if result.returncode == 0:
|
||||
lines = result.stdout.split("\n")
|
||||
version_line = lines[0] if lines else "unknown"
|
||||
|
||||
# Check for hardware acceleration support
|
||||
hw_accel = {}
|
||||
config_result = subprocess.run(
|
||||
["ffmpeg", "-hwaccels"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
if config_result.returncode == 0:
|
||||
hw_lines = config_result.stdout.strip().split("\n")
|
||||
hw_accel["available"] = [h.strip() for h in hw_lines[1:] if h.strip()]
|
||||
|
||||
# Check encoder availability
|
||||
encoders = {}
|
||||
encoder_result = subprocess.run(
|
||||
["ffmpeg", "-encoders"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
if encoder_result.returncode == 0:
|
||||
encoder_output = encoder_result.stdout
|
||||
# Check for specific encoders we care about
|
||||
encoders["libx264"] = "libx264" in encoder_output
|
||||
encoders["h264_nvenc"] = "h264_nvenc" in encoder_output
|
||||
encoders["h264_vaapi"] = "h264_vaapi" in encoder_output
|
||||
encoders["hevc_nvenc"] = "hevc_nvenc" in encoder_output
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"version": version_line,
|
||||
"hw_acceleration": hw_accel,
|
||||
"encoders": encoders,
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"available": False,
|
||||
"error": result.stderr[:500] if result.stderr else "Unknown error",
|
||||
}
|
||||
except subprocess.TimeoutExpired:
|
||||
return {"available": False, "error": "Timeout getting FFmpeg info"}
|
||||
except FileNotFoundError:
|
||||
return {"available": False, "error": "FFmpeg not found"}
|
||||
except Exception as e:
|
||||
return {"available": False, "error": str(e)}
|
||||
|
||||
|
||||
def check_transmission_status() -> Dict[str, Any]:
|
||||
"""Check if Transmission daemon is available and responsive."""
|
||||
try:
|
||||
@@ -99,34 +209,192 @@ async def check_flacfetch_service_status() -> Dict[str, Any]:
|
||||
}
|
||||
|
||||
|
||||
async def check_encoding_worker_status() -> Dict[str, Any]:
|
||||
"""Check if GCE encoding worker is available and healthy."""
|
||||
encoding_service = get_encoding_service()
|
||||
|
||||
if not encoding_service.is_configured:
|
||||
return {
|
||||
"configured": False,
|
||||
"enabled": encoding_service.settings.use_gce_encoding,
|
||||
"message": "GCE encoding worker not configured (ENCODING_WORKER_URL or API key not set)",
|
||||
}
|
||||
|
||||
try:
|
||||
health = await encoding_service.health_check()
|
||||
return {
|
||||
"configured": True,
|
||||
"enabled": encoding_service.is_enabled,
|
||||
"available": health.get("status") == "ok",
|
||||
"status": health.get("status"),
|
||||
"active_jobs": health.get("active_jobs", 0),
|
||||
"queue_length": health.get("queue_length", 0),
|
||||
"ffmpeg_version": health.get("ffmpeg_version"),
|
||||
"wheel_version": health.get("wheel_version"),
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"configured": True,
|
||||
"enabled": encoding_service.is_enabled,
|
||||
"available": False,
|
||||
"error": str(e),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/encoding-worker")
|
||||
async def encoding_worker_health() -> Dict[str, Any]:
|
||||
"""
|
||||
Lightweight endpoint to check encoding worker status.
|
||||
|
||||
Returns minimal info for frontend footer display.
|
||||
No authentication required.
|
||||
"""
|
||||
status = await check_encoding_worker_status()
|
||||
|
||||
# Return simplified response for frontend
|
||||
if not status.get("configured"):
|
||||
return {
|
||||
"available": False,
|
||||
"status": "not_configured",
|
||||
}
|
||||
|
||||
if not status.get("available"):
|
||||
return {
|
||||
"available": False,
|
||||
"status": "offline",
|
||||
"error": status.get("error"),
|
||||
}
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"status": "ok",
|
||||
"version": status.get("wheel_version"),
|
||||
"active_jobs": status.get("active_jobs", 0),
|
||||
"queue_length": status.get("queue_length", 0),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/detailed")
|
||||
async def detailed_health_check() -> Dict[str, Any]:
|
||||
"""
|
||||
Detailed health check including dependencies.
|
||||
|
||||
|
||||
Use this to debug issues with Transmission, flacfetch service, etc.
|
||||
"""
|
||||
transmission_status = check_transmission_status()
|
||||
flacfetch_status = await check_flacfetch_service_status()
|
||||
|
||||
encoding_status = await check_encoding_worker_status()
|
||||
|
||||
# Check email service
|
||||
email_service = get_email_service()
|
||||
email_status = {
|
||||
"configured": email_service.is_configured(),
|
||||
"provider": type(email_service.provider).__name__,
|
||||
}
|
||||
|
||||
# Check Stripe service
|
||||
stripe_service = get_stripe_service()
|
||||
stripe_status = {
|
||||
"configured": stripe_service.is_configured(),
|
||||
}
|
||||
|
||||
# Log for debugging
|
||||
if not transmission_status.get("available"):
|
||||
logger.warning(f"Local Transmission not available: {transmission_status.get('error')}")
|
||||
else:
|
||||
logger.info(f"Local Transmission available at {transmission_status.get('host')}:{transmission_status.get('port')}")
|
||||
|
||||
|
||||
if flacfetch_status.get("configured") and not flacfetch_status.get("available"):
|
||||
logger.warning(f"Remote flacfetch service not available: {flacfetch_status.get('error')}")
|
||||
elif flacfetch_status.get("available"):
|
||||
logger.info(f"Remote flacfetch service healthy: {flacfetch_status.get('status')}")
|
||||
|
||||
|
||||
if not email_status["configured"]:
|
||||
logger.warning("Email service not configured - magic links will not work")
|
||||
|
||||
if not stripe_status["configured"]:
|
||||
logger.warning("Stripe service not configured - payments will not work")
|
||||
|
||||
# Get system and FFmpeg info for performance analysis
|
||||
system_info = get_system_info()
|
||||
ffmpeg_info = get_ffmpeg_info()
|
||||
|
||||
return {
|
||||
"status": "healthy",
|
||||
"service": "karaoke-gen-backend",
|
||||
"version": VERSION,
|
||||
"system": system_info,
|
||||
"ffmpeg": ffmpeg_info,
|
||||
"dependencies": {
|
||||
"transmission_local": transmission_status,
|
||||
"flacfetch_remote": flacfetch_status,
|
||||
"encoding_worker": encoding_status,
|
||||
},
|
||||
"services": {
|
||||
"email": email_status,
|
||||
"stripe": stripe_status,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/preload-status")
|
||||
async def preload_status() -> Dict[str, Any]:
|
||||
"""
|
||||
Check status of preloaded resources for performance optimization.
|
||||
|
||||
Use this endpoint to verify that NLTK, SpaCy, and Langfuse resources
|
||||
were successfully preloaded at container startup. If any show as
|
||||
not preloaded, check Cloud Run startup logs for errors.
|
||||
|
||||
Expected state after successful deployment:
|
||||
- spacy.preloaded: true
|
||||
- nltk.preloaded: true
|
||||
- langfuse.preloaded: true (if configured) or configured: false
|
||||
"""
|
||||
# SpaCy status
|
||||
spacy_model = get_preloaded_model("en_core_web_sm")
|
||||
spacy_status = {
|
||||
"preloaded": is_model_preloaded("en_core_web_sm"),
|
||||
"model": "en_core_web_sm",
|
||||
}
|
||||
if spacy_model:
|
||||
spacy_status["vocab_size"] = len(spacy_model.vocab)
|
||||
|
||||
# NLTK status
|
||||
cmudict = get_preloaded_cmudict()
|
||||
nltk_status = {
|
||||
"preloaded": is_cmudict_preloaded(),
|
||||
"resource": "cmudict",
|
||||
}
|
||||
if cmudict:
|
||||
nltk_status["entries"] = len(cmudict)
|
||||
|
||||
# Langfuse status
|
||||
langfuse_handler = get_preloaded_langfuse_handler()
|
||||
langfuse_status = {
|
||||
"configured": is_langfuse_configured(),
|
||||
"preloaded": is_langfuse_preloaded(),
|
||||
}
|
||||
if langfuse_handler:
|
||||
langfuse_status["handler_type"] = type(langfuse_handler).__name__
|
||||
|
||||
# Overall status
|
||||
all_preloaded = (
|
||||
spacy_status["preloaded"]
|
||||
and nltk_status["preloaded"]
|
||||
and (langfuse_status["preloaded"] or not langfuse_status["configured"])
|
||||
)
|
||||
|
||||
return {
|
||||
"status": "ok" if all_preloaded else "degraded",
|
||||
"message": "All resources preloaded" if all_preloaded else "Some resources not preloaded - check startup logs",
|
||||
"spacy": spacy_status,
|
||||
"nltk": nltk_status,
|
||||
"langfuse": langfuse_status,
|
||||
"performance_impact": {
|
||||
"spacy_preload": "Saves ~60s on first lyrics correction",
|
||||
"nltk_preload": "Saves ~100-150s on SyllablesMatchHandler init",
|
||||
"langfuse_preload": "Saves ~200s on AgenticCorrector init",
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -316,13 +316,118 @@ async def trigger_render_video_worker(
|
||||
)
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/check-idle-reminder")
|
||||
async def check_idle_reminder(
|
||||
job_id: str,
|
||||
http_request: Request,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Check if a job needs an idle reminder email.
|
||||
|
||||
This endpoint is called by a Cloud Tasks scheduled task 5 minutes after
|
||||
a job enters a blocking state (AWAITING_REVIEW or AWAITING_INSTRUMENTAL_SELECTION).
|
||||
|
||||
If the job is still in the blocking state and no reminder has been sent yet,
|
||||
sends a reminder email to the user.
|
||||
|
||||
Idempotency: Only one reminder per job (tracked via reminder_sent flag).
|
||||
"""
|
||||
from backend.models.job import JobStatus
|
||||
from backend.services.job_notification_service import get_job_notification_service
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] IDLE_REMINDER_CHECK starting")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("operation", "idle_reminder_check")
|
||||
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
logger.warning(f"[job:{job_id}] Job not found for idle reminder check")
|
||||
add_span_event("job_not_found")
|
||||
return {"status": "not_found", "job_id": job_id, "message": "Job not found"}
|
||||
|
||||
# Check if job is still in a blocking state
|
||||
blocking_states = [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION]
|
||||
if job.status not in [s.value for s in blocking_states]:
|
||||
logger.info(f"[job:{job_id}] Job no longer in blocking state ({job.status}), skipping reminder")
|
||||
add_span_event("not_blocking", {"current_status": job.status})
|
||||
return {
|
||||
"status": "skipped",
|
||||
"job_id": job_id,
|
||||
"message": f"Job not in blocking state (current: {job.status})"
|
||||
}
|
||||
|
||||
# Normalize state_data to prevent None errors
|
||||
state_data = job.state_data or {}
|
||||
|
||||
# Check if reminder was already sent (idempotency)
|
||||
if state_data.get('reminder_sent'):
|
||||
logger.info(f"[job:{job_id}] Reminder already sent, skipping")
|
||||
add_span_event("already_sent")
|
||||
return {"status": "already_sent", "job_id": job_id, "message": "Reminder already sent"}
|
||||
|
||||
# Check if user has an email
|
||||
if not job.user_email:
|
||||
logger.warning(f"[job:{job_id}] No user email, cannot send reminder")
|
||||
add_span_event("no_email")
|
||||
return {"status": "no_email", "job_id": job_id, "message": "No user email configured"}
|
||||
|
||||
# Determine action type
|
||||
action_type = state_data.get('blocking_action_type')
|
||||
if not action_type:
|
||||
action_type = "lyrics" if job.status == JobStatus.AWAITING_REVIEW.value else "instrumental"
|
||||
|
||||
# Send the reminder email
|
||||
try:
|
||||
notification_service = get_job_notification_service()
|
||||
|
||||
success = await notification_service.send_action_reminder_email(
|
||||
job_id=job.job_id,
|
||||
user_email=job.user_email,
|
||||
action_type=action_type,
|
||||
user_name=None, # Could fetch from user service if needed
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
audio_hash=job.audio_hash,
|
||||
review_token=job.review_token,
|
||||
instrumental_token=job.instrumental_token,
|
||||
)
|
||||
|
||||
if success:
|
||||
# Mark reminder as sent (prevents duplicate sends)
|
||||
job_manager.firestore.update_job(job_id, {
|
||||
'state_data': {**state_data, 'reminder_sent': True}
|
||||
})
|
||||
logger.info(f"[job:{job_id}] Sent {action_type} reminder email to {job.user_email}")
|
||||
add_span_event("reminder_sent", {"action_type": action_type})
|
||||
return {
|
||||
"status": "sent",
|
||||
"job_id": job_id,
|
||||
"message": f"Sent {action_type} reminder to {job.user_email}"
|
||||
}
|
||||
else:
|
||||
logger.error(f"[job:{job_id}] Failed to send reminder email")
|
||||
add_span_event("send_failed")
|
||||
return {"status": "failed", "job_id": job_id, "message": "Failed to send reminder"}
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"[job:{job_id}] Error sending reminder: {e}")
|
||||
add_span_event("error", {"error": str(e)})
|
||||
return {"status": "error", "job_id": job_id, "message": str(e)}
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def internal_health(
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Internal health check endpoint.
|
||||
|
||||
|
||||
Used to verify the internal API is responsive.
|
||||
Requires admin authentication.
|
||||
"""
|
||||
|
||||
+356
-63
@@ -25,10 +25,12 @@ from backend.models.requests import (
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.worker_service import get_worker_service
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.theme_service import get_theme_service
|
||||
from backend.config import get_settings
|
||||
from backend.api.dependencies import require_admin, require_auth, require_instrumental_auth
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.auth_service import UserType, AuthResult
|
||||
from backend.services.metrics import metrics
|
||||
from backend.utils.test_data import is_test_email
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -57,7 +59,7 @@ async def _trigger_workers_parallel(job_id: str) -> None:
|
||||
async def create_job(
|
||||
request: URLSubmissionRequest,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> JobResponse:
|
||||
"""
|
||||
Create a new karaoke generation job from a URL.
|
||||
@@ -69,17 +71,51 @@ async def create_job(
|
||||
4. When both complete, job transitions to AWAITING_REVIEW
|
||||
"""
|
||||
try:
|
||||
# Determine job owner email:
|
||||
# All authentication methods must provide a user_email for job ownership
|
||||
if auth_result.user_email:
|
||||
# Use authenticated user's email (standard case)
|
||||
user_email = auth_result.user_email
|
||||
else:
|
||||
# This should never happen - all auth methods now require user_email
|
||||
logger.error("Authentication succeeded but no user_email provided")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Authentication error: no user identity available"
|
||||
)
|
||||
|
||||
# Admins can optionally create jobs on behalf of other users
|
||||
if request.user_email and auth_result.is_admin and request.user_email != auth_result.user_email:
|
||||
user_email = request.user_email
|
||||
logger.info(f"Admin {auth_result.user_email} creating job on behalf of {user_email}")
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_enable_youtube_upload = request.enable_youtube_upload if request.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
|
||||
# Apply default theme - all jobs require a theme
|
||||
theme_service = get_theme_service()
|
||||
effective_theme_id = theme_service.get_default_theme_id()
|
||||
if not effective_theme_id:
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail="No default theme configured. Please contact support or specify a theme_id."
|
||||
)
|
||||
logger.info(f"Applying default theme: {effective_theme_id}")
|
||||
|
||||
# Create job with all preferences
|
||||
job_create = JobCreate(
|
||||
url=str(request.url),
|
||||
artist=request.artist,
|
||||
title=request.title,
|
||||
theme_id=effective_theme_id, # Required - all jobs must have a theme
|
||||
enable_cdg=request.enable_cdg,
|
||||
enable_txt=request.enable_txt,
|
||||
enable_youtube_upload=request.enable_youtube_upload,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=request.youtube_description,
|
||||
webhook_url=request.webhook_url,
|
||||
user_email=request.user_email
|
||||
user_email=user_email
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
|
||||
@@ -109,20 +145,49 @@ async def create_job(
|
||||
@router.get("/{job_id}", response_model=Job)
|
||||
async def get_job(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> Job:
|
||||
"""Get job status and details."""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Check ownership - users can only see their own jobs, admins can see all
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to access this job")
|
||||
|
||||
# If job is complete, include download URLs
|
||||
if job.status == JobStatus.COMPLETE:
|
||||
job.download_urls = job_manager.get_output_urls(job_id)
|
||||
|
||||
|
||||
return job
|
||||
|
||||
|
||||
def _check_job_ownership(job: Job, auth_result: AuthResult) -> bool:
|
||||
"""
|
||||
Check if the authenticated user owns the job or has admin access.
|
||||
|
||||
Returns:
|
||||
True if user can access the job, False otherwise
|
||||
"""
|
||||
# Admins can access all jobs
|
||||
if auth_result.is_admin:
|
||||
return True
|
||||
|
||||
# Check if user owns the job
|
||||
if auth_result.user_email and job.user_email:
|
||||
return auth_result.user_email.lower() == job.user_email.lower()
|
||||
|
||||
# If no user_email on auth (token auth without email), deny access to jobs with user_email
|
||||
# This prevents token-based auth from accessing user jobs
|
||||
if job.user_email:
|
||||
return False
|
||||
|
||||
# Legacy jobs without user_email - allow access for backward compatibility
|
||||
# TODO: Consider restricting this in the future
|
||||
return True
|
||||
|
||||
|
||||
@router.get("", response_model=List[Job])
|
||||
async def list_jobs(
|
||||
status: Optional[JobStatus] = None,
|
||||
@@ -130,50 +195,73 @@ async def list_jobs(
|
||||
client_id: Optional[str] = None,
|
||||
created_after: Optional[str] = None,
|
||||
created_before: Optional[str] = None,
|
||||
exclude_test: bool = True,
|
||||
limit: int = 100,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> List[Job]:
|
||||
"""
|
||||
List jobs with optional filters.
|
||||
|
||||
|
||||
Regular users only see their own jobs. Admins see all jobs.
|
||||
|
||||
Args:
|
||||
status: Filter by job status (pending, complete, failed, etc.)
|
||||
environment: Filter by request_metadata.environment (test/production/development)
|
||||
client_id: Filter by request_metadata.client_id (customer identifier)
|
||||
created_after: Filter jobs created after this ISO datetime (e.g., 2024-01-01T00:00:00Z)
|
||||
created_before: Filter jobs created before this ISO datetime
|
||||
exclude_test: If True (default), exclude jobs from test users (admin only)
|
||||
limit: Maximum number of jobs to return (default 100)
|
||||
|
||||
|
||||
Returns:
|
||||
List of jobs matching filters, ordered by created_at descending
|
||||
"""
|
||||
from datetime import datetime
|
||||
|
||||
|
||||
try:
|
||||
# Parse datetime strings if provided
|
||||
created_after_dt = None
|
||||
created_before_dt = None
|
||||
|
||||
|
||||
if created_after:
|
||||
try:
|
||||
created_after_dt = datetime.fromisoformat(created_after.replace('Z', '+00:00'))
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid created_after format: {created_after}")
|
||||
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid created_after format: {created_after}") from e
|
||||
|
||||
if created_before:
|
||||
try:
|
||||
created_before_dt = datetime.fromisoformat(created_before.replace('Z', '+00:00'))
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid created_before format: {created_before}")
|
||||
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid created_before format: {created_before}") from e
|
||||
|
||||
# Determine user_email filter based on admin status
|
||||
# Admins see all jobs, regular users only see their own
|
||||
user_email_filter = None
|
||||
if not auth_result.is_admin:
|
||||
if auth_result.user_email:
|
||||
user_email_filter = auth_result.user_email
|
||||
logger.debug(f"Filtering jobs for user: {user_email_filter}")
|
||||
else:
|
||||
# Token-based auth without user email - show no jobs for security
|
||||
logger.warning("Non-admin auth without user_email, returning empty job list")
|
||||
return []
|
||||
|
||||
jobs = job_manager.list_jobs(
|
||||
status=status,
|
||||
environment=environment,
|
||||
client_id=client_id,
|
||||
created_after=created_after_dt,
|
||||
created_before=created_before_dt,
|
||||
user_email=user_email_filter,
|
||||
limit=limit
|
||||
)
|
||||
|
||||
# Filter out test user jobs if exclude_test is True (admin only)
|
||||
if exclude_test and auth_result.is_admin:
|
||||
jobs = [j for j in jobs if not is_test_email(j.user_email or "")]
|
||||
|
||||
logger.debug(f"Listed {len(jobs)} jobs for user={auth_result.user_email}, admin={auth_result.is_admin}")
|
||||
return jobs
|
||||
except HTTPException:
|
||||
raise
|
||||
@@ -186,16 +274,20 @@ async def list_jobs(
|
||||
async def delete_job(
|
||||
job_id: str,
|
||||
delete_files: bool = True,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""Delete a job and optionally its output files."""
|
||||
try:
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Check ownership - users can only delete their own jobs
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to delete this job")
|
||||
|
||||
job_manager.delete_job(job_id, delete_files=delete_files)
|
||||
|
||||
|
||||
return {"status": "success", "message": f"Job {job_id} deleted"}
|
||||
except HTTPException:
|
||||
raise
|
||||
@@ -212,7 +304,7 @@ async def bulk_delete_jobs(
|
||||
created_before: Optional[str] = None,
|
||||
delete_files: bool = True,
|
||||
confirm: bool = False,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
_auth_result: AuthResult = Depends(require_admin)
|
||||
) -> dict:
|
||||
"""
|
||||
Delete multiple jobs matching filter criteria.
|
||||
@@ -316,18 +408,22 @@ async def bulk_delete_jobs(
|
||||
@router.get("/{job_id}/review-data")
|
||||
async def get_review_data(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Get data needed for lyrics review interface.
|
||||
|
||||
|
||||
Returns corrections JSON URL and audio URL.
|
||||
Frontend loads these to render the review UI.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Check ownership
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to access this job")
|
||||
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
@@ -376,13 +472,21 @@ async def get_review_data(
|
||||
async def start_review(
|
||||
job_id: str,
|
||||
request: StartReviewRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""
|
||||
Mark job as IN_REVIEW (user opened review interface).
|
||||
|
||||
|
||||
This helps track that the user is actively working on the review.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
# Check ownership
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to access this job")
|
||||
|
||||
success = job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.IN_REVIEW,
|
||||
@@ -400,20 +504,24 @@ async def submit_corrections(
|
||||
job_id: str,
|
||||
submission: CorrectionsSubmission,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""
|
||||
Save corrected lyrics during human review.
|
||||
|
||||
|
||||
This endpoint saves review progress but does NOT complete the review.
|
||||
Call POST /{job_id}/complete-review to finish and trigger video rendering.
|
||||
|
||||
|
||||
Can be called multiple times to save progress.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Check ownership
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to modify this job")
|
||||
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
@@ -459,11 +567,11 @@ async def submit_corrections(
|
||||
async def complete_review(
|
||||
job_id: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""
|
||||
Complete the human review and trigger video rendering.
|
||||
|
||||
|
||||
This is the FIRST critical human-in-the-loop completion point.
|
||||
After this:
|
||||
1. Job transitions to REVIEW_COMPLETE
|
||||
@@ -474,7 +582,11 @@ async def complete_review(
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Check ownership
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to modify this job")
|
||||
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
@@ -948,18 +1060,22 @@ async def select_instrumental(
|
||||
@router.get("/{job_id}/download-urls")
|
||||
async def get_download_urls(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""
|
||||
Get download URLs for all job output files.
|
||||
|
||||
|
||||
Returns a dictionary mapping file types to download URLs.
|
||||
Uses the streaming download endpoint which proxies through the backend.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Check ownership
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to access this job")
|
||||
|
||||
if job.status != JobStatus.COMPLETE:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
@@ -990,26 +1106,42 @@ async def get_download_urls(
|
||||
}
|
||||
|
||||
|
||||
# Map file keys to human-readable suffixes (matching Dropbox naming from karaoke_finalise.py)
|
||||
DOWNLOAD_FILENAME_SUFFIXES = {
|
||||
"lossless_4k_mp4": " (Final Karaoke Lossless 4k).mp4",
|
||||
"lossless_4k_mkv": " (Final Karaoke Lossless 4k).mkv",
|
||||
"lossy_4k_mp4": " (Final Karaoke Lossy 4k).mp4",
|
||||
"lossy_720p_mp4": " (Final Karaoke Lossy 720p).mp4",
|
||||
"cdg_zip": " (Final Karaoke CDG).zip",
|
||||
"txt_zip": " (Final Karaoke TXT).zip",
|
||||
"with_vocals": " (With Vocals).mkv",
|
||||
}
|
||||
|
||||
|
||||
@router.get("/{job_id}/download/{category}/{file_key}")
|
||||
async def download_file(
|
||||
job_id: str,
|
||||
category: str,
|
||||
file_key: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Stream download a specific file from a completed job.
|
||||
|
||||
|
||||
This endpoint proxies the file from GCS through the backend,
|
||||
so no client-side authentication is required.
|
||||
"""
|
||||
from fastapi.responses import StreamingResponse
|
||||
import tempfile
|
||||
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Check ownership
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to access this job")
|
||||
|
||||
if job.status != JobStatus.COMPLETE:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
@@ -1050,8 +1182,18 @@ async def download_file(
|
||||
}
|
||||
content_type = content_types.get(ext, 'application/octet-stream')
|
||||
|
||||
# Get filename for Content-Disposition
|
||||
filename = gcs_path.split('/')[-1]
|
||||
# Build proper filename: "Artist - Title (Final Karaoke Lossy 4k).mp4"
|
||||
# Use sanitize_filename to handle Unicode characters (curly quotes, em dashes, etc.)
|
||||
# that cause HTTP header encoding issues (Content-Disposition uses latin-1)
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
artist_clean = sanitize_filename(job.artist) if job.artist else None
|
||||
title_clean = sanitize_filename(job.title) if job.title else None
|
||||
base_name = f"{artist_clean} - {title_clean}" if artist_clean and title_clean else None
|
||||
|
||||
if base_name and file_key in DOWNLOAD_FILENAME_SUFFIXES:
|
||||
filename = f"{base_name}{DOWNLOAD_FILENAME_SUFFIXES[file_key]}"
|
||||
else:
|
||||
filename = gcs_path.split('/')[-1] # Fallback to original
|
||||
|
||||
try:
|
||||
# Download to temp file and stream
|
||||
@@ -1086,13 +1228,21 @@ async def download_file(
|
||||
async def cancel_job(
|
||||
job_id: str,
|
||||
request: CancelJobRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""
|
||||
Cancel a job.
|
||||
|
||||
|
||||
Jobs can be cancelled at any stage before completion.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
# Check ownership
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to cancel this job")
|
||||
|
||||
success = job_manager.cancel_job(job_id, reason=request.reason)
|
||||
|
||||
if not success:
|
||||
@@ -1109,7 +1259,7 @@ async def cancel_job(
|
||||
async def retry_job(
|
||||
job_id: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""
|
||||
Retry a failed or cancelled job from the last successful checkpoint.
|
||||
@@ -1127,6 +1277,10 @@ async def retry_job(
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
# Check ownership
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to retry this job")
|
||||
|
||||
if job.status not in [JobStatus.FAILED, JobStatus.CANCELLED]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
@@ -1151,12 +1305,13 @@ async def retry_job(
|
||||
|
||||
logger.info(f"Job {job_id}: Has rendered video and instrumental selection, retrying video generation")
|
||||
|
||||
# Clear error state
|
||||
# Clear error state and reset worker progress for idempotency
|
||||
job_manager.update_job(job_id, {
|
||||
'error_message': None,
|
||||
'error_details': None,
|
||||
})
|
||||
|
||||
job_manager.update_state_data(job_id, 'video_progress', {'stage': 'pending'})
|
||||
|
||||
# Reset to INSTRUMENTAL_SELECTED and trigger video worker
|
||||
if not job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
@@ -1184,13 +1339,14 @@ async def retry_job(
|
||||
file_urls.get('screens', {}).get('title')):
|
||||
|
||||
logger.info(f"Job {job_id}: Has corrections and screens, retrying from render stage")
|
||||
|
||||
# Clear error state
|
||||
|
||||
# Clear error state and reset worker progress for idempotency
|
||||
job_manager.update_job(job_id, {
|
||||
'error_message': None,
|
||||
'error_details': None,
|
||||
})
|
||||
|
||||
job_manager.update_state_data(job_id, 'render_progress', {'stage': 'pending'})
|
||||
|
||||
# Reset to REVIEW_COMPLETE and trigger render worker
|
||||
if not job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
@@ -1218,13 +1374,14 @@ async def retry_job(
|
||||
file_urls.get('lyrics', {}).get('corrections')):
|
||||
|
||||
logger.info(f"Job {job_id}: Has stems and corrections, retrying from screens stage")
|
||||
|
||||
# Clear error state
|
||||
|
||||
# Clear error state and reset worker progress for idempotency
|
||||
job_manager.update_job(job_id, {
|
||||
'error_message': None,
|
||||
'error_details': None,
|
||||
})
|
||||
|
||||
job_manager.update_state_data(job_id, 'screens_progress', {'stage': 'pending'})
|
||||
|
||||
# Reset to a state before screens and trigger screens worker
|
||||
if not job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
@@ -1301,19 +1458,23 @@ async def get_worker_logs(
|
||||
job_id: str,
|
||||
since_index: int = 0,
|
||||
worker: Optional[str] = None,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Get worker logs for debugging.
|
||||
|
||||
|
||||
This endpoint returns worker logs stored in Firestore.
|
||||
Use `since_index` for efficient polling (returns only new logs).
|
||||
|
||||
|
||||
Logs are stored in a subcollection (jobs/{job_id}/logs) to avoid
|
||||
the 1MB document size limit. Older jobs may have logs in an embedded
|
||||
array (worker_logs field) - this endpoint handles both transparently.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
since_index: Return only logs after this index (for pagination/polling)
|
||||
worker: Filter by worker name (audio, lyrics, screens, video, render)
|
||||
|
||||
worker: Filter by worker name (audio, lyrics, screens, video, render, distribution)
|
||||
|
||||
Returns:
|
||||
{
|
||||
"logs": [{"timestamp": "...", "level": "INFO", "worker": "audio", "message": "..."}],
|
||||
@@ -1324,13 +1485,145 @@ async def get_worker_logs(
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Check ownership - users can only see logs for their own jobs
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to access logs for this job")
|
||||
|
||||
logs = job_manager.get_worker_logs(job_id, since_index=since_index, worker=worker)
|
||||
total = len(job.worker_logs) if job.worker_logs else 0
|
||||
|
||||
total = job_manager.get_worker_logs_count(job_id)
|
||||
|
||||
return {
|
||||
"logs": logs,
|
||||
"next_index": total,
|
||||
"next_index": since_index + len(logs),
|
||||
"total_logs": total
|
||||
}
|
||||
|
||||
|
||||
@router.post("/{job_id}/cleanup-distribution")
|
||||
async def cleanup_distribution(
|
||||
job_id: str,
|
||||
delete_job: bool = True,
|
||||
auth_result: AuthResult = Depends(require_admin)
|
||||
) -> dict:
|
||||
"""
|
||||
Clean up all distributed content for a job (YouTube, Dropbox, Google Drive).
|
||||
|
||||
This admin-only endpoint is designed for E2E test cleanup. It:
|
||||
1. Deletes YouTube video (if uploaded)
|
||||
2. Deletes Dropbox folder (if uploaded)
|
||||
3. Deletes Google Drive files (if uploaded)
|
||||
4. Optionally deletes the job itself
|
||||
|
||||
Args:
|
||||
job_id: Job ID to clean up
|
||||
delete_job: If True, also delete the job after cleaning up distribution
|
||||
|
||||
Returns:
|
||||
Cleanup results for each service
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
state_data = job.state_data or {}
|
||||
results = {
|
||||
"job_id": job_id,
|
||||
"youtube": {"status": "skipped", "reason": "no youtube_url in state_data"},
|
||||
"dropbox": {"status": "skipped", "reason": "no brand_code or dropbox_path"},
|
||||
"gdrive": {"status": "skipped", "reason": "no gdrive_files in state_data"},
|
||||
"job_deleted": False
|
||||
}
|
||||
|
||||
# Clean up YouTube
|
||||
youtube_url = state_data.get('youtube_url')
|
||||
if youtube_url:
|
||||
try:
|
||||
# Extract video ID from URL (format: https://youtu.be/VIDEO_ID or https://www.youtube.com/watch?v=VIDEO_ID)
|
||||
import re
|
||||
video_id_match = re.search(r'(?:youtu\.be/|youtube\.com/watch\?v=)([^&\s]+)', youtube_url)
|
||||
if video_id_match:
|
||||
video_id = video_id_match.group(1)
|
||||
|
||||
# Import and use karaoke_finalise for YouTube deletion
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
from backend.services.youtube_service import get_youtube_service
|
||||
|
||||
youtube_service = get_youtube_service()
|
||||
if youtube_service.is_configured:
|
||||
# Create minimal KaraokeFinalise instance for deletion
|
||||
finalise = KaraokeFinalise(
|
||||
dry_run=False,
|
||||
non_interactive=True,
|
||||
enable_youtube=True,
|
||||
user_youtube_credentials=youtube_service.get_credentials_dict()
|
||||
)
|
||||
success = finalise.delete_youtube_video(video_id)
|
||||
results["youtube"] = {
|
||||
"status": "success" if success else "failed",
|
||||
"video_id": video_id
|
||||
}
|
||||
else:
|
||||
results["youtube"] = {"status": "failed", "reason": "YouTube credentials not configured"}
|
||||
else:
|
||||
results["youtube"] = {"status": "failed", "reason": f"Could not extract video ID from {youtube_url}"}
|
||||
except Exception as e:
|
||||
logger.error(f"Error cleaning up YouTube for job {job_id}: {e}", exc_info=True)
|
||||
results["youtube"] = {"status": "error", "error": str(e)}
|
||||
|
||||
# Clean up Dropbox
|
||||
brand_code = state_data.get('brand_code')
|
||||
dropbox_path = getattr(job, 'dropbox_path', None)
|
||||
if brand_code and dropbox_path:
|
||||
try:
|
||||
from backend.services.dropbox_service import get_dropbox_service
|
||||
dropbox = get_dropbox_service()
|
||||
if dropbox.is_configured:
|
||||
base_name = f"{job.artist} - {job.title}"
|
||||
folder_name = f"{brand_code} - {base_name}"
|
||||
full_path = f"{dropbox_path}/{folder_name}"
|
||||
success = dropbox.delete_folder(full_path)
|
||||
results["dropbox"] = {
|
||||
"status": "success" if success else "failed",
|
||||
"path": full_path
|
||||
}
|
||||
else:
|
||||
results["dropbox"] = {"status": "failed", "reason": "Dropbox credentials not configured"}
|
||||
except Exception as e:
|
||||
logger.error(f"Error cleaning up Dropbox for job {job_id}: {e}", exc_info=True)
|
||||
results["dropbox"] = {"status": "error", "error": str(e)}
|
||||
|
||||
# Clean up Google Drive
|
||||
gdrive_files = state_data.get('gdrive_files')
|
||||
if gdrive_files:
|
||||
try:
|
||||
from backend.services.gdrive_service import get_gdrive_service
|
||||
gdrive = get_gdrive_service()
|
||||
if gdrive.is_configured:
|
||||
# gdrive_files is a dict like {"mp4": "file_id", "mp4_720p": "file_id", "cdg": "file_id"}
|
||||
file_ids = list(gdrive_files.values()) if isinstance(gdrive_files, dict) else []
|
||||
delete_results = gdrive.delete_files(file_ids)
|
||||
all_success = all(delete_results.values())
|
||||
results["gdrive"] = {
|
||||
"status": "success" if all_success else "partial",
|
||||
"files": delete_results
|
||||
}
|
||||
else:
|
||||
results["gdrive"] = {"status": "failed", "reason": "Google Drive credentials not configured"}
|
||||
except Exception as e:
|
||||
logger.error(f"Error cleaning up Google Drive for job {job_id}: {e}", exc_info=True)
|
||||
results["gdrive"] = {"status": "error", "error": str(e)}
|
||||
|
||||
# Delete the job if requested
|
||||
if delete_job:
|
||||
try:
|
||||
job_manager.delete_job(job_id, delete_files=True)
|
||||
results["job_deleted"] = True
|
||||
logger.info(f"Deleted job {job_id} after distribution cleanup")
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting job {job_id}: {e}", exc_info=True)
|
||||
results["job_deleted"] = False
|
||||
results["job_delete_error"] = str(e)
|
||||
|
||||
return results
|
||||
|
||||
+122
-40
@@ -27,8 +27,10 @@ from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.job_logging import job_log_context, JobLogger
|
||||
from backend.services.tracing import create_span, add_span_attribute, add_span_event
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
from backend.api.dependencies import require_auth, require_review_auth
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.config import get_settings
|
||||
|
||||
# LyricsTranscriber imports for preview generation
|
||||
from lyrics_transcriber.types import CorrectionResult
|
||||
@@ -385,28 +387,35 @@ async def generate_preview_video(
|
||||
):
|
||||
"""
|
||||
Generate a preview video with the current corrections.
|
||||
|
||||
|
||||
Uses the LyricsTranscriber's CorrectionOperations to generate a 360p preview
|
||||
video with the user's current corrections applied.
|
||||
|
||||
When USE_GCE_PREVIEW_ENCODING is enabled, video encoding is offloaded to
|
||||
the high-performance GCE worker for faster generation (15-20s vs 60+s).
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
settings = get_settings()
|
||||
encoding_service = get_encoding_service()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Job must be in review state to generate preview
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job not in review state (current status: {job.status})"
|
||||
)
|
||||
|
||||
logger.info(f"Job {job_id}: Generating preview video")
|
||||
|
||||
|
||||
# Check if GCE preview encoding is enabled
|
||||
use_gce_preview = encoding_service.is_preview_enabled
|
||||
logger.info(f"Job {job_id}: Generating preview video (GCE preview: {use_gce_preview})")
|
||||
|
||||
# Use tracing and job_log_context for full observability
|
||||
with create_span("generate-preview-video", {"job_id": job_id}) as span:
|
||||
with create_span("generate-preview-video", {"job_id": job_id, "use_gce": use_gce_preview}) as span:
|
||||
with job_log_context(job_id, worker="preview"):
|
||||
try:
|
||||
# Create temp directory for this preview operation
|
||||
@@ -416,19 +425,19 @@ async def generate_preview_video(
|
||||
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
corrections_path = os.path.join(temp_dir, "corrections.json")
|
||||
storage.download_file(corrections_gcs, corrections_path)
|
||||
|
||||
|
||||
with open(corrections_path, 'r', encoding='utf-8') as f:
|
||||
original_data = json.load(f)
|
||||
|
||||
|
||||
# 2. Download input audio
|
||||
audio_path = os.path.join(temp_dir, "audio.flac")
|
||||
storage.download_file(job.input_media_gcs_path, audio_path)
|
||||
download_span.set_attribute("audio_gcs_path", job.input_media_gcs_path)
|
||||
|
||||
|
||||
# 3. Load original as CorrectionResult
|
||||
correction_result = CorrectionResult.from_dict(original_data)
|
||||
add_span_event("corrections_loaded")
|
||||
|
||||
|
||||
# 4. Get or create styles file for preview using unified style loader
|
||||
with create_span("load-styles") as styles_span:
|
||||
styles_path, _ = load_styles_from_gcs(
|
||||
@@ -439,54 +448,127 @@ async def generate_preview_video(
|
||||
logger=logger,
|
||||
)
|
||||
styles_span.set_attribute("styles_path", styles_path)
|
||||
|
||||
|
||||
# 5. Set up output config for preview
|
||||
output_dir = os.path.join(temp_dir, "output")
|
||||
cache_dir = os.path.join(temp_dir, "cache")
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
|
||||
|
||||
output_config = OutputConfig(
|
||||
output_styles_json=styles_path,
|
||||
output_dir=output_dir,
|
||||
cache_dir=cache_dir,
|
||||
video_resolution="360p",
|
||||
)
|
||||
|
||||
# 6. Generate preview video using CorrectionOperations (heavy operation)
|
||||
with create_span("render-preview-video") as render_span:
|
||||
render_span.set_attribute("resolution", "360p")
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
correction_result=correction_result,
|
||||
updated_data=updated_data,
|
||||
output_config=output_config,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
logger=logger
|
||||
)
|
||||
add_span_event("render_complete")
|
||||
|
||||
# 7. Upload preview video to GCS for serving
|
||||
preview_hash = result["preview_hash"]
|
||||
video_path = result["video_path"]
|
||||
|
||||
with create_span("upload-preview-video") as upload_span:
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
storage.upload_file(video_path, preview_gcs_path)
|
||||
upload_span.set_attribute("gcs_path", preview_gcs_path)
|
||||
|
||||
|
||||
# 6. Generate preview (ASS-only if using GCE, or full video if local)
|
||||
preview_gcs_path = None
|
||||
|
||||
if use_gce_preview:
|
||||
# GCE path: Generate ASS only, then offload encoding to GCE
|
||||
try:
|
||||
with create_span("generate-ass-subtitles") as ass_span:
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
correction_result=correction_result,
|
||||
updated_data=updated_data,
|
||||
output_config=output_config,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
logger=logger,
|
||||
ass_only=True, # Only generate ASS, skip video encoding
|
||||
)
|
||||
preview_hash = result["preview_hash"]
|
||||
ass_path = result["ass_path"]
|
||||
ass_span.set_attribute("ass_path", ass_path)
|
||||
add_span_event("ass_generated")
|
||||
|
||||
# Upload ASS to GCS
|
||||
with create_span("upload-ass-to-gcs") as upload_ass_span:
|
||||
ass_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.ass"
|
||||
storage.upload_file(ass_path, ass_gcs_path)
|
||||
upload_ass_span.set_attribute("ass_gcs_path", ass_gcs_path)
|
||||
|
||||
# Call GCE encoding service
|
||||
with create_span("gce-preview-encoding") as gce_span:
|
||||
bucket_name = settings.gcs_bucket_name
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
|
||||
# Get background image and font from style assets if available
|
||||
style_assets = job.style_assets or {}
|
||||
|
||||
background_image_gcs_path = None
|
||||
for key in ["karaoke_background", "style_karaoke_background"]:
|
||||
if key in style_assets:
|
||||
background_image_gcs_path = f"gs://{bucket_name}/{style_assets[key]}"
|
||||
gce_span.set_attribute("background_image", background_image_gcs_path)
|
||||
break
|
||||
|
||||
font_gcs_path = None
|
||||
for key in ["font", "style_font"]:
|
||||
if key in style_assets:
|
||||
font_gcs_path = f"gs://{bucket_name}/{style_assets[key]}"
|
||||
gce_span.set_attribute("font", font_gcs_path)
|
||||
break
|
||||
|
||||
gce_result = await encoding_service.encode_preview_video(
|
||||
job_id=f"preview_{job_id}_{preview_hash}",
|
||||
ass_gcs_path=f"gs://{bucket_name}/{ass_gcs_path}",
|
||||
audio_gcs_path=f"gs://{bucket_name}/{job.input_media_gcs_path}",
|
||||
output_gcs_path=f"gs://{bucket_name}/{preview_gcs_path}",
|
||||
background_color="black",
|
||||
background_image_gcs_path=background_image_gcs_path,
|
||||
font_gcs_path=font_gcs_path,
|
||||
)
|
||||
gce_span.set_attribute("gce_status", gce_result.get("status"))
|
||||
add_span_event("gce_encoding_complete")
|
||||
|
||||
logger.info(f"Job {job_id}: Preview generated via GCE: {preview_hash}")
|
||||
|
||||
except Exception as gce_error:
|
||||
# Fall back to local encoding if GCE fails
|
||||
logger.warning(
|
||||
f"Job {job_id}: GCE preview encoding failed, falling back to local: {gce_error}"
|
||||
)
|
||||
span.set_attribute("gce_fallback", True)
|
||||
use_gce_preview = False # Fall through to local encoding below
|
||||
|
||||
if not use_gce_preview:
|
||||
# Local path: Generate full preview video locally
|
||||
with create_span("render-preview-video-local") as render_span:
|
||||
render_span.set_attribute("resolution", "360p")
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
correction_result=correction_result,
|
||||
updated_data=updated_data,
|
||||
output_config=output_config,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
logger=logger,
|
||||
ass_only=False, # Generate full video locally
|
||||
)
|
||||
preview_hash = result["preview_hash"]
|
||||
video_path = result["video_path"]
|
||||
add_span_event("render_complete")
|
||||
|
||||
# Upload preview video to GCS
|
||||
with create_span("upload-preview-video") as upload_span:
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
storage.upload_file(video_path, preview_gcs_path)
|
||||
upload_span.set_attribute("gcs_path", preview_gcs_path)
|
||||
|
||||
# Store the GCS path for serving
|
||||
if job_id not in _preview_videos:
|
||||
_preview_videos[job_id] = {}
|
||||
_preview_videos[job_id][preview_hash] = preview_gcs_path
|
||||
|
||||
|
||||
logger.info(f"Job {job_id}: Preview video generated: {preview_hash}")
|
||||
span.set_attribute("preview_hash", preview_hash)
|
||||
span.set_attribute("success", True)
|
||||
|
||||
|
||||
return {"status": "success", "preview_hash": preview_hash}
|
||||
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Failed to generate preview video: {e}", exc_info=True)
|
||||
span.set_attribute("error", str(e))
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
+29
-1
@@ -40,7 +40,15 @@ class Settings(BaseSettings):
|
||||
# Processing
|
||||
max_concurrent_jobs: int = int(os.getenv("MAX_CONCURRENT_JOBS", "5"))
|
||||
job_timeout_seconds: int = int(os.getenv("JOB_TIMEOUT_SECONDS", "3600"))
|
||||
|
||||
|
||||
# Agentic AI Correction (for lyrics correction via LLM)
|
||||
# When enabled, uses Gemini via Vertex AI for intelligent lyrics correction
|
||||
use_agentic_ai: bool = os.getenv("USE_AGENTIC_AI", "true").lower() in ("true", "1", "yes")
|
||||
agentic_ai_model: str = os.getenv("AGENTIC_AI_MODEL", "vertexai/gemini-3-flash-preview")
|
||||
# Timeout for agentic correction in seconds. If correction takes longer, abort and
|
||||
# use uncorrected transcription - human review will fix any issues.
|
||||
agentic_correction_timeout_seconds: int = int(os.getenv("AGENTIC_CORRECTION_TIMEOUT_SECONDS", "180"))
|
||||
|
||||
# Cloud Tasks (for scalable worker coordination)
|
||||
# When enabled, workers are triggered via Cloud Tasks for guaranteed delivery
|
||||
# When disabled (default), workers are triggered via direct HTTP (for development)
|
||||
@@ -52,9 +60,29 @@ class Settings(BaseSettings):
|
||||
# instead of Cloud Tasks. This supports encoding times >30 minutes (up to 24 hours).
|
||||
# Default is false - Cloud Tasks is sufficient for most videos (15-20 min).
|
||||
use_cloud_run_jobs_for_video: bool = os.getenv("USE_CLOUD_RUN_JOBS_FOR_VIDEO", "false").lower() in ("true", "1", "yes")
|
||||
|
||||
# GCE Encoding Worker (for high-performance video encoding)
|
||||
# When enabled, video encoding is offloaded to a dedicated C4 GCE instance
|
||||
# with faster CPU (Intel Granite Rapids 3.9 GHz) instead of Cloud Run.
|
||||
# This provides 2-3x faster encoding times for CPU-bound FFmpeg libx264 encoding.
|
||||
use_gce_encoding: bool = os.getenv("USE_GCE_ENCODING", "false").lower() in ("true", "1", "yes")
|
||||
encoding_worker_url: Optional[str] = os.getenv("ENCODING_WORKER_URL") # e.g., http://136.119.50.148:8080
|
||||
encoding_worker_api_key: Optional[str] = os.getenv("ENCODING_WORKER_API_KEY")
|
||||
|
||||
# GCE Preview Encoding (for faster preview video generation)
|
||||
# When enabled, preview video encoding during lyrics review is offloaded to the GCE worker.
|
||||
# This reduces preview generation time from 60+ seconds to ~15-20 seconds.
|
||||
# Requires use_gce_encoding to be enabled and the GCE worker to support /encode-preview endpoint.
|
||||
use_gce_preview_encoding: bool = os.getenv("USE_GCE_PREVIEW_ENCODING", "false").lower() in ("true", "1", "yes")
|
||||
|
||||
# Storage paths
|
||||
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
|
||||
|
||||
# Worker logs storage mode
|
||||
# When enabled, worker logs are stored in a Firestore subcollection (jobs/{job_id}/logs)
|
||||
# instead of an embedded array. This avoids the 1MB document size limit.
|
||||
# Default is true for new deployments.
|
||||
use_log_subcollection: bool = os.getenv("USE_LOG_SUBCOLLECTION", "true").lower() in ("true", "1", "yes")
|
||||
|
||||
# Flacfetch remote service (for torrent downloads)
|
||||
# When configured, audio search uses the remote flacfetch HTTP API instead of local flacfetch.
|
||||
|
||||
+32
-2
@@ -7,9 +7,13 @@ from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from backend.config import settings
|
||||
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes
|
||||
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin
|
||||
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
|
||||
from backend.services.structured_logging import setup_structured_logging
|
||||
from backend.services.spacy_preloader import preload_spacy_model
|
||||
from backend.services.nltk_preloader import preload_all_nltk_resources
|
||||
from backend.services.langfuse_preloader import preload_langfuse_handler
|
||||
from backend.middleware.audit_logging import AuditLoggingMiddleware
|
||||
|
||||
|
||||
from backend.version import VERSION
|
||||
@@ -66,7 +70,28 @@ async def lifespan(app: FastAPI):
|
||||
logger.info(f"Environment: {settings.environment}")
|
||||
logger.info(f"GCS Bucket: {settings.gcs_bucket_name}")
|
||||
logger.info(f"Tracing enabled: {tracing_enabled}")
|
||||
|
||||
|
||||
# Preload NLP models and resources to avoid cold start delays
|
||||
# See docs/archive/2026-01-08-performance-investigation.md for background
|
||||
|
||||
# 1. SpaCy model (60+ second delay without preload)
|
||||
try:
|
||||
preload_spacy_model("en_core_web_sm")
|
||||
except Exception as e:
|
||||
logger.warning(f"SpaCy preload failed (will load lazily): {e}")
|
||||
|
||||
# 2. NLTK cmudict (50-100+ second delay without preload)
|
||||
try:
|
||||
preload_all_nltk_resources()
|
||||
except Exception as e:
|
||||
logger.warning(f"NLTK preload failed (will load lazily): {e}")
|
||||
|
||||
# 3. Langfuse callback handler (200+ second delay without preload)
|
||||
try:
|
||||
preload_langfuse_handler()
|
||||
except Exception as e:
|
||||
logger.warning(f"Langfuse preload failed (will initialize lazily): {e}")
|
||||
|
||||
# Validate OAuth credentials (non-blocking)
|
||||
try:
|
||||
await validate_credentials_on_startup()
|
||||
@@ -100,6 +125,9 @@ app.add_middleware(
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
# Add audit logging middleware (captures all requests with request_id for correlation)
|
||||
app.add_middleware(AuditLoggingMiddleware)
|
||||
|
||||
# Include routers
|
||||
app.include_router(health.router, prefix="/api")
|
||||
app.include_router(jobs.router, prefix="/api")
|
||||
@@ -109,6 +137,8 @@ app.include_router(review.router, prefix="/api") # Review UI compatibility endp
|
||||
app.include_router(auth.router, prefix="/api") # OAuth credential management
|
||||
app.include_router(audio_search.router, prefix="/api") # Audio search (artist+title mode)
|
||||
app.include_router(themes.router, prefix="/api") # Theme selection for styles
|
||||
app.include_router(users.router, prefix="/api") # User auth, credits, and Stripe webhooks
|
||||
app.include_router(admin.router, prefix="/api") # Admin dashboard and management
|
||||
|
||||
|
||||
@app.get("/")
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
"""Middleware package for FastAPI application."""
|
||||
|
||||
from backend.middleware.audit_logging import AuditLoggingMiddleware
|
||||
|
||||
__all__ = ["AuditLoggingMiddleware"]
|
||||
@@ -0,0 +1,124 @@
|
||||
"""
|
||||
Audit logging middleware for tracking all HTTP requests.
|
||||
|
||||
This middleware captures request metadata and logs it to Cloud Logging
|
||||
for audit trail and user activity investigation purposes.
|
||||
"""
|
||||
import logging
|
||||
import time
|
||||
import uuid
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.requests import Request
|
||||
from starlette.responses import Response
|
||||
|
||||
|
||||
logger = logging.getLogger("audit")
|
||||
|
||||
# Endpoints to exclude from audit logging (high-frequency health checks)
|
||||
EXCLUDED_PATHS = {
|
||||
"/",
|
||||
"/api/health",
|
||||
"/api/health/detailed",
|
||||
"/api/readiness",
|
||||
"/healthz",
|
||||
"/ready",
|
||||
}
|
||||
|
||||
|
||||
class AuditLoggingMiddleware(BaseHTTPMiddleware):
|
||||
"""
|
||||
Middleware that logs all HTTP requests for audit purposes.
|
||||
|
||||
Captures:
|
||||
- request_id: UUID for correlating with auth logs
|
||||
- method: HTTP method
|
||||
- path: Request path
|
||||
- status_code: Response status
|
||||
- latency_ms: Request duration
|
||||
- client_ip: Client IP (from X-Forwarded-For for proxied requests)
|
||||
- user_agent: Browser/client identifier
|
||||
|
||||
The request_id is stored in request.state and added to response headers,
|
||||
allowing correlation with auth logs that capture user_email.
|
||||
"""
|
||||
|
||||
async def dispatch(self, request: Request, call_next) -> Response:
|
||||
# Skip excluded paths (health checks)
|
||||
if request.url.path in EXCLUDED_PATHS:
|
||||
return await call_next(request)
|
||||
|
||||
# Generate unique request ID for correlation
|
||||
request_id = str(uuid.uuid4())
|
||||
request.state.request_id = request_id
|
||||
|
||||
# Capture start time
|
||||
start_time = time.time()
|
||||
|
||||
# Extract client info
|
||||
client_ip = self._get_client_ip(request)
|
||||
user_agent = request.headers.get("user-agent", "")
|
||||
|
||||
# Process request
|
||||
try:
|
||||
response = await call_next(request)
|
||||
except Exception:
|
||||
# Log failed requests too (exception() auto-includes stack trace)
|
||||
latency_ms = int((time.time() - start_time) * 1000)
|
||||
logger.exception(
|
||||
"request_audit_error",
|
||||
extra={
|
||||
"request_id": request_id,
|
||||
"method": request.method,
|
||||
"path": request.url.path,
|
||||
"query_string": str(request.query_params) if request.query_params else None,
|
||||
"latency_ms": latency_ms,
|
||||
"client_ip": client_ip,
|
||||
"user_agent": user_agent[:200] if user_agent else None,
|
||||
"audit_type": "http_request",
|
||||
}
|
||||
)
|
||||
raise
|
||||
|
||||
# Calculate latency
|
||||
latency_ms = int((time.time() - start_time) * 1000)
|
||||
|
||||
# Log audit entry (INFO level for successful requests)
|
||||
log_level = logging.WARNING if response.status_code >= 400 else logging.INFO
|
||||
logger.log(
|
||||
log_level,
|
||||
"request_audit",
|
||||
extra={
|
||||
"request_id": request_id,
|
||||
"method": request.method,
|
||||
"path": request.url.path,
|
||||
"query_string": str(request.query_params) if request.query_params else None,
|
||||
"status_code": response.status_code,
|
||||
"latency_ms": latency_ms,
|
||||
"client_ip": client_ip,
|
||||
"user_agent": user_agent[:200] if user_agent else None,
|
||||
"audit_type": "http_request",
|
||||
}
|
||||
)
|
||||
|
||||
# Add request_id to response headers for debugging/correlation
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
|
||||
return response
|
||||
|
||||
def _get_client_ip(self, request: Request) -> str:
|
||||
"""
|
||||
Extract client IP address, handling proxy scenarios.
|
||||
|
||||
Cloud Run and other proxies set X-Forwarded-For header.
|
||||
"""
|
||||
# Check X-Forwarded-For for proxy scenarios (Cloud Run, load balancers)
|
||||
forwarded = request.headers.get("x-forwarded-for", "")
|
||||
if forwarded:
|
||||
# First IP is the original client
|
||||
return forwarded.split(",")[0].strip()
|
||||
|
||||
# Fall back to direct connection
|
||||
if request.client:
|
||||
return request.client.host
|
||||
|
||||
return "unknown"
|
||||
+23
-3
@@ -9,6 +9,8 @@ from datetime import datetime
|
||||
from enum import Enum
|
||||
from pydantic import BaseModel, Field, validator
|
||||
|
||||
from karaoke_gen.utils import normalize_text
|
||||
|
||||
|
||||
class JobStatus(str, Enum):
|
||||
"""
|
||||
@@ -204,6 +206,7 @@ class Job(BaseModel):
|
||||
youtube_description: Optional[str] = None # YouTube video description
|
||||
webhook_url: Optional[str] = None # Webhook for notifications
|
||||
user_email: Optional[str] = None # Email for notifications
|
||||
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
|
||||
|
||||
# Theme configuration (pre-made themes from GCS)
|
||||
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
|
||||
@@ -419,6 +422,7 @@ class JobCreate(BaseModel):
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
|
||||
|
||||
# Theme configuration (pre-made themes from GCS)
|
||||
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
|
||||
@@ -483,13 +487,29 @@ class JobCreate(BaseModel):
|
||||
- custom_headers: All X-* headers
|
||||
"""
|
||||
|
||||
@validator('url', 'artist', 'title')
|
||||
def validate_inputs(cls, v):
|
||||
"""Validate string inputs are not empty."""
|
||||
@validator('url')
|
||||
def validate_url(cls, v):
|
||||
"""Validate URL is not empty."""
|
||||
if v is not None and isinstance(v, str) and not v.strip():
|
||||
raise ValueError("Field cannot be empty string")
|
||||
return v.strip() if isinstance(v, str) else v
|
||||
|
||||
@validator('artist', 'title')
|
||||
def normalize_artist_title(cls, v):
|
||||
"""Normalize artist/title text to standardize Unicode characters.
|
||||
|
||||
This ensures consistent data storage by converting:
|
||||
- Curly quotes -> straight quotes
|
||||
- Various dashes -> hyphen
|
||||
- Unusual whitespace -> regular space
|
||||
"""
|
||||
if v is not None and isinstance(v, str):
|
||||
if not v.strip():
|
||||
raise ValueError("Field cannot be empty string")
|
||||
# normalize_text handles stripping and Unicode normalization
|
||||
return normalize_text(v)
|
||||
return v
|
||||
|
||||
|
||||
class JobResponse(BaseModel):
|
||||
"""Job response model."""
|
||||
|
||||
@@ -14,7 +14,7 @@ class URLSubmissionRequest(BaseModel):
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: bool = False
|
||||
enable_youtube_upload: Optional[bool] = None # None = use server default
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
@@ -28,7 +28,7 @@ class UploadSubmissionRequest(BaseModel):
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: bool = False
|
||||
enable_youtube_upload: Optional[bool] = None # None = use server default
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
|
||||
@@ -0,0 +1,254 @@
|
||||
"""
|
||||
User model for authentication and credits.
|
||||
|
||||
Supports:
|
||||
- Magic link authentication (email-based)
|
||||
- Credit system for karaoke generation
|
||||
- Role-based access control (user/admin)
|
||||
- Stripe integration for payments
|
||||
- Beta tester program with feedback collection
|
||||
"""
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
from typing import Optional, List
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class UserRole(str, Enum):
|
||||
"""User roles for access control."""
|
||||
USER = "user"
|
||||
ADMIN = "admin"
|
||||
|
||||
|
||||
class BetaTesterStatus(str, Enum):
|
||||
"""Status of beta tester participation."""
|
||||
ACTIVE = "active" # Enrolled, free credits available
|
||||
PENDING_FEEDBACK = "pending_feedback" # Job completed, awaiting feedback
|
||||
COMPLETED = "completed" # Feedback submitted
|
||||
EXPIRED = "expired" # 24hr deadline passed without feedback
|
||||
|
||||
|
||||
class CreditTransaction(BaseModel):
|
||||
"""Record of a credit transaction."""
|
||||
id: str
|
||||
amount: int # Positive = add, negative = deduct
|
||||
reason: str # e.g., "purchase", "refund", "admin_grant", "job_creation"
|
||||
job_id: Optional[str] = None
|
||||
stripe_session_id: Optional[str] = None
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
created_by: Optional[str] = None # Admin email if granted by admin
|
||||
|
||||
|
||||
class User(BaseModel):
|
||||
"""
|
||||
User model stored in Firestore.
|
||||
|
||||
Users are identified by email address. Authentication is via magic links
|
||||
(no passwords). Credits are consumed when creating karaoke jobs.
|
||||
"""
|
||||
email: str # Primary identifier
|
||||
role: UserRole = UserRole.USER
|
||||
credits: int = 0
|
||||
|
||||
# Stripe integration
|
||||
stripe_customer_id: Optional[str] = None
|
||||
|
||||
# Account state
|
||||
is_active: bool = True
|
||||
email_verified: bool = False
|
||||
|
||||
# Timestamps
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
updated_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
last_login_at: Optional[datetime] = None
|
||||
|
||||
# Credit history (last 100 transactions)
|
||||
credit_transactions: List[CreditTransaction] = Field(default_factory=list)
|
||||
|
||||
# Job tracking
|
||||
total_jobs_created: int = 0
|
||||
total_jobs_completed: int = 0
|
||||
|
||||
# Optional profile fields for future use
|
||||
display_name: Optional[str] = None
|
||||
|
||||
# Beta tester program
|
||||
is_beta_tester: bool = False
|
||||
beta_tester_status: Optional[BetaTesterStatus] = None
|
||||
beta_enrolled_at: Optional[datetime] = None
|
||||
beta_promise_text: Optional[str] = None # User's promise statement
|
||||
beta_feedback_due_at: Optional[datetime] = None # 24hr after job completion
|
||||
beta_feedback_email_sent: bool = False
|
||||
|
||||
|
||||
class MagicLinkToken(BaseModel):
|
||||
"""
|
||||
Magic link token for passwordless authentication.
|
||||
|
||||
Tokens are short-lived (15 minutes) and single-use.
|
||||
Stored in Firestore with automatic TTL.
|
||||
"""
|
||||
token: str # Secure random token
|
||||
email: str
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
expires_at: datetime
|
||||
used: bool = False
|
||||
used_at: Optional[datetime] = None
|
||||
ip_address: Optional[str] = None
|
||||
user_agent: Optional[str] = None
|
||||
|
||||
|
||||
class Session(BaseModel):
|
||||
"""
|
||||
User session for authenticated requests.
|
||||
|
||||
Sessions are created after successful magic link verification.
|
||||
Sessions expire after 7 days of inactivity or 30 days absolute.
|
||||
"""
|
||||
token: str # Secure random session token
|
||||
user_email: str
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
expires_at: datetime
|
||||
last_activity_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
ip_address: Optional[str] = None
|
||||
user_agent: Optional[str] = None
|
||||
is_active: bool = True
|
||||
|
||||
|
||||
# Pydantic models for API requests/responses
|
||||
|
||||
class SendMagicLinkRequest(BaseModel):
|
||||
"""Request to send a magic link email."""
|
||||
email: str
|
||||
|
||||
|
||||
class SendMagicLinkResponse(BaseModel):
|
||||
"""Response after sending magic link."""
|
||||
status: str
|
||||
message: str
|
||||
|
||||
|
||||
class VerifyMagicLinkRequest(BaseModel):
|
||||
"""Request to verify a magic link token."""
|
||||
token: str
|
||||
|
||||
|
||||
class VerifyMagicLinkResponse(BaseModel):
|
||||
"""Response after verifying magic link."""
|
||||
status: str
|
||||
session_token: str
|
||||
user: "UserPublic"
|
||||
message: str
|
||||
|
||||
|
||||
class UserPublic(BaseModel):
|
||||
"""Public user information (safe to expose to frontend)."""
|
||||
email: str
|
||||
role: UserRole
|
||||
credits: int
|
||||
display_name: Optional[str] = None
|
||||
total_jobs_created: int = 0
|
||||
total_jobs_completed: int = 0
|
||||
|
||||
|
||||
class AddCreditsRequest(BaseModel):
|
||||
"""Admin request to add credits to a user."""
|
||||
email: str
|
||||
amount: int
|
||||
reason: str = "admin_grant"
|
||||
|
||||
|
||||
class AddCreditsResponse(BaseModel):
|
||||
"""Response after adding credits."""
|
||||
status: str
|
||||
email: str
|
||||
credits_added: int
|
||||
new_balance: int
|
||||
message: str
|
||||
|
||||
|
||||
class UserListResponse(BaseModel):
|
||||
"""Response for listing users (admin only)."""
|
||||
users: List[UserPublic]
|
||||
total: int
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Beta Tester Program Models
|
||||
# ============================================================================
|
||||
|
||||
class BetaTesterFeedback(BaseModel):
|
||||
"""
|
||||
Feedback from a beta tester after using the service.
|
||||
|
||||
Stored in Firestore 'beta_feedback' collection.
|
||||
"""
|
||||
id: str # Unique feedback ID
|
||||
user_email: str
|
||||
job_id: Optional[str] = None # The job they're providing feedback on
|
||||
|
||||
# Ratings (1-5 scale)
|
||||
overall_rating: int = Field(ge=1, le=5)
|
||||
ease_of_use_rating: int = Field(ge=1, le=5)
|
||||
lyrics_accuracy_rating: int = Field(ge=1, le=5)
|
||||
correction_experience_rating: int = Field(ge=1, le=5)
|
||||
|
||||
# Open-ended feedback
|
||||
what_went_well: Optional[str] = None
|
||||
what_could_improve: Optional[str] = None
|
||||
additional_comments: Optional[str] = None
|
||||
|
||||
# Would they recommend / use again?
|
||||
would_recommend: bool = True
|
||||
would_use_again: bool = True
|
||||
|
||||
# Metadata
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
submitted_via: str = "web" # 'web' or 'email'
|
||||
|
||||
|
||||
class BetaTesterEnrollRequest(BaseModel):
|
||||
"""Request to enroll as a beta tester."""
|
||||
email: str
|
||||
promise_text: str = Field(
|
||||
min_length=10,
|
||||
description="User's promise to provide feedback (min 10 chars)"
|
||||
)
|
||||
accept_corrections_work: bool = Field(
|
||||
description="User accepts they may need to review/correct lyrics"
|
||||
)
|
||||
|
||||
|
||||
class BetaTesterEnrollResponse(BaseModel):
|
||||
"""Response after enrolling as beta tester."""
|
||||
status: str
|
||||
message: str
|
||||
credits_granted: int
|
||||
session_token: Optional[str] = None # If new user, provide session
|
||||
|
||||
|
||||
class BetaFeedbackRequest(BaseModel):
|
||||
"""Request to submit beta tester feedback."""
|
||||
job_id: Optional[str] = None
|
||||
|
||||
# Ratings (1-5 scale)
|
||||
overall_rating: int = Field(ge=1, le=5)
|
||||
ease_of_use_rating: int = Field(ge=1, le=5)
|
||||
lyrics_accuracy_rating: int = Field(ge=1, le=5)
|
||||
correction_experience_rating: int = Field(ge=1, le=5)
|
||||
|
||||
# Open-ended feedback
|
||||
what_went_well: Optional[str] = None
|
||||
what_could_improve: Optional[str] = None
|
||||
additional_comments: Optional[str] = None
|
||||
|
||||
# Would they recommend / use again?
|
||||
would_recommend: bool = True
|
||||
would_use_again: bool = True
|
||||
|
||||
|
||||
class BetaFeedbackResponse(BaseModel):
|
||||
"""Response after submitting feedback."""
|
||||
status: str
|
||||
message: str
|
||||
bonus_credits: int = 0 # Bonus credits for great feedback
|
||||
@@ -0,0 +1,164 @@
|
||||
"""
|
||||
Worker log entry model for subcollection storage.
|
||||
|
||||
This module defines the LogEntry model for storing worker logs in a Firestore
|
||||
subcollection instead of an embedded array. This avoids the 1MB document size
|
||||
limit that caused job 501258e1 to fail when logs reached 1.26 MB.
|
||||
"""
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from typing import Optional, Dict, Any
|
||||
import uuid
|
||||
|
||||
|
||||
# Default TTL for log entries (30 days)
|
||||
DEFAULT_LOG_TTL_DAYS = 30
|
||||
|
||||
|
||||
@dataclass
|
||||
class WorkerLogEntry:
|
||||
"""
|
||||
Worker log entry for Firestore subcollection storage.
|
||||
|
||||
Stored at: jobs/{job_id}/logs/{log_id}
|
||||
|
||||
This model is separate from the legacy LogEntry (in job.py) which is
|
||||
stored as an embedded array in the job document. This new model supports
|
||||
the subcollection approach with TTL and richer metadata.
|
||||
"""
|
||||
# Core fields (required)
|
||||
timestamp: datetime
|
||||
level: str # "DEBUG", "INFO", "WARNING", "ERROR"
|
||||
worker: str # "audio", "lyrics", "screens", "video", "render", "distribution"
|
||||
message: str
|
||||
|
||||
# Identifiers
|
||||
id: str = field(default_factory=lambda: str(uuid.uuid4()))
|
||||
job_id: str = "" # Set when writing to Firestore
|
||||
|
||||
# TTL for automatic cleanup
|
||||
ttl_expiry: datetime = field(
|
||||
default_factory=lambda: datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
|
||||
)
|
||||
|
||||
# Optional metadata for debugging
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
|
||||
@classmethod
|
||||
def create(
|
||||
cls,
|
||||
job_id: str,
|
||||
worker: str,
|
||||
level: str,
|
||||
message: str,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
ttl_days: int = DEFAULT_LOG_TTL_DAYS
|
||||
) -> "WorkerLogEntry":
|
||||
"""
|
||||
Factory method to create a new log entry.
|
||||
|
||||
Args:
|
||||
job_id: Job ID this log belongs to
|
||||
worker: Worker name (audio, lyrics, screens, video, render, distribution)
|
||||
level: Log level (DEBUG, INFO, WARNING, ERROR)
|
||||
message: Log message (truncated to 1000 chars)
|
||||
metadata: Optional additional metadata
|
||||
ttl_days: Days until log expires (default 30)
|
||||
|
||||
Returns:
|
||||
New WorkerLogEntry instance
|
||||
"""
|
||||
now = datetime.now(timezone.utc)
|
||||
return cls(
|
||||
id=str(uuid.uuid4()),
|
||||
job_id=job_id,
|
||||
timestamp=now,
|
||||
level=level.upper(),
|
||||
worker=worker,
|
||||
message=message[:1000], # Truncate long messages
|
||||
metadata=metadata,
|
||||
ttl_expiry=now + timedelta(days=ttl_days)
|
||||
)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert to dictionary for Firestore storage.
|
||||
|
||||
Returns:
|
||||
Dictionary representation for Firestore
|
||||
"""
|
||||
result = {
|
||||
"id": self.id,
|
||||
"job_id": self.job_id,
|
||||
"timestamp": self.timestamp,
|
||||
"level": self.level,
|
||||
"worker": self.worker,
|
||||
"message": self.message,
|
||||
"ttl_expiry": self.ttl_expiry
|
||||
}
|
||||
if self.metadata:
|
||||
result["metadata"] = self.metadata
|
||||
return result
|
||||
|
||||
def to_legacy_dict(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert to legacy format for backward compatibility with API responses.
|
||||
|
||||
The existing API returns logs in this format:
|
||||
{"timestamp": "...", "level": "INFO", "worker": "audio", "message": "..."}
|
||||
|
||||
Returns:
|
||||
Dictionary in legacy format
|
||||
"""
|
||||
# Format timestamp: use "Z" suffix for UTC, isoformat() for others
|
||||
if self.timestamp.tzinfo is None:
|
||||
# Naive datetime - use as-is
|
||||
timestamp_str = self.timestamp.isoformat()
|
||||
elif self.timestamp.utcoffset() == timedelta(0):
|
||||
# UTC datetime - replace +00:00 with Z for cleaner format
|
||||
timestamp_str = self.timestamp.replace(tzinfo=None).isoformat() + "Z"
|
||||
else:
|
||||
# Non-UTC timezone-aware - use full isoformat with offset
|
||||
timestamp_str = self.timestamp.isoformat()
|
||||
|
||||
return {
|
||||
"timestamp": timestamp_str,
|
||||
"level": self.level,
|
||||
"worker": self.worker,
|
||||
"message": self.message
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, Any]) -> "WorkerLogEntry":
|
||||
"""
|
||||
Create from Firestore document data.
|
||||
|
||||
Args:
|
||||
data: Dictionary from Firestore document
|
||||
|
||||
Returns:
|
||||
WorkerLogEntry instance
|
||||
"""
|
||||
# Handle timestamp conversion (Firestore returns datetime objects)
|
||||
timestamp = data.get("timestamp")
|
||||
if isinstance(timestamp, str):
|
||||
# Parse ISO format string
|
||||
timestamp = datetime.fromisoformat(timestamp.rstrip("Z")).replace(tzinfo=timezone.utc)
|
||||
|
||||
ttl_expiry = data.get("ttl_expiry")
|
||||
if isinstance(ttl_expiry, str):
|
||||
ttl_expiry = datetime.fromisoformat(ttl_expiry.rstrip("Z")).replace(tzinfo=timezone.utc)
|
||||
elif ttl_expiry is None:
|
||||
# Default TTL if not set
|
||||
ttl_expiry = datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
|
||||
|
||||
return cls(
|
||||
id=data.get("id", str(uuid.uuid4())),
|
||||
job_id=data.get("job_id", ""),
|
||||
timestamp=timestamp or datetime.now(timezone.utc),
|
||||
level=data.get("level", "INFO"),
|
||||
worker=data.get("worker", "unknown"),
|
||||
message=data.get("message", ""),
|
||||
metadata=data.get("metadata"),
|
||||
ttl_expiry=ttl_expiry
|
||||
)
|
||||
@@ -276,20 +276,35 @@ class AudioSearchService:
|
||||
Raises:
|
||||
DownloadError: If download fails or no cached result for index
|
||||
"""
|
||||
if result_index < 0 or result_index >= len(self._cached_results):
|
||||
raise DownloadError(
|
||||
f"No cached result for index {result_index}. "
|
||||
f"Available indices: 0-{len(self._cached_results) - 1}. "
|
||||
"Run search() first."
|
||||
)
|
||||
# Use provided search_id or fall back to service-level cache
|
||||
effective_search_id = remote_search_id or self._remote_search_id
|
||||
|
||||
# Check if we have local cached results
|
||||
has_local_cache = result_index >= 0 and result_index < len(self._cached_results)
|
||||
|
||||
# If no local cache but we have remote search_id, use remote download
|
||||
# This handles multi-instance deployments where the cache doesn't persist
|
||||
if not has_local_cache:
|
||||
if effective_search_id and self._remote_client:
|
||||
logger.info(f"No local cache, using remote download with search_id={effective_search_id}, index={result_index}")
|
||||
return self._download_remote(result_index, output_dir, output_filename, gcs_path, effective_search_id)
|
||||
else:
|
||||
# Provide clear error message based on whether cache is empty or index out of bounds
|
||||
if len(self._cached_results) == 0:
|
||||
raise DownloadError(
|
||||
f"No cached result for index {result_index}. "
|
||||
"No cached results available. Run search() first."
|
||||
)
|
||||
raise DownloadError(
|
||||
f"No cached result for index {result_index}. "
|
||||
f"Available indices: 0-{len(self._cached_results) - 1}. "
|
||||
"Run search() first."
|
||||
)
|
||||
|
||||
result = self._cached_results[result_index]
|
||||
|
||||
logger.info(f"Downloading: {result.artist} - {result.title} from {result.provider}")
|
||||
|
||||
# Use provided search_id or fall back to service-level cache
|
||||
effective_search_id = remote_search_id or self._remote_search_id
|
||||
|
||||
# Check if this was a remote search (torrent sources need remote download)
|
||||
if effective_search_id and self._remote_client:
|
||||
# Check if this is a torrent source that needs remote handling
|
||||
@@ -481,8 +496,12 @@ class AudioSearchService:
|
||||
if not effective_search_id:
|
||||
raise DownloadError("No remote search ID - run search() first")
|
||||
|
||||
result = self._cached_results[result_index]
|
||||
|
||||
# Try to get local result for metadata, but it's optional
|
||||
# The remote service maintains its own cache by search_id
|
||||
result = None
|
||||
if result_index >= 0 and result_index < len(self._cached_results):
|
||||
result = self._cached_results[result_index]
|
||||
|
||||
try:
|
||||
# Enable nested event loops (needed when called from FastAPI async context)
|
||||
try:
|
||||
@@ -529,13 +548,15 @@ class AudioSearchService:
|
||||
raise DownloadError("Remote download completed but no file path returned")
|
||||
|
||||
logger.info(f"Remote download complete: {filepath}")
|
||||
|
||||
|
||||
# Use local result metadata if available, otherwise use empty strings
|
||||
# (the actual download is handled by the remote service using search_id)
|
||||
return AudioDownloadResult(
|
||||
filepath=filepath,
|
||||
artist=result.artist,
|
||||
title=result.title,
|
||||
provider=result.provider,
|
||||
quality=result.quality,
|
||||
artist=result.artist if result else "",
|
||||
title=result.title if result else "",
|
||||
provider=result.provider if result else "remote",
|
||||
quality=result.quality if result else "",
|
||||
)
|
||||
|
||||
except FlacfetchServiceError as e:
|
||||
|
||||
@@ -3,50 +3,81 @@ Authentication service for karaoke backend.
|
||||
|
||||
Provides token-based authentication with support for:
|
||||
- Admin tokens (hardcoded in environment)
|
||||
- User tokens (stored in Firestore)
|
||||
- Admin by email domain (@nomadkaraoke.com)
|
||||
- User tokens (stored in Firestore auth_tokens collection)
|
||||
- Session tokens (stored in Firestore sessions collection, from magic link auth)
|
||||
- API keys for business users
|
||||
- Usage tracking and limits
|
||||
- Token management API
|
||||
|
||||
This is designed to be extensible for future Stripe integration
|
||||
and admin dashboard functionality.
|
||||
"""
|
||||
import logging
|
||||
import hashlib
|
||||
import time
|
||||
import secrets
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from typing import Optional, Tuple, Dict, Any
|
||||
from typing import Optional, Tuple, Dict, Any, List
|
||||
from datetime import datetime
|
||||
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
from backend.config import get_settings
|
||||
from backend.models.user import UserRole
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Admin email domains - users with these domains are automatically admin
|
||||
ADMIN_EMAIL_DOMAINS = ["nomadkaraoke.com"]
|
||||
|
||||
|
||||
class UserType(str, Enum):
|
||||
"""User access levels."""
|
||||
ADMIN = "admin" # Unlimited access, can manage tokens
|
||||
UNLIMITED = "unlimited" # Unlimited karaoke generation
|
||||
LIMITED = "limited" # Limited number of uses
|
||||
STRIPE = "stripe" # Paid access via Stripe (future)
|
||||
STRIPE = "stripe" # Paid access via Stripe
|
||||
API_KEY = "api_key" # Business API key access
|
||||
|
||||
|
||||
@dataclass
|
||||
class AuthResult:
|
||||
"""Result of authentication validation."""
|
||||
is_valid: bool
|
||||
user_type: UserType
|
||||
remaining_uses: int # -1 = unlimited, 0 = exhausted, >0 = remaining
|
||||
message: str
|
||||
user_email: Optional[str] = None # Email if authenticated via session/API key
|
||||
is_admin: bool = False # True if admin token or admin email domain
|
||||
api_key_id: Optional[str] = None # API key ID if authenticated via API key
|
||||
|
||||
def __iter__(self):
|
||||
"""Allow unpacking as tuple for backward compatibility."""
|
||||
return iter((self.is_valid, self.user_type, self.remaining_uses, self.message))
|
||||
|
||||
|
||||
def is_admin_email(email: str) -> bool:
|
||||
"""Check if an email belongs to an admin domain."""
|
||||
if not email:
|
||||
return False
|
||||
email_lower = email.lower()
|
||||
return any(email_lower.endswith(f"@{domain}") for domain in ADMIN_EMAIL_DOMAINS)
|
||||
|
||||
|
||||
class AuthService:
|
||||
"""Service for authentication and token management."""
|
||||
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize auth service."""
|
||||
self.firestore = FirestoreService()
|
||||
self.settings = get_settings()
|
||||
self._init_admin_tokens()
|
||||
|
||||
|
||||
def _init_admin_tokens(self):
|
||||
"""Load admin tokens from environment variables."""
|
||||
# Get admin tokens from secret manager or env vars
|
||||
admin_tokens_str = self.settings.admin_tokens or ""
|
||||
self.admin_tokens = [t.strip() for t in admin_tokens_str.split(",") if t.strip()]
|
||||
|
||||
|
||||
if self.admin_tokens:
|
||||
logger.info(f"Loaded {len(self.admin_tokens)} admin token(s)")
|
||||
else:
|
||||
@@ -78,10 +109,30 @@ class AuthService:
|
||||
f"expected len={len(self.admin_tokens[0])}, got len={len(token)}"
|
||||
)
|
||||
|
||||
# Check stored tokens in Firestore
|
||||
# Check stored tokens in Firestore (auth_tokens collection)
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
|
||||
if not token_data:
|
||||
# Fall back to checking session tokens (from magic link auth)
|
||||
# Import here to avoid circular dependency
|
||||
from backend.services.user_service import get_user_service
|
||||
|
||||
user_service = get_user_service()
|
||||
is_valid, user, message = user_service.validate_session(token)
|
||||
|
||||
if is_valid and user:
|
||||
# Session is valid - user is authenticated via magic link
|
||||
# Return their credits as remaining uses
|
||||
credits = user.credits
|
||||
logger.info(f"Session token validated for user {user.email} (credits: {credits})")
|
||||
|
||||
if credits <= 0:
|
||||
# User has no credits but is authenticated
|
||||
return True, UserType.STRIPE, 0, f"Authenticated but no credits remaining"
|
||||
|
||||
return True, UserType.STRIPE, credits, f"Session valid: {credits} credits"
|
||||
|
||||
# Neither auth_token nor session found
|
||||
return False, UserType.LIMITED, 0, "Invalid token"
|
||||
|
||||
# Check if token is active
|
||||
@@ -129,35 +180,319 @@ class AuthService:
|
||||
return True, token_type, -1, "Stripe access granted"
|
||||
|
||||
return False, UserType.LIMITED, 0, "Unknown token type"
|
||||
|
||||
|
||||
def validate_token_full(self, token: str) -> AuthResult:
|
||||
"""
|
||||
Validate an access token and return full authentication details.
|
||||
|
||||
This is the preferred method for new code as it returns complete
|
||||
authentication context including user email and admin status.
|
||||
|
||||
Returns:
|
||||
AuthResult with full authentication details
|
||||
"""
|
||||
if not token:
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=UserType.LIMITED,
|
||||
remaining_uses=0,
|
||||
message="No token provided"
|
||||
)
|
||||
|
||||
# Check for admin tokens (highest priority)
|
||||
# Admin tokens from env var are associated with the system admin email
|
||||
if token in self.admin_tokens:
|
||||
logger.info("Admin token validated")
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=-1,
|
||||
message="Admin access granted",
|
||||
user_email="admin@nomadkaraoke.com",
|
||||
is_admin=True
|
||||
)
|
||||
|
||||
# Check stored tokens in Firestore (auth_tokens collection)
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
if not token_data:
|
||||
# Fall back to checking session tokens (from magic link auth)
|
||||
from backend.services.user_service import get_user_service
|
||||
|
||||
user_service = get_user_service()
|
||||
is_valid, user, _message = user_service.validate_session(token)
|
||||
|
||||
if is_valid and user:
|
||||
# Session is valid - user is authenticated via magic link
|
||||
credits = user.credits
|
||||
user_email = user.email
|
||||
|
||||
# Check if user is admin by email domain or role
|
||||
user_is_admin = is_admin_email(user_email) or user.role == UserRole.ADMIN
|
||||
|
||||
if user_is_admin:
|
||||
logger.info(f"Admin session validated for {user_email}")
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=-1,
|
||||
message="Admin access granted (domain)",
|
||||
user_email=user_email,
|
||||
is_admin=True
|
||||
)
|
||||
|
||||
logger.info(f"Session token validated for user {user_email} (credits: {credits})")
|
||||
|
||||
if credits <= 0:
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.STRIPE,
|
||||
remaining_uses=0,
|
||||
message="Authenticated but no credits remaining",
|
||||
user_email=user_email,
|
||||
is_admin=False
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.STRIPE,
|
||||
remaining_uses=credits,
|
||||
message=f"Session valid: {credits} credits",
|
||||
user_email=user_email,
|
||||
is_admin=False
|
||||
)
|
||||
|
||||
# Neither auth_token nor session found
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=UserType.LIMITED,
|
||||
remaining_uses=0,
|
||||
message="Invalid token"
|
||||
)
|
||||
|
||||
# Check if token is active
|
||||
if not token_data.get("active", True):
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=UserType(token_data["type"]),
|
||||
remaining_uses=0,
|
||||
message="Token has been revoked"
|
||||
)
|
||||
|
||||
token_type = UserType(token_data["type"])
|
||||
max_uses = token_data.get("max_uses", -1)
|
||||
# All auth_tokens must have an associated user_email for job ownership
|
||||
token_user_email = token_data.get("user_email")
|
||||
api_key_id = token_data.get("api_key_id")
|
||||
|
||||
# Require user_email on all auth_tokens (no anonymous token auth)
|
||||
if not token_user_email:
|
||||
logger.warning("Auth token missing required user_email field")
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=token_type,
|
||||
remaining_uses=0,
|
||||
message="Token configuration error: missing user_email. Please contact support."
|
||||
)
|
||||
|
||||
# Check if token's user is an admin (by email domain)
|
||||
token_is_admin = is_admin_email(token_user_email)
|
||||
|
||||
# UNLIMITED tokens: no usage limits
|
||||
if token_type == UserType.UNLIMITED:
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=-1,
|
||||
message="Unlimited access granted",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
# LIMITED tokens: check usage count
|
||||
if token_type == UserType.LIMITED:
|
||||
if max_uses <= 0: # -1 means unlimited
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=-1,
|
||||
message="Limited token with unlimited uses",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
current_uses = token_data.get("usage_count", 0)
|
||||
remaining = max_uses - current_uses
|
||||
|
||||
if remaining <= 0:
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=token_type,
|
||||
remaining_uses=0,
|
||||
message="Token usage limit exceeded",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=remaining,
|
||||
message=f"Limited token: {remaining} uses remaining",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
# STRIPE tokens: check expiration and usage
|
||||
if token_type == UserType.STRIPE:
|
||||
expires_at = token_data.get("expires_at")
|
||||
|
||||
if expires_at and time.time() > expires_at:
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=token_type,
|
||||
remaining_uses=0,
|
||||
message="Token has expired",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
if max_uses > 0:
|
||||
current_uses = token_data.get("usage_count", 0)
|
||||
remaining = max_uses - current_uses
|
||||
|
||||
if remaining <= 0:
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=token_type,
|
||||
remaining_uses=0,
|
||||
message="Token usage limit exceeded",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=remaining,
|
||||
message=f"Stripe token: {remaining} uses remaining",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=-1,
|
||||
message="Stripe access granted",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
# API_KEY tokens
|
||||
if token_type == UserType.API_KEY:
|
||||
if max_uses > 0:
|
||||
current_uses = token_data.get("usage_count", 0)
|
||||
remaining = max_uses - current_uses
|
||||
if remaining <= 0:
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=token_type,
|
||||
remaining_uses=0,
|
||||
message="API key usage limit exceeded",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=remaining,
|
||||
message=f"API key valid: {remaining} uses remaining",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=-1,
|
||||
message="API key access granted",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=UserType.LIMITED,
|
||||
remaining_uses=0,
|
||||
message="Unknown token type"
|
||||
)
|
||||
|
||||
def increment_token_usage(self, token: str, job_id: str) -> bool:
|
||||
"""
|
||||
Increment usage count for a token and track the job.
|
||||
|
||||
|
||||
For session tokens (magic link auth), this deducts a credit from the user.
|
||||
For auth_tokens, this increments the usage count.
|
||||
|
||||
Args:
|
||||
token: The access token
|
||||
job_id: The job ID being created
|
||||
|
||||
|
||||
Returns:
|
||||
True if usage was tracked, False otherwise
|
||||
"""
|
||||
# Validate token first
|
||||
is_valid, user_type, remaining_uses, message = self.validate_token(token)
|
||||
|
||||
|
||||
if not is_valid:
|
||||
logger.warning(f"Cannot increment usage for invalid token: {message}")
|
||||
return False
|
||||
|
||||
|
||||
# Don't track usage for admin or unlimited tokens
|
||||
if user_type in [UserType.ADMIN, UserType.UNLIMITED]:
|
||||
logger.debug(f"Skipping usage tracking for {user_type} token")
|
||||
return True
|
||||
|
||||
|
||||
# For admin tokens (not in Firestore), no tracking needed
|
||||
if token in self.admin_tokens:
|
||||
return True
|
||||
|
||||
# Increment usage in Firestore
|
||||
|
||||
# Check if this is a session token (STRIPE type from validate_token means it's a session)
|
||||
# Session tokens are not in auth_tokens collection
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
if not token_data:
|
||||
# This is a session token - deduct credit from user
|
||||
from backend.services.user_service import get_user_service
|
||||
user_service = get_user_service()
|
||||
|
||||
# Get the user email from the session
|
||||
is_valid, user, _ = user_service.validate_session(token)
|
||||
if not is_valid or not user:
|
||||
logger.error(f"Session token validation failed during usage increment")
|
||||
return False
|
||||
|
||||
# Deduct one credit
|
||||
success, new_balance, deduct_message = user_service.deduct_credit(
|
||||
user.email, job_id, reason="job_creation"
|
||||
)
|
||||
|
||||
if success:
|
||||
logger.info(f"Deducted credit for user {user.email} (remaining: {new_balance})")
|
||||
return True
|
||||
else:
|
||||
logger.error(f"Failed to deduct credit for user {user.email}: {deduct_message}")
|
||||
return False
|
||||
|
||||
# Regular auth_token - increment usage in Firestore
|
||||
try:
|
||||
self.firestore.increment_token_usage(token, job_id)
|
||||
logger.info(f"Incremented usage for token (remaining: {remaining_uses - 1})")
|
||||
|
||||
@@ -0,0 +1,172 @@
|
||||
"""
|
||||
Discord Notification Service.
|
||||
|
||||
Provides Discord webhook notification functionality, extracted from KaraokeFinalise
|
||||
for use by both the cloud backend (video_worker) and local CLI.
|
||||
|
||||
This service handles:
|
||||
- Posting messages to Discord webhooks
|
||||
- Video upload notifications
|
||||
- Validation of webhook URLs
|
||||
"""
|
||||
|
||||
import logging
|
||||
import re
|
||||
from typing import Optional
|
||||
|
||||
import requests
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class DiscordNotificationService:
|
||||
"""
|
||||
Service for sending Discord webhook notifications.
|
||||
|
||||
Supports posting messages to Discord channels via webhooks,
|
||||
commonly used to notify when new karaoke videos are uploaded.
|
||||
"""
|
||||
|
||||
# Discord webhook URL pattern
|
||||
WEBHOOK_URL_PATTERN = re.compile(
|
||||
r"^https://discord\.com/api/webhooks/\d+/[a-zA-Z0-9_-]+$"
|
||||
)
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
webhook_url: Optional[str] = None,
|
||||
dry_run: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the Discord notification service.
|
||||
|
||||
Args:
|
||||
webhook_url: Discord webhook URL for posting notifications
|
||||
dry_run: If True, log actions without performing them
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.webhook_url = webhook_url
|
||||
self.dry_run = dry_run
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
# Validate webhook URL if provided
|
||||
if webhook_url:
|
||||
self._validate_webhook_url(webhook_url)
|
||||
|
||||
def _validate_webhook_url(self, url: str) -> None:
|
||||
"""
|
||||
Validate that the webhook URL is a valid Discord webhook URL.
|
||||
|
||||
Args:
|
||||
url: URL to validate
|
||||
|
||||
Raises:
|
||||
ValueError: If URL is not a valid Discord webhook URL
|
||||
"""
|
||||
url = url.strip()
|
||||
if not url.startswith("https://discord.com/api/webhooks/"):
|
||||
raise ValueError(
|
||||
f"Invalid Discord webhook URL: {url}. "
|
||||
"URL must start with 'https://discord.com/api/webhooks/'"
|
||||
)
|
||||
|
||||
def post_message(
|
||||
self,
|
||||
message: str,
|
||||
webhook_url: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Post a message to a Discord channel via webhook.
|
||||
|
||||
Args:
|
||||
message: The message content to post
|
||||
webhook_url: Optional webhook URL (uses instance URL if not provided)
|
||||
|
||||
Returns:
|
||||
True if message was posted successfully, False otherwise
|
||||
|
||||
Raises:
|
||||
ValueError: If no webhook URL is available
|
||||
requests.HTTPError: If the webhook request fails
|
||||
"""
|
||||
url = webhook_url or self.webhook_url
|
||||
if not url:
|
||||
raise ValueError("No Discord webhook URL provided")
|
||||
|
||||
url = url.strip()
|
||||
self._validate_webhook_url(url)
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(
|
||||
f"DRY RUN: Would post Discord message: '{message}' to {url}"
|
||||
)
|
||||
return True
|
||||
|
||||
self.logger.info(f"Posting message to Discord webhook...")
|
||||
data = {"content": message}
|
||||
response = requests.post(url, json=data, timeout=30)
|
||||
response.raise_for_status()
|
||||
self.logger.info("Message posted to Discord successfully")
|
||||
return True
|
||||
|
||||
def post_video_notification(
|
||||
self,
|
||||
youtube_url: str,
|
||||
webhook_url: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Post a notification about a new video upload.
|
||||
|
||||
Args:
|
||||
youtube_url: The YouTube URL of the uploaded video
|
||||
webhook_url: Optional webhook URL (uses instance URL if not provided)
|
||||
|
||||
Returns:
|
||||
True if notification was posted successfully, False otherwise
|
||||
"""
|
||||
if not youtube_url:
|
||||
self.logger.info("Skipping Discord notification - no YouTube URL available")
|
||||
return False
|
||||
|
||||
message = f"New upload: {youtube_url}"
|
||||
return self.post_message(message, webhook_url)
|
||||
|
||||
def is_enabled(self) -> bool:
|
||||
"""
|
||||
Check if Discord notifications are enabled.
|
||||
|
||||
Returns:
|
||||
True if a valid webhook URL is configured
|
||||
"""
|
||||
return bool(self.webhook_url)
|
||||
|
||||
|
||||
# Singleton instance and factory function (following existing service pattern)
|
||||
_discord_notification_service: Optional[DiscordNotificationService] = None
|
||||
|
||||
|
||||
def get_discord_notification_service(
|
||||
webhook_url: Optional[str] = None,
|
||||
**kwargs
|
||||
) -> DiscordNotificationService:
|
||||
"""
|
||||
Get a Discord notification service instance.
|
||||
|
||||
Args:
|
||||
webhook_url: Discord webhook URL for notifications
|
||||
**kwargs: Additional arguments passed to DiscordNotificationService
|
||||
|
||||
Returns:
|
||||
DiscordNotificationService instance
|
||||
"""
|
||||
global _discord_notification_service
|
||||
|
||||
# Create new instance if webhook URL changed or not yet created
|
||||
if _discord_notification_service is None or webhook_url:
|
||||
_discord_notification_service = DiscordNotificationService(
|
||||
webhook_url=webhook_url,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
return _discord_notification_service
|
||||
@@ -199,7 +199,7 @@ class DropboxService:
|
||||
|
||||
def upload_folder(self, local_dir: str, remote_path: str) -> None:
|
||||
"""
|
||||
Upload all files in a directory to Dropbox folder.
|
||||
Recursively upload all files and subdirectories to Dropbox folder.
|
||||
|
||||
Args:
|
||||
local_dir: Local directory to upload
|
||||
@@ -212,10 +212,17 @@ class DropboxService:
|
||||
logger.info(f"Uploading folder {local_dir} to {remote_path}")
|
||||
|
||||
uploaded_count = 0
|
||||
for filename in os.listdir(local_dir):
|
||||
local_file = os.path.join(local_dir, filename)
|
||||
if os.path.isfile(local_file):
|
||||
remote_file = f"{remote_path}/{filename}"
|
||||
for root, _dirs, files in os.walk(local_dir):
|
||||
# Calculate the relative path from local_dir to current root
|
||||
rel_root = os.path.relpath(root, local_dir)
|
||||
if rel_root == ".":
|
||||
current_remote = remote_path
|
||||
else:
|
||||
current_remote = f"{remote_path}/{rel_root}"
|
||||
|
||||
for filename in files:
|
||||
local_file = os.path.join(root, filename)
|
||||
remote_file = f"{current_remote}/{filename}"
|
||||
self.upload_file(local_file, remote_file)
|
||||
uploaded_count += 1
|
||||
|
||||
@@ -256,6 +263,38 @@ class DropboxService:
|
||||
return links.links[0].url
|
||||
raise
|
||||
|
||||
def delete_folder(self, path: str) -> bool:
|
||||
"""
|
||||
Delete a folder and all its contents from Dropbox.
|
||||
|
||||
Args:
|
||||
path: Dropbox path to delete (e.g., "/Karaoke/Tracks-Organized/NOMAD-1234 - Artist - Title")
|
||||
|
||||
Returns:
|
||||
True if deleted successfully, False otherwise
|
||||
"""
|
||||
from dropbox.exceptions import ApiError
|
||||
|
||||
# Ensure path starts with /
|
||||
if not path.startswith("/"):
|
||||
path = f"/{path}"
|
||||
|
||||
logger.info(f"Deleting Dropbox folder: {path}")
|
||||
|
||||
try:
|
||||
self.client.files_delete_v2(path)
|
||||
logger.info(f"Successfully deleted: {path}")
|
||||
return True
|
||||
except ApiError as e:
|
||||
if e.error.is_path_lookup() and e.error.get_path_lookup().is_not_found():
|
||||
logger.warning(f"Folder not found (already deleted?): {path}")
|
||||
return True # Consider it success if already gone
|
||||
logger.error(f"Failed to delete Dropbox folder: {e}")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting Dropbox folder: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def get_dropbox_service() -> DropboxService:
|
||||
"""Factory function to get a DropboxService instance."""
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,454 @@
|
||||
"""
|
||||
Encoding Interface.
|
||||
|
||||
Defines the abstract interface for video encoding backends, allowing
|
||||
the video worker orchestrator to use either GCE or local encoding
|
||||
interchangeably.
|
||||
|
||||
This follows the Strategy pattern - different encoding implementations
|
||||
can be swapped without changing the orchestration logic.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from abc import ABC, abstractmethod
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Dict, List, Optional, Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingInput:
|
||||
"""
|
||||
Input configuration for video encoding.
|
||||
|
||||
Contains all the paths and metadata needed to encode a karaoke video.
|
||||
"""
|
||||
# Required input files
|
||||
title_video_path: str # Title card video (MOV)
|
||||
karaoke_video_path: str # Main karaoke video with vocals (MOV/MKV)
|
||||
instrumental_audio_path: str # Instrumental audio track (FLAC)
|
||||
|
||||
# Optional input files
|
||||
end_video_path: Optional[str] = None # End credits video (MOV)
|
||||
|
||||
# Metadata for output naming
|
||||
artist: str = ""
|
||||
title: str = ""
|
||||
brand_code: Optional[str] = None
|
||||
|
||||
# Output directory
|
||||
output_dir: str = ""
|
||||
|
||||
# Additional options
|
||||
options: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingOutput:
|
||||
"""
|
||||
Output from video encoding.
|
||||
|
||||
Contains paths to all generated video files and status information.
|
||||
"""
|
||||
success: bool
|
||||
error_message: Optional[str] = None
|
||||
|
||||
# Output file paths (relative to output_dir or absolute)
|
||||
karaoke_mp4_path: Optional[str] = None # Karaoke video with instrumental audio
|
||||
with_vocals_mp4_path: Optional[str] = None # With vocals MP4 version
|
||||
lossless_4k_mp4_path: Optional[str] = None # Final lossless 4K MP4
|
||||
lossy_4k_mp4_path: Optional[str] = None # Final lossy 4K MP4
|
||||
lossless_mkv_path: Optional[str] = None # MKV with FLAC (for YouTube)
|
||||
lossy_720p_mp4_path: Optional[str] = None # 720p web version
|
||||
|
||||
# All output files as a dict for convenience
|
||||
output_files: Dict[str, str] = field(default_factory=dict)
|
||||
|
||||
# Encoding metadata
|
||||
encoding_time_seconds: Optional[float] = None
|
||||
encoding_backend: Optional[str] = None # "gce" or "local"
|
||||
|
||||
|
||||
class EncodingBackend(ABC):
|
||||
"""
|
||||
Abstract base class for video encoding backends.
|
||||
|
||||
Implementations:
|
||||
- GCEEncodingBackend: Cloud-based encoding using Google Compute Engine
|
||||
- LocalEncodingBackend: Local FFmpeg-based encoding
|
||||
"""
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def name(self) -> str:
|
||||
"""Return the name of this encoding backend."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
|
||||
"""
|
||||
Encode video files according to the input configuration.
|
||||
|
||||
This method should:
|
||||
1. Create karaoke video with instrumental audio
|
||||
2. Convert with-vocals video to MP4 if needed
|
||||
3. Encode lossless 4K MP4 (concatenating title + karaoke + end)
|
||||
4. Encode lossy 4K MP4 with AAC audio
|
||||
5. Create MKV with FLAC audio for YouTube
|
||||
6. Encode 720p version for web
|
||||
|
||||
Args:
|
||||
input_config: EncodingInput with all required paths and options
|
||||
|
||||
Returns:
|
||||
EncodingOutput with paths to generated files and status
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def is_available(self) -> bool:
|
||||
"""
|
||||
Check if this encoding backend is available and configured.
|
||||
|
||||
Returns:
|
||||
True if the backend can be used, False otherwise
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def get_status(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Get the current status of the encoding backend.
|
||||
|
||||
Returns:
|
||||
Dict with status information (availability, queue length, etc.)
|
||||
"""
|
||||
pass
|
||||
|
||||
|
||||
class LocalEncodingBackend(EncodingBackend):
|
||||
"""
|
||||
Local FFmpeg-based encoding backend.
|
||||
|
||||
Wraps the LocalEncodingService to implement the EncodingBackend interface.
|
||||
Uses asyncio.to_thread() to run synchronous FFmpeg operations asynchronously.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
dry_run: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the local encoding backend.
|
||||
|
||||
Args:
|
||||
dry_run: If True, log operations without executing
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.dry_run = dry_run
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
self._service = None
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return "local"
|
||||
|
||||
def _get_service(self):
|
||||
"""Lazy-load the local encoding service."""
|
||||
if self._service is None:
|
||||
from backend.services.local_encoding_service import LocalEncodingService
|
||||
self._service = LocalEncodingService(
|
||||
dry_run=self.dry_run,
|
||||
logger=self.logger
|
||||
)
|
||||
return self._service
|
||||
|
||||
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
|
||||
"""
|
||||
Encode video using local FFmpeg.
|
||||
|
||||
Args:
|
||||
input_config: Encoding input configuration
|
||||
|
||||
Returns:
|
||||
EncodingOutput with results
|
||||
"""
|
||||
import asyncio
|
||||
import time
|
||||
from backend.services.local_encoding_service import EncodingConfig
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
# Build output file paths
|
||||
base_name = f"{input_config.artist} - {input_config.title}"
|
||||
output_dir = input_config.output_dir or "."
|
||||
|
||||
import os
|
||||
config = EncodingConfig(
|
||||
title_video=input_config.title_video_path,
|
||||
karaoke_video=input_config.karaoke_video_path,
|
||||
instrumental_audio=input_config.instrumental_audio_path,
|
||||
end_video=input_config.end_video_path,
|
||||
output_karaoke_mp4=os.path.join(output_dir, f"{base_name} (Karaoke).mp4"),
|
||||
output_with_vocals_mp4=os.path.join(output_dir, f"{base_name} (With Vocals).mp4"),
|
||||
output_lossless_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mp4"),
|
||||
output_lossy_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 4k).mp4"),
|
||||
output_lossless_mkv=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mkv"),
|
||||
output_720p_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 720p).mp4"),
|
||||
)
|
||||
|
||||
# Run encoding in thread pool to avoid blocking
|
||||
service = self._get_service()
|
||||
result = await asyncio.to_thread(service.encode_all_formats, config)
|
||||
|
||||
encoding_time = time.time() - start_time
|
||||
|
||||
if result.success:
|
||||
return EncodingOutput(
|
||||
success=True,
|
||||
karaoke_mp4_path=config.output_karaoke_mp4,
|
||||
with_vocals_mp4_path=config.output_with_vocals_mp4,
|
||||
lossless_4k_mp4_path=config.output_lossless_4k_mp4,
|
||||
lossy_4k_mp4_path=config.output_lossy_4k_mp4,
|
||||
lossless_mkv_path=config.output_lossless_mkv,
|
||||
lossy_720p_mp4_path=config.output_720p_mp4,
|
||||
output_files=result.output_files,
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
)
|
||||
else:
|
||||
return EncodingOutput(
|
||||
success=False,
|
||||
error_message=result.error,
|
||||
output_files=result.output_files,
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
)
|
||||
|
||||
async def is_available(self) -> bool:
|
||||
"""Check if local encoding is available (FFmpeg installed)."""
|
||||
import subprocess
|
||||
import asyncio
|
||||
|
||||
try:
|
||||
result = await asyncio.to_thread(
|
||||
subprocess.run,
|
||||
["ffmpeg", "-version"],
|
||||
capture_output=True,
|
||||
timeout=10
|
||||
)
|
||||
return result.returncode == 0
|
||||
except (FileNotFoundError, subprocess.TimeoutExpired):
|
||||
return False
|
||||
|
||||
async def get_status(self) -> Dict[str, Any]:
|
||||
"""Get local encoding status."""
|
||||
available = await self.is_available()
|
||||
service = self._get_service()
|
||||
|
||||
return {
|
||||
"backend": self.name,
|
||||
"available": available,
|
||||
"hwaccel_available": service.hwaccel_available if available else False,
|
||||
"video_encoder": service.video_encoder if available else None,
|
||||
}
|
||||
|
||||
|
||||
class GCEEncodingBackend(EncodingBackend):
|
||||
"""
|
||||
GCE-based encoding backend.
|
||||
|
||||
Wraps the existing EncodingService to implement the EncodingBackend interface.
|
||||
Submits jobs to a remote GCE worker for high-performance encoding.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
dry_run: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the GCE encoding backend.
|
||||
|
||||
Args:
|
||||
dry_run: Ignored for GCE backend (remote service doesn't support dry run)
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.dry_run = dry_run # Stored but not used (GCE is remote)
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
self._service = None
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return "gce"
|
||||
|
||||
def _get_service(self):
|
||||
"""Lazy-load the GCE encoding service."""
|
||||
if self._service is None:
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
self._service = get_encoding_service()
|
||||
return self._service
|
||||
|
||||
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
|
||||
"""
|
||||
Encode video using GCE worker.
|
||||
|
||||
Note: GCE encoding requires files to be in GCS, so this method
|
||||
expects input_config.options to contain 'input_gcs_path' and
|
||||
'output_gcs_path'.
|
||||
|
||||
Args:
|
||||
input_config: Encoding input configuration
|
||||
|
||||
Returns:
|
||||
EncodingOutput with results
|
||||
"""
|
||||
import time
|
||||
|
||||
start_time = time.time()
|
||||
service = self._get_service()
|
||||
|
||||
# GCE requires GCS paths
|
||||
input_gcs_path = input_config.options.get("input_gcs_path")
|
||||
output_gcs_path = input_config.options.get("output_gcs_path")
|
||||
job_id = input_config.options.get("job_id", input_config.brand_code or "unknown")
|
||||
|
||||
if not input_gcs_path or not output_gcs_path:
|
||||
return EncodingOutput(
|
||||
success=False,
|
||||
error_message="GCE encoding requires input_gcs_path and output_gcs_path in options",
|
||||
encoding_backend=self.name
|
||||
)
|
||||
|
||||
try:
|
||||
# Build encoding config
|
||||
encoding_config = {
|
||||
"formats": ["mp4_4k_lossless", "mp4_4k_lossy", "mkv_4k", "mp4_720p"],
|
||||
"artist": input_config.artist,
|
||||
"title": input_config.title,
|
||||
}
|
||||
|
||||
# Submit and wait for completion
|
||||
result = await service.encode_videos(
|
||||
job_id=job_id,
|
||||
input_gcs_path=input_gcs_path,
|
||||
output_gcs_path=output_gcs_path,
|
||||
encoding_config=encoding_config,
|
||||
)
|
||||
|
||||
encoding_time = time.time() - start_time
|
||||
|
||||
# Extract output file paths from result
|
||||
# Handle case where GCE worker returns a list or unexpected format
|
||||
if isinstance(result, list):
|
||||
# If result is a list, try to find the output_files in the first dict
|
||||
self.logger.warning(f"GCE returned list instead of dict: {result}")
|
||||
result = result[0] if result and isinstance(result[0], dict) else {}
|
||||
if not isinstance(result, dict):
|
||||
self.logger.error(f"Unexpected GCE result type: {type(result)}")
|
||||
result = {}
|
||||
raw_output_files = result.get("output_files", {})
|
||||
|
||||
# Convert output_files from list of paths to dict
|
||||
# GCE worker returns list like: ["path/Artist - Title (Final Karaoke Lossless 4k).mp4", ...]
|
||||
# We need dict like: {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
|
||||
if isinstance(raw_output_files, list):
|
||||
output_files = {}
|
||||
for path in raw_output_files:
|
||||
if not isinstance(path, str):
|
||||
continue
|
||||
filename = path.split("/")[-1] if "/" in path else path
|
||||
filename_lower = filename.lower()
|
||||
# Map filename patterns to output format keys
|
||||
# Files are named like "Artist - Title (Final Karaoke Lossless 4k).mp4"
|
||||
if "lossless 4k" in filename_lower:
|
||||
if filename.endswith(".mkv"):
|
||||
output_files["mkv_4k"] = path
|
||||
else:
|
||||
output_files["mp4_4k_lossless"] = path
|
||||
elif "lossy 4k" in filename_lower:
|
||||
output_files["mp4_4k_lossy"] = path
|
||||
elif "720p" in filename_lower:
|
||||
output_files["mp4_720p"] = path
|
||||
self.logger.info(f"Converted output_files list to dict: {output_files}")
|
||||
else:
|
||||
output_files = raw_output_files if isinstance(raw_output_files, dict) else {}
|
||||
|
||||
return EncodingOutput(
|
||||
success=True,
|
||||
lossless_4k_mp4_path=output_files.get("mp4_4k_lossless"),
|
||||
lossy_4k_mp4_path=output_files.get("mp4_4k_lossy"),
|
||||
lossless_mkv_path=output_files.get("mkv_4k"),
|
||||
lossy_720p_mp4_path=output_files.get("mp4_720p"),
|
||||
output_files=output_files,
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
encoding_time = time.time() - start_time
|
||||
self.logger.error(f"GCE encoding failed: {e}")
|
||||
return EncodingOutput(
|
||||
success=False,
|
||||
error_message=str(e),
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
)
|
||||
|
||||
async def is_available(self) -> bool:
|
||||
"""Check if GCE encoding is available and configured."""
|
||||
service = self._get_service()
|
||||
return service.is_enabled
|
||||
|
||||
async def get_status(self) -> Dict[str, Any]:
|
||||
"""Get GCE encoding status."""
|
||||
service = self._get_service()
|
||||
|
||||
status = {
|
||||
"backend": self.name,
|
||||
"available": service.is_enabled,
|
||||
"configured": service.is_configured,
|
||||
}
|
||||
|
||||
if service.is_configured:
|
||||
try:
|
||||
health = await service.health_check()
|
||||
status["health"] = health
|
||||
except Exception as e:
|
||||
status["health_error"] = str(e)
|
||||
|
||||
return status
|
||||
|
||||
|
||||
# Factory function to get an encoding backend
|
||||
def get_encoding_backend(
|
||||
backend_type: str = "auto",
|
||||
**kwargs
|
||||
) -> EncodingBackend:
|
||||
"""
|
||||
Get an encoding backend instance.
|
||||
|
||||
Args:
|
||||
backend_type: Type of backend - "local", "gce", or "auto"
|
||||
**kwargs: Additional arguments for the backend
|
||||
|
||||
Returns:
|
||||
EncodingBackend instance
|
||||
|
||||
Raises:
|
||||
ValueError: If backend_type is unknown
|
||||
"""
|
||||
if backend_type == "local":
|
||||
return LocalEncodingBackend(**kwargs)
|
||||
elif backend_type == "gce":
|
||||
return GCEEncodingBackend(**kwargs)
|
||||
elif backend_type == "auto":
|
||||
# Check if GCE is available, otherwise use local
|
||||
gce_backend = GCEEncodingBackend(**kwargs)
|
||||
if gce_backend._get_service().is_enabled:
|
||||
return gce_backend
|
||||
return LocalEncodingBackend(**kwargs)
|
||||
else:
|
||||
raise ValueError(f"Unknown encoding backend type: {backend_type}")
|
||||
@@ -0,0 +1,502 @@
|
||||
"""
|
||||
GCE Encoding Worker Service.
|
||||
|
||||
This service dispatches video encoding jobs to a dedicated high-performance
|
||||
GCE instance (C4-standard with Intel Granite Rapids CPU) for faster encoding.
|
||||
|
||||
The GCE worker provides:
|
||||
- 3.9 GHz all-core frequency (vs 3.7 GHz on Cloud Run)
|
||||
- Dedicated vCPUs (no contention)
|
||||
- 2-3x faster FFmpeg libx264 encoding
|
||||
|
||||
Usage:
|
||||
encoding_service = get_encoding_service()
|
||||
if encoding_service.is_configured:
|
||||
result = await encoding_service.encode_videos(job_id, input_gcs_path, config)
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
import aiohttp
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Retry configuration for handling transient failures (e.g., worker restarts)
|
||||
MAX_RETRIES = 3
|
||||
INITIAL_BACKOFF_SECONDS = 2.0
|
||||
MAX_BACKOFF_SECONDS = 10.0
|
||||
|
||||
|
||||
class EncodingService:
|
||||
"""Service for dispatching encoding jobs to GCE worker."""
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
self._url = None
|
||||
self._api_key = None
|
||||
self._initialized = False
|
||||
|
||||
def _load_credentials(self):
|
||||
"""Load encoding worker URL and API key from config/secrets."""
|
||||
if self._initialized:
|
||||
return
|
||||
|
||||
# Try environment variables first, then Secret Manager
|
||||
self._url = self.settings.encoding_worker_url
|
||||
self._api_key = self.settings.encoding_worker_api_key
|
||||
|
||||
# Fall back to Secret Manager
|
||||
if not self._url:
|
||||
self._url = self.settings.get_secret("encoding-worker-url")
|
||||
if not self._api_key:
|
||||
self._api_key = self.settings.get_secret("encoding-worker-api-key")
|
||||
|
||||
self._initialized = True
|
||||
|
||||
@property
|
||||
def is_configured(self) -> bool:
|
||||
"""Check if encoding service is configured with URL and API key."""
|
||||
self._load_credentials()
|
||||
return bool(self._url and self._api_key)
|
||||
|
||||
@property
|
||||
def is_enabled(self) -> bool:
|
||||
"""Check if GCE encoding is enabled and configured."""
|
||||
return self.settings.use_gce_encoding and self.is_configured
|
||||
|
||||
@property
|
||||
def is_preview_enabled(self) -> bool:
|
||||
"""Check if GCE preview encoding is enabled and configured."""
|
||||
return self.settings.use_gce_preview_encoding and self.is_configured
|
||||
|
||||
async def _request_with_retry(
|
||||
self,
|
||||
method: str,
|
||||
url: str,
|
||||
headers: Dict[str, str],
|
||||
json_payload: Optional[Dict[str, Any]] = None,
|
||||
timeout: float = 30.0,
|
||||
job_id: str = "unknown",
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Make an HTTP request with retry logic for transient failures.
|
||||
|
||||
This handles connection errors that occur when the GCE worker is
|
||||
restarting (e.g., during deployments) by retrying with exponential backoff.
|
||||
|
||||
Args:
|
||||
method: HTTP method (GET, POST)
|
||||
url: Request URL
|
||||
headers: Request headers
|
||||
json_payload: JSON body for POST requests
|
||||
timeout: Request timeout in seconds
|
||||
job_id: Job ID for logging
|
||||
|
||||
Returns:
|
||||
Dict with keys:
|
||||
- status (int): HTTP status code
|
||||
- json (Any): Parsed JSON response body (if status 200, else None)
|
||||
- text (str): Raw response text (if status != 200, else None)
|
||||
|
||||
Raises:
|
||||
aiohttp.ClientConnectorError: If all retries fail due to connection errors
|
||||
aiohttp.ServerDisconnectedError: If all retries fail due to server disconnect
|
||||
asyncio.TimeoutError: If all retries fail due to timeout
|
||||
"""
|
||||
last_exception = None
|
||||
backoff = INITIAL_BACKOFF_SECONDS
|
||||
|
||||
for attempt in range(MAX_RETRIES + 1):
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
if method.upper() == "POST":
|
||||
async with session.post(
|
||||
url, json=json_payload, headers=headers, timeout=timeout
|
||||
) as resp:
|
||||
# Return a copy of the response data since we exit the context
|
||||
return {
|
||||
"status": resp.status,
|
||||
"json": await resp.json() if resp.status == 200 else None,
|
||||
"text": await resp.text() if resp.status != 200 else None,
|
||||
}
|
||||
else: # GET
|
||||
async with session.get(
|
||||
url, headers=headers, timeout=timeout
|
||||
) as resp:
|
||||
return {
|
||||
"status": resp.status,
|
||||
"json": await resp.json() if resp.status == 200 else None,
|
||||
"text": await resp.text() if resp.status != 200 else None,
|
||||
}
|
||||
except (aiohttp.ClientConnectorError, aiohttp.ServerDisconnectedError, asyncio.TimeoutError) as e:
|
||||
last_exception = e
|
||||
if attempt < MAX_RETRIES:
|
||||
logger.warning(
|
||||
f"[job:{job_id}] GCE worker connection failed (attempt {attempt + 1}/{MAX_RETRIES + 1}): {e}. "
|
||||
f"Retrying in {backoff:.1f}s..."
|
||||
)
|
||||
await asyncio.sleep(backoff)
|
||||
backoff = min(backoff * 2, MAX_BACKOFF_SECONDS)
|
||||
else:
|
||||
logger.error(
|
||||
f"[job:{job_id}] GCE worker connection failed after {MAX_RETRIES + 1} attempts: {e}"
|
||||
)
|
||||
|
||||
raise last_exception
|
||||
|
||||
async def submit_encoding_job(
|
||||
self,
|
||||
job_id: str,
|
||||
input_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
encoding_config: Dict[str, Any],
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit an encoding job to the GCE worker.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
input_gcs_path: GCS path to input files (gs://bucket/path/)
|
||||
output_gcs_path: GCS path for output files (gs://bucket/path/)
|
||||
encoding_config: Configuration for encoding (formats, quality, etc.)
|
||||
|
||||
Returns:
|
||||
Response from the encoding worker
|
||||
|
||||
Raises:
|
||||
Exception: If submission fails
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._url}/encode"
|
||||
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
|
||||
payload = {
|
||||
"job_id": job_id,
|
||||
"input_gcs_path": input_gcs_path,
|
||||
"output_gcs_path": output_gcs_path,
|
||||
"encoding_config": encoding_config,
|
||||
}
|
||||
|
||||
logger.info(f"[job:{job_id}] Submitting encoding job to GCE worker: {url}")
|
||||
|
||||
resp = await self._request_with_retry(
|
||||
method="POST",
|
||||
url=url,
|
||||
headers=headers,
|
||||
json_payload=payload,
|
||||
timeout=30.0,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
if resp["status"] == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp["status"] == 409:
|
||||
raise RuntimeError(f"Encoding job {job_id} already exists")
|
||||
if resp["status"] != 200:
|
||||
raise RuntimeError(f"Failed to submit encoding job: {resp['status']} - {resp['text']}")
|
||||
|
||||
return resp["json"]
|
||||
|
||||
async def get_job_status(self, job_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Get the status of an encoding job.
|
||||
|
||||
Args:
|
||||
job_id: Job identifier
|
||||
|
||||
Returns:
|
||||
Job status including: status, progress, error, output_files
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._url}/status/{job_id}"
|
||||
headers = {"X-API-Key": self._api_key}
|
||||
|
||||
resp = await self._request_with_retry(
|
||||
method="GET",
|
||||
url=url,
|
||||
headers=headers,
|
||||
timeout=30.0,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
if resp["status"] == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp["status"] == 404:
|
||||
raise RuntimeError(f"Encoding job {job_id} not found")
|
||||
if resp["status"] != 200:
|
||||
raise RuntimeError(f"Failed to get job status: {resp['status']} - {resp['text']}")
|
||||
|
||||
return resp["json"]
|
||||
|
||||
async def wait_for_completion(
|
||||
self,
|
||||
job_id: str,
|
||||
poll_interval: float = 10.0,
|
||||
timeout: float = 3600.0,
|
||||
progress_callback=None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Poll for encoding job completion.
|
||||
|
||||
Args:
|
||||
job_id: Job identifier
|
||||
poll_interval: Seconds between status checks
|
||||
timeout: Maximum time to wait (default 1 hour)
|
||||
progress_callback: Optional callback(progress: int) for progress updates
|
||||
|
||||
Returns:
|
||||
Final job status with output files
|
||||
|
||||
Raises:
|
||||
TimeoutError: If job doesn't complete within timeout
|
||||
RuntimeError: If job fails
|
||||
"""
|
||||
logger.info(f"[job:{job_id}] Waiting for GCE encoding to complete...")
|
||||
|
||||
start_time = asyncio.get_event_loop().time()
|
||||
last_progress = 0
|
||||
|
||||
while True:
|
||||
elapsed = asyncio.get_event_loop().time() - start_time
|
||||
if elapsed > timeout:
|
||||
raise TimeoutError(f"Encoding job {job_id} timed out after {timeout}s")
|
||||
|
||||
status = await self.get_job_status(job_id)
|
||||
|
||||
# Handle case where GCE worker returns a list instead of dict
|
||||
if isinstance(status, list):
|
||||
logger.warning(f"[job:{job_id}] GCE returned list instead of dict: {status}")
|
||||
status = status[0] if status and isinstance(status[0], dict) else {}
|
||||
if not isinstance(status, dict):
|
||||
logger.error(f"[job:{job_id}] Unexpected status type: {type(status)}")
|
||||
status = {}
|
||||
|
||||
job_status = status.get("status", "unknown")
|
||||
progress = status.get("progress", 0)
|
||||
|
||||
# Report progress
|
||||
if progress != last_progress:
|
||||
logger.info(f"[job:{job_id}] Encoding progress: {progress}%")
|
||||
last_progress = progress
|
||||
if progress_callback:
|
||||
try:
|
||||
progress_callback(progress)
|
||||
except Exception as e:
|
||||
logger.warning(f"Progress callback failed: {e}")
|
||||
|
||||
if job_status == "complete":
|
||||
logger.info(f"[job:{job_id}] GCE encoding complete in {elapsed:.1f}s")
|
||||
return status
|
||||
|
||||
if job_status == "failed":
|
||||
error = status.get("error", "Unknown error")
|
||||
raise RuntimeError(f"Encoding job {job_id} failed: {error}")
|
||||
|
||||
await asyncio.sleep(poll_interval)
|
||||
|
||||
async def encode_videos(
|
||||
self,
|
||||
job_id: str,
|
||||
input_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
encoding_config: Optional[Dict[str, Any]] = None,
|
||||
progress_callback=None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit encoding job and wait for completion.
|
||||
|
||||
This is a convenience method that combines submit + wait.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
input_gcs_path: GCS path to input files
|
||||
output_gcs_path: GCS path for output files
|
||||
encoding_config: Optional encoding configuration
|
||||
progress_callback: Optional callback for progress updates
|
||||
|
||||
Returns:
|
||||
Final job status with output files
|
||||
"""
|
||||
config = encoding_config or {"formats": ["mp4_4k", "mp4_720p"]}
|
||||
|
||||
# Submit the job
|
||||
await self.submit_encoding_job(job_id, input_gcs_path, output_gcs_path, config)
|
||||
|
||||
# Wait for completion
|
||||
return await self.wait_for_completion(
|
||||
job_id, progress_callback=progress_callback
|
||||
)
|
||||
|
||||
async def submit_preview_encoding_job(
|
||||
self,
|
||||
job_id: str,
|
||||
ass_gcs_path: str,
|
||||
audio_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
background_color: str = "black",
|
||||
background_image_gcs_path: Optional[str] = None,
|
||||
font_gcs_path: Optional[str] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit a preview video encoding job to the GCE worker.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
ass_gcs_path: GCS path to ASS subtitles file (gs://bucket/path/file.ass)
|
||||
audio_gcs_path: GCS path to audio file
|
||||
output_gcs_path: GCS path for output video
|
||||
background_color: Background color (default: black)
|
||||
background_image_gcs_path: Optional GCS path to background image
|
||||
font_gcs_path: Optional GCS path to custom font file
|
||||
|
||||
Returns:
|
||||
Response from the encoding worker
|
||||
|
||||
Raises:
|
||||
Exception: If submission fails
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._url}/encode-preview"
|
||||
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
|
||||
payload = {
|
||||
"job_id": job_id,
|
||||
"ass_gcs_path": ass_gcs_path,
|
||||
"audio_gcs_path": audio_gcs_path,
|
||||
"output_gcs_path": output_gcs_path,
|
||||
"background_color": background_color,
|
||||
}
|
||||
if background_image_gcs_path:
|
||||
payload["background_image_gcs_path"] = background_image_gcs_path
|
||||
if font_gcs_path:
|
||||
payload["font_gcs_path"] = font_gcs_path
|
||||
|
||||
logger.info(f"[job:{job_id}] Submitting preview encoding job to GCE worker: {url}")
|
||||
|
||||
resp = await self._request_with_retry(
|
||||
method="POST",
|
||||
url=url,
|
||||
headers=headers,
|
||||
json_payload=payload,
|
||||
timeout=30.0,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
if resp["status"] == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp["status"] == 409:
|
||||
raise RuntimeError(f"Preview encoding job {job_id} already exists")
|
||||
if resp["status"] != 200:
|
||||
raise RuntimeError(f"Failed to submit preview encoding job: {resp['status']} - {resp['text']}")
|
||||
|
||||
return resp["json"]
|
||||
|
||||
async def encode_preview_video(
|
||||
self,
|
||||
job_id: str,
|
||||
ass_gcs_path: str,
|
||||
audio_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
background_color: str = "black",
|
||||
background_image_gcs_path: Optional[str] = None,
|
||||
font_gcs_path: Optional[str] = None,
|
||||
timeout: float = 90.0,
|
||||
poll_interval: float = 2.0,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit preview encoding job and wait for completion.
|
||||
|
||||
This is a convenience method that combines submit + wait with shorter
|
||||
timeouts suitable for preview videos.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
ass_gcs_path: GCS path to ASS subtitles file
|
||||
audio_gcs_path: GCS path to audio file
|
||||
output_gcs_path: GCS path for output video
|
||||
background_color: Background color (default: black)
|
||||
background_image_gcs_path: Optional GCS path to background image
|
||||
font_gcs_path: Optional GCS path to custom font file
|
||||
timeout: Maximum time to wait (default 90s for preview)
|
||||
poll_interval: Seconds between status checks (default 2s)
|
||||
|
||||
Returns:
|
||||
Final job status with output files
|
||||
"""
|
||||
# Submit the job
|
||||
submit_result = await self.submit_preview_encoding_job(
|
||||
job_id=job_id,
|
||||
ass_gcs_path=ass_gcs_path,
|
||||
audio_gcs_path=audio_gcs_path,
|
||||
output_gcs_path=output_gcs_path,
|
||||
background_color=background_color,
|
||||
background_image_gcs_path=background_image_gcs_path,
|
||||
font_gcs_path=font_gcs_path,
|
||||
)
|
||||
|
||||
# If cached, return immediately - video already exists in GCS
|
||||
submit_status = submit_result.get("status")
|
||||
if submit_status == "cached":
|
||||
logger.info(f"[job:{job_id}] Preview already cached, returning immediately")
|
||||
return {"status": "complete", "output_path": submit_result.get("output_path")}
|
||||
|
||||
# If in_progress, another request is encoding it - just wait for that
|
||||
if submit_status == "in_progress":
|
||||
logger.info(f"[job:{job_id}] Preview encoding already in progress, waiting")
|
||||
|
||||
# Wait for completion with shorter timeout
|
||||
return await self.wait_for_completion(
|
||||
job_id=job_id,
|
||||
poll_interval=poll_interval,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
async def health_check(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Check the health of the encoding worker.
|
||||
|
||||
Returns:
|
||||
Health status including active jobs and FFmpeg version
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
return {"status": "not_configured"}
|
||||
|
||||
url = f"{self._url}/health"
|
||||
headers = {"X-API-Key": self._api_key}
|
||||
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(url, headers=headers, timeout=10) as resp:
|
||||
if resp.status == 200:
|
||||
return await resp.json()
|
||||
return {"status": "error", "code": resp.status}
|
||||
except Exception as e:
|
||||
return {"status": "error", "error": str(e)}
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_encoding_service: Optional[EncodingService] = None
|
||||
|
||||
|
||||
def get_encoding_service() -> EncodingService:
|
||||
"""Get the singleton encoding service instance."""
|
||||
global _encoding_service
|
||||
if _encoding_service is None:
|
||||
_encoding_service = EncodingService()
|
||||
return _encoding_service
|
||||
@@ -9,6 +9,7 @@ from google.cloud.firestore_v1 import FieldFilter
|
||||
|
||||
from backend.config import settings
|
||||
from backend.models.job import Job, JobStatus, TimelineEvent
|
||||
from backend.models.worker_log import WorkerLogEntry
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -114,47 +115,53 @@ class FirestoreService:
|
||||
client_id: Optional[str] = None,
|
||||
created_after: Optional[datetime] = None,
|
||||
created_before: Optional[datetime] = None,
|
||||
user_email: Optional[str] = None,
|
||||
limit: int = 100
|
||||
) -> List[Job]:
|
||||
"""
|
||||
List jobs with optional filters.
|
||||
|
||||
|
||||
Args:
|
||||
status: Filter by job status
|
||||
environment: Filter by request_metadata.environment (test/production/development)
|
||||
client_id: Filter by request_metadata.client_id
|
||||
created_after: Filter jobs created after this datetime
|
||||
created_before: Filter jobs created before this datetime
|
||||
user_email: Filter by user_email (owner of the job)
|
||||
limit: Maximum number of jobs to return
|
||||
|
||||
|
||||
Returns:
|
||||
List of Job objects matching filters, ordered by created_at descending
|
||||
"""
|
||||
try:
|
||||
query = self.db.collection(self.collection)
|
||||
|
||||
|
||||
if status:
|
||||
query = query.where(filter=FieldFilter('status', '==', status.value))
|
||||
|
||||
|
||||
# Filter by request_metadata fields using dot notation
|
||||
if environment:
|
||||
query = query.where(filter=FieldFilter('request_metadata.environment', '==', environment))
|
||||
|
||||
|
||||
if client_id:
|
||||
query = query.where(filter=FieldFilter('request_metadata.client_id', '==', client_id))
|
||||
|
||||
|
||||
# Filter by user_email (job owner)
|
||||
if user_email:
|
||||
query = query.where(filter=FieldFilter('user_email', '==', user_email.lower()))
|
||||
|
||||
# Date range filters
|
||||
if created_after:
|
||||
query = query.where(filter=FieldFilter('created_at', '>=', created_after))
|
||||
|
||||
|
||||
if created_before:
|
||||
query = query.where(filter=FieldFilter('created_at', '<=', created_before))
|
||||
|
||||
|
||||
query = query.order_by('created_at', direction=firestore.Query.DESCENDING).limit(limit)
|
||||
|
||||
|
||||
docs = query.stream()
|
||||
jobs = [Job(**doc.to_dict()) for doc in docs]
|
||||
|
||||
|
||||
return jobs
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing jobs: {e}")
|
||||
@@ -239,10 +246,13 @@ class FirestoreService:
|
||||
def append_worker_log(self, job_id: str, log_entry: Dict[str, Any]) -> None:
|
||||
"""
|
||||
Atomically append a log entry to worker_logs using ArrayUnion.
|
||||
|
||||
|
||||
This avoids the race condition of read-modify-write when multiple
|
||||
workers are logging concurrently.
|
||||
|
||||
|
||||
DEPRECATED: Use append_log_to_subcollection() instead to avoid
|
||||
the 1MB document size limit.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
log_entry: Log entry dict with timestamp, level, worker, message
|
||||
@@ -257,6 +267,165 @@ class FirestoreService:
|
||||
except Exception as e:
|
||||
# Log but don't raise - logging shouldn't break workers
|
||||
logger.debug(f"Error appending worker log for job {job_id}: {e}")
|
||||
|
||||
# ============================================
|
||||
# Worker Log Subcollection Methods
|
||||
# ============================================
|
||||
# These methods store logs in a subcollection (jobs/{job_id}/logs)
|
||||
# instead of an embedded array to avoid the 1MB document size limit.
|
||||
|
||||
def append_log_to_subcollection(self, job_id: str, log_entry: WorkerLogEntry) -> None:
|
||||
"""
|
||||
Append a log entry to the logs subcollection.
|
||||
|
||||
Stores logs at: jobs/{job_id}/logs/{log_id}
|
||||
|
||||
This approach avoids the 1MB document size limit by storing each
|
||||
log entry as a separate document in a subcollection.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
log_entry: WorkerLogEntry instance
|
||||
"""
|
||||
try:
|
||||
# Ensure job_id is set on the log entry
|
||||
log_entry.job_id = job_id
|
||||
|
||||
# Get subcollection reference
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
|
||||
# Add document with auto-generated ID or use log_entry.id
|
||||
doc_ref = logs_ref.document(log_entry.id)
|
||||
doc_ref.set(log_entry.to_dict())
|
||||
|
||||
# Don't log every append - too spammy
|
||||
except Exception as e:
|
||||
# Log but don't raise - logging shouldn't break workers
|
||||
logger.debug(f"Error appending log to subcollection for job {job_id}: {e}")
|
||||
|
||||
def get_logs_from_subcollection(
|
||||
self,
|
||||
job_id: str,
|
||||
limit: int = 500,
|
||||
since_timestamp: Optional[datetime] = None,
|
||||
worker: Optional[str] = None,
|
||||
offset: int = 0
|
||||
) -> List[WorkerLogEntry]:
|
||||
"""
|
||||
Get log entries from the logs subcollection.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
limit: Maximum number of logs to return (default 500)
|
||||
since_timestamp: Return only logs after this timestamp
|
||||
worker: Filter by worker name (optional)
|
||||
offset: Number of logs to skip (for pagination)
|
||||
|
||||
Returns:
|
||||
List of WorkerLogEntry instances, ordered by timestamp ascending
|
||||
"""
|
||||
try:
|
||||
# Get subcollection reference
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
|
||||
# Build query
|
||||
query = logs_ref.order_by("timestamp", direction=firestore.Query.ASCENDING)
|
||||
|
||||
if since_timestamp:
|
||||
query = query.where(filter=FieldFilter("timestamp", ">", since_timestamp))
|
||||
|
||||
if worker:
|
||||
query = query.where(filter=FieldFilter("worker", "==", worker))
|
||||
|
||||
# Apply offset and limit
|
||||
if offset > 0:
|
||||
query = query.offset(offset)
|
||||
|
||||
query = query.limit(limit)
|
||||
|
||||
# Execute query
|
||||
docs = query.stream()
|
||||
logs = [WorkerLogEntry.from_dict(doc.to_dict()) for doc in docs]
|
||||
|
||||
return logs
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting logs from subcollection for job {job_id}: {e}")
|
||||
return []
|
||||
|
||||
def get_logs_count_from_subcollection(self, job_id: str) -> int:
|
||||
"""
|
||||
Get the total count of log entries in the subcollection.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
|
||||
Returns:
|
||||
Total count of log entries
|
||||
"""
|
||||
try:
|
||||
# Get subcollection reference
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
|
||||
# Use aggregation query for efficient counting
|
||||
count_query = logs_ref.count()
|
||||
result = count_query.get()
|
||||
|
||||
# Result is a list of AggregationResult, we want the first one's count
|
||||
if result and len(result) > 0:
|
||||
return result[0][0].value
|
||||
return 0
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error counting logs for job {job_id}: {e}")
|
||||
return 0
|
||||
|
||||
def delete_logs_subcollection(self, job_id: str, batch_size: int = 500) -> int:
|
||||
"""
|
||||
Delete all log entries in the logs subcollection.
|
||||
|
||||
This is used when deleting a job to clean up its logs.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
batch_size: Number of documents to delete per batch
|
||||
|
||||
Returns:
|
||||
Number of logs deleted
|
||||
"""
|
||||
try:
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
deleted_count = 0
|
||||
|
||||
while True:
|
||||
# Get a batch of documents
|
||||
docs = logs_ref.limit(batch_size).stream()
|
||||
deleted_in_batch = 0
|
||||
|
||||
# Delete in a batch
|
||||
batch = self.db.batch()
|
||||
for doc in docs:
|
||||
batch.delete(doc.reference)
|
||||
deleted_in_batch += 1
|
||||
|
||||
if deleted_in_batch == 0:
|
||||
break
|
||||
|
||||
batch.commit()
|
||||
deleted_count += deleted_in_batch
|
||||
|
||||
# If we deleted less than batch_size, we're done
|
||||
if deleted_in_batch < batch_size:
|
||||
break
|
||||
|
||||
if deleted_count > 0:
|
||||
logger.info(f"Deleted {deleted_count} logs for job {job_id}")
|
||||
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting logs subcollection for job {job_id}: {e}")
|
||||
return 0
|
||||
|
||||
# ============================================
|
||||
# Token Management Methods
|
||||
|
||||
@@ -408,11 +408,11 @@ class FlacfetchClient:
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Trigger disk cleanup.
|
||||
|
||||
|
||||
Args:
|
||||
strategy: Cleanup strategy (oldest, largest, lowest_ratio)
|
||||
target_free_gb: Target free space to achieve
|
||||
|
||||
|
||||
Returns:
|
||||
Cleanup response with removed count and freed space
|
||||
"""
|
||||
@@ -429,12 +429,112 @@ class FlacfetchClient:
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Cleanup failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Cleanup failed: {e.response.status_code}")
|
||||
|
||||
# =========================================================================
|
||||
# Cache Management
|
||||
# =========================================================================
|
||||
|
||||
async def clear_search_cache(self, artist: str, title: str) -> bool:
|
||||
"""
|
||||
Clear a specific cached search result by artist and title.
|
||||
|
||||
Args:
|
||||
artist: Artist name
|
||||
title: Track title
|
||||
|
||||
Returns:
|
||||
True if a cache entry was deleted, False if no entry existed
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On request failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.request(
|
||||
"DELETE",
|
||||
f"{self.base_url}/cache/search",
|
||||
headers=self._headers(),
|
||||
json={"artist": artist, "title": title},
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
deleted = data.get("deleted", False)
|
||||
logger.info(
|
||||
f"Cleared flacfetch cache for '{artist}' - '{title}': "
|
||||
f"{'deleted' if deleted else 'no entry found'}"
|
||||
)
|
||||
return deleted
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Clear search cache request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Clear search cache failed: {e.response.status_code} - {e.response.text}"
|
||||
)
|
||||
|
||||
async def clear_all_cache(self) -> int:
|
||||
"""
|
||||
Clear all cached search results.
|
||||
|
||||
Returns:
|
||||
Number of cache entries deleted
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On request failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.delete(
|
||||
f"{self.base_url}/cache",
|
||||
headers=self._headers(),
|
||||
timeout=60,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
deleted_count = data.get("deleted_count", 0)
|
||||
logger.info(f"Cleared all flacfetch cache: {deleted_count} entries deleted")
|
||||
return deleted_count
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Clear all cache request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Clear all cache failed: {e.response.status_code} - {e.response.text}"
|
||||
)
|
||||
|
||||
async def get_cache_stats(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Get statistics about the cache.
|
||||
|
||||
Returns:
|
||||
Dict with count, total_size_bytes, oldest_entry, newest_entry, configured
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On request failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.get(
|
||||
f"{self.base_url}/cache/stats",
|
||||
headers=self._headers(),
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Get cache stats request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Get cache stats failed: {e.response.status_code} - {e.response.text}"
|
||||
)
|
||||
|
||||
|
||||
# Singleton client instance
|
||||
_client: Optional[FlacfetchClient] = None
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
# GCE Encoding Worker
|
||||
|
||||
HTTP API service that runs on the GCE encoding worker VM for video encoding jobs.
|
||||
|
||||
## Overview
|
||||
|
||||
This service provides HTTP endpoints for submitting and monitoring video encoding jobs. It runs as a systemd service on the `encoding-worker` GCE VM instance.
|
||||
|
||||
## Endpoints
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/encode` | POST | Submit a full encoding job |
|
||||
| `/encode-preview` | POST | Submit a preview encoding job (480x270, fast) |
|
||||
| `/status/{job_id}` | GET | Get job status and progress |
|
||||
| `/health` | GET | Health check |
|
||||
|
||||
## Authentication
|
||||
|
||||
All endpoints (except `/health`) require an API key via the `X-API-Key` header. The key is stored in Secret Manager (`encoding-worker-api-key`).
|
||||
|
||||
## Encoding Process
|
||||
|
||||
### Full Encoding (`/encode`)
|
||||
1. Downloads input files from GCS (title screen, karaoke video, end screen, instrumental audio)
|
||||
2. Uses `LocalEncodingService` from the karaoke-gen wheel to produce:
|
||||
- Lossless 4K MP4
|
||||
- Lossy 4K MP4
|
||||
- Lossless MKV
|
||||
- 720p MP4
|
||||
3. Uploads results to GCS
|
||||
4. Reports progress via status endpoint
|
||||
|
||||
### Preview Encoding (`/encode-preview`)
|
||||
1. Downloads ASS subtitle and audio files from GCS
|
||||
2. Runs FFmpeg to create a quick 480x270 preview video
|
||||
3. Uploads result to GCS
|
||||
4. Used for real-time preview in the web UI
|
||||
|
||||
## Dependencies
|
||||
|
||||
- **FFmpeg**: Static build from John Van Sickle
|
||||
- **Python 3.13**: Built from source (required for karaoke-gen)
|
||||
- **karaoke-gen wheel**: Downloaded from GCS at job start (enables hot updates)
|
||||
|
||||
## Deployment
|
||||
|
||||
This service is deployed via the VM's startup script. The script:
|
||||
1. Installs system dependencies (FFmpeg, fonts, Python build deps)
|
||||
2. Builds Python 3.13 from source
|
||||
3. Creates venv and installs dependencies
|
||||
4. Writes `main.py` to `/opt/encoding-worker/`
|
||||
5. Creates systemd service
|
||||
6. Starts the service on port 8080
|
||||
|
||||
## Local Development
|
||||
|
||||
This code is extracted from the infrastructure for maintainability. To test locally:
|
||||
|
||||
```bash
|
||||
cd backend/services/gce_encoding
|
||||
pip install -r requirements.txt
|
||||
pip install ../../../ # Install karaoke-gen wheel
|
||||
ENCODING_API_KEY=test python main.py
|
||||
```
|
||||
|
||||
## Architecture Notes
|
||||
|
||||
- **Hot code updates**: The karaoke-gen wheel is re-downloaded at the start of each job, allowing code updates without VM restart
|
||||
- **In-memory job tracking**: Job state is stored in memory. Restart clears queue.
|
||||
- **Parallel encoding**: ThreadPoolExecutor with 4 workers
|
||||
- **GCS integration**: Direct download/upload via google-cloud-storage client
|
||||
@@ -0,0 +1,22 @@
|
||||
"""
|
||||
GCE Encoding Worker Service.
|
||||
|
||||
This package contains the FastAPI application that runs on the GCE encoding worker VM.
|
||||
It provides HTTP endpoints for video encoding jobs.
|
||||
|
||||
The service:
|
||||
- Downloads input files from GCS
|
||||
- Runs FFmpeg encoding via LocalEncodingService
|
||||
- Uploads results back to GCS
|
||||
- Provides job status tracking
|
||||
|
||||
Endpoints:
|
||||
- POST /encode - Submit a full encoding job
|
||||
- POST /encode-preview - Submit a preview encoding job (fast, low-res)
|
||||
- GET /status/{job_id} - Get job status
|
||||
- GET /health - Health check
|
||||
"""
|
||||
|
||||
from .main import app
|
||||
|
||||
__all__ = ["app"]
|
||||
@@ -0,0 +1,589 @@
|
||||
import asyncio
|
||||
import glob as glob_module
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import uuid
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import FastAPI, BackgroundTasks, HTTPException, Header, Depends
|
||||
from google.cloud import storage
|
||||
from pydantic import BaseModel
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
app = FastAPI(title="Encoding Worker", version="1.0.0")
|
||||
|
||||
# API key authentication
|
||||
API_KEY = os.environ.get("ENCODING_API_KEY", "")
|
||||
|
||||
|
||||
async def verify_api_key(x_api_key: str = Header(None)):
|
||||
# Verify API key for authentication
|
||||
if not API_KEY:
|
||||
logger.warning("No API key configured - authentication disabled")
|
||||
return True
|
||||
if x_api_key != API_KEY:
|
||||
raise HTTPException(status_code=401, detail="Invalid API key")
|
||||
return True
|
||||
|
||||
# Job tracking
|
||||
jobs: dict[str, dict] = {}
|
||||
executor = ThreadPoolExecutor(max_workers=4) # 4 parallel encoding jobs
|
||||
|
||||
# GCS client
|
||||
storage_client = storage.Client()
|
||||
|
||||
class EncodeRequest(BaseModel):
|
||||
job_id: str
|
||||
input_gcs_path: str # gs://bucket/path/to/inputs/
|
||||
output_gcs_path: str # gs://bucket/path/to/outputs/
|
||||
encoding_config: dict # Video formats to generate
|
||||
|
||||
|
||||
class EncodePreviewRequest(BaseModel):
|
||||
job_id: str
|
||||
ass_gcs_path: str # gs://bucket/path/to/subtitles.ass
|
||||
audio_gcs_path: str # gs://bucket/path/to/audio.flac
|
||||
output_gcs_path: str # gs://bucket/path/to/output.mp4
|
||||
background_color: str = "black"
|
||||
background_image_gcs_path: Optional[str] = None
|
||||
font_gcs_path: Optional[str] = None # gs://bucket/path/to/custom-font.ttf
|
||||
|
||||
|
||||
class JobStatus(BaseModel):
|
||||
job_id: str
|
||||
status: str # pending, running, complete, failed
|
||||
progress: int # 0-100
|
||||
error: Optional[str] = None
|
||||
output_files: Optional[list[str]] = None
|
||||
|
||||
|
||||
def download_from_gcs(gcs_uri: str, local_path: Path):
|
||||
# Download a file or folder from GCS
|
||||
# Parse gs://bucket/path
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
prefix = parts[1] if len(parts) > 1 else ""
|
||||
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
blobs = list(bucket.list_blobs(prefix=prefix))
|
||||
|
||||
for blob in blobs:
|
||||
# Get relative path from prefix
|
||||
rel_path = blob.name[len(prefix):].lstrip("/")
|
||||
if not rel_path:
|
||||
continue
|
||||
dest = local_path / rel_path
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
blob.download_to_filename(str(dest))
|
||||
logger.info(f"Downloaded: {blob.name} -> {dest}")
|
||||
|
||||
|
||||
def upload_to_gcs(local_path: Path, gcs_uri: str):
|
||||
# Upload a file or folder to GCS
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
|
||||
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
|
||||
if local_path.is_file():
|
||||
blob_name = f"{prefix}/{local_path.name}" if prefix else local_path.name
|
||||
blob = bucket.blob(blob_name)
|
||||
blob.upload_from_filename(str(local_path))
|
||||
logger.info(f"Uploaded: {local_path} -> gs://{bucket_name}/{blob_name}")
|
||||
else:
|
||||
for file in local_path.rglob("*"):
|
||||
if file.is_file():
|
||||
rel_path = file.relative_to(local_path)
|
||||
blob_name = f"{prefix}/{rel_path}" if prefix else str(rel_path)
|
||||
blob = bucket.blob(blob_name)
|
||||
blob.upload_from_filename(str(file))
|
||||
logger.info(f"Uploaded: {file} -> gs://{bucket_name}/{blob_name}")
|
||||
|
||||
|
||||
def download_single_file_from_gcs(gcs_uri: str, local_path: Path):
|
||||
# Download a single file from GCS
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
blob_name = parts[1] if len(parts) > 1 else ""
|
||||
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
blob = bucket.blob(blob_name)
|
||||
local_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
blob.download_to_filename(str(local_path))
|
||||
logger.info(f"Downloaded: {gcs_uri} -> {local_path}")
|
||||
|
||||
|
||||
def upload_single_file_to_gcs(local_path: Path, gcs_uri: str):
|
||||
# Upload a single file to a specific GCS path
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
blob_name = parts[1] if len(parts) > 1 else ""
|
||||
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
blob = bucket.blob(blob_name)
|
||||
blob.upload_from_filename(str(local_path))
|
||||
logger.info(f"Uploaded: {local_path} -> {gcs_uri}")
|
||||
|
||||
|
||||
def run_preview_encoding(job_id: str, work_dir: Path, request: "EncodePreviewRequest"):
|
||||
# Run FFmpeg encoding for preview video (480x270, fast settings)
|
||||
jobs[job_id]["status"] = "running"
|
||||
jobs[job_id]["progress"] = 10
|
||||
|
||||
try:
|
||||
# Download input files
|
||||
ass_path = work_dir / "subtitles.ass"
|
||||
audio_path = work_dir / "audio.flac"
|
||||
|
||||
download_single_file_from_gcs(request.ass_gcs_path, ass_path)
|
||||
jobs[job_id]["progress"] = 20
|
||||
|
||||
download_single_file_from_gcs(request.audio_gcs_path, audio_path)
|
||||
jobs[job_id]["progress"] = 30
|
||||
|
||||
# Download background image if provided
|
||||
bg_image_path = None
|
||||
if request.background_image_gcs_path:
|
||||
bg_image_path = work_dir / "background.png"
|
||||
download_single_file_from_gcs(request.background_image_gcs_path, bg_image_path)
|
||||
|
||||
# Download custom font if provided and register with fontconfig
|
||||
if request.font_gcs_path:
|
||||
# Use standard fontconfig location that's already in the search path
|
||||
fonts_dir = Path("/usr/local/share/fonts/custom")
|
||||
fonts_dir.mkdir(parents=True, exist_ok=True)
|
||||
font_filename = request.font_gcs_path.split("/")[-1]
|
||||
font_path = fonts_dir / font_filename
|
||||
download_single_file_from_gcs(request.font_gcs_path, font_path)
|
||||
logger.info(f"Downloaded custom font: {font_path}")
|
||||
# Update fontconfig cache so libass can find the font
|
||||
subprocess.run(["fc-cache", "-fv"], capture_output=True)
|
||||
logger.info(f"Updated fontconfig cache with custom font: {font_filename}")
|
||||
|
||||
# Build FFmpeg command
|
||||
output_path = work_dir / "preview.mp4"
|
||||
|
||||
# Escape special characters in path for FFmpeg filter syntax
|
||||
# FFmpeg filter parsing requires escaping: \ : , [ ] ;
|
||||
def escape_ffmpeg_filter_path(path: str) -> str:
|
||||
# Note: Extra escaping needed since this is inside a triple-quoted string in Pulumi
|
||||
return path.replace("\\", "\\\\").replace(":", "\\:").replace(",", "\\,").replace("[", "\\[").replace("]", "\\]").replace(";", "\\;")
|
||||
|
||||
escaped_ass_path = escape_ffmpeg_filter_path(str(ass_path))
|
||||
|
||||
# Base command with frame rate
|
||||
cmd = ["ffmpeg", "-y", "-r", "24"]
|
||||
|
||||
# Video input: background image or solid color
|
||||
if bg_image_path and bg_image_path.exists():
|
||||
cmd.extend(["-loop", "1", "-i", str(bg_image_path)])
|
||||
# Scale and pad background to 480x270
|
||||
vf = f"scale=480:270:force_original_aspect_ratio=decrease,pad=480:270:(ow-iw)/2:(oh-ih)/2,ass={escaped_ass_path}"
|
||||
else:
|
||||
# Solid color background
|
||||
color = request.background_color or "black"
|
||||
cmd.extend(["-f", "lavfi", "-i", f"color=c={color}:s=480x270:r=24"])
|
||||
vf = f"ass={escaped_ass_path}"
|
||||
|
||||
# Audio input
|
||||
cmd.extend(["-i", str(audio_path)])
|
||||
|
||||
# Video filter and encoding settings
|
||||
cmd.extend([
|
||||
"-vf", vf,
|
||||
"-c:a", "aac", "-b:a", "96k",
|
||||
"-c:v", "libx264",
|
||||
"-preset", "superfast",
|
||||
"-crf", "28",
|
||||
"-pix_fmt", "yuv420p",
|
||||
"-movflags", "+faststart",
|
||||
"-threads", "8",
|
||||
"-shortest",
|
||||
str(output_path)
|
||||
])
|
||||
|
||||
jobs[job_id]["progress"] = 40
|
||||
logger.info(f"Running preview encoding: {' '.join(cmd)}")
|
||||
|
||||
result = subprocess.run(cmd, capture_output=True, text=True)
|
||||
|
||||
if result.returncode != 0:
|
||||
logger.error(f"FFmpeg failed: {result.stderr}")
|
||||
raise RuntimeError(f"FFmpeg preview encoding failed: {result.stderr[-500:]}")
|
||||
|
||||
jobs[job_id]["progress"] = 80
|
||||
logger.info(f"Preview encoded: {output_path}")
|
||||
|
||||
# Upload output to GCS
|
||||
upload_single_file_to_gcs(output_path, request.output_gcs_path)
|
||||
jobs[job_id]["progress"] = 95
|
||||
|
||||
jobs[job_id]["output_files"] = [request.output_gcs_path]
|
||||
return output_path
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Preview encoding failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
raise
|
||||
|
||||
|
||||
def ensure_latest_wheel():
|
||||
'''Download and install latest karaoke-gen wheel from GCS.
|
||||
|
||||
Called at the start of each job to enable hot code updates without restart.
|
||||
In-progress jobs continue with their version, new jobs get latest code.
|
||||
'''
|
||||
try:
|
||||
logger.info("Checking for latest karaoke-gen wheel in GCS...")
|
||||
|
||||
# Download latest wheel
|
||||
result = subprocess.run(
|
||||
["gsutil", "cp", "gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-*.whl", "/tmp/"],
|
||||
capture_output=True, text=True, timeout=60
|
||||
)
|
||||
|
||||
# Find the downloaded wheel (get the latest by version sorting)
|
||||
wheels = glob_module.glob("/tmp/karaoke_gen-*.whl")
|
||||
if not wheels:
|
||||
logger.warning("No wheel found in GCS, using fallback encoding logic")
|
||||
return False
|
||||
|
||||
# Sort to get latest version
|
||||
wheel_path = sorted(wheels)[-1]
|
||||
logger.info(f"Installing wheel: {wheel_path}")
|
||||
|
||||
# Install (or upgrade) the wheel
|
||||
# Use 5-minute timeout - first install at job start may need to resolve dependencies
|
||||
# Subsequent installs are faster since dependencies are cached
|
||||
install_result = subprocess.run(
|
||||
[sys.executable, "-m", "pip", "install", "--upgrade", "--quiet", wheel_path],
|
||||
capture_output=True, text=True, timeout=300
|
||||
)
|
||||
|
||||
if install_result.returncode != 0:
|
||||
logger.warning(f"Wheel installation failed: {install_result.stderr}")
|
||||
return False
|
||||
|
||||
logger.info(f"Successfully installed wheel: {wheel_path}")
|
||||
return True
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning("Wheel download/install timed out, using fallback")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to ensure latest wheel: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def find_file(work_dir: Path, *patterns):
|
||||
'''Find a file matching any of the given glob patterns.'''
|
||||
for pattern in patterns:
|
||||
matches = list(work_dir.glob(f"**/{pattern}"))
|
||||
if matches:
|
||||
return matches[0]
|
||||
return None
|
||||
|
||||
|
||||
def run_encoding(job_id: str, work_dir: Path, config: dict):
|
||||
'''Run encoding using LocalEncodingService (single source of truth).
|
||||
|
||||
Uses LocalEncodingService from the installed karaoke-gen wheel to ensure
|
||||
output files match local CLI exactly:
|
||||
- Proper names like "Artist - Title (Final Karaoke Lossless 4k).mp4"
|
||||
- Concatenated title + karaoke + end screens
|
||||
- All formats: lossless 4K MP4, lossy 4K MP4, lossless MKV, 720p MP4
|
||||
|
||||
Requires the karaoke-gen wheel to be installed (done by ensure_latest_wheel).
|
||||
'''
|
||||
jobs[job_id]["status"] = "running"
|
||||
jobs[job_id]["progress"] = 10
|
||||
|
||||
try:
|
||||
# Import LocalEncodingService from installed wheel (required, no fallback)
|
||||
from backend.services.local_encoding_service import LocalEncodingService, EncodingConfig
|
||||
logger.info("Using LocalEncodingService from installed wheel")
|
||||
|
||||
# Get artist/title from config for proper naming
|
||||
artist = config.get("artist", "Unknown Artist")
|
||||
title = config.get("title", "Unknown Title")
|
||||
base_name = f"{artist} - {title}"
|
||||
logger.info(f"Encoding for: {base_name}")
|
||||
|
||||
# Find input files in work_dir
|
||||
# Title/end screens are in screens/ subdirectory
|
||||
title_video = find_file(work_dir, "screens/title.mov", "*Title*.mov", "*title*.mov")
|
||||
end_video = find_file(work_dir, "screens/end.mov", "*End*.mov", "*end*.mov")
|
||||
|
||||
# Karaoke video - search for With Vocals or main karaoke video
|
||||
karaoke_video = find_file(
|
||||
work_dir,
|
||||
"*With Vocals*.mov", "*With Vocals*.mkv",
|
||||
"*Vocals*.mov", "*Vocals*.mkv",
|
||||
"*.mkv", "*.mov"
|
||||
)
|
||||
# Exclude title/end/output videos
|
||||
if karaoke_video:
|
||||
name_lower = karaoke_video.name.lower()
|
||||
if "title" in name_lower or "end" in name_lower or "outputs" in str(karaoke_video):
|
||||
# Search more specifically for karaoke video
|
||||
karaoke_video = find_file(work_dir, "*Karaoke*.mkv", "*Karaoke*.mov", "*vocals*.mkv")
|
||||
|
||||
# Instrumental audio
|
||||
instrumental = find_file(
|
||||
work_dir,
|
||||
"*instrumental_clean*.flac", "*Instrumental Clean*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
|
||||
logger.info(f"Found files:")
|
||||
logger.info(f" Title video: {title_video}")
|
||||
logger.info(f" Karaoke video: {karaoke_video}")
|
||||
logger.info(f" End video: {end_video}")
|
||||
logger.info(f" Instrumental: {instrumental}")
|
||||
|
||||
# Validate required files
|
||||
if not title_video:
|
||||
raise ValueError(f"No title video found in {work_dir}. Check screens/ subdirectory.")
|
||||
if not karaoke_video:
|
||||
raise ValueError(f"No karaoke video found in {work_dir}")
|
||||
if not instrumental:
|
||||
raise ValueError(f"No instrumental audio found in {work_dir}")
|
||||
|
||||
output_dir = work_dir / "outputs"
|
||||
output_dir.mkdir(exist_ok=True)
|
||||
|
||||
jobs[job_id]["progress"] = 20
|
||||
|
||||
# Build encoding config with proper file names
|
||||
encoding_config = EncodingConfig(
|
||||
title_video=str(title_video),
|
||||
karaoke_video=str(karaoke_video),
|
||||
instrumental_audio=str(instrumental),
|
||||
end_video=str(end_video) if end_video else None,
|
||||
output_karaoke_mp4=str(output_dir / f"{base_name} (Karaoke).mp4"),
|
||||
output_with_vocals_mp4=str(output_dir / f"{base_name} (With Vocals).mp4"),
|
||||
output_lossless_4k_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossless 4k).mp4"),
|
||||
output_lossy_4k_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossy 4k).mp4"),
|
||||
output_lossless_mkv=str(output_dir / f"{base_name} (Final Karaoke Lossless 4k).mkv"),
|
||||
output_720p_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossy 720p).mp4"),
|
||||
)
|
||||
|
||||
# Create service and run encoding
|
||||
service = LocalEncodingService(logger=logger)
|
||||
|
||||
jobs[job_id]["progress"] = 30
|
||||
logger.info("Starting LocalEncodingService.encode_all_formats()")
|
||||
|
||||
result = service.encode_all_formats(encoding_config)
|
||||
|
||||
if not result.success:
|
||||
raise RuntimeError(f"Encoding failed: {result.error}")
|
||||
|
||||
jobs[job_id]["progress"] = 90
|
||||
|
||||
# Collect output files
|
||||
output_files = [str(f) for f in output_dir.glob("*") if f.is_file()]
|
||||
jobs[job_id]["output_files"] = output_files
|
||||
|
||||
logger.info(f"Encoding complete. Output files: {output_files}")
|
||||
return output_dir
|
||||
|
||||
except ImportError as e:
|
||||
# No fallback - wheel must be installed
|
||||
error_msg = (
|
||||
f"LocalEncodingService not available: {e}. "
|
||||
"The karaoke-gen wheel must be installed. "
|
||||
"Check that ensure_latest_wheel() succeeded and wheel exists in GCS."
|
||||
)
|
||||
logger.error(error_msg)
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = error_msg
|
||||
raise RuntimeError(error_msg) from e
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Encoding failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
raise
|
||||
|
||||
|
||||
async def process_job(job_id: str, request: EncodeRequest):
|
||||
# Process an encoding job asynchronously
|
||||
try:
|
||||
# Download and install latest wheel at job start (allows hot updates without restart)
|
||||
# This means in-progress jobs continue with their version, new jobs get latest code
|
||||
ensure_latest_wheel()
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
work_dir = Path(temp_dir) / "work"
|
||||
work_dir.mkdir()
|
||||
|
||||
# Download input files
|
||||
jobs[job_id]["progress"] = 5
|
||||
logger.info(f"Downloading from {request.input_gcs_path}")
|
||||
download_from_gcs(request.input_gcs_path, work_dir)
|
||||
|
||||
# Run encoding in thread pool (CPU-bound)
|
||||
loop = asyncio.get_event_loop()
|
||||
output_dir = await loop.run_in_executor(
|
||||
executor,
|
||||
run_encoding,
|
||||
job_id,
|
||||
work_dir,
|
||||
request.encoding_config
|
||||
)
|
||||
|
||||
# Upload outputs
|
||||
jobs[job_id]["progress"] = 95
|
||||
logger.info(f"Uploading to {request.output_gcs_path}")
|
||||
upload_to_gcs(output_dir, request.output_gcs_path)
|
||||
|
||||
# Convert local paths to blob paths (backend expects blob paths, not full gs:// URIs)
|
||||
# output_gcs_path is like "gs://bucket/jobs/id/encoded/"
|
||||
# We need paths like "jobs/id/encoded/Artist - Title (Final Karaoke Lossless 4k).mp4"
|
||||
gcs_path = request.output_gcs_path.replace("gs://", "")
|
||||
parts = gcs_path.split("/", 1)
|
||||
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
|
||||
local_output_files = jobs[job_id].get("output_files", [])
|
||||
blob_paths = []
|
||||
for local_path in local_output_files:
|
||||
filename = Path(local_path).name
|
||||
blob_path = f"{prefix}/{filename}" if prefix else filename
|
||||
blob_paths.append(blob_path)
|
||||
jobs[job_id]["output_files"] = blob_paths
|
||||
logger.info(f"Output files (blob paths): {blob_paths}")
|
||||
|
||||
jobs[job_id]["status"] = "complete"
|
||||
jobs[job_id]["progress"] = 100
|
||||
logger.info(f"Job {job_id} complete")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id} failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
|
||||
|
||||
async def process_preview_job(job_id: str, request: EncodePreviewRequest):
|
||||
# Process a preview encoding job asynchronously
|
||||
try:
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
work_dir = Path(temp_dir) / "work"
|
||||
work_dir.mkdir()
|
||||
|
||||
# Run preview encoding in thread pool (CPU-bound)
|
||||
# Note: run_preview_encoding handles download/upload internally
|
||||
loop = asyncio.get_event_loop()
|
||||
await loop.run_in_executor(
|
||||
executor,
|
||||
run_preview_encoding,
|
||||
job_id,
|
||||
work_dir,
|
||||
request
|
||||
)
|
||||
|
||||
jobs[job_id]["status"] = "complete"
|
||||
jobs[job_id]["progress"] = 100
|
||||
logger.info(f"Preview job {job_id} complete")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Preview job {job_id} failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
|
||||
|
||||
@app.post("/encode-preview")
|
||||
async def submit_preview_encode_job(request: EncodePreviewRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
|
||||
# Submit a preview encoding job
|
||||
job_id = request.job_id
|
||||
|
||||
# If job already exists, return cached result or current status
|
||||
if job_id in jobs:
|
||||
existing_job = jobs[job_id]
|
||||
if existing_job["status"] == "complete":
|
||||
# Return cached result - preview already encoded
|
||||
return {"status": "cached", "job_id": job_id, "output_path": existing_job.get("output_path")}
|
||||
elif existing_job["status"] == "failed":
|
||||
# Previous attempt failed, allow retry by replacing the job
|
||||
pass
|
||||
else:
|
||||
# Job is still in progress
|
||||
return {"status": "in_progress", "job_id": job_id}
|
||||
|
||||
jobs[job_id] = {
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"progress": 0,
|
||||
"error": None,
|
||||
"output_files": None,
|
||||
}
|
||||
|
||||
background_tasks.add_task(process_preview_job, job_id, request)
|
||||
|
||||
return {"status": "accepted", "job_id": job_id}
|
||||
|
||||
|
||||
@app.post("/encode")
|
||||
async def submit_encode_job(request: EncodeRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
|
||||
# Submit an encoding job
|
||||
job_id = request.job_id
|
||||
|
||||
if job_id in jobs:
|
||||
raise HTTPException(status_code=409, detail=f"Job {job_id} already exists")
|
||||
|
||||
jobs[job_id] = {
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"progress": 0,
|
||||
"error": None,
|
||||
"output_files": None,
|
||||
}
|
||||
|
||||
background_tasks.add_task(process_job, job_id, request)
|
||||
|
||||
return {"status": "accepted", "job_id": job_id}
|
||||
|
||||
|
||||
@app.get("/status/{job_id}")
|
||||
async def get_job_status(job_id: str, _auth: bool = Depends(verify_api_key)) -> JobStatus:
|
||||
# Get the status of an encoding job
|
||||
if job_id not in jobs:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
return JobStatus(**jobs[job_id])
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
async def health_check():
|
||||
# Health check endpoint
|
||||
active_jobs = sum(1 for j in jobs.values() if j["status"] == "running")
|
||||
|
||||
# Get karaoke-gen wheel version if installed
|
||||
wheel_version = None
|
||||
try:
|
||||
from importlib.metadata import version as get_version
|
||||
wheel_version = get_version("karaoke-gen")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return {
|
||||
"status": "ok",
|
||||
"active_jobs": active_jobs,
|
||||
"queue_length": sum(1 for j in jobs.values() if j["status"] == "pending"),
|
||||
"ffmpeg_version": subprocess.run(["ffmpeg", "-version"], capture_output=True, text=True).stdout.split("\n")[0],
|
||||
"wheel_version": wheel_version,
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=8080)
|
||||
@@ -0,0 +1,16 @@
|
||||
# GCE Encoding Worker dependencies
|
||||
# These are installed in the VM's Python virtual environment
|
||||
|
||||
# Web framework
|
||||
fastapi>=0.109.0
|
||||
uvicorn>=0.27.0
|
||||
|
||||
# GCS client
|
||||
google-cloud-storage>=2.14.0
|
||||
|
||||
# Async utilities (optional, for future improvements)
|
||||
aiofiles>=23.2.0
|
||||
aiohttp>=3.9.0
|
||||
|
||||
# Note: The karaoke-gen wheel is installed separately from GCS.
|
||||
# It provides LocalEncodingService which does the actual encoding.
|
||||
@@ -15,6 +15,7 @@ import os
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from backend.config import get_settings
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -136,8 +137,10 @@ class GoogleDriveService:
|
||||
logger.info(f"Looking for folder '{folder_name}' in parent {parent_id}")
|
||||
|
||||
# Search for existing folder
|
||||
# Escape single quotes in folder name for Google Drive API query syntax
|
||||
escaped_folder_name = folder_name.replace("'", "\\'")
|
||||
query = (
|
||||
f"name='{folder_name}' and '{parent_id}' in parents "
|
||||
f"name='{escaped_folder_name}' and '{parent_id}' in parents "
|
||||
f"and mimeType='application/vnd.google-apps.folder' and trashed=false"
|
||||
)
|
||||
results = self.service.files().list(q=query, fields="files(id, name)").execute()
|
||||
@@ -203,8 +206,10 @@ class GoogleDriveService:
|
||||
|
||||
# Check for existing file with same name
|
||||
if replace_existing:
|
||||
# Escape single quotes in filename for Google Drive API query syntax
|
||||
escaped_filename = filename.replace("'", "\\'")
|
||||
query = (
|
||||
f"name='{filename}' and '{parent_id}' in parents and trashed=false"
|
||||
f"name='{escaped_filename}' and '{parent_id}' in parents and trashed=false"
|
||||
)
|
||||
results = self.service.files().list(q=query, fields="files(id)").execute()
|
||||
for existing_file in results.get("files", []):
|
||||
@@ -249,7 +254,10 @@ class GoogleDriveService:
|
||||
Returns:
|
||||
Dictionary mapping category to uploaded file ID
|
||||
"""
|
||||
filename_base = f"{brand_code} - {base_name}"
|
||||
# Sanitize base_name to handle Unicode characters (curly quotes, em dashes, etc.)
|
||||
# that could cause issues with Google Drive API queries and file naming
|
||||
safe_base_name = sanitize_filename(base_name) if base_name else base_name
|
||||
filename_base = f"{brand_code} - {safe_base_name}"
|
||||
uploaded_files = {}
|
||||
|
||||
logger.info(
|
||||
@@ -296,6 +304,45 @@ class GoogleDriveService:
|
||||
logger.info(f"Public share upload complete: {len(uploaded_files)} files uploaded")
|
||||
return uploaded_files
|
||||
|
||||
def delete_file(self, file_id: str) -> bool:
|
||||
"""
|
||||
Delete a file from Google Drive.
|
||||
|
||||
Args:
|
||||
file_id: Google Drive file ID to delete
|
||||
|
||||
Returns:
|
||||
True if deleted successfully, False otherwise
|
||||
"""
|
||||
logger.info(f"Deleting Google Drive file: {file_id}")
|
||||
|
||||
try:
|
||||
self.service.files().delete(fileId=file_id).execute()
|
||||
logger.info(f"Successfully deleted file: {file_id}")
|
||||
return True
|
||||
except Exception as e:
|
||||
# Check if it's a 404 (already deleted)
|
||||
if hasattr(e, 'resp') and e.resp.status == 404:
|
||||
logger.warning(f"File not found (already deleted?): {file_id}")
|
||||
return True
|
||||
logger.error(f"Failed to delete Google Drive file: {e}")
|
||||
return False
|
||||
|
||||
def delete_files(self, file_ids: list[str]) -> dict[str, bool]:
|
||||
"""
|
||||
Delete multiple files from Google Drive.
|
||||
|
||||
Args:
|
||||
file_ids: List of Google Drive file IDs to delete
|
||||
|
||||
Returns:
|
||||
Dictionary mapping file_id to success status
|
||||
"""
|
||||
results = {}
|
||||
for file_id in file_ids:
|
||||
results[file_id] = self.delete_file(file_id)
|
||||
return results
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_gdrive_service: Optional[GoogleDriveService] = None
|
||||
|
||||
+246
-32
@@ -13,7 +13,9 @@ import uuid
|
||||
from datetime import datetime
|
||||
from typing import Optional, Dict, Any, List
|
||||
|
||||
from backend.config import settings
|
||||
from backend.models.job import Job, JobStatus, JobCreate, STATE_TRANSITIONS
|
||||
from backend.models.worker_log import WorkerLogEntry
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
@@ -32,10 +34,21 @@ class JobManager:
|
||||
def create_job(self, job_create: JobCreate) -> Job:
|
||||
"""
|
||||
Create a new job with initial state PENDING.
|
||||
|
||||
|
||||
Jobs start in PENDING state and transition to DOWNLOADING
|
||||
when a worker picks them up.
|
||||
|
||||
Raises:
|
||||
ValueError: If theme_id is not provided (all jobs require a theme)
|
||||
"""
|
||||
# Enforce theme requirement - all jobs must have a theme
|
||||
# This prevents unstyled videos from ever being generated
|
||||
if not job_create.theme_id:
|
||||
raise ValueError(
|
||||
"theme_id is required for all jobs. "
|
||||
"Use get_theme_service().get_default_theme_id() to get the default theme."
|
||||
)
|
||||
|
||||
job_id = str(uuid.uuid4())[:8]
|
||||
|
||||
now = datetime.utcnow()
|
||||
@@ -111,19 +124,21 @@ class JobManager:
|
||||
client_id: Optional[str] = None,
|
||||
created_after: Optional[datetime] = None,
|
||||
created_before: Optional[datetime] = None,
|
||||
user_email: Optional[str] = None,
|
||||
limit: int = 100
|
||||
) -> List[Job]:
|
||||
"""
|
||||
List jobs with optional filtering.
|
||||
|
||||
|
||||
Args:
|
||||
status: Filter by job status
|
||||
environment: Filter by request_metadata.environment (test/production/development)
|
||||
client_id: Filter by request_metadata.client_id (customer identifier)
|
||||
created_after: Filter jobs created after this datetime
|
||||
created_before: Filter jobs created before this datetime
|
||||
user_email: Filter by user_email (job owner)
|
||||
limit: Maximum number of jobs to return
|
||||
|
||||
|
||||
Returns:
|
||||
List of Job objects matching filters
|
||||
"""
|
||||
@@ -133,6 +148,7 @@ class JobManager:
|
||||
client_id=client_id,
|
||||
created_after=created_after,
|
||||
created_before=created_before,
|
||||
user_email=user_email,
|
||||
limit=limit
|
||||
)
|
||||
|
||||
@@ -221,7 +237,7 @@ class JobManager:
|
||||
return urls
|
||||
|
||||
def delete_job(self, job_id: str, delete_files: bool = True) -> None:
|
||||
"""Delete a job and optionally its files."""
|
||||
"""Delete a job, its files, and its logs subcollection."""
|
||||
if delete_files:
|
||||
job = self.get_job(job_id)
|
||||
if job and job.output_files:
|
||||
@@ -230,7 +246,15 @@ class JobManager:
|
||||
self.storage.delete_file(gcs_path)
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting file {gcs_path}: {e}")
|
||||
|
||||
|
||||
# Delete logs subcollection first (must be done before deleting parent doc)
|
||||
try:
|
||||
deleted_logs = self.firestore.delete_logs_subcollection(job_id)
|
||||
if deleted_logs > 0:
|
||||
logger.info(f"Deleted {deleted_logs} log entries for job {job_id}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Error deleting logs subcollection for job {job_id}: {e}")
|
||||
|
||||
self.firestore.delete_job(job_id)
|
||||
logger.info(f"Deleted job {job_id}")
|
||||
|
||||
@@ -346,9 +370,147 @@ class JobManager:
|
||||
'instrumental_token': updates['instrumental_token'],
|
||||
'instrumental_token_expires_at': updates['instrumental_token_expires_at']
|
||||
})
|
||||
|
||||
|
||||
logger.info(f"Job {job_id} transitioned to {new_status}")
|
||||
|
||||
# Trigger notifications asynchronously (fire-and-forget)
|
||||
self._trigger_state_notifications(job_id, new_status)
|
||||
|
||||
return True
|
||||
|
||||
def _trigger_state_notifications(self, job_id: str, new_status: JobStatus) -> None:
|
||||
"""
|
||||
Trigger email notifications based on state transitions.
|
||||
|
||||
This is fire-and-forget - notification failures don't affect job processing.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
new_status: New job status
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
try:
|
||||
# Get the job to access user info
|
||||
job = self.get_job(job_id)
|
||||
if not job or not job.user_email:
|
||||
logger.debug(f"No user email for job {job_id}, skipping notifications")
|
||||
return
|
||||
|
||||
# Job completion notification
|
||||
if new_status == JobStatus.COMPLETE:
|
||||
self._schedule_completion_email(job)
|
||||
|
||||
# Idle reminder scheduling for blocking states
|
||||
elif new_status in [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION]:
|
||||
self._schedule_idle_reminder(job, new_status)
|
||||
|
||||
except Exception as e:
|
||||
# Never let notification failures affect job processing
|
||||
logger.error(f"Error triggering notifications for job {job_id}: {e}")
|
||||
|
||||
def _schedule_completion_email(self, job: Job) -> None:
|
||||
"""
|
||||
Schedule sending a job completion email.
|
||||
|
||||
Uses asyncio to fire-and-forget the email sending.
|
||||
"""
|
||||
import asyncio
|
||||
import threading
|
||||
|
||||
try:
|
||||
from backend.services.job_notification_service import get_job_notification_service
|
||||
|
||||
notification_service = get_job_notification_service()
|
||||
|
||||
# Get youtube, dropbox URLs, and brand_code from state_data (may be None)
|
||||
state_data = job.state_data or {}
|
||||
youtube_url = state_data.get('youtube_url')
|
||||
dropbox_url = state_data.get('dropbox_link')
|
||||
brand_code = state_data.get('brand_code')
|
||||
|
||||
# Create async task (fire-and-forget)
|
||||
async def send_email():
|
||||
await notification_service.send_job_completion_email(
|
||||
job_id=job.job_id,
|
||||
user_email=job.user_email,
|
||||
user_name=None, # Could fetch from user service if needed
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
youtube_url=youtube_url,
|
||||
dropbox_url=dropbox_url,
|
||||
brand_code=brand_code,
|
||||
)
|
||||
|
||||
# Try to get existing event loop, create new one if none exists
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
# If we're in an async context, create a task
|
||||
loop.create_task(send_email())
|
||||
except RuntimeError:
|
||||
# No event loop - we're likely in a sync context
|
||||
# Use daemon thread to avoid blocking job completion
|
||||
def run_in_thread():
|
||||
asyncio.run(send_email())
|
||||
thread = threading.Thread(target=run_in_thread, daemon=True)
|
||||
thread.start()
|
||||
|
||||
logger.info(f"Scheduled completion email for job {job.job_id}")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to schedule completion email for job {job.job_id}: {e}")
|
||||
|
||||
def _schedule_idle_reminder(self, job: Job, new_status: JobStatus) -> None:
|
||||
"""
|
||||
Schedule an idle reminder for a blocking state.
|
||||
|
||||
Records the timestamp when the blocking state was entered and
|
||||
schedules a Cloud Tasks task for 5 minutes later.
|
||||
"""
|
||||
import asyncio
|
||||
import threading
|
||||
|
||||
try:
|
||||
# Record when we entered blocking state (for idle detection)
|
||||
blocking_entered_at = datetime.utcnow().isoformat()
|
||||
|
||||
action_type = "lyrics" if new_status == JobStatus.AWAITING_REVIEW else "instrumental"
|
||||
|
||||
# Update state_data with blocking state info (handle None state_data)
|
||||
existing_state_data = job.state_data or {}
|
||||
state_data_update = {
|
||||
'blocking_state_entered_at': blocking_entered_at,
|
||||
'blocking_action_type': action_type,
|
||||
'reminder_sent': False, # Will be set to True after reminder is sent
|
||||
}
|
||||
|
||||
self.firestore.update_job(job.job_id, {
|
||||
'state_data': {**existing_state_data, **state_data_update}
|
||||
})
|
||||
|
||||
# Schedule the idle reminder check via worker service (5 min delay)
|
||||
from backend.services.worker_service import get_worker_service
|
||||
|
||||
async def schedule_reminder():
|
||||
worker_service = get_worker_service()
|
||||
await worker_service.schedule_idle_reminder(job.job_id)
|
||||
|
||||
# Try to get existing event loop, create new one if none exists
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
loop.create_task(schedule_reminder())
|
||||
except RuntimeError:
|
||||
# No event loop - we're in a sync context
|
||||
# Use daemon thread to avoid blocking job processing
|
||||
def run_in_thread():
|
||||
asyncio.run(schedule_reminder())
|
||||
thread = threading.Thread(target=run_in_thread, daemon=True)
|
||||
thread.start()
|
||||
|
||||
logger.info(f"Scheduled idle reminder for job {job.job_id} ({action_type})")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to schedule idle reminder for job {job.job_id}: {e}")
|
||||
|
||||
def update_state_data(self, job_id: str, key: str, value: Any) -> None:
|
||||
"""
|
||||
@@ -576,33 +738,45 @@ class JobManager:
|
||||
worker: str,
|
||||
level: str,
|
||||
message: str,
|
||||
max_logs: int = 500
|
||||
max_logs: int = 500,
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
) -> None:
|
||||
"""
|
||||
Append a log entry to the job's worker_logs atomically.
|
||||
|
||||
Uses Firestore ArrayUnion for atomic append to avoid race conditions
|
||||
when multiple workers (audio + lyrics) are logging concurrently.
|
||||
|
||||
Note: max_logs is not enforced per-append (would require transactions).
|
||||
Logs may grow beyond max_logs; trim periodically if needed.
|
||||
|
||||
Append a log entry to the job's logs.
|
||||
|
||||
By default (USE_LOG_SUBCOLLECTION=true), logs are stored in a Firestore
|
||||
subcollection (jobs/{job_id}/logs) to avoid the 1MB document size limit.
|
||||
|
||||
For backward compatibility (USE_LOG_SUBCOLLECTION=false), logs are stored
|
||||
in the embedded worker_logs array using atomic ArrayUnion.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
worker: Worker name (audio, lyrics, screens, video, render)
|
||||
worker: Worker name (audio, lyrics, screens, video, render, distribution)
|
||||
level: Log level (DEBUG, INFO, WARNING, ERROR)
|
||||
message: Log message
|
||||
max_logs: Not used (kept for API compatibility)
|
||||
metadata: Optional additional metadata dict
|
||||
"""
|
||||
log_entry = {
|
||||
'timestamp': datetime.utcnow().isoformat() + 'Z',
|
||||
'level': level,
|
||||
'worker': worker,
|
||||
'message': message[:1000] # Truncate long messages
|
||||
}
|
||||
|
||||
# Use atomic ArrayUnion to avoid race conditions
|
||||
self.firestore.append_worker_log(job_id, log_entry)
|
||||
if settings.use_log_subcollection:
|
||||
# New subcollection approach - avoids 1MB limit
|
||||
log_entry = WorkerLogEntry.create(
|
||||
job_id=job_id,
|
||||
worker=worker,
|
||||
level=level,
|
||||
message=message,
|
||||
metadata=metadata
|
||||
)
|
||||
self.firestore.append_log_to_subcollection(job_id, log_entry)
|
||||
else:
|
||||
# Legacy embedded array approach
|
||||
log_entry = {
|
||||
'timestamp': datetime.utcnow().isoformat() + 'Z',
|
||||
'level': level,
|
||||
'worker': worker,
|
||||
'message': message[:1000] # Truncate long messages
|
||||
}
|
||||
self.firestore.append_worker_log(job_id, log_entry)
|
||||
|
||||
def get_worker_logs(
|
||||
self,
|
||||
@@ -612,28 +786,68 @@ class JobManager:
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Get worker logs for a job, optionally filtered by worker and index.
|
||||
|
||||
|
||||
By default (USE_LOG_SUBCOLLECTION=true), logs are read from the
|
||||
subcollection. Falls back to embedded array for older jobs.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
since_index: Return only logs after this index (for pagination)
|
||||
worker: Filter by worker name (optional)
|
||||
|
||||
|
||||
Returns:
|
||||
List of log entries as dicts
|
||||
List of log entries as dicts (in legacy format for API compatibility)
|
||||
"""
|
||||
if settings.use_log_subcollection:
|
||||
# Try subcollection first
|
||||
subcollection_logs = self.firestore.get_logs_from_subcollection(
|
||||
job_id=job_id,
|
||||
offset=since_index,
|
||||
worker=worker,
|
||||
limit=500
|
||||
)
|
||||
if subcollection_logs:
|
||||
# Convert to legacy format for API compatibility
|
||||
return [log.to_legacy_dict() for log in subcollection_logs]
|
||||
# Fall through to check embedded array for older jobs
|
||||
|
||||
# Embedded array approach (legacy jobs or fallback)
|
||||
job = self.get_job(job_id)
|
||||
if not job or not job.worker_logs:
|
||||
return []
|
||||
|
||||
|
||||
logs = [log.dict() if hasattr(log, 'dict') else log for log in job.worker_logs]
|
||||
|
||||
|
||||
# Filter by index
|
||||
if since_index > 0:
|
||||
logs = logs[since_index:]
|
||||
|
||||
|
||||
# Filter by worker
|
||||
if worker:
|
||||
logs = [log for log in logs if log.get('worker') == worker]
|
||||
|
||||
|
||||
return logs
|
||||
|
||||
def get_worker_logs_count(self, job_id: str) -> int:
|
||||
"""
|
||||
Get the total count of worker logs for a job.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
|
||||
Returns:
|
||||
Total count of logs
|
||||
"""
|
||||
if settings.use_log_subcollection:
|
||||
# Try subcollection first
|
||||
count = self.firestore.get_logs_count_from_subcollection(job_id)
|
||||
if count > 0:
|
||||
return count
|
||||
# Fall through to check embedded array
|
||||
|
||||
# Embedded array (legacy jobs or fallback)
|
||||
job = self.get_job(job_id)
|
||||
if not job or not job.worker_logs:
|
||||
return 0
|
||||
return len(job.worker_logs)
|
||||
|
||||
@@ -0,0 +1,271 @@
|
||||
"""
|
||||
Job notification service for sending emails on job state changes.
|
||||
|
||||
Handles:
|
||||
- Job completion emails (when job enters COMPLETE status)
|
||||
- Action reminder emails (when user is idle at blocking states)
|
||||
|
||||
This service orchestrates between the template service and email service,
|
||||
building the complete notification flow for jobs.
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
from backend.services.email_service import get_email_service
|
||||
from backend.services.template_service import get_template_service
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Environment variable to enable/disable auto emails
|
||||
ENABLE_AUTO_EMAILS = os.getenv("ENABLE_AUTO_EMAILS", "true").lower() == "true"
|
||||
|
||||
# Feedback form URL (configured per environment, empty by default to avoid placeholder in emails)
|
||||
FEEDBACK_FORM_URL = os.getenv("FEEDBACK_FORM_URL", "")
|
||||
|
||||
|
||||
def _mask_email(email: str) -> str:
|
||||
"""Mask email for logging to protect PII. Shows first char + domain."""
|
||||
if not email or "@" not in email:
|
||||
return "***"
|
||||
local, domain = email.split("@", 1)
|
||||
if len(local) <= 1:
|
||||
return f"*@{domain}"
|
||||
return f"{local[0]}***@{domain}"
|
||||
|
||||
|
||||
class JobNotificationService:
|
||||
"""
|
||||
Service for sending job-related email notifications.
|
||||
|
||||
Coordinates between template service (for message rendering) and
|
||||
email service (for sending).
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize notification service."""
|
||||
self.email_service = get_email_service()
|
||||
self.template_service = get_template_service()
|
||||
self.frontend_url = os.getenv("FRONTEND_URL", "https://gen.nomadkaraoke.com")
|
||||
self.backend_url = os.getenv("BACKEND_URL", "https://api.nomadkaraoke.com")
|
||||
|
||||
def _build_review_url(self, job_id: str, audio_hash: Optional[str] = None, review_token: Optional[str] = None) -> str:
|
||||
"""Build the lyrics review URL for a job."""
|
||||
import urllib.parse
|
||||
|
||||
review_ui_url = os.getenv("REVIEW_UI_URL", f"{self.frontend_url}/lyrics/")
|
||||
base_api_url = f"{self.backend_url}/api/review/{job_id}"
|
||||
encoded_api_url = urllib.parse.quote(base_api_url, safe='')
|
||||
|
||||
url = f"{review_ui_url}?baseApiUrl={encoded_api_url}"
|
||||
if audio_hash:
|
||||
url += f"&audioHash={urllib.parse.quote(audio_hash, safe='')}"
|
||||
if review_token:
|
||||
url += f"&reviewToken={urllib.parse.quote(review_token, safe='')}"
|
||||
return url
|
||||
|
||||
def _build_instrumental_url(self, job_id: str, instrumental_token: Optional[str] = None) -> str:
|
||||
"""Build the instrumental selection URL for a job."""
|
||||
base_api_url = f"{self.backend_url}/api/jobs/{job_id}"
|
||||
|
||||
import urllib.parse
|
||||
encoded_api_url = urllib.parse.quote(base_api_url, safe='')
|
||||
|
||||
url = f"{self.frontend_url}/instrumental/?baseApiUrl={encoded_api_url}"
|
||||
if instrumental_token:
|
||||
url += f"&instrumentalToken={urllib.parse.quote(instrumental_token, safe='')}"
|
||||
return url
|
||||
|
||||
async def send_job_completion_email(
|
||||
self,
|
||||
job_id: str,
|
||||
user_email: str,
|
||||
user_name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
youtube_url: Optional[str] = None,
|
||||
dropbox_url: Optional[str] = None,
|
||||
brand_code: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Send job completion email to user.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
user_email: User's email address
|
||||
user_name: User's display name (optional)
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
youtube_url: YouTube video URL
|
||||
dropbox_url: Dropbox folder URL
|
||||
brand_code: Release ID (e.g., "NOMAD-1178")
|
||||
|
||||
Returns:
|
||||
True if email was sent successfully
|
||||
"""
|
||||
if not ENABLE_AUTO_EMAILS:
|
||||
logger.info(f"Auto emails disabled, skipping completion email for job {job_id}")
|
||||
return False
|
||||
|
||||
if not user_email:
|
||||
logger.warning(f"No user email for job {job_id}, skipping completion email")
|
||||
return False
|
||||
|
||||
try:
|
||||
# Render the completion message using template service
|
||||
message_content = self.template_service.render_job_completion(
|
||||
name=user_name,
|
||||
youtube_url=youtube_url,
|
||||
dropbox_url=dropbox_url,
|
||||
artist=artist,
|
||||
title=title,
|
||||
job_id=job_id,
|
||||
feedback_url=FEEDBACK_FORM_URL,
|
||||
)
|
||||
|
||||
# Send the email with CC to admin
|
||||
success = self.email_service.send_job_completion(
|
||||
to_email=user_email,
|
||||
message_content=message_content,
|
||||
artist=artist,
|
||||
title=title,
|
||||
brand_code=brand_code,
|
||||
cc_admin=True,
|
||||
)
|
||||
|
||||
if success:
|
||||
logger.info(f"Sent completion email for job {job_id} to {_mask_email(user_email)}")
|
||||
else:
|
||||
logger.error(f"Failed to send completion email for job {job_id}")
|
||||
|
||||
return success
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"Error sending completion email for job {job_id}: {e}")
|
||||
return False
|
||||
|
||||
async def send_action_reminder_email(
|
||||
self,
|
||||
job_id: str,
|
||||
user_email: str,
|
||||
action_type: str,
|
||||
user_name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
audio_hash: Optional[str] = None,
|
||||
review_token: Optional[str] = None,
|
||||
instrumental_token: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Send action-needed reminder email to user.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
user_email: User's email address
|
||||
action_type: Type of action needed ("lyrics" or "instrumental")
|
||||
user_name: User's display name
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
audio_hash: Audio hash for review URL
|
||||
review_token: Review token for URL
|
||||
instrumental_token: Instrumental token for URL
|
||||
|
||||
Returns:
|
||||
True if email was sent successfully
|
||||
"""
|
||||
if not ENABLE_AUTO_EMAILS:
|
||||
logger.info(f"Auto emails disabled, skipping reminder for job {job_id}")
|
||||
return False
|
||||
|
||||
if not user_email:
|
||||
logger.warning(f"No user email for job {job_id}, skipping reminder")
|
||||
return False
|
||||
|
||||
try:
|
||||
# Render the appropriate template
|
||||
if action_type == "lyrics":
|
||||
review_url = self._build_review_url(job_id, audio_hash, review_token)
|
||||
message_content = self.template_service.render_action_needed_lyrics(
|
||||
name=user_name,
|
||||
artist=artist,
|
||||
title=title,
|
||||
review_url=review_url,
|
||||
)
|
||||
elif action_type == "instrumental":
|
||||
instrumental_url = self._build_instrumental_url(job_id, instrumental_token)
|
||||
message_content = self.template_service.render_action_needed_instrumental(
|
||||
name=user_name,
|
||||
artist=artist,
|
||||
title=title,
|
||||
instrumental_url=instrumental_url,
|
||||
)
|
||||
else:
|
||||
logger.error(f"Unknown action type: {action_type}")
|
||||
return False
|
||||
|
||||
# Send the reminder email (no CC)
|
||||
success = self.email_service.send_action_reminder(
|
||||
to_email=user_email,
|
||||
message_content=message_content,
|
||||
action_type=action_type,
|
||||
artist=artist,
|
||||
title=title,
|
||||
)
|
||||
|
||||
if success:
|
||||
logger.info(f"Sent {action_type} reminder for job {job_id} to {_mask_email(user_email)}")
|
||||
else:
|
||||
logger.error(f"Failed to send {action_type} reminder for job {job_id}")
|
||||
|
||||
return success
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"Error sending {action_type} reminder for job {job_id}: {e}")
|
||||
return False
|
||||
|
||||
def get_completion_message(
|
||||
self,
|
||||
job_id: str,
|
||||
user_name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
youtube_url: Optional[str] = None,
|
||||
dropbox_url: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Get the rendered completion message for a job (for admin copy functionality).
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
user_name: User's display name
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
youtube_url: YouTube video URL
|
||||
dropbox_url: Dropbox folder URL
|
||||
|
||||
Returns:
|
||||
Rendered message content as plain text
|
||||
"""
|
||||
return self.template_service.render_job_completion(
|
||||
name=user_name,
|
||||
youtube_url=youtube_url,
|
||||
dropbox_url=dropbox_url,
|
||||
artist=artist,
|
||||
title=title,
|
||||
job_id=job_id,
|
||||
feedback_url=FEEDBACK_FORM_URL,
|
||||
)
|
||||
|
||||
|
||||
# Global instance
|
||||
_job_notification_service: Optional[JobNotificationService] = None
|
||||
|
||||
|
||||
def get_job_notification_service() -> JobNotificationService:
|
||||
"""Get the global job notification service instance."""
|
||||
global _job_notification_service
|
||||
if _job_notification_service is None:
|
||||
_job_notification_service = JobNotificationService()
|
||||
return _job_notification_service
|
||||
@@ -0,0 +1,98 @@
|
||||
"""Langfuse callback handler preloader for container startup.
|
||||
|
||||
Initializes the Langfuse callback handler at container startup to avoid
|
||||
slow initialization during request processing. The CallbackHandler()
|
||||
constructor makes blocking network calls to the Langfuse API, which can
|
||||
take 3+ minutes on Cloud Run cold starts.
|
||||
|
||||
See docs/archive/2026-01-08-performance-investigation.md for background.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from typing import Optional, Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Singleton storage for preloaded handler
|
||||
_preloaded_handler: Optional[Any] = None
|
||||
_initialization_attempted: bool = False
|
||||
|
||||
|
||||
def preload_langfuse_handler() -> Optional[Any]:
|
||||
"""Preload Langfuse callback handler at startup.
|
||||
|
||||
Only initializes if LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY
|
||||
environment variables are set.
|
||||
|
||||
Returns:
|
||||
The preloaded CallbackHandler, or None if not configured
|
||||
"""
|
||||
global _preloaded_handler, _initialization_attempted
|
||||
|
||||
if _initialization_attempted:
|
||||
logger.debug("Langfuse initialization already attempted")
|
||||
return _preloaded_handler
|
||||
|
||||
_initialization_attempted = True
|
||||
|
||||
# Check if Langfuse is configured
|
||||
public_key = os.getenv("LANGFUSE_PUBLIC_KEY")
|
||||
secret_key = os.getenv("LANGFUSE_SECRET_KEY")
|
||||
|
||||
if not (public_key and secret_key):
|
||||
logger.info("Langfuse not configured (missing keys), skipping preload")
|
||||
return None
|
||||
|
||||
logger.info("Preloading Langfuse callback handler...")
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
from langfuse.langchain import CallbackHandler
|
||||
|
||||
# Initialize the handler - this is the slow part that makes network calls
|
||||
_preloaded_handler = CallbackHandler()
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
host = os.getenv("LANGFUSE_HOST", "cloud.langfuse.com")
|
||||
logger.info(f"Langfuse handler preloaded in {elapsed:.2f}s (host: {host})")
|
||||
|
||||
return _preloaded_handler
|
||||
|
||||
except ImportError:
|
||||
logger.warning("langfuse package not installed, skipping preload")
|
||||
return None
|
||||
except Exception as e:
|
||||
elapsed = time.time() - start_time
|
||||
logger.error(f"Failed to preload Langfuse handler after {elapsed:.2f}s: {e}")
|
||||
# Don't raise - Langfuse is optional
|
||||
return None
|
||||
|
||||
|
||||
def get_preloaded_langfuse_handler() -> Optional[Any]:
|
||||
"""Get the preloaded Langfuse callback handler if available.
|
||||
|
||||
Returns:
|
||||
The preloaded CallbackHandler, or None if not preloaded/configured
|
||||
"""
|
||||
return _preloaded_handler
|
||||
|
||||
|
||||
def is_langfuse_preloaded() -> bool:
|
||||
"""Check if Langfuse handler has been preloaded."""
|
||||
return _preloaded_handler is not None
|
||||
|
||||
|
||||
def is_langfuse_configured() -> bool:
|
||||
"""Check if Langfuse environment variables are configured."""
|
||||
return bool(
|
||||
os.getenv("LANGFUSE_PUBLIC_KEY") and os.getenv("LANGFUSE_SECRET_KEY")
|
||||
)
|
||||
|
||||
|
||||
def clear_preloaded_handler() -> None:
|
||||
"""Clear preloaded handler. Useful for testing."""
|
||||
global _preloaded_handler, _initialization_attempted
|
||||
_preloaded_handler = None
|
||||
_initialization_attempted = False
|
||||
@@ -0,0 +1,590 @@
|
||||
"""
|
||||
Local Encoding Service.
|
||||
|
||||
Provides local FFmpeg-based video encoding functionality, extracted from KaraokeFinalise
|
||||
for use by both the cloud backend (video_worker as a fallback) and local CLI.
|
||||
|
||||
This service handles:
|
||||
- Concatenating title, karaoke, and end videos
|
||||
- Encoding to multiple output formats (4K lossless, 4K lossy, 720p, MKV)
|
||||
- Hardware acceleration detection and fallback to software encoding
|
||||
- Remuxing video with instrumental audio
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional, List, Dict, Any, Tuple
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingConfig:
|
||||
"""Configuration for video encoding."""
|
||||
title_video: str # Path to title video
|
||||
karaoke_video: str # Path to karaoke video (with vocals)
|
||||
instrumental_audio: str # Path to instrumental audio
|
||||
end_video: Optional[str] = None # Optional path to end credits video
|
||||
|
||||
# Output paths
|
||||
output_karaoke_mp4: Optional[str] = None
|
||||
output_with_vocals_mp4: Optional[str] = None
|
||||
output_lossless_4k_mp4: Optional[str] = None
|
||||
output_lossy_4k_mp4: Optional[str] = None
|
||||
output_lossless_mkv: Optional[str] = None
|
||||
output_720p_mp4: Optional[str] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingResult:
|
||||
"""Result of video encoding operation."""
|
||||
success: bool
|
||||
output_files: Dict[str, str]
|
||||
error: Optional[str] = None
|
||||
|
||||
|
||||
class LocalEncodingService:
|
||||
"""
|
||||
Service for local FFmpeg-based video encoding.
|
||||
|
||||
Supports hardware acceleration (NVENC) with automatic fallback
|
||||
to software encoding (libx264) when unavailable.
|
||||
"""
|
||||
|
||||
# MP4 flags for better compatibility
|
||||
MP4_FLAGS = "-movflags +faststart"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
dry_run: bool = False,
|
||||
log_level: int = logging.INFO,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the local encoding service.
|
||||
|
||||
Args:
|
||||
dry_run: If True, log commands without executing them
|
||||
log_level: Logging level (affects FFmpeg verbosity)
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.dry_run = dry_run
|
||||
self.log_level = log_level
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
# Hardware acceleration settings (detected on first use)
|
||||
self._hwaccel_available: Optional[bool] = None
|
||||
self._video_encoder: Optional[str] = None
|
||||
self._scale_filter: Optional[str] = None
|
||||
self._hwaccel_decode_flags: Optional[str] = None
|
||||
self._aac_codec: Optional[str] = None
|
||||
|
||||
# Build FFmpeg base command
|
||||
self._ffmpeg_base_command = self._build_ffmpeg_base_command()
|
||||
|
||||
def _build_ffmpeg_base_command(self) -> str:
|
||||
"""Build the FFmpeg base command with appropriate flags."""
|
||||
# Use bundled FFmpeg for frozen builds
|
||||
ffmpeg_path = os.path.join(sys._MEIPASS, "ffmpeg.exe") if getattr(sys, "frozen", False) else "ffmpeg"
|
||||
|
||||
base_cmd = f"{ffmpeg_path} -hide_banner -nostats -y"
|
||||
|
||||
if self.log_level == logging.DEBUG:
|
||||
base_cmd += " -loglevel verbose"
|
||||
else:
|
||||
base_cmd += " -loglevel fatal"
|
||||
|
||||
return base_cmd
|
||||
|
||||
def _detect_hardware_acceleration(self) -> Tuple[bool, str, str, str, str]:
|
||||
"""
|
||||
Detect available hardware acceleration.
|
||||
|
||||
Returns:
|
||||
Tuple of (hwaccel_available, video_encoder, scale_filter,
|
||||
hwaccel_decode_flags, aac_codec)
|
||||
"""
|
||||
self.logger.info("Detecting hardware acceleration capabilities...")
|
||||
|
||||
# Try NVENC (NVIDIA)
|
||||
try:
|
||||
test_cmd = f"{self._ffmpeg_base_command} -hide_banner -loglevel error " \
|
||||
f"-f lavfi -i testsrc=duration=1:size=320x240:rate=1 " \
|
||||
f"-c:v h264_nvenc -f null -"
|
||||
subprocess.run(
|
||||
test_cmd, shell=True, check=True,
|
||||
capture_output=True, timeout=30
|
||||
)
|
||||
self.logger.info("NVIDIA NVENC hardware acceleration available")
|
||||
return (
|
||||
True,
|
||||
"h264_nvenc",
|
||||
"scale_cuda",
|
||||
"-hwaccel cuda -hwaccel_output_format cuda",
|
||||
"aac"
|
||||
)
|
||||
except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError):
|
||||
pass
|
||||
|
||||
# No hardware acceleration available
|
||||
self.logger.info("No hardware acceleration available, using software encoding")
|
||||
return (False, "libx264", "scale", "", "aac")
|
||||
|
||||
@property
|
||||
def hwaccel_available(self) -> bool:
|
||||
"""Check if hardware acceleration is available."""
|
||||
if self._hwaccel_available is None:
|
||||
self._detect_and_set_hwaccel()
|
||||
return self._hwaccel_available
|
||||
|
||||
@property
|
||||
def video_encoder(self) -> str:
|
||||
"""Get the video encoder to use."""
|
||||
if self._video_encoder is None:
|
||||
self._detect_and_set_hwaccel()
|
||||
return self._video_encoder
|
||||
|
||||
@property
|
||||
def scale_filter(self) -> str:
|
||||
"""Get the scale filter to use."""
|
||||
if self._scale_filter is None:
|
||||
self._detect_and_set_hwaccel()
|
||||
return self._scale_filter
|
||||
|
||||
@property
|
||||
def hwaccel_decode_flags(self) -> str:
|
||||
"""Get hardware acceleration decode flags."""
|
||||
if self._hwaccel_decode_flags is None:
|
||||
self._detect_and_set_hwaccel()
|
||||
return self._hwaccel_decode_flags
|
||||
|
||||
@property
|
||||
def aac_codec(self) -> str:
|
||||
"""Get the AAC codec to use."""
|
||||
if self._aac_codec is None:
|
||||
self._detect_and_set_hwaccel()
|
||||
return self._aac_codec
|
||||
|
||||
def _detect_and_set_hwaccel(self) -> None:
|
||||
"""Detect and set hardware acceleration settings."""
|
||||
(
|
||||
self._hwaccel_available,
|
||||
self._video_encoder,
|
||||
self._scale_filter,
|
||||
self._hwaccel_decode_flags,
|
||||
self._aac_codec
|
||||
) = self._detect_hardware_acceleration()
|
||||
|
||||
def _get_nvenc_quality_settings(self, preset: str = "medium") -> str:
|
||||
"""Get NVENC quality settings for different presets."""
|
||||
if not self.hwaccel_available:
|
||||
return ""
|
||||
|
||||
settings = {
|
||||
"lossless": "-preset p7 -tune hq -rc vbr -cq 0 -qmin 0 -qmax 0",
|
||||
"high": "-preset p7 -tune hq -rc vbr -cq 19 -b:v 0",
|
||||
"medium": "-preset p4 -tune hq -rc vbr -cq 23 -b:v 0",
|
||||
"fast": "-preset p2 -tune ll -rc vbr -cq 28 -b:v 0",
|
||||
}
|
||||
return settings.get(preset, settings["medium"])
|
||||
|
||||
def _execute_command(
|
||||
self,
|
||||
command: str,
|
||||
description: str,
|
||||
timeout: int = 3600, # 1 hour default
|
||||
) -> bool:
|
||||
"""
|
||||
Execute an FFmpeg command.
|
||||
|
||||
Args:
|
||||
command: The FFmpeg command to execute
|
||||
description: Human-readable description of the operation
|
||||
timeout: Command timeout in seconds
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
self.logger.info(f"Executing: {description}")
|
||||
self.logger.debug(f"Command: {command}")
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(f"DRY RUN: Would execute: {command}")
|
||||
return True
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
command,
|
||||
shell=True,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout
|
||||
)
|
||||
self.logger.info(f"Completed: {description}")
|
||||
return True
|
||||
except subprocess.CalledProcessError as e:
|
||||
self.logger.error(f"Failed: {description}")
|
||||
self.logger.error(f"Error: {e.stderr}")
|
||||
return False
|
||||
except subprocess.TimeoutExpired:
|
||||
self.logger.error(f"Timeout: {description}")
|
||||
return False
|
||||
|
||||
def _execute_command_with_fallback(
|
||||
self,
|
||||
gpu_command: str,
|
||||
cpu_command: str,
|
||||
description: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Execute a command with GPU, falling back to CPU if it fails.
|
||||
|
||||
Args:
|
||||
gpu_command: Hardware-accelerated command
|
||||
cpu_command: Software fallback command
|
||||
description: Human-readable description
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
if self.hwaccel_available:
|
||||
self.logger.info(f"Trying hardware-accelerated encoding for: {description}")
|
||||
if self._execute_command(gpu_command, description):
|
||||
return True
|
||||
self.logger.warning(f"Hardware encoding failed, falling back to software")
|
||||
|
||||
return self._execute_command(cpu_command, description)
|
||||
|
||||
def remux_with_instrumental(
|
||||
self,
|
||||
input_video: str,
|
||||
instrumental_audio: str,
|
||||
output_file: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Remux video with instrumental audio track.
|
||||
|
||||
Args:
|
||||
input_video: Path to input video file
|
||||
instrumental_audio: Path to instrumental audio file
|
||||
output_file: Path for output file
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
command = (
|
||||
f'{self._ffmpeg_base_command} -i "{input_video}" '
|
||||
f'-i "{instrumental_audio}" -map 0:v -map 1:a -c copy '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
return self._execute_command(command, "Remuxing with instrumental audio")
|
||||
|
||||
def convert_mov_to_mp4(
|
||||
self,
|
||||
input_file: str,
|
||||
output_file: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Convert MOV to MP4 format.
|
||||
|
||||
Args:
|
||||
input_file: Path to input MOV file
|
||||
output_file: Path for output MP4 file
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
gpu_command = (
|
||||
f'{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i "{input_file}" '
|
||||
f'-c:v {self.video_encoder} -c:a copy {self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
cpu_command = (
|
||||
f'{self._ffmpeg_base_command} -i "{input_file}" '
|
||||
f'-c:v libx264 -c:a copy {self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
return self._execute_command_with_fallback(
|
||||
gpu_command, cpu_command, "Converting MOV to MP4"
|
||||
)
|
||||
|
||||
def encode_lossless_mp4(
|
||||
self,
|
||||
title_video: str,
|
||||
karaoke_video: str,
|
||||
output_file: str,
|
||||
end_video: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Create lossless 4K MP4 by concatenating title, karaoke, and optionally end videos.
|
||||
|
||||
Args:
|
||||
title_video: Path to title video
|
||||
karaoke_video: Path to karaoke video
|
||||
output_file: Path for output file
|
||||
end_video: Optional path to end credits video
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
# Quote file paths
|
||||
title_quoted = shlex.quote(os.path.abspath(title_video))
|
||||
karaoke_quoted = shlex.quote(os.path.abspath(karaoke_video))
|
||||
|
||||
# Build filter and inputs for concatenation
|
||||
if end_video and os.path.isfile(end_video):
|
||||
end_quoted = shlex.quote(os.path.abspath(end_video))
|
||||
extra_input = f"-i {end_quoted}"
|
||||
concat_filter = (
|
||||
'-filter_complex "[0:v:0][0:a:0][1:v:0][1:a:0][2:v:0][2:a:0]'
|
||||
'concat=n=3:v=1:a=1[outv][outa]"'
|
||||
)
|
||||
else:
|
||||
extra_input = ""
|
||||
concat_filter = (
|
||||
'-filter_complex "[0:v:0][0:a:0][1:v:0][1:a:0]'
|
||||
'concat=n=2:v=1:a=1[outv][outa]"'
|
||||
)
|
||||
|
||||
gpu_command = (
|
||||
f"{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i {title_quoted} "
|
||||
f"{self.hwaccel_decode_flags} -i {karaoke_quoted} {extra_input} "
|
||||
f'{concat_filter} -map "[outv]" -map "[outa]" -c:v {self.video_encoder} '
|
||||
f'{self._get_nvenc_quality_settings("lossless")} -c:a pcm_s16le '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
cpu_command = (
|
||||
f"{self._ffmpeg_base_command} -i {title_quoted} -i {karaoke_quoted} {extra_input} "
|
||||
f'{concat_filter} -map "[outv]" -map "[outa]" -c:v libx264 -c:a pcm_s16le '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
|
||||
return self._execute_command_with_fallback(
|
||||
gpu_command, cpu_command, "Encoding lossless 4K MP4"
|
||||
)
|
||||
|
||||
def encode_lossy_mp4(
|
||||
self,
|
||||
input_file: str,
|
||||
output_file: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Create lossy 4K MP4 with AAC audio.
|
||||
|
||||
Args:
|
||||
input_file: Path to input file (typically lossless MP4)
|
||||
output_file: Path for output file
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
command = (
|
||||
f'{self._ffmpeg_base_command} -i "{input_file}" '
|
||||
f'-c:v copy -c:a {self.aac_codec} -ar 48000 -b:a 320k '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
return self._execute_command(command, "Encoding lossy 4K MP4 with AAC")
|
||||
|
||||
def encode_lossless_mkv(
|
||||
self,
|
||||
input_file: str,
|
||||
output_file: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Create MKV with FLAC audio (for YouTube upload).
|
||||
|
||||
Args:
|
||||
input_file: Path to input file
|
||||
output_file: Path for output file
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
command = (
|
||||
f'{self._ffmpeg_base_command} -i "{input_file}" '
|
||||
f'-c:v copy -c:a flac "{output_file}"'
|
||||
)
|
||||
return self._execute_command(command, "Creating MKV with FLAC for YouTube")
|
||||
|
||||
def encode_720p(
|
||||
self,
|
||||
input_file: str,
|
||||
output_file: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Create 720p MP4 with AAC audio.
|
||||
|
||||
Args:
|
||||
input_file: Path to input file
|
||||
output_file: Path for output file
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
gpu_command = (
|
||||
f'{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i "{input_file}" '
|
||||
f'-c:v {self.video_encoder} -vf "{self.scale_filter}=1280:720" '
|
||||
f'{self._get_nvenc_quality_settings("medium")} -b:v 2000k '
|
||||
f'-c:a {self.aac_codec} -ar 48000 -b:a 128k '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
cpu_command = (
|
||||
f'{self._ffmpeg_base_command} -i "{input_file}" '
|
||||
f'-c:v libx264 -vf "scale=1280:720" -b:v 2000k -preset medium -tune animation '
|
||||
f'-c:a {self.aac_codec} -ar 48000 -b:a 128k '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
return self._execute_command_with_fallback(
|
||||
gpu_command, cpu_command, "Encoding 720p MP4"
|
||||
)
|
||||
|
||||
def encode_all_formats(
|
||||
self,
|
||||
config: EncodingConfig,
|
||||
) -> EncodingResult:
|
||||
"""
|
||||
Encode video to all output formats.
|
||||
|
||||
This performs the full encoding pipeline:
|
||||
1. Remux with instrumental audio
|
||||
2. Convert to MP4 if needed
|
||||
3. Encode lossless 4K MP4 (concatenated)
|
||||
4. Encode lossy 4K MP4
|
||||
5. Encode lossless MKV (for YouTube)
|
||||
6. Encode 720p MP4
|
||||
|
||||
Args:
|
||||
config: Encoding configuration with input/output paths
|
||||
|
||||
Returns:
|
||||
EncodingResult with success status and output file paths
|
||||
"""
|
||||
output_files = {}
|
||||
|
||||
try:
|
||||
# Step 1: Remux with instrumental audio
|
||||
if config.output_karaoke_mp4:
|
||||
self.logger.info("[Step 1/6] Remuxing video with instrumental audio...")
|
||||
if not self.remux_with_instrumental(
|
||||
config.karaoke_video,
|
||||
config.instrumental_audio,
|
||||
config.output_karaoke_mp4
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to remux with instrumental audio"
|
||||
)
|
||||
output_files["karaoke_mp4"] = config.output_karaoke_mp4
|
||||
|
||||
# Step 2: Convert to MP4 if needed
|
||||
if config.output_with_vocals_mp4:
|
||||
if not config.karaoke_video.endswith(".mp4"):
|
||||
self.logger.info("[Step 2/6] Converting karaoke video to MP4...")
|
||||
if not self.convert_mov_to_mp4(
|
||||
config.karaoke_video,
|
||||
config.output_with_vocals_mp4
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to convert to MP4"
|
||||
)
|
||||
output_files["with_vocals_mp4"] = config.output_with_vocals_mp4
|
||||
else:
|
||||
self.logger.info("[Step 2/6] Skipped - video already MP4")
|
||||
|
||||
# Step 3: Encode lossless 4K MP4
|
||||
if config.output_lossless_4k_mp4:
|
||||
self.logger.info("[Step 3/6] Encoding lossless 4K MP4...")
|
||||
karaoke_for_concat = config.output_karaoke_mp4 or config.karaoke_video
|
||||
if not self.encode_lossless_mp4(
|
||||
config.title_video,
|
||||
karaoke_for_concat,
|
||||
config.output_lossless_4k_mp4,
|
||||
config.end_video
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to encode lossless 4K MP4"
|
||||
)
|
||||
output_files["lossless_4k_mp4"] = config.output_lossless_4k_mp4
|
||||
|
||||
# Step 4: Encode lossy 4K MP4
|
||||
if config.output_lossy_4k_mp4 and config.output_lossless_4k_mp4:
|
||||
self.logger.info("[Step 4/6] Encoding lossy 4K MP4...")
|
||||
if not self.encode_lossy_mp4(
|
||||
config.output_lossless_4k_mp4,
|
||||
config.output_lossy_4k_mp4
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to encode lossy 4K MP4"
|
||||
)
|
||||
output_files["lossy_4k_mp4"] = config.output_lossy_4k_mp4
|
||||
|
||||
# Step 5: Create MKV with FLAC audio
|
||||
if config.output_lossless_mkv and config.output_lossless_4k_mp4:
|
||||
self.logger.info("[Step 5/6] Creating MKV with FLAC audio...")
|
||||
if not self.encode_lossless_mkv(
|
||||
config.output_lossless_4k_mp4,
|
||||
config.output_lossless_mkv
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to create MKV"
|
||||
)
|
||||
output_files["lossless_mkv"] = config.output_lossless_mkv
|
||||
|
||||
# Step 6: Encode 720p version
|
||||
if config.output_720p_mp4 and config.output_lossless_4k_mp4:
|
||||
self.logger.info("[Step 6/6] Encoding 720p MP4...")
|
||||
if not self.encode_720p(
|
||||
config.output_lossless_4k_mp4,
|
||||
config.output_720p_mp4
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to encode 720p"
|
||||
)
|
||||
output_files["720p_mp4"] = config.output_720p_mp4
|
||||
|
||||
self.logger.info("All encoding steps completed successfully")
|
||||
return EncodingResult(success=True, output_files=output_files)
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f"Encoding failed with exception: {e}")
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
|
||||
# Singleton instance and factory function (following existing service pattern)
|
||||
_local_encoding_service: Optional[LocalEncodingService] = None
|
||||
|
||||
|
||||
def get_local_encoding_service(**kwargs) -> LocalEncodingService:
|
||||
"""
|
||||
Get a local encoding service instance.
|
||||
|
||||
Args:
|
||||
**kwargs: Arguments passed to LocalEncodingService
|
||||
|
||||
Returns:
|
||||
LocalEncodingService instance
|
||||
"""
|
||||
global _local_encoding_service
|
||||
|
||||
if _local_encoding_service is None:
|
||||
_local_encoding_service = LocalEncodingService(**kwargs)
|
||||
|
||||
return _local_encoding_service
|
||||
@@ -0,0 +1,407 @@
|
||||
"""
|
||||
Local Preview Encoding Service.
|
||||
|
||||
Provides local FFmpeg-based preview video encoding functionality, extracted from
|
||||
VideoGenerator for use by the GCE worker. This ensures the same encoding logic
|
||||
is used across local CLI, Cloud Run, and GCE worker environments.
|
||||
|
||||
This service handles:
|
||||
- Generating preview videos with ASS subtitle overlay
|
||||
- Hardware acceleration detection (NVENC) with fallback to software encoding
|
||||
- Background image/color support
|
||||
- Custom font support
|
||||
- Optimized settings for fast preview generation
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Optional, List
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class PreviewEncodingConfig:
|
||||
"""Configuration for preview video encoding."""
|
||||
ass_path: str # Path to ASS subtitles file
|
||||
audio_path: str # Path to audio file
|
||||
output_path: str # Path for output video file
|
||||
|
||||
# Optional background settings
|
||||
background_image_path: Optional[str] = None # Path to background image
|
||||
background_color: str = "black" # Fallback background color
|
||||
|
||||
# Optional font settings
|
||||
font_path: Optional[str] = None # Path to custom font file
|
||||
|
||||
|
||||
@dataclass
|
||||
class PreviewEncodingResult:
|
||||
"""Result of preview encoding operation."""
|
||||
success: bool
|
||||
output_path: Optional[str] = None
|
||||
error: Optional[str] = None
|
||||
|
||||
|
||||
class LocalPreviewEncodingService:
|
||||
"""
|
||||
Service for local FFmpeg-based preview video encoding.
|
||||
|
||||
This is the single source of truth for preview encoding logic, used by:
|
||||
- Local CLI (via VideoGenerator which delegates here)
|
||||
- Cloud Run (when GCE is unavailable)
|
||||
- GCE worker (via installed wheel)
|
||||
|
||||
Supports hardware acceleration (NVENC) with automatic fallback
|
||||
to software encoding (libx264) when unavailable.
|
||||
"""
|
||||
|
||||
# Preview video settings - these are the canonical values
|
||||
PREVIEW_WIDTH = 480
|
||||
PREVIEW_HEIGHT = 270
|
||||
PREVIEW_FPS = 24
|
||||
PREVIEW_AUDIO_BITRATE = "96k"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the local preview encoding service.
|
||||
|
||||
Args:
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
# Hardware acceleration settings (detected on first use)
|
||||
self._nvenc_available: Optional[bool] = None
|
||||
self._video_encoder: Optional[str] = None
|
||||
self._hwaccel_flags: Optional[List[str]] = None
|
||||
|
||||
def _detect_nvenc_support(self) -> bool:
|
||||
"""
|
||||
Detect if NVENC hardware encoding is available.
|
||||
|
||||
Returns:
|
||||
True if NVENC is available, False otherwise
|
||||
"""
|
||||
try:
|
||||
self.logger.info("Detecting NVENC hardware acceleration...")
|
||||
|
||||
# Test h264_nvenc encoder directly
|
||||
test_cmd = [
|
||||
"ffmpeg", "-hide_banner", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", "testsrc=duration=1:size=320x240:rate=1",
|
||||
"-c:v", "h264_nvenc", "-f", "null", "-"
|
||||
]
|
||||
|
||||
result = subprocess.run(
|
||||
test_cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
self.logger.info("NVENC hardware acceleration available")
|
||||
return True
|
||||
|
||||
# Try alternative test with different source
|
||||
alt_test_cmd = [
|
||||
"ffmpeg", "-hide_banner", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", "color=red:size=320x240:duration=0.1",
|
||||
"-c:v", "h264_nvenc", "-preset", "fast", "-f", "null", "-"
|
||||
]
|
||||
|
||||
alt_result = subprocess.run(
|
||||
alt_test_cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30
|
||||
)
|
||||
|
||||
if alt_result.returncode == 0:
|
||||
self.logger.info("NVENC hardware acceleration available")
|
||||
return True
|
||||
|
||||
self.logger.info("NVENC not available, using software encoding")
|
||||
return False
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
self.logger.debug("NVENC detection timed out")
|
||||
return False
|
||||
except Exception as e:
|
||||
self.logger.debug(f"NVENC detection failed: {e}")
|
||||
return False
|
||||
|
||||
def _configure_hardware_acceleration(self) -> None:
|
||||
"""Configure hardware acceleration settings based on detected capabilities."""
|
||||
if self._nvenc_available is None:
|
||||
self._nvenc_available = self._detect_nvenc_support()
|
||||
|
||||
if self._nvenc_available:
|
||||
self._video_encoder = "h264_nvenc"
|
||||
self._hwaccel_flags = ["-hwaccel", "cuda", "-hwaccel_output_format", "cuda"]
|
||||
self.logger.info("Using NVENC hardware acceleration for preview encoding")
|
||||
else:
|
||||
self._video_encoder = "libx264"
|
||||
self._hwaccel_flags = []
|
||||
self.logger.info("Using software encoding (libx264) for preview")
|
||||
|
||||
@property
|
||||
def nvenc_available(self) -> bool:
|
||||
"""Check if NVENC hardware acceleration is available."""
|
||||
if self._nvenc_available is None:
|
||||
self._configure_hardware_acceleration()
|
||||
return self._nvenc_available
|
||||
|
||||
@property
|
||||
def video_encoder(self) -> str:
|
||||
"""Get the video encoder to use."""
|
||||
if self._video_encoder is None:
|
||||
self._configure_hardware_acceleration()
|
||||
return self._video_encoder
|
||||
|
||||
@property
|
||||
def hwaccel_flags(self) -> List[str]:
|
||||
"""Get hardware acceleration flags."""
|
||||
if self._hwaccel_flags is None:
|
||||
self._configure_hardware_acceleration()
|
||||
return self._hwaccel_flags
|
||||
|
||||
def _escape_ffmpeg_filter_path(self, path: str) -> str:
|
||||
"""
|
||||
Escape a path for FFmpeg filter expressions (for subprocess without shell).
|
||||
|
||||
When using subprocess with a command list (no shell), FFmpeg receives the
|
||||
filter string directly. FFmpeg's filter parser requires escaping:
|
||||
- Backslashes: double them (\\ -> \\\\)
|
||||
- Single quotes/apostrophes: escape with three backslashes (' -> \\\\')
|
||||
- Spaces: escape with backslash ( -> \\ )
|
||||
- Special characters: :,[];
|
||||
|
||||
Example: "I'm With You" becomes "I\\\\'m\\ With\\ You"
|
||||
"""
|
||||
# First escape existing backslashes
|
||||
escaped = path.replace("\\", "\\\\")
|
||||
# Escape single quotes
|
||||
escaped = escaped.replace("'", "\\\\\\'")
|
||||
# Escape spaces
|
||||
escaped = escaped.replace(" ", "\\ ")
|
||||
# Escape FFmpeg filter special characters
|
||||
escaped = escaped.replace(":", "\\:")
|
||||
escaped = escaped.replace(",", "\\,")
|
||||
escaped = escaped.replace("[", "\\[")
|
||||
escaped = escaped.replace("]", "\\]")
|
||||
escaped = escaped.replace(";", "\\;")
|
||||
return escaped
|
||||
|
||||
def _build_ass_filter(self, ass_path: str, font_path: Optional[str] = None) -> str:
|
||||
"""
|
||||
Build ASS filter with optional font directory support.
|
||||
|
||||
Args:
|
||||
ass_path: Path to ASS subtitles file
|
||||
font_path: Optional path to custom font file
|
||||
|
||||
Returns:
|
||||
FFmpeg ASS filter string
|
||||
"""
|
||||
escaped_ass_path = self._escape_ffmpeg_filter_path(ass_path)
|
||||
ass_filter = f"ass={escaped_ass_path}"
|
||||
|
||||
if font_path and os.path.isfile(font_path):
|
||||
font_dir = os.path.dirname(font_path)
|
||||
escaped_font_dir = self._escape_ffmpeg_filter_path(font_dir)
|
||||
ass_filter += f":fontsdir={escaped_font_dir}"
|
||||
self.logger.debug(f"Using font directory: {font_dir}")
|
||||
|
||||
return ass_filter
|
||||
|
||||
def _build_preview_ffmpeg_command(self, config: PreviewEncodingConfig) -> List[str]:
|
||||
"""
|
||||
Build FFmpeg command for preview video generation.
|
||||
|
||||
This is the canonical preview encoding command, ensuring consistency
|
||||
across all environments (local CLI, Cloud Run, GCE worker).
|
||||
|
||||
Args:
|
||||
config: Preview encoding configuration
|
||||
|
||||
Returns:
|
||||
FFmpeg command as a list of arguments
|
||||
"""
|
||||
width, height = self.PREVIEW_WIDTH, self.PREVIEW_HEIGHT
|
||||
|
||||
cmd = [
|
||||
"ffmpeg",
|
||||
"-hide_banner",
|
||||
"-loglevel", "error",
|
||||
"-r", str(self.PREVIEW_FPS),
|
||||
]
|
||||
|
||||
# Add hardware acceleration flags if available
|
||||
cmd.extend(self.hwaccel_flags)
|
||||
|
||||
# Input source (background image or solid color)
|
||||
if config.background_image_path and os.path.isfile(config.background_image_path):
|
||||
self.logger.debug(f"Using background image: {config.background_image_path}")
|
||||
cmd.extend([
|
||||
"-loop", "1",
|
||||
"-i", config.background_image_path,
|
||||
])
|
||||
# Build video filter with scaling and ASS subtitles
|
||||
video_filter = (
|
||||
f"scale={width}:{height}:force_original_aspect_ratio=decrease,"
|
||||
f"pad={width}:{height}:(ow-iw)/2:(oh-ih)/2,"
|
||||
f"{self._build_ass_filter(config.ass_path, config.font_path)}"
|
||||
)
|
||||
else:
|
||||
self.logger.debug(f"Using solid {config.background_color} background")
|
||||
cmd.extend([
|
||||
"-f", "lavfi",
|
||||
"-i", f"color=c={config.background_color}:s={width}x{height}:r={self.PREVIEW_FPS}",
|
||||
])
|
||||
# Just ASS subtitles, no scaling needed
|
||||
video_filter = self._build_ass_filter(config.ass_path, config.font_path)
|
||||
|
||||
cmd.extend([
|
||||
"-i", config.audio_path,
|
||||
"-vf", video_filter,
|
||||
"-c:a", "aac",
|
||||
"-b:a", self.PREVIEW_AUDIO_BITRATE,
|
||||
"-c:v", self.video_encoder,
|
||||
])
|
||||
|
||||
# Add encoder-specific settings optimized for speed
|
||||
if self.nvenc_available:
|
||||
cmd.extend([
|
||||
"-preset", "p1", # Fastest NVENC preset
|
||||
"-tune", "ll", # Low latency
|
||||
"-rc", "cbr", # Constant bitrate for speed
|
||||
"-b:v", "800k", # Lower bitrate for speed
|
||||
"-profile:v", "baseline", # Most compatible profile
|
||||
"-level", "3.1", # Lower level for speed
|
||||
])
|
||||
self.logger.debug("Using NVENC with maximum speed settings")
|
||||
else:
|
||||
cmd.extend([
|
||||
"-profile:v", "baseline",
|
||||
"-level", "3.0",
|
||||
"-preset", "superfast",
|
||||
"-tune", "fastdecode",
|
||||
"-b:v", "600k",
|
||||
"-maxrate", "800k",
|
||||
"-bufsize", "1200k",
|
||||
"-crf", "28",
|
||||
])
|
||||
self.logger.debug("Using software encoding with maximum speed settings")
|
||||
|
||||
cmd.extend([
|
||||
"-pix_fmt", "yuv420p", # Required for browser compatibility
|
||||
"-movflags", "+faststart+frag_keyframe+empty_moov+dash",
|
||||
"-g", "48", # Keyframe every 48 frames (2 seconds at 24fps)
|
||||
"-keyint_min", "48",
|
||||
"-sc_threshold", "0", # Disable scene change detection for speed
|
||||
"-threads", "0", # Use all available CPU threads
|
||||
"-shortest",
|
||||
"-y",
|
||||
config.output_path,
|
||||
])
|
||||
|
||||
return cmd
|
||||
|
||||
def encode_preview(self, config: PreviewEncodingConfig) -> PreviewEncodingResult:
|
||||
"""
|
||||
Encode a preview video.
|
||||
|
||||
Args:
|
||||
config: Preview encoding configuration
|
||||
|
||||
Returns:
|
||||
PreviewEncodingResult with success status and output path
|
||||
"""
|
||||
self.logger.info(f"Encoding preview video: {config.output_path}")
|
||||
|
||||
# Validate input files
|
||||
if not os.path.isfile(config.ass_path):
|
||||
return PreviewEncodingResult(
|
||||
success=False,
|
||||
error=f"ASS subtitles file not found: {config.ass_path}"
|
||||
)
|
||||
|
||||
if not os.path.isfile(config.audio_path):
|
||||
return PreviewEncodingResult(
|
||||
success=False,
|
||||
error=f"Audio file not found: {config.audio_path}"
|
||||
)
|
||||
|
||||
# Ensure output directory exists
|
||||
output_dir = os.path.dirname(config.output_path)
|
||||
if output_dir:
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
|
||||
try:
|
||||
# Build and execute FFmpeg command
|
||||
cmd = self._build_preview_ffmpeg_command(config)
|
||||
self.logger.debug(f"FFmpeg command: {' '.join(cmd)}")
|
||||
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=300 # 5 minute timeout for preview encoding
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
error_msg = result.stderr[-500:] if result.stderr else "Unknown error"
|
||||
self.logger.error(f"FFmpeg failed: {error_msg}")
|
||||
return PreviewEncodingResult(
|
||||
success=False,
|
||||
error=f"FFmpeg preview encoding failed: {error_msg}"
|
||||
)
|
||||
|
||||
self.logger.info(f"Preview encoded successfully: {config.output_path}")
|
||||
return PreviewEncodingResult(
|
||||
success=True,
|
||||
output_path=config.output_path
|
||||
)
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
self.logger.error("Preview encoding timed out")
|
||||
return PreviewEncodingResult(
|
||||
success=False,
|
||||
error="Preview encoding timed out after 5 minutes"
|
||||
)
|
||||
except Exception as e:
|
||||
self.logger.error(f"Preview encoding failed: {e}")
|
||||
return PreviewEncodingResult(
|
||||
success=False,
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
|
||||
# Singleton instance and factory function
|
||||
_local_preview_encoding_service: Optional[LocalPreviewEncodingService] = None
|
||||
|
||||
|
||||
def get_local_preview_encoding_service(**kwargs) -> LocalPreviewEncodingService:
|
||||
"""
|
||||
Get a local preview encoding service instance.
|
||||
|
||||
Args:
|
||||
**kwargs: Arguments passed to LocalPreviewEncodingService
|
||||
|
||||
Returns:
|
||||
LocalPreviewEncodingService instance
|
||||
"""
|
||||
global _local_preview_encoding_service
|
||||
|
||||
if _local_preview_encoding_service is None:
|
||||
_local_preview_encoding_service = LocalPreviewEncodingService(**kwargs)
|
||||
|
||||
return _local_preview_encoding_service
|
||||
@@ -0,0 +1,216 @@
|
||||
"""
|
||||
LyricsTranscriber cache synchronization with GCS.
|
||||
|
||||
This service persists LyricsTranscriber's cache files to GCS so that
|
||||
cloud workers (Cloud Run instances) can share cache across containers.
|
||||
|
||||
Cache files are stored flat in GCS under lyrics-transcriber-cache/ prefix:
|
||||
- Transcription: {provider}_{audio_hash}_raw.json, {provider}_{audio_hash}_converted.json
|
||||
- Lyrics: {provider}_{artist_title_hash}_raw.json, {provider}_{artist_title_hash}_converted.json
|
||||
|
||||
Hash computation matches LyricsTranscriber's implementation exactly:
|
||||
- Audio hash: MD5 of audio file bytes
|
||||
- Lyrics hash: MD5 of "{artist.lower()}_{title.lower()}"
|
||||
"""
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Providers that use audio file hash as cache key
|
||||
TRANSCRIPTION_PROVIDERS = ["audioshake", "whisper", "localwhisper"]
|
||||
|
||||
# Providers that use artist+title hash as cache key
|
||||
LYRICS_PROVIDERS = ["genius", "spotify", "lrclib", "musixmatch"]
|
||||
|
||||
# Cache file suffixes
|
||||
CACHE_SUFFIXES = ["raw", "converted"]
|
||||
|
||||
|
||||
class LyricsCacheService:
|
||||
"""Service to sync LyricsTranscriber cache with GCS."""
|
||||
|
||||
GCS_CACHE_PREFIX = "lyrics-transcriber-cache/"
|
||||
|
||||
def __init__(self, storage: Optional[StorageService] = None):
|
||||
"""Initialize the cache service.
|
||||
|
||||
Args:
|
||||
storage: StorageService instance. If None, creates a new one.
|
||||
"""
|
||||
self.storage = storage or StorageService()
|
||||
|
||||
def compute_audio_hash(self, audio_path: str) -> str:
|
||||
"""Compute MD5 hash of audio file bytes.
|
||||
|
||||
This matches LyricsTranscriber's _get_file_hash() method exactly.
|
||||
|
||||
Args:
|
||||
audio_path: Path to the audio file.
|
||||
|
||||
Returns:
|
||||
MD5 hex digest of the file contents.
|
||||
"""
|
||||
md5_hash = hashlib.md5()
|
||||
with open(audio_path, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(4096), b""):
|
||||
md5_hash.update(chunk)
|
||||
return md5_hash.hexdigest()
|
||||
|
||||
def compute_lyrics_hash(self, artist: str, title: str) -> str:
|
||||
"""Compute MD5 hash of artist and title.
|
||||
|
||||
This matches LyricsTranscriber's _get_artist_title_hash() method exactly.
|
||||
|
||||
Args:
|
||||
artist: Artist name.
|
||||
title: Track title.
|
||||
|
||||
Returns:
|
||||
MD5 hex digest of "{artist.lower()}_{title.lower()}".
|
||||
"""
|
||||
combined = f"{artist.lower()}_{title.lower()}"
|
||||
return hashlib.md5(combined.encode()).hexdigest()
|
||||
|
||||
def _get_cache_filenames(
|
||||
self, providers: List[str], hash_value: str
|
||||
) -> List[str]:
|
||||
"""Generate list of possible cache filenames for given providers and hash.
|
||||
|
||||
Args:
|
||||
providers: List of provider names (e.g., ["audioshake", "whisper"]).
|
||||
hash_value: The hash to use in filenames.
|
||||
|
||||
Returns:
|
||||
List of filenames like ["audioshake_abc123_raw.json", ...].
|
||||
"""
|
||||
filenames = []
|
||||
for provider in providers:
|
||||
for suffix in CACHE_SUFFIXES:
|
||||
filenames.append(f"{provider}_{hash_value}_{suffix}.json")
|
||||
return filenames
|
||||
|
||||
def sync_cache_from_gcs(
|
||||
self,
|
||||
local_cache_dir: str,
|
||||
audio_hash: str,
|
||||
lyrics_hash: str,
|
||||
) -> Dict[str, int]:
|
||||
"""Download relevant cache files from GCS to local directory.
|
||||
|
||||
Downloads cache files for both transcription (audio hash) and
|
||||
lyrics (artist+title hash) providers.
|
||||
|
||||
Args:
|
||||
local_cache_dir: Local directory to download cache files to.
|
||||
audio_hash: MD5 hash of audio file.
|
||||
lyrics_hash: MD5 hash of artist+title.
|
||||
|
||||
Returns:
|
||||
Dict with counts: {"downloaded": N, "not_found": M, "errors": E}
|
||||
"""
|
||||
os.makedirs(local_cache_dir, exist_ok=True)
|
||||
|
||||
stats = {"downloaded": 0, "not_found": 0, "errors": 0}
|
||||
|
||||
# Get all possible cache filenames
|
||||
transcription_files = self._get_cache_filenames(
|
||||
TRANSCRIPTION_PROVIDERS, audio_hash
|
||||
)
|
||||
lyrics_files = self._get_cache_filenames(LYRICS_PROVIDERS, lyrics_hash)
|
||||
all_files = transcription_files + lyrics_files
|
||||
|
||||
for filename in all_files:
|
||||
gcs_path = f"{self.GCS_CACHE_PREFIX}{filename}"
|
||||
local_path = os.path.join(local_cache_dir, filename)
|
||||
|
||||
try:
|
||||
if self.storage.file_exists(gcs_path):
|
||||
self.storage.download_file(gcs_path, local_path)
|
||||
logger.info(f"Cache hit: downloaded {filename}")
|
||||
stats["downloaded"] += 1
|
||||
else:
|
||||
logger.debug(f"Cache miss: {filename} not in GCS")
|
||||
stats["not_found"] += 1
|
||||
except Exception as e:
|
||||
logger.warning(f"Error downloading cache file {filename}: {e}")
|
||||
stats["errors"] += 1
|
||||
|
||||
logger.info(
|
||||
f"Cache sync from GCS complete: "
|
||||
f"{stats['downloaded']} downloaded, "
|
||||
f"{stats['not_found']} not found, "
|
||||
f"{stats['errors']} errors"
|
||||
)
|
||||
return stats
|
||||
|
||||
def sync_cache_to_gcs(
|
||||
self,
|
||||
local_cache_dir: str,
|
||||
audio_hash: str,
|
||||
lyrics_hash: str,
|
||||
) -> Dict[str, int]:
|
||||
"""Upload new cache files from local directory to GCS.
|
||||
|
||||
Only uploads files that match expected cache patterns for the given
|
||||
hashes and don't already exist in GCS.
|
||||
|
||||
Args:
|
||||
local_cache_dir: Local directory with cache files.
|
||||
audio_hash: MD5 hash of audio file.
|
||||
lyrics_hash: MD5 hash of artist+title.
|
||||
|
||||
Returns:
|
||||
Dict with counts: {"uploaded": N, "skipped": M, "errors": E}
|
||||
"""
|
||||
stats = {"uploaded": 0, "skipped": 0, "errors": 0}
|
||||
|
||||
if not os.path.exists(local_cache_dir):
|
||||
logger.warning(f"Local cache dir does not exist: {local_cache_dir}")
|
||||
return stats
|
||||
|
||||
# Get all possible cache filenames we're interested in
|
||||
transcription_files = self._get_cache_filenames(
|
||||
TRANSCRIPTION_PROVIDERS, audio_hash
|
||||
)
|
||||
lyrics_files = self._get_cache_filenames(LYRICS_PROVIDERS, lyrics_hash)
|
||||
expected_files = set(transcription_files + lyrics_files)
|
||||
|
||||
# Check each file in local cache dir
|
||||
for filename in os.listdir(local_cache_dir):
|
||||
# Only process files we expect (matching our hash patterns)
|
||||
if filename not in expected_files:
|
||||
continue
|
||||
|
||||
local_path = os.path.join(local_cache_dir, filename)
|
||||
if not os.path.isfile(local_path):
|
||||
continue
|
||||
|
||||
gcs_path = f"{self.GCS_CACHE_PREFIX}{filename}"
|
||||
|
||||
try:
|
||||
# Skip if already exists in GCS (same hash = same content)
|
||||
if self.storage.file_exists(gcs_path):
|
||||
logger.debug(f"Cache file already in GCS: {filename}")
|
||||
stats["skipped"] += 1
|
||||
continue
|
||||
|
||||
self.storage.upload_file(local_path, gcs_path)
|
||||
logger.info(f"Uploaded cache file: {filename}")
|
||||
stats["uploaded"] += 1
|
||||
except Exception as e:
|
||||
logger.warning(f"Error uploading cache file {filename}: {e}")
|
||||
stats["errors"] += 1
|
||||
|
||||
logger.info(
|
||||
f"Cache sync to GCS complete: "
|
||||
f"{stats['uploaded']} uploaded, "
|
||||
f"{stats['skipped']} skipped (already exist), "
|
||||
f"{stats['errors']} errors"
|
||||
)
|
||||
return stats
|
||||
@@ -0,0 +1,122 @@
|
||||
"""NLTK resource preloader for container startup.
|
||||
|
||||
Loads NLTK data at container startup to avoid slow downloads during request processing.
|
||||
Cloud Run's ephemeral filesystem means NLTK data must be re-downloaded on each cold start,
|
||||
which can take 30-100+ seconds for cmudict.
|
||||
|
||||
See docs/archive/2026-01-08-performance-investigation.md for background.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Singleton storage for preloaded resources
|
||||
_preloaded_resources: Dict[str, Any] = {}
|
||||
|
||||
|
||||
def preload_nltk_cmudict() -> None:
|
||||
"""Preload NLTK's CMU Pronouncing Dictionary at startup.
|
||||
|
||||
The cmudict is used by SyllablesMatchHandler for syllable counting.
|
||||
Without preloading, each SyllablesMatchHandler init downloads ~30MB,
|
||||
which took 50-100+ seconds in Cloud Run.
|
||||
"""
|
||||
global _preloaded_resources
|
||||
|
||||
if "cmudict" in _preloaded_resources:
|
||||
logger.info("NLTK cmudict already preloaded")
|
||||
return
|
||||
|
||||
logger.info("Preloading NLTK cmudict...")
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
import nltk
|
||||
|
||||
# Ensure the data is downloaded
|
||||
try:
|
||||
from nltk.corpus import cmudict
|
||||
|
||||
# Try to access it - will raise LookupError if not downloaded
|
||||
_ = cmudict.dict()
|
||||
except LookupError:
|
||||
logger.info("Downloading NLTK cmudict data...")
|
||||
nltk.download("cmudict", quiet=True)
|
||||
from nltk.corpus import cmudict
|
||||
|
||||
# Load into memory
|
||||
cmu_dict = cmudict.dict()
|
||||
_preloaded_resources["cmudict"] = cmu_dict
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
logger.info(f"NLTK cmudict preloaded in {elapsed:.2f}s ({len(cmu_dict)} entries)")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to preload NLTK cmudict: {e}")
|
||||
raise
|
||||
|
||||
|
||||
def get_preloaded_cmudict() -> Optional[Dict]:
|
||||
"""Get the preloaded CMU dictionary if available.
|
||||
|
||||
Returns:
|
||||
The preloaded cmudict dictionary, or None if not preloaded
|
||||
"""
|
||||
return _preloaded_resources.get("cmudict")
|
||||
|
||||
|
||||
def is_cmudict_preloaded() -> bool:
|
||||
"""Check if cmudict has been preloaded."""
|
||||
return "cmudict" in _preloaded_resources
|
||||
|
||||
|
||||
def preload_nltk_punkt() -> None:
|
||||
"""Preload NLTK's punkt tokenizer (optional, used for sentence tokenization)."""
|
||||
global _preloaded_resources
|
||||
|
||||
if "punkt" in _preloaded_resources:
|
||||
logger.info("NLTK punkt already preloaded")
|
||||
return
|
||||
|
||||
logger.info("Preloading NLTK punkt tokenizer...")
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
import nltk
|
||||
|
||||
try:
|
||||
from nltk.tokenize import word_tokenize
|
||||
|
||||
# Test it works
|
||||
_ = word_tokenize("test")
|
||||
except LookupError:
|
||||
logger.info("Downloading NLTK punkt data...")
|
||||
nltk.download("punkt", quiet=True)
|
||||
nltk.download("punkt_tab", quiet=True)
|
||||
|
||||
_preloaded_resources["punkt"] = True
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
logger.info(f"NLTK punkt preloaded in {elapsed:.2f}s")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to preload NLTK punkt (non-critical): {e}")
|
||||
|
||||
|
||||
def preload_all_nltk_resources() -> None:
|
||||
"""Preload all NLTK resources used by the application."""
|
||||
preload_nltk_cmudict()
|
||||
# punkt is optional and less critical
|
||||
try:
|
||||
preload_nltk_punkt()
|
||||
except Exception:
|
||||
pass # Non-critical
|
||||
|
||||
|
||||
def clear_preloaded_resources() -> None:
|
||||
"""Clear all preloaded resources. Useful for testing."""
|
||||
global _preloaded_resources
|
||||
_preloaded_resources.clear()
|
||||
@@ -0,0 +1,287 @@
|
||||
"""
|
||||
Packaging Service.
|
||||
|
||||
Provides CDG and TXT package generation functionality, extracted from KaraokeFinalise
|
||||
for use by both the cloud backend (video_worker) and local CLI.
|
||||
|
||||
This service handles:
|
||||
- CDG (CD+G) file generation from LRC files
|
||||
- TXT lyric file generation from LRC files
|
||||
- ZIP packaging of CDG/MP3 and TXT/MP3 pairs
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import zipfile
|
||||
from typing import Optional, Dict, Any, Tuple
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class PackagingService:
|
||||
"""
|
||||
Service for creating CDG and TXT karaoke packages.
|
||||
|
||||
CDG (CD+Graphics) is a format used by karaoke machines.
|
||||
TXT packages are used by software karaoke players.
|
||||
Both formats are packaged as ZIP files with an MP3 audio track.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
cdg_styles: Optional[Dict[str, Any]] = None,
|
||||
dry_run: bool = False,
|
||||
non_interactive: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the packaging service.
|
||||
|
||||
Args:
|
||||
cdg_styles: CDG style configuration for CDG generation
|
||||
dry_run: If True, log actions without performing them
|
||||
non_interactive: If True, skip interactive prompts
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.cdg_styles = cdg_styles
|
||||
self.dry_run = dry_run
|
||||
self.non_interactive = non_interactive
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
def create_cdg_package(
|
||||
self,
|
||||
lrc_file: str,
|
||||
audio_file: str,
|
||||
output_zip_path: str,
|
||||
artist: str,
|
||||
title: str,
|
||||
output_mp3_path: Optional[str] = None,
|
||||
output_cdg_path: Optional[str] = None,
|
||||
) -> Tuple[str, Optional[str], Optional[str]]:
|
||||
"""
|
||||
Create a CDG package (ZIP containing CDG and MP3 files).
|
||||
|
||||
Args:
|
||||
lrc_file: Path to the LRC lyrics file
|
||||
audio_file: Path to the instrumental audio file
|
||||
output_zip_path: Path for the output ZIP file
|
||||
artist: Artist name for metadata
|
||||
title: Song title for metadata
|
||||
output_mp3_path: Optional path for the extracted MP3 file
|
||||
output_cdg_path: Optional path for the extracted CDG file
|
||||
|
||||
Returns:
|
||||
Tuple of (zip_path, mp3_path, cdg_path)
|
||||
|
||||
Raises:
|
||||
ValueError: If CDG styles are not configured
|
||||
FileNotFoundError: If input files are missing
|
||||
Exception: If CDG generation fails
|
||||
"""
|
||||
self.logger.info(f"Creating CDG package for {artist} - {title}")
|
||||
|
||||
# Validate inputs
|
||||
if not os.path.isfile(lrc_file):
|
||||
raise FileNotFoundError(f"LRC file not found: {lrc_file}")
|
||||
if not os.path.isfile(audio_file):
|
||||
raise FileNotFoundError(f"Audio file not found: {audio_file}")
|
||||
|
||||
# Check if ZIP already exists
|
||||
if os.path.isfile(output_zip_path):
|
||||
if self.non_interactive:
|
||||
self.logger.info(
|
||||
f"CDG ZIP exists, will be overwritten: {output_zip_path}"
|
||||
)
|
||||
else:
|
||||
self.logger.info(f"CDG ZIP already exists: {output_zip_path}")
|
||||
|
||||
# Check if individual files exist (allows skipping generation)
|
||||
if output_mp3_path and output_cdg_path:
|
||||
if os.path.isfile(output_mp3_path) and os.path.isfile(output_cdg_path):
|
||||
self.logger.info("Found existing MP3 and CDG files, creating ZIP directly")
|
||||
if not self.dry_run:
|
||||
self._create_zip_from_files(
|
||||
output_zip_path,
|
||||
[(output_mp3_path, os.path.basename(output_mp3_path)),
|
||||
(output_cdg_path, os.path.basename(output_cdg_path))]
|
||||
)
|
||||
return output_zip_path, output_mp3_path, output_cdg_path
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(
|
||||
f"DRY RUN: Would generate CDG package: {output_zip_path}"
|
||||
)
|
||||
return output_zip_path, output_mp3_path, output_cdg_path
|
||||
|
||||
# Generate CDG files
|
||||
if self.cdg_styles is None:
|
||||
raise ValueError(
|
||||
"CDG styles configuration is required for CDG generation"
|
||||
)
|
||||
|
||||
self.logger.info("Generating CDG and MP3 files")
|
||||
from lyrics_transcriber.output.cdg import CDGGenerator
|
||||
|
||||
output_dir = os.path.dirname(output_zip_path) or os.getcwd()
|
||||
generator = CDGGenerator(output_dir=output_dir, logger=self.logger)
|
||||
|
||||
cdg_file, mp3_file, zip_file = generator.generate_cdg_from_lrc(
|
||||
lrc_file=lrc_file,
|
||||
audio_file=audio_file,
|
||||
title=title,
|
||||
artist=artist,
|
||||
cdg_styles=self.cdg_styles,
|
||||
)
|
||||
|
||||
# Rename ZIP to expected output path if different
|
||||
if os.path.isfile(zip_file) and zip_file != output_zip_path:
|
||||
os.rename(zip_file, output_zip_path)
|
||||
self.logger.info(f"Renamed CDG ZIP: {zip_file} -> {output_zip_path}")
|
||||
|
||||
if not os.path.isfile(output_zip_path):
|
||||
raise Exception(f"Failed to create CDG ZIP file: {output_zip_path}")
|
||||
|
||||
# Extract the ZIP to get individual files if paths provided
|
||||
extracted_mp3 = None
|
||||
extracted_cdg = None
|
||||
if output_mp3_path or output_cdg_path:
|
||||
self.logger.info(f"Extracting CDG ZIP file: {output_zip_path}")
|
||||
with zipfile.ZipFile(output_zip_path, "r") as zip_ref:
|
||||
zip_ref.extractall(output_dir)
|
||||
|
||||
# Find extracted files
|
||||
if output_mp3_path and os.path.isfile(output_mp3_path):
|
||||
extracted_mp3 = output_mp3_path
|
||||
self.logger.info(f"Extracted MP3: {extracted_mp3}")
|
||||
if output_cdg_path and os.path.isfile(output_cdg_path):
|
||||
extracted_cdg = output_cdg_path
|
||||
self.logger.info(f"Extracted CDG: {extracted_cdg}")
|
||||
|
||||
self.logger.info(f"CDG package created: {output_zip_path}")
|
||||
return output_zip_path, extracted_mp3, extracted_cdg
|
||||
|
||||
def create_txt_package(
|
||||
self,
|
||||
lrc_file: str,
|
||||
mp3_file: str,
|
||||
output_zip_path: str,
|
||||
output_txt_path: Optional[str] = None,
|
||||
) -> Tuple[str, Optional[str]]:
|
||||
"""
|
||||
Create a TXT package (ZIP containing TXT lyrics and MP3 files).
|
||||
|
||||
Args:
|
||||
lrc_file: Path to the LRC lyrics file
|
||||
mp3_file: Path to the MP3 audio file
|
||||
output_zip_path: Path for the output ZIP file
|
||||
output_txt_path: Optional path for the generated TXT file
|
||||
|
||||
Returns:
|
||||
Tuple of (zip_path, txt_path)
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: If input files are missing
|
||||
Exception: If TXT generation fails
|
||||
"""
|
||||
self.logger.info(f"Creating TXT package from {lrc_file}")
|
||||
|
||||
# Validate inputs
|
||||
if not os.path.isfile(lrc_file):
|
||||
raise FileNotFoundError(f"LRC file not found: {lrc_file}")
|
||||
if not os.path.isfile(mp3_file):
|
||||
raise FileNotFoundError(f"MP3 file not found: {mp3_file}")
|
||||
|
||||
# Check if ZIP already exists
|
||||
if os.path.isfile(output_zip_path):
|
||||
if self.non_interactive:
|
||||
self.logger.info(
|
||||
f"TXT ZIP exists, will be overwritten: {output_zip_path}"
|
||||
)
|
||||
else:
|
||||
self.logger.info(f"TXT ZIP already exists: {output_zip_path}")
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(
|
||||
f"DRY RUN: Would create TXT package: {output_zip_path}"
|
||||
)
|
||||
return output_zip_path, output_txt_path
|
||||
|
||||
# Generate TXT from LRC
|
||||
self.logger.info(f"Converting LRC to TXT format: {lrc_file}")
|
||||
from lyrics_converter import LyricsConverter
|
||||
|
||||
txt_converter = LyricsConverter(output_format="txt", filepath=lrc_file)
|
||||
converted_txt = txt_converter.convert_file()
|
||||
|
||||
# Write TXT file
|
||||
if output_txt_path is None:
|
||||
# Default to same name as ZIP but with .txt extension
|
||||
output_txt_path = output_zip_path.replace(".zip", ".txt")
|
||||
|
||||
with open(output_txt_path, "w") as txt_file:
|
||||
txt_file.write(converted_txt)
|
||||
self.logger.info(f"TXT file written: {output_txt_path}")
|
||||
|
||||
# Create ZIP containing MP3 and TXT
|
||||
self.logger.info(f"Creating TXT ZIP: {output_zip_path}")
|
||||
self._create_zip_from_files(
|
||||
output_zip_path,
|
||||
[(mp3_file, os.path.basename(mp3_file)),
|
||||
(output_txt_path, os.path.basename(output_txt_path))]
|
||||
)
|
||||
|
||||
if not os.path.isfile(output_zip_path):
|
||||
raise Exception(f"Failed to create TXT ZIP file: {output_zip_path}")
|
||||
|
||||
self.logger.info(f"TXT package created: {output_zip_path}")
|
||||
return output_zip_path, output_txt_path
|
||||
|
||||
def _create_zip_from_files(
|
||||
self,
|
||||
zip_path: str,
|
||||
files: list,
|
||||
) -> None:
|
||||
"""
|
||||
Create a ZIP file from a list of files.
|
||||
|
||||
Args:
|
||||
zip_path: Path for the output ZIP file
|
||||
files: List of (file_path, archive_name) tuples
|
||||
"""
|
||||
with zipfile.ZipFile(zip_path, "w") as zipf:
|
||||
for file_path, archive_name in files:
|
||||
if os.path.isfile(file_path):
|
||||
zipf.write(file_path, archive_name)
|
||||
self.logger.debug(f"Added to ZIP: {archive_name}")
|
||||
else:
|
||||
self.logger.warning(f"File not found for ZIP: {file_path}")
|
||||
|
||||
|
||||
# Singleton instance and factory function (following existing service pattern)
|
||||
_packaging_service: Optional[PackagingService] = None
|
||||
|
||||
|
||||
def get_packaging_service(
|
||||
cdg_styles: Optional[Dict[str, Any]] = None,
|
||||
**kwargs
|
||||
) -> PackagingService:
|
||||
"""
|
||||
Get a packaging service instance.
|
||||
|
||||
Args:
|
||||
cdg_styles: CDG style configuration
|
||||
**kwargs: Additional arguments passed to PackagingService
|
||||
|
||||
Returns:
|
||||
PackagingService instance
|
||||
"""
|
||||
global _packaging_service
|
||||
|
||||
# Create new instance if settings changed
|
||||
if _packaging_service is None or cdg_styles:
|
||||
_packaging_service = PackagingService(
|
||||
cdg_styles=cdg_styles,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
return _packaging_service
|
||||
@@ -0,0 +1,65 @@
|
||||
"""SpaCy model preloader for container startup.
|
||||
|
||||
Loads SpaCy models at container startup to avoid slow loading during request processing.
|
||||
Cloud Run filesystem I/O can cause 60+ second delays when loading SpaCy models lazily.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Singleton storage for preloaded models
|
||||
_preloaded_models: dict = {}
|
||||
|
||||
|
||||
def preload_spacy_model(model_name: str = "en_core_web_sm") -> None:
|
||||
"""Preload a SpaCy model at startup.
|
||||
|
||||
Args:
|
||||
model_name: The SpaCy model to load (default: en_core_web_sm)
|
||||
"""
|
||||
global _preloaded_models
|
||||
|
||||
if model_name in _preloaded_models:
|
||||
logger.info(f"SpaCy model '{model_name}' already preloaded")
|
||||
return
|
||||
|
||||
logger.info(f"Preloading SpaCy model '{model_name}'...")
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
import spacy
|
||||
|
||||
nlp = spacy.load(model_name)
|
||||
_preloaded_models[model_name] = nlp
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
logger.info(f"SpaCy model '{model_name}' preloaded in {elapsed:.2f}s")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to preload SpaCy model '{model_name}': {e}")
|
||||
raise
|
||||
|
||||
|
||||
def get_preloaded_model(model_name: str = "en_core_web_sm") -> Optional[object]:
|
||||
"""Get a preloaded SpaCy model if available.
|
||||
|
||||
Args:
|
||||
model_name: The SpaCy model name
|
||||
|
||||
Returns:
|
||||
The preloaded SpaCy Language object, or None if not preloaded
|
||||
"""
|
||||
return _preloaded_models.get(model_name)
|
||||
|
||||
|
||||
def is_model_preloaded(model_name: str = "en_core_web_sm") -> bool:
|
||||
"""Check if a SpaCy model has been preloaded."""
|
||||
return model_name in _preloaded_models
|
||||
|
||||
|
||||
def clear_preloaded_models() -> None:
|
||||
"""Clear all preloaded models. Useful for testing."""
|
||||
global _preloaded_models
|
||||
_preloaded_models.clear()
|
||||
@@ -0,0 +1,371 @@
|
||||
"""
|
||||
Stripe service for payment processing.
|
||||
|
||||
Handles:
|
||||
- Creating checkout sessions for credit purchases
|
||||
- Processing webhook events
|
||||
- Managing customer records
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
from typing import Optional, Dict, Any, Tuple
|
||||
|
||||
import stripe
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Credit packages available for purchase
|
||||
CREDIT_PACKAGES = {
|
||||
"1_credit": {
|
||||
"credits": 1,
|
||||
"price_cents": 500, # $5.00
|
||||
"name": "1 Karaoke Credit",
|
||||
"description": "Create 1 professional karaoke video",
|
||||
},
|
||||
"3_credits": {
|
||||
"credits": 3,
|
||||
"price_cents": 1200, # $12.00 (20% discount)
|
||||
"name": "3 Karaoke Credits",
|
||||
"description": "Create 3 professional karaoke videos (Save 20%)",
|
||||
},
|
||||
"5_credits": {
|
||||
"credits": 5,
|
||||
"price_cents": 1750, # $17.50 (30% discount)
|
||||
"name": "5 Karaoke Credits",
|
||||
"description": "Create 5 professional karaoke videos (Save 30%)",
|
||||
},
|
||||
"10_credits": {
|
||||
"credits": 10,
|
||||
"price_cents": 3000, # $30.00 (40% discount)
|
||||
"name": "10 Karaoke Credits",
|
||||
"description": "Create 10 professional karaoke videos (Save 40%)",
|
||||
},
|
||||
}
|
||||
|
||||
# Done-for-you service package (not a credit package - creates a job directly)
|
||||
DONE_FOR_YOU_PACKAGE = {
|
||||
"price_cents": 1500, # $15.00
|
||||
"name": "Done For You Karaoke Video",
|
||||
"description": "Full-service karaoke video creation with 24-hour delivery",
|
||||
}
|
||||
|
||||
|
||||
class StripeService:
|
||||
"""Service for Stripe payment processing."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize Stripe with API key."""
|
||||
self.settings = get_settings()
|
||||
self.secret_key = os.getenv("STRIPE_SECRET_KEY")
|
||||
self.webhook_secret = os.getenv("STRIPE_WEBHOOK_SECRET")
|
||||
self.frontend_url = os.getenv("FRONTEND_URL", "https://gen.nomadkaraoke.com")
|
||||
# After consolidation, buy URL is the same as frontend URL
|
||||
self.buy_url = os.getenv("BUY_URL", self.frontend_url)
|
||||
|
||||
if self.secret_key:
|
||||
stripe.api_key = self.secret_key
|
||||
logger.info("Stripe initialized with API key")
|
||||
else:
|
||||
logger.warning("STRIPE_SECRET_KEY not set - payments disabled")
|
||||
|
||||
def is_configured(self) -> bool:
|
||||
"""Check if Stripe is properly configured."""
|
||||
return bool(self.secret_key)
|
||||
|
||||
def get_credit_packages(self) -> Dict[str, Dict[str, Any]]:
|
||||
"""Get available credit packages."""
|
||||
return CREDIT_PACKAGES
|
||||
|
||||
def create_checkout_session(
|
||||
self,
|
||||
package_id: str,
|
||||
user_email: str,
|
||||
success_url: Optional[str] = None,
|
||||
cancel_url: Optional[str] = None,
|
||||
) -> Tuple[bool, Optional[str], str]:
|
||||
"""
|
||||
Create a Stripe Checkout session for purchasing credits.
|
||||
|
||||
Args:
|
||||
package_id: ID of the credit package to purchase
|
||||
user_email: Email of the purchasing user
|
||||
success_url: URL to redirect to on success (optional)
|
||||
cancel_url: URL to redirect to on cancel (optional)
|
||||
|
||||
Returns:
|
||||
(success, checkout_url, message)
|
||||
"""
|
||||
if not self.is_configured():
|
||||
return False, None, "Payment processing is not configured"
|
||||
|
||||
package = CREDIT_PACKAGES.get(package_id)
|
||||
if not package:
|
||||
return False, None, f"Invalid package: {package_id}"
|
||||
|
||||
try:
|
||||
# Default URLs
|
||||
if not success_url:
|
||||
success_url = f"{self.frontend_url}/payment/success?session_id={{CHECKOUT_SESSION_ID}}"
|
||||
if not cancel_url:
|
||||
cancel_url = f"{self.buy_url}?cancelled=true"
|
||||
|
||||
# Create checkout session
|
||||
session = stripe.checkout.Session.create(
|
||||
payment_method_types=['card'],
|
||||
line_items=[{
|
||||
'price_data': {
|
||||
'currency': 'usd',
|
||||
'product_data': {
|
||||
'name': package['name'],
|
||||
'description': package['description'],
|
||||
},
|
||||
'unit_amount': package['price_cents'],
|
||||
},
|
||||
'quantity': 1,
|
||||
}],
|
||||
mode='payment',
|
||||
success_url=success_url,
|
||||
cancel_url=cancel_url,
|
||||
customer_email=user_email,
|
||||
metadata={
|
||||
'package_id': package_id,
|
||||
'credits': str(package['credits']),
|
||||
'user_email': user_email,
|
||||
},
|
||||
# Allow promotion codes
|
||||
allow_promotion_codes=True,
|
||||
)
|
||||
|
||||
logger.info(f"Created checkout session {session.id} for {user_email}, package {package_id}")
|
||||
return True, session.url, "Checkout session created"
|
||||
|
||||
except stripe.error.StripeError as e:
|
||||
logger.error(f"Stripe error creating checkout session: {e}")
|
||||
return False, None, f"Payment error: {str(e)}"
|
||||
except Exception as e:
|
||||
logger.error(f"Error creating checkout session: {e}")
|
||||
return False, None, "Failed to create checkout session"
|
||||
|
||||
def create_done_for_you_checkout_session(
|
||||
self,
|
||||
customer_email: str,
|
||||
artist: str,
|
||||
title: str,
|
||||
source_type: str = "search",
|
||||
youtube_url: Optional[str] = None,
|
||||
notes: Optional[str] = None,
|
||||
success_url: Optional[str] = None,
|
||||
cancel_url: Optional[str] = None,
|
||||
) -> Tuple[bool, Optional[str], str]:
|
||||
"""
|
||||
Create a Stripe Checkout session for a done-for-you order.
|
||||
|
||||
This is for the full-service karaoke video creation where Nomad Karaoke
|
||||
handles all the work (lyrics review, instrumental selection, etc.).
|
||||
|
||||
Args:
|
||||
customer_email: Customer's email for delivery
|
||||
artist: Song artist
|
||||
title: Song title
|
||||
source_type: Audio source type (search, youtube, upload)
|
||||
youtube_url: YouTube URL if source_type is youtube
|
||||
notes: Any special requests from customer
|
||||
success_url: URL to redirect to on success (optional)
|
||||
cancel_url: URL to redirect to on cancel (optional)
|
||||
|
||||
Returns:
|
||||
(success, checkout_url, message)
|
||||
"""
|
||||
if not self.is_configured():
|
||||
return False, None, "Payment processing is not configured"
|
||||
|
||||
try:
|
||||
# Default URLs - redirect to homepage success page
|
||||
if not success_url:
|
||||
success_url = "https://nomadkaraoke.com/order/success/?session_id={CHECKOUT_SESSION_ID}"
|
||||
if not cancel_url:
|
||||
cancel_url = "https://nomadkaraoke.com/#do-it-for-me"
|
||||
|
||||
# Build metadata for job creation after payment
|
||||
metadata = {
|
||||
'order_type': 'done_for_you',
|
||||
'customer_email': customer_email,
|
||||
'artist': artist,
|
||||
'title': title,
|
||||
'source_type': source_type,
|
||||
}
|
||||
if youtube_url:
|
||||
metadata['youtube_url'] = youtube_url
|
||||
if notes:
|
||||
# Truncate notes to fit Stripe's 500 char limit per metadata value
|
||||
metadata['notes'] = notes[:500] if len(notes) > 500 else notes
|
||||
|
||||
# Create checkout session
|
||||
session = stripe.checkout.Session.create(
|
||||
payment_method_types=['card'],
|
||||
line_items=[{
|
||||
'price_data': {
|
||||
'currency': 'usd',
|
||||
'product_data': {
|
||||
'name': DONE_FOR_YOU_PACKAGE['name'],
|
||||
'description': f"{artist} - {title}",
|
||||
},
|
||||
'unit_amount': DONE_FOR_YOU_PACKAGE['price_cents'],
|
||||
},
|
||||
'quantity': 1,
|
||||
}],
|
||||
mode='payment',
|
||||
success_url=success_url,
|
||||
cancel_url=cancel_url,
|
||||
customer_email=customer_email,
|
||||
metadata=metadata,
|
||||
allow_promotion_codes=True,
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"Created done-for-you checkout session {session.id} for {customer_email}, "
|
||||
f"song: {artist} - {title}"
|
||||
)
|
||||
return True, session.url, "Checkout session created"
|
||||
|
||||
except stripe.error.StripeError as e:
|
||||
logger.error(f"Stripe error creating done-for-you checkout session: {e}")
|
||||
return False, None, f"Payment error: {str(e)}"
|
||||
except Exception as e:
|
||||
logger.error(f"Error creating done-for-you checkout session: {e}")
|
||||
return False, None, "Failed to create checkout session"
|
||||
|
||||
def verify_webhook_signature(self, payload: bytes, signature: str) -> Tuple[bool, Optional[Dict], str]:
|
||||
"""
|
||||
Verify a Stripe webhook signature.
|
||||
|
||||
Args:
|
||||
payload: Raw request body
|
||||
signature: Stripe-Signature header value
|
||||
|
||||
Returns:
|
||||
(valid, event_data, message)
|
||||
"""
|
||||
if not self.webhook_secret:
|
||||
logger.error("STRIPE_WEBHOOK_SECRET not configured")
|
||||
return False, None, "Webhook secret not configured"
|
||||
|
||||
try:
|
||||
event = stripe.Webhook.construct_event(
|
||||
payload, signature, self.webhook_secret
|
||||
)
|
||||
return True, event, "Webhook verified"
|
||||
except stripe.error.SignatureVerificationError as e:
|
||||
logger.error(f"Invalid webhook signature: {e}")
|
||||
return False, None, "Invalid signature"
|
||||
except Exception as e:
|
||||
logger.error(f"Error verifying webhook: {e}")
|
||||
return False, None, str(e)
|
||||
|
||||
def handle_checkout_completed(self, session: Dict) -> Tuple[bool, Optional[str], int, str]:
|
||||
"""
|
||||
Handle a completed checkout session.
|
||||
|
||||
Args:
|
||||
session: Stripe checkout session object
|
||||
|
||||
Returns:
|
||||
(success, user_email, credits_to_add, message)
|
||||
"""
|
||||
try:
|
||||
metadata = session.get('metadata', {})
|
||||
user_email = metadata.get('user_email') or session.get('customer_email')
|
||||
try:
|
||||
credits = int(metadata.get('credits', 0))
|
||||
except (ValueError, TypeError):
|
||||
logger.error(f"Invalid credits metadata in session {session.get('id')}: {metadata.get('credits')}")
|
||||
return False, user_email, 0, "Invalid credit amount in session metadata"
|
||||
package_id = metadata.get('package_id')
|
||||
|
||||
if not user_email:
|
||||
logger.error(f"No user email in checkout session {session.get('id')}")
|
||||
return False, None, 0, "No user email found"
|
||||
|
||||
if credits <= 0:
|
||||
logger.error(f"Invalid credits in checkout session {session.get('id')}")
|
||||
return False, user_email, 0, "Invalid credit amount"
|
||||
|
||||
logger.info(
|
||||
f"Checkout completed: {user_email} purchased {credits} credits "
|
||||
f"(package: {package_id}, session: {session.get('id')})"
|
||||
)
|
||||
|
||||
return True, user_email, credits, f"Successfully purchased {credits} credits"
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error handling checkout completed: {e}")
|
||||
return False, None, 0, str(e)
|
||||
|
||||
def get_session(self, session_id: str) -> Optional[Dict]:
|
||||
"""
|
||||
Get a checkout session by ID.
|
||||
|
||||
Args:
|
||||
session_id: Stripe checkout session ID
|
||||
|
||||
Returns:
|
||||
Session data or None
|
||||
"""
|
||||
if not self.is_configured():
|
||||
return None
|
||||
|
||||
try:
|
||||
session = stripe.checkout.Session.retrieve(session_id)
|
||||
return dict(session)
|
||||
except Exception as e:
|
||||
logger.error(f"Error retrieving session {session_id}: {e}")
|
||||
return None
|
||||
|
||||
def create_customer(self, email: str, name: Optional[str] = None) -> Optional[str]:
|
||||
"""
|
||||
Create or get a Stripe customer for a user.
|
||||
|
||||
Args:
|
||||
email: User's email
|
||||
name: User's display name (optional)
|
||||
|
||||
Returns:
|
||||
Stripe customer ID or None
|
||||
"""
|
||||
if not self.is_configured():
|
||||
return None
|
||||
|
||||
try:
|
||||
# Check if customer already exists
|
||||
customers = stripe.Customer.list(email=email, limit=1)
|
||||
if customers.data:
|
||||
return customers.data[0].id
|
||||
|
||||
# Create new customer
|
||||
customer = stripe.Customer.create(
|
||||
email=email,
|
||||
name=name,
|
||||
metadata={'source': 'nomad_karaoke'},
|
||||
)
|
||||
logger.info(f"Created Stripe customer {customer.id} for {email}")
|
||||
return customer.id
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error creating Stripe customer: {e}")
|
||||
return None
|
||||
|
||||
|
||||
# Global instance
|
||||
_stripe_service = None
|
||||
|
||||
|
||||
def get_stripe_service() -> StripeService:
|
||||
"""Get the global Stripe service instance."""
|
||||
global _stripe_service
|
||||
if _stripe_service is None:
|
||||
_stripe_service = StripeService()
|
||||
return _stripe_service
|
||||
@@ -91,7 +91,15 @@ class StructuredFormatter(logging.Formatter):
|
||||
|
||||
# Add custom fields from 'extra' dict
|
||||
# Common fields we want to extract from log records
|
||||
custom_fields = ["job_id", "worker", "operation", "duration", "status", "error"]
|
||||
custom_fields = [
|
||||
# Job-related fields
|
||||
"job_id", "worker", "operation", "duration", "status", "error",
|
||||
# Audit logging fields (from middleware and auth)
|
||||
"request_id", "user_email", "client_ip", "latency_ms",
|
||||
"audit_type", "method", "path", "status_code", "query_string",
|
||||
"user_agent", "user_type", "is_admin", "remaining_uses",
|
||||
"auth_message", "token_provided", "token_length", "auth_header_present",
|
||||
]
|
||||
for field in custom_fields:
|
||||
value = getattr(record, field, None)
|
||||
if value is not None:
|
||||
|
||||
@@ -0,0 +1,330 @@
|
||||
"""
|
||||
Template service for managing email templates stored in GCS.
|
||||
|
||||
Templates are stored in GCS bucket and rendered with job-specific variables.
|
||||
This allows updating email content without code deployment.
|
||||
|
||||
Template locations:
|
||||
- gs://{bucket}/templates/job-completion.txt - Job completion email (plain text)
|
||||
- gs://{bucket}/templates/action-needed-lyrics.txt - Lyrics review reminder
|
||||
- gs://{bucket}/templates/action-needed-instrumental.txt - Instrumental selection reminder
|
||||
"""
|
||||
import logging
|
||||
import re
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Default templates (fallback if GCS fetch fails)
|
||||
DEFAULT_JOB_COMPLETION_TEMPLATE = """Hi {name},
|
||||
|
||||
Thanks for your order!
|
||||
|
||||
Here's the link for the karaoke video published to YouTube:
|
||||
{youtube_url}
|
||||
|
||||
Here's the dropbox folder with all the finished files and source files, including:
|
||||
- "(Final Karaoke Lossless).mkv": combined karaoke video in 4k H264 with lossless FLAC audio
|
||||
- "(Final Karaoke).mp4": combined karaoke video with title/end screen in 4k H264/AAC
|
||||
- "(Final Karaoke 720p).mp4": combined karaoke video in 720p H264/AAC (smaller file for older systems)
|
||||
- "(With Vocals).mp4": sing along video in 4k H264/AAC with original vocals
|
||||
- "(Karaoke).mov": karaoke video output from MidiCo (no title/end screen)
|
||||
- "(Title).mov"/"(End).mov": title card and end screen videos
|
||||
- "(Final Karaoke CDG).zip": CDG+MP3 format for older/commercial karaoke systems
|
||||
- "(Final Karaoke TXT).zip": TXT+MP3 format for Power Karaoke
|
||||
- stems/*.flac: various separated instrumental and vocal audio stems in lossless format
|
||||
- lyrics/*.txt song lyrics from various sources in plain text format
|
||||
|
||||
{dropbox_url}
|
||||
|
||||
Let me know if anything isn't perfect and I'll happily tweak / fix, or if you need it in any other format I can probably convert it for you!
|
||||
|
||||
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
|
||||
{feedback_url}
|
||||
|
||||
Thanks again and have a great day!
|
||||
-Andrew
|
||||
"""
|
||||
|
||||
DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE = """Hi {name},
|
||||
|
||||
Your karaoke video for "{artist} - {title}" is ready for lyrics review!
|
||||
|
||||
Our system has transcribed and synchronized the lyrics, but they may need some corrections. Please review and make any corrections necessary so we can finish generating your video.
|
||||
|
||||
Review your lyrics here:
|
||||
{review_url}
|
||||
|
||||
This usually takes just a few minutes.
|
||||
|
||||
Thanks!
|
||||
-Andrew
|
||||
"""
|
||||
|
||||
DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE = """Hi {name},
|
||||
|
||||
Your karaoke video for "{artist} - {title}" is almost done!
|
||||
|
||||
We've separated the audio into different versions. Please select which instrumental track you'd like to use for the final video.
|
||||
|
||||
Select your instrumental here:
|
||||
{instrumental_url}
|
||||
|
||||
Thanks!
|
||||
-Andrew
|
||||
"""
|
||||
|
||||
|
||||
class TemplateService:
|
||||
"""
|
||||
Service for fetching and rendering email templates from GCS.
|
||||
|
||||
Templates support the following variables:
|
||||
- {name} - User's display name or "there" if unknown
|
||||
- {youtube_url} - YouTube video URL
|
||||
- {dropbox_url} - Dropbox folder URL
|
||||
- {artist} - Artist name
|
||||
- {title} - Song title
|
||||
- {job_id} - Job ID
|
||||
- {review_url} - Lyrics review URL
|
||||
- {instrumental_url} - Instrumental selection URL
|
||||
- {feedback_url} - Feedback form URL
|
||||
"""
|
||||
|
||||
TEMPLATE_PREFIX = "templates/"
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize template service."""
|
||||
self.settings = get_settings()
|
||||
self._storage_client = None
|
||||
self._bucket = None
|
||||
|
||||
@property
|
||||
def storage_client(self):
|
||||
"""Lazy-initialize storage client."""
|
||||
if self._storage_client is None:
|
||||
from google.cloud import storage
|
||||
self._storage_client = storage.Client(project=self.settings.google_cloud_project)
|
||||
return self._storage_client
|
||||
|
||||
@property
|
||||
def bucket(self):
|
||||
"""Get the GCS bucket for templates."""
|
||||
if self._bucket is None:
|
||||
self._bucket = self.storage_client.bucket(self.settings.gcs_bucket_name)
|
||||
return self._bucket
|
||||
|
||||
def _fetch_template_from_gcs(self, template_name: str) -> Optional[str]:
|
||||
"""
|
||||
Fetch a template from GCS.
|
||||
|
||||
Args:
|
||||
template_name: Name of template file (e.g., "job-completion.txt")
|
||||
|
||||
Returns:
|
||||
Template content as string, or None if not found
|
||||
"""
|
||||
blob_path = f"{self.TEMPLATE_PREFIX}{template_name}"
|
||||
try:
|
||||
blob = self.bucket.blob(blob_path)
|
||||
if blob.exists():
|
||||
content = blob.download_as_text()
|
||||
logger.debug(f"Fetched template from GCS: {blob_path}")
|
||||
return content
|
||||
else:
|
||||
logger.warning(f"Template not found in GCS: {blob_path}")
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to fetch template {blob_path}: {e}")
|
||||
return None
|
||||
|
||||
def get_job_completion_template(self) -> str:
|
||||
"""Get the job completion email template."""
|
||||
template = self._fetch_template_from_gcs("job-completion.txt")
|
||||
if template is None:
|
||||
logger.info("Using default job completion template")
|
||||
return DEFAULT_JOB_COMPLETION_TEMPLATE
|
||||
return template
|
||||
|
||||
def get_action_needed_lyrics_template(self) -> str:
|
||||
"""Get the lyrics review reminder template."""
|
||||
template = self._fetch_template_from_gcs("action-needed-lyrics.txt")
|
||||
if template is None:
|
||||
logger.info("Using default lyrics reminder template")
|
||||
return DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE
|
||||
return template
|
||||
|
||||
def get_action_needed_instrumental_template(self) -> str:
|
||||
"""Get the instrumental selection reminder template."""
|
||||
template = self._fetch_template_from_gcs("action-needed-instrumental.txt")
|
||||
if template is None:
|
||||
logger.info("Using default instrumental reminder template")
|
||||
return DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE
|
||||
return template
|
||||
|
||||
def render_template(self, template: str, variables: Dict[str, Any]) -> str:
|
||||
"""
|
||||
Render a template with the given variables.
|
||||
|
||||
Missing variables are replaced with empty strings.
|
||||
Handles conditional sections like feedback URL.
|
||||
|
||||
Args:
|
||||
template: Template string with {variable} placeholders
|
||||
variables: Dictionary of variable values
|
||||
|
||||
Returns:
|
||||
Rendered template string
|
||||
"""
|
||||
result = template
|
||||
|
||||
# Handle feedback URL section - remove if not provided
|
||||
if not variables.get("feedback_url"):
|
||||
# Remove feedback section (lines containing feedback_url placeholder and surrounding text)
|
||||
result = re.sub(
|
||||
r'\n*If you have a moment.*?\{feedback_url\}\n*',
|
||||
'\n',
|
||||
result,
|
||||
flags=re.DOTALL
|
||||
)
|
||||
variables["feedback_url"] = ""
|
||||
|
||||
# Replace all variables
|
||||
for key, value in variables.items():
|
||||
placeholder = "{" + key + "}"
|
||||
result = result.replace(placeholder, str(value) if value else "")
|
||||
|
||||
# Clean up any remaining unreplaced placeholders
|
||||
result = re.sub(r'\{[a-z_]+\}', '', result)
|
||||
|
||||
return result.strip()
|
||||
|
||||
def render_job_completion(
|
||||
self,
|
||||
name: Optional[str] = None,
|
||||
youtube_url: Optional[str] = None,
|
||||
dropbox_url: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
job_id: Optional[str] = None,
|
||||
feedback_url: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Render the job completion email template.
|
||||
|
||||
Args:
|
||||
name: User's display name (defaults to "there")
|
||||
youtube_url: YouTube video URL
|
||||
dropbox_url: Dropbox folder URL
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
job_id: Job ID
|
||||
feedback_url: Feedback form URL (optional)
|
||||
|
||||
Returns:
|
||||
Rendered email content
|
||||
"""
|
||||
template = self.get_job_completion_template()
|
||||
variables = {
|
||||
"name": name or "there",
|
||||
"youtube_url": youtube_url or "[YouTube URL not available]",
|
||||
"dropbox_url": dropbox_url or "[Dropbox URL not available]",
|
||||
"artist": artist or "Unknown Artist",
|
||||
"title": title or "Unknown Title",
|
||||
"job_id": job_id or "",
|
||||
"feedback_url": feedback_url,
|
||||
}
|
||||
return self.render_template(template, variables)
|
||||
|
||||
def render_action_needed_lyrics(
|
||||
self,
|
||||
name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
review_url: str = "",
|
||||
) -> str:
|
||||
"""
|
||||
Render the lyrics review reminder template.
|
||||
|
||||
Args:
|
||||
name: User's display name
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
review_url: Lyrics review URL
|
||||
|
||||
Returns:
|
||||
Rendered email content
|
||||
"""
|
||||
template = self.get_action_needed_lyrics_template()
|
||||
variables = {
|
||||
"name": name or "there",
|
||||
"artist": artist or "Unknown Artist",
|
||||
"title": title or "Unknown Title",
|
||||
"review_url": review_url,
|
||||
}
|
||||
return self.render_template(template, variables)
|
||||
|
||||
def render_action_needed_instrumental(
|
||||
self,
|
||||
name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
instrumental_url: str = "",
|
||||
) -> str:
|
||||
"""
|
||||
Render the instrumental selection reminder template.
|
||||
|
||||
Args:
|
||||
name: User's display name
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
instrumental_url: Instrumental selection URL
|
||||
|
||||
Returns:
|
||||
Rendered email content
|
||||
"""
|
||||
template = self.get_action_needed_instrumental_template()
|
||||
variables = {
|
||||
"name": name or "there",
|
||||
"artist": artist or "Unknown Artist",
|
||||
"title": title or "Unknown Title",
|
||||
"instrumental_url": instrumental_url,
|
||||
}
|
||||
return self.render_template(template, variables)
|
||||
|
||||
def upload_template(self, template_name: str, content: str) -> bool:
|
||||
"""
|
||||
Upload a template to GCS.
|
||||
|
||||
Args:
|
||||
template_name: Name of template file (e.g., "job-completion.txt")
|
||||
content: Template content
|
||||
|
||||
Returns:
|
||||
True if upload successful
|
||||
"""
|
||||
blob_path = f"{self.TEMPLATE_PREFIX}{template_name}"
|
||||
try:
|
||||
blob = self.bucket.blob(blob_path)
|
||||
blob.upload_from_string(content, content_type="text/plain")
|
||||
logger.info(f"Uploaded template to GCS: {blob_path}")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to upload template {blob_path}: {e}")
|
||||
return False
|
||||
|
||||
|
||||
# Global instance
|
||||
_template_service: Optional[TemplateService] = None
|
||||
|
||||
|
||||
def get_template_service() -> TemplateService:
|
||||
"""Get the global template service instance."""
|
||||
global _template_service
|
||||
if _template_service is None:
|
||||
_template_service = TemplateService()
|
||||
return _template_service
|
||||
@@ -0,0 +1,721 @@
|
||||
"""
|
||||
User service for authentication, credits, and user management.
|
||||
|
||||
Handles:
|
||||
- Magic link authentication (send, verify)
|
||||
- Session management
|
||||
- Credit operations (add, deduct, refund)
|
||||
- User CRUD operations
|
||||
"""
|
||||
import logging
|
||||
import secrets
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Optional, List, Tuple
|
||||
|
||||
from google.cloud import firestore
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
from google.cloud.firestore_v1 import Increment
|
||||
|
||||
from backend.config import get_settings
|
||||
from backend.models.user import (
|
||||
User,
|
||||
UserRole,
|
||||
UserPublic,
|
||||
MagicLinkToken,
|
||||
Session,
|
||||
CreditTransaction,
|
||||
)
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Collection names
|
||||
USERS_COLLECTION = "gen_users"
|
||||
MAGIC_LINKS_COLLECTION = "magic_links"
|
||||
SESSIONS_COLLECTION = "sessions"
|
||||
PROCESSED_STRIPE_SESSIONS_COLLECTION = "processed_stripe_sessions"
|
||||
|
||||
# Token/session configuration
|
||||
MAGIC_LINK_EXPIRY_MINUTES = 15
|
||||
SESSION_EXPIRY_DAYS = 7
|
||||
SESSION_ABSOLUTE_EXPIRY_DAYS = 30
|
||||
MAX_CREDIT_TRANSACTIONS = 100 # Keep last N transactions
|
||||
|
||||
|
||||
class UserService:
|
||||
"""Service for user management and authentication."""
|
||||
|
||||
# Number of free credits granted to new users
|
||||
NEW_USER_FREE_CREDITS = 1
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize user service with Firestore client."""
|
||||
self.settings = get_settings()
|
||||
self.db = firestore.Client(project=self.settings.google_cloud_project)
|
||||
|
||||
# =========================================================================
|
||||
# User CRUD Operations
|
||||
# =========================================================================
|
||||
|
||||
def get_user(self, email: str) -> Optional[User]:
|
||||
"""Get a user by email."""
|
||||
try:
|
||||
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
return None
|
||||
|
||||
return User(**doc.to_dict())
|
||||
except Exception:
|
||||
logger.exception(f"Error getting user {email}")
|
||||
return None
|
||||
|
||||
def get_or_create_user(self, email: str) -> User:
|
||||
"""
|
||||
Get existing user or create a new one.
|
||||
|
||||
New users receive a welcome credit to try the service.
|
||||
"""
|
||||
email = email.lower()
|
||||
user = self.get_user(email)
|
||||
|
||||
if user:
|
||||
return user
|
||||
|
||||
# Create new user with welcome credit
|
||||
welcome_credit = self.NEW_USER_FREE_CREDITS
|
||||
welcome_transaction = CreditTransaction(
|
||||
id=str(uuid.uuid4()),
|
||||
amount=welcome_credit,
|
||||
reason="welcome_credit",
|
||||
)
|
||||
|
||||
user = User(
|
||||
email=email,
|
||||
credits=welcome_credit,
|
||||
credit_transactions=[welcome_transaction],
|
||||
)
|
||||
self._save_user(user)
|
||||
logger.info(f"Created new user: {email} with {welcome_credit} welcome credit(s)")
|
||||
return user
|
||||
|
||||
def _save_user(self, user: User) -> None:
|
||||
"""Save user to Firestore."""
|
||||
try:
|
||||
user.updated_at = datetime.utcnow()
|
||||
doc_ref = self.db.collection(USERS_COLLECTION).document(user.email.lower())
|
||||
doc_ref.set(user.model_dump(mode='json'))
|
||||
except Exception:
|
||||
logger.exception(f"Error saving user {user.email}")
|
||||
raise
|
||||
|
||||
def update_user(self, email: str, **updates) -> Optional[User]:
|
||||
"""Update user fields."""
|
||||
try:
|
||||
updates['updated_at'] = datetime.utcnow()
|
||||
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
|
||||
doc_ref.update(updates)
|
||||
return self.get_user(email)
|
||||
except Exception:
|
||||
logger.exception(f"Error updating user {email}")
|
||||
return None
|
||||
|
||||
def list_users(self, limit: int = 100, include_inactive: bool = False) -> List[User]:
|
||||
"""List all users (admin only)."""
|
||||
try:
|
||||
query = self.db.collection(USERS_COLLECTION)
|
||||
|
||||
if not include_inactive:
|
||||
query = query.where(filter=FieldFilter('is_active', '==', True))
|
||||
|
||||
query = query.order_by('created_at', direction=firestore.Query.DESCENDING)
|
||||
query = query.limit(limit)
|
||||
|
||||
docs = query.stream()
|
||||
return [User(**doc.to_dict()) for doc in docs]
|
||||
except Exception:
|
||||
logger.exception("Error listing users")
|
||||
return []
|
||||
|
||||
def get_user_public(self, email: str) -> Optional[UserPublic]:
|
||||
"""Get public user info."""
|
||||
user = self.get_user(email)
|
||||
if not user:
|
||||
return None
|
||||
|
||||
return UserPublic(
|
||||
email=user.email,
|
||||
role=user.role,
|
||||
credits=user.credits,
|
||||
display_name=user.display_name,
|
||||
total_jobs_created=user.total_jobs_created,
|
||||
total_jobs_completed=user.total_jobs_completed,
|
||||
)
|
||||
|
||||
# =========================================================================
|
||||
# Magic Link Authentication
|
||||
# =========================================================================
|
||||
|
||||
def create_magic_link(
|
||||
self,
|
||||
email: str,
|
||||
ip_address: Optional[str] = None,
|
||||
user_agent: Optional[str] = None
|
||||
) -> MagicLinkToken:
|
||||
"""
|
||||
Create a magic link token for email authentication.
|
||||
|
||||
Returns the token object. The actual sending of the email
|
||||
should be handled by the caller (or an email service).
|
||||
"""
|
||||
email = email.lower()
|
||||
|
||||
# Ensure user exists
|
||||
self.get_or_create_user(email)
|
||||
|
||||
# Generate secure token
|
||||
token = secrets.token_urlsafe(32)
|
||||
|
||||
magic_link = MagicLinkToken(
|
||||
token=token,
|
||||
email=email,
|
||||
expires_at=datetime.utcnow() + timedelta(minutes=MAGIC_LINK_EXPIRY_MINUTES),
|
||||
ip_address=ip_address,
|
||||
user_agent=user_agent,
|
||||
)
|
||||
|
||||
# Save to Firestore
|
||||
doc_ref = self.db.collection(MAGIC_LINKS_COLLECTION).document(token)
|
||||
doc_ref.set(magic_link.model_dump(mode='json'))
|
||||
|
||||
logger.info(f"Created magic link for {email}")
|
||||
return magic_link
|
||||
|
||||
def verify_magic_link(self, token: str) -> Tuple[bool, Optional[User], str]:
|
||||
"""
|
||||
Verify a magic link token using a Firestore transaction to prevent race conditions.
|
||||
|
||||
Returns:
|
||||
(success, user, message)
|
||||
"""
|
||||
try:
|
||||
doc_ref = self.db.collection(MAGIC_LINKS_COLLECTION).document(token)
|
||||
|
||||
@firestore.transactional
|
||||
def verify_in_transaction(transaction):
|
||||
"""Atomically verify and mark magic link as used."""
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
|
||||
if not doc.exists:
|
||||
return False, None, "Invalid or expired link"
|
||||
|
||||
magic_link = MagicLinkToken(**doc.to_dict())
|
||||
|
||||
# Check if already used
|
||||
if magic_link.used:
|
||||
return False, None, "This link has already been used"
|
||||
|
||||
# Check expiry
|
||||
if datetime.utcnow() > magic_link.expires_at:
|
||||
return False, None, "This link has expired"
|
||||
|
||||
# Mark as used atomically within transaction
|
||||
transaction.update(doc_ref, {
|
||||
'used': True,
|
||||
'used_at': datetime.utcnow()
|
||||
})
|
||||
|
||||
return True, magic_link.email, "Success"
|
||||
|
||||
# Execute the transaction
|
||||
transaction = self.db.transaction()
|
||||
success, email_or_error, message = verify_in_transaction(transaction)
|
||||
|
||||
if not success:
|
||||
return False, None, message
|
||||
|
||||
# Get user and mark email as verified (outside transaction)
|
||||
user = self.get_user(email_or_error)
|
||||
if user:
|
||||
self.update_user(
|
||||
email_or_error,
|
||||
email_verified=True,
|
||||
last_login_at=datetime.utcnow()
|
||||
)
|
||||
user = self.get_user(email_or_error) # Refresh
|
||||
|
||||
logger.info(f"Magic link verified for {email_or_error}")
|
||||
return True, user, "Successfully authenticated"
|
||||
|
||||
except Exception:
|
||||
logger.exception("Error verifying magic link")
|
||||
return False, None, "An error occurred during verification"
|
||||
|
||||
# =========================================================================
|
||||
# Session Management
|
||||
# =========================================================================
|
||||
|
||||
def create_session(
|
||||
self,
|
||||
user_email: str,
|
||||
ip_address: Optional[str] = None,
|
||||
user_agent: Optional[str] = None
|
||||
) -> Session:
|
||||
"""Create a new session for an authenticated user."""
|
||||
token = secrets.token_urlsafe(32)
|
||||
token_prefix = token[:12]
|
||||
|
||||
session = Session(
|
||||
token=token,
|
||||
user_email=user_email.lower(),
|
||||
expires_at=datetime.utcnow() + timedelta(days=SESSION_ABSOLUTE_EXPIRY_DAYS),
|
||||
ip_address=ip_address,
|
||||
user_agent=user_agent,
|
||||
)
|
||||
|
||||
# Serialize and write to Firestore
|
||||
session_data = session.model_dump(mode='json')
|
||||
doc_ref = self.db.collection(SESSIONS_COLLECTION).document(token)
|
||||
doc_ref.set(session_data)
|
||||
|
||||
# Verify the write succeeded by reading back
|
||||
verify_doc = doc_ref.get()
|
||||
if not verify_doc.exists:
|
||||
logger.error(f"Session write verification FAILED for {user_email}: {token_prefix}...")
|
||||
raise RuntimeError(f"Failed to persist session for {user_email}")
|
||||
|
||||
logger.info(f"Created and verified session for {user_email}: {token_prefix}... (expires: {session.expires_at})")
|
||||
|
||||
return session
|
||||
|
||||
def validate_session(self, token: str) -> Tuple[bool, Optional[User], str]:
|
||||
"""
|
||||
Validate a session token.
|
||||
|
||||
Returns:
|
||||
(valid, user, message)
|
||||
"""
|
||||
# Log token info for debugging (only prefix for security)
|
||||
token_prefix = token[:12] if token and len(token) >= 12 else token
|
||||
logger.debug(f"Validating session token: {token_prefix}... (len={len(token) if token else 0})")
|
||||
|
||||
try:
|
||||
doc_ref = self.db.collection(SESSIONS_COLLECTION).document(token)
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
logger.warning(f"Session not found in Firestore: {token_prefix}...")
|
||||
return False, None, "Invalid session"
|
||||
|
||||
raw_data = doc.to_dict()
|
||||
logger.debug(f"Session data found for {token_prefix}...: user_email={raw_data.get('user_email')}, is_active={raw_data.get('is_active')}")
|
||||
|
||||
session = Session(**raw_data)
|
||||
|
||||
# Check if active
|
||||
if not session.is_active:
|
||||
logger.warning(f"Session revoked for {token_prefix}... (user: {session.user_email})")
|
||||
return False, None, "Session has been revoked"
|
||||
|
||||
# Use timezone-aware datetime for all comparisons
|
||||
# Firestore returns timezone-aware datetimes, so we must use aware datetimes
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
# Normalize session datetimes to be timezone-aware for comparison
|
||||
# (handles legacy naive datetimes that may exist in Firestore)
|
||||
expires_at = session.expires_at
|
||||
if expires_at.tzinfo is None:
|
||||
expires_at = expires_at.replace(tzinfo=timezone.utc)
|
||||
|
||||
last_activity = session.last_activity_at
|
||||
if last_activity.tzinfo is None:
|
||||
last_activity = last_activity.replace(tzinfo=timezone.utc)
|
||||
|
||||
# Check expiry
|
||||
if now > expires_at:
|
||||
logger.warning(f"Session expired for {token_prefix}... (user: {session.user_email}, expired_at: {expires_at}, now: {now})")
|
||||
return False, None, "Session has expired"
|
||||
|
||||
# Check inactivity (7 days)
|
||||
inactivity_limit = now - timedelta(days=SESSION_EXPIRY_DAYS)
|
||||
if last_activity < inactivity_limit:
|
||||
logger.warning(f"Session inactive for {token_prefix}... (user: {session.user_email}, last_activity: {last_activity}, limit: {inactivity_limit})")
|
||||
return False, None, "Session expired due to inactivity"
|
||||
|
||||
# Update last activity (use timezone-aware datetime)
|
||||
doc_ref.update({'last_activity_at': now})
|
||||
|
||||
# Get user
|
||||
user = self.get_user(session.user_email)
|
||||
if not user:
|
||||
logger.warning(f"User not found for session {token_prefix}...: {session.user_email}")
|
||||
return False, None, "User not found"
|
||||
|
||||
if not user.is_active:
|
||||
logger.warning(f"User account disabled for session {token_prefix}...: {session.user_email}")
|
||||
return False, None, "User account is disabled"
|
||||
|
||||
logger.debug(f"Session valid for {token_prefix}... (user: {user.email}, credits: {user.credits})")
|
||||
return True, user, "Valid session"
|
||||
|
||||
except Exception:
|
||||
logger.exception(f"Error validating session {token_prefix}...")
|
||||
return False, None, "An error occurred during validation"
|
||||
|
||||
def revoke_session(self, token: str) -> bool:
|
||||
"""Revoke a session (logout)."""
|
||||
try:
|
||||
doc_ref = self.db.collection(SESSIONS_COLLECTION).document(token)
|
||||
doc_ref.update({'is_active': False})
|
||||
logger.info("Session revoked")
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("Error revoking session")
|
||||
return False
|
||||
|
||||
def revoke_all_sessions(self, user_email: str) -> int:
|
||||
"""Revoke all sessions for a user."""
|
||||
try:
|
||||
query = self.db.collection(SESSIONS_COLLECTION).where(
|
||||
filter=FieldFilter('user_email', '==', user_email.lower())
|
||||
).where(
|
||||
filter=FieldFilter('is_active', '==', True)
|
||||
)
|
||||
|
||||
count = 0
|
||||
for doc in query.stream():
|
||||
doc.reference.update({'is_active': False})
|
||||
count += 1
|
||||
|
||||
logger.info(f"Revoked {count} sessions for {user_email}")
|
||||
return count
|
||||
except Exception:
|
||||
logger.exception("Error revoking sessions")
|
||||
return 0
|
||||
|
||||
def list_sessions_for_user(
|
||||
self,
|
||||
user_email: str,
|
||||
include_revoked: bool = False,
|
||||
limit: int = 50
|
||||
) -> List[Session]:
|
||||
"""
|
||||
List all sessions for a user.
|
||||
|
||||
Args:
|
||||
user_email: User's email address
|
||||
include_revoked: If True, include revoked/inactive sessions
|
||||
limit: Maximum number of sessions to return
|
||||
|
||||
Returns:
|
||||
List of Session objects, ordered by created_at descending
|
||||
"""
|
||||
try:
|
||||
query = self.db.collection(SESSIONS_COLLECTION).where(
|
||||
filter=FieldFilter('user_email', '==', user_email.lower())
|
||||
)
|
||||
|
||||
if not include_revoked:
|
||||
query = query.where(filter=FieldFilter('is_active', '==', True))
|
||||
|
||||
query = query.order_by('created_at', direction=firestore.Query.DESCENDING)
|
||||
query = query.limit(limit)
|
||||
|
||||
sessions = []
|
||||
for doc in query.stream():
|
||||
try:
|
||||
sessions.append(Session(**doc.to_dict()))
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to parse session document: {e}")
|
||||
|
||||
logger.debug(f"Found {len(sessions)} sessions for {user_email}")
|
||||
return sessions
|
||||
except Exception:
|
||||
logger.exception(f"Error listing sessions for {user_email}")
|
||||
return []
|
||||
|
||||
# =========================================================================
|
||||
# Credit Operations
|
||||
# =========================================================================
|
||||
|
||||
def add_credits(
|
||||
self,
|
||||
email: str,
|
||||
amount: int,
|
||||
reason: str,
|
||||
job_id: Optional[str] = None,
|
||||
stripe_session_id: Optional[str] = None,
|
||||
admin_email: Optional[str] = None
|
||||
) -> Tuple[bool, int, str]:
|
||||
"""
|
||||
Add credits to a user account.
|
||||
|
||||
Returns:
|
||||
(success, new_balance, message)
|
||||
"""
|
||||
try:
|
||||
# For Stripe sessions, atomically check and mark as processed
|
||||
# Uses create() which fails if document exists - ensures idempotency
|
||||
if stripe_session_id:
|
||||
doc_ref = self.db.collection(PROCESSED_STRIPE_SESSIONS_COLLECTION).document(stripe_session_id)
|
||||
try:
|
||||
doc_ref.create({
|
||||
'stripe_session_id': stripe_session_id,
|
||||
'email': email.lower(),
|
||||
'amount': amount,
|
||||
'processed_at': datetime.utcnow()
|
||||
})
|
||||
except Exception:
|
||||
# Document already exists - this session was already processed
|
||||
logger.info(f"Stripe session {stripe_session_id} already processed (idempotent skip)")
|
||||
return False, 0, "Session already processed"
|
||||
|
||||
email = email.lower()
|
||||
user = self.get_or_create_user(email)
|
||||
|
||||
# Create transaction record
|
||||
transaction = CreditTransaction(
|
||||
id=str(uuid.uuid4()),
|
||||
amount=amount,
|
||||
reason=reason,
|
||||
job_id=job_id,
|
||||
stripe_session_id=stripe_session_id,
|
||||
created_by=admin_email,
|
||||
)
|
||||
|
||||
# Add to transaction history (keep last N)
|
||||
transactions = user.credit_transactions[-MAX_CREDIT_TRANSACTIONS + 1:]
|
||||
transactions.append(transaction)
|
||||
|
||||
# Update user
|
||||
new_balance = user.credits + amount
|
||||
self.update_user(
|
||||
email,
|
||||
credits=new_balance,
|
||||
credit_transactions=[t.model_dump(mode='json') for t in transactions]
|
||||
)
|
||||
|
||||
logger.info(f"Added {amount} credits to {email} ({reason}). New balance: {new_balance}")
|
||||
return True, new_balance, f"Added {amount} credits"
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"Error adding credits to {email}")
|
||||
return False, 0, f"Failed to add credits: {e}"
|
||||
|
||||
def deduct_credit(
|
||||
self,
|
||||
email: str,
|
||||
job_id: str,
|
||||
reason: str = "job_creation"
|
||||
) -> Tuple[bool, int, str]:
|
||||
"""
|
||||
Deduct one credit from a user account.
|
||||
|
||||
Uses Firestore transaction to prevent race conditions.
|
||||
|
||||
Returns:
|
||||
(success, remaining_credits, message)
|
||||
"""
|
||||
try:
|
||||
email = email.lower()
|
||||
doc_ref = self.db.collection(USERS_COLLECTION).document(email)
|
||||
|
||||
@firestore.transactional
|
||||
def deduct_in_transaction(transaction):
|
||||
"""Atomically check and deduct credit."""
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
|
||||
if not doc.exists:
|
||||
return False, 0, "User not found"
|
||||
|
||||
user_data = doc.to_dict()
|
||||
current_credits = user_data.get('credits', 0)
|
||||
|
||||
if current_credits <= 0:
|
||||
return False, 0, "Insufficient credits"
|
||||
|
||||
# Create transaction record
|
||||
credit_txn = CreditTransaction(
|
||||
id=str(uuid.uuid4()),
|
||||
amount=-1,
|
||||
reason=reason,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
# Get existing transactions and add new one
|
||||
existing_transactions = user_data.get('credit_transactions', [])
|
||||
# Keep last N-1 transactions to make room for new one
|
||||
transactions = existing_transactions[-(MAX_CREDIT_TRANSACTIONS - 1):]
|
||||
transactions.append(credit_txn.model_dump(mode='json'))
|
||||
|
||||
# Calculate new values
|
||||
new_balance = current_credits - 1
|
||||
total_jobs = user_data.get('total_jobs_created', 0) + 1
|
||||
|
||||
# Update atomically within transaction
|
||||
transaction.update(doc_ref, {
|
||||
'credits': new_balance,
|
||||
'credit_transactions': transactions,
|
||||
'total_jobs_created': total_jobs,
|
||||
'updated_at': datetime.utcnow()
|
||||
})
|
||||
|
||||
return True, new_balance, f"Credit deducted. {new_balance} remaining"
|
||||
|
||||
# Execute the transaction
|
||||
fs_transaction = self.db.transaction()
|
||||
success, new_balance, message = deduct_in_transaction(fs_transaction)
|
||||
|
||||
if success:
|
||||
logger.info(f"Deducted 1 credit from {email} for job {job_id}. Remaining: {new_balance}")
|
||||
|
||||
return success, new_balance, message
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"Error deducting credit from {email}")
|
||||
return False, 0, f"Failed to deduct credit: {e}"
|
||||
|
||||
def refund_credit(
|
||||
self,
|
||||
email: str,
|
||||
job_id: str,
|
||||
reason: str = "job_failed"
|
||||
) -> Tuple[bool, int, str]:
|
||||
"""
|
||||
Refund one credit to a user account (e.g., after job failure).
|
||||
|
||||
Returns:
|
||||
(success, new_balance, message)
|
||||
"""
|
||||
return self.add_credits(email, 1, reason, job_id=job_id)
|
||||
|
||||
def check_credits(self, email: str) -> int:
|
||||
"""Check user's credit balance."""
|
||||
user = self.get_user(email)
|
||||
return user.credits if user else 0
|
||||
|
||||
def has_credits(self, email: str) -> bool:
|
||||
"""Check if user has at least one credit."""
|
||||
return self.check_credits(email) > 0
|
||||
|
||||
def is_stripe_session_processed(self, stripe_session_id: str) -> bool:
|
||||
"""
|
||||
Check if a Stripe session ID has already been processed.
|
||||
|
||||
Used to ensure idempotency of webhook processing - prevents
|
||||
duplicate credit additions if Stripe sends the same webhook twice.
|
||||
|
||||
Uses a dedicated collection for O(1) lookup instead of scanning all users.
|
||||
|
||||
Args:
|
||||
stripe_session_id: The Stripe checkout session ID
|
||||
|
||||
Returns:
|
||||
True if this session was already processed
|
||||
"""
|
||||
try:
|
||||
doc_ref = self.db.collection(PROCESSED_STRIPE_SESSIONS_COLLECTION).document(stripe_session_id)
|
||||
doc = doc_ref.get()
|
||||
if doc.exists:
|
||||
logger.info(f"Stripe session {stripe_session_id} already processed")
|
||||
return True
|
||||
return False
|
||||
except Exception:
|
||||
logger.exception(f"Error checking if stripe session {stripe_session_id} was processed")
|
||||
# On error, return False to allow processing (better to risk duplicate than block)
|
||||
return False
|
||||
|
||||
def _mark_stripe_session_processed(self, stripe_session_id: str, email: str, amount: int) -> None:
|
||||
"""
|
||||
Mark a Stripe session as processed.
|
||||
|
||||
Called after successfully adding credits from a Stripe webhook.
|
||||
|
||||
Args:
|
||||
stripe_session_id: The Stripe checkout session ID
|
||||
email: User email who received the credits
|
||||
amount: Number of credits added
|
||||
"""
|
||||
try:
|
||||
doc_ref = self.db.collection(PROCESSED_STRIPE_SESSIONS_COLLECTION).document(stripe_session_id)
|
||||
doc_ref.set({
|
||||
'stripe_session_id': stripe_session_id,
|
||||
'email': email,
|
||||
'amount': amount,
|
||||
'processed_at': datetime.utcnow()
|
||||
})
|
||||
except Exception:
|
||||
# Log but don't fail - the credits were already added
|
||||
logger.exception(f"Error marking stripe session {stripe_session_id} as processed")
|
||||
|
||||
# =========================================================================
|
||||
# Admin Operations
|
||||
# =========================================================================
|
||||
|
||||
def set_user_role(self, email: str, role: UserRole, admin_email: str) -> bool:
|
||||
"""Set a user's role (admin only)."""
|
||||
try:
|
||||
user = self.get_user(email)
|
||||
if not user:
|
||||
return False
|
||||
|
||||
self.update_user(email, role=role.value)
|
||||
logger.info(f"Admin {admin_email} set role for {email} to {role.value}")
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("Error setting user role")
|
||||
return False
|
||||
|
||||
def disable_user(self, email: str, admin_email: str) -> bool:
|
||||
"""Disable a user account (admin only)."""
|
||||
try:
|
||||
self.update_user(email, is_active=False)
|
||||
self.revoke_all_sessions(email)
|
||||
logger.info(f"Admin {admin_email} disabled user {email}")
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("Error disabling user")
|
||||
return False
|
||||
|
||||
def enable_user(self, email: str, admin_email: str) -> bool:
|
||||
"""Enable a user account (admin only)."""
|
||||
try:
|
||||
self.update_user(email, is_active=True)
|
||||
logger.info(f"Admin {admin_email} enabled user {email}")
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("Error enabling user")
|
||||
return False
|
||||
|
||||
def increment_jobs_completed(self, email: str) -> bool:
|
||||
"""Increment the completed jobs counter for a user using atomic increment."""
|
||||
try:
|
||||
# Check user exists first
|
||||
if not self.get_user(email):
|
||||
return False
|
||||
|
||||
# Use atomic increment to prevent race conditions
|
||||
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
|
||||
doc_ref.update({
|
||||
'total_jobs_completed': Increment(1),
|
||||
'updated_at': datetime.utcnow()
|
||||
})
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception(f"Error incrementing jobs completed for {email}")
|
||||
return False
|
||||
|
||||
|
||||
# Global instance
|
||||
_user_service = None
|
||||
|
||||
|
||||
def get_user_service() -> UserService:
|
||||
"""Get the global user service instance."""
|
||||
global _user_service
|
||||
if _user_service is None:
|
||||
_user_service = UserService()
|
||||
return _user_service
|
||||
@@ -20,6 +20,7 @@ import os
|
||||
import json
|
||||
from typing import Optional
|
||||
import httpx
|
||||
from google.protobuf import duration_pb2
|
||||
|
||||
from backend.config import get_settings
|
||||
from backend.services.tracing import inject_trace_context
|
||||
@@ -36,8 +37,25 @@ WORKER_QUEUES = {
|
||||
"screens": "screens-worker-queue",
|
||||
"render-video": "render-worker-queue",
|
||||
"video": "video-worker-queue",
|
||||
"idle-reminder": "idle-reminder-queue", # For delayed idle reminder checks
|
||||
}
|
||||
|
||||
# Dispatch deadlines for each worker type (in seconds)
|
||||
# Cloud Tasks max is 1800s (30 min). These set how long Cloud Tasks waits
|
||||
# for the HTTP handler to respond before considering it failed.
|
||||
# Must be >= actual worker timeout + buffer for startup/upload.
|
||||
WORKER_DISPATCH_DEADLINES = {
|
||||
"audio": 1800, # 30 min - Modal separation can take 15-20 min
|
||||
"lyrics": 1500, # 25 min - TRANSCRIPTION_TIMEOUT_SECONDS is 1200s (20 min)
|
||||
"screens": 600, # 10 min - Screen generation is fast
|
||||
"render-video": 1800, # 30 min - Video encoding can be slow
|
||||
"video": 1800, # 30 min - Video encoding can be slow
|
||||
"idle-reminder": 60, # 1 min - Quick check and potential email send
|
||||
}
|
||||
|
||||
# Default delay for idle reminders (seconds)
|
||||
IDLE_REMINDER_DELAY_SECONDS = 5 * 60 # 5 minutes
|
||||
|
||||
|
||||
class WorkerService:
|
||||
"""
|
||||
@@ -234,6 +252,9 @@ class WorkerService:
|
||||
# This allows worker spans to link back to the original request trace
|
||||
headers = inject_trace_context(headers)
|
||||
|
||||
# Get dispatch deadline for this worker type (how long Cloud Tasks waits for response)
|
||||
dispatch_deadline_seconds = WORKER_DISPATCH_DEADLINES.get(worker_type, 600)
|
||||
|
||||
# Build task payload
|
||||
task = {
|
||||
"http_request": {
|
||||
@@ -247,6 +268,10 @@ class WorkerService:
|
||||
"service_account_email": f"karaoke-backend@{project}.iam.gserviceaccount.com",
|
||||
},
|
||||
},
|
||||
# Set dispatch_deadline - how long Cloud Tasks waits for the handler to respond
|
||||
# Default is 10 min, but audio separation can take 30+ min
|
||||
# Max is 1800s (30 min) for Cloud Tasks
|
||||
"dispatch_deadline": duration_pb2.Duration(seconds=dispatch_deadline_seconds),
|
||||
}
|
||||
|
||||
# Create task
|
||||
@@ -254,7 +279,8 @@ class WorkerService:
|
||||
# This prevents duplicate task errors on retries
|
||||
response = self.tasks_client.create_task(parent=parent, task=task)
|
||||
logger.info(
|
||||
f"[job:{job_id}] Created Cloud Task for {worker_type} worker: {response.name}"
|
||||
f"[job:{job_id}] Created Cloud Task for {worker_type} worker: {response.name} "
|
||||
f"(dispatch_deadline={dispatch_deadline_seconds}s)"
|
||||
)
|
||||
return True
|
||||
|
||||
@@ -409,6 +435,99 @@ class WorkerService:
|
||||
"""Trigger render video worker (post-review)."""
|
||||
return await self.trigger_worker("render-video", job_id)
|
||||
|
||||
async def schedule_idle_reminder(
|
||||
self,
|
||||
job_id: str,
|
||||
delay_seconds: int = IDLE_REMINDER_DELAY_SECONDS
|
||||
) -> bool:
|
||||
"""
|
||||
Schedule an idle reminder check for a job.
|
||||
|
||||
The reminder task will be delivered after the specified delay.
|
||||
When the task executes, it checks if the job is still in a blocking
|
||||
state and sends a reminder email if the user hasn't taken action.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to check
|
||||
delay_seconds: Delay before executing the check (default: 5 minutes)
|
||||
|
||||
Returns:
|
||||
True if task was scheduled successfully, False otherwise
|
||||
"""
|
||||
if not self._use_cloud_tasks:
|
||||
# In development mode, log and skip (no delayed execution support)
|
||||
logger.info(
|
||||
f"[job:{job_id}] Idle reminder not scheduled (Cloud Tasks disabled). "
|
||||
f"Would have fired in {delay_seconds}s."
|
||||
)
|
||||
return True
|
||||
|
||||
try:
|
||||
from google.cloud import tasks_v2
|
||||
from google.protobuf import timestamp_pb2
|
||||
import time
|
||||
|
||||
queue_name = WORKER_QUEUES.get("idle-reminder")
|
||||
if not queue_name:
|
||||
logger.error("Idle reminder queue not configured")
|
||||
return False
|
||||
|
||||
project = self.settings.google_cloud_project
|
||||
if not project:
|
||||
logger.error("GOOGLE_CLOUD_PROJECT not set")
|
||||
return False
|
||||
|
||||
location = self.settings.gcp_region
|
||||
|
||||
# Build queue path
|
||||
parent = self.tasks_client.queue_path(project, location, queue_name)
|
||||
|
||||
# Build headers
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
if self._admin_token:
|
||||
headers["X-Admin-Token"] = self._admin_token
|
||||
|
||||
# Inject trace context
|
||||
headers = inject_trace_context(headers)
|
||||
|
||||
# Calculate schedule time
|
||||
schedule_time = timestamp_pb2.Timestamp()
|
||||
schedule_time.FromSeconds(int(time.time()) + delay_seconds)
|
||||
|
||||
dispatch_deadline_seconds = WORKER_DISPATCH_DEADLINES.get("idle-reminder", 60)
|
||||
|
||||
# Build task payload
|
||||
task = {
|
||||
"http_request": {
|
||||
"http_method": tasks_v2.HttpMethod.POST,
|
||||
"url": f"{self._base_url}/api/internal/jobs/{job_id}/check-idle-reminder",
|
||||
"headers": headers,
|
||||
"body": json.dumps({"job_id": job_id}).encode(),
|
||||
"oidc_token": {
|
||||
"service_account_email": f"karaoke-backend@{project}.iam.gserviceaccount.com",
|
||||
},
|
||||
},
|
||||
"schedule_time": schedule_time,
|
||||
"dispatch_deadline": duration_pb2.Duration(seconds=dispatch_deadline_seconds),
|
||||
}
|
||||
|
||||
# Create task
|
||||
response = self.tasks_client.create_task(parent=parent, task=task)
|
||||
logger.info(
|
||||
f"[job:{job_id}] Scheduled idle reminder check in {delay_seconds}s: {response.name}"
|
||||
)
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"[job:{job_id}] Failed to schedule idle reminder: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
return False
|
||||
|
||||
|
||||
# Global worker service instance
|
||||
_worker_service: Optional[WorkerService] = None
|
||||
|
||||
@@ -0,0 +1,445 @@
|
||||
"""
|
||||
YouTube Upload Service.
|
||||
|
||||
Provides video upload functionality for YouTube, extracted from KaraokeFinalise
|
||||
for use by both the cloud backend (video_worker) and local CLI.
|
||||
|
||||
This service handles:
|
||||
- Video uploads with metadata (title, description, tags)
|
||||
- Thumbnail uploads
|
||||
- Duplicate video detection and replacement
|
||||
- Authentication via pre-stored credentials or client secrets file
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import Optional, Dict, Any, Tuple
|
||||
|
||||
from thefuzz import fuzz
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class YouTubeUploadService:
|
||||
"""
|
||||
Service for uploading videos to YouTube.
|
||||
|
||||
Supports two authentication modes:
|
||||
1. Pre-stored credentials (for server-side/non-interactive use)
|
||||
2. Client secrets file (for interactive CLI use with browser OAuth)
|
||||
"""
|
||||
|
||||
YOUTUBE_URL_PREFIX = "https://www.youtube.com/watch?v="
|
||||
SCOPES = ["https://www.googleapis.com/auth/youtube"]
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
credentials: Optional[Dict[str, Any]] = None,
|
||||
client_secrets_file: Optional[str] = None,
|
||||
non_interactive: bool = False,
|
||||
server_side_mode: bool = False,
|
||||
dry_run: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the YouTube upload service.
|
||||
|
||||
Args:
|
||||
credentials: Pre-stored OAuth credentials dict with keys:
|
||||
- token: Access token
|
||||
- refresh_token: Refresh token
|
||||
- token_uri: Token endpoint URL
|
||||
- client_id: OAuth client ID
|
||||
- client_secret: OAuth client secret
|
||||
- scopes: List of OAuth scopes
|
||||
client_secrets_file: Path to OAuth client secrets JSON file
|
||||
(for interactive authentication)
|
||||
non_interactive: If True, skip interactive prompts
|
||||
server_side_mode: If True, use exact matching for duplicate detection
|
||||
dry_run: If True, log actions without performing them
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.credentials = credentials
|
||||
self.client_secrets_file = client_secrets_file
|
||||
self.non_interactive = non_interactive
|
||||
self.server_side_mode = server_side_mode
|
||||
self.dry_run = dry_run
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
self._youtube_service = None
|
||||
self._channel_id = None
|
||||
|
||||
def _authenticate(self):
|
||||
"""
|
||||
Authenticate with YouTube API and return service object.
|
||||
|
||||
Returns:
|
||||
YouTube API service object
|
||||
|
||||
Raises:
|
||||
Exception: If authentication fails or is required in non-interactive mode
|
||||
"""
|
||||
from google.auth.transport.requests import Request
|
||||
from google.oauth2.credentials import Credentials
|
||||
from googleapiclient.discovery import build
|
||||
from google_auth_oauthlib.flow import InstalledAppFlow
|
||||
import pickle
|
||||
|
||||
# Check if we have pre-stored credentials (for non-interactive mode)
|
||||
if self.credentials and self.non_interactive:
|
||||
try:
|
||||
creds = Credentials(
|
||||
token=self.credentials.get('token'),
|
||||
refresh_token=self.credentials.get('refresh_token'),
|
||||
token_uri=self.credentials.get('token_uri'),
|
||||
client_id=self.credentials.get('client_id'),
|
||||
client_secret=self.credentials.get('client_secret'),
|
||||
scopes=self.credentials.get('scopes', self.SCOPES)
|
||||
)
|
||||
|
||||
# Refresh token if needed
|
||||
if creds.expired and creds.refresh_token:
|
||||
creds.refresh(Request())
|
||||
|
||||
youtube = build('youtube', 'v3', credentials=creds)
|
||||
self.logger.info("Successfully authenticated with YouTube using pre-stored credentials")
|
||||
return youtube
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to authenticate with pre-stored credentials: {str(e)}")
|
||||
# Fall through to original authentication if pre-stored credentials fail
|
||||
|
||||
# For non-interactive mode without pre-stored credentials, raise error
|
||||
if self.non_interactive:
|
||||
raise Exception(
|
||||
"YouTube authentication required but running in non-interactive mode. "
|
||||
"Please pre-authenticate or disable YouTube upload."
|
||||
)
|
||||
|
||||
# Interactive authentication using client secrets file
|
||||
if not self.client_secrets_file:
|
||||
raise Exception("No YouTube credentials or client secrets file provided")
|
||||
|
||||
# Token file stores the user's access and refresh tokens
|
||||
youtube_token_file = "/tmp/karaoke-finalise-youtube-token.pickle"
|
||||
|
||||
creds = None
|
||||
|
||||
# Check if we have saved credentials
|
||||
if os.path.exists(youtube_token_file):
|
||||
with open(youtube_token_file, "rb") as token:
|
||||
creds = pickle.load(token)
|
||||
|
||||
# If there are no valid credentials, let the user log in
|
||||
if not creds or not creds.valid:
|
||||
if creds and creds.expired and creds.refresh_token:
|
||||
creds.refresh(Request())
|
||||
else:
|
||||
flow = InstalledAppFlow.from_client_secrets_file(
|
||||
self.client_secrets_file, scopes=self.SCOPES
|
||||
)
|
||||
creds = flow.run_local_server(port=0)
|
||||
|
||||
# Save the credentials for the next run
|
||||
with open(youtube_token_file, "wb") as token:
|
||||
pickle.dump(creds, token)
|
||||
|
||||
return build("youtube", "v3", credentials=creds)
|
||||
|
||||
@property
|
||||
def youtube_service(self):
|
||||
"""Lazy-load YouTube service on first access."""
|
||||
if self._youtube_service is None:
|
||||
self._youtube_service = self._authenticate()
|
||||
return self._youtube_service
|
||||
|
||||
def get_channel_id(self) -> Optional[str]:
|
||||
"""
|
||||
Get the authenticated user's YouTube channel ID.
|
||||
|
||||
Returns:
|
||||
Channel ID string, or None if not found
|
||||
"""
|
||||
if self._channel_id:
|
||||
return self._channel_id
|
||||
|
||||
request = self.youtube_service.channels().list(part="snippet", mine=True)
|
||||
response = request.execute()
|
||||
|
||||
if "items" in response and len(response["items"]) > 0:
|
||||
self._channel_id = response["items"][0]["id"]
|
||||
return self._channel_id
|
||||
|
||||
return None
|
||||
|
||||
def check_duplicate(
|
||||
self,
|
||||
title: str,
|
||||
exact_match: Optional[bool] = None
|
||||
) -> Tuple[bool, Optional[str], Optional[str]]:
|
||||
"""
|
||||
Check if a video with the given title already exists on the channel.
|
||||
|
||||
Args:
|
||||
title: Video title to search for
|
||||
exact_match: If True, require exact title match. If None, uses
|
||||
server_side_mode setting (exact in server mode, fuzzy in CLI)
|
||||
|
||||
Returns:
|
||||
Tuple of (exists: bool, video_id: Optional[str], video_url: Optional[str])
|
||||
"""
|
||||
channel_id = self.get_channel_id()
|
||||
if not channel_id:
|
||||
self.logger.warning("Could not get channel ID, skipping duplicate check")
|
||||
return False, None, None
|
||||
|
||||
use_exact_match = exact_match if exact_match is not None else self.server_side_mode
|
||||
|
||||
self.logger.info(f"Searching YouTube channel {channel_id} for title: {title}")
|
||||
request = self.youtube_service.search().list(
|
||||
part="snippet",
|
||||
channelId=channel_id,
|
||||
q=title,
|
||||
type="video",
|
||||
maxResults=10
|
||||
)
|
||||
response = request.execute()
|
||||
|
||||
if "items" not in response or len(response["items"]) == 0:
|
||||
self.logger.info(f"No matching video found with title: {title}")
|
||||
return False, None, None
|
||||
|
||||
for item in response["items"]:
|
||||
# Verify the video actually belongs to our channel
|
||||
result_channel_id = item["snippet"]["channelId"]
|
||||
if result_channel_id != channel_id:
|
||||
self.logger.debug(
|
||||
f"Skipping video from different channel: {item['snippet']['title']} "
|
||||
f"(channel: {result_channel_id})"
|
||||
)
|
||||
continue
|
||||
|
||||
found_title = item["snippet"]["title"]
|
||||
|
||||
# Determine if this is a match
|
||||
if use_exact_match:
|
||||
is_match = title.lower() == found_title.lower()
|
||||
similarity_score = 100 if is_match else 0
|
||||
else:
|
||||
similarity_score = fuzz.ratio(title.lower(), found_title.lower())
|
||||
is_match = similarity_score >= 70
|
||||
|
||||
if is_match:
|
||||
video_id = item["id"]["videoId"]
|
||||
video_url = f"{self.YOUTUBE_URL_PREFIX}{video_id}"
|
||||
self.logger.info(
|
||||
f"Potential match found on YouTube channel with ID: {video_id} "
|
||||
f"and title: {found_title} (similarity: {similarity_score}%)"
|
||||
)
|
||||
|
||||
# In non-interactive mode, return the match directly
|
||||
if self.non_interactive:
|
||||
self.logger.info("Non-interactive mode, found a match.")
|
||||
return True, video_id, video_url
|
||||
|
||||
# Interactive confirmation
|
||||
confirmation = input(
|
||||
f"Is '{found_title}' the video you are finalising? (y/n): "
|
||||
).strip().lower()
|
||||
if confirmation == "y":
|
||||
return True, video_id, video_url
|
||||
|
||||
self.logger.info(f"No matching video found with title: {title}")
|
||||
return False, None, None
|
||||
|
||||
def delete_video(self, video_id: str) -> bool:
|
||||
"""
|
||||
Delete a YouTube video by its ID.
|
||||
|
||||
Args:
|
||||
video_id: The YouTube video ID to delete
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
self.logger.info(f"Deleting YouTube video with ID: {video_id}")
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(f"DRY RUN: Would delete YouTube video with ID: {video_id}")
|
||||
return True
|
||||
|
||||
try:
|
||||
self.youtube_service.videos().delete(id=video_id).execute()
|
||||
self.logger.info(f"Successfully deleted YouTube video with ID: {video_id}")
|
||||
return True
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to delete YouTube video with ID {video_id}: {e}")
|
||||
return False
|
||||
|
||||
@staticmethod
|
||||
def truncate_title(title: str, max_length: int = 95) -> str:
|
||||
"""
|
||||
Truncate title to the nearest whole word within max_length.
|
||||
|
||||
Args:
|
||||
title: Title to truncate
|
||||
max_length: Maximum length (default 95 for YouTube)
|
||||
|
||||
Returns:
|
||||
Truncated title with "..." if needed
|
||||
"""
|
||||
if len(title) <= max_length:
|
||||
return title
|
||||
truncated_title = title[:max_length].rsplit(" ", 1)[0]
|
||||
if len(truncated_title) < max_length:
|
||||
truncated_title += " ..."
|
||||
return truncated_title
|
||||
|
||||
def upload_video(
|
||||
self,
|
||||
video_path: str,
|
||||
title: str,
|
||||
description: str,
|
||||
thumbnail_path: Optional[str] = None,
|
||||
tags: Optional[list] = None,
|
||||
category_id: str = "10", # Music category
|
||||
privacy_status: str = "public",
|
||||
replace_existing: bool = False,
|
||||
) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""
|
||||
Upload a video to YouTube with metadata and optional thumbnail.
|
||||
|
||||
Args:
|
||||
video_path: Path to the video file to upload
|
||||
title: Video title (will be truncated to 95 chars if needed)
|
||||
description: Video description
|
||||
thumbnail_path: Optional path to thumbnail image
|
||||
tags: Optional list of tags/keywords
|
||||
category_id: YouTube category ID (default "10" for Music)
|
||||
privacy_status: "public", "private", or "unlisted"
|
||||
replace_existing: If True, delete existing video with same title
|
||||
|
||||
Returns:
|
||||
Tuple of (video_id, video_url) or (None, None) if upload failed/skipped
|
||||
"""
|
||||
from googleapiclient.http import MediaFileUpload
|
||||
|
||||
# Truncate title if needed
|
||||
youtube_title = self.truncate_title(title)
|
||||
|
||||
self.logger.info(f"Uploading video to YouTube: {youtube_title}")
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(
|
||||
f"DRY RUN: Would upload {video_path} to YouTube with title: {youtube_title}"
|
||||
)
|
||||
return "dry_run_video_id", f"{self.YOUTUBE_URL_PREFIX}dry_run_video_id"
|
||||
|
||||
# Check for existing video
|
||||
should_replace = True if self.server_side_mode else replace_existing
|
||||
exists, existing_id, existing_url = self.check_duplicate(youtube_title)
|
||||
|
||||
if exists:
|
||||
if should_replace:
|
||||
self.logger.info(f"Video already exists on YouTube, deleting before re-upload: {existing_url}")
|
||||
if self.delete_video(existing_id):
|
||||
self.logger.info("Successfully deleted existing video, proceeding with upload")
|
||||
else:
|
||||
self.logger.error("Failed to delete existing video, aborting upload")
|
||||
return None, None
|
||||
else:
|
||||
self.logger.warning(f"Video already exists on YouTube, skipping upload: {existing_url}")
|
||||
return existing_id, existing_url
|
||||
|
||||
# Prepare video metadata
|
||||
body = {
|
||||
"snippet": {
|
||||
"title": youtube_title,
|
||||
"description": description,
|
||||
"tags": tags or [],
|
||||
"categoryId": category_id,
|
||||
},
|
||||
"status": {"privacyStatus": privacy_status},
|
||||
}
|
||||
|
||||
# Determine MIME type based on file extension
|
||||
ext = os.path.splitext(video_path)[1].lower()
|
||||
mime_type = {
|
||||
".mkv": "video/x-matroska",
|
||||
".mp4": "video/mp4",
|
||||
".mov": "video/quicktime",
|
||||
".avi": "video/x-msvideo",
|
||||
}.get(ext, "video/*")
|
||||
|
||||
# Upload video
|
||||
self.logger.info(f"Authenticating with YouTube...")
|
||||
media_file = MediaFileUpload(video_path, mimetype=mime_type, resumable=True)
|
||||
|
||||
self.logger.info(f"Uploading video to YouTube...")
|
||||
request = self.youtube_service.videos().insert(
|
||||
part="snippet,status",
|
||||
body=body,
|
||||
media_body=media_file
|
||||
)
|
||||
response = request.execute()
|
||||
|
||||
video_id = response.get("id")
|
||||
video_url = f"{self.YOUTUBE_URL_PREFIX}{video_id}"
|
||||
self.logger.info(f"Uploaded video to YouTube: {video_url}")
|
||||
|
||||
# Upload thumbnail if provided
|
||||
if thumbnail_path and os.path.isfile(thumbnail_path):
|
||||
try:
|
||||
self.logger.info(f"Uploading thumbnail from: {thumbnail_path}")
|
||||
media_thumbnail = MediaFileUpload(thumbnail_path, mimetype="image/jpeg")
|
||||
self.youtube_service.thumbnails().set(
|
||||
videoId=video_id,
|
||||
media_body=media_thumbnail
|
||||
).execute()
|
||||
self.logger.info(f"Uploaded thumbnail for video ID {video_id}")
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to upload thumbnail: {e}")
|
||||
self.logger.warning(
|
||||
"Video uploaded but thumbnail not set. "
|
||||
"You may need to set it manually on YouTube."
|
||||
)
|
||||
elif thumbnail_path:
|
||||
self.logger.warning(f"Thumbnail file not found, skipping: {thumbnail_path}")
|
||||
|
||||
return video_id, video_url
|
||||
|
||||
|
||||
# Singleton instance and factory function (following existing service pattern)
|
||||
_youtube_upload_service: Optional[YouTubeUploadService] = None
|
||||
|
||||
|
||||
def get_youtube_upload_service(
|
||||
credentials: Optional[Dict[str, Any]] = None,
|
||||
client_secrets_file: Optional[str] = None,
|
||||
**kwargs
|
||||
) -> YouTubeUploadService:
|
||||
"""
|
||||
Get a YouTube upload service instance.
|
||||
|
||||
For server-side use, pass credentials from YouTubeService.
|
||||
For CLI use, pass client_secrets_file.
|
||||
|
||||
Args:
|
||||
credentials: Pre-stored OAuth credentials dict
|
||||
client_secrets_file: Path to OAuth client secrets JSON file
|
||||
**kwargs: Additional arguments passed to YouTubeUploadService
|
||||
|
||||
Returns:
|
||||
YouTubeUploadService instance
|
||||
"""
|
||||
global _youtube_upload_service
|
||||
|
||||
# Create new instance if credentials/settings changed
|
||||
if _youtube_upload_service is None or credentials or client_secrets_file:
|
||||
_youtube_upload_service = YouTubeUploadService(
|
||||
credentials=credentials,
|
||||
client_secrets_file=client_secrets_file,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
return _youtube_upload_service
|
||||
@@ -6,23 +6,29 @@ Provides common mocks and test utilities across all test modules.
|
||||
NOTE: Module-level mocks are NOT applied here to allow emulator tests to work.
|
||||
Individual tests must mock dependencies as needed.
|
||||
"""
|
||||
# IMPORTANT: Set environment variables FIRST, before ANY imports that might
|
||||
# trigger backend module loading and Settings singleton creation.
|
||||
import os
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, AsyncMock, patch
|
||||
from datetime import datetime, UTC
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
# Mock google.auth.default AND firestore for unit tests if not using emulator
|
||||
# This prevents DefaultCredentialsError and FirestoreClient initialization during imports
|
||||
# Set up test environment variables BEFORE importing any backend modules
|
||||
# This must happen before Settings is instantiated, which occurs on first import
|
||||
if 'FIRESTORE_EMULATOR_HOST' not in os.environ:
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
# Set up test environment variables BEFORE importing any backend modules
|
||||
os.environ.setdefault('ADMIN_TOKENS', 'test-admin-token')
|
||||
os.environ.setdefault('GOOGLE_CLOUD_PROJECT', 'test-project')
|
||||
os.environ.setdefault('GCS_BUCKET_NAME', 'test-bucket')
|
||||
os.environ.setdefault('FIRESTORE_COLLECTION', 'jobs')
|
||||
os.environ.setdefault('ENVIRONMENT', 'test')
|
||||
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, AsyncMock, patch
|
||||
from datetime import datetime, UTC
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
|
||||
# Mock google.auth.default AND firestore for unit tests if not using emulator
|
||||
# This prevents DefaultCredentialsError and FirestoreClient initialization during imports
|
||||
if 'FIRESTORE_EMULATOR_HOST' not in os.environ:
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
# Mock google.auth.default - prevents credential errors
|
||||
try:
|
||||
|
||||
@@ -39,6 +39,24 @@ os.environ["ADMIN_TOKENS"] = "test-admin-token"
|
||||
|
||||
# Only import app if emulators are running
|
||||
if emulators_running():
|
||||
from unittest.mock import Mock
|
||||
|
||||
# Mock theme service BEFORE importing app to ensure all jobs get theme_id="nomad"
|
||||
_mock_theme_service = Mock()
|
||||
_mock_theme_service.get_default_theme_id.return_value = "nomad"
|
||||
_mock_theme_service.get_theme.return_value = None
|
||||
|
||||
# Apply patch at module load time (before app imports the service)
|
||||
_theme_patches = [
|
||||
patch("backend.services.theme_service.get_theme_service", return_value=_mock_theme_service),
|
||||
patch("backend.api.routes.audio_search.get_theme_service", return_value=_mock_theme_service),
|
||||
patch("backend.api.routes.file_upload.get_theme_service", return_value=_mock_theme_service),
|
||||
patch("backend.api.routes.users.get_theme_service", return_value=_mock_theme_service),
|
||||
patch("backend.api.routes.jobs.get_theme_service", return_value=_mock_theme_service),
|
||||
]
|
||||
for p in _theme_patches:
|
||||
p.start()
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
from backend.main import app
|
||||
else:
|
||||
@@ -76,7 +94,10 @@ def mock_worker_service():
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def client(mock_worker_service):
|
||||
"""Create FastAPI test client with mocked workers."""
|
||||
"""Create FastAPI test client with mocked workers.
|
||||
|
||||
Theme service is mocked at module load time (see above).
|
||||
"""
|
||||
with patch("backend.api.routes.file_upload.worker_service", mock_worker_service):
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@@ -0,0 +1,443 @@
|
||||
"""
|
||||
Integration tests for worker log subcollection storage.
|
||||
|
||||
Tests the subcollection approach using real Firestore emulator to verify:
|
||||
- Logs are stored in subcollection (jobs/{job_id}/logs)
|
||||
- TTL expiry field is set correctly
|
||||
- Large log volumes work without 1MB limit issues
|
||||
- Concurrent writes are handled correctly
|
||||
- Logs can be queried efficiently
|
||||
|
||||
Run with: ./scripts/run-emulator-tests.sh
|
||||
"""
|
||||
import pytest
|
||||
import time
|
||||
import requests
|
||||
import os
|
||||
import threading
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from datetime import datetime, timezone, timedelta
|
||||
|
||||
|
||||
def emulators_running() -> bool:
|
||||
"""Check if GCP emulators are running."""
|
||||
try:
|
||||
requests.get("http://127.0.0.1:8080", timeout=1)
|
||||
return True
|
||||
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
|
||||
return False
|
||||
|
||||
|
||||
# Skip all tests in this module if emulators aren't running
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not emulators_running(),
|
||||
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
|
||||
)
|
||||
|
||||
# Set up environment for emulator
|
||||
os.environ["FIRESTORE_EMULATOR_HOST"] = "127.0.0.1:8080"
|
||||
os.environ["GOOGLE_CLOUD_PROJECT"] = "test-project"
|
||||
|
||||
|
||||
class TestWorkerLogsSubcollectionDirect:
|
||||
"""Direct Firestore tests for worker logs subcollection."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup_firestore(self):
|
||||
"""Set up Firestore client for each test."""
|
||||
from google.cloud import firestore
|
||||
self.db = firestore.Client(project="test-project")
|
||||
self.collection = "test-subcollection-jobs"
|
||||
yield
|
||||
|
||||
def _create_test_job(self):
|
||||
"""Create a test job document."""
|
||||
job_id = f"test-sub-{int(time.time() * 1000)}"
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.set({
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"created_at": datetime.now(timezone.utc),
|
||||
"worker_logs": [] # Legacy field - kept empty for new jobs
|
||||
})
|
||||
return job_id
|
||||
|
||||
def _append_log_to_subcollection(self, job_id: str, worker: str, message: str, ttl_days: int = 30):
|
||||
"""Add log to subcollection at jobs/{job_id}/logs."""
|
||||
import uuid
|
||||
log_id = str(uuid.uuid4())
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
logs_ref.document(log_id).set({
|
||||
"id": log_id,
|
||||
"job_id": job_id,
|
||||
"timestamp": now,
|
||||
"level": "INFO",
|
||||
"worker": worker,
|
||||
"message": message,
|
||||
"ttl_expiry": now + timedelta(days=ttl_days)
|
||||
})
|
||||
return log_id
|
||||
|
||||
def _get_logs_from_subcollection(self, job_id: str, worker: str = None, limit: int = 500):
|
||||
"""Get logs from subcollection."""
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
query = logs_ref.order_by("timestamp")
|
||||
|
||||
if worker:
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
query = query.where(filter=FieldFilter("worker", "==", worker))
|
||||
|
||||
query = query.limit(limit)
|
||||
|
||||
return [doc.to_dict() for doc in query.stream()]
|
||||
|
||||
def _count_logs_in_subcollection(self, job_id: str) -> int:
|
||||
"""Count logs in subcollection."""
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
count_query = logs_ref.count()
|
||||
result = count_query.get()
|
||||
if result and len(result) > 0:
|
||||
return result[0][0].value
|
||||
return 0
|
||||
|
||||
def _delete_logs_subcollection(self, job_id: str, batch_size: int = 100) -> int:
|
||||
"""Delete all logs in subcollection."""
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
deleted_count = 0
|
||||
|
||||
while True:
|
||||
docs = list(logs_ref.limit(batch_size).stream())
|
||||
if not docs:
|
||||
break
|
||||
|
||||
batch = self.db.batch()
|
||||
for doc in docs:
|
||||
batch.delete(doc.reference)
|
||||
deleted_count += 1
|
||||
|
||||
batch.commit()
|
||||
|
||||
return deleted_count
|
||||
|
||||
def test_subcollection_single_write(self):
|
||||
"""Test writing single log to subcollection."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
log_id = self._append_log_to_subcollection(job_id, "test", "Single log message")
|
||||
|
||||
time.sleep(0.1)
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(logs) == 1
|
||||
assert logs[0]["message"] == "Single log message"
|
||||
assert logs[0]["worker"] == "test"
|
||||
assert logs[0]["id"] == log_id
|
||||
print("Single write to subcollection works")
|
||||
|
||||
def test_subcollection_ttl_field_is_set(self):
|
||||
"""Test TTL expiry field is set correctly."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
self._append_log_to_subcollection(job_id, "test", "Log with TTL", ttl_days=7)
|
||||
|
||||
time.sleep(0.1)
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(logs) == 1
|
||||
ttl_expiry = logs[0]["ttl_expiry"]
|
||||
assert ttl_expiry is not None
|
||||
|
||||
# TTL should be ~7 days from now
|
||||
expected_ttl = datetime.now(timezone.utc) + timedelta(days=7)
|
||||
if hasattr(ttl_expiry, 'timestamp'):
|
||||
# Firestore datetime object
|
||||
diff = abs((ttl_expiry.replace(tzinfo=timezone.utc) - expected_ttl).total_seconds())
|
||||
else:
|
||||
diff = 0 # If already datetime
|
||||
assert diff < 60, f"TTL expiry should be ~7 days from now, diff was {diff}s"
|
||||
print("TTL field is set correctly")
|
||||
|
||||
def test_subcollection_sequential_writes(self):
|
||||
"""Test multiple sequential writes to subcollection."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
for i in range(20):
|
||||
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
|
||||
|
||||
time.sleep(0.2)
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(logs) == 20, f"Expected 20 logs, got {len(logs)}"
|
||||
print(f"Sequential writes: {len(logs)} logs created")
|
||||
|
||||
def test_subcollection_concurrent_writes(self):
|
||||
"""Test concurrent writes to subcollection - no race conditions."""
|
||||
job_id = self._create_test_job()
|
||||
num_writes = 50
|
||||
|
||||
def write_log(index):
|
||||
self._append_log_to_subcollection(job_id, "worker", f"Concurrent Log {index}")
|
||||
|
||||
with ThreadPoolExecutor(max_workers=10) as executor:
|
||||
futures = [executor.submit(write_log, i) for i in range(num_writes)]
|
||||
for future in as_completed(futures):
|
||||
future.result()
|
||||
|
||||
time.sleep(0.5)
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(logs) == num_writes, f"Expected {num_writes} logs, got {len(logs)}"
|
||||
print(f"Concurrent writes: All {num_writes} logs preserved")
|
||||
|
||||
def test_subcollection_large_volume(self):
|
||||
"""Test large volume of logs - demonstrates no 1MB limit."""
|
||||
job_id = self._create_test_job()
|
||||
num_logs = 500 # Would exceed 1MB in embedded array
|
||||
|
||||
# Write logs in batches
|
||||
batch_size = 50
|
||||
for batch_start in range(0, num_logs, batch_size):
|
||||
with ThreadPoolExecutor(max_workers=10) as executor:
|
||||
futures = []
|
||||
for i in range(batch_start, min(batch_start + batch_size, num_logs)):
|
||||
futures.append(executor.submit(
|
||||
self._append_log_to_subcollection,
|
||||
job_id,
|
||||
"stress",
|
||||
f"Large volume test log {i} - " + "x" * 500 # ~500 bytes per log
|
||||
))
|
||||
for future in as_completed(futures):
|
||||
future.result()
|
||||
|
||||
time.sleep(1)
|
||||
count = self._count_logs_in_subcollection(job_id)
|
||||
|
||||
assert count == num_logs, f"Expected {num_logs} logs, got {count}"
|
||||
print(f"Large volume: {count} logs created (~{count * 500 / 1024:.1f}KB would exceed 1MB if embedded)")
|
||||
|
||||
def test_subcollection_filter_by_worker(self):
|
||||
"""Test filtering logs by worker type."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
# Add logs from different workers
|
||||
for i in range(10):
|
||||
self._append_log_to_subcollection(job_id, "audio", f"Audio log {i}")
|
||||
self._append_log_to_subcollection(job_id, "lyrics", f"Lyrics log {i}")
|
||||
self._append_log_to_subcollection(job_id, "video", f"Video log {i}")
|
||||
|
||||
time.sleep(0.3)
|
||||
|
||||
# Query by worker
|
||||
audio_logs = self._get_logs_from_subcollection(job_id, worker="audio")
|
||||
lyrics_logs = self._get_logs_from_subcollection(job_id, worker="lyrics")
|
||||
all_logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(audio_logs) == 10, f"Expected 10 audio logs, got {len(audio_logs)}"
|
||||
assert len(lyrics_logs) == 10, f"Expected 10 lyrics logs, got {len(lyrics_logs)}"
|
||||
assert len(all_logs) == 30, f"Expected 30 total logs, got {len(all_logs)}"
|
||||
print("Worker filtering works correctly")
|
||||
|
||||
def test_subcollection_ordered_by_timestamp(self):
|
||||
"""Test logs are returned in timestamp order."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
# Add logs with slight delays to ensure different timestamps
|
||||
for i in range(5):
|
||||
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
|
||||
time.sleep(0.02)
|
||||
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
# Verify order (timestamps should be ascending)
|
||||
for i in range(len(logs) - 1):
|
||||
assert logs[i]["timestamp"] <= logs[i + 1]["timestamp"], \
|
||||
f"Logs not in order at index {i}"
|
||||
print("Logs are ordered by timestamp")
|
||||
|
||||
def test_subcollection_delete_all_logs(self):
|
||||
"""Test deleting all logs in subcollection."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
# Add logs
|
||||
for i in range(25):
|
||||
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
|
||||
|
||||
time.sleep(0.2)
|
||||
count_before = self._count_logs_in_subcollection(job_id)
|
||||
assert count_before == 25
|
||||
|
||||
# Delete all logs
|
||||
deleted = self._delete_logs_subcollection(job_id)
|
||||
|
||||
time.sleep(0.2)
|
||||
count_after = self._count_logs_in_subcollection(job_id)
|
||||
|
||||
assert deleted == 25, f"Expected to delete 25 logs, deleted {deleted}"
|
||||
assert count_after == 0, f"Expected 0 logs after delete, got {count_after}"
|
||||
print(f"Deleted {deleted} logs from subcollection")
|
||||
|
||||
def test_subcollection_concurrent_workers_interleaved(self):
|
||||
"""Test simulating audio and lyrics workers writing concurrently."""
|
||||
job_id = self._create_test_job()
|
||||
logs_per_worker = 20
|
||||
|
||||
def audio_worker():
|
||||
for i in range(logs_per_worker):
|
||||
self._append_log_to_subcollection(job_id, "audio", f"Audio processing step {i}")
|
||||
time.sleep(0.005)
|
||||
|
||||
def lyrics_worker():
|
||||
for i in range(logs_per_worker):
|
||||
self._append_log_to_subcollection(job_id, "lyrics", f"Lyrics processing step {i}")
|
||||
time.sleep(0.005)
|
||||
|
||||
def video_worker():
|
||||
for i in range(logs_per_worker):
|
||||
self._append_log_to_subcollection(job_id, "video", f"Video encoding step {i}")
|
||||
time.sleep(0.005)
|
||||
|
||||
# Start all workers
|
||||
threads = [
|
||||
threading.Thread(target=audio_worker),
|
||||
threading.Thread(target=lyrics_worker),
|
||||
threading.Thread(target=video_worker)
|
||||
]
|
||||
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
|
||||
time.sleep(0.5)
|
||||
|
||||
audio_logs = self._get_logs_from_subcollection(job_id, worker="audio")
|
||||
lyrics_logs = self._get_logs_from_subcollection(job_id, worker="lyrics")
|
||||
video_logs = self._get_logs_from_subcollection(job_id, worker="video")
|
||||
total = self._count_logs_in_subcollection(job_id)
|
||||
|
||||
assert len(audio_logs) == logs_per_worker
|
||||
assert len(lyrics_logs) == logs_per_worker
|
||||
assert len(video_logs) == logs_per_worker
|
||||
assert total == logs_per_worker * 3
|
||||
|
||||
print(f"Concurrent workers: {len(audio_logs)} audio + {len(lyrics_logs)} lyrics + {len(video_logs)} video = {total}")
|
||||
|
||||
def test_subcollection_job_deletion_cleans_up_logs(self):
|
||||
"""Test that deleting job document doesn't orphan logs."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
# Add logs
|
||||
for i in range(10):
|
||||
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
|
||||
|
||||
time.sleep(0.2)
|
||||
assert self._count_logs_in_subcollection(job_id) == 10
|
||||
|
||||
# Delete logs first (must happen before parent doc deletion in real code)
|
||||
deleted = self._delete_logs_subcollection(job_id)
|
||||
assert deleted == 10
|
||||
|
||||
# Now delete job document
|
||||
self.db.collection(self.collection).document(job_id).delete()
|
||||
|
||||
# Verify subcollection is empty (Firestore doesn't cascade delete subcollections)
|
||||
time.sleep(0.2)
|
||||
assert self._count_logs_in_subcollection(job_id) == 0
|
||||
print("Job deletion with log cleanup works correctly")
|
||||
|
||||
|
||||
class TestSubcollectionVsEmbeddedArray:
|
||||
"""Compare subcollection approach with embedded array."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup_firestore(self):
|
||||
"""Set up Firestore client for each test."""
|
||||
from google.cloud import firestore
|
||||
self.db = firestore.Client(project="test-project")
|
||||
self.collection = "test-comparison-jobs"
|
||||
yield
|
||||
|
||||
def _create_test_job(self, use_array: bool):
|
||||
"""Create test job."""
|
||||
job_id = f"test-cmp-{int(time.time() * 1000)}-{'arr' if use_array else 'sub'}"
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.set({
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"created_at": datetime.now(timezone.utc),
|
||||
"worker_logs": []
|
||||
})
|
||||
return job_id
|
||||
|
||||
def _append_log_array(self, job_id: str, message: str):
|
||||
"""Add log to embedded array."""
|
||||
from google.cloud import firestore
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.update({
|
||||
"worker_logs": firestore.ArrayUnion([{
|
||||
"timestamp": datetime.now(timezone.utc).isoformat() + "Z",
|
||||
"level": "INFO",
|
||||
"worker": "test",
|
||||
"message": message
|
||||
}])
|
||||
})
|
||||
|
||||
def _append_log_subcollection(self, job_id: str, message: str):
|
||||
"""Add log to subcollection."""
|
||||
import uuid
|
||||
now = datetime.now(timezone.utc)
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
logs_ref.document(str(uuid.uuid4())).set({
|
||||
"timestamp": now,
|
||||
"level": "INFO",
|
||||
"worker": "test",
|
||||
"message": message,
|
||||
"ttl_expiry": now + timedelta(days=30)
|
||||
})
|
||||
|
||||
def test_document_size_comparison(self):
|
||||
"""Compare document sizes between approaches."""
|
||||
array_job_id = self._create_test_job(use_array=True)
|
||||
sub_job_id = self._create_test_job(use_array=False)
|
||||
|
||||
num_logs = 100
|
||||
large_message = "x" * 5000 # 5KB per log
|
||||
|
||||
# Add logs to both
|
||||
print("\nAdding logs to both approaches...")
|
||||
for i in range(num_logs):
|
||||
self._append_log_array(array_job_id, f"Array log {i}: {large_message}")
|
||||
self._append_log_subcollection(sub_job_id, f"Sub log {i}: {large_message}")
|
||||
|
||||
time.sleep(0.5)
|
||||
|
||||
# Get document sizes (approximate via raw data)
|
||||
array_doc = self.db.collection(self.collection).document(array_job_id).get()
|
||||
sub_doc = self.db.collection(self.collection).document(sub_job_id).get()
|
||||
|
||||
array_data = array_doc.to_dict()
|
||||
sub_data = sub_doc.to_dict()
|
||||
|
||||
array_logs = array_data.get("worker_logs", [])
|
||||
sub_logs = sub_data.get("worker_logs", [])
|
||||
|
||||
print(f"\nEmbedded array: {len(array_logs)} logs in document")
|
||||
print(f"Subcollection: {len(sub_logs)} logs in document (0 expected)")
|
||||
|
||||
# Embedded array should have all logs in document
|
||||
assert len(array_logs) == num_logs
|
||||
|
||||
# Subcollection should have empty array in main document
|
||||
assert len(sub_logs) == 0
|
||||
|
||||
# Subcollection logs are in subcollection
|
||||
sub_count = self.db.collection(self.collection).document(sub_job_id).collection("logs").count().get()[0][0].value
|
||||
assert sub_count == num_logs
|
||||
|
||||
print(f"Subcollection logs stored separately: {sub_count}")
|
||||
print("Subcollection approach keeps main document small")
|
||||
|
||||
|
||||
print("Worker logs subcollection integration tests ready")
|
||||
@@ -0,0 +1,411 @@
|
||||
"""
|
||||
Unit tests for admin email endpoints.
|
||||
|
||||
Tests the completion message and send email API endpoints.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, MagicMock, AsyncMock
|
||||
from fastapi.testclient import TestClient
|
||||
from fastapi import FastAPI
|
||||
|
||||
from backend.api.routes.admin import router
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
# Create a test app with the admin router
|
||||
# The router already has prefix="/admin", so we add /api prefix
|
||||
app = FastAPI()
|
||||
app.include_router(router, prefix="/api")
|
||||
|
||||
|
||||
def get_mock_admin():
|
||||
"""Override for require_admin dependency."""
|
||||
return ("admin@example.com", "admin", 1)
|
||||
|
||||
|
||||
# Override the require_admin dependency
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
"""Create a test client."""
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job():
|
||||
"""Create a mock job."""
|
||||
job = Mock(spec=Job)
|
||||
job.job_id = "test-job-123"
|
||||
job.user_email = "user@example.com"
|
||||
job.artist = "Test Artist"
|
||||
job.title = "Test Song"
|
||||
job.status = JobStatus.COMPLETE
|
||||
job.audio_hash = "hash123"
|
||||
job.review_token = "review123"
|
||||
job.instrumental_token = "inst123"
|
||||
job.state_data = {
|
||||
"youtube_url": "https://youtube.com/watch?v=test123",
|
||||
"dropbox_link": "https://dropbox.com/folder/test",
|
||||
"brand_code": "NOMAD-1234",
|
||||
}
|
||||
return job
|
||||
|
||||
|
||||
class TestGetCompletionMessage:
|
||||
"""Tests for GET /api/admin/jobs/{job_id}/completion-message endpoint."""
|
||||
|
||||
def test_returns_completion_message(self, client, mock_job):
|
||||
"""Test successful completion message retrieval."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
|
||||
|
||||
# Setup mocks
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Your video is ready!"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
response = client.get(
|
||||
"/api/admin/jobs/test-job-123/completion-message",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["job_id"] == "test-job-123"
|
||||
assert data["message"] == "Your video is ready!"
|
||||
# Subject format: "NOMAD-1234: Test Artist - Test Song (Your karaoke video is ready!)"
|
||||
assert data["subject"] == "NOMAD-1234: Test Artist - Test Song (Your karaoke video is ready!)"
|
||||
assert data["youtube_url"] == "https://youtube.com/watch?v=test123"
|
||||
assert data["dropbox_url"] == "https://dropbox.com/folder/test"
|
||||
|
||||
def test_returns_404_when_job_not_found(self, client):
|
||||
"""Test 404 when job doesn't exist."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = None
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.get(
|
||||
"/api/admin/jobs/nonexistent-job/completion-message",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
def test_handles_none_state_data(self, client, mock_job):
|
||||
"""Test handling of job with None state_data."""
|
||||
mock_job.state_data = None
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Your video is ready!"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
response = client.get(
|
||||
"/api/admin/jobs/test-job-123/completion-message",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["youtube_url"] is None
|
||||
assert data["dropbox_url"] is None
|
||||
|
||||
def test_default_subject_without_song_info(self, client, mock_job):
|
||||
"""Test default subject when no artist/title."""
|
||||
mock_job.artist = None
|
||||
mock_job.title = None
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Message"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
response = client.get(
|
||||
"/api/admin/jobs/test-job-123/completion-message",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["subject"] == "Your karaoke video is ready!"
|
||||
|
||||
|
||||
class TestSendCompletionEmail:
|
||||
"""Tests for POST /api/admin/jobs/{job_id}/send-completion-email endpoint."""
|
||||
|
||||
def test_sends_email_successfully(self, client, mock_job):
|
||||
"""Test successful email sending."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
|
||||
patch('backend.services.email_service.get_email_service') as mock_get_es:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Your video is ready!"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
mock_es = Mock()
|
||||
mock_es.send_job_completion.return_value = True
|
||||
mock_get_es.return_value = mock_es
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/send-completion-email",
|
||||
json={"to_email": "customer@example.com", "cc_admin": True},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["success"] is True
|
||||
assert data["job_id"] == "test-job-123"
|
||||
assert data["to_email"] == "customer@example.com"
|
||||
|
||||
# Verify email was sent with correct params
|
||||
mock_es.send_job_completion.assert_called_once_with(
|
||||
to_email="customer@example.com",
|
||||
message_content="Your video is ready!",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
brand_code="NOMAD-1234",
|
||||
cc_admin=True,
|
||||
)
|
||||
|
||||
def test_sends_email_without_cc(self, client, mock_job):
|
||||
"""Test email sending without CC."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
|
||||
patch('backend.services.email_service.get_email_service') as mock_get_es:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Message"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
mock_es = Mock()
|
||||
mock_es.send_job_completion.return_value = True
|
||||
mock_get_es.return_value = mock_es
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/send-completion-email",
|
||||
json={"to_email": "customer@example.com", "cc_admin": False},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
# Verify CC was not included
|
||||
call_kwargs = mock_es.send_job_completion.call_args.kwargs
|
||||
assert call_kwargs["cc_admin"] is False
|
||||
|
||||
def test_returns_404_when_job_not_found(self, client):
|
||||
"""Test 404 when job doesn't exist."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = None
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/nonexistent-job/send-completion-email",
|
||||
json={"to_email": "customer@example.com"},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
def test_returns_500_when_email_fails(self, client, mock_job):
|
||||
"""Test 500 when email sending fails."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
|
||||
patch('backend.services.email_service.get_email_service') as mock_get_es:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Message"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
mock_es = Mock()
|
||||
mock_es.send_job_completion.return_value = False # Email failed
|
||||
mock_get_es.return_value = mock_es
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/send-completion-email",
|
||||
json={"to_email": "customer@example.com"},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 500
|
||||
assert "Failed to send email" in response.json()["detail"]
|
||||
|
||||
def test_handles_none_state_data(self, client, mock_job):
|
||||
"""Test handling of job with None state_data."""
|
||||
mock_job.state_data = None
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
|
||||
patch('backend.services.email_service.get_email_service') as mock_get_es:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Message"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
mock_es = Mock()
|
||||
mock_es.send_job_completion.return_value = True
|
||||
mock_get_es.return_value = mock_es
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/send-completion-email",
|
||||
json={"to_email": "customer@example.com"},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_accepts_any_email_string(self, client, mock_job):
|
||||
"""Test that any string is currently accepted as email (no validation).
|
||||
|
||||
Documents current behavior: FastAPI/Pydantic doesn't validate email
|
||||
format by default. If validation is added later, this test should
|
||||
be updated to expect 422 for invalid emails.
|
||||
"""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
|
||||
patch('backend.services.email_service.get_email_service') as mock_get_es:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Message"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
mock_es = Mock()
|
||||
mock_es.send_job_completion.return_value = True
|
||||
mock_get_es.return_value = mock_es
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/send-completion-email",
|
||||
json={"to_email": "invalid-email"}, # Not a valid email format
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
# Currently accepts any string - endpoint succeeds
|
||||
assert response.status_code == 200
|
||||
# Verify the invalid email was passed through
|
||||
mock_es.send_job_completion.assert_called_once()
|
||||
call_kwargs = mock_es.send_job_completion.call_args.kwargs
|
||||
assert call_kwargs["to_email"] == "invalid-email"
|
||||
|
||||
|
||||
class TestIdleReminderEndpoint:
|
||||
"""Tests for the internal idle reminder endpoint."""
|
||||
|
||||
def test_sends_reminder_when_still_idle(self, mock_job):
|
||||
"""Test that reminder is sent when user is still idle."""
|
||||
from backend.api.routes.internal import router as internal_router
|
||||
from backend.api.dependencies import require_admin
|
||||
|
||||
# Create test app for internal router
|
||||
# The router already has prefix="/internal", so we add /api prefix
|
||||
test_app = FastAPI()
|
||||
test_app.include_router(internal_router, prefix="/api")
|
||||
test_app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
mock_job.status = JobStatus.AWAITING_REVIEW.value # Use .value for string comparison
|
||||
mock_job.state_data = {
|
||||
"blocking_state_entered_at": "2024-01-01T00:00:00",
|
||||
"blocking_action_type": "lyrics",
|
||||
"reminder_sent": False,
|
||||
}
|
||||
|
||||
with patch('backend.api.routes.internal.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.firestore = Mock() # Mock firestore for update_job call
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.send_action_reminder_email = AsyncMock(return_value=True)
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
test_client = TestClient(test_app)
|
||||
response = test_client.post(
|
||||
"/api/internal/jobs/test-job-123/check-idle-reminder",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
# Should succeed and return "sent" status
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "sent"
|
||||
assert data["job_id"] == "test-job-123"
|
||||
|
||||
def test_skips_reminder_when_already_sent(self, mock_job):
|
||||
"""Test that reminder is skipped if already sent."""
|
||||
from backend.api.routes.internal import router as internal_router
|
||||
from backend.api.dependencies import require_admin
|
||||
|
||||
# Create test app for internal router
|
||||
# The router already has prefix="/internal", so we add /api prefix
|
||||
test_app = FastAPI()
|
||||
test_app.include_router(internal_router, prefix="/api")
|
||||
test_app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
mock_job.status = JobStatus.AWAITING_REVIEW.value # Use .value for string comparison
|
||||
mock_job.state_data = {
|
||||
"blocking_state_entered_at": "2024-01-01T00:00:00",
|
||||
"blocking_action_type": "lyrics",
|
||||
"reminder_sent": True, # Already sent
|
||||
}
|
||||
|
||||
with patch('backend.api.routes.internal.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
test_client = TestClient(test_app)
|
||||
response = test_client.post(
|
||||
"/api/internal/jobs/test-job-123/check-idle-reminder",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
# Should succeed with "already_sent" status
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "already_sent"
|
||||
assert data["job_id"] == "test-job-123"
|
||||
@@ -460,16 +460,18 @@ class TestAudioSearchServiceDownload:
|
||||
url="https://youtube.com/watch?v=abc123",
|
||||
index=0,
|
||||
)
|
||||
|
||||
|
||||
service = AudioSearchService(red_api_key=None, red_api_url=None, ops_api_key=None, ops_api_url=None)
|
||||
service._fetcher = Mock()
|
||||
service._fetcher.search.return_value = [mock_result]
|
||||
|
||||
# Ensure no remote client so we test the local cache path
|
||||
service._remote_client = None
|
||||
|
||||
service.search("ABBA", "Waterloo")
|
||||
|
||||
|
||||
with pytest.raises(DownloadError) as exc_info:
|
||||
service.download(99, "/tmp")
|
||||
|
||||
|
||||
assert "No cached result for index 99" in str(exc_info.value)
|
||||
|
||||
|
||||
|
||||
@@ -114,9 +114,97 @@ class TestWorkerServiceBasics:
|
||||
assert 'http' in url
|
||||
|
||||
|
||||
class TestSessionTokenValidation:
|
||||
"""Tests for session token validation in AuthService.
|
||||
|
||||
PR #120: Fixed authentication to recognize session tokens from magic link auth.
|
||||
Previously, validate_token() only checked admin tokens and auth_tokens collection.
|
||||
Session tokens (created via magic link) are stored in the sessions collection,
|
||||
so they were incorrectly rejected as "Invalid token".
|
||||
"""
|
||||
|
||||
def test_validate_token_checks_session_on_fallback(self):
|
||||
"""Test that validate_token falls back to checking session tokens."""
|
||||
mock_firestore = MagicMock()
|
||||
mock_firestore.get_token.return_value = None # Token not in auth_tokens
|
||||
|
||||
mock_user = MagicMock()
|
||||
mock_user.email = "test@example.com"
|
||||
mock_user.credits = 5
|
||||
|
||||
mock_user_service = MagicMock()
|
||||
mock_user_service.validate_session.return_value = (True, mock_user, "Valid session")
|
||||
|
||||
with patch('backend.services.auth_service.FirestoreService', return_value=mock_firestore), \
|
||||
patch('backend.services.user_service.get_user_service', return_value=mock_user_service), \
|
||||
patch.dict('os.environ', {'ADMIN_TOKENS': 'admin-token-123'}):
|
||||
# Clear the cached instance to get fresh state
|
||||
import backend.services.auth_service
|
||||
backend.services.auth_service._auth_service = None
|
||||
from backend.services.auth_service import AuthService, UserType
|
||||
|
||||
service = AuthService()
|
||||
is_valid, user_type, remaining, msg = service.validate_token("session-token-xyz")
|
||||
|
||||
assert is_valid is True
|
||||
assert user_type == UserType.STRIPE
|
||||
assert remaining == 5 # User's credits
|
||||
mock_user_service.validate_session.assert_called_once_with("session-token-xyz")
|
||||
|
||||
def test_validate_token_returns_invalid_when_session_invalid(self):
|
||||
"""Test that validate_token returns invalid when session validation fails."""
|
||||
mock_firestore = MagicMock()
|
||||
mock_firestore.get_token.return_value = None # Token not in auth_tokens
|
||||
|
||||
mock_user_service = MagicMock()
|
||||
mock_user_service.validate_session.return_value = (False, None, "Invalid session")
|
||||
|
||||
with patch('backend.services.auth_service.FirestoreService', return_value=mock_firestore), \
|
||||
patch('backend.services.user_service.get_user_service', return_value=mock_user_service), \
|
||||
patch.dict('os.environ', {'ADMIN_TOKENS': 'admin-token-123'}):
|
||||
import backend.services.auth_service
|
||||
backend.services.auth_service._auth_service = None
|
||||
from backend.services.auth_service import AuthService, UserType
|
||||
|
||||
service = AuthService()
|
||||
is_valid, user_type, remaining, msg = service.validate_token("invalid-session")
|
||||
|
||||
assert is_valid is False
|
||||
assert remaining == 0
|
||||
assert "Invalid token" in msg
|
||||
|
||||
def test_validate_token_returns_zero_credits_when_user_has_none(self):
|
||||
"""Test that validate_token handles users with 0 credits correctly."""
|
||||
mock_firestore = MagicMock()
|
||||
mock_firestore.get_token.return_value = None
|
||||
|
||||
mock_user = MagicMock()
|
||||
mock_user.email = "test@example.com"
|
||||
mock_user.credits = 0
|
||||
|
||||
mock_user_service = MagicMock()
|
||||
mock_user_service.validate_session.return_value = (True, mock_user, "Valid session")
|
||||
|
||||
with patch('backend.services.auth_service.FirestoreService', return_value=mock_firestore), \
|
||||
patch('backend.services.user_service.get_user_service', return_value=mock_user_service), \
|
||||
patch.dict('os.environ', {'ADMIN_TOKENS': 'admin-token-123'}):
|
||||
import backend.services.auth_service
|
||||
backend.services.auth_service._auth_service = None
|
||||
from backend.services.auth_service import AuthService, UserType
|
||||
|
||||
service = AuthService()
|
||||
is_valid, user_type, remaining, msg = service.validate_token("session-token")
|
||||
|
||||
# User is authenticated but has no credits
|
||||
assert is_valid is True
|
||||
assert user_type == UserType.STRIPE
|
||||
assert remaining == 0
|
||||
assert "no credits" in msg.lower()
|
||||
|
||||
|
||||
class TestJobManagerBasics:
|
||||
"""Basic tests for JobManager."""
|
||||
|
||||
|
||||
def test_job_manager_can_import(self):
|
||||
"""Test JobManager can be imported."""
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
@@ -0,0 +1,244 @@
|
||||
"""
|
||||
Tests for DiscordNotificationService.
|
||||
|
||||
Tests cover:
|
||||
- Service initialization
|
||||
- Webhook URL validation
|
||||
- Message posting
|
||||
- Video notification posting
|
||||
- Dry run mode
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from backend.services.discord_service import (
|
||||
DiscordNotificationService,
|
||||
get_discord_notification_service,
|
||||
)
|
||||
|
||||
|
||||
class TestDiscordNotificationServiceInit:
|
||||
"""Test service initialization."""
|
||||
|
||||
def test_init_with_webhook_url(self):
|
||||
"""Test initialization with valid webhook URL."""
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef123456"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
assert service.webhook_url == url
|
||||
assert service.is_enabled() is True
|
||||
|
||||
def test_init_without_webhook_url(self):
|
||||
"""Test initialization without webhook URL."""
|
||||
service = DiscordNotificationService()
|
||||
assert service.webhook_url is None
|
||||
assert service.is_enabled() is False
|
||||
|
||||
def test_init_with_dry_run(self):
|
||||
"""Test initialization with dry run mode."""
|
||||
service = DiscordNotificationService(dry_run=True)
|
||||
assert service.dry_run is True
|
||||
|
||||
def test_init_with_invalid_webhook_url_raises(self):
|
||||
"""Test that invalid webhook URL raises ValueError."""
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
DiscordNotificationService(webhook_url="https://example.com/webhook")
|
||||
assert "Invalid Discord webhook URL" in str(exc_info.value)
|
||||
|
||||
def test_init_with_non_discord_url_raises(self):
|
||||
"""Test that non-Discord URL raises ValueError."""
|
||||
with pytest.raises(ValueError):
|
||||
DiscordNotificationService(
|
||||
webhook_url="https://slack.com/api/webhooks/123"
|
||||
)
|
||||
|
||||
|
||||
class TestDiscordNotificationServiceValidation:
|
||||
"""Test webhook URL validation."""
|
||||
|
||||
def test_validate_valid_webhook_url(self):
|
||||
"""Test that valid webhook URL passes validation."""
|
||||
service = DiscordNotificationService()
|
||||
# Should not raise
|
||||
service._validate_webhook_url(
|
||||
"https://discord.com/api/webhooks/123456789/abcdef"
|
||||
)
|
||||
|
||||
def test_validate_invalid_webhook_url(self):
|
||||
"""Test that invalid webhook URL raises ValueError."""
|
||||
service = DiscordNotificationService()
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
service._validate_webhook_url("https://example.com/webhook")
|
||||
assert "Invalid Discord webhook URL" in str(exc_info.value)
|
||||
|
||||
def test_validate_strips_whitespace(self):
|
||||
"""Test that whitespace is stripped from URL."""
|
||||
service = DiscordNotificationService()
|
||||
# Should not raise - whitespace is stripped
|
||||
service._validate_webhook_url(
|
||||
" https://discord.com/api/webhooks/123456789/abcdef "
|
||||
)
|
||||
|
||||
|
||||
class TestDiscordNotificationServicePostMessage:
|
||||
"""Test message posting."""
|
||||
|
||||
@patch("backend.services.discord_service.requests.post")
|
||||
def test_post_message_success(self, mock_post):
|
||||
"""Test successful message posting."""
|
||||
mock_response = MagicMock()
|
||||
mock_response.raise_for_status = MagicMock()
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
|
||||
result = service.post_message("Test message")
|
||||
|
||||
assert result is True
|
||||
mock_post.assert_called_once_with(
|
||||
url,
|
||||
json={"content": "Test message"},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
@patch("backend.services.discord_service.requests.post")
|
||||
def test_post_message_with_custom_webhook(self, mock_post):
|
||||
"""Test message posting with custom webhook URL."""
|
||||
mock_response = MagicMock()
|
||||
mock_response.raise_for_status = MagicMock()
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
service = DiscordNotificationService()
|
||||
custom_url = "https://discord.com/api/webhooks/987654321/fedcba"
|
||||
|
||||
result = service.post_message("Test message", webhook_url=custom_url)
|
||||
|
||||
assert result is True
|
||||
mock_post.assert_called_once_with(
|
||||
custom_url,
|
||||
json={"content": "Test message"},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
def test_post_message_no_webhook_raises(self):
|
||||
"""Test that posting without webhook URL raises ValueError."""
|
||||
service = DiscordNotificationService()
|
||||
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
service.post_message("Test message")
|
||||
assert "No Discord webhook URL provided" in str(exc_info.value)
|
||||
|
||||
def test_post_message_dry_run(self):
|
||||
"""Test message posting in dry run mode."""
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url, dry_run=True)
|
||||
|
||||
result = service.post_message("Test message")
|
||||
|
||||
assert result is True
|
||||
# In dry run mode, no actual request should be made
|
||||
|
||||
@patch("backend.services.discord_service.requests.post")
|
||||
def test_post_message_http_error(self, mock_post):
|
||||
"""Test handling of HTTP errors."""
|
||||
import requests as req
|
||||
mock_response = MagicMock()
|
||||
mock_response.raise_for_status.side_effect = req.HTTPError("404 Not Found")
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
|
||||
with pytest.raises(req.HTTPError):
|
||||
service.post_message("Test message")
|
||||
|
||||
|
||||
class TestDiscordNotificationServiceVideoNotification:
|
||||
"""Test video notification posting."""
|
||||
|
||||
@patch("backend.services.discord_service.requests.post")
|
||||
def test_post_video_notification_success(self, mock_post):
|
||||
"""Test successful video notification."""
|
||||
mock_response = MagicMock()
|
||||
mock_response.raise_for_status = MagicMock()
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
|
||||
result = service.post_video_notification(
|
||||
"https://www.youtube.com/watch?v=abc123"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
mock_post.assert_called_once()
|
||||
call_args = mock_post.call_args
|
||||
assert "New upload:" in call_args[1]["json"]["content"]
|
||||
assert "abc123" in call_args[1]["json"]["content"]
|
||||
|
||||
def test_post_video_notification_no_youtube_url(self):
|
||||
"""Test video notification with empty YouTube URL."""
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
|
||||
result = service.post_video_notification("")
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_post_video_notification_none_youtube_url(self):
|
||||
"""Test video notification with None YouTube URL."""
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
|
||||
result = service.post_video_notification(None)
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_post_video_notification_dry_run(self):
|
||||
"""Test video notification in dry run mode."""
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url, dry_run=True)
|
||||
|
||||
result = service.post_video_notification(
|
||||
"https://www.youtube.com/watch?v=abc123"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
|
||||
|
||||
class TestGetDiscordNotificationService:
|
||||
"""Test factory function."""
|
||||
|
||||
def test_get_service_creates_instance(self):
|
||||
"""Test that factory function creates a new instance."""
|
||||
import backend.services.discord_service as module
|
||||
module._discord_notification_service = None
|
||||
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = get_discord_notification_service(webhook_url=url)
|
||||
|
||||
assert service is not None
|
||||
assert isinstance(service, DiscordNotificationService)
|
||||
assert service.webhook_url == url
|
||||
|
||||
def test_get_service_without_webhook(self):
|
||||
"""Test factory function without webhook URL."""
|
||||
import backend.services.discord_service as module
|
||||
module._discord_notification_service = None
|
||||
|
||||
service = get_discord_notification_service()
|
||||
|
||||
assert service is not None
|
||||
assert service.webhook_url is None
|
||||
assert service.is_enabled() is False
|
||||
|
||||
def test_get_service_with_dry_run(self):
|
||||
"""Test factory function with dry run mode."""
|
||||
import backend.services.discord_service as module
|
||||
module._discord_notification_service = None
|
||||
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = get_discord_notification_service(webhook_url=url, dry_run=True)
|
||||
|
||||
assert service.dry_run is True
|
||||
@@ -321,31 +321,95 @@ class TestUploadFile:
|
||||
|
||||
class TestUploadFolder:
|
||||
"""Test upload_folder method."""
|
||||
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_upload_folder(self, mock_sm_client_class, tmp_path):
|
||||
"""Test uploading a folder with multiple files."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
# Create test folder with files
|
||||
|
||||
# Create test folder with files (no subdirs for this test)
|
||||
(tmp_path / "file1.txt").write_text("content1")
|
||||
(tmp_path / "file2.txt").write_text("content2")
|
||||
(tmp_path / "subdir").mkdir() # Should be skipped
|
||||
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
|
||||
with patch.object(service, "upload_file") as mock_upload:
|
||||
service.upload_folder(str(tmp_path), "/Uploads/folder")
|
||||
|
||||
# Should upload 2 files (not subdir)
|
||||
|
||||
# Should upload 2 files
|
||||
assert mock_upload.call_count == 2
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_upload_folder_recursive(self, mock_sm_client_class, tmp_path):
|
||||
"""Test uploading a folder recursively includes subdirectories."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
# Create test folder with files and subdirectories
|
||||
(tmp_path / "root_file.txt").write_text("root content")
|
||||
(tmp_path / "stems").mkdir()
|
||||
(tmp_path / "stems" / "vocals.flac").write_text("vocals")
|
||||
(tmp_path / "stems" / "instrumental.flac").write_text("instrumental")
|
||||
(tmp_path / "lyrics").mkdir()
|
||||
(tmp_path / "lyrics" / "song.lrc").write_text("lyrics")
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
uploaded_files = []
|
||||
def capture_upload(local_path, remote_path):
|
||||
uploaded_files.append(remote_path)
|
||||
|
||||
with patch.object(service, "upload_file", side_effect=capture_upload) as mock_upload:
|
||||
service.upload_folder(str(tmp_path), "/Uploads/folder")
|
||||
|
||||
# Should upload 4 files (1 root + 2 stems + 1 lyrics)
|
||||
assert mock_upload.call_count == 4
|
||||
|
||||
# Check that subdirectory structure is preserved
|
||||
assert "/Uploads/folder/root_file.txt" in uploaded_files
|
||||
assert "/Uploads/folder/stems/vocals.flac" in uploaded_files
|
||||
assert "/Uploads/folder/stems/instrumental.flac" in uploaded_files
|
||||
assert "/Uploads/folder/lyrics/song.lrc" in uploaded_files
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_upload_folder_deeply_nested(self, mock_sm_client_class, tmp_path):
|
||||
"""Test uploading deeply nested folder structure."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
# Create deeply nested structure
|
||||
(tmp_path / "level1").mkdir()
|
||||
(tmp_path / "level1" / "level2").mkdir()
|
||||
(tmp_path / "level1" / "level2" / "deep_file.txt").write_text("deep")
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
uploaded_files = []
|
||||
def capture_upload(local_path, remote_path):
|
||||
uploaded_files.append(remote_path)
|
||||
|
||||
with patch.object(service, "upload_file", side_effect=capture_upload):
|
||||
service.upload_folder(str(tmp_path), "/Uploads/folder")
|
||||
|
||||
# Check deeply nested file is uploaded with correct path
|
||||
assert "/Uploads/folder/level1/level2/deep_file.txt" in uploaded_files
|
||||
|
||||
|
||||
class TestCreateSharedLink:
|
||||
"""Test create_shared_link method."""
|
||||
|
||||
@@ -0,0 +1,492 @@
|
||||
"""
|
||||
Unit tests for email service.
|
||||
|
||||
Tests the new job completion and action reminder email methods,
|
||||
as well as CC support.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, MagicMock
|
||||
|
||||
from backend.services.email_service import (
|
||||
EmailService,
|
||||
EmailProvider,
|
||||
ConsoleEmailProvider,
|
||||
SendGridEmailProvider,
|
||||
get_email_service,
|
||||
)
|
||||
|
||||
|
||||
class TestConsoleEmailProvider:
|
||||
"""Tests for console email provider."""
|
||||
|
||||
def test_send_email_basic(self):
|
||||
"""Test basic email logging."""
|
||||
provider = ConsoleEmailProvider()
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test content</p>",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
|
||||
def test_send_email_with_text_content(self):
|
||||
"""Test email with text content."""
|
||||
provider = ConsoleEmailProvider()
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
text_content="Test plain text",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
|
||||
def test_send_email_with_cc(self):
|
||||
"""Test email with CC recipients."""
|
||||
provider = ConsoleEmailProvider()
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
cc_emails=["cc1@example.com", "cc2@example.com"],
|
||||
)
|
||||
|
||||
assert result is True
|
||||
|
||||
|
||||
class TestSendGridEmailProvider:
|
||||
"""Tests for SendGrid email provider."""
|
||||
|
||||
def test_send_email_success(self):
|
||||
"""Test successful email sending via SendGrid."""
|
||||
provider = SendGridEmailProvider(
|
||||
api_key="test-api-key",
|
||||
from_email="from@example.com",
|
||||
from_name="Test Sender",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 202
|
||||
|
||||
# Patch at the sendgrid module level since it's imported inline
|
||||
with patch('sendgrid.SendGridAPIClient') as mock_sg:
|
||||
mock_client = Mock()
|
||||
mock_client.send.return_value = mock_response
|
||||
mock_sg.return_value = mock_client
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
mock_client.send.assert_called_once()
|
||||
|
||||
def test_send_email_with_cc(self):
|
||||
"""Test email sending with CC via SendGrid."""
|
||||
provider = SendGridEmailProvider(
|
||||
api_key="test-api-key",
|
||||
from_email="from@example.com",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 202
|
||||
|
||||
with patch('sendgrid.SendGridAPIClient') as mock_sg:
|
||||
mock_client = Mock()
|
||||
mock_client.send.return_value = mock_response
|
||||
mock_sg.return_value = mock_client
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
cc_emails=["cc@example.com"],
|
||||
)
|
||||
|
||||
assert result is True
|
||||
|
||||
def test_send_email_failure_status(self):
|
||||
"""Test email sending failure due to bad status."""
|
||||
provider = SendGridEmailProvider(
|
||||
api_key="test-api-key",
|
||||
from_email="from@example.com",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 400
|
||||
|
||||
with patch('sendgrid.SendGridAPIClient') as mock_sg:
|
||||
mock_client = Mock()
|
||||
mock_client.send.return_value = mock_response
|
||||
mock_sg.return_value = mock_client
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_send_email_exception(self):
|
||||
"""Test email sending exception handling."""
|
||||
provider = SendGridEmailProvider(
|
||||
api_key="test-api-key",
|
||||
from_email="from@example.com",
|
||||
)
|
||||
|
||||
with patch('sendgrid.SendGridAPIClient') as mock_sg:
|
||||
mock_sg.return_value.send.side_effect = Exception("API error")
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
|
||||
class TestEmailServiceJobCompletion:
|
||||
"""Tests for job completion email method."""
|
||||
|
||||
def test_send_job_completion_basic(self):
|
||||
"""Test basic job completion email."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
result = service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
service.provider.send_email.assert_called_once()
|
||||
|
||||
def test_send_job_completion_with_song_info(self):
|
||||
"""Test job completion email includes song in subject."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
)
|
||||
|
||||
call_args = service.provider.send_email.call_args
|
||||
subject = call_args.kwargs.get('subject') or call_args[0][1]
|
||||
assert "Test Artist" in subject
|
||||
assert "Test Song" in subject
|
||||
|
||||
def test_send_job_completion_with_brand_code(self):
|
||||
"""Test job completion email includes brand code in subject."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
artist="Seether",
|
||||
title="Tonight",
|
||||
brand_code="NOMAD-1178",
|
||||
)
|
||||
|
||||
call_args = service.provider.send_email.call_args
|
||||
subject = call_args.kwargs.get('subject') or call_args[0][1]
|
||||
# Subject format: "NOMAD-1178: Seether - Tonight (Your karaoke video is ready!)"
|
||||
assert subject == "NOMAD-1178: Seether - Tonight (Your karaoke video is ready!)"
|
||||
|
||||
def test_send_job_completion_default_subject(self):
|
||||
"""Test job completion email default subject without song info."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
)
|
||||
|
||||
call_args = service.provider.send_email.call_args
|
||||
subject = call_args.kwargs.get('subject') or call_args[0][1]
|
||||
assert "karaoke video is ready" in subject.lower()
|
||||
|
||||
def test_send_job_completion_with_cc(self):
|
||||
"""Test job completion email with CC to admin."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
cc_admin=True,
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
assert "gen@nomadkaraoke.com" in call_kwargs.get('cc_emails', [])
|
||||
|
||||
def test_send_job_completion_without_cc(self):
|
||||
"""Test job completion email without CC."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
cc_admin=False,
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
assert call_kwargs.get('cc_emails') is None
|
||||
|
||||
def test_send_job_completion_escapes_html(self):
|
||||
"""Test that message content is HTML-escaped."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Test <script>alert('xss')</script>",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
html_content = call_kwargs.get('html_content')
|
||||
assert "<script>" not in html_content
|
||||
assert "<script>" in html_content
|
||||
|
||||
def test_send_job_completion_includes_plain_text(self):
|
||||
"""Test that plain text content is included."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
message = "Your video is ready!"
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content=message,
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
assert call_kwargs.get('text_content') == message
|
||||
|
||||
|
||||
class TestEmailServiceActionReminder:
|
||||
"""Tests for action reminder email method."""
|
||||
|
||||
def test_send_action_reminder_lyrics(self):
|
||||
"""Test lyrics action reminder email."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
result = service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Please review your lyrics",
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
subject = call_kwargs.get('subject')
|
||||
assert "lyrics" in subject.lower()
|
||||
|
||||
def test_send_action_reminder_instrumental(self):
|
||||
"""Test instrumental action reminder email."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
result = service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Please select instrumental",
|
||||
action_type="instrumental",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
subject = call_kwargs.get('subject')
|
||||
assert "instrumental" in subject.lower()
|
||||
|
||||
def test_send_action_reminder_with_song_info(self):
|
||||
"""Test action reminder includes song in subject."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Please review",
|
||||
action_type="lyrics",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
subject = call_kwargs.get('subject')
|
||||
assert "Test Artist" in subject
|
||||
assert "Test Song" in subject
|
||||
|
||||
def test_send_action_reminder_unknown_type(self):
|
||||
"""Test action reminder with unknown type still sends."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
result = service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Please take action",
|
||||
action_type="unknown",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
subject = call_kwargs.get('subject')
|
||||
assert "Action needed" in subject
|
||||
|
||||
def test_send_action_reminder_escapes_html(self):
|
||||
"""Test that message content is HTML-escaped."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Test <script>alert('xss')</script>",
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
html_content = call_kwargs.get('html_content')
|
||||
assert "<script>" not in html_content
|
||||
assert "<script>" in html_content
|
||||
|
||||
def test_send_action_reminder_no_cc(self):
|
||||
"""Test that action reminders don't have CC."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Please review",
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
# Action reminders should not have CC
|
||||
assert call_kwargs.get('cc_emails') is None
|
||||
|
||||
|
||||
class TestEmailServiceConfiguration:
|
||||
"""Tests for email service configuration."""
|
||||
|
||||
def test_uses_sendgrid_when_configured(self):
|
||||
"""Test that SendGrid is used when API key is set."""
|
||||
with patch.dict('os.environ', {'SENDGRID_API_KEY': 'test-key'}):
|
||||
service = EmailService()
|
||||
assert isinstance(service.provider, SendGridEmailProvider)
|
||||
|
||||
def test_uses_console_when_not_configured(self):
|
||||
"""Test that console is used when no API key."""
|
||||
with patch.dict('os.environ', {}, clear=True):
|
||||
# Remove SENDGRID_API_KEY if it exists
|
||||
import os
|
||||
original = os.environ.pop('SENDGRID_API_KEY', None)
|
||||
try:
|
||||
service = EmailService()
|
||||
assert isinstance(service.provider, ConsoleEmailProvider)
|
||||
finally:
|
||||
if original:
|
||||
os.environ['SENDGRID_API_KEY'] = original
|
||||
|
||||
def test_is_configured_true_for_sendgrid(self):
|
||||
"""Test is_configured returns True for SendGrid."""
|
||||
service = EmailService()
|
||||
service.provider = SendGridEmailProvider("key", "from@example.com")
|
||||
assert service.is_configured() is True
|
||||
|
||||
def test_is_configured_false_for_console(self):
|
||||
"""Test is_configured returns False for console."""
|
||||
service = EmailService()
|
||||
service.provider = ConsoleEmailProvider()
|
||||
assert service.is_configured() is False
|
||||
|
||||
|
||||
class TestGlobalInstance:
|
||||
"""Tests for global instance management."""
|
||||
|
||||
def test_get_email_service_returns_same_instance(self):
|
||||
"""Test that get_email_service returns singleton."""
|
||||
# Reset global
|
||||
import backend.services.email_service as es
|
||||
es._email_service = None
|
||||
|
||||
service1 = get_email_service()
|
||||
service2 = get_email_service()
|
||||
|
||||
assert service1 is service2
|
||||
|
||||
|
||||
class TestCCFunctionality:
|
||||
"""Integration tests for CC functionality across providers."""
|
||||
|
||||
def test_console_provider_logs_cc(self, caplog):
|
||||
"""Test that console provider logs CC recipients."""
|
||||
import logging
|
||||
caplog.set_level(logging.INFO)
|
||||
|
||||
provider = ConsoleEmailProvider()
|
||||
provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test",
|
||||
html_content="<p>Test</p>",
|
||||
cc_emails=["cc1@example.com", "cc2@example.com"],
|
||||
)
|
||||
|
||||
# Check that CC was logged
|
||||
assert "cc1@example.com" in caplog.text or "CC:" in caplog.text
|
||||
|
||||
def test_sendgrid_provider_adds_cc_recipients(self):
|
||||
"""Test that SendGrid provider adds CC recipients to message."""
|
||||
provider = SendGridEmailProvider(
|
||||
api_key="test-api-key",
|
||||
from_email="from@example.com",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 202
|
||||
|
||||
# Patch at sendgrid module level since imports are inline
|
||||
with patch('sendgrid.SendGridAPIClient') as mock_sg:
|
||||
mock_client = Mock()
|
||||
mock_client.send.return_value = mock_response
|
||||
mock_sg.return_value = mock_client
|
||||
|
||||
# We need to capture the Mail object
|
||||
with patch('sendgrid.helpers.mail.Mail') as mock_mail_class:
|
||||
mock_mail = MagicMock()
|
||||
mock_mail_class.return_value = mock_mail
|
||||
|
||||
provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test",
|
||||
html_content="<p>Test</p>",
|
||||
cc_emails=["cc@example.com"],
|
||||
)
|
||||
|
||||
# Verify add_cc was called
|
||||
mock_mail.add_cc.assert_called()
|
||||
@@ -0,0 +1,412 @@
|
||||
"""
|
||||
Tests for Encoding Interface.
|
||||
|
||||
Tests cover:
|
||||
- EncodingInput and EncodingOutput dataclasses
|
||||
- LocalEncodingBackend implementation
|
||||
- Backend factory function
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch, AsyncMock
|
||||
|
||||
from backend.services.encoding_interface import (
|
||||
EncodingInput,
|
||||
EncodingOutput,
|
||||
EncodingBackend,
|
||||
LocalEncodingBackend,
|
||||
GCEEncodingBackend,
|
||||
get_encoding_backend,
|
||||
)
|
||||
|
||||
|
||||
class TestEncodingInput:
|
||||
"""Test EncodingInput dataclass."""
|
||||
|
||||
def test_required_fields(self):
|
||||
"""Test that required fields must be provided."""
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
)
|
||||
assert input_config.title_video_path == "/path/title.mov"
|
||||
assert input_config.end_video_path is None
|
||||
|
||||
def test_all_fields(self):
|
||||
"""Test all fields including optional ones."""
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
end_video_path="/path/end.mov",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
brand_code="NOMAD-1234",
|
||||
output_dir="/output",
|
||||
options={"quality": "high"}
|
||||
)
|
||||
assert input_config.artist == "Test Artist"
|
||||
assert input_config.brand_code == "NOMAD-1234"
|
||||
assert input_config.options["quality"] == "high"
|
||||
|
||||
|
||||
class TestEncodingOutput:
|
||||
"""Test EncodingOutput dataclass."""
|
||||
|
||||
def test_success_output(self):
|
||||
"""Test successful output."""
|
||||
output = EncodingOutput(
|
||||
success=True,
|
||||
lossless_4k_mp4_path="/output/video.mp4",
|
||||
encoding_backend="local"
|
||||
)
|
||||
assert output.success is True
|
||||
assert output.error_message is None
|
||||
|
||||
def test_failure_output(self):
|
||||
"""Test failure output."""
|
||||
output = EncodingOutput(
|
||||
success=False,
|
||||
error_message="Encoding failed",
|
||||
encoding_backend="gce"
|
||||
)
|
||||
assert output.success is False
|
||||
assert output.error_message == "Encoding failed"
|
||||
|
||||
def test_output_files_dict(self):
|
||||
"""Test output_files dictionary."""
|
||||
output = EncodingOutput(
|
||||
success=True,
|
||||
output_files={
|
||||
"lossless_4k_mp4": "/output/lossless.mp4",
|
||||
"720p_mp4": "/output/720p.mp4"
|
||||
}
|
||||
)
|
||||
assert "lossless_4k_mp4" in output.output_files
|
||||
assert output.output_files["720p_mp4"] == "/output/720p.mp4"
|
||||
|
||||
|
||||
class TestLocalEncodingBackend:
|
||||
"""Test LocalEncodingBackend implementation."""
|
||||
|
||||
def test_name(self):
|
||||
"""Test backend name."""
|
||||
backend = LocalEncodingBackend()
|
||||
assert backend.name == "local"
|
||||
|
||||
def test_init_with_dry_run(self):
|
||||
"""Test initialization with dry run."""
|
||||
backend = LocalEncodingBackend(dry_run=True)
|
||||
assert backend.dry_run is True
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch("subprocess.run")
|
||||
async def test_is_available_success(self, mock_run):
|
||||
"""Test availability check when FFmpeg is installed."""
|
||||
mock_run.return_value = MagicMock(returncode=0)
|
||||
|
||||
backend = LocalEncodingBackend()
|
||||
available = await backend.is_available()
|
||||
|
||||
assert available is True
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch("subprocess.run")
|
||||
async def test_is_available_not_found(self, mock_run):
|
||||
"""Test availability check when FFmpeg is not installed."""
|
||||
mock_run.side_effect = FileNotFoundError()
|
||||
|
||||
backend = LocalEncodingBackend()
|
||||
available = await backend.is_available()
|
||||
|
||||
assert available is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch.object(LocalEncodingBackend, "is_available")
|
||||
@patch.object(LocalEncodingBackend, "_get_service")
|
||||
async def test_get_status(self, mock_get_service, mock_is_available):
|
||||
"""Test status retrieval."""
|
||||
mock_is_available.return_value = True
|
||||
mock_service = MagicMock()
|
||||
mock_service.hwaccel_available = True
|
||||
mock_service.video_encoder = "h264_nvenc"
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = LocalEncodingBackend()
|
||||
status = await backend.get_status()
|
||||
|
||||
assert status["backend"] == "local"
|
||||
assert status["available"] is True
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch("asyncio.to_thread")
|
||||
@patch.object(LocalEncodingBackend, "_get_service")
|
||||
async def test_encode_success(self, mock_get_service, mock_to_thread):
|
||||
"""Test successful encoding."""
|
||||
from backend.services.local_encoding_service import EncodingResult
|
||||
|
||||
mock_result = EncodingResult(
|
||||
success=True,
|
||||
output_files={"key": "/path/file.mp4"}
|
||||
)
|
||||
mock_to_thread.return_value = mock_result
|
||||
|
||||
mock_service = MagicMock()
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = LocalEncodingBackend()
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/input/title.mov",
|
||||
karaoke_video_path="/input/karaoke.mov",
|
||||
instrumental_audio_path="/input/audio.flac",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
output_dir="/output"
|
||||
)
|
||||
|
||||
output = await backend.encode(input_config)
|
||||
|
||||
assert output.success is True
|
||||
assert output.encoding_backend == "local"
|
||||
assert output.encoding_time_seconds is not None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch("asyncio.to_thread")
|
||||
@patch.object(LocalEncodingBackend, "_get_service")
|
||||
async def test_encode_failure(self, mock_get_service, mock_to_thread):
|
||||
"""Test encoding failure."""
|
||||
from backend.services.local_encoding_service import EncodingResult
|
||||
|
||||
mock_result = EncodingResult(
|
||||
success=False,
|
||||
output_files={},
|
||||
error="FFmpeg failed"
|
||||
)
|
||||
mock_to_thread.return_value = mock_result
|
||||
|
||||
mock_service = MagicMock()
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = LocalEncodingBackend()
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/input/title.mov",
|
||||
karaoke_video_path="/input/karaoke.mov",
|
||||
instrumental_audio_path="/input/audio.flac",
|
||||
artist="Test",
|
||||
title="Test",
|
||||
)
|
||||
|
||||
output = await backend.encode(input_config)
|
||||
|
||||
assert output.success is False
|
||||
assert output.error_message == "FFmpeg failed"
|
||||
|
||||
|
||||
class TestGCEEncodingBackend:
|
||||
"""Test GCEEncodingBackend implementation."""
|
||||
|
||||
def test_name(self):
|
||||
"""Test backend name."""
|
||||
backend = GCEEncodingBackend()
|
||||
assert backend.name == "gce"
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
@pytest.mark.asyncio
|
||||
async def test_is_available_enabled(self, mock_get_service):
|
||||
"""Test availability when GCE is enabled."""
|
||||
mock_service = MagicMock()
|
||||
mock_service.is_enabled = True
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = GCEEncodingBackend()
|
||||
available = await backend.is_available()
|
||||
|
||||
assert available is True
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
@pytest.mark.asyncio
|
||||
async def test_is_available_disabled(self, mock_get_service):
|
||||
"""Test availability when GCE is disabled."""
|
||||
mock_service = MagicMock()
|
||||
mock_service.is_enabled = False
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = GCEEncodingBackend()
|
||||
available = await backend.is_available()
|
||||
|
||||
assert available is False
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
@pytest.mark.asyncio
|
||||
async def test_encode_missing_gcs_paths(self, mock_get_service):
|
||||
"""Test encoding fails without GCS paths."""
|
||||
backend = GCEEncodingBackend()
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/input/title.mov",
|
||||
karaoke_video_path="/input/karaoke.mov",
|
||||
instrumental_audio_path="/input/audio.flac",
|
||||
# Missing GCS paths in options
|
||||
)
|
||||
|
||||
output = await backend.encode(input_config)
|
||||
|
||||
assert output.success is False
|
||||
assert "gcs_path" in output.error_message.lower()
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
@pytest.mark.asyncio
|
||||
async def test_encode_success(self, mock_get_service):
|
||||
"""Test successful GCE encoding."""
|
||||
mock_service = MagicMock()
|
||||
mock_service.encode_videos = AsyncMock(return_value={
|
||||
"status": "complete",
|
||||
"output_files": {
|
||||
"mp4_4k_lossless": "gs://bucket/output/lossless.mp4",
|
||||
"mp4_720p": "gs://bucket/output/720p.mp4",
|
||||
}
|
||||
})
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = GCEEncodingBackend()
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/input/title.mov",
|
||||
karaoke_video_path="/input/karaoke.mov",
|
||||
instrumental_audio_path="/input/audio.flac",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
options={
|
||||
"job_id": "test-job",
|
||||
"input_gcs_path": "gs://bucket/input/",
|
||||
"output_gcs_path": "gs://bucket/output/",
|
||||
}
|
||||
)
|
||||
|
||||
output = await backend.encode(input_config)
|
||||
|
||||
assert output.success is True
|
||||
assert output.encoding_backend == "gce"
|
||||
mock_service.encode_videos.assert_called_once()
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
@pytest.mark.asyncio
|
||||
async def test_encode_failure(self, mock_get_service):
|
||||
"""Test GCE encoding failure handling."""
|
||||
mock_service = MagicMock()
|
||||
mock_service.encode_videos = AsyncMock(side_effect=Exception("GCE worker error"))
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = GCEEncodingBackend()
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/input/title.mov",
|
||||
karaoke_video_path="/input/karaoke.mov",
|
||||
instrumental_audio_path="/input/audio.flac",
|
||||
options={
|
||||
"job_id": "test-job",
|
||||
"input_gcs_path": "gs://bucket/input/",
|
||||
"output_gcs_path": "gs://bucket/output/",
|
||||
}
|
||||
)
|
||||
|
||||
output = await backend.encode(input_config)
|
||||
|
||||
assert output.success is False
|
||||
assert "GCE worker error" in output.error_message
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
@pytest.mark.asyncio
|
||||
async def test_encode_handles_list_result(self, mock_get_service):
|
||||
"""Test GCE encoding handles list response gracefully.
|
||||
|
||||
This would have caught: 'list' object has no attribute 'get' error
|
||||
when GCE worker returns a list instead of a dict.
|
||||
"""
|
||||
mock_service = MagicMock()
|
||||
# Simulate GCE worker returning a list instead of dict
|
||||
mock_service.encode_videos = AsyncMock(return_value=[
|
||||
{"output_files": {"mp4_4k_lossless": "gs://bucket/output/lossless.mp4"}}
|
||||
])
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = GCEEncodingBackend()
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/input/title.mov",
|
||||
karaoke_video_path="/input/karaoke.mov",
|
||||
instrumental_audio_path="/input/audio.flac",
|
||||
options={
|
||||
"job_id": "test-job",
|
||||
"input_gcs_path": "gs://bucket/input/",
|
||||
"output_gcs_path": "gs://bucket/output/",
|
||||
}
|
||||
)
|
||||
|
||||
# This should not raise an error
|
||||
output = await backend.encode(input_config)
|
||||
|
||||
# Should still succeed by extracting from the list
|
||||
assert output.success is True
|
||||
assert output.lossless_4k_mp4_path == "gs://bucket/output/lossless.mp4"
|
||||
|
||||
|
||||
class TestGetEncodingBackend:
|
||||
"""Test encoding backend factory function."""
|
||||
|
||||
def test_get_local_backend(self):
|
||||
"""Test getting local backend."""
|
||||
backend = get_encoding_backend("local")
|
||||
assert isinstance(backend, LocalEncodingBackend)
|
||||
assert backend.name == "local"
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
def test_get_auto_backend_gce_disabled(self, mock_get_service):
|
||||
"""Test getting auto backend falls back to local when GCE disabled."""
|
||||
mock_service = MagicMock()
|
||||
mock_service.is_enabled = False
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = get_encoding_backend("auto")
|
||||
assert isinstance(backend, LocalEncodingBackend)
|
||||
|
||||
def test_get_local_backend_with_options(self):
|
||||
"""Test getting local backend with options."""
|
||||
backend = get_encoding_backend("local", dry_run=True)
|
||||
assert backend.dry_run is True
|
||||
|
||||
def test_get_gce_backend(self):
|
||||
"""Test getting GCE backend."""
|
||||
backend = get_encoding_backend("gce")
|
||||
assert isinstance(backend, GCEEncodingBackend)
|
||||
assert backend.name == "gce"
|
||||
|
||||
def test_get_unknown_backend_raises(self):
|
||||
"""Test that unknown backend raises ValueError."""
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
get_encoding_backend("unknown")
|
||||
assert "Unknown encoding backend type" in str(exc_info.value)
|
||||
|
||||
def test_get_gce_backend_with_options(self):
|
||||
"""Test getting GCE backend with common options like dry_run.
|
||||
|
||||
This ensures all backends accept the same kwargs, preventing
|
||||
errors when get_encoding_backend() passes **kwargs to different backends.
|
||||
"""
|
||||
# This would have caught: GCEEncodingBackend.__init__() got an unexpected keyword argument 'dry_run'
|
||||
backend = get_encoding_backend("gce", dry_run=True)
|
||||
assert isinstance(backend, GCEEncodingBackend)
|
||||
assert backend.dry_run is True
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
def test_all_backends_accept_dry_run(self, mock_get_service):
|
||||
"""Test that all backend types accept dry_run parameter.
|
||||
|
||||
This is an integration test to ensure the factory function
|
||||
can pass dry_run to any backend without TypeError.
|
||||
"""
|
||||
mock_service = MagicMock()
|
||||
mock_service.is_enabled = False
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
for backend_type in ["local", "gce", "auto"]:
|
||||
# This should not raise TypeError
|
||||
backend = get_encoding_backend(backend_type, dry_run=True)
|
||||
assert backend is not None
|
||||
@@ -1128,7 +1128,8 @@ class TestFinaliseOnlyModels:
|
||||
assert request.enable_txt is None
|
||||
assert request.brand_prefix is None
|
||||
assert request.keep_brand_code is None
|
||||
assert request.enable_youtube_upload is False
|
||||
# YouTube upload default is None (server applies default_enable_youtube_upload)
|
||||
assert request.enable_youtube_upload is None
|
||||
assert request.dropbox_path is None
|
||||
assert request.gdrive_folder_id is None
|
||||
|
||||
@@ -1606,5 +1607,133 @@ class TestIsUrlFunction:
|
||||
assert is_url("") is False
|
||||
|
||||
|
||||
class TestUploadEndpointThemeSupport:
|
||||
"""Test that /jobs/upload endpoint supports theme configuration.
|
||||
|
||||
CRITICAL: These tests verify that the /api/jobs/upload endpoint correctly handles
|
||||
theme_id and color_overrides parameters, ensuring preview videos have themed
|
||||
backgrounds instead of black backgrounds.
|
||||
|
||||
This addresses a bug where:
|
||||
- The frontend sends theme_id and color_overrides when uploading files
|
||||
- But the backend /api/jobs/upload endpoint was ignoring these parameters
|
||||
- Result: Jobs created via file upload had black backgrounds instead of themed ones
|
||||
"""
|
||||
|
||||
def test_upload_endpoint_accepts_theme_id_parameter(self):
|
||||
"""Verify the upload endpoint has theme_id as a form parameter.
|
||||
|
||||
CRITICAL: The frontend sends theme_id when uploading files with a theme.
|
||||
If this parameter is missing from the endpoint, the theme is silently ignored
|
||||
and preview videos will have black backgrounds instead of themed ones.
|
||||
"""
|
||||
from backend.api.routes import file_upload as file_upload_module
|
||||
|
||||
with open(file_upload_module.__file__, 'r') as f:
|
||||
source_code = f.read()
|
||||
|
||||
has_theme_id_param = 'theme_id: Optional[str] = Form(' in source_code
|
||||
|
||||
assert has_theme_id_param, (
|
||||
"file_upload.py /jobs/upload endpoint does not have theme_id as a Form parameter. "
|
||||
"The frontend sends theme_id when uploading files with a theme, but the backend "
|
||||
"ignores it. Add: theme_id: Optional[str] = Form(None, description='Theme ID...')"
|
||||
)
|
||||
|
||||
def test_upload_endpoint_accepts_color_overrides_parameter(self):
|
||||
"""Verify the upload endpoint has color_overrides as a form parameter."""
|
||||
from backend.api.routes import file_upload as file_upload_module
|
||||
|
||||
with open(file_upload_module.__file__, 'r') as f:
|
||||
source_code = f.read()
|
||||
|
||||
has_color_overrides_param = 'color_overrides: Optional[str] = Form(' in source_code
|
||||
|
||||
assert has_color_overrides_param, (
|
||||
"file_upload.py /jobs/upload endpoint does not have color_overrides as a Form parameter. "
|
||||
"The frontend sends color_overrides when customizing theme colors."
|
||||
)
|
||||
|
||||
def test_upload_endpoint_calls_prepare_theme_for_job(self):
|
||||
"""Verify the upload endpoint calls _prepare_theme_for_job when theme_id is set.
|
||||
|
||||
CRITICAL: When a job is created via the upload endpoint with a theme_id
|
||||
(and no custom style files), the code must call _prepare_theme_for_job() to set:
|
||||
1. style_params_gcs_path (pointing to the copied style_params.json)
|
||||
2. style_assets (populated with asset mappings)
|
||||
|
||||
Without this, LyricsTranscriber won't have access to the theme's styles
|
||||
and preview videos will have black backgrounds instead of themed ones.
|
||||
"""
|
||||
from backend.api.routes import file_upload as file_upload_module
|
||||
|
||||
with open(file_upload_module.__file__, 'r') as f:
|
||||
source_code = f.read()
|
||||
|
||||
# The function should be called in upload_and_create_job
|
||||
has_theme_prep_call = '_prepare_theme_for_job(' in source_code
|
||||
|
||||
assert has_theme_prep_call, (
|
||||
"file_upload.py does not call _prepare_theme_for_job(). "
|
||||
"When theme_id is provided to /jobs/upload without custom style files, "
|
||||
"the endpoint MUST call _prepare_theme_for_job() to copy the theme's "
|
||||
"style_params.json to the job folder."
|
||||
)
|
||||
|
||||
def test_upload_endpoint_uses_resolve_cdg_txt_defaults(self):
|
||||
"""Verify the upload endpoint uses _resolve_cdg_txt_defaults for theme-based defaults."""
|
||||
from backend.api.routes import file_upload as file_upload_module
|
||||
|
||||
with open(file_upload_module.__file__, 'r') as f:
|
||||
source_code = f.read()
|
||||
|
||||
has_resolve_call = '_resolve_cdg_txt_defaults(' in source_code
|
||||
|
||||
assert has_resolve_call, (
|
||||
"file_upload.py does not call _resolve_cdg_txt_defaults(). "
|
||||
"When theme_id is set, enable_cdg and enable_txt should default to True."
|
||||
)
|
||||
|
||||
def test_upload_endpoint_has_optional_cdg_txt_params(self):
|
||||
"""Verify enable_cdg and enable_txt are Optional[bool] to support theme defaults.
|
||||
|
||||
CRITICAL: If enable_cdg/enable_txt are bool instead of Optional[bool],
|
||||
they will default to False and override the theme-based defaults.
|
||||
"""
|
||||
from backend.api.routes import file_upload as file_upload_module
|
||||
|
||||
with open(file_upload_module.__file__, 'r') as f:
|
||||
source_code = f.read()
|
||||
|
||||
has_optional_cdg = 'enable_cdg: Optional[bool] = Form(' in source_code
|
||||
has_optional_txt = 'enable_txt: Optional[bool] = Form(' in source_code
|
||||
|
||||
assert has_optional_cdg, (
|
||||
"file_upload.py has enable_cdg as bool instead of Optional[bool]. "
|
||||
"This prevents theme-based defaults from working."
|
||||
)
|
||||
assert has_optional_txt, (
|
||||
"file_upload.py has enable_txt as bool instead of Optional[bool]. "
|
||||
"This prevents theme-based defaults from working."
|
||||
)
|
||||
|
||||
def test_job_create_includes_theme_id_and_color_overrides(self):
|
||||
"""Verify JobCreate is called with theme_id and color_overrides."""
|
||||
from backend.api.routes import file_upload as file_upload_module
|
||||
|
||||
with open(file_upload_module.__file__, 'r') as f:
|
||||
source_code = f.read()
|
||||
|
||||
has_theme_id_in_job_create = 'theme_id=theme_id,' in source_code
|
||||
has_color_overrides_in_job_create = 'color_overrides=parsed_color_overrides' in source_code
|
||||
|
||||
assert has_theme_id_in_job_create, (
|
||||
"file_upload.py does not pass theme_id to JobCreate."
|
||||
)
|
||||
assert has_color_overrides_in_job_create, (
|
||||
"file_upload.py does not pass color_overrides to JobCreate."
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
@@ -0,0 +1,583 @@
|
||||
"""
|
||||
Regression tests for job creation issues discovered 2024-12-31.
|
||||
|
||||
These tests ensure:
|
||||
1. user_email is properly extracted from AuthResult and set on jobs
|
||||
2. URL jobs trigger workers sequentially (audio first, then lyrics)
|
||||
3. Audio search downloads work without relying on in-memory cache
|
||||
4. Transcription has proper timeout handling
|
||||
|
||||
See docs/archive/2024-12-31-job-failure-investigation.md for details.
|
||||
"""
|
||||
import asyncio
|
||||
import pytest
|
||||
from unittest.mock import Mock, AsyncMock, patch, MagicMock
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from backend.services.auth_service import AuthResult, UserType
|
||||
|
||||
|
||||
class TestUserEmailExtraction:
|
||||
"""
|
||||
Issue 1: user_email not being set on jobs.
|
||||
|
||||
Root cause: Job creation endpoints used require_auth but didn't extract
|
||||
auth_result.user_email and set it on the job.
|
||||
|
||||
These tests verify that authenticated user's email is properly set on jobs.
|
||||
"""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_auth_result_with_email(self):
|
||||
"""AuthResult for a logged-in user with email."""
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.UNLIMITED, # Session-based users typically have UNLIMITED
|
||||
remaining_uses=-1,
|
||||
message="Valid session",
|
||||
user_email="testuser@example.com",
|
||||
is_admin=False,
|
||||
)
|
||||
|
||||
@pytest.fixture
|
||||
def mock_auth_result_admin_token(self):
|
||||
"""AuthResult for admin token without email."""
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=-1,
|
||||
message="Valid admin token",
|
||||
user_email=None,
|
||||
is_admin=True,
|
||||
)
|
||||
|
||||
def test_auth_result_has_user_email_field(self):
|
||||
"""AuthResult must have user_email field for job association."""
|
||||
result = AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.UNLIMITED,
|
||||
remaining_uses=-1,
|
||||
message="test",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
assert hasattr(result, 'user_email')
|
||||
assert result.user_email == "user@example.com"
|
||||
|
||||
def test_auth_result_user_email_can_be_none(self):
|
||||
"""AuthResult user_email can be None for token-based auth."""
|
||||
result = AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=-1,
|
||||
message="test",
|
||||
user_email=None,
|
||||
)
|
||||
assert result.user_email is None
|
||||
|
||||
def test_upload_endpoint_extracts_user_email_from_auth(self):
|
||||
"""
|
||||
/jobs/upload endpoint must set user_email from authenticated user.
|
||||
|
||||
This verifies the code pattern by inspecting the source.
|
||||
Integration tests verify the full behavior.
|
||||
"""
|
||||
from backend.api.routes import file_upload
|
||||
import inspect
|
||||
|
||||
# Get the source code of upload_and_create_job
|
||||
source = inspect.getsource(file_upload.upload_and_create_job)
|
||||
|
||||
# Verify the endpoint extracts user_email from auth_result
|
||||
assert 'auth_result.user_email' in source, \
|
||||
"upload_and_create_job must extract user_email from auth_result"
|
||||
assert 'effective_user_email' in source, \
|
||||
"upload_and_create_job must use effective_user_email pattern"
|
||||
assert 'user_email=effective_user_email' in source or 'user_email=' in source, \
|
||||
"upload_and_create_job must pass user_email to JobCreate"
|
||||
|
||||
def test_all_job_creation_endpoints_extract_user_email(self):
|
||||
"""
|
||||
All job creation endpoints must extract user_email from AuthResult.
|
||||
|
||||
This is a regression guard for Issue 1 (user_email not being set).
|
||||
"""
|
||||
from backend.api.routes import file_upload, audio_search
|
||||
import inspect
|
||||
|
||||
endpoints_to_check = [
|
||||
(file_upload, 'upload_and_create_job'),
|
||||
(file_upload, 'create_job_from_url'),
|
||||
(audio_search, 'search_audio'),
|
||||
]
|
||||
|
||||
for module, func_name in endpoints_to_check:
|
||||
func = getattr(module, func_name)
|
||||
source = inspect.getsource(func)
|
||||
|
||||
assert 'auth_result.user_email' in source, \
|
||||
f"{func_name} must extract user_email from auth_result"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_create_from_url_endpoint_sets_user_email(self, mock_auth_result_with_email):
|
||||
"""
|
||||
/jobs/create-from-url endpoint must set user_email from authenticated user.
|
||||
"""
|
||||
from backend.api.routes.file_upload import create_job_from_url, CreateJobFromUrlRequest
|
||||
|
||||
mock_request = Mock()
|
||||
mock_request.headers = {}
|
||||
mock_request.client = Mock(host="127.0.0.1")
|
||||
mock_request.url = Mock(path="/api/jobs/create-from-url")
|
||||
|
||||
mock_background_tasks = Mock()
|
||||
body = CreateJobFromUrlRequest(
|
||||
url="https://www.youtube.com/watch?v=dQw4w9WgXcQ",
|
||||
artist="Rick Astley",
|
||||
title="Never Gonna Give You Up",
|
||||
)
|
||||
|
||||
with patch('backend.api.routes.file_upload.job_manager') as mock_jm, \
|
||||
patch('backend.api.routes.file_upload.worker_service') as mock_worker, \
|
||||
patch('backend.api.routes.file_upload.get_credential_manager') as mock_cred:
|
||||
|
||||
mock_job = Mock()
|
||||
mock_job.job_id = "test-job-456"
|
||||
mock_jm.create_job.return_value = mock_job
|
||||
mock_cred.return_value.check_youtube_credentials.return_value = Mock(status=Mock(value="valid"))
|
||||
|
||||
await create_job_from_url(
|
||||
request=mock_request,
|
||||
background_tasks=mock_background_tasks,
|
||||
body=body,
|
||||
auth_result=mock_auth_result_with_email,
|
||||
)
|
||||
|
||||
mock_jm.create_job.assert_called_once()
|
||||
job_create_arg = mock_jm.create_job.call_args[0][0]
|
||||
assert job_create_arg.user_email == "testuser@example.com", \
|
||||
"create-from-url must set user_email from AuthResult"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_audio_search_endpoint_sets_user_email(self, mock_auth_result_with_email):
|
||||
"""
|
||||
/audio-search/search endpoint must set user_email from authenticated user.
|
||||
"""
|
||||
from backend.api.routes.audio_search import search_audio, AudioSearchRequest
|
||||
|
||||
mock_request = Mock()
|
||||
mock_request.headers = {}
|
||||
mock_request.client = Mock(host="127.0.0.1")
|
||||
mock_request.url = Mock(path="/api/audio-search/search")
|
||||
|
||||
mock_background_tasks = Mock()
|
||||
body = AudioSearchRequest(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
)
|
||||
|
||||
with patch('backend.api.routes.audio_search.job_manager') as mock_jm, \
|
||||
patch('backend.api.routes.audio_search.get_audio_search_service') as mock_search, \
|
||||
patch('backend.api.routes.audio_search.get_credential_manager') as mock_cred:
|
||||
|
||||
mock_job = Mock()
|
||||
mock_job.job_id = "test-job-789"
|
||||
mock_job.state_data = {}
|
||||
mock_jm.create_job.return_value = mock_job
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_cred.return_value.check_youtube_credentials.return_value = Mock(status=Mock(value="valid"))
|
||||
mock_search.return_value.search.return_value = []
|
||||
|
||||
await search_audio(
|
||||
request=mock_request,
|
||||
background_tasks=mock_background_tasks,
|
||||
body=body,
|
||||
auth_result=mock_auth_result_with_email,
|
||||
)
|
||||
|
||||
mock_jm.create_job.assert_called_once()
|
||||
job_create_arg = mock_jm.create_job.call_args[0][0]
|
||||
assert job_create_arg.user_email == "testuser@example.com", \
|
||||
"audio-search must set user_email from AuthResult"
|
||||
|
||||
def test_effective_user_email_prefers_auth_over_form(self):
|
||||
"""
|
||||
When both auth_result.user_email and form user_email are provided,
|
||||
the authenticated user's email should take precedence.
|
||||
"""
|
||||
auth_email = "authenticated@example.com"
|
||||
form_email = "form@example.com"
|
||||
|
||||
# This is the logic used in the endpoints
|
||||
effective_user_email = auth_email or form_email
|
||||
|
||||
assert effective_user_email == auth_email, \
|
||||
"Authenticated user's email should take precedence over form parameter"
|
||||
|
||||
def test_effective_user_email_falls_back_to_form(self):
|
||||
"""
|
||||
When auth_result.user_email is None (e.g., admin token),
|
||||
form user_email should be used as fallback.
|
||||
"""
|
||||
auth_email = None
|
||||
form_email = "form@example.com"
|
||||
|
||||
effective_user_email = auth_email or form_email
|
||||
|
||||
assert effective_user_email == form_email, \
|
||||
"Form parameter should be used when auth has no email"
|
||||
|
||||
|
||||
class TestUrlJobWorkerSequencing:
|
||||
"""
|
||||
Issue 2: YouTube URL download race condition.
|
||||
|
||||
Root cause: Both audio and lyrics workers were triggered in parallel.
|
||||
For URL jobs, lyrics worker would timeout waiting for audio to download.
|
||||
|
||||
These tests verify that URL jobs trigger workers sequentially.
|
||||
"""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_url_job_triggers_only_audio_worker_initially(self):
|
||||
"""
|
||||
create-from-url must only trigger audio worker, not lyrics worker.
|
||||
The audio worker will trigger lyrics after download completes.
|
||||
"""
|
||||
from backend.api.routes.file_upload import _trigger_audio_worker_only
|
||||
|
||||
with patch('backend.api.routes.file_upload.worker_service') as mock_ws:
|
||||
mock_ws.trigger_audio_worker = AsyncMock()
|
||||
mock_ws.trigger_lyrics_worker = AsyncMock()
|
||||
|
||||
await _trigger_audio_worker_only("test-job-id")
|
||||
|
||||
mock_ws.trigger_audio_worker.assert_called_once_with("test-job-id")
|
||||
mock_ws.trigger_lyrics_worker.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_parallel_worker_triggers_both_workers(self):
|
||||
"""
|
||||
_trigger_workers_parallel should trigger both workers (for uploaded files).
|
||||
"""
|
||||
from backend.api.routes.file_upload import _trigger_workers_parallel
|
||||
|
||||
with patch('backend.api.routes.file_upload.worker_service') as mock_ws:
|
||||
mock_ws.trigger_audio_worker = AsyncMock()
|
||||
mock_ws.trigger_lyrics_worker = AsyncMock()
|
||||
|
||||
await _trigger_workers_parallel("test-job-id")
|
||||
|
||||
mock_ws.trigger_audio_worker.assert_called_once_with("test-job-id")
|
||||
mock_ws.trigger_lyrics_worker.assert_called_once_with("test-job-id")
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_audio_worker_triggers_lyrics_after_url_download(self):
|
||||
"""
|
||||
Audio worker must trigger lyrics worker after successful URL download.
|
||||
"""
|
||||
from backend.workers.audio_worker import _trigger_lyrics_worker_after_url_download
|
||||
|
||||
# Mock at the source module where it's imported from
|
||||
with patch('backend.services.worker_service.get_worker_service') as mock_get_ws:
|
||||
mock_ws = Mock()
|
||||
mock_ws.trigger_lyrics_worker = AsyncMock()
|
||||
mock_get_ws.return_value = mock_ws
|
||||
|
||||
await _trigger_lyrics_worker_after_url_download("test-job-id")
|
||||
|
||||
mock_ws.trigger_lyrics_worker.assert_called_once_with("test-job-id")
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_audio_worker_lyrics_trigger_handles_errors_gracefully(self):
|
||||
"""
|
||||
If lyrics worker trigger fails, audio processing should continue.
|
||||
"""
|
||||
from backend.workers.audio_worker import _trigger_lyrics_worker_after_url_download
|
||||
|
||||
# Mock at the source module where it's imported from
|
||||
with patch('backend.services.worker_service.get_worker_service') as mock_get_ws:
|
||||
mock_ws = Mock()
|
||||
mock_ws.trigger_lyrics_worker = AsyncMock(side_effect=Exception("Network error"))
|
||||
mock_get_ws.return_value = mock_ws
|
||||
|
||||
# Should not raise exception
|
||||
await _trigger_lyrics_worker_after_url_download("test-job-id")
|
||||
|
||||
|
||||
class TestAudioSearchCacheIndependence:
|
||||
"""
|
||||
Issue 3: Audio search cache not persisting across Cloud Run instances.
|
||||
|
||||
Root cause: AudioSearchService used in-memory cache that doesn't persist
|
||||
across horizontally scaled instances.
|
||||
|
||||
These tests verify downloads work using state_data, not in-memory cache.
|
||||
"""
|
||||
|
||||
def test_search_results_stored_in_job_state_data(self):
|
||||
"""
|
||||
Search results must be stored in job.state_data for persistence.
|
||||
This tests the Job model's ability to store search results.
|
||||
"""
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
job = Job(
|
||||
job_id="test-123",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
state_data={
|
||||
'audio_search_results': [
|
||||
{
|
||||
'provider': 'YouTube',
|
||||
'title': 'Test Song',
|
||||
'artist': 'Test Artist',
|
||||
'url': 'https://youtube.com/watch?v=abc123',
|
||||
'source_id': 'abc123',
|
||||
}
|
||||
]
|
||||
}
|
||||
)
|
||||
|
||||
assert 'audio_search_results' in job.state_data
|
||||
assert len(job.state_data['audio_search_results']) == 1
|
||||
assert job.state_data['audio_search_results'][0]['url'] == 'https://youtube.com/watch?v=abc123'
|
||||
|
||||
def test_audio_search_endpoint_stores_results_in_state_data(self):
|
||||
"""
|
||||
Verify that the search_audio code path stores results in job.state_data.
|
||||
|
||||
This tests that the audio_search route has the correct pattern for
|
||||
persisting search results. The actual integration is tested elsewhere.
|
||||
"""
|
||||
# Verify the import and module structure exists
|
||||
from backend.api.routes import audio_search
|
||||
import inspect
|
||||
|
||||
# Get the source code of search_audio
|
||||
source = inspect.getsource(audio_search.search_audio)
|
||||
|
||||
# Verify the endpoint stores results in state_data
|
||||
# The code uses job_manager.update_job with state_data dict
|
||||
assert 'audio_search_results' in source, \
|
||||
"search_audio must store results under 'audio_search_results' key"
|
||||
assert 'state_data' in source, \
|
||||
"search_audio must use state_data for persistence"
|
||||
|
||||
def test_download_code_has_youtube_direct_download_branch(self):
|
||||
"""
|
||||
Verify that _download_and_start_processing has the YouTube direct download path.
|
||||
|
||||
This checks that the code path exists for downloading YouTube audio directly
|
||||
using the URL from state_data, avoiding the in-memory cache dependency.
|
||||
"""
|
||||
from backend.api.routes import audio_search
|
||||
import inspect
|
||||
|
||||
# Get the source of _download_and_start_processing
|
||||
source = inspect.getsource(audio_search._download_and_start_processing)
|
||||
|
||||
# Verify the YouTube direct download branch exists
|
||||
assert "source_name == 'YouTube'" in source, \
|
||||
"_download_and_start_processing must check for YouTube source"
|
||||
assert 'download_url' in source, \
|
||||
"_download_and_start_processing must use download_url from state_data"
|
||||
assert 'download_from_url' in source, \
|
||||
"_download_and_start_processing must call download_from_url for YouTube"
|
||||
|
||||
def test_youtube_download_branch_avoids_cache_dependency(self):
|
||||
"""
|
||||
Verify that the YouTube download path uses URL from state_data,
|
||||
not the in-memory cache that doesn't persist across Cloud Run instances.
|
||||
"""
|
||||
from backend.api.routes import audio_search
|
||||
import inspect
|
||||
|
||||
source = inspect.getsource(audio_search._download_and_start_processing)
|
||||
|
||||
# The fix specifically uses selected.get('url') which comes from state_data
|
||||
# rather than audio_search_service cache
|
||||
assert "selected.get('url')" in source or "download_url" in source, \
|
||||
"YouTube download must use URL from selected result (state_data)"
|
||||
|
||||
# Verify it doesn't rely on cache for YouTube
|
||||
# The branch is: elif source_name == 'YouTube' and download_url:
|
||||
assert "YouTube" in source and "download_url" in source, \
|
||||
"YouTube branch must check for download_url availability"
|
||||
|
||||
|
||||
class TestTranscriptionTimeout:
|
||||
"""
|
||||
Issue 4: Jobs stuck in downloading state.
|
||||
|
||||
Root cause: Lyrics worker's AudioShake transcription could hang forever
|
||||
without proper timeout, leaving jobs stuck.
|
||||
|
||||
These tests verify timeout handling for transcription.
|
||||
"""
|
||||
|
||||
def test_transcription_timeout_constant_exists(self):
|
||||
"""
|
||||
Verify transcription timeout constant is defined in lyrics_worker.
|
||||
"""
|
||||
from backend.workers.lyrics_worker import TRANSCRIPTION_TIMEOUT_SECONDS
|
||||
|
||||
assert TRANSCRIPTION_TIMEOUT_SECONDS > 0
|
||||
assert TRANSCRIPTION_TIMEOUT_SECONDS >= 300, "Timeout should be at least 5 minutes for long songs"
|
||||
assert TRANSCRIPTION_TIMEOUT_SECONDS <= 1500, "Timeout shouldn't be more than 25 minutes"
|
||||
|
||||
def test_transcription_timeout_value(self):
|
||||
"""
|
||||
Verify the specific timeout value (20 minutes = 1200 seconds).
|
||||
|
||||
This accounts for:
|
||||
- Cloud Run cold start / worker initialization (1-5 min)
|
||||
- AudioShake transcription (1-2 min)
|
||||
- spaCy model loading for correction (2-3 min on cold start)
|
||||
- Agentic AI correction (1-3 min)
|
||||
"""
|
||||
from backend.workers.lyrics_worker import TRANSCRIPTION_TIMEOUT_SECONDS
|
||||
|
||||
assert TRANSCRIPTION_TIMEOUT_SECONDS == 1200, "Transcription timeout should be 20 minutes (1200 seconds)"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_asyncio_wait_for_raises_timeout_error(self):
|
||||
"""
|
||||
Verify asyncio.wait_for properly raises TimeoutError.
|
||||
This is a sanity check for the timeout mechanism we use.
|
||||
"""
|
||||
async def slow_operation():
|
||||
await asyncio.sleep(10)
|
||||
return "completed"
|
||||
|
||||
with pytest.raises(asyncio.TimeoutError):
|
||||
await asyncio.wait_for(slow_operation(), timeout=0.1)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_lyrics_worker_timeout_converts_to_descriptive_error(self):
|
||||
"""
|
||||
Verify that timeout is converted to a descriptive exception message.
|
||||
Tests the actual error conversion logic pattern used in lyrics_worker.
|
||||
"""
|
||||
from backend.workers.lyrics_worker import TRANSCRIPTION_TIMEOUT_SECONDS
|
||||
|
||||
# This simulates what happens in lyrics_worker when timeout occurs
|
||||
error_message = None
|
||||
try:
|
||||
# Simulate the timeout handling pattern from lyrics_worker
|
||||
raise asyncio.TimeoutError()
|
||||
except asyncio.TimeoutError:
|
||||
error_message = f"Transcription timed out after {TRANSCRIPTION_TIMEOUT_SECONDS} seconds"
|
||||
|
||||
assert error_message is not None
|
||||
assert "timed out" in error_message.lower()
|
||||
assert "1200" in error_message
|
||||
|
||||
def test_lyrics_worker_exception_marks_job_failed(self):
|
||||
"""
|
||||
Any exception in lyrics worker should mark job as failed.
|
||||
"""
|
||||
from backend.models.job import JobStatus
|
||||
|
||||
# Simulate the exception handling in lyrics_worker
|
||||
job_status = JobStatus.DOWNLOADING
|
||||
error_occurred = True
|
||||
|
||||
if error_occurred:
|
||||
# This is what happens in the except block
|
||||
job_status = JobStatus.FAILED
|
||||
error_message = "Lyrics transcription failed: Transcription timed out after 1200 seconds"
|
||||
|
||||
assert job_status == JobStatus.FAILED
|
||||
assert "timed out" in error_message.lower()
|
||||
|
||||
|
||||
class TestJobOwnershipFiltering:
|
||||
"""
|
||||
Additional tests for job ownership and filtering.
|
||||
|
||||
These verify that jobs are properly associated with users
|
||||
and can be filtered by user_email.
|
||||
"""
|
||||
|
||||
def test_job_model_has_user_email_field(self):
|
||||
"""Job model must have user_email field."""
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
job = Job(
|
||||
job_id="test-123",
|
||||
artist="Test",
|
||||
title="Test",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
assert hasattr(job, 'user_email')
|
||||
assert job.user_email == "user@example.com"
|
||||
|
||||
def test_job_create_model_has_user_email_field(self):
|
||||
"""JobCreate model must have user_email field."""
|
||||
from backend.models.job import JobCreate
|
||||
|
||||
job_create = JobCreate(
|
||||
artist="Test",
|
||||
title="Test",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
assert hasattr(job_create, 'user_email')
|
||||
assert job_create.user_email == "user@example.com"
|
||||
|
||||
def test_job_create_user_email_is_optional(self):
|
||||
"""JobCreate user_email should be optional for backward compatibility."""
|
||||
from backend.models.job import JobCreate
|
||||
|
||||
job_create = JobCreate(
|
||||
artist="Test",
|
||||
title="Test",
|
||||
)
|
||||
|
||||
assert job_create.user_email is None
|
||||
|
||||
def test_jobs_can_be_filtered_by_user_email(self):
|
||||
"""Jobs should be filterable by user_email."""
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
jobs = [
|
||||
Job(
|
||||
job_id="job-1",
|
||||
artist="Test",
|
||||
title="Test 1",
|
||||
status=JobStatus.COMPLETE,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
user_email="user1@example.com",
|
||||
),
|
||||
Job(
|
||||
job_id="job-2",
|
||||
artist="Test",
|
||||
title="Test 2",
|
||||
status=JobStatus.COMPLETE,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
user_email="user2@example.com",
|
||||
),
|
||||
Job(
|
||||
job_id="job-3",
|
||||
artist="Test",
|
||||
title="Test 3",
|
||||
status=JobStatus.COMPLETE,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
user_email="user1@example.com",
|
||||
),
|
||||
]
|
||||
|
||||
user1_jobs = [j for j in jobs if j.user_email == "user1@example.com"]
|
||||
|
||||
assert len(user1_jobs) == 2
|
||||
assert all(j.user_email == "user1@example.com" for j in user1_jobs)
|
||||
@@ -33,13 +33,28 @@ def job_manager(mock_firestore_service):
|
||||
|
||||
class TestJobCreation:
|
||||
"""Test job creation logic."""
|
||||
|
||||
|
||||
def test_create_job_requires_theme_id(self, job_manager, mock_firestore_service):
|
||||
"""Test that jobs without theme_id are rejected."""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
# No theme_id - should fail
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError, match="theme_id is required"):
|
||||
job_manager.create_job(job_create)
|
||||
|
||||
# Verify Firestore was NOT called
|
||||
mock_firestore_service.create_job.assert_not_called()
|
||||
|
||||
def test_create_job_with_url(self, job_manager, mock_firestore_service):
|
||||
"""Test creating a job with YouTube URL."""
|
||||
job_create = JobCreate(
|
||||
url="https://youtube.com/watch?v=test",
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
title="Test Song",
|
||||
theme_id="nomad" # Required for all jobs
|
||||
)
|
||||
|
||||
# The actual create_job method creates the job and returns it
|
||||
@@ -59,7 +74,8 @@ class TestJobCreation:
|
||||
"""Test creating a job without URL (for file upload)."""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
title="Test Song",
|
||||
theme_id="nomad" # Required for all jobs
|
||||
)
|
||||
|
||||
job = job_manager.create_job(job_create)
|
||||
@@ -70,8 +86,8 @@ class TestJobCreation:
|
||||
|
||||
def test_create_job_generates_unique_id(self, job_manager, mock_firestore_service):
|
||||
"""Test that each job gets a unique ID."""
|
||||
job_create = JobCreate()
|
||||
|
||||
job_create = JobCreate(theme_id="nomad") # Required for all jobs
|
||||
|
||||
# Create multiple jobs
|
||||
ids = []
|
||||
for i in range(5):
|
||||
@@ -89,8 +105,8 @@ class TestJobCreation:
|
||||
|
||||
def test_create_job_sets_initial_status(self, job_manager, mock_firestore_service):
|
||||
"""Test that new jobs start with PENDING status."""
|
||||
job_create = JobCreate()
|
||||
|
||||
job_create = JobCreate(theme_id="nomad") # Required for all jobs
|
||||
|
||||
mock_firestore_service.create_job.return_value = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
@@ -105,13 +121,14 @@ class TestJobCreation:
|
||||
|
||||
def test_create_job_with_distribution_settings(self, job_manager, mock_firestore_service):
|
||||
"""Test that distribution settings are passed from JobCreate to Job.
|
||||
|
||||
|
||||
This was a bug where brand_prefix, dropbox_path, gdrive_folder_id, and
|
||||
discord_webhook_url were NOT being passed to the Job constructor.
|
||||
"""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad", # Required for all jobs
|
||||
brand_prefix="NOMAD",
|
||||
discord_webhook_url="https://discord.com/webhook/test",
|
||||
dropbox_path="/Karaoke/Tracks-Organized",
|
||||
|
||||
@@ -0,0 +1,329 @@
|
||||
"""
|
||||
Unit tests for job manager notification methods.
|
||||
|
||||
Tests the state transition triggers for email notifications.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, MagicMock, AsyncMock
|
||||
from datetime import datetime
|
||||
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
class TestTriggerStateNotifications:
|
||||
"""Tests for _trigger_state_notifications method."""
|
||||
|
||||
def test_triggers_completion_email_on_complete(self):
|
||||
"""Test that completion email is triggered when job completes."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {"youtube_url": "https://youtube.com/123"}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_schedule_completion_email') as mock_schedule:
|
||||
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
|
||||
|
||||
mock_schedule.assert_called_once_with(mock_job)
|
||||
|
||||
def test_triggers_idle_reminder_on_awaiting_review(self):
|
||||
"""Test that idle reminder is triggered when job enters awaiting review."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_schedule_idle_reminder') as mock_schedule:
|
||||
manager._trigger_state_notifications("job-123", JobStatus.AWAITING_REVIEW)
|
||||
|
||||
mock_schedule.assert_called_once_with(mock_job, JobStatus.AWAITING_REVIEW)
|
||||
|
||||
def test_triggers_idle_reminder_on_awaiting_instrumental(self):
|
||||
"""Test that idle reminder is triggered when job enters awaiting instrumental."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_schedule_idle_reminder') as mock_schedule:
|
||||
manager._trigger_state_notifications(
|
||||
"job-123",
|
||||
JobStatus.AWAITING_INSTRUMENTAL_SELECTION
|
||||
)
|
||||
|
||||
mock_schedule.assert_called_once_with(
|
||||
mock_job,
|
||||
JobStatus.AWAITING_INSTRUMENTAL_SELECTION
|
||||
)
|
||||
|
||||
def test_skips_notification_when_no_job(self):
|
||||
"""Test that notification is skipped when job not found."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
manager.get_job = Mock(return_value=None)
|
||||
|
||||
with patch.object(manager, '_schedule_completion_email') as mock_schedule:
|
||||
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
|
||||
|
||||
mock_schedule.assert_not_called()
|
||||
|
||||
def test_skips_notification_when_no_user_email(self):
|
||||
"""Test that notification is skipped when job has no user email."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = None
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_schedule_completion_email') as mock_schedule:
|
||||
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
|
||||
|
||||
mock_schedule.assert_not_called()
|
||||
|
||||
def test_does_not_trigger_on_other_states(self):
|
||||
"""Test that no notification is triggered for non-notification states."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_schedule_completion_email') as mock_complete, \
|
||||
patch.object(manager, '_schedule_idle_reminder') as mock_idle:
|
||||
|
||||
# Test various non-notification states
|
||||
for status in [JobStatus.PENDING, JobStatus.DOWNLOADING, JobStatus.TRANSCRIBING,
|
||||
JobStatus.RENDERING_VIDEO, JobStatus.FAILED]:
|
||||
manager._trigger_state_notifications("job-123", status)
|
||||
|
||||
mock_complete.assert_not_called()
|
||||
mock_idle.assert_not_called()
|
||||
|
||||
def test_handles_exception_gracefully(self):
|
||||
"""Test that exceptions don't propagate from notifications."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
manager.get_job = Mock(side_effect=Exception("Database error"))
|
||||
|
||||
# Should not raise
|
||||
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
|
||||
|
||||
|
||||
class TestScheduleCompletionEmail:
|
||||
"""Tests for _schedule_completion_email method."""
|
||||
|
||||
def test_extracts_urls_from_state_data(self):
|
||||
"""Test that YouTube and Dropbox URLs are extracted from state_data."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.artist = "Test Artist"
|
||||
mock_job.title = "Test Song"
|
||||
mock_job.state_data = {
|
||||
"youtube_url": "https://youtube.com/watch?v=123",
|
||||
"dropbox_link": "https://dropbox.com/folder/abc",
|
||||
}
|
||||
|
||||
with patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.send_job_completion_email = AsyncMock(return_value=True)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
# Run in sync context - this will start a daemon thread
|
||||
manager._schedule_completion_email(mock_job)
|
||||
|
||||
# Give the thread a moment to execute
|
||||
import time
|
||||
time.sleep(0.1)
|
||||
|
||||
def test_handles_none_state_data(self):
|
||||
"""Test that None state_data is handled gracefully."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.artist = None
|
||||
mock_job.title = None
|
||||
mock_job.state_data = None # None state_data
|
||||
|
||||
with patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.send_job_completion_email = AsyncMock(return_value=True)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
# Should not raise
|
||||
manager._schedule_completion_email(mock_job)
|
||||
|
||||
def test_handles_exception_gracefully(self):
|
||||
"""Test that exceptions don't propagate."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
|
||||
with patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_service:
|
||||
mock_get_service.side_effect = Exception("Service error")
|
||||
|
||||
# Should not raise
|
||||
manager._schedule_completion_email(mock_job)
|
||||
|
||||
|
||||
class TestScheduleIdleReminder:
|
||||
"""Tests for _schedule_idle_reminder method."""
|
||||
|
||||
def test_updates_state_data_for_lyrics_review(self):
|
||||
"""Test that state_data is updated with blocking state info for lyrics."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {"existing_key": "value"}
|
||||
|
||||
with patch('backend.services.worker_service.get_worker_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.schedule_idle_reminder = AsyncMock(return_value=True)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
manager._schedule_idle_reminder(mock_job, JobStatus.AWAITING_REVIEW)
|
||||
|
||||
# Verify state_data was updated
|
||||
update_call = manager.firestore.update_job.call_args
|
||||
assert update_call is not None
|
||||
updated_data = update_call[0][1] # Second positional arg
|
||||
state_data = updated_data.get('state_data', {})
|
||||
|
||||
assert state_data.get('blocking_action_type') == 'lyrics'
|
||||
assert state_data.get('reminder_sent') is False
|
||||
assert 'blocking_state_entered_at' in state_data
|
||||
assert state_data.get('existing_key') == 'value' # Preserved
|
||||
|
||||
def test_updates_state_data_for_instrumental_selection(self):
|
||||
"""Test that state_data is updated with blocking state info for instrumental."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
|
||||
with patch('backend.services.worker_service.get_worker_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.schedule_idle_reminder = AsyncMock(return_value=True)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
manager._schedule_idle_reminder(
|
||||
mock_job,
|
||||
JobStatus.AWAITING_INSTRUMENTAL_SELECTION
|
||||
)
|
||||
|
||||
# Verify state_data was updated
|
||||
update_call = manager.firestore.update_job.call_args
|
||||
updated_data = update_call[0][1]
|
||||
state_data = updated_data.get('state_data', {})
|
||||
|
||||
assert state_data.get('blocking_action_type') == 'instrumental'
|
||||
|
||||
def test_handles_none_state_data(self):
|
||||
"""Test that None state_data is handled gracefully."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = None
|
||||
|
||||
with patch('backend.services.worker_service.get_worker_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.schedule_idle_reminder = AsyncMock(return_value=True)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
# Should not raise
|
||||
manager._schedule_idle_reminder(mock_job, JobStatus.AWAITING_REVIEW)
|
||||
|
||||
# Verify update was still called
|
||||
manager.firestore.update_job.assert_called_once()
|
||||
|
||||
def test_handles_exception_gracefully(self):
|
||||
"""Test that exceptions don't propagate."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
|
||||
# Make firestore raise an exception
|
||||
manager.firestore.update_job.side_effect = Exception("Database error")
|
||||
|
||||
# Should not raise
|
||||
manager._schedule_idle_reminder(mock_job, JobStatus.AWAITING_REVIEW)
|
||||
|
||||
|
||||
class TestTransitionTriggersNotifications:
|
||||
"""Integration tests for transition_to_state triggering notifications."""
|
||||
|
||||
def test_transition_to_complete_triggers_notification(self):
|
||||
"""Test that transitioning to COMPLETE triggers notification."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
# Use ENCODING as the starting state since ENCODING -> COMPLETE is valid
|
||||
mock_job.status = JobStatus.ENCODING
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_trigger_state_notifications') as mock_trigger:
|
||||
manager.transition_to_state("job-123", JobStatus.COMPLETE)
|
||||
|
||||
mock_trigger.assert_called_once_with("job-123", JobStatus.COMPLETE)
|
||||
|
||||
def test_transition_to_awaiting_review_triggers_notification(self):
|
||||
"""Test that transitioning to AWAITING_REVIEW triggers notification."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
# Use APPLYING_PADDING as the starting state since APPLYING_PADDING -> AWAITING_REVIEW is valid
|
||||
mock_job.status = JobStatus.APPLYING_PADDING
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_trigger_state_notifications') as mock_trigger:
|
||||
manager.transition_to_state("job-123", JobStatus.AWAITING_REVIEW)
|
||||
|
||||
mock_trigger.assert_called_once_with("job-123", JobStatus.AWAITING_REVIEW)
|
||||
@@ -0,0 +1,443 @@
|
||||
"""
|
||||
Unit tests for job notification service.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, AsyncMock
|
||||
import urllib.parse
|
||||
|
||||
from backend.services.job_notification_service import (
|
||||
JobNotificationService,
|
||||
get_job_notification_service,
|
||||
ENABLE_AUTO_EMAILS,
|
||||
FEEDBACK_FORM_URL,
|
||||
)
|
||||
|
||||
|
||||
class TestURLBuilding:
|
||||
"""Tests for URL building methods."""
|
||||
|
||||
def test_build_review_url_basic(self):
|
||||
"""Test basic review URL building."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_review_url("job-123")
|
||||
|
||||
assert "gen.nomadkaraoke.com/lyrics/" in url
|
||||
assert "baseApiUrl=" in url
|
||||
# The API URL should be URL-encoded
|
||||
assert urllib.parse.quote("https://api.nomadkaraoke.com/api/review/job-123", safe='') in url
|
||||
|
||||
def test_build_review_url_with_audio_hash(self):
|
||||
"""Test review URL with audio hash parameter."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_review_url("job-123", audio_hash="abc123")
|
||||
|
||||
assert "audioHash=abc123" in url
|
||||
|
||||
def test_build_review_url_with_review_token(self):
|
||||
"""Test review URL with review token parameter."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_review_url("job-123", review_token="token456")
|
||||
|
||||
assert "reviewToken=token456" in url
|
||||
|
||||
def test_build_review_url_with_all_params(self):
|
||||
"""Test review URL with all parameters."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_review_url(
|
||||
"job-123",
|
||||
audio_hash="hash789",
|
||||
review_token="token456"
|
||||
)
|
||||
|
||||
assert "baseApiUrl=" in url
|
||||
assert "audioHash=hash789" in url
|
||||
assert "reviewToken=token456" in url
|
||||
|
||||
def test_build_review_url_encodes_special_chars(self):
|
||||
"""Test that special characters in job ID are encoded."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_review_url("job/with/slashes")
|
||||
|
||||
# The baseApiUrl parameter should have encoded slashes
|
||||
assert "%2F" in url
|
||||
|
||||
def test_build_instrumental_url_basic(self):
|
||||
"""Test basic instrumental URL building."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_instrumental_url("job-123")
|
||||
|
||||
assert "gen.nomadkaraoke.com/instrumental/" in url
|
||||
assert "baseApiUrl=" in url
|
||||
|
||||
def test_build_instrumental_url_with_token(self):
|
||||
"""Test instrumental URL with token parameter."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_instrumental_url("job-123", instrumental_token="inst-token")
|
||||
|
||||
assert "instrumentalToken=inst-token" in url
|
||||
|
||||
|
||||
class TestCompletionEmail:
|
||||
"""Tests for job completion email sending."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_success(self):
|
||||
"""Test successful completion email sending."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_job_completion.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Test message"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
user_name="Test User",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
youtube_url="https://youtube.com/watch?v=123",
|
||||
dropbox_url="https://dropbox.com/folder/abc",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
service.template_service.render_job_completion.assert_called_once()
|
||||
service.email_service.send_job_completion.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_disabled(self):
|
||||
"""Test that completion email is skipped when auto emails are disabled."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.template_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', False):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
service.email_service.send_job_completion.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_no_email(self):
|
||||
"""Test that completion email is skipped when no user email."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.template_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email=None,
|
||||
)
|
||||
|
||||
assert result is False
|
||||
service.email_service.send_job_completion.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_empty_email(self):
|
||||
"""Test that completion email is skipped when user email is empty."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.template_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
service.email_service.send_job_completion.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_send_failure(self):
|
||||
"""Test handling of email send failure."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_job_completion.return_value = False
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Test message"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_exception(self):
|
||||
"""Test handling of exceptions during email sending."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_job_completion.side_effect = Exception("Send error")
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Test message"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_passes_cc_admin(self):
|
||||
"""Test that completion email is sent with CC to admin."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_job_completion.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Test message"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
# Verify cc_admin=True was passed
|
||||
call_kwargs = service.email_service.send_job_completion.call_args.kwargs
|
||||
assert call_kwargs.get('cc_admin') is True
|
||||
|
||||
|
||||
class TestActionReminderEmail:
|
||||
"""Tests for action reminder email sending."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_lyrics_reminder_success(self):
|
||||
"""Test successful lyrics reminder email sending."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_action_reminder.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_action_needed_lyrics.return_value = "Review your lyrics"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="lyrics",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
service.template_service.render_action_needed_lyrics.assert_called_once()
|
||||
service.email_service.send_action_reminder.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_instrumental_reminder_success(self):
|
||||
"""Test successful instrumental reminder email sending."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_action_reminder.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_action_needed_instrumental.return_value = "Select instrumental"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="instrumental",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
service.template_service.render_action_needed_instrumental.assert_called_once()
|
||||
service.email_service.send_action_reminder.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_reminder_disabled(self):
|
||||
"""Test that reminder email is skipped when auto emails are disabled."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', False):
|
||||
result = await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
service.email_service.send_action_reminder.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_reminder_no_email(self):
|
||||
"""Test that reminder email is skipped when no user email."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email=None,
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_reminder_unknown_action_type(self):
|
||||
"""Test handling of unknown action type."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.template_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="unknown",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
service.email_service.send_action_reminder.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_lyrics_reminder_includes_review_url(self):
|
||||
"""Test that lyrics reminder includes correct review URL."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_action_reminder.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_action_needed_lyrics.return_value = "Review"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="lyrics",
|
||||
audio_hash="hash123",
|
||||
review_token="token456",
|
||||
)
|
||||
|
||||
# Verify the review URL was passed to template
|
||||
call_kwargs = service.template_service.render_action_needed_lyrics.call_args.kwargs
|
||||
review_url = call_kwargs.get('review_url')
|
||||
assert "audioHash=hash123" in review_url
|
||||
assert "reviewToken=token456" in review_url
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_instrumental_reminder_includes_url(self):
|
||||
"""Test that instrumental reminder includes correct URL."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_action_reminder.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_action_needed_instrumental.return_value = "Select"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="instrumental",
|
||||
instrumental_token="inst-token",
|
||||
)
|
||||
|
||||
# Verify the instrumental URL was passed to template
|
||||
call_kwargs = service.template_service.render_action_needed_instrumental.call_args.kwargs
|
||||
instrumental_url = call_kwargs.get('instrumental_url')
|
||||
assert "instrumentalToken=inst-token" in instrumental_url
|
||||
|
||||
|
||||
class TestGetCompletionMessage:
|
||||
"""Tests for get_completion_message method."""
|
||||
|
||||
def test_get_completion_message_basic(self):
|
||||
"""Test basic completion message retrieval."""
|
||||
service = JobNotificationService()
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Your video is ready!"
|
||||
|
||||
result = service.get_completion_message(
|
||||
job_id="job-123",
|
||||
user_name="Test User",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
youtube_url="https://youtube.com/watch?v=123",
|
||||
dropbox_url="https://dropbox.com/folder/abc",
|
||||
)
|
||||
|
||||
assert result == "Your video is ready!"
|
||||
service.template_service.render_job_completion.assert_called_once()
|
||||
|
||||
def test_get_completion_message_includes_feedback_url(self):
|
||||
"""Test that completion message includes feedback URL."""
|
||||
service = JobNotificationService()
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Message"
|
||||
|
||||
service.get_completion_message(job_id="job-123")
|
||||
|
||||
# Verify feedback URL was passed
|
||||
call_kwargs = service.template_service.render_job_completion.call_args.kwargs
|
||||
assert 'feedback_url' in call_kwargs
|
||||
|
||||
def test_get_completion_message_with_defaults(self):
|
||||
"""Test completion message with default values."""
|
||||
service = JobNotificationService()
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Message"
|
||||
|
||||
service.get_completion_message(job_id="job-123")
|
||||
|
||||
call_kwargs = service.template_service.render_job_completion.call_args.kwargs
|
||||
assert call_kwargs.get('job_id') == "job-123"
|
||||
# Other params should be None/default
|
||||
assert call_kwargs.get('name') is None
|
||||
assert call_kwargs.get('artist') is None
|
||||
|
||||
|
||||
class TestGlobalInstance:
|
||||
"""Tests for global instance management."""
|
||||
|
||||
def test_get_job_notification_service_returns_same_instance(self):
|
||||
"""Test that get_job_notification_service returns singleton."""
|
||||
# Reset global
|
||||
import backend.services.job_notification_service as jns
|
||||
jns._job_notification_service = None
|
||||
|
||||
service1 = get_job_notification_service()
|
||||
service2 = get_job_notification_service()
|
||||
|
||||
assert service1 is service2
|
||||
|
||||
def test_service_initializes_with_dependencies(self):
|
||||
"""Test that service initializes with email and template services."""
|
||||
service = JobNotificationService()
|
||||
|
||||
assert service.email_service is not None
|
||||
assert service.template_service is not None
|
||||
assert service.frontend_url is not None
|
||||
assert service.backend_url is not None
|
||||
@@ -50,14 +50,30 @@ def mock_worker_service():
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(mock_job_manager, mock_worker_service):
|
||||
def mock_theme_service():
|
||||
"""Create a mock ThemeService that returns 'nomad' as default theme."""
|
||||
service = MagicMock()
|
||||
service.get_default_theme_id.return_value = "nomad"
|
||||
service.get_theme.return_value = None
|
||||
return service
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(mock_job_manager, mock_worker_service, mock_theme_service):
|
||||
"""Create TestClient with mocked dependencies."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.universe_domain = 'googleapis.com'
|
||||
|
||||
|
||||
# Create a JobManager class that returns our mock instance
|
||||
def mock_job_manager_factory(*args, **kwargs):
|
||||
return mock_job_manager
|
||||
|
||||
# Patch at the module level where jobs.py imports them
|
||||
# Also patch JobManager class used in dependencies.py for auth checks
|
||||
with patch('backend.api.routes.jobs.job_manager', mock_job_manager), \
|
||||
patch('backend.api.routes.jobs.worker_service', mock_worker_service), \
|
||||
patch('backend.api.routes.jobs.get_theme_service', return_value=mock_theme_service), \
|
||||
patch('backend.services.job_manager.JobManager', mock_job_manager_factory), \
|
||||
patch('backend.services.firestore_service.firestore'), \
|
||||
patch('backend.services.storage_service.storage'), \
|
||||
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
||||
@@ -87,9 +103,13 @@ class TestGetJob:
|
||||
mock_job_manager.get_job.return_value = None
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.universe_domain = 'googleapis.com'
|
||||
|
||||
|
||||
def mock_job_manager_factory(*args, **kwargs):
|
||||
return mock_job_manager
|
||||
|
||||
with patch('backend.api.routes.jobs.job_manager', mock_job_manager), \
|
||||
patch('backend.api.routes.jobs.worker_service', mock_worker_service), \
|
||||
patch('backend.services.job_manager.JobManager', mock_job_manager_factory), \
|
||||
patch('backend.services.firestore_service.firestore'), \
|
||||
patch('backend.services.storage_service.storage'), \
|
||||
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
||||
@@ -173,12 +193,17 @@ class TestDeleteJob:
|
||||
def test_delete_nonexistent_job(self, mock_worker_service, auth_headers):
|
||||
"""Test deleting non-existent job."""
|
||||
mock_job_manager = MagicMock()
|
||||
mock_job_manager.get_job.return_value = None # Job doesn't exist
|
||||
mock_job_manager.delete_job.return_value = False
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.universe_domain = 'googleapis.com'
|
||||
|
||||
|
||||
def mock_job_manager_factory(*args, **kwargs):
|
||||
return mock_job_manager
|
||||
|
||||
with patch('backend.api.routes.jobs.job_manager', mock_job_manager), \
|
||||
patch('backend.api.routes.jobs.worker_service', mock_worker_service), \
|
||||
patch('backend.services.job_manager.JobManager', mock_job_manager_factory), \
|
||||
patch('backend.services.firestore_service.firestore'), \
|
||||
patch('backend.services.storage_service.storage'), \
|
||||
patch('google.auth.default', return_value=(mock_creds, 'test-project')):
|
||||
|
||||
@@ -0,0 +1,423 @@
|
||||
"""
|
||||
Tests for LocalEncodingService.
|
||||
|
||||
Tests cover:
|
||||
- Service initialization
|
||||
- Hardware acceleration detection
|
||||
- FFmpeg command execution
|
||||
- Individual encoding methods
|
||||
- Full encoding pipeline
|
||||
- Dry run mode
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch, call
|
||||
|
||||
from backend.services.local_encoding_service import (
|
||||
LocalEncodingService,
|
||||
EncodingConfig,
|
||||
EncodingResult,
|
||||
get_local_encoding_service,
|
||||
)
|
||||
|
||||
|
||||
class TestLocalEncodingServiceInit:
|
||||
"""Test service initialization."""
|
||||
|
||||
def test_init_default_values(self):
|
||||
"""Test default initialization."""
|
||||
service = LocalEncodingService()
|
||||
assert service.dry_run is False
|
||||
assert "ffmpeg" in service._ffmpeg_base_command
|
||||
|
||||
def test_init_with_dry_run(self):
|
||||
"""Test initialization with dry run mode."""
|
||||
service = LocalEncodingService(dry_run=True)
|
||||
assert service.dry_run is True
|
||||
|
||||
def test_init_with_debug_logging(self):
|
||||
"""Test initialization with debug logging level."""
|
||||
import logging
|
||||
service = LocalEncodingService(log_level=logging.DEBUG)
|
||||
assert "-loglevel verbose" in service._ffmpeg_base_command
|
||||
|
||||
def test_init_with_info_logging(self):
|
||||
"""Test initialization with info logging level."""
|
||||
import logging
|
||||
service = LocalEncodingService(log_level=logging.INFO)
|
||||
assert "-loglevel fatal" in service._ffmpeg_base_command
|
||||
|
||||
|
||||
class TestLocalEncodingServiceHWAccel:
|
||||
"""Test hardware acceleration detection."""
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_detect_nvenc_available(self, mock_run):
|
||||
"""Test NVENC detection when available."""
|
||||
mock_run.return_value = MagicMock(returncode=0)
|
||||
|
||||
service = LocalEncodingService()
|
||||
# Force detection
|
||||
service._detect_and_set_hwaccel()
|
||||
|
||||
assert service._hwaccel_available is True
|
||||
assert service._video_encoder == "h264_nvenc"
|
||||
assert service._scale_filter == "scale_cuda"
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_detect_no_hwaccel(self, mock_run):
|
||||
"""Test fallback when no hardware acceleration available."""
|
||||
import subprocess
|
||||
mock_run.side_effect = subprocess.CalledProcessError(1, "test")
|
||||
|
||||
service = LocalEncodingService()
|
||||
# Force detection
|
||||
service._detect_and_set_hwaccel()
|
||||
|
||||
assert service._hwaccel_available is False
|
||||
assert service._video_encoder == "libx264"
|
||||
assert service._scale_filter == "scale"
|
||||
|
||||
def test_nvenc_quality_settings(self):
|
||||
"""Test NVENC quality settings for different presets."""
|
||||
service = LocalEncodingService()
|
||||
service._hwaccel_available = True
|
||||
|
||||
lossless = service._get_nvenc_quality_settings("lossless")
|
||||
assert "cq 0" in lossless
|
||||
|
||||
medium = service._get_nvenc_quality_settings("medium")
|
||||
assert "p4" in medium
|
||||
|
||||
def test_nvenc_quality_settings_disabled(self):
|
||||
"""Test NVENC quality settings when hwaccel is disabled."""
|
||||
service = LocalEncodingService()
|
||||
service._hwaccel_available = False
|
||||
|
||||
settings = service._get_nvenc_quality_settings("lossless")
|
||||
assert settings == ""
|
||||
|
||||
|
||||
class TestLocalEncodingServiceExecuteCommand:
|
||||
"""Test command execution."""
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_execute_command_success(self, mock_run):
|
||||
"""Test successful command execution."""
|
||||
mock_run.return_value = MagicMock(returncode=0)
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service._execute_command("echo test", "Test command")
|
||||
|
||||
assert result is True
|
||||
mock_run.assert_called_once()
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_execute_command_failure(self, mock_run):
|
||||
"""Test command execution failure."""
|
||||
import subprocess
|
||||
mock_run.side_effect = subprocess.CalledProcessError(
|
||||
1, "test", stderr="Error message"
|
||||
)
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service._execute_command("echo test", "Test command")
|
||||
|
||||
assert result is False
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_execute_command_timeout(self, mock_run):
|
||||
"""Test command execution timeout."""
|
||||
import subprocess
|
||||
mock_run.side_effect = subprocess.TimeoutExpired("test", 30)
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service._execute_command("echo test", "Test command", timeout=30)
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_execute_command_dry_run(self):
|
||||
"""Test command execution in dry run mode."""
|
||||
service = LocalEncodingService(dry_run=True)
|
||||
result = service._execute_command("echo test", "Test command")
|
||||
|
||||
assert result is True
|
||||
# No actual subprocess should be called in dry run mode
|
||||
|
||||
|
||||
class TestLocalEncodingServiceEncodingMethods:
|
||||
"""Test individual encoding methods."""
|
||||
|
||||
@patch.object(LocalEncodingService, "_execute_command")
|
||||
def test_remux_with_instrumental(self, mock_execute):
|
||||
"""Test remuxing with instrumental audio."""
|
||||
mock_execute.return_value = True
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service.remux_with_instrumental(
|
||||
"/input/video.mov",
|
||||
"/input/audio.flac",
|
||||
"/output/video.mp4"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
mock_execute.assert_called_once()
|
||||
call_args = mock_execute.call_args[0][0]
|
||||
assert "/input/video.mov" in call_args
|
||||
assert "/input/audio.flac" in call_args
|
||||
assert "-map 0:v -map 1:a" in call_args
|
||||
|
||||
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
|
||||
def test_convert_mov_to_mp4(self, mock_execute):
|
||||
"""Test MOV to MP4 conversion."""
|
||||
mock_execute.return_value = True
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service.convert_mov_to_mp4(
|
||||
"/input/video.mov",
|
||||
"/output/video.mp4"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
mock_execute.assert_called_once()
|
||||
|
||||
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
|
||||
def test_encode_lossless_mp4_without_end(self, mock_execute):
|
||||
"""Test lossless 4K MP4 encoding without end credits."""
|
||||
mock_execute.return_value = True
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service.encode_lossless_mp4(
|
||||
"/input/title.mov",
|
||||
"/input/karaoke.mp4",
|
||||
"/output/lossless.mp4"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
mock_execute.assert_called_once()
|
||||
call_args = mock_execute.call_args
|
||||
assert "concat=n=2" in call_args[0][0] # GPU command
|
||||
|
||||
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
|
||||
def test_encode_lossless_mp4_with_end(self, mock_execute):
|
||||
"""Test lossless 4K MP4 encoding with end credits."""
|
||||
mock_execute.return_value = True
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
end_file = os.path.join(tmpdir, "end.mov")
|
||||
with open(end_file, "w") as f:
|
||||
f.write("fake video")
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service.encode_lossless_mp4(
|
||||
"/input/title.mov",
|
||||
"/input/karaoke.mp4",
|
||||
"/output/lossless.mp4",
|
||||
end_video=end_file
|
||||
)
|
||||
|
||||
assert result is True
|
||||
call_args = mock_execute.call_args
|
||||
assert "concat=n=3" in call_args[0][0] # 3 videos
|
||||
|
||||
@patch.object(LocalEncodingService, "_execute_command")
|
||||
def test_encode_lossy_mp4(self, mock_execute):
|
||||
"""Test lossy 4K MP4 encoding."""
|
||||
mock_execute.return_value = True
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service.encode_lossy_mp4(
|
||||
"/input/lossless.mp4",
|
||||
"/output/lossy.mp4"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
call_args = mock_execute.call_args[0][0]
|
||||
assert "-c:v copy" in call_args
|
||||
assert "aac" in call_args.lower()
|
||||
|
||||
@patch.object(LocalEncodingService, "_execute_command")
|
||||
def test_encode_lossless_mkv(self, mock_execute):
|
||||
"""Test MKV encoding with FLAC audio."""
|
||||
mock_execute.return_value = True
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service.encode_lossless_mkv(
|
||||
"/input/lossless.mp4",
|
||||
"/output/video.mkv"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
call_args = mock_execute.call_args[0][0]
|
||||
assert "-c:v copy" in call_args
|
||||
assert "-c:a flac" in call_args
|
||||
|
||||
@patch.object(LocalEncodingService, "_execute_command_with_fallback")
|
||||
def test_encode_720p(self, mock_execute):
|
||||
"""Test 720p encoding."""
|
||||
mock_execute.return_value = True
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service.encode_720p(
|
||||
"/input/lossless.mp4",
|
||||
"/output/720p.mp4"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
call_args = mock_execute.call_args
|
||||
# Should contain scale filter
|
||||
assert "1280:720" in call_args[0][0] or "1280:720" in call_args[0][1]
|
||||
|
||||
|
||||
class TestLocalEncodingServiceFullPipeline:
|
||||
"""Test full encoding pipeline."""
|
||||
|
||||
@patch.object(LocalEncodingService, "remux_with_instrumental")
|
||||
@patch.object(LocalEncodingService, "convert_mov_to_mp4")
|
||||
@patch.object(LocalEncodingService, "encode_lossless_mp4")
|
||||
@patch.object(LocalEncodingService, "encode_lossy_mp4")
|
||||
@patch.object(LocalEncodingService, "encode_lossless_mkv")
|
||||
@patch.object(LocalEncodingService, "encode_720p")
|
||||
def test_encode_all_formats_success(
|
||||
self, mock_720p, mock_mkv, mock_lossy, mock_lossless, mock_convert, mock_remux
|
||||
):
|
||||
"""Test successful full encoding pipeline."""
|
||||
mock_remux.return_value = True
|
||||
mock_convert.return_value = True
|
||||
mock_lossless.return_value = True
|
||||
mock_lossy.return_value = True
|
||||
mock_mkv.return_value = True
|
||||
mock_720p.return_value = True
|
||||
|
||||
service = LocalEncodingService()
|
||||
config = EncodingConfig(
|
||||
title_video="/input/title.mov",
|
||||
karaoke_video="/input/karaoke.mov",
|
||||
instrumental_audio="/input/instrumental.flac",
|
||||
output_karaoke_mp4="/output/karaoke.mp4",
|
||||
output_with_vocals_mp4="/output/with_vocals.mp4",
|
||||
output_lossless_4k_mp4="/output/lossless_4k.mp4",
|
||||
output_lossy_4k_mp4="/output/lossy_4k.mp4",
|
||||
output_lossless_mkv="/output/lossless.mkv",
|
||||
output_720p_mp4="/output/720p.mp4",
|
||||
)
|
||||
|
||||
result = service.encode_all_formats(config)
|
||||
|
||||
assert result.success is True
|
||||
assert "karaoke_mp4" in result.output_files
|
||||
assert "lossless_4k_mp4" in result.output_files
|
||||
assert "720p_mp4" in result.output_files
|
||||
|
||||
@patch.object(LocalEncodingService, "remux_with_instrumental")
|
||||
def test_encode_all_formats_failure_early(self, mock_remux):
|
||||
"""Test encoding pipeline failure at early step."""
|
||||
mock_remux.return_value = False
|
||||
|
||||
service = LocalEncodingService()
|
||||
config = EncodingConfig(
|
||||
title_video="/input/title.mov",
|
||||
karaoke_video="/input/karaoke.mov",
|
||||
instrumental_audio="/input/instrumental.flac",
|
||||
output_karaoke_mp4="/output/karaoke.mp4",
|
||||
output_lossless_4k_mp4="/output/lossless_4k.mp4",
|
||||
)
|
||||
|
||||
result = service.encode_all_formats(config)
|
||||
|
||||
assert result.success is False
|
||||
assert "Failed to remux" in result.error
|
||||
|
||||
def test_encode_all_formats_dry_run(self):
|
||||
"""Test encoding pipeline in dry run mode."""
|
||||
service = LocalEncodingService(dry_run=True)
|
||||
config = EncodingConfig(
|
||||
title_video="/input/title.mov",
|
||||
karaoke_video="/input/karaoke.mp4", # Already MP4
|
||||
instrumental_audio="/input/instrumental.flac",
|
||||
output_karaoke_mp4="/output/karaoke.mp4",
|
||||
output_lossless_4k_mp4="/output/lossless_4k.mp4",
|
||||
output_lossy_4k_mp4="/output/lossy_4k.mp4",
|
||||
output_lossless_mkv="/output/lossless.mkv",
|
||||
output_720p_mp4="/output/720p.mp4",
|
||||
)
|
||||
|
||||
result = service.encode_all_formats(config)
|
||||
|
||||
# In dry run mode, all operations should "succeed"
|
||||
assert result.success is True
|
||||
|
||||
|
||||
class TestEncodingConfig:
|
||||
"""Test EncodingConfig dataclass."""
|
||||
|
||||
def test_config_required_fields(self):
|
||||
"""Test that required fields must be provided."""
|
||||
config = EncodingConfig(
|
||||
title_video="/path/title.mov",
|
||||
karaoke_video="/path/karaoke.mov",
|
||||
instrumental_audio="/path/audio.flac",
|
||||
)
|
||||
assert config.title_video == "/path/title.mov"
|
||||
assert config.end_video is None # Optional field
|
||||
|
||||
def test_config_all_fields(self):
|
||||
"""Test config with all fields."""
|
||||
config = EncodingConfig(
|
||||
title_video="/path/title.mov",
|
||||
karaoke_video="/path/karaoke.mov",
|
||||
instrumental_audio="/path/audio.flac",
|
||||
end_video="/path/end.mov",
|
||||
output_karaoke_mp4="/output/karaoke.mp4",
|
||||
output_720p_mp4="/output/720p.mp4",
|
||||
)
|
||||
assert config.end_video == "/path/end.mov"
|
||||
assert config.output_karaoke_mp4 == "/output/karaoke.mp4"
|
||||
|
||||
|
||||
class TestEncodingResult:
|
||||
"""Test EncodingResult dataclass."""
|
||||
|
||||
def test_result_success(self):
|
||||
"""Test successful result."""
|
||||
result = EncodingResult(
|
||||
success=True,
|
||||
output_files={"key": "/path/file.mp4"}
|
||||
)
|
||||
assert result.success is True
|
||||
assert result.error is None
|
||||
|
||||
def test_result_failure(self):
|
||||
"""Test failure result."""
|
||||
result = EncodingResult(
|
||||
success=False,
|
||||
output_files={},
|
||||
error="Something went wrong"
|
||||
)
|
||||
assert result.success is False
|
||||
assert result.error == "Something went wrong"
|
||||
|
||||
|
||||
class TestGetLocalEncodingService:
|
||||
"""Test factory function."""
|
||||
|
||||
def test_get_service_creates_instance(self):
|
||||
"""Test that factory function creates a new instance."""
|
||||
import backend.services.local_encoding_service as module
|
||||
module._local_encoding_service = None
|
||||
|
||||
service = get_local_encoding_service()
|
||||
|
||||
assert service is not None
|
||||
assert isinstance(service, LocalEncodingService)
|
||||
|
||||
def test_get_service_with_dry_run(self):
|
||||
"""Test factory function with dry run option."""
|
||||
import backend.services.local_encoding_service as module
|
||||
module._local_encoding_service = None
|
||||
|
||||
service = get_local_encoding_service(dry_run=True)
|
||||
|
||||
assert service.dry_run is True
|
||||
@@ -0,0 +1,567 @@
|
||||
"""
|
||||
Tests for LocalPreviewEncodingService.
|
||||
|
||||
Tests cover:
|
||||
- Service initialization
|
||||
- Hardware acceleration detection (NVENC)
|
||||
- FFmpeg filter path escaping
|
||||
- ASS filter building
|
||||
- Preview FFmpeg command construction
|
||||
- Preview encoding execution
|
||||
- Error handling
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from backend.services.local_preview_encoding_service import (
|
||||
LocalPreviewEncodingService,
|
||||
PreviewEncodingConfig,
|
||||
PreviewEncodingResult,
|
||||
get_local_preview_encoding_service,
|
||||
)
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServiceInit:
|
||||
"""Test service initialization."""
|
||||
|
||||
def test_init_default_values(self):
|
||||
"""Test default initialization."""
|
||||
service = LocalPreviewEncodingService()
|
||||
assert service._nvenc_available is None # Lazy detection
|
||||
assert service._video_encoder is None # Lazy detection
|
||||
assert service._hwaccel_flags is None # Lazy detection
|
||||
|
||||
def test_init_with_custom_logger(self):
|
||||
"""Test initialization with custom logger."""
|
||||
import logging
|
||||
custom_logger = logging.getLogger("test")
|
||||
service = LocalPreviewEncodingService(logger=custom_logger)
|
||||
assert service.logger is custom_logger
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServiceHWAccel:
|
||||
"""Test hardware acceleration detection."""
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_detect_nvenc_available(self, mock_run):
|
||||
"""Test NVENC detection when available."""
|
||||
mock_run.return_value = MagicMock(returncode=0)
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._detect_nvenc_support()
|
||||
|
||||
assert result is True
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_detect_nvenc_not_available(self, mock_run):
|
||||
"""Test fallback when NVENC not available."""
|
||||
import subprocess
|
||||
mock_run.side_effect = subprocess.CalledProcessError(1, "test")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._detect_nvenc_support()
|
||||
|
||||
assert result is False
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_detect_nvenc_timeout(self, mock_run):
|
||||
"""Test NVENC detection timeout handling."""
|
||||
import subprocess
|
||||
mock_run.side_effect = subprocess.TimeoutExpired("test", 30)
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._detect_nvenc_support()
|
||||
|
||||
assert result is False
|
||||
|
||||
@patch.object(LocalPreviewEncodingService, "_detect_nvenc_support")
|
||||
def test_configure_nvenc_available(self, mock_detect):
|
||||
"""Test configuration when NVENC is available."""
|
||||
mock_detect.return_value = True
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
service._configure_hardware_acceleration()
|
||||
|
||||
assert service._nvenc_available is True
|
||||
assert service._video_encoder == "h264_nvenc"
|
||||
assert "-hwaccel" in service._hwaccel_flags
|
||||
|
||||
@patch.object(LocalPreviewEncodingService, "_detect_nvenc_support")
|
||||
def test_configure_no_nvenc(self, mock_detect):
|
||||
"""Test configuration when NVENC not available."""
|
||||
mock_detect.return_value = False
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
service._configure_hardware_acceleration()
|
||||
|
||||
assert service._nvenc_available is False
|
||||
assert service._video_encoder == "libx264"
|
||||
assert service._hwaccel_flags == []
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServiceProperties:
|
||||
"""Test property accessors that trigger lazy detection."""
|
||||
|
||||
@patch.object(LocalPreviewEncodingService, "_configure_hardware_acceleration")
|
||||
def test_nvenc_available_triggers_detection(self, mock_config):
|
||||
"""Test that nvenc_available property triggers detection."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = None
|
||||
|
||||
# Access property
|
||||
_ = service.nvenc_available
|
||||
|
||||
mock_config.assert_called_once()
|
||||
|
||||
@patch.object(LocalPreviewEncodingService, "_configure_hardware_acceleration")
|
||||
def test_video_encoder_triggers_detection(self, mock_config):
|
||||
"""Test that video_encoder property triggers detection."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._video_encoder = None
|
||||
|
||||
# Access property
|
||||
_ = service.video_encoder
|
||||
|
||||
mock_config.assert_called_once()
|
||||
|
||||
@patch.object(LocalPreviewEncodingService, "_configure_hardware_acceleration")
|
||||
def test_hwaccel_flags_triggers_detection(self, mock_config):
|
||||
"""Test that hwaccel_flags property triggers detection."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._hwaccel_flags = None
|
||||
|
||||
# Access property
|
||||
_ = service.hwaccel_flags
|
||||
|
||||
mock_config.assert_called_once()
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServicePathEscaping:
|
||||
"""Test FFmpeg filter path escaping."""
|
||||
|
||||
def test_escape_simple_path(self):
|
||||
"""Test escaping a simple path."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._escape_ffmpeg_filter_path("/simple/path.ass")
|
||||
assert result == "/simple/path.ass"
|
||||
|
||||
def test_escape_path_with_spaces(self):
|
||||
"""Test escaping path with spaces."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._escape_ffmpeg_filter_path("/path/with spaces/file.ass")
|
||||
assert "\\ " in result # Spaces should be escaped
|
||||
|
||||
def test_escape_path_with_apostrophe(self):
|
||||
"""Test escaping path with apostrophe (common in song titles)."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._escape_ffmpeg_filter_path("/path/I'm With You/file.ass")
|
||||
assert "\\\\\\'" in result # Apostrophe should be triple-backslash escaped
|
||||
|
||||
def test_escape_path_with_special_chars(self):
|
||||
"""Test escaping path with FFmpeg special characters."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._escape_ffmpeg_filter_path("/path:with[special];chars,here.ass")
|
||||
# Should escape :,[];
|
||||
assert "\\:" in result
|
||||
assert "\\[" in result
|
||||
assert "\\]" in result
|
||||
assert "\\;" in result
|
||||
assert "\\," in result
|
||||
|
||||
def test_escape_path_with_backslashes(self):
|
||||
"""Test escaping path with existing backslashes."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._escape_ffmpeg_filter_path("/path\\with\\backslashes.ass")
|
||||
assert "\\\\" in result
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServiceASSFilter:
|
||||
"""Test ASS filter building."""
|
||||
|
||||
def test_build_ass_filter_simple(self):
|
||||
"""Test building ASS filter without font."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._build_ass_filter("/path/to/file.ass")
|
||||
assert result.startswith("ass=")
|
||||
assert "fontsdir" not in result
|
||||
|
||||
def test_build_ass_filter_with_font(self):
|
||||
"""Test building ASS filter with custom font."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
font_path = os.path.join(tmpdir, "custom.ttf")
|
||||
with open(font_path, "w") as f:
|
||||
f.write("fake font")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._build_ass_filter("/path/to/file.ass", font_path)
|
||||
|
||||
assert "fontsdir=" in result
|
||||
|
||||
def test_build_ass_filter_nonexistent_font(self):
|
||||
"""Test building ASS filter with nonexistent font file."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._build_ass_filter("/path/to/file.ass", "/nonexistent/font.ttf")
|
||||
# Should not include fontsdir for nonexistent font
|
||||
assert "fontsdir" not in result
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServiceFFmpegCommand:
|
||||
"""Test FFmpeg command building."""
|
||||
|
||||
def test_build_command_solid_background_libx264(self):
|
||||
"""Test command building with solid background and libx264."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4",
|
||||
background_color="black"
|
||||
)
|
||||
|
||||
cmd = service._build_preview_ffmpeg_command(config)
|
||||
|
||||
assert "ffmpeg" in cmd
|
||||
assert "-r" in cmd and "24" in cmd # Frame rate
|
||||
assert "-c:v" in cmd
|
||||
assert "libx264" in cmd
|
||||
assert "-preset" in cmd and "superfast" in cmd
|
||||
assert "-crf" in cmd and "28" in cmd
|
||||
assert "color=c=black" in " ".join(cmd)
|
||||
|
||||
def test_build_command_with_nvenc(self):
|
||||
"""Test command building with NVENC hardware acceleration."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = True
|
||||
service._video_encoder = "h264_nvenc"
|
||||
service._hwaccel_flags = ["-hwaccel", "cuda", "-hwaccel_output_format", "cuda"]
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4"
|
||||
)
|
||||
|
||||
cmd = service._build_preview_ffmpeg_command(config)
|
||||
|
||||
assert "h264_nvenc" in cmd
|
||||
assert "-hwaccel" in cmd
|
||||
assert "-preset" in cmd and "p1" in cmd # Fastest NVENC preset
|
||||
|
||||
def test_build_command_with_background_image(self):
|
||||
"""Test command building with background image."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
bg_path = os.path.join(tmpdir, "background.png")
|
||||
with open(bg_path, "wb") as f:
|
||||
f.write(b"fake png")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4",
|
||||
background_image_path=bg_path
|
||||
)
|
||||
|
||||
cmd = service._build_preview_ffmpeg_command(config)
|
||||
|
||||
assert "-loop" in cmd
|
||||
assert "1" in cmd
|
||||
assert bg_path in cmd
|
||||
# Should have scale filter for background image
|
||||
vf_index = cmd.index("-vf")
|
||||
vf_value = cmd[vf_index + 1]
|
||||
assert "scale=" in vf_value
|
||||
assert "pad=" in vf_value
|
||||
|
||||
def test_build_command_resolution(self):
|
||||
"""Test that command uses correct preview resolution."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4"
|
||||
)
|
||||
|
||||
cmd = service._build_preview_ffmpeg_command(config)
|
||||
cmd_str = " ".join(cmd)
|
||||
|
||||
# Resolution should be 480x270
|
||||
assert "480" in cmd_str
|
||||
assert "270" in cmd_str
|
||||
|
||||
def test_build_command_audio_settings(self):
|
||||
"""Test that command uses correct audio settings."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4"
|
||||
)
|
||||
|
||||
cmd = service._build_preview_ffmpeg_command(config)
|
||||
|
||||
assert "-c:a" in cmd
|
||||
assert "aac" in cmd
|
||||
assert "-b:a" in cmd
|
||||
assert "96k" in cmd
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServiceEncode:
|
||||
"""Test preview encoding execution."""
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_encode_preview_success(self, mock_run):
|
||||
"""Test successful preview encoding."""
|
||||
mock_run.return_value = MagicMock(returncode=0, stderr="")
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
ass_path = os.path.join(tmpdir, "subs.ass")
|
||||
audio_path = os.path.join(tmpdir, "audio.flac")
|
||||
output_path = os.path.join(tmpdir, "output.mp4")
|
||||
|
||||
# Create input files
|
||||
with open(ass_path, "w") as f:
|
||||
f.write("[Script Info]\n")
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(b"fake audio")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path=ass_path,
|
||||
audio_path=audio_path,
|
||||
output_path=output_path
|
||||
)
|
||||
|
||||
result = service.encode_preview(config)
|
||||
|
||||
assert result.success is True
|
||||
assert result.output_path == output_path
|
||||
assert result.error is None
|
||||
mock_run.assert_called_once()
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_encode_preview_ffmpeg_failure(self, mock_run):
|
||||
"""Test preview encoding with FFmpeg failure."""
|
||||
mock_run.return_value = MagicMock(returncode=1, stderr="FFmpeg error message")
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
ass_path = os.path.join(tmpdir, "subs.ass")
|
||||
audio_path = os.path.join(tmpdir, "audio.flac")
|
||||
output_path = os.path.join(tmpdir, "output.mp4")
|
||||
|
||||
with open(ass_path, "w") as f:
|
||||
f.write("[Script Info]\n")
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(b"fake audio")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path=ass_path,
|
||||
audio_path=audio_path,
|
||||
output_path=output_path
|
||||
)
|
||||
|
||||
result = service.encode_preview(config)
|
||||
|
||||
assert result.success is False
|
||||
assert "FFmpeg preview encoding failed" in result.error
|
||||
|
||||
def test_encode_preview_missing_ass_file(self):
|
||||
"""Test preview encoding with missing ASS file."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
audio_path = os.path.join(tmpdir, "audio.flac")
|
||||
output_path = os.path.join(tmpdir, "output.mp4")
|
||||
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(b"fake audio")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/nonexistent/subs.ass",
|
||||
audio_path=audio_path,
|
||||
output_path=output_path
|
||||
)
|
||||
|
||||
result = service.encode_preview(config)
|
||||
|
||||
assert result.success is False
|
||||
assert "not found" in result.error
|
||||
|
||||
def test_encode_preview_missing_audio_file(self):
|
||||
"""Test preview encoding with missing audio file."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
ass_path = os.path.join(tmpdir, "subs.ass")
|
||||
output_path = os.path.join(tmpdir, "output.mp4")
|
||||
|
||||
with open(ass_path, "w") as f:
|
||||
f.write("[Script Info]\n")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path=ass_path,
|
||||
audio_path="/nonexistent/audio.flac",
|
||||
output_path=output_path
|
||||
)
|
||||
|
||||
result = service.encode_preview(config)
|
||||
|
||||
assert result.success is False
|
||||
assert "not found" in result.error
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_encode_preview_timeout(self, mock_run):
|
||||
"""Test preview encoding timeout."""
|
||||
import subprocess
|
||||
mock_run.side_effect = subprocess.TimeoutExpired("ffmpeg", 300)
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
ass_path = os.path.join(tmpdir, "subs.ass")
|
||||
audio_path = os.path.join(tmpdir, "audio.flac")
|
||||
output_path = os.path.join(tmpdir, "output.mp4")
|
||||
|
||||
with open(ass_path, "w") as f:
|
||||
f.write("[Script Info]\n")
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(b"fake audio")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path=ass_path,
|
||||
audio_path=audio_path,
|
||||
output_path=output_path
|
||||
)
|
||||
|
||||
result = service.encode_preview(config)
|
||||
|
||||
assert result.success is False
|
||||
assert "timed out" in result.error
|
||||
|
||||
|
||||
class TestPreviewEncodingConfig:
|
||||
"""Test PreviewEncodingConfig dataclass."""
|
||||
|
||||
def test_config_required_fields(self):
|
||||
"""Test that required fields must be provided."""
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4"
|
||||
)
|
||||
assert config.ass_path == "/path/to/subs.ass"
|
||||
assert config.background_color == "black" # Default
|
||||
assert config.background_image_path is None # Optional
|
||||
|
||||
def test_config_all_fields(self):
|
||||
"""Test config with all fields."""
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4",
|
||||
background_image_path="/path/to/bg.png",
|
||||
background_color="red",
|
||||
font_path="/path/to/font.ttf"
|
||||
)
|
||||
assert config.background_image_path == "/path/to/bg.png"
|
||||
assert config.background_color == "red"
|
||||
assert config.font_path == "/path/to/font.ttf"
|
||||
|
||||
|
||||
class TestPreviewEncodingResult:
|
||||
"""Test PreviewEncodingResult dataclass."""
|
||||
|
||||
def test_result_success(self):
|
||||
"""Test successful result."""
|
||||
result = PreviewEncodingResult(
|
||||
success=True,
|
||||
output_path="/path/to/output.mp4"
|
||||
)
|
||||
assert result.success is True
|
||||
assert result.error is None
|
||||
assert result.output_path == "/path/to/output.mp4"
|
||||
|
||||
def test_result_failure(self):
|
||||
"""Test failure result."""
|
||||
result = PreviewEncodingResult(
|
||||
success=False,
|
||||
error="Something went wrong"
|
||||
)
|
||||
assert result.success is False
|
||||
assert result.error == "Something went wrong"
|
||||
assert result.output_path is None
|
||||
|
||||
|
||||
class TestGetLocalPreviewEncodingService:
|
||||
"""Test factory function."""
|
||||
|
||||
def test_get_service_creates_instance(self):
|
||||
"""Test that factory function creates a new instance."""
|
||||
import backend.services.local_preview_encoding_service as module
|
||||
module._local_preview_encoding_service = None
|
||||
|
||||
service = get_local_preview_encoding_service()
|
||||
|
||||
assert service is not None
|
||||
assert isinstance(service, LocalPreviewEncodingService)
|
||||
|
||||
def test_get_service_returns_singleton(self):
|
||||
"""Test that factory function returns the same instance."""
|
||||
import backend.services.local_preview_encoding_service as module
|
||||
module._local_preview_encoding_service = None
|
||||
|
||||
service1 = get_local_preview_encoding_service()
|
||||
service2 = get_local_preview_encoding_service()
|
||||
|
||||
assert service1 is service2
|
||||
|
||||
|
||||
class TestPreviewEncodingConstants:
|
||||
"""Test that encoding constants match expected values."""
|
||||
|
||||
def test_preview_resolution(self):
|
||||
"""Test preview resolution constants."""
|
||||
service = LocalPreviewEncodingService()
|
||||
assert service.PREVIEW_WIDTH == 480
|
||||
assert service.PREVIEW_HEIGHT == 270
|
||||
|
||||
def test_preview_fps(self):
|
||||
"""Test preview frame rate constant."""
|
||||
service = LocalPreviewEncodingService()
|
||||
assert service.PREVIEW_FPS == 24
|
||||
|
||||
def test_preview_audio_bitrate(self):
|
||||
"""Test preview audio bitrate constant."""
|
||||
service = LocalPreviewEncodingService()
|
||||
assert service.PREVIEW_AUDIO_BITRATE == "96k"
|
||||
@@ -0,0 +1,382 @@
|
||||
"""
|
||||
Tests for PackagingService.
|
||||
|
||||
Tests cover:
|
||||
- Service initialization
|
||||
- CDG package creation
|
||||
- TXT package creation
|
||||
- Dry run mode
|
||||
- Error handling
|
||||
"""
|
||||
|
||||
import os
|
||||
import zipfile
|
||||
import tempfile
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from backend.services.packaging_service import (
|
||||
PackagingService,
|
||||
get_packaging_service,
|
||||
)
|
||||
|
||||
|
||||
class TestPackagingServiceInit:
|
||||
"""Test service initialization."""
|
||||
|
||||
def test_init_default_values(self):
|
||||
"""Test default initialization."""
|
||||
service = PackagingService()
|
||||
assert service.cdg_styles is None
|
||||
assert service.dry_run is False
|
||||
assert service.non_interactive is False
|
||||
|
||||
def test_init_with_cdg_styles(self):
|
||||
"""Test initialization with CDG styles."""
|
||||
styles = {"font_size": 24, "bg_color": "black"}
|
||||
service = PackagingService(cdg_styles=styles)
|
||||
assert service.cdg_styles == styles
|
||||
|
||||
def test_init_with_dry_run(self):
|
||||
"""Test initialization with dry run mode."""
|
||||
service = PackagingService(dry_run=True)
|
||||
assert service.dry_run is True
|
||||
|
||||
def test_init_with_non_interactive(self):
|
||||
"""Test initialization with non-interactive mode."""
|
||||
service = PackagingService(non_interactive=True)
|
||||
assert service.non_interactive is True
|
||||
|
||||
|
||||
class TestPackagingServiceCreateZipFromFiles:
|
||||
"""Test internal ZIP creation."""
|
||||
|
||||
def test_create_zip_from_files(self):
|
||||
"""Test creating a ZIP file from multiple files."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
# Create test files
|
||||
file1 = os.path.join(tmpdir, "test1.txt")
|
||||
file2 = os.path.join(tmpdir, "test2.txt")
|
||||
zip_path = os.path.join(tmpdir, "output.zip")
|
||||
|
||||
with open(file1, "w") as f:
|
||||
f.write("content1")
|
||||
with open(file2, "w") as f:
|
||||
f.write("content2")
|
||||
|
||||
service = PackagingService()
|
||||
service._create_zip_from_files(
|
||||
zip_path,
|
||||
[(file1, "test1.txt"), (file2, "test2.txt")]
|
||||
)
|
||||
|
||||
assert os.path.isfile(zip_path)
|
||||
with zipfile.ZipFile(zip_path, "r") as zf:
|
||||
names = zf.namelist()
|
||||
assert "test1.txt" in names
|
||||
assert "test2.txt" in names
|
||||
|
||||
|
||||
class TestPackagingServiceCDGPackage:
|
||||
"""Test CDG package creation."""
|
||||
|
||||
def test_create_cdg_package_missing_lrc_file(self):
|
||||
"""Test that missing LRC file raises FileNotFoundError."""
|
||||
service = PackagingService()
|
||||
|
||||
with pytest.raises(FileNotFoundError) as exc_info:
|
||||
service.create_cdg_package(
|
||||
lrc_file="/nonexistent/file.lrc",
|
||||
audio_file="/some/audio.flac",
|
||||
output_zip_path="/output/test.zip",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
)
|
||||
assert "LRC file not found" in str(exc_info.value)
|
||||
|
||||
def test_create_cdg_package_missing_audio_file(self):
|
||||
"""Test that missing audio file raises FileNotFoundError."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
lrc_file = os.path.join(tmpdir, "test.lrc")
|
||||
with open(lrc_file, "w") as f:
|
||||
f.write("[00:00.00]Test lyrics")
|
||||
|
||||
service = PackagingService()
|
||||
|
||||
with pytest.raises(FileNotFoundError) as exc_info:
|
||||
service.create_cdg_package(
|
||||
lrc_file=lrc_file,
|
||||
audio_file="/nonexistent/audio.flac",
|
||||
output_zip_path="/output/test.zip",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
)
|
||||
assert "Audio file not found" in str(exc_info.value)
|
||||
|
||||
def test_create_cdg_package_no_styles_raises(self):
|
||||
"""Test that CDG generation without styles raises ValueError."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
lrc_file = os.path.join(tmpdir, "test.lrc")
|
||||
audio_file = os.path.join(tmpdir, "test.flac")
|
||||
zip_path = os.path.join(tmpdir, "output.zip")
|
||||
|
||||
with open(lrc_file, "w") as f:
|
||||
f.write("[00:00.00]Test lyrics")
|
||||
with open(audio_file, "w") as f:
|
||||
f.write("fake audio")
|
||||
|
||||
# Service without CDG styles
|
||||
service = PackagingService(cdg_styles=None)
|
||||
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
service.create_cdg_package(
|
||||
lrc_file=lrc_file,
|
||||
audio_file=audio_file,
|
||||
output_zip_path=zip_path,
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
)
|
||||
assert "CDG styles configuration is required" in str(exc_info.value)
|
||||
|
||||
def test_create_cdg_package_dry_run(self):
|
||||
"""Test CDG package creation in dry run mode."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
lrc_file = os.path.join(tmpdir, "test.lrc")
|
||||
audio_file = os.path.join(tmpdir, "test.flac")
|
||||
zip_path = os.path.join(tmpdir, "output.zip")
|
||||
|
||||
with open(lrc_file, "w") as f:
|
||||
f.write("[00:00.00]Test lyrics")
|
||||
with open(audio_file, "w") as f:
|
||||
f.write("fake audio")
|
||||
|
||||
service = PackagingService(dry_run=True)
|
||||
|
||||
result = service.create_cdg_package(
|
||||
lrc_file=lrc_file,
|
||||
audio_file=audio_file,
|
||||
output_zip_path=zip_path,
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
)
|
||||
|
||||
# In dry run, ZIP should not be created
|
||||
assert not os.path.isfile(zip_path)
|
||||
assert result[0] == zip_path
|
||||
|
||||
def test_create_cdg_package_existing_files(self):
|
||||
"""Test CDG package creation with existing MP3 and CDG files."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
lrc_file = os.path.join(tmpdir, "test.lrc")
|
||||
audio_file = os.path.join(tmpdir, "test.flac")
|
||||
mp3_file = os.path.join(tmpdir, "test.mp3")
|
||||
cdg_file = os.path.join(tmpdir, "test.cdg")
|
||||
zip_path = os.path.join(tmpdir, "output.zip")
|
||||
|
||||
# Create all files
|
||||
with open(lrc_file, "w") as f:
|
||||
f.write("[00:00.00]Test lyrics")
|
||||
with open(audio_file, "w") as f:
|
||||
f.write("fake audio")
|
||||
with open(mp3_file, "w") as f:
|
||||
f.write("fake mp3")
|
||||
with open(cdg_file, "w") as f:
|
||||
f.write("fake cdg")
|
||||
|
||||
service = PackagingService()
|
||||
|
||||
result = service.create_cdg_package(
|
||||
lrc_file=lrc_file,
|
||||
audio_file=audio_file,
|
||||
output_zip_path=zip_path,
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
output_mp3_path=mp3_file,
|
||||
output_cdg_path=cdg_file,
|
||||
)
|
||||
|
||||
# ZIP should be created from existing files
|
||||
assert os.path.isfile(zip_path)
|
||||
assert result[0] == zip_path
|
||||
assert result[1] == mp3_file
|
||||
assert result[2] == cdg_file
|
||||
|
||||
@patch("lyrics_transcriber.output.cdg.CDGGenerator")
|
||||
def test_create_cdg_package_with_generator(self, mock_cdg_generator_class):
|
||||
"""Test CDG package creation using CDGGenerator."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
lrc_file = os.path.join(tmpdir, "test.lrc")
|
||||
audio_file = os.path.join(tmpdir, "test.flac")
|
||||
zip_path = os.path.join(tmpdir, "output.zip")
|
||||
generated_zip = os.path.join(tmpdir, "generated.zip")
|
||||
|
||||
with open(lrc_file, "w") as f:
|
||||
f.write("[00:00.00]Test lyrics")
|
||||
with open(audio_file, "w") as f:
|
||||
f.write("fake audio")
|
||||
|
||||
# Create the zip file that would be generated
|
||||
with zipfile.ZipFile(generated_zip, "w") as zf:
|
||||
zf.writestr("test.mp3", "fake mp3")
|
||||
zf.writestr("test.cdg", "fake cdg")
|
||||
|
||||
mock_generator = MagicMock()
|
||||
mock_generator.generate_cdg_from_lrc.return_value = (
|
||||
"test.cdg", "test.mp3", generated_zip
|
||||
)
|
||||
mock_cdg_generator_class.return_value = mock_generator
|
||||
|
||||
styles = {"font_size": 24}
|
||||
service = PackagingService(cdg_styles=styles)
|
||||
|
||||
result = service.create_cdg_package(
|
||||
lrc_file=lrc_file,
|
||||
audio_file=audio_file,
|
||||
output_zip_path=zip_path,
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
)
|
||||
|
||||
mock_generator.generate_cdg_from_lrc.assert_called_once_with(
|
||||
lrc_file=lrc_file,
|
||||
audio_file=audio_file,
|
||||
title="Test Title",
|
||||
artist="Test Artist",
|
||||
cdg_styles=styles,
|
||||
)
|
||||
assert os.path.isfile(zip_path)
|
||||
|
||||
|
||||
class TestPackagingServiceTXTPackage:
|
||||
"""Test TXT package creation."""
|
||||
|
||||
def test_create_txt_package_missing_lrc_file(self):
|
||||
"""Test that missing LRC file raises FileNotFoundError."""
|
||||
service = PackagingService()
|
||||
|
||||
with pytest.raises(FileNotFoundError) as exc_info:
|
||||
service.create_txt_package(
|
||||
lrc_file="/nonexistent/file.lrc",
|
||||
mp3_file="/some/audio.mp3",
|
||||
output_zip_path="/output/test.zip",
|
||||
)
|
||||
assert "LRC file not found" in str(exc_info.value)
|
||||
|
||||
def test_create_txt_package_missing_mp3_file(self):
|
||||
"""Test that missing MP3 file raises FileNotFoundError."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
lrc_file = os.path.join(tmpdir, "test.lrc")
|
||||
with open(lrc_file, "w") as f:
|
||||
f.write("[00:00.00]Test lyrics")
|
||||
|
||||
service = PackagingService()
|
||||
|
||||
with pytest.raises(FileNotFoundError) as exc_info:
|
||||
service.create_txt_package(
|
||||
lrc_file=lrc_file,
|
||||
mp3_file="/nonexistent/audio.mp3",
|
||||
output_zip_path="/output/test.zip",
|
||||
)
|
||||
assert "MP3 file not found" in str(exc_info.value)
|
||||
|
||||
def test_create_txt_package_dry_run(self):
|
||||
"""Test TXT package creation in dry run mode."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
lrc_file = os.path.join(tmpdir, "test.lrc")
|
||||
mp3_file = os.path.join(tmpdir, "test.mp3")
|
||||
zip_path = os.path.join(tmpdir, "output.zip")
|
||||
|
||||
with open(lrc_file, "w") as f:
|
||||
f.write("[00:00.00]Test lyrics")
|
||||
with open(mp3_file, "w") as f:
|
||||
f.write("fake mp3")
|
||||
|
||||
service = PackagingService(dry_run=True)
|
||||
|
||||
result = service.create_txt_package(
|
||||
lrc_file=lrc_file,
|
||||
mp3_file=mp3_file,
|
||||
output_zip_path=zip_path,
|
||||
)
|
||||
|
||||
# In dry run, ZIP should not be created
|
||||
assert not os.path.isfile(zip_path)
|
||||
assert result[0] == zip_path
|
||||
|
||||
@patch("lyrics_converter.LyricsConverter")
|
||||
def test_create_txt_package_success(self, mock_converter_class):
|
||||
"""Test successful TXT package creation."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
lrc_file = os.path.join(tmpdir, "test.lrc")
|
||||
mp3_file = os.path.join(tmpdir, "test.mp3")
|
||||
zip_path = os.path.join(tmpdir, "output.zip")
|
||||
txt_path = os.path.join(tmpdir, "output.txt")
|
||||
|
||||
with open(lrc_file, "w") as f:
|
||||
f.write("[00:00.00]Test lyrics")
|
||||
with open(mp3_file, "w") as f:
|
||||
f.write("fake mp3")
|
||||
|
||||
mock_converter = MagicMock()
|
||||
mock_converter.convert_file.return_value = "Converted lyrics text"
|
||||
mock_converter_class.return_value = mock_converter
|
||||
|
||||
service = PackagingService()
|
||||
|
||||
result = service.create_txt_package(
|
||||
lrc_file=lrc_file,
|
||||
mp3_file=mp3_file,
|
||||
output_zip_path=zip_path,
|
||||
output_txt_path=txt_path,
|
||||
)
|
||||
|
||||
mock_converter_class.assert_called_once_with(
|
||||
output_format="txt",
|
||||
filepath=lrc_file
|
||||
)
|
||||
mock_converter.convert_file.assert_called_once()
|
||||
|
||||
assert os.path.isfile(zip_path)
|
||||
assert os.path.isfile(txt_path)
|
||||
assert result[0] == zip_path
|
||||
assert result[1] == txt_path
|
||||
|
||||
# Verify ZIP contents
|
||||
with zipfile.ZipFile(zip_path, "r") as zf:
|
||||
names = zf.namelist()
|
||||
assert "test.mp3" in names
|
||||
assert "output.txt" in names
|
||||
|
||||
|
||||
class TestGetPackagingService:
|
||||
"""Test factory function."""
|
||||
|
||||
def test_get_service_creates_instance(self):
|
||||
"""Test that factory function creates a new instance."""
|
||||
import backend.services.packaging_service as module
|
||||
module._packaging_service = None
|
||||
|
||||
service = get_packaging_service()
|
||||
|
||||
assert service is not None
|
||||
assert isinstance(service, PackagingService)
|
||||
|
||||
def test_get_service_with_cdg_styles(self):
|
||||
"""Test factory function with CDG styles."""
|
||||
import backend.services.packaging_service as module
|
||||
module._packaging_service = None
|
||||
|
||||
styles = {"font_size": 24}
|
||||
service = get_packaging_service(cdg_styles=styles)
|
||||
|
||||
assert service.cdg_styles == styles
|
||||
|
||||
def test_get_service_with_options(self):
|
||||
"""Test factory function with additional options."""
|
||||
import backend.services.packaging_service as module
|
||||
module._packaging_service = None
|
||||
|
||||
service = get_packaging_service(dry_run=True, non_interactive=True)
|
||||
|
||||
assert service.dry_run is True
|
||||
assert service.non_interactive is True
|
||||
@@ -149,7 +149,8 @@ class TestUploadSubmissionRequest:
|
||||
# CDG/TXT disabled by default (requires style config)
|
||||
assert request.enable_cdg is False
|
||||
assert request.enable_txt is False
|
||||
assert request.enable_youtube_upload is False
|
||||
# YouTube upload default is None (use server default)
|
||||
assert request.enable_youtube_upload is None
|
||||
|
||||
|
||||
class TestStartReviewRequest:
|
||||
|
||||
@@ -99,7 +99,7 @@ class TestStylesConfigRequirements:
|
||||
"karaoke": {
|
||||
"background_color": "#000000",
|
||||
"font_path": "", # MUST be string, NOT None
|
||||
"font": "Arial",
|
||||
"font": "Noto Sans",
|
||||
"ass_name": "Default", # REQUIRED
|
||||
"primary_color": "112, 112, 247, 255",
|
||||
"secondary_color": "255, 255, 255, 255",
|
||||
@@ -241,7 +241,7 @@ class TestPreviewStyleLoading:
|
||||
"background_image": "/original/path/background.png",
|
||||
"font_path": "/original/path/font.ttf",
|
||||
"background_color": "#000000",
|
||||
"font": "Arial",
|
||||
"font": "Noto Sans",
|
||||
"ass_name": "Default",
|
||||
}
|
||||
}
|
||||
@@ -308,7 +308,7 @@ class TestPreviewStyleLoading:
|
||||
# Should have karaoke section with minimal/default values
|
||||
assert "karaoke" in result_styles
|
||||
assert result_styles["karaoke"]["background_color"] == "#000000"
|
||||
assert result_styles["karaoke"]["font"] == "Arial"
|
||||
assert result_styles["karaoke"]["font"] == "Noto Sans"
|
||||
# Minimal styles have background_image as None (default)
|
||||
assert result_styles["karaoke"].get("background_image") is None
|
||||
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
"""Tests for SpaCy preloader service."""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import patch, MagicMock
|
||||
|
||||
from backend.services.spacy_preloader import (
|
||||
preload_spacy_model,
|
||||
get_preloaded_model,
|
||||
is_model_preloaded,
|
||||
clear_preloaded_models,
|
||||
)
|
||||
|
||||
|
||||
class TestSpacyPreloader:
|
||||
"""Tests for SpaCy preloading functionality."""
|
||||
|
||||
def setup_method(self):
|
||||
"""Clear preloaded models before each test."""
|
||||
clear_preloaded_models()
|
||||
|
||||
def teardown_method(self):
|
||||
"""Clear preloaded models after each test."""
|
||||
clear_preloaded_models()
|
||||
|
||||
def test_preload_spacy_model_loads_and_stores(self):
|
||||
"""GIVEN no preloaded models
|
||||
WHEN preload_spacy_model is called
|
||||
THEN model should be loaded and stored in singleton."""
|
||||
mock_nlp = MagicMock()
|
||||
|
||||
with patch("spacy.load", return_value=mock_nlp) as mock_load:
|
||||
preload_spacy_model("en_core_web_sm")
|
||||
|
||||
mock_load.assert_called_once_with("en_core_web_sm")
|
||||
assert is_model_preloaded("en_core_web_sm")
|
||||
assert get_preloaded_model("en_core_web_sm") is mock_nlp
|
||||
|
||||
def test_preload_is_idempotent(self):
|
||||
"""GIVEN a model already preloaded
|
||||
WHEN preload_spacy_model is called again
|
||||
THEN model should not be reloaded."""
|
||||
mock_nlp = MagicMock()
|
||||
|
||||
with patch("spacy.load", return_value=mock_nlp) as mock_load:
|
||||
preload_spacy_model("en_core_web_sm")
|
||||
preload_spacy_model("en_core_web_sm") # Second call
|
||||
|
||||
# Should only load once
|
||||
assert mock_load.call_count == 1
|
||||
|
||||
def test_get_preloaded_model_returns_none_if_not_loaded(self):
|
||||
"""GIVEN no preloaded models
|
||||
WHEN get_preloaded_model is called
|
||||
THEN should return None."""
|
||||
assert get_preloaded_model("en_core_web_sm") is None
|
||||
assert not is_model_preloaded("en_core_web_sm")
|
||||
|
||||
def test_preload_different_models(self):
|
||||
"""GIVEN no preloaded models
|
||||
WHEN preload_spacy_model is called with different model names
|
||||
THEN each model should be loaded and stored separately."""
|
||||
mock_nlp_sm = MagicMock(name="en_core_web_sm")
|
||||
mock_nlp_md = MagicMock(name="en_core_web_md")
|
||||
|
||||
def mock_load(model_name):
|
||||
if model_name == "en_core_web_sm":
|
||||
return mock_nlp_sm
|
||||
elif model_name == "en_core_web_md":
|
||||
return mock_nlp_md
|
||||
raise ValueError(f"Unknown model: {model_name}")
|
||||
|
||||
with patch("spacy.load", side_effect=mock_load):
|
||||
preload_spacy_model("en_core_web_sm")
|
||||
preload_spacy_model("en_core_web_md")
|
||||
|
||||
assert is_model_preloaded("en_core_web_sm")
|
||||
assert is_model_preloaded("en_core_web_md")
|
||||
assert get_preloaded_model("en_core_web_sm") is mock_nlp_sm
|
||||
assert get_preloaded_model("en_core_web_md") is mock_nlp_md
|
||||
|
||||
def test_clear_preloaded_models(self):
|
||||
"""GIVEN preloaded models
|
||||
WHEN clear_preloaded_models is called
|
||||
THEN all models should be removed."""
|
||||
mock_nlp = MagicMock()
|
||||
|
||||
with patch("spacy.load", return_value=mock_nlp):
|
||||
preload_spacy_model("en_core_web_sm")
|
||||
assert is_model_preloaded("en_core_web_sm")
|
||||
|
||||
clear_preloaded_models()
|
||||
assert not is_model_preloaded("en_core_web_sm")
|
||||
assert get_preloaded_model("en_core_web_sm") is None
|
||||
|
||||
def test_preload_failure_raises_exception(self):
|
||||
"""GIVEN spacy.load raises an exception
|
||||
WHEN preload_spacy_model is called
|
||||
THEN exception should be propagated."""
|
||||
with patch(
|
||||
"spacy.load",
|
||||
side_effect=OSError("Model not found"),
|
||||
):
|
||||
with pytest.raises(OSError, match="Model not found"):
|
||||
preload_spacy_model("nonexistent_model")
|
||||
|
||||
# Model should not be marked as preloaded
|
||||
assert not is_model_preloaded("nonexistent_model")
|
||||
|
||||
def test_preload_uses_default_model_name(self):
|
||||
"""GIVEN no model name specified
|
||||
WHEN preload_spacy_model is called without arguments
|
||||
THEN should use en_core_web_sm as default."""
|
||||
mock_nlp = MagicMock()
|
||||
|
||||
with patch("spacy.load", return_value=mock_nlp) as mock_load:
|
||||
preload_spacy_model()
|
||||
|
||||
mock_load.assert_called_once_with("en_core_web_sm")
|
||||
assert is_model_preloaded("en_core_web_sm")
|
||||
@@ -0,0 +1,295 @@
|
||||
"""
|
||||
Unit tests for template service.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, MagicMock
|
||||
|
||||
from backend.services.template_service import (
|
||||
TemplateService,
|
||||
get_template_service,
|
||||
DEFAULT_JOB_COMPLETION_TEMPLATE,
|
||||
DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE,
|
||||
DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE,
|
||||
)
|
||||
|
||||
|
||||
class TestTemplateRendering:
|
||||
"""Tests for template rendering functionality."""
|
||||
|
||||
def test_render_template_basic(self):
|
||||
"""Test basic template variable replacement."""
|
||||
service = TemplateService()
|
||||
template = "Hello {name}, your job {job_id} is ready!"
|
||||
variables = {"name": "Alice", "job_id": "123"}
|
||||
|
||||
result = service.render_template(template, variables)
|
||||
|
||||
assert result == "Hello Alice, your job 123 is ready!"
|
||||
|
||||
def test_render_template_missing_variables(self):
|
||||
"""Test that missing variables are replaced with empty strings."""
|
||||
service = TemplateService()
|
||||
template = "Hello {name}, your {thing} is {status}!"
|
||||
variables = {"name": "Bob"}
|
||||
|
||||
result = service.render_template(template, variables)
|
||||
|
||||
assert result == "Hello Bob, your is !"
|
||||
|
||||
def test_render_template_none_values(self):
|
||||
"""Test that None values are handled correctly."""
|
||||
service = TemplateService()
|
||||
template = "Hello {name}!"
|
||||
variables = {"name": None}
|
||||
|
||||
result = service.render_template(template, variables)
|
||||
|
||||
assert result == "Hello !"
|
||||
|
||||
def test_render_template_removes_feedback_section_when_no_url(self):
|
||||
"""Test that feedback section is removed when feedback_url is not provided."""
|
||||
service = TemplateService()
|
||||
template = """Thanks for your order!
|
||||
|
||||
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
|
||||
{feedback_url}
|
||||
|
||||
Have a great day!"""
|
||||
variables = {"feedback_url": None}
|
||||
|
||||
result = service.render_template(template, variables)
|
||||
|
||||
assert "feedback" not in result.lower()
|
||||
assert "Thanks for your order!" in result
|
||||
assert "Have a great day!" in result
|
||||
|
||||
def test_render_template_keeps_feedback_section_when_url_provided(self):
|
||||
"""Test that feedback section is kept when feedback_url is provided."""
|
||||
service = TemplateService()
|
||||
template = """Thanks!
|
||||
|
||||
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
|
||||
{feedback_url}
|
||||
|
||||
Bye!"""
|
||||
variables = {"feedback_url": "https://example.com/feedback"}
|
||||
|
||||
result = service.render_template(template, variables)
|
||||
|
||||
assert "https://example.com/feedback" in result
|
||||
assert "feedback" in result.lower()
|
||||
|
||||
|
||||
class TestJobCompletionTemplate:
|
||||
"""Tests for job completion template rendering."""
|
||||
|
||||
def test_render_job_completion_all_fields(self):
|
||||
"""Test rendering job completion with all fields."""
|
||||
service = TemplateService()
|
||||
|
||||
# Mock _fetch_template_from_gcs to return None (use default)
|
||||
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
|
||||
result = service.render_job_completion(
|
||||
name="Alice",
|
||||
youtube_url="https://youtube.com/watch?v=123",
|
||||
dropbox_url="https://dropbox.com/folder/abc",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="job-123",
|
||||
feedback_url="https://example.com/feedback",
|
||||
)
|
||||
|
||||
assert "Alice" in result
|
||||
assert "https://youtube.com/watch?v=123" in result
|
||||
assert "https://dropbox.com/folder/abc" in result
|
||||
assert "https://example.com/feedback" in result
|
||||
|
||||
def test_render_job_completion_defaults(self):
|
||||
"""Test rendering job completion with default values."""
|
||||
service = TemplateService()
|
||||
|
||||
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
|
||||
result = service.render_job_completion()
|
||||
|
||||
assert "there" in result # Default name
|
||||
assert "[YouTube URL not available]" in result
|
||||
assert "[Dropbox URL not available]" in result
|
||||
|
||||
def test_render_job_completion_no_feedback_url(self):
|
||||
"""Test rendering job completion without feedback URL removes section."""
|
||||
service = TemplateService()
|
||||
|
||||
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
|
||||
result = service.render_job_completion(
|
||||
name="Bob",
|
||||
youtube_url="https://youtube.com/123",
|
||||
dropbox_url="https://dropbox.com/abc",
|
||||
)
|
||||
|
||||
assert "Bob" in result
|
||||
# Feedback section should be removed
|
||||
assert "really appreciate your feedback" not in result
|
||||
|
||||
|
||||
class TestActionNeededTemplates:
|
||||
"""Tests for action-needed template rendering."""
|
||||
|
||||
def test_render_action_needed_lyrics(self):
|
||||
"""Test rendering lyrics review reminder."""
|
||||
service = TemplateService()
|
||||
|
||||
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
|
||||
result = service.render_action_needed_lyrics(
|
||||
name="Charlie",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
review_url="https://example.com/review",
|
||||
)
|
||||
|
||||
assert "Charlie" in result
|
||||
assert "Test Artist" in result
|
||||
assert "Test Song" in result
|
||||
assert "https://example.com/review" in result
|
||||
|
||||
def test_render_action_needed_instrumental(self):
|
||||
"""Test rendering instrumental selection reminder."""
|
||||
service = TemplateService()
|
||||
|
||||
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
|
||||
result = service.render_action_needed_instrumental(
|
||||
name="Diana",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
instrumental_url="https://example.com/instrumental",
|
||||
)
|
||||
|
||||
assert "Diana" in result
|
||||
assert "Test Artist" in result
|
||||
assert "Test Song" in result
|
||||
assert "https://example.com/instrumental" in result
|
||||
|
||||
|
||||
class TestGCSFetching:
|
||||
"""Tests for GCS template fetching."""
|
||||
|
||||
def test_fetch_template_from_gcs_success(self):
|
||||
"""Test successful template fetch from GCS."""
|
||||
service = TemplateService()
|
||||
|
||||
# Mock the bucket and blob
|
||||
mock_blob = Mock()
|
||||
mock_blob.exists.return_value = True
|
||||
mock_blob.download_as_text.return_value = "Custom template {name}"
|
||||
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
|
||||
with patch.object(service, "_bucket", mock_bucket):
|
||||
result = service._fetch_template_from_gcs("test.txt")
|
||||
|
||||
assert result == "Custom template {name}"
|
||||
mock_bucket.blob.assert_called_once_with("templates/test.txt")
|
||||
|
||||
def test_fetch_template_from_gcs_not_found(self):
|
||||
"""Test template fetch when file doesn't exist in GCS."""
|
||||
service = TemplateService()
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_blob.exists.return_value = False
|
||||
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
|
||||
with patch.object(service, "_bucket", mock_bucket):
|
||||
result = service._fetch_template_from_gcs("nonexistent.txt")
|
||||
|
||||
assert result is None
|
||||
|
||||
def test_fetch_template_from_gcs_error(self):
|
||||
"""Test template fetch handles errors gracefully."""
|
||||
service = TemplateService()
|
||||
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.side_effect = Exception("GCS error")
|
||||
|
||||
with patch.object(service, "_bucket", mock_bucket):
|
||||
result = service._fetch_template_from_gcs("test.txt")
|
||||
|
||||
assert result is None
|
||||
|
||||
def test_get_job_completion_template_fallback(self):
|
||||
"""Test that default template is used when GCS fetch fails."""
|
||||
service = TemplateService()
|
||||
|
||||
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
|
||||
result = service.get_job_completion_template()
|
||||
|
||||
assert result == DEFAULT_JOB_COMPLETION_TEMPLATE
|
||||
|
||||
def test_get_action_needed_lyrics_template_fallback(self):
|
||||
"""Test that default lyrics template is used when GCS fetch fails."""
|
||||
service = TemplateService()
|
||||
|
||||
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
|
||||
result = service.get_action_needed_lyrics_template()
|
||||
|
||||
assert result == DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE
|
||||
|
||||
def test_get_action_needed_instrumental_template_fallback(self):
|
||||
"""Test that default instrumental template is used when GCS fetch fails."""
|
||||
service = TemplateService()
|
||||
|
||||
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
|
||||
result = service.get_action_needed_instrumental_template()
|
||||
|
||||
assert result == DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE
|
||||
|
||||
|
||||
class TestTemplateUpload:
|
||||
"""Tests for template upload functionality."""
|
||||
|
||||
def test_upload_template_success(self):
|
||||
"""Test successful template upload."""
|
||||
service = TemplateService()
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
|
||||
with patch.object(service, "_bucket", mock_bucket):
|
||||
result = service.upload_template("test.txt", "Test content")
|
||||
|
||||
assert result is True
|
||||
mock_bucket.blob.assert_called_once_with("templates/test.txt")
|
||||
mock_blob.upload_from_string.assert_called_once_with(
|
||||
"Test content", content_type="text/plain"
|
||||
)
|
||||
|
||||
def test_upload_template_error(self):
|
||||
"""Test template upload handles errors."""
|
||||
service = TemplateService()
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_blob.upload_from_string.side_effect = Exception("Upload failed")
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
|
||||
with patch.object(service, "_bucket", mock_bucket):
|
||||
result = service.upload_template("test.txt", "Test content")
|
||||
|
||||
assert result is False
|
||||
|
||||
|
||||
class TestGlobalInstance:
|
||||
"""Tests for global instance management."""
|
||||
|
||||
def test_get_template_service_returns_same_instance(self):
|
||||
"""Test that get_template_service returns singleton."""
|
||||
# Reset global
|
||||
import backend.services.template_service as ts
|
||||
ts._template_service = None
|
||||
|
||||
service1 = get_template_service()
|
||||
service2 = get_template_service()
|
||||
|
||||
assert service1 is service2
|
||||
@@ -0,0 +1,522 @@
|
||||
"""
|
||||
Comprehensive tests for Unicode and special character handling.
|
||||
|
||||
This test file verifies that artist/title with Unicode characters (curly quotes,
|
||||
em dashes, non-ASCII characters) are properly sanitized throughout the codebase
|
||||
to prevent:
|
||||
1. HTTP header encoding failures (Content-Disposition, email subjects)
|
||||
2. API query failures (Google Drive)
|
||||
3. Filename encoding issues (Modal API, filesystem)
|
||||
|
||||
The root cause was job d49efab1 which had title "Mama Says (You Can't Back Down)"
|
||||
with a curly apostrophe (U+2019) that caused the audio separation to fail because
|
||||
the Modal API couldn't handle the non-latin-1 character in HTTP headers.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from karaoke_gen.utils import (
|
||||
sanitize_filename,
|
||||
normalize_text,
|
||||
UNICODE_REPLACEMENTS,
|
||||
TEXT_NORMALIZATIONS,
|
||||
APOSTROPHE_REPLACEMENTS,
|
||||
DOUBLE_QUOTE_REPLACEMENTS,
|
||||
DASH_REPLACEMENTS,
|
||||
WHITESPACE_REPLACEMENTS,
|
||||
)
|
||||
|
||||
|
||||
class TestSanitizeFilename:
|
||||
"""Test the sanitize_filename function handles all edge cases."""
|
||||
|
||||
def test_curly_single_quotes(self):
|
||||
"""Test that curly single quotes are converted to straight quotes."""
|
||||
# The exact character that caused job d49efab1 to fail
|
||||
assert sanitize_filename("Can't") == "Can't"
|
||||
assert sanitize_filename("It's") == "It's"
|
||||
# Left single quote
|
||||
assert sanitize_filename("'Hello'") == "'Hello'"
|
||||
|
||||
def test_curly_double_quotes(self):
|
||||
"""Test that curly double quotes are converted to underscores.
|
||||
|
||||
Note: Double quotes are filesystem-unsafe so they become underscores."""
|
||||
# Curly quotes first become straight, then straight quotes become underscore
|
||||
assert sanitize_filename("\u201cHello\u201d") == "_Hello_"
|
||||
|
||||
def test_em_dash(self):
|
||||
"""Test that em dashes are converted to regular hyphens."""
|
||||
assert sanitize_filename("Artist — Title") == "Artist - Title"
|
||||
assert sanitize_filename("Song—Name") == "Song-Name"
|
||||
|
||||
def test_en_dash(self):
|
||||
"""Test that en dashes are converted to regular hyphens."""
|
||||
assert sanitize_filename("1990–2000") == "1990-2000"
|
||||
|
||||
def test_ellipsis(self):
|
||||
"""Test that horizontal ellipsis is converted to three dots."""
|
||||
# Note: The ellipsis becomes "..." but trailing dots are stripped
|
||||
assert sanitize_filename("Wait\u2026") == "Wait"
|
||||
# But in the middle of a string they're preserved
|
||||
assert sanitize_filename("Wait\u2026Here") == "Wait...Here"
|
||||
|
||||
def test_non_breaking_space(self):
|
||||
"""Test that non-breaking spaces are converted to regular spaces."""
|
||||
# U+00A0 is non-breaking space
|
||||
assert sanitize_filename("Hello\u00A0World") == "Hello World"
|
||||
|
||||
def test_filesystem_unsafe_characters(self):
|
||||
"""Test that filesystem-unsafe characters are replaced."""
|
||||
assert sanitize_filename("file/name") == "file_name"
|
||||
assert sanitize_filename("file\\name") == "file_name"
|
||||
assert sanitize_filename("file:name") == "file_name"
|
||||
assert sanitize_filename("file*name") == "file_name"
|
||||
assert sanitize_filename("file?name") == "file_name"
|
||||
assert sanitize_filename('file"name') == "file_name"
|
||||
assert sanitize_filename("file<name") == "file_name"
|
||||
assert sanitize_filename("file>name") == "file_name"
|
||||
assert sanitize_filename("file|name") == "file_name"
|
||||
|
||||
def test_trailing_periods_and_spaces(self):
|
||||
"""Test that trailing periods and spaces are removed."""
|
||||
assert sanitize_filename("filename.") == "filename"
|
||||
assert sanitize_filename("filename...") == "filename"
|
||||
assert sanitize_filename("filename ") == "filename"
|
||||
assert sanitize_filename("filename . ") == "filename"
|
||||
|
||||
def test_leading_periods_and_spaces(self):
|
||||
"""Test that leading periods and spaces are removed."""
|
||||
assert sanitize_filename(".filename") == "filename"
|
||||
assert sanitize_filename("...filename") == "filename"
|
||||
assert sanitize_filename(" filename") == "filename"
|
||||
assert sanitize_filename(" . filename") == "filename"
|
||||
|
||||
def test_multiple_underscores_collapsed(self):
|
||||
"""Test that multiple consecutive underscores are collapsed to one."""
|
||||
assert sanitize_filename("file___name") == "file_name"
|
||||
# Multiple unsafe chars in a row
|
||||
assert sanitize_filename("file?*:name") == "file_name"
|
||||
|
||||
def test_multiple_spaces_collapsed(self):
|
||||
"""Test that multiple consecutive spaces are collapsed to one."""
|
||||
assert sanitize_filename("file name") == "file name"
|
||||
|
||||
def test_none_input(self):
|
||||
"""Test that None input returns None."""
|
||||
assert sanitize_filename(None) is None
|
||||
|
||||
def test_empty_string(self):
|
||||
"""Test that empty string returns empty string."""
|
||||
assert sanitize_filename("") == ""
|
||||
|
||||
def test_real_world_examples(self):
|
||||
"""Test real-world examples that have caused issues."""
|
||||
# The exact title from job d49efab1
|
||||
assert sanitize_filename("Mama Says (You Can't Back Down)") == "Mama Says (You Can't Back Down)"
|
||||
|
||||
# Broadway cast with smart quotes
|
||||
assert sanitize_filename("Footloose (Broadway Cast) \u2014 \u201cFinal Song\u201d") == "Footloose (Broadway Cast) - _Final Song_"
|
||||
|
||||
# Japanese artist with em dash
|
||||
assert sanitize_filename("宇多田ヒカル — First Love") == "宇多田ヒカル - First Love"
|
||||
|
||||
# Korean title (should pass through unchanged)
|
||||
assert sanitize_filename("아이유 - 좋은 날") == "아이유 - 좋은 날"
|
||||
|
||||
# Mixed content
|
||||
assert sanitize_filename("L'Arc~en~Ciel - Driver's High") == "L'Arc~en~Ciel - Driver's High"
|
||||
|
||||
def test_combination_of_issues(self):
|
||||
"""Test strings with multiple problematic characters."""
|
||||
result = sanitize_filename("It\u2019s \u201cMy\u201d Song \u2014 Volume\u20261")
|
||||
# Curly ' -> ', curly " -> _ (via filesystem check), em dash -> -, ellipsis -> ...
|
||||
assert result == "It's _My_ Song - Volume...1"
|
||||
|
||||
def test_unicode_replacements_dict_complete(self):
|
||||
"""Verify all expected Unicode characters are in the replacements dict."""
|
||||
# Curly quotes
|
||||
assert "\u2018" in UNICODE_REPLACEMENTS # LEFT SINGLE QUOTATION MARK
|
||||
assert "\u2019" in UNICODE_REPLACEMENTS # RIGHT SINGLE QUOTATION MARK
|
||||
assert "\u201A" in UNICODE_REPLACEMENTS # SINGLE LOW-9 QUOTATION MARK
|
||||
assert "\u201B" in UNICODE_REPLACEMENTS # SINGLE HIGH-REVERSED-9 QUOTATION MARK
|
||||
assert "\u201C" in UNICODE_REPLACEMENTS # LEFT DOUBLE QUOTATION MARK
|
||||
assert "\u201D" in UNICODE_REPLACEMENTS # RIGHT DOUBLE QUOTATION MARK
|
||||
assert "\u201E" in UNICODE_REPLACEMENTS # DOUBLE LOW-9 QUOTATION MARK
|
||||
assert "\u201F" in UNICODE_REPLACEMENTS # DOUBLE HIGH-REVERSED-9 QUOTATION MARK
|
||||
# Dashes
|
||||
assert "\u2013" in UNICODE_REPLACEMENTS # EN DASH
|
||||
assert "\u2014" in UNICODE_REPLACEMENTS # EM DASH
|
||||
# Other
|
||||
assert "\u2026" in UNICODE_REPLACEMENTS # HORIZONTAL ELLIPSIS
|
||||
assert "\u00A0" in UNICODE_REPLACEMENTS # NON-BREAKING SPACE
|
||||
|
||||
|
||||
class TestSanitizationInContext:
|
||||
"""Test that sanitization is applied correctly in various contexts.
|
||||
|
||||
These tests verify that the fixes we applied work in the actual
|
||||
code paths that use artist/title data.
|
||||
"""
|
||||
|
||||
def test_artist_title_format_for_audio_separation(self):
|
||||
"""Verify artist-title formatting is sanitized for Modal API."""
|
||||
# Simulate what audio_worker.py does
|
||||
artist = "Footloose (Broadway Cast)"
|
||||
title = "Mama Says (You Can\u2019t Back Down)" # Curly apostrophe U+2019
|
||||
|
||||
safe_artist = sanitize_filename(artist) if artist else "Unknown"
|
||||
safe_title = sanitize_filename(title) if title else "Unknown"
|
||||
artist_title = f"{safe_artist} - {safe_title}"
|
||||
|
||||
# The curly apostrophe should be converted to straight
|
||||
assert "'" in artist_title # Straight apostrophe
|
||||
assert "\u2019" not in artist_title # No curly apostrophe (U+2019)
|
||||
|
||||
def test_email_subject_safe_for_latin1(self):
|
||||
"""Verify email subjects can be encoded to latin-1 after sanitization."""
|
||||
artist = "Café del Mar" # Has accented e
|
||||
title = "It\u2019s a \u201cBeautiful\u201d Day \u2014 Remix"
|
||||
|
||||
safe_artist = sanitize_filename(artist) if artist else None
|
||||
safe_title = sanitize_filename(title) if title else None
|
||||
subject = f"{safe_artist} - {safe_title}"
|
||||
|
||||
# Accented characters should pass through (they're valid latin-1)
|
||||
assert "é" in subject
|
||||
# But curly quotes and em dash should be converted
|
||||
assert "'" in subject # Straight apostrophe
|
||||
assert "-" in subject # Regular hyphen
|
||||
# No curly apostrophe (U+2019)
|
||||
assert "\u2019" not in subject
|
||||
|
||||
def test_content_disposition_safe(self):
|
||||
"""Verify filenames are safe for HTTP Content-Disposition headers."""
|
||||
artist = "Artist\u2019s \u201cName\u201d"
|
||||
title = "Song\u2014Title"
|
||||
|
||||
safe_artist = sanitize_filename(artist) if artist else None
|
||||
safe_title = sanitize_filename(title) if title else None
|
||||
filename = f"{safe_artist} - {safe_title} (Final Karaoke).mp4"
|
||||
|
||||
# Should be able to encode to latin-1 for HTTP headers
|
||||
try:
|
||||
filename.encode('latin-1')
|
||||
can_encode = True
|
||||
except UnicodeEncodeError:
|
||||
can_encode = False
|
||||
|
||||
# Standard ASCII chars and latin-1 safe accented chars should work
|
||||
# Note: The test above with 'é' passes latin-1; smart quotes do not
|
||||
assert can_encode
|
||||
|
||||
def test_google_drive_query_safe(self):
|
||||
"""Verify filenames don't break Google Drive API queries."""
|
||||
base_name = "Artist\u2019s \u201cSong\u201d"
|
||||
|
||||
safe_base_name = sanitize_filename(base_name) if base_name else base_name
|
||||
filename = f"NOMAD-1234 - {safe_base_name}.mp4"
|
||||
|
||||
# Google Drive queries use single quotes - our sanitized string
|
||||
# should have straight single quotes, not curly
|
||||
# The query escaping handles straight quotes: '
|
||||
escaped = filename.replace("'", "\\'")
|
||||
query = f"name='{escaped}'"
|
||||
|
||||
# Should be a valid query string (no curly quotes to break syntax)
|
||||
assert "\u2019" not in query # No curly single quote (U+2019)
|
||||
assert "\u2018" not in query # No curly single quote (U+2018)
|
||||
|
||||
def test_dropbox_path_safe(self):
|
||||
"""Verify Dropbox paths don't have problematic characters."""
|
||||
artist = "Don\u2019t Stop"
|
||||
title = "Believin\u2019"
|
||||
|
||||
safe_artist = sanitize_filename(artist) if artist else "Unknown"
|
||||
safe_title = sanitize_filename(title) if title else "Unknown"
|
||||
folder_name = f"NOMAD-1234 - {safe_artist} - {safe_title}"
|
||||
remote_path = f"/Karaoke/{folder_name}"
|
||||
|
||||
# Path should not have curly quotes (U+2018 and U+2019)
|
||||
assert "\u2018" not in remote_path
|
||||
assert "\u2019" not in remote_path
|
||||
# But should have the converted straight apostrophe
|
||||
assert "'" in remote_path
|
||||
|
||||
|
||||
class TestInternationalCharacters:
|
||||
"""Test that international/non-ASCII characters are handled properly.
|
||||
|
||||
The goal is NOT to strip all non-ASCII, but to convert problematic
|
||||
Unicode to safe equivalents while preserving legitimate international text.
|
||||
"""
|
||||
|
||||
def test_japanese_preserved(self):
|
||||
"""Japanese characters should pass through unchanged."""
|
||||
assert sanitize_filename("宇多田ヒカル") == "宇多田ヒカル"
|
||||
assert sanitize_filename("君が代") == "君が代"
|
||||
|
||||
def test_korean_preserved(self):
|
||||
"""Korean characters should pass through unchanged."""
|
||||
assert sanitize_filename("방탄소년단") == "방탄소년단"
|
||||
assert sanitize_filename("좋은 날") == "좋은 날"
|
||||
|
||||
def test_chinese_preserved(self):
|
||||
"""Chinese characters should pass through unchanged."""
|
||||
assert sanitize_filename("周杰倫") == "周杰倫"
|
||||
assert sanitize_filename("青花瓷") == "青花瓷"
|
||||
|
||||
def test_cyrillic_preserved(self):
|
||||
"""Cyrillic characters should pass through unchanged."""
|
||||
assert sanitize_filename("Тату") == "Тату"
|
||||
assert sanitize_filename("Не верь, не бойся") == "Не верь, не бойся"
|
||||
|
||||
def test_arabic_preserved(self):
|
||||
"""Arabic characters should pass through unchanged."""
|
||||
assert sanitize_filename("فيروز") == "فيروز"
|
||||
|
||||
def test_accented_latin_preserved(self):
|
||||
"""Accented Latin characters should pass through unchanged."""
|
||||
assert sanitize_filename("Café") == "Café"
|
||||
assert sanitize_filename("Señorita") == "Señorita"
|
||||
assert sanitize_filename("Naïve") == "Naïve"
|
||||
assert sanitize_filename("Björk") == "Björk"
|
||||
|
||||
def test_mixed_script_preserved(self):
|
||||
"""Mixed script text should work correctly."""
|
||||
# Japanese artist, English title
|
||||
assert sanitize_filename("宇多田ヒカル - First Love") == "宇多田ヒカル - First Love"
|
||||
# K-pop with English
|
||||
assert sanitize_filename("BTS 방탄소년단 - Dynamite") == "BTS 방탄소년단 - Dynamite"
|
||||
|
||||
|
||||
class TestEdgeCasesAndRegression:
|
||||
"""Edge cases and regression tests for specific bugs."""
|
||||
|
||||
def test_job_d49efab1_exact_title(self):
|
||||
"""Exact reproduction of job d49efab1 failure case.
|
||||
|
||||
The job had:
|
||||
- artist: "Footloose (Broadway Cast)" (from display_artist override)
|
||||
- title: "Mama Says (You Can't Back Down)" (with curly apostrophe U+2019)
|
||||
|
||||
This caused Stage 2 audio separation to fail because the filename
|
||||
with the curly apostrophe couldn't be encoded in HTTP headers.
|
||||
"""
|
||||
artist = "Footloose (Broadway Cast)"
|
||||
title = "Mama Says (You Can\u2019t Back Down)" # Explicit U+2019
|
||||
|
||||
safe_artist = sanitize_filename(artist)
|
||||
safe_title = sanitize_filename(title)
|
||||
artist_title = f"{safe_artist} - {safe_title}"
|
||||
|
||||
# The result should use straight apostrophe
|
||||
assert "Can't" in artist_title
|
||||
assert "\u2019" not in artist_title # No curly apostrophe
|
||||
|
||||
# Should be HTTP-header safe
|
||||
try:
|
||||
artist_title.encode('latin-1')
|
||||
header_safe = True
|
||||
except UnicodeEncodeError:
|
||||
header_safe = False
|
||||
assert header_safe
|
||||
|
||||
def test_double_sanitization_idempotent(self):
|
||||
"""Sanitizing twice should give the same result as once."""
|
||||
original = "It\u2019s \u201cMy\u201d Song \u2014 Test\u2026"
|
||||
once = sanitize_filename(original)
|
||||
twice = sanitize_filename(once)
|
||||
assert once == twice
|
||||
|
||||
def test_only_problematic_chars_string(self):
|
||||
"""Test string made entirely of problematic characters."""
|
||||
result = sanitize_filename("\u2018\u2019\u201c\u201d\u2014\u2026")
|
||||
# Should become: ''""--... then quotes -> underscores, collapses
|
||||
# Actually: ' ' " " - ... -> underscores for filesystem chars
|
||||
assert result # Should not be empty
|
||||
|
||||
def test_very_long_filename(self):
|
||||
"""Test that very long filenames are handled."""
|
||||
long_name = "A" * 1000
|
||||
result = sanitize_filename(long_name)
|
||||
assert result == long_name # No truncation in sanitize_filename itself
|
||||
|
||||
def test_special_musical_characters(self):
|
||||
"""Test musical symbols and special characters."""
|
||||
# These should pass through as they're not in our replacement list
|
||||
assert "♪" in sanitize_filename("♪ Intro ♪")
|
||||
assert "♫" in sanitize_filename("♫ Music ♫")
|
||||
assert "♯" in sanitize_filename("C♯ Minor")
|
||||
assert "♭" in sanitize_filename("B♭ Major")
|
||||
|
||||
|
||||
class TestNormalizeText:
|
||||
"""Test the normalize_text function for data consistency normalization."""
|
||||
|
||||
def test_curly_single_quotes_normalized(self):
|
||||
"""Test that curly single quotes are converted to straight apostrophe."""
|
||||
assert normalize_text("Can\u2019t") == "Can't"
|
||||
assert normalize_text("It\u2018s") == "It's"
|
||||
assert normalize_text("\u201Aquote\u201B") == "'quote'"
|
||||
|
||||
def test_curly_double_quotes_normalized(self):
|
||||
"""Test that curly double quotes are converted to straight double quotes."""
|
||||
assert normalize_text("\u201CHello\u201D") == '"Hello"'
|
||||
assert normalize_text("\u201Equote\u201F") == '"quote"'
|
||||
|
||||
def test_backticks_normalized(self):
|
||||
"""Test that backticks and similar marks are converted to apostrophe."""
|
||||
assert normalize_text("code`here") == "code'here"
|
||||
assert normalize_text("acute\u00B4accent") == "acute'accent"
|
||||
assert normalize_text("prime\u2032mark") == "prime'mark"
|
||||
|
||||
def test_dashes_normalized(self):
|
||||
"""Test that various dashes are converted to hyphen-minus."""
|
||||
# EN DASH
|
||||
assert normalize_text("1990\u20132000") == "1990-2000"
|
||||
# EM DASH
|
||||
assert normalize_text("word\u2014word") == "word-word"
|
||||
# MINUS SIGN
|
||||
assert normalize_text("a\u2212b") == "a-b"
|
||||
# FIGURE DASH
|
||||
assert normalize_text("phone\u2012number") == "phone-number"
|
||||
|
||||
def test_whitespace_normalized(self):
|
||||
"""Test that various whitespace characters are normalized."""
|
||||
# NON-BREAKING SPACE
|
||||
assert normalize_text("hello\u00A0world") == "hello world"
|
||||
# EM SPACE
|
||||
assert normalize_text("hello\u2003world") == "hello world"
|
||||
# IDEOGRAPHIC SPACE (CJK full-width)
|
||||
assert normalize_text("hello\u3000world") == "hello world"
|
||||
# ZERO WIDTH SPACE (removed entirely)
|
||||
assert normalize_text("hello\u200Bworld") == "helloworld"
|
||||
|
||||
def test_ellipsis_normalized(self):
|
||||
"""Test that ellipsis character is converted to three dots."""
|
||||
assert normalize_text("Wait\u2026") == "Wait..."
|
||||
assert normalize_text("Loading\u2026please wait") == "Loading...please wait"
|
||||
|
||||
def test_multiple_spaces_collapsed(self):
|
||||
"""Test that multiple spaces are collapsed to one."""
|
||||
assert normalize_text("hello world") == "hello world"
|
||||
# After normalizing multiple whitespace chars
|
||||
assert normalize_text("hello\u00A0\u00A0\u00A0world") == "hello world"
|
||||
|
||||
def test_leading_trailing_whitespace_stripped(self):
|
||||
"""Test that leading/trailing whitespace is stripped."""
|
||||
assert normalize_text(" hello ") == "hello"
|
||||
assert normalize_text("\u00A0hello\u00A0") == "hello"
|
||||
|
||||
def test_none_input(self):
|
||||
"""Test that None input returns None."""
|
||||
assert normalize_text(None) is None
|
||||
|
||||
def test_non_string_input(self):
|
||||
"""Test that non-string input is returned unchanged."""
|
||||
assert normalize_text(123) == 123
|
||||
assert normalize_text(["list"]) == ["list"]
|
||||
|
||||
def test_international_characters_preserved(self):
|
||||
"""Test that international characters are NOT normalized away."""
|
||||
assert normalize_text("日本語") == "日本語"
|
||||
assert normalize_text("한국어") == "한국어"
|
||||
assert normalize_text("Café") == "Café"
|
||||
assert normalize_text("Björk") == "Björk"
|
||||
|
||||
def test_real_world_examples(self):
|
||||
"""Test real-world examples with mixed content."""
|
||||
# Job d49efab1's title
|
||||
assert normalize_text("Mama Says (You Can\u2019t Back Down)") == "Mama Says (You Can't Back Down)"
|
||||
# Broadway cast with smart quotes and em dash
|
||||
assert normalize_text("Footloose \u2014 \u201CFinal Song\u201D") == 'Footloose - "Final Song"'
|
||||
# Japanese with em dash
|
||||
assert normalize_text("宇多田ヒカル \u2014 First Love") == "宇多田ヒカル - First Love"
|
||||
|
||||
def test_idempotent(self):
|
||||
"""Test that normalizing twice gives the same result."""
|
||||
original = "It\u2019s \u201CMy\u201D Song \u2014 Test\u2026"
|
||||
once = normalize_text(original)
|
||||
twice = normalize_text(once)
|
||||
assert once == twice
|
||||
|
||||
def test_text_normalizations_dict_complete(self):
|
||||
"""Verify TEXT_NORMALIZATIONS includes all expected categories."""
|
||||
# Check apostrophe-like characters
|
||||
for char in APOSTROPHE_REPLACEMENTS:
|
||||
assert char in TEXT_NORMALIZATIONS
|
||||
|
||||
# Check double quote-like characters
|
||||
for char in DOUBLE_QUOTE_REPLACEMENTS:
|
||||
assert char in TEXT_NORMALIZATIONS
|
||||
|
||||
# Check dash-like characters
|
||||
for char in DASH_REPLACEMENTS:
|
||||
assert char in TEXT_NORMALIZATIONS
|
||||
|
||||
# Check whitespace characters
|
||||
for char in WHITESPACE_REPLACEMENTS:
|
||||
assert char in TEXT_NORMALIZATIONS
|
||||
|
||||
|
||||
class TestModelValidatorNormalization:
|
||||
"""Test that model validators correctly normalize input text."""
|
||||
|
||||
def test_job_create_normalizes_artist_title(self):
|
||||
"""Test that JobCreate model normalizes artist and title fields."""
|
||||
from backend.models.job import JobCreate
|
||||
|
||||
job = JobCreate(
|
||||
artist="Don\u2019t Stop",
|
||||
title="Believin\u2019 \u2014 Live",
|
||||
)
|
||||
assert job.artist == "Don't Stop"
|
||||
assert job.title == "Believin' - Live"
|
||||
|
||||
def test_job_create_preserves_international_chars(self):
|
||||
"""Test that JobCreate preserves international characters."""
|
||||
from backend.models.job import JobCreate
|
||||
|
||||
job = JobCreate(
|
||||
artist="宇多田ヒカル",
|
||||
title="First Love",
|
||||
)
|
||||
assert job.artist == "宇多田ヒカル"
|
||||
assert job.title == "First Love"
|
||||
|
||||
def test_job_create_handles_none(self):
|
||||
"""Test that JobCreate handles None values correctly."""
|
||||
from backend.models.job import JobCreate
|
||||
|
||||
job = JobCreate(
|
||||
artist=None,
|
||||
title="Some Title",
|
||||
)
|
||||
assert job.artist is None
|
||||
assert job.title == "Some Title"
|
||||
|
||||
def test_audio_search_normalizes_fields(self):
|
||||
"""Test that AudioSearchRequest normalizes text fields."""
|
||||
from backend.api.routes.audio_search import AudioSearchRequest
|
||||
|
||||
request = AudioSearchRequest(
|
||||
artist="Artist\u2019s Name",
|
||||
title="Song \u2014 Remix",
|
||||
display_artist="Display\u2019s Artist",
|
||||
display_title="Display\u2019s Title",
|
||||
)
|
||||
assert request.artist == "Artist's Name"
|
||||
assert request.title == "Song - Remix"
|
||||
assert request.display_artist == "Display's Artist"
|
||||
assert request.display_title == "Display's Title"
|
||||
|
||||
def test_audio_search_preserves_international_chars(self):
|
||||
"""Test that AudioSearchRequest preserves international characters."""
|
||||
from backend.api.routes.audio_search import AudioSearchRequest
|
||||
|
||||
request = AudioSearchRequest(
|
||||
artist="방탄소년단",
|
||||
title="Dynamite",
|
||||
)
|
||||
assert request.artist == "방탄소년단"
|
||||
assert request.title == "Dynamite"
|
||||
@@ -0,0 +1,847 @@
|
||||
"""
|
||||
Tests for Video Worker Orchestrator.
|
||||
|
||||
Tests cover:
|
||||
- OrchestratorConfig and OrchestratorResult dataclasses
|
||||
- VideoWorkerOrchestrator initialization
|
||||
- Individual stage methods
|
||||
- Full pipeline execution
|
||||
- Error handling and recovery
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch, AsyncMock
|
||||
from dataclasses import asdict
|
||||
|
||||
from backend.workers.video_worker_orchestrator import (
|
||||
OrchestratorConfig,
|
||||
OrchestratorResult,
|
||||
VideoWorkerOrchestrator,
|
||||
create_orchestrator_config_from_job,
|
||||
)
|
||||
|
||||
|
||||
class TestOrchestratorConfig:
|
||||
"""Test OrchestratorConfig dataclass."""
|
||||
|
||||
def test_required_fields(self):
|
||||
"""Test that required fields must be provided."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
)
|
||||
assert config.job_id == "test-job"
|
||||
assert config.artist == "Test Artist"
|
||||
assert config.encoding_backend == "auto"
|
||||
|
||||
def test_default_values(self):
|
||||
"""Test default values for optional fields."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
)
|
||||
assert config.enable_cdg is False
|
||||
assert config.enable_txt is False
|
||||
assert config.enable_youtube_upload is False
|
||||
assert config.dry_run is False
|
||||
assert config.non_interactive is True
|
||||
assert config.end_video_path is None
|
||||
|
||||
def test_all_fields(self):
|
||||
"""Test all fields including optional ones."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
end_video_path="/path/end.mov",
|
||||
enable_cdg=True,
|
||||
enable_txt=True,
|
||||
enable_youtube_upload=True,
|
||||
brand_prefix="TEST",
|
||||
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
|
||||
encoding_backend="gce",
|
||||
)
|
||||
assert config.enable_cdg is True
|
||||
assert config.encoding_backend == "gce"
|
||||
assert config.discord_webhook_url is not None
|
||||
|
||||
|
||||
class TestOrchestratorResult:
|
||||
"""Test OrchestratorResult dataclass."""
|
||||
|
||||
def test_success_result(self):
|
||||
"""Test successful result."""
|
||||
result = OrchestratorResult(
|
||||
success=True,
|
||||
final_video="/output/video.mp4",
|
||||
brand_code="TEST-1234",
|
||||
youtube_url="https://youtube.com/watch?v=test",
|
||||
)
|
||||
assert result.success is True
|
||||
assert result.error_message is None
|
||||
assert result.brand_code == "TEST-1234"
|
||||
|
||||
def test_failure_result(self):
|
||||
"""Test failure result."""
|
||||
result = OrchestratorResult(
|
||||
success=False,
|
||||
error_message="Encoding failed",
|
||||
)
|
||||
assert result.success is False
|
||||
assert result.error_message == "Encoding failed"
|
||||
assert result.final_video is None
|
||||
|
||||
def test_all_output_files(self):
|
||||
"""Test all output file fields."""
|
||||
result = OrchestratorResult(
|
||||
success=True,
|
||||
final_video="/output/lossless.mp4",
|
||||
final_video_mkv="/output/lossless.mkv",
|
||||
final_video_lossy="/output/lossy.mp4",
|
||||
final_video_720p="/output/720p.mp4",
|
||||
final_karaoke_cdg_zip="/output/cdg.zip",
|
||||
final_karaoke_txt_zip="/output/txt.zip",
|
||||
)
|
||||
assert result.final_video == "/output/lossless.mp4"
|
||||
assert result.final_video_mkv == "/output/lossless.mkv"
|
||||
|
||||
|
||||
class TestVideoWorkerOrchestratorInit:
|
||||
"""Test VideoWorkerOrchestrator initialization."""
|
||||
|
||||
def test_init_with_config(self):
|
||||
"""Test initialization with config."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
assert orchestrator.config == config
|
||||
assert orchestrator.result.success is False
|
||||
|
||||
def test_init_with_job_manager(self):
|
||||
"""Test initialization with job manager."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
)
|
||||
job_manager = MagicMock()
|
||||
orchestrator = VideoWorkerOrchestrator(config, job_manager=job_manager)
|
||||
|
||||
assert orchestrator.job_manager == job_manager
|
||||
|
||||
|
||||
class TestVideoWorkerOrchestratorServices:
|
||||
"""Test service lazy-loading."""
|
||||
|
||||
def test_get_encoding_backend(self):
|
||||
"""Test encoding backend lazy loading."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
encoding_backend="local",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
with patch("backend.services.encoding_interface.get_encoding_backend") as mock_get:
|
||||
mock_backend = MagicMock()
|
||||
mock_backend.name = "local"
|
||||
mock_get.return_value = mock_backend
|
||||
|
||||
backend = orchestrator._get_encoding_backend()
|
||||
|
||||
assert backend == mock_backend
|
||||
mock_get.assert_called_once()
|
||||
|
||||
def test_get_packaging_service(self):
|
||||
"""Test packaging service lazy loading."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
cdg_styles={"background_color": "black"},
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
with patch("backend.services.packaging_service.PackagingService") as MockService:
|
||||
mock_service = MagicMock()
|
||||
MockService.return_value = mock_service
|
||||
|
||||
service = orchestrator._get_packaging_service()
|
||||
|
||||
assert service == mock_service
|
||||
MockService.assert_called_once()
|
||||
|
||||
def test_get_youtube_service(self):
|
||||
"""Test YouTube service lazy loading."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
youtube_credentials={"token": "test"},
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
with patch("backend.services.youtube_upload_service.YouTubeUploadService") as MockService:
|
||||
mock_service = MagicMock()
|
||||
MockService.return_value = mock_service
|
||||
|
||||
service = orchestrator._get_youtube_service()
|
||||
|
||||
assert service == mock_service
|
||||
MockService.assert_called_once()
|
||||
|
||||
def test_get_discord_service(self):
|
||||
"""Test Discord service lazy loading."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
with patch("backend.services.discord_service.DiscordNotificationService") as MockService:
|
||||
mock_service = MagicMock()
|
||||
MockService.return_value = mock_service
|
||||
|
||||
service = orchestrator._get_discord_service()
|
||||
|
||||
assert service == mock_service
|
||||
MockService.assert_called_once()
|
||||
|
||||
|
||||
class TestVideoWorkerOrchestratorPackaging:
|
||||
"""Test packaging stage."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_packaging_no_lrc(self):
|
||||
"""Test packaging stage with no LRC file."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
enable_cdg=True,
|
||||
lrc_file_path=None,
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
# Should not raise, just skip
|
||||
await orchestrator._run_packaging()
|
||||
|
||||
assert orchestrator.result.final_karaoke_cdg_zip is None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_packaging_cdg(self):
|
||||
"""Test CDG packaging."""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
lrc_file = os.path.join(temp_dir, "test.lrc")
|
||||
audio_file = os.path.join(temp_dir, "test.flac")
|
||||
|
||||
# Create dummy files
|
||||
with open(lrc_file, "w") as f:
|
||||
f.write("[00:00.00]Test lyrics")
|
||||
with open(audio_file, "w") as f:
|
||||
f.write("dummy audio")
|
||||
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path=audio_file,
|
||||
lrc_file_path=lrc_file,
|
||||
output_dir=temp_dir,
|
||||
enable_cdg=True,
|
||||
cdg_styles={"background_color": "black"},
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
with patch.object(orchestrator, "_get_packaging_service") as mock_get:
|
||||
mock_service = MagicMock()
|
||||
mock_service.create_cdg_package.return_value = (
|
||||
f"{temp_dir}/cdg.zip",
|
||||
f"{temp_dir}/test.mp3",
|
||||
f"{temp_dir}/test.cdg",
|
||||
)
|
||||
mock_get.return_value = mock_service
|
||||
|
||||
await orchestrator._run_packaging()
|
||||
|
||||
mock_service.create_cdg_package.assert_called_once()
|
||||
assert orchestrator.result.final_karaoke_cdg_zip == f"{temp_dir}/cdg.zip"
|
||||
|
||||
|
||||
class TestVideoWorkerOrchestratorEncoding:
|
||||
"""Test encoding stage."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_encoding_success(self):
|
||||
"""Test successful encoding."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
output_dir="/output",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
|
||||
from backend.services.encoding_interface import EncodingOutput
|
||||
|
||||
mock_backend = MagicMock()
|
||||
mock_backend.name = "local"
|
||||
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
|
||||
success=True,
|
||||
lossless_4k_mp4_path="/output/lossless.mp4",
|
||||
lossy_4k_mp4_path="/output/lossy.mp4",
|
||||
lossy_720p_mp4_path="/output/720p.mp4",
|
||||
lossless_mkv_path="/output/lossless.mkv",
|
||||
encoding_time_seconds=120.5,
|
||||
encoding_backend="local",
|
||||
))
|
||||
mock_get.return_value = mock_backend
|
||||
|
||||
await orchestrator._run_encoding()
|
||||
|
||||
mock_backend.encode.assert_called_once()
|
||||
assert orchestrator.result.final_video == "/output/lossless.mp4"
|
||||
assert orchestrator.result.final_video_720p == "/output/720p.mp4"
|
||||
assert orchestrator.result.encoding_time_seconds == 120.5
|
||||
|
||||
def test_encoding_input_gcs_paths_pattern(self):
|
||||
"""Test that EncodingInput.options contains proper GCS paths structure.
|
||||
|
||||
This test verifies the GCS path construction pattern that GCE encoding
|
||||
requires. It tests the structure without running _run_encoding().
|
||||
|
||||
This would have caught: 'GCE encoding requires input_gcs_path and
|
||||
output_gcs_path in options' error when the orchestrator didn't pass
|
||||
the required paths for GCE encoding.
|
||||
"""
|
||||
from backend.services.encoding_interface import EncodingInput
|
||||
|
||||
# Test that EncodingInput can hold GCS paths in options
|
||||
# This mirrors what the orchestrator should build
|
||||
job_id = "test-job-123"
|
||||
bucket = "test-bucket"
|
||||
input_gcs_path = f"gs://{bucket}/jobs/{job_id}/"
|
||||
output_gcs_path = f"gs://{bucket}/jobs/{job_id}/finals/"
|
||||
|
||||
encoding_input = EncodingInput(
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
brand_code="TEST-001",
|
||||
output_dir="/output",
|
||||
options={
|
||||
"input_gcs_path": input_gcs_path,
|
||||
"output_gcs_path": output_gcs_path,
|
||||
},
|
||||
)
|
||||
|
||||
# Verify the structure that GCEEncodingBackend expects
|
||||
assert "input_gcs_path" in encoding_input.options, \
|
||||
"EncodingInput.options must include input_gcs_path for GCE encoding"
|
||||
assert "output_gcs_path" in encoding_input.options, \
|
||||
"EncodingInput.options must include output_gcs_path for GCE encoding"
|
||||
# Verify path format
|
||||
assert encoding_input.options["input_gcs_path"].startswith("gs://")
|
||||
assert encoding_input.options["output_gcs_path"].startswith("gs://")
|
||||
assert job_id in encoding_input.options["input_gcs_path"]
|
||||
assert job_id in encoding_input.options["output_gcs_path"]
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_encoding_failure(self):
|
||||
"""Test encoding failure."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
|
||||
from backend.services.encoding_interface import EncodingOutput
|
||||
|
||||
mock_backend = MagicMock()
|
||||
mock_backend.name = "local"
|
||||
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
|
||||
success=False,
|
||||
error_message="FFmpeg failed",
|
||||
encoding_backend="local",
|
||||
))
|
||||
mock_get.return_value = mock_backend
|
||||
|
||||
with pytest.raises(Exception) as exc_info:
|
||||
await orchestrator._run_encoding()
|
||||
|
||||
assert "Encoding failed" in str(exc_info.value)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_encoding_gce_downloads_files(self):
|
||||
"""Test that GCE encoding downloads files from GCS to local directory.
|
||||
|
||||
This test verifies the fix for YouTube upload failure when using GCE encoding.
|
||||
GCE encoding returns GCS blob paths, which need to be downloaded locally
|
||||
before YouTube upload can access them.
|
||||
"""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path=os.path.join(temp_dir, "title.mov"),
|
||||
karaoke_video_path=os.path.join(temp_dir, "karaoke.mov"),
|
||||
instrumental_audio_path=os.path.join(temp_dir, "audio.flac"),
|
||||
output_dir=temp_dir,
|
||||
)
|
||||
|
||||
# Mock storage service
|
||||
mock_storage = MagicMock()
|
||||
mock_storage.download_file = MagicMock()
|
||||
|
||||
orchestrator = VideoWorkerOrchestrator(
|
||||
config,
|
||||
storage=mock_storage,
|
||||
)
|
||||
|
||||
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
|
||||
from backend.services.encoding_interface import EncodingOutput
|
||||
|
||||
# GCE backend returns GCS blob paths (not local paths)
|
||||
mock_backend = MagicMock()
|
||||
mock_backend.name = "gce" # Important: must be "gce" to trigger download
|
||||
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
|
||||
success=True,
|
||||
lossless_4k_mp4_path="jobs/test-job/finals/output_4k_lossless.mp4",
|
||||
lossy_4k_mp4_path="jobs/test-job/finals/output_4k_lossy.mp4",
|
||||
lossy_720p_mp4_path="jobs/test-job/finals/output_720p.mp4",
|
||||
lossless_mkv_path="jobs/test-job/finals/output_4k.mkv",
|
||||
encoding_time_seconds=60.0,
|
||||
encoding_backend="gce",
|
||||
))
|
||||
mock_get.return_value = mock_backend
|
||||
|
||||
await orchestrator._run_encoding()
|
||||
|
||||
# Verify download_file was called for each output file
|
||||
assert mock_storage.download_file.call_count == 4
|
||||
|
||||
# Verify the result paths were updated to local paths
|
||||
assert orchestrator.result.final_video.startswith(temp_dir)
|
||||
assert orchestrator.result.final_video_mkv.startswith(temp_dir)
|
||||
assert orchestrator.result.final_video_lossy.startswith(temp_dir)
|
||||
assert orchestrator.result.final_video_720p.startswith(temp_dir)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_encoding_local_does_not_download(self):
|
||||
"""Test that local encoding does NOT trigger GCS download.
|
||||
|
||||
Local encoding produces files directly in the output directory,
|
||||
so no download is needed.
|
||||
"""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path=os.path.join(temp_dir, "title.mov"),
|
||||
karaoke_video_path=os.path.join(temp_dir, "karaoke.mov"),
|
||||
instrumental_audio_path=os.path.join(temp_dir, "audio.flac"),
|
||||
output_dir=temp_dir,
|
||||
)
|
||||
|
||||
# Mock storage service
|
||||
mock_storage = MagicMock()
|
||||
mock_storage.download_file = MagicMock()
|
||||
|
||||
orchestrator = VideoWorkerOrchestrator(
|
||||
config,
|
||||
storage=mock_storage,
|
||||
)
|
||||
|
||||
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
|
||||
from backend.services.encoding_interface import EncodingOutput
|
||||
|
||||
# Local backend returns local paths directly
|
||||
mock_backend = MagicMock()
|
||||
mock_backend.name = "local" # Local backend
|
||||
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
|
||||
success=True,
|
||||
lossless_4k_mp4_path=os.path.join(temp_dir, "lossless.mp4"),
|
||||
lossy_4k_mp4_path=os.path.join(temp_dir, "lossy.mp4"),
|
||||
lossy_720p_mp4_path=os.path.join(temp_dir, "720p.mp4"),
|
||||
lossless_mkv_path=os.path.join(temp_dir, "lossless.mkv"),
|
||||
encoding_time_seconds=120.0,
|
||||
encoding_backend="local",
|
||||
))
|
||||
mock_get.return_value = mock_backend
|
||||
|
||||
await orchestrator._run_encoding()
|
||||
|
||||
# Verify download_file was NOT called for local encoding
|
||||
mock_storage.download_file.assert_not_called()
|
||||
|
||||
|
||||
class TestVideoWorkerOrchestratorOrganization:
|
||||
"""Test organization stage."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_organization_keep_brand_code(self):
|
||||
"""Test organization with existing brand code."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
keep_brand_code="NOMAD-9999",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
await orchestrator._run_organization()
|
||||
|
||||
assert orchestrator.result.brand_code == "NOMAD-9999"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_organization_generate_brand_code(self):
|
||||
"""Test brand code generation from Dropbox."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
dropbox_path="/Karaoke/Tracks",
|
||||
brand_prefix="TEST",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
with patch("backend.services.dropbox_service.get_dropbox_service") as mock_get:
|
||||
mock_dropbox = MagicMock()
|
||||
mock_dropbox.is_configured = True
|
||||
mock_dropbox.get_next_brand_code.return_value = "TEST-0001"
|
||||
mock_get.return_value = mock_dropbox
|
||||
|
||||
await orchestrator._run_organization()
|
||||
|
||||
assert orchestrator.result.brand_code == "TEST-0001"
|
||||
|
||||
|
||||
class TestVideoWorkerOrchestratorDistribution:
|
||||
"""Test distribution stage."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_upload_to_youtube(self):
|
||||
"""Test YouTube upload."""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
video_file = os.path.join(temp_dir, "test.mp4")
|
||||
with open(video_file, "w") as f:
|
||||
f.write("dummy video")
|
||||
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
enable_youtube_upload=True,
|
||||
youtube_credentials={"token": "test"},
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
orchestrator.result.final_video_lossy = video_file
|
||||
|
||||
with patch.object(orchestrator, "_get_youtube_service") as mock_get:
|
||||
mock_service = MagicMock()
|
||||
mock_service.upload_video.return_value = (
|
||||
"video123",
|
||||
"https://youtube.com/watch?v=video123"
|
||||
)
|
||||
mock_get.return_value = mock_service
|
||||
|
||||
await orchestrator._upload_to_youtube()
|
||||
|
||||
mock_service.upload_video.assert_called_once()
|
||||
assert orchestrator.result.youtube_url == "https://youtube.com/watch?v=video123"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_upload_to_dropbox(self):
|
||||
"""Test Dropbox upload."""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
output_dir=temp_dir,
|
||||
dropbox_path="/Karaoke/Tracks",
|
||||
brand_prefix="TEST",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
orchestrator.result.brand_code = "TEST-0001"
|
||||
|
||||
with patch("backend.services.dropbox_service.get_dropbox_service") as mock_get:
|
||||
mock_dropbox = MagicMock()
|
||||
mock_dropbox.is_configured = True
|
||||
mock_dropbox.create_shared_link.return_value = "https://dropbox.com/link"
|
||||
mock_get.return_value = mock_dropbox
|
||||
|
||||
await orchestrator._upload_to_dropbox()
|
||||
|
||||
mock_dropbox.upload_folder.assert_called_once()
|
||||
assert orchestrator.result.dropbox_link == "https://dropbox.com/link"
|
||||
|
||||
|
||||
class TestVideoWorkerOrchestratorNotifications:
|
||||
"""Test notifications stage."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_notifications_with_youtube_url(self):
|
||||
"""Test Discord notification with YouTube URL."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
orchestrator.result.youtube_url = "https://youtube.com/watch?v=test"
|
||||
|
||||
with patch.object(orchestrator, "_get_discord_service") as mock_get:
|
||||
mock_service = MagicMock()
|
||||
mock_service.post_video_notification.return_value = True
|
||||
mock_get.return_value = mock_service
|
||||
|
||||
await orchestrator._run_notifications()
|
||||
|
||||
mock_service.post_video_notification.assert_called_once_with(
|
||||
"https://youtube.com/watch?v=test"
|
||||
)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_notifications_no_youtube_url(self):
|
||||
"""Test notification skipped without YouTube URL."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
discord_webhook_url="https://discord.com/api/webhooks/123/abc",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
# No youtube_url set
|
||||
|
||||
with patch.object(orchestrator, "_get_discord_service") as mock_get:
|
||||
mock_service = MagicMock()
|
||||
mock_get.return_value = mock_service
|
||||
|
||||
await orchestrator._run_notifications()
|
||||
|
||||
mock_service.post_video_notification.assert_not_called()
|
||||
|
||||
|
||||
class TestVideoWorkerOrchestratorFullPipeline:
|
||||
"""Test full pipeline execution."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_full_pipeline_success(self):
|
||||
"""Test successful full pipeline."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
enable_cdg=True, # Enable to trigger packaging stage
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
# Mock all stages
|
||||
with patch.object(orchestrator, "_run_packaging", new_callable=AsyncMock) as mock_packaging, \
|
||||
patch.object(orchestrator, "_run_encoding", new_callable=AsyncMock) as mock_encoding, \
|
||||
patch.object(orchestrator, "_run_organization", new_callable=AsyncMock) as mock_org, \
|
||||
patch.object(orchestrator, "_run_distribution", new_callable=AsyncMock) as mock_dist, \
|
||||
patch.object(orchestrator, "_run_notifications", new_callable=AsyncMock) as mock_notify:
|
||||
|
||||
result = await orchestrator.run()
|
||||
|
||||
assert result.success is True
|
||||
mock_packaging.assert_called_once()
|
||||
mock_encoding.assert_called_once()
|
||||
mock_org.assert_called_once()
|
||||
mock_dist.assert_called_once()
|
||||
mock_notify.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_pipeline_skips_packaging_when_disabled(self):
|
||||
"""Test pipeline skips packaging when CDG/TXT disabled."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
enable_cdg=False,
|
||||
enable_txt=False,
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
# Mock all stages
|
||||
with patch.object(orchestrator, "_run_packaging", new_callable=AsyncMock) as mock_packaging, \
|
||||
patch.object(orchestrator, "_run_encoding", new_callable=AsyncMock) as mock_encoding, \
|
||||
patch.object(orchestrator, "_run_organization", new_callable=AsyncMock), \
|
||||
patch.object(orchestrator, "_run_distribution", new_callable=AsyncMock), \
|
||||
patch.object(orchestrator, "_run_notifications", new_callable=AsyncMock):
|
||||
|
||||
result = await orchestrator.run()
|
||||
|
||||
assert result.success is True
|
||||
mock_packaging.assert_not_called()
|
||||
mock_encoding.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_full_pipeline_encoding_failure(self):
|
||||
"""Test pipeline failure during encoding."""
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
)
|
||||
orchestrator = VideoWorkerOrchestrator(config)
|
||||
|
||||
with patch.object(orchestrator, "_run_packaging", new_callable=AsyncMock), \
|
||||
patch.object(orchestrator, "_run_encoding", new_callable=AsyncMock) as mock_encoding, \
|
||||
patch.object(orchestrator, "_run_organization", new_callable=AsyncMock) as mock_org:
|
||||
|
||||
mock_encoding.side_effect = Exception("Encoding failed")
|
||||
|
||||
result = await orchestrator.run()
|
||||
|
||||
assert result.success is False
|
||||
assert "Encoding failed" in result.error_message
|
||||
mock_org.assert_not_called()
|
||||
|
||||
|
||||
class TestCreateOrchestratorConfigFromJob:
|
||||
"""Test helper function for creating config from job."""
|
||||
|
||||
def test_create_config_from_job(self):
|
||||
"""Test creating config from a job object."""
|
||||
job = MagicMock()
|
||||
job.job_id = "test-123"
|
||||
job.artist = "Test Artist"
|
||||
job.title = "Test Title"
|
||||
job.state_data = {"instrumental_selection": "clean"}
|
||||
job.enable_cdg = True
|
||||
job.enable_txt = False
|
||||
job.enable_youtube_upload = True
|
||||
job.brand_prefix = "NOMAD"
|
||||
job.discord_webhook_url = "https://discord.com/api/webhooks/123/abc"
|
||||
job.youtube_description_template = "Test description"
|
||||
job.dropbox_path = "/Karaoke"
|
||||
job.gdrive_folder_id = None
|
||||
job.keep_brand_code = None
|
||||
job.existing_instrumental_gcs_path = None
|
||||
|
||||
config = create_orchestrator_config_from_job(
|
||||
job=job,
|
||||
temp_dir="/tmp/test",
|
||||
youtube_credentials={"token": "test"},
|
||||
cdg_styles={"background": "black"},
|
||||
)
|
||||
|
||||
assert config.job_id == "test-123"
|
||||
assert config.artist == "Test Artist"
|
||||
assert config.title == "Test Title"
|
||||
assert config.enable_cdg is True
|
||||
assert config.enable_youtube_upload is True
|
||||
assert config.title_video_path == "/tmp/test/Test Artist - Test Title (Title).mov"
|
||||
assert config.instrumental_audio_path == "/tmp/test/Test Artist - Test Title (Instrumental Clean).flac"
|
||||
|
||||
def test_create_config_from_job_with_existing_instrumental(self):
|
||||
"""Test config with user-provided instrumental."""
|
||||
job = MagicMock()
|
||||
job.job_id = "test-123"
|
||||
job.artist = "Test Artist"
|
||||
job.title = "Test Title"
|
||||
job.state_data = {"instrumental_selection": "custom"}
|
||||
job.enable_cdg = False
|
||||
job.enable_txt = False
|
||||
job.enable_youtube_upload = False
|
||||
job.brand_prefix = None
|
||||
job.discord_webhook_url = None
|
||||
job.youtube_description_template = None
|
||||
job.dropbox_path = None
|
||||
job.gdrive_folder_id = None
|
||||
job.keep_brand_code = None
|
||||
job.existing_instrumental_gcs_path = "gs://bucket/instrumental.mp3"
|
||||
|
||||
config = create_orchestrator_config_from_job(
|
||||
job=job,
|
||||
temp_dir="/tmp/test",
|
||||
)
|
||||
|
||||
assert config.instrumental_audio_path == "/tmp/test/Test Artist - Test Title (Instrumental User).mp3"
|
||||
@@ -0,0 +1,509 @@
|
||||
"""
|
||||
Unit tests for worker log subcollection functionality.
|
||||
|
||||
Tests the WorkerLogEntry model, FirestoreService subcollection methods,
|
||||
and JobManager log operations with the feature flag.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, patch
|
||||
from datetime import datetime, timezone, timedelta
|
||||
import uuid
|
||||
|
||||
# Mock Firestore before imports
|
||||
import sys
|
||||
sys.modules['google.cloud.firestore'] = MagicMock()
|
||||
sys.modules['google.cloud.firestore_v1'] = MagicMock()
|
||||
|
||||
from backend.models.worker_log import WorkerLogEntry, DEFAULT_LOG_TTL_DAYS
|
||||
|
||||
|
||||
class TestWorkerLogEntry:
|
||||
"""Tests for WorkerLogEntry dataclass."""
|
||||
|
||||
def test_create_log_entry(self):
|
||||
"""Test creating a log entry with factory method."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test message"
|
||||
)
|
||||
|
||||
assert entry.job_id == "job123"
|
||||
assert entry.worker == "audio"
|
||||
assert entry.level == "INFO"
|
||||
assert entry.message == "Test message"
|
||||
assert entry.id is not None
|
||||
assert entry.timestamp is not None
|
||||
assert entry.ttl_expiry is not None
|
||||
# TTL should be ~30 days from now
|
||||
expected_ttl = datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
|
||||
assert abs((entry.ttl_expiry - expected_ttl).total_seconds()) < 5
|
||||
|
||||
def test_create_log_entry_truncates_long_message(self):
|
||||
"""Test that messages longer than 1000 chars are truncated."""
|
||||
long_message = "x" * 2000
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message=long_message
|
||||
)
|
||||
|
||||
assert len(entry.message) == 1000
|
||||
|
||||
def test_create_log_entry_normalizes_level(self):
|
||||
"""Test that log level is normalized to uppercase."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="info",
|
||||
message="Test"
|
||||
)
|
||||
|
||||
assert entry.level == "INFO"
|
||||
|
||||
def test_create_log_entry_custom_ttl(self):
|
||||
"""Test creating entry with custom TTL."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test",
|
||||
ttl_days=7
|
||||
)
|
||||
|
||||
expected_ttl = datetime.now(timezone.utc) + timedelta(days=7)
|
||||
assert abs((entry.ttl_expiry - expected_ttl).total_seconds()) < 5
|
||||
|
||||
def test_create_log_entry_with_metadata(self):
|
||||
"""Test creating entry with metadata."""
|
||||
metadata = {"file_size": 1024, "duration": 300}
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test",
|
||||
metadata=metadata
|
||||
)
|
||||
|
||||
assert entry.metadata == metadata
|
||||
|
||||
def test_to_dict(self):
|
||||
"""Test converting entry to dict for Firestore."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test message",
|
||||
metadata={"key": "value"}
|
||||
)
|
||||
|
||||
d = entry.to_dict()
|
||||
|
||||
assert d["job_id"] == "job123"
|
||||
assert d["worker"] == "audio"
|
||||
assert d["level"] == "INFO"
|
||||
assert d["message"] == "Test message"
|
||||
assert d["metadata"] == {"key": "value"}
|
||||
assert "id" in d
|
||||
assert "timestamp" in d
|
||||
assert "ttl_expiry" in d
|
||||
|
||||
def test_to_dict_without_metadata(self):
|
||||
"""Test to_dict doesn't include metadata when None."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test"
|
||||
)
|
||||
|
||||
d = entry.to_dict()
|
||||
assert "metadata" not in d
|
||||
|
||||
def test_to_legacy_dict(self):
|
||||
"""Test converting to legacy format for API compatibility."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test message"
|
||||
)
|
||||
|
||||
d = entry.to_legacy_dict()
|
||||
|
||||
assert "timestamp" in d
|
||||
assert d["level"] == "INFO"
|
||||
assert d["worker"] == "audio"
|
||||
assert d["message"] == "Test message"
|
||||
# Should not include id, job_id, ttl_expiry, metadata
|
||||
assert "id" not in d
|
||||
assert "job_id" not in d
|
||||
assert "ttl_expiry" not in d
|
||||
|
||||
def test_from_dict(self):
|
||||
"""Test creating entry from Firestore document."""
|
||||
timestamp = datetime.now(timezone.utc)
|
||||
ttl_expiry = timestamp + timedelta(days=30)
|
||||
|
||||
data = {
|
||||
"id": "log123",
|
||||
"job_id": "job123",
|
||||
"timestamp": timestamp,
|
||||
"level": "WARNING",
|
||||
"worker": "video",
|
||||
"message": "Warning message",
|
||||
"metadata": {"error_code": 500},
|
||||
"ttl_expiry": ttl_expiry
|
||||
}
|
||||
|
||||
entry = WorkerLogEntry.from_dict(data)
|
||||
|
||||
assert entry.id == "log123"
|
||||
assert entry.job_id == "job123"
|
||||
assert entry.level == "WARNING"
|
||||
assert entry.worker == "video"
|
||||
assert entry.message == "Warning message"
|
||||
assert entry.metadata == {"error_code": 500}
|
||||
|
||||
def test_from_dict_with_iso_strings(self):
|
||||
"""Test creating entry from dict with ISO format strings."""
|
||||
data = {
|
||||
"timestamp": "2026-01-04T12:00:00Z",
|
||||
"level": "INFO",
|
||||
"worker": "audio",
|
||||
"message": "Test"
|
||||
}
|
||||
|
||||
entry = WorkerLogEntry.from_dict(data)
|
||||
|
||||
assert entry.timestamp.tzinfo == timezone.utc
|
||||
assert entry.message == "Test"
|
||||
|
||||
def test_from_dict_missing_fields(self):
|
||||
"""Test creating entry with missing fields uses defaults."""
|
||||
data = {
|
||||
"message": "Minimal log"
|
||||
}
|
||||
|
||||
entry = WorkerLogEntry.from_dict(data)
|
||||
|
||||
assert entry.message == "Minimal log"
|
||||
assert entry.level == "INFO" # Default
|
||||
assert entry.worker == "unknown" # Default
|
||||
assert entry.id is not None # Generated
|
||||
|
||||
|
||||
class TestFirestoreServiceSubcollection:
|
||||
"""Tests for FirestoreService log subcollection methods."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_db(self):
|
||||
"""Create mock Firestore database."""
|
||||
return MagicMock()
|
||||
|
||||
@pytest.fixture
|
||||
def firestore_service(self, mock_db):
|
||||
"""Create FirestoreService with mocked DB."""
|
||||
with patch('backend.services.firestore_service.firestore') as mock_firestore:
|
||||
mock_firestore.Client.return_value = mock_db
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
service = FirestoreService()
|
||||
service.db = mock_db
|
||||
return service
|
||||
|
||||
def test_append_log_to_subcollection(self, firestore_service, mock_db):
|
||||
"""Test appending log to subcollection."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test message"
|
||||
)
|
||||
|
||||
# Setup mock chain
|
||||
mock_collection = MagicMock()
|
||||
mock_doc = MagicMock()
|
||||
mock_logs_ref = MagicMock()
|
||||
mock_log_doc = MagicMock()
|
||||
|
||||
mock_db.collection.return_value = mock_collection
|
||||
mock_collection.document.return_value = mock_doc
|
||||
mock_doc.collection.return_value = mock_logs_ref
|
||||
mock_logs_ref.document.return_value = mock_log_doc
|
||||
|
||||
firestore_service.append_log_to_subcollection("job123", entry)
|
||||
|
||||
# Verify correct path: jobs/{job_id}/logs/{log_id}
|
||||
mock_db.collection.assert_called_with("jobs")
|
||||
mock_collection.document.assert_called_with("job123")
|
||||
mock_doc.collection.assert_called_with("logs")
|
||||
mock_logs_ref.document.assert_called_with(entry.id)
|
||||
mock_log_doc.set.assert_called_once()
|
||||
|
||||
def test_get_logs_from_subcollection(self, firestore_service, mock_db):
|
||||
"""Test getting logs from subcollection."""
|
||||
# Setup mock chain
|
||||
mock_collection = MagicMock()
|
||||
mock_doc = MagicMock()
|
||||
mock_logs_ref = MagicMock()
|
||||
mock_query = MagicMock()
|
||||
|
||||
mock_db.collection.return_value = mock_collection
|
||||
mock_collection.document.return_value = mock_doc
|
||||
mock_doc.collection.return_value = mock_logs_ref
|
||||
mock_logs_ref.order_by.return_value = mock_query
|
||||
mock_query.limit.return_value = mock_query
|
||||
|
||||
# Mock document data
|
||||
mock_doc1 = MagicMock()
|
||||
mock_doc1.to_dict.return_value = {
|
||||
"timestamp": datetime.now(timezone.utc),
|
||||
"level": "INFO",
|
||||
"worker": "audio",
|
||||
"message": "Log 1"
|
||||
}
|
||||
|
||||
mock_query.stream.return_value = iter([mock_doc1])
|
||||
|
||||
logs = firestore_service.get_logs_from_subcollection("job123")
|
||||
|
||||
assert len(logs) == 1
|
||||
assert logs[0].message == "Log 1"
|
||||
mock_logs_ref.order_by.assert_called_once()
|
||||
|
||||
def test_get_logs_from_subcollection_with_worker_filter(self, firestore_service, mock_db):
|
||||
"""Test filtering logs by worker."""
|
||||
mock_collection = MagicMock()
|
||||
mock_doc = MagicMock()
|
||||
mock_logs_ref = MagicMock()
|
||||
mock_query = MagicMock()
|
||||
|
||||
mock_db.collection.return_value = mock_collection
|
||||
mock_collection.document.return_value = mock_doc
|
||||
mock_doc.collection.return_value = mock_logs_ref
|
||||
mock_logs_ref.order_by.return_value = mock_query
|
||||
mock_query.where.return_value = mock_query
|
||||
mock_query.limit.return_value = mock_query
|
||||
mock_query.stream.return_value = iter([])
|
||||
|
||||
firestore_service.get_logs_from_subcollection("job123", worker="audio")
|
||||
|
||||
# Verify where clause was added
|
||||
mock_query.where.assert_called()
|
||||
|
||||
def test_delete_logs_subcollection(self, firestore_service, mock_db):
|
||||
"""Test deleting all logs in subcollection."""
|
||||
mock_collection = MagicMock()
|
||||
mock_doc = MagicMock()
|
||||
mock_logs_ref = MagicMock()
|
||||
mock_batch = MagicMock()
|
||||
|
||||
mock_db.collection.return_value = mock_collection
|
||||
mock_collection.document.return_value = mock_doc
|
||||
mock_doc.collection.return_value = mock_logs_ref
|
||||
mock_db.batch.return_value = mock_batch
|
||||
|
||||
# First call returns 2 docs, second call returns empty
|
||||
mock_log_doc1 = MagicMock()
|
||||
mock_log_doc2 = MagicMock()
|
||||
|
||||
call_count = [0]
|
||||
def stream_side_effect(*args, **kwargs):
|
||||
call_count[0] += 1
|
||||
if call_count[0] == 1:
|
||||
return iter([mock_log_doc1, mock_log_doc2])
|
||||
return iter([])
|
||||
|
||||
mock_logs_ref.limit.return_value = mock_logs_ref
|
||||
mock_logs_ref.stream.side_effect = stream_side_effect
|
||||
|
||||
deleted = firestore_service.delete_logs_subcollection("job123")
|
||||
|
||||
assert deleted == 2
|
||||
mock_batch.commit.assert_called()
|
||||
|
||||
|
||||
class TestJobManagerLogging:
|
||||
"""Tests for JobManager log methods with feature flag."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_firestore_service(self):
|
||||
"""Mock FirestoreService."""
|
||||
with patch('backend.services.job_manager.FirestoreService') as mock:
|
||||
service = MagicMock()
|
||||
mock.return_value = service
|
||||
yield service
|
||||
|
||||
@pytest.fixture
|
||||
def mock_storage_service(self):
|
||||
"""Mock StorageService."""
|
||||
with patch('backend.services.job_manager.StorageService') as mock:
|
||||
service = MagicMock()
|
||||
mock.return_value = service
|
||||
yield service
|
||||
|
||||
def test_append_worker_log_uses_subcollection_when_enabled(
|
||||
self, mock_firestore_service, mock_storage_service
|
||||
):
|
||||
"""Test that logs go to subcollection when feature is enabled."""
|
||||
with patch('backend.services.job_manager.settings') as mock_settings:
|
||||
mock_settings.use_log_subcollection = True
|
||||
|
||||
from backend.services.job_manager import JobManager
|
||||
manager = JobManager()
|
||||
|
||||
manager.append_worker_log(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test message"
|
||||
)
|
||||
|
||||
# Should call subcollection method
|
||||
mock_firestore_service.append_log_to_subcollection.assert_called_once()
|
||||
# Should NOT call legacy method
|
||||
mock_firestore_service.append_worker_log.assert_not_called()
|
||||
|
||||
def test_append_worker_log_uses_array_when_disabled(
|
||||
self, mock_firestore_service, mock_storage_service
|
||||
):
|
||||
"""Test that logs go to embedded array when feature is disabled."""
|
||||
with patch('backend.services.job_manager.settings') as mock_settings:
|
||||
mock_settings.use_log_subcollection = False
|
||||
|
||||
from backend.services.job_manager import JobManager
|
||||
manager = JobManager()
|
||||
|
||||
manager.append_worker_log(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test message"
|
||||
)
|
||||
|
||||
# Should call legacy method
|
||||
mock_firestore_service.append_worker_log.assert_called_once()
|
||||
# Should NOT call subcollection method
|
||||
mock_firestore_service.append_log_to_subcollection.assert_not_called()
|
||||
|
||||
def test_get_worker_logs_from_subcollection(
|
||||
self, mock_firestore_service, mock_storage_service
|
||||
):
|
||||
"""Test getting logs from subcollection."""
|
||||
with patch('backend.services.job_manager.settings') as mock_settings:
|
||||
mock_settings.use_log_subcollection = True
|
||||
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test"
|
||||
)
|
||||
mock_firestore_service.get_logs_from_subcollection.return_value = [entry]
|
||||
|
||||
from backend.services.job_manager import JobManager
|
||||
manager = JobManager()
|
||||
|
||||
logs = manager.get_worker_logs("job123")
|
||||
|
||||
assert len(logs) == 1
|
||||
assert logs[0]["message"] == "Test"
|
||||
mock_firestore_service.get_logs_from_subcollection.assert_called_once()
|
||||
|
||||
def test_get_worker_logs_falls_back_to_array(
|
||||
self, mock_firestore_service, mock_storage_service
|
||||
):
|
||||
"""Test fallback to embedded array for legacy jobs."""
|
||||
with patch('backend.services.job_manager.settings') as mock_settings:
|
||||
mock_settings.use_log_subcollection = True
|
||||
|
||||
# Subcollection returns empty
|
||||
mock_firestore_service.get_logs_from_subcollection.return_value = []
|
||||
|
||||
# Mock job with legacy logs
|
||||
from backend.models.job import Job, JobStatus
|
||||
mock_job = Job(
|
||||
job_id="job123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
worker_logs=[
|
||||
{"timestamp": "2026-01-04T12:00:00Z", "level": "INFO", "worker": "audio", "message": "Legacy log"}
|
||||
]
|
||||
)
|
||||
mock_firestore_service.get_job.return_value = mock_job
|
||||
|
||||
from backend.services.job_manager import JobManager
|
||||
manager = JobManager()
|
||||
|
||||
logs = manager.get_worker_logs("job123")
|
||||
|
||||
assert len(logs) == 1
|
||||
assert logs[0]["message"] == "Legacy log"
|
||||
|
||||
def test_delete_job_deletes_logs_subcollection(
|
||||
self, mock_firestore_service, mock_storage_service
|
||||
):
|
||||
"""Test that deleting a job also deletes its logs subcollection."""
|
||||
mock_firestore_service.get_job.return_value = None
|
||||
mock_firestore_service.delete_logs_subcollection.return_value = 5
|
||||
|
||||
from backend.services.job_manager import JobManager
|
||||
manager = JobManager()
|
||||
|
||||
manager.delete_job("job123", delete_files=False)
|
||||
|
||||
mock_firestore_service.delete_logs_subcollection.assert_called_once_with("job123")
|
||||
mock_firestore_service.delete_job.assert_called_once_with("job123")
|
||||
|
||||
|
||||
class TestWorkerLogEntryEdgeCases:
|
||||
"""Edge case tests for WorkerLogEntry."""
|
||||
|
||||
def test_create_with_empty_message(self):
|
||||
"""Test creating entry with empty message."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message=""
|
||||
)
|
||||
assert entry.message == ""
|
||||
|
||||
def test_create_with_unicode_message(self):
|
||||
"""Test creating entry with Unicode characters."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Processing song: 日本語の曲 - アーティスト"
|
||||
)
|
||||
assert "日本語" in entry.message
|
||||
|
||||
def test_create_with_newlines_in_message(self):
|
||||
"""Test creating entry with newlines."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="ERROR",
|
||||
message="Error occurred:\nLine 1\nLine 2"
|
||||
)
|
||||
assert "\n" in entry.message
|
||||
|
||||
def test_from_dict_handles_missing_ttl_expiry(self):
|
||||
"""Test from_dict creates default TTL when missing."""
|
||||
data = {
|
||||
"timestamp": datetime.now(timezone.utc),
|
||||
"level": "INFO",
|
||||
"worker": "audio",
|
||||
"message": "Test"
|
||||
}
|
||||
|
||||
entry = WorkerLogEntry.from_dict(data)
|
||||
|
||||
assert entry.ttl_expiry is not None
|
||||
# Should be ~30 days from now
|
||||
expected = datetime.now(timezone.utc) + timedelta(days=30)
|
||||
assert abs((entry.ttl_expiry - expected).total_seconds()) < 10
|
||||
@@ -970,10 +970,147 @@ class TestBackingVocalsAnalysis:
|
||||
def test_analysis_service_can_be_instantiated(self):
|
||||
"""Test that AudioAnalysisService can be instantiated."""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
|
||||
mock_storage = MagicMock()
|
||||
service = AudioAnalysisService(storage_service=mock_storage)
|
||||
|
||||
|
||||
assert service is not None
|
||||
assert service.storage_service == mock_storage
|
||||
|
||||
|
||||
class TestModelNamesStorage:
|
||||
"""Tests for model names storage in audio_worker.
|
||||
|
||||
These tests verify that model names are stored in job state_data
|
||||
for use by video_worker in distribution directory preparation.
|
||||
"""
|
||||
|
||||
def test_effective_model_names_defaults(self):
|
||||
"""Test that default model names are used when not specified on job."""
|
||||
from backend.workers.audio_worker import (
|
||||
DEFAULT_CLEAN_MODEL,
|
||||
DEFAULT_BACKING_MODELS,
|
||||
DEFAULT_OTHER_MODELS,
|
||||
)
|
||||
|
||||
# Simulate the logic from process_audio_separation
|
||||
job_clean_model = None
|
||||
job_backing_models = None
|
||||
job_other_models = None
|
||||
|
||||
effective_model_names = {
|
||||
'clean_instrumental_model': job_clean_model or DEFAULT_CLEAN_MODEL,
|
||||
'backing_vocals_models': job_backing_models or DEFAULT_BACKING_MODELS,
|
||||
'other_stems_models': job_other_models or DEFAULT_OTHER_MODELS,
|
||||
}
|
||||
|
||||
assert effective_model_names['clean_instrumental_model'] == DEFAULT_CLEAN_MODEL
|
||||
assert effective_model_names['backing_vocals_models'] == DEFAULT_BACKING_MODELS
|
||||
assert effective_model_names['other_stems_models'] == DEFAULT_OTHER_MODELS
|
||||
|
||||
def test_effective_model_names_custom(self):
|
||||
"""Test that custom model names override defaults."""
|
||||
from backend.workers.audio_worker import (
|
||||
DEFAULT_CLEAN_MODEL,
|
||||
DEFAULT_BACKING_MODELS,
|
||||
DEFAULT_OTHER_MODELS,
|
||||
)
|
||||
|
||||
custom_clean = "custom_clean_model.ckpt"
|
||||
custom_backing = ["custom_backing.ckpt"]
|
||||
custom_other = ["custom_demucs.yaml"]
|
||||
|
||||
# Simulate the logic from process_audio_separation
|
||||
effective_model_names = {
|
||||
'clean_instrumental_model': custom_clean or DEFAULT_CLEAN_MODEL,
|
||||
'backing_vocals_models': custom_backing or DEFAULT_BACKING_MODELS,
|
||||
'other_stems_models': custom_other or DEFAULT_OTHER_MODELS,
|
||||
}
|
||||
|
||||
assert effective_model_names['clean_instrumental_model'] == custom_clean
|
||||
assert effective_model_names['backing_vocals_models'] == custom_backing
|
||||
assert effective_model_names['other_stems_models'] == custom_other
|
||||
|
||||
|
||||
class TestDistributionDirectoryPreparation:
|
||||
"""Tests for _prepare_distribution_directory in video_worker.
|
||||
|
||||
These tests verify that the distribution directory is prepared with:
|
||||
- stems/ subfolder containing all audio stems with model names
|
||||
- lyrics/ subfolder containing intermediate lyrics files
|
||||
- Properly named instrumentals at root level
|
||||
"""
|
||||
|
||||
def test_distribution_directory_creates_stems_folder(self, tmp_path):
|
||||
"""Test that stems directory is created."""
|
||||
stems_dir = tmp_path / "stems"
|
||||
stems_dir.mkdir()
|
||||
|
||||
assert stems_dir.exists()
|
||||
assert stems_dir.is_dir()
|
||||
|
||||
def test_distribution_directory_creates_lyrics_folder(self, tmp_path):
|
||||
"""Test that lyrics directory is created."""
|
||||
lyrics_dir = tmp_path / "lyrics"
|
||||
lyrics_dir.mkdir()
|
||||
|
||||
assert lyrics_dir.exists()
|
||||
assert lyrics_dir.is_dir()
|
||||
|
||||
def test_instrumental_naming_with_model(self):
|
||||
"""Test that instrumental files are named with model names."""
|
||||
base_name = "Artist - Song"
|
||||
clean_model = "model_bs_roformer_ep_317_sdr_12.9755.ckpt"
|
||||
backing_model = "mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt"
|
||||
|
||||
clean_instrumental_name = f"{base_name} (Instrumental {clean_model}).flac"
|
||||
backing_instrumental_name = f"{base_name} (Instrumental +BV {backing_model}).flac"
|
||||
|
||||
# Verify expected format
|
||||
assert "model_bs_roformer" in clean_instrumental_name
|
||||
assert "+BV" in backing_instrumental_name
|
||||
assert backing_model in backing_instrumental_name
|
||||
|
||||
def test_stem_naming_convention(self):
|
||||
"""Test that stems are named with proper model suffixes."""
|
||||
base_name = "Artist - Song"
|
||||
clean_model = "model_bs_roformer_ep_317_sdr_12.9755.ckpt"
|
||||
backing_model = "mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt"
|
||||
other_model = "htdemucs_6s.yaml"
|
||||
|
||||
# Expected stem filenames
|
||||
expected_stems = {
|
||||
'vocals_clean': f"{base_name} (Vocals {clean_model}).flac",
|
||||
'lead_vocals': f"{base_name} (Lead Vocals {backing_model}).flac",
|
||||
'backing_vocals': f"{base_name} (Backing Vocals {backing_model}).flac",
|
||||
'bass': f"{base_name} (Bass {other_model}).flac",
|
||||
'drums': f"{base_name} (Drums {other_model}).flac",
|
||||
}
|
||||
|
||||
# Verify naming convention
|
||||
for _key, name in expected_stems.items():
|
||||
assert ".flac" in name
|
||||
assert base_name in name
|
||||
|
||||
def test_simplified_instrumental_cleanup(self, tmp_path):
|
||||
"""Test that simplified instrumental names are cleaned up."""
|
||||
base_name = "Artist - Song"
|
||||
|
||||
# Create simplified-named files (as created by _setup_working_directory)
|
||||
simplified_clean = tmp_path / f"{base_name} (Instrumental Clean).flac"
|
||||
simplified_backing = tmp_path / f"{base_name} (Instrumental Backing).flac"
|
||||
simplified_clean.write_text("fake")
|
||||
simplified_backing.write_text("fake")
|
||||
|
||||
# Verify they exist
|
||||
assert simplified_clean.exists()
|
||||
assert simplified_backing.exists()
|
||||
|
||||
# Simulate cleanup
|
||||
simplified_clean.unlink()
|
||||
simplified_backing.unlink()
|
||||
|
||||
# Verify they're removed
|
||||
assert not simplified_clean.exists()
|
||||
assert not simplified_backing.exists()
|
||||
|
||||
@@ -0,0 +1,568 @@
|
||||
"""
|
||||
Tests for YouTubeUploadService.
|
||||
|
||||
Tests cover:
|
||||
- Authentication with pre-stored credentials
|
||||
- Authentication with client secrets file
|
||||
- Duplicate video detection (exact and fuzzy matching)
|
||||
- Video deletion
|
||||
- Video upload with metadata and thumbnail
|
||||
- Title truncation
|
||||
- Dry run mode
|
||||
"""
|
||||
|
||||
import os
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch, mock_open
|
||||
|
||||
from backend.services.youtube_upload_service import YouTubeUploadService, get_youtube_upload_service
|
||||
|
||||
|
||||
class TestYouTubeUploadServiceInit:
|
||||
"""Test service initialization."""
|
||||
|
||||
def test_init_with_credentials(self):
|
||||
"""Test initialization with pre-stored credentials."""
|
||||
creds = {
|
||||
"token": "test_token",
|
||||
"refresh_token": "test_refresh",
|
||||
"token_uri": "https://oauth2.googleapis.com/token",
|
||||
"client_id": "test_client_id",
|
||||
"client_secret": "test_secret",
|
||||
}
|
||||
service = YouTubeUploadService(
|
||||
credentials=creds,
|
||||
non_interactive=True
|
||||
)
|
||||
assert service.credentials == creds
|
||||
assert service.non_interactive is True
|
||||
|
||||
def test_init_with_client_secrets(self):
|
||||
"""Test initialization with client secrets file."""
|
||||
service = YouTubeUploadService(
|
||||
client_secrets_file="/path/to/secrets.json"
|
||||
)
|
||||
assert service.client_secrets_file == "/path/to/secrets.json"
|
||||
assert service.non_interactive is False
|
||||
|
||||
def test_init_default_values(self):
|
||||
"""Test default values on initialization."""
|
||||
service = YouTubeUploadService()
|
||||
assert service.credentials is None
|
||||
assert service.client_secrets_file is None
|
||||
assert service.non_interactive is False
|
||||
assert service.server_side_mode is False
|
||||
assert service.dry_run is False
|
||||
|
||||
|
||||
class TestYouTubeUploadServiceAuthentication:
|
||||
"""Test authentication methods."""
|
||||
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
def test_authenticate_with_prestored_credentials(
|
||||
self, mock_request, mock_credentials_class, mock_build
|
||||
):
|
||||
"""Test authentication using pre-stored credentials."""
|
||||
# Setup mocks
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = False
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
mock_youtube = MagicMock()
|
||||
mock_build.return_value = mock_youtube
|
||||
|
||||
creds = {
|
||||
"token": "test_token",
|
||||
"refresh_token": "test_refresh",
|
||||
"token_uri": "https://oauth2.googleapis.com/token",
|
||||
"client_id": "test_client_id",
|
||||
"client_secret": "test_secret",
|
||||
"scopes": ["https://www.googleapis.com/auth/youtube"],
|
||||
}
|
||||
service = YouTubeUploadService(credentials=creds, non_interactive=True)
|
||||
|
||||
# Access youtube_service to trigger authentication
|
||||
result = service.youtube_service
|
||||
|
||||
assert result == mock_youtube
|
||||
mock_credentials_class.assert_called_once()
|
||||
mock_build.assert_called_once_with('youtube', 'v3', credentials=mock_creds)
|
||||
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
def test_authenticate_refreshes_expired_token(
|
||||
self, mock_request, mock_credentials_class, mock_build
|
||||
):
|
||||
"""Test that expired tokens are refreshed."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = True
|
||||
mock_creds.refresh_token = "test_refresh"
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
|
||||
creds = {
|
||||
"token": "expired_token",
|
||||
"refresh_token": "test_refresh",
|
||||
"token_uri": "https://oauth2.googleapis.com/token",
|
||||
"client_id": "test_client_id",
|
||||
"client_secret": "test_secret",
|
||||
}
|
||||
service = YouTubeUploadService(credentials=creds, non_interactive=True)
|
||||
service.youtube_service
|
||||
|
||||
mock_creds.refresh.assert_called_once()
|
||||
|
||||
def test_authenticate_non_interactive_no_credentials_raises(self):
|
||||
"""Test that non-interactive mode without credentials raises error."""
|
||||
service = YouTubeUploadService(non_interactive=True)
|
||||
|
||||
with pytest.raises(Exception) as exc_info:
|
||||
service._authenticate()
|
||||
|
||||
assert "non-interactive mode" in str(exc_info.value).lower()
|
||||
|
||||
|
||||
class TestYouTubeUploadServiceChannelId:
|
||||
"""Test channel ID retrieval."""
|
||||
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
def test_get_channel_id_success(
|
||||
self, mock_request, mock_credentials_class, mock_build
|
||||
):
|
||||
"""Test successful channel ID retrieval."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = False
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
|
||||
mock_youtube = MagicMock()
|
||||
mock_channels = MagicMock()
|
||||
mock_list = MagicMock()
|
||||
mock_list.execute.return_value = {
|
||||
"items": [{"id": "UC123456"}]
|
||||
}
|
||||
mock_channels.list.return_value = mock_list
|
||||
mock_youtube.channels.return_value = mock_channels
|
||||
mock_build.return_value = mock_youtube
|
||||
|
||||
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
|
||||
"client_id": "id", "client_secret": "secret"}
|
||||
service = YouTubeUploadService(credentials=creds, non_interactive=True)
|
||||
|
||||
channel_id = service.get_channel_id()
|
||||
|
||||
assert channel_id == "UC123456"
|
||||
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
def test_get_channel_id_no_items(
|
||||
self, mock_request, mock_credentials_class, mock_build
|
||||
):
|
||||
"""Test channel ID returns None when no items found."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = False
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
|
||||
mock_youtube = MagicMock()
|
||||
mock_channels = MagicMock()
|
||||
mock_list = MagicMock()
|
||||
mock_list.execute.return_value = {"items": []}
|
||||
mock_channels.list.return_value = mock_list
|
||||
mock_youtube.channels.return_value = mock_channels
|
||||
mock_build.return_value = mock_youtube
|
||||
|
||||
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
|
||||
"client_id": "id", "client_secret": "secret"}
|
||||
service = YouTubeUploadService(credentials=creds, non_interactive=True)
|
||||
|
||||
channel_id = service.get_channel_id()
|
||||
|
||||
assert channel_id is None
|
||||
|
||||
|
||||
class TestYouTubeUploadServiceDuplicateCheck:
|
||||
"""Test duplicate video detection."""
|
||||
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
def test_check_duplicate_exact_match_found(
|
||||
self, mock_request, mock_credentials_class, mock_build
|
||||
):
|
||||
"""Test duplicate detection with exact match in server mode."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = False
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
|
||||
mock_youtube = MagicMock()
|
||||
# Mock channels().list() for get_channel_id
|
||||
mock_channels = MagicMock()
|
||||
mock_channels_list = MagicMock()
|
||||
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
|
||||
mock_channels.list.return_value = mock_channels_list
|
||||
mock_youtube.channels.return_value = mock_channels
|
||||
|
||||
# Mock search().list() for duplicate check
|
||||
mock_search = MagicMock()
|
||||
mock_search_list = MagicMock()
|
||||
mock_search_list.execute.return_value = {
|
||||
"items": [{
|
||||
"id": {"videoId": "VIDEO123"},
|
||||
"snippet": {
|
||||
"channelId": "UC123456",
|
||||
"title": "Test Artist - Test Song (Karaoke)"
|
||||
}
|
||||
}]
|
||||
}
|
||||
mock_search.list.return_value = mock_search_list
|
||||
mock_youtube.search.return_value = mock_search
|
||||
mock_build.return_value = mock_youtube
|
||||
|
||||
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
|
||||
"client_id": "id", "client_secret": "secret"}
|
||||
service = YouTubeUploadService(
|
||||
credentials=creds,
|
||||
non_interactive=True,
|
||||
server_side_mode=True
|
||||
)
|
||||
|
||||
exists, video_id, video_url = service.check_duplicate(
|
||||
"Test Artist - Test Song (Karaoke)"
|
||||
)
|
||||
|
||||
assert exists is True
|
||||
assert video_id == "VIDEO123"
|
||||
assert "VIDEO123" in video_url
|
||||
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
def test_check_duplicate_no_match(
|
||||
self, mock_request, mock_credentials_class, mock_build
|
||||
):
|
||||
"""Test duplicate detection when no match found."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = False
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
|
||||
mock_youtube = MagicMock()
|
||||
mock_channels = MagicMock()
|
||||
mock_channels_list = MagicMock()
|
||||
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
|
||||
mock_channels.list.return_value = mock_channels_list
|
||||
mock_youtube.channels.return_value = mock_channels
|
||||
|
||||
mock_search = MagicMock()
|
||||
mock_search_list = MagicMock()
|
||||
mock_search_list.execute.return_value = {"items": []}
|
||||
mock_search.list.return_value = mock_search_list
|
||||
mock_youtube.search.return_value = mock_search
|
||||
mock_build.return_value = mock_youtube
|
||||
|
||||
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
|
||||
"client_id": "id", "client_secret": "secret"}
|
||||
service = YouTubeUploadService(credentials=creds, non_interactive=True)
|
||||
|
||||
exists, video_id, video_url = service.check_duplicate("Some Title")
|
||||
|
||||
assert exists is False
|
||||
assert video_id is None
|
||||
assert video_url is None
|
||||
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
def test_check_duplicate_skips_other_channels(
|
||||
self, mock_request, mock_credentials_class, mock_build
|
||||
):
|
||||
"""Test that videos from other channels are skipped."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = False
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
|
||||
mock_youtube = MagicMock()
|
||||
mock_channels = MagicMock()
|
||||
mock_channels_list = MagicMock()
|
||||
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
|
||||
mock_channels.list.return_value = mock_channels_list
|
||||
mock_youtube.channels.return_value = mock_channels
|
||||
|
||||
# Return video from different channel
|
||||
mock_search = MagicMock()
|
||||
mock_search_list = MagicMock()
|
||||
mock_search_list.execute.return_value = {
|
||||
"items": [{
|
||||
"id": {"videoId": "VIDEO123"},
|
||||
"snippet": {
|
||||
"channelId": "UC_DIFFERENT", # Different channel
|
||||
"title": "Test Title"
|
||||
}
|
||||
}]
|
||||
}
|
||||
mock_search.list.return_value = mock_search_list
|
||||
mock_youtube.search.return_value = mock_search
|
||||
mock_build.return_value = mock_youtube
|
||||
|
||||
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
|
||||
"client_id": "id", "client_secret": "secret"}
|
||||
service = YouTubeUploadService(credentials=creds, non_interactive=True)
|
||||
|
||||
exists, video_id, video_url = service.check_duplicate("Test Title")
|
||||
|
||||
assert exists is False
|
||||
|
||||
|
||||
class TestYouTubeUploadServiceDelete:
|
||||
"""Test video deletion."""
|
||||
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
def test_delete_video_success(
|
||||
self, mock_request, mock_credentials_class, mock_build
|
||||
):
|
||||
"""Test successful video deletion."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = False
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
|
||||
mock_youtube = MagicMock()
|
||||
mock_videos = MagicMock()
|
||||
mock_delete = MagicMock()
|
||||
mock_delete.execute.return_value = None
|
||||
mock_videos.delete.return_value = mock_delete
|
||||
mock_youtube.videos.return_value = mock_videos
|
||||
mock_build.return_value = mock_youtube
|
||||
|
||||
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
|
||||
"client_id": "id", "client_secret": "secret"}
|
||||
service = YouTubeUploadService(credentials=creds, non_interactive=True)
|
||||
|
||||
result = service.delete_video("VIDEO123")
|
||||
|
||||
assert result is True
|
||||
mock_videos.delete.assert_called_once_with(id="VIDEO123")
|
||||
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
def test_delete_video_failure(
|
||||
self, mock_request, mock_credentials_class, mock_build
|
||||
):
|
||||
"""Test video deletion failure handling."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = False
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
|
||||
mock_youtube = MagicMock()
|
||||
mock_videos = MagicMock()
|
||||
mock_delete = MagicMock()
|
||||
mock_delete.execute.side_effect = Exception("API Error")
|
||||
mock_videos.delete.return_value = mock_delete
|
||||
mock_youtube.videos.return_value = mock_videos
|
||||
mock_build.return_value = mock_youtube
|
||||
|
||||
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
|
||||
"client_id": "id", "client_secret": "secret"}
|
||||
service = YouTubeUploadService(credentials=creds, non_interactive=True)
|
||||
|
||||
result = service.delete_video("VIDEO123")
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_delete_video_dry_run(self):
|
||||
"""Test video deletion in dry run mode."""
|
||||
service = YouTubeUploadService(dry_run=True)
|
||||
|
||||
result = service.delete_video("VIDEO123")
|
||||
|
||||
assert result is True
|
||||
|
||||
|
||||
class TestYouTubeUploadServiceTitleTruncation:
|
||||
"""Test title truncation."""
|
||||
|
||||
def test_truncate_title_short_title(self):
|
||||
"""Test that short titles are not truncated."""
|
||||
title = "Short Title"
|
||||
result = YouTubeUploadService.truncate_title(title)
|
||||
assert result == title
|
||||
|
||||
def test_truncate_title_exact_length(self):
|
||||
"""Test title at exact max length."""
|
||||
title = "A" * 95
|
||||
result = YouTubeUploadService.truncate_title(title, max_length=95)
|
||||
assert result == title
|
||||
assert len(result) == 95
|
||||
|
||||
def test_truncate_title_long_title(self):
|
||||
"""Test that long titles are truncated at word boundary."""
|
||||
title = "This is a very long title that exceeds the maximum length and needs to be truncated properly at a word boundary"
|
||||
result = YouTubeUploadService.truncate_title(title, max_length=50)
|
||||
assert len(result) <= 50
|
||||
assert result.endswith("...")
|
||||
|
||||
def test_truncate_title_no_space(self):
|
||||
"""Test truncation of title without spaces."""
|
||||
title = "A" * 100
|
||||
result = YouTubeUploadService.truncate_title(title, max_length=50)
|
||||
assert len(result) <= 50
|
||||
|
||||
|
||||
class TestYouTubeUploadServiceUpload:
|
||||
"""Test video upload."""
|
||||
|
||||
@patch("googleapiclient.http.MediaFileUpload")
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
def test_upload_video_success(
|
||||
self, mock_request, mock_credentials_class, mock_build, mock_media_upload
|
||||
):
|
||||
"""Test successful video upload."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = False
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
|
||||
mock_youtube = MagicMock()
|
||||
# Mock channels for get_channel_id
|
||||
mock_channels = MagicMock()
|
||||
mock_channels_list = MagicMock()
|
||||
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
|
||||
mock_channels.list.return_value = mock_channels_list
|
||||
mock_youtube.channels.return_value = mock_channels
|
||||
|
||||
# Mock search for duplicate check (no duplicates)
|
||||
mock_search = MagicMock()
|
||||
mock_search_list = MagicMock()
|
||||
mock_search_list.execute.return_value = {"items": []}
|
||||
mock_search.list.return_value = mock_search_list
|
||||
mock_youtube.search.return_value = mock_search
|
||||
|
||||
# Mock video insert
|
||||
mock_videos = MagicMock()
|
||||
mock_insert = MagicMock()
|
||||
mock_insert.execute.return_value = {"id": "NEW_VIDEO_ID"}
|
||||
mock_videos.insert.return_value = mock_insert
|
||||
mock_youtube.videos.return_value = mock_videos
|
||||
mock_build.return_value = mock_youtube
|
||||
|
||||
mock_media_upload.return_value = MagicMock()
|
||||
|
||||
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
|
||||
"client_id": "id", "client_secret": "secret"}
|
||||
service = YouTubeUploadService(credentials=creds, non_interactive=True)
|
||||
|
||||
video_id, video_url = service.upload_video(
|
||||
video_path="/path/to/video.mkv",
|
||||
title="Test Artist - Test Song (Karaoke)",
|
||||
description="Test description",
|
||||
tags=["karaoke", "test"]
|
||||
)
|
||||
|
||||
assert video_id == "NEW_VIDEO_ID"
|
||||
assert "NEW_VIDEO_ID" in video_url
|
||||
|
||||
def test_upload_video_dry_run(self):
|
||||
"""Test video upload in dry run mode."""
|
||||
service = YouTubeUploadService(dry_run=True)
|
||||
|
||||
video_id, video_url = service.upload_video(
|
||||
video_path="/path/to/video.mkv",
|
||||
title="Test Title",
|
||||
description="Test description"
|
||||
)
|
||||
|
||||
assert video_id == "dry_run_video_id"
|
||||
assert "dry_run_video_id" in video_url
|
||||
|
||||
@patch("googleapiclient.http.MediaFileUpload")
|
||||
@patch("googleapiclient.discovery.build")
|
||||
@patch("google.oauth2.credentials.Credentials")
|
||||
@patch("google.auth.transport.requests.Request")
|
||||
@patch("os.path.isfile")
|
||||
def test_upload_video_with_thumbnail(
|
||||
self, mock_isfile, mock_request, mock_credentials_class,
|
||||
mock_build, mock_media_upload
|
||||
):
|
||||
"""Test video upload with thumbnail."""
|
||||
mock_isfile.return_value = True
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.expired = False
|
||||
mock_credentials_class.return_value = mock_creds
|
||||
|
||||
mock_youtube = MagicMock()
|
||||
mock_channels = MagicMock()
|
||||
mock_channels_list = MagicMock()
|
||||
mock_channels_list.execute.return_value = {"items": [{"id": "UC123456"}]}
|
||||
mock_channels.list.return_value = mock_channels_list
|
||||
mock_youtube.channels.return_value = mock_channels
|
||||
|
||||
mock_search = MagicMock()
|
||||
mock_search_list = MagicMock()
|
||||
mock_search_list.execute.return_value = {"items": []}
|
||||
mock_search.list.return_value = mock_search_list
|
||||
mock_youtube.search.return_value = mock_search
|
||||
|
||||
mock_videos = MagicMock()
|
||||
mock_insert = MagicMock()
|
||||
mock_insert.execute.return_value = {"id": "NEW_VIDEO_ID"}
|
||||
mock_videos.insert.return_value = mock_insert
|
||||
mock_youtube.videos.return_value = mock_videos
|
||||
|
||||
# Mock thumbnail upload
|
||||
mock_thumbnails = MagicMock()
|
||||
mock_set = MagicMock()
|
||||
mock_set.execute.return_value = None
|
||||
mock_thumbnails.set.return_value = mock_set
|
||||
mock_youtube.thumbnails.return_value = mock_thumbnails
|
||||
|
||||
mock_build.return_value = mock_youtube
|
||||
mock_media_upload.return_value = MagicMock()
|
||||
|
||||
creds = {"token": "test", "refresh_token": "test", "token_uri": "uri",
|
||||
"client_id": "id", "client_secret": "secret"}
|
||||
service = YouTubeUploadService(credentials=creds, non_interactive=True)
|
||||
|
||||
video_id, video_url = service.upload_video(
|
||||
video_path="/path/to/video.mkv",
|
||||
title="Test Title",
|
||||
description="Test description",
|
||||
thumbnail_path="/path/to/thumbnail.jpg"
|
||||
)
|
||||
|
||||
assert video_id == "NEW_VIDEO_ID"
|
||||
mock_thumbnails.set.assert_called_once()
|
||||
|
||||
|
||||
class TestGetYouTubeUploadService:
|
||||
"""Test factory function."""
|
||||
|
||||
def test_get_service_creates_instance(self):
|
||||
"""Test that factory function creates a new instance."""
|
||||
# Reset global
|
||||
import backend.services.youtube_upload_service as module
|
||||
module._youtube_upload_service = None
|
||||
|
||||
service = get_youtube_upload_service(
|
||||
credentials={"token": "test"},
|
||||
non_interactive=True
|
||||
)
|
||||
|
||||
assert service is not None
|
||||
assert isinstance(service, YouTubeUploadService)
|
||||
|
||||
def test_get_service_with_client_secrets(self):
|
||||
"""Test factory function with client secrets file."""
|
||||
import backend.services.youtube_upload_service as module
|
||||
module._youtube_upload_service = None
|
||||
|
||||
service = get_youtube_upload_service(
|
||||
client_secrets_file="/path/to/secrets.json"
|
||||
)
|
||||
|
||||
assert service is not None
|
||||
assert service.client_secrets_file == "/path/to/secrets.json"
|
||||
@@ -0,0 +1,27 @@
|
||||
"""
|
||||
Utilities for identifying test data.
|
||||
|
||||
Test data is generated by automated E2E tests and should be filterable
|
||||
in admin dashboards to show only real user data by default.
|
||||
"""
|
||||
|
||||
# Email domains used by automated testing frameworks
|
||||
TEST_EMAIL_DOMAINS = [
|
||||
"inbox.testmail.app", # Used by E2E happy path tests
|
||||
]
|
||||
|
||||
|
||||
def is_test_email(email: str) -> bool:
|
||||
"""
|
||||
Check if an email address belongs to automated test data.
|
||||
|
||||
Args:
|
||||
email: Email address to check
|
||||
|
||||
Returns:
|
||||
True if the email matches a test email domain pattern
|
||||
"""
|
||||
if not email:
|
||||
return False
|
||||
email_lower = email.lower()
|
||||
return any(email_lower.endswith(f"@{domain}") for domain in TEST_EMAIL_DOMAINS)
|
||||
@@ -36,6 +36,35 @@ from karaoke_gen.audio_processor import AudioProcessor
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def _trigger_lyrics_worker_after_url_download(job_id: str) -> None:
|
||||
"""
|
||||
Trigger lyrics worker after URL audio download completes.
|
||||
|
||||
For URL jobs, we use sequential triggering:
|
||||
1. Audio worker downloads and uploads audio to GCS
|
||||
2. Audio worker triggers lyrics worker (this function)
|
||||
3. Both workers then proceed in parallel (audio separation + lyrics transcription)
|
||||
|
||||
This prevents the race condition where lyrics worker times out waiting for audio.
|
||||
"""
|
||||
from backend.services.worker_service import get_worker_service
|
||||
|
||||
try:
|
||||
worker_service = get_worker_service()
|
||||
await worker_service.trigger_lyrics_worker(job_id)
|
||||
logger.info(f"Job {job_id}: Triggered lyrics worker after URL download")
|
||||
except Exception as e:
|
||||
# Log but don't fail - audio processing can still continue
|
||||
# The job will eventually timeout if lyrics worker doesn't run
|
||||
logger.error(f"Job {job_id}: Failed to trigger lyrics worker: {e}")
|
||||
|
||||
|
||||
# Default model names - used by create_audio_processor and stored in state_data
|
||||
DEFAULT_CLEAN_MODEL = "model_bs_roformer_ep_317_sdr_12.9755.ckpt"
|
||||
DEFAULT_BACKING_MODELS = ["mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt"]
|
||||
DEFAULT_OTHER_MODELS = ["htdemucs_6s.yaml"]
|
||||
|
||||
|
||||
# Loggers to capture for audio worker
|
||||
AUDIO_WORKER_LOGGERS = [
|
||||
"karaoke_gen.audio_processor",
|
||||
@@ -181,10 +210,10 @@ def create_audio_processor(
|
||||
audio_logger = logging.getLogger("karaoke_gen.audio_processor")
|
||||
audio_logger.setLevel(logging.INFO)
|
||||
|
||||
# Model configurations - use provided values or CLI defaults
|
||||
effective_clean_model = clean_instrumental_model or "model_bs_roformer_ep_317_sdr_12.9755.ckpt"
|
||||
effective_backing_models = backing_vocals_models or ["mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt"]
|
||||
effective_other_models = other_stems_models or ["htdemucs_6s.yaml"] # For 6-stem separation (bass, drums, etc.)
|
||||
# Model configurations - use provided values or defaults from module constants
|
||||
effective_clean_model = clean_instrumental_model or DEFAULT_CLEAN_MODEL
|
||||
effective_backing_models = backing_vocals_models or DEFAULT_BACKING_MODELS
|
||||
effective_other_models = other_stems_models or DEFAULT_OTHER_MODELS # For 6-stem separation (bass, drums, etc.)
|
||||
|
||||
# FFmpeg command for combining audio files (must be a string, not a list)
|
||||
ffmpeg_base_command = "ffmpeg -hide_banner -loglevel error -nostats -y"
|
||||
@@ -303,9 +332,24 @@ async def process_audio_separation(job_id: str) -> bool:
|
||||
backing_vocals_models=job.backing_vocals_models,
|
||||
other_stems_models=job.other_stems_models
|
||||
)
|
||||
|
||||
|
||||
# Store effective model names in state_data for video_worker to use in file naming
|
||||
# This ensures output filenames match local CLI behavior (e.g., "Instrumental model_bs_roformer_ep_317_sdr_12.9755.ckpt")
|
||||
effective_model_names = {
|
||||
'clean_instrumental_model': job.clean_instrumental_model or DEFAULT_CLEAN_MODEL,
|
||||
'backing_vocals_models': job.backing_vocals_models or DEFAULT_BACKING_MODELS,
|
||||
'other_stems_models': job.other_stems_models or DEFAULT_OTHER_MODELS,
|
||||
}
|
||||
job_manager.update_state_data(job_id, 'model_names', effective_model_names)
|
||||
job_log.info(f"Stored effective model names: clean={effective_model_names['clean_instrumental_model']}")
|
||||
|
||||
# Format artist-title for file naming (matches CLI behavior)
|
||||
artist_title = f"{job.artist} - {job.title}"
|
||||
# Sanitize to handle Unicode characters (curly quotes, em dashes, etc.)
|
||||
# that cause HTTP header encoding issues with the remote API
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
safe_artist = sanitize_filename(job.artist) if job.artist else "Unknown"
|
||||
safe_title = sanitize_filename(job.title) if job.title else "Unknown"
|
||||
artist_title = f"{safe_artist} - {safe_title}"
|
||||
|
||||
# Run audio separation (calls Modal API internally)
|
||||
# This returns a dict with paths to all separated stems
|
||||
@@ -429,29 +473,34 @@ async def download_audio(
|
||||
# Case 3: Fresh URL that needs downloading
|
||||
if job.url:
|
||||
logger.info(f"Job {job_id}: Downloading from URL: {job.url}")
|
||||
|
||||
|
||||
# Use provided job_manager or create new one
|
||||
jm = job_manager_instance or JobManager()
|
||||
|
||||
|
||||
local_path = await download_from_url(
|
||||
job.url,
|
||||
temp_dir,
|
||||
job.artist,
|
||||
job.url,
|
||||
temp_dir,
|
||||
job.artist,
|
||||
job.title,
|
||||
job_manager=jm,
|
||||
job_id=job_id
|
||||
)
|
||||
|
||||
|
||||
if local_path and os.path.exists(local_path):
|
||||
# Upload to GCS and update job
|
||||
gcs_path = f"jobs/{job_id}/input/{os.path.basename(local_path)}"
|
||||
url = storage.upload_file(local_path, gcs_path)
|
||||
|
||||
|
||||
# Update job with GCS path for lyrics worker
|
||||
jm.update_job(job_id, {'input_media_gcs_path': gcs_path})
|
||||
jm.update_file_url(job_id, 'input', 'audio', url)
|
||||
|
||||
|
||||
logger.info(f"Job {job_id}: Downloaded and uploaded audio to GCS: {gcs_path}")
|
||||
|
||||
# For URL jobs, trigger lyrics worker now that audio is available
|
||||
# This is the sequential trigger pattern - audio first, then lyrics
|
||||
await _trigger_lyrics_worker_after_url_download(job_id)
|
||||
|
||||
return local_path
|
||||
else:
|
||||
logger.error(f"Job {job_id}: Failed to download from URL: {job.url}")
|
||||
|
||||
@@ -32,6 +32,7 @@ from backend.models.job import JobStatus
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.lyrics_cache_service import LyricsCacheService
|
||||
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
|
||||
from backend.workers.style_helper import load_style_config
|
||||
from backend.services.tracing import job_span, add_span_event, add_span_attribute
|
||||
@@ -39,10 +40,43 @@ from backend.services.metrics import metrics
|
||||
|
||||
# Import from karaoke_gen package
|
||||
from karaoke_gen.lyrics_processor import LyricsProcessor
|
||||
from backend.config import get_settings
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Timeout for entire transcription process including agentic AI correction (20 minutes)
|
||||
# This needs to account for:
|
||||
# - Cloud Run cold start / worker initialization (1-5 min)
|
||||
# - AudioShake transcription (1-2 min)
|
||||
# - spaCy model loading for correction (2-3 min on cold start)
|
||||
# - Agentic AI correction (1-3 min)
|
||||
TRANSCRIPTION_TIMEOUT_SECONDS = 1200
|
||||
|
||||
# Default agentic correction timeout (3 minutes)
|
||||
# Configurable via AGENTIC_CORRECTION_TIMEOUT_SECONDS environment variable
|
||||
# If agentic AI takes longer than this, abort and use uncorrected transcription
|
||||
# Human review will correct any issues
|
||||
DEFAULT_AGENTIC_TIMEOUT_SECONDS = 180
|
||||
|
||||
|
||||
def _configure_agentic_ai():
|
||||
"""Configure environment variables for agentic AI correction.
|
||||
|
||||
The lyrics_transcriber library reads these directly from environment.
|
||||
This ensures the settings from backend config are available.
|
||||
"""
|
||||
settings = get_settings()
|
||||
|
||||
# Enable agentic AI if configured
|
||||
if settings.use_agentic_ai:
|
||||
os.environ["USE_AGENTIC_AI"] = "1"
|
||||
os.environ["AGENTIC_AI_MODEL"] = settings.agentic_ai_model
|
||||
logger.info(f"Agentic AI enabled with model: {settings.agentic_ai_model}")
|
||||
else:
|
||||
os.environ["USE_AGENTIC_AI"] = "0"
|
||||
logger.info("Agentic AI disabled")
|
||||
|
||||
|
||||
# Loggers to capture for lyrics worker (top-level package loggers only)
|
||||
# Note: We only capture at top-level package loggers to avoid duplicate log entries.
|
||||
@@ -119,7 +153,10 @@ async def process_lyrics_transcription(job_id: str) -> bool:
|
||||
start_time = time.time()
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
|
||||
# Configure agentic AI before any lyrics processing
|
||||
_configure_agentic_ai()
|
||||
|
||||
# Create job logger for remote debugging FIRST
|
||||
job_log = create_job_logger(job_id, "lyrics")
|
||||
|
||||
@@ -193,7 +230,40 @@ async def process_lyrics_transcription(job_id: str) -> bool:
|
||||
raise Exception("Failed to download audio file")
|
||||
job_log.info(f"Audio downloaded: {os.path.basename(audio_path)}")
|
||||
download_span.set_attribute("audio_file", os.path.basename(audio_path))
|
||||
|
||||
|
||||
# Set up LyricsTranscriber cache directory and sync from GCS
|
||||
# This allows reusing cached AudioShake/lyrics API responses across Cloud Run instances
|
||||
cache_dir = os.path.join(temp_dir, "lyrics-cache")
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
os.environ["LYRICS_TRANSCRIBER_CACHE_DIR"] = cache_dir
|
||||
job_log.info(f"LyricsTranscriber cache dir: {cache_dir}")
|
||||
|
||||
# Sync cache from GCS (download any existing cached API responses)
|
||||
# Note: Cache sync is best-effort - failures should not fail the job
|
||||
cache_service = LyricsCacheService(storage)
|
||||
audio_hash = None
|
||||
lyrics_hash = None
|
||||
with job_span("sync-cache-from-gcs", job_id):
|
||||
try:
|
||||
# Compute cache keys
|
||||
audio_hash = cache_service.compute_audio_hash(audio_path)
|
||||
# Use lyrics_artist/lyrics_title for lyrics hash (these are what LyricsTranscriber uses)
|
||||
lyrics_search_artist = getattr(job, 'lyrics_artist', None) or job.artist
|
||||
lyrics_search_title = getattr(job, 'lyrics_title', None) or job.title
|
||||
lyrics_hash = cache_service.compute_lyrics_hash(lyrics_search_artist, lyrics_search_title)
|
||||
|
||||
job_log.info(f"Cache keys: audio_hash={audio_hash[:8]}..., lyrics_hash={lyrics_hash[:8]}...")
|
||||
add_span_attribute("audio_hash", audio_hash[:8])
|
||||
add_span_attribute("lyrics_hash", lyrics_hash[:8])
|
||||
|
||||
# Download relevant cache files from GCS
|
||||
cache_stats = cache_service.sync_cache_from_gcs(cache_dir, audio_hash, lyrics_hash)
|
||||
job_log.info(f"Cache sync from GCS: {cache_stats['downloaded']} downloaded, {cache_stats['not_found']} not found")
|
||||
add_span_attribute("cache_hits", cache_stats["downloaded"])
|
||||
except Exception as e:
|
||||
job_log.warning(f"Cache sync from GCS failed (non-fatal): {e}", exc_info=True)
|
||||
add_span_attribute("cache_sync_error", str(e)[:100])
|
||||
|
||||
# Update progress using state_data (don't change status during parallel processing)
|
||||
# The status is managed at a higher level - workers just track their progress
|
||||
job_manager.update_state_data(job_id, 'lyrics_progress', {
|
||||
@@ -263,26 +333,73 @@ async def process_lyrics_transcription(job_id: str) -> bool:
|
||||
if lyrics_search_title != job.title:
|
||||
job_log.info(f" Lyrics search title override: {lyrics_search_title}")
|
||||
logger.info(f"[job:{job_id}] Calling lyrics_processor.transcribe_lyrics()")
|
||||
|
||||
# Run transcription + correction
|
||||
|
||||
# Run transcription + correction with timeout
|
||||
# AudioShake typically takes 1-2 minutes, but can hang.
|
||||
with job_span("audioshake-transcription", job_id) as trans_span:
|
||||
trans_start = time.time()
|
||||
add_span_event("transcription_started")
|
||||
|
||||
# Run transcription in thread pool to avoid blocking the event loop
|
||||
# (transcribe_lyrics is synchronous and takes 1-2 minutes)
|
||||
# Note: transcribe_lyrics internally calls AudioShake and lyrics APIs
|
||||
with metrics.time_external_api("audioshake", job_id):
|
||||
result = await asyncio.to_thread(
|
||||
lyrics_processor.transcribe_lyrics,
|
||||
input_audio_wav=audio_path,
|
||||
artist=job.artist, # Original artist for file naming
|
||||
title=job.title, # Original title for file naming
|
||||
track_output_dir=temp_dir,
|
||||
lyrics_artist=lyrics_search_artist, # Override for lyrics search
|
||||
lyrics_title=lyrics_search_title # Override for lyrics search
|
||||
|
||||
# Calculate deadline for agentic correction
|
||||
# If agentic AI is enabled, the correction loop will check this deadline
|
||||
# and abort early if exceeded, returning uncorrected transcription for human review
|
||||
settings = get_settings()
|
||||
agentic_deadline = None
|
||||
timeout_seconds = settings.agentic_correction_timeout_seconds
|
||||
if settings.use_agentic_ai:
|
||||
agentic_deadline = time.time() + timeout_seconds
|
||||
job_log.info(
|
||||
f"Agentic AI enabled with {timeout_seconds}s timeout"
|
||||
)
|
||||
|
||||
|
||||
# Run transcription in thread pool
|
||||
# When agentic AI is enabled, wrap with outer timeout as safety net
|
||||
# Inner deadline check in corrector.py will break out of gap loop gracefully
|
||||
# Outer timeout catches case where single LLM call hangs for too long
|
||||
transcription_coro = asyncio.to_thread(
|
||||
lyrics_processor.transcribe_lyrics,
|
||||
input_audio_wav=audio_path,
|
||||
artist=job.artist, # Original artist for file naming
|
||||
title=job.title, # Original title for file naming
|
||||
track_output_dir=temp_dir,
|
||||
lyrics_artist=lyrics_search_artist, # Override for lyrics search
|
||||
lyrics_title=lyrics_search_title, # Override for lyrics search
|
||||
agentic_deadline=agentic_deadline, # Deadline for agentic timeout
|
||||
)
|
||||
|
||||
with metrics.time_external_api("audioshake", job_id):
|
||||
if settings.use_agentic_ai:
|
||||
# Outer timeout must be generous to allow for AudioShake time
|
||||
# (can take 2-5+ minutes for long songs) PLUS agentic correction.
|
||||
# The inner deadline check in corrector.py handles the 3-minute
|
||||
# correction limit gracefully by returning partial results.
|
||||
# This outer timeout is a safety net for completely hung LLM calls.
|
||||
outer_timeout = TRANSCRIPTION_TIMEOUT_SECONDS # Same as non-agentic
|
||||
try:
|
||||
result = await asyncio.wait_for(
|
||||
transcription_coro,
|
||||
timeout=outer_timeout
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
# This should rarely trigger since inner deadline check breaks first
|
||||
# But provides hard stop if e.g. single LLM call hangs completely
|
||||
job_log.error(
|
||||
f"HARD TIMEOUT: Transcription exceeded {outer_timeout}s. "
|
||||
"This indicates the inner deadline check failed to trigger."
|
||||
)
|
||||
raise RuntimeError(
|
||||
f"Lyrics transcription timed out after {outer_timeout}s"
|
||||
) from None
|
||||
else:
|
||||
# Non-agentic mode: use general timeout (10 minutes)
|
||||
try:
|
||||
result = await asyncio.wait_for(
|
||||
transcription_coro,
|
||||
timeout=TRANSCRIPTION_TIMEOUT_SECONDS
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
raise Exception(f"Transcription timed out after {TRANSCRIPTION_TIMEOUT_SECONDS} seconds")
|
||||
|
||||
trans_duration = time.time() - trans_start
|
||||
trans_span.set_attribute("duration_seconds", trans_duration)
|
||||
add_span_event("transcription_completed", {"duration_seconds": trans_duration})
|
||||
@@ -297,7 +414,23 @@ async def process_lyrics_transcription(job_id: str) -> bool:
|
||||
|
||||
job_log.info("All lyrics data uploaded successfully")
|
||||
logger.info(f"[job:{job_id}] All lyrics data uploaded successfully")
|
||||
|
||||
|
||||
# Sync new cache files to GCS (upload any newly created cache entries)
|
||||
# This persists cache across Cloud Run instances for future jobs
|
||||
# Note: Cache upload is best-effort - failures should not fail the job
|
||||
with job_span("sync-cache-to-gcs", job_id):
|
||||
try:
|
||||
if audio_hash and lyrics_hash:
|
||||
upload_stats = cache_service.sync_cache_to_gcs(cache_dir, audio_hash, lyrics_hash)
|
||||
job_log.info(f"Cache sync to GCS: {upload_stats['uploaded']} uploaded, {upload_stats['skipped']} already existed")
|
||||
add_span_attribute("cache_uploads", upload_stats["uploaded"])
|
||||
else:
|
||||
job_log.warning("Skipping cache upload: missing audio_hash or lyrics_hash")
|
||||
add_span_attribute("cache_upload_skipped", "missing_hashes")
|
||||
except Exception as e:
|
||||
job_log.warning(f"Cache upload to GCS failed (non-fatal): {e}")
|
||||
add_span_attribute("cache_upload_error", str(e)[:100])
|
||||
|
||||
# Update progress using state_data (don't change status during parallel processing)
|
||||
job_manager.update_state_data(job_id, 'lyrics_progress', {
|
||||
'stage': 'lyrics_complete',
|
||||
@@ -506,40 +639,45 @@ async def upload_lyrics_results(
|
||||
logger.error(f"Job {job_id}: {error_msg}")
|
||||
raise Exception(error_msg)
|
||||
|
||||
# Upload reference lyrics if available
|
||||
# Upload ALL reference lyrics files (not just first) so they're available for distribution
|
||||
# Note: Source names use .title() so "lrclib" -> "Lrclib", "genius" -> "Genius"
|
||||
# Use sanitized artist/title to match LyricsProcessor's file naming
|
||||
reference_files = [
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Genius).txt", "Genius"),
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Spotify).txt", "Spotify"),
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Musixmatch).txt", "Musixmatch"),
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Lrclib).txt", "LRCLib"), # Note: source is 'lrclib', .title() = 'Lrclib'
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Genius).txt", "genius"),
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Spotify).txt", "spotify"),
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Musixmatch).txt", "musixmatch"),
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Lrclib).txt", "lrclib"),
|
||||
]
|
||||
|
||||
found_reference = False
|
||||
for ref_filename, source_name in reference_files:
|
||||
|
||||
found_references = []
|
||||
for ref_filename, source_key in reference_files:
|
||||
ref_path = os.path.join(lyrics_dir, ref_filename)
|
||||
if os.path.exists(ref_path):
|
||||
# Upload with original filename to preserve proper naming
|
||||
gcs_path = f"jobs/{job_id}/lyrics/{ref_filename}"
|
||||
url = storage.upload_file(ref_path, gcs_path)
|
||||
# Track in job.files so video_worker can download for distribution
|
||||
job_manager.update_file_url(job_id, 'lyrics', f'reference_{source_key}', url)
|
||||
if job_log:
|
||||
job_log.info(f"Found reference lyrics from {source_name}")
|
||||
job_log.info(f"Found reference lyrics from {source_key}")
|
||||
logger.info(f"Job {job_id}: Uploaded reference lyrics: {ref_filename}")
|
||||
found_reference = True
|
||||
break # Only upload first available reference
|
||||
|
||||
if not found_reference:
|
||||
found_references.append(source_key)
|
||||
|
||||
if not found_references:
|
||||
if job_log:
|
||||
job_log.warning("No reference lyrics found from any source (Genius, Spotify, Musixmatch, LRCLib)")
|
||||
logger.warning(f"Job {job_id}: No reference lyrics found from any source")
|
||||
else:
|
||||
logger.info(f"Job {job_id}: Found {len(found_references)} reference lyrics sources: {found_references}")
|
||||
|
||||
# Upload uncorrected transcription if available (use sanitized names)
|
||||
uncorrected_file = os.path.join(lyrics_dir, f"{safe_artist} - {safe_title} (Lyrics Uncorrected).txt")
|
||||
# Upload uncorrected transcription if available (preserve original filename for distribution)
|
||||
uncorrected_filename = f"{safe_artist} - {safe_title} (Lyrics Uncorrected).txt"
|
||||
uncorrected_file = os.path.join(lyrics_dir, uncorrected_filename)
|
||||
if os.path.exists(uncorrected_file):
|
||||
gcs_path = f"jobs/{job_id}/lyrics/uncorrected.txt"
|
||||
gcs_path = f"jobs/{job_id}/lyrics/{uncorrected_filename}"
|
||||
url = storage.upload_file(uncorrected_file, gcs_path)
|
||||
job_manager.update_file_url(job_id, 'lyrics', 'uncorrected', url)
|
||||
logger.info(f"Job {job_id}: Uploaded uncorrected transcription")
|
||||
logger.info(f"Job {job_id}: Uploaded uncorrected transcription: {uncorrected_filename}")
|
||||
|
||||
logger.info(f"Job {job_id}: All lyrics results uploaded successfully")
|
||||
|
||||
@@ -43,6 +43,7 @@ from lyrics_transcriber.core.config import OutputConfig
|
||||
|
||||
# Import from the unified style loader
|
||||
from karaoke_gen.style_loader import load_styles_from_gcs
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -242,7 +243,10 @@ async def process_render_video(job_id: str) -> bool:
|
||||
output_generator = OutputGenerator(config, logger)
|
||||
|
||||
# 8. Generate outputs (video, LRC, ASS, etc.)
|
||||
output_prefix = f"{job.artist or 'Unknown'} - {job.title or 'Unknown'}"
|
||||
# Sanitize artist/title to handle Unicode characters (curly quotes, em dashes, etc.)
|
||||
safe_artist = sanitize_filename(job.artist) if job.artist else "Unknown"
|
||||
safe_title = sanitize_filename(job.title) if job.title else "Unknown"
|
||||
output_prefix = f"{safe_artist} - {safe_title}"
|
||||
job_log.info(f"Generating outputs with prefix '{output_prefix}'")
|
||||
logger.info(f"Job {job_id}: Generating outputs with prefix '{output_prefix}'")
|
||||
|
||||
|
||||
@@ -39,6 +39,7 @@ from backend.services.tracing import job_span, add_span_event, add_span_attribut
|
||||
|
||||
# Import from karaoke_gen package
|
||||
from karaoke_gen.video_generator import VideoGenerator
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -217,26 +218,36 @@ async def generate_screens(job_id: str) -> bool:
|
||||
def _validate_prerequisites(job) -> bool:
|
||||
"""
|
||||
Validate that both audio and lyrics processing are complete.
|
||||
|
||||
|
||||
Single Responsibility: Validation logic separated from main flow.
|
||||
|
||||
|
||||
Args:
|
||||
job: Job object
|
||||
|
||||
|
||||
Returns:
|
||||
True if prerequisites met, False otherwise
|
||||
"""
|
||||
# SAFETY NET: Enforce theme requirement at processing time
|
||||
# This catches any jobs that somehow bypassed JobManager.create_job() validation
|
||||
if not job.theme_id:
|
||||
logger.error(
|
||||
f"Job {job.job_id}: CRITICAL - No theme_id configured. "
|
||||
"All jobs must have a theme to generate styled videos. "
|
||||
"This job should have been rejected at creation time."
|
||||
)
|
||||
return False
|
||||
|
||||
audio_complete = job.state_data.get('audio_complete', False)
|
||||
lyrics_complete = job.state_data.get('lyrics_complete', False)
|
||||
|
||||
|
||||
if not audio_complete:
|
||||
logger.error(f"Job {job.job_id}: Audio processing not complete")
|
||||
return False
|
||||
|
||||
|
||||
if not lyrics_complete:
|
||||
logger.error(f"Job {job.job_id}: Lyrics processing not complete")
|
||||
return False
|
||||
|
||||
|
||||
if not job.artist or not job.title:
|
||||
logger.error(f"Job {job.job_id}: Missing artist or title")
|
||||
return False
|
||||
@@ -294,9 +305,12 @@ async def _generate_title_screen(
|
||||
"""
|
||||
try:
|
||||
logger.info(f"Job {job_id}: Generating title screen")
|
||||
|
||||
|
||||
# Set up output paths
|
||||
artist_title = f"{job.artist} - {job.title}"
|
||||
# Sanitize artist/title to handle Unicode characters (curly quotes, em dashes, etc.)
|
||||
safe_artist = sanitize_filename(job.artist) if job.artist else "Unknown"
|
||||
safe_title = sanitize_filename(job.title) if job.title else "Unknown"
|
||||
artist_title = f"{safe_artist} - {safe_title}"
|
||||
output_image_filepath_noext = os.path.join(temp_dir, f"{artist_title} (Title)")
|
||||
output_video_filepath = os.path.join(temp_dir, f"{artist_title} (Title).mov")
|
||||
|
||||
@@ -372,9 +386,12 @@ async def _generate_end_screen(
|
||||
"""
|
||||
try:
|
||||
logger.info(f"Job {job_id}: Generating end screen")
|
||||
|
||||
|
||||
# Set up output paths
|
||||
artist_title = f"{job.artist} - {job.title}"
|
||||
# Sanitize artist/title to handle Unicode characters (curly quotes, em dashes, etc.)
|
||||
safe_artist = sanitize_filename(job.artist) if job.artist else "Unknown"
|
||||
safe_title = sanitize_filename(job.title) if job.title else "Unknown"
|
||||
artist_title = f"{safe_artist} - {safe_title}"
|
||||
output_image_filepath_noext = os.path.join(temp_dir, f"{artist_title} (End)")
|
||||
output_video_filepath = os.path.join(temp_dir, f"{artist_title} (End).mov")
|
||||
|
||||
|
||||
+693
-111
@@ -1,20 +1,21 @@
|
||||
"""
|
||||
Video generation and finalization worker.
|
||||
|
||||
This worker uses KaraokeFinalise from the karaoke_gen package to handle:
|
||||
1. Video encoding (4 formats)
|
||||
2. CDG/TXT package generation
|
||||
3. Discord notifications
|
||||
4. Brand code generation
|
||||
5. All other finalisation features
|
||||
This worker has two code paths controlled by USE_NEW_ORCHESTRATOR:
|
||||
|
||||
The worker's job is simply to:
|
||||
1. Download files from GCS and set up the directory structure KaraokeFinalise expects
|
||||
2. Instantiate KaraokeFinalise with parameters from the job
|
||||
3. Call process()
|
||||
4. Upload results back to GCS
|
||||
1. NEW PATH (USE_NEW_ORCHESTRATOR=true):
|
||||
Uses VideoWorkerOrchestrator which provides a unified pipeline for all
|
||||
encoding backends (GCE or local). This ensures features like YouTube upload,
|
||||
Discord notifications, and CDG/TXT packaging work regardless of encoding backend.
|
||||
|
||||
This reuses 100% of the existing karaoke_gen functionality.
|
||||
2. LEGACY PATH (USE_NEW_ORCHESTRATOR=false):
|
||||
Uses KaraokeFinalise from the karaoke_gen package. This path has divergent
|
||||
behavior between GCE and local encoding - GCE encoding bypasses many features.
|
||||
|
||||
The orchestrator-based approach is preferred as it:
|
||||
- Eliminates GCE vs local code path divergence
|
||||
- Ensures all features work consistently
|
||||
- Provides better testability and maintainability
|
||||
|
||||
Observability:
|
||||
- All operations wrapped in tracing spans for Cloud Trace visibility
|
||||
@@ -35,6 +36,7 @@ from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.rclone_service import get_rclone_service
|
||||
from backend.services.youtube_service import get_youtube_service
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
from backend.config import get_settings
|
||||
from backend.workers.style_helper import load_style_config
|
||||
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
|
||||
@@ -42,11 +44,18 @@ from backend.services.tracing import job_span, add_span_event, add_span_attribut
|
||||
|
||||
# Import from karaoke_gen package - reuse existing implementation
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Feature flag for new orchestrator-based pipeline
|
||||
# Set to True to use the new unified pipeline that works with any encoding backend
|
||||
# Set to False to use the legacy KaraokeFinalise-based pipeline
|
||||
USE_NEW_ORCHESTRATOR = os.environ.get("USE_NEW_ORCHESTRATOR", "true").lower() == "true"
|
||||
|
||||
|
||||
# Loggers to capture for video worker
|
||||
# Include the full module path to properly capture KaraokeFinalise logs
|
||||
VIDEO_WORKER_LOGGERS = [
|
||||
@@ -55,16 +64,370 @@ VIDEO_WORKER_LOGGERS = [
|
||||
]
|
||||
|
||||
|
||||
async def generate_video(job_id: str) -> bool:
|
||||
async def _encode_via_gce(
|
||||
job_id: str,
|
||||
job,
|
||||
job_manager: JobManager,
|
||||
storage: StorageService,
|
||||
temp_dir: str,
|
||||
base_name: str,
|
||||
job_log,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Generate final karaoke videos using KaraokeFinalise.
|
||||
|
||||
This sets up the directory structure and calls KaraokeFinalise.process()
|
||||
which handles all encoding, packaging, and notifications.
|
||||
|
||||
Encode videos using the high-performance GCE encoding worker.
|
||||
|
||||
This offloads FFmpeg encoding to a dedicated C4 GCE instance with
|
||||
Intel Granite Rapids 3.9 GHz CPU for 2-3x faster encoding.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
job: Job object
|
||||
job_manager: Job manager instance
|
||||
storage: Storage service instance
|
||||
temp_dir: Temporary directory for output files
|
||||
base_name: Base name for output files (e.g., "Artist - Title")
|
||||
job_log: Job-specific logger
|
||||
|
||||
Returns:
|
||||
Result dict with paths to encoded files
|
||||
"""
|
||||
settings = get_settings()
|
||||
encoding_service = get_encoding_service()
|
||||
|
||||
job_log.info("Using GCE encoding worker for high-performance encoding")
|
||||
logger.info(f"[job:{job_id}] Using GCE encoding worker")
|
||||
|
||||
# Construct GCS paths for the encoding worker
|
||||
bucket_name = settings.gcs_bucket_name
|
||||
input_gcs_path = f"gs://{bucket_name}/jobs/{job_id}/"
|
||||
output_gcs_path = f"gs://{bucket_name}/jobs/{job_id}/encoded/"
|
||||
|
||||
# Determine instrumental selection for encoding config
|
||||
instrumental_selection = job.state_data.get('instrumental_selection', 'clean')
|
||||
existing_instrumental = getattr(job, 'existing_instrumental_gcs_path', None)
|
||||
|
||||
encoding_config = {
|
||||
"formats": ["mp4_4k_lossless", "mp4_4k_lossy", "mp4_720p"],
|
||||
"base_name": base_name,
|
||||
"instrumental_selection": instrumental_selection,
|
||||
"existing_instrumental": existing_instrumental,
|
||||
"ffmpeg_threads": 8, # c4-standard-8 has 8 vCPUs
|
||||
}
|
||||
|
||||
job_log.info(f"Submitting encoding job to GCE worker")
|
||||
job_log.info(f" Input: {input_gcs_path}")
|
||||
job_log.info(f" Output: {output_gcs_path}")
|
||||
|
||||
def progress_callback(progress: int):
|
||||
# Update job progress (encoding is 75-95% of total)
|
||||
scaled_progress = 75 + int(progress * 0.2) # Map 0-100 to 75-95
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.ENCODING,
|
||||
progress=scaled_progress,
|
||||
message=f"Encoding videos ({progress}%)"
|
||||
)
|
||||
|
||||
try:
|
||||
# Submit and wait for encoding
|
||||
with job_span("gce-encoding", job_id) as encoding_span:
|
||||
encode_start = time.time()
|
||||
add_span_event("gce_encoding_started")
|
||||
|
||||
result = await encoding_service.encode_videos(
|
||||
job_id=job_id,
|
||||
input_gcs_path=input_gcs_path,
|
||||
output_gcs_path=output_gcs_path,
|
||||
encoding_config=encoding_config,
|
||||
progress_callback=progress_callback,
|
||||
)
|
||||
|
||||
encode_duration = time.time() - encode_start
|
||||
encoding_span.set_attribute("duration_seconds", encode_duration)
|
||||
add_span_event("gce_encoding_completed", {"duration_seconds": encode_duration})
|
||||
|
||||
job_log.info(f"GCE encoding complete in {encode_duration:.1f}s")
|
||||
logger.info(f"[job:{job_id}] GCE encoding complete in {encode_duration:.1f}s")
|
||||
|
||||
# Download encoded files from GCS to temp_dir
|
||||
output_files = result.get("output_files", [])
|
||||
local_files = {}
|
||||
|
||||
for gcs_path in output_files:
|
||||
filename = os.path.basename(gcs_path)
|
||||
local_path = os.path.join(temp_dir, filename)
|
||||
|
||||
job_log.info(f"Downloading encoded file: {filename}")
|
||||
storage.download_file(gcs_path, local_path)
|
||||
|
||||
# Map to result keys expected by _upload_results
|
||||
# Files are named like "Artist - Title (Final Karaoke Lossless 4k).mp4"
|
||||
filename_lower = filename.lower()
|
||||
if "lossless 4k" in filename_lower:
|
||||
if filename.endswith(".mkv"):
|
||||
local_files["final_video_mkv"] = local_path
|
||||
else:
|
||||
local_files["final_video"] = local_path
|
||||
elif "lossy 4k" in filename_lower:
|
||||
local_files["final_video_lossy"] = local_path
|
||||
elif "720p" in filename_lower:
|
||||
local_files["final_video_720p"] = local_path
|
||||
|
||||
job_log.info(f"Downloaded {len(local_files)} encoded files")
|
||||
|
||||
return local_files
|
||||
|
||||
except Exception as e:
|
||||
job_log.error(f"GCE encoding failed: {e}")
|
||||
logger.error(f"[job:{job_id}] GCE encoding failed: {e}")
|
||||
raise
|
||||
|
||||
|
||||
async def generate_video_orchestrated(job_id: str) -> bool:
|
||||
"""
|
||||
Generate final karaoke videos using the new VideoWorkerOrchestrator.
|
||||
|
||||
This provides a unified pipeline that works with any encoding backend
|
||||
(GCE or local), ensuring all features like YouTube upload, Discord
|
||||
notifications, and CDG/TXT packaging work consistently.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to process
|
||||
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
from backend.workers.video_worker_orchestrator import (
|
||||
VideoWorkerOrchestrator,
|
||||
create_orchestrator_config_from_job,
|
||||
)
|
||||
|
||||
start_time = time.time()
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
# Create job logger for remote debugging
|
||||
job_log = create_job_logger(job_id, "video")
|
||||
|
||||
# Log with structured markers for easy Cloud Logging queries
|
||||
logger.info(f"[job:{job_id}] WORKER_START worker=video orchestrator=true")
|
||||
|
||||
# Set up log capture for KaraokeFinalise (still used by some services)
|
||||
log_handler = setup_job_logging(job_id, "video", *VIDEO_WORKER_LOGGERS)
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
logger.error(f"[job:{job_id}] Job not found")
|
||||
return False
|
||||
|
||||
# Validate prerequisites
|
||||
if not _validate_prerequisites(job):
|
||||
logger.error(f"[job:{job_id}] Prerequisites not met for video generation")
|
||||
return False
|
||||
|
||||
# Create temporary working directory
|
||||
temp_dir = tempfile.mkdtemp(prefix=f"karaoke_video_{job_id}_")
|
||||
original_cwd = os.getcwd()
|
||||
|
||||
# Set up rclone config if needed (for legacy Dropbox upload path)
|
||||
rclone_service = None
|
||||
if getattr(job, 'organised_dir_rclone_root', None):
|
||||
rclone_service = get_rclone_service()
|
||||
if rclone_service.setup_rclone_config():
|
||||
job_log.info("Rclone config loaded for Dropbox upload")
|
||||
|
||||
# Load YouTube credentials if needed
|
||||
youtube_credentials = None
|
||||
if getattr(job, 'enable_youtube_upload', False):
|
||||
youtube_service = get_youtube_service()
|
||||
if youtube_service.is_configured:
|
||||
youtube_credentials = youtube_service.get_credentials_dict()
|
||||
job_log.info("YouTube credentials loaded for video upload")
|
||||
else:
|
||||
job_log.warning("YouTube credentials not available - upload will be skipped")
|
||||
|
||||
try:
|
||||
# Wrap entire worker in a tracing span
|
||||
with job_span("video-worker-orchestrated", job_id, {"artist": job.artist, "title": job.title}) as root_span:
|
||||
with job_logging_context(job_id):
|
||||
job_log.info(f"Starting orchestrated video generation for {job.artist} - {job.title}")
|
||||
logger.info(f"[job:{job_id}] Starting orchestrated video generation")
|
||||
|
||||
# Transition to GENERATING_VIDEO state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.GENERATING_VIDEO,
|
||||
progress=70,
|
||||
message="Preparing files for video generation"
|
||||
)
|
||||
|
||||
# Download and set up files
|
||||
# Sanitize artist/title to handle Unicode characters (curly quotes, em dashes, etc.)
|
||||
safe_artist = sanitize_filename(job.artist) if job.artist else "Unknown"
|
||||
safe_title = sanitize_filename(job.title) if job.title else "Unknown"
|
||||
base_name = f"{safe_artist} - {safe_title}"
|
||||
with job_span("download-files", job_id):
|
||||
job_log.info("Downloading files from GCS...")
|
||||
await _setup_working_directory(job_id, job, storage, temp_dir, base_name, job_log)
|
||||
job_log.info("All files downloaded successfully")
|
||||
|
||||
# Load style config for CDG styles
|
||||
style_config = await load_style_config(job, storage, temp_dir)
|
||||
cdg_styles = style_config.get_cdg_styles()
|
||||
|
||||
# Change to working directory
|
||||
os.chdir(temp_dir)
|
||||
|
||||
# Create orchestrator config from job
|
||||
config = create_orchestrator_config_from_job(
|
||||
job=job,
|
||||
temp_dir=temp_dir,
|
||||
youtube_credentials=youtube_credentials,
|
||||
cdg_styles=cdg_styles,
|
||||
)
|
||||
|
||||
# Run the orchestrated pipeline
|
||||
orchestrator = VideoWorkerOrchestrator(
|
||||
config=config,
|
||||
job_manager=job_manager,
|
||||
storage=storage,
|
||||
job_logger=job_log,
|
||||
)
|
||||
|
||||
with job_span("orchestrator-run", job_id):
|
||||
result = await orchestrator.run()
|
||||
|
||||
if not result.success:
|
||||
raise Exception(result.error_message or "Orchestrator failed")
|
||||
|
||||
# Prepare distribution directory for native uploads
|
||||
with job_span("distribution", job_id):
|
||||
await _handle_native_distribution(
|
||||
job_id=job_id,
|
||||
job=job,
|
||||
job_log=job_log,
|
||||
job_manager=job_manager,
|
||||
temp_dir=temp_dir,
|
||||
result={
|
||||
'brand_code': result.brand_code,
|
||||
'youtube_url': result.youtube_url,
|
||||
'final_video': result.final_video,
|
||||
'final_video_lossy': result.final_video_lossy,
|
||||
'final_video_720p': result.final_video_720p,
|
||||
'final_karaoke_cdg_zip': result.final_karaoke_cdg_zip,
|
||||
'dropbox_link': result.dropbox_link,
|
||||
'gdrive_files': result.gdrive_files,
|
||||
},
|
||||
storage=storage,
|
||||
)
|
||||
|
||||
# Upload generated files to GCS
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.PACKAGING,
|
||||
progress=95,
|
||||
message="Uploading final files"
|
||||
)
|
||||
|
||||
with job_span("upload-results", job_id):
|
||||
await _upload_results(job_id, job_manager, storage, temp_dir, {
|
||||
'final_video': result.final_video,
|
||||
'final_video_mkv': result.final_video_mkv,
|
||||
'final_video_lossy': result.final_video_lossy,
|
||||
'final_video_720p': result.final_video_720p,
|
||||
'final_karaoke_cdg_zip': result.final_karaoke_cdg_zip,
|
||||
'final_karaoke_txt_zip': result.final_karaoke_txt_zip,
|
||||
})
|
||||
|
||||
# Store result metadata in job BEFORE transitioning to COMPLETE
|
||||
# This ensures youtube_url is available when completion email is sent
|
||||
logger.info(f"[job:{job_id}] Video generation complete")
|
||||
job_manager.update_job(job_id, {
|
||||
'state_data': {
|
||||
**job.state_data,
|
||||
'brand_code': result.brand_code,
|
||||
'youtube_url': result.youtube_url,
|
||||
'dropbox_link': result.dropbox_link,
|
||||
'gdrive_files': result.gdrive_files,
|
||||
}
|
||||
})
|
||||
|
||||
# Mark job as complete (triggers completion email with youtube_url now available)
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.COMPLETE,
|
||||
progress=100,
|
||||
message="Karaoke generation complete!"
|
||||
)
|
||||
|
||||
duration = time.time() - start_time
|
||||
root_span.set_attribute("duration_seconds", duration)
|
||||
root_span.set_attribute("brand_code", result.brand_code or '')
|
||||
logger.info(f"[job:{job_id}] WORKER_END worker=video orchestrator=true status=success duration={duration:.1f}s")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
duration = time.time() - start_time
|
||||
logger.error(f"[job:{job_id}] WORKER_END worker=video orchestrator=true status=error duration={duration:.1f}s error={e}")
|
||||
job_manager.mark_job_failed(
|
||||
job_id=job_id,
|
||||
error_message=f"Video generation failed: {str(e)}",
|
||||
error_details={"stage": "video_generation", "error": str(e)}
|
||||
)
|
||||
return False
|
||||
|
||||
finally:
|
||||
# Restore original working directory
|
||||
os.chdir(original_cwd)
|
||||
|
||||
# Remove log handler
|
||||
for logger_name in VIDEO_WORKER_LOGGERS:
|
||||
try:
|
||||
logging.getLogger(logger_name).removeHandler(log_handler)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Cleanup rclone config file
|
||||
if rclone_service:
|
||||
rclone_service.cleanup()
|
||||
|
||||
# Cleanup temporary directory
|
||||
if os.path.exists(temp_dir):
|
||||
shutil.rmtree(temp_dir)
|
||||
logger.debug(f"Cleaned up temp directory: {temp_dir}")
|
||||
|
||||
|
||||
async def generate_video(job_id: str) -> bool:
|
||||
"""
|
||||
Generate final karaoke videos.
|
||||
|
||||
Routes to either the new orchestrator-based pipeline or the legacy
|
||||
KaraokeFinalise-based pipeline based on USE_NEW_ORCHESTRATOR flag.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to process
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
if USE_NEW_ORCHESTRATOR:
|
||||
logger.info(f"[job:{job_id}] Using new orchestrator-based pipeline")
|
||||
return await generate_video_orchestrated(job_id)
|
||||
else:
|
||||
logger.info(f"[job:{job_id}] Using legacy KaraokeFinalise pipeline")
|
||||
return await generate_video_legacy(job_id)
|
||||
|
||||
|
||||
async def generate_video_legacy(job_id: str) -> bool:
|
||||
"""
|
||||
Generate final karaoke videos using KaraokeFinalise (legacy path).
|
||||
|
||||
This is the original implementation that has divergent behavior between
|
||||
GCE and local encoding paths. Kept for rollback purposes.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to process
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
@@ -134,7 +497,10 @@ async def generate_video(job_id: str) -> bool:
|
||||
)
|
||||
|
||||
# Download and set up files in the format KaraokeFinalise expects
|
||||
base_name = f"{job.artist} - {job.title}"
|
||||
# Sanitize artist/title to handle Unicode characters (curly quotes, em dashes, etc.)
|
||||
safe_artist = sanitize_filename(job.artist) if job.artist else "Unknown"
|
||||
safe_title = sanitize_filename(job.title) if job.title else "Unknown"
|
||||
base_name = f"{safe_artist} - {safe_title}"
|
||||
with job_span("download-files", job_id):
|
||||
job_log.info("Downloading files from GCS (this may take a few minutes for large files)...")
|
||||
await _setup_working_directory(job_id, job, storage, temp_dir, base_name, job_log)
|
||||
@@ -184,80 +550,105 @@ async def generate_video(job_id: str) -> bool:
|
||||
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
|
||||
job_log.info(f"Countdown padding detected: {countdown_padding_seconds}s - instrumental will be padded if needed")
|
||||
|
||||
# Log finalization parameters
|
||||
job_log.info("Creating KaraokeFinalise with parameters:")
|
||||
job_log.info(f" enable_cdg: {getattr(job, 'enable_cdg', False)}")
|
||||
job_log.info(f" enable_txt: {getattr(job, 'enable_txt', False)}")
|
||||
job_log.info(f" brand_prefix: {getattr(job, 'brand_prefix', None)}")
|
||||
job_log.info(f" discord_webhook: {'configured' if getattr(job, 'discord_webhook_url', None) else 'not configured'}")
|
||||
job_log.info(f" instrumental source: {instrumental_source}")
|
||||
job_log.info(f" countdown_padding_seconds: {countdown_padding_seconds}")
|
||||
if existing_instrumental_path:
|
||||
job_log.info(f" using user-provided instrumental (selection was: {instrumental_selection})")
|
||||
|
||||
# Create KaraokeFinalise with ALL the parameters from the job
|
||||
# This reuses all existing functionality!
|
||||
|
||||
# Set up YouTube description file if template is provided
|
||||
youtube_desc_path = None
|
||||
if youtube_credentials and getattr(job, 'youtube_description_template', None):
|
||||
youtube_desc_path = os.path.join(temp_dir, "youtube_description.txt")
|
||||
with open(youtube_desc_path, 'w') as f:
|
||||
f.write(job.youtube_description_template)
|
||||
job_log.info("YouTube description template written to temp file")
|
||||
|
||||
finalise = KaraokeFinalise(
|
||||
logger=logger,
|
||||
log_level=logging.INFO,
|
||||
dry_run=False,
|
||||
instrumental_format="flac",
|
||||
# CDG/TXT generation
|
||||
enable_cdg=getattr(job, 'enable_cdg', False),
|
||||
enable_txt=getattr(job, 'enable_txt', False),
|
||||
cdg_styles=cdg_styles,
|
||||
# Brand code and organization (server-side mode uses rclone)
|
||||
brand_prefix=getattr(job, 'brand_prefix', None),
|
||||
organised_dir=None, # Not used in cloud - files stay in GCS
|
||||
organised_dir_rclone_root=getattr(job, 'organised_dir_rclone_root', None),
|
||||
public_share_dir=None, # Not used in cloud
|
||||
# Notifications
|
||||
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
|
||||
# YouTube upload (server-side with pre-loaded credentials)
|
||||
youtube_client_secrets_file=None, # Not used with pre-stored credentials
|
||||
youtube_description_file=youtube_desc_path,
|
||||
user_youtube_credentials=youtube_credentials, # Pre-loaded from Secret Manager
|
||||
rclone_destination=None,
|
||||
email_template_file=None,
|
||||
# Server-side optimizations
|
||||
non_interactive=True,
|
||||
server_side_mode=True,
|
||||
selected_instrumental_file=instrumental_file,
|
||||
# Audio synchronization - ensure instrumental matches vocal padding
|
||||
countdown_padding_seconds=countdown_padding_seconds,
|
||||
)
|
||||
|
||||
# Call process() - this does ALL the work:
|
||||
# - Encodes to 4 video formats
|
||||
# - Generates CDG/TXT packages if enabled
|
||||
# - Posts Discord notification if configured
|
||||
# - Handles brand code generation
|
||||
with job_span("karaoke-finalise", job_id) as finalise_span:
|
||||
finalise_start = time.time()
|
||||
job_log.info("Starting KaraokeFinalise.process() - this may take 15-20 minutes...")
|
||||
logger.info(f"[job:{job_id}] Starting KaraokeFinalise.process()")
|
||||
add_span_event("finalise_started")
|
||||
|
||||
result = finalise.process()
|
||||
|
||||
finalise_duration = time.time() - finalise_start
|
||||
finalise_span.set_attribute("duration_seconds", finalise_duration)
|
||||
add_span_event("finalise_completed", {"duration_seconds": finalise_duration})
|
||||
|
||||
job_log.info("KaraokeFinalise.process() complete!")
|
||||
if result.get('brand_code'):
|
||||
job_log.info(f"Brand code: {result.get('brand_code')}")
|
||||
logger.info(f"[job:{job_id}] KaraokeFinalise.process() complete in {finalise_duration:.1f}s")
|
||||
logger.info(f"[job:{job_id}] Brand code: {result.get('brand_code')}")
|
||||
# Check if GCE encoding is enabled for high-performance encoding
|
||||
encoding_service = get_encoding_service()
|
||||
use_gce_encoding = encoding_service.is_enabled
|
||||
|
||||
if use_gce_encoding:
|
||||
# ============ GCE ENCODING PATH ============
|
||||
# Use dedicated C4 GCE instance for 2-3x faster FFmpeg encoding
|
||||
job_log.info("GCE encoding enabled - using high-performance encoding worker")
|
||||
job_log.info(f" instrumental source: {instrumental_source}")
|
||||
|
||||
result = await _encode_via_gce(
|
||||
job_id=job_id,
|
||||
job=job,
|
||||
job_manager=job_manager,
|
||||
storage=storage,
|
||||
temp_dir=temp_dir,
|
||||
base_name=base_name,
|
||||
job_log=job_log,
|
||||
)
|
||||
|
||||
# GCE encoding doesn't generate brand_code, CDG/TXT, or Discord notifications
|
||||
# These will be handled by distribution if needed
|
||||
job_log.info("GCE encoding complete - encoded files downloaded")
|
||||
|
||||
else:
|
||||
# ============ STANDARD ENCODING PATH ============
|
||||
# Use KaraokeFinalise for encoding (runs on Cloud Run)
|
||||
|
||||
# Log finalization parameters
|
||||
job_log.info("Using KaraokeFinalise for encoding (standard path)")
|
||||
job_log.info(f" enable_cdg: {getattr(job, 'enable_cdg', False)}")
|
||||
job_log.info(f" enable_txt: {getattr(job, 'enable_txt', False)}")
|
||||
job_log.info(f" brand_prefix: {getattr(job, 'brand_prefix', None)}")
|
||||
job_log.info(f" discord_webhook: {'configured' if getattr(job, 'discord_webhook_url', None) else 'not configured'}")
|
||||
job_log.info(f" instrumental source: {instrumental_source}")
|
||||
job_log.info(f" countdown_padding_seconds: {countdown_padding_seconds}")
|
||||
if existing_instrumental_path:
|
||||
job_log.info(f" using user-provided instrumental (selection was: {instrumental_selection})")
|
||||
|
||||
# Set up YouTube description file if template is provided
|
||||
youtube_desc_path = None
|
||||
if youtube_credentials and getattr(job, 'youtube_description_template', None):
|
||||
youtube_desc_path = os.path.join(temp_dir, "youtube_description.txt")
|
||||
with open(youtube_desc_path, 'w') as f:
|
||||
f.write(job.youtube_description_template)
|
||||
job_log.info("YouTube description template written to temp file")
|
||||
|
||||
finalise = KaraokeFinalise(
|
||||
logger=logger,
|
||||
log_level=logging.INFO,
|
||||
dry_run=False,
|
||||
instrumental_format="flac",
|
||||
# CDG/TXT generation
|
||||
enable_cdg=getattr(job, 'enable_cdg', False),
|
||||
enable_txt=getattr(job, 'enable_txt', False),
|
||||
cdg_styles=cdg_styles,
|
||||
# Brand code and organization (server-side mode uses rclone)
|
||||
brand_prefix=getattr(job, 'brand_prefix', None),
|
||||
organised_dir=None, # Not used in cloud - files stay in GCS
|
||||
organised_dir_rclone_root=getattr(job, 'organised_dir_rclone_root', None),
|
||||
public_share_dir=None, # Not used in cloud
|
||||
# Notifications
|
||||
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
|
||||
# YouTube upload (server-side with pre-loaded credentials)
|
||||
youtube_client_secrets_file=None, # Not used with pre-stored credentials
|
||||
youtube_description_file=youtube_desc_path,
|
||||
user_youtube_credentials=youtube_credentials, # Pre-loaded from Secret Manager
|
||||
rclone_destination=None,
|
||||
email_template_file=None,
|
||||
# Server-side optimizations
|
||||
non_interactive=True,
|
||||
server_side_mode=True,
|
||||
selected_instrumental_file=instrumental_file,
|
||||
# Audio synchronization - ensure instrumental matches vocal padding
|
||||
countdown_padding_seconds=countdown_padding_seconds,
|
||||
)
|
||||
|
||||
# Call process() - this does ALL the work:
|
||||
# - Encodes to 4 video formats
|
||||
# - Generates CDG/TXT packages if enabled
|
||||
# - Posts Discord notification if configured
|
||||
# - Handles brand code generation
|
||||
with job_span("karaoke-finalise", job_id) as finalise_span:
|
||||
finalise_start = time.time()
|
||||
job_log.info("Starting KaraokeFinalise.process() - this may take 15-20 minutes...")
|
||||
logger.info(f"[job:{job_id}] Starting KaraokeFinalise.process()")
|
||||
add_span_event("finalise_started")
|
||||
|
||||
result = finalise.process()
|
||||
|
||||
finalise_duration = time.time() - finalise_start
|
||||
finalise_span.set_attribute("duration_seconds", finalise_duration)
|
||||
add_span_event("finalise_completed", {"duration_seconds": finalise_duration})
|
||||
|
||||
job_log.info("KaraokeFinalise.process() complete!")
|
||||
if result.get('brand_code'):
|
||||
job_log.info(f"Brand code: {result.get('brand_code')}")
|
||||
logger.info(f"[job:{job_id}] KaraokeFinalise.process() complete in {finalise_duration:.1f}s")
|
||||
logger.info(f"[job:{job_id}] Brand code: {result.get('brand_code')}")
|
||||
|
||||
# Native API distribution uploads (used by remote CLI instead of rclone)
|
||||
with job_span("distribution", job_id):
|
||||
@@ -268,6 +659,7 @@ async def generate_video(job_id: str) -> bool:
|
||||
job_manager=job_manager,
|
||||
temp_dir=temp_dir,
|
||||
result=result,
|
||||
storage=storage,
|
||||
)
|
||||
|
||||
# Upload generated files to GCS
|
||||
@@ -281,24 +673,30 @@ async def generate_video(job_id: str) -> bool:
|
||||
with job_span("upload-results", job_id):
|
||||
await _upload_results(job_id, job_manager, storage, temp_dir, result)
|
||||
|
||||
# Mark job as complete
|
||||
# Store result metadata in job BEFORE transitioning to COMPLETE
|
||||
# This ensures youtube_url is available when completion email is sent
|
||||
# NOTE: Must include dropbox_link and gdrive_files from result, since
|
||||
# _handle_native_distribution already saved them but job.state_data
|
||||
# is stale (fetched before distribution ran)
|
||||
logger.info(f"[job:{job_id}] Video generation complete")
|
||||
job_manager.update_job(job_id, {
|
||||
'state_data': {
|
||||
**job.state_data,
|
||||
'brand_code': result.get('brand_code'),
|
||||
'youtube_url': result.get('youtube_url'),
|
||||
'dropbox_link': result.get('dropbox_link'),
|
||||
'gdrive_files': result.get('gdrive_files'),
|
||||
}
|
||||
})
|
||||
|
||||
# Mark job as complete (triggers completion email with youtube_url now available)
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.COMPLETE,
|
||||
progress=100,
|
||||
message="Karaoke generation complete!"
|
||||
)
|
||||
|
||||
# Store result metadata in job
|
||||
job_manager.update_job(job_id, {
|
||||
'state_data': {
|
||||
**job.state_data,
|
||||
'brand_code': result.get('brand_code'),
|
||||
'youtube_url': result.get('youtube_url'),
|
||||
}
|
||||
})
|
||||
|
||||
|
||||
duration = time.time() - start_time
|
||||
root_span.set_attribute("duration_seconds", duration)
|
||||
root_span.set_attribute("brand_code", result.get('brand_code', ''))
|
||||
@@ -343,16 +741,17 @@ async def _handle_native_distribution(
|
||||
job_manager: JobManager,
|
||||
temp_dir: str,
|
||||
result: Dict[str, Any],
|
||||
storage: StorageService = None,
|
||||
) -> None:
|
||||
"""
|
||||
Handle distribution uploads using native APIs (Dropbox SDK, Google Drive API).
|
||||
|
||||
|
||||
This is used by the remote CLI instead of rclone-based uploads.
|
||||
The native APIs provide:
|
||||
- Better error handling and retry logic
|
||||
- No need for rclone binary in container
|
||||
- Credentials managed via Secret Manager
|
||||
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
job: Job object with dropbox_path and gdrive_folder_id fields
|
||||
@@ -360,10 +759,14 @@ async def _handle_native_distribution(
|
||||
job_manager: Job manager for updating job state
|
||||
temp_dir: Temporary directory with output files
|
||||
result: Result dict from KaraokeFinalise.process()
|
||||
storage: StorageService instance for downloading stems/lyrics
|
||||
"""
|
||||
brand_code = result.get('brand_code')
|
||||
base_name = f"{job.artist} - {job.title}"
|
||||
|
||||
# Sanitize artist/title to handle Unicode characters (curly quotes, em dashes, etc.)
|
||||
safe_artist = sanitize_filename(job.artist) if job.artist else "Unknown"
|
||||
safe_title = sanitize_filename(job.title) if job.title else "Unknown"
|
||||
base_name = f"{safe_artist} - {safe_title}"
|
||||
|
||||
# Check if we should preserve existing brand code (Batch 6: --keep-brand-code)
|
||||
keep_brand_code = getattr(job, 'keep_brand_code', None)
|
||||
if keep_brand_code:
|
||||
@@ -392,10 +795,22 @@ async def _handle_native_distribution(
|
||||
result['brand_code'] = brand_code
|
||||
job_log.info(f"Generated brand code: {brand_code}")
|
||||
|
||||
# Prepare full distribution directory with stems/ and lyrics/ subfolders
|
||||
# This must be done before upload to ensure complete output structure
|
||||
if storage:
|
||||
await _prepare_distribution_directory(
|
||||
job_id=job_id,
|
||||
job=job,
|
||||
storage=storage,
|
||||
temp_dir=temp_dir,
|
||||
base_name=base_name,
|
||||
job_log=job_log,
|
||||
)
|
||||
|
||||
# Create folder name and upload files
|
||||
folder_name = f"{brand_code} - {base_name}"
|
||||
remote_folder = f"{dropbox_path}/{folder_name}"
|
||||
|
||||
|
||||
job_log.info(f"Uploading to Dropbox folder: {remote_folder}")
|
||||
dropbox.upload_folder(temp_dir, remote_folder)
|
||||
|
||||
@@ -601,6 +1016,173 @@ async def _setup_working_directory(
|
||||
log_progress("Downloaded end JPG")
|
||||
|
||||
|
||||
async def _prepare_distribution_directory(
|
||||
job_id: str,
|
||||
job,
|
||||
storage: StorageService,
|
||||
temp_dir: str,
|
||||
base_name: str,
|
||||
job_log=None
|
||||
) -> None:
|
||||
"""
|
||||
Prepare the full distribution directory structure matching local CLI output.
|
||||
|
||||
Creates:
|
||||
- stems/ subfolder with all audio stems (proper model names)
|
||||
- lyrics/ subfolder with intermediate lyrics files
|
||||
- Root-level instrumentals with proper model names
|
||||
|
||||
This ensures the Dropbox upload contains the complete output structure.
|
||||
"""
|
||||
def log_progress(message: str):
|
||||
"""Log to both module logger and job logger if available."""
|
||||
logger.info(f"Job {job_id}: {message}")
|
||||
if job_log:
|
||||
job_log.info(message)
|
||||
|
||||
log_progress("Preparing full distribution directory structure")
|
||||
|
||||
# Get model names from state_data (stored by audio_worker)
|
||||
model_names = job.state_data.get('model_names', {})
|
||||
clean_model = model_names.get('clean_instrumental_model', 'model_bs_roformer_ep_317_sdr_12.9755.ckpt')
|
||||
backing_models = model_names.get('backing_vocals_models', ['mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt'])
|
||||
other_models = model_names.get('other_stems_models', ['htdemucs_6s.yaml'])
|
||||
backing_model = backing_models[0] if backing_models else 'mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt'
|
||||
other_model = other_models[0] if other_models else 'htdemucs_6s.yaml'
|
||||
|
||||
# Create stems subdirectory
|
||||
stems_dir = os.path.join(temp_dir, "stems")
|
||||
os.makedirs(stems_dir, exist_ok=True)
|
||||
|
||||
# Create lyrics subdirectory
|
||||
lyrics_dir = os.path.join(temp_dir, "lyrics")
|
||||
os.makedirs(lyrics_dir, exist_ok=True)
|
||||
|
||||
stems = job.file_urls.get('stems', {})
|
||||
lyrics = job.file_urls.get('lyrics', {})
|
||||
|
||||
# --- Download stems to stems/ subfolder with proper model names ---
|
||||
|
||||
# Clean up simplified-named instrumentals from _setup_working_directory
|
||||
# These were needed for KaraokeFinalise but shouldn't go to distribution
|
||||
simplified_instrumental_patterns = [
|
||||
f"{base_name} (Instrumental Clean).flac",
|
||||
f"{base_name} (Instrumental Backing).flac",
|
||||
]
|
||||
for pattern in simplified_instrumental_patterns:
|
||||
simplified_path = os.path.join(temp_dir, pattern)
|
||||
if os.path.exists(simplified_path):
|
||||
os.remove(simplified_path)
|
||||
log_progress(f"Removed simplified instrumental: {pattern}")
|
||||
|
||||
# Clean instrumental (from clean_instrumental_model)
|
||||
if stems.get('instrumental_clean'):
|
||||
dest_path = os.path.join(stems_dir, f"{base_name} (Instrumental {clean_model}).flac")
|
||||
storage.download_file(stems['instrumental_clean'], dest_path)
|
||||
log_progress("Downloaded clean instrumental to stems/")
|
||||
|
||||
# Also copy to root level for distribution
|
||||
root_path = os.path.join(temp_dir, f"{base_name} (Instrumental {clean_model}).flac")
|
||||
shutil.copy2(dest_path, root_path)
|
||||
|
||||
# Clean vocals (from clean_instrumental_model)
|
||||
if stems.get('vocals_clean'):
|
||||
dest_path = os.path.join(stems_dir, f"{base_name} (Vocals {clean_model}).flac")
|
||||
storage.download_file(stems['vocals_clean'], dest_path)
|
||||
log_progress("Downloaded clean vocals to stems/")
|
||||
|
||||
# Instrumental with backing vocals (from backing_vocals_model)
|
||||
if stems.get('instrumental_with_backing'):
|
||||
dest_path = os.path.join(stems_dir, f"{base_name} (Instrumental +BV {backing_model}).flac")
|
||||
storage.download_file(stems['instrumental_with_backing'], dest_path)
|
||||
log_progress("Downloaded instrumental+BV to stems/")
|
||||
|
||||
# Also copy to root level for distribution
|
||||
root_path = os.path.join(temp_dir, f"{base_name} (Instrumental +BV {backing_model}).flac")
|
||||
shutil.copy2(dest_path, root_path)
|
||||
|
||||
# Lead vocals (from backing_vocals_model)
|
||||
if stems.get('lead_vocals'):
|
||||
dest_path = os.path.join(stems_dir, f"{base_name} (Lead Vocals {backing_model}).flac")
|
||||
storage.download_file(stems['lead_vocals'], dest_path)
|
||||
log_progress("Downloaded lead vocals to stems/")
|
||||
|
||||
# Backing vocals (from backing_vocals_model)
|
||||
if stems.get('backing_vocals'):
|
||||
dest_path = os.path.join(stems_dir, f"{base_name} (Backing Vocals {backing_model}).flac")
|
||||
storage.download_file(stems['backing_vocals'], dest_path)
|
||||
log_progress("Downloaded backing vocals to stems/")
|
||||
|
||||
# Other stems (from other_stems_model - typically htdemucs_6s)
|
||||
other_stem_keys = ['bass', 'drums', 'guitar', 'piano', 'other']
|
||||
for stem_key in other_stem_keys:
|
||||
if stems.get(stem_key):
|
||||
stem_name = stem_key.capitalize()
|
||||
dest_path = os.path.join(stems_dir, f"{base_name} ({stem_name} {other_model}).flac")
|
||||
storage.download_file(stems[stem_key], dest_path)
|
||||
log_progress(f"Downloaded {stem_key} to stems/")
|
||||
|
||||
# --- Download lyrics files to lyrics/ subfolder ---
|
||||
# Use proper filenames matching local CLI output structure
|
||||
|
||||
# Map job.files.lyrics keys to proper local filenames
|
||||
# Format: (job_files_key, local_filename_pattern)
|
||||
lyrics_file_mappings = [
|
||||
# Karaoke output files
|
||||
('lrc', f"{base_name} (Karaoke).lrc"),
|
||||
('ass', f"{base_name} (Karaoke).ass"),
|
||||
# Corrections and transcription files
|
||||
('corrections', f"{base_name} (Lyrics Corrections).json"),
|
||||
('corrected_txt', f"{base_name} (Lyrics Corrected).txt"),
|
||||
('uncorrected', f"{base_name} (Lyrics Uncorrected).txt"),
|
||||
# Reference lyrics from various sources
|
||||
('reference_genius', f"{base_name} (Lyrics Genius).txt"),
|
||||
('reference_spotify', f"{base_name} (Lyrics Spotify).txt"),
|
||||
('reference_musixmatch', f"{base_name} (Lyrics Musixmatch).txt"),
|
||||
('reference_lrclib', f"{base_name} (Lyrics Lrclib).txt"),
|
||||
]
|
||||
|
||||
for lyrics_key, local_filename in lyrics_file_mappings:
|
||||
if lyrics.get(lyrics_key):
|
||||
dest_path = os.path.join(lyrics_dir, local_filename)
|
||||
try:
|
||||
storage.download_file(lyrics[lyrics_key], dest_path)
|
||||
log_progress(f"Downloaded {lyrics_key} to lyrics/{local_filename}")
|
||||
except Exception as e:
|
||||
log_progress(f"Could not download {lyrics_key}: {e}")
|
||||
|
||||
# --- Download (With Vocals).mkv to lyrics/ subfolder ---
|
||||
# Local CLI places the karaoke video with vocals in the lyrics folder
|
||||
videos = job.file_urls.get('videos', {})
|
||||
if videos.get('with_vocals'):
|
||||
with_vocals_dest = os.path.join(lyrics_dir, f"{base_name} (With Vocals).mkv")
|
||||
try:
|
||||
storage.download_file(videos['with_vocals'], with_vocals_dest)
|
||||
log_progress(f"Downloaded with_vocals to lyrics/{base_name} (With Vocals).mkv")
|
||||
except Exception as e:
|
||||
log_progress(f"Could not download with_vocals to lyrics/: {e}")
|
||||
|
||||
# --- Download preview ASS files to lyrics/previews/ subfolder ---
|
||||
# Preview files are stored in jobs/{job_id}/previews/ in GCS
|
||||
previews_dir = os.path.join(lyrics_dir, "previews")
|
||||
os.makedirs(previews_dir, exist_ok=True)
|
||||
|
||||
# List all preview files from GCS and download them
|
||||
try:
|
||||
preview_prefix = f"jobs/{job_id}/previews/"
|
||||
preview_files = storage.list_files(preview_prefix)
|
||||
for blob_name in preview_files:
|
||||
if blob_name.endswith('.ass'):
|
||||
filename = os.path.basename(blob_name)
|
||||
dest_path = os.path.join(previews_dir, filename)
|
||||
storage.download_file(blob_name, dest_path)
|
||||
log_progress(f"Downloaded preview {filename} to lyrics/previews/")
|
||||
except Exception as e:
|
||||
log_progress(f"Could not download preview files: {e}")
|
||||
|
||||
log_progress("Distribution directory prepared with stems/ and lyrics/ subfolders")
|
||||
|
||||
|
||||
async def _upload_results(
|
||||
job_id: str,
|
||||
job_manager: JobManager,
|
||||
@@ -609,7 +1191,7 @@ async def _upload_results(
|
||||
result: Dict[str, Any]
|
||||
) -> None:
|
||||
"""Upload generated files to GCS."""
|
||||
|
||||
|
||||
# Map of result keys to GCS paths
|
||||
file_mappings = [
|
||||
('final_video', 'finals', 'lossless_4k_mp4'),
|
||||
|
||||
@@ -0,0 +1,701 @@
|
||||
"""
|
||||
Video Worker Orchestrator.
|
||||
|
||||
Coordinates the video generation pipeline stages in a unified way,
|
||||
regardless of whether encoding happens locally or on GCE.
|
||||
|
||||
This resolves the code path divergence where GCE encoding bypassed
|
||||
features like YouTube upload, Discord notifications, and CDG/TXT packaging.
|
||||
|
||||
Pipeline stages:
|
||||
1. Setup - Download files, prepare directories
|
||||
2. Packaging - CDG/TXT generation (if enabled)
|
||||
3. Encoding - GCE or Local via EncodingBackend interface
|
||||
4. Organization - Brand code, folder structure
|
||||
5. Distribution - YouTube, Dropbox, Google Drive uploads
|
||||
6. Notification - Discord notifications
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Optional, Dict, Any, List
|
||||
from pathlib import Path
|
||||
|
||||
from backend.models.job import JobStatus
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.tracing import job_span, add_span_event
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class OrchestratorConfig:
|
||||
"""Configuration for the video worker orchestrator."""
|
||||
job_id: str
|
||||
artist: str
|
||||
title: str
|
||||
|
||||
# Input file paths (in temp_dir)
|
||||
title_video_path: str
|
||||
karaoke_video_path: str
|
||||
instrumental_audio_path: str
|
||||
end_video_path: Optional[str] = None
|
||||
lrc_file_path: Optional[str] = None
|
||||
title_jpg_path: Optional[str] = None
|
||||
|
||||
# Output directory
|
||||
output_dir: str = ""
|
||||
|
||||
# Feature flags
|
||||
enable_cdg: bool = False
|
||||
enable_txt: bool = False
|
||||
enable_youtube_upload: bool = False
|
||||
|
||||
# Service configurations
|
||||
brand_prefix: Optional[str] = None
|
||||
discord_webhook_url: Optional[str] = None
|
||||
youtube_credentials: Optional[Dict[str, Any]] = None
|
||||
youtube_description_template: Optional[str] = None
|
||||
cdg_styles: Optional[Dict[str, Any]] = None
|
||||
|
||||
# Dropbox/GDrive configuration
|
||||
dropbox_path: Optional[str] = None
|
||||
gdrive_folder_id: Optional[str] = None
|
||||
|
||||
# Keep existing brand code (for re-processing)
|
||||
keep_brand_code: Optional[str] = None
|
||||
|
||||
# Encoding backend preference
|
||||
encoding_backend: str = "auto" # "auto", "local", "gce"
|
||||
|
||||
# Additional options
|
||||
dry_run: bool = False
|
||||
non_interactive: bool = True
|
||||
|
||||
|
||||
@dataclass
|
||||
class OrchestratorResult:
|
||||
"""Result from the video worker orchestrator."""
|
||||
success: bool
|
||||
error_message: Optional[str] = None
|
||||
|
||||
# Generated files
|
||||
final_video: Optional[str] = None # Lossless 4K MP4
|
||||
final_video_mkv: Optional[str] = None # Lossless 4K MKV
|
||||
final_video_lossy: Optional[str] = None # Lossy 4K MP4
|
||||
final_video_720p: Optional[str] = None # Lossy 720p MP4
|
||||
final_karaoke_cdg_zip: Optional[str] = None
|
||||
final_karaoke_txt_zip: Optional[str] = None
|
||||
|
||||
# Organization
|
||||
brand_code: Optional[str] = None
|
||||
|
||||
# Distribution results
|
||||
youtube_url: Optional[str] = None
|
||||
dropbox_link: Optional[str] = None
|
||||
gdrive_files: Optional[Dict[str, str]] = field(default_factory=dict)
|
||||
|
||||
# Timing
|
||||
encoding_time_seconds: Optional[float] = None
|
||||
total_time_seconds: Optional[float] = None
|
||||
|
||||
|
||||
class VideoWorkerOrchestrator:
|
||||
"""
|
||||
Orchestrates the video generation pipeline.
|
||||
|
||||
This class coordinates all stages of video generation in a unified way,
|
||||
ensuring that features like YouTube upload and Discord notifications
|
||||
work regardless of whether GCE or local encoding is used.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
config: OrchestratorConfig,
|
||||
job_manager: Optional[JobManager] = None,
|
||||
storage: Optional[StorageService] = None,
|
||||
job_logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the orchestrator.
|
||||
|
||||
Args:
|
||||
config: Orchestrator configuration
|
||||
job_manager: Job manager for state updates (optional)
|
||||
storage: Storage service for file downloads (optional)
|
||||
job_logger: Job-specific logger (optional)
|
||||
"""
|
||||
self.config = config
|
||||
self.job_manager = job_manager
|
||||
self.storage = storage
|
||||
self.job_log = job_logger or logging.getLogger(__name__)
|
||||
|
||||
# Services (lazy-loaded)
|
||||
self._encoding_backend = None
|
||||
self._packaging_service = None
|
||||
self._youtube_service = None
|
||||
self._discord_service = None
|
||||
|
||||
# State
|
||||
self.result = OrchestratorResult(success=False)
|
||||
|
||||
def _get_encoding_backend(self):
|
||||
"""Lazy-load the encoding backend."""
|
||||
if self._encoding_backend is None:
|
||||
from backend.services.encoding_interface import get_encoding_backend
|
||||
self._encoding_backend = get_encoding_backend(
|
||||
backend_type=self.config.encoding_backend,
|
||||
dry_run=self.config.dry_run,
|
||||
logger=self.job_log,
|
||||
)
|
||||
return self._encoding_backend
|
||||
|
||||
def _get_packaging_service(self):
|
||||
"""Lazy-load the packaging service."""
|
||||
if self._packaging_service is None:
|
||||
from backend.services.packaging_service import PackagingService
|
||||
self._packaging_service = PackagingService(
|
||||
cdg_styles=self.config.cdg_styles,
|
||||
dry_run=self.config.dry_run,
|
||||
non_interactive=self.config.non_interactive,
|
||||
logger=self.job_log,
|
||||
)
|
||||
return self._packaging_service
|
||||
|
||||
def _get_youtube_service(self):
|
||||
"""Lazy-load the YouTube upload service."""
|
||||
if self._youtube_service is None:
|
||||
from backend.services.youtube_upload_service import YouTubeUploadService
|
||||
self._youtube_service = YouTubeUploadService(
|
||||
credentials=self.config.youtube_credentials,
|
||||
non_interactive=self.config.non_interactive,
|
||||
server_side_mode=True,
|
||||
dry_run=self.config.dry_run,
|
||||
logger=self.job_log,
|
||||
)
|
||||
return self._youtube_service
|
||||
|
||||
def _get_discord_service(self):
|
||||
"""Lazy-load the Discord notification service."""
|
||||
if self._discord_service is None:
|
||||
from backend.services.discord_service import DiscordNotificationService
|
||||
self._discord_service = DiscordNotificationService(
|
||||
webhook_url=self.config.discord_webhook_url,
|
||||
dry_run=self.config.dry_run,
|
||||
logger=self.job_log,
|
||||
)
|
||||
return self._discord_service
|
||||
|
||||
def _update_progress(self, status: JobStatus, progress: int, message: str):
|
||||
"""Update job progress if job_manager is available."""
|
||||
if self.job_manager:
|
||||
self.job_manager.transition_to_state(
|
||||
job_id=self.config.job_id,
|
||||
new_status=status,
|
||||
progress=progress,
|
||||
message=message
|
||||
)
|
||||
|
||||
async def run(self) -> OrchestratorResult:
|
||||
"""
|
||||
Run the full video generation pipeline.
|
||||
|
||||
Returns:
|
||||
OrchestratorResult with generated files and metadata
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
with job_span("orchestrator", self.config.job_id) as span:
|
||||
self.job_log.info(f"Starting orchestrated video generation for {self.config.artist} - {self.config.title}")
|
||||
|
||||
# Stage 1: Packaging (CDG/TXT) - runs BEFORE encoding
|
||||
# This depends on LRC file, not on encoded videos
|
||||
if self.config.enable_cdg or self.config.enable_txt:
|
||||
await self._run_packaging()
|
||||
|
||||
# Stage 2: Encoding
|
||||
await self._run_encoding()
|
||||
|
||||
# Stage 3: Organization (brand code)
|
||||
await self._run_organization()
|
||||
|
||||
# Stage 4: Distribution (YouTube, Dropbox, GDrive)
|
||||
await self._run_distribution()
|
||||
|
||||
# Stage 5: Notifications (Discord)
|
||||
await self._run_notifications()
|
||||
|
||||
self.result.success = True
|
||||
self.result.total_time_seconds = time.time() - start_time
|
||||
|
||||
self.job_log.info(f"Orchestrated video generation complete in {self.result.total_time_seconds:.1f}s")
|
||||
|
||||
except Exception as e:
|
||||
self.result.success = False
|
||||
self.result.error_message = str(e)
|
||||
self.result.total_time_seconds = time.time() - start_time
|
||||
self.job_log.error(f"Orchestrated video generation failed: {e}")
|
||||
logger.error(f"[job:{self.config.job_id}] Orchestration failed: {e}")
|
||||
|
||||
return self.result
|
||||
|
||||
async def _run_packaging(self):
|
||||
"""Run the packaging stage (CDG/TXT generation)."""
|
||||
self.job_log.info("Starting packaging stage (CDG/TXT)")
|
||||
|
||||
if not self.config.lrc_file_path or not os.path.isfile(self.config.lrc_file_path):
|
||||
self.job_log.warning("No LRC file available, skipping CDG/TXT packaging")
|
||||
return
|
||||
|
||||
base_name = f"{self.config.artist} - {self.config.title}"
|
||||
packaging_service = self._get_packaging_service()
|
||||
|
||||
# Generate CDG package
|
||||
if self.config.enable_cdg:
|
||||
self.job_log.info("Generating CDG package")
|
||||
try:
|
||||
cdg_zip_path = os.path.join(
|
||||
self.config.output_dir,
|
||||
f"{base_name} (Final Karaoke CDG).zip"
|
||||
)
|
||||
mp3_path = os.path.join(
|
||||
self.config.output_dir,
|
||||
f"{base_name} (Karaoke).mp3"
|
||||
)
|
||||
cdg_path = os.path.join(
|
||||
self.config.output_dir,
|
||||
f"{base_name} (Karaoke).cdg"
|
||||
)
|
||||
|
||||
zip_file, mp3_file, cdg_file = packaging_service.create_cdg_package(
|
||||
lrc_file=self.config.lrc_file_path,
|
||||
audio_file=self.config.instrumental_audio_path,
|
||||
output_zip_path=cdg_zip_path,
|
||||
artist=self.config.artist,
|
||||
title=self.config.title,
|
||||
output_mp3_path=mp3_path,
|
||||
output_cdg_path=cdg_path,
|
||||
)
|
||||
|
||||
self.result.final_karaoke_cdg_zip = zip_file
|
||||
self.job_log.info(f"CDG package created: {zip_file}")
|
||||
|
||||
except Exception as e:
|
||||
self.job_log.error(f"CDG generation failed: {e}")
|
||||
# Don't fail the pipeline, CDG is optional
|
||||
|
||||
# Generate TXT package
|
||||
if self.config.enable_txt:
|
||||
self.job_log.info("Generating TXT package")
|
||||
try:
|
||||
# TXT package needs MP3 file (from CDG generation or create it)
|
||||
mp3_path = os.path.join(
|
||||
self.config.output_dir,
|
||||
f"{base_name} (Karaoke).mp3"
|
||||
)
|
||||
|
||||
if not os.path.isfile(mp3_path):
|
||||
self.job_log.info("MP3 not found, CDG must be enabled first for TXT")
|
||||
return
|
||||
|
||||
txt_zip_path = os.path.join(
|
||||
self.config.output_dir,
|
||||
f"{base_name} (Final Karaoke TXT).zip"
|
||||
)
|
||||
|
||||
zip_file, txt_file = packaging_service.create_txt_package(
|
||||
lrc_file=self.config.lrc_file_path,
|
||||
mp3_file=mp3_path,
|
||||
output_zip_path=txt_zip_path,
|
||||
)
|
||||
|
||||
self.result.final_karaoke_txt_zip = zip_file
|
||||
self.job_log.info(f"TXT package created: {zip_file}")
|
||||
|
||||
except Exception as e:
|
||||
self.job_log.error(f"TXT generation failed: {e}")
|
||||
# Don't fail the pipeline, TXT is optional
|
||||
|
||||
async def _run_encoding(self):
|
||||
"""Run the encoding stage."""
|
||||
self.job_log.info("Starting encoding stage")
|
||||
self._update_progress(JobStatus.ENCODING, 75, "Encoding videos")
|
||||
|
||||
encoding_backend = self._get_encoding_backend()
|
||||
self.job_log.info(f"Using encoding backend: {encoding_backend.name}")
|
||||
|
||||
# Build encoding input
|
||||
from backend.services.encoding_interface import EncodingInput
|
||||
from backend.config import settings
|
||||
|
||||
# For GCE encoding, we need to provide GCS paths
|
||||
gcs_bucket = settings.gcs_bucket_name
|
||||
input_gcs_path = f"gs://{gcs_bucket}/jobs/{self.config.job_id}/"
|
||||
output_gcs_path = f"gs://{gcs_bucket}/jobs/{self.config.job_id}/finals/"
|
||||
|
||||
encoding_input = EncodingInput(
|
||||
title_video_path=self.config.title_video_path,
|
||||
karaoke_video_path=self.config.karaoke_video_path,
|
||||
instrumental_audio_path=self.config.instrumental_audio_path,
|
||||
end_video_path=self.config.end_video_path,
|
||||
artist=self.config.artist,
|
||||
title=self.config.title,
|
||||
brand_code=self.config.keep_brand_code,
|
||||
output_dir=self.config.output_dir,
|
||||
options={
|
||||
"job_id": self.config.job_id,
|
||||
"input_gcs_path": input_gcs_path,
|
||||
"output_gcs_path": output_gcs_path,
|
||||
},
|
||||
)
|
||||
|
||||
# Run encoding
|
||||
with job_span("encoding", self.config.job_id) as span:
|
||||
add_span_event("encoding_started", {"backend": encoding_backend.name})
|
||||
|
||||
output = await encoding_backend.encode(encoding_input)
|
||||
|
||||
add_span_event("encoding_completed", {
|
||||
"success": output.success,
|
||||
"duration": output.encoding_time_seconds or 0
|
||||
})
|
||||
|
||||
if not output.success:
|
||||
raise Exception(f"Encoding failed: {output.error_message}")
|
||||
|
||||
# Store results - for GCE backend, these are GCS blob paths that need to be downloaded
|
||||
self.result.final_video = output.lossless_4k_mp4_path
|
||||
self.result.final_video_mkv = output.lossless_mkv_path
|
||||
self.result.final_video_lossy = output.lossy_4k_mp4_path
|
||||
self.result.final_video_720p = output.lossy_720p_mp4_path
|
||||
self.result.encoding_time_seconds = output.encoding_time_seconds
|
||||
|
||||
# For GCE encoding, download the encoded files from GCS to local directory
|
||||
# This is required for YouTube upload and other local file operations
|
||||
if encoding_backend.name == "gce" and self.storage:
|
||||
await self._download_gce_encoded_files(output)
|
||||
|
||||
self.job_log.info(f"Encoding complete ({encoding_backend.name}) in {output.encoding_time_seconds:.1f}s")
|
||||
|
||||
async def _download_gce_encoded_files(self, output):
|
||||
"""
|
||||
Download GCE-encoded files from GCS to the local output directory.
|
||||
|
||||
GCE encoding stores files in GCS and returns blob paths like:
|
||||
'jobs/{job_id}/finals/Artist - Title (Final Karaoke Lossless 4k).mp4'
|
||||
|
||||
This method downloads those files locally so that subsequent stages
|
||||
(YouTube upload, etc.) can access them as local files.
|
||||
|
||||
Args:
|
||||
output: EncodingOutput from the GCE backend with GCS blob paths
|
||||
"""
|
||||
self.job_log.info("Downloading GCE-encoded files from GCS")
|
||||
|
||||
# Map of result attributes to download
|
||||
file_mappings = [
|
||||
('lossless_4k_mp4_path', 'final_video'),
|
||||
('lossless_mkv_path', 'final_video_mkv'),
|
||||
('lossy_4k_mp4_path', 'final_video_lossy'),
|
||||
('lossy_720p_mp4_path', 'final_video_720p'),
|
||||
]
|
||||
|
||||
downloaded_count = 0
|
||||
for output_attr, result_attr in file_mappings:
|
||||
gcs_path = getattr(output, output_attr, None)
|
||||
if not gcs_path:
|
||||
continue
|
||||
|
||||
# Extract filename from GCS path
|
||||
filename = os.path.basename(gcs_path)
|
||||
local_path = os.path.join(self.config.output_dir, filename)
|
||||
|
||||
try:
|
||||
self.job_log.info(f"Downloading {filename} from GCS")
|
||||
self.storage.download_file(gcs_path, local_path)
|
||||
|
||||
# Update the result to point to local file
|
||||
setattr(self.result, result_attr, local_path)
|
||||
downloaded_count += 1
|
||||
self.job_log.info(f"Downloaded {filename} to {local_path}")
|
||||
|
||||
except Exception as e:
|
||||
self.job_log.error(f"Failed to download {filename}: {e}")
|
||||
# Clear the result attribute so downstream doesn't try to use invalid GCS path
|
||||
setattr(self.result, result_attr, None)
|
||||
# Don't fail - some formats might not be generated
|
||||
|
||||
self.job_log.info(f"Downloaded {downloaded_count} encoded files from GCS")
|
||||
|
||||
async def _run_organization(self):
|
||||
"""Run the organization stage (brand code generation)."""
|
||||
self.job_log.info("Starting organization stage")
|
||||
|
||||
# Use existing brand code if provided
|
||||
if self.config.keep_brand_code:
|
||||
self.result.brand_code = self.config.keep_brand_code
|
||||
self.job_log.info(f"Using preserved brand code: {self.result.brand_code}")
|
||||
return
|
||||
|
||||
# Generate brand code from Dropbox if configured
|
||||
if self.config.dropbox_path and self.config.brand_prefix:
|
||||
try:
|
||||
from backend.services.dropbox_service import get_dropbox_service
|
||||
|
||||
dropbox = get_dropbox_service()
|
||||
if dropbox.is_configured:
|
||||
brand_code = dropbox.get_next_brand_code(
|
||||
self.config.dropbox_path,
|
||||
self.config.brand_prefix
|
||||
)
|
||||
self.result.brand_code = brand_code
|
||||
self.job_log.info(f"Generated brand code: {brand_code}")
|
||||
else:
|
||||
self.job_log.warning("Dropbox not configured, skipping brand code generation")
|
||||
|
||||
except Exception as e:
|
||||
self.job_log.error(f"Brand code generation failed: {e}")
|
||||
# Don't fail - brand code is optional
|
||||
|
||||
async def _run_distribution(self):
|
||||
"""Run the distribution stage (YouTube, Dropbox, GDrive uploads)."""
|
||||
self.job_log.info("Starting distribution stage")
|
||||
self._update_progress(JobStatus.PACKAGING, 90, "Uploading files")
|
||||
|
||||
# YouTube upload
|
||||
if self.config.enable_youtube_upload and self.config.youtube_credentials:
|
||||
await self._upload_to_youtube()
|
||||
|
||||
# Dropbox upload
|
||||
if self.config.dropbox_path and self.config.brand_prefix:
|
||||
await self._upload_to_dropbox()
|
||||
|
||||
# Google Drive upload
|
||||
if self.config.gdrive_folder_id:
|
||||
await self._upload_to_gdrive()
|
||||
|
||||
async def _upload_to_youtube(self):
|
||||
"""Upload video to YouTube."""
|
||||
self.job_log.info("Uploading to YouTube")
|
||||
|
||||
# Find the best video file to upload (prefer MKV for FLAC audio, then lossless MP4)
|
||||
video_to_upload = None
|
||||
if self.result.final_video_mkv and os.path.isfile(self.result.final_video_mkv):
|
||||
video_to_upload = self.result.final_video_mkv
|
||||
elif self.result.final_video and os.path.isfile(self.result.final_video):
|
||||
video_to_upload = self.result.final_video
|
||||
elif self.result.final_video_lossy and os.path.isfile(self.result.final_video_lossy):
|
||||
video_to_upload = self.result.final_video_lossy
|
||||
|
||||
if not video_to_upload:
|
||||
self.job_log.warning("No video file available for YouTube upload")
|
||||
return
|
||||
|
||||
try:
|
||||
youtube_service = self._get_youtube_service()
|
||||
|
||||
# Build video title
|
||||
title = f"{self.config.artist} - {self.config.title} (Karaoke)"
|
||||
|
||||
# Build description
|
||||
description = self.config.youtube_description_template or ""
|
||||
if self.result.brand_code:
|
||||
description = f"{description}\n\nBrand Code: {self.result.brand_code}".strip()
|
||||
|
||||
# Upload
|
||||
video_id, video_url = youtube_service.upload_video(
|
||||
video_path=video_to_upload,
|
||||
title=title,
|
||||
description=description,
|
||||
thumbnail_path=self.config.title_jpg_path,
|
||||
tags=["karaoke", self.config.artist, self.config.title],
|
||||
replace_existing=True, # Server-side always replaces
|
||||
)
|
||||
|
||||
if video_url:
|
||||
self.result.youtube_url = video_url
|
||||
self.job_log.info(f"Uploaded to YouTube: {video_url}")
|
||||
else:
|
||||
self.job_log.warning("YouTube upload did not return a URL")
|
||||
|
||||
except Exception as e:
|
||||
self.job_log.error(f"YouTube upload failed: {e}")
|
||||
# Don't fail the pipeline - YouTube is optional
|
||||
|
||||
async def _upload_to_dropbox(self):
|
||||
"""Upload files to Dropbox."""
|
||||
self.job_log.info("Uploading to Dropbox")
|
||||
|
||||
try:
|
||||
from backend.services.dropbox_service import get_dropbox_service
|
||||
|
||||
dropbox = get_dropbox_service()
|
||||
if not dropbox.is_configured:
|
||||
self.job_log.warning("Dropbox not configured, skipping upload")
|
||||
return
|
||||
|
||||
# Sanitize artist/title to handle Unicode characters (curly quotes, em dashes, etc.)
|
||||
safe_artist = sanitize_filename(self.config.artist) if self.config.artist else "Unknown"
|
||||
safe_title = sanitize_filename(self.config.title) if self.config.title else "Unknown"
|
||||
base_name = f"{safe_artist} - {safe_title}"
|
||||
folder_name = f"{self.result.brand_code or 'TRACK-0000'} - {base_name}"
|
||||
remote_folder = f"{self.config.dropbox_path}/{folder_name}"
|
||||
|
||||
# Upload entire output directory
|
||||
dropbox.upload_folder(self.config.output_dir, remote_folder)
|
||||
|
||||
# Create sharing link
|
||||
try:
|
||||
sharing_link = dropbox.create_shared_link(remote_folder)
|
||||
self.result.dropbox_link = sharing_link
|
||||
self.job_log.info(f"Dropbox sharing link: {sharing_link}")
|
||||
except Exception as e:
|
||||
self.job_log.warning(f"Failed to create Dropbox sharing link: {e}")
|
||||
|
||||
self.job_log.info("Dropbox upload complete")
|
||||
|
||||
except Exception as e:
|
||||
self.job_log.error(f"Dropbox upload failed: {e}")
|
||||
# Don't fail the pipeline - Dropbox is optional
|
||||
|
||||
async def _upload_to_gdrive(self):
|
||||
"""Upload files to Google Drive."""
|
||||
self.job_log.info("Uploading to Google Drive")
|
||||
|
||||
try:
|
||||
from backend.services.gdrive_service import get_gdrive_service
|
||||
|
||||
gdrive = get_gdrive_service()
|
||||
if not gdrive.is_configured:
|
||||
self.job_log.warning("Google Drive not configured, skipping upload")
|
||||
return
|
||||
|
||||
base_name = f"{self.config.artist} - {self.config.title}"
|
||||
brand_code = self.result.brand_code or f"{self.config.brand_prefix or 'TRACK'}-0000"
|
||||
|
||||
# Map result files to expected keys
|
||||
output_files = {
|
||||
'final_karaoke_lossy_mp4': self.result.final_video_lossy,
|
||||
'final_karaoke_lossy_720p_mp4': self.result.final_video_720p,
|
||||
'final_karaoke_cdg_zip': self.result.final_karaoke_cdg_zip,
|
||||
}
|
||||
|
||||
uploaded = gdrive.upload_to_public_share(
|
||||
root_folder_id=self.config.gdrive_folder_id,
|
||||
brand_code=brand_code,
|
||||
base_name=base_name,
|
||||
output_files=output_files,
|
||||
)
|
||||
|
||||
self.result.gdrive_files = uploaded
|
||||
self.job_log.info(f"Google Drive upload complete: {len(uploaded)} files")
|
||||
|
||||
except Exception as e:
|
||||
self.job_log.error(f"Google Drive upload failed: {e}")
|
||||
# Don't fail the pipeline - GDrive is optional
|
||||
|
||||
async def _run_notifications(self):
|
||||
"""Run the notifications stage (Discord)."""
|
||||
self.job_log.info("Starting notifications stage")
|
||||
|
||||
if not self.config.discord_webhook_url:
|
||||
self.job_log.debug("No Discord webhook configured, skipping notification")
|
||||
return
|
||||
|
||||
if not self.result.youtube_url:
|
||||
self.job_log.info("No YouTube URL available, skipping Discord notification")
|
||||
return
|
||||
|
||||
try:
|
||||
discord_service = self._get_discord_service()
|
||||
discord_service.post_video_notification(self.result.youtube_url)
|
||||
self.job_log.info("Discord notification sent")
|
||||
except Exception as e:
|
||||
self.job_log.error(f"Discord notification failed: {e}")
|
||||
# Don't fail the pipeline - notifications are optional
|
||||
|
||||
|
||||
def create_orchestrator_config_from_job(
|
||||
job,
|
||||
temp_dir: str,
|
||||
youtube_credentials: Optional[Dict[str, Any]] = None,
|
||||
cdg_styles: Optional[Dict[str, Any]] = None,
|
||||
) -> OrchestratorConfig:
|
||||
"""
|
||||
Create an OrchestratorConfig from a job object.
|
||||
|
||||
This is a helper function to bridge the existing job structure
|
||||
with the new orchestrator configuration.
|
||||
|
||||
Args:
|
||||
job: Job object from Firestore
|
||||
temp_dir: Temporary directory with downloaded files
|
||||
youtube_credentials: Pre-loaded YouTube credentials
|
||||
cdg_styles: CDG style configuration
|
||||
|
||||
Returns:
|
||||
OrchestratorConfig for the orchestrator
|
||||
"""
|
||||
# Sanitize artist/title to handle Unicode characters (curly quotes, em dashes, etc.)
|
||||
safe_artist = sanitize_filename(job.artist) if job.artist else "Unknown"
|
||||
safe_title = sanitize_filename(job.title) if job.title else "Unknown"
|
||||
base_name = f"{safe_artist} - {safe_title}"
|
||||
|
||||
# Determine instrumental file path
|
||||
instrumental_selection = job.state_data.get('instrumental_selection', 'clean')
|
||||
existing_instrumental = getattr(job, 'existing_instrumental_gcs_path', None)
|
||||
|
||||
if existing_instrumental:
|
||||
ext = Path(existing_instrumental).suffix.lower()
|
||||
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental User){ext}")
|
||||
else:
|
||||
instrumental_suffix = "Clean" if instrumental_selection == 'clean' else "Backing"
|
||||
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental {instrumental_suffix}).flac")
|
||||
|
||||
return OrchestratorConfig(
|
||||
job_id=job.job_id,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
|
||||
# Input files
|
||||
title_video_path=os.path.join(temp_dir, f"{base_name} (Title).mov"),
|
||||
karaoke_video_path=os.path.join(temp_dir, f"{base_name} (With Vocals).mov"),
|
||||
instrumental_audio_path=instrumental_path,
|
||||
end_video_path=os.path.join(temp_dir, f"{base_name} (End).mov"),
|
||||
lrc_file_path=os.path.join(temp_dir, f"{base_name} (Karaoke).lrc"),
|
||||
title_jpg_path=os.path.join(temp_dir, f"{base_name} (Title).jpg"),
|
||||
|
||||
# Output directory
|
||||
output_dir=temp_dir,
|
||||
|
||||
# Feature flags
|
||||
enable_cdg=getattr(job, 'enable_cdg', False),
|
||||
enable_txt=getattr(job, 'enable_txt', False),
|
||||
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
|
||||
|
||||
# Service configurations
|
||||
brand_prefix=getattr(job, 'brand_prefix', None),
|
||||
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
|
||||
youtube_credentials=youtube_credentials,
|
||||
youtube_description_template=getattr(job, 'youtube_description_template', None),
|
||||
cdg_styles=cdg_styles,
|
||||
|
||||
# Dropbox/GDrive
|
||||
dropbox_path=getattr(job, 'dropbox_path', None),
|
||||
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
|
||||
|
||||
# Keep existing brand code
|
||||
keep_brand_code=getattr(job, 'keep_brand_code', None),
|
||||
|
||||
# Encoding backend - auto selects GCE if available
|
||||
encoding_backend="auto",
|
||||
|
||||
# Server-side defaults
|
||||
dry_run=False,
|
||||
non_interactive=True,
|
||||
)
|
||||
+1
-1
@@ -75,7 +75,7 @@ steps:
|
||||
--concurrency 1 \
|
||||
--max-instances 50 \
|
||||
--min-instances 0 \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=$PROJECT_ID,GCS_BUCKET_NAME=karaoke-gen-storage-$PROJECT_ID,FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD" \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=$PROJECT_ID,GCS_BUCKET_NAME=karaoke-gen-storage-$PROJECT_ID,FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_CLOUD_RUN_JOBS_FOR_VIDEO=true" \
|
||||
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest"
|
||||
|
||||
images:
|
||||
|
||||
+479
-3
@@ -20,6 +20,30 @@ GET /health
|
||||
|
||||
No auth required. Returns service status.
|
||||
|
||||
#### Encoding Worker Health
|
||||
|
||||
```http
|
||||
GET /health/encoding-worker
|
||||
```
|
||||
|
||||
No auth required. Returns GCE encoding worker status for frontend display.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"available": true,
|
||||
"status": "ok",
|
||||
"version": "0.95.1",
|
||||
"active_jobs": 0,
|
||||
"queue_length": 0
|
||||
}
|
||||
```
|
||||
|
||||
Status values:
|
||||
- `ok` - Worker is healthy and available
|
||||
- `offline` - Worker is unavailable (includes `error` field)
|
||||
- `not_configured` - Encoding worker not configured on backend
|
||||
|
||||
### Jobs
|
||||
|
||||
#### Create Job with Upload
|
||||
@@ -53,8 +77,14 @@ Response includes:
|
||||
GET /api/jobs
|
||||
GET /api/jobs?status=complete
|
||||
GET /api/jobs?limit=10&offset=0
|
||||
GET /api/jobs?exclude_test=false
|
||||
```
|
||||
|
||||
Query parameters:
|
||||
- `exclude_test` (bool, default: true) - Admin only: filter out jobs from test users
|
||||
- `status` - Filter by job status
|
||||
- `limit` / `offset` - Pagination
|
||||
|
||||
#### Delete Job
|
||||
|
||||
```http
|
||||
@@ -114,11 +144,52 @@ Content-Type: application/json
|
||||
|
||||
### Audio Search
|
||||
|
||||
#### Search for Audio
|
||||
|
||||
```http
|
||||
GET /api/audio-search?q=song+name
|
||||
POST /api/audio-search/search
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"artist": "Artist Name",
|
||||
"title": "Song Title",
|
||||
"auto_download": false,
|
||||
"theme_id": "nomad",
|
||||
"display_artist": "Display Artist (optional)",
|
||||
"display_title": "Display Title (optional)"
|
||||
}
|
||||
```
|
||||
|
||||
Search for songs (flacfetch integration).
|
||||
Searches for audio sources (flacfetch integration). Creates a job and returns search results.
|
||||
|
||||
**Key fields:**
|
||||
- `artist`, `title` - Used to search for audio on trackers
|
||||
- `display_artist`, `display_title` - Optional overrides for how artist/title appear on title screens and filenames. If omitted, search values are used.
|
||||
- `auto_download` - If true, automatically selects best result and starts processing
|
||||
- `theme_id` - Video theme to use
|
||||
|
||||
**Use case:** Search for "Jeremy Kushnier" but display "Footloose (Broadway Cast)" on the karaoke video.
|
||||
|
||||
#### Get Search Results
|
||||
|
||||
```http
|
||||
GET /api/audio-search/{job_id}/results
|
||||
```
|
||||
|
||||
Returns cached search results for a job awaiting audio selection.
|
||||
|
||||
#### Select Audio Source
|
||||
|
||||
```http
|
||||
POST /api/audio-search/{job_id}/select
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"selection_index": 0
|
||||
}
|
||||
```
|
||||
|
||||
Selects an audio source from the search results and starts processing.
|
||||
|
||||
### Themes
|
||||
|
||||
@@ -174,10 +245,415 @@ Common status codes:
|
||||
- `404` - Job not found
|
||||
- `500` - Server error
|
||||
|
||||
## User Authentication
|
||||
|
||||
### Send Magic Link
|
||||
|
||||
```http
|
||||
POST /api/users/auth/magic-link
|
||||
Content-Type: application/json
|
||||
|
||||
{"email": "user@example.com"}
|
||||
```
|
||||
|
||||
### Verify Magic Link
|
||||
|
||||
```http
|
||||
GET /api/users/auth/verify?token=TOKEN
|
||||
```
|
||||
|
||||
Returns session token and user info.
|
||||
|
||||
### Get Current User
|
||||
|
||||
```http
|
||||
GET /api/users/me
|
||||
Authorization: Bearer SESSION_TOKEN
|
||||
```
|
||||
|
||||
### Logout
|
||||
|
||||
```http
|
||||
POST /api/users/auth/logout
|
||||
Authorization: Bearer SESSION_TOKEN
|
||||
```
|
||||
|
||||
## Credits & Payments (PR #90)
|
||||
|
||||
### List Credit Packages
|
||||
|
||||
```http
|
||||
GET /api/users/credits/packages
|
||||
```
|
||||
|
||||
No auth required. Returns available packages and prices.
|
||||
|
||||
### Create Checkout Session
|
||||
|
||||
```http
|
||||
POST /api/users/credits/checkout
|
||||
Content-Type: application/json
|
||||
|
||||
{"package_id": "5_credits", "email": "user@example.com"}
|
||||
```
|
||||
|
||||
Returns Stripe checkout URL.
|
||||
|
||||
### Stripe Webhook
|
||||
|
||||
```http
|
||||
POST /api/users/webhooks/stripe
|
||||
```
|
||||
|
||||
Handles `checkout.session.completed` events.
|
||||
|
||||
## Beta Tester Program (PR #90)
|
||||
|
||||
### Enroll as Beta Tester
|
||||
|
||||
```http
|
||||
POST /api/users/beta/enroll
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"email": "user@example.com",
|
||||
"promise_text": "I promise to provide feedback...",
|
||||
"accept_corrections_work": true
|
||||
}
|
||||
```
|
||||
|
||||
Returns 1 free credit and session token.
|
||||
|
||||
### Submit Feedback
|
||||
|
||||
```http
|
||||
POST /api/users/beta/feedback
|
||||
Authorization: Bearer SESSION_TOKEN
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"overall_rating": 4,
|
||||
"ease_of_use_rating": 5,
|
||||
"lyrics_accuracy_rating": 4,
|
||||
"correction_experience_rating": 3,
|
||||
"what_went_well": "...",
|
||||
"what_could_improve": "..."
|
||||
}
|
||||
```
|
||||
|
||||
Bonus credit for detailed feedback (50+ chars).
|
||||
|
||||
## Admin Endpoints
|
||||
|
||||
All admin endpoints require an admin-role session token.
|
||||
|
||||
### Test Data Filtering
|
||||
|
||||
Most admin list/stats endpoints support an `exclude_test` query parameter (default: `true`) to filter out E2E test data:
|
||||
- Test users: email addresses ending in `@inbox.testmail.app`
|
||||
- Test jobs: jobs created by test users
|
||||
|
||||
```http
|
||||
GET /api/admin/stats/overview?exclude_test=true # Default - hide test data
|
||||
GET /api/admin/stats/overview?exclude_test=false # Show all data including tests
|
||||
```
|
||||
|
||||
The frontend admin dashboard includes a "Show test data" toggle that controls this filter globally.
|
||||
|
||||
### Admin Dashboard Stats
|
||||
|
||||
```http
|
||||
GET /api/admin/stats/overview
|
||||
GET /api/admin/stats/overview?exclude_test=false
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Returns platform statistics:
|
||||
```json
|
||||
{
|
||||
"total_users": 100,
|
||||
"active_users_7d": 25,
|
||||
"active_users_30d": 60,
|
||||
"total_jobs": 500,
|
||||
"jobs_last_7d": 50,
|
||||
"jobs_last_30d": 150,
|
||||
"jobs_by_status": {
|
||||
"pending": 5,
|
||||
"processing": 3,
|
||||
"awaiting_review": 10,
|
||||
"complete": 400,
|
||||
"failed": 20
|
||||
},
|
||||
"total_credits_issued_30d": 200,
|
||||
"total_beta_testers": 30
|
||||
}
|
||||
```
|
||||
|
||||
### List Users (Admin)
|
||||
|
||||
```http
|
||||
GET /api/users/admin/users
|
||||
GET /api/users/admin/users?search=user@example.com
|
||||
GET /api/users/admin/users?limit=20&offset=0&sort_by=created_at&sort_order=desc
|
||||
GET /api/users/admin/users?exclude_test=false
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Query parameters:
|
||||
- `exclude_test` (bool, default: true) - Filter out test users
|
||||
- `limit` / `offset` - Pagination
|
||||
- `search` - Email prefix search
|
||||
- `sort_by` / `sort_order` - Sorting
|
||||
|
||||
Returns paginated user list with total count.
|
||||
|
||||
### User Detail (Admin)
|
||||
|
||||
```http
|
||||
GET /api/users/admin/users/{email}/detail
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Returns full user profile including:
|
||||
- User info and stats
|
||||
- Recent credit transactions (last 20)
|
||||
- Recent jobs (last 10)
|
||||
- Active sessions count
|
||||
|
||||
### Add Credits (Admin)
|
||||
|
||||
```http
|
||||
POST /api/users/admin/credits
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"email": "user@example.com",
|
||||
"amount": 5,
|
||||
"reason": "Beta reward"
|
||||
}
|
||||
```
|
||||
|
||||
### Set User Role (Admin)
|
||||
|
||||
```http
|
||||
POST /api/users/admin/users/{email}/role
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
Content-Type: application/json
|
||||
|
||||
{"role": "admin"}
|
||||
```
|
||||
|
||||
### Enable/Disable User (Admin)
|
||||
|
||||
```http
|
||||
POST /api/users/admin/users/{email}/enable
|
||||
POST /api/users/admin/users/{email}/disable
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
### Beta Stats (Admin)
|
||||
|
||||
```http
|
||||
GET /api/users/admin/beta/stats
|
||||
GET /api/users/admin/beta/stats?exclude_test=false
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Query parameters:
|
||||
- `exclude_test` (bool, default: true) - Filter out test users from beta statistics
|
||||
|
||||
### Beta Feedback List (Admin)
|
||||
|
||||
```http
|
||||
GET /api/users/admin/beta/feedback?limit=50
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
### Delete Job (Admin)
|
||||
|
||||
```http
|
||||
DELETE /api/jobs/{job_id}?delete_files=true
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Admins can delete any job. Regular users can only delete their own jobs.
|
||||
|
||||
### Audio Search Management (Admin)
|
||||
|
||||
#### List Audio Searches
|
||||
|
||||
```http
|
||||
GET /api/admin/audio-searches
|
||||
GET /api/admin/audio-searches?limit=50&status_filter=awaiting_audio_selection
|
||||
GET /api/admin/audio-searches?exclude_test=false
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Query parameters:
|
||||
- `exclude_test` (bool, default: true) - Filter out searches from test users
|
||||
- `limit` - Max results to return
|
||||
- `status_filter` - Filter by job status
|
||||
|
||||
Returns jobs with cached audio search results. Useful for:
|
||||
- Monitoring search activity
|
||||
- Identifying stale cached results (YouTube-only when lossless should be available)
|
||||
- Clearing cache for specific jobs
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"jobs": [
|
||||
{
|
||||
"job_id": "abc123",
|
||||
"status": "awaiting_audio_selection",
|
||||
"user_email": "user@example.com",
|
||||
"audio_search_artist": "Artist Name",
|
||||
"audio_search_title": "Song Title",
|
||||
"created_at": "2026-01-03T12:00:00Z",
|
||||
"results_count": 5,
|
||||
"has_lossless": false,
|
||||
"providers": ["YouTube"],
|
||||
"results_summary": [...]
|
||||
}
|
||||
],
|
||||
"total": 1
|
||||
}
|
||||
```
|
||||
|
||||
#### Clear Audio Search Cache
|
||||
|
||||
```http
|
||||
POST /api/admin/audio-searches/{job_id}/clear-cache
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Clears cached search results and resets job to `pending` status, allowing a new search.
|
||||
|
||||
Use when:
|
||||
- Cached results are stale (e.g., flacfetch was updated with new providers)
|
||||
- User wants to search again
|
||||
- Results appear incomplete (YouTube-only when lossless should exist)
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"status": "success",
|
||||
"job_id": "abc123",
|
||||
"message": "Cleared 5 cached search results. Job reset to pending. Flacfetch cache also cleared.",
|
||||
"previous_status": "awaiting_audio_selection",
|
||||
"new_status": "pending",
|
||||
"results_cleared": 5,
|
||||
"flacfetch_cache_cleared": true,
|
||||
"flacfetch_error": null
|
||||
}
|
||||
```
|
||||
|
||||
Note: This endpoint now also clears the flacfetch GCS cache for the artist/title combination, ensuring the next search hits trackers fresh.
|
||||
|
||||
#### Clear All Flacfetch Cache
|
||||
|
||||
```http
|
||||
DELETE /api/admin/cache
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Clears the entire flacfetch search cache (GCS-backed). Use when flacfetch has been updated and you want all subsequent searches to use fresh tracker results.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"status": "success",
|
||||
"message": "Cleared 15 cache entries from flacfetch.",
|
||||
"deleted_count": 15
|
||||
}
|
||||
```
|
||||
|
||||
#### Get Flacfetch Cache Stats
|
||||
|
||||
```http
|
||||
GET /api/admin/cache/stats
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Returns statistics about the flacfetch search cache.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"count": 42,
|
||||
"total_size_bytes": 128000,
|
||||
"oldest_entry": "2025-12-15T10:30:00Z",
|
||||
"newest_entry": "2026-01-03T15:45:00Z",
|
||||
"configured": true
|
||||
}
|
||||
```
|
||||
|
||||
### Email Notifications (Admin)
|
||||
|
||||
#### Get Completion Message
|
||||
|
||||
```http
|
||||
GET /api/admin/jobs/{job_id}/completion-message
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Returns the rendered completion message for a job (for copying to clipboard).
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"job_id": "abc123",
|
||||
"message": "Hi there! Your karaoke video is ready...",
|
||||
"subject": "Your Karaoke Video is Ready! 🎤",
|
||||
"youtube_url": "https://youtube.com/watch?v=...",
|
||||
"dropbox_url": "https://dropbox.com/..."
|
||||
}
|
||||
```
|
||||
|
||||
#### Send Completion Email
|
||||
|
||||
```http
|
||||
POST /api/admin/jobs/{job_id}/send-completion-email
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"to_email": "customer@example.com",
|
||||
"cc_admin": true
|
||||
}
|
||||
```
|
||||
|
||||
Sends the job completion email to the specified address. When `cc_admin` is true (default), CCs gen@nomadkaraoke.com.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"job_id": "abc123",
|
||||
"to_email": "customer@example.com",
|
||||
"message": "Completion email sent successfully"
|
||||
}
|
||||
```
|
||||
|
||||
### Internal Email Endpoints
|
||||
|
||||
These endpoints are used by Cloud Tasks for automated notifications.
|
||||
|
||||
#### Check Idle Reminder
|
||||
|
||||
```http
|
||||
POST /api/internal/jobs/{job_id}/check-idle-reminder
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Called by Cloud Tasks 5 minutes after a job enters a blocking state (awaiting_review or awaiting_instrumental_selection). Sends a reminder email if the user is still idle and no reminder has been sent yet.
|
||||
|
||||
## Rate Limits
|
||||
|
||||
No rate limits currently implemented.
|
||||
|
||||
## Webhooks
|
||||
|
||||
Not yet implemented. Jobs must be polled for status.
|
||||
Stripe webhooks implemented at `/api/users/webhooks/stripe`.
|
||||
Job status webhooks not yet implemented.
|
||||
+75
-7
@@ -23,12 +23,17 @@
|
||||
│ • Video Worker - Final assembly and encoding │
|
||||
└───────────────────────────┬─────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────┼─────────────┐
|
||||
▼ ▼ ▼
|
||||
┌──────────┐ ┌──────────┐ ┌──────────────┐
|
||||
│ Firestore│ │ GCS │ │Secret Manager│
|
||||
│ (jobs) │ │ (files) │ │ (API keys) │
|
||||
└──────────┘ └──────────┘ └──────────────┘
|
||||
┌─────────────┼─────────────┬───────────────┐
|
||||
▼ ▼ ▼ ▼
|
||||
┌──────────┐ ┌──────────┐ ┌──────────────┐ ┌──────────────┐
|
||||
│ Firestore│ │ GCS │ │Secret Manager│ │GCE Encoding │
|
||||
│ (jobs) │ │ (files) │ │ (API keys) │ │ Worker (*) │
|
||||
└──────────┘ └──────────┘ └──────────────┘ └──────────────┘
|
||||
|
||||
(*) GCE Encoding Worker: c4d-highcpu-32 VM with AMD EPYC 9B45 (Turin) for
|
||||
high-performance FFmpeg encoding (4.92x faster than previous c4-standard-8).
|
||||
Used for both final video encoding and preview video generation.
|
||||
Uses LocalEncodingService via GCS wheel deployment.
|
||||
```
|
||||
|
||||
## Processing Pipeline
|
||||
@@ -54,7 +59,7 @@
|
||||
▼
|
||||
4. HUMAN REVIEW (AWAITING_REVIEW)
|
||||
• React UI for lyrics correction
|
||||
• Preview video generation
|
||||
• Preview video generation (optionally via GCE for speed)
|
||||
• Submit corrections
|
||||
▼
|
||||
5. VIDEO RENDERING
|
||||
@@ -148,8 +153,69 @@ LyricsTranscriber LyricsTranscriber
|
||||
|---------|---------|----------|
|
||||
| Modal | GPU audio separation | Yes |
|
||||
| AudioShake | Lyrics transcription | Yes |
|
||||
| Vertex AI | Agentic AI correction (Gemini 3 Flash) | Default on |
|
||||
| Genius | Reference lyrics | Yes |
|
||||
| YouTube API | Video upload | Optional |
|
||||
| SendGrid | Email notifications | Optional |
|
||||
| Cloud Tasks | Delayed task scheduling (idle reminders) | Optional |
|
||||
|
||||
## Firestore Collections
|
||||
|
||||
karaoke-gen shares a GCP project (`nomadkaraoke`) with karaoke-decide, but uses separate Firestore collections:
|
||||
|
||||
| Collection | Purpose | Key Fields |
|
||||
|------------|---------|------------|
|
||||
| `gen_users` | karaoke-gen user accounts | email, credits, role, is_active |
|
||||
| `jobs` | Karaoke generation jobs | job_id, user_email, status, state_data |
|
||||
| `jobs/{job_id}/logs` | Worker log entries (subcollection) | timestamp, level, worker, message, ttl_expiry |
|
||||
| `sessions` | Magic link auth sessions | user_email, token, expires_at |
|
||||
| `magic_links` | Passwordless auth tokens | email, token, expires_at, used |
|
||||
| `beta_feedback` | Beta program feedback | user_email, ratings, comments |
|
||||
|
||||
**Note**: The `users` collection in the same Firestore instance belongs to karaoke-decide (different schema: user_id, is_guest, quiz_* fields). Don't use it for karaoke-gen.
|
||||
|
||||
**Worker Logs**: Stored in subcollection (`jobs/{job_id}/logs`) instead of embedded array to avoid Firestore's 1MB document size limit. TTL policy auto-deletes logs after 30 days. Feature flag: `USE_LOG_SUBCOLLECTION` (default: true). See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#firestore-document-1mb-limit-with-embedded-arrays).
|
||||
|
||||
## Video Worker Orchestrator
|
||||
|
||||
The Video Worker uses an orchestrator pattern to ensure all features work regardless of encoding backend (local Cloud Run or GCE VM).
|
||||
|
||||
```text
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ VideoWorkerOrchestrator │
|
||||
│ │
|
||||
│ Stage 1: PACKAGING Stage 2: ENCODING │
|
||||
│ ├─ CDG file generation ├─ Local (FFmpeg) │
|
||||
│ └─ TXT lyrics export └─ GCE (remote VM) │
|
||||
│ │
|
||||
│ Stage 3: ORGANIZATION Stage 4: DISTRIBUTION │
|
||||
│ ├─ Final path setup ├─ Dropbox upload │
|
||||
│ └─ File organization ├─ Google Drive upload │
|
||||
│ └─ YouTube upload │
|
||||
│ │
|
||||
│ Stage 5: NOTIFICATIONS │
|
||||
│ ├─ Discord webhooks │
|
||||
│ └─ Email notifications (completion, reminders) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Key Design Decision**: The orchestrator coordinates stages rather than embedding logic in monolithic functions. This:
|
||||
- Ensures all features work with both local and GCE encoding
|
||||
- Makes each stage independently testable (139 new tests)
|
||||
- Allows feature flag rollback via `USE_NEW_ORCHESTRATOR` env var
|
||||
|
||||
**Service Modules**:
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `youtube_upload_service.py` | YouTube OAuth flow and upload |
|
||||
| `discord_service.py` | Discord webhook notifications |
|
||||
| `packaging_service.py` | CDG/TXT package generation |
|
||||
| `local_encoding_service.py` | FFmpeg encoding (Cloud Run) |
|
||||
| `encoding_interface.py` | Abstract interface + GCE backend |
|
||||
| `video_worker_orchestrator.py` | Stage coordination |
|
||||
| `email_service.py` | SendGrid email delivery with CC support |
|
||||
| `template_service.py` | GCS-backed email templates |
|
||||
| `job_notification_service.py` | Email orchestration (completion, reminders) |
|
||||
|
||||
## Tech Stack
|
||||
|
||||
@@ -158,5 +224,7 @@ LyricsTranscriber LyricsTranscriber
|
||||
- **Database**: Firestore
|
||||
- **Storage**: Google Cloud Storage
|
||||
- **Secrets**: Google Secret Manager
|
||||
- **Task Queues**: Cloud Tasks (idle reminders)
|
||||
- **Email**: SendGrid
|
||||
- **IaC**: Pulumi
|
||||
- **CI/CD**: GitHub Actions
|
||||
@@ -0,0 +1,709 @@
|
||||
# Nomad Karaoke Brand Style Guide
|
||||
|
||||
> **Version**: 1.2
|
||||
> **Last Updated**: 2026-01-08
|
||||
> **Purpose**: Unified brand identity for all Nomad Karaoke products and communications
|
||||
|
||||
This style guide is designed for both human designers and LLM agents implementing the Nomad Karaoke brand across web applications, emails, marketing materials, and other touchpoints.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Brand Overview](#brand-overview)
|
||||
2. [Color Palette](#color-palette)
|
||||
3. [Typography](#typography)
|
||||
4. [Logo Usage](#logo-usage)
|
||||
5. [Iconography](#iconography)
|
||||
6. [UI Components](#ui-components)
|
||||
7. [Layout & Spacing](#layout--spacing)
|
||||
8. [Email Design](#email-design)
|
||||
9. [Tone & Voice](#tone--voice)
|
||||
10. [Light/Dark Theme Requirements](#lightdark-theme-requirements)
|
||||
11. [Implementation Reference](#implementation-reference)
|
||||
|
||||
---
|
||||
|
||||
## Brand Overview
|
||||
|
||||
### Brand Personality
|
||||
Nomad Karaoke is **approachable**, **professional**, and **modern**. We make sophisticated technology feel accessible and fun. Our brand bridges the gap between serious audio engineering and the joy of karaoke.
|
||||
|
||||
### Core Values
|
||||
- **Approachable**: Technology that anyone can use
|
||||
- **Professional**: Studio-quality output
|
||||
- **Playful**: Karaoke should be fun
|
||||
- **Trustworthy**: Reliable service and results
|
||||
|
||||
### Brand Keywords
|
||||
`modern` `vibrant` `accessible` `professional` `musical` `AI-powered` `creative`
|
||||
|
||||
---
|
||||
|
||||
## Color Palette
|
||||
|
||||
### Primary Colors
|
||||
|
||||
| Name | Hex | RGB | Usage |
|
||||
|------|-----|-----|-------|
|
||||
| **Hot Pink** | `#ff5bb8` | rgb(255, 91, 184) | Primary brand color, buttons with white text (better contrast) |
|
||||
| **Hot Pink Light** | `#ff7acc` | rgb(255, 122, 204) | Logo, hover states, accents where contrast is less critical |
|
||||
| **Gold** | `#ffdf6b` | rgb(255, 223, 107) | Secondary accent, highlights, success states, premium feel |
|
||||
|
||||
> **Accessibility Note**: We use the darker pink (`#ff5bb8`) as the default for UI elements with white text because it provides better contrast ratio (WCAG compliance). The lighter pink (`#ff7acc`) is used for hover states and decorative accents where white text contrast is less critical.
|
||||
|
||||
### Supporting Colors
|
||||
|
||||
| Name | Hex | RGB | Usage |
|
||||
|------|-----|-----|-------|
|
||||
| **Deep Purple** | `#8b5cf6` | rgb(139, 92, 246) | Gradients, special features, beta badges |
|
||||
| **Electric Blue** | `#3b82f6` | rgb(59, 130, 246) | Links, interactive elements, info states |
|
||||
| **Soft Pink** | `#ec4899` | rgb(236, 72, 153) | Gradient end points, hover states |
|
||||
|
||||
### Semantic Colors
|
||||
|
||||
| Name | Hex | Usage |
|
||||
|------|-----|-------|
|
||||
| **Success** | `#22c55e` | Completed states, positive feedback |
|
||||
| **Warning** | `#f59e0b` | Attention needed, caution states |
|
||||
| **Error** | `#ef4444` | Errors, destructive actions |
|
||||
| **Info** | `#3b82f6` | Informational messages |
|
||||
|
||||
### Dark Theme (Default)
|
||||
|
||||
| Name | Hex | Usage |
|
||||
|------|-----|-------|
|
||||
| **Background** | `#0f0f0f` | Page background |
|
||||
| **Card Background** | `#1a1a1a` | Card surfaces |
|
||||
| **Card Border** | `#2a2a2a` | Subtle borders |
|
||||
| **Text Primary** | `#e5e5e5` | Main content text |
|
||||
| **Text Muted** | `#888888` | Secondary text, labels |
|
||||
| **Text Subtle** | `#666666` | Tertiary text, hints |
|
||||
|
||||
### Light Theme
|
||||
|
||||
| Name | Hex | Usage |
|
||||
|------|-----|-------|
|
||||
| **Background** | `#f8fafc` | Page background |
|
||||
| **Card Background** | `#ffffff` | Card surfaces |
|
||||
| **Card Border** | `#e2e8f0` | Subtle borders |
|
||||
| **Text Primary** | `#1e293b` | Main content text |
|
||||
| **Text Muted** | `#64748b` | Secondary text, labels |
|
||||
|
||||
### Gradients
|
||||
|
||||
#### Primary Gradient (Brand Gradient)
|
||||
```css
|
||||
background: linear-gradient(135deg, #3b82f6 0%, #8b5cf6 50%, #ff7acc 100%);
|
||||
```
|
||||
Use for: Hero text, feature highlights, premium elements. The gradient flows from blue through purple to the brand pink.
|
||||
|
||||
#### Radial Background Gradient (Landing Pages)
|
||||
```css
|
||||
/* Layered radial gradients using brand colors */
|
||||
body::after {
|
||||
content: '';
|
||||
position: fixed;
|
||||
inset: 0;
|
||||
z-index: -1;
|
||||
background:
|
||||
radial-gradient(ellipse at 20% 0%, rgba(139, 92, 246, 0.12) 0%, transparent 50%), /* Purple glow top-left */
|
||||
radial-gradient(ellipse at 80% 100%, rgba(255, 122, 204, 0.10) 0%, transparent 50%), /* Pink glow bottom-right */
|
||||
radial-gradient(ellipse at 50% 50%, rgba(255, 223, 107, 0.03) 0%, transparent 70%); /* Gold subtle center */
|
||||
pointer-events: none;
|
||||
}
|
||||
|
||||
/* Optional: Add subtle pulsing animation for ambient glow effect */
|
||||
@keyframes ambient-glow {
|
||||
0%, 100% { opacity: 1; }
|
||||
50% { opacity: 0.7; }
|
||||
}
|
||||
body::after {
|
||||
animation: ambient-glow 8s ease-in-out infinite;
|
||||
}
|
||||
```
|
||||
Use for: Landing page backgrounds, hero sections. Creates depth with brand colors without being distracting.
|
||||
|
||||
**Light theme variant**: Reduce opacity values (0.08, 0.06, 0.02) for subtler effect on light backgrounds.
|
||||
|
||||
#### Beta/Special Gradient
|
||||
```css
|
||||
background: linear-gradient(135deg, #8b5cf6, #ec4899);
|
||||
```
|
||||
Use for: Beta badges, special promotions, exclusive features.
|
||||
|
||||
---
|
||||
|
||||
## Typography
|
||||
|
||||
### Font Stack
|
||||
|
||||
#### Primary (Sans-serif)
|
||||
```css
|
||||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;
|
||||
```
|
||||
Use system fonts for optimal performance and native feel.
|
||||
|
||||
#### Monospace (Code/Technical)
|
||||
```css
|
||||
font-family: "SF Mono", Monaco, "Cascadia Code", "Roboto Mono", monospace;
|
||||
```
|
||||
Use for: Code snippets, technical information, job IDs.
|
||||
|
||||
### Type Scale
|
||||
|
||||
| Name | Size | Weight | Line Height | Usage |
|
||||
|------|------|--------|-------------|-------|
|
||||
| **Display** | 48-60px | 800 (extrabold) | 1.1 | Hero headlines |
|
||||
| **H1** | 36-48px | 700 (bold) | 1.2 | Page titles |
|
||||
| **H2** | 28-32px | 700 (bold) | 1.3 | Section headers |
|
||||
| **H3** | 20-24px | 600 (semibold) | 1.4 | Card titles |
|
||||
| **Body** | 16px | 400 (regular) | 1.6 | Main content |
|
||||
| **Body Small** | 14px | 400 (regular) | 1.5 | Secondary content |
|
||||
| **Caption** | 12px | 400 (regular) | 1.4 | Labels, hints |
|
||||
|
||||
### Text Styling
|
||||
|
||||
- **Gradient Text**: Apply brand gradient to hero headlines
|
||||
- **Antialiasing**: Always use `-webkit-font-smoothing: antialiased`
|
||||
- **Letter Spacing**: Default, slightly increased for ALL CAPS text
|
||||
|
||||
---
|
||||
|
||||
## Logo Usage
|
||||
|
||||
**Important**: Always use the Nomad Karaoke logo rather than generic emojis (like 🎤) in headers, navigation, and brand touchpoints.
|
||||
|
||||
### Primary Logo
|
||||
The Nomad Karaoke logo consists of:
|
||||
1. **Wordmark**: "NOMAD" stylized letters (pink #ff7acc)
|
||||
2. **Tagline**: "KARAOKE" below (pink #ff7acc)
|
||||
3. **Subline**: "WHERE EVERY SONG'S YOURS, SINGING!" (gold #ffdf6b)
|
||||
|
||||
### Logo Format Priority
|
||||
|
||||
1. **SVG (Preferred)**: Use `nomad-karaoke-logo.svg` for all web applications
|
||||
- Vector format scales perfectly at any size
|
||||
- Smaller file size
|
||||
- Supports both light and dark themes
|
||||
- Located in: `frontend/public/` directory
|
||||
|
||||
2. **PNG**: Use for contexts that don't support SVG
|
||||
- `nomad-logo.png` - Standard PNG fallback
|
||||
|
||||
3. **GIF (Email Only)**: Use the hosted GIF **only** for email templates
|
||||
- URL: `https://beveradb.github.io/public-images/Nomad-Karaoke-Logo-small-indexed-websafe-rectangle.gif`
|
||||
- This hosted URL exists because emails cannot embed local assets
|
||||
- Do NOT use this URL in web applications - use the local SVG instead
|
||||
|
||||
### Logo Files
|
||||
- `nomad-karaoke-logo.svg` - Full vector logo (**primary choice**)
|
||||
- `nomad-logo.png` - PNG fallback
|
||||
- `favicon.ico`, `favicon-16x16.png`, `favicon-32x32.png` - Favicons
|
||||
- `apple-touch-icon.png` - iOS home screen icon
|
||||
|
||||
### Logo Placement
|
||||
- **Navigation**: Left-aligned, 32-40px height, use SVG
|
||||
- **Footer**: Smaller, 24-32px height, use SVG
|
||||
- **Email Header**: Centered, 130-180px width, use hosted GIF URL
|
||||
|
||||
### Logo Clear Space
|
||||
Maintain clear space equal to the "N" letter height around all sides of the logo.
|
||||
|
||||
### Logo Don'ts
|
||||
- Don't use emojis (🎤, 🎵) instead of the logo
|
||||
- Don't change the logo colors
|
||||
- Don't stretch or distort the logo
|
||||
- Don't add effects (shadows, gradients, etc.)
|
||||
- Don't place on busy backgrounds without contrast
|
||||
- Don't use the logo smaller than 24px height
|
||||
- Don't use the hosted GIF URL in web applications (use SVG)
|
||||
|
||||
---
|
||||
|
||||
## Iconography
|
||||
|
||||
### Icon Library
|
||||
Use [Lucide Icons](https://lucide.dev/) for UI icons. They are:
|
||||
- Open source
|
||||
- Consistent stroke width
|
||||
- Good visual balance
|
||||
|
||||
### Common Icons
|
||||
|
||||
| Icon | Name | Usage |
|
||||
|------|------|-------|
|
||||
| `MicVocal` | Microphone | Brand representation in nav (when logo not used) |
|
||||
| `Music` | Music note | Audio/song related |
|
||||
| `Music2` | Double notes | Empty states for songs |
|
||||
| `Sparkles` | Stars | AI/magic features |
|
||||
| `Video` | Video | Video output |
|
||||
| `Youtube` | YouTube logo | YouTube integration |
|
||||
| `Gift` | Gift box | Free credits, promotions |
|
||||
| `Check` | Checkmark | Success, confirmation |
|
||||
| `Loader2` | Spinner | Loading states |
|
||||
|
||||
### Icon Sizing
|
||||
- **Small**: 16px (inline with text)
|
||||
- **Medium**: 20-24px (buttons, lists)
|
||||
- **Large**: 32-48px (feature highlights)
|
||||
- **XL**: 64px+ (empty states, hero)
|
||||
|
||||
### Icon Colors
|
||||
- Match text color in context
|
||||
- Use brand pink (#ff7acc) for emphasis
|
||||
- Use muted color for decorative icons
|
||||
|
||||
---
|
||||
|
||||
## UI Components
|
||||
|
||||
### Buttons
|
||||
|
||||
#### Primary Button (CTA)
|
||||
```css
|
||||
/* Default state uses darker pink for better white text contrast */
|
||||
background-color: #ff5bb8;
|
||||
color: #ffffff;
|
||||
padding: 14px 28px;
|
||||
border-radius: 12px;
|
||||
font-weight: 600;
|
||||
box-shadow: 0 0 20px rgba(255, 122, 204, 0.4);
|
||||
transition: all 0.2s ease;
|
||||
|
||||
/* Hover - lightens to original brand pink */
|
||||
background-color: #ff7acc;
|
||||
box-shadow: 0 0 30px rgba(255, 122, 204, 0.6);
|
||||
```
|
||||
|
||||
#### Secondary Button
|
||||
```css
|
||||
background-color: #252525;
|
||||
color: #e5e5e5;
|
||||
padding: 12px 24px;
|
||||
border-radius: 8px;
|
||||
border: 1px solid #2a2a2a;
|
||||
|
||||
/* Hover */
|
||||
background-color: #333333;
|
||||
```
|
||||
|
||||
#### Ghost Button
|
||||
```css
|
||||
background-color: transparent;
|
||||
color: #888888;
|
||||
padding: 8px 16px;
|
||||
|
||||
/* Hover */
|
||||
color: #e5e5e5;
|
||||
background-color: rgba(255, 255, 255, 0.05);
|
||||
```
|
||||
|
||||
### Cards
|
||||
|
||||
```css
|
||||
background-color: var(--card); /* #1a1a1a dark / #ffffff light */
|
||||
border: 1px solid var(--card-border); /* #2a2a2a dark / #e2e8f0 light */
|
||||
border-radius: 12px;
|
||||
padding: 24px;
|
||||
|
||||
/* Hover effect (optional) */
|
||||
transition: transform 0.2s ease, box-shadow 0.2s ease;
|
||||
transform: translateY(-4px);
|
||||
box-shadow: 0 20px 40px rgba(0, 0, 0, 0.3);
|
||||
```
|
||||
|
||||
### Form Inputs
|
||||
|
||||
```css
|
||||
background-color: #0f0f0f; /* Darker than card */
|
||||
border: 1px solid #2a2a2a;
|
||||
border-radius: 8px;
|
||||
padding: 12px 16px;
|
||||
color: #e5e5e5;
|
||||
|
||||
/* Focus */
|
||||
border-color: #ff7acc;
|
||||
box-shadow: 0 0 0 2px rgba(255, 122, 204, 0.2);
|
||||
outline: none;
|
||||
|
||||
/* Placeholder */
|
||||
color: #666666;
|
||||
```
|
||||
|
||||
### Badges
|
||||
|
||||
#### Beta Badge
|
||||
```css
|
||||
background: linear-gradient(135deg, #8b5cf6, #ec4899);
|
||||
color: #ffffff;
|
||||
padding: 4px 12px;
|
||||
border-radius: 20px;
|
||||
font-size: 12px;
|
||||
font-weight: 600;
|
||||
```
|
||||
|
||||
#### Status Badges
|
||||
Use semantic colors with 20% opacity backgrounds:
|
||||
- Success: `background: rgba(34, 197, 94, 0.2); color: #22c55e;`
|
||||
- Warning: `background: rgba(245, 158, 11, 0.2); color: #f59e0b;`
|
||||
- Error: `background: rgba(239, 68, 68, 0.2); color: #ef4444;`
|
||||
|
||||
---
|
||||
|
||||
## Layout & Spacing
|
||||
|
||||
### Spacing Scale
|
||||
Use a consistent 4px base unit:
|
||||
|
||||
| Name | Value | Usage |
|
||||
|------|-------|-------|
|
||||
| `xs` | 4px | Tight spacing |
|
||||
| `sm` | 8px | Small gaps |
|
||||
| `md` | 16px | Default gaps |
|
||||
| `lg` | 24px | Section padding |
|
||||
| `xl` | 32px | Large sections |
|
||||
| `2xl` | 48px | Major sections |
|
||||
| `3xl` | 64px | Page sections |
|
||||
|
||||
### Container Widths
|
||||
- **Max content width**: 1200px (6xl)
|
||||
- **Text content**: 672px (2xl) for readability
|
||||
- **Form width**: 448px (md) for focused input
|
||||
|
||||
### Grid
|
||||
Use CSS Grid or Flexbox with responsive breakpoints:
|
||||
- Mobile: 1 column
|
||||
- Tablet (md: 768px): 2 columns
|
||||
- Desktop (lg: 1024px): 2-4 columns
|
||||
|
||||
### Border Radius
|
||||
- **Small** (inputs, small buttons): 8px
|
||||
- **Medium** (cards, buttons): 12px
|
||||
- **Large** (feature cards, modals): 16px
|
||||
- **Pill** (badges, tags): 20px or 9999px
|
||||
|
||||
---
|
||||
|
||||
## Email Design
|
||||
|
||||
### Email Structure
|
||||
|
||||
```html
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<style>
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
|
||||
line-height: 1.6;
|
||||
color: #333333;
|
||||
max-width: 600px;
|
||||
margin: 0 auto;
|
||||
padding: 20px;
|
||||
background-color: #ffffff;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<!-- Header with logo -->
|
||||
<div style="text-align: center; padding: 20px 0;">
|
||||
<img src="logo-url" alt="Nomad Karaoke" width="150" />
|
||||
</div>
|
||||
|
||||
<!-- Content -->
|
||||
<div>...</div>
|
||||
|
||||
<!-- Footer -->
|
||||
<div style="margin-top: 30px; padding-top: 20px; border-top: 1px solid #ff7acc; font-size: 12px; color: #666;">
|
||||
...
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
```
|
||||
|
||||
### Email Colors
|
||||
- **Background**: `#ffffff` (white for email client compatibility)
|
||||
- **Text**: `#333333`
|
||||
- **Primary Button**: `#ff7acc` with white text
|
||||
- **Links**: `#3b82f6`
|
||||
- **Dividers**: `#ff7acc` (brand pink) or `#eeeeee`
|
||||
- **Footer text**: `#666666`
|
||||
|
||||
### Email Button
|
||||
```html
|
||||
<a href="URL" style="display: inline-block; background-color: #ff7acc; color: #ffffff; padding: 14px 28px; text-decoration: none; border-radius: 8px; font-weight: 600;">
|
||||
Button Text
|
||||
</a>
|
||||
```
|
||||
|
||||
### Email Header
|
||||
Always use the Nomad Karaoke logo image in email headers, not an emoji.
|
||||
|
||||
```html
|
||||
<div style="text-align: center; padding: 20px 0;">
|
||||
<a href="https://nomadkaraoke.com">
|
||||
<img src="https://beveradb.github.io/public-images/Nomad-Karaoke-Logo-small-indexed-websafe-rectangle.gif"
|
||||
alt="Nomad Karaoke"
|
||||
style="max-width: 180px; height: auto; border-radius: 10px;" />
|
||||
</a>
|
||||
</div>
|
||||
```
|
||||
|
||||
### Email Signature
|
||||
Include consistent signature with:
|
||||
- Profile image
|
||||
- Name and title
|
||||
- Social media icons
|
||||
- Contact information
|
||||
- Pink (#ff7acc) accent lines
|
||||
|
||||
---
|
||||
|
||||
## Tone & Voice
|
||||
|
||||
### Writing Style
|
||||
- **Friendly but professional**: "Hi there!" not "Dear Customer"
|
||||
- **Clear and concise**: Get to the point
|
||||
- **Encouraging**: Celebrate user accomplishments
|
||||
- **Helpful**: Guide users through actions
|
||||
|
||||
### Example Phrases
|
||||
|
||||
| Instead of... | Use... |
|
||||
|---------------|--------|
|
||||
| "Your order has been processed" | "Your karaoke video is ready!" |
|
||||
| "Error occurred" | "Something went wrong - we're on it!" |
|
||||
| "Please wait" | "Just a moment..." |
|
||||
| "Submit" | "Create Karaoke" or "Get Started" |
|
||||
|
||||
### Emoji Usage
|
||||
- Use sparingly for warmth
|
||||
- Common: 🎤 (brand), 🎵 (music), ✨ (magic), 🎉 (celebration)
|
||||
- Avoid in error messages
|
||||
- Email subject lines: One emoji max
|
||||
|
||||
---
|
||||
|
||||
## Light/Dark Theme Requirements
|
||||
|
||||
### Overview
|
||||
All Nomad Karaoke web application screens must support both light and dark themes. Dark theme is the default, but users should have easy access to toggle between themes.
|
||||
|
||||
### Implementation Requirements
|
||||
|
||||
1. **Theme Toggle**: Every screen/page must include a visible theme toggle
|
||||
- Location: Typically in the header or settings area
|
||||
- Use `next-themes` library for Next.js applications
|
||||
- Toggle should be easily discoverable
|
||||
|
||||
2. **Default Theme**: Dark mode
|
||||
- Respects system preference on first visit
|
||||
- User preference is persisted
|
||||
|
||||
3. **Color Consistency**: All brand colors must work in both themes
|
||||
- Brand pink (#ff7acc) remains consistent
|
||||
- Background, card, and text colors adapt per theme tables above
|
||||
- Ensure sufficient contrast in both modes
|
||||
|
||||
### next-themes Setup (Next.js)
|
||||
|
||||
```tsx
|
||||
// app/layout.tsx or _app.tsx
|
||||
import { ThemeProvider } from 'next-themes'
|
||||
|
||||
export default function RootLayout({ children }) {
|
||||
return (
|
||||
<html suppressHydrationWarning>
|
||||
<body>
|
||||
<ThemeProvider attribute="class" defaultTheme="dark" enableSystem>
|
||||
{children}
|
||||
</ThemeProvider>
|
||||
</body>
|
||||
</html>
|
||||
)
|
||||
}
|
||||
|
||||
// Theme toggle component
|
||||
import { useTheme } from 'next-themes'
|
||||
import { Sun, Moon } from 'lucide-react'
|
||||
|
||||
export function ThemeToggle() {
|
||||
const { theme, setTheme } = useTheme()
|
||||
|
||||
return (
|
||||
<button
|
||||
onClick={() => setTheme(theme === 'dark' ? 'light' : 'dark')}
|
||||
className="p-2 rounded-lg hover:bg-white/10"
|
||||
aria-label="Toggle theme"
|
||||
>
|
||||
{theme === 'dark' ? <Sun size={20} /> : <Moon size={20} />}
|
||||
</button>
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
### CSS Variables for Theming
|
||||
|
||||
```css
|
||||
/* Dark theme (default) */
|
||||
:root {
|
||||
--background: #0f0f0f;
|
||||
--card: #1a1a1a;
|
||||
--card-border: #2a2a2a;
|
||||
--text: #e5e5e5;
|
||||
--text-muted: #888888;
|
||||
}
|
||||
|
||||
/* Light theme */
|
||||
.light {
|
||||
--background: #f8fafc;
|
||||
--card: #ffffff;
|
||||
--card-border: #e2e8f0;
|
||||
--text: #1e293b;
|
||||
--text-muted: #64748b;
|
||||
}
|
||||
|
||||
/* Brand colors stay consistent across themes */
|
||||
:root {
|
||||
--brand-pink: #ff5bb8; /* Darker - better white text contrast */
|
||||
--brand-pink-hover: #ff7acc; /* Lighter - for hover states */
|
||||
--brand-gold: #ffdf6b;
|
||||
--brand-purple: #8b5cf6;
|
||||
}
|
||||
```
|
||||
|
||||
### Testing Checklist
|
||||
- [ ] Theme toggle is visible on all pages
|
||||
- [ ] Toggle persists preference across sessions
|
||||
- [ ] All text is readable in both themes
|
||||
- [ ] Brand colors render correctly in both themes
|
||||
- [ ] Interactive elements have proper hover/focus states in both themes
|
||||
- [ ] Radial background gradient adjusts opacity for light theme
|
||||
|
||||
---
|
||||
|
||||
## Implementation Reference
|
||||
|
||||
### CSS Variables (Recommended)
|
||||
|
||||
```css
|
||||
:root {
|
||||
/* Brand Colors */
|
||||
/* Note: darker pink used as default for better white text contrast */
|
||||
--brand-pink: #ff5bb8; /* Default - better contrast with white text */
|
||||
--brand-pink-hover: #ff7acc; /* Hover - lighter, original brand pink */
|
||||
--brand-gold: #ffdf6b;
|
||||
--brand-purple: #8b5cf6;
|
||||
--brand-blue: #3b82f6;
|
||||
|
||||
/* Semantic Colors */
|
||||
--success: #22c55e;
|
||||
--warning: #f59e0b;
|
||||
--danger: #ef4444;
|
||||
|
||||
/* Dark Theme */
|
||||
--bg: #0f0f0f;
|
||||
--card: #1a1a1a;
|
||||
--card-border: #2a2a2a;
|
||||
--text: #e5e5e5;
|
||||
--text-muted: #888888;
|
||||
|
||||
/* Spacing */
|
||||
--space-xs: 4px;
|
||||
--space-sm: 8px;
|
||||
--space-md: 16px;
|
||||
--space-lg: 24px;
|
||||
--space-xl: 32px;
|
||||
|
||||
/* Border Radius */
|
||||
--radius-sm: 8px;
|
||||
--radius-md: 12px;
|
||||
--radius-lg: 16px;
|
||||
--radius-pill: 9999px;
|
||||
}
|
||||
```
|
||||
|
||||
### Tailwind Configuration
|
||||
|
||||
```javascript
|
||||
// tailwind.config.js
|
||||
module.exports = {
|
||||
theme: {
|
||||
extend: {
|
||||
colors: {
|
||||
brand: {
|
||||
// Darker pink as default for better white text contrast
|
||||
pink: '#ff5bb8',
|
||||
'pink-hover': '#ff7acc', // Lighter for hover states
|
||||
gold: '#ffdf6b',
|
||||
purple: '#8b5cf6',
|
||||
blue: '#3b82f6',
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
### React Component Example
|
||||
|
||||
```tsx
|
||||
// Primary Button Component
|
||||
export function PrimaryButton({ children, ...props }) {
|
||||
return (
|
||||
<button
|
||||
className="bg-brand-pink hover:bg-brand-pink-hover text-white font-semibold px-7 py-3.5 rounded-xl transition-all shadow-[0_0_20px_rgba(255,122,204,0.4)] hover:shadow-[0_0_30px_rgba(255,122,204,0.6)]"
|
||||
{...props}
|
||||
>
|
||||
{children}
|
||||
</button>
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Checklist for Brand Compliance
|
||||
|
||||
### Landing Pages
|
||||
- [ ] Uses brand pink (#ff5bb8) for primary CTAs with white text
|
||||
- [ ] Logo displayed correctly
|
||||
- [ ] Dark theme as default
|
||||
- [ ] Gradient text on hero headlines
|
||||
- [ ] Consistent typography scale
|
||||
- [ ] Proper spacing (multiples of 4px)
|
||||
|
||||
### Application UI
|
||||
- [ ] Logo in header (32-40px height)
|
||||
- [ ] Brand pink for primary actions
|
||||
- [ ] Consistent card styling
|
||||
- [ ] Proper focus states with brand pink
|
||||
- [ ] Dark/light theme toggle working
|
||||
|
||||
### Emails
|
||||
- [ ] Header with logo image (not emoji)
|
||||
- [ ] Brand pink buttons
|
||||
- [ ] Pink divider lines
|
||||
- [ ] Consistent footer with signature
|
||||
- [ ] White background for compatibility
|
||||
|
||||
### Marketing Materials
|
||||
- [ ] Logo with proper clear space
|
||||
- [ ] Brand colors only
|
||||
- [ ] Consistent typography
|
||||
- [ ] No logo modifications
|
||||
|
||||
---
|
||||
|
||||
## Version History
|
||||
|
||||
| Version | Date | Changes |
|
||||
|---------|------|---------|
|
||||
| 1.2 | 2026-01-08 | **Accessibility improvement**: Swapped brand pink colors for better white text contrast. Default `--brand-pink` is now `#ff5bb8` (darker) for buttons/CTAs, hover state `--brand-pink-hover` is now `#ff7acc` (lighter). The original lighter pink had insufficient contrast ratio with white text. |
|
||||
| 1.1 | 2026-01-07 | Updated radial background gradient (now uses layered radial gradients with optional animation); Updated logo usage (SVG preferred, hosted GIF URL for emails only); Updated email header (use logo image, not emoji); Added Light/Dark Theme Requirements section; Fixed success color consistency (#22c55e) |
|
||||
| 1.0 | 2026-01-07 | Initial brand guide created |
|
||||
|
||||
---
|
||||
|
||||
*This guide is the source of truth for Nomad Karaoke brand identity. For questions or clarifications, contact the design team.*
|
||||
+40
-6
@@ -110,17 +110,41 @@ Push to `main` triggers GitHub Actions:
|
||||
3. Deploy to Cloud Run
|
||||
4. Deploy frontend to Cloudflare Pages
|
||||
|
||||
### Manual Backend Deploy
|
||||
### Infrastructure Changes (Pulumi)
|
||||
|
||||
**All infrastructure changes must go through PRs.** The CI pipeline automatically deploys Pulumi changes when PRs merge to main.
|
||||
|
||||
```bash
|
||||
# Preview infrastructure changes
|
||||
# Preview changes locally (requires GCP auth)
|
||||
cd infrastructure
|
||||
pulumi preview
|
||||
pulumi preview --stack nomadkaraoke/karaoke-gen-infrastructure/prod
|
||||
|
||||
# Apply changes
|
||||
pulumi up
|
||||
# DO NOT run `pulumi up` locally - let CI handle deployment
|
||||
# CI uses --skip-preview due to limited service account permissions
|
||||
```
|
||||
|
||||
**What's managed by Pulumi:**
|
||||
- Firestore database and indexes
|
||||
- GCS buckets
|
||||
- Cloud Run service configuration
|
||||
- Cloud Tasks queues
|
||||
- Secret Manager secrets
|
||||
- Service accounts and IAM bindings
|
||||
- Workload Identity Federation
|
||||
- GCE instances (GitHub runners, encoding worker)
|
||||
|
||||
**Workflow:**
|
||||
1. Make changes in the appropriate module under `infrastructure/`
|
||||
- `modules/` - Core GCP resources (database, storage, secrets, IAM, etc.)
|
||||
- `compute/` - VM definitions and startup scripts
|
||||
- `config.py` - Shared configuration constants
|
||||
- `__main__.py` - Entry point that wires modules together
|
||||
2. Run `pulumi preview` locally to verify
|
||||
3. Create PR and merge
|
||||
4. CI deploys automatically
|
||||
|
||||
See `docs/LESSONS-LEARNED.md` for Pulumi CI gotchas.
|
||||
|
||||
### Manual Frontend Deploy
|
||||
|
||||
```bash
|
||||
@@ -151,7 +175,11 @@ ADMIN_TOKENS=token1,token2
|
||||
|
||||
```bash
|
||||
NEXT_PUBLIC_API_URL=http://localhost:8000
|
||||
KARAOKE_ACCESS_TOKEN=... # For E2E tests
|
||||
|
||||
# For E2E tests (see frontend/.env.local.example)
|
||||
KARAOKE_ACCESS_TOKEN=... # Skip email enrollment
|
||||
TESTMAIL_API_KEY=... # testmail.app API key
|
||||
TESTMAIL_NAMESPACE=... # testmail.app namespace
|
||||
```
|
||||
|
||||
## Key Scripts
|
||||
@@ -162,6 +190,12 @@ KARAOKE_ACCESS_TOKEN=... # For E2E tests
|
||||
| `./scripts/run-backend-local.sh` | Start backend with emulators |
|
||||
| `./scripts/start-emulators.sh` | Start emulators only |
|
||||
| `./scripts/stop-emulators.sh` | Stop emulators |
|
||||
| `./scripts/debug-job.sh <job_id>` | Debug a cloud job (status, logs, files) |
|
||||
| `./scripts/get_job.py <job_id>` | Fetch job data from Firestore |
|
||||
| `./scripts/fetch_job_logs.py <job_id>` | Fetch cloud job logs to local file |
|
||||
| `./scripts/analyze_log_timing.py <log>` | Profile karaoke-gen performance from logs |
|
||||
| `./scripts/benchmark_ffmpeg.py` | Isolated FFmpeg encoding benchmark |
|
||||
| `./scripts/compare_local_vs_remote.py` | Compare local vs cloud karaoke-gen performance |
|
||||
|
||||
## Project Structure
|
||||
|
||||
|
||||
@@ -0,0 +1,239 @@
|
||||
# E2E Happy Path Test Specification
|
||||
|
||||
> **STATUS**: ✅ **PASSING** - Test completes all 12 steps in ~38 minutes. Last verified 2026-01-02.
|
||||
>
|
||||
> **IMPORTANT**: This document defines the requirements for a comprehensive end-to-end test that validates the entire karaoke generation flow with REAL user interactions only. NO API shortcuts allowed.
|
||||
|
||||
## Overview
|
||||
|
||||
**Goal**: A single Playwright test that validates the complete karaoke generation journey from first-time user signup through to distribution, using only browser interactions like a real user would.
|
||||
|
||||
**Environment**: Production (`gen.nomadkaraoke.com` / `api.nomadkaraoke.com`)
|
||||
|
||||
**Automation**: Runs daily via GitHub Actions and can be triggered on-demand.
|
||||
|
||||
## Test Requirements
|
||||
|
||||
### 1. New User Registration (testmail.app)
|
||||
|
||||
- [ ] Navigate to landing page (`gen.nomadkaraoke.com`)
|
||||
- [ ] Click "Sign In" or "Get Started" button
|
||||
- [ ] Generate a unique test email address (testmail.app namespace)
|
||||
- [ ] Enter the test email in the beta enrollment form
|
||||
- [ ] Submit the form
|
||||
- [ ] Wait for welcome/magic-link email via testmail.app API
|
||||
- [ ] Extract the magic link from the email
|
||||
- [ ] Click/navigate to the magic link to authenticate
|
||||
- [ ] Verify user is logged in and has credits
|
||||
|
||||
**Note**: When `use_test_token=true` workflow input is set, steps 1-9 above are skipped and the test uses a pre-authenticated token instead.
|
||||
|
||||
### 2. Create Karaoke Job (UI Only)
|
||||
|
||||
- [ ] Navigate to the karaoke creation page
|
||||
- [ ] Enter artist name: `piri` (test song with cached results)
|
||||
- [ ] Enter song title: `dog`
|
||||
- [ ] Click "Search" or equivalent button
|
||||
- [ ] Wait for search results to appear in the UI
|
||||
|
||||
### 3. Audio Selection (UI Only)
|
||||
|
||||
- [ ] Audio selection dialog/panel should appear
|
||||
- [ ] View available audio options in the UI
|
||||
- [ ] Click to select the first/best audio option
|
||||
- [ ] Confirm the selection via UI button
|
||||
- [ ] Wait for job to transition to processing state
|
||||
|
||||
### 4. Wait for Lyrics Transcription
|
||||
|
||||
- [ ] Monitor job status in the UI (not via API polling)
|
||||
- [ ] Wait for job to reach "awaiting lyrics review" state
|
||||
- [ ] This may take 10-15+ minutes for audio download + transcription
|
||||
|
||||
### 5. Lyrics Review (FULL UI Interaction)
|
||||
|
||||
> **CRITICAL**: This MUST be done through the UI, not via API calls
|
||||
|
||||
- [ ] Click to open the lyrics review interface
|
||||
- [ ] Verify lyrics are displayed in the editor
|
||||
- [ ] Interact with the lyrics editor:
|
||||
- [ ] View synced lyrics with timestamps
|
||||
- [ ] Optionally make a small edit to prove UI interaction works
|
||||
- [ ] Preview the video if preview functionality exists
|
||||
- [ ] Click "Approve" / "Submit" / "Complete Review" button in UI
|
||||
- [ ] Wait for job to transition to next state
|
||||
|
||||
### 6. Instrumental Selection (FULL UI Interaction)
|
||||
|
||||
> **CRITICAL**: This MUST be done through the UI, not via API calls
|
||||
|
||||
- [ ] Wait for instrumental options to become available
|
||||
- [ ] Instrumental selection dialog/panel should appear in UI
|
||||
- [ ] View available instrumental options (e.g., "clean", "with backing vocals")
|
||||
- [ ] Click to select the preferred instrumental option
|
||||
- [ ] Confirm selection via UI button
|
||||
- [ ] Wait for job to transition to video rendering state
|
||||
|
||||
### 7. Wait for Video Rendering & Encoding
|
||||
|
||||
- [ ] Monitor job progress in the UI
|
||||
- [ ] Wait for video rendering to complete
|
||||
- [ ] Wait for final encoding to complete
|
||||
- [ ] Job should reach "complete" status
|
||||
|
||||
### 8. Verify Completion & Downloads
|
||||
|
||||
- [ ] Verify job shows as "complete" in the UI
|
||||
- [ ] Verify download links are visible and clickable
|
||||
- [ ] Click at least one download link to verify it works
|
||||
- [ ] Verify the downloaded file is valid (non-zero size, correct format)
|
||||
|
||||
### 9. Distribution Verification & Cleanup
|
||||
|
||||
> **CRITICAL**: Test MUST clean up after itself to avoid polluting production distribution channels
|
||||
|
||||
- [ ] Verify job.state_data contains distribution results:
|
||||
- [ ] `brand_code` - The generated brand code (e.g., "NOMAD-1234")
|
||||
- [ ] `youtube_url` - YouTube video URL (if YouTube enabled)
|
||||
- [ ] `dropbox_link` - Dropbox shared link (if Dropbox enabled)
|
||||
- [ ] `gdrive_files` - Google Drive file IDs (if GDrive enabled)
|
||||
- [ ] **CLEANUP** - After verification, delete all distributed content:
|
||||
- [ ] Delete YouTube video via API (using stored youtube_url)
|
||||
- [ ] Delete Dropbox folder via API (using brand_code + dropbox_path)
|
||||
- [ ] Delete Google Drive files via API (using gdrive_files IDs)
|
||||
- [ ] Delete the test job from the system (GCS + Firestore)
|
||||
|
||||
## Technical Requirements
|
||||
|
||||
### Timeouts
|
||||
|
||||
| Stage | Timeout |
|
||||
|-------|---------|
|
||||
| Page navigation | 30 seconds |
|
||||
| Email arrival | 120 seconds |
|
||||
| Audio search | 60 seconds |
|
||||
| Lyrics transcription | 20 minutes |
|
||||
| Video rendering | 15 minutes |
|
||||
| Final encoding | 10 minutes |
|
||||
| **Total test** | **60 minutes** |
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
TESTMAIL_NAMESPACE=xxx # Required - for email testing (testmail.app)
|
||||
TESTMAIL_API_KEY=xxx # Required - for email testing (testmail.app)
|
||||
E2E_TEST_TOKEN=xxx # Optional - skip signup, use existing account
|
||||
# When use_test_token=true, skips email signup and uses pre-authenticated token
|
||||
```
|
||||
|
||||
### Test Song
|
||||
|
||||
Use `piri - dog` because:
|
||||
- Has cached flacfetch results (faster audio search)
|
||||
- Known to work reliably
|
||||
- Short song = faster processing
|
||||
|
||||
### Screenshots & Artifacts
|
||||
|
||||
Capture screenshots at each major milestone:
|
||||
1. Landing page loaded
|
||||
2. Beta form submitted
|
||||
3. Email received
|
||||
4. Logged in successfully
|
||||
5. Job created
|
||||
6. Audio selected
|
||||
7. Lyrics review opened
|
||||
8. Lyrics approved
|
||||
9. Instrumental selected
|
||||
10. Job completed
|
||||
11. Download verified
|
||||
|
||||
### Logging
|
||||
|
||||
Log detailed progress at each step with timestamps for debugging.
|
||||
|
||||
## GitHub Actions Workflow
|
||||
|
||||
**File**: `.github/workflows/e2e-happy-path.yml`
|
||||
|
||||
### Triggers
|
||||
|
||||
1. **Daily schedule**: Run at 6 AM UTC every day
|
||||
2. **Manual dispatch**: Can be triggered on-demand with optional parameters
|
||||
3. **Label trigger**: When `e2e-full` label added to PR
|
||||
|
||||
### Secrets Required
|
||||
|
||||
- `TESTMAIL_API_KEY` - For email testing (testmail.app)
|
||||
- `TESTMAIL_NAMESPACE` - testmail.app namespace for unique email addresses
|
||||
|
||||
### Workflow Steps
|
||||
|
||||
1. Checkout code
|
||||
2. Setup Node.js
|
||||
3. Install dependencies
|
||||
4. Install Playwright browsers
|
||||
5. Run the e2e happy path test
|
||||
6. Upload artifacts (screenshots, videos, logs)
|
||||
7. Report results
|
||||
|
||||
### Success Criteria
|
||||
|
||||
- All UI interactions complete successfully
|
||||
- Job reaches "complete" status
|
||||
- Download link works
|
||||
- No JavaScript errors in console
|
||||
- All screenshots captured
|
||||
|
||||
## What This Test Proves
|
||||
|
||||
When this test passes, it proves:
|
||||
|
||||
1. **User signup works** - New users can register via magic link
|
||||
2. **Authentication works** - Email verification and login flow functional
|
||||
3. **Job creation works** - Can search and create karaoke jobs
|
||||
4. **Audio pipeline works** - Audio search, selection, and download functional
|
||||
5. **Transcription works** - Lyrics transcription completes successfully
|
||||
6. **Lyrics review UI works** - Users can review and approve lyrics
|
||||
7. **Instrumental selection UI works** - Users can select instrumental versions
|
||||
8. **Video rendering works** - Full video generation pipeline functional
|
||||
9. **Downloads work** - Users can download their completed karaoke videos
|
||||
10. **End-to-end integration** - All systems work together correctly
|
||||
|
||||
## Anti-Patterns to Avoid
|
||||
|
||||
1. **NO API calls to complete review** - Must click buttons in UI
|
||||
2. **NO API calls to select instrumental** - Must click buttons in UI
|
||||
3. **NO pre-authenticated tokens** - Must go through full signup
|
||||
4. **NO skipping steps** - Every user-facing interaction must happen
|
||||
5. **NO hardcoded waits** - Use proper Playwright waiting strategies
|
||||
|
||||
## File Location
|
||||
|
||||
**Test file**: `frontend/e2e/production/happy-path-real-user.spec.ts`
|
||||
|
||||
This is separate from the existing `full-user-journey.spec.ts` which uses API shortcuts.
|
||||
|
||||
## Iteration Plan
|
||||
|
||||
1. ~~Create initial test with all steps stubbed~~
|
||||
2. ~~Implement each section one at a time~~
|
||||
3. ~~Run via GHA workflow~~
|
||||
4. ~~Debug failures using artifacts~~
|
||||
5. ~~Iterate until fully passing~~ ✅ Completed 2026-01-02
|
||||
6. Add to required CI checks once stable
|
||||
|
||||
## Key Fixes Applied
|
||||
|
||||
During iteration, three issues were fixed:
|
||||
|
||||
1. **STEP 8 status detection**: "rendering" status comes BEFORE instrumental selection, not after. Changed to only skip on "encoding".
|
||||
2. **Instrumental page navigation**: Changed `waitUntil: 'networkidle'` to `'load'` because audio players prevent networkidle.
|
||||
3. **Download verification**: Changed HEAD to GET request with `maxRedirects: 0` since download endpoint doesn't support HEAD.
|
||||
|
||||
See [LESSONS-LEARNED.md](LESSONS-LEARNED.md) for details.
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2026-01-02*
|
||||
*Author: Claude (via Andrew's request)*
|
||||
Loaded 100 of 339 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user