mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-21 13:28:08 +09:00
Compare commits
331
Commits
No files matched your search
@@ -0,0 +1,109 @@
|
||||
# Plan: Admin Feedback Visibility
|
||||
|
||||
**Created:** 2026-03-23
|
||||
**Branch:** feat/sess-20260323-1454-feedback-email-debug
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
User feedback submissions are stored in Firestore (`user_feedback` collection) but are completely invisible to the admin. The admin panel only shows a boolean "Feedback Submitted: Yes/No" badge per user. This plan adds:
|
||||
|
||||
1. **Admin UI page** to browse and review all submitted feedback
|
||||
2. **Email notification** to admin@nomadkaraoke.com on each new submission
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] Admin page at `/admin/feedback` listing all feedback with ratings, text, timestamps, and user info
|
||||
- [ ] Sortable by date (newest first by default) and filterable by search (email)
|
||||
- [ ] Star ratings rendered visually (not just numbers)
|
||||
- [ ] Email sent to admin@nomadkaraoke.com immediately when feedback is submitted
|
||||
- [ ] Email includes all feedback content (ratings, text fields, user email)
|
||||
- [ ] Test data filtering works (exclude_test toggle)
|
||||
- [ ] Sidebar navigation includes new "Feedback" item
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Admin API Endpoint
|
||||
Add `GET /api/admin/feedback` to `backend/api/routes/admin.py`. Follows the standard admin endpoint pattern: `require_admin` dependency, pagination with `limit`/`offset`/`search`/`exclude_test`, returns Pydantic response model. Queries `user_feedback` Firestore collection.
|
||||
|
||||
### Admin UI Page
|
||||
New page at `frontend/app/admin/feedback/page.tsx`. Follows existing admin page patterns (Card, Table, Badge, pagination, search input, loading states). Uses `adminApi.listFeedback()` to fetch data.
|
||||
|
||||
### Email Notification
|
||||
Add a new method to `EmailService` (or call `send_email` directly) from the `submit_user_feedback` endpoint in `users.py`. Uses the existing SendGrid infrastructure — no new template service templates needed since this is an internal admin notification (simple HTML table with the feedback data).
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Step 1: Backend — Admin feedback list endpoint
|
||||
|
||||
1. [ ] Add `AdminFeedbackItem` and `AdminFeedbackListResponse` Pydantic models to `admin.py`
|
||||
2. [ ] Add `GET /api/admin/feedback` endpoint with:
|
||||
- Pagination: `limit=50`, `offset=0`
|
||||
- Search by user email
|
||||
- `exclude_test` filter (reuse `is_test_email`)
|
||||
- Sort by `created_at` descending
|
||||
- Return: items, total, offset, limit, has_more
|
||||
3. [ ] Add aggregate stats in response: average ratings, total count
|
||||
|
||||
### Step 2: Backend — Email notification on submission
|
||||
|
||||
4. [ ] Add `send_feedback_notification` method to `EmailService` that sends an HTML email to `admin@nomadkaraoke.com` containing:
|
||||
- User email
|
||||
- All 4 star ratings
|
||||
- All text fields
|
||||
- Would recommend / would use again
|
||||
- Submitted timestamp
|
||||
- Link to admin feedback page
|
||||
5. [ ] Call `send_feedback_notification` from `submit_user_feedback` in `users.py` (fire-and-forget, don't block the user response on email success)
|
||||
|
||||
### Step 3: Frontend — Admin API client
|
||||
|
||||
6. [ ] Add `listFeedback` method to `adminApi` in `frontend/lib/api.ts`
|
||||
7. [ ] Add TypeScript types for `AdminFeedbackItem` and `AdminFeedbackListResponse`
|
||||
|
||||
### Step 4: Frontend — Admin feedback page
|
||||
|
||||
8. [ ] Create `frontend/app/admin/feedback/page.tsx` with:
|
||||
- Table: user email, date, overall rating (stars), ease of use, lyrics accuracy, correction experience, would recommend, would use again
|
||||
- Expandable rows or click-to-detail for text fields (what went well, what could improve, additional comments)
|
||||
- Search by email
|
||||
- Pagination (50 per page)
|
||||
- Refresh button
|
||||
- Test data toggle (via `useAdminSettings`)
|
||||
9. [ ] Add "Feedback" item to admin sidebar in `frontend/components/admin/admin-sidebar.tsx` with `MessageSquare` icon
|
||||
|
||||
### Step 5: Testing
|
||||
|
||||
10. [ ] Backend unit test for `GET /api/admin/feedback` endpoint (pagination, search, exclude_test)
|
||||
11. [ ] Backend unit test for `send_feedback_notification` email method
|
||||
12. [ ] Frontend component test for feedback page (render, empty state)
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/api/routes/admin.py` | Modify | Add feedback list endpoint + response models |
|
||||
| `backend/api/routes/users.py` | Modify | Add email notification call in `submit_user_feedback` |
|
||||
| `backend/services/email_service.py` | Modify | Add `send_feedback_notification` method |
|
||||
| `frontend/lib/api.ts` | Modify | Add `adminApi.listFeedback()` method + types |
|
||||
| `frontend/app/admin/feedback/page.tsx` | Create | Admin feedback review page |
|
||||
| `frontend/components/admin/admin-sidebar.tsx` | Modify | Add "Feedback" nav item |
|
||||
| `backend/tests/test_admin_feedback.py` | Create | Tests for admin feedback endpoint |
|
||||
| `backend/tests/test_feedback_email.py` | Create | Tests for feedback email notification |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests**: Admin endpoint (pagination, filtering, sorting), email method (content formatting, send call)
|
||||
- **Integration**: Submit feedback → verify email service called
|
||||
- **Manual verification**: Check email arrives in admin@nomadkaraoke.com inbox after submission
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [ ] Should the admin page show feedback from the old `beta_feedback` collection too, or just `user_feedback`? (Recommendation: just `user_feedback` — the beta collection is legacy)
|
||||
- [ ] Should there be any spam/quality detection flagging? (Some submissions appear to game the system for credits)
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
- Backend: Remove endpoint from admin.py, revert email call in users.py
|
||||
- Frontend: Delete feedback page, remove sidebar item
|
||||
- No data migration needed — Firestore collection unchanged
|
||||
@@ -0,0 +1,120 @@
|
||||
# Plan: Admin Lyrics Review Dashboard
|
||||
|
||||
**Created:** 2026-03-05
|
||||
**Branch:** feat/sess-20260305-0002-admin-lyrics-review
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Add an "Edit Reviews" section to the admin dashboard for reviewing completed lyrics reviews. Three goals:
|
||||
|
||||
1. **View what users saw** - Jump into a read-only lyrics review UI showing original vs corrected lyrics
|
||||
2. **Replay user edits** - Step through edit log entries to see exactly what each user changed
|
||||
3. **Review raw data** - View human edits data to evaluate quality for AudioShake data sharing
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] New admin sidebar item "Edit Reviews" linking to `/admin/edit-reviews`
|
||||
- [ ] Backend endpoint to list recent jobs that have completed lyrics review (with edit logs)
|
||||
- [ ] Backend endpoint to serve immutable correction data for a job (read-only, cannot modify job)
|
||||
- [ ] Admin page showing list of recently reviewed jobs with key metrics
|
||||
- [ ] Ability to open the existing LyricsAnalyzer in read-only mode with snapshot data
|
||||
- [ ] Edit log replay/stepper UI to walk through user changes one by one
|
||||
- [ ] Raw data viewer for edit logs, annotations, and correction diffs
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Backend: 2 new admin endpoints
|
||||
|
||||
1. **`GET /api/admin/edit-reviews`** - List jobs with completed reviews
|
||||
- Query Firestore for jobs with `state_data.last_edit_log_path` set (indicating review was performed)
|
||||
- Filter by status (completed jobs, or optionally all post-review statuses)
|
||||
- Return summary: job_id, artist, title, user_email, review_completed_at, edit_count, feedback_count
|
||||
- Support pagination, search, exclude_test filtering
|
||||
|
||||
2. **`GET /api/admin/edit-reviews/{job_id}`** - Get full review snapshot
|
||||
- Returns immutable copy of:
|
||||
- Original corrections (`corrections.json` from GCS)
|
||||
- Updated corrections (`corrections_updated.json` from GCS)
|
||||
- Edit log (`edit_log_{session_id}.json` from GCS)
|
||||
- Annotations (`annotations.json` from GCS)
|
||||
- Job metadata (artist, title, user_email, timestamps)
|
||||
- Audio URL (signed, for playback in review UI)
|
||||
- This is a **read-only** endpoint - no job state mutations
|
||||
- Uses `require_admin` dependency (not `require_review_auth`)
|
||||
- Does NOT transition job status or modify any state
|
||||
|
||||
### Frontend: New admin page + Read-only review mode
|
||||
|
||||
1. **`/admin/edit-reviews/page.tsx`** - List page
|
||||
- Table of recently reviewed jobs
|
||||
- Columns: Artist/Title, User, Review Date, Edits Made, Feedback Given, Actions
|
||||
- "View Review" button → opens read-only lyrics review
|
||||
- "View Edit Log" button → opens edit replay view
|
||||
- "View Raw Data" button → shows raw JSON
|
||||
|
||||
2. **Read-only review viewer** - Reuse LyricsAnalyzer
|
||||
- Load data from the new admin endpoint (not the regular review endpoint)
|
||||
- Pass `isReadOnly={true}` and `apiClient={null}` to LyricsAnalyzer
|
||||
- Header shows "Admin Review - Read Only" indicator
|
||||
- No submit/save buttons, no state transitions
|
||||
- Audio playback still works (via signed URL)
|
||||
|
||||
3. **Edit log replay** - New component
|
||||
- Timeline/stepper showing each edit entry
|
||||
- For each entry: operation type, text before → text after, timestamp
|
||||
- Step forward/backward through edits
|
||||
- Show feedback reason if user provided one
|
||||
- Summary stats: total edits, edits with feedback, correction types breakdown
|
||||
|
||||
4. **Raw data viewer** - JSON viewer tabs
|
||||
- Tab 1: Edit Log (formatted JSON with syntax highlighting)
|
||||
- Tab 2: Annotations
|
||||
- Tab 3: Corrections diff (original vs updated)
|
||||
|
||||
### Key safety guarantee
|
||||
|
||||
The admin review endpoint serves a **snapshot** of the data. It reads from GCS files that were already written. The LyricsAnalyzer receives `apiClient={null}` so it physically cannot call any mutation endpoints. Even if code changes in the future, the admin endpoint itself never writes to Firestore or GCS.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. [ ] **Backend: Add `GET /api/admin/edit-reviews` endpoint** - List reviewed jobs with edit log metadata
|
||||
2. [ ] **Backend: Add `GET /api/admin/edit-reviews/{job_id}` endpoint** - Serve full immutable review snapshot
|
||||
3. [ ] **Frontend: Add admin API client methods** - `adminApi.listEditReviews()`, `adminApi.getEditReview(jobId)`
|
||||
4. [ ] **Frontend: Create `/admin/edit-reviews/page.tsx`** - List page with table
|
||||
5. [ ] **Frontend: Add sidebar nav item** - "Edit Reviews" with BookOpen icon
|
||||
6. [ ] **Frontend: Create read-only review viewer** - Load snapshot data into LyricsAnalyzer with isReadOnly
|
||||
7. [ ] **Frontend: Create edit log replay component** - Step-through UI for edit entries
|
||||
8. [ ] **Frontend: Create raw data viewer** - JSON viewer with tabs
|
||||
9. [ ] **Tests: Backend unit tests** for new admin endpoints
|
||||
10. [ ] **Tests: Frontend component tests** for new page
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/api/routes/admin.py` | Modify | Add edit-reviews endpoints |
|
||||
| `frontend/lib/api.ts` | Modify | Add adminApi methods for edit reviews |
|
||||
| `frontend/app/admin/edit-reviews/page.tsx` | Create | Main edit reviews list page |
|
||||
| `frontend/components/admin/edit-review-viewer.tsx` | Create | Read-only review wrapper using LyricsAnalyzer |
|
||||
| `frontend/components/admin/edit-log-replay.tsx` | Create | Edit log stepper/replay component |
|
||||
| `frontend/components/admin/raw-data-viewer.tsx` | Create | JSON viewer with tabs |
|
||||
| `frontend/components/admin/admin-sidebar.tsx` | Modify | Add Edit Reviews nav item |
|
||||
| `backend/tests/test_admin_edit_reviews.py` | Create | Backend tests |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- Unit tests for backend endpoints (mock Firestore + GCS)
|
||||
- Verify admin auth is required on all new endpoints
|
||||
- Verify no write operations occur on read-only endpoints
|
||||
- Frontend component tests for key interactions
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [ ] Should we show ALL jobs that went through review, or only ones with edit logs? (Recommendation: only those with edit logs, since those are the interesting ones)
|
||||
- [ ] Do we need audio playback in the admin review viewer? (Recommendation: yes, it helps understand what the user heard)
|
||||
- [ ] Should the raw data be downloadable? (Nice-to-have, can add later)
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
All changes are additive (new endpoints, new pages). Removing the sidebar link effectively hides the feature. No existing behavior is modified.
|
||||
@@ -0,0 +1,151 @@
|
||||
# Plan: AI-Powered Credit Gating
|
||||
|
||||
**Created:** 2026-03-23
|
||||
**Branch:** feat/sess-20260323-1716-ai-credit-gating
|
||||
**Status:** Implemented
|
||||
|
||||
## Overview
|
||||
|
||||
Replace the automatic welcome/feedback credit grants with an AI-powered evaluation step. When a user triggers a credit grant (email verification or feedback submission), instead of immediately granting credits, we collect abuse signals and ask Gemini to make a grant/deny decision. Suspected abusers get a friendly rejection email (CC'd to Andrew) instead of free credits.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] Welcome credit grants go through AI evaluation before granting
|
||||
- [ ] Feedback credit grants go through AI evaluation before granting
|
||||
- [ ] AI evaluates: fingerprint correlations, IP correlations, IP geo info, user agent, signup recency patterns
|
||||
- [ ] Approved users get credits as before (seamless experience)
|
||||
- [ ] Rejected users get a friendly-but-firm email explaining the situation, CC'd to andrew@beveridge.uk
|
||||
- [ ] Rejection email invites them to reply if it's a mistake
|
||||
- [ ] Decision is logged to Firestore for audit trail
|
||||
- [ ] Fail-open: if AI evaluation fails, grant credits (don't punish users for our errors)
|
||||
- [ ] Admin can override decisions via existing admin credit grant endpoint
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
User verifies email / submits feedback
|
||||
↓
|
||||
Collect abuse signals (sync, fast):
|
||||
- User's fingerprint + IP
|
||||
- Other accounts with same fingerprint (from correlations)
|
||||
- Other accounts with same IP
|
||||
- IP geolocation info
|
||||
- Recent signup count from same IP/fingerprint (last 24h)
|
||||
- User agent string
|
||||
- For feedback: the feedback content itself
|
||||
↓
|
||||
Call Gemini (vertexai/gemini-2.5-pro-preview) with structured prompt:
|
||||
- Present all signals
|
||||
- Ask for JSON response: {decision: "grant" | "deny", reasoning: string, confidence: number}
|
||||
↓
|
||||
If "grant": grant credits normally, log decision
|
||||
If "deny": skip credit grant, send rejection email, log decision
|
||||
If AI call fails: grant credits (fail-open), log error
|
||||
```
|
||||
|
||||
### Key Design Decisions
|
||||
|
||||
1. **Two-phase verification**: The magic link verify endpoint returns immediately with session + a `credit_status: "evaluating"` field. The frontend shows a "preparing your account" loading screen, then polls/waits for the evaluation result via a new lightweight endpoint. This keeps the auth flow snappy while the AI evaluation runs in the background of the same request or via a short follow-up call. The frontend then shows either a celebratory "2 free credits!" interstitial or a "sorry, couldn't grant credits" interstitial before navigating to the dashboard.
|
||||
|
||||
2. **Fail-open**: If Gemini is down or returns an error, we grant credits. Better to let an occasional abuser through than block legitimate users.
|
||||
|
||||
3. **Use google-genai directly**: The project already has `langchain-google-genai` installed. We can use the simpler `google.generativeai` library directly for a focused prompt/response, or use the existing LangChain setup. We'll use `google.generativeai` for simplicity since this is a single-turn evaluation, not an agentic workflow.
|
||||
|
||||
4. **Structured JSON output**: Ask Gemini for structured output with decision, reasoning, and confidence to make decisions parseable and auditable.
|
||||
|
||||
5. **Log all decisions**: Store every evaluation in a `credit_evaluations` Firestore collection for audit trail and to tune the prompt over time.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Step 1: Add google-generativeai dependency
|
||||
- Add `google-generativeai` to pyproject.toml (lightweight, direct Gemini access)
|
||||
- Verify it works with Vertex AI auth in the nomadkaraoke project
|
||||
|
||||
### Step 2: Create credit evaluation service
|
||||
- New file: `backend/services/credit_evaluation_service.py`
|
||||
- `CreditEvaluationService` class with:
|
||||
- `evaluate_credit_grant(email, grant_type, feedback_content=None) -> CreditEvaluation`
|
||||
- Collects all abuse signals by calling existing services (user_service, ip_geolocation_service)
|
||||
- Builds prompt with signals
|
||||
- Calls Gemini for decision
|
||||
- Logs decision to Firestore
|
||||
- Returns structured result
|
||||
|
||||
### Step 3: Build the Gemini prompt
|
||||
- System prompt explaining the role: "You are an anti-abuse evaluator for a karaoke video service..."
|
||||
- Present all signals in a structured format
|
||||
- Request JSON response: `{decision: "grant"|"deny", reasoning: "...", confidence: 0.0-1.0}`
|
||||
- Include clear guidelines: same fingerprint across accounts = very suspicious, same IP = somewhat suspicious, etc.
|
||||
|
||||
### Step 4: Create rejection email template
|
||||
- Add `send_credit_denied_email(email, reasoning)` to EmailService
|
||||
- Friendly-but-firm tone
|
||||
- CC andrew@beveridge.uk
|
||||
- Invite reply if it's a mistake
|
||||
- Use existing email template infrastructure (branding, styling)
|
||||
|
||||
### Step 5: Integrate into welcome credit flow
|
||||
- Modify `grant_welcome_credits_if_eligible()` in user_service.py
|
||||
- Before granting: call `CreditEvaluationService.evaluate_credit_grant(email, "welcome")`
|
||||
- If denied: send rejection email, return False
|
||||
- If error: log warning, grant anyway (fail-open)
|
||||
|
||||
### Step 6: Integrate into feedback credit flow
|
||||
- Modify feedback endpoint in users.py
|
||||
- Before granting credits: evaluate with feedback content included
|
||||
- If denied: return different response message, send rejection email
|
||||
- If error: grant anyway
|
||||
|
||||
### Step 7: Add Firestore collection for audit trail
|
||||
- Collection: `credit_evaluations`
|
||||
- Document fields: email, grant_type, decision, reasoning, confidence, signals_snapshot, model_used, evaluated_at
|
||||
- TTL: keep indefinitely (small volume, high audit value)
|
||||
|
||||
### Step 8: Tests
|
||||
- Unit tests for signal collection
|
||||
- Unit tests for prompt building
|
||||
- Unit tests for decision parsing (grant/deny/error cases)
|
||||
- Integration test for the evaluation service with mocked Gemini
|
||||
- Integration test for welcome credit flow with evaluation
|
||||
- Integration test for feedback credit flow with evaluation
|
||||
- Test fail-open behavior
|
||||
|
||||
### Step 9: Configuration
|
||||
- Add env vars: `CREDIT_EVAL_MODEL` (default: "gemini-2.5-pro-preview"), `CREDIT_EVAL_ENABLED` (default: true)
|
||||
- Add to config.py
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `pyproject.toml` | Modify | Add google-generativeai dependency |
|
||||
| `backend/config.py` | Modify | Add credit evaluation config vars |
|
||||
| `backend/services/credit_evaluation_service.py` | Create | New service: collect signals, call Gemini, log decision |
|
||||
| `backend/services/user_service.py` | Modify | Integrate evaluation into welcome credit grant |
|
||||
| `backend/api/routes/users.py` | Modify | Integrate evaluation into feedback credit grant |
|
||||
| `backend/services/email_service.py` | Modify | Add credit denied email template |
|
||||
| `backend/tests/test_credit_evaluation.py` | Create | Tests for evaluation service |
|
||||
| `backend/tests/test_anti_abuse.py` | Modify | Update welcome credit tests for evaluation |
|
||||
| `frontend/app/auth/verify/page.tsx` | Modify | Add "preparing account" loading + credit result interstitial |
|
||||
| `frontend/lib/api.ts` | Modify | Add credit evaluation status endpoint |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests**: Signal collection, prompt building, response parsing, fail-open behavior
|
||||
- **Integration tests**: Full evaluation flow with mocked Gemini responses
|
||||
- **E2E tests**: Manual verification in production with test accounts
|
||||
- **Edge cases**: Gemini timeout, malformed response, missing signals, first-ever user (no correlations)
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Gemini model**: Using `gemini-3.1-pro-preview` as specified — already available in the nomadkaraoke GCP project.
|
||||
|
||||
2. **Latency UX**: Frontend shows a friendly "preparing your account" loading screen while evaluation runs. Then transitions to either a celebratory "2 free credits!" message (with gentle reminder not to abuse) or a "sorry, couldn't grant free credits" message with a buy credits CTA. User clicks through to dashboard from either.
|
||||
|
||||
3. **Should we also gate the existing rate limiter?** Currently `is_signup_rate_limited()` silently blocks at 2 accounts per IP/fingerprint per 24h. The AI evaluation is complementary — rate limiter blocks obvious rapid abuse, AI evaluation catches more subtle patterns.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Set `CREDIT_EVAL_ENABLED=false` in Cloud Run env vars → immediate rollback to auto-grant behavior. No data model changes needed. The evaluation service is purely additive.
|
||||
@@ -0,0 +1,150 @@
|
||||
# Plan: Block Disposable Email Registration
|
||||
|
||||
**Created:** 2026-03-19
|
||||
**Branch:** feat/sess-20260319-1332-disposable-email-abuse
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Users are signing up with disposable email addresses (yopmail.com, emailhook.site, etc.) to get unlimited free credits (2 per account). The `EmailValidationService` already exists with 160+ disposable domains but is **never called during signup**. We need to wire it into the magic link flow and show a friendly rejection message.
|
||||
|
||||
## Context
|
||||
|
||||
- The `EmailValidationService` (`backend/services/email_validation_service.py`) is fully implemented with disposable domain detection, email/IP blocklists, and admin management APIs
|
||||
- The admin blocklist management routes exist (`backend/api/routes/rate_limits.py`) and work
|
||||
- The magic link endpoint (`POST /api/users/auth/magic-link` in `backend/api/routes/users.py:133`) **never calls** the validation service
|
||||
- Currently 1 yopmail user and 1 emailhook.site user in `gen_users`
|
||||
- Each new account gets 2 free welcome credits automatically
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Reject disposable email domains at the magic link endpoint (backend)
|
||||
- [x] Return a **specific, distinguishable error** so the frontend can show a friendly message (not the generic "email enumeration" response)
|
||||
- [x] Show a warm, friendly message in the AuthDialog explaining why and encouraging real email use
|
||||
- [x] Add `emailhook.site` to the default disposable domains list
|
||||
- [x] Add backend unit tests for the integration
|
||||
- [x] Add frontend test coverage for the new error state
|
||||
|
||||
## Design Decision: Explicit Rejection vs Silent Failure
|
||||
|
||||
The current magic link endpoint returns a generic success message to prevent email enumeration. For disposable emails, we will **explicitly reject** with a 422 status and a specific error message. This is the right trade-off because:
|
||||
|
||||
1. We *want* the user to know their disposable email won't work (so they can use a real one)
|
||||
2. Disposable domain detection is not sensitive information — the blocklist is public knowledge
|
||||
3. The friendly message serves our goal of converting them to real email users
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Step 1: Backend — Add disposable email check to magic link endpoint
|
||||
|
||||
**File:** `backend/api/routes/users.py`
|
||||
|
||||
Add the `EmailValidationService` check **after** email lowercasing (line 158) and **before** tenant validation (line 164). Return a 422 with a friendly message.
|
||||
|
||||
```python
|
||||
# Check for disposable email domains
|
||||
email_validation = get_email_validation_service()
|
||||
if email_validation.is_disposable_domain(email):
|
||||
logger.warning(f"Blocked disposable email signup attempt: {_mask_email(email)}")
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail="disposable_email_not_allowed"
|
||||
)
|
||||
```
|
||||
|
||||
Use a machine-readable error code (`disposable_email_not_allowed`) so the frontend can match on it and show the custom friendly message. This keeps the backend response lean and puts the UX copy in the frontend where it belongs.
|
||||
|
||||
### Step 2: Backend — Also check email and IP blocklists
|
||||
|
||||
In the same location, add checks for explicitly blocked emails and IPs:
|
||||
|
||||
```python
|
||||
if email_validation.is_email_blocked(email):
|
||||
logger.warning(f"Blocked email signup attempt: {_mask_email(email)}")
|
||||
# Silent reject — return generic success to prevent enumeration
|
||||
return SendMagicLinkResponse(
|
||||
status="success",
|
||||
message="If this email is registered, you will receive a sign-in link shortly."
|
||||
)
|
||||
|
||||
ip_address = http_request.client.host if http_request.client else None
|
||||
if ip_address and email_validation.is_ip_blocked(ip_address):
|
||||
logger.warning(f"Blocked IP signup attempt: {ip_address}")
|
||||
return SendMagicLinkResponse(
|
||||
status="success",
|
||||
message="If this email is registered, you will receive a sign-in link shortly."
|
||||
)
|
||||
```
|
||||
|
||||
Note: Blocked emails/IPs get silent rejection (anti-enumeration). Disposable domains get explicit rejection (we want the user to switch to a real email).
|
||||
|
||||
### Step 3: Backend — Add `emailhook.site` to default disposable domains
|
||||
|
||||
**File:** `backend/services/email_validation_service.py`
|
||||
|
||||
Add `emailhook.site` to `DEFAULT_DISPOSABLE_DOMAINS` set.
|
||||
|
||||
### Step 4: Frontend — Handle the disposable email error in AuthDialog
|
||||
|
||||
**File:** `frontend/components/auth/AuthDialog.tsx`
|
||||
|
||||
The `sendMagicLink` call in `auth.ts` already catches errors and sets `error` from the API response message. The `handleResponse` in `api.ts` throws an `ApiError` with the `detail` string for non-200 responses.
|
||||
|
||||
Update `handleSendMagicLink` to detect the `disposable_email_not_allowed` error code and show a friendly multi-line message instead of the terse error string:
|
||||
|
||||
```tsx
|
||||
const success = await sendMagicLink(email.trim().toLowerCase())
|
||||
if (success) {
|
||||
setStep("sent")
|
||||
}
|
||||
// If error contains our specific code, override with friendly message
|
||||
// (the error state is already set by the auth store)
|
||||
```
|
||||
|
||||
We need to detect the specific error. The cleanest approach: check the error string in the `displayError` rendering. If it matches `disposable_email_not_allowed`, render a custom friendly block instead of the red error text.
|
||||
|
||||
The friendly message:
|
||||
|
||||
> **We don't support disposable email addresses**
|
||||
>
|
||||
> We'd love to help you create karaoke videos! Please use your regular email address so we can send you a notification when your video is ready.
|
||||
>
|
||||
> We promise — no spam, no marketing emails, ever. We only email you about your karaoke videos.
|
||||
|
||||
### Step 5: Backend tests
|
||||
|
||||
**File:** `backend/tests/test_email_validation_integration.py` (new)
|
||||
|
||||
Test cases:
|
||||
1. Magic link with disposable email returns 422 with `disposable_email_not_allowed`
|
||||
2. Magic link with legitimate email proceeds normally
|
||||
3. Magic link with blocked email returns silent success (no actual email sent)
|
||||
4. Magic link with blocked IP returns silent success
|
||||
|
||||
### Step 6: Frontend tests
|
||||
|
||||
**File:** `frontend/components/__tests__/AuthDialog.test.tsx` (new or extend existing)
|
||||
|
||||
Test cases:
|
||||
1. Submitting a disposable email shows the friendly rejection message
|
||||
2. The friendly message contains the key copy (no spam promise, use real email)
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/api/routes/users.py` | Modify | Add email validation checks to magic link endpoint |
|
||||
| `backend/services/email_validation_service.py` | Modify | Add `emailhook.site` to default domains |
|
||||
| `frontend/components/auth/AuthDialog.tsx` | Modify | Handle disposable email error with friendly message |
|
||||
| `backend/tests/test_email_validation_integration.py` | Create | Integration tests for the validation in signup flow |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests:** Validate `is_disposable_domain` catches yopmail, emailhook.site (already covered in existing tests)
|
||||
- **Integration tests:** New test file verifying the magic link endpoint rejects disposable emails with 422
|
||||
- **Frontend tests:** Verify the AuthDialog shows friendly message on disposable email error
|
||||
- **Manual verification:** Try signing up with `test@yopmail.com` on dev server to see the UX
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Remove the validation check from `send_magic_link()` in `users.py`. The `EmailValidationService` itself remains unchanged (it's already deployed and used by admin routes).
|
||||
@@ -0,0 +1,111 @@
|
||||
# Plan: Credit Refund on Job Deletion and Cancellation
|
||||
|
||||
**Created:** 2026-03-04
|
||||
**Branch:** feat/sess-20260304-1308-investigate-user-credits
|
||||
**Status:** Implemented
|
||||
|
||||
## Overview
|
||||
|
||||
When a user deletes or cancels a job, their credit is not refunded. This was discovered when user john2011@sky.com deleted a job 16 seconds after creation (accidental duplicate) and lost the credit. Job *failures* already auto-refund via `mark_job_failed()` → `refund_credit()`, but deletions and cancellations do not.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Job failures already refund credits (existing behavior, no change needed)
|
||||
- [ ] Job cancellation refunds 1 credit
|
||||
- [ ] Job deletion refunds 1 credit (if not already completed)
|
||||
- [ ] No refund for completed jobs (user got their video)
|
||||
- [ ] No refund for admin-owned jobs (consistent with failure refund logic)
|
||||
- [ ] Idempotency: prevent double-refund if a job is cancelled then deleted, or fails then deleted
|
||||
- [ ] Credit transaction records the reason clearly ("job_cancelled", "job_deleted")
|
||||
- [ ] Refund failure must not block the cancel/delete operation
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Key Insight: Idempotency via Job Field
|
||||
|
||||
The current `refund_credit()` has no guard against double-refund for the same job. Since a job can transition through cancelled → deleted, or failed → deleted, we need a way to track that a refund was already issued.
|
||||
|
||||
**Approach**: Add a `credit_refunded` boolean field to the Job model. Check it before refunding. This is simpler and more reliable than scanning the user's transaction history.
|
||||
|
||||
### What Should and Shouldn't Be Refunded
|
||||
|
||||
| Job Status at Time of Action | Delete | Cancel | Notes |
|
||||
|------------------------------|--------|--------|-------|
|
||||
| PENDING through pre-COMPLETE | Refund | Refund | Job didn't deliver value |
|
||||
| COMPLETE / PREP_COMPLETE | No refund | N/A (can't cancel terminal) | User got their output |
|
||||
| FAILED | No refund (already refunded) | N/A (can't cancel terminal) | Already refunded by fail_job |
|
||||
| CANCELLED | No refund (already refunded) | N/A (already cancelled) | Already refunded by cancel |
|
||||
|
||||
For deletion: we need to read the job status *before* deleting it to decide whether to refund. The delete endpoint already fetches the job for ownership check, so this is straightforward.
|
||||
|
||||
### Refund Helper
|
||||
|
||||
Extract the refund logic (currently in `mark_job_failed`) into a reusable private method `_refund_credit_for_job()` that:
|
||||
1. Checks `job.credit_refunded` — skip if already refunded
|
||||
2. Checks `job.user_email` — skip if no user
|
||||
3. Checks if admin — skip if admin job
|
||||
4. Calls `user_service.refund_credit()` with appropriate reason
|
||||
5. Sets `credit_refunded = True` on the job (for cancel; for delete the job is about to be removed but we set it anyway for the brief window)
|
||||
6. Wraps everything in try/except so failures don't block the parent operation
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. [ ] **Add `credit_refunded` field to Job model** (`backend/models/job.py`)
|
||||
- Add `credit_refunded: bool = False` to the Job model
|
||||
- No migration needed — Firestore is schemaless, defaults to False for existing docs
|
||||
|
||||
2. [ ] **Extract refund helper in JobManager** (`backend/services/job_manager.py`)
|
||||
- Create `_refund_credit_for_job(self, job_id: str, job: Job, reason: str) -> bool`
|
||||
- Move the refund logic from `mark_job_failed` (lines 821-839) into this helper
|
||||
- Add `credit_refunded` check at the top
|
||||
- Update `mark_job_failed` to call the helper with `reason="job_failed"`
|
||||
|
||||
3. [ ] **Add refund to `cancel_job()`** (`backend/services/job_manager.py`)
|
||||
- After setting status to CANCELLED, call `_refund_credit_for_job(job_id, job, reason="job_cancelled")`
|
||||
- The job object is already fetched at line 1017
|
||||
|
||||
4. [ ] **Add refund to `delete_job()`** (`backend/services/job_manager.py`)
|
||||
- Need to fetch the job (already done when `delete_files=True`, but not always)
|
||||
- Before deletion, check if job status is NOT in `[COMPLETE, PREP_COMPLETE, FAILED, CANCELLED]`
|
||||
- If eligible, call `_refund_credit_for_job(job_id, job, reason="job_deleted")`
|
||||
- For FAILED/CANCELLED: skip because refund was already issued during that transition
|
||||
- For COMPLETE/PREP_COMPLETE: skip because user received their output
|
||||
|
||||
5. [ ] **Update `mark_job_failed` to set `credit_refunded`**
|
||||
- The helper handles this, but also update the Firestore doc
|
||||
|
||||
6. [ ] **Add unit tests** (`backend/tests/test_credit_enforcement.py`)
|
||||
- Test: cancel_job refunds credit for non-admin user
|
||||
- Test: cancel_job skips refund for admin jobs
|
||||
- Test: delete_job refunds credit for non-complete, non-terminal jobs
|
||||
- Test: delete_job does NOT refund for completed jobs
|
||||
- Test: delete_job does NOT double-refund for already-failed jobs
|
||||
- Test: delete_job does NOT double-refund for already-cancelled jobs
|
||||
- Test: refund failure doesn't block cancel/delete
|
||||
- Test: credit_refunded flag prevents double refund
|
||||
|
||||
7. [ ] **Backfill: Refund john2011@sky.com** (manual one-time)
|
||||
- Grant 1 credit for the deleted job 40dcca89 via admin API or direct Firestore update
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/job.py` | Modify | Add `credit_refunded: bool = False` field |
|
||||
| `backend/services/job_manager.py` | Modify | Extract `_refund_credit_for_job()` helper, add refund to `cancel_job()` and `delete_job()` |
|
||||
| `backend/tests/test_credit_enforcement.py` | Modify | Add tests for cancel/delete refund scenarios |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests**: Extend `test_credit_enforcement.py` with new test classes `TestCreditRefundOnJobCancellation` and `TestCreditRefundOnJobDeletion`, following the exact pattern of existing `TestCreditRefundOnJobFailure`
|
||||
- **Manual verification**: After deploy, cancel and delete test jobs and verify credit balance updates
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [ ] Should we notify the user (email/in-app) when a credit is refunded? (Suggest: no, keep it simple for now — the credit just appears back in their balance)
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
- The `credit_refunded` field is additive (defaults to False for old jobs)
|
||||
- If issues arise, revert the PR — old behavior (no refund on cancel/delete) is restored
|
||||
- Any refunds already issued are just bonus credits to users, no harm done
|
||||
@@ -0,0 +1,407 @@
|
||||
# Plan: Add Retry Logic for Flacfetch Download Failures
|
||||
|
||||
**Created:** 2026-02-10
|
||||
**Branch:** feat/sess-20260210-1158-investigate-job-749141f8
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Job 749141f8 failed with error: "Audio download failed: Remote download by ID failed: Download by ID request failed:"
|
||||
|
||||
Investigation revealed **multiple related issues** when flacfetch network calls fail:
|
||||
|
||||
1. **No retry logic** - A single network hiccup causes immediate job failure
|
||||
2. **Missing source info** - When download fails, source_id/source_name aren't saved to job
|
||||
3. **Broken retry button** - Can't retry because no source info was persisted
|
||||
4. **Silent UI failures** - Frontend doesn't show retry error messages to user
|
||||
|
||||
### Root Cause
|
||||
|
||||
Audio search flow saves selection info to `state_data`, but doesn't save download parameters (source_id, source_name, etc.) to the job document until AFTER download succeeds. If the flacfetch call fails (network error), the job fails with no way to retry.
|
||||
|
||||
### Solution
|
||||
|
||||
This plan fixes all four issues:
|
||||
- Add retry logic to flacfetch client (network resilience)
|
||||
- Persist download info before attempting download (enables retry)
|
||||
- Update retry endpoint to handle audio search jobs
|
||||
- Show error messages in frontend when retry fails
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements - Flacfetch Retry Logic
|
||||
- [x] Retry `download_by_id()` on `httpx.RequestError` (network failures)
|
||||
- [x] Use exponential backoff spread over ~5 minutes
|
||||
- [x] Configurable max retries via environment variable
|
||||
- [x] Log each retry attempt with context
|
||||
- [x] Do NOT retry on HTTP 4xx errors (client errors - immediate failure)
|
||||
- [x] Do retry on HTTP 5xx errors (server errors - transient)
|
||||
|
||||
### Functional Requirements - Persist Download Info
|
||||
- [x] Save source_id, source_name, target_file, download_url to job document
|
||||
- [x] Save BEFORE attempting download (not after)
|
||||
- [x] Store in top-level fields for easy retry access
|
||||
- [x] Preserve existing state_data fields (selected_audio_index, etc.)
|
||||
|
||||
### Functional Requirements - Retry Endpoint
|
||||
- [x] Add case for audio search jobs in `/jobs/{id}/retry` endpoint
|
||||
- [x] Check for saved download params (source_id, source_name)
|
||||
- [x] Re-download using saved params
|
||||
- [x] Clear error state before retrying
|
||||
- [x] Trigger appropriate workers after successful download
|
||||
|
||||
### Functional Requirements - Frontend Error Handling
|
||||
- [x] Show toast notification when retry fails
|
||||
- [x] Display actual error message from API
|
||||
- [x] Don't just console.error - make it visible to user
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [x] Minimal code changes (use existing patterns)
|
||||
- [x] Clear error messages that include retry context
|
||||
- [x] Backwards compatible (no breaking changes)
|
||||
- [x] Should not significantly increase total timeout duration
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Architecture Decisions
|
||||
|
||||
**Option 1: Tenacity library (Recommended)**
|
||||
- ✅ Industry standard, well-tested
|
||||
- ✅ Declarative retry configuration
|
||||
- ✅ Built-in exponential backoff
|
||||
- ✅ Already used in project (check `poetry.lock`)
|
||||
- ❌ New dependency if not already present
|
||||
|
||||
**Option 2: Manual retry loop**
|
||||
- ✅ No new dependencies
|
||||
- ✅ Full control over logic
|
||||
- ❌ More code to maintain
|
||||
- ❌ Easy to get wrong (edge cases)
|
||||
|
||||
**Decision:** Use **tenacity** if already in dependencies, otherwise implement manual retry loop.
|
||||
|
||||
### Retry Strategy
|
||||
|
||||
```python
|
||||
# Retry on network errors only (RequestError)
|
||||
# Do NOT retry on:
|
||||
# - HTTP 4xx (client errors like 400, 401, 404)
|
||||
# - Already completed downloads
|
||||
# Do retry on:
|
||||
# - httpx.RequestError (connection timeout, refused, DNS failure)
|
||||
# - HTTP 5xx (server errors - service temporarily down)
|
||||
|
||||
Max retries: 8 (total 9 attempts)
|
||||
Backoff: Exponential with cap, multiplier=2, min=10s, max=60s
|
||||
- Attempt 1: immediate
|
||||
- Attempt 2: wait 10s
|
||||
- Attempt 3: wait 20s
|
||||
- Attempt 4: wait 40s
|
||||
- Attempt 5: wait 60s (capped)
|
||||
- Attempt 6: wait 60s (capped)
|
||||
- Attempt 7: wait 60s (capped)
|
||||
- Attempt 8: wait 60s (capped)
|
||||
- Attempt 9: wait 60s (capped)
|
||||
Total max time: ~370s (6+ minutes) of backoff + request timeouts
|
||||
|
||||
This gives the flacfetch service time to recover from restarts,
|
||||
network issues, or temporary VM problems.
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Add to `backend/config.py`:
|
||||
```python
|
||||
# Flacfetch client retry settings
|
||||
flacfetch_retry_max_attempts: int = int(os.getenv("FLACFETCH_RETRY_MAX_ATTEMPTS", "9")) # Total attempts (1 initial + 8 retries)
|
||||
flacfetch_retry_min_wait: float = float(os.getenv("FLACFETCH_RETRY_MIN_WAIT", "10.0")) # seconds
|
||||
flacfetch_retry_max_wait: float = float(os.getenv("FLACFETCH_RETRY_MAX_WAIT", "60.0")) # seconds
|
||||
```
|
||||
|
||||
### Error Message Improvements
|
||||
|
||||
Change from:
|
||||
```
|
||||
Download by ID request failed:
|
||||
```
|
||||
|
||||
To:
|
||||
```
|
||||
Download by ID request failed after 4 attempts: [ConnectTimeout] Connection to http://10.0.0.5:8080 timed out
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Check Dependencies
|
||||
1. [ ] Check if `tenacity` is in `pyproject.toml` dependencies
|
||||
2. [ ] If not present, add `tenacity = "^8.2.0"` to dependencies
|
||||
3. [ ] Run `poetry lock` if dependencies changed
|
||||
|
||||
### Phase 2: Add Configuration
|
||||
4. [ ] Add retry configuration to `backend/config.py`
|
||||
- `flacfetch_retry_max_attempts = 9`
|
||||
- `flacfetch_retry_min_wait = 10.0`
|
||||
- `flacfetch_retry_max_wait = 60.0`
|
||||
|
||||
### Phase 3: Implement Flacfetch Retry Logic
|
||||
5. [ ] Import tenacity in `backend/services/flacfetch_client.py`
|
||||
6. [ ] Create retry decorator with:
|
||||
- Retry on `httpx.RequestError`
|
||||
- Retry on `httpx.HTTPStatusError` with 5xx status
|
||||
- Stop on `httpx.HTTPStatusError` with 4xx status
|
||||
- Exponential backoff with cap at 60s
|
||||
- Before-sleep logging callback
|
||||
7. [ ] Apply decorator to `download_by_id()` method
|
||||
8. [ ] Improve error messages to include:
|
||||
- Number of attempts made
|
||||
- Original exception type and message
|
||||
|
||||
### Phase 4: Persist Download Info Before Download
|
||||
9. [ ] Update `audio_search.py:_download_and_start_processing()`
|
||||
10. [ ] After getting source info from selected result (line 307-311)
|
||||
11. [ ] **Save to job document BEFORE download attempt:**
|
||||
```python
|
||||
job_manager.update_job(job_id, {
|
||||
'audio_source_type': 'audio_search',
|
||||
'source_name': source_name,
|
||||
'source_id': source_id,
|
||||
'target_file': target_file,
|
||||
'download_url': download_url,
|
||||
})
|
||||
```
|
||||
12. [ ] Then proceed with download (existing code)
|
||||
13. [ ] Add to Job model if needed (check if fields exist)
|
||||
|
||||
### Phase 5: Update Retry Endpoint for Audio Search
|
||||
14. [ ] Update `backend/api/routes/jobs.py:retry_job()`
|
||||
15. [ ] Add new case before "no input audio" error (line ~1088)
|
||||
16. [ ] Check if job has saved download params:
|
||||
```python
|
||||
elif job.audio_source_type == 'audio_search' and job.source_id:
|
||||
# Has audio search download info - retry download
|
||||
```
|
||||
17. [ ] Re-call audio search download logic with saved params
|
||||
18. [ ] Or refactor download into reusable function
|
||||
|
||||
### Phase 6: Frontend Error Handling
|
||||
19. [ ] Update `frontend/components/job/JobActions.tsx`
|
||||
20. [ ] Add toast notification library (check if exists)
|
||||
21. [ ] In `handleRetry()` catch block, show toast:
|
||||
```typescript
|
||||
catch (error) {
|
||||
const message = error.response?.data?.detail || "Failed to retry job"
|
||||
toast.error(message)
|
||||
console.error("Failed to retry job:", error)
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 7: Testing - Flacfetch Retry
|
||||
22. [ ] Add unit tests to `backend/tests/test_flacfetch_client.py`
|
||||
23. [ ] Test retry on RequestError (mock fails N times then succeeds)
|
||||
24. [ ] Test no retry on 4xx errors
|
||||
25. [ ] Test retry on 5xx errors
|
||||
26. [ ] Test max retries exhausted
|
||||
27. [ ] Verify logging output includes attempt numbers
|
||||
|
||||
### Phase 8: Testing - End-to-End
|
||||
28. [ ] Test audio search job with flacfetch down
|
||||
29. [ ] Verify source info is saved even when download fails
|
||||
30. [ ] Verify retry button works after source info saved
|
||||
31. [ ] Test retry succeeds when flacfetch comes back up
|
||||
32. [ ] Verify error toast appears when retry fails
|
||||
|
||||
### Phase 9: Documentation
|
||||
33. [ ] Update `backend/services/flacfetch_client.py` docstring
|
||||
34. [ ] Add to `docs/LESSONS-LEARNED.md`:
|
||||
- "Always add retries for external service calls"
|
||||
- "Save operation parameters before attempting operations"
|
||||
- "Persist enough state to enable retry from any failure point"
|
||||
35. [ ] Update `docs/ARCHITECTURE.md` if needed
|
||||
36. [ ] Document the new job fields in API.md
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/config.py` | Modify | Add retry configuration variables |
|
||||
| `backend/services/flacfetch_client.py` | Modify | Add retry logic to `download_by_id()` |
|
||||
| `backend/api/routes/audio_search.py` | Modify | Save download info before attempting download |
|
||||
| `backend/api/routes/jobs.py` | Modify | Add audio search case to retry endpoint |
|
||||
| `backend/models/job.py` | Modify (maybe) | Add fields: source_name, source_id, target_file, download_url |
|
||||
| `frontend/components/job/JobActions.tsx` | Modify | Show error toast when retry fails |
|
||||
| `backend/tests/test_flacfetch_client.py` | Modify | Add retry tests |
|
||||
| `backend/tests/test_audio_search.py` | Modify | Test download info persistence |
|
||||
| `backend/tests/test_jobs.py` | Modify | Test retry with audio search jobs |
|
||||
| `pyproject.toml` | Modify (maybe) | Add tenacity if not present |
|
||||
| `docs/LESSONS-LEARNED.md` | Modify | Document patterns |
|
||||
| `docs/API.md` | Modify | Document new job fields |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests (`backend/tests/test_flacfetch_client.py`)
|
||||
|
||||
1. **Test retry on RequestError**
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_by_id_retries_on_network_error():
|
||||
"""Should retry 3 times on connection errors."""
|
||||
client = FlacfetchClient(base_url="http://test", api_key="key")
|
||||
|
||||
with patch("httpx.AsyncClient.post") as mock_post:
|
||||
# Fail 2 times, then succeed
|
||||
mock_post.side_effect = [
|
||||
httpx.RequestError("Connection timeout"),
|
||||
httpx.RequestError("Connection refused"),
|
||||
Mock(json=lambda: {"download_id": "test123"}, status_code=200),
|
||||
]
|
||||
|
||||
result = await client.download_by_id("YouTube", "abc123")
|
||||
|
||||
assert result == "test123"
|
||||
assert mock_post.call_count == 3
|
||||
```
|
||||
|
||||
2. **Test no retry on 4xx errors**
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_by_id_no_retry_on_client_error():
|
||||
"""Should NOT retry on 400/401/404 errors."""
|
||||
client = FlacfetchClient(base_url="http://test", api_key="key")
|
||||
|
||||
with patch("httpx.AsyncClient.post") as mock_post:
|
||||
response = Mock(status_code=404, text="Not found")
|
||||
response.raise_for_status.side_effect = httpx.HTTPStatusError(
|
||||
"Not found", request=Mock(), response=response
|
||||
)
|
||||
mock_post.return_value = response
|
||||
|
||||
with pytest.raises(FlacfetchServiceError, match="404"):
|
||||
await client.download_by_id("YouTube", "abc123")
|
||||
|
||||
assert mock_post.call_count == 1 # No retries
|
||||
```
|
||||
|
||||
3. **Test retry on 5xx errors**
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_by_id_retries_on_server_error():
|
||||
"""Should retry on 500/503 errors."""
|
||||
# Similar to test 1 but with HTTP 503 errors
|
||||
```
|
||||
|
||||
4. **Test max retries exhausted**
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_by_id_fails_after_max_retries():
|
||||
"""Should fail with clear message after max retries."""
|
||||
# All attempts fail, verify error message includes attempt count
|
||||
```
|
||||
|
||||
### Integration Tests
|
||||
|
||||
5. **Test with actual flacfetch service** (if available in test environment)
|
||||
- Verify retry doesn't break normal operation
|
||||
- Test with deliberate network issues
|
||||
|
||||
### Manual Testing
|
||||
|
||||
6. **Simulate network failure**
|
||||
```bash
|
||||
# Temporarily stop flacfetch VM
|
||||
gcloud compute instances stop flacfetch-service --zone=us-central1-a
|
||||
|
||||
# Submit a test job
|
||||
# Observe logs showing retry attempts
|
||||
|
||||
# Restart VM during retries
|
||||
gcloud compute instances start flacfetch-service --zone=us-central1-a
|
||||
|
||||
# Job should succeed
|
||||
```
|
||||
|
||||
7. **Test with intentional delays**
|
||||
- Add sleep in flacfetch API handler
|
||||
- Verify retries don't cause issues
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] ~~Is tenacity already in dependencies?~~ (Will check during implementation)
|
||||
- [x] ~~What's the appropriate total timeout budget?~~ - 9 attempts over ~6 minutes
|
||||
- [x] ~~Should we fix the Retry button UX in this PR?~~ - YES, it's the same root cause
|
||||
- [x] ~~Why did job 749141f8 fail before saving audio source info?~~ - Download info saved AFTER download, not before
|
||||
- [ ] Should we also add retries to `wait_for_download()`?
|
||||
- Probably NO - that already has its own polling/timeout logic
|
||||
- [ ] Should we add circuit breaker pattern?
|
||||
- Probably NO for now - keep it simple
|
||||
- Could add in future if we see cascading failures
|
||||
- [ ] Should retry logic be applied to other flacfetch methods?
|
||||
- `start_download()` - probably YES (same pattern)
|
||||
- `search()` - maybe YES (search can also have network issues)
|
||||
- `get_download_status()` - probably NO (polling already handles this)
|
||||
- **Decision**: Apply to `start_download()` too, skip `search()` for now
|
||||
- [ ] Which toast library should frontend use?
|
||||
- Check if already exists (react-hot-toast, sonner, etc.)
|
||||
- If not, add lightweight library
|
||||
- [ ] Should we save download params to state_data or top-level fields?
|
||||
- Top-level fields more visible and easier to query
|
||||
- **Decision**: Top-level fields (source_id, source_name, etc.)
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If this causes issues:
|
||||
|
||||
1. **Quick rollback**: Set env var `FLACFETCH_RETRY_MAX_ATTEMPTS=1` in Cloud Run
|
||||
- Disables retries without code change
|
||||
- Can deploy in ~2 minutes
|
||||
|
||||
2. **Full rollback**: Revert the PR
|
||||
- Git revert the merge commit
|
||||
- Deploy previous version
|
||||
|
||||
3. **Monitoring**: Watch for:
|
||||
- Increased request duration to flacfetch
|
||||
- Jobs timing out at higher layer (Cloud Tasks, Cloud Run)
|
||||
- Duplicate download attempts causing issues
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Jobs with transient flacfetch failures succeed instead of failing
|
||||
- [ ] Error messages are more informative (show attempts and root cause)
|
||||
- [ ] No increase in false positives (jobs that should fail, now hang)
|
||||
- [ ] All tests pass
|
||||
- [ ] No impact on successful jobs (retry overhead is minimal)
|
||||
|
||||
## Related Issues
|
||||
|
||||
### Job 749141f8 - The Incident
|
||||
|
||||
**What happened:**
|
||||
- User searched for audio via audio search
|
||||
- Selected "Elizabeth Moen - Songbird"
|
||||
- Backend tried to download via flacfetch
|
||||
- Flacfetch call failed with network error (connection timeout/refused)
|
||||
- Job immediately failed
|
||||
- No source info saved → Retry button broken
|
||||
- User sees "Retry" button but clicking does nothing (silent 400 error)
|
||||
|
||||
**How this fix helps:**
|
||||
1. ✅ Flacfetch retry logic → Download would have succeeded after retry
|
||||
2. ✅ Persist download info → Even if all retries fail, user can retry later
|
||||
3. ✅ Retry endpoint fix → Retry button will actually work
|
||||
4. ✅ Frontend toast → User sees helpful error if retry still fails
|
||||
|
||||
**Note:** Job 749141f8 cannot be fixed retroactively (no source info saved). But future jobs with same issue will be fixable via retry button.
|
||||
|
||||
### Future Work (Not in This PR)
|
||||
|
||||
Similar pattern should be applied to other external service calls:
|
||||
- AudioShake API (lyrics transcription)
|
||||
- Modal API (audio separation)
|
||||
- YouTube direct API calls (not via flacfetch)
|
||||
- Email sending (SendGrid)
|
||||
|
||||
## References
|
||||
|
||||
- [Tenacity documentation](https://tenacity.readthedocs.io/)
|
||||
- [LESSONS-LEARNED.md](docs/LESSONS-LEARNED.md) - External service response format mismatches
|
||||
- [flacfetch README](../../flacfetch/README.md) - Understanding the service
|
||||
@@ -0,0 +1,149 @@
|
||||
# Plan: Improve Job Observability
|
||||
|
||||
**Created:** 2026-03-07
|
||||
**Branch:** feat/sess-20260307-2216-improve-job-observability
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
After shipping the "edit completed tracks" feature, we found it very difficult to trace what actually happened to edited jobs. Timeline events are too sparse (no output details), cleanup operations aren't logged to Cloud Logging or Firestore, state_data gets fully overwritten during re-processing (losing history of previous outputs), and API endpoints don't write to the job's Firestore log subcollection.
|
||||
|
||||
The goal is to make it trivially easy for future agents and humans to reconstruct the complete lifecycle of any job by reading its timeline and Firestore logs.
|
||||
|
||||
## Problem Analysis
|
||||
|
||||
### What we found investigating two edited jobs:
|
||||
|
||||
1. **Timeline events only have `status`, `timestamp`, `progress`, `message`** — no output details like YouTube video IDs, brand codes, Dropbox links, or GDrive file IDs
|
||||
2. **Edit endpoint cleanup results never logged** — the `cleanup_results` dict is built and returned in the HTTP response but never logged to Cloud Logging or the job's Firestore log subcollection
|
||||
3. **Distribution details incomplete in logs** — YouTube logs the URL but not video_id separately; Dropbox doesn't log folder path; GDrive doesn't log file IDs
|
||||
4. **`state_data` fully overwritten on re-processing** — `video_worker.py:342-354` does `{**job.state_data, ...}`, so any `DELETE_FIELD` operations from the edit endpoint get lost when old keys are re-merged from the in-memory copy
|
||||
5. **`outputs_deleted_at` cleared by video worker** — intentionally, but loses audit trail
|
||||
6. **No Firestore job logs from API endpoints** — edit/admin endpoints only log to Cloud Run stdout, not to `jobs/{job_id}/logs/`
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] Add `metadata` dict to `TimelineEvent` for structured output details
|
||||
- [ ] Log all cleanup operations (success AND failure) to both Cloud Logging and Firestore job logs
|
||||
- [ ] Log distribution output details (YouTube video ID, Dropbox path, brand code, GDrive file IDs) in timeline events and Firestore logs
|
||||
- [ ] Preserve previous output details in timeline when a job is edited (snapshot before cleanup)
|
||||
- [ ] Add helper to write Firestore job logs from API endpoints (not just workers)
|
||||
- [ ] Ensure edit/admin delete endpoints write to Firestore job logs
|
||||
- [ ] No breaking changes to existing timeline format (additive only)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### 1. Add `metadata` field to `TimelineEvent`
|
||||
|
||||
The `TimelineEvent` model gains an optional `metadata: Optional[Dict[str, Any]]` field. This allows any timeline event to carry structured data without changing the schema. The field is additive — existing events without metadata continue to work.
|
||||
|
||||
Key events that gain metadata:
|
||||
- **`complete`**: `{youtube_url, youtube_video_id, dropbox_link, brand_code, gdrive_file_ids}`
|
||||
- **Edit initiated**: `{previous_outputs: {youtube_url, dropbox_link, brand_code, gdrive_files}, cleanup_results: {...}}`
|
||||
- **Outputs deleted**: `{deleted_outputs: {...}, cleanup_results: {...}}`
|
||||
|
||||
### 2. Write Firestore job logs from API endpoints
|
||||
|
||||
Create a simple helper function that API route handlers can call to append logs to the job's Firestore subcollection. Workers already have `JobLogAdapter` — we just need a lightweight equivalent for API routes:
|
||||
|
||||
```python
|
||||
def log_to_job(job_id: str, worker: str, level: str, message: str, metadata: dict = None):
|
||||
"""Write a log entry to the job's Firestore log subcollection."""
|
||||
entry = WorkerLogEntry.create(job_id, worker, level, message, metadata)
|
||||
FirestoreService().append_log_to_subcollection(job_id, entry)
|
||||
```
|
||||
|
||||
### 3. Enhance distribution logging in video_worker_orchestrator
|
||||
|
||||
After each distribution upload succeeds, log the details with structured metadata:
|
||||
- YouTube: video_id, video_url, channel
|
||||
- Dropbox: folder_path, sharing_link
|
||||
- GDrive: file_ids dict
|
||||
- Brand code: code, prefix, allocated vs recycled
|
||||
|
||||
### 4. Snapshot previous outputs on edit
|
||||
|
||||
Before the edit endpoint clears state_data distribution keys, capture the current values in the timeline event's metadata. This preserves the history permanently in the timeline array (which is append-only via `ArrayUnion`).
|
||||
|
||||
### 5. Log cleanup results from edit/admin endpoints
|
||||
|
||||
After cleanup runs, log each step's result to both Cloud Logging and the Firestore job log subcollection with structured metadata.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Step 1: Add `metadata` to `TimelineEvent` model
|
||||
- [ ] Add `metadata: Optional[Dict[str, Any]] = None` to `TimelineEvent` in `backend/models/job.py`
|
||||
- [ ] No migration needed — Firestore is schemaless, existing events just won't have the field
|
||||
|
||||
### Step 2: Create `log_to_job` helper
|
||||
- [ ] Add `log_to_job()` function in `backend/services/firestore_service.py` (or a small utility module)
|
||||
- [ ] Takes `job_id`, `worker`, `level`, `message`, optional `metadata`
|
||||
- [ ] Wraps `WorkerLogEntry.create()` + `append_log_to_subcollection()`
|
||||
- [ ] Catches and suppresses errors (logging should never break the caller)
|
||||
|
||||
### Step 3: Update `transition_to_state` to accept metadata
|
||||
- [ ] Add optional `timeline_metadata: Optional[Dict[str, Any]] = None` parameter to `job_manager.transition_to_state()`
|
||||
- [ ] Pass through to `update_job_status()` → `TimelineEvent` construction
|
||||
- [ ] Update `firestore_service.update_job_status()` to accept and pass `timeline_metadata`
|
||||
|
||||
### Step 4: Enhance edit endpoint logging
|
||||
- [ ] Before cleanup: snapshot current distribution outputs (`youtube_url`, `dropbox_link`, `brand_code`, `gdrive_files`) into `previous_outputs` dict
|
||||
- [ ] After each cleanup step: log result to Cloud Logging AND call `log_to_job()`
|
||||
- [ ] After all cleanup: include `previous_outputs` and `cleanup_results` in the timeline event metadata
|
||||
- [ ] The "Track edit initiated" timeline event gains metadata: `{initiated_by, previous_outputs, cleanup_results}`
|
||||
|
||||
### Step 5: Enhance admin delete-outputs endpoint logging
|
||||
- [ ] Same pattern as edit endpoint: snapshot outputs, log each cleanup step, include in timeline metadata
|
||||
- [ ] The "Outputs deleted by admin" timeline event gains metadata: `{deleted_by, deleted_outputs, cleanup_results}`
|
||||
|
||||
### Step 6: Enhance distribution logging in video_worker_orchestrator
|
||||
- [ ] YouTube upload success: log video_id and video_url with metadata `{video_id, video_url}`
|
||||
- [ ] Dropbox upload success: log folder_path and sharing_link with metadata `{folder_path, sharing_link}`
|
||||
- [ ] GDrive upload success: log file_ids with metadata `{file_ids: {...}}`
|
||||
- [ ] Brand code allocation: log with metadata `{brand_code, source: "allocated"|"recycled"}`
|
||||
|
||||
### Step 7: Add output details to `complete` timeline event
|
||||
- [ ] When `video_worker.py` calls `transition_to_state(COMPLETE)`, pass `timeline_metadata` with distribution output details
|
||||
- [ ] Include: `youtube_url`, `youtube_video_id`, `dropbox_link`, `brand_code`, `gdrive_file_ids`, `edit_count`
|
||||
|
||||
### Step 8: Tests
|
||||
- [ ] Unit test: `TimelineEvent` with metadata serializes correctly
|
||||
- [ ] Unit test: `log_to_job` writes to subcollection
|
||||
- [ ] Unit test: edit endpoint includes `previous_outputs` and `cleanup_results` in timeline event
|
||||
- [ ] Unit test: `complete` transition includes output metadata
|
||||
- [ ] Verify existing tests still pass
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/job.py` | Modify | Add `metadata` field to `TimelineEvent` |
|
||||
| `backend/services/firestore_service.py` | Modify | Add `log_to_job()` helper, update `update_job_status()` to accept timeline metadata |
|
||||
| `backend/services/job_manager.py` | Modify | Add `timeline_metadata` param to `transition_to_state()` |
|
||||
| `backend/api/routes/jobs.py` | Modify | Enhance edit endpoint with cleanup logging, output snapshots, timeline metadata |
|
||||
| `backend/api/routes/admin.py` | Modify | Enhance delete-outputs endpoint similarly |
|
||||
| `backend/workers/video_worker_orchestrator.py` | Modify | Add structured metadata to distribution log messages |
|
||||
| `backend/workers/video_worker.py` | Modify | Pass output metadata in `complete` transition (both orchestrator and legacy paths) |
|
||||
| `backend/tests/test_edit_completed_track.py` | Modify | Verify timeline metadata in edit tests |
|
||||
| `backend/tests/test_observability.py` | Create | Tests for TimelineEvent metadata and log_to_job |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests**: TimelineEvent metadata serialization, log_to_job helper, edit endpoint timeline metadata assertions
|
||||
- **Existing test verification**: `make test` must pass — changes are additive so existing tests should be unaffected
|
||||
- **No E2E tests needed**: This is purely backend logging/observability — no UI changes
|
||||
|
||||
## What This Does NOT Include
|
||||
|
||||
- **Dedicated admin UI for viewing job logs** — existing admin page already shows timeline
|
||||
- **Log aggregation/search UI** — Cloud Logging and Firestore viewer already serve this purpose
|
||||
- **Changing state_data overwrite behavior** — the `{**job.state_data, ...}` pattern in video_worker.py is intentional; instead we preserve history in timeline events
|
||||
- **Changing outputs_deleted_at clearing** — video worker clearing it after re-processing is correct behavior; the edit history lives in timeline metadata instead
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
All changes are additive (new optional field on TimelineEvent, additional log writes). Rolling back just means deploying the previous version — no data migration needed.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None — the approach is straightforward and low-risk.
|
||||
@@ -0,0 +1,279 @@
|
||||
# Plan: Private (Non-Published) Tracks
|
||||
|
||||
**Created:** 2026-02-24
|
||||
**Branch:** feat/sess-20260224-1149-private-tracks
|
||||
**Status:** Implemented
|
||||
|
||||
## Overview
|
||||
|
||||
Add a `is_private` flag to jobs that changes distribution behavior:
|
||||
- **Dropbox**: Uses `Tracks-NonPublished` folder (not `Tracks-Organized`) with brand code `NOMADNP` (not `NOMAD`)
|
||||
- **YouTube**: Skipped entirely
|
||||
- **Google Drive**: Skipped entirely
|
||||
- **Email**: Completion email includes Dropbox link but no YouTube section
|
||||
- **Discord**: Still notified (for internal tracking)
|
||||
|
||||
Available to all users at job creation and toggleable by admin on existing jobs.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] All users can mark a job as "Private (Non-Published)" when creating it
|
||||
- [ ] Admin can toggle `is_private` on any existing job via admin dashboard
|
||||
- [ ] Private jobs upload to Dropbox at `Tracks-NonPublished` with `NOMADNP-XXXX` brand codes
|
||||
- [ ] Private jobs skip YouTube and Google Drive uploads entirely
|
||||
- [ ] Completion emails omit YouTube section when no YouTube URL (private jobs + any other edge case)
|
||||
- [ ] When admin toggles a completed job to private, existing outputs are auto-deleted
|
||||
- [ ] Discord notifications still fire for private jobs
|
||||
- [ ] Existing (non-private) jobs are completely unaffected (backward compatible)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Core Design: Single Boolean Flag with Runtime Overrides
|
||||
|
||||
Store `is_private: bool = False` on the job model. At distribution time, a helper function returns effective distribution parameters based on this flag. This means:
|
||||
|
||||
- **Toggling is_private automatically changes all distribution behavior** - no need to manually update dropbox_path, brand_prefix, enable_youtube_upload, gdrive_folder_id
|
||||
- **Job fields still store the "normal" defaults** - the override happens at the config-to-orchestrator boundary
|
||||
- **Admin can flip one field** and everything adapts
|
||||
|
||||
### Key Helper Function
|
||||
|
||||
```python
|
||||
# In job_defaults_service.py
|
||||
def get_effective_distribution_for_job(job) -> EffectiveDistributionSettings:
|
||||
"""Return distribution settings, applying private overrides if is_private=True."""
|
||||
if getattr(job, 'is_private', False):
|
||||
settings = get_settings()
|
||||
return EffectiveDistributionSettings(
|
||||
dropbox_path=settings.default_private_dropbox_path,
|
||||
brand_prefix=settings.default_private_brand_prefix,
|
||||
enable_youtube_upload=False,
|
||||
gdrive_folder_id=None,
|
||||
# Keep other settings from job (discord, youtube_description, etc.)
|
||||
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
|
||||
youtube_description=None,
|
||||
)
|
||||
# Non-private: return job's own settings
|
||||
return EffectiveDistributionSettings(
|
||||
dropbox_path=getattr(job, 'dropbox_path', None),
|
||||
brand_prefix=getattr(job, 'brand_prefix', None),
|
||||
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
|
||||
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
|
||||
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
|
||||
youtube_description=getattr(job, 'youtube_description_template', None),
|
||||
)
|
||||
```
|
||||
|
||||
This helper is used in both distribution paths (orchestrator and native fallback).
|
||||
|
||||
### Distribution Paths That Need Changes
|
||||
|
||||
Two code paths handle distribution (see LESSONS-LEARNED: "Fix Both Sides of Dual Code Paths"):
|
||||
|
||||
1. **Orchestrator path** (`video_worker_orchestrator.py`): Primary path. `create_orchestrator_config_from_job()` uses the helper to build config.
|
||||
2. **Native distribution path** (`video_worker.py` → `_handle_native_distribution()`): Fallback path. Also uses the helper instead of reading job fields directly.
|
||||
|
||||
### Email Template Change
|
||||
|
||||
Add conditional YouTube section removal in `render_template()`, matching the existing pattern for `feedback_url`:
|
||||
|
||||
```python
|
||||
# Remove YouTube section if no URL provided
|
||||
if not variables.get("youtube_url"):
|
||||
result = re.sub(
|
||||
r"\n*Here's the link for the karaoke video published to YouTube:\n\{youtube_url\}\n*",
|
||||
'\n',
|
||||
result,
|
||||
flags=re.DOTALL
|
||||
)
|
||||
variables["youtube_url"] = ""
|
||||
```
|
||||
|
||||
And change `render_job_completion()` to pass `None` instead of `"[YouTube URL not available]"` when youtube_url is empty.
|
||||
|
||||
### Admin Toggle Auto-Delete
|
||||
|
||||
When admin PATCH updates `is_private` from False to True on a job that has existing outputs, reuse the existing delete-outputs logic to remove YouTube + GDrive + Dropbox outputs. Admin can then "Reprocess" to re-distribute to the private Dropbox path.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Backend Model & Config (foundation)
|
||||
|
||||
1. [ ] **Add `is_private` to Job model** (`backend/models/job.py`)
|
||||
- Add `is_private: bool = False` field after `made_for_you` field
|
||||
- Add to `JobCreate` model as well
|
||||
|
||||
2. [ ] **Add private distribution config** (`backend/config.py`)
|
||||
- Add `default_private_dropbox_path: Optional[str] = os.getenv("DEFAULT_PRIVATE_DROPBOX_PATH")`
|
||||
- Add `default_private_brand_prefix: Optional[str] = os.getenv("DEFAULT_PRIVATE_BRAND_PREFIX", "NOMADNP")`
|
||||
|
||||
3. [ ] **Add helper function** (`backend/services/job_defaults_service.py`)
|
||||
- Add `get_effective_distribution_for_job(job)` function that returns distribution settings with private overrides applied
|
||||
|
||||
### Phase 2: Backend API (job creation + admin)
|
||||
|
||||
4. [ ] **Add `is_private` to request models** (`backend/models/requests.py`)
|
||||
- Add `is_private: Optional[bool] = None` to `URLSubmissionRequest`
|
||||
- Add `is_private: Optional[bool] = None` to `AudioSearchRequest`
|
||||
|
||||
5. [ ] **Update URL creation endpoint** (`backend/api/routes/jobs.py`)
|
||||
- Accept and pass through `is_private` field to `JobCreate`
|
||||
|
||||
6. [ ] **Update file upload endpoint** (`backend/api/routes/file_upload.py`)
|
||||
- Accept `is_private` Form parameter
|
||||
- Pass through to `JobCreate`
|
||||
|
||||
7. [ ] **Update audio search endpoint** (`backend/api/routes/audio_search.py`)
|
||||
- Accept `is_private` in `AudioSearchRequest`
|
||||
- Pass through to `JobCreate`
|
||||
|
||||
8. [ ] **Update admin endpoints** (`backend/api/routes/admin.py`)
|
||||
- Add `is_private: Optional[bool] = None` to `JobUpdateRequest`
|
||||
- Add `"is_private"` to `EDITABLE_JOB_FIELDS`
|
||||
- In the PATCH handler: detect when `is_private` changes from False→True on a completed job, auto-delete existing outputs (call existing delete-outputs service logic)
|
||||
|
||||
### Phase 3: Distribution Pipeline
|
||||
|
||||
9. [ ] **Update orchestrator config creation** (`backend/workers/video_worker_orchestrator.py`)
|
||||
- In `create_orchestrator_config_from_job()`: Use `get_effective_distribution_for_job()` helper to determine dropbox_path, brand_prefix, enable_youtube_upload, gdrive_folder_id instead of reading directly from job
|
||||
|
||||
10. [ ] **Update native distribution fallback** (`backend/workers/video_worker.py`)
|
||||
- In `_handle_native_distribution()`: Use `get_effective_distribution_for_job()` helper instead of reading directly from job fields
|
||||
|
||||
### Phase 4: Email Template
|
||||
|
||||
11. [ ] **Add YouTube section conditional removal** (`backend/services/template_service.py`)
|
||||
- In `render_template()`: Add regex to remove YouTube paragraph when `youtube_url` is empty (matching existing `feedback_url` pattern)
|
||||
- In `render_job_completion()`: Pass `None` for youtube_url when empty (instead of `"[YouTube URL not available]"`)
|
||||
|
||||
### Phase 5: Frontend
|
||||
|
||||
12. [ ] **Add private checkbox to job creation form** (`frontend/components/job/JobSubmission.tsx`)
|
||||
- Add "Private (Non-Published)" checkbox below the existing "Skip lyrics review" checkbox
|
||||
- Show for all users (not just admin)
|
||||
- Include brief description text: "Private tracks go to Dropbox only, not YouTube or Google Drive"
|
||||
|
||||
13. [ ] **Update API interfaces** (`frontend/lib/api.ts`)
|
||||
- Add `is_private?: boolean` to all job creation option interfaces
|
||||
- Add `is_private` to `JobUpdateRequest` interface
|
||||
- Pass through in `uploadJob()`, `createJobFromUrl()`, `searchAudio()` methods
|
||||
|
||||
14. [ ] **Add private toggle to admin job detail** (`frontend/app/admin/jobs/page.tsx`)
|
||||
- Add `is_private` as an inline-editable field in the job overview section (use Switch component)
|
||||
- Show current state with visual indicator (e.g., lock icon for private jobs)
|
||||
- When toggling to private on a completed job, show confirmation dialog about auto-deleting outputs
|
||||
|
||||
### Phase 6: Environment Configuration
|
||||
|
||||
15. [ ] **Update CI/deployment config** (`.github/workflows/ci.yml` or Cloud Run env)
|
||||
- Add `DEFAULT_PRIVATE_DROPBOX_PATH` environment variable (e.g., `/Karaoke/Tracks-NonPublished`)
|
||||
- Add `DEFAULT_PRIVATE_BRAND_PREFIX` environment variable (e.g., `NOMADNP`)
|
||||
- These should be configured in Cloud Run service and CI test environment
|
||||
|
||||
### Phase 7: Testing
|
||||
|
||||
16. [ ] **Unit tests for private distribution logic** (`backend/tests/test_private_tracks.py`)
|
||||
- Test `get_effective_distribution_for_job()` with private and non-private jobs
|
||||
- Test `create_orchestrator_config_from_job()` applies private overrides correctly
|
||||
- Test `_run_distribution()` skips YouTube and GDrive when private config applied
|
||||
- Test `_upload_to_dropbox()` uses private path and brand prefix
|
||||
- Test brand code generation with NOMADNP prefix in Tracks-NonPublished
|
||||
- Test `_handle_native_distribution()` respects private overrides
|
||||
- Test non-private jobs are completely unaffected (regression tests)
|
||||
|
||||
17. [ ] **Unit tests for email changes** (add to `backend/tests/test_email_service.py` or `test_template_service.py`)
|
||||
- Test YouTube section removed from email when youtube_url is None
|
||||
- Test YouTube section present when youtube_url is provided
|
||||
- Test Dropbox section still present for private jobs
|
||||
- Test full email rendering for private job scenario
|
||||
|
||||
18. [ ] **Unit tests for admin toggle** (add to `backend/tests/test_admin.py` or new file)
|
||||
- Test admin can update is_private field
|
||||
- Test toggling is_private to True on completed job triggers output deletion
|
||||
- Test toggling is_private to True on non-completed job does NOT delete outputs
|
||||
- Test toggling is_private to False does nothing special
|
||||
|
||||
19. [ ] **Integration test** (`backend/tests/emulator/` or `test_distribution_services.py`)
|
||||
- Test full distribution pipeline with is_private=True: verify Dropbox path, brand prefix, YouTube skip, GDrive skip
|
||||
|
||||
20. [ ] **Frontend E2E regression test** (`frontend/e2e/regression/`)
|
||||
- Test private checkbox appears in job creation form
|
||||
- Test private toggle in admin dashboard
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/job.py` | Modify | Add `is_private: bool = False` to Job and JobCreate |
|
||||
| `backend/config.py` | Modify | Add `default_private_dropbox_path` and `default_private_brand_prefix` |
|
||||
| `backend/services/job_defaults_service.py` | Modify | Add `get_effective_distribution_for_job()` helper |
|
||||
| `backend/models/requests.py` | Modify | Add `is_private` to URL and AudioSearch request models |
|
||||
| `backend/api/routes/jobs.py` | Modify | Accept and pass `is_private` |
|
||||
| `backend/api/routes/file_upload.py` | Modify | Accept and pass `is_private` |
|
||||
| `backend/api/routes/audio_search.py` | Modify | Accept and pass `is_private` |
|
||||
| `backend/api/routes/admin.py` | Modify | Add `is_private` to editable fields + auto-delete logic |
|
||||
| `backend/workers/video_worker_orchestrator.py` | Modify | Use helper in `create_orchestrator_config_from_job()` |
|
||||
| `backend/workers/video_worker.py` | Modify | Use helper in `_handle_native_distribution()` |
|
||||
| `backend/services/template_service.py` | Modify | Conditional YouTube section removal |
|
||||
| `frontend/components/job/JobSubmission.tsx` | Modify | Add private checkbox |
|
||||
| `frontend/lib/api.ts` | Modify | Add `is_private` to interfaces and API calls |
|
||||
| `frontend/app/admin/jobs/page.tsx` | Modify | Add private toggle to admin detail |
|
||||
| `backend/tests/test_private_tracks.py` | Create | Comprehensive unit tests for private track logic |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests (~25-30 tests in `test_private_tracks.py`)
|
||||
- `get_effective_distribution_for_job()` with private=True and False
|
||||
- `create_orchestrator_config_from_job()` private overrides (YouTube=False, GDrive=None, private Dropbox path, NOMADNP brand)
|
||||
- Distribution stage skips: YouTube not called, GDrive not called, Dropbox IS called with correct path
|
||||
- Brand code generation uses NOMADNP prefix and Tracks-NonPublished path
|
||||
- Native distribution fallback applies same overrides
|
||||
- **Regression tests**: Non-private jobs have unchanged behavior (same paths, same brand code, YouTube+GDrive+Dropbox all fire)
|
||||
|
||||
### Email Template Tests (~5 tests)
|
||||
- YouTube section stripped when youtube_url is None
|
||||
- YouTube section preserved when youtube_url has value
|
||||
- Dropbox section always present
|
||||
- Full private job email rendering (Dropbox + no YouTube)
|
||||
- Full non-private job email rendering (both present) - regression
|
||||
|
||||
### Admin Toggle Tests (~5 tests)
|
||||
- is_private editable via PATCH
|
||||
- Toggle to private on completed job deletes outputs
|
||||
- Toggle to private on pending/in-progress job does NOT delete
|
||||
- Toggle to non-private does nothing special
|
||||
|
||||
### Frontend Tests
|
||||
- E2E: Private checkbox visible and functional in job creation
|
||||
- E2E: Admin toggle visible and includes confirmation dialog
|
||||
|
||||
### Manual Verification
|
||||
- Create a private job end-to-end, verify Dropbox folder in Tracks-NonPublished with NOMADNP code
|
||||
- Verify completion email has Dropbox link but no YouTube section
|
||||
- Toggle existing completed job to private, verify outputs deleted
|
||||
|
||||
## Regression Risk Analysis
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| Non-private jobs affected | `is_private` defaults to `False`; helper returns unchanged values for non-private. Explicit regression tests. |
|
||||
| Dual code paths (orchestrator vs native) | Both paths use same helper function. Tests cover both paths. (Lesson: "Fix Both Sides of Dual Code Paths") |
|
||||
| Email template regex breaks other content | Regex targets specific YouTube paragraph text. Tests verify non-YouTube content untouched. |
|
||||
| Brand code namespace collision | NOMADNP codes are in a completely separate Dropbox folder (Tracks-NonPublished), no overlap with NOMAD codes in Tracks-Organized |
|
||||
| Admin auto-delete on wrong job | Only triggers when `is_private` changes from False→True AND job has terminal status (complete). Confirmation dialog in frontend. |
|
||||
| Made-for-you orders | MFY webhook doesn't set is_private (defaults False). Admin can toggle after. No change to MFY flow. |
|
||||
| Existing Firestore documents | Missing `is_private` field defaults to `False` via model default. No migration needed. |
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] Who sees private toggle? → **Everyone** (all users at creation, admin on existing)
|
||||
- [x] Auto-delete on toggle? → **Yes**, auto-delete all outputs when toggling to private on completed job
|
||||
- [x] Email format? → **Remove YouTube section** when no YouTube URL (improves all edge cases)
|
||||
- [x] Discord? → **Still notify** (useful for internal tracking)
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. **Feature flag**: The `is_private` field defaults to `False`, so setting `DEFAULT_PRIVATE_DROPBOX_PATH` to empty/None effectively disables private distribution
|
||||
2. **No schema migration needed**: Missing `is_private` field on existing Firestore documents defaults to `False`
|
||||
3. **Email change is backward-compatible**: The YouTube section removal only happens when youtube_url is None, which is already an edge case showing "[YouTube URL not available]"
|
||||
4. **Frontend**: Private checkbox is additive UI, removing it just means users can't create new private jobs
|
||||
@@ -0,0 +1,239 @@
|
||||
# Plan: Song Lookup & Autocomplete + Community Version Detection
|
||||
|
||||
**Created:** 2026-03-02
|
||||
**Branch:** feat/sess-20260302-2324-song-lookup-autocomplete
|
||||
**Status:** Implementation Complete
|
||||
|
||||
## Overview
|
||||
|
||||
Add two features to the karaoke-gen job creation flow:
|
||||
|
||||
1. **Artist/title autocomplete** using karaoke-decide's MusicBrainz + Spotify catalog data, so users get properly-styled artist/title text (e.g. "Fleetwood Mac" instead of "fleetwood mac")
|
||||
2. **Community karaoke version detection** that warns users when a high-quality community karaoke version already exists on YouTube (via karaokenerds data), saving them credits and time
|
||||
|
||||
Both features are **optional and non-blocking** — users can still type whatever they want and proceed even if a community version exists.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] Artist field autocomplete across all 3 job creation tabs (Search, Upload, URL)
|
||||
- [ ] Title field autocomplete (optionally filtered by selected artist) across all 3 tabs
|
||||
- [ ] Autocomplete uses canonical MusicBrainz/Spotify text styling
|
||||
- [ ] Autocomplete is optional — users can ignore suggestions and type freely
|
||||
- [ ] Community version check triggers when both artist and title are filled
|
||||
- [ ] If community karaoke version exists, show a dismissible banner with YouTube link(s)
|
||||
- [ ] Community check does not block job submission — just informs the user
|
||||
- [ ] Graceful degradation if karaoke-decide API or karaokenerds.com is down
|
||||
|
||||
## Architecture Decisions
|
||||
|
||||
### How to access karaoke-decide catalog data
|
||||
|
||||
**Decision: Proxy through karaoke-gen backend**
|
||||
|
||||
Both services share the same GCP project (`nomadkaraoke`), but the frontend at `gen.nomadkaraoke.com` can't directly call `decide.nomadkaraoke.com/api/` due to CORS (different origins). Options considered:
|
||||
|
||||
| Option | Pros | Cons |
|
||||
|--------|------|------|
|
||||
| **Proxy via karaoke-gen backend** | No CORS issues, no changes to karaoke-decide, frontend already calls api.nomadkaraoke.com | Added latency (extra hop), karaoke-decide dependency |
|
||||
| Direct frontend calls + CORS | Simpler, no proxy code | Requires karaoke-decide CORS changes, cross-project coupling |
|
||||
| Query BigQuery directly from karaoke-gen | No dependency on karaoke-decide service | Duplicates query logic, BigQuery setup needed, higher latency |
|
||||
|
||||
The proxy approach is cleanest: karaoke-gen frontend → karaoke-gen backend → karaoke-decide API. Response caching in the proxy minimizes latency impact.
|
||||
|
||||
### How to detect community karaoke versions
|
||||
|
||||
**Decision: Scrape karaokenerds.com from karaoke-gen backend** (same approach as kjbox)
|
||||
|
||||
The `karaokenerds_raw` BigQuery table has the catalog but NOT community status. Community status is determined by an `<img class="check">` badge in karaokenerds.com's HTML, only available via web scraping. The kjbox codebase has proven, well-tested scraping logic we can port directly.
|
||||
|
||||
Search URL: `https://karaokenerds.com/Search?query={artist}+{title}&webFilter=OnlyWeb`
|
||||
|
||||
Returns: songs with tracks, each track having `brand_name`, `brand_code`, `youtube_url`, `is_community`.
|
||||
|
||||
### Autocomplete UI component
|
||||
|
||||
**Decision: Use existing shadcn Command (cmdk) + Popover components**
|
||||
|
||||
The frontend already has `command.tsx` and `popover.tsx` from shadcn/ui. The standard combobox pattern (Popover + Command) provides:
|
||||
- Keyboard navigation (arrow keys, enter to select)
|
||||
- Debounced search-as-you-type
|
||||
- Clean dropdown with loading states
|
||||
- Works inline within forms (not modal)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Data flow
|
||||
|
||||
```
|
||||
User types in artist field
|
||||
→ debounced (300ms) API call to GET /api/catalog/artists?q=fle
|
||||
→ karaoke-gen backend proxies to decide.nomadkaraoke.com/api/catalog/artists?q=fle
|
||||
→ returns [{ name: "Fleetwood Mac", mbid: "...", popularity: 85 }, ...]
|
||||
→ user selects "Fleetwood Mac" → field populated with canonical name
|
||||
|
||||
User types in title field
|
||||
→ debounced (300ms) API call to GET /api/catalog/tracks?q=dreams&artist=Fleetwood+Mac
|
||||
→ returns [{ track_name: "Dreams", artist_name: "Fleetwood Mac", popularity: 82 }, ...]
|
||||
→ user selects "Dreams" → field populated with canonical title
|
||||
|
||||
Both fields filled → community check fires
|
||||
→ POST /api/catalog/community-check { artist: "Fleetwood Mac", title: "Dreams" }
|
||||
→ backend scrapes karaokenerds.com for "Fleetwood Mac Dreams" with webFilter=OnlyWeb
|
||||
→ if community tracks found with YouTube URLs, return them
|
||||
→ frontend shows banner: "A community karaoke version already exists! [Watch on YouTube →]"
|
||||
```
|
||||
|
||||
### Caching strategy
|
||||
|
||||
| Endpoint | Cache TTL | Scope |
|
||||
|----------|-----------|-------|
|
||||
| Artist search proxy | 5 min | In-memory (backend) |
|
||||
| Track search proxy | 5 min | In-memory (backend) |
|
||||
| Community check | 1 hour | In-memory (backend) |
|
||||
|
||||
### karaoke-decide API endpoints used
|
||||
|
||||
These are all **public, no auth required**:
|
||||
|
||||
- `GET /api/catalog/artists?q={query}&limit=10` — MBID-first artist search
|
||||
- Returns: `[{ mbid, name, disambiguation, artist_type, tags, spotify_id, popularity, genres }]`
|
||||
- `GET /api/catalog/tracks?q={query}&limit=10` — Track search (Spotify data)
|
||||
- Returns: `[{ track_id, track_name, artist_name, artist_id, popularity, duration_ms, explicit }]`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Backend — Catalog proxy + community check endpoints
|
||||
|
||||
1. [ ] **Add catalog proxy service** — `backend/services/catalog_proxy_service.py`
|
||||
- HTTP client (httpx async) calling karaoke-decide API
|
||||
- In-memory TTL cache for responses (5 min)
|
||||
- Configurable karaoke-decide base URL
|
||||
- Graceful error handling (return empty results on failure)
|
||||
|
||||
2. [ ] **Port karaokenerds scraping logic** — `backend/services/karaokenerds_service.py`
|
||||
- Port `search()`, `parse_results()`, `_parse_tracks()`, `_parse_single_track()`, `_clean_youtube_url()` from `kjbox/kj-controller/karaoke_nerds.py`
|
||||
- Add async support (httpx instead of requests)
|
||||
- In-memory TTL cache (1 hour)
|
||||
- Return only community tracks with YouTube URLs
|
||||
|
||||
3. [ ] **Add catalog routes** — `backend/api/routes/catalog.py`
|
||||
- `GET /api/catalog/artists?q=&limit=10` — proxy to karaoke-decide
|
||||
- `GET /api/catalog/tracks?q=&artist=&limit=10` — proxy to karaoke-decide
|
||||
- `POST /api/catalog/community-check` — scrape karaokenerds.com
|
||||
- All endpoints require auth (same as job creation)
|
||||
- Pydantic request/response models
|
||||
|
||||
4. [ ] **Register routes in main.py** — add `catalog.router` to app
|
||||
|
||||
5. [ ] **Add config** — `backend/config.py`
|
||||
- `karaoke_decide_api_url` setting (default: `https://decide.nomadkaraoke.com`)
|
||||
|
||||
6. [ ] **Add `beautifulsoup4` + `httpx` dependencies** to `pyproject.toml` (if not already present)
|
||||
|
||||
### Phase 2: Frontend — Autocomplete component
|
||||
|
||||
7. [ ] **Create `AutocompleteInput` component** — `frontend/components/ui/autocomplete-input.tsx`
|
||||
- Wraps an Input field with a Popover + Command dropdown
|
||||
- Props: `value`, `onChange`, `onSelect`, `fetchSuggestions(query) => Promise<Suggestion[]>`, `placeholder`, `disabled`, `renderItem`
|
||||
- Debounced search (300ms) triggered on input change
|
||||
- Shows loading spinner while fetching
|
||||
- Shows "No results" when search returns empty
|
||||
- Keyboard navigation (up/down arrows, enter to select, escape to close)
|
||||
- Clicking outside closes dropdown
|
||||
- When user selects a suggestion, calls `onSelect(suggestion)` and populates input
|
||||
- When user types without selecting, uses their raw text (autocomplete is optional)
|
||||
|
||||
8. [ ] **Add catalog API functions** — `frontend/lib/api.ts`
|
||||
- `searchArtists(query: string, limit?: number)` → `GET /api/catalog/artists`
|
||||
- `searchTracks(query: string, artist?: string, limit?: number)` → `GET /api/catalog/tracks`
|
||||
- `checkCommunityVersions(artist: string, title: string)` → `POST /api/catalog/community-check`
|
||||
|
||||
### Phase 3: Frontend — Integrate autocomplete into job creation
|
||||
|
||||
9. [ ] **Replace artist Input fields with AutocompleteInput** in `JobSubmission.tsx`
|
||||
- All 3 tabs: `searchArtist`, `uploadArtist`, `youtubeArtist`
|
||||
- Fetch suggestions from `api.searchArtists(query)`
|
||||
- On select: populate artist field with canonical name
|
||||
- Show artist name + optional disambiguation in dropdown items
|
||||
|
||||
10. [ ] **Replace title Input fields with AutocompleteInput** in `JobSubmission.tsx`
|
||||
- All 3 tabs: `searchTitle`, `uploadTitle`, `youtubeTitle`
|
||||
- Fetch suggestions from `api.searchTracks(query, selectedArtist)`
|
||||
- On select: populate title field with canonical name
|
||||
- Show track name + artist name + duration in dropdown items
|
||||
|
||||
### Phase 4: Frontend — Community version detection
|
||||
|
||||
11. [ ] **Create `CommunityVersionBanner` component** — `frontend/components/job/CommunityVersionBanner.tsx`
|
||||
- Shows when community karaoke version(s) found
|
||||
- Displays: "A karaoke version of this song already exists!"
|
||||
- Lists community tracks with brand name and YouTube link
|
||||
- "Watch on YouTube" button for top community version
|
||||
- Dismissible (X button or "Continue anyway" link)
|
||||
- Styled with green border (matching kjbox community track styling)
|
||||
|
||||
12. [ ] **Add community check logic to `JobSubmission.tsx`** (Search tab only)
|
||||
- Debounced check (500ms) when both artist AND title fields have values
|
||||
- Only triggers when values change (not on every keystroke)
|
||||
- Shows `CommunityVersionBanner` between the form fields and the submit button
|
||||
- Does NOT block form submission
|
||||
- Clears when artist or title changes
|
||||
|
||||
### Phase 5: Testing
|
||||
|
||||
13. [ ] **Backend unit tests** — `backend/tests/unit/test_catalog_routes.py`
|
||||
- Test proxy endpoints with mocked karaoke-decide responses
|
||||
- Test community check with mocked karaokenerds HTML
|
||||
- Test error handling (karaoke-decide down, karaokenerds down)
|
||||
- Test caching behavior
|
||||
|
||||
14. [ ] **Backend unit tests** — `backend/tests/unit/test_karaokenerds_service.py`
|
||||
- Port relevant tests from `kjbox/kj-controller/tests/unit/test_karaoke_nerds.py`
|
||||
- Test HTML parsing, community detection, YouTube URL cleanup
|
||||
|
||||
15. [ ] **Frontend component tests** — `frontend/components/ui/__tests__/autocomplete-input.test.tsx`
|
||||
- Test suggestion fetching and display
|
||||
- Test keyboard navigation
|
||||
- Test selection behavior
|
||||
- Test free-text input (no selection)
|
||||
|
||||
16. [ ] **E2E tests** — `frontend/e2e/job-creation-autocomplete.spec.ts`
|
||||
- Test autocomplete appears on typing
|
||||
- Test selecting from autocomplete populates field
|
||||
- Test community banner appears for known songs
|
||||
- Test form submission works with and without autocomplete selection
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/services/catalog_proxy_service.py` | Create | HTTP client for karaoke-decide API with caching |
|
||||
| `backend/services/karaokenerds_service.py` | Create | Karaokenerds.com scraper (ported from kjbox) |
|
||||
| `backend/api/routes/catalog.py` | Create | Proxy endpoints for artist/track search + community check |
|
||||
| `backend/main.py` | Modify | Register catalog router |
|
||||
| `backend/config.py` | Modify | Add karaoke_decide_api_url setting |
|
||||
| `pyproject.toml` | Modify | Add beautifulsoup4 dependency (if missing) |
|
||||
| `frontend/components/ui/autocomplete-input.tsx` | Create | Reusable autocomplete input component |
|
||||
| `frontend/components/job/CommunityVersionBanner.tsx` | Create | Community version warning banner |
|
||||
| `frontend/components/job/JobSubmission.tsx` | Modify | Replace Input with AutocompleteInput, add community check |
|
||||
| `frontend/lib/api.ts` | Modify | Add catalog API functions |
|
||||
| `backend/tests/unit/test_catalog_routes.py` | Create | Backend route tests |
|
||||
| `backend/tests/unit/test_karaokenerds_service.py` | Create | Scraper unit tests |
|
||||
| `frontend/components/ui/__tests__/autocomplete-input.test.tsx` | Create | Component tests |
|
||||
| `frontend/e2e/job-creation-autocomplete.spec.ts` | Create | E2E tests |
|
||||
|
||||
## Decisions (Resolved)
|
||||
|
||||
- **Track search scope**: All Spotify tracks (256M) — better coverage for new/obscure songs
|
||||
- **Community check scope**: Search tab only (Upload/URL users already have their audio)
|
||||
- **Rate limiting**: 20 requests/min per user on catalog proxy endpoints (prevents abuse of karaokenerds scraping). All endpoints require auth.
|
||||
- **"Display As" fields**: No autocomplete — they're for deliberate overrides like soundtrack names
|
||||
- **CORS**: Proxy approach (no changes needed to karaoke-decide)
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
- Feature is entirely additive — no existing functionality is modified
|
||||
- Autocomplete fields fall back to plain text Input if API calls fail
|
||||
- Community check is non-blocking — if it fails, form works normally
|
||||
- All new endpoints are behind the existing auth middleware
|
||||
- Can disable by reverting the frontend changes (backend proxy is harmless if unused)
|
||||
Executable
+37
@@ -0,0 +1,37 @@
|
||||
#!/bin/bash
|
||||
# Auto-translate when English translation files change.
|
||||
# Uses GCS cache so this is near-instant for small changes.
|
||||
# Skip with: git commit --no-verify
|
||||
|
||||
set -e
|
||||
|
||||
STAGED_EN=$(git diff --cached --name-only --diff-filter=ACM | grep -E '(messages|translations)/en\.json$' || true)
|
||||
|
||||
if [ -z "$STAGED_EN" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "i18n: English translations changed — running translation pipeline..."
|
||||
|
||||
for en_file in $STAGED_EN; do
|
||||
dir=$(dirname "$en_file")
|
||||
|
||||
if [[ "$dir" == frontend/messages ]]; then
|
||||
script="frontend/scripts/translate.py"
|
||||
elif [[ "$dir" == backend/translations ]]; then
|
||||
script="frontend/scripts/translate.py"
|
||||
else
|
||||
echo " Warning: Unknown messages dir $dir, skipping"
|
||||
continue
|
||||
fi
|
||||
|
||||
echo " Translating $dir..."
|
||||
if python "$script" --messages-dir "$dir" --target all --skip-review 2>&1 | tail -5; then
|
||||
git add "$dir"/*.json
|
||||
echo " Done: $dir"
|
||||
else
|
||||
echo " Warning: Translation failed for $dir (GCP auth may be missing). CI will catch this."
|
||||
fi
|
||||
done
|
||||
|
||||
echo "i18n: Translation complete."
|
||||
+576
-215
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,337 @@
|
||||
name: E2E Daily Test (Payment + Happy Path)
|
||||
|
||||
# Full production E2E: buy credits via real Stripe, then use them for karaoke generation.
|
||||
# Replaces the previous e2e-happy-path.yml with an additional payment stage.
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 6 * * *'
|
||||
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
debug_mode:
|
||||
description: 'Enable debug mode (slower, more screenshots)'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
browser:
|
||||
description: 'Browser to test with'
|
||||
required: false
|
||||
default: 'chromium'
|
||||
type: choice
|
||||
options:
|
||||
- chromium
|
||||
- firefox
|
||||
- webkit
|
||||
skip_payment:
|
||||
description: 'Skip Stage 1 (payment) — use admin-granted credits for happy path only'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
use_test_token:
|
||||
description: 'Skip signup entirely — use E2E_TEST_TOKEN (for quick iterations)'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
# =========================================================================
|
||||
# Stage 1: Purchase credits via real Stripe Checkout
|
||||
# =========================================================================
|
||||
e2e-credit-purchase:
|
||||
name: "Stage 1: Credit Purchase"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
if: ${{ github.event.inputs.skip_payment != 'true' && github.event.inputs.use_test_token != 'true' }}
|
||||
|
||||
outputs:
|
||||
session_token: ${{ steps.extract_token.outputs.token }}
|
||||
result: ${{ steps.test_result.outputs.result }}
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Install Playwright browsers
|
||||
working-directory: frontend
|
||||
run: npx playwright install --with-deps ${{ inputs.browser || 'chromium' }}
|
||||
|
||||
- name: Create test-results directory
|
||||
working-directory: frontend
|
||||
run: mkdir -p test-results
|
||||
|
||||
- name: Ensure testmail.app domain is allowlisted
|
||||
run: |
|
||||
curl -sf -X POST \
|
||||
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
|
||||
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
|
||||
|
||||
- name: Run Credit Purchase Test
|
||||
working-directory: frontend
|
||||
env:
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
E2E_STRIPE_CARD_NUMBER: ${{ secrets.E2E_STRIPE_CARD_NUMBER }}
|
||||
E2E_STRIPE_CARD_EXPIRY: ${{ secrets.E2E_STRIPE_CARD_EXPIRY }}
|
||||
E2E_STRIPE_CARD_CVC: ${{ secrets.E2E_STRIPE_CARD_CVC }}
|
||||
E2E_STRIPE_CARDHOLDER_NAME: ${{ secrets.E2E_STRIPE_CARDHOLDER_NAME }}
|
||||
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
|
||||
run: |
|
||||
echo "=== Stage 1: Credit Purchase via Real Stripe ==="
|
||||
echo "Time: $(date)"
|
||||
npx playwright test credit-purchase-real.spec.ts \
|
||||
--config=playwright.production.config.ts \
|
||||
--project=${{ inputs.browser || 'chromium' }} \
|
||||
--reporter=list \
|
||||
--timeout=600000 \
|
||||
2>&1 | tee test-output-payment.log
|
||||
continue-on-error: true
|
||||
|
||||
- name: Check test result
|
||||
id: test_result
|
||||
working-directory: frontend
|
||||
run: |
|
||||
if grep -q "CREDIT PURCHASE TEST PASSED" test-output-payment.log; then
|
||||
echo "result=success" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "result=failure" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
- name: Extract session token
|
||||
id: extract_token
|
||||
if: steps.test_result.outputs.result == 'success'
|
||||
working-directory: frontend
|
||||
run: |
|
||||
if [ -f test-results/e2e-session-token.txt ]; then
|
||||
TOKEN=$(cat test-results/e2e-session-token.txt)
|
||||
echo "::add-mask::$TOKEN"
|
||||
echo "token=$TOKEN" >> $GITHUB_OUTPUT
|
||||
echo "Session token extracted for Stage 2"
|
||||
else
|
||||
echo "No session token file found"
|
||||
echo "token=" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
- name: Upload artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-payment-results-${{ github.run_number }}
|
||||
path: |
|
||||
frontend/test-results/
|
||||
frontend/test-output-payment.log
|
||||
retention-days: 30
|
||||
|
||||
- name: Discord notification (failure)
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
run: |
|
||||
curl -s -H "Content-Type: application/json" \
|
||||
-d "{\"embeds\": [{\"title\": \"🚨 E2E Daily Test FAILED — Stage 1: Credit Purchase\", \"description\": \"Payment flow is broken. Customers may not be receiving credits.\n\n**Run:** #${{ github.run_number }}\n**Details:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": 15158332}]}" \
|
||||
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
|
||||
|
||||
- name: Fail if test failed
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
run: exit 1
|
||||
|
||||
# =========================================================================
|
||||
# Stage 2: Happy Path (uses credit from Stage 1)
|
||||
# =========================================================================
|
||||
e2e-happy-path:
|
||||
name: "Stage 2: Happy Path"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 90
|
||||
needs: e2e-credit-purchase
|
||||
if: |
|
||||
always() && (
|
||||
needs.e2e-credit-purchase.result == 'success' ||
|
||||
github.event.inputs.skip_payment == 'true' ||
|
||||
github.event.inputs.use_test_token == 'true'
|
||||
)
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Install Playwright browsers
|
||||
working-directory: frontend
|
||||
run: npx playwright install --with-deps ${{ inputs.browser || 'chromium' }}
|
||||
|
||||
- name: Create test-results directory
|
||||
working-directory: frontend
|
||||
run: mkdir -p test-results
|
||||
|
||||
- name: Ensure testmail.app domain is allowlisted
|
||||
run: |
|
||||
curl -sf -X POST \
|
||||
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
|
||||
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
|
||||
|
||||
- name: Refresh E2E session token (when use_test_token enabled)
|
||||
id: refresh_token
|
||||
if: github.event.inputs.use_test_token == 'true'
|
||||
run: |
|
||||
RESPONSE=$(curl -sf -X POST \
|
||||
"https://api.nomadkaraoke.com/api/admin/users/e2e-test-runner@nomadkaraoke.com/impersonate" \
|
||||
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
|
||||
-H "Content-Type: application/json")
|
||||
TOKEN=$(echo "$RESPONSE" | python3 -c 'import json,sys; print(json.load(sys.stdin)["session_token"])')
|
||||
echo "::add-mask::$TOKEN"
|
||||
echo "token=$TOKEN" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Run Happy Path Test
|
||||
working-directory: frontend
|
||||
env:
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
E2E_TEST_TOKEN: ${{ steps.refresh_token.outputs.token || needs.e2e-credit-purchase.outputs.session_token || '' }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
|
||||
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
|
||||
run: |
|
||||
echo "=== Stage 2: Happy Path ==="
|
||||
echo "Time: $(date)"
|
||||
TOKEN_SOURCE="none"
|
||||
if [ -n "${{ steps.refresh_token.outputs.token }}" ]; then
|
||||
TOKEN_SOURCE="use_test_token (impersonation)"
|
||||
elif [ -n "${{ needs.e2e-credit-purchase.outputs.session_token }}" ]; then
|
||||
TOKEN_SOURCE="Stage 1 (credit purchase)"
|
||||
fi
|
||||
echo "Token source: $TOKEN_SOURCE"
|
||||
|
||||
npx playwright test happy-path-real-user.spec.ts \
|
||||
--config=playwright.production.config.ts \
|
||||
--project=${{ inputs.browser || 'chromium' }} \
|
||||
--reporter=list \
|
||||
--timeout=3600000 \
|
||||
2>&1 | tee test-output-happy-path.log
|
||||
continue-on-error: true
|
||||
|
||||
- name: Check test result
|
||||
id: test_result
|
||||
working-directory: frontend
|
||||
run: |
|
||||
if grep -q "passed" test-output-happy-path.log && ! grep -q "failed" test-output-happy-path.log; then
|
||||
echo "result=success" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "result=failure" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
- name: Upload test artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-results-${{ github.run_number }}
|
||||
path: |
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
frontend/test-output-happy-path.log
|
||||
retention-days: 30
|
||||
|
||||
- name: Upload video recordings
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-videos-${{ github.run_number }}
|
||||
path: frontend/test-results/**/*.webm
|
||||
retention-days: 30
|
||||
if-no-files-found: ignore
|
||||
|
||||
- name: Create summary
|
||||
if: always()
|
||||
run: |
|
||||
STAGE1="${{ needs.e2e-credit-purchase.result || 'skipped' }}"
|
||||
STAGE2="${{ steps.test_result.outputs.result || 'unknown' }}"
|
||||
echo "## E2E Daily Test Results" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Stage | Result |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "|-------|--------|" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Stage 1: Credit Purchase | ${STAGE1} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Stage 2: Happy Path | ${STAGE2} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
- name: Discord notification (failure)
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
run: |
|
||||
curl -s -H "Content-Type: application/json" \
|
||||
-d "{\"embeds\": [{\"title\": \"🚨 E2E Daily Test FAILED — Stage 2: Happy Path\", \"description\": \"Karaoke generation flow is broken.\n\n**Run:** #${{ github.run_number }}\n**Details:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": 15158332}]}" \
|
||||
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
|
||||
|
||||
- name: Fail if test failed
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
run: exit 1
|
||||
|
||||
# =========================================================================
|
||||
# Final: Send notifications
|
||||
# =========================================================================
|
||||
notify:
|
||||
name: "Notify"
|
||||
runs-on: ubuntu-latest
|
||||
needs: [e2e-credit-purchase, e2e-happy-path]
|
||||
if: always()
|
||||
|
||||
steps:
|
||||
- name: Discord notification (success)
|
||||
if: needs.e2e-happy-path.result == 'success'
|
||||
run: |
|
||||
STAGE1="${{ needs.e2e-credit-purchase.result || 'skipped' }}"
|
||||
curl -s -H "Content-Type: application/json" \
|
||||
-d "{\"embeds\": [{\"title\": \"✅ E2E Daily Test PASSED\", \"description\": \"Payment: ${STAGE1} | Happy Path: success\n\n**Run:** #${{ github.run_number }}\", \"color\": 3066993}]}" \
|
||||
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
|
||||
|
||||
- name: Send email notification
|
||||
if: github.event_name == 'schedule'
|
||||
uses: dawidd6/action-send-mail@v3
|
||||
with:
|
||||
server_address: smtp.sendgrid.net
|
||||
server_port: 587
|
||||
username: apikey
|
||||
password: ${{ secrets.SENDGRID_API_KEY }}
|
||||
subject: "E2E Daily Test ${{ needs.e2e-happy-path.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
|
||||
to: andrew@nomadkaraoke.com
|
||||
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
|
||||
body: |
|
||||
E2E Daily Test Results
|
||||
======================
|
||||
|
||||
Stage 1 (Credit Purchase): ${{ needs.e2e-credit-purchase.result || 'skipped' }}
|
||||
Stage 2 (Happy Path): ${{ needs.e2e-happy-path.result || 'skipped' }}
|
||||
|
||||
Run: #${{ github.run_number }}
|
||||
Trigger: ${{ github.event_name }}
|
||||
|
||||
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||
@@ -1,174 +0,0 @@
|
||||
name: E2E Happy Path Test
|
||||
|
||||
# Comprehensive end-to-end test that validates the full karaoke generation journey
|
||||
# using ONLY browser interactions (no API shortcuts)
|
||||
#
|
||||
# See docs/E2E-HAPPY-PATH-TEST-SPEC.md for full requirements
|
||||
|
||||
on:
|
||||
# Daily run at 6 AM UTC
|
||||
schedule:
|
||||
- cron: '0 6 * * *'
|
||||
|
||||
# Manual trigger with optional parameters
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
debug_mode:
|
||||
description: 'Enable debug mode (slower, more screenshots)'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
browser:
|
||||
description: 'Browser to test with'
|
||||
required: false
|
||||
default: 'chromium'
|
||||
type: choice
|
||||
options:
|
||||
- chromium
|
||||
- firefox
|
||||
- webkit
|
||||
use_test_token:
|
||||
description: 'Skip signup and use E2E_TEST_TOKEN (for iterations)'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
e2e-happy-path:
|
||||
name: "Full Happy Path (Real User)"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 90 # 1.5 hours max (test itself is 60 min)
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Install Playwright browsers
|
||||
working-directory: frontend
|
||||
run: |
|
||||
BROWSER=${{ inputs.browser || 'chromium' }}
|
||||
npx playwright install --with-deps $BROWSER
|
||||
|
||||
- name: Create test-results directory
|
||||
working-directory: frontend
|
||||
run: mkdir -p test-results
|
||||
|
||||
- name: Run E2E Happy Path Test
|
||||
working-directory: frontend
|
||||
env:
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
E2E_TEST_TOKEN: ${{ inputs.use_test_token && secrets.E2E_TEST_TOKEN || '' }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
|
||||
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
|
||||
run: |
|
||||
echo "Starting E2E Happy Path Test"
|
||||
echo "Browser: ${{ inputs.browser || 'chromium' }}"
|
||||
echo "Debug mode: ${{ inputs.debug_mode || 'false' }}"
|
||||
echo "Use test token: ${{ inputs.use_test_token || 'false' }}"
|
||||
echo "Time: $(date)"
|
||||
echo ""
|
||||
|
||||
# Run the real user happy path test
|
||||
npx playwright test happy-path-real-user.spec.ts \
|
||||
--config=playwright.production.config.ts \
|
||||
--project=${{ inputs.browser || 'chromium' }} \
|
||||
--reporter=list \
|
||||
--timeout=3600000 \
|
||||
2>&1 | tee test-output.log
|
||||
continue-on-error: true
|
||||
|
||||
- name: Check test result
|
||||
id: test_result
|
||||
working-directory: frontend
|
||||
run: |
|
||||
if grep -q "passed" test-output.log && ! grep -q "failed" test-output.log; then
|
||||
echo "result=success" >> $GITHUB_OUTPUT
|
||||
echo "Test PASSED"
|
||||
else
|
||||
echo "result=failure" >> $GITHUB_OUTPUT
|
||||
echo "Test FAILED - see artifacts for details"
|
||||
fi
|
||||
|
||||
- name: Upload test artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-results-${{ github.run_number }}
|
||||
path: |
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
frontend/test-output.log
|
||||
retention-days: 30
|
||||
|
||||
- name: Upload video recordings
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-videos-${{ github.run_number }}
|
||||
path: frontend/test-results/**/*.webm
|
||||
retention-days: 30
|
||||
if-no-files-found: ignore
|
||||
|
||||
- name: Create summary
|
||||
if: always()
|
||||
run: |
|
||||
echo "## E2E Happy Path Test Results" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Result | ${{ steps.test_result.outputs.result || 'unknown' }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
if [ -f frontend/test-output.log ]; then
|
||||
echo "### Test Output (last 100 lines)" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
tail -100 frontend/test-output.log >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
fi
|
||||
|
||||
- name: Send email notification
|
||||
if: always() && github.event_name == 'schedule'
|
||||
uses: dawidd6/action-send-mail@v3
|
||||
with:
|
||||
server_address: smtp.sendgrid.net
|
||||
server_port: 587
|
||||
username: apikey
|
||||
password: ${{ secrets.SENDGRID_API_KEY }}
|
||||
subject: "E2E Happy Path Test ${{ steps.test_result.outputs.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
|
||||
to: andrew@nomadkaraoke.com
|
||||
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
|
||||
body: |
|
||||
E2E Happy Path Test Results
|
||||
===========================
|
||||
|
||||
Status: ${{ steps.test_result.outputs.result == 'success' && 'PASSED ✅' || 'FAILED ❌' }}
|
||||
Run: #${{ github.run_number }}
|
||||
Trigger: ${{ github.event_name }}
|
||||
Time: ${{ github.event.repository.updated_at }}
|
||||
|
||||
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||
|
||||
This is an automated notification from the daily E2E happy path test.
|
||||
|
||||
- name: Fail if test failed
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
run: exit 1
|
||||
@@ -232,3 +232,4 @@ flacfetch
|
||||
|
||||
# Test output directories
|
||||
Rick Astley*/
|
||||
.superpowers/
|
||||
@@ -0,0 +1 @@
|
||||
[{"name":"generate-buildid","duration":272,"timestamp":244518298649,"id":4,"parentId":1,"tags":{},"startTime":1775262331361,"traceId":"b3ef0d179e8acedf"},{"name":"load-custom-routes","duration":314,"timestamp":244518299029,"id":5,"parentId":1,"tags":{},"startTime":1775262331361,"traceId":"b3ef0d179e8acedf"},{"name":"create-dist-dir","duration":333,"timestamp":244518299363,"id":6,"parentId":1,"tags":{},"startTime":1775262331362,"traceId":"b3ef0d179e8acedf"},{"name":"clean","duration":358,"timestamp":244518300558,"id":7,"parentId":1,"tags":{},"startTime":1775262331363,"traceId":"b3ef0d179e8acedf"},{"name":"next-build","duration":1670851,"timestamp":244516630180,"id":1,"tags":{"buildMode":"default","version":"16.2.2","bundler":"turbopack","failed":true},"startTime":1775262329693,"traceId":"b3ef0d179e8acedf"}]
|
||||
@@ -0,0 +1 @@
|
||||
[{"name":"next-build","duration":1670851,"timestamp":244516630180,"id":1,"tags":{"buildMode":"default","version":"16.2.2","bundler":"turbopack","failed":true},"startTime":1775262329693,"traceId":"b3ef0d179e8acedf"}]
|
||||
@@ -19,6 +19,8 @@
|
||||
| **Testing & code quality** | `docs/TESTING.md` |
|
||||
| API reference | `docs/API.md` |
|
||||
| Past learnings | `docs/LESSONS-LEARNED.md` |
|
||||
| **Operational runbooks** | `docs/TROUBLESHOOTING.md` |
|
||||
| GDrive validator & gaps | `docs/GDRIVE-VALIDATOR.md` |
|
||||
| **Product communication** | `docs/PRODUCT-COMMUNICATION-GUIDE.md` |
|
||||
| Brand style guide | `docs/BRAND-STYLE-GUIDE.md` |
|
||||
|
||||
@@ -26,7 +28,7 @@
|
||||
|
||||
- **Backend**: FastAPI on Cloud Run, Firestore, GCS, Secret Manager
|
||||
- **Frontend**: Next.js on Cloudflare Pages
|
||||
- **Processing**: Modal (GPU audio separation), AudioShake/Whisper (transcription)
|
||||
- **Processing**: Cloud Run GPU (L4 audio separation), AudioShake/Whisper (transcription)
|
||||
- **Infra**: Pulumi IaC, GitHub Actions CI/CD
|
||||
|
||||
## Essential Rules
|
||||
@@ -90,11 +92,25 @@ See `docs/TESTING.md` § "Ad-Hoc Production Debugging" for full details.
|
||||
|
||||
### Infrastructure
|
||||
- **All GCP changes via Pulumi PRs** - changes in `infrastructure/` deploy automatically on merge to main
|
||||
- **Always run `pulumi up` locally BEFORE merging PRs** - CI deploys run on spot VMs that can be preempted mid-apply, causing partial state. Apply locally first, then merge (CI re-runs as no-op)
|
||||
- Never modify GCP resources directly via console or `gcloud` CLI
|
||||
- `gcloud` CLI for reading/debugging only (e.g., checking logs, SSH to VMs)
|
||||
- Local `pulumi preview` works but `pulumi up` requires broader permissions than CI
|
||||
- Stop and notify user on auth issues
|
||||
|
||||
### Internationalization (i18n)
|
||||
|
||||
- All user-facing strings live in `frontend/messages/{locale}.json` (33 locales), NOT in components
|
||||
- Backend strings live in `backend/translations/{locale}.json` (currently 3 locales: en, es, de)
|
||||
- Components use `useTranslations('namespace')` from next-intl
|
||||
- Pages are under `frontend/src/app/[locale]/` — locale-aware routing
|
||||
- Admin pages (under `/admin/`, `/app/`) use DefaultIntlProvider (English-only, excluded from i18n)
|
||||
- Internal links use `Link` from `@/i18n/routing` (locale-aware)
|
||||
- After adding/changing English strings: `python frontend/scripts/translate.py --messages-dir frontend/messages --target all`
|
||||
- Translation uses Gemini 3.1 Pro via Vertex AI with GCS cache for efficiency
|
||||
- CI validates all locale files have matching keys — **PR will fail if translations are missing**
|
||||
- Pre-commit hook auto-translates when en.json changes (enable: `git config core.hooksPath .githooks`)
|
||||
- Don't hardcode user-facing strings — add to `messages/en.json` and use `t('key')`
|
||||
|
||||
## API Authentication (for Agents)
|
||||
|
||||
When you need to call backend APIs programmatically:
|
||||
@@ -128,15 +144,17 @@ EOF
|
||||
|
||||
### GCE Encoding Worker
|
||||
```bash
|
||||
# SSH to restart service (clears in-memory job queue)
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
|
||||
--command="sudo systemctl restart encoding-worker"
|
||||
|
||||
# Check health
|
||||
# Check health (includes wheel_version)
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
|
||||
--command="curl -s http://localhost:8080/health"
|
||||
|
||||
# Restart service (pick up new wheel after a deploy)
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
|
||||
--command="sudo systemctl restart encoding-worker"
|
||||
```
|
||||
|
||||
For job stuck at `encoding` status, see [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md).
|
||||
|
||||
### What Doesn't Work
|
||||
- `gcloud auth print-identity-token` - Wrong token type for internal endpoints
|
||||
- Cloud Tasks without proper service account - Gets 401 from Cloud Run
|
||||
|
||||
@@ -1,18 +1,25 @@
|
||||
# Makefile for karaoke-gen project
|
||||
# Run `make help` to see available commands
|
||||
|
||||
.PHONY: help install install-backend install-frontend build-frontend test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop
|
||||
.PHONY: help install install-backend install-frontend build-frontend dev-install test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop
|
||||
|
||||
# Per-stage timeout in seconds (default: 10 minutes)
|
||||
# Prevents hangs from stale dev servers, broken emulators, etc.
|
||||
# Override with: make test TEST_TIMEOUT=300
|
||||
TEST_TIMEOUT ?= 600
|
||||
|
||||
# Default target
|
||||
help:
|
||||
@echo "Available commands:"
|
||||
@echo " make test - Run ALL tests (backend + frontend, installs deps automatically)"
|
||||
@echo " make test-backend - Run backend tests only (unit + emulator)"
|
||||
@echo " make test-frontend - Run frontend tests only (unit + E2E)"
|
||||
@echo " make test-frontend - Run frontend unit tests (Jest)"
|
||||
@echo " make test-frontend-e2e - Run frontend E2E tests (Playwright, requires dev server)"
|
||||
@echo " make test-unit - Run unit tests only (karaoke_gen package)"
|
||||
@echo " make test-e2e - Run emulator tests with auto-start/stop"
|
||||
@echo " make install - Install all dependencies (backend + frontend)"
|
||||
@echo " make build-frontend - Build frontend and copy to Python package (for local testing)"
|
||||
@echo " make build-frontend - Build frontend and copy to Python package"
|
||||
@echo " make dev-install - Build frontend + pip install -e . (for local CLI testing)"
|
||||
@echo " make lint - Run linter checks"
|
||||
@echo " make emulators-start - Start GCP emulators for local development"
|
||||
@echo " make emulators-stop - Stop GCP emulators"
|
||||
@@ -34,23 +41,34 @@ install-frontend:
|
||||
|
||||
install: install-backend install-frontend
|
||||
|
||||
# Build frontend and copy to Python package (for local CLI testing)
|
||||
# Build frontend and copy to Python package
|
||||
build-frontend: install-frontend
|
||||
@echo "=== Building frontend ==="
|
||||
cd frontend && npm run build
|
||||
@echo "=== Copying build to Python package ==="
|
||||
rm -rf karaoke_gen/nextjs_frontend/out
|
||||
mkdir -p karaoke_gen/nextjs_frontend/out
|
||||
cp -r frontend/out/* karaoke_gen/nextjs_frontend/out/
|
||||
@echo "✅ Frontend built and ready for local testing"
|
||||
@echo "✅ Frontend built and copied to karaoke_gen/nextjs_frontend/out/"
|
||||
|
||||
# Build frontend + install package in editable mode for local CLI testing
|
||||
# After running this, `karaoke-gen` CLI will use the freshly built frontend.
|
||||
dev-install: build-frontend
|
||||
@echo "=== Installing package in editable mode ==="
|
||||
pip install -e .
|
||||
@echo ""
|
||||
@echo "✅ Ready! You can now run karaoke-gen with the updated frontend."
|
||||
@echo " Example: karaoke-gen input.flac \"Artist\" \"Title\""
|
||||
|
||||
# Run unit tests for karaoke_gen package
|
||||
test-unit: install-backend
|
||||
@echo "=== Running karaoke_gen unit tests ==="
|
||||
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
|
||||
timeout $(TEST_TIMEOUT) poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
|
||||
|
||||
# Run backend unit tests (excludes emulator tests)
|
||||
test-backend-unit: install-backend
|
||||
@echo "=== Running backend unit tests ==="
|
||||
poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
|
||||
timeout $(TEST_TIMEOUT) poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
|
||||
|
||||
# Run E2E integration tests with emulators
|
||||
test-e2e: install-backend
|
||||
@@ -62,7 +80,7 @@ test-e2e: install-backend
|
||||
export GCS_BUCKET_NAME=test-bucket && \
|
||||
export ADMIN_TOKENS=test-admin-token && \
|
||||
export ENVIRONMENT=test && \
|
||||
poetry run pytest backend/tests/emulator/ -v; \
|
||||
timeout $(TEST_TIMEOUT) poetry run pytest backend/tests/emulator/ -v; \
|
||||
TEST_RESULT=$$?; \
|
||||
./scripts/stop-emulators.sh; \
|
||||
exit $$TEST_RESULT
|
||||
@@ -72,10 +90,15 @@ test-backend: test-unit test-backend-unit test-e2e
|
||||
@echo ""
|
||||
@echo "✅ Backend tests passed!"
|
||||
|
||||
# Run frontend tests (unit + E2E)
|
||||
# Run frontend unit tests (Jest only - E2E requires manual setup)
|
||||
test-frontend: install-frontend
|
||||
@echo "=== Running frontend tests ==="
|
||||
cd frontend && npm run test:all
|
||||
@echo "=== Running frontend unit tests ==="
|
||||
cd frontend && timeout $(TEST_TIMEOUT) npm run test:unit
|
||||
|
||||
# Run frontend E2E tests (requires dev server - run manually, not in make test)
|
||||
test-frontend-e2e: install-frontend
|
||||
@echo "=== Running frontend E2E tests ==="
|
||||
cd frontend && timeout $(TEST_TIMEOUT) npm run test:e2e
|
||||
|
||||
# Run ALL tests (backend + frontend) - use this before committing!
|
||||
test: test-backend test-frontend
|
||||
|
||||
@@ -29,6 +29,8 @@ tmp/
|
||||
# Google Cloud
|
||||
.gcloud/
|
||||
*.json
|
||||
# Allow i18n translation files
|
||||
!translations/*.json
|
||||
|
||||
# Testing
|
||||
.pytest_cache/
|
||||
|
||||
+9
-1
@@ -26,7 +26,15 @@ COPY backend /app/backend
|
||||
# Note: lyrics_transcriber is now a subpackage of karaoke_gen
|
||||
COPY karaoke_gen /app/karaoke_gen
|
||||
COPY pyproject.toml README.md LICENSE /app/
|
||||
RUN pip install --no-cache-dir -e .
|
||||
# GPU images (MODEL_DIR set in base): use --no-deps to preserve pinned torch cu126
|
||||
# and audio-separator[gpu]. Without this, pip upgrades torch to cu130 (breaks CUDA
|
||||
# on Cloud Run L4 drivers) and reinstalls audio-separator[cpu] (breaks GPU ONNX).
|
||||
# CPU images: normal install resolves all deps from pyproject.toml.
|
||||
RUN if [ -n "$MODEL_DIR" ]; then \
|
||||
pip install --no-cache-dir --no-deps -e .; \
|
||||
else \
|
||||
pip install --no-cache-dir -e .; \
|
||||
fi
|
||||
|
||||
# Set environment variables
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
|
||||
+26
-7
@@ -1,8 +1,15 @@
|
||||
# Base image with all Python dependencies pre-installed
|
||||
# This image is rebuilt only when poetry.lock changes
|
||||
#
|
||||
# This image is rebuilt only when poetry.lock or this Dockerfile changes
|
||||
#
|
||||
# Build with: gcloud builds submit --config=cloudbuild-base.yaml
|
||||
# Or manually: docker build -f backend/Dockerfile.base -t karaoke-backend-base .
|
||||
#
|
||||
# Layer caching strategy:
|
||||
# 1. System deps + fonts + ffmpeg (rarely change, always cached)
|
||||
# 2. pyproject.toml + package stubs (changes with dep updates)
|
||||
# 3. pip install (the slow part - cached unless pyproject.toml changes)
|
||||
# 4. Source code copy (changes every rebuild, but AFTER deps are installed)
|
||||
# 5. spaCy model download (cached unless pip install layer changes)
|
||||
|
||||
FROM python:3.11-slim
|
||||
|
||||
@@ -16,6 +23,7 @@ RUN apt-get update && apt-get install -y \
|
||||
build-essential \
|
||||
curl \
|
||||
xz-utils \
|
||||
chromium \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install comprehensive font support for international lyrics and symbols
|
||||
@@ -50,16 +58,27 @@ RUN curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-stati
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy dependency files and source code needed for pip install
|
||||
# Copy ONLY dependency specification files first (changes less often than source code)
|
||||
COPY pyproject.toml README.md LICENSE /app/
|
||||
|
||||
# Create minimal package stubs so pip can resolve the editable install
|
||||
# pip install -e . needs package directories to exist but doesn't validate content
|
||||
RUN mkdir -p /app/karaoke_gen /app/lyrics_transcriber_temp /app/backend && \
|
||||
touch /app/karaoke_gen/__init__.py /app/lyrics_transcriber_temp/__init__.py /app/backend/__init__.py
|
||||
|
||||
# Install all Python dependencies (the slow part)
|
||||
# BuildKit cache mount preserves downloaded wheels across builds, so partial
|
||||
# dependency updates only download changed packages instead of everything
|
||||
RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
pip install --upgrade pip && \
|
||||
pip install -e .
|
||||
|
||||
# Now copy actual source code (AFTER deps installed, so code changes don't
|
||||
# invalidate the expensive pip install layer in registry cache)
|
||||
COPY karaoke_gen /app/karaoke_gen
|
||||
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
|
||||
COPY backend /app/backend
|
||||
|
||||
# Install all Python dependencies (the slow part - cached in base image)
|
||||
RUN pip install --no-cache-dir --upgrade pip && \
|
||||
pip install --no-cache-dir -e .
|
||||
|
||||
# Download spaCy language model for lyrics transcription
|
||||
# This must be done at build time, not runtime (Cloud Run network restrictions)
|
||||
RUN python -m spacy download en_core_web_sm
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
# GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models
|
||||
# This image is rebuilt only when deps or models change (infrequently).
|
||||
#
|
||||
# Build with: gcloud builds submit --config=backend/cloudbuild-gpu-base.yaml --project=nomadkaraoke --timeout=3600s
|
||||
# Or manually: docker build -f backend/Dockerfile.gpu-base -t karaoke-backend-gpu-base .
|
||||
#
|
||||
# Layer caching strategy:
|
||||
# 1. System deps + fonts + ffmpeg (rarely change, always cached)
|
||||
# 2. CUDA library path configuration
|
||||
# 3. pyproject.toml + package stubs (changes with dep updates)
|
||||
# 4. pip install karaoke-gen deps + audio-separator[gpu] (slow, cached unless pyproject.toml changes)
|
||||
# 5. Source code copy (changes every rebuild, but AFTER deps are installed)
|
||||
# 6. spaCy model download (cached unless pip install layer changes)
|
||||
# 7. Pre-downloaded ensemble preset models (~1-1.5 GB, cached unless pip layer changes)
|
||||
|
||||
FROM nvidia/cuda:12.6.3-runtime-ubuntu22.04
|
||||
|
||||
# Prevent interactive prompts during package installation
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
# Install Python 3.11 from deadsnakes PPA and system dependencies
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
software-properties-common \
|
||||
&& add-apt-repository -y ppa:deadsnakes/ppa \
|
||||
&& apt-get update && apt-get install -y --no-install-recommends \
|
||||
# Python 3.11
|
||||
python3.11 \
|
||||
python3.11-dev \
|
||||
python3.11-venv \
|
||||
# Audio libraries
|
||||
libsndfile1 \
|
||||
libsox-dev \
|
||||
sox \
|
||||
# Build tools
|
||||
build-essential \
|
||||
# Utilities
|
||||
curl \
|
||||
xz-utils \
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& python3.11 --version
|
||||
|
||||
# Install comprehensive font support for international lyrics and symbols
|
||||
# Noto fonts provide full Unicode coverage including:
|
||||
# - Latin, Greek, Cyrillic (core)
|
||||
# - CJK: Chinese, Japanese, Korean (~150MB)
|
||||
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
|
||||
# - Color emoji (limited support in libass, but available)
|
||||
# - Musical symbols (♪ ♫ etc.)
|
||||
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
fonts-noto-core \
|
||||
fonts-noto-cjk \
|
||||
fonts-noto-extra \
|
||||
fonts-noto-color-emoji \
|
||||
fontconfig \
|
||||
&& fc-cache -f \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install FFmpeg from Ubuntu package
|
||||
# The audio worker only uses FFmpeg for format conversion and padding, not
|
||||
# encoding-intensive tasks, so the Debian package is sufficient. The static
|
||||
# build from johnvansickle.com is used for the encoding worker instead.
|
||||
RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Set Python 3.11 as default and install pip
|
||||
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1 \
|
||||
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.11 1 \
|
||||
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11 \
|
||||
&& python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel
|
||||
|
||||
# Set up CUDA library paths so GPU-accelerated packages can find CUDA libs at runtime
|
||||
RUN echo '/usr/local/cuda/lib64' >> /etc/ld.so.conf.d/cuda.conf && ldconfig
|
||||
|
||||
# Environment configuration for CUDA and Python
|
||||
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH \
|
||||
PATH=/usr/local/cuda/bin:$PATH \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
MODEL_DIR=/models
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy ONLY dependency specification files first (changes less often than source code)
|
||||
COPY pyproject.toml README.md LICENSE /app/
|
||||
|
||||
# Create minimal package stubs so pip can resolve the editable install
|
||||
# pip install -e . needs package directories to exist but doesn't validate content
|
||||
RUN mkdir -p /app/karaoke_gen /app/lyrics_transcriber_temp /app/backend && \
|
||||
touch /app/karaoke_gen/__init__.py /app/lyrics_transcriber_temp/__init__.py /app/backend/__init__.py
|
||||
|
||||
# Install all karaoke-gen Python dependencies (the slow part)
|
||||
# BuildKit cache mount preserves downloaded wheels across builds
|
||||
RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
pip install --upgrade pip && \
|
||||
pip install -e .
|
||||
|
||||
# Install audio-separator with GPU support (PyTorch + CUDA + onnxruntime-gpu)
|
||||
# This is separate from pip install -e . to keep layer caching granular
|
||||
#
|
||||
# IMPORTANT: Install PyTorch cu126 FIRST, then audio-separator[gpu].
|
||||
# The default audio-separator[gpu] pulls PyTorch cu130, but Cloud Run L4 GPUs
|
||||
# have NVIDIA driver 570 which only supports up to CUDA 12.8. PyTorch cu130
|
||||
# fails with "NVIDIA driver is too old". PyTorch cu126 works with 12.8 drivers.
|
||||
# Installing torch first means audio-separator[gpu] sees it as already satisfied.
|
||||
RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
pip install torch==2.6.0+cu126 torchvision==0.21.0+cu126 --index-url https://download.pytorch.org/whl/cu126 && \
|
||||
pip install "audio-separator[gpu]"
|
||||
|
||||
# Now copy actual source code (AFTER deps installed, so code changes don't
|
||||
# invalidate the expensive pip install layer in registry cache)
|
||||
COPY karaoke_gen /app/karaoke_gen
|
||||
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
|
||||
COPY backend /app/backend
|
||||
|
||||
# Download spaCy language model for lyrics transcription
|
||||
# Must be done at build time, not runtime (Cloud Run network restrictions)
|
||||
RUN python -m spacy download en_core_web_sm
|
||||
|
||||
# Create models directory and pre-download ensemble preset models
|
||||
# Baking models into the image eliminates cold-start model download latency.
|
||||
# Total model size: ~1-1.5 GB for instrumental_clean + karaoke presets.
|
||||
RUN mkdir -p /models
|
||||
COPY backend/scripts/download_models.py /tmp/download_models.py
|
||||
RUN python /tmp/download_models.py /models && \
|
||||
rm /tmp/download_models.py && \
|
||||
ls -lh /models/
|
||||
|
||||
# Label with build info
|
||||
ARG BUILD_DATE
|
||||
ARG BUILD_ID
|
||||
ARG BASE_CONTENT_HASH
|
||||
LABEL org.opencontainers.image.created="${BUILD_DATE}"
|
||||
LABEL org.opencontainers.image.revision="${BUILD_ID}"
|
||||
LABEL org.opencontainers.image.title="karaoke-backend-gpu-base"
|
||||
LABEL org.opencontainers.image.description="GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models for karaoke-backend audio worker"
|
||||
LABEL base.content.hash="${BASE_CONTENT_HASH}"
|
||||
@@ -113,6 +113,19 @@ async def require_auth(
|
||||
detail=f"Authentication failed: {auth_result.message}"
|
||||
)
|
||||
|
||||
# If token has a tenant_id, override the hostname-based tenant detection.
|
||||
# This allows tenant-scoped tokens to enforce tenant context regardless of domain.
|
||||
# We must also load tenant_config so downstream handlers (e.g. job creation)
|
||||
# that rely on tenant_config for setting tenant_id on jobs work correctly.
|
||||
if auth_result.tenant_id:
|
||||
request.state.tenant_id = auth_result.tenant_id
|
||||
if not getattr(request.state, "tenant_config", None):
|
||||
from backend.services.tenant_service import get_tenant_service
|
||||
tenant_service = get_tenant_service()
|
||||
tenant_config = tenant_service.get_tenant_config(auth_result.tenant_id)
|
||||
if tenant_config and tenant_config.is_active:
|
||||
request.state.tenant_config = tenant_config
|
||||
|
||||
# Log successful auth with request_id for correlation with request audit
|
||||
logger.info(
|
||||
"auth_success",
|
||||
@@ -122,6 +135,7 @@ async def require_auth(
|
||||
"user_type": auth_result.user_type.value if auth_result.user_type else None,
|
||||
"is_admin": auth_result.is_admin,
|
||||
"remaining_uses": auth_result.remaining_uses,
|
||||
"tenant_id": auth_result.tenant_id,
|
||||
"audit_type": "auth_event",
|
||||
}
|
||||
)
|
||||
|
||||
+2116
-28
File diff suppressed because it is too large.
Load diff
@@ -17,6 +17,8 @@ import mimetypes
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import uuid
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Optional, List, Dict, Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request, Depends
|
||||
@@ -35,6 +37,7 @@ from backend.services.audio_search_service import (
|
||||
NoResultsError,
|
||||
DownloadError,
|
||||
)
|
||||
from backend.services.flacfetch_client import get_flacfetch_client, FlacfetchServiceError
|
||||
from backend.services.youtube_download_service import (
|
||||
get_youtube_download_service,
|
||||
YouTubeDownloadError,
|
||||
@@ -46,7 +49,11 @@ from backend.version import VERSION
|
||||
from backend.api.dependencies import require_auth
|
||||
from backend.services.auth_service import UserType, AuthResult
|
||||
from backend.middleware.tenant import get_tenant_config_from_request
|
||||
from backend.exceptions import InsufficientCreditsError
|
||||
from backend.services.tracing import add_span_attribute
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
from pathlib import Path
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -134,6 +141,9 @@ class AudioSearchRequest(BaseModel):
|
||||
# Non-interactive mode
|
||||
non_interactive: bool = Field(False, description="Skip interactive steps (lyrics review, instrumental selection)")
|
||||
|
||||
# Private (non-published) track mode
|
||||
is_private: bool = Field(False, description="Private track: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive")
|
||||
|
||||
@validator('artist', 'title', 'lyrics_artist', 'lyrics_title', 'display_artist', 'display_title')
|
||||
def normalize_text_fields(cls, v):
|
||||
"""Normalize text fields to standardize Unicode characters."""
|
||||
@@ -192,9 +202,32 @@ class AudioSearchResponse(BaseModel):
|
||||
style_upload_urls: Optional[List[StyleUploadUrl]] = None
|
||||
|
||||
|
||||
class StandaloneSearchRequest(BaseModel):
|
||||
"""Request for standalone audio search (no job created)."""
|
||||
artist: str = Field(..., description="Artist name to search for")
|
||||
title: str = Field(..., description="Song title to search for")
|
||||
|
||||
@validator('artist', 'title')
|
||||
def normalize_text_fields(cls, v):
|
||||
if v is not None and isinstance(v, str):
|
||||
return normalize_text(v)
|
||||
return v
|
||||
|
||||
|
||||
class StandaloneSearchResponse(BaseModel):
|
||||
"""Response from standalone audio search."""
|
||||
search_session_id: str
|
||||
results: List[AudioSearchResultResponse]
|
||||
results_count: int
|
||||
|
||||
|
||||
class AudioSelectRequest(BaseModel):
|
||||
"""Request to select an audio source."""
|
||||
selection_index: int = Field(..., description="Index of the selected audio source from search results")
|
||||
# Optional overrides — sent from the guided flow's Customize step (Step 3)
|
||||
is_private: Optional[bool] = Field(None, description="Override privacy flag (guided flow Step 3)")
|
||||
display_artist: Optional[str] = Field(None, description="Override display artist name")
|
||||
display_title: Optional[str] = Field(None, description="Override display title")
|
||||
|
||||
|
||||
class AudioSelectResponse(BaseModel):
|
||||
@@ -208,24 +241,24 @@ class AudioSelectResponse(BaseModel):
|
||||
selected_provider: str
|
||||
|
||||
|
||||
def extract_request_metadata(request: Request, created_from: str = "audio_search") -> Dict[str, Any]:
|
||||
def extract_request_metadata(request: Request, created_from: str = "audio_search", auth_result=None) -> Dict[str, Any]:
|
||||
"""Extract metadata from request for job tracking."""
|
||||
headers = dict(request.headers)
|
||||
|
||||
|
||||
client_ip = headers.get('x-forwarded-for', '').split(',')[0].strip()
|
||||
if not client_ip and request.client:
|
||||
client_ip = request.client.host
|
||||
|
||||
|
||||
user_agent = headers.get('user-agent', '')
|
||||
environment = headers.get('x-environment', '')
|
||||
client_id = headers.get('x-client-id', '')
|
||||
|
||||
|
||||
custom_headers = {}
|
||||
for key, value in headers.items():
|
||||
if key.lower().startswith('x-') and key.lower() not in ('x-forwarded-for', 'x-forwarded-proto', 'x-forwarded-host'):
|
||||
custom_headers[key] = value
|
||||
|
||||
return {
|
||||
|
||||
metadata = {
|
||||
'client_ip': client_ip,
|
||||
'user_agent': user_agent,
|
||||
'environment': environment,
|
||||
@@ -235,13 +268,16 @@ def extract_request_metadata(request: Request, created_from: str = "audio_search
|
||||
'custom_headers': custom_headers,
|
||||
}
|
||||
|
||||
# Add auth context for abuse investigation and analytics
|
||||
if auth_result is not None:
|
||||
metadata['auth_method'] = getattr(auth_result, 'auth_method', None)
|
||||
user_type = getattr(auth_result, 'user_type', None)
|
||||
metadata['user_type'] = user_type.value if hasattr(user_type, 'value') else str(user_type) if user_type else None
|
||||
remaining = getattr(auth_result, 'remaining_uses', None)
|
||||
if remaining is not None:
|
||||
metadata['credits_at_creation'] = remaining
|
||||
|
||||
async def _trigger_workers_parallel(job_id: str) -> None:
|
||||
"""Trigger both audio and lyrics workers in parallel."""
|
||||
await asyncio.gather(
|
||||
worker_service.trigger_audio_worker(job_id),
|
||||
worker_service.trigger_lyrics_worker(job_id)
|
||||
)
|
||||
return metadata
|
||||
|
||||
|
||||
def _extract_gcs_path(filepath: str) -> str:
|
||||
@@ -262,43 +298,33 @@ def _extract_gcs_path(filepath: str) -> str:
|
||||
return filepath
|
||||
|
||||
|
||||
async def _download_and_start_processing(
|
||||
def _validate_and_prepare_selection(
|
||||
job_id: str,
|
||||
selection_index: int,
|
||||
audio_search_service,
|
||||
background_tasks: BackgroundTasks,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Download selected audio and start job processing.
|
||||
|
||||
This is called either:
|
||||
- Immediately after search if auto_download=True
|
||||
- When user calls the select endpoint
|
||||
|
||||
For remote flacfetch downloads (torrent sources), the file is downloaded
|
||||
on the flacfetch VM and uploaded directly to GCS. For local downloads
|
||||
(YouTube), the file is downloaded locally and then uploaded to GCS.
|
||||
Validate selection and transition job to DOWNLOADING_AUDIO state.
|
||||
|
||||
Returns selection info dict with the selected result details.
|
||||
Raises HTTPException on validation errors.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
raise HTTPException(status_code=404, detail=t("en", "audioSearch.jobNotFound", job_id=job_id))
|
||||
|
||||
# Get search results from state_data
|
||||
search_results = job.state_data.get('audio_search_results', [])
|
||||
if not search_results:
|
||||
raise HTTPException(status_code=400, detail="No search results available")
|
||||
raise HTTPException(status_code=400, detail=t("en", "audioSearch.noSearchResults"))
|
||||
|
||||
if selection_index < 0 or selection_index >= len(search_results):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid selection index {selection_index}. Valid range: 0-{len(search_results)-1}"
|
||||
detail=t("en", "audioSearch.invalidSelectionIndex", selection_index=selection_index, max=len(search_results)-1)
|
||||
)
|
||||
|
||||
selected = search_results[selection_index]
|
||||
|
||||
# Get remote_search_id from state_data (stored during initial search)
|
||||
remote_search_id = job.state_data.get('remote_search_id')
|
||||
|
||||
# Transition to downloading state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
@@ -310,8 +336,59 @@ async def _download_and_start_processing(
|
||||
'selected_audio_provider': selected['provider'],
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# Save download params to job BEFORE attempting download
|
||||
# This enables retry even if download fails
|
||||
source_name = selected.get('provider')
|
||||
source_id = selected.get('source_id')
|
||||
target_file = selected.get('target_file')
|
||||
download_url = selected.get('url')
|
||||
|
||||
job_manager.update_job(job_id, {
|
||||
'audio_source_type': 'audio_search',
|
||||
'source_name': source_name,
|
||||
'source_id': source_id,
|
||||
'target_file': target_file,
|
||||
'download_url': download_url,
|
||||
})
|
||||
logger.info(
|
||||
f"Job {job_id}: Saved download params before attempt: "
|
||||
f"source_name={source_name}, source_id={source_id}"
|
||||
)
|
||||
|
||||
return {
|
||||
'selected_index': selection_index,
|
||||
'selected_title': selected['title'],
|
||||
'selected_artist': selected['artist'],
|
||||
'selected_provider': selected['provider'],
|
||||
}
|
||||
|
||||
|
||||
async def _download_audio_and_trigger_workers(
|
||||
job_id: str,
|
||||
selection_index: int,
|
||||
audio_search_service,
|
||||
) -> None:
|
||||
"""
|
||||
Download audio and trigger processing workers.
|
||||
|
||||
This runs as a background task after the select endpoint returns.
|
||||
Also called synchronously from the auto_download path during search.
|
||||
|
||||
For remote flacfetch downloads (torrent sources), the file is downloaded
|
||||
on the flacfetch VM and uploaded directly to GCS. For local downloads
|
||||
(YouTube), the file is downloaded locally and then uploaded to GCS.
|
||||
"""
|
||||
try:
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
logger.error(f"Job {job_id} not found during background download")
|
||||
return
|
||||
|
||||
search_results = job.state_data.get('audio_search_results', [])
|
||||
selected = search_results[selection_index]
|
||||
remote_search_id = job.state_data.get('remote_search_id')
|
||||
|
||||
# Get source info from selected result
|
||||
source_name = selected.get('provider')
|
||||
source_id = selected.get('source_id')
|
||||
@@ -345,8 +422,9 @@ async def _download_and_start_processing(
|
||||
raise DownloadError(f"YouTube download failed: {e}")
|
||||
|
||||
elif source_name in ['RED', 'OPS']:
|
||||
# Torrent sources - must use remote flacfetch
|
||||
if not audio_search_service.is_remote_enabled():
|
||||
# Torrent sources - must use remote flacfetch (async, no event loop blocking)
|
||||
flacfetch_client = get_flacfetch_client()
|
||||
if not flacfetch_client:
|
||||
raise DownloadError(
|
||||
f"Cannot download from {source_name} without remote flacfetch service. "
|
||||
"Configure FLACFETCH_API_URL."
|
||||
@@ -356,31 +434,48 @@ async def _download_and_start_processing(
|
||||
|
||||
if source_id:
|
||||
logger.info(f"Torrent download via download_by_id: {source_name} ID={source_id}")
|
||||
result = audio_search_service.download_by_id(
|
||||
download_id = await flacfetch_client.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir="",
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_destination,
|
||||
)
|
||||
else:
|
||||
if not remote_search_id:
|
||||
raise DownloadError(
|
||||
f"No source_id or remote_search_id available for {source_name} download"
|
||||
)
|
||||
logger.info(f"Torrent download via search-based download")
|
||||
result = audio_search_service.download(
|
||||
download_id = await flacfetch_client.download(
|
||||
search_id=remote_search_id,
|
||||
result_index=selection_index,
|
||||
output_dir="",
|
||||
gcs_path=gcs_destination,
|
||||
remote_search_id=remote_search_id,
|
||||
)
|
||||
|
||||
# Extract GCS path from result
|
||||
audio_gcs_path = _extract_gcs_path(result.filepath)
|
||||
filename = os.path.basename(result.filepath)
|
||||
def log_progress(status):
|
||||
progress = status.get("progress", 0)
|
||||
speed = status.get("download_speed_kbps", 0)
|
||||
logger.debug(f"Download progress: {progress:.1f}% ({speed:.1f} KB/s)")
|
||||
|
||||
final_status = await flacfetch_client.wait_for_download(
|
||||
download_id,
|
||||
timeout=600,
|
||||
progress_callback=log_progress,
|
||||
)
|
||||
|
||||
filepath = final_status.get("gcs_path") or final_status.get("output_path")
|
||||
if not filepath:
|
||||
raise DownloadError("Remote download completed but no file path returned")
|
||||
|
||||
audio_gcs_path = _extract_gcs_path(filepath)
|
||||
filename = os.path.basename(filepath)
|
||||
logger.info(f"Torrent download complete: {audio_gcs_path}")
|
||||
|
||||
elif source_name == 'Spotify':
|
||||
# Spotify downloads - use audio_search_service
|
||||
if not audio_search_service.is_remote_enabled():
|
||||
# Spotify downloads - use async FlacfetchClient directly (no event loop blocking)
|
||||
flacfetch_client = get_flacfetch_client()
|
||||
if not flacfetch_client:
|
||||
raise DownloadError(
|
||||
f"Cannot download from Spotify without remote flacfetch service. "
|
||||
"Configure FLACFETCH_API_URL."
|
||||
@@ -389,25 +484,40 @@ async def _download_and_start_processing(
|
||||
gcs_destination = f"uploads/{job_id}/audio/"
|
||||
logger.info(f"Spotify download: source_id={source_id}")
|
||||
|
||||
result = audio_search_service.download_by_id(
|
||||
download_id = await flacfetch_client.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir="",
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_destination,
|
||||
)
|
||||
|
||||
audio_gcs_path = _extract_gcs_path(result.filepath)
|
||||
filename = os.path.basename(result.filepath)
|
||||
def log_progress(status):
|
||||
progress = status.get("progress", 0)
|
||||
speed = status.get("download_speed_kbps", 0)
|
||||
logger.debug(f"Download progress: {progress:.1f}% ({speed:.1f} KB/s)")
|
||||
|
||||
final_status = await flacfetch_client.wait_for_download(
|
||||
download_id,
|
||||
timeout=600,
|
||||
progress_callback=log_progress,
|
||||
)
|
||||
|
||||
filepath = final_status.get("gcs_path") or final_status.get("output_path")
|
||||
if not filepath:
|
||||
raise DownloadError("Remote download completed but no file path returned")
|
||||
|
||||
audio_gcs_path = _extract_gcs_path(filepath)
|
||||
filename = os.path.basename(filepath)
|
||||
logger.info(f"Spotify download complete: {audio_gcs_path}")
|
||||
|
||||
else:
|
||||
# Unknown source - try generic download
|
||||
# Unknown source - try generic download (run sync code in thread to avoid blocking)
|
||||
logger.warning(f"Unknown source type: {source_name}, attempting generic download")
|
||||
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
|
||||
|
||||
try:
|
||||
result = audio_search_service.download(
|
||||
result = await asyncio.to_thread(
|
||||
audio_search_service.download,
|
||||
result_index=selection_index,
|
||||
output_dir=temp_dir,
|
||||
remote_search_id=remote_search_id,
|
||||
@@ -433,36 +543,36 @@ async def _download_and_start_processing(
|
||||
shutil.rmtree(temp_dir)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to clean up temp files: {e}")
|
||||
|
||||
# Update job with GCS path and transition to DOWNLOADING
|
||||
|
||||
# Update job with GCS path
|
||||
job_manager.update_job(job_id, {
|
||||
'input_media_gcs_path': audio_gcs_path,
|
||||
'filename': filename,
|
||||
})
|
||||
|
||||
|
||||
# Transition from DOWNLOADING_AUDIO to DOWNLOADING and trigger workers
|
||||
# Note: Can't use start_job_processing() here because that expects PENDING status,
|
||||
# but we're in DOWNLOADING_AUDIO state from the audio search flow.
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.DOWNLOADING,
|
||||
progress=15,
|
||||
message="Audio downloaded, starting processing"
|
||||
)
|
||||
|
||||
# Trigger workers
|
||||
background_tasks.add_task(_trigger_workers_parallel, job_id)
|
||||
|
||||
return {
|
||||
'selected_index': selection_index,
|
||||
'selected_title': selected['title'],
|
||||
'selected_artist': selected['artist'],
|
||||
'selected_provider': selected['provider'],
|
||||
}
|
||||
|
||||
except DownloadError as e:
|
||||
|
||||
# Trigger workers directly
|
||||
worker_service = get_worker_service()
|
||||
await asyncio.gather(
|
||||
worker_service.trigger_audio_worker(job_id),
|
||||
worker_service.trigger_lyrics_worker(job_id)
|
||||
)
|
||||
|
||||
except (DownloadError, FlacfetchServiceError) as e:
|
||||
logger.error(f"Download failed for job {job_id}: {e}")
|
||||
job_manager.fail_job(job_id, f"Audio download failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Download failed: {e}")
|
||||
except Exception as e:
|
||||
logger.error(f"Download failed for job {job_id}: {e}", exc_info=True)
|
||||
job_manager.fail_job(job_id, f"Audio download failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Download failed: {e}")
|
||||
|
||||
|
||||
@router.post("/audio-search/search", response_model=AudioSearchResponse)
|
||||
@@ -485,12 +595,14 @@ async def search_audio(
|
||||
- Interactive mode (default): Returns results, user calls /select endpoint
|
||||
- Auto mode (auto_download=True): Automatically selects and downloads best
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
|
||||
# Check tenant feature flag
|
||||
tenant_config = get_tenant_config_from_request(request)
|
||||
if tenant_config and not tenant_config.features.audio_search:
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="Audio search is not available for this portal"
|
||||
detail=t(locale, "audioSearch.notEnabled")
|
||||
)
|
||||
|
||||
try:
|
||||
@@ -535,8 +647,9 @@ async def search_audio(
|
||||
}
|
||||
)
|
||||
|
||||
# Extract request metadata
|
||||
request_metadata = extract_request_metadata(request, created_from="audio_search")
|
||||
# Extract request metadata (includes auth context + search query info)
|
||||
request_metadata = extract_request_metadata(request, created_from="audio_search", auth_result=auth_result)
|
||||
request_metadata['search_query'] = {'artist': body.artist, 'title': body.title}
|
||||
|
||||
# Apply default theme if none specified
|
||||
# This ensures all karaoke videos use the Nomad theme by default
|
||||
@@ -588,12 +701,18 @@ async def search_audio(
|
||||
auto_download=body.auto_download,
|
||||
request_metadata=request_metadata,
|
||||
non_interactive=body.non_interactive,
|
||||
is_private=body.is_private,
|
||||
# Tenant scoping
|
||||
tenant_id=tenant_config.id if tenant_config else None,
|
||||
tenant_id=tenant_config.id if tenant_config else "",
|
||||
)
|
||||
job = job_manager.create_job(job_create, is_admin=auth_result.is_admin)
|
||||
job_id = job.job_id
|
||||
|
||||
# Trace attributes for observability
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("job.source", "audio_search")
|
||||
add_span_attribute("job.is_private", body.is_private)
|
||||
|
||||
logger.info(f"Created job {job_id} for audio search: {body.artist} - {body.title}")
|
||||
|
||||
# Update job with audio search fields
|
||||
@@ -650,11 +769,11 @@ async def search_audio(
|
||||
|
||||
# Generate GCS path
|
||||
if file_info.file_type == 'style_params':
|
||||
gcs_path = f"uploads/{job_id}/style/style_params.json"
|
||||
gcs_path = f"jobs/{job_id}/style/style_params.json"
|
||||
else:
|
||||
# style_intro_background -> intro_background, etc.
|
||||
asset_key = file_info.file_type.replace('style_', '')
|
||||
gcs_path = f"uploads/{job_id}/style/{asset_key}{ext}"
|
||||
gcs_path = f"jobs/{job_id}/style/{asset_key}{ext}"
|
||||
|
||||
# Generate signed upload URL
|
||||
signed_url = storage_service.generate_signed_upload_url(
|
||||
@@ -696,7 +815,7 @@ async def search_audio(
|
||||
audio_search_service = get_audio_search_service()
|
||||
|
||||
try:
|
||||
search_results = audio_search_service.search(body.artist, body.title)
|
||||
search_results = await audio_search_service.search_async(body.artist, body.title)
|
||||
except NoResultsError as e:
|
||||
job_manager.fail_job(job_id, f"No audio sources found for: {body.artist} - {body.title}")
|
||||
raise HTTPException(
|
||||
@@ -727,16 +846,21 @@ async def search_audio(
|
||||
# If auto_download, select best and start processing
|
||||
if body.auto_download:
|
||||
best_index = audio_search_service.select_best(search_results)
|
||||
|
||||
|
||||
logger.info(f"Auto-download enabled, selecting result {best_index}")
|
||||
|
||||
selection_info = await _download_and_start_processing(
|
||||
|
||||
selection_info = _validate_and_prepare_selection(
|
||||
job_id=job_id,
|
||||
selection_index=best_index,
|
||||
audio_search_service=audio_search_service,
|
||||
background_tasks=background_tasks,
|
||||
)
|
||||
|
||||
|
||||
# Trigger audio download as a Cloud Run Job (survives instance shutdown)
|
||||
worker_service = get_worker_service()
|
||||
triggered = await worker_service.trigger_audio_download_worker(job_id)
|
||||
if not triggered:
|
||||
job_manager.fail_job(job_id, "Failed to trigger audio download worker")
|
||||
raise HTTPException(status_code=500, detail="Failed to trigger audio download worker")
|
||||
|
||||
return AudioSearchResponse(
|
||||
status="success",
|
||||
job_id=job_id,
|
||||
@@ -824,10 +948,149 @@ async def search_audio(
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
if isinstance(e, InsufficientCreditsError):
|
||||
raise
|
||||
logger.error(f"Error in audio search: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.post("/audio-search/search-standalone", response_model=StandaloneSearchResponse)
|
||||
async def search_audio_standalone(
|
||||
request: Request,
|
||||
body: StandaloneSearchRequest,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Search for audio by artist and title WITHOUT creating a job.
|
||||
|
||||
Returns a search session ID and results. Call POST /api/jobs/create-from-search
|
||||
with the session ID and selected result index to create the actual job.
|
||||
|
||||
Credits are checked here (early feedback) but NOT deducted — deduction
|
||||
happens when the job is created via create-from-search.
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
|
||||
# Check tenant feature flag
|
||||
tenant_config = get_tenant_config_from_request(request)
|
||||
if tenant_config and not tenant_config.features.audio_search:
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail=t(locale, "audioSearch.notEnabled")
|
||||
)
|
||||
|
||||
try:
|
||||
user_email = auth_result.user_email
|
||||
if not user_email:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
|
||||
# Credit check — give early feedback but do NOT deduct here
|
||||
if not auth_result.is_admin:
|
||||
from backend.services.user_service import get_user_service
|
||||
user_service = get_user_service()
|
||||
if not user_service.has_credits(user_email):
|
||||
credits_available = user_service.check_credits(user_email)
|
||||
raise InsufficientCreditsError(
|
||||
message="You're out of credits. Buy more to continue creating karaoke videos.",
|
||||
credits_available=credits_available,
|
||||
credits_required=1,
|
||||
)
|
||||
|
||||
# Perform the search (runs in thread to avoid blocking the event loop)
|
||||
audio_search_service = get_audio_search_service()
|
||||
try:
|
||||
search_results = await audio_search_service.search_async(body.artist, body.title)
|
||||
except NoResultsError:
|
||||
# No results is a normal outcome — return empty list, not an error
|
||||
search_results = []
|
||||
except AudioSearchError as e:
|
||||
raise HTTPException(status_code=500, detail=f"Search failed: {e}")
|
||||
|
||||
# Build result response objects
|
||||
result_responses: List[AudioSearchResultResponse] = []
|
||||
results_dicts: List[Dict[str, Any]] = []
|
||||
for r in search_results:
|
||||
raw_dict: Dict[str, Any] = {}
|
||||
if r.raw_result:
|
||||
if isinstance(r.raw_result, dict):
|
||||
raw_dict = r.raw_result
|
||||
else:
|
||||
try:
|
||||
raw_dict = r.raw_result.to_dict()
|
||||
except AttributeError:
|
||||
pass
|
||||
|
||||
result_responses.append(
|
||||
AudioSearchResultResponse(
|
||||
index=r.index,
|
||||
title=r.title,
|
||||
artist=r.artist,
|
||||
provider=r.provider,
|
||||
url=r.url,
|
||||
duration=r.duration,
|
||||
quality=r.quality,
|
||||
source_id=r.source_id,
|
||||
seeders=raw_dict.get('seeders') or r.seeders,
|
||||
target_file=raw_dict.get('target_file') or r.target_file,
|
||||
year=raw_dict.get('year'),
|
||||
label=raw_dict.get('label'),
|
||||
edition_info=raw_dict.get('edition_info'),
|
||||
release_type=raw_dict.get('release_type'),
|
||||
channel=raw_dict.get('channel'),
|
||||
view_count=raw_dict.get('view_count'),
|
||||
size_bytes=raw_dict.get('size_bytes'),
|
||||
target_file_size=raw_dict.get('target_file_size'),
|
||||
track_pattern=raw_dict.get('track_pattern'),
|
||||
match_score=raw_dict.get('match_score'),
|
||||
formatted_size=raw_dict.get('formatted_size'),
|
||||
formatted_duration=raw_dict.get('formatted_duration'),
|
||||
formatted_views=raw_dict.get('formatted_views'),
|
||||
is_lossless=raw_dict.get('is_lossless'),
|
||||
quality_str=raw_dict.get('quality_str'),
|
||||
quality_data=raw_dict.get('quality_data') or raw_dict.get('quality'),
|
||||
)
|
||||
)
|
||||
results_dicts.append(r.to_dict())
|
||||
|
||||
# Store session in Firestore (TTL: 7 days)
|
||||
# Long TTL so users who leave a tab open mid-flow don't hit "Search expired".
|
||||
# Sessions are tiny docs (~2 KB) and cleaned up by Firestore TTL policy.
|
||||
session_id = str(uuid.uuid4())
|
||||
now = datetime.utcnow()
|
||||
session_data = {
|
||||
'session_id': session_id,
|
||||
'user_email': user_email,
|
||||
'tenant_id': tenant_config.id if tenant_config else None,
|
||||
'artist': body.artist,
|
||||
'title': body.title,
|
||||
'results': results_dicts,
|
||||
'remote_search_id': audio_search_service.last_remote_search_id,
|
||||
'created_at': now.isoformat(),
|
||||
'ttl_expiry': now + timedelta(days=7),
|
||||
}
|
||||
firestore_service = FirestoreService()
|
||||
firestore_service.create_search_session(session_data)
|
||||
|
||||
logger.info(
|
||||
f"Standalone search complete for {body.artist} - {body.title}: "
|
||||
f"{len(search_results)} results, session={session_id}"
|
||||
)
|
||||
|
||||
return StandaloneSearchResponse(
|
||||
search_session_id=session_id,
|
||||
results=result_responses,
|
||||
results_count=len(result_responses),
|
||||
)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except InsufficientCreditsError:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error in standalone audio search: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.get("/audio-search/{job_id}/results")
|
||||
async def get_audio_search_results(
|
||||
job_id: str,
|
||||
@@ -840,14 +1103,14 @@ async def get_audio_search_results(
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
raise HTTPException(status_code=404, detail=t("en", "audioSearch.jobNotFound", job_id=job_id))
|
||||
|
||||
search_results = job.state_data.get('audio_search_results', [])
|
||||
|
||||
|
||||
if not search_results:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="No search results available for this job"
|
||||
detail=t("en", "audioSearch.noSearchResults")
|
||||
)
|
||||
|
||||
return {
|
||||
@@ -870,45 +1133,64 @@ async def select_audio_source(
|
||||
):
|
||||
"""
|
||||
Select an audio source and start job processing.
|
||||
|
||||
This endpoint:
|
||||
1. Validates the job is awaiting selection
|
||||
2. Downloads the selected audio
|
||||
3. Uploads to GCS
|
||||
4. Triggers audio and lyrics workers
|
||||
|
||||
This endpoint returns immediately after validating the selection and
|
||||
transitioning the job to DOWNLOADING_AUDIO state. The actual download
|
||||
runs as a background task so the UI can close the selection dialog
|
||||
without waiting for the download to complete.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
raise HTTPException(status_code=404, detail=t("en", "audioSearch.jobNotFound", job_id=job_id))
|
||||
|
||||
# Verify job is awaiting selection
|
||||
if job.status != JobStatus.AWAITING_AUDIO_SELECTION:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job is not awaiting audio selection (status: {job.status})"
|
||||
)
|
||||
|
||||
|
||||
# Get search service instance
|
||||
# Note: With download_by_id, we no longer need to re-search to populate the cache.
|
||||
# The source_id stored in job.state_data['audio_search_results'] is sufficient.
|
||||
audio_search_service = get_audio_search_service()
|
||||
|
||||
# Apply optional overrides from guided flow Step 3 (Customize)
|
||||
# These fields are set AFTER the job is created (Step 2), so they must be
|
||||
# patched here before the pipeline starts.
|
||||
overrides = {}
|
||||
if body.is_private is not None:
|
||||
overrides["is_private"] = body.is_private
|
||||
if body.display_artist is not None:
|
||||
overrides["display_artist"] = body.display_artist
|
||||
if body.display_title is not None:
|
||||
overrides["display_title"] = body.display_title
|
||||
if overrides:
|
||||
job_manager.update_job(job_id, overrides)
|
||||
logger.info(f"[job:{job_id}] Applied selection overrides: {list(overrides.keys())}")
|
||||
|
||||
# Validate search results exist in job state_data
|
||||
search_results = job.state_data.get('audio_search_results', [])
|
||||
if not search_results:
|
||||
raise HTTPException(status_code=400, detail="No search results cached for this job")
|
||||
|
||||
selection_info = await _download_and_start_processing(
|
||||
|
||||
# Validate selection and transition to DOWNLOADING_AUDIO (returns immediately)
|
||||
selection_info = _validate_and_prepare_selection(
|
||||
job_id=job_id,
|
||||
selection_index=body.selection_index,
|
||||
audio_search_service=audio_search_service,
|
||||
background_tasks=background_tasks,
|
||||
)
|
||||
|
||||
|
||||
# Trigger audio download as a Cloud Run Job (survives instance shutdown)
|
||||
worker_service = get_worker_service()
|
||||
triggered = await worker_service.trigger_audio_download_worker(job_id)
|
||||
if not triggered:
|
||||
job_manager.fail_job(job_id, "Failed to trigger audio download worker")
|
||||
raise HTTPException(status_code=500, detail="Failed to trigger audio download worker")
|
||||
|
||||
return AudioSelectResponse(
|
||||
status="success",
|
||||
job_id=job_id,
|
||||
message="Audio selected and download started",
|
||||
message="Audio selected, download starting",
|
||||
selected_index=selection_info['selected_index'],
|
||||
selected_title=selection_info['selected_title'],
|
||||
selected_artist=selection_info['selected_artist'],
|
||||
|
||||
+41
-35
@@ -9,7 +9,7 @@ Provides endpoints for:
|
||||
import logging
|
||||
from typing import Optional
|
||||
from datetime import datetime
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.services.credential_manager import (
|
||||
@@ -18,6 +18,7 @@ from backend.services.credential_manager import (
|
||||
CredentialCheckResult,
|
||||
)
|
||||
from backend.config import get_settings
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/auth", tags=["auth"])
|
||||
@@ -125,15 +126,16 @@ async def get_credentials_status():
|
||||
|
||||
|
||||
@router.get("/status/{service}", response_model=CredentialStatusResponse)
|
||||
async def get_service_credential_status(service: str):
|
||||
async def get_service_credential_status(service: str, request: Request):
|
||||
"""
|
||||
Get the status of a specific service's OAuth credentials.
|
||||
|
||||
|
||||
Args:
|
||||
service: One of "youtube", "gdrive", "dropbox"
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
manager = get_credential_manager()
|
||||
|
||||
|
||||
if service == "youtube":
|
||||
result = manager.check_youtube_credentials()
|
||||
elif service == "gdrive":
|
||||
@@ -141,7 +143,7 @@ async def get_service_credential_status(service: str):
|
||||
elif service == "dropbox":
|
||||
result = manager.check_dropbox_credentials()
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail=f"Unknown service: {service}")
|
||||
raise HTTPException(status_code=400, detail=t(locale, "auth.unknownService", service=service))
|
||||
|
||||
return CredentialStatusResponse(
|
||||
service=result.service,
|
||||
@@ -153,42 +155,43 @@ async def get_service_credential_status(service: str):
|
||||
|
||||
|
||||
@router.post("/validate", response_model=CredentialValidationResponse)
|
||||
async def validate_credentials(request: CredentialValidationRequest):
|
||||
async def validate_credentials(request: CredentialValidationRequest, http_request: Request):
|
||||
"""
|
||||
Validate that credentials are available for requested services.
|
||||
|
||||
|
||||
Use this before submitting a job to ensure all required
|
||||
credentials are valid.
|
||||
"""
|
||||
locale = get_locale_from_request(http_request)
|
||||
manager = get_credential_manager()
|
||||
invalid_services = []
|
||||
|
||||
|
||||
if request.youtube:
|
||||
result = manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append("youtube")
|
||||
|
||||
|
||||
if request.gdrive:
|
||||
result = manager.check_gdrive_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append("gdrive")
|
||||
|
||||
|
||||
if request.dropbox:
|
||||
result = manager.check_dropbox_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append("dropbox")
|
||||
|
||||
|
||||
if invalid_services:
|
||||
return CredentialValidationResponse(
|
||||
valid=False,
|
||||
invalid_services=invalid_services,
|
||||
message=f"The following services need re-authorization: {', '.join(invalid_services)}"
|
||||
message=t(locale, "auth.invalidCredentials", services=", ".join(invalid_services))
|
||||
)
|
||||
|
||||
|
||||
return CredentialValidationResponse(
|
||||
valid=True,
|
||||
invalid_services=[],
|
||||
message="All requested credentials are valid"
|
||||
message=t(locale, "auth.validCredentials")
|
||||
)
|
||||
|
||||
|
||||
@@ -197,40 +200,41 @@ async def validate_credentials(request: CredentialValidationRequest):
|
||||
# ===========================================================================
|
||||
|
||||
@router.post("/youtube/device", response_model=DeviceAuthStartResponse)
|
||||
async def start_youtube_device_auth(request: Optional[DeviceAuthStartRequest] = None):
|
||||
async def start_youtube_device_auth(request: Optional[DeviceAuthStartRequest] = None, http_request: Request = None):
|
||||
"""
|
||||
Start YouTube device authorization flow.
|
||||
|
||||
|
||||
This initiates a device auth flow that allows authorization
|
||||
from any device. The user must visit the verification URL
|
||||
and enter the user code.
|
||||
|
||||
|
||||
Client credentials are loaded from Secret Manager ('youtube-client-credentials')
|
||||
unless explicitly provided in the request body.
|
||||
|
||||
|
||||
After starting, poll /auth/youtube/device/{device_code} to
|
||||
check for completion.
|
||||
"""
|
||||
locale = get_locale_from_request(http_request) if http_request else "en"
|
||||
manager = get_credential_manager()
|
||||
|
||||
|
||||
try:
|
||||
device_info = manager.start_youtube_device_auth(
|
||||
client_id=request.client_id if request else None,
|
||||
client_secret=request.client_secret if request else None
|
||||
)
|
||||
|
||||
|
||||
return DeviceAuthStartResponse(
|
||||
device_code=device_info.device_code,
|
||||
user_code=device_info.user_code,
|
||||
verification_url=device_info.verification_url,
|
||||
expires_in=device_info.expires_in,
|
||||
interval=device_info.interval,
|
||||
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
|
||||
instructions=t(locale, "auth.youtubeAuthInstructions", verification_url=device_info.verification_url, user_code=device_info.user_code)
|
||||
)
|
||||
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to start YouTube device auth: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
raise HTTPException(status_code=500, detail=t(locale, "auth.youtubeAuthStartFailed", error=str(e)))
|
||||
|
||||
|
||||
@router.get("/youtube/device/{device_code}", response_model=DeviceAuthPollResponse)
|
||||
@@ -259,35 +263,36 @@ async def poll_youtube_device_auth(device_code: str):
|
||||
|
||||
|
||||
@router.post("/gdrive/device", response_model=DeviceAuthStartResponse)
|
||||
async def start_gdrive_device_auth(request: Optional[DeviceAuthStartRequest] = None):
|
||||
async def start_gdrive_device_auth(request: Optional[DeviceAuthStartRequest] = None, http_request: Request = None):
|
||||
"""
|
||||
Start Google Drive device authorization flow.
|
||||
|
||||
|
||||
Client credentials are loaded from Secret Manager ('gdrive-client-credentials')
|
||||
unless explicitly provided in the request body.
|
||||
|
||||
|
||||
After starting, poll /auth/gdrive/device/{device_code} to check for completion.
|
||||
"""
|
||||
locale = get_locale_from_request(http_request) if http_request else "en"
|
||||
manager = get_credential_manager()
|
||||
|
||||
|
||||
try:
|
||||
device_info = manager.start_gdrive_device_auth(
|
||||
client_id=request.client_id if request else None,
|
||||
client_secret=request.client_secret if request else None
|
||||
)
|
||||
|
||||
|
||||
return DeviceAuthStartResponse(
|
||||
device_code=device_info.device_code,
|
||||
user_code=device_info.user_code,
|
||||
verification_url=device_info.verification_url,
|
||||
expires_in=device_info.expires_in,
|
||||
interval=device_info.interval,
|
||||
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
|
||||
instructions=t(locale, "auth.gdriveLinkInstructions", verification_url=device_info.verification_url, user_code=device_info.user_code)
|
||||
)
|
||||
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to start Google Drive device auth: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
raise HTTPException(status_code=500, detail=t(locale, "auth.gdriveAuthStartFailed", error=str(e)))
|
||||
|
||||
|
||||
@router.get("/gdrive/device/{device_code}", response_model=DeviceAuthPollResponse)
|
||||
@@ -311,19 +316,20 @@ async def poll_gdrive_device_auth(device_code: str):
|
||||
# ===========================================================================
|
||||
|
||||
@router.post("/test-alert")
|
||||
async def test_credential_alert(background_tasks: BackgroundTasks):
|
||||
async def test_credential_alert(background_tasks: BackgroundTasks, request: Request):
|
||||
"""
|
||||
Test the credential alert mechanism.
|
||||
|
||||
|
||||
Sends a test alert to Discord if configured.
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
settings = get_settings()
|
||||
discord_url = settings.get_secret("discord-alert-webhook")
|
||||
|
||||
|
||||
if not discord_url:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="Discord alert webhook not configured"
|
||||
detail=t(locale, "auth.discordAlertNotConfigured")
|
||||
)
|
||||
|
||||
manager = get_credential_manager()
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
"""
|
||||
Catalog API routes for song/artist lookup and community version detection.
|
||||
|
||||
Provides autocomplete data from karaoke-decide's MusicBrainz + Spotify catalog,
|
||||
and checks karaokenerds.com for existing community karaoke versions.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from backend.api.dependencies import require_auth
|
||||
from backend.services.auth_service import AuthResult
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/catalog", tags=["catalog"])
|
||||
|
||||
# Simple per-user rate limiting: 20 requests/minute
|
||||
# { user_email: [timestamp, timestamp, ...] }
|
||||
_rate_limit_windows: dict[str, list[float]] = {}
|
||||
RATE_LIMIT_MAX_REQUESTS = 20
|
||||
RATE_LIMIT_WINDOW_SECONDS = 60
|
||||
|
||||
|
||||
def _check_rate_limit(user_email: str, locale: str = "en") -> None:
|
||||
"""Check per-user rate limit. Raises 429 if exceeded."""
|
||||
now = time.monotonic()
|
||||
window_start = now - RATE_LIMIT_WINDOW_SECONDS
|
||||
|
||||
# Get or create window for this user
|
||||
if user_email not in _rate_limit_windows:
|
||||
_rate_limit_windows[user_email] = []
|
||||
|
||||
# Remove expired entries
|
||||
timestamps = _rate_limit_windows[user_email]
|
||||
_rate_limit_windows[user_email] = [ts for ts in timestamps if ts > window_start]
|
||||
|
||||
if len(_rate_limit_windows[user_email]) >= RATE_LIMIT_MAX_REQUESTS:
|
||||
raise HTTPException(
|
||||
status_code=429,
|
||||
detail=t(locale, "catalog.rateLimitExceeded", max_requests=RATE_LIMIT_MAX_REQUESTS),
|
||||
)
|
||||
|
||||
_rate_limit_windows[user_email].append(now)
|
||||
|
||||
|
||||
# --- Response Models ---
|
||||
|
||||
|
||||
class ArtistResult(BaseModel):
|
||||
"""Artist search result from MusicBrainz/Spotify catalog."""
|
||||
name: str
|
||||
mbid: Optional[str] = None
|
||||
disambiguation: Optional[str] = None
|
||||
artist_type: Optional[str] = None
|
||||
spotify_id: Optional[str] = None
|
||||
popularity: Optional[int] = None
|
||||
genres: Optional[list[str]] = None
|
||||
tags: Optional[list[str]] = None
|
||||
|
||||
|
||||
class TrackResult(BaseModel):
|
||||
"""Track search result from Spotify catalog."""
|
||||
track_name: str
|
||||
artist_name: str
|
||||
track_id: Optional[str] = None
|
||||
artist_id: Optional[str] = None
|
||||
popularity: Optional[int] = None
|
||||
duration_ms: Optional[int] = None
|
||||
explicit: Optional[bool] = None
|
||||
|
||||
|
||||
class CommunityTrack(BaseModel):
|
||||
"""A community-approved karaoke track from karaokenerds."""
|
||||
brand_name: str
|
||||
brand_code: str
|
||||
youtube_url: str
|
||||
is_community: bool = True
|
||||
|
||||
|
||||
class CommunityCheckSong(BaseModel):
|
||||
"""A song with community karaoke tracks."""
|
||||
title: str
|
||||
artist: str
|
||||
community_tracks: list[CommunityTrack]
|
||||
|
||||
|
||||
class CommunityCheckRequest(BaseModel):
|
||||
"""Request body for community version check."""
|
||||
artist: str = Field(..., min_length=1)
|
||||
title: str = Field(..., min_length=1)
|
||||
|
||||
|
||||
class CommunityCheckResponse(BaseModel):
|
||||
"""Response for community version check."""
|
||||
has_community: bool
|
||||
songs: list[CommunityCheckSong]
|
||||
best_youtube_url: Optional[str] = None
|
||||
|
||||
|
||||
# --- Routes ---
|
||||
|
||||
|
||||
@router.get("/artists", response_model=list[ArtistResult])
|
||||
async def search_artists(
|
||||
request: Request,
|
||||
q: str = Query(..., min_length=2, description="Artist name search query"),
|
||||
limit: int = Query(10, ge=1, le=50, description="Maximum results"),
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""
|
||||
Search for artists by name. Returns canonical MusicBrainz/Spotify artist data
|
||||
for autocomplete in job creation forms.
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
|
||||
|
||||
from backend.services.catalog_proxy_service import search_artists
|
||||
results = await search_artists(q, limit)
|
||||
return results
|
||||
|
||||
|
||||
@router.get("/tracks", response_model=list[TrackResult])
|
||||
async def search_tracks(
|
||||
request: Request,
|
||||
q: str = Query(..., min_length=2, description="Track name search query"),
|
||||
artist: Optional[str] = Query(None, description="Filter by artist name"),
|
||||
limit: int = Query(10, ge=1, le=50, description="Maximum results"),
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""
|
||||
Search for tracks by name, optionally filtered by artist.
|
||||
Returns canonical Spotify track data for autocomplete in job creation forms.
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
|
||||
|
||||
from backend.services.catalog_proxy_service import search_tracks
|
||||
results = await search_tracks(q, artist, limit)
|
||||
return results
|
||||
|
||||
|
||||
@router.post("/community-check", response_model=CommunityCheckResponse)
|
||||
async def check_community_versions(
|
||||
request: Request,
|
||||
body: CommunityCheckRequest,
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""
|
||||
Check if a song already has community-approved karaoke versions on YouTube
|
||||
via karaokenerds.com. Non-blocking — just informs the user.
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
|
||||
|
||||
from backend.services.karaokenerds_service import check_community_versions
|
||||
result = await check_community_versions(body.artist, body.title)
|
||||
return result
|
||||
@@ -0,0 +1,65 @@
|
||||
"""
|
||||
Encoding worker lifecycle endpoints.
|
||||
|
||||
Provides warmup and heartbeat endpoints for the blue-green
|
||||
encoding worker VMs. Called by the frontend to ensure the
|
||||
primary VM is running before encoding requests.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from fastapi import APIRouter, Depends
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.encoding_worker_manager import EncodingWorkerManager
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/internal/encoding-worker", tags=["encoding-worker"])
|
||||
|
||||
# Lazy-loaded singleton
|
||||
_manager = None
|
||||
|
||||
|
||||
def get_worker_manager():
|
||||
global _manager
|
||||
if _manager is None:
|
||||
from google.cloud import compute_v1, firestore
|
||||
from backend.config import get_settings
|
||||
settings = get_settings()
|
||||
db = firestore.Client(project=settings.google_cloud_project)
|
||||
compute_client = compute_v1.InstancesClient()
|
||||
_manager = EncodingWorkerManager(
|
||||
db=db,
|
||||
compute_client=compute_client,
|
||||
project_id=settings.google_cloud_project,
|
||||
)
|
||||
return _manager
|
||||
|
||||
|
||||
@router.post("/warmup")
|
||||
async def warmup_encoding_worker(
|
||||
_admin=Depends(require_admin),
|
||||
manager=Depends(get_worker_manager),
|
||||
):
|
||||
"""Start the primary encoding worker VM if it's stopped."""
|
||||
try:
|
||||
result = manager.ensure_primary_running()
|
||||
if result["started"]:
|
||||
logger.info(f"Started encoding worker VM: {result['vm_name']}")
|
||||
return result
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to warm up encoding worker: {e}")
|
||||
return {"started": False, "error": str(e)}
|
||||
|
||||
|
||||
@router.post("/heartbeat")
|
||||
async def heartbeat_encoding_worker(
|
||||
_admin=Depends(require_admin),
|
||||
manager=Depends(get_worker_manager),
|
||||
):
|
||||
"""Update activity timestamp to prevent idle shutdown."""
|
||||
try:
|
||||
manager.update_activity()
|
||||
return {"status": "ok"}
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to update encoding worker heartbeat: {e}")
|
||||
return {"status": "error", "error": str(e)}
|
||||
+505
-114
File diff suppressed because it is too large.
Load diff
@@ -5,10 +5,12 @@ import os
|
||||
import logging
|
||||
import subprocess
|
||||
import platform
|
||||
from fastapi import APIRouter
|
||||
from typing import Dict, Any, Optional
|
||||
from fastapi import APIRouter, Depends
|
||||
from typing import Dict, Any, Optional, Tuple
|
||||
|
||||
from backend.version import VERSION
|
||||
from backend.version import VERSION, COMMIT_SHA, PR_NUMBER, PR_TITLE, STARTUP_TIME
|
||||
from backend.api.dependencies import optional_auth
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.flacfetch_client import get_flacfetch_client
|
||||
from backend.services.email_service import get_email_service
|
||||
from backend.services.stripe_service import get_stripe_service
|
||||
@@ -16,6 +18,9 @@ from backend.services.encoding_service import get_encoding_service
|
||||
from backend.services.spacy_preloader import get_preloaded_model, is_model_preloaded
|
||||
from backend.services.nltk_preloader import get_preloaded_cmudict, is_cmudict_preloaded
|
||||
from backend.services.langfuse_preloader import get_preloaded_langfuse_handler, is_langfuse_preloaded, is_langfuse_configured
|
||||
from backend.services.job_health_service import check_job_consistency_detailed
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.models.job import JobStatus
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -305,6 +310,175 @@ async def flacfetch_health() -> Dict[str, Any]:
|
||||
}
|
||||
|
||||
|
||||
def check_audio_separator_status() -> Dict[str, Any]:
|
||||
"""Check audio separator availability without calling the remote GPU service."""
|
||||
from backend.config import get_settings
|
||||
|
||||
settings = get_settings()
|
||||
if not settings.audio_separator_api_url:
|
||||
return {
|
||||
"available": False,
|
||||
"status": "not_configured",
|
||||
}
|
||||
|
||||
try:
|
||||
from importlib.metadata import version
|
||||
pkg_version = version("audio-separator")
|
||||
except Exception:
|
||||
pkg_version = None
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"status": "ok",
|
||||
"version": pkg_version,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/audio-separator")
|
||||
async def audio_separator_health() -> Dict[str, Any]:
|
||||
"""
|
||||
Return the installed audio-separator package version.
|
||||
|
||||
Does NOT call the remote GPU service — that would cold-start an expensive
|
||||
Cloud Run GPU instance (~$0.07 per wake). Instead reports the locally
|
||||
installed package version, which matches what the audio worker uses.
|
||||
|
||||
No authentication required.
|
||||
"""
|
||||
from backend.config import get_settings
|
||||
|
||||
settings = get_settings()
|
||||
if not settings.audio_separator_api_url:
|
||||
return {
|
||||
"available": False,
|
||||
"status": "not_configured",
|
||||
}
|
||||
|
||||
try:
|
||||
from importlib.metadata import version
|
||||
pkg_version = version("audio-separator")
|
||||
except Exception:
|
||||
pkg_version = None
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"status": "ok",
|
||||
"version": pkg_version,
|
||||
}
|
||||
|
||||
|
||||
def _get_encoding_worker_manager():
|
||||
"""Get the EncodingWorkerManager instance. Separated for testability."""
|
||||
try:
|
||||
encoding_service = get_encoding_service()
|
||||
if hasattr(encoding_service, '_worker_manager') and encoding_service._worker_manager:
|
||||
return encoding_service._worker_manager
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
@router.get("/health/system-status")
|
||||
async def system_status(
|
||||
auth: Optional[Tuple] = Depends(optional_auth),
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Aggregated system status for all services.
|
||||
|
||||
Returns basic info for everyone. Admin users get additional
|
||||
blue-green deployment details for the encoder.
|
||||
"""
|
||||
is_admin = auth is not None and len(auth) >= 2 and auth[1] == UserType.ADMIN
|
||||
|
||||
# Fetch all health statuses concurrently
|
||||
import asyncio
|
||||
encoding_task = check_encoding_worker_status()
|
||||
flacfetch_task = check_flacfetch_service_status()
|
||||
encoding_status, flacfetch_status = await asyncio.gather(
|
||||
encoding_task, flacfetch_task
|
||||
)
|
||||
separator_status = check_audio_separator_status()
|
||||
|
||||
# Build frontend service info (from env vars baked at build time — returned as-is)
|
||||
frontend_svc = {
|
||||
"status": "ok",
|
||||
"version": VERSION, # Frontend and backend share the same version from pyproject.toml
|
||||
}
|
||||
|
||||
# Build backend service info
|
||||
backend_svc = {
|
||||
"status": "ok",
|
||||
"version": VERSION,
|
||||
"deployed_at": STARTUP_TIME,
|
||||
}
|
||||
if COMMIT_SHA:
|
||||
backend_svc["commit_sha"] = COMMIT_SHA
|
||||
if PR_NUMBER:
|
||||
backend_svc["pr_number"] = PR_NUMBER
|
||||
if PR_TITLE:
|
||||
backend_svc["pr_title"] = PR_TITLE
|
||||
|
||||
# Build encoder service info
|
||||
encoder_svc = {
|
||||
"status": "ok" if encoding_status.get("available") else "offline",
|
||||
"version": encoding_status.get("wheel_version"),
|
||||
"active_jobs": encoding_status.get("active_jobs", 0),
|
||||
}
|
||||
|
||||
# Build flacfetch service info
|
||||
flacfetch_svc = {
|
||||
"status": "ok" if flacfetch_status.get("available") else "offline",
|
||||
"version": flacfetch_status.get("version"),
|
||||
}
|
||||
|
||||
# Build separator service info
|
||||
separator_svc = {
|
||||
"status": "ok" if separator_status.get("available") else "offline",
|
||||
"version": separator_status.get("version"),
|
||||
}
|
||||
|
||||
# Admin-only: blue-green encoder details from Firestore
|
||||
if is_admin:
|
||||
manager = _get_encoding_worker_manager()
|
||||
if manager:
|
||||
try:
|
||||
config = manager.get_config()
|
||||
encoder_svc["admin_details"] = {
|
||||
"primary_vm": config.primary_vm,
|
||||
"primary_ip": config.primary_ip,
|
||||
"primary_version": config.primary_version,
|
||||
"primary_deployed_at": config.primary_deployed_at,
|
||||
"secondary_vm": config.secondary_vm,
|
||||
"secondary_ip": config.secondary_ip,
|
||||
"secondary_version": config.secondary_version,
|
||||
"secondary_deployed_at": config.secondary_deployed_at,
|
||||
"last_swap_at": config.last_swap_at,
|
||||
"deploy_in_progress": config.deploy_in_progress,
|
||||
"active_jobs": encoding_status.get("active_jobs", 0),
|
||||
"queue_length": encoding_status.get("queue_length", 0),
|
||||
}
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to get blue-green config: {e}")
|
||||
|
||||
# Add error details for offline services
|
||||
if not flacfetch_status.get("available"):
|
||||
flacfetch_svc["admin_details"] = {"error": flacfetch_status.get("error")}
|
||||
if not separator_status.get("available"):
|
||||
separator_svc["admin_details"] = {"error": separator_status.get("error")}
|
||||
if not encoding_status.get("available") and "admin_details" not in encoder_svc:
|
||||
encoder_svc["admin_details"] = {"error": encoding_status.get("error")}
|
||||
|
||||
return {
|
||||
"services": {
|
||||
"frontend": frontend_svc,
|
||||
"backend": backend_svc,
|
||||
"encoder": encoder_svc,
|
||||
"flacfetch": flacfetch_svc,
|
||||
"separator": separator_svc,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/detailed")
|
||||
async def detailed_health_check() -> Dict[str, Any]:
|
||||
"""
|
||||
@@ -430,6 +604,69 @@ async def preload_status() -> Dict[str, Any]:
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/job-consistency")
|
||||
async def job_consistency_check(
|
||||
status: Optional[str] = None,
|
||||
limit: int = 50
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Check job consistency across the system.
|
||||
|
||||
This endpoint identifies jobs with state inconsistencies that may indicate
|
||||
bugs in the state machine or trigger logic. Use this to find stuck jobs
|
||||
or jobs in invalid states.
|
||||
|
||||
Args:
|
||||
status: Optional filter by job status (e.g., "pending", "downloading")
|
||||
limit: Maximum number of jobs to check (default 50)
|
||||
|
||||
Returns:
|
||||
Dictionary with:
|
||||
- total_checked: Number of jobs checked
|
||||
- inconsistent_count: Number of jobs with issues
|
||||
- inconsistent_jobs: List of job details with their issues
|
||||
- summary: Summary of issue types found
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
|
||||
# Query jobs to check
|
||||
query_params = {}
|
||||
if status:
|
||||
try:
|
||||
job_status = JobStatus(status)
|
||||
query_params['status'] = job_status
|
||||
except ValueError:
|
||||
return {
|
||||
"error": f"Invalid status: {status}",
|
||||
"valid_statuses": [s.value for s in JobStatus]
|
||||
}
|
||||
|
||||
# Get jobs from Firestore
|
||||
jobs = job_manager.list_jobs(limit=limit, **query_params)
|
||||
|
||||
# Check each job for consistency
|
||||
inconsistent_jobs = []
|
||||
issue_summary = {}
|
||||
|
||||
for job in jobs:
|
||||
result = check_job_consistency_detailed(job)
|
||||
if not result['is_healthy']:
|
||||
inconsistent_jobs.append(result)
|
||||
# Count issue types
|
||||
for issue in result['issues']:
|
||||
issue_type = issue.split(':')[0] if ':' in issue else issue
|
||||
issue_summary[issue_type] = issue_summary.get(issue_type, 0) + 1
|
||||
|
||||
return {
|
||||
"total_checked": len(jobs),
|
||||
"inconsistent_count": len(inconsistent_jobs),
|
||||
"inconsistent_jobs": inconsistent_jobs,
|
||||
"summary": issue_summary,
|
||||
"status_filter": status,
|
||||
"limit": limit,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/readiness")
|
||||
async def readiness_check() -> Dict[str, str]:
|
||||
"""Readiness check endpoint for Cloud Run."""
|
||||
|
||||
@@ -25,7 +25,7 @@ from backend.workers.screens_worker import generate_screens
|
||||
from backend.workers.video_worker import generate_video
|
||||
from backend.workers.render_video_worker import process_render_video
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.auth_service import AuthResult, UserType
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.tracing import (
|
||||
extract_trace_context,
|
||||
@@ -76,6 +76,18 @@ def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerR
|
||||
message=f"Job {job_id} not found"
|
||||
)
|
||||
|
||||
# Reject workers for jobs in terminal states (failed, cancelled, complete)
|
||||
# This prevents Cloud Tasks retries from triggering workers on dead jobs
|
||||
terminal_statuses = {'failed', 'cancelled', 'complete', 'prep_complete', 'error'}
|
||||
job_status = job.status.value if hasattr(job.status, 'value') else str(job.status)
|
||||
if job_status in terminal_statuses:
|
||||
logger.info(f"[job:{job_id}] Job in terminal state '{job_status}', skipping {worker_name} worker")
|
||||
return WorkerResponse(
|
||||
status="skipped",
|
||||
job_id=job_id,
|
||||
message=f"Job is in terminal state '{job_status}', {worker_name} worker not needed"
|
||||
)
|
||||
|
||||
# Check worker-specific progress in state_data
|
||||
progress_key = f"{worker_name}_progress"
|
||||
worker_progress = job.state_data.get(progress_key, {})
|
||||
@@ -102,6 +114,31 @@ def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerR
|
||||
return None
|
||||
|
||||
|
||||
@router.post("/sync-disposable-domains", response_model=WorkerResponse)
|
||||
async def sync_disposable_domains_endpoint(
|
||||
request: Request,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Sync external disposable domain blocklist. Called by Cloud Scheduler daily."""
|
||||
import asyncio
|
||||
from backend.services.disposable_domain_sync_service import (
|
||||
fetch_external_blocklist, sync_disposable_domains
|
||||
)
|
||||
from backend.services.firestore_service import get_firestore_client
|
||||
from backend.services.email_validation_service import EmailValidationService
|
||||
|
||||
domains = await fetch_external_blocklist()
|
||||
db = get_firestore_client()
|
||||
result = await asyncio.to_thread(sync_disposable_domains, db, domains)
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
return WorkerResponse(
|
||||
status="completed",
|
||||
job_id="sync-disposable-domains",
|
||||
message=f"Synced {result['external_count']} domains"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/workers/audio", response_model=WorkerResponse)
|
||||
async def trigger_audio_worker(
|
||||
request: WorkerRequest,
|
||||
@@ -428,6 +465,175 @@ async def check_idle_reminder(
|
||||
return {"status": "error", "job_id": job_id, "message": str(e)}
|
||||
|
||||
|
||||
@router.post("/youtube-queue/process")
|
||||
async def process_youtube_upload_queue(
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Process queued YouTube uploads.
|
||||
|
||||
Called by Cloud Scheduler (hourly) to retry uploads that were deferred
|
||||
due to YouTube API quota exhaustion. Can also be triggered manually
|
||||
from the admin dashboard.
|
||||
|
||||
Processes queued uploads one at a time, stopping if quota is exhausted.
|
||||
"""
|
||||
from backend.workers.youtube_queue_processor import process_youtube_upload_queue as process_queue
|
||||
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info("YOUTUBE_QUEUE_PROCESS starting")
|
||||
add_span_attribute("operation", "youtube_queue_process")
|
||||
|
||||
# Run in background so the HTTP response returns quickly
|
||||
# (Cloud Scheduler has a 30min timeout but we don't want to block)
|
||||
async def _process():
|
||||
try:
|
||||
result = await process_queue()
|
||||
logger.info(f"YOUTUBE_QUEUE_PROCESS complete: {result}")
|
||||
except Exception as e:
|
||||
logger.exception(f"YOUTUBE_QUEUE_PROCESS failed: {e}")
|
||||
|
||||
background_tasks.add_task(_process)
|
||||
|
||||
add_span_event("queue_processing_started")
|
||||
return {
|
||||
"status": "started",
|
||||
"message": "YouTube upload queue processing started in background"
|
||||
}
|
||||
|
||||
|
||||
@router.post("/process-stale-reviews")
|
||||
async def process_stale_reviews_endpoint(
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Process stale review jobs — send reminders and expire old ones.
|
||||
|
||||
Called by Cloud Scheduler (hourly). Can also be triggered manually
|
||||
from the admin dashboard.
|
||||
|
||||
- Jobs in review for >= 24h get a reminder email
|
||||
- Jobs in review for >= 48h are auto-cancelled with credit refund
|
||||
"""
|
||||
from backend.workers.stale_review_processor import process_stale_reviews
|
||||
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info("STALE_REVIEW_PROCESS starting")
|
||||
add_span_attribute("operation", "stale_review_process")
|
||||
|
||||
async def _process():
|
||||
try:
|
||||
result = await process_stale_reviews()
|
||||
logger.info(f"STALE_REVIEW_PROCESS complete: {result}")
|
||||
except Exception as e:
|
||||
logger.exception(f"STALE_REVIEW_PROCESS failed: {e}")
|
||||
|
||||
background_tasks.add_task(_process)
|
||||
|
||||
add_span_event("stale_review_processing_started")
|
||||
return {
|
||||
"status": "started",
|
||||
"message": "Stale review processing started in background"
|
||||
}
|
||||
|
||||
|
||||
@router.post("/trigger-gdrive-validation")
|
||||
async def trigger_gdrive_validation_endpoint(
|
||||
http_request: Request,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Trigger GDrive validation via the Cloud Function.
|
||||
|
||||
Called by a Cloud Tasks delayed task after a job completes and uploads
|
||||
to the public GDrive share. The 5-minute delay ensures E2E test cleanup
|
||||
finishes before the validator checks for sequence gaps.
|
||||
"""
|
||||
from backend.services.gdrive_validator_client import trigger_gdrive_validation
|
||||
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info("GDRIVE_VALIDATION_TRIGGER starting (delayed post-job)")
|
||||
add_span_attribute("operation", "gdrive_validation_trigger")
|
||||
|
||||
try:
|
||||
result = trigger_gdrive_validation()
|
||||
if result is None:
|
||||
return {"status": "skipped", "message": "GDRIVE_VALIDATOR_URL not configured"}
|
||||
|
||||
status = result.get("status", "unknown")
|
||||
add_span_event("validation_complete", {"result_status": status})
|
||||
return {"status": status, "message": "GDrive validation completed", "result": result}
|
||||
except Exception as e:
|
||||
logger.exception(f"GDrive validation trigger failed: {e}")
|
||||
add_span_event("error", {"error": str(e)})
|
||||
return {"status": "error", "message": str(e)}
|
||||
|
||||
|
||||
@router.post("/recover-stuck-jobs")
|
||||
async def recover_stuck_jobs(
|
||||
http_request: Request,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Detect and recover jobs stuck in downloading_audio status.
|
||||
|
||||
Called by Cloud Scheduler (every 5 minutes) or manually from admin.
|
||||
Finds jobs stuck in DOWNLOADING_AUDIO for >10 minutes and fails them
|
||||
so they can be retried.
|
||||
"""
|
||||
from backend.services.job_health_service import check_job_consistency
|
||||
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
logger.info("RECOVER_STUCK_JOBS starting")
|
||||
add_span_attribute("operation", "recover_stuck_jobs")
|
||||
|
||||
job_manager = JobManager()
|
||||
|
||||
# Query jobs in downloading_audio status
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
jobs_ref = job_manager.firestore.db.collection("jobs")
|
||||
stuck_query = jobs_ref.where(
|
||||
filter=FieldFilter("status", "==", "downloading_audio")
|
||||
).stream()
|
||||
|
||||
recovered = []
|
||||
for doc in stuck_query:
|
||||
job_data = doc.to_dict()
|
||||
job_id = job_data.get("job_id", doc.id)
|
||||
|
||||
# Use the health service to check if actually stuck
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
continue
|
||||
|
||||
issues = check_job_consistency(job)
|
||||
stuck_issues = [i for i in issues if "downloading_audio_stuck" in i]
|
||||
|
||||
if stuck_issues:
|
||||
logger.warning(f"[job:{job_id}] Recovering stuck download: {stuck_issues[0]}")
|
||||
job_manager.fail_job(
|
||||
job_id,
|
||||
"Audio download timed out (stuck >10 minutes). Use retry to re-attempt."
|
||||
)
|
||||
recovered.append(job_id)
|
||||
|
||||
logger.info(f"RECOVER_STUCK_JOBS complete: recovered={len(recovered)} jobs")
|
||||
add_span_event("recovery_complete", {"recovered_count": len(recovered)})
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"recovered_jobs": recovered,
|
||||
"recovered_count": len(recovered),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def internal_health(
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
|
||||
+1318
-100
File diff suppressed because it is too large.
Load diff
+43
-18
@@ -11,13 +11,14 @@ Provides endpoints for managing Web Push notification subscriptions:
|
||||
import logging
|
||||
from typing import Optional, Dict, List
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.config import get_settings
|
||||
from backend.api.dependencies import require_auth, require_admin
|
||||
from backend.services.auth_service import AuthResult
|
||||
from backend.services.push_notification_service import get_push_notification_service
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -37,6 +38,7 @@ class SubscribeRequest(BaseModel):
|
||||
endpoint: str
|
||||
keys: Dict[str, str] # p256dh and auth
|
||||
device_name: Optional[str] = None
|
||||
tenant_id: Optional[str] = None # Tenant scope for notification isolation
|
||||
|
||||
|
||||
class SubscribeResponse(BaseModel):
|
||||
@@ -111,7 +113,8 @@ async def get_vapid_public_key():
|
||||
|
||||
@router.post("/subscribe", response_model=SubscribeResponse)
|
||||
async def subscribe_push(
|
||||
request: SubscribeRequest,
|
||||
subscribe_request: SubscribeRequest,
|
||||
http_request: Request,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
@@ -119,39 +122,53 @@ async def subscribe_push(
|
||||
|
||||
Requires authentication. Users can have up to 5 subscriptions
|
||||
(configurable via MAX_PUSH_SUBSCRIPTIONS_PER_USER).
|
||||
|
||||
Subscriptions are scoped by tenant_id to prevent cross-tenant
|
||||
notification leakage. The tenant is resolved from:
|
||||
1. Request body tenant_id field (explicit from frontend)
|
||||
2. Tenant middleware (X-Tenant-ID header or subdomain detection)
|
||||
"""
|
||||
locale = get_locale_from_request(http_request)
|
||||
|
||||
settings = get_settings()
|
||||
if not settings.enable_push_notifications:
|
||||
raise HTTPException(status_code=503, detail="Push notifications are not enabled")
|
||||
raise HTTPException(status_code=503, detail=t(locale, "push.disabled"))
|
||||
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail="User email not available")
|
||||
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
# Validate keys
|
||||
if "p256dh" not in request.keys or "auth" not in request.keys:
|
||||
raise HTTPException(status_code=400, detail="Missing required keys (p256dh, auth)")
|
||||
if "p256dh" not in subscribe_request.keys or "auth" not in subscribe_request.keys:
|
||||
raise HTTPException(status_code=400, detail=t(locale, "push.missingEncryptionKeys"))
|
||||
|
||||
# Resolve tenant_id: prefer explicit from body, fall back to middleware
|
||||
tenant_id = subscribe_request.tenant_id
|
||||
if tenant_id is None:
|
||||
tenant_id = getattr(http_request.state, "tenant_id", None)
|
||||
|
||||
success = await push_service.add_subscription(
|
||||
user_email=auth_result.user_email,
|
||||
endpoint=request.endpoint,
|
||||
keys=request.keys,
|
||||
device_name=request.device_name
|
||||
endpoint=subscribe_request.endpoint,
|
||||
keys=subscribe_request.keys,
|
||||
device_name=subscribe_request.device_name,
|
||||
tenant_id=tenant_id or None,
|
||||
)
|
||||
|
||||
if not success:
|
||||
raise HTTPException(status_code=500, detail="Failed to save subscription")
|
||||
raise HTTPException(status_code=500, detail=t(locale, "push.subscriptionSaveFailed"))
|
||||
|
||||
return SubscribeResponse(
|
||||
status="success",
|
||||
message="Push subscription registered successfully"
|
||||
message=t(locale, "push.subscriptionRegistered")
|
||||
)
|
||||
|
||||
|
||||
@router.post("/unsubscribe", response_model=UnsubscribeResponse)
|
||||
async def unsubscribe_push(
|
||||
request: UnsubscribeRequest,
|
||||
http_request: Request,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
@@ -159,8 +176,10 @@ async def unsubscribe_push(
|
||||
|
||||
Requires authentication. Users can only remove their own subscriptions.
|
||||
"""
|
||||
locale = get_locale_from_request(http_request)
|
||||
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail="User email not available")
|
||||
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
@@ -173,17 +192,18 @@ async def unsubscribe_push(
|
||||
# Don't error if subscription wasn't found - might already be removed
|
||||
return UnsubscribeResponse(
|
||||
status="success",
|
||||
message="Subscription removed (or was not found)"
|
||||
message=t(locale, "push.subscriptionNotFound")
|
||||
)
|
||||
|
||||
return UnsubscribeResponse(
|
||||
status="success",
|
||||
message="Push subscription removed successfully"
|
||||
message=t(locale, "push.subscriptionRemoved")
|
||||
)
|
||||
|
||||
|
||||
@router.get("/subscriptions", response_model=SubscriptionsListResponse)
|
||||
async def list_subscriptions(
|
||||
request: Request,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
@@ -191,8 +211,10 @@ async def list_subscriptions(
|
||||
|
||||
Requires authentication.
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail="User email not available")
|
||||
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
@@ -207,6 +229,7 @@ async def list_subscriptions(
|
||||
@router.post("/test", response_model=TestNotificationResponse)
|
||||
async def send_test_notification(
|
||||
request: TestNotificationRequest,
|
||||
http_request: Request,
|
||||
auth_result: AuthResult = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
@@ -214,12 +237,14 @@ async def send_test_notification(
|
||||
|
||||
Admin only. Useful for testing push notification setup.
|
||||
"""
|
||||
locale = get_locale_from_request(http_request)
|
||||
|
||||
settings = get_settings()
|
||||
if not settings.enable_push_notifications:
|
||||
raise HTTPException(status_code=503, detail="Push notifications are not enabled")
|
||||
raise HTTPException(status_code=503, detail=t(locale, "push.disabled"))
|
||||
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail="User email not available")
|
||||
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
@@ -234,5 +259,5 @@ async def send_test_notification(
|
||||
return TestNotificationResponse(
|
||||
status="success",
|
||||
sent_count=sent_count,
|
||||
message=f"Test notification sent to {sent_count} device(s)"
|
||||
message=t(locale, "push.testNotificationSent", sent_count=sent_count)
|
||||
)
|
||||
+160
-242
@@ -1,24 +1,22 @@
|
||||
"""
|
||||
Admin API routes for rate limit management.
|
||||
Admin API routes for blocklist and YouTube queue management.
|
||||
|
||||
Handles:
|
||||
- Rate limit statistics and monitoring
|
||||
- Blocklist management (disposable domains, blocked emails, blocked IPs)
|
||||
- User override management (whitelist/bypass permissions)
|
||||
- YouTube upload queue management
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from datetime import datetime
|
||||
from typing import List, Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, EmailStr
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.auth_service import AuthResult
|
||||
from backend.services.rate_limit_service import get_rate_limit_service, RateLimitService
|
||||
from backend.services.email_validation_service import get_email_validation_service, EmailValidationService
|
||||
from backend.config import settings
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/admin/rate-limits", tags=["admin", "rate-limits"])
|
||||
@@ -28,45 +26,40 @@ router = APIRouter(prefix="/admin/rate-limits", tags=["admin", "rate-limits"])
|
||||
# Response Models
|
||||
# =============================================================================
|
||||
|
||||
class RateLimitStatsResponse(BaseModel):
|
||||
"""Current rate limit statistics."""
|
||||
# Configuration
|
||||
jobs_per_day_limit: int
|
||||
youtube_uploads_per_day_limit: int
|
||||
beta_ip_per_day_limit: int
|
||||
rate_limiting_enabled: bool
|
||||
|
||||
# Current usage
|
||||
youtube_uploads_today: int
|
||||
youtube_uploads_remaining: int
|
||||
|
||||
# Blocklist stats
|
||||
disposable_domains_count: int
|
||||
blocked_emails_count: int
|
||||
blocked_ips_count: int
|
||||
|
||||
# User override stats
|
||||
total_overrides: int
|
||||
class YouTubeQueueEntry(BaseModel):
|
||||
"""A queued YouTube upload entry."""
|
||||
job_id: str
|
||||
status: str
|
||||
reason: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
artist: Optional[str] = None
|
||||
title: Optional[str] = None
|
||||
brand_code: Optional[str] = None
|
||||
queued_at: Optional[datetime] = None
|
||||
attempts: int = 0
|
||||
max_attempts: int = 5
|
||||
last_error: Optional[str] = None
|
||||
youtube_url: Optional[str] = None
|
||||
notification_sent: bool = False
|
||||
|
||||
|
||||
class UserRateLimitStatusResponse(BaseModel):
|
||||
"""Rate limit status for a specific user."""
|
||||
email: str
|
||||
jobs_today: int
|
||||
jobs_limit: int
|
||||
jobs_remaining: int
|
||||
has_bypass: bool
|
||||
custom_limit: Optional[int]
|
||||
bypass_reason: Optional[str]
|
||||
class YouTubeQueueListResponse(BaseModel):
|
||||
"""List of YouTube upload queue entries."""
|
||||
entries: List[YouTubeQueueEntry]
|
||||
stats: dict
|
||||
|
||||
|
||||
class BlocklistsResponse(BaseModel):
|
||||
"""All blocklist data."""
|
||||
disposable_domains: List[str]
|
||||
"""All blocklist data with domain source separation."""
|
||||
external_domains: List[str]
|
||||
manual_domains: List[str]
|
||||
allowlisted_domains: List[str]
|
||||
blocked_emails: List[str]
|
||||
blocked_ips: List[str]
|
||||
updated_at: Optional[datetime]
|
||||
updated_by: Optional[str]
|
||||
last_sync_at: Optional[datetime] = None
|
||||
last_sync_count: Optional[int] = None
|
||||
updated_at: Optional[datetime] = None
|
||||
updated_by: Optional[str] = None
|
||||
|
||||
|
||||
class DomainRequest(BaseModel):
|
||||
@@ -84,115 +77,12 @@ class IPRequest(BaseModel):
|
||||
ip_address: str
|
||||
|
||||
|
||||
class UserOverride(BaseModel):
|
||||
"""User override configuration."""
|
||||
email: str
|
||||
bypass_job_limit: bool
|
||||
custom_daily_job_limit: Optional[int]
|
||||
reason: str
|
||||
created_by: str
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class UserOverrideRequest(BaseModel):
|
||||
"""Request to set/update a user override."""
|
||||
bypass_job_limit: bool = False
|
||||
custom_daily_job_limit: Optional[int] = None
|
||||
reason: str
|
||||
|
||||
|
||||
class UserOverridesListResponse(BaseModel):
|
||||
"""List of all user overrides."""
|
||||
overrides: List[UserOverride]
|
||||
total: int
|
||||
|
||||
|
||||
class SuccessResponse(BaseModel):
|
||||
"""Generic success response."""
|
||||
success: bool
|
||||
message: str
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Statistics Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/stats", response_model=RateLimitStatsResponse)
|
||||
async def get_rate_limit_stats(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Get current rate limit statistics.
|
||||
|
||||
Returns configuration values, current usage counts, and blocklist stats.
|
||||
"""
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
# Get YouTube uploads today
|
||||
youtube_today = rate_limit_service.get_youtube_uploads_today()
|
||||
youtube_limit = settings.rate_limit_youtube_uploads_per_day
|
||||
youtube_remaining = max(0, youtube_limit - youtube_today) if youtube_limit > 0 else -1
|
||||
|
||||
# Get blocklist stats
|
||||
blocklist_stats = email_validation.get_blocklist_stats()
|
||||
|
||||
# Get override count
|
||||
overrides = rate_limit_service.get_all_overrides()
|
||||
|
||||
return RateLimitStatsResponse(
|
||||
jobs_per_day_limit=settings.rate_limit_jobs_per_day,
|
||||
youtube_uploads_per_day_limit=settings.rate_limit_youtube_uploads_per_day,
|
||||
beta_ip_per_day_limit=settings.rate_limit_beta_ip_per_day,
|
||||
rate_limiting_enabled=settings.enable_rate_limiting,
|
||||
youtube_uploads_today=youtube_today,
|
||||
youtube_uploads_remaining=youtube_remaining,
|
||||
disposable_domains_count=blocklist_stats["disposable_domains_count"],
|
||||
blocked_emails_count=blocklist_stats["blocked_emails_count"],
|
||||
blocked_ips_count=blocklist_stats["blocked_ips_count"],
|
||||
total_overrides=len(overrides),
|
||||
)
|
||||
|
||||
|
||||
@router.get("/users/{email}", response_model=UserRateLimitStatusResponse)
|
||||
async def get_user_rate_limit_status(
|
||||
email: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Get rate limit status for a specific user.
|
||||
|
||||
Returns their current usage and any override settings.
|
||||
"""
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
|
||||
jobs_today = rate_limit_service.get_user_job_count_today(email)
|
||||
override = rate_limit_service.get_user_override(email)
|
||||
|
||||
if override:
|
||||
has_bypass = override.get("bypass_job_limit", False)
|
||||
custom_limit = override.get("custom_daily_job_limit")
|
||||
bypass_reason = override.get("reason")
|
||||
jobs_limit = custom_limit if custom_limit is not None else settings.rate_limit_jobs_per_day
|
||||
else:
|
||||
has_bypass = False
|
||||
custom_limit = None
|
||||
bypass_reason = None
|
||||
jobs_limit = settings.rate_limit_jobs_per_day
|
||||
|
||||
jobs_remaining = max(0, jobs_limit - jobs_today) if not has_bypass else -1
|
||||
|
||||
return UserRateLimitStatusResponse(
|
||||
email=email,
|
||||
jobs_today=jobs_today,
|
||||
jobs_limit=jobs_limit,
|
||||
jobs_remaining=jobs_remaining,
|
||||
has_bypass=has_bypass,
|
||||
custom_limit=custom_limit,
|
||||
bypass_reason=bypass_reason,
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Blocklist Management Endpoints
|
||||
# =============================================================================
|
||||
@@ -201,38 +91,10 @@ async def get_user_rate_limit_status(
|
||||
async def get_blocklists(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Get all blocklist data.
|
||||
|
||||
Returns disposable domains, blocked emails, and blocked IPs.
|
||||
"""
|
||||
"""Get all blocklist data with domain source separation."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
# Force refresh to get latest data
|
||||
config = email_validation.get_blocklist_config(force_refresh=True)
|
||||
|
||||
# Get the raw document for metadata
|
||||
from google.cloud import firestore
|
||||
from backend.services.firestore_service import get_firestore_client
|
||||
from backend.services.email_validation_service import BLOCKLISTS_COLLECTION, BLOCKLIST_CONFIG_DOC
|
||||
|
||||
db = get_firestore_client()
|
||||
doc = db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC).get()
|
||||
|
||||
updated_at = None
|
||||
updated_by = None
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
updated_at = data.get("updated_at")
|
||||
updated_by = data.get("updated_by")
|
||||
|
||||
return BlocklistsResponse(
|
||||
disposable_domains=sorted(list(config["disposable_domains"])),
|
||||
blocked_emails=sorted(list(config["blocked_emails"])),
|
||||
blocked_ips=sorted(list(config["blocked_ips"])),
|
||||
updated_at=updated_at,
|
||||
updated_by=updated_by,
|
||||
)
|
||||
raw_data = email_validation.get_blocklist_raw_data()
|
||||
return BlocklistsResponse(**raw_data)
|
||||
|
||||
|
||||
@router.post("/blocklists/disposable-domains", response_model=SuccessResponse)
|
||||
@@ -245,13 +107,13 @@ async def add_disposable_domain(
|
||||
|
||||
domain = request.domain.lower().strip()
|
||||
if not domain or "." not in domain:
|
||||
raise HTTPException(status_code=400, detail="Invalid domain format")
|
||||
raise HTTPException(status_code=400, detail=t("en", "admin.invalidDomainFormat"))
|
||||
|
||||
email_validation.add_disposable_domain(domain, auth_result.user_email)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Domain '{domain}' added to disposable domains blocklist"
|
||||
message=t("en", "admin.domainAddedToBl", domain=domain)
|
||||
)
|
||||
|
||||
|
||||
@@ -265,11 +127,62 @@ async def remove_disposable_domain(
|
||||
|
||||
domain = domain.lower().strip()
|
||||
if not email_validation.remove_disposable_domain(domain, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail="Domain not found in blocklist")
|
||||
raise HTTPException(status_code=404, detail=t("en", "admin.domainNotFoundInBl"))
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Domain '{domain}' removed from disposable domains blocklist"
|
||||
message=t("en", "admin.domainRemovedFromBl", domain=domain)
|
||||
)
|
||||
|
||||
|
||||
@router.post("/blocklists/allowlisted-domains", response_model=SuccessResponse)
|
||||
async def add_allowlisted_domain(
|
||||
request: DomainRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Add a domain to the allowlist (overrides external blocklist)."""
|
||||
email_validation = get_email_validation_service()
|
||||
domain = request.domain.lower().strip()
|
||||
if not domain or "." not in domain:
|
||||
raise HTTPException(status_code=400, detail=t("en", "admin.invalidDomainFormat"))
|
||||
email_validation.add_allowlisted_domain(domain, auth_result.user_email)
|
||||
return SuccessResponse(success=True, message=t("en", "admin.domainAddedToAllowlist", domain=domain))
|
||||
|
||||
|
||||
@router.delete("/blocklists/allowlisted-domains/{domain}", response_model=SuccessResponse)
|
||||
async def remove_allowlisted_domain(
|
||||
domain: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove a domain from the allowlist."""
|
||||
email_validation = get_email_validation_service()
|
||||
domain = domain.lower().strip()
|
||||
if not email_validation.remove_allowlisted_domain(domain, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail=t("en", "admin.domainNotFoundInAllowlist"))
|
||||
return SuccessResponse(success=True, message=t("en", "admin.domainRemovedFromAllowlist", domain=domain))
|
||||
|
||||
|
||||
@router.post("/blocklists/sync", response_model=SuccessResponse)
|
||||
async def trigger_sync(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Manually trigger a sync of the external disposable domain blocklist."""
|
||||
import asyncio
|
||||
from backend.services.disposable_domain_sync_service import (
|
||||
fetch_external_blocklist, sync_disposable_domains
|
||||
)
|
||||
from backend.services.firestore_service import get_firestore_client
|
||||
|
||||
domains = await fetch_external_blocklist()
|
||||
db = get_firestore_client()
|
||||
result = await asyncio.to_thread(sync_disposable_domains, db, domains)
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=t("en", "admin.syncComplete", external_count=result['external_count'], added=result['added'], removed=result['removed'])
|
||||
)
|
||||
|
||||
|
||||
@@ -283,13 +196,13 @@ async def add_blocked_email(
|
||||
|
||||
email = request.email.lower().strip()
|
||||
if not email or "@" not in email:
|
||||
raise HTTPException(status_code=400, detail="Invalid email format")
|
||||
raise HTTPException(status_code=400, detail=t("en", "admin.invalidEmailFormat"))
|
||||
|
||||
email_validation.add_blocked_email(email, auth_result.user_email)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Email '{email}' added to blocked emails list"
|
||||
message=t("en", "admin.emailAddedToBl", email=email)
|
||||
)
|
||||
|
||||
|
||||
@@ -303,11 +216,11 @@ async def remove_blocked_email(
|
||||
|
||||
email = email.lower().strip()
|
||||
if not email_validation.remove_blocked_email(email, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail="Email not found in blocklist")
|
||||
raise HTTPException(status_code=404, detail=t("en", "admin.emailNotFoundInBl"))
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Email '{email}' removed from blocked emails list"
|
||||
message=t("en", "admin.emailRemovedFromBl", email=email)
|
||||
)
|
||||
|
||||
|
||||
@@ -321,13 +234,13 @@ async def add_blocked_ip(
|
||||
|
||||
ip_address = request.ip_address.strip()
|
||||
if not ip_address:
|
||||
raise HTTPException(status_code=400, detail="Invalid IP address")
|
||||
raise HTTPException(status_code=400, detail=t("en", "admin.invalidIPAddress"))
|
||||
|
||||
email_validation.add_blocked_ip(ip_address, auth_result.user_email)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"IP '{ip_address}' added to blocked IPs list"
|
||||
message=t("en", "admin.ipAddedToBl", ip_address=ip_address)
|
||||
)
|
||||
|
||||
|
||||
@@ -341,88 +254,93 @@ async def remove_blocked_ip(
|
||||
|
||||
ip_address = ip_address.strip()
|
||||
if not email_validation.remove_blocked_ip(ip_address, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail="IP not found in blocklist")
|
||||
raise HTTPException(status_code=404, detail=t("en", "admin.ipNotFoundInBl"))
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"IP '{ip_address}' removed from blocked IPs list"
|
||||
message=t("en", "admin.ipRemovedFromBl", ip_address=ip_address)
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# User Override Management Endpoints
|
||||
# YouTube Upload Queue Management Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/overrides", response_model=UserOverridesListResponse)
|
||||
async def get_all_overrides(
|
||||
@router.get("/youtube-queue", response_model=YouTubeQueueListResponse)
|
||||
async def get_youtube_queue(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Get all user overrides."""
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
"""
|
||||
Get YouTube upload queue entries and stats.
|
||||
|
||||
overrides_data = rate_limit_service.get_all_overrides()
|
||||
Returns all queue entries (any status) and aggregate statistics.
|
||||
"""
|
||||
from backend.services.youtube_upload_queue_service import get_youtube_upload_queue_service
|
||||
|
||||
overrides = [
|
||||
UserOverride(
|
||||
email=email,
|
||||
bypass_job_limit=data.get("bypass_job_limit", False),
|
||||
custom_daily_job_limit=data.get("custom_daily_job_limit"),
|
||||
reason=data.get("reason", ""),
|
||||
created_by=data.get("created_by", "unknown"),
|
||||
created_at=data.get("created_at", datetime.now(timezone.utc)),
|
||||
queue_service = get_youtube_upload_queue_service()
|
||||
entries = queue_service.get_all_queue_entries(limit=50)
|
||||
stats = queue_service.get_queue_stats()
|
||||
|
||||
return YouTubeQueueListResponse(
|
||||
entries=[YouTubeQueueEntry(**entry) for entry in entries],
|
||||
stats=stats,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/youtube-queue/{job_id}/retry", response_model=SuccessResponse)
|
||||
async def retry_youtube_upload(
|
||||
job_id: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Manually retry a failed YouTube upload.
|
||||
|
||||
Resets the entry back to 'queued' status with 0 attempts.
|
||||
"""
|
||||
from backend.services.youtube_upload_queue_service import get_youtube_upload_queue_service
|
||||
|
||||
queue_service = get_youtube_upload_queue_service()
|
||||
|
||||
if not queue_service.retry_upload(job_id):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=t("en", "admin.youtubeUploadNotFound", job_id=job_id)
|
||||
)
|
||||
for email, data in overrides_data.items()
|
||||
]
|
||||
|
||||
return UserOverridesListResponse(
|
||||
overrides=overrides,
|
||||
total=len(overrides),
|
||||
)
|
||||
|
||||
|
||||
@router.put("/overrides/{email}", response_model=SuccessResponse)
|
||||
async def set_user_override(
|
||||
email: str,
|
||||
request: UserOverrideRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Set or update a user override."""
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
|
||||
email = email.lower().strip()
|
||||
if not email or "@" not in email:
|
||||
raise HTTPException(status_code=400, detail="Invalid email format")
|
||||
|
||||
if not request.reason or len(request.reason.strip()) < 3:
|
||||
raise HTTPException(status_code=400, detail="Reason is required (min 3 characters)")
|
||||
|
||||
rate_limit_service.set_user_override(
|
||||
user_email=email,
|
||||
bypass_job_limit=request.bypass_job_limit,
|
||||
custom_daily_job_limit=request.custom_daily_job_limit,
|
||||
reason=request.reason,
|
||||
admin_email=auth_result.user_email,
|
||||
)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Override set for user '{email}'"
|
||||
message=t("en", "admin.youtubeUploadQueuedForRetry", job_id=job_id)
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/overrides/{email}", response_model=SuccessResponse)
|
||||
async def remove_user_override(
|
||||
email: str,
|
||||
@router.post("/youtube-queue/process", response_model=SuccessResponse)
|
||||
async def trigger_youtube_queue_processing(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove a user override."""
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
"""
|
||||
Manually trigger YouTube upload queue processing.
|
||||
|
||||
email = email.lower().strip()
|
||||
if not rate_limit_service.remove_user_override(email, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail="Override not found for user")
|
||||
Same as the scheduled processor but triggered from the admin dashboard.
|
||||
Runs in the background and returns immediately.
|
||||
"""
|
||||
from backend.workers.youtube_queue_processor import process_youtube_upload_queue
|
||||
import asyncio
|
||||
|
||||
async def _process():
|
||||
try:
|
||||
result = await process_youtube_upload_queue()
|
||||
logger.info(f"Manual YouTube queue processing complete: {result}")
|
||||
except Exception as e:
|
||||
logger.exception(f"Manual YouTube queue processing failed: {e}")
|
||||
|
||||
# Fire and forget
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
loop.create_task(_process())
|
||||
except RuntimeError:
|
||||
pass
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Override removed for user '{email}'"
|
||||
message=t("en", "admin.youtubeQueueProcessingStarted")
|
||||
)
|
||||
@@ -0,0 +1,374 @@
|
||||
"""
|
||||
Referral API routes.
|
||||
|
||||
Handles:
|
||||
- Public referral interstitial (click tracking)
|
||||
- Authenticated referral dashboard
|
||||
- Stripe Connect onboarding
|
||||
- Admin vanity link creation and link management
|
||||
"""
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request
|
||||
from fastapi.responses import Response
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from backend.api.dependencies import require_auth, require_admin
|
||||
from backend.models.referral import (
|
||||
CreateReferralLinkRequest,
|
||||
UpdateReferralLinkRequest,
|
||||
ReferralInterstitialResponse,
|
||||
)
|
||||
from backend.services.flyer_service import get_flyer_service, FlyerError
|
||||
from backend.services.referral_service import get_referral_service
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/referrals", tags=["referrals"])
|
||||
|
||||
|
||||
class GenerateFlyerRequest(BaseModel):
|
||||
theme: str = Field(..., pattern="^(light|dark)$")
|
||||
qr_data_url: str = Field(..., max_length=500_000)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Public endpoints
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@router.get("/r/{code}", response_model=ReferralInterstitialResponse)
|
||||
async def get_referral_interstitial(code: str, no_track: bool = False):
|
||||
"""
|
||||
Public endpoint: get interstitial data for a referral link.
|
||||
|
||||
Increments click count unless no_track=true. Returns valid=False if code not found.
|
||||
"""
|
||||
service = get_referral_service()
|
||||
link = service.get_link_by_code(code)
|
||||
|
||||
if not link:
|
||||
return ReferralInterstitialResponse(valid=False)
|
||||
|
||||
# Track the click (skip when fetching for display purposes)
|
||||
if not no_track:
|
||||
try:
|
||||
service.increment_clicks(code)
|
||||
except Exception:
|
||||
logger.warning("Failed to increment clicks for code %s", code, exc_info=True)
|
||||
|
||||
return ReferralInterstitialResponse(
|
||||
valid=True,
|
||||
referral_code=link.code,
|
||||
referrer_display_name=link.display_name,
|
||||
discount_percent=link.discount_percent,
|
||||
discount_duration_days=link.discount_duration_days,
|
||||
custom_message=link.custom_message,
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Authenticated user endpoints
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@router.get("/me")
|
||||
async def get_my_dashboard(auth=Depends(require_auth)):
|
||||
"""Get referral dashboard data for the authenticated user."""
|
||||
service = get_referral_service()
|
||||
return service.get_dashboard_data(auth.user_email)
|
||||
|
||||
|
||||
@router.put("/me")
|
||||
async def update_my_link(
|
||||
updates: UpdateReferralLinkRequest,
|
||||
auth=Depends(require_auth),
|
||||
):
|
||||
"""Update display_name and custom_message on the user's own referral link."""
|
||||
service = get_referral_service()
|
||||
link = service.get_or_create_link(auth.user_email)
|
||||
|
||||
update_fields = {}
|
||||
if updates.display_name is not None:
|
||||
update_fields["display_name"] = updates.display_name
|
||||
if updates.custom_message is not None:
|
||||
update_fields["custom_message"] = updates.custom_message
|
||||
|
||||
if not update_fields:
|
||||
return {"ok": True, "message": "No changes"}
|
||||
|
||||
success, message = service.update_link(link.code, **update_fields)
|
||||
if not success:
|
||||
raise HTTPException(status_code=400, detail=message)
|
||||
|
||||
return {"ok": True, "message": message}
|
||||
|
||||
|
||||
@router.post("/me/connect")
|
||||
async def start_stripe_connect(auth=Depends(require_auth)):
|
||||
"""Start Stripe Connect onboarding for the authenticated user."""
|
||||
# Check if user already has a Connect account
|
||||
from backend.services.user_service import get_user_service
|
||||
user_service = get_user_service()
|
||||
user = user_service.get_user(auth.user_email)
|
||||
if user and user.stripe_connect_account_id:
|
||||
raise HTTPException(status_code=400, detail="Stripe Connect account already configured")
|
||||
|
||||
service = get_referral_service()
|
||||
account_id, onboarding_url = service.create_connect_account(auth.user_email)
|
||||
|
||||
if not account_id:
|
||||
raise HTTPException(status_code=500, detail="Failed to create Stripe Connect account")
|
||||
|
||||
user_service.update_user(auth.user_email, stripe_connect_account_id=account_id)
|
||||
|
||||
return {"account_id": account_id, "onboarding_url": onboarding_url}
|
||||
|
||||
|
||||
@router.post("/me/connect/dashboard-link")
|
||||
async def get_connect_dashboard_link(auth=Depends(require_auth)):
|
||||
"""Get a login link to the Stripe Express dashboard for managing payout settings."""
|
||||
from backend.services.user_service import get_user_service
|
||||
user = get_user_service().get_user(auth.user_email)
|
||||
if not user or not user.stripe_connect_account_id:
|
||||
raise HTTPException(status_code=400, detail="No Stripe Connect account configured")
|
||||
|
||||
service = get_referral_service()
|
||||
url = service.create_connect_login_link(user.stripe_connect_account_id)
|
||||
if not url:
|
||||
raise HTTPException(status_code=500, detail="Failed to create dashboard link")
|
||||
|
||||
return {"url": url}
|
||||
|
||||
|
||||
@router.post("/me/connect/update-link")
|
||||
async def get_connect_update_link(auth=Depends(require_auth)):
|
||||
"""Get a link to update Stripe Connect account info (re-onboarding)."""
|
||||
from backend.services.user_service import get_user_service
|
||||
user = get_user_service().get_user(auth.user_email)
|
||||
if not user or not user.stripe_connect_account_id:
|
||||
raise HTTPException(status_code=400, detail="No Stripe Connect account configured")
|
||||
|
||||
service = get_referral_service()
|
||||
url = service.create_connect_update_link(user.stripe_connect_account_id)
|
||||
if not url:
|
||||
raise HTTPException(status_code=500, detail="Failed to create update link")
|
||||
|
||||
return {"url": url}
|
||||
|
||||
|
||||
class VanityRequest(BaseModel):
|
||||
desired_code: str = Field(..., min_length=2, max_length=30, pattern="^[a-z0-9-]+$")
|
||||
|
||||
|
||||
@router.post("/me/vanity-request")
|
||||
async def request_vanity_url(
|
||||
request: VanityRequest,
|
||||
auth=Depends(require_auth),
|
||||
):
|
||||
"""Request a custom vanity referral URL. Sends email to admin for review."""
|
||||
from backend.services.email_service import get_email_service
|
||||
|
||||
service = get_referral_service()
|
||||
link = service.get_or_create_link(auth.user_email)
|
||||
|
||||
# Check if code is already taken
|
||||
existing = service.get_link_by_code(request.desired_code)
|
||||
if existing:
|
||||
raise HTTPException(status_code=409, detail="That code is already taken. Please try another.")
|
||||
|
||||
email_service = get_email_service()
|
||||
email_service.provider.send_email(
|
||||
to_email="andrew@nomadkaraoke.com",
|
||||
subject=f"Vanity referral URL request: {request.desired_code}",
|
||||
html_content=f"""
|
||||
<h2>Vanity Referral URL Request</h2>
|
||||
<p><strong>User:</strong> {auth.user_email}</p>
|
||||
<p><strong>Current code:</strong> {link.code}</p>
|
||||
<p><strong>Desired vanity code:</strong> {request.desired_code}</p>
|
||||
<p>To approve, create the vanity link in the <a href="https://gen.nomadkaraoke.com/admin/referrals">admin dashboard</a>.</p>
|
||||
""",
|
||||
text_content=f"Vanity URL request from {auth.user_email}: {request.desired_code} (current: {link.code})",
|
||||
)
|
||||
|
||||
logger.info("Vanity URL request from %s: %s", auth.user_email, request.desired_code)
|
||||
return {"ok": True, "message": "Request sent"}
|
||||
|
||||
|
||||
@router.post("/me/flyer")
|
||||
async def generate_flyer(
|
||||
request: GenerateFlyerRequest,
|
||||
auth=Depends(require_auth),
|
||||
):
|
||||
"""Generate a personalized referral flyer PDF."""
|
||||
service = get_referral_service()
|
||||
link = service.get_or_create_link(auth.user_email)
|
||||
|
||||
try:
|
||||
flyer_service = get_flyer_service()
|
||||
pdf_bytes = flyer_service.generate_pdf(
|
||||
theme=request.theme,
|
||||
referral_code=link.code,
|
||||
discount_percent=link.discount_percent,
|
||||
qr_data_url=request.qr_data_url,
|
||||
)
|
||||
except FlyerError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
|
||||
return Response(
|
||||
content=pdf_bytes,
|
||||
media_type="application/pdf",
|
||||
headers={
|
||||
"Content-Disposition": 'attachment; filename="nomad-karaoke-referral-flyer.pdf"',
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Admin endpoints
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@router.post("/admin/vanity")
|
||||
async def create_vanity_link(
|
||||
request: CreateReferralLinkRequest,
|
||||
auth=Depends(require_admin),
|
||||
):
|
||||
"""Admin: create a vanity referral link."""
|
||||
if not request.vanity_code:
|
||||
raise HTTPException(status_code=400, detail="vanity_code is required")
|
||||
|
||||
service = get_referral_service()
|
||||
success, link, message = service.create_vanity_link(
|
||||
code=request.vanity_code,
|
||||
owner_email=request.owner_email,
|
||||
display_name=request.display_name,
|
||||
custom_message=request.custom_message,
|
||||
discount_percent=request.discount_percent,
|
||||
kickback_percent=request.kickback_percent,
|
||||
discount_duration_days=request.discount_duration_days,
|
||||
earning_duration_days=request.earning_duration_days,
|
||||
)
|
||||
|
||||
if not success:
|
||||
raise HTTPException(status_code=400, detail=message)
|
||||
|
||||
return {"ok": True, "code": link.code, "message": message}
|
||||
|
||||
|
||||
@router.get("/admin/links")
|
||||
async def list_links(
|
||||
limit: int = Query(50, ge=1, le=200),
|
||||
offset: int = Query(0, ge=0),
|
||||
auth=Depends(require_admin),
|
||||
):
|
||||
"""Admin: list all referral links."""
|
||||
service = get_referral_service()
|
||||
links = service.list_links(limit=limit, offset=offset)
|
||||
return {
|
||||
"links": [link.model_dump(mode="json") for link in links],
|
||||
"count": len(links),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/admin/apply-discount")
|
||||
async def admin_apply_discount(
|
||||
request: Request,
|
||||
auth=Depends(require_admin),
|
||||
):
|
||||
"""Admin: apply a referral discount to any user, even without a referral link."""
|
||||
from backend.services.user_service import get_user_service
|
||||
|
||||
body = await request.json()
|
||||
user_email = body.get("email", "").lower()
|
||||
discount_percent = body.get("discount_percent", 10)
|
||||
duration_days = body.get("duration_days", 30)
|
||||
referral_code = body.get("referral_code", "admin-grant")
|
||||
|
||||
if not user_email:
|
||||
raise HTTPException(status_code=400, detail="Email is required")
|
||||
|
||||
user_service = get_user_service()
|
||||
user = user_service.get_user(user_email)
|
||||
if not user:
|
||||
raise HTTPException(status_code=404, detail="User not found")
|
||||
|
||||
from datetime import datetime, timedelta
|
||||
now = datetime.utcnow()
|
||||
|
||||
user_service.update_user(
|
||||
user_email,
|
||||
referred_by_code=referral_code,
|
||||
referred_at=now,
|
||||
referral_discount_expires_at=now + timedelta(days=duration_days),
|
||||
)
|
||||
|
||||
# Create/ensure the Stripe coupon exists for this discount percent
|
||||
service = get_referral_service()
|
||||
service.get_or_create_stripe_coupon(discount_percent)
|
||||
|
||||
logger.info(f"Admin applied {discount_percent}% discount for {duration_days} days to {user_email}")
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"message": f"Applied {discount_percent}% discount for {duration_days} days",
|
||||
"discount_expires_at": (now + timedelta(days=duration_days)).isoformat(),
|
||||
}
|
||||
|
||||
|
||||
@router.put("/admin/links/{code}")
|
||||
async def update_link(
|
||||
code: str,
|
||||
updates: UpdateReferralLinkRequest,
|
||||
auth=Depends(require_admin),
|
||||
):
|
||||
"""Admin: update any referral link's settings."""
|
||||
service = get_referral_service()
|
||||
|
||||
update_fields = {
|
||||
k: v for k, v in updates.model_dump().items() if v is not None
|
||||
}
|
||||
|
||||
if not update_fields:
|
||||
return {"ok": True, "message": "No changes"}
|
||||
|
||||
success, message = service.update_link(code, **update_fields)
|
||||
if not success:
|
||||
raise HTTPException(status_code=404, detail=message)
|
||||
|
||||
return {"ok": True, "message": message}
|
||||
|
||||
|
||||
@router.post("/admin/links/{code}/flyer")
|
||||
async def admin_generate_flyer(
|
||||
code: str,
|
||||
request: GenerateFlyerRequest,
|
||||
auth=Depends(require_admin),
|
||||
):
|
||||
"""Admin: generate a personalized referral flyer PDF for any code."""
|
||||
service = get_referral_service()
|
||||
link = service.get_link_by_code(code)
|
||||
|
||||
if not link:
|
||||
raise HTTPException(status_code=404, detail=f"Referral link '{code}' not found")
|
||||
|
||||
try:
|
||||
flyer_service = get_flyer_service()
|
||||
pdf_bytes = flyer_service.generate_pdf(
|
||||
theme=request.theme,
|
||||
referral_code=link.code,
|
||||
discount_percent=link.discount_percent,
|
||||
qr_data_url=request.qr_data_url,
|
||||
)
|
||||
except FlyerError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
|
||||
return Response(
|
||||
content=pdf_bytes,
|
||||
media_type="application/pdf",
|
||||
headers={
|
||||
"Content-Disposition": f'attachment; filename="nomad-karaoke-flyer-{link.code}.pdf"',
|
||||
},
|
||||
)
|
||||
+1174
-100
File diff suppressed because it is too large.
Load diff
@@ -12,6 +12,7 @@ from fastapi import APIRouter, Query, Request
|
||||
|
||||
from backend.models.tenant import TenantConfigResponse, TenantPublicConfig
|
||||
from backend.services.tenant_service import get_tenant_service
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -115,6 +116,6 @@ async def get_tenant_asset(tenant_id: str, asset_name: str):
|
||||
if not url:
|
||||
from fastapi import HTTPException
|
||||
|
||||
raise HTTPException(status_code=404, detail="Asset not found")
|
||||
raise HTTPException(status_code=404, detail=t("en", "tenant.assetNotFound"))
|
||||
|
||||
return RedirectResponse(url=url)
|
||||
@@ -9,10 +9,11 @@ before job creation.
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
|
||||
from backend.models.theme import ThemeDetailResponse, ThemesListResponse
|
||||
from backend.services.theme_service import get_theme_service
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -20,7 +21,7 @@ router = APIRouter(prefix="/themes", tags=["themes"])
|
||||
|
||||
|
||||
@router.get("", response_model=ThemesListResponse)
|
||||
async def list_themes() -> ThemesListResponse:
|
||||
async def list_themes(request: Request) -> ThemesListResponse:
|
||||
"""
|
||||
List all available themes.
|
||||
|
||||
@@ -29,6 +30,7 @@ async def list_themes() -> ThemesListResponse:
|
||||
|
||||
This endpoint is public and does not require authentication.
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
themes = theme_service.list_themes()
|
||||
@@ -37,12 +39,12 @@ async def list_themes() -> ThemesListResponse:
|
||||
logger.error(f"Error listing themes: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load themes"
|
||||
detail=t(locale, "themes.themesLoadError")
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}", response_model=ThemeDetailResponse)
|
||||
async def get_theme(theme_id: str) -> ThemeDetailResponse:
|
||||
async def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
|
||||
"""
|
||||
Get detailed information about a specific theme.
|
||||
|
||||
@@ -58,6 +60,7 @@ async def get_theme(theme_id: str) -> ThemeDetailResponse:
|
||||
Raises:
|
||||
404: Theme not found
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
theme = theme_service.get_theme(theme_id)
|
||||
@@ -65,7 +68,7 @@ async def get_theme(theme_id: str) -> ThemeDetailResponse:
|
||||
if theme is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
detail=t(locale, "themes.themeNotFound", theme_id=theme_id)
|
||||
)
|
||||
|
||||
return ThemeDetailResponse(theme=theme)
|
||||
@@ -75,12 +78,12 @@ async def get_theme(theme_id: str) -> ThemeDetailResponse:
|
||||
logger.error(f"Error getting theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load theme"
|
||||
detail=t(locale, "themes.themeLoadError")
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}/preview")
|
||||
async def get_theme_preview(theme_id: str) -> dict:
|
||||
async def get_theme_preview(theme_id: str, request: Request) -> dict:
|
||||
"""
|
||||
Get a signed URL for the theme's preview image.
|
||||
|
||||
@@ -96,20 +99,21 @@ async def get_theme_preview(theme_id: str) -> dict:
|
||||
Raises:
|
||||
404: Theme not found or no preview available
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
|
||||
if not theme_service.theme_exists(theme_id):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
detail=t(locale, "themes.themeNotFound", theme_id=theme_id)
|
||||
)
|
||||
|
||||
theme = theme_service.get_theme(theme_id)
|
||||
if theme is None or theme.preview_url is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"No preview available for theme: {theme_id}"
|
||||
detail=t(locale, "themes.previewNotAvailable", theme_id=theme_id)
|
||||
)
|
||||
|
||||
return {"preview_url": theme.preview_url}
|
||||
@@ -119,12 +123,12 @@ async def get_theme_preview(theme_id: str) -> dict:
|
||||
logger.error(f"Error getting preview for theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load theme preview"
|
||||
detail=t(locale, "themes.previewLoadError")
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}/youtube-description")
|
||||
async def get_theme_youtube_description(theme_id: str) -> dict:
|
||||
async def get_theme_youtube_description(theme_id: str, request: Request) -> dict:
|
||||
"""
|
||||
Get the YouTube description template for a theme.
|
||||
|
||||
@@ -141,13 +145,14 @@ async def get_theme_youtube_description(theme_id: str) -> dict:
|
||||
Raises:
|
||||
404: Theme not found
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
|
||||
if not theme_service.theme_exists(theme_id):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
detail=t(locale, "themes.youtubeDescriptionNotFound", theme_id=theme_id)
|
||||
)
|
||||
|
||||
description = theme_service.get_youtube_description(theme_id)
|
||||
@@ -158,5 +163,5 @@ async def get_theme_youtube_description(theme_id: str) -> dict:
|
||||
logger.error(f"Error getting YouTube description for theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load YouTube description"
|
||||
detail=t(locale, "themes.youtubeDescriptionLoadError")
|
||||
)
|
||||
+763
-477
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,35 @@
|
||||
# Cloud Build config for GPU base image
|
||||
#
|
||||
# Builds the CUDA + PyTorch + models base image used by the audio worker.
|
||||
# This image is large (~10GB) so builds take 15-30 minutes.
|
||||
#
|
||||
# Trigger manually:
|
||||
# gcloud builds submit --config=backend/cloudbuild-gpu-base.yaml --project=nomadkaraoke --timeout=3600s
|
||||
#
|
||||
# The app-layer Dockerfile uses this as its BASE_IMAGE for audio worker deploys.
|
||||
|
||||
steps:
|
||||
- name: 'gcr.io/cloud-builders/docker'
|
||||
args:
|
||||
- 'build'
|
||||
- '-f'
|
||||
- 'backend/Dockerfile.gpu-base'
|
||||
- '-t'
|
||||
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:latest'
|
||||
- '-t'
|
||||
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:$BUILD_ID'
|
||||
- '--build-arg'
|
||||
- 'BUILD_DATE=$_BUILD_DATE'
|
||||
- '--build-arg'
|
||||
- 'BUILD_ID=$BUILD_ID'
|
||||
- '.'
|
||||
|
||||
images:
|
||||
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:latest'
|
||||
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:$BUILD_ID'
|
||||
|
||||
timeout: '3600s'
|
||||
|
||||
options:
|
||||
machineType: 'E2_HIGHCPU_32'
|
||||
diskSizeGb: 100
|
||||
+30
-10
@@ -32,6 +32,10 @@ class Settings(BaseSettings):
|
||||
|
||||
# Authentication
|
||||
admin_tokens: Optional[str] = os.getenv("ADMIN_TOKENS") # Comma-separated list
|
||||
scheduler_service_account: str = os.getenv(
|
||||
"SCHEDULER_SERVICE_ACCOUNT",
|
||||
"karaoke-backend@nomadkaraoke.iam.gserviceaccount.com"
|
||||
)
|
||||
|
||||
# Application
|
||||
environment: str = os.getenv("ENVIRONMENT", "development")
|
||||
@@ -57,6 +61,12 @@ class Settings(BaseSettings):
|
||||
# use uncorrected transcription - human review will fix any issues.
|
||||
agentic_correction_timeout_seconds: int = int(os.getenv("AGENTIC_CORRECTION_TIMEOUT_SECONDS", "180"))
|
||||
|
||||
# AI Credit Evaluation (anti-abuse gating for free credits)
|
||||
# When enabled, welcome and feedback credit grants are evaluated by Gemini
|
||||
# before granting. Fail-open: if evaluation fails, credits are granted anyway.
|
||||
credit_eval_enabled: bool = os.getenv("CREDIT_EVAL_ENABLED", "true").lower() in ("true", "1", "yes")
|
||||
credit_eval_model: str = os.getenv("CREDIT_EVAL_MODEL", "gemini-3.1-pro-preview")
|
||||
|
||||
# Cloud Tasks (for scalable worker coordination)
|
||||
# When enabled, workers are triggered via Cloud Tasks for guaranteed delivery
|
||||
# When disabled (default), workers are triggered via direct HTTP (for development)
|
||||
@@ -83,6 +93,9 @@ class Settings(BaseSettings):
|
||||
# Requires use_gce_encoding to be enabled and the GCE worker to support /encode-preview endpoint.
|
||||
use_gce_preview_encoding: bool = os.getenv("USE_GCE_PREVIEW_ENCODING", "false").lower() in ("true", "1", "yes")
|
||||
|
||||
# Karaoke-Decide catalog API (for song/artist autocomplete)
|
||||
karaoke_decide_api_url: str = os.getenv("KARAOKE_DECIDE_API_URL", "https://decide.nomadkaraoke.com")
|
||||
|
||||
# Storage paths
|
||||
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
|
||||
|
||||
@@ -97,7 +110,13 @@ class Settings(BaseSettings):
|
||||
# This is required for torrent downloads since Cloud Run doesn't support BitTorrent.
|
||||
flacfetch_api_url: Optional[str] = os.getenv("FLACFETCH_API_URL") # e.g., http://10.0.0.5:8080
|
||||
flacfetch_api_key: Optional[str] = os.getenv("FLACFETCH_API_KEY")
|
||||
|
||||
|
||||
# Flacfetch client retry settings
|
||||
# Retries help handle transient network failures when calling the flacfetch service
|
||||
flacfetch_retry_max_attempts: int = int(os.getenv("FLACFETCH_RETRY_MAX_ATTEMPTS", "9")) # Total attempts (1 initial + 8 retries)
|
||||
flacfetch_retry_min_wait: float = float(os.getenv("FLACFETCH_RETRY_MIN_WAIT", "10.0")) # seconds
|
||||
flacfetch_retry_max_wait: float = float(os.getenv("FLACFETCH_RETRY_MAX_WAIT", "60.0")) # seconds
|
||||
|
||||
# Default distribution settings (can be overridden per-request)
|
||||
default_dropbox_path: Optional[str] = os.getenv("DEFAULT_DROPBOX_PATH")
|
||||
default_gdrive_folder_id: Optional[str] = os.getenv("DEFAULT_GDRIVE_FOLDER_ID")
|
||||
@@ -110,16 +129,17 @@ class Settings(BaseSettings):
|
||||
default_enable_youtube_upload: bool = os.getenv("DEFAULT_ENABLE_YOUTUBE_UPLOAD", "false").lower() in ("true", "1", "yes")
|
||||
default_brand_prefix: Optional[str] = os.getenv("DEFAULT_BRAND_PREFIX")
|
||||
|
||||
# Rate Limiting Configuration
|
||||
# Enable/disable rate limiting system-wide (useful for development)
|
||||
enable_rate_limiting: bool = os.getenv("ENABLE_RATE_LIMITING", "true").lower() in ("true", "1", "yes")
|
||||
# Maximum jobs a user can create per day (0 = unlimited)
|
||||
rate_limit_jobs_per_day: int = int(os.getenv("RATE_LIMIT_JOBS_PER_DAY", "5"))
|
||||
# Maximum YouTube uploads system-wide per day (0 = unlimited)
|
||||
rate_limit_youtube_uploads_per_day: int = int(os.getenv("RATE_LIMIT_YOUTUBE_UPLOADS_PER_DAY", "10"))
|
||||
# Maximum beta enrollments from same IP per day (0 = unlimited)
|
||||
rate_limit_beta_ip_per_day: int = int(os.getenv("RATE_LIMIT_BETA_IP_PER_DAY", "1"))
|
||||
# Private (non-published) track distribution settings
|
||||
default_private_dropbox_path: Optional[str] = os.getenv("DEFAULT_PRIVATE_DROPBOX_PATH")
|
||||
default_private_brand_prefix: Optional[str] = os.getenv("DEFAULT_PRIVATE_BRAND_PREFIX", "NOMADNP")
|
||||
|
||||
# YouTube Data API v3 Quota Configuration
|
||||
# Daily quota limit (YouTube grants 10,000 units/day by default)
|
||||
youtube_quota_daily_limit: int = int(os.getenv("YOUTUBE_QUOTA_DAILY_LIMIT", "10000"))
|
||||
# Estimated units consumed per upload (videos.insert=100 + search.list=100 + thumbnails.set=50 + buffer)
|
||||
youtube_quota_upload_cost: int = int(os.getenv("YOUTUBE_QUOTA_UPLOAD_COST", "300"))
|
||||
# Safety margin to reserve (avoid hitting exact limit)
|
||||
youtube_quota_safety_margin: int = int(os.getenv("YOUTUBE_QUOTA_SAFETY_MARGIN", "500"))
|
||||
# E2E test bypass key for rate limiting (set via secret in production)
|
||||
e2e_bypass_key: str = os.getenv("E2E_BYPASS_KEY", "")
|
||||
default_youtube_description: str = os.getenv(
|
||||
|
||||
+59
-31
@@ -2,40 +2,10 @@
|
||||
Custom exceptions for the karaoke generation backend.
|
||||
|
||||
These exceptions are used for structured error handling across the application,
|
||||
particularly for rate limiting and validation errors.
|
||||
particularly for validation errors and credit checks.
|
||||
"""
|
||||
|
||||
|
||||
class RateLimitExceededError(Exception):
|
||||
"""
|
||||
Raised when a rate limit is exceeded.
|
||||
|
||||
Includes information about when the limit will reset for client retry logic.
|
||||
|
||||
Attributes:
|
||||
message: Human-readable error message
|
||||
limit_type: Type of limit exceeded (e.g., "jobs_per_day", "youtube_uploads")
|
||||
remaining_seconds: Seconds until the limit resets (for Retry-After header)
|
||||
current_count: Current usage count
|
||||
limit_value: The limit that was exceeded
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
message: str,
|
||||
limit_type: str = "unknown",
|
||||
remaining_seconds: int = 0,
|
||||
current_count: int = 0,
|
||||
limit_value: int = 0
|
||||
):
|
||||
self.message = message
|
||||
self.limit_type = limit_type
|
||||
self.remaining_seconds = remaining_seconds
|
||||
self.current_count = current_count
|
||||
self.limit_value = limit_value
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
class EmailValidationError(Exception):
|
||||
"""
|
||||
Raised when email validation fails.
|
||||
@@ -64,3 +34,61 @@ class IPBlockedError(Exception):
|
||||
self.message = message
|
||||
self.ip_address = ip_address
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
class InsufficientCreditsError(Exception):
|
||||
"""
|
||||
Raised when a user attempts to create a job without sufficient credits.
|
||||
|
||||
Attributes:
|
||||
message: Human-readable error message
|
||||
credits_available: Number of credits the user currently has
|
||||
credits_required: Number of credits required (always 1 for now)
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
message: str,
|
||||
credits_available: int = 0,
|
||||
credits_required: int = 1
|
||||
):
|
||||
self.message = message
|
||||
self.credits_available = credits_available
|
||||
self.credits_required = credits_required
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
class InvalidStateTransitionError(Exception):
|
||||
"""
|
||||
Raised when an invalid job state transition is attempted.
|
||||
|
||||
This exception is raised by JobManager.transition_to_state() when the
|
||||
requested transition is not allowed by the state machine defined in
|
||||
STATE_TRANSITIONS.
|
||||
|
||||
Example: Trying to transition from PENDING to GENERATING_SCREENS is invalid
|
||||
because PENDING can only transition to DOWNLOADING, SEARCHING_AUDIO, FAILED,
|
||||
or CANCELLED.
|
||||
|
||||
Attributes:
|
||||
message: Human-readable error message
|
||||
job_id: The job ID that failed to transition
|
||||
from_status: Current status of the job
|
||||
to_status: Attempted target status
|
||||
valid_transitions: List of valid transitions from current status
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
message: str,
|
||||
job_id: str = "",
|
||||
from_status: str = "",
|
||||
to_status: str = "",
|
||||
valid_transitions: list = None
|
||||
):
|
||||
self.message = message
|
||||
self.job_id = job_id
|
||||
self.from_status = from_status
|
||||
self.to_status = to_status
|
||||
self.valid_transitions = valid_transitions or []
|
||||
super().__init__(message)
|
||||
@@ -0,0 +1,99 @@
|
||||
"""
|
||||
Lightweight i18n utility for backend translations.
|
||||
|
||||
Usage:
|
||||
from backend.i18n import t, get_locale_from_request, SUPPORTED_LOCALES
|
||||
|
||||
# In a route handler:
|
||||
locale = get_locale_from_request(request)
|
||||
message = t(locale, "auth.userNotFound")
|
||||
message = t(locale, "jobs.notFound", job_id="abc123")
|
||||
"""
|
||||
|
||||
import json
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from starlette.requests import Request
|
||||
|
||||
SUPPORTED_LOCALES = ("en", "es", "de")
|
||||
DEFAULT_LOCALE = "en"
|
||||
TRANSLATIONS_DIR = Path(__file__).parent / "translations"
|
||||
|
||||
|
||||
@lru_cache(maxsize=len(SUPPORTED_LOCALES))
|
||||
def _load_translations(locale: str) -> dict[str, Any]:
|
||||
"""Load and cache translation file for a locale."""
|
||||
path = TRANSLATIONS_DIR / f"{locale}.json"
|
||||
if not path.exists():
|
||||
if locale != DEFAULT_LOCALE:
|
||||
return _load_translations(DEFAULT_LOCALE)
|
||||
return {}
|
||||
with open(path, encoding="utf-8") as f:
|
||||
result: dict[str, Any] = json.load(f)
|
||||
return result
|
||||
|
||||
|
||||
def t(locale: str, key: str, **kwargs: Any) -> str:
|
||||
"""Get a translated string by dotted key path.
|
||||
|
||||
Args:
|
||||
locale: Language code (en, es, de)
|
||||
key: Dotted path like "auth.userNotFound" or "emails.magicLink.subject"
|
||||
**kwargs: Interpolation variables (e.g., job_id="abc123")
|
||||
|
||||
Returns:
|
||||
Translated string with variables interpolated, or the key itself if not found.
|
||||
"""
|
||||
if locale not in SUPPORTED_LOCALES:
|
||||
locale = DEFAULT_LOCALE
|
||||
|
||||
translations = _load_translations(locale)
|
||||
|
||||
# Walk the dotted path
|
||||
value: Any = translations
|
||||
for part in key.split("."):
|
||||
if isinstance(value, dict) and part in value:
|
||||
value = value[part]
|
||||
else:
|
||||
# Key not found — fall back to English
|
||||
if locale != DEFAULT_LOCALE:
|
||||
return t(DEFAULT_LOCALE, key, **kwargs)
|
||||
return key # Return the key itself as last resort
|
||||
|
||||
if not isinstance(value, str):
|
||||
return key
|
||||
|
||||
result: str = value
|
||||
# Interpolate variables
|
||||
if kwargs:
|
||||
try:
|
||||
result = result.format(**kwargs)
|
||||
except KeyError:
|
||||
pass # Missing variable — return template as-is
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def get_locale_from_request(request: Request) -> str:
|
||||
"""Extract locale from Accept-Language header.
|
||||
|
||||
Returns the first supported locale found, or DEFAULT_LOCALE.
|
||||
"""
|
||||
accept_lang = request.headers.get("accept-language", "")
|
||||
|
||||
# Simple parsing — Accept-Language can be "de", "de-DE", "en-US,en;q=0.9,de;q=0.8"
|
||||
for part in accept_lang.split(","):
|
||||
lang = part.split(";")[0].strip().split("-")[0].lower()
|
||||
if lang in SUPPORTED_LOCALES:
|
||||
return lang
|
||||
|
||||
return DEFAULT_LOCALE
|
||||
|
||||
|
||||
def get_locale_prefix(locale: str) -> str:
|
||||
"""Get URL locale prefix for frontend URLs (e.g., '/en', '/es')."""
|
||||
if locale not in SUPPORTED_LOCALES:
|
||||
locale = DEFAULT_LOCALE
|
||||
return f"/{locale}"
|
||||
+19
-16
@@ -7,7 +7,7 @@ from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from backend.config import settings
|
||||
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, rate_limits, push
|
||||
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, rate_limits, push, catalog, encoding_worker
|
||||
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
|
||||
from backend.services.structured_logging import setup_structured_logging
|
||||
from backend.services.spacy_preloader import preload_spacy_model
|
||||
@@ -136,8 +136,8 @@ if tracing_enabled:
|
||||
# Configure CORS
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"], # Configure this properly for production
|
||||
allow_credentials=True,
|
||||
allow_origins=["*"],
|
||||
allow_credentials=False,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
@@ -153,37 +153,40 @@ app.include_router(health.router, prefix="/api")
|
||||
app.include_router(jobs.router, prefix="/api")
|
||||
app.include_router(file_upload.router, prefix="/api") # File upload endpoint
|
||||
app.include_router(internal.router, prefix="/api") # Internal worker endpoints
|
||||
app.include_router(encoding_worker.router, prefix="/api") # Encoding worker lifecycle
|
||||
app.include_router(review.router, prefix="/api") # Review UI compatibility endpoints
|
||||
app.include_router(auth.router, prefix="/api") # OAuth credential management
|
||||
app.include_router(audio_search.router, prefix="/api") # Audio search (artist+title mode)
|
||||
app.include_router(themes.router, prefix="/api") # Theme selection for styles
|
||||
app.include_router(users.router, prefix="/api") # User auth, credits, and Stripe webhooks
|
||||
from backend.api.routes import referrals
|
||||
app.include_router(referrals.router, prefix="/api") # Referral links and payouts
|
||||
app.include_router(admin.router, prefix="/api") # Admin dashboard and management
|
||||
app.include_router(rate_limits.router, prefix="/api") # Rate limits admin management
|
||||
app.include_router(push.router, prefix="/api") # Push notification subscription management
|
||||
app.include_router(catalog.router, prefix="/api") # Catalog proxy for song/artist autocomplete
|
||||
app.include_router(tenant.router) # Tenant/white-label configuration (no /api prefix, router has it)
|
||||
|
||||
|
||||
# Exception handler for rate limiting
|
||||
# Exception handlers
|
||||
from fastapi import Request
|
||||
from fastapi.responses import JSONResponse
|
||||
from backend.exceptions import RateLimitExceededError
|
||||
from backend.exceptions import InsufficientCreditsError
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
|
||||
|
||||
@app.exception_handler(RateLimitExceededError)
|
||||
async def rate_limit_exception_handler(request: Request, exc: RateLimitExceededError):
|
||||
"""Handle rate limit exceeded errors with 429 status."""
|
||||
@app.exception_handler(InsufficientCreditsError)
|
||||
async def insufficient_credits_exception_handler(request: Request, exc: InsufficientCreditsError):
|
||||
"""Handle insufficient credits errors with 402 Payment Required status."""
|
||||
locale = get_locale_from_request(request)
|
||||
return JSONResponse(
|
||||
status_code=429,
|
||||
status_code=402,
|
||||
content={
|
||||
"detail": exc.message,
|
||||
"limit_type": exc.limit_type,
|
||||
"current_count": exc.current_count,
|
||||
"limit_value": exc.limit_value,
|
||||
"detail": t(locale, "errors.insufficientCredits"),
|
||||
"credits_available": exc.credits_available,
|
||||
"credits_required": exc.credits_required,
|
||||
"buy_url": t(locale, "errors.insufficientCreditsBuyUrl"),
|
||||
},
|
||||
headers={
|
||||
"Retry-After": str(exc.remaining_seconds),
|
||||
} if exc.remaining_seconds > 0 else None,
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
"""
|
||||
Audio edit session model for persisting input audio editing progress.
|
||||
|
||||
Stored as a Firestore subcollection: jobs/{job_id}/audio_edit_sessions/{session_id}
|
||||
Full edit data is stored in GCS to avoid Firestore's 1MB document limit.
|
||||
"""
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional, Dict, Any, List
|
||||
import uuid
|
||||
|
||||
|
||||
@dataclass
|
||||
class AudioEditSessionSummary:
|
||||
"""Compact summary of operations in an audio edit session, for list views."""
|
||||
total_operations: int = 0
|
||||
operations_breakdown: Dict[str, int] = field(default_factory=dict)
|
||||
duration_change_seconds: float = 0.0
|
||||
net_duration_seconds: float = 0.0
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"total_operations": self.total_operations,
|
||||
"operations_breakdown": self.operations_breakdown,
|
||||
"duration_change_seconds": self.duration_change_seconds,
|
||||
"net_duration_seconds": self.net_duration_seconds,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, Any]) -> "AudioEditSessionSummary":
|
||||
return cls(
|
||||
total_operations=data.get("total_operations", 0),
|
||||
operations_breakdown=data.get("operations_breakdown", {}),
|
||||
duration_change_seconds=data.get("duration_change_seconds", 0.0),
|
||||
net_duration_seconds=data.get("net_duration_seconds", 0.0),
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class AudioEditSession:
|
||||
"""
|
||||
A saved snapshot of an audio editing session.
|
||||
|
||||
Stored at: jobs/{job_id}/audio_edit_sessions/{session_id}
|
||||
Edit data stored at: jobs/{job_id}/audio_edit_sessions/{session_id}.json (GCS)
|
||||
"""
|
||||
session_id: str = field(default_factory=lambda: str(uuid.uuid4()))
|
||||
job_id: str = ""
|
||||
user_email: str = ""
|
||||
created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
|
||||
updated_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
|
||||
edit_count: int = 0
|
||||
trigger: str = "auto" # "auto" | "manual" | "submit"
|
||||
audio_duration_seconds: Optional[float] = None
|
||||
original_duration_seconds: Optional[float] = None
|
||||
artist: Optional[str] = None
|
||||
title: Optional[str] = None
|
||||
summary: AudioEditSessionSummary = field(default_factory=AudioEditSessionSummary)
|
||||
edit_data_gcs_path: str = ""
|
||||
|
||||
# Hash of the edit data JSON, used to skip duplicate saves
|
||||
data_hash: str = ""
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"session_id": self.session_id,
|
||||
"job_id": self.job_id,
|
||||
"user_email": self.user_email,
|
||||
"created_at": self.created_at,
|
||||
"updated_at": self.updated_at,
|
||||
"edit_count": self.edit_count,
|
||||
"trigger": self.trigger,
|
||||
"audio_duration_seconds": self.audio_duration_seconds,
|
||||
"original_duration_seconds": self.original_duration_seconds,
|
||||
"artist": self.artist,
|
||||
"title": self.title,
|
||||
"summary": self.summary.to_dict(),
|
||||
"edit_data_gcs_path": self.edit_data_gcs_path,
|
||||
"data_hash": self.data_hash,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, Any]) -> "AudioEditSession":
|
||||
created_at = data.get("created_at")
|
||||
if isinstance(created_at, str):
|
||||
created_at = datetime.fromisoformat(created_at.rstrip("Z")).replace(tzinfo=timezone.utc)
|
||||
|
||||
updated_at = data.get("updated_at")
|
||||
if isinstance(updated_at, str):
|
||||
updated_at = datetime.fromisoformat(updated_at.rstrip("Z")).replace(tzinfo=timezone.utc)
|
||||
|
||||
summary_data = data.get("summary", {})
|
||||
summary = AudioEditSessionSummary.from_dict(summary_data) if isinstance(summary_data, dict) else AudioEditSessionSummary()
|
||||
|
||||
return cls(
|
||||
session_id=data.get("session_id", str(uuid.uuid4())),
|
||||
job_id=data.get("job_id", ""),
|
||||
user_email=data.get("user_email", ""),
|
||||
created_at=created_at or datetime.now(timezone.utc),
|
||||
updated_at=updated_at or datetime.now(timezone.utc),
|
||||
edit_count=data.get("edit_count", 0),
|
||||
trigger=data.get("trigger", "auto"),
|
||||
audio_duration_seconds=data.get("audio_duration_seconds"),
|
||||
original_duration_seconds=data.get("original_duration_seconds"),
|
||||
artist=data.get("artist"),
|
||||
title=data.get("title"),
|
||||
summary=summary,
|
||||
edit_data_gcs_path=data.get("edit_data_gcs_path", ""),
|
||||
data_hash=data.get("data_hash", ""),
|
||||
)
|
||||
+89
-11
@@ -16,8 +16,9 @@ class JobStatus(str, Enum):
|
||||
"""
|
||||
Job status enumeration - Complete state machine.
|
||||
|
||||
The workflow has 7 main stages with 2 human interaction points:
|
||||
The workflow has 8 main stages with up to 3 human interaction points:
|
||||
1. Input & Setup (may include audio source selection)
|
||||
1b. Optional Audio Editing (BLOCKING) - trim/cut/mute/join input audio
|
||||
2. Parallel Processing (audio + lyrics)
|
||||
3. Title/End Screen Generation + Backing Vocals Analysis
|
||||
4. Countdown Padding Synchronization
|
||||
@@ -39,6 +40,11 @@ class JobStatus(str, Enum):
|
||||
|
||||
DOWNLOADING = "downloading" # Downloading from URL or processing upload
|
||||
|
||||
# Optional: Audio editing (before processing begins)
|
||||
AWAITING_AUDIO_EDIT = "awaiting_audio_edit" # ⚠️ WAITING FOR USER - edit input audio
|
||||
IN_AUDIO_EDIT = "in_audio_edit" # User is actively editing audio
|
||||
AUDIO_EDIT_COMPLETE = "audio_edit_complete" # User submitted edited audio
|
||||
|
||||
# Stage 2a: Audio separation (parallel track 1)
|
||||
SEPARATING_STAGE1 = "separating_stage1" # Clean instrumental separation (Modal API)
|
||||
SEPARATING_STAGE2 = "separating_stage2" # Backing vocals separation (Modal API)
|
||||
@@ -93,8 +99,9 @@ class JobStatus(str, Enum):
|
||||
|
||||
# Valid state transitions
|
||||
STATE_TRANSITIONS = {
|
||||
# PENDING can go to DOWNLOADING (file upload) or SEARCHING_AUDIO (artist+title search)
|
||||
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
# PENDING can go to DOWNLOADING (file upload), SEARCHING_AUDIO (artist+title search),
|
||||
# or DOWNLOADING_AUDIO (create-from-search where audio is pre-selected)
|
||||
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
|
||||
# Audio search flow (for artist+title search mode)
|
||||
JobStatus.SEARCHING_AUDIO: [JobStatus.AWAITING_AUDIO_SELECTION, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED],
|
||||
@@ -102,7 +109,13 @@ STATE_TRANSITIONS = {
|
||||
JobStatus.DOWNLOADING_AUDIO: [JobStatus.DOWNLOADING, JobStatus.FAILED],
|
||||
|
||||
# DOWNLOADING allows parallel processing (audio + lyrics) and then screens when both complete
|
||||
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
# Can also enter optional audio edit phase if requires_audio_edit flag is set
|
||||
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.AWAITING_AUDIO_EDIT, JobStatus.FAILED],
|
||||
|
||||
# Optional audio edit phase (trim, cut, mute, join input audio before processing)
|
||||
JobStatus.AWAITING_AUDIO_EDIT: [JobStatus.IN_AUDIO_EDIT, JobStatus.AUDIO_EDIT_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.IN_AUDIO_EDIT: [JobStatus.AWAITING_AUDIO_EDIT, JobStatus.AUDIO_EDIT_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.AUDIO_EDIT_COMPLETE: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
|
||||
# Audio separation flow
|
||||
JobStatus.SEPARATING_STAGE1: [JobStatus.SEPARATING_STAGE2, JobStatus.FAILED],
|
||||
@@ -141,13 +154,14 @@ STATE_TRANSITIONS = {
|
||||
JobStatus.UPLOADING: [JobStatus.NOTIFYING, JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.NOTIFYING: [JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
|
||||
# Terminal states - COMPLETE, PREP_COMPLETE have no transitions
|
||||
# Terminal states - COMPLETE allows edit (reopen review) and visibility change
|
||||
# FAILED and CANCELLED allow retry transitions to resume from checkpoints
|
||||
# PREP_COMPLETE allows continuation from combined review
|
||||
JobStatus.COMPLETE: [],
|
||||
JobStatus.COMPLETE: [JobStatus.AWAITING_REVIEW, JobStatus.LYRICS_COMPLETE], # Edit track or visibility change
|
||||
JobStatus.PREP_COMPLETE: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED], # Continue from combined review
|
||||
JobStatus.FAILED: [
|
||||
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
|
||||
JobStatus.DOWNLOADING_AUDIO, # Retry audio download (audio search flow)
|
||||
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
|
||||
JobStatus.REVIEW_COMPLETE, # Retry from render stage
|
||||
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
|
||||
@@ -155,6 +169,7 @@ STATE_TRANSITIONS = {
|
||||
],
|
||||
JobStatus.CANCELLED: [
|
||||
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
|
||||
JobStatus.DOWNLOADING_AUDIO, # Retry audio download (audio search flow)
|
||||
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
|
||||
JobStatus.REVIEW_COMPLETE, # Retry from render stage
|
||||
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
|
||||
@@ -176,6 +191,7 @@ class TimelineEvent(BaseModel):
|
||||
timestamp: str
|
||||
progress: Optional[int] = None
|
||||
message: Optional[str] = None
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
|
||||
|
||||
class LogEntry(BaseModel):
|
||||
@@ -205,7 +221,14 @@ class Job(BaseModel):
|
||||
title: Optional[str] = None
|
||||
filename: Optional[str] = None # Original uploaded filename
|
||||
input_media_gcs_path: Optional[str] = None # GCS path to uploaded file
|
||||
|
||||
|
||||
# Audio source tracking (for retry capability)
|
||||
audio_source_type: Optional[str] = None # Source type: 'audio_search', 'youtube_url', 'file_upload'
|
||||
source_name: Optional[str] = None # Provider name (RED, OPS, YouTube, Spotify) - for audio_search
|
||||
source_id: Optional[str] = None # Source-specific ID (torrent ID, video ID, track ID) - for audio_search
|
||||
target_file: Optional[str] = None # For torrents, specific file to extract - for audio_search
|
||||
download_url: Optional[str] = None # Direct download URL if available - for audio_search
|
||||
|
||||
# User preferences
|
||||
enable_cdg: bool = False # Generate CDG+MP3 package (requires style config)
|
||||
enable_txt: bool = False # Generate TXT+MP3 package (requires style config)
|
||||
@@ -215,8 +238,8 @@ class Job(BaseModel):
|
||||
user_email: Optional[str] = None # Email for notifications
|
||||
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
|
||||
|
||||
# Multi-tenant support (None = default Nomad Karaoke)
|
||||
tenant_id: Optional[str] = None # Tenant ID for white-label portal scoping
|
||||
# Multi-tenant support ("" = consumer portal, "vocalstar"/etc = tenant portal)
|
||||
tenant_id: str = "" # Tenant ID for white-label portal scoping
|
||||
|
||||
# Theme configuration (pre-made themes from GCS)
|
||||
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
|
||||
@@ -292,10 +315,25 @@ class Job(BaseModel):
|
||||
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
|
||||
customer_notes: Optional[str] = None # Notes provided by customer with their order
|
||||
|
||||
# Private (non-published) track mode
|
||||
is_private: bool = False # Private tracks: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive
|
||||
|
||||
# Creation-time decision params (for observability — what did the user actually choose?)
|
||||
creation_params: Dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
# Anti-abuse: IP address at job creation time (for cross-account correlation)
|
||||
creation_ip: Optional[str] = None
|
||||
|
||||
# Output deletion tracking (for admin cleanup without deleting job)
|
||||
outputs_deleted_at: Optional[datetime] = None # Timestamp when outputs were deleted by admin
|
||||
outputs_deleted_by: Optional[str] = None # Admin email who deleted outputs
|
||||
|
||||
# Credit refund tracking (prevents double-refund on cancel→delete or fail→delete)
|
||||
credit_refunded: bool = False
|
||||
|
||||
# Edit tracking (how many times a completed track has been re-edited)
|
||||
edit_count: int = 0
|
||||
|
||||
# Processing state
|
||||
track_output_dir: Optional[str] = None # Local output directory (temp)
|
||||
audio_hash: Optional[str] = None # Hash for deduplication
|
||||
@@ -386,6 +424,43 @@ class Job(BaseModel):
|
||||
}
|
||||
"""
|
||||
|
||||
# Processing metadata (immutable provenance data written by workers during processing)
|
||||
processing_metadata: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
Immutable processing provenance data. Written once by each worker during processing.
|
||||
Never modified after initial write. Used for analytics, debugging, and data export.
|
||||
|
||||
Structure:
|
||||
{
|
||||
"transcription": {
|
||||
"provider": "audioshake",
|
||||
"audioshake_task_id": "...",
|
||||
"audioshake_asset_id": "...",
|
||||
"language_detected": "en",
|
||||
"word_count": 358,
|
||||
"segment_count": 42,
|
||||
},
|
||||
"correction": {
|
||||
"handlers_applied": [...],
|
||||
"corrections_made": 12,
|
||||
"correction_ratio": 0.034,
|
||||
"reference_sources_found": ["genius", "spotify"],
|
||||
"agentic_routing": "rule-based",
|
||||
"duration_seconds": 33.2,
|
||||
},
|
||||
"separation": {
|
||||
"provider": "modal",
|
||||
"clean_model": "...",
|
||||
"duration_seconds": 420.5,
|
||||
},
|
||||
"timing": {
|
||||
"lyrics_worker_seconds": 125.3,
|
||||
"audio_worker_seconds": 420.5,
|
||||
...
|
||||
},
|
||||
}
|
||||
"""
|
||||
|
||||
# Request metadata (captured at job creation for tracking and filtering)
|
||||
request_metadata: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
@@ -499,6 +574,9 @@ class JobCreate(BaseModel):
|
||||
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
|
||||
customer_notes: Optional[str] = None # Notes provided by customer with their order
|
||||
|
||||
# Private (non-published) track mode
|
||||
is_private: bool = False # Private tracks: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive
|
||||
|
||||
# Request metadata (set by API endpoint from request headers)
|
||||
request_metadata: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
@@ -511,8 +589,8 @@ class JobCreate(BaseModel):
|
||||
- custom_headers: All X-* headers
|
||||
"""
|
||||
|
||||
# Tenant scoping for white-label portals
|
||||
tenant_id: Optional[str] = None # Tenant ID for job scoping
|
||||
# Tenant scoping for white-label portals ("" = consumer, "vocalstar"/etc = tenant)
|
||||
tenant_id: str = "" # Tenant ID for job scoping
|
||||
|
||||
@validator('url')
|
||||
def validate_url(cls, v):
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
"""
|
||||
Referral system models.
|
||||
|
||||
Supports:
|
||||
- Referral link creation and management
|
||||
- Earnings tracking from referred purchases
|
||||
- Payout processing via Stripe Connect
|
||||
- API request/response shapes for referral endpoints
|
||||
"""
|
||||
from datetime import datetime
|
||||
from typing import Optional, List
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Firestore Document Models
|
||||
# ============================================================================
|
||||
|
||||
class ReferralLinkStats(BaseModel):
|
||||
"""Aggregated statistics for a referral link."""
|
||||
clicks: int = 0
|
||||
signups: int = 0
|
||||
purchases: int = 0
|
||||
total_earned_cents: int = 0
|
||||
|
||||
|
||||
class ReferralLink(BaseModel):
|
||||
"""
|
||||
Referral link stored in Firestore.
|
||||
|
||||
Each user can have one referral link. Vanity links use a custom code
|
||||
(e.g., "karaoke-king") while standard links use a generated code.
|
||||
"""
|
||||
code: str # Unique referral code (URL slug)
|
||||
owner_email: str # Email of the referrer
|
||||
display_name: Optional[str] = None # Public name shown on interstitial
|
||||
custom_message: Optional[str] = None # Optional message on interstitial
|
||||
|
||||
# Reward configuration
|
||||
discount_percent: int = 10 # Discount for referred user
|
||||
kickback_percent: int = 20 # Earnings for referrer
|
||||
discount_duration_days: int = 30 # How long the discount lasts
|
||||
earning_duration_days: int = 365 # How long referrer earns from this link
|
||||
|
||||
# Stripe
|
||||
stripe_coupon_id: Optional[str] = None # Stripe coupon for this link
|
||||
|
||||
# Flags
|
||||
is_vanity: bool = False # Custom vanity code vs generated
|
||||
enabled: bool = True # Can be disabled without deleting
|
||||
|
||||
# Timestamps
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
updated_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
|
||||
# Aggregated stats
|
||||
stats: ReferralLinkStats = Field(default_factory=ReferralLinkStats)
|
||||
|
||||
|
||||
class ReferralEarning(BaseModel):
|
||||
"""
|
||||
Record of a single earning from a referred purchase.
|
||||
|
||||
Created when a referred user completes a Stripe checkout.
|
||||
"""
|
||||
id: str
|
||||
referrer_email: str
|
||||
referred_email: str
|
||||
referral_code: str
|
||||
stripe_session_id: str
|
||||
purchase_amount_cents: int
|
||||
earning_amount_cents: int
|
||||
status: str = "pending" # pending, paid, cancelled
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
paid_at: Optional[datetime] = None
|
||||
payout_id: Optional[str] = None
|
||||
|
||||
|
||||
class ReferralPayout(BaseModel):
|
||||
"""
|
||||
Record of a payout to a referrer via Stripe Connect.
|
||||
|
||||
Groups multiple earnings into a single transfer.
|
||||
"""
|
||||
id: str
|
||||
referrer_email: str
|
||||
stripe_transfer_id: str
|
||||
amount_cents: int
|
||||
earnings_included: List[str] # List of ReferralEarning IDs
|
||||
status: str = "processing" # processing, completed, failed
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# API Request Models
|
||||
# ============================================================================
|
||||
|
||||
class CreateReferralLinkRequest(BaseModel):
|
||||
"""Admin request to create a vanity referral link."""
|
||||
vanity_code: Optional[str] = None
|
||||
owner_email: str
|
||||
display_name: Optional[str] = None
|
||||
custom_message: Optional[str] = None
|
||||
discount_percent: int = 10
|
||||
kickback_percent: int = 20
|
||||
discount_duration_days: int = 30
|
||||
earning_duration_days: int = 365
|
||||
|
||||
|
||||
class UpdateReferralLinkRequest(BaseModel):
|
||||
"""Request to update a referral link (user: name/message only, admin: all fields)."""
|
||||
display_name: Optional[str] = None
|
||||
custom_message: Optional[str] = None
|
||||
discount_percent: Optional[int] = None
|
||||
kickback_percent: Optional[int] = None
|
||||
discount_duration_days: Optional[int] = None
|
||||
earning_duration_days: Optional[int] = None
|
||||
enabled: Optional[bool] = None
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# API Response Models
|
||||
# ============================================================================
|
||||
|
||||
class ReferralLinkResponse(BaseModel):
|
||||
"""Public referral link information."""
|
||||
code: str
|
||||
owner_email: str
|
||||
display_name: Optional[str] = None
|
||||
custom_message: Optional[str] = None
|
||||
discount_percent: int
|
||||
kickback_percent: int
|
||||
is_vanity: bool
|
||||
enabled: bool
|
||||
stats: ReferralLinkStats
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class ReferralEarningResponse(BaseModel):
|
||||
"""Public earning information (for referrer dashboard)."""
|
||||
id: str
|
||||
referred_email: str
|
||||
referral_code: str
|
||||
purchase_amount_cents: int
|
||||
earning_amount_cents: int
|
||||
status: str
|
||||
created_at: datetime
|
||||
paid_at: Optional[datetime] = None
|
||||
|
||||
|
||||
class ReferralPayoutResponse(BaseModel):
|
||||
"""Public payout information (for referrer dashboard)."""
|
||||
id: str
|
||||
amount_cents: int
|
||||
earnings_count: int
|
||||
status: str
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class ReferralDashboardResponse(BaseModel):
|
||||
"""Full referral dashboard data for the referrer."""
|
||||
referral_link: ReferralLinkResponse
|
||||
recent_earnings: List[ReferralEarningResponse]
|
||||
recent_payouts: List[ReferralPayoutResponse]
|
||||
total_earned_cents: int
|
||||
total_paid_cents: int
|
||||
pending_balance_cents: int
|
||||
stripe_connect_onboarded: bool
|
||||
|
||||
|
||||
class ReferralInterstitialResponse(BaseModel):
|
||||
"""Data for the referral landing/interstitial page."""
|
||||
valid: bool = True
|
||||
referral_code: str = ""
|
||||
referrer_display_name: Optional[str] = None
|
||||
discount_percent: int = 0
|
||||
discount_duration_days: int = 0
|
||||
custom_message: Optional[str] = None
|
||||
@@ -10,7 +10,7 @@ class URLSubmissionRequest(BaseModel):
|
||||
url: HttpUrl
|
||||
artist: Optional[str] = None # Auto-detected if not provided
|
||||
title: Optional[str] = None # Auto-detected if not provided
|
||||
|
||||
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
@@ -18,6 +18,7 @@ class URLSubmissionRequest(BaseModel):
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
is_private: Optional[bool] = None # Private (non-published) track mode
|
||||
|
||||
|
||||
class UploadSubmissionRequest(BaseModel):
|
||||
@@ -141,3 +142,24 @@ class RetryJobRequest(BaseModel):
|
||||
"""Request to retry a failed job."""
|
||||
from_stage: Optional[str] = None # Optional: restart from specific stage
|
||||
|
||||
|
||||
class ChangeVisibilityRequest(BaseModel):
|
||||
"""Request to change job visibility (public/private)."""
|
||||
target_visibility: str # "public" or "private"
|
||||
|
||||
@validator('target_visibility')
|
||||
def validate_target(cls, v):
|
||||
if v not in ('public', 'private'):
|
||||
raise ValueError("target_visibility must be 'public' or 'private'")
|
||||
return v
|
||||
|
||||
|
||||
class ChangeVisibilityResponse(BaseModel):
|
||||
"""Response from change visibility endpoint."""
|
||||
status: str # "success", "processing", "error"
|
||||
job_id: str
|
||||
message: str
|
||||
previous_visibility: str # "public" or "private"
|
||||
new_visibility: str # "public" or "private"
|
||||
reprocessing_required: bool
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
"""
|
||||
Review session model for persisting lyrics review progress.
|
||||
|
||||
Stored as a Firestore subcollection: jobs/{job_id}/review_sessions/{session_id}
|
||||
Full correction data is stored in GCS to avoid Firestore's 1MB document limit.
|
||||
"""
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional, Dict, Any, List
|
||||
import uuid
|
||||
|
||||
|
||||
@dataclass
|
||||
class ReviewSessionSummary:
|
||||
"""Compact summary of edits in a review session, for list views."""
|
||||
total_segments: int = 0
|
||||
total_words: int = 0
|
||||
corrections_made: int = 0
|
||||
changed_words: List[Dict[str, Any]] = field(default_factory=list)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"total_segments": self.total_segments,
|
||||
"total_words": self.total_words,
|
||||
"corrections_made": self.corrections_made,
|
||||
"changed_words": self.changed_words,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, Any]) -> "ReviewSessionSummary":
|
||||
return cls(
|
||||
total_segments=data.get("total_segments", 0),
|
||||
total_words=data.get("total_words", 0),
|
||||
corrections_made=data.get("corrections_made", 0),
|
||||
changed_words=data.get("changed_words", []),
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ReviewSession:
|
||||
"""
|
||||
A saved snapshot of a lyrics review editing session.
|
||||
|
||||
Stored at: jobs/{job_id}/review_sessions/{session_id}
|
||||
Correction data stored at: jobs/{job_id}/review_sessions/{session_id}.json (GCS)
|
||||
"""
|
||||
session_id: str = field(default_factory=lambda: str(uuid.uuid4()))
|
||||
job_id: str = ""
|
||||
user_email: str = ""
|
||||
created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
|
||||
updated_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
|
||||
edit_count: int = 0
|
||||
trigger: str = "auto" # "auto" | "preview" | "manual"
|
||||
audio_duration_seconds: Optional[float] = None
|
||||
artist: Optional[str] = None
|
||||
title: Optional[str] = None
|
||||
summary: ReviewSessionSummary = field(default_factory=ReviewSessionSummary)
|
||||
correction_data_gcs_path: str = ""
|
||||
|
||||
# Hash of the correction data JSON, used to skip duplicate saves
|
||||
data_hash: str = ""
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"session_id": self.session_id,
|
||||
"job_id": self.job_id,
|
||||
"user_email": self.user_email,
|
||||
"created_at": self.created_at,
|
||||
"updated_at": self.updated_at,
|
||||
"edit_count": self.edit_count,
|
||||
"trigger": self.trigger,
|
||||
"audio_duration_seconds": self.audio_duration_seconds,
|
||||
"artist": self.artist,
|
||||
"title": self.title,
|
||||
"summary": self.summary.to_dict(),
|
||||
"correction_data_gcs_path": self.correction_data_gcs_path,
|
||||
"data_hash": self.data_hash,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, Any]) -> "ReviewSession":
|
||||
created_at = data.get("created_at")
|
||||
if isinstance(created_at, str):
|
||||
created_at = datetime.fromisoformat(created_at.rstrip("Z")).replace(tzinfo=timezone.utc)
|
||||
|
||||
updated_at = data.get("updated_at")
|
||||
if isinstance(updated_at, str):
|
||||
updated_at = datetime.fromisoformat(updated_at.rstrip("Z")).replace(tzinfo=timezone.utc)
|
||||
|
||||
summary_data = data.get("summary", {})
|
||||
summary = ReviewSessionSummary.from_dict(summary_data) if isinstance(summary_data, dict) else ReviewSessionSummary()
|
||||
|
||||
return cls(
|
||||
session_id=data.get("session_id", str(uuid.uuid4())),
|
||||
job_id=data.get("job_id", ""),
|
||||
user_email=data.get("user_email", ""),
|
||||
created_at=created_at or datetime.now(timezone.utc),
|
||||
updated_at=updated_at or datetime.now(timezone.utc),
|
||||
edit_count=data.get("edit_count", 0),
|
||||
trigger=data.get("trigger", "auto"),
|
||||
audio_duration_seconds=data.get("audio_duration_seconds"),
|
||||
artist=data.get("artist"),
|
||||
title=data.get("title"),
|
||||
summary=summary,
|
||||
correction_data_gcs_path=data.get("correction_data_gcs_path", ""),
|
||||
data_hash=data.get("data_hash", ""),
|
||||
)
|
||||
@@ -95,6 +95,12 @@ class TenantDefaults(BaseModel):
|
||||
brand_prefix: Optional[str] = Field(
|
||||
None, description="Prefix for output filenames (e.g., 'VSTAR')"
|
||||
)
|
||||
dropbox_path: Optional[str] = Field(
|
||||
None, description="Tenant-specific Dropbox folder path (e.g., '/Karaoke/Vocal-Star')"
|
||||
)
|
||||
gdrive_folder_id: Optional[str] = Field(
|
||||
None, description="Tenant-specific Google Drive folder ID for output uploads"
|
||||
)
|
||||
youtube_description_template: Optional[str] = Field(
|
||||
None, description="Default YouTube description template"
|
||||
)
|
||||
@@ -139,6 +145,10 @@ class TenantConfig(BaseModel):
|
||||
created_at: Optional[datetime] = None
|
||||
updated_at: Optional[datetime] = None
|
||||
|
||||
def get_frontend_url(self) -> str:
|
||||
"""Get the frontend URL for this tenant's portal."""
|
||||
return f"https://{self.subdomain}"
|
||||
|
||||
def get_sender_email(self) -> str:
|
||||
"""Get the email sender address for this tenant."""
|
||||
if self.auth.sender_email:
|
||||
|
||||
+71
-50
@@ -6,7 +6,6 @@ Supports:
|
||||
- Credit system for karaoke generation
|
||||
- Role-based access control (user/admin)
|
||||
- Stripe integration for payments
|
||||
- Beta tester program with feedback collection
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
from enum import Enum
|
||||
@@ -20,14 +19,6 @@ class UserRole(str, Enum):
|
||||
ADMIN = "admin"
|
||||
|
||||
|
||||
class BetaTesterStatus(str, Enum):
|
||||
"""Status of beta tester participation."""
|
||||
ACTIVE = "active" # Enrolled, free credits available
|
||||
PENDING_FEEDBACK = "pending_feedback" # Job completed, awaiting feedback
|
||||
COMPLETED = "completed" # Feedback submitted
|
||||
EXPIRED = "expired" # 24hr deadline passed without feedback
|
||||
|
||||
|
||||
class CreditTransaction(BaseModel):
|
||||
"""Record of a credit transaction."""
|
||||
id: str
|
||||
@@ -45,10 +36,15 @@ class PushSubscription(BaseModel):
|
||||
|
||||
Stores the push subscription endpoint and encryption keys needed
|
||||
to send push notifications to the user's browser/device.
|
||||
|
||||
Subscriptions are scoped by tenant_id to prevent cross-tenant
|
||||
notification leakage (e.g., a consumer job notification appearing
|
||||
on a tenant portal's service worker).
|
||||
"""
|
||||
endpoint: str # Push service endpoint URL
|
||||
keys: Dict[str, str] # p256dh and auth keys for encryption
|
||||
device_name: Optional[str] = None # e.g., "iPhone", "Chrome on Windows"
|
||||
tenant_id: Optional[str] = None # Tenant scope (None = consumer/default portal)
|
||||
created_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
|
||||
last_used_at: Optional[datetime] = None # Last time a notification was sent
|
||||
|
||||
@@ -89,21 +85,36 @@ class User(BaseModel):
|
||||
total_jobs_created: int = 0
|
||||
total_jobs_completed: int = 0
|
||||
|
||||
# Payment tracking (cached from stripe_payments, in cents)
|
||||
total_spent: int = 0
|
||||
|
||||
# Optional profile fields for future use
|
||||
display_name: Optional[str] = None
|
||||
|
||||
# Beta tester program
|
||||
is_beta_tester: bool = False
|
||||
beta_tester_status: Optional[BetaTesterStatus] = None
|
||||
beta_enrolled_at: Optional[datetime] = None
|
||||
beta_promise_text: Optional[str] = None # User's promise statement
|
||||
beta_feedback_due_at: Optional[datetime] = None # 24hr after job completion
|
||||
beta_feedback_email_sent: bool = False
|
||||
# i18n locale preference (set from Accept-Language on login, used for emails/push)
|
||||
locale: Optional[str] = None
|
||||
|
||||
# Push notification subscriptions (Web Push API)
|
||||
# Users can subscribe from multiple devices/browsers
|
||||
push_subscriptions: List[PushSubscription] = Field(default_factory=list)
|
||||
|
||||
# Feedback-for-credits program
|
||||
has_submitted_feedback: bool = False
|
||||
|
||||
# Anti-abuse: IP address and device fingerprint used during signup
|
||||
signup_ip: Optional[str] = None
|
||||
device_fingerprint: Optional[str] = None
|
||||
|
||||
# Flag to prevent duplicate welcome credit grants (idempotency)
|
||||
welcome_credits_granted: bool = False
|
||||
|
||||
# Referral system
|
||||
referral_code: Optional[str] = None # User's own referral code
|
||||
referred_by_code: Optional[str] = None # Code used when this user signed up
|
||||
referred_at: Optional[datetime] = None # When the referral was applied
|
||||
referral_discount_expires_at: Optional[datetime] = None # When referral discount ends
|
||||
stripe_connect_account_id: Optional[str] = None # For receiving referral payouts
|
||||
|
||||
|
||||
class MagicLinkToken(BaseModel):
|
||||
"""
|
||||
@@ -123,10 +134,19 @@ class MagicLinkToken(BaseModel):
|
||||
used_at: Optional[datetime] = None
|
||||
ip_address: Optional[str] = None
|
||||
user_agent: Optional[str] = None
|
||||
device_fingerprint: Optional[str] = None
|
||||
|
||||
# Multi-tenant support (None = default Nomad Karaoke)
|
||||
tenant_id: Optional[str] = None
|
||||
|
||||
# Referral attribution (travels with magic link for cross-device support)
|
||||
referral_code: Optional[str] = None
|
||||
|
||||
# Pre-computed credit evaluation (set asynchronously after magic link is sent)
|
||||
credit_eval_decision: Optional[str] = None # "grant", "deny", "pending_review"
|
||||
credit_eval_reasoning: Optional[str] = None
|
||||
credit_eval_error: Optional[str] = None
|
||||
|
||||
|
||||
class Session(BaseModel):
|
||||
"""
|
||||
@@ -144,6 +164,7 @@ class Session(BaseModel):
|
||||
last_activity_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
ip_address: Optional[str] = None
|
||||
user_agent: Optional[str] = None
|
||||
device_fingerprint: Optional[str] = None
|
||||
is_active: bool = True
|
||||
|
||||
# Multi-tenant support (None = default Nomad Karaoke)
|
||||
@@ -155,6 +176,8 @@ class Session(BaseModel):
|
||||
class SendMagicLinkRequest(BaseModel):
|
||||
"""Request to send a magic link email."""
|
||||
email: str
|
||||
device_fingerprint: Optional[str] = None
|
||||
referral_code: Optional[str] = None # From referral interstitial
|
||||
|
||||
|
||||
class SendMagicLinkResponse(BaseModel):
|
||||
@@ -174,6 +197,9 @@ class VerifyMagicLinkResponse(BaseModel):
|
||||
session_token: str
|
||||
user: "UserPublic"
|
||||
message: str
|
||||
tenant_subdomain: Optional[str] = None
|
||||
credits_granted: int = 0
|
||||
credit_status: str = "not_applicable" # "granted", "denied", "pending_review", "already_granted", "not_applicable"
|
||||
|
||||
|
||||
class UserPublic(BaseModel):
|
||||
@@ -185,6 +211,15 @@ class UserPublic(BaseModel):
|
||||
total_jobs_created: int = 0
|
||||
total_jobs_completed: int = 0
|
||||
tenant_id: Optional[str] = None
|
||||
feedback_eligible: bool = False
|
||||
total_spent: int = 0
|
||||
referral_code: Optional[str] = None
|
||||
has_active_referral_discount: bool = False
|
||||
referral_discount_percent: Optional[int] = None
|
||||
referral_discount_expires_at: Optional[str] = None # ISO string
|
||||
referred_by_code: Optional[str] = None
|
||||
created_at: Optional[str] = None
|
||||
last_login_at: Optional[str] = None
|
||||
|
||||
|
||||
class AddCreditsRequest(BaseModel):
|
||||
@@ -210,18 +245,18 @@ class UserListResponse(BaseModel):
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Beta Tester Program Models
|
||||
# User Feedback-for-Credits Models
|
||||
# ============================================================================
|
||||
|
||||
class BetaTesterFeedback(BaseModel):
|
||||
class UserFeedback(BaseModel):
|
||||
"""
|
||||
Feedback from a beta tester after using the service.
|
||||
Feedback from any user (not just beta testers).
|
||||
|
||||
Stored in Firestore 'beta_feedback' collection.
|
||||
Stored in Firestore 'user_feedback' collection.
|
||||
Users earn 1 free credit for submitting detailed feedback.
|
||||
"""
|
||||
id: str # Unique feedback ID
|
||||
id: str
|
||||
user_email: str
|
||||
job_id: Optional[str] = None # The job they're providing feedback on
|
||||
|
||||
# Ratings (1-5 scale)
|
||||
overall_rating: int = Field(ge=1, le=5)
|
||||
@@ -240,33 +275,11 @@ class BetaTesterFeedback(BaseModel):
|
||||
|
||||
# Metadata
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
submitted_via: str = "web" # 'web' or 'email'
|
||||
submitted_via: str = "web"
|
||||
|
||||
|
||||
class BetaTesterEnrollRequest(BaseModel):
|
||||
"""Request to enroll as a beta tester."""
|
||||
email: str
|
||||
promise_text: str = Field(
|
||||
min_length=10,
|
||||
description="User's promise to provide feedback (min 10 chars)"
|
||||
)
|
||||
accept_corrections_work: bool = Field(
|
||||
description="User accepts they may need to review/correct lyrics"
|
||||
)
|
||||
|
||||
|
||||
class BetaTesterEnrollResponse(BaseModel):
|
||||
"""Response after enrolling as beta tester."""
|
||||
status: str
|
||||
message: str
|
||||
credits_granted: int
|
||||
session_token: Optional[str] = None # If new user, provide session
|
||||
|
||||
|
||||
class BetaFeedbackRequest(BaseModel):
|
||||
"""Request to submit beta tester feedback."""
|
||||
job_id: Optional[str] = None
|
||||
|
||||
class UserFeedbackRequest(BaseModel):
|
||||
"""Request to submit user feedback for credits."""
|
||||
# Ratings (1-5 scale)
|
||||
overall_rating: int = Field(ge=1, le=5)
|
||||
ease_of_use_rating: int = Field(ge=1, le=5)
|
||||
@@ -283,8 +296,16 @@ class BetaFeedbackRequest(BaseModel):
|
||||
would_use_again: bool = True
|
||||
|
||||
|
||||
class BetaFeedbackResponse(BaseModel):
|
||||
"""Response after submitting feedback."""
|
||||
class UserFeedbackResponse(BaseModel):
|
||||
"""Response after submitting user feedback."""
|
||||
status: str
|
||||
message: str
|
||||
bonus_credits: int = 0 # Bonus credits for great feedback
|
||||
credits_granted: int = 0
|
||||
|
||||
|
||||
class FeedbackEligibilityResponse(BaseModel):
|
||||
"""Response for feedback eligibility check."""
|
||||
eligible: bool
|
||||
has_submitted: bool
|
||||
jobs_completed: int
|
||||
credits_reward: int = 1
|
||||
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Download ensemble preset models at Docker build time.
|
||||
|
||||
Bakes models into the image so they're available immediately at runtime,
|
||||
avoiding cold-start model download latency in Cloud Run Jobs.
|
||||
|
||||
Usage: python backend/scripts/download_models.py /models
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
import logging
|
||||
import importlib.resources as resources
|
||||
from audio_separator.separator import Separator
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Presets to bake into the GPU image
|
||||
PRESETS_TO_DOWNLOAD = ["instrumental_clean", "karaoke"]
|
||||
|
||||
|
||||
def download_preset_models(model_dir: str) -> None:
|
||||
"""Download all models for the configured presets individually.
|
||||
|
||||
Ensemble presets resolve to multiple model filenames. We must download
|
||||
each model individually via load_model(model_filename) — calling
|
||||
load_model() with no args on an ensemble preset doesn't trigger downloads.
|
||||
"""
|
||||
with resources.open_text("audio_separator", "ensemble_presets.json") as f:
|
||||
presets = json.load(f)["presets"]
|
||||
|
||||
models_to_download = set()
|
||||
for preset_name in PRESETS_TO_DOWNLOAD:
|
||||
models_to_download.update(presets[preset_name]["models"])
|
||||
|
||||
logger.info(f"Downloading {len(models_to_download)} models for presets: {PRESETS_TO_DOWNLOAD}")
|
||||
for model in sorted(models_to_download):
|
||||
logger.info(f" Downloading: {model}")
|
||||
sep = Separator(model_file_dir=model_dir)
|
||||
sep.load_model(model)
|
||||
logger.info(f" Done: {model}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
model_dir = sys.argv[1] if len(sys.argv) > 1 else "/models"
|
||||
download_preset_models(model_dir)
|
||||
logger.info(f"All preset models downloaded to {model_dir}")
|
||||
@@ -0,0 +1,265 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Generate a Mermaid state diagram from the job state machine.
|
||||
|
||||
This script reads the STATE_TRANSITIONS dictionary from models/job.py
|
||||
and generates a Mermaid diagram that can be embedded in documentation.
|
||||
|
||||
Usage:
|
||||
python scripts/generate_state_diagram.py > docs/state-machine.mmd
|
||||
# Or to update ARCHITECTURE.md directly:
|
||||
python scripts/generate_state_diagram.py --update-docs
|
||||
|
||||
Created as part of state machine robustness improvements (2026-02-02).
|
||||
"""
|
||||
import sys
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
# Add backend to path for imports
|
||||
backend_dir = Path(__file__).parent.parent
|
||||
sys.path.insert(0, str(backend_dir))
|
||||
|
||||
# Import the state machine data
|
||||
# Note: Must be run from an environment where karaoke_gen is installed
|
||||
try:
|
||||
from models.job import STATE_TRANSITIONS, JobStatus
|
||||
except ImportError:
|
||||
# Fallback: define the state machine inline for documentation purposes
|
||||
from enum import Enum
|
||||
|
||||
class JobStatus(str, Enum):
|
||||
PENDING = "pending"
|
||||
SEARCHING_AUDIO = "searching_audio"
|
||||
AWAITING_AUDIO_SELECTION = "awaiting_audio_selection"
|
||||
DOWNLOADING_AUDIO = "downloading_audio"
|
||||
DOWNLOADING = "downloading"
|
||||
SEPARATING_STAGE1 = "separating_stage1"
|
||||
SEPARATING_STAGE2 = "separating_stage2"
|
||||
AUDIO_COMPLETE = "audio_complete"
|
||||
TRANSCRIBING = "transcribing"
|
||||
CORRECTING = "correcting"
|
||||
LYRICS_COMPLETE = "lyrics_complete"
|
||||
GENERATING_SCREENS = "generating_screens"
|
||||
APPLYING_PADDING = "applying_padding"
|
||||
AWAITING_REVIEW = "awaiting_review"
|
||||
IN_REVIEW = "in_review"
|
||||
REVIEW_COMPLETE = "review_complete"
|
||||
RENDERING_VIDEO = "rendering_video"
|
||||
AWAITING_INSTRUMENTAL_SELECTION = "awaiting_instrumental_selection"
|
||||
INSTRUMENTAL_SELECTED = "instrumental_selected"
|
||||
GENERATING_VIDEO = "generating_video"
|
||||
ENCODING = "encoding"
|
||||
PACKAGING = "packaging"
|
||||
UPLOADING = "uploading"
|
||||
NOTIFYING = "notifying"
|
||||
COMPLETE = "complete"
|
||||
PREP_COMPLETE = "prep_complete"
|
||||
FAILED = "failed"
|
||||
CANCELLED = "cancelled"
|
||||
QUEUED = "queued"
|
||||
PROCESSING = "processing"
|
||||
ERROR = "error"
|
||||
READY_FOR_FINALIZATION = "ready_for_finalization"
|
||||
FINALIZING = "finalizing"
|
||||
|
||||
STATE_TRANSITIONS = {
|
||||
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.SEARCHING_AUDIO: [JobStatus.AWAITING_AUDIO_SELECTION, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED],
|
||||
JobStatus.AWAITING_AUDIO_SELECTION: [JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.DOWNLOADING_AUDIO: [JobStatus.DOWNLOADING, JobStatus.FAILED],
|
||||
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
JobStatus.SEPARATING_STAGE1: [JobStatus.SEPARATING_STAGE2, JobStatus.FAILED],
|
||||
JobStatus.SEPARATING_STAGE2: [JobStatus.AUDIO_COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.AUDIO_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
JobStatus.TRANSCRIBING: [JobStatus.CORRECTING, JobStatus.FAILED],
|
||||
JobStatus.CORRECTING: [JobStatus.LYRICS_COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.LYRICS_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
JobStatus.GENERATING_SCREENS: [JobStatus.APPLYING_PADDING, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
|
||||
JobStatus.APPLYING_PADDING: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
|
||||
JobStatus.AWAITING_REVIEW: [JobStatus.IN_REVIEW, JobStatus.REVIEW_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.IN_REVIEW: [JobStatus.REVIEW_COMPLETE, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
|
||||
JobStatus.REVIEW_COMPLETE: [JobStatus.RENDERING_VIDEO, JobStatus.FAILED],
|
||||
JobStatus.RENDERING_VIDEO: [JobStatus.INSTRUMENTAL_SELECTED, JobStatus.FAILED],
|
||||
JobStatus.INSTRUMENTAL_SELECTED: [JobStatus.GENERATING_VIDEO, JobStatus.FAILED],
|
||||
JobStatus.GENERATING_VIDEO: [JobStatus.ENCODING, JobStatus.FAILED],
|
||||
JobStatus.ENCODING: [JobStatus.PACKAGING, JobStatus.COMPLETE, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.PACKAGING: [JobStatus.UPLOADING, JobStatus.COMPLETE, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.UPLOADING: [JobStatus.NOTIFYING, JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.NOTIFYING: [JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
}
|
||||
|
||||
|
||||
def generate_mermaid() -> str:
|
||||
"""
|
||||
Generate a Mermaid state diagram from STATE_TRANSITIONS.
|
||||
|
||||
Returns:
|
||||
Mermaid diagram as a string
|
||||
"""
|
||||
lines = ["stateDiagram-v2"]
|
||||
lines.append(" direction TB")
|
||||
lines.append("")
|
||||
|
||||
# Group states by category for better visualization
|
||||
categories = {
|
||||
"Initial": [JobStatus.PENDING],
|
||||
"Audio Search": [
|
||||
JobStatus.SEARCHING_AUDIO,
|
||||
JobStatus.AWAITING_AUDIO_SELECTION,
|
||||
JobStatus.DOWNLOADING_AUDIO,
|
||||
],
|
||||
"Processing": [
|
||||
JobStatus.DOWNLOADING,
|
||||
JobStatus.SEPARATING_STAGE1,
|
||||
JobStatus.SEPARATING_STAGE2,
|
||||
JobStatus.AUDIO_COMPLETE,
|
||||
JobStatus.TRANSCRIBING,
|
||||
JobStatus.CORRECTING,
|
||||
JobStatus.LYRICS_COMPLETE,
|
||||
],
|
||||
"Screens": [
|
||||
JobStatus.GENERATING_SCREENS,
|
||||
JobStatus.APPLYING_PADDING,
|
||||
],
|
||||
"Review": [
|
||||
JobStatus.AWAITING_REVIEW,
|
||||
JobStatus.IN_REVIEW,
|
||||
JobStatus.REVIEW_COMPLETE,
|
||||
],
|
||||
"Video": [
|
||||
JobStatus.RENDERING_VIDEO,
|
||||
JobStatus.INSTRUMENTAL_SELECTED,
|
||||
JobStatus.GENERATING_VIDEO,
|
||||
JobStatus.ENCODING,
|
||||
JobStatus.PACKAGING,
|
||||
],
|
||||
"Distribution": [
|
||||
JobStatus.UPLOADING,
|
||||
JobStatus.NOTIFYING,
|
||||
],
|
||||
"Terminal": [
|
||||
JobStatus.COMPLETE,
|
||||
JobStatus.PREP_COMPLETE,
|
||||
JobStatus.FAILED,
|
||||
JobStatus.CANCELLED,
|
||||
],
|
||||
}
|
||||
|
||||
# Add state definitions with notes for key states
|
||||
lines.append(" %% State definitions")
|
||||
key_states = {
|
||||
JobStatus.PENDING: "Job created",
|
||||
JobStatus.AWAITING_AUDIO_SELECTION: "⚠️ USER ACTION",
|
||||
JobStatus.AWAITING_REVIEW: "⚠️ USER ACTION",
|
||||
JobStatus.COMPLETE: "✅ Success",
|
||||
JobStatus.FAILED: "❌ Error",
|
||||
}
|
||||
|
||||
for status, note in key_states.items():
|
||||
lines.append(f" {status.value}: {status.value}")
|
||||
lines.append(f" note right of {status.value}: {note}")
|
||||
|
||||
lines.append("")
|
||||
lines.append(" %% Transitions")
|
||||
|
||||
# Generate transitions
|
||||
for from_status, to_statuses in STATE_TRANSITIONS.items():
|
||||
for to_status in to_statuses:
|
||||
# Skip deprecated statuses
|
||||
if to_status in [JobStatus.QUEUED, JobStatus.PROCESSING, JobStatus.ERROR,
|
||||
JobStatus.READY_FOR_FINALIZATION, JobStatus.FINALIZING]:
|
||||
continue
|
||||
if from_status in [JobStatus.QUEUED, JobStatus.PROCESSING, JobStatus.ERROR,
|
||||
JobStatus.READY_FOR_FINALIZATION, JobStatus.FINALIZING]:
|
||||
continue
|
||||
|
||||
lines.append(f" {from_status.value} --> {to_status.value}")
|
||||
|
||||
# Add start and end markers
|
||||
lines.append("")
|
||||
lines.append(" %% Start and end")
|
||||
lines.append(" [*] --> pending")
|
||||
lines.append(" complete --> [*]")
|
||||
lines.append(" prep_complete --> [*]")
|
||||
lines.append(" failed --> [*]")
|
||||
lines.append(" cancelled --> [*]")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def generate_markdown_section() -> str:
|
||||
"""
|
||||
Generate a markdown section with the state diagram.
|
||||
|
||||
Returns:
|
||||
Markdown string with the diagram
|
||||
"""
|
||||
diagram = generate_mermaid()
|
||||
|
||||
return f"""## Job State Machine
|
||||
|
||||
The following diagram shows all valid state transitions for a karaoke generation job.
|
||||
|
||||
Key states:
|
||||
- **PENDING**: Initial state when job is created
|
||||
- **AWAITING_AUDIO_SELECTION**: Human selects audio source (for artist+title search)
|
||||
- **AWAITING_REVIEW**: Human reviews lyrics and selects instrumental
|
||||
- **COMPLETE**: All processing finished successfully
|
||||
|
||||
```mermaid
|
||||
{diagram}
|
||||
```
|
||||
|
||||
### Parallel Processing
|
||||
|
||||
Jobs use parallel processing for audio separation and lyrics transcription:
|
||||
|
||||
1. **DOWNLOADING** triggers both:
|
||||
- Audio worker → SEPARATING_STAGE1 → SEPARATING_STAGE2 → AUDIO_COMPLETE
|
||||
- Lyrics worker → TRANSCRIBING → CORRECTING → LYRICS_COMPLETE
|
||||
|
||||
2. When BOTH complete, the screens worker is triggered automatically.
|
||||
|
||||
### Human Review Points
|
||||
|
||||
Jobs wait for human input at these points:
|
||||
- **AWAITING_AUDIO_SELECTION**: User selects audio source from search results
|
||||
- **AWAITING_REVIEW**: User reviews/corrects lyrics AND selects instrumental
|
||||
|
||||
### Error Handling
|
||||
|
||||
- Any state can transition to **FAILED** on unrecoverable errors
|
||||
- **CANCELLED** is set when user cancels the job
|
||||
|
||||
"""
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Generate state machine diagram")
|
||||
parser.add_argument(
|
||||
"--update-docs",
|
||||
action="store_true",
|
||||
help="Update ARCHITECTURE.md with the diagram"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--markdown",
|
||||
action="store_true",
|
||||
help="Output markdown section instead of raw Mermaid"
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.update_docs:
|
||||
# TODO: Implement auto-update of ARCHITECTURE.md
|
||||
print("Auto-update of ARCHITECTURE.md not yet implemented")
|
||||
print("Please copy the output below to the appropriate section:")
|
||||
print()
|
||||
print(generate_markdown_section())
|
||||
elif args.markdown:
|
||||
print(generate_markdown_section())
|
||||
else:
|
||||
print(generate_mermaid())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -5,6 +5,7 @@ This service wraps the shared karaoke_gen.instrumental_review module
|
||||
to provide GCS-integrated audio analysis capabilities.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
@@ -187,7 +188,78 @@ class AudioAnalysisService:
|
||||
)
|
||||
|
||||
return amplitudes, duration
|
||||
|
||||
|
||||
def cache_waveform_data(
|
||||
self,
|
||||
gcs_audio_path: str,
|
||||
job_id: str,
|
||||
cache_gcs_path: str,
|
||||
num_points: int = 1000,
|
||||
) -> tuple[list[float], float]:
|
||||
"""
|
||||
Generate waveform data and cache it as JSON in GCS.
|
||||
|
||||
Args:
|
||||
gcs_audio_path: Path to the audio file in GCS
|
||||
job_id: Job ID for logging
|
||||
cache_gcs_path: GCS path to store the cached waveform JSON
|
||||
num_points: Number of data points to generate
|
||||
|
||||
Returns:
|
||||
Tuple of (amplitude_values, duration_seconds)
|
||||
"""
|
||||
logger.info(f"[{job_id}] Generating and caching waveform data: {gcs_audio_path}")
|
||||
|
||||
amplitudes, duration = self.get_waveform_data(
|
||||
gcs_audio_path=gcs_audio_path,
|
||||
job_id=job_id,
|
||||
num_points=num_points,
|
||||
)
|
||||
|
||||
cache_data = {
|
||||
"amplitudes": list(amplitudes),
|
||||
"duration_seconds": duration,
|
||||
"num_points": num_points,
|
||||
"source_gcs_path": gcs_audio_path,
|
||||
}
|
||||
self.storage_service.upload_json(cache_gcs_path, cache_data)
|
||||
logger.info(f"[{job_id}] Waveform data cached at {cache_gcs_path}")
|
||||
|
||||
return amplitudes, duration
|
||||
|
||||
def load_cached_waveform(
|
||||
self,
|
||||
cache_gcs_path: str,
|
||||
) -> Optional[tuple[list[float], float]]:
|
||||
"""
|
||||
Load cached waveform data from GCS.
|
||||
|
||||
Args:
|
||||
cache_gcs_path: GCS path to the cached waveform JSON
|
||||
|
||||
Returns:
|
||||
Tuple of (amplitude_values, duration_seconds) or None if not found
|
||||
"""
|
||||
try:
|
||||
if not self.storage_service.file_exists(cache_gcs_path):
|
||||
return None
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
local_path = os.path.join(temp_dir, "waveform.json")
|
||||
self.storage_service.download_file(cache_gcs_path, local_path)
|
||||
with open(local_path) as f:
|
||||
data = json.load(f)
|
||||
|
||||
amplitudes = data.get("amplitudes", [])
|
||||
duration = data.get("duration_seconds", 0.0)
|
||||
if not amplitudes:
|
||||
return None
|
||||
|
||||
return amplitudes, duration
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to load cached waveform from {cache_gcs_path}: {e}")
|
||||
return None
|
||||
|
||||
def generate_waveform_with_mutes(
|
||||
self,
|
||||
gcs_audio_path: str,
|
||||
|
||||
@@ -0,0 +1,180 @@
|
||||
"""
|
||||
Audio editing service — server-side FFmpeg operations for trim, cut, mute, join.
|
||||
|
||||
All operations are lossless (FLAC in, FLAC out). Playback URLs use OGG Opus
|
||||
via the existing AudioTranscodingService.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import json
|
||||
import subprocess
|
||||
import tempfile
|
||||
from dataclasses import dataclass, asdict
|
||||
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class AudioMetadata:
|
||||
duration_seconds: float
|
||||
sample_rate: int
|
||||
channels: int
|
||||
format: str
|
||||
file_size_bytes: int
|
||||
|
||||
|
||||
class AudioEditService:
|
||||
"""Server-side audio editing via FFmpeg."""
|
||||
|
||||
def __init__(self, storage_service: StorageService | None = None):
|
||||
self.storage = storage_service or StorageService()
|
||||
|
||||
def get_metadata(self, audio_path: str) -> AudioMetadata:
|
||||
"""Get audio metadata from a local file using ffprobe."""
|
||||
cmd = [
|
||||
"ffprobe", "-v", "quiet", "-print_format", "json",
|
||||
"-show_format", "-show_streams", audio_path,
|
||||
]
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(f"ffprobe failed: {result.stderr}")
|
||||
|
||||
info = json.loads(result.stdout)
|
||||
fmt = info.get("format", {})
|
||||
streams = info.get("streams", [])
|
||||
audio_stream = next((s for s in streams if s.get("codec_type") == "audio"), {})
|
||||
|
||||
return AudioMetadata(
|
||||
duration_seconds=float(fmt.get("duration", 0)),
|
||||
sample_rate=int(audio_stream.get("sample_rate", 44100)),
|
||||
channels=int(audio_stream.get("channels", 2)),
|
||||
format=fmt.get("format_name", "unknown"),
|
||||
file_size_bytes=int(fmt.get("size", 0)),
|
||||
)
|
||||
|
||||
def get_metadata_from_gcs(self, gcs_path: str) -> AudioMetadata:
|
||||
"""Download from GCS and get metadata."""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
local_path = os.path.join(temp_dir, "audio")
|
||||
self.storage.download_file(gcs_path, local_path)
|
||||
return self.get_metadata(local_path)
|
||||
|
||||
def trim_start(self, input_path: str, end_seconds: float, output_path: str) -> AudioMetadata:
|
||||
"""Remove audio from 0 to end_seconds (skip the first N seconds)."""
|
||||
self._run_ffmpeg([
|
||||
"-ss", str(end_seconds),
|
||||
"-i", input_path,
|
||||
"-c:a", "flac",
|
||||
output_path,
|
||||
])
|
||||
return self.get_metadata(output_path)
|
||||
|
||||
def trim_end(self, input_path: str, start_seconds: float, output_path: str) -> AudioMetadata:
|
||||
"""Keep audio from 0 to start_seconds (remove from start_seconds to end)."""
|
||||
self._run_ffmpeg([
|
||||
"-i", input_path,
|
||||
"-t", str(start_seconds),
|
||||
"-c:a", "flac",
|
||||
output_path,
|
||||
])
|
||||
return self.get_metadata(output_path)
|
||||
|
||||
def cut_region(self, input_path: str, start: float, end: float, output_path: str) -> AudioMetadata:
|
||||
"""Remove a region from start to end, joining the remaining parts."""
|
||||
self._run_ffmpeg([
|
||||
"-i", input_path,
|
||||
"-filter_complex",
|
||||
f"[0]atrim=0:{start},asetpts=PTS-STARTPTS[a];"
|
||||
f"[0]atrim={end},asetpts=PTS-STARTPTS[b];"
|
||||
f"[a][b]concat=n=2:v=0:a=1[out]",
|
||||
"-map", "[out]",
|
||||
"-c:a", "flac",
|
||||
output_path,
|
||||
])
|
||||
return self.get_metadata(output_path)
|
||||
|
||||
def mute_region(self, input_path: str, start: float, end: float, output_path: str) -> AudioMetadata:
|
||||
"""Silence a region (preserve duration)."""
|
||||
self._run_ffmpeg([
|
||||
"-i", input_path,
|
||||
"-af", f"volume=enable='between(t,{start},{end})':volume=0",
|
||||
"-c:a", "flac",
|
||||
output_path,
|
||||
])
|
||||
return self.get_metadata(output_path)
|
||||
|
||||
def join_audio(self, input_path: str, other_path: str, position: str, output_path: str) -> AudioMetadata:
|
||||
"""Join two audio files. position: 'start' (prepend other) or 'end' (append other)."""
|
||||
if position == "start":
|
||||
first, second = other_path, input_path
|
||||
else:
|
||||
first, second = input_path, other_path
|
||||
|
||||
self._run_ffmpeg([
|
||||
"-i", first,
|
||||
"-i", second,
|
||||
"-filter_complex", "[0:a][1:a]concat=n=2:v=0:a=1[out]",
|
||||
"-map", "[out]",
|
||||
"-c:a", "flac",
|
||||
output_path,
|
||||
])
|
||||
return self.get_metadata(output_path)
|
||||
|
||||
def apply_edit(
|
||||
self,
|
||||
input_gcs_path: str,
|
||||
operation: str,
|
||||
params: dict,
|
||||
output_gcs_path: str,
|
||||
job_id: str,
|
||||
) -> tuple[AudioMetadata, str]:
|
||||
"""
|
||||
Apply an edit operation to an audio file in GCS.
|
||||
|
||||
Returns (metadata, output_gcs_path).
|
||||
"""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
local_input = os.path.join(temp_dir, "input.flac")
|
||||
local_output = os.path.join(temp_dir, "output.flac")
|
||||
|
||||
# Download current audio
|
||||
self.storage.download_file(input_gcs_path, local_input)
|
||||
|
||||
# Apply the operation
|
||||
if operation == "trim_start":
|
||||
metadata = self.trim_start(local_input, params["end_seconds"], local_output)
|
||||
elif operation == "trim_end":
|
||||
metadata = self.trim_end(local_input, params["start_seconds"], local_output)
|
||||
elif operation == "cut":
|
||||
metadata = self.cut_region(
|
||||
local_input, params["start_seconds"], params["end_seconds"], local_output
|
||||
)
|
||||
elif operation == "mute":
|
||||
metadata = self.mute_region(
|
||||
local_input, params["start_seconds"], params["end_seconds"], local_output
|
||||
)
|
||||
elif operation in ("join_start", "join_end"):
|
||||
# Download the upload file
|
||||
upload_gcs_path = params["upload_gcs_path"]
|
||||
local_other = os.path.join(temp_dir, "other.flac")
|
||||
self.storage.download_file(upload_gcs_path, local_other)
|
||||
position = "start" if operation == "join_start" else "end"
|
||||
metadata = self.join_audio(local_input, local_other, position, local_output)
|
||||
else:
|
||||
raise ValueError(f"Unknown operation: {operation}")
|
||||
|
||||
# Upload edited audio
|
||||
self.storage.upload_file(local_output, output_gcs_path)
|
||||
logger.info(f"[{job_id}] Applied {operation}, uploaded to {output_gcs_path}")
|
||||
|
||||
return metadata, output_gcs_path
|
||||
|
||||
def _run_ffmpeg(self, args: list[str]) -> None:
|
||||
"""Run an ffmpeg command with standard options."""
|
||||
cmd = ["ffmpeg", "-hide_banner", "-loglevel", "error", "-y"] + args
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(f"ffmpeg failed (exit {result.returncode}): {result.stderr}")
|
||||
@@ -612,13 +612,13 @@ class AudioSearchService:
|
||||
async def search_async(self, artist: str, title: str) -> List[AudioSearchResult]:
|
||||
"""
|
||||
Async version of search for use in async routes.
|
||||
|
||||
Note: Currently wraps sync search in executor. Future optimization
|
||||
could make this fully async.
|
||||
|
||||
Runs the sync search in a thread executor to avoid blocking
|
||||
the event loop.
|
||||
"""
|
||||
loop = asyncio.get_event_loop()
|
||||
loop = asyncio.get_running_loop()
|
||||
return await loop.run_in_executor(None, self.search, artist, title)
|
||||
|
||||
|
||||
async def download_async(
|
||||
self,
|
||||
result_index: int,
|
||||
@@ -630,10 +630,10 @@ class AudioSearchService:
|
||||
"""
|
||||
Async version of download for use in async routes.
|
||||
|
||||
Note: Currently wraps sync download in executor. Future optimization
|
||||
could make this fully async.
|
||||
Runs the sync download in a thread executor to avoid blocking
|
||||
the event loop.
|
||||
"""
|
||||
loop = asyncio.get_event_loop()
|
||||
loop = asyncio.get_running_loop()
|
||||
return await loop.run_in_executor(
|
||||
None,
|
||||
lambda: self.download(result_index, output_dir, output_filename, gcs_path, remote_search_id)
|
||||
@@ -651,8 +651,11 @@ class AudioSearchService:
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Async version of download_by_id for use in async routes.
|
||||
|
||||
Runs the sync download_by_id in a thread executor to avoid blocking
|
||||
the event loop.
|
||||
"""
|
||||
loop = asyncio.get_event_loop()
|
||||
loop = asyncio.get_running_loop()
|
||||
return await loop.run_in_executor(
|
||||
None,
|
||||
lambda: self.download_by_id(
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
"""
|
||||
Audio transcoding service for review UI optimization.
|
||||
|
||||
Transcodes FLAC audio files to OGG Opus for efficient browser playback.
|
||||
Original FLAC files are never modified — transcoded versions are cached
|
||||
in GCS at jobs/{job_id}/review-audio/{filename}.ogg.
|
||||
|
||||
Eager transcoding: called from screens_worker before AWAITING_REVIEW
|
||||
so all files are cached before the user opens the review UI.
|
||||
Falls back to original FLAC signed URLs if transcoded version is missing.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AudioTranscodingService:
|
||||
"""
|
||||
Transcodes audio files to OGG Opus and caches them in GCS.
|
||||
|
||||
Used by the review UI to serve ~3 MB OGG files instead of 35+ MB FLACs.
|
||||
"""
|
||||
|
||||
def __init__(self, storage_service: Optional[StorageService] = None):
|
||||
self.storage = storage_service or StorageService()
|
||||
|
||||
def _get_cache_path(self, source_gcs_path: str) -> str:
|
||||
"""
|
||||
Derive the cache path for a transcoded file.
|
||||
|
||||
Examples:
|
||||
jobs/abc123/input/song.flac -> jobs/abc123/review-audio/song.ogg
|
||||
jobs/abc123/stems/instrumental_clean.flac -> jobs/abc123/review-audio/instrumental_clean.ogg
|
||||
"""
|
||||
parts = Path(source_gcs_path).parts
|
||||
# Find "jobs/{job_id}" prefix
|
||||
try:
|
||||
jobs_idx = list(parts).index("jobs")
|
||||
job_id = parts[jobs_idx + 1]
|
||||
except (ValueError, IndexError):
|
||||
# Fallback: use a hash of the full path to avoid collisions
|
||||
job_id = "_" + hashlib.sha256(source_gcs_path.encode()).hexdigest()[:12]
|
||||
|
||||
filename = Path(source_gcs_path).stem + ".ogg"
|
||||
return f"jobs/{job_id}/review-audio/{filename}"
|
||||
|
||||
def _transcode_and_upload(self, source_gcs_path: str, cache_path: str) -> str:
|
||||
"""
|
||||
Download FLAC from GCS, transcode to OGG Opus, upload to cache path.
|
||||
|
||||
Returns the cache GCS path on success.
|
||||
Raises on failure.
|
||||
"""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
local_input = os.path.join(temp_dir, "input.flac")
|
||||
local_output = os.path.join(temp_dir, "output.ogg")
|
||||
|
||||
# Download source
|
||||
self.storage.download_file(source_gcs_path, local_input)
|
||||
|
||||
# Transcode: FLAC -> OGG Opus 128kbps
|
||||
cmd = [
|
||||
"ffmpeg", "-hide_banner", "-loglevel", "error",
|
||||
"-y",
|
||||
"-i", local_input,
|
||||
"-c:a", "libopus",
|
||||
"-b:a", "128k",
|
||||
"-vn",
|
||||
local_output,
|
||||
]
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(
|
||||
f"ffmpeg transcoding failed (exit {result.returncode}): {result.stderr}"
|
||||
)
|
||||
|
||||
# Upload transcoded file
|
||||
self.storage.upload_file(local_output, cache_path)
|
||||
|
||||
return cache_path
|
||||
|
||||
def transcode_if_needed(self, source_gcs_path: str) -> str:
|
||||
"""
|
||||
Return the GCS path of the transcoded OGG file, transcoding if not cached.
|
||||
|
||||
Returns the cache path (always an OGG file in review-audio/).
|
||||
Raises if transcoding fails.
|
||||
"""
|
||||
cache_path = self._get_cache_path(source_gcs_path)
|
||||
|
||||
if self.storage.file_exists(cache_path):
|
||||
logger.debug(f"Cache hit: {cache_path}")
|
||||
return cache_path
|
||||
|
||||
logger.info(f"Cache miss, transcoding: {source_gcs_path} -> {cache_path}")
|
||||
return self._transcode_and_upload(source_gcs_path, cache_path)
|
||||
|
||||
def get_review_audio_url(
|
||||
self, source_gcs_path: str, expiration_minutes: int = 120
|
||||
) -> str:
|
||||
"""
|
||||
Get a signed URL for the transcoded audio, falling back to original FLAC.
|
||||
"""
|
||||
try:
|
||||
cache_path = self.transcode_if_needed(source_gcs_path)
|
||||
return self.storage.generate_signed_url(cache_path, expiration_minutes)
|
||||
except Exception as e:
|
||||
# Source may be deleted but cache from earlier transcode persists
|
||||
try:
|
||||
cache_path = self._get_cache_path(source_gcs_path)
|
||||
if self.storage.file_exists(cache_path):
|
||||
logger.info(f"Source gone but cache exists, serving: {cache_path}")
|
||||
return self.storage.generate_signed_url(cache_path, expiration_minutes)
|
||||
except Exception:
|
||||
pass # Fall through to FLAC fallback below
|
||||
logger.warning(
|
||||
f"Transcoding failed for {source_gcs_path}, falling back to FLAC: {e}"
|
||||
)
|
||||
return self.storage.generate_signed_url(source_gcs_path, expiration_minutes)
|
||||
|
||||
async def get_review_audio_url_async(
|
||||
self, source_gcs_path: str, expiration_minutes: int = 120
|
||||
) -> str:
|
||||
"""Async wrapper around get_review_audio_url via asyncio.to_thread."""
|
||||
return await asyncio.to_thread(
|
||||
self.get_review_audio_url, source_gcs_path, expiration_minutes
|
||||
)
|
||||
|
||||
def prepare_review_audio_for_job(self, job) -> list[str]:
|
||||
"""
|
||||
Transcode all review audio files for a job (eager transcoding).
|
||||
|
||||
Called from screens_worker before AWAITING_REVIEW transition
|
||||
and from the admin backfill endpoint.
|
||||
|
||||
Returns list of cache paths that were transcoded.
|
||||
"""
|
||||
job_id = job.job_id
|
||||
transcoded = []
|
||||
|
||||
# Collect all audio paths to transcode
|
||||
paths_to_transcode = []
|
||||
|
||||
# Main input audio
|
||||
if job.input_media_gcs_path:
|
||||
paths_to_transcode.append(("input", job.input_media_gcs_path))
|
||||
|
||||
# Stems
|
||||
stems = job.file_urls.get("stems", {})
|
||||
for stem_key in ["instrumental_clean", "instrumental_with_backing", "backing_vocals"]:
|
||||
path = stems.get(stem_key)
|
||||
if path:
|
||||
paths_to_transcode.append((stem_key, path))
|
||||
|
||||
for label, source_path in paths_to_transcode:
|
||||
try:
|
||||
cache_path = self.transcode_if_needed(source_path)
|
||||
transcoded.append(cache_path)
|
||||
logger.info(f"[{job_id}] Transcoded {label}: {cache_path}")
|
||||
except Exception as e:
|
||||
# Non-fatal: review UI will fall back to FLAC
|
||||
logger.warning(
|
||||
f"[{job_id}] Failed to transcode {label} ({source_path}): {e}"
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"[{job_id}] Review audio preparation complete: "
|
||||
f"{len(transcoded)}/{len(paths_to_transcode)} files transcoded"
|
||||
)
|
||||
return transcoded
|
||||
@@ -3,6 +3,7 @@ Authentication service for karaoke backend.
|
||||
|
||||
Provides token-based authentication with support for:
|
||||
- Admin tokens (hardcoded in environment)
|
||||
- Google OIDC tokens (from Cloud Scheduler / Cloud Tasks service accounts)
|
||||
- Admin by email domain (@nomadkaraoke.com)
|
||||
- User tokens (stored in Firestore auth_tokens collection)
|
||||
- Session tokens (stored in Firestore sessions collection, from magic link auth)
|
||||
@@ -19,6 +20,9 @@ from enum import Enum
|
||||
from typing import Optional, Tuple, Dict, Any, List
|
||||
from datetime import datetime
|
||||
|
||||
from google.auth.transport import requests as google_auth_requests
|
||||
from google.oauth2 import id_token as google_id_token
|
||||
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
from backend.config import get_settings
|
||||
from backend.models.user import UserRole
|
||||
@@ -49,6 +53,7 @@ class AuthResult:
|
||||
user_email: Optional[str] = None # Email if authenticated via session/API key
|
||||
is_admin: bool = False # True if admin token or admin email domain
|
||||
api_key_id: Optional[str] = None # API key ID if authenticated via API key
|
||||
tenant_id: Optional[str] = None # Tenant scope from token or session
|
||||
|
||||
def __iter__(self):
|
||||
"""Allow unpacking as tuple for backward compatibility."""
|
||||
@@ -71,6 +76,7 @@ class AuthService:
|
||||
self.firestore = FirestoreService()
|
||||
self.settings = get_settings()
|
||||
self._init_admin_tokens()
|
||||
self._scheduler_service_account = self.settings.scheduler_service_account
|
||||
|
||||
def _init_admin_tokens(self):
|
||||
"""Load admin tokens from environment variables."""
|
||||
@@ -82,6 +88,56 @@ class AuthService:
|
||||
logger.info(f"Loaded {len(self.admin_tokens)} admin token(s)")
|
||||
else:
|
||||
logger.warning("No admin tokens configured! Set ADMIN_TOKENS environment variable.")
|
||||
|
||||
def _validate_oidc_token(self, token: str) -> Optional[AuthResult]:
|
||||
"""
|
||||
Validate a Google OIDC token (from Cloud Scheduler or Cloud Tasks).
|
||||
|
||||
Returns AuthResult if token is a valid Google OIDC JWT from an allowed
|
||||
service account, None if the token is not a JWT or validation fails.
|
||||
"""
|
||||
# Quick check: JWTs have 3 dot-separated parts
|
||||
if token.count('.') != 2:
|
||||
return None
|
||||
|
||||
try:
|
||||
# Verify the token signature and claims with Google
|
||||
# audience=None skips audience check — Cloud Scheduler sets audience
|
||||
# to the endpoint URL, which varies per endpoint
|
||||
claims = google_id_token.verify_oauth2_token(
|
||||
token,
|
||||
google_auth_requests.Request(),
|
||||
audience=None,
|
||||
)
|
||||
|
||||
token_email = claims.get("email", "")
|
||||
|
||||
# Only accept tokens from the configured service account
|
||||
if token_email != self._scheduler_service_account:
|
||||
logger.warning(
|
||||
f"OIDC token from unexpected service account: {token_email} "
|
||||
f"(expected {self._scheduler_service_account})"
|
||||
)
|
||||
return None
|
||||
|
||||
logger.info(f"OIDC token validated for service account: {token_email}")
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=-1,
|
||||
message="OIDC service account access granted",
|
||||
user_email=token_email,
|
||||
is_admin=True,
|
||||
)
|
||||
|
||||
except ValueError as e:
|
||||
# verify_oauth2_token raises ValueError for invalid/expired tokens
|
||||
logger.debug(f"OIDC token validation failed: {e}")
|
||||
return None
|
||||
except Exception as e:
|
||||
# Network errors fetching Google certs, etc.
|
||||
logger.warning(f"OIDC token verification error: {e}")
|
||||
return None
|
||||
|
||||
def validate_token(self, token: str) -> Tuple[bool, UserType, int, str]:
|
||||
"""
|
||||
@@ -212,6 +268,12 @@ class AuthService:
|
||||
is_admin=True
|
||||
)
|
||||
|
||||
# Check for Google OIDC tokens (Cloud Scheduler / Cloud Tasks)
|
||||
# Done before Firestore lookup to avoid wasting a read on JWT strings
|
||||
oidc_result = self._validate_oidc_token(token)
|
||||
if oidc_result is not None:
|
||||
return oidc_result
|
||||
|
||||
# Check stored tokens in Firestore (auth_tokens collection)
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
@@ -284,6 +346,7 @@ class AuthService:
|
||||
# All auth_tokens must have an associated user_email for job ownership
|
||||
token_user_email = token_data.get("user_email")
|
||||
api_key_id = token_data.get("api_key_id")
|
||||
token_tenant_id = token_data.get("tenant_id")
|
||||
|
||||
# Require user_email on all auth_tokens (no anonymous token auth)
|
||||
if not token_user_email:
|
||||
@@ -307,7 +370,8 @@ class AuthService:
|
||||
message="Unlimited access granted",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
)
|
||||
|
||||
# LIMITED tokens: check usage count
|
||||
@@ -320,7 +384,8 @@ class AuthService:
|
||||
message="Limited token with unlimited uses",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
)
|
||||
|
||||
current_uses = token_data.get("usage_count", 0)
|
||||
@@ -333,7 +398,8 @@ class AuthService:
|
||||
remaining_uses=0,
|
||||
message="Token usage limit exceeded",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
@@ -343,7 +409,8 @@ class AuthService:
|
||||
message=f"Limited token: {remaining} uses remaining",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
)
|
||||
|
||||
# STRIPE tokens: check expiration and usage
|
||||
@@ -357,7 +424,8 @@ class AuthService:
|
||||
remaining_uses=0,
|
||||
message="Token has expired",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
)
|
||||
|
||||
if max_uses > 0:
|
||||
@@ -371,7 +439,8 @@ class AuthService:
|
||||
remaining_uses=0,
|
||||
message="Token usage limit exceeded",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
@@ -381,7 +450,8 @@ class AuthService:
|
||||
message=f"Stripe token: {remaining} uses remaining",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
@@ -391,7 +461,8 @@ class AuthService:
|
||||
message="Stripe access granted",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
)
|
||||
|
||||
# API_KEY tokens
|
||||
@@ -406,7 +477,8 @@ class AuthService:
|
||||
remaining_uses=0,
|
||||
message="API key usage limit exceeded",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
)
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
@@ -415,7 +487,8 @@ class AuthService:
|
||||
message=f"API key valid: {remaining} uses remaining",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
@@ -425,7 +498,8 @@ class AuthService:
|
||||
message="API key access granted",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
@@ -437,10 +511,11 @@ class AuthService:
|
||||
|
||||
def increment_token_usage(self, token: str, job_id: str) -> bool:
|
||||
"""
|
||||
Increment usage count for a token and track the job.
|
||||
Increment usage count for an auth_token and track the job.
|
||||
|
||||
For session tokens (magic link auth), this deducts a credit from the user.
|
||||
For auth_tokens, this increments the usage count.
|
||||
Note: Credit deduction for session-based users is handled in
|
||||
JobManager.create_job(), not here. This method only tracks usage
|
||||
for legacy auth_tokens stored in Firestore.
|
||||
|
||||
Args:
|
||||
token: The access token
|
||||
@@ -465,32 +540,11 @@ class AuthService:
|
||||
if token in self.admin_tokens:
|
||||
return True
|
||||
|
||||
# Check if this is a session token (STRIPE type from validate_token means it's a session)
|
||||
# Session tokens are not in auth_tokens collection
|
||||
# Only increment usage for auth_tokens in Firestore
|
||||
# Session tokens (magic link auth) have credits managed by JobManager.create_job()
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
if not token_data:
|
||||
# This is a session token - deduct credit from user
|
||||
from backend.services.user_service import get_user_service
|
||||
user_service = get_user_service()
|
||||
|
||||
# Get the user email from the session
|
||||
is_valid, user, _ = user_service.validate_session(token)
|
||||
if not is_valid or not user:
|
||||
logger.error(f"Session token validation failed during usage increment")
|
||||
return False
|
||||
|
||||
# Deduct one credit
|
||||
success, new_balance, deduct_message = user_service.deduct_credit(
|
||||
user.email, job_id, reason="job_creation"
|
||||
)
|
||||
|
||||
if success:
|
||||
logger.info(f"Deducted credit for user {user.email} (remaining: {new_balance})")
|
||||
return True
|
||||
else:
|
||||
logger.error(f"Failed to deduct credit for user {user.email}: {deduct_message}")
|
||||
return False
|
||||
return True
|
||||
|
||||
# Regular auth_token - increment usage in Firestore
|
||||
try:
|
||||
|
||||
@@ -0,0 +1,225 @@
|
||||
"""
|
||||
Brand code allocation service using Firestore atomic counters.
|
||||
|
||||
Replaces the previous Dropbox folder-scanning approach which was susceptible to
|
||||
TOCTOU race conditions when concurrent jobs read the same folder state.
|
||||
|
||||
Uses Firestore transactions to atomically allocate unique brand codes.
|
||||
Supports recycling numbers when E2E cleanup deletes test brand codes.
|
||||
"""
|
||||
import logging
|
||||
import re
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional
|
||||
|
||||
from google.cloud import firestore
|
||||
|
||||
from backend.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
BRAND_CODE_COUNTERS_COLLECTION = "brand_code_counters"
|
||||
|
||||
# Singleton instance
|
||||
_brand_code_service: Optional["BrandCodeService"] = None
|
||||
|
||||
|
||||
class BrandCodeService:
|
||||
"""Atomic brand code allocation using Firestore counters."""
|
||||
|
||||
def __init__(self, db: Optional[firestore.Client] = None):
|
||||
if db is None:
|
||||
self.db = firestore.Client(project=settings.google_cloud_project)
|
||||
else:
|
||||
self.db = db
|
||||
|
||||
def allocate_brand_code(self, prefix: str, dropbox_path: str) -> str:
|
||||
"""
|
||||
Atomically allocate the next brand code for a given prefix.
|
||||
|
||||
Uses a Firestore transaction to ensure no two concurrent callers
|
||||
receive the same number. If recycled numbers are available (from
|
||||
E2E cleanup), the smallest recycled number is reused first.
|
||||
|
||||
On first use for a prefix, initializes the counter by scanning
|
||||
Dropbox to find the current max number.
|
||||
|
||||
Args:
|
||||
prefix: Brand prefix (e.g., "NOMAD" or "NOMADNP")
|
||||
dropbox_path: Dropbox path for one-time initialization scan
|
||||
|
||||
Returns:
|
||||
Brand code string (e.g., "NOMAD-1264")
|
||||
"""
|
||||
doc_ref = self.db.collection(BRAND_CODE_COUNTERS_COLLECTION).document(prefix)
|
||||
|
||||
@firestore.transactional
|
||||
def allocate_in_transaction(transaction):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
|
||||
if not doc.exists:
|
||||
# First use — initialize from Dropbox scan
|
||||
next_number = self._get_initial_next_number(prefix, dropbox_path)
|
||||
allocated = next_number
|
||||
transaction.set(doc_ref, {
|
||||
"prefix": prefix,
|
||||
"next_number": next_number + 1,
|
||||
"recycled": [],
|
||||
"initialized_at": datetime.now(timezone.utc),
|
||||
"last_updated": datetime.now(timezone.utc),
|
||||
})
|
||||
return allocated
|
||||
|
||||
data = doc.to_dict()
|
||||
recycled = data.get("recycled", [])
|
||||
|
||||
if recycled:
|
||||
# Reuse smallest recycled number
|
||||
recycled.sort()
|
||||
allocated = recycled.pop(0)
|
||||
transaction.update(doc_ref, {
|
||||
"recycled": recycled,
|
||||
"last_updated": datetime.now(timezone.utc),
|
||||
})
|
||||
else:
|
||||
# Allocate next sequential number
|
||||
allocated = data["next_number"]
|
||||
transaction.update(doc_ref, {
|
||||
"next_number": allocated + 1,
|
||||
"last_updated": datetime.now(timezone.utc),
|
||||
})
|
||||
|
||||
return allocated
|
||||
|
||||
transaction = self.db.transaction()
|
||||
number = allocate_in_transaction(transaction)
|
||||
brand_code = f"{prefix}-{number:04d}"
|
||||
logger.info(f"Allocated brand code: {brand_code}")
|
||||
return brand_code
|
||||
|
||||
def recycle_brand_code(self, prefix: str, number: int) -> None:
|
||||
"""
|
||||
Return a brand code number to the recycled pool.
|
||||
|
||||
Called when E2E cleanup deletes a brand code's distribution.
|
||||
The number will be reused on the next allocation for this prefix.
|
||||
|
||||
Args:
|
||||
prefix: Brand prefix (e.g., "NOMAD")
|
||||
number: The numeric portion to recycle (e.g., 1234)
|
||||
"""
|
||||
doc_ref = self.db.collection(BRAND_CODE_COUNTERS_COLLECTION).document(prefix)
|
||||
|
||||
@firestore.transactional
|
||||
def recycle_in_transaction(transaction):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
|
||||
if not doc.exists:
|
||||
# Counter doesn't exist yet — nothing to recycle into.
|
||||
# The number will naturally be below next_number when
|
||||
# the counter is initialized, so it would create a gap.
|
||||
# Just log and skip.
|
||||
logger.warning(
|
||||
f"Cannot recycle {prefix}-{number:04d}: counter doc doesn't exist"
|
||||
)
|
||||
return
|
||||
|
||||
data = doc.to_dict()
|
||||
next_number = data.get("next_number", 0)
|
||||
|
||||
if number <= 0 or number >= next_number:
|
||||
logger.warning(
|
||||
f"Cannot recycle {prefix}-{number:04d}: "
|
||||
f"invalid (next_number={next_number})"
|
||||
)
|
||||
return
|
||||
|
||||
recycled = data.get("recycled", [])
|
||||
|
||||
if number not in recycled:
|
||||
recycled.append(number)
|
||||
transaction.update(doc_ref, {
|
||||
"recycled": recycled,
|
||||
"last_updated": datetime.now(timezone.utc),
|
||||
})
|
||||
logger.info(f"Recycled brand code number: {prefix}-{number:04d}")
|
||||
else:
|
||||
logger.warning(f"Number {number} already in recycled pool for {prefix}")
|
||||
|
||||
transaction = self.db.transaction()
|
||||
recycle_in_transaction(transaction)
|
||||
|
||||
def _get_initial_next_number(self, prefix: str, dropbox_path: str) -> int:
|
||||
"""
|
||||
Scan Dropbox once to determine the starting next_number.
|
||||
|
||||
This is only called when the counter document doesn't exist yet
|
||||
(one-time migration from the old Dropbox-scanning approach).
|
||||
|
||||
Args:
|
||||
prefix: Brand prefix to scan for
|
||||
dropbox_path: Dropbox path containing organized folders
|
||||
|
||||
Returns:
|
||||
The next number to allocate (max existing + 1, minimum 1)
|
||||
"""
|
||||
try:
|
||||
from backend.services.dropbox_service import get_dropbox_service
|
||||
|
||||
dropbox = get_dropbox_service()
|
||||
if not dropbox.is_configured:
|
||||
logger.warning(
|
||||
f"Dropbox not configured for initialization of {prefix} counter, starting at 1"
|
||||
)
|
||||
return 1
|
||||
|
||||
folders = dropbox.list_folders(dropbox_path)
|
||||
pattern = re.compile(rf"^{re.escape(prefix)}-(\d{{4}})")
|
||||
|
||||
existing_nums: set[int] = set()
|
||||
for folder in folders:
|
||||
match = pattern.match(folder)
|
||||
if match:
|
||||
existing_nums.add(int(match.group(1)))
|
||||
|
||||
if existing_nums:
|
||||
next_num = max(existing_nums) + 1
|
||||
logger.info(
|
||||
f"Initialized {prefix} counter from Dropbox: "
|
||||
f"max existing={max(existing_nums)}, next_number={next_num}"
|
||||
)
|
||||
return next_num
|
||||
|
||||
logger.info(f"No existing {prefix} codes found in Dropbox, starting at 1")
|
||||
return 1
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to initialize {prefix} counter from Dropbox: {e}")
|
||||
raise
|
||||
|
||||
@staticmethod
|
||||
def parse_brand_code(brand_code: str) -> tuple[str, int]:
|
||||
"""
|
||||
Parse a brand code string into prefix and number.
|
||||
|
||||
Args:
|
||||
brand_code: e.g., "NOMAD-1234" or "NOMADNP-0012"
|
||||
|
||||
Returns:
|
||||
Tuple of (prefix, number), e.g., ("NOMAD", 1234)
|
||||
|
||||
Raises:
|
||||
ValueError: If brand_code doesn't match expected format
|
||||
"""
|
||||
match = re.match(r"^([A-Z]+)-(\d+)$", brand_code)
|
||||
if not match:
|
||||
raise ValueError(f"Invalid brand code format: {brand_code}")
|
||||
return match.group(1), int(match.group(2))
|
||||
|
||||
|
||||
def get_brand_code_service() -> BrandCodeService:
|
||||
"""Get or create the singleton BrandCodeService instance."""
|
||||
global _brand_code_service
|
||||
if _brand_code_service is None:
|
||||
_brand_code_service = BrandCodeService()
|
||||
return _brand_code_service
|
||||
@@ -0,0 +1,105 @@
|
||||
"""
|
||||
Proxy service for karaoke-decide catalog API.
|
||||
|
||||
Forwards artist and track search requests to the karaoke-decide backend,
|
||||
providing autocomplete data from MusicBrainz + Spotify catalogs.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
from backend.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# In-memory TTL cache: { cache_key: (expiry_timestamp, data) }
|
||||
_cache: dict[str, tuple[float, Any]] = {}
|
||||
CACHE_TTL_SECONDS = 300 # 5 minutes
|
||||
|
||||
KARAOKE_DECIDE_BASE_URL = "https://decide.nomadkaraoke.com"
|
||||
REQUEST_TIMEOUT = 10 # seconds
|
||||
|
||||
|
||||
def _get_base_url() -> str:
|
||||
"""Get the karaoke-decide API base URL from config or default."""
|
||||
return getattr(settings, "karaoke_decide_api_url", None) or KARAOKE_DECIDE_BASE_URL
|
||||
|
||||
|
||||
def _cache_get(key: str) -> Any | None:
|
||||
"""Get a value from cache if it exists and hasn't expired."""
|
||||
if key in _cache:
|
||||
expiry, data = _cache[key]
|
||||
if time.monotonic() < expiry:
|
||||
return data
|
||||
del _cache[key]
|
||||
return None
|
||||
|
||||
|
||||
def _cache_set(key: str, data: Any, ttl: float = CACHE_TTL_SECONDS) -> None:
|
||||
"""Store a value in cache with TTL."""
|
||||
_cache[key] = (time.monotonic() + ttl, data)
|
||||
|
||||
|
||||
async def search_artists(query: str, limit: int = 10) -> list[dict]:
|
||||
"""
|
||||
Search for artists via karaoke-decide catalog API.
|
||||
|
||||
Returns list of artist dicts with canonical MusicBrainz names.
|
||||
Falls back to empty list on any error.
|
||||
"""
|
||||
cache_key = f"artists:{query.lower()}:{limit}"
|
||||
cached = _cache_get(cache_key)
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
base_url = _get_base_url()
|
||||
url = f"{base_url}/api/catalog/artists"
|
||||
params = {"q": query, "limit": limit}
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=REQUEST_TIMEOUT) as client:
|
||||
resp = await client.get(url, params=params)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
# karaoke-decide wraps results: {"artists": [...], "total": N}
|
||||
results = data.get("artists", data) if isinstance(data, dict) else data
|
||||
_cache_set(cache_key, results)
|
||||
return results
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to search artists from karaoke-decide: {e}")
|
||||
return []
|
||||
|
||||
|
||||
async def search_tracks(query: str, artist: str | None = None, limit: int = 10) -> list[dict]:
|
||||
"""
|
||||
Search for tracks via karaoke-decide catalog API.
|
||||
|
||||
Returns list of track dicts with canonical Spotify/MusicBrainz names.
|
||||
Falls back to empty list on any error.
|
||||
"""
|
||||
cache_key = f"tracks:{query.lower()}:{(artist or '').lower()}:{limit}"
|
||||
cached = _cache_get(cache_key)
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
base_url = _get_base_url()
|
||||
url = f"{base_url}/api/catalog/tracks"
|
||||
params: dict[str, Any] = {"q": query, "limit": limit}
|
||||
if artist:
|
||||
params["artist"] = artist
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=REQUEST_TIMEOUT) as client:
|
||||
resp = await client.get(url, params=params)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
# karaoke-decide wraps results: {"tracks": [...], "total": N}
|
||||
results = data.get("tracks", data) if isinstance(data, dict) else data
|
||||
_cache_set(cache_key, results)
|
||||
return results
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to search tracks from karaoke-decide: {e}")
|
||||
return []
|
||||
@@ -0,0 +1,441 @@
|
||||
"""
|
||||
AI-powered credit evaluation service.
|
||||
|
||||
Evaluates whether to grant free credits (welcome or feedback) by collecting
|
||||
abuse signals and asking Gemini for a grant/deny decision. Fail-open: if
|
||||
the AI call fails for any reason, credits are granted.
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import uuid
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
CREDIT_EVALUATIONS_COLLECTION = "credit_evaluations"
|
||||
|
||||
|
||||
@dataclass
|
||||
class CreditEvaluation:
|
||||
"""Result of an AI credit evaluation."""
|
||||
decision: str # "grant" or "deny"
|
||||
reasoning: str
|
||||
confidence: float
|
||||
error: Optional[str] = None # Set if evaluation failed (decision defaults to "grant")
|
||||
signals: Optional[dict] = None # Collected abuse signals (for admin review emails)
|
||||
|
||||
|
||||
SYSTEM_PROMPT = """You are an anti-abuse evaluator for Nomad Karaoke, a web service that creates professional karaoke videos. Each job costs the service real money in API credits and cloud compute.
|
||||
|
||||
New users get 1 free welcome credit to try the service. Users who complete 2 jobs and submit feedback get 1 more credit. Some users abuse this by creating multiple accounts to get unlimited free karaoke generation.
|
||||
|
||||
Your job: Given the signals below, decide whether to GRANT or DENY free credits to this user.
|
||||
|
||||
## Decision Guidelines
|
||||
|
||||
**DENY if any of these are true:**
|
||||
- The user's device fingerprint matches other accounts (strong signal of multi-accounting)
|
||||
- The user's signup IP matches other accounts AND those accounts also have suspicious patterns (e.g., no spend, same user agent)
|
||||
- There's a clear pattern of account creation from the same device/network to farm free credits
|
||||
|
||||
**GRANT if:**
|
||||
- This appears to be a genuine new user with no suspicious correlations
|
||||
- The IP match is explainable (e.g., different fingerprints suggest different actual users on a shared network like a school or office)
|
||||
- There's insufficient evidence to confidently flag abuse
|
||||
|
||||
**When in doubt, GRANT.** False positives (blocking real users) are worse than false negatives (letting an occasional abuser through). We'd rather lose a few dollars than alienate a real customer.
|
||||
|
||||
## Response Format
|
||||
|
||||
Respond with ONLY a JSON object, no other text:
|
||||
{"decision": "grant" or "deny", "reasoning": "Brief explanation", "confidence": 0.0 to 1.0}
|
||||
"""
|
||||
|
||||
|
||||
def _build_user_prompt(
|
||||
email: str,
|
||||
grant_type: str,
|
||||
user_data: dict,
|
||||
fingerprint_matches: list,
|
||||
ip_matches: list,
|
||||
ip_geo: Optional[dict],
|
||||
recent_signups_ip: int,
|
||||
recent_signups_fp: int,
|
||||
user_agent: Optional[str],
|
||||
feedback_content: Optional[dict] = None,
|
||||
) -> str:
|
||||
"""Build the user prompt with all abuse signals."""
|
||||
lines = [
|
||||
f"## Credit Grant Request",
|
||||
f"- **Email:** {email}",
|
||||
f"- **Grant type:** {grant_type} credits",
|
||||
f"- **User's device fingerprint:** {user_data.get('device_fingerprint', 'unknown')}",
|
||||
f"- **User's signup IP:** {user_data.get('signup_ip', 'unknown')}",
|
||||
f"- **User agent:** {user_agent or 'unknown'}",
|
||||
f"- **Account created:** {user_data.get('created_at', 'unknown')}",
|
||||
f"- **Jobs created:** {user_data.get('total_jobs_created', 0)}",
|
||||
f"- **Jobs completed:** {user_data.get('total_jobs_completed', 0)}",
|
||||
f"- **Total spent (real money):** ${user_data.get('total_spent', 0) / 100:.2f}",
|
||||
f"- **Current credits:** {user_data.get('credits', 0)}",
|
||||
"",
|
||||
]
|
||||
|
||||
# IP geolocation
|
||||
if ip_geo and ip_geo.get("status") == "success":
|
||||
lines.extend([
|
||||
f"## IP Geolocation",
|
||||
f"- **Country:** {ip_geo.get('country', 'unknown')} ({ip_geo.get('country_code', '')})",
|
||||
f"- **City/Region:** {ip_geo.get('city', '')}, {ip_geo.get('region', '')}",
|
||||
f"- **ISP:** {ip_geo.get('isp', 'unknown')}",
|
||||
f"- **Organization:** {ip_geo.get('org', 'unknown')}",
|
||||
f"- **ASN:** {ip_geo.get('as_number', '')} {ip_geo.get('as_name', '')}",
|
||||
"",
|
||||
])
|
||||
|
||||
# Recent signup patterns
|
||||
lines.extend([
|
||||
f"## Recent Signup Patterns (last 24 hours)",
|
||||
f"- **New accounts from same IP:** {recent_signups_ip}",
|
||||
f"- **New accounts from same fingerprint:** {recent_signups_fp}",
|
||||
"",
|
||||
])
|
||||
|
||||
# Fingerprint correlations
|
||||
if fingerprint_matches:
|
||||
lines.append("## Other Accounts with Same Device Fingerprint (RED FLAG)")
|
||||
for match in fingerprint_matches:
|
||||
lines.append(
|
||||
f"- {match.get('email', '?')} | "
|
||||
f"Jobs: {match.get('total_jobs_created', 0)} | "
|
||||
f"Spent: ${match.get('total_spent', 0) / 100:.2f} | "
|
||||
f"Created: {match.get('created_at', '?')}"
|
||||
)
|
||||
lines.append("")
|
||||
else:
|
||||
lines.extend(["## Other Accounts with Same Device Fingerprint", "None found.", ""])
|
||||
|
||||
# IP correlations
|
||||
if ip_matches:
|
||||
lines.append("## Other Accounts from Same Signup IP")
|
||||
for match in ip_matches:
|
||||
lines.append(
|
||||
f"- {match.get('email', '?')} | "
|
||||
f"Fingerprint: {match.get('device_fingerprint', 'unknown')} | "
|
||||
f"Jobs: {match.get('total_jobs_created', 0)} | "
|
||||
f"Spent: ${match.get('total_spent', 0) / 100:.2f} | "
|
||||
f"Created: {match.get('created_at', '?')}"
|
||||
)
|
||||
lines.append("")
|
||||
else:
|
||||
lines.extend(["## Other Accounts from Same Signup IP", "None found.", ""])
|
||||
|
||||
# Feedback content (if applicable)
|
||||
if feedback_content:
|
||||
lines.extend([
|
||||
"## Feedback Submitted",
|
||||
f"- **Overall rating:** {feedback_content.get('overall_rating', '?')}/5",
|
||||
f"- **What went well:** {feedback_content.get('what_went_well', 'N/A')[:200]}",
|
||||
f"- **What could improve:** {feedback_content.get('what_could_improve', 'N/A')[:200]}",
|
||||
f"- **Additional comments:** {feedback_content.get('additional_comments', 'N/A')[:200]}",
|
||||
"",
|
||||
])
|
||||
|
||||
lines.append("Based on these signals, should we GRANT or DENY free credits to this user?")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _parse_gemini_response(response_text: str) -> CreditEvaluation:
|
||||
"""Parse Gemini's JSON response into a CreditEvaluation."""
|
||||
try:
|
||||
# Strip markdown code fences if present
|
||||
text = response_text.strip()
|
||||
if text.startswith("```"):
|
||||
text = text.split("\n", 1)[1] if "\n" in text else text[3:]
|
||||
if text.endswith("```"):
|
||||
text = text.rsplit("```", 1)[0]
|
||||
text = text.strip()
|
||||
if text.startswith("json"):
|
||||
text = text[4:].strip()
|
||||
|
||||
data = json.loads(text)
|
||||
decision = data.get("decision", "").lower()
|
||||
if decision not in ("grant", "deny"):
|
||||
decision = "pending_review" # fail-closed: unknown decision → manual review
|
||||
|
||||
return CreditEvaluation(
|
||||
decision=decision,
|
||||
reasoning=str(data.get("reasoning", "No reasoning provided")),
|
||||
confidence=float(data.get("confidence", 0.5)),
|
||||
)
|
||||
except (json.JSONDecodeError, ValueError, TypeError) as e:
|
||||
logger.warning(f"Failed to parse Gemini response: {e}. Response: {response_text[:200]}")
|
||||
return CreditEvaluation(
|
||||
decision="pending_review",
|
||||
reasoning="Failed to parse AI response — pending manual review",
|
||||
confidence=0.0,
|
||||
error=f"Parse error: {e}",
|
||||
)
|
||||
|
||||
|
||||
class CreditEvaluationService:
|
||||
"""Evaluates credit grant requests using Gemini AI."""
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
|
||||
def _collect_signals(self, email: str) -> dict:
|
||||
"""Collect all abuse signals for evaluation."""
|
||||
from backend.services.user_service import get_user_service
|
||||
from backend.services.ip_geolocation_service import get_ip_geolocation_service
|
||||
|
||||
user_service = get_user_service()
|
||||
user = user_service.get_user(email)
|
||||
|
||||
if not user:
|
||||
return {
|
||||
"user_data": {},
|
||||
"fingerprint_matches": [],
|
||||
"ip_matches": [],
|
||||
"ip_geo": None,
|
||||
"recent_signups_ip": 0,
|
||||
"recent_signups_fp": 0,
|
||||
"user_agent": None,
|
||||
}
|
||||
|
||||
user_data = {
|
||||
"email": user.email,
|
||||
"device_fingerprint": user.device_fingerprint,
|
||||
"signup_ip": user.signup_ip,
|
||||
"credits": user.credits,
|
||||
"total_jobs_created": user.total_jobs_created,
|
||||
"total_jobs_completed": user.total_jobs_completed,
|
||||
"total_spent": user.total_spent,
|
||||
"created_at": str(user.created_at),
|
||||
}
|
||||
|
||||
# Find other accounts with same fingerprint
|
||||
fingerprint_matches = []
|
||||
if user.device_fingerprint:
|
||||
fp_users = user_service.find_users_by_fingerprint(user.device_fingerprint)
|
||||
fingerprint_matches = [
|
||||
{
|
||||
"email": u.email,
|
||||
"total_jobs_created": u.total_jobs_created,
|
||||
"total_spent": u.total_spent,
|
||||
"created_at": str(u.created_at),
|
||||
}
|
||||
for u in fp_users if u.email != email
|
||||
]
|
||||
|
||||
# Find other accounts from same IP
|
||||
ip_matches = []
|
||||
if user.signup_ip:
|
||||
ip_users = user_service.find_users_by_signup_ip(user.signup_ip)
|
||||
ip_matches = [
|
||||
{
|
||||
"email": u.email,
|
||||
"device_fingerprint": u.device_fingerprint,
|
||||
"total_jobs_created": u.total_jobs_created,
|
||||
"total_spent": u.total_spent,
|
||||
"created_at": str(u.created_at),
|
||||
}
|
||||
for u in ip_users if u.email != email
|
||||
]
|
||||
|
||||
# IP geolocation
|
||||
ip_geo = None
|
||||
if user.signup_ip:
|
||||
try:
|
||||
geo_service = get_ip_geolocation_service()
|
||||
ip_geo = geo_service.lookup_ip(user.signup_ip)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Recent signup counts
|
||||
recent_signups_ip = 0
|
||||
recent_signups_fp = 0
|
||||
if user.signup_ip:
|
||||
recent_signups_ip = user_service.count_recent_signups_from_ip(user.signup_ip)
|
||||
if user.device_fingerprint:
|
||||
recent_signups_fp = user_service.count_recent_signups_from_fingerprint(user.device_fingerprint)
|
||||
|
||||
# User agent from latest session
|
||||
user_agent = None
|
||||
try:
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
from google.cloud import firestore as fs
|
||||
sessions = (
|
||||
user_service.db.collection("sessions")
|
||||
.where(filter=FieldFilter("user_email", "==", email))
|
||||
.order_by("created_at", direction=fs.Query.DESCENDING)
|
||||
.limit(1)
|
||||
.stream()
|
||||
)
|
||||
for session_doc in sessions:
|
||||
user_agent = session_doc.to_dict().get("user_agent")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return {
|
||||
"user_data": user_data,
|
||||
"fingerprint_matches": fingerprint_matches,
|
||||
"ip_matches": ip_matches,
|
||||
"ip_geo": ip_geo,
|
||||
"recent_signups_ip": recent_signups_ip,
|
||||
"recent_signups_fp": recent_signups_fp,
|
||||
"user_agent": user_agent,
|
||||
}
|
||||
|
||||
def _call_gemini(self, prompt: str) -> str:
|
||||
"""Call Gemini and return the response text."""
|
||||
from google import genai
|
||||
|
||||
client = genai.Client(
|
||||
vertexai=True,
|
||||
project=self.settings.google_cloud_project,
|
||||
location="global",
|
||||
)
|
||||
response = client.models.generate_content(
|
||||
model=self.settings.credit_eval_model,
|
||||
contents=prompt,
|
||||
config=genai.types.GenerateContentConfig(
|
||||
system_instruction=SYSTEM_PROMPT,
|
||||
),
|
||||
)
|
||||
return response.text
|
||||
|
||||
def _log_evaluation(
|
||||
self,
|
||||
email: str,
|
||||
grant_type: str,
|
||||
evaluation: CreditEvaluation,
|
||||
signals: dict,
|
||||
):
|
||||
"""Log the evaluation decision to Firestore for audit trail."""
|
||||
try:
|
||||
from google.cloud import firestore as fs
|
||||
db = fs.Client(project=self.settings.google_cloud_project)
|
||||
doc_id = str(uuid.uuid4())
|
||||
db.collection(CREDIT_EVALUATIONS_COLLECTION).document(doc_id).set({
|
||||
"id": doc_id,
|
||||
"email": email,
|
||||
"grant_type": grant_type,
|
||||
"decision": evaluation.decision,
|
||||
"reasoning": evaluation.reasoning,
|
||||
"confidence": evaluation.confidence,
|
||||
"error": evaluation.error,
|
||||
"model": self.settings.credit_eval_model,
|
||||
"signals_snapshot": {
|
||||
"fingerprint_match_count": len(signals.get("fingerprint_matches", [])),
|
||||
"ip_match_count": len(signals.get("ip_matches", [])),
|
||||
"recent_signups_ip": signals.get("recent_signups_ip", 0),
|
||||
"recent_signups_fp": signals.get("recent_signups_fp", 0),
|
||||
"user_ip": signals.get("user_data", {}).get("signup_ip"),
|
||||
"user_fingerprint": signals.get("user_data", {}).get("device_fingerprint"),
|
||||
},
|
||||
"evaluated_at": datetime.now(timezone.utc).isoformat(),
|
||||
})
|
||||
except Exception:
|
||||
logger.exception(f"Failed to log credit evaluation for {email}")
|
||||
|
||||
def evaluate(
|
||||
self,
|
||||
email: str,
|
||||
grant_type: str,
|
||||
feedback_content: Optional[dict] = None,
|
||||
) -> CreditEvaluation:
|
||||
"""
|
||||
Evaluate whether to grant free credits.
|
||||
|
||||
Args:
|
||||
email: User's email
|
||||
grant_type: "welcome" or "feedback"
|
||||
feedback_content: Feedback form data (for feedback grants only)
|
||||
|
||||
Returns:
|
||||
CreditEvaluation with decision, reasoning, confidence.
|
||||
Fail-closed: returns "pending_review" on any error (admin notified).
|
||||
"""
|
||||
if not self.settings.credit_eval_enabled:
|
||||
return CreditEvaluation(
|
||||
decision="grant",
|
||||
reasoning="Credit evaluation disabled — auto-granting",
|
||||
confidence=1.0,
|
||||
)
|
||||
|
||||
try:
|
||||
# Collect signals
|
||||
signals = self._collect_signals(email)
|
||||
|
||||
# Quick-grant: if no correlations found at all, skip AI call
|
||||
if (
|
||||
not signals["fingerprint_matches"]
|
||||
and not signals["ip_matches"]
|
||||
and signals["recent_signups_ip"] <= 1
|
||||
and signals["recent_signups_fp"] <= 1
|
||||
):
|
||||
evaluation = CreditEvaluation(
|
||||
decision="grant",
|
||||
reasoning="No suspicious correlations found — clean user",
|
||||
confidence=1.0,
|
||||
signals=signals,
|
||||
)
|
||||
self._log_evaluation(email, grant_type, evaluation, signals)
|
||||
return evaluation
|
||||
|
||||
# Build prompt and call Gemini
|
||||
prompt = _build_user_prompt(
|
||||
email=email,
|
||||
grant_type=grant_type,
|
||||
user_data=signals["user_data"],
|
||||
fingerprint_matches=signals["fingerprint_matches"],
|
||||
ip_matches=signals["ip_matches"],
|
||||
ip_geo=signals["ip_geo"],
|
||||
recent_signups_ip=signals["recent_signups_ip"],
|
||||
recent_signups_fp=signals["recent_signups_fp"],
|
||||
user_agent=signals["user_agent"],
|
||||
feedback_content=feedback_content,
|
||||
)
|
||||
|
||||
response_text = self._call_gemini(prompt)
|
||||
evaluation = _parse_gemini_response(response_text)
|
||||
evaluation.signals = signals
|
||||
|
||||
self._log_evaluation(email, grant_type, evaluation, signals)
|
||||
logger.info(
|
||||
f"Credit evaluation for {email} ({grant_type}): "
|
||||
f"{evaluation.decision} (confidence={evaluation.confidence:.2f})"
|
||||
)
|
||||
return evaluation
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"Credit evaluation failed for {email} — pending manual review (fail-closed)")
|
||||
# Include whatever signals were collected before the failure
|
||||
collected_signals = locals().get("signals")
|
||||
evaluation = CreditEvaluation(
|
||||
decision="pending_review",
|
||||
reasoning="Evaluation failed — pending manual review",
|
||||
confidence=0.0,
|
||||
error=str(e),
|
||||
signals=collected_signals,
|
||||
)
|
||||
try:
|
||||
self._log_evaluation(email, grant_type, evaluation, collected_signals or {})
|
||||
except Exception:
|
||||
pass
|
||||
return evaluation
|
||||
|
||||
|
||||
_service_instance: Optional[CreditEvaluationService] = None
|
||||
|
||||
|
||||
def get_credit_evaluation_service() -> CreditEvaluationService:
|
||||
"""Get or create the singleton credit evaluation service."""
|
||||
global _service_instance
|
||||
if _service_instance is None:
|
||||
_service_instance = CreditEvaluationService()
|
||||
return _service_instance
|
||||
@@ -0,0 +1,129 @@
|
||||
"""
|
||||
Disposable Domain Sync Service.
|
||||
|
||||
Fetches and syncs disposable email domain blocklists from external sources.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Set
|
||||
|
||||
import httpx
|
||||
|
||||
from backend.services.email_validation_service import (
|
||||
BLOCKLISTS_COLLECTION,
|
||||
BLOCKLIST_CONFIG_DOC,
|
||||
DEFAULT_DISPOSABLE_DOMAINS,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
EXTERNAL_BLOCKLIST_URL = (
|
||||
"https://raw.githubusercontent.com/disposable-email-domains/"
|
||||
"disposable-email-domains/refs/heads/main/disposable_email_blocklist.conf"
|
||||
)
|
||||
FETCH_TIMEOUT_SECONDS = 30
|
||||
MAX_RESPONSE_BYTES = 2 * 1024 * 1024 # 2 MB
|
||||
MAX_DOMAIN_COUNT = 50_000
|
||||
|
||||
|
||||
def parse_blocklist_text(text: str) -> Set[str]:
|
||||
"""Parse newline-delimited blocklist text into a set of lowercase domains."""
|
||||
domains = set()
|
||||
for line in text.splitlines():
|
||||
domain = line.strip().lower()
|
||||
if domain:
|
||||
domains.add(domain)
|
||||
return domains
|
||||
|
||||
|
||||
async def fetch_external_blocklist() -> Set[str]:
|
||||
"""Fetch the external disposable domain blocklist from GitHub.
|
||||
|
||||
Checks Content-Length header and response body size against safety limits.
|
||||
"""
|
||||
async with httpx.AsyncClient(timeout=FETCH_TIMEOUT_SECONDS) as client:
|
||||
response = await client.get(EXTERNAL_BLOCKLIST_URL)
|
||||
response.raise_for_status()
|
||||
|
||||
content_length = response.headers.get("content-length")
|
||||
if content_length and int(content_length) > MAX_RESPONSE_BYTES:
|
||||
raise ValueError(
|
||||
f"Response size {content_length} exceeds maximum {MAX_RESPONSE_BYTES}"
|
||||
)
|
||||
|
||||
text = response.text
|
||||
if len(text.encode()) > MAX_RESPONSE_BYTES:
|
||||
raise ValueError(
|
||||
f"Response body size exceeds maximum {MAX_RESPONSE_BYTES}"
|
||||
)
|
||||
|
||||
domains = parse_blocklist_text(text)
|
||||
|
||||
if len(domains) > MAX_DOMAIN_COUNT:
|
||||
raise ValueError(
|
||||
f"Domain count {len(domains)} exceeds maximum {MAX_DOMAIN_COUNT}"
|
||||
)
|
||||
|
||||
logger.info(f"Fetched {len(domains)} domains from external blocklist")
|
||||
return domains
|
||||
|
||||
|
||||
def sync_disposable_domains(db, external_domains: Set[str]) -> dict:
|
||||
"""
|
||||
Sync external domains into Firestore.
|
||||
|
||||
Handles first-run migration from old disposable_domains field,
|
||||
and cleans up redundant manual domains on subsequent runs.
|
||||
|
||||
Note: This is a synchronous function (Firestore SDK is sync).
|
||||
Call from async code with `await asyncio.to_thread(sync_disposable_domains, db, domains)`.
|
||||
|
||||
Returns a summary dict with counts.
|
||||
"""
|
||||
doc_ref = db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
doc = doc_ref.get()
|
||||
|
||||
data = doc.to_dict() if doc.exists else {}
|
||||
result = {"external_count": len(external_domains), "added": 0, "removed": 0, "migrated_to_manual": []}
|
||||
|
||||
is_migration = "external_domains" not in data
|
||||
|
||||
if is_migration:
|
||||
old_domains = set(data.get("disposable_domains", [])) | DEFAULT_DISPOSABLE_DOMAINS
|
||||
manual_only = old_domains - external_domains
|
||||
result["migrated_to_manual"] = sorted(manual_only)
|
||||
|
||||
new_data = {
|
||||
**data, # preserve all existing fields
|
||||
"external_domains": sorted(external_domains),
|
||||
"manual_domains": sorted(manual_only),
|
||||
"allowlisted_domains": data.get("allowlisted_domains", []),
|
||||
"last_sync_at": datetime.now(timezone.utc),
|
||||
"last_sync_count": len(external_domains),
|
||||
}
|
||||
new_data.pop("disposable_domains", None) # remove legacy field
|
||||
doc_ref.set(new_data)
|
||||
else:
|
||||
old_external = set(data.get("external_domains", []))
|
||||
manual_domains = set(data.get("manual_domains", []))
|
||||
|
||||
result["added"] = len(external_domains - old_external)
|
||||
result["removed"] = len(old_external - external_domains)
|
||||
|
||||
cleaned_manual = manual_domains - external_domains
|
||||
|
||||
new_data = {
|
||||
**data,
|
||||
"external_domains": sorted(external_domains),
|
||||
"manual_domains": sorted(cleaned_manual),
|
||||
"last_sync_at": datetime.now(timezone.utc),
|
||||
"last_sync_count": len(external_domains),
|
||||
}
|
||||
doc_ref.set(new_data)
|
||||
|
||||
logger.info(
|
||||
f"Sync complete: {result['external_count']} external domains, "
|
||||
f"{result['added']} added, {result['removed']} removed"
|
||||
)
|
||||
return result
|
||||
@@ -111,6 +111,10 @@ class DropboxService:
|
||||
"""
|
||||
Calculate next brand code from existing folders, filling gaps in the sequence.
|
||||
|
||||
NOTE: This method is only used for one-time initialization of the Firestore
|
||||
counter (see brand_code_service.py). All runtime brand code allocation now
|
||||
goes through BrandCodeService.allocate_brand_code() for atomic safety.
|
||||
|
||||
Scans the folder for existing brand codes like "NOMAD-0001" and
|
||||
returns the first available code, filling any gaps left by deleted outputs.
|
||||
|
||||
|
||||
+725
-293
File diff suppressed because it is too large.
Load diff
@@ -2,21 +2,43 @@
|
||||
Email Validation Service.
|
||||
|
||||
Provides email validation, normalization, and blocklist checking
|
||||
to prevent abuse during beta enrollment and other flows.
|
||||
to prevent abuse during enrollment and other flows.
|
||||
|
||||
Uses a tiered disposable domain detection strategy:
|
||||
1. Static blocklist (instant, from Firestore)
|
||||
2. Verified-clean domain cache (skip API calls for 7 days)
|
||||
3. DeBounce API (free, <50ms)
|
||||
4. verifymail.io API (paid, only if DeBounce says clean)
|
||||
|
||||
Auto-learns: flagged domains are persisted to manual_domains,
|
||||
clean domains are cached for 7 days in verified_clean_domains.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import re
|
||||
from datetime import datetime, timezone
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Optional, Tuple, Set
|
||||
|
||||
import httpx
|
||||
from google.cloud import firestore
|
||||
|
||||
from backend.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Module-level HTTP client for connection pooling (external API checks)
|
||||
_http_client: Optional[httpx.Client] = None
|
||||
_HTTP_TIMEOUT = 2.0 # seconds per external API call
|
||||
|
||||
|
||||
def _get_http_client() -> httpx.Client:
|
||||
"""Get or create the module-level HTTP client."""
|
||||
global _http_client
|
||||
if _http_client is None or _http_client.is_closed:
|
||||
_http_client = httpx.Client(timeout=_HTTP_TIMEOUT)
|
||||
return _http_client
|
||||
|
||||
|
||||
def _mask_email(email: str) -> str:
|
||||
"""Mask an email address for privacy-safe logging."""
|
||||
@@ -34,164 +56,12 @@ def _mask_email(email: str) -> str:
|
||||
BLOCKLISTS_COLLECTION = "blocklists"
|
||||
BLOCKLIST_CONFIG_DOC = "config"
|
||||
|
||||
# Default disposable email domains (can be extended via admin UI)
|
||||
DEFAULT_DISPOSABLE_DOMAINS = {
|
||||
# Popular temporary email services
|
||||
"10minutemail.com",
|
||||
"10minutemail.net",
|
||||
"guerrillamail.com",
|
||||
"guerrillamail.org",
|
||||
"guerrillamail.net",
|
||||
"guerrillamail.biz",
|
||||
"guerrillamailblock.com",
|
||||
"tempmail.com",
|
||||
"tempmail.net",
|
||||
"temp-mail.org",
|
||||
"temp-mail.io",
|
||||
"throwawaymail.com",
|
||||
"mailinator.com",
|
||||
"mailinator.net",
|
||||
"mailinator.org",
|
||||
"mailinator.info",
|
||||
"maildrop.cc",
|
||||
"yopmail.com",
|
||||
"yopmail.fr",
|
||||
"yopmail.net",
|
||||
"fakeinbox.com",
|
||||
"fakemailgenerator.com",
|
||||
"dispostable.com",
|
||||
"getairmail.com",
|
||||
"getnada.com",
|
||||
"mohmal.com",
|
||||
"trashmail.com",
|
||||
"trashmail.net",
|
||||
"trashmail.org",
|
||||
"sharklasers.com",
|
||||
"grr.la",
|
||||
"guerrillamail.de",
|
||||
"pokemail.net",
|
||||
"spam4.me",
|
||||
"spamgourmet.com",
|
||||
"mytrashmail.com",
|
||||
"mailnesia.com",
|
||||
"mailcatch.com",
|
||||
"mintemail.com",
|
||||
"tempr.email",
|
||||
"tempail.com",
|
||||
"emailondeck.com",
|
||||
"incognitomail.com",
|
||||
"inboxalias.com",
|
||||
"33mail.com",
|
||||
"spamex.com",
|
||||
"spamfree24.org",
|
||||
"spamspot.com",
|
||||
"mailnull.com",
|
||||
"mailsac.com",
|
||||
"emailfake.com",
|
||||
"fakemail.fr",
|
||||
"tempinbox.com",
|
||||
"throwaway.email",
|
||||
"burnermail.io",
|
||||
"anonbox.net",
|
||||
"anonymbox.com",
|
||||
"discard.email",
|
||||
"discardmail.com",
|
||||
"discardmail.de",
|
||||
"mailexpire.com",
|
||||
"mailforspam.com",
|
||||
"meltmail.com",
|
||||
"mt2009.com",
|
||||
"mt2014.com",
|
||||
"nospam.ze.tc",
|
||||
"nospamfor.us",
|
||||
"nowmymail.com",
|
||||
"receiveee.com",
|
||||
"safe-mail.net",
|
||||
"spamavert.com",
|
||||
"spambob.com",
|
||||
"spambog.com",
|
||||
"spambox.us",
|
||||
"spamcannon.com",
|
||||
"spamcannon.net",
|
||||
"spamcon.org",
|
||||
"spamcorptastic.com",
|
||||
"spamday.com",
|
||||
"spamfree.eu",
|
||||
"spamherelots.com",
|
||||
"spamhereplease.com",
|
||||
"spamhole.com",
|
||||
"spamify.com",
|
||||
"spaminator.de",
|
||||
"spamkill.info",
|
||||
"spaml.com",
|
||||
"spaml.de",
|
||||
"spamoff.de",
|
||||
"spamobox.com",
|
||||
"spamslicer.com",
|
||||
"spamstack.net",
|
||||
"spamthis.co.uk",
|
||||
"spamthisplease.com",
|
||||
"supergreatmail.com",
|
||||
"suremail.info",
|
||||
"teleworm.us",
|
||||
"tempemail.co.za",
|
||||
"tempemail.net",
|
||||
"tempmailaddress.com",
|
||||
"tempmailo.com",
|
||||
"thankyou2010.com",
|
||||
"thisisnotmyrealemail.com",
|
||||
"tm.slsrs.ru",
|
||||
"tmpeml.info",
|
||||
"trash-mail.at",
|
||||
"trash-mail.de",
|
||||
"trash2009.com",
|
||||
"trashemail.de",
|
||||
"trashmail.at",
|
||||
"trashmailer.com",
|
||||
"wegwerfmail.de",
|
||||
"wegwerfmail.net",
|
||||
"wegwerfmail.org",
|
||||
"wh4f.org",
|
||||
"willhackforfood.biz",
|
||||
"willselfdestruct.com",
|
||||
"xmaily.com",
|
||||
"xyzfree.net",
|
||||
"yep.it",
|
||||
"yogamaven.com",
|
||||
"yuurok.com",
|
||||
"zehnminutenmail.de",
|
||||
"zippymail.info",
|
||||
# Additional common ones
|
||||
"mailnator.com",
|
||||
"bugmenot.com",
|
||||
"dodgeit.com",
|
||||
"dodgit.com",
|
||||
"e4ward.com",
|
||||
"emailsensei.com",
|
||||
"hushmail.com",
|
||||
"jetable.org",
|
||||
"kasmail.com",
|
||||
"mailblock.net",
|
||||
"mailcatch.com",
|
||||
"mymailoasis.com",
|
||||
"nervmich.net",
|
||||
"nervtmansen.de",
|
||||
"oneoffemail.com",
|
||||
"pookmail.com",
|
||||
"shortmail.net",
|
||||
"sneakemail.com",
|
||||
"sogetthis.com",
|
||||
"tempemailaddress.com",
|
||||
"tempomail.fr",
|
||||
"temporaryemail.net",
|
||||
"temporaryforwarding.com",
|
||||
"temporaryinbox.com",
|
||||
"thankyou2010.com",
|
||||
"tyldd.com",
|
||||
"uggsrock.com",
|
||||
"veryrealemail.com",
|
||||
"yourewronghereswhy.com",
|
||||
}
|
||||
# Legacy default set — kept as empty set for import compatibility.
|
||||
# The external sync (disposable_domain_sync_service) replaces this with ~4,800
|
||||
# domains from the community-curated disposable-email-domains repo.
|
||||
# The migration path in sync_disposable_domains() references this set to ensure
|
||||
# no domains are lost during the transition.
|
||||
DEFAULT_DISPOSABLE_DOMAINS: Set[str] = set()
|
||||
|
||||
# Gmail-like domains that support alias normalization
|
||||
GMAIL_LIKE_DOMAINS = {
|
||||
@@ -199,6 +69,47 @@ GMAIL_LIKE_DOMAINS = {
|
||||
"googlemail.com",
|
||||
}
|
||||
|
||||
# Well-known email providers that are never disposable.
|
||||
# These skip all external API checks for zero-latency validation.
|
||||
WELL_KNOWN_PROVIDERS = {
|
||||
"gmail.com",
|
||||
"googlemail.com",
|
||||
"yahoo.com",
|
||||
"yahoo.co.uk",
|
||||
"yahoo.co.jp",
|
||||
"yahoo.fr",
|
||||
"hotmail.com",
|
||||
"hotmail.co.uk",
|
||||
"outlook.com",
|
||||
"outlook.fr",
|
||||
"live.com",
|
||||
"live.co.uk",
|
||||
"msn.com",
|
||||
"icloud.com",
|
||||
"me.com",
|
||||
"mac.com",
|
||||
"aol.com",
|
||||
"protonmail.com",
|
||||
"proton.me",
|
||||
"zoho.com",
|
||||
"yandex.com",
|
||||
"yandex.ru",
|
||||
"mail.com",
|
||||
"gmx.com",
|
||||
"gmx.de",
|
||||
"fastmail.com",
|
||||
"hey.com",
|
||||
"tutanota.com",
|
||||
"tuta.io",
|
||||
}
|
||||
|
||||
# How long a verified-clean domain result is trusted before re-checking
|
||||
VERIFIED_CLEAN_TTL = timedelta(days=7)
|
||||
|
||||
# External API URLs
|
||||
DEBOUNCE_API_URL = "https://disposable.debounce.io/"
|
||||
VERIFYMAIL_API_URL = "https://verifymail.io/api"
|
||||
|
||||
|
||||
class EmailValidationService:
|
||||
"""
|
||||
@@ -293,17 +204,30 @@ class EmailValidationService:
|
||||
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
if "external_domains" in data:
|
||||
# New data model: effective = (external + manual) - allowlisted
|
||||
external = set(data.get("external_domains", []))
|
||||
manual = set(data.get("manual_domains", []))
|
||||
allowlisted = set(data.get("allowlisted_domains", []))
|
||||
effective_domains = (external | manual) - allowlisted
|
||||
else:
|
||||
# Old data model (pre-migration): use disposable_domains + defaults
|
||||
allowlisted = set()
|
||||
effective_domains = set(data.get("disposable_domains", [])) | DEFAULT_DISPOSABLE_DOMAINS
|
||||
config = {
|
||||
"disposable_domains": set(data.get("disposable_domains", [])) | DEFAULT_DISPOSABLE_DOMAINS,
|
||||
"disposable_domains": effective_domains,
|
||||
"allowlisted_domains": allowlisted,
|
||||
"blocked_emails": set(data.get("blocked_emails", [])),
|
||||
"blocked_ips": set(data.get("blocked_ips", [])),
|
||||
"verified_clean_domains": data.get("verified_clean_domains", {}),
|
||||
}
|
||||
else:
|
||||
# Use defaults if no config exists
|
||||
config = {
|
||||
"disposable_domains": DEFAULT_DISPOSABLE_DOMAINS.copy(),
|
||||
"allowlisted_domains": set(),
|
||||
"blocked_emails": set(),
|
||||
"blocked_ips": set(),
|
||||
"verified_clean_domains": {},
|
||||
}
|
||||
|
||||
# Update cache
|
||||
@@ -316,6 +240,16 @@ class EmailValidationService:
|
||||
"""
|
||||
Check if an email uses a disposable domain.
|
||||
|
||||
Uses a tiered checking strategy:
|
||||
0. Well-known providers (gmail, yahoo, etc.) → instant False
|
||||
1. Static blocklist from Firestore → instant
|
||||
2. Verified-clean cache (7-day TTL) → skip API calls
|
||||
3. DeBounce API (free) → fast external check
|
||||
4. verifymail.io API (paid) → only if DeBounce says clean
|
||||
|
||||
Flagged domains are auto-learned to manual_domains.
|
||||
Clean domains are cached for 7 days.
|
||||
|
||||
Args:
|
||||
email: Email address to check
|
||||
|
||||
@@ -326,8 +260,194 @@ class EmailValidationService:
|
||||
return False
|
||||
|
||||
domain = email.lower().split("@")[-1]
|
||||
|
||||
# Tier 0: Well-known providers — skip everything
|
||||
if domain in WELL_KNOWN_PROVIDERS:
|
||||
return False
|
||||
|
||||
# Tier 1: Static blocklist (instant, always available)
|
||||
config = self.get_blocklist_config()
|
||||
return domain in config["disposable_domains"]
|
||||
if domain in config["disposable_domains"]:
|
||||
return True
|
||||
|
||||
# Tier 1.5: Explicitly allowlisted domains skip external API checks.
|
||||
# Without this, external APIs can override admin allowlist decisions
|
||||
# (e.g. inbox.testmail.app is a legitimate testing domain we allow).
|
||||
if domain in config.get("allowlisted_domains", set()):
|
||||
return False
|
||||
|
||||
# Tier 2: Recently verified clean? Skip API calls
|
||||
if self._is_domain_verified_clean(domain, config):
|
||||
return False
|
||||
|
||||
# Tier 3: DeBounce API (free, fast)
|
||||
debounce_result = self._check_debounce(email)
|
||||
if debounce_result is True:
|
||||
self._auto_learn_domain(domain)
|
||||
return True
|
||||
|
||||
# Tier 4: verifymail.io (paid, only if DeBounce gave definitive clean)
|
||||
if debounce_result is False:
|
||||
verifymail_result = self._check_verifymail(email)
|
||||
if verifymail_result is True:
|
||||
self._auto_learn_domain(domain)
|
||||
return True
|
||||
|
||||
# Both APIs say clean (or errored) — cache clean result
|
||||
if debounce_result is False:
|
||||
self._mark_domain_clean(domain)
|
||||
|
||||
return False
|
||||
|
||||
def _check_debounce(self, email: str) -> Optional[bool]:
|
||||
"""
|
||||
Check email against DeBounce disposable email API.
|
||||
|
||||
Returns:
|
||||
True if disposable, False if clean, None if error/timeout.
|
||||
"""
|
||||
try:
|
||||
client = _get_http_client()
|
||||
response = client.get(DEBOUNCE_API_URL, params={"email": email})
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
result = str(data.get("disposable", "")).lower() == "true"
|
||||
logger.info(
|
||||
f"DeBounce check for {_mask_email(email)}: "
|
||||
f"disposable={result}"
|
||||
)
|
||||
return result
|
||||
except httpx.TimeoutException:
|
||||
logger.warning(f"DeBounce API timeout for {_mask_email(email)}")
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.warning(f"DeBounce API error for {_mask_email(email)}: {e}")
|
||||
return None
|
||||
|
||||
def _check_verifymail(self, email: str) -> Optional[bool]:
|
||||
"""
|
||||
Check email against verifymail.io API.
|
||||
|
||||
Only called when DeBounce says clean (to minimize paid API usage).
|
||||
|
||||
Returns:
|
||||
True if disposable/blocked, False if clean, None if error/timeout
|
||||
or API key not configured.
|
||||
"""
|
||||
api_key = settings.get_secret("verifymail-api-key")
|
||||
if not api_key:
|
||||
logger.debug("verifymail.io API key not configured, skipping")
|
||||
return None
|
||||
|
||||
try:
|
||||
client = _get_http_client()
|
||||
response = client.get(
|
||||
f"{VERIFYMAIL_API_URL}/{email}",
|
||||
params={"key": api_key},
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
# verifymail.io returns both "disposable" and "block" flags
|
||||
is_disposable = data.get("disposable") is True
|
||||
is_blocked = data.get("block") is True
|
||||
result = is_disposable or is_blocked
|
||||
logger.info(
|
||||
f"verifymail.io check for {_mask_email(email)}: "
|
||||
f"disposable={is_disposable}, block={is_blocked}"
|
||||
)
|
||||
return result
|
||||
except httpx.TimeoutException:
|
||||
logger.warning(f"verifymail.io API timeout for {_mask_email(email)}")
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.warning(f"verifymail.io API error for {_mask_email(email)}: {e}")
|
||||
return None
|
||||
|
||||
def _is_domain_verified_clean(self, domain: str, config: dict) -> bool:
|
||||
"""Check if a domain was verified clean within the TTL period."""
|
||||
verified = config.get("verified_clean_domains", {})
|
||||
entry = verified.get(domain)
|
||||
if not entry:
|
||||
return False
|
||||
|
||||
checked_at = entry.get("checked_at")
|
||||
if not checked_at:
|
||||
return False
|
||||
|
||||
# Handle both datetime objects and ISO strings
|
||||
if isinstance(checked_at, str):
|
||||
try:
|
||||
checked_at = datetime.fromisoformat(checked_at)
|
||||
except (ValueError, TypeError):
|
||||
return False
|
||||
|
||||
# Make timezone-aware if naive
|
||||
if checked_at.tzinfo is None:
|
||||
checked_at = checked_at.replace(tzinfo=timezone.utc)
|
||||
|
||||
return (datetime.now(timezone.utc) - checked_at) < VERIFIED_CLEAN_TTL
|
||||
|
||||
def _auto_learn_domain(self, domain: str) -> None:
|
||||
"""
|
||||
Add a domain flagged by external API to manual_domains in Firestore.
|
||||
|
||||
This persists the result so future checks don't need the API call.
|
||||
Fire-and-forget: errors are logged but don't affect the caller.
|
||||
"""
|
||||
try:
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(
|
||||
BLOCKLIST_CONFIG_DOC
|
||||
)
|
||||
doc = doc_ref.get()
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
manual = set(data.get("manual_domains", []))
|
||||
else:
|
||||
manual = set()
|
||||
|
||||
if domain not in manual:
|
||||
manual.add(domain)
|
||||
doc_ref.set(
|
||||
{
|
||||
"manual_domains": list(manual),
|
||||
"updated_at": datetime.now(timezone.utc).isoformat(),
|
||||
"updated_by": "auto-learn",
|
||||
},
|
||||
merge=True,
|
||||
)
|
||||
logger.info(f"Auto-learned disposable domain: {domain}")
|
||||
|
||||
# Invalidate cache so the domain is blocked immediately
|
||||
EmailValidationService._blocklist_cache = None
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to auto-learn domain {domain}: {e}")
|
||||
|
||||
def _mark_domain_clean(self, domain: str) -> None:
|
||||
"""
|
||||
Record that a domain was verified clean by external APIs.
|
||||
|
||||
Stored in verified_clean_domains dict keyed by domain.
|
||||
Future checks within VERIFIED_CLEAN_TTL will skip API calls.
|
||||
|
||||
Note: Uses read-modify-write instead of dot notation because
|
||||
domain names contain dots which Firestore interprets as nested paths.
|
||||
"""
|
||||
try:
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(
|
||||
BLOCKLIST_CONFIG_DOC
|
||||
)
|
||||
doc = doc_ref.get()
|
||||
verified = doc.to_dict().get("verified_clean_domains", {}) if doc.exists else {}
|
||||
verified[domain] = {
|
||||
"checked_at": datetime.now(timezone.utc).isoformat(),
|
||||
}
|
||||
doc_ref.set({"verified_clean_domains": verified}, merge=True)
|
||||
logger.debug(f"Marked domain as verified clean: {domain}")
|
||||
|
||||
# Invalidate cache so new data is picked up
|
||||
EmailValidationService._blocklist_cache = None
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to mark domain clean {domain}: {e}")
|
||||
|
||||
def is_email_blocked(self, email: str) -> bool:
|
||||
"""
|
||||
@@ -369,36 +489,6 @@ class EmailValidationService:
|
||||
config = self.get_blocklist_config()
|
||||
return ip_address in config["blocked_ips"]
|
||||
|
||||
def validate_email_for_beta(self, email: str) -> Tuple[bool, str]:
|
||||
"""
|
||||
Validate an email for beta enrollment.
|
||||
|
||||
Performs all validation checks:
|
||||
1. Basic format validation
|
||||
2. Disposable domain check
|
||||
3. Blocked email check
|
||||
|
||||
Args:
|
||||
email: Email address to validate
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, error_message)
|
||||
"""
|
||||
if not email or "@" not in email:
|
||||
return False, "Invalid email format"
|
||||
|
||||
# Check disposable domain
|
||||
if self.is_disposable_domain(email):
|
||||
logger.warning(f"Beta enrollment blocked - disposable domain: {_mask_email(email)}")
|
||||
return False, "Disposable email addresses are not allowed"
|
||||
|
||||
# Check blocked email
|
||||
if self.is_email_blocked(email):
|
||||
logger.warning(f"Beta enrollment blocked - email blocked: {_mask_email(email)}")
|
||||
return False, "This email address is not allowed"
|
||||
|
||||
return True, ""
|
||||
|
||||
def hash_ip(self, ip_address: str) -> str:
|
||||
"""
|
||||
Hash an IP address for privacy-preserving storage.
|
||||
@@ -428,13 +518,13 @@ class EmailValidationService:
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
domains = set(data.get("disposable_domains", []))
|
||||
domains = set(data.get("manual_domains", []))
|
||||
else:
|
||||
data = {}
|
||||
domains = set()
|
||||
|
||||
domains.add(domain)
|
||||
data["disposable_domains"] = list(domains)
|
||||
data["manual_domains"] = list(domains)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
@@ -462,14 +552,22 @@ class EmailValidationService:
|
||||
return
|
||||
|
||||
data = doc.to_dict()
|
||||
domains = set(data.get("disposable_domains", []))
|
||||
|
||||
if domain not in domains:
|
||||
manual = set(data.get("manual_domains", []))
|
||||
external = set(data.get("external_domains", []))
|
||||
|
||||
if domain in manual:
|
||||
result["found"] = True
|
||||
manual.discard(domain)
|
||||
data["manual_domains"] = list(manual)
|
||||
elif domain in external:
|
||||
result["found"] = True
|
||||
allowlisted = set(data.get("allowlisted_domains", []))
|
||||
allowlisted.add(domain)
|
||||
data["allowlisted_domains"] = list(allowlisted)
|
||||
else:
|
||||
return
|
||||
|
||||
result["found"] = True
|
||||
domains.discard(domain)
|
||||
data["disposable_domains"] = list(domains)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
@@ -628,14 +726,126 @@ class EmailValidationService:
|
||||
logger.info(f"Removed blocked IP: {ip_address} by {admin_email}")
|
||||
return True
|
||||
|
||||
def add_allowlisted_domain(self, domain: str, admin_email: str) -> bool:
|
||||
"""Add a domain to the allowlisted domains list."""
|
||||
domain = domain.lower().strip()
|
||||
if not domain:
|
||||
return False
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
domains = set(data.get("allowlisted_domains", []))
|
||||
else:
|
||||
data = {}
|
||||
domains = set()
|
||||
|
||||
domains.add(domain)
|
||||
data["allowlisted_domains"] = list(domains)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Added allowlisted domain: {domain} by {admin_email}")
|
||||
return True
|
||||
|
||||
def remove_allowlisted_domain(self, domain: str, admin_email: str) -> bool:
|
||||
"""Remove a domain from the allowlisted domains list."""
|
||||
domain = domain.lower().strip()
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
result = {"found": False}
|
||||
|
||||
@firestore.transactional
|
||||
def remove_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if not doc.exists:
|
||||
return
|
||||
|
||||
data = doc.to_dict()
|
||||
domains = set(data.get("allowlisted_domains", []))
|
||||
|
||||
if domain not in domains:
|
||||
return
|
||||
|
||||
result["found"] = True
|
||||
domains.discard(domain)
|
||||
data["allowlisted_domains"] = list(domains)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
remove_in_transaction(transaction, doc_ref)
|
||||
|
||||
if not result["found"]:
|
||||
return False
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Removed allowlisted domain: {domain} by {admin_email}")
|
||||
return True
|
||||
|
||||
def get_blocklist_raw_data(self) -> dict:
|
||||
"""Get raw blocklist data for admin display (not the effective set)."""
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
doc = doc_ref.get()
|
||||
if not doc.exists:
|
||||
return {
|
||||
"external_domains": [], "manual_domains": [], "allowlisted_domains": [],
|
||||
"blocked_emails": [], "blocked_ips": [],
|
||||
"last_sync_at": None, "last_sync_count": None,
|
||||
"updated_at": None, "updated_by": None,
|
||||
}
|
||||
data = doc.to_dict()
|
||||
return {
|
||||
"external_domains": sorted(data.get("external_domains", [])),
|
||||
"manual_domains": sorted(data.get("manual_domains", [])),
|
||||
"allowlisted_domains": sorted(data.get("allowlisted_domains", [])),
|
||||
"blocked_emails": sorted(data.get("blocked_emails", [])),
|
||||
"blocked_ips": sorted(data.get("blocked_ips", [])),
|
||||
"last_sync_at": data.get("last_sync_at"),
|
||||
"last_sync_count": data.get("last_sync_count"),
|
||||
"updated_at": data.get("updated_at"),
|
||||
"updated_by": data.get("updated_by"),
|
||||
}
|
||||
|
||||
def get_blocklist_stats(self) -> dict:
|
||||
"""Get statistics about current blocklists."""
|
||||
config = self.get_blocklist_config(force_refresh=True)
|
||||
|
||||
# Also get raw data for detailed counts
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
doc = doc_ref.get()
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
external_count = len(data.get("external_domains", []))
|
||||
manual_count = len(data.get("manual_domains", []))
|
||||
allowlisted_count = len(data.get("allowlisted_domains", []))
|
||||
else:
|
||||
external_count = 0
|
||||
manual_count = 0
|
||||
allowlisted_count = 0
|
||||
|
||||
return {
|
||||
"disposable_domains_count": len(config["disposable_domains"]),
|
||||
"blocked_emails_count": len(config["blocked_emails"]),
|
||||
"blocked_ips_count": len(config["blocked_ips"]),
|
||||
"default_disposable_domains_count": len(DEFAULT_DISPOSABLE_DOMAINS),
|
||||
"external_domains_count": external_count,
|
||||
"manual_domains_count": manual_count,
|
||||
"allowlisted_domains_count": allowlisted_count,
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -383,6 +383,8 @@ class GCEEncodingBackend(EncodingBackend):
|
||||
output_files["mp4_4k_lossy"] = path
|
||||
elif "720p" in filename_lower:
|
||||
output_files["mp4_720p"] = path
|
||||
elif "with vocals" in filename_lower and filename.endswith(".mp4"):
|
||||
output_files["with_vocals_mp4"] = path
|
||||
self.logger.info(f"Converted output_files list to dict: {output_files}")
|
||||
else:
|
||||
output_files = raw_output_files if isinstance(raw_output_files, dict) else {}
|
||||
@@ -393,6 +395,7 @@ class GCEEncodingBackend(EncodingBackend):
|
||||
lossy_4k_mp4_path=output_files.get("mp4_4k_lossy"),
|
||||
lossless_mkv_path=output_files.get("mkv_4k"),
|
||||
lossy_720p_mp4_path=output_files.get("mp4_720p"),
|
||||
with_vocals_mp4_path=output_files.get("with_vocals_mp4"),
|
||||
output_files=output_files,
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
|
||||
@@ -17,6 +17,7 @@ Usage:
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
import aiohttp
|
||||
@@ -25,10 +26,23 @@ from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Retry configuration for handling transient failures (e.g., worker restarts)
|
||||
MAX_RETRIES = 3
|
||||
INITIAL_BACKOFF_SECONDS = 2.0
|
||||
MAX_BACKOFF_SECONDS = 10.0
|
||||
# Retry configuration for handling transient failures (e.g., worker restarts during deployments).
|
||||
#
|
||||
# During a CI deployment, the encoding worker is restarted via systemctl. The restart
|
||||
# involves downloading a new wheel from GCS, installing it, and starting uvicorn —
|
||||
# which can take 30-90 seconds. The retry window must exceed this to avoid failing
|
||||
# jobs that happen to hit the encoding stage during a deploy.
|
||||
#
|
||||
# With 7 retries and exponential backoff (5s, 10s, 15s, 15s, 15s, 15s, 15s),
|
||||
# total retry window is ~90s, sufficient to survive a full worker restart.
|
||||
MAX_RETRIES = 7
|
||||
INITIAL_BACKOFF_SECONDS = 5.0
|
||||
MAX_BACKOFF_SECONDS = 15.0
|
||||
|
||||
# Poll failure tolerance: number of consecutive poll failures allowed before giving up.
|
||||
# This prevents a single transient network blip during status polling from killing a
|
||||
# long-running encoding job. Similar pattern to flacfetch status polling (PR #446).
|
||||
MAX_CONSECUTIVE_POLL_FAILURES = 5
|
||||
|
||||
|
||||
class EncodingService:
|
||||
@@ -39,6 +53,34 @@ class EncodingService:
|
||||
self._url = None
|
||||
self._api_key = None
|
||||
self._initialized = False
|
||||
self._worker_manager = None
|
||||
self._cached_url = None
|
||||
self._url_cached_at = 0
|
||||
self._URL_CACHE_TTL = 30 # seconds
|
||||
|
||||
def set_worker_manager(self, manager):
|
||||
"""Set the worker manager for dynamic URL resolution from Firestore."""
|
||||
self._worker_manager = manager
|
||||
|
||||
def _get_worker_url(self) -> str:
|
||||
"""Get the current primary worker URL from Firestore (with TTL cache).
|
||||
|
||||
Falls back to static URL from config if worker_manager is not set.
|
||||
"""
|
||||
now = time.time()
|
||||
if self._cached_url and (now - self._url_cached_at) < self._URL_CACHE_TTL:
|
||||
return self._cached_url
|
||||
|
||||
if self._worker_manager:
|
||||
config = self._worker_manager.get_config()
|
||||
self._cached_url = config.primary_url
|
||||
self._url_cached_at = now
|
||||
return self._cached_url
|
||||
|
||||
# Fallback to static URL
|
||||
if not self._initialized:
|
||||
self._load_credentials()
|
||||
return self._url
|
||||
|
||||
def _load_credentials(self):
|
||||
"""Load encoding worker URL and API key from config/secrets."""
|
||||
@@ -73,6 +115,28 @@ class EncodingService:
|
||||
"""Check if GCE preview encoding is enabled and configured."""
|
||||
return self.settings.use_gce_preview_encoding and self.is_configured
|
||||
|
||||
def _warmup_encoding_worker_fallback(self, job_id: str) -> None:
|
||||
"""Safety net: try to start the encoding worker VM on first connection failure.
|
||||
|
||||
If the worker_manager is available, calls ensure_primary_running() so the VM
|
||||
starts booting during the retry window. If worker_manager is not set (e.g. dev
|
||||
mode with static URL), this is a no-op.
|
||||
"""
|
||||
if not self._worker_manager:
|
||||
return
|
||||
try:
|
||||
result = self._worker_manager.ensure_primary_running()
|
||||
if result["started"]:
|
||||
logger.warning(
|
||||
f"[job:{job_id}] Encoding worker unreachable — started VM {result['vm_name']} as fallback"
|
||||
)
|
||||
else:
|
||||
logger.info(
|
||||
f"[job:{job_id}] Encoding worker unreachable — VM {result['vm_name']} already running/starting"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"[job:{job_id}] Encoding worker warmup fallback failed (non-fatal): {e}")
|
||||
|
||||
async def _request_with_retry(
|
||||
self,
|
||||
method: str,
|
||||
@@ -134,6 +198,8 @@ class EncodingService:
|
||||
}
|
||||
except (aiohttp.ClientConnectorError, aiohttp.ServerDisconnectedError, asyncio.TimeoutError) as e:
|
||||
last_exception = e
|
||||
if attempt == 0:
|
||||
self._warmup_encoding_worker_fallback(job_id)
|
||||
if attempt < MAX_RETRIES:
|
||||
logger.warning(
|
||||
f"[job:{job_id}] GCE worker connection failed (attempt {attempt + 1}/{MAX_RETRIES + 1}): {e}. "
|
||||
@@ -175,7 +241,7 @@ class EncodingService:
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._url}/encode"
|
||||
url = f"{self._get_worker_url()}/encode"
|
||||
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
|
||||
payload = {
|
||||
"job_id": job_id,
|
||||
@@ -198,7 +264,24 @@ class EncodingService:
|
||||
if resp["status"] == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp["status"] == 409:
|
||||
raise RuntimeError(f"Encoding job {job_id} already exists")
|
||||
# Job already exists on worker — check if it already completed
|
||||
logger.warning(f"[job:{job_id}] GCE worker returned 409, checking job status")
|
||||
try:
|
||||
status = await self.get_job_status(job_id)
|
||||
job_status = status.get("status", "unknown")
|
||||
if job_status == "complete":
|
||||
logger.info(f"[job:{job_id}] Job already complete on GCE worker, returning cached result")
|
||||
return {"status": "cached", "job_id": job_id, "output_files": status.get("output_files")}
|
||||
elif job_status in ("pending", "running"):
|
||||
logger.info(f"[job:{job_id}] Job still in progress on GCE worker")
|
||||
return {"status": "in_progress", "job_id": job_id}
|
||||
else:
|
||||
raise RuntimeError(f"Encoding job {job_id} already exists with status: {job_status}")
|
||||
except RuntimeError as e:
|
||||
if "not found" in str(e).lower():
|
||||
# Job was in worker memory but got cleared (restart) — safe to raise original error
|
||||
raise RuntimeError(f"Encoding job {job_id} conflict: 409 but job not found on status check")
|
||||
raise
|
||||
if resp["status"] != 200:
|
||||
raise RuntimeError(f"Failed to submit encoding job: {resp['status']} - {resp['text']}")
|
||||
|
||||
@@ -219,7 +302,7 @@ class EncodingService:
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._url}/status/{job_id}"
|
||||
url = f"{self._get_worker_url()}/status/{job_id}"
|
||||
headers = {"X-API-Key": self._api_key}
|
||||
|
||||
resp = await self._request_with_retry(
|
||||
@@ -247,7 +330,12 @@ class EncodingService:
|
||||
progress_callback=None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Poll for encoding job completion.
|
||||
Poll for encoding job completion with tolerance for transient failures.
|
||||
|
||||
During a deployment, the encoding worker may restart while we're polling.
|
||||
Rather than failing the entire job on a single poll error, we tolerate up to
|
||||
MAX_CONSECUTIVE_POLL_FAILURES consecutive failures before giving up. This is
|
||||
the same pattern used in flacfetch status polling (PR #446).
|
||||
|
||||
Args:
|
||||
job_id: Job identifier
|
||||
@@ -260,19 +348,40 @@ class EncodingService:
|
||||
|
||||
Raises:
|
||||
TimeoutError: If job doesn't complete within timeout
|
||||
RuntimeError: If job fails
|
||||
RuntimeError: If job fails or too many consecutive poll failures
|
||||
"""
|
||||
logger.info(f"[job:{job_id}] Waiting for GCE encoding to complete...")
|
||||
|
||||
start_time = asyncio.get_event_loop().time()
|
||||
last_progress = 0
|
||||
consecutive_failures = 0
|
||||
|
||||
while True:
|
||||
elapsed = asyncio.get_event_loop().time() - start_time
|
||||
if elapsed > timeout:
|
||||
raise TimeoutError(f"Encoding job {job_id} timed out after {timeout}s")
|
||||
|
||||
status = await self.get_job_status(job_id)
|
||||
try:
|
||||
status = await self.get_job_status(job_id)
|
||||
# Reset failure counter on successful poll
|
||||
consecutive_failures = 0
|
||||
except (aiohttp.ClientConnectorError, aiohttp.ServerDisconnectedError,
|
||||
asyncio.TimeoutError, RuntimeError) as e:
|
||||
consecutive_failures += 1
|
||||
if consecutive_failures >= MAX_CONSECUTIVE_POLL_FAILURES:
|
||||
logger.error(
|
||||
f"[job:{job_id}] Status polling failed {consecutive_failures} consecutive times, giving up: {e}"
|
||||
)
|
||||
raise RuntimeError(
|
||||
f"Encoding job {job_id} lost contact with worker after "
|
||||
f"{consecutive_failures} consecutive poll failures: {e}"
|
||||
)
|
||||
logger.warning(
|
||||
f"[job:{job_id}] Status poll failed ({consecutive_failures}/{MAX_CONSECUTIVE_POLL_FAILURES}): {e}. "
|
||||
f"Worker may be restarting, will retry..."
|
||||
)
|
||||
await asyncio.sleep(poll_interval)
|
||||
continue
|
||||
|
||||
# Handle case where GCE worker returns a list instead of dict
|
||||
if isinstance(status, list):
|
||||
@@ -331,7 +440,17 @@ class EncodingService:
|
||||
config = encoding_config or {"formats": ["mp4_4k", "mp4_720p"]}
|
||||
|
||||
# Submit the job
|
||||
await self.submit_encoding_job(job_id, input_gcs_path, output_gcs_path, config)
|
||||
submit_result = await self.submit_encoding_job(job_id, input_gcs_path, output_gcs_path, config)
|
||||
|
||||
# If cached, return immediately — encoding already done
|
||||
submit_status = submit_result.get("status")
|
||||
if submit_status == "cached":
|
||||
logger.info(f"[job:{job_id}] Encoding already cached, returning immediately")
|
||||
return {"status": "complete", "output_files": submit_result.get("output_files")}
|
||||
|
||||
# If in_progress, another request is encoding it — just wait for that
|
||||
if submit_status == "in_progress":
|
||||
logger.info(f"[job:{job_id}] Encoding already in progress, joining poll")
|
||||
|
||||
# Wait for completion
|
||||
return await self.wait_for_completion(
|
||||
@@ -371,7 +490,7 @@ class EncodingService:
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._url}/encode-preview"
|
||||
url = f"{self._get_worker_url()}/encode-preview"
|
||||
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
|
||||
payload = {
|
||||
"job_id": job_id,
|
||||
@@ -465,6 +584,116 @@ class EncodingService:
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
async def submit_render_video_job(
|
||||
self,
|
||||
job_id: str,
|
||||
render_config: Dict[str, Any],
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit a render-video job to the GCE worker.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
render_config: Configuration dict containing original_corrections_gcs_path,
|
||||
audio_gcs_path, output_gcs_prefix, artist, title, and any other render params.
|
||||
|
||||
Returns:
|
||||
Response from the encoding worker
|
||||
|
||||
Raises:
|
||||
RuntimeError: If submission fails
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._get_worker_url()}/render-video"
|
||||
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
|
||||
payload = {"job_id": job_id, **render_config}
|
||||
|
||||
logger.info(f"[job:{job_id}] Submitting render-video job to GCE worker: {url}")
|
||||
|
||||
resp = await self._request_with_retry(
|
||||
method="POST",
|
||||
url=url,
|
||||
headers=headers,
|
||||
json_payload=payload,
|
||||
timeout=30.0,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
if resp["status"] == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp["status"] == 409:
|
||||
# Job already exists on worker — check if it already completed
|
||||
logger.warning(f"[job:{job_id}] GCE worker returned 409, checking job status")
|
||||
try:
|
||||
status = await self.get_job_status(job_id)
|
||||
job_status = status.get("status", "unknown")
|
||||
if job_status == "complete":
|
||||
logger.info(f"[job:{job_id}] Render-video job already complete on GCE worker, returning cached result")
|
||||
return {
|
||||
"status": "cached",
|
||||
"job_id": job_id,
|
||||
"output_files": status.get("output_files"),
|
||||
"metadata": status.get("metadata"),
|
||||
}
|
||||
elif job_status in ("pending", "running"):
|
||||
logger.info(f"[job:{job_id}] Render-video job still in progress on GCE worker")
|
||||
return {"status": "in_progress", "job_id": job_id}
|
||||
else:
|
||||
raise RuntimeError(f"Render-video job {job_id} already exists with status: {job_status}")
|
||||
except RuntimeError as e:
|
||||
if "not found" in str(e).lower():
|
||||
raise RuntimeError(f"Render-video job {job_id} conflict: 409 but job not found on status check")
|
||||
raise
|
||||
if resp["status"] != 200:
|
||||
raise RuntimeError(f"Failed to submit render-video job: {resp['status']} - {resp['text']}")
|
||||
|
||||
return resp["json"]
|
||||
|
||||
async def render_video_on_gce(
|
||||
self,
|
||||
job_id: str,
|
||||
render_config: Dict[str, Any],
|
||||
progress_callback=None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit render-video job and wait for completion.
|
||||
|
||||
This is a convenience method that combines submit + wait.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
render_config: Configuration dict for the render-video worker endpoint
|
||||
progress_callback: Optional callback for progress updates
|
||||
|
||||
Returns:
|
||||
Final job status with output files and metadata
|
||||
"""
|
||||
# Submit the job
|
||||
submit_result = await self.submit_render_video_job(job_id, render_config)
|
||||
|
||||
# If cached, return immediately — rendering already done
|
||||
submit_status = submit_result.get("status")
|
||||
if submit_status == "cached":
|
||||
logger.info(f"[job:{job_id}] Render-video already cached, returning immediately")
|
||||
return {
|
||||
"status": "complete",
|
||||
"output_files": submit_result.get("output_files"),
|
||||
"metadata": submit_result.get("metadata"),
|
||||
}
|
||||
|
||||
# If in_progress, another request is rendering it — just wait for that
|
||||
if submit_status == "in_progress":
|
||||
logger.info(f"[job:{job_id}] Render-video already in progress, joining poll")
|
||||
|
||||
# Wait for completion
|
||||
return await self.wait_for_completion(
|
||||
job_id, progress_callback=progress_callback
|
||||
)
|
||||
|
||||
async def health_check(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Check the health of the encoding worker.
|
||||
@@ -477,7 +706,7 @@ class EncodingService:
|
||||
if not self.is_configured:
|
||||
return {"status": "not_configured"}
|
||||
|
||||
url = f"{self._url}/health"
|
||||
url = f"{self._get_worker_url()}/health"
|
||||
headers = {"X-API-Key": self._api_key}
|
||||
|
||||
try:
|
||||
@@ -499,4 +728,20 @@ def get_encoding_service() -> EncodingService:
|
||||
global _encoding_service
|
||||
if _encoding_service is None:
|
||||
_encoding_service = EncodingService()
|
||||
# Wire up worker manager for dynamic URL resolution
|
||||
try:
|
||||
from backend.services.encoding_worker_manager import EncodingWorkerManager
|
||||
from google.cloud import compute_v1, firestore
|
||||
settings = get_settings()
|
||||
db = firestore.Client(project=settings.google_cloud_project)
|
||||
compute_client = compute_v1.InstancesClient()
|
||||
manager = EncodingWorkerManager(
|
||||
db=db,
|
||||
compute_client=compute_client,
|
||||
project_id=settings.google_cloud_project,
|
||||
)
|
||||
_encoding_service.set_worker_manager(manager)
|
||||
except Exception:
|
||||
# Fallback to static URL if worker manager setup fails
|
||||
pass
|
||||
return _encoding_service
|
||||
@@ -0,0 +1,239 @@
|
||||
"""
|
||||
Blue-Green Encoding Worker Manager.
|
||||
|
||||
Manages the lifecycle and role assignment of two GCE encoding worker VMs
|
||||
(blue/green) via a Firestore config document as the single source of truth.
|
||||
|
||||
The config document at `config/encoding-worker` tracks which VM is primary
|
||||
(actively serving encoding requests) and which is secondary (idle or being
|
||||
deployed to). This enables zero-downtime deployments by swapping roles
|
||||
after deploying new code to the secondary.
|
||||
|
||||
Usage:
|
||||
from google.cloud import firestore
|
||||
from google.cloud.compute_v1 import InstancesClient
|
||||
|
||||
db = firestore.Client()
|
||||
compute = InstancesClient()
|
||||
manager = EncodingWorkerManager(db, compute)
|
||||
|
||||
# Get current config
|
||||
config = manager.get_config()
|
||||
print(f"Primary: {config.primary_vm} at {config.primary_url}")
|
||||
|
||||
# Ensure primary VM is running before dispatching work
|
||||
result = manager.ensure_primary_running()
|
||||
"""
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, UTC
|
||||
from typing import Optional
|
||||
|
||||
from google.cloud import firestore
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
CONFIG_COLLECTION = "config"
|
||||
CONFIG_DOCUMENT = "encoding-worker"
|
||||
ENCODING_WORKER_PORT = 8080
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingWorkerConfig:
|
||||
"""Represents the Firestore config/encoding-worker document."""
|
||||
|
||||
primary_vm: str
|
||||
primary_ip: str
|
||||
primary_version: str
|
||||
primary_deployed_at: Optional[str]
|
||||
secondary_vm: str
|
||||
secondary_ip: str
|
||||
secondary_version: Optional[str]
|
||||
secondary_deployed_at: Optional[str]
|
||||
last_swap_at: Optional[str]
|
||||
last_activity_at: Optional[str]
|
||||
deploy_in_progress: bool
|
||||
deploy_in_progress_since: Optional[str]
|
||||
|
||||
@property
|
||||
def primary_url(self) -> str:
|
||||
return f"http://{self.primary_ip}:{ENCODING_WORKER_PORT}"
|
||||
|
||||
@property
|
||||
def secondary_url(self) -> str:
|
||||
return f"http://{self.secondary_ip}:{ENCODING_WORKER_PORT}"
|
||||
|
||||
|
||||
class EncodingWorkerManager:
|
||||
"""Manages blue-green encoding worker VMs via Firestore config and Compute API."""
|
||||
|
||||
def __init__(self, db, compute_client=None, project_id="nomadkaraoke", zone="us-central1-c"):
|
||||
self._db = db
|
||||
self._compute = compute_client
|
||||
self._project_id = project_id
|
||||
self._zone = zone
|
||||
|
||||
def _doc_ref(self):
|
||||
"""Return a reference to the config/encoding-worker document."""
|
||||
return self._db.collection(CONFIG_COLLECTION).document(CONFIG_DOCUMENT)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Firestore config operations (Task 1)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def get_config(self) -> EncodingWorkerConfig:
|
||||
"""Read config/encoding-worker from Firestore and return as dataclass."""
|
||||
snapshot = self._doc_ref().get()
|
||||
if not snapshot.exists:
|
||||
raise ValueError(
|
||||
f"Encoding worker config document not found at "
|
||||
f"{CONFIG_COLLECTION}/{CONFIG_DOCUMENT}"
|
||||
)
|
||||
data = snapshot.to_dict()
|
||||
return EncodingWorkerConfig(
|
||||
primary_vm=data["primary_vm"],
|
||||
primary_ip=data["primary_ip"],
|
||||
primary_version=data["primary_version"],
|
||||
primary_deployed_at=data.get("primary_deployed_at"),
|
||||
secondary_vm=data["secondary_vm"],
|
||||
secondary_ip=data["secondary_ip"],
|
||||
secondary_version=data.get("secondary_version"),
|
||||
secondary_deployed_at=data.get("secondary_deployed_at"),
|
||||
last_swap_at=data.get("last_swap_at"),
|
||||
last_activity_at=data.get("last_activity_at"),
|
||||
deploy_in_progress=data.get("deploy_in_progress", False),
|
||||
deploy_in_progress_since=data.get("deploy_in_progress_since"),
|
||||
)
|
||||
|
||||
def update_activity(self) -> None:
|
||||
"""Update last_activity_at timestamp to record recent usage."""
|
||||
now = datetime.now(UTC).isoformat()
|
||||
self._doc_ref().update({"last_activity_at": now})
|
||||
logger.info("Updated encoding worker last_activity_at to %s", now)
|
||||
|
||||
def swap_roles(
|
||||
self,
|
||||
new_primary_vm: str,
|
||||
new_primary_ip: str,
|
||||
new_primary_version: str,
|
||||
old_primary_vm: str,
|
||||
old_primary_ip: str,
|
||||
old_primary_version: str,
|
||||
) -> None:
|
||||
"""Atomically swap primary and secondary roles using a Firestore transaction.
|
||||
|
||||
Args:
|
||||
new_primary_vm: VM name that will become the new primary.
|
||||
new_primary_ip: Internal IP of the new primary.
|
||||
new_primary_version: Version deployed on the new primary.
|
||||
old_primary_vm: Current primary VM name (becomes secondary).
|
||||
old_primary_ip: Internal IP of the old primary.
|
||||
old_primary_version: Version on the old primary.
|
||||
"""
|
||||
transaction = self._db.transaction()
|
||||
doc_ref = self._doc_ref()
|
||||
now = datetime.now(UTC).isoformat()
|
||||
|
||||
@firestore.transactional
|
||||
def _swap_in_transaction(txn, ref):
|
||||
txn.update(ref, {
|
||||
"primary_vm": new_primary_vm,
|
||||
"primary_ip": new_primary_ip,
|
||||
"primary_version": new_primary_version,
|
||||
"primary_deployed_at": now,
|
||||
"secondary_vm": old_primary_vm,
|
||||
"secondary_ip": old_primary_ip,
|
||||
"secondary_version": old_primary_version,
|
||||
"secondary_deployed_at": None,
|
||||
"last_swap_at": now,
|
||||
"deploy_in_progress": False,
|
||||
"deploy_in_progress_since": None,
|
||||
})
|
||||
|
||||
_swap_in_transaction(transaction, doc_ref)
|
||||
logger.info(
|
||||
"Swapped encoding worker roles: %s -> primary, %s -> secondary",
|
||||
new_primary_vm, old_primary_vm,
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# VM lifecycle operations (Task 2)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def get_vm_status(self, vm_name: str) -> str:
|
||||
"""Get the current status of a GCE VM (e.g. RUNNING, TERMINATED, STAGING)."""
|
||||
instance = self._compute.get(
|
||||
project=self._project_id,
|
||||
zone=self._zone,
|
||||
instance=vm_name,
|
||||
)
|
||||
return instance.status
|
||||
|
||||
def start_vm(self, vm_name: str) -> None:
|
||||
"""Start a VM if it is not already running or starting.
|
||||
|
||||
Fire-and-forget: returns immediately. Caller should poll get_vm_status
|
||||
if they need to wait for the VM to be fully RUNNING.
|
||||
"""
|
||||
status = self.get_vm_status(vm_name)
|
||||
if status in ("RUNNING", "STAGING"):
|
||||
logger.info("VM %s is already %s, skipping start", vm_name, status)
|
||||
return
|
||||
logger.info("Starting VM %s (current status: %s)", vm_name, status)
|
||||
self._compute.start(
|
||||
project=self._project_id,
|
||||
zone=self._zone,
|
||||
instance=vm_name,
|
||||
)
|
||||
|
||||
def stop_vm(self, vm_name: str) -> None:
|
||||
"""Stop a VM if it is not already stopped or stopping."""
|
||||
status = self.get_vm_status(vm_name)
|
||||
if status in ("TERMINATED", "STOPPING"):
|
||||
logger.info("VM %s is already %s, skipping stop", vm_name, status)
|
||||
return
|
||||
logger.info("Stopping VM %s (current status: %s)", vm_name, status)
|
||||
self._compute.stop(
|
||||
project=self._project_id,
|
||||
zone=self._zone,
|
||||
instance=vm_name,
|
||||
)
|
||||
|
||||
def ensure_primary_running(self) -> dict:
|
||||
"""Ensure the primary encoding worker VM is running.
|
||||
|
||||
Reads config, starts primary VM if stopped, and updates activity timestamp.
|
||||
|
||||
Returns:
|
||||
dict with keys:
|
||||
- started (bool): True if VM was started, False if already running
|
||||
- vm_name (str): Name of the primary VM
|
||||
- primary_url (str): URL of the primary encoding worker
|
||||
"""
|
||||
config = self.get_config()
|
||||
vm_name = config.primary_vm
|
||||
|
||||
status = self.get_vm_status(vm_name)
|
||||
started = False
|
||||
|
||||
if status not in ("RUNNING", "STAGING"):
|
||||
logger.info("Primary VM %s is %s, starting it", vm_name, status)
|
||||
self._compute.start(
|
||||
project=self._project_id,
|
||||
zone=self._zone,
|
||||
instance=vm_name,
|
||||
)
|
||||
started = True
|
||||
else:
|
||||
logger.info("Primary VM %s is already %s", vm_name, status)
|
||||
|
||||
self.update_activity()
|
||||
|
||||
return {
|
||||
"started": started,
|
||||
"vm_name": vm_name,
|
||||
"primary_url": config.primary_url,
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
"""Production error monitoring service for Nomad Karaoke."""
|
||||
@@ -0,0 +1,177 @@
|
||||
"""Configuration for the production error monitoring service.
|
||||
|
||||
All constants, service lists, and environment-driven configuration
|
||||
functions used across the error monitor modules.
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# GCP project settings
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
GCP_PROJECT: str = os.environ.get("GCP_PROJECT", "nomadkaraoke")
|
||||
GCP_REGION: str = os.environ.get("GCP_REGION", "us-central1")
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Log querying settings
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: How far back to look for errors on each run (minutes).
|
||||
LOOKBACK_MINUTES: int = 15
|
||||
|
||||
#: Maximum number of log entries to fetch per service per run.
|
||||
MAX_LOG_ENTRIES: int = 500
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Monitored services
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
MONITORED_CLOUD_RUN_SERVICES: list[str] = [
|
||||
"karaoke-backend",
|
||||
"karaoke-decide",
|
||||
"audio-separator",
|
||||
# Gen2 Cloud Functions appear as Cloud Run services in logs
|
||||
"gdrive-validator",
|
||||
"github-runner-manager",
|
||||
"backup-to-aws",
|
||||
"divebar-mirror",
|
||||
"kn-data-sync",
|
||||
"divebar-lookup",
|
||||
"encoding-worker-idle-shutdown",
|
||||
]
|
||||
|
||||
MONITORED_CLOUD_RUN_JOBS: list[str] = [
|
||||
"video-encoding-job",
|
||||
"lyrics-transcription-job",
|
||||
"audio-separation-job",
|
||||
"audio-download-job",
|
||||
]
|
||||
|
||||
# Gen2 Cloud Functions log as cloud_run_revision, not cloud_function.
|
||||
# Their names are in MONITORED_CLOUD_RUN_SERVICES above.
|
||||
MONITORED_CLOUD_FUNCTIONS: list[str] = []
|
||||
|
||||
MONITORED_GCE_INSTANCES: list[str] = [
|
||||
"encoding-worker-a",
|
||||
"encoding-worker-b",
|
||||
"flacfetch-vm",
|
||||
"divebar-sync-vm",
|
||||
]
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Service dependency map
|
||||
# Human-readable description of which services depend on which workers.
|
||||
# Used as LLM context when generating root-cause analysis.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
SERVICE_DEPENDENCY_MAP: str = """\
|
||||
karaoke-backend (main API)
|
||||
→ video-encoding-job (Cloud Run Job, triggered per order)
|
||||
→ lyrics-transcription-job (Cloud Run Job, triggered after audio-sep)
|
||||
→ audio-separation-job (Cloud Run Job, runs on Cloud Run GPU)
|
||||
→ audio-download-job (Cloud Run Job, uses flacfetch-vm)
|
||||
→ gdrive-validator (Cloud Function, validates output files)
|
||||
→ encoding-worker-a/b (GCE VMs, long-running encoding workers)
|
||||
|
||||
karaoke-decide (catalog / discovery API)
|
||||
→ kn_data_sync (Cloud Function, syncs KaraokeNerds catalog)
|
||||
→ divebar_mirror (Cloud Function, mirrors Divebar catalog)
|
||||
→ divebar_lookup (Cloud Function, single-song lookup)
|
||||
|
||||
audio-separator (Cloud Run service, GPU stem separation)
|
||||
→ audio-separation-job (Cloud Run Job, launched by audio-separator)
|
||||
|
||||
flacfetch-vm (GCE VM)
|
||||
→ audio-download-job (initiated by karaoke-backend)
|
||||
|
||||
divebar-sync-vm (GCE VM)
|
||||
→ divebar_mirror (Cloud Function, reads from this VM)
|
||||
|
||||
Shared infrastructure:
|
||||
→ runner_manager (Cloud Function, manages encoding worker lifecycle)
|
||||
→ encoding_worker_idle (Cloud Function, shuts down idle encoding workers)
|
||||
→ backup_to_aws (Cloud Function, daily GCS → S3 backup)
|
||||
"""
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# LLM analysis settings
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
LLM_ANALYSIS_MODEL: str = os.environ.get("LLM_ANALYSIS_MODEL", "gemini-2.0-flash-001")
|
||||
LLM_VERTEX_LOCATION: str = os.environ.get("LLM_VERTEX_LOCATION", "us-central1")
|
||||
|
||||
#: Minimum number of active error patterns required before triggering LLM analysis.
|
||||
MIN_PATTERNS_FOR_ANALYSIS: int = 3
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Spike detection
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: An error count is a spike if it exceeds SPIKE_MULTIPLIER × baseline average.
|
||||
SPIKE_MULTIPLIER: float = 5.0
|
||||
|
||||
#: Minimum absolute error count before spike detection fires (avoids noise).
|
||||
SPIKE_MIN_COUNT: int = 5
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Auto-resolution settings
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: Multiply LOOKBACK_MINUTES by this to get auto-resolve window in minutes.
|
||||
AUTO_RESOLVE_MULTIPLIER: int = 8
|
||||
|
||||
#: Minimum hours before a pattern can be auto-resolved.
|
||||
AUTO_RESOLVE_MIN_HOURS: int = 6
|
||||
|
||||
#: Maximum hours before a pattern is force-resolved even if still active.
|
||||
AUTO_RESOLVE_MAX_HOURS: int = 168 # 1 week
|
||||
|
||||
#: Fallback resolve window (hours) if baseline data is unavailable.
|
||||
AUTO_RESOLVE_FALLBACK_HOURS: int = 48
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Discord settings
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: Maximum Discord messages to send per monitor run (rate-limit guard).
|
||||
MAX_DISCORD_MESSAGES_PER_RUN: int = 10
|
||||
|
||||
#: Discord API hard limit on message length.
|
||||
DISCORD_MAX_MESSAGE_LENGTH: int = 2000
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pattern tracking settings
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: Maximum number of active error patterns to track in Firestore.
|
||||
MAX_ACTIVE_PATTERNS: int = 500
|
||||
|
||||
#: Rolling window used when computing error baselines.
|
||||
ROLLING_WINDOW_DAYS: int = 7
|
||||
|
||||
#: Maximum length of a normalized error message stored in Firestore.
|
||||
MAX_NORMALIZED_MESSAGE_LENGTH: int = 200
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helper functions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def get_llm_enabled() -> bool:
|
||||
"""Return True if LLM analysis is enabled via the LLM_ANALYSIS_ENABLED env var."""
|
||||
return os.environ.get("LLM_ANALYSIS_ENABLED", "false").strip().lower() == "true"
|
||||
|
||||
|
||||
def get_discord_webhook_secret_name() -> str:
|
||||
"""Return the Secret Manager secret name for the Discord webhook URL."""
|
||||
return os.environ.get("DISCORD_WEBHOOK_SECRET_NAME", "error-monitor-discord-webhook")
|
||||
|
||||
|
||||
def get_digest_mode() -> bool:
|
||||
"""Return True if the monitor should run in digest mode (batch alerts instead of immediate).
|
||||
|
||||
In digest mode all alerts are bundled into a single Discord message sent at
|
||||
the end of the run, rather than sending one message per error pattern.
|
||||
"""
|
||||
return os.environ.get("DIGEST_MODE", "false").strip().lower() == "true"
|
||||
@@ -0,0 +1,323 @@
|
||||
"""Discord alert client for the production error monitoring service.
|
||||
|
||||
Provides formatting functions for each alert type, and an ErrorMonitorDiscord
|
||||
webhook client that rate-limits outbound messages to MAX_DISCORD_MESSAGES_PER_RUN
|
||||
per monitor run.
|
||||
|
||||
Usage:
|
||||
from backend.services.error_monitor.discord import (
|
||||
ErrorMonitorDiscord,
|
||||
format_new_pattern_alert,
|
||||
format_incident_alert,
|
||||
format_auto_resolved_alert,
|
||||
format_spike_alert,
|
||||
format_daily_digest,
|
||||
)
|
||||
|
||||
discord = ErrorMonitorDiscord(webhook_url="https://discord.com/api/webhooks/...")
|
||||
msg = format_new_pattern_alert(...)
|
||||
discord.send_message(msg)
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
import requests
|
||||
|
||||
from backend.services.error_monitor.config import (
|
||||
DISCORD_MAX_MESSAGE_LENGTH,
|
||||
MAX_DISCORD_MESSAGES_PER_RUN,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Constants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
SEVERITY_EMOJI: dict[str, str] = {
|
||||
"P0": "🚨",
|
||||
"P1": "🔴",
|
||||
"P2": "⚠️",
|
||||
"P3": "🟡",
|
||||
}
|
||||
|
||||
_EASTERN = ZoneInfo("America/New_York")
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Private helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _now_eastern() -> str:
|
||||
"""Return the current Eastern time as 'HH:MM AM/PM ET' (DST-aware)."""
|
||||
now = datetime.now(tz=_EASTERN)
|
||||
return now.strftime("%I:%M %p ET").lstrip("0")
|
||||
|
||||
|
||||
def _truncate(msg: str) -> str:
|
||||
"""Truncate *msg* to DISCORD_MAX_MESSAGE_LENGTH characters."""
|
||||
if len(msg) <= DISCORD_MAX_MESSAGE_LENGTH:
|
||||
return msg
|
||||
suffix = "…"
|
||||
return msg[: DISCORD_MAX_MESSAGE_LENGTH - len(suffix)] + suffix
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Formatting functions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def format_new_pattern_alert(
|
||||
service: str,
|
||||
resource_type: str,
|
||||
normalized_message: str,
|
||||
sample_message: str,
|
||||
count: int,
|
||||
first_seen: str,
|
||||
) -> str:
|
||||
"""Format a Discord alert for a newly-detected error pattern.
|
||||
|
||||
Args:
|
||||
service: Name of the service emitting the error.
|
||||
resource_type: GCP resource type (e.g. 'cloud_run_service', 'gce_instance').
|
||||
normalized_message: Normalised error pattern string.
|
||||
sample_message: A representative raw log message.
|
||||
count: Number of occurrences seen so far.
|
||||
first_seen: ISO 8601 timestamp when the pattern was first seen.
|
||||
|
||||
Returns:
|
||||
Formatted Discord message string, truncated to 2000 chars.
|
||||
"""
|
||||
lines = [
|
||||
f"🔴 **New Error Pattern Detected**",
|
||||
f"",
|
||||
f"**Service:** {service}",
|
||||
f"**Resource type:** {resource_type}",
|
||||
f"**Pattern:** {normalized_message}",
|
||||
f"**Sample:** {sample_message}",
|
||||
f"**Count:** {count}",
|
||||
f"**First seen:** {first_seen}",
|
||||
f"**Alerted at:** {_now_eastern()}",
|
||||
f"",
|
||||
f"To investigate: `/prod-investigate {service} \"{normalized_message}\"`",
|
||||
]
|
||||
return _truncate("\n".join(lines))
|
||||
|
||||
|
||||
def format_incident_alert(
|
||||
title: str,
|
||||
severity: str,
|
||||
root_cause: str,
|
||||
suggested_fix: str,
|
||||
primary_service: str,
|
||||
patterns: list[dict],
|
||||
) -> str:
|
||||
"""Format a Discord alert for an analysed incident (LLM-generated).
|
||||
|
||||
Args:
|
||||
title: Short incident title.
|
||||
severity: Severity level — one of P0, P1, P2, P3.
|
||||
root_cause: Human-readable root-cause analysis.
|
||||
suggested_fix: Recommended remediation steps.
|
||||
primary_service: The main service involved in the incident.
|
||||
patterns: List of dicts, each with keys: service, normalized_message, count.
|
||||
Only the first 5 are shown; normalized_message is truncated to 80 chars.
|
||||
|
||||
Returns:
|
||||
Formatted Discord message string, truncated to 2000 chars.
|
||||
"""
|
||||
emoji = SEVERITY_EMOJI.get(severity, "⚠️")
|
||||
lines = [
|
||||
f"{emoji} **[{severity}] {title}**",
|
||||
f"",
|
||||
f"**Primary service:** {primary_service}",
|
||||
f"**Root cause:** {root_cause}",
|
||||
f"**Suggested fix:** {suggested_fix}",
|
||||
]
|
||||
|
||||
shown_patterns = patterns[:5]
|
||||
if shown_patterns:
|
||||
lines.append("")
|
||||
lines.append(f"**Patterns ({len(shown_patterns)} of {len(patterns)}):**")
|
||||
for p in shown_patterns:
|
||||
svc = p.get("service", "unknown")
|
||||
msg = p.get("normalized_message", "")[:80]
|
||||
cnt = p.get("count", 0)
|
||||
lines.append(f" • `{svc}` — {msg} ({cnt}x)")
|
||||
|
||||
lines.append("")
|
||||
lines.append(f"**Alerted at:** {_now_eastern()}")
|
||||
|
||||
return _truncate("\n".join(lines))
|
||||
|
||||
|
||||
def format_auto_resolved_alert(resolved_patterns: list[dict]) -> str:
|
||||
"""Format a consolidated auto-resolved alert.
|
||||
|
||||
Args:
|
||||
resolved_patterns: List of dicts, each with keys: service, message, hours_silent.
|
||||
Only the first 8 are shown; message is truncated to 60 chars.
|
||||
|
||||
Returns:
|
||||
Formatted Discord message string, truncated to 2000 chars.
|
||||
"""
|
||||
lines = [
|
||||
f"✅ **Patterns Auto-Resolved**",
|
||||
f"",
|
||||
]
|
||||
|
||||
shown = resolved_patterns[:8]
|
||||
if shown:
|
||||
for p in shown:
|
||||
svc = p.get("service", "unknown")
|
||||
msg = p.get("message", "")[:60]
|
||||
hours = p.get("hours_silent", 0)
|
||||
lines.append(f" • `{svc}` — {msg} (silent {hours}h)")
|
||||
else:
|
||||
lines.append(" *(no patterns)*")
|
||||
|
||||
if len(resolved_patterns) > 8:
|
||||
lines.append(f" … and {len(resolved_patterns) - 8} more")
|
||||
|
||||
lines.append("")
|
||||
lines.append(f"**Resolved at:** {_now_eastern()}")
|
||||
|
||||
return _truncate("\n".join(lines))
|
||||
|
||||
|
||||
def format_spike_alert(
|
||||
service: str,
|
||||
normalized_message: str,
|
||||
current_count: int,
|
||||
rolling_average: float,
|
||||
) -> str:
|
||||
"""Format a Discord alert for an error rate spike.
|
||||
|
||||
Args:
|
||||
service: Name of the service.
|
||||
normalized_message: Normalised error pattern string.
|
||||
current_count: Error count in the current window.
|
||||
rolling_average: Baseline rolling average count.
|
||||
|
||||
Returns:
|
||||
Formatted Discord message string, truncated to 2000 chars.
|
||||
"""
|
||||
if rolling_average > 0:
|
||||
multiplier = current_count / rolling_average
|
||||
multiplier_str = f"{multiplier:.1f}x"
|
||||
else:
|
||||
multiplier_str = "∞x"
|
||||
|
||||
lines = [
|
||||
f"⚠️ **Error Rate Spike Detected**",
|
||||
f"",
|
||||
f"**Service:** {service}",
|
||||
f"**Pattern:** {normalized_message}",
|
||||
f"**Current count:** {current_count}",
|
||||
f"**Normal (rolling avg):** {rolling_average}",
|
||||
f"**Multiplier:** {multiplier_str} above normal",
|
||||
f"",
|
||||
f"**Alerted at:** {_now_eastern()}",
|
||||
]
|
||||
return _truncate("\n".join(lines))
|
||||
|
||||
|
||||
def format_daily_digest(
|
||||
total_errors: int,
|
||||
new_patterns: int,
|
||||
resolved_patterns: int,
|
||||
active_patterns: int,
|
||||
per_service_counts: dict,
|
||||
) -> str:
|
||||
"""Format the daily 24h error digest.
|
||||
|
||||
Args:
|
||||
total_errors: Total error count in the last 24 hours.
|
||||
new_patterns: Number of new error patterns detected.
|
||||
resolved_patterns: Number of patterns auto-resolved.
|
||||
active_patterns: Total number of currently active patterns.
|
||||
per_service_counts: Mapping of service name → error count.
|
||||
Displayed sorted by count descending.
|
||||
|
||||
Returns:
|
||||
Formatted Discord message string, truncated to 2000 chars.
|
||||
"""
|
||||
lines = [
|
||||
f"📊 **Daily Error Digest (24h)**",
|
||||
f"",
|
||||
f"**Total errors:** {total_errors}",
|
||||
f"**New patterns:** {new_patterns}",
|
||||
f"**Resolved patterns:** {resolved_patterns}",
|
||||
f"**Active patterns:** {active_patterns}",
|
||||
]
|
||||
|
||||
if per_service_counts:
|
||||
lines.append("")
|
||||
lines.append("**Per-service breakdown:**")
|
||||
sorted_services = sorted(per_service_counts.items(), key=lambda kv: kv[1], reverse=True)
|
||||
for svc, cnt in sorted_services:
|
||||
lines.append(f" • `{svc}`: {cnt}")
|
||||
|
||||
lines.append("")
|
||||
lines.append(f"**Generated at:** {_now_eastern()}")
|
||||
|
||||
return _truncate("\n".join(lines))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Discord webhook client
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class ErrorMonitorDiscord:
|
||||
"""Webhook client for sending error monitor alerts to Discord.
|
||||
|
||||
Enforces a per-run rate limit of MAX_DISCORD_MESSAGES_PER_RUN to avoid
|
||||
flooding the channel during high-volume incident windows.
|
||||
|
||||
Attributes:
|
||||
webhook_url: The Discord incoming webhook URL.
|
||||
messages_sent: Counter of messages attempted this run (success or failure).
|
||||
"""
|
||||
|
||||
def __init__(self, webhook_url: str) -> None:
|
||||
self.webhook_url = webhook_url
|
||||
self.messages_sent: int = 0
|
||||
|
||||
def send_message(self, content: str) -> bool:
|
||||
"""Send a message to the configured Discord webhook.
|
||||
|
||||
Rate-limits: if messages_sent >= MAX_DISCORD_MESSAGES_PER_RUN, the
|
||||
message is silently skipped and False is returned without incrementing
|
||||
the counter.
|
||||
|
||||
Args:
|
||||
content: The message text to send (should be pre-formatted and
|
||||
pre-truncated to ≤2000 chars).
|
||||
|
||||
Returns:
|
||||
True if the message was sent successfully, False otherwise.
|
||||
"""
|
||||
if self.messages_sent >= MAX_DISCORD_MESSAGES_PER_RUN:
|
||||
logger.warning(
|
||||
"Discord rate limit reached (%d messages sent this run); skipping message.",
|
||||
self.messages_sent,
|
||||
)
|
||||
return False
|
||||
|
||||
try:
|
||||
response = requests.post(
|
||||
self.webhook_url,
|
||||
json={"content": content, "flags": 4}, # SUPPRESS_EMBEDS
|
||||
timeout=30,
|
||||
)
|
||||
response.raise_for_status()
|
||||
logger.debug("Discord message sent successfully.")
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.error("Failed to send Discord message: %s", exc)
|
||||
return False
|
||||
finally:
|
||||
self.messages_sent += 1
|
||||
@@ -0,0 +1,489 @@
|
||||
"""Firestore adapter for the production error monitoring service.
|
||||
|
||||
Handles all CRUD for three Firestore collections:
|
||||
- error_patterns: normalized error patterns (pattern_id as doc ID)
|
||||
- error_incidents: groups of related patterns (incident_id as doc ID)
|
||||
- discord_alerts: audit trail for Discord messages (alert_id as doc ID)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import secrets
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Any
|
||||
|
||||
from backend.services.error_monitor.config import (
|
||||
AUTO_RESOLVE_FALLBACK_HOURS,
|
||||
AUTO_RESOLVE_MAX_HOURS,
|
||||
AUTO_RESOLVE_MIN_HOURS,
|
||||
AUTO_RESOLVE_MULTIPLIER,
|
||||
ROLLING_WINDOW_DAYS,
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Data classes
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class PatternData:
|
||||
"""Input data for creating or updating an error pattern."""
|
||||
|
||||
pattern_id: str
|
||||
service: str
|
||||
resource_type: str
|
||||
normalized_message: str
|
||||
sample_message: str
|
||||
count: int
|
||||
timestamp: datetime
|
||||
|
||||
|
||||
@dataclass
|
||||
class UpsertResult:
|
||||
"""Result of an upsert_pattern operation."""
|
||||
|
||||
pattern_id: str
|
||||
is_new: bool
|
||||
previous_status: str | None = None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Collection names
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_COL_PATTERNS = "error_patterns"
|
||||
_COL_INCIDENTS = "error_incidents"
|
||||
_COL_ALERTS = "discord_alerts"
|
||||
|
||||
# Statuses that can be queried as "active"
|
||||
_ACTIVE_STATUSES = ["new", "acknowledged", "known"]
|
||||
|
||||
# Statuses from which a pattern is reactivated when the error recurs
|
||||
_REACTIVATION_STATUSES = {"auto_resolved", "fixed"}
|
||||
|
||||
# Statuses eligible for auto-resolve checking
|
||||
_AUTO_RESOLVE_ELIGIBLE = {"new", "acknowledged"}
|
||||
|
||||
|
||||
def _utcnow() -> datetime:
|
||||
"""Return current UTC datetime."""
|
||||
return datetime.now(tz=timezone.utc)
|
||||
|
||||
|
||||
def _iso(dt: datetime) -> str:
|
||||
"""Serialise a datetime to ISO-8601 string."""
|
||||
return dt.isoformat()
|
||||
|
||||
|
||||
def _generate_id(prefix: str, fmt: str) -> str:
|
||||
"""Generate a time-stamped ID with 8 hex random suffix.
|
||||
|
||||
Args:
|
||||
prefix: e.g. "inc" or "alert"
|
||||
fmt: strftime format string applied to utcnow(), e.g. "%Y%m%d_%H%M"
|
||||
"""
|
||||
ts = _utcnow().strftime(fmt)
|
||||
suffix = secrets.token_hex(4) # 8 hex chars
|
||||
return f"{prefix}_{ts}_{suffix}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Adapter
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class ErrorPatternsAdapter:
|
||||
"""Firestore CRUD adapter for error monitoring collections.
|
||||
|
||||
Args:
|
||||
db: Optional Firestore client. If omitted, a default client for the
|
||||
``nomadkaraoke`` GCP project is constructed on first use.
|
||||
"""
|
||||
|
||||
def __init__(self, db=None) -> None:
|
||||
if db is None:
|
||||
from google.cloud import firestore # type: ignore[import]
|
||||
|
||||
db = firestore.Client(project="nomadkaraoke")
|
||||
self._db = db
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Patterns — read
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def get_pattern(self, pattern_id: str) -> dict | None:
|
||||
"""Return the pattern document as a dict, or None if it doesn't exist.
|
||||
|
||||
Args:
|
||||
pattern_id: Firestore document ID in the ``error_patterns`` collection.
|
||||
"""
|
||||
snap = self._db.collection(_COL_PATTERNS).document(pattern_id).get()
|
||||
if not snap.exists:
|
||||
return None
|
||||
return snap.to_dict()
|
||||
|
||||
def get_active_patterns(self) -> list[dict]:
|
||||
"""Return all patterns with status in [new, acknowledged, known]."""
|
||||
query = self._db.collection(_COL_PATTERNS).where(
|
||||
"status", "in", _ACTIVE_STATUSES
|
||||
)
|
||||
return [snap.to_dict() for snap in query.stream()]
|
||||
|
||||
def get_patterns_for_auto_resolve(self) -> list[dict]:
|
||||
"""Return active patterns eligible for auto-resolution (new or acknowledged)."""
|
||||
query = self._db.collection(_COL_PATTERNS).where(
|
||||
"status", "in", list(_AUTO_RESOLVE_ELIGIBLE)
|
||||
)
|
||||
return [snap.to_dict() for snap in query.stream()]
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Patterns — upsert
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def upsert_pattern(self, data: PatternData) -> UpsertResult:
|
||||
"""Create a new pattern or update an existing one.
|
||||
|
||||
New pattern:
|
||||
- status="new", severity="P3"
|
||||
- Initialize ``rolling_counts`` with one entry for this run.
|
||||
|
||||
Existing pattern:
|
||||
- Increment ``total_count`` and append to ``rolling_counts``.
|
||||
- If status was ``auto_resolved`` or ``fixed`` → reactivate (set status="new").
|
||||
- Prune ``rolling_counts`` entries older than ROLLING_WINDOW_DAYS.
|
||||
|
||||
Returns:
|
||||
UpsertResult with ``is_new=True`` for brand-new patterns *and* for
|
||||
reactivated patterns (auto_resolved → new, fixed → new).
|
||||
"""
|
||||
doc_ref = self._db.collection(_COL_PATTERNS).document(data.pattern_id)
|
||||
snap = doc_ref.get()
|
||||
ts_iso = _iso(data.timestamp)
|
||||
|
||||
new_rolling_entry = {"ts": ts_iso, "count": data.count}
|
||||
|
||||
if not snap.exists:
|
||||
# ── brand new pattern ──────────────────────────────────────────
|
||||
doc_ref.set(
|
||||
{
|
||||
"pattern_id": data.pattern_id,
|
||||
"service": data.service,
|
||||
"resource_type": data.resource_type,
|
||||
"normalized_message": data.normalized_message,
|
||||
"sample_message": data.sample_message,
|
||||
"status": "new",
|
||||
"severity": "P3",
|
||||
"total_count": data.count,
|
||||
"rolling_counts": [new_rolling_entry],
|
||||
"first_seen": ts_iso,
|
||||
"last_seen": ts_iso,
|
||||
"alerted_at": None,
|
||||
"created_at": ts_iso,
|
||||
"updated_at": ts_iso,
|
||||
}
|
||||
)
|
||||
return UpsertResult(pattern_id=data.pattern_id, is_new=True)
|
||||
|
||||
# ── existing pattern ───────────────────────────────────────────────
|
||||
existing = snap.to_dict()
|
||||
prev_status = existing.get("status", "new")
|
||||
|
||||
rolling_counts: list[dict] = list(existing.get("rolling_counts") or [])
|
||||
rolling_counts.append(new_rolling_entry)
|
||||
rolling_counts = self._prune_rolling_counts(rolling_counts)
|
||||
|
||||
update: dict[str, Any] = {
|
||||
"total_count": existing.get("total_count", 0) + data.count,
|
||||
"rolling_counts": rolling_counts,
|
||||
"last_seen": ts_iso,
|
||||
"sample_message": data.sample_message,
|
||||
"updated_at": ts_iso,
|
||||
}
|
||||
|
||||
# Reactivation: pattern was considered resolved/fixed but error recurred
|
||||
is_new = False
|
||||
if prev_status in _REACTIVATION_STATUSES:
|
||||
update["status"] = "new"
|
||||
update["alerted_at"] = None
|
||||
is_new = True
|
||||
|
||||
doc_ref.update(update)
|
||||
return UpsertResult(
|
||||
pattern_id=data.pattern_id,
|
||||
is_new=is_new,
|
||||
previous_status=prev_status,
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Patterns — status changes
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def auto_resolve_pattern(self, pattern_id: str, hours_silent: float) -> None:
|
||||
"""Mark a pattern as auto-resolved.
|
||||
|
||||
Args:
|
||||
pattern_id: Firestore document ID.
|
||||
hours_silent: How many hours the pattern has been silent.
|
||||
"""
|
||||
self._db.collection(_COL_PATTERNS).document(pattern_id).update(
|
||||
{
|
||||
"status": "auto_resolved",
|
||||
"auto_resolve_hours_silent": hours_silent,
|
||||
"auto_resolved_at": _iso(_utcnow()),
|
||||
"updated_at": _iso(_utcnow()),
|
||||
}
|
||||
)
|
||||
|
||||
def resolve_pattern(
|
||||
self,
|
||||
pattern_id: str,
|
||||
pr_url: str | None = None,
|
||||
note: str = "",
|
||||
) -> None:
|
||||
"""Mark a pattern as fixed (human-resolved).
|
||||
|
||||
Args:
|
||||
pattern_id: Firestore document ID.
|
||||
pr_url: Optional URL of the PR that fixed this pattern.
|
||||
note: Optional human-readable note about the fix.
|
||||
"""
|
||||
now_iso = _iso(_utcnow())
|
||||
self._db.collection(_COL_PATTERNS).document(pattern_id).update(
|
||||
{
|
||||
"status": "fixed",
|
||||
"fixed_by": {
|
||||
"pr_url": pr_url,
|
||||
"note": note,
|
||||
"fixed_at": now_iso,
|
||||
},
|
||||
"updated_at": now_iso,
|
||||
}
|
||||
)
|
||||
|
||||
def update_pattern_alerted(self, pattern_id: str) -> None:
|
||||
"""Set the ``alerted_at`` timestamp on a pattern.
|
||||
|
||||
Called after a Discord alert has been successfully sent for the pattern.
|
||||
"""
|
||||
self._db.collection(_COL_PATTERNS).document(pattern_id).update(
|
||||
{
|
||||
"alerted_at": _iso(_utcnow()),
|
||||
"updated_at": _iso(_utcnow()),
|
||||
}
|
||||
)
|
||||
|
||||
def merge_pattern(self, source_id: str, target_id: str, reason: str) -> None:
|
||||
"""Merge *source* into *target* by muting the source pattern.
|
||||
|
||||
Args:
|
||||
source_id: Pattern to mute.
|
||||
target_id: Pattern that the source is being merged into.
|
||||
reason: Human-readable rationale for the merge.
|
||||
"""
|
||||
self._db.collection(_COL_PATTERNS).document(source_id).update(
|
||||
{
|
||||
"status": "muted",
|
||||
"merged_into": target_id,
|
||||
"merge_reason": reason,
|
||||
"updated_at": _iso(_utcnow()),
|
||||
}
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Auto-resolve logic
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def check_auto_resolve(self, pattern: dict) -> float | None:
|
||||
"""Determine whether *pattern* should be auto-resolved.
|
||||
|
||||
Computes the frequency-aware silence threshold and compares it against
|
||||
how long the pattern has been silent.
|
||||
|
||||
Args:
|
||||
pattern: A pattern document dict (as returned by get_pattern).
|
||||
|
||||
Returns:
|
||||
The number of hours the pattern has been silent if the threshold is
|
||||
exceeded, or ``None`` if the pattern should not yet be resolved.
|
||||
"""
|
||||
status = pattern.get("status", "")
|
||||
if status not in _AUTO_RESOLVE_ELIGIBLE:
|
||||
return None
|
||||
|
||||
last_seen_str = pattern.get("last_seen")
|
||||
if not last_seen_str:
|
||||
return None
|
||||
|
||||
last_seen = datetime.fromisoformat(last_seen_str)
|
||||
now = datetime.now(tz=timezone.utc)
|
||||
|
||||
# Ensure both datetimes are tz-aware for subtraction
|
||||
if last_seen.tzinfo is None:
|
||||
last_seen = last_seen.replace(tzinfo=timezone.utc)
|
||||
|
||||
hours_silent = (now - last_seen).total_seconds() / 3600.0
|
||||
|
||||
rolling_counts = pattern.get("rolling_counts") or []
|
||||
threshold = self._compute_resolve_threshold(rolling_counts)
|
||||
|
||||
if hours_silent >= threshold:
|
||||
return hours_silent
|
||||
return None
|
||||
|
||||
def _compute_resolve_threshold(self, rolling_counts: list[dict]) -> float:
|
||||
"""Compute the frequency-aware auto-resolve threshold in hours.
|
||||
|
||||
Algorithm:
|
||||
1. If fewer than 3 data points → return AUTO_RESOLVE_FALLBACK_HOURS (48).
|
||||
2. Compute mean interval between consecutive timestamps.
|
||||
3. threshold = mean_interval_hours * AUTO_RESOLVE_MULTIPLIER.
|
||||
4. Clamp result to [AUTO_RESOLVE_MIN_HOURS, AUTO_RESOLVE_MAX_HOURS].
|
||||
|
||||
Args:
|
||||
rolling_counts: List of ``{"ts": ISO-string, "count": int}`` dicts.
|
||||
|
||||
Returns:
|
||||
Threshold in hours (float).
|
||||
"""
|
||||
if len(rolling_counts) < 3:
|
||||
return float(AUTO_RESOLVE_FALLBACK_HOURS)
|
||||
|
||||
# Parse timestamps and sort ascending
|
||||
timestamps: list[datetime] = []
|
||||
for entry in rolling_counts:
|
||||
ts = datetime.fromisoformat(entry["ts"])
|
||||
if ts.tzinfo is None:
|
||||
ts = ts.replace(tzinfo=timezone.utc)
|
||||
timestamps.append(ts)
|
||||
timestamps.sort()
|
||||
|
||||
# Compute pairwise intervals
|
||||
intervals_hours: list[float] = []
|
||||
for i in range(1, len(timestamps)):
|
||||
delta = (timestamps[i] - timestamps[i - 1]).total_seconds() / 3600.0
|
||||
intervals_hours.append(delta)
|
||||
|
||||
if not intervals_hours:
|
||||
return float(AUTO_RESOLVE_FALLBACK_HOURS)
|
||||
|
||||
mean_interval = sum(intervals_hours) / len(intervals_hours)
|
||||
threshold = mean_interval * AUTO_RESOLVE_MULTIPLIER
|
||||
|
||||
# Clamp to configured bounds
|
||||
threshold = max(float(AUTO_RESOLVE_MIN_HOURS), threshold)
|
||||
threshold = min(float(AUTO_RESOLVE_MAX_HOURS), threshold)
|
||||
return threshold
|
||||
|
||||
def _prune_rolling_counts(self, rolling_counts: list[dict]) -> list[dict]:
|
||||
"""Remove rolling_count entries older than ROLLING_WINDOW_DAYS.
|
||||
|
||||
Args:
|
||||
rolling_counts: List of ``{"ts": ISO-string, "count": int}`` dicts.
|
||||
|
||||
Returns:
|
||||
Filtered list with only recent entries.
|
||||
"""
|
||||
if not rolling_counts:
|
||||
return []
|
||||
|
||||
cutoff = _utcnow() - timedelta(days=ROLLING_WINDOW_DAYS)
|
||||
result = []
|
||||
for entry in rolling_counts:
|
||||
ts = datetime.fromisoformat(entry["ts"])
|
||||
if ts.tzinfo is None:
|
||||
ts = ts.replace(tzinfo=timezone.utc)
|
||||
if ts >= cutoff:
|
||||
result.append(entry)
|
||||
return result
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Incidents
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def create_incident(
|
||||
self,
|
||||
title: str,
|
||||
root_cause: str,
|
||||
severity: str,
|
||||
suggested_fix: str,
|
||||
primary_service: str,
|
||||
pattern_ids: list[str],
|
||||
used_llm: bool,
|
||||
) -> str:
|
||||
"""Create a new incident document and link associated patterns.
|
||||
|
||||
Args:
|
||||
title: Short human-readable title.
|
||||
root_cause: LLM-generated or manual root-cause description.
|
||||
severity: Severity string (e.g. "P1", "P2", "P3").
|
||||
suggested_fix: Suggested remediation steps.
|
||||
primary_service: The primary affected service.
|
||||
pattern_ids: List of pattern_ids that belong to this incident.
|
||||
used_llm: Whether LLM analysis was used to generate this incident.
|
||||
|
||||
Returns:
|
||||
The generated incident_id string.
|
||||
"""
|
||||
incident_id = _generate_id("inc", "%Y%m%d_%H%M")
|
||||
now_iso = _iso(_utcnow())
|
||||
|
||||
self._db.collection(_COL_INCIDENTS).document(incident_id).set(
|
||||
{
|
||||
"incident_id": incident_id,
|
||||
"title": title,
|
||||
"root_cause": root_cause,
|
||||
"severity": severity,
|
||||
"suggested_fix": suggested_fix,
|
||||
"primary_service": primary_service,
|
||||
"pattern_ids": pattern_ids,
|
||||
"used_llm": used_llm,
|
||||
"created_at": now_iso,
|
||||
"updated_at": now_iso,
|
||||
"status": "open",
|
||||
}
|
||||
)
|
||||
|
||||
# Link each pattern back to this incident
|
||||
for pattern_id in pattern_ids:
|
||||
self._db.collection(_COL_PATTERNS).document(pattern_id).update(
|
||||
{
|
||||
"incident_id": incident_id,
|
||||
"updated_at": now_iso,
|
||||
}
|
||||
)
|
||||
|
||||
return incident_id
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Discord audit trail
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def log_discord_alert(
|
||||
self,
|
||||
alert_type: str,
|
||||
content: str,
|
||||
success: bool,
|
||||
metadata: dict | None = None,
|
||||
) -> None:
|
||||
"""Write an audit record to the ``discord_alerts`` collection.
|
||||
|
||||
Args:
|
||||
alert_type: Category of alert (e.g. "new_pattern", "digest").
|
||||
content: The message body that was (or would have been) sent.
|
||||
success: Whether the Discord webhook call succeeded.
|
||||
metadata: Optional additional data (pattern_ids, incident_id, etc.).
|
||||
"""
|
||||
alert_id = _generate_id("alert", "%Y%m%d_%H%M%S")
|
||||
now_iso = _iso(_utcnow())
|
||||
|
||||
self._db.collection(_COL_ALERTS).document(alert_id).set(
|
||||
{
|
||||
"alert_id": alert_id,
|
||||
"alert_type": alert_type,
|
||||
"content": content,
|
||||
"success": success,
|
||||
"metadata": metadata or {},
|
||||
"created_at": now_iso,
|
||||
}
|
||||
)
|
||||
@@ -0,0 +1,106 @@
|
||||
"""Known infrastructure noise patterns for the production error monitoring service.
|
||||
|
||||
Filters out transient, expected log messages so the error monitor is not flooded
|
||||
with Cloud Run cold-start probes, deploy events, spot VM preemptions, and similar
|
||||
operational noise.
|
||||
|
||||
Usage:
|
||||
from backend.services.error_monitor.known_issues import should_ignore
|
||||
|
||||
reason = should_ignore(service, message)
|
||||
if reason:
|
||||
# skip this log entry — it's known infrastructure noise
|
||||
...
|
||||
"""
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class IgnoreReason:
|
||||
"""Describes why a log entry should be suppressed.
|
||||
|
||||
Attributes:
|
||||
pattern_name: Machine-readable name of the matching ignore pattern.
|
||||
reason: Human-readable explanation of why this is infrastructure noise.
|
||||
"""
|
||||
|
||||
pattern_name: str
|
||||
reason: str
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Compiled ignore patterns (applied in order)
|
||||
# All patterns use re.IGNORECASE for case-insensitive matching.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_IGNORE_PATTERNS: list[tuple[str, re.Pattern, str]] = [
|
||||
(
|
||||
"startup_probe",
|
||||
re.compile(r"startup probe failed", re.IGNORECASE),
|
||||
"Cloud Run cold start — startup probe failure is transient and expected",
|
||||
),
|
||||
(
|
||||
"ready_condition",
|
||||
re.compile(r"ready condition status changed", re.IGNORECASE),
|
||||
"Cloud Run deploy event — readiness condition change is expected on deploy",
|
||||
),
|
||||
(
|
||||
"spot_preemption",
|
||||
re.compile(r"instance was preempted", re.IGNORECASE),
|
||||
"Spot/preemptible VM preemption — expected behaviour for spot instances",
|
||||
),
|
||||
(
|
||||
"idle_shutdown",
|
||||
re.compile(r"stopping idle encoding worker", re.IGNORECASE),
|
||||
"Encoding worker idle shutdown — expected behaviour when no jobs are queued",
|
||||
),
|
||||
(
|
||||
"health_check_404",
|
||||
re.compile(
|
||||
r"(?:GET|HEAD)\b[^\n]*?/health[^\n]*?\b(?:404|not found)\b",
|
||||
re.IGNORECASE,
|
||||
),
|
||||
"Load balancer health check returning 404 — transient during cold start",
|
||||
),
|
||||
(
|
||||
"scheduler_retry",
|
||||
re.compile(r"cloud scheduler.*retry", re.IGNORECASE),
|
||||
"Cloud Scheduler transient retry — expected on intermittent job failures",
|
||||
),
|
||||
(
|
||||
"runner_startup",
|
||||
re.compile(r"github\.runner\..*(?:starting|stopping|idle)", re.IGNORECASE),
|
||||
"GitHub Actions runner lifecycle event — expected during runner start/stop",
|
||||
),
|
||||
(
|
||||
"container_shutdown",
|
||||
re.compile(r"container called exit\(0\)", re.IGNORECASE),
|
||||
"Container clean exit — exit code 0 indicates a graceful, expected shutdown",
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def should_ignore(service: str, message: str) -> IgnoreReason | None:
|
||||
"""Check whether a log entry matches a known infrastructure noise pattern.
|
||||
|
||||
The check is purely message-based; ``service`` is accepted for API symmetry
|
||||
and future service-specific overrides but does not gate any pattern today.
|
||||
|
||||
Args:
|
||||
service: The originating service name (e.g. ``"karaoke-backend"``).
|
||||
message: The raw log message string to evaluate.
|
||||
|
||||
Returns:
|
||||
An :class:`IgnoreReason` if the message matches a known noise pattern,
|
||||
or ``None`` if the message should be processed normally.
|
||||
"""
|
||||
if not message:
|
||||
return None
|
||||
|
||||
for pattern_name, pattern, reason in _IGNORE_PATTERNS:
|
||||
if pattern.search(message):
|
||||
return IgnoreReason(pattern_name=pattern_name, reason=reason)
|
||||
|
||||
return None
|
||||
@@ -0,0 +1,429 @@
|
||||
"""LLM-powered incident analysis and duplicate detection for the error monitor.
|
||||
|
||||
Uses Gemini Flash via the google-generativeai library to:
|
||||
1. Group related error patterns into incidents with root-cause analysis.
|
||||
2. Identify near-duplicate patterns that the regex normalizer missed.
|
||||
|
||||
Both functions fall back gracefully when the LLM is unavailable or returns
|
||||
unparseable output.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from collections import defaultdict
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from google import genai
|
||||
from google.genai import types as genai_types
|
||||
|
||||
from backend.services.error_monitor.config import (
|
||||
GCP_PROJECT,
|
||||
LLM_ANALYSIS_MODEL,
|
||||
LLM_VERTEX_LOCATION,
|
||||
MIN_PATTERNS_FOR_ANALYSIS,
|
||||
SERVICE_DEPENDENCY_MAP,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Dataclasses
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class Incident:
|
||||
"""A grouped incident produced by LLM analysis or the fallback grouper."""
|
||||
|
||||
title: str
|
||||
root_cause: str | None
|
||||
severity: str # P0 / P1 / P2 / P3
|
||||
suggested_fix: str | None
|
||||
primary_service: str
|
||||
pattern_indices: list[int]
|
||||
|
||||
|
||||
@dataclass
|
||||
class IncidentAnalysis:
|
||||
"""The result of an incident grouping pass."""
|
||||
|
||||
incidents: list[Incident]
|
||||
used_llm: bool = True
|
||||
|
||||
|
||||
@dataclass
|
||||
class DuplicateGroup:
|
||||
"""A canonical pattern plus the new patterns that duplicate it."""
|
||||
|
||||
canonical_index: int
|
||||
duplicate_indices: list[int]
|
||||
reason: str
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# LLM call helper
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_RETRY_ATTEMPTS = 3
|
||||
_RETRY_BASE_DELAY = 2.0 # seconds
|
||||
|
||||
|
||||
def _call_llm(
|
||||
system_prompt: str,
|
||||
user_prompt: str,
|
||||
temperature: float = 0.2,
|
||||
) -> str:
|
||||
"""Call Gemini and return the response text.
|
||||
|
||||
Retries up to _RETRY_ATTEMPTS times with exponential back-off on failure.
|
||||
|
||||
Args:
|
||||
system_prompt: System instruction for the model.
|
||||
user_prompt: User-facing prompt content.
|
||||
temperature: Sampling temperature (lower = more deterministic).
|
||||
|
||||
Returns:
|
||||
The model's response text.
|
||||
|
||||
Raises:
|
||||
Exception: Re-raises the last exception after all retries are exhausted.
|
||||
"""
|
||||
last_exc: Exception | None = None
|
||||
|
||||
for attempt in range(_RETRY_ATTEMPTS):
|
||||
try:
|
||||
client = genai.Client(
|
||||
vertexai=True,
|
||||
project=GCP_PROJECT,
|
||||
location=LLM_VERTEX_LOCATION,
|
||||
)
|
||||
response = client.models.generate_content(
|
||||
model=LLM_ANALYSIS_MODEL,
|
||||
contents=user_prompt,
|
||||
config=genai_types.GenerateContentConfig(
|
||||
system_instruction=system_prompt,
|
||||
temperature=temperature,
|
||||
),
|
||||
)
|
||||
return response.text
|
||||
except Exception as exc: # noqa: BLE001
|
||||
last_exc = exc
|
||||
delay = _RETRY_BASE_DELAY * (2**attempt)
|
||||
logger.warning(
|
||||
"LLM call failed (attempt %d/%d): %s — retrying in %.1fs",
|
||||
attempt + 1,
|
||||
_RETRY_ATTEMPTS,
|
||||
exc,
|
||||
delay,
|
||||
)
|
||||
time.sleep(delay)
|
||||
|
||||
raise last_exc # type: ignore[misc]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Response parsing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_MD_CODE_BLOCK_RE = re.compile(r"```(?:json)?\s*([\s\S]*?)```", re.IGNORECASE)
|
||||
|
||||
|
||||
def _strip_markdown_code_block(text: str) -> str:
|
||||
"""Remove surrounding ```json ... ``` or ``` ... ``` fences if present."""
|
||||
match = _MD_CODE_BLOCK_RE.search(text)
|
||||
if match:
|
||||
return match.group(1).strip()
|
||||
return text.strip()
|
||||
|
||||
|
||||
def _parse_llm_response(text: str) -> IncidentAnalysis | None:
|
||||
"""Parse the LLM response text into an IncidentAnalysis.
|
||||
|
||||
Args:
|
||||
text: Raw LLM output, possibly wrapped in a markdown code block.
|
||||
|
||||
Returns:
|
||||
IncidentAnalysis on success, None on any parse failure.
|
||||
"""
|
||||
if not text:
|
||||
return None
|
||||
|
||||
cleaned = _strip_markdown_code_block(text)
|
||||
|
||||
try:
|
||||
data = json.loads(cleaned)
|
||||
except json.JSONDecodeError:
|
||||
logger.warning("LLM returned non-JSON response (first 200 chars): %.200s", text)
|
||||
return None
|
||||
|
||||
if "incidents" not in data:
|
||||
logger.warning("LLM response missing 'incidents' key: %s", list(data.keys()))
|
||||
return None
|
||||
|
||||
incidents: list[Incident] = []
|
||||
for raw in data["incidents"]:
|
||||
try:
|
||||
incidents.append(
|
||||
Incident(
|
||||
title=raw.get("title", "Unknown incident"),
|
||||
root_cause=raw.get("root_cause"),
|
||||
severity=raw.get("severity", "P2"),
|
||||
suggested_fix=raw.get("suggested_fix"),
|
||||
primary_service=raw.get("primary_service", "unknown"),
|
||||
pattern_indices=raw.get("pattern_indices", []),
|
||||
)
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("Skipping malformed incident entry: %s — %s", raw, exc)
|
||||
|
||||
return IncidentAnalysis(incidents=incidents, used_llm=True)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fallback grouper
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _fallback_group_by_service(patterns: list[dict]) -> IncidentAnalysis:
|
||||
"""Group patterns by service name when the LLM is unavailable.
|
||||
|
||||
Each unique service becomes one P2 incident containing all of that
|
||||
service's pattern indices.
|
||||
|
||||
Args:
|
||||
patterns: List of error pattern dicts (must include 'service' key).
|
||||
|
||||
Returns:
|
||||
IncidentAnalysis with used_llm=False.
|
||||
"""
|
||||
service_to_indices: dict[str, list[int]] = defaultdict(list)
|
||||
for idx, pattern in enumerate(patterns):
|
||||
service = pattern.get("service", "unknown")
|
||||
service_to_indices[service].append(idx)
|
||||
|
||||
incidents = [
|
||||
Incident(
|
||||
title=f"Errors in {service}",
|
||||
root_cause=None,
|
||||
severity="P2",
|
||||
suggested_fix=None,
|
||||
primary_service=service,
|
||||
pattern_indices=indices,
|
||||
)
|
||||
for service, indices in service_to_indices.items()
|
||||
]
|
||||
|
||||
return IncidentAnalysis(incidents=incidents, used_llm=False)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main analysis functions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_ANALYZE_SYSTEM_PROMPT = f"""\
|
||||
You are an expert site-reliability engineer for Nomad Karaoke, a karaoke video
|
||||
generation platform running on Google Cloud Platform.
|
||||
|
||||
Your task is to analyse a list of active error patterns detected across
|
||||
production services, group them into meaningful incidents, and identify root
|
||||
causes.
|
||||
|
||||
## Service dependency map
|
||||
|
||||
{SERVICE_DEPENDENCY_MAP}
|
||||
|
||||
## Severity definitions
|
||||
|
||||
- P0: Service is completely down / total outage
|
||||
- P1: Major pipeline broken — orders cannot complete
|
||||
- P2: Degraded performance — some orders affected
|
||||
- P3: Minor / cosmetic issue with no direct user impact
|
||||
|
||||
## Output format
|
||||
|
||||
Respond with a single JSON object (no markdown fencing) with this schema:
|
||||
|
||||
{{
|
||||
"incidents": [
|
||||
{{
|
||||
"title": "<short human-readable title>",
|
||||
"root_cause": "<root cause or null>",
|
||||
"severity": "P0|P1|P2|P3",
|
||||
"suggested_fix": "<actionable suggestion or null>",
|
||||
"primary_service": "<service name>",
|
||||
"pattern_indices": [<list of 0-based indices from the input list>]
|
||||
}}
|
||||
]
|
||||
}}
|
||||
|
||||
Group related patterns together into a single incident where they share a
|
||||
common root cause (e.g., a downstream dependency failure). Unrelated patterns
|
||||
should be in separate incidents. Every input pattern must appear in exactly
|
||||
one incident.
|
||||
"""
|
||||
|
||||
_DEDUP_SYSTEM_PROMPT = """\
|
||||
You are an expert at identifying near-duplicate error patterns in production
|
||||
monitoring systems.
|
||||
|
||||
Your task is to compare a list of NEW error patterns against a list of
|
||||
EXISTING (canonical) patterns and identify which new patterns are essentially
|
||||
duplicates of existing ones — i.e., they describe the same underlying error
|
||||
but with slightly different wording due to variation in log messages.
|
||||
|
||||
## Output format
|
||||
|
||||
Respond with a single JSON object (no markdown fencing) with this schema:
|
||||
|
||||
{
|
||||
"duplicates": [
|
||||
{
|
||||
"canonical_index": <0-based index into the EXISTING patterns list>,
|
||||
"duplicate_indices": [<0-based indices into the NEW patterns list>],
|
||||
"reason": "<brief explanation>"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
If no duplicates are found, return {"duplicates": []}.
|
||||
"""
|
||||
|
||||
|
||||
def analyze_patterns(patterns: list[dict]) -> IncidentAnalysis | None:
|
||||
"""Group error patterns into incidents using Gemini LLM analysis.
|
||||
|
||||
Returns None if there are fewer than MIN_PATTERNS_FOR_ANALYSIS patterns.
|
||||
Falls back to service-based grouping when the LLM fails or returns an
|
||||
unparseable response.
|
||||
|
||||
Args:
|
||||
patterns: List of error pattern dicts to analyse.
|
||||
|
||||
Returns:
|
||||
IncidentAnalysis, or None if insufficient patterns.
|
||||
"""
|
||||
if len(patterns) < MIN_PATTERNS_FOR_ANALYSIS:
|
||||
return None
|
||||
|
||||
user_prompt = _build_analyze_user_prompt(patterns)
|
||||
|
||||
try:
|
||||
raw_response = _call_llm(_ANALYZE_SYSTEM_PROMPT, user_prompt, temperature=0.2)
|
||||
result = _parse_llm_response(raw_response)
|
||||
if result is not None:
|
||||
return result
|
||||
|
||||
logger.warning("LLM returned unparseable response; falling back to service grouping")
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("LLM analysis failed: %s — falling back to service grouping", exc)
|
||||
|
||||
return _fallback_group_by_service(patterns)
|
||||
|
||||
|
||||
def find_duplicate_patterns(
|
||||
new_patterns: list[dict],
|
||||
existing_patterns: list[dict],
|
||||
) -> list[DuplicateGroup]:
|
||||
"""Identify near-duplicate patterns that the regex normalizer missed.
|
||||
|
||||
Returns an empty list if either input list is empty, or if the LLM call
|
||||
fails for any reason.
|
||||
|
||||
Args:
|
||||
new_patterns: Newly detected error patterns to check.
|
||||
existing_patterns: Currently tracked (canonical) error patterns.
|
||||
|
||||
Returns:
|
||||
List of DuplicateGroup objects (may be empty).
|
||||
"""
|
||||
if not new_patterns or not existing_patterns:
|
||||
return []
|
||||
|
||||
user_prompt = _build_dedup_user_prompt(new_patterns, existing_patterns)
|
||||
|
||||
try:
|
||||
raw_response = _call_llm(_DEDUP_SYSTEM_PROMPT, user_prompt, temperature=0.1)
|
||||
return _parse_duplicate_response(raw_response)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("Duplicate detection LLM call failed: %s", exc)
|
||||
return []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Prompt builders
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_analyze_user_prompt(patterns: list[dict]) -> str:
|
||||
"""Build the user prompt listing patterns for incident grouping."""
|
||||
lines = ["Error patterns to analyse:\n"]
|
||||
for idx, pattern in enumerate(patterns):
|
||||
service = pattern.get("service", "unknown")
|
||||
message = pattern.get("normalized_message", "")
|
||||
count = pattern.get("count", 0)
|
||||
lines.append(f"[{idx}] service={service!r} count={count} message={message!r}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _build_dedup_user_prompt(
|
||||
new_patterns: list[dict],
|
||||
existing_patterns: list[dict],
|
||||
) -> str:
|
||||
"""Build the user prompt listing existing and new patterns for dedup."""
|
||||
lines = ["EXISTING (canonical) patterns:\n"]
|
||||
for idx, pattern in enumerate(existing_patterns):
|
||||
service = pattern.get("service", "unknown")
|
||||
message = pattern.get("normalized_message", "")
|
||||
lines.append(f"[{idx}] service={service!r} message={message!r}")
|
||||
|
||||
lines.append("\nNEW patterns to check:\n")
|
||||
for idx, pattern in enumerate(new_patterns):
|
||||
service = pattern.get("service", "unknown")
|
||||
message = pattern.get("normalized_message", "")
|
||||
lines.append(f"[{idx}] service={service!r} message={message!r}")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _parse_duplicate_response(text: str) -> list[DuplicateGroup]:
|
||||
"""Parse the LLM response for duplicate detection.
|
||||
|
||||
Args:
|
||||
text: Raw LLM output.
|
||||
|
||||
Returns:
|
||||
List of DuplicateGroup objects, or [] on any failure.
|
||||
"""
|
||||
if not text:
|
||||
return []
|
||||
|
||||
cleaned = _strip_markdown_code_block(text)
|
||||
|
||||
try:
|
||||
data = json.loads(cleaned)
|
||||
except json.JSONDecodeError:
|
||||
logger.warning("Dedup LLM returned non-JSON: %.200s", text)
|
||||
return []
|
||||
|
||||
if "duplicates" not in data:
|
||||
logger.warning("Dedup LLM response missing 'duplicates' key")
|
||||
return []
|
||||
|
||||
groups: list[DuplicateGroup] = []
|
||||
for raw in data["duplicates"]:
|
||||
try:
|
||||
groups.append(
|
||||
DuplicateGroup(
|
||||
canonical_index=int(raw["canonical_index"]),
|
||||
duplicate_indices=list(raw.get("duplicate_indices", [])),
|
||||
reason=raw.get("reason", ""),
|
||||
)
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("Skipping malformed duplicate entry: %s — %s", raw, exc)
|
||||
|
||||
return groups
|
||||
@@ -0,0 +1,641 @@
|
||||
"""Monitor orchestrator for the production error monitoring service.
|
||||
|
||||
This is the main entry point. Run as::
|
||||
|
||||
python -m backend.services.error_monitor.monitor
|
||||
|
||||
The ``ErrorMonitor.run()`` method executes the full pipeline:
|
||||
|
||||
* Monitor cycle (default) — query Cloud Logging, detect new patterns, send
|
||||
Discord alerts, check auto-resolution.
|
||||
* Daily digest mode (``DIGEST_MODE=true``) — bundle 24h stats into one message.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Any
|
||||
|
||||
from google.cloud import logging as cloud_logging
|
||||
from google.cloud import secretmanager
|
||||
|
||||
from backend.services.error_monitor.config import (
|
||||
GCP_PROJECT,
|
||||
LOOKBACK_MINUTES,
|
||||
MAX_LOG_ENTRIES,
|
||||
MONITORED_CLOUD_FUNCTIONS,
|
||||
MONITORED_CLOUD_RUN_JOBS,
|
||||
MONITORED_CLOUD_RUN_SERVICES,
|
||||
MONITORED_GCE_INSTANCES,
|
||||
SPIKE_MIN_COUNT,
|
||||
SPIKE_MULTIPLIER,
|
||||
get_digest_mode,
|
||||
get_discord_webhook_secret_name,
|
||||
get_llm_enabled,
|
||||
)
|
||||
from backend.services.error_monitor.discord import (
|
||||
ErrorMonitorDiscord,
|
||||
format_auto_resolved_alert,
|
||||
format_daily_digest,
|
||||
format_incident_alert,
|
||||
format_new_pattern_alert,
|
||||
format_spike_alert,
|
||||
)
|
||||
from backend.services.error_monitor.firestore_adapter import (
|
||||
ErrorPatternsAdapter,
|
||||
PatternData,
|
||||
UpsertResult,
|
||||
)
|
||||
from backend.services.error_monitor.known_issues import should_ignore
|
||||
from backend.services.error_monitor.llm_analysis import analyze_patterns, find_duplicate_patterns
|
||||
from backend.services.error_monitor.normalizer import compute_pattern_hash, normalize_message
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Module-level constants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: Maps log resource type → (label_key, service_name_list)
|
||||
RESOURCE_TYPE_FILTERS: dict[str, tuple[str, list[str]]] = {
|
||||
"cloud_run_revision": ("resource.labels.service_name", MONITORED_CLOUD_RUN_SERVICES),
|
||||
"cloud_run_job": ("resource.labels.job_name", MONITORED_CLOUD_RUN_JOBS),
|
||||
"cloud_function": ("resource.labels.function_name", MONITORED_CLOUD_FUNCTIONS),
|
||||
"gce_instance": ('labels."compute.googleapis.com/resource_name"', MONITORED_GCE_INSTANCES),
|
||||
}
|
||||
|
||||
#: Maps log resource type → canonical resource_type string stored in Firestore.
|
||||
RESOURCE_TYPE_MAP: dict[str, str] = {
|
||||
"cloud_run_revision": "cloud_run_service",
|
||||
"cloud_run_job": "cloud_run_job",
|
||||
"cloud_function": "cloud_function",
|
||||
"gce_instance": "gce_instance",
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Free functions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_log_filter(
|
||||
resource_type: str,
|
||||
resource_names: list[str],
|
||||
lookback_minutes: int,
|
||||
) -> str:
|
||||
"""Build a Cloud Logging filter string.
|
||||
|
||||
Args:
|
||||
resource_type: GCP resource type (e.g. ``"cloud_run_revision"``).
|
||||
resource_names: List of service/job/function/instance names to include.
|
||||
lookback_minutes: How many minutes back to query.
|
||||
|
||||
Returns:
|
||||
A filter string suitable for ``logging_client.list_entries(filter_=...)``.
|
||||
"""
|
||||
label_key = RESOURCE_TYPE_FILTERS[resource_type][0]
|
||||
cutoff = datetime.now(tz=timezone.utc) - timedelta(minutes=lookback_minutes)
|
||||
cutoff_iso = cutoff.strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
name_clause = " OR ".join(f'{label_key}="{name}"' for name in resource_names)
|
||||
if len(resource_names) > 1:
|
||||
name_clause = f"({name_clause})"
|
||||
|
||||
return (
|
||||
f'resource.type="{resource_type}" '
|
||||
f"severity>=ERROR "
|
||||
f'timestamp>="{cutoff_iso}" '
|
||||
f"{name_clause}"
|
||||
)
|
||||
|
||||
|
||||
def _extract_message(entry: Any) -> str | None:
|
||||
"""Extract the log message text from a Cloud Logging entry.
|
||||
|
||||
Handles several payload shapes:
|
||||
|
||||
* ``str`` payload — returned directly.
|
||||
* ``dict`` payload — tries ``"message"``, ``"textPayload"``, ``"error"``,
|
||||
``"msg"`` keys in that order.
|
||||
* ``None`` payload — returns ``None``.
|
||||
|
||||
Args:
|
||||
entry: A Cloud Logging ``LogEntry`` object.
|
||||
|
||||
Returns:
|
||||
The message string, or ``None`` if not extractable.
|
||||
"""
|
||||
payload = entry.payload
|
||||
if payload is None:
|
||||
return None
|
||||
if isinstance(payload, str):
|
||||
return payload if payload else None
|
||||
if isinstance(payload, dict):
|
||||
for key in ("message", "textPayload", "error", "msg"):
|
||||
value = payload.get(key)
|
||||
if value:
|
||||
return str(value)
|
||||
return None
|
||||
# Fallback: convert to string representation
|
||||
text = str(payload)
|
||||
return text if text else None
|
||||
|
||||
|
||||
def _extract_service_name(entry: Any, resource_type: str) -> str:
|
||||
"""Extract the service/job/function/instance name from a log entry.
|
||||
|
||||
Args:
|
||||
entry: A Cloud Logging ``LogEntry`` object.
|
||||
resource_type: GCP log resource type string.
|
||||
|
||||
Returns:
|
||||
The service name string (falls back to ``"unknown"`` if not found).
|
||||
"""
|
||||
labels = getattr(entry.resource, "labels", {}) or {}
|
||||
entry_labels = getattr(entry, "labels", {}) or {}
|
||||
|
||||
if resource_type == "cloud_run_revision":
|
||||
return labels.get("service_name") or "unknown"
|
||||
if resource_type == "cloud_run_job":
|
||||
return labels.get("job_name") or "unknown"
|
||||
if resource_type == "cloud_function":
|
||||
return labels.get("function_name") or "unknown"
|
||||
if resource_type == "gce_instance":
|
||||
# Prefer the human-readable resource name label
|
||||
name = entry_labels.get("compute.googleapis.com/resource_name")
|
||||
if name:
|
||||
return name
|
||||
return labels.get("instance_id") or "unknown"
|
||||
return "unknown"
|
||||
|
||||
|
||||
def _classify_resource_type(log_resource_type: str) -> str:
|
||||
"""Map a Cloud Logging resource type to the canonical type stored in Firestore.
|
||||
|
||||
Args:
|
||||
log_resource_type: Raw GCP resource type (e.g. ``"cloud_run_revision"``).
|
||||
|
||||
Returns:
|
||||
Canonical resource type string (falls back to the input if not in map).
|
||||
"""
|
||||
return RESOURCE_TYPE_MAP.get(log_resource_type, log_resource_type)
|
||||
|
||||
|
||||
def _group_by_pattern(entries: list[dict]) -> dict[str, dict]:
|
||||
"""Group normalised log entries by their pattern hash.
|
||||
|
||||
Args:
|
||||
entries: List of dicts with ``service``, ``normalized``, and ``message`` keys.
|
||||
|
||||
Returns:
|
||||
Dict mapping ``pattern_id`` → group dict containing ``service``,
|
||||
``resource_type``, ``normalized``, ``sample_message``, ``count``.
|
||||
"""
|
||||
groups: dict[str, dict] = {}
|
||||
for entry in entries:
|
||||
service = entry["service"]
|
||||
normalized = entry["normalized"]
|
||||
pattern_id = compute_pattern_hash(service, normalized)
|
||||
|
||||
if pattern_id not in groups:
|
||||
groups[pattern_id] = {
|
||||
"pattern_id": pattern_id,
|
||||
"service": service,
|
||||
"resource_type": entry.get("resource_type", "unknown"),
|
||||
"normalized": normalized,
|
||||
"sample_message": entry["message"],
|
||||
"count": 0,
|
||||
}
|
||||
groups[pattern_id]["count"] += 1
|
||||
return groups
|
||||
|
||||
|
||||
def _rolling_average(rolling_counts: list[dict]) -> float:
|
||||
"""Compute the mean count from a list of rolling_count dicts.
|
||||
|
||||
Args:
|
||||
rolling_counts: List of ``{"count": int, ...}`` dicts.
|
||||
|
||||
Returns:
|
||||
The arithmetic mean as a float, or ``0.0`` for an empty list.
|
||||
"""
|
||||
if not rolling_counts:
|
||||
return 0.0
|
||||
total = sum(entry.get("count", 0) for entry in rolling_counts)
|
||||
return total / len(rolling_counts)
|
||||
|
||||
|
||||
def _is_spike(current_count: int, rolling_counts: list[dict]) -> bool:
|
||||
"""Determine whether ``current_count`` represents an error rate spike.
|
||||
|
||||
A spike is detected when:
|
||||
- ``current_count`` >= ``SPIKE_MIN_COUNT`` (avoids noise from rare errors), AND
|
||||
- ``current_count`` > ``rolling_average * SPIKE_MULTIPLIER``
|
||||
|
||||
Args:
|
||||
current_count: Error count in the current monitoring window.
|
||||
rolling_counts: Historical rolling counts from Firestore.
|
||||
|
||||
Returns:
|
||||
``True`` if a spike is detected, ``False`` otherwise.
|
||||
"""
|
||||
if not rolling_counts:
|
||||
return False
|
||||
if current_count < SPIKE_MIN_COUNT:
|
||||
return False
|
||||
avg = _rolling_average(rolling_counts)
|
||||
if avg == 0.0:
|
||||
return False
|
||||
return current_count > avg * SPIKE_MULTIPLIER
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ErrorMonitor class
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class ErrorMonitor:
|
||||
"""Orchestrates the full production error monitoring pipeline.
|
||||
|
||||
On each run (typically every 15 minutes via Cloud Scheduler), it:
|
||||
|
||||
1. Queries Cloud Logging for recent errors across all monitored services.
|
||||
2. Groups errors into patterns using normalisation + hashing.
|
||||
3. Upserts patterns to Firestore, tracking new vs. recurrent.
|
||||
4. Optionally uses an LLM to deduplicate patterns and create incidents.
|
||||
5. Sends Discord alerts for new patterns, incidents, and spikes.
|
||||
6. Auto-resolves patterns that have been silent long enough.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.logging_client = cloud_logging.Client(project=GCP_PROJECT)
|
||||
self.firestore_adapter = ErrorPatternsAdapter()
|
||||
self.discord = ErrorMonitorDiscord(webhook_url=self._get_discord_webhook())
|
||||
|
||||
def _get_discord_webhook(self) -> str:
|
||||
"""Retrieve the Discord webhook URL.
|
||||
|
||||
Checks ``DISCORD_WEBHOOK_URL`` environment variable first; falls back
|
||||
to Secret Manager.
|
||||
|
||||
Returns:
|
||||
Discord webhook URL string.
|
||||
"""
|
||||
url = os.environ.get("DISCORD_WEBHOOK_URL")
|
||||
if url:
|
||||
return url
|
||||
|
||||
secret_name = get_discord_webhook_secret_name()
|
||||
client = secretmanager.SecretManagerServiceClient()
|
||||
secret_path = f"projects/{GCP_PROJECT}/secrets/{secret_name}/versions/latest"
|
||||
response = client.access_secret_version(name=secret_path)
|
||||
return response.payload.data.decode("utf-8").strip()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Main entry point
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def run(self) -> None:
|
||||
"""Execute the monitor pipeline (digest or cycle mode)."""
|
||||
if get_digest_mode():
|
||||
self._run_daily_digest()
|
||||
else:
|
||||
self._run_monitor_cycle()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Monitor cycle
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _run_monitor_cycle(self) -> None:
|
||||
"""Execute one monitor cycle: collect → group → upsert → alert → auto-resolve."""
|
||||
now = datetime.now(tz=timezone.utc)
|
||||
now_iso = now.isoformat()
|
||||
|
||||
# ── Step 1: Collect log entries ────────────────────────────────────
|
||||
raw_entries: list[dict] = []
|
||||
for resource_type, (_, resource_names) in RESOURCE_TYPE_FILTERS.items():
|
||||
log_filter = _build_log_filter(resource_type, resource_names, LOOKBACK_MINUTES)
|
||||
logger.info("Querying %s with filter: %s", resource_type, log_filter)
|
||||
try:
|
||||
entries = self.logging_client.list_entries(
|
||||
filter_=log_filter,
|
||||
max_results=MAX_LOG_ENTRIES,
|
||||
)
|
||||
for entry in entries:
|
||||
message = _extract_message(entry)
|
||||
if not message:
|
||||
continue
|
||||
service = _extract_service_name(entry, resource_type)
|
||||
ignore_reason = should_ignore(service, message)
|
||||
if ignore_reason:
|
||||
logger.debug(
|
||||
"Ignoring entry from %s (%s): %s",
|
||||
service,
|
||||
ignore_reason.pattern_name,
|
||||
ignore_reason.reason,
|
||||
)
|
||||
continue
|
||||
normalized = normalize_message(message)
|
||||
if not normalized:
|
||||
continue
|
||||
raw_entries.append(
|
||||
{
|
||||
"service": service,
|
||||
"resource_type": _classify_resource_type(resource_type),
|
||||
"normalized": normalized,
|
||||
"message": message,
|
||||
}
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.error("Failed to query %s logs: %s", resource_type, exc)
|
||||
|
||||
logger.info("Collected %d log entries after filtering", len(raw_entries))
|
||||
|
||||
# ── Step 2: Group by pattern ───────────────────────────────────────
|
||||
groups = _group_by_pattern(raw_entries)
|
||||
logger.info("Grouped into %d distinct patterns", len(groups))
|
||||
|
||||
if not groups:
|
||||
logger.info("No new error entries found this cycle; checking auto-resolve.")
|
||||
self._check_auto_resolve()
|
||||
return
|
||||
|
||||
# ── Step 3: Upsert to Firestore ────────────────────────────────────
|
||||
new_pattern_ids: list[str] = []
|
||||
spike_patterns: list[dict] = []
|
||||
|
||||
for pattern_id, group in groups.items():
|
||||
data = PatternData(
|
||||
pattern_id=pattern_id,
|
||||
service=group["service"],
|
||||
resource_type=group["resource_type"],
|
||||
normalized_message=group["normalized"],
|
||||
sample_message=group["sample_message"],
|
||||
count=group["count"],
|
||||
timestamp=now,
|
||||
)
|
||||
result: UpsertResult = self.firestore_adapter.upsert_pattern(data)
|
||||
if result.is_new:
|
||||
new_pattern_ids.append(pattern_id)
|
||||
logger.info("New pattern detected: %s (%s)", pattern_id, group["service"])
|
||||
|
||||
# Check for spike on existing patterns
|
||||
existing = self.firestore_adapter.get_pattern(pattern_id)
|
||||
if existing and not result.is_new:
|
||||
rolling = existing.get("rolling_counts") or []
|
||||
if _is_spike(group["count"], rolling):
|
||||
spike_patterns.append(
|
||||
{
|
||||
"service": group["service"],
|
||||
"normalized_message": group["normalized"],
|
||||
"current_count": group["count"],
|
||||
"rolling_average": _rolling_average(rolling),
|
||||
}
|
||||
)
|
||||
logger.info("Spike detected for pattern %s", pattern_id)
|
||||
|
||||
# ── Step 4: LLM dedup ─────────────────────────────────────────────
|
||||
if new_pattern_ids and get_llm_enabled():
|
||||
new_patterns_data = [
|
||||
{
|
||||
"service": groups[pid]["service"],
|
||||
"normalized_message": groups[pid]["normalized"],
|
||||
"count": groups[pid]["count"],
|
||||
}
|
||||
for pid in new_pattern_ids
|
||||
]
|
||||
existing_patterns = self.firestore_adapter.get_active_patterns()
|
||||
# Exclude newly added patterns from the canonical list
|
||||
canonical = [p for p in existing_patterns if p.get("pattern_id") not in new_pattern_ids]
|
||||
|
||||
if canonical:
|
||||
duplicate_groups = find_duplicate_patterns(new_patterns_data, canonical)
|
||||
for dup_group in duplicate_groups:
|
||||
# Merge each duplicate new pattern into the canonical
|
||||
canonical_pattern = canonical[dup_group.canonical_index]
|
||||
canonical_id = canonical_pattern.get("pattern_id", "")
|
||||
for dup_idx in dup_group.duplicate_indices:
|
||||
if dup_idx < len(new_pattern_ids):
|
||||
dup_id = new_pattern_ids[dup_idx]
|
||||
logger.info(
|
||||
"Merging duplicate pattern %s into %s: %s",
|
||||
dup_id,
|
||||
canonical_id,
|
||||
dup_group.reason,
|
||||
)
|
||||
self.firestore_adapter.merge_pattern(dup_id, canonical_id, dup_group.reason)
|
||||
# Remove from new_pattern_ids since it was merged
|
||||
new_pattern_ids = [pid for pid in new_pattern_ids if pid != dup_id]
|
||||
|
||||
# ── Step 5: LLM incident analysis ─────────────────────────────────
|
||||
incident_pattern_ids: set[str] = set()
|
||||
|
||||
if len(new_pattern_ids) >= 2 and get_llm_enabled():
|
||||
patterns_for_analysis = [
|
||||
{
|
||||
"service": groups[pid]["service"],
|
||||
"normalized_message": groups[pid]["normalized"],
|
||||
"count": groups[pid]["count"],
|
||||
}
|
||||
for pid in new_pattern_ids
|
||||
if pid in groups
|
||||
]
|
||||
analysis = analyze_patterns(patterns_for_analysis)
|
||||
|
||||
if analysis and analysis.incidents:
|
||||
for incident in analysis.incidents:
|
||||
incident_pids = [
|
||||
new_pattern_ids[idx]
|
||||
for idx in incident.pattern_indices
|
||||
if idx < len(new_pattern_ids)
|
||||
]
|
||||
if not incident_pids:
|
||||
continue
|
||||
|
||||
incident_id = self.firestore_adapter.create_incident(
|
||||
title=incident.title,
|
||||
root_cause=incident.root_cause or "",
|
||||
severity=incident.severity,
|
||||
suggested_fix=incident.suggested_fix or "",
|
||||
primary_service=incident.primary_service,
|
||||
pattern_ids=incident_pids,
|
||||
used_llm=analysis.used_llm,
|
||||
)
|
||||
logger.info(
|
||||
"Created incident %s (%s) with %d patterns",
|
||||
incident_id,
|
||||
incident.severity,
|
||||
len(incident_pids),
|
||||
)
|
||||
|
||||
# Build pattern list for the Discord alert
|
||||
pattern_dicts = [
|
||||
{
|
||||
"service": groups[pid]["service"],
|
||||
"normalized_message": groups[pid]["normalized"],
|
||||
"count": groups[pid]["count"],
|
||||
}
|
||||
for pid in incident_pids
|
||||
if pid in groups
|
||||
]
|
||||
msg = format_incident_alert(
|
||||
title=incident.title,
|
||||
severity=incident.severity,
|
||||
root_cause=incident.root_cause or "Unknown",
|
||||
suggested_fix=incident.suggested_fix or "Investigate logs",
|
||||
primary_service=incident.primary_service,
|
||||
patterns=pattern_dicts,
|
||||
)
|
||||
if self.discord.send_message(msg):
|
||||
for pid in incident_pids:
|
||||
try:
|
||||
self.firestore_adapter.update_pattern_alerted(pid)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("Could not update alerted_at for %s: %s", pid, exc)
|
||||
|
||||
incident_pattern_ids.update(incident_pids)
|
||||
|
||||
# ── Step 6: Individual alerts for new patterns not in incidents ────
|
||||
for pattern_id in new_pattern_ids:
|
||||
if pattern_id in incident_pattern_ids:
|
||||
continue
|
||||
if pattern_id not in groups:
|
||||
continue
|
||||
group = groups[pattern_id]
|
||||
|
||||
# Fetch fresh copy to get first_seen timestamp
|
||||
existing = self.firestore_adapter.get_pattern(pattern_id)
|
||||
first_seen = existing.get("first_seen", now_iso) if existing else now_iso
|
||||
|
||||
msg = format_new_pattern_alert(
|
||||
service=group["service"],
|
||||
resource_type=group["resource_type"],
|
||||
normalized_message=group["normalized"],
|
||||
sample_message=group["sample_message"],
|
||||
count=group["count"],
|
||||
first_seen=first_seen,
|
||||
)
|
||||
if self.discord.send_message(msg):
|
||||
try:
|
||||
self.firestore_adapter.update_pattern_alerted(pattern_id)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("Could not update alerted_at for %s: %s", pattern_id, exc)
|
||||
|
||||
# ── Step 7: Spike alerts ───────────────────────────────────────────
|
||||
for spike in spike_patterns:
|
||||
msg = format_spike_alert(
|
||||
service=spike["service"],
|
||||
normalized_message=spike["normalized_message"],
|
||||
current_count=spike["current_count"],
|
||||
rolling_average=spike["rolling_average"],
|
||||
)
|
||||
self.discord.send_message(msg)
|
||||
|
||||
# ── Step 8: Auto-resolve check ─────────────────────────────────────
|
||||
self._check_auto_resolve()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Auto-resolve helper
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _check_auto_resolve(self) -> None:
|
||||
"""Check all active patterns for auto-resolution and send a consolidated alert."""
|
||||
patterns = self.firestore_adapter.get_patterns_for_auto_resolve()
|
||||
resolved: list[dict] = []
|
||||
|
||||
for pattern in patterns:
|
||||
pattern_id = pattern.get("pattern_id", "")
|
||||
hours_silent = self.firestore_adapter.check_auto_resolve(pattern)
|
||||
if hours_silent is not None:
|
||||
logger.info(
|
||||
"Auto-resolving pattern %s (silent %.1f hours)", pattern_id, hours_silent
|
||||
)
|
||||
self.firestore_adapter.auto_resolve_pattern(pattern_id, round(hours_silent, 1))
|
||||
resolved.append(
|
||||
{
|
||||
"service": pattern.get("service", "unknown"),
|
||||
"message": pattern.get("normalized_message", ""),
|
||||
"hours_silent": round(hours_silent, 1),
|
||||
}
|
||||
)
|
||||
|
||||
if resolved:
|
||||
msg = format_auto_resolved_alert(resolved)
|
||||
self.discord.send_message(msg)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Daily digest
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _run_daily_digest(self) -> None:
|
||||
"""Gather 24h stats and send a daily digest to Discord."""
|
||||
# Optional dedup sweep (best-effort; LLM errors are non-fatal)
|
||||
if get_llm_enabled():
|
||||
try:
|
||||
active = self.firestore_adapter.get_active_patterns()
|
||||
if len(active) >= 2:
|
||||
mid = len(active) // 2
|
||||
duplicates = find_duplicate_patterns(active[:mid], active[mid:])
|
||||
for dup in duplicates:
|
||||
for idx in dup.duplicate_indices:
|
||||
target_half = active[mid:]
|
||||
source_half = active[:mid]
|
||||
if idx < len(target_half) and dup.canonical_index < len(source_half):
|
||||
self.firestore_adapter.merge_pattern(
|
||||
target_half[idx]["pattern_id"],
|
||||
source_half[dup.canonical_index]["pattern_id"],
|
||||
dup.reason,
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("Digest dedup sweep failed (non-fatal): %s", exc)
|
||||
|
||||
# Gather stats from active patterns
|
||||
active_patterns = self.firestore_adapter.get_active_patterns()
|
||||
per_service_counts: dict[str, int] = {}
|
||||
new_count = 0
|
||||
total_errors = 0
|
||||
|
||||
for pattern in active_patterns:
|
||||
service = pattern.get("service", "unknown")
|
||||
count = pattern.get("total_count", 0)
|
||||
per_service_counts[service] = per_service_counts.get(service, 0) + count
|
||||
total_errors += count
|
||||
if pattern.get("status") == "new":
|
||||
new_count += 1
|
||||
|
||||
# Count recently auto-resolved patterns
|
||||
resolved_patterns = self.firestore_adapter.get_patterns_for_auto_resolve()
|
||||
resolved_count = sum(
|
||||
1
|
||||
for p in resolved_patterns
|
||||
if self.firestore_adapter.check_auto_resolve(p) is not None
|
||||
)
|
||||
|
||||
msg = format_daily_digest(
|
||||
total_errors=total_errors,
|
||||
new_patterns=new_count,
|
||||
resolved_patterns=resolved_count,
|
||||
active_patterns=len(active_patterns),
|
||||
per_service_counts=per_service_counts,
|
||||
)
|
||||
self.discord.send_message(msg)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Entry point
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Configure logging and run the error monitor."""
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
|
||||
)
|
||||
monitor = ErrorMonitor()
|
||||
monitor.run()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,137 @@
|
||||
"""Message normalizer for the production error monitoring service.
|
||||
|
||||
Replaces variable parts of error messages (IDs, timestamps, paths, etc.)
|
||||
with stable placeholders so the same logical error always produces the same
|
||||
pattern hash, regardless of which specific resource or timestamp was involved.
|
||||
|
||||
Normalizer rules are applied in order — more specific patterns first.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import re
|
||||
|
||||
from backend.services.error_monitor.config import MAX_NORMALIZED_MESSAGE_LENGTH
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Compiled regex patterns (applied in order — more specific first)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_NORMALIZERS: list[tuple[re.Pattern, str]] = [
|
||||
# 1. GCS paths: gs://bucket/path/to/object → gs://<BUCKET>/<PATH>
|
||||
# Must come before Firestore and URL normalization.
|
||||
(
|
||||
re.compile(r"gs://([^/\s]+)/([^\s]+)"),
|
||||
r"gs://<BUCKET>/<PATH>",
|
||||
),
|
||||
# 1b. GCS bucket root (no object path): gs://bucket/ or gs://bucket
|
||||
# Uses a lookahead so the slash/end-of-string is not consumed, and uses
|
||||
# a character class that excludes '<' to avoid re-matching placeholders.
|
||||
(
|
||||
re.compile(r"gs://([^/<\s]+)(?=[/\s]|$)"),
|
||||
r"gs://<BUCKET>",
|
||||
),
|
||||
# 2. Firestore document paths (collection/doc/…/doc — at least 2 segments)
|
||||
# Negative lookbehind for '.' prevents matching inside domain names (URLs).
|
||||
# Negative lookbehind for '/' prevents matching inside already-replaced paths.
|
||||
(
|
||||
re.compile(
|
||||
r"(?<![./])\b([a-zA-Z_][a-zA-Z0-9_-]*/[a-zA-Z0-9_-]+(?:/[a-zA-Z0-9_-]+){1,})\b"
|
||||
),
|
||||
r"<DOC_PATH>",
|
||||
),
|
||||
# 3. UUIDs (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
|
||||
(
|
||||
re.compile(
|
||||
r"\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\b"
|
||||
),
|
||||
"<ID>",
|
||||
),
|
||||
# 4. ISO 8601 timestamps (2024-01-15T10:30:45Z / …+05:30 / ….123Z)
|
||||
(
|
||||
re.compile(
|
||||
r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:\d{2})"
|
||||
),
|
||||
"<TS>",
|
||||
),
|
||||
# 5. Epoch timestamps: 10+ digit number optionally followed by fractional seconds
|
||||
(
|
||||
re.compile(r"\b\d{10,}(?:\.\d+)?\b"),
|
||||
"<EPOCH>",
|
||||
),
|
||||
# 6. Email addresses
|
||||
(
|
||||
re.compile(r"\b[A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}\b"),
|
||||
"<EMAIL>",
|
||||
),
|
||||
# 7. IP addresses (IPv4) — must come before large-number replacement
|
||||
(
|
||||
re.compile(r"\b(?:\d{1,3}\.){3}\d{1,3}\b"),
|
||||
"<IP>",
|
||||
),
|
||||
# 8. URLs — strip to domain only (https://host/path → https://host)
|
||||
# Handles http and https; preserves the scheme+domain for context.
|
||||
(
|
||||
re.compile(r"https?://([A-Za-z0-9.\-]+(:\d+)?)/[^\s]*"),
|
||||
r"https://\1",
|
||||
),
|
||||
# 9. Job IDs in paths (/jobs/<alphanumeric>)
|
||||
(
|
||||
re.compile(r"/jobs/[A-Za-z0-9]+"),
|
||||
"/jobs/<ID>",
|
||||
),
|
||||
# 10. Firebase UIDs (20–28 mixed-case alphanumeric characters)
|
||||
# Firebase Auth UIDs are typically 28 chars; allow 20-28 for flexibility.
|
||||
# Must contain both letters and digits (to avoid replacing plain words).
|
||||
(
|
||||
re.compile(r"\b(?=[A-Za-z0-9]{20,28}\b)(?=[A-Za-z0-9]*[A-Z])(?=[A-Za-z0-9]*[a-z])(?=[A-Za-z0-9]*\d)[A-Za-z0-9]{20,28}\b"),
|
||||
"<UID>",
|
||||
),
|
||||
# 11. Hex/alphanumeric IDs: 8+ chars containing both letters and digits
|
||||
# (e.g. git SHAs, Cloud Run revision IDs, job IDs)
|
||||
# Uses a lookahead to require at least one digit AND one letter.
|
||||
(
|
||||
re.compile(r"\b(?=[a-fA-F0-9]*[a-fA-F][a-fA-F0-9]*\d|[a-fA-F0-9]*\d[a-fA-F0-9]*[a-fA-F])[a-fA-F0-9]{8,}\b"),
|
||||
"<ID>",
|
||||
),
|
||||
# 12. Large numbers (4+ digits) — must come after epoch/IP/UUID replacements
|
||||
(
|
||||
re.compile(r"\b\d{4,}\b"),
|
||||
"<NUM>",
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def normalize_message(message: str) -> str:
|
||||
"""Normalize a raw log message by replacing variable parts with placeholders.
|
||||
|
||||
Applies each normalizer rule in order (more specific patterns first) and
|
||||
truncates the result to MAX_NORMALIZED_MESSAGE_LENGTH characters.
|
||||
|
||||
Args:
|
||||
message: Raw log message string.
|
||||
|
||||
Returns:
|
||||
Normalized message string, at most MAX_NORMALIZED_MESSAGE_LENGTH chars.
|
||||
"""
|
||||
if not message:
|
||||
return ""
|
||||
|
||||
result = message
|
||||
for pattern, replacement in _NORMALIZERS:
|
||||
result = pattern.sub(replacement, result)
|
||||
|
||||
return result[:MAX_NORMALIZED_MESSAGE_LENGTH]
|
||||
|
||||
|
||||
def compute_pattern_hash(service: str, normalized_message: str) -> str:
|
||||
"""Compute a stable SHA-256 fingerprint for an error pattern.
|
||||
|
||||
Args:
|
||||
service: The originating service name (e.g. "karaoke-backend").
|
||||
normalized_message: The output of normalize_message().
|
||||
|
||||
Returns:
|
||||
64-character lowercase hex SHA-256 digest.
|
||||
"""
|
||||
payload = f"{service}:{normalized_message}"
|
||||
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
|
||||
@@ -10,6 +10,8 @@ from google.cloud.firestore_v1 import FieldFilter
|
||||
from backend.config import settings
|
||||
from backend.models.job import Job, JobStatus, TimelineEvent
|
||||
from backend.models.worker_log import WorkerLogEntry
|
||||
from backend.models.review_session import ReviewSession
|
||||
from backend.models.audio_edit_session import AudioEditSession
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -75,18 +77,20 @@ class FirestoreService:
|
||||
status: JobStatus,
|
||||
progress: Optional[int] = None,
|
||||
message: Optional[str] = None,
|
||||
timeline_metadata: Optional[Dict[str, Any]] = None,
|
||||
**additional_fields
|
||||
) -> None:
|
||||
"""Update job status and add timeline event."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
|
||||
|
||||
# Create timeline event
|
||||
timeline_event = TimelineEvent(
|
||||
status=status.value,
|
||||
timestamp=datetime.utcnow().isoformat(),
|
||||
progress=progress,
|
||||
message=message
|
||||
message=message,
|
||||
metadata=timeline_metadata,
|
||||
)
|
||||
|
||||
# Prepare updates
|
||||
@@ -153,7 +157,8 @@ class FirestoreService:
|
||||
query = query.where(filter=FieldFilter('user_email', '==', user_email.lower()))
|
||||
|
||||
# Filter by tenant_id (white-label portal scoping)
|
||||
if tenant_id:
|
||||
# Always filter: "" for consumer jobs, "vocalstar"/etc for tenant jobs
|
||||
if tenant_id is not None:
|
||||
query = query.where(filter=FieldFilter('tenant_id', '==', tenant_id))
|
||||
|
||||
# Date range filters
|
||||
@@ -173,6 +178,86 @@ class FirestoreService:
|
||||
logger.error(f"Error listing jobs: {e}")
|
||||
raise
|
||||
|
||||
# Fields needed by the dashboard summary view
|
||||
SUMMARY_FIELD_PATHS = [
|
||||
'job_id', 'status', 'progress', 'created_at', 'artist', 'title',
|
||||
'error_message', 'non_interactive', 'outputs_deleted_at', 'user_email', 'is_private',
|
||||
'url', 'filename', 'audio_search_artist', 'audio_search_title',
|
||||
'audio_source_type', 'source_name', 'source_id', 'target_file', 'download_url',
|
||||
'state_data.brand_code', 'state_data.youtube_url', 'state_data.dropbox_link',
|
||||
'state_data.audio_progress', 'state_data.lyrics_progress',
|
||||
'state_data.audio_complete', 'state_data.lyrics_complete',
|
||||
'state_data.backing_vocals_analysis', 'state_data.visibility_change_in_progress',
|
||||
'file_urls.finals', 'file_urls.videos', 'file_urls.packages',
|
||||
'processing_metadata',
|
||||
]
|
||||
|
||||
def list_jobs_summary(
|
||||
self,
|
||||
status: Optional[JobStatus] = None,
|
||||
exclude_statuses: Optional[List[str]] = None,
|
||||
environment: Optional[str] = None,
|
||||
client_id: Optional[str] = None,
|
||||
created_after: Optional[datetime] = None,
|
||||
created_before: Optional[datetime] = None,
|
||||
user_email: Optional[str] = None,
|
||||
tenant_id: Optional[str] = None,
|
||||
limit: int = 100,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
List jobs with Firestore field projection for reduced payload.
|
||||
|
||||
Uses ``query.select()`` so Firestore only reads the requested fields,
|
||||
drastically cutting bandwidth and deserialization cost.
|
||||
|
||||
Returns raw dicts (not Job models) because projected documents would
|
||||
fail Pydantic validation for missing required fields.
|
||||
"""
|
||||
try:
|
||||
query = self.db.collection(self.collection)
|
||||
|
||||
if status:
|
||||
query = query.where(filter=FieldFilter('status', '==', status.value))
|
||||
|
||||
# NOTE: exclude_statuses is filtered in Python after the query
|
||||
# because Firestore's not-in filter combined with order_by(created_at)
|
||||
# requires a composite index. Since documents are tiny after select(),
|
||||
# Python filtering is fast and avoids the index requirement.
|
||||
|
||||
if environment:
|
||||
query = query.where(filter=FieldFilter('request_metadata.environment', '==', environment))
|
||||
|
||||
if client_id:
|
||||
query = query.where(filter=FieldFilter('request_metadata.client_id', '==', client_id))
|
||||
|
||||
if user_email:
|
||||
query = query.where(filter=FieldFilter('user_email', '==', user_email.lower()))
|
||||
|
||||
# Always filter: "" for consumer jobs, "vocalstar"/etc for tenant jobs
|
||||
if tenant_id is not None:
|
||||
query = query.where(filter=FieldFilter('tenant_id', '==', tenant_id))
|
||||
|
||||
if created_after:
|
||||
query = query.where(filter=FieldFilter('created_at', '>=', created_after))
|
||||
|
||||
if created_before:
|
||||
query = query.where(filter=FieldFilter('created_at', '<=', created_before))
|
||||
|
||||
query = query.order_by('created_at', direction=firestore.Query.DESCENDING).limit(limit)
|
||||
query = query.select(self.SUMMARY_FIELD_PATHS)
|
||||
|
||||
docs = query.stream()
|
||||
results = [doc.to_dict() for doc in docs]
|
||||
|
||||
# Apply exclude_statuses filter in Python (see note above)
|
||||
if exclude_statuses:
|
||||
results = [r for r in results if r.get('status') not in exclude_statuses]
|
||||
|
||||
return results
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing jobs summary: {e}")
|
||||
raise
|
||||
|
||||
def delete_jobs_by_filter(
|
||||
self,
|
||||
environment: Optional[str] = None,
|
||||
@@ -516,6 +601,426 @@ class FirestoreService:
|
||||
logger.error(f"Error listing tokens: {e}")
|
||||
return []
|
||||
|
||||
# ============================================
|
||||
# Search Session Methods
|
||||
# ============================================
|
||||
|
||||
def create_search_session(self, session_data: dict) -> str:
|
||||
"""Store a search session for the guided job creation flow.
|
||||
|
||||
Sessions are short-lived (TTL enforced via Firestore TTL policy) and
|
||||
represent a completed audio search that hasn't yet been converted into
|
||||
a job. The document is consumed (deleted) when create-from-search runs.
|
||||
|
||||
Args:
|
||||
session_data: Dict with session_id, user_email, tenant_id, artist,
|
||||
title, results, remote_search_id, ttl_expiry.
|
||||
|
||||
Returns:
|
||||
session_id from session_data.
|
||||
"""
|
||||
try:
|
||||
session_id = session_data['session_id']
|
||||
doc_ref = self.db.collection('search_sessions').document(session_id)
|
||||
doc_ref.set(session_data)
|
||||
logger.info(f"Created search session {session_id}")
|
||||
return session_id
|
||||
except Exception as e:
|
||||
logger.error(f"Error creating search session: {e}")
|
||||
raise
|
||||
|
||||
def get_search_session(self, session_id: str) -> Optional[dict]:
|
||||
"""Retrieve a search session by ID.
|
||||
|
||||
Returns None only if the document does not exist.
|
||||
Raises on Firestore errors so callers can distinguish a missing session
|
||||
from a transient datastore failure.
|
||||
|
||||
TTL-expired documents may still appear briefly; callers should check
|
||||
ttl_expiry and treat expired sessions as missing.
|
||||
|
||||
Args:
|
||||
session_id: UUID string identifying the session.
|
||||
|
||||
Returns:
|
||||
Session dict or None.
|
||||
"""
|
||||
doc_ref = self.db.collection('search_sessions').document(session_id)
|
||||
doc = doc_ref.get()
|
||||
if not doc.exists:
|
||||
return None
|
||||
return doc.to_dict()
|
||||
|
||||
def consume_search_session(self, session_id: str) -> Optional[dict]:
|
||||
"""Atomically read and delete a search session.
|
||||
|
||||
Uses a Firestore transaction to prevent two concurrent job-creation
|
||||
requests from both reading the same session and creating duplicate jobs.
|
||||
|
||||
Returns the session dict if it existed, None if it did not exist.
|
||||
Raises on Firestore errors.
|
||||
|
||||
Args:
|
||||
session_id: UUID string identifying the session.
|
||||
|
||||
Returns:
|
||||
Session dict or None.
|
||||
"""
|
||||
doc_ref = self.db.collection('search_sessions').document(session_id)
|
||||
|
||||
@firestore.transactional
|
||||
def _read_and_delete(transaction, ref):
|
||||
doc = ref.get(transaction=transaction)
|
||||
if not doc.exists:
|
||||
return None
|
||||
session_data = doc.to_dict()
|
||||
transaction.delete(ref)
|
||||
return session_data
|
||||
|
||||
transaction = self.db.transaction()
|
||||
return _read_and_delete(transaction, doc_ref)
|
||||
|
||||
# ============================================
|
||||
# Review Session Subcollection Methods
|
||||
# ============================================
|
||||
# These methods store review session snapshots in a subcollection
|
||||
# (jobs/{job_id}/review_sessions) for lyrics review backup/restore.
|
||||
|
||||
def save_review_session(self, job_id: str, session: ReviewSession) -> str:
|
||||
"""
|
||||
Save a review session to the subcollection.
|
||||
|
||||
Stores at: jobs/{job_id}/review_sessions/{session_id}
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
session: ReviewSession instance
|
||||
|
||||
Returns:
|
||||
session_id
|
||||
"""
|
||||
try:
|
||||
session.job_id = job_id
|
||||
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
|
||||
doc_ref = sessions_ref.document(session.session_id)
|
||||
doc_ref.set(session.to_dict())
|
||||
logger.info(f"Saved review session {session.session_id} for job {job_id}")
|
||||
return session.session_id
|
||||
except Exception as e:
|
||||
logger.error(f"Error saving review session for job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def list_review_sessions(self, job_id: str) -> List[ReviewSession]:
|
||||
"""
|
||||
List all review sessions for a job, ordered by updated_at descending.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
|
||||
Returns:
|
||||
List of ReviewSession instances (metadata only, no correction_data)
|
||||
"""
|
||||
try:
|
||||
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
|
||||
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING)
|
||||
docs = query.stream()
|
||||
return [ReviewSession.from_dict(doc.to_dict()) for doc in docs]
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing review sessions for job {job_id}: {e}")
|
||||
return []
|
||||
|
||||
def get_review_session(self, job_id: str, session_id: str) -> Optional[ReviewSession]:
|
||||
"""
|
||||
Get a single review session by ID.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
session_id: Session ID
|
||||
|
||||
Returns:
|
||||
ReviewSession or None
|
||||
"""
|
||||
try:
|
||||
doc_ref = (
|
||||
self.db.collection(self.collection)
|
||||
.document(job_id)
|
||||
.collection("review_sessions")
|
||||
.document(session_id)
|
||||
)
|
||||
doc = doc_ref.get()
|
||||
if not doc.exists:
|
||||
return None
|
||||
return ReviewSession.from_dict(doc.to_dict())
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting review session {session_id} for job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def delete_review_session(self, job_id: str, session_id: str) -> None:
|
||||
"""
|
||||
Delete a review session.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
session_id: Session ID
|
||||
"""
|
||||
try:
|
||||
doc_ref = (
|
||||
self.db.collection(self.collection)
|
||||
.document(job_id)
|
||||
.collection("review_sessions")
|
||||
.document(session_id)
|
||||
)
|
||||
doc_ref.delete()
|
||||
logger.info(f"Deleted review session {session_id} for job {job_id}")
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting review session {session_id} for job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def delete_review_sessions_subcollection(self, job_id: str, batch_size: int = 500) -> int:
|
||||
"""
|
||||
Delete all review sessions for a job (used when deleting a job).
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
batch_size: Number of documents to delete per batch
|
||||
|
||||
Returns:
|
||||
Number of sessions deleted
|
||||
"""
|
||||
try:
|
||||
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
|
||||
deleted_count = 0
|
||||
|
||||
while True:
|
||||
docs = sessions_ref.limit(batch_size).stream()
|
||||
deleted_in_batch = 0
|
||||
|
||||
batch = self.db.batch()
|
||||
for doc in docs:
|
||||
batch.delete(doc.reference)
|
||||
deleted_in_batch += 1
|
||||
|
||||
if deleted_in_batch == 0:
|
||||
break
|
||||
|
||||
batch.commit()
|
||||
deleted_count += deleted_in_batch
|
||||
|
||||
if deleted_in_batch < batch_size:
|
||||
break
|
||||
|
||||
if deleted_count > 0:
|
||||
logger.info(f"Deleted {deleted_count} review sessions for job {job_id}")
|
||||
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting review sessions subcollection for job {job_id}: {e}")
|
||||
return 0
|
||||
|
||||
def get_latest_review_session_hash(self, job_id: str) -> Optional[str]:
|
||||
"""
|
||||
Get the data_hash of the most recent review session for deduplication.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
|
||||
Returns:
|
||||
data_hash string or None if no sessions exist
|
||||
"""
|
||||
try:
|
||||
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
|
||||
query = (
|
||||
sessions_ref
|
||||
.order_by("updated_at", direction=firestore.Query.DESCENDING)
|
||||
.limit(1)
|
||||
.select(["data_hash"])
|
||||
)
|
||||
docs = list(query.stream())
|
||||
if docs:
|
||||
return docs[0].to_dict().get("data_hash", "")
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting latest review session hash for job {job_id}: {e}")
|
||||
return None
|
||||
|
||||
def search_review_sessions(
|
||||
self,
|
||||
query_text: str = "",
|
||||
job_id: Optional[str] = None,
|
||||
limit: int = 20,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Search review sessions across all jobs using collection group query.
|
||||
|
||||
Args:
|
||||
query_text: Search text (prefix-matched against artist_lower or title_lower)
|
||||
job_id: If provided, search only within this job
|
||||
limit: Maximum results
|
||||
|
||||
Returns:
|
||||
List of session dicts with job context
|
||||
"""
|
||||
try:
|
||||
if job_id:
|
||||
# Search within a specific job
|
||||
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
|
||||
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING).limit(limit)
|
||||
docs = query.stream()
|
||||
return [doc.to_dict() for doc in docs]
|
||||
|
||||
# Collection group query across all jobs
|
||||
sessions_ref = self.db.collection_group("review_sessions")
|
||||
|
||||
if query_text:
|
||||
# Over-fetch since we filter in Python (Firestore doesn't support LIKE/contains)
|
||||
fetch_limit = max(limit * 25, 500)
|
||||
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING).limit(fetch_limit)
|
||||
docs = query.stream()
|
||||
results = [doc.to_dict() for doc in docs]
|
||||
|
||||
q_lower = query_text.lower()
|
||||
results = [
|
||||
r for r in results
|
||||
if q_lower in (r.get("artist") or "").lower()
|
||||
or q_lower in (r.get("title") or "").lower()
|
||||
or q_lower in (r.get("job_id") or "").lower()
|
||||
]
|
||||
return results[:limit]
|
||||
else:
|
||||
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING).limit(limit)
|
||||
docs = query.stream()
|
||||
return [doc.to_dict() for doc in docs]
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error searching review sessions: {e}")
|
||||
return []
|
||||
|
||||
# ---- Audio Edit Sessions ----
|
||||
|
||||
def save_audio_edit_session(self, job_id: str, session: AudioEditSession) -> str:
|
||||
"""Save an audio edit session to the subcollection."""
|
||||
try:
|
||||
session.job_id = job_id
|
||||
sessions_ref = self.db.collection(self.collection).document(job_id).collection("audio_edit_sessions")
|
||||
doc_ref = sessions_ref.document(session.session_id)
|
||||
doc_ref.set(session.to_dict())
|
||||
logger.info(f"Saved audio edit session {session.session_id} for job {job_id}")
|
||||
return session.session_id
|
||||
except Exception as e:
|
||||
logger.error(f"Error saving audio edit session for job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def list_audio_edit_sessions(self, job_id: str) -> List[AudioEditSession]:
|
||||
"""List all audio edit sessions for a job, ordered by updated_at descending."""
|
||||
try:
|
||||
sessions_ref = self.db.collection(self.collection).document(job_id).collection("audio_edit_sessions")
|
||||
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING)
|
||||
docs = query.stream()
|
||||
return [AudioEditSession.from_dict(doc.to_dict()) for doc in docs]
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing audio edit sessions for job {job_id}: {e}")
|
||||
return []
|
||||
|
||||
def get_audio_edit_session(self, job_id: str, session_id: str) -> Optional[AudioEditSession]:
|
||||
"""Get a single audio edit session by ID."""
|
||||
try:
|
||||
doc_ref = (
|
||||
self.db.collection(self.collection)
|
||||
.document(job_id)
|
||||
.collection("audio_edit_sessions")
|
||||
.document(session_id)
|
||||
)
|
||||
doc = doc_ref.get()
|
||||
if not doc.exists:
|
||||
return None
|
||||
return AudioEditSession.from_dict(doc.to_dict())
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting audio edit session {session_id} for job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def delete_audio_edit_session(self, job_id: str, session_id: str) -> None:
|
||||
"""Delete a single audio edit session."""
|
||||
try:
|
||||
doc_ref = (
|
||||
self.db.collection(self.collection)
|
||||
.document(job_id)
|
||||
.collection("audio_edit_sessions")
|
||||
.document(session_id)
|
||||
)
|
||||
doc_ref.delete()
|
||||
logger.info(f"Deleted audio edit session {session_id} for job {job_id}")
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting audio edit session {session_id} for job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def delete_audio_edit_sessions_subcollection(self, job_id: str, batch_size: int = 500) -> int:
|
||||
"""Delete all audio edit sessions for a job (used when deleting a job)."""
|
||||
try:
|
||||
sessions_ref = self.db.collection(self.collection).document(job_id).collection("audio_edit_sessions")
|
||||
deleted_count = 0
|
||||
|
||||
while True:
|
||||
docs = sessions_ref.limit(batch_size).stream()
|
||||
deleted_in_batch = 0
|
||||
|
||||
batch = self.db.batch()
|
||||
for doc in docs:
|
||||
batch.delete(doc.reference)
|
||||
deleted_in_batch += 1
|
||||
|
||||
if deleted_in_batch == 0:
|
||||
break
|
||||
|
||||
batch.commit()
|
||||
deleted_count += deleted_in_batch
|
||||
|
||||
if deleted_in_batch < batch_size:
|
||||
break
|
||||
|
||||
if deleted_count > 0:
|
||||
logger.info(f"Deleted {deleted_count} audio edit sessions for job {job_id}")
|
||||
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting audio edit sessions subcollection for job {job_id}: {e}")
|
||||
return 0
|
||||
|
||||
def get_latest_audio_edit_session_hash(self, job_id: str) -> Optional[str]:
|
||||
"""Get the data_hash of the most recent audio edit session for deduplication."""
|
||||
try:
|
||||
sessions_ref = self.db.collection(self.collection).document(job_id).collection("audio_edit_sessions")
|
||||
query = (
|
||||
sessions_ref
|
||||
.order_by("updated_at", direction=firestore.Query.DESCENDING)
|
||||
.limit(1)
|
||||
.select(["data_hash"])
|
||||
)
|
||||
docs = list(query.stream())
|
||||
if docs:
|
||||
return docs[0].to_dict().get("data_hash", "")
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting latest audio edit session hash for job {job_id}: {e}")
|
||||
return None
|
||||
|
||||
def delete_search_session(self, session_id: str) -> None:
|
||||
"""Delete a search session after it has been consumed by job creation.
|
||||
|
||||
Errors are logged but not re-raised — a missing or already-deleted
|
||||
session is a no-op.
|
||||
|
||||
Args:
|
||||
session_id: UUID string identifying the session.
|
||||
"""
|
||||
try:
|
||||
doc_ref = self.db.collection('search_sessions').document(session_id)
|
||||
doc_ref.delete()
|
||||
logger.debug(f"Deleted search session {session_id}")
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting search session {session_id}: {e}")
|
||||
|
||||
|
||||
# Singleton client instance
|
||||
_firestore_client: Optional[firestore.Client] = None
|
||||
@@ -537,3 +1042,35 @@ def get_firestore_client() -> firestore.Client:
|
||||
_firestore_client = firestore.Client(project=settings.google_cloud_project)
|
||||
return _firestore_client
|
||||
|
||||
|
||||
_log_service: Optional["FirestoreService"] = None
|
||||
|
||||
|
||||
def _get_log_service() -> "FirestoreService":
|
||||
"""Return a cached FirestoreService instance for log_to_job."""
|
||||
global _log_service
|
||||
if _log_service is None:
|
||||
_log_service = FirestoreService()
|
||||
return _log_service
|
||||
|
||||
|
||||
def log_to_job(job_id: str, worker: str, level: str, message: str, metadata: Optional[Dict[str, Any]] = None) -> None:
|
||||
"""
|
||||
Write a log entry to a job's Firestore log subcollection.
|
||||
|
||||
This is a convenience function for API endpoints that need to log
|
||||
to a job's log history (workers already have JobLogAdapter for this).
|
||||
|
||||
Args:
|
||||
job_id: Job ID to log to
|
||||
worker: Worker/source name (e.g., "api", "edit", "admin")
|
||||
level: Log level (INFO, WARNING, ERROR)
|
||||
message: Log message
|
||||
metadata: Optional structured metadata dict
|
||||
"""
|
||||
try:
|
||||
entry = WorkerLogEntry.create(job_id, worker, level, message, metadata)
|
||||
_get_log_service().append_log_to_subcollection(job_id, entry)
|
||||
except Exception as e:
|
||||
logger.debug(f"Failed to write job log for {job_id}: {e}")
|
||||
|
||||
@@ -34,6 +34,8 @@ from typing import Any, Dict, List, Optional
|
||||
|
||||
import httpx
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@@ -42,6 +44,104 @@ class FlacfetchServiceError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def _format_httpx_error(e: Exception) -> str:
|
||||
"""
|
||||
Format an httpx exception into a descriptive error string.
|
||||
|
||||
httpx.RequestError subclasses (ConnectError, ReadError, etc.) often have
|
||||
empty str() representations, which produces useless messages like
|
||||
"Status request failed: ". This helper always returns something meaningful.
|
||||
"""
|
||||
error_str = str(e)
|
||||
error_type = type(e).__name__
|
||||
|
||||
if error_str:
|
||||
return f"{error_type}: {error_str}"
|
||||
|
||||
# str(e) is empty - try to include request URL for context
|
||||
if isinstance(e, httpx.RequestError):
|
||||
try:
|
||||
request = e.request
|
||||
if request:
|
||||
return f"{error_type} for {request.method} {request.url}"
|
||||
except RuntimeError:
|
||||
pass # .request not set
|
||||
return error_type
|
||||
|
||||
|
||||
def should_retry_flacfetch_error(exception: Exception) -> bool:
|
||||
"""
|
||||
Determine if a flacfetch exception should be retried.
|
||||
|
||||
Retry on:
|
||||
- httpx.RequestError (network failures: timeout, connection refused, etc.)
|
||||
- httpx.HTTPStatusError with 5xx status (server errors)
|
||||
|
||||
Don't retry on:
|
||||
- httpx.HTTPStatusError with 4xx status (client errors: bad request, auth, not found)
|
||||
"""
|
||||
if isinstance(exception, httpx.RequestError):
|
||||
# Network-level failures - always retry
|
||||
return True
|
||||
|
||||
if isinstance(exception, httpx.HTTPStatusError):
|
||||
# HTTP errors - only retry on 5xx (server errors)
|
||||
return exception.response.status_code >= 500
|
||||
|
||||
# Don't retry on other exceptions (e.g., FlacfetchServiceError after wrapping)
|
||||
return False
|
||||
|
||||
|
||||
async def with_retry(func, *args, **kwargs):
|
||||
"""
|
||||
Execute an async function with retry logic using exponential backoff.
|
||||
|
||||
This implementation uses manual retry logic instead of tenacity decorators
|
||||
for better testability and runtime configuration.
|
||||
"""
|
||||
settings = get_settings()
|
||||
max_attempts = settings.flacfetch_retry_max_attempts
|
||||
min_wait = settings.flacfetch_retry_min_wait
|
||||
max_wait = settings.flacfetch_retry_max_wait
|
||||
|
||||
last_exception = None
|
||||
|
||||
for attempt in range(1, max_attempts + 1):
|
||||
try:
|
||||
return await func(*args, **kwargs)
|
||||
except Exception as e:
|
||||
last_exception = e
|
||||
|
||||
# Check if we should retry this exception
|
||||
if not should_retry_flacfetch_error(e):
|
||||
# Don't retry - raise immediately
|
||||
raise
|
||||
|
||||
# Check if we have more attempts left
|
||||
if attempt >= max_attempts:
|
||||
# No more attempts - raise the last exception
|
||||
logger.warning(
|
||||
f"Flacfetch request failed after {max_attempts} attempts: "
|
||||
f"{_format_httpx_error(e) if isinstance(e, httpx.RequestError) else e}"
|
||||
)
|
||||
raise
|
||||
|
||||
# Calculate wait time with exponential backoff
|
||||
wait_time = min(min_wait * (2 ** (attempt - 1)), max_wait)
|
||||
|
||||
logger.warning(
|
||||
f"Flacfetch request failed (attempt {attempt}/{max_attempts}): "
|
||||
f"{_format_httpx_error(e) if isinstance(e, httpx.RequestError) else e}. "
|
||||
f"Retrying in {wait_time:.1f}s..."
|
||||
)
|
||||
|
||||
# Wait before next attempt
|
||||
await asyncio.sleep(wait_time)
|
||||
|
||||
# Should never reach here, but just in case
|
||||
raise last_exception
|
||||
|
||||
|
||||
class FlacfetchClient:
|
||||
"""
|
||||
Client for remote flacfetch HTTP API.
|
||||
@@ -100,7 +200,7 @@ class FlacfetchClient:
|
||||
|
||||
return data
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Cannot reach flacfetch service: {e}")
|
||||
raise FlacfetchServiceError(f"Cannot reach flacfetch service: {_format_httpx_error(e)}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Health check failed: {e.response.status_code}")
|
||||
|
||||
@@ -145,10 +245,10 @@ class FlacfetchClient:
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Search request failed: {e}")
|
||||
raise FlacfetchServiceError(f"Search request failed: {_format_httpx_error(e)}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Search failed: {e.response.status_code} - {e.response.text}")
|
||||
|
||||
|
||||
async def download(
|
||||
self,
|
||||
search_id: str,
|
||||
@@ -158,47 +258,59 @@ class FlacfetchClient:
|
||||
) -> str:
|
||||
"""
|
||||
Start downloading an audio file.
|
||||
|
||||
|
||||
Retries automatically on network errors and server errors (5xx) using
|
||||
exponential backoff. Configured via FLACFETCH_RETRY_* environment variables.
|
||||
|
||||
Args:
|
||||
search_id: Search ID from previous search
|
||||
result_index: Index of result to download
|
||||
output_filename: Optional custom filename (without extension)
|
||||
gcs_path: GCS path for upload (e.g., "uploads/job123/audio/")
|
||||
|
||||
|
||||
Returns:
|
||||
Download ID for tracking progress
|
||||
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On download start failure
|
||||
FlacfetchServiceError: On download start failure (after all retries exhausted)
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
payload = {
|
||||
"search_id": search_id,
|
||||
"result_index": result_index,
|
||||
}
|
||||
if output_filename:
|
||||
payload["output_filename"] = output_filename
|
||||
if gcs_path:
|
||||
payload["upload_to_gcs"] = True
|
||||
payload["gcs_path"] = gcs_path
|
||||
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/download",
|
||||
headers=self._headers(),
|
||||
json=payload,
|
||||
timeout=self.timeout,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
data = resp.json()
|
||||
return data["download_id"]
|
||||
|
||||
return await with_retry(self._download_impl, search_id, result_index, output_filename, gcs_path)
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Download request failed: {e}")
|
||||
raise FlacfetchServiceError(f"Download request failed: {_format_httpx_error(e)}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Download start failed: {e.response.status_code} - {e.response.text}")
|
||||
|
||||
|
||||
async def _download_impl(
|
||||
self,
|
||||
search_id: str,
|
||||
result_index: int,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Implementation of download without error wrapping - lets httpx exceptions bubble up."""
|
||||
async with httpx.AsyncClient() as client:
|
||||
payload = {
|
||||
"search_id": search_id,
|
||||
"result_index": result_index,
|
||||
}
|
||||
if output_filename:
|
||||
payload["output_filename"] = output_filename
|
||||
if gcs_path:
|
||||
payload["upload_to_gcs"] = True
|
||||
payload["gcs_path"] = gcs_path
|
||||
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/download",
|
||||
headers=self._headers(),
|
||||
json=payload,
|
||||
timeout=self.timeout,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
data = resp.json()
|
||||
return data["download_id"]
|
||||
|
||||
async def download_by_id(
|
||||
self,
|
||||
source_name: str,
|
||||
@@ -214,6 +326,9 @@ class FlacfetchClient:
|
||||
This is useful when you have stored the source_id from a previous search
|
||||
and want to download later without re-searching.
|
||||
|
||||
Retries automatically on network errors and server errors (5xx) using
|
||||
exponential backoff. Configured via FLACFETCH_RETRY_* environment variables.
|
||||
|
||||
Args:
|
||||
source_name: Provider name (RED, OPS, YouTube, Spotify)
|
||||
source_id: Source-specific ID (torrent ID, video ID, track ID)
|
||||
@@ -226,52 +341,68 @@ class FlacfetchClient:
|
||||
Download ID for tracking progress
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On download start failure
|
||||
FlacfetchServiceError: On download start failure (after all retries exhausted)
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
payload = {
|
||||
"source_name": source_name,
|
||||
"source_id": source_id,
|
||||
}
|
||||
if output_filename:
|
||||
payload["output_filename"] = output_filename
|
||||
if target_file:
|
||||
payload["target_file"] = target_file
|
||||
if download_url:
|
||||
payload["download_url"] = download_url
|
||||
if gcs_path:
|
||||
payload["upload_to_gcs"] = True
|
||||
payload["gcs_path"] = gcs_path
|
||||
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/download-by-id",
|
||||
headers=self._headers(),
|
||||
json=payload,
|
||||
timeout=self.timeout,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
data = resp.json()
|
||||
return data["download_id"]
|
||||
|
||||
return await with_retry(
|
||||
self._download_by_id_impl,
|
||||
source_name, source_id, output_filename, target_file, download_url, gcs_path
|
||||
)
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Download by ID request failed: {e}")
|
||||
raise FlacfetchServiceError(f"Download by ID request failed: {_format_httpx_error(e)}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Download by ID failed: {e.response.status_code} - {e.response.text}")
|
||||
|
||||
async def _download_by_id_impl(
|
||||
self,
|
||||
source_name: str,
|
||||
source_id: str,
|
||||
output_filename: Optional[str] = None,
|
||||
target_file: Optional[str] = None,
|
||||
download_url: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Implementation of download_by_id without error wrapping - lets httpx exceptions bubble up."""
|
||||
async with httpx.AsyncClient() as client:
|
||||
payload = {
|
||||
"source_name": source_name,
|
||||
"source_id": source_id,
|
||||
}
|
||||
if output_filename:
|
||||
payload["output_filename"] = output_filename
|
||||
if target_file:
|
||||
payload["target_file"] = target_file
|
||||
if download_url:
|
||||
payload["download_url"] = download_url
|
||||
if gcs_path:
|
||||
payload["upload_to_gcs"] = True
|
||||
payload["gcs_path"] = gcs_path
|
||||
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/download-by-id",
|
||||
headers=self._headers(),
|
||||
json=payload,
|
||||
timeout=self.timeout,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
data = resp.json()
|
||||
return data["download_id"]
|
||||
|
||||
async def get_download_status(self, download_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Get the current status of a download.
|
||||
|
||||
|
||||
Args:
|
||||
download_id: Download ID from start_download
|
||||
|
||||
|
||||
Returns:
|
||||
Status dict with progress, speed, path info
|
||||
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On status check failure
|
||||
httpx.RequestError: On network-level failures (connection refused, timeout, etc.)
|
||||
These are NOT wrapped so wait_for_download can handle retries.
|
||||
FlacfetchServiceError: On application-level failures (404, other HTTP errors)
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
@@ -282,9 +413,10 @@ class FlacfetchClient:
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Status request failed: {e}")
|
||||
|
||||
except httpx.RequestError:
|
||||
# Let network errors bubble up so wait_for_download can handle retries
|
||||
raise
|
||||
except httpx.HTTPStatusError as e:
|
||||
if e.response.status_code == 404:
|
||||
raise FlacfetchServiceError(f"Download not found: {download_id}")
|
||||
@@ -296,35 +428,59 @@ class FlacfetchClient:
|
||||
timeout: int = 600,
|
||||
poll_interval: float = 2.0,
|
||||
progress_callback: Optional[callable] = None,
|
||||
max_consecutive_errors: int = 3,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Wait for a download to complete.
|
||||
|
||||
|
||||
Tolerates transient network errors during status polling. A single
|
||||
connection hiccup won't fail the entire download -- only N consecutive
|
||||
poll failures will.
|
||||
|
||||
Args:
|
||||
download_id: Download ID to wait for
|
||||
timeout: Maximum wait time in seconds
|
||||
poll_interval: Time between status checks
|
||||
progress_callback: Optional callback(status_dict) for progress updates
|
||||
|
||||
max_consecutive_errors: Number of consecutive poll failures before giving up
|
||||
|
||||
Returns:
|
||||
Final status dict with output_path or gcs_path
|
||||
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On download failure or timeout
|
||||
FlacfetchServiceError: On download failure, timeout, or too many consecutive poll errors
|
||||
"""
|
||||
elapsed = 0.0
|
||||
|
||||
consecutive_errors = 0
|
||||
|
||||
while elapsed < timeout:
|
||||
status = await self.get_download_status(download_id)
|
||||
|
||||
try:
|
||||
status = await self.get_download_status(download_id)
|
||||
consecutive_errors = 0 # Reset on success
|
||||
except httpx.RequestError as e:
|
||||
consecutive_errors += 1
|
||||
error_desc = _format_httpx_error(e)
|
||||
logger.warning(
|
||||
f"Status poll failed for download {download_id} "
|
||||
f"({consecutive_errors}/{max_consecutive_errors}): {error_desc}"
|
||||
)
|
||||
if consecutive_errors >= max_consecutive_errors:
|
||||
raise FlacfetchServiceError(
|
||||
f"Status polling failed after {consecutive_errors} consecutive "
|
||||
f"network errors (last: {error_desc})"
|
||||
)
|
||||
await asyncio.sleep(poll_interval)
|
||||
elapsed += poll_interval
|
||||
continue
|
||||
|
||||
if progress_callback:
|
||||
try:
|
||||
progress_callback(status)
|
||||
except Exception as e:
|
||||
logger.warning(f"Progress callback error: {e}")
|
||||
|
||||
|
||||
download_status = status.get("status")
|
||||
|
||||
|
||||
if download_status in ["complete", "seeding"]:
|
||||
logger.info(f"Download {download_id} complete: {status.get('gcs_path') or status.get('output_path')}")
|
||||
return status
|
||||
@@ -333,7 +489,7 @@ class FlacfetchClient:
|
||||
raise FlacfetchServiceError(f"Download failed: {error}")
|
||||
elif download_status == "cancelled":
|
||||
raise FlacfetchServiceError("Download was cancelled")
|
||||
|
||||
|
||||
# Log progress
|
||||
progress = status.get("progress", 0)
|
||||
speed = status.get("download_speed_kbps", 0)
|
||||
@@ -342,10 +498,10 @@ class FlacfetchClient:
|
||||
f"Download {download_id}: {progress:.1f}% "
|
||||
f"({speed:.1f} KB/s, {peers} peers)"
|
||||
)
|
||||
|
||||
|
||||
await asyncio.sleep(poll_interval)
|
||||
elapsed += poll_interval
|
||||
|
||||
|
||||
raise FlacfetchServiceError(f"Download timed out after {timeout}s")
|
||||
|
||||
async def list_torrents(self) -> Dict[str, Any]:
|
||||
@@ -366,7 +522,7 @@ class FlacfetchClient:
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"List torrents failed: {e}")
|
||||
raise FlacfetchServiceError(f"List torrents failed: {_format_httpx_error(e)}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"List torrents failed: {e.response.status_code}")
|
||||
|
||||
@@ -397,7 +553,7 @@ class FlacfetchClient:
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Delete torrent failed: {e}")
|
||||
raise FlacfetchServiceError(f"Delete torrent failed: {_format_httpx_error(e)}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Delete torrent failed: {e.response.status_code}")
|
||||
|
||||
@@ -431,7 +587,7 @@ class FlacfetchClient:
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Cleanup failed: {e}")
|
||||
raise FlacfetchServiceError(f"Cleanup failed: {_format_httpx_error(e)}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Cleanup failed: {e.response.status_code}")
|
||||
|
||||
@@ -472,7 +628,7 @@ class FlacfetchClient:
|
||||
return deleted
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Clear search cache request failed: {e}")
|
||||
raise FlacfetchServiceError(f"Clear search cache request failed: {_format_httpx_error(e)}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Clear search cache failed: {e.response.status_code} - {e.response.text}"
|
||||
@@ -502,12 +658,55 @@ class FlacfetchClient:
|
||||
return deleted_count
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Clear all cache request failed: {e}")
|
||||
raise FlacfetchServiceError(f"Clear all cache request failed: {_format_httpx_error(e)}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Clear all cache failed: {e.response.status_code} - {e.response.text}"
|
||||
)
|
||||
|
||||
# =========================================================================
|
||||
# YouTube Availability Check
|
||||
# =========================================================================
|
||||
|
||||
async def check_youtube(self, url: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Check if a YouTube video is available for download from this server's location.
|
||||
|
||||
Calls the flacfetch /check-youtube endpoint which uses yt-dlp to detect
|
||||
geo-restrictions, private/removed videos, and other unavailability reasons.
|
||||
|
||||
Args:
|
||||
url: YouTube video URL or video ID
|
||||
|
||||
Returns:
|
||||
Dict with keys: available, video_id, title, error,
|
||||
is_geo_restricted, is_age_restricted, is_private, is_removed
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On request failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/check-youtube",
|
||||
headers=self._headers(),
|
||||
json={"url": url},
|
||||
timeout=15,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"YouTube availability check failed: {_format_httpx_error(e)}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"YouTube availability check failed: {e.response.status_code} - {e.response.text}"
|
||||
)
|
||||
|
||||
# =========================================================================
|
||||
# Cache Management
|
||||
# =========================================================================
|
||||
|
||||
async def get_cache_stats(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Get statistics about the cache.
|
||||
@@ -529,7 +728,7 @@ class FlacfetchClient:
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Get cache stats request failed: {e}")
|
||||
raise FlacfetchServiceError(f"Get cache stats request failed: {_format_httpx_error(e)}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Get cache stats failed: {e.response.status_code} - {e.response.text}"
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
"""Service for generating personalized referral flyer PDFs."""
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
TEMPLATE_DIR = os.path.join(os.path.dirname(__file__), "..", "templates", "printables")
|
||||
|
||||
CHROMIUM_PATHS = [
|
||||
"/usr/bin/chromium",
|
||||
"/usr/bin/chromium-browser",
|
||||
"/usr/bin/google-chrome",
|
||||
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
|
||||
]
|
||||
|
||||
VALID_THEMES = {"light", "dark"}
|
||||
MAX_QR_DATA_URL_BYTES = 500_000
|
||||
|
||||
|
||||
class FlyerError(Exception):
|
||||
"""Error during flyer generation."""
|
||||
|
||||
|
||||
class FlyerService:
|
||||
"""Generates personalized referral flyer PDFs."""
|
||||
|
||||
def __init__(self):
|
||||
self._chromium_path: str | None = None
|
||||
|
||||
def _get_chromium(self) -> str:
|
||||
"""Lazily find Chromium binary. Only needed when generating PDFs."""
|
||||
if self._chromium_path:
|
||||
return self._chromium_path
|
||||
for path in CHROMIUM_PATHS:
|
||||
if os.path.exists(path):
|
||||
self._chromium_path = path
|
||||
return path
|
||||
for name in ["chromium", "chromium-browser", "google-chrome"]:
|
||||
found = shutil.which(name)
|
||||
if found:
|
||||
self._chromium_path = found
|
||||
return found
|
||||
raise FlyerError("Chromium not found. Install chromium to generate flyer PDFs.")
|
||||
|
||||
def _validate_qr_data_url(self, qr_data_url: str) -> None:
|
||||
if not qr_data_url.startswith("data:image/"):
|
||||
raise FlyerError("Invalid QR data URL: must be a data:image/ URL")
|
||||
if len(qr_data_url) > MAX_QR_DATA_URL_BYTES:
|
||||
raise FlyerError(f"QR data URL too large (>{MAX_QR_DATA_URL_BYTES} bytes)")
|
||||
|
||||
def _render_template(
|
||||
self,
|
||||
theme: str,
|
||||
referral_code: str,
|
||||
discount_percent: int,
|
||||
qr_data_url: str,
|
||||
) -> str:
|
||||
if theme not in VALID_THEMES:
|
||||
raise FlyerError(f"Invalid theme: {theme}. Must be one of: {VALID_THEMES}")
|
||||
|
||||
suffix = "-dark" if theme == "dark" else ""
|
||||
template_path = os.path.join(TEMPLATE_DIR, f"website-referral-flyer{suffix}.html")
|
||||
|
||||
with open(template_path) as f:
|
||||
html = f.read()
|
||||
|
||||
html = html.replace("{{REFERRAL_CODE}}", referral_code.upper())
|
||||
html = html.replace("{{REFERRAL_CODE_LOWER}}", referral_code.lower())
|
||||
html = html.replace("{{DISCOUNT_PERCENT}}", str(discount_percent))
|
||||
html = html.replace("{{QR_DATA_URL}}", qr_data_url)
|
||||
|
||||
return html
|
||||
|
||||
def generate_pdf(
|
||||
self,
|
||||
theme: str,
|
||||
referral_code: str,
|
||||
discount_percent: int,
|
||||
qr_data_url: str,
|
||||
) -> bytes:
|
||||
"""Generate a personalized flyer PDF. Returns raw PDF bytes."""
|
||||
self._validate_qr_data_url(qr_data_url)
|
||||
html = self._render_template(theme, referral_code, discount_percent, qr_data_url)
|
||||
|
||||
with tempfile.NamedTemporaryFile(
|
||||
suffix=".html", delete=False, mode="w", encoding="utf-8"
|
||||
) as tmp_html:
|
||||
tmp_html.write(html)
|
||||
tmp_html_path = tmp_html.name
|
||||
|
||||
tmp_pdf_path = tmp_html_path.replace(".html", ".pdf")
|
||||
|
||||
try:
|
||||
cmd = [
|
||||
self._get_chromium(),
|
||||
"--headless",
|
||||
f"--print-to-pdf={tmp_pdf_path}",
|
||||
"--no-margins",
|
||||
"--print-background",
|
||||
"--no-pdf-header-footer",
|
||||
"--virtual-time-budget=5000",
|
||||
"--paper-width=8.5",
|
||||
"--paper-height=11",
|
||||
"--disable-gpu",
|
||||
"--no-sandbox",
|
||||
tmp_html_path,
|
||||
]
|
||||
|
||||
result = subprocess.run(
|
||||
cmd, capture_output=True, text=True, timeout=30
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
logger.error("Chromium PDF generation failed: %s", result.stderr)
|
||||
raise FlyerError(f"PDF generation failed: {result.stderr[:200]}")
|
||||
|
||||
if not os.path.exists(tmp_pdf_path):
|
||||
raise FlyerError("PDF generation failed: no output file created")
|
||||
|
||||
with open(tmp_pdf_path, "rb") as f:
|
||||
return f.read()
|
||||
|
||||
finally:
|
||||
for path in [tmp_html_path, tmp_pdf_path]:
|
||||
try:
|
||||
os.unlink(path)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
_flyer_service = None
|
||||
|
||||
|
||||
def get_flyer_service() -> FlyerService:
|
||||
global _flyer_service
|
||||
if _flyer_service is None:
|
||||
_flyer_service = FlyerService()
|
||||
return _flyer_service
|
||||
@@ -1,8 +1,11 @@
|
||||
import asyncio
|
||||
import glob as glob_module
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
@@ -13,6 +16,7 @@ from typing import Optional
|
||||
|
||||
from fastapi import FastAPI, BackgroundTasks, HTTPException, Header, Depends
|
||||
from google.cloud import storage
|
||||
from packaging.version import Version
|
||||
from pydantic import BaseModel
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
@@ -57,12 +61,27 @@ class EncodePreviewRequest(BaseModel):
|
||||
font_gcs_path: Optional[str] = None # gs://bucket/path/to/custom-font.ttf
|
||||
|
||||
|
||||
class RenderVideoRequest(BaseModel):
|
||||
job_id: str
|
||||
original_corrections_gcs_path: str
|
||||
updated_corrections_gcs_path: Optional[str] = None
|
||||
audio_gcs_path: str
|
||||
style_params_gcs_path: Optional[str] = None
|
||||
style_assets: Optional[dict] = None
|
||||
output_gcs_prefix: str
|
||||
artist: str
|
||||
title: str
|
||||
subtitle_offset_ms: int = 0
|
||||
video_resolution: str = "4k"
|
||||
|
||||
|
||||
class JobStatus(BaseModel):
|
||||
job_id: str
|
||||
status: str # pending, running, complete, failed
|
||||
progress: int # 0-100
|
||||
error: Optional[str] = None
|
||||
output_files: Optional[list[str]] = None
|
||||
metadata: Optional[dict] = None
|
||||
|
||||
|
||||
def download_from_gcs(gcs_uri: str, local_path: Path):
|
||||
@@ -134,6 +153,59 @@ def upload_single_file_to_gcs(local_path: Path, gcs_uri: str):
|
||||
logger.info(f"Uploaded: {local_path} -> {gcs_uri}")
|
||||
|
||||
|
||||
# Style asset disk cache directory
|
||||
# Persists across jobs to avoid re-downloading the same theme assets (fonts, backgrounds)
|
||||
STYLE_CACHE_DIR = Path("/var/cache/karaoke-gen/styles")
|
||||
|
||||
|
||||
def download_with_cache(gcs_uri: str, local_path: Path, cache_dir: Optional[Path] = STYLE_CACHE_DIR):
|
||||
"""Download a file from GCS, using a local disk cache to skip repeated downloads.
|
||||
|
||||
Cache key is SHA-256 of the GCS URI. Cache hits are validated by comparing
|
||||
the cached file size against the GCS object size (via a lightweight metadata
|
||||
call). On size mismatch the stale entry is replaced.
|
||||
When cache_dir is None, downloads directly (no caching).
|
||||
"""
|
||||
if cache_dir is None:
|
||||
download_single_file_from_gcs(gcs_uri, local_path)
|
||||
return
|
||||
|
||||
cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
cache_key = hashlib.sha256(gcs_uri.encode()).hexdigest()
|
||||
cached_path = cache_dir / cache_key
|
||||
|
||||
if cached_path.exists():
|
||||
# Validate cached file against GCS object size
|
||||
try:
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
blob_name = parts[1] if len(parts) > 1 else ""
|
||||
blob = storage_client.bucket(bucket_name).blob(blob_name)
|
||||
blob.reload() # Fetch metadata (lightweight HEAD request)
|
||||
gcs_size = blob.size
|
||||
|
||||
cached_size = cached_path.stat().st_size
|
||||
if gcs_size is not None and cached_size != gcs_size:
|
||||
logger.warning(
|
||||
f"Cache stale for {gcs_uri}: cached={cached_size}B, GCS={gcs_size}B. Re-downloading."
|
||||
)
|
||||
cached_path.unlink()
|
||||
else:
|
||||
logger.info(f"Cache hit for {gcs_uri} -> {cached_path}")
|
||||
shutil.copy2(str(cached_path), str(local_path))
|
||||
return
|
||||
except Exception as e:
|
||||
# On transient GCS errors, serve from cache rather than failing the job
|
||||
logger.warning(f"Cache validation failed for {gcs_uri}: {e}. Serving from cache.")
|
||||
shutil.copy2(str(cached_path), str(local_path))
|
||||
return
|
||||
|
||||
logger.info(f"Cache miss for {gcs_uri}, downloading...")
|
||||
download_single_file_from_gcs(gcs_uri, local_path)
|
||||
# Store in cache
|
||||
shutil.copy2(str(local_path), str(cached_path))
|
||||
|
||||
|
||||
def run_preview_encoding(job_id: str, work_dir: Path, request: "EncodePreviewRequest"):
|
||||
# Run FFmpeg encoding for preview video (480x270, fast settings)
|
||||
jobs[job_id]["status"] = "running"
|
||||
@@ -237,6 +309,192 @@ def run_preview_encoding(job_id: str, work_dir: Path, request: "EncodePreviewReq
|
||||
raise
|
||||
|
||||
|
||||
def run_render_video(job_id: str, work_dir: Path, request: "RenderVideoRequest"):
|
||||
"""Run video rendering using OutputGenerator from the karaoke-gen wheel.
|
||||
|
||||
Downloads corrections, audio, and style assets from GCS, runs OutputGenerator
|
||||
to produce with_vocals.mkv and subtitle files, then uploads results.
|
||||
Style assets use disk cache to avoid repeated downloads of the same theme.
|
||||
|
||||
This mirrors the logic in render_video_worker.py but runs on the GCE VM
|
||||
instead of Cloud Run, providing more memory and CPU for ffmpeg/libass.
|
||||
"""
|
||||
jobs[job_id]["status"] = "running"
|
||||
jobs[job_id]["progress"] = 5
|
||||
|
||||
try:
|
||||
# Import from installed karaoke-gen wheel
|
||||
from karaoke_gen.lyrics_transcriber.output.generator import OutputGenerator
|
||||
from karaoke_gen.lyrics_transcriber.output.countdown_processor import CountdownProcessor
|
||||
from karaoke_gen.lyrics_transcriber.types import CorrectionResult
|
||||
from karaoke_gen.lyrics_transcriber.correction.operations import CorrectionOperations
|
||||
from karaoke_gen.lyrics_transcriber.core.config import OutputConfig
|
||||
from karaoke_gen.style_loader import load_styles_from_gcs
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
|
||||
# 1. Download original corrections
|
||||
corrections_path = work_dir / "corrections.json"
|
||||
download_single_file_from_gcs(request.original_corrections_gcs_path, corrections_path)
|
||||
jobs[job_id]["progress"] = 15
|
||||
|
||||
with open(corrections_path, 'r', encoding='utf-8') as f:
|
||||
original_data = json.load(f)
|
||||
base_result = CorrectionResult.from_dict(original_data)
|
||||
|
||||
# 2. Apply user corrections if available
|
||||
if request.updated_corrections_gcs_path:
|
||||
updated_path = work_dir / "corrections_updated.json"
|
||||
download_single_file_from_gcs(request.updated_corrections_gcs_path, updated_path)
|
||||
with open(updated_path, 'r', encoding='utf-8') as f:
|
||||
updated_data = json.load(f)
|
||||
if isinstance(updated_data, dict) and "corrections" in updated_data:
|
||||
correction_result = CorrectionOperations.update_correction_result_with_data(
|
||||
base_result, updated_data
|
||||
)
|
||||
logger.info(f"[job:{job_id}] Applied user corrections")
|
||||
else:
|
||||
logger.warning(f"[job:{job_id}] corrections_updated.json exists but has no 'corrections' key, using base result")
|
||||
correction_result = base_result
|
||||
else:
|
||||
correction_result = base_result
|
||||
|
||||
jobs[job_id]["progress"] = 25
|
||||
|
||||
# 3. Download audio
|
||||
audio_path = work_dir / "audio.flac"
|
||||
download_single_file_from_gcs(request.audio_gcs_path, audio_path)
|
||||
jobs[job_id]["progress"] = 35
|
||||
|
||||
# 4. Process countdown intro
|
||||
countdown_processor = CountdownProcessor(cache_dir=str(work_dir), logger=logger)
|
||||
correction_result, audio_path_str, padding_added, padding_seconds = countdown_processor.process(
|
||||
correction_result=correction_result,
|
||||
audio_filepath=str(audio_path),
|
||||
)
|
||||
audio_path = Path(audio_path_str)
|
||||
jobs[job_id]["progress"] = 40
|
||||
|
||||
# 5. Download style assets (with disk cache for repeated themes)
|
||||
def cached_download(gcs_path, local_path):
|
||||
download_with_cache(gcs_path, Path(local_path), STYLE_CACHE_DIR)
|
||||
|
||||
styles_path, style_data = load_styles_from_gcs(
|
||||
style_params_gcs_path=request.style_params_gcs_path,
|
||||
style_assets=request.style_assets,
|
||||
temp_dir=str(work_dir),
|
||||
download_func=cached_download,
|
||||
logger=logger,
|
||||
)
|
||||
|
||||
# Register any downloaded fonts with fontconfig
|
||||
for asset_key, gcs_path in (request.style_assets or {}).items():
|
||||
if 'font' in asset_key.lower() and gcs_path.endswith(('.ttf', '.otf', '.woff', '.woff2')):
|
||||
fonts_dir = Path("/usr/local/share/fonts/custom")
|
||||
fonts_dir.mkdir(parents=True, exist_ok=True)
|
||||
font_filename = gcs_path.split("/")[-1]
|
||||
font_dest = fonts_dir / font_filename
|
||||
if not font_dest.exists():
|
||||
ext = os.path.splitext(gcs_path)[1]
|
||||
style_font = work_dir / "style" / f"{asset_key}{ext}"
|
||||
if style_font.exists():
|
||||
shutil.copy2(str(style_font), str(font_dest))
|
||||
subprocess.run(["fc-cache", "-fv"], capture_output=True)
|
||||
logger.info(f"Registered font: {font_filename}")
|
||||
|
||||
jobs[job_id]["progress"] = 50
|
||||
|
||||
# 6. Configure and run OutputGenerator
|
||||
output_dir = work_dir / "output"
|
||||
cache_dir = work_dir / "cache"
|
||||
output_dir.mkdir(exist_ok=True)
|
||||
cache_dir.mkdir(exist_ok=True)
|
||||
|
||||
config = OutputConfig(
|
||||
output_dir=str(output_dir),
|
||||
cache_dir=str(cache_dir),
|
||||
output_styles_json=styles_path,
|
||||
render_video=True,
|
||||
generate_cdg=False,
|
||||
generate_plain_text=True,
|
||||
generate_lrc=True,
|
||||
video_resolution=request.video_resolution,
|
||||
subtitle_offset_ms=request.subtitle_offset_ms,
|
||||
)
|
||||
|
||||
output_generator = OutputGenerator(config, logger)
|
||||
|
||||
safe_artist = sanitize_filename(request.artist) if request.artist else "Unknown"
|
||||
safe_title = sanitize_filename(request.title) if request.title else "Unknown"
|
||||
output_prefix = f"{safe_artist} - {safe_title}"
|
||||
|
||||
logger.info(f"[job:{job_id}] Generating outputs with prefix '{output_prefix}'")
|
||||
jobs[job_id]["progress"] = 55
|
||||
|
||||
outputs = output_generator.generate_outputs(
|
||||
transcription_corrected=correction_result,
|
||||
lyrics_results={},
|
||||
output_prefix=output_prefix,
|
||||
audio_filepath=str(audio_path),
|
||||
artist=request.artist,
|
||||
title=request.title,
|
||||
)
|
||||
|
||||
jobs[job_id]["progress"] = 85
|
||||
|
||||
# 7. Upload outputs to GCS
|
||||
output_prefix_gcs = request.output_gcs_prefix.rstrip("/")
|
||||
output_files = []
|
||||
|
||||
if outputs.video and os.path.exists(outputs.video):
|
||||
gcs_path = f"{output_prefix_gcs}/videos/with_vocals.mkv"
|
||||
upload_single_file_to_gcs(Path(outputs.video), gcs_path)
|
||||
output_files.append(gcs_path)
|
||||
logger.info(f"[job:{job_id}] Uploaded with_vocals.mkv ({os.path.getsize(outputs.video)} bytes)")
|
||||
else:
|
||||
raise RuntimeError("Video generation failed - no output file produced")
|
||||
|
||||
if outputs.ass and os.path.exists(outputs.ass):
|
||||
gcs_path = f"{output_prefix_gcs}/lyrics/karaoke.ass"
|
||||
upload_single_file_to_gcs(Path(outputs.ass), gcs_path)
|
||||
output_files.append(gcs_path)
|
||||
|
||||
if outputs.lrc and os.path.exists(outputs.lrc):
|
||||
gcs_path = f"{output_prefix_gcs}/lyrics/karaoke.lrc"
|
||||
upload_single_file_to_gcs(Path(outputs.lrc), gcs_path)
|
||||
output_files.append(gcs_path)
|
||||
|
||||
if outputs.corrected_txt and os.path.exists(outputs.corrected_txt):
|
||||
gcs_path = f"{output_prefix_gcs}/lyrics/corrected.txt"
|
||||
upload_single_file_to_gcs(Path(outputs.corrected_txt), gcs_path)
|
||||
output_files.append(gcs_path)
|
||||
|
||||
jobs[job_id]["progress"] = 95
|
||||
jobs[job_id]["output_files"] = output_files
|
||||
jobs[job_id]["metadata"] = {
|
||||
"countdown_padding_added": padding_added,
|
||||
"countdown_padding_seconds": padding_seconds if padding_added else 0,
|
||||
}
|
||||
|
||||
logger.info(f"[job:{job_id}] Render video complete. Output files: {output_files}")
|
||||
|
||||
except ImportError as e:
|
||||
error_msg = (
|
||||
f"OutputGenerator not available: {e}. "
|
||||
"The karaoke-gen wheel must be installed. "
|
||||
"Check that ensure_latest_wheel() succeeded."
|
||||
)
|
||||
logger.error(error_msg)
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = error_msg
|
||||
raise RuntimeError(error_msg) from e
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"[job:{job_id}] Render video failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
raise
|
||||
|
||||
|
||||
def ensure_latest_wheel():
|
||||
'''Download and install latest karaoke-gen wheel from GCS.
|
||||
|
||||
@@ -254,12 +512,23 @@ def ensure_latest_wheel():
|
||||
|
||||
# Find the downloaded wheel (get the latest by version sorting)
|
||||
wheels = glob_module.glob("/tmp/karaoke_gen-*.whl")
|
||||
# Filter out karaoke_gen-current.whl (not a valid PEP 427 wheel name)
|
||||
wheels = [w for w in wheels if '-current' not in w]
|
||||
|
||||
if not wheels:
|
||||
logger.warning("No wheel found in GCS, using fallback encoding logic")
|
||||
return False
|
||||
|
||||
# Sort to get latest version
|
||||
wheel_path = sorted(wheels)[-1]
|
||||
# Sort to get latest version (by semantic version, not alphabetically)
|
||||
def extract_version(wheel_path):
|
||||
"""Extract version from wheel filename like karaoke_gen-0.116.1-py3-none-any.whl"""
|
||||
match = re.search(r'karaoke_gen-([0-9.]+)-', wheel_path)
|
||||
if match:
|
||||
return Version(match.group(1))
|
||||
return Version("0.0.0") # Fallback for unparseable filenames
|
||||
|
||||
wheels.sort(key=extract_version, reverse=True)
|
||||
wheel_path = wheels[0]
|
||||
logger.info(f"Installing wheel: {wheel_path}")
|
||||
|
||||
# Install (or upgrade) the wheel
|
||||
@@ -294,6 +563,43 @@ def find_file(work_dir: Path, *patterns):
|
||||
return None
|
||||
|
||||
|
||||
def generate_mov_from_png(png_path: Path, mov_path: Path, duration: int = 5) -> Path:
|
||||
"""Generate a MOV video from a static PNG image using FFmpeg.
|
||||
|
||||
This runs on the GCE encoding worker (not Cloud Run) because 4K H.264
|
||||
encoding requires more memory than the right-sized Cloud Run container
|
||||
provides (2Gi). See PR #640 / #647.
|
||||
|
||||
Args:
|
||||
png_path: Path to input PNG image
|
||||
mov_path: Path for output MOV video
|
||||
duration: Video duration in seconds (default 5)
|
||||
|
||||
Returns:
|
||||
Path to generated MOV file
|
||||
|
||||
Raises:
|
||||
RuntimeError: If FFmpeg fails
|
||||
"""
|
||||
command = [
|
||||
"ffmpeg", "-hide_banner", "-loglevel", "error", "-nostats", "-y",
|
||||
"-loop", "1", "-framerate", "30", "-i", str(png_path),
|
||||
"-f", "lavfi", "-i", "anullsrc",
|
||||
"-c:v", "libx264", "-r", "30", "-t", str(duration),
|
||||
"-pix_fmt", "yuv420p", "-vf", "scale=3840:2160",
|
||||
"-c:a", "aac", "-shortest",
|
||||
str(mov_path),
|
||||
]
|
||||
logger.info(f"Generating MOV from PNG: {png_path.name} -> {mov_path.name}")
|
||||
result = subprocess.run(command, capture_output=True, text=True, timeout=120)
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(f"FFmpeg failed to generate {mov_path.name}: {result.stderr}")
|
||||
if not mov_path.exists() or mov_path.stat().st_size < 1024:
|
||||
raise RuntimeError(f"FFmpeg produced invalid output for {mov_path.name}")
|
||||
logger.info(f"Generated MOV: {mov_path.name} ({mov_path.stat().st_size} bytes)")
|
||||
return mov_path
|
||||
|
||||
|
||||
def run_encoding(job_id: str, work_dir: Path, config: dict):
|
||||
'''Run encoding using LocalEncodingService (single source of truth).
|
||||
|
||||
@@ -316,14 +622,43 @@ def run_encoding(job_id: str, work_dir: Path, config: dict):
|
||||
# Get artist/title from config for proper naming
|
||||
artist = config.get("artist", "Unknown Artist")
|
||||
title = config.get("title", "Unknown Title")
|
||||
base_name = f"{artist} - {title}"
|
||||
# Sanitize for use in file paths (e.g., title "PORTRAIT/WALK/BORN" → "PORTRAIT_WALK_BORN")
|
||||
safe_artist = re.sub(r'[<>:"/\\|?*]', '_', artist)
|
||||
safe_title = re.sub(r'[<>:"/\\|?*]', '_', title)
|
||||
base_name = f"{safe_artist} - {safe_title}"
|
||||
logger.info(f"Encoding for: {base_name}")
|
||||
|
||||
# Find input files in work_dir
|
||||
# Title/end screens are in screens/ subdirectory
|
||||
# Title/end screens: look for MOV first, fall back to generating from PNG.
|
||||
# Since PR #640 right-sized Cloud Run to 2Gi, the screens_worker only generates
|
||||
# PNG/JPG images — MOV video generation now happens here on the GCE worker.
|
||||
title_video = find_file(work_dir, "screens/title.mov", "*Title*.mov", "*title*.mov")
|
||||
end_video = find_file(work_dir, "screens/end.mov", "*End*.mov", "*end*.mov")
|
||||
|
||||
# Get intro/end video durations from style config (default 5s)
|
||||
intro_duration = config.get("intro_video_duration", 5)
|
||||
end_duration = config.get("end_video_duration", 5)
|
||||
|
||||
# Generate title MOV from PNG if missing or invalid (< 1KB = empty shell)
|
||||
if not title_video or title_video.stat().st_size < 1024:
|
||||
title_png = find_file(work_dir, "screens/title.png", "*Title*.png", "*title*.png")
|
||||
if title_png:
|
||||
title_mov_path = title_png.parent / "title.mov"
|
||||
logger.info(f"Title MOV missing/invalid, generating from PNG: {title_png}")
|
||||
title_video = generate_mov_from_png(title_png, title_mov_path, duration=intro_duration)
|
||||
elif title_video:
|
||||
logger.warning(f"Title MOV is only {title_video.stat().st_size} bytes and no PNG found")
|
||||
|
||||
# Generate end MOV from PNG if missing or invalid
|
||||
if not end_video or (end_video and end_video.stat().st_size < 1024):
|
||||
end_png = find_file(work_dir, "screens/end.png", "*End*.png", "*end*.png")
|
||||
if end_png:
|
||||
end_mov_path = end_png.parent / "end.mov"
|
||||
logger.info(f"End MOV missing/invalid, generating from PNG: {end_png}")
|
||||
end_video = generate_mov_from_png(end_png, end_mov_path, duration=end_duration)
|
||||
elif end_video:
|
||||
logger.warning(f"End MOV is only {end_video.stat().st_size} bytes and no PNG found")
|
||||
|
||||
# Karaoke video - search for With Vocals or main karaoke video
|
||||
# IMPORTANT: Search order matters! Search specific paths first to avoid picking up
|
||||
# old encoding outputs from finals/ directory when re-encoding after reset
|
||||
@@ -357,9 +692,26 @@ def run_encoding(job_id: str, work_dir: Path, config: dict):
|
||||
|
||||
# Instrumental audio - respect user's selection from encoding config
|
||||
instrumental_selection = config.get("instrumental_selection", "clean")
|
||||
existing_instrumental = config.get("existing_instrumental")
|
||||
logger.info(f"Instrumental selection from config: {instrumental_selection}")
|
||||
if existing_instrumental:
|
||||
logger.info(f"Existing instrumental from config: {existing_instrumental}")
|
||||
|
||||
if instrumental_selection == "with_backing":
|
||||
if existing_instrumental:
|
||||
# User-provided instrumental uploaded at job creation
|
||||
# Downloaded to work_dir by process_job before run_encoding is called
|
||||
instrumental = find_file(
|
||||
work_dir,
|
||||
"*existing_instrumental*", "*Instrumental User*",
|
||||
)
|
||||
elif instrumental_selection == "custom":
|
||||
# Custom instrumental created via mute-region editing in review UI
|
||||
instrumental = find_file(
|
||||
work_dir,
|
||||
"*custom_instrumental*.flac", "*Instrumental Custom*.flac",
|
||||
"*custom_instrumental*.mp3",
|
||||
)
|
||||
elif instrumental_selection == "with_backing":
|
||||
# User selected instrumental with backing vocals
|
||||
instrumental = find_file(
|
||||
work_dir,
|
||||
@@ -472,6 +824,17 @@ async def process_job(job_id: str, request: EncodeRequest):
|
||||
logger.info(f"Downloading from {request.input_gcs_path}")
|
||||
download_from_gcs(request.input_gcs_path, work_dir)
|
||||
|
||||
# Download existing instrumental if present (stored under uploads/, not jobs/)
|
||||
existing_instrumental = request.encoding_config.get("existing_instrumental") if request.encoding_config else None
|
||||
if existing_instrumental:
|
||||
# Extract bucket name from input_gcs_path (gs://bucket/jobs/...)
|
||||
bucket_name = request.input_gcs_path.replace("gs://", "").split("/", 1)[0]
|
||||
ext = Path(existing_instrumental).suffix.lower()
|
||||
dest = work_dir / f"existing_instrumental{ext}"
|
||||
gcs_uri = f"gs://{bucket_name}/{existing_instrumental}"
|
||||
logger.info(f"Downloading existing instrumental: {gcs_uri} -> {dest}")
|
||||
download_single_file_from_gcs(gcs_uri, dest)
|
||||
|
||||
# Run encoding in thread pool (CPU-bound)
|
||||
loop = asyncio.get_event_loop()
|
||||
output_dir = await loop.run_in_executor(
|
||||
@@ -540,6 +903,36 @@ async def process_preview_job(job_id: str, request: EncodePreviewRequest):
|
||||
jobs[job_id]["error"] = str(e)
|
||||
|
||||
|
||||
async def process_render_video_job(job_id: str, request: RenderVideoRequest):
|
||||
# Process a render-video job asynchronously
|
||||
try:
|
||||
# Download and install latest wheel at job start (allows hot updates without restart)
|
||||
ensure_latest_wheel()
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
work_dir = Path(temp_dir) / "work"
|
||||
work_dir.mkdir()
|
||||
|
||||
# Run render-video in thread pool (CPU-bound)
|
||||
loop = asyncio.get_event_loop()
|
||||
await loop.run_in_executor(
|
||||
executor,
|
||||
run_render_video,
|
||||
job_id,
|
||||
work_dir,
|
||||
request
|
||||
)
|
||||
|
||||
jobs[job_id]["status"] = "complete"
|
||||
jobs[job_id]["progress"] = 100
|
||||
logger.info(f"Render video job {job_id} complete")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Render video job {job_id} failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
|
||||
|
||||
@app.post("/encode-preview")
|
||||
async def submit_preview_encode_job(request: EncodePreviewRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
|
||||
# Submit a preview encoding job
|
||||
@@ -571,13 +964,54 @@ async def submit_preview_encode_job(request: EncodePreviewRequest, background_ta
|
||||
return {"status": "accepted", "job_id": job_id}
|
||||
|
||||
|
||||
@app.post("/render-video")
|
||||
async def submit_render_video_job(request: RenderVideoRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
|
||||
# Submit a render-video job
|
||||
job_id = request.job_id
|
||||
|
||||
# If job already exists, return cached result or current status
|
||||
if job_id in jobs:
|
||||
existing_job = jobs[job_id]
|
||||
if existing_job["status"] == "complete":
|
||||
return {"status": "cached", "job_id": job_id, "output_files": existing_job.get("output_files"), "metadata": existing_job.get("metadata")}
|
||||
elif existing_job["status"] == "failed":
|
||||
# Previous attempt failed, allow retry by replacing the job
|
||||
pass
|
||||
else:
|
||||
# Job is still in progress
|
||||
return {"status": "in_progress", "job_id": job_id}
|
||||
|
||||
jobs[job_id] = {
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"progress": 0,
|
||||
"error": None,
|
||||
"output_files": None,
|
||||
"metadata": None,
|
||||
}
|
||||
|
||||
background_tasks.add_task(process_render_video_job, job_id, request)
|
||||
|
||||
return {"status": "accepted", "job_id": job_id}
|
||||
|
||||
|
||||
@app.post("/encode")
|
||||
async def submit_encode_job(request: EncodeRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
|
||||
# Submit an encoding job
|
||||
job_id = request.job_id
|
||||
|
||||
# If job already exists, return cached result or current status
|
||||
if job_id in jobs:
|
||||
raise HTTPException(status_code=409, detail=f"Job {job_id} already exists")
|
||||
existing_job = jobs[job_id]
|
||||
if existing_job["status"] == "complete":
|
||||
# Return cached result - encoding already done
|
||||
return {"status": "cached", "job_id": job_id, "output_files": existing_job.get("output_files")}
|
||||
elif existing_job["status"] == "failed":
|
||||
# Previous attempt failed, allow retry by replacing the job
|
||||
pass
|
||||
else:
|
||||
# Job is still in progress
|
||||
return {"status": "in_progress", "job_id": job_id}
|
||||
|
||||
jobs[job_id] = {
|
||||
"job_id": job_id,
|
||||
|
||||
@@ -12,13 +12,19 @@ shared with YouTube credentials if scopes include drive.file.
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import ssl
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from tenacity import Retrying, stop_after_attempt, wait_exponential, retry_if_exception_type
|
||||
|
||||
from backend.config import get_settings
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Transient errors that indicate a stale HTTP connection (Cloud Run idle containers)
|
||||
TRANSIENT_ERRORS = (BrokenPipeError, ConnectionResetError, ConnectionError, ssl.SSLError)
|
||||
|
||||
|
||||
class GoogleDriveService:
|
||||
"""Google Drive operations using native API."""
|
||||
@@ -33,6 +39,11 @@ class GoogleDriveService:
|
||||
self._credentials_data: Optional[Dict[str, Any]] = None
|
||||
self._loaded = False
|
||||
|
||||
def _reset_service(self):
|
||||
"""Reset the cached Drive service to force a fresh connection on next use."""
|
||||
logger.info("Resetting Google Drive service connection")
|
||||
self._service = None
|
||||
|
||||
def _load_credentials(self) -> Optional[Dict[str, Any]]:
|
||||
"""Load OAuth credentials from Secret Manager."""
|
||||
if self._loaded:
|
||||
@@ -127,6 +138,9 @@ class GoogleDriveService:
|
||||
"""
|
||||
Get existing folder or create new one, return folder ID.
|
||||
|
||||
Retries on transient connection errors (BrokenPipeError, SSLError)
|
||||
that occur when Cloud Run containers sit idle between jobs.
|
||||
|
||||
Args:
|
||||
parent_id: Parent folder ID
|
||||
folder_name: Name of folder to find or create
|
||||
@@ -134,33 +148,41 @@ class GoogleDriveService:
|
||||
Returns:
|
||||
Folder ID
|
||||
"""
|
||||
logger.info(f"Looking for folder '{folder_name}' in parent {parent_id}")
|
||||
for attempt in Retrying(
|
||||
stop=stop_after_attempt(3),
|
||||
wait=wait_exponential(multiplier=1, min=2, max=8),
|
||||
retry=retry_if_exception_type(TRANSIENT_ERRORS),
|
||||
before_sleep=lambda retry_state: self._reset_service(),
|
||||
reraise=True,
|
||||
):
|
||||
with attempt:
|
||||
logger.info(f"Looking for folder '{folder_name}' in parent {parent_id}")
|
||||
|
||||
# Search for existing folder
|
||||
# Escape single quotes in folder name for Google Drive API query syntax
|
||||
escaped_folder_name = folder_name.replace("'", "\\'")
|
||||
query = (
|
||||
f"name='{escaped_folder_name}' and '{parent_id}' in parents "
|
||||
f"and mimeType='application/vnd.google-apps.folder' and trashed=false"
|
||||
)
|
||||
results = self.service.files().list(q=query, fields="files(id, name)").execute()
|
||||
# Search for existing folder
|
||||
# Escape single quotes in folder name for Google Drive API query syntax
|
||||
escaped_folder_name = folder_name.replace("'", "\\'")
|
||||
query = (
|
||||
f"name='{escaped_folder_name}' and '{parent_id}' in parents "
|
||||
f"and mimeType='application/vnd.google-apps.folder' and trashed=false"
|
||||
)
|
||||
results = self.service.files().list(q=query, fields="files(id, name)").execute()
|
||||
|
||||
if results.get("files"):
|
||||
folder_id = results["files"][0]["id"]
|
||||
logger.info(f"Found existing folder '{folder_name}': {folder_id}")
|
||||
return folder_id
|
||||
if results.get("files"):
|
||||
folder_id = results["files"][0]["id"]
|
||||
logger.info(f"Found existing folder '{folder_name}': {folder_id}")
|
||||
return folder_id
|
||||
|
||||
# Create folder
|
||||
logger.info(f"Creating new folder '{folder_name}'")
|
||||
metadata = {
|
||||
"name": folder_name,
|
||||
"mimeType": "application/vnd.google-apps.folder",
|
||||
"parents": [parent_id],
|
||||
}
|
||||
folder = self.service.files().create(body=metadata, fields="id").execute()
|
||||
folder_id = folder["id"]
|
||||
logger.info(f"Created folder '{folder_name}': {folder_id}")
|
||||
return folder_id
|
||||
# Create folder
|
||||
logger.info(f"Creating new folder '{folder_name}'")
|
||||
metadata = {
|
||||
"name": folder_name,
|
||||
"mimeType": "application/vnd.google-apps.folder",
|
||||
"parents": [parent_id],
|
||||
}
|
||||
folder = self.service.files().create(body=metadata, fields="id").execute()
|
||||
folder_id = folder["id"]
|
||||
logger.info(f"Created folder '{folder_name}': {folder_id}")
|
||||
return folder_id
|
||||
|
||||
def upload_file(
|
||||
self,
|
||||
@@ -172,6 +194,9 @@ class GoogleDriveService:
|
||||
"""
|
||||
Upload a file to a specific Drive folder.
|
||||
|
||||
Retries on transient connection errors (BrokenPipeError, SSLError)
|
||||
that occur when Cloud Run containers sit idle between jobs.
|
||||
|
||||
Args:
|
||||
local_path: Local file path
|
||||
parent_id: Parent folder ID in Google Drive
|
||||
@@ -183,49 +208,57 @@ class GoogleDriveService:
|
||||
"""
|
||||
from googleapiclient.http import MediaFileUpload
|
||||
|
||||
file_size = os.path.getsize(local_path)
|
||||
logger.info(
|
||||
f"Uploading {local_path} ({file_size / 1024 / 1024:.1f} MB) "
|
||||
f"as '{filename}' to folder {parent_id}"
|
||||
)
|
||||
for attempt in Retrying(
|
||||
stop=stop_after_attempt(3),
|
||||
wait=wait_exponential(multiplier=1, min=2, max=8),
|
||||
retry=retry_if_exception_type(TRANSIENT_ERRORS),
|
||||
before_sleep=lambda retry_state: self._reset_service(),
|
||||
reraise=True,
|
||||
):
|
||||
with attempt:
|
||||
file_size = os.path.getsize(local_path)
|
||||
logger.info(
|
||||
f"Uploading {local_path} ({file_size / 1024 / 1024:.1f} MB) "
|
||||
f"as '{filename}' to folder {parent_id}"
|
||||
)
|
||||
|
||||
# Determine MIME type
|
||||
ext = os.path.splitext(local_path)[1].lower()
|
||||
mime_types = {
|
||||
".mp4": "video/mp4",
|
||||
".mkv": "video/x-matroska",
|
||||
".zip": "application/zip",
|
||||
".flac": "audio/flac",
|
||||
".mp3": "audio/mpeg",
|
||||
".wav": "audio/wav",
|
||||
".png": "image/png",
|
||||
".jpg": "image/jpeg",
|
||||
".jpeg": "image/jpeg",
|
||||
}
|
||||
mime_type = mime_types.get(ext, "application/octet-stream")
|
||||
# Determine MIME type
|
||||
ext = os.path.splitext(local_path)[1].lower()
|
||||
mime_types = {
|
||||
".mp4": "video/mp4",
|
||||
".mkv": "video/x-matroska",
|
||||
".zip": "application/zip",
|
||||
".flac": "audio/flac",
|
||||
".mp3": "audio/mpeg",
|
||||
".wav": "audio/wav",
|
||||
".png": "image/png",
|
||||
".jpg": "image/jpeg",
|
||||
".jpeg": "image/jpeg",
|
||||
}
|
||||
mime_type = mime_types.get(ext, "application/octet-stream")
|
||||
|
||||
# Check for existing file with same name
|
||||
if replace_existing:
|
||||
# Escape single quotes in filename for Google Drive API query syntax
|
||||
escaped_filename = filename.replace("'", "\\'")
|
||||
query = (
|
||||
f"name='{escaped_filename}' and '{parent_id}' in parents and trashed=false"
|
||||
)
|
||||
results = self.service.files().list(q=query, fields="files(id)").execute()
|
||||
for existing_file in results.get("files", []):
|
||||
logger.info(f"Deleting existing file: {existing_file['id']}")
|
||||
self.service.files().delete(fileId=existing_file["id"]).execute()
|
||||
# Check for existing file with same name
|
||||
if replace_existing:
|
||||
# Escape single quotes in filename for Google Drive API query syntax
|
||||
escaped_filename = filename.replace("'", "\\'")
|
||||
query = (
|
||||
f"name='{escaped_filename}' and '{parent_id}' in parents and trashed=false"
|
||||
)
|
||||
results = self.service.files().list(q=query, fields="files(id)").execute()
|
||||
for existing_file in results.get("files", []):
|
||||
logger.info(f"Deleting existing file: {existing_file['id']}")
|
||||
self.service.files().delete(fileId=existing_file["id"]).execute()
|
||||
|
||||
# Upload file
|
||||
metadata = {"name": filename, "parents": [parent_id]}
|
||||
media = MediaFileUpload(local_path, mimetype=mime_type, resumable=True)
|
||||
# Upload file
|
||||
metadata = {"name": filename, "parents": [parent_id]}
|
||||
media = MediaFileUpload(local_path, mimetype=mime_type, resumable=True)
|
||||
|
||||
file_result = (
|
||||
self.service.files().create(body=metadata, media_body=media, fields="id").execute()
|
||||
)
|
||||
file_id = file_result["id"]
|
||||
logger.info(f"Successfully uploaded '{filename}': {file_id}")
|
||||
return file_id
|
||||
file_result = (
|
||||
self.service.files().create(body=metadata, media_body=media, fields="id").execute()
|
||||
)
|
||||
file_id = file_result["id"]
|
||||
logger.info(f"Successfully uploaded '{filename}': {file_id}")
|
||||
return file_id
|
||||
|
||||
def upload_to_public_share(
|
||||
self,
|
||||
@@ -308,6 +341,9 @@ class GoogleDriveService:
|
||||
"""
|
||||
Delete a file from Google Drive.
|
||||
|
||||
Retries on transient connection errors (BrokenPipeError, SSLError)
|
||||
that occur when Cloud Run containers sit idle between jobs.
|
||||
|
||||
Args:
|
||||
file_id: Google Drive file ID to delete
|
||||
|
||||
@@ -317,9 +353,17 @@ class GoogleDriveService:
|
||||
logger.info(f"Deleting Google Drive file: {file_id}")
|
||||
|
||||
try:
|
||||
self.service.files().delete(fileId=file_id).execute()
|
||||
logger.info(f"Successfully deleted file: {file_id}")
|
||||
return True
|
||||
for attempt in Retrying(
|
||||
stop=stop_after_attempt(3),
|
||||
wait=wait_exponential(multiplier=1, min=2, max=8),
|
||||
retry=retry_if_exception_type(TRANSIENT_ERRORS),
|
||||
before_sleep=lambda retry_state: self._reset_service(),
|
||||
reraise=True,
|
||||
):
|
||||
with attempt:
|
||||
self.service.files().delete(fileId=file_id).execute()
|
||||
logger.info(f"Successfully deleted file: {file_id}")
|
||||
return True
|
||||
except Exception as e:
|
||||
# Check if it's a 404 (already deleted)
|
||||
if hasattr(e, 'resp') and e.resp.status == 404:
|
||||
@@ -343,6 +387,85 @@ class GoogleDriveService:
|
||||
results[file_id] = self.delete_file(file_id)
|
||||
return results
|
||||
|
||||
def find_files_by_brand_code(self, root_folder_id: str, brand_code: str) -> list[str]:
|
||||
"""
|
||||
Search for files in public share subfolders matching a brand code prefix.
|
||||
|
||||
Used as a fallback when gdrive_files is not tracked in state_data (e.g., old
|
||||
jobs uploaded before file ID tracking was implemented, or jobs where GDrive
|
||||
upload succeeded but file IDs were not saved due to a bug).
|
||||
|
||||
Searches CDG/, MP4/, MP4-720p/ subfolders for files whose name starts with
|
||||
'{brand_code} - '.
|
||||
|
||||
Args:
|
||||
root_folder_id: Root Google Drive folder ID for the public share
|
||||
brand_code: Brand code prefix to search for (e.g., "NOMAD-1271")
|
||||
|
||||
Returns:
|
||||
List of file IDs found matching the brand code
|
||||
"""
|
||||
found_ids = []
|
||||
subfolders = ["CDG", "MP4", "MP4-720p"]
|
||||
search_errors: list[Exception] = []
|
||||
|
||||
for subfolder in subfolders:
|
||||
try:
|
||||
# Find the subfolder
|
||||
escaped_subfolder = subfolder.replace("'", "\\'")
|
||||
query = (
|
||||
f"name='{escaped_subfolder}' and '{root_folder_id}' in parents "
|
||||
f"and mimeType='application/vnd.google-apps.folder' and trashed=false"
|
||||
)
|
||||
results = self.service.files().list(
|
||||
q=query, fields="files(id, name)", supportsAllDrives=True,
|
||||
includeItemsFromAllDrives=True
|
||||
).execute()
|
||||
|
||||
if not results.get("files"):
|
||||
logger.debug(f"Subfolder '{subfolder}' not found in {root_folder_id}")
|
||||
continue
|
||||
|
||||
subfolder_id = results["files"][0]["id"]
|
||||
|
||||
# Search for files whose name starts with the brand code
|
||||
# Use contains for efficiency, then filter by exact prefix
|
||||
escaped_brand = brand_code.replace("'", "\\'")
|
||||
file_query = (
|
||||
f"name contains '{escaped_brand}' and '{subfolder_id}' in parents "
|
||||
f"and trashed=false"
|
||||
)
|
||||
file_results = self.service.files().list(
|
||||
q=file_query, fields="files(id, name)", supportsAllDrives=True,
|
||||
includeItemsFromAllDrives=True
|
||||
).execute()
|
||||
|
||||
for f in file_results.get("files", []):
|
||||
if f["name"].startswith(f"{brand_code} - "):
|
||||
found_ids.append(f["id"])
|
||||
logger.info(
|
||||
f"Found GDrive file to clean up in {subfolder}/: "
|
||||
f"{f['name']} ({f['id']})"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Error searching for brand_code '{brand_code}' in '{subfolder}': {e}",
|
||||
exc_info=True,
|
||||
)
|
||||
search_errors.append(e)
|
||||
|
||||
if search_errors:
|
||||
# Raise so the caller knows the search was incomplete.
|
||||
# This prevents the cleanup endpoint from treating an uncertain result
|
||||
# as "GDrive is clean" and incorrectly recycling the brand code.
|
||||
raise RuntimeError(
|
||||
f"GDrive brand_code search incomplete: "
|
||||
f"{len(search_errors)}/{len(subfolders)} subfolder(s) failed"
|
||||
)
|
||||
|
||||
return found_ids
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_gdrive_service: Optional[GoogleDriveService] = None
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
"""
|
||||
GDrive Validator Cloud Function client.
|
||||
|
||||
Lightweight HTTP client for invoking the gdrive-validator Cloud Function
|
||||
from the backend (e.g., after job completion). Uses OIDC authentication
|
||||
via the backend's service account.
|
||||
|
||||
Fire-and-forget pattern: never blocks or fails the caller.
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
|
||||
import requests
|
||||
import google.auth.transport.requests
|
||||
import google.oauth2.id_token
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Cloud Function URL - set via environment variable
|
||||
GDRIVE_VALIDATOR_URL = os.environ.get("GDRIVE_VALIDATOR_URL", "")
|
||||
|
||||
|
||||
def trigger_gdrive_validation() -> dict | None:
|
||||
"""
|
||||
Invoke the GDrive validator Cloud Function.
|
||||
|
||||
Uses the backend service account's OIDC token for authentication.
|
||||
|
||||
Returns:
|
||||
Parsed JSON response from the function, or None on error.
|
||||
"""
|
||||
if not GDRIVE_VALIDATOR_URL:
|
||||
logger.debug("GDRIVE_VALIDATOR_URL not set, skipping GDrive validation trigger")
|
||||
return None
|
||||
|
||||
try:
|
||||
# Get OIDC token for the Cloud Function URL
|
||||
auth_request = google.auth.transport.requests.Request()
|
||||
token = google.oauth2.id_token.fetch_id_token(auth_request, GDRIVE_VALIDATOR_URL)
|
||||
|
||||
response = requests.post(
|
||||
GDRIVE_VALIDATOR_URL,
|
||||
headers={"Authorization": f"Bearer {token}"},
|
||||
timeout=60,
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
||||
result = response.json()
|
||||
status = result.get("status", "unknown")
|
||||
|
||||
if status == "issues_found":
|
||||
logger.warning("GDrive validation found issues (notification sent)")
|
||||
elif status == "ok":
|
||||
logger.info("GDrive validation: all clear")
|
||||
else:
|
||||
logger.info(f"GDrive validation returned status: {status}")
|
||||
|
||||
return result
|
||||
|
||||
except Exception:
|
||||
logger.exception("Failed to trigger GDrive validation (non-fatal)")
|
||||
return None
|
||||
@@ -0,0 +1,228 @@
|
||||
"""
|
||||
IP geolocation service with Firestore caching.
|
||||
|
||||
Looks up IP addresses via ip-api.com (free, no key needed) and caches
|
||||
results in Firestore to avoid repeated lookups. Used by admin UI to
|
||||
enrich IP displays with country, city, ISP, and ASN info.
|
||||
"""
|
||||
import ipaddress
|
||||
import logging
|
||||
import time
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Optional, Dict, List
|
||||
|
||||
import httpx
|
||||
from google.cloud import firestore
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
IP_GEOLOCATION_COLLECTION = "ip_geolocation"
|
||||
CACHE_TTL_DAYS = 90 # Cache results for 90 days
|
||||
IP_API_URL = "http://ip-api.com/json"
|
||||
IP_API_BATCH_URL = "http://ip-api.com/batch"
|
||||
IP_API_FIELDS = "status,country,countryCode,regionName,city,isp,org,as,timezone,query"
|
||||
RATE_LIMIT_PER_MINUTE = 45
|
||||
|
||||
|
||||
def _is_private_ip(ip: str) -> bool:
|
||||
"""Check if an IP is private/reserved (not routable on the internet)."""
|
||||
try:
|
||||
addr = ipaddress.ip_address(ip)
|
||||
return addr.is_private or addr.is_reserved or addr.is_loopback or addr.is_link_local
|
||||
except ValueError:
|
||||
return True # Invalid IP
|
||||
|
||||
|
||||
def _parse_ip_api_response(data: dict) -> dict:
|
||||
"""Parse ip-api.com response into our standard format."""
|
||||
if data.get("status") != "success":
|
||||
return {
|
||||
"status": "fail",
|
||||
"ip": data.get("query", ""),
|
||||
"cached_at": datetime.utcnow().isoformat(),
|
||||
}
|
||||
|
||||
# Parse AS field: "AS14615 Comporium Inc" -> number + name
|
||||
as_field = data.get("as", "")
|
||||
as_number = ""
|
||||
as_name = ""
|
||||
if as_field:
|
||||
parts = as_field.split(" ", 1)
|
||||
as_number = parts[0] if parts else ""
|
||||
as_name = parts[1] if len(parts) > 1 else ""
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"ip": data.get("query", ""),
|
||||
"country": data.get("country", ""),
|
||||
"country_code": data.get("countryCode", ""),
|
||||
"region": data.get("regionName", ""),
|
||||
"city": data.get("city", ""),
|
||||
"isp": data.get("isp", ""),
|
||||
"org": data.get("org", ""),
|
||||
"as_number": as_number,
|
||||
"as_name": as_name,
|
||||
"timezone": data.get("timezone", ""),
|
||||
"cached_at": datetime.utcnow().isoformat(),
|
||||
}
|
||||
|
||||
|
||||
class IpGeolocationService:
|
||||
"""IP geolocation lookup with Firestore caching."""
|
||||
|
||||
def __init__(self):
|
||||
settings = get_settings()
|
||||
self.db = firestore.Client(project=settings.google_cloud_project)
|
||||
self._last_request_times: List[float] = []
|
||||
|
||||
def _check_rate_limit(self):
|
||||
"""Simple rate limiter — sleep if approaching 45 req/min."""
|
||||
now = time.time()
|
||||
# Remove entries older than 60 seconds
|
||||
self._last_request_times = [
|
||||
t for t in self._last_request_times if now - t < 60
|
||||
]
|
||||
if len(self._last_request_times) >= RATE_LIMIT_PER_MINUTE - 5:
|
||||
# Getting close to limit, wait
|
||||
sleep_time = 60 - (now - self._last_request_times[0])
|
||||
if sleep_time > 0:
|
||||
logger.info(f"IP geolocation rate limit: sleeping {sleep_time:.1f}s")
|
||||
time.sleep(sleep_time)
|
||||
self._last_request_times.append(time.time())
|
||||
|
||||
def _get_cached(self, ip: str) -> Optional[dict]:
|
||||
"""Get cached geolocation data for an IP."""
|
||||
try:
|
||||
# Firestore doc IDs can't contain '/' so IPv6 with :: is fine,
|
||||
# but we need to sanitize for doc ID
|
||||
doc_id = ip.replace("/", "_")
|
||||
doc = self.db.collection(IP_GEOLOCATION_COLLECTION).document(doc_id).get()
|
||||
if not doc.exists:
|
||||
return None
|
||||
|
||||
data = doc.to_dict()
|
||||
|
||||
# Check TTL
|
||||
cached_at = data.get("cached_at", "")
|
||||
if cached_at:
|
||||
try:
|
||||
cached_dt = datetime.fromisoformat(cached_at)
|
||||
if datetime.utcnow() - cached_dt > timedelta(days=CACHE_TTL_DAYS):
|
||||
return None # Expired
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
return data
|
||||
except Exception:
|
||||
logger.exception(f"Error reading IP geo cache for {ip}")
|
||||
return None
|
||||
|
||||
def _cache_result(self, ip: str, data: dict):
|
||||
"""Store geolocation result in Firestore cache."""
|
||||
try:
|
||||
doc_id = ip.replace("/", "_")
|
||||
self.db.collection(IP_GEOLOCATION_COLLECTION).document(doc_id).set(data)
|
||||
except Exception:
|
||||
logger.exception(f"Error caching IP geo for {ip}")
|
||||
|
||||
def lookup_ip(self, ip: str) -> dict:
|
||||
"""
|
||||
Look up geolocation for a single IP address.
|
||||
|
||||
Returns cached result if available, otherwise queries ip-api.com.
|
||||
"""
|
||||
if not ip or _is_private_ip(ip):
|
||||
return {"status": "private", "ip": ip or ""}
|
||||
|
||||
# Check cache
|
||||
cached = self._get_cached(ip)
|
||||
if cached:
|
||||
return cached
|
||||
|
||||
# Query ip-api.com
|
||||
try:
|
||||
self._check_rate_limit()
|
||||
with httpx.Client(timeout=10) as client:
|
||||
response = client.get(
|
||||
f"{IP_API_URL}/{ip}",
|
||||
params={"fields": IP_API_FIELDS},
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
result = _parse_ip_api_response(data)
|
||||
self._cache_result(ip, result)
|
||||
return result
|
||||
|
||||
except Exception:
|
||||
logger.exception(f"Error looking up IP {ip}")
|
||||
return {"status": "error", "ip": ip}
|
||||
|
||||
def lookup_ips_batch(self, ips: List[str]) -> Dict[str, dict]:
|
||||
"""
|
||||
Look up geolocation for multiple IPs.
|
||||
|
||||
Returns dict of ip -> geo data. Uses cache for known IPs,
|
||||
batches uncached lookups to ip-api.com.
|
||||
"""
|
||||
results = {}
|
||||
uncached = []
|
||||
|
||||
for ip in set(ips): # Deduplicate
|
||||
if not ip or _is_private_ip(ip):
|
||||
results[ip] = {"status": "private", "ip": ip or ""}
|
||||
continue
|
||||
|
||||
cached = self._get_cached(ip)
|
||||
if cached:
|
||||
results[ip] = cached
|
||||
else:
|
||||
uncached.append(ip)
|
||||
|
||||
if not uncached:
|
||||
return results
|
||||
|
||||
# ip-api.com batch endpoint accepts up to 100 IPs per request
|
||||
for i in range(0, len(uncached), 100):
|
||||
batch = uncached[i:i + 100]
|
||||
try:
|
||||
self._check_rate_limit()
|
||||
# Batch endpoint takes a JSON array of IPs
|
||||
payload = [
|
||||
{"query": ip, "fields": IP_API_FIELDS}
|
||||
for ip in batch
|
||||
]
|
||||
with httpx.Client(timeout=15) as client:
|
||||
response = client.post(IP_API_BATCH_URL, json=payload)
|
||||
response.raise_for_status()
|
||||
batch_results = response.json()
|
||||
|
||||
for data in batch_results:
|
||||
result = _parse_ip_api_response(data)
|
||||
ip = result.get("ip", data.get("query", ""))
|
||||
if ip:
|
||||
results[ip] = result
|
||||
self._cache_result(ip, result)
|
||||
|
||||
except Exception:
|
||||
logger.exception(f"Error in batch IP lookup ({len(batch)} IPs)")
|
||||
# Mark failed IPs
|
||||
for ip in batch:
|
||||
if ip not in results:
|
||||
results[ip] = {"status": "error", "ip": ip}
|
||||
|
||||
return results
|
||||
|
||||
|
||||
# Global instance
|
||||
_service = None
|
||||
|
||||
|
||||
def get_ip_geolocation_service() -> IpGeolocationService:
|
||||
"""Get the global IP geolocation service instance."""
|
||||
global _service
|
||||
if _service is None:
|
||||
_service = IpGeolocationService()
|
||||
return _service
|
||||
@@ -61,6 +61,56 @@ def get_effective_distribution_settings(
|
||||
)
|
||||
|
||||
|
||||
def get_effective_distribution_for_job(job) -> EffectiveDistributionSettings:
|
||||
"""
|
||||
Get effective distribution settings for a job, applying private overrides if is_private=True.
|
||||
|
||||
For private (non-published) jobs, this overrides:
|
||||
- dropbox_path → private Dropbox path (Tracks-NonPublished)
|
||||
- brand_prefix → private brand prefix (NOMADNP)
|
||||
- enable_youtube_upload → False (no YouTube for private)
|
||||
- gdrive_folder_id → None (no Google Drive for private)
|
||||
|
||||
For non-private jobs, returns the job's own distribution settings unchanged.
|
||||
|
||||
Args:
|
||||
job: Job object (or any object with distribution fields)
|
||||
|
||||
Returns:
|
||||
EffectiveDistributionSettings with private overrides applied if needed
|
||||
"""
|
||||
if getattr(job, 'is_private', False):
|
||||
# Tenant jobs: use the job's own distribution fields (set at creation from tenant config)
|
||||
# Non-tenant private jobs: override to NonPublished paths
|
||||
if getattr(job, 'tenant_id', None):
|
||||
return EffectiveDistributionSettings(
|
||||
dropbox_path=getattr(job, 'dropbox_path', None),
|
||||
brand_prefix=getattr(job, 'brand_prefix', None),
|
||||
enable_youtube_upload=False,
|
||||
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
|
||||
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
|
||||
youtube_description=None,
|
||||
)
|
||||
settings = get_settings()
|
||||
return EffectiveDistributionSettings(
|
||||
dropbox_path=settings.default_private_dropbox_path,
|
||||
brand_prefix=settings.default_private_brand_prefix,
|
||||
enable_youtube_upload=False,
|
||||
gdrive_folder_id=None,
|
||||
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
|
||||
youtube_description=None,
|
||||
)
|
||||
# Non-private: return job's own settings
|
||||
return EffectiveDistributionSettings(
|
||||
dropbox_path=getattr(job, 'dropbox_path', None),
|
||||
brand_prefix=getattr(job, 'brand_prefix', None),
|
||||
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
|
||||
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
|
||||
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
|
||||
youtube_description=getattr(job, 'youtube_description_template', None),
|
||||
)
|
||||
|
||||
|
||||
def resolve_cdg_txt_defaults(
|
||||
theme_id: Optional[str],
|
||||
enable_cdg: Optional[bool] = None,
|
||||
|
||||
@@ -0,0 +1,262 @@
|
||||
"""
|
||||
Job health and consistency checking service.
|
||||
|
||||
Provides functions to detect state inconsistencies in jobs that might indicate
|
||||
bugs in the state machine or trigger logic.
|
||||
|
||||
Created as part of state machine robustness improvements (2026-02-02).
|
||||
"""
|
||||
import logging
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from typing import List, Dict, Any, Optional
|
||||
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Jobs in these statuses should NOT have audio_complete flag set
|
||||
# (unless they're actively processing or beyond)
|
||||
STATUSES_BEFORE_AUDIO_START = {
|
||||
JobStatus.PENDING,
|
||||
JobStatus.SEARCHING_AUDIO,
|
||||
JobStatus.AWAITING_AUDIO_SELECTION,
|
||||
JobStatus.AWAITING_AUDIO_EDIT,
|
||||
JobStatus.IN_AUDIO_EDIT,
|
||||
JobStatus.AUDIO_EDIT_COMPLETE,
|
||||
}
|
||||
|
||||
# Jobs in these statuses should NOT have lyrics_complete flag set
|
||||
STATUSES_BEFORE_LYRICS_START = {
|
||||
JobStatus.PENDING,
|
||||
JobStatus.SEARCHING_AUDIO,
|
||||
JobStatus.AWAITING_AUDIO_SELECTION,
|
||||
JobStatus.AWAITING_AUDIO_EDIT,
|
||||
JobStatus.IN_AUDIO_EDIT,
|
||||
JobStatus.AUDIO_EDIT_COMPLETE,
|
||||
}
|
||||
|
||||
# Statuses where screens_progress.stage='complete' is valid
|
||||
STATUSES_AFTER_SCREENS_COMPLETE = {
|
||||
JobStatus.AWAITING_REVIEW,
|
||||
JobStatus.IN_REVIEW,
|
||||
JobStatus.REVIEW_COMPLETE,
|
||||
JobStatus.RENDERING_VIDEO,
|
||||
JobStatus.AWAITING_INSTRUMENTAL_SELECTION, # Legacy
|
||||
JobStatus.INSTRUMENTAL_SELECTED,
|
||||
JobStatus.GENERATING_VIDEO,
|
||||
JobStatus.ENCODING,
|
||||
JobStatus.PACKAGING,
|
||||
JobStatus.UPLOADING,
|
||||
JobStatus.NOTIFYING,
|
||||
JobStatus.COMPLETE,
|
||||
JobStatus.PREP_COMPLETE,
|
||||
}
|
||||
|
||||
|
||||
def check_job_consistency(job: Job) -> List[str]:
|
||||
"""
|
||||
Check for state inconsistencies in a job.
|
||||
|
||||
This function detects situations where state_data flags indicate
|
||||
progress that doesn't match the job status - which usually indicates
|
||||
a bug in the state machine or trigger logic.
|
||||
|
||||
Args:
|
||||
job: Job object to check
|
||||
|
||||
Returns:
|
||||
List of issue descriptions found. Empty list means no issues.
|
||||
"""
|
||||
issues = []
|
||||
state_data = job.state_data or {}
|
||||
|
||||
# Normalize status to JobStatus enum if it's a string
|
||||
status = job.status
|
||||
if isinstance(status, str):
|
||||
try:
|
||||
status = JobStatus(status)
|
||||
except ValueError:
|
||||
issues.append(f"invalid_status: '{job.status}' is not a valid JobStatus")
|
||||
return issues
|
||||
|
||||
# Check: audio_complete=True but status still in pre-audio states
|
||||
if state_data.get('audio_complete') and status in STATUSES_BEFORE_AUDIO_START:
|
||||
issues.append(
|
||||
f"audio_complete_status_mismatch: audio_complete=True but status={status.value}"
|
||||
)
|
||||
|
||||
# Check: lyrics_complete=True but status still in pre-lyrics states
|
||||
if state_data.get('lyrics_complete') and status in STATUSES_BEFORE_LYRICS_START:
|
||||
issues.append(
|
||||
f"lyrics_complete_status_mismatch: lyrics_complete=True but status={status.value}"
|
||||
)
|
||||
|
||||
# Check: screens_progress.stage='complete' but status not in post-screens states
|
||||
screens_progress = state_data.get('screens_progress', {})
|
||||
if isinstance(screens_progress, dict) and screens_progress.get('stage') == 'complete':
|
||||
if status not in STATUSES_AFTER_SCREENS_COMPLETE:
|
||||
issues.append(
|
||||
f"screens_complete_status_mismatch: screens_progress.stage=complete but status={status.value}"
|
||||
)
|
||||
|
||||
# Check: job has input_media_gcs_path but is stuck at pending
|
||||
# This was the exact bug that got jobs 06cfea29 and 984da08b stuck
|
||||
if job.input_media_gcs_path and status == JobStatus.PENDING:
|
||||
# This could be legitimate if job was just created and workers haven't started
|
||||
# But if it's been more than a few minutes, it's likely stuck
|
||||
# We flag it as a warning, not an error
|
||||
issues.append(
|
||||
f"pending_with_input_media: has input_media_gcs_path but status=pending (may be stuck)"
|
||||
)
|
||||
|
||||
# Check: job stuck in encoding status beyond 50 minutes
|
||||
# Encoding timeout is 1 hour; if updated_at hasn't advanced in 50 min,
|
||||
# the poller likely died (e.g., Cloud Run deployment killed the instance)
|
||||
if status == JobStatus.ENCODING and job.updated_at:
|
||||
now = datetime.now(timezone.utc)
|
||||
updated_at = job.updated_at
|
||||
if updated_at.tzinfo is None:
|
||||
updated_at = updated_at.replace(tzinfo=timezone.utc)
|
||||
encoding_age = now - updated_at
|
||||
if encoding_age > timedelta(minutes=50):
|
||||
minutes = int(encoding_age.total_seconds() / 60)
|
||||
issues.append(
|
||||
f"encoding_stuck: status=encoding for {minutes} min without update (updated_at={updated_at.isoformat()})"
|
||||
)
|
||||
|
||||
# Check: job stuck in downloading_audio status beyond 10 minutes
|
||||
# Audio downloads typically complete in 30s-5min. If stuck longer,
|
||||
# the Cloud Run Job likely failed silently.
|
||||
if status == JobStatus.DOWNLOADING_AUDIO and job.updated_at:
|
||||
now = datetime.now(timezone.utc)
|
||||
updated_at = job.updated_at
|
||||
if updated_at.tzinfo is None:
|
||||
updated_at = updated_at.replace(tzinfo=timezone.utc)
|
||||
download_age = now - updated_at
|
||||
if download_age > timedelta(minutes=10):
|
||||
minutes = int(download_age.total_seconds() / 60)
|
||||
issues.append(
|
||||
f"downloading_audio_stuck: status=downloading_audio for {minutes} min without update (updated_at={updated_at.isoformat()})"
|
||||
)
|
||||
|
||||
return issues
|
||||
|
||||
|
||||
def check_job_consistency_detailed(job: Job) -> Dict[str, Any]:
|
||||
"""
|
||||
Check job consistency with detailed information for debugging.
|
||||
|
||||
Returns a dictionary with:
|
||||
- job_id: The job ID
|
||||
- status: Current job status
|
||||
- issues: List of issue descriptions
|
||||
- state_data_summary: Summary of relevant state_data fields
|
||||
- is_healthy: Boolean indicating if no issues were found
|
||||
|
||||
Args:
|
||||
job: Job object to check
|
||||
|
||||
Returns:
|
||||
Dictionary with detailed consistency information
|
||||
"""
|
||||
issues = check_job_consistency(job)
|
||||
state_data = job.state_data or {}
|
||||
|
||||
return {
|
||||
"job_id": job.job_id,
|
||||
"status": job.status.value if hasattr(job.status, 'value') else str(job.status),
|
||||
"issues": issues,
|
||||
"state_data_summary": {
|
||||
"audio_complete": state_data.get('audio_complete', False),
|
||||
"lyrics_complete": state_data.get('lyrics_complete', False),
|
||||
"screens_progress_stage": state_data.get('screens_progress', {}).get('stage'),
|
||||
"has_input_media_gcs_path": bool(job.input_media_gcs_path),
|
||||
},
|
||||
"is_healthy": len(issues) == 0,
|
||||
}
|
||||
|
||||
|
||||
def get_worker_valid_statuses() -> Dict[str, List[JobStatus]]:
|
||||
"""
|
||||
Get mapping of worker names to their valid job statuses.
|
||||
|
||||
This is used by workers to validate they should run for a given job.
|
||||
|
||||
Returns:
|
||||
Dictionary mapping worker name to list of valid JobStatus values
|
||||
"""
|
||||
return {
|
||||
"audio_worker": [
|
||||
JobStatus.DOWNLOADING,
|
||||
JobStatus.DOWNLOADING_AUDIO,
|
||||
JobStatus.AUDIO_EDIT_COMPLETE,
|
||||
JobStatus.SEPARATING_STAGE1,
|
||||
JobStatus.SEPARATING_STAGE2,
|
||||
],
|
||||
"lyrics_worker": [
|
||||
JobStatus.DOWNLOADING,
|
||||
JobStatus.AUDIO_EDIT_COMPLETE,
|
||||
JobStatus.TRANSCRIBING,
|
||||
JobStatus.CORRECTING,
|
||||
],
|
||||
"screens_worker": [
|
||||
# Screens worker can be triggered when both audio+lyrics complete
|
||||
# which can happen in any of these states
|
||||
JobStatus.DOWNLOADING,
|
||||
JobStatus.AUDIO_COMPLETE,
|
||||
JobStatus.LYRICS_COMPLETE,
|
||||
# Also valid if re-running after a failure
|
||||
JobStatus.GENERATING_SCREENS,
|
||||
],
|
||||
"video_worker": [
|
||||
JobStatus.INSTRUMENTAL_SELECTED,
|
||||
JobStatus.REVIEW_COMPLETE,
|
||||
# Note: RENDERING_VIDEO and GENERATING_VIDEO are intentionally excluded.
|
||||
# If the job is already in those states, another worker is actively processing it.
|
||||
# Including them caused a race condition where Cloud Run Job retries + manual retries
|
||||
# could launch concurrent workers that corrupt each other's state transitions.
|
||||
],
|
||||
"render_video_worker": [
|
||||
JobStatus.REVIEW_COMPLETE,
|
||||
JobStatus.RENDERING_VIDEO,
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def validate_worker_can_run(worker_name: str, job: Job) -> Optional[str]:
|
||||
"""
|
||||
Validate that a worker should run for the given job.
|
||||
|
||||
Args:
|
||||
worker_name: Name of the worker (e.g., "screens_worker")
|
||||
job: Job object to validate
|
||||
|
||||
Returns:
|
||||
None if valid, or an error message string if invalid
|
||||
"""
|
||||
valid_statuses_map = get_worker_valid_statuses()
|
||||
|
||||
if worker_name not in valid_statuses_map:
|
||||
# Unknown worker, can't validate
|
||||
return None
|
||||
|
||||
valid_statuses = valid_statuses_map[worker_name]
|
||||
|
||||
# Normalize status
|
||||
status = job.status
|
||||
if isinstance(status, str):
|
||||
try:
|
||||
status = JobStatus(status)
|
||||
except ValueError:
|
||||
return f"Invalid job status: {job.status}"
|
||||
|
||||
if status not in valid_statuses:
|
||||
valid_names = [s.value for s in valid_statuses]
|
||||
return (
|
||||
f"{worker_name} called but job status is {status.value}, "
|
||||
f"expected one of {valid_names}. This may indicate a bug in the trigger logic."
|
||||
)
|
||||
|
||||
return None
|
||||
+348
-98
@@ -14,7 +14,7 @@ from datetime import datetime
|
||||
from typing import Optional, Dict, Any, List
|
||||
|
||||
from backend.config import settings
|
||||
from backend.exceptions import RateLimitExceededError
|
||||
from backend.exceptions import InvalidStateTransitionError, InsufficientCreditsError
|
||||
from backend.models.job import Job, JobStatus, JobCreate, STATE_TRANSITIONS
|
||||
from backend.models.worker_log import WorkerLogEntry
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
@@ -51,33 +51,21 @@ class JobManager:
|
||||
|
||||
Args:
|
||||
job_create: Job creation parameters
|
||||
is_admin: Whether the requesting user is an admin (bypasses rate limits)
|
||||
is_admin: Whether the requesting user is an admin (bypasses credit checks)
|
||||
|
||||
Raises:
|
||||
ValueError: If theme_id is not provided (all jobs require a theme)
|
||||
RateLimitExceededError: If user has exceeded their daily job limit
|
||||
"""
|
||||
# Check rate limit FIRST (before any other validation)
|
||||
# This prevents wasted work if user is rate limited
|
||||
if job_create.user_email:
|
||||
from backend.services.rate_limit_service import get_rate_limit_service
|
||||
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
allowed, remaining, message = rate_limit_service.check_user_job_limit(
|
||||
user_email=job_create.user_email,
|
||||
is_admin=is_admin
|
||||
)
|
||||
if not allowed:
|
||||
from backend.services.rate_limit_service import _seconds_until_midnight_utc
|
||||
|
||||
# Get actual current count - remaining is clamped to 0 which loses info
|
||||
current_count = rate_limit_service.get_user_job_count_today(job_create.user_email)
|
||||
raise RateLimitExceededError(
|
||||
message=message,
|
||||
limit_type="jobs_per_day",
|
||||
remaining_seconds=_seconds_until_midnight_utc(),
|
||||
current_count=current_count,
|
||||
limit_value=settings.rate_limit_jobs_per_day
|
||||
# Check credits (skip for admins)
|
||||
if job_create.user_email and not is_admin:
|
||||
from backend.services.user_service import get_user_service
|
||||
user_service = get_user_service()
|
||||
if not user_service.has_credits(job_create.user_email):
|
||||
credits_available = user_service.check_credits(job_create.user_email)
|
||||
raise InsufficientCreditsError(
|
||||
message="You're out of credits. Buy more to continue creating karaoke videos.",
|
||||
credits_available=credits_available,
|
||||
credits_required=1,
|
||||
)
|
||||
|
||||
# Enforce theme requirement - all jobs must have a theme
|
||||
@@ -130,20 +118,38 @@ class JobManager:
|
||||
made_for_you=job_create.made_for_you,
|
||||
customer_email=job_create.customer_email,
|
||||
customer_notes=job_create.customer_notes,
|
||||
# Private (non-published) track mode
|
||||
is_private=job_create.is_private,
|
||||
# Capture creation-time decision params for observability
|
||||
creation_params={
|
||||
"is_private": job_create.is_private,
|
||||
"is_admin": is_admin,
|
||||
"created_from": job_create.request_metadata.get("created_from", "unknown"),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
self.firestore.create_job(job)
|
||||
logger.info(f"Created new job {job_id} with status PENDING")
|
||||
|
||||
# Record job creation for rate limiting (after successful persistence)
|
||||
if job_create.user_email:
|
||||
try:
|
||||
from backend.services.rate_limit_service import get_rate_limit_service
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
rate_limit_service.record_job_creation(job_create.user_email, job_id)
|
||||
except Exception as e:
|
||||
# Don't fail job creation if rate limit recording fails
|
||||
logger.warning(f"Failed to record job creation for rate limiting: {e}")
|
||||
# Deduct credit atomically (after job is persisted so we have job_id for transaction record)
|
||||
if job_create.user_email and not is_admin:
|
||||
from backend.services.user_service import get_user_service
|
||||
user_service = get_user_service()
|
||||
success, _remaining, deduct_msg = user_service.deduct_credit(
|
||||
job_create.user_email, job_id, reason="job_creation"
|
||||
)
|
||||
if not success:
|
||||
# Deduction failed (e.g., race condition) - delete the job and raise error
|
||||
logger.error(f"Credit deduction failed for job {job_id}: {deduct_msg}")
|
||||
try:
|
||||
self.firestore.delete_job(job_id)
|
||||
except Exception:
|
||||
logger.exception(f"Failed to delete job {job_id} after credit deduction failure")
|
||||
raise InsufficientCreditsError(
|
||||
message="Failed to deduct credit. Please try again.",
|
||||
credits_available=0,
|
||||
credits_required=1,
|
||||
)
|
||||
|
||||
return job
|
||||
|
||||
@@ -210,6 +216,31 @@ class JobManager:
|
||||
limit=limit
|
||||
)
|
||||
|
||||
def list_jobs_summary(
|
||||
self,
|
||||
status: Optional[JobStatus] = None,
|
||||
exclude_statuses: Optional[List[str]] = None,
|
||||
environment: Optional[str] = None,
|
||||
client_id: Optional[str] = None,
|
||||
created_after: Optional[datetime] = None,
|
||||
created_before: Optional[datetime] = None,
|
||||
user_email: Optional[str] = None,
|
||||
tenant_id: Optional[str] = None,
|
||||
limit: int = 100,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""List jobs with field projection (summary mode)."""
|
||||
return self.firestore.list_jobs_summary(
|
||||
status=status,
|
||||
exclude_statuses=exclude_statuses,
|
||||
environment=environment,
|
||||
client_id=client_id,
|
||||
created_after=created_after,
|
||||
created_before=created_before,
|
||||
user_email=user_email,
|
||||
tenant_id=tenant_id,
|
||||
limit=limit,
|
||||
)
|
||||
|
||||
def delete_jobs_by_filter(
|
||||
self,
|
||||
environment: Optional[str] = None,
|
||||
@@ -296,14 +327,20 @@ class JobManager:
|
||||
|
||||
def delete_job(self, job_id: str, delete_files: bool = True) -> None:
|
||||
"""Delete a job, its files, and its logs subcollection."""
|
||||
if delete_files:
|
||||
job = self.get_job(job_id)
|
||||
if job and job.output_files:
|
||||
for gcs_path in job.output_files.values():
|
||||
try:
|
||||
self.storage.delete_file(gcs_path)
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting file {gcs_path}: {e}")
|
||||
job = self.get_job(job_id)
|
||||
|
||||
# Refund credit if job never completed (and wasn't already refunded via fail/cancel)
|
||||
if job:
|
||||
no_refund_statuses = [JobStatus.COMPLETE, JobStatus.PREP_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED]
|
||||
if job.status not in no_refund_statuses:
|
||||
self._refund_credit_for_job(job_id, job, reason="job_deleted")
|
||||
|
||||
if delete_files and job and job.output_files:
|
||||
for gcs_path in job.output_files.values():
|
||||
try:
|
||||
self.storage.delete_file(gcs_path)
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting file {gcs_path}: {e}")
|
||||
|
||||
# Delete logs subcollection first (must be done before deleting parent doc)
|
||||
try:
|
||||
@@ -313,82 +350,126 @@ class JobManager:
|
||||
except Exception as e:
|
||||
logger.warning(f"Error deleting logs subcollection for job {job_id}: {e}")
|
||||
|
||||
# Delete audio edit sessions subcollection
|
||||
try:
|
||||
deleted_sessions = self.firestore.delete_audio_edit_sessions_subcollection(job_id)
|
||||
if deleted_sessions > 0:
|
||||
logger.info(f"Deleted {deleted_sessions} audio edit sessions for job {job_id}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Error deleting audio edit sessions for job {job_id}: {e}")
|
||||
|
||||
self.firestore.delete_job(job_id)
|
||||
logger.info(f"Deleted job {job_id}")
|
||||
|
||||
def validate_state_transition(self, job_id: str, new_status: JobStatus) -> bool:
|
||||
def validate_state_transition(
|
||||
self,
|
||||
job_id: str,
|
||||
new_status: JobStatus,
|
||||
raise_on_invalid: bool = True
|
||||
) -> bool:
|
||||
"""
|
||||
Validate that a state transition is legal.
|
||||
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
new_status: Target state
|
||||
raise_on_invalid: If True, raise InvalidStateTransitionError on invalid
|
||||
transitions. If False, return False (legacy behavior).
|
||||
|
||||
Returns:
|
||||
True if transition is valid, False otherwise
|
||||
True if transition is valid
|
||||
|
||||
Raises:
|
||||
InvalidStateTransitionError: If transition is invalid and raise_on_invalid=True
|
||||
ValueError: If job not found and raise_on_invalid=True
|
||||
"""
|
||||
job = self.get_job(job_id)
|
||||
if not job:
|
||||
logger.error(f"Job {job_id} not found")
|
||||
error_msg = f"Job {job_id} not found"
|
||||
logger.error(error_msg)
|
||||
if raise_on_invalid:
|
||||
raise ValueError(error_msg)
|
||||
return False
|
||||
|
||||
|
||||
current_status = job.status
|
||||
valid_transitions = STATE_TRANSITIONS.get(current_status, [])
|
||||
|
||||
|
||||
if new_status not in valid_transitions:
|
||||
logger.error(
|
||||
error_msg = (
|
||||
f"Invalid state transition for job {job_id}: "
|
||||
f"{current_status} -> {new_status}. "
|
||||
f"Valid transitions: {valid_transitions}"
|
||||
f"Valid transitions: {[s.value for s in valid_transitions]}"
|
||||
)
|
||||
logger.error(error_msg)
|
||||
|
||||
if raise_on_invalid:
|
||||
raise InvalidStateTransitionError(
|
||||
message=error_msg,
|
||||
job_id=job_id,
|
||||
from_status=current_status,
|
||||
to_status=new_status.value if hasattr(new_status, 'value') else str(new_status),
|
||||
valid_transitions=[s.value for s in valid_transitions]
|
||||
)
|
||||
return False
|
||||
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def transition_to_state(
|
||||
self,
|
||||
job_id: str,
|
||||
new_status: JobStatus,
|
||||
progress: Optional[int] = None,
|
||||
message: Optional[str] = None,
|
||||
state_data_updates: Optional[Dict[str, Any]] = None
|
||||
state_data_updates: Optional[Dict[str, Any]] = None,
|
||||
raise_on_invalid: bool = True,
|
||||
timeline_metadata: Optional[Dict[str, Any]] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Transition job to new state with validation.
|
||||
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
new_status: Target state
|
||||
progress: Progress percentage (0-100)
|
||||
message: Timeline message
|
||||
state_data_updates: Updates to state_data field
|
||||
|
||||
raise_on_invalid: If True (default), raise InvalidStateTransitionError
|
||||
on invalid transitions. If False, return False silently.
|
||||
timeline_metadata: Optional structured metadata to attach to the timeline event
|
||||
|
||||
Returns:
|
||||
True if transition succeeded, False otherwise
|
||||
True if transition succeeded, False if failed (only when raise_on_invalid=False)
|
||||
|
||||
Raises:
|
||||
InvalidStateTransitionError: If transition is invalid and raise_on_invalid=True
|
||||
"""
|
||||
if not self.validate_state_transition(job_id, new_status):
|
||||
if not self.validate_state_transition(job_id, new_status, raise_on_invalid=raise_on_invalid):
|
||||
return False
|
||||
|
||||
|
||||
updates = {
|
||||
'status': new_status,
|
||||
'updated_at': datetime.utcnow()
|
||||
}
|
||||
|
||||
|
||||
if progress is not None:
|
||||
updates['progress'] = progress
|
||||
|
||||
# Generate review token when entering AWAITING_REVIEW state
|
||||
|
||||
# Generate review token when entering blocking states that need user action
|
||||
# Tokens don't expire - they're job-scoped so low risk, and natural expiry happens when job completes
|
||||
if new_status == JobStatus.AWAITING_REVIEW:
|
||||
if new_status in (JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_AUDIO_EDIT):
|
||||
from backend.api.dependencies import generate_review_token
|
||||
review_token = generate_review_token()
|
||||
updates['review_token'] = review_token
|
||||
updates['review_token_expires_at'] = None # No expiry - token is job-scoped
|
||||
logger.info(f"Generated review token for job {job_id} (no expiry)")
|
||||
|
||||
|
||||
# If we have state_data_updates, merge them with existing state_data
|
||||
merged_state_data = None
|
||||
if state_data_updates:
|
||||
job = self.get_job(job_id)
|
||||
if job:
|
||||
merged_state_data = {**job.state_data, **state_data_updates}
|
||||
|
||||
|
||||
# Update job status (includes timeline event), passing state_data if present
|
||||
if merged_state_data is not None:
|
||||
self.update_job_status(
|
||||
@@ -396,6 +477,7 @@ class JobManager:
|
||||
status=new_status,
|
||||
progress=progress,
|
||||
message=message,
|
||||
timeline_metadata=timeline_metadata,
|
||||
state_data=merged_state_data
|
||||
)
|
||||
else:
|
||||
@@ -403,7 +485,8 @@ class JobManager:
|
||||
job_id=job_id,
|
||||
status=new_status,
|
||||
progress=progress,
|
||||
message=message
|
||||
message=message,
|
||||
timeline_metadata=timeline_metadata,
|
||||
)
|
||||
|
||||
# Apply review token update separately if generated
|
||||
@@ -415,6 +498,17 @@ class JobManager:
|
||||
|
||||
logger.info(f"Job {job_id} transitioned to {new_status}")
|
||||
|
||||
# Increment user's completed jobs counter on terminal success states
|
||||
if new_status in (JobStatus.COMPLETE, JobStatus.PREP_COMPLETE):
|
||||
try:
|
||||
job = self.get_job(job_id)
|
||||
if job and job.user_email:
|
||||
from backend.services.user_service import get_user_service
|
||||
user_service = get_user_service()
|
||||
user_service.increment_jobs_completed(job.user_email)
|
||||
except Exception:
|
||||
logger.exception(f"Failed to increment jobs_completed for job {job_id}")
|
||||
|
||||
# Trigger notifications asynchronously (fire-and-forget)
|
||||
self._trigger_state_notifications(job_id, new_status)
|
||||
|
||||
@@ -445,10 +539,15 @@ class JobManager:
|
||||
self._send_push_notification(job, "complete")
|
||||
|
||||
# Idle reminder scheduling for blocking states
|
||||
elif new_status in [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION]:
|
||||
elif new_status in [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.AWAITING_AUDIO_EDIT]:
|
||||
self._schedule_idle_reminder(job, new_status)
|
||||
# Send push notification for blocking states
|
||||
action_type = "lyrics" if new_status == JobStatus.AWAITING_REVIEW else "instrumental"
|
||||
if new_status == JobStatus.AWAITING_AUDIO_EDIT:
|
||||
action_type = "audio_edit"
|
||||
elif new_status == JobStatus.AWAITING_REVIEW:
|
||||
action_type = "lyrics"
|
||||
else:
|
||||
action_type = "instrumental"
|
||||
self._send_push_notification(job, action_type)
|
||||
|
||||
except Exception as e:
|
||||
@@ -475,6 +574,7 @@ class JobManager:
|
||||
youtube_url = state_data.get('youtube_url')
|
||||
dropbox_url = state_data.get('dropbox_link')
|
||||
brand_code = state_data.get('brand_code')
|
||||
youtube_queued = state_data.get('youtube_upload_queued', False)
|
||||
|
||||
# For made-for-you orders, send to customer and transfer ownership
|
||||
recipient_email = job.user_email
|
||||
@@ -498,6 +598,8 @@ class JobManager:
|
||||
youtube_url=youtube_url,
|
||||
dropbox_url=dropbox_url,
|
||||
brand_code=brand_code,
|
||||
is_private=getattr(job, 'is_private', False),
|
||||
youtube_queued=youtube_queued,
|
||||
)
|
||||
|
||||
# Try to get existing event loop, create new one if none exists
|
||||
@@ -532,7 +634,12 @@ class JobManager:
|
||||
# Record when we entered blocking state (for idle detection)
|
||||
blocking_entered_at = datetime.utcnow().isoformat()
|
||||
|
||||
action_type = "lyrics" if new_status == JobStatus.AWAITING_REVIEW else "instrumental"
|
||||
if new_status == JobStatus.AWAITING_AUDIO_EDIT:
|
||||
action_type = "audio_edit"
|
||||
elif new_status == JobStatus.AWAITING_REVIEW:
|
||||
action_type = "lyrics"
|
||||
else:
|
||||
action_type = "instrumental"
|
||||
|
||||
# Update state_data with blocking state info (handle None state_data)
|
||||
existing_state_data = job.state_data or {}
|
||||
@@ -599,6 +706,7 @@ class JobManager:
|
||||
"user_email": job.user_email,
|
||||
"artist": job.artist,
|
||||
"title": job.title,
|
||||
"tenant_id": job.tenant_id or None,
|
||||
}
|
||||
|
||||
async def send_notification():
|
||||
@@ -640,6 +748,31 @@ class JobManager:
|
||||
self.update_job(job_id, {'state_data': state_data})
|
||||
logger.debug(f"Job {job_id} state_data updated: {key} = {value}")
|
||||
|
||||
def update_processing_metadata(self, job_id: str, section: str, data) -> None:
|
||||
"""
|
||||
Write data into processing_metadata[section] using Firestore dot-notation.
|
||||
|
||||
Each worker writes its own section once. This uses Firestore's nested
|
||||
field update to avoid overwriting other workers' sections.
|
||||
|
||||
The section can use dot-notation for deeper nesting, e.g.
|
||||
'timing.lyrics_worker_seconds' writes to processing_metadata.timing.lyrics_worker_seconds
|
||||
without overwriting other timing entries.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
section: Metadata section path (e.g., 'transcription', 'timing.audio_worker_seconds')
|
||||
data: Value to store (dict for sections, scalar for individual fields)
|
||||
"""
|
||||
try:
|
||||
self.firestore.update_job(job_id, {
|
||||
f"processing_metadata.{section}": data,
|
||||
})
|
||||
logger.debug(f"Job {job_id} processing_metadata.{section} updated")
|
||||
except Exception as e:
|
||||
# Log but don't raise - metadata storage should never fail a job
|
||||
logger.warning(f"Job {job_id}: Failed to update processing_metadata.{section}: {e}")
|
||||
|
||||
def delete_state_data_key(self, job_id: str, key: str) -> bool:
|
||||
"""
|
||||
Delete a specific key from the job's state_data field.
|
||||
@@ -694,6 +827,46 @@ class JobManager:
|
||||
logger.error(f"Job {job_id}: Failed to delete state_data keys {keys}: {e}")
|
||||
return []
|
||||
|
||||
def _refund_credit_for_job(self, job_id: str, job: Job, reason: str) -> bool:
|
||||
"""
|
||||
Refund one credit for a job if eligible.
|
||||
|
||||
Skips refund if: already refunded, no user email, or admin-owned job.
|
||||
Never raises — refund failure must not block the calling operation.
|
||||
|
||||
Returns:
|
||||
True if credit was refunded, False otherwise.
|
||||
"""
|
||||
try:
|
||||
if job.credit_refunded:
|
||||
logger.info(f"Credit already refunded for job {job_id}, skipping")
|
||||
return False
|
||||
|
||||
if not job.user_email:
|
||||
return False
|
||||
|
||||
from backend.services.auth_service import is_admin_email
|
||||
from backend.services.user_service import get_user_service
|
||||
|
||||
if is_admin_email(job.user_email):
|
||||
return False
|
||||
|
||||
user_service = get_user_service()
|
||||
refund_ok, new_balance, refund_msg = user_service.refund_credit(
|
||||
job.user_email, job_id, reason=reason
|
||||
)
|
||||
if refund_ok:
|
||||
# Mark job as refunded to prevent double-refund
|
||||
self.update_job(job_id, {'credit_refunded': True})
|
||||
logger.info(f"Refunded credit for {reason} job {job_id} to {_mask_email(job.user_email)} (balance: {new_balance})")
|
||||
return True
|
||||
else:
|
||||
logger.warning(f"Credit refund failed for job {job_id}: {refund_msg}")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.warning(f"Error refunding credit for job {job_id}: {e}")
|
||||
return False
|
||||
|
||||
def fail_job(self, job_id: str, error_message: str, error_details: Optional[Dict[str, Any]] = None) -> bool:
|
||||
"""
|
||||
Mark a job as failed with error information.
|
||||
@@ -721,11 +894,17 @@ class JobManager:
|
||||
)
|
||||
|
||||
logger.error(f"Job {job_id} failed: {error_message}")
|
||||
|
||||
# Auto-refund credit on failure
|
||||
job = self.get_job(job_id)
|
||||
if job:
|
||||
self._refund_credit_for_job(job_id, job, reason="job_failed")
|
||||
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Error marking job {job_id} as failed: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def update_file_url(self, job_id: str, category: str, file_type: str, url: str) -> None:
|
||||
"""
|
||||
Update a file URL in the job's file_urls structure.
|
||||
@@ -752,49 +931,116 @@ class JobManager:
|
||||
|
||||
def check_parallel_processing_complete(self, job_id: str) -> bool:
|
||||
"""
|
||||
Check if both parallel tracks (audio + lyrics) are complete.
|
||||
|
||||
This is called after audio_complete or lyrics_complete to determine
|
||||
if we can proceed to screen generation.
|
||||
|
||||
Check if lyrics processing is complete (sufficient for screen generation).
|
||||
|
||||
Audio separation is decoupled from the critical path — users can start
|
||||
reviewing lyrics while separation continues in the background. The
|
||||
instrumental review page handles the case where separation isn't done yet.
|
||||
|
||||
Returns:
|
||||
True if both tracks complete, False otherwise
|
||||
True if lyrics are complete, False otherwise
|
||||
"""
|
||||
job = self.get_job(job_id)
|
||||
if not job:
|
||||
return False
|
||||
|
||||
audio_complete = job.state_data.get('audio_complete', False)
|
||||
|
||||
lyrics_complete = job.state_data.get('lyrics_complete', False)
|
||||
|
||||
return audio_complete and lyrics_complete
|
||||
|
||||
|
||||
return lyrics_complete
|
||||
|
||||
async def start_job_processing(
|
||||
self,
|
||||
job_id: str,
|
||||
progress: int = 15,
|
||||
message: str = "Starting audio and lyrics processing"
|
||||
) -> None:
|
||||
"""
|
||||
Transition job to DOWNLOADING and trigger audio + lyrics workers.
|
||||
|
||||
This is the single entry point for starting job processing after audio
|
||||
is available. All handlers (file_upload, webhook, audio_search) should
|
||||
use this method to ensure consistent state transitions.
|
||||
|
||||
The method:
|
||||
1. Validates job exists and has input_media_gcs_path
|
||||
2. Transitions to DOWNLOADING status (raises if invalid)
|
||||
3. Triggers audio and lyrics workers in parallel
|
||||
|
||||
Args:
|
||||
job_id: Job ID to start processing
|
||||
progress: Progress percentage to set (default 15)
|
||||
message: Timeline message (default "Starting audio and lyrics processing")
|
||||
|
||||
Raises:
|
||||
InvalidStateTransitionError: If job can't transition to DOWNLOADING
|
||||
ValueError: If job not found or missing input_media_gcs_path
|
||||
|
||||
Example:
|
||||
# After downloading audio to GCS:
|
||||
job_manager.update_job(job_id, {'input_media_gcs_path': audio_path})
|
||||
await job_manager.start_job_processing(job_id)
|
||||
"""
|
||||
import asyncio
|
||||
from backend.services.worker_service import get_worker_service
|
||||
|
||||
job = self.get_job(job_id)
|
||||
if not job:
|
||||
raise ValueError(f"Job {job_id} not found")
|
||||
|
||||
if not job.input_media_gcs_path:
|
||||
raise ValueError(
|
||||
f"Job {job_id} missing input_media_gcs_path. "
|
||||
"Audio must be downloaded to GCS before starting processing."
|
||||
)
|
||||
|
||||
# Transition to DOWNLOADING - this will raise if invalid
|
||||
self.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.DOWNLOADING,
|
||||
progress=progress,
|
||||
message=message
|
||||
)
|
||||
|
||||
# Check if audio separation can be skipped
|
||||
# Jobs with an existing instrumental (e.g. tenant uploads) don't need
|
||||
# GPU-powered audio separation — saves 5-10 minutes of processing time
|
||||
worker_service = get_worker_service()
|
||||
|
||||
if job.existing_instrumental_gcs_path:
|
||||
logger.info(f"Job {job_id}: Skipping audio separation (existing instrumental provided)")
|
||||
# Mark audio as immediately complete and only trigger lyrics worker
|
||||
self.update_state_data(job_id, 'audio_complete', True)
|
||||
await worker_service.trigger_lyrics_worker(job_id)
|
||||
logger.info(f"Job {job_id}: Started processing (lyrics worker only, audio separation skipped)")
|
||||
else:
|
||||
# Trigger both workers in parallel
|
||||
await asyncio.gather(
|
||||
worker_service.trigger_audio_worker(job_id),
|
||||
worker_service.trigger_lyrics_worker(job_id)
|
||||
)
|
||||
logger.info(f"Job {job_id}: Started processing (audio + lyrics workers triggered)")
|
||||
|
||||
def mark_audio_complete(self, job_id: str) -> None:
|
||||
"""
|
||||
Mark audio processing as complete and check if can proceed.
|
||||
|
||||
If lyrics are also complete, automatically triggers screens worker.
|
||||
Mark audio processing as complete.
|
||||
|
||||
Audio completion is decoupled from the lyrics review path — it no longer
|
||||
triggers screens. The instrumental review page polls for this flag.
|
||||
"""
|
||||
self.update_state_data(job_id, 'audio_complete', True)
|
||||
|
||||
if self.check_parallel_processing_complete(job_id):
|
||||
logger.info(f"Job {job_id}: Both audio and lyrics complete, triggering screens worker")
|
||||
# Transition happens in screens worker
|
||||
# We just trigger it here
|
||||
self._trigger_screens_worker(job_id)
|
||||
|
||||
logger.info(f"Job {job_id}: Audio processing complete (separation finished)")
|
||||
|
||||
def mark_lyrics_complete(self, job_id: str) -> None:
|
||||
"""
|
||||
Mark lyrics processing as complete and check if can proceed.
|
||||
|
||||
If audio is also complete, automatically triggers screens worker.
|
||||
Mark lyrics processing as complete and trigger screens if ready.
|
||||
|
||||
Lyrics completion is the sole gate for screen generation and review.
|
||||
Audio separation runs independently in the background.
|
||||
"""
|
||||
self.update_state_data(job_id, 'lyrics_complete', True)
|
||||
|
||||
|
||||
if self.check_parallel_processing_complete(job_id):
|
||||
logger.info(f"Job {job_id}: Both audio and lyrics complete, triggering screens worker")
|
||||
# Transition happens in screens worker
|
||||
# We just trigger it here
|
||||
logger.info(f"Job {job_id}: Lyrics complete, triggering screens worker")
|
||||
self._trigger_screens_worker(job_id)
|
||||
|
||||
def _trigger_screens_worker(self, job_id: str) -> None:
|
||||
@@ -853,8 +1099,12 @@ class JobManager:
|
||||
)
|
||||
|
||||
logger.info(f"Job {job_id} cancelled")
|
||||
|
||||
# Refund credit for cancelled job
|
||||
self._refund_credit_for_job(job_id, job, reason="job_cancelled")
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def mark_job_failed(
|
||||
self,
|
||||
job_id: str,
|
||||
|
||||
@@ -12,8 +12,10 @@ import logging
|
||||
import os
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
from backend.i18n import get_locale_prefix
|
||||
from backend.services.email_service import get_email_service
|
||||
from backend.services.template_service import get_template_service
|
||||
from backend.services.user_service import get_user_service
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -51,15 +53,30 @@ class JobNotificationService:
|
||||
self.frontend_url = os.getenv("FRONTEND_URL", "https://gen.nomadkaraoke.com")
|
||||
self.backend_url = os.getenv("BACKEND_URL", "https://api.nomadkaraoke.com")
|
||||
|
||||
def _build_review_url(self, job_id: str, audio_hash: Optional[str] = None, review_token: Optional[str] = None) -> str:
|
||||
def _get_user_locale(self, user_email: str) -> str:
|
||||
"""Look up user's locale preference from Firestore, defaulting to 'en'."""
|
||||
try:
|
||||
user_service = get_user_service()
|
||||
user = user_service.get_user(user_email)
|
||||
if user and user.locale:
|
||||
return user.locale
|
||||
except Exception:
|
||||
pass # Fall back to English
|
||||
return "en"
|
||||
|
||||
def _build_review_url(self, job_id: str, audio_hash: Optional[str] = None, review_token: Optional[str] = None, locale: str = "en") -> str:
|
||||
"""Build the lyrics review URL for a job."""
|
||||
# Use hash-based routing for static hosting compatibility
|
||||
return f"{self.frontend_url}/app/jobs#/{job_id}/review"
|
||||
return f"{self.frontend_url}{get_locale_prefix(locale)}/app/jobs#/{job_id}/review"
|
||||
|
||||
def _build_instrumental_url(self, job_id: str, instrumental_token: Optional[str] = None) -> str:
|
||||
def _build_instrumental_url(self, job_id: str, instrumental_token: Optional[str] = None, locale: str = "en") -> str:
|
||||
"""Build the instrumental selection URL for a job."""
|
||||
# Use hash-based routing for static hosting compatibility
|
||||
return f"{self.frontend_url}/app/jobs#/{job_id}/instrumental"
|
||||
return f"{self.frontend_url}{get_locale_prefix(locale)}/app/jobs#/{job_id}/instrumental"
|
||||
|
||||
def _build_audio_edit_url(self, job_id: str, locale: str = "en") -> str:
|
||||
"""Build the audio edit URL for a job."""
|
||||
return f"{self.frontend_url}{get_locale_prefix(locale)}/app/jobs#/{job_id}/audio-edit"
|
||||
|
||||
async def send_job_completion_email(
|
||||
self,
|
||||
@@ -71,6 +88,8 @@ class JobNotificationService:
|
||||
youtube_url: Optional[str] = None,
|
||||
dropbox_url: Optional[str] = None,
|
||||
brand_code: Optional[str] = None,
|
||||
is_private: bool = False,
|
||||
youtube_queued: bool = False,
|
||||
) -> bool:
|
||||
"""
|
||||
Send job completion email to user.
|
||||
@@ -84,6 +103,8 @@ class JobNotificationService:
|
||||
youtube_url: YouTube video URL
|
||||
dropbox_url: Dropbox folder URL
|
||||
brand_code: Release ID (e.g., "NOMAD-1178")
|
||||
is_private: If True, omit YouTube section (private/non-published tracks)
|
||||
youtube_queued: If True, YouTube upload was deferred due to quota
|
||||
|
||||
Returns:
|
||||
True if email was sent successfully
|
||||
@@ -97,6 +118,9 @@ class JobNotificationService:
|
||||
return False
|
||||
|
||||
try:
|
||||
# Get user locale for email
|
||||
user_locale = self._get_user_locale(user_email)
|
||||
|
||||
# Render the completion message using template service
|
||||
message_content = self.template_service.render_job_completion(
|
||||
name=user_name,
|
||||
@@ -106,6 +130,9 @@ class JobNotificationService:
|
||||
title=title,
|
||||
job_id=job_id,
|
||||
feedback_url=FEEDBACK_FORM_URL,
|
||||
is_private=is_private,
|
||||
youtube_queued=youtube_queued,
|
||||
locale=user_locale,
|
||||
)
|
||||
|
||||
# Send the email with CC to admin
|
||||
@@ -116,6 +143,7 @@ class JobNotificationService:
|
||||
title=title,
|
||||
brand_code=brand_code,
|
||||
cc_admin=True,
|
||||
locale=user_locale,
|
||||
)
|
||||
|
||||
if success:
|
||||
@@ -129,6 +157,68 @@ class JobNotificationService:
|
||||
logger.exception(f"Error sending completion email for job {job_id}: {e}")
|
||||
return False
|
||||
|
||||
async def send_youtube_upload_complete_email(
|
||||
self,
|
||||
job_id: str,
|
||||
user_email: str,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
youtube_url: Optional[str] = None,
|
||||
brand_code: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Send follow-up email when a deferred YouTube upload completes.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
user_email: User's email address
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
youtube_url: YouTube video URL
|
||||
brand_code: Release ID
|
||||
|
||||
Returns:
|
||||
True if email was sent successfully
|
||||
"""
|
||||
if not ENABLE_AUTO_EMAILS:
|
||||
logger.info(f"Auto emails disabled, skipping YouTube notification for job {job_id}")
|
||||
return False
|
||||
|
||||
if not user_email:
|
||||
logger.warning(f"No user email for job {job_id}, skipping YouTube notification")
|
||||
return False
|
||||
|
||||
try:
|
||||
# Get user locale for email
|
||||
user_locale = self._get_user_locale(user_email)
|
||||
|
||||
message_content = self.template_service.render_youtube_upload_complete(
|
||||
artist=artist,
|
||||
title=title,
|
||||
youtube_url=youtube_url,
|
||||
locale=user_locale,
|
||||
)
|
||||
|
||||
success = self.email_service.send_youtube_upload_complete(
|
||||
to_email=user_email,
|
||||
message_content=message_content,
|
||||
artist=artist,
|
||||
title=title,
|
||||
brand_code=brand_code,
|
||||
locale=user_locale,
|
||||
)
|
||||
|
||||
if success:
|
||||
logger.info(f"Sent YouTube upload notification for job {job_id} to {_mask_email(user_email)}")
|
||||
else:
|
||||
logger.error(f"Failed to send YouTube upload notification for job {job_id}")
|
||||
|
||||
return success
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"Error sending YouTube upload notification for job {job_id}: {e}")
|
||||
return False
|
||||
|
||||
async def send_action_reminder_email(
|
||||
self,
|
||||
job_id: str,
|
||||
@@ -150,6 +240,7 @@ class JobNotificationService:
|
||||
action_type: Type of action needed:
|
||||
- "lyrics": Combined review (lyrics + instrumental selection) for normal jobs
|
||||
- "instrumental": Instrumental-only selection for finalise-only jobs
|
||||
- "audio_edit": Audio editing before processing begins
|
||||
user_name: User's display name
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
@@ -169,22 +260,37 @@ class JobNotificationService:
|
||||
return False
|
||||
|
||||
try:
|
||||
# Get user locale for email
|
||||
user_locale = self._get_user_locale(user_email)
|
||||
|
||||
# Render the appropriate template
|
||||
if action_type == "lyrics":
|
||||
review_url = self._build_review_url(job_id, audio_hash, review_token)
|
||||
review_url = self._build_review_url(job_id, audio_hash, review_token, locale=user_locale)
|
||||
message_content = self.template_service.render_action_needed_lyrics(
|
||||
name=user_name,
|
||||
artist=artist,
|
||||
title=title,
|
||||
review_url=review_url,
|
||||
locale=user_locale,
|
||||
)
|
||||
elif action_type == "instrumental":
|
||||
instrumental_url = self._build_instrumental_url(job_id, instrumental_token)
|
||||
instrumental_url = self._build_instrumental_url(job_id, instrumental_token, locale=user_locale)
|
||||
message_content = self.template_service.render_action_needed_instrumental(
|
||||
name=user_name,
|
||||
artist=artist,
|
||||
title=title,
|
||||
instrumental_url=instrumental_url,
|
||||
locale=user_locale,
|
||||
)
|
||||
elif action_type == "audio_edit":
|
||||
audio_edit_url = self._build_audio_edit_url(job_id, locale=user_locale)
|
||||
# Reuse the lyrics template but with the audio edit URL and different subject
|
||||
message_content = self.template_service.render_action_needed_lyrics(
|
||||
name=user_name,
|
||||
artist=artist,
|
||||
title=title,
|
||||
review_url=audio_edit_url,
|
||||
locale=user_locale,
|
||||
)
|
||||
else:
|
||||
logger.error(f"Unknown action type: {action_type}")
|
||||
@@ -197,6 +303,7 @@ class JobNotificationService:
|
||||
action_type=action_type,
|
||||
artist=artist,
|
||||
title=title,
|
||||
locale=user_locale,
|
||||
)
|
||||
|
||||
if success:
|
||||
@@ -218,6 +325,8 @@ class JobNotificationService:
|
||||
title: Optional[str] = None,
|
||||
youtube_url: Optional[str] = None,
|
||||
dropbox_url: Optional[str] = None,
|
||||
is_private: bool = False,
|
||||
youtube_queued: bool = False,
|
||||
) -> str:
|
||||
"""
|
||||
Get the rendered completion message for a job (for admin copy functionality).
|
||||
@@ -229,6 +338,8 @@ class JobNotificationService:
|
||||
title: Song title
|
||||
youtube_url: YouTube video URL
|
||||
dropbox_url: Dropbox folder URL
|
||||
is_private: If True, omit YouTube section (private/non-published tracks)
|
||||
youtube_queued: If True, YouTube upload was deferred
|
||||
|
||||
Returns:
|
||||
Rendered message content as plain text
|
||||
@@ -241,6 +352,8 @@ class JobNotificationService:
|
||||
title=title,
|
||||
job_id=job_id,
|
||||
feedback_url=FEEDBACK_FORM_URL,
|
||||
is_private=is_private,
|
||||
youtube_queued=youtube_queued,
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,203 @@
|
||||
"""
|
||||
KaraokeNerds community version detection service.
|
||||
|
||||
Scrapes karaokenerds.com to check if a song already has community-approved
|
||||
karaoke versions available on YouTube. Ported from kjbox/kj-controller/karaoke_nerds.py.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from typing import Any
|
||||
from urllib.parse import urlencode
|
||||
|
||||
import httpx
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SEARCH_URL = "https://karaokenerds.com/Search"
|
||||
REQUEST_TIMEOUT = 8
|
||||
USER_AGENT = "NomadKaraokeGen/1.0"
|
||||
|
||||
# In-memory TTL cache: { cache_key: (expiry_timestamp, data) }
|
||||
_cache: dict[str, tuple[float, Any]] = {}
|
||||
CACHE_TTL_SECONDS = 3600 # 1 hour
|
||||
|
||||
|
||||
def _cache_get(key: str) -> Any | None:
|
||||
"""Get a value from cache if it exists and hasn't expired."""
|
||||
if key in _cache:
|
||||
expiry, data = _cache[key]
|
||||
if time.monotonic() < expiry:
|
||||
return data
|
||||
del _cache[key]
|
||||
return None
|
||||
|
||||
|
||||
def _cache_set(key: str, data: Any, ttl: float = CACHE_TTL_SECONDS) -> None:
|
||||
"""Store a value in cache with TTL."""
|
||||
_cache[key] = (time.monotonic() + ttl, data)
|
||||
|
||||
|
||||
def _clean_youtube_url(url: str) -> str:
|
||||
"""Strip playlist params from YouTube URLs, keep just the video URL."""
|
||||
return re.sub(r"&list=[^&]*", "", url)
|
||||
|
||||
|
||||
def _parse_single_track(li) -> dict | None:
|
||||
"""Parse a single track <li> element from karaokenerds search results."""
|
||||
# Brand name: first <a> in the li
|
||||
brand_link = li.find("a")
|
||||
brand_name = brand_link.get_text(strip=True) if brand_link else ""
|
||||
|
||||
# Brand code: text inside .badge span
|
||||
badge = li.find("span", class_="badge")
|
||||
brand_code = ""
|
||||
if badge:
|
||||
brand_code = badge.get_text(strip=True)
|
||||
|
||||
# YouTube URL: link containing youtube.com
|
||||
youtube_url = None
|
||||
for a in li.find_all("a", href=True):
|
||||
href = a["href"]
|
||||
if "youtube.com" in href:
|
||||
youtube_url = _clean_youtube_url(href)
|
||||
break
|
||||
|
||||
# Community: presence of img.check
|
||||
is_community = bool(li.find("img", class_="check"))
|
||||
|
||||
if not youtube_url:
|
||||
return None
|
||||
|
||||
return {
|
||||
"brand_name": brand_name,
|
||||
"brand_code": brand_code,
|
||||
"youtube_url": youtube_url,
|
||||
"is_community": is_community,
|
||||
}
|
||||
|
||||
|
||||
def _parse_tracks(details_row) -> list[dict]:
|
||||
"""Parse track list items from a details row."""
|
||||
tracks = []
|
||||
for li in details_row.find_all("li", class_="track"):
|
||||
track = _parse_single_track(li)
|
||||
if track:
|
||||
tracks.append(track)
|
||||
return tracks
|
||||
|
||||
|
||||
def parse_results(html: str) -> list[dict]:
|
||||
"""Parse karaokenerds.com search results HTML into structured data."""
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
table = soup.find("table")
|
||||
if not table:
|
||||
return []
|
||||
|
||||
tbody = table.find("tbody")
|
||||
if not tbody:
|
||||
return []
|
||||
|
||||
songs = []
|
||||
rows = tbody.find_all("tr", recursive=False)
|
||||
|
||||
i = 0
|
||||
while i < len(rows):
|
||||
row = rows[i]
|
||||
|
||||
# Song rows have class "group"
|
||||
if "group" not in row.get("class", []):
|
||||
i += 1
|
||||
continue
|
||||
|
||||
# Extract title and artist from the song row
|
||||
cells = row.find_all("td")
|
||||
if len(cells) < 3:
|
||||
i += 1
|
||||
continue
|
||||
|
||||
title_link = cells[0].find("a")
|
||||
artist_link = cells[1].find("a")
|
||||
title = title_link.get_text(strip=True) if title_link else ""
|
||||
artist = artist_link.get_text(strip=True) if artist_link else ""
|
||||
|
||||
# The next row should be the details row with tracks
|
||||
tracks = []
|
||||
if i + 1 < len(rows):
|
||||
details_row = rows[i + 1]
|
||||
if "details" in details_row.get("class", []):
|
||||
tracks = _parse_tracks(details_row)
|
||||
i += 2
|
||||
else:
|
||||
i += 1
|
||||
else:
|
||||
i += 1
|
||||
|
||||
if title:
|
||||
songs.append({
|
||||
"title": title,
|
||||
"artist": artist,
|
||||
"tracks": tracks,
|
||||
})
|
||||
|
||||
return songs
|
||||
|
||||
|
||||
async def check_community_versions(artist: str, title: str) -> dict:
|
||||
"""
|
||||
Check if a song has community-approved karaoke versions on karaokenerds.
|
||||
|
||||
Returns a dict with:
|
||||
- has_community: bool
|
||||
- songs: list of matched songs with community tracks
|
||||
- best_youtube_url: URL of the top community version (if any)
|
||||
"""
|
||||
query = f"{artist} {title}"
|
||||
cache_key = f"community:{query.lower().strip()}"
|
||||
cached = _cache_get(cache_key)
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
params = urlencode({"query": query, "webFilter": "OnlyWeb"})
|
||||
url = f"{SEARCH_URL}?{params}"
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=REQUEST_TIMEOUT) as client:
|
||||
resp = await client.get(
|
||||
url,
|
||||
headers={"User-Agent": USER_AGENT},
|
||||
follow_redirects=True,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
except Exception as e:
|
||||
logger.warning(f"KaraokeNerds search failed for '{query}': {e}")
|
||||
result = {"has_community": False, "songs": [], "best_youtube_url": None}
|
||||
_cache_set(cache_key, result)
|
||||
return result
|
||||
|
||||
songs = parse_results(resp.text)
|
||||
|
||||
# Filter to only songs that have community tracks with YouTube URLs
|
||||
community_songs = []
|
||||
best_youtube_url = None
|
||||
|
||||
for song in songs:
|
||||
community_tracks = [t for t in song["tracks"] if t["is_community"]]
|
||||
if community_tracks:
|
||||
community_songs.append({
|
||||
"title": song["title"],
|
||||
"artist": song["artist"],
|
||||
"community_tracks": community_tracks,
|
||||
})
|
||||
if best_youtube_url is None:
|
||||
best_youtube_url = community_tracks[0]["youtube_url"]
|
||||
|
||||
result = {
|
||||
"has_community": len(community_songs) > 0,
|
||||
"songs": community_songs,
|
||||
"best_youtube_url": best_youtube_url,
|
||||
}
|
||||
_cache_set(cache_key, result)
|
||||
return result
|
||||
@@ -545,7 +545,10 @@ class LocalEncodingService:
|
||||
)
|
||||
output_files["with_vocals_mp4"] = config.output_with_vocals_mp4
|
||||
else:
|
||||
self.logger.info("[Step 2/6] Skipped - video already MP4")
|
||||
self.logger.info("[Step 2/6] Skipped - video already MP4, copying as-is")
|
||||
import shutil
|
||||
shutil.copy2(config.karaoke_video, config.output_with_vocals_mp4)
|
||||
output_files["with_vocals_mp4"] = config.output_with_vocals_mp4
|
||||
|
||||
# Step 3: Encode lossless 4K MP4
|
||||
if config.output_lossless_4k_mp4:
|
||||
|
||||
@@ -57,6 +57,8 @@ class PackagingService:
|
||||
title: str,
|
||||
output_mp3_path: Optional[str] = None,
|
||||
output_cdg_path: Optional[str] = None,
|
||||
lrc_has_countdown_padding: bool = False,
|
||||
countdown_padding_seconds: float = 3.0,
|
||||
) -> Tuple[str, Optional[str], Optional[str]]:
|
||||
"""
|
||||
Create a CDG package (ZIP containing CDG and MP3 files).
|
||||
@@ -69,6 +71,9 @@ class PackagingService:
|
||||
title: Song title for metadata
|
||||
output_mp3_path: Optional path for the extracted MP3 file
|
||||
output_cdg_path: Optional path for the extracted CDG file
|
||||
lrc_has_countdown_padding: If True, LRC contains countdown-shifted timestamps
|
||||
that need adjustment since CDG uses instrumental audio without padding
|
||||
countdown_padding_seconds: Duration of countdown padding to compensate for
|
||||
|
||||
Returns:
|
||||
Tuple of (zip_path, mp3_path, cdg_path)
|
||||
@@ -131,6 +136,8 @@ class PackagingService:
|
||||
title=title,
|
||||
artist=artist,
|
||||
cdg_styles=self.cdg_styles,
|
||||
lrc_has_countdown_padding=lrc_has_countdown_padding,
|
||||
countdown_padding_seconds=countdown_padding_seconds,
|
||||
)
|
||||
|
||||
# Rename ZIP to expected output path if different
|
||||
|
||||
@@ -12,6 +12,7 @@ from pywebpush import webpush, WebPushException
|
||||
|
||||
from backend.config import get_settings
|
||||
from backend.models.user import User, PushSubscription
|
||||
from backend.i18n import t
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -85,10 +86,14 @@ class PushNotificationService:
|
||||
title: str,
|
||||
body: str,
|
||||
url: str = "/app/",
|
||||
tag: Optional[str] = None
|
||||
tag: Optional[str] = None,
|
||||
tenant_id: Optional[str] = None
|
||||
) -> int:
|
||||
"""
|
||||
Send a push notification to all of a user's subscribed devices.
|
||||
Send a push notification to a user's subscribed devices for a specific tenant.
|
||||
|
||||
Only sends to subscriptions matching the given tenant_id to prevent
|
||||
cross-tenant notification leakage.
|
||||
|
||||
Args:
|
||||
user_email: The user's email address
|
||||
@@ -96,6 +101,8 @@ class PushNotificationService:
|
||||
body: Notification body text
|
||||
url: URL to open when notification is clicked
|
||||
tag: Optional tag for notification grouping (replaces notifications with same tag)
|
||||
tenant_id: Tenant scope - only send to subscriptions for this tenant
|
||||
(None or "" = consumer/default portal)
|
||||
|
||||
Returns:
|
||||
Number of notifications successfully sent
|
||||
@@ -111,7 +118,30 @@ class PushNotificationService:
|
||||
return 0
|
||||
|
||||
user_data = user_doc.to_dict()
|
||||
subscriptions = user_data.get("push_subscriptions", [])
|
||||
all_subscriptions = user_data.get("push_subscriptions", [])
|
||||
|
||||
# Filter subscriptions to only those matching the target tenant
|
||||
# Normalize: None and "" both mean "consumer/default portal"
|
||||
target_tenant = tenant_id or None
|
||||
subscriptions = [
|
||||
sub for sub in all_subscriptions
|
||||
if (sub.get("tenant_id") or None) == target_tenant
|
||||
]
|
||||
|
||||
if not subscriptions and all_subscriptions:
|
||||
# Migration path: if no subscriptions match and none have tenant_id set,
|
||||
# it means they're legacy subscriptions created before tenant scoping.
|
||||
# Send to legacy subscriptions (those without tenant_id) only if
|
||||
# the target is the consumer portal (no tenant).
|
||||
has_any_tenant_scoped = any(sub.get("tenant_id") for sub in all_subscriptions)
|
||||
if not has_any_tenant_scoped and target_tenant is None:
|
||||
subscriptions = all_subscriptions
|
||||
logger.debug(f"Using legacy unscoped subscriptions for {user_email}")
|
||||
else:
|
||||
logger.debug(
|
||||
f"No matching subscriptions for tenant={target_tenant} "
|
||||
f"(user has {len(all_subscriptions)} total subscriptions)"
|
||||
)
|
||||
|
||||
if not subscriptions:
|
||||
logger.debug(f"User {user_email} has no push subscriptions")
|
||||
@@ -196,7 +226,7 @@ class PushNotificationService:
|
||||
Send notification when a job enters a blocking state requiring user action.
|
||||
|
||||
Args:
|
||||
job: Job dictionary with job_id, user_email, artist, title
|
||||
job: Job dictionary with job_id, user_email, artist, title, tenant_id
|
||||
action_type: Type of action needed ("lyrics" or "instrumental")
|
||||
|
||||
Returns:
|
||||
@@ -209,16 +239,18 @@ class PushNotificationService:
|
||||
job_id = job.get("job_id", "unknown")
|
||||
artist = job.get("artist", "Unknown Artist")
|
||||
title = job.get("title", "Unknown Title")
|
||||
tenant_id = job.get("tenant_id") or None
|
||||
locale = job.get("locale", "en")
|
||||
|
||||
if action_type == "lyrics":
|
||||
notif_title = "Review Lyrics"
|
||||
notif_body = f'"{title}" by {artist} needs lyrics review'
|
||||
url = f"/review/{job_id}"
|
||||
notif_title = t(locale, "pushNotifications.reviewLyricsTitle")
|
||||
notif_body = t(locale, "pushNotifications.reviewLyricsBody", title=title, artist=artist)
|
||||
url = f"/app/jobs#/{job_id}/review"
|
||||
tag = f"lyrics-{job_id}"
|
||||
else: # instrumental
|
||||
notif_title = "Select Instrumental"
|
||||
notif_body = f'"{title}" by {artist} needs instrumental selection'
|
||||
url = f"/instrumental/{job_id}"
|
||||
notif_title = t(locale, "pushNotifications.selectInstrumentalTitle")
|
||||
notif_body = t(locale, "pushNotifications.selectInstrumentalBody", title=title, artist=artist)
|
||||
url = f"/app/jobs#/{job_id}/instrumental"
|
||||
tag = f"instrumental-{job_id}"
|
||||
|
||||
return await self.send_push(
|
||||
@@ -226,7 +258,8 @@ class PushNotificationService:
|
||||
title=notif_title,
|
||||
body=notif_body,
|
||||
url=url,
|
||||
tag=tag
|
||||
tag=tag,
|
||||
tenant_id=tenant_id,
|
||||
)
|
||||
|
||||
async def send_completion_notification(self, job: dict) -> int:
|
||||
@@ -234,7 +267,7 @@ class PushNotificationService:
|
||||
Send notification when a job completes successfully.
|
||||
|
||||
Args:
|
||||
job: Job dictionary with job_id, user_email, artist, title
|
||||
job: Job dictionary with job_id, user_email, artist, title, tenant_id
|
||||
|
||||
Returns:
|
||||
Number of notifications sent
|
||||
@@ -246,13 +279,16 @@ class PushNotificationService:
|
||||
job_id = job.get("job_id", "unknown")
|
||||
artist = job.get("artist", "Unknown Artist")
|
||||
title = job.get("title", "Unknown Title")
|
||||
tenant_id = job.get("tenant_id") or None
|
||||
locale = job.get("locale", "en")
|
||||
|
||||
return await self.send_push(
|
||||
user_email=user_email,
|
||||
title="Video Ready!",
|
||||
body=f'Your karaoke video for "{title}" by {artist} is ready to download',
|
||||
title=t(locale, "pushNotifications.videoReadyTitle"),
|
||||
body=t(locale, "pushNotifications.videoReadyBody", title=title, artist=artist),
|
||||
url=f"/app/?job={job_id}",
|
||||
tag=f"complete-{job_id}"
|
||||
tag=f"complete-{job_id}",
|
||||
tenant_id=tenant_id,
|
||||
)
|
||||
|
||||
async def add_subscription(
|
||||
@@ -260,10 +296,11 @@ class PushNotificationService:
|
||||
user_email: str,
|
||||
endpoint: str,
|
||||
keys: dict,
|
||||
device_name: Optional[str] = None
|
||||
device_name: Optional[str] = None,
|
||||
tenant_id: Optional[str] = None
|
||||
) -> bool:
|
||||
"""
|
||||
Add a push subscription for a user.
|
||||
Add a push subscription for a user, scoped to a tenant.
|
||||
|
||||
Enforces max subscriptions per user - oldest removed if limit exceeded.
|
||||
|
||||
@@ -272,6 +309,7 @@ class PushNotificationService:
|
||||
endpoint: Push service endpoint URL
|
||||
keys: Encryption keys (p256dh, auth)
|
||||
device_name: Optional device identifier
|
||||
tenant_id: Tenant scope (None = consumer/default portal)
|
||||
|
||||
Returns:
|
||||
True if subscription was added successfully
|
||||
@@ -298,6 +336,7 @@ class PushNotificationService:
|
||||
"endpoint": endpoint,
|
||||
"keys": keys,
|
||||
"device_name": device_name,
|
||||
"tenant_id": tenant_id or None,
|
||||
"created_at": datetime.now(timezone.utc).isoformat(),
|
||||
"last_used_at": None
|
||||
}
|
||||
|
||||
@@ -1,641 +0,0 @@
|
||||
"""
|
||||
Rate limiting service for job creation, YouTube uploads, and beta enrollment.
|
||||
|
||||
Uses Firestore for distributed rate limit tracking with date-based document IDs
|
||||
that automatically reset at UTC midnight.
|
||||
"""
|
||||
import hashlib
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Tuple, Optional, List, Dict, Any
|
||||
|
||||
from google.cloud import firestore
|
||||
|
||||
from backend.config import settings
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Firestore collection names
|
||||
RATE_LIMITS_COLLECTION = "rate_limits"
|
||||
BLOCKLISTS_COLLECTION = "blocklists"
|
||||
OVERRIDES_COLLECTION = "overrides"
|
||||
|
||||
|
||||
def _get_today_date_str() -> str:
|
||||
"""Get today's date in UTC as YYYY-MM-DD string."""
|
||||
return datetime.now(timezone.utc).strftime("%Y-%m-%d")
|
||||
|
||||
|
||||
def _seconds_until_midnight_utc() -> int:
|
||||
"""Calculate seconds until UTC midnight (when rate limits reset)."""
|
||||
now = datetime.now(timezone.utc)
|
||||
midnight = now.replace(hour=0, minute=0, second=0, microsecond=0)
|
||||
# Add 1 day to get next midnight
|
||||
from datetime import timedelta
|
||||
next_midnight = midnight + timedelta(days=1)
|
||||
return int((next_midnight - now).total_seconds())
|
||||
|
||||
|
||||
def _hash_ip(ip_address: str) -> str:
|
||||
"""Hash an IP address for storage (privacy-preserving)."""
|
||||
return hashlib.sha256(ip_address.encode()).hexdigest()[:16]
|
||||
|
||||
|
||||
class RateLimitService:
|
||||
"""
|
||||
Service for managing rate limits across the platform.
|
||||
|
||||
Rate limits use Firestore for distributed tracking with automatic
|
||||
reset at UTC midnight via date-based document IDs.
|
||||
"""
|
||||
|
||||
def __init__(self, db: Optional[firestore.Client] = None):
|
||||
"""Initialize rate limit service with Firestore client."""
|
||||
if db is None:
|
||||
self.db = firestore.Client(project=settings.google_cloud_project)
|
||||
else:
|
||||
self.db = db
|
||||
self._blocklist_cache: Optional[Dict[str, Any]] = None
|
||||
self._blocklist_cache_time: Optional[datetime] = None
|
||||
self._cache_ttl_seconds = 300 # 5 minute cache
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Job Rate Limiting
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def check_user_job_limit(self, user_email: str, is_admin: bool = False) -> Tuple[bool, int, str]:
|
||||
"""
|
||||
Check if a user can create a new job.
|
||||
|
||||
Args:
|
||||
user_email: User's email address
|
||||
is_admin: Whether the user is an admin (admins bypass limits)
|
||||
|
||||
Returns:
|
||||
Tuple of (allowed, remaining, message)
|
||||
- allowed: True if user can create job
|
||||
- remaining: Number of jobs remaining today
|
||||
- message: Human-readable status message
|
||||
"""
|
||||
if not settings.enable_rate_limiting:
|
||||
return True, -1, "Rate limiting disabled"
|
||||
|
||||
if is_admin:
|
||||
return True, -1, "Admin users bypass rate limits"
|
||||
|
||||
limit = settings.rate_limit_jobs_per_day
|
||||
if limit == 0:
|
||||
return True, -1, "No job limit configured"
|
||||
|
||||
# Check for user override
|
||||
override = self.get_user_override(user_email)
|
||||
if override and override.get("bypass_job_limit"):
|
||||
return True, -1, "User has rate limit bypass"
|
||||
if override and override.get("custom_daily_job_limit"):
|
||||
limit = override["custom_daily_job_limit"]
|
||||
|
||||
# Get current count
|
||||
current_count = self.get_user_job_count_today(user_email)
|
||||
remaining = max(0, limit - current_count)
|
||||
|
||||
if current_count >= limit:
|
||||
seconds = _seconds_until_midnight_utc()
|
||||
logger.warning(
|
||||
f"Rate limit exceeded for {user_email}: {current_count}/{limit} jobs today"
|
||||
)
|
||||
return (
|
||||
False,
|
||||
0,
|
||||
f"Daily job limit reached ({limit} jobs per day). Resets in {seconds // 3600}h {(seconds % 3600) // 60}m."
|
||||
)
|
||||
|
||||
return True, remaining, f"{remaining} jobs remaining today"
|
||||
|
||||
def get_user_job_count_today(self, user_email: str) -> int:
|
||||
"""Get the number of jobs created by a user today."""
|
||||
date_str = _get_today_date_str()
|
||||
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
|
||||
f"user_jobs_{user_email}_{date_str}"
|
||||
)
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
return 0
|
||||
|
||||
return doc.to_dict().get("count", 0)
|
||||
|
||||
def record_job_creation(self, user_email: str, job_id: str) -> None:
|
||||
"""
|
||||
Record a job creation for rate limiting.
|
||||
|
||||
Uses Firestore transactions for atomic increment.
|
||||
"""
|
||||
if not settings.enable_rate_limiting:
|
||||
return
|
||||
|
||||
date_str = _get_today_date_str()
|
||||
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
|
||||
f"user_jobs_{user_email}_{date_str}"
|
||||
)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
count = data.get("count", 0) + 1
|
||||
job_ids = data.get("job_ids", [])
|
||||
job_ids.append(job_id)
|
||||
else:
|
||||
count = 1
|
||||
job_ids = [job_id]
|
||||
|
||||
transaction.set(doc_ref, {
|
||||
"user_email": user_email,
|
||||
"date": date_str,
|
||||
"count": count,
|
||||
"job_ids": job_ids,
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
})
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
logger.info(f"Recorded job {job_id} for user {user_email} rate limiting")
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# YouTube Upload Rate Limiting
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def check_youtube_upload_limit(self) -> Tuple[bool, int, str]:
|
||||
"""
|
||||
Check if the system can perform a YouTube upload.
|
||||
|
||||
YouTube uploads are limited system-wide due to API quota constraints.
|
||||
|
||||
Returns:
|
||||
Tuple of (allowed, remaining, message)
|
||||
"""
|
||||
if not settings.enable_rate_limiting:
|
||||
return True, -1, "Rate limiting disabled"
|
||||
|
||||
limit = settings.rate_limit_youtube_uploads_per_day
|
||||
if limit == 0:
|
||||
return True, -1, "No YouTube upload limit configured"
|
||||
|
||||
current_count = self.get_youtube_uploads_today()
|
||||
remaining = max(0, limit - current_count)
|
||||
|
||||
if current_count >= limit:
|
||||
seconds = _seconds_until_midnight_utc()
|
||||
logger.warning(
|
||||
f"YouTube upload limit exceeded: {current_count}/{limit} uploads today"
|
||||
)
|
||||
return (
|
||||
False,
|
||||
0,
|
||||
f"Daily YouTube upload limit reached ({limit} uploads per day). Resets in {seconds // 3600}h {(seconds % 3600) // 60}m."
|
||||
)
|
||||
|
||||
return True, remaining, f"{remaining} YouTube uploads remaining today"
|
||||
|
||||
def get_youtube_uploads_today(self) -> int:
|
||||
"""Get the number of YouTube uploads performed today (system-wide)."""
|
||||
date_str = _get_today_date_str()
|
||||
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
|
||||
f"youtube_uploads_{date_str}"
|
||||
)
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
return 0
|
||||
|
||||
return doc.to_dict().get("count", 0)
|
||||
|
||||
def record_youtube_upload(self, job_id: str, user_email: str) -> None:
|
||||
"""
|
||||
Record a YouTube upload for rate limiting.
|
||||
|
||||
Uses Firestore transactions for atomic increment.
|
||||
"""
|
||||
if not settings.enable_rate_limiting:
|
||||
return
|
||||
|
||||
date_str = _get_today_date_str()
|
||||
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
|
||||
f"youtube_uploads_{date_str}"
|
||||
)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
count = data.get("count", 0) + 1
|
||||
uploads = data.get("uploads", [])
|
||||
else:
|
||||
count = 1
|
||||
uploads = []
|
||||
|
||||
uploads.append({
|
||||
"job_id": job_id,
|
||||
"user_email": user_email,
|
||||
"timestamp": datetime.now(timezone.utc),
|
||||
})
|
||||
|
||||
transaction.set(doc_ref, {
|
||||
"date": date_str,
|
||||
"count": count,
|
||||
"uploads": uploads,
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
})
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
logger.info(f"Recorded YouTube upload for job {job_id}")
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Beta Enrollment IP Rate Limiting
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def check_beta_ip_limit(self, ip_address: str) -> Tuple[bool, int, str]:
|
||||
"""
|
||||
Check if an IP address can enroll in the beta program.
|
||||
|
||||
Args:
|
||||
ip_address: Client IP address
|
||||
|
||||
Returns:
|
||||
Tuple of (allowed, remaining, message)
|
||||
"""
|
||||
if not settings.enable_rate_limiting:
|
||||
return True, -1, "Rate limiting disabled"
|
||||
|
||||
limit = settings.rate_limit_beta_ip_per_day
|
||||
if limit == 0:
|
||||
return True, -1, "No beta IP limit configured"
|
||||
|
||||
# Check today's enrollment count for this IP
|
||||
ip_hash = _hash_ip(ip_address)
|
||||
date_str = _get_today_date_str()
|
||||
|
||||
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
|
||||
f"beta_ip_{ip_hash}_{date_str}"
|
||||
)
|
||||
doc = doc_ref.get()
|
||||
|
||||
current_count = 0
|
||||
if doc.exists:
|
||||
current_count = doc.to_dict().get("count", 0)
|
||||
|
||||
remaining = max(0, limit - current_count)
|
||||
|
||||
if current_count >= limit:
|
||||
seconds = _seconds_until_midnight_utc()
|
||||
logger.warning(
|
||||
f"Beta enrollment IP limit exceeded for {ip_hash}: {current_count}/{limit} enrollments today"
|
||||
)
|
||||
return (
|
||||
False,
|
||||
0,
|
||||
f"Too many beta enrollments from this network today. Please try again tomorrow."
|
||||
)
|
||||
|
||||
return True, remaining, f"{remaining} beta enrollments remaining from this IP today"
|
||||
|
||||
def record_beta_enrollment(self, ip_address: str, email: str) -> None:
|
||||
"""
|
||||
Record a beta enrollment for IP rate limiting.
|
||||
|
||||
Uses Firestore transactions for atomic increment.
|
||||
"""
|
||||
if not settings.enable_rate_limiting:
|
||||
return
|
||||
|
||||
ip_hash = _hash_ip(ip_address)
|
||||
date_str = _get_today_date_str()
|
||||
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
|
||||
f"beta_ip_{ip_hash}_{date_str}"
|
||||
)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
count = data.get("count", 0) + 1
|
||||
enrollments = data.get("enrollments", [])
|
||||
else:
|
||||
count = 1
|
||||
enrollments = []
|
||||
|
||||
enrollments.append({
|
||||
"email": email,
|
||||
"timestamp": datetime.now(timezone.utc),
|
||||
})
|
||||
|
||||
transaction.set(doc_ref, {
|
||||
"ip_hash": ip_hash,
|
||||
"date": date_str,
|
||||
"count": count,
|
||||
"enrollments": enrollments,
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
})
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
logger.info(f"Recorded beta enrollment from IP {ip_hash} for {email}")
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# User Overrides (Whitelist)
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def get_user_override(self, user_email: str) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Get rate limit override settings for a user.
|
||||
|
||||
Returns:
|
||||
Override settings dict or None if no override exists
|
||||
"""
|
||||
doc_ref = self.db.collection(OVERRIDES_COLLECTION).document(user_email.lower())
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
return None
|
||||
|
||||
return doc.to_dict()
|
||||
|
||||
def set_user_override(
|
||||
self,
|
||||
user_email: str,
|
||||
bypass_job_limit: bool = False,
|
||||
custom_daily_job_limit: Optional[int] = None,
|
||||
reason: str = "",
|
||||
admin_email: str = ""
|
||||
) -> None:
|
||||
"""
|
||||
Set rate limit override for a user.
|
||||
|
||||
Args:
|
||||
user_email: User to override
|
||||
bypass_job_limit: If True, user bypasses all job limits
|
||||
custom_daily_job_limit: Custom limit (None = use default)
|
||||
reason: Reason for override
|
||||
admin_email: Admin who set the override
|
||||
"""
|
||||
doc_ref = self.db.collection(OVERRIDES_COLLECTION).document(user_email.lower())
|
||||
doc_ref.set({
|
||||
"email": user_email.lower(),
|
||||
"bypass_job_limit": bypass_job_limit,
|
||||
"custom_daily_job_limit": custom_daily_job_limit,
|
||||
"reason": reason,
|
||||
"created_by": admin_email,
|
||||
"created_at": datetime.now(timezone.utc),
|
||||
})
|
||||
logger.info(f"Set rate limit override for {user_email} by {admin_email}")
|
||||
|
||||
def remove_user_override(self, user_email: str, admin_email: str = "") -> bool:
|
||||
"""Remove rate limit override for a user."""
|
||||
doc_ref = self.db.collection(OVERRIDES_COLLECTION).document(user_email.lower())
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
return False
|
||||
|
||||
doc_ref.delete()
|
||||
logger.info(f"Removed rate limit override for {user_email} by {admin_email}")
|
||||
return True
|
||||
|
||||
def list_user_overrides(self) -> List[Dict[str, Any]]:
|
||||
"""List all user rate limit overrides."""
|
||||
docs = self.db.collection(OVERRIDES_COLLECTION).stream()
|
||||
return [doc.to_dict() for doc in docs]
|
||||
|
||||
def get_all_overrides(self) -> Dict[str, Dict[str, Any]]:
|
||||
"""Get all user rate limit overrides as a dict keyed by email."""
|
||||
docs = self.db.collection(OVERRIDES_COLLECTION).stream()
|
||||
return {doc.id: doc.to_dict() for doc in docs}
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Blocklist Management
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def _load_blocklist(self, force_refresh: bool = False) -> Dict[str, Any]:
|
||||
"""
|
||||
Load blocklist from Firestore with caching.
|
||||
|
||||
Returns:
|
||||
Dict with keys: disposable_domains, blocked_emails, blocked_ips
|
||||
"""
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
# Check cache
|
||||
if not force_refresh and self._blocklist_cache and self._blocklist_cache_time:
|
||||
cache_age = (now - self._blocklist_cache_time).total_seconds()
|
||||
if cache_age < self._cache_ttl_seconds:
|
||||
return self._blocklist_cache
|
||||
|
||||
# Load from Firestore
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
# Initialize with empty blocklist
|
||||
self._blocklist_cache = {
|
||||
"disposable_domains": [],
|
||||
"blocked_emails": [],
|
||||
"blocked_ips": [],
|
||||
}
|
||||
else:
|
||||
self._blocklist_cache = doc.to_dict()
|
||||
|
||||
self._blocklist_cache_time = now
|
||||
return self._blocklist_cache
|
||||
|
||||
def is_disposable_domain(self, domain: str) -> bool:
|
||||
"""Check if a domain is in the disposable domains blocklist."""
|
||||
blocklist = self._load_blocklist()
|
||||
return domain.lower() in [d.lower() for d in blocklist.get("disposable_domains", [])]
|
||||
|
||||
def is_blocked_email(self, email: str) -> bool:
|
||||
"""Check if an email is explicitly blocked."""
|
||||
blocklist = self._load_blocklist()
|
||||
return email.lower() in [e.lower() for e in blocklist.get("blocked_emails", [])]
|
||||
|
||||
def is_blocked_ip(self, ip_address: str) -> bool:
|
||||
"""Check if an IP address is blocked."""
|
||||
blocklist = self._load_blocklist()
|
||||
return ip_address in blocklist.get("blocked_ips", [])
|
||||
|
||||
def get_blocklist(self) -> Dict[str, Any]:
|
||||
"""Get all blocklist data."""
|
||||
return self._load_blocklist(force_refresh=True)
|
||||
|
||||
def add_disposable_domain(self, domain: str, admin_email: str) -> None:
|
||||
"""Add a domain to the disposable domains blocklist."""
|
||||
blocklist = self._load_blocklist(force_refresh=True)
|
||||
domains = set(blocklist.get("disposable_domains", []))
|
||||
domains.add(domain.lower())
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc_ref.set({
|
||||
**blocklist,
|
||||
"disposable_domains": list(domains),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
"updated_by": admin_email,
|
||||
}, merge=True)
|
||||
|
||||
self._blocklist_cache = None # Invalidate cache
|
||||
logger.info(f"Added disposable domain {domain} by {admin_email}")
|
||||
|
||||
def remove_disposable_domain(self, domain: str) -> bool:
|
||||
"""Remove a domain from the disposable domains blocklist."""
|
||||
blocklist = self._load_blocklist(force_refresh=True)
|
||||
domains = set(blocklist.get("disposable_domains", []))
|
||||
|
||||
if domain.lower() not in [d.lower() for d in domains]:
|
||||
return False
|
||||
|
||||
domains = {d for d in domains if d.lower() != domain.lower()}
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc_ref.set({
|
||||
**blocklist,
|
||||
"disposable_domains": list(domains),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
}, merge=True)
|
||||
|
||||
self._blocklist_cache = None
|
||||
logger.info(f"Removed disposable domain {domain}")
|
||||
return True
|
||||
|
||||
def add_blocked_email(self, email: str, admin_email: str) -> None:
|
||||
"""Add an email to the blocked emails list."""
|
||||
blocklist = self._load_blocklist(force_refresh=True)
|
||||
emails = set(blocklist.get("blocked_emails", []))
|
||||
emails.add(email.lower())
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc_ref.set({
|
||||
**blocklist,
|
||||
"blocked_emails": list(emails),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
"updated_by": admin_email,
|
||||
}, merge=True)
|
||||
|
||||
self._blocklist_cache = None
|
||||
logger.info(f"Added blocked email {email} by {admin_email}")
|
||||
|
||||
def remove_blocked_email(self, email: str) -> bool:
|
||||
"""Remove an email from the blocked emails list."""
|
||||
blocklist = self._load_blocklist(force_refresh=True)
|
||||
emails = set(blocklist.get("blocked_emails", []))
|
||||
|
||||
if email.lower() not in [e.lower() for e in emails]:
|
||||
return False
|
||||
|
||||
emails = {e for e in emails if e.lower() != email.lower()}
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc_ref.set({
|
||||
**blocklist,
|
||||
"blocked_emails": list(emails),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
}, merge=True)
|
||||
|
||||
self._blocklist_cache = None
|
||||
logger.info(f"Removed blocked email {email}")
|
||||
return True
|
||||
|
||||
def add_blocked_ip(self, ip_address: str, admin_email: str) -> None:
|
||||
"""Add an IP address to the blocked IPs list."""
|
||||
blocklist = self._load_blocklist(force_refresh=True)
|
||||
ips = set(blocklist.get("blocked_ips", []))
|
||||
ips.add(ip_address)
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc_ref.set({
|
||||
**blocklist,
|
||||
"blocked_ips": list(ips),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
"updated_by": admin_email,
|
||||
}, merge=True)
|
||||
|
||||
self._blocklist_cache = None
|
||||
logger.info(f"Added blocked IP {ip_address} by {admin_email}")
|
||||
|
||||
def remove_blocked_ip(self, ip_address: str) -> bool:
|
||||
"""Remove an IP address from the blocked IPs list."""
|
||||
blocklist = self._load_blocklist(force_refresh=True)
|
||||
ips = set(blocklist.get("blocked_ips", []))
|
||||
|
||||
if ip_address not in ips:
|
||||
return False
|
||||
|
||||
ips.discard(ip_address)
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc_ref.set({
|
||||
**blocklist,
|
||||
"blocked_ips": list(ips),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
}, merge=True)
|
||||
|
||||
self._blocklist_cache = None
|
||||
logger.info(f"Removed blocked IP {ip_address}")
|
||||
return True
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Stats
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def get_rate_limit_stats(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Get current rate limit statistics.
|
||||
|
||||
Returns:
|
||||
Dict with current usage stats
|
||||
"""
|
||||
date_str = _get_today_date_str()
|
||||
|
||||
# Get YouTube upload count
|
||||
youtube_count = self.get_youtube_uploads_today()
|
||||
|
||||
# Count unique users with jobs today
|
||||
users_with_jobs = set()
|
||||
total_jobs_today = 0
|
||||
|
||||
# Query all user job documents for today
|
||||
docs = self.db.collection(RATE_LIMITS_COLLECTION).where(
|
||||
"date", "==", date_str
|
||||
).stream()
|
||||
|
||||
for doc in docs:
|
||||
data = doc.to_dict()
|
||||
if data.get("user_email"):
|
||||
users_with_jobs.add(data["user_email"])
|
||||
total_jobs_today += data.get("count", 0)
|
||||
|
||||
return {
|
||||
"date": date_str,
|
||||
"youtube_uploads_today": youtube_count,
|
||||
"youtube_uploads_limit": settings.rate_limit_youtube_uploads_per_day,
|
||||
"total_jobs_today": total_jobs_today,
|
||||
"users_with_jobs_today": len(users_with_jobs),
|
||||
"job_limit_per_user": settings.rate_limit_jobs_per_day,
|
||||
"beta_ip_limit_per_day": settings.rate_limit_beta_ip_per_day,
|
||||
"rate_limiting_enabled": settings.enable_rate_limiting,
|
||||
"seconds_until_reset": _seconds_until_midnight_utc(),
|
||||
}
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_rate_limit_service: Optional[RateLimitService] = None
|
||||
|
||||
|
||||
def get_rate_limit_service() -> RateLimitService:
|
||||
"""Get the singleton RateLimitService instance."""
|
||||
global _rate_limit_service
|
||||
if _rate_limit_service is None:
|
||||
_rate_limit_service = RateLimitService()
|
||||
return _rate_limit_service
|
||||
@@ -0,0 +1,526 @@
|
||||
"""
|
||||
Referral service for link management, earnings tracking, and payouts.
|
||||
|
||||
Handles:
|
||||
- Referral link creation (generated and vanity codes)
|
||||
- Link lookup and validation
|
||||
- Click tracking
|
||||
- Admin listing
|
||||
"""
|
||||
import logging
|
||||
import re
|
||||
import secrets
|
||||
import string
|
||||
from datetime import datetime, timedelta
|
||||
from typing import List, Optional, Tuple
|
||||
|
||||
from google.cloud import firestore
|
||||
|
||||
from backend.models.referral import (
|
||||
ReferralLink,
|
||||
ReferralLinkStats,
|
||||
ReferralEarning,
|
||||
ReferralPayout,
|
||||
)
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Constants
|
||||
# ============================================================================
|
||||
|
||||
REFERRAL_LINKS_COLLECTION = "referral_links"
|
||||
REFERRAL_EARNINGS_COLLECTION = "referral_earnings"
|
||||
REFERRAL_PAYOUTS_COLLECTION = "referral_payouts"
|
||||
|
||||
RESERVED_CODES = frozenset({
|
||||
"admin", "app", "api", "r", "pricing", "order", "payment",
|
||||
"login", "auth", "webhook", "webhooks", "internal", "health",
|
||||
"static", "assets", "public", "private",
|
||||
})
|
||||
|
||||
PAYOUT_THRESHOLD_CENTS = 2000
|
||||
|
||||
VANITY_CODE_PATTERN = re.compile(r"^[a-z0-9][a-z0-9\-]{1,28}[a-z0-9]$")
|
||||
|
||||
# Characters for generated codes
|
||||
_CODE_ALPHABET = string.ascii_lowercase + string.digits
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Service
|
||||
# ============================================================================
|
||||
|
||||
class ReferralService:
|
||||
"""Service for referral link management and earnings."""
|
||||
|
||||
def __init__(self, db=None, stripe_service=None):
|
||||
if db is None:
|
||||
db = firestore.Client()
|
||||
self.db = db
|
||||
self.stripe_service = stripe_service
|
||||
|
||||
# ========================================================================
|
||||
# Code Generation & Validation
|
||||
# ========================================================================
|
||||
|
||||
def _generate_code(self) -> str:
|
||||
"""Generate an 8-character lowercase alphanumeric referral code."""
|
||||
return "".join(secrets.choice(_CODE_ALPHABET) for _ in range(8))
|
||||
|
||||
def _validate_vanity_code(self, code: str) -> Tuple[bool, str]:
|
||||
"""Validate a vanity code against reserved words and pattern rules.
|
||||
|
||||
Returns (is_valid, message).
|
||||
"""
|
||||
if code.lower() in RESERVED_CODES:
|
||||
return False, f"'{code}' is a reserved code"
|
||||
if not VANITY_CODE_PATTERN.match(code.lower()):
|
||||
return False, (
|
||||
"Code must be 3-30 characters, start/end with alphanumeric, "
|
||||
"and contain only lowercase letters, digits, and hyphens"
|
||||
)
|
||||
return True, "Valid"
|
||||
|
||||
# ========================================================================
|
||||
# Link CRUD
|
||||
# ========================================================================
|
||||
|
||||
def get_link_by_code(self, code: str) -> Optional[ReferralLink]:
|
||||
"""Get a referral link by its code. Returns None if not found or disabled."""
|
||||
doc = self.db.collection(REFERRAL_LINKS_COLLECTION).document(code).get()
|
||||
if not doc.exists:
|
||||
return None
|
||||
data = doc.to_dict()
|
||||
if not data.get("enabled", True):
|
||||
return None
|
||||
return ReferralLink(**data)
|
||||
|
||||
def get_or_create_link(self, owner_email: str) -> ReferralLink:
|
||||
"""Get existing referral link for owner, or create a new one.
|
||||
|
||||
Retries up to 10 times if generated code collides with existing doc.
|
||||
"""
|
||||
# Check for existing link
|
||||
existing = (
|
||||
self.db.collection(REFERRAL_LINKS_COLLECTION)
|
||||
.where("owner_email", "==", owner_email)
|
||||
.limit(1)
|
||||
.stream()
|
||||
)
|
||||
for doc in existing:
|
||||
return ReferralLink(**doc.to_dict())
|
||||
|
||||
# Create new link with retry for code collision
|
||||
for attempt in range(10):
|
||||
code = self._generate_code()
|
||||
doc_ref = self.db.collection(REFERRAL_LINKS_COLLECTION).document(code)
|
||||
doc_snapshot = doc_ref.get()
|
||||
if doc_snapshot.exists:
|
||||
logger.warning("Code collision on attempt %d: %s", attempt + 1, code)
|
||||
continue
|
||||
|
||||
now = datetime.utcnow()
|
||||
link = ReferralLink(
|
||||
code=code,
|
||||
owner_email=owner_email,
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
)
|
||||
doc_ref.set(link.model_dump())
|
||||
return link
|
||||
|
||||
raise RuntimeError("Failed to generate unique referral code after 10 attempts")
|
||||
|
||||
def create_vanity_link(
|
||||
self,
|
||||
code: str,
|
||||
owner_email: str,
|
||||
display_name: Optional[str] = None,
|
||||
custom_message: Optional[str] = None,
|
||||
discount_percent: int = 10,
|
||||
kickback_percent: int = 20,
|
||||
discount_duration_days: int = 30,
|
||||
earning_duration_days: int = 365,
|
||||
) -> Tuple[bool, Optional[ReferralLink], str]:
|
||||
"""Create a vanity referral link.
|
||||
|
||||
Returns (success, link_or_none, message).
|
||||
"""
|
||||
valid, msg = self._validate_vanity_code(code)
|
||||
if not valid:
|
||||
return False, None, msg
|
||||
|
||||
normalized = code.lower()
|
||||
doc_ref = self.db.collection(REFERRAL_LINKS_COLLECTION).document(normalized)
|
||||
doc_snapshot = doc_ref.get()
|
||||
if doc_snapshot.exists:
|
||||
return False, None, f"Code '{normalized}' is already taken"
|
||||
|
||||
now = datetime.utcnow()
|
||||
link = ReferralLink(
|
||||
code=normalized,
|
||||
owner_email=owner_email,
|
||||
display_name=display_name,
|
||||
custom_message=custom_message,
|
||||
discount_percent=discount_percent,
|
||||
kickback_percent=kickback_percent,
|
||||
discount_duration_days=discount_duration_days,
|
||||
earning_duration_days=earning_duration_days,
|
||||
is_vanity=True,
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
)
|
||||
doc_ref.set(link.model_dump())
|
||||
return True, link, "Vanity link created"
|
||||
|
||||
def update_link(self, code: str, **updates) -> Tuple[bool, str]:
|
||||
"""Update fields on an existing referral link.
|
||||
|
||||
Returns (success, message).
|
||||
"""
|
||||
doc_ref = self.db.collection(REFERRAL_LINKS_COLLECTION).document(code)
|
||||
doc_snapshot = doc_ref.get()
|
||||
if not doc_snapshot.exists:
|
||||
return False, f"Link '{code}' not found"
|
||||
|
||||
updates["updated_at"] = datetime.utcnow()
|
||||
doc_ref.update(updates)
|
||||
return True, "Link updated"
|
||||
|
||||
def increment_clicks(self, code: str) -> None:
|
||||
"""Increment the click counter for a referral link."""
|
||||
doc_ref = self.db.collection(REFERRAL_LINKS_COLLECTION).document(code)
|
||||
doc_ref.update({"stats.clicks": firestore.Increment(1)})
|
||||
|
||||
# ========================================================================
|
||||
# Attribution
|
||||
# ========================================================================
|
||||
|
||||
def attribute_referral(self, referred_email: str, referral_code: str) -> Tuple[bool, str]:
|
||||
"""Attribute a referral to a user. Called during first login/signup."""
|
||||
referred_email = referred_email.lower()
|
||||
referral_code = referral_code.lower()
|
||||
|
||||
link = self.get_link_by_code(referral_code)
|
||||
if not link:
|
||||
return False, "Invalid or disabled referral code"
|
||||
|
||||
if link.owner_email == referred_email:
|
||||
return False, "Cannot use your own referral code"
|
||||
|
||||
# Increment signup count
|
||||
self.db.collection(REFERRAL_LINKS_COLLECTION).document(referral_code).update({
|
||||
"stats.signups": firestore.Increment(1),
|
||||
})
|
||||
|
||||
return True, "Referral attributed"
|
||||
|
||||
def get_attribution_data(self, referral_code: str) -> Optional[dict]:
|
||||
"""Get the data to set on user doc for referral attribution."""
|
||||
link = self.get_link_by_code(referral_code)
|
||||
if not link:
|
||||
return None
|
||||
now = datetime.utcnow()
|
||||
return {
|
||||
"referred_by_code": referral_code.lower(),
|
||||
"referred_at": now,
|
||||
"referral_discount_expires_at": now + timedelta(days=link.discount_duration_days),
|
||||
}
|
||||
|
||||
def get_or_create_stripe_coupon(self, discount_percent: int) -> Optional[str]:
|
||||
if not self.stripe_service:
|
||||
return None
|
||||
return self.stripe_service.get_or_create_referral_coupon(discount_percent)
|
||||
|
||||
def should_apply_discount(self, user_data: dict) -> bool:
|
||||
if not user_data.get("referred_by_code"):
|
||||
return False
|
||||
expires_at = user_data.get("referral_discount_expires_at")
|
||||
if not expires_at:
|
||||
return False
|
||||
if isinstance(expires_at, datetime):
|
||||
now = datetime.utcnow()
|
||||
# Firestore returns timezone-aware datetimes; ensure comparison is compatible
|
||||
if expires_at.tzinfo is not None:
|
||||
now = now.replace(tzinfo=expires_at.tzinfo)
|
||||
return expires_at > now
|
||||
return False
|
||||
|
||||
def get_discount_for_checkout(self, user_email: str) -> Optional[dict]:
|
||||
from backend.services.user_service import get_user_service
|
||||
user_service = get_user_service()
|
||||
user = user_service.get_user(user_email)
|
||||
if not user or not user.referred_by_code:
|
||||
return None
|
||||
user_data = {
|
||||
"referred_by_code": user.referred_by_code,
|
||||
"referral_discount_expires_at": user.referral_discount_expires_at,
|
||||
}
|
||||
if not self.should_apply_discount(user_data):
|
||||
return None
|
||||
link = self.get_link_by_code(user.referred_by_code)
|
||||
if not link:
|
||||
return None
|
||||
coupon_id = self.get_or_create_stripe_coupon(link.discount_percent)
|
||||
if not coupon_id:
|
||||
return None
|
||||
return {"coupon_id": coupon_id, "discount_percent": link.discount_percent}
|
||||
|
||||
# ========================================================================
|
||||
# Earnings
|
||||
# ========================================================================
|
||||
|
||||
def record_earning(self, referred_email: str, stripe_session_id: str, purchase_amount_cents: int) -> Optional[dict]:
|
||||
"""Record a referral earning after a credit purchase."""
|
||||
referred_email = referred_email.lower()
|
||||
|
||||
# Look up the referred user
|
||||
user_doc = self.db.collection("gen_users").document(referred_email).get()
|
||||
if not user_doc.exists:
|
||||
return None
|
||||
|
||||
user_data = user_doc.to_dict()
|
||||
referral_code = user_data.get("referred_by_code")
|
||||
if not referral_code:
|
||||
return None
|
||||
|
||||
referred_at = user_data.get("referred_at")
|
||||
if not referred_at:
|
||||
return None
|
||||
|
||||
link = self.get_link_by_code(referral_code)
|
||||
if not link:
|
||||
return None
|
||||
|
||||
# Check earning window
|
||||
earning_expires = referred_at + timedelta(days=link.earning_duration_days)
|
||||
if datetime.utcnow() > earning_expires:
|
||||
return None
|
||||
|
||||
# Calculate earning
|
||||
earning_amount = int(purchase_amount_cents * link.kickback_percent / 100)
|
||||
if earning_amount <= 0:
|
||||
return None
|
||||
|
||||
import uuid
|
||||
earning_id = str(uuid.uuid4())
|
||||
earning = ReferralEarning(
|
||||
id=earning_id,
|
||||
referrer_email=link.owner_email,
|
||||
referred_email=referred_email,
|
||||
referral_code=referral_code,
|
||||
stripe_session_id=stripe_session_id,
|
||||
purchase_amount_cents=purchase_amount_cents,
|
||||
earning_amount_cents=earning_amount,
|
||||
)
|
||||
|
||||
self.db.collection(REFERRAL_EARNINGS_COLLECTION).document(earning_id).set(
|
||||
earning.model_dump(mode="json")
|
||||
)
|
||||
|
||||
self.db.collection(REFERRAL_LINKS_COLLECTION).document(referral_code).update({
|
||||
"stats.purchases": firestore.Increment(1),
|
||||
"stats.total_earned_cents": firestore.Increment(earning_amount),
|
||||
})
|
||||
|
||||
logger.info(f"Referral earning recorded: ${earning_amount / 100:.2f} for {link.owner_email}")
|
||||
return {"earning_id": earning_id, "referrer_email": link.owner_email, "earning_amount_cents": earning_amount}
|
||||
|
||||
# ========================================================================
|
||||
# Stripe Connect & Payouts
|
||||
# ========================================================================
|
||||
|
||||
def create_connect_account(self, email: str) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""Create Stripe Connect account for a referrer."""
|
||||
if not self.stripe_service:
|
||||
return None, None
|
||||
return self.stripe_service.create_connect_account(email)
|
||||
|
||||
def get_connect_account_status(self, account_id: str) -> Optional[dict]:
|
||||
"""Get Stripe Connect account status."""
|
||||
if not self.stripe_service:
|
||||
return None
|
||||
return self.stripe_service.get_connect_account_status(account_id)
|
||||
|
||||
def create_connect_login_link(self, account_id: str) -> Optional[str]:
|
||||
"""Create login link for Stripe Express dashboard."""
|
||||
if not self.stripe_service:
|
||||
return None
|
||||
return self.stripe_service.create_connect_login_link(account_id)
|
||||
|
||||
def create_connect_update_link(self, account_id: str) -> Optional[str]:
|
||||
"""Create account link for updating Connect info."""
|
||||
if not self.stripe_service:
|
||||
return None
|
||||
return self.stripe_service.create_connect_update_link(account_id)
|
||||
|
||||
def get_pending_earnings(self, referrer_email: str) -> List[dict]:
|
||||
"""Get all pending earnings for a referrer."""
|
||||
query = (
|
||||
self.db.collection(REFERRAL_EARNINGS_COLLECTION)
|
||||
.where("referrer_email", "==", referrer_email.lower())
|
||||
.where("status", "==", "pending")
|
||||
)
|
||||
return [
|
||||
{**doc.to_dict(), "_ref": doc.reference}
|
||||
for doc in query.get()
|
||||
]
|
||||
|
||||
def check_and_trigger_payout(self, referrer_email: str, stripe_connect_account_id: str) -> bool:
|
||||
"""Check if pending earnings meet threshold and trigger payout."""
|
||||
pending = self.get_pending_earnings(referrer_email)
|
||||
total_pending = sum(e.get("earning_amount_cents", 0) for e in pending)
|
||||
|
||||
if total_pending < PAYOUT_THRESHOLD_CENTS:
|
||||
return False
|
||||
|
||||
if not self.stripe_service:
|
||||
return False
|
||||
|
||||
transfer_id = self.stripe_service.create_transfer(
|
||||
amount_cents=total_pending,
|
||||
destination_account_id=stripe_connect_account_id,
|
||||
)
|
||||
if not transfer_id:
|
||||
return False
|
||||
|
||||
import uuid
|
||||
payout_id = str(uuid.uuid4())
|
||||
earning_ids = [e["id"] for e in pending]
|
||||
|
||||
payout = ReferralPayout(
|
||||
id=payout_id,
|
||||
referrer_email=referrer_email.lower(),
|
||||
stripe_transfer_id=transfer_id,
|
||||
amount_cents=total_pending,
|
||||
earnings_included=earning_ids,
|
||||
)
|
||||
self.db.collection(REFERRAL_PAYOUTS_COLLECTION).document(payout_id).set(
|
||||
payout.model_dump(mode="json")
|
||||
)
|
||||
|
||||
for earning in pending:
|
||||
if "_ref" in earning:
|
||||
earning["_ref"].update({
|
||||
"status": "paid",
|
||||
"paid_at": datetime.utcnow(),
|
||||
"payout_id": payout_id,
|
||||
})
|
||||
|
||||
logger.info(f"Payout triggered: ${total_pending / 100:.2f} to {referrer_email} (transfer: {transfer_id})")
|
||||
return True
|
||||
|
||||
def get_dashboard_data(self, user_email: str) -> dict:
|
||||
"""Get full referral dashboard data for a user."""
|
||||
user_email = user_email.lower()
|
||||
link = self.get_or_create_link(user_email)
|
||||
|
||||
# Get recent earnings (last 20)
|
||||
earnings_query = (
|
||||
self.db.collection(REFERRAL_EARNINGS_COLLECTION)
|
||||
.where("referrer_email", "==", user_email)
|
||||
.order_by("created_at", direction=firestore.Query.DESCENDING)
|
||||
.limit(20)
|
||||
)
|
||||
earnings = [doc.to_dict() for doc in earnings_query.stream()]
|
||||
|
||||
# Get recent payouts (last 10)
|
||||
payouts_query = (
|
||||
self.db.collection(REFERRAL_PAYOUTS_COLLECTION)
|
||||
.where("referrer_email", "==", user_email)
|
||||
.order_by("created_at", direction=firestore.Query.DESCENDING)
|
||||
.limit(10)
|
||||
)
|
||||
payouts = [doc.to_dict() for doc in payouts_query.stream()]
|
||||
|
||||
# Compute pending balance from ALL pending earnings (not limited to recent)
|
||||
pending_earnings = self.get_pending_earnings(user_email)
|
||||
pending_balance = sum(e.get("earning_amount_cents", 0) for e in pending_earnings)
|
||||
total_earned = link.stats.total_earned_cents
|
||||
# total_paid = total_earned - pending (since total_earned tracks all time)
|
||||
total_paid = total_earned - pending_balance
|
||||
|
||||
# Check Connect status and fetch account details
|
||||
from backend.services.user_service import get_user_service
|
||||
user = get_user_service().get_user(user_email)
|
||||
has_connect = bool(user and user.stripe_connect_account_id)
|
||||
|
||||
connect_account = None
|
||||
if has_connect and self.stripe_service:
|
||||
connect_account = self.stripe_service.get_connect_account_status(
|
||||
user.stripe_connect_account_id
|
||||
)
|
||||
|
||||
return {
|
||||
"link": {
|
||||
"code": link.code,
|
||||
"display_name": link.display_name,
|
||||
"custom_message": link.custom_message,
|
||||
"discount_percent": link.discount_percent,
|
||||
"kickback_percent": link.kickback_percent,
|
||||
"discount_duration_days": link.discount_duration_days,
|
||||
"earning_duration_days": link.earning_duration_days,
|
||||
"stats": link.stats.model_dump(),
|
||||
"enabled": link.enabled,
|
||||
"is_vanity": link.is_vanity,
|
||||
},
|
||||
"pending_balance_cents": pending_balance,
|
||||
"total_earned_cents": total_earned,
|
||||
"total_paid_cents": total_paid,
|
||||
"recent_earnings": [
|
||||
{
|
||||
"id": e["id"],
|
||||
"referred_email": e.get("referred_email", ""),
|
||||
"purchase_amount_cents": e.get("purchase_amount_cents", 0),
|
||||
"earning_amount_cents": e.get("earning_amount_cents", 0),
|
||||
"status": e.get("status", "pending"),
|
||||
"created_at": str(e.get("created_at", "")),
|
||||
}
|
||||
for e in earnings
|
||||
],
|
||||
"recent_payouts": [
|
||||
{
|
||||
"id": p["id"],
|
||||
"amount_cents": p.get("amount_cents", 0),
|
||||
"status": p.get("status", "processing"),
|
||||
"created_at": str(p.get("created_at", "")),
|
||||
}
|
||||
for p in payouts
|
||||
],
|
||||
"stripe_connect_configured": has_connect,
|
||||
"stripe_connect_account": connect_account,
|
||||
}
|
||||
|
||||
def list_links(self, limit: int = 50, offset: int = 0) -> list[ReferralLink]:
|
||||
"""List referral links for admin view."""
|
||||
query = (
|
||||
self.db.collection(REFERRAL_LINKS_COLLECTION)
|
||||
.order_by("created_at", direction=firestore.Query.DESCENDING)
|
||||
.offset(offset)
|
||||
.limit(limit)
|
||||
)
|
||||
links = []
|
||||
for doc in query.stream():
|
||||
try:
|
||||
links.append(ReferralLink(**doc.to_dict()))
|
||||
except Exception:
|
||||
logger.warning("Skipping malformed referral link doc: %s", doc.id)
|
||||
return links
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Singleton accessor
|
||||
# ============================================================================
|
||||
|
||||
_referral_service = None
|
||||
|
||||
|
||||
def get_referral_service() -> ReferralService:
|
||||
global _referral_service
|
||||
if _referral_service is None:
|
||||
from backend.services.stripe_service import get_stripe_service
|
||||
_referral_service = ReferralService(stripe_service=get_stripe_service())
|
||||
return _referral_service
|
||||
@@ -0,0 +1,759 @@
|
||||
"""
|
||||
Stripe admin service for payment analytics and management.
|
||||
|
||||
Handles:
|
||||
- Storing enriched payment data from webhooks
|
||||
- Revenue analytics and reporting
|
||||
- Payment listing and detail views
|
||||
- Stripe balance and payout queries
|
||||
- Refund processing
|
||||
- Dispute monitoring
|
||||
- Webhook event logging
|
||||
"""
|
||||
import logging
|
||||
import time
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Optional, Dict, Any, List, Tuple
|
||||
|
||||
import stripe
|
||||
from google.cloud import firestore
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
STRIPE_PAYMENTS_COLLECTION = "stripe_payments"
|
||||
STRIPE_WEBHOOK_EVENTS_COLLECTION = "stripe_webhook_events"
|
||||
|
||||
|
||||
class StripeAdminService:
|
||||
"""Service for Stripe admin analytics and payment management."""
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
self.db = firestore.Client()
|
||||
self._revenue_cache: Optional[Dict[str, Any]] = None
|
||||
self._revenue_cache_time: float = 0
|
||||
self._revenue_cache_ttl: float = 300 # 5 minutes
|
||||
|
||||
# =========================================================================
|
||||
# Payment Storage
|
||||
# =========================================================================
|
||||
|
||||
def store_payment(self, session: Dict[str, Any]) -> bool:
|
||||
"""
|
||||
Store enriched payment data from a completed Checkout Session.
|
||||
|
||||
Args:
|
||||
session: Stripe Checkout Session object (expanded with payment_intent,
|
||||
charge, and balance_transaction)
|
||||
|
||||
Returns:
|
||||
True if stored successfully
|
||||
"""
|
||||
try:
|
||||
session_id = session.get("id", "")
|
||||
metadata = session.get("metadata", {})
|
||||
|
||||
# Extract payment intent and charge details
|
||||
payment_intent = session.get("payment_intent", {})
|
||||
if isinstance(payment_intent, str):
|
||||
# Not expanded - just store the ID
|
||||
payment_intent_id = payment_intent
|
||||
charge_data = {}
|
||||
balance_txn = {}
|
||||
else:
|
||||
payment_intent_id = payment_intent.get("id", "")
|
||||
charges = payment_intent.get("latest_charge", {})
|
||||
if isinstance(charges, str):
|
||||
charge_data = {}
|
||||
balance_txn = {}
|
||||
else:
|
||||
charge_data = charges or {}
|
||||
bt = charge_data.get("balance_transaction", {})
|
||||
balance_txn = bt if isinstance(bt, dict) else {}
|
||||
|
||||
# Extract payment method details
|
||||
payment_method_details = charge_data.get("payment_method_details", {})
|
||||
pm_type = payment_method_details.get("type", "unknown")
|
||||
card_info = payment_method_details.get("card", {})
|
||||
|
||||
# Determine order type
|
||||
order_type = metadata.get("order_type", "credit_purchase")
|
||||
package_id = metadata.get("package_id")
|
||||
credits = 0
|
||||
try:
|
||||
credits = int(metadata.get("credits", 0))
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
# Build product description
|
||||
if order_type == "made_for_you":
|
||||
artist = metadata.get("artist", "")
|
||||
title = metadata.get("title", "")
|
||||
product_description = f"Made For You: {artist} - {title}"
|
||||
else:
|
||||
from backend.services.stripe_service import CREDIT_PACKAGES
|
||||
pkg = CREDIT_PACKAGES.get(package_id, {})
|
||||
product_description = pkg.get("name", f"{credits} Credits")
|
||||
|
||||
# Extract discount info
|
||||
total_details = session.get("total_details", {})
|
||||
discount_amount = 0
|
||||
promotion_code = None
|
||||
if total_details:
|
||||
discount_obj = total_details.get("amount_discount", 0)
|
||||
discount_amount = discount_obj if isinstance(discount_obj, int) else 0
|
||||
|
||||
# Determine if test payment
|
||||
is_test = not (session.get("livemode", False))
|
||||
|
||||
payment_data = {
|
||||
"session_id": session_id,
|
||||
"payment_intent_id": payment_intent_id,
|
||||
"charge_id": charge_data.get("id", ""),
|
||||
# Financial (cents)
|
||||
"amount_total": session.get("amount_total", 0),
|
||||
"currency": session.get("currency", "usd"),
|
||||
"stripe_fee": balance_txn.get("fee", 0),
|
||||
"net_amount": balance_txn.get("net", 0),
|
||||
# Customer
|
||||
"customer_email": (
|
||||
metadata.get("user_email")
|
||||
or metadata.get("customer_email")
|
||||
or session.get("customer_email", "")
|
||||
).lower().strip(),
|
||||
"customer_name": session.get("customer_details", {}).get("name", ""),
|
||||
"stripe_customer_id": session.get("customer", ""),
|
||||
# Payment method
|
||||
"payment_method_type": pm_type,
|
||||
"card_brand": card_info.get("brand", ""),
|
||||
"card_last4": card_info.get("last4", ""),
|
||||
# Order details
|
||||
"order_type": order_type,
|
||||
"package_id": package_id,
|
||||
"credits_granted": credits,
|
||||
"product_description": product_description,
|
||||
# Made-for-you fields
|
||||
"artist": metadata.get("artist"),
|
||||
"title": metadata.get("title"),
|
||||
"job_id": metadata.get("job_id"),
|
||||
# Status
|
||||
"status": "succeeded",
|
||||
"refund_amount": 0,
|
||||
"refund_id": None,
|
||||
"refunded_at": None,
|
||||
"refund_reason": None,
|
||||
# Timestamps
|
||||
"created_at": datetime.utcfromtimestamp(session["created"]) if session.get("created") else datetime.utcnow(),
|
||||
"processed_at": datetime.utcnow(),
|
||||
# Metadata
|
||||
"is_test": is_test,
|
||||
"promotion_code": promotion_code,
|
||||
"discount_amount": discount_amount,
|
||||
}
|
||||
|
||||
doc_ref = self.db.collection(STRIPE_PAYMENTS_COLLECTION).document(session_id)
|
||||
doc_ref.set(payment_data)
|
||||
|
||||
# Invalidate revenue cache
|
||||
self._revenue_cache = None
|
||||
|
||||
logger.info(
|
||||
f"Stored payment data for session {session_id}: "
|
||||
f"${payment_data['amount_total'] / 100:.2f} from {payment_data['customer_email']}"
|
||||
)
|
||||
return True
|
||||
|
||||
except Exception:
|
||||
logger.exception(f"Error storing payment data for session {session.get('id', 'unknown')}")
|
||||
return False
|
||||
|
||||
def update_payment_status(self, session_id: str, updates: Dict[str, Any]) -> bool:
|
||||
"""Update payment status fields (for refunds/disputes)."""
|
||||
try:
|
||||
doc_ref = self.db.collection(STRIPE_PAYMENTS_COLLECTION).document(session_id)
|
||||
doc = doc_ref.get()
|
||||
if not doc.exists:
|
||||
logger.warning(f"Payment {session_id} not found for status update")
|
||||
return False
|
||||
|
||||
doc_ref.update(updates)
|
||||
self._revenue_cache = None
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception(f"Error updating payment status for {session_id}")
|
||||
return False
|
||||
|
||||
# =========================================================================
|
||||
# Revenue Analytics
|
||||
# =========================================================================
|
||||
|
||||
def get_revenue_summary(
|
||||
self, days: int = 30, exclude_test: bool = True
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Get revenue summary metrics.
|
||||
|
||||
Returns total gross, fees, net, refunds, count, AOV, revenue by product type.
|
||||
Uses in-memory cache with 5-minute TTL.
|
||||
"""
|
||||
cache_key = f"{days}_{exclude_test}"
|
||||
now = time.time()
|
||||
if (
|
||||
self._revenue_cache
|
||||
and self._revenue_cache.get("_key") == cache_key
|
||||
and (now - self._revenue_cache_time) < self._revenue_cache_ttl
|
||||
):
|
||||
return self._revenue_cache
|
||||
|
||||
try:
|
||||
collection = self.db.collection(STRIPE_PAYMENTS_COLLECTION)
|
||||
query = collection
|
||||
|
||||
if days > 0:
|
||||
cutoff = datetime.utcnow() - timedelta(days=days)
|
||||
query = query.where("created_at", ">=", cutoff)
|
||||
|
||||
total_gross = 0
|
||||
total_fees = 0
|
||||
total_net = 0
|
||||
total_refunds = 0
|
||||
count = 0
|
||||
by_type: Dict[str, int] = {}
|
||||
|
||||
for doc in query.stream():
|
||||
data = doc.to_dict()
|
||||
if exclude_test and data.get("is_test"):
|
||||
continue
|
||||
|
||||
amount = data.get("amount_total", 0)
|
||||
total_gross += amount
|
||||
total_fees += data.get("stripe_fee", 0)
|
||||
total_net += data.get("net_amount", 0)
|
||||
total_refunds += data.get("refund_amount", 0)
|
||||
count += 1
|
||||
|
||||
order_type = data.get("order_type", "credit_purchase")
|
||||
by_type[order_type] = by_type.get(order_type, 0) + amount
|
||||
|
||||
aov = total_gross / count if count > 0 else 0
|
||||
|
||||
result = {
|
||||
"_key": cache_key,
|
||||
"total_gross": total_gross,
|
||||
"total_fees": total_fees,
|
||||
"total_net": total_net,
|
||||
"total_refunds": total_refunds,
|
||||
"transaction_count": count,
|
||||
"average_order_value": round(aov),
|
||||
"revenue_by_type": by_type,
|
||||
}
|
||||
|
||||
self._revenue_cache = result
|
||||
self._revenue_cache_time = now
|
||||
return result
|
||||
|
||||
except Exception:
|
||||
logger.exception("Error calculating revenue summary")
|
||||
return {
|
||||
"total_gross": 0,
|
||||
"total_fees": 0,
|
||||
"total_net": 0,
|
||||
"total_refunds": 0,
|
||||
"transaction_count": 0,
|
||||
"average_order_value": 0,
|
||||
"revenue_by_type": {},
|
||||
}
|
||||
|
||||
def get_revenue_by_period(
|
||||
self,
|
||||
days: int = 30,
|
||||
group_by: str = "day",
|
||||
exclude_test: bool = True,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Get revenue grouped by time period for charting.
|
||||
|
||||
Args:
|
||||
days: Number of days to look back
|
||||
group_by: 'day' or 'week' or 'month'
|
||||
exclude_test: Exclude test transactions
|
||||
|
||||
Returns:
|
||||
List of {date, gross, net, fees, count} dicts
|
||||
"""
|
||||
try:
|
||||
collection = self.db.collection(STRIPE_PAYMENTS_COLLECTION)
|
||||
cutoff = datetime.utcnow() - timedelta(days=days)
|
||||
query = collection.where("created_at", ">=", cutoff)
|
||||
|
||||
buckets: Dict[str, Dict[str, Any]] = {}
|
||||
|
||||
for doc in query.stream():
|
||||
data = doc.to_dict()
|
||||
if exclude_test and data.get("is_test"):
|
||||
continue
|
||||
|
||||
created = data.get("created_at")
|
||||
if not created:
|
||||
continue
|
||||
if isinstance(created, str):
|
||||
try:
|
||||
created = datetime.fromisoformat(created.replace("Z", "+00:00"))
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
if group_by == "week":
|
||||
# Start of week (Monday)
|
||||
start = created - timedelta(days=created.weekday())
|
||||
key = start.strftime("%Y-%m-%d")
|
||||
elif group_by == "month":
|
||||
key = created.strftime("%Y-%m-01")
|
||||
else:
|
||||
key = created.strftime("%Y-%m-%d")
|
||||
|
||||
if key not in buckets:
|
||||
buckets[key] = {"date": key, "gross": 0, "net": 0, "fees": 0, "count": 0}
|
||||
|
||||
buckets[key]["gross"] += data.get("amount_total", 0)
|
||||
buckets[key]["net"] += data.get("net_amount", 0)
|
||||
buckets[key]["fees"] += data.get("stripe_fee", 0)
|
||||
buckets[key]["count"] += 1
|
||||
|
||||
result = sorted(buckets.values(), key=lambda x: x["date"])
|
||||
return result
|
||||
|
||||
except Exception:
|
||||
logger.exception("Error getting revenue by period")
|
||||
return []
|
||||
|
||||
# =========================================================================
|
||||
# Payment Listing & Details
|
||||
# =========================================================================
|
||||
|
||||
def list_payments(
|
||||
self,
|
||||
limit: int = 50,
|
||||
offset: int = 0,
|
||||
order_type: Optional[str] = None,
|
||||
status: Optional[str] = None,
|
||||
email: Optional[str] = None,
|
||||
exclude_test: bool = True,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
List payments with pagination and filters.
|
||||
|
||||
Returns:
|
||||
{payments: [...], total: int, has_more: bool}
|
||||
"""
|
||||
try:
|
||||
collection = self.db.collection(STRIPE_PAYMENTS_COLLECTION)
|
||||
query = collection.order_by("created_at", direction=firestore.Query.DESCENDING)
|
||||
|
||||
if order_type:
|
||||
query = collection.where("order_type", "==", order_type).order_by(
|
||||
"created_at", direction=firestore.Query.DESCENDING
|
||||
)
|
||||
if status:
|
||||
query = collection.where("status", "==", status).order_by(
|
||||
"created_at", direction=firestore.Query.DESCENDING
|
||||
)
|
||||
if email:
|
||||
query = collection.where("customer_email", "==", email.lower()).order_by(
|
||||
"created_at", direction=firestore.Query.DESCENDING
|
||||
)
|
||||
|
||||
# Stream and filter (Firestore doesn't support multiple inequality filters)
|
||||
payments = []
|
||||
skipped = 0
|
||||
total = 0
|
||||
|
||||
for doc in query.limit(1000).stream():
|
||||
data = doc.to_dict()
|
||||
data["session_id"] = doc.id
|
||||
|
||||
if exclude_test and data.get("is_test"):
|
||||
continue
|
||||
|
||||
# Apply filters that couldn't be done in query
|
||||
if order_type and data.get("order_type") != order_type:
|
||||
continue
|
||||
if status and data.get("status") != status:
|
||||
continue
|
||||
if email and data.get("customer_email", "").lower() != email.lower():
|
||||
continue
|
||||
|
||||
total += 1
|
||||
if skipped < offset:
|
||||
skipped += 1
|
||||
continue
|
||||
if len(payments) < limit:
|
||||
# Convert datetime objects for JSON serialization
|
||||
for key in ["created_at", "processed_at", "refunded_at"]:
|
||||
if isinstance(data.get(key), datetime):
|
||||
data[key] = data[key].isoformat()
|
||||
payments.append(data)
|
||||
|
||||
return {
|
||||
"payments": payments,
|
||||
"total": total,
|
||||
"has_more": total > offset + limit,
|
||||
}
|
||||
|
||||
except Exception:
|
||||
logger.exception("Error listing payments")
|
||||
return {"payments": [], "total": 0, "has_more": False}
|
||||
|
||||
def get_payment_detail(self, session_id: str) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Get full payment details for a single payment.
|
||||
|
||||
Combines Firestore data with on-demand Stripe API call for receipt_url.
|
||||
"""
|
||||
try:
|
||||
doc_ref = self.db.collection(STRIPE_PAYMENTS_COLLECTION).document(session_id)
|
||||
doc = doc_ref.get()
|
||||
if not doc.exists:
|
||||
return None
|
||||
|
||||
data = doc.to_dict()
|
||||
data["session_id"] = doc.id
|
||||
|
||||
# Convert datetime objects
|
||||
for key in ["created_at", "processed_at", "refunded_at"]:
|
||||
if isinstance(data.get(key), datetime):
|
||||
data[key] = data[key].isoformat()
|
||||
|
||||
# On-demand: fetch receipt URL from Stripe
|
||||
charge_id = data.get("charge_id")
|
||||
if charge_id:
|
||||
try:
|
||||
charge = stripe.Charge.retrieve(charge_id)
|
||||
data["receipt_url"] = charge.get("receipt_url")
|
||||
data["stripe_dashboard_url"] = (
|
||||
f"https://dashboard.stripe.com/payments/{data.get('payment_intent_id', '')}"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not fetch charge {charge_id}: {e}")
|
||||
data["receipt_url"] = None
|
||||
|
||||
return data
|
||||
|
||||
except Exception:
|
||||
logger.exception(f"Error getting payment detail for {session_id}")
|
||||
return None
|
||||
|
||||
def get_user_payment_history(self, email: str) -> Dict[str, Any]:
|
||||
"""Get all payments for a user with total_spent."""
|
||||
try:
|
||||
collection = self.db.collection(STRIPE_PAYMENTS_COLLECTION)
|
||||
# Try with order_by first (requires composite index on customer_email + created_at)
|
||||
try:
|
||||
query = (
|
||||
collection.where("customer_email", "==", email.lower())
|
||||
.order_by("created_at", direction=firestore.Query.DESCENDING)
|
||||
)
|
||||
docs = list(query.stream())
|
||||
except Exception:
|
||||
# Fallback: query without order_by if composite index doesn't exist yet
|
||||
logger.warning(
|
||||
f"Composite index missing for stripe_payments(customer_email, created_at). "
|
||||
f"Falling back to unordered query for {email}"
|
||||
)
|
||||
query = collection.where("customer_email", "==", email.lower())
|
||||
docs = list(query.stream())
|
||||
|
||||
payments = []
|
||||
total_spent = 0
|
||||
total_refunded = 0
|
||||
|
||||
for doc in docs:
|
||||
data = doc.to_dict()
|
||||
data["session_id"] = doc.id
|
||||
total_spent += data.get("amount_total", 0)
|
||||
total_refunded += data.get("refund_amount", 0)
|
||||
|
||||
for key in ["created_at", "processed_at", "refunded_at"]:
|
||||
if isinstance(data.get(key), datetime):
|
||||
data[key] = data[key].isoformat()
|
||||
|
||||
payments.append(data)
|
||||
|
||||
# Sort by created_at descending (handles both indexed and fallback paths)
|
||||
payments.sort(key=lambda p: p.get("created_at", ""), reverse=True)
|
||||
|
||||
first_payment = payments[-1]["created_at"] if payments else None
|
||||
last_payment = payments[0]["created_at"] if payments else None
|
||||
|
||||
return {
|
||||
"email": email,
|
||||
"payments": payments,
|
||||
"total_spent": total_spent,
|
||||
"total_refunded": total_refunded,
|
||||
"net_spent": total_spent - total_refunded,
|
||||
"payment_count": len(payments),
|
||||
"first_payment_at": first_payment,
|
||||
"last_payment_at": last_payment,
|
||||
}
|
||||
|
||||
except Exception:
|
||||
logger.exception(f"Error getting payment history for {email}")
|
||||
return {
|
||||
"email": email,
|
||||
"payments": [],
|
||||
"total_spent": 0,
|
||||
"total_refunded": 0,
|
||||
"net_spent": 0,
|
||||
"payment_count": 0,
|
||||
"first_payment_at": None,
|
||||
"last_payment_at": None,
|
||||
}
|
||||
|
||||
# =========================================================================
|
||||
# Stripe Balance & Payouts
|
||||
# =========================================================================
|
||||
|
||||
def get_stripe_balance(self) -> Dict[str, Any]:
|
||||
"""Get current Stripe account balance (available + pending)."""
|
||||
try:
|
||||
balance = stripe.Balance.retrieve()
|
||||
available = sum(b.get("amount", 0) for b in balance.get("available", []))
|
||||
pending = sum(b.get("amount", 0) for b in balance.get("pending", []))
|
||||
return {
|
||||
"available": available,
|
||||
"pending": pending,
|
||||
"currency": "usd",
|
||||
}
|
||||
except Exception:
|
||||
logger.exception("Error fetching Stripe balance")
|
||||
return {"available": 0, "pending": 0, "currency": "usd"}
|
||||
|
||||
def get_recent_payouts(self, limit: int = 10) -> List[Dict[str, Any]]:
|
||||
"""Get recent payouts from Stripe."""
|
||||
try:
|
||||
payouts = stripe.Payout.list(limit=limit)
|
||||
result = []
|
||||
for p in payouts.get("data", []):
|
||||
result.append({
|
||||
"id": p.get("id"),
|
||||
"amount": p.get("amount"),
|
||||
"currency": p.get("currency"),
|
||||
"status": p.get("status"),
|
||||
"arrival_date": p.get("arrival_date"),
|
||||
"created": p.get("created"),
|
||||
"description": p.get("description"),
|
||||
"method": p.get("method"),
|
||||
})
|
||||
return result
|
||||
except Exception:
|
||||
logger.exception("Error fetching payouts")
|
||||
return []
|
||||
|
||||
# =========================================================================
|
||||
# Disputes
|
||||
# =========================================================================
|
||||
|
||||
def get_recent_disputes(self) -> List[Dict[str, Any]]:
|
||||
"""Get open and recent disputes from Stripe."""
|
||||
try:
|
||||
disputes = stripe.Dispute.list(limit=20)
|
||||
result = []
|
||||
for d in disputes.get("data", []):
|
||||
result.append({
|
||||
"id": d.get("id"),
|
||||
"amount": d.get("amount"),
|
||||
"currency": d.get("currency"),
|
||||
"status": d.get("status"),
|
||||
"reason": d.get("reason"),
|
||||
"charge_id": d.get("charge"),
|
||||
"created": d.get("created"),
|
||||
"evidence_due_by": d.get("evidence_details", {}).get("due_by"),
|
||||
"payment_intent_id": d.get("payment_intent"),
|
||||
})
|
||||
return result
|
||||
except Exception:
|
||||
logger.exception("Error fetching disputes")
|
||||
return []
|
||||
|
||||
# =========================================================================
|
||||
# Refund Processing
|
||||
# =========================================================================
|
||||
|
||||
def process_refund(
|
||||
self,
|
||||
session_id: str,
|
||||
amount: Optional[int] = None,
|
||||
reason: str = "requested_by_customer",
|
||||
) -> Tuple[bool, str]:
|
||||
"""
|
||||
Process a refund for a payment.
|
||||
|
||||
Args:
|
||||
session_id: Checkout session ID
|
||||
amount: Refund amount in cents (None for full refund)
|
||||
reason: Refund reason
|
||||
|
||||
Returns:
|
||||
(success, message)
|
||||
"""
|
||||
try:
|
||||
# Get payment record
|
||||
doc_ref = self.db.collection(STRIPE_PAYMENTS_COLLECTION).document(session_id)
|
||||
doc = doc_ref.get()
|
||||
if not doc.exists:
|
||||
return False, "Payment not found"
|
||||
|
||||
data = doc.to_dict()
|
||||
payment_intent_id = data.get("payment_intent_id")
|
||||
if not payment_intent_id:
|
||||
return False, "No payment intent ID found"
|
||||
|
||||
current_refund = data.get("refund_amount", 0)
|
||||
original_amount = data.get("amount_total", 0)
|
||||
refund_amount = amount or (original_amount - current_refund)
|
||||
|
||||
if refund_amount <= 0:
|
||||
return False, "Invalid refund amount"
|
||||
if current_refund + refund_amount > original_amount:
|
||||
return False, f"Refund would exceed original amount (${original_amount / 100:.2f})"
|
||||
|
||||
# Create Stripe refund
|
||||
refund_params: Dict[str, Any] = {
|
||||
"payment_intent": payment_intent_id,
|
||||
"amount": refund_amount,
|
||||
"reason": reason,
|
||||
}
|
||||
refund = stripe.Refund.create(**refund_params)
|
||||
|
||||
# Update Firestore record
|
||||
new_total_refund = current_refund + refund_amount
|
||||
new_status = "refunded" if new_total_refund >= original_amount else "partially_refunded"
|
||||
|
||||
doc_ref.update({
|
||||
"status": new_status,
|
||||
"refund_amount": new_total_refund,
|
||||
"refund_id": refund.get("id"),
|
||||
"refunded_at": datetime.utcnow(),
|
||||
"refund_reason": reason,
|
||||
})
|
||||
|
||||
self._revenue_cache = None
|
||||
|
||||
logger.info(
|
||||
f"Processed refund for {session_id}: "
|
||||
f"${refund_amount / 100:.2f} ({new_status})"
|
||||
)
|
||||
return True, f"Refund of ${refund_amount / 100:.2f} processed successfully"
|
||||
|
||||
except stripe.error.StripeError as e:
|
||||
logger.error(f"Stripe refund error for {session_id}: {e}")
|
||||
return False, f"Stripe error: {str(e)}"
|
||||
except Exception:
|
||||
logger.exception(f"Error processing refund for {session_id}")
|
||||
return False, "Unexpected error processing refund"
|
||||
|
||||
# =========================================================================
|
||||
# Webhook Event Logging
|
||||
# =========================================================================
|
||||
|
||||
def log_webhook_event(
|
||||
self,
|
||||
event_id: str,
|
||||
event_type: str,
|
||||
status: str = "processing",
|
||||
session_id: Optional[str] = None,
|
||||
customer_email: Optional[str] = None,
|
||||
summary: Optional[str] = None,
|
||||
error_message: Optional[str] = None,
|
||||
) -> None:
|
||||
"""Log a webhook event for audit trail."""
|
||||
try:
|
||||
doc_ref = self.db.collection(STRIPE_WEBHOOK_EVENTS_COLLECTION).document(event_id)
|
||||
data = {
|
||||
"event_id": event_id,
|
||||
"event_type": event_type,
|
||||
"created_at": datetime.utcnow(),
|
||||
"status": status,
|
||||
"session_id": session_id,
|
||||
"customer_email": customer_email,
|
||||
"summary": summary,
|
||||
"error_message": error_message,
|
||||
}
|
||||
if status in ("success", "error"):
|
||||
data["processed_at"] = datetime.utcnow()
|
||||
doc_ref.set(data, merge=True)
|
||||
except Exception:
|
||||
logger.exception(f"Error logging webhook event {event_id}")
|
||||
|
||||
def list_webhook_events(
|
||||
self,
|
||||
limit: int = 50,
|
||||
event_type: Optional[str] = None,
|
||||
status: Optional[str] = None,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""List recent webhook events."""
|
||||
try:
|
||||
collection = self.db.collection(STRIPE_WEBHOOK_EVENTS_COLLECTION)
|
||||
query = collection.order_by("created_at", direction=firestore.Query.DESCENDING)
|
||||
|
||||
if event_type:
|
||||
query = collection.where("event_type", "==", event_type).order_by(
|
||||
"created_at", direction=firestore.Query.DESCENDING
|
||||
)
|
||||
|
||||
events = []
|
||||
for doc in query.limit(limit).stream():
|
||||
data = doc.to_dict()
|
||||
if status and data.get("status") != status:
|
||||
continue
|
||||
for key in ["created_at", "processed_at"]:
|
||||
if isinstance(data.get(key), datetime):
|
||||
data[key] = data[key].isoformat()
|
||||
events.append(data)
|
||||
|
||||
return events
|
||||
|
||||
except Exception:
|
||||
logger.exception("Error listing webhook events")
|
||||
return []
|
||||
|
||||
def normalize_customer_emails(self) -> Dict[str, Any]:
|
||||
"""Backfill: lowercase all customer_email values in stripe_payments."""
|
||||
try:
|
||||
collection = self.db.collection(STRIPE_PAYMENTS_COLLECTION)
|
||||
updated = 0
|
||||
skipped = 0
|
||||
|
||||
for doc in collection.stream():
|
||||
data = doc.to_dict()
|
||||
email = data.get("customer_email", "")
|
||||
normalized = email.lower().strip()
|
||||
if email != normalized:
|
||||
doc.reference.update({"customer_email": normalized})
|
||||
updated += 1
|
||||
else:
|
||||
skipped += 1
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"updated": updated,
|
||||
"skipped": skipped,
|
||||
"total": updated + skipped,
|
||||
}
|
||||
except Exception:
|
||||
logger.exception("Error normalizing customer emails")
|
||||
raise
|
||||
|
||||
|
||||
# Global instance
|
||||
_stripe_admin_service = None
|
||||
|
||||
|
||||
def get_stripe_admin_service() -> StripeAdminService:
|
||||
"""Get the global Stripe admin service instance."""
|
||||
global _stripe_admin_service
|
||||
if _stripe_admin_service is None:
|
||||
_stripe_admin_service = StripeAdminService()
|
||||
return _stripe_admin_service
|
||||
@@ -6,6 +6,7 @@ Handles:
|
||||
- Processing webhook events
|
||||
- Managing customer records
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from typing import Optional, Dict, Any, Tuple
|
||||
@@ -22,25 +23,25 @@ logger = logging.getLogger(__name__)
|
||||
CREDIT_PACKAGES = {
|
||||
"1_credit": {
|
||||
"credits": 1,
|
||||
"price_cents": 500, # $5.00
|
||||
"price_cents": 1000, # $10.00
|
||||
"name": "1 Karaoke Credit",
|
||||
"description": "Create 1 professional karaoke video",
|
||||
},
|
||||
"3_credits": {
|
||||
"credits": 3,
|
||||
"price_cents": 1200, # $12.00 (20% discount)
|
||||
"price_cents": 2400, # $24.00 (20% discount)
|
||||
"name": "3 Karaoke Credits",
|
||||
"description": "Create 3 professional karaoke videos (Save 20%)",
|
||||
},
|
||||
"5_credits": {
|
||||
"credits": 5,
|
||||
"price_cents": 1750, # $17.50 (30% discount)
|
||||
"price_cents": 3500, # $35.00 (30% discount)
|
||||
"name": "5 Karaoke Credits",
|
||||
"description": "Create 5 professional karaoke videos (Save 30%)",
|
||||
},
|
||||
"10_credits": {
|
||||
"credits": 10,
|
||||
"price_cents": 3000, # $30.00 (40% discount)
|
||||
"price_cents": 6000, # $60.00 (40% discount)
|
||||
"name": "10 Karaoke Credits",
|
||||
"description": "Create 10 professional karaoke videos (Save 40%)",
|
||||
},
|
||||
@@ -48,7 +49,7 @@ CREDIT_PACKAGES = {
|
||||
|
||||
# Made-for-you service package (not a credit package - creates a job directly)
|
||||
MADE_FOR_YOU_PACKAGE = {
|
||||
"price_cents": 1500, # $15.00
|
||||
"price_cents": 5000, # $50.00
|
||||
"name": "Made For You Karaoke Video",
|
||||
"description": "Professional 4K karaoke video with perfectly synced lyrics, delivered to your email within 24 hours",
|
||||
"images": ["https://gen.nomadkaraoke.com/nomad-logo.png"],
|
||||
@@ -96,6 +97,7 @@ class StripeService:
|
||||
user_email: str,
|
||||
success_url: Optional[str] = None,
|
||||
cancel_url: Optional[str] = None,
|
||||
coupon_id: Optional[str] = None,
|
||||
) -> Tuple[bool, Optional[str], str]:
|
||||
"""
|
||||
Create a Stripe Checkout session for purchasing credits.
|
||||
@@ -151,6 +153,11 @@ class StripeService:
|
||||
'allow_promotion_codes': True,
|
||||
}
|
||||
|
||||
# If a referral coupon is applied, use discounts instead of allow_promotion_codes
|
||||
if coupon_id:
|
||||
session_params.pop('allow_promotion_codes', None)
|
||||
session_params['discounts'] = [{'coupon': coupon_id}]
|
||||
|
||||
# Add payment method configuration if set (enables Google Pay, Link, etc.)
|
||||
if self.payment_method_config:
|
||||
session_params['payment_method_configuration'] = self.payment_method_config
|
||||
@@ -284,7 +291,10 @@ class StripeService:
|
||||
event = stripe.Webhook.construct_event(
|
||||
payload, signature, self.webhook_secret
|
||||
)
|
||||
return True, event, "Webhook verified"
|
||||
# Convert Stripe Event object to plain dict — Stripe SDK v14+
|
||||
# returns StripeObject instances that don't support .get()
|
||||
event_dict = json.loads(str(event))
|
||||
return True, event_dict, "Webhook verified"
|
||||
except stripe.error.SignatureVerificationError as e:
|
||||
logger.error(f"Invalid webhook signature: {e}")
|
||||
return False, None, "Invalid signature"
|
||||
@@ -351,6 +361,112 @@ class StripeService:
|
||||
logger.error(f"Error retrieving session {session_id}: {e}")
|
||||
return None
|
||||
|
||||
def get_or_create_referral_coupon(self, discount_percent: int) -> Optional[str]:
|
||||
"""Get or create a Stripe coupon for a referral discount percentage."""
|
||||
if not self.is_configured():
|
||||
return None
|
||||
coupon_id = f"referral-{discount_percent}pct"
|
||||
try:
|
||||
stripe.Coupon.retrieve(coupon_id)
|
||||
return coupon_id
|
||||
except stripe.error.InvalidRequestError:
|
||||
try:
|
||||
stripe.Coupon.create(
|
||||
id=coupon_id,
|
||||
percent_off=discount_percent,
|
||||
duration="once",
|
||||
name=f"Referral {discount_percent}% off",
|
||||
)
|
||||
logger.info(f"Created Stripe coupon: {coupon_id}")
|
||||
return coupon_id
|
||||
except stripe.error.StripeError as e:
|
||||
logger.error(f"Failed to create coupon {coupon_id}: {e}")
|
||||
return None
|
||||
|
||||
def create_connect_account(self, email: str) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""Create a Stripe Connect Express account. Returns (account_id, onboarding_url)."""
|
||||
if not self.is_configured():
|
||||
return None, None
|
||||
try:
|
||||
account = stripe.Account.create(
|
||||
type="express",
|
||||
email=email,
|
||||
capabilities={"transfers": {"requested": True}},
|
||||
metadata={"source": "nomad_karaoke_referrals"},
|
||||
)
|
||||
account_link = stripe.AccountLink.create(
|
||||
account=account.id,
|
||||
refresh_url=f"{self.frontend_url}/app/referrals?connect=refresh",
|
||||
return_url=f"{self.frontend_url}/app/referrals?connect=complete",
|
||||
type="account_onboarding",
|
||||
)
|
||||
logger.info(f"Created Connect account {account.id} for {email}")
|
||||
return account.id, account_link.url
|
||||
except stripe.error.StripeError as e:
|
||||
logger.error(f"Failed to create Connect account for {email}: {e}")
|
||||
return None, None
|
||||
|
||||
def get_connect_account_status(self, account_id: str) -> Optional[dict]:
|
||||
"""Retrieve Stripe Connect account status. Returns dict with account details."""
|
||||
if not self.is_configured():
|
||||
return None
|
||||
try:
|
||||
account = stripe.Account.retrieve(account_id)
|
||||
return {
|
||||
"account_id": account.id,
|
||||
"charges_enabled": account.charges_enabled,
|
||||
"payouts_enabled": account.payouts_enabled,
|
||||
"details_submitted": account.details_submitted,
|
||||
"email": account.email,
|
||||
}
|
||||
except stripe.error.StripeError as e:
|
||||
logger.error(f"Failed to retrieve Connect account {account_id}: {e}")
|
||||
return None
|
||||
|
||||
def create_connect_login_link(self, account_id: str) -> Optional[str]:
|
||||
"""Create a login link to the Stripe Express dashboard. Returns URL."""
|
||||
if not self.is_configured():
|
||||
return None
|
||||
try:
|
||||
link = stripe.Account.create_login_link(account_id)
|
||||
return link.url
|
||||
except stripe.error.StripeError as e:
|
||||
logger.error(f"Failed to create login link for {account_id}: {e}")
|
||||
return None
|
||||
|
||||
def create_connect_update_link(self, account_id: str) -> Optional[str]:
|
||||
"""Create an account link for updating Connect account info. Returns URL."""
|
||||
if not self.is_configured():
|
||||
return None
|
||||
try:
|
||||
account_link = stripe.AccountLink.create(
|
||||
account=account_id,
|
||||
refresh_url=f"{self.frontend_url}/app/referrals?connect=refresh",
|
||||
return_url=f"{self.frontend_url}/app/referrals?connect=complete",
|
||||
type="account_onboarding",
|
||||
)
|
||||
return account_link.url
|
||||
except stripe.error.StripeError as e:
|
||||
logger.error(f"Failed to create update link for {account_id}: {e}")
|
||||
return None
|
||||
|
||||
def create_transfer(self, amount_cents: int, destination_account_id: str, description: str = "Nomad Karaoke referral payout") -> Optional[str]:
|
||||
"""Create a transfer to a Connect account. Returns transfer ID."""
|
||||
if not self.is_configured():
|
||||
return None
|
||||
try:
|
||||
transfer = stripe.Transfer.create(
|
||||
amount=amount_cents,
|
||||
currency="usd",
|
||||
destination=destination_account_id,
|
||||
description=description,
|
||||
)
|
||||
logger.info(f"Created transfer {transfer.id}: ${amount_cents / 100:.2f} to {destination_account_id}")
|
||||
return transfer.id
|
||||
except stripe.error.StripeError as e:
|
||||
logger.error(f"Failed to create transfer to {destination_account_id}: {e}")
|
||||
return None
|
||||
|
||||
def create_customer(self, email: str, name: Optional[str] = None) -> Optional[str]:
|
||||
"""
|
||||
Create or get a Stripe customer for a user.
|
||||
|
||||
@@ -14,6 +14,7 @@ import re
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
from backend.config import get_settings
|
||||
from backend.i18n import t
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -79,6 +80,18 @@ Thanks!
|
||||
"""
|
||||
|
||||
|
||||
DEFAULT_YOUTUBE_UPLOAD_COMPLETE_TEMPLATE = """Hi {name},
|
||||
|
||||
Great news! Your karaoke video for "{artist} - {title}" has been uploaded to YouTube:
|
||||
{youtube_url}
|
||||
|
||||
This upload was briefly delayed while our systems processed other uploads, but everything is all set now.
|
||||
|
||||
Thanks!
|
||||
-Andrew
|
||||
"""
|
||||
|
||||
|
||||
class TemplateService:
|
||||
"""
|
||||
Service for fetching and rendering email templates from GCS.
|
||||
@@ -142,29 +155,48 @@ class TemplateService:
|
||||
logger.error(f"Failed to fetch template {blob_path}: {e}")
|
||||
return None
|
||||
|
||||
def get_job_completion_template(self) -> str:
|
||||
def get_job_completion_template(self, locale: str = "en") -> str:
|
||||
"""Get the job completion email template."""
|
||||
# Try locale-specific GCS template first, then default GCS, then localized default
|
||||
if locale != "en":
|
||||
template = self._fetch_template_from_gcs(f"{locale}/job-completion.txt")
|
||||
if template is not None:
|
||||
return template
|
||||
template = self._fetch_template_from_gcs("job-completion.txt")
|
||||
if template is None:
|
||||
logger.info("Using default job completion template")
|
||||
if template is not None:
|
||||
return template
|
||||
logger.info("Using default job completion template")
|
||||
if locale == "en":
|
||||
return DEFAULT_JOB_COMPLETION_TEMPLATE
|
||||
return template
|
||||
return self._build_localized_job_completion_template(locale)
|
||||
|
||||
def get_action_needed_lyrics_template(self) -> str:
|
||||
def get_action_needed_lyrics_template(self, locale: str = "en") -> str:
|
||||
"""Get the lyrics review reminder template."""
|
||||
if locale != "en":
|
||||
template = self._fetch_template_from_gcs(f"{locale}/action-needed-lyrics.txt")
|
||||
if template is not None:
|
||||
return template
|
||||
template = self._fetch_template_from_gcs("action-needed-lyrics.txt")
|
||||
if template is None:
|
||||
logger.info("Using default lyrics reminder template")
|
||||
if template is not None:
|
||||
return template
|
||||
logger.info("Using default lyrics reminder template")
|
||||
if locale == "en":
|
||||
return DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE
|
||||
return template
|
||||
return self._build_localized_action_lyrics_template(locale)
|
||||
|
||||
def get_action_needed_instrumental_template(self) -> str:
|
||||
def get_action_needed_instrumental_template(self, locale: str = "en") -> str:
|
||||
"""Get the instrumental selection reminder template."""
|
||||
if locale != "en":
|
||||
template = self._fetch_template_from_gcs(f"{locale}/action-needed-instrumental.txt")
|
||||
if template is not None:
|
||||
return template
|
||||
template = self._fetch_template_from_gcs("action-needed-instrumental.txt")
|
||||
if template is None:
|
||||
logger.info("Using default instrumental reminder template")
|
||||
if template is not None:
|
||||
return template
|
||||
logger.info("Using default instrumental reminder template")
|
||||
if locale == "en":
|
||||
return DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE
|
||||
return template
|
||||
return self._build_localized_action_instrumental_template(locale)
|
||||
|
||||
def render_template(self, template: str, variables: Dict[str, Any]) -> str:
|
||||
"""
|
||||
@@ -203,6 +235,74 @@ class TemplateService:
|
||||
|
||||
return result.strip()
|
||||
|
||||
def _build_localized_job_completion_template(self, locale: str) -> str:
|
||||
"""Build a localized job completion template from translation keys."""
|
||||
ns = "emails.templates.jobCompletion"
|
||||
return f"""{t(locale, f"{ns}.greeting")}
|
||||
|
||||
{t(locale, f"{ns}.thanks")}
|
||||
|
||||
{t(locale, f"{ns}.youtubeLink")}
|
||||
{{youtube_url}}
|
||||
|
||||
{t(locale, f"{ns}.dropboxIntro")}
|
||||
{t(locale, f"{ns}.fileList")}
|
||||
|
||||
{{dropbox_url}}
|
||||
|
||||
{t(locale, f"{ns}.fixOffer")}
|
||||
|
||||
{t(locale, f"{ns}.feedbackRequest")}
|
||||
{{feedback_url}}
|
||||
|
||||
{t(locale, f"{ns}.signOff")}
|
||||
"""
|
||||
|
||||
def _build_localized_action_lyrics_template(self, locale: str) -> str:
|
||||
"""Build a localized action-needed lyrics template."""
|
||||
ns = "emails.templates.actionLyrics"
|
||||
return f"""{t(locale, f"{ns}.greeting")}
|
||||
|
||||
{t(locale, f"{ns}.ready")}
|
||||
|
||||
{t(locale, f"{ns}.instructions")}
|
||||
|
||||
{t(locale, f"{ns}.reviewPrompt")}
|
||||
{{review_url}}
|
||||
|
||||
{t(locale, f"{ns}.duration")}
|
||||
|
||||
{t(locale, f"{ns}.signOff")}
|
||||
"""
|
||||
|
||||
def _build_localized_action_instrumental_template(self, locale: str) -> str:
|
||||
"""Build a localized action-needed instrumental template."""
|
||||
ns = "emails.templates.actionInstrumental"
|
||||
return f"""{t(locale, f"{ns}.greeting")}
|
||||
|
||||
{t(locale, f"{ns}.almostDone")}
|
||||
|
||||
{t(locale, f"{ns}.instructions")}
|
||||
|
||||
{t(locale, f"{ns}.selectPrompt")}
|
||||
{{instrumental_url}}
|
||||
|
||||
{t(locale, f"{ns}.signOff")}
|
||||
"""
|
||||
|
||||
def _build_localized_youtube_complete_template(self, locale: str) -> str:
|
||||
"""Build a localized YouTube upload complete template."""
|
||||
ns = "emails.templates.youtubeComplete"
|
||||
return f"""{t(locale, f"{ns}.greeting")}
|
||||
|
||||
{t(locale, f"{ns}.greatNews")}
|
||||
{{youtube_url}}
|
||||
|
||||
{t(locale, f"{ns}.delayNote")}
|
||||
|
||||
{t(locale, f"{ns}.signOff")}
|
||||
"""
|
||||
|
||||
def render_job_completion(
|
||||
self,
|
||||
name: Optional[str] = None,
|
||||
@@ -212,6 +312,9 @@ class TemplateService:
|
||||
title: Optional[str] = None,
|
||||
job_id: Optional[str] = None,
|
||||
feedback_url: Optional[str] = None,
|
||||
is_private: bool = False,
|
||||
youtube_queued: bool = False,
|
||||
locale: str = "en",
|
||||
) -> str:
|
||||
"""
|
||||
Render the job completion email template.
|
||||
@@ -224,11 +327,36 @@ class TemplateService:
|
||||
title: Song title
|
||||
job_id: Job ID
|
||||
feedback_url: Feedback form URL (optional)
|
||||
is_private: If True, remove YouTube section entirely (private/non-published tracks)
|
||||
youtube_queued: If True, YouTube upload was deferred - show pending message
|
||||
|
||||
Returns:
|
||||
Rendered email content
|
||||
"""
|
||||
template = self.get_job_completion_template()
|
||||
template = self.get_job_completion_template(locale)
|
||||
|
||||
# For private tracks, remove the YouTube section entirely
|
||||
if is_private:
|
||||
template = re.sub(
|
||||
r"\n*Here's the link for the karaoke video published to YouTube:\n\{youtube_url\}\n*",
|
||||
'\n',
|
||||
template,
|
||||
)
|
||||
elif youtube_queued and not youtube_url:
|
||||
# Replace YouTube URL with pending message
|
||||
queued_msg = t(locale, "emails.templates.jobCompletion.youtubeQueued")
|
||||
youtube_link_text = t(locale, "emails.templates.jobCompletion.youtubeLink")
|
||||
template = template.replace(
|
||||
f"{youtube_link_text}\n{{youtube_url}}",
|
||||
queued_msg
|
||||
)
|
||||
# Also handle the English default template pattern
|
||||
if "{youtube_url}" in template:
|
||||
template = template.replace(
|
||||
"Here's the link for the karaoke video published to YouTube:\n{youtube_url}",
|
||||
queued_msg
|
||||
)
|
||||
|
||||
variables = {
|
||||
"name": name or "there",
|
||||
"youtube_url": youtube_url or "[YouTube URL not available]",
|
||||
@@ -246,6 +374,7 @@ class TemplateService:
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
review_url: str = "",
|
||||
locale: str = "en",
|
||||
) -> str:
|
||||
"""
|
||||
Render the combined review reminder template.
|
||||
@@ -262,7 +391,7 @@ class TemplateService:
|
||||
Returns:
|
||||
Rendered email content
|
||||
"""
|
||||
template = self.get_action_needed_lyrics_template()
|
||||
template = self.get_action_needed_lyrics_template(locale)
|
||||
variables = {
|
||||
"name": name or "there",
|
||||
"artist": artist or "Unknown Artist",
|
||||
@@ -277,6 +406,7 @@ class TemplateService:
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
instrumental_url: str = "",
|
||||
locale: str = "en",
|
||||
) -> str:
|
||||
"""
|
||||
Render the instrumental selection reminder template.
|
||||
@@ -294,7 +424,7 @@ class TemplateService:
|
||||
Returns:
|
||||
Rendered email content
|
||||
"""
|
||||
template = self.get_action_needed_instrumental_template()
|
||||
template = self.get_action_needed_instrumental_template(locale)
|
||||
variables = {
|
||||
"name": name or "there",
|
||||
"artist": artist or "Unknown Artist",
|
||||
@@ -303,6 +433,47 @@ class TemplateService:
|
||||
}
|
||||
return self.render_template(template, variables)
|
||||
|
||||
def render_youtube_upload_complete(
|
||||
self,
|
||||
name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
youtube_url: Optional[str] = None,
|
||||
locale: str = "en",
|
||||
) -> str:
|
||||
"""
|
||||
Render the YouTube upload complete follow-up email template.
|
||||
|
||||
Sent when a deferred YouTube upload completes after quota becomes available.
|
||||
|
||||
Args:
|
||||
name: User's display name
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
youtube_url: YouTube video URL
|
||||
|
||||
Returns:
|
||||
Rendered email content
|
||||
"""
|
||||
if locale != "en":
|
||||
template = self._fetch_template_from_gcs(f"{locale}/youtube-upload-complete.txt")
|
||||
if template is not None:
|
||||
pass # Use locale-specific GCS template
|
||||
else:
|
||||
template = self._fetch_template_from_gcs("youtube-upload-complete.txt")
|
||||
else:
|
||||
template = self._fetch_template_from_gcs("youtube-upload-complete.txt")
|
||||
if template is None:
|
||||
template = DEFAULT_YOUTUBE_UPLOAD_COMPLETE_TEMPLATE if locale == "en" else self._build_localized_youtube_complete_template(locale)
|
||||
|
||||
variables = {
|
||||
"name": name or "there",
|
||||
"artist": artist or "Unknown Artist",
|
||||
"title": title or "Unknown Title",
|
||||
"youtube_url": youtube_url or "[YouTube URL not available]",
|
||||
}
|
||||
return self.render_template(template, variables)
|
||||
|
||||
def upload_template(self, template_name: str, content: str) -> bool:
|
||||
"""
|
||||
Upload a template to GCS.
|
||||
|
||||
@@ -334,7 +334,8 @@ class ThemeService:
|
||||
style_params = self._update_asset_paths_in_style(theme_id, style_params)
|
||||
|
||||
# Upload modified style_params.json to job's style folder
|
||||
job_style_path = f"uploads/{job_id}/style/style_params.json"
|
||||
# Uses jobs/ prefix (not uploads/) to avoid GCS lifecycle deletion rule
|
||||
job_style_path = f"jobs/{job_id}/style/style_params.json"
|
||||
self.storage.upload_json(job_style_path, style_params)
|
||||
|
||||
logger.info(f"Prepared job {job_id} style from theme {theme_id}")
|
||||
|
||||
@@ -13,6 +13,7 @@ import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Optional, List, Tuple
|
||||
|
||||
from google.api_core import exceptions as google_exceptions
|
||||
from google.cloud import firestore
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
from google.cloud.firestore_v1 import Increment
|
||||
@@ -59,29 +60,71 @@ class UserService:
|
||||
# User CRUD Operations
|
||||
# =========================================================================
|
||||
|
||||
def get_user_doc_ref(self, email: str):
|
||||
"""Get the Firestore document reference for a user by email.
|
||||
|
||||
Handles both email-keyed docs (doc ID = email) and hash-keyed docs
|
||||
(doc ID = hash, email stored as field). Returns None if not found.
|
||||
"""
|
||||
try:
|
||||
# First try direct lookup by email as doc ID
|
||||
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
|
||||
doc = doc_ref.get()
|
||||
if doc.exists:
|
||||
return doc_ref
|
||||
|
||||
# Fallback: query by email field (for hash-keyed docs)
|
||||
query = self.db.collection(USERS_COLLECTION).where(
|
||||
filter=FieldFilter('email', '==', email.lower())
|
||||
).limit(1)
|
||||
results = list(query.stream())
|
||||
if results:
|
||||
return results[0].reference
|
||||
|
||||
return None
|
||||
except Exception:
|
||||
logger.exception(f"Error finding user doc ref for {email}")
|
||||
return None
|
||||
|
||||
def get_user(self, email: str) -> Optional[User]:
|
||||
"""Get a user by email."""
|
||||
try:
|
||||
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
return None
|
||||
if doc.exists:
|
||||
return User(**doc.to_dict())
|
||||
|
||||
return User(**doc.to_dict())
|
||||
# Fallback: query by email field (for hash-keyed docs)
|
||||
query = self.db.collection(USERS_COLLECTION).where(
|
||||
filter=FieldFilter('email', '==', email.lower())
|
||||
).limit(1)
|
||||
results = list(query.stream())
|
||||
if results:
|
||||
return User(**results[0].to_dict())
|
||||
|
||||
return None
|
||||
except Exception:
|
||||
logger.exception(f"Error getting user {email}")
|
||||
return None
|
||||
|
||||
def get_or_create_user(self, email: str, tenant_id: Optional[str] = None) -> User:
|
||||
def get_or_create_user(
|
||||
self,
|
||||
email: str,
|
||||
tenant_id: Optional[str] = None,
|
||||
signup_ip: Optional[str] = None,
|
||||
device_fingerprint: Optional[str] = None,
|
||||
) -> User:
|
||||
"""
|
||||
Get existing user or create a new one.
|
||||
|
||||
New users receive a welcome credit to try the service.
|
||||
Welcome credits are granted on first magic link verification, not here.
|
||||
|
||||
Args:
|
||||
email: User's email address
|
||||
tenant_id: Tenant ID for white-label portals (None = default Nomad Karaoke)
|
||||
signup_ip: Client IP address at signup (for rate limiting)
|
||||
device_fingerprint: Browser fingerprint at signup (for rate limiting)
|
||||
|
||||
Note: If user exists but has a different tenant_id, the existing user is returned.
|
||||
Users are uniquely identified by email, not email+tenant.
|
||||
@@ -95,22 +138,17 @@ class UserService:
|
||||
# (though features may be restricted per-tenant)
|
||||
return user
|
||||
|
||||
# Create new user with welcome credit
|
||||
welcome_credit = self.NEW_USER_FREE_CREDITS
|
||||
welcome_transaction = CreditTransaction(
|
||||
id=str(uuid.uuid4()),
|
||||
amount=welcome_credit,
|
||||
reason="welcome_credit",
|
||||
)
|
||||
|
||||
# Create new user with zero credits (welcome credits granted on first verification)
|
||||
user = User(
|
||||
email=email,
|
||||
credits=welcome_credit,
|
||||
credit_transactions=[welcome_transaction],
|
||||
credits=0,
|
||||
credit_transactions=[],
|
||||
tenant_id=tenant_id, # Associate with tenant on creation
|
||||
signup_ip=signup_ip,
|
||||
device_fingerprint=device_fingerprint,
|
||||
)
|
||||
self._save_user(user)
|
||||
logger.info(f"Created new user: {email} with {welcome_credit} welcome credit(s) (tenant: {tenant_id or 'default'})")
|
||||
logger.info(f"Created new user: {email} (tenant: {tenant_id or 'default'}) — credits pending verification")
|
||||
return user
|
||||
|
||||
def _save_user(self, user: User) -> None:
|
||||
@@ -127,7 +165,10 @@ class UserService:
|
||||
"""Update user fields."""
|
||||
try:
|
||||
updates['updated_at'] = datetime.utcnow()
|
||||
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
|
||||
doc_ref = self.get_user_doc_ref(email)
|
||||
if not doc_ref:
|
||||
logger.warning(f"Cannot update user {email}: doc not found")
|
||||
return None
|
||||
doc_ref.update(updates)
|
||||
return self.get_user(email)
|
||||
except Exception:
|
||||
@@ -176,7 +217,9 @@ class UserService:
|
||||
email: str,
|
||||
ip_address: Optional[str] = None,
|
||||
user_agent: Optional[str] = None,
|
||||
tenant_id: Optional[str] = None
|
||||
tenant_id: Optional[str] = None,
|
||||
device_fingerprint: Optional[str] = None,
|
||||
referral_code: Optional[str] = None,
|
||||
) -> MagicLinkToken:
|
||||
"""
|
||||
Create a magic link token for email authentication.
|
||||
@@ -189,11 +232,17 @@ class UserService:
|
||||
ip_address: Client IP for auditing
|
||||
user_agent: Client user agent for auditing
|
||||
tenant_id: Tenant ID for white-label portals (None = default Nomad Karaoke)
|
||||
device_fingerprint: Browser fingerprint for anti-abuse
|
||||
"""
|
||||
email = email.lower()
|
||||
|
||||
# Ensure user exists (with tenant association)
|
||||
self.get_or_create_user(email, tenant_id=tenant_id)
|
||||
# Ensure user exists (with tenant association, signup IP, and fingerprint)
|
||||
self.get_or_create_user(
|
||||
email,
|
||||
tenant_id=tenant_id,
|
||||
signup_ip=ip_address,
|
||||
device_fingerprint=device_fingerprint,
|
||||
)
|
||||
|
||||
# Generate secure token
|
||||
token = secrets.token_urlsafe(32)
|
||||
@@ -205,6 +254,8 @@ class UserService:
|
||||
ip_address=ip_address,
|
||||
user_agent=user_agent,
|
||||
tenant_id=tenant_id,
|
||||
device_fingerprint=device_fingerprint,
|
||||
referral_code=referral_code,
|
||||
)
|
||||
|
||||
# Save to Firestore
|
||||
@@ -322,6 +373,406 @@ class UserService:
|
||||
logger.exception("Error verifying magic link")
|
||||
return False, None, "An error occurred during verification"
|
||||
|
||||
def grant_welcome_credits_if_eligible(
|
||||
self,
|
||||
email: str,
|
||||
precomputed_eval: Optional[dict] = None,
|
||||
) -> Tuple[bool, str]:
|
||||
"""
|
||||
Grant welcome credits on first magic link verification.
|
||||
|
||||
Uses AI evaluation to check for abuse before granting. Fail-open:
|
||||
if evaluation fails, credits are granted anyway.
|
||||
|
||||
Uses a dedicated welcome_credits_granted flag as primary idempotency
|
||||
check (survives credit_transactions list trimming). Falls back to
|
||||
checking for welcome_credit transaction as secondary defense.
|
||||
|
||||
Args:
|
||||
email: User's email address.
|
||||
precomputed_eval: Optional pre-computed evaluation from magic link doc.
|
||||
Keys: credit_eval_decision, credit_eval_reasoning, credit_eval_error.
|
||||
If provided and has a decision, skips the AI call.
|
||||
|
||||
Returns:
|
||||
Tuple of (credits_granted: bool, status: str).
|
||||
status is one of: "granted", "denied", "already_granted", "not_found", "error"
|
||||
"""
|
||||
try:
|
||||
email = email.lower()
|
||||
user = self.get_user(email)
|
||||
if not user:
|
||||
return False, "not_found"
|
||||
|
||||
# Primary check: dedicated flag (survives transaction list trimming)
|
||||
if user.welcome_credits_granted:
|
||||
return False, "already_granted"
|
||||
|
||||
# Secondary check: look for existing welcome_credit transaction
|
||||
for txn in user.credit_transactions:
|
||||
if txn.reason == "welcome_credit":
|
||||
return False, "already_granted"
|
||||
|
||||
# Skip AI evaluation for test emails (e.g. E2E test accounts)
|
||||
# to avoid wasting LLM credits and sending unnecessary denial emails
|
||||
from backend.utils.test_data import is_test_email
|
||||
if not is_test_email(email):
|
||||
# AI evaluation: check for abuse signals before granting
|
||||
# Use pre-computed result from magic link if available
|
||||
try:
|
||||
from backend.services.credit_evaluation_service import (
|
||||
get_credit_evaluation_service,
|
||||
CreditEvaluation,
|
||||
)
|
||||
precomputed_decision = (precomputed_eval or {}).get("credit_eval_decision")
|
||||
if precomputed_decision:
|
||||
logger.info(f"Using pre-computed credit eval for {email}: {precomputed_decision}")
|
||||
evaluation = CreditEvaluation(
|
||||
decision=precomputed_decision,
|
||||
reasoning=(precomputed_eval or {}).get("credit_eval_reasoning", ""),
|
||||
confidence=1.0,
|
||||
error=(precomputed_eval or {}).get("credit_eval_error"),
|
||||
)
|
||||
else:
|
||||
eval_service = get_credit_evaluation_service()
|
||||
evaluation = eval_service.evaluate(email, "welcome")
|
||||
|
||||
if evaluation.decision == "deny":
|
||||
# Mark as evaluated but not granted so we don't re-evaluate
|
||||
self.update_user(email, welcome_credits_granted=True)
|
||||
logger.info(
|
||||
f"Denied welcome credits to {email}: {evaluation.reasoning}"
|
||||
)
|
||||
try:
|
||||
from backend.services.email_service import get_email_service
|
||||
get_email_service().send_credit_denied_email(email, "welcome")
|
||||
except Exception:
|
||||
logger.exception(f"Failed to send credit denied email to {email}")
|
||||
return False, "denied"
|
||||
|
||||
if evaluation.decision == "pending_review":
|
||||
# Don't mark welcome_credits_granted — allow retry after manual review
|
||||
logger.info(
|
||||
f"Welcome credits pending review for {email}: {evaluation.reasoning}"
|
||||
)
|
||||
try:
|
||||
from backend.services.email_service import get_email_service
|
||||
email_service = get_email_service()
|
||||
email_service.send_credit_review_needed_email(
|
||||
email, "welcome", evaluation.reasoning, signals=evaluation.signals,
|
||||
)
|
||||
except Exception:
|
||||
logger.exception(f"Failed to send review needed email for {email}")
|
||||
return False, "pending_review"
|
||||
except Exception:
|
||||
logger.exception(f"Credit evaluation failed for {email} — pending review (fail-closed)")
|
||||
try:
|
||||
from backend.services.email_service import get_email_service
|
||||
email_service = get_email_service()
|
||||
email_service.send_credit_review_needed_email(
|
||||
email, "welcome", f"Evaluation error: {email}",
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
return False, "pending_review"
|
||||
else:
|
||||
logger.info(f"Skipping credit evaluation for test email {email}")
|
||||
|
||||
# Grant welcome credits
|
||||
welcome_credit = self.NEW_USER_FREE_CREDITS
|
||||
transaction = CreditTransaction(
|
||||
id=str(uuid.uuid4()),
|
||||
amount=welcome_credit,
|
||||
reason="welcome_credit",
|
||||
)
|
||||
|
||||
transactions = user.credit_transactions[-MAX_CREDIT_TRANSACTIONS + 1:]
|
||||
transactions.append(transaction)
|
||||
|
||||
new_balance = user.credits + welcome_credit
|
||||
self.update_user(
|
||||
email,
|
||||
credits=new_balance,
|
||||
credit_transactions=[t.model_dump(mode='json') for t in transactions],
|
||||
welcome_credits_granted=True,
|
||||
)
|
||||
|
||||
logger.info(f"Granted {welcome_credit} welcome credits to {email}")
|
||||
return True, "granted"
|
||||
|
||||
except Exception:
|
||||
logger.exception(f"Error granting welcome credits to {email}")
|
||||
return False, "error"
|
||||
|
||||
# =========================================================================
|
||||
# Anti-Abuse Rate Limiting
|
||||
# =========================================================================
|
||||
|
||||
# Maximum new signups allowed per IP address in the rate limit window
|
||||
MAX_SIGNUPS_PER_IP = 2
|
||||
SIGNUP_RATE_LIMIT_HOURS = 24
|
||||
|
||||
def count_recent_signups_from_ip(self, ip_address: str, hours: int = 24) -> int:
|
||||
"""
|
||||
Count new user accounts created from this IP in the last N hours.
|
||||
|
||||
Queries gen_users where signup_ip matches and created_at is recent.
|
||||
Used to rate-limit new account creation per IP address.
|
||||
|
||||
Args:
|
||||
ip_address: Client IP to check
|
||||
hours: Lookback window in hours (default: 24)
|
||||
|
||||
Returns:
|
||||
Number of recent signups from this IP
|
||||
"""
|
||||
try:
|
||||
cutoff = datetime.utcnow() - timedelta(hours=hours)
|
||||
query = (
|
||||
self.db.collection(USERS_COLLECTION)
|
||||
.where(filter=FieldFilter("signup_ip", "==", ip_address))
|
||||
.where(filter=FieldFilter("created_at", ">=", cutoff))
|
||||
)
|
||||
|
||||
count = 0
|
||||
for _ in query.stream():
|
||||
count += 1
|
||||
return count
|
||||
|
||||
except Exception:
|
||||
logger.exception(f"Error counting signups from IP {ip_address}")
|
||||
# On error, don't block — return 0 to allow the request
|
||||
return 0
|
||||
|
||||
def count_recent_signups_from_fingerprint(
|
||||
self, device_fingerprint: str, hours: int = 24
|
||||
) -> int:
|
||||
"""
|
||||
Count new user accounts created with this device fingerprint in the last N hours.
|
||||
|
||||
Args:
|
||||
device_fingerprint: Browser fingerprint to check
|
||||
hours: Lookback window in hours (default: 24)
|
||||
|
||||
Returns:
|
||||
Number of recent signups from this fingerprint
|
||||
"""
|
||||
try:
|
||||
cutoff = datetime.utcnow() - timedelta(hours=hours)
|
||||
query = (
|
||||
self.db.collection(USERS_COLLECTION)
|
||||
.where(filter=FieldFilter("device_fingerprint", "==", device_fingerprint))
|
||||
.where(filter=FieldFilter("created_at", ">=", cutoff))
|
||||
)
|
||||
|
||||
count = 0
|
||||
for _ in query.stream():
|
||||
count += 1
|
||||
return count
|
||||
|
||||
except Exception:
|
||||
logger.exception(f"Error counting signups from fingerprint")
|
||||
return 0
|
||||
|
||||
def is_signup_rate_limited(
|
||||
self, ip_address: Optional[str] = None, device_fingerprint: Optional[str] = None
|
||||
) -> bool:
|
||||
"""
|
||||
Check if a signup should be rate limited based on IP and/or fingerprint.
|
||||
|
||||
Either signal hitting the limit triggers a block.
|
||||
"""
|
||||
if ip_address:
|
||||
ip_count = self.count_recent_signups_from_ip(
|
||||
ip_address, hours=self.SIGNUP_RATE_LIMIT_HOURS
|
||||
)
|
||||
if ip_count >= self.MAX_SIGNUPS_PER_IP:
|
||||
return True
|
||||
|
||||
if device_fingerprint:
|
||||
fp_count = self.count_recent_signups_from_fingerprint(
|
||||
device_fingerprint, hours=self.SIGNUP_RATE_LIMIT_HOURS
|
||||
)
|
||||
if fp_count >= self.MAX_SIGNUPS_PER_IP:
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
# =========================================================================
|
||||
# Anti-Abuse Investigation
|
||||
# =========================================================================
|
||||
|
||||
def find_users_by_signup_ip(self, ip_address: str) -> List[User]:
|
||||
"""Find all users who signed up from this IP address."""
|
||||
try:
|
||||
query = (
|
||||
self.db.collection(USERS_COLLECTION)
|
||||
.where(filter=FieldFilter("signup_ip", "==", ip_address))
|
||||
.order_by("created_at", direction=firestore.Query.DESCENDING)
|
||||
.limit(50)
|
||||
)
|
||||
return [User(**doc.to_dict()) for doc in query.stream()]
|
||||
except Exception:
|
||||
logger.exception(f"Error finding users by IP {ip_address}")
|
||||
return []
|
||||
|
||||
def find_users_by_fingerprint(self, device_fingerprint: str) -> List[User]:
|
||||
"""Find all users who signed up with this device fingerprint."""
|
||||
try:
|
||||
query = (
|
||||
self.db.collection(USERS_COLLECTION)
|
||||
.where(filter=FieldFilter("device_fingerprint", "==", device_fingerprint))
|
||||
.order_by("created_at", direction=firestore.Query.DESCENDING)
|
||||
.limit(50)
|
||||
)
|
||||
return [User(**doc.to_dict()) for doc in query.stream()]
|
||||
except Exception:
|
||||
logger.exception(f"Error finding users by fingerprint")
|
||||
return []
|
||||
|
||||
def find_suspicious_accounts(self, min_jobs: int = 2, max_spend: int = 0) -> List[User]:
|
||||
"""
|
||||
Find accounts with many jobs but no spend (potential free-credit abusers).
|
||||
|
||||
Args:
|
||||
min_jobs: Minimum total_jobs_created to flag (default: 2)
|
||||
max_spend: Maximum total_spent in cents to include (default: 0 = free only)
|
||||
"""
|
||||
from backend.utils.test_data import is_internal_or_test_email
|
||||
|
||||
try:
|
||||
query = (
|
||||
self.db.collection(USERS_COLLECTION)
|
||||
.where(filter=FieldFilter("total_jobs_created", ">=", min_jobs))
|
||||
.order_by("total_jobs_created", direction=firestore.Query.DESCENDING)
|
||||
.limit(100)
|
||||
)
|
||||
users = []
|
||||
for doc in query.stream():
|
||||
user = User(**doc.to_dict())
|
||||
if user.total_spent <= max_spend and not is_internal_or_test_email(user.email):
|
||||
users.append(user)
|
||||
return users
|
||||
except Exception:
|
||||
logger.exception("Error finding suspicious accounts")
|
||||
return []
|
||||
|
||||
def find_related_accounts(self, email: str) -> dict:
|
||||
"""
|
||||
Given a user email, find all accounts that share the same signup IP
|
||||
or device fingerprint. Returns a dict with grouped results.
|
||||
"""
|
||||
user = self.get_user(email)
|
||||
if not user:
|
||||
return {"user": None, "by_ip": [], "by_fingerprint": []}
|
||||
|
||||
by_ip = []
|
||||
if user.signup_ip:
|
||||
by_ip = [
|
||||
u for u in self.find_users_by_signup_ip(user.signup_ip)
|
||||
if u.email != user.email
|
||||
]
|
||||
|
||||
by_fingerprint = []
|
||||
if user.device_fingerprint:
|
||||
by_fingerprint = [
|
||||
u for u in self.find_users_by_fingerprint(user.device_fingerprint)
|
||||
if u.email != user.email
|
||||
]
|
||||
|
||||
return {
|
||||
"user": user,
|
||||
"by_ip": by_ip,
|
||||
"by_fingerprint": by_fingerprint,
|
||||
}
|
||||
|
||||
def find_account_correlations(self) -> dict:
|
||||
"""
|
||||
Find clusters of accounts sharing the same fingerprint or signup IP.
|
||||
|
||||
Returns groups of 2+ accounts that share the same identifier,
|
||||
sorted by group size (largest first). Excludes internal/test accounts.
|
||||
"""
|
||||
from collections import defaultdict
|
||||
from backend.utils.test_data import is_internal_or_test_email
|
||||
|
||||
# Known bad IPs (load balancer, metadata, placeholder) that don't represent real client IPs
|
||||
BAD_IPS = {"169.254.169.126", "unknown", ""}
|
||||
|
||||
try:
|
||||
all_users = []
|
||||
for doc in self.db.collection(USERS_COLLECTION).stream():
|
||||
user = User(**doc.to_dict())
|
||||
if not is_internal_or_test_email(user.email):
|
||||
all_users.append(user)
|
||||
|
||||
# Group by fingerprint
|
||||
fp_groups: dict[str, list[User]] = defaultdict(list)
|
||||
for user in all_users:
|
||||
if user.device_fingerprint:
|
||||
fp_groups[user.device_fingerprint].append(user)
|
||||
|
||||
# Group by IP (exclude known bad IPs)
|
||||
ip_groups: dict[str, list[User]] = defaultdict(list)
|
||||
for user in all_users:
|
||||
if user.signup_ip and user.signup_ip not in BAD_IPS:
|
||||
ip_groups[user.signup_ip].append(user)
|
||||
|
||||
# Collect user agents from latest sessions for users in clusters
|
||||
cluster_emails = set()
|
||||
for fp, users in fp_groups.items():
|
||||
if len(users) >= 2:
|
||||
cluster_emails.update(u.email for u in users)
|
||||
for ip, users in ip_groups.items():
|
||||
if len(users) >= 2:
|
||||
cluster_emails.update(u.email for u in users)
|
||||
|
||||
user_agents: dict[str, str] = {}
|
||||
for email in cluster_emails:
|
||||
try:
|
||||
sessions = (
|
||||
self.db.collection(SESSIONS_COLLECTION)
|
||||
.where(filter=FieldFilter("user_email", "==", email))
|
||||
.order_by("created_at", direction=firestore.Query.DESCENDING)
|
||||
.limit(1)
|
||||
.stream()
|
||||
)
|
||||
for session_doc in sessions:
|
||||
session_data = session_doc.to_dict()
|
||||
ua = session_data.get("user_agent")
|
||||
if ua:
|
||||
user_agents[email] = ua
|
||||
except Exception:
|
||||
pass # Skip if session lookup fails
|
||||
|
||||
# Build response - only clusters with 2+ users
|
||||
fingerprint_clusters = []
|
||||
for fp, users in fp_groups.items():
|
||||
if len(users) >= 2:
|
||||
fingerprint_clusters.append({
|
||||
"fingerprint": fp,
|
||||
"users": sorted(users, key=lambda u: u.created_at or "", reverse=True),
|
||||
})
|
||||
fingerprint_clusters.sort(key=lambda c: len(c["users"]), reverse=True)
|
||||
|
||||
ip_clusters = []
|
||||
for ip, users in ip_groups.items():
|
||||
if len(users) >= 2:
|
||||
ip_clusters.append({
|
||||
"ip": ip,
|
||||
"users": sorted(users, key=lambda u: u.created_at or "", reverse=True),
|
||||
})
|
||||
ip_clusters.sort(key=lambda c: len(c["users"]), reverse=True)
|
||||
|
||||
return {
|
||||
"fingerprint_clusters": fingerprint_clusters,
|
||||
"ip_clusters": ip_clusters,
|
||||
"user_agents": user_agents,
|
||||
}
|
||||
except Exception:
|
||||
logger.exception("Error finding account correlations")
|
||||
return {"fingerprint_clusters": [], "ip_clusters": [], "user_agents": {}}
|
||||
|
||||
# =========================================================================
|
||||
# Session Management
|
||||
# =========================================================================
|
||||
@@ -331,7 +782,8 @@ class UserService:
|
||||
user_email: str,
|
||||
ip_address: Optional[str] = None,
|
||||
user_agent: Optional[str] = None,
|
||||
tenant_id: Optional[str] = None
|
||||
tenant_id: Optional[str] = None,
|
||||
device_fingerprint: Optional[str] = None,
|
||||
) -> Session:
|
||||
"""
|
||||
Create a new session for an authenticated user.
|
||||
@@ -341,6 +793,7 @@ class UserService:
|
||||
ip_address: Client IP for auditing
|
||||
user_agent: Client user agent for auditing
|
||||
tenant_id: Tenant ID for white-label portals (None = default Nomad Karaoke)
|
||||
device_fingerprint: Browser fingerprint for cross-account correlation
|
||||
"""
|
||||
token = secrets.token_urlsafe(32)
|
||||
token_prefix = token[:12]
|
||||
@@ -352,6 +805,7 @@ class UserService:
|
||||
ip_address=ip_address,
|
||||
user_agent=user_agent,
|
||||
tenant_id=tenant_id,
|
||||
device_fingerprint=device_fingerprint,
|
||||
)
|
||||
|
||||
# Serialize and write to Firestore
|
||||
@@ -546,7 +1000,7 @@ class UserService:
|
||||
'amount': amount,
|
||||
'processed_at': datetime.utcnow()
|
||||
})
|
||||
except Exception:
|
||||
except google_exceptions.AlreadyExists:
|
||||
# Document already exists - this session was already processed
|
||||
logger.info(f"Stripe session {stripe_session_id} already processed (idempotent skip)")
|
||||
return False, 0, "Session already processed"
|
||||
@@ -768,15 +1222,62 @@ class UserService:
|
||||
logger.exception("Error enabling user")
|
||||
return False
|
||||
|
||||
def delete_user(self, email: str, admin_email: str) -> bool:
|
||||
"""
|
||||
Permanently delete a user and all their associated data.
|
||||
|
||||
Deletes: user document, sessions, magic links.
|
||||
Does NOT delete jobs (they remain as historical records).
|
||||
|
||||
Args:
|
||||
email: Email of the user to delete
|
||||
admin_email: Email/ID of the admin performing the action
|
||||
|
||||
Returns:
|
||||
True if user was deleted, False if user not found
|
||||
|
||||
Raises:
|
||||
ValueError: If trying to delete an admin user
|
||||
"""
|
||||
try:
|
||||
email = email.lower()
|
||||
user = self.get_user(email)
|
||||
|
||||
if not user:
|
||||
return False
|
||||
|
||||
if user.role == UserRole.ADMIN:
|
||||
raise ValueError("Cannot delete admin users")
|
||||
|
||||
# Revoke all active sessions
|
||||
self.revoke_all_sessions(email)
|
||||
|
||||
# Delete magic links for the user
|
||||
magic_links = self.db.collection(MAGIC_LINKS_COLLECTION).where(
|
||||
filter=FieldFilter('email', '==', email)
|
||||
).stream()
|
||||
for doc in magic_links:
|
||||
doc.reference.delete()
|
||||
|
||||
# Delete the user document
|
||||
self.db.collection(USERS_COLLECTION).document(email).delete()
|
||||
|
||||
logger.info(f"Admin {admin_email} permanently deleted user {email}")
|
||||
return True
|
||||
|
||||
except ValueError:
|
||||
raise
|
||||
except Exception:
|
||||
logger.exception(f"Error deleting user {email}")
|
||||
return False
|
||||
|
||||
def increment_jobs_completed(self, email: str) -> bool:
|
||||
"""Increment the completed jobs counter for a user using atomic increment."""
|
||||
try:
|
||||
# Check user exists first
|
||||
if not self.get_user(email):
|
||||
doc_ref = self.get_user_doc_ref(email)
|
||||
if not doc_ref:
|
||||
return False
|
||||
|
||||
# Use atomic increment to prevent race conditions
|
||||
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
|
||||
doc_ref.update({
|
||||
'total_jobs_completed': Increment(1),
|
||||
'updated_at': datetime.utcnow()
|
||||
|
||||
@@ -0,0 +1,300 @@
|
||||
"""
|
||||
Visibility change service for completed jobs.
|
||||
|
||||
Handles changing a job's visibility between public and private:
|
||||
- Public -> Private: redistribute existing finals to private destination (fast, ~1-2 min)
|
||||
- Private -> Public: reset styles, regenerate screens, re-render, re-encode (slow, ~15-30 min)
|
||||
"""
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional
|
||||
|
||||
from google.cloud.firestore_v1 import DELETE_FIELD, ArrayUnion
|
||||
|
||||
from backend.models.job import JobStatus
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class VisibilityChangeService:
|
||||
"""Service for changing job visibility after completion."""
|
||||
|
||||
def __init__(self, job_manager: Optional[JobManager] = None):
|
||||
self.job_manager = job_manager or JobManager()
|
||||
|
||||
def validate_change(self, job, user_email: str, is_admin: bool, target_visibility: str) -> Optional[str]:
|
||||
"""Validate a visibility change request. Returns error message or None if valid."""
|
||||
if not job:
|
||||
return "Job not found"
|
||||
|
||||
if job.status != JobStatus.COMPLETE.value and job.status != "complete":
|
||||
return f"Can only change visibility on completed jobs. Current status: {job.status}"
|
||||
|
||||
if getattr(job, 'tenant_id', None):
|
||||
return "Cannot change visibility on tenant jobs"
|
||||
|
||||
current_is_private = getattr(job, 'is_private', False)
|
||||
target_is_private = target_visibility == "private"
|
||||
|
||||
if current_is_private == target_is_private:
|
||||
current = "private" if current_is_private else "public"
|
||||
return f"Job is already {current}"
|
||||
|
||||
# Check user owns the job or is admin
|
||||
if not is_admin and job.user_email != user_email:
|
||||
return "You can only change visibility on your own jobs"
|
||||
|
||||
# Check for in-progress visibility change
|
||||
state_data = job.state_data or {}
|
||||
if state_data.get('visibility_change_in_progress'):
|
||||
return "A visibility change is already in progress for this job"
|
||||
|
||||
return None
|
||||
|
||||
async def change_to_private(self, job_id: str, job, user_email: str) -> dict:
|
||||
"""
|
||||
Change a public job to private (fast path).
|
||||
|
||||
Deletes distributed outputs (YouTube/Dropbox/GDrive) but keeps GCS finals,
|
||||
then redistributes to private destination.
|
||||
"""
|
||||
logger.info(f"[job:{job_id}] Starting visibility change: public -> private (by {user_email})")
|
||||
|
||||
db = self.job_manager.firestore.db
|
||||
job_ref = db.collection("jobs").document(job_id)
|
||||
|
||||
# Set guard flag
|
||||
job_ref.update({
|
||||
"state_data.visibility_change_in_progress": True,
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
})
|
||||
|
||||
try:
|
||||
# Step 1: Delete distributed outputs (keep GCS finals)
|
||||
await self._delete_distributed_outputs(job_id, job, keep_gcs_finals=True)
|
||||
|
||||
# Step 2: Update job to private
|
||||
job_ref.update({
|
||||
"is_private": True,
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
"timeline": ArrayUnion([{
|
||||
"status": "complete",
|
||||
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||
"message": f"Visibility changed to private by {user_email}",
|
||||
}]),
|
||||
})
|
||||
|
||||
# Step 3: Trigger redistribution via video worker
|
||||
from backend.workers.video_worker import redistribute_video
|
||||
success = await redistribute_video(job_id)
|
||||
|
||||
if not success:
|
||||
raise RuntimeError("Redistribution failed - check worker logs")
|
||||
|
||||
# Clear guard flag (redistribution updates state_data with new brand_code etc.)
|
||||
job_ref.update({
|
||||
"state_data.visibility_change_in_progress": DELETE_FIELD,
|
||||
})
|
||||
|
||||
logger.info(f"[job:{job_id}] Visibility change to private complete")
|
||||
return {
|
||||
"status": "success",
|
||||
"message": "Job changed to private. Outputs redistributed to private destination.",
|
||||
"reprocessing_required": False,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"[job:{job_id}] Visibility change to private failed: {e}", exc_info=True)
|
||||
# Clear guard flag on failure
|
||||
job_ref.update({
|
||||
"state_data.visibility_change_in_progress": DELETE_FIELD,
|
||||
})
|
||||
raise
|
||||
|
||||
async def change_to_public(self, job_id: str, job, user_email: str) -> dict:
|
||||
"""
|
||||
Change a private job to public (slow path).
|
||||
|
||||
Resets custom styles, deletes all outputs/screens/rendered video,
|
||||
and triggers full re-processing from screens worker onward.
|
||||
"""
|
||||
logger.info(f"[job:{job_id}] Starting visibility change: private -> public (by {user_email})")
|
||||
|
||||
db = self.job_manager.firestore.db
|
||||
job_ref = db.collection("jobs").document(job_id)
|
||||
storage = StorageService()
|
||||
|
||||
# Step 1: Delete distributed outputs + GCS finals
|
||||
await self._delete_distributed_outputs(job_id, job, keep_gcs_finals=False)
|
||||
|
||||
# Step 2: Delete screens and rendered video from GCS
|
||||
for path in [
|
||||
f"jobs/{job_id}/screens/title.mov",
|
||||
f"jobs/{job_id}/screens/title.jpg",
|
||||
f"jobs/{job_id}/screens/title.png",
|
||||
f"jobs/{job_id}/screens/end.mov",
|
||||
f"jobs/{job_id}/screens/end.jpg",
|
||||
f"jobs/{job_id}/screens/end.png",
|
||||
f"jobs/{job_id}/videos/with_vocals.mkv",
|
||||
f"jobs/{job_id}/videos/with_vocals.mov",
|
||||
]:
|
||||
try:
|
||||
storage.delete_file(path)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Delete custom style files from GCS
|
||||
try:
|
||||
style_prefix = f"jobs/{job_id}/style/"
|
||||
storage.delete_folder(style_prefix)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Step 3: Prepare nomad theme style files for the job
|
||||
from backend.api.routes.file_upload import _prepare_theme_for_job
|
||||
style_params_path, style_assets_dict, _youtube_desc = _prepare_theme_for_job(
|
||||
job_id=job_id, theme_id="nomad"
|
||||
)
|
||||
|
||||
# Step 4: Reset styles and update job fields
|
||||
# Step 5: Set status to LYRICS_COMPLETE with regen_restore_status
|
||||
# Step 6: Clear progress keys and set guard flag
|
||||
update_payload = {
|
||||
"is_private": False,
|
||||
"theme_id": "nomad",
|
||||
"color_overrides": {},
|
||||
"style_assets": style_assets_dict,
|
||||
"style_params_gcs_path": style_params_path,
|
||||
"status": JobStatus.LYRICS_COMPLETE.value,
|
||||
"state_data.visibility_change_in_progress": True,
|
||||
"state_data.regen_restore_status": "review_complete",
|
||||
"state_data.audio_complete": True,
|
||||
"state_data.lyrics_complete": True,
|
||||
"state_data.screens_progress": DELETE_FIELD,
|
||||
"state_data.render_progress": DELETE_FIELD,
|
||||
"state_data.video_progress": DELETE_FIELD,
|
||||
"state_data.encoding_progress": DELETE_FIELD,
|
||||
"outputs_deleted_at": None,
|
||||
"outputs_deleted_by": None,
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
"timeline": ArrayUnion([{
|
||||
"status": "lyrics_complete",
|
||||
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||
"message": f"Visibility change to public initiated by {user_email}. "
|
||||
"Re-processing from screens generation.",
|
||||
}]),
|
||||
}
|
||||
job_ref.update(update_payload)
|
||||
|
||||
# Step 6: Trigger screens worker
|
||||
from backend.services.worker_service import get_worker_service
|
||||
worker_service = get_worker_service()
|
||||
triggered = await worker_service.trigger_screens_worker(job_id)
|
||||
|
||||
if not triggered:
|
||||
logger.error(f"[job:{job_id}] Failed to trigger screens worker for visibility change")
|
||||
# Restore to complete status on failure
|
||||
job_ref.update({
|
||||
"status": "complete",
|
||||
"is_private": True,
|
||||
"state_data.visibility_change_in_progress": DELETE_FIELD,
|
||||
"state_data.regen_restore_status": DELETE_FIELD,
|
||||
})
|
||||
raise RuntimeError("Failed to trigger screens worker")
|
||||
|
||||
logger.info(f"[job:{job_id}] Visibility change to public initiated, screens worker triggered")
|
||||
return {
|
||||
"status": "processing",
|
||||
"message": "Visibility change to public started. The video will be re-rendered "
|
||||
"with default branding and published publicly. This takes ~15-30 minutes.",
|
||||
"reprocessing_required": True,
|
||||
}
|
||||
|
||||
async def _delete_distributed_outputs(self, job_id: str, job, keep_gcs_finals: bool = False):
|
||||
"""Delete distributed outputs (YouTube, Dropbox, GDrive) and recycle brand code."""
|
||||
import re
|
||||
|
||||
state_data = job.state_data or {}
|
||||
|
||||
# Delete YouTube video
|
||||
youtube_url = state_data.get('youtube_url')
|
||||
if youtube_url:
|
||||
try:
|
||||
video_id_match = re.search(r'(?:youtu\.be/|youtube\.com/watch\?v=)([^&\s]+)', youtube_url)
|
||||
if video_id_match:
|
||||
video_id = video_id_match.group(1)
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
from backend.services.youtube_service import get_youtube_service
|
||||
youtube_service = get_youtube_service()
|
||||
if youtube_service.is_configured:
|
||||
finalise = KaraokeFinalise(
|
||||
dry_run=False, non_interactive=True,
|
||||
user_youtube_credentials=youtube_service.get_credentials_dict()
|
||||
)
|
||||
finalise.delete_youtube_video(video_id)
|
||||
logger.info(f"[job:{job_id}] Deleted YouTube video {video_id}")
|
||||
except Exception as e:
|
||||
logger.warning(f"[job:{job_id}] Failed to delete YouTube video: {e}")
|
||||
|
||||
# Delete Dropbox folder
|
||||
brand_code = state_data.get('brand_code')
|
||||
dropbox_path = getattr(job, 'dropbox_path', None)
|
||||
if brand_code and dropbox_path:
|
||||
try:
|
||||
from backend.services.dropbox_service import get_dropbox_service
|
||||
dropbox = get_dropbox_service()
|
||||
if dropbox.is_configured:
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
safe_artist = sanitize_filename(job.artist) if job.artist else "Unknown"
|
||||
safe_title = sanitize_filename(job.title) if job.title else "Unknown"
|
||||
folder_name = f"{brand_code} - {safe_artist} - {safe_title}"
|
||||
full_path = f"{dropbox_path}/{folder_name}"
|
||||
dropbox.delete_folder(full_path)
|
||||
logger.info(f"[job:{job_id}] Deleted Dropbox folder {full_path}")
|
||||
except Exception as e:
|
||||
logger.warning(f"[job:{job_id}] Failed to delete Dropbox folder: {e}")
|
||||
|
||||
# Delete Google Drive files
|
||||
gdrive_files = state_data.get('gdrive_files')
|
||||
if gdrive_files:
|
||||
try:
|
||||
from backend.services.gdrive_service import get_gdrive_service
|
||||
gdrive = get_gdrive_service()
|
||||
if gdrive.is_configured:
|
||||
file_ids = list(gdrive_files.values()) if isinstance(gdrive_files, dict) else []
|
||||
gdrive.delete_files(file_ids)
|
||||
logger.info(f"[job:{job_id}] Deleted Google Drive files")
|
||||
except Exception as e:
|
||||
logger.warning(f"[job:{job_id}] Failed to delete Google Drive files: {e}")
|
||||
|
||||
# Delete GCS finals if requested
|
||||
if not keep_gcs_finals:
|
||||
try:
|
||||
storage = StorageService()
|
||||
deleted_count = storage.delete_folder(f"jobs/{job_id}/finals/")
|
||||
if deleted_count > 0:
|
||||
logger.info(f"[job:{job_id}] Deleted {deleted_count} GCS finals files")
|
||||
except Exception as e:
|
||||
logger.warning(f"[job:{job_id}] Failed to delete GCS finals: {e}")
|
||||
|
||||
# Recycle brand code
|
||||
if brand_code:
|
||||
try:
|
||||
from backend.services.brand_code_service import BrandCodeService, get_brand_code_service
|
||||
prefix, number = BrandCodeService.parse_brand_code(brand_code)
|
||||
get_brand_code_service().recycle_brand_code(prefix, number)
|
||||
logger.info(f"[job:{job_id}] Recycled brand code {brand_code}")
|
||||
except Exception as e:
|
||||
logger.warning(f"[job:{job_id}] Failed to recycle brand code: {e}")
|
||||
|
||||
# Clear distribution state_data
|
||||
db = self.job_manager.firestore.db
|
||||
job_ref = db.collection("jobs").document(job_id)
|
||||
update = {}
|
||||
for key in ['youtube_url', 'brand_code', 'dropbox_link', 'gdrive_files']:
|
||||
if key in state_data:
|
||||
update[f"state_data.{key}"] = DELETE_FIELD
|
||||
if update:
|
||||
job_ref.update(update)
|
||||
@@ -38,6 +38,7 @@ WORKER_QUEUES = {
|
||||
"render-video": "render-worker-queue",
|
||||
"video": "video-worker-queue",
|
||||
"idle-reminder": "idle-reminder-queue", # For delayed idle reminder checks
|
||||
"gdrive-validation": "gdrive-validation-queue", # For delayed post-job validation
|
||||
}
|
||||
|
||||
# Dispatch deadlines for each worker type (in seconds)
|
||||
@@ -51,10 +52,15 @@ WORKER_DISPATCH_DEADLINES = {
|
||||
"render-video": 1800, # 30 min - Video encoding can be slow
|
||||
"video": 1800, # 30 min - Video encoding can be slow
|
||||
"idle-reminder": 60, # 1 min - Quick check and potential email send
|
||||
"gdrive-validation": 120, # 2 min - Calls Cloud Function which validates GDrive
|
||||
}
|
||||
|
||||
# Default delay for idle reminders (seconds)
|
||||
IDLE_REMINDER_DELAY_SECONDS = 5 * 60 # 5 minutes
|
||||
IDLE_REMINDER_DELAY_SECONDS = 2 * 60 # 2 minutes
|
||||
|
||||
# Default delay for post-job GDrive validation (seconds)
|
||||
# Gives E2E tests time to verify output and clean up before validator runs
|
||||
GDRIVE_VALIDATION_DELAY_SECONDS = 5 * 60 # 5 minutes
|
||||
|
||||
|
||||
class WorkerService:
|
||||
@@ -355,19 +361,35 @@ class WorkerService:
|
||||
|
||||
# Convenience methods for specific workers
|
||||
# These provide a cleaner API and better IDE autocomplete
|
||||
|
||||
|
||||
async def trigger_audio_download_worker(self, job_id: str) -> bool:
|
||||
"""
|
||||
Trigger audio download worker.
|
||||
|
||||
Downloads audio from the selected source (Spotify, YouTube, RED, OPS)
|
||||
and then triggers audio separation + lyrics workers.
|
||||
|
||||
Uses Cloud Run Jobs to avoid instance termination during long-running
|
||||
downloads. This replaces the BackgroundTasks approach that caused
|
||||
jobs to get stuck when Cloud Run scaled down instances.
|
||||
"""
|
||||
return await self._trigger_worker_cloud_run_job(
|
||||
job_id=job_id,
|
||||
cloud_run_job_name="audio-download-job",
|
||||
worker_module="audio_download_worker",
|
||||
)
|
||||
|
||||
async def trigger_audio_worker(self, job_id: str) -> bool:
|
||||
"""
|
||||
Trigger audio separation worker.
|
||||
|
||||
Always uses Cloud Run Jobs to avoid instance termination during long-running
|
||||
Modal API calls. The BackgroundTasks approach caused Cloud Run to terminate
|
||||
instances mid-processing when no HTTP requests were active.
|
||||
Uses Cloud Run Jobs with L4 GPU in us-east4 for direct audio separation.
|
||||
"""
|
||||
return await self._trigger_worker_cloud_run_job(
|
||||
job_id=job_id,
|
||||
cloud_run_job_name="audio-separation-job",
|
||||
worker_module="audio_worker",
|
||||
location="us-east4", # GPU job runs in us-east4 (L4 quota)
|
||||
)
|
||||
|
||||
async def trigger_lyrics_worker(self, job_id: str) -> bool:
|
||||
@@ -389,14 +411,40 @@ class WorkerService:
|
||||
"""Trigger screen generation worker."""
|
||||
return await self.trigger_worker("screens", job_id)
|
||||
|
||||
def _warmup_encoding_worker(self, job_id: str) -> None:
|
||||
"""Fire-and-forget warmup of the encoding worker VM.
|
||||
|
||||
Called before dispatching video/render workers so the GCE encoding
|
||||
VM starts booting while the worker downloads files and prepares.
|
||||
Failures are logged but never block worker dispatch.
|
||||
"""
|
||||
try:
|
||||
from backend.services.encoding_worker_manager import EncodingWorkerManager
|
||||
from google.cloud import compute_v1, firestore
|
||||
db = firestore.Client(project=self.settings.google_cloud_project)
|
||||
compute_client = compute_v1.InstancesClient()
|
||||
manager = EncodingWorkerManager(
|
||||
db=db,
|
||||
compute_client=compute_client,
|
||||
project_id=self.settings.google_cloud_project,
|
||||
)
|
||||
result = manager.ensure_primary_running()
|
||||
if result["started"]:
|
||||
logger.info(f"[job:{job_id}] Encoding worker warmup: started VM {result['vm_name']}")
|
||||
else:
|
||||
logger.info(f"[job:{job_id}] Encoding worker warmup: VM {result['vm_name']} already running")
|
||||
except Exception as e:
|
||||
logger.warning(f"[job:{job_id}] Encoding worker warmup failed (non-fatal): {e}")
|
||||
|
||||
async def trigger_video_worker(self, job_id: str) -> bool:
|
||||
"""
|
||||
Trigger video generation worker.
|
||||
|
||||
|
||||
When USE_CLOUD_RUN_JOBS_FOR_VIDEO=true and ENABLE_CLOUD_TASKS=true,
|
||||
uses Cloud Run Jobs for execution (supports >30 min encoding).
|
||||
Otherwise, uses Cloud Tasks or direct HTTP.
|
||||
"""
|
||||
self._warmup_encoding_worker(job_id)
|
||||
if self._use_cloud_tasks and self.settings.use_cloud_run_jobs_for_video:
|
||||
return await self._trigger_cloud_run_job(job_id)
|
||||
return await self.trigger_worker("video", job_id)
|
||||
@@ -425,6 +473,7 @@ class WorkerService:
|
||||
job_id: str,
|
||||
cloud_run_job_name: str,
|
||||
worker_module: str,
|
||||
location: str | None = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Trigger a worker as a Cloud Run Job.
|
||||
@@ -437,6 +486,7 @@ class WorkerService:
|
||||
job_id: Job ID to process
|
||||
cloud_run_job_name: Name of the Cloud Run Job (e.g., "lyrics-transcription-job")
|
||||
worker_module: Worker module name (e.g., "lyrics_worker")
|
||||
location: GCP region for the job (defaults to settings.gcp_region)
|
||||
|
||||
Returns:
|
||||
True if job was triggered successfully, False otherwise
|
||||
@@ -449,7 +499,7 @@ class WorkerService:
|
||||
logger.error("GOOGLE_CLOUD_PROJECT not set, cannot trigger Cloud Run Job")
|
||||
return False
|
||||
|
||||
location = self.settings.gcp_region
|
||||
location = location or self.settings.gcp_region
|
||||
job_name = f"projects/{project}/locations/{location}/jobs/{cloud_run_job_name}"
|
||||
|
||||
# Create Cloud Run Jobs client
|
||||
@@ -487,6 +537,7 @@ class WorkerService:
|
||||
|
||||
async def trigger_render_video_worker(self, job_id: str) -> bool:
|
||||
"""Trigger render video worker (post-review)."""
|
||||
self._warmup_encoding_worker(job_id)
|
||||
return await self.trigger_worker("render-video", job_id)
|
||||
|
||||
async def schedule_idle_reminder(
|
||||
@@ -582,6 +633,93 @@ class WorkerService:
|
||||
)
|
||||
return False
|
||||
|
||||
async def schedule_gdrive_validation(
|
||||
self,
|
||||
delay_seconds: int = GDRIVE_VALIDATION_DELAY_SECONDS
|
||||
) -> bool:
|
||||
"""
|
||||
Schedule a delayed GDrive validation check.
|
||||
|
||||
Fires after the specified delay so that E2E test cleanup completes
|
||||
before the validator runs, avoiding false "all clear" reports.
|
||||
|
||||
Args:
|
||||
delay_seconds: Delay before executing the validation (default: 5 minutes)
|
||||
|
||||
Returns:
|
||||
True if task was scheduled successfully, False otherwise
|
||||
"""
|
||||
if not self._use_cloud_tasks:
|
||||
logger.info(
|
||||
f"GDrive validation not scheduled (Cloud Tasks disabled). "
|
||||
f"Would have fired in {delay_seconds}s."
|
||||
)
|
||||
return True
|
||||
|
||||
try:
|
||||
from google.cloud import tasks_v2
|
||||
from google.protobuf import timestamp_pb2
|
||||
import time
|
||||
|
||||
queue_name = WORKER_QUEUES.get("gdrive-validation")
|
||||
if not queue_name:
|
||||
logger.error("GDrive validation queue not configured")
|
||||
return False
|
||||
|
||||
project = self.settings.google_cloud_project
|
||||
if not project:
|
||||
logger.error("GOOGLE_CLOUD_PROJECT not set")
|
||||
return False
|
||||
|
||||
location = self.settings.gcp_region
|
||||
|
||||
# Build queue path
|
||||
parent = self.tasks_client.queue_path(project, location, queue_name)
|
||||
|
||||
# Build headers
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
if self._admin_token:
|
||||
headers["X-Admin-Token"] = self._admin_token
|
||||
|
||||
headers = inject_trace_context(headers)
|
||||
|
||||
# Calculate schedule time
|
||||
schedule_time = timestamp_pb2.Timestamp()
|
||||
schedule_time.FromSeconds(int(time.time()) + delay_seconds)
|
||||
|
||||
dispatch_deadline_seconds = WORKER_DISPATCH_DEADLINES.get("gdrive-validation", 120)
|
||||
|
||||
# Build task payload
|
||||
task = {
|
||||
"http_request": {
|
||||
"http_method": tasks_v2.HttpMethod.POST,
|
||||
"url": f"{self._base_url}/api/internal/trigger-gdrive-validation",
|
||||
"headers": headers,
|
||||
"body": b"{}",
|
||||
"oidc_token": {
|
||||
"service_account_email": f"karaoke-backend@{project}.iam.gserviceaccount.com",
|
||||
},
|
||||
},
|
||||
"schedule_time": schedule_time,
|
||||
"dispatch_deadline": duration_pb2.Duration(seconds=dispatch_deadline_seconds),
|
||||
}
|
||||
|
||||
response = self.tasks_client.create_task(parent=parent, task=task)
|
||||
logger.info(
|
||||
f"Scheduled GDrive validation in {delay_seconds}s: {response.name}"
|
||||
)
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Failed to schedule GDrive validation: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
return False
|
||||
|
||||
|
||||
# Global worker service instance
|
||||
_worker_service: Optional[WorkerService] = None
|
||||
|
||||
Loaded 100 of 1255 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user