Compare commits

..
Author SHA1 Message Date
Andrew BeveridgeandClaude Opus 4.5 275fc909f6 fix: E2E test issues - add Firestore index and fix job endpoint URL
- Add missing Firestore composite index for customer_email + made_for_you queries
  (required for test-webhook made-for-you order processing)
- Fix getJobDetails helper to use /api/jobs/{job_id} instead of /api/admin/jobs/{job_id}
  (the admin route doesn't have a GET method)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-29 21:52:34 -08:00
1254 changed files with 12505 additions and 231335 deletions

No files matched your search

@@ -1,109 +0,0 @@
# Plan: Admin Feedback Visibility
**Created:** 2026-03-23
**Branch:** feat/sess-20260323-1454-feedback-email-debug
**Status:** Draft
## Overview
User feedback submissions are stored in Firestore (`user_feedback` collection) but are completely invisible to the admin. The admin panel only shows a boolean "Feedback Submitted: Yes/No" badge per user. This plan adds:
1. **Admin UI page** to browse and review all submitted feedback
2. **Email notification** to admin@nomadkaraoke.com on each new submission
## Requirements
- [ ] Admin page at `/admin/feedback` listing all feedback with ratings, text, timestamps, and user info
- [ ] Sortable by date (newest first by default) and filterable by search (email)
- [ ] Star ratings rendered visually (not just numbers)
- [ ] Email sent to admin@nomadkaraoke.com immediately when feedback is submitted
- [ ] Email includes all feedback content (ratings, text fields, user email)
- [ ] Test data filtering works (exclude_test toggle)
- [ ] Sidebar navigation includes new "Feedback" item
## Technical Approach
### Admin API Endpoint
Add `GET /api/admin/feedback` to `backend/api/routes/admin.py`. Follows the standard admin endpoint pattern: `require_admin` dependency, pagination with `limit`/`offset`/`search`/`exclude_test`, returns Pydantic response model. Queries `user_feedback` Firestore collection.
### Admin UI Page
New page at `frontend/app/admin/feedback/page.tsx`. Follows existing admin page patterns (Card, Table, Badge, pagination, search input, loading states). Uses `adminApi.listFeedback()` to fetch data.
### Email Notification
Add a new method to `EmailService` (or call `send_email` directly) from the `submit_user_feedback` endpoint in `users.py`. Uses the existing SendGrid infrastructure — no new template service templates needed since this is an internal admin notification (simple HTML table with the feedback data).
## Implementation Steps
### Step 1: Backend — Admin feedback list endpoint
1. [ ] Add `AdminFeedbackItem` and `AdminFeedbackListResponse` Pydantic models to `admin.py`
2. [ ] Add `GET /api/admin/feedback` endpoint with:
- Pagination: `limit=50`, `offset=0`
- Search by user email
- `exclude_test` filter (reuse `is_test_email`)
- Sort by `created_at` descending
- Return: items, total, offset, limit, has_more
3. [ ] Add aggregate stats in response: average ratings, total count
### Step 2: Backend — Email notification on submission
4. [ ] Add `send_feedback_notification` method to `EmailService` that sends an HTML email to `admin@nomadkaraoke.com` containing:
- User email
- All 4 star ratings
- All text fields
- Would recommend / would use again
- Submitted timestamp
- Link to admin feedback page
5. [ ] Call `send_feedback_notification` from `submit_user_feedback` in `users.py` (fire-and-forget, don't block the user response on email success)
### Step 3: Frontend — Admin API client
6. [ ] Add `listFeedback` method to `adminApi` in `frontend/lib/api.ts`
7. [ ] Add TypeScript types for `AdminFeedbackItem` and `AdminFeedbackListResponse`
### Step 4: Frontend — Admin feedback page
8. [ ] Create `frontend/app/admin/feedback/page.tsx` with:
- Table: user email, date, overall rating (stars), ease of use, lyrics accuracy, correction experience, would recommend, would use again
- Expandable rows or click-to-detail for text fields (what went well, what could improve, additional comments)
- Search by email
- Pagination (50 per page)
- Refresh button
- Test data toggle (via `useAdminSettings`)
9. [ ] Add "Feedback" item to admin sidebar in `frontend/components/admin/admin-sidebar.tsx` with `MessageSquare` icon
### Step 5: Testing
10. [ ] Backend unit test for `GET /api/admin/feedback` endpoint (pagination, search, exclude_test)
11. [ ] Backend unit test for `send_feedback_notification` email method
12. [ ] Frontend component test for feedback page (render, empty state)
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/api/routes/admin.py` | Modify | Add feedback list endpoint + response models |
| `backend/api/routes/users.py` | Modify | Add email notification call in `submit_user_feedback` |
| `backend/services/email_service.py` | Modify | Add `send_feedback_notification` method |
| `frontend/lib/api.ts` | Modify | Add `adminApi.listFeedback()` method + types |
| `frontend/app/admin/feedback/page.tsx` | Create | Admin feedback review page |
| `frontend/components/admin/admin-sidebar.tsx` | Modify | Add "Feedback" nav item |
| `backend/tests/test_admin_feedback.py` | Create | Tests for admin feedback endpoint |
| `backend/tests/test_feedback_email.py` | Create | Tests for feedback email notification |
## Testing Strategy
- **Unit tests**: Admin endpoint (pagination, filtering, sorting), email method (content formatting, send call)
- **Integration**: Submit feedback → verify email service called
- **Manual verification**: Check email arrives in admin@nomadkaraoke.com inbox after submission
## Open Questions
- [ ] Should the admin page show feedback from the old `beta_feedback` collection too, or just `user_feedback`? (Recommendation: just `user_feedback` — the beta collection is legacy)
- [ ] Should there be any spam/quality detection flagging? (Some submissions appear to game the system for credits)
## Rollback Plan
- Backend: Remove endpoint from admin.py, revert email call in users.py
- Frontend: Delete feedback page, remove sidebar item
- No data migration needed — Firestore collection unchanged
-120
View File
@@ -1,120 +0,0 @@
# Plan: Admin Lyrics Review Dashboard
**Created:** 2026-03-05
**Branch:** feat/sess-20260305-0002-admin-lyrics-review
**Status:** Draft
## Overview
Add an "Edit Reviews" section to the admin dashboard for reviewing completed lyrics reviews. Three goals:
1. **View what users saw** - Jump into a read-only lyrics review UI showing original vs corrected lyrics
2. **Replay user edits** - Step through edit log entries to see exactly what each user changed
3. **Review raw data** - View human edits data to evaluate quality for AudioShake data sharing
## Requirements
- [ ] New admin sidebar item "Edit Reviews" linking to `/admin/edit-reviews`
- [ ] Backend endpoint to list recent jobs that have completed lyrics review (with edit logs)
- [ ] Backend endpoint to serve immutable correction data for a job (read-only, cannot modify job)
- [ ] Admin page showing list of recently reviewed jobs with key metrics
- [ ] Ability to open the existing LyricsAnalyzer in read-only mode with snapshot data
- [ ] Edit log replay/stepper UI to walk through user changes one by one
- [ ] Raw data viewer for edit logs, annotations, and correction diffs
## Technical Approach
### Backend: 2 new admin endpoints
1. **`GET /api/admin/edit-reviews`** - List jobs with completed reviews
- Query Firestore for jobs with `state_data.last_edit_log_path` set (indicating review was performed)
- Filter by status (completed jobs, or optionally all post-review statuses)
- Return summary: job_id, artist, title, user_email, review_completed_at, edit_count, feedback_count
- Support pagination, search, exclude_test filtering
2. **`GET /api/admin/edit-reviews/{job_id}`** - Get full review snapshot
- Returns immutable copy of:
- Original corrections (`corrections.json` from GCS)
- Updated corrections (`corrections_updated.json` from GCS)
- Edit log (`edit_log_{session_id}.json` from GCS)
- Annotations (`annotations.json` from GCS)
- Job metadata (artist, title, user_email, timestamps)
- Audio URL (signed, for playback in review UI)
- This is a **read-only** endpoint - no job state mutations
- Uses `require_admin` dependency (not `require_review_auth`)
- Does NOT transition job status or modify any state
### Frontend: New admin page + Read-only review mode
1. **`/admin/edit-reviews/page.tsx`** - List page
- Table of recently reviewed jobs
- Columns: Artist/Title, User, Review Date, Edits Made, Feedback Given, Actions
- "View Review" button → opens read-only lyrics review
- "View Edit Log" button → opens edit replay view
- "View Raw Data" button → shows raw JSON
2. **Read-only review viewer** - Reuse LyricsAnalyzer
- Load data from the new admin endpoint (not the regular review endpoint)
- Pass `isReadOnly={true}` and `apiClient={null}` to LyricsAnalyzer
- Header shows "Admin Review - Read Only" indicator
- No submit/save buttons, no state transitions
- Audio playback still works (via signed URL)
3. **Edit log replay** - New component
- Timeline/stepper showing each edit entry
- For each entry: operation type, text before → text after, timestamp
- Step forward/backward through edits
- Show feedback reason if user provided one
- Summary stats: total edits, edits with feedback, correction types breakdown
4. **Raw data viewer** - JSON viewer tabs
- Tab 1: Edit Log (formatted JSON with syntax highlighting)
- Tab 2: Annotations
- Tab 3: Corrections diff (original vs updated)
### Key safety guarantee
The admin review endpoint serves a **snapshot** of the data. It reads from GCS files that were already written. The LyricsAnalyzer receives `apiClient={null}` so it physically cannot call any mutation endpoints. Even if code changes in the future, the admin endpoint itself never writes to Firestore or GCS.
## Implementation Steps
1. [ ] **Backend: Add `GET /api/admin/edit-reviews` endpoint** - List reviewed jobs with edit log metadata
2. [ ] **Backend: Add `GET /api/admin/edit-reviews/{job_id}` endpoint** - Serve full immutable review snapshot
3. [ ] **Frontend: Add admin API client methods** - `adminApi.listEditReviews()`, `adminApi.getEditReview(jobId)`
4. [ ] **Frontend: Create `/admin/edit-reviews/page.tsx`** - List page with table
5. [ ] **Frontend: Add sidebar nav item** - "Edit Reviews" with BookOpen icon
6. [ ] **Frontend: Create read-only review viewer** - Load snapshot data into LyricsAnalyzer with isReadOnly
7. [ ] **Frontend: Create edit log replay component** - Step-through UI for edit entries
8. [ ] **Frontend: Create raw data viewer** - JSON viewer with tabs
9. [ ] **Tests: Backend unit tests** for new admin endpoints
10. [ ] **Tests: Frontend component tests** for new page
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/api/routes/admin.py` | Modify | Add edit-reviews endpoints |
| `frontend/lib/api.ts` | Modify | Add adminApi methods for edit reviews |
| `frontend/app/admin/edit-reviews/page.tsx` | Create | Main edit reviews list page |
| `frontend/components/admin/edit-review-viewer.tsx` | Create | Read-only review wrapper using LyricsAnalyzer |
| `frontend/components/admin/edit-log-replay.tsx` | Create | Edit log stepper/replay component |
| `frontend/components/admin/raw-data-viewer.tsx` | Create | JSON viewer with tabs |
| `frontend/components/admin/admin-sidebar.tsx` | Modify | Add Edit Reviews nav item |
| `backend/tests/test_admin_edit_reviews.py` | Create | Backend tests |
## Testing Strategy
- Unit tests for backend endpoints (mock Firestore + GCS)
- Verify admin auth is required on all new endpoints
- Verify no write operations occur on read-only endpoints
- Frontend component tests for key interactions
## Open Questions
- [ ] Should we show ALL jobs that went through review, or only ones with edit logs? (Recommendation: only those with edit logs, since those are the interesting ones)
- [ ] Do we need audio playback in the admin review viewer? (Recommendation: yes, it helps understand what the user heard)
- [ ] Should the raw data be downloadable? (Nice-to-have, can add later)
## Rollback Plan
All changes are additive (new endpoints, new pages). Removing the sidebar link effectively hides the feature. No existing behavior is modified.
-151
View File
@@ -1,151 +0,0 @@
# Plan: AI-Powered Credit Gating
**Created:** 2026-03-23
**Branch:** feat/sess-20260323-1716-ai-credit-gating
**Status:** Implemented
## Overview
Replace the automatic welcome/feedback credit grants with an AI-powered evaluation step. When a user triggers a credit grant (email verification or feedback submission), instead of immediately granting credits, we collect abuse signals and ask Gemini to make a grant/deny decision. Suspected abusers get a friendly rejection email (CC'd to Andrew) instead of free credits.
## Requirements
- [ ] Welcome credit grants go through AI evaluation before granting
- [ ] Feedback credit grants go through AI evaluation before granting
- [ ] AI evaluates: fingerprint correlations, IP correlations, IP geo info, user agent, signup recency patterns
- [ ] Approved users get credits as before (seamless experience)
- [ ] Rejected users get a friendly-but-firm email explaining the situation, CC'd to andrew@beveridge.uk
- [ ] Rejection email invites them to reply if it's a mistake
- [ ] Decision is logged to Firestore for audit trail
- [ ] Fail-open: if AI evaluation fails, grant credits (don't punish users for our errors)
- [ ] Admin can override decisions via existing admin credit grant endpoint
## Technical Approach
### Architecture
```
User verifies email / submits feedback
↓
Collect abuse signals (sync, fast):
- User's fingerprint + IP
- Other accounts with same fingerprint (from correlations)
- Other accounts with same IP
- IP geolocation info
- Recent signup count from same IP/fingerprint (last 24h)
- User agent string
- For feedback: the feedback content itself
↓
Call Gemini (vertexai/gemini-2.5-pro-preview) with structured prompt:
- Present all signals
- Ask for JSON response: {decision: "grant" | "deny", reasoning: string, confidence: number}
↓
If "grant": grant credits normally, log decision
If "deny": skip credit grant, send rejection email, log decision
If AI call fails: grant credits (fail-open), log error
```
### Key Design Decisions
1. **Two-phase verification**: The magic link verify endpoint returns immediately with session + a `credit_status: "evaluating"` field. The frontend shows a "preparing your account" loading screen, then polls/waits for the evaluation result via a new lightweight endpoint. This keeps the auth flow snappy while the AI evaluation runs in the background of the same request or via a short follow-up call. The frontend then shows either a celebratory "2 free credits!" interstitial or a "sorry, couldn't grant credits" interstitial before navigating to the dashboard.
2. **Fail-open**: If Gemini is down or returns an error, we grant credits. Better to let an occasional abuser through than block legitimate users.
3. **Use google-genai directly**: The project already has `langchain-google-genai` installed. We can use the simpler `google.generativeai` library directly for a focused prompt/response, or use the existing LangChain setup. We'll use `google.generativeai` for simplicity since this is a single-turn evaluation, not an agentic workflow.
4. **Structured JSON output**: Ask Gemini for structured output with decision, reasoning, and confidence to make decisions parseable and auditable.
5. **Log all decisions**: Store every evaluation in a `credit_evaluations` Firestore collection for audit trail and to tune the prompt over time.
## Implementation Steps
### Step 1: Add google-generativeai dependency
- Add `google-generativeai` to pyproject.toml (lightweight, direct Gemini access)
- Verify it works with Vertex AI auth in the nomadkaraoke project
### Step 2: Create credit evaluation service
- New file: `backend/services/credit_evaluation_service.py`
- `CreditEvaluationService` class with:
- `evaluate_credit_grant(email, grant_type, feedback_content=None) -> CreditEvaluation`
- Collects all abuse signals by calling existing services (user_service, ip_geolocation_service)
- Builds prompt with signals
- Calls Gemini for decision
- Logs decision to Firestore
- Returns structured result
### Step 3: Build the Gemini prompt
- System prompt explaining the role: "You are an anti-abuse evaluator for a karaoke video service..."
- Present all signals in a structured format
- Request JSON response: `{decision: "grant"|"deny", reasoning: "...", confidence: 0.0-1.0}`
- Include clear guidelines: same fingerprint across accounts = very suspicious, same IP = somewhat suspicious, etc.
### Step 4: Create rejection email template
- Add `send_credit_denied_email(email, reasoning)` to EmailService
- Friendly-but-firm tone
- CC andrew@beveridge.uk
- Invite reply if it's a mistake
- Use existing email template infrastructure (branding, styling)
### Step 5: Integrate into welcome credit flow
- Modify `grant_welcome_credits_if_eligible()` in user_service.py
- Before granting: call `CreditEvaluationService.evaluate_credit_grant(email, "welcome")`
- If denied: send rejection email, return False
- If error: log warning, grant anyway (fail-open)
### Step 6: Integrate into feedback credit flow
- Modify feedback endpoint in users.py
- Before granting credits: evaluate with feedback content included
- If denied: return different response message, send rejection email
- If error: grant anyway
### Step 7: Add Firestore collection for audit trail
- Collection: `credit_evaluations`
- Document fields: email, grant_type, decision, reasoning, confidence, signals_snapshot, model_used, evaluated_at
- TTL: keep indefinitely (small volume, high audit value)
### Step 8: Tests
- Unit tests for signal collection
- Unit tests for prompt building
- Unit tests for decision parsing (grant/deny/error cases)
- Integration test for the evaluation service with mocked Gemini
- Integration test for welcome credit flow with evaluation
- Integration test for feedback credit flow with evaluation
- Test fail-open behavior
### Step 9: Configuration
- Add env vars: `CREDIT_EVAL_MODEL` (default: "gemini-2.5-pro-preview"), `CREDIT_EVAL_ENABLED` (default: true)
- Add to config.py
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `pyproject.toml` | Modify | Add google-generativeai dependency |
| `backend/config.py` | Modify | Add credit evaluation config vars |
| `backend/services/credit_evaluation_service.py` | Create | New service: collect signals, call Gemini, log decision |
| `backend/services/user_service.py` | Modify | Integrate evaluation into welcome credit grant |
| `backend/api/routes/users.py` | Modify | Integrate evaluation into feedback credit grant |
| `backend/services/email_service.py` | Modify | Add credit denied email template |
| `backend/tests/test_credit_evaluation.py` | Create | Tests for evaluation service |
| `backend/tests/test_anti_abuse.py` | Modify | Update welcome credit tests for evaluation |
| `frontend/app/auth/verify/page.tsx` | Modify | Add "preparing account" loading + credit result interstitial |
| `frontend/lib/api.ts` | Modify | Add credit evaluation status endpoint |
## Testing Strategy
- **Unit tests**: Signal collection, prompt building, response parsing, fail-open behavior
- **Integration tests**: Full evaluation flow with mocked Gemini responses
- **E2E tests**: Manual verification in production with test accounts
- **Edge cases**: Gemini timeout, malformed response, missing signals, first-ever user (no correlations)
## Open Questions
1. **Gemini model**: Using `gemini-3.1-pro-preview` as specified — already available in the nomadkaraoke GCP project.
2. **Latency UX**: Frontend shows a friendly "preparing your account" loading screen while evaluation runs. Then transitions to either a celebratory "2 free credits!" message (with gentle reminder not to abuse) or a "sorry, couldn't grant free credits" message with a buy credits CTA. User clicks through to dashboard from either.
3. **Should we also gate the existing rate limiter?** Currently `is_signup_rate_limited()` silently blocks at 2 accounts per IP/fingerprint per 24h. The AI evaluation is complementary — rate limiter blocks obvious rapid abuse, AI evaluation catches more subtle patterns.
## Rollback Plan
Set `CREDIT_EVAL_ENABLED=false` in Cloud Run env vars → immediate rollback to auto-grant behavior. No data model changes needed. The evaluation service is purely additive.
@@ -1,150 +0,0 @@
# Plan: Block Disposable Email Registration
**Created:** 2026-03-19
**Branch:** feat/sess-20260319-1332-disposable-email-abuse
**Status:** Draft
## Overview
Users are signing up with disposable email addresses (yopmail.com, emailhook.site, etc.) to get unlimited free credits (2 per account). The `EmailValidationService` already exists with 160+ disposable domains but is **never called during signup**. We need to wire it into the magic link flow and show a friendly rejection message.
## Context
- The `EmailValidationService` (`backend/services/email_validation_service.py`) is fully implemented with disposable domain detection, email/IP blocklists, and admin management APIs
- The admin blocklist management routes exist (`backend/api/routes/rate_limits.py`) and work
- The magic link endpoint (`POST /api/users/auth/magic-link` in `backend/api/routes/users.py:133`) **never calls** the validation service
- Currently 1 yopmail user and 1 emailhook.site user in `gen_users`
- Each new account gets 2 free welcome credits automatically
## Requirements
- [x] Reject disposable email domains at the magic link endpoint (backend)
- [x] Return a **specific, distinguishable error** so the frontend can show a friendly message (not the generic "email enumeration" response)
- [x] Show a warm, friendly message in the AuthDialog explaining why and encouraging real email use
- [x] Add `emailhook.site` to the default disposable domains list
- [x] Add backend unit tests for the integration
- [x] Add frontend test coverage for the new error state
## Design Decision: Explicit Rejection vs Silent Failure
The current magic link endpoint returns a generic success message to prevent email enumeration. For disposable emails, we will **explicitly reject** with a 422 status and a specific error message. This is the right trade-off because:
1. We *want* the user to know their disposable email won't work (so they can use a real one)
2. Disposable domain detection is not sensitive information — the blocklist is public knowledge
3. The friendly message serves our goal of converting them to real email users
## Implementation Steps
### Step 1: Backend — Add disposable email check to magic link endpoint
**File:** `backend/api/routes/users.py`
Add the `EmailValidationService` check **after** email lowercasing (line 158) and **before** tenant validation (line 164). Return a 422 with a friendly message.
```python
# Check for disposable email domains
email_validation = get_email_validation_service()
if email_validation.is_disposable_domain(email):
logger.warning(f"Blocked disposable email signup attempt: {_mask_email(email)}")
raise HTTPException(
status_code=422,
detail="disposable_email_not_allowed"
)
```
Use a machine-readable error code (`disposable_email_not_allowed`) so the frontend can match on it and show the custom friendly message. This keeps the backend response lean and puts the UX copy in the frontend where it belongs.
### Step 2: Backend — Also check email and IP blocklists
In the same location, add checks for explicitly blocked emails and IPs:
```python
if email_validation.is_email_blocked(email):
logger.warning(f"Blocked email signup attempt: {_mask_email(email)}")
# Silent reject — return generic success to prevent enumeration
return SendMagicLinkResponse(
status="success",
message="If this email is registered, you will receive a sign-in link shortly."
)
ip_address = http_request.client.host if http_request.client else None
if ip_address and email_validation.is_ip_blocked(ip_address):
logger.warning(f"Blocked IP signup attempt: {ip_address}")
return SendMagicLinkResponse(
status="success",
message="If this email is registered, you will receive a sign-in link shortly."
)
```
Note: Blocked emails/IPs get silent rejection (anti-enumeration). Disposable domains get explicit rejection (we want the user to switch to a real email).
### Step 3: Backend — Add `emailhook.site` to default disposable domains
**File:** `backend/services/email_validation_service.py`
Add `emailhook.site` to `DEFAULT_DISPOSABLE_DOMAINS` set.
### Step 4: Frontend — Handle the disposable email error in AuthDialog
**File:** `frontend/components/auth/AuthDialog.tsx`
The `sendMagicLink` call in `auth.ts` already catches errors and sets `error` from the API response message. The `handleResponse` in `api.ts` throws an `ApiError` with the `detail` string for non-200 responses.
Update `handleSendMagicLink` to detect the `disposable_email_not_allowed` error code and show a friendly multi-line message instead of the terse error string:
```tsx
const success = await sendMagicLink(email.trim().toLowerCase())
if (success) {
setStep("sent")
}
// If error contains our specific code, override with friendly message
// (the error state is already set by the auth store)
```
We need to detect the specific error. The cleanest approach: check the error string in the `displayError` rendering. If it matches `disposable_email_not_allowed`, render a custom friendly block instead of the red error text.
The friendly message:
> **We don't support disposable email addresses**
>
> We'd love to help you create karaoke videos! Please use your regular email address so we can send you a notification when your video is ready.
>
> We promise — no spam, no marketing emails, ever. We only email you about your karaoke videos.
### Step 5: Backend tests
**File:** `backend/tests/test_email_validation_integration.py` (new)
Test cases:
1. Magic link with disposable email returns 422 with `disposable_email_not_allowed`
2. Magic link with legitimate email proceeds normally
3. Magic link with blocked email returns silent success (no actual email sent)
4. Magic link with blocked IP returns silent success
### Step 6: Frontend tests
**File:** `frontend/components/__tests__/AuthDialog.test.tsx` (new or extend existing)
Test cases:
1. Submitting a disposable email shows the friendly rejection message
2. The friendly message contains the key copy (no spam promise, use real email)
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/api/routes/users.py` | Modify | Add email validation checks to magic link endpoint |
| `backend/services/email_validation_service.py` | Modify | Add `emailhook.site` to default domains |
| `frontend/components/auth/AuthDialog.tsx` | Modify | Handle disposable email error with friendly message |
| `backend/tests/test_email_validation_integration.py` | Create | Integration tests for the validation in signup flow |
## Testing Strategy
- **Unit tests:** Validate `is_disposable_domain` catches yopmail, emailhook.site (already covered in existing tests)
- **Integration tests:** New test file verifying the magic link endpoint rejects disposable emails with 422
- **Frontend tests:** Verify the AuthDialog shows friendly message on disposable email error
- **Manual verification:** Try signing up with `test@yopmail.com` on dev server to see the UX
## Rollback Plan
Remove the validation check from `send_magic_link()` in `users.py`. The `EmailValidationService` itself remains unchanged (it's already deployed and used by admin routes).
@@ -1,111 +0,0 @@
# Plan: Credit Refund on Job Deletion and Cancellation
**Created:** 2026-03-04
**Branch:** feat/sess-20260304-1308-investigate-user-credits
**Status:** Implemented
## Overview
When a user deletes or cancels a job, their credit is not refunded. This was discovered when user john2011@sky.com deleted a job 16 seconds after creation (accidental duplicate) and lost the credit. Job *failures* already auto-refund via `mark_job_failed()` → `refund_credit()`, but deletions and cancellations do not.
## Requirements
- [x] Job failures already refund credits (existing behavior, no change needed)
- [ ] Job cancellation refunds 1 credit
- [ ] Job deletion refunds 1 credit (if not already completed)
- [ ] No refund for completed jobs (user got their video)
- [ ] No refund for admin-owned jobs (consistent with failure refund logic)
- [ ] Idempotency: prevent double-refund if a job is cancelled then deleted, or fails then deleted
- [ ] Credit transaction records the reason clearly ("job_cancelled", "job_deleted")
- [ ] Refund failure must not block the cancel/delete operation
## Technical Approach
### Key Insight: Idempotency via Job Field
The current `refund_credit()` has no guard against double-refund for the same job. Since a job can transition through cancelled → deleted, or failed → deleted, we need a way to track that a refund was already issued.
**Approach**: Add a `credit_refunded` boolean field to the Job model. Check it before refunding. This is simpler and more reliable than scanning the user's transaction history.
### What Should and Shouldn't Be Refunded
| Job Status at Time of Action | Delete | Cancel | Notes |
|------------------------------|--------|--------|-------|
| PENDING through pre-COMPLETE | Refund | Refund | Job didn't deliver value |
| COMPLETE / PREP_COMPLETE | No refund | N/A (can't cancel terminal) | User got their output |
| FAILED | No refund (already refunded) | N/A (can't cancel terminal) | Already refunded by fail_job |
| CANCELLED | No refund (already refunded) | N/A (already cancelled) | Already refunded by cancel |
For deletion: we need to read the job status *before* deleting it to decide whether to refund. The delete endpoint already fetches the job for ownership check, so this is straightforward.
### Refund Helper
Extract the refund logic (currently in `mark_job_failed`) into a reusable private method `_refund_credit_for_job()` that:
1. Checks `job.credit_refunded` — skip if already refunded
2. Checks `job.user_email` — skip if no user
3. Checks if admin — skip if admin job
4. Calls `user_service.refund_credit()` with appropriate reason
5. Sets `credit_refunded = True` on the job (for cancel; for delete the job is about to be removed but we set it anyway for the brief window)
6. Wraps everything in try/except so failures don't block the parent operation
## Implementation Steps
1. [ ] **Add `credit_refunded` field to Job model** (`backend/models/job.py`)
- Add `credit_refunded: bool = False` to the Job model
- No migration needed — Firestore is schemaless, defaults to False for existing docs
2. [ ] **Extract refund helper in JobManager** (`backend/services/job_manager.py`)
- Create `_refund_credit_for_job(self, job_id: str, job: Job, reason: str) -> bool`
- Move the refund logic from `mark_job_failed` (lines 821-839) into this helper
- Add `credit_refunded` check at the top
- Update `mark_job_failed` to call the helper with `reason="job_failed"`
3. [ ] **Add refund to `cancel_job()`** (`backend/services/job_manager.py`)
- After setting status to CANCELLED, call `_refund_credit_for_job(job_id, job, reason="job_cancelled")`
- The job object is already fetched at line 1017
4. [ ] **Add refund to `delete_job()`** (`backend/services/job_manager.py`)
- Need to fetch the job (already done when `delete_files=True`, but not always)
- Before deletion, check if job status is NOT in `[COMPLETE, PREP_COMPLETE, FAILED, CANCELLED]`
- If eligible, call `_refund_credit_for_job(job_id, job, reason="job_deleted")`
- For FAILED/CANCELLED: skip because refund was already issued during that transition
- For COMPLETE/PREP_COMPLETE: skip because user received their output
5. [ ] **Update `mark_job_failed` to set `credit_refunded`**
- The helper handles this, but also update the Firestore doc
6. [ ] **Add unit tests** (`backend/tests/test_credit_enforcement.py`)
- Test: cancel_job refunds credit for non-admin user
- Test: cancel_job skips refund for admin jobs
- Test: delete_job refunds credit for non-complete, non-terminal jobs
- Test: delete_job does NOT refund for completed jobs
- Test: delete_job does NOT double-refund for already-failed jobs
- Test: delete_job does NOT double-refund for already-cancelled jobs
- Test: refund failure doesn't block cancel/delete
- Test: credit_refunded flag prevents double refund
7. [ ] **Backfill: Refund john2011@sky.com** (manual one-time)
- Grant 1 credit for the deleted job 40dcca89 via admin API or direct Firestore update
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Add `credit_refunded: bool = False` field |
| `backend/services/job_manager.py` | Modify | Extract `_refund_credit_for_job()` helper, add refund to `cancel_job()` and `delete_job()` |
| `backend/tests/test_credit_enforcement.py` | Modify | Add tests for cancel/delete refund scenarios |
## Testing Strategy
- **Unit tests**: Extend `test_credit_enforcement.py` with new test classes `TestCreditRefundOnJobCancellation` and `TestCreditRefundOnJobDeletion`, following the exact pattern of existing `TestCreditRefundOnJobFailure`
- **Manual verification**: After deploy, cancel and delete test jobs and verify credit balance updates
## Open Questions
- [ ] Should we notify the user (email/in-app) when a credit is refunded? (Suggest: no, keep it simple for now — the credit just appears back in their balance)
## Rollback Plan
- The `credit_refunded` field is additive (defaults to False for old jobs)
- If issues arise, revert the PR — old behavior (no refund on cancel/delete) is restored
- Any refunds already issued are just bonus credits to users, no harm done
-407
View File
@@ -1,407 +0,0 @@
# Plan: Add Retry Logic for Flacfetch Download Failures
**Created:** 2026-02-10
**Branch:** feat/sess-20260210-1158-investigate-job-749141f8
**Status:** Draft
## Overview
Job 749141f8 failed with error: "Audio download failed: Remote download by ID failed: Download by ID request failed:"
Investigation revealed **multiple related issues** when flacfetch network calls fail:
1. **No retry logic** - A single network hiccup causes immediate job failure
2. **Missing source info** - When download fails, source_id/source_name aren't saved to job
3. **Broken retry button** - Can't retry because no source info was persisted
4. **Silent UI failures** - Frontend doesn't show retry error messages to user
### Root Cause
Audio search flow saves selection info to `state_data`, but doesn't save download parameters (source_id, source_name, etc.) to the job document until AFTER download succeeds. If the flacfetch call fails (network error), the job fails with no way to retry.
### Solution
This plan fixes all four issues:
- Add retry logic to flacfetch client (network resilience)
- Persist download info before attempting download (enables retry)
- Update retry endpoint to handle audio search jobs
- Show error messages in frontend when retry fails
## Requirements
### Functional Requirements - Flacfetch Retry Logic
- [x] Retry `download_by_id()` on `httpx.RequestError` (network failures)
- [x] Use exponential backoff spread over ~5 minutes
- [x] Configurable max retries via environment variable
- [x] Log each retry attempt with context
- [x] Do NOT retry on HTTP 4xx errors (client errors - immediate failure)
- [x] Do retry on HTTP 5xx errors (server errors - transient)
### Functional Requirements - Persist Download Info
- [x] Save source_id, source_name, target_file, download_url to job document
- [x] Save BEFORE attempting download (not after)
- [x] Store in top-level fields for easy retry access
- [x] Preserve existing state_data fields (selected_audio_index, etc.)
### Functional Requirements - Retry Endpoint
- [x] Add case for audio search jobs in `/jobs/{id}/retry` endpoint
- [x] Check for saved download params (source_id, source_name)
- [x] Re-download using saved params
- [x] Clear error state before retrying
- [x] Trigger appropriate workers after successful download
### Functional Requirements - Frontend Error Handling
- [x] Show toast notification when retry fails
- [x] Display actual error message from API
- [x] Don't just console.error - make it visible to user
### Non-Functional Requirements
- [x] Minimal code changes (use existing patterns)
- [x] Clear error messages that include retry context
- [x] Backwards compatible (no breaking changes)
- [x] Should not significantly increase total timeout duration
## Technical Approach
### Architecture Decisions
**Option 1: Tenacity library (Recommended)**
- ✅ Industry standard, well-tested
- ✅ Declarative retry configuration
- ✅ Built-in exponential backoff
- ✅ Already used in project (check `poetry.lock`)
- ❌ New dependency if not already present
**Option 2: Manual retry loop**
- ✅ No new dependencies
- ✅ Full control over logic
- ❌ More code to maintain
- ❌ Easy to get wrong (edge cases)
**Decision:** Use **tenacity** if already in dependencies, otherwise implement manual retry loop.
### Retry Strategy
```python
# Retry on network errors only (RequestError)
# Do NOT retry on:
# - HTTP 4xx (client errors like 400, 401, 404)
# - Already completed downloads
# Do retry on:
# - httpx.RequestError (connection timeout, refused, DNS failure)
# - HTTP 5xx (server errors - service temporarily down)
Max retries: 8 (total 9 attempts)
Backoff: Exponential with cap, multiplier=2, min=10s, max=60s
- Attempt 1: immediate
- Attempt 2: wait 10s
- Attempt 3: wait 20s
- Attempt 4: wait 40s
- Attempt 5: wait 60s (capped)
- Attempt 6: wait 60s (capped)
- Attempt 7: wait 60s (capped)
- Attempt 8: wait 60s (capped)
- Attempt 9: wait 60s (capped)
Total max time: ~370s (6+ minutes) of backoff + request timeouts
This gives the flacfetch service time to recover from restarts,
network issues, or temporary VM problems.
```
### Configuration
Add to `backend/config.py`:
```python
# Flacfetch client retry settings
flacfetch_retry_max_attempts: int = int(os.getenv("FLACFETCH_RETRY_MAX_ATTEMPTS", "9")) # Total attempts (1 initial + 8 retries)
flacfetch_retry_min_wait: float = float(os.getenv("FLACFETCH_RETRY_MIN_WAIT", "10.0")) # seconds
flacfetch_retry_max_wait: float = float(os.getenv("FLACFETCH_RETRY_MAX_WAIT", "60.0")) # seconds
```
### Error Message Improvements
Change from:
```
Download by ID request failed:
```
To:
```
Download by ID request failed after 4 attempts: [ConnectTimeout] Connection to http://10.0.0.5:8080 timed out
```
## Implementation Steps
### Phase 1: Check Dependencies
1. [ ] Check if `tenacity` is in `pyproject.toml` dependencies
2. [ ] If not present, add `tenacity = "^8.2.0"` to dependencies
3. [ ] Run `poetry lock` if dependencies changed
### Phase 2: Add Configuration
4. [ ] Add retry configuration to `backend/config.py`
- `flacfetch_retry_max_attempts = 9`
- `flacfetch_retry_min_wait = 10.0`
- `flacfetch_retry_max_wait = 60.0`
### Phase 3: Implement Flacfetch Retry Logic
5. [ ] Import tenacity in `backend/services/flacfetch_client.py`
6. [ ] Create retry decorator with:
- Retry on `httpx.RequestError`
- Retry on `httpx.HTTPStatusError` with 5xx status
- Stop on `httpx.HTTPStatusError` with 4xx status
- Exponential backoff with cap at 60s
- Before-sleep logging callback
7. [ ] Apply decorator to `download_by_id()` method
8. [ ] Improve error messages to include:
- Number of attempts made
- Original exception type and message
### Phase 4: Persist Download Info Before Download
9. [ ] Update `audio_search.py:_download_and_start_processing()`
10. [ ] After getting source info from selected result (line 307-311)
11. [ ] **Save to job document BEFORE download attempt:**
```python
job_manager.update_job(job_id, {
'audio_source_type': 'audio_search',
'source_name': source_name,
'source_id': source_id,
'target_file': target_file,
'download_url': download_url,
})
```
12. [ ] Then proceed with download (existing code)
13. [ ] Add to Job model if needed (check if fields exist)
### Phase 5: Update Retry Endpoint for Audio Search
14. [ ] Update `backend/api/routes/jobs.py:retry_job()`
15. [ ] Add new case before "no input audio" error (line ~1088)
16. [ ] Check if job has saved download params:
```python
elif job.audio_source_type == 'audio_search' and job.source_id:
# Has audio search download info - retry download
```
17. [ ] Re-call audio search download logic with saved params
18. [ ] Or refactor download into reusable function
### Phase 6: Frontend Error Handling
19. [ ] Update `frontend/components/job/JobActions.tsx`
20. [ ] Add toast notification library (check if exists)
21. [ ] In `handleRetry()` catch block, show toast:
```typescript
catch (error) {
const message = error.response?.data?.detail || "Failed to retry job"
toast.error(message)
console.error("Failed to retry job:", error)
}
```
### Phase 7: Testing - Flacfetch Retry
22. [ ] Add unit tests to `backend/tests/test_flacfetch_client.py`
23. [ ] Test retry on RequestError (mock fails N times then succeeds)
24. [ ] Test no retry on 4xx errors
25. [ ] Test retry on 5xx errors
26. [ ] Test max retries exhausted
27. [ ] Verify logging output includes attempt numbers
### Phase 8: Testing - End-to-End
28. [ ] Test audio search job with flacfetch down
29. [ ] Verify source info is saved even when download fails
30. [ ] Verify retry button works after source info saved
31. [ ] Test retry succeeds when flacfetch comes back up
32. [ ] Verify error toast appears when retry fails
### Phase 9: Documentation
33. [ ] Update `backend/services/flacfetch_client.py` docstring
34. [ ] Add to `docs/LESSONS-LEARNED.md`:
- "Always add retries for external service calls"
- "Save operation parameters before attempting operations"
- "Persist enough state to enable retry from any failure point"
35. [ ] Update `docs/ARCHITECTURE.md` if needed
36. [ ] Document the new job fields in API.md
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/config.py` | Modify | Add retry configuration variables |
| `backend/services/flacfetch_client.py` | Modify | Add retry logic to `download_by_id()` |
| `backend/api/routes/audio_search.py` | Modify | Save download info before attempting download |
| `backend/api/routes/jobs.py` | Modify | Add audio search case to retry endpoint |
| `backend/models/job.py` | Modify (maybe) | Add fields: source_name, source_id, target_file, download_url |
| `frontend/components/job/JobActions.tsx` | Modify | Show error toast when retry fails |
| `backend/tests/test_flacfetch_client.py` | Modify | Add retry tests |
| `backend/tests/test_audio_search.py` | Modify | Test download info persistence |
| `backend/tests/test_jobs.py` | Modify | Test retry with audio search jobs |
| `pyproject.toml` | Modify (maybe) | Add tenacity if not present |
| `docs/LESSONS-LEARNED.md` | Modify | Document patterns |
| `docs/API.md` | Modify | Document new job fields |
## Testing Strategy
### Unit Tests (`backend/tests/test_flacfetch_client.py`)
1. **Test retry on RequestError**
```python
@pytest.mark.asyncio
async def test_download_by_id_retries_on_network_error():
"""Should retry 3 times on connection errors."""
client = FlacfetchClient(base_url="http://test", api_key="key")
with patch("httpx.AsyncClient.post") as mock_post:
# Fail 2 times, then succeed
mock_post.side_effect = [
httpx.RequestError("Connection timeout"),
httpx.RequestError("Connection refused"),
Mock(json=lambda: {"download_id": "test123"}, status_code=200),
]
result = await client.download_by_id("YouTube", "abc123")
assert result == "test123"
assert mock_post.call_count == 3
```
2. **Test no retry on 4xx errors**
```python
@pytest.mark.asyncio
async def test_download_by_id_no_retry_on_client_error():
"""Should NOT retry on 400/401/404 errors."""
client = FlacfetchClient(base_url="http://test", api_key="key")
with patch("httpx.AsyncClient.post") as mock_post:
response = Mock(status_code=404, text="Not found")
response.raise_for_status.side_effect = httpx.HTTPStatusError(
"Not found", request=Mock(), response=response
)
mock_post.return_value = response
with pytest.raises(FlacfetchServiceError, match="404"):
await client.download_by_id("YouTube", "abc123")
assert mock_post.call_count == 1 # No retries
```
3. **Test retry on 5xx errors**
```python
@pytest.mark.asyncio
async def test_download_by_id_retries_on_server_error():
"""Should retry on 500/503 errors."""
# Similar to test 1 but with HTTP 503 errors
```
4. **Test max retries exhausted**
```python
@pytest.mark.asyncio
async def test_download_by_id_fails_after_max_retries():
"""Should fail with clear message after max retries."""
# All attempts fail, verify error message includes attempt count
```
### Integration Tests
5. **Test with actual flacfetch service** (if available in test environment)
- Verify retry doesn't break normal operation
- Test with deliberate network issues
### Manual Testing
6. **Simulate network failure**
```bash
# Temporarily stop flacfetch VM
gcloud compute instances stop flacfetch-service --zone=us-central1-a
# Submit a test job
# Observe logs showing retry attempts
# Restart VM during retries
gcloud compute instances start flacfetch-service --zone=us-central1-a
# Job should succeed
```
7. **Test with intentional delays**
- Add sleep in flacfetch API handler
- Verify retries don't cause issues
## Open Questions
- [x] ~~Is tenacity already in dependencies?~~ (Will check during implementation)
- [x] ~~What's the appropriate total timeout budget?~~ - 9 attempts over ~6 minutes
- [x] ~~Should we fix the Retry button UX in this PR?~~ - YES, it's the same root cause
- [x] ~~Why did job 749141f8 fail before saving audio source info?~~ - Download info saved AFTER download, not before
- [ ] Should we also add retries to `wait_for_download()`?
- Probably NO - that already has its own polling/timeout logic
- [ ] Should we add circuit breaker pattern?
- Probably NO for now - keep it simple
- Could add in future if we see cascading failures
- [ ] Should retry logic be applied to other flacfetch methods?
- `start_download()` - probably YES (same pattern)
- `search()` - maybe YES (search can also have network issues)
- `get_download_status()` - probably NO (polling already handles this)
- **Decision**: Apply to `start_download()` too, skip `search()` for now
- [ ] Which toast library should frontend use?
- Check if already exists (react-hot-toast, sonner, etc.)
- If not, add lightweight library
- [ ] Should we save download params to state_data or top-level fields?
- Top-level fields more visible and easier to query
- **Decision**: Top-level fields (source_id, source_name, etc.)
## Rollback Plan
If this causes issues:
1. **Quick rollback**: Set env var `FLACFETCH_RETRY_MAX_ATTEMPTS=1` in Cloud Run
- Disables retries without code change
- Can deploy in ~2 minutes
2. **Full rollback**: Revert the PR
- Git revert the merge commit
- Deploy previous version
3. **Monitoring**: Watch for:
- Increased request duration to flacfetch
- Jobs timing out at higher layer (Cloud Tasks, Cloud Run)
- Duplicate download attempts causing issues
## Success Criteria
- [ ] Jobs with transient flacfetch failures succeed instead of failing
- [ ] Error messages are more informative (show attempts and root cause)
- [ ] No increase in false positives (jobs that should fail, now hang)
- [ ] All tests pass
- [ ] No impact on successful jobs (retry overhead is minimal)
## Related Issues
### Job 749141f8 - The Incident
**What happened:**
- User searched for audio via audio search
- Selected "Elizabeth Moen - Songbird"
- Backend tried to download via flacfetch
- Flacfetch call failed with network error (connection timeout/refused)
- Job immediately failed
- No source info saved → Retry button broken
- User sees "Retry" button but clicking does nothing (silent 400 error)
**How this fix helps:**
1. ✅ Flacfetch retry logic → Download would have succeeded after retry
2. ✅ Persist download info → Even if all retries fail, user can retry later
3. ✅ Retry endpoint fix → Retry button will actually work
4. ✅ Frontend toast → User sees helpful error if retry still fails
**Note:** Job 749141f8 cannot be fixed retroactively (no source info saved). But future jobs with same issue will be fixable via retry button.
### Future Work (Not in This PR)
Similar pattern should be applied to other external service calls:
- AudioShake API (lyrics transcription)
- Modal API (audio separation)
- YouTube direct API calls (not via flacfetch)
- Email sending (SendGrid)
## References
- [Tenacity documentation](https://tenacity.readthedocs.io/)
- [LESSONS-LEARNED.md](docs/LESSONS-LEARNED.md) - External service response format mismatches
- [flacfetch README](../../flacfetch/README.md) - Understanding the service
@@ -1,149 +0,0 @@
# Plan: Improve Job Observability
**Created:** 2026-03-07
**Branch:** feat/sess-20260307-2216-improve-job-observability
**Status:** Draft
## Overview
After shipping the "edit completed tracks" feature, we found it very difficult to trace what actually happened to edited jobs. Timeline events are too sparse (no output details), cleanup operations aren't logged to Cloud Logging or Firestore, state_data gets fully overwritten during re-processing (losing history of previous outputs), and API endpoints don't write to the job's Firestore log subcollection.
The goal is to make it trivially easy for future agents and humans to reconstruct the complete lifecycle of any job by reading its timeline and Firestore logs.
## Problem Analysis
### What we found investigating two edited jobs:
1. **Timeline events only have `status`, `timestamp`, `progress`, `message`** — no output details like YouTube video IDs, brand codes, Dropbox links, or GDrive file IDs
2. **Edit endpoint cleanup results never logged** — the `cleanup_results` dict is built and returned in the HTTP response but never logged to Cloud Logging or the job's Firestore log subcollection
3. **Distribution details incomplete in logs** — YouTube logs the URL but not video_id separately; Dropbox doesn't log folder path; GDrive doesn't log file IDs
4. **`state_data` fully overwritten on re-processing** — `video_worker.py:342-354` does `{**job.state_data, ...}`, so any `DELETE_FIELD` operations from the edit endpoint get lost when old keys are re-merged from the in-memory copy
5. **`outputs_deleted_at` cleared by video worker** — intentionally, but loses audit trail
6. **No Firestore job logs from API endpoints** — edit/admin endpoints only log to Cloud Run stdout, not to `jobs/{job_id}/logs/`
## Requirements
- [ ] Add `metadata` dict to `TimelineEvent` for structured output details
- [ ] Log all cleanup operations (success AND failure) to both Cloud Logging and Firestore job logs
- [ ] Log distribution output details (YouTube video ID, Dropbox path, brand code, GDrive file IDs) in timeline events and Firestore logs
- [ ] Preserve previous output details in timeline when a job is edited (snapshot before cleanup)
- [ ] Add helper to write Firestore job logs from API endpoints (not just workers)
- [ ] Ensure edit/admin delete endpoints write to Firestore job logs
- [ ] No breaking changes to existing timeline format (additive only)
## Technical Approach
### 1. Add `metadata` field to `TimelineEvent`
The `TimelineEvent` model gains an optional `metadata: Optional[Dict[str, Any]]` field. This allows any timeline event to carry structured data without changing the schema. The field is additive — existing events without metadata continue to work.
Key events that gain metadata:
- **`complete`**: `{youtube_url, youtube_video_id, dropbox_link, brand_code, gdrive_file_ids}`
- **Edit initiated**: `{previous_outputs: {youtube_url, dropbox_link, brand_code, gdrive_files}, cleanup_results: {...}}`
- **Outputs deleted**: `{deleted_outputs: {...}, cleanup_results: {...}}`
### 2. Write Firestore job logs from API endpoints
Create a simple helper function that API route handlers can call to append logs to the job's Firestore subcollection. Workers already have `JobLogAdapter` — we just need a lightweight equivalent for API routes:
```python
def log_to_job(job_id: str, worker: str, level: str, message: str, metadata: dict = None):
"""Write a log entry to the job's Firestore log subcollection."""
entry = WorkerLogEntry.create(job_id, worker, level, message, metadata)
FirestoreService().append_log_to_subcollection(job_id, entry)
```
### 3. Enhance distribution logging in video_worker_orchestrator
After each distribution upload succeeds, log the details with structured metadata:
- YouTube: video_id, video_url, channel
- Dropbox: folder_path, sharing_link
- GDrive: file_ids dict
- Brand code: code, prefix, allocated vs recycled
### 4. Snapshot previous outputs on edit
Before the edit endpoint clears state_data distribution keys, capture the current values in the timeline event's metadata. This preserves the history permanently in the timeline array (which is append-only via `ArrayUnion`).
### 5. Log cleanup results from edit/admin endpoints
After cleanup runs, log each step's result to both Cloud Logging and the Firestore job log subcollection with structured metadata.
## Implementation Steps
### Step 1: Add `metadata` to `TimelineEvent` model
- [ ] Add `metadata: Optional[Dict[str, Any]] = None` to `TimelineEvent` in `backend/models/job.py`
- [ ] No migration needed — Firestore is schemaless, existing events just won't have the field
### Step 2: Create `log_to_job` helper
- [ ] Add `log_to_job()` function in `backend/services/firestore_service.py` (or a small utility module)
- [ ] Takes `job_id`, `worker`, `level`, `message`, optional `metadata`
- [ ] Wraps `WorkerLogEntry.create()` + `append_log_to_subcollection()`
- [ ] Catches and suppresses errors (logging should never break the caller)
### Step 3: Update `transition_to_state` to accept metadata
- [ ] Add optional `timeline_metadata: Optional[Dict[str, Any]] = None` parameter to `job_manager.transition_to_state()`
- [ ] Pass through to `update_job_status()` → `TimelineEvent` construction
- [ ] Update `firestore_service.update_job_status()` to accept and pass `timeline_metadata`
### Step 4: Enhance edit endpoint logging
- [ ] Before cleanup: snapshot current distribution outputs (`youtube_url`, `dropbox_link`, `brand_code`, `gdrive_files`) into `previous_outputs` dict
- [ ] After each cleanup step: log result to Cloud Logging AND call `log_to_job()`
- [ ] After all cleanup: include `previous_outputs` and `cleanup_results` in the timeline event metadata
- [ ] The "Track edit initiated" timeline event gains metadata: `{initiated_by, previous_outputs, cleanup_results}`
### Step 5: Enhance admin delete-outputs endpoint logging
- [ ] Same pattern as edit endpoint: snapshot outputs, log each cleanup step, include in timeline metadata
- [ ] The "Outputs deleted by admin" timeline event gains metadata: `{deleted_by, deleted_outputs, cleanup_results}`
### Step 6: Enhance distribution logging in video_worker_orchestrator
- [ ] YouTube upload success: log video_id and video_url with metadata `{video_id, video_url}`
- [ ] Dropbox upload success: log folder_path and sharing_link with metadata `{folder_path, sharing_link}`
- [ ] GDrive upload success: log file_ids with metadata `{file_ids: {...}}`
- [ ] Brand code allocation: log with metadata `{brand_code, source: "allocated"|"recycled"}`
### Step 7: Add output details to `complete` timeline event
- [ ] When `video_worker.py` calls `transition_to_state(COMPLETE)`, pass `timeline_metadata` with distribution output details
- [ ] Include: `youtube_url`, `youtube_video_id`, `dropbox_link`, `brand_code`, `gdrive_file_ids`, `edit_count`
### Step 8: Tests
- [ ] Unit test: `TimelineEvent` with metadata serializes correctly
- [ ] Unit test: `log_to_job` writes to subcollection
- [ ] Unit test: edit endpoint includes `previous_outputs` and `cleanup_results` in timeline event
- [ ] Unit test: `complete` transition includes output metadata
- [ ] Verify existing tests still pass
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Add `metadata` field to `TimelineEvent` |
| `backend/services/firestore_service.py` | Modify | Add `log_to_job()` helper, update `update_job_status()` to accept timeline metadata |
| `backend/services/job_manager.py` | Modify | Add `timeline_metadata` param to `transition_to_state()` |
| `backend/api/routes/jobs.py` | Modify | Enhance edit endpoint with cleanup logging, output snapshots, timeline metadata |
| `backend/api/routes/admin.py` | Modify | Enhance delete-outputs endpoint similarly |
| `backend/workers/video_worker_orchestrator.py` | Modify | Add structured metadata to distribution log messages |
| `backend/workers/video_worker.py` | Modify | Pass output metadata in `complete` transition (both orchestrator and legacy paths) |
| `backend/tests/test_edit_completed_track.py` | Modify | Verify timeline metadata in edit tests |
| `backend/tests/test_observability.py` | Create | Tests for TimelineEvent metadata and log_to_job |
## Testing Strategy
- **Unit tests**: TimelineEvent metadata serialization, log_to_job helper, edit endpoint timeline metadata assertions
- **Existing test verification**: `make test` must pass — changes are additive so existing tests should be unaffected
- **No E2E tests needed**: This is purely backend logging/observability — no UI changes
## What This Does NOT Include
- **Dedicated admin UI for viewing job logs** — existing admin page already shows timeline
- **Log aggregation/search UI** — Cloud Logging and Firestore viewer already serve this purpose
- **Changing state_data overwrite behavior** — the `{**job.state_data, ...}` pattern in video_worker.py is intentional; instead we preserve history in timeline events
- **Changing outputs_deleted_at clearing** — video worker clearing it after re-processing is correct behavior; the edit history lives in timeline metadata instead
## Rollback Plan
All changes are additive (new optional field on TimelineEvent, additional log writes). Rolling back just means deploying the previous version — no data migration needed.
## Open Questions
None — the approach is straightforward and low-risk.
-279
View File
@@ -1,279 +0,0 @@
# Plan: Private (Non-Published) Tracks
**Created:** 2026-02-24
**Branch:** feat/sess-20260224-1149-private-tracks
**Status:** Implemented
## Overview
Add a `is_private` flag to jobs that changes distribution behavior:
- **Dropbox**: Uses `Tracks-NonPublished` folder (not `Tracks-Organized`) with brand code `NOMADNP` (not `NOMAD`)
- **YouTube**: Skipped entirely
- **Google Drive**: Skipped entirely
- **Email**: Completion email includes Dropbox link but no YouTube section
- **Discord**: Still notified (for internal tracking)
Available to all users at job creation and toggleable by admin on existing jobs.
## Requirements
- [ ] All users can mark a job as "Private (Non-Published)" when creating it
- [ ] Admin can toggle `is_private` on any existing job via admin dashboard
- [ ] Private jobs upload to Dropbox at `Tracks-NonPublished` with `NOMADNP-XXXX` brand codes
- [ ] Private jobs skip YouTube and Google Drive uploads entirely
- [ ] Completion emails omit YouTube section when no YouTube URL (private jobs + any other edge case)
- [ ] When admin toggles a completed job to private, existing outputs are auto-deleted
- [ ] Discord notifications still fire for private jobs
- [ ] Existing (non-private) jobs are completely unaffected (backward compatible)
## Technical Approach
### Core Design: Single Boolean Flag with Runtime Overrides
Store `is_private: bool = False` on the job model. At distribution time, a helper function returns effective distribution parameters based on this flag. This means:
- **Toggling is_private automatically changes all distribution behavior** - no need to manually update dropbox_path, brand_prefix, enable_youtube_upload, gdrive_folder_id
- **Job fields still store the "normal" defaults** - the override happens at the config-to-orchestrator boundary
- **Admin can flip one field** and everything adapts
### Key Helper Function
```python
# In job_defaults_service.py
def get_effective_distribution_for_job(job) -> EffectiveDistributionSettings:
"""Return distribution settings, applying private overrides if is_private=True."""
if getattr(job, 'is_private', False):
settings = get_settings()
return EffectiveDistributionSettings(
dropbox_path=settings.default_private_dropbox_path,
brand_prefix=settings.default_private_brand_prefix,
enable_youtube_upload=False,
gdrive_folder_id=None,
# Keep other settings from job (discord, youtube_description, etc.)
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=None,
)
# Non-private: return job's own settings
return EffectiveDistributionSettings(
dropbox_path=getattr(job, 'dropbox_path', None),
brand_prefix=getattr(job, 'brand_prefix', None),
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=getattr(job, 'youtube_description_template', None),
)
```
This helper is used in both distribution paths (orchestrator and native fallback).
### Distribution Paths That Need Changes
Two code paths handle distribution (see LESSONS-LEARNED: "Fix Both Sides of Dual Code Paths"):
1. **Orchestrator path** (`video_worker_orchestrator.py`): Primary path. `create_orchestrator_config_from_job()` uses the helper to build config.
2. **Native distribution path** (`video_worker.py` → `_handle_native_distribution()`): Fallback path. Also uses the helper instead of reading job fields directly.
### Email Template Change
Add conditional YouTube section removal in `render_template()`, matching the existing pattern for `feedback_url`:
```python
# Remove YouTube section if no URL provided
if not variables.get("youtube_url"):
result = re.sub(
r"\n*Here's the link for the karaoke video published to YouTube:\n\{youtube_url\}\n*",
'\n',
result,
flags=re.DOTALL
)
variables["youtube_url"] = ""
```
And change `render_job_completion()` to pass `None` instead of `"[YouTube URL not available]"` when youtube_url is empty.
### Admin Toggle Auto-Delete
When admin PATCH updates `is_private` from False to True on a job that has existing outputs, reuse the existing delete-outputs logic to remove YouTube + GDrive + Dropbox outputs. Admin can then "Reprocess" to re-distribute to the private Dropbox path.
## Implementation Steps
### Phase 1: Backend Model & Config (foundation)
1. [ ] **Add `is_private` to Job model** (`backend/models/job.py`)
- Add `is_private: bool = False` field after `made_for_you` field
- Add to `JobCreate` model as well
2. [ ] **Add private distribution config** (`backend/config.py`)
- Add `default_private_dropbox_path: Optional[str] = os.getenv("DEFAULT_PRIVATE_DROPBOX_PATH")`
- Add `default_private_brand_prefix: Optional[str] = os.getenv("DEFAULT_PRIVATE_BRAND_PREFIX", "NOMADNP")`
3. [ ] **Add helper function** (`backend/services/job_defaults_service.py`)
- Add `get_effective_distribution_for_job(job)` function that returns distribution settings with private overrides applied
### Phase 2: Backend API (job creation + admin)
4. [ ] **Add `is_private` to request models** (`backend/models/requests.py`)
- Add `is_private: Optional[bool] = None` to `URLSubmissionRequest`
- Add `is_private: Optional[bool] = None` to `AudioSearchRequest`
5. [ ] **Update URL creation endpoint** (`backend/api/routes/jobs.py`)
- Accept and pass through `is_private` field to `JobCreate`
6. [ ] **Update file upload endpoint** (`backend/api/routes/file_upload.py`)
- Accept `is_private` Form parameter
- Pass through to `JobCreate`
7. [ ] **Update audio search endpoint** (`backend/api/routes/audio_search.py`)
- Accept `is_private` in `AudioSearchRequest`
- Pass through to `JobCreate`
8. [ ] **Update admin endpoints** (`backend/api/routes/admin.py`)
- Add `is_private: Optional[bool] = None` to `JobUpdateRequest`
- Add `"is_private"` to `EDITABLE_JOB_FIELDS`
- In the PATCH handler: detect when `is_private` changes from False→True on a completed job, auto-delete existing outputs (call existing delete-outputs service logic)
### Phase 3: Distribution Pipeline
9. [ ] **Update orchestrator config creation** (`backend/workers/video_worker_orchestrator.py`)
- In `create_orchestrator_config_from_job()`: Use `get_effective_distribution_for_job()` helper to determine dropbox_path, brand_prefix, enable_youtube_upload, gdrive_folder_id instead of reading directly from job
10. [ ] **Update native distribution fallback** (`backend/workers/video_worker.py`)
- In `_handle_native_distribution()`: Use `get_effective_distribution_for_job()` helper instead of reading directly from job fields
### Phase 4: Email Template
11. [ ] **Add YouTube section conditional removal** (`backend/services/template_service.py`)
- In `render_template()`: Add regex to remove YouTube paragraph when `youtube_url` is empty (matching existing `feedback_url` pattern)
- In `render_job_completion()`: Pass `None` for youtube_url when empty (instead of `"[YouTube URL not available]"`)
### Phase 5: Frontend
12. [ ] **Add private checkbox to job creation form** (`frontend/components/job/JobSubmission.tsx`)
- Add "Private (Non-Published)" checkbox below the existing "Skip lyrics review" checkbox
- Show for all users (not just admin)
- Include brief description text: "Private tracks go to Dropbox only, not YouTube or Google Drive"
13. [ ] **Update API interfaces** (`frontend/lib/api.ts`)
- Add `is_private?: boolean` to all job creation option interfaces
- Add `is_private` to `JobUpdateRequest` interface
- Pass through in `uploadJob()`, `createJobFromUrl()`, `searchAudio()` methods
14. [ ] **Add private toggle to admin job detail** (`frontend/app/admin/jobs/page.tsx`)
- Add `is_private` as an inline-editable field in the job overview section (use Switch component)
- Show current state with visual indicator (e.g., lock icon for private jobs)
- When toggling to private on a completed job, show confirmation dialog about auto-deleting outputs
### Phase 6: Environment Configuration
15. [ ] **Update CI/deployment config** (`.github/workflows/ci.yml` or Cloud Run env)
- Add `DEFAULT_PRIVATE_DROPBOX_PATH` environment variable (e.g., `/Karaoke/Tracks-NonPublished`)
- Add `DEFAULT_PRIVATE_BRAND_PREFIX` environment variable (e.g., `NOMADNP`)
- These should be configured in Cloud Run service and CI test environment
### Phase 7: Testing
16. [ ] **Unit tests for private distribution logic** (`backend/tests/test_private_tracks.py`)
- Test `get_effective_distribution_for_job()` with private and non-private jobs
- Test `create_orchestrator_config_from_job()` applies private overrides correctly
- Test `_run_distribution()` skips YouTube and GDrive when private config applied
- Test `_upload_to_dropbox()` uses private path and brand prefix
- Test brand code generation with NOMADNP prefix in Tracks-NonPublished
- Test `_handle_native_distribution()` respects private overrides
- Test non-private jobs are completely unaffected (regression tests)
17. [ ] **Unit tests for email changes** (add to `backend/tests/test_email_service.py` or `test_template_service.py`)
- Test YouTube section removed from email when youtube_url is None
- Test YouTube section present when youtube_url is provided
- Test Dropbox section still present for private jobs
- Test full email rendering for private job scenario
18. [ ] **Unit tests for admin toggle** (add to `backend/tests/test_admin.py` or new file)
- Test admin can update is_private field
- Test toggling is_private to True on completed job triggers output deletion
- Test toggling is_private to True on non-completed job does NOT delete outputs
- Test toggling is_private to False does nothing special
19. [ ] **Integration test** (`backend/tests/emulator/` or `test_distribution_services.py`)
- Test full distribution pipeline with is_private=True: verify Dropbox path, brand prefix, YouTube skip, GDrive skip
20. [ ] **Frontend E2E regression test** (`frontend/e2e/regression/`)
- Test private checkbox appears in job creation form
- Test private toggle in admin dashboard
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Add `is_private: bool = False` to Job and JobCreate |
| `backend/config.py` | Modify | Add `default_private_dropbox_path` and `default_private_brand_prefix` |
| `backend/services/job_defaults_service.py` | Modify | Add `get_effective_distribution_for_job()` helper |
| `backend/models/requests.py` | Modify | Add `is_private` to URL and AudioSearch request models |
| `backend/api/routes/jobs.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/file_upload.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/audio_search.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/admin.py` | Modify | Add `is_private` to editable fields + auto-delete logic |
| `backend/workers/video_worker_orchestrator.py` | Modify | Use helper in `create_orchestrator_config_from_job()` |
| `backend/workers/video_worker.py` | Modify | Use helper in `_handle_native_distribution()` |
| `backend/services/template_service.py` | Modify | Conditional YouTube section removal |
| `frontend/components/job/JobSubmission.tsx` | Modify | Add private checkbox |
| `frontend/lib/api.ts` | Modify | Add `is_private` to interfaces and API calls |
| `frontend/app/admin/jobs/page.tsx` | Modify | Add private toggle to admin detail |
| `backend/tests/test_private_tracks.py` | Create | Comprehensive unit tests for private track logic |
## Testing Strategy
### Unit Tests (~25-30 tests in `test_private_tracks.py`)
- `get_effective_distribution_for_job()` with private=True and False
- `create_orchestrator_config_from_job()` private overrides (YouTube=False, GDrive=None, private Dropbox path, NOMADNP brand)
- Distribution stage skips: YouTube not called, GDrive not called, Dropbox IS called with correct path
- Brand code generation uses NOMADNP prefix and Tracks-NonPublished path
- Native distribution fallback applies same overrides
- **Regression tests**: Non-private jobs have unchanged behavior (same paths, same brand code, YouTube+GDrive+Dropbox all fire)
### Email Template Tests (~5 tests)
- YouTube section stripped when youtube_url is None
- YouTube section preserved when youtube_url has value
- Dropbox section always present
- Full private job email rendering (Dropbox + no YouTube)
- Full non-private job email rendering (both present) - regression
### Admin Toggle Tests (~5 tests)
- is_private editable via PATCH
- Toggle to private on completed job deletes outputs
- Toggle to private on pending/in-progress job does NOT delete
- Toggle to non-private does nothing special
### Frontend Tests
- E2E: Private checkbox visible and functional in job creation
- E2E: Admin toggle visible and includes confirmation dialog
### Manual Verification
- Create a private job end-to-end, verify Dropbox folder in Tracks-NonPublished with NOMADNP code
- Verify completion email has Dropbox link but no YouTube section
- Toggle existing completed job to private, verify outputs deleted
## Regression Risk Analysis
| Risk | Mitigation |
|------|-----------|
| Non-private jobs affected | `is_private` defaults to `False`; helper returns unchanged values for non-private. Explicit regression tests. |
| Dual code paths (orchestrator vs native) | Both paths use same helper function. Tests cover both paths. (Lesson: "Fix Both Sides of Dual Code Paths") |
| Email template regex breaks other content | Regex targets specific YouTube paragraph text. Tests verify non-YouTube content untouched. |
| Brand code namespace collision | NOMADNP codes are in a completely separate Dropbox folder (Tracks-NonPublished), no overlap with NOMAD codes in Tracks-Organized |
| Admin auto-delete on wrong job | Only triggers when `is_private` changes from False→True AND job has terminal status (complete). Confirmation dialog in frontend. |
| Made-for-you orders | MFY webhook doesn't set is_private (defaults False). Admin can toggle after. No change to MFY flow. |
| Existing Firestore documents | Missing `is_private` field defaults to `False` via model default. No migration needed. |
## Open Questions
- [x] Who sees private toggle? → **Everyone** (all users at creation, admin on existing)
- [x] Auto-delete on toggle? → **Yes**, auto-delete all outputs when toggling to private on completed job
- [x] Email format? → **Remove YouTube section** when no YouTube URL (improves all edge cases)
- [x] Discord? → **Still notify** (useful for internal tracking)
## Rollback Plan
1. **Feature flag**: The `is_private` field defaults to `False`, so setting `DEFAULT_PRIVATE_DROPBOX_PATH` to empty/None effectively disables private distribution
2. **No schema migration needed**: Missing `is_private` field on existing Firestore documents defaults to `False`
3. **Email change is backward-compatible**: The YouTube section removal only happens when youtube_url is None, which is already an edge case showing "[YouTube URL not available]"
4. **Frontend**: Private checkbox is additive UI, removing it just means users can't create new private jobs
@@ -1,239 +0,0 @@
# Plan: Song Lookup & Autocomplete + Community Version Detection
**Created:** 2026-03-02
**Branch:** feat/sess-20260302-2324-song-lookup-autocomplete
**Status:** Implementation Complete
## Overview
Add two features to the karaoke-gen job creation flow:
1. **Artist/title autocomplete** using karaoke-decide's MusicBrainz + Spotify catalog data, so users get properly-styled artist/title text (e.g. "Fleetwood Mac" instead of "fleetwood mac")
2. **Community karaoke version detection** that warns users when a high-quality community karaoke version already exists on YouTube (via karaokenerds data), saving them credits and time
Both features are **optional and non-blocking** — users can still type whatever they want and proceed even if a community version exists.
## Requirements
- [ ] Artist field autocomplete across all 3 job creation tabs (Search, Upload, URL)
- [ ] Title field autocomplete (optionally filtered by selected artist) across all 3 tabs
- [ ] Autocomplete uses canonical MusicBrainz/Spotify text styling
- [ ] Autocomplete is optional — users can ignore suggestions and type freely
- [ ] Community version check triggers when both artist and title are filled
- [ ] If community karaoke version exists, show a dismissible banner with YouTube link(s)
- [ ] Community check does not block job submission — just informs the user
- [ ] Graceful degradation if karaoke-decide API or karaokenerds.com is down
## Architecture Decisions
### How to access karaoke-decide catalog data
**Decision: Proxy through karaoke-gen backend**
Both services share the same GCP project (`nomadkaraoke`), but the frontend at `gen.nomadkaraoke.com` can't directly call `decide.nomadkaraoke.com/api/` due to CORS (different origins). Options considered:
| Option | Pros | Cons |
|--------|------|------|
| **Proxy via karaoke-gen backend** | No CORS issues, no changes to karaoke-decide, frontend already calls api.nomadkaraoke.com | Added latency (extra hop), karaoke-decide dependency |
| Direct frontend calls + CORS | Simpler, no proxy code | Requires karaoke-decide CORS changes, cross-project coupling |
| Query BigQuery directly from karaoke-gen | No dependency on karaoke-decide service | Duplicates query logic, BigQuery setup needed, higher latency |
The proxy approach is cleanest: karaoke-gen frontend → karaoke-gen backend → karaoke-decide API. Response caching in the proxy minimizes latency impact.
### How to detect community karaoke versions
**Decision: Scrape karaokenerds.com from karaoke-gen backend** (same approach as kjbox)
The `karaokenerds_raw` BigQuery table has the catalog but NOT community status. Community status is determined by an `<img class="check">` badge in karaokenerds.com's HTML, only available via web scraping. The kjbox codebase has proven, well-tested scraping logic we can port directly.
Search URL: `https://karaokenerds.com/Search?query={artist}+{title}&webFilter=OnlyWeb`
Returns: songs with tracks, each track having `brand_name`, `brand_code`, `youtube_url`, `is_community`.
### Autocomplete UI component
**Decision: Use existing shadcn Command (cmdk) + Popover components**
The frontend already has `command.tsx` and `popover.tsx` from shadcn/ui. The standard combobox pattern (Popover + Command) provides:
- Keyboard navigation (arrow keys, enter to select)
- Debounced search-as-you-type
- Clean dropdown with loading states
- Works inline within forms (not modal)
## Technical Approach
### Data flow
```
User types in artist field
→ debounced (300ms) API call to GET /api/catalog/artists?q=fle
→ karaoke-gen backend proxies to decide.nomadkaraoke.com/api/catalog/artists?q=fle
→ returns [{ name: "Fleetwood Mac", mbid: "...", popularity: 85 }, ...]
→ user selects "Fleetwood Mac" → field populated with canonical name
User types in title field
→ debounced (300ms) API call to GET /api/catalog/tracks?q=dreams&artist=Fleetwood+Mac
→ returns [{ track_name: "Dreams", artist_name: "Fleetwood Mac", popularity: 82 }, ...]
→ user selects "Dreams" → field populated with canonical title
Both fields filled → community check fires
→ POST /api/catalog/community-check { artist: "Fleetwood Mac", title: "Dreams" }
→ backend scrapes karaokenerds.com for "Fleetwood Mac Dreams" with webFilter=OnlyWeb
→ if community tracks found with YouTube URLs, return them
→ frontend shows banner: "A community karaoke version already exists! [Watch on YouTube →]"
```
### Caching strategy
| Endpoint | Cache TTL | Scope |
|----------|-----------|-------|
| Artist search proxy | 5 min | In-memory (backend) |
| Track search proxy | 5 min | In-memory (backend) |
| Community check | 1 hour | In-memory (backend) |
### karaoke-decide API endpoints used
These are all **public, no auth required**:
- `GET /api/catalog/artists?q={query}&limit=10` — MBID-first artist search
- Returns: `[{ mbid, name, disambiguation, artist_type, tags, spotify_id, popularity, genres }]`
- `GET /api/catalog/tracks?q={query}&limit=10` — Track search (Spotify data)
- Returns: `[{ track_id, track_name, artist_name, artist_id, popularity, duration_ms, explicit }]`
## Implementation Steps
### Phase 1: Backend — Catalog proxy + community check endpoints
1. [ ] **Add catalog proxy service** — `backend/services/catalog_proxy_service.py`
- HTTP client (httpx async) calling karaoke-decide API
- In-memory TTL cache for responses (5 min)
- Configurable karaoke-decide base URL
- Graceful error handling (return empty results on failure)
2. [ ] **Port karaokenerds scraping logic** — `backend/services/karaokenerds_service.py`
- Port `search()`, `parse_results()`, `_parse_tracks()`, `_parse_single_track()`, `_clean_youtube_url()` from `kjbox/kj-controller/karaoke_nerds.py`
- Add async support (httpx instead of requests)
- In-memory TTL cache (1 hour)
- Return only community tracks with YouTube URLs
3. [ ] **Add catalog routes** — `backend/api/routes/catalog.py`
- `GET /api/catalog/artists?q=&limit=10` — proxy to karaoke-decide
- `GET /api/catalog/tracks?q=&artist=&limit=10` — proxy to karaoke-decide
- `POST /api/catalog/community-check` — scrape karaokenerds.com
- All endpoints require auth (same as job creation)
- Pydantic request/response models
4. [ ] **Register routes in main.py** — add `catalog.router` to app
5. [ ] **Add config** — `backend/config.py`
- `karaoke_decide_api_url` setting (default: `https://decide.nomadkaraoke.com`)
6. [ ] **Add `beautifulsoup4` + `httpx` dependencies** to `pyproject.toml` (if not already present)
### Phase 2: Frontend — Autocomplete component
7. [ ] **Create `AutocompleteInput` component** — `frontend/components/ui/autocomplete-input.tsx`
- Wraps an Input field with a Popover + Command dropdown
- Props: `value`, `onChange`, `onSelect`, `fetchSuggestions(query) => Promise<Suggestion[]>`, `placeholder`, `disabled`, `renderItem`
- Debounced search (300ms) triggered on input change
- Shows loading spinner while fetching
- Shows "No results" when search returns empty
- Keyboard navigation (up/down arrows, enter to select, escape to close)
- Clicking outside closes dropdown
- When user selects a suggestion, calls `onSelect(suggestion)` and populates input
- When user types without selecting, uses their raw text (autocomplete is optional)
8. [ ] **Add catalog API functions** — `frontend/lib/api.ts`
- `searchArtists(query: string, limit?: number)` → `GET /api/catalog/artists`
- `searchTracks(query: string, artist?: string, limit?: number)` → `GET /api/catalog/tracks`
- `checkCommunityVersions(artist: string, title: string)` → `POST /api/catalog/community-check`
### Phase 3: Frontend — Integrate autocomplete into job creation
9. [ ] **Replace artist Input fields with AutocompleteInput** in `JobSubmission.tsx`
- All 3 tabs: `searchArtist`, `uploadArtist`, `youtubeArtist`
- Fetch suggestions from `api.searchArtists(query)`
- On select: populate artist field with canonical name
- Show artist name + optional disambiguation in dropdown items
10. [ ] **Replace title Input fields with AutocompleteInput** in `JobSubmission.tsx`
- All 3 tabs: `searchTitle`, `uploadTitle`, `youtubeTitle`
- Fetch suggestions from `api.searchTracks(query, selectedArtist)`
- On select: populate title field with canonical name
- Show track name + artist name + duration in dropdown items
### Phase 4: Frontend — Community version detection
11. [ ] **Create `CommunityVersionBanner` component** — `frontend/components/job/CommunityVersionBanner.tsx`
- Shows when community karaoke version(s) found
- Displays: "A karaoke version of this song already exists!"
- Lists community tracks with brand name and YouTube link
- "Watch on YouTube" button for top community version
- Dismissible (X button or "Continue anyway" link)
- Styled with green border (matching kjbox community track styling)
12. [ ] **Add community check logic to `JobSubmission.tsx`** (Search tab only)
- Debounced check (500ms) when both artist AND title fields have values
- Only triggers when values change (not on every keystroke)
- Shows `CommunityVersionBanner` between the form fields and the submit button
- Does NOT block form submission
- Clears when artist or title changes
### Phase 5: Testing
13. [ ] **Backend unit tests** — `backend/tests/unit/test_catalog_routes.py`
- Test proxy endpoints with mocked karaoke-decide responses
- Test community check with mocked karaokenerds HTML
- Test error handling (karaoke-decide down, karaokenerds down)
- Test caching behavior
14. [ ] **Backend unit tests** — `backend/tests/unit/test_karaokenerds_service.py`
- Port relevant tests from `kjbox/kj-controller/tests/unit/test_karaoke_nerds.py`
- Test HTML parsing, community detection, YouTube URL cleanup
15. [ ] **Frontend component tests** — `frontend/components/ui/__tests__/autocomplete-input.test.tsx`
- Test suggestion fetching and display
- Test keyboard navigation
- Test selection behavior
- Test free-text input (no selection)
16. [ ] **E2E tests** — `frontend/e2e/job-creation-autocomplete.spec.ts`
- Test autocomplete appears on typing
- Test selecting from autocomplete populates field
- Test community banner appears for known songs
- Test form submission works with and without autocomplete selection
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/services/catalog_proxy_service.py` | Create | HTTP client for karaoke-decide API with caching |
| `backend/services/karaokenerds_service.py` | Create | Karaokenerds.com scraper (ported from kjbox) |
| `backend/api/routes/catalog.py` | Create | Proxy endpoints for artist/track search + community check |
| `backend/main.py` | Modify | Register catalog router |
| `backend/config.py` | Modify | Add karaoke_decide_api_url setting |
| `pyproject.toml` | Modify | Add beautifulsoup4 dependency (if missing) |
| `frontend/components/ui/autocomplete-input.tsx` | Create | Reusable autocomplete input component |
| `frontend/components/job/CommunityVersionBanner.tsx` | Create | Community version warning banner |
| `frontend/components/job/JobSubmission.tsx` | Modify | Replace Input with AutocompleteInput, add community check |
| `frontend/lib/api.ts` | Modify | Add catalog API functions |
| `backend/tests/unit/test_catalog_routes.py` | Create | Backend route tests |
| `backend/tests/unit/test_karaokenerds_service.py` | Create | Scraper unit tests |
| `frontend/components/ui/__tests__/autocomplete-input.test.tsx` | Create | Component tests |
| `frontend/e2e/job-creation-autocomplete.spec.ts` | Create | E2E tests |
## Decisions (Resolved)
- **Track search scope**: All Spotify tracks (256M) — better coverage for new/obscure songs
- **Community check scope**: Search tab only (Upload/URL users already have their audio)
- **Rate limiting**: 20 requests/min per user on catalog proxy endpoints (prevents abuse of karaokenerds scraping). All endpoints require auth.
- **"Display As" fields**: No autocomplete — they're for deliberate overrides like soundtrack names
- **CORS**: Proxy approach (no changes needed to karaoke-decide)
## Rollback Plan
- Feature is entirely additive — no existing functionality is modified
- Autocomplete fields fall back to plain text Input if API calls fail
- Community check is non-blocking — if it fails, form works normally
- All new endpoints are behind the existing auth middleware
- Can disable by reverting the frontend changes (backend proxy is harmless if unused)
-37
View File
@@ -1,37 +0,0 @@
#!/bin/bash
# Auto-translate when English translation files change.
# Uses GCS cache so this is near-instant for small changes.
# Skip with: git commit --no-verify
set -e
STAGED_EN=$(git diff --cached --name-only --diff-filter=ACM | grep -E '(messages|translations)/en\.json$' || true)
if [ -z "$STAGED_EN" ]; then
exit 0
fi
echo "i18n: English translations changed — running translation pipeline..."
for en_file in $STAGED_EN; do
dir=$(dirname "$en_file")
if [[ "$dir" == frontend/messages ]]; then
script="frontend/scripts/translate.py"
elif [[ "$dir" == backend/translations ]]; then
script="frontend/scripts/translate.py"
else
echo " Warning: Unknown messages dir $dir, skipping"
continue
fi
echo " Translating $dir..."
if python "$script" --messages-dir "$dir" --target all --skip-review 2>&1 | tail -5; then
git add "$dir"/*.json
echo " Done: $dir"
else
echo " Warning: Translation failed for $dir (GCP auth may be missing). CI will catch this."
fi
done
echo "i18n: Translation complete."
+215 -576
View File
File diff suppressed because it is too large. Load diff
-337
View File
@@ -1,337 +0,0 @@
name: E2E Daily Test (Payment + Happy Path)
# Full production E2E: buy credits via real Stripe, then use them for karaoke generation.
# Replaces the previous e2e-happy-path.yml with an additional payment stage.
on:
schedule:
- cron: '0 6 * * *'
workflow_dispatch:
inputs:
debug_mode:
description: 'Enable debug mode (slower, more screenshots)'
required: false
default: 'false'
type: boolean
browser:
description: 'Browser to test with'
required: false
default: 'chromium'
type: choice
options:
- chromium
- firefox
- webkit
skip_payment:
description: 'Skip Stage 1 (payment) — use admin-granted credits for happy path only'
required: false
default: 'false'
type: boolean
use_test_token:
description: 'Skip signup entirely — use E2E_TEST_TOKEN (for quick iterations)'
required: false
default: 'false'
type: boolean
permissions:
contents: read
jobs:
# =========================================================================
# Stage 1: Purchase credits via real Stripe Checkout
# =========================================================================
e2e-credit-purchase:
name: "Stage 1: Credit Purchase"
runs-on: ubuntu-latest
timeout-minutes: 15
if: ${{ github.event.inputs.skip_payment != 'true' && github.event.inputs.use_test_token != 'true' }}
outputs:
session_token: ${{ steps.extract_token.outputs.token }}
result: ${{ steps.test_result.outputs.result }}
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: npx playwright install --with-deps ${{ inputs.browser || 'chromium' }}
- name: Create test-results directory
working-directory: frontend
run: mkdir -p test-results
- name: Ensure testmail.app domain is allowlisted
run: |
curl -sf -X POST \
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
-H "Content-Type: application/json" \
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
- name: Run Credit Purchase Test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
E2E_STRIPE_CARD_NUMBER: ${{ secrets.E2E_STRIPE_CARD_NUMBER }}
E2E_STRIPE_CARD_EXPIRY: ${{ secrets.E2E_STRIPE_CARD_EXPIRY }}
E2E_STRIPE_CARD_CVC: ${{ secrets.E2E_STRIPE_CARD_CVC }}
E2E_STRIPE_CARDHOLDER_NAME: ${{ secrets.E2E_STRIPE_CARDHOLDER_NAME }}
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
run: |
echo "=== Stage 1: Credit Purchase via Real Stripe ==="
echo "Time: $(date)"
npx playwright test credit-purchase-real.spec.ts \
--config=playwright.production.config.ts \
--project=${{ inputs.browser || 'chromium' }} \
--reporter=list \
--timeout=600000 \
2>&1 | tee test-output-payment.log
continue-on-error: true
- name: Check test result
id: test_result
working-directory: frontend
run: |
if grep -q "CREDIT PURCHASE TEST PASSED" test-output-payment.log; then
echo "result=success" >> $GITHUB_OUTPUT
else
echo "result=failure" >> $GITHUB_OUTPUT
fi
- name: Extract session token
id: extract_token
if: steps.test_result.outputs.result == 'success'
working-directory: frontend
run: |
if [ -f test-results/e2e-session-token.txt ]; then
TOKEN=$(cat test-results/e2e-session-token.txt)
echo "::add-mask::$TOKEN"
echo "token=$TOKEN" >> $GITHUB_OUTPUT
echo "Session token extracted for Stage 2"
else
echo "No session token file found"
echo "token=" >> $GITHUB_OUTPUT
fi
- name: Upload artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-payment-results-${{ github.run_number }}
path: |
frontend/test-results/
frontend/test-output-payment.log
retention-days: 30
- name: Discord notification (failure)
if: steps.test_result.outputs.result == 'failure'
run: |
curl -s -H "Content-Type: application/json" \
-d "{\"embeds\": [{\"title\": \"🚨 E2E Daily Test FAILED — Stage 1: Credit Purchase\", \"description\": \"Payment flow is broken. Customers may not be receiving credits.\n\n**Run:** #${{ github.run_number }}\n**Details:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": 15158332}]}" \
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
- name: Fail if test failed
if: steps.test_result.outputs.result == 'failure'
run: exit 1
# =========================================================================
# Stage 2: Happy Path (uses credit from Stage 1)
# =========================================================================
e2e-happy-path:
name: "Stage 2: Happy Path"
runs-on: ubuntu-latest
timeout-minutes: 90
needs: e2e-credit-purchase
if: |
always() && (
needs.e2e-credit-purchase.result == 'success' ||
github.event.inputs.skip_payment == 'true' ||
github.event.inputs.use_test_token == 'true'
)
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: npx playwright install --with-deps ${{ inputs.browser || 'chromium' }}
- name: Create test-results directory
working-directory: frontend
run: mkdir -p test-results
- name: Ensure testmail.app domain is allowlisted
run: |
curl -sf -X POST \
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
-H "Content-Type: application/json" \
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
- name: Refresh E2E session token (when use_test_token enabled)
id: refresh_token
if: github.event.inputs.use_test_token == 'true'
run: |
RESPONSE=$(curl -sf -X POST \
"https://api.nomadkaraoke.com/api/admin/users/e2e-test-runner@nomadkaraoke.com/impersonate" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
-H "Content-Type: application/json")
TOKEN=$(echo "$RESPONSE" | python3 -c 'import json,sys; print(json.load(sys.stdin)["session_token"])')
echo "::add-mask::$TOKEN"
echo "token=$TOKEN" >> $GITHUB_OUTPUT
- name: Run Happy Path Test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_TEST_TOKEN: ${{ steps.refresh_token.outputs.token || needs.e2e-credit-purchase.outputs.session_token || '' }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
run: |
echo "=== Stage 2: Happy Path ==="
echo "Time: $(date)"
TOKEN_SOURCE="none"
if [ -n "${{ steps.refresh_token.outputs.token }}" ]; then
TOKEN_SOURCE="use_test_token (impersonation)"
elif [ -n "${{ needs.e2e-credit-purchase.outputs.session_token }}" ]; then
TOKEN_SOURCE="Stage 1 (credit purchase)"
fi
echo "Token source: $TOKEN_SOURCE"
npx playwright test happy-path-real-user.spec.ts \
--config=playwright.production.config.ts \
--project=${{ inputs.browser || 'chromium' }} \
--reporter=list \
--timeout=3600000 \
2>&1 | tee test-output-happy-path.log
continue-on-error: true
- name: Check test result
id: test_result
working-directory: frontend
run: |
if grep -q "passed" test-output-happy-path.log && ! grep -q "failed" test-output-happy-path.log; then
echo "result=success" >> $GITHUB_OUTPUT
else
echo "result=failure" >> $GITHUB_OUTPUT
fi
- name: Upload test artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-results-${{ github.run_number }}
path: |
frontend/test-results/
frontend/playwright-report/
frontend/test-output-happy-path.log
retention-days: 30
- name: Upload video recordings
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-videos-${{ github.run_number }}
path: frontend/test-results/**/*.webm
retention-days: 30
if-no-files-found: ignore
- name: Create summary
if: always()
run: |
STAGE1="${{ needs.e2e-credit-purchase.result || 'skipped' }}"
STAGE2="${{ steps.test_result.outputs.result || 'unknown' }}"
echo "## E2E Daily Test Results" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "| Stage | Result |" >> $GITHUB_STEP_SUMMARY
echo "|-------|--------|" >> $GITHUB_STEP_SUMMARY
echo "| Stage 1: Credit Purchase | ${STAGE1} |" >> $GITHUB_STEP_SUMMARY
echo "| Stage 2: Happy Path | ${STAGE2} |" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
- name: Discord notification (failure)
if: steps.test_result.outputs.result == 'failure'
run: |
curl -s -H "Content-Type: application/json" \
-d "{\"embeds\": [{\"title\": \"🚨 E2E Daily Test FAILED — Stage 2: Happy Path\", \"description\": \"Karaoke generation flow is broken.\n\n**Run:** #${{ github.run_number }}\n**Details:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": 15158332}]}" \
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
- name: Fail if test failed
if: steps.test_result.outputs.result == 'failure'
run: exit 1
# =========================================================================
# Final: Send notifications
# =========================================================================
notify:
name: "Notify"
runs-on: ubuntu-latest
needs: [e2e-credit-purchase, e2e-happy-path]
if: always()
steps:
- name: Discord notification (success)
if: needs.e2e-happy-path.result == 'success'
run: |
STAGE1="${{ needs.e2e-credit-purchase.result || 'skipped' }}"
curl -s -H "Content-Type: application/json" \
-d "{\"embeds\": [{\"title\": \"✅ E2E Daily Test PASSED\", \"description\": \"Payment: ${STAGE1} | Happy Path: success\n\n**Run:** #${{ github.run_number }}\", \"color\": 3066993}]}" \
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
- name: Send email notification
if: github.event_name == 'schedule'
uses: dawidd6/action-send-mail@v3
with:
server_address: smtp.sendgrid.net
server_port: 587
username: apikey
password: ${{ secrets.SENDGRID_API_KEY }}
subject: "E2E Daily Test ${{ needs.e2e-happy-path.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
to: andrew@nomadkaraoke.com
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
body: |
E2E Daily Test Results
======================
Stage 1 (Credit Purchase): ${{ needs.e2e-credit-purchase.result || 'skipped' }}
Stage 2 (Happy Path): ${{ needs.e2e-happy-path.result || 'skipped' }}
Run: #${{ github.run_number }}
Trigger: ${{ github.event_name }}
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+174
View File
@@ -0,0 +1,174 @@
name: E2E Happy Path Test
# Comprehensive end-to-end test that validates the full karaoke generation journey
# using ONLY browser interactions (no API shortcuts)
#
# See docs/E2E-HAPPY-PATH-TEST-SPEC.md for full requirements
on:
# Daily run at 6 AM UTC
schedule:
- cron: '0 6 * * *'
# Manual trigger with optional parameters
workflow_dispatch:
inputs:
debug_mode:
description: 'Enable debug mode (slower, more screenshots)'
required: false
default: 'false'
type: boolean
browser:
description: 'Browser to test with'
required: false
default: 'chromium'
type: choice
options:
- chromium
- firefox
- webkit
use_test_token:
description: 'Skip signup and use E2E_TEST_TOKEN (for iterations)'
required: false
default: 'false'
type: boolean
permissions:
contents: read
jobs:
e2e-happy-path:
name: "Full Happy Path (Real User)"
runs-on: ubuntu-latest
timeout-minutes: 90 # 1.5 hours max (test itself is 60 min)
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: |
BROWSER=${{ inputs.browser || 'chromium' }}
npx playwright install --with-deps $BROWSER
- name: Create test-results directory
working-directory: frontend
run: mkdir -p test-results
- name: Run E2E Happy Path Test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_TEST_TOKEN: ${{ inputs.use_test_token && secrets.E2E_TEST_TOKEN || '' }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
run: |
echo "Starting E2E Happy Path Test"
echo "Browser: ${{ inputs.browser || 'chromium' }}"
echo "Debug mode: ${{ inputs.debug_mode || 'false' }}"
echo "Use test token: ${{ inputs.use_test_token || 'false' }}"
echo "Time: $(date)"
echo ""
# Run the real user happy path test
npx playwright test happy-path-real-user.spec.ts \
--config=playwright.production.config.ts \
--project=${{ inputs.browser || 'chromium' }} \
--reporter=list \
--timeout=3600000 \
2>&1 | tee test-output.log
continue-on-error: true
- name: Check test result
id: test_result
working-directory: frontend
run: |
if grep -q "passed" test-output.log && ! grep -q "failed" test-output.log; then
echo "result=success" >> $GITHUB_OUTPUT
echo "Test PASSED"
else
echo "result=failure" >> $GITHUB_OUTPUT
echo "Test FAILED - see artifacts for details"
fi
- name: Upload test artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-results-${{ github.run_number }}
path: |
frontend/test-results/
frontend/playwright-report/
frontend/test-output.log
retention-days: 30
- name: Upload video recordings
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-videos-${{ github.run_number }}
path: frontend/test-results/**/*.webm
retention-days: 30
if-no-files-found: ignore
- name: Create summary
if: always()
run: |
echo "## E2E Happy Path Test Results" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
echo "| Result | ${{ steps.test_result.outputs.result || 'unknown' }} |" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
if [ -f frontend/test-output.log ]; then
echo "### Test Output (last 100 lines)" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
tail -100 frontend/test-output.log >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
fi
- name: Send email notification
if: always() && github.event_name == 'schedule'
uses: dawidd6/action-send-mail@v3
with:
server_address: smtp.sendgrid.net
server_port: 587
username: apikey
password: ${{ secrets.SENDGRID_API_KEY }}
subject: "E2E Happy Path Test ${{ steps.test_result.outputs.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
to: andrew@nomadkaraoke.com
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
body: |
E2E Happy Path Test Results
===========================
Status: ${{ steps.test_result.outputs.result == 'success' && 'PASSED ✅' || 'FAILED ❌' }}
Run: #${{ github.run_number }}
Trigger: ${{ github.event_name }}
Time: ${{ github.event.repository.updated_at }}
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
This is an automated notification from the daily E2E happy path test.
- name: Fail if test failed
if: steps.test_result.outputs.result == 'failure'
run: exit 1
-1
View File
@@ -232,4 +232,3 @@ flacfetch
# Test output directories
Rick Astley*/
.superpowers/
-1
View File
@@ -1 +0,0 @@
[{"name":"generate-buildid","duration":272,"timestamp":244518298649,"id":4,"parentId":1,"tags":{},"startTime":1775262331361,"traceId":"b3ef0d179e8acedf"},{"name":"load-custom-routes","duration":314,"timestamp":244518299029,"id":5,"parentId":1,"tags":{},"startTime":1775262331361,"traceId":"b3ef0d179e8acedf"},{"name":"create-dist-dir","duration":333,"timestamp":244518299363,"id":6,"parentId":1,"tags":{},"startTime":1775262331362,"traceId":"b3ef0d179e8acedf"},{"name":"clean","duration":358,"timestamp":244518300558,"id":7,"parentId":1,"tags":{},"startTime":1775262331363,"traceId":"b3ef0d179e8acedf"},{"name":"next-build","duration":1670851,"timestamp":244516630180,"id":1,"tags":{"buildMode":"default","version":"16.2.2","bundler":"turbopack","failed":true},"startTime":1775262329693,"traceId":"b3ef0d179e8acedf"}]
-1
View File
@@ -1 +0,0 @@
[{"name":"next-build","duration":1670851,"timestamp":244516630180,"id":1,"tags":{"buildMode":"default","version":"16.2.2","bundler":"turbopack","failed":true},"startTime":1775262329693,"traceId":"b3ef0d179e8acedf"}]
+7 -25
View File
@@ -19,8 +19,6 @@
| **Testing & code quality** | `docs/TESTING.md` |
| API reference | `docs/API.md` |
| Past learnings | `docs/LESSONS-LEARNED.md` |
| **Operational runbooks** | `docs/TROUBLESHOOTING.md` |
| GDrive validator & gaps | `docs/GDRIVE-VALIDATOR.md` |
| **Product communication** | `docs/PRODUCT-COMMUNICATION-GUIDE.md` |
| Brand style guide | `docs/BRAND-STYLE-GUIDE.md` |
@@ -28,7 +26,7 @@
- **Backend**: FastAPI on Cloud Run, Firestore, GCS, Secret Manager
- **Frontend**: Next.js on Cloudflare Pages
- **Processing**: Cloud Run GPU (L4 audio separation), AudioShake/Whisper (transcription)
- **Processing**: Modal (GPU audio separation), AudioShake/Whisper (transcription)
- **Infra**: Pulumi IaC, GitHub Actions CI/CD
## Essential Rules
@@ -92,25 +90,11 @@ See `docs/TESTING.md` § "Ad-Hoc Production Debugging" for full details.
### Infrastructure
- **All GCP changes via Pulumi PRs** - changes in `infrastructure/` deploy automatically on merge to main
- **Always run `pulumi up` locally BEFORE merging PRs** - CI deploys run on spot VMs that can be preempted mid-apply, causing partial state. Apply locally first, then merge (CI re-runs as no-op)
- Never modify GCP resources directly via console or `gcloud` CLI
- `gcloud` CLI for reading/debugging only (e.g., checking logs, SSH to VMs)
- Local `pulumi preview` works but `pulumi up` requires broader permissions than CI
- Stop and notify user on auth issues
### Internationalization (i18n)
- All user-facing strings live in `frontend/messages/{locale}.json` (33 locales), NOT in components
- Backend strings live in `backend/translations/{locale}.json` (currently 3 locales: en, es, de)
- Components use `useTranslations('namespace')` from next-intl
- Pages are under `frontend/src/app/[locale]/` — locale-aware routing
- Admin pages (under `/admin/`, `/app/`) use DefaultIntlProvider (English-only, excluded from i18n)
- Internal links use `Link` from `@/i18n/routing` (locale-aware)
- After adding/changing English strings: `python frontend/scripts/translate.py --messages-dir frontend/messages --target all`
- Translation uses Gemini 3.1 Pro via Vertex AI with GCS cache for efficiency
- CI validates all locale files have matching keys — **PR will fail if translations are missing**
- Pre-commit hook auto-translates when en.json changes (enable: `git config core.hooksPath .githooks`)
- Don't hardcode user-facing strings — add to `messages/en.json` and use `t('key')`
## API Authentication (for Agents)
When you need to call backend APIs programmatically:
@@ -144,16 +128,14 @@ EOF
### GCE Encoding Worker
```bash
# Check health (includes wheel_version)
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
--command="curl -s http://localhost:8080/health"
# Restart service (pick up new wheel after a deploy)
# SSH to restart service (clears in-memory job queue)
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
--command="sudo systemctl restart encoding-worker"
```
For job stuck at `encoding` status, see [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md).
# Check health
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
--command="curl -s http://localhost:8080/health"
```
### What Doesn't Work
- `gcloud auth print-identity-token` - Wrong token type for internal endpoints
+11 -34
View File
@@ -1,25 +1,18 @@
# Makefile for karaoke-gen project
# Run `make help` to see available commands
.PHONY: help install install-backend install-frontend build-frontend dev-install test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop
# Per-stage timeout in seconds (default: 10 minutes)
# Prevents hangs from stale dev servers, broken emulators, etc.
# Override with: make test TEST_TIMEOUT=300
TEST_TIMEOUT ?= 600
.PHONY: help install install-backend install-frontend build-frontend test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop
# Default target
help:
@echo "Available commands:"
@echo " make test - Run ALL tests (backend + frontend, installs deps automatically)"
@echo " make test-backend - Run backend tests only (unit + emulator)"
@echo " make test-frontend - Run frontend unit tests (Jest)"
@echo " make test-frontend-e2e - Run frontend E2E tests (Playwright, requires dev server)"
@echo " make test-frontend - Run frontend tests only (unit + E2E)"
@echo " make test-unit - Run unit tests only (karaoke_gen package)"
@echo " make test-e2e - Run emulator tests with auto-start/stop"
@echo " make install - Install all dependencies (backend + frontend)"
@echo " make build-frontend - Build frontend and copy to Python package"
@echo " make dev-install - Build frontend + pip install -e . (for local CLI testing)"
@echo " make build-frontend - Build frontend and copy to Python package (for local testing)"
@echo " make lint - Run linter checks"
@echo " make emulators-start - Start GCP emulators for local development"
@echo " make emulators-stop - Stop GCP emulators"
@@ -41,34 +34,23 @@ install-frontend:
install: install-backend install-frontend
# Build frontend and copy to Python package
# Build frontend and copy to Python package (for local CLI testing)
build-frontend: install-frontend
@echo "=== Building frontend ==="
cd frontend && npm run build
@echo "=== Copying build to Python package ==="
rm -rf karaoke_gen/nextjs_frontend/out
mkdir -p karaoke_gen/nextjs_frontend/out
cp -r frontend/out/* karaoke_gen/nextjs_frontend/out/
@echo "✅ Frontend built and copied to karaoke_gen/nextjs_frontend/out/"
# Build frontend + install package in editable mode for local CLI testing
# After running this, `karaoke-gen` CLI will use the freshly built frontend.
dev-install: build-frontend
@echo "=== Installing package in editable mode ==="
pip install -e .
@echo ""
@echo "✅ Ready! You can now run karaoke-gen with the updated frontend."
@echo " Example: karaoke-gen input.flac \"Artist\" \"Title\""
@echo "✅ Frontend built and ready for local testing"
# Run unit tests for karaoke_gen package
test-unit: install-backend
@echo "=== Running karaoke_gen unit tests ==="
timeout $(TEST_TIMEOUT) poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
# Run backend unit tests (excludes emulator tests)
test-backend-unit: install-backend
@echo "=== Running backend unit tests ==="
timeout $(TEST_TIMEOUT) poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
# Run E2E integration tests with emulators
test-e2e: install-backend
@@ -80,7 +62,7 @@ test-e2e: install-backend
export GCS_BUCKET_NAME=test-bucket && \
export ADMIN_TOKENS=test-admin-token && \
export ENVIRONMENT=test && \
timeout $(TEST_TIMEOUT) poetry run pytest backend/tests/emulator/ -v; \
poetry run pytest backend/tests/emulator/ -v; \
TEST_RESULT=$$?; \
./scripts/stop-emulators.sh; \
exit $$TEST_RESULT
@@ -90,15 +72,10 @@ test-backend: test-unit test-backend-unit test-e2e
@echo ""
@echo "✅ Backend tests passed!"
# Run frontend unit tests (Jest only - E2E requires manual setup)
# Run frontend tests (unit + E2E)
test-frontend: install-frontend
@echo "=== Running frontend unit tests ==="
cd frontend && timeout $(TEST_TIMEOUT) npm run test:unit
# Run frontend E2E tests (requires dev server - run manually, not in make test)
test-frontend-e2e: install-frontend
@echo "=== Running frontend E2E tests ==="
cd frontend && timeout $(TEST_TIMEOUT) npm run test:e2e
@echo "=== Running frontend tests ==="
cd frontend && npm run test:all
# Run ALL tests (backend + frontend) - use this before committing!
test: test-backend test-frontend
-2
View File
@@ -29,8 +29,6 @@ tmp/
# Google Cloud
.gcloud/
*.json
# Allow i18n translation files
!translations/*.json
# Testing
.pytest_cache/
+1 -9
View File
@@ -26,15 +26,7 @@ COPY backend /app/backend
# Note: lyrics_transcriber is now a subpackage of karaoke_gen
COPY karaoke_gen /app/karaoke_gen
COPY pyproject.toml README.md LICENSE /app/
# GPU images (MODEL_DIR set in base): use --no-deps to preserve pinned torch cu126
# and audio-separator[gpu]. Without this, pip upgrades torch to cu130 (breaks CUDA
# on Cloud Run L4 drivers) and reinstalls audio-separator[cpu] (breaks GPU ONNX).
# CPU images: normal install resolves all deps from pyproject.toml.
RUN if [ -n "$MODEL_DIR" ]; then \
pip install --no-cache-dir --no-deps -e .; \
else \
pip install --no-cache-dir -e .; \
fi
RUN pip install --no-cache-dir -e .
# Set environment variables
ENV PYTHONUNBUFFERED=1
+7 -26
View File
@@ -1,15 +1,8 @@
# Base image with all Python dependencies pre-installed
# This image is rebuilt only when poetry.lock or this Dockerfile changes
#
# This image is rebuilt only when poetry.lock changes
#
# Build with: gcloud builds submit --config=cloudbuild-base.yaml
# Or manually: docker build -f backend/Dockerfile.base -t karaoke-backend-base .
#
# Layer caching strategy:
# 1. System deps + fonts + ffmpeg (rarely change, always cached)
# 2. pyproject.toml + package stubs (changes with dep updates)
# 3. pip install (the slow part - cached unless pyproject.toml changes)
# 4. Source code copy (changes every rebuild, but AFTER deps are installed)
# 5. spaCy model download (cached unless pip install layer changes)
FROM python:3.11-slim
@@ -23,7 +16,6 @@ RUN apt-get update && apt-get install -y \
build-essential \
curl \
xz-utils \
chromium \
&& rm -rf /var/lib/apt/lists/*
# Install comprehensive font support for international lyrics and symbols
@@ -58,27 +50,16 @@ RUN curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-stati
# Set working directory
WORKDIR /app
# Copy ONLY dependency specification files first (changes less often than source code)
# Copy dependency files and source code needed for pip install
COPY pyproject.toml README.md LICENSE /app/
# Create minimal package stubs so pip can resolve the editable install
# pip install -e . needs package directories to exist but doesn't validate content
RUN mkdir -p /app/karaoke_gen /app/lyrics_transcriber_temp /app/backend && \
touch /app/karaoke_gen/__init__.py /app/lyrics_transcriber_temp/__init__.py /app/backend/__init__.py
# Install all Python dependencies (the slow part)
# BuildKit cache mount preserves downloaded wheels across builds, so partial
# dependency updates only download changed packages instead of everything
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --upgrade pip && \
pip install -e .
# Now copy actual source code (AFTER deps installed, so code changes don't
# invalidate the expensive pip install layer in registry cache)
COPY karaoke_gen /app/karaoke_gen
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
COPY backend /app/backend
# Install all Python dependencies (the slow part - cached in base image)
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir -e .
# Download spaCy language model for lyrics transcription
# This must be done at build time, not runtime (Cloud Run network restrictions)
RUN python -m spacy download en_core_web_sm
-136
View File
@@ -1,136 +0,0 @@
# GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models
# This image is rebuilt only when deps or models change (infrequently).
#
# Build with: gcloud builds submit --config=backend/cloudbuild-gpu-base.yaml --project=nomadkaraoke --timeout=3600s
# Or manually: docker build -f backend/Dockerfile.gpu-base -t karaoke-backend-gpu-base .
#
# Layer caching strategy:
# 1. System deps + fonts + ffmpeg (rarely change, always cached)
# 2. CUDA library path configuration
# 3. pyproject.toml + package stubs (changes with dep updates)
# 4. pip install karaoke-gen deps + audio-separator[gpu] (slow, cached unless pyproject.toml changes)
# 5. Source code copy (changes every rebuild, but AFTER deps are installed)
# 6. spaCy model download (cached unless pip install layer changes)
# 7. Pre-downloaded ensemble preset models (~1-1.5 GB, cached unless pip layer changes)
FROM nvidia/cuda:12.6.3-runtime-ubuntu22.04
# Prevent interactive prompts during package installation
ENV DEBIAN_FRONTEND=noninteractive
# Install Python 3.11 from deadsnakes PPA and system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
software-properties-common \
&& add-apt-repository -y ppa:deadsnakes/ppa \
&& apt-get update && apt-get install -y --no-install-recommends \
# Python 3.11
python3.11 \
python3.11-dev \
python3.11-venv \
# Audio libraries
libsndfile1 \
libsox-dev \
sox \
# Build tools
build-essential \
# Utilities
curl \
xz-utils \
&& rm -rf /var/lib/apt/lists/* \
&& python3.11 --version
# Install comprehensive font support for international lyrics and symbols
# Noto fonts provide full Unicode coverage including:
# - Latin, Greek, Cyrillic (core)
# - CJK: Chinese, Japanese, Korean (~150MB)
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
# - Color emoji (limited support in libass, but available)
# - Musical symbols (♪ ♫ etc.)
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
RUN apt-get update && apt-get install -y --no-install-recommends \
fonts-noto-core \
fonts-noto-cjk \
fonts-noto-extra \
fonts-noto-color-emoji \
fontconfig \
&& fc-cache -f \
&& rm -rf /var/lib/apt/lists/*
# Install FFmpeg from Ubuntu package
# The audio worker only uses FFmpeg for format conversion and padding, not
# encoding-intensive tasks, so the Debian package is sufficient. The static
# build from johnvansickle.com is used for the encoding worker instead.
RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*
# Set Python 3.11 as default and install pip
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1 \
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.11 1 \
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11 \
&& python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel
# Set up CUDA library paths so GPU-accelerated packages can find CUDA libs at runtime
RUN echo '/usr/local/cuda/lib64' >> /etc/ld.so.conf.d/cuda.conf && ldconfig
# Environment configuration for CUDA and Python
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH \
PATH=/usr/local/cuda/bin:$PATH \
PYTHONUNBUFFERED=1 \
MODEL_DIR=/models
# Set working directory
WORKDIR /app
# Copy ONLY dependency specification files first (changes less often than source code)
COPY pyproject.toml README.md LICENSE /app/
# Create minimal package stubs so pip can resolve the editable install
# pip install -e . needs package directories to exist but doesn't validate content
RUN mkdir -p /app/karaoke_gen /app/lyrics_transcriber_temp /app/backend && \
touch /app/karaoke_gen/__init__.py /app/lyrics_transcriber_temp/__init__.py /app/backend/__init__.py
# Install all karaoke-gen Python dependencies (the slow part)
# BuildKit cache mount preserves downloaded wheels across builds
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --upgrade pip && \
pip install -e .
# Install audio-separator with GPU support (PyTorch + CUDA + onnxruntime-gpu)
# This is separate from pip install -e . to keep layer caching granular
#
# IMPORTANT: Install PyTorch cu126 FIRST, then audio-separator[gpu].
# The default audio-separator[gpu] pulls PyTorch cu130, but Cloud Run L4 GPUs
# have NVIDIA driver 570 which only supports up to CUDA 12.8. PyTorch cu130
# fails with "NVIDIA driver is too old". PyTorch cu126 works with 12.8 drivers.
# Installing torch first means audio-separator[gpu] sees it as already satisfied.
RUN --mount=type=cache,target=/root/.cache/pip \
pip install torch==2.6.0+cu126 torchvision==0.21.0+cu126 --index-url https://download.pytorch.org/whl/cu126 && \
pip install "audio-separator[gpu]"
# Now copy actual source code (AFTER deps installed, so code changes don't
# invalidate the expensive pip install layer in registry cache)
COPY karaoke_gen /app/karaoke_gen
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
COPY backend /app/backend
# Download spaCy language model for lyrics transcription
# Must be done at build time, not runtime (Cloud Run network restrictions)
RUN python -m spacy download en_core_web_sm
# Create models directory and pre-download ensemble preset models
# Baking models into the image eliminates cold-start model download latency.
# Total model size: ~1-1.5 GB for instrumental_clean + karaoke presets.
RUN mkdir -p /models
COPY backend/scripts/download_models.py /tmp/download_models.py
RUN python /tmp/download_models.py /models && \
rm /tmp/download_models.py && \
ls -lh /models/
# Label with build info
ARG BUILD_DATE
ARG BUILD_ID
ARG BASE_CONTENT_HASH
LABEL org.opencontainers.image.created="${BUILD_DATE}"
LABEL org.opencontainers.image.revision="${BUILD_ID}"
LABEL org.opencontainers.image.title="karaoke-backend-gpu-base"
LABEL org.opencontainers.image.description="GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models for karaoke-backend audio worker"
LABEL base.content.hash="${BASE_CONTENT_HASH}"
-14
View File
@@ -113,19 +113,6 @@ async def require_auth(
detail=f"Authentication failed: {auth_result.message}"
)
# If token has a tenant_id, override the hostname-based tenant detection.
# This allows tenant-scoped tokens to enforce tenant context regardless of domain.
# We must also load tenant_config so downstream handlers (e.g. job creation)
# that rely on tenant_config for setting tenant_id on jobs work correctly.
if auth_result.tenant_id:
request.state.tenant_id = auth_result.tenant_id
if not getattr(request.state, "tenant_config", None):
from backend.services.tenant_service import get_tenant_service
tenant_service = get_tenant_service()
tenant_config = tenant_service.get_tenant_config(auth_result.tenant_id)
if tenant_config and tenant_config.is_active:
request.state.tenant_config = tenant_config
# Log successful auth with request_id for correlation with request audit
logger.info(
"auth_success",
@@ -135,7 +122,6 @@ async def require_auth(
"user_type": auth_result.user_type.value if auth_result.user_type else None,
"is_admin": auth_result.is_admin,
"remaining_uses": auth_result.remaining_uses,
"tenant_id": auth_result.tenant_id,
"audit_type": "auth_event",
}
)
File diff suppressed because it is too large. Load diff
+99 -381
View File
@@ -17,8 +17,6 @@ import mimetypes
import os
import shutil
import tempfile
import uuid
from datetime import datetime, timedelta
from typing import Optional, List, Dict, Any
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request, Depends
@@ -37,7 +35,6 @@ from backend.services.audio_search_service import (
NoResultsError,
DownloadError,
)
from backend.services.flacfetch_client import get_flacfetch_client, FlacfetchServiceError
from backend.services.youtube_download_service import (
get_youtube_download_service,
YouTubeDownloadError,
@@ -49,11 +46,7 @@ from backend.version import VERSION
from backend.api.dependencies import require_auth
from backend.services.auth_service import UserType, AuthResult
from backend.middleware.tenant import get_tenant_config_from_request
from backend.exceptions import InsufficientCreditsError
from backend.services.tracing import add_span_attribute
from backend.services.firestore_service import FirestoreService
from pathlib import Path
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
@@ -141,9 +134,6 @@ class AudioSearchRequest(BaseModel):
# Non-interactive mode
non_interactive: bool = Field(False, description="Skip interactive steps (lyrics review, instrumental selection)")
# Private (non-published) track mode
is_private: bool = Field(False, description="Private track: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive")
@validator('artist', 'title', 'lyrics_artist', 'lyrics_title', 'display_artist', 'display_title')
def normalize_text_fields(cls, v):
"""Normalize text fields to standardize Unicode characters."""
@@ -202,32 +192,9 @@ class AudioSearchResponse(BaseModel):
style_upload_urls: Optional[List[StyleUploadUrl]] = None
class StandaloneSearchRequest(BaseModel):
"""Request for standalone audio search (no job created)."""
artist: str = Field(..., description="Artist name to search for")
title: str = Field(..., description="Song title to search for")
@validator('artist', 'title')
def normalize_text_fields(cls, v):
if v is not None and isinstance(v, str):
return normalize_text(v)
return v
class StandaloneSearchResponse(BaseModel):
"""Response from standalone audio search."""
search_session_id: str
results: List[AudioSearchResultResponse]
results_count: int
class AudioSelectRequest(BaseModel):
"""Request to select an audio source."""
selection_index: int = Field(..., description="Index of the selected audio source from search results")
# Optional overrides — sent from the guided flow's Customize step (Step 3)
is_private: Optional[bool] = Field(None, description="Override privacy flag (guided flow Step 3)")
display_artist: Optional[str] = Field(None, description="Override display artist name")
display_title: Optional[str] = Field(None, description="Override display title")
class AudioSelectResponse(BaseModel):
@@ -241,24 +208,24 @@ class AudioSelectResponse(BaseModel):
selected_provider: str
def extract_request_metadata(request: Request, created_from: str = "audio_search", auth_result=None) -> Dict[str, Any]:
def extract_request_metadata(request: Request, created_from: str = "audio_search") -> Dict[str, Any]:
"""Extract metadata from request for job tracking."""
headers = dict(request.headers)
client_ip = headers.get('x-forwarded-for', '').split(',')[0].strip()
if not client_ip and request.client:
client_ip = request.client.host
user_agent = headers.get('user-agent', '')
environment = headers.get('x-environment', '')
client_id = headers.get('x-client-id', '')
custom_headers = {}
for key, value in headers.items():
if key.lower().startswith('x-') and key.lower() not in ('x-forwarded-for', 'x-forwarded-proto', 'x-forwarded-host'):
custom_headers[key] = value
metadata = {
return {
'client_ip': client_ip,
'user_agent': user_agent,
'environment': environment,
@@ -268,16 +235,13 @@ def extract_request_metadata(request: Request, created_from: str = "audio_search
'custom_headers': custom_headers,
}
# Add auth context for abuse investigation and analytics
if auth_result is not None:
metadata['auth_method'] = getattr(auth_result, 'auth_method', None)
user_type = getattr(auth_result, 'user_type', None)
metadata['user_type'] = user_type.value if hasattr(user_type, 'value') else str(user_type) if user_type else None
remaining = getattr(auth_result, 'remaining_uses', None)
if remaining is not None:
metadata['credits_at_creation'] = remaining
return metadata
async def _trigger_workers_parallel(job_id: str) -> None:
"""Trigger both audio and lyrics workers in parallel."""
await asyncio.gather(
worker_service.trigger_audio_worker(job_id),
worker_service.trigger_lyrics_worker(job_id)
)
def _extract_gcs_path(filepath: str) -> str:
@@ -298,33 +262,43 @@ def _extract_gcs_path(filepath: str) -> str:
return filepath
def _validate_and_prepare_selection(
async def _download_and_start_processing(
job_id: str,
selection_index: int,
audio_search_service,
background_tasks: BackgroundTasks,
) -> Dict[str, Any]:
"""
Validate selection and transition job to DOWNLOADING_AUDIO state.
Returns selection info dict with the selected result details.
Raises HTTPException on validation errors.
Download selected audio and start job processing.
This is called either:
- Immediately after search if auto_download=True
- When user calls the select endpoint
For remote flacfetch downloads (torrent sources), the file is downloaded
on the flacfetch VM and uploaded directly to GCS. For local downloads
(YouTube), the file is downloaded locally and then uploaded to GCS.
"""
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=t("en", "audioSearch.jobNotFound", job_id=job_id))
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Get search results from state_data
search_results = job.state_data.get('audio_search_results', [])
if not search_results:
raise HTTPException(status_code=400, detail=t("en", "audioSearch.noSearchResults"))
raise HTTPException(status_code=400, detail="No search results available")
if selection_index < 0 or selection_index >= len(search_results):
raise HTTPException(
status_code=400,
detail=t("en", "audioSearch.invalidSelectionIndex", selection_index=selection_index, max=len(search_results)-1)
detail=f"Invalid selection index {selection_index}. Valid range: 0-{len(search_results)-1}"
)
selected = search_results[selection_index]
# Get remote_search_id from state_data (stored during initial search)
remote_search_id = job.state_data.get('remote_search_id')
# Transition to downloading state
job_manager.transition_to_state(
job_id=job_id,
@@ -336,59 +310,8 @@ def _validate_and_prepare_selection(
'selected_audio_provider': selected['provider'],
}
)
# Save download params to job BEFORE attempting download
# This enables retry even if download fails
source_name = selected.get('provider')
source_id = selected.get('source_id')
target_file = selected.get('target_file')
download_url = selected.get('url')
job_manager.update_job(job_id, {
'audio_source_type': 'audio_search',
'source_name': source_name,
'source_id': source_id,
'target_file': target_file,
'download_url': download_url,
})
logger.info(
f"Job {job_id}: Saved download params before attempt: "
f"source_name={source_name}, source_id={source_id}"
)
return {
'selected_index': selection_index,
'selected_title': selected['title'],
'selected_artist': selected['artist'],
'selected_provider': selected['provider'],
}
async def _download_audio_and_trigger_workers(
job_id: str,
selection_index: int,
audio_search_service,
) -> None:
"""
Download audio and trigger processing workers.
This runs as a background task after the select endpoint returns.
Also called synchronously from the auto_download path during search.
For remote flacfetch downloads (torrent sources), the file is downloaded
on the flacfetch VM and uploaded directly to GCS. For local downloads
(YouTube), the file is downloaded locally and then uploaded to GCS.
"""
try:
job = job_manager.get_job(job_id)
if not job:
logger.error(f"Job {job_id} not found during background download")
return
search_results = job.state_data.get('audio_search_results', [])
selected = search_results[selection_index]
remote_search_id = job.state_data.get('remote_search_id')
# Get source info from selected result
source_name = selected.get('provider')
source_id = selected.get('source_id')
@@ -422,9 +345,8 @@ async def _download_audio_and_trigger_workers(
raise DownloadError(f"YouTube download failed: {e}")
elif source_name in ['RED', 'OPS']:
# Torrent sources - must use remote flacfetch (async, no event loop blocking)
flacfetch_client = get_flacfetch_client()
if not flacfetch_client:
# Torrent sources - must use remote flacfetch
if not audio_search_service.is_remote_enabled():
raise DownloadError(
f"Cannot download from {source_name} without remote flacfetch service. "
"Configure FLACFETCH_API_URL."
@@ -434,48 +356,31 @@ async def _download_audio_and_trigger_workers(
if source_id:
logger.info(f"Torrent download via download_by_id: {source_name} ID={source_id}")
download_id = await flacfetch_client.download_by_id(
result = audio_search_service.download_by_id(
source_name=source_name,
source_id=source_id,
output_dir="",
target_file=target_file,
download_url=download_url,
gcs_path=gcs_destination,
)
else:
if not remote_search_id:
raise DownloadError(
f"No source_id or remote_search_id available for {source_name} download"
)
logger.info(f"Torrent download via search-based download")
download_id = await flacfetch_client.download(
search_id=remote_search_id,
result = audio_search_service.download(
result_index=selection_index,
output_dir="",
gcs_path=gcs_destination,
remote_search_id=remote_search_id,
)
def log_progress(status):
progress = status.get("progress", 0)
speed = status.get("download_speed_kbps", 0)
logger.debug(f"Download progress: {progress:.1f}% ({speed:.1f} KB/s)")
final_status = await flacfetch_client.wait_for_download(
download_id,
timeout=600,
progress_callback=log_progress,
)
filepath = final_status.get("gcs_path") or final_status.get("output_path")
if not filepath:
raise DownloadError("Remote download completed but no file path returned")
audio_gcs_path = _extract_gcs_path(filepath)
filename = os.path.basename(filepath)
# Extract GCS path from result
audio_gcs_path = _extract_gcs_path(result.filepath)
filename = os.path.basename(result.filepath)
logger.info(f"Torrent download complete: {audio_gcs_path}")
elif source_name == 'Spotify':
# Spotify downloads - use async FlacfetchClient directly (no event loop blocking)
flacfetch_client = get_flacfetch_client()
if not flacfetch_client:
# Spotify downloads - use audio_search_service
if not audio_search_service.is_remote_enabled():
raise DownloadError(
f"Cannot download from Spotify without remote flacfetch service. "
"Configure FLACFETCH_API_URL."
@@ -484,40 +389,25 @@ async def _download_audio_and_trigger_workers(
gcs_destination = f"uploads/{job_id}/audio/"
logger.info(f"Spotify download: source_id={source_id}")
download_id = await flacfetch_client.download_by_id(
result = audio_search_service.download_by_id(
source_name=source_name,
source_id=source_id,
output_dir="",
download_url=download_url,
gcs_path=gcs_destination,
)
def log_progress(status):
progress = status.get("progress", 0)
speed = status.get("download_speed_kbps", 0)
logger.debug(f"Download progress: {progress:.1f}% ({speed:.1f} KB/s)")
final_status = await flacfetch_client.wait_for_download(
download_id,
timeout=600,
progress_callback=log_progress,
)
filepath = final_status.get("gcs_path") or final_status.get("output_path")
if not filepath:
raise DownloadError("Remote download completed but no file path returned")
audio_gcs_path = _extract_gcs_path(filepath)
filename = os.path.basename(filepath)
audio_gcs_path = _extract_gcs_path(result.filepath)
filename = os.path.basename(result.filepath)
logger.info(f"Spotify download complete: {audio_gcs_path}")
else:
# Unknown source - try generic download (run sync code in thread to avoid blocking)
# Unknown source - try generic download
logger.warning(f"Unknown source type: {source_name}, attempting generic download")
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
try:
result = await asyncio.to_thread(
audio_search_service.download,
result = audio_search_service.download(
result_index=selection_index,
output_dir=temp_dir,
remote_search_id=remote_search_id,
@@ -543,36 +433,36 @@ async def _download_audio_and_trigger_workers(
shutil.rmtree(temp_dir)
except Exception as e:
logger.warning(f"Failed to clean up temp files: {e}")
# Update job with GCS path
# Update job with GCS path and transition to DOWNLOADING
job_manager.update_job(job_id, {
'input_media_gcs_path': audio_gcs_path,
'filename': filename,
})
# Transition from DOWNLOADING_AUDIO to DOWNLOADING and trigger workers
# Note: Can't use start_job_processing() here because that expects PENDING status,
# but we're in DOWNLOADING_AUDIO state from the audio search flow.
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.DOWNLOADING,
progress=15,
message="Audio downloaded, starting processing"
)
# Trigger workers directly
worker_service = get_worker_service()
await asyncio.gather(
worker_service.trigger_audio_worker(job_id),
worker_service.trigger_lyrics_worker(job_id)
)
except (DownloadError, FlacfetchServiceError) as e:
logger.error(f"Download failed for job {job_id}: {e}")
# Trigger workers
background_tasks.add_task(_trigger_workers_parallel, job_id)
return {
'selected_index': selection_index,
'selected_title': selected['title'],
'selected_artist': selected['artist'],
'selected_provider': selected['provider'],
}
except DownloadError as e:
job_manager.fail_job(job_id, f"Audio download failed: {e}")
raise HTTPException(status_code=500, detail=f"Download failed: {e}")
except Exception as e:
logger.error(f"Download failed for job {job_id}: {e}", exc_info=True)
job_manager.fail_job(job_id, f"Audio download failed: {e}")
raise HTTPException(status_code=500, detail=f"Download failed: {e}")
@router.post("/audio-search/search", response_model=AudioSearchResponse)
@@ -595,14 +485,12 @@ async def search_audio(
- Interactive mode (default): Returns results, user calls /select endpoint
- Auto mode (auto_download=True): Automatically selects and downloads best
"""
locale = get_locale_from_request(request)
# Check tenant feature flag
tenant_config = get_tenant_config_from_request(request)
if tenant_config and not tenant_config.features.audio_search:
raise HTTPException(
status_code=403,
detail=t(locale, "audioSearch.notEnabled")
detail="Audio search is not available for this portal"
)
try:
@@ -647,9 +535,8 @@ async def search_audio(
}
)
# Extract request metadata (includes auth context + search query info)
request_metadata = extract_request_metadata(request, created_from="audio_search", auth_result=auth_result)
request_metadata['search_query'] = {'artist': body.artist, 'title': body.title}
# Extract request metadata
request_metadata = extract_request_metadata(request, created_from="audio_search")
# Apply default theme if none specified
# This ensures all karaoke videos use the Nomad theme by default
@@ -701,18 +588,12 @@ async def search_audio(
auto_download=body.auto_download,
request_metadata=request_metadata,
non_interactive=body.non_interactive,
is_private=body.is_private,
# Tenant scoping
tenant_id=tenant_config.id if tenant_config else "",
tenant_id=tenant_config.id if tenant_config else None,
)
job = job_manager.create_job(job_create, is_admin=auth_result.is_admin)
job_id = job.job_id
# Trace attributes for observability
add_span_attribute("job_id", job_id)
add_span_attribute("job.source", "audio_search")
add_span_attribute("job.is_private", body.is_private)
logger.info(f"Created job {job_id} for audio search: {body.artist} - {body.title}")
# Update job with audio search fields
@@ -769,11 +650,11 @@ async def search_audio(
# Generate GCS path
if file_info.file_type == 'style_params':
gcs_path = f"jobs/{job_id}/style/style_params.json"
gcs_path = f"uploads/{job_id}/style/style_params.json"
else:
# style_intro_background -> intro_background, etc.
asset_key = file_info.file_type.replace('style_', '')
gcs_path = f"jobs/{job_id}/style/{asset_key}{ext}"
gcs_path = f"uploads/{job_id}/style/{asset_key}{ext}"
# Generate signed upload URL
signed_url = storage_service.generate_signed_upload_url(
@@ -815,7 +696,7 @@ async def search_audio(
audio_search_service = get_audio_search_service()
try:
search_results = await audio_search_service.search_async(body.artist, body.title)
search_results = audio_search_service.search(body.artist, body.title)
except NoResultsError as e:
job_manager.fail_job(job_id, f"No audio sources found for: {body.artist} - {body.title}")
raise HTTPException(
@@ -846,21 +727,16 @@ async def search_audio(
# If auto_download, select best and start processing
if body.auto_download:
best_index = audio_search_service.select_best(search_results)
logger.info(f"Auto-download enabled, selecting result {best_index}")
selection_info = _validate_and_prepare_selection(
selection_info = await _download_and_start_processing(
job_id=job_id,
selection_index=best_index,
audio_search_service=audio_search_service,
background_tasks=background_tasks,
)
# Trigger audio download as a Cloud Run Job (survives instance shutdown)
worker_service = get_worker_service()
triggered = await worker_service.trigger_audio_download_worker(job_id)
if not triggered:
job_manager.fail_job(job_id, "Failed to trigger audio download worker")
raise HTTPException(status_code=500, detail="Failed to trigger audio download worker")
return AudioSearchResponse(
status="success",
job_id=job_id,
@@ -948,149 +824,10 @@ async def search_audio(
except HTTPException:
raise
except Exception as e:
if isinstance(e, InsufficientCreditsError):
raise
logger.error(f"Error in audio search: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e))
@router.post("/audio-search/search-standalone", response_model=StandaloneSearchResponse)
async def search_audio_standalone(
request: Request,
body: StandaloneSearchRequest,
auth_result: AuthResult = Depends(require_auth)
):
"""
Search for audio by artist and title WITHOUT creating a job.
Returns a search session ID and results. Call POST /api/jobs/create-from-search
with the session ID and selected result index to create the actual job.
Credits are checked here (early feedback) but NOT deducted — deduction
happens when the job is created via create-from-search.
"""
locale = get_locale_from_request(request)
# Check tenant feature flag
tenant_config = get_tenant_config_from_request(request)
if tenant_config and not tenant_config.features.audio_search:
raise HTTPException(
status_code=403,
detail=t(locale, "audioSearch.notEnabled")
)
try:
user_email = auth_result.user_email
if not user_email:
raise HTTPException(status_code=401, detail="Authentication required")
# Credit check — give early feedback but do NOT deduct here
if not auth_result.is_admin:
from backend.services.user_service import get_user_service
user_service = get_user_service()
if not user_service.has_credits(user_email):
credits_available = user_service.check_credits(user_email)
raise InsufficientCreditsError(
message="You're out of credits. Buy more to continue creating karaoke videos.",
credits_available=credits_available,
credits_required=1,
)
# Perform the search (runs in thread to avoid blocking the event loop)
audio_search_service = get_audio_search_service()
try:
search_results = await audio_search_service.search_async(body.artist, body.title)
except NoResultsError:
# No results is a normal outcome — return empty list, not an error
search_results = []
except AudioSearchError as e:
raise HTTPException(status_code=500, detail=f"Search failed: {e}")
# Build result response objects
result_responses: List[AudioSearchResultResponse] = []
results_dicts: List[Dict[str, Any]] = []
for r in search_results:
raw_dict: Dict[str, Any] = {}
if r.raw_result:
if isinstance(r.raw_result, dict):
raw_dict = r.raw_result
else:
try:
raw_dict = r.raw_result.to_dict()
except AttributeError:
pass
result_responses.append(
AudioSearchResultResponse(
index=r.index,
title=r.title,
artist=r.artist,
provider=r.provider,
url=r.url,
duration=r.duration,
quality=r.quality,
source_id=r.source_id,
seeders=raw_dict.get('seeders') or r.seeders,
target_file=raw_dict.get('target_file') or r.target_file,
year=raw_dict.get('year'),
label=raw_dict.get('label'),
edition_info=raw_dict.get('edition_info'),
release_type=raw_dict.get('release_type'),
channel=raw_dict.get('channel'),
view_count=raw_dict.get('view_count'),
size_bytes=raw_dict.get('size_bytes'),
target_file_size=raw_dict.get('target_file_size'),
track_pattern=raw_dict.get('track_pattern'),
match_score=raw_dict.get('match_score'),
formatted_size=raw_dict.get('formatted_size'),
formatted_duration=raw_dict.get('formatted_duration'),
formatted_views=raw_dict.get('formatted_views'),
is_lossless=raw_dict.get('is_lossless'),
quality_str=raw_dict.get('quality_str'),
quality_data=raw_dict.get('quality_data') or raw_dict.get('quality'),
)
)
results_dicts.append(r.to_dict())
# Store session in Firestore (TTL: 7 days)
# Long TTL so users who leave a tab open mid-flow don't hit "Search expired".
# Sessions are tiny docs (~2 KB) and cleaned up by Firestore TTL policy.
session_id = str(uuid.uuid4())
now = datetime.utcnow()
session_data = {
'session_id': session_id,
'user_email': user_email,
'tenant_id': tenant_config.id if tenant_config else None,
'artist': body.artist,
'title': body.title,
'results': results_dicts,
'remote_search_id': audio_search_service.last_remote_search_id,
'created_at': now.isoformat(),
'ttl_expiry': now + timedelta(days=7),
}
firestore_service = FirestoreService()
firestore_service.create_search_session(session_data)
logger.info(
f"Standalone search complete for {body.artist} - {body.title}: "
f"{len(search_results)} results, session={session_id}"
)
return StandaloneSearchResponse(
search_session_id=session_id,
results=result_responses,
results_count=len(result_responses),
)
except HTTPException:
raise
except InsufficientCreditsError:
raise
except Exception as e:
logger.error(f"Error in standalone audio search: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e))
@router.get("/audio-search/{job_id}/results")
async def get_audio_search_results(
job_id: str,
@@ -1103,14 +840,14 @@ async def get_audio_search_results(
"""
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=t("en", "audioSearch.jobNotFound", job_id=job_id))
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
search_results = job.state_data.get('audio_search_results', [])
if not search_results:
raise HTTPException(
status_code=400,
detail=t("en", "audioSearch.noSearchResults")
detail="No search results available for this job"
)
return {
@@ -1133,64 +870,45 @@ async def select_audio_source(
):
"""
Select an audio source and start job processing.
This endpoint returns immediately after validating the selection and
transitioning the job to DOWNLOADING_AUDIO state. The actual download
runs as a background task so the UI can close the selection dialog
without waiting for the download to complete.
This endpoint:
1. Validates the job is awaiting selection
2. Downloads the selected audio
3. Uploads to GCS
4. Triggers audio and lyrics workers
"""
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=t("en", "audioSearch.jobNotFound", job_id=job_id))
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Verify job is awaiting selection
if job.status != JobStatus.AWAITING_AUDIO_SELECTION:
raise HTTPException(
status_code=400,
detail=f"Job is not awaiting audio selection (status: {job.status})"
)
# Get search service instance
# Note: With download_by_id, we no longer need to re-search to populate the cache.
# The source_id stored in job.state_data['audio_search_results'] is sufficient.
audio_search_service = get_audio_search_service()
# Apply optional overrides from guided flow Step 3 (Customize)
# These fields are set AFTER the job is created (Step 2), so they must be
# patched here before the pipeline starts.
overrides = {}
if body.is_private is not None:
overrides["is_private"] = body.is_private
if body.display_artist is not None:
overrides["display_artist"] = body.display_artist
if body.display_title is not None:
overrides["display_title"] = body.display_title
if overrides:
job_manager.update_job(job_id, overrides)
logger.info(f"[job:{job_id}] Applied selection overrides: {list(overrides.keys())}")
# Validate search results exist in job state_data
search_results = job.state_data.get('audio_search_results', [])
if not search_results:
raise HTTPException(status_code=400, detail="No search results cached for this job")
# Validate selection and transition to DOWNLOADING_AUDIO (returns immediately)
selection_info = _validate_and_prepare_selection(
selection_info = await _download_and_start_processing(
job_id=job_id,
selection_index=body.selection_index,
audio_search_service=audio_search_service,
background_tasks=background_tasks,
)
# Trigger audio download as a Cloud Run Job (survives instance shutdown)
worker_service = get_worker_service()
triggered = await worker_service.trigger_audio_download_worker(job_id)
if not triggered:
job_manager.fail_job(job_id, "Failed to trigger audio download worker")
raise HTTPException(status_code=500, detail="Failed to trigger audio download worker")
return AudioSelectResponse(
status="success",
job_id=job_id,
message="Audio selected, download starting",
message="Audio selected and download started",
selected_index=selection_info['selected_index'],
selected_title=selection_info['selected_title'],
selected_artist=selection_info['selected_artist'],
+35 -41
View File
@@ -9,7 +9,7 @@ Provides endpoints for:
import logging
from typing import Optional
from datetime import datetime
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request
from fastapi import APIRouter, HTTPException, BackgroundTasks
from pydantic import BaseModel
from backend.services.credential_manager import (
@@ -18,7 +18,6 @@ from backend.services.credential_manager import (
CredentialCheckResult,
)
from backend.config import get_settings
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/auth", tags=["auth"])
@@ -126,16 +125,15 @@ async def get_credentials_status():
@router.get("/status/{service}", response_model=CredentialStatusResponse)
async def get_service_credential_status(service: str, request: Request):
async def get_service_credential_status(service: str):
"""
Get the status of a specific service's OAuth credentials.
Args:
service: One of "youtube", "gdrive", "dropbox"
"""
locale = get_locale_from_request(request)
manager = get_credential_manager()
if service == "youtube":
result = manager.check_youtube_credentials()
elif service == "gdrive":
@@ -143,7 +141,7 @@ async def get_service_credential_status(service: str, request: Request):
elif service == "dropbox":
result = manager.check_dropbox_credentials()
else:
raise HTTPException(status_code=400, detail=t(locale, "auth.unknownService", service=service))
raise HTTPException(status_code=400, detail=f"Unknown service: {service}")
return CredentialStatusResponse(
service=result.service,
@@ -155,43 +153,42 @@ async def get_service_credential_status(service: str, request: Request):
@router.post("/validate", response_model=CredentialValidationResponse)
async def validate_credentials(request: CredentialValidationRequest, http_request: Request):
async def validate_credentials(request: CredentialValidationRequest):
"""
Validate that credentials are available for requested services.
Use this before submitting a job to ensure all required
credentials are valid.
"""
locale = get_locale_from_request(http_request)
manager = get_credential_manager()
invalid_services = []
if request.youtube:
result = manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append("youtube")
if request.gdrive:
result = manager.check_gdrive_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append("gdrive")
if request.dropbox:
result = manager.check_dropbox_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append("dropbox")
if invalid_services:
return CredentialValidationResponse(
valid=False,
invalid_services=invalid_services,
message=t(locale, "auth.invalidCredentials", services=", ".join(invalid_services))
message=f"The following services need re-authorization: {', '.join(invalid_services)}"
)
return CredentialValidationResponse(
valid=True,
invalid_services=[],
message=t(locale, "auth.validCredentials")
message="All requested credentials are valid"
)
@@ -200,41 +197,40 @@ async def validate_credentials(request: CredentialValidationRequest, http_reques
# ===========================================================================
@router.post("/youtube/device", response_model=DeviceAuthStartResponse)
async def start_youtube_device_auth(request: Optional[DeviceAuthStartRequest] = None, http_request: Request = None):
async def start_youtube_device_auth(request: Optional[DeviceAuthStartRequest] = None):
"""
Start YouTube device authorization flow.
This initiates a device auth flow that allows authorization
from any device. The user must visit the verification URL
and enter the user code.
Client credentials are loaded from Secret Manager ('youtube-client-credentials')
unless explicitly provided in the request body.
After starting, poll /auth/youtube/device/{device_code} to
check for completion.
"""
locale = get_locale_from_request(http_request) if http_request else "en"
manager = get_credential_manager()
try:
device_info = manager.start_youtube_device_auth(
client_id=request.client_id if request else None,
client_secret=request.client_secret if request else None
)
return DeviceAuthStartResponse(
device_code=device_info.device_code,
user_code=device_info.user_code,
verification_url=device_info.verification_url,
expires_in=device_info.expires_in,
interval=device_info.interval,
instructions=t(locale, "auth.youtubeAuthInstructions", verification_url=device_info.verification_url, user_code=device_info.user_code)
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
)
except Exception as e:
logger.error(f"Failed to start YouTube device auth: {e}")
raise HTTPException(status_code=500, detail=t(locale, "auth.youtubeAuthStartFailed", error=str(e)))
raise HTTPException(status_code=500, detail=str(e))
@router.get("/youtube/device/{device_code}", response_model=DeviceAuthPollResponse)
@@ -263,36 +259,35 @@ async def poll_youtube_device_auth(device_code: str):
@router.post("/gdrive/device", response_model=DeviceAuthStartResponse)
async def start_gdrive_device_auth(request: Optional[DeviceAuthStartRequest] = None, http_request: Request = None):
async def start_gdrive_device_auth(request: Optional[DeviceAuthStartRequest] = None):
"""
Start Google Drive device authorization flow.
Client credentials are loaded from Secret Manager ('gdrive-client-credentials')
unless explicitly provided in the request body.
After starting, poll /auth/gdrive/device/{device_code} to check for completion.
"""
locale = get_locale_from_request(http_request) if http_request else "en"
manager = get_credential_manager()
try:
device_info = manager.start_gdrive_device_auth(
client_id=request.client_id if request else None,
client_secret=request.client_secret if request else None
)
return DeviceAuthStartResponse(
device_code=device_info.device_code,
user_code=device_info.user_code,
verification_url=device_info.verification_url,
expires_in=device_info.expires_in,
interval=device_info.interval,
instructions=t(locale, "auth.gdriveLinkInstructions", verification_url=device_info.verification_url, user_code=device_info.user_code)
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
)
except Exception as e:
logger.error(f"Failed to start Google Drive device auth: {e}")
raise HTTPException(status_code=500, detail=t(locale, "auth.gdriveAuthStartFailed", error=str(e)))
raise HTTPException(status_code=500, detail=str(e))
@router.get("/gdrive/device/{device_code}", response_model=DeviceAuthPollResponse)
@@ -316,20 +311,19 @@ async def poll_gdrive_device_auth(device_code: str):
# ===========================================================================
@router.post("/test-alert")
async def test_credential_alert(background_tasks: BackgroundTasks, request: Request):
async def test_credential_alert(background_tasks: BackgroundTasks):
"""
Test the credential alert mechanism.
Sends a test alert to Discord if configured.
"""
locale = get_locale_from_request(request)
settings = get_settings()
discord_url = settings.get_secret("discord-alert-webhook")
if not discord_url:
raise HTTPException(
status_code=400,
detail=t(locale, "auth.discordAlertNotConfigured")
detail="Discord alert webhook not configured"
)
manager = get_credential_manager()
-163
View File
@@ -1,163 +0,0 @@
"""
Catalog API routes for song/artist lookup and community version detection.
Provides autocomplete data from karaoke-decide's MusicBrainz + Spotify catalog,
and checks karaokenerds.com for existing community karaoke versions.
"""
import logging
import time
from typing import Optional
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from pydantic import BaseModel, Field
from backend.api.dependencies import require_auth
from backend.services.auth_service import AuthResult
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/catalog", tags=["catalog"])
# Simple per-user rate limiting: 20 requests/minute
# { user_email: [timestamp, timestamp, ...] }
_rate_limit_windows: dict[str, list[float]] = {}
RATE_LIMIT_MAX_REQUESTS = 20
RATE_LIMIT_WINDOW_SECONDS = 60
def _check_rate_limit(user_email: str, locale: str = "en") -> None:
"""Check per-user rate limit. Raises 429 if exceeded."""
now = time.monotonic()
window_start = now - RATE_LIMIT_WINDOW_SECONDS
# Get or create window for this user
if user_email not in _rate_limit_windows:
_rate_limit_windows[user_email] = []
# Remove expired entries
timestamps = _rate_limit_windows[user_email]
_rate_limit_windows[user_email] = [ts for ts in timestamps if ts > window_start]
if len(_rate_limit_windows[user_email]) >= RATE_LIMIT_MAX_REQUESTS:
raise HTTPException(
status_code=429,
detail=t(locale, "catalog.rateLimitExceeded", max_requests=RATE_LIMIT_MAX_REQUESTS),
)
_rate_limit_windows[user_email].append(now)
# --- Response Models ---
class ArtistResult(BaseModel):
"""Artist search result from MusicBrainz/Spotify catalog."""
name: str
mbid: Optional[str] = None
disambiguation: Optional[str] = None
artist_type: Optional[str] = None
spotify_id: Optional[str] = None
popularity: Optional[int] = None
genres: Optional[list[str]] = None
tags: Optional[list[str]] = None
class TrackResult(BaseModel):
"""Track search result from Spotify catalog."""
track_name: str
artist_name: str
track_id: Optional[str] = None
artist_id: Optional[str] = None
popularity: Optional[int] = None
duration_ms: Optional[int] = None
explicit: Optional[bool] = None
class CommunityTrack(BaseModel):
"""A community-approved karaoke track from karaokenerds."""
brand_name: str
brand_code: str
youtube_url: str
is_community: bool = True
class CommunityCheckSong(BaseModel):
"""A song with community karaoke tracks."""
title: str
artist: str
community_tracks: list[CommunityTrack]
class CommunityCheckRequest(BaseModel):
"""Request body for community version check."""
artist: str = Field(..., min_length=1)
title: str = Field(..., min_length=1)
class CommunityCheckResponse(BaseModel):
"""Response for community version check."""
has_community: bool
songs: list[CommunityCheckSong]
best_youtube_url: Optional[str] = None
# --- Routes ---
@router.get("/artists", response_model=list[ArtistResult])
async def search_artists(
request: Request,
q: str = Query(..., min_length=2, description="Artist name search query"),
limit: int = Query(10, ge=1, le=50, description="Maximum results"),
auth_result: AuthResult = Depends(require_auth),
):
"""
Search for artists by name. Returns canonical MusicBrainz/Spotify artist data
for autocomplete in job creation forms.
"""
locale = get_locale_from_request(request)
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
from backend.services.catalog_proxy_service import search_artists
results = await search_artists(q, limit)
return results
@router.get("/tracks", response_model=list[TrackResult])
async def search_tracks(
request: Request,
q: str = Query(..., min_length=2, description="Track name search query"),
artist: Optional[str] = Query(None, description="Filter by artist name"),
limit: int = Query(10, ge=1, le=50, description="Maximum results"),
auth_result: AuthResult = Depends(require_auth),
):
"""
Search for tracks by name, optionally filtered by artist.
Returns canonical Spotify track data for autocomplete in job creation forms.
"""
locale = get_locale_from_request(request)
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
from backend.services.catalog_proxy_service import search_tracks
results = await search_tracks(q, artist, limit)
return results
@router.post("/community-check", response_model=CommunityCheckResponse)
async def check_community_versions(
request: Request,
body: CommunityCheckRequest,
auth_result: AuthResult = Depends(require_auth),
):
"""
Check if a song already has community-approved karaoke versions on YouTube
via karaokenerds.com. Non-blocking — just informs the user.
"""
locale = get_locale_from_request(request)
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
from backend.services.karaokenerds_service import check_community_versions
result = await check_community_versions(body.artist, body.title)
return result
-65
View File
@@ -1,65 +0,0 @@
"""
Encoding worker lifecycle endpoints.
Provides warmup and heartbeat endpoints for the blue-green
encoding worker VMs. Called by the frontend to ensure the
primary VM is running before encoding requests.
"""
import logging
from fastapi import APIRouter, Depends
from backend.api.dependencies import require_admin
from backend.services.encoding_worker_manager import EncodingWorkerManager
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/internal/encoding-worker", tags=["encoding-worker"])
# Lazy-loaded singleton
_manager = None
def get_worker_manager():
global _manager
if _manager is None:
from google.cloud import compute_v1, firestore
from backend.config import get_settings
settings = get_settings()
db = firestore.Client(project=settings.google_cloud_project)
compute_client = compute_v1.InstancesClient()
_manager = EncodingWorkerManager(
db=db,
compute_client=compute_client,
project_id=settings.google_cloud_project,
)
return _manager
@router.post("/warmup")
async def warmup_encoding_worker(
_admin=Depends(require_admin),
manager=Depends(get_worker_manager),
):
"""Start the primary encoding worker VM if it's stopped."""
try:
result = manager.ensure_primary_running()
if result["started"]:
logger.info(f"Started encoding worker VM: {result['vm_name']}")
return result
except Exception as e:
logger.error(f"Failed to warm up encoding worker: {e}")
return {"started": False, "error": str(e)}
@router.post("/heartbeat")
async def heartbeat_encoding_worker(
_admin=Depends(require_admin),
manager=Depends(get_worker_manager),
):
"""Update activity timestamp to prevent idle shutdown."""
try:
manager.update_activity()
return {"status": "ok"}
except Exception as e:
logger.error(f"Failed to update encoding worker heartbeat: {e}")
return {"status": "error", "error": str(e)}
File diff suppressed because it is too large. Load diff
+3 -240
View File
@@ -5,12 +5,10 @@ import os
import logging
import subprocess
import platform
from fastapi import APIRouter, Depends
from typing import Dict, Any, Optional, Tuple
from fastapi import APIRouter
from typing import Dict, Any, Optional
from backend.version import VERSION, COMMIT_SHA, PR_NUMBER, PR_TITLE, STARTUP_TIME
from backend.api.dependencies import optional_auth
from backend.services.auth_service import UserType
from backend.version import VERSION
from backend.services.flacfetch_client import get_flacfetch_client
from backend.services.email_service import get_email_service
from backend.services.stripe_service import get_stripe_service
@@ -18,9 +16,6 @@ from backend.services.encoding_service import get_encoding_service
from backend.services.spacy_preloader import get_preloaded_model, is_model_preloaded
from backend.services.nltk_preloader import get_preloaded_cmudict, is_cmudict_preloaded
from backend.services.langfuse_preloader import get_preloaded_langfuse_handler, is_langfuse_preloaded, is_langfuse_configured
from backend.services.job_health_service import check_job_consistency_detailed
from backend.services.job_manager import JobManager
from backend.models.job import JobStatus
router = APIRouter()
logger = logging.getLogger(__name__)
@@ -310,175 +305,6 @@ async def flacfetch_health() -> Dict[str, Any]:
}
def check_audio_separator_status() -> Dict[str, Any]:
"""Check audio separator availability without calling the remote GPU service."""
from backend.config import get_settings
settings = get_settings()
if not settings.audio_separator_api_url:
return {
"available": False,
"status": "not_configured",
}
try:
from importlib.metadata import version
pkg_version = version("audio-separator")
except Exception:
pkg_version = None
return {
"available": True,
"status": "ok",
"version": pkg_version,
}
@router.get("/health/audio-separator")
async def audio_separator_health() -> Dict[str, Any]:
"""
Return the installed audio-separator package version.
Does NOT call the remote GPU service — that would cold-start an expensive
Cloud Run GPU instance (~$0.07 per wake). Instead reports the locally
installed package version, which matches what the audio worker uses.
No authentication required.
"""
from backend.config import get_settings
settings = get_settings()
if not settings.audio_separator_api_url:
return {
"available": False,
"status": "not_configured",
}
try:
from importlib.metadata import version
pkg_version = version("audio-separator")
except Exception:
pkg_version = None
return {
"available": True,
"status": "ok",
"version": pkg_version,
}
def _get_encoding_worker_manager():
"""Get the EncodingWorkerManager instance. Separated for testability."""
try:
encoding_service = get_encoding_service()
if hasattr(encoding_service, '_worker_manager') and encoding_service._worker_manager:
return encoding_service._worker_manager
except Exception:
pass
return None
@router.get("/health/system-status")
async def system_status(
auth: Optional[Tuple] = Depends(optional_auth),
) -> Dict[str, Any]:
"""
Aggregated system status for all services.
Returns basic info for everyone. Admin users get additional
blue-green deployment details for the encoder.
"""
is_admin = auth is not None and len(auth) >= 2 and auth[1] == UserType.ADMIN
# Fetch all health statuses concurrently
import asyncio
encoding_task = check_encoding_worker_status()
flacfetch_task = check_flacfetch_service_status()
encoding_status, flacfetch_status = await asyncio.gather(
encoding_task, flacfetch_task
)
separator_status = check_audio_separator_status()
# Build frontend service info (from env vars baked at build time — returned as-is)
frontend_svc = {
"status": "ok",
"version": VERSION, # Frontend and backend share the same version from pyproject.toml
}
# Build backend service info
backend_svc = {
"status": "ok",
"version": VERSION,
"deployed_at": STARTUP_TIME,
}
if COMMIT_SHA:
backend_svc["commit_sha"] = COMMIT_SHA
if PR_NUMBER:
backend_svc["pr_number"] = PR_NUMBER
if PR_TITLE:
backend_svc["pr_title"] = PR_TITLE
# Build encoder service info
encoder_svc = {
"status": "ok" if encoding_status.get("available") else "offline",
"version": encoding_status.get("wheel_version"),
"active_jobs": encoding_status.get("active_jobs", 0),
}
# Build flacfetch service info
flacfetch_svc = {
"status": "ok" if flacfetch_status.get("available") else "offline",
"version": flacfetch_status.get("version"),
}
# Build separator service info
separator_svc = {
"status": "ok" if separator_status.get("available") else "offline",
"version": separator_status.get("version"),
}
# Admin-only: blue-green encoder details from Firestore
if is_admin:
manager = _get_encoding_worker_manager()
if manager:
try:
config = manager.get_config()
encoder_svc["admin_details"] = {
"primary_vm": config.primary_vm,
"primary_ip": config.primary_ip,
"primary_version": config.primary_version,
"primary_deployed_at": config.primary_deployed_at,
"secondary_vm": config.secondary_vm,
"secondary_ip": config.secondary_ip,
"secondary_version": config.secondary_version,
"secondary_deployed_at": config.secondary_deployed_at,
"last_swap_at": config.last_swap_at,
"deploy_in_progress": config.deploy_in_progress,
"active_jobs": encoding_status.get("active_jobs", 0),
"queue_length": encoding_status.get("queue_length", 0),
}
except Exception as e:
logger.warning(f"Failed to get blue-green config: {e}")
# Add error details for offline services
if not flacfetch_status.get("available"):
flacfetch_svc["admin_details"] = {"error": flacfetch_status.get("error")}
if not separator_status.get("available"):
separator_svc["admin_details"] = {"error": separator_status.get("error")}
if not encoding_status.get("available") and "admin_details" not in encoder_svc:
encoder_svc["admin_details"] = {"error": encoding_status.get("error")}
return {
"services": {
"frontend": frontend_svc,
"backend": backend_svc,
"encoder": encoder_svc,
"flacfetch": flacfetch_svc,
"separator": separator_svc,
}
}
@router.get("/health/detailed")
async def detailed_health_check() -> Dict[str, Any]:
"""
@@ -604,69 +430,6 @@ async def preload_status() -> Dict[str, Any]:
}
@router.get("/health/job-consistency")
async def job_consistency_check(
status: Optional[str] = None,
limit: int = 50
) -> Dict[str, Any]:
"""
Check job consistency across the system.
This endpoint identifies jobs with state inconsistencies that may indicate
bugs in the state machine or trigger logic. Use this to find stuck jobs
or jobs in invalid states.
Args:
status: Optional filter by job status (e.g., "pending", "downloading")
limit: Maximum number of jobs to check (default 50)
Returns:
Dictionary with:
- total_checked: Number of jobs checked
- inconsistent_count: Number of jobs with issues
- inconsistent_jobs: List of job details with their issues
- summary: Summary of issue types found
"""
job_manager = JobManager()
# Query jobs to check
query_params = {}
if status:
try:
job_status = JobStatus(status)
query_params['status'] = job_status
except ValueError:
return {
"error": f"Invalid status: {status}",
"valid_statuses": [s.value for s in JobStatus]
}
# Get jobs from Firestore
jobs = job_manager.list_jobs(limit=limit, **query_params)
# Check each job for consistency
inconsistent_jobs = []
issue_summary = {}
for job in jobs:
result = check_job_consistency_detailed(job)
if not result['is_healthy']:
inconsistent_jobs.append(result)
# Count issue types
for issue in result['issues']:
issue_type = issue.split(':')[0] if ':' in issue else issue
issue_summary[issue_type] = issue_summary.get(issue_type, 0) + 1
return {
"total_checked": len(jobs),
"inconsistent_count": len(inconsistent_jobs),
"inconsistent_jobs": inconsistent_jobs,
"summary": issue_summary,
"status_filter": status,
"limit": limit,
}
@router.get("/readiness")
async def readiness_check() -> Dict[str, str]:
"""Readiness check endpoint for Cloud Run."""
+1 -207
View File
@@ -25,7 +25,7 @@ from backend.workers.screens_worker import generate_screens
from backend.workers.video_worker import generate_video
from backend.workers.render_video_worker import process_render_video
from backend.api.dependencies import require_admin
from backend.services.auth_service import AuthResult, UserType
from backend.services.auth_service import UserType
from backend.services.job_manager import JobManager
from backend.services.tracing import (
extract_trace_context,
@@ -76,18 +76,6 @@ def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerR
message=f"Job {job_id} not found"
)
# Reject workers for jobs in terminal states (failed, cancelled, complete)
# This prevents Cloud Tasks retries from triggering workers on dead jobs
terminal_statuses = {'failed', 'cancelled', 'complete', 'prep_complete', 'error'}
job_status = job.status.value if hasattr(job.status, 'value') else str(job.status)
if job_status in terminal_statuses:
logger.info(f"[job:{job_id}] Job in terminal state '{job_status}', skipping {worker_name} worker")
return WorkerResponse(
status="skipped",
job_id=job_id,
message=f"Job is in terminal state '{job_status}', {worker_name} worker not needed"
)
# Check worker-specific progress in state_data
progress_key = f"{worker_name}_progress"
worker_progress = job.state_data.get(progress_key, {})
@@ -114,31 +102,6 @@ def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerR
return None
@router.post("/sync-disposable-domains", response_model=WorkerResponse)
async def sync_disposable_domains_endpoint(
request: Request,
auth_result: AuthResult = Depends(require_admin),
):
"""Sync external disposable domain blocklist. Called by Cloud Scheduler daily."""
import asyncio
from backend.services.disposable_domain_sync_service import (
fetch_external_blocklist, sync_disposable_domains
)
from backend.services.firestore_service import get_firestore_client
from backend.services.email_validation_service import EmailValidationService
domains = await fetch_external_blocklist()
db = get_firestore_client()
result = await asyncio.to_thread(sync_disposable_domains, db, domains)
EmailValidationService._blocklist_cache = None
return WorkerResponse(
status="completed",
job_id="sync-disposable-domains",
message=f"Synced {result['external_count']} domains"
)
@router.post("/workers/audio", response_model=WorkerResponse)
async def trigger_audio_worker(
request: WorkerRequest,
@@ -465,175 +428,6 @@ async def check_idle_reminder(
return {"status": "error", "job_id": job_id, "message": str(e)}
@router.post("/youtube-queue/process")
async def process_youtube_upload_queue(
http_request: Request,
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Process queued YouTube uploads.
Called by Cloud Scheduler (hourly) to retry uploads that were deferred
due to YouTube API quota exhaustion. Can also be triggered manually
from the admin dashboard.
Processes queued uploads one at a time, stopping if quota is exhausted.
"""
from backend.workers.youtube_queue_processor import process_youtube_upload_queue as process_queue
trace_context = extract_trace_context(dict(http_request.headers))
logger.info("YOUTUBE_QUEUE_PROCESS starting")
add_span_attribute("operation", "youtube_queue_process")
# Run in background so the HTTP response returns quickly
# (Cloud Scheduler has a 30min timeout but we don't want to block)
async def _process():
try:
result = await process_queue()
logger.info(f"YOUTUBE_QUEUE_PROCESS complete: {result}")
except Exception as e:
logger.exception(f"YOUTUBE_QUEUE_PROCESS failed: {e}")
background_tasks.add_task(_process)
add_span_event("queue_processing_started")
return {
"status": "started",
"message": "YouTube upload queue processing started in background"
}
@router.post("/process-stale-reviews")
async def process_stale_reviews_endpoint(
http_request: Request,
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Process stale review jobs — send reminders and expire old ones.
Called by Cloud Scheduler (hourly). Can also be triggered manually
from the admin dashboard.
- Jobs in review for >= 24h get a reminder email
- Jobs in review for >= 48h are auto-cancelled with credit refund
"""
from backend.workers.stale_review_processor import process_stale_reviews
trace_context = extract_trace_context(dict(http_request.headers))
logger.info("STALE_REVIEW_PROCESS starting")
add_span_attribute("operation", "stale_review_process")
async def _process():
try:
result = await process_stale_reviews()
logger.info(f"STALE_REVIEW_PROCESS complete: {result}")
except Exception as e:
logger.exception(f"STALE_REVIEW_PROCESS failed: {e}")
background_tasks.add_task(_process)
add_span_event("stale_review_processing_started")
return {
"status": "started",
"message": "Stale review processing started in background"
}
@router.post("/trigger-gdrive-validation")
async def trigger_gdrive_validation_endpoint(
http_request: Request,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Trigger GDrive validation via the Cloud Function.
Called by a Cloud Tasks delayed task after a job completes and uploads
to the public GDrive share. The 5-minute delay ensures E2E test cleanup
finishes before the validator checks for sequence gaps.
"""
from backend.services.gdrive_validator_client import trigger_gdrive_validation
trace_context = extract_trace_context(dict(http_request.headers))
logger.info("GDRIVE_VALIDATION_TRIGGER starting (delayed post-job)")
add_span_attribute("operation", "gdrive_validation_trigger")
try:
result = trigger_gdrive_validation()
if result is None:
return {"status": "skipped", "message": "GDRIVE_VALIDATOR_URL not configured"}
status = result.get("status", "unknown")
add_span_event("validation_complete", {"result_status": status})
return {"status": status, "message": "GDrive validation completed", "result": result}
except Exception as e:
logger.exception(f"GDrive validation trigger failed: {e}")
add_span_event("error", {"error": str(e)})
return {"status": "error", "message": str(e)}
@router.post("/recover-stuck-jobs")
async def recover_stuck_jobs(
http_request: Request,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Detect and recover jobs stuck in downloading_audio status.
Called by Cloud Scheduler (every 5 minutes) or manually from admin.
Finds jobs stuck in DOWNLOADING_AUDIO for >10 minutes and fails them
so they can be retried.
"""
from backend.services.job_health_service import check_job_consistency
trace_context = extract_trace_context(dict(http_request.headers))
logger.info("RECOVER_STUCK_JOBS starting")
add_span_attribute("operation", "recover_stuck_jobs")
job_manager = JobManager()
# Query jobs in downloading_audio status
from google.cloud.firestore_v1 import FieldFilter
jobs_ref = job_manager.firestore.db.collection("jobs")
stuck_query = jobs_ref.where(
filter=FieldFilter("status", "==", "downloading_audio")
).stream()
recovered = []
for doc in stuck_query:
job_data = doc.to_dict()
job_id = job_data.get("job_id", doc.id)
# Use the health service to check if actually stuck
job = job_manager.get_job(job_id)
if not job:
continue
issues = check_job_consistency(job)
stuck_issues = [i for i in issues if "downloading_audio_stuck" in i]
if stuck_issues:
logger.warning(f"[job:{job_id}] Recovering stuck download: {stuck_issues[0]}")
job_manager.fail_job(
job_id,
"Audio download timed out (stuck >10 minutes). Use retry to re-attempt."
)
recovered.append(job_id)
logger.info(f"RECOVER_STUCK_JOBS complete: recovered={len(recovered)} jobs")
add_span_event("recovery_complete", {"recovered_count": len(recovered)})
return {
"status": "success",
"recovered_jobs": recovered,
"recovered_count": len(recovered),
}
@router.get("/health")
async def internal_health(
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
+99 -1317
View File
File diff suppressed because it is too large. Load diff
+18 -43
View File
@@ -11,14 +11,13 @@ Provides endpoints for managing Web Push notification subscriptions:
import logging
from typing import Optional, Dict, List
from fastapi import APIRouter, Depends, HTTPException, Request
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from backend.config import get_settings
from backend.api.dependencies import require_auth, require_admin
from backend.services.auth_service import AuthResult
from backend.services.push_notification_service import get_push_notification_service
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
@@ -38,7 +37,6 @@ class SubscribeRequest(BaseModel):
endpoint: str
keys: Dict[str, str] # p256dh and auth
device_name: Optional[str] = None
tenant_id: Optional[str] = None # Tenant scope for notification isolation
class SubscribeResponse(BaseModel):
@@ -113,8 +111,7 @@ async def get_vapid_public_key():
@router.post("/subscribe", response_model=SubscribeResponse)
async def subscribe_push(
subscribe_request: SubscribeRequest,
http_request: Request,
request: SubscribeRequest,
auth_result: AuthResult = Depends(require_auth)
):
"""
@@ -122,53 +119,39 @@ async def subscribe_push(
Requires authentication. Users can have up to 5 subscriptions
(configurable via MAX_PUSH_SUBSCRIPTIONS_PER_USER).
Subscriptions are scoped by tenant_id to prevent cross-tenant
notification leakage. The tenant is resolved from:
1. Request body tenant_id field (explicit from frontend)
2. Tenant middleware (X-Tenant-ID header or subdomain detection)
"""
locale = get_locale_from_request(http_request)
settings = get_settings()
if not settings.enable_push_notifications:
raise HTTPException(status_code=503, detail=t(locale, "push.disabled"))
raise HTTPException(status_code=503, detail="Push notifications are not enabled")
if not auth_result.user_email:
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
raise HTTPException(status_code=401, detail="User email not available")
push_service = get_push_notification_service()
# Validate keys
if "p256dh" not in subscribe_request.keys or "auth" not in subscribe_request.keys:
raise HTTPException(status_code=400, detail=t(locale, "push.missingEncryptionKeys"))
# Resolve tenant_id: prefer explicit from body, fall back to middleware
tenant_id = subscribe_request.tenant_id
if tenant_id is None:
tenant_id = getattr(http_request.state, "tenant_id", None)
if "p256dh" not in request.keys or "auth" not in request.keys:
raise HTTPException(status_code=400, detail="Missing required keys (p256dh, auth)")
success = await push_service.add_subscription(
user_email=auth_result.user_email,
endpoint=subscribe_request.endpoint,
keys=subscribe_request.keys,
device_name=subscribe_request.device_name,
tenant_id=tenant_id or None,
endpoint=request.endpoint,
keys=request.keys,
device_name=request.device_name
)
if not success:
raise HTTPException(status_code=500, detail=t(locale, "push.subscriptionSaveFailed"))
raise HTTPException(status_code=500, detail="Failed to save subscription")
return SubscribeResponse(
status="success",
message=t(locale, "push.subscriptionRegistered")
message="Push subscription registered successfully"
)
@router.post("/unsubscribe", response_model=UnsubscribeResponse)
async def unsubscribe_push(
request: UnsubscribeRequest,
http_request: Request,
auth_result: AuthResult = Depends(require_auth)
):
"""
@@ -176,10 +159,8 @@ async def unsubscribe_push(
Requires authentication. Users can only remove their own subscriptions.
"""
locale = get_locale_from_request(http_request)
if not auth_result.user_email:
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
raise HTTPException(status_code=401, detail="User email not available")
push_service = get_push_notification_service()
@@ -192,18 +173,17 @@ async def unsubscribe_push(
# Don't error if subscription wasn't found - might already be removed
return UnsubscribeResponse(
status="success",
message=t(locale, "push.subscriptionNotFound")
message="Subscription removed (or was not found)"
)
return UnsubscribeResponse(
status="success",
message=t(locale, "push.subscriptionRemoved")
message="Push subscription removed successfully"
)
@router.get("/subscriptions", response_model=SubscriptionsListResponse)
async def list_subscriptions(
request: Request,
auth_result: AuthResult = Depends(require_auth)
):
"""
@@ -211,10 +191,8 @@ async def list_subscriptions(
Requires authentication.
"""
locale = get_locale_from_request(request)
if not auth_result.user_email:
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
raise HTTPException(status_code=401, detail="User email not available")
push_service = get_push_notification_service()
@@ -229,7 +207,6 @@ async def list_subscriptions(
@router.post("/test", response_model=TestNotificationResponse)
async def send_test_notification(
request: TestNotificationRequest,
http_request: Request,
auth_result: AuthResult = Depends(require_admin)
):
"""
@@ -237,14 +214,12 @@ async def send_test_notification(
Admin only. Useful for testing push notification setup.
"""
locale = get_locale_from_request(http_request)
settings = get_settings()
if not settings.enable_push_notifications:
raise HTTPException(status_code=503, detail=t(locale, "push.disabled"))
raise HTTPException(status_code=503, detail="Push notifications are not enabled")
if not auth_result.user_email:
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
raise HTTPException(status_code=401, detail="User email not available")
push_service = get_push_notification_service()
@@ -259,5 +234,5 @@ async def send_test_notification(
return TestNotificationResponse(
status="success",
sent_count=sent_count,
message=t(locale, "push.testNotificationSent", sent_count=sent_count)
message=f"Test notification sent to {sent_count} device(s)"
)
+242 -160
View File
@@ -1,22 +1,24 @@
"""
Admin API routes for blocklist and YouTube queue management.
Admin API routes for rate limit management.
Handles:
- Rate limit statistics and monitoring
- Blocklist management (disposable domains, blocked emails, blocked IPs)
- YouTube upload queue management
- User override management (whitelist/bypass permissions)
"""
import logging
from datetime import datetime
from datetime import datetime, timezone
from typing import List, Optional
from fastapi import APIRouter, Depends, HTTPException, Request
from pydantic import BaseModel
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, EmailStr
from backend.api.dependencies import require_admin
from backend.services.auth_service import AuthResult
from backend.services.rate_limit_service import get_rate_limit_service, RateLimitService
from backend.services.email_validation_service import get_email_validation_service, EmailValidationService
from backend.i18n import t, get_locale_from_request
from backend.config import settings
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/admin/rate-limits", tags=["admin", "rate-limits"])
@@ -26,40 +28,45 @@ router = APIRouter(prefix="/admin/rate-limits", tags=["admin", "rate-limits"])
# Response Models
# =============================================================================
class YouTubeQueueEntry(BaseModel):
"""A queued YouTube upload entry."""
job_id: str
status: str
reason: Optional[str] = None
user_email: Optional[str] = None
artist: Optional[str] = None
title: Optional[str] = None
brand_code: Optional[str] = None
queued_at: Optional[datetime] = None
attempts: int = 0
max_attempts: int = 5
last_error: Optional[str] = None
youtube_url: Optional[str] = None
notification_sent: bool = False
class RateLimitStatsResponse(BaseModel):
"""Current rate limit statistics."""
# Configuration
jobs_per_day_limit: int
youtube_uploads_per_day_limit: int
beta_ip_per_day_limit: int
rate_limiting_enabled: bool
# Current usage
youtube_uploads_today: int
youtube_uploads_remaining: int
# Blocklist stats
disposable_domains_count: int
blocked_emails_count: int
blocked_ips_count: int
# User override stats
total_overrides: int
class YouTubeQueueListResponse(BaseModel):
"""List of YouTube upload queue entries."""
entries: List[YouTubeQueueEntry]
stats: dict
class UserRateLimitStatusResponse(BaseModel):
"""Rate limit status for a specific user."""
email: str
jobs_today: int
jobs_limit: int
jobs_remaining: int
has_bypass: bool
custom_limit: Optional[int]
bypass_reason: Optional[str]
class BlocklistsResponse(BaseModel):
"""All blocklist data with domain source separation."""
external_domains: List[str]
manual_domains: List[str]
allowlisted_domains: List[str]
"""All blocklist data."""
disposable_domains: List[str]
blocked_emails: List[str]
blocked_ips: List[str]
last_sync_at: Optional[datetime] = None
last_sync_count: Optional[int] = None
updated_at: Optional[datetime] = None
updated_by: Optional[str] = None
updated_at: Optional[datetime]
updated_by: Optional[str]
class DomainRequest(BaseModel):
@@ -77,12 +84,115 @@ class IPRequest(BaseModel):
ip_address: str
class UserOverride(BaseModel):
"""User override configuration."""
email: str
bypass_job_limit: bool
custom_daily_job_limit: Optional[int]
reason: str
created_by: str
created_at: datetime
class UserOverrideRequest(BaseModel):
"""Request to set/update a user override."""
bypass_job_limit: bool = False
custom_daily_job_limit: Optional[int] = None
reason: str
class UserOverridesListResponse(BaseModel):
"""List of all user overrides."""
overrides: List[UserOverride]
total: int
class SuccessResponse(BaseModel):
"""Generic success response."""
success: bool
message: str
# =============================================================================
# Statistics Endpoints
# =============================================================================
@router.get("/stats", response_model=RateLimitStatsResponse)
async def get_rate_limit_stats(
auth_result: AuthResult = Depends(require_admin),
):
"""
Get current rate limit statistics.
Returns configuration values, current usage counts, and blocklist stats.
"""
rate_limit_service = get_rate_limit_service()
email_validation = get_email_validation_service()
# Get YouTube uploads today
youtube_today = rate_limit_service.get_youtube_uploads_today()
youtube_limit = settings.rate_limit_youtube_uploads_per_day
youtube_remaining = max(0, youtube_limit - youtube_today) if youtube_limit > 0 else -1
# Get blocklist stats
blocklist_stats = email_validation.get_blocklist_stats()
# Get override count
overrides = rate_limit_service.get_all_overrides()
return RateLimitStatsResponse(
jobs_per_day_limit=settings.rate_limit_jobs_per_day,
youtube_uploads_per_day_limit=settings.rate_limit_youtube_uploads_per_day,
beta_ip_per_day_limit=settings.rate_limit_beta_ip_per_day,
rate_limiting_enabled=settings.enable_rate_limiting,
youtube_uploads_today=youtube_today,
youtube_uploads_remaining=youtube_remaining,
disposable_domains_count=blocklist_stats["disposable_domains_count"],
blocked_emails_count=blocklist_stats["blocked_emails_count"],
blocked_ips_count=blocklist_stats["blocked_ips_count"],
total_overrides=len(overrides),
)
@router.get("/users/{email}", response_model=UserRateLimitStatusResponse)
async def get_user_rate_limit_status(
email: str,
auth_result: AuthResult = Depends(require_admin),
):
"""
Get rate limit status for a specific user.
Returns their current usage and any override settings.
"""
rate_limit_service = get_rate_limit_service()
jobs_today = rate_limit_service.get_user_job_count_today(email)
override = rate_limit_service.get_user_override(email)
if override:
has_bypass = override.get("bypass_job_limit", False)
custom_limit = override.get("custom_daily_job_limit")
bypass_reason = override.get("reason")
jobs_limit = custom_limit if custom_limit is not None else settings.rate_limit_jobs_per_day
else:
has_bypass = False
custom_limit = None
bypass_reason = None
jobs_limit = settings.rate_limit_jobs_per_day
jobs_remaining = max(0, jobs_limit - jobs_today) if not has_bypass else -1
return UserRateLimitStatusResponse(
email=email,
jobs_today=jobs_today,
jobs_limit=jobs_limit,
jobs_remaining=jobs_remaining,
has_bypass=has_bypass,
custom_limit=custom_limit,
bypass_reason=bypass_reason,
)
# =============================================================================
# Blocklist Management Endpoints
# =============================================================================
@@ -91,10 +201,38 @@ class SuccessResponse(BaseModel):
async def get_blocklists(
auth_result: AuthResult = Depends(require_admin),
):
"""Get all blocklist data with domain source separation."""
"""
Get all blocklist data.
Returns disposable domains, blocked emails, and blocked IPs.
"""
email_validation = get_email_validation_service()
raw_data = email_validation.get_blocklist_raw_data()
return BlocklistsResponse(**raw_data)
# Force refresh to get latest data
config = email_validation.get_blocklist_config(force_refresh=True)
# Get the raw document for metadata
from google.cloud import firestore
from backend.services.firestore_service import get_firestore_client
from backend.services.email_validation_service import BLOCKLISTS_COLLECTION, BLOCKLIST_CONFIG_DOC
db = get_firestore_client()
doc = db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC).get()
updated_at = None
updated_by = None
if doc.exists:
data = doc.to_dict()
updated_at = data.get("updated_at")
updated_by = data.get("updated_by")
return BlocklistsResponse(
disposable_domains=sorted(list(config["disposable_domains"])),
blocked_emails=sorted(list(config["blocked_emails"])),
blocked_ips=sorted(list(config["blocked_ips"])),
updated_at=updated_at,
updated_by=updated_by,
)
@router.post("/blocklists/disposable-domains", response_model=SuccessResponse)
@@ -107,13 +245,13 @@ async def add_disposable_domain(
domain = request.domain.lower().strip()
if not domain or "." not in domain:
raise HTTPException(status_code=400, detail=t("en", "admin.invalidDomainFormat"))
raise HTTPException(status_code=400, detail="Invalid domain format")
email_validation.add_disposable_domain(domain, auth_result.user_email)
return SuccessResponse(
success=True,
message=t("en", "admin.domainAddedToBl", domain=domain)
message=f"Domain '{domain}' added to disposable domains blocklist"
)
@@ -127,62 +265,11 @@ async def remove_disposable_domain(
domain = domain.lower().strip()
if not email_validation.remove_disposable_domain(domain, auth_result.user_email):
raise HTTPException(status_code=404, detail=t("en", "admin.domainNotFoundInBl"))
raise HTTPException(status_code=404, detail="Domain not found in blocklist")
return SuccessResponse(
success=True,
message=t("en", "admin.domainRemovedFromBl", domain=domain)
)
@router.post("/blocklists/allowlisted-domains", response_model=SuccessResponse)
async def add_allowlisted_domain(
request: DomainRequest,
auth_result: AuthResult = Depends(require_admin),
):
"""Add a domain to the allowlist (overrides external blocklist)."""
email_validation = get_email_validation_service()
domain = request.domain.lower().strip()
if not domain or "." not in domain:
raise HTTPException(status_code=400, detail=t("en", "admin.invalidDomainFormat"))
email_validation.add_allowlisted_domain(domain, auth_result.user_email)
return SuccessResponse(success=True, message=t("en", "admin.domainAddedToAllowlist", domain=domain))
@router.delete("/blocklists/allowlisted-domains/{domain}", response_model=SuccessResponse)
async def remove_allowlisted_domain(
domain: str,
auth_result: AuthResult = Depends(require_admin),
):
"""Remove a domain from the allowlist."""
email_validation = get_email_validation_service()
domain = domain.lower().strip()
if not email_validation.remove_allowlisted_domain(domain, auth_result.user_email):
raise HTTPException(status_code=404, detail=t("en", "admin.domainNotFoundInAllowlist"))
return SuccessResponse(success=True, message=t("en", "admin.domainRemovedFromAllowlist", domain=domain))
@router.post("/blocklists/sync", response_model=SuccessResponse)
async def trigger_sync(
auth_result: AuthResult = Depends(require_admin),
):
"""Manually trigger a sync of the external disposable domain blocklist."""
import asyncio
from backend.services.disposable_domain_sync_service import (
fetch_external_blocklist, sync_disposable_domains
)
from backend.services.firestore_service import get_firestore_client
domains = await fetch_external_blocklist()
db = get_firestore_client()
result = await asyncio.to_thread(sync_disposable_domains, db, domains)
# Invalidate cache
EmailValidationService._blocklist_cache = None
return SuccessResponse(
success=True,
message=t("en", "admin.syncComplete", external_count=result['external_count'], added=result['added'], removed=result['removed'])
message=f"Domain '{domain}' removed from disposable domains blocklist"
)
@@ -196,13 +283,13 @@ async def add_blocked_email(
email = request.email.lower().strip()
if not email or "@" not in email:
raise HTTPException(status_code=400, detail=t("en", "admin.invalidEmailFormat"))
raise HTTPException(status_code=400, detail="Invalid email format")
email_validation.add_blocked_email(email, auth_result.user_email)
return SuccessResponse(
success=True,
message=t("en", "admin.emailAddedToBl", email=email)
message=f"Email '{email}' added to blocked emails list"
)
@@ -216,11 +303,11 @@ async def remove_blocked_email(
email = email.lower().strip()
if not email_validation.remove_blocked_email(email, auth_result.user_email):
raise HTTPException(status_code=404, detail=t("en", "admin.emailNotFoundInBl"))
raise HTTPException(status_code=404, detail="Email not found in blocklist")
return SuccessResponse(
success=True,
message=t("en", "admin.emailRemovedFromBl", email=email)
message=f"Email '{email}' removed from blocked emails list"
)
@@ -234,13 +321,13 @@ async def add_blocked_ip(
ip_address = request.ip_address.strip()
if not ip_address:
raise HTTPException(status_code=400, detail=t("en", "admin.invalidIPAddress"))
raise HTTPException(status_code=400, detail="Invalid IP address")
email_validation.add_blocked_ip(ip_address, auth_result.user_email)
return SuccessResponse(
success=True,
message=t("en", "admin.ipAddedToBl", ip_address=ip_address)
message=f"IP '{ip_address}' added to blocked IPs list"
)
@@ -254,93 +341,88 @@ async def remove_blocked_ip(
ip_address = ip_address.strip()
if not email_validation.remove_blocked_ip(ip_address, auth_result.user_email):
raise HTTPException(status_code=404, detail=t("en", "admin.ipNotFoundInBl"))
raise HTTPException(status_code=404, detail="IP not found in blocklist")
return SuccessResponse(
success=True,
message=t("en", "admin.ipRemovedFromBl", ip_address=ip_address)
message=f"IP '{ip_address}' removed from blocked IPs list"
)
# =============================================================================
# YouTube Upload Queue Management Endpoints
# User Override Management Endpoints
# =============================================================================
@router.get("/youtube-queue", response_model=YouTubeQueueListResponse)
async def get_youtube_queue(
@router.get("/overrides", response_model=UserOverridesListResponse)
async def get_all_overrides(
auth_result: AuthResult = Depends(require_admin),
):
"""
Get YouTube upload queue entries and stats.
"""Get all user overrides."""
rate_limit_service = get_rate_limit_service()
Returns all queue entries (any status) and aggregate statistics.
"""
from backend.services.youtube_upload_queue_service import get_youtube_upload_queue_service
overrides_data = rate_limit_service.get_all_overrides()
queue_service = get_youtube_upload_queue_service()
entries = queue_service.get_all_queue_entries(limit=50)
stats = queue_service.get_queue_stats()
return YouTubeQueueListResponse(
entries=[YouTubeQueueEntry(**entry) for entry in entries],
stats=stats,
)
@router.post("/youtube-queue/{job_id}/retry", response_model=SuccessResponse)
async def retry_youtube_upload(
job_id: str,
auth_result: AuthResult = Depends(require_admin),
):
"""
Manually retry a failed YouTube upload.
Resets the entry back to 'queued' status with 0 attempts.
"""
from backend.services.youtube_upload_queue_service import get_youtube_upload_queue_service
queue_service = get_youtube_upload_queue_service()
if not queue_service.retry_upload(job_id):
raise HTTPException(
status_code=404,
detail=t("en", "admin.youtubeUploadNotFound", job_id=job_id)
overrides = [
UserOverride(
email=email,
bypass_job_limit=data.get("bypass_job_limit", False),
custom_daily_job_limit=data.get("custom_daily_job_limit"),
reason=data.get("reason", ""),
created_by=data.get("created_by", "unknown"),
created_at=data.get("created_at", datetime.now(timezone.utc)),
)
for email, data in overrides_data.items()
]
return SuccessResponse(
success=True,
message=t("en", "admin.youtubeUploadQueuedForRetry", job_id=job_id)
return UserOverridesListResponse(
overrides=overrides,
total=len(overrides),
)
@router.post("/youtube-queue/process", response_model=SuccessResponse)
async def trigger_youtube_queue_processing(
@router.put("/overrides/{email}", response_model=SuccessResponse)
async def set_user_override(
email: str,
request: UserOverrideRequest,
auth_result: AuthResult = Depends(require_admin),
):
"""
Manually trigger YouTube upload queue processing.
"""Set or update a user override."""
rate_limit_service = get_rate_limit_service()
Same as the scheduled processor but triggered from the admin dashboard.
Runs in the background and returns immediately.
"""
from backend.workers.youtube_queue_processor import process_youtube_upload_queue
import asyncio
email = email.lower().strip()
if not email or "@" not in email:
raise HTTPException(status_code=400, detail="Invalid email format")
async def _process():
try:
result = await process_youtube_upload_queue()
logger.info(f"Manual YouTube queue processing complete: {result}")
except Exception as e:
logger.exception(f"Manual YouTube queue processing failed: {e}")
if not request.reason or len(request.reason.strip()) < 3:
raise HTTPException(status_code=400, detail="Reason is required (min 3 characters)")
# Fire and forget
try:
loop = asyncio.get_running_loop()
loop.create_task(_process())
except RuntimeError:
pass
rate_limit_service.set_user_override(
user_email=email,
bypass_job_limit=request.bypass_job_limit,
custom_daily_job_limit=request.custom_daily_job_limit,
reason=request.reason,
admin_email=auth_result.user_email,
)
return SuccessResponse(
success=True,
message=t("en", "admin.youtubeQueueProcessingStarted")
message=f"Override set for user '{email}'"
)
@router.delete("/overrides/{email}", response_model=SuccessResponse)
async def remove_user_override(
email: str,
auth_result: AuthResult = Depends(require_admin),
):
"""Remove a user override."""
rate_limit_service = get_rate_limit_service()
email = email.lower().strip()
if not rate_limit_service.remove_user_override(email, auth_result.user_email):
raise HTTPException(status_code=404, detail="Override not found for user")
return SuccessResponse(
success=True,
message=f"Override removed for user '{email}'"
)
-374
View File
@@ -1,374 +0,0 @@
"""
Referral API routes.
Handles:
- Public referral interstitial (click tracking)
- Authenticated referral dashboard
- Stripe Connect onboarding
- Admin vanity link creation and link management
"""
import logging
from typing import Optional
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi.responses import Response
from pydantic import BaseModel, Field
from backend.api.dependencies import require_auth, require_admin
from backend.models.referral import (
CreateReferralLinkRequest,
UpdateReferralLinkRequest,
ReferralInterstitialResponse,
)
from backend.services.flyer_service import get_flyer_service, FlyerError
from backend.services.referral_service import get_referral_service
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/referrals", tags=["referrals"])
class GenerateFlyerRequest(BaseModel):
theme: str = Field(..., pattern="^(light|dark)$")
qr_data_url: str = Field(..., max_length=500_000)
# ============================================================================
# Public endpoints
# ============================================================================
@router.get("/r/{code}", response_model=ReferralInterstitialResponse)
async def get_referral_interstitial(code: str, no_track: bool = False):
"""
Public endpoint: get interstitial data for a referral link.
Increments click count unless no_track=true. Returns valid=False if code not found.
"""
service = get_referral_service()
link = service.get_link_by_code(code)
if not link:
return ReferralInterstitialResponse(valid=False)
# Track the click (skip when fetching for display purposes)
if not no_track:
try:
service.increment_clicks(code)
except Exception:
logger.warning("Failed to increment clicks for code %s", code, exc_info=True)
return ReferralInterstitialResponse(
valid=True,
referral_code=link.code,
referrer_display_name=link.display_name,
discount_percent=link.discount_percent,
discount_duration_days=link.discount_duration_days,
custom_message=link.custom_message,
)
# ============================================================================
# Authenticated user endpoints
# ============================================================================
@router.get("/me")
async def get_my_dashboard(auth=Depends(require_auth)):
"""Get referral dashboard data for the authenticated user."""
service = get_referral_service()
return service.get_dashboard_data(auth.user_email)
@router.put("/me")
async def update_my_link(
updates: UpdateReferralLinkRequest,
auth=Depends(require_auth),
):
"""Update display_name and custom_message on the user's own referral link."""
service = get_referral_service()
link = service.get_or_create_link(auth.user_email)
update_fields = {}
if updates.display_name is not None:
update_fields["display_name"] = updates.display_name
if updates.custom_message is not None:
update_fields["custom_message"] = updates.custom_message
if not update_fields:
return {"ok": True, "message": "No changes"}
success, message = service.update_link(link.code, **update_fields)
if not success:
raise HTTPException(status_code=400, detail=message)
return {"ok": True, "message": message}
@router.post("/me/connect")
async def start_stripe_connect(auth=Depends(require_auth)):
"""Start Stripe Connect onboarding for the authenticated user."""
# Check if user already has a Connect account
from backend.services.user_service import get_user_service
user_service = get_user_service()
user = user_service.get_user(auth.user_email)
if user and user.stripe_connect_account_id:
raise HTTPException(status_code=400, detail="Stripe Connect account already configured")
service = get_referral_service()
account_id, onboarding_url = service.create_connect_account(auth.user_email)
if not account_id:
raise HTTPException(status_code=500, detail="Failed to create Stripe Connect account")
user_service.update_user(auth.user_email, stripe_connect_account_id=account_id)
return {"account_id": account_id, "onboarding_url": onboarding_url}
@router.post("/me/connect/dashboard-link")
async def get_connect_dashboard_link(auth=Depends(require_auth)):
"""Get a login link to the Stripe Express dashboard for managing payout settings."""
from backend.services.user_service import get_user_service
user = get_user_service().get_user(auth.user_email)
if not user or not user.stripe_connect_account_id:
raise HTTPException(status_code=400, detail="No Stripe Connect account configured")
service = get_referral_service()
url = service.create_connect_login_link(user.stripe_connect_account_id)
if not url:
raise HTTPException(status_code=500, detail="Failed to create dashboard link")
return {"url": url}
@router.post("/me/connect/update-link")
async def get_connect_update_link(auth=Depends(require_auth)):
"""Get a link to update Stripe Connect account info (re-onboarding)."""
from backend.services.user_service import get_user_service
user = get_user_service().get_user(auth.user_email)
if not user or not user.stripe_connect_account_id:
raise HTTPException(status_code=400, detail="No Stripe Connect account configured")
service = get_referral_service()
url = service.create_connect_update_link(user.stripe_connect_account_id)
if not url:
raise HTTPException(status_code=500, detail="Failed to create update link")
return {"url": url}
class VanityRequest(BaseModel):
desired_code: str = Field(..., min_length=2, max_length=30, pattern="^[a-z0-9-]+$")
@router.post("/me/vanity-request")
async def request_vanity_url(
request: VanityRequest,
auth=Depends(require_auth),
):
"""Request a custom vanity referral URL. Sends email to admin for review."""
from backend.services.email_service import get_email_service
service = get_referral_service()
link = service.get_or_create_link(auth.user_email)
# Check if code is already taken
existing = service.get_link_by_code(request.desired_code)
if existing:
raise HTTPException(status_code=409, detail="That code is already taken. Please try another.")
email_service = get_email_service()
email_service.provider.send_email(
to_email="andrew@nomadkaraoke.com",
subject=f"Vanity referral URL request: {request.desired_code}",
html_content=f"""
<h2>Vanity Referral URL Request</h2>
<p><strong>User:</strong> {auth.user_email}</p>
<p><strong>Current code:</strong> {link.code}</p>
<p><strong>Desired vanity code:</strong> {request.desired_code}</p>
<p>To approve, create the vanity link in the <a href="https://gen.nomadkaraoke.com/admin/referrals">admin dashboard</a>.</p>
""",
text_content=f"Vanity URL request from {auth.user_email}: {request.desired_code} (current: {link.code})",
)
logger.info("Vanity URL request from %s: %s", auth.user_email, request.desired_code)
return {"ok": True, "message": "Request sent"}
@router.post("/me/flyer")
async def generate_flyer(
request: GenerateFlyerRequest,
auth=Depends(require_auth),
):
"""Generate a personalized referral flyer PDF."""
service = get_referral_service()
link = service.get_or_create_link(auth.user_email)
try:
flyer_service = get_flyer_service()
pdf_bytes = flyer_service.generate_pdf(
theme=request.theme,
referral_code=link.code,
discount_percent=link.discount_percent,
qr_data_url=request.qr_data_url,
)
except FlyerError as e:
raise HTTPException(status_code=400, detail=str(e))
return Response(
content=pdf_bytes,
media_type="application/pdf",
headers={
"Content-Disposition": 'attachment; filename="nomad-karaoke-referral-flyer.pdf"',
},
)
# ============================================================================
# Admin endpoints
# ============================================================================
@router.post("/admin/vanity")
async def create_vanity_link(
request: CreateReferralLinkRequest,
auth=Depends(require_admin),
):
"""Admin: create a vanity referral link."""
if not request.vanity_code:
raise HTTPException(status_code=400, detail="vanity_code is required")
service = get_referral_service()
success, link, message = service.create_vanity_link(
code=request.vanity_code,
owner_email=request.owner_email,
display_name=request.display_name,
custom_message=request.custom_message,
discount_percent=request.discount_percent,
kickback_percent=request.kickback_percent,
discount_duration_days=request.discount_duration_days,
earning_duration_days=request.earning_duration_days,
)
if not success:
raise HTTPException(status_code=400, detail=message)
return {"ok": True, "code": link.code, "message": message}
@router.get("/admin/links")
async def list_links(
limit: int = Query(50, ge=1, le=200),
offset: int = Query(0, ge=0),
auth=Depends(require_admin),
):
"""Admin: list all referral links."""
service = get_referral_service()
links = service.list_links(limit=limit, offset=offset)
return {
"links": [link.model_dump(mode="json") for link in links],
"count": len(links),
}
@router.post("/admin/apply-discount")
async def admin_apply_discount(
request: Request,
auth=Depends(require_admin),
):
"""Admin: apply a referral discount to any user, even without a referral link."""
from backend.services.user_service import get_user_service
body = await request.json()
user_email = body.get("email", "").lower()
discount_percent = body.get("discount_percent", 10)
duration_days = body.get("duration_days", 30)
referral_code = body.get("referral_code", "admin-grant")
if not user_email:
raise HTTPException(status_code=400, detail="Email is required")
user_service = get_user_service()
user = user_service.get_user(user_email)
if not user:
raise HTTPException(status_code=404, detail="User not found")
from datetime import datetime, timedelta
now = datetime.utcnow()
user_service.update_user(
user_email,
referred_by_code=referral_code,
referred_at=now,
referral_discount_expires_at=now + timedelta(days=duration_days),
)
# Create/ensure the Stripe coupon exists for this discount percent
service = get_referral_service()
service.get_or_create_stripe_coupon(discount_percent)
logger.info(f"Admin applied {discount_percent}% discount for {duration_days} days to {user_email}")
return {
"ok": True,
"message": f"Applied {discount_percent}% discount for {duration_days} days",
"discount_expires_at": (now + timedelta(days=duration_days)).isoformat(),
}
@router.put("/admin/links/{code}")
async def update_link(
code: str,
updates: UpdateReferralLinkRequest,
auth=Depends(require_admin),
):
"""Admin: update any referral link's settings."""
service = get_referral_service()
update_fields = {
k: v for k, v in updates.model_dump().items() if v is not None
}
if not update_fields:
return {"ok": True, "message": "No changes"}
success, message = service.update_link(code, **update_fields)
if not success:
raise HTTPException(status_code=404, detail=message)
return {"ok": True, "message": message}
@router.post("/admin/links/{code}/flyer")
async def admin_generate_flyer(
code: str,
request: GenerateFlyerRequest,
auth=Depends(require_admin),
):
"""Admin: generate a personalized referral flyer PDF for any code."""
service = get_referral_service()
link = service.get_link_by_code(code)
if not link:
raise HTTPException(status_code=404, detail=f"Referral link '{code}' not found")
try:
flyer_service = get_flyer_service()
pdf_bytes = flyer_service.generate_pdf(
theme=request.theme,
referral_code=link.code,
discount_percent=link.discount_percent,
qr_data_url=request.qr_data_url,
)
except FlyerError as e:
raise HTTPException(status_code=400, detail=str(e))
return Response(
content=pdf_bytes,
media_type="application/pdf",
headers={
"Content-Disposition": f'attachment; filename="nomad-karaoke-flyer-{link.code}.pdf"',
},
)
+100 -1174
View File
File diff suppressed because it is too large. Load diff
+1 -2
View File
@@ -12,7 +12,6 @@ from fastapi import APIRouter, Query, Request
from backend.models.tenant import TenantConfigResponse, TenantPublicConfig
from backend.services.tenant_service import get_tenant_service
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
@@ -116,6 +115,6 @@ async def get_tenant_asset(tenant_id: str, asset_name: str):
if not url:
from fastapi import HTTPException
raise HTTPException(status_code=404, detail=t("en", "tenant.assetNotFound"))
raise HTTPException(status_code=404, detail="Asset not found")
return RedirectResponse(url=url)
+13 -18
View File
@@ -9,11 +9,10 @@ before job creation.
import logging
from typing import Optional
from fastapi import APIRouter, HTTPException, Request
from fastapi import APIRouter, HTTPException
from backend.models.theme import ThemeDetailResponse, ThemesListResponse
from backend.services.theme_service import get_theme_service
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
@@ -21,7 +20,7 @@ router = APIRouter(prefix="/themes", tags=["themes"])
@router.get("", response_model=ThemesListResponse)
async def list_themes(request: Request) -> ThemesListResponse:
async def list_themes() -> ThemesListResponse:
"""
List all available themes.
@@ -30,7 +29,6 @@ async def list_themes(request: Request) -> ThemesListResponse:
This endpoint is public and does not require authentication.
"""
locale = get_locale_from_request(request)
try:
theme_service = get_theme_service()
themes = theme_service.list_themes()
@@ -39,12 +37,12 @@ async def list_themes(request: Request) -> ThemesListResponse:
logger.error(f"Error listing themes: {e}")
raise HTTPException(
status_code=500,
detail=t(locale, "themes.themesLoadError")
detail="Failed to load themes"
)
@router.get("/{theme_id}", response_model=ThemeDetailResponse)
async def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
async def get_theme(theme_id: str) -> ThemeDetailResponse:
"""
Get detailed information about a specific theme.
@@ -60,7 +58,6 @@ async def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
Raises:
404: Theme not found
"""
locale = get_locale_from_request(request)
try:
theme_service = get_theme_service()
theme = theme_service.get_theme(theme_id)
@@ -68,7 +65,7 @@ async def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
if theme is None:
raise HTTPException(
status_code=404,
detail=t(locale, "themes.themeNotFound", theme_id=theme_id)
detail=f"Theme not found: {theme_id}"
)
return ThemeDetailResponse(theme=theme)
@@ -78,12 +75,12 @@ async def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
logger.error(f"Error getting theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail=t(locale, "themes.themeLoadError")
detail="Failed to load theme"
)
@router.get("/{theme_id}/preview")
async def get_theme_preview(theme_id: str, request: Request) -> dict:
async def get_theme_preview(theme_id: str) -> dict:
"""
Get a signed URL for the theme's preview image.
@@ -99,21 +96,20 @@ async def get_theme_preview(theme_id: str, request: Request) -> dict:
Raises:
404: Theme not found or no preview available
"""
locale = get_locale_from_request(request)
try:
theme_service = get_theme_service()
if not theme_service.theme_exists(theme_id):
raise HTTPException(
status_code=404,
detail=t(locale, "themes.themeNotFound", theme_id=theme_id)
detail=f"Theme not found: {theme_id}"
)
theme = theme_service.get_theme(theme_id)
if theme is None or theme.preview_url is None:
raise HTTPException(
status_code=404,
detail=t(locale, "themes.previewNotAvailable", theme_id=theme_id)
detail=f"No preview available for theme: {theme_id}"
)
return {"preview_url": theme.preview_url}
@@ -123,12 +119,12 @@ async def get_theme_preview(theme_id: str, request: Request) -> dict:
logger.error(f"Error getting preview for theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail=t(locale, "themes.previewLoadError")
detail="Failed to load theme preview"
)
@router.get("/{theme_id}/youtube-description")
async def get_theme_youtube_description(theme_id: str, request: Request) -> dict:
async def get_theme_youtube_description(theme_id: str) -> dict:
"""
Get the YouTube description template for a theme.
@@ -145,14 +141,13 @@ async def get_theme_youtube_description(theme_id: str, request: Request) -> dict
Raises:
404: Theme not found
"""
locale = get_locale_from_request(request)
try:
theme_service = get_theme_service()
if not theme_service.theme_exists(theme_id):
raise HTTPException(
status_code=404,
detail=t(locale, "themes.youtubeDescriptionNotFound", theme_id=theme_id)
detail=f"Theme not found: {theme_id}"
)
description = theme_service.get_youtube_description(theme_id)
@@ -163,5 +158,5 @@ async def get_theme_youtube_description(theme_id: str, request: Request) -> dict
logger.error(f"Error getting YouTube description for theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail=t(locale, "themes.youtubeDescriptionLoadError")
detail="Failed to load YouTube description"
)
File diff suppressed because it is too large. Load diff
-35
View File
@@ -1,35 +0,0 @@
# Cloud Build config for GPU base image
#
# Builds the CUDA + PyTorch + models base image used by the audio worker.
# This image is large (~10GB) so builds take 15-30 minutes.
#
# Trigger manually:
# gcloud builds submit --config=backend/cloudbuild-gpu-base.yaml --project=nomadkaraoke --timeout=3600s
#
# The app-layer Dockerfile uses this as its BASE_IMAGE for audio worker deploys.
steps:
- name: 'gcr.io/cloud-builders/docker'
args:
- 'build'
- '-f'
- 'backend/Dockerfile.gpu-base'
- '-t'
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:latest'
- '-t'
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:$BUILD_ID'
- '--build-arg'
- 'BUILD_DATE=$_BUILD_DATE'
- '--build-arg'
- 'BUILD_ID=$BUILD_ID'
- '.'
images:
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:latest'
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:$BUILD_ID'
timeout: '3600s'
options:
machineType: 'E2_HIGHCPU_32'
diskSizeGb: 100
+10 -30
View File
@@ -32,10 +32,6 @@ class Settings(BaseSettings):
# Authentication
admin_tokens: Optional[str] = os.getenv("ADMIN_TOKENS") # Comma-separated list
scheduler_service_account: str = os.getenv(
"SCHEDULER_SERVICE_ACCOUNT",
"karaoke-backend@nomadkaraoke.iam.gserviceaccount.com"
)
# Application
environment: str = os.getenv("ENVIRONMENT", "development")
@@ -61,12 +57,6 @@ class Settings(BaseSettings):
# use uncorrected transcription - human review will fix any issues.
agentic_correction_timeout_seconds: int = int(os.getenv("AGENTIC_CORRECTION_TIMEOUT_SECONDS", "180"))
# AI Credit Evaluation (anti-abuse gating for free credits)
# When enabled, welcome and feedback credit grants are evaluated by Gemini
# before granting. Fail-open: if evaluation fails, credits are granted anyway.
credit_eval_enabled: bool = os.getenv("CREDIT_EVAL_ENABLED", "true").lower() in ("true", "1", "yes")
credit_eval_model: str = os.getenv("CREDIT_EVAL_MODEL", "gemini-3.1-pro-preview")
# Cloud Tasks (for scalable worker coordination)
# When enabled, workers are triggered via Cloud Tasks for guaranteed delivery
# When disabled (default), workers are triggered via direct HTTP (for development)
@@ -93,9 +83,6 @@ class Settings(BaseSettings):
# Requires use_gce_encoding to be enabled and the GCE worker to support /encode-preview endpoint.
use_gce_preview_encoding: bool = os.getenv("USE_GCE_PREVIEW_ENCODING", "false").lower() in ("true", "1", "yes")
# Karaoke-Decide catalog API (for song/artist autocomplete)
karaoke_decide_api_url: str = os.getenv("KARAOKE_DECIDE_API_URL", "https://decide.nomadkaraoke.com")
# Storage paths
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
@@ -110,13 +97,7 @@ class Settings(BaseSettings):
# This is required for torrent downloads since Cloud Run doesn't support BitTorrent.
flacfetch_api_url: Optional[str] = os.getenv("FLACFETCH_API_URL") # e.g., http://10.0.0.5:8080
flacfetch_api_key: Optional[str] = os.getenv("FLACFETCH_API_KEY")
# Flacfetch client retry settings
# Retries help handle transient network failures when calling the flacfetch service
flacfetch_retry_max_attempts: int = int(os.getenv("FLACFETCH_RETRY_MAX_ATTEMPTS", "9")) # Total attempts (1 initial + 8 retries)
flacfetch_retry_min_wait: float = float(os.getenv("FLACFETCH_RETRY_MIN_WAIT", "10.0")) # seconds
flacfetch_retry_max_wait: float = float(os.getenv("FLACFETCH_RETRY_MAX_WAIT", "60.0")) # seconds
# Default distribution settings (can be overridden per-request)
default_dropbox_path: Optional[str] = os.getenv("DEFAULT_DROPBOX_PATH")
default_gdrive_folder_id: Optional[str] = os.getenv("DEFAULT_GDRIVE_FOLDER_ID")
@@ -129,17 +110,16 @@ class Settings(BaseSettings):
default_enable_youtube_upload: bool = os.getenv("DEFAULT_ENABLE_YOUTUBE_UPLOAD", "false").lower() in ("true", "1", "yes")
default_brand_prefix: Optional[str] = os.getenv("DEFAULT_BRAND_PREFIX")
# Private (non-published) track distribution settings
default_private_dropbox_path: Optional[str] = os.getenv("DEFAULT_PRIVATE_DROPBOX_PATH")
default_private_brand_prefix: Optional[str] = os.getenv("DEFAULT_PRIVATE_BRAND_PREFIX", "NOMADNP")
# Rate Limiting Configuration
# Enable/disable rate limiting system-wide (useful for development)
enable_rate_limiting: bool = os.getenv("ENABLE_RATE_LIMITING", "true").lower() in ("true", "1", "yes")
# Maximum jobs a user can create per day (0 = unlimited)
rate_limit_jobs_per_day: int = int(os.getenv("RATE_LIMIT_JOBS_PER_DAY", "5"))
# Maximum YouTube uploads system-wide per day (0 = unlimited)
rate_limit_youtube_uploads_per_day: int = int(os.getenv("RATE_LIMIT_YOUTUBE_UPLOADS_PER_DAY", "10"))
# Maximum beta enrollments from same IP per day (0 = unlimited)
rate_limit_beta_ip_per_day: int = int(os.getenv("RATE_LIMIT_BETA_IP_PER_DAY", "1"))
# YouTube Data API v3 Quota Configuration
# Daily quota limit (YouTube grants 10,000 units/day by default)
youtube_quota_daily_limit: int = int(os.getenv("YOUTUBE_QUOTA_DAILY_LIMIT", "10000"))
# Estimated units consumed per upload (videos.insert=100 + search.list=100 + thumbnails.set=50 + buffer)
youtube_quota_upload_cost: int = int(os.getenv("YOUTUBE_QUOTA_UPLOAD_COST", "300"))
# Safety margin to reserve (avoid hitting exact limit)
youtube_quota_safety_margin: int = int(os.getenv("YOUTUBE_QUOTA_SAFETY_MARGIN", "500"))
# E2E test bypass key for rate limiting (set via secret in production)
e2e_bypass_key: str = os.getenv("E2E_BYPASS_KEY", "")
default_youtube_description: str = os.getenv(
+31 -59
View File
@@ -2,10 +2,40 @@
Custom exceptions for the karaoke generation backend.
These exceptions are used for structured error handling across the application,
particularly for validation errors and credit checks.
particularly for rate limiting and validation errors.
"""
class RateLimitExceededError(Exception):
"""
Raised when a rate limit is exceeded.
Includes information about when the limit will reset for client retry logic.
Attributes:
message: Human-readable error message
limit_type: Type of limit exceeded (e.g., "jobs_per_day", "youtube_uploads")
remaining_seconds: Seconds until the limit resets (for Retry-After header)
current_count: Current usage count
limit_value: The limit that was exceeded
"""
def __init__(
self,
message: str,
limit_type: str = "unknown",
remaining_seconds: int = 0,
current_count: int = 0,
limit_value: int = 0
):
self.message = message
self.limit_type = limit_type
self.remaining_seconds = remaining_seconds
self.current_count = current_count
self.limit_value = limit_value
super().__init__(message)
class EmailValidationError(Exception):
"""
Raised when email validation fails.
@@ -34,61 +64,3 @@ class IPBlockedError(Exception):
self.message = message
self.ip_address = ip_address
super().__init__(message)
class InsufficientCreditsError(Exception):
"""
Raised when a user attempts to create a job without sufficient credits.
Attributes:
message: Human-readable error message
credits_available: Number of credits the user currently has
credits_required: Number of credits required (always 1 for now)
"""
def __init__(
self,
message: str,
credits_available: int = 0,
credits_required: int = 1
):
self.message = message
self.credits_available = credits_available
self.credits_required = credits_required
super().__init__(message)
class InvalidStateTransitionError(Exception):
"""
Raised when an invalid job state transition is attempted.
This exception is raised by JobManager.transition_to_state() when the
requested transition is not allowed by the state machine defined in
STATE_TRANSITIONS.
Example: Trying to transition from PENDING to GENERATING_SCREENS is invalid
because PENDING can only transition to DOWNLOADING, SEARCHING_AUDIO, FAILED,
or CANCELLED.
Attributes:
message: Human-readable error message
job_id: The job ID that failed to transition
from_status: Current status of the job
to_status: Attempted target status
valid_transitions: List of valid transitions from current status
"""
def __init__(
self,
message: str,
job_id: str = "",
from_status: str = "",
to_status: str = "",
valid_transitions: list = None
):
self.message = message
self.job_id = job_id
self.from_status = from_status
self.to_status = to_status
self.valid_transitions = valid_transitions or []
super().__init__(message)
-99
View File
@@ -1,99 +0,0 @@
"""
Lightweight i18n utility for backend translations.
Usage:
from backend.i18n import t, get_locale_from_request, SUPPORTED_LOCALES
# In a route handler:
locale = get_locale_from_request(request)
message = t(locale, "auth.userNotFound")
message = t(locale, "jobs.notFound", job_id="abc123")
"""
import json
from functools import lru_cache
from pathlib import Path
from typing import Any
from starlette.requests import Request
SUPPORTED_LOCALES = ("en", "es", "de")
DEFAULT_LOCALE = "en"
TRANSLATIONS_DIR = Path(__file__).parent / "translations"
@lru_cache(maxsize=len(SUPPORTED_LOCALES))
def _load_translations(locale: str) -> dict[str, Any]:
"""Load and cache translation file for a locale."""
path = TRANSLATIONS_DIR / f"{locale}.json"
if not path.exists():
if locale != DEFAULT_LOCALE:
return _load_translations(DEFAULT_LOCALE)
return {}
with open(path, encoding="utf-8") as f:
result: dict[str, Any] = json.load(f)
return result
def t(locale: str, key: str, **kwargs: Any) -> str:
"""Get a translated string by dotted key path.
Args:
locale: Language code (en, es, de)
key: Dotted path like "auth.userNotFound" or "emails.magicLink.subject"
**kwargs: Interpolation variables (e.g., job_id="abc123")
Returns:
Translated string with variables interpolated, or the key itself if not found.
"""
if locale not in SUPPORTED_LOCALES:
locale = DEFAULT_LOCALE
translations = _load_translations(locale)
# Walk the dotted path
value: Any = translations
for part in key.split("."):
if isinstance(value, dict) and part in value:
value = value[part]
else:
# Key not found — fall back to English
if locale != DEFAULT_LOCALE:
return t(DEFAULT_LOCALE, key, **kwargs)
return key # Return the key itself as last resort
if not isinstance(value, str):
return key
result: str = value
# Interpolate variables
if kwargs:
try:
result = result.format(**kwargs)
except KeyError:
pass # Missing variable — return template as-is
return result
def get_locale_from_request(request: Request) -> str:
"""Extract locale from Accept-Language header.
Returns the first supported locale found, or DEFAULT_LOCALE.
"""
accept_lang = request.headers.get("accept-language", "")
# Simple parsing — Accept-Language can be "de", "de-DE", "en-US,en;q=0.9,de;q=0.8"
for part in accept_lang.split(","):
lang = part.split(";")[0].strip().split("-")[0].lower()
if lang in SUPPORTED_LOCALES:
return lang
return DEFAULT_LOCALE
def get_locale_prefix(locale: str) -> str:
"""Get URL locale prefix for frontend URLs (e.g., '/en', '/es')."""
if locale not in SUPPORTED_LOCALES:
locale = DEFAULT_LOCALE
return f"/{locale}"
+16 -19
View File
@@ -7,7 +7,7 @@ from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from backend.config import settings
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, rate_limits, push, catalog, encoding_worker
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, rate_limits, push
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
from backend.services.structured_logging import setup_structured_logging
from backend.services.spacy_preloader import preload_spacy_model
@@ -136,8 +136,8 @@ if tracing_enabled:
# Configure CORS
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=False,
allow_origins=["*"], # Configure this properly for production
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
@@ -153,40 +153,37 @@ app.include_router(health.router, prefix="/api")
app.include_router(jobs.router, prefix="/api")
app.include_router(file_upload.router, prefix="/api") # File upload endpoint
app.include_router(internal.router, prefix="/api") # Internal worker endpoints
app.include_router(encoding_worker.router, prefix="/api") # Encoding worker lifecycle
app.include_router(review.router, prefix="/api") # Review UI compatibility endpoints
app.include_router(auth.router, prefix="/api") # OAuth credential management
app.include_router(audio_search.router, prefix="/api") # Audio search (artist+title mode)
app.include_router(themes.router, prefix="/api") # Theme selection for styles
app.include_router(users.router, prefix="/api") # User auth, credits, and Stripe webhooks
from backend.api.routes import referrals
app.include_router(referrals.router, prefix="/api") # Referral links and payouts
app.include_router(admin.router, prefix="/api") # Admin dashboard and management
app.include_router(rate_limits.router, prefix="/api") # Rate limits admin management
app.include_router(push.router, prefix="/api") # Push notification subscription management
app.include_router(catalog.router, prefix="/api") # Catalog proxy for song/artist autocomplete
app.include_router(tenant.router) # Tenant/white-label configuration (no /api prefix, router has it)
# Exception handlers
# Exception handler for rate limiting
from fastapi import Request
from fastapi.responses import JSONResponse
from backend.exceptions import InsufficientCreditsError
from backend.i18n import t, get_locale_from_request
from backend.exceptions import RateLimitExceededError
@app.exception_handler(InsufficientCreditsError)
async def insufficient_credits_exception_handler(request: Request, exc: InsufficientCreditsError):
"""Handle insufficient credits errors with 402 Payment Required status."""
locale = get_locale_from_request(request)
@app.exception_handler(RateLimitExceededError)
async def rate_limit_exception_handler(request: Request, exc: RateLimitExceededError):
"""Handle rate limit exceeded errors with 429 status."""
return JSONResponse(
status_code=402,
status_code=429,
content={
"detail": t(locale, "errors.insufficientCredits"),
"credits_available": exc.credits_available,
"credits_required": exc.credits_required,
"buy_url": t(locale, "errors.insufficientCreditsBuyUrl"),
"detail": exc.message,
"limit_type": exc.limit_type,
"current_count": exc.current_count,
"limit_value": exc.limit_value,
},
headers={
"Retry-After": str(exc.remaining_seconds),
} if exc.remaining_seconds > 0 else None,
)
-110
View File
@@ -1,110 +0,0 @@
"""
Audio edit session model for persisting input audio editing progress.
Stored as a Firestore subcollection: jobs/{job_id}/audio_edit_sessions/{session_id}
Full edit data is stored in GCS to avoid Firestore's 1MB document limit.
"""
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Optional, Dict, Any, List
import uuid
@dataclass
class AudioEditSessionSummary:
"""Compact summary of operations in an audio edit session, for list views."""
total_operations: int = 0
operations_breakdown: Dict[str, int] = field(default_factory=dict)
duration_change_seconds: float = 0.0
net_duration_seconds: float = 0.0
def to_dict(self) -> Dict[str, Any]:
return {
"total_operations": self.total_operations,
"operations_breakdown": self.operations_breakdown,
"duration_change_seconds": self.duration_change_seconds,
"net_duration_seconds": self.net_duration_seconds,
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "AudioEditSessionSummary":
return cls(
total_operations=data.get("total_operations", 0),
operations_breakdown=data.get("operations_breakdown", {}),
duration_change_seconds=data.get("duration_change_seconds", 0.0),
net_duration_seconds=data.get("net_duration_seconds", 0.0),
)
@dataclass
class AudioEditSession:
"""
A saved snapshot of an audio editing session.
Stored at: jobs/{job_id}/audio_edit_sessions/{session_id}
Edit data stored at: jobs/{job_id}/audio_edit_sessions/{session_id}.json (GCS)
"""
session_id: str = field(default_factory=lambda: str(uuid.uuid4()))
job_id: str = ""
user_email: str = ""
created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
updated_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
edit_count: int = 0
trigger: str = "auto" # "auto" | "manual" | "submit"
audio_duration_seconds: Optional[float] = None
original_duration_seconds: Optional[float] = None
artist: Optional[str] = None
title: Optional[str] = None
summary: AudioEditSessionSummary = field(default_factory=AudioEditSessionSummary)
edit_data_gcs_path: str = ""
# Hash of the edit data JSON, used to skip duplicate saves
data_hash: str = ""
def to_dict(self) -> Dict[str, Any]:
return {
"session_id": self.session_id,
"job_id": self.job_id,
"user_email": self.user_email,
"created_at": self.created_at,
"updated_at": self.updated_at,
"edit_count": self.edit_count,
"trigger": self.trigger,
"audio_duration_seconds": self.audio_duration_seconds,
"original_duration_seconds": self.original_duration_seconds,
"artist": self.artist,
"title": self.title,
"summary": self.summary.to_dict(),
"edit_data_gcs_path": self.edit_data_gcs_path,
"data_hash": self.data_hash,
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "AudioEditSession":
created_at = data.get("created_at")
if isinstance(created_at, str):
created_at = datetime.fromisoformat(created_at.rstrip("Z")).replace(tzinfo=timezone.utc)
updated_at = data.get("updated_at")
if isinstance(updated_at, str):
updated_at = datetime.fromisoformat(updated_at.rstrip("Z")).replace(tzinfo=timezone.utc)
summary_data = data.get("summary", {})
summary = AudioEditSessionSummary.from_dict(summary_data) if isinstance(summary_data, dict) else AudioEditSessionSummary()
return cls(
session_id=data.get("session_id", str(uuid.uuid4())),
job_id=data.get("job_id", ""),
user_email=data.get("user_email", ""),
created_at=created_at or datetime.now(timezone.utc),
updated_at=updated_at or datetime.now(timezone.utc),
edit_count=data.get("edit_count", 0),
trigger=data.get("trigger", "auto"),
audio_duration_seconds=data.get("audio_duration_seconds"),
original_duration_seconds=data.get("original_duration_seconds"),
artist=data.get("artist"),
title=data.get("title"),
summary=summary,
edit_data_gcs_path=data.get("edit_data_gcs_path", ""),
data_hash=data.get("data_hash", ""),
)
+11 -89
View File
@@ -16,9 +16,8 @@ class JobStatus(str, Enum):
"""
Job status enumeration - Complete state machine.
The workflow has 8 main stages with up to 3 human interaction points:
The workflow has 7 main stages with 2 human interaction points:
1. Input & Setup (may include audio source selection)
1b. Optional Audio Editing (BLOCKING) - trim/cut/mute/join input audio
2. Parallel Processing (audio + lyrics)
3. Title/End Screen Generation + Backing Vocals Analysis
4. Countdown Padding Synchronization
@@ -40,11 +39,6 @@ class JobStatus(str, Enum):
DOWNLOADING = "downloading" # Downloading from URL or processing upload
# Optional: Audio editing (before processing begins)
AWAITING_AUDIO_EDIT = "awaiting_audio_edit" # ⚠️ WAITING FOR USER - edit input audio
IN_AUDIO_EDIT = "in_audio_edit" # User is actively editing audio
AUDIO_EDIT_COMPLETE = "audio_edit_complete" # User submitted edited audio
# Stage 2a: Audio separation (parallel track 1)
SEPARATING_STAGE1 = "separating_stage1" # Clean instrumental separation (Modal API)
SEPARATING_STAGE2 = "separating_stage2" # Backing vocals separation (Modal API)
@@ -99,9 +93,8 @@ class JobStatus(str, Enum):
# Valid state transitions
STATE_TRANSITIONS = {
# PENDING can go to DOWNLOADING (file upload), SEARCHING_AUDIO (artist+title search),
# or DOWNLOADING_AUDIO (create-from-search where audio is pre-selected)
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
# PENDING can go to DOWNLOADING (file upload) or SEARCHING_AUDIO (artist+title search)
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
# Audio search flow (for artist+title search mode)
JobStatus.SEARCHING_AUDIO: [JobStatus.AWAITING_AUDIO_SELECTION, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED],
@@ -109,13 +102,7 @@ STATE_TRANSITIONS = {
JobStatus.DOWNLOADING_AUDIO: [JobStatus.DOWNLOADING, JobStatus.FAILED],
# DOWNLOADING allows parallel processing (audio + lyrics) and then screens when both complete
# Can also enter optional audio edit phase if requires_audio_edit flag is set
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.AWAITING_AUDIO_EDIT, JobStatus.FAILED],
# Optional audio edit phase (trim, cut, mute, join input audio before processing)
JobStatus.AWAITING_AUDIO_EDIT: [JobStatus.IN_AUDIO_EDIT, JobStatus.AUDIO_EDIT_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.IN_AUDIO_EDIT: [JobStatus.AWAITING_AUDIO_EDIT, JobStatus.AUDIO_EDIT_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.AUDIO_EDIT_COMPLETE: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
# Audio separation flow
JobStatus.SEPARATING_STAGE1: [JobStatus.SEPARATING_STAGE2, JobStatus.FAILED],
@@ -154,14 +141,13 @@ STATE_TRANSITIONS = {
JobStatus.UPLOADING: [JobStatus.NOTIFYING, JobStatus.COMPLETE, JobStatus.FAILED],
JobStatus.NOTIFYING: [JobStatus.COMPLETE, JobStatus.FAILED],
# Terminal states - COMPLETE allows edit (reopen review) and visibility change
# Terminal states - COMPLETE, PREP_COMPLETE have no transitions
# FAILED and CANCELLED allow retry transitions to resume from checkpoints
# PREP_COMPLETE allows continuation from combined review
JobStatus.COMPLETE: [JobStatus.AWAITING_REVIEW, JobStatus.LYRICS_COMPLETE], # Edit track or visibility change
JobStatus.COMPLETE: [],
JobStatus.PREP_COMPLETE: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED], # Continue from combined review
JobStatus.FAILED: [
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
JobStatus.DOWNLOADING_AUDIO, # Retry audio download (audio search flow)
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
JobStatus.REVIEW_COMPLETE, # Retry from render stage
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
@@ -169,7 +155,6 @@ STATE_TRANSITIONS = {
],
JobStatus.CANCELLED: [
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
JobStatus.DOWNLOADING_AUDIO, # Retry audio download (audio search flow)
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
JobStatus.REVIEW_COMPLETE, # Retry from render stage
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
@@ -191,7 +176,6 @@ class TimelineEvent(BaseModel):
timestamp: str
progress: Optional[int] = None
message: Optional[str] = None
metadata: Optional[Dict[str, Any]] = None
class LogEntry(BaseModel):
@@ -221,14 +205,7 @@ class Job(BaseModel):
title: Optional[str] = None
filename: Optional[str] = None # Original uploaded filename
input_media_gcs_path: Optional[str] = None # GCS path to uploaded file
# Audio source tracking (for retry capability)
audio_source_type: Optional[str] = None # Source type: 'audio_search', 'youtube_url', 'file_upload'
source_name: Optional[str] = None # Provider name (RED, OPS, YouTube, Spotify) - for audio_search
source_id: Optional[str] = None # Source-specific ID (torrent ID, video ID, track ID) - for audio_search
target_file: Optional[str] = None # For torrents, specific file to extract - for audio_search
download_url: Optional[str] = None # Direct download URL if available - for audio_search
# User preferences
enable_cdg: bool = False # Generate CDG+MP3 package (requires style config)
enable_txt: bool = False # Generate TXT+MP3 package (requires style config)
@@ -238,8 +215,8 @@ class Job(BaseModel):
user_email: Optional[str] = None # Email for notifications
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
# Multi-tenant support ("" = consumer portal, "vocalstar"/etc = tenant portal)
tenant_id: str = "" # Tenant ID for white-label portal scoping
# Multi-tenant support (None = default Nomad Karaoke)
tenant_id: Optional[str] = None # Tenant ID for white-label portal scoping
# Theme configuration (pre-made themes from GCS)
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
@@ -315,25 +292,10 @@ class Job(BaseModel):
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
customer_notes: Optional[str] = None # Notes provided by customer with their order
# Private (non-published) track mode
is_private: bool = False # Private tracks: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive
# Creation-time decision params (for observability — what did the user actually choose?)
creation_params: Dict[str, Any] = Field(default_factory=dict)
# Anti-abuse: IP address at job creation time (for cross-account correlation)
creation_ip: Optional[str] = None
# Output deletion tracking (for admin cleanup without deleting job)
outputs_deleted_at: Optional[datetime] = None # Timestamp when outputs were deleted by admin
outputs_deleted_by: Optional[str] = None # Admin email who deleted outputs
# Credit refund tracking (prevents double-refund on cancel→delete or fail→delete)
credit_refunded: bool = False
# Edit tracking (how many times a completed track has been re-edited)
edit_count: int = 0
# Processing state
track_output_dir: Optional[str] = None # Local output directory (temp)
audio_hash: Optional[str] = None # Hash for deduplication
@@ -424,43 +386,6 @@ class Job(BaseModel):
}
"""
# Processing metadata (immutable provenance data written by workers during processing)
processing_metadata: Dict[str, Any] = Field(default_factory=dict)
"""
Immutable processing provenance data. Written once by each worker during processing.
Never modified after initial write. Used for analytics, debugging, and data export.
Structure:
{
"transcription": {
"provider": "audioshake",
"audioshake_task_id": "...",
"audioshake_asset_id": "...",
"language_detected": "en",
"word_count": 358,
"segment_count": 42,
},
"correction": {
"handlers_applied": [...],
"corrections_made": 12,
"correction_ratio": 0.034,
"reference_sources_found": ["genius", "spotify"],
"agentic_routing": "rule-based",
"duration_seconds": 33.2,
},
"separation": {
"provider": "modal",
"clean_model": "...",
"duration_seconds": 420.5,
},
"timing": {
"lyrics_worker_seconds": 125.3,
"audio_worker_seconds": 420.5,
...
},
}
"""
# Request metadata (captured at job creation for tracking and filtering)
request_metadata: Dict[str, Any] = Field(default_factory=dict)
"""
@@ -574,9 +499,6 @@ class JobCreate(BaseModel):
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
customer_notes: Optional[str] = None # Notes provided by customer with their order
# Private (non-published) track mode
is_private: bool = False # Private tracks: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive
# Request metadata (set by API endpoint from request headers)
request_metadata: Dict[str, Any] = Field(default_factory=dict)
"""
@@ -589,8 +511,8 @@ class JobCreate(BaseModel):
- custom_headers: All X-* headers
"""
# Tenant scoping for white-label portals ("" = consumer, "vocalstar"/etc = tenant)
tenant_id: str = "" # Tenant ID for job scoping
# Tenant scoping for white-label portals
tenant_id: Optional[str] = None # Tenant ID for job scoping
@validator('url')
def validate_url(cls, v):
-178
View File
@@ -1,178 +0,0 @@
"""
Referral system models.
Supports:
- Referral link creation and management
- Earnings tracking from referred purchases
- Payout processing via Stripe Connect
- API request/response shapes for referral endpoints
"""
from datetime import datetime
from typing import Optional, List
from pydantic import BaseModel, Field
# ============================================================================
# Firestore Document Models
# ============================================================================
class ReferralLinkStats(BaseModel):
"""Aggregated statistics for a referral link."""
clicks: int = 0
signups: int = 0
purchases: int = 0
total_earned_cents: int = 0
class ReferralLink(BaseModel):
"""
Referral link stored in Firestore.
Each user can have one referral link. Vanity links use a custom code
(e.g., "karaoke-king") while standard links use a generated code.
"""
code: str # Unique referral code (URL slug)
owner_email: str # Email of the referrer
display_name: Optional[str] = None # Public name shown on interstitial
custom_message: Optional[str] = None # Optional message on interstitial
# Reward configuration
discount_percent: int = 10 # Discount for referred user
kickback_percent: int = 20 # Earnings for referrer
discount_duration_days: int = 30 # How long the discount lasts
earning_duration_days: int = 365 # How long referrer earns from this link
# Stripe
stripe_coupon_id: Optional[str] = None # Stripe coupon for this link
# Flags
is_vanity: bool = False # Custom vanity code vs generated
enabled: bool = True # Can be disabled without deleting
# Timestamps
created_at: datetime = Field(default_factory=datetime.utcnow)
updated_at: datetime = Field(default_factory=datetime.utcnow)
# Aggregated stats
stats: ReferralLinkStats = Field(default_factory=ReferralLinkStats)
class ReferralEarning(BaseModel):
"""
Record of a single earning from a referred purchase.
Created when a referred user completes a Stripe checkout.
"""
id: str
referrer_email: str
referred_email: str
referral_code: str
stripe_session_id: str
purchase_amount_cents: int
earning_amount_cents: int
status: str = "pending" # pending, paid, cancelled
created_at: datetime = Field(default_factory=datetime.utcnow)
paid_at: Optional[datetime] = None
payout_id: Optional[str] = None
class ReferralPayout(BaseModel):
"""
Record of a payout to a referrer via Stripe Connect.
Groups multiple earnings into a single transfer.
"""
id: str
referrer_email: str
stripe_transfer_id: str
amount_cents: int
earnings_included: List[str] # List of ReferralEarning IDs
status: str = "processing" # processing, completed, failed
created_at: datetime = Field(default_factory=datetime.utcnow)
# ============================================================================
# API Request Models
# ============================================================================
class CreateReferralLinkRequest(BaseModel):
"""Admin request to create a vanity referral link."""
vanity_code: Optional[str] = None
owner_email: str
display_name: Optional[str] = None
custom_message: Optional[str] = None
discount_percent: int = 10
kickback_percent: int = 20
discount_duration_days: int = 30
earning_duration_days: int = 365
class UpdateReferralLinkRequest(BaseModel):
"""Request to update a referral link (user: name/message only, admin: all fields)."""
display_name: Optional[str] = None
custom_message: Optional[str] = None
discount_percent: Optional[int] = None
kickback_percent: Optional[int] = None
discount_duration_days: Optional[int] = None
earning_duration_days: Optional[int] = None
enabled: Optional[bool] = None
# ============================================================================
# API Response Models
# ============================================================================
class ReferralLinkResponse(BaseModel):
"""Public referral link information."""
code: str
owner_email: str
display_name: Optional[str] = None
custom_message: Optional[str] = None
discount_percent: int
kickback_percent: int
is_vanity: bool
enabled: bool
stats: ReferralLinkStats
created_at: datetime
class ReferralEarningResponse(BaseModel):
"""Public earning information (for referrer dashboard)."""
id: str
referred_email: str
referral_code: str
purchase_amount_cents: int
earning_amount_cents: int
status: str
created_at: datetime
paid_at: Optional[datetime] = None
class ReferralPayoutResponse(BaseModel):
"""Public payout information (for referrer dashboard)."""
id: str
amount_cents: int
earnings_count: int
status: str
created_at: datetime
class ReferralDashboardResponse(BaseModel):
"""Full referral dashboard data for the referrer."""
referral_link: ReferralLinkResponse
recent_earnings: List[ReferralEarningResponse]
recent_payouts: List[ReferralPayoutResponse]
total_earned_cents: int
total_paid_cents: int
pending_balance_cents: int
stripe_connect_onboarded: bool
class ReferralInterstitialResponse(BaseModel):
"""Data for the referral landing/interstitial page."""
valid: bool = True
referral_code: str = ""
referrer_display_name: Optional[str] = None
discount_percent: int = 0
discount_duration_days: int = 0
custom_message: Optional[str] = None
+1 -23
View File
@@ -10,7 +10,7 @@ class URLSubmissionRequest(BaseModel):
url: HttpUrl
artist: Optional[str] = None # Auto-detected if not provided
title: Optional[str] = None # Auto-detected if not provided
# Optional preferences
enable_cdg: bool = False # Requires style config
enable_txt: bool = False # Requires style config
@@ -18,7 +18,6 @@ class URLSubmissionRequest(BaseModel):
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
is_private: Optional[bool] = None # Private (non-published) track mode
class UploadSubmissionRequest(BaseModel):
@@ -142,24 +141,3 @@ class RetryJobRequest(BaseModel):
"""Request to retry a failed job."""
from_stage: Optional[str] = None # Optional: restart from specific stage
class ChangeVisibilityRequest(BaseModel):
"""Request to change job visibility (public/private)."""
target_visibility: str # "public" or "private"
@validator('target_visibility')
def validate_target(cls, v):
if v not in ('public', 'private'):
raise ValueError("target_visibility must be 'public' or 'private'")
return v
class ChangeVisibilityResponse(BaseModel):
"""Response from change visibility endpoint."""
status: str # "success", "processing", "error"
job_id: str
message: str
previous_visibility: str # "public" or "private"
new_visibility: str # "public" or "private"
reprocessing_required: bool
-107
View File
@@ -1,107 +0,0 @@
"""
Review session model for persisting lyrics review progress.
Stored as a Firestore subcollection: jobs/{job_id}/review_sessions/{session_id}
Full correction data is stored in GCS to avoid Firestore's 1MB document limit.
"""
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Optional, Dict, Any, List
import uuid
@dataclass
class ReviewSessionSummary:
"""Compact summary of edits in a review session, for list views."""
total_segments: int = 0
total_words: int = 0
corrections_made: int = 0
changed_words: List[Dict[str, Any]] = field(default_factory=list)
def to_dict(self) -> Dict[str, Any]:
return {
"total_segments": self.total_segments,
"total_words": self.total_words,
"corrections_made": self.corrections_made,
"changed_words": self.changed_words,
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "ReviewSessionSummary":
return cls(
total_segments=data.get("total_segments", 0),
total_words=data.get("total_words", 0),
corrections_made=data.get("corrections_made", 0),
changed_words=data.get("changed_words", []),
)
@dataclass
class ReviewSession:
"""
A saved snapshot of a lyrics review editing session.
Stored at: jobs/{job_id}/review_sessions/{session_id}
Correction data stored at: jobs/{job_id}/review_sessions/{session_id}.json (GCS)
"""
session_id: str = field(default_factory=lambda: str(uuid.uuid4()))
job_id: str = ""
user_email: str = ""
created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
updated_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
edit_count: int = 0
trigger: str = "auto" # "auto" | "preview" | "manual"
audio_duration_seconds: Optional[float] = None
artist: Optional[str] = None
title: Optional[str] = None
summary: ReviewSessionSummary = field(default_factory=ReviewSessionSummary)
correction_data_gcs_path: str = ""
# Hash of the correction data JSON, used to skip duplicate saves
data_hash: str = ""
def to_dict(self) -> Dict[str, Any]:
return {
"session_id": self.session_id,
"job_id": self.job_id,
"user_email": self.user_email,
"created_at": self.created_at,
"updated_at": self.updated_at,
"edit_count": self.edit_count,
"trigger": self.trigger,
"audio_duration_seconds": self.audio_duration_seconds,
"artist": self.artist,
"title": self.title,
"summary": self.summary.to_dict(),
"correction_data_gcs_path": self.correction_data_gcs_path,
"data_hash": self.data_hash,
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "ReviewSession":
created_at = data.get("created_at")
if isinstance(created_at, str):
created_at = datetime.fromisoformat(created_at.rstrip("Z")).replace(tzinfo=timezone.utc)
updated_at = data.get("updated_at")
if isinstance(updated_at, str):
updated_at = datetime.fromisoformat(updated_at.rstrip("Z")).replace(tzinfo=timezone.utc)
summary_data = data.get("summary", {})
summary = ReviewSessionSummary.from_dict(summary_data) if isinstance(summary_data, dict) else ReviewSessionSummary()
return cls(
session_id=data.get("session_id", str(uuid.uuid4())),
job_id=data.get("job_id", ""),
user_email=data.get("user_email", ""),
created_at=created_at or datetime.now(timezone.utc),
updated_at=updated_at or datetime.now(timezone.utc),
edit_count=data.get("edit_count", 0),
trigger=data.get("trigger", "auto"),
audio_duration_seconds=data.get("audio_duration_seconds"),
artist=data.get("artist"),
title=data.get("title"),
summary=summary,
correction_data_gcs_path=data.get("correction_data_gcs_path", ""),
data_hash=data.get("data_hash", ""),
)
-10
View File
@@ -95,12 +95,6 @@ class TenantDefaults(BaseModel):
brand_prefix: Optional[str] = Field(
None, description="Prefix for output filenames (e.g., 'VSTAR')"
)
dropbox_path: Optional[str] = Field(
None, description="Tenant-specific Dropbox folder path (e.g., '/Karaoke/Vocal-Star')"
)
gdrive_folder_id: Optional[str] = Field(
None, description="Tenant-specific Google Drive folder ID for output uploads"
)
youtube_description_template: Optional[str] = Field(
None, description="Default YouTube description template"
)
@@ -145,10 +139,6 @@ class TenantConfig(BaseModel):
created_at: Optional[datetime] = None
updated_at: Optional[datetime] = None
def get_frontend_url(self) -> str:
"""Get the frontend URL for this tenant's portal."""
return f"https://{self.subdomain}"
def get_sender_email(self) -> str:
"""Get the email sender address for this tenant."""
if self.auth.sender_email:
+50 -71
View File
@@ -6,6 +6,7 @@ Supports:
- Credit system for karaoke generation
- Role-based access control (user/admin)
- Stripe integration for payments
- Beta tester program with feedback collection
"""
from datetime import datetime, timezone
from enum import Enum
@@ -19,6 +20,14 @@ class UserRole(str, Enum):
ADMIN = "admin"
class BetaTesterStatus(str, Enum):
"""Status of beta tester participation."""
ACTIVE = "active" # Enrolled, free credits available
PENDING_FEEDBACK = "pending_feedback" # Job completed, awaiting feedback
COMPLETED = "completed" # Feedback submitted
EXPIRED = "expired" # 24hr deadline passed without feedback
class CreditTransaction(BaseModel):
"""Record of a credit transaction."""
id: str
@@ -36,15 +45,10 @@ class PushSubscription(BaseModel):
Stores the push subscription endpoint and encryption keys needed
to send push notifications to the user's browser/device.
Subscriptions are scoped by tenant_id to prevent cross-tenant
notification leakage (e.g., a consumer job notification appearing
on a tenant portal's service worker).
"""
endpoint: str # Push service endpoint URL
keys: Dict[str, str] # p256dh and auth keys for encryption
device_name: Optional[str] = None # e.g., "iPhone", "Chrome on Windows"
tenant_id: Optional[str] = None # Tenant scope (None = consumer/default portal)
created_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
last_used_at: Optional[datetime] = None # Last time a notification was sent
@@ -85,36 +89,21 @@ class User(BaseModel):
total_jobs_created: int = 0
total_jobs_completed: int = 0
# Payment tracking (cached from stripe_payments, in cents)
total_spent: int = 0
# Optional profile fields for future use
display_name: Optional[str] = None
# i18n locale preference (set from Accept-Language on login, used for emails/push)
locale: Optional[str] = None
# Beta tester program
is_beta_tester: bool = False
beta_tester_status: Optional[BetaTesterStatus] = None
beta_enrolled_at: Optional[datetime] = None
beta_promise_text: Optional[str] = None # User's promise statement
beta_feedback_due_at: Optional[datetime] = None # 24hr after job completion
beta_feedback_email_sent: bool = False
# Push notification subscriptions (Web Push API)
# Users can subscribe from multiple devices/browsers
push_subscriptions: List[PushSubscription] = Field(default_factory=list)
# Feedback-for-credits program
has_submitted_feedback: bool = False
# Anti-abuse: IP address and device fingerprint used during signup
signup_ip: Optional[str] = None
device_fingerprint: Optional[str] = None
# Flag to prevent duplicate welcome credit grants (idempotency)
welcome_credits_granted: bool = False
# Referral system
referral_code: Optional[str] = None # User's own referral code
referred_by_code: Optional[str] = None # Code used when this user signed up
referred_at: Optional[datetime] = None # When the referral was applied
referral_discount_expires_at: Optional[datetime] = None # When referral discount ends
stripe_connect_account_id: Optional[str] = None # For receiving referral payouts
class MagicLinkToken(BaseModel):
"""
@@ -134,19 +123,10 @@ class MagicLinkToken(BaseModel):
used_at: Optional[datetime] = None
ip_address: Optional[str] = None
user_agent: Optional[str] = None
device_fingerprint: Optional[str] = None
# Multi-tenant support (None = default Nomad Karaoke)
tenant_id: Optional[str] = None
# Referral attribution (travels with magic link for cross-device support)
referral_code: Optional[str] = None
# Pre-computed credit evaluation (set asynchronously after magic link is sent)
credit_eval_decision: Optional[str] = None # "grant", "deny", "pending_review"
credit_eval_reasoning: Optional[str] = None
credit_eval_error: Optional[str] = None
class Session(BaseModel):
"""
@@ -164,7 +144,6 @@ class Session(BaseModel):
last_activity_at: datetime = Field(default_factory=datetime.utcnow)
ip_address: Optional[str] = None
user_agent: Optional[str] = None
device_fingerprint: Optional[str] = None
is_active: bool = True
# Multi-tenant support (None = default Nomad Karaoke)
@@ -176,8 +155,6 @@ class Session(BaseModel):
class SendMagicLinkRequest(BaseModel):
"""Request to send a magic link email."""
email: str
device_fingerprint: Optional[str] = None
referral_code: Optional[str] = None # From referral interstitial
class SendMagicLinkResponse(BaseModel):
@@ -197,9 +174,6 @@ class VerifyMagicLinkResponse(BaseModel):
session_token: str
user: "UserPublic"
message: str
tenant_subdomain: Optional[str] = None
credits_granted: int = 0
credit_status: str = "not_applicable" # "granted", "denied", "pending_review", "already_granted", "not_applicable"
class UserPublic(BaseModel):
@@ -211,15 +185,6 @@ class UserPublic(BaseModel):
total_jobs_created: int = 0
total_jobs_completed: int = 0
tenant_id: Optional[str] = None
feedback_eligible: bool = False
total_spent: int = 0
referral_code: Optional[str] = None
has_active_referral_discount: bool = False
referral_discount_percent: Optional[int] = None
referral_discount_expires_at: Optional[str] = None # ISO string
referred_by_code: Optional[str] = None
created_at: Optional[str] = None
last_login_at: Optional[str] = None
class AddCreditsRequest(BaseModel):
@@ -245,18 +210,18 @@ class UserListResponse(BaseModel):
# ============================================================================
# User Feedback-for-Credits Models
# Beta Tester Program Models
# ============================================================================
class UserFeedback(BaseModel):
class BetaTesterFeedback(BaseModel):
"""
Feedback from any user (not just beta testers).
Feedback from a beta tester after using the service.
Stored in Firestore 'user_feedback' collection.
Users earn 1 free credit for submitting detailed feedback.
Stored in Firestore 'beta_feedback' collection.
"""
id: str
id: str # Unique feedback ID
user_email: str
job_id: Optional[str] = None # The job they're providing feedback on
# Ratings (1-5 scale)
overall_rating: int = Field(ge=1, le=5)
@@ -275,11 +240,33 @@ class UserFeedback(BaseModel):
# Metadata
created_at: datetime = Field(default_factory=datetime.utcnow)
submitted_via: str = "web"
submitted_via: str = "web" # 'web' or 'email'
class UserFeedbackRequest(BaseModel):
"""Request to submit user feedback for credits."""
class BetaTesterEnrollRequest(BaseModel):
"""Request to enroll as a beta tester."""
email: str
promise_text: str = Field(
min_length=10,
description="User's promise to provide feedback (min 10 chars)"
)
accept_corrections_work: bool = Field(
description="User accepts they may need to review/correct lyrics"
)
class BetaTesterEnrollResponse(BaseModel):
"""Response after enrolling as beta tester."""
status: str
message: str
credits_granted: int
session_token: Optional[str] = None # If new user, provide session
class BetaFeedbackRequest(BaseModel):
"""Request to submit beta tester feedback."""
job_id: Optional[str] = None
# Ratings (1-5 scale)
overall_rating: int = Field(ge=1, le=5)
ease_of_use_rating: int = Field(ge=1, le=5)
@@ -296,16 +283,8 @@ class UserFeedbackRequest(BaseModel):
would_use_again: bool = True
class UserFeedbackResponse(BaseModel):
"""Response after submitting user feedback."""
class BetaFeedbackResponse(BaseModel):
"""Response after submitting feedback."""
status: str
message: str
credits_granted: int = 0
class FeedbackEligibilityResponse(BaseModel):
"""Response for feedback eligibility check."""
eligible: bool
has_submitted: bool
jobs_completed: int
credits_reward: int = 1
bonus_credits: int = 0 # Bonus credits for great feedback
-48
View File
@@ -1,48 +0,0 @@
#!/usr/bin/env python3
"""
Download ensemble preset models at Docker build time.
Bakes models into the image so they're available immediately at runtime,
avoiding cold-start model download latency in Cloud Run Jobs.
Usage: python backend/scripts/download_models.py /models
"""
import json
import sys
import logging
import importlib.resources as resources
from audio_separator.separator import Separator
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
# Presets to bake into the GPU image
PRESETS_TO_DOWNLOAD = ["instrumental_clean", "karaoke"]
def download_preset_models(model_dir: str) -> None:
"""Download all models for the configured presets individually.
Ensemble presets resolve to multiple model filenames. We must download
each model individually via load_model(model_filename) — calling
load_model() with no args on an ensemble preset doesn't trigger downloads.
"""
with resources.open_text("audio_separator", "ensemble_presets.json") as f:
presets = json.load(f)["presets"]
models_to_download = set()
for preset_name in PRESETS_TO_DOWNLOAD:
models_to_download.update(presets[preset_name]["models"])
logger.info(f"Downloading {len(models_to_download)} models for presets: {PRESETS_TO_DOWNLOAD}")
for model in sorted(models_to_download):
logger.info(f" Downloading: {model}")
sep = Separator(model_file_dir=model_dir)
sep.load_model(model)
logger.info(f" Done: {model}")
if __name__ == "__main__":
model_dir = sys.argv[1] if len(sys.argv) > 1 else "/models"
download_preset_models(model_dir)
logger.info(f"All preset models downloaded to {model_dir}")
-265
View File
@@ -1,265 +0,0 @@
#!/usr/bin/env python3
"""
Generate a Mermaid state diagram from the job state machine.
This script reads the STATE_TRANSITIONS dictionary from models/job.py
and generates a Mermaid diagram that can be embedded in documentation.
Usage:
python scripts/generate_state_diagram.py > docs/state-machine.mmd
# Or to update ARCHITECTURE.md directly:
python scripts/generate_state_diagram.py --update-docs
Created as part of state machine robustness improvements (2026-02-02).
"""
import sys
import argparse
from pathlib import Path
# Add backend to path for imports
backend_dir = Path(__file__).parent.parent
sys.path.insert(0, str(backend_dir))
# Import the state machine data
# Note: Must be run from an environment where karaoke_gen is installed
try:
from models.job import STATE_TRANSITIONS, JobStatus
except ImportError:
# Fallback: define the state machine inline for documentation purposes
from enum import Enum
class JobStatus(str, Enum):
PENDING = "pending"
SEARCHING_AUDIO = "searching_audio"
AWAITING_AUDIO_SELECTION = "awaiting_audio_selection"
DOWNLOADING_AUDIO = "downloading_audio"
DOWNLOADING = "downloading"
SEPARATING_STAGE1 = "separating_stage1"
SEPARATING_STAGE2 = "separating_stage2"
AUDIO_COMPLETE = "audio_complete"
TRANSCRIBING = "transcribing"
CORRECTING = "correcting"
LYRICS_COMPLETE = "lyrics_complete"
GENERATING_SCREENS = "generating_screens"
APPLYING_PADDING = "applying_padding"
AWAITING_REVIEW = "awaiting_review"
IN_REVIEW = "in_review"
REVIEW_COMPLETE = "review_complete"
RENDERING_VIDEO = "rendering_video"
AWAITING_INSTRUMENTAL_SELECTION = "awaiting_instrumental_selection"
INSTRUMENTAL_SELECTED = "instrumental_selected"
GENERATING_VIDEO = "generating_video"
ENCODING = "encoding"
PACKAGING = "packaging"
UPLOADING = "uploading"
NOTIFYING = "notifying"
COMPLETE = "complete"
PREP_COMPLETE = "prep_complete"
FAILED = "failed"
CANCELLED = "cancelled"
QUEUED = "queued"
PROCESSING = "processing"
ERROR = "error"
READY_FOR_FINALIZATION = "ready_for_finalization"
FINALIZING = "finalizing"
STATE_TRANSITIONS = {
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.SEARCHING_AUDIO: [JobStatus.AWAITING_AUDIO_SELECTION, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED],
JobStatus.AWAITING_AUDIO_SELECTION: [JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.DOWNLOADING_AUDIO: [JobStatus.DOWNLOADING, JobStatus.FAILED],
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
JobStatus.SEPARATING_STAGE1: [JobStatus.SEPARATING_STAGE2, JobStatus.FAILED],
JobStatus.SEPARATING_STAGE2: [JobStatus.AUDIO_COMPLETE, JobStatus.FAILED],
JobStatus.AUDIO_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
JobStatus.TRANSCRIBING: [JobStatus.CORRECTING, JobStatus.FAILED],
JobStatus.CORRECTING: [JobStatus.LYRICS_COMPLETE, JobStatus.FAILED],
JobStatus.LYRICS_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
JobStatus.GENERATING_SCREENS: [JobStatus.APPLYING_PADDING, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
JobStatus.APPLYING_PADDING: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
JobStatus.AWAITING_REVIEW: [JobStatus.IN_REVIEW, JobStatus.REVIEW_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.IN_REVIEW: [JobStatus.REVIEW_COMPLETE, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
JobStatus.REVIEW_COMPLETE: [JobStatus.RENDERING_VIDEO, JobStatus.FAILED],
JobStatus.RENDERING_VIDEO: [JobStatus.INSTRUMENTAL_SELECTED, JobStatus.FAILED],
JobStatus.INSTRUMENTAL_SELECTED: [JobStatus.GENERATING_VIDEO, JobStatus.FAILED],
JobStatus.GENERATING_VIDEO: [JobStatus.ENCODING, JobStatus.FAILED],
JobStatus.ENCODING: [JobStatus.PACKAGING, JobStatus.COMPLETE, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
JobStatus.PACKAGING: [JobStatus.UPLOADING, JobStatus.COMPLETE, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
JobStatus.UPLOADING: [JobStatus.NOTIFYING, JobStatus.COMPLETE, JobStatus.FAILED],
JobStatus.NOTIFYING: [JobStatus.COMPLETE, JobStatus.FAILED],
}
def generate_mermaid() -> str:
"""
Generate a Mermaid state diagram from STATE_TRANSITIONS.
Returns:
Mermaid diagram as a string
"""
lines = ["stateDiagram-v2"]
lines.append(" direction TB")
lines.append("")
# Group states by category for better visualization
categories = {
"Initial": [JobStatus.PENDING],
"Audio Search": [
JobStatus.SEARCHING_AUDIO,
JobStatus.AWAITING_AUDIO_SELECTION,
JobStatus.DOWNLOADING_AUDIO,
],
"Processing": [
JobStatus.DOWNLOADING,
JobStatus.SEPARATING_STAGE1,
JobStatus.SEPARATING_STAGE2,
JobStatus.AUDIO_COMPLETE,
JobStatus.TRANSCRIBING,
JobStatus.CORRECTING,
JobStatus.LYRICS_COMPLETE,
],
"Screens": [
JobStatus.GENERATING_SCREENS,
JobStatus.APPLYING_PADDING,
],
"Review": [
JobStatus.AWAITING_REVIEW,
JobStatus.IN_REVIEW,
JobStatus.REVIEW_COMPLETE,
],
"Video": [
JobStatus.RENDERING_VIDEO,
JobStatus.INSTRUMENTAL_SELECTED,
JobStatus.GENERATING_VIDEO,
JobStatus.ENCODING,
JobStatus.PACKAGING,
],
"Distribution": [
JobStatus.UPLOADING,
JobStatus.NOTIFYING,
],
"Terminal": [
JobStatus.COMPLETE,
JobStatus.PREP_COMPLETE,
JobStatus.FAILED,
JobStatus.CANCELLED,
],
}
# Add state definitions with notes for key states
lines.append(" %% State definitions")
key_states = {
JobStatus.PENDING: "Job created",
JobStatus.AWAITING_AUDIO_SELECTION: "⚠️ USER ACTION",
JobStatus.AWAITING_REVIEW: "⚠️ USER ACTION",
JobStatus.COMPLETE: "✅ Success",
JobStatus.FAILED: "❌ Error",
}
for status, note in key_states.items():
lines.append(f" {status.value}: {status.value}")
lines.append(f" note right of {status.value}: {note}")
lines.append("")
lines.append(" %% Transitions")
# Generate transitions
for from_status, to_statuses in STATE_TRANSITIONS.items():
for to_status in to_statuses:
# Skip deprecated statuses
if to_status in [JobStatus.QUEUED, JobStatus.PROCESSING, JobStatus.ERROR,
JobStatus.READY_FOR_FINALIZATION, JobStatus.FINALIZING]:
continue
if from_status in [JobStatus.QUEUED, JobStatus.PROCESSING, JobStatus.ERROR,
JobStatus.READY_FOR_FINALIZATION, JobStatus.FINALIZING]:
continue
lines.append(f" {from_status.value} --> {to_status.value}")
# Add start and end markers
lines.append("")
lines.append(" %% Start and end")
lines.append(" [*] --> pending")
lines.append(" complete --> [*]")
lines.append(" prep_complete --> [*]")
lines.append(" failed --> [*]")
lines.append(" cancelled --> [*]")
return "\n".join(lines)
def generate_markdown_section() -> str:
"""
Generate a markdown section with the state diagram.
Returns:
Markdown string with the diagram
"""
diagram = generate_mermaid()
return f"""## Job State Machine
The following diagram shows all valid state transitions for a karaoke generation job.
Key states:
- **PENDING**: Initial state when job is created
- **AWAITING_AUDIO_SELECTION**: Human selects audio source (for artist+title search)
- **AWAITING_REVIEW**: Human reviews lyrics and selects instrumental
- **COMPLETE**: All processing finished successfully
```mermaid
{diagram}
```
### Parallel Processing
Jobs use parallel processing for audio separation and lyrics transcription:
1. **DOWNLOADING** triggers both:
- Audio worker → SEPARATING_STAGE1 → SEPARATING_STAGE2 → AUDIO_COMPLETE
- Lyrics worker → TRANSCRIBING → CORRECTING → LYRICS_COMPLETE
2. When BOTH complete, the screens worker is triggered automatically.
### Human Review Points
Jobs wait for human input at these points:
- **AWAITING_AUDIO_SELECTION**: User selects audio source from search results
- **AWAITING_REVIEW**: User reviews/corrects lyrics AND selects instrumental
### Error Handling
- Any state can transition to **FAILED** on unrecoverable errors
- **CANCELLED** is set when user cancels the job
"""
def main():
parser = argparse.ArgumentParser(description="Generate state machine diagram")
parser.add_argument(
"--update-docs",
action="store_true",
help="Update ARCHITECTURE.md with the diagram"
)
parser.add_argument(
"--markdown",
action="store_true",
help="Output markdown section instead of raw Mermaid"
)
args = parser.parse_args()
if args.update_docs:
# TODO: Implement auto-update of ARCHITECTURE.md
print("Auto-update of ARCHITECTURE.md not yet implemented")
print("Please copy the output below to the appropriate section:")
print()
print(generate_markdown_section())
elif args.markdown:
print(generate_markdown_section())
else:
print(generate_mermaid())
if __name__ == "__main__":
main()
+1 -73
View File
@@ -5,7 +5,6 @@ This service wraps the shared karaoke_gen.instrumental_review module
to provide GCS-integrated audio analysis capabilities.
"""
import json
import logging
import os
import tempfile
@@ -188,78 +187,7 @@ class AudioAnalysisService:
)
return amplitudes, duration
def cache_waveform_data(
self,
gcs_audio_path: str,
job_id: str,
cache_gcs_path: str,
num_points: int = 1000,
) -> tuple[list[float], float]:
"""
Generate waveform data and cache it as JSON in GCS.
Args:
gcs_audio_path: Path to the audio file in GCS
job_id: Job ID for logging
cache_gcs_path: GCS path to store the cached waveform JSON
num_points: Number of data points to generate
Returns:
Tuple of (amplitude_values, duration_seconds)
"""
logger.info(f"[{job_id}] Generating and caching waveform data: {gcs_audio_path}")
amplitudes, duration = self.get_waveform_data(
gcs_audio_path=gcs_audio_path,
job_id=job_id,
num_points=num_points,
)
cache_data = {
"amplitudes": list(amplitudes),
"duration_seconds": duration,
"num_points": num_points,
"source_gcs_path": gcs_audio_path,
}
self.storage_service.upload_json(cache_gcs_path, cache_data)
logger.info(f"[{job_id}] Waveform data cached at {cache_gcs_path}")
return amplitudes, duration
def load_cached_waveform(
self,
cache_gcs_path: str,
) -> Optional[tuple[list[float], float]]:
"""
Load cached waveform data from GCS.
Args:
cache_gcs_path: GCS path to the cached waveform JSON
Returns:
Tuple of (amplitude_values, duration_seconds) or None if not found
"""
try:
if not self.storage_service.file_exists(cache_gcs_path):
return None
with tempfile.TemporaryDirectory() as temp_dir:
local_path = os.path.join(temp_dir, "waveform.json")
self.storage_service.download_file(cache_gcs_path, local_path)
with open(local_path) as f:
data = json.load(f)
amplitudes = data.get("amplitudes", [])
duration = data.get("duration_seconds", 0.0)
if not amplitudes:
return None
return amplitudes, duration
except Exception as e:
logger.warning(f"Failed to load cached waveform from {cache_gcs_path}: {e}")
return None
def generate_waveform_with_mutes(
self,
gcs_audio_path: str,
-180
View File
@@ -1,180 +0,0 @@
"""
Audio editing service — server-side FFmpeg operations for trim, cut, mute, join.
All operations are lossless (FLAC in, FLAC out). Playback URLs use OGG Opus
via the existing AudioTranscodingService.
"""
import logging
import os
import json
import subprocess
import tempfile
from dataclasses import dataclass, asdict
from backend.services.storage_service import StorageService
logger = logging.getLogger(__name__)
@dataclass
class AudioMetadata:
duration_seconds: float
sample_rate: int
channels: int
format: str
file_size_bytes: int
class AudioEditService:
"""Server-side audio editing via FFmpeg."""
def __init__(self, storage_service: StorageService | None = None):
self.storage = storage_service or StorageService()
def get_metadata(self, audio_path: str) -> AudioMetadata:
"""Get audio metadata from a local file using ffprobe."""
cmd = [
"ffprobe", "-v", "quiet", "-print_format", "json",
"-show_format", "-show_streams", audio_path,
]
result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
if result.returncode != 0:
raise RuntimeError(f"ffprobe failed: {result.stderr}")
info = json.loads(result.stdout)
fmt = info.get("format", {})
streams = info.get("streams", [])
audio_stream = next((s for s in streams if s.get("codec_type") == "audio"), {})
return AudioMetadata(
duration_seconds=float(fmt.get("duration", 0)),
sample_rate=int(audio_stream.get("sample_rate", 44100)),
channels=int(audio_stream.get("channels", 2)),
format=fmt.get("format_name", "unknown"),
file_size_bytes=int(fmt.get("size", 0)),
)
def get_metadata_from_gcs(self, gcs_path: str) -> AudioMetadata:
"""Download from GCS and get metadata."""
with tempfile.TemporaryDirectory() as temp_dir:
local_path = os.path.join(temp_dir, "audio")
self.storage.download_file(gcs_path, local_path)
return self.get_metadata(local_path)
def trim_start(self, input_path: str, end_seconds: float, output_path: str) -> AudioMetadata:
"""Remove audio from 0 to end_seconds (skip the first N seconds)."""
self._run_ffmpeg([
"-ss", str(end_seconds),
"-i", input_path,
"-c:a", "flac",
output_path,
])
return self.get_metadata(output_path)
def trim_end(self, input_path: str, start_seconds: float, output_path: str) -> AudioMetadata:
"""Keep audio from 0 to start_seconds (remove from start_seconds to end)."""
self._run_ffmpeg([
"-i", input_path,
"-t", str(start_seconds),
"-c:a", "flac",
output_path,
])
return self.get_metadata(output_path)
def cut_region(self, input_path: str, start: float, end: float, output_path: str) -> AudioMetadata:
"""Remove a region from start to end, joining the remaining parts."""
self._run_ffmpeg([
"-i", input_path,
"-filter_complex",
f"[0]atrim=0:{start},asetpts=PTS-STARTPTS[a];"
f"[0]atrim={end},asetpts=PTS-STARTPTS[b];"
f"[a][b]concat=n=2:v=0:a=1[out]",
"-map", "[out]",
"-c:a", "flac",
output_path,
])
return self.get_metadata(output_path)
def mute_region(self, input_path: str, start: float, end: float, output_path: str) -> AudioMetadata:
"""Silence a region (preserve duration)."""
self._run_ffmpeg([
"-i", input_path,
"-af", f"volume=enable='between(t,{start},{end})':volume=0",
"-c:a", "flac",
output_path,
])
return self.get_metadata(output_path)
def join_audio(self, input_path: str, other_path: str, position: str, output_path: str) -> AudioMetadata:
"""Join two audio files. position: 'start' (prepend other) or 'end' (append other)."""
if position == "start":
first, second = other_path, input_path
else:
first, second = input_path, other_path
self._run_ffmpeg([
"-i", first,
"-i", second,
"-filter_complex", "[0:a][1:a]concat=n=2:v=0:a=1[out]",
"-map", "[out]",
"-c:a", "flac",
output_path,
])
return self.get_metadata(output_path)
def apply_edit(
self,
input_gcs_path: str,
operation: str,
params: dict,
output_gcs_path: str,
job_id: str,
) -> tuple[AudioMetadata, str]:
"""
Apply an edit operation to an audio file in GCS.
Returns (metadata, output_gcs_path).
"""
with tempfile.TemporaryDirectory() as temp_dir:
local_input = os.path.join(temp_dir, "input.flac")
local_output = os.path.join(temp_dir, "output.flac")
# Download current audio
self.storage.download_file(input_gcs_path, local_input)
# Apply the operation
if operation == "trim_start":
metadata = self.trim_start(local_input, params["end_seconds"], local_output)
elif operation == "trim_end":
metadata = self.trim_end(local_input, params["start_seconds"], local_output)
elif operation == "cut":
metadata = self.cut_region(
local_input, params["start_seconds"], params["end_seconds"], local_output
)
elif operation == "mute":
metadata = self.mute_region(
local_input, params["start_seconds"], params["end_seconds"], local_output
)
elif operation in ("join_start", "join_end"):
# Download the upload file
upload_gcs_path = params["upload_gcs_path"]
local_other = os.path.join(temp_dir, "other.flac")
self.storage.download_file(upload_gcs_path, local_other)
position = "start" if operation == "join_start" else "end"
metadata = self.join_audio(local_input, local_other, position, local_output)
else:
raise ValueError(f"Unknown operation: {operation}")
# Upload edited audio
self.storage.upload_file(local_output, output_gcs_path)
logger.info(f"[{job_id}] Applied {operation}, uploaded to {output_gcs_path}")
return metadata, output_gcs_path
def _run_ffmpeg(self, args: list[str]) -> None:
"""Run an ffmpeg command with standard options."""
cmd = ["ffmpeg", "-hide_banner", "-loglevel", "error", "-y"] + args
result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
if result.returncode != 0:
raise RuntimeError(f"ffmpeg failed (exit {result.returncode}): {result.stderr}")
+9 -12
View File
@@ -612,13 +612,13 @@ class AudioSearchService:
async def search_async(self, artist: str, title: str) -> List[AudioSearchResult]:
"""
Async version of search for use in async routes.
Runs the sync search in a thread executor to avoid blocking
the event loop.
Note: Currently wraps sync search in executor. Future optimization
could make this fully async.
"""
loop = asyncio.get_running_loop()
loop = asyncio.get_event_loop()
return await loop.run_in_executor(None, self.search, artist, title)
async def download_async(
self,
result_index: int,
@@ -630,10 +630,10 @@ class AudioSearchService:
"""
Async version of download for use in async routes.
Runs the sync download in a thread executor to avoid blocking
the event loop.
Note: Currently wraps sync download in executor. Future optimization
could make this fully async.
"""
loop = asyncio.get_running_loop()
loop = asyncio.get_event_loop()
return await loop.run_in_executor(
None,
lambda: self.download(result_index, output_dir, output_filename, gcs_path, remote_search_id)
@@ -651,11 +651,8 @@ class AudioSearchService:
) -> AudioDownloadResult:
"""
Async version of download_by_id for use in async routes.
Runs the sync download_by_id in a thread executor to avoid blocking
the event loop.
"""
loop = asyncio.get_running_loop()
loop = asyncio.get_event_loop()
return await loop.run_in_executor(
None,
lambda: self.download_by_id(
@@ -1,181 +0,0 @@
"""
Audio transcoding service for review UI optimization.
Transcodes FLAC audio files to OGG Opus for efficient browser playback.
Original FLAC files are never modified — transcoded versions are cached
in GCS at jobs/{job_id}/review-audio/{filename}.ogg.
Eager transcoding: called from screens_worker before AWAITING_REVIEW
so all files are cached before the user opens the review UI.
Falls back to original FLAC signed URLs if transcoded version is missing.
"""
import asyncio
import hashlib
import logging
import os
import subprocess
import tempfile
from pathlib import Path
from typing import Optional
from backend.services.storage_service import StorageService
logger = logging.getLogger(__name__)
class AudioTranscodingService:
"""
Transcodes audio files to OGG Opus and caches them in GCS.
Used by the review UI to serve ~3 MB OGG files instead of 35+ MB FLACs.
"""
def __init__(self, storage_service: Optional[StorageService] = None):
self.storage = storage_service or StorageService()
def _get_cache_path(self, source_gcs_path: str) -> str:
"""
Derive the cache path for a transcoded file.
Examples:
jobs/abc123/input/song.flac -> jobs/abc123/review-audio/song.ogg
jobs/abc123/stems/instrumental_clean.flac -> jobs/abc123/review-audio/instrumental_clean.ogg
"""
parts = Path(source_gcs_path).parts
# Find "jobs/{job_id}" prefix
try:
jobs_idx = list(parts).index("jobs")
job_id = parts[jobs_idx + 1]
except (ValueError, IndexError):
# Fallback: use a hash of the full path to avoid collisions
job_id = "_" + hashlib.sha256(source_gcs_path.encode()).hexdigest()[:12]
filename = Path(source_gcs_path).stem + ".ogg"
return f"jobs/{job_id}/review-audio/{filename}"
def _transcode_and_upload(self, source_gcs_path: str, cache_path: str) -> str:
"""
Download FLAC from GCS, transcode to OGG Opus, upload to cache path.
Returns the cache GCS path on success.
Raises on failure.
"""
with tempfile.TemporaryDirectory() as temp_dir:
local_input = os.path.join(temp_dir, "input.flac")
local_output = os.path.join(temp_dir, "output.ogg")
# Download source
self.storage.download_file(source_gcs_path, local_input)
# Transcode: FLAC -> OGG Opus 128kbps
cmd = [
"ffmpeg", "-hide_banner", "-loglevel", "error",
"-y",
"-i", local_input,
"-c:a", "libopus",
"-b:a", "128k",
"-vn",
local_output,
]
result = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
if result.returncode != 0:
raise RuntimeError(
f"ffmpeg transcoding failed (exit {result.returncode}): {result.stderr}"
)
# Upload transcoded file
self.storage.upload_file(local_output, cache_path)
return cache_path
def transcode_if_needed(self, source_gcs_path: str) -> str:
"""
Return the GCS path of the transcoded OGG file, transcoding if not cached.
Returns the cache path (always an OGG file in review-audio/).
Raises if transcoding fails.
"""
cache_path = self._get_cache_path(source_gcs_path)
if self.storage.file_exists(cache_path):
logger.debug(f"Cache hit: {cache_path}")
return cache_path
logger.info(f"Cache miss, transcoding: {source_gcs_path} -> {cache_path}")
return self._transcode_and_upload(source_gcs_path, cache_path)
def get_review_audio_url(
self, source_gcs_path: str, expiration_minutes: int = 120
) -> str:
"""
Get a signed URL for the transcoded audio, falling back to original FLAC.
"""
try:
cache_path = self.transcode_if_needed(source_gcs_path)
return self.storage.generate_signed_url(cache_path, expiration_minutes)
except Exception as e:
# Source may be deleted but cache from earlier transcode persists
try:
cache_path = self._get_cache_path(source_gcs_path)
if self.storage.file_exists(cache_path):
logger.info(f"Source gone but cache exists, serving: {cache_path}")
return self.storage.generate_signed_url(cache_path, expiration_minutes)
except Exception:
pass # Fall through to FLAC fallback below
logger.warning(
f"Transcoding failed for {source_gcs_path}, falling back to FLAC: {e}"
)
return self.storage.generate_signed_url(source_gcs_path, expiration_minutes)
async def get_review_audio_url_async(
self, source_gcs_path: str, expiration_minutes: int = 120
) -> str:
"""Async wrapper around get_review_audio_url via asyncio.to_thread."""
return await asyncio.to_thread(
self.get_review_audio_url, source_gcs_path, expiration_minutes
)
def prepare_review_audio_for_job(self, job) -> list[str]:
"""
Transcode all review audio files for a job (eager transcoding).
Called from screens_worker before AWAITING_REVIEW transition
and from the admin backfill endpoint.
Returns list of cache paths that were transcoded.
"""
job_id = job.job_id
transcoded = []
# Collect all audio paths to transcode
paths_to_transcode = []
# Main input audio
if job.input_media_gcs_path:
paths_to_transcode.append(("input", job.input_media_gcs_path))
# Stems
stems = job.file_urls.get("stems", {})
for stem_key in ["instrumental_clean", "instrumental_with_backing", "backing_vocals"]:
path = stems.get(stem_key)
if path:
paths_to_transcode.append((stem_key, path))
for label, source_path in paths_to_transcode:
try:
cache_path = self.transcode_if_needed(source_path)
transcoded.append(cache_path)
logger.info(f"[{job_id}] Transcoded {label}: {cache_path}")
except Exception as e:
# Non-fatal: review UI will fall back to FLAC
logger.warning(
f"[{job_id}] Failed to transcode {label} ({source_path}): {e}"
)
logger.info(
f"[{job_id}] Review audio preparation complete: "
f"{len(transcoded)}/{len(paths_to_transcode)} files transcoded"
)
return transcoded
+38 -92
View File
@@ -3,7 +3,6 @@ Authentication service for karaoke backend.
Provides token-based authentication with support for:
- Admin tokens (hardcoded in environment)
- Google OIDC tokens (from Cloud Scheduler / Cloud Tasks service accounts)
- Admin by email domain (@nomadkaraoke.com)
- User tokens (stored in Firestore auth_tokens collection)
- Session tokens (stored in Firestore sessions collection, from magic link auth)
@@ -20,9 +19,6 @@ from enum import Enum
from typing import Optional, Tuple, Dict, Any, List
from datetime import datetime
from google.auth.transport import requests as google_auth_requests
from google.oauth2 import id_token as google_id_token
from backend.services.firestore_service import FirestoreService
from backend.config import get_settings
from backend.models.user import UserRole
@@ -53,7 +49,6 @@ class AuthResult:
user_email: Optional[str] = None # Email if authenticated via session/API key
is_admin: bool = False # True if admin token or admin email domain
api_key_id: Optional[str] = None # API key ID if authenticated via API key
tenant_id: Optional[str] = None # Tenant scope from token or session
def __iter__(self):
"""Allow unpacking as tuple for backward compatibility."""
@@ -76,7 +71,6 @@ class AuthService:
self.firestore = FirestoreService()
self.settings = get_settings()
self._init_admin_tokens()
self._scheduler_service_account = self.settings.scheduler_service_account
def _init_admin_tokens(self):
"""Load admin tokens from environment variables."""
@@ -88,56 +82,6 @@ class AuthService:
logger.info(f"Loaded {len(self.admin_tokens)} admin token(s)")
else:
logger.warning("No admin tokens configured! Set ADMIN_TOKENS environment variable.")
def _validate_oidc_token(self, token: str) -> Optional[AuthResult]:
"""
Validate a Google OIDC token (from Cloud Scheduler or Cloud Tasks).
Returns AuthResult if token is a valid Google OIDC JWT from an allowed
service account, None if the token is not a JWT or validation fails.
"""
# Quick check: JWTs have 3 dot-separated parts
if token.count('.') != 2:
return None
try:
# Verify the token signature and claims with Google
# audience=None skips audience check — Cloud Scheduler sets audience
# to the endpoint URL, which varies per endpoint
claims = google_id_token.verify_oauth2_token(
token,
google_auth_requests.Request(),
audience=None,
)
token_email = claims.get("email", "")
# Only accept tokens from the configured service account
if token_email != self._scheduler_service_account:
logger.warning(
f"OIDC token from unexpected service account: {token_email} "
f"(expected {self._scheduler_service_account})"
)
return None
logger.info(f"OIDC token validated for service account: {token_email}")
return AuthResult(
is_valid=True,
user_type=UserType.ADMIN,
remaining_uses=-1,
message="OIDC service account access granted",
user_email=token_email,
is_admin=True,
)
except ValueError as e:
# verify_oauth2_token raises ValueError for invalid/expired tokens
logger.debug(f"OIDC token validation failed: {e}")
return None
except Exception as e:
# Network errors fetching Google certs, etc.
logger.warning(f"OIDC token verification error: {e}")
return None
def validate_token(self, token: str) -> Tuple[bool, UserType, int, str]:
"""
@@ -268,12 +212,6 @@ class AuthService:
is_admin=True
)
# Check for Google OIDC tokens (Cloud Scheduler / Cloud Tasks)
# Done before Firestore lookup to avoid wasting a read on JWT strings
oidc_result = self._validate_oidc_token(token)
if oidc_result is not None:
return oidc_result
# Check stored tokens in Firestore (auth_tokens collection)
token_data = self.firestore.get_token(token)
@@ -346,7 +284,6 @@ class AuthService:
# All auth_tokens must have an associated user_email for job ownership
token_user_email = token_data.get("user_email")
api_key_id = token_data.get("api_key_id")
token_tenant_id = token_data.get("tenant_id")
# Require user_email on all auth_tokens (no anonymous token auth)
if not token_user_email:
@@ -370,8 +307,7 @@ class AuthService:
message="Unlimited access granted",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
# LIMITED tokens: check usage count
@@ -384,8 +320,7 @@ class AuthService:
message="Limited token with unlimited uses",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
current_uses = token_data.get("usage_count", 0)
@@ -398,8 +333,7 @@ class AuthService:
remaining_uses=0,
message="Token usage limit exceeded",
user_email=token_user_email,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
return AuthResult(
@@ -409,8 +343,7 @@ class AuthService:
message=f"Limited token: {remaining} uses remaining",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
# STRIPE tokens: check expiration and usage
@@ -424,8 +357,7 @@ class AuthService:
remaining_uses=0,
message="Token has expired",
user_email=token_user_email,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
if max_uses > 0:
@@ -439,8 +371,7 @@ class AuthService:
remaining_uses=0,
message="Token usage limit exceeded",
user_email=token_user_email,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
return AuthResult(
@@ -450,8 +381,7 @@ class AuthService:
message=f"Stripe token: {remaining} uses remaining",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
return AuthResult(
@@ -461,8 +391,7 @@ class AuthService:
message="Stripe access granted",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
# API_KEY tokens
@@ -477,8 +406,7 @@ class AuthService:
remaining_uses=0,
message="API key usage limit exceeded",
user_email=token_user_email,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
return AuthResult(
is_valid=True,
@@ -487,8 +415,7 @@ class AuthService:
message=f"API key valid: {remaining} uses remaining",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
return AuthResult(
@@ -498,8 +425,7 @@ class AuthService:
message="API key access granted",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
return AuthResult(
@@ -511,11 +437,10 @@ class AuthService:
def increment_token_usage(self, token: str, job_id: str) -> bool:
"""
Increment usage count for an auth_token and track the job.
Increment usage count for a token and track the job.
Note: Credit deduction for session-based users is handled in
JobManager.create_job(), not here. This method only tracks usage
for legacy auth_tokens stored in Firestore.
For session tokens (magic link auth), this deducts a credit from the user.
For auth_tokens, this increments the usage count.
Args:
token: The access token
@@ -540,11 +465,32 @@ class AuthService:
if token in self.admin_tokens:
return True
# Only increment usage for auth_tokens in Firestore
# Session tokens (magic link auth) have credits managed by JobManager.create_job()
# Check if this is a session token (STRIPE type from validate_token means it's a session)
# Session tokens are not in auth_tokens collection
token_data = self.firestore.get_token(token)
if not token_data:
return True
# This is a session token - deduct credit from user
from backend.services.user_service import get_user_service
user_service = get_user_service()
# Get the user email from the session
is_valid, user, _ = user_service.validate_session(token)
if not is_valid or not user:
logger.error(f"Session token validation failed during usage increment")
return False
# Deduct one credit
success, new_balance, deduct_message = user_service.deduct_credit(
user.email, job_id, reason="job_creation"
)
if success:
logger.info(f"Deducted credit for user {user.email} (remaining: {new_balance})")
return True
else:
logger.error(f"Failed to deduct credit for user {user.email}: {deduct_message}")
return False
# Regular auth_token - increment usage in Firestore
try:
-225
View File
@@ -1,225 +0,0 @@
"""
Brand code allocation service using Firestore atomic counters.
Replaces the previous Dropbox folder-scanning approach which was susceptible to
TOCTOU race conditions when concurrent jobs read the same folder state.
Uses Firestore transactions to atomically allocate unique brand codes.
Supports recycling numbers when E2E cleanup deletes test brand codes.
"""
import logging
import re
from datetime import datetime, timezone
from typing import Optional
from google.cloud import firestore
from backend.config import settings
logger = logging.getLogger(__name__)
BRAND_CODE_COUNTERS_COLLECTION = "brand_code_counters"
# Singleton instance
_brand_code_service: Optional["BrandCodeService"] = None
class BrandCodeService:
"""Atomic brand code allocation using Firestore counters."""
def __init__(self, db: Optional[firestore.Client] = None):
if db is None:
self.db = firestore.Client(project=settings.google_cloud_project)
else:
self.db = db
def allocate_brand_code(self, prefix: str, dropbox_path: str) -> str:
"""
Atomically allocate the next brand code for a given prefix.
Uses a Firestore transaction to ensure no two concurrent callers
receive the same number. If recycled numbers are available (from
E2E cleanup), the smallest recycled number is reused first.
On first use for a prefix, initializes the counter by scanning
Dropbox to find the current max number.
Args:
prefix: Brand prefix (e.g., "NOMAD" or "NOMADNP")
dropbox_path: Dropbox path for one-time initialization scan
Returns:
Brand code string (e.g., "NOMAD-1264")
"""
doc_ref = self.db.collection(BRAND_CODE_COUNTERS_COLLECTION).document(prefix)
@firestore.transactional
def allocate_in_transaction(transaction):
doc = doc_ref.get(transaction=transaction)
if not doc.exists:
# First use — initialize from Dropbox scan
next_number = self._get_initial_next_number(prefix, dropbox_path)
allocated = next_number
transaction.set(doc_ref, {
"prefix": prefix,
"next_number": next_number + 1,
"recycled": [],
"initialized_at": datetime.now(timezone.utc),
"last_updated": datetime.now(timezone.utc),
})
return allocated
data = doc.to_dict()
recycled = data.get("recycled", [])
if recycled:
# Reuse smallest recycled number
recycled.sort()
allocated = recycled.pop(0)
transaction.update(doc_ref, {
"recycled": recycled,
"last_updated": datetime.now(timezone.utc),
})
else:
# Allocate next sequential number
allocated = data["next_number"]
transaction.update(doc_ref, {
"next_number": allocated + 1,
"last_updated": datetime.now(timezone.utc),
})
return allocated
transaction = self.db.transaction()
number = allocate_in_transaction(transaction)
brand_code = f"{prefix}-{number:04d}"
logger.info(f"Allocated brand code: {brand_code}")
return brand_code
def recycle_brand_code(self, prefix: str, number: int) -> None:
"""
Return a brand code number to the recycled pool.
Called when E2E cleanup deletes a brand code's distribution.
The number will be reused on the next allocation for this prefix.
Args:
prefix: Brand prefix (e.g., "NOMAD")
number: The numeric portion to recycle (e.g., 1234)
"""
doc_ref = self.db.collection(BRAND_CODE_COUNTERS_COLLECTION).document(prefix)
@firestore.transactional
def recycle_in_transaction(transaction):
doc = doc_ref.get(transaction=transaction)
if not doc.exists:
# Counter doesn't exist yet — nothing to recycle into.
# The number will naturally be below next_number when
# the counter is initialized, so it would create a gap.
# Just log and skip.
logger.warning(
f"Cannot recycle {prefix}-{number:04d}: counter doc doesn't exist"
)
return
data = doc.to_dict()
next_number = data.get("next_number", 0)
if number <= 0 or number >= next_number:
logger.warning(
f"Cannot recycle {prefix}-{number:04d}: "
f"invalid (next_number={next_number})"
)
return
recycled = data.get("recycled", [])
if number not in recycled:
recycled.append(number)
transaction.update(doc_ref, {
"recycled": recycled,
"last_updated": datetime.now(timezone.utc),
})
logger.info(f"Recycled brand code number: {prefix}-{number:04d}")
else:
logger.warning(f"Number {number} already in recycled pool for {prefix}")
transaction = self.db.transaction()
recycle_in_transaction(transaction)
def _get_initial_next_number(self, prefix: str, dropbox_path: str) -> int:
"""
Scan Dropbox once to determine the starting next_number.
This is only called when the counter document doesn't exist yet
(one-time migration from the old Dropbox-scanning approach).
Args:
prefix: Brand prefix to scan for
dropbox_path: Dropbox path containing organized folders
Returns:
The next number to allocate (max existing + 1, minimum 1)
"""
try:
from backend.services.dropbox_service import get_dropbox_service
dropbox = get_dropbox_service()
if not dropbox.is_configured:
logger.warning(
f"Dropbox not configured for initialization of {prefix} counter, starting at 1"
)
return 1
folders = dropbox.list_folders(dropbox_path)
pattern = re.compile(rf"^{re.escape(prefix)}-(\d{{4}})")
existing_nums: set[int] = set()
for folder in folders:
match = pattern.match(folder)
if match:
existing_nums.add(int(match.group(1)))
if existing_nums:
next_num = max(existing_nums) + 1
logger.info(
f"Initialized {prefix} counter from Dropbox: "
f"max existing={max(existing_nums)}, next_number={next_num}"
)
return next_num
logger.info(f"No existing {prefix} codes found in Dropbox, starting at 1")
return 1
except Exception as e:
logger.error(f"Failed to initialize {prefix} counter from Dropbox: {e}")
raise
@staticmethod
def parse_brand_code(brand_code: str) -> tuple[str, int]:
"""
Parse a brand code string into prefix and number.
Args:
brand_code: e.g., "NOMAD-1234" or "NOMADNP-0012"
Returns:
Tuple of (prefix, number), e.g., ("NOMAD", 1234)
Raises:
ValueError: If brand_code doesn't match expected format
"""
match = re.match(r"^([A-Z]+)-(\d+)$", brand_code)
if not match:
raise ValueError(f"Invalid brand code format: {brand_code}")
return match.group(1), int(match.group(2))
def get_brand_code_service() -> BrandCodeService:
"""Get or create the singleton BrandCodeService instance."""
global _brand_code_service
if _brand_code_service is None:
_brand_code_service = BrandCodeService()
return _brand_code_service
-105
View File
@@ -1,105 +0,0 @@
"""
Proxy service for karaoke-decide catalog API.
Forwards artist and track search requests to the karaoke-decide backend,
providing autocomplete data from MusicBrainz + Spotify catalogs.
"""
import logging
import time
from typing import Any
import httpx
from backend.config import settings
logger = logging.getLogger(__name__)
# In-memory TTL cache: { cache_key: (expiry_timestamp, data) }
_cache: dict[str, tuple[float, Any]] = {}
CACHE_TTL_SECONDS = 300 # 5 minutes
KARAOKE_DECIDE_BASE_URL = "https://decide.nomadkaraoke.com"
REQUEST_TIMEOUT = 10 # seconds
def _get_base_url() -> str:
"""Get the karaoke-decide API base URL from config or default."""
return getattr(settings, "karaoke_decide_api_url", None) or KARAOKE_DECIDE_BASE_URL
def _cache_get(key: str) -> Any | None:
"""Get a value from cache if it exists and hasn't expired."""
if key in _cache:
expiry, data = _cache[key]
if time.monotonic() < expiry:
return data
del _cache[key]
return None
def _cache_set(key: str, data: Any, ttl: float = CACHE_TTL_SECONDS) -> None:
"""Store a value in cache with TTL."""
_cache[key] = (time.monotonic() + ttl, data)
async def search_artists(query: str, limit: int = 10) -> list[dict]:
"""
Search for artists via karaoke-decide catalog API.
Returns list of artist dicts with canonical MusicBrainz names.
Falls back to empty list on any error.
"""
cache_key = f"artists:{query.lower()}:{limit}"
cached = _cache_get(cache_key)
if cached is not None:
return cached
base_url = _get_base_url()
url = f"{base_url}/api/catalog/artists"
params = {"q": query, "limit": limit}
try:
async with httpx.AsyncClient(timeout=REQUEST_TIMEOUT) as client:
resp = await client.get(url, params=params)
resp.raise_for_status()
data = resp.json()
# karaoke-decide wraps results: {"artists": [...], "total": N}
results = data.get("artists", data) if isinstance(data, dict) else data
_cache_set(cache_key, results)
return results
except Exception as e:
logger.warning(f"Failed to search artists from karaoke-decide: {e}")
return []
async def search_tracks(query: str, artist: str | None = None, limit: int = 10) -> list[dict]:
"""
Search for tracks via karaoke-decide catalog API.
Returns list of track dicts with canonical Spotify/MusicBrainz names.
Falls back to empty list on any error.
"""
cache_key = f"tracks:{query.lower()}:{(artist or '').lower()}:{limit}"
cached = _cache_get(cache_key)
if cached is not None:
return cached
base_url = _get_base_url()
url = f"{base_url}/api/catalog/tracks"
params: dict[str, Any] = {"q": query, "limit": limit}
if artist:
params["artist"] = artist
try:
async with httpx.AsyncClient(timeout=REQUEST_TIMEOUT) as client:
resp = await client.get(url, params=params)
resp.raise_for_status()
data = resp.json()
# karaoke-decide wraps results: {"tracks": [...], "total": N}
results = data.get("tracks", data) if isinstance(data, dict) else data
_cache_set(cache_key, results)
return results
except Exception as e:
logger.warning(f"Failed to search tracks from karaoke-decide: {e}")
return []
@@ -1,441 +0,0 @@
"""
AI-powered credit evaluation service.
Evaluates whether to grant free credits (welcome or feedback) by collecting
abuse signals and asking Gemini for a grant/deny decision. Fail-open: if
the AI call fails for any reason, credits are granted.
"""
import json
import logging
import uuid
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Optional
from backend.config import get_settings
logger = logging.getLogger(__name__)
CREDIT_EVALUATIONS_COLLECTION = "credit_evaluations"
@dataclass
class CreditEvaluation:
"""Result of an AI credit evaluation."""
decision: str # "grant" or "deny"
reasoning: str
confidence: float
error: Optional[str] = None # Set if evaluation failed (decision defaults to "grant")
signals: Optional[dict] = None # Collected abuse signals (for admin review emails)
SYSTEM_PROMPT = """You are an anti-abuse evaluator for Nomad Karaoke, a web service that creates professional karaoke videos. Each job costs the service real money in API credits and cloud compute.
New users get 1 free welcome credit to try the service. Users who complete 2 jobs and submit feedback get 1 more credit. Some users abuse this by creating multiple accounts to get unlimited free karaoke generation.
Your job: Given the signals below, decide whether to GRANT or DENY free credits to this user.
## Decision Guidelines
**DENY if any of these are true:**
- The user's device fingerprint matches other accounts (strong signal of multi-accounting)
- The user's signup IP matches other accounts AND those accounts also have suspicious patterns (e.g., no spend, same user agent)
- There's a clear pattern of account creation from the same device/network to farm free credits
**GRANT if:**
- This appears to be a genuine new user with no suspicious correlations
- The IP match is explainable (e.g., different fingerprints suggest different actual users on a shared network like a school or office)
- There's insufficient evidence to confidently flag abuse
**When in doubt, GRANT.** False positives (blocking real users) are worse than false negatives (letting an occasional abuser through). We'd rather lose a few dollars than alienate a real customer.
## Response Format
Respond with ONLY a JSON object, no other text:
{"decision": "grant" or "deny", "reasoning": "Brief explanation", "confidence": 0.0 to 1.0}
"""
def _build_user_prompt(
email: str,
grant_type: str,
user_data: dict,
fingerprint_matches: list,
ip_matches: list,
ip_geo: Optional[dict],
recent_signups_ip: int,
recent_signups_fp: int,
user_agent: Optional[str],
feedback_content: Optional[dict] = None,
) -> str:
"""Build the user prompt with all abuse signals."""
lines = [
f"## Credit Grant Request",
f"- **Email:** {email}",
f"- **Grant type:** {grant_type} credits",
f"- **User's device fingerprint:** {user_data.get('device_fingerprint', 'unknown')}",
f"- **User's signup IP:** {user_data.get('signup_ip', 'unknown')}",
f"- **User agent:** {user_agent or 'unknown'}",
f"- **Account created:** {user_data.get('created_at', 'unknown')}",
f"- **Jobs created:** {user_data.get('total_jobs_created', 0)}",
f"- **Jobs completed:** {user_data.get('total_jobs_completed', 0)}",
f"- **Total spent (real money):** ${user_data.get('total_spent', 0) / 100:.2f}",
f"- **Current credits:** {user_data.get('credits', 0)}",
"",
]
# IP geolocation
if ip_geo and ip_geo.get("status") == "success":
lines.extend([
f"## IP Geolocation",
f"- **Country:** {ip_geo.get('country', 'unknown')} ({ip_geo.get('country_code', '')})",
f"- **City/Region:** {ip_geo.get('city', '')}, {ip_geo.get('region', '')}",
f"- **ISP:** {ip_geo.get('isp', 'unknown')}",
f"- **Organization:** {ip_geo.get('org', 'unknown')}",
f"- **ASN:** {ip_geo.get('as_number', '')} {ip_geo.get('as_name', '')}",
"",
])
# Recent signup patterns
lines.extend([
f"## Recent Signup Patterns (last 24 hours)",
f"- **New accounts from same IP:** {recent_signups_ip}",
f"- **New accounts from same fingerprint:** {recent_signups_fp}",
"",
])
# Fingerprint correlations
if fingerprint_matches:
lines.append("## Other Accounts with Same Device Fingerprint (RED FLAG)")
for match in fingerprint_matches:
lines.append(
f"- {match.get('email', '?')} | "
f"Jobs: {match.get('total_jobs_created', 0)} | "
f"Spent: ${match.get('total_spent', 0) / 100:.2f} | "
f"Created: {match.get('created_at', '?')}"
)
lines.append("")
else:
lines.extend(["## Other Accounts with Same Device Fingerprint", "None found.", ""])
# IP correlations
if ip_matches:
lines.append("## Other Accounts from Same Signup IP")
for match in ip_matches:
lines.append(
f"- {match.get('email', '?')} | "
f"Fingerprint: {match.get('device_fingerprint', 'unknown')} | "
f"Jobs: {match.get('total_jobs_created', 0)} | "
f"Spent: ${match.get('total_spent', 0) / 100:.2f} | "
f"Created: {match.get('created_at', '?')}"
)
lines.append("")
else:
lines.extend(["## Other Accounts from Same Signup IP", "None found.", ""])
# Feedback content (if applicable)
if feedback_content:
lines.extend([
"## Feedback Submitted",
f"- **Overall rating:** {feedback_content.get('overall_rating', '?')}/5",
f"- **What went well:** {feedback_content.get('what_went_well', 'N/A')[:200]}",
f"- **What could improve:** {feedback_content.get('what_could_improve', 'N/A')[:200]}",
f"- **Additional comments:** {feedback_content.get('additional_comments', 'N/A')[:200]}",
"",
])
lines.append("Based on these signals, should we GRANT or DENY free credits to this user?")
return "\n".join(lines)
def _parse_gemini_response(response_text: str) -> CreditEvaluation:
"""Parse Gemini's JSON response into a CreditEvaluation."""
try:
# Strip markdown code fences if present
text = response_text.strip()
if text.startswith("```"):
text = text.split("\n", 1)[1] if "\n" in text else text[3:]
if text.endswith("```"):
text = text.rsplit("```", 1)[0]
text = text.strip()
if text.startswith("json"):
text = text[4:].strip()
data = json.loads(text)
decision = data.get("decision", "").lower()
if decision not in ("grant", "deny"):
decision = "pending_review" # fail-closed: unknown decision → manual review
return CreditEvaluation(
decision=decision,
reasoning=str(data.get("reasoning", "No reasoning provided")),
confidence=float(data.get("confidence", 0.5)),
)
except (json.JSONDecodeError, ValueError, TypeError) as e:
logger.warning(f"Failed to parse Gemini response: {e}. Response: {response_text[:200]}")
return CreditEvaluation(
decision="pending_review",
reasoning="Failed to parse AI response — pending manual review",
confidence=0.0,
error=f"Parse error: {e}",
)
class CreditEvaluationService:
"""Evaluates credit grant requests using Gemini AI."""
def __init__(self):
self.settings = get_settings()
def _collect_signals(self, email: str) -> dict:
"""Collect all abuse signals for evaluation."""
from backend.services.user_service import get_user_service
from backend.services.ip_geolocation_service import get_ip_geolocation_service
user_service = get_user_service()
user = user_service.get_user(email)
if not user:
return {
"user_data": {},
"fingerprint_matches": [],
"ip_matches": [],
"ip_geo": None,
"recent_signups_ip": 0,
"recent_signups_fp": 0,
"user_agent": None,
}
user_data = {
"email": user.email,
"device_fingerprint": user.device_fingerprint,
"signup_ip": user.signup_ip,
"credits": user.credits,
"total_jobs_created": user.total_jobs_created,
"total_jobs_completed": user.total_jobs_completed,
"total_spent": user.total_spent,
"created_at": str(user.created_at),
}
# Find other accounts with same fingerprint
fingerprint_matches = []
if user.device_fingerprint:
fp_users = user_service.find_users_by_fingerprint(user.device_fingerprint)
fingerprint_matches = [
{
"email": u.email,
"total_jobs_created": u.total_jobs_created,
"total_spent": u.total_spent,
"created_at": str(u.created_at),
}
for u in fp_users if u.email != email
]
# Find other accounts from same IP
ip_matches = []
if user.signup_ip:
ip_users = user_service.find_users_by_signup_ip(user.signup_ip)
ip_matches = [
{
"email": u.email,
"device_fingerprint": u.device_fingerprint,
"total_jobs_created": u.total_jobs_created,
"total_spent": u.total_spent,
"created_at": str(u.created_at),
}
for u in ip_users if u.email != email
]
# IP geolocation
ip_geo = None
if user.signup_ip:
try:
geo_service = get_ip_geolocation_service()
ip_geo = geo_service.lookup_ip(user.signup_ip)
except Exception:
pass
# Recent signup counts
recent_signups_ip = 0
recent_signups_fp = 0
if user.signup_ip:
recent_signups_ip = user_service.count_recent_signups_from_ip(user.signup_ip)
if user.device_fingerprint:
recent_signups_fp = user_service.count_recent_signups_from_fingerprint(user.device_fingerprint)
# User agent from latest session
user_agent = None
try:
from google.cloud.firestore_v1 import FieldFilter
from google.cloud import firestore as fs
sessions = (
user_service.db.collection("sessions")
.where(filter=FieldFilter("user_email", "==", email))
.order_by("created_at", direction=fs.Query.DESCENDING)
.limit(1)
.stream()
)
for session_doc in sessions:
user_agent = session_doc.to_dict().get("user_agent")
except Exception:
pass
return {
"user_data": user_data,
"fingerprint_matches": fingerprint_matches,
"ip_matches": ip_matches,
"ip_geo": ip_geo,
"recent_signups_ip": recent_signups_ip,
"recent_signups_fp": recent_signups_fp,
"user_agent": user_agent,
}
def _call_gemini(self, prompt: str) -> str:
"""Call Gemini and return the response text."""
from google import genai
client = genai.Client(
vertexai=True,
project=self.settings.google_cloud_project,
location="global",
)
response = client.models.generate_content(
model=self.settings.credit_eval_model,
contents=prompt,
config=genai.types.GenerateContentConfig(
system_instruction=SYSTEM_PROMPT,
),
)
return response.text
def _log_evaluation(
self,
email: str,
grant_type: str,
evaluation: CreditEvaluation,
signals: dict,
):
"""Log the evaluation decision to Firestore for audit trail."""
try:
from google.cloud import firestore as fs
db = fs.Client(project=self.settings.google_cloud_project)
doc_id = str(uuid.uuid4())
db.collection(CREDIT_EVALUATIONS_COLLECTION).document(doc_id).set({
"id": doc_id,
"email": email,
"grant_type": grant_type,
"decision": evaluation.decision,
"reasoning": evaluation.reasoning,
"confidence": evaluation.confidence,
"error": evaluation.error,
"model": self.settings.credit_eval_model,
"signals_snapshot": {
"fingerprint_match_count": len(signals.get("fingerprint_matches", [])),
"ip_match_count": len(signals.get("ip_matches", [])),
"recent_signups_ip": signals.get("recent_signups_ip", 0),
"recent_signups_fp": signals.get("recent_signups_fp", 0),
"user_ip": signals.get("user_data", {}).get("signup_ip"),
"user_fingerprint": signals.get("user_data", {}).get("device_fingerprint"),
},
"evaluated_at": datetime.now(timezone.utc).isoformat(),
})
except Exception:
logger.exception(f"Failed to log credit evaluation for {email}")
def evaluate(
self,
email: str,
grant_type: str,
feedback_content: Optional[dict] = None,
) -> CreditEvaluation:
"""
Evaluate whether to grant free credits.
Args:
email: User's email
grant_type: "welcome" or "feedback"
feedback_content: Feedback form data (for feedback grants only)
Returns:
CreditEvaluation with decision, reasoning, confidence.
Fail-closed: returns "pending_review" on any error (admin notified).
"""
if not self.settings.credit_eval_enabled:
return CreditEvaluation(
decision="grant",
reasoning="Credit evaluation disabled — auto-granting",
confidence=1.0,
)
try:
# Collect signals
signals = self._collect_signals(email)
# Quick-grant: if no correlations found at all, skip AI call
if (
not signals["fingerprint_matches"]
and not signals["ip_matches"]
and signals["recent_signups_ip"] <= 1
and signals["recent_signups_fp"] <= 1
):
evaluation = CreditEvaluation(
decision="grant",
reasoning="No suspicious correlations found — clean user",
confidence=1.0,
signals=signals,
)
self._log_evaluation(email, grant_type, evaluation, signals)
return evaluation
# Build prompt and call Gemini
prompt = _build_user_prompt(
email=email,
grant_type=grant_type,
user_data=signals["user_data"],
fingerprint_matches=signals["fingerprint_matches"],
ip_matches=signals["ip_matches"],
ip_geo=signals["ip_geo"],
recent_signups_ip=signals["recent_signups_ip"],
recent_signups_fp=signals["recent_signups_fp"],
user_agent=signals["user_agent"],
feedback_content=feedback_content,
)
response_text = self._call_gemini(prompt)
evaluation = _parse_gemini_response(response_text)
evaluation.signals = signals
self._log_evaluation(email, grant_type, evaluation, signals)
logger.info(
f"Credit evaluation for {email} ({grant_type}): "
f"{evaluation.decision} (confidence={evaluation.confidence:.2f})"
)
return evaluation
except Exception as e:
logger.exception(f"Credit evaluation failed for {email} — pending manual review (fail-closed)")
# Include whatever signals were collected before the failure
collected_signals = locals().get("signals")
evaluation = CreditEvaluation(
decision="pending_review",
reasoning="Evaluation failed — pending manual review",
confidence=0.0,
error=str(e),
signals=collected_signals,
)
try:
self._log_evaluation(email, grant_type, evaluation, collected_signals or {})
except Exception:
pass
return evaluation
_service_instance: Optional[CreditEvaluationService] = None
def get_credit_evaluation_service() -> CreditEvaluationService:
"""Get or create the singleton credit evaluation service."""
global _service_instance
if _service_instance is None:
_service_instance = CreditEvaluationService()
return _service_instance
@@ -1,129 +0,0 @@
"""
Disposable Domain Sync Service.
Fetches and syncs disposable email domain blocklists from external sources.
"""
import logging
from datetime import datetime, timezone
from typing import Set
import httpx
from backend.services.email_validation_service import (
BLOCKLISTS_COLLECTION,
BLOCKLIST_CONFIG_DOC,
DEFAULT_DISPOSABLE_DOMAINS,
)
logger = logging.getLogger(__name__)
EXTERNAL_BLOCKLIST_URL = (
"https://raw.githubusercontent.com/disposable-email-domains/"
"disposable-email-domains/refs/heads/main/disposable_email_blocklist.conf"
)
FETCH_TIMEOUT_SECONDS = 30
MAX_RESPONSE_BYTES = 2 * 1024 * 1024 # 2 MB
MAX_DOMAIN_COUNT = 50_000
def parse_blocklist_text(text: str) -> Set[str]:
"""Parse newline-delimited blocklist text into a set of lowercase domains."""
domains = set()
for line in text.splitlines():
domain = line.strip().lower()
if domain:
domains.add(domain)
return domains
async def fetch_external_blocklist() -> Set[str]:
"""Fetch the external disposable domain blocklist from GitHub.
Checks Content-Length header and response body size against safety limits.
"""
async with httpx.AsyncClient(timeout=FETCH_TIMEOUT_SECONDS) as client:
response = await client.get(EXTERNAL_BLOCKLIST_URL)
response.raise_for_status()
content_length = response.headers.get("content-length")
if content_length and int(content_length) > MAX_RESPONSE_BYTES:
raise ValueError(
f"Response size {content_length} exceeds maximum {MAX_RESPONSE_BYTES}"
)
text = response.text
if len(text.encode()) > MAX_RESPONSE_BYTES:
raise ValueError(
f"Response body size exceeds maximum {MAX_RESPONSE_BYTES}"
)
domains = parse_blocklist_text(text)
if len(domains) > MAX_DOMAIN_COUNT:
raise ValueError(
f"Domain count {len(domains)} exceeds maximum {MAX_DOMAIN_COUNT}"
)
logger.info(f"Fetched {len(domains)} domains from external blocklist")
return domains
def sync_disposable_domains(db, external_domains: Set[str]) -> dict:
"""
Sync external domains into Firestore.
Handles first-run migration from old disposable_domains field,
and cleans up redundant manual domains on subsequent runs.
Note: This is a synchronous function (Firestore SDK is sync).
Call from async code with `await asyncio.to_thread(sync_disposable_domains, db, domains)`.
Returns a summary dict with counts.
"""
doc_ref = db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
doc = doc_ref.get()
data = doc.to_dict() if doc.exists else {}
result = {"external_count": len(external_domains), "added": 0, "removed": 0, "migrated_to_manual": []}
is_migration = "external_domains" not in data
if is_migration:
old_domains = set(data.get("disposable_domains", [])) | DEFAULT_DISPOSABLE_DOMAINS
manual_only = old_domains - external_domains
result["migrated_to_manual"] = sorted(manual_only)
new_data = {
**data, # preserve all existing fields
"external_domains": sorted(external_domains),
"manual_domains": sorted(manual_only),
"allowlisted_domains": data.get("allowlisted_domains", []),
"last_sync_at": datetime.now(timezone.utc),
"last_sync_count": len(external_domains),
}
new_data.pop("disposable_domains", None) # remove legacy field
doc_ref.set(new_data)
else:
old_external = set(data.get("external_domains", []))
manual_domains = set(data.get("manual_domains", []))
result["added"] = len(external_domains - old_external)
result["removed"] = len(old_external - external_domains)
cleaned_manual = manual_domains - external_domains
new_data = {
**data,
"external_domains": sorted(external_domains),
"manual_domains": sorted(cleaned_manual),
"last_sync_at": datetime.now(timezone.utc),
"last_sync_count": len(external_domains),
}
doc_ref.set(new_data)
logger.info(
f"Sync complete: {result['external_count']} external domains, "
f"{result['added']} added, {result['removed']} removed"
)
return result
-4
View File
@@ -111,10 +111,6 @@ class DropboxService:
"""
Calculate next brand code from existing folders, filling gaps in the sequence.
NOTE: This method is only used for one-time initialization of the Firestore
counter (see brand_code_service.py). All runtime brand code allocation now
goes through BrandCodeService.allocate_brand_code() for atomic safety.
Scans the folder for existing brand codes like "NOMAD-0001" and
returns the first available code, filling any gaps left by deleted outputs.
File diff suppressed because it is too large. Load diff
+200 -410
View File
@@ -2,43 +2,21 @@
Email Validation Service.
Provides email validation, normalization, and blocklist checking
to prevent abuse during enrollment and other flows.
Uses a tiered disposable domain detection strategy:
1. Static blocklist (instant, from Firestore)
2. Verified-clean domain cache (skip API calls for 7 days)
3. DeBounce API (free, <50ms)
4. verifymail.io API (paid, only if DeBounce says clean)
Auto-learns: flagged domains are persisted to manual_domains,
clean domains are cached for 7 days in verified_clean_domains.
to prevent abuse during beta enrollment and other flows.
"""
import hashlib
import logging
import re
from datetime import datetime, timedelta, timezone
from datetime import datetime, timezone
from typing import Optional, Tuple, Set
import httpx
from google.cloud import firestore
from backend.config import settings
logger = logging.getLogger(__name__)
# Module-level HTTP client for connection pooling (external API checks)
_http_client: Optional[httpx.Client] = None
_HTTP_TIMEOUT = 2.0 # seconds per external API call
def _get_http_client() -> httpx.Client:
"""Get or create the module-level HTTP client."""
global _http_client
if _http_client is None or _http_client.is_closed:
_http_client = httpx.Client(timeout=_HTTP_TIMEOUT)
return _http_client
def _mask_email(email: str) -> str:
"""Mask an email address for privacy-safe logging."""
@@ -56,12 +34,164 @@ def _mask_email(email: str) -> str:
BLOCKLISTS_COLLECTION = "blocklists"
BLOCKLIST_CONFIG_DOC = "config"
# Legacy default set — kept as empty set for import compatibility.
# The external sync (disposable_domain_sync_service) replaces this with ~4,800
# domains from the community-curated disposable-email-domains repo.
# The migration path in sync_disposable_domains() references this set to ensure
# no domains are lost during the transition.
DEFAULT_DISPOSABLE_DOMAINS: Set[str] = set()
# Default disposable email domains (can be extended via admin UI)
DEFAULT_DISPOSABLE_DOMAINS = {
# Popular temporary email services
"10minutemail.com",
"10minutemail.net",
"guerrillamail.com",
"guerrillamail.org",
"guerrillamail.net",
"guerrillamail.biz",
"guerrillamailblock.com",
"tempmail.com",
"tempmail.net",
"temp-mail.org",
"temp-mail.io",
"throwawaymail.com",
"mailinator.com",
"mailinator.net",
"mailinator.org",
"mailinator.info",
"maildrop.cc",
"yopmail.com",
"yopmail.fr",
"yopmail.net",
"fakeinbox.com",
"fakemailgenerator.com",
"dispostable.com",
"getairmail.com",
"getnada.com",
"mohmal.com",
"trashmail.com",
"trashmail.net",
"trashmail.org",
"sharklasers.com",
"grr.la",
"guerrillamail.de",
"pokemail.net",
"spam4.me",
"spamgourmet.com",
"mytrashmail.com",
"mailnesia.com",
"mailcatch.com",
"mintemail.com",
"tempr.email",
"tempail.com",
"emailondeck.com",
"incognitomail.com",
"inboxalias.com",
"33mail.com",
"spamex.com",
"spamfree24.org",
"spamspot.com",
"mailnull.com",
"mailsac.com",
"emailfake.com",
"fakemail.fr",
"tempinbox.com",
"throwaway.email",
"burnermail.io",
"anonbox.net",
"anonymbox.com",
"discard.email",
"discardmail.com",
"discardmail.de",
"mailexpire.com",
"mailforspam.com",
"meltmail.com",
"mt2009.com",
"mt2014.com",
"nospam.ze.tc",
"nospamfor.us",
"nowmymail.com",
"receiveee.com",
"safe-mail.net",
"spamavert.com",
"spambob.com",
"spambog.com",
"spambox.us",
"spamcannon.com",
"spamcannon.net",
"spamcon.org",
"spamcorptastic.com",
"spamday.com",
"spamfree.eu",
"spamherelots.com",
"spamhereplease.com",
"spamhole.com",
"spamify.com",
"spaminator.de",
"spamkill.info",
"spaml.com",
"spaml.de",
"spamoff.de",
"spamobox.com",
"spamslicer.com",
"spamstack.net",
"spamthis.co.uk",
"spamthisplease.com",
"supergreatmail.com",
"suremail.info",
"teleworm.us",
"tempemail.co.za",
"tempemail.net",
"tempmailaddress.com",
"tempmailo.com",
"thankyou2010.com",
"thisisnotmyrealemail.com",
"tm.slsrs.ru",
"tmpeml.info",
"trash-mail.at",
"trash-mail.de",
"trash2009.com",
"trashemail.de",
"trashmail.at",
"trashmailer.com",
"wegwerfmail.de",
"wegwerfmail.net",
"wegwerfmail.org",
"wh4f.org",
"willhackforfood.biz",
"willselfdestruct.com",
"xmaily.com",
"xyzfree.net",
"yep.it",
"yogamaven.com",
"yuurok.com",
"zehnminutenmail.de",
"zippymail.info",
# Additional common ones
"mailnator.com",
"bugmenot.com",
"dodgeit.com",
"dodgit.com",
"e4ward.com",
"emailsensei.com",
"hushmail.com",
"jetable.org",
"kasmail.com",
"mailblock.net",
"mailcatch.com",
"mymailoasis.com",
"nervmich.net",
"nervtmansen.de",
"oneoffemail.com",
"pookmail.com",
"shortmail.net",
"sneakemail.com",
"sogetthis.com",
"tempemailaddress.com",
"tempomail.fr",
"temporaryemail.net",
"temporaryforwarding.com",
"temporaryinbox.com",
"thankyou2010.com",
"tyldd.com",
"uggsrock.com",
"veryrealemail.com",
"yourewronghereswhy.com",
}
# Gmail-like domains that support alias normalization
GMAIL_LIKE_DOMAINS = {
@@ -69,47 +199,6 @@ GMAIL_LIKE_DOMAINS = {
"googlemail.com",
}
# Well-known email providers that are never disposable.
# These skip all external API checks for zero-latency validation.
WELL_KNOWN_PROVIDERS = {
"gmail.com",
"googlemail.com",
"yahoo.com",
"yahoo.co.uk",
"yahoo.co.jp",
"yahoo.fr",
"hotmail.com",
"hotmail.co.uk",
"outlook.com",
"outlook.fr",
"live.com",
"live.co.uk",
"msn.com",
"icloud.com",
"me.com",
"mac.com",
"aol.com",
"protonmail.com",
"proton.me",
"zoho.com",
"yandex.com",
"yandex.ru",
"mail.com",
"gmx.com",
"gmx.de",
"fastmail.com",
"hey.com",
"tutanota.com",
"tuta.io",
}
# How long a verified-clean domain result is trusted before re-checking
VERIFIED_CLEAN_TTL = timedelta(days=7)
# External API URLs
DEBOUNCE_API_URL = "https://disposable.debounce.io/"
VERIFYMAIL_API_URL = "https://verifymail.io/api"
class EmailValidationService:
"""
@@ -204,30 +293,17 @@ class EmailValidationService:
if doc.exists:
data = doc.to_dict()
if "external_domains" in data:
# New data model: effective = (external + manual) - allowlisted
external = set(data.get("external_domains", []))
manual = set(data.get("manual_domains", []))
allowlisted = set(data.get("allowlisted_domains", []))
effective_domains = (external | manual) - allowlisted
else:
# Old data model (pre-migration): use disposable_domains + defaults
allowlisted = set()
effective_domains = set(data.get("disposable_domains", [])) | DEFAULT_DISPOSABLE_DOMAINS
config = {
"disposable_domains": effective_domains,
"allowlisted_domains": allowlisted,
"disposable_domains": set(data.get("disposable_domains", [])) | DEFAULT_DISPOSABLE_DOMAINS,
"blocked_emails": set(data.get("blocked_emails", [])),
"blocked_ips": set(data.get("blocked_ips", [])),
"verified_clean_domains": data.get("verified_clean_domains", {}),
}
else:
# Use defaults if no config exists
config = {
"disposable_domains": DEFAULT_DISPOSABLE_DOMAINS.copy(),
"allowlisted_domains": set(),
"blocked_emails": set(),
"blocked_ips": set(),
"verified_clean_domains": {},
}
# Update cache
@@ -240,16 +316,6 @@ class EmailValidationService:
"""
Check if an email uses a disposable domain.
Uses a tiered checking strategy:
0. Well-known providers (gmail, yahoo, etc.) → instant False
1. Static blocklist from Firestore → instant
2. Verified-clean cache (7-day TTL) → skip API calls
3. DeBounce API (free) → fast external check
4. verifymail.io API (paid) → only if DeBounce says clean
Flagged domains are auto-learned to manual_domains.
Clean domains are cached for 7 days.
Args:
email: Email address to check
@@ -260,194 +326,8 @@ class EmailValidationService:
return False
domain = email.lower().split("@")[-1]
# Tier 0: Well-known providers — skip everything
if domain in WELL_KNOWN_PROVIDERS:
return False
# Tier 1: Static blocklist (instant, always available)
config = self.get_blocklist_config()
if domain in config["disposable_domains"]:
return True
# Tier 1.5: Explicitly allowlisted domains skip external API checks.
# Without this, external APIs can override admin allowlist decisions
# (e.g. inbox.testmail.app is a legitimate testing domain we allow).
if domain in config.get("allowlisted_domains", set()):
return False
# Tier 2: Recently verified clean? Skip API calls
if self._is_domain_verified_clean(domain, config):
return False
# Tier 3: DeBounce API (free, fast)
debounce_result = self._check_debounce(email)
if debounce_result is True:
self._auto_learn_domain(domain)
return True
# Tier 4: verifymail.io (paid, only if DeBounce gave definitive clean)
if debounce_result is False:
verifymail_result = self._check_verifymail(email)
if verifymail_result is True:
self._auto_learn_domain(domain)
return True
# Both APIs say clean (or errored) — cache clean result
if debounce_result is False:
self._mark_domain_clean(domain)
return False
def _check_debounce(self, email: str) -> Optional[bool]:
"""
Check email against DeBounce disposable email API.
Returns:
True if disposable, False if clean, None if error/timeout.
"""
try:
client = _get_http_client()
response = client.get(DEBOUNCE_API_URL, params={"email": email})
response.raise_for_status()
data = response.json()
result = str(data.get("disposable", "")).lower() == "true"
logger.info(
f"DeBounce check for {_mask_email(email)}: "
f"disposable={result}"
)
return result
except httpx.TimeoutException:
logger.warning(f"DeBounce API timeout for {_mask_email(email)}")
return None
except Exception as e:
logger.warning(f"DeBounce API error for {_mask_email(email)}: {e}")
return None
def _check_verifymail(self, email: str) -> Optional[bool]:
"""
Check email against verifymail.io API.
Only called when DeBounce says clean (to minimize paid API usage).
Returns:
True if disposable/blocked, False if clean, None if error/timeout
or API key not configured.
"""
api_key = settings.get_secret("verifymail-api-key")
if not api_key:
logger.debug("verifymail.io API key not configured, skipping")
return None
try:
client = _get_http_client()
response = client.get(
f"{VERIFYMAIL_API_URL}/{email}",
params={"key": api_key},
)
response.raise_for_status()
data = response.json()
# verifymail.io returns both "disposable" and "block" flags
is_disposable = data.get("disposable") is True
is_blocked = data.get("block") is True
result = is_disposable or is_blocked
logger.info(
f"verifymail.io check for {_mask_email(email)}: "
f"disposable={is_disposable}, block={is_blocked}"
)
return result
except httpx.TimeoutException:
logger.warning(f"verifymail.io API timeout for {_mask_email(email)}")
return None
except Exception as e:
logger.warning(f"verifymail.io API error for {_mask_email(email)}: {e}")
return None
def _is_domain_verified_clean(self, domain: str, config: dict) -> bool:
"""Check if a domain was verified clean within the TTL period."""
verified = config.get("verified_clean_domains", {})
entry = verified.get(domain)
if not entry:
return False
checked_at = entry.get("checked_at")
if not checked_at:
return False
# Handle both datetime objects and ISO strings
if isinstance(checked_at, str):
try:
checked_at = datetime.fromisoformat(checked_at)
except (ValueError, TypeError):
return False
# Make timezone-aware if naive
if checked_at.tzinfo is None:
checked_at = checked_at.replace(tzinfo=timezone.utc)
return (datetime.now(timezone.utc) - checked_at) < VERIFIED_CLEAN_TTL
def _auto_learn_domain(self, domain: str) -> None:
"""
Add a domain flagged by external API to manual_domains in Firestore.
This persists the result so future checks don't need the API call.
Fire-and-forget: errors are logged but don't affect the caller.
"""
try:
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(
BLOCKLIST_CONFIG_DOC
)
doc = doc_ref.get()
if doc.exists:
data = doc.to_dict()
manual = set(data.get("manual_domains", []))
else:
manual = set()
if domain not in manual:
manual.add(domain)
doc_ref.set(
{
"manual_domains": list(manual),
"updated_at": datetime.now(timezone.utc).isoformat(),
"updated_by": "auto-learn",
},
merge=True,
)
logger.info(f"Auto-learned disposable domain: {domain}")
# Invalidate cache so the domain is blocked immediately
EmailValidationService._blocklist_cache = None
except Exception as e:
logger.error(f"Failed to auto-learn domain {domain}: {e}")
def _mark_domain_clean(self, domain: str) -> None:
"""
Record that a domain was verified clean by external APIs.
Stored in verified_clean_domains dict keyed by domain.
Future checks within VERIFIED_CLEAN_TTL will skip API calls.
Note: Uses read-modify-write instead of dot notation because
domain names contain dots which Firestore interprets as nested paths.
"""
try:
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(
BLOCKLIST_CONFIG_DOC
)
doc = doc_ref.get()
verified = doc.to_dict().get("verified_clean_domains", {}) if doc.exists else {}
verified[domain] = {
"checked_at": datetime.now(timezone.utc).isoformat(),
}
doc_ref.set({"verified_clean_domains": verified}, merge=True)
logger.debug(f"Marked domain as verified clean: {domain}")
# Invalidate cache so new data is picked up
EmailValidationService._blocklist_cache = None
except Exception as e:
logger.error(f"Failed to mark domain clean {domain}: {e}")
return domain in config["disposable_domains"]
def is_email_blocked(self, email: str) -> bool:
"""
@@ -489,6 +369,36 @@ class EmailValidationService:
config = self.get_blocklist_config()
return ip_address in config["blocked_ips"]
def validate_email_for_beta(self, email: str) -> Tuple[bool, str]:
"""
Validate an email for beta enrollment.
Performs all validation checks:
1. Basic format validation
2. Disposable domain check
3. Blocked email check
Args:
email: Email address to validate
Returns:
Tuple of (is_valid, error_message)
"""
if not email or "@" not in email:
return False, "Invalid email format"
# Check disposable domain
if self.is_disposable_domain(email):
logger.warning(f"Beta enrollment blocked - disposable domain: {_mask_email(email)}")
return False, "Disposable email addresses are not allowed"
# Check blocked email
if self.is_email_blocked(email):
logger.warning(f"Beta enrollment blocked - email blocked: {_mask_email(email)}")
return False, "This email address is not allowed"
return True, ""
def hash_ip(self, ip_address: str) -> str:
"""
Hash an IP address for privacy-preserving storage.
@@ -518,13 +428,13 @@ class EmailValidationService:
doc = doc_ref.get(transaction=transaction)
if doc.exists:
data = doc.to_dict()
domains = set(data.get("manual_domains", []))
domains = set(data.get("disposable_domains", []))
else:
data = {}
domains = set()
domains.add(domain)
data["manual_domains"] = list(domains)
data["disposable_domains"] = list(domains)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
@@ -552,22 +462,14 @@ class EmailValidationService:
return
data = doc.to_dict()
domains = set(data.get("disposable_domains", []))
manual = set(data.get("manual_domains", []))
external = set(data.get("external_domains", []))
if domain in manual:
result["found"] = True
manual.discard(domain)
data["manual_domains"] = list(manual)
elif domain in external:
result["found"] = True
allowlisted = set(data.get("allowlisted_domains", []))
allowlisted.add(domain)
data["allowlisted_domains"] = list(allowlisted)
else:
if domain not in domains:
return
result["found"] = True
domains.discard(domain)
data["disposable_domains"] = list(domains)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
@@ -726,126 +628,14 @@ class EmailValidationService:
logger.info(f"Removed blocked IP: {ip_address} by {admin_email}")
return True
def add_allowlisted_domain(self, domain: str, admin_email: str) -> bool:
"""Add a domain to the allowlisted domains list."""
domain = domain.lower().strip()
if not domain:
return False
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
@firestore.transactional
def update_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if doc.exists:
data = doc.to_dict()
domains = set(data.get("allowlisted_domains", []))
else:
data = {}
domains = set()
domains.add(domain)
data["allowlisted_domains"] = list(domains)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
transaction = self.db.transaction()
update_in_transaction(transaction, doc_ref)
# Invalidate cache
EmailValidationService._blocklist_cache = None
logger.info(f"Added allowlisted domain: {domain} by {admin_email}")
return True
def remove_allowlisted_domain(self, domain: str, admin_email: str) -> bool:
"""Remove a domain from the allowlisted domains list."""
domain = domain.lower().strip()
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
result = {"found": False}
@firestore.transactional
def remove_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if not doc.exists:
return
data = doc.to_dict()
domains = set(data.get("allowlisted_domains", []))
if domain not in domains:
return
result["found"] = True
domains.discard(domain)
data["allowlisted_domains"] = list(domains)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
transaction = self.db.transaction()
remove_in_transaction(transaction, doc_ref)
if not result["found"]:
return False
# Invalidate cache
EmailValidationService._blocklist_cache = None
logger.info(f"Removed allowlisted domain: {domain} by {admin_email}")
return True
def get_blocklist_raw_data(self) -> dict:
"""Get raw blocklist data for admin display (not the effective set)."""
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
doc = doc_ref.get()
if not doc.exists:
return {
"external_domains": [], "manual_domains": [], "allowlisted_domains": [],
"blocked_emails": [], "blocked_ips": [],
"last_sync_at": None, "last_sync_count": None,
"updated_at": None, "updated_by": None,
}
data = doc.to_dict()
return {
"external_domains": sorted(data.get("external_domains", [])),
"manual_domains": sorted(data.get("manual_domains", [])),
"allowlisted_domains": sorted(data.get("allowlisted_domains", [])),
"blocked_emails": sorted(data.get("blocked_emails", [])),
"blocked_ips": sorted(data.get("blocked_ips", [])),
"last_sync_at": data.get("last_sync_at"),
"last_sync_count": data.get("last_sync_count"),
"updated_at": data.get("updated_at"),
"updated_by": data.get("updated_by"),
}
def get_blocklist_stats(self) -> dict:
"""Get statistics about current blocklists."""
config = self.get_blocklist_config(force_refresh=True)
# Also get raw data for detailed counts
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
doc = doc_ref.get()
if doc.exists:
data = doc.to_dict()
external_count = len(data.get("external_domains", []))
manual_count = len(data.get("manual_domains", []))
allowlisted_count = len(data.get("allowlisted_domains", []))
else:
external_count = 0
manual_count = 0
allowlisted_count = 0
return {
"disposable_domains_count": len(config["disposable_domains"]),
"blocked_emails_count": len(config["blocked_emails"]),
"blocked_ips_count": len(config["blocked_ips"]),
"default_disposable_domains_count": len(DEFAULT_DISPOSABLE_DOMAINS),
"external_domains_count": external_count,
"manual_domains_count": manual_count,
"allowlisted_domains_count": allowlisted_count,
}
-3
View File
@@ -383,8 +383,6 @@ class GCEEncodingBackend(EncodingBackend):
output_files["mp4_4k_lossy"] = path
elif "720p" in filename_lower:
output_files["mp4_720p"] = path
elif "with vocals" in filename_lower and filename.endswith(".mp4"):
output_files["with_vocals_mp4"] = path
self.logger.info(f"Converted output_files list to dict: {output_files}")
else:
output_files = raw_output_files if isinstance(raw_output_files, dict) else {}
@@ -395,7 +393,6 @@ class GCEEncodingBackend(EncodingBackend):
lossy_4k_mp4_path=output_files.get("mp4_4k_lossy"),
lossless_mkv_path=output_files.get("mkv_4k"),
lossy_720p_mp4_path=output_files.get("mp4_720p"),
with_vocals_mp4_path=output_files.get("with_vocals_mp4"),
output_files=output_files,
encoding_time_seconds=encoding_time,
encoding_backend=self.name
+13 -258
View File
@@ -17,7 +17,6 @@ Usage:
import asyncio
import logging
import time
from typing import Optional, Dict, Any
import aiohttp
@@ -26,23 +25,10 @@ from backend.config import get_settings
logger = logging.getLogger(__name__)
# Retry configuration for handling transient failures (e.g., worker restarts during deployments).
#
# During a CI deployment, the encoding worker is restarted via systemctl. The restart
# involves downloading a new wheel from GCS, installing it, and starting uvicorn —
# which can take 30-90 seconds. The retry window must exceed this to avoid failing
# jobs that happen to hit the encoding stage during a deploy.
#
# With 7 retries and exponential backoff (5s, 10s, 15s, 15s, 15s, 15s, 15s),
# total retry window is ~90s, sufficient to survive a full worker restart.
MAX_RETRIES = 7
INITIAL_BACKOFF_SECONDS = 5.0
MAX_BACKOFF_SECONDS = 15.0
# Poll failure tolerance: number of consecutive poll failures allowed before giving up.
# This prevents a single transient network blip during status polling from killing a
# long-running encoding job. Similar pattern to flacfetch status polling (PR #446).
MAX_CONSECUTIVE_POLL_FAILURES = 5
# Retry configuration for handling transient failures (e.g., worker restarts)
MAX_RETRIES = 3
INITIAL_BACKOFF_SECONDS = 2.0
MAX_BACKOFF_SECONDS = 10.0
class EncodingService:
@@ -53,34 +39,6 @@ class EncodingService:
self._url = None
self._api_key = None
self._initialized = False
self._worker_manager = None
self._cached_url = None
self._url_cached_at = 0
self._URL_CACHE_TTL = 30 # seconds
def set_worker_manager(self, manager):
"""Set the worker manager for dynamic URL resolution from Firestore."""
self._worker_manager = manager
def _get_worker_url(self) -> str:
"""Get the current primary worker URL from Firestore (with TTL cache).
Falls back to static URL from config if worker_manager is not set.
"""
now = time.time()
if self._cached_url and (now - self._url_cached_at) < self._URL_CACHE_TTL:
return self._cached_url
if self._worker_manager:
config = self._worker_manager.get_config()
self._cached_url = config.primary_url
self._url_cached_at = now
return self._cached_url
# Fallback to static URL
if not self._initialized:
self._load_credentials()
return self._url
def _load_credentials(self):
"""Load encoding worker URL and API key from config/secrets."""
@@ -115,28 +73,6 @@ class EncodingService:
"""Check if GCE preview encoding is enabled and configured."""
return self.settings.use_gce_preview_encoding and self.is_configured
def _warmup_encoding_worker_fallback(self, job_id: str) -> None:
"""Safety net: try to start the encoding worker VM on first connection failure.
If the worker_manager is available, calls ensure_primary_running() so the VM
starts booting during the retry window. If worker_manager is not set (e.g. dev
mode with static URL), this is a no-op.
"""
if not self._worker_manager:
return
try:
result = self._worker_manager.ensure_primary_running()
if result["started"]:
logger.warning(
f"[job:{job_id}] Encoding worker unreachable — started VM {result['vm_name']} as fallback"
)
else:
logger.info(
f"[job:{job_id}] Encoding worker unreachable — VM {result['vm_name']} already running/starting"
)
except Exception as e:
logger.warning(f"[job:{job_id}] Encoding worker warmup fallback failed (non-fatal): {e}")
async def _request_with_retry(
self,
method: str,
@@ -198,8 +134,6 @@ class EncodingService:
}
except (aiohttp.ClientConnectorError, aiohttp.ServerDisconnectedError, asyncio.TimeoutError) as e:
last_exception = e
if attempt == 0:
self._warmup_encoding_worker_fallback(job_id)
if attempt < MAX_RETRIES:
logger.warning(
f"[job:{job_id}] GCE worker connection failed (attempt {attempt + 1}/{MAX_RETRIES + 1}): {e}. "
@@ -241,7 +175,7 @@ class EncodingService:
if not self.is_configured:
raise RuntimeError("Encoding service not configured")
url = f"{self._get_worker_url()}/encode"
url = f"{self._url}/encode"
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
payload = {
"job_id": job_id,
@@ -264,24 +198,7 @@ class EncodingService:
if resp["status"] == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp["status"] == 409:
# Job already exists on worker — check if it already completed
logger.warning(f"[job:{job_id}] GCE worker returned 409, checking job status")
try:
status = await self.get_job_status(job_id)
job_status = status.get("status", "unknown")
if job_status == "complete":
logger.info(f"[job:{job_id}] Job already complete on GCE worker, returning cached result")
return {"status": "cached", "job_id": job_id, "output_files": status.get("output_files")}
elif job_status in ("pending", "running"):
logger.info(f"[job:{job_id}] Job still in progress on GCE worker")
return {"status": "in_progress", "job_id": job_id}
else:
raise RuntimeError(f"Encoding job {job_id} already exists with status: {job_status}")
except RuntimeError as e:
if "not found" in str(e).lower():
# Job was in worker memory but got cleared (restart) — safe to raise original error
raise RuntimeError(f"Encoding job {job_id} conflict: 409 but job not found on status check")
raise
raise RuntimeError(f"Encoding job {job_id} already exists")
if resp["status"] != 200:
raise RuntimeError(f"Failed to submit encoding job: {resp['status']} - {resp['text']}")
@@ -302,7 +219,7 @@ class EncodingService:
if not self.is_configured:
raise RuntimeError("Encoding service not configured")
url = f"{self._get_worker_url()}/status/{job_id}"
url = f"{self._url}/status/{job_id}"
headers = {"X-API-Key": self._api_key}
resp = await self._request_with_retry(
@@ -330,12 +247,7 @@ class EncodingService:
progress_callback=None,
) -> Dict[str, Any]:
"""
Poll for encoding job completion with tolerance for transient failures.
During a deployment, the encoding worker may restart while we're polling.
Rather than failing the entire job on a single poll error, we tolerate up to
MAX_CONSECUTIVE_POLL_FAILURES consecutive failures before giving up. This is
the same pattern used in flacfetch status polling (PR #446).
Poll for encoding job completion.
Args:
job_id: Job identifier
@@ -348,40 +260,19 @@ class EncodingService:
Raises:
TimeoutError: If job doesn't complete within timeout
RuntimeError: If job fails or too many consecutive poll failures
RuntimeError: If job fails
"""
logger.info(f"[job:{job_id}] Waiting for GCE encoding to complete...")
start_time = asyncio.get_event_loop().time()
last_progress = 0
consecutive_failures = 0
while True:
elapsed = asyncio.get_event_loop().time() - start_time
if elapsed > timeout:
raise TimeoutError(f"Encoding job {job_id} timed out after {timeout}s")
try:
status = await self.get_job_status(job_id)
# Reset failure counter on successful poll
consecutive_failures = 0
except (aiohttp.ClientConnectorError, aiohttp.ServerDisconnectedError,
asyncio.TimeoutError, RuntimeError) as e:
consecutive_failures += 1
if consecutive_failures >= MAX_CONSECUTIVE_POLL_FAILURES:
logger.error(
f"[job:{job_id}] Status polling failed {consecutive_failures} consecutive times, giving up: {e}"
)
raise RuntimeError(
f"Encoding job {job_id} lost contact with worker after "
f"{consecutive_failures} consecutive poll failures: {e}"
)
logger.warning(
f"[job:{job_id}] Status poll failed ({consecutive_failures}/{MAX_CONSECUTIVE_POLL_FAILURES}): {e}. "
f"Worker may be restarting, will retry..."
)
await asyncio.sleep(poll_interval)
continue
status = await self.get_job_status(job_id)
# Handle case where GCE worker returns a list instead of dict
if isinstance(status, list):
@@ -440,17 +331,7 @@ class EncodingService:
config = encoding_config or {"formats": ["mp4_4k", "mp4_720p"]}
# Submit the job
submit_result = await self.submit_encoding_job(job_id, input_gcs_path, output_gcs_path, config)
# If cached, return immediately — encoding already done
submit_status = submit_result.get("status")
if submit_status == "cached":
logger.info(f"[job:{job_id}] Encoding already cached, returning immediately")
return {"status": "complete", "output_files": submit_result.get("output_files")}
# If in_progress, another request is encoding it — just wait for that
if submit_status == "in_progress":
logger.info(f"[job:{job_id}] Encoding already in progress, joining poll")
await self.submit_encoding_job(job_id, input_gcs_path, output_gcs_path, config)
# Wait for completion
return await self.wait_for_completion(
@@ -490,7 +371,7 @@ class EncodingService:
if not self.is_configured:
raise RuntimeError("Encoding service not configured")
url = f"{self._get_worker_url()}/encode-preview"
url = f"{self._url}/encode-preview"
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
payload = {
"job_id": job_id,
@@ -584,116 +465,6 @@ class EncodingService:
timeout=timeout,
)
async def submit_render_video_job(
self,
job_id: str,
render_config: Dict[str, Any],
) -> Dict[str, Any]:
"""
Submit a render-video job to the GCE worker.
Args:
job_id: Unique job identifier
render_config: Configuration dict containing original_corrections_gcs_path,
audio_gcs_path, output_gcs_prefix, artist, title, and any other render params.
Returns:
Response from the encoding worker
Raises:
RuntimeError: If submission fails
"""
self._load_credentials()
if not self.is_configured:
raise RuntimeError("Encoding service not configured")
url = f"{self._get_worker_url()}/render-video"
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
payload = {"job_id": job_id, **render_config}
logger.info(f"[job:{job_id}] Submitting render-video job to GCE worker: {url}")
resp = await self._request_with_retry(
method="POST",
url=url,
headers=headers,
json_payload=payload,
timeout=30.0,
job_id=job_id,
)
if resp["status"] == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp["status"] == 409:
# Job already exists on worker — check if it already completed
logger.warning(f"[job:{job_id}] GCE worker returned 409, checking job status")
try:
status = await self.get_job_status(job_id)
job_status = status.get("status", "unknown")
if job_status == "complete":
logger.info(f"[job:{job_id}] Render-video job already complete on GCE worker, returning cached result")
return {
"status": "cached",
"job_id": job_id,
"output_files": status.get("output_files"),
"metadata": status.get("metadata"),
}
elif job_status in ("pending", "running"):
logger.info(f"[job:{job_id}] Render-video job still in progress on GCE worker")
return {"status": "in_progress", "job_id": job_id}
else:
raise RuntimeError(f"Render-video job {job_id} already exists with status: {job_status}")
except RuntimeError as e:
if "not found" in str(e).lower():
raise RuntimeError(f"Render-video job {job_id} conflict: 409 but job not found on status check")
raise
if resp["status"] != 200:
raise RuntimeError(f"Failed to submit render-video job: {resp['status']} - {resp['text']}")
return resp["json"]
async def render_video_on_gce(
self,
job_id: str,
render_config: Dict[str, Any],
progress_callback=None,
) -> Dict[str, Any]:
"""
Submit render-video job and wait for completion.
This is a convenience method that combines submit + wait.
Args:
job_id: Unique job identifier
render_config: Configuration dict for the render-video worker endpoint
progress_callback: Optional callback for progress updates
Returns:
Final job status with output files and metadata
"""
# Submit the job
submit_result = await self.submit_render_video_job(job_id, render_config)
# If cached, return immediately — rendering already done
submit_status = submit_result.get("status")
if submit_status == "cached":
logger.info(f"[job:{job_id}] Render-video already cached, returning immediately")
return {
"status": "complete",
"output_files": submit_result.get("output_files"),
"metadata": submit_result.get("metadata"),
}
# If in_progress, another request is rendering it — just wait for that
if submit_status == "in_progress":
logger.info(f"[job:{job_id}] Render-video already in progress, joining poll")
# Wait for completion
return await self.wait_for_completion(
job_id, progress_callback=progress_callback
)
async def health_check(self) -> Dict[str, Any]:
"""
Check the health of the encoding worker.
@@ -706,7 +477,7 @@ class EncodingService:
if not self.is_configured:
return {"status": "not_configured"}
url = f"{self._get_worker_url()}/health"
url = f"{self._url}/health"
headers = {"X-API-Key": self._api_key}
try:
@@ -728,20 +499,4 @@ def get_encoding_service() -> EncodingService:
global _encoding_service
if _encoding_service is None:
_encoding_service = EncodingService()
# Wire up worker manager for dynamic URL resolution
try:
from backend.services.encoding_worker_manager import EncodingWorkerManager
from google.cloud import compute_v1, firestore
settings = get_settings()
db = firestore.Client(project=settings.google_cloud_project)
compute_client = compute_v1.InstancesClient()
manager = EncodingWorkerManager(
db=db,
compute_client=compute_client,
project_id=settings.google_cloud_project,
)
_encoding_service.set_worker_manager(manager)
except Exception:
# Fallback to static URL if worker manager setup fails
pass
return _encoding_service
-239
View File
@@ -1,239 +0,0 @@
"""
Blue-Green Encoding Worker Manager.
Manages the lifecycle and role assignment of two GCE encoding worker VMs
(blue/green) via a Firestore config document as the single source of truth.
The config document at `config/encoding-worker` tracks which VM is primary
(actively serving encoding requests) and which is secondary (idle or being
deployed to). This enables zero-downtime deployments by swapping roles
after deploying new code to the secondary.
Usage:
from google.cloud import firestore
from google.cloud.compute_v1 import InstancesClient
db = firestore.Client()
compute = InstancesClient()
manager = EncodingWorkerManager(db, compute)
# Get current config
config = manager.get_config()
print(f"Primary: {config.primary_vm} at {config.primary_url}")
# Ensure primary VM is running before dispatching work
result = manager.ensure_primary_running()
"""
import logging
from dataclasses import dataclass
from datetime import datetime, UTC
from typing import Optional
from google.cloud import firestore
logger = logging.getLogger(__name__)
CONFIG_COLLECTION = "config"
CONFIG_DOCUMENT = "encoding-worker"
ENCODING_WORKER_PORT = 8080
@dataclass
class EncodingWorkerConfig:
"""Represents the Firestore config/encoding-worker document."""
primary_vm: str
primary_ip: str
primary_version: str
primary_deployed_at: Optional[str]
secondary_vm: str
secondary_ip: str
secondary_version: Optional[str]
secondary_deployed_at: Optional[str]
last_swap_at: Optional[str]
last_activity_at: Optional[str]
deploy_in_progress: bool
deploy_in_progress_since: Optional[str]
@property
def primary_url(self) -> str:
return f"http://{self.primary_ip}:{ENCODING_WORKER_PORT}"
@property
def secondary_url(self) -> str:
return f"http://{self.secondary_ip}:{ENCODING_WORKER_PORT}"
class EncodingWorkerManager:
"""Manages blue-green encoding worker VMs via Firestore config and Compute API."""
def __init__(self, db, compute_client=None, project_id="nomadkaraoke", zone="us-central1-c"):
self._db = db
self._compute = compute_client
self._project_id = project_id
self._zone = zone
def _doc_ref(self):
"""Return a reference to the config/encoding-worker document."""
return self._db.collection(CONFIG_COLLECTION).document(CONFIG_DOCUMENT)
# ------------------------------------------------------------------
# Firestore config operations (Task 1)
# ------------------------------------------------------------------
def get_config(self) -> EncodingWorkerConfig:
"""Read config/encoding-worker from Firestore and return as dataclass."""
snapshot = self._doc_ref().get()
if not snapshot.exists:
raise ValueError(
f"Encoding worker config document not found at "
f"{CONFIG_COLLECTION}/{CONFIG_DOCUMENT}"
)
data = snapshot.to_dict()
return EncodingWorkerConfig(
primary_vm=data["primary_vm"],
primary_ip=data["primary_ip"],
primary_version=data["primary_version"],
primary_deployed_at=data.get("primary_deployed_at"),
secondary_vm=data["secondary_vm"],
secondary_ip=data["secondary_ip"],
secondary_version=data.get("secondary_version"),
secondary_deployed_at=data.get("secondary_deployed_at"),
last_swap_at=data.get("last_swap_at"),
last_activity_at=data.get("last_activity_at"),
deploy_in_progress=data.get("deploy_in_progress", False),
deploy_in_progress_since=data.get("deploy_in_progress_since"),
)
def update_activity(self) -> None:
"""Update last_activity_at timestamp to record recent usage."""
now = datetime.now(UTC).isoformat()
self._doc_ref().update({"last_activity_at": now})
logger.info("Updated encoding worker last_activity_at to %s", now)
def swap_roles(
self,
new_primary_vm: str,
new_primary_ip: str,
new_primary_version: str,
old_primary_vm: str,
old_primary_ip: str,
old_primary_version: str,
) -> None:
"""Atomically swap primary and secondary roles using a Firestore transaction.
Args:
new_primary_vm: VM name that will become the new primary.
new_primary_ip: Internal IP of the new primary.
new_primary_version: Version deployed on the new primary.
old_primary_vm: Current primary VM name (becomes secondary).
old_primary_ip: Internal IP of the old primary.
old_primary_version: Version on the old primary.
"""
transaction = self._db.transaction()
doc_ref = self._doc_ref()
now = datetime.now(UTC).isoformat()
@firestore.transactional
def _swap_in_transaction(txn, ref):
txn.update(ref, {
"primary_vm": new_primary_vm,
"primary_ip": new_primary_ip,
"primary_version": new_primary_version,
"primary_deployed_at": now,
"secondary_vm": old_primary_vm,
"secondary_ip": old_primary_ip,
"secondary_version": old_primary_version,
"secondary_deployed_at": None,
"last_swap_at": now,
"deploy_in_progress": False,
"deploy_in_progress_since": None,
})
_swap_in_transaction(transaction, doc_ref)
logger.info(
"Swapped encoding worker roles: %s -> primary, %s -> secondary",
new_primary_vm, old_primary_vm,
)
# ------------------------------------------------------------------
# VM lifecycle operations (Task 2)
# ------------------------------------------------------------------
def get_vm_status(self, vm_name: str) -> str:
"""Get the current status of a GCE VM (e.g. RUNNING, TERMINATED, STAGING)."""
instance = self._compute.get(
project=self._project_id,
zone=self._zone,
instance=vm_name,
)
return instance.status
def start_vm(self, vm_name: str) -> None:
"""Start a VM if it is not already running or starting.
Fire-and-forget: returns immediately. Caller should poll get_vm_status
if they need to wait for the VM to be fully RUNNING.
"""
status = self.get_vm_status(vm_name)
if status in ("RUNNING", "STAGING"):
logger.info("VM %s is already %s, skipping start", vm_name, status)
return
logger.info("Starting VM %s (current status: %s)", vm_name, status)
self._compute.start(
project=self._project_id,
zone=self._zone,
instance=vm_name,
)
def stop_vm(self, vm_name: str) -> None:
"""Stop a VM if it is not already stopped or stopping."""
status = self.get_vm_status(vm_name)
if status in ("TERMINATED", "STOPPING"):
logger.info("VM %s is already %s, skipping stop", vm_name, status)
return
logger.info("Stopping VM %s (current status: %s)", vm_name, status)
self._compute.stop(
project=self._project_id,
zone=self._zone,
instance=vm_name,
)
def ensure_primary_running(self) -> dict:
"""Ensure the primary encoding worker VM is running.
Reads config, starts primary VM if stopped, and updates activity timestamp.
Returns:
dict with keys:
- started (bool): True if VM was started, False if already running
- vm_name (str): Name of the primary VM
- primary_url (str): URL of the primary encoding worker
"""
config = self.get_config()
vm_name = config.primary_vm
status = self.get_vm_status(vm_name)
started = False
if status not in ("RUNNING", "STAGING"):
logger.info("Primary VM %s is %s, starting it", vm_name, status)
self._compute.start(
project=self._project_id,
zone=self._zone,
instance=vm_name,
)
started = True
else:
logger.info("Primary VM %s is already %s", vm_name, status)
self.update_activity()
return {
"started": started,
"vm_name": vm_name,
"primary_url": config.primary_url,
}
@@ -1 +0,0 @@
"""Production error monitoring service for Nomad Karaoke."""
-177
View File
@@ -1,177 +0,0 @@
"""Configuration for the production error monitoring service.
All constants, service lists, and environment-driven configuration
functions used across the error monitor modules.
"""
import os
# ---------------------------------------------------------------------------
# GCP project settings
# ---------------------------------------------------------------------------
GCP_PROJECT: str = os.environ.get("GCP_PROJECT", "nomadkaraoke")
GCP_REGION: str = os.environ.get("GCP_REGION", "us-central1")
# ---------------------------------------------------------------------------
# Log querying settings
# ---------------------------------------------------------------------------
#: How far back to look for errors on each run (minutes).
LOOKBACK_MINUTES: int = 15
#: Maximum number of log entries to fetch per service per run.
MAX_LOG_ENTRIES: int = 500
# ---------------------------------------------------------------------------
# Monitored services
# ---------------------------------------------------------------------------
MONITORED_CLOUD_RUN_SERVICES: list[str] = [
"karaoke-backend",
"karaoke-decide",
"audio-separator",
# Gen2 Cloud Functions appear as Cloud Run services in logs
"gdrive-validator",
"github-runner-manager",
"backup-to-aws",
"divebar-mirror",
"kn-data-sync",
"divebar-lookup",
"encoding-worker-idle-shutdown",
]
MONITORED_CLOUD_RUN_JOBS: list[str] = [
"video-encoding-job",
"lyrics-transcription-job",
"audio-separation-job",
"audio-download-job",
]
# Gen2 Cloud Functions log as cloud_run_revision, not cloud_function.
# Their names are in MONITORED_CLOUD_RUN_SERVICES above.
MONITORED_CLOUD_FUNCTIONS: list[str] = []
MONITORED_GCE_INSTANCES: list[str] = [
"encoding-worker-a",
"encoding-worker-b",
"flacfetch-vm",
"divebar-sync-vm",
]
# ---------------------------------------------------------------------------
# Service dependency map
# Human-readable description of which services depend on which workers.
# Used as LLM context when generating root-cause analysis.
# ---------------------------------------------------------------------------
SERVICE_DEPENDENCY_MAP: str = """\
karaoke-backend (main API)
→ video-encoding-job (Cloud Run Job, triggered per order)
→ lyrics-transcription-job (Cloud Run Job, triggered after audio-sep)
→ audio-separation-job (Cloud Run Job, runs on Cloud Run GPU)
→ audio-download-job (Cloud Run Job, uses flacfetch-vm)
→ gdrive-validator (Cloud Function, validates output files)
→ encoding-worker-a/b (GCE VMs, long-running encoding workers)
karaoke-decide (catalog / discovery API)
→ kn_data_sync (Cloud Function, syncs KaraokeNerds catalog)
→ divebar_mirror (Cloud Function, mirrors Divebar catalog)
→ divebar_lookup (Cloud Function, single-song lookup)
audio-separator (Cloud Run service, GPU stem separation)
→ audio-separation-job (Cloud Run Job, launched by audio-separator)
flacfetch-vm (GCE VM)
→ audio-download-job (initiated by karaoke-backend)
divebar-sync-vm (GCE VM)
→ divebar_mirror (Cloud Function, reads from this VM)
Shared infrastructure:
→ runner_manager (Cloud Function, manages encoding worker lifecycle)
→ encoding_worker_idle (Cloud Function, shuts down idle encoding workers)
→ backup_to_aws (Cloud Function, daily GCS → S3 backup)
"""
# ---------------------------------------------------------------------------
# LLM analysis settings
# ---------------------------------------------------------------------------
LLM_ANALYSIS_MODEL: str = os.environ.get("LLM_ANALYSIS_MODEL", "gemini-2.0-flash-001")
LLM_VERTEX_LOCATION: str = os.environ.get("LLM_VERTEX_LOCATION", "us-central1")
#: Minimum number of active error patterns required before triggering LLM analysis.
MIN_PATTERNS_FOR_ANALYSIS: int = 3
# ---------------------------------------------------------------------------
# Spike detection
# ---------------------------------------------------------------------------
#: An error count is a spike if it exceeds SPIKE_MULTIPLIER × baseline average.
SPIKE_MULTIPLIER: float = 5.0
#: Minimum absolute error count before spike detection fires (avoids noise).
SPIKE_MIN_COUNT: int = 5
# ---------------------------------------------------------------------------
# Auto-resolution settings
# ---------------------------------------------------------------------------
#: Multiply LOOKBACK_MINUTES by this to get auto-resolve window in minutes.
AUTO_RESOLVE_MULTIPLIER: int = 8
#: Minimum hours before a pattern can be auto-resolved.
AUTO_RESOLVE_MIN_HOURS: int = 6
#: Maximum hours before a pattern is force-resolved even if still active.
AUTO_RESOLVE_MAX_HOURS: int = 168 # 1 week
#: Fallback resolve window (hours) if baseline data is unavailable.
AUTO_RESOLVE_FALLBACK_HOURS: int = 48
# ---------------------------------------------------------------------------
# Discord settings
# ---------------------------------------------------------------------------
#: Maximum Discord messages to send per monitor run (rate-limit guard).
MAX_DISCORD_MESSAGES_PER_RUN: int = 10
#: Discord API hard limit on message length.
DISCORD_MAX_MESSAGE_LENGTH: int = 2000
# ---------------------------------------------------------------------------
# Pattern tracking settings
# ---------------------------------------------------------------------------
#: Maximum number of active error patterns to track in Firestore.
MAX_ACTIVE_PATTERNS: int = 500
#: Rolling window used when computing error baselines.
ROLLING_WINDOW_DAYS: int = 7
#: Maximum length of a normalized error message stored in Firestore.
MAX_NORMALIZED_MESSAGE_LENGTH: int = 200
# ---------------------------------------------------------------------------
# Helper functions
# ---------------------------------------------------------------------------
def get_llm_enabled() -> bool:
"""Return True if LLM analysis is enabled via the LLM_ANALYSIS_ENABLED env var."""
return os.environ.get("LLM_ANALYSIS_ENABLED", "false").strip().lower() == "true"
def get_discord_webhook_secret_name() -> str:
"""Return the Secret Manager secret name for the Discord webhook URL."""
return os.environ.get("DISCORD_WEBHOOK_SECRET_NAME", "error-monitor-discord-webhook")
def get_digest_mode() -> bool:
"""Return True if the monitor should run in digest mode (batch alerts instead of immediate).
In digest mode all alerts are bundled into a single Discord message sent at
the end of the run, rather than sending one message per error pattern.
"""
return os.environ.get("DIGEST_MODE", "false").strip().lower() == "true"
-323
View File
@@ -1,323 +0,0 @@
"""Discord alert client for the production error monitoring service.
Provides formatting functions for each alert type, and an ErrorMonitorDiscord
webhook client that rate-limits outbound messages to MAX_DISCORD_MESSAGES_PER_RUN
per monitor run.
Usage:
from backend.services.error_monitor.discord import (
ErrorMonitorDiscord,
format_new_pattern_alert,
format_incident_alert,
format_auto_resolved_alert,
format_spike_alert,
format_daily_digest,
)
discord = ErrorMonitorDiscord(webhook_url="https://discord.com/api/webhooks/...")
msg = format_new_pattern_alert(...)
discord.send_message(msg)
"""
import logging
from datetime import datetime
from zoneinfo import ZoneInfo
import requests
from backend.services.error_monitor.config import (
DISCORD_MAX_MESSAGE_LENGTH,
MAX_DISCORD_MESSAGES_PER_RUN,
)
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Constants
# ---------------------------------------------------------------------------
SEVERITY_EMOJI: dict[str, str] = {
"P0": "🚨",
"P1": "🔴",
"P2": "⚠️",
"P3": "🟡",
}
_EASTERN = ZoneInfo("America/New_York")
# ---------------------------------------------------------------------------
# Private helpers
# ---------------------------------------------------------------------------
def _now_eastern() -> str:
"""Return the current Eastern time as 'HH:MM AM/PM ET' (DST-aware)."""
now = datetime.now(tz=_EASTERN)
return now.strftime("%I:%M %p ET").lstrip("0")
def _truncate(msg: str) -> str:
"""Truncate *msg* to DISCORD_MAX_MESSAGE_LENGTH characters."""
if len(msg) <= DISCORD_MAX_MESSAGE_LENGTH:
return msg
suffix = "…"
return msg[: DISCORD_MAX_MESSAGE_LENGTH - len(suffix)] + suffix
# ---------------------------------------------------------------------------
# Formatting functions
# ---------------------------------------------------------------------------
def format_new_pattern_alert(
service: str,
resource_type: str,
normalized_message: str,
sample_message: str,
count: int,
first_seen: str,
) -> str:
"""Format a Discord alert for a newly-detected error pattern.
Args:
service: Name of the service emitting the error.
resource_type: GCP resource type (e.g. 'cloud_run_service', 'gce_instance').
normalized_message: Normalised error pattern string.
sample_message: A representative raw log message.
count: Number of occurrences seen so far.
first_seen: ISO 8601 timestamp when the pattern was first seen.
Returns:
Formatted Discord message string, truncated to 2000 chars.
"""
lines = [
f"🔴 **New Error Pattern Detected**",
f"",
f"**Service:** {service}",
f"**Resource type:** {resource_type}",
f"**Pattern:** {normalized_message}",
f"**Sample:** {sample_message}",
f"**Count:** {count}",
f"**First seen:** {first_seen}",
f"**Alerted at:** {_now_eastern()}",
f"",
f"To investigate: `/prod-investigate {service} \"{normalized_message}\"`",
]
return _truncate("\n".join(lines))
def format_incident_alert(
title: str,
severity: str,
root_cause: str,
suggested_fix: str,
primary_service: str,
patterns: list[dict],
) -> str:
"""Format a Discord alert for an analysed incident (LLM-generated).
Args:
title: Short incident title.
severity: Severity level — one of P0, P1, P2, P3.
root_cause: Human-readable root-cause analysis.
suggested_fix: Recommended remediation steps.
primary_service: The main service involved in the incident.
patterns: List of dicts, each with keys: service, normalized_message, count.
Only the first 5 are shown; normalized_message is truncated to 80 chars.
Returns:
Formatted Discord message string, truncated to 2000 chars.
"""
emoji = SEVERITY_EMOJI.get(severity, "⚠️")
lines = [
f"{emoji} **[{severity}] {title}**",
f"",
f"**Primary service:** {primary_service}",
f"**Root cause:** {root_cause}",
f"**Suggested fix:** {suggested_fix}",
]
shown_patterns = patterns[:5]
if shown_patterns:
lines.append("")
lines.append(f"**Patterns ({len(shown_patterns)} of {len(patterns)}):**")
for p in shown_patterns:
svc = p.get("service", "unknown")
msg = p.get("normalized_message", "")[:80]
cnt = p.get("count", 0)
lines.append(f" • `{svc}` — {msg} ({cnt}x)")
lines.append("")
lines.append(f"**Alerted at:** {_now_eastern()}")
return _truncate("\n".join(lines))
def format_auto_resolved_alert(resolved_patterns: list[dict]) -> str:
"""Format a consolidated auto-resolved alert.
Args:
resolved_patterns: List of dicts, each with keys: service, message, hours_silent.
Only the first 8 are shown; message is truncated to 60 chars.
Returns:
Formatted Discord message string, truncated to 2000 chars.
"""
lines = [
f"✅ **Patterns Auto-Resolved**",
f"",
]
shown = resolved_patterns[:8]
if shown:
for p in shown:
svc = p.get("service", "unknown")
msg = p.get("message", "")[:60]
hours = p.get("hours_silent", 0)
lines.append(f" • `{svc}` — {msg} (silent {hours}h)")
else:
lines.append(" *(no patterns)*")
if len(resolved_patterns) > 8:
lines.append(f" … and {len(resolved_patterns) - 8} more")
lines.append("")
lines.append(f"**Resolved at:** {_now_eastern()}")
return _truncate("\n".join(lines))
def format_spike_alert(
service: str,
normalized_message: str,
current_count: int,
rolling_average: float,
) -> str:
"""Format a Discord alert for an error rate spike.
Args:
service: Name of the service.
normalized_message: Normalised error pattern string.
current_count: Error count in the current window.
rolling_average: Baseline rolling average count.
Returns:
Formatted Discord message string, truncated to 2000 chars.
"""
if rolling_average > 0:
multiplier = current_count / rolling_average
multiplier_str = f"{multiplier:.1f}x"
else:
multiplier_str = "∞x"
lines = [
f"⚠️ **Error Rate Spike Detected**",
f"",
f"**Service:** {service}",
f"**Pattern:** {normalized_message}",
f"**Current count:** {current_count}",
f"**Normal (rolling avg):** {rolling_average}",
f"**Multiplier:** {multiplier_str} above normal",
f"",
f"**Alerted at:** {_now_eastern()}",
]
return _truncate("\n".join(lines))
def format_daily_digest(
total_errors: int,
new_patterns: int,
resolved_patterns: int,
active_patterns: int,
per_service_counts: dict,
) -> str:
"""Format the daily 24h error digest.
Args:
total_errors: Total error count in the last 24 hours.
new_patterns: Number of new error patterns detected.
resolved_patterns: Number of patterns auto-resolved.
active_patterns: Total number of currently active patterns.
per_service_counts: Mapping of service name → error count.
Displayed sorted by count descending.
Returns:
Formatted Discord message string, truncated to 2000 chars.
"""
lines = [
f"📊 **Daily Error Digest (24h)**",
f"",
f"**Total errors:** {total_errors}",
f"**New patterns:** {new_patterns}",
f"**Resolved patterns:** {resolved_patterns}",
f"**Active patterns:** {active_patterns}",
]
if per_service_counts:
lines.append("")
lines.append("**Per-service breakdown:**")
sorted_services = sorted(per_service_counts.items(), key=lambda kv: kv[1], reverse=True)
for svc, cnt in sorted_services:
lines.append(f" • `{svc}`: {cnt}")
lines.append("")
lines.append(f"**Generated at:** {_now_eastern()}")
return _truncate("\n".join(lines))
# ---------------------------------------------------------------------------
# Discord webhook client
# ---------------------------------------------------------------------------
class ErrorMonitorDiscord:
"""Webhook client for sending error monitor alerts to Discord.
Enforces a per-run rate limit of MAX_DISCORD_MESSAGES_PER_RUN to avoid
flooding the channel during high-volume incident windows.
Attributes:
webhook_url: The Discord incoming webhook URL.
messages_sent: Counter of messages attempted this run (success or failure).
"""
def __init__(self, webhook_url: str) -> None:
self.webhook_url = webhook_url
self.messages_sent: int = 0
def send_message(self, content: str) -> bool:
"""Send a message to the configured Discord webhook.
Rate-limits: if messages_sent >= MAX_DISCORD_MESSAGES_PER_RUN, the
message is silently skipped and False is returned without incrementing
the counter.
Args:
content: The message text to send (should be pre-formatted and
pre-truncated to ≤2000 chars).
Returns:
True if the message was sent successfully, False otherwise.
"""
if self.messages_sent >= MAX_DISCORD_MESSAGES_PER_RUN:
logger.warning(
"Discord rate limit reached (%d messages sent this run); skipping message.",
self.messages_sent,
)
return False
try:
response = requests.post(
self.webhook_url,
json={"content": content, "flags": 4}, # SUPPRESS_EMBEDS
timeout=30,
)
response.raise_for_status()
logger.debug("Discord message sent successfully.")
return True
except Exception as exc:
logger.error("Failed to send Discord message: %s", exc)
return False
finally:
self.messages_sent += 1
@@ -1,489 +0,0 @@
"""Firestore adapter for the production error monitoring service.
Handles all CRUD for three Firestore collections:
- error_patterns: normalized error patterns (pattern_id as doc ID)
- error_incidents: groups of related patterns (incident_id as doc ID)
- discord_alerts: audit trail for Discord messages (alert_id as doc ID)
"""
from __future__ import annotations
import secrets
from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
from typing import Any
from backend.services.error_monitor.config import (
AUTO_RESOLVE_FALLBACK_HOURS,
AUTO_RESOLVE_MAX_HOURS,
AUTO_RESOLVE_MIN_HOURS,
AUTO_RESOLVE_MULTIPLIER,
ROLLING_WINDOW_DAYS,
)
# ---------------------------------------------------------------------------
# Data classes
# ---------------------------------------------------------------------------
@dataclass
class PatternData:
"""Input data for creating or updating an error pattern."""
pattern_id: str
service: str
resource_type: str
normalized_message: str
sample_message: str
count: int
timestamp: datetime
@dataclass
class UpsertResult:
"""Result of an upsert_pattern operation."""
pattern_id: str
is_new: bool
previous_status: str | None = None
# ---------------------------------------------------------------------------
# Collection names
# ---------------------------------------------------------------------------
_COL_PATTERNS = "error_patterns"
_COL_INCIDENTS = "error_incidents"
_COL_ALERTS = "discord_alerts"
# Statuses that can be queried as "active"
_ACTIVE_STATUSES = ["new", "acknowledged", "known"]
# Statuses from which a pattern is reactivated when the error recurs
_REACTIVATION_STATUSES = {"auto_resolved", "fixed"}
# Statuses eligible for auto-resolve checking
_AUTO_RESOLVE_ELIGIBLE = {"new", "acknowledged"}
def _utcnow() -> datetime:
"""Return current UTC datetime."""
return datetime.now(tz=timezone.utc)
def _iso(dt: datetime) -> str:
"""Serialise a datetime to ISO-8601 string."""
return dt.isoformat()
def _generate_id(prefix: str, fmt: str) -> str:
"""Generate a time-stamped ID with 8 hex random suffix.
Args:
prefix: e.g. "inc" or "alert"
fmt: strftime format string applied to utcnow(), e.g. "%Y%m%d_%H%M"
"""
ts = _utcnow().strftime(fmt)
suffix = secrets.token_hex(4) # 8 hex chars
return f"{prefix}_{ts}_{suffix}"
# ---------------------------------------------------------------------------
# Adapter
# ---------------------------------------------------------------------------
class ErrorPatternsAdapter:
"""Firestore CRUD adapter for error monitoring collections.
Args:
db: Optional Firestore client. If omitted, a default client for the
``nomadkaraoke`` GCP project is constructed on first use.
"""
def __init__(self, db=None) -> None:
if db is None:
from google.cloud import firestore # type: ignore[import]
db = firestore.Client(project="nomadkaraoke")
self._db = db
# ------------------------------------------------------------------
# Patterns — read
# ------------------------------------------------------------------
def get_pattern(self, pattern_id: str) -> dict | None:
"""Return the pattern document as a dict, or None if it doesn't exist.
Args:
pattern_id: Firestore document ID in the ``error_patterns`` collection.
"""
snap = self._db.collection(_COL_PATTERNS).document(pattern_id).get()
if not snap.exists:
return None
return snap.to_dict()
def get_active_patterns(self) -> list[dict]:
"""Return all patterns with status in [new, acknowledged, known]."""
query = self._db.collection(_COL_PATTERNS).where(
"status", "in", _ACTIVE_STATUSES
)
return [snap.to_dict() for snap in query.stream()]
def get_patterns_for_auto_resolve(self) -> list[dict]:
"""Return active patterns eligible for auto-resolution (new or acknowledged)."""
query = self._db.collection(_COL_PATTERNS).where(
"status", "in", list(_AUTO_RESOLVE_ELIGIBLE)
)
return [snap.to_dict() for snap in query.stream()]
# ------------------------------------------------------------------
# Patterns — upsert
# ------------------------------------------------------------------
def upsert_pattern(self, data: PatternData) -> UpsertResult:
"""Create a new pattern or update an existing one.
New pattern:
- status="new", severity="P3"
- Initialize ``rolling_counts`` with one entry for this run.
Existing pattern:
- Increment ``total_count`` and append to ``rolling_counts``.
- If status was ``auto_resolved`` or ``fixed`` → reactivate (set status="new").
- Prune ``rolling_counts`` entries older than ROLLING_WINDOW_DAYS.
Returns:
UpsertResult with ``is_new=True`` for brand-new patterns *and* for
reactivated patterns (auto_resolved → new, fixed → new).
"""
doc_ref = self._db.collection(_COL_PATTERNS).document(data.pattern_id)
snap = doc_ref.get()
ts_iso = _iso(data.timestamp)
new_rolling_entry = {"ts": ts_iso, "count": data.count}
if not snap.exists:
# ── brand new pattern ──────────────────────────────────────────
doc_ref.set(
{
"pattern_id": data.pattern_id,
"service": data.service,
"resource_type": data.resource_type,
"normalized_message": data.normalized_message,
"sample_message": data.sample_message,
"status": "new",
"severity": "P3",
"total_count": data.count,
"rolling_counts": [new_rolling_entry],
"first_seen": ts_iso,
"last_seen": ts_iso,
"alerted_at": None,
"created_at": ts_iso,
"updated_at": ts_iso,
}
)
return UpsertResult(pattern_id=data.pattern_id, is_new=True)
# ── existing pattern ───────────────────────────────────────────────
existing = snap.to_dict()
prev_status = existing.get("status", "new")
rolling_counts: list[dict] = list(existing.get("rolling_counts") or [])
rolling_counts.append(new_rolling_entry)
rolling_counts = self._prune_rolling_counts(rolling_counts)
update: dict[str, Any] = {
"total_count": existing.get("total_count", 0) + data.count,
"rolling_counts": rolling_counts,
"last_seen": ts_iso,
"sample_message": data.sample_message,
"updated_at": ts_iso,
}
# Reactivation: pattern was considered resolved/fixed but error recurred
is_new = False
if prev_status in _REACTIVATION_STATUSES:
update["status"] = "new"
update["alerted_at"] = None
is_new = True
doc_ref.update(update)
return UpsertResult(
pattern_id=data.pattern_id,
is_new=is_new,
previous_status=prev_status,
)
# ------------------------------------------------------------------
# Patterns — status changes
# ------------------------------------------------------------------
def auto_resolve_pattern(self, pattern_id: str, hours_silent: float) -> None:
"""Mark a pattern as auto-resolved.
Args:
pattern_id: Firestore document ID.
hours_silent: How many hours the pattern has been silent.
"""
self._db.collection(_COL_PATTERNS).document(pattern_id).update(
{
"status": "auto_resolved",
"auto_resolve_hours_silent": hours_silent,
"auto_resolved_at": _iso(_utcnow()),
"updated_at": _iso(_utcnow()),
}
)
def resolve_pattern(
self,
pattern_id: str,
pr_url: str | None = None,
note: str = "",
) -> None:
"""Mark a pattern as fixed (human-resolved).
Args:
pattern_id: Firestore document ID.
pr_url: Optional URL of the PR that fixed this pattern.
note: Optional human-readable note about the fix.
"""
now_iso = _iso(_utcnow())
self._db.collection(_COL_PATTERNS).document(pattern_id).update(
{
"status": "fixed",
"fixed_by": {
"pr_url": pr_url,
"note": note,
"fixed_at": now_iso,
},
"updated_at": now_iso,
}
)
def update_pattern_alerted(self, pattern_id: str) -> None:
"""Set the ``alerted_at`` timestamp on a pattern.
Called after a Discord alert has been successfully sent for the pattern.
"""
self._db.collection(_COL_PATTERNS).document(pattern_id).update(
{
"alerted_at": _iso(_utcnow()),
"updated_at": _iso(_utcnow()),
}
)
def merge_pattern(self, source_id: str, target_id: str, reason: str) -> None:
"""Merge *source* into *target* by muting the source pattern.
Args:
source_id: Pattern to mute.
target_id: Pattern that the source is being merged into.
reason: Human-readable rationale for the merge.
"""
self._db.collection(_COL_PATTERNS).document(source_id).update(
{
"status": "muted",
"merged_into": target_id,
"merge_reason": reason,
"updated_at": _iso(_utcnow()),
}
)
# ------------------------------------------------------------------
# Auto-resolve logic
# ------------------------------------------------------------------
def check_auto_resolve(self, pattern: dict) -> float | None:
"""Determine whether *pattern* should be auto-resolved.
Computes the frequency-aware silence threshold and compares it against
how long the pattern has been silent.
Args:
pattern: A pattern document dict (as returned by get_pattern).
Returns:
The number of hours the pattern has been silent if the threshold is
exceeded, or ``None`` if the pattern should not yet be resolved.
"""
status = pattern.get("status", "")
if status not in _AUTO_RESOLVE_ELIGIBLE:
return None
last_seen_str = pattern.get("last_seen")
if not last_seen_str:
return None
last_seen = datetime.fromisoformat(last_seen_str)
now = datetime.now(tz=timezone.utc)
# Ensure both datetimes are tz-aware for subtraction
if last_seen.tzinfo is None:
last_seen = last_seen.replace(tzinfo=timezone.utc)
hours_silent = (now - last_seen).total_seconds() / 3600.0
rolling_counts = pattern.get("rolling_counts") or []
threshold = self._compute_resolve_threshold(rolling_counts)
if hours_silent >= threshold:
return hours_silent
return None
def _compute_resolve_threshold(self, rolling_counts: list[dict]) -> float:
"""Compute the frequency-aware auto-resolve threshold in hours.
Algorithm:
1. If fewer than 3 data points → return AUTO_RESOLVE_FALLBACK_HOURS (48).
2. Compute mean interval between consecutive timestamps.
3. threshold = mean_interval_hours * AUTO_RESOLVE_MULTIPLIER.
4. Clamp result to [AUTO_RESOLVE_MIN_HOURS, AUTO_RESOLVE_MAX_HOURS].
Args:
rolling_counts: List of ``{"ts": ISO-string, "count": int}`` dicts.
Returns:
Threshold in hours (float).
"""
if len(rolling_counts) < 3:
return float(AUTO_RESOLVE_FALLBACK_HOURS)
# Parse timestamps and sort ascending
timestamps: list[datetime] = []
for entry in rolling_counts:
ts = datetime.fromisoformat(entry["ts"])
if ts.tzinfo is None:
ts = ts.replace(tzinfo=timezone.utc)
timestamps.append(ts)
timestamps.sort()
# Compute pairwise intervals
intervals_hours: list[float] = []
for i in range(1, len(timestamps)):
delta = (timestamps[i] - timestamps[i - 1]).total_seconds() / 3600.0
intervals_hours.append(delta)
if not intervals_hours:
return float(AUTO_RESOLVE_FALLBACK_HOURS)
mean_interval = sum(intervals_hours) / len(intervals_hours)
threshold = mean_interval * AUTO_RESOLVE_MULTIPLIER
# Clamp to configured bounds
threshold = max(float(AUTO_RESOLVE_MIN_HOURS), threshold)
threshold = min(float(AUTO_RESOLVE_MAX_HOURS), threshold)
return threshold
def _prune_rolling_counts(self, rolling_counts: list[dict]) -> list[dict]:
"""Remove rolling_count entries older than ROLLING_WINDOW_DAYS.
Args:
rolling_counts: List of ``{"ts": ISO-string, "count": int}`` dicts.
Returns:
Filtered list with only recent entries.
"""
if not rolling_counts:
return []
cutoff = _utcnow() - timedelta(days=ROLLING_WINDOW_DAYS)
result = []
for entry in rolling_counts:
ts = datetime.fromisoformat(entry["ts"])
if ts.tzinfo is None:
ts = ts.replace(tzinfo=timezone.utc)
if ts >= cutoff:
result.append(entry)
return result
# ------------------------------------------------------------------
# Incidents
# ------------------------------------------------------------------
def create_incident(
self,
title: str,
root_cause: str,
severity: str,
suggested_fix: str,
primary_service: str,
pattern_ids: list[str],
used_llm: bool,
) -> str:
"""Create a new incident document and link associated patterns.
Args:
title: Short human-readable title.
root_cause: LLM-generated or manual root-cause description.
severity: Severity string (e.g. "P1", "P2", "P3").
suggested_fix: Suggested remediation steps.
primary_service: The primary affected service.
pattern_ids: List of pattern_ids that belong to this incident.
used_llm: Whether LLM analysis was used to generate this incident.
Returns:
The generated incident_id string.
"""
incident_id = _generate_id("inc", "%Y%m%d_%H%M")
now_iso = _iso(_utcnow())
self._db.collection(_COL_INCIDENTS).document(incident_id).set(
{
"incident_id": incident_id,
"title": title,
"root_cause": root_cause,
"severity": severity,
"suggested_fix": suggested_fix,
"primary_service": primary_service,
"pattern_ids": pattern_ids,
"used_llm": used_llm,
"created_at": now_iso,
"updated_at": now_iso,
"status": "open",
}
)
# Link each pattern back to this incident
for pattern_id in pattern_ids:
self._db.collection(_COL_PATTERNS).document(pattern_id).update(
{
"incident_id": incident_id,
"updated_at": now_iso,
}
)
return incident_id
# ------------------------------------------------------------------
# Discord audit trail
# ------------------------------------------------------------------
def log_discord_alert(
self,
alert_type: str,
content: str,
success: bool,
metadata: dict | None = None,
) -> None:
"""Write an audit record to the ``discord_alerts`` collection.
Args:
alert_type: Category of alert (e.g. "new_pattern", "digest").
content: The message body that was (or would have been) sent.
success: Whether the Discord webhook call succeeded.
metadata: Optional additional data (pattern_ids, incident_id, etc.).
"""
alert_id = _generate_id("alert", "%Y%m%d_%H%M%S")
now_iso = _iso(_utcnow())
self._db.collection(_COL_ALERTS).document(alert_id).set(
{
"alert_id": alert_id,
"alert_type": alert_type,
"content": content,
"success": success,
"metadata": metadata or {},
"created_at": now_iso,
}
)
@@ -1,106 +0,0 @@
"""Known infrastructure noise patterns for the production error monitoring service.
Filters out transient, expected log messages so the error monitor is not flooded
with Cloud Run cold-start probes, deploy events, spot VM preemptions, and similar
operational noise.
Usage:
from backend.services.error_monitor.known_issues import should_ignore
reason = should_ignore(service, message)
if reason:
# skip this log entry — it's known infrastructure noise
...
"""
import re
from dataclasses import dataclass
@dataclass
class IgnoreReason:
"""Describes why a log entry should be suppressed.
Attributes:
pattern_name: Machine-readable name of the matching ignore pattern.
reason: Human-readable explanation of why this is infrastructure noise.
"""
pattern_name: str
reason: str
# ---------------------------------------------------------------------------
# Compiled ignore patterns (applied in order)
# All patterns use re.IGNORECASE for case-insensitive matching.
# ---------------------------------------------------------------------------
_IGNORE_PATTERNS: list[tuple[str, re.Pattern, str]] = [
(
"startup_probe",
re.compile(r"startup probe failed", re.IGNORECASE),
"Cloud Run cold start — startup probe failure is transient and expected",
),
(
"ready_condition",
re.compile(r"ready condition status changed", re.IGNORECASE),
"Cloud Run deploy event — readiness condition change is expected on deploy",
),
(
"spot_preemption",
re.compile(r"instance was preempted", re.IGNORECASE),
"Spot/preemptible VM preemption — expected behaviour for spot instances",
),
(
"idle_shutdown",
re.compile(r"stopping idle encoding worker", re.IGNORECASE),
"Encoding worker idle shutdown — expected behaviour when no jobs are queued",
),
(
"health_check_404",
re.compile(
r"(?:GET|HEAD)\b[^\n]*?/health[^\n]*?\b(?:404|not found)\b",
re.IGNORECASE,
),
"Load balancer health check returning 404 — transient during cold start",
),
(
"scheduler_retry",
re.compile(r"cloud scheduler.*retry", re.IGNORECASE),
"Cloud Scheduler transient retry — expected on intermittent job failures",
),
(
"runner_startup",
re.compile(r"github\.runner\..*(?:starting|stopping|idle)", re.IGNORECASE),
"GitHub Actions runner lifecycle event — expected during runner start/stop",
),
(
"container_shutdown",
re.compile(r"container called exit\(0\)", re.IGNORECASE),
"Container clean exit — exit code 0 indicates a graceful, expected shutdown",
),
]
def should_ignore(service: str, message: str) -> IgnoreReason | None:
"""Check whether a log entry matches a known infrastructure noise pattern.
The check is purely message-based; ``service`` is accepted for API symmetry
and future service-specific overrides but does not gate any pattern today.
Args:
service: The originating service name (e.g. ``"karaoke-backend"``).
message: The raw log message string to evaluate.
Returns:
An :class:`IgnoreReason` if the message matches a known noise pattern,
or ``None`` if the message should be processed normally.
"""
if not message:
return None
for pattern_name, pattern, reason in _IGNORE_PATTERNS:
if pattern.search(message):
return IgnoreReason(pattern_name=pattern_name, reason=reason)
return None
@@ -1,429 +0,0 @@
"""LLM-powered incident analysis and duplicate detection for the error monitor.
Uses Gemini Flash via the google-generativeai library to:
1. Group related error patterns into incidents with root-cause analysis.
2. Identify near-duplicate patterns that the regex normalizer missed.
Both functions fall back gracefully when the LLM is unavailable or returns
unparseable output.
"""
from __future__ import annotations
import json
import logging
import re
import time
from collections import defaultdict
from dataclasses import dataclass, field
from google import genai
from google.genai import types as genai_types
from backend.services.error_monitor.config import (
GCP_PROJECT,
LLM_ANALYSIS_MODEL,
LLM_VERTEX_LOCATION,
MIN_PATTERNS_FOR_ANALYSIS,
SERVICE_DEPENDENCY_MAP,
)
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Dataclasses
# ---------------------------------------------------------------------------
@dataclass
class Incident:
"""A grouped incident produced by LLM analysis or the fallback grouper."""
title: str
root_cause: str | None
severity: str # P0 / P1 / P2 / P3
suggested_fix: str | None
primary_service: str
pattern_indices: list[int]
@dataclass
class IncidentAnalysis:
"""The result of an incident grouping pass."""
incidents: list[Incident]
used_llm: bool = True
@dataclass
class DuplicateGroup:
"""A canonical pattern plus the new patterns that duplicate it."""
canonical_index: int
duplicate_indices: list[int]
reason: str
# ---------------------------------------------------------------------------
# LLM call helper
# ---------------------------------------------------------------------------
_RETRY_ATTEMPTS = 3
_RETRY_BASE_DELAY = 2.0 # seconds
def _call_llm(
system_prompt: str,
user_prompt: str,
temperature: float = 0.2,
) -> str:
"""Call Gemini and return the response text.
Retries up to _RETRY_ATTEMPTS times with exponential back-off on failure.
Args:
system_prompt: System instruction for the model.
user_prompt: User-facing prompt content.
temperature: Sampling temperature (lower = more deterministic).
Returns:
The model's response text.
Raises:
Exception: Re-raises the last exception after all retries are exhausted.
"""
last_exc: Exception | None = None
for attempt in range(_RETRY_ATTEMPTS):
try:
client = genai.Client(
vertexai=True,
project=GCP_PROJECT,
location=LLM_VERTEX_LOCATION,
)
response = client.models.generate_content(
model=LLM_ANALYSIS_MODEL,
contents=user_prompt,
config=genai_types.GenerateContentConfig(
system_instruction=system_prompt,
temperature=temperature,
),
)
return response.text
except Exception as exc: # noqa: BLE001
last_exc = exc
delay = _RETRY_BASE_DELAY * (2**attempt)
logger.warning(
"LLM call failed (attempt %d/%d): %s — retrying in %.1fs",
attempt + 1,
_RETRY_ATTEMPTS,
exc,
delay,
)
time.sleep(delay)
raise last_exc # type: ignore[misc]
# ---------------------------------------------------------------------------
# Response parsing
# ---------------------------------------------------------------------------
_MD_CODE_BLOCK_RE = re.compile(r"```(?:json)?\s*([\s\S]*?)```", re.IGNORECASE)
def _strip_markdown_code_block(text: str) -> str:
"""Remove surrounding ```json ... ``` or ``` ... ``` fences if present."""
match = _MD_CODE_BLOCK_RE.search(text)
if match:
return match.group(1).strip()
return text.strip()
def _parse_llm_response(text: str) -> IncidentAnalysis | None:
"""Parse the LLM response text into an IncidentAnalysis.
Args:
text: Raw LLM output, possibly wrapped in a markdown code block.
Returns:
IncidentAnalysis on success, None on any parse failure.
"""
if not text:
return None
cleaned = _strip_markdown_code_block(text)
try:
data = json.loads(cleaned)
except json.JSONDecodeError:
logger.warning("LLM returned non-JSON response (first 200 chars): %.200s", text)
return None
if "incidents" not in data:
logger.warning("LLM response missing 'incidents' key: %s", list(data.keys()))
return None
incidents: list[Incident] = []
for raw in data["incidents"]:
try:
incidents.append(
Incident(
title=raw.get("title", "Unknown incident"),
root_cause=raw.get("root_cause"),
severity=raw.get("severity", "P2"),
suggested_fix=raw.get("suggested_fix"),
primary_service=raw.get("primary_service", "unknown"),
pattern_indices=raw.get("pattern_indices", []),
)
)
except Exception as exc: # noqa: BLE001
logger.warning("Skipping malformed incident entry: %s — %s", raw, exc)
return IncidentAnalysis(incidents=incidents, used_llm=True)
# ---------------------------------------------------------------------------
# Fallback grouper
# ---------------------------------------------------------------------------
def _fallback_group_by_service(patterns: list[dict]) -> IncidentAnalysis:
"""Group patterns by service name when the LLM is unavailable.
Each unique service becomes one P2 incident containing all of that
service's pattern indices.
Args:
patterns: List of error pattern dicts (must include 'service' key).
Returns:
IncidentAnalysis with used_llm=False.
"""
service_to_indices: dict[str, list[int]] = defaultdict(list)
for idx, pattern in enumerate(patterns):
service = pattern.get("service", "unknown")
service_to_indices[service].append(idx)
incidents = [
Incident(
title=f"Errors in {service}",
root_cause=None,
severity="P2",
suggested_fix=None,
primary_service=service,
pattern_indices=indices,
)
for service, indices in service_to_indices.items()
]
return IncidentAnalysis(incidents=incidents, used_llm=False)
# ---------------------------------------------------------------------------
# Main analysis functions
# ---------------------------------------------------------------------------
_ANALYZE_SYSTEM_PROMPT = f"""\
You are an expert site-reliability engineer for Nomad Karaoke, a karaoke video
generation platform running on Google Cloud Platform.
Your task is to analyse a list of active error patterns detected across
production services, group them into meaningful incidents, and identify root
causes.
## Service dependency map
{SERVICE_DEPENDENCY_MAP}
## Severity definitions
- P0: Service is completely down / total outage
- P1: Major pipeline broken — orders cannot complete
- P2: Degraded performance — some orders affected
- P3: Minor / cosmetic issue with no direct user impact
## Output format
Respond with a single JSON object (no markdown fencing) with this schema:
{{
"incidents": [
{{
"title": "<short human-readable title>",
"root_cause": "<root cause or null>",
"severity": "P0|P1|P2|P3",
"suggested_fix": "<actionable suggestion or null>",
"primary_service": "<service name>",
"pattern_indices": [<list of 0-based indices from the input list>]
}}
]
}}
Group related patterns together into a single incident where they share a
common root cause (e.g., a downstream dependency failure). Unrelated patterns
should be in separate incidents. Every input pattern must appear in exactly
one incident.
"""
_DEDUP_SYSTEM_PROMPT = """\
You are an expert at identifying near-duplicate error patterns in production
monitoring systems.
Your task is to compare a list of NEW error patterns against a list of
EXISTING (canonical) patterns and identify which new patterns are essentially
duplicates of existing ones — i.e., they describe the same underlying error
but with slightly different wording due to variation in log messages.
## Output format
Respond with a single JSON object (no markdown fencing) with this schema:
{
"duplicates": [
{
"canonical_index": <0-based index into the EXISTING patterns list>,
"duplicate_indices": [<0-based indices into the NEW patterns list>],
"reason": "<brief explanation>"
}
]
}
If no duplicates are found, return {"duplicates": []}.
"""
def analyze_patterns(patterns: list[dict]) -> IncidentAnalysis | None:
"""Group error patterns into incidents using Gemini LLM analysis.
Returns None if there are fewer than MIN_PATTERNS_FOR_ANALYSIS patterns.
Falls back to service-based grouping when the LLM fails or returns an
unparseable response.
Args:
patterns: List of error pattern dicts to analyse.
Returns:
IncidentAnalysis, or None if insufficient patterns.
"""
if len(patterns) < MIN_PATTERNS_FOR_ANALYSIS:
return None
user_prompt = _build_analyze_user_prompt(patterns)
try:
raw_response = _call_llm(_ANALYZE_SYSTEM_PROMPT, user_prompt, temperature=0.2)
result = _parse_llm_response(raw_response)
if result is not None:
return result
logger.warning("LLM returned unparseable response; falling back to service grouping")
except Exception as exc: # noqa: BLE001
logger.warning("LLM analysis failed: %s — falling back to service grouping", exc)
return _fallback_group_by_service(patterns)
def find_duplicate_patterns(
new_patterns: list[dict],
existing_patterns: list[dict],
) -> list[DuplicateGroup]:
"""Identify near-duplicate patterns that the regex normalizer missed.
Returns an empty list if either input list is empty, or if the LLM call
fails for any reason.
Args:
new_patterns: Newly detected error patterns to check.
existing_patterns: Currently tracked (canonical) error patterns.
Returns:
List of DuplicateGroup objects (may be empty).
"""
if not new_patterns or not existing_patterns:
return []
user_prompt = _build_dedup_user_prompt(new_patterns, existing_patterns)
try:
raw_response = _call_llm(_DEDUP_SYSTEM_PROMPT, user_prompt, temperature=0.1)
return _parse_duplicate_response(raw_response)
except Exception as exc: # noqa: BLE001
logger.warning("Duplicate detection LLM call failed: %s", exc)
return []
# ---------------------------------------------------------------------------
# Prompt builders
# ---------------------------------------------------------------------------
def _build_analyze_user_prompt(patterns: list[dict]) -> str:
"""Build the user prompt listing patterns for incident grouping."""
lines = ["Error patterns to analyse:\n"]
for idx, pattern in enumerate(patterns):
service = pattern.get("service", "unknown")
message = pattern.get("normalized_message", "")
count = pattern.get("count", 0)
lines.append(f"[{idx}] service={service!r} count={count} message={message!r}")
return "\n".join(lines)
def _build_dedup_user_prompt(
new_patterns: list[dict],
existing_patterns: list[dict],
) -> str:
"""Build the user prompt listing existing and new patterns for dedup."""
lines = ["EXISTING (canonical) patterns:\n"]
for idx, pattern in enumerate(existing_patterns):
service = pattern.get("service", "unknown")
message = pattern.get("normalized_message", "")
lines.append(f"[{idx}] service={service!r} message={message!r}")
lines.append("\nNEW patterns to check:\n")
for idx, pattern in enumerate(new_patterns):
service = pattern.get("service", "unknown")
message = pattern.get("normalized_message", "")
lines.append(f"[{idx}] service={service!r} message={message!r}")
return "\n".join(lines)
def _parse_duplicate_response(text: str) -> list[DuplicateGroup]:
"""Parse the LLM response for duplicate detection.
Args:
text: Raw LLM output.
Returns:
List of DuplicateGroup objects, or [] on any failure.
"""
if not text:
return []
cleaned = _strip_markdown_code_block(text)
try:
data = json.loads(cleaned)
except json.JSONDecodeError:
logger.warning("Dedup LLM returned non-JSON: %.200s", text)
return []
if "duplicates" not in data:
logger.warning("Dedup LLM response missing 'duplicates' key")
return []
groups: list[DuplicateGroup] = []
for raw in data["duplicates"]:
try:
groups.append(
DuplicateGroup(
canonical_index=int(raw["canonical_index"]),
duplicate_indices=list(raw.get("duplicate_indices", [])),
reason=raw.get("reason", ""),
)
)
except Exception as exc: # noqa: BLE001
logger.warning("Skipping malformed duplicate entry: %s — %s", raw, exc)
return groups
-641
View File
@@ -1,641 +0,0 @@
"""Monitor orchestrator for the production error monitoring service.
This is the main entry point. Run as::
python -m backend.services.error_monitor.monitor
The ``ErrorMonitor.run()`` method executes the full pipeline:
* Monitor cycle (default) — query Cloud Logging, detect new patterns, send
Discord alerts, check auto-resolution.
* Daily digest mode (``DIGEST_MODE=true``) — bundle 24h stats into one message.
"""
from __future__ import annotations
import logging
import os
from datetime import datetime, timedelta, timezone
from typing import Any
from google.cloud import logging as cloud_logging
from google.cloud import secretmanager
from backend.services.error_monitor.config import (
GCP_PROJECT,
LOOKBACK_MINUTES,
MAX_LOG_ENTRIES,
MONITORED_CLOUD_FUNCTIONS,
MONITORED_CLOUD_RUN_JOBS,
MONITORED_CLOUD_RUN_SERVICES,
MONITORED_GCE_INSTANCES,
SPIKE_MIN_COUNT,
SPIKE_MULTIPLIER,
get_digest_mode,
get_discord_webhook_secret_name,
get_llm_enabled,
)
from backend.services.error_monitor.discord import (
ErrorMonitorDiscord,
format_auto_resolved_alert,
format_daily_digest,
format_incident_alert,
format_new_pattern_alert,
format_spike_alert,
)
from backend.services.error_monitor.firestore_adapter import (
ErrorPatternsAdapter,
PatternData,
UpsertResult,
)
from backend.services.error_monitor.known_issues import should_ignore
from backend.services.error_monitor.llm_analysis import analyze_patterns, find_duplicate_patterns
from backend.services.error_monitor.normalizer import compute_pattern_hash, normalize_message
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Module-level constants
# ---------------------------------------------------------------------------
#: Maps log resource type → (label_key, service_name_list)
RESOURCE_TYPE_FILTERS: dict[str, tuple[str, list[str]]] = {
"cloud_run_revision": ("resource.labels.service_name", MONITORED_CLOUD_RUN_SERVICES),
"cloud_run_job": ("resource.labels.job_name", MONITORED_CLOUD_RUN_JOBS),
"cloud_function": ("resource.labels.function_name", MONITORED_CLOUD_FUNCTIONS),
"gce_instance": ('labels."compute.googleapis.com/resource_name"', MONITORED_GCE_INSTANCES),
}
#: Maps log resource type → canonical resource_type string stored in Firestore.
RESOURCE_TYPE_MAP: dict[str, str] = {
"cloud_run_revision": "cloud_run_service",
"cloud_run_job": "cloud_run_job",
"cloud_function": "cloud_function",
"gce_instance": "gce_instance",
}
# ---------------------------------------------------------------------------
# Free functions
# ---------------------------------------------------------------------------
def _build_log_filter(
resource_type: str,
resource_names: list[str],
lookback_minutes: int,
) -> str:
"""Build a Cloud Logging filter string.
Args:
resource_type: GCP resource type (e.g. ``"cloud_run_revision"``).
resource_names: List of service/job/function/instance names to include.
lookback_minutes: How many minutes back to query.
Returns:
A filter string suitable for ``logging_client.list_entries(filter_=...)``.
"""
label_key = RESOURCE_TYPE_FILTERS[resource_type][0]
cutoff = datetime.now(tz=timezone.utc) - timedelta(minutes=lookback_minutes)
cutoff_iso = cutoff.strftime("%Y-%m-%dT%H:%M:%SZ")
name_clause = " OR ".join(f'{label_key}="{name}"' for name in resource_names)
if len(resource_names) > 1:
name_clause = f"({name_clause})"
return (
f'resource.type="{resource_type}" '
f"severity>=ERROR "
f'timestamp>="{cutoff_iso}" '
f"{name_clause}"
)
def _extract_message(entry: Any) -> str | None:
"""Extract the log message text from a Cloud Logging entry.
Handles several payload shapes:
* ``str`` payload — returned directly.
* ``dict`` payload — tries ``"message"``, ``"textPayload"``, ``"error"``,
``"msg"`` keys in that order.
* ``None`` payload — returns ``None``.
Args:
entry: A Cloud Logging ``LogEntry`` object.
Returns:
The message string, or ``None`` if not extractable.
"""
payload = entry.payload
if payload is None:
return None
if isinstance(payload, str):
return payload if payload else None
if isinstance(payload, dict):
for key in ("message", "textPayload", "error", "msg"):
value = payload.get(key)
if value:
return str(value)
return None
# Fallback: convert to string representation
text = str(payload)
return text if text else None
def _extract_service_name(entry: Any, resource_type: str) -> str:
"""Extract the service/job/function/instance name from a log entry.
Args:
entry: A Cloud Logging ``LogEntry`` object.
resource_type: GCP log resource type string.
Returns:
The service name string (falls back to ``"unknown"`` if not found).
"""
labels = getattr(entry.resource, "labels", {}) or {}
entry_labels = getattr(entry, "labels", {}) or {}
if resource_type == "cloud_run_revision":
return labels.get("service_name") or "unknown"
if resource_type == "cloud_run_job":
return labels.get("job_name") or "unknown"
if resource_type == "cloud_function":
return labels.get("function_name") or "unknown"
if resource_type == "gce_instance":
# Prefer the human-readable resource name label
name = entry_labels.get("compute.googleapis.com/resource_name")
if name:
return name
return labels.get("instance_id") or "unknown"
return "unknown"
def _classify_resource_type(log_resource_type: str) -> str:
"""Map a Cloud Logging resource type to the canonical type stored in Firestore.
Args:
log_resource_type: Raw GCP resource type (e.g. ``"cloud_run_revision"``).
Returns:
Canonical resource type string (falls back to the input if not in map).
"""
return RESOURCE_TYPE_MAP.get(log_resource_type, log_resource_type)
def _group_by_pattern(entries: list[dict]) -> dict[str, dict]:
"""Group normalised log entries by their pattern hash.
Args:
entries: List of dicts with ``service``, ``normalized``, and ``message`` keys.
Returns:
Dict mapping ``pattern_id`` → group dict containing ``service``,
``resource_type``, ``normalized``, ``sample_message``, ``count``.
"""
groups: dict[str, dict] = {}
for entry in entries:
service = entry["service"]
normalized = entry["normalized"]
pattern_id = compute_pattern_hash(service, normalized)
if pattern_id not in groups:
groups[pattern_id] = {
"pattern_id": pattern_id,
"service": service,
"resource_type": entry.get("resource_type", "unknown"),
"normalized": normalized,
"sample_message": entry["message"],
"count": 0,
}
groups[pattern_id]["count"] += 1
return groups
def _rolling_average(rolling_counts: list[dict]) -> float:
"""Compute the mean count from a list of rolling_count dicts.
Args:
rolling_counts: List of ``{"count": int, ...}`` dicts.
Returns:
The arithmetic mean as a float, or ``0.0`` for an empty list.
"""
if not rolling_counts:
return 0.0
total = sum(entry.get("count", 0) for entry in rolling_counts)
return total / len(rolling_counts)
def _is_spike(current_count: int, rolling_counts: list[dict]) -> bool:
"""Determine whether ``current_count`` represents an error rate spike.
A spike is detected when:
- ``current_count`` >= ``SPIKE_MIN_COUNT`` (avoids noise from rare errors), AND
- ``current_count`` > ``rolling_average * SPIKE_MULTIPLIER``
Args:
current_count: Error count in the current monitoring window.
rolling_counts: Historical rolling counts from Firestore.
Returns:
``True`` if a spike is detected, ``False`` otherwise.
"""
if not rolling_counts:
return False
if current_count < SPIKE_MIN_COUNT:
return False
avg = _rolling_average(rolling_counts)
if avg == 0.0:
return False
return current_count > avg * SPIKE_MULTIPLIER
# ---------------------------------------------------------------------------
# ErrorMonitor class
# ---------------------------------------------------------------------------
class ErrorMonitor:
"""Orchestrates the full production error monitoring pipeline.
On each run (typically every 15 minutes via Cloud Scheduler), it:
1. Queries Cloud Logging for recent errors across all monitored services.
2. Groups errors into patterns using normalisation + hashing.
3. Upserts patterns to Firestore, tracking new vs. recurrent.
4. Optionally uses an LLM to deduplicate patterns and create incidents.
5. Sends Discord alerts for new patterns, incidents, and spikes.
6. Auto-resolves patterns that have been silent long enough.
"""
def __init__(self) -> None:
self.logging_client = cloud_logging.Client(project=GCP_PROJECT)
self.firestore_adapter = ErrorPatternsAdapter()
self.discord = ErrorMonitorDiscord(webhook_url=self._get_discord_webhook())
def _get_discord_webhook(self) -> str:
"""Retrieve the Discord webhook URL.
Checks ``DISCORD_WEBHOOK_URL`` environment variable first; falls back
to Secret Manager.
Returns:
Discord webhook URL string.
"""
url = os.environ.get("DISCORD_WEBHOOK_URL")
if url:
return url
secret_name = get_discord_webhook_secret_name()
client = secretmanager.SecretManagerServiceClient()
secret_path = f"projects/{GCP_PROJECT}/secrets/{secret_name}/versions/latest"
response = client.access_secret_version(name=secret_path)
return response.payload.data.decode("utf-8").strip()
# ------------------------------------------------------------------
# Main entry point
# ------------------------------------------------------------------
def run(self) -> None:
"""Execute the monitor pipeline (digest or cycle mode)."""
if get_digest_mode():
self._run_daily_digest()
else:
self._run_monitor_cycle()
# ------------------------------------------------------------------
# Monitor cycle
# ------------------------------------------------------------------
def _run_monitor_cycle(self) -> None:
"""Execute one monitor cycle: collect → group → upsert → alert → auto-resolve."""
now = datetime.now(tz=timezone.utc)
now_iso = now.isoformat()
# ── Step 1: Collect log entries ────────────────────────────────────
raw_entries: list[dict] = []
for resource_type, (_, resource_names) in RESOURCE_TYPE_FILTERS.items():
log_filter = _build_log_filter(resource_type, resource_names, LOOKBACK_MINUTES)
logger.info("Querying %s with filter: %s", resource_type, log_filter)
try:
entries = self.logging_client.list_entries(
filter_=log_filter,
max_results=MAX_LOG_ENTRIES,
)
for entry in entries:
message = _extract_message(entry)
if not message:
continue
service = _extract_service_name(entry, resource_type)
ignore_reason = should_ignore(service, message)
if ignore_reason:
logger.debug(
"Ignoring entry from %s (%s): %s",
service,
ignore_reason.pattern_name,
ignore_reason.reason,
)
continue
normalized = normalize_message(message)
if not normalized:
continue
raw_entries.append(
{
"service": service,
"resource_type": _classify_resource_type(resource_type),
"normalized": normalized,
"message": message,
}
)
except Exception as exc: # noqa: BLE001
logger.error("Failed to query %s logs: %s", resource_type, exc)
logger.info("Collected %d log entries after filtering", len(raw_entries))
# ── Step 2: Group by pattern ───────────────────────────────────────
groups = _group_by_pattern(raw_entries)
logger.info("Grouped into %d distinct patterns", len(groups))
if not groups:
logger.info("No new error entries found this cycle; checking auto-resolve.")
self._check_auto_resolve()
return
# ── Step 3: Upsert to Firestore ────────────────────────────────────
new_pattern_ids: list[str] = []
spike_patterns: list[dict] = []
for pattern_id, group in groups.items():
data = PatternData(
pattern_id=pattern_id,
service=group["service"],
resource_type=group["resource_type"],
normalized_message=group["normalized"],
sample_message=group["sample_message"],
count=group["count"],
timestamp=now,
)
result: UpsertResult = self.firestore_adapter.upsert_pattern(data)
if result.is_new:
new_pattern_ids.append(pattern_id)
logger.info("New pattern detected: %s (%s)", pattern_id, group["service"])
# Check for spike on existing patterns
existing = self.firestore_adapter.get_pattern(pattern_id)
if existing and not result.is_new:
rolling = existing.get("rolling_counts") or []
if _is_spike(group["count"], rolling):
spike_patterns.append(
{
"service": group["service"],
"normalized_message": group["normalized"],
"current_count": group["count"],
"rolling_average": _rolling_average(rolling),
}
)
logger.info("Spike detected for pattern %s", pattern_id)
# ── Step 4: LLM dedup ─────────────────────────────────────────────
if new_pattern_ids and get_llm_enabled():
new_patterns_data = [
{
"service": groups[pid]["service"],
"normalized_message": groups[pid]["normalized"],
"count": groups[pid]["count"],
}
for pid in new_pattern_ids
]
existing_patterns = self.firestore_adapter.get_active_patterns()
# Exclude newly added patterns from the canonical list
canonical = [p for p in existing_patterns if p.get("pattern_id") not in new_pattern_ids]
if canonical:
duplicate_groups = find_duplicate_patterns(new_patterns_data, canonical)
for dup_group in duplicate_groups:
# Merge each duplicate new pattern into the canonical
canonical_pattern = canonical[dup_group.canonical_index]
canonical_id = canonical_pattern.get("pattern_id", "")
for dup_idx in dup_group.duplicate_indices:
if dup_idx < len(new_pattern_ids):
dup_id = new_pattern_ids[dup_idx]
logger.info(
"Merging duplicate pattern %s into %s: %s",
dup_id,
canonical_id,
dup_group.reason,
)
self.firestore_adapter.merge_pattern(dup_id, canonical_id, dup_group.reason)
# Remove from new_pattern_ids since it was merged
new_pattern_ids = [pid for pid in new_pattern_ids if pid != dup_id]
# ── Step 5: LLM incident analysis ─────────────────────────────────
incident_pattern_ids: set[str] = set()
if len(new_pattern_ids) >= 2 and get_llm_enabled():
patterns_for_analysis = [
{
"service": groups[pid]["service"],
"normalized_message": groups[pid]["normalized"],
"count": groups[pid]["count"],
}
for pid in new_pattern_ids
if pid in groups
]
analysis = analyze_patterns(patterns_for_analysis)
if analysis and analysis.incidents:
for incident in analysis.incidents:
incident_pids = [
new_pattern_ids[idx]
for idx in incident.pattern_indices
if idx < len(new_pattern_ids)
]
if not incident_pids:
continue
incident_id = self.firestore_adapter.create_incident(
title=incident.title,
root_cause=incident.root_cause or "",
severity=incident.severity,
suggested_fix=incident.suggested_fix or "",
primary_service=incident.primary_service,
pattern_ids=incident_pids,
used_llm=analysis.used_llm,
)
logger.info(
"Created incident %s (%s) with %d patterns",
incident_id,
incident.severity,
len(incident_pids),
)
# Build pattern list for the Discord alert
pattern_dicts = [
{
"service": groups[pid]["service"],
"normalized_message": groups[pid]["normalized"],
"count": groups[pid]["count"],
}
for pid in incident_pids
if pid in groups
]
msg = format_incident_alert(
title=incident.title,
severity=incident.severity,
root_cause=incident.root_cause or "Unknown",
suggested_fix=incident.suggested_fix or "Investigate logs",
primary_service=incident.primary_service,
patterns=pattern_dicts,
)
if self.discord.send_message(msg):
for pid in incident_pids:
try:
self.firestore_adapter.update_pattern_alerted(pid)
except Exception as exc: # noqa: BLE001
logger.warning("Could not update alerted_at for %s: %s", pid, exc)
incident_pattern_ids.update(incident_pids)
# ── Step 6: Individual alerts for new patterns not in incidents ────
for pattern_id in new_pattern_ids:
if pattern_id in incident_pattern_ids:
continue
if pattern_id not in groups:
continue
group = groups[pattern_id]
# Fetch fresh copy to get first_seen timestamp
existing = self.firestore_adapter.get_pattern(pattern_id)
first_seen = existing.get("first_seen", now_iso) if existing else now_iso
msg = format_new_pattern_alert(
service=group["service"],
resource_type=group["resource_type"],
normalized_message=group["normalized"],
sample_message=group["sample_message"],
count=group["count"],
first_seen=first_seen,
)
if self.discord.send_message(msg):
try:
self.firestore_adapter.update_pattern_alerted(pattern_id)
except Exception as exc: # noqa: BLE001
logger.warning("Could not update alerted_at for %s: %s", pattern_id, exc)
# ── Step 7: Spike alerts ───────────────────────────────────────────
for spike in spike_patterns:
msg = format_spike_alert(
service=spike["service"],
normalized_message=spike["normalized_message"],
current_count=spike["current_count"],
rolling_average=spike["rolling_average"],
)
self.discord.send_message(msg)
# ── Step 8: Auto-resolve check ─────────────────────────────────────
self._check_auto_resolve()
# ------------------------------------------------------------------
# Auto-resolve helper
# ------------------------------------------------------------------
def _check_auto_resolve(self) -> None:
"""Check all active patterns for auto-resolution and send a consolidated alert."""
patterns = self.firestore_adapter.get_patterns_for_auto_resolve()
resolved: list[dict] = []
for pattern in patterns:
pattern_id = pattern.get("pattern_id", "")
hours_silent = self.firestore_adapter.check_auto_resolve(pattern)
if hours_silent is not None:
logger.info(
"Auto-resolving pattern %s (silent %.1f hours)", pattern_id, hours_silent
)
self.firestore_adapter.auto_resolve_pattern(pattern_id, round(hours_silent, 1))
resolved.append(
{
"service": pattern.get("service", "unknown"),
"message": pattern.get("normalized_message", ""),
"hours_silent": round(hours_silent, 1),
}
)
if resolved:
msg = format_auto_resolved_alert(resolved)
self.discord.send_message(msg)
# ------------------------------------------------------------------
# Daily digest
# ------------------------------------------------------------------
def _run_daily_digest(self) -> None:
"""Gather 24h stats and send a daily digest to Discord."""
# Optional dedup sweep (best-effort; LLM errors are non-fatal)
if get_llm_enabled():
try:
active = self.firestore_adapter.get_active_patterns()
if len(active) >= 2:
mid = len(active) // 2
duplicates = find_duplicate_patterns(active[:mid], active[mid:])
for dup in duplicates:
for idx in dup.duplicate_indices:
target_half = active[mid:]
source_half = active[:mid]
if idx < len(target_half) and dup.canonical_index < len(source_half):
self.firestore_adapter.merge_pattern(
target_half[idx]["pattern_id"],
source_half[dup.canonical_index]["pattern_id"],
dup.reason,
)
except Exception as exc: # noqa: BLE001
logger.warning("Digest dedup sweep failed (non-fatal): %s", exc)
# Gather stats from active patterns
active_patterns = self.firestore_adapter.get_active_patterns()
per_service_counts: dict[str, int] = {}
new_count = 0
total_errors = 0
for pattern in active_patterns:
service = pattern.get("service", "unknown")
count = pattern.get("total_count", 0)
per_service_counts[service] = per_service_counts.get(service, 0) + count
total_errors += count
if pattern.get("status") == "new":
new_count += 1
# Count recently auto-resolved patterns
resolved_patterns = self.firestore_adapter.get_patterns_for_auto_resolve()
resolved_count = sum(
1
for p in resolved_patterns
if self.firestore_adapter.check_auto_resolve(p) is not None
)
msg = format_daily_digest(
total_errors=total_errors,
new_patterns=new_count,
resolved_patterns=resolved_count,
active_patterns=len(active_patterns),
per_service_counts=per_service_counts,
)
self.discord.send_message(msg)
# ---------------------------------------------------------------------------
# Entry point
# ---------------------------------------------------------------------------
def main() -> None:
"""Configure logging and run the error monitor."""
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
)
monitor = ErrorMonitor()
monitor.run()
if __name__ == "__main__":
main()
@@ -1,137 +0,0 @@
"""Message normalizer for the production error monitoring service.
Replaces variable parts of error messages (IDs, timestamps, paths, etc.)
with stable placeholders so the same logical error always produces the same
pattern hash, regardless of which specific resource or timestamp was involved.
Normalizer rules are applied in order — more specific patterns first.
"""
import hashlib
import re
from backend.services.error_monitor.config import MAX_NORMALIZED_MESSAGE_LENGTH
# ---------------------------------------------------------------------------
# Compiled regex patterns (applied in order — more specific first)
# ---------------------------------------------------------------------------
_NORMALIZERS: list[tuple[re.Pattern, str]] = [
# 1. GCS paths: gs://bucket/path/to/object → gs://<BUCKET>/<PATH>
# Must come before Firestore and URL normalization.
(
re.compile(r"gs://([^/\s]+)/([^\s]+)"),
r"gs://<BUCKET>/<PATH>",
),
# 1b. GCS bucket root (no object path): gs://bucket/ or gs://bucket
# Uses a lookahead so the slash/end-of-string is not consumed, and uses
# a character class that excludes '<' to avoid re-matching placeholders.
(
re.compile(r"gs://([^/<\s]+)(?=[/\s]|$)"),
r"gs://<BUCKET>",
),
# 2. Firestore document paths (collection/doc/…/doc — at least 2 segments)
# Negative lookbehind for '.' prevents matching inside domain names (URLs).
# Negative lookbehind for '/' prevents matching inside already-replaced paths.
(
re.compile(
r"(?<![./])\b([a-zA-Z_][a-zA-Z0-9_-]*/[a-zA-Z0-9_-]+(?:/[a-zA-Z0-9_-]+){1,})\b"
),
r"<DOC_PATH>",
),
# 3. UUIDs (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
(
re.compile(
r"\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\b"
),
"<ID>",
),
# 4. ISO 8601 timestamps (2024-01-15T10:30:45Z / …+05:30 / ….123Z)
(
re.compile(
r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:\d{2})"
),
"<TS>",
),
# 5. Epoch timestamps: 10+ digit number optionally followed by fractional seconds
(
re.compile(r"\b\d{10,}(?:\.\d+)?\b"),
"<EPOCH>",
),
# 6. Email addresses
(
re.compile(r"\b[A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}\b"),
"<EMAIL>",
),
# 7. IP addresses (IPv4) — must come before large-number replacement
(
re.compile(r"\b(?:\d{1,3}\.){3}\d{1,3}\b"),
"<IP>",
),
# 8. URLs — strip to domain only (https://host/path → https://host)
# Handles http and https; preserves the scheme+domain for context.
(
re.compile(r"https?://([A-Za-z0-9.\-]+(:\d+)?)/[^\s]*"),
r"https://\1",
),
# 9. Job IDs in paths (/jobs/<alphanumeric>)
(
re.compile(r"/jobs/[A-Za-z0-9]+"),
"/jobs/<ID>",
),
# 10. Firebase UIDs (20–28 mixed-case alphanumeric characters)
# Firebase Auth UIDs are typically 28 chars; allow 20-28 for flexibility.
# Must contain both letters and digits (to avoid replacing plain words).
(
re.compile(r"\b(?=[A-Za-z0-9]{20,28}\b)(?=[A-Za-z0-9]*[A-Z])(?=[A-Za-z0-9]*[a-z])(?=[A-Za-z0-9]*\d)[A-Za-z0-9]{20,28}\b"),
"<UID>",
),
# 11. Hex/alphanumeric IDs: 8+ chars containing both letters and digits
# (e.g. git SHAs, Cloud Run revision IDs, job IDs)
# Uses a lookahead to require at least one digit AND one letter.
(
re.compile(r"\b(?=[a-fA-F0-9]*[a-fA-F][a-fA-F0-9]*\d|[a-fA-F0-9]*\d[a-fA-F0-9]*[a-fA-F])[a-fA-F0-9]{8,}\b"),
"<ID>",
),
# 12. Large numbers (4+ digits) — must come after epoch/IP/UUID replacements
(
re.compile(r"\b\d{4,}\b"),
"<NUM>",
),
]
def normalize_message(message: str) -> str:
"""Normalize a raw log message by replacing variable parts with placeholders.
Applies each normalizer rule in order (more specific patterns first) and
truncates the result to MAX_NORMALIZED_MESSAGE_LENGTH characters.
Args:
message: Raw log message string.
Returns:
Normalized message string, at most MAX_NORMALIZED_MESSAGE_LENGTH chars.
"""
if not message:
return ""
result = message
for pattern, replacement in _NORMALIZERS:
result = pattern.sub(replacement, result)
return result[:MAX_NORMALIZED_MESSAGE_LENGTH]
def compute_pattern_hash(service: str, normalized_message: str) -> str:
"""Compute a stable SHA-256 fingerprint for an error pattern.
Args:
service: The originating service name (e.g. "karaoke-backend").
normalized_message: The output of normalize_message().
Returns:
64-character lowercase hex SHA-256 digest.
"""
payload = f"{service}:{normalized_message}"
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
+3 -540
View File
@@ -10,8 +10,6 @@ from google.cloud.firestore_v1 import FieldFilter
from backend.config import settings
from backend.models.job import Job, JobStatus, TimelineEvent
from backend.models.worker_log import WorkerLogEntry
from backend.models.review_session import ReviewSession
from backend.models.audio_edit_session import AudioEditSession
logger = logging.getLogger(__name__)
@@ -77,20 +75,18 @@ class FirestoreService:
status: JobStatus,
progress: Optional[int] = None,
message: Optional[str] = None,
timeline_metadata: Optional[Dict[str, Any]] = None,
**additional_fields
) -> None:
"""Update job status and add timeline event."""
try:
doc_ref = self.db.collection(self.collection).document(job_id)
# Create timeline event
timeline_event = TimelineEvent(
status=status.value,
timestamp=datetime.utcnow().isoformat(),
progress=progress,
message=message,
metadata=timeline_metadata,
message=message
)
# Prepare updates
@@ -157,8 +153,7 @@ class FirestoreService:
query = query.where(filter=FieldFilter('user_email', '==', user_email.lower()))
# Filter by tenant_id (white-label portal scoping)
# Always filter: "" for consumer jobs, "vocalstar"/etc for tenant jobs
if tenant_id is not None:
if tenant_id:
query = query.where(filter=FieldFilter('tenant_id', '==', tenant_id))
# Date range filters
@@ -178,86 +173,6 @@ class FirestoreService:
logger.error(f"Error listing jobs: {e}")
raise
# Fields needed by the dashboard summary view
SUMMARY_FIELD_PATHS = [
'job_id', 'status', 'progress', 'created_at', 'artist', 'title',
'error_message', 'non_interactive', 'outputs_deleted_at', 'user_email', 'is_private',
'url', 'filename', 'audio_search_artist', 'audio_search_title',
'audio_source_type', 'source_name', 'source_id', 'target_file', 'download_url',
'state_data.brand_code', 'state_data.youtube_url', 'state_data.dropbox_link',
'state_data.audio_progress', 'state_data.lyrics_progress',
'state_data.audio_complete', 'state_data.lyrics_complete',
'state_data.backing_vocals_analysis', 'state_data.visibility_change_in_progress',
'file_urls.finals', 'file_urls.videos', 'file_urls.packages',
'processing_metadata',
]
def list_jobs_summary(
self,
status: Optional[JobStatus] = None,
exclude_statuses: Optional[List[str]] = None,
environment: Optional[str] = None,
client_id: Optional[str] = None,
created_after: Optional[datetime] = None,
created_before: Optional[datetime] = None,
user_email: Optional[str] = None,
tenant_id: Optional[str] = None,
limit: int = 100,
) -> List[Dict[str, Any]]:
"""
List jobs with Firestore field projection for reduced payload.
Uses ``query.select()`` so Firestore only reads the requested fields,
drastically cutting bandwidth and deserialization cost.
Returns raw dicts (not Job models) because projected documents would
fail Pydantic validation for missing required fields.
"""
try:
query = self.db.collection(self.collection)
if status:
query = query.where(filter=FieldFilter('status', '==', status.value))
# NOTE: exclude_statuses is filtered in Python after the query
# because Firestore's not-in filter combined with order_by(created_at)
# requires a composite index. Since documents are tiny after select(),
# Python filtering is fast and avoids the index requirement.
if environment:
query = query.where(filter=FieldFilter('request_metadata.environment', '==', environment))
if client_id:
query = query.where(filter=FieldFilter('request_metadata.client_id', '==', client_id))
if user_email:
query = query.where(filter=FieldFilter('user_email', '==', user_email.lower()))
# Always filter: "" for consumer jobs, "vocalstar"/etc for tenant jobs
if tenant_id is not None:
query = query.where(filter=FieldFilter('tenant_id', '==', tenant_id))
if created_after:
query = query.where(filter=FieldFilter('created_at', '>=', created_after))
if created_before:
query = query.where(filter=FieldFilter('created_at', '<=', created_before))
query = query.order_by('created_at', direction=firestore.Query.DESCENDING).limit(limit)
query = query.select(self.SUMMARY_FIELD_PATHS)
docs = query.stream()
results = [doc.to_dict() for doc in docs]
# Apply exclude_statuses filter in Python (see note above)
if exclude_statuses:
results = [r for r in results if r.get('status') not in exclude_statuses]
return results
except Exception as e:
logger.error(f"Error listing jobs summary: {e}")
raise
def delete_jobs_by_filter(
self,
environment: Optional[str] = None,
@@ -601,426 +516,6 @@ class FirestoreService:
logger.error(f"Error listing tokens: {e}")
return []
# ============================================
# Search Session Methods
# ============================================
def create_search_session(self, session_data: dict) -> str:
"""Store a search session for the guided job creation flow.
Sessions are short-lived (TTL enforced via Firestore TTL policy) and
represent a completed audio search that hasn't yet been converted into
a job. The document is consumed (deleted) when create-from-search runs.
Args:
session_data: Dict with session_id, user_email, tenant_id, artist,
title, results, remote_search_id, ttl_expiry.
Returns:
session_id from session_data.
"""
try:
session_id = session_data['session_id']
doc_ref = self.db.collection('search_sessions').document(session_id)
doc_ref.set(session_data)
logger.info(f"Created search session {session_id}")
return session_id
except Exception as e:
logger.error(f"Error creating search session: {e}")
raise
def get_search_session(self, session_id: str) -> Optional[dict]:
"""Retrieve a search session by ID.
Returns None only if the document does not exist.
Raises on Firestore errors so callers can distinguish a missing session
from a transient datastore failure.
TTL-expired documents may still appear briefly; callers should check
ttl_expiry and treat expired sessions as missing.
Args:
session_id: UUID string identifying the session.
Returns:
Session dict or None.
"""
doc_ref = self.db.collection('search_sessions').document(session_id)
doc = doc_ref.get()
if not doc.exists:
return None
return doc.to_dict()
def consume_search_session(self, session_id: str) -> Optional[dict]:
"""Atomically read and delete a search session.
Uses a Firestore transaction to prevent two concurrent job-creation
requests from both reading the same session and creating duplicate jobs.
Returns the session dict if it existed, None if it did not exist.
Raises on Firestore errors.
Args:
session_id: UUID string identifying the session.
Returns:
Session dict or None.
"""
doc_ref = self.db.collection('search_sessions').document(session_id)
@firestore.transactional
def _read_and_delete(transaction, ref):
doc = ref.get(transaction=transaction)
if not doc.exists:
return None
session_data = doc.to_dict()
transaction.delete(ref)
return session_data
transaction = self.db.transaction()
return _read_and_delete(transaction, doc_ref)
# ============================================
# Review Session Subcollection Methods
# ============================================
# These methods store review session snapshots in a subcollection
# (jobs/{job_id}/review_sessions) for lyrics review backup/restore.
def save_review_session(self, job_id: str, session: ReviewSession) -> str:
"""
Save a review session to the subcollection.
Stores at: jobs/{job_id}/review_sessions/{session_id}
Args:
job_id: Job ID
session: ReviewSession instance
Returns:
session_id
"""
try:
session.job_id = job_id
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
doc_ref = sessions_ref.document(session.session_id)
doc_ref.set(session.to_dict())
logger.info(f"Saved review session {session.session_id} for job {job_id}")
return session.session_id
except Exception as e:
logger.error(f"Error saving review session for job {job_id}: {e}")
raise
def list_review_sessions(self, job_id: str) -> List[ReviewSession]:
"""
List all review sessions for a job, ordered by updated_at descending.
Args:
job_id: Job ID
Returns:
List of ReviewSession instances (metadata only, no correction_data)
"""
try:
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING)
docs = query.stream()
return [ReviewSession.from_dict(doc.to_dict()) for doc in docs]
except Exception as e:
logger.error(f"Error listing review sessions for job {job_id}: {e}")
return []
def get_review_session(self, job_id: str, session_id: str) -> Optional[ReviewSession]:
"""
Get a single review session by ID.
Args:
job_id: Job ID
session_id: Session ID
Returns:
ReviewSession or None
"""
try:
doc_ref = (
self.db.collection(self.collection)
.document(job_id)
.collection("review_sessions")
.document(session_id)
)
doc = doc_ref.get()
if not doc.exists:
return None
return ReviewSession.from_dict(doc.to_dict())
except Exception as e:
logger.error(f"Error getting review session {session_id} for job {job_id}: {e}")
raise
def delete_review_session(self, job_id: str, session_id: str) -> None:
"""
Delete a review session.
Args:
job_id: Job ID
session_id: Session ID
"""
try:
doc_ref = (
self.db.collection(self.collection)
.document(job_id)
.collection("review_sessions")
.document(session_id)
)
doc_ref.delete()
logger.info(f"Deleted review session {session_id} for job {job_id}")
except Exception as e:
logger.error(f"Error deleting review session {session_id} for job {job_id}: {e}")
raise
def delete_review_sessions_subcollection(self, job_id: str, batch_size: int = 500) -> int:
"""
Delete all review sessions for a job (used when deleting a job).
Args:
job_id: Job ID
batch_size: Number of documents to delete per batch
Returns:
Number of sessions deleted
"""
try:
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
deleted_count = 0
while True:
docs = sessions_ref.limit(batch_size).stream()
deleted_in_batch = 0
batch = self.db.batch()
for doc in docs:
batch.delete(doc.reference)
deleted_in_batch += 1
if deleted_in_batch == 0:
break
batch.commit()
deleted_count += deleted_in_batch
if deleted_in_batch < batch_size:
break
if deleted_count > 0:
logger.info(f"Deleted {deleted_count} review sessions for job {job_id}")
return deleted_count
except Exception as e:
logger.error(f"Error deleting review sessions subcollection for job {job_id}: {e}")
return 0
def get_latest_review_session_hash(self, job_id: str) -> Optional[str]:
"""
Get the data_hash of the most recent review session for deduplication.
Args:
job_id: Job ID
Returns:
data_hash string or None if no sessions exist
"""
try:
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
query = (
sessions_ref
.order_by("updated_at", direction=firestore.Query.DESCENDING)
.limit(1)
.select(["data_hash"])
)
docs = list(query.stream())
if docs:
return docs[0].to_dict().get("data_hash", "")
return None
except Exception as e:
logger.error(f"Error getting latest review session hash for job {job_id}: {e}")
return None
def search_review_sessions(
self,
query_text: str = "",
job_id: Optional[str] = None,
limit: int = 20,
) -> List[Dict[str, Any]]:
"""
Search review sessions across all jobs using collection group query.
Args:
query_text: Search text (prefix-matched against artist_lower or title_lower)
job_id: If provided, search only within this job
limit: Maximum results
Returns:
List of session dicts with job context
"""
try:
if job_id:
# Search within a specific job
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING).limit(limit)
docs = query.stream()
return [doc.to_dict() for doc in docs]
# Collection group query across all jobs
sessions_ref = self.db.collection_group("review_sessions")
if query_text:
# Over-fetch since we filter in Python (Firestore doesn't support LIKE/contains)
fetch_limit = max(limit * 25, 500)
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING).limit(fetch_limit)
docs = query.stream()
results = [doc.to_dict() for doc in docs]
q_lower = query_text.lower()
results = [
r for r in results
if q_lower in (r.get("artist") or "").lower()
or q_lower in (r.get("title") or "").lower()
or q_lower in (r.get("job_id") or "").lower()
]
return results[:limit]
else:
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING).limit(limit)
docs = query.stream()
return [doc.to_dict() for doc in docs]
except Exception as e:
logger.error(f"Error searching review sessions: {e}")
return []
# ---- Audio Edit Sessions ----
def save_audio_edit_session(self, job_id: str, session: AudioEditSession) -> str:
"""Save an audio edit session to the subcollection."""
try:
session.job_id = job_id
sessions_ref = self.db.collection(self.collection).document(job_id).collection("audio_edit_sessions")
doc_ref = sessions_ref.document(session.session_id)
doc_ref.set(session.to_dict())
logger.info(f"Saved audio edit session {session.session_id} for job {job_id}")
return session.session_id
except Exception as e:
logger.error(f"Error saving audio edit session for job {job_id}: {e}")
raise
def list_audio_edit_sessions(self, job_id: str) -> List[AudioEditSession]:
"""List all audio edit sessions for a job, ordered by updated_at descending."""
try:
sessions_ref = self.db.collection(self.collection).document(job_id).collection("audio_edit_sessions")
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING)
docs = query.stream()
return [AudioEditSession.from_dict(doc.to_dict()) for doc in docs]
except Exception as e:
logger.error(f"Error listing audio edit sessions for job {job_id}: {e}")
return []
def get_audio_edit_session(self, job_id: str, session_id: str) -> Optional[AudioEditSession]:
"""Get a single audio edit session by ID."""
try:
doc_ref = (
self.db.collection(self.collection)
.document(job_id)
.collection("audio_edit_sessions")
.document(session_id)
)
doc = doc_ref.get()
if not doc.exists:
return None
return AudioEditSession.from_dict(doc.to_dict())
except Exception as e:
logger.error(f"Error getting audio edit session {session_id} for job {job_id}: {e}")
raise
def delete_audio_edit_session(self, job_id: str, session_id: str) -> None:
"""Delete a single audio edit session."""
try:
doc_ref = (
self.db.collection(self.collection)
.document(job_id)
.collection("audio_edit_sessions")
.document(session_id)
)
doc_ref.delete()
logger.info(f"Deleted audio edit session {session_id} for job {job_id}")
except Exception as e:
logger.error(f"Error deleting audio edit session {session_id} for job {job_id}: {e}")
raise
def delete_audio_edit_sessions_subcollection(self, job_id: str, batch_size: int = 500) -> int:
"""Delete all audio edit sessions for a job (used when deleting a job)."""
try:
sessions_ref = self.db.collection(self.collection).document(job_id).collection("audio_edit_sessions")
deleted_count = 0
while True:
docs = sessions_ref.limit(batch_size).stream()
deleted_in_batch = 0
batch = self.db.batch()
for doc in docs:
batch.delete(doc.reference)
deleted_in_batch += 1
if deleted_in_batch == 0:
break
batch.commit()
deleted_count += deleted_in_batch
if deleted_in_batch < batch_size:
break
if deleted_count > 0:
logger.info(f"Deleted {deleted_count} audio edit sessions for job {job_id}")
return deleted_count
except Exception as e:
logger.error(f"Error deleting audio edit sessions subcollection for job {job_id}: {e}")
return 0
def get_latest_audio_edit_session_hash(self, job_id: str) -> Optional[str]:
"""Get the data_hash of the most recent audio edit session for deduplication."""
try:
sessions_ref = self.db.collection(self.collection).document(job_id).collection("audio_edit_sessions")
query = (
sessions_ref
.order_by("updated_at", direction=firestore.Query.DESCENDING)
.limit(1)
.select(["data_hash"])
)
docs = list(query.stream())
if docs:
return docs[0].to_dict().get("data_hash", "")
return None
except Exception as e:
logger.error(f"Error getting latest audio edit session hash for job {job_id}: {e}")
return None
def delete_search_session(self, session_id: str) -> None:
"""Delete a search session after it has been consumed by job creation.
Errors are logged but not re-raised — a missing or already-deleted
session is a no-op.
Args:
session_id: UUID string identifying the session.
"""
try:
doc_ref = self.db.collection('search_sessions').document(session_id)
doc_ref.delete()
logger.debug(f"Deleted search session {session_id}")
except Exception as e:
logger.error(f"Error deleting search session {session_id}: {e}")
# Singleton client instance
_firestore_client: Optional[firestore.Client] = None
@@ -1042,35 +537,3 @@ def get_firestore_client() -> firestore.Client:
_firestore_client = firestore.Client(project=settings.google_cloud_project)
return _firestore_client
_log_service: Optional["FirestoreService"] = None
def _get_log_service() -> "FirestoreService":
"""Return a cached FirestoreService instance for log_to_job."""
global _log_service
if _log_service is None:
_log_service = FirestoreService()
return _log_service
def log_to_job(job_id: str, worker: str, level: str, message: str, metadata: Optional[Dict[str, Any]] = None) -> None:
"""
Write a log entry to a job's Firestore log subcollection.
This is a convenience function for API endpoints that need to log
to a job's log history (workers already have JobLogAdapter for this).
Args:
job_id: Job ID to log to
worker: Worker/source name (e.g., "api", "edit", "admin")
level: Log level (INFO, WARNING, ERROR)
message: Log message
metadata: Optional structured metadata dict
"""
try:
entry = WorkerLogEntry.create(job_id, worker, level, message, metadata)
_get_log_service().append_log_to_subcollection(job_id, entry)
except Exception as e:
logger.debug(f"Failed to write job log for {job_id}: {e}")
+84 -283
View File
@@ -34,8 +34,6 @@ from typing import Any, Dict, List, Optional
import httpx
from backend.config import get_settings
logger = logging.getLogger(__name__)
@@ -44,104 +42,6 @@ class FlacfetchServiceError(Exception):
pass
def _format_httpx_error(e: Exception) -> str:
"""
Format an httpx exception into a descriptive error string.
httpx.RequestError subclasses (ConnectError, ReadError, etc.) often have
empty str() representations, which produces useless messages like
"Status request failed: ". This helper always returns something meaningful.
"""
error_str = str(e)
error_type = type(e).__name__
if error_str:
return f"{error_type}: {error_str}"
# str(e) is empty - try to include request URL for context
if isinstance(e, httpx.RequestError):
try:
request = e.request
if request:
return f"{error_type} for {request.method} {request.url}"
except RuntimeError:
pass # .request not set
return error_type
def should_retry_flacfetch_error(exception: Exception) -> bool:
"""
Determine if a flacfetch exception should be retried.
Retry on:
- httpx.RequestError (network failures: timeout, connection refused, etc.)
- httpx.HTTPStatusError with 5xx status (server errors)
Don't retry on:
- httpx.HTTPStatusError with 4xx status (client errors: bad request, auth, not found)
"""
if isinstance(exception, httpx.RequestError):
# Network-level failures - always retry
return True
if isinstance(exception, httpx.HTTPStatusError):
# HTTP errors - only retry on 5xx (server errors)
return exception.response.status_code >= 500
# Don't retry on other exceptions (e.g., FlacfetchServiceError after wrapping)
return False
async def with_retry(func, *args, **kwargs):
"""
Execute an async function with retry logic using exponential backoff.
This implementation uses manual retry logic instead of tenacity decorators
for better testability and runtime configuration.
"""
settings = get_settings()
max_attempts = settings.flacfetch_retry_max_attempts
min_wait = settings.flacfetch_retry_min_wait
max_wait = settings.flacfetch_retry_max_wait
last_exception = None
for attempt in range(1, max_attempts + 1):
try:
return await func(*args, **kwargs)
except Exception as e:
last_exception = e
# Check if we should retry this exception
if not should_retry_flacfetch_error(e):
# Don't retry - raise immediately
raise
# Check if we have more attempts left
if attempt >= max_attempts:
# No more attempts - raise the last exception
logger.warning(
f"Flacfetch request failed after {max_attempts} attempts: "
f"{_format_httpx_error(e) if isinstance(e, httpx.RequestError) else e}"
)
raise
# Calculate wait time with exponential backoff
wait_time = min(min_wait * (2 ** (attempt - 1)), max_wait)
logger.warning(
f"Flacfetch request failed (attempt {attempt}/{max_attempts}): "
f"{_format_httpx_error(e) if isinstance(e, httpx.RequestError) else e}. "
f"Retrying in {wait_time:.1f}s..."
)
# Wait before next attempt
await asyncio.sleep(wait_time)
# Should never reach here, but just in case
raise last_exception
class FlacfetchClient:
"""
Client for remote flacfetch HTTP API.
@@ -200,7 +100,7 @@ class FlacfetchClient:
return data
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Cannot reach flacfetch service: {_format_httpx_error(e)}")
raise FlacfetchServiceError(f"Cannot reach flacfetch service: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Health check failed: {e.response.status_code}")
@@ -245,10 +145,10 @@ class FlacfetchClient:
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Search request failed: {_format_httpx_error(e)}")
raise FlacfetchServiceError(f"Search request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Search failed: {e.response.status_code} - {e.response.text}")
async def download(
self,
search_id: str,
@@ -258,59 +158,47 @@ class FlacfetchClient:
) -> str:
"""
Start downloading an audio file.
Retries automatically on network errors and server errors (5xx) using
exponential backoff. Configured via FLACFETCH_RETRY_* environment variables.
Args:
search_id: Search ID from previous search
result_index: Index of result to download
output_filename: Optional custom filename (without extension)
gcs_path: GCS path for upload (e.g., "uploads/job123/audio/")
Returns:
Download ID for tracking progress
Raises:
FlacfetchServiceError: On download start failure (after all retries exhausted)
FlacfetchServiceError: On download start failure
"""
try:
return await with_retry(self._download_impl, search_id, result_index, output_filename, gcs_path)
async with httpx.AsyncClient() as client:
payload = {
"search_id": search_id,
"result_index": result_index,
}
if output_filename:
payload["output_filename"] = output_filename
if gcs_path:
payload["upload_to_gcs"] = True
payload["gcs_path"] = gcs_path
resp = await client.post(
f"{self.base_url}/download",
headers=self._headers(),
json=payload,
timeout=self.timeout,
)
resp.raise_for_status()
data = resp.json()
return data["download_id"]
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Download request failed: {_format_httpx_error(e)}")
raise FlacfetchServiceError(f"Download request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Download start failed: {e.response.status_code} - {e.response.text}")
async def _download_impl(
self,
search_id: str,
result_index: int,
output_filename: Optional[str] = None,
gcs_path: Optional[str] = None,
) -> str:
"""Implementation of download without error wrapping - lets httpx exceptions bubble up."""
async with httpx.AsyncClient() as client:
payload = {
"search_id": search_id,
"result_index": result_index,
}
if output_filename:
payload["output_filename"] = output_filename
if gcs_path:
payload["upload_to_gcs"] = True
payload["gcs_path"] = gcs_path
resp = await client.post(
f"{self.base_url}/download",
headers=self._headers(),
json=payload,
timeout=self.timeout,
)
resp.raise_for_status()
data = resp.json()
return data["download_id"]
async def download_by_id(
self,
source_name: str,
@@ -326,9 +214,6 @@ class FlacfetchClient:
This is useful when you have stored the source_id from a previous search
and want to download later without re-searching.
Retries automatically on network errors and server errors (5xx) using
exponential backoff. Configured via FLACFETCH_RETRY_* environment variables.
Args:
source_name: Provider name (RED, OPS, YouTube, Spotify)
source_id: Source-specific ID (torrent ID, video ID, track ID)
@@ -341,68 +226,52 @@ class FlacfetchClient:
Download ID for tracking progress
Raises:
FlacfetchServiceError: On download start failure (after all retries exhausted)
FlacfetchServiceError: On download start failure
"""
try:
return await with_retry(
self._download_by_id_impl,
source_name, source_id, output_filename, target_file, download_url, gcs_path
)
async with httpx.AsyncClient() as client:
payload = {
"source_name": source_name,
"source_id": source_id,
}
if output_filename:
payload["output_filename"] = output_filename
if target_file:
payload["target_file"] = target_file
if download_url:
payload["download_url"] = download_url
if gcs_path:
payload["upload_to_gcs"] = True
payload["gcs_path"] = gcs_path
resp = await client.post(
f"{self.base_url}/download-by-id",
headers=self._headers(),
json=payload,
timeout=self.timeout,
)
resp.raise_for_status()
data = resp.json()
return data["download_id"]
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Download by ID request failed: {_format_httpx_error(e)}")
raise FlacfetchServiceError(f"Download by ID request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Download by ID failed: {e.response.status_code} - {e.response.text}")
async def _download_by_id_impl(
self,
source_name: str,
source_id: str,
output_filename: Optional[str] = None,
target_file: Optional[str] = None,
download_url: Optional[str] = None,
gcs_path: Optional[str] = None,
) -> str:
"""Implementation of download_by_id without error wrapping - lets httpx exceptions bubble up."""
async with httpx.AsyncClient() as client:
payload = {
"source_name": source_name,
"source_id": source_id,
}
if output_filename:
payload["output_filename"] = output_filename
if target_file:
payload["target_file"] = target_file
if download_url:
payload["download_url"] = download_url
if gcs_path:
payload["upload_to_gcs"] = True
payload["gcs_path"] = gcs_path
resp = await client.post(
f"{self.base_url}/download-by-id",
headers=self._headers(),
json=payload,
timeout=self.timeout,
)
resp.raise_for_status()
data = resp.json()
return data["download_id"]
async def get_download_status(self, download_id: str) -> Dict[str, Any]:
"""
Get the current status of a download.
Args:
download_id: Download ID from start_download
Returns:
Status dict with progress, speed, path info
Raises:
httpx.RequestError: On network-level failures (connection refused, timeout, etc.)
These are NOT wrapped so wait_for_download can handle retries.
FlacfetchServiceError: On application-level failures (404, other HTTP errors)
FlacfetchServiceError: On status check failure
"""
try:
async with httpx.AsyncClient() as client:
@@ -413,10 +282,9 @@ class FlacfetchClient:
)
resp.raise_for_status()
return resp.json()
except httpx.RequestError:
# Let network errors bubble up so wait_for_download can handle retries
raise
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Status request failed: {e}")
except httpx.HTTPStatusError as e:
if e.response.status_code == 404:
raise FlacfetchServiceError(f"Download not found: {download_id}")
@@ -428,59 +296,35 @@ class FlacfetchClient:
timeout: int = 600,
poll_interval: float = 2.0,
progress_callback: Optional[callable] = None,
max_consecutive_errors: int = 3,
) -> Dict[str, Any]:
"""
Wait for a download to complete.
Tolerates transient network errors during status polling. A single
connection hiccup won't fail the entire download -- only N consecutive
poll failures will.
Args:
download_id: Download ID to wait for
timeout: Maximum wait time in seconds
poll_interval: Time between status checks
progress_callback: Optional callback(status_dict) for progress updates
max_consecutive_errors: Number of consecutive poll failures before giving up
Returns:
Final status dict with output_path or gcs_path
Raises:
FlacfetchServiceError: On download failure, timeout, or too many consecutive poll errors
FlacfetchServiceError: On download failure or timeout
"""
elapsed = 0.0
consecutive_errors = 0
while elapsed < timeout:
try:
status = await self.get_download_status(download_id)
consecutive_errors = 0 # Reset on success
except httpx.RequestError as e:
consecutive_errors += 1
error_desc = _format_httpx_error(e)
logger.warning(
f"Status poll failed for download {download_id} "
f"({consecutive_errors}/{max_consecutive_errors}): {error_desc}"
)
if consecutive_errors >= max_consecutive_errors:
raise FlacfetchServiceError(
f"Status polling failed after {consecutive_errors} consecutive "
f"network errors (last: {error_desc})"
)
await asyncio.sleep(poll_interval)
elapsed += poll_interval
continue
status = await self.get_download_status(download_id)
if progress_callback:
try:
progress_callback(status)
except Exception as e:
logger.warning(f"Progress callback error: {e}")
download_status = status.get("status")
if download_status in ["complete", "seeding"]:
logger.info(f"Download {download_id} complete: {status.get('gcs_path') or status.get('output_path')}")
return status
@@ -489,7 +333,7 @@ class FlacfetchClient:
raise FlacfetchServiceError(f"Download failed: {error}")
elif download_status == "cancelled":
raise FlacfetchServiceError("Download was cancelled")
# Log progress
progress = status.get("progress", 0)
speed = status.get("download_speed_kbps", 0)
@@ -498,10 +342,10 @@ class FlacfetchClient:
f"Download {download_id}: {progress:.1f}% "
f"({speed:.1f} KB/s, {peers} peers)"
)
await asyncio.sleep(poll_interval)
elapsed += poll_interval
raise FlacfetchServiceError(f"Download timed out after {timeout}s")
async def list_torrents(self) -> Dict[str, Any]:
@@ -522,7 +366,7 @@ class FlacfetchClient:
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"List torrents failed: {_format_httpx_error(e)}")
raise FlacfetchServiceError(f"List torrents failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"List torrents failed: {e.response.status_code}")
@@ -553,7 +397,7 @@ class FlacfetchClient:
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Delete torrent failed: {_format_httpx_error(e)}")
raise FlacfetchServiceError(f"Delete torrent failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Delete torrent failed: {e.response.status_code}")
@@ -587,7 +431,7 @@ class FlacfetchClient:
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Cleanup failed: {_format_httpx_error(e)}")
raise FlacfetchServiceError(f"Cleanup failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Cleanup failed: {e.response.status_code}")
@@ -628,7 +472,7 @@ class FlacfetchClient:
return deleted
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Clear search cache request failed: {_format_httpx_error(e)}")
raise FlacfetchServiceError(f"Clear search cache request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"Clear search cache failed: {e.response.status_code} - {e.response.text}"
@@ -658,55 +502,12 @@ class FlacfetchClient:
return deleted_count
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Clear all cache request failed: {_format_httpx_error(e)}")
raise FlacfetchServiceError(f"Clear all cache request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"Clear all cache failed: {e.response.status_code} - {e.response.text}"
)
# =========================================================================
# YouTube Availability Check
# =========================================================================
async def check_youtube(self, url: str) -> Dict[str, Any]:
"""
Check if a YouTube video is available for download from this server's location.
Calls the flacfetch /check-youtube endpoint which uses yt-dlp to detect
geo-restrictions, private/removed videos, and other unavailability reasons.
Args:
url: YouTube video URL or video ID
Returns:
Dict with keys: available, video_id, title, error,
is_geo_restricted, is_age_restricted, is_private, is_removed
Raises:
FlacfetchServiceError: On request failure
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.post(
f"{self.base_url}/check-youtube",
headers=self._headers(),
json={"url": url},
timeout=15,
)
resp.raise_for_status()
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"YouTube availability check failed: {_format_httpx_error(e)}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"YouTube availability check failed: {e.response.status_code} - {e.response.text}"
)
# =========================================================================
# Cache Management
# =========================================================================
async def get_cache_stats(self) -> Dict[str, Any]:
"""
Get statistics about the cache.
@@ -728,7 +529,7 @@ class FlacfetchClient:
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Get cache stats request failed: {_format_httpx_error(e)}")
raise FlacfetchServiceError(f"Get cache stats request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"Get cache stats failed: {e.response.status_code} - {e.response.text}"
-141
View File
@@ -1,141 +0,0 @@
"""Service for generating personalized referral flyer PDFs."""
import logging
import os
import shutil
import subprocess
import tempfile
logger = logging.getLogger(__name__)
TEMPLATE_DIR = os.path.join(os.path.dirname(__file__), "..", "templates", "printables")
CHROMIUM_PATHS = [
"/usr/bin/chromium",
"/usr/bin/chromium-browser",
"/usr/bin/google-chrome",
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
]
VALID_THEMES = {"light", "dark"}
MAX_QR_DATA_URL_BYTES = 500_000
class FlyerError(Exception):
"""Error during flyer generation."""
class FlyerService:
"""Generates personalized referral flyer PDFs."""
def __init__(self):
self._chromium_path: str | None = None
def _get_chromium(self) -> str:
"""Lazily find Chromium binary. Only needed when generating PDFs."""
if self._chromium_path:
return self._chromium_path
for path in CHROMIUM_PATHS:
if os.path.exists(path):
self._chromium_path = path
return path
for name in ["chromium", "chromium-browser", "google-chrome"]:
found = shutil.which(name)
if found:
self._chromium_path = found
return found
raise FlyerError("Chromium not found. Install chromium to generate flyer PDFs.")
def _validate_qr_data_url(self, qr_data_url: str) -> None:
if not qr_data_url.startswith("data:image/"):
raise FlyerError("Invalid QR data URL: must be a data:image/ URL")
if len(qr_data_url) > MAX_QR_DATA_URL_BYTES:
raise FlyerError(f"QR data URL too large (>{MAX_QR_DATA_URL_BYTES} bytes)")
def _render_template(
self,
theme: str,
referral_code: str,
discount_percent: int,
qr_data_url: str,
) -> str:
if theme not in VALID_THEMES:
raise FlyerError(f"Invalid theme: {theme}. Must be one of: {VALID_THEMES}")
suffix = "-dark" if theme == "dark" else ""
template_path = os.path.join(TEMPLATE_DIR, f"website-referral-flyer{suffix}.html")
with open(template_path) as f:
html = f.read()
html = html.replace("{{REFERRAL_CODE}}", referral_code.upper())
html = html.replace("{{REFERRAL_CODE_LOWER}}", referral_code.lower())
html = html.replace("{{DISCOUNT_PERCENT}}", str(discount_percent))
html = html.replace("{{QR_DATA_URL}}", qr_data_url)
return html
def generate_pdf(
self,
theme: str,
referral_code: str,
discount_percent: int,
qr_data_url: str,
) -> bytes:
"""Generate a personalized flyer PDF. Returns raw PDF bytes."""
self._validate_qr_data_url(qr_data_url)
html = self._render_template(theme, referral_code, discount_percent, qr_data_url)
with tempfile.NamedTemporaryFile(
suffix=".html", delete=False, mode="w", encoding="utf-8"
) as tmp_html:
tmp_html.write(html)
tmp_html_path = tmp_html.name
tmp_pdf_path = tmp_html_path.replace(".html", ".pdf")
try:
cmd = [
self._get_chromium(),
"--headless",
f"--print-to-pdf={tmp_pdf_path}",
"--no-margins",
"--print-background",
"--no-pdf-header-footer",
"--virtual-time-budget=5000",
"--paper-width=8.5",
"--paper-height=11",
"--disable-gpu",
"--no-sandbox",
tmp_html_path,
]
result = subprocess.run(
cmd, capture_output=True, text=True, timeout=30
)
if result.returncode != 0:
logger.error("Chromium PDF generation failed: %s", result.stderr)
raise FlyerError(f"PDF generation failed: {result.stderr[:200]}")
if not os.path.exists(tmp_pdf_path):
raise FlyerError("PDF generation failed: no output file created")
with open(tmp_pdf_path, "rb") as f:
return f.read()
finally:
for path in [tmp_html_path, tmp_pdf_path]:
try:
os.unlink(path)
except OSError:
pass
_flyer_service = None
def get_flyer_service() -> FlyerService:
global _flyer_service
if _flyer_service is None:
_flyer_service = FlyerService()
return _flyer_service
+6 -440
View File
@@ -1,11 +1,8 @@
import asyncio
import glob as glob_module
import hashlib
import json
import logging
import os
import re
import shutil
import subprocess
import sys
import tempfile
@@ -16,7 +13,6 @@ from typing import Optional
from fastapi import FastAPI, BackgroundTasks, HTTPException, Header, Depends
from google.cloud import storage
from packaging.version import Version
from pydantic import BaseModel
logging.basicConfig(level=logging.INFO)
@@ -61,27 +57,12 @@ class EncodePreviewRequest(BaseModel):
font_gcs_path: Optional[str] = None # gs://bucket/path/to/custom-font.ttf
class RenderVideoRequest(BaseModel):
job_id: str
original_corrections_gcs_path: str
updated_corrections_gcs_path: Optional[str] = None
audio_gcs_path: str
style_params_gcs_path: Optional[str] = None
style_assets: Optional[dict] = None
output_gcs_prefix: str
artist: str
title: str
subtitle_offset_ms: int = 0
video_resolution: str = "4k"
class JobStatus(BaseModel):
job_id: str
status: str # pending, running, complete, failed
progress: int # 0-100
error: Optional[str] = None
output_files: Optional[list[str]] = None
metadata: Optional[dict] = None
def download_from_gcs(gcs_uri: str, local_path: Path):
@@ -153,59 +134,6 @@ def upload_single_file_to_gcs(local_path: Path, gcs_uri: str):
logger.info(f"Uploaded: {local_path} -> {gcs_uri}")
# Style asset disk cache directory
# Persists across jobs to avoid re-downloading the same theme assets (fonts, backgrounds)
STYLE_CACHE_DIR = Path("/var/cache/karaoke-gen/styles")
def download_with_cache(gcs_uri: str, local_path: Path, cache_dir: Optional[Path] = STYLE_CACHE_DIR):
"""Download a file from GCS, using a local disk cache to skip repeated downloads.
Cache key is SHA-256 of the GCS URI. Cache hits are validated by comparing
the cached file size against the GCS object size (via a lightweight metadata
call). On size mismatch the stale entry is replaced.
When cache_dir is None, downloads directly (no caching).
"""
if cache_dir is None:
download_single_file_from_gcs(gcs_uri, local_path)
return
cache_dir.mkdir(parents=True, exist_ok=True)
cache_key = hashlib.sha256(gcs_uri.encode()).hexdigest()
cached_path = cache_dir / cache_key
if cached_path.exists():
# Validate cached file against GCS object size
try:
parts = gcs_uri.replace("gs://", "").split("/", 1)
bucket_name = parts[0]
blob_name = parts[1] if len(parts) > 1 else ""
blob = storage_client.bucket(bucket_name).blob(blob_name)
blob.reload() # Fetch metadata (lightweight HEAD request)
gcs_size = blob.size
cached_size = cached_path.stat().st_size
if gcs_size is not None and cached_size != gcs_size:
logger.warning(
f"Cache stale for {gcs_uri}: cached={cached_size}B, GCS={gcs_size}B. Re-downloading."
)
cached_path.unlink()
else:
logger.info(f"Cache hit for {gcs_uri} -> {cached_path}")
shutil.copy2(str(cached_path), str(local_path))
return
except Exception as e:
# On transient GCS errors, serve from cache rather than failing the job
logger.warning(f"Cache validation failed for {gcs_uri}: {e}. Serving from cache.")
shutil.copy2(str(cached_path), str(local_path))
return
logger.info(f"Cache miss for {gcs_uri}, downloading...")
download_single_file_from_gcs(gcs_uri, local_path)
# Store in cache
shutil.copy2(str(local_path), str(cached_path))
def run_preview_encoding(job_id: str, work_dir: Path, request: "EncodePreviewRequest"):
# Run FFmpeg encoding for preview video (480x270, fast settings)
jobs[job_id]["status"] = "running"
@@ -309,192 +237,6 @@ def run_preview_encoding(job_id: str, work_dir: Path, request: "EncodePreviewReq
raise
def run_render_video(job_id: str, work_dir: Path, request: "RenderVideoRequest"):
"""Run video rendering using OutputGenerator from the karaoke-gen wheel.
Downloads corrections, audio, and style assets from GCS, runs OutputGenerator
to produce with_vocals.mkv and subtitle files, then uploads results.
Style assets use disk cache to avoid repeated downloads of the same theme.
This mirrors the logic in render_video_worker.py but runs on the GCE VM
instead of Cloud Run, providing more memory and CPU for ffmpeg/libass.
"""
jobs[job_id]["status"] = "running"
jobs[job_id]["progress"] = 5
try:
# Import from installed karaoke-gen wheel
from karaoke_gen.lyrics_transcriber.output.generator import OutputGenerator
from karaoke_gen.lyrics_transcriber.output.countdown_processor import CountdownProcessor
from karaoke_gen.lyrics_transcriber.types import CorrectionResult
from karaoke_gen.lyrics_transcriber.correction.operations import CorrectionOperations
from karaoke_gen.lyrics_transcriber.core.config import OutputConfig
from karaoke_gen.style_loader import load_styles_from_gcs
from karaoke_gen.utils import sanitize_filename
# 1. Download original corrections
corrections_path = work_dir / "corrections.json"
download_single_file_from_gcs(request.original_corrections_gcs_path, corrections_path)
jobs[job_id]["progress"] = 15
with open(corrections_path, 'r', encoding='utf-8') as f:
original_data = json.load(f)
base_result = CorrectionResult.from_dict(original_data)
# 2. Apply user corrections if available
if request.updated_corrections_gcs_path:
updated_path = work_dir / "corrections_updated.json"
download_single_file_from_gcs(request.updated_corrections_gcs_path, updated_path)
with open(updated_path, 'r', encoding='utf-8') as f:
updated_data = json.load(f)
if isinstance(updated_data, dict) and "corrections" in updated_data:
correction_result = CorrectionOperations.update_correction_result_with_data(
base_result, updated_data
)
logger.info(f"[job:{job_id}] Applied user corrections")
else:
logger.warning(f"[job:{job_id}] corrections_updated.json exists but has no 'corrections' key, using base result")
correction_result = base_result
else:
correction_result = base_result
jobs[job_id]["progress"] = 25
# 3. Download audio
audio_path = work_dir / "audio.flac"
download_single_file_from_gcs(request.audio_gcs_path, audio_path)
jobs[job_id]["progress"] = 35
# 4. Process countdown intro
countdown_processor = CountdownProcessor(cache_dir=str(work_dir), logger=logger)
correction_result, audio_path_str, padding_added, padding_seconds = countdown_processor.process(
correction_result=correction_result,
audio_filepath=str(audio_path),
)
audio_path = Path(audio_path_str)
jobs[job_id]["progress"] = 40
# 5. Download style assets (with disk cache for repeated themes)
def cached_download(gcs_path, local_path):
download_with_cache(gcs_path, Path(local_path), STYLE_CACHE_DIR)
styles_path, style_data = load_styles_from_gcs(
style_params_gcs_path=request.style_params_gcs_path,
style_assets=request.style_assets,
temp_dir=str(work_dir),
download_func=cached_download,
logger=logger,
)
# Register any downloaded fonts with fontconfig
for asset_key, gcs_path in (request.style_assets or {}).items():
if 'font' in asset_key.lower() and gcs_path.endswith(('.ttf', '.otf', '.woff', '.woff2')):
fonts_dir = Path("/usr/local/share/fonts/custom")
fonts_dir.mkdir(parents=True, exist_ok=True)
font_filename = gcs_path.split("/")[-1]
font_dest = fonts_dir / font_filename
if not font_dest.exists():
ext = os.path.splitext(gcs_path)[1]
style_font = work_dir / "style" / f"{asset_key}{ext}"
if style_font.exists():
shutil.copy2(str(style_font), str(font_dest))
subprocess.run(["fc-cache", "-fv"], capture_output=True)
logger.info(f"Registered font: {font_filename}")
jobs[job_id]["progress"] = 50
# 6. Configure and run OutputGenerator
output_dir = work_dir / "output"
cache_dir = work_dir / "cache"
output_dir.mkdir(exist_ok=True)
cache_dir.mkdir(exist_ok=True)
config = OutputConfig(
output_dir=str(output_dir),
cache_dir=str(cache_dir),
output_styles_json=styles_path,
render_video=True,
generate_cdg=False,
generate_plain_text=True,
generate_lrc=True,
video_resolution=request.video_resolution,
subtitle_offset_ms=request.subtitle_offset_ms,
)
output_generator = OutputGenerator(config, logger)
safe_artist = sanitize_filename(request.artist) if request.artist else "Unknown"
safe_title = sanitize_filename(request.title) if request.title else "Unknown"
output_prefix = f"{safe_artist} - {safe_title}"
logger.info(f"[job:{job_id}] Generating outputs with prefix '{output_prefix}'")
jobs[job_id]["progress"] = 55
outputs = output_generator.generate_outputs(
transcription_corrected=correction_result,
lyrics_results={},
output_prefix=output_prefix,
audio_filepath=str(audio_path),
artist=request.artist,
title=request.title,
)
jobs[job_id]["progress"] = 85
# 7. Upload outputs to GCS
output_prefix_gcs = request.output_gcs_prefix.rstrip("/")
output_files = []
if outputs.video and os.path.exists(outputs.video):
gcs_path = f"{output_prefix_gcs}/videos/with_vocals.mkv"
upload_single_file_to_gcs(Path(outputs.video), gcs_path)
output_files.append(gcs_path)
logger.info(f"[job:{job_id}] Uploaded with_vocals.mkv ({os.path.getsize(outputs.video)} bytes)")
else:
raise RuntimeError("Video generation failed - no output file produced")
if outputs.ass and os.path.exists(outputs.ass):
gcs_path = f"{output_prefix_gcs}/lyrics/karaoke.ass"
upload_single_file_to_gcs(Path(outputs.ass), gcs_path)
output_files.append(gcs_path)
if outputs.lrc and os.path.exists(outputs.lrc):
gcs_path = f"{output_prefix_gcs}/lyrics/karaoke.lrc"
upload_single_file_to_gcs(Path(outputs.lrc), gcs_path)
output_files.append(gcs_path)
if outputs.corrected_txt and os.path.exists(outputs.corrected_txt):
gcs_path = f"{output_prefix_gcs}/lyrics/corrected.txt"
upload_single_file_to_gcs(Path(outputs.corrected_txt), gcs_path)
output_files.append(gcs_path)
jobs[job_id]["progress"] = 95
jobs[job_id]["output_files"] = output_files
jobs[job_id]["metadata"] = {
"countdown_padding_added": padding_added,
"countdown_padding_seconds": padding_seconds if padding_added else 0,
}
logger.info(f"[job:{job_id}] Render video complete. Output files: {output_files}")
except ImportError as e:
error_msg = (
f"OutputGenerator not available: {e}. "
"The karaoke-gen wheel must be installed. "
"Check that ensure_latest_wheel() succeeded."
)
logger.error(error_msg)
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = error_msg
raise RuntimeError(error_msg) from e
except Exception as e:
logger.error(f"[job:{job_id}] Render video failed: {e}")
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = str(e)
raise
def ensure_latest_wheel():
'''Download and install latest karaoke-gen wheel from GCS.
@@ -512,23 +254,12 @@ def ensure_latest_wheel():
# Find the downloaded wheel (get the latest by version sorting)
wheels = glob_module.glob("/tmp/karaoke_gen-*.whl")
# Filter out karaoke_gen-current.whl (not a valid PEP 427 wheel name)
wheels = [w for w in wheels if '-current' not in w]
if not wheels:
logger.warning("No wheel found in GCS, using fallback encoding logic")
return False
# Sort to get latest version (by semantic version, not alphabetically)
def extract_version(wheel_path):
"""Extract version from wheel filename like karaoke_gen-0.116.1-py3-none-any.whl"""
match = re.search(r'karaoke_gen-([0-9.]+)-', wheel_path)
if match:
return Version(match.group(1))
return Version("0.0.0") # Fallback for unparseable filenames
wheels.sort(key=extract_version, reverse=True)
wheel_path = wheels[0]
# Sort to get latest version
wheel_path = sorted(wheels)[-1]
logger.info(f"Installing wheel: {wheel_path}")
# Install (or upgrade) the wheel
@@ -563,43 +294,6 @@ def find_file(work_dir: Path, *patterns):
return None
def generate_mov_from_png(png_path: Path, mov_path: Path, duration: int = 5) -> Path:
"""Generate a MOV video from a static PNG image using FFmpeg.
This runs on the GCE encoding worker (not Cloud Run) because 4K H.264
encoding requires more memory than the right-sized Cloud Run container
provides (2Gi). See PR #640 / #647.
Args:
png_path: Path to input PNG image
mov_path: Path for output MOV video
duration: Video duration in seconds (default 5)
Returns:
Path to generated MOV file
Raises:
RuntimeError: If FFmpeg fails
"""
command = [
"ffmpeg", "-hide_banner", "-loglevel", "error", "-nostats", "-y",
"-loop", "1", "-framerate", "30", "-i", str(png_path),
"-f", "lavfi", "-i", "anullsrc",
"-c:v", "libx264", "-r", "30", "-t", str(duration),
"-pix_fmt", "yuv420p", "-vf", "scale=3840:2160",
"-c:a", "aac", "-shortest",
str(mov_path),
]
logger.info(f"Generating MOV from PNG: {png_path.name} -> {mov_path.name}")
result = subprocess.run(command, capture_output=True, text=True, timeout=120)
if result.returncode != 0:
raise RuntimeError(f"FFmpeg failed to generate {mov_path.name}: {result.stderr}")
if not mov_path.exists() or mov_path.stat().st_size < 1024:
raise RuntimeError(f"FFmpeg produced invalid output for {mov_path.name}")
logger.info(f"Generated MOV: {mov_path.name} ({mov_path.stat().st_size} bytes)")
return mov_path
def run_encoding(job_id: str, work_dir: Path, config: dict):
'''Run encoding using LocalEncodingService (single source of truth).
@@ -622,43 +316,14 @@ def run_encoding(job_id: str, work_dir: Path, config: dict):
# Get artist/title from config for proper naming
artist = config.get("artist", "Unknown Artist")
title = config.get("title", "Unknown Title")
# Sanitize for use in file paths (e.g., title "PORTRAIT/WALK/BORN" → "PORTRAIT_WALK_BORN")
safe_artist = re.sub(r'[<>:"/\\|?*]', '_', artist)
safe_title = re.sub(r'[<>:"/\\|?*]', '_', title)
base_name = f"{safe_artist} - {safe_title}"
base_name = f"{artist} - {title}"
logger.info(f"Encoding for: {base_name}")
# Find input files in work_dir
# Title/end screens: look for MOV first, fall back to generating from PNG.
# Since PR #640 right-sized Cloud Run to 2Gi, the screens_worker only generates
# PNG/JPG images — MOV video generation now happens here on the GCE worker.
# Title/end screens are in screens/ subdirectory
title_video = find_file(work_dir, "screens/title.mov", "*Title*.mov", "*title*.mov")
end_video = find_file(work_dir, "screens/end.mov", "*End*.mov", "*end*.mov")
# Get intro/end video durations from style config (default 5s)
intro_duration = config.get("intro_video_duration", 5)
end_duration = config.get("end_video_duration", 5)
# Generate title MOV from PNG if missing or invalid (< 1KB = empty shell)
if not title_video or title_video.stat().st_size < 1024:
title_png = find_file(work_dir, "screens/title.png", "*Title*.png", "*title*.png")
if title_png:
title_mov_path = title_png.parent / "title.mov"
logger.info(f"Title MOV missing/invalid, generating from PNG: {title_png}")
title_video = generate_mov_from_png(title_png, title_mov_path, duration=intro_duration)
elif title_video:
logger.warning(f"Title MOV is only {title_video.stat().st_size} bytes and no PNG found")
# Generate end MOV from PNG if missing or invalid
if not end_video or (end_video and end_video.stat().st_size < 1024):
end_png = find_file(work_dir, "screens/end.png", "*End*.png", "*end*.png")
if end_png:
end_mov_path = end_png.parent / "end.mov"
logger.info(f"End MOV missing/invalid, generating from PNG: {end_png}")
end_video = generate_mov_from_png(end_png, end_mov_path, duration=end_duration)
elif end_video:
logger.warning(f"End MOV is only {end_video.stat().st_size} bytes and no PNG found")
# Karaoke video - search for With Vocals or main karaoke video
# IMPORTANT: Search order matters! Search specific paths first to avoid picking up
# old encoding outputs from finals/ directory when re-encoding after reset
@@ -692,26 +357,9 @@ def run_encoding(job_id: str, work_dir: Path, config: dict):
# Instrumental audio - respect user's selection from encoding config
instrumental_selection = config.get("instrumental_selection", "clean")
existing_instrumental = config.get("existing_instrumental")
logger.info(f"Instrumental selection from config: {instrumental_selection}")
if existing_instrumental:
logger.info(f"Existing instrumental from config: {existing_instrumental}")
if existing_instrumental:
# User-provided instrumental uploaded at job creation
# Downloaded to work_dir by process_job before run_encoding is called
instrumental = find_file(
work_dir,
"*existing_instrumental*", "*Instrumental User*",
)
elif instrumental_selection == "custom":
# Custom instrumental created via mute-region editing in review UI
instrumental = find_file(
work_dir,
"*custom_instrumental*.flac", "*Instrumental Custom*.flac",
"*custom_instrumental*.mp3",
)
elif instrumental_selection == "with_backing":
if instrumental_selection == "with_backing":
# User selected instrumental with backing vocals
instrumental = find_file(
work_dir,
@@ -824,17 +472,6 @@ async def process_job(job_id: str, request: EncodeRequest):
logger.info(f"Downloading from {request.input_gcs_path}")
download_from_gcs(request.input_gcs_path, work_dir)
# Download existing instrumental if present (stored under uploads/, not jobs/)
existing_instrumental = request.encoding_config.get("existing_instrumental") if request.encoding_config else None
if existing_instrumental:
# Extract bucket name from input_gcs_path (gs://bucket/jobs/...)
bucket_name = request.input_gcs_path.replace("gs://", "").split("/", 1)[0]
ext = Path(existing_instrumental).suffix.lower()
dest = work_dir / f"existing_instrumental{ext}"
gcs_uri = f"gs://{bucket_name}/{existing_instrumental}"
logger.info(f"Downloading existing instrumental: {gcs_uri} -> {dest}")
download_single_file_from_gcs(gcs_uri, dest)
# Run encoding in thread pool (CPU-bound)
loop = asyncio.get_event_loop()
output_dir = await loop.run_in_executor(
@@ -903,36 +540,6 @@ async def process_preview_job(job_id: str, request: EncodePreviewRequest):
jobs[job_id]["error"] = str(e)
async def process_render_video_job(job_id: str, request: RenderVideoRequest):
# Process a render-video job asynchronously
try:
# Download and install latest wheel at job start (allows hot updates without restart)
ensure_latest_wheel()
with tempfile.TemporaryDirectory() as temp_dir:
work_dir = Path(temp_dir) / "work"
work_dir.mkdir()
# Run render-video in thread pool (CPU-bound)
loop = asyncio.get_event_loop()
await loop.run_in_executor(
executor,
run_render_video,
job_id,
work_dir,
request
)
jobs[job_id]["status"] = "complete"
jobs[job_id]["progress"] = 100
logger.info(f"Render video job {job_id} complete")
except Exception as e:
logger.error(f"Render video job {job_id} failed: {e}")
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = str(e)
@app.post("/encode-preview")
async def submit_preview_encode_job(request: EncodePreviewRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
# Submit a preview encoding job
@@ -964,54 +571,13 @@ async def submit_preview_encode_job(request: EncodePreviewRequest, background_ta
return {"status": "accepted", "job_id": job_id}
@app.post("/render-video")
async def submit_render_video_job(request: RenderVideoRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
# Submit a render-video job
job_id = request.job_id
# If job already exists, return cached result or current status
if job_id in jobs:
existing_job = jobs[job_id]
if existing_job["status"] == "complete":
return {"status": "cached", "job_id": job_id, "output_files": existing_job.get("output_files"), "metadata": existing_job.get("metadata")}
elif existing_job["status"] == "failed":
# Previous attempt failed, allow retry by replacing the job
pass
else:
# Job is still in progress
return {"status": "in_progress", "job_id": job_id}
jobs[job_id] = {
"job_id": job_id,
"status": "pending",
"progress": 0,
"error": None,
"output_files": None,
"metadata": None,
}
background_tasks.add_task(process_render_video_job, job_id, request)
return {"status": "accepted", "job_id": job_id}
@app.post("/encode")
async def submit_encode_job(request: EncodeRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
# Submit an encoding job
job_id = request.job_id
# If job already exists, return cached result or current status
if job_id in jobs:
existing_job = jobs[job_id]
if existing_job["status"] == "complete":
# Return cached result - encoding already done
return {"status": "cached", "job_id": job_id, "output_files": existing_job.get("output_files")}
elif existing_job["status"] == "failed":
# Previous attempt failed, allow retry by replacing the job
pass
else:
# Job is still in progress
return {"status": "in_progress", "job_id": job_id}
raise HTTPException(status_code=409, detail=f"Job {job_id} already exists")
jobs[job_id] = {
"job_id": job_id,
+66 -189
View File
@@ -12,19 +12,13 @@ shared with YouTube credentials if scopes include drive.file.
import json
import logging
import os
import ssl
from typing import Any, Dict, Optional
from tenacity import Retrying, stop_after_attempt, wait_exponential, retry_if_exception_type
from backend.config import get_settings
from karaoke_gen.utils import sanitize_filename
logger = logging.getLogger(__name__)
# Transient errors that indicate a stale HTTP connection (Cloud Run idle containers)
TRANSIENT_ERRORS = (BrokenPipeError, ConnectionResetError, ConnectionError, ssl.SSLError)
class GoogleDriveService:
"""Google Drive operations using native API."""
@@ -39,11 +33,6 @@ class GoogleDriveService:
self._credentials_data: Optional[Dict[str, Any]] = None
self._loaded = False
def _reset_service(self):
"""Reset the cached Drive service to force a fresh connection on next use."""
logger.info("Resetting Google Drive service connection")
self._service = None
def _load_credentials(self) -> Optional[Dict[str, Any]]:
"""Load OAuth credentials from Secret Manager."""
if self._loaded:
@@ -138,9 +127,6 @@ class GoogleDriveService:
"""
Get existing folder or create new one, return folder ID.
Retries on transient connection errors (BrokenPipeError, SSLError)
that occur when Cloud Run containers sit idle between jobs.
Args:
parent_id: Parent folder ID
folder_name: Name of folder to find or create
@@ -148,41 +134,33 @@ class GoogleDriveService:
Returns:
Folder ID
"""
for attempt in Retrying(
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=2, max=8),
retry=retry_if_exception_type(TRANSIENT_ERRORS),
before_sleep=lambda retry_state: self._reset_service(),
reraise=True,
):
with attempt:
logger.info(f"Looking for folder '{folder_name}' in parent {parent_id}")
logger.info(f"Looking for folder '{folder_name}' in parent {parent_id}")
# Search for existing folder
# Escape single quotes in folder name for Google Drive API query syntax
escaped_folder_name = folder_name.replace("'", "\\'")
query = (
f"name='{escaped_folder_name}' and '{parent_id}' in parents "
f"and mimeType='application/vnd.google-apps.folder' and trashed=false"
)
results = self.service.files().list(q=query, fields="files(id, name)").execute()
# Search for existing folder
# Escape single quotes in folder name for Google Drive API query syntax
escaped_folder_name = folder_name.replace("'", "\\'")
query = (
f"name='{escaped_folder_name}' and '{parent_id}' in parents "
f"and mimeType='application/vnd.google-apps.folder' and trashed=false"
)
results = self.service.files().list(q=query, fields="files(id, name)").execute()
if results.get("files"):
folder_id = results["files"][0]["id"]
logger.info(f"Found existing folder '{folder_name}': {folder_id}")
return folder_id
if results.get("files"):
folder_id = results["files"][0]["id"]
logger.info(f"Found existing folder '{folder_name}': {folder_id}")
return folder_id
# Create folder
logger.info(f"Creating new folder '{folder_name}'")
metadata = {
"name": folder_name,
"mimeType": "application/vnd.google-apps.folder",
"parents": [parent_id],
}
folder = self.service.files().create(body=metadata, fields="id").execute()
folder_id = folder["id"]
logger.info(f"Created folder '{folder_name}': {folder_id}")
return folder_id
# Create folder
logger.info(f"Creating new folder '{folder_name}'")
metadata = {
"name": folder_name,
"mimeType": "application/vnd.google-apps.folder",
"parents": [parent_id],
}
folder = self.service.files().create(body=metadata, fields="id").execute()
folder_id = folder["id"]
logger.info(f"Created folder '{folder_name}': {folder_id}")
return folder_id
def upload_file(
self,
@@ -194,9 +172,6 @@ class GoogleDriveService:
"""
Upload a file to a specific Drive folder.
Retries on transient connection errors (BrokenPipeError, SSLError)
that occur when Cloud Run containers sit idle between jobs.
Args:
local_path: Local file path
parent_id: Parent folder ID in Google Drive
@@ -208,57 +183,49 @@ class GoogleDriveService:
"""
from googleapiclient.http import MediaFileUpload
for attempt in Retrying(
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=2, max=8),
retry=retry_if_exception_type(TRANSIENT_ERRORS),
before_sleep=lambda retry_state: self._reset_service(),
reraise=True,
):
with attempt:
file_size = os.path.getsize(local_path)
logger.info(
f"Uploading {local_path} ({file_size / 1024 / 1024:.1f} MB) "
f"as '{filename}' to folder {parent_id}"
)
file_size = os.path.getsize(local_path)
logger.info(
f"Uploading {local_path} ({file_size / 1024 / 1024:.1f} MB) "
f"as '{filename}' to folder {parent_id}"
)
# Determine MIME type
ext = os.path.splitext(local_path)[1].lower()
mime_types = {
".mp4": "video/mp4",
".mkv": "video/x-matroska",
".zip": "application/zip",
".flac": "audio/flac",
".mp3": "audio/mpeg",
".wav": "audio/wav",
".png": "image/png",
".jpg": "image/jpeg",
".jpeg": "image/jpeg",
}
mime_type = mime_types.get(ext, "application/octet-stream")
# Determine MIME type
ext = os.path.splitext(local_path)[1].lower()
mime_types = {
".mp4": "video/mp4",
".mkv": "video/x-matroska",
".zip": "application/zip",
".flac": "audio/flac",
".mp3": "audio/mpeg",
".wav": "audio/wav",
".png": "image/png",
".jpg": "image/jpeg",
".jpeg": "image/jpeg",
}
mime_type = mime_types.get(ext, "application/octet-stream")
# Check for existing file with same name
if replace_existing:
# Escape single quotes in filename for Google Drive API query syntax
escaped_filename = filename.replace("'", "\\'")
query = (
f"name='{escaped_filename}' and '{parent_id}' in parents and trashed=false"
)
results = self.service.files().list(q=query, fields="files(id)").execute()
for existing_file in results.get("files", []):
logger.info(f"Deleting existing file: {existing_file['id']}")
self.service.files().delete(fileId=existing_file["id"]).execute()
# Check for existing file with same name
if replace_existing:
# Escape single quotes in filename for Google Drive API query syntax
escaped_filename = filename.replace("'", "\\'")
query = (
f"name='{escaped_filename}' and '{parent_id}' in parents and trashed=false"
)
results = self.service.files().list(q=query, fields="files(id)").execute()
for existing_file in results.get("files", []):
logger.info(f"Deleting existing file: {existing_file['id']}")
self.service.files().delete(fileId=existing_file["id"]).execute()
# Upload file
metadata = {"name": filename, "parents": [parent_id]}
media = MediaFileUpload(local_path, mimetype=mime_type, resumable=True)
# Upload file
metadata = {"name": filename, "parents": [parent_id]}
media = MediaFileUpload(local_path, mimetype=mime_type, resumable=True)
file_result = (
self.service.files().create(body=metadata, media_body=media, fields="id").execute()
)
file_id = file_result["id"]
logger.info(f"Successfully uploaded '{filename}': {file_id}")
return file_id
file_result = (
self.service.files().create(body=metadata, media_body=media, fields="id").execute()
)
file_id = file_result["id"]
logger.info(f"Successfully uploaded '{filename}': {file_id}")
return file_id
def upload_to_public_share(
self,
@@ -341,9 +308,6 @@ class GoogleDriveService:
"""
Delete a file from Google Drive.
Retries on transient connection errors (BrokenPipeError, SSLError)
that occur when Cloud Run containers sit idle between jobs.
Args:
file_id: Google Drive file ID to delete
@@ -353,17 +317,9 @@ class GoogleDriveService:
logger.info(f"Deleting Google Drive file: {file_id}")
try:
for attempt in Retrying(
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=2, max=8),
retry=retry_if_exception_type(TRANSIENT_ERRORS),
before_sleep=lambda retry_state: self._reset_service(),
reraise=True,
):
with attempt:
self.service.files().delete(fileId=file_id).execute()
logger.info(f"Successfully deleted file: {file_id}")
return True
self.service.files().delete(fileId=file_id).execute()
logger.info(f"Successfully deleted file: {file_id}")
return True
except Exception as e:
# Check if it's a 404 (already deleted)
if hasattr(e, 'resp') and e.resp.status == 404:
@@ -387,85 +343,6 @@ class GoogleDriveService:
results[file_id] = self.delete_file(file_id)
return results
def find_files_by_brand_code(self, root_folder_id: str, brand_code: str) -> list[str]:
"""
Search for files in public share subfolders matching a brand code prefix.
Used as a fallback when gdrive_files is not tracked in state_data (e.g., old
jobs uploaded before file ID tracking was implemented, or jobs where GDrive
upload succeeded but file IDs were not saved due to a bug).
Searches CDG/, MP4/, MP4-720p/ subfolders for files whose name starts with
'{brand_code} - '.
Args:
root_folder_id: Root Google Drive folder ID for the public share
brand_code: Brand code prefix to search for (e.g., "NOMAD-1271")
Returns:
List of file IDs found matching the brand code
"""
found_ids = []
subfolders = ["CDG", "MP4", "MP4-720p"]
search_errors: list[Exception] = []
for subfolder in subfolders:
try:
# Find the subfolder
escaped_subfolder = subfolder.replace("'", "\\'")
query = (
f"name='{escaped_subfolder}' and '{root_folder_id}' in parents "
f"and mimeType='application/vnd.google-apps.folder' and trashed=false"
)
results = self.service.files().list(
q=query, fields="files(id, name)", supportsAllDrives=True,
includeItemsFromAllDrives=True
).execute()
if not results.get("files"):
logger.debug(f"Subfolder '{subfolder}' not found in {root_folder_id}")
continue
subfolder_id = results["files"][0]["id"]
# Search for files whose name starts with the brand code
# Use contains for efficiency, then filter by exact prefix
escaped_brand = brand_code.replace("'", "\\'")
file_query = (
f"name contains '{escaped_brand}' and '{subfolder_id}' in parents "
f"and trashed=false"
)
file_results = self.service.files().list(
q=file_query, fields="files(id, name)", supportsAllDrives=True,
includeItemsFromAllDrives=True
).execute()
for f in file_results.get("files", []):
if f["name"].startswith(f"{brand_code} - "):
found_ids.append(f["id"])
logger.info(
f"Found GDrive file to clean up in {subfolder}/: "
f"{f['name']} ({f['id']})"
)
except Exception as e:
logger.error(
f"Error searching for brand_code '{brand_code}' in '{subfolder}': {e}",
exc_info=True,
)
search_errors.append(e)
if search_errors:
# Raise so the caller knows the search was incomplete.
# This prevents the cleanup endpoint from treating an uncertain result
# as "GDrive is clean" and incorrectly recycling the brand code.
raise RuntimeError(
f"GDrive brand_code search incomplete: "
f"{len(search_errors)}/{len(subfolders)} subfolder(s) failed"
)
return found_ids
# Singleton instance
_gdrive_service: Optional[GoogleDriveService] = None
@@ -1,62 +0,0 @@
"""
GDrive Validator Cloud Function client.
Lightweight HTTP client for invoking the gdrive-validator Cloud Function
from the backend (e.g., after job completion). Uses OIDC authentication
via the backend's service account.
Fire-and-forget pattern: never blocks or fails the caller.
"""
import logging
import os
import requests
import google.auth.transport.requests
import google.oauth2.id_token
logger = logging.getLogger(__name__)
# Cloud Function URL - set via environment variable
GDRIVE_VALIDATOR_URL = os.environ.get("GDRIVE_VALIDATOR_URL", "")
def trigger_gdrive_validation() -> dict | None:
"""
Invoke the GDrive validator Cloud Function.
Uses the backend service account's OIDC token for authentication.
Returns:
Parsed JSON response from the function, or None on error.
"""
if not GDRIVE_VALIDATOR_URL:
logger.debug("GDRIVE_VALIDATOR_URL not set, skipping GDrive validation trigger")
return None
try:
# Get OIDC token for the Cloud Function URL
auth_request = google.auth.transport.requests.Request()
token = google.oauth2.id_token.fetch_id_token(auth_request, GDRIVE_VALIDATOR_URL)
response = requests.post(
GDRIVE_VALIDATOR_URL,
headers={"Authorization": f"Bearer {token}"},
timeout=60,
)
response.raise_for_status()
result = response.json()
status = result.get("status", "unknown")
if status == "issues_found":
logger.warning("GDrive validation found issues (notification sent)")
elif status == "ok":
logger.info("GDrive validation: all clear")
else:
logger.info(f"GDrive validation returned status: {status}")
return result
except Exception:
logger.exception("Failed to trigger GDrive validation (non-fatal)")
return None
-228
View File
@@ -1,228 +0,0 @@
"""
IP geolocation service with Firestore caching.
Looks up IP addresses via ip-api.com (free, no key needed) and caches
results in Firestore to avoid repeated lookups. Used by admin UI to
enrich IP displays with country, city, ISP, and ASN info.
"""
import ipaddress
import logging
import time
from datetime import datetime, timedelta
from typing import Optional, Dict, List
import httpx
from google.cloud import firestore
from backend.config import get_settings
logger = logging.getLogger(__name__)
IP_GEOLOCATION_COLLECTION = "ip_geolocation"
CACHE_TTL_DAYS = 90 # Cache results for 90 days
IP_API_URL = "http://ip-api.com/json"
IP_API_BATCH_URL = "http://ip-api.com/batch"
IP_API_FIELDS = "status,country,countryCode,regionName,city,isp,org,as,timezone,query"
RATE_LIMIT_PER_MINUTE = 45
def _is_private_ip(ip: str) -> bool:
"""Check if an IP is private/reserved (not routable on the internet)."""
try:
addr = ipaddress.ip_address(ip)
return addr.is_private or addr.is_reserved or addr.is_loopback or addr.is_link_local
except ValueError:
return True # Invalid IP
def _parse_ip_api_response(data: dict) -> dict:
"""Parse ip-api.com response into our standard format."""
if data.get("status") != "success":
return {
"status": "fail",
"ip": data.get("query", ""),
"cached_at": datetime.utcnow().isoformat(),
}
# Parse AS field: "AS14615 Comporium Inc" -> number + name
as_field = data.get("as", "")
as_number = ""
as_name = ""
if as_field:
parts = as_field.split(" ", 1)
as_number = parts[0] if parts else ""
as_name = parts[1] if len(parts) > 1 else ""
return {
"status": "success",
"ip": data.get("query", ""),
"country": data.get("country", ""),
"country_code": data.get("countryCode", ""),
"region": data.get("regionName", ""),
"city": data.get("city", ""),
"isp": data.get("isp", ""),
"org": data.get("org", ""),
"as_number": as_number,
"as_name": as_name,
"timezone": data.get("timezone", ""),
"cached_at": datetime.utcnow().isoformat(),
}
class IpGeolocationService:
"""IP geolocation lookup with Firestore caching."""
def __init__(self):
settings = get_settings()
self.db = firestore.Client(project=settings.google_cloud_project)
self._last_request_times: List[float] = []
def _check_rate_limit(self):
"""Simple rate limiter — sleep if approaching 45 req/min."""
now = time.time()
# Remove entries older than 60 seconds
self._last_request_times = [
t for t in self._last_request_times if now - t < 60
]
if len(self._last_request_times) >= RATE_LIMIT_PER_MINUTE - 5:
# Getting close to limit, wait
sleep_time = 60 - (now - self._last_request_times[0])
if sleep_time > 0:
logger.info(f"IP geolocation rate limit: sleeping {sleep_time:.1f}s")
time.sleep(sleep_time)
self._last_request_times.append(time.time())
def _get_cached(self, ip: str) -> Optional[dict]:
"""Get cached geolocation data for an IP."""
try:
# Firestore doc IDs can't contain '/' so IPv6 with :: is fine,
# but we need to sanitize for doc ID
doc_id = ip.replace("/", "_")
doc = self.db.collection(IP_GEOLOCATION_COLLECTION).document(doc_id).get()
if not doc.exists:
return None
data = doc.to_dict()
# Check TTL
cached_at = data.get("cached_at", "")
if cached_at:
try:
cached_dt = datetime.fromisoformat(cached_at)
if datetime.utcnow() - cached_dt > timedelta(days=CACHE_TTL_DAYS):
return None # Expired
except (ValueError, TypeError):
pass
return data
except Exception:
logger.exception(f"Error reading IP geo cache for {ip}")
return None
def _cache_result(self, ip: str, data: dict):
"""Store geolocation result in Firestore cache."""
try:
doc_id = ip.replace("/", "_")
self.db.collection(IP_GEOLOCATION_COLLECTION).document(doc_id).set(data)
except Exception:
logger.exception(f"Error caching IP geo for {ip}")
def lookup_ip(self, ip: str) -> dict:
"""
Look up geolocation for a single IP address.
Returns cached result if available, otherwise queries ip-api.com.
"""
if not ip or _is_private_ip(ip):
return {"status": "private", "ip": ip or ""}
# Check cache
cached = self._get_cached(ip)
if cached:
return cached
# Query ip-api.com
try:
self._check_rate_limit()
with httpx.Client(timeout=10) as client:
response = client.get(
f"{IP_API_URL}/{ip}",
params={"fields": IP_API_FIELDS},
)
response.raise_for_status()
data = response.json()
result = _parse_ip_api_response(data)
self._cache_result(ip, result)
return result
except Exception:
logger.exception(f"Error looking up IP {ip}")
return {"status": "error", "ip": ip}
def lookup_ips_batch(self, ips: List[str]) -> Dict[str, dict]:
"""
Look up geolocation for multiple IPs.
Returns dict of ip -> geo data. Uses cache for known IPs,
batches uncached lookups to ip-api.com.
"""
results = {}
uncached = []
for ip in set(ips): # Deduplicate
if not ip or _is_private_ip(ip):
results[ip] = {"status": "private", "ip": ip or ""}
continue
cached = self._get_cached(ip)
if cached:
results[ip] = cached
else:
uncached.append(ip)
if not uncached:
return results
# ip-api.com batch endpoint accepts up to 100 IPs per request
for i in range(0, len(uncached), 100):
batch = uncached[i:i + 100]
try:
self._check_rate_limit()
# Batch endpoint takes a JSON array of IPs
payload = [
{"query": ip, "fields": IP_API_FIELDS}
for ip in batch
]
with httpx.Client(timeout=15) as client:
response = client.post(IP_API_BATCH_URL, json=payload)
response.raise_for_status()
batch_results = response.json()
for data in batch_results:
result = _parse_ip_api_response(data)
ip = result.get("ip", data.get("query", ""))
if ip:
results[ip] = result
self._cache_result(ip, result)
except Exception:
logger.exception(f"Error in batch IP lookup ({len(batch)} IPs)")
# Mark failed IPs
for ip in batch:
if ip not in results:
results[ip] = {"status": "error", "ip": ip}
return results
# Global instance
_service = None
def get_ip_geolocation_service() -> IpGeolocationService:
"""Get the global IP geolocation service instance."""
global _service
if _service is None:
_service = IpGeolocationService()
return _service
-50
View File
@@ -61,56 +61,6 @@ def get_effective_distribution_settings(
)
def get_effective_distribution_for_job(job) -> EffectiveDistributionSettings:
"""
Get effective distribution settings for a job, applying private overrides if is_private=True.
For private (non-published) jobs, this overrides:
- dropbox_path → private Dropbox path (Tracks-NonPublished)
- brand_prefix → private brand prefix (NOMADNP)
- enable_youtube_upload → False (no YouTube for private)
- gdrive_folder_id → None (no Google Drive for private)
For non-private jobs, returns the job's own distribution settings unchanged.
Args:
job: Job object (or any object with distribution fields)
Returns:
EffectiveDistributionSettings with private overrides applied if needed
"""
if getattr(job, 'is_private', False):
# Tenant jobs: use the job's own distribution fields (set at creation from tenant config)
# Non-tenant private jobs: override to NonPublished paths
if getattr(job, 'tenant_id', None):
return EffectiveDistributionSettings(
dropbox_path=getattr(job, 'dropbox_path', None),
brand_prefix=getattr(job, 'brand_prefix', None),
enable_youtube_upload=False,
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=None,
)
settings = get_settings()
return EffectiveDistributionSettings(
dropbox_path=settings.default_private_dropbox_path,
brand_prefix=settings.default_private_brand_prefix,
enable_youtube_upload=False,
gdrive_folder_id=None,
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=None,
)
# Non-private: return job's own settings
return EffectiveDistributionSettings(
dropbox_path=getattr(job, 'dropbox_path', None),
brand_prefix=getattr(job, 'brand_prefix', None),
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=getattr(job, 'youtube_description_template', None),
)
def resolve_cdg_txt_defaults(
theme_id: Optional[str],
enable_cdg: Optional[bool] = None,
-262
View File
@@ -1,262 +0,0 @@
"""
Job health and consistency checking service.
Provides functions to detect state inconsistencies in jobs that might indicate
bugs in the state machine or trigger logic.
Created as part of state machine robustness improvements (2026-02-02).
"""
import logging
from datetime import datetime, timezone, timedelta
from typing import List, Dict, Any, Optional
from backend.models.job import Job, JobStatus
logger = logging.getLogger(__name__)
# Jobs in these statuses should NOT have audio_complete flag set
# (unless they're actively processing or beyond)
STATUSES_BEFORE_AUDIO_START = {
JobStatus.PENDING,
JobStatus.SEARCHING_AUDIO,
JobStatus.AWAITING_AUDIO_SELECTION,
JobStatus.AWAITING_AUDIO_EDIT,
JobStatus.IN_AUDIO_EDIT,
JobStatus.AUDIO_EDIT_COMPLETE,
}
# Jobs in these statuses should NOT have lyrics_complete flag set
STATUSES_BEFORE_LYRICS_START = {
JobStatus.PENDING,
JobStatus.SEARCHING_AUDIO,
JobStatus.AWAITING_AUDIO_SELECTION,
JobStatus.AWAITING_AUDIO_EDIT,
JobStatus.IN_AUDIO_EDIT,
JobStatus.AUDIO_EDIT_COMPLETE,
}
# Statuses where screens_progress.stage='complete' is valid
STATUSES_AFTER_SCREENS_COMPLETE = {
JobStatus.AWAITING_REVIEW,
JobStatus.IN_REVIEW,
JobStatus.REVIEW_COMPLETE,
JobStatus.RENDERING_VIDEO,
JobStatus.AWAITING_INSTRUMENTAL_SELECTION, # Legacy
JobStatus.INSTRUMENTAL_SELECTED,
JobStatus.GENERATING_VIDEO,
JobStatus.ENCODING,
JobStatus.PACKAGING,
JobStatus.UPLOADING,
JobStatus.NOTIFYING,
JobStatus.COMPLETE,
JobStatus.PREP_COMPLETE,
}
def check_job_consistency(job: Job) -> List[str]:
"""
Check for state inconsistencies in a job.
This function detects situations where state_data flags indicate
progress that doesn't match the job status - which usually indicates
a bug in the state machine or trigger logic.
Args:
job: Job object to check
Returns:
List of issue descriptions found. Empty list means no issues.
"""
issues = []
state_data = job.state_data or {}
# Normalize status to JobStatus enum if it's a string
status = job.status
if isinstance(status, str):
try:
status = JobStatus(status)
except ValueError:
issues.append(f"invalid_status: '{job.status}' is not a valid JobStatus")
return issues
# Check: audio_complete=True but status still in pre-audio states
if state_data.get('audio_complete') and status in STATUSES_BEFORE_AUDIO_START:
issues.append(
f"audio_complete_status_mismatch: audio_complete=True but status={status.value}"
)
# Check: lyrics_complete=True but status still in pre-lyrics states
if state_data.get('lyrics_complete') and status in STATUSES_BEFORE_LYRICS_START:
issues.append(
f"lyrics_complete_status_mismatch: lyrics_complete=True but status={status.value}"
)
# Check: screens_progress.stage='complete' but status not in post-screens states
screens_progress = state_data.get('screens_progress', {})
if isinstance(screens_progress, dict) and screens_progress.get('stage') == 'complete':
if status not in STATUSES_AFTER_SCREENS_COMPLETE:
issues.append(
f"screens_complete_status_mismatch: screens_progress.stage=complete but status={status.value}"
)
# Check: job has input_media_gcs_path but is stuck at pending
# This was the exact bug that got jobs 06cfea29 and 984da08b stuck
if job.input_media_gcs_path and status == JobStatus.PENDING:
# This could be legitimate if job was just created and workers haven't started
# But if it's been more than a few minutes, it's likely stuck
# We flag it as a warning, not an error
issues.append(
f"pending_with_input_media: has input_media_gcs_path but status=pending (may be stuck)"
)
# Check: job stuck in encoding status beyond 50 minutes
# Encoding timeout is 1 hour; if updated_at hasn't advanced in 50 min,
# the poller likely died (e.g., Cloud Run deployment killed the instance)
if status == JobStatus.ENCODING and job.updated_at:
now = datetime.now(timezone.utc)
updated_at = job.updated_at
if updated_at.tzinfo is None:
updated_at = updated_at.replace(tzinfo=timezone.utc)
encoding_age = now - updated_at
if encoding_age > timedelta(minutes=50):
minutes = int(encoding_age.total_seconds() / 60)
issues.append(
f"encoding_stuck: status=encoding for {minutes} min without update (updated_at={updated_at.isoformat()})"
)
# Check: job stuck in downloading_audio status beyond 10 minutes
# Audio downloads typically complete in 30s-5min. If stuck longer,
# the Cloud Run Job likely failed silently.
if status == JobStatus.DOWNLOADING_AUDIO and job.updated_at:
now = datetime.now(timezone.utc)
updated_at = job.updated_at
if updated_at.tzinfo is None:
updated_at = updated_at.replace(tzinfo=timezone.utc)
download_age = now - updated_at
if download_age > timedelta(minutes=10):
minutes = int(download_age.total_seconds() / 60)
issues.append(
f"downloading_audio_stuck: status=downloading_audio for {minutes} min without update (updated_at={updated_at.isoformat()})"
)
return issues
def check_job_consistency_detailed(job: Job) -> Dict[str, Any]:
"""
Check job consistency with detailed information for debugging.
Returns a dictionary with:
- job_id: The job ID
- status: Current job status
- issues: List of issue descriptions
- state_data_summary: Summary of relevant state_data fields
- is_healthy: Boolean indicating if no issues were found
Args:
job: Job object to check
Returns:
Dictionary with detailed consistency information
"""
issues = check_job_consistency(job)
state_data = job.state_data or {}
return {
"job_id": job.job_id,
"status": job.status.value if hasattr(job.status, 'value') else str(job.status),
"issues": issues,
"state_data_summary": {
"audio_complete": state_data.get('audio_complete', False),
"lyrics_complete": state_data.get('lyrics_complete', False),
"screens_progress_stage": state_data.get('screens_progress', {}).get('stage'),
"has_input_media_gcs_path": bool(job.input_media_gcs_path),
},
"is_healthy": len(issues) == 0,
}
def get_worker_valid_statuses() -> Dict[str, List[JobStatus]]:
"""
Get mapping of worker names to their valid job statuses.
This is used by workers to validate they should run for a given job.
Returns:
Dictionary mapping worker name to list of valid JobStatus values
"""
return {
"audio_worker": [
JobStatus.DOWNLOADING,
JobStatus.DOWNLOADING_AUDIO,
JobStatus.AUDIO_EDIT_COMPLETE,
JobStatus.SEPARATING_STAGE1,
JobStatus.SEPARATING_STAGE2,
],
"lyrics_worker": [
JobStatus.DOWNLOADING,
JobStatus.AUDIO_EDIT_COMPLETE,
JobStatus.TRANSCRIBING,
JobStatus.CORRECTING,
],
"screens_worker": [
# Screens worker can be triggered when both audio+lyrics complete
# which can happen in any of these states
JobStatus.DOWNLOADING,
JobStatus.AUDIO_COMPLETE,
JobStatus.LYRICS_COMPLETE,
# Also valid if re-running after a failure
JobStatus.GENERATING_SCREENS,
],
"video_worker": [
JobStatus.INSTRUMENTAL_SELECTED,
JobStatus.REVIEW_COMPLETE,
# Note: RENDERING_VIDEO and GENERATING_VIDEO are intentionally excluded.
# If the job is already in those states, another worker is actively processing it.
# Including them caused a race condition where Cloud Run Job retries + manual retries
# could launch concurrent workers that corrupt each other's state transitions.
],
"render_video_worker": [
JobStatus.REVIEW_COMPLETE,
JobStatus.RENDERING_VIDEO,
],
}
def validate_worker_can_run(worker_name: str, job: Job) -> Optional[str]:
"""
Validate that a worker should run for the given job.
Args:
worker_name: Name of the worker (e.g., "screens_worker")
job: Job object to validate
Returns:
None if valid, or an error message string if invalid
"""
valid_statuses_map = get_worker_valid_statuses()
if worker_name not in valid_statuses_map:
# Unknown worker, can't validate
return None
valid_statuses = valid_statuses_map[worker_name]
# Normalize status
status = job.status
if isinstance(status, str):
try:
status = JobStatus(status)
except ValueError:
return f"Invalid job status: {job.status}"
if status not in valid_statuses:
valid_names = [s.value for s in valid_statuses]
return (
f"{worker_name} called but job status is {status.value}, "
f"expected one of {valid_names}. This may indicate a bug in the trigger logic."
)
return None
+98 -348
View File
@@ -14,7 +14,7 @@ from datetime import datetime
from typing import Optional, Dict, Any, List
from backend.config import settings
from backend.exceptions import InvalidStateTransitionError, InsufficientCreditsError
from backend.exceptions import RateLimitExceededError
from backend.models.job import Job, JobStatus, JobCreate, STATE_TRANSITIONS
from backend.models.worker_log import WorkerLogEntry
from backend.services.firestore_service import FirestoreService
@@ -51,21 +51,33 @@ class JobManager:
Args:
job_create: Job creation parameters
is_admin: Whether the requesting user is an admin (bypasses credit checks)
is_admin: Whether the requesting user is an admin (bypasses rate limits)
Raises:
ValueError: If theme_id is not provided (all jobs require a theme)
RateLimitExceededError: If user has exceeded their daily job limit
"""
# Check credits (skip for admins)
if job_create.user_email and not is_admin:
from backend.services.user_service import get_user_service
user_service = get_user_service()
if not user_service.has_credits(job_create.user_email):
credits_available = user_service.check_credits(job_create.user_email)
raise InsufficientCreditsError(
message="You're out of credits. Buy more to continue creating karaoke videos.",
credits_available=credits_available,
credits_required=1,
# Check rate limit FIRST (before any other validation)
# This prevents wasted work if user is rate limited
if job_create.user_email:
from backend.services.rate_limit_service import get_rate_limit_service
rate_limit_service = get_rate_limit_service()
allowed, remaining, message = rate_limit_service.check_user_job_limit(
user_email=job_create.user_email,
is_admin=is_admin
)
if not allowed:
from backend.services.rate_limit_service import _seconds_until_midnight_utc
# Get actual current count - remaining is clamped to 0 which loses info
current_count = rate_limit_service.get_user_job_count_today(job_create.user_email)
raise RateLimitExceededError(
message=message,
limit_type="jobs_per_day",
remaining_seconds=_seconds_until_midnight_utc(),
current_count=current_count,
limit_value=settings.rate_limit_jobs_per_day
)
# Enforce theme requirement - all jobs must have a theme
@@ -118,38 +130,20 @@ class JobManager:
made_for_you=job_create.made_for_you,
customer_email=job_create.customer_email,
customer_notes=job_create.customer_notes,
# Private (non-published) track mode
is_private=job_create.is_private,
# Capture creation-time decision params for observability
creation_params={
"is_private": job_create.is_private,
"is_admin": is_admin,
"created_from": job_create.request_metadata.get("created_from", "unknown"),
},
)
self.firestore.create_job(job)
logger.info(f"Created new job {job_id} with status PENDING")
# Deduct credit atomically (after job is persisted so we have job_id for transaction record)
if job_create.user_email and not is_admin:
from backend.services.user_service import get_user_service
user_service = get_user_service()
success, _remaining, deduct_msg = user_service.deduct_credit(
job_create.user_email, job_id, reason="job_creation"
)
if not success:
# Deduction failed (e.g., race condition) - delete the job and raise error
logger.error(f"Credit deduction failed for job {job_id}: {deduct_msg}")
try:
self.firestore.delete_job(job_id)
except Exception:
logger.exception(f"Failed to delete job {job_id} after credit deduction failure")
raise InsufficientCreditsError(
message="Failed to deduct credit. Please try again.",
credits_available=0,
credits_required=1,
)
# Record job creation for rate limiting (after successful persistence)
if job_create.user_email:
try:
from backend.services.rate_limit_service import get_rate_limit_service
rate_limit_service = get_rate_limit_service()
rate_limit_service.record_job_creation(job_create.user_email, job_id)
except Exception as e:
# Don't fail job creation if rate limit recording fails
logger.warning(f"Failed to record job creation for rate limiting: {e}")
return job
@@ -216,31 +210,6 @@ class JobManager:
limit=limit
)
def list_jobs_summary(
self,
status: Optional[JobStatus] = None,
exclude_statuses: Optional[List[str]] = None,
environment: Optional[str] = None,
client_id: Optional[str] = None,
created_after: Optional[datetime] = None,
created_before: Optional[datetime] = None,
user_email: Optional[str] = None,
tenant_id: Optional[str] = None,
limit: int = 100,
) -> List[Dict[str, Any]]:
"""List jobs with field projection (summary mode)."""
return self.firestore.list_jobs_summary(
status=status,
exclude_statuses=exclude_statuses,
environment=environment,
client_id=client_id,
created_after=created_after,
created_before=created_before,
user_email=user_email,
tenant_id=tenant_id,
limit=limit,
)
def delete_jobs_by_filter(
self,
environment: Optional[str] = None,
@@ -327,20 +296,14 @@ class JobManager:
def delete_job(self, job_id: str, delete_files: bool = True) -> None:
"""Delete a job, its files, and its logs subcollection."""
job = self.get_job(job_id)
# Refund credit if job never completed (and wasn't already refunded via fail/cancel)
if job:
no_refund_statuses = [JobStatus.COMPLETE, JobStatus.PREP_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED]
if job.status not in no_refund_statuses:
self._refund_credit_for_job(job_id, job, reason="job_deleted")
if delete_files and job and job.output_files:
for gcs_path in job.output_files.values():
try:
self.storage.delete_file(gcs_path)
except Exception as e:
logger.error(f"Error deleting file {gcs_path}: {e}")
if delete_files:
job = self.get_job(job_id)
if job and job.output_files:
for gcs_path in job.output_files.values():
try:
self.storage.delete_file(gcs_path)
except Exception as e:
logger.error(f"Error deleting file {gcs_path}: {e}")
# Delete logs subcollection first (must be done before deleting parent doc)
try:
@@ -350,126 +313,82 @@ class JobManager:
except Exception as e:
logger.warning(f"Error deleting logs subcollection for job {job_id}: {e}")
# Delete audio edit sessions subcollection
try:
deleted_sessions = self.firestore.delete_audio_edit_sessions_subcollection(job_id)
if deleted_sessions > 0:
logger.info(f"Deleted {deleted_sessions} audio edit sessions for job {job_id}")
except Exception as e:
logger.warning(f"Error deleting audio edit sessions for job {job_id}: {e}")
self.firestore.delete_job(job_id)
logger.info(f"Deleted job {job_id}")
def validate_state_transition(
self,
job_id: str,
new_status: JobStatus,
raise_on_invalid: bool = True
) -> bool:
def validate_state_transition(self, job_id: str, new_status: JobStatus) -> bool:
"""
Validate that a state transition is legal.
Args:
job_id: Job ID
new_status: Target state
raise_on_invalid: If True, raise InvalidStateTransitionError on invalid
transitions. If False, return False (legacy behavior).
Returns:
True if transition is valid
Raises:
InvalidStateTransitionError: If transition is invalid and raise_on_invalid=True
ValueError: If job not found and raise_on_invalid=True
True if transition is valid, False otherwise
"""
job = self.get_job(job_id)
if not job:
error_msg = f"Job {job_id} not found"
logger.error(error_msg)
if raise_on_invalid:
raise ValueError(error_msg)
logger.error(f"Job {job_id} not found")
return False
current_status = job.status
valid_transitions = STATE_TRANSITIONS.get(current_status, [])
if new_status not in valid_transitions:
error_msg = (
logger.error(
f"Invalid state transition for job {job_id}: "
f"{current_status} -> {new_status}. "
f"Valid transitions: {[s.value for s in valid_transitions]}"
f"Valid transitions: {valid_transitions}"
)
logger.error(error_msg)
if raise_on_invalid:
raise InvalidStateTransitionError(
message=error_msg,
job_id=job_id,
from_status=current_status,
to_status=new_status.value if hasattr(new_status, 'value') else str(new_status),
valid_transitions=[s.value for s in valid_transitions]
)
return False
return True
def transition_to_state(
self,
job_id: str,
new_status: JobStatus,
progress: Optional[int] = None,
message: Optional[str] = None,
state_data_updates: Optional[Dict[str, Any]] = None,
raise_on_invalid: bool = True,
timeline_metadata: Optional[Dict[str, Any]] = None,
state_data_updates: Optional[Dict[str, Any]] = None
) -> bool:
"""
Transition job to new state with validation.
Args:
job_id: Job ID
new_status: Target state
progress: Progress percentage (0-100)
message: Timeline message
state_data_updates: Updates to state_data field
raise_on_invalid: If True (default), raise InvalidStateTransitionError
on invalid transitions. If False, return False silently.
timeline_metadata: Optional structured metadata to attach to the timeline event
Returns:
True if transition succeeded, False if failed (only when raise_on_invalid=False)
Raises:
InvalidStateTransitionError: If transition is invalid and raise_on_invalid=True
True if transition succeeded, False otherwise
"""
if not self.validate_state_transition(job_id, new_status, raise_on_invalid=raise_on_invalid):
if not self.validate_state_transition(job_id, new_status):
return False
updates = {
'status': new_status,
'updated_at': datetime.utcnow()
}
if progress is not None:
updates['progress'] = progress
# Generate review token when entering blocking states that need user action
# Generate review token when entering AWAITING_REVIEW state
# Tokens don't expire - they're job-scoped so low risk, and natural expiry happens when job completes
if new_status in (JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_AUDIO_EDIT):
if new_status == JobStatus.AWAITING_REVIEW:
from backend.api.dependencies import generate_review_token
review_token = generate_review_token()
updates['review_token'] = review_token
updates['review_token_expires_at'] = None # No expiry - token is job-scoped
logger.info(f"Generated review token for job {job_id} (no expiry)")
# If we have state_data_updates, merge them with existing state_data
merged_state_data = None
if state_data_updates:
job = self.get_job(job_id)
if job:
merged_state_data = {**job.state_data, **state_data_updates}
# Update job status (includes timeline event), passing state_data if present
if merged_state_data is not None:
self.update_job_status(
@@ -477,7 +396,6 @@ class JobManager:
status=new_status,
progress=progress,
message=message,
timeline_metadata=timeline_metadata,
state_data=merged_state_data
)
else:
@@ -485,8 +403,7 @@ class JobManager:
job_id=job_id,
status=new_status,
progress=progress,
message=message,
timeline_metadata=timeline_metadata,
message=message
)
# Apply review token update separately if generated
@@ -498,17 +415,6 @@ class JobManager:
logger.info(f"Job {job_id} transitioned to {new_status}")
# Increment user's completed jobs counter on terminal success states
if new_status in (JobStatus.COMPLETE, JobStatus.PREP_COMPLETE):
try:
job = self.get_job(job_id)
if job and job.user_email:
from backend.services.user_service import get_user_service
user_service = get_user_service()
user_service.increment_jobs_completed(job.user_email)
except Exception:
logger.exception(f"Failed to increment jobs_completed for job {job_id}")
# Trigger notifications asynchronously (fire-and-forget)
self._trigger_state_notifications(job_id, new_status)
@@ -539,15 +445,10 @@ class JobManager:
self._send_push_notification(job, "complete")
# Idle reminder scheduling for blocking states
elif new_status in [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.AWAITING_AUDIO_EDIT]:
elif new_status in [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION]:
self._schedule_idle_reminder(job, new_status)
# Send push notification for blocking states
if new_status == JobStatus.AWAITING_AUDIO_EDIT:
action_type = "audio_edit"
elif new_status == JobStatus.AWAITING_REVIEW:
action_type = "lyrics"
else:
action_type = "instrumental"
action_type = "lyrics" if new_status == JobStatus.AWAITING_REVIEW else "instrumental"
self._send_push_notification(job, action_type)
except Exception as e:
@@ -574,7 +475,6 @@ class JobManager:
youtube_url = state_data.get('youtube_url')
dropbox_url = state_data.get('dropbox_link')
brand_code = state_data.get('brand_code')
youtube_queued = state_data.get('youtube_upload_queued', False)
# For made-for-you orders, send to customer and transfer ownership
recipient_email = job.user_email
@@ -598,8 +498,6 @@ class JobManager:
youtube_url=youtube_url,
dropbox_url=dropbox_url,
brand_code=brand_code,
is_private=getattr(job, 'is_private', False),
youtube_queued=youtube_queued,
)
# Try to get existing event loop, create new one if none exists
@@ -634,12 +532,7 @@ class JobManager:
# Record when we entered blocking state (for idle detection)
blocking_entered_at = datetime.utcnow().isoformat()
if new_status == JobStatus.AWAITING_AUDIO_EDIT:
action_type = "audio_edit"
elif new_status == JobStatus.AWAITING_REVIEW:
action_type = "lyrics"
else:
action_type = "instrumental"
action_type = "lyrics" if new_status == JobStatus.AWAITING_REVIEW else "instrumental"
# Update state_data with blocking state info (handle None state_data)
existing_state_data = job.state_data or {}
@@ -706,7 +599,6 @@ class JobManager:
"user_email": job.user_email,
"artist": job.artist,
"title": job.title,
"tenant_id": job.tenant_id or None,
}
async def send_notification():
@@ -748,31 +640,6 @@ class JobManager:
self.update_job(job_id, {'state_data': state_data})
logger.debug(f"Job {job_id} state_data updated: {key} = {value}")
def update_processing_metadata(self, job_id: str, section: str, data) -> None:
"""
Write data into processing_metadata[section] using Firestore dot-notation.
Each worker writes its own section once. This uses Firestore's nested
field update to avoid overwriting other workers' sections.
The section can use dot-notation for deeper nesting, e.g.
'timing.lyrics_worker_seconds' writes to processing_metadata.timing.lyrics_worker_seconds
without overwriting other timing entries.
Args:
job_id: Job ID
section: Metadata section path (e.g., 'transcription', 'timing.audio_worker_seconds')
data: Value to store (dict for sections, scalar for individual fields)
"""
try:
self.firestore.update_job(job_id, {
f"processing_metadata.{section}": data,
})
logger.debug(f"Job {job_id} processing_metadata.{section} updated")
except Exception as e:
# Log but don't raise - metadata storage should never fail a job
logger.warning(f"Job {job_id}: Failed to update processing_metadata.{section}: {e}")
def delete_state_data_key(self, job_id: str, key: str) -> bool:
"""
Delete a specific key from the job's state_data field.
@@ -827,46 +694,6 @@ class JobManager:
logger.error(f"Job {job_id}: Failed to delete state_data keys {keys}: {e}")
return []
def _refund_credit_for_job(self, job_id: str, job: Job, reason: str) -> bool:
"""
Refund one credit for a job if eligible.
Skips refund if: already refunded, no user email, or admin-owned job.
Never raises — refund failure must not block the calling operation.
Returns:
True if credit was refunded, False otherwise.
"""
try:
if job.credit_refunded:
logger.info(f"Credit already refunded for job {job_id}, skipping")
return False
if not job.user_email:
return False
from backend.services.auth_service import is_admin_email
from backend.services.user_service import get_user_service
if is_admin_email(job.user_email):
return False
user_service = get_user_service()
refund_ok, new_balance, refund_msg = user_service.refund_credit(
job.user_email, job_id, reason=reason
)
if refund_ok:
# Mark job as refunded to prevent double-refund
self.update_job(job_id, {'credit_refunded': True})
logger.info(f"Refunded credit for {reason} job {job_id} to {_mask_email(job.user_email)} (balance: {new_balance})")
return True
else:
logger.warning(f"Credit refund failed for job {job_id}: {refund_msg}")
return False
except Exception as e:
logger.warning(f"Error refunding credit for job {job_id}: {e}")
return False
def fail_job(self, job_id: str, error_message: str, error_details: Optional[Dict[str, Any]] = None) -> bool:
"""
Mark a job as failed with error information.
@@ -894,17 +721,11 @@ class JobManager:
)
logger.error(f"Job {job_id} failed: {error_message}")
# Auto-refund credit on failure
job = self.get_job(job_id)
if job:
self._refund_credit_for_job(job_id, job, reason="job_failed")
return True
except Exception as e:
logger.error(f"Error marking job {job_id} as failed: {e}")
return False
def update_file_url(self, job_id: str, category: str, file_type: str, url: str) -> None:
"""
Update a file URL in the job's file_urls structure.
@@ -931,116 +752,49 @@ class JobManager:
def check_parallel_processing_complete(self, job_id: str) -> bool:
"""
Check if lyrics processing is complete (sufficient for screen generation).
Audio separation is decoupled from the critical path — users can start
reviewing lyrics while separation continues in the background. The
instrumental review page handles the case where separation isn't done yet.
Check if both parallel tracks (audio + lyrics) are complete.
This is called after audio_complete or lyrics_complete to determine
if we can proceed to screen generation.
Returns:
True if lyrics are complete, False otherwise
True if both tracks complete, False otherwise
"""
job = self.get_job(job_id)
if not job:
return False
audio_complete = job.state_data.get('audio_complete', False)
lyrics_complete = job.state_data.get('lyrics_complete', False)
return lyrics_complete
async def start_job_processing(
self,
job_id: str,
progress: int = 15,
message: str = "Starting audio and lyrics processing"
) -> None:
"""
Transition job to DOWNLOADING and trigger audio + lyrics workers.
This is the single entry point for starting job processing after audio
is available. All handlers (file_upload, webhook, audio_search) should
use this method to ensure consistent state transitions.
The method:
1. Validates job exists and has input_media_gcs_path
2. Transitions to DOWNLOADING status (raises if invalid)
3. Triggers audio and lyrics workers in parallel
Args:
job_id: Job ID to start processing
progress: Progress percentage to set (default 15)
message: Timeline message (default "Starting audio and lyrics processing")
Raises:
InvalidStateTransitionError: If job can't transition to DOWNLOADING
ValueError: If job not found or missing input_media_gcs_path
Example:
# After downloading audio to GCS:
job_manager.update_job(job_id, {'input_media_gcs_path': audio_path})
await job_manager.start_job_processing(job_id)
"""
import asyncio
from backend.services.worker_service import get_worker_service
job = self.get_job(job_id)
if not job:
raise ValueError(f"Job {job_id} not found")
if not job.input_media_gcs_path:
raise ValueError(
f"Job {job_id} missing input_media_gcs_path. "
"Audio must be downloaded to GCS before starting processing."
)
# Transition to DOWNLOADING - this will raise if invalid
self.transition_to_state(
job_id=job_id,
new_status=JobStatus.DOWNLOADING,
progress=progress,
message=message
)
# Check if audio separation can be skipped
# Jobs with an existing instrumental (e.g. tenant uploads) don't need
# GPU-powered audio separation — saves 5-10 minutes of processing time
worker_service = get_worker_service()
if job.existing_instrumental_gcs_path:
logger.info(f"Job {job_id}: Skipping audio separation (existing instrumental provided)")
# Mark audio as immediately complete and only trigger lyrics worker
self.update_state_data(job_id, 'audio_complete', True)
await worker_service.trigger_lyrics_worker(job_id)
logger.info(f"Job {job_id}: Started processing (lyrics worker only, audio separation skipped)")
else:
# Trigger both workers in parallel
await asyncio.gather(
worker_service.trigger_audio_worker(job_id),
worker_service.trigger_lyrics_worker(job_id)
)
logger.info(f"Job {job_id}: Started processing (audio + lyrics workers triggered)")
return audio_complete and lyrics_complete
def mark_audio_complete(self, job_id: str) -> None:
"""
Mark audio processing as complete.
Audio completion is decoupled from the lyrics review path — it no longer
triggers screens. The instrumental review page polls for this flag.
Mark audio processing as complete and check if can proceed.
If lyrics are also complete, automatically triggers screens worker.
"""
self.update_state_data(job_id, 'audio_complete', True)
logger.info(f"Job {job_id}: Audio processing complete (separation finished)")
if self.check_parallel_processing_complete(job_id):
logger.info(f"Job {job_id}: Both audio and lyrics complete, triggering screens worker")
# Transition happens in screens worker
# We just trigger it here
self._trigger_screens_worker(job_id)
def mark_lyrics_complete(self, job_id: str) -> None:
"""
Mark lyrics processing as complete and trigger screens if ready.
Lyrics completion is the sole gate for screen generation and review.
Audio separation runs independently in the background.
Mark lyrics processing as complete and check if can proceed.
If audio is also complete, automatically triggers screens worker.
"""
self.update_state_data(job_id, 'lyrics_complete', True)
if self.check_parallel_processing_complete(job_id):
logger.info(f"Job {job_id}: Lyrics complete, triggering screens worker")
logger.info(f"Job {job_id}: Both audio and lyrics complete, triggering screens worker")
# Transition happens in screens worker
# We just trigger it here
self._trigger_screens_worker(job_id)
def _trigger_screens_worker(self, job_id: str) -> None:
@@ -1099,12 +853,8 @@ class JobManager:
)
logger.info(f"Job {job_id} cancelled")
# Refund credit for cancelled job
self._refund_credit_for_job(job_id, job, reason="job_cancelled")
return True
def mark_job_failed(
self,
job_id: str,
+6 -119
View File
@@ -12,10 +12,8 @@ import logging
import os
from typing import Optional, Dict, Any
from backend.i18n import get_locale_prefix
from backend.services.email_service import get_email_service
from backend.services.template_service import get_template_service
from backend.services.user_service import get_user_service
logger = logging.getLogger(__name__)
@@ -53,30 +51,15 @@ class JobNotificationService:
self.frontend_url = os.getenv("FRONTEND_URL", "https://gen.nomadkaraoke.com")
self.backend_url = os.getenv("BACKEND_URL", "https://api.nomadkaraoke.com")
def _get_user_locale(self, user_email: str) -> str:
"""Look up user's locale preference from Firestore, defaulting to 'en'."""
try:
user_service = get_user_service()
user = user_service.get_user(user_email)
if user and user.locale:
return user.locale
except Exception:
pass # Fall back to English
return "en"
def _build_review_url(self, job_id: str, audio_hash: Optional[str] = None, review_token: Optional[str] = None, locale: str = "en") -> str:
def _build_review_url(self, job_id: str, audio_hash: Optional[str] = None, review_token: Optional[str] = None) -> str:
"""Build the lyrics review URL for a job."""
# Use hash-based routing for static hosting compatibility
return f"{self.frontend_url}{get_locale_prefix(locale)}/app/jobs#/{job_id}/review"
return f"{self.frontend_url}/app/jobs#/{job_id}/review"
def _build_instrumental_url(self, job_id: str, instrumental_token: Optional[str] = None, locale: str = "en") -> str:
def _build_instrumental_url(self, job_id: str, instrumental_token: Optional[str] = None) -> str:
"""Build the instrumental selection URL for a job."""
# Use hash-based routing for static hosting compatibility
return f"{self.frontend_url}{get_locale_prefix(locale)}/app/jobs#/{job_id}/instrumental"
def _build_audio_edit_url(self, job_id: str, locale: str = "en") -> str:
"""Build the audio edit URL for a job."""
return f"{self.frontend_url}{get_locale_prefix(locale)}/app/jobs#/{job_id}/audio-edit"
return f"{self.frontend_url}/app/jobs#/{job_id}/instrumental"
async def send_job_completion_email(
self,
@@ -88,8 +71,6 @@ class JobNotificationService:
youtube_url: Optional[str] = None,
dropbox_url: Optional[str] = None,
brand_code: Optional[str] = None,
is_private: bool = False,
youtube_queued: bool = False,
) -> bool:
"""
Send job completion email to user.
@@ -103,8 +84,6 @@ class JobNotificationService:
youtube_url: YouTube video URL
dropbox_url: Dropbox folder URL
brand_code: Release ID (e.g., "NOMAD-1178")
is_private: If True, omit YouTube section (private/non-published tracks)
youtube_queued: If True, YouTube upload was deferred due to quota
Returns:
True if email was sent successfully
@@ -118,9 +97,6 @@ class JobNotificationService:
return False
try:
# Get user locale for email
user_locale = self._get_user_locale(user_email)
# Render the completion message using template service
message_content = self.template_service.render_job_completion(
name=user_name,
@@ -130,9 +106,6 @@ class JobNotificationService:
title=title,
job_id=job_id,
feedback_url=FEEDBACK_FORM_URL,
is_private=is_private,
youtube_queued=youtube_queued,
locale=user_locale,
)
# Send the email with CC to admin
@@ -143,7 +116,6 @@ class JobNotificationService:
title=title,
brand_code=brand_code,
cc_admin=True,
locale=user_locale,
)
if success:
@@ -157,68 +129,6 @@ class JobNotificationService:
logger.exception(f"Error sending completion email for job {job_id}: {e}")
return False
async def send_youtube_upload_complete_email(
self,
job_id: str,
user_email: str,
artist: Optional[str] = None,
title: Optional[str] = None,
youtube_url: Optional[str] = None,
brand_code: Optional[str] = None,
) -> bool:
"""
Send follow-up email when a deferred YouTube upload completes.
Args:
job_id: Job ID
user_email: User's email address
artist: Artist name
title: Song title
youtube_url: YouTube video URL
brand_code: Release ID
Returns:
True if email was sent successfully
"""
if not ENABLE_AUTO_EMAILS:
logger.info(f"Auto emails disabled, skipping YouTube notification for job {job_id}")
return False
if not user_email:
logger.warning(f"No user email for job {job_id}, skipping YouTube notification")
return False
try:
# Get user locale for email
user_locale = self._get_user_locale(user_email)
message_content = self.template_service.render_youtube_upload_complete(
artist=artist,
title=title,
youtube_url=youtube_url,
locale=user_locale,
)
success = self.email_service.send_youtube_upload_complete(
to_email=user_email,
message_content=message_content,
artist=artist,
title=title,
brand_code=brand_code,
locale=user_locale,
)
if success:
logger.info(f"Sent YouTube upload notification for job {job_id} to {_mask_email(user_email)}")
else:
logger.error(f"Failed to send YouTube upload notification for job {job_id}")
return success
except Exception as e:
logger.exception(f"Error sending YouTube upload notification for job {job_id}: {e}")
return False
async def send_action_reminder_email(
self,
job_id: str,
@@ -240,7 +150,6 @@ class JobNotificationService:
action_type: Type of action needed:
- "lyrics": Combined review (lyrics + instrumental selection) for normal jobs
- "instrumental": Instrumental-only selection for finalise-only jobs
- "audio_edit": Audio editing before processing begins
user_name: User's display name
artist: Artist name
title: Song title
@@ -260,37 +169,22 @@ class JobNotificationService:
return False
try:
# Get user locale for email
user_locale = self._get_user_locale(user_email)
# Render the appropriate template
if action_type == "lyrics":
review_url = self._build_review_url(job_id, audio_hash, review_token, locale=user_locale)
review_url = self._build_review_url(job_id, audio_hash, review_token)
message_content = self.template_service.render_action_needed_lyrics(
name=user_name,
artist=artist,
title=title,
review_url=review_url,
locale=user_locale,
)
elif action_type == "instrumental":
instrumental_url = self._build_instrumental_url(job_id, instrumental_token, locale=user_locale)
instrumental_url = self._build_instrumental_url(job_id, instrumental_token)
message_content = self.template_service.render_action_needed_instrumental(
name=user_name,
artist=artist,
title=title,
instrumental_url=instrumental_url,
locale=user_locale,
)
elif action_type == "audio_edit":
audio_edit_url = self._build_audio_edit_url(job_id, locale=user_locale)
# Reuse the lyrics template but with the audio edit URL and different subject
message_content = self.template_service.render_action_needed_lyrics(
name=user_name,
artist=artist,
title=title,
review_url=audio_edit_url,
locale=user_locale,
)
else:
logger.error(f"Unknown action type: {action_type}")
@@ -303,7 +197,6 @@ class JobNotificationService:
action_type=action_type,
artist=artist,
title=title,
locale=user_locale,
)
if success:
@@ -325,8 +218,6 @@ class JobNotificationService:
title: Optional[str] = None,
youtube_url: Optional[str] = None,
dropbox_url: Optional[str] = None,
is_private: bool = False,
youtube_queued: bool = False,
) -> str:
"""
Get the rendered completion message for a job (for admin copy functionality).
@@ -338,8 +229,6 @@ class JobNotificationService:
title: Song title
youtube_url: YouTube video URL
dropbox_url: Dropbox folder URL
is_private: If True, omit YouTube section (private/non-published tracks)
youtube_queued: If True, YouTube upload was deferred
Returns:
Rendered message content as plain text
@@ -352,8 +241,6 @@ class JobNotificationService:
title=title,
job_id=job_id,
feedback_url=FEEDBACK_FORM_URL,
is_private=is_private,
youtube_queued=youtube_queued,
)
-203
View File
@@ -1,203 +0,0 @@
"""
KaraokeNerds community version detection service.
Scrapes karaokenerds.com to check if a song already has community-approved
karaoke versions available on YouTube. Ported from kjbox/kj-controller/karaoke_nerds.py.
"""
import logging
import re
import time
from typing import Any
from urllib.parse import urlencode
import httpx
from bs4 import BeautifulSoup
logger = logging.getLogger(__name__)
SEARCH_URL = "https://karaokenerds.com/Search"
REQUEST_TIMEOUT = 8
USER_AGENT = "NomadKaraokeGen/1.0"
# In-memory TTL cache: { cache_key: (expiry_timestamp, data) }
_cache: dict[str, tuple[float, Any]] = {}
CACHE_TTL_SECONDS = 3600 # 1 hour
def _cache_get(key: str) -> Any | None:
"""Get a value from cache if it exists and hasn't expired."""
if key in _cache:
expiry, data = _cache[key]
if time.monotonic() < expiry:
return data
del _cache[key]
return None
def _cache_set(key: str, data: Any, ttl: float = CACHE_TTL_SECONDS) -> None:
"""Store a value in cache with TTL."""
_cache[key] = (time.monotonic() + ttl, data)
def _clean_youtube_url(url: str) -> str:
"""Strip playlist params from YouTube URLs, keep just the video URL."""
return re.sub(r"&list=[^&]*", "", url)
def _parse_single_track(li) -> dict | None:
"""Parse a single track <li> element from karaokenerds search results."""
# Brand name: first <a> in the li
brand_link = li.find("a")
brand_name = brand_link.get_text(strip=True) if brand_link else ""
# Brand code: text inside .badge span
badge = li.find("span", class_="badge")
brand_code = ""
if badge:
brand_code = badge.get_text(strip=True)
# YouTube URL: link containing youtube.com
youtube_url = None
for a in li.find_all("a", href=True):
href = a["href"]
if "youtube.com" in href:
youtube_url = _clean_youtube_url(href)
break
# Community: presence of img.check
is_community = bool(li.find("img", class_="check"))
if not youtube_url:
return None
return {
"brand_name": brand_name,
"brand_code": brand_code,
"youtube_url": youtube_url,
"is_community": is_community,
}
def _parse_tracks(details_row) -> list[dict]:
"""Parse track list items from a details row."""
tracks = []
for li in details_row.find_all("li", class_="track"):
track = _parse_single_track(li)
if track:
tracks.append(track)
return tracks
def parse_results(html: str) -> list[dict]:
"""Parse karaokenerds.com search results HTML into structured data."""
soup = BeautifulSoup(html, "html.parser")
table = soup.find("table")
if not table:
return []
tbody = table.find("tbody")
if not tbody:
return []
songs = []
rows = tbody.find_all("tr", recursive=False)
i = 0
while i < len(rows):
row = rows[i]
# Song rows have class "group"
if "group" not in row.get("class", []):
i += 1
continue
# Extract title and artist from the song row
cells = row.find_all("td")
if len(cells) < 3:
i += 1
continue
title_link = cells[0].find("a")
artist_link = cells[1].find("a")
title = title_link.get_text(strip=True) if title_link else ""
artist = artist_link.get_text(strip=True) if artist_link else ""
# The next row should be the details row with tracks
tracks = []
if i + 1 < len(rows):
details_row = rows[i + 1]
if "details" in details_row.get("class", []):
tracks = _parse_tracks(details_row)
i += 2
else:
i += 1
else:
i += 1
if title:
songs.append({
"title": title,
"artist": artist,
"tracks": tracks,
})
return songs
async def check_community_versions(artist: str, title: str) -> dict:
"""
Check if a song has community-approved karaoke versions on karaokenerds.
Returns a dict with:
- has_community: bool
- songs: list of matched songs with community tracks
- best_youtube_url: URL of the top community version (if any)
"""
query = f"{artist} {title}"
cache_key = f"community:{query.lower().strip()}"
cached = _cache_get(cache_key)
if cached is not None:
return cached
params = urlencode({"query": query, "webFilter": "OnlyWeb"})
url = f"{SEARCH_URL}?{params}"
try:
async with httpx.AsyncClient(timeout=REQUEST_TIMEOUT) as client:
resp = await client.get(
url,
headers={"User-Agent": USER_AGENT},
follow_redirects=True,
)
resp.raise_for_status()
except Exception as e:
logger.warning(f"KaraokeNerds search failed for '{query}': {e}")
result = {"has_community": False, "songs": [], "best_youtube_url": None}
_cache_set(cache_key, result)
return result
songs = parse_results(resp.text)
# Filter to only songs that have community tracks with YouTube URLs
community_songs = []
best_youtube_url = None
for song in songs:
community_tracks = [t for t in song["tracks"] if t["is_community"]]
if community_tracks:
community_songs.append({
"title": song["title"],
"artist": song["artist"],
"community_tracks": community_tracks,
})
if best_youtube_url is None:
best_youtube_url = community_tracks[0]["youtube_url"]
result = {
"has_community": len(community_songs) > 0,
"songs": community_songs,
"best_youtube_url": best_youtube_url,
}
_cache_set(cache_key, result)
return result
+1 -4
View File
@@ -545,10 +545,7 @@ class LocalEncodingService:
)
output_files["with_vocals_mp4"] = config.output_with_vocals_mp4
else:
self.logger.info("[Step 2/6] Skipped - video already MP4, copying as-is")
import shutil
shutil.copy2(config.karaoke_video, config.output_with_vocals_mp4)
output_files["with_vocals_mp4"] = config.output_with_vocals_mp4
self.logger.info("[Step 2/6] Skipped - video already MP4")
# Step 3: Encode lossless 4K MP4
if config.output_lossless_4k_mp4:
-7
View File
@@ -57,8 +57,6 @@ class PackagingService:
title: str,
output_mp3_path: Optional[str] = None,
output_cdg_path: Optional[str] = None,
lrc_has_countdown_padding: bool = False,
countdown_padding_seconds: float = 3.0,
) -> Tuple[str, Optional[str], Optional[str]]:
"""
Create a CDG package (ZIP containing CDG and MP3 files).
@@ -71,9 +69,6 @@ class PackagingService:
title: Song title for metadata
output_mp3_path: Optional path for the extracted MP3 file
output_cdg_path: Optional path for the extracted CDG file
lrc_has_countdown_padding: If True, LRC contains countdown-shifted timestamps
that need adjustment since CDG uses instrumental audio without padding
countdown_padding_seconds: Duration of countdown padding to compensate for
Returns:
Tuple of (zip_path, mp3_path, cdg_path)
@@ -136,8 +131,6 @@ class PackagingService:
title=title,
artist=artist,
cdg_styles=self.cdg_styles,
lrc_has_countdown_padding=lrc_has_countdown_padding,
countdown_padding_seconds=countdown_padding_seconds,
)
# Rename ZIP to expected output path if different
+17 -56
View File
@@ -12,7 +12,6 @@ from pywebpush import webpush, WebPushException
from backend.config import get_settings
from backend.models.user import User, PushSubscription
from backend.i18n import t
logger = logging.getLogger(__name__)
@@ -86,14 +85,10 @@ class PushNotificationService:
title: str,
body: str,
url: str = "/app/",
tag: Optional[str] = None,
tenant_id: Optional[str] = None
tag: Optional[str] = None
) -> int:
"""
Send a push notification to a user's subscribed devices for a specific tenant.
Only sends to subscriptions matching the given tenant_id to prevent
cross-tenant notification leakage.
Send a push notification to all of a user's subscribed devices.
Args:
user_email: The user's email address
@@ -101,8 +96,6 @@ class PushNotificationService:
body: Notification body text
url: URL to open when notification is clicked
tag: Optional tag for notification grouping (replaces notifications with same tag)
tenant_id: Tenant scope - only send to subscriptions for this tenant
(None or "" = consumer/default portal)
Returns:
Number of notifications successfully sent
@@ -118,30 +111,7 @@ class PushNotificationService:
return 0
user_data = user_doc.to_dict()
all_subscriptions = user_data.get("push_subscriptions", [])
# Filter subscriptions to only those matching the target tenant
# Normalize: None and "" both mean "consumer/default portal"
target_tenant = tenant_id or None
subscriptions = [
sub for sub in all_subscriptions
if (sub.get("tenant_id") or None) == target_tenant
]
if not subscriptions and all_subscriptions:
# Migration path: if no subscriptions match and none have tenant_id set,
# it means they're legacy subscriptions created before tenant scoping.
# Send to legacy subscriptions (those without tenant_id) only if
# the target is the consumer portal (no tenant).
has_any_tenant_scoped = any(sub.get("tenant_id") for sub in all_subscriptions)
if not has_any_tenant_scoped and target_tenant is None:
subscriptions = all_subscriptions
logger.debug(f"Using legacy unscoped subscriptions for {user_email}")
else:
logger.debug(
f"No matching subscriptions for tenant={target_tenant} "
f"(user has {len(all_subscriptions)} total subscriptions)"
)
subscriptions = user_data.get("push_subscriptions", [])
if not subscriptions:
logger.debug(f"User {user_email} has no push subscriptions")
@@ -226,7 +196,7 @@ class PushNotificationService:
Send notification when a job enters a blocking state requiring user action.
Args:
job: Job dictionary with job_id, user_email, artist, title, tenant_id
job: Job dictionary with job_id, user_email, artist, title
action_type: Type of action needed ("lyrics" or "instrumental")
Returns:
@@ -239,18 +209,16 @@ class PushNotificationService:
job_id = job.get("job_id", "unknown")
artist = job.get("artist", "Unknown Artist")
title = job.get("title", "Unknown Title")
tenant_id = job.get("tenant_id") or None
locale = job.get("locale", "en")
if action_type == "lyrics":
notif_title = t(locale, "pushNotifications.reviewLyricsTitle")
notif_body = t(locale, "pushNotifications.reviewLyricsBody", title=title, artist=artist)
url = f"/app/jobs#/{job_id}/review"
notif_title = "Review Lyrics"
notif_body = f'"{title}" by {artist} needs lyrics review'
url = f"/review/{job_id}"
tag = f"lyrics-{job_id}"
else: # instrumental
notif_title = t(locale, "pushNotifications.selectInstrumentalTitle")
notif_body = t(locale, "pushNotifications.selectInstrumentalBody", title=title, artist=artist)
url = f"/app/jobs#/{job_id}/instrumental"
notif_title = "Select Instrumental"
notif_body = f'"{title}" by {artist} needs instrumental selection'
url = f"/instrumental/{job_id}"
tag = f"instrumental-{job_id}"
return await self.send_push(
@@ -258,8 +226,7 @@ class PushNotificationService:
title=notif_title,
body=notif_body,
url=url,
tag=tag,
tenant_id=tenant_id,
tag=tag
)
async def send_completion_notification(self, job: dict) -> int:
@@ -267,7 +234,7 @@ class PushNotificationService:
Send notification when a job completes successfully.
Args:
job: Job dictionary with job_id, user_email, artist, title, tenant_id
job: Job dictionary with job_id, user_email, artist, title
Returns:
Number of notifications sent
@@ -279,16 +246,13 @@ class PushNotificationService:
job_id = job.get("job_id", "unknown")
artist = job.get("artist", "Unknown Artist")
title = job.get("title", "Unknown Title")
tenant_id = job.get("tenant_id") or None
locale = job.get("locale", "en")
return await self.send_push(
user_email=user_email,
title=t(locale, "pushNotifications.videoReadyTitle"),
body=t(locale, "pushNotifications.videoReadyBody", title=title, artist=artist),
title="Video Ready!",
body=f'Your karaoke video for "{title}" by {artist} is ready to download',
url=f"/app/?job={job_id}",
tag=f"complete-{job_id}",
tenant_id=tenant_id,
tag=f"complete-{job_id}"
)
async def add_subscription(
@@ -296,11 +260,10 @@ class PushNotificationService:
user_email: str,
endpoint: str,
keys: dict,
device_name: Optional[str] = None,
tenant_id: Optional[str] = None
device_name: Optional[str] = None
) -> bool:
"""
Add a push subscription for a user, scoped to a tenant.
Add a push subscription for a user.
Enforces max subscriptions per user - oldest removed if limit exceeded.
@@ -309,7 +272,6 @@ class PushNotificationService:
endpoint: Push service endpoint URL
keys: Encryption keys (p256dh, auth)
device_name: Optional device identifier
tenant_id: Tenant scope (None = consumer/default portal)
Returns:
True if subscription was added successfully
@@ -336,7 +298,6 @@ class PushNotificationService:
"endpoint": endpoint,
"keys": keys,
"device_name": device_name,
"tenant_id": tenant_id or None,
"created_at": datetime.now(timezone.utc).isoformat(),
"last_used_at": None
}
+641
View File
@@ -0,0 +1,641 @@
"""
Rate limiting service for job creation, YouTube uploads, and beta enrollment.
Uses Firestore for distributed rate limit tracking with date-based document IDs
that automatically reset at UTC midnight.
"""
import hashlib
import logging
from datetime import datetime, timezone
from typing import Tuple, Optional, List, Dict, Any
from google.cloud import firestore
from backend.config import settings
logger = logging.getLogger(__name__)
# Firestore collection names
RATE_LIMITS_COLLECTION = "rate_limits"
BLOCKLISTS_COLLECTION = "blocklists"
OVERRIDES_COLLECTION = "overrides"
def _get_today_date_str() -> str:
"""Get today's date in UTC as YYYY-MM-DD string."""
return datetime.now(timezone.utc).strftime("%Y-%m-%d")
def _seconds_until_midnight_utc() -> int:
"""Calculate seconds until UTC midnight (when rate limits reset)."""
now = datetime.now(timezone.utc)
midnight = now.replace(hour=0, minute=0, second=0, microsecond=0)
# Add 1 day to get next midnight
from datetime import timedelta
next_midnight = midnight + timedelta(days=1)
return int((next_midnight - now).total_seconds())
def _hash_ip(ip_address: str) -> str:
"""Hash an IP address for storage (privacy-preserving)."""
return hashlib.sha256(ip_address.encode()).hexdigest()[:16]
class RateLimitService:
"""
Service for managing rate limits across the platform.
Rate limits use Firestore for distributed tracking with automatic
reset at UTC midnight via date-based document IDs.
"""
def __init__(self, db: Optional[firestore.Client] = None):
"""Initialize rate limit service with Firestore client."""
if db is None:
self.db = firestore.Client(project=settings.google_cloud_project)
else:
self.db = db
self._blocklist_cache: Optional[Dict[str, Any]] = None
self._blocklist_cache_time: Optional[datetime] = None
self._cache_ttl_seconds = 300 # 5 minute cache
# -------------------------------------------------------------------------
# Job Rate Limiting
# -------------------------------------------------------------------------
def check_user_job_limit(self, user_email: str, is_admin: bool = False) -> Tuple[bool, int, str]:
"""
Check if a user can create a new job.
Args:
user_email: User's email address
is_admin: Whether the user is an admin (admins bypass limits)
Returns:
Tuple of (allowed, remaining, message)
- allowed: True if user can create job
- remaining: Number of jobs remaining today
- message: Human-readable status message
"""
if not settings.enable_rate_limiting:
return True, -1, "Rate limiting disabled"
if is_admin:
return True, -1, "Admin users bypass rate limits"
limit = settings.rate_limit_jobs_per_day
if limit == 0:
return True, -1, "No job limit configured"
# Check for user override
override = self.get_user_override(user_email)
if override and override.get("bypass_job_limit"):
return True, -1, "User has rate limit bypass"
if override and override.get("custom_daily_job_limit"):
limit = override["custom_daily_job_limit"]
# Get current count
current_count = self.get_user_job_count_today(user_email)
remaining = max(0, limit - current_count)
if current_count >= limit:
seconds = _seconds_until_midnight_utc()
logger.warning(
f"Rate limit exceeded for {user_email}: {current_count}/{limit} jobs today"
)
return (
False,
0,
f"Daily job limit reached ({limit} jobs per day). Resets in {seconds // 3600}h {(seconds % 3600) // 60}m."
)
return True, remaining, f"{remaining} jobs remaining today"
def get_user_job_count_today(self, user_email: str) -> int:
"""Get the number of jobs created by a user today."""
date_str = _get_today_date_str()
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
f"user_jobs_{user_email}_{date_str}"
)
doc = doc_ref.get()
if not doc.exists:
return 0
return doc.to_dict().get("count", 0)
def record_job_creation(self, user_email: str, job_id: str) -> None:
"""
Record a job creation for rate limiting.
Uses Firestore transactions for atomic increment.
"""
if not settings.enable_rate_limiting:
return
date_str = _get_today_date_str()
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
f"user_jobs_{user_email}_{date_str}"
)
@firestore.transactional
def update_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if doc.exists:
data = doc.to_dict()
count = data.get("count", 0) + 1
job_ids = data.get("job_ids", [])
job_ids.append(job_id)
else:
count = 1
job_ids = [job_id]
transaction.set(doc_ref, {
"user_email": user_email,
"date": date_str,
"count": count,
"job_ids": job_ids,
"updated_at": datetime.now(timezone.utc),
})
transaction = self.db.transaction()
update_in_transaction(transaction, doc_ref)
logger.info(f"Recorded job {job_id} for user {user_email} rate limiting")
# -------------------------------------------------------------------------
# YouTube Upload Rate Limiting
# -------------------------------------------------------------------------
def check_youtube_upload_limit(self) -> Tuple[bool, int, str]:
"""
Check if the system can perform a YouTube upload.
YouTube uploads are limited system-wide due to API quota constraints.
Returns:
Tuple of (allowed, remaining, message)
"""
if not settings.enable_rate_limiting:
return True, -1, "Rate limiting disabled"
limit = settings.rate_limit_youtube_uploads_per_day
if limit == 0:
return True, -1, "No YouTube upload limit configured"
current_count = self.get_youtube_uploads_today()
remaining = max(0, limit - current_count)
if current_count >= limit:
seconds = _seconds_until_midnight_utc()
logger.warning(
f"YouTube upload limit exceeded: {current_count}/{limit} uploads today"
)
return (
False,
0,
f"Daily YouTube upload limit reached ({limit} uploads per day). Resets in {seconds // 3600}h {(seconds % 3600) // 60}m."
)
return True, remaining, f"{remaining} YouTube uploads remaining today"
def get_youtube_uploads_today(self) -> int:
"""Get the number of YouTube uploads performed today (system-wide)."""
date_str = _get_today_date_str()
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
f"youtube_uploads_{date_str}"
)
doc = doc_ref.get()
if not doc.exists:
return 0
return doc.to_dict().get("count", 0)
def record_youtube_upload(self, job_id: str, user_email: str) -> None:
"""
Record a YouTube upload for rate limiting.
Uses Firestore transactions for atomic increment.
"""
if not settings.enable_rate_limiting:
return
date_str = _get_today_date_str()
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
f"youtube_uploads_{date_str}"
)
@firestore.transactional
def update_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if doc.exists:
data = doc.to_dict()
count = data.get("count", 0) + 1
uploads = data.get("uploads", [])
else:
count = 1
uploads = []
uploads.append({
"job_id": job_id,
"user_email": user_email,
"timestamp": datetime.now(timezone.utc),
})
transaction.set(doc_ref, {
"date": date_str,
"count": count,
"uploads": uploads,
"updated_at": datetime.now(timezone.utc),
})
transaction = self.db.transaction()
update_in_transaction(transaction, doc_ref)
logger.info(f"Recorded YouTube upload for job {job_id}")
# -------------------------------------------------------------------------
# Beta Enrollment IP Rate Limiting
# -------------------------------------------------------------------------
def check_beta_ip_limit(self, ip_address: str) -> Tuple[bool, int, str]:
"""
Check if an IP address can enroll in the beta program.
Args:
ip_address: Client IP address
Returns:
Tuple of (allowed, remaining, message)
"""
if not settings.enable_rate_limiting:
return True, -1, "Rate limiting disabled"
limit = settings.rate_limit_beta_ip_per_day
if limit == 0:
return True, -1, "No beta IP limit configured"
# Check today's enrollment count for this IP
ip_hash = _hash_ip(ip_address)
date_str = _get_today_date_str()
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
f"beta_ip_{ip_hash}_{date_str}"
)
doc = doc_ref.get()
current_count = 0
if doc.exists:
current_count = doc.to_dict().get("count", 0)
remaining = max(0, limit - current_count)
if current_count >= limit:
seconds = _seconds_until_midnight_utc()
logger.warning(
f"Beta enrollment IP limit exceeded for {ip_hash}: {current_count}/{limit} enrollments today"
)
return (
False,
0,
f"Too many beta enrollments from this network today. Please try again tomorrow."
)
return True, remaining, f"{remaining} beta enrollments remaining from this IP today"
def record_beta_enrollment(self, ip_address: str, email: str) -> None:
"""
Record a beta enrollment for IP rate limiting.
Uses Firestore transactions for atomic increment.
"""
if not settings.enable_rate_limiting:
return
ip_hash = _hash_ip(ip_address)
date_str = _get_today_date_str()
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
f"beta_ip_{ip_hash}_{date_str}"
)
@firestore.transactional
def update_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if doc.exists:
data = doc.to_dict()
count = data.get("count", 0) + 1
enrollments = data.get("enrollments", [])
else:
count = 1
enrollments = []
enrollments.append({
"email": email,
"timestamp": datetime.now(timezone.utc),
})
transaction.set(doc_ref, {
"ip_hash": ip_hash,
"date": date_str,
"count": count,
"enrollments": enrollments,
"updated_at": datetime.now(timezone.utc),
})
transaction = self.db.transaction()
update_in_transaction(transaction, doc_ref)
logger.info(f"Recorded beta enrollment from IP {ip_hash} for {email}")
# -------------------------------------------------------------------------
# User Overrides (Whitelist)
# -------------------------------------------------------------------------
def get_user_override(self, user_email: str) -> Optional[Dict[str, Any]]:
"""
Get rate limit override settings for a user.
Returns:
Override settings dict or None if no override exists
"""
doc_ref = self.db.collection(OVERRIDES_COLLECTION).document(user_email.lower())
doc = doc_ref.get()
if not doc.exists:
return None
return doc.to_dict()
def set_user_override(
self,
user_email: str,
bypass_job_limit: bool = False,
custom_daily_job_limit: Optional[int] = None,
reason: str = "",
admin_email: str = ""
) -> None:
"""
Set rate limit override for a user.
Args:
user_email: User to override
bypass_job_limit: If True, user bypasses all job limits
custom_daily_job_limit: Custom limit (None = use default)
reason: Reason for override
admin_email: Admin who set the override
"""
doc_ref = self.db.collection(OVERRIDES_COLLECTION).document(user_email.lower())
doc_ref.set({
"email": user_email.lower(),
"bypass_job_limit": bypass_job_limit,
"custom_daily_job_limit": custom_daily_job_limit,
"reason": reason,
"created_by": admin_email,
"created_at": datetime.now(timezone.utc),
})
logger.info(f"Set rate limit override for {user_email} by {admin_email}")
def remove_user_override(self, user_email: str, admin_email: str = "") -> bool:
"""Remove rate limit override for a user."""
doc_ref = self.db.collection(OVERRIDES_COLLECTION).document(user_email.lower())
doc = doc_ref.get()
if not doc.exists:
return False
doc_ref.delete()
logger.info(f"Removed rate limit override for {user_email} by {admin_email}")
return True
def list_user_overrides(self) -> List[Dict[str, Any]]:
"""List all user rate limit overrides."""
docs = self.db.collection(OVERRIDES_COLLECTION).stream()
return [doc.to_dict() for doc in docs]
def get_all_overrides(self) -> Dict[str, Dict[str, Any]]:
"""Get all user rate limit overrides as a dict keyed by email."""
docs = self.db.collection(OVERRIDES_COLLECTION).stream()
return {doc.id: doc.to_dict() for doc in docs}
# -------------------------------------------------------------------------
# Blocklist Management
# -------------------------------------------------------------------------
def _load_blocklist(self, force_refresh: bool = False) -> Dict[str, Any]:
"""
Load blocklist from Firestore with caching.
Returns:
Dict with keys: disposable_domains, blocked_emails, blocked_ips
"""
now = datetime.now(timezone.utc)
# Check cache
if not force_refresh and self._blocklist_cache and self._blocklist_cache_time:
cache_age = (now - self._blocklist_cache_time).total_seconds()
if cache_age < self._cache_ttl_seconds:
return self._blocklist_cache
# Load from Firestore
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc = doc_ref.get()
if not doc.exists:
# Initialize with empty blocklist
self._blocklist_cache = {
"disposable_domains": [],
"blocked_emails": [],
"blocked_ips": [],
}
else:
self._blocklist_cache = doc.to_dict()
self._blocklist_cache_time = now
return self._blocklist_cache
def is_disposable_domain(self, domain: str) -> bool:
"""Check if a domain is in the disposable domains blocklist."""
blocklist = self._load_blocklist()
return domain.lower() in [d.lower() for d in blocklist.get("disposable_domains", [])]
def is_blocked_email(self, email: str) -> bool:
"""Check if an email is explicitly blocked."""
blocklist = self._load_blocklist()
return email.lower() in [e.lower() for e in blocklist.get("blocked_emails", [])]
def is_blocked_ip(self, ip_address: str) -> bool:
"""Check if an IP address is blocked."""
blocklist = self._load_blocklist()
return ip_address in blocklist.get("blocked_ips", [])
def get_blocklist(self) -> Dict[str, Any]:
"""Get all blocklist data."""
return self._load_blocklist(force_refresh=True)
def add_disposable_domain(self, domain: str, admin_email: str) -> None:
"""Add a domain to the disposable domains blocklist."""
blocklist = self._load_blocklist(force_refresh=True)
domains = set(blocklist.get("disposable_domains", []))
domains.add(domain.lower())
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc_ref.set({
**blocklist,
"disposable_domains": list(domains),
"updated_at": datetime.now(timezone.utc),
"updated_by": admin_email,
}, merge=True)
self._blocklist_cache = None # Invalidate cache
logger.info(f"Added disposable domain {domain} by {admin_email}")
def remove_disposable_domain(self, domain: str) -> bool:
"""Remove a domain from the disposable domains blocklist."""
blocklist = self._load_blocklist(force_refresh=True)
domains = set(blocklist.get("disposable_domains", []))
if domain.lower() not in [d.lower() for d in domains]:
return False
domains = {d for d in domains if d.lower() != domain.lower()}
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc_ref.set({
**blocklist,
"disposable_domains": list(domains),
"updated_at": datetime.now(timezone.utc),
}, merge=True)
self._blocklist_cache = None
logger.info(f"Removed disposable domain {domain}")
return True
def add_blocked_email(self, email: str, admin_email: str) -> None:
"""Add an email to the blocked emails list."""
blocklist = self._load_blocklist(force_refresh=True)
emails = set(blocklist.get("blocked_emails", []))
emails.add(email.lower())
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc_ref.set({
**blocklist,
"blocked_emails": list(emails),
"updated_at": datetime.now(timezone.utc),
"updated_by": admin_email,
}, merge=True)
self._blocklist_cache = None
logger.info(f"Added blocked email {email} by {admin_email}")
def remove_blocked_email(self, email: str) -> bool:
"""Remove an email from the blocked emails list."""
blocklist = self._load_blocklist(force_refresh=True)
emails = set(blocklist.get("blocked_emails", []))
if email.lower() not in [e.lower() for e in emails]:
return False
emails = {e for e in emails if e.lower() != email.lower()}
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc_ref.set({
**blocklist,
"blocked_emails": list(emails),
"updated_at": datetime.now(timezone.utc),
}, merge=True)
self._blocklist_cache = None
logger.info(f"Removed blocked email {email}")
return True
def add_blocked_ip(self, ip_address: str, admin_email: str) -> None:
"""Add an IP address to the blocked IPs list."""
blocklist = self._load_blocklist(force_refresh=True)
ips = set(blocklist.get("blocked_ips", []))
ips.add(ip_address)
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc_ref.set({
**blocklist,
"blocked_ips": list(ips),
"updated_at": datetime.now(timezone.utc),
"updated_by": admin_email,
}, merge=True)
self._blocklist_cache = None
logger.info(f"Added blocked IP {ip_address} by {admin_email}")
def remove_blocked_ip(self, ip_address: str) -> bool:
"""Remove an IP address from the blocked IPs list."""
blocklist = self._load_blocklist(force_refresh=True)
ips = set(blocklist.get("blocked_ips", []))
if ip_address not in ips:
return False
ips.discard(ip_address)
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc_ref.set({
**blocklist,
"blocked_ips": list(ips),
"updated_at": datetime.now(timezone.utc),
}, merge=True)
self._blocklist_cache = None
logger.info(f"Removed blocked IP {ip_address}")
return True
# -------------------------------------------------------------------------
# Stats
# -------------------------------------------------------------------------
def get_rate_limit_stats(self) -> Dict[str, Any]:
"""
Get current rate limit statistics.
Returns:
Dict with current usage stats
"""
date_str = _get_today_date_str()
# Get YouTube upload count
youtube_count = self.get_youtube_uploads_today()
# Count unique users with jobs today
users_with_jobs = set()
total_jobs_today = 0
# Query all user job documents for today
docs = self.db.collection(RATE_LIMITS_COLLECTION).where(
"date", "==", date_str
).stream()
for doc in docs:
data = doc.to_dict()
if data.get("user_email"):
users_with_jobs.add(data["user_email"])
total_jobs_today += data.get("count", 0)
return {
"date": date_str,
"youtube_uploads_today": youtube_count,
"youtube_uploads_limit": settings.rate_limit_youtube_uploads_per_day,
"total_jobs_today": total_jobs_today,
"users_with_jobs_today": len(users_with_jobs),
"job_limit_per_user": settings.rate_limit_jobs_per_day,
"beta_ip_limit_per_day": settings.rate_limit_beta_ip_per_day,
"rate_limiting_enabled": settings.enable_rate_limiting,
"seconds_until_reset": _seconds_until_midnight_utc(),
}
# Singleton instance
_rate_limit_service: Optional[RateLimitService] = None
def get_rate_limit_service() -> RateLimitService:
"""Get the singleton RateLimitService instance."""
global _rate_limit_service
if _rate_limit_service is None:
_rate_limit_service = RateLimitService()
return _rate_limit_service
-526
View File
@@ -1,526 +0,0 @@
"""
Referral service for link management, earnings tracking, and payouts.
Handles:
- Referral link creation (generated and vanity codes)
- Link lookup and validation
- Click tracking
- Admin listing
"""
import logging
import re
import secrets
import string
from datetime import datetime, timedelta
from typing import List, Optional, Tuple
from google.cloud import firestore
from backend.models.referral import (
ReferralLink,
ReferralLinkStats,
ReferralEarning,
ReferralPayout,
)
logger = logging.getLogger(__name__)
# ============================================================================
# Constants
# ============================================================================
REFERRAL_LINKS_COLLECTION = "referral_links"
REFERRAL_EARNINGS_COLLECTION = "referral_earnings"
REFERRAL_PAYOUTS_COLLECTION = "referral_payouts"
RESERVED_CODES = frozenset({
"admin", "app", "api", "r", "pricing", "order", "payment",
"login", "auth", "webhook", "webhooks", "internal", "health",
"static", "assets", "public", "private",
})
PAYOUT_THRESHOLD_CENTS = 2000
VANITY_CODE_PATTERN = re.compile(r"^[a-z0-9][a-z0-9\-]{1,28}[a-z0-9]$")
# Characters for generated codes
_CODE_ALPHABET = string.ascii_lowercase + string.digits
# ============================================================================
# Service
# ============================================================================
class ReferralService:
"""Service for referral link management and earnings."""
def __init__(self, db=None, stripe_service=None):
if db is None:
db = firestore.Client()
self.db = db
self.stripe_service = stripe_service
# ========================================================================
# Code Generation & Validation
# ========================================================================
def _generate_code(self) -> str:
"""Generate an 8-character lowercase alphanumeric referral code."""
return "".join(secrets.choice(_CODE_ALPHABET) for _ in range(8))
def _validate_vanity_code(self, code: str) -> Tuple[bool, str]:
"""Validate a vanity code against reserved words and pattern rules.
Returns (is_valid, message).
"""
if code.lower() in RESERVED_CODES:
return False, f"'{code}' is a reserved code"
if not VANITY_CODE_PATTERN.match(code.lower()):
return False, (
"Code must be 3-30 characters, start/end with alphanumeric, "
"and contain only lowercase letters, digits, and hyphens"
)
return True, "Valid"
# ========================================================================
# Link CRUD
# ========================================================================
def get_link_by_code(self, code: str) -> Optional[ReferralLink]:
"""Get a referral link by its code. Returns None if not found or disabled."""
doc = self.db.collection(REFERRAL_LINKS_COLLECTION).document(code).get()
if not doc.exists:
return None
data = doc.to_dict()
if not data.get("enabled", True):
return None
return ReferralLink(**data)
def get_or_create_link(self, owner_email: str) -> ReferralLink:
"""Get existing referral link for owner, or create a new one.
Retries up to 10 times if generated code collides with existing doc.
"""
# Check for existing link
existing = (
self.db.collection(REFERRAL_LINKS_COLLECTION)
.where("owner_email", "==", owner_email)
.limit(1)
.stream()
)
for doc in existing:
return ReferralLink(**doc.to_dict())
# Create new link with retry for code collision
for attempt in range(10):
code = self._generate_code()
doc_ref = self.db.collection(REFERRAL_LINKS_COLLECTION).document(code)
doc_snapshot = doc_ref.get()
if doc_snapshot.exists:
logger.warning("Code collision on attempt %d: %s", attempt + 1, code)
continue
now = datetime.utcnow()
link = ReferralLink(
code=code,
owner_email=owner_email,
created_at=now,
updated_at=now,
)
doc_ref.set(link.model_dump())
return link
raise RuntimeError("Failed to generate unique referral code after 10 attempts")
def create_vanity_link(
self,
code: str,
owner_email: str,
display_name: Optional[str] = None,
custom_message: Optional[str] = None,
discount_percent: int = 10,
kickback_percent: int = 20,
discount_duration_days: int = 30,
earning_duration_days: int = 365,
) -> Tuple[bool, Optional[ReferralLink], str]:
"""Create a vanity referral link.
Returns (success, link_or_none, message).
"""
valid, msg = self._validate_vanity_code(code)
if not valid:
return False, None, msg
normalized = code.lower()
doc_ref = self.db.collection(REFERRAL_LINKS_COLLECTION).document(normalized)
doc_snapshot = doc_ref.get()
if doc_snapshot.exists:
return False, None, f"Code '{normalized}' is already taken"
now = datetime.utcnow()
link = ReferralLink(
code=normalized,
owner_email=owner_email,
display_name=display_name,
custom_message=custom_message,
discount_percent=discount_percent,
kickback_percent=kickback_percent,
discount_duration_days=discount_duration_days,
earning_duration_days=earning_duration_days,
is_vanity=True,
created_at=now,
updated_at=now,
)
doc_ref.set(link.model_dump())
return True, link, "Vanity link created"
def update_link(self, code: str, **updates) -> Tuple[bool, str]:
"""Update fields on an existing referral link.
Returns (success, message).
"""
doc_ref = self.db.collection(REFERRAL_LINKS_COLLECTION).document(code)
doc_snapshot = doc_ref.get()
if not doc_snapshot.exists:
return False, f"Link '{code}' not found"
updates["updated_at"] = datetime.utcnow()
doc_ref.update(updates)
return True, "Link updated"
def increment_clicks(self, code: str) -> None:
"""Increment the click counter for a referral link."""
doc_ref = self.db.collection(REFERRAL_LINKS_COLLECTION).document(code)
doc_ref.update({"stats.clicks": firestore.Increment(1)})
# ========================================================================
# Attribution
# ========================================================================
def attribute_referral(self, referred_email: str, referral_code: str) -> Tuple[bool, str]:
"""Attribute a referral to a user. Called during first login/signup."""
referred_email = referred_email.lower()
referral_code = referral_code.lower()
link = self.get_link_by_code(referral_code)
if not link:
return False, "Invalid or disabled referral code"
if link.owner_email == referred_email:
return False, "Cannot use your own referral code"
# Increment signup count
self.db.collection(REFERRAL_LINKS_COLLECTION).document(referral_code).update({
"stats.signups": firestore.Increment(1),
})
return True, "Referral attributed"
def get_attribution_data(self, referral_code: str) -> Optional[dict]:
"""Get the data to set on user doc for referral attribution."""
link = self.get_link_by_code(referral_code)
if not link:
return None
now = datetime.utcnow()
return {
"referred_by_code": referral_code.lower(),
"referred_at": now,
"referral_discount_expires_at": now + timedelta(days=link.discount_duration_days),
}
def get_or_create_stripe_coupon(self, discount_percent: int) -> Optional[str]:
if not self.stripe_service:
return None
return self.stripe_service.get_or_create_referral_coupon(discount_percent)
def should_apply_discount(self, user_data: dict) -> bool:
if not user_data.get("referred_by_code"):
return False
expires_at = user_data.get("referral_discount_expires_at")
if not expires_at:
return False
if isinstance(expires_at, datetime):
now = datetime.utcnow()
# Firestore returns timezone-aware datetimes; ensure comparison is compatible
if expires_at.tzinfo is not None:
now = now.replace(tzinfo=expires_at.tzinfo)
return expires_at > now
return False
def get_discount_for_checkout(self, user_email: str) -> Optional[dict]:
from backend.services.user_service import get_user_service
user_service = get_user_service()
user = user_service.get_user(user_email)
if not user or not user.referred_by_code:
return None
user_data = {
"referred_by_code": user.referred_by_code,
"referral_discount_expires_at": user.referral_discount_expires_at,
}
if not self.should_apply_discount(user_data):
return None
link = self.get_link_by_code(user.referred_by_code)
if not link:
return None
coupon_id = self.get_or_create_stripe_coupon(link.discount_percent)
if not coupon_id:
return None
return {"coupon_id": coupon_id, "discount_percent": link.discount_percent}
# ========================================================================
# Earnings
# ========================================================================
def record_earning(self, referred_email: str, stripe_session_id: str, purchase_amount_cents: int) -> Optional[dict]:
"""Record a referral earning after a credit purchase."""
referred_email = referred_email.lower()
# Look up the referred user
user_doc = self.db.collection("gen_users").document(referred_email).get()
if not user_doc.exists:
return None
user_data = user_doc.to_dict()
referral_code = user_data.get("referred_by_code")
if not referral_code:
return None
referred_at = user_data.get("referred_at")
if not referred_at:
return None
link = self.get_link_by_code(referral_code)
if not link:
return None
# Check earning window
earning_expires = referred_at + timedelta(days=link.earning_duration_days)
if datetime.utcnow() > earning_expires:
return None
# Calculate earning
earning_amount = int(purchase_amount_cents * link.kickback_percent / 100)
if earning_amount <= 0:
return None
import uuid
earning_id = str(uuid.uuid4())
earning = ReferralEarning(
id=earning_id,
referrer_email=link.owner_email,
referred_email=referred_email,
referral_code=referral_code,
stripe_session_id=stripe_session_id,
purchase_amount_cents=purchase_amount_cents,
earning_amount_cents=earning_amount,
)
self.db.collection(REFERRAL_EARNINGS_COLLECTION).document(earning_id).set(
earning.model_dump(mode="json")
)
self.db.collection(REFERRAL_LINKS_COLLECTION).document(referral_code).update({
"stats.purchases": firestore.Increment(1),
"stats.total_earned_cents": firestore.Increment(earning_amount),
})
logger.info(f"Referral earning recorded: ${earning_amount / 100:.2f} for {link.owner_email}")
return {"earning_id": earning_id, "referrer_email": link.owner_email, "earning_amount_cents": earning_amount}
# ========================================================================
# Stripe Connect & Payouts
# ========================================================================
def create_connect_account(self, email: str) -> Tuple[Optional[str], Optional[str]]:
"""Create Stripe Connect account for a referrer."""
if not self.stripe_service:
return None, None
return self.stripe_service.create_connect_account(email)
def get_connect_account_status(self, account_id: str) -> Optional[dict]:
"""Get Stripe Connect account status."""
if not self.stripe_service:
return None
return self.stripe_service.get_connect_account_status(account_id)
def create_connect_login_link(self, account_id: str) -> Optional[str]:
"""Create login link for Stripe Express dashboard."""
if not self.stripe_service:
return None
return self.stripe_service.create_connect_login_link(account_id)
def create_connect_update_link(self, account_id: str) -> Optional[str]:
"""Create account link for updating Connect info."""
if not self.stripe_service:
return None
return self.stripe_service.create_connect_update_link(account_id)
def get_pending_earnings(self, referrer_email: str) -> List[dict]:
"""Get all pending earnings for a referrer."""
query = (
self.db.collection(REFERRAL_EARNINGS_COLLECTION)
.where("referrer_email", "==", referrer_email.lower())
.where("status", "==", "pending")
)
return [
{**doc.to_dict(), "_ref": doc.reference}
for doc in query.get()
]
def check_and_trigger_payout(self, referrer_email: str, stripe_connect_account_id: str) -> bool:
"""Check if pending earnings meet threshold and trigger payout."""
pending = self.get_pending_earnings(referrer_email)
total_pending = sum(e.get("earning_amount_cents", 0) for e in pending)
if total_pending < PAYOUT_THRESHOLD_CENTS:
return False
if not self.stripe_service:
return False
transfer_id = self.stripe_service.create_transfer(
amount_cents=total_pending,
destination_account_id=stripe_connect_account_id,
)
if not transfer_id:
return False
import uuid
payout_id = str(uuid.uuid4())
earning_ids = [e["id"] for e in pending]
payout = ReferralPayout(
id=payout_id,
referrer_email=referrer_email.lower(),
stripe_transfer_id=transfer_id,
amount_cents=total_pending,
earnings_included=earning_ids,
)
self.db.collection(REFERRAL_PAYOUTS_COLLECTION).document(payout_id).set(
payout.model_dump(mode="json")
)
for earning in pending:
if "_ref" in earning:
earning["_ref"].update({
"status": "paid",
"paid_at": datetime.utcnow(),
"payout_id": payout_id,
})
logger.info(f"Payout triggered: ${total_pending / 100:.2f} to {referrer_email} (transfer: {transfer_id})")
return True
def get_dashboard_data(self, user_email: str) -> dict:
"""Get full referral dashboard data for a user."""
user_email = user_email.lower()
link = self.get_or_create_link(user_email)
# Get recent earnings (last 20)
earnings_query = (
self.db.collection(REFERRAL_EARNINGS_COLLECTION)
.where("referrer_email", "==", user_email)
.order_by("created_at", direction=firestore.Query.DESCENDING)
.limit(20)
)
earnings = [doc.to_dict() for doc in earnings_query.stream()]
# Get recent payouts (last 10)
payouts_query = (
self.db.collection(REFERRAL_PAYOUTS_COLLECTION)
.where("referrer_email", "==", user_email)
.order_by("created_at", direction=firestore.Query.DESCENDING)
.limit(10)
)
payouts = [doc.to_dict() for doc in payouts_query.stream()]
# Compute pending balance from ALL pending earnings (not limited to recent)
pending_earnings = self.get_pending_earnings(user_email)
pending_balance = sum(e.get("earning_amount_cents", 0) for e in pending_earnings)
total_earned = link.stats.total_earned_cents
# total_paid = total_earned - pending (since total_earned tracks all time)
total_paid = total_earned - pending_balance
# Check Connect status and fetch account details
from backend.services.user_service import get_user_service
user = get_user_service().get_user(user_email)
has_connect = bool(user and user.stripe_connect_account_id)
connect_account = None
if has_connect and self.stripe_service:
connect_account = self.stripe_service.get_connect_account_status(
user.stripe_connect_account_id
)
return {
"link": {
"code": link.code,
"display_name": link.display_name,
"custom_message": link.custom_message,
"discount_percent": link.discount_percent,
"kickback_percent": link.kickback_percent,
"discount_duration_days": link.discount_duration_days,
"earning_duration_days": link.earning_duration_days,
"stats": link.stats.model_dump(),
"enabled": link.enabled,
"is_vanity": link.is_vanity,
},
"pending_balance_cents": pending_balance,
"total_earned_cents": total_earned,
"total_paid_cents": total_paid,
"recent_earnings": [
{
"id": e["id"],
"referred_email": e.get("referred_email", ""),
"purchase_amount_cents": e.get("purchase_amount_cents", 0),
"earning_amount_cents": e.get("earning_amount_cents", 0),
"status": e.get("status", "pending"),
"created_at": str(e.get("created_at", "")),
}
for e in earnings
],
"recent_payouts": [
{
"id": p["id"],
"amount_cents": p.get("amount_cents", 0),
"status": p.get("status", "processing"),
"created_at": str(p.get("created_at", "")),
}
for p in payouts
],
"stripe_connect_configured": has_connect,
"stripe_connect_account": connect_account,
}
def list_links(self, limit: int = 50, offset: int = 0) -> list[ReferralLink]:
"""List referral links for admin view."""
query = (
self.db.collection(REFERRAL_LINKS_COLLECTION)
.order_by("created_at", direction=firestore.Query.DESCENDING)
.offset(offset)
.limit(limit)
)
links = []
for doc in query.stream():
try:
links.append(ReferralLink(**doc.to_dict()))
except Exception:
logger.warning("Skipping malformed referral link doc: %s", doc.id)
return links
# ============================================================================
# Singleton accessor
# ============================================================================
_referral_service = None
def get_referral_service() -> ReferralService:
global _referral_service
if _referral_service is None:
from backend.services.stripe_service import get_stripe_service
_referral_service = ReferralService(stripe_service=get_stripe_service())
return _referral_service
-759
View File
@@ -1,759 +0,0 @@
"""
Stripe admin service for payment analytics and management.
Handles:
- Storing enriched payment data from webhooks
- Revenue analytics and reporting
- Payment listing and detail views
- Stripe balance and payout queries
- Refund processing
- Dispute monitoring
- Webhook event logging
"""
import logging
import time
from datetime import datetime, timedelta
from typing import Optional, Dict, Any, List, Tuple
import stripe
from google.cloud import firestore
from backend.config import get_settings
logger = logging.getLogger(__name__)
STRIPE_PAYMENTS_COLLECTION = "stripe_payments"
STRIPE_WEBHOOK_EVENTS_COLLECTION = "stripe_webhook_events"
class StripeAdminService:
"""Service for Stripe admin analytics and payment management."""
def __init__(self):
self.settings = get_settings()
self.db = firestore.Client()
self._revenue_cache: Optional[Dict[str, Any]] = None
self._revenue_cache_time: float = 0
self._revenue_cache_ttl: float = 300 # 5 minutes
# =========================================================================
# Payment Storage
# =========================================================================
def store_payment(self, session: Dict[str, Any]) -> bool:
"""
Store enriched payment data from a completed Checkout Session.
Args:
session: Stripe Checkout Session object (expanded with payment_intent,
charge, and balance_transaction)
Returns:
True if stored successfully
"""
try:
session_id = session.get("id", "")
metadata = session.get("metadata", {})
# Extract payment intent and charge details
payment_intent = session.get("payment_intent", {})
if isinstance(payment_intent, str):
# Not expanded - just store the ID
payment_intent_id = payment_intent
charge_data = {}
balance_txn = {}
else:
payment_intent_id = payment_intent.get("id", "")
charges = payment_intent.get("latest_charge", {})
if isinstance(charges, str):
charge_data = {}
balance_txn = {}
else:
charge_data = charges or {}
bt = charge_data.get("balance_transaction", {})
balance_txn = bt if isinstance(bt, dict) else {}
# Extract payment method details
payment_method_details = charge_data.get("payment_method_details", {})
pm_type = payment_method_details.get("type", "unknown")
card_info = payment_method_details.get("card", {})
# Determine order type
order_type = metadata.get("order_type", "credit_purchase")
package_id = metadata.get("package_id")
credits = 0
try:
credits = int(metadata.get("credits", 0))
except (ValueError, TypeError):
pass
# Build product description
if order_type == "made_for_you":
artist = metadata.get("artist", "")
title = metadata.get("title", "")
product_description = f"Made For You: {artist} - {title}"
else:
from backend.services.stripe_service import CREDIT_PACKAGES
pkg = CREDIT_PACKAGES.get(package_id, {})
product_description = pkg.get("name", f"{credits} Credits")
# Extract discount info
total_details = session.get("total_details", {})
discount_amount = 0
promotion_code = None
if total_details:
discount_obj = total_details.get("amount_discount", 0)
discount_amount = discount_obj if isinstance(discount_obj, int) else 0
# Determine if test payment
is_test = not (session.get("livemode", False))
payment_data = {
"session_id": session_id,
"payment_intent_id": payment_intent_id,
"charge_id": charge_data.get("id", ""),
# Financial (cents)
"amount_total": session.get("amount_total", 0),
"currency": session.get("currency", "usd"),
"stripe_fee": balance_txn.get("fee", 0),
"net_amount": balance_txn.get("net", 0),
# Customer
"customer_email": (
metadata.get("user_email")
or metadata.get("customer_email")
or session.get("customer_email", "")
).lower().strip(),
"customer_name": session.get("customer_details", {}).get("name", ""),
"stripe_customer_id": session.get("customer", ""),
# Payment method
"payment_method_type": pm_type,
"card_brand": card_info.get("brand", ""),
"card_last4": card_info.get("last4", ""),
# Order details
"order_type": order_type,
"package_id": package_id,
"credits_granted": credits,
"product_description": product_description,
# Made-for-you fields
"artist": metadata.get("artist"),
"title": metadata.get("title"),
"job_id": metadata.get("job_id"),
# Status
"status": "succeeded",
"refund_amount": 0,
"refund_id": None,
"refunded_at": None,
"refund_reason": None,
# Timestamps
"created_at": datetime.utcfromtimestamp(session["created"]) if session.get("created") else datetime.utcnow(),
"processed_at": datetime.utcnow(),
# Metadata
"is_test": is_test,
"promotion_code": promotion_code,
"discount_amount": discount_amount,
}
doc_ref = self.db.collection(STRIPE_PAYMENTS_COLLECTION).document(session_id)
doc_ref.set(payment_data)
# Invalidate revenue cache
self._revenue_cache = None
logger.info(
f"Stored payment data for session {session_id}: "
f"${payment_data['amount_total'] / 100:.2f} from {payment_data['customer_email']}"
)
return True
except Exception:
logger.exception(f"Error storing payment data for session {session.get('id', 'unknown')}")
return False
def update_payment_status(self, session_id: str, updates: Dict[str, Any]) -> bool:
"""Update payment status fields (for refunds/disputes)."""
try:
doc_ref = self.db.collection(STRIPE_PAYMENTS_COLLECTION).document(session_id)
doc = doc_ref.get()
if not doc.exists:
logger.warning(f"Payment {session_id} not found for status update")
return False
doc_ref.update(updates)
self._revenue_cache = None
return True
except Exception:
logger.exception(f"Error updating payment status for {session_id}")
return False
# =========================================================================
# Revenue Analytics
# =========================================================================
def get_revenue_summary(
self, days: int = 30, exclude_test: bool = True
) -> Dict[str, Any]:
"""
Get revenue summary metrics.
Returns total gross, fees, net, refunds, count, AOV, revenue by product type.
Uses in-memory cache with 5-minute TTL.
"""
cache_key = f"{days}_{exclude_test}"
now = time.time()
if (
self._revenue_cache
and self._revenue_cache.get("_key") == cache_key
and (now - self._revenue_cache_time) < self._revenue_cache_ttl
):
return self._revenue_cache
try:
collection = self.db.collection(STRIPE_PAYMENTS_COLLECTION)
query = collection
if days > 0:
cutoff = datetime.utcnow() - timedelta(days=days)
query = query.where("created_at", ">=", cutoff)
total_gross = 0
total_fees = 0
total_net = 0
total_refunds = 0
count = 0
by_type: Dict[str, int] = {}
for doc in query.stream():
data = doc.to_dict()
if exclude_test and data.get("is_test"):
continue
amount = data.get("amount_total", 0)
total_gross += amount
total_fees += data.get("stripe_fee", 0)
total_net += data.get("net_amount", 0)
total_refunds += data.get("refund_amount", 0)
count += 1
order_type = data.get("order_type", "credit_purchase")
by_type[order_type] = by_type.get(order_type, 0) + amount
aov = total_gross / count if count > 0 else 0
result = {
"_key": cache_key,
"total_gross": total_gross,
"total_fees": total_fees,
"total_net": total_net,
"total_refunds": total_refunds,
"transaction_count": count,
"average_order_value": round(aov),
"revenue_by_type": by_type,
}
self._revenue_cache = result
self._revenue_cache_time = now
return result
except Exception:
logger.exception("Error calculating revenue summary")
return {
"total_gross": 0,
"total_fees": 0,
"total_net": 0,
"total_refunds": 0,
"transaction_count": 0,
"average_order_value": 0,
"revenue_by_type": {},
}
def get_revenue_by_period(
self,
days: int = 30,
group_by: str = "day",
exclude_test: bool = True,
) -> List[Dict[str, Any]]:
"""
Get revenue grouped by time period for charting.
Args:
days: Number of days to look back
group_by: 'day' or 'week' or 'month'
exclude_test: Exclude test transactions
Returns:
List of {date, gross, net, fees, count} dicts
"""
try:
collection = self.db.collection(STRIPE_PAYMENTS_COLLECTION)
cutoff = datetime.utcnow() - timedelta(days=days)
query = collection.where("created_at", ">=", cutoff)
buckets: Dict[str, Dict[str, Any]] = {}
for doc in query.stream():
data = doc.to_dict()
if exclude_test and data.get("is_test"):
continue
created = data.get("created_at")
if not created:
continue
if isinstance(created, str):
try:
created = datetime.fromisoformat(created.replace("Z", "+00:00"))
except Exception:
continue
if group_by == "week":
# Start of week (Monday)
start = created - timedelta(days=created.weekday())
key = start.strftime("%Y-%m-%d")
elif group_by == "month":
key = created.strftime("%Y-%m-01")
else:
key = created.strftime("%Y-%m-%d")
if key not in buckets:
buckets[key] = {"date": key, "gross": 0, "net": 0, "fees": 0, "count": 0}
buckets[key]["gross"] += data.get("amount_total", 0)
buckets[key]["net"] += data.get("net_amount", 0)
buckets[key]["fees"] += data.get("stripe_fee", 0)
buckets[key]["count"] += 1
result = sorted(buckets.values(), key=lambda x: x["date"])
return result
except Exception:
logger.exception("Error getting revenue by period")
return []
# =========================================================================
# Payment Listing & Details
# =========================================================================
def list_payments(
self,
limit: int = 50,
offset: int = 0,
order_type: Optional[str] = None,
status: Optional[str] = None,
email: Optional[str] = None,
exclude_test: bool = True,
) -> Dict[str, Any]:
"""
List payments with pagination and filters.
Returns:
{payments: [...], total: int, has_more: bool}
"""
try:
collection = self.db.collection(STRIPE_PAYMENTS_COLLECTION)
query = collection.order_by("created_at", direction=firestore.Query.DESCENDING)
if order_type:
query = collection.where("order_type", "==", order_type).order_by(
"created_at", direction=firestore.Query.DESCENDING
)
if status:
query = collection.where("status", "==", status).order_by(
"created_at", direction=firestore.Query.DESCENDING
)
if email:
query = collection.where("customer_email", "==", email.lower()).order_by(
"created_at", direction=firestore.Query.DESCENDING
)
# Stream and filter (Firestore doesn't support multiple inequality filters)
payments = []
skipped = 0
total = 0
for doc in query.limit(1000).stream():
data = doc.to_dict()
data["session_id"] = doc.id
if exclude_test and data.get("is_test"):
continue
# Apply filters that couldn't be done in query
if order_type and data.get("order_type") != order_type:
continue
if status and data.get("status") != status:
continue
if email and data.get("customer_email", "").lower() != email.lower():
continue
total += 1
if skipped < offset:
skipped += 1
continue
if len(payments) < limit:
# Convert datetime objects for JSON serialization
for key in ["created_at", "processed_at", "refunded_at"]:
if isinstance(data.get(key), datetime):
data[key] = data[key].isoformat()
payments.append(data)
return {
"payments": payments,
"total": total,
"has_more": total > offset + limit,
}
except Exception:
logger.exception("Error listing payments")
return {"payments": [], "total": 0, "has_more": False}
def get_payment_detail(self, session_id: str) -> Optional[Dict[str, Any]]:
"""
Get full payment details for a single payment.
Combines Firestore data with on-demand Stripe API call for receipt_url.
"""
try:
doc_ref = self.db.collection(STRIPE_PAYMENTS_COLLECTION).document(session_id)
doc = doc_ref.get()
if not doc.exists:
return None
data = doc.to_dict()
data["session_id"] = doc.id
# Convert datetime objects
for key in ["created_at", "processed_at", "refunded_at"]:
if isinstance(data.get(key), datetime):
data[key] = data[key].isoformat()
# On-demand: fetch receipt URL from Stripe
charge_id = data.get("charge_id")
if charge_id:
try:
charge = stripe.Charge.retrieve(charge_id)
data["receipt_url"] = charge.get("receipt_url")
data["stripe_dashboard_url"] = (
f"https://dashboard.stripe.com/payments/{data.get('payment_intent_id', '')}"
)
except Exception as e:
logger.warning(f"Could not fetch charge {charge_id}: {e}")
data["receipt_url"] = None
return data
except Exception:
logger.exception(f"Error getting payment detail for {session_id}")
return None
def get_user_payment_history(self, email: str) -> Dict[str, Any]:
"""Get all payments for a user with total_spent."""
try:
collection = self.db.collection(STRIPE_PAYMENTS_COLLECTION)
# Try with order_by first (requires composite index on customer_email + created_at)
try:
query = (
collection.where("customer_email", "==", email.lower())
.order_by("created_at", direction=firestore.Query.DESCENDING)
)
docs = list(query.stream())
except Exception:
# Fallback: query without order_by if composite index doesn't exist yet
logger.warning(
f"Composite index missing for stripe_payments(customer_email, created_at). "
f"Falling back to unordered query for {email}"
)
query = collection.where("customer_email", "==", email.lower())
docs = list(query.stream())
payments = []
total_spent = 0
total_refunded = 0
for doc in docs:
data = doc.to_dict()
data["session_id"] = doc.id
total_spent += data.get("amount_total", 0)
total_refunded += data.get("refund_amount", 0)
for key in ["created_at", "processed_at", "refunded_at"]:
if isinstance(data.get(key), datetime):
data[key] = data[key].isoformat()
payments.append(data)
# Sort by created_at descending (handles both indexed and fallback paths)
payments.sort(key=lambda p: p.get("created_at", ""), reverse=True)
first_payment = payments[-1]["created_at"] if payments else None
last_payment = payments[0]["created_at"] if payments else None
return {
"email": email,
"payments": payments,
"total_spent": total_spent,
"total_refunded": total_refunded,
"net_spent": total_spent - total_refunded,
"payment_count": len(payments),
"first_payment_at": first_payment,
"last_payment_at": last_payment,
}
except Exception:
logger.exception(f"Error getting payment history for {email}")
return {
"email": email,
"payments": [],
"total_spent": 0,
"total_refunded": 0,
"net_spent": 0,
"payment_count": 0,
"first_payment_at": None,
"last_payment_at": None,
}
# =========================================================================
# Stripe Balance & Payouts
# =========================================================================
def get_stripe_balance(self) -> Dict[str, Any]:
"""Get current Stripe account balance (available + pending)."""
try:
balance = stripe.Balance.retrieve()
available = sum(b.get("amount", 0) for b in balance.get("available", []))
pending = sum(b.get("amount", 0) for b in balance.get("pending", []))
return {
"available": available,
"pending": pending,
"currency": "usd",
}
except Exception:
logger.exception("Error fetching Stripe balance")
return {"available": 0, "pending": 0, "currency": "usd"}
def get_recent_payouts(self, limit: int = 10) -> List[Dict[str, Any]]:
"""Get recent payouts from Stripe."""
try:
payouts = stripe.Payout.list(limit=limit)
result = []
for p in payouts.get("data", []):
result.append({
"id": p.get("id"),
"amount": p.get("amount"),
"currency": p.get("currency"),
"status": p.get("status"),
"arrival_date": p.get("arrival_date"),
"created": p.get("created"),
"description": p.get("description"),
"method": p.get("method"),
})
return result
except Exception:
logger.exception("Error fetching payouts")
return []
# =========================================================================
# Disputes
# =========================================================================
def get_recent_disputes(self) -> List[Dict[str, Any]]:
"""Get open and recent disputes from Stripe."""
try:
disputes = stripe.Dispute.list(limit=20)
result = []
for d in disputes.get("data", []):
result.append({
"id": d.get("id"),
"amount": d.get("amount"),
"currency": d.get("currency"),
"status": d.get("status"),
"reason": d.get("reason"),
"charge_id": d.get("charge"),
"created": d.get("created"),
"evidence_due_by": d.get("evidence_details", {}).get("due_by"),
"payment_intent_id": d.get("payment_intent"),
})
return result
except Exception:
logger.exception("Error fetching disputes")
return []
# =========================================================================
# Refund Processing
# =========================================================================
def process_refund(
self,
session_id: str,
amount: Optional[int] = None,
reason: str = "requested_by_customer",
) -> Tuple[bool, str]:
"""
Process a refund for a payment.
Args:
session_id: Checkout session ID
amount: Refund amount in cents (None for full refund)
reason: Refund reason
Returns:
(success, message)
"""
try:
# Get payment record
doc_ref = self.db.collection(STRIPE_PAYMENTS_COLLECTION).document(session_id)
doc = doc_ref.get()
if not doc.exists:
return False, "Payment not found"
data = doc.to_dict()
payment_intent_id = data.get("payment_intent_id")
if not payment_intent_id:
return False, "No payment intent ID found"
current_refund = data.get("refund_amount", 0)
original_amount = data.get("amount_total", 0)
refund_amount = amount or (original_amount - current_refund)
if refund_amount <= 0:
return False, "Invalid refund amount"
if current_refund + refund_amount > original_amount:
return False, f"Refund would exceed original amount (${original_amount / 100:.2f})"
# Create Stripe refund
refund_params: Dict[str, Any] = {
"payment_intent": payment_intent_id,
"amount": refund_amount,
"reason": reason,
}
refund = stripe.Refund.create(**refund_params)
# Update Firestore record
new_total_refund = current_refund + refund_amount
new_status = "refunded" if new_total_refund >= original_amount else "partially_refunded"
doc_ref.update({
"status": new_status,
"refund_amount": new_total_refund,
"refund_id": refund.get("id"),
"refunded_at": datetime.utcnow(),
"refund_reason": reason,
})
self._revenue_cache = None
logger.info(
f"Processed refund for {session_id}: "
f"${refund_amount / 100:.2f} ({new_status})"
)
return True, f"Refund of ${refund_amount / 100:.2f} processed successfully"
except stripe.error.StripeError as e:
logger.error(f"Stripe refund error for {session_id}: {e}")
return False, f"Stripe error: {str(e)}"
except Exception:
logger.exception(f"Error processing refund for {session_id}")
return False, "Unexpected error processing refund"
# =========================================================================
# Webhook Event Logging
# =========================================================================
def log_webhook_event(
self,
event_id: str,
event_type: str,
status: str = "processing",
session_id: Optional[str] = None,
customer_email: Optional[str] = None,
summary: Optional[str] = None,
error_message: Optional[str] = None,
) -> None:
"""Log a webhook event for audit trail."""
try:
doc_ref = self.db.collection(STRIPE_WEBHOOK_EVENTS_COLLECTION).document(event_id)
data = {
"event_id": event_id,
"event_type": event_type,
"created_at": datetime.utcnow(),
"status": status,
"session_id": session_id,
"customer_email": customer_email,
"summary": summary,
"error_message": error_message,
}
if status in ("success", "error"):
data["processed_at"] = datetime.utcnow()
doc_ref.set(data, merge=True)
except Exception:
logger.exception(f"Error logging webhook event {event_id}")
def list_webhook_events(
self,
limit: int = 50,
event_type: Optional[str] = None,
status: Optional[str] = None,
) -> List[Dict[str, Any]]:
"""List recent webhook events."""
try:
collection = self.db.collection(STRIPE_WEBHOOK_EVENTS_COLLECTION)
query = collection.order_by("created_at", direction=firestore.Query.DESCENDING)
if event_type:
query = collection.where("event_type", "==", event_type).order_by(
"created_at", direction=firestore.Query.DESCENDING
)
events = []
for doc in query.limit(limit).stream():
data = doc.to_dict()
if status and data.get("status") != status:
continue
for key in ["created_at", "processed_at"]:
if isinstance(data.get(key), datetime):
data[key] = data[key].isoformat()
events.append(data)
return events
except Exception:
logger.exception("Error listing webhook events")
return []
def normalize_customer_emails(self) -> Dict[str, Any]:
"""Backfill: lowercase all customer_email values in stripe_payments."""
try:
collection = self.db.collection(STRIPE_PAYMENTS_COLLECTION)
updated = 0
skipped = 0
for doc in collection.stream():
data = doc.to_dict()
email = data.get("customer_email", "")
normalized = email.lower().strip()
if email != normalized:
doc.reference.update({"customer_email": normalized})
updated += 1
else:
skipped += 1
return {
"status": "success",
"updated": updated,
"skipped": skipped,
"total": updated + skipped,
}
except Exception:
logger.exception("Error normalizing customer emails")
raise
# Global instance
_stripe_admin_service = None
def get_stripe_admin_service() -> StripeAdminService:
"""Get the global Stripe admin service instance."""
global _stripe_admin_service
if _stripe_admin_service is None:
_stripe_admin_service = StripeAdminService()
return _stripe_admin_service
+6 -122
View File
@@ -6,7 +6,6 @@ Handles:
- Processing webhook events
- Managing customer records
"""
import json
import logging
import os
from typing import Optional, Dict, Any, Tuple
@@ -23,25 +22,25 @@ logger = logging.getLogger(__name__)
CREDIT_PACKAGES = {
"1_credit": {
"credits": 1,
"price_cents": 1000, # $10.00
"price_cents": 500, # $5.00
"name": "1 Karaoke Credit",
"description": "Create 1 professional karaoke video",
},
"3_credits": {
"credits": 3,
"price_cents": 2400, # $24.00 (20% discount)
"price_cents": 1200, # $12.00 (20% discount)
"name": "3 Karaoke Credits",
"description": "Create 3 professional karaoke videos (Save 20%)",
},
"5_credits": {
"credits": 5,
"price_cents": 3500, # $35.00 (30% discount)
"price_cents": 1750, # $17.50 (30% discount)
"name": "5 Karaoke Credits",
"description": "Create 5 professional karaoke videos (Save 30%)",
},
"10_credits": {
"credits": 10,
"price_cents": 6000, # $60.00 (40% discount)
"price_cents": 3000, # $30.00 (40% discount)
"name": "10 Karaoke Credits",
"description": "Create 10 professional karaoke videos (Save 40%)",
},
@@ -49,7 +48,7 @@ CREDIT_PACKAGES = {
# Made-for-you service package (not a credit package - creates a job directly)
MADE_FOR_YOU_PACKAGE = {
"price_cents": 5000, # $50.00
"price_cents": 1500, # $15.00
"name": "Made For You Karaoke Video",
"description": "Professional 4K karaoke video with perfectly synced lyrics, delivered to your email within 24 hours",
"images": ["https://gen.nomadkaraoke.com/nomad-logo.png"],
@@ -97,7 +96,6 @@ class StripeService:
user_email: str,
success_url: Optional[str] = None,
cancel_url: Optional[str] = None,
coupon_id: Optional[str] = None,
) -> Tuple[bool, Optional[str], str]:
"""
Create a Stripe Checkout session for purchasing credits.
@@ -153,11 +151,6 @@ class StripeService:
'allow_promotion_codes': True,
}
# If a referral coupon is applied, use discounts instead of allow_promotion_codes
if coupon_id:
session_params.pop('allow_promotion_codes', None)
session_params['discounts'] = [{'coupon': coupon_id}]
# Add payment method configuration if set (enables Google Pay, Link, etc.)
if self.payment_method_config:
session_params['payment_method_configuration'] = self.payment_method_config
@@ -291,10 +284,7 @@ class StripeService:
event = stripe.Webhook.construct_event(
payload, signature, self.webhook_secret
)
# Convert Stripe Event object to plain dict — Stripe SDK v14+
# returns StripeObject instances that don't support .get()
event_dict = json.loads(str(event))
return True, event_dict, "Webhook verified"
return True, event, "Webhook verified"
except stripe.error.SignatureVerificationError as e:
logger.error(f"Invalid webhook signature: {e}")
return False, None, "Invalid signature"
@@ -361,112 +351,6 @@ class StripeService:
logger.error(f"Error retrieving session {session_id}: {e}")
return None
def get_or_create_referral_coupon(self, discount_percent: int) -> Optional[str]:
"""Get or create a Stripe coupon for a referral discount percentage."""
if not self.is_configured():
return None
coupon_id = f"referral-{discount_percent}pct"
try:
stripe.Coupon.retrieve(coupon_id)
return coupon_id
except stripe.error.InvalidRequestError:
try:
stripe.Coupon.create(
id=coupon_id,
percent_off=discount_percent,
duration="once",
name=f"Referral {discount_percent}% off",
)
logger.info(f"Created Stripe coupon: {coupon_id}")
return coupon_id
except stripe.error.StripeError as e:
logger.error(f"Failed to create coupon {coupon_id}: {e}")
return None
def create_connect_account(self, email: str) -> Tuple[Optional[str], Optional[str]]:
"""Create a Stripe Connect Express account. Returns (account_id, onboarding_url)."""
if not self.is_configured():
return None, None
try:
account = stripe.Account.create(
type="express",
email=email,
capabilities={"transfers": {"requested": True}},
metadata={"source": "nomad_karaoke_referrals"},
)
account_link = stripe.AccountLink.create(
account=account.id,
refresh_url=f"{self.frontend_url}/app/referrals?connect=refresh",
return_url=f"{self.frontend_url}/app/referrals?connect=complete",
type="account_onboarding",
)
logger.info(f"Created Connect account {account.id} for {email}")
return account.id, account_link.url
except stripe.error.StripeError as e:
logger.error(f"Failed to create Connect account for {email}: {e}")
return None, None
def get_connect_account_status(self, account_id: str) -> Optional[dict]:
"""Retrieve Stripe Connect account status. Returns dict with account details."""
if not self.is_configured():
return None
try:
account = stripe.Account.retrieve(account_id)
return {
"account_id": account.id,
"charges_enabled": account.charges_enabled,
"payouts_enabled": account.payouts_enabled,
"details_submitted": account.details_submitted,
"email": account.email,
}
except stripe.error.StripeError as e:
logger.error(f"Failed to retrieve Connect account {account_id}: {e}")
return None
def create_connect_login_link(self, account_id: str) -> Optional[str]:
"""Create a login link to the Stripe Express dashboard. Returns URL."""
if not self.is_configured():
return None
try:
link = stripe.Account.create_login_link(account_id)
return link.url
except stripe.error.StripeError as e:
logger.error(f"Failed to create login link for {account_id}: {e}")
return None
def create_connect_update_link(self, account_id: str) -> Optional[str]:
"""Create an account link for updating Connect account info. Returns URL."""
if not self.is_configured():
return None
try:
account_link = stripe.AccountLink.create(
account=account_id,
refresh_url=f"{self.frontend_url}/app/referrals?connect=refresh",
return_url=f"{self.frontend_url}/app/referrals?connect=complete",
type="account_onboarding",
)
return account_link.url
except stripe.error.StripeError as e:
logger.error(f"Failed to create update link for {account_id}: {e}")
return None
def create_transfer(self, amount_cents: int, destination_account_id: str, description: str = "Nomad Karaoke referral payout") -> Optional[str]:
"""Create a transfer to a Connect account. Returns transfer ID."""
if not self.is_configured():
return None
try:
transfer = stripe.Transfer.create(
amount=amount_cents,
currency="usd",
destination=destination_account_id,
description=description,
)
logger.info(f"Created transfer {transfer.id}: ${amount_cents / 100:.2f} to {destination_account_id}")
return transfer.id
except stripe.error.StripeError as e:
logger.error(f"Failed to create transfer to {destination_account_id}: {e}")
return None
def create_customer(self, email: str, name: Optional[str] = None) -> Optional[str]:
"""
Create or get a Stripe customer for a user.
+15 -186
View File
@@ -14,7 +14,6 @@ import re
from typing import Optional, Dict, Any
from backend.config import get_settings
from backend.i18n import t
logger = logging.getLogger(__name__)
@@ -80,18 +79,6 @@ Thanks!
"""
DEFAULT_YOUTUBE_UPLOAD_COMPLETE_TEMPLATE = """Hi {name},
Great news! Your karaoke video for "{artist} - {title}" has been uploaded to YouTube:
{youtube_url}
This upload was briefly delayed while our systems processed other uploads, but everything is all set now.
Thanks!
-Andrew
"""
class TemplateService:
"""
Service for fetching and rendering email templates from GCS.
@@ -155,48 +142,29 @@ class TemplateService:
logger.error(f"Failed to fetch template {blob_path}: {e}")
return None
def get_job_completion_template(self, locale: str = "en") -> str:
def get_job_completion_template(self) -> str:
"""Get the job completion email template."""
# Try locale-specific GCS template first, then default GCS, then localized default
if locale != "en":
template = self._fetch_template_from_gcs(f"{locale}/job-completion.txt")
if template is not None:
return template
template = self._fetch_template_from_gcs("job-completion.txt")
if template is not None:
return template
logger.info("Using default job completion template")
if locale == "en":
if template is None:
logger.info("Using default job completion template")
return DEFAULT_JOB_COMPLETION_TEMPLATE
return self._build_localized_job_completion_template(locale)
return template
def get_action_needed_lyrics_template(self, locale: str = "en") -> str:
def get_action_needed_lyrics_template(self) -> str:
"""Get the lyrics review reminder template."""
if locale != "en":
template = self._fetch_template_from_gcs(f"{locale}/action-needed-lyrics.txt")
if template is not None:
return template
template = self._fetch_template_from_gcs("action-needed-lyrics.txt")
if template is not None:
return template
logger.info("Using default lyrics reminder template")
if locale == "en":
if template is None:
logger.info("Using default lyrics reminder template")
return DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE
return self._build_localized_action_lyrics_template(locale)
return template
def get_action_needed_instrumental_template(self, locale: str = "en") -> str:
def get_action_needed_instrumental_template(self) -> str:
"""Get the instrumental selection reminder template."""
if locale != "en":
template = self._fetch_template_from_gcs(f"{locale}/action-needed-instrumental.txt")
if template is not None:
return template
template = self._fetch_template_from_gcs("action-needed-instrumental.txt")
if template is not None:
return template
logger.info("Using default instrumental reminder template")
if locale == "en":
if template is None:
logger.info("Using default instrumental reminder template")
return DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE
return self._build_localized_action_instrumental_template(locale)
return template
def render_template(self, template: str, variables: Dict[str, Any]) -> str:
"""
@@ -235,74 +203,6 @@ class TemplateService:
return result.strip()
def _build_localized_job_completion_template(self, locale: str) -> str:
"""Build a localized job completion template from translation keys."""
ns = "emails.templates.jobCompletion"
return f"""{t(locale, f"{ns}.greeting")}
{t(locale, f"{ns}.thanks")}
{t(locale, f"{ns}.youtubeLink")}
{{youtube_url}}
{t(locale, f"{ns}.dropboxIntro")}
{t(locale, f"{ns}.fileList")}
{{dropbox_url}}
{t(locale, f"{ns}.fixOffer")}
{t(locale, f"{ns}.feedbackRequest")}
{{feedback_url}}
{t(locale, f"{ns}.signOff")}
"""
def _build_localized_action_lyrics_template(self, locale: str) -> str:
"""Build a localized action-needed lyrics template."""
ns = "emails.templates.actionLyrics"
return f"""{t(locale, f"{ns}.greeting")}
{t(locale, f"{ns}.ready")}
{t(locale, f"{ns}.instructions")}
{t(locale, f"{ns}.reviewPrompt")}
{{review_url}}
{t(locale, f"{ns}.duration")}
{t(locale, f"{ns}.signOff")}
"""
def _build_localized_action_instrumental_template(self, locale: str) -> str:
"""Build a localized action-needed instrumental template."""
ns = "emails.templates.actionInstrumental"
return f"""{t(locale, f"{ns}.greeting")}
{t(locale, f"{ns}.almostDone")}
{t(locale, f"{ns}.instructions")}
{t(locale, f"{ns}.selectPrompt")}
{{instrumental_url}}
{t(locale, f"{ns}.signOff")}
"""
def _build_localized_youtube_complete_template(self, locale: str) -> str:
"""Build a localized YouTube upload complete template."""
ns = "emails.templates.youtubeComplete"
return f"""{t(locale, f"{ns}.greeting")}
{t(locale, f"{ns}.greatNews")}
{{youtube_url}}
{t(locale, f"{ns}.delayNote")}
{t(locale, f"{ns}.signOff")}
"""
def render_job_completion(
self,
name: Optional[str] = None,
@@ -312,9 +212,6 @@ class TemplateService:
title: Optional[str] = None,
job_id: Optional[str] = None,
feedback_url: Optional[str] = None,
is_private: bool = False,
youtube_queued: bool = False,
locale: str = "en",
) -> str:
"""
Render the job completion email template.
@@ -327,36 +224,11 @@ class TemplateService:
title: Song title
job_id: Job ID
feedback_url: Feedback form URL (optional)
is_private: If True, remove YouTube section entirely (private/non-published tracks)
youtube_queued: If True, YouTube upload was deferred - show pending message
Returns:
Rendered email content
"""
template = self.get_job_completion_template(locale)
# For private tracks, remove the YouTube section entirely
if is_private:
template = re.sub(
r"\n*Here's the link for the karaoke video published to YouTube:\n\{youtube_url\}\n*",
'\n',
template,
)
elif youtube_queued and not youtube_url:
# Replace YouTube URL with pending message
queued_msg = t(locale, "emails.templates.jobCompletion.youtubeQueued")
youtube_link_text = t(locale, "emails.templates.jobCompletion.youtubeLink")
template = template.replace(
f"{youtube_link_text}\n{{youtube_url}}",
queued_msg
)
# Also handle the English default template pattern
if "{youtube_url}" in template:
template = template.replace(
"Here's the link for the karaoke video published to YouTube:\n{youtube_url}",
queued_msg
)
template = self.get_job_completion_template()
variables = {
"name": name or "there",
"youtube_url": youtube_url or "[YouTube URL not available]",
@@ -374,7 +246,6 @@ class TemplateService:
artist: Optional[str] = None,
title: Optional[str] = None,
review_url: str = "",
locale: str = "en",
) -> str:
"""
Render the combined review reminder template.
@@ -391,7 +262,7 @@ class TemplateService:
Returns:
Rendered email content
"""
template = self.get_action_needed_lyrics_template(locale)
template = self.get_action_needed_lyrics_template()
variables = {
"name": name or "there",
"artist": artist or "Unknown Artist",
@@ -406,7 +277,6 @@ class TemplateService:
artist: Optional[str] = None,
title: Optional[str] = None,
instrumental_url: str = "",
locale: str = "en",
) -> str:
"""
Render the instrumental selection reminder template.
@@ -424,7 +294,7 @@ class TemplateService:
Returns:
Rendered email content
"""
template = self.get_action_needed_instrumental_template(locale)
template = self.get_action_needed_instrumental_template()
variables = {
"name": name or "there",
"artist": artist or "Unknown Artist",
@@ -433,47 +303,6 @@ class TemplateService:
}
return self.render_template(template, variables)
def render_youtube_upload_complete(
self,
name: Optional[str] = None,
artist: Optional[str] = None,
title: Optional[str] = None,
youtube_url: Optional[str] = None,
locale: str = "en",
) -> str:
"""
Render the YouTube upload complete follow-up email template.
Sent when a deferred YouTube upload completes after quota becomes available.
Args:
name: User's display name
artist: Artist name
title: Song title
youtube_url: YouTube video URL
Returns:
Rendered email content
"""
if locale != "en":
template = self._fetch_template_from_gcs(f"{locale}/youtube-upload-complete.txt")
if template is not None:
pass # Use locale-specific GCS template
else:
template = self._fetch_template_from_gcs("youtube-upload-complete.txt")
else:
template = self._fetch_template_from_gcs("youtube-upload-complete.txt")
if template is None:
template = DEFAULT_YOUTUBE_UPLOAD_COMPLETE_TEMPLATE if locale == "en" else self._build_localized_youtube_complete_template(locale)
variables = {
"name": name or "there",
"artist": artist or "Unknown Artist",
"title": title or "Unknown Title",
"youtube_url": youtube_url or "[YouTube URL not available]",
}
return self.render_template(template, variables)
def upload_template(self, template_name: str, content: str) -> bool:
"""
Upload a template to GCS.
+1 -2
View File
@@ -334,8 +334,7 @@ class ThemeService:
style_params = self._update_asset_paths_in_style(theme_id, style_params)
# Upload modified style_params.json to job's style folder
# Uses jobs/ prefix (not uploads/) to avoid GCS lifecycle deletion rule
job_style_path = f"jobs/{job_id}/style/style_params.json"
job_style_path = f"uploads/{job_id}/style/style_params.json"
self.storage.upload_json(job_style_path, style_params)
logger.info(f"Prepared job {job_id} style from theme {theme_id}")
+26 -527
View File
@@ -13,7 +13,6 @@ import uuid
from datetime import datetime, timedelta, timezone
from typing import Optional, List, Tuple
from google.api_core import exceptions as google_exceptions
from google.cloud import firestore
from google.cloud.firestore_v1 import FieldFilter
from google.cloud.firestore_v1 import Increment
@@ -60,71 +59,29 @@ class UserService:
# User CRUD Operations
# =========================================================================
def get_user_doc_ref(self, email: str):
"""Get the Firestore document reference for a user by email.
Handles both email-keyed docs (doc ID = email) and hash-keyed docs
(doc ID = hash, email stored as field). Returns None if not found.
"""
try:
# First try direct lookup by email as doc ID
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
doc = doc_ref.get()
if doc.exists:
return doc_ref
# Fallback: query by email field (for hash-keyed docs)
query = self.db.collection(USERS_COLLECTION).where(
filter=FieldFilter('email', '==', email.lower())
).limit(1)
results = list(query.stream())
if results:
return results[0].reference
return None
except Exception:
logger.exception(f"Error finding user doc ref for {email}")
return None
def get_user(self, email: str) -> Optional[User]:
"""Get a user by email."""
try:
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
doc = doc_ref.get()
if doc.exists:
return User(**doc.to_dict())
if not doc.exists:
return None
# Fallback: query by email field (for hash-keyed docs)
query = self.db.collection(USERS_COLLECTION).where(
filter=FieldFilter('email', '==', email.lower())
).limit(1)
results = list(query.stream())
if results:
return User(**results[0].to_dict())
return None
return User(**doc.to_dict())
except Exception:
logger.exception(f"Error getting user {email}")
return None
def get_or_create_user(
self,
email: str,
tenant_id: Optional[str] = None,
signup_ip: Optional[str] = None,
device_fingerprint: Optional[str] = None,
) -> User:
def get_or_create_user(self, email: str, tenant_id: Optional[str] = None) -> User:
"""
Get existing user or create a new one.
Welcome credits are granted on first magic link verification, not here.
New users receive a welcome credit to try the service.
Args:
email: User's email address
tenant_id: Tenant ID for white-label portals (None = default Nomad Karaoke)
signup_ip: Client IP address at signup (for rate limiting)
device_fingerprint: Browser fingerprint at signup (for rate limiting)
Note: If user exists but has a different tenant_id, the existing user is returned.
Users are uniquely identified by email, not email+tenant.
@@ -138,17 +95,22 @@ class UserService:
# (though features may be restricted per-tenant)
return user
# Create new user with zero credits (welcome credits granted on first verification)
# Create new user with welcome credit
welcome_credit = self.NEW_USER_FREE_CREDITS
welcome_transaction = CreditTransaction(
id=str(uuid.uuid4()),
amount=welcome_credit,
reason="welcome_credit",
)
user = User(
email=email,
credits=0,
credit_transactions=[],
credits=welcome_credit,
credit_transactions=[welcome_transaction],
tenant_id=tenant_id, # Associate with tenant on creation
signup_ip=signup_ip,
device_fingerprint=device_fingerprint,
)
self._save_user(user)
logger.info(f"Created new user: {email} (tenant: {tenant_id or 'default'}) — credits pending verification")
logger.info(f"Created new user: {email} with {welcome_credit} welcome credit(s) (tenant: {tenant_id or 'default'})")
return user
def _save_user(self, user: User) -> None:
@@ -165,10 +127,7 @@ class UserService:
"""Update user fields."""
try:
updates['updated_at'] = datetime.utcnow()
doc_ref = self.get_user_doc_ref(email)
if not doc_ref:
logger.warning(f"Cannot update user {email}: doc not found")
return None
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
doc_ref.update(updates)
return self.get_user(email)
except Exception:
@@ -217,9 +176,7 @@ class UserService:
email: str,
ip_address: Optional[str] = None,
user_agent: Optional[str] = None,
tenant_id: Optional[str] = None,
device_fingerprint: Optional[str] = None,
referral_code: Optional[str] = None,
tenant_id: Optional[str] = None
) -> MagicLinkToken:
"""
Create a magic link token for email authentication.
@@ -232,17 +189,11 @@ class UserService:
ip_address: Client IP for auditing
user_agent: Client user agent for auditing
tenant_id: Tenant ID for white-label portals (None = default Nomad Karaoke)
device_fingerprint: Browser fingerprint for anti-abuse
"""
email = email.lower()
# Ensure user exists (with tenant association, signup IP, and fingerprint)
self.get_or_create_user(
email,
tenant_id=tenant_id,
signup_ip=ip_address,
device_fingerprint=device_fingerprint,
)
# Ensure user exists (with tenant association)
self.get_or_create_user(email, tenant_id=tenant_id)
# Generate secure token
token = secrets.token_urlsafe(32)
@@ -254,8 +205,6 @@ class UserService:
ip_address=ip_address,
user_agent=user_agent,
tenant_id=tenant_id,
device_fingerprint=device_fingerprint,
referral_code=referral_code,
)
# Save to Firestore
@@ -373,406 +322,6 @@ class UserService:
logger.exception("Error verifying magic link")
return False, None, "An error occurred during verification"
def grant_welcome_credits_if_eligible(
self,
email: str,
precomputed_eval: Optional[dict] = None,
) -> Tuple[bool, str]:
"""
Grant welcome credits on first magic link verification.
Uses AI evaluation to check for abuse before granting. Fail-open:
if evaluation fails, credits are granted anyway.
Uses a dedicated welcome_credits_granted flag as primary idempotency
check (survives credit_transactions list trimming). Falls back to
checking for welcome_credit transaction as secondary defense.
Args:
email: User's email address.
precomputed_eval: Optional pre-computed evaluation from magic link doc.
Keys: credit_eval_decision, credit_eval_reasoning, credit_eval_error.
If provided and has a decision, skips the AI call.
Returns:
Tuple of (credits_granted: bool, status: str).
status is one of: "granted", "denied", "already_granted", "not_found", "error"
"""
try:
email = email.lower()
user = self.get_user(email)
if not user:
return False, "not_found"
# Primary check: dedicated flag (survives transaction list trimming)
if user.welcome_credits_granted:
return False, "already_granted"
# Secondary check: look for existing welcome_credit transaction
for txn in user.credit_transactions:
if txn.reason == "welcome_credit":
return False, "already_granted"
# Skip AI evaluation for test emails (e.g. E2E test accounts)
# to avoid wasting LLM credits and sending unnecessary denial emails
from backend.utils.test_data import is_test_email
if not is_test_email(email):
# AI evaluation: check for abuse signals before granting
# Use pre-computed result from magic link if available
try:
from backend.services.credit_evaluation_service import (
get_credit_evaluation_service,
CreditEvaluation,
)
precomputed_decision = (precomputed_eval or {}).get("credit_eval_decision")
if precomputed_decision:
logger.info(f"Using pre-computed credit eval for {email}: {precomputed_decision}")
evaluation = CreditEvaluation(
decision=precomputed_decision,
reasoning=(precomputed_eval or {}).get("credit_eval_reasoning", ""),
confidence=1.0,
error=(precomputed_eval or {}).get("credit_eval_error"),
)
else:
eval_service = get_credit_evaluation_service()
evaluation = eval_service.evaluate(email, "welcome")
if evaluation.decision == "deny":
# Mark as evaluated but not granted so we don't re-evaluate
self.update_user(email, welcome_credits_granted=True)
logger.info(
f"Denied welcome credits to {email}: {evaluation.reasoning}"
)
try:
from backend.services.email_service import get_email_service
get_email_service().send_credit_denied_email(email, "welcome")
except Exception:
logger.exception(f"Failed to send credit denied email to {email}")
return False, "denied"
if evaluation.decision == "pending_review":
# Don't mark welcome_credits_granted — allow retry after manual review
logger.info(
f"Welcome credits pending review for {email}: {evaluation.reasoning}"
)
try:
from backend.services.email_service import get_email_service
email_service = get_email_service()
email_service.send_credit_review_needed_email(
email, "welcome", evaluation.reasoning, signals=evaluation.signals,
)
except Exception:
logger.exception(f"Failed to send review needed email for {email}")
return False, "pending_review"
except Exception:
logger.exception(f"Credit evaluation failed for {email} — pending review (fail-closed)")
try:
from backend.services.email_service import get_email_service
email_service = get_email_service()
email_service.send_credit_review_needed_email(
email, "welcome", f"Evaluation error: {email}",
)
except Exception:
pass
return False, "pending_review"
else:
logger.info(f"Skipping credit evaluation for test email {email}")
# Grant welcome credits
welcome_credit = self.NEW_USER_FREE_CREDITS
transaction = CreditTransaction(
id=str(uuid.uuid4()),
amount=welcome_credit,
reason="welcome_credit",
)
transactions = user.credit_transactions[-MAX_CREDIT_TRANSACTIONS + 1:]
transactions.append(transaction)
new_balance = user.credits + welcome_credit
self.update_user(
email,
credits=new_balance,
credit_transactions=[t.model_dump(mode='json') for t in transactions],
welcome_credits_granted=True,
)
logger.info(f"Granted {welcome_credit} welcome credits to {email}")
return True, "granted"
except Exception:
logger.exception(f"Error granting welcome credits to {email}")
return False, "error"
# =========================================================================
# Anti-Abuse Rate Limiting
# =========================================================================
# Maximum new signups allowed per IP address in the rate limit window
MAX_SIGNUPS_PER_IP = 2
SIGNUP_RATE_LIMIT_HOURS = 24
def count_recent_signups_from_ip(self, ip_address: str, hours: int = 24) -> int:
"""
Count new user accounts created from this IP in the last N hours.
Queries gen_users where signup_ip matches and created_at is recent.
Used to rate-limit new account creation per IP address.
Args:
ip_address: Client IP to check
hours: Lookback window in hours (default: 24)
Returns:
Number of recent signups from this IP
"""
try:
cutoff = datetime.utcnow() - timedelta(hours=hours)
query = (
self.db.collection(USERS_COLLECTION)
.where(filter=FieldFilter("signup_ip", "==", ip_address))
.where(filter=FieldFilter("created_at", ">=", cutoff))
)
count = 0
for _ in query.stream():
count += 1
return count
except Exception:
logger.exception(f"Error counting signups from IP {ip_address}")
# On error, don't block — return 0 to allow the request
return 0
def count_recent_signups_from_fingerprint(
self, device_fingerprint: str, hours: int = 24
) -> int:
"""
Count new user accounts created with this device fingerprint in the last N hours.
Args:
device_fingerprint: Browser fingerprint to check
hours: Lookback window in hours (default: 24)
Returns:
Number of recent signups from this fingerprint
"""
try:
cutoff = datetime.utcnow() - timedelta(hours=hours)
query = (
self.db.collection(USERS_COLLECTION)
.where(filter=FieldFilter("device_fingerprint", "==", device_fingerprint))
.where(filter=FieldFilter("created_at", ">=", cutoff))
)
count = 0
for _ in query.stream():
count += 1
return count
except Exception:
logger.exception(f"Error counting signups from fingerprint")
return 0
def is_signup_rate_limited(
self, ip_address: Optional[str] = None, device_fingerprint: Optional[str] = None
) -> bool:
"""
Check if a signup should be rate limited based on IP and/or fingerprint.
Either signal hitting the limit triggers a block.
"""
if ip_address:
ip_count = self.count_recent_signups_from_ip(
ip_address, hours=self.SIGNUP_RATE_LIMIT_HOURS
)
if ip_count >= self.MAX_SIGNUPS_PER_IP:
return True
if device_fingerprint:
fp_count = self.count_recent_signups_from_fingerprint(
device_fingerprint, hours=self.SIGNUP_RATE_LIMIT_HOURS
)
if fp_count >= self.MAX_SIGNUPS_PER_IP:
return True
return False
# =========================================================================
# Anti-Abuse Investigation
# =========================================================================
def find_users_by_signup_ip(self, ip_address: str) -> List[User]:
"""Find all users who signed up from this IP address."""
try:
query = (
self.db.collection(USERS_COLLECTION)
.where(filter=FieldFilter("signup_ip", "==", ip_address))
.order_by("created_at", direction=firestore.Query.DESCENDING)
.limit(50)
)
return [User(**doc.to_dict()) for doc in query.stream()]
except Exception:
logger.exception(f"Error finding users by IP {ip_address}")
return []
def find_users_by_fingerprint(self, device_fingerprint: str) -> List[User]:
"""Find all users who signed up with this device fingerprint."""
try:
query = (
self.db.collection(USERS_COLLECTION)
.where(filter=FieldFilter("device_fingerprint", "==", device_fingerprint))
.order_by("created_at", direction=firestore.Query.DESCENDING)
.limit(50)
)
return [User(**doc.to_dict()) for doc in query.stream()]
except Exception:
logger.exception(f"Error finding users by fingerprint")
return []
def find_suspicious_accounts(self, min_jobs: int = 2, max_spend: int = 0) -> List[User]:
"""
Find accounts with many jobs but no spend (potential free-credit abusers).
Args:
min_jobs: Minimum total_jobs_created to flag (default: 2)
max_spend: Maximum total_spent in cents to include (default: 0 = free only)
"""
from backend.utils.test_data import is_internal_or_test_email
try:
query = (
self.db.collection(USERS_COLLECTION)
.where(filter=FieldFilter("total_jobs_created", ">=", min_jobs))
.order_by("total_jobs_created", direction=firestore.Query.DESCENDING)
.limit(100)
)
users = []
for doc in query.stream():
user = User(**doc.to_dict())
if user.total_spent <= max_spend and not is_internal_or_test_email(user.email):
users.append(user)
return users
except Exception:
logger.exception("Error finding suspicious accounts")
return []
def find_related_accounts(self, email: str) -> dict:
"""
Given a user email, find all accounts that share the same signup IP
or device fingerprint. Returns a dict with grouped results.
"""
user = self.get_user(email)
if not user:
return {"user": None, "by_ip": [], "by_fingerprint": []}
by_ip = []
if user.signup_ip:
by_ip = [
u for u in self.find_users_by_signup_ip(user.signup_ip)
if u.email != user.email
]
by_fingerprint = []
if user.device_fingerprint:
by_fingerprint = [
u for u in self.find_users_by_fingerprint(user.device_fingerprint)
if u.email != user.email
]
return {
"user": user,
"by_ip": by_ip,
"by_fingerprint": by_fingerprint,
}
def find_account_correlations(self) -> dict:
"""
Find clusters of accounts sharing the same fingerprint or signup IP.
Returns groups of 2+ accounts that share the same identifier,
sorted by group size (largest first). Excludes internal/test accounts.
"""
from collections import defaultdict
from backend.utils.test_data import is_internal_or_test_email
# Known bad IPs (load balancer, metadata, placeholder) that don't represent real client IPs
BAD_IPS = {"169.254.169.126", "unknown", ""}
try:
all_users = []
for doc in self.db.collection(USERS_COLLECTION).stream():
user = User(**doc.to_dict())
if not is_internal_or_test_email(user.email):
all_users.append(user)
# Group by fingerprint
fp_groups: dict[str, list[User]] = defaultdict(list)
for user in all_users:
if user.device_fingerprint:
fp_groups[user.device_fingerprint].append(user)
# Group by IP (exclude known bad IPs)
ip_groups: dict[str, list[User]] = defaultdict(list)
for user in all_users:
if user.signup_ip and user.signup_ip not in BAD_IPS:
ip_groups[user.signup_ip].append(user)
# Collect user agents from latest sessions for users in clusters
cluster_emails = set()
for fp, users in fp_groups.items():
if len(users) >= 2:
cluster_emails.update(u.email for u in users)
for ip, users in ip_groups.items():
if len(users) >= 2:
cluster_emails.update(u.email for u in users)
user_agents: dict[str, str] = {}
for email in cluster_emails:
try:
sessions = (
self.db.collection(SESSIONS_COLLECTION)
.where(filter=FieldFilter("user_email", "==", email))
.order_by("created_at", direction=firestore.Query.DESCENDING)
.limit(1)
.stream()
)
for session_doc in sessions:
session_data = session_doc.to_dict()
ua = session_data.get("user_agent")
if ua:
user_agents[email] = ua
except Exception:
pass # Skip if session lookup fails
# Build response - only clusters with 2+ users
fingerprint_clusters = []
for fp, users in fp_groups.items():
if len(users) >= 2:
fingerprint_clusters.append({
"fingerprint": fp,
"users": sorted(users, key=lambda u: u.created_at or "", reverse=True),
})
fingerprint_clusters.sort(key=lambda c: len(c["users"]), reverse=True)
ip_clusters = []
for ip, users in ip_groups.items():
if len(users) >= 2:
ip_clusters.append({
"ip": ip,
"users": sorted(users, key=lambda u: u.created_at or "", reverse=True),
})
ip_clusters.sort(key=lambda c: len(c["users"]), reverse=True)
return {
"fingerprint_clusters": fingerprint_clusters,
"ip_clusters": ip_clusters,
"user_agents": user_agents,
}
except Exception:
logger.exception("Error finding account correlations")
return {"fingerprint_clusters": [], "ip_clusters": [], "user_agents": {}}
# =========================================================================
# Session Management
# =========================================================================
@@ -782,8 +331,7 @@ class UserService:
user_email: str,
ip_address: Optional[str] = None,
user_agent: Optional[str] = None,
tenant_id: Optional[str] = None,
device_fingerprint: Optional[str] = None,
tenant_id: Optional[str] = None
) -> Session:
"""
Create a new session for an authenticated user.
@@ -793,7 +341,6 @@ class UserService:
ip_address: Client IP for auditing
user_agent: Client user agent for auditing
tenant_id: Tenant ID for white-label portals (None = default Nomad Karaoke)
device_fingerprint: Browser fingerprint for cross-account correlation
"""
token = secrets.token_urlsafe(32)
token_prefix = token[:12]
@@ -805,7 +352,6 @@ class UserService:
ip_address=ip_address,
user_agent=user_agent,
tenant_id=tenant_id,
device_fingerprint=device_fingerprint,
)
# Serialize and write to Firestore
@@ -1000,7 +546,7 @@ class UserService:
'amount': amount,
'processed_at': datetime.utcnow()
})
except google_exceptions.AlreadyExists:
except Exception:
# Document already exists - this session was already processed
logger.info(f"Stripe session {stripe_session_id} already processed (idempotent skip)")
return False, 0, "Session already processed"
@@ -1222,62 +768,15 @@ class UserService:
logger.exception("Error enabling user")
return False
def delete_user(self, email: str, admin_email: str) -> bool:
"""
Permanently delete a user and all their associated data.
Deletes: user document, sessions, magic links.
Does NOT delete jobs (they remain as historical records).
Args:
email: Email of the user to delete
admin_email: Email/ID of the admin performing the action
Returns:
True if user was deleted, False if user not found
Raises:
ValueError: If trying to delete an admin user
"""
try:
email = email.lower()
user = self.get_user(email)
if not user:
return False
if user.role == UserRole.ADMIN:
raise ValueError("Cannot delete admin users")
# Revoke all active sessions
self.revoke_all_sessions(email)
# Delete magic links for the user
magic_links = self.db.collection(MAGIC_LINKS_COLLECTION).where(
filter=FieldFilter('email', '==', email)
).stream()
for doc in magic_links:
doc.reference.delete()
# Delete the user document
self.db.collection(USERS_COLLECTION).document(email).delete()
logger.info(f"Admin {admin_email} permanently deleted user {email}")
return True
except ValueError:
raise
except Exception:
logger.exception(f"Error deleting user {email}")
return False
def increment_jobs_completed(self, email: str) -> bool:
"""Increment the completed jobs counter for a user using atomic increment."""
try:
doc_ref = self.get_user_doc_ref(email)
if not doc_ref:
# Check user exists first
if not self.get_user(email):
return False
# Use atomic increment to prevent race conditions
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
doc_ref.update({
'total_jobs_completed': Increment(1),
'updated_at': datetime.utcnow()
@@ -1,300 +0,0 @@
"""
Visibility change service for completed jobs.
Handles changing a job's visibility between public and private:
- Public -> Private: redistribute existing finals to private destination (fast, ~1-2 min)
- Private -> Public: reset styles, regenerate screens, re-render, re-encode (slow, ~15-30 min)
"""
import logging
from datetime import datetime, timezone
from typing import Optional
from google.cloud.firestore_v1 import DELETE_FIELD, ArrayUnion
from backend.models.job import JobStatus
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
logger = logging.getLogger(__name__)
class VisibilityChangeService:
"""Service for changing job visibility after completion."""
def __init__(self, job_manager: Optional[JobManager] = None):
self.job_manager = job_manager or JobManager()
def validate_change(self, job, user_email: str, is_admin: bool, target_visibility: str) -> Optional[str]:
"""Validate a visibility change request. Returns error message or None if valid."""
if not job:
return "Job not found"
if job.status != JobStatus.COMPLETE.value and job.status != "complete":
return f"Can only change visibility on completed jobs. Current status: {job.status}"
if getattr(job, 'tenant_id', None):
return "Cannot change visibility on tenant jobs"
current_is_private = getattr(job, 'is_private', False)
target_is_private = target_visibility == "private"
if current_is_private == target_is_private:
current = "private" if current_is_private else "public"
return f"Job is already {current}"
# Check user owns the job or is admin
if not is_admin and job.user_email != user_email:
return "You can only change visibility on your own jobs"
# Check for in-progress visibility change
state_data = job.state_data or {}
if state_data.get('visibility_change_in_progress'):
return "A visibility change is already in progress for this job"
return None
async def change_to_private(self, job_id: str, job, user_email: str) -> dict:
"""
Change a public job to private (fast path).
Deletes distributed outputs (YouTube/Dropbox/GDrive) but keeps GCS finals,
then redistributes to private destination.
"""
logger.info(f"[job:{job_id}] Starting visibility change: public -> private (by {user_email})")
db = self.job_manager.firestore.db
job_ref = db.collection("jobs").document(job_id)
# Set guard flag
job_ref.update({
"state_data.visibility_change_in_progress": True,
"updated_at": datetime.now(timezone.utc),
})
try:
# Step 1: Delete distributed outputs (keep GCS finals)
await self._delete_distributed_outputs(job_id, job, keep_gcs_finals=True)
# Step 2: Update job to private
job_ref.update({
"is_private": True,
"updated_at": datetime.now(timezone.utc),
"timeline": ArrayUnion([{
"status": "complete",
"timestamp": datetime.now(timezone.utc).isoformat(),
"message": f"Visibility changed to private by {user_email}",
}]),
})
# Step 3: Trigger redistribution via video worker
from backend.workers.video_worker import redistribute_video
success = await redistribute_video(job_id)
if not success:
raise RuntimeError("Redistribution failed - check worker logs")
# Clear guard flag (redistribution updates state_data with new brand_code etc.)
job_ref.update({
"state_data.visibility_change_in_progress": DELETE_FIELD,
})
logger.info(f"[job:{job_id}] Visibility change to private complete")
return {
"status": "success",
"message": "Job changed to private. Outputs redistributed to private destination.",
"reprocessing_required": False,
}
except Exception as e:
logger.error(f"[job:{job_id}] Visibility change to private failed: {e}", exc_info=True)
# Clear guard flag on failure
job_ref.update({
"state_data.visibility_change_in_progress": DELETE_FIELD,
})
raise
async def change_to_public(self, job_id: str, job, user_email: str) -> dict:
"""
Change a private job to public (slow path).
Resets custom styles, deletes all outputs/screens/rendered video,
and triggers full re-processing from screens worker onward.
"""
logger.info(f"[job:{job_id}] Starting visibility change: private -> public (by {user_email})")
db = self.job_manager.firestore.db
job_ref = db.collection("jobs").document(job_id)
storage = StorageService()
# Step 1: Delete distributed outputs + GCS finals
await self._delete_distributed_outputs(job_id, job, keep_gcs_finals=False)
# Step 2: Delete screens and rendered video from GCS
for path in [
f"jobs/{job_id}/screens/title.mov",
f"jobs/{job_id}/screens/title.jpg",
f"jobs/{job_id}/screens/title.png",
f"jobs/{job_id}/screens/end.mov",
f"jobs/{job_id}/screens/end.jpg",
f"jobs/{job_id}/screens/end.png",
f"jobs/{job_id}/videos/with_vocals.mkv",
f"jobs/{job_id}/videos/with_vocals.mov",
]:
try:
storage.delete_file(path)
except Exception:
pass
# Delete custom style files from GCS
try:
style_prefix = f"jobs/{job_id}/style/"
storage.delete_folder(style_prefix)
except Exception:
pass
# Step 3: Prepare nomad theme style files for the job
from backend.api.routes.file_upload import _prepare_theme_for_job
style_params_path, style_assets_dict, _youtube_desc = _prepare_theme_for_job(
job_id=job_id, theme_id="nomad"
)
# Step 4: Reset styles and update job fields
# Step 5: Set status to LYRICS_COMPLETE with regen_restore_status
# Step 6: Clear progress keys and set guard flag
update_payload = {
"is_private": False,
"theme_id": "nomad",
"color_overrides": {},
"style_assets": style_assets_dict,
"style_params_gcs_path": style_params_path,
"status": JobStatus.LYRICS_COMPLETE.value,
"state_data.visibility_change_in_progress": True,
"state_data.regen_restore_status": "review_complete",
"state_data.audio_complete": True,
"state_data.lyrics_complete": True,
"state_data.screens_progress": DELETE_FIELD,
"state_data.render_progress": DELETE_FIELD,
"state_data.video_progress": DELETE_FIELD,
"state_data.encoding_progress": DELETE_FIELD,
"outputs_deleted_at": None,
"outputs_deleted_by": None,
"updated_at": datetime.now(timezone.utc),
"timeline": ArrayUnion([{
"status": "lyrics_complete",
"timestamp": datetime.now(timezone.utc).isoformat(),
"message": f"Visibility change to public initiated by {user_email}. "
"Re-processing from screens generation.",
}]),
}
job_ref.update(update_payload)
# Step 6: Trigger screens worker
from backend.services.worker_service import get_worker_service
worker_service = get_worker_service()
triggered = await worker_service.trigger_screens_worker(job_id)
if not triggered:
logger.error(f"[job:{job_id}] Failed to trigger screens worker for visibility change")
# Restore to complete status on failure
job_ref.update({
"status": "complete",
"is_private": True,
"state_data.visibility_change_in_progress": DELETE_FIELD,
"state_data.regen_restore_status": DELETE_FIELD,
})
raise RuntimeError("Failed to trigger screens worker")
logger.info(f"[job:{job_id}] Visibility change to public initiated, screens worker triggered")
return {
"status": "processing",
"message": "Visibility change to public started. The video will be re-rendered "
"with default branding and published publicly. This takes ~15-30 minutes.",
"reprocessing_required": True,
}
async def _delete_distributed_outputs(self, job_id: str, job, keep_gcs_finals: bool = False):
"""Delete distributed outputs (YouTube, Dropbox, GDrive) and recycle brand code."""
import re
state_data = job.state_data or {}
# Delete YouTube video
youtube_url = state_data.get('youtube_url')
if youtube_url:
try:
video_id_match = re.search(r'(?:youtu\.be/|youtube\.com/watch\?v=)([^&\s]+)', youtube_url)
if video_id_match:
video_id = video_id_match.group(1)
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
from backend.services.youtube_service import get_youtube_service
youtube_service = get_youtube_service()
if youtube_service.is_configured:
finalise = KaraokeFinalise(
dry_run=False, non_interactive=True,
user_youtube_credentials=youtube_service.get_credentials_dict()
)
finalise.delete_youtube_video(video_id)
logger.info(f"[job:{job_id}] Deleted YouTube video {video_id}")
except Exception as e:
logger.warning(f"[job:{job_id}] Failed to delete YouTube video: {e}")
# Delete Dropbox folder
brand_code = state_data.get('brand_code')
dropbox_path = getattr(job, 'dropbox_path', None)
if brand_code and dropbox_path:
try:
from backend.services.dropbox_service import get_dropbox_service
dropbox = get_dropbox_service()
if dropbox.is_configured:
from karaoke_gen.utils import sanitize_filename
safe_artist = sanitize_filename(job.artist) if job.artist else "Unknown"
safe_title = sanitize_filename(job.title) if job.title else "Unknown"
folder_name = f"{brand_code} - {safe_artist} - {safe_title}"
full_path = f"{dropbox_path}/{folder_name}"
dropbox.delete_folder(full_path)
logger.info(f"[job:{job_id}] Deleted Dropbox folder {full_path}")
except Exception as e:
logger.warning(f"[job:{job_id}] Failed to delete Dropbox folder: {e}")
# Delete Google Drive files
gdrive_files = state_data.get('gdrive_files')
if gdrive_files:
try:
from backend.services.gdrive_service import get_gdrive_service
gdrive = get_gdrive_service()
if gdrive.is_configured:
file_ids = list(gdrive_files.values()) if isinstance(gdrive_files, dict) else []
gdrive.delete_files(file_ids)
logger.info(f"[job:{job_id}] Deleted Google Drive files")
except Exception as e:
logger.warning(f"[job:{job_id}] Failed to delete Google Drive files: {e}")
# Delete GCS finals if requested
if not keep_gcs_finals:
try:
storage = StorageService()
deleted_count = storage.delete_folder(f"jobs/{job_id}/finals/")
if deleted_count > 0:
logger.info(f"[job:{job_id}] Deleted {deleted_count} GCS finals files")
except Exception as e:
logger.warning(f"[job:{job_id}] Failed to delete GCS finals: {e}")
# Recycle brand code
if brand_code:
try:
from backend.services.brand_code_service import BrandCodeService, get_brand_code_service
prefix, number = BrandCodeService.parse_brand_code(brand_code)
get_brand_code_service().recycle_brand_code(prefix, number)
logger.info(f"[job:{job_id}] Recycled brand code {brand_code}")
except Exception as e:
logger.warning(f"[job:{job_id}] Failed to recycle brand code: {e}")
# Clear distribution state_data
db = self.job_manager.firestore.db
job_ref = db.collection("jobs").document(job_id)
update = {}
for key in ['youtube_url', 'brand_code', 'dropbox_link', 'gdrive_files']:
if key in state_data:
update[f"state_data.{key}"] = DELETE_FIELD
if update:
job_ref.update(update)
+7 -145
View File
@@ -38,7 +38,6 @@ WORKER_QUEUES = {
"render-video": "render-worker-queue",
"video": "video-worker-queue",
"idle-reminder": "idle-reminder-queue", # For delayed idle reminder checks
"gdrive-validation": "gdrive-validation-queue", # For delayed post-job validation
}
# Dispatch deadlines for each worker type (in seconds)
@@ -52,15 +51,10 @@ WORKER_DISPATCH_DEADLINES = {
"render-video": 1800, # 30 min - Video encoding can be slow
"video": 1800, # 30 min - Video encoding can be slow
"idle-reminder": 60, # 1 min - Quick check and potential email send
"gdrive-validation": 120, # 2 min - Calls Cloud Function which validates GDrive
}
# Default delay for idle reminders (seconds)
IDLE_REMINDER_DELAY_SECONDS = 2 * 60 # 2 minutes
# Default delay for post-job GDrive validation (seconds)
# Gives E2E tests time to verify output and clean up before validator runs
GDRIVE_VALIDATION_DELAY_SECONDS = 5 * 60 # 5 minutes
IDLE_REMINDER_DELAY_SECONDS = 5 * 60 # 5 minutes
class WorkerService:
@@ -361,35 +355,19 @@ class WorkerService:
# Convenience methods for specific workers
# These provide a cleaner API and better IDE autocomplete
async def trigger_audio_download_worker(self, job_id: str) -> bool:
"""
Trigger audio download worker.
Downloads audio from the selected source (Spotify, YouTube, RED, OPS)
and then triggers audio separation + lyrics workers.
Uses Cloud Run Jobs to avoid instance termination during long-running
downloads. This replaces the BackgroundTasks approach that caused
jobs to get stuck when Cloud Run scaled down instances.
"""
return await self._trigger_worker_cloud_run_job(
job_id=job_id,
cloud_run_job_name="audio-download-job",
worker_module="audio_download_worker",
)
async def trigger_audio_worker(self, job_id: str) -> bool:
"""
Trigger audio separation worker.
Uses Cloud Run Jobs with L4 GPU in us-east4 for direct audio separation.
Always uses Cloud Run Jobs to avoid instance termination during long-running
Modal API calls. The BackgroundTasks approach caused Cloud Run to terminate
instances mid-processing when no HTTP requests were active.
"""
return await self._trigger_worker_cloud_run_job(
job_id=job_id,
cloud_run_job_name="audio-separation-job",
worker_module="audio_worker",
location="us-east4", # GPU job runs in us-east4 (L4 quota)
)
async def trigger_lyrics_worker(self, job_id: str) -> bool:
@@ -411,40 +389,14 @@ class WorkerService:
"""Trigger screen generation worker."""
return await self.trigger_worker("screens", job_id)
def _warmup_encoding_worker(self, job_id: str) -> None:
"""Fire-and-forget warmup of the encoding worker VM.
Called before dispatching video/render workers so the GCE encoding
VM starts booting while the worker downloads files and prepares.
Failures are logged but never block worker dispatch.
"""
try:
from backend.services.encoding_worker_manager import EncodingWorkerManager
from google.cloud import compute_v1, firestore
db = firestore.Client(project=self.settings.google_cloud_project)
compute_client = compute_v1.InstancesClient()
manager = EncodingWorkerManager(
db=db,
compute_client=compute_client,
project_id=self.settings.google_cloud_project,
)
result = manager.ensure_primary_running()
if result["started"]:
logger.info(f"[job:{job_id}] Encoding worker warmup: started VM {result['vm_name']}")
else:
logger.info(f"[job:{job_id}] Encoding worker warmup: VM {result['vm_name']} already running")
except Exception as e:
logger.warning(f"[job:{job_id}] Encoding worker warmup failed (non-fatal): {e}")
async def trigger_video_worker(self, job_id: str) -> bool:
"""
Trigger video generation worker.
When USE_CLOUD_RUN_JOBS_FOR_VIDEO=true and ENABLE_CLOUD_TASKS=true,
uses Cloud Run Jobs for execution (supports >30 min encoding).
Otherwise, uses Cloud Tasks or direct HTTP.
"""
self._warmup_encoding_worker(job_id)
if self._use_cloud_tasks and self.settings.use_cloud_run_jobs_for_video:
return await self._trigger_cloud_run_job(job_id)
return await self.trigger_worker("video", job_id)
@@ -473,7 +425,6 @@ class WorkerService:
job_id: str,
cloud_run_job_name: str,
worker_module: str,
location: str | None = None,
) -> bool:
"""
Trigger a worker as a Cloud Run Job.
@@ -486,7 +437,6 @@ class WorkerService:
job_id: Job ID to process
cloud_run_job_name: Name of the Cloud Run Job (e.g., "lyrics-transcription-job")
worker_module: Worker module name (e.g., "lyrics_worker")
location: GCP region for the job (defaults to settings.gcp_region)
Returns:
True if job was triggered successfully, False otherwise
@@ -499,7 +449,7 @@ class WorkerService:
logger.error("GOOGLE_CLOUD_PROJECT not set, cannot trigger Cloud Run Job")
return False
location = location or self.settings.gcp_region
location = self.settings.gcp_region
job_name = f"projects/{project}/locations/{location}/jobs/{cloud_run_job_name}"
# Create Cloud Run Jobs client
@@ -537,7 +487,6 @@ class WorkerService:
async def trigger_render_video_worker(self, job_id: str) -> bool:
"""Trigger render video worker (post-review)."""
self._warmup_encoding_worker(job_id)
return await self.trigger_worker("render-video", job_id)
async def schedule_idle_reminder(
@@ -633,93 +582,6 @@ class WorkerService:
)
return False
async def schedule_gdrive_validation(
self,
delay_seconds: int = GDRIVE_VALIDATION_DELAY_SECONDS
) -> bool:
"""
Schedule a delayed GDrive validation check.
Fires after the specified delay so that E2E test cleanup completes
before the validator runs, avoiding false "all clear" reports.
Args:
delay_seconds: Delay before executing the validation (default: 5 minutes)
Returns:
True if task was scheduled successfully, False otherwise
"""
if not self._use_cloud_tasks:
logger.info(
f"GDrive validation not scheduled (Cloud Tasks disabled). "
f"Would have fired in {delay_seconds}s."
)
return True
try:
from google.cloud import tasks_v2
from google.protobuf import timestamp_pb2
import time
queue_name = WORKER_QUEUES.get("gdrive-validation")
if not queue_name:
logger.error("GDrive validation queue not configured")
return False
project = self.settings.google_cloud_project
if not project:
logger.error("GOOGLE_CLOUD_PROJECT not set")
return False
location = self.settings.gcp_region
# Build queue path
parent = self.tasks_client.queue_path(project, location, queue_name)
# Build headers
headers = {
"Content-Type": "application/json",
}
if self._admin_token:
headers["X-Admin-Token"] = self._admin_token
headers = inject_trace_context(headers)
# Calculate schedule time
schedule_time = timestamp_pb2.Timestamp()
schedule_time.FromSeconds(int(time.time()) + delay_seconds)
dispatch_deadline_seconds = WORKER_DISPATCH_DEADLINES.get("gdrive-validation", 120)
# Build task payload
task = {
"http_request": {
"http_method": tasks_v2.HttpMethod.POST,
"url": f"{self._base_url}/api/internal/trigger-gdrive-validation",
"headers": headers,
"body": b"{}",
"oidc_token": {
"service_account_email": f"karaoke-backend@{project}.iam.gserviceaccount.com",
},
},
"schedule_time": schedule_time,
"dispatch_deadline": duration_pb2.Duration(seconds=dispatch_deadline_seconds),
}
response = self.tasks_client.create_task(parent=parent, task=task)
logger.info(
f"Scheduled GDrive validation in {delay_seconds}s: {response.name}"
)
return True
except Exception as e:
logger.error(
f"Failed to schedule GDrive validation: {e}",
exc_info=True
)
return False
# Global worker service instance
_worker_service: Optional[WorkerService] = None
Loaded 100 of 1254 files, more files were not shown because too many files have changed in this diff. Show more