Compare commits

..
Author SHA1 Message Date
Andrew BeveridgeandClaude Opus 4.5 275fc909f6 fix: E2E test issues - add Firestore index and fix job endpoint URL
- Add missing Firestore composite index for customer_email + made_for_you queries
  (required for test-webhook made-for-you order processing)
- Fix getJobDetails helper to use /api/jobs/{job_id} instead of /api/admin/jobs/{job_id}
  (the admin route doesn't have a GET method)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-29 21:52:34 -08:00
2186 changed files with 18010 additions and 510339 deletions

No files matched your search

@@ -1,109 +0,0 @@
# Plan: Admin Feedback Visibility
**Created:** 2026-03-23
**Branch:** feat/sess-20260323-1454-feedback-email-debug
**Status:** Draft
## Overview
User feedback submissions are stored in Firestore (`user_feedback` collection) but are completely invisible to the admin. The admin panel only shows a boolean "Feedback Submitted: Yes/No" badge per user. This plan adds:
1. **Admin UI page** to browse and review all submitted feedback
2. **Email notification** to admin@nomadkaraoke.com on each new submission
## Requirements
- [ ] Admin page at `/admin/feedback` listing all feedback with ratings, text, timestamps, and user info
- [ ] Sortable by date (newest first by default) and filterable by search (email)
- [ ] Star ratings rendered visually (not just numbers)
- [ ] Email sent to admin@nomadkaraoke.com immediately when feedback is submitted
- [ ] Email includes all feedback content (ratings, text fields, user email)
- [ ] Test data filtering works (exclude_test toggle)
- [ ] Sidebar navigation includes new "Feedback" item
## Technical Approach
### Admin API Endpoint
Add `GET /api/admin/feedback` to `backend/api/routes/admin.py`. Follows the standard admin endpoint pattern: `require_admin` dependency, pagination with `limit`/`offset`/`search`/`exclude_test`, returns Pydantic response model. Queries `user_feedback` Firestore collection.
### Admin UI Page
New page at `frontend/app/admin/feedback/page.tsx`. Follows existing admin page patterns (Card, Table, Badge, pagination, search input, loading states). Uses `adminApi.listFeedback()` to fetch data.
### Email Notification
Add a new method to `EmailService` (or call `send_email` directly) from the `submit_user_feedback` endpoint in `users.py`. Uses the existing SendGrid infrastructure — no new template service templates needed since this is an internal admin notification (simple HTML table with the feedback data).
## Implementation Steps
### Step 1: Backend — Admin feedback list endpoint
1. [ ] Add `AdminFeedbackItem` and `AdminFeedbackListResponse` Pydantic models to `admin.py`
2. [ ] Add `GET /api/admin/feedback` endpoint with:
- Pagination: `limit=50`, `offset=0`
- Search by user email
- `exclude_test` filter (reuse `is_test_email`)
- Sort by `created_at` descending
- Return: items, total, offset, limit, has_more
3. [ ] Add aggregate stats in response: average ratings, total count
### Step 2: Backend — Email notification on submission
4. [ ] Add `send_feedback_notification` method to `EmailService` that sends an HTML email to `admin@nomadkaraoke.com` containing:
- User email
- All 4 star ratings
- All text fields
- Would recommend / would use again
- Submitted timestamp
- Link to admin feedback page
5. [ ] Call `send_feedback_notification` from `submit_user_feedback` in `users.py` (fire-and-forget, don't block the user response on email success)
### Step 3: Frontend — Admin API client
6. [ ] Add `listFeedback` method to `adminApi` in `frontend/lib/api.ts`
7. [ ] Add TypeScript types for `AdminFeedbackItem` and `AdminFeedbackListResponse`
### Step 4: Frontend — Admin feedback page
8. [ ] Create `frontend/app/admin/feedback/page.tsx` with:
- Table: user email, date, overall rating (stars), ease of use, lyrics accuracy, correction experience, would recommend, would use again
- Expandable rows or click-to-detail for text fields (what went well, what could improve, additional comments)
- Search by email
- Pagination (50 per page)
- Refresh button
- Test data toggle (via `useAdminSettings`)
9. [ ] Add "Feedback" item to admin sidebar in `frontend/components/admin/admin-sidebar.tsx` with `MessageSquare` icon
### Step 5: Testing
10. [ ] Backend unit test for `GET /api/admin/feedback` endpoint (pagination, search, exclude_test)
11. [ ] Backend unit test for `send_feedback_notification` email method
12. [ ] Frontend component test for feedback page (render, empty state)
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/api/routes/admin.py` | Modify | Add feedback list endpoint + response models |
| `backend/api/routes/users.py` | Modify | Add email notification call in `submit_user_feedback` |
| `backend/services/email_service.py` | Modify | Add `send_feedback_notification` method |
| `frontend/lib/api.ts` | Modify | Add `adminApi.listFeedback()` method + types |
| `frontend/app/admin/feedback/page.tsx` | Create | Admin feedback review page |
| `frontend/components/admin/admin-sidebar.tsx` | Modify | Add "Feedback" nav item |
| `backend/tests/test_admin_feedback.py` | Create | Tests for admin feedback endpoint |
| `backend/tests/test_feedback_email.py` | Create | Tests for feedback email notification |
## Testing Strategy
- **Unit tests**: Admin endpoint (pagination, filtering, sorting), email method (content formatting, send call)
- **Integration**: Submit feedback → verify email service called
- **Manual verification**: Check email arrives in admin@nomadkaraoke.com inbox after submission
## Open Questions
- [ ] Should the admin page show feedback from the old `beta_feedback` collection too, or just `user_feedback`? (Recommendation: just `user_feedback` — the beta collection is legacy)
- [ ] Should there be any spam/quality detection flagging? (Some submissions appear to game the system for credits)
## Rollback Plan
- Backend: Remove endpoint from admin.py, revert email call in users.py
- Frontend: Delete feedback page, remove sidebar item
- No data migration needed — Firestore collection unchanged
-120
View File
@@ -1,120 +0,0 @@
# Plan: Admin Lyrics Review Dashboard
**Created:** 2026-03-05
**Branch:** feat/sess-20260305-0002-admin-lyrics-review
**Status:** Draft
## Overview
Add an "Edit Reviews" section to the admin dashboard for reviewing completed lyrics reviews. Three goals:
1. **View what users saw** - Jump into a read-only lyrics review UI showing original vs corrected lyrics
2. **Replay user edits** - Step through edit log entries to see exactly what each user changed
3. **Review raw data** - View human edits data to evaluate quality for AudioShake data sharing
## Requirements
- [ ] New admin sidebar item "Edit Reviews" linking to `/admin/edit-reviews`
- [ ] Backend endpoint to list recent jobs that have completed lyrics review (with edit logs)
- [ ] Backend endpoint to serve immutable correction data for a job (read-only, cannot modify job)
- [ ] Admin page showing list of recently reviewed jobs with key metrics
- [ ] Ability to open the existing LyricsAnalyzer in read-only mode with snapshot data
- [ ] Edit log replay/stepper UI to walk through user changes one by one
- [ ] Raw data viewer for edit logs, annotations, and correction diffs
## Technical Approach
### Backend: 2 new admin endpoints
1. **`GET /api/admin/edit-reviews`** - List jobs with completed reviews
- Query Firestore for jobs with `state_data.last_edit_log_path` set (indicating review was performed)
- Filter by status (completed jobs, or optionally all post-review statuses)
- Return summary: job_id, artist, title, user_email, review_completed_at, edit_count, feedback_count
- Support pagination, search, exclude_test filtering
2. **`GET /api/admin/edit-reviews/{job_id}`** - Get full review snapshot
- Returns immutable copy of:
- Original corrections (`corrections.json` from GCS)
- Updated corrections (`corrections_updated.json` from GCS)
- Edit log (`edit_log_{session_id}.json` from GCS)
- Annotations (`annotations.json` from GCS)
- Job metadata (artist, title, user_email, timestamps)
- Audio URL (signed, for playback in review UI)
- This is a **read-only** endpoint - no job state mutations
- Uses `require_admin` dependency (not `require_review_auth`)
- Does NOT transition job status or modify any state
### Frontend: New admin page + Read-only review mode
1. **`/admin/edit-reviews/page.tsx`** - List page
- Table of recently reviewed jobs
- Columns: Artist/Title, User, Review Date, Edits Made, Feedback Given, Actions
- "View Review" button → opens read-only lyrics review
- "View Edit Log" button → opens edit replay view
- "View Raw Data" button → shows raw JSON
2. **Read-only review viewer** - Reuse LyricsAnalyzer
- Load data from the new admin endpoint (not the regular review endpoint)
- Pass `isReadOnly={true}` and `apiClient={null}` to LyricsAnalyzer
- Header shows "Admin Review - Read Only" indicator
- No submit/save buttons, no state transitions
- Audio playback still works (via signed URL)
3. **Edit log replay** - New component
- Timeline/stepper showing each edit entry
- For each entry: operation type, text before → text after, timestamp
- Step forward/backward through edits
- Show feedback reason if user provided one
- Summary stats: total edits, edits with feedback, correction types breakdown
4. **Raw data viewer** - JSON viewer tabs
- Tab 1: Edit Log (formatted JSON with syntax highlighting)
- Tab 2: Annotations
- Tab 3: Corrections diff (original vs updated)
### Key safety guarantee
The admin review endpoint serves a **snapshot** of the data. It reads from GCS files that were already written. The LyricsAnalyzer receives `apiClient={null}` so it physically cannot call any mutation endpoints. Even if code changes in the future, the admin endpoint itself never writes to Firestore or GCS.
## Implementation Steps
1. [ ] **Backend: Add `GET /api/admin/edit-reviews` endpoint** - List reviewed jobs with edit log metadata
2. [ ] **Backend: Add `GET /api/admin/edit-reviews/{job_id}` endpoint** - Serve full immutable review snapshot
3. [ ] **Frontend: Add admin API client methods** - `adminApi.listEditReviews()`, `adminApi.getEditReview(jobId)`
4. [ ] **Frontend: Create `/admin/edit-reviews/page.tsx`** - List page with table
5. [ ] **Frontend: Add sidebar nav item** - "Edit Reviews" with BookOpen icon
6. [ ] **Frontend: Create read-only review viewer** - Load snapshot data into LyricsAnalyzer with isReadOnly
7. [ ] **Frontend: Create edit log replay component** - Step-through UI for edit entries
8. [ ] **Frontend: Create raw data viewer** - JSON viewer with tabs
9. [ ] **Tests: Backend unit tests** for new admin endpoints
10. [ ] **Tests: Frontend component tests** for new page
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/api/routes/admin.py` | Modify | Add edit-reviews endpoints |
| `frontend/lib/api.ts` | Modify | Add adminApi methods for edit reviews |
| `frontend/app/admin/edit-reviews/page.tsx` | Create | Main edit reviews list page |
| `frontend/components/admin/edit-review-viewer.tsx` | Create | Read-only review wrapper using LyricsAnalyzer |
| `frontend/components/admin/edit-log-replay.tsx` | Create | Edit log stepper/replay component |
| `frontend/components/admin/raw-data-viewer.tsx` | Create | JSON viewer with tabs |
| `frontend/components/admin/admin-sidebar.tsx` | Modify | Add Edit Reviews nav item |
| `backend/tests/test_admin_edit_reviews.py` | Create | Backend tests |
## Testing Strategy
- Unit tests for backend endpoints (mock Firestore + GCS)
- Verify admin auth is required on all new endpoints
- Verify no write operations occur on read-only endpoints
- Frontend component tests for key interactions
## Open Questions
- [ ] Should we show ALL jobs that went through review, or only ones with edit logs? (Recommendation: only those with edit logs, since those are the interesting ones)
- [ ] Do we need audio playback in the admin review viewer? (Recommendation: yes, it helps understand what the user heard)
- [ ] Should the raw data be downloadable? (Nice-to-have, can add later)
## Rollback Plan
All changes are additive (new endpoints, new pages). Removing the sidebar link effectively hides the feature. No existing behavior is modified.
-151
View File
@@ -1,151 +0,0 @@
# Plan: AI-Powered Credit Gating
**Created:** 2026-03-23
**Branch:** feat/sess-20260323-1716-ai-credit-gating
**Status:** Implemented
## Overview
Replace the automatic welcome/feedback credit grants with an AI-powered evaluation step. When a user triggers a credit grant (email verification or feedback submission), instead of immediately granting credits, we collect abuse signals and ask Gemini to make a grant/deny decision. Suspected abusers get a friendly rejection email (CC'd to Andrew) instead of free credits.
## Requirements
- [ ] Welcome credit grants go through AI evaluation before granting
- [ ] Feedback credit grants go through AI evaluation before granting
- [ ] AI evaluates: fingerprint correlations, IP correlations, IP geo info, user agent, signup recency patterns
- [ ] Approved users get credits as before (seamless experience)
- [ ] Rejected users get a friendly-but-firm email explaining the situation, CC'd to andrew@beveridge.uk
- [ ] Rejection email invites them to reply if it's a mistake
- [ ] Decision is logged to Firestore for audit trail
- [ ] Fail-open: if AI evaluation fails, grant credits (don't punish users for our errors)
- [ ] Admin can override decisions via existing admin credit grant endpoint
## Technical Approach
### Architecture
```
User verifies email / submits feedback
↓
Collect abuse signals (sync, fast):
- User's fingerprint + IP
- Other accounts with same fingerprint (from correlations)
- Other accounts with same IP
- IP geolocation info
- Recent signup count from same IP/fingerprint (last 24h)
- User agent string
- For feedback: the feedback content itself
↓
Call Gemini (vertexai/gemini-2.5-pro-preview) with structured prompt:
- Present all signals
- Ask for JSON response: {decision: "grant" | "deny", reasoning: string, confidence: number}
↓
If "grant": grant credits normally, log decision
If "deny": skip credit grant, send rejection email, log decision
If AI call fails: grant credits (fail-open), log error
```
### Key Design Decisions
1. **Two-phase verification**: The magic link verify endpoint returns immediately with session + a `credit_status: "evaluating"` field. The frontend shows a "preparing your account" loading screen, then polls/waits for the evaluation result via a new lightweight endpoint. This keeps the auth flow snappy while the AI evaluation runs in the background of the same request or via a short follow-up call. The frontend then shows either a celebratory "2 free credits!" interstitial or a "sorry, couldn't grant credits" interstitial before navigating to the dashboard.
2. **Fail-open**: If Gemini is down or returns an error, we grant credits. Better to let an occasional abuser through than block legitimate users.
3. **Use google-genai directly**: The project already has `langchain-google-genai` installed. We can use the simpler `google.generativeai` library directly for a focused prompt/response, or use the existing LangChain setup. We'll use `google.generativeai` for simplicity since this is a single-turn evaluation, not an agentic workflow.
4. **Structured JSON output**: Ask Gemini for structured output with decision, reasoning, and confidence to make decisions parseable and auditable.
5. **Log all decisions**: Store every evaluation in a `credit_evaluations` Firestore collection for audit trail and to tune the prompt over time.
## Implementation Steps
### Step 1: Add google-generativeai dependency
- Add `google-generativeai` to pyproject.toml (lightweight, direct Gemini access)
- Verify it works with Vertex AI auth in the nomadkaraoke project
### Step 2: Create credit evaluation service
- New file: `backend/services/credit_evaluation_service.py`
- `CreditEvaluationService` class with:
- `evaluate_credit_grant(email, grant_type, feedback_content=None) -> CreditEvaluation`
- Collects all abuse signals by calling existing services (user_service, ip_geolocation_service)
- Builds prompt with signals
- Calls Gemini for decision
- Logs decision to Firestore
- Returns structured result
### Step 3: Build the Gemini prompt
- System prompt explaining the role: "You are an anti-abuse evaluator for a karaoke video service..."
- Present all signals in a structured format
- Request JSON response: `{decision: "grant"|"deny", reasoning: "...", confidence: 0.0-1.0}`
- Include clear guidelines: same fingerprint across accounts = very suspicious, same IP = somewhat suspicious, etc.
### Step 4: Create rejection email template
- Add `send_credit_denied_email(email, reasoning)` to EmailService
- Friendly-but-firm tone
- CC andrew@beveridge.uk
- Invite reply if it's a mistake
- Use existing email template infrastructure (branding, styling)
### Step 5: Integrate into welcome credit flow
- Modify `grant_welcome_credits_if_eligible()` in user_service.py
- Before granting: call `CreditEvaluationService.evaluate_credit_grant(email, "welcome")`
- If denied: send rejection email, return False
- If error: log warning, grant anyway (fail-open)
### Step 6: Integrate into feedback credit flow
- Modify feedback endpoint in users.py
- Before granting credits: evaluate with feedback content included
- If denied: return different response message, send rejection email
- If error: grant anyway
### Step 7: Add Firestore collection for audit trail
- Collection: `credit_evaluations`
- Document fields: email, grant_type, decision, reasoning, confidence, signals_snapshot, model_used, evaluated_at
- TTL: keep indefinitely (small volume, high audit value)
### Step 8: Tests
- Unit tests for signal collection
- Unit tests for prompt building
- Unit tests for decision parsing (grant/deny/error cases)
- Integration test for the evaluation service with mocked Gemini
- Integration test for welcome credit flow with evaluation
- Integration test for feedback credit flow with evaluation
- Test fail-open behavior
### Step 9: Configuration
- Add env vars: `CREDIT_EVAL_MODEL` (default: "gemini-2.5-pro-preview"), `CREDIT_EVAL_ENABLED` (default: true)
- Add to config.py
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `pyproject.toml` | Modify | Add google-generativeai dependency |
| `backend/config.py` | Modify | Add credit evaluation config vars |
| `backend/services/credit_evaluation_service.py` | Create | New service: collect signals, call Gemini, log decision |
| `backend/services/user_service.py` | Modify | Integrate evaluation into welcome credit grant |
| `backend/api/routes/users.py` | Modify | Integrate evaluation into feedback credit grant |
| `backend/services/email_service.py` | Modify | Add credit denied email template |
| `backend/tests/test_credit_evaluation.py` | Create | Tests for evaluation service |
| `backend/tests/test_anti_abuse.py` | Modify | Update welcome credit tests for evaluation |
| `frontend/app/auth/verify/page.tsx` | Modify | Add "preparing account" loading + credit result interstitial |
| `frontend/lib/api.ts` | Modify | Add credit evaluation status endpoint |
## Testing Strategy
- **Unit tests**: Signal collection, prompt building, response parsing, fail-open behavior
- **Integration tests**: Full evaluation flow with mocked Gemini responses
- **E2E tests**: Manual verification in production with test accounts
- **Edge cases**: Gemini timeout, malformed response, missing signals, first-ever user (no correlations)
## Open Questions
1. **Gemini model**: Using `gemini-3.1-pro-preview` as specified — already available in the nomadkaraoke GCP project.
2. **Latency UX**: Frontend shows a friendly "preparing your account" loading screen while evaluation runs. Then transitions to either a celebratory "2 free credits!" message (with gentle reminder not to abuse) or a "sorry, couldn't grant free credits" message with a buy credits CTA. User clicks through to dashboard from either.
3. **Should we also gate the existing rate limiter?** Currently `is_signup_rate_limited()` silently blocks at 2 accounts per IP/fingerprint per 24h. The AI evaluation is complementary — rate limiter blocks obvious rapid abuse, AI evaluation catches more subtle patterns.
## Rollback Plan
Set `CREDIT_EVAL_ENABLED=false` in Cloud Run env vars → immediate rollback to auto-grant behavior. No data model changes needed. The evaluation service is purely additive.
@@ -1,150 +0,0 @@
# Plan: Block Disposable Email Registration
**Created:** 2026-03-19
**Branch:** feat/sess-20260319-1332-disposable-email-abuse
**Status:** Draft
## Overview
Users are signing up with disposable email addresses (yopmail.com, emailhook.site, etc.) to get unlimited free credits (2 per account). The `EmailValidationService` already exists with 160+ disposable domains but is **never called during signup**. We need to wire it into the magic link flow and show a friendly rejection message.
## Context
- The `EmailValidationService` (`backend/services/email_validation_service.py`) is fully implemented with disposable domain detection, email/IP blocklists, and admin management APIs
- The admin blocklist management routes exist (`backend/api/routes/rate_limits.py`) and work
- The magic link endpoint (`POST /api/users/auth/magic-link` in `backend/api/routes/users.py:133`) **never calls** the validation service
- Currently 1 yopmail user and 1 emailhook.site user in `gen_users`
- Each new account gets 2 free welcome credits automatically
## Requirements
- [x] Reject disposable email domains at the magic link endpoint (backend)
- [x] Return a **specific, distinguishable error** so the frontend can show a friendly message (not the generic "email enumeration" response)
- [x] Show a warm, friendly message in the AuthDialog explaining why and encouraging real email use
- [x] Add `emailhook.site` to the default disposable domains list
- [x] Add backend unit tests for the integration
- [x] Add frontend test coverage for the new error state
## Design Decision: Explicit Rejection vs Silent Failure
The current magic link endpoint returns a generic success message to prevent email enumeration. For disposable emails, we will **explicitly reject** with a 422 status and a specific error message. This is the right trade-off because:
1. We *want* the user to know their disposable email won't work (so they can use a real one)
2. Disposable domain detection is not sensitive information — the blocklist is public knowledge
3. The friendly message serves our goal of converting them to real email users
## Implementation Steps
### Step 1: Backend — Add disposable email check to magic link endpoint
**File:** `backend/api/routes/users.py`
Add the `EmailValidationService` check **after** email lowercasing (line 158) and **before** tenant validation (line 164). Return a 422 with a friendly message.
```python
# Check for disposable email domains
email_validation = get_email_validation_service()
if email_validation.is_disposable_domain(email):
logger.warning(f"Blocked disposable email signup attempt: {_mask_email(email)}")
raise HTTPException(
status_code=422,
detail="disposable_email_not_allowed"
)
```
Use a machine-readable error code (`disposable_email_not_allowed`) so the frontend can match on it and show the custom friendly message. This keeps the backend response lean and puts the UX copy in the frontend where it belongs.
### Step 2: Backend — Also check email and IP blocklists
In the same location, add checks for explicitly blocked emails and IPs:
```python
if email_validation.is_email_blocked(email):
logger.warning(f"Blocked email signup attempt: {_mask_email(email)}")
# Silent reject — return generic success to prevent enumeration
return SendMagicLinkResponse(
status="success",
message="If this email is registered, you will receive a sign-in link shortly."
)
ip_address = http_request.client.host if http_request.client else None
if ip_address and email_validation.is_ip_blocked(ip_address):
logger.warning(f"Blocked IP signup attempt: {ip_address}")
return SendMagicLinkResponse(
status="success",
message="If this email is registered, you will receive a sign-in link shortly."
)
```
Note: Blocked emails/IPs get silent rejection (anti-enumeration). Disposable domains get explicit rejection (we want the user to switch to a real email).
### Step 3: Backend — Add `emailhook.site` to default disposable domains
**File:** `backend/services/email_validation_service.py`
Add `emailhook.site` to `DEFAULT_DISPOSABLE_DOMAINS` set.
### Step 4: Frontend — Handle the disposable email error in AuthDialog
**File:** `frontend/components/auth/AuthDialog.tsx`
The `sendMagicLink` call in `auth.ts` already catches errors and sets `error` from the API response message. The `handleResponse` in `api.ts` throws an `ApiError` with the `detail` string for non-200 responses.
Update `handleSendMagicLink` to detect the `disposable_email_not_allowed` error code and show a friendly multi-line message instead of the terse error string:
```tsx
const success = await sendMagicLink(email.trim().toLowerCase())
if (success) {
setStep("sent")
}
// If error contains our specific code, override with friendly message
// (the error state is already set by the auth store)
```
We need to detect the specific error. The cleanest approach: check the error string in the `displayError` rendering. If it matches `disposable_email_not_allowed`, render a custom friendly block instead of the red error text.
The friendly message:
> **We don't support disposable email addresses**
>
> We'd love to help you create karaoke videos! Please use your regular email address so we can send you a notification when your video is ready.
>
> We promise — no spam, no marketing emails, ever. We only email you about your karaoke videos.
### Step 5: Backend tests
**File:** `backend/tests/test_email_validation_integration.py` (new)
Test cases:
1. Magic link with disposable email returns 422 with `disposable_email_not_allowed`
2. Magic link with legitimate email proceeds normally
3. Magic link with blocked email returns silent success (no actual email sent)
4. Magic link with blocked IP returns silent success
### Step 6: Frontend tests
**File:** `frontend/components/__tests__/AuthDialog.test.tsx` (new or extend existing)
Test cases:
1. Submitting a disposable email shows the friendly rejection message
2. The friendly message contains the key copy (no spam promise, use real email)
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/api/routes/users.py` | Modify | Add email validation checks to magic link endpoint |
| `backend/services/email_validation_service.py` | Modify | Add `emailhook.site` to default domains |
| `frontend/components/auth/AuthDialog.tsx` | Modify | Handle disposable email error with friendly message |
| `backend/tests/test_email_validation_integration.py` | Create | Integration tests for the validation in signup flow |
## Testing Strategy
- **Unit tests:** Validate `is_disposable_domain` catches yopmail, emailhook.site (already covered in existing tests)
- **Integration tests:** New test file verifying the magic link endpoint rejects disposable emails with 422
- **Frontend tests:** Verify the AuthDialog shows friendly message on disposable email error
- **Manual verification:** Try signing up with `test@yopmail.com` on dev server to see the UX
## Rollback Plan
Remove the validation check from `send_magic_link()` in `users.py`. The `EmailValidationService` itself remains unchanged (it's already deployed and used by admin routes).
@@ -1,111 +0,0 @@
# Plan: Credit Refund on Job Deletion and Cancellation
**Created:** 2026-03-04
**Branch:** feat/sess-20260304-1308-investigate-user-credits
**Status:** Implemented
## Overview
When a user deletes or cancels a job, their credit is not refunded. This was discovered when user john2011@sky.com deleted a job 16 seconds after creation (accidental duplicate) and lost the credit. Job *failures* already auto-refund via `mark_job_failed()` → `refund_credit()`, but deletions and cancellations do not.
## Requirements
- [x] Job failures already refund credits (existing behavior, no change needed)
- [ ] Job cancellation refunds 1 credit
- [ ] Job deletion refunds 1 credit (if not already completed)
- [ ] No refund for completed jobs (user got their video)
- [ ] No refund for admin-owned jobs (consistent with failure refund logic)
- [ ] Idempotency: prevent double-refund if a job is cancelled then deleted, or fails then deleted
- [ ] Credit transaction records the reason clearly ("job_cancelled", "job_deleted")
- [ ] Refund failure must not block the cancel/delete operation
## Technical Approach
### Key Insight: Idempotency via Job Field
The current `refund_credit()` has no guard against double-refund for the same job. Since a job can transition through cancelled → deleted, or failed → deleted, we need a way to track that a refund was already issued.
**Approach**: Add a `credit_refunded` boolean field to the Job model. Check it before refunding. This is simpler and more reliable than scanning the user's transaction history.
### What Should and Shouldn't Be Refunded
| Job Status at Time of Action | Delete | Cancel | Notes |
|------------------------------|--------|--------|-------|
| PENDING through pre-COMPLETE | Refund | Refund | Job didn't deliver value |
| COMPLETE / PREP_COMPLETE | No refund | N/A (can't cancel terminal) | User got their output |
| FAILED | No refund (already refunded) | N/A (can't cancel terminal) | Already refunded by fail_job |
| CANCELLED | No refund (already refunded) | N/A (already cancelled) | Already refunded by cancel |
For deletion: we need to read the job status *before* deleting it to decide whether to refund. The delete endpoint already fetches the job for ownership check, so this is straightforward.
### Refund Helper
Extract the refund logic (currently in `mark_job_failed`) into a reusable private method `_refund_credit_for_job()` that:
1. Checks `job.credit_refunded` — skip if already refunded
2. Checks `job.user_email` — skip if no user
3. Checks if admin — skip if admin job
4. Calls `user_service.refund_credit()` with appropriate reason
5. Sets `credit_refunded = True` on the job (for cancel; for delete the job is about to be removed but we set it anyway for the brief window)
6. Wraps everything in try/except so failures don't block the parent operation
## Implementation Steps
1. [ ] **Add `credit_refunded` field to Job model** (`backend/models/job.py`)
- Add `credit_refunded: bool = False` to the Job model
- No migration needed — Firestore is schemaless, defaults to False for existing docs
2. [ ] **Extract refund helper in JobManager** (`backend/services/job_manager.py`)
- Create `_refund_credit_for_job(self, job_id: str, job: Job, reason: str) -> bool`
- Move the refund logic from `mark_job_failed` (lines 821-839) into this helper
- Add `credit_refunded` check at the top
- Update `mark_job_failed` to call the helper with `reason="job_failed"`
3. [ ] **Add refund to `cancel_job()`** (`backend/services/job_manager.py`)
- After setting status to CANCELLED, call `_refund_credit_for_job(job_id, job, reason="job_cancelled")`
- The job object is already fetched at line 1017
4. [ ] **Add refund to `delete_job()`** (`backend/services/job_manager.py`)
- Need to fetch the job (already done when `delete_files=True`, but not always)
- Before deletion, check if job status is NOT in `[COMPLETE, PREP_COMPLETE, FAILED, CANCELLED]`
- If eligible, call `_refund_credit_for_job(job_id, job, reason="job_deleted")`
- For FAILED/CANCELLED: skip because refund was already issued during that transition
- For COMPLETE/PREP_COMPLETE: skip because user received their output
5. [ ] **Update `mark_job_failed` to set `credit_refunded`**
- The helper handles this, but also update the Firestore doc
6. [ ] **Add unit tests** (`backend/tests/test_credit_enforcement.py`)
- Test: cancel_job refunds credit for non-admin user
- Test: cancel_job skips refund for admin jobs
- Test: delete_job refunds credit for non-complete, non-terminal jobs
- Test: delete_job does NOT refund for completed jobs
- Test: delete_job does NOT double-refund for already-failed jobs
- Test: delete_job does NOT double-refund for already-cancelled jobs
- Test: refund failure doesn't block cancel/delete
- Test: credit_refunded flag prevents double refund
7. [ ] **Backfill: Refund john2011@sky.com** (manual one-time)
- Grant 1 credit for the deleted job 40dcca89 via admin API or direct Firestore update
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Add `credit_refunded: bool = False` field |
| `backend/services/job_manager.py` | Modify | Extract `_refund_credit_for_job()` helper, add refund to `cancel_job()` and `delete_job()` |
| `backend/tests/test_credit_enforcement.py` | Modify | Add tests for cancel/delete refund scenarios |
## Testing Strategy
- **Unit tests**: Extend `test_credit_enforcement.py` with new test classes `TestCreditRefundOnJobCancellation` and `TestCreditRefundOnJobDeletion`, following the exact pattern of existing `TestCreditRefundOnJobFailure`
- **Manual verification**: After deploy, cancel and delete test jobs and verify credit balance updates
## Open Questions
- [ ] Should we notify the user (email/in-app) when a credit is refunded? (Suggest: no, keep it simple for now — the credit just appears back in their balance)
## Rollback Plan
- The `credit_refunded` field is additive (defaults to False for old jobs)
- If issues arise, revert the PR — old behavior (no refund on cancel/delete) is restored
- Any refunds already issued are just bonus credits to users, no harm done
-407
View File
@@ -1,407 +0,0 @@
# Plan: Add Retry Logic for Flacfetch Download Failures
**Created:** 2026-02-10
**Branch:** feat/sess-20260210-1158-investigate-job-749141f8
**Status:** Draft
## Overview
Job 749141f8 failed with error: "Audio download failed: Remote download by ID failed: Download by ID request failed:"
Investigation revealed **multiple related issues** when flacfetch network calls fail:
1. **No retry logic** - A single network hiccup causes immediate job failure
2. **Missing source info** - When download fails, source_id/source_name aren't saved to job
3. **Broken retry button** - Can't retry because no source info was persisted
4. **Silent UI failures** - Frontend doesn't show retry error messages to user
### Root Cause
Audio search flow saves selection info to `state_data`, but doesn't save download parameters (source_id, source_name, etc.) to the job document until AFTER download succeeds. If the flacfetch call fails (network error), the job fails with no way to retry.
### Solution
This plan fixes all four issues:
- Add retry logic to flacfetch client (network resilience)
- Persist download info before attempting download (enables retry)
- Update retry endpoint to handle audio search jobs
- Show error messages in frontend when retry fails
## Requirements
### Functional Requirements - Flacfetch Retry Logic
- [x] Retry `download_by_id()` on `httpx.RequestError` (network failures)
- [x] Use exponential backoff spread over ~5 minutes
- [x] Configurable max retries via environment variable
- [x] Log each retry attempt with context
- [x] Do NOT retry on HTTP 4xx errors (client errors - immediate failure)
- [x] Do retry on HTTP 5xx errors (server errors - transient)
### Functional Requirements - Persist Download Info
- [x] Save source_id, source_name, target_file, download_url to job document
- [x] Save BEFORE attempting download (not after)
- [x] Store in top-level fields for easy retry access
- [x] Preserve existing state_data fields (selected_audio_index, etc.)
### Functional Requirements - Retry Endpoint
- [x] Add case for audio search jobs in `/jobs/{id}/retry` endpoint
- [x] Check for saved download params (source_id, source_name)
- [x] Re-download using saved params
- [x] Clear error state before retrying
- [x] Trigger appropriate workers after successful download
### Functional Requirements - Frontend Error Handling
- [x] Show toast notification when retry fails
- [x] Display actual error message from API
- [x] Don't just console.error - make it visible to user
### Non-Functional Requirements
- [x] Minimal code changes (use existing patterns)
- [x] Clear error messages that include retry context
- [x] Backwards compatible (no breaking changes)
- [x] Should not significantly increase total timeout duration
## Technical Approach
### Architecture Decisions
**Option 1: Tenacity library (Recommended)**
- ✅ Industry standard, well-tested
- ✅ Declarative retry configuration
- ✅ Built-in exponential backoff
- ✅ Already used in project (check `poetry.lock`)
- ❌ New dependency if not already present
**Option 2: Manual retry loop**
- ✅ No new dependencies
- ✅ Full control over logic
- ❌ More code to maintain
- ❌ Easy to get wrong (edge cases)
**Decision:** Use **tenacity** if already in dependencies, otherwise implement manual retry loop.
### Retry Strategy
```python
# Retry on network errors only (RequestError)
# Do NOT retry on:
# - HTTP 4xx (client errors like 400, 401, 404)
# - Already completed downloads
# Do retry on:
# - httpx.RequestError (connection timeout, refused, DNS failure)
# - HTTP 5xx (server errors - service temporarily down)
Max retries: 8 (total 9 attempts)
Backoff: Exponential with cap, multiplier=2, min=10s, max=60s
- Attempt 1: immediate
- Attempt 2: wait 10s
- Attempt 3: wait 20s
- Attempt 4: wait 40s
- Attempt 5: wait 60s (capped)
- Attempt 6: wait 60s (capped)
- Attempt 7: wait 60s (capped)
- Attempt 8: wait 60s (capped)
- Attempt 9: wait 60s (capped)
Total max time: ~370s (6+ minutes) of backoff + request timeouts
This gives the flacfetch service time to recover from restarts,
network issues, or temporary VM problems.
```
### Configuration
Add to `backend/config.py`:
```python
# Flacfetch client retry settings
flacfetch_retry_max_attempts: int = int(os.getenv("FLACFETCH_RETRY_MAX_ATTEMPTS", "9")) # Total attempts (1 initial + 8 retries)
flacfetch_retry_min_wait: float = float(os.getenv("FLACFETCH_RETRY_MIN_WAIT", "10.0")) # seconds
flacfetch_retry_max_wait: float = float(os.getenv("FLACFETCH_RETRY_MAX_WAIT", "60.0")) # seconds
```
### Error Message Improvements
Change from:
```
Download by ID request failed:
```
To:
```
Download by ID request failed after 4 attempts: [ConnectTimeout] Connection to http://10.0.0.5:8080 timed out
```
## Implementation Steps
### Phase 1: Check Dependencies
1. [ ] Check if `tenacity` is in `pyproject.toml` dependencies
2. [ ] If not present, add `tenacity = "^8.2.0"` to dependencies
3. [ ] Run `poetry lock` if dependencies changed
### Phase 2: Add Configuration
4. [ ] Add retry configuration to `backend/config.py`
- `flacfetch_retry_max_attempts = 9`
- `flacfetch_retry_min_wait = 10.0`
- `flacfetch_retry_max_wait = 60.0`
### Phase 3: Implement Flacfetch Retry Logic
5. [ ] Import tenacity in `backend/services/flacfetch_client.py`
6. [ ] Create retry decorator with:
- Retry on `httpx.RequestError`
- Retry on `httpx.HTTPStatusError` with 5xx status
- Stop on `httpx.HTTPStatusError` with 4xx status
- Exponential backoff with cap at 60s
- Before-sleep logging callback
7. [ ] Apply decorator to `download_by_id()` method
8. [ ] Improve error messages to include:
- Number of attempts made
- Original exception type and message
### Phase 4: Persist Download Info Before Download
9. [ ] Update `audio_search.py:_download_and_start_processing()`
10. [ ] After getting source info from selected result (line 307-311)
11. [ ] **Save to job document BEFORE download attempt:**
```python
job_manager.update_job(job_id, {
'audio_source_type': 'audio_search',
'source_name': source_name,
'source_id': source_id,
'target_file': target_file,
'download_url': download_url,
})
```
12. [ ] Then proceed with download (existing code)
13. [ ] Add to Job model if needed (check if fields exist)
### Phase 5: Update Retry Endpoint for Audio Search
14. [ ] Update `backend/api/routes/jobs.py:retry_job()`
15. [ ] Add new case before "no input audio" error (line ~1088)
16. [ ] Check if job has saved download params:
```python
elif job.audio_source_type == 'audio_search' and job.source_id:
# Has audio search download info - retry download
```
17. [ ] Re-call audio search download logic with saved params
18. [ ] Or refactor download into reusable function
### Phase 6: Frontend Error Handling
19. [ ] Update `frontend/components/job/JobActions.tsx`
20. [ ] Add toast notification library (check if exists)
21. [ ] In `handleRetry()` catch block, show toast:
```typescript
catch (error) {
const message = error.response?.data?.detail || "Failed to retry job"
toast.error(message)
console.error("Failed to retry job:", error)
}
```
### Phase 7: Testing - Flacfetch Retry
22. [ ] Add unit tests to `backend/tests/test_flacfetch_client.py`
23. [ ] Test retry on RequestError (mock fails N times then succeeds)
24. [ ] Test no retry on 4xx errors
25. [ ] Test retry on 5xx errors
26. [ ] Test max retries exhausted
27. [ ] Verify logging output includes attempt numbers
### Phase 8: Testing - End-to-End
28. [ ] Test audio search job with flacfetch down
29. [ ] Verify source info is saved even when download fails
30. [ ] Verify retry button works after source info saved
31. [ ] Test retry succeeds when flacfetch comes back up
32. [ ] Verify error toast appears when retry fails
### Phase 9: Documentation
33. [ ] Update `backend/services/flacfetch_client.py` docstring
34. [ ] Add to `docs/LESSONS-LEARNED.md`:
- "Always add retries for external service calls"
- "Save operation parameters before attempting operations"
- "Persist enough state to enable retry from any failure point"
35. [ ] Update `docs/ARCHITECTURE.md` if needed
36. [ ] Document the new job fields in API.md
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/config.py` | Modify | Add retry configuration variables |
| `backend/services/flacfetch_client.py` | Modify | Add retry logic to `download_by_id()` |
| `backend/api/routes/audio_search.py` | Modify | Save download info before attempting download |
| `backend/api/routes/jobs.py` | Modify | Add audio search case to retry endpoint |
| `backend/models/job.py` | Modify (maybe) | Add fields: source_name, source_id, target_file, download_url |
| `frontend/components/job/JobActions.tsx` | Modify | Show error toast when retry fails |
| `backend/tests/test_flacfetch_client.py` | Modify | Add retry tests |
| `backend/tests/test_audio_search.py` | Modify | Test download info persistence |
| `backend/tests/test_jobs.py` | Modify | Test retry with audio search jobs |
| `pyproject.toml` | Modify (maybe) | Add tenacity if not present |
| `docs/LESSONS-LEARNED.md` | Modify | Document patterns |
| `docs/API.md` | Modify | Document new job fields |
## Testing Strategy
### Unit Tests (`backend/tests/test_flacfetch_client.py`)
1. **Test retry on RequestError**
```python
@pytest.mark.asyncio
async def test_download_by_id_retries_on_network_error():
"""Should retry 3 times on connection errors."""
client = FlacfetchClient(base_url="http://test", api_key="key")
with patch("httpx.AsyncClient.post") as mock_post:
# Fail 2 times, then succeed
mock_post.side_effect = [
httpx.RequestError("Connection timeout"),
httpx.RequestError("Connection refused"),
Mock(json=lambda: {"download_id": "test123"}, status_code=200),
]
result = await client.download_by_id("YouTube", "abc123")
assert result == "test123"
assert mock_post.call_count == 3
```
2. **Test no retry on 4xx errors**
```python
@pytest.mark.asyncio
async def test_download_by_id_no_retry_on_client_error():
"""Should NOT retry on 400/401/404 errors."""
client = FlacfetchClient(base_url="http://test", api_key="key")
with patch("httpx.AsyncClient.post") as mock_post:
response = Mock(status_code=404, text="Not found")
response.raise_for_status.side_effect = httpx.HTTPStatusError(
"Not found", request=Mock(), response=response
)
mock_post.return_value = response
with pytest.raises(FlacfetchServiceError, match="404"):
await client.download_by_id("YouTube", "abc123")
assert mock_post.call_count == 1 # No retries
```
3. **Test retry on 5xx errors**
```python
@pytest.mark.asyncio
async def test_download_by_id_retries_on_server_error():
"""Should retry on 500/503 errors."""
# Similar to test 1 but with HTTP 503 errors
```
4. **Test max retries exhausted**
```python
@pytest.mark.asyncio
async def test_download_by_id_fails_after_max_retries():
"""Should fail with clear message after max retries."""
# All attempts fail, verify error message includes attempt count
```
### Integration Tests
5. **Test with actual flacfetch service** (if available in test environment)
- Verify retry doesn't break normal operation
- Test with deliberate network issues
### Manual Testing
6. **Simulate network failure**
```bash
# Temporarily stop flacfetch VM
gcloud compute instances stop flacfetch-service --zone=us-central1-a
# Submit a test job
# Observe logs showing retry attempts
# Restart VM during retries
gcloud compute instances start flacfetch-service --zone=us-central1-a
# Job should succeed
```
7. **Test with intentional delays**
- Add sleep in flacfetch API handler
- Verify retries don't cause issues
## Open Questions
- [x] ~~Is tenacity already in dependencies?~~ (Will check during implementation)
- [x] ~~What's the appropriate total timeout budget?~~ - 9 attempts over ~6 minutes
- [x] ~~Should we fix the Retry button UX in this PR?~~ - YES, it's the same root cause
- [x] ~~Why did job 749141f8 fail before saving audio source info?~~ - Download info saved AFTER download, not before
- [ ] Should we also add retries to `wait_for_download()`?
- Probably NO - that already has its own polling/timeout logic
- [ ] Should we add circuit breaker pattern?
- Probably NO for now - keep it simple
- Could add in future if we see cascading failures
- [ ] Should retry logic be applied to other flacfetch methods?
- `start_download()` - probably YES (same pattern)
- `search()` - maybe YES (search can also have network issues)
- `get_download_status()` - probably NO (polling already handles this)
- **Decision**: Apply to `start_download()` too, skip `search()` for now
- [ ] Which toast library should frontend use?
- Check if already exists (react-hot-toast, sonner, etc.)
- If not, add lightweight library
- [ ] Should we save download params to state_data or top-level fields?
- Top-level fields more visible and easier to query
- **Decision**: Top-level fields (source_id, source_name, etc.)
## Rollback Plan
If this causes issues:
1. **Quick rollback**: Set env var `FLACFETCH_RETRY_MAX_ATTEMPTS=1` in Cloud Run
- Disables retries without code change
- Can deploy in ~2 minutes
2. **Full rollback**: Revert the PR
- Git revert the merge commit
- Deploy previous version
3. **Monitoring**: Watch for:
- Increased request duration to flacfetch
- Jobs timing out at higher layer (Cloud Tasks, Cloud Run)
- Duplicate download attempts causing issues
## Success Criteria
- [ ] Jobs with transient flacfetch failures succeed instead of failing
- [ ] Error messages are more informative (show attempts and root cause)
- [ ] No increase in false positives (jobs that should fail, now hang)
- [ ] All tests pass
- [ ] No impact on successful jobs (retry overhead is minimal)
## Related Issues
### Job 749141f8 - The Incident
**What happened:**
- User searched for audio via audio search
- Selected "Elizabeth Moen - Songbird"
- Backend tried to download via flacfetch
- Flacfetch call failed with network error (connection timeout/refused)
- Job immediately failed
- No source info saved → Retry button broken
- User sees "Retry" button but clicking does nothing (silent 400 error)
**How this fix helps:**
1. ✅ Flacfetch retry logic → Download would have succeeded after retry
2. ✅ Persist download info → Even if all retries fail, user can retry later
3. ✅ Retry endpoint fix → Retry button will actually work
4. ✅ Frontend toast → User sees helpful error if retry still fails
**Note:** Job 749141f8 cannot be fixed retroactively (no source info saved). But future jobs with same issue will be fixable via retry button.
### Future Work (Not in This PR)
Similar pattern should be applied to other external service calls:
- AudioShake API (lyrics transcription)
- Modal API (audio separation)
- YouTube direct API calls (not via flacfetch)
- Email sending (SendGrid)
## References
- [Tenacity documentation](https://tenacity.readthedocs.io/)
- [LESSONS-LEARNED.md](docs/LESSONS-LEARNED.md) - External service response format mismatches
- [flacfetch README](../../flacfetch/README.md) - Understanding the service
@@ -1,149 +0,0 @@
# Plan: Improve Job Observability
**Created:** 2026-03-07
**Branch:** feat/sess-20260307-2216-improve-job-observability
**Status:** Draft
## Overview
After shipping the "edit completed tracks" feature, we found it very difficult to trace what actually happened to edited jobs. Timeline events are too sparse (no output details), cleanup operations aren't logged to Cloud Logging or Firestore, state_data gets fully overwritten during re-processing (losing history of previous outputs), and API endpoints don't write to the job's Firestore log subcollection.
The goal is to make it trivially easy for future agents and humans to reconstruct the complete lifecycle of any job by reading its timeline and Firestore logs.
## Problem Analysis
### What we found investigating two edited jobs:
1. **Timeline events only have `status`, `timestamp`, `progress`, `message`** — no output details like YouTube video IDs, brand codes, Dropbox links, or GDrive file IDs
2. **Edit endpoint cleanup results never logged** — the `cleanup_results` dict is built and returned in the HTTP response but never logged to Cloud Logging or the job's Firestore log subcollection
3. **Distribution details incomplete in logs** — YouTube logs the URL but not video_id separately; Dropbox doesn't log folder path; GDrive doesn't log file IDs
4. **`state_data` fully overwritten on re-processing** — `video_worker.py:342-354` does `{**job.state_data, ...}`, so any `DELETE_FIELD` operations from the edit endpoint get lost when old keys are re-merged from the in-memory copy
5. **`outputs_deleted_at` cleared by video worker** — intentionally, but loses audit trail
6. **No Firestore job logs from API endpoints** — edit/admin endpoints only log to Cloud Run stdout, not to `jobs/{job_id}/logs/`
## Requirements
- [ ] Add `metadata` dict to `TimelineEvent` for structured output details
- [ ] Log all cleanup operations (success AND failure) to both Cloud Logging and Firestore job logs
- [ ] Log distribution output details (YouTube video ID, Dropbox path, brand code, GDrive file IDs) in timeline events and Firestore logs
- [ ] Preserve previous output details in timeline when a job is edited (snapshot before cleanup)
- [ ] Add helper to write Firestore job logs from API endpoints (not just workers)
- [ ] Ensure edit/admin delete endpoints write to Firestore job logs
- [ ] No breaking changes to existing timeline format (additive only)
## Technical Approach
### 1. Add `metadata` field to `TimelineEvent`
The `TimelineEvent` model gains an optional `metadata: Optional[Dict[str, Any]]` field. This allows any timeline event to carry structured data without changing the schema. The field is additive — existing events without metadata continue to work.
Key events that gain metadata:
- **`complete`**: `{youtube_url, youtube_video_id, dropbox_link, brand_code, gdrive_file_ids}`
- **Edit initiated**: `{previous_outputs: {youtube_url, dropbox_link, brand_code, gdrive_files}, cleanup_results: {...}}`
- **Outputs deleted**: `{deleted_outputs: {...}, cleanup_results: {...}}`
### 2. Write Firestore job logs from API endpoints
Create a simple helper function that API route handlers can call to append logs to the job's Firestore subcollection. Workers already have `JobLogAdapter` — we just need a lightweight equivalent for API routes:
```python
def log_to_job(job_id: str, worker: str, level: str, message: str, metadata: dict = None):
"""Write a log entry to the job's Firestore log subcollection."""
entry = WorkerLogEntry.create(job_id, worker, level, message, metadata)
FirestoreService().append_log_to_subcollection(job_id, entry)
```
### 3. Enhance distribution logging in video_worker_orchestrator
After each distribution upload succeeds, log the details with structured metadata:
- YouTube: video_id, video_url, channel
- Dropbox: folder_path, sharing_link
- GDrive: file_ids dict
- Brand code: code, prefix, allocated vs recycled
### 4. Snapshot previous outputs on edit
Before the edit endpoint clears state_data distribution keys, capture the current values in the timeline event's metadata. This preserves the history permanently in the timeline array (which is append-only via `ArrayUnion`).
### 5. Log cleanup results from edit/admin endpoints
After cleanup runs, log each step's result to both Cloud Logging and the Firestore job log subcollection with structured metadata.
## Implementation Steps
### Step 1: Add `metadata` to `TimelineEvent` model
- [ ] Add `metadata: Optional[Dict[str, Any]] = None` to `TimelineEvent` in `backend/models/job.py`
- [ ] No migration needed — Firestore is schemaless, existing events just won't have the field
### Step 2: Create `log_to_job` helper
- [ ] Add `log_to_job()` function in `backend/services/firestore_service.py` (or a small utility module)
- [ ] Takes `job_id`, `worker`, `level`, `message`, optional `metadata`
- [ ] Wraps `WorkerLogEntry.create()` + `append_log_to_subcollection()`
- [ ] Catches and suppresses errors (logging should never break the caller)
### Step 3: Update `transition_to_state` to accept metadata
- [ ] Add optional `timeline_metadata: Optional[Dict[str, Any]] = None` parameter to `job_manager.transition_to_state()`
- [ ] Pass through to `update_job_status()` → `TimelineEvent` construction
- [ ] Update `firestore_service.update_job_status()` to accept and pass `timeline_metadata`
### Step 4: Enhance edit endpoint logging
- [ ] Before cleanup: snapshot current distribution outputs (`youtube_url`, `dropbox_link`, `brand_code`, `gdrive_files`) into `previous_outputs` dict
- [ ] After each cleanup step: log result to Cloud Logging AND call `log_to_job()`
- [ ] After all cleanup: include `previous_outputs` and `cleanup_results` in the timeline event metadata
- [ ] The "Track edit initiated" timeline event gains metadata: `{initiated_by, previous_outputs, cleanup_results}`
### Step 5: Enhance admin delete-outputs endpoint logging
- [ ] Same pattern as edit endpoint: snapshot outputs, log each cleanup step, include in timeline metadata
- [ ] The "Outputs deleted by admin" timeline event gains metadata: `{deleted_by, deleted_outputs, cleanup_results}`
### Step 6: Enhance distribution logging in video_worker_orchestrator
- [ ] YouTube upload success: log video_id and video_url with metadata `{video_id, video_url}`
- [ ] Dropbox upload success: log folder_path and sharing_link with metadata `{folder_path, sharing_link}`
- [ ] GDrive upload success: log file_ids with metadata `{file_ids: {...}}`
- [ ] Brand code allocation: log with metadata `{brand_code, source: "allocated"|"recycled"}`
### Step 7: Add output details to `complete` timeline event
- [ ] When `video_worker.py` calls `transition_to_state(COMPLETE)`, pass `timeline_metadata` with distribution output details
- [ ] Include: `youtube_url`, `youtube_video_id`, `dropbox_link`, `brand_code`, `gdrive_file_ids`, `edit_count`
### Step 8: Tests
- [ ] Unit test: `TimelineEvent` with metadata serializes correctly
- [ ] Unit test: `log_to_job` writes to subcollection
- [ ] Unit test: edit endpoint includes `previous_outputs` and `cleanup_results` in timeline event
- [ ] Unit test: `complete` transition includes output metadata
- [ ] Verify existing tests still pass
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Add `metadata` field to `TimelineEvent` |
| `backend/services/firestore_service.py` | Modify | Add `log_to_job()` helper, update `update_job_status()` to accept timeline metadata |
| `backend/services/job_manager.py` | Modify | Add `timeline_metadata` param to `transition_to_state()` |
| `backend/api/routes/jobs.py` | Modify | Enhance edit endpoint with cleanup logging, output snapshots, timeline metadata |
| `backend/api/routes/admin.py` | Modify | Enhance delete-outputs endpoint similarly |
| `backend/workers/video_worker_orchestrator.py` | Modify | Add structured metadata to distribution log messages |
| `backend/workers/video_worker.py` | Modify | Pass output metadata in `complete` transition (both orchestrator and legacy paths) |
| `backend/tests/test_edit_completed_track.py` | Modify | Verify timeline metadata in edit tests |
| `backend/tests/test_observability.py` | Create | Tests for TimelineEvent metadata and log_to_job |
## Testing Strategy
- **Unit tests**: TimelineEvent metadata serialization, log_to_job helper, edit endpoint timeline metadata assertions
- **Existing test verification**: `make test` must pass — changes are additive so existing tests should be unaffected
- **No E2E tests needed**: This is purely backend logging/observability — no UI changes
## What This Does NOT Include
- **Dedicated admin UI for viewing job logs** — existing admin page already shows timeline
- **Log aggregation/search UI** — Cloud Logging and Firestore viewer already serve this purpose
- **Changing state_data overwrite behavior** — the `{**job.state_data, ...}` pattern in video_worker.py is intentional; instead we preserve history in timeline events
- **Changing outputs_deleted_at clearing** — video worker clearing it after re-processing is correct behavior; the edit history lives in timeline metadata instead
## Rollback Plan
All changes are additive (new optional field on TimelineEvent, additional log writes). Rolling back just means deploying the previous version — no data migration needed.
## Open Questions
None — the approach is straightforward and low-risk.
-279
View File
@@ -1,279 +0,0 @@
# Plan: Private (Non-Published) Tracks
**Created:** 2026-02-24
**Branch:** feat/sess-20260224-1149-private-tracks
**Status:** Implemented
## Overview
Add a `is_private` flag to jobs that changes distribution behavior:
- **Dropbox**: Uses `Tracks-NonPublished` folder (not `Tracks-Organized`) with brand code `NOMADNP` (not `NOMAD`)
- **YouTube**: Skipped entirely
- **Google Drive**: Skipped entirely
- **Email**: Completion email includes Dropbox link but no YouTube section
- **Discord**: Still notified (for internal tracking)
Available to all users at job creation and toggleable by admin on existing jobs.
## Requirements
- [ ] All users can mark a job as "Private (Non-Published)" when creating it
- [ ] Admin can toggle `is_private` on any existing job via admin dashboard
- [ ] Private jobs upload to Dropbox at `Tracks-NonPublished` with `NOMADNP-XXXX` brand codes
- [ ] Private jobs skip YouTube and Google Drive uploads entirely
- [ ] Completion emails omit YouTube section when no YouTube URL (private jobs + any other edge case)
- [ ] When admin toggles a completed job to private, existing outputs are auto-deleted
- [ ] Discord notifications still fire for private jobs
- [ ] Existing (non-private) jobs are completely unaffected (backward compatible)
## Technical Approach
### Core Design: Single Boolean Flag with Runtime Overrides
Store `is_private: bool = False` on the job model. At distribution time, a helper function returns effective distribution parameters based on this flag. This means:
- **Toggling is_private automatically changes all distribution behavior** - no need to manually update dropbox_path, brand_prefix, enable_youtube_upload, gdrive_folder_id
- **Job fields still store the "normal" defaults** - the override happens at the config-to-orchestrator boundary
- **Admin can flip one field** and everything adapts
### Key Helper Function
```python
# In job_defaults_service.py
def get_effective_distribution_for_job(job) -> EffectiveDistributionSettings:
"""Return distribution settings, applying private overrides if is_private=True."""
if getattr(job, 'is_private', False):
settings = get_settings()
return EffectiveDistributionSettings(
dropbox_path=settings.default_private_dropbox_path,
brand_prefix=settings.default_private_brand_prefix,
enable_youtube_upload=False,
gdrive_folder_id=None,
# Keep other settings from job (discord, youtube_description, etc.)
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=None,
)
# Non-private: return job's own settings
return EffectiveDistributionSettings(
dropbox_path=getattr(job, 'dropbox_path', None),
brand_prefix=getattr(job, 'brand_prefix', None),
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=getattr(job, 'youtube_description_template', None),
)
```
This helper is used in both distribution paths (orchestrator and native fallback).
### Distribution Paths That Need Changes
Two code paths handle distribution (see LESSONS-LEARNED: "Fix Both Sides of Dual Code Paths"):
1. **Orchestrator path** (`video_worker_orchestrator.py`): Primary path. `create_orchestrator_config_from_job()` uses the helper to build config.
2. **Native distribution path** (`video_worker.py` → `_handle_native_distribution()`): Fallback path. Also uses the helper instead of reading job fields directly.
### Email Template Change
Add conditional YouTube section removal in `render_template()`, matching the existing pattern for `feedback_url`:
```python
# Remove YouTube section if no URL provided
if not variables.get("youtube_url"):
result = re.sub(
r"\n*Here's the link for the karaoke video published to YouTube:\n\{youtube_url\}\n*",
'\n',
result,
flags=re.DOTALL
)
variables["youtube_url"] = ""
```
And change `render_job_completion()` to pass `None` instead of `"[YouTube URL not available]"` when youtube_url is empty.
### Admin Toggle Auto-Delete
When admin PATCH updates `is_private` from False to True on a job that has existing outputs, reuse the existing delete-outputs logic to remove YouTube + GDrive + Dropbox outputs. Admin can then "Reprocess" to re-distribute to the private Dropbox path.
## Implementation Steps
### Phase 1: Backend Model & Config (foundation)
1. [ ] **Add `is_private` to Job model** (`backend/models/job.py`)
- Add `is_private: bool = False` field after `made_for_you` field
- Add to `JobCreate` model as well
2. [ ] **Add private distribution config** (`backend/config.py`)
- Add `default_private_dropbox_path: Optional[str] = os.getenv("DEFAULT_PRIVATE_DROPBOX_PATH")`
- Add `default_private_brand_prefix: Optional[str] = os.getenv("DEFAULT_PRIVATE_BRAND_PREFIX", "NOMADNP")`
3. [ ] **Add helper function** (`backend/services/job_defaults_service.py`)
- Add `get_effective_distribution_for_job(job)` function that returns distribution settings with private overrides applied
### Phase 2: Backend API (job creation + admin)
4. [ ] **Add `is_private` to request models** (`backend/models/requests.py`)
- Add `is_private: Optional[bool] = None` to `URLSubmissionRequest`
- Add `is_private: Optional[bool] = None` to `AudioSearchRequest`
5. [ ] **Update URL creation endpoint** (`backend/api/routes/jobs.py`)
- Accept and pass through `is_private` field to `JobCreate`
6. [ ] **Update file upload endpoint** (`backend/api/routes/file_upload.py`)
- Accept `is_private` Form parameter
- Pass through to `JobCreate`
7. [ ] **Update audio search endpoint** (`backend/api/routes/audio_search.py`)
- Accept `is_private` in `AudioSearchRequest`
- Pass through to `JobCreate`
8. [ ] **Update admin endpoints** (`backend/api/routes/admin.py`)
- Add `is_private: Optional[bool] = None` to `JobUpdateRequest`
- Add `"is_private"` to `EDITABLE_JOB_FIELDS`
- In the PATCH handler: detect when `is_private` changes from False→True on a completed job, auto-delete existing outputs (call existing delete-outputs service logic)
### Phase 3: Distribution Pipeline
9. [ ] **Update orchestrator config creation** (`backend/workers/video_worker_orchestrator.py`)
- In `create_orchestrator_config_from_job()`: Use `get_effective_distribution_for_job()` helper to determine dropbox_path, brand_prefix, enable_youtube_upload, gdrive_folder_id instead of reading directly from job
10. [ ] **Update native distribution fallback** (`backend/workers/video_worker.py`)
- In `_handle_native_distribution()`: Use `get_effective_distribution_for_job()` helper instead of reading directly from job fields
### Phase 4: Email Template
11. [ ] **Add YouTube section conditional removal** (`backend/services/template_service.py`)
- In `render_template()`: Add regex to remove YouTube paragraph when `youtube_url` is empty (matching existing `feedback_url` pattern)
- In `render_job_completion()`: Pass `None` for youtube_url when empty (instead of `"[YouTube URL not available]"`)
### Phase 5: Frontend
12. [ ] **Add private checkbox to job creation form** (`frontend/components/job/JobSubmission.tsx`)
- Add "Private (Non-Published)" checkbox below the existing "Skip lyrics review" checkbox
- Show for all users (not just admin)
- Include brief description text: "Private tracks go to Dropbox only, not YouTube or Google Drive"
13. [ ] **Update API interfaces** (`frontend/lib/api.ts`)
- Add `is_private?: boolean` to all job creation option interfaces
- Add `is_private` to `JobUpdateRequest` interface
- Pass through in `uploadJob()`, `createJobFromUrl()`, `searchAudio()` methods
14. [ ] **Add private toggle to admin job detail** (`frontend/app/admin/jobs/page.tsx`)
- Add `is_private` as an inline-editable field in the job overview section (use Switch component)
- Show current state with visual indicator (e.g., lock icon for private jobs)
- When toggling to private on a completed job, show confirmation dialog about auto-deleting outputs
### Phase 6: Environment Configuration
15. [ ] **Update CI/deployment config** (`.github/workflows/ci.yml` or Cloud Run env)
- Add `DEFAULT_PRIVATE_DROPBOX_PATH` environment variable (e.g., `/Karaoke/Tracks-NonPublished`)
- Add `DEFAULT_PRIVATE_BRAND_PREFIX` environment variable (e.g., `NOMADNP`)
- These should be configured in Cloud Run service and CI test environment
### Phase 7: Testing
16. [ ] **Unit tests for private distribution logic** (`backend/tests/test_private_tracks.py`)
- Test `get_effective_distribution_for_job()` with private and non-private jobs
- Test `create_orchestrator_config_from_job()` applies private overrides correctly
- Test `_run_distribution()` skips YouTube and GDrive when private config applied
- Test `_upload_to_dropbox()` uses private path and brand prefix
- Test brand code generation with NOMADNP prefix in Tracks-NonPublished
- Test `_handle_native_distribution()` respects private overrides
- Test non-private jobs are completely unaffected (regression tests)
17. [ ] **Unit tests for email changes** (add to `backend/tests/test_email_service.py` or `test_template_service.py`)
- Test YouTube section removed from email when youtube_url is None
- Test YouTube section present when youtube_url is provided
- Test Dropbox section still present for private jobs
- Test full email rendering for private job scenario
18. [ ] **Unit tests for admin toggle** (add to `backend/tests/test_admin.py` or new file)
- Test admin can update is_private field
- Test toggling is_private to True on completed job triggers output deletion
- Test toggling is_private to True on non-completed job does NOT delete outputs
- Test toggling is_private to False does nothing special
19. [ ] **Integration test** (`backend/tests/emulator/` or `test_distribution_services.py`)
- Test full distribution pipeline with is_private=True: verify Dropbox path, brand prefix, YouTube skip, GDrive skip
20. [ ] **Frontend E2E regression test** (`frontend/e2e/regression/`)
- Test private checkbox appears in job creation form
- Test private toggle in admin dashboard
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Add `is_private: bool = False` to Job and JobCreate |
| `backend/config.py` | Modify | Add `default_private_dropbox_path` and `default_private_brand_prefix` |
| `backend/services/job_defaults_service.py` | Modify | Add `get_effective_distribution_for_job()` helper |
| `backend/models/requests.py` | Modify | Add `is_private` to URL and AudioSearch request models |
| `backend/api/routes/jobs.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/file_upload.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/audio_search.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/admin.py` | Modify | Add `is_private` to editable fields + auto-delete logic |
| `backend/workers/video_worker_orchestrator.py` | Modify | Use helper in `create_orchestrator_config_from_job()` |
| `backend/workers/video_worker.py` | Modify | Use helper in `_handle_native_distribution()` |
| `backend/services/template_service.py` | Modify | Conditional YouTube section removal |
| `frontend/components/job/JobSubmission.tsx` | Modify | Add private checkbox |
| `frontend/lib/api.ts` | Modify | Add `is_private` to interfaces and API calls |
| `frontend/app/admin/jobs/page.tsx` | Modify | Add private toggle to admin detail |
| `backend/tests/test_private_tracks.py` | Create | Comprehensive unit tests for private track logic |
## Testing Strategy
### Unit Tests (~25-30 tests in `test_private_tracks.py`)
- `get_effective_distribution_for_job()` with private=True and False
- `create_orchestrator_config_from_job()` private overrides (YouTube=False, GDrive=None, private Dropbox path, NOMADNP brand)
- Distribution stage skips: YouTube not called, GDrive not called, Dropbox IS called with correct path
- Brand code generation uses NOMADNP prefix and Tracks-NonPublished path
- Native distribution fallback applies same overrides
- **Regression tests**: Non-private jobs have unchanged behavior (same paths, same brand code, YouTube+GDrive+Dropbox all fire)
### Email Template Tests (~5 tests)
- YouTube section stripped when youtube_url is None
- YouTube section preserved when youtube_url has value
- Dropbox section always present
- Full private job email rendering (Dropbox + no YouTube)
- Full non-private job email rendering (both present) - regression
### Admin Toggle Tests (~5 tests)
- is_private editable via PATCH
- Toggle to private on completed job deletes outputs
- Toggle to private on pending/in-progress job does NOT delete
- Toggle to non-private does nothing special
### Frontend Tests
- E2E: Private checkbox visible and functional in job creation
- E2E: Admin toggle visible and includes confirmation dialog
### Manual Verification
- Create a private job end-to-end, verify Dropbox folder in Tracks-NonPublished with NOMADNP code
- Verify completion email has Dropbox link but no YouTube section
- Toggle existing completed job to private, verify outputs deleted
## Regression Risk Analysis
| Risk | Mitigation |
|------|-----------|
| Non-private jobs affected | `is_private` defaults to `False`; helper returns unchanged values for non-private. Explicit regression tests. |
| Dual code paths (orchestrator vs native) | Both paths use same helper function. Tests cover both paths. (Lesson: "Fix Both Sides of Dual Code Paths") |
| Email template regex breaks other content | Regex targets specific YouTube paragraph text. Tests verify non-YouTube content untouched. |
| Brand code namespace collision | NOMADNP codes are in a completely separate Dropbox folder (Tracks-NonPublished), no overlap with NOMAD codes in Tracks-Organized |
| Admin auto-delete on wrong job | Only triggers when `is_private` changes from False→True AND job has terminal status (complete). Confirmation dialog in frontend. |
| Made-for-you orders | MFY webhook doesn't set is_private (defaults False). Admin can toggle after. No change to MFY flow. |
| Existing Firestore documents | Missing `is_private` field defaults to `False` via model default. No migration needed. |
## Open Questions
- [x] Who sees private toggle? → **Everyone** (all users at creation, admin on existing)
- [x] Auto-delete on toggle? → **Yes**, auto-delete all outputs when toggling to private on completed job
- [x] Email format? → **Remove YouTube section** when no YouTube URL (improves all edge cases)
- [x] Discord? → **Still notify** (useful for internal tracking)
## Rollback Plan
1. **Feature flag**: The `is_private` field defaults to `False`, so setting `DEFAULT_PRIVATE_DROPBOX_PATH` to empty/None effectively disables private distribution
2. **No schema migration needed**: Missing `is_private` field on existing Firestore documents defaults to `False`
3. **Email change is backward-compatible**: The YouTube section removal only happens when youtube_url is None, which is already an edge case showing "[YouTube URL not available]"
4. **Frontend**: Private checkbox is additive UI, removing it just means users can't create new private jobs
@@ -1,239 +0,0 @@
# Plan: Song Lookup & Autocomplete + Community Version Detection
**Created:** 2026-03-02
**Branch:** feat/sess-20260302-2324-song-lookup-autocomplete
**Status:** Implementation Complete
## Overview
Add two features to the karaoke-gen job creation flow:
1. **Artist/title autocomplete** using karaoke-decide's MusicBrainz + Spotify catalog data, so users get properly-styled artist/title text (e.g. "Fleetwood Mac" instead of "fleetwood mac")
2. **Community karaoke version detection** that warns users when a high-quality community karaoke version already exists on YouTube (via karaokenerds data), saving them credits and time
Both features are **optional and non-blocking** — users can still type whatever they want and proceed even if a community version exists.
## Requirements
- [ ] Artist field autocomplete across all 3 job creation tabs (Search, Upload, URL)
- [ ] Title field autocomplete (optionally filtered by selected artist) across all 3 tabs
- [ ] Autocomplete uses canonical MusicBrainz/Spotify text styling
- [ ] Autocomplete is optional — users can ignore suggestions and type freely
- [ ] Community version check triggers when both artist and title are filled
- [ ] If community karaoke version exists, show a dismissible banner with YouTube link(s)
- [ ] Community check does not block job submission — just informs the user
- [ ] Graceful degradation if karaoke-decide API or karaokenerds.com is down
## Architecture Decisions
### How to access karaoke-decide catalog data
**Decision: Proxy through karaoke-gen backend**
Both services share the same GCP project (`nomadkaraoke`), but the frontend at `gen.nomadkaraoke.com` can't directly call `decide.nomadkaraoke.com/api/` due to CORS (different origins). Options considered:
| Option | Pros | Cons |
|--------|------|------|
| **Proxy via karaoke-gen backend** | No CORS issues, no changes to karaoke-decide, frontend already calls api.nomadkaraoke.com | Added latency (extra hop), karaoke-decide dependency |
| Direct frontend calls + CORS | Simpler, no proxy code | Requires karaoke-decide CORS changes, cross-project coupling |
| Query BigQuery directly from karaoke-gen | No dependency on karaoke-decide service | Duplicates query logic, BigQuery setup needed, higher latency |
The proxy approach is cleanest: karaoke-gen frontend → karaoke-gen backend → karaoke-decide API. Response caching in the proxy minimizes latency impact.
### How to detect community karaoke versions
**Decision: Scrape karaokenerds.com from karaoke-gen backend** (same approach as kjbox)
The `karaokenerds_raw` BigQuery table has the catalog but NOT community status. Community status is determined by an `<img class="check">` badge in karaokenerds.com's HTML, only available via web scraping. The kjbox codebase has proven, well-tested scraping logic we can port directly.
Search URL: `https://karaokenerds.com/Search?query={artist}+{title}&webFilter=OnlyWeb`
Returns: songs with tracks, each track having `brand_name`, `brand_code`, `youtube_url`, `is_community`.
### Autocomplete UI component
**Decision: Use existing shadcn Command (cmdk) + Popover components**
The frontend already has `command.tsx` and `popover.tsx` from shadcn/ui. The standard combobox pattern (Popover + Command) provides:
- Keyboard navigation (arrow keys, enter to select)
- Debounced search-as-you-type
- Clean dropdown with loading states
- Works inline within forms (not modal)
## Technical Approach
### Data flow
```
User types in artist field
→ debounced (300ms) API call to GET /api/catalog/artists?q=fle
→ karaoke-gen backend proxies to decide.nomadkaraoke.com/api/catalog/artists?q=fle
→ returns [{ name: "Fleetwood Mac", mbid: "...", popularity: 85 }, ...]
→ user selects "Fleetwood Mac" → field populated with canonical name
User types in title field
→ debounced (300ms) API call to GET /api/catalog/tracks?q=dreams&artist=Fleetwood+Mac
→ returns [{ track_name: "Dreams", artist_name: "Fleetwood Mac", popularity: 82 }, ...]
→ user selects "Dreams" → field populated with canonical title
Both fields filled → community check fires
→ POST /api/catalog/community-check { artist: "Fleetwood Mac", title: "Dreams" }
→ backend scrapes karaokenerds.com for "Fleetwood Mac Dreams" with webFilter=OnlyWeb
→ if community tracks found with YouTube URLs, return them
→ frontend shows banner: "A community karaoke version already exists! [Watch on YouTube →]"
```
### Caching strategy
| Endpoint | Cache TTL | Scope |
|----------|-----------|-------|
| Artist search proxy | 5 min | In-memory (backend) |
| Track search proxy | 5 min | In-memory (backend) |
| Community check | 1 hour | In-memory (backend) |
### karaoke-decide API endpoints used
These are all **public, no auth required**:
- `GET /api/catalog/artists?q={query}&limit=10` — MBID-first artist search
- Returns: `[{ mbid, name, disambiguation, artist_type, tags, spotify_id, popularity, genres }]`
- `GET /api/catalog/tracks?q={query}&limit=10` — Track search (Spotify data)
- Returns: `[{ track_id, track_name, artist_name, artist_id, popularity, duration_ms, explicit }]`
## Implementation Steps
### Phase 1: Backend — Catalog proxy + community check endpoints
1. [ ] **Add catalog proxy service** — `backend/services/catalog_proxy_service.py`
- HTTP client (httpx async) calling karaoke-decide API
- In-memory TTL cache for responses (5 min)
- Configurable karaoke-decide base URL
- Graceful error handling (return empty results on failure)
2. [ ] **Port karaokenerds scraping logic** — `backend/services/karaokenerds_service.py`
- Port `search()`, `parse_results()`, `_parse_tracks()`, `_parse_single_track()`, `_clean_youtube_url()` from `kjbox/kj-controller/karaoke_nerds.py`
- Add async support (httpx instead of requests)
- In-memory TTL cache (1 hour)
- Return only community tracks with YouTube URLs
3. [ ] **Add catalog routes** — `backend/api/routes/catalog.py`
- `GET /api/catalog/artists?q=&limit=10` — proxy to karaoke-decide
- `GET /api/catalog/tracks?q=&artist=&limit=10` — proxy to karaoke-decide
- `POST /api/catalog/community-check` — scrape karaokenerds.com
- All endpoints require auth (same as job creation)
- Pydantic request/response models
4. [ ] **Register routes in main.py** — add `catalog.router` to app
5. [ ] **Add config** — `backend/config.py`
- `karaoke_decide_api_url` setting (default: `https://decide.nomadkaraoke.com`)
6. [ ] **Add `beautifulsoup4` + `httpx` dependencies** to `pyproject.toml` (if not already present)
### Phase 2: Frontend — Autocomplete component
7. [ ] **Create `AutocompleteInput` component** — `frontend/components/ui/autocomplete-input.tsx`
- Wraps an Input field with a Popover + Command dropdown
- Props: `value`, `onChange`, `onSelect`, `fetchSuggestions(query) => Promise<Suggestion[]>`, `placeholder`, `disabled`, `renderItem`
- Debounced search (300ms) triggered on input change
- Shows loading spinner while fetching
- Shows "No results" when search returns empty
- Keyboard navigation (up/down arrows, enter to select, escape to close)
- Clicking outside closes dropdown
- When user selects a suggestion, calls `onSelect(suggestion)` and populates input
- When user types without selecting, uses their raw text (autocomplete is optional)
8. [ ] **Add catalog API functions** — `frontend/lib/api.ts`
- `searchArtists(query: string, limit?: number)` → `GET /api/catalog/artists`
- `searchTracks(query: string, artist?: string, limit?: number)` → `GET /api/catalog/tracks`
- `checkCommunityVersions(artist: string, title: string)` → `POST /api/catalog/community-check`
### Phase 3: Frontend — Integrate autocomplete into job creation
9. [ ] **Replace artist Input fields with AutocompleteInput** in `JobSubmission.tsx`
- All 3 tabs: `searchArtist`, `uploadArtist`, `youtubeArtist`
- Fetch suggestions from `api.searchArtists(query)`
- On select: populate artist field with canonical name
- Show artist name + optional disambiguation in dropdown items
10. [ ] **Replace title Input fields with AutocompleteInput** in `JobSubmission.tsx`
- All 3 tabs: `searchTitle`, `uploadTitle`, `youtubeTitle`
- Fetch suggestions from `api.searchTracks(query, selectedArtist)`
- On select: populate title field with canonical name
- Show track name + artist name + duration in dropdown items
### Phase 4: Frontend — Community version detection
11. [ ] **Create `CommunityVersionBanner` component** — `frontend/components/job/CommunityVersionBanner.tsx`
- Shows when community karaoke version(s) found
- Displays: "A karaoke version of this song already exists!"
- Lists community tracks with brand name and YouTube link
- "Watch on YouTube" button for top community version
- Dismissible (X button or "Continue anyway" link)
- Styled with green border (matching kjbox community track styling)
12. [ ] **Add community check logic to `JobSubmission.tsx`** (Search tab only)
- Debounced check (500ms) when both artist AND title fields have values
- Only triggers when values change (not on every keystroke)
- Shows `CommunityVersionBanner` between the form fields and the submit button
- Does NOT block form submission
- Clears when artist or title changes
### Phase 5: Testing
13. [ ] **Backend unit tests** — `backend/tests/unit/test_catalog_routes.py`
- Test proxy endpoints with mocked karaoke-decide responses
- Test community check with mocked karaokenerds HTML
- Test error handling (karaoke-decide down, karaokenerds down)
- Test caching behavior
14. [ ] **Backend unit tests** — `backend/tests/unit/test_karaokenerds_service.py`
- Port relevant tests from `kjbox/kj-controller/tests/unit/test_karaoke_nerds.py`
- Test HTML parsing, community detection, YouTube URL cleanup
15. [ ] **Frontend component tests** — `frontend/components/ui/__tests__/autocomplete-input.test.tsx`
- Test suggestion fetching and display
- Test keyboard navigation
- Test selection behavior
- Test free-text input (no selection)
16. [ ] **E2E tests** — `frontend/e2e/job-creation-autocomplete.spec.ts`
- Test autocomplete appears on typing
- Test selecting from autocomplete populates field
- Test community banner appears for known songs
- Test form submission works with and without autocomplete selection
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/services/catalog_proxy_service.py` | Create | HTTP client for karaoke-decide API with caching |
| `backend/services/karaokenerds_service.py` | Create | Karaokenerds.com scraper (ported from kjbox) |
| `backend/api/routes/catalog.py` | Create | Proxy endpoints for artist/track search + community check |
| `backend/main.py` | Modify | Register catalog router |
| `backend/config.py` | Modify | Add karaoke_decide_api_url setting |
| `pyproject.toml` | Modify | Add beautifulsoup4 dependency (if missing) |
| `frontend/components/ui/autocomplete-input.tsx` | Create | Reusable autocomplete input component |
| `frontend/components/job/CommunityVersionBanner.tsx` | Create | Community version warning banner |
| `frontend/components/job/JobSubmission.tsx` | Modify | Replace Input with AutocompleteInput, add community check |
| `frontend/lib/api.ts` | Modify | Add catalog API functions |
| `backend/tests/unit/test_catalog_routes.py` | Create | Backend route tests |
| `backend/tests/unit/test_karaokenerds_service.py` | Create | Scraper unit tests |
| `frontend/components/ui/__tests__/autocomplete-input.test.tsx` | Create | Component tests |
| `frontend/e2e/job-creation-autocomplete.spec.ts` | Create | E2E tests |
## Decisions (Resolved)
- **Track search scope**: All Spotify tracks (256M) — better coverage for new/obscure songs
- **Community check scope**: Search tab only (Upload/URL users already have their audio)
- **Rate limiting**: 20 requests/min per user on catalog proxy endpoints (prevents abuse of karaokenerds scraping). All endpoints require auth.
- **"Display As" fields**: No autocomplete — they're for deliberate overrides like soundtrack names
- **CORS**: Proxy approach (no changes needed to karaoke-decide)
## Rollback Plan
- Feature is entirely additive — no existing functionality is modified
- Autocomplete fields fall back to plain text Input if API calls fail
- Community check is non-blocking — if it fails, form works normally
- All new endpoints are behind the existing auth middleware
- Can disable by reverting the frontend changes (backend proxy is harmless if unused)
-42
View File
@@ -1,42 +0,0 @@
# karaoke-gen backend environment — copy to .env and adjust.
#
# NOTE: scripts/run-backend-local.sh --with-emulators sets all the required
# variables itself; you only need a .env for optional API keys or overrides.
# None of the keys below are required to boot the backend locally.
# --- Core (defaults shown are what run-backend-local.sh uses in emulator mode) ---
# ENVIRONMENT=development
# GOOGLE_CLOUD_PROJECT=test-project
# GCS_BUCKET_NAME=test-bucket
# FIRESTORE_COLLECTION=test-jobs
# ADMIN_TOKENS=local-dev-token
# PORT=8000
# FRONTEND_URL=http://localhost:3000
# --- Emulators (set automatically by run-backend-local.sh --with-emulators) ---
# FIRESTORE_EMULATOR_HOST=127.0.0.1:8080
# STORAGE_EMULATOR_HOST=http://127.0.0.1:4443
# --- Audio separation (pick one; without either, jobs stall at separation) ---
# Local CPU/GPU separation via the audio-separator package. Models are
# downloaded into this directory on first use (~2GB). CPU works but is slow.
# MODEL_DIR=./models
# ...or point at a remote audio-separator API service:
# AUDIO_SEPARATOR_API_URL=
# --- Lyrics transcription ---
# The lyrics worker reads AUDIOSHAKE_API_TOKEN (not AUDIOSHAKE_API_KEY).
# Without it, transcription falls back to local Whisper if installed
# (poetry install --extras local-whisper).
# AUDIOSHAKE_API_TOKEN=
# --- Lyrics sources (optional; improves reference lyrics) ---
# GENIUS_API_KEY=
# SPOTIFY_COOKIE_SP_DC=
# --- Email (optional; without it, emails are logged to the console) ---
# POSTMARK_SERVER_TOKEN=
# --- Payments (optional; only needed to exercise payment endpoints) ---
# STRIPE_SECRET_KEY=
# STRIPE_WEBHOOK_SECRET=
-58
View File
@@ -1,58 +0,0 @@
#!/usr/bin/env bash
# Nomad Karaoke pre-commit hook (karaoke-gen variant)
# 1. Gitleaks secret scan (fail fast)
# 2. i18n auto-translation when en.json changes
# Enable: git config core.hooksPath .githooks
set -eu
# ---------------------------------------------------------------------------
# 1. Gitleaks secret scan
# ---------------------------------------------------------------------------
if command -v gitleaks >/dev/null 2>&1; then
if ! gitleaks protect --staged --redact --no-banner; then
printf '\n\033[31m✗ gitleaks detected a potential secret in staged changes.\033[0m\n' >&2
printf ' • If real: remove it, rotate the credential, then re-commit.\n' >&2
printf ' • If false positive: add an allowlist entry to .gitleaks.toml.\n' >&2
printf ' • Emergency override (not recommended): git commit --no-verify\n' >&2
exit 1
fi
else
printf '\033[33mwarn:\033[0m gitleaks not installed — skipping secret scan\n' >&2
printf ' install: brew install gitleaks\n' >&2
fi
# ---------------------------------------------------------------------------
# 2. i18n auto-translation
# ---------------------------------------------------------------------------
STAGED_EN=$(git diff --cached --name-only --diff-filter=ACM | grep -E '(messages|translations)/en\.json$' || true)
if [ -z "$STAGED_EN" ]; then
exit 0
fi
echo "i18n: English translations changed — running translation pipeline..."
for en_file in $STAGED_EN; do
dir=$(dirname "$en_file")
if [[ "$dir" == frontend/messages ]]; then
script="frontend/scripts/translate.py"
elif [[ "$dir" == backend/translations ]]; then
script="frontend/scripts/translate.py"
else
echo " Warning: Unknown messages dir $dir, skipping"
continue
fi
echo " Translating $dir..."
if python "$script" --messages-dir "$dir" --target all --skip-review 2>&1 | tail -5; then
git add "$dir"/*.json
echo " Done: $dir"
else
echo " Warning: Translation failed for $dir (GCP auth may be missing). CI will catch this."
fi
done
echo "i18n: Translation complete."
-327
View File
@@ -1,327 +0,0 @@
name: Build GHA Runner Images
# Produces the custom GCE images used by the ephemeral GitHub Actions runner
# dispatcher: gha-runner-general, gha-runner-build, gha-runner-gpu,
# gha-runner-gpu-windows.
#
# Flow per variant:
# 1. Create a temporary build VM from the variant's base image with the
# variant's provisioning script as its startup script:
# - Linux variants: debian-12 + runner-image-provision.sh (startup-script)
# - gpu-windows: windows-2022 + runner-image-provision.ps1
# (windows-startup-script-ps1)
# 2. Poll the serial console for the READY/FAILED marker (survives reboots).
# 3. Stop the VM, snapshot its boot disk to a new image in the `gha-runner-<variant>`
# image family with name gha-runner-<variant>-<timestamp>.
# 4. Delete the build VM (boot disk auto-deletes).
# 5. Delete older images in the family, keeping only the newest (GPU/windows images
# are 15-21 GiB each; roll back by re-running this workflow at an older ref).
on:
workflow_dispatch:
inputs:
variants:
description: 'Comma-separated variants to build (general,build,gpu,gpu-windows) or "all" (= all except build)'
required: false
default: 'all'
skip_prune:
description: 'Skip pruning (deletion) of old images (debugging)'
type: boolean
required: false
default: false
schedule:
# Monthly on the 1st at 02:00 UTC
- cron: '0 2 1 * *'
permissions:
contents: read
id-token: write
env:
GCP_PROJECT: nomadkaraoke
BUILD_REGION: us-central1
BUILD_ZONE: us-central1-a
BASE_IMAGE: projects/debian-cloud/global/images/family/debian-12
KEEP_IMAGES_PER_FAMILY: 1
jobs:
resolve-matrix:
name: Resolve variant matrix
runs-on: ubuntu-latest
outputs:
variants: ${{ steps.resolve.outputs.variants }}
steps:
- id: resolve
run: |
requested='${{ github.event.inputs.variants || 'all' }}'
if [[ "$requested" == "all" || -z "$requested" ]]; then
# "build" (docker-build label) is excluded from "all": no job uses it
# since deploy-backend moved to ubuntu-latest + Cloud Build
# (2026-10-01). Still buildable via an explicit variants=build.
echo 'variants=["general","gpu","gpu-windows"]' >> "$GITHUB_OUTPUT"
else
json="$(echo "$requested" | tr ',' '\n' | jq -R . | jq -sc .)"
echo "variants=$json" >> "$GITHUB_OUTPUT"
fi
build:
name: Build ${{ matrix.variant }} image
needs: resolve-matrix
runs-on: ubuntu-latest
timeout-minutes: 90
strategy:
fail-fast: false
matrix:
variant: ${{ fromJSON(needs.resolve-matrix.outputs.variants) }}
steps:
- uses: actions/checkout@v4
- name: Authenticate to GCP
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
with:
project_id: ${{ env.GCP_PROJECT }}
- name: Resolve build parameters
id: params
run: |
variant='${{ matrix.variant }}'
ts="$(date -u +%Y%m%d-%H%M%S)"
name="gha-runner-${variant}-${ts}"
family="gha-runner-${variant}"
vm="gha-image-build-${variant}-${ts}"
# Per-variant base image + startup-script metadata key: Windows
# VMs only execute scripts under windows-startup-script-ps1.
image_flag="--image=${{ env.BASE_IMAGE }}"
metadata_file="startup-script=infrastructure/scripts/runner-image-provision.sh"
case "$variant" in
general|build)
machine="n2-standard-8"
disk_size="50"
accel=""
;;
gpu)
# Build with a T4 attached so DKMS module is built and validated
# against the kernel that ships in the image.
machine="n1-standard-4"
disk_size="200"
accel="--maintenance-policy=TERMINATE --accelerator=type=nvidia-tesla-t4,count=1"
;;
gpu-windows)
# T4 attached so the GRID driver install can validate WDDM mode
# (required for DirectML). Keep disk in sync with the
# gpu-windows FamilySpec in runner_manager/ephemeral.py.
machine="n1-standard-4"
disk_size="100"
accel="--maintenance-policy=TERMINATE --accelerator=type=nvidia-tesla-t4,count=1"
image_flag="--image-family=windows-2022 --image-project=windows-cloud"
metadata_file="windows-startup-script-ps1=infrastructure/scripts/runner-image-provision.ps1"
;;
*)
echo "Unknown variant: $variant" >&2
exit 1
;;
esac
{
echo "name=$name"
echo "family=$family"
echo "vm=$vm"
echo "machine=$machine"
echo "disk_size=$disk_size"
echo "accel=$accel"
echo "image_flag=$image_flag"
echo "metadata_file=$metadata_file"
} >> "$GITHUB_OUTPUT"
- name: Create build VM
run: |
set -euo pipefail
gcloud compute instances create '${{ steps.params.outputs.vm }}' \
--project='${{ env.GCP_PROJECT }}' \
--zone='${{ env.BUILD_ZONE }}' \
--machine-type='${{ steps.params.outputs.machine }}' \
${{ steps.params.outputs.image_flag }} \
--boot-disk-size='${{ steps.params.outputs.disk_size }}GB' \
--boot-disk-type=pd-balanced \
--boot-disk-auto-delete \
--service-account="${{ secrets.GCP_SERVICE_ACCOUNT }}" \
--scopes=https://www.googleapis.com/auth/cloud-platform \
--metadata=enable-oslogin=TRUE,image-variant='${{ matrix.variant }}' \
--metadata-from-file='${{ steps.params.outputs.metadata_file }}' \
--labels=purpose=gha-runner-image-builder,variant='${{ matrix.variant }}' \
${{ steps.params.outputs.accel }}
- name: Wait for provisioning to complete
env:
VM: ${{ steps.params.outputs.vm }}
ZONE: ${{ env.BUILD_ZONE }}
PROJECT: ${{ env.GCP_PROJECT }}
run: |
set -uo pipefail
# Read the serial console — NOT SSH — to detect completion.
#
# The provisioner script tees its full output to /dev/ttyS0 (the GCE
# serial console). `gcloud compute instances get-serial-port-output`
# returns the full console buffer with zero SSH/IAP dependency. This
# makes the wait robust against the IAP tunnel issues that previously
# caused both false-positive readiness and false-negative timeouts.
#
# The script writes one of two distinctive marker lines:
# "### runner-image: READY: ..." — successful completion
# "### runner-image-provision FAILED ..." — fatal error (with line+rc)
#
# We also stream live progress to the GHA log so the operator can
# see what's happening without leaving the run page.
deadline=$(( $(date +%s) + 75 * 60 ))
last_size=0
attempt=0
while (( $(date +%s) < deadline )); do
attempt=$((attempt+1))
console=$(gcloud compute instances get-serial-port-output "$VM" \
--zone="$ZONE" --project="$PROJECT" --port=1 2>/dev/null || true)
cur_size=${#console}
# Stream new provisioner lines (filter syslog noise).
if (( cur_size > last_size )); then
echo "$console" | tail -c +$((last_size+1)) \
| grep -E "runner-image|### |Attempt " || true
last_size=$cur_size
fi
# Success marker
if echo "$console" | grep -qE 'runner-image: READY:'; then
echo
echo "### Image ready (attempt $attempt)"
echo "$console" | grep -E 'runner-image: READY:' | tail -1
exit 0
fi
# Failure marker
if echo "$console" | grep -qE 'runner-image-provision FAILED'; then
echo
echo "### Provisioner reported FAILURE — failure block + 40 lines of preceding context:" >&2
# -B 40: include 40 lines BEFORE the marker (where the actual
# error output of the failing command lives — python tracebacks,
# apt errors, curl messages, etc).
# -A 5: include the variant/line/cmd metadata lines after the marker.
echo "$console" | grep -B 40 -A 5 'runner-image-provision FAILED' \
| tail -80 >&2
exit 1
fi
sleep 30
done
echo
echo "### Timed out waiting for READY marker after 75 minutes" >&2
echo "### Last serial console output (tail):" >&2
gcloud compute instances get-serial-port-output "$VM" \
--zone="$ZONE" --project="$PROJECT" --port=1 2>/dev/null \
| grep -E "runner-image|### " | tail -50 >&2 || true
exit 1
- name: Stop build VM
env:
VM: ${{ steps.params.outputs.vm }}
ZONE: ${{ env.BUILD_ZONE }}
PROJECT: ${{ env.GCP_PROJECT }}
run: |
set -euo pipefail
gcloud compute instances stop "$VM" --zone="$ZONE" --project="$PROJECT" --quiet
# Belt-and-suspenders wait for TERMINATED.
for _ in $(seq 1 60); do
status=$(gcloud compute instances describe "$VM" --zone="$ZONE" --project="$PROJECT" --format='value(status)')
[[ "$status" == "TERMINATED" ]] && break
sleep 5
done
- name: Create image
env:
VM: ${{ steps.params.outputs.vm }}
ZONE: ${{ env.BUILD_ZONE }}
PROJECT: ${{ env.GCP_PROJECT }}
IMAGE: ${{ steps.params.outputs.name }}
FAMILY: ${{ steps.params.outputs.family }}
run: |
set -euo pipefail
# The "Create build VM" step doesn't pass --device-name or
# --boot-disk-name, so GCE auto-names the boot disk after the
# instance. We rely on that here instead of extracting the name
# via a format expression — the previous
# `disks[0].source.scope(disks).segment(1)` selector silently
# returned empty and gcloud rejected the call with
# "You cannot specify [--source-disk-zone] unless you are
# specifying [--source-disk]".
DISK="$VM"
echo "Creating image $IMAGE in family $FAMILY from disk $DISK in $ZONE..."
gcloud compute images create "$IMAGE" \
--project="$PROJECT" \
--source-disk="$DISK" \
--source-disk-zone="$ZONE" \
--family="$FAMILY" \
--description="GHA ephemeral runner image (${{ matrix.variant }}) built ${{ github.run_id }}" \
--labels=purpose=gha-ephemeral-runner,variant=${{ matrix.variant }},built-by=gha
echo "Created $IMAGE in family $FAMILY"
- name: Delete build VM
if: always()
env:
VM: ${{ steps.params.outputs.vm }}
ZONE: ${{ env.BUILD_ZONE }}
PROJECT: ${{ env.GCP_PROJECT }}
run: |
gcloud compute instances delete "$VM" --zone="$ZONE" --project="$PROJECT" --quiet || true
- name: Prune old images (keep newest ${{ env.KEEP_IMAGES_PER_FAMILY }})
if: ${{ github.event.inputs.skip_prune != 'true' }}
env:
PROJECT: ${{ env.GCP_PROJECT }}
FAMILY: ${{ steps.params.outputs.family }}
KEEP: ${{ env.KEEP_IMAGES_PER_FAMILY }}
run: |
set -euo pipefail
# DELETE (not deprecate — deprecated images still bill for storage)
# every image in this family beyond the newest $KEEP. The dispatcher
# boots from the family URL, which always resolves to the newest image,
# so older ones are only rollback insurance (~$0.05/GB-mo; gpu images
# are ~21 GB each). Previously this step used an invalid
# `-deprecated.state:*` filter: gcloud errored to stderr, stdout was
# empty, and it silently reported "Nothing to deprecate" every month
# while images piled up (31 runner images, ~$18/mo, by 2026-09).
# Match the family EXACTLY client-side: a gcloud `family=gha-runner-gpu`
# filter is a word-boundary regex that ALSO matches gha-runner-gpu-windows.
# Rank by ACTIVE (non-deprecated) images: the family URL skips deprecated
# images, so the newest $KEEP active ones are the real rollback path.
# Deprecated images never count toward $KEEP.
rows=$(gcloud compute images list \
--project="$PROJECT" \
--no-standard-images \
--show-deprecated \
--sort-by='~creationTimestamp' \
--format='value(name,family,deprecated.state)' | awk -v f="$FAMILY" '$2==f')
active=$(echo "$rows" | awk 'NF==2 {print $1}')
if [[ -z "$active" ]]; then
echo "::error::No active images found in family $FAMILY (list failed?) — refusing to prune."
exit 1
fi
keep=$(echo "$active" | head -n "$KEEP")
to_delete=$(echo "$rows" | awk '{print $1}' | grep -vxF "$keep" || true)
if [[ -z "$to_delete" ]]; then
echo "Nothing to prune (<= $KEEP images in $FAMILY)."
exit 0
fi
# Never delete an image some existing disk was created from (e.g. a
# long-lived VM still on an older image).
in_use=$(gcloud compute disks list --project="$PROJECT" \
--format='value(sourceImage.basename())' | sort -u)
for img in $to_delete; do
if grep -qx "$img" <<<"$in_use"; then
echo "Keeping $img (source image of an existing disk)"
continue
fi
echo "Deleting $img"
gcloud compute images delete "$img" --project="$PROJECT" --quiet
done
+469 -1488
View File
File diff suppressed because it is too large. Load diff
-180
View File
@@ -1,180 +0,0 @@
name: DR backup freshness monitor
# Off-platform monitor: runs in GitHub (not GCP) so it survives a full GCP
# compromise. If the nightly backup pipeline dies silently, this catches it.
#
# Required secrets:
# AWS_BACKUP_READONLY_ACCESS_KEY_ID
# AWS_BACKUP_READONLY_SECRET_ACCESS_KEY
# DR_MONITOR_DISCORD_WEBHOOK
#
# Setup: see docs/DISASTER-RECOVERY.md § "Backup freshness monitor".
on:
schedule:
# Daily at 14:00 UTC (~9h after the 1am ET backup window).
- cron: "0 14 * * *"
workflow_dispatch: {}
permissions:
contents: read
jobs:
check-freshness:
runs-on: ubuntu-latest
timeout-minutes: 5
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_BACKUP_READONLY_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_BACKUP_READONLY_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: us-east-1
BUCKET: nomadkaraoke-backup
DISCORD_WEBHOOK: ${{ secrets.DR_MONITOR_DISCORD_WEBHOOK }}
steps:
- name: Verify required secrets are set
run: |
missing=0
for var in AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY DISCORD_WEBHOOK; do
if [ -z "${!var}" ]; then
echo "::error::Required secret $var is empty"
missing=1
fi
done
[ "$missing" = "0" ] || exit 1
- name: Install boto3
run: pip install --quiet boto3
- name: Check freshness of each tracked prefix
id: check
shell: bash
run: |
python3 - <<'PY'
import os, sys, datetime, boto3
BUCKET = os.environ["BUCKET"]
# (label, prefix, max_age_hours, require_suffix)
# require_suffix: only objects with this key suffix count toward
# freshness (None = any object).
# NOTE: Firestore and BigQuery export to GCS staging nightly but are
# only uploaded to S3 *weekly* (Sundays) to cut cross-cloud egress —
# see backup_to_aws/main.py (firestore_to_s3_today). This monitor only
# sees the S3 (off-site) copy, so their limits must allow a full week
# plus slack (192h ≈ 8 days), NOT the 36h nightly cadence. Secrets DO
# upload to S3 nightly, so they stay at 36h. GCS job files are NOT
# tracked: their off-site copy was stopped 2026-10-01 (egress cost
# cut) — they rely on GCS versioning + soft-delete only.
# Git repos are also weekly (Sundays) AND incremental since
# 2026-09-26 — only repos pushed that week get a fresh bundle, so a
# quiet week legitimately leaves every .bundle older than 8 days.
# Freshness is judged on git-repos/manifest.json instead (rewritten
# every weekly run), and the manifest's contents are checked below so
# a run where every repo failed still reads as stale.
TARGETS = [
("Firestore", "firestore/", 192, None),
("Encrypted secrets", "secrets/", 36, None),
("BigQuery weekly", "bigquery/daily-refresh/", 192, None),
("Git repos", "git-repos/manifest.json", 192, None),
]
s3 = boto3.client("s3")
now = datetime.datetime.now(datetime.timezone.utc)
stale, ok = [], []
# Paginated scan — JMESPath --query on `aws s3api list-objects-v2`
# operates per-page, which silently breaks "newest across all pages"
# queries on prefixes with >1000 objects (e.g. firestore/ has ~22k).
paginator = s3.get_paginator("list_objects_v2")
for label, prefix, max_hours, require_suffix in TARGETS:
latest = None
for page in paginator.paginate(Bucket=BUCKET, Prefix=prefix):
for obj in page.get("Contents", []):
if require_suffix and not obj["Key"].endswith(require_suffix):
continue
if latest is None or obj["LastModified"] > latest:
latest = obj["LastModified"]
if latest is None:
stale.append(f"{label}: NO OBJECTS in s3://{BUCKET}/{prefix}")
continue
age = (now - latest).total_seconds()
age_h = int(age // 3600)
line = f"{label}: {age_h}h old (limit {max_hours}h) — last {latest.isoformat()}"
(stale if age > max_hours * 3600 else ok).append(line)
# Git manifest contents: ANY bundle error alerts, even though the
# manifest itself is fresh. The git step is weekly + incremental, so
# a repo that failed keeps an ever-older S3 bundle until the next
# Sunday — silence here would let it drift past the 7-day RPO.
try:
import json as _json
m = _json.loads(s3.get_object(Bucket=BUCKET, Key="git-repos/manifest.json")["Body"].read())
if int(m.get("errors", 0)) > 0:
failed = [r.get("full_name") for r in m.get("repos", []) if r.get("status") == "error"]
stale.append(f"Git repos: last run had {m.get('errors')} bundle errors: {', '.join(failed[:10])}")
except Exception as e: # noqa: BLE001
stale.append(f"Git repos: could not read manifest.json ({e})")
summary = ["## DR backup freshness", ""]
if stale:
summary.append("### Stale")
summary.extend(f"- {s}" for s in stale)
summary.append("")
summary.append("### Fresh")
summary.extend(f"- {o}" for o in ok)
with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as f:
f.write("\n".join(summary) + "\n")
with open(os.environ["GITHUB_OUTPUT"], "a") as f:
f.write(f"ok_count={len(ok)}\n")
f.write(f"stale_count={len(stale)}\n")
if stale:
f.write("STALE_LIST<<EOF\n")
f.write("\n".join(stale) + "\n")
f.write("EOF\n")
for o in ok:
print("OK:", o)
if stale:
for s in stale:
print("STALE:", s, file=sys.stderr)
sys.exit(1)
PY
- name: Alert Discord on stale backup
if: failure() && steps.check.outputs.stale_count != ''
env:
STALE: ${{ steps.check.outputs.STALE_LIST }}
run: |
payload=$(jq -n \
--arg title "🚨 DR backup is stale" \
--arg desc "$STALE" \
--arg url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" \
'{
embeds: [{
title: $title,
description: $desc,
color: 15158332,
fields: [{name: "Workflow run", value: $url}]
}]
}')
curl -fsS -X POST -H "Content-Type: application/json" \
-d "$payload" "$DISCORD_WEBHOOK"
- name: Alert Discord on workflow failure (catch-all)
if: failure() && steps.check.outputs.stale_count == ''
run: |
payload=$(jq -n \
--arg title "⚠️ DR freshness monitor failed to run" \
--arg desc "Could not list S3 backup bucket — credentials, network, or bucket policy may be broken." \
--arg url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" \
'{
embeds: [{
title: $title,
description: $desc,
color: 15844367,
fields: [{name: "Workflow run", value: $url}]
}]
}')
curl -fsS -X POST -H "Content-Type: application/json" \
-d "$payload" "$DISCORD_WEBHOOK"
-646
View File
@@ -1,646 +0,0 @@
name: E2E Daily Test (Payment + Happy Path)
# Full production E2E: buy credits via real Stripe, then use them for karaoke generation.
# Replaces the previous e2e-happy-path.yml with an additional payment stage.
on:
schedule:
- cron: '0 6 * * *'
workflow_dispatch:
inputs:
debug_mode:
description: 'Enable debug mode (slower, more screenshots)'
required: false
default: 'false'
type: boolean
browser:
description: 'Browser to test with'
required: false
default: 'chromium'
type: choice
options:
- chromium
- firefox
- webkit
skip_payment:
description: 'Skip Stage 1 (payment) — use admin-granted credits for happy path only'
required: false
default: 'false'
type: boolean
use_test_token:
description: 'Skip signup entirely — use E2E_TEST_TOKEN (for quick iterations)'
required: false
default: 'false'
type: boolean
self_heal_force_llm:
description: 'Self-healing test: comma-separated Stripe step ids to force through the LLM fallback (e.g. selectCard). Learned recipes still open an auto-merge PR.'
required: false
default: ''
type: string
permissions:
contents: read
jobs:
# =========================================================================
# Stage 1: Purchase credits via real Stripe Checkout
# =========================================================================
e2e-credit-purchase:
name: "Stage 1: Credit Purchase"
runs-on: ubuntu-24.04 # Playwright 1.57 can't install chromium on ubuntu-latest (26.04)
timeout-minutes: 15
if: ${{ github.event.inputs.skip_payment != 'true' && github.event.inputs.use_test_token != 'true' }}
permissions:
contents: read
id-token: write # keyless GCP auth, only to read the Gemini key for self-healing
outputs:
session_token: ${{ steps.extract_token.outputs.token }}
result: ${{ steps.test_result.outputs.result }}
failure_reason: ${{ steps.test_result.outputs.failure_reason }}
healed: ${{ steps.self_heal.outputs.healed }}
env:
HAS_GEMINI_SECRET: ${{ secrets.GEMINI_API_KEY != '' }}
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: npx playwright install --with-deps ${{ inputs.browser || 'chromium' }}
- name: Create test-results directory
working-directory: frontend
run: mkdir -p test-results
# ── Self-healing Stripe Checkout: LLM fallback key ───────────────────
# The Stripe step runner (frontend/e2e/helpers/self-healing) falls back
# to Gemini when every deterministic variant of a step fails. The key is
# the AI Studio key (billed outside the nomadkaraoke project). Prefer a
# GEMINI_API_KEY repo secret; otherwise read it from Secret Manager via
# keyless auth. All of this is best-effort: without a key the test runs
# deterministic-only, exactly as before.
- name: Authenticate to Google Cloud (Gemini key)
id: gcp_auth
if: ${{ env.HAS_GEMINI_SECRET != 'true' }}
continue-on-error: true
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Set up Cloud SDK
if: steps.gcp_auth.outcome == 'success'
continue-on-error: true
uses: google-github-actions/setup-gcloud@v2
- name: Load Gemini key for self-healing fallback
continue-on-error: true
env:
GEMINI_SECRET: ${{ secrets.GEMINI_API_KEY }}
GCP_AUTHED: ${{ steps.gcp_auth.outcome == 'success' }}
run: |
KEY="$GEMINI_SECRET"
if [ -z "$KEY" ] && [ "$GCP_AUTHED" = "true" ]; then
KEY=$(gcloud secrets versions access latest --secret=gemini-api-key --project=nomadkaraoke 2>/dev/null || true)
fi
# Drop the deployer credentials before any test code runs.
if [ -n "${GOOGLE_GHA_CREDS_PATH:-}" ]; then rm -f "$GOOGLE_GHA_CREDS_PATH"; fi
if [ -n "${CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE:-}" ]; then rm -f "$CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE"; fi
if [ -n "$KEY" ]; then
echo "::add-mask::$KEY"
echo "GEMINI_API_KEY=$KEY" >> "$GITHUB_ENV"
echo "Self-healing LLM fallback: enabled"
else
echo "::warning::Self-healing LLM fallback disabled (no Gemini key) — Stripe steps run deterministic-only"
fi
- name: Ensure testmail.app domain is allowlisted
run: |
curl -sf -X POST \
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
-H "Content-Type: application/json" \
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
- name: Run Credit Purchase Test
id: run_test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
E2E_STRIPE_CARD_NUMBER: ${{ secrets.E2E_STRIPE_CARD_NUMBER }}
E2E_STRIPE_CARD_EXPIRY: ${{ secrets.E2E_STRIPE_CARD_EXPIRY }}
E2E_STRIPE_CARD_CVC: ${{ secrets.E2E_STRIPE_CARD_CVC }}
E2E_STRIPE_CARDHOLDER_NAME: ${{ secrets.E2E_STRIPE_CARDHOLDER_NAME }}
E2E_STRIPE_ZIP: ${{ secrets.E2E_STRIPE_ZIP }}
E2E_SELF_HEAL_FORCE_LLM: ${{ inputs.self_heal_force_llm || '' }}
# No real card available → 'checkout-only' (no charge). Set the repo variable
# E2E_REAL_PAYMENT=true to go back to buying a credit with E2E_STRIPE_CARD_*.
E2E_PAYMENT_MODE: ${{ vars.E2E_REAL_PAYMENT == 'true' && 'real' || 'checkout-only' }}
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
run: |
echo "=== Stage 1: Credit Purchase via Real Stripe (mode: $E2E_PAYMENT_MODE) ==="
echo "Time: $(date)"
npx playwright test credit-purchase-real.spec.ts \
--config=playwright.production.config.ts \
--project=${{ inputs.browser || 'chromium' }} \
--reporter=list \
--timeout=600000 \
2>&1 | tee test-output-payment.log
# Capture Playwright's real exit code, not tee's (0) — the
# authoritative pass/fail signal. See Stage 2 for rationale.
echo "exit_code=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
continue-on-error: true
- name: Check test result
id: test_result
run: |
if [ "${{ steps.run_test.outputs.exit_code }}" = "0" ]; then
echo "result=success" >> "$GITHUB_OUTPUT"
else
echo "result=failure" >> "$GITHUB_OUTPUT"
# A declined card is an issuer/card problem, not a broken checkout —
# say so in the alerts (marker comes from e2e/helpers/stripe-checkout.ts).
REASON=$(grep -m1 -o 'STRIPE_CARD_DECLINED:.*' frontend/test-output-payment.log 2>/dev/null | sed 's/ — .*//' | tr -d '"\\' || true)
if [ -n "$REASON" ]; then
echo "failure_reason=Test card declined, not a checkout bug (${REASON#STRIPE_CARD_DECLINED: }). Check the card/issuer for a fraud hold, then re-run." >> "$GITHUB_OUTPUT"
echo "::error::$REASON"
fi
fi
- name: Detect self-healed Stripe steps
id: self_heal
if: always()
working-directory: frontend
run: |
REPORT=test-results/self-heal/self-heal-report.json
LEARNED=test-results/self-heal/learned-variants.json
if [ -f "$REPORT" ]; then
{
echo "### Stripe Checkout self-healing"
echo '```json'
jq '{flowVerified, llmCalls, learnedWritten, added, outcomes: [.outcomes[] | {stepId, status, via, variant, model, reason}]}' "$REPORT"
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
fi
# Only promote recipes from a run whose purchase was verified server-side.
if [ "${{ steps.test_result.outputs.result }}" = "success" ] && [ -f "$LEARNED" ]; then
echo "healed=true" >> "$GITHUB_OUTPUT"
echo "::warning::Stripe Checkout changed — LLM fallback healed it; an auto-merge PR will record the new variant"
else
echo "healed=false" >> "$GITHUB_OUTPUT"
fi
- name: Upload self-heal learned variants
if: steps.self_heal.outputs.healed == 'true'
uses: actions/upload-artifact@v4
with:
name: self-heal-learned-${{ github.run_number }}
path: frontend/test-results/self-heal/
retention-days: 30
- name: Extract session token
id: extract_token
if: steps.test_result.outputs.result == 'success'
working-directory: frontend
run: |
if [ -f test-results/e2e-session-token.txt ]; then
TOKEN=$(cat test-results/e2e-session-token.txt)
echo "::add-mask::$TOKEN"
echo "token=$TOKEN" >> $GITHUB_OUTPUT
echo "Session token extracted for Stage 2"
else
echo "No session token file found"
echo "token=" >> $GITHUB_OUTPUT
fi
- name: Scrub secrets from artifacts (public repo)
id: scrub
if: always()
working-directory: frontend
env:
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
E2E_STRIPE_CARD_NUMBER: ${{ secrets.E2E_STRIPE_CARD_NUMBER }}
E2E_STRIPE_CARD_EXPIRY: ${{ secrets.E2E_STRIPE_CARD_EXPIRY }}
E2E_STRIPE_CARD_CVC: ${{ secrets.E2E_STRIPE_CARD_CVC }}
E2E_SESSION_TOKEN: ${{ steps.extract_token.outputs.token }}
run: node e2e/scripts/scrub-artifacts.mjs --drop-videos --blank-inputs test-results test-output-payment.log
- name: Upload artifacts
if: always() && steps.scrub.outcome == 'success'
uses: actions/upload-artifact@v4
with:
name: e2e-payment-results-${{ github.run_number }}
path: |
frontend/test-results/
frontend/test-output-payment.log
retention-days: 30
- name: Discord notification (failure)
if: steps.test_result.outputs.result == 'failure'
env:
# Via env + jq, never ${{ }} in the script: the reason contains Stripe's message text.
REASON: ${{ steps.test_result.outputs.failure_reason || 'Payment flow is broken. Customers may not be receiving credits.' }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
RUN_NUMBER: ${{ github.run_number }}
WEBHOOK: ${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}
run: |
jq -n --arg reason "$REASON" --arg run "$RUN_NUMBER" --arg url "$RUN_URL" \
'{embeds: [{title: "🚨 E2E Daily Test FAILED — Stage 1: Credit Purchase", description: "\($reason)\n\n**Run:** #\($run)\n**Details:** \($url)", color: 15158332}]}' |
curl -s -H "Content-Type: application/json" -d @- "$WEBHOOK"
- name: Fail if test failed
if: steps.test_result.outputs.result == 'failure'
run: exit 1
# =========================================================================
# Self-heal: record LLM-discovered Stripe variants via an auto-merging PR
# =========================================================================
# One Stripe layout change → one LLM-healed run → one data-only PR (only
# learned-variants.json) that auto-merges, so the next run replays the
# recipe deterministically. No human in the loop.
#
# Merge gates for a bot PR (all verified live with #1117, 2026-10-03):
# - its `pull_request` CI run is created "action_required" and, until run,
# supersedes other CI Gate results → we approve it via the API;
# - the `protect main` ruleset's require_extra_approval_for_unattributed_changes
# (GitHub default true since ~Aug 2026) would demand a human review for
# bot-authored changes → set to false on ruleset 11423976;
# - review threads must be resolved → `@coderabbitai ignore` in the body.
self-heal-pr:
name: "Self-heal: auto-merge PR"
runs-on: ubuntu-latest
timeout-minutes: 10
needs: e2e-credit-purchase
# Only learn from main — branch runs (e.g. testing self_heal_force_llm on a
# feature branch) must not open PRs against main.
if: always() && needs.e2e-credit-purchase.outputs.healed == 'true' && github.ref == 'refs/heads/main'
permissions:
contents: write
pull-requests: write
actions: write
outputs:
pr_url: ${{ steps.pr.outputs.pr_url }}
unchanged: ${{ steps.pr.outputs.unchanged }}
steps:
- name: Checkout main
uses: actions/checkout@v4
with:
ref: main
- name: Download learned variants
uses: actions/download-artifact@v4
with:
name: self-heal-learned-${{ github.run_number }}
path: /tmp/self-heal
- name: Open auto-merge PR
id: pr
env:
GH_TOKEN: ${{ github.token }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
set -euo pipefail
TARGET=frontend/e2e/helpers/self-healing/learned-variants.json
# A bot-authored PR's `pull_request` CI run is created as
# "action_required" (awaiting approval) and, until it runs, supersedes
# any other CI Gate result for the ruleset. Approve it; succeed once a
# pull_request run exists that is past approval.
approve_ci() {
local branch=$1 runs pending active
for _ in $(seq 1 18); do
runs=$(gh run list --workflow ci.yml --branch "$branch" --event pull_request --json databaseId,status,conclusion) || { sleep 10; continue; }
pending=$(jq -r '.[] | select(.conclusion == "action_required" or .status == "waiting" or .status == "action_required") | .databaseId' <<<"$runs") || pending=""
active=$(jq '[.[] | select(.conclusion != "action_required" and .status != "waiting" and .status != "action_required")] | length' <<<"$runs") || active=0
for id in $pending; do
if gh api -X POST "repos/${{ github.repository }}/actions/runs/$id/approve" >/dev/null; then
echo "Approved CI run $id"
else
echo "::warning::Could not approve CI run $id with GITHUB_TOKEN"
fi
done
if [ -z "$pending" ] && [ "$active" -gt 0 ]; then
echo "CI is running for $branch"
return 0
fi
sleep 10
done
return 1
}
REPORT=/tmp/self-heal/self-heal-report.json
OPEN=$(gh pr list --label e2e-self-heal --state open --json url --jq '.[0].url // ""')
if [ -n "$OPEN" ]; then
echo "A self-heal PR is already open ($OPEN) — not opening another; making sure its CI can run"
echo "pr_url=$OPEN" >> "$GITHUB_OUTPUT"
OPEN_BRANCH=$(gh pr view "$OPEN" --json headRefName --jq .headRefName || true)
if [ -z "$OPEN_BRANCH" ] || ! approve_ci "$OPEN_BRANCH"; then
echo "ci_needs_approval=true" >> "$GITHUB_OUTPUT"
fi
exit 0
fi
cp /tmp/self-heal/learned-variants.json "$TARGET"
if git diff --quiet -- "$TARGET"; then
echo "learned-variants.json unchanged vs main — nothing to do"
echo "unchanged=true" >> "$GITHUB_OUTPUT"
exit 0
fi
STEPS=$(jq -r '[.outcomes[] | select(.status == "healed") | .stepId] | unique | join(", ")' "$REPORT")
BRANCH="self-heal/stripe-checkout-${{ github.run_id }}"
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git checkout -b "$BRANCH"
git add "$TARGET"
git commit -m "test(e2e): self-heal Stripe Checkout steps: $STEPS" -m "Learned by the LLM fallback in $RUN_URL (purchase verified server-side)."
git push origin "$BRANCH"
gh label create e2e-self-heal --color FBCA04 --description "Auto-generated by the self-healing Stripe E2E" 2>/dev/null || true
{
echo "## 🩹 Stripe Checkout self-heal"
echo
echo "Stripe changed its hosted checkout page. In [run #${{ github.run_number }}]($RUN_URL) every deterministic variant of **$STEPS** failed; the Gemini fallback completed the step(s), and the purchase was then **verified server-side** (credit granted)."
echo
echo "This PR only adds the recipe(s) to \`learned-variants.json\`, so future runs replay them deterministically (no LLM). It auto-merges once CI passes."
echo
echo "### Learned recipes"
echo '```json'
jq '[.outcomes[] | select(.status == "healed") | {stepId, model, llmCalls, actions}]' "$REPORT"
echo '```'
echo
echo "@coderabbitai ignore"
echo
echo "🤖 Generated by the E2E Daily self-healing workflow"
} > /tmp/pr-body.md
PR_URL=$(gh pr create --base main --head "$BRANCH" --label e2e-self-heal \
--title "test(e2e): self-heal Stripe Checkout ($STEPS)" --body-file /tmp/pr-body.md)
echo "pr_url=$PR_URL" >> "$GITHUB_OUTPUT"
gh pr merge "$PR_URL" --auto --squash
if approve_ci "$BRANCH"; then
echo "Opened $PR_URL (auto-merge enabled, CI approved)"
else
# Fallback: dispatched CI (allowed from GITHUB_TOKEN) puts a CI Gate
# check on the head SHA if no pull_request run ever appears.
gh workflow run ci.yml --ref "$BRANCH"
echo "ci_needs_approval=true" >> "$GITHUB_OUTPUT"
echo "::warning::CI for $PR_URL may need a one-click approval (PR → Checks → Approve and run)"
fi
- name: Discord notification (self-heal)
# Recipe already on main (e.g. merged between checkout and now) → nothing to report.
if: always() && steps.pr.outputs.unchanged != 'true'
run: |
PR="${{ steps.pr.outputs.pr_url }}"
# #alerts = needs a human; #noise = informational (auto-merge handles it).
# Noise falls back to the alert webhook if its secret is unset.
ALERT_WEBHOOK="${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
NOISE_WEBHOOK="${{ secrets.DISCORD_NOISE_WEBHOOK_URL || secrets.DISCORD_ALERT_WEBHOOK_URL }}"
if [ -n "$PR" ] && [ "${{ steps.pr.outputs.ci_needs_approval }}" = "true" ]; then
TITLE="🩹 E2E self-healed Stripe — PR needs a CI approval click"
DESC="LLM fallback healed a Stripe Checkout change. The auto-merge PR's CI run awaits approval (open the PR → Checks → Approve and run): $PR"
COLOR=16776960
WEBHOOK="$ALERT_WEBHOOK"
elif [ -n "$PR" ]; then
TITLE="🩹 E2E self-healed a Stripe Checkout change"
DESC="Stripe changed its checkout page; the LLM fallback got through and the purchase was verified. Auto-merge PR: $PR"
COLOR=16776960
WEBHOOK="$NOISE_WEBHOOK"
else
TITLE="⚠️ E2E self-heal PR could not be opened"
DESC="The LLM fallback healed a Stripe step but opening the PR failed — see the run."
COLOR=15105570
WEBHOOK="$ALERT_WEBHOOK"
fi
PAYLOAD="{\"embeds\": [{\"title\": \"$TITLE\", \"description\": \"$DESC\n\n**Run:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": $COLOR}]}"
# A failed #noise post retries on #alerts so it is never silently lost.
curl -fsS -H "Content-Type: application/json" -d "$PAYLOAD" "$WEBHOOK" \
|| { [ "$WEBHOOK" != "$ALERT_WEBHOOK" ] && curl -s -H "Content-Type: application/json" -d "$PAYLOAD" "$ALERT_WEBHOOK"; } \
|| true
# =========================================================================
# Stage 2: Happy Path (uses credit from Stage 1)
# =========================================================================
e2e-happy-path:
name: "Stage 2: Happy Path"
runs-on: ubuntu-24.04 # Playwright 1.57 can't install chromium on ubuntu-latest (26.04)
timeout-minutes: 90
needs: e2e-credit-purchase
if: |
always() && (
needs.e2e-credit-purchase.result == 'success' ||
github.event.inputs.skip_payment == 'true' ||
github.event.inputs.use_test_token == 'true'
)
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: npx playwright install --with-deps ${{ inputs.browser || 'chromium' }}
- name: Create test-results directory
working-directory: frontend
run: mkdir -p test-results
- name: Ensure testmail.app domain is allowlisted
run: |
curl -sf -X POST \
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
-H "Content-Type: application/json" \
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
- name: Refresh E2E session token (when use_test_token enabled)
id: refresh_token
if: github.event.inputs.use_test_token == 'true'
run: |
RESPONSE=$(curl -sf -X POST \
"https://api.nomadkaraoke.com/api/admin/users/e2e-test-runner@nomadkaraoke.com/impersonate" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
-H "Content-Type: application/json")
TOKEN=$(echo "$RESPONSE" | python3 -c 'import json,sys; print(json.load(sys.stdin)["session_token"])')
echo "::add-mask::$TOKEN"
echo "token=$TOKEN" >> $GITHUB_OUTPUT
- name: Run Happy Path Test
id: run_test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_TEST_TOKEN: ${{ steps.refresh_token.outputs.token || needs.e2e-credit-purchase.outputs.session_token || '' }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
run: |
echo "=== Stage 2: Happy Path ==="
echo "Time: $(date)"
TOKEN_SOURCE="none"
if [ -n "${{ steps.refresh_token.outputs.token }}" ]; then
TOKEN_SOURCE="use_test_token (impersonation)"
elif [ -n "${{ needs.e2e-credit-purchase.outputs.session_token }}" ]; then
TOKEN_SOURCE="Stage 1 (credit purchase)"
fi
echo "Token source: $TOKEN_SOURCE"
npx playwright test happy-path-real-user.spec.ts \
--config=playwright.production.config.ts \
--project=${{ inputs.browser || 'chromium' }} \
--reporter=list \
--timeout=3600000 \
2>&1 | tee test-output-happy-path.log
# Capture Playwright's real exit code, not tee's (0). This is the
# authoritative pass/fail signal — grepping the log for "failed" is
# fragile because benign retry warnings (e.g. "goto attempt 1/3
# failed") contain that substring and produce false alarms.
echo "exit_code=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
continue-on-error: true
- name: Check test result
id: test_result
run: |
if [ "${{ steps.run_test.outputs.exit_code }}" = "0" ]; then
echo "result=success" >> "$GITHUB_OUTPUT"
else
echo "result=failure" >> "$GITHUB_OUTPUT"
fi
- name: Scrub secrets from artifacts (public repo)
id: scrub
if: always()
working-directory: frontend
env:
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
E2E_TEST_TOKEN: ${{ steps.refresh_token.outputs.token || needs.e2e-credit-purchase.outputs.session_token || '' }}
run: node e2e/scripts/scrub-artifacts.mjs test-results playwright-report test-output-happy-path.log
- name: Upload test artifacts
if: always() && steps.scrub.outcome == 'success'
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-results-${{ github.run_number }}
path: |
frontend/test-results/
frontend/playwright-report/
frontend/test-output-happy-path.log
retention-days: 30
- name: Upload video recordings
if: always() && steps.scrub.outcome == 'success'
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-videos-${{ github.run_number }}
path: frontend/test-results/**/*.webm
retention-days: 30
if-no-files-found: ignore
- name: Create summary
if: always()
run: |
STAGE1="${{ needs.e2e-credit-purchase.result || 'skipped' }}"
STAGE2="${{ steps.test_result.outputs.result || 'unknown' }}"
echo "## E2E Daily Test Results" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "| Stage | Result |" >> $GITHUB_STEP_SUMMARY
echo "|-------|--------|" >> $GITHUB_STEP_SUMMARY
echo "| Stage 1: Credit Purchase | ${STAGE1} |" >> $GITHUB_STEP_SUMMARY
echo "| Stage 2: Happy Path | ${STAGE2} |" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
- name: Discord notification (failure)
if: steps.test_result.outputs.result == 'failure'
run: |
curl -s -H "Content-Type: application/json" \
-d "{\"embeds\": [{\"title\": \"🚨 E2E Daily Test FAILED — Stage 2: Happy Path\", \"description\": \"Karaoke generation flow is broken.\n\n**Run:** #${{ github.run_number }}\n**Details:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": 15158332}]}" \
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
- name: Fail if test failed
if: steps.test_result.outputs.result == 'failure'
run: exit 1
# =========================================================================
# Final: Send notifications
# =========================================================================
notify:
name: "Notify"
runs-on: ubuntu-latest
needs: [e2e-credit-purchase, e2e-happy-path, self-heal-pr]
if: always()
steps:
- name: Discord notification (success)
# Passes are not signal: they go to #noise (fallback: #alerts if the
# noise secret is unset, so nothing is ever silently dropped).
if: needs.e2e-happy-path.result == 'success'
run: |
STAGE1="${{ needs.e2e-credit-purchase.result || 'skipped' }}"
ALERT_WEBHOOK="${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
NOISE_WEBHOOK="${{ secrets.DISCORD_NOISE_WEBHOOK_URL || secrets.DISCORD_ALERT_WEBHOOK_URL }}"
PAYLOAD="{\"embeds\": [{\"title\": \"✅ E2E Daily Test PASSED\", \"description\": \"Payment: ${STAGE1} | Happy Path: success\n\n**Run:** #${{ github.run_number }}\", \"color\": 3066993}]}"
curl -fsS -H "Content-Type: application/json" -d "$PAYLOAD" "$NOISE_WEBHOOK" \
|| { [ "$NOISE_WEBHOOK" != "$ALERT_WEBHOOK" ] && curl -s -H "Content-Type: application/json" -d "$PAYLOAD" "$ALERT_WEBHOOK"; } \
|| true
- name: Send email notification
if: github.event_name == 'schedule'
uses: dawidd6/action-send-mail@v3
with:
server_address: smtp.postmarkapp.com
server_port: 587
username: ${{ secrets.POSTMARK_SERVER_TOKEN }}
password: ${{ secrets.POSTMARK_SERVER_TOKEN }}
subject: "E2E Daily Test ${{ needs.e2e-happy-path.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
to: andrew@nomadkaraoke.com
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
body: |
E2E Daily Test Results
======================
Stage 1 (Credit Purchase): ${{ needs.e2e-credit-purchase.result || 'skipped' }}${{ vars.E2E_REAL_PAYMENT != 'true' && ' [checkout-only: no real payment]' || '' }}${{ needs.e2e-credit-purchase.outputs.failure_reason && format(' — {0}', needs.e2e-credit-purchase.outputs.failure_reason) || '' }}
Stage 2 (Happy Path): ${{ needs.e2e-happy-path.result || 'skipped' }}
Stripe self-heal: ${{ needs.e2e-credit-purchase.outputs.healed == 'true' && format('Stripe changed its checkout; LLM fallback healed it. Auto-merge PR: {0}', needs.self-heal-pr.outputs.pr_url || (needs.self-heal-pr.outputs.unchanged == 'true' && '(already on main)' || '(failed to open — see run)')) || 'not needed' }}
Run: #${{ github.run_number }}
Trigger: ${{ github.event_name }}
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+174
View File
@@ -0,0 +1,174 @@
name: E2E Happy Path Test
# Comprehensive end-to-end test that validates the full karaoke generation journey
# using ONLY browser interactions (no API shortcuts)
#
# See docs/E2E-HAPPY-PATH-TEST-SPEC.md for full requirements
on:
# Daily run at 6 AM UTC
schedule:
- cron: '0 6 * * *'
# Manual trigger with optional parameters
workflow_dispatch:
inputs:
debug_mode:
description: 'Enable debug mode (slower, more screenshots)'
required: false
default: 'false'
type: boolean
browser:
description: 'Browser to test with'
required: false
default: 'chromium'
type: choice
options:
- chromium
- firefox
- webkit
use_test_token:
description: 'Skip signup and use E2E_TEST_TOKEN (for iterations)'
required: false
default: 'false'
type: boolean
permissions:
contents: read
jobs:
e2e-happy-path:
name: "Full Happy Path (Real User)"
runs-on: ubuntu-latest
timeout-minutes: 90 # 1.5 hours max (test itself is 60 min)
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: |
BROWSER=${{ inputs.browser || 'chromium' }}
npx playwright install --with-deps $BROWSER
- name: Create test-results directory
working-directory: frontend
run: mkdir -p test-results
- name: Run E2E Happy Path Test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_TEST_TOKEN: ${{ inputs.use_test_token && secrets.E2E_TEST_TOKEN || '' }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
run: |
echo "Starting E2E Happy Path Test"
echo "Browser: ${{ inputs.browser || 'chromium' }}"
echo "Debug mode: ${{ inputs.debug_mode || 'false' }}"
echo "Use test token: ${{ inputs.use_test_token || 'false' }}"
echo "Time: $(date)"
echo ""
# Run the real user happy path test
npx playwright test happy-path-real-user.spec.ts \
--config=playwright.production.config.ts \
--project=${{ inputs.browser || 'chromium' }} \
--reporter=list \
--timeout=3600000 \
2>&1 | tee test-output.log
continue-on-error: true
- name: Check test result
id: test_result
working-directory: frontend
run: |
if grep -q "passed" test-output.log && ! grep -q "failed" test-output.log; then
echo "result=success" >> $GITHUB_OUTPUT
echo "Test PASSED"
else
echo "result=failure" >> $GITHUB_OUTPUT
echo "Test FAILED - see artifacts for details"
fi
- name: Upload test artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-results-${{ github.run_number }}
path: |
frontend/test-results/
frontend/playwright-report/
frontend/test-output.log
retention-days: 30
- name: Upload video recordings
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-videos-${{ github.run_number }}
path: frontend/test-results/**/*.webm
retention-days: 30
if-no-files-found: ignore
- name: Create summary
if: always()
run: |
echo "## E2E Happy Path Test Results" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
echo "| Result | ${{ steps.test_result.outputs.result || 'unknown' }} |" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
if [ -f frontend/test-output.log ]; then
echo "### Test Output (last 100 lines)" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
tail -100 frontend/test-output.log >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
fi
- name: Send email notification
if: always() && github.event_name == 'schedule'
uses: dawidd6/action-send-mail@v3
with:
server_address: smtp.sendgrid.net
server_port: 587
username: apikey
password: ${{ secrets.SENDGRID_API_KEY }}
subject: "E2E Happy Path Test ${{ steps.test_result.outputs.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
to: andrew@nomadkaraoke.com
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
body: |
E2E Happy Path Test Results
===========================
Status: ${{ steps.test_result.outputs.result == 'success' && 'PASSED ✅' || 'FAILED ❌' }}
Run: #${{ github.run_number }}
Trigger: ${{ github.event_name }}
Time: ${{ github.event.repository.updated_at }}
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
This is an automated notification from the daily E2E happy path test.
- name: Fail if test failed
if: steps.test_result.outputs.result == 'failure'
run: exit 1
-141
View File
@@ -1,141 +0,0 @@
name: E2E Daily Test (Tenant Portals)
# Full production E2E per white-label tenant: submit a job with the tenant's own
# mixed + instrumental audio, approve the lyrics review, wait for the render, and assert
# the tenant distribution (locked theme, no YouTube, no Google Drive, Dropbox folder +
# downloadable outputs). Mirrors the consumer e2e-daily.yml. Each run cleans up its job.
on:
schedule:
# Daily at 00:07 UTC = ~8pm US Eastern (8:07pm EDT / 7:07pm EST). Cron is fixed
# UTC so the local time shifts 1h across DST. Off the :00 mark to avoid the
# top-of-hour scheduler herd.
- cron: '7 0 * * *'
workflow_dispatch: {}
permissions:
contents: read
id-token: write # for Workload Identity Federation
jobs:
e2e-tenant:
name: "Tenant E2E: ${{ matrix.tenant }}"
runs-on: ubuntu-24.04 # Playwright 1.57 can't install chromium on ubuntu-latest (26.04)
timeout-minutes: 60
strategy:
fail-fast: false
# Run tenants sequentially: the GCE encoder is a single shared worker, and two
# concurrent full renders overwhelm it ("lost contact with worker"). One at a time.
max-parallel: 1
matrix:
tenant: [vocalstar, singa]
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Authenticate to GCP
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: npx playwright install --with-deps chromium
- name: Fetch test audio (private GCS)
run: bash scripts/e2e/fetch_test_audio.sh frontend/e2e/fixtures
- name: Ensure testmail.app domain is allowlisted (rate limit)
run: |
curl -sf -X POST \
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
-H "Content-Type: application/json" \
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
- name: Run tenant E2E (frontend journey)
id: run_test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
TENANT_ID: ${{ matrix.tenant }}
TENANT_PORTAL_URL: https://${{ matrix.tenant }}.nomadkaraoke.com
run: |
echo "=== Tenant E2E (frontend): ${{ matrix.tenant }} ==="
npx playwright test tenant-happy-path.spec.ts \
--config=playwright.production.config.ts --project=chromium \
--reporter=list --timeout=2400000 2>&1 | tee e2e-output.log
echo "exit_code=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
continue-on-error: true
- name: Scrub secrets from artifacts (public repo)
id: scrub
if: always()
working-directory: frontend
env:
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
run: node e2e/scripts/scrub-artifacts.mjs test-results playwright-report e2e-output.log
- name: Upload logs
if: always() && steps.scrub.outcome == 'success'
uses: actions/upload-artifact@v4
with:
name: e2e-tenant-${{ matrix.tenant }}-${{ github.run_number }}
path: |
frontend/e2e-output.log
frontend/test-results/
frontend/playwright-report/
retention-days: 14
- name: Discord notification (failure)
if: ${{ steps.run_test.outputs.exit_code != '0' }}
env:
DISCORD_WEBHOOK: ${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}
run: |
if [ -n "$DISCORD_WEBHOOK" ]; then
curl -s -H "Content-Type: application/json" \
-d "{\"embeds\": [{\"title\": \"❌ Tenant E2E FAILED — ${{ matrix.tenant }}\", \"description\": \"Run: #${{ github.run_number }}\", \"color\": 15158332}]}" \
"$DISCORD_WEBHOOK" || true
fi
- name: Fail the job if the test failed
if: ${{ steps.run_test.outputs.exit_code != '0' }}
run: exit 1
notify-email:
name: "Email summary"
runs-on: ubuntu-latest
needs: e2e-tenant
if: ${{ always() && github.event_name == 'schedule' }}
steps:
- name: Send email notification
uses: dawidd6/action-send-mail@v3
with:
server_address: smtp.postmarkapp.com
server_port: 587
username: ${{ secrets.POSTMARK_SERVER_TOKEN }}
password: ${{ secrets.POSTMARK_SERVER_TOKEN }}
subject: "Tenant E2E Daily ${{ needs.e2e-tenant.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
to: andrew@nomadkaraoke.com
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
body: |
Tenant E2E Daily Test (Vocal Star + Singa)
Result: ${{ needs.e2e-tenant.result }}
Run: #${{ github.run_number }}
Details: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}
-88
View File
@@ -1,88 +0,0 @@
name: PyPI prune reminder
# PyPI caps total *project* size at 10 GB. karaoke-gen publishes a ~62 MiB wheel
# whenever CLI code changes (scripts/pypi_publish_gate.py), so the project slowly
# refills the cap and `poetry publish` then fails with "Project size too large".
#
# PyPI has NO deletion API (upload tokens are upload-only), so this can't
# auto-delete. Instead it runs monthly, computes the retention plan
# (scripts/prune_pypi_releases.py), and opens/updates a single tracking issue
# containing a paste-into-browser-console snippet you run on your logged-in PyPI
# releases page to actually prune. No secrets are stored.
#
# See docs/PYPI-STORAGE-PRUNE.md.
on:
schedule:
# 09:00 UTC on the 1st of each month.
- cron: "0 9 1 * *"
workflow_dispatch: {}
permissions:
contents: read
issues: write
jobs:
prune-reminder:
runs-on: ubuntu-latest
timeout-minutes: 5
env:
GH_TOKEN: ${{ github.token }}
PROJECT: karaoke-gen
KEEP_DAYS: "30"
# Only nag once the project crosses this watermark — pruning holds it at
# ~4-7 GB, and there are always old releases to consolidate, so without a
# threshold this would open an issue every month for no real reason. 7 GB
# leaves ~1 month of ~3 GB/month growth before the 10 GB cap.
NAG_THRESHOLD_GB: "7"
ISSUE_TITLE: "🧹 PyPI storage prune due: karaoke-gen"
steps:
- uses: actions/checkout@v4
- name: Set up Python 3.13
uses: actions/setup-python@v5
with:
python-version: "3.13"
- name: Compute prune plan
id: plan
run: |
python scripts/prune_pypi_releases.py \
--project "$PROJECT" --keep-days "$KEEP_DAYS" \
--format markdown --output plan.md
python scripts/prune_pypi_releases.py \
--project "$PROJECT" --keep-days "$KEEP_DAYS" \
--format json --output plan.json
python - <<'PY' >> "$GITHUB_OUTPUT"
import json, os
d = json.load(open("plan.json"))
threshold = float(os.environ["NAG_THRESHOLD_GB"])
should_nag = d["delete_count"] > 0 and d["current_gb"] >= threshold
print(f"delete_count={d['delete_count']}")
print(f"reclaimable_gb={d['reclaimable_gb']:.2f}")
print(f"current_gb={d['current_gb']:.2f}")
print(f"should_nag={'true' if should_nag else 'false'}")
PY
- name: Job summary
run: cat plan.md >> "$GITHUB_STEP_SUMMARY"
- name: Open or update tracking issue
if: steps.plan.outputs.should_nag == 'true'
run: |
# Search by a stable ASCII substring (emoji in search queries is
# flaky); the exact-title jq filter below is the real dedupe guard.
existing=$(gh issue list --repo "$GITHUB_REPOSITORY" \
--state open --search "PyPI storage prune due in:title" \
--json number,title \
--jq "map(select(.title == \"$ISSUE_TITLE\")) | .[0].number // empty")
note="🔁 Refreshed $(date -u +%Y-%m-%d): ${{ steps.plan.outputs.delete_count }} releases prunable, ~${{ steps.plan.outputs.reclaimable_gb }} GB reclaimable (project at ${{ steps.plan.outputs.current_gb }} GB)."
if [ -n "$existing" ]; then
echo "Updating existing issue #$existing"
gh issue edit "$existing" --repo "$GITHUB_REPOSITORY" --body-file plan.md
gh issue comment "$existing" --repo "$GITHUB_REPOSITORY" --body "$note"
else
echo "Creating new tracking issue"
gh issue create --repo "$GITHUB_REPOSITORY" \
--title "$ISSUE_TITLE" --body-file plan.md
fi
-10
View File
@@ -1,10 +0,0 @@
name: Security
on:
pull_request:
push:
branches: [main]
jobs:
gitleaks:
uses: nomadkaraoke/.github/.github/workflows/gitleaks.yml@main
-8
View File
@@ -13,12 +13,8 @@ tests/outputs
/input
/output/
# Local karaoke-gen CLI test output dirs (created during manual validation)
/test-output-*/
# Cloud Infra
.emulator-logs
.local-stack
# Next.js / React Frontend (./frontend)
node_modules/
@@ -29,9 +25,6 @@ frontend/.swc/
frontend/.turbo/
frontend/build/
frontend/dist/
# Generated Next.js static export copied into the Python package at build time (#876).
# Bundled into the wheel via pyproject `include`; must not be tracked in git.
karaoke_gen/nextjs_frontend/out/
# Next.js specific
frontend/*.tsbuildinfo
@@ -239,4 +232,3 @@ flacfetch
# Test output directories
Rick Astley*/
.superpowers/
-63
View File
@@ -1,63 +0,0 @@
# Nomad Karaoke shared gitleaks config.
# Extends the default ruleset with one custom rule for Discord webhooks
# and an allowlist covering known false-positive paths.
title = "Nomad Karaoke"
[extend]
useDefault = true
# ---------------------------------------------------------------------------
# Custom rules
# ---------------------------------------------------------------------------
[[rules]]
id = "discord-webhook-url"
description = "Discord webhook URL (token embedded in URL)"
regex = '''https://(?:discord|discordapp)\.com/api/webhooks/\d{17,20}/[A-Za-z0-9_-]{60,}'''
tags = ["discord", "webhook", "secret"]
# ---------------------------------------------------------------------------
# Global allowlist
# ---------------------------------------------------------------------------
[allowlist]
description = "Paths and patterns that are known false positives"
paths = [
'''(?i)\.(jpg|jpeg|png|gif|bmp|svg|webp|pdf|ico|mp3|mp4|wav|flac|m4a)$''',
'''(^|/)(package-lock\.json|yarn\.lock|pnpm-lock\.yaml|poetry\.lock|Podfile\.lock|Cargo\.lock|go\.sum|composer\.lock|Gemfile\.lock|uv\.lock)$''',
'''(^|/)node_modules/''',
'''(^|/)vendor/''',
'''(^|/)wp-content/''',
'''(^|/)\.dart_tool/''',
'''(^|/)\.venv/''',
'''(^|/)\.next/''',
'''(^|/)build/''',
'''(^|/)dist/''',
'''(^|/)\.flutter-plugins''',
'''(^|/)test[-_]?fixtures?/''',
'''(^|/)tests?/.*conftest\.py$''',
'''(^|/)tests?/.*test_[^/]+\.py$''',
'''(^|/)tests?/.*_test\.py$''',
'''(^|/).*\.(test|spec)\.(js|mjs|cjs|ts|tsx|jsx)$''',
'''(^|/).*\.test\.local\.(js|mjs|cjs|ts|tsx|jsx|py)$''',
'''(^|/).*\.local\.(js|mjs|cjs|ts|tsx|jsx|py)$''',
]
regexes = [
# Explicit placeholder patterns
'''YOUR_[A-Z_]{2,}''',
'''EXAMPLE_[A-Z_]{2,}''',
'''PLACEHOLDER_[A-Z_]{2,}''',
'''CHANGE[_-]?ME''',
'''<[A-Z_]{2,}>''',
'''\$\{[A-Z_]{2,}\}''',
# Firebase / GCP client API keys — public by design (identify the project,
# access is gated by Firebase Security Rules).
'''AIzaSy[A-Za-z0-9_-]{33}''',
# Documented local-development placeholder token (CONTRIBUTING.md, dev scripts)
'''local-dev-token''',
]
-1
View File
@@ -1 +0,0 @@
[{"name":"generate-buildid","duration":272,"timestamp":244518298649,"id":4,"parentId":1,"tags":{},"startTime":1775262331361,"traceId":"b3ef0d179e8acedf"},{"name":"load-custom-routes","duration":314,"timestamp":244518299029,"id":5,"parentId":1,"tags":{},"startTime":1775262331361,"traceId":"b3ef0d179e8acedf"},{"name":"create-dist-dir","duration":333,"timestamp":244518299363,"id":6,"parentId":1,"tags":{},"startTime":1775262331362,"traceId":"b3ef0d179e8acedf"},{"name":"clean","duration":358,"timestamp":244518300558,"id":7,"parentId":1,"tags":{},"startTime":1775262331363,"traceId":"b3ef0d179e8acedf"},{"name":"next-build","duration":1670851,"timestamp":244516630180,"id":1,"tags":{"buildMode":"default","version":"16.2.2","bundler":"turbopack","failed":true},"startTime":1775262329693,"traceId":"b3ef0d179e8acedf"}]
-1
View File
@@ -1 +0,0 @@
[{"name":"next-build","duration":1670851,"timestamp":244516630180,"id":1,"tags":{"buildMode":"default","version":"16.2.2","bundler":"turbopack","failed":true},"startTime":1775262329693,"traceId":"b3ef0d179e8acedf"}]
+12 -54
View File
@@ -19,17 +19,14 @@
| **Testing & code quality** | `docs/TESTING.md` |
| API reference | `docs/API.md` |
| Past learnings | `docs/LESSONS-LEARNED.md` |
| **Operational runbooks** | `docs/TROUBLESHOOTING.md` |
| GDrive validator & gaps | `docs/GDRIVE-VALIDATOR.md` |
| **Product communication** | `docs/PRODUCT-COMMUNICATION-GUIDE.md` |
| Brand style guide | `docs/BRAND-STYLE-GUIDE.md` |
| **Making medleys** (multi-song karaoke) | `docs/MEDLEYS.md` + `scripts/medley/make_medley.py` |
## Tech Stack
- **Backend**: FastAPI on Cloud Run, Firestore, GCS, Secret Manager
- **Frontend**: Next.js on Cloudflare Pages
- **Processing**: Cloud Run GPU (L4 audio separation), AudioShake/Whisper (transcription)
- **Processing**: Modal (GPU audio separation), AudioShake/Whisper (transcription)
- **Infra**: Pulumi IaC, GitHub Actions CI/CD
## Essential Rules
@@ -68,9 +65,8 @@ This single command:
When asked to "test it yourself in prod" or verify a fix in production:
```bash
# Get admin token. claude-readonly@ can read all secrets (Pulumi:
# infrastructure/modules/iam/claude_readonly_sa.py), so no gcloud login is needed.
export KARAOKE_ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke --account=claude-readonly@nomadkaraoke.iam.gserviceaccount.com | cut -d',' -f1)
# Get admin token
export KARAOKE_ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke | cut -d',' -f1)
# Option 1: Use the debug template directly
cd frontend && node e2e/helpers/debug-prod-template.mjs
@@ -93,32 +89,11 @@ See `docs/TESTING.md` § "Ad-Hoc Production Debugging" for full details.
- Skip for docs-only changes
### Infrastructure
- **All GCP changes via Pulumi PRs** - but CI does **not** run Pulumi (see the note in `.github/workflows/ci.yml`): a merged `infrastructure/` change is not live until someone runs `pulumi up` locally. Claude sessions have read-only ADC, so they need Andrew's `/gcp-admin` grant to apply. Ask for it early, and vet with a targeted `pulumi preview --diff` first
- **`infrastructure/` is the shared platform IaC home**: the GCP project (`infrastructure/`) and the Nomad Karaoke Cloudflare account (`infrastructure/cloudflare/`, own Pulumi project + stack, see its README). Never hand-edit Cloudflare config that stack manages; app repos keep only app-specific stacks
- **Run `pulumi up` locally BEFORE merging infra PRs** - CI only validates; it never applies. If you merge first, main and the live state drift until someone applies
- **All GCP changes via Pulumi PRs** - changes in `infrastructure/` deploy automatically on merge to main
- Never modify GCP resources directly via console or `gcloud` CLI
- `gcloud` CLI for reading/debugging only (e.g., checking logs, SSH to VMs)
- Local `pulumi preview` works but `pulumi up` requires broader permissions than CI
- Stop and notify user on auth issues
- **GCS ops on macOS: never use `gsutil -m`** — its fork-based multiprocessing segfaults
on macOS (crash-report popups + the parent process hangs forever). Use `gcloud storage`
instead (`gcloud storage cp/rm/rsync …`) — no fork, and faster. Plain `gsutil` (no `-m`)
is safe but slower. `~/.boto` pins `parallel_process_count=1` as a safety net; don't remove it.
### Internationalization (i18n)
- All user-facing strings live in `frontend/messages/{locale}.json` (33 locales), NOT in components
- Backend strings live in `backend/translations/{locale}.json` (currently 3 locales: en, es, de)
- Components use `useTranslations('namespace')` from next-intl
- Pages are under `frontend/app/[locale]/` — locale-aware routing (customer-facing: `/`, `/app/*`, `/order/*`, `/auth/*`, `/payment/*`, `/r/*`)
- Non-locale routes at those same paths (e.g. `frontend/app/app/jobs/[[...slug]]/page.tsx`) are `<LocaleRedirect />` shims that redirect into `[locale]` — they are NOT English-only surfaces
- The `/app/jobs/#/{id}/review`, `/instrumental`, and `/audio-edit` review UIs are fully multilingual (components under `components/lyrics-review/`, `components/instrumental-review/`, `components/audio-editor/` use `useTranslations` heavily)
- Only `/admin/*` is English-only — it has no `[locale]/admin` counterpart and inherits `DefaultIntlProvider` from the root layout
- Internal links use `Link` from `@/i18n/routing` (locale-aware)
- After adding/changing English strings: `python frontend/scripts/translate.py --messages-dir frontend/messages --target all`
- Translation uses Gemini 3.8 Flash via the **Gemini Developer API** (key: `GEMINI_API_KEY`, else Secret Manager `gemini-api-key` via your gcloud login — `frontend/scripts/gemini_client.py`) with GCS cache for efficiency. **Vertex AI is disabled in the GCP project — never use `vertexai=True`; backend code uses `backend/services/gemini_client.get_genai_client()`.**
- CI validates all locale files have matching keys — **PR will fail if translations are missing**
- Pre-commit hook auto-translates when en.json changes (enable: `git config core.hooksPath .githooks`)
- Don't hardcode user-facing strings — add to `messages/en.json` and use `t('key')`
## API Authentication (for Agents)
@@ -129,7 +104,7 @@ Use the `X-Admin-Token` header with the token from Secret Manager:
```bash
# Trigger video worker
curl -X POST "https://api.nomadkaraoke.com/api/internal/workers/video" \
-H "X-Admin-Token: $(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke --account=claude-readonly@nomadkaraoke.iam.gserviceaccount.com)" \
-H "X-Admin-Token: $(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke)" \
-H "Content-Type: application/json" \
-d '{"job_id": "YOUR_JOB_ID"}'
@@ -152,33 +127,16 @@ EOF
```
### GCE Encoding Worker
Since 2026-11 the only GCE encoding VM is the stopped on-demand fallback
`encoding-worker-fallback-c2df` (c2d-highcpu-16, us-central1-f); the c4d Spot pair
(`encoding-worker-a`/`-b`) was retired and deleted, and `primary_vm`/`secondary_vm` in
`config/encoding-worker` are empty. The serving worker is `active_override_vm` (normally the
AWS box below). Idle VMs stop after 5 min. CI's "blue-green" deploy now only runs when worker
code changed (`infrastructure/encoding-worker/worker_code_paths.txt`, diffed against
`validated_version`): it boots c2df, runs a test encode, records `validated_version` and stops
it again, leaving the AWS override serving. Force it via Actions → CI → Run workflow →
"Force a blue-green redeploy of the GCE encoding worker".
```bash
# Check health (includes wheel_version) on the GCE fallback.
gcloud compute ssh encoding-worker-fallback-c2df --zone=us-central1-f --project=nomadkaraoke \
--command="curl -s http://localhost:8080/health"
# Restart the service (pick up new worker-side code after a deploy)
gcloud compute ssh encoding-worker-fallback-c2df --zone=us-central1-f --project=nomadkaraoke \
# SSH to restart service (clears in-memory job queue)
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
--command="sudo systemctl restart encoding-worker"
# Check health
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
--command="curl -s http://localhost:8080/health"
```
For job stuck at `encoding` status, see [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md).
**AWS encoding worker** (`aws-encoding-worker-1`, EC2 c7a.4xlarge on-demand, us-east-2) is a pool
candidate too (`provider: aws` in `encoding-worker-fallback-vms`): last resort by default,
preferred when Firestore `config/encoding-worker.speed_rank_overrides = {"c7a": 5}`. Not
blue-green deployed, idle-stops itself, shell via SSM, logs are on the box (journald). See
`infrastructure/aws/README.md`.
### What Doesn't Work
- `gcloud auth print-identity-token` - Wrong token type for internal endpoints
- Cloud Tasks without proper service account - Gets 401 from Cloud Run
-109
View File
@@ -1,109 +0,0 @@
# Contributing to karaoke-gen
Thanks for your interest in contributing! This guide focuses on getting a
working **local development environment with no Google Cloud credentials or
paid API keys** — everything runs against local emulators.
## What runs locally (no credentials needed)
| Component | Local story |
|-----------|-------------|
| FastAPI backend | Runs directly via uvicorn |
| Firestore | Google's Firestore emulator |
| Cloud Storage | [fake-gcs-server](https://github.com/fsouza/fake-gcs-server) (Docker) |
| Background workers (audio, lyrics, screens, video…) | Run as local subprocesses / in-process tasks |
| Audio separation | Local CPU/GPU via `audio-separator` (set `MODEL_DIR`; models download on first use) |
| Lyrics transcription | Local Whisper (`poetry install --extras local-whisper`); AudioShake used only if you set a token |
| Login (magic links) | Printed to the backend console instead of emailed |
| Signed URLs (review audio, downloads, uploads) | Rewritten to plain emulator URLs |
| Frontend (Next.js) | `npm run dev`, pointed at the local backend |
Things that do NOT run locally: payments (Stripe), production email
(Postmark), YouTube/Drive/Dropbox distribution, and the `flacfetch`
URL-download service — **jobs from URLs won't process locally; use file
upload instead.**
## Prerequisites
- Python 3.12/3.13 + [Poetry](https://python-poetry.org/)
- Node.js 20+ (for the frontend)
- Docker (for the GCS emulator)
- Java 21+ and the Google Cloud SDK with the Firestore emulator:
`gcloud components install cloud-firestore-emulator`
(no login/account needed — the emulator runs entirely offline)
- ffmpeg (`brew install ffmpeg` / `apt install ffmpeg`)
## Backend quickstart
```bash
poetry install # heavy: torch etc, ~10 min first time
./scripts/run-backend-local.sh --with-emulators
```
This starts the Firestore emulator (port 8080), fake-gcs-server (port 4443),
creates the bucket, seeds a default theme (`scripts/seed-local-data.py`), and
runs the backend at <http://localhost:8000> (API docs at `/docs`).
Optional API keys go in `.env` (see `.env.example`) — none are required to boot.
### Try it
```bash
# Health
curl http://localhost:8000/api/health
# Create a job from a local audio file (admin token is "local-dev-token")
curl -X POST http://localhost:8000/api/jobs/upload \
-H 'Authorization: Bearer local-dev-token' \
-F "file=@/path/to/song.flac" -F "artist=Some Artist" -F "title=Some Song"
# Watch it process; when status hits awaiting_review, open the review UI
curl http://localhost:8000/api/jobs/<job_id> -H 'Authorization: Bearer local-dev-token'
```
For audio separation to run, set `MODEL_DIR` (e.g. `MODEL_DIR=./models`) —
separation models (~2 GB) are downloaded on first use. Without it the job
fails at the separation step with a clear error.
## Frontend quickstart
```bash
cd frontend
npm install
NEXT_PUBLIC_API_URL=http://localhost:8000 npx next dev
```
Then open <http://localhost:3000>. To log in, enter any email address — the
magic link is printed in the **backend** console (look for `EMAIL TO:`); open
that link in your browser.
### Review UI without running the full pipeline
If you're working on the lyrics review UI specifically, there's a fixtures
harness that serves real correction data through the review API shape without
processing any audio:
```bash
python scripts/review_test_fixtures.py # serves on :8765
```
## Tests
```bash
make test # everything (backend + frontend)
make test-backend # backend only (unit + emulator tests)
```
Emulator-backed integration tests live in `backend/tests/emulator/` and use
the same two emulators as the local dev setup.
## Pull requests
- Keep PRs focused; include tests for behavior changes.
- Run `make test` before pushing — CI runs the same suite.
- Bump `tool.poetry.version` in `pyproject.toml` for code changes.
- CI must pass before merge; a maintainer will review from there.
See `docs/DEVELOPMENT.md` for deeper docs (architecture, deployment,
observability) — most of that file concerns the production GCP deployment,
which contributors don't need.
+11 -53
View File
@@ -1,25 +1,18 @@
# Makefile for karaoke-gen project
# Run `make help` to see available commands
.PHONY: help install install-backend install-frontend build-frontend dev-install test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop local-stack-up local-stack-down test-local-stack eval-custom-lyrics
# Per-stage timeout in seconds (default: 10 minutes)
# Prevents hangs from stale dev servers, broken emulators, etc.
# Override with: make test TEST_TIMEOUT=300
TEST_TIMEOUT ?= 600
.PHONY: help install install-backend install-frontend build-frontend test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop
# Default target
help:
@echo "Available commands:"
@echo " make test - Run ALL tests (backend + frontend, installs deps automatically)"
@echo " make test-backend - Run backend tests only (unit + emulator)"
@echo " make test-frontend - Run frontend unit tests (Jest)"
@echo " make test-frontend-e2e - Run frontend E2E tests (Playwright, requires dev server)"
@echo " make test-frontend - Run frontend tests only (unit + E2E)"
@echo " make test-unit - Run unit tests only (karaoke_gen package)"
@echo " make test-e2e - Run emulator tests with auto-start/stop"
@echo " make install - Install all dependencies (backend + frontend)"
@echo " make build-frontend - Build frontend and copy to Python package"
@echo " make dev-install - Build frontend + pip install -e . (for local CLI testing)"
@echo " make build-frontend - Build frontend and copy to Python package (for local testing)"
@echo " make lint - Run linter checks"
@echo " make emulators-start - Start GCP emulators for local development"
@echo " make emulators-stop - Stop GCP emulators"
@@ -41,38 +34,23 @@ install-frontend:
install: install-backend install-frontend
# Build frontend and copy to Python package
# Build frontend and copy to Python package (for local CLI testing)
build-frontend: install-frontend
@echo "=== Building frontend ==="
cd frontend && npm run build
@echo "=== Copying build to Python package ==="
rm -rf karaoke_gen/nextjs_frontend/out
mkdir -p karaoke_gen/nextjs_frontend/out
cp -r frontend/out/* karaoke_gen/nextjs_frontend/out/
@# Drop Next.js RSC payloads (*.txt, ~66 MB across 33 locales): the CLI review
@# server never serves them (only HTML entrypoints, /_next and /static), and they
@# pushed the wheel past PyPI's 100 MiB per-file limit (2026-10-27..29).
find karaoke_gen/nextjs_frontend/out -name '*.txt' -type f -delete
@echo "✅ Frontend built and copied to karaoke_gen/nextjs_frontend/out/"
# Build frontend + install package in editable mode for local CLI testing
# After running this, `karaoke-gen` CLI will use the freshly built frontend.
dev-install: build-frontend
@echo "=== Installing package in editable mode ==="
pip install -e .
@echo ""
@echo "✅ Ready! You can now run karaoke-gen with the updated frontend."
@echo " Example: karaoke-gen input.flac \"Artist\" \"Title\""
@echo "✅ Frontend built and ready for local testing"
# Run unit tests for karaoke_gen package
test-unit: install-backend
@echo "=== Running karaoke_gen unit tests ==="
timeout $(TEST_TIMEOUT) poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
# Run backend unit tests (excludes emulator tests)
test-backend-unit: install-backend
@echo "=== Running backend unit tests ==="
timeout $(TEST_TIMEOUT) poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
# Run E2E integration tests with emulators
test-e2e: install-backend
@@ -84,7 +62,7 @@ test-e2e: install-backend
export GCS_BUCKET_NAME=test-bucket && \
export ADMIN_TOKENS=test-admin-token && \
export ENVIRONMENT=test && \
timeout $(TEST_TIMEOUT) poetry run pytest backend/tests/emulator/ -v; \
poetry run pytest backend/tests/emulator/ -v; \
TEST_RESULT=$$?; \
./scripts/stop-emulators.sh; \
exit $$TEST_RESULT
@@ -94,15 +72,10 @@ test-backend: test-unit test-backend-unit test-e2e
@echo ""
@echo "✅ Backend tests passed!"
# Run frontend unit tests (Jest only - E2E requires manual setup)
# Run frontend tests (unit + E2E)
test-frontend: install-frontend
@echo "=== Running frontend unit tests ==="
cd frontend && timeout $(TEST_TIMEOUT) npm run test:unit
# Run frontend E2E tests (requires dev server - run manually, not in make test)
test-frontend-e2e: install-frontend
@echo "=== Running frontend E2E tests ==="
cd frontend && timeout $(TEST_TIMEOUT) npm run test:e2e
@echo "=== Running frontend tests ==="
cd frontend && npm run test:all
# Run ALL tests (backend + frontend) - use this before committing!
test: test-backend test-frontend
@@ -128,17 +101,6 @@ emulators-stop:
@echo "=== Stopping GCP emulators ==="
./scripts/stop-emulators.sh
# Local E2E stack: real backend on emulators + fake workers (see docs/TESTING.md)
local-stack-up:
./scripts/local-stack.sh up --frontend
local-stack-down:
./scripts/local-stack.sh down
# Full-journey Playwright suite against the local stack (starts it if needed)
test-local-stack: install-frontend
cd frontend && timeout $(TEST_TIMEOUT) npm run test:e2e:local-stack
# Clean up temporary files
clean:
find . -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
@@ -148,7 +110,3 @@ clean:
rm -rf htmlcov/ .coverage coverage.xml 2>/dev/null || true
@echo "Cleaned up temporary files"
.PHONY: eval-custom-lyrics
eval-custom-lyrics:
cd backend && poetry run python -m backend.eval.custom_lyrics.run
+1 -3
View File
@@ -675,6 +675,4 @@ MIT
## 🤝 Contributing
Contributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) — the whole
backend runs locally against emulators with no Google Cloud credentials or
paid API keys required.
Contributions are welcome! Please see our contributing guidelines.
-2
View File
@@ -29,8 +29,6 @@ tmp/
# Google Cloud
.gcloud/
*.json
# Allow i18n translation files
!translations/*.json
# Testing
.pytest_cache/
+1 -13
View File
@@ -26,19 +26,7 @@ COPY backend /app/backend
# Note: lyrics_transcriber is now a subpackage of karaoke_gen
COPY karaoke_gen /app/karaoke_gen
COPY pyproject.toml README.md LICENSE /app/
# GPU images (MODEL_DIR set in base): use --no-deps to preserve pinned torch cu126
# and audio-separator[gpu]. Without this, pip upgrades torch to cu130 (breaks CUDA
# on Cloud Run L4 drivers) and reinstalls audio-separator[cpu] (breaks GPU ONNX).
# CPU images: normal install resolves all deps from pyproject.toml.
RUN if [ -n "$MODEL_DIR" ]; then \
pip install --no-cache-dir --no-deps -e .; \
else \
pip install --no-cache-dir -e .; \
fi
# Precompile bytecode so lazy imports never write .pyc at runtime: a cold
# import that compiles + writes on the event loop has stalled requests 10-15s.
RUN python -m compileall -q /app/backend /app/karaoke_gen
RUN pip install --no-cache-dir -e .
# Set environment variables
ENV PYTHONUNBUFFERED=1
+7 -38
View File
@@ -1,15 +1,8 @@
# Base image with all Python dependencies pre-installed
# This image is rebuilt only when poetry.lock or this Dockerfile changes
#
# This image is rebuilt only when poetry.lock changes
#
# Build with: gcloud builds submit --config=cloudbuild-base.yaml
# Or manually: docker build -f backend/Dockerfile.base -t karaoke-backend-base .
#
# Layer caching strategy:
# 1. System deps + fonts + ffmpeg (rarely change, always cached)
# 2. pyproject.toml + package stubs (changes with dep updates)
# 3. pip install (the slow part - cached unless pyproject.toml changes)
# 4. Source code copy (changes every rebuild, but AFTER deps are installed)
# 5. spaCy model download (cached unless pip install layer changes)
FROM python:3.11-slim
@@ -23,7 +16,6 @@ RUN apt-get update && apt-get install -y \
build-essential \
curl \
xz-utils \
chromium \
&& rm -rf /var/lib/apt/lists/*
# Install comprehensive font support for international lyrics and symbols
@@ -58,39 +50,16 @@ RUN curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-stati
# Set working directory
WORKDIR /app
# Copy ONLY dependency specification files first (changes less often than source code)
# Copy dependency files and source code needed for pip install
COPY pyproject.toml README.md LICENSE /app/
# Create minimal package stubs so pip can resolve the editable install
# pip install -e . needs package directories to exist but doesn't validate content
RUN mkdir -p /app/karaoke_gen /app/lyrics_transcriber_temp /app/backend && \
touch /app/karaoke_gen/__init__.py /app/lyrics_transcriber_temp/__init__.py /app/backend/__init__.py
# Install all Python dependencies (the slow part)
# BuildKit cache mount preserves downloaded wheels across builds, so partial
# dependency updates only download changed packages instead of everything
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --upgrade pip && \
pip install -e .
# Pre-build matplotlib's font cache at BUILD time. Without it, the first
# `import matplotlib.pyplot` on every new Cloud Run instance scanned all the
# Noto fonts above (20-40s of GIL-holding CPU) and froze the API's event loop
# — prod logged "Matplotlib is building the font cache" in 17 of 28
# "servers unavailable" banner episodes (2026-10-03 analysis). The cache lives
# in MPLCONFIGDIR so it ships in the image instead of the in-memory $HOME.
ENV MPLCONFIGDIR=/opt/mplconfig
RUN mkdir -p /opt/mplconfig && \
python -c "import matplotlib; matplotlib.use('Agg'); import matplotlib.font_manager as fm; print(len(fm.fontManager.ttflist), 'fonts cached')" && \
ls /opt/mplconfig/fontlist-*.json && \
chmod -R a+rwX /opt/mplconfig
# Now copy actual source code (AFTER deps installed, so code changes don't
# invalidate the expensive pip install layer in registry cache)
COPY karaoke_gen /app/karaoke_gen
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
COPY backend /app/backend
# Install all Python dependencies (the slow part - cached in base image)
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir -e .
# Download spaCy language model for lyrics transcription
# This must be done at build time, not runtime (Cloud Run network restrictions)
RUN python -m spacy download en_core_web_sm
-144
View File
@@ -1,144 +0,0 @@
# GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models
# This image is rebuilt only when deps or models change (infrequently).
#
# Build with: gcloud builds submit --config=backend/cloudbuild-gpu-base.yaml --project=nomadkaraoke --timeout=3600s
# Or manually: docker build -f backend/Dockerfile.gpu-base -t karaoke-backend-gpu-base .
#
# Layer caching strategy:
# 1. System deps + fonts + ffmpeg (rarely change, always cached)
# 2. CUDA library path configuration
# 3. pyproject.toml + package stubs (changes with dep updates)
# 4. pip install karaoke-gen deps + audio-separator[gpu] (slow, cached unless pyproject.toml changes)
# 5. Source code copy (changes every rebuild, but AFTER deps are installed)
# 6. spaCy model download (cached unless pip install layer changes)
# 7. Pre-downloaded ensemble preset models (~1-1.5 GB, cached unless pip layer changes)
FROM nvidia/cuda:12.6.3-runtime-ubuntu22.04
# Prevent interactive prompts during package installation
ENV DEBIAN_FRONTEND=noninteractive
# Install Python 3.11 from deadsnakes PPA and system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
software-properties-common \
&& add-apt-repository -y ppa:deadsnakes/ppa \
&& apt-get update && apt-get install -y --no-install-recommends \
# Python 3.11
python3.11 \
python3.11-dev \
python3.11-venv \
# Audio libraries
libsndfile1 \
libsox-dev \
sox \
# Build tools
build-essential \
# Utilities
curl \
xz-utils \
&& rm -rf /var/lib/apt/lists/* \
&& python3.11 --version
# Install comprehensive font support for international lyrics and symbols
# Noto fonts provide full Unicode coverage including:
# - Latin, Greek, Cyrillic (core)
# - CJK: Chinese, Japanese, Korean (~150MB)
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
# - Color emoji (limited support in libass, but available)
# - Musical symbols (♪ ♫ etc.)
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
RUN apt-get update && apt-get install -y --no-install-recommends \
fonts-noto-core \
fonts-noto-cjk \
fonts-noto-extra \
fonts-noto-color-emoji \
fontconfig \
&& fc-cache -f \
&& rm -rf /var/lib/apt/lists/*
# Install FFmpeg from Ubuntu package
# The audio worker only uses FFmpeg for format conversion and padding, not
# encoding-intensive tasks, so the Debian package is sufficient. The static
# build from johnvansickle.com is used for the encoding worker instead.
RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*
# Set Python 3.11 as default and install pip
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1 \
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.11 1 \
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11 \
&& python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel
# Set up CUDA library paths so GPU-accelerated packages can find CUDA libs at runtime
RUN echo '/usr/local/cuda/lib64' >> /etc/ld.so.conf.d/cuda.conf && ldconfig
# Environment configuration for CUDA and Python
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH \
PATH=/usr/local/cuda/bin:$PATH \
PYTHONUNBUFFERED=1 \
MODEL_DIR=/models
# Set working directory
WORKDIR /app
# Copy ONLY dependency specification files first (changes less often than source code)
COPY pyproject.toml README.md LICENSE /app/
# Create minimal package stubs so pip can resolve the editable install
# pip install -e . needs package directories to exist but doesn't validate content
RUN mkdir -p /app/karaoke_gen /app/lyrics_transcriber_temp /app/backend && \
touch /app/karaoke_gen/__init__.py /app/lyrics_transcriber_temp/__init__.py /app/backend/__init__.py
# Install all karaoke-gen Python dependencies (the slow part)
# BuildKit cache mount preserves downloaded wheels across builds
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --upgrade pip && \
pip install -e .
# Install audio-separator with GPU support (PyTorch + CUDA + onnxruntime-gpu)
# This is separate from pip install -e . to keep layer caching granular
#
# IMPORTANT: Install PyTorch cu126 FIRST, then audio-separator[gpu].
# The default audio-separator[gpu] pulls PyTorch cu130, but Cloud Run L4 GPUs
# have NVIDIA driver 570 which only supports up to CUDA 12.8. PyTorch cu130
# fails with "NVIDIA driver is too old". PyTorch cu126 works with 12.8 drivers.
# Installing torch first means audio-separator[gpu] sees it as already satisfied.
#
# PINNED to an exact version: an unpinned install silently picked up 0.47.0 on the
# 2026-10-17 rebuild and separation went from ~5 to ~10-16 min. Cause: 0.47 (#299) made
# RoFormer `overlap` a window count defaulting to each model YAML's num_overlap (4-8 for
# the karaoke models = 4-6x the forward passes of 0.44's fixed 8 s step). karaoke_gen's
# audio_processor now passes overlap=2 on >= 0.47 (SEPARATOR_MDXC_PARAMS). Time a real
# job before bumping this, and read the audio-separator release notes for compute changes.
ARG AUDIO_SEPARATOR_VERSION=0.47.1
RUN --mount=type=cache,target=/root/.cache/pip \
pip install torch==2.6.0+cu126 torchvision==0.21.0+cu126 --index-url https://download.pytorch.org/whl/cu126 && \
pip install "audio-separator[gpu]==${AUDIO_SEPARATOR_VERSION}"
# Now copy actual source code (AFTER deps installed, so code changes don't
# invalidate the expensive pip install layer in registry cache)
COPY karaoke_gen /app/karaoke_gen
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
COPY backend /app/backend
# Download spaCy language model for lyrics transcription
# Must be done at build time, not runtime (Cloud Run network restrictions)
RUN python -m spacy download en_core_web_sm
# Create models directory and pre-download ensemble preset models
# Baking models into the image eliminates cold-start model download latency.
# Total model size: ~1-1.5 GB for instrumental_clean + karaoke presets.
RUN mkdir -p /models
COPY backend/scripts/download_models.py /tmp/download_models.py
RUN python /tmp/download_models.py /models && \
rm /tmp/download_models.py && \
ls -lh /models/
# Label with build info
ARG BUILD_DATE
ARG BUILD_ID
ARG BASE_CONTENT_HASH
LABEL org.opencontainers.image.created="${BUILD_DATE}"
LABEL org.opencontainers.image.revision="${BUILD_ID}"
LABEL org.opencontainers.image.title="karaoke-backend-gpu-base"
LABEL org.opencontainers.image.description="GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models for karaoke-backend audio worker"
LABEL base.content.hash="${BASE_CONTENT_HASH}"
+23 -40
View File
@@ -1,7 +1,6 @@
"""
FastAPI dependencies for authentication and authorization.
"""
import asyncio
import logging
import secrets
from datetime import datetime, timedelta, UTC
@@ -90,8 +89,7 @@ async def require_auth(
)
# Validate token using the full method
# Sync Firestore reads: run off the event loop (every authed request passes here).
auth_result = await asyncio.to_thread(auth_service.validate_token_full, token_str)
auth_result = auth_service.validate_token_full(token_str)
# Get request_id from middleware for correlation
request_id = getattr(request.state, "request_id", None)
@@ -115,19 +113,6 @@ async def require_auth(
detail=f"Authentication failed: {auth_result.message}"
)
# If token has a tenant_id, override the hostname-based tenant detection.
# This allows tenant-scoped tokens to enforce tenant context regardless of domain.
# We must also load tenant_config so downstream handlers (e.g. job creation)
# that rely on tenant_config for setting tenant_id on jobs work correctly.
if auth_result.tenant_id:
request.state.tenant_id = auth_result.tenant_id
if not getattr(request.state, "tenant_config", None):
from backend.services.tenant_service import get_tenant_service
tenant_service = get_tenant_service()
tenant_config = await asyncio.to_thread(tenant_service.get_tenant_config, auth_result.tenant_id)
if tenant_config and tenant_config.is_active:
request.state.tenant_config = tenant_config
# Log successful auth with request_id for correlation with request audit
logger.info(
"auth_success",
@@ -137,7 +122,6 @@ async def require_auth(
"user_type": auth_result.user_type.value if auth_result.user_type else None,
"is_admin": auth_result.is_admin,
"remaining_uses": auth_result.remaining_uses,
"tenant_id": auth_result.tenant_id,
"audit_type": "auth_event",
}
)
@@ -167,7 +151,7 @@ async def require_auth_legacy(
detail="Authentication required. Provide token via Authorization header or ?token= parameter"
)
is_valid, user_type, remaining_uses, message = await asyncio.to_thread(auth_service.validate_token, token_str)
is_valid, user_type, remaining_uses, message = auth_service.validate_token(token_str)
if not is_valid:
raise HTTPException(
@@ -286,39 +270,38 @@ def require_review_auth_factory(job_id_param: str = "job_id"):
full_token = token
if full_token:
auth_result = await asyncio.to_thread(auth_service.validate_token_full, full_token)
auth_result = auth_service.validate_token_full(full_token)
if auth_result.is_valid:
# For full auth, also verify job ownership
job = await asyncio.to_thread(job_manager.get_job, job_id)
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Admin, owner, or a member of the job's tenant portal
# (backend/services/job_access.py).
from backend.services.job_access import can_access_job
if not await asyncio.to_thread(
can_access_job, job, auth_result.user_email, bool(auth_result.is_admin)
):
logger.warning(
f"Review access denied: user {auth_result.user_email} "
f"tried to access job {job_id} owned by {job.user_email}"
)
raise HTTPException(
status_code=403,
detail="You don't have permission to access this job's review"
)
# Check ownership: admin can access all, users only their own jobs
if not auth_result.is_admin:
if auth_result.user_email and job.user_email:
if auth_result.user_email.lower() != job.user_email.lower():
logger.warning(
f"Review access denied: user {auth_result.user_email} "
f"tried to access job {job_id} owned by {job.user_email}"
)
raise HTTPException(
status_code=403,
detail="You don't have permission to access this job's review"
)
elif job.user_email:
# Token auth without email trying to access a job with owner
raise HTTPException(
status_code=403,
detail="You don't have permission to access this job's review"
)
logger.info(f"Review access granted via full auth ({auth_result.user_type}) for job {job_id}")
# Who is reviewing — handlers record it on first open
# (state_data.review_started_by, see JobManager.record_review_started).
request.state.review_auth_email = auth_result.user_email
request.state.review_auth_is_admin = bool(auth_result.is_admin)
return job_id, "full"
# Try review token
if review_token:
job = await asyncio.to_thread(job_manager.get_job, job_id)
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
+148 -2870
View File
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+38 -44
View File
@@ -9,7 +9,7 @@ Provides endpoints for:
import logging
from typing import Optional
from datetime import datetime
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request
from fastapi import APIRouter, HTTPException, BackgroundTasks
from pydantic import BaseModel
from backend.services.credential_manager import (
@@ -18,7 +18,6 @@ from backend.services.credential_manager import (
CredentialCheckResult,
)
from backend.config import get_settings
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/auth", tags=["auth"])
@@ -91,7 +90,7 @@ class CredentialValidationResponse(BaseModel):
# ===========================================================================
@router.get("/status", response_model=AllCredentialsStatusResponse)
def get_credentials_status():
async def get_credentials_status():
"""
Get the status of all OAuth credentials.
@@ -126,16 +125,15 @@ def get_credentials_status():
@router.get("/status/{service}", response_model=CredentialStatusResponse)
def get_service_credential_status(service: str, request: Request):
async def get_service_credential_status(service: str):
"""
Get the status of a specific service's OAuth credentials.
Args:
service: One of "youtube", "gdrive", "dropbox"
"""
locale = get_locale_from_request(request)
manager = get_credential_manager()
if service == "youtube":
result = manager.check_youtube_credentials()
elif service == "gdrive":
@@ -143,7 +141,7 @@ def get_service_credential_status(service: str, request: Request):
elif service == "dropbox":
result = manager.check_dropbox_credentials()
else:
raise HTTPException(status_code=400, detail=t(locale, "auth.unknownService", service=service))
raise HTTPException(status_code=400, detail=f"Unknown service: {service}")
return CredentialStatusResponse(
service=result.service,
@@ -155,43 +153,42 @@ def get_service_credential_status(service: str, request: Request):
@router.post("/validate", response_model=CredentialValidationResponse)
def validate_credentials(request: CredentialValidationRequest, http_request: Request):
async def validate_credentials(request: CredentialValidationRequest):
"""
Validate that credentials are available for requested services.
Use this before submitting a job to ensure all required
credentials are valid.
"""
locale = get_locale_from_request(http_request)
manager = get_credential_manager()
invalid_services = []
if request.youtube:
result = manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append("youtube")
if request.gdrive:
result = manager.check_gdrive_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append("gdrive")
if request.dropbox:
result = manager.check_dropbox_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append("dropbox")
if invalid_services:
return CredentialValidationResponse(
valid=False,
invalid_services=invalid_services,
message=t(locale, "auth.invalidCredentials", services=", ".join(invalid_services))
message=f"The following services need re-authorization: {', '.join(invalid_services)}"
)
return CredentialValidationResponse(
valid=True,
invalid_services=[],
message=t(locale, "auth.validCredentials")
message="All requested credentials are valid"
)
@@ -200,45 +197,44 @@ def validate_credentials(request: CredentialValidationRequest, http_request: Req
# ===========================================================================
@router.post("/youtube/device", response_model=DeviceAuthStartResponse)
def start_youtube_device_auth(request: Optional[DeviceAuthStartRequest] = None, http_request: Request = None):
async def start_youtube_device_auth(request: Optional[DeviceAuthStartRequest] = None):
"""
Start YouTube device authorization flow.
This initiates a device auth flow that allows authorization
from any device. The user must visit the verification URL
and enter the user code.
Client credentials are loaded from Secret Manager ('youtube-client-credentials')
unless explicitly provided in the request body.
After starting, poll /auth/youtube/device/{device_code} to
check for completion.
"""
locale = get_locale_from_request(http_request) if http_request else "en"
manager = get_credential_manager()
try:
device_info = manager.start_youtube_device_auth(
client_id=request.client_id if request else None,
client_secret=request.client_secret if request else None
)
return DeviceAuthStartResponse(
device_code=device_info.device_code,
user_code=device_info.user_code,
verification_url=device_info.verification_url,
expires_in=device_info.expires_in,
interval=device_info.interval,
instructions=t(locale, "auth.youtubeAuthInstructions", verification_url=device_info.verification_url, user_code=device_info.user_code)
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
)
except Exception as e:
logger.error(f"Failed to start YouTube device auth: {e}")
raise HTTPException(status_code=500, detail=t(locale, "auth.youtubeAuthStartFailed", error=str(e)))
raise HTTPException(status_code=500, detail=str(e))
@router.get("/youtube/device/{device_code}", response_model=DeviceAuthPollResponse)
def poll_youtube_device_auth(device_code: str):
async def poll_youtube_device_auth(device_code: str):
"""
Poll for YouTube device authorization completion.
@@ -263,40 +259,39 @@ def poll_youtube_device_auth(device_code: str):
@router.post("/gdrive/device", response_model=DeviceAuthStartResponse)
def start_gdrive_device_auth(request: Optional[DeviceAuthStartRequest] = None, http_request: Request = None):
async def start_gdrive_device_auth(request: Optional[DeviceAuthStartRequest] = None):
"""
Start Google Drive device authorization flow.
Client credentials are loaded from Secret Manager ('gdrive-client-credentials')
unless explicitly provided in the request body.
After starting, poll /auth/gdrive/device/{device_code} to check for completion.
"""
locale = get_locale_from_request(http_request) if http_request else "en"
manager = get_credential_manager()
try:
device_info = manager.start_gdrive_device_auth(
client_id=request.client_id if request else None,
client_secret=request.client_secret if request else None
)
return DeviceAuthStartResponse(
device_code=device_info.device_code,
user_code=device_info.user_code,
verification_url=device_info.verification_url,
expires_in=device_info.expires_in,
interval=device_info.interval,
instructions=t(locale, "auth.gdriveLinkInstructions", verification_url=device_info.verification_url, user_code=device_info.user_code)
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
)
except Exception as e:
logger.error(f"Failed to start Google Drive device auth: {e}")
raise HTTPException(status_code=500, detail=t(locale, "auth.gdriveAuthStartFailed", error=str(e)))
raise HTTPException(status_code=500, detail=str(e))
@router.get("/gdrive/device/{device_code}", response_model=DeviceAuthPollResponse)
def poll_gdrive_device_auth(device_code: str):
async def poll_gdrive_device_auth(device_code: str):
"""
Poll for Google Drive device authorization completion.
"""
@@ -316,20 +311,19 @@ def poll_gdrive_device_auth(device_code: str):
# ===========================================================================
@router.post("/test-alert")
def test_credential_alert(background_tasks: BackgroundTasks, request: Request):
async def test_credential_alert(background_tasks: BackgroundTasks):
"""
Test the credential alert mechanism.
Sends a test alert to Discord if configured.
"""
locale = get_locale_from_request(request)
settings = get_settings()
discord_url = settings.get_secret("discord-alert-webhook")
if not discord_url:
raise HTTPException(
status_code=400,
detail=t(locale, "auth.discordAlertNotConfigured")
detail="Discord alert webhook not configured"
)
manager = get_credential_manager()
-528
View File
@@ -1,528 +0,0 @@
"""Bulk Mode API routes.
Lets a user submit up to 100 karaoke jobs at once, by text (artist/title rows)
or by album (MusicBrainz release lookup). Album tracklists and text rows are
enriched with KaraokeNerds community-version availability so the user can skip
tracks that already exist.
Submission (POST /bulk/submit) and progress (GET /bulk/{batch_id}) live here too
(added in later phases).
"""
import asyncio
import logging
import uuid
from typing import Literal, Optional
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from pydantic import BaseModel, Field
from backend.api.dependencies import require_auth
from backend.i18n import get_full_locale_from_request
from backend.services.auth_service import AuthResult
logger = logging.getLogger(__name__)
# Strong refs to shielded tasks so they aren't garbage-collected if the request is cancelled.
_BACKGROUND_TASKS: set = set()
router = APIRouter(prefix="/bulk", tags=["bulk"])
MAX_BULK_SONGS = 100
# --- Models -----------------------------------------------------------------
class BulkArtist(BaseModel):
mbid: Optional[str] = None
name: str
disambiguation: str = ""
type: Optional[str] = None
country: Optional[str] = None
# Set when only a MusicBrainz alias matched the query (e.g. "Hikaru Utada").
alias: Optional[str] = None
class BulkAlbum(BaseModel):
release_group_mbid: str
title: str
primary_type: Optional[str] = None
secondary_types: list[str] = []
first_release_date: str = ""
is_studio: bool = True
class BulkEdition(BaseModel):
release_mbid: str
title: Optional[str] = None
status: Optional[str] = None
date: str = ""
country: Optional[str] = None
track_count: int = 0
class BulkEditionVariant(BaseModel):
"""A distinct tracklist variant (country pressings collapsed by track count)."""
representative_release_mbid: str
label: Literal["Original", "Reissue"] = "Original"
track_count: int = 0
year: str = ""
pressing_count: int = 0
delta_vs_original: int = 0
class CommunityVersion(BaseModel):
"""An existing community karaoke version the user can preview on YouTube."""
brand: str
url: str
class BulkTrack(BaseModel):
position: Optional[int] = None
title: str
recording_mbid: Optional[str] = None
length_ms: Optional[int] = None
is_extra: bool = False
extra_reason: str = ""
# Availability (filled from KaraokeNerds); available=True => already exists
available: bool = False
brands: list[str] = []
versions: list[CommunityVersion] = []
class BulkTracklistResponse(BaseModel):
release_mbid: Optional[str] = None
canonical_release_mbid: Optional[str] = None
selected_variant_mbid: Optional[str] = None
title: Optional[str] = None
date: str = ""
tracks: list[BulkTrack] = []
editions: list[BulkEdition] = []
variants: list[BulkEditionVariant] = []
class AvailabilityItem(BaseModel):
artist: str = Field(..., min_length=1)
title: str = Field(..., min_length=1)
class AvailabilityRequest(BaseModel):
tracks: list[AvailabilityItem]
class AvailabilityResult(BaseModel):
artist: str
title: str
available: bool
brands: list[str] = []
brand_count: int = 0
versions: list[CommunityVersion] = []
class AvailabilityResponse(BaseModel):
results: list[AvailabilityResult]
# --- Album lookup routes ----------------------------------------------------
@router.get("/album/artists", response_model=list[BulkArtist])
async def album_artists(
request: Request,
q: str = Query(..., min_length=2, description="Artist name search"),
limit: int = Query(8, ge=1, le=25),
auth_result: AuthResult = Depends(require_auth),
):
"""Search MusicBrainz for artists (album-mode step 1)."""
from backend.services.musicbrainz_service import get_musicbrainz_service
try:
results = await get_musicbrainz_service().search_artists(q, limit=limit)
except Exception as e: # noqa: BLE001
logger.warning("MusicBrainz artist search failed: %r", e, exc_info=True)
raise HTTPException(status_code=502, detail="Artist lookup is temporarily unavailable")
return results
@router.get("/album/albums", response_model=list[BulkAlbum])
async def album_albums(
request: Request,
artist_mbid: str = Query(..., min_length=1),
auth_result: AuthResult = Depends(require_auth),
):
"""List an artist's albums (release-groups) for the album picker."""
from backend.services.musicbrainz_service import get_musicbrainz_service
try:
return await get_musicbrainz_service().get_albums(artist_mbid)
except Exception as e: # noqa: BLE001
logger.warning("MusicBrainz album list failed: %r", e, exc_info=True)
raise HTTPException(status_code=502, detail="Album lookup is temporarily unavailable")
@router.get("/album/tracklist", response_model=BulkTracklistResponse)
async def album_tracklist(
request: Request,
artist: str = Query(..., min_length=1, description="Artist name for availability checks"),
release_group_mbid: Optional[str] = Query(None),
release_mbid: Optional[str] = Query(None, description="Specific edition to load"),
locale: Optional[str] = Query(None, description="UI locale → biases representative-pressing choice"),
auth_result: AuthResult = Depends(require_auth),
):
"""Resolve an album's canonical tracklist (or a specific edition) and enrich
each track with KaraokeNerds community-version availability.
Response shape note: ``variants`` is only populated on the ``release_group_mbid``
path (the initial album load). When loading a specific edition via
``release_mbid`` it is intentionally ``[]`` — the client already holds the
variant list from the initial load and only needs the chosen edition's tracks.
"""
from backend.services.musicbrainz_service import (
get_musicbrainz_service,
country_prefs_for_locale,
)
from backend.services.karaokenerds_service import check_community_versions_batch
if not release_group_mbid and not release_mbid:
raise HTTPException(status_code=422, detail="release_group_mbid or release_mbid required")
mb = get_musicbrainz_service()
try:
if release_mbid:
# Loading a specific variant — variants already known to the client.
tracklist = await mb.get_release_tracklist(release_mbid, release_group_mbid)
tracklist.setdefault("editions", [])
tracklist.setdefault("canonical_release_mbid", release_mbid)
tracklist.setdefault("variants", [])
tracklist.setdefault("selected_variant_mbid", release_mbid)
else:
tracklist = await mb.get_album_tracklist(
release_group_mbid, preferred_countries=country_prefs_for_locale(locale)
)
except LookupError:
# Not in our mirror (or not part of that album) — retrying won't help.
raise HTTPException(status_code=404, detail="Release not found")
except Exception as e: # noqa: BLE001
logger.warning("MusicBrainz tracklist failed: %r", e, exc_info=True)
raise HTTPException(status_code=502, detail="Tracklist lookup is temporarily unavailable")
tracks = tracklist.get("tracks", [])
# Availability enrichment — never fatal.
try:
avail = await check_community_versions_batch(
[{"artist": artist, "title": t["title"]} for t in tracks]
)
for t, a in zip(tracks, avail):
t["available"] = a["available"]
t["brands"] = a["brands"]
t["versions"] = a.get("versions", [])
except Exception as e: # noqa: BLE001
logger.warning("Availability enrichment failed (non-fatal): %s", e)
return BulkTracklistResponse(
release_mbid=tracklist.get("release_mbid"),
canonical_release_mbid=tracklist.get("canonical_release_mbid"),
selected_variant_mbid=tracklist.get("selected_variant_mbid"),
title=tracklist.get("title"),
date=tracklist.get("date", ""),
tracks=[BulkTrack(**t) for t in tracks],
editions=[BulkEdition(**e) for e in tracklist.get("editions", [])],
variants=[BulkEditionVariant(**v) for v in tracklist.get("variants", [])],
)
@router.post("/availability", response_model=AvailabilityResponse)
async def bulk_availability(
request: Request,
body: AvailabilityRequest,
auth_result: AuthResult = Depends(require_auth),
):
"""Batch KaraokeNerds availability check for text-mode rows."""
from backend.services.karaokenerds_service import check_community_versions_batch
if len(body.tracks) > MAX_BULK_SONGS:
raise HTTPException(status_code=422, detail=f"At most {MAX_BULK_SONGS} tracks per request")
results = await check_community_versions_batch(
[{"artist": t.artist, "title": t.title} for t in body.tracks]
)
return AvailabilityResponse(results=[AvailabilityResult(**r) for r in results])
# --- Submission -------------------------------------------------------------
class BulkSong(BaseModel):
artist: str = Field(..., min_length=1)
title: str = Field(..., min_length=1)
display_artist: Optional[str] = None
display_title: Optional[str] = None
# Album mode: the track's number on its disc (album-search tie-break).
position: Optional[int] = None
class BulkAlbumRef(BaseModel):
"""The album an album-mode batch was picked from. Lets the search worker find
the album's torrent once instead of searching every track separately."""
artist: str = Field(..., min_length=1)
title: str = Field(..., min_length=1)
release_group_mbid: Optional[str] = None
release_mbid: Optional[str] = None
class BulkSettings(BaseModel):
auto_select_if_lossless: bool = True
is_private: bool = False
skip_audio_edit: bool = True
skip_customization: bool = True
# Full-auto review (workstream C): batch default autonomy + backing preference.
review_mode: str = "auto"
backing_preference: str = "auto"
translation_language: Optional[str] = None
class BulkSubmitRequest(BaseModel):
songs: list[BulkSong]
settings: BulkSettings = Field(default_factory=BulkSettings)
album: Optional[BulkAlbumRef] = None
class BulkSubmitResponse(BaseModel):
batch_id: str
job_ids: list[str]
total: int
@router.post("/submit", response_model=BulkSubmitResponse)
async def bulk_submit(
request: Request,
body: BulkSubmitRequest,
auth_result: AuthResult = Depends(require_auth),
):
"""Create one job per song, gate on credits upfront, and dispatch the batch
search worker. Jobs start parked (auto_download=False) and the worker either
auto-selects a confident lossless match or leaves them awaiting selection."""
from backend.config import get_settings
from backend.exceptions import InsufficientCreditsError
from backend.models.job import JobCreate
from backend.services.job_manager import JobManager
from backend.services.job_defaults_service import resolve_cdg_txt_defaults
from backend.services.theme_service import get_theme_service
from backend.services.worker_service import get_worker_service
n = len(body.songs)
if n == 0:
raise HTTPException(status_code=422, detail="At least one song is required")
if n > MAX_BULK_SONGS:
raise HTTPException(status_code=422, detail=f"At most {MAX_BULK_SONGS} songs per batch")
user_email = auth_result.user_email
is_admin = bool(auth_result.is_admin)
locale = get_full_locale_from_request(request)
# Everything up to dispatch is sync Firestore/GCS work (credit check, default
# theme, one create_job + update_job per song). Run it off the event loop: a
# 2026-10-22 album batch stalled the loop 26s and 9 other requests with it.
def _create_batch_jobs() -> tuple[str, list[str]]:
# Upfront credit gate (1 credit/song minimum). Admins bypass.
if not is_admin and user_email:
from backend.services.user_service import get_user_service
available = get_user_service().check_credits(user_email)
if available < n:
raise HTTPException(
status_code=402,
detail={
"error": "insufficient_credits",
"message": "You don't have enough credits for this batch.",
"credits_available": available,
"credits_required": n,
},
)
settings = get_settings()
theme_service = get_theme_service()
theme_id = theme_service.get_default_theme_id()
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(theme_id, None, None)
batch_id = uuid.uuid4().hex[:12]
bulk_settings = body.settings.model_dump()
bulk_album = body.album.model_dump() if body.album else None
job_manager = JobManager()
job_ids: list[str] = []
for song in body.songs:
display_artist = (song.display_artist or song.artist).strip()
display_title = (song.display_title or song.title).strip()
request_metadata = {
"created_from": "bulk",
"batch_id": batch_id,
"user_email": user_email,
}
job_create = JobCreate(
artist=display_artist,
title=display_title,
theme_id=theme_id,
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
user_email=user_email,
audio_search_artist=song.artist.strip(),
audio_search_title=song.title.strip(),
auto_download=False,
is_private=body.settings.is_private,
review_mode=body.settings.review_mode,
backing_preference=body.settings.backing_preference,
translation_language=body.settings.translation_language,
brand_prefix=settings.default_brand_prefix,
enable_youtube_upload=settings.default_enable_youtube_upload,
youtube_description=settings.default_youtube_description,
discord_webhook_url=settings.default_discord_webhook_url,
dropbox_path=settings.default_dropbox_path,
gdrive_folder_id=settings.default_gdrive_folder_id,
request_metadata=request_metadata,
locale=locale,
)
try:
job = job_manager.create_job(job_create, is_admin=is_admin)
except InsufficientCreditsError:
# Ran out mid-batch (e.g. concurrent spend after the gate). Stop here;
# already-created jobs are valid and will proceed.
logger.warning("[batch:%s] ran out of credits after %d/%d jobs", batch_id, len(job_ids), n)
break
bulk_state = {
"state_data.batch_id": batch_id,
"state_data.bulk_settings": bulk_settings,
"state_data.bulk_pending_search": True,
}
if bulk_album is not None:
bulk_state["state_data.bulk_album"] = bulk_album
bulk_state["state_data.bulk_track_position"] = song.position
job_manager.update_job(job.job_id, bulk_state)
job_ids.append(job.job_id)
return batch_id, job_ids
# Dispatch the batch search worker. Retry once on transient failure. If it
# still fails the jobs exist (and are charged) but won't process, so surface
# an explicit error with the batch_id — and tell the user NOT to resubmit
# (that would create+charge a second batch); the batch can be re-dispatched.
worker_service = get_worker_service()
async def _create_and_dispatch() -> tuple[str, list[str], bool]:
batch_id, job_ids = await asyncio.to_thread(_create_batch_jobs)
if not job_ids:
return batch_id, job_ids, False
triggered = await worker_service.trigger_bulk_search_worker(batch_id)
if not triggered:
triggered = await worker_service.trigger_bulk_search_worker(batch_id)
return batch_id, job_ids, triggered
# Shielded: if the request is cancelled (client disconnect) mid-way, the jobs
# being created and charged in the thread still get dispatched.
task = asyncio.create_task(_create_and_dispatch())
_BACKGROUND_TASKS.add(task)
task.add_done_callback(_BACKGROUND_TASKS.discard)
batch_id, job_ids, triggered = await asyncio.shield(task)
if not job_ids:
raise HTTPException(status_code=402, detail="No jobs could be created")
if not triggered:
logger.critical(
"[batch:%s] created %d jobs but FAILED to dispatch bulk-search worker",
batch_id, len(job_ids),
)
raise HTTPException(
status_code=503,
detail={
"error": "dispatch_failed",
"message": "Your batch was created but processing couldn't start. "
"Please don't resubmit — contact support with this batch ID and we'll start it.",
"batch_id": batch_id,
"job_ids": job_ids,
},
)
logger.info("[batch:%s] created %d jobs, dispatched bulk-search worker", batch_id, len(job_ids))
return BulkSubmitResponse(batch_id=batch_id, job_ids=job_ids, total=len(job_ids))
# --- Progress ---------------------------------------------------------------
class BulkJobStatus(BaseModel):
job_id: str
artist: str
title: str
status: str
auto_selected: bool = False
class BulkBatchResponse(BaseModel):
batch_id: str
total: int
counts: dict
jobs: list[BulkJobStatus]
# Status -> progress bucket for the batch summary.
_AWAITING = "awaiting_audio_selection"
_TERMINAL_OK = "complete"
_TERMINAL_FAIL = "failed"
_SEARCHING = {"pending", "searching_audio"}
def _bucket(status: str, pending_search: bool) -> str:
if pending_search or status in _SEARCHING:
return "searching"
if status == _AWAITING:
return "awaiting_selection"
if status == _TERMINAL_OK:
return "complete"
if status == _TERMINAL_FAIL:
return "failed"
return "processing"
@router.get("/{batch_id}", response_model=BulkBatchResponse)
def bulk_batch(
request: Request,
batch_id: str,
auth_result: AuthResult = Depends(require_auth),
):
"""Progress summary for a batch: counts by bucket + per-job status."""
from backend.services.job_manager import JobManager
job_manager = JobManager()
jobs = job_manager.list_jobs_by_batch_id(batch_id)
if not bool(auth_result.is_admin):
email = (auth_result.user_email or "").lower()
jobs = [j for j in jobs if (j.user_email or "").lower() == email]
if not jobs:
raise HTTPException(status_code=404, detail="Batch not found")
counts = {"searching": 0, "awaiting_selection": 0, "processing": 0, "complete": 0, "failed": 0}
job_statuses: list[BulkJobStatus] = []
for j in jobs:
status = j.status.value if hasattr(j.status, "value") else str(j.status)
sd = j.state_data or {}
bucket = _bucket(status, bool(sd.get("bulk_pending_search")))
counts[bucket] += 1
job_statuses.append(
BulkJobStatus(
job_id=j.job_id,
artist=j.audio_search_artist or j.artist or "",
title=j.audio_search_title or j.title or "",
status=status,
auto_selected=bool(sd.get("bulk_auto_selected")),
)
)
return BulkBatchResponse(
batch_id=batch_id, total=len(jobs), counts=counts, jobs=job_statuses
)
-229
View File
@@ -1,229 +0,0 @@
"""
Catalog API routes for song/artist lookup and community version detection.
Provides autocomplete data from karaoke-decide's MusicBrainz + Spotify catalog,
and checks karaokenerds.com for existing community karaoke versions.
"""
import logging
import time
from typing import Literal, Optional
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from pydantic import BaseModel, Field
from backend.api.dependencies import require_auth
from backend.services.auth_service import AuthResult
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/catalog", tags=["catalog"])
# Simple per-user rate limiting: 20 requests/minute
# { user_email: [timestamp, timestamp, ...] }
_rate_limit_windows: dict[str, list[float]] = {}
RATE_LIMIT_MAX_REQUESTS = 20
RATE_LIMIT_WINDOW_SECONDS = 60
def _check_rate_limit(user_email: str, locale: str = "en") -> None:
"""Check per-user rate limit. Raises 429 if exceeded."""
now = time.monotonic()
window_start = now - RATE_LIMIT_WINDOW_SECONDS
# Get or create window for this user
if user_email not in _rate_limit_windows:
_rate_limit_windows[user_email] = []
# Remove expired entries
timestamps = _rate_limit_windows[user_email]
_rate_limit_windows[user_email] = [ts for ts in timestamps if ts > window_start]
if len(_rate_limit_windows[user_email]) >= RATE_LIMIT_MAX_REQUESTS:
raise HTTPException(
status_code=429,
detail=t(locale, "catalog.rateLimitExceeded", max_requests=RATE_LIMIT_MAX_REQUESTS),
)
_rate_limit_windows[user_email].append(now)
# --- Response Models ---
class ArtistResult(BaseModel):
"""Artist search result from MusicBrainz/Spotify catalog."""
name: str
mbid: Optional[str] = None
disambiguation: Optional[str] = None
artist_type: Optional[str] = None
spotify_id: Optional[str] = None
popularity: Optional[int] = None
genres: Optional[list[str]] = None
tags: Optional[list[str]] = None
class TrackResult(BaseModel):
"""Track search result from Spotify catalog."""
track_name: str
artist_name: str
track_id: Optional[str] = None
artist_id: Optional[str] = None
popularity: Optional[int] = None
duration_ms: Optional[int] = None
explicit: Optional[bool] = None
class CommunityTrack(BaseModel):
"""A community-approved karaoke track from karaokenerds."""
brand_name: str
brand_code: str
youtube_url: str
is_community: bool = True
class CommunityCheckSong(BaseModel):
"""A song with community karaoke tracks."""
title: str
artist: str
community_tracks: list[CommunityTrack]
class CommunityCheckRequest(BaseModel):
"""Request body for community version check."""
artist: str = Field(..., min_length=1)
title: str = Field(..., min_length=1)
class CommunityCheckResponse(BaseModel):
"""Response for community version check."""
has_community: bool
songs: list[CommunityCheckSong]
best_youtube_url: Optional[str] = None
class MatchJudgeRequest(BaseModel):
"""Request body for the artist/title match judge."""
artist: str = Field(..., min_length=1)
title: str = Field(..., min_length=1)
# The audio search's confidence tier (1=strong..3=weak), so the judge can
# tell whether weak results hint at a typo. Optional.
audio_confidence_tier: Optional[int] = Field(None, ge=1, le=3)
# "fast" = catalog-only pass (no AI, no tier needed) the client fires in
# parallel with the audio search; "full" = complete pipeline once the tier is
# known. Defaults to "full" so a single legacy call still works.
stage: Literal["fast", "full"] = "full"
class MatchJudgeAlternative(BaseModel):
artist: str
title: str
class MatchJudgeResponse(BaseModel):
"""Verdict on a typed artist/title — formatting + match acceptance."""
kind: str # "cosmetic" | "content" | "ambiguous" | "none"
confident: bool
canonical_artist: str
canonical_title: str
alternatives: list[MatchJudgeAlternative] = []
engine: str # "deterministic" | "catalog" | "ai"
reason: str = ""
# Set by a "fast" pass that couldn't decide — the client should follow up with
# a "full" call once the audio tier is known.
needs_ai: bool = False
# --- Routes ---
@router.get("/artists", response_model=list[ArtistResult])
async def search_artists(
request: Request,
q: str = Query(..., min_length=2, description="Artist name search query"),
limit: int = Query(10, ge=1, le=50, description="Maximum results"),
auth_result: AuthResult = Depends(require_auth),
):
"""
Search for artists by name. Returns canonical MusicBrainz/Spotify artist data
for autocomplete in job creation forms.
"""
locale = get_locale_from_request(request)
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
from backend.services.catalog_proxy_service import search_artists
results = await search_artists(q, limit)
return results
@router.get("/tracks", response_model=list[TrackResult])
async def search_tracks(
request: Request,
q: str = Query(..., min_length=2, description="Track name search query"),
artist: Optional[str] = Query(None, description="Filter by artist name"),
limit: int = Query(10, ge=1, le=50, description="Maximum results"),
auth_result: AuthResult = Depends(require_auth),
):
"""
Search for tracks by name, optionally filtered by artist.
Returns canonical Spotify track data for autocomplete in job creation forms.
"""
locale = get_locale_from_request(request)
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
from backend.services.catalog_proxy_service import search_tracks
results = await search_tracks(q, artist, limit)
return results
@router.post("/community-check", response_model=CommunityCheckResponse)
async def check_community_versions(
request: Request,
body: CommunityCheckRequest,
auth_result: AuthResult = Depends(require_auth),
):
"""
Check if a song already has community-approved karaoke versions on YouTube
via karaokenerds.com. Non-blocking — just informs the user.
"""
locale = get_locale_from_request(request)
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
from backend.services.karaokenerds_service import check_community_versions
result = await check_community_versions(body.artist, body.title)
return result
@router.post("/match-judge", response_model=MatchJudgeResponse)
async def match_judge(
request: Request,
body: MatchJudgeRequest,
auth_result: AuthResult = Depends(require_auth),
):
"""
Judge a typed artist/title: decide official formatting + whether it matches a
real song. Deterministic + catalog layers run first; a light AI model fires
only when those aren't confident. Never blocks the submission flow — failures
degrade to a "none" verdict.
"""
locale = get_locale_from_request(request)
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
from backend.services.fake_workers import fake_workers_enabled
if fake_workers_enabled():
# Local E2E stack: scripted verdict (per-user fake config), no catalog/Gemini.
from backend.services.fake_workers.judge import fake_match_verdict
return fake_match_verdict(body.artist, body.title, auth_result.user_email, stage=body.stage).to_dict()
from backend.services.match_judge.service import judge_match
verdict = await judge_match(
body.artist, body.title,
audio_tier=body.audio_confidence_tier,
stage=body.stage,
)
logger.info(
"match-judge stage=%s kind=%s engine=%s confident=%s needs_ai=%s",
body.stage, verdict.kind, verdict.engine, verdict.confident, verdict.needs_ai,
)
return verdict.to_dict()
-136
View File
@@ -1,136 +0,0 @@
"""POST /api/client-errors — ingest frontend crash reports.
Unauthenticated (logged-out users' crashes still matter). Deduped and alerted
via the existing error-monitor pipeline — see
``backend/services/error_monitor``.
"""
from __future__ import annotations
import logging
import time
from typing import Optional
from fastapi import APIRouter, Request, HTTPException
from pydantic import BaseModel, Field, field_validator
from backend.services.error_monitor.firestore_adapter import ErrorPatternsAdapter
from backend.services.error_monitor.frontend_ingestion import (
FrontendErrorReport,
RateLimiter,
build_pattern_data,
is_bot_user_agent,
is_host_app_injection_error,
is_opaque_script_error,
sanitize_url,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/client-errors", tags=["client-errors"])
_limiter = RateLimiter(max_per_minute=60)
_adapter_singleton: ErrorPatternsAdapter | None = None
def _get_adapter() -> ErrorPatternsAdapter:
global _adapter_singleton
if _adapter_singleton is None:
_adapter_singleton = ErrorPatternsAdapter()
return _adapter_singleton
class ClientErrorPayload(BaseModel):
message: str = Field(..., min_length=1, max_length=4000)
stack: Optional[str] = Field(None, max_length=1_000_000)
url: str = Field("", max_length=2048)
user_agent: str = Field("", max_length=1024)
release: str = Field("", max_length=64)
user_email: Optional[str] = Field(None, max_length=320)
viewport: Optional[dict] = None
locale: str = Field("en", max_length=10)
source: str = Field("unknown", max_length=64)
# optional free-form extra for debugging. Capped server-side.
extra: Optional[dict] = None
@field_validator("message", "url", "user_agent", "locale", "source")
@classmethod
def strip(cls, v: str) -> str:
return (v or "").strip()
class ClientErrorResponse(BaseModel):
pattern_id: str
is_new: bool
@router.post("", response_model=ClientErrorResponse, status_code=202)
def report_client_error(payload: ClientErrorPayload, request: Request) -> ClientErrorResponse:
client_ip = request.client.host if request.client else "unknown"
if not _limiter.allow(client_ip, time.monotonic()):
raise HTTPException(status_code=429, detail="too many reports")
# Crawler/bot crashes are pure alert noise (bots run headless Chrome and trip
# DOM errors no real user hits). Accept the report so the client doesn't retry,
# but don't persist/alert on it.
if is_bot_user_agent(payload.user_agent):
# Log the bot UA (not sensitive) + sanitized URL (query/token stripped)
# for visibility into what's being dropped, without persisting/alerting.
logger.info(
"frontend_crash_ignored_bot ua=%r url=%s source=%s",
payload.user_agent[:120],
sanitize_url(payload.url),
payload.source,
)
return ClientErrorResponse(pattern_id="bot-ignored", is_new=False)
# Opaque cross-origin "Script error." — no file/line, not our code.
if is_opaque_script_error(payload.message, payload.source, payload.extra):
logger.info(
"frontend_crash_ignored_opaque_script_error url=%s source=%s",
sanitize_url(payload.url),
payload.source,
)
return ClientErrorResponse(pattern_id="opaque-script-error-ignored", is_new=False)
# A script the in-app browser's host app injected (Zalo), not our code.
if is_host_app_injection_error(payload.message, payload.user_agent):
logger.info(
"frontend_crash_ignored_host_app_injection url=%s source=%s",
sanitize_url(payload.url),
payload.source,
)
return ClientErrorResponse(pattern_id="host-app-injection-ignored", is_new=False)
report = FrontendErrorReport(
message=payload.message,
stack=payload.stack,
url=payload.url,
user_agent=payload.user_agent,
release=payload.release,
user_email=payload.user_email,
viewport=payload.viewport,
locale=payload.locale,
extra=payload.extra,
)
pattern_data = build_pattern_data(report)
try:
result = _get_adapter().upsert_pattern(pattern_data)
except Exception: # pragma: no cover - Firestore transient errors
logger.exception("Failed to upsert frontend error pattern")
raise HTTPException(status_code=503, detail="storage unavailable")
logger.info(
"frontend_crash_reported pattern_id=%s is_new=%s source=%s",
result.pattern_id,
result.is_new,
payload.source,
# Full client diagnostics + breadcrumbs, searchable in Cloud Logging
# (the alert sample only carries a trimmed summary).
extra={
"error_pattern_id": result.pattern_id,
"client_url": sanitize_url(payload.url),
"client_release": payload.release,
"client_extra": payload.extra,
},
)
return ClientErrorResponse(pattern_id=result.pattern_id, is_new=result.is_new)
-212
View File
@@ -1,212 +0,0 @@
"""POST /api/client-events — persistent telemetry for user-visible degradation.
Records every time the frontend shows a user a degraded-service surface (the
orange "trouble reaching our servers" banner, the stronger "temporarily
unavailable" error, a failed lyrics load, a slow/failed waveform), with enough
metadata to review frequency and context later. Two sinks per event:
1. A structured INFO log line (``client_event type=…``) — queryable in Cloud
Logging / Log Analytics alongside backend symptoms from the same window.
2. A Firestore doc in ``client_events`` — survives log retention, cheap to
aggregate for "how often are users seeing this?" reviews.
Identity (2026-10-03): the client sends its session token (if any) and the
server resolves ``user_email`` / ``is_admin`` / ``is_internal`` / ``is_test``
from it — a client-supplied email is never trusted (and is ignored). Each
event also carries the FingerprintJS ``device_fingerprint`` (same visitorId the
magic-link abuse checks use), a per-tab ``tab_id`` and an ``episode_id`` that
groups one degradation episode (first banner → ``banner_recovered``).
The watchdog in ``backend/services/loop_watchdog.py`` writes server-side
``server_loop_stall`` docs into the same collection; clients cannot submit
that type.
Unauthenticated (review-token users must be able to report) and rate-limited
per IP, mirroring /api/client-errors. The limiter is process-local, so the
effective ceiling is 120/min/IP × live instances — acceptable for this threat
model (non-malicious browsers, 60s client-side per-type throttle); Cloudflare's
zone-wide rate limit sits in front as a flood backstop. Events are dropped,
never queued, when the limiter trips — telemetry, not a delivery guarantee.
"""
from __future__ import annotations
import logging
import time
from datetime import datetime, timezone
from typing import Any, Literal, Optional
from urllib.parse import urlparse
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel, Field, field_validator
from backend.utils.test_data import is_internal_email, is_test_email
from backend.services.error_monitor.frontend_ingestion import (
RateLimiter,
is_bot_user_agent,
sanitize_url,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/client-events", tags=["client-events"])
# Sized for a legitimate worst case from ONE IP: a many-tab burst (e.g. 16 review
# tabs at once) where every tab reports a banner + a waveform event — that must be
# fully recorded, not self-suppressed. Each tab throttles per-type client-side.
_limiter = RateLimiter(max_per_minute=120)
_db_singleton = None
# The full closed set of reportable events. Reject anything else so the
# collection stays reviewable (no free-form types accumulating).
EventType = Literal[
"banner_reconnecting",
"banner_unavailable",
"banner_waking",
"banner_recovered",
"lyrics_load_failed",
"waveform_slow",
"waveform_failed",
]
def _get_db():
global _db_singleton
if _db_singleton is None:
from google.cloud import firestore # type: ignore[import]
_db_singleton = firestore.Client(project="nomadkaraoke")
return _db_singleton
class ClientEventPayload(BaseModel):
type: EventType
url: str = Field("", max_length=2048)
job_id: Optional[str] = Field(None, max_length=64)
# Legacy field from pre-2026-10-03 frontends — accepted but IGNORED (the
# server resolves identity from the bearer token instead).
user_email: Optional[str] = Field(None, max_length=320)
device_fingerprint: Optional[str] = Field(None, max_length=128)
tab_id: Optional[str] = Field(None, max_length=64)
episode_id: Optional[str] = Field(None, max_length=64)
locale: str = Field("en", max_length=10)
release: str = Field("", max_length=64)
# Small free-form context (stall age, http status, elapsed ms, probe state…).
# Size-capped server-side; values are stored as-is.
detail: Optional[dict] = None
@field_validator("url", "locale", "release")
@classmethod
def strip(cls, v: str) -> str:
return (v or "").strip()
def _capped_detail(detail: Optional[dict]) -> dict[str, Any]:
"""Keep detail reviewable: at most 12 keys, scalar-ish values, short strings."""
if not isinstance(detail, dict):
return {}
out: dict[str, Any] = {}
for key, value in list(detail.items())[:12]:
k = str(key)[:40]
if isinstance(value, (int, float, bool)) or value is None:
out[k] = value
else:
out[k] = str(value)[:200]
return out
_TENANT_HOST_EXCLUDE = {"gen", "www", "api", "app", "decide", "localhost"}
def _tenant_from_url(url: str) -> Optional[str]:
"""``randy-vild.nomadkaraoke.com/...`` → ``randy-vild``; consumer hosts → None."""
try:
host = (urlparse(url).hostname or "").lower()
except ValueError:
return None
if not host.endswith(".nomadkaraoke.com"):
return None
sub = host[: -len(".nomadkaraoke.com")]
if not sub or "." in sub or sub in _TENANT_HOST_EXCLUDE:
return None
return sub
def _resolve_identity(request: Request) -> dict[str, Any]:
"""Resolve the caller from its bearer token. Never raises; anonymous on failure."""
identity: dict[str, Any] = {
"user_email": None, "is_admin": False, "is_internal": False, "is_test": False,
}
auth = request.headers.get("authorization", "")
if not auth.lower().startswith("bearer "):
return identity
token = auth[7:].strip()
if not token:
return identity
try:
from backend.services.auth_service import get_auth_service
result = get_auth_service().validate_token_full(token)
except Exception:
logger.debug("client_event token resolution failed", exc_info=True)
return identity
if not result.is_valid:
return identity
email = result.user_email
identity["user_email"] = email
identity["is_admin"] = bool(result.is_admin)
identity["is_internal"] = bool(email and is_internal_email(email))
identity["is_test"] = bool(email and is_test_email(email))
return identity
@router.post("", status_code=202)
def report_client_event(payload: ClientEventPayload, request: Request) -> dict:
client_ip = request.client.host if request.client else "unknown"
if not _limiter.allow(client_ip, time.monotonic()):
raise HTTPException(status_code=429, detail="too many events")
user_agent = request.headers.get("user-agent", "")[:1024]
if is_bot_user_agent(user_agent):
return {"status": "ignored"}
# Sync handler → runs in FastAPI's threadpool, so the token lookup
# (Firestore) never touches the event loop.
identity = _resolve_identity(request)
event = {
"type": payload.type,
"source": "client",
"url": sanitize_url(payload.url),
"tenant": _tenant_from_url(payload.url),
"job_id": payload.job_id,
**identity,
"device_fingerprint": payload.device_fingerprint,
"tab_id": payload.tab_id,
"episode_id": payload.episode_id,
"locale": payload.locale,
"release": payload.release,
"user_agent": user_agent,
"detail": _capped_detail(payload.detail),
"created_at": datetime.now(timezone.utc),
}
# Sink 1: structured log line — correlate with backend symptoms in Logging.
logger.info(
"client_event type=%s job_id=%s user=%s admin=%s fp=%s episode=%s url=%s release=%s detail=%s",
event["type"],
event["job_id"],
event["user_email"],
event["is_admin"],
event["device_fingerprint"],
event["episode_id"],
event["url"],
event["release"],
event["detail"],
)
# Sink 2: Firestore — best-effort; telemetry must never error back to users.
try:
_get_db().collection("client_events").add(event)
except Exception:
logger.exception("Failed to persist client event (non-fatal)")
return {"status": "recorded"}
-104
View File
@@ -1,104 +0,0 @@
"""
Encoding worker lifecycle endpoints.
Provides warmup and heartbeat endpoints for the blue-green
encoding worker VMs. Called by the lyrics-review frontend to ensure the
primary VM is running before the reviewer previews a render.
Auth: these are gated on ``require_review_auth`` (review access to a specific
job), NOT ``require_admin``. The callers are ordinary customers on the
lyrics-review page — gating on admin makes every non-admin caller 403, which
silently defeats the JIT pre-warm. Scoping to the job being reviewed keeps the
abuse posture identical to the rest of the review surface: you can only warm
the shared encoding VM if you already have review access to a real job.
"""
import asyncio
import logging
from typing import Tuple
from fastapi import APIRouter, Depends
from backend.api.dependencies import require_review_auth
from backend.services.encoding_worker_manager import EncodingWorkerManager
from backend.services.encoding_errors import EncodingWorkerStartError
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/internal/encoding-worker", tags=["encoding-worker"])
# Lazy-loaded singleton
_manager = None
def get_worker_manager():
global _manager
if _manager is None:
from google.cloud import compute_v1, firestore
from backend.config import get_settings
settings = get_settings()
db = firestore.Client(project=settings.google_cloud_project)
compute_client = compute_v1.InstancesClient()
_manager = EncodingWorkerManager(
db=db,
compute_client=compute_client,
project_id=settings.google_cloud_project,
)
return _manager
@router.post("/warmup/{job_id}")
async def warmup_encoding_worker(
job_id: str,
_auth: Tuple[str, str] = Depends(require_review_auth),
manager=Depends(get_worker_manager),
):
"""Start the preferred encoding worker if it's stopped (normally the GCE primary).
Called on lyrics-review page load so the VM is warm by the time the
reviewer previews a render. ``job_id`` scopes auth to the job under review.
"""
try:
# ensure_primary_running does Firestore reads + GCE compute-API calls
# (instances.get / instances.start) that can take 10s+. Run it off the
# event loop: on the single-process server a sync call here froze the
# WHOLE instance for ~11s on every review-page open with a cold VM,
# stalling that same page's audio/lyrics requests behind it
# (2026-09-19, follow-up to the concurrent-load investigation).
result = await asyncio.to_thread(manager.ensure_preferred_running)
if result["started"]:
logger.info(f"Started encoding worker VM: {result['vm_name']}")
return result
except EncodingWorkerStartError as e:
# The primary VM couldn't start (capacity exhaustion or a transient
# 503 SERVICE_UNAVAILABLE from the GCE backend). This is NOT fatal:
# the encoding flow's own warmup (ensure_any_running) falls back to
# the alternate-zone workers, so the job still completes. Log at
# WARNING so it doesn't trip the production error monitor and page us
# for self-healing capacity events.
logger.warning(
f"Primary encoding worker warmup failed ({e}); "
f"encoding will fall back to an alternate-zone worker"
)
return {"started": False, "error": str(e)}
except Exception as e:
logger.error(f"Failed to warm up encoding worker: {e}")
return {"started": False, "error": str(e)}
@router.post("/heartbeat/{job_id}")
async def heartbeat_encoding_worker(
job_id: str,
_auth: Tuple[str, str] = Depends(require_review_auth),
manager=Depends(get_worker_manager),
):
"""Update activity timestamp to prevent idle shutdown.
Called periodically while the reviewer works. ``job_id`` scopes auth to the
job under review.
"""
try:
# Sync Firestore write — keep it off the event loop (see warmup above).
await asyncio.to_thread(manager.update_activity)
return {"status": "ok"}
except Exception as e:
logger.error(f"Failed to update encoding worker heartbeat: {e}")
return {"status": "error", "error": str(e)}
File diff suppressed because it is too large. Load diff
+5 -246
View File
@@ -1,28 +1,21 @@
"""
Health check routes.
"""
import asyncio
import os
import logging
import subprocess
import platform
from fastapi import APIRouter, Depends
from typing import Dict, Any, Optional, Tuple
from fastapi import APIRouter
from typing import Dict, Any, Optional
from backend.version import VERSION, COMMIT_SHA, PR_NUMBER, PR_TITLE, STARTUP_TIME
from backend.api.dependencies import optional_auth
from backend.services.auth_service import UserType
from backend.version import VERSION
from backend.services.flacfetch_client import get_flacfetch_client
from backend.services.email_service import get_email_service
from backend.services.stripe_service import get_stripe_service
from backend.services.encoding_service import get_encoding_service
from backend.services.encoding_worker_preference import infer_machine_type, PRIMARY_MACHINE_TYPE
from backend.services.spacy_preloader import get_preloaded_model, is_model_preloaded
from backend.services.nltk_preloader import get_preloaded_cmudict, is_cmudict_preloaded
from backend.services.langfuse_preloader import get_preloaded_langfuse_handler, is_langfuse_preloaded, is_langfuse_configured
from backend.services.job_health_service import check_job_consistency_detailed
from backend.services.job_manager import JobManager
from backend.models.job import JobStatus
router = APIRouter()
logger = logging.getLogger(__name__)
@@ -218,7 +211,7 @@ async def check_flacfetch_service_status() -> Dict[str, Any]:
async def check_encoding_worker_status() -> Dict[str, Any]:
"""Check if GCE encoding worker is available and healthy."""
encoding_service = await asyncio.to_thread(get_encoding_service)
encoding_service = get_encoding_service()
if not encoding_service.is_configured:
return {
@@ -312,177 +305,6 @@ async def flacfetch_health() -> Dict[str, Any]:
}
def check_audio_separator_status() -> Dict[str, Any]:
"""Report audio separation availability.
Separation runs in the ``audio-separation-job`` GPU Cloud Run Job (local
GPU mode, models baked into the image). The standalone ``audio-separator``
Cloud Run GPU service was shut down on 2026-10-01 (cost cut), so there is
no remote service to probe — this reports the installed package version
only and never calls out to anything.
"""
try:
from importlib.metadata import version
pkg_version = version("audio-separator")
except Exception:
pkg_version = None
return {
"available": True,
"status": "ok",
"mode": "gpu_job",
"version": pkg_version,
}
@router.get("/health/audio-separator")
def audio_separator_health() -> Dict[str, Any]:
"""
Return audio separation status (installed audio-separator package version).
Does NOT call any remote service — separation runs in the
audio-separation-job GPU Cloud Run Job. No authentication required.
"""
return check_audio_separator_status()
def _get_encoding_worker_manager():
"""Get the EncodingWorkerManager instance. Separated for testability."""
try:
encoding_service = get_encoding_service()
if hasattr(encoding_service, '_worker_manager') and encoding_service._worker_manager:
return encoding_service._worker_manager
except Exception:
pass
return None
@router.get("/health/system-status")
async def system_status(
auth: Optional[Tuple] = Depends(optional_auth),
) -> Dict[str, Any]:
"""
Aggregated system status for all services.
Returns basic info for everyone. Admin users get additional
blue-green deployment details for the encoder.
"""
is_admin = auth is not None and len(auth) >= 2 and auth[1] == UserType.ADMIN
# Fetch all health statuses concurrently
import asyncio
encoding_task = check_encoding_worker_status()
flacfetch_task = check_flacfetch_service_status()
encoding_status, flacfetch_status = await asyncio.gather(
encoding_task, flacfetch_task
)
separator_status = check_audio_separator_status()
# Build frontend service info (from env vars baked at build time — returned as-is)
frontend_svc = {
"status": "ok",
"version": VERSION, # Frontend and backend share the same version from pyproject.toml
}
# Build backend service info
backend_svc = {
"status": "ok",
"version": VERSION,
"deployed_at": STARTUP_TIME,
}
if COMMIT_SHA:
backend_svc["commit_sha"] = COMMIT_SHA
if PR_NUMBER:
backend_svc["pr_number"] = PR_NUMBER
if PR_TITLE:
backend_svc["pr_title"] = PR_TITLE
# Build encoder service info
encoder_svc = {
"status": "ok" if encoding_status.get("available") else "offline",
"version": encoding_status.get("wheel_version"),
"active_jobs": encoding_status.get("active_jobs", 0),
}
# Build flacfetch service info
flacfetch_svc = {
"status": "ok" if flacfetch_status.get("available") else "offline",
"version": flacfetch_status.get("version"),
}
# Build separator service info
separator_svc = {
"status": "ok" if separator_status.get("available") else "offline",
"version": separator_status.get("version"),
}
# Admin-only: blue-green encoder details from Firestore
if is_admin:
manager = _get_encoding_worker_manager()
if manager:
try:
config = manager.get_config()
# Which VM is actually serving right now? When a capacity
# fallback is active (e.g. a c4d stockout routed us to an
# alternate family) the override points at the live VM;
# otherwise it's the blue-green primary. The primary pair is
# always c4d — fallbacks carry their family in the VM name
# (…-fallback-c2df), which infer_machine_type
# decodes when no explicit machine_type is stored on the config.
on_fallback = bool(config.active_override_vm)
active_vm = config.active_override_vm or config.primary_vm
default_zone = getattr(manager, "_zone", "us-central1-c")
active_zone = config.active_override_zone or default_zone
if on_fallback:
active_machine_type = infer_machine_type({"vm": active_vm})
else:
active_machine_type = (
getattr(config, "primary_machine_type", None)
or PRIMARY_MACHINE_TYPE
)
encoder_svc["admin_details"] = {
"primary_vm": config.primary_vm,
"primary_ip": config.primary_ip,
"primary_version": config.primary_version,
"primary_deployed_at": config.primary_deployed_at,
"secondary_vm": config.secondary_vm,
"secondary_ip": config.secondary_ip,
"secondary_version": config.secondary_version,
"secondary_deployed_at": config.secondary_deployed_at,
"last_swap_at": config.last_swap_at,
"deploy_in_progress": config.deploy_in_progress,
"active_jobs": encoding_status.get("active_jobs", 0),
"queue_length": encoding_status.get("queue_length", 0),
# Currently-serving worker (fallback-aware)
"active_vm": active_vm,
"active_zone": active_zone,
"active_machine_type": active_machine_type,
"on_fallback": on_fallback,
}
except Exception as e:
logger.warning(f"Failed to get blue-green config: {e}")
# Add error details for offline services
if not flacfetch_status.get("available"):
flacfetch_svc["admin_details"] = {"error": flacfetch_status.get("error")}
if not separator_status.get("available"):
separator_svc["admin_details"] = {"error": separator_status.get("error")}
if not encoding_status.get("available") and "admin_details" not in encoder_svc:
encoder_svc["admin_details"] = {"error": encoding_status.get("error")}
return {
"services": {
"frontend": frontend_svc,
"backend": backend_svc,
"encoder": encoder_svc,
"flacfetch": flacfetch_svc,
"separator": separator_svc,
}
}
@router.get("/health/detailed")
async def detailed_health_check() -> Dict[str, Any]:
"""
@@ -547,7 +369,7 @@ async def detailed_health_check() -> Dict[str, Any]:
@router.get("/health/preload-status")
def preload_status() -> Dict[str, Any]:
async def preload_status() -> Dict[str, Any]:
"""
Check status of preloaded resources for performance optimization.
@@ -608,69 +430,6 @@ def preload_status() -> Dict[str, Any]:
}
@router.get("/health/job-consistency")
def job_consistency_check(
status: Optional[str] = None,
limit: int = 50
) -> Dict[str, Any]:
"""
Check job consistency across the system.
This endpoint identifies jobs with state inconsistencies that may indicate
bugs in the state machine or trigger logic. Use this to find stuck jobs
or jobs in invalid states.
Args:
status: Optional filter by job status (e.g., "pending", "downloading")
limit: Maximum number of jobs to check (default 50)
Returns:
Dictionary with:
- total_checked: Number of jobs checked
- inconsistent_count: Number of jobs with issues
- inconsistent_jobs: List of job details with their issues
- summary: Summary of issue types found
"""
job_manager = JobManager()
# Query jobs to check
query_params = {}
if status:
try:
job_status = JobStatus(status)
query_params['status'] = job_status
except ValueError:
return {
"error": f"Invalid status: {status}",
"valid_statuses": [s.value for s in JobStatus]
}
# Get jobs from Firestore
jobs = job_manager.list_jobs(limit=limit, **query_params)
# Check each job for consistency
inconsistent_jobs = []
issue_summary = {}
for job in jobs:
result = check_job_consistency_detailed(job)
if not result['is_healthy']:
inconsistent_jobs.append(result)
# Count issue types
for issue in result['issues']:
issue_type = issue.split(':')[0] if ':' in issue else issue
issue_summary[issue_type] = issue_summary.get(issue_type, 0) + 1
return {
"total_checked": len(jobs),
"inconsistent_count": len(inconsistent_jobs),
"inconsistent_jobs": inconsistent_jobs,
"summary": issue_summary,
"status_filter": status,
"limit": limit,
}
@router.get("/readiness")
async def readiness_check() -> Dict[str, str]:
"""Readiness check endpoint for Cloud Run."""
File diff suppressed because it is too large. Load diff
+183 -2412
View File
File diff suppressed because it is too large. Load diff
-94
View File
@@ -1,94 +0,0 @@
"""POST /api/karaokehunt/request — intake for retired-KaraokeHunt-app requests.
A Cloudflare Worker on create.karaokehunt.com (the URL baked into the
unchangeable app binary) answers the app instantly and forwards each payload
here with a shared secret. This route validates + logs the payload, then runs
the full conversion synchronously (see ``workers/karaokehunt_conversion``) —
the Worker doesn't wait for us, so latency is a non-issue and uvicorn finishes
the handler even if the forwarding subrequest is reaped.
Unauthenticated by necessity (the app can't auth), gated three ways: the
shared secret (503 until configured — deploys dark), a per-IP rate limit, and
the conversion worker's own daily job cap + per-song dedup.
"""
from __future__ import annotations
import hmac
import logging
import time
from fastapi import APIRouter, Depends, HTTPException, Request
from backend.api.dependencies import require_admin
from backend.config import get_settings
from backend.services.error_monitor.frontend_ingestion import RateLimiter
from backend.workers import karaokehunt_conversion
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/karaokehunt", tags=["karaokehunt"])
# ~1 legit request every 3 days historically; 10/min/IP is already generous.
_limiter = RateLimiter(max_per_minute=10)
def _client_ip(request: Request) -> str:
# The Worker forwards the app's real IP; direct callers fall back to peer IP.
return (request.headers.get("x-kh-client-ip")
or (request.client.host if request.client else "unknown"))
@router.post("/request", status_code=200)
async def intake_karaokehunt_request(request: Request) -> dict:
settings = get_settings()
if not settings.karaokehunt_forwarder_secret:
raise HTTPException(status_code=503, detail="not configured")
provided = request.headers.get("x-kh-forwarder-secret", "")
if not hmac.compare_digest(provided, settings.karaokehunt_forwarder_secret):
raise HTTPException(status_code=403, detail="forbidden")
if not _limiter.allow(_client_ip(request), time.monotonic()):
raise HTTPException(status_code=429, detail="too many requests")
# The live app binary's exact schema is unverifiable (built from a
# FlutterFlow export newer than any repo branch), so parse leniently:
# any JSON object is accepted and logged; unusable ones become
# outcome="invalid" audit rows rather than errors.
try:
payload = await request.json()
if not isinstance(payload, dict):
raise ValueError("payload must be a JSON object")
except Exception: # noqa: BLE001
raise HTTPException(status_code=400, detail="invalid JSON")
doc = karaokehunt_conversion.create_intake(
payload,
client_ip=_client_ip(request),
user_agent=request.headers.get("user-agent", "")[:512],
)
if doc["outcome"] == "invalid":
return {"status": "success", "id": doc["id"]}
try:
result = await karaokehunt_conversion.process_intake(doc["id"])
logger.info("karaokehunt: processed %s -> %s", doc["id"], result.get("status"))
except Exception: # noqa: BLE001 — the intake doc + reprocess endpoint carry retries
logger.exception("karaokehunt: conversion failed for %s", doc["id"])
try:
karaokehunt_conversion._get_db().collection(
karaokehunt_conversion.COLLECTION
).document(doc["id"]).update({"outcome": "error", "error": "unhandled exception"})
except Exception: # noqa: BLE001
logger.exception("karaokehunt: failed to mark %s as error", doc["id"])
# The app ignores the body; always succeed so nothing user-visible breaks.
return {"status": "success", "id": doc["id"]}
@router.post("/internal/reprocess/{doc_id}")
async def reprocess_karaokehunt_request(
doc_id: str, auth_result=Depends(require_admin)
) -> dict:
"""Admin retry for a failed/errored intake (idempotent markers prevent
double credits/jobs). Also accepts force=true via query for terminal docs."""
return await karaokehunt_conversion.process_intake(doc_id, force=True)
-625
View File
@@ -1,625 +0,0 @@
"""kjbox partner API — singer email-code sign-in + "free at the show" credits.
kjbox (the KJ device app at live karaoke nights) lets a patron verify their
email inside the kjbox singer web page, then creates gen jobs AS that real gen
user (with the returned session token) so they get normal gen delivery emails.
Every endpoint requires ``X-Kjbox-Secret`` (503 until ``KJBOX_PARTNER_SECRET``
is configured — deploys dark; 403 if wrong). Contract:
- POST /api/kjbox/auth/send-code {email, locale?, venue?} → {"status": "sent"}
- POST /api/kjbox/auth/verify-code {email, code} → {session_token, user,
credits_granted, credit_status}
- POST /api/kjbox/auth/google | apple {id_token, locale?, venue?} → same as
verify-code (one-tap sign-in in the singer page)
- POST /api/kjbox/credits/show-credit (+ Authorization: Bearer <session>)
{idempotency_key, venue?, only_if_empty?}
→ {granted, credits}
A venue is many singers behind one IP, so gen's 2-signups-per-IP cap does NOT
apply here; instead a partner-wide rolling-24h cap on new VERIFIED accounts
(accounts are only created at verify-code), a per-email code throttle, and a
per-user daily show-credit cap bound abuse.
"""
from __future__ import annotations
import asyncio
import hmac
import logging
import threading
from datetime import datetime
from typing import Optional
from fastapi import APIRouter, Depends, Header, HTTPException, Query, Request, Response
from pydantic import BaseModel, Field
from backend.api.routes.users import (
_build_user_public,
_mask_email,
_precompute_credit_eval,
complete_verified_login,
)
from backend.config import get_settings, is_production
from backend.i18n import SUPPORTED_LOCALES, t
from backend.services.account_deletion_service import (
AccountDeletionService,
get_account_deletion_service,
)
from backend.services.email_service import EmailService, get_email_service
from backend.services.email_validation_service import get_email_validation_service
from backend.services.kjbox_partner_service import (
CODE_EXPIRY_MINUTES,
LOGIN_CODES_COLLECTION,
KjboxPartnerService,
CodeThrottled,
ShowCreditCapReached,
TransactionContention,
VerifyOutcome,
)
from backend.services.user_service import UserService, get_user_service
from backend.utils.request_helpers import get_client_ip
from backend.utils.test_data import is_test_email
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/kjbox", tags=["kjbox"])
SIGNUP_SOURCE = "kjbox"
_MAX_EMAIL_LEN = 254
_MAX_VENUE_LEN = 200
# ------------------------------------------------------------------ auth gate
def require_kjbox_partner(x_kjbox_secret: Optional[str] = Header(default=None)) -> str:
"""Shared-secret gate. Returns the secret (also used as the code-hash pepper)."""
secret = get_settings().kjbox_partner_secret
if not secret:
raise HTTPException(status_code=503, detail="not configured")
if not hmac.compare_digest((x_kjbox_secret or "").encode(), secret.encode()):
raise HTTPException(status_code=403, detail="forbidden")
return secret
def require_resolve_partner(
x_kjbox_secret: Optional[str] = Header(default=None),
x_partner_secret: Optional[str] = Header(default=None),
) -> str:
"""Gate for ``catalog/resolve``: kjbox (``X-Kjbox-Secret``) or karaoke-decide
(``X-Partner-Secret``). Returns the partner name ("kjbox" / "decide"), which
picks the rate-budget bucket. 503 when neither secret is configured."""
settings = get_settings()
kjbox_secret = settings.kjbox_partner_secret
decide_secret = getattr(settings, "decide_partner_secret", "")
if not kjbox_secret and not decide_secret:
raise HTTPException(status_code=503, detail="not configured")
if kjbox_secret and x_kjbox_secret and hmac.compare_digest(
x_kjbox_secret.encode(), kjbox_secret.encode()):
return "kjbox"
if decide_secret and x_partner_secret and hmac.compare_digest(
x_partner_secret.encode(), decide_secret.encode()):
return "decide"
raise HTTPException(status_code=403, detail="forbidden")
def get_kjbox_service(
secret: str = Depends(require_kjbox_partner),
user_service: UserService = Depends(get_user_service),
) -> KjboxPartnerService:
return KjboxPartnerService(user_service.db, pepper=secret)
# --------------------------------------------------------------------- models
class SendCodeRequest(BaseModel):
email: str
locale: Optional[str] = None
venue: Optional[str] = Field(default=None, max_length=_MAX_VENUE_LEN)
class SendCodeResponse(BaseModel):
status: str
class VerifyCodeRequest(BaseModel):
email: str
code: str = Field(max_length=32)
class SocialSignInRequest(BaseModel):
# Google Identity Services / Sign in with Apple JS ID token from the singer page
# (or ``fake:<email>`` on the local stack).
id_token: str = Field(min_length=1, max_length=8192)
locale: Optional[str] = None
venue: Optional[str] = Field(default=None, max_length=_MAX_VENUE_LEN)
class VerifyCodeResponse(BaseModel):
session_token: str
user: dict
credits_granted: int
credit_status: str
class ShowCreditRequest(BaseModel):
idempotency_key: str = Field(min_length=1, max_length=128)
venue: Optional[str] = Field(default=None, max_length=_MAX_VENUE_LEN)
# Grant only if the singer currently has no credits (< 1). The key is claimed
# only when a credit is actually granted, so a later call with the same key
# and only_if_empty=false grants iff this one didn't.
only_if_empty: bool = False
class ShowCreditResponse(BaseModel):
granted: bool
credits: int
# -------------------------------------------------------------------- helpers
def _normalize_email(raw: str) -> str:
email = (raw or "").strip().lower()
local, _, domain = email.partition("@")
if (not local or "." not in domain or len(email) > _MAX_EMAIL_LEN
or any(c.isspace() for c in email) or email.count("@") != 1):
raise HTTPException(status_code=422, detail="invalid_email")
return email
def _split_locale(raw: Optional[str]) -> tuple[Optional[str], Optional[str]]:
"""Return (email_locale in en/es/de or None, ui_locale primary subtag or None)."""
if not raw:
return None, None
primary = raw.strip().replace("_", "-").split("-")[0].lower()
if not primary.isalpha() or len(primary) < 2:
return None, None
return (primary if primary in SUPPORTED_LOCALES else None), primary
def _pretend_sent() -> SendCodeResponse:
return SendCodeResponse(status="sent")
# ------------------------------------------------------------------ endpoints
@router.post("/auth/send-code", response_model=SendCodeResponse)
def send_code(
body: SendCodeRequest,
http_request: Request,
svc: KjboxPartnerService = Depends(get_kjbox_service),
user_service: UserService = Depends(get_user_service),
email_service: EmailService = Depends(get_email_service),
):
settings = get_settings()
email = _normalize_email(body.email)
email_locale, ui_locale = _split_locale(body.locale)
locale = email_locale or "en"
venue = (body.venue or "").strip() or None
email_validation = get_email_validation_service()
if email_validation.is_disposable_domain(email):
logger.warning(f"kjbox: blocked disposable email {_mask_email(email)}")
raise HTTPException(status_code=422, detail=t(locale, "users.disposableEmailBlocked"))
if email_validation.is_email_blocked(email):
logger.warning(f"kjbox: blocked email {_mask_email(email)} — pretending success")
return _pretend_sent()
ip_address = get_client_ip(http_request)
if ip_address and email_validation.is_ip_blocked(ip_address):
logger.warning(f"kjbox: blocked IP {ip_address} — pretending success")
return _pretend_sent()
if not email_service.is_configured() and is_production():
logger.error("kjbox: email service not configured - cannot send codes")
raise HTTPException(status_code=503, detail="email not configured")
# No account is created here — only once the email is verified (verify-code),
# so unverified addresses can neither burn the signup cap nor create accounts
# for other people's emails. Venue/locale ride on the code record until then.
user = user_service.get_user(email)
try:
issued = svc.issue_code(
email,
max_per_hour=settings.kjbox_codes_per_email_per_hour,
needs_credit_eval=(
not (user and getattr(user, "welcome_credits_granted", False))
and not is_test_email(email)
),
locale=email_locale,
ui_locale=ui_locale,
venue=venue,
ip_address=ip_address,
user_agent=http_request.headers.get("user-agent"),
)
except (CodeThrottled, TransactionContention):
# Contention on one email's code doc = a burst of parallel sends to the
# same address, which is exactly what the throttle exists to stop.
raise HTTPException(status_code=429, detail="too_many_codes")
sent = email_service.send_kjbox_login_code(
email, issued.code, expiry_minutes=CODE_EXPIRY_MINUTES, locale=locale,
)
if not sent:
if getattr(email_service, "last_send_suppressed", False):
logger.info(f"kjbox: code recipient {_mask_email(email)} is Postmark-suppressed")
else:
logger.error(f"kjbox: failed to send code email to {_mask_email(email)}")
# Warm the welcome-credit AI evaluation while the singer reads their email,
# so verify-code is instant (same optimisation as magic links).
if issued.needs_credit_eval:
threading.Thread(
target=_precompute_credit_eval,
args=(issued.doc_id, email, LOGIN_CODES_COLLECTION),
daemon=True,
).start()
return SendCodeResponse(status="sent")
@router.post("/auth/verify-code", response_model=VerifyCodeResponse)
def verify_code(
body: VerifyCodeRequest,
http_request: Request,
svc: KjboxPartnerService = Depends(get_kjbox_service),
user_service: UserService = Depends(get_user_service),
email_service: EmailService = Depends(get_email_service),
):
settings = get_settings()
email = _normalize_email(body.email)
# New accounts are created (and counted against the partner-wide cap) only
# here, after the email is proven. Check the cap BEFORE consuming the code so
# a capped singer can retry the same code later.
existing = user_service.get_user(email)
if existing is None and svc.count_recent_signups() >= settings.kjbox_signup_cap_per_24h:
logger.warning("kjbox: partner-wide verified-signup cap reached")
raise HTTPException(status_code=429, detail="signup_cap")
outcome, record = svc.verify_code(email, body.code)
if outcome == VerifyOutcome.TOO_MANY_ATTEMPTS:
raise HTTPException(status_code=429, detail="too_many_attempts")
if outcome == VerifyOutcome.INVALID:
raise HTTPException(status_code=401, detail="invalid_code")
if outcome != VerifyOutcome.OK or record is None:
raise HTTPException(status_code=401, detail="expired")
precomputed_eval = {
k: record.get(k)
for k in ("credit_eval_decision", "credit_eval_reasoning", "credit_eval_error")
if record.get(k) is not None
} or None
return _finish_sign_in(
email, http_request, svc=svc, user_service=user_service, email_service=email_service,
venue=record.get("venue"), email_locale=record.get("locale"),
ui_locale=record.get("ui_locale"), precomputed_eval=precomputed_eval, how="verified",
)
def _finish_sign_in(email, http_request, *, svc, user_service, email_service, venue,
email_locale, ui_locale, precomputed_eval, how):
"""A proven email (code, or a Google/Apple ID token) → the singer's gen session."""
user = user_service.get_user(email)
if user is None:
# No signup_ip: the caller is the kjbox box, whose IP is the whole
# venue's — recording it would trip gen's per-IP cap for everyone there.
user = user_service.get_or_create_user(
email, signup_source=SIGNUP_SOURCE, signup_venue=venue,
)
svc.record_signup(email, venue)
logger.info(f"kjbox: created verified gen user {_mask_email(email)} (venue={venue!r}, via {how})")
# Same first-login definition as magic-link verify (checked before last_login_at is set).
is_first_login = user.total_jobs_created == 0 and not user.last_login_at
user = user_service.update_user(
email, email_verified=True, last_login_at=datetime.utcnow(),
) or user
result = complete_verified_login(
user,
user_service=user_service,
email_service=email_service,
is_first_login=is_first_login,
precomputed_eval=precomputed_eval,
grant_welcome_credit=True,
referral_code=None,
locale=email_locale,
ui_locale=ui_locale,
ip_address=get_client_ip(http_request),
user_agent=http_request.headers.get("user-agent"),
tenant_id=None,
device_fingerprint=None,
email_locale=email_locale or user.locale or "en",
)
logger.info(f"kjbox: {how} {_mask_email(email)} (credit_status={result.credit_status})")
return VerifyCodeResponse(
session_token=result.session.token,
user=_build_user_public(result.user).model_dump(mode="json"),
credits_granted=result.credits_granted,
credit_status=result.credit_status,
)
def _social_sign_in(provider, body, http_request, svc, user_service, email_service):
"""Google / Apple one-tap: the provider has proven the email, so no code.
Same venue rules as verify-code: no per-IP signup cap, the partner-wide 24h
cap on new accounts instead. Apple "Hide My Email" relays are their own account."""
from backend.api.routes.login_code import APPLE_RELAY_DOMAIN, _verify_identity
identity = _verify_identity(provider, body.id_token) # 401 invalid_token / 503
email = _normalize_email(identity.email)
email_locale, ui_locale = _split_locale(body.locale)
# The same eligibility checks as send-code. There's no email to hide a
# refusal behind, so (like the app's social sign-in) refusals are explicit.
email_validation = get_email_validation_service()
is_relay = provider == "apple" and email.endswith("@" + APPLE_RELAY_DOMAIN)
if not is_relay and email_validation.is_disposable_domain(email):
logger.warning(f"kjbox: {provider} sign-in refused, disposable email {_mask_email(email)}")
raise HTTPException(status_code=422, detail=t(email_locale or "en", "users.disposableEmailBlocked"))
ip_address = get_client_ip(http_request)
if email_validation.is_email_blocked(email) or (ip_address and email_validation.is_ip_blocked(ip_address)):
logger.warning(f"kjbox: {provider} sign-in refused, blocked email/IP {_mask_email(email)}")
raise HTTPException(status_code=422, detail="sign_in_not_allowed")
if user_service.get_user(email) is None and \
svc.count_recent_signups() >= get_settings().kjbox_signup_cap_per_24h:
logger.warning("kjbox: partner-wide verified-signup cap reached")
raise HTTPException(status_code=429, detail="signup_cap")
return _finish_sign_in(
email, http_request, svc=svc, user_service=user_service, email_service=email_service,
venue=(body.venue or None), email_locale=email_locale, ui_locale=ui_locale,
precomputed_eval=None, how=f"{provider} sign-in",
)
@router.post("/auth/google", response_model=VerifyCodeResponse)
def google_sign_in(
body: SocialSignInRequest,
http_request: Request,
svc: KjboxPartnerService = Depends(get_kjbox_service),
user_service: UserService = Depends(get_user_service),
email_service: EmailService = Depends(get_email_service),
):
"""Google ID token (audience ``GOOGLE_OAUTH_CLIENT_IDS``) → singer session."""
return _social_sign_in("google", body, http_request, svc, user_service, email_service)
@router.post("/auth/apple", response_model=VerifyCodeResponse)
def apple_sign_in(
body: SocialSignInRequest,
http_request: Request,
svc: KjboxPartnerService = Depends(get_kjbox_service),
user_service: UserService = Depends(get_user_service),
email_service: EmailService = Depends(get_email_service),
):
"""Apple ID token (audience ``APPLE_SIGNIN_AUDIENCES``, incl. the web Services ID) → singer session."""
return _social_sign_in("apple", body, http_request, svc, user_service, email_service)
@router.post("/credits/show-credit", response_model=ShowCreditResponse)
def show_credit(
body: ShowCreditRequest,
svc: KjboxPartnerService = Depends(get_kjbox_service),
user_service: UserService = Depends(get_user_service),
authorization: Optional[str] = Header(default=None),
):
"""Quietly grant +1 credit so a singer's make-it job is free at the show.
No "credits added" email. Idempotent per ``idempotency_key`` (claimed only
when a credit is granted); ``only_if_empty`` grants only at a 0 balance;
capped per user per rolling 24h. All decisions + the increment are one
Firestore transaction.
"""
settings = get_settings()
scheme, _, token = (authorization or "").partition(" ")
token = token.strip()
if scheme.lower() != "bearer" or not token:
raise HTTPException(status_code=401, detail="invalid_session")
valid, user, _msg = user_service.validate_session(token)
if not valid or not user:
raise HTTPException(status_code=401, detail="invalid_session")
email = user.email.lower()
venue = (body.venue or "").strip() or None
user_ref = user_service.get_user_doc_ref(email)
if user_ref is None:
raise HTTPException(status_code=401, detail="invalid_session")
try:
result = svc.grant_show_credit(
user_ref,
email,
body.idempotency_key,
venue=venue,
only_if_empty=body.only_if_empty,
max_per_24h=settings.kjbox_show_credits_per_user_per_24h,
)
except ShowCreditCapReached:
raise HTTPException(status_code=429, detail="show_credit_cap")
except LookupError:
raise HTTPException(status_code=401, detail="invalid_session")
except TransactionContention:
# Nothing was written; kjbox retries with the same idempotency key.
raise HTTPException(status_code=409, detail="busy_retry")
if result.granted:
logger.info(f"kjbox: show credit granted to {_mask_email(email)} (venue={venue!r}) → {result.credits}")
return ShowCreditResponse(granted=result.granted, credits=result.credits)
# ------------------------------------------------------------- review link
class ReviewLinkRequest(BaseModel):
locale: Optional[str] = None
class ReviewLinkResponse(BaseModel):
url: str
status: str
# "admin" = the KJ / support already opened the review; "owner" = the
# singer did; None = nobody has opened it yet.
review_started_by: Optional[str] = None
_REVIEW_LINK_STATUSES = ("awaiting_review", "in_review")
# ------------------------------------------------- account deletion queue
@router.get("/account-deletions")
def list_account_deletions(
since: Optional[datetime] = Query(default=None),
_secret: str = Depends(require_kjbox_partner),
deletion: AccountDeletionService = Depends(get_account_deletion_service),
):
"""Deleted gen accounts the box still has to erase (it pulls: it may be offline).
Only the sha256 of the email is sent; the box hashes the ``gen_email`` values
it stores to match. Lists entries still pending for kjbox, created after ``since``.
"""
return {"deletions": deletion.list_kjbox_pending(since)}
@router.post("/account-deletions/{entry_id}/ack", status_code=204)
def ack_account_deletion(
entry_id: str,
_secret: str = Depends(require_kjbox_partner),
deletion: AccountDeletionService = Depends(get_account_deletion_service),
):
"""The box erased this account's data; stop listing it. ``entry_id`` = sha256(email)."""
if not deletion.ack_kjbox(entry_id):
raise HTTPException(status_code=404, detail="not_found")
return Response(status_code=204)
@router.post("/jobs/{job_id}/review-link", response_model=ReviewLinkResponse)
async def review_link(
job_id: str,
body: ReviewLinkRequest,
_secret: str = Depends(require_kjbox_partner),
user_service: UserService = Depends(get_user_service),
authorization: Optional[str] = Header(default=None),
):
"""One-click sign-in link straight to the singer's own job's lyrics review.
Backs the "Tap here to review it yourself" link on the kjbox singer page:
the same single-use ``job_review:<id>`` magic link the kjbox review email
carries, minted on demand. Only the job's owner (the Bearer session) may
get one, and only while the job is in review (409 ``not_in_review``).
"""
scheme, _, token = (authorization or "").partition(" ")
token = token.strip()
if scheme.lower() != "bearer" or not token:
raise HTTPException(status_code=401, detail="invalid_session")
locale = body.locale if body.locale in SUPPORTED_LOCALES else "en"
# Session check, job read and token mint are blocking Firestore calls —
# keep them off the event loop (it serves every in-flight request).
url, status, started_by = await asyncio.to_thread(
_mint_review_link, user_service, token, job_id, locale)
return ReviewLinkResponse(url=url, status=status, review_started_by=started_by)
def get_owned_review_job(user_service: UserService, session_token: str, job_id: str):
"""Blocking: the session owner's job, checked to be in review → (job, status).
Shared by kjbox's review link and the mobile app's web handoff
(``POST /api/users/auth/handoff-token``). Raises 401 ``invalid_session``,
404 ``not_found`` (missing or not yours — same answer, so other users' job
ids aren't confirmed) or 409 ``not_in_review``.
"""
from backend.services.job_manager import JobManager
valid, user, _msg = user_service.validate_session(session_token)
if not valid or not user:
raise HTTPException(status_code=401, detail="invalid_session")
job = JobManager().get_job(job_id)
if not job or (job.user_email or "").lower() != user.email.lower():
raise HTTPException(status_code=404, detail="not_found")
status = str(getattr(job.status, "value", job.status))
if status not in _REVIEW_LINK_STATUSES:
raise HTTPException(status_code=409, detail="not_in_review")
return job, status
def _mint_review_link(user_service: UserService, token: str, job_id: str, locale: str):
"""Blocking half of ``review_link`` → (url, status, review_started_by)."""
from backend.services.job_notification_service import build_review_login_url
job, status = get_owned_review_job(user_service, token, job_id)
url = build_review_login_url(job.job_id, job.user_email, locale=locale)
if not url:
raise HTTPException(status_code=502, detail="link_unavailable")
started_by = (job.state_data or {}).get("review_started_by") if status == "in_review" else None
return url, status, started_by
# ---------------------------------------------------------- search resolve
class ResolveRequest(BaseModel):
query: str = Field(min_length=1, max_length=200)
_RESOLVE_CACHE_MAX = 2000
_RESOLVE_PER_MINUTE = 120
_resolve_cache: "dict[str, dict]" = {}
_RESOLVE_DECIDE_PER_MINUTE = 60 # karaoke-decide's own bucket, so kjbox nights are never starved
_resolve_calls: "list[float]" = []
_resolve_calls_decide: "list[float]" = []
_resolve_lock = threading.Lock()
def _resolve_budget_ok(partner: str = "kjbox") -> bool:
"""Per-partner in-process cap on AI calls (cache hits are free)."""
import time
now = time.monotonic()
if partner == "decide":
calls, limit = _resolve_calls_decide, _RESOLVE_DECIDE_PER_MINUTE
else:
calls, limit = _resolve_calls, _RESOLVE_PER_MINUTE
with _resolve_lock:
while calls and calls[0] < now - 60:
calls.pop(0)
if len(calls) >= limit:
return False
calls.append(now)
return True
@router.post("/catalog/resolve")
async def catalog_resolve(body: ResolveRequest, partner: str = Depends(require_resolve_partner)):
"""Split + auto-correct a singer's free-text song search.
kjbox calls this only when its own catalogue search found nothing, then
shows gen's AudioSourceStep "Tidied to … / keep what I typed" (re-searching
when the song changed, and pre-filling its generate form). The AI splits the
one-line query; the job flow's judge_match catalog pass then tidies it, so
the canonical artist/title match what gen's own job form would use.
Response = the match judge verdict shape + typed_artist / typed_title.
Never 5xx on model trouble: kind "none".
"""
from backend.services.match_judge.free_text import _none, resolve_and_tidy
query = " ".join(body.query.split())
if not query:
raise HTTPException(status_code=422, detail="empty_query")
from backend.services.fake_workers import fake_workers_enabled
if fake_workers_enabled(get_settings()):
# Local E2E stack: scripted verdict, no Gemini, no cache/budget (tests change the knobs).
from backend.services.fake_workers.judge import fake_resolve
return fake_resolve(query)
key = query.casefold() # shared by both partners: same query -> same verdict
with _resolve_lock:
hit = _resolve_cache.get(key)
if hit is not None:
# Same query modulo case/spacing: the verdict applies, but echo THIS
# request's text where the verdict fell back to the typed query.
if not hit.get("typed_artist"):
hit = {**hit, "typed_title": query}
return hit
if not _resolve_budget_ok(partner):
raise HTTPException(status_code=429, detail="rate_limited")
verdict = await resolve_and_tidy(query)
if verdict.get("reason") != "unavailable": # don't cache transient failures
with _resolve_lock:
if len(_resolve_cache) >= _RESOLVE_CACHE_MAX:
_resolve_cache.pop(next(iter(_resolve_cache)))
_resolve_cache[key] = verdict
return verdict or _none(query)
-628
View File
@@ -1,628 +0,0 @@
"""gen app sign-in: email code (``send-code`` + ``verify-code``), Google / Apple
ID tokens, and the app → web handoff link.
An alternative to the magic link for clients that can't follow an emailed link
back into the same session (the mobile app, a second device, local E2E). The
email carries BOTH a 6-digit code and a regular magic link; either signs in.
- POST /api/users/auth/send-code {email, locale?, device_fingerprint?,
referral_code?, attribution?} → 202 {"status": "sent"}
- POST /api/users/auth/verify-code {email, code} → VerifyMagicLinkResponse
- POST /api/users/auth/google | apple {id_token, locale?, device_fingerprint?,
referral_code?, attribution?} → VerifyMagicLinkResponse
- POST /api/users/auth/handoff-token {target, job_id, locale?} (Bearer)
→ {url, expires_in_seconds}: the mobile app's
2-minute, single-use signed-in web link
Code mechanics come from :class:`LoginCodeStore` (shared with kjbox): HMAC-hashed
at rest with a server pepper, 10-minute TTL, 5 verify attempts then 429,
single-use marker, and per-email + per-IP rolling-hour send throttles checked in
the same Firestore transaction as the code write.
Abuse checks mirror ``POST /auth/magic-link``: disposable domain → 422 (unless a
tenant explicitly allowlisted the address), blocked email / IP → pretend
success, new-user per-IP/fingerprint signup cap → pretend success, tenant
email-domain restriction → pretend success, no email provider in production →
503. Pretend-success responses are indistinguishable from a real send.
Account creation: the email's magic link is minted with the normal
``UserService.create_magic_link`` machinery, which (exactly like the magic-link
endpoint) creates the account at send time, recording signup IP, device
fingerprint and first-touch attribution. That keeps magic-link parity (same
signup caps, same audit fields) instead of inventing a lazily-minted link.
verify-code still re-creates the account from the code record if it's gone.
"""
from __future__ import annotations
import hashlib
import logging
import threading
from dataclasses import dataclass
from datetime import datetime
from typing import Annotated, Dict, Literal, Optional
from fastapi import APIRouter, Depends, Header, HTTPException, Request
from pydantic import BaseModel, Field, StringConstraints
from backend.api.routes.kjbox import _normalize_email, _split_locale, get_owned_review_job
from backend.api.routes.users import (
_build_user_public,
_mask_email,
_precompute_credit_eval,
complete_verified_login,
)
from backend.config import get_settings, is_production
from backend.i18n import WEB_LOCALES, get_full_locale_from_request, get_locale_from_request, t
from backend.models.user import VerifyMagicLinkResponse
from backend.services.email_service import EmailService, get_email_service
from backend.services.email_validation_service import get_email_validation_service
from backend.services.fake_workers import fake_workers_enabled
from backend.services import social_identity
from backend.services.login_code_store import (
CODE_EXPIRY_MINUTES,
CodeThrottled,
LoginCodeStore,
TransactionContention,
VerifyOutcome,
)
from backend.services.user_service import MAGIC_LINKS_COLLECTION, UserService, get_user_service
from backend.utils.request_helpers import get_client_ip
from backend.utils.test_data import is_test_email
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/users", tags=["users"])
LOGIN_CODES_COLLECTION = "login_codes"
LOGIN_CODE_USES_COLLECTION = "login_code_uses"
LOGIN_CODE_IP_COLLECTION = "login_code_ip_sends"
# Used only outside production when neither LOGIN_CODE_PEPPER nor an admin token
# is configured (local dev / unit tests). Never used in production.
_DEV_PEPPER = "gen-login-code-dev-pepper"
def _resolve_pepper() -> Optional[str]:
"""HMAC pepper for stored codes.
Order: ``LOGIN_CODE_PEPPER`` → sha256("gen-login-code-pepper:" + first admin
token) → (non-production only) a fixed dev pepper. Returns None in production
with nothing configured (endpoints then 503). Rotating the first admin token
without setting LOGIN_CODE_PEPPER invalidates outstanding codes (≤10 min).
"""
settings = get_settings()
if settings.login_code_pepper:
return settings.login_code_pepper
tokens = [tok.strip() for tok in (settings.admin_tokens or "").split(",") if tok.strip()]
if tokens:
return hashlib.sha256(f"gen-login-code-pepper:{tokens[0]}".encode("utf-8")).hexdigest()
if not is_production():
return _DEV_PEPPER
return None
def get_login_code_store(user_service: UserService = Depends(get_user_service)) -> LoginCodeStore:
pepper = _resolve_pepper()
if not pepper:
logger.error("login-code: no LOGIN_CODE_PEPPER / admin token configured in production")
raise HTTPException(status_code=503, detail="not configured")
return LoginCodeStore(
user_service.db,
pepper,
LOGIN_CODES_COLLECTION,
LOGIN_CODE_USES_COLLECTION,
ip_collection=LOGIN_CODE_IP_COLLECTION,
)
# --------------------------------------------------------------------- models
class SendLoginCodeRequest(BaseModel):
email: str
locale: Optional[str] = None
device_fingerprint: Optional[str] = Field(default=None, max_length=256)
referral_code: Optional[str] = Field(default=None, max_length=64)
# First-touch attribution, same shape/limits as SendMagicLinkRequest.attribution.
attribution: Optional[
Dict[Annotated[str, StringConstraints(max_length=32)], Optional[Annotated[str, StringConstraints(max_length=500)]]]
] = Field(default=None, max_length=16)
class SendLoginCodeResponse(BaseModel):
status: str
class VerifyLoginCodeRequest(BaseModel):
email: str
code: str = Field(max_length=32)
def _sent() -> SendLoginCodeResponse:
return SendLoginCodeResponse(status="sent")
# ------------------------------------------------------------------ endpoints
@router.post("/auth/send-code", response_model=SendLoginCodeResponse, status_code=202)
def send_login_code(
body: SendLoginCodeRequest,
http_request: Request,
store: LoginCodeStore = Depends(get_login_code_store),
user_service: UserService = Depends(get_user_service),
email_service: EmailService = Depends(get_email_service),
):
"""Email a 6-digit sign-in code (plus a magic link) to ``email``."""
from backend.middleware.tenant import get_tenant_config_from_request, get_tenant_from_request
settings = get_settings()
email = _normalize_email(body.email)
email_locale, ui_locale = _split_locale(body.locale)
has_accept_language = bool(http_request.headers.get("accept-language"))
locale = email_locale or get_locale_from_request(http_request)
ui_locale = ui_locale or get_full_locale_from_request(http_request)
# Locale persisted on the user at verify: only an explicit choice (body or
# Accept-Language), never the "en" fallback — that would clobber a stored one.
persist_locale = email_locale or (locale if has_accept_language else None)
tenant_id = get_tenant_from_request(http_request)
tenant_config = get_tenant_config_from_request(http_request)
ip_address = get_client_ip(http_request)
device_fingerprint = body.device_fingerprint
check = _sign_in_checks(
email, http_request, user_service,
device_fingerprint=device_fingerprint, tenant_config=tenant_config, log_prefix="login-code",
)
if check.refusal == "disposable":
raise HTTPException(status_code=422, detail=t(locale, "users.disposableEmailBlocked"))
if check.refusal == "tenant_disallowed":
logger.warning(f"login-code: email not allowed for tenant {tenant_id}: {_mask_email(email)}")
return _sent()
if check.refusal: # blocked email / IP, new-user signup cap: indistinguishable from a send
return _sent()
existing_user = check.existing_user
if not email_service.is_configured() and is_production():
logger.error("login-code: email service not configured - cannot send codes")
raise HTTPException(status_code=503, detail=t(locale, "users.emailServiceNotConfigured"))
user_agent = http_request.headers.get("user-agent")
try:
issued = store.issue_code(
email,
max_per_hour=settings.login_codes_per_email_per_hour,
needs_credit_eval=(
not (existing_user and getattr(existing_user, "welcome_credits_granted", False))
and not is_test_email(email)
and not tenant_id
),
fields={
"locale": persist_locale,
"email_locale": locale,
"ui_locale": ui_locale,
"ip_address": ip_address,
"user_agent": user_agent,
"device_fingerprint": device_fingerprint,
"tenant_id": tenant_id,
"referral_code": body.referral_code,
},
throttle_ip=ip_address,
max_per_ip_per_hour=settings.login_codes_per_ip_per_hour,
)
except (CodeThrottled, TransactionContention):
# Contention on one email's code doc = a burst of parallel sends to the
# same address, which is exactly what the throttle exists to stop.
raise HTTPException(status_code=429, detail="too_many_codes")
# Same machinery as POST /auth/magic-link (creates the account at send time,
# recording signup IP / fingerprint / attribution) — see module docstring.
magic_link = user_service.create_magic_link(
email,
ip_address=ip_address,
user_agent=user_agent,
tenant_id=tenant_id,
device_fingerprint=device_fingerprint,
referral_code=body.referral_code,
attribution=body.attribution,
)
magic_link_url = email_service.build_magic_link_url(
magic_link.token,
locale=locale,
tenant_frontend_url=tenant_config.get_frontend_url() if tenant_config else None,
referral_code=body.referral_code,
)
sent = email_service.send_login_code(
email,
issued.code,
magic_link_url=magic_link_url,
expiry_minutes=CODE_EXPIRY_MINUTES,
locale=locale,
sender_email=tenant_config.get_sender_email() if tenant_config else None,
tenant_name=tenant_config.name if tenant_config else None,
)
if not sent:
if getattr(email_service, "last_send_suppressed", False):
logger.info(f"login-code: recipient {_mask_email(email)} is Postmark-suppressed")
else:
logger.error(f"login-code: failed to send code email to {_mask_email(email)}")
# Don't reveal failure (anti-enumeration).
# Warm the welcome-credit AI evaluation while the user reads their email, on
# both sign-in docs (code record + magic link) so either path is instant.
if issued.needs_credit_eval:
threading.Thread(
target=_precompute_credit_eval,
args=(issued.doc_id, email, LOGIN_CODES_COLLECTION, ((MAGIC_LINKS_COLLECTION, magic_link.token),)),
daemon=True,
).start()
return _sent()
@router.post("/auth/verify-code", response_model=VerifyMagicLinkResponse)
def verify_login_code(
body: VerifyLoginCodeRequest,
http_request: Request,
store: LoginCodeStore = Depends(get_login_code_store),
user_service: UserService = Depends(get_user_service),
email_service: EmailService = Depends(get_email_service),
):
"""Exchange an emailed code for a session (same response as GET /auth/verify)."""
email = _normalize_email(body.email)
settings = get_settings()
demo_email = (settings.app_review_demo_email or "").strip().lower()
demo_code = settings.app_review_demo_code or ""
if demo_email and demo_code and email == demo_email:
# App Review demo account: reviewers can't receive email, so this one
# address signs in with the fixed code. Attempt limits still apply.
outcome, record = store.verify_fixed_code(email, body.code, demo_code)
if outcome == VerifyOutcome.OK:
logger.info("app review demo login")
else:
outcome, record = store.verify_code(email, body.code)
if outcome == VerifyOutcome.TOO_MANY_ATTEMPTS:
raise HTTPException(status_code=429, detail="too_many_attempts")
if outcome == VerifyOutcome.INVALID:
raise HTTPException(status_code=401, detail="invalid_code")
if outcome != VerifyOutcome.OK or record is None:
raise HTTPException(status_code=401, detail="expired")
tenant_id = record.get("tenant_id")
user = user_service.get_user(email)
if user is None:
# Normally created at send time by create_magic_link; recreate from the
# code record (same signup audit fields) if it was removed meanwhile.
user = user_service.get_or_create_user(
email,
tenant_id=tenant_id,
signup_ip=record.get("ip_address"),
device_fingerprint=record.get("device_fingerprint"),
)
precomputed_eval = {
k: record.get(k)
for k in ("credit_eval_decision", "credit_eval_reasoning", "credit_eval_error")
if record.get(k) is not None
} or None
persist_locale = record.get("locale")
return finish_verified_sign_in(
user,
http_request,
user_service=user_service,
email_service=email_service,
tenant_id=tenant_id,
precomputed_eval=precomputed_eval,
referral_code=http_request.headers.get("x-referral-code") or record.get("referral_code"),
persist_locale=persist_locale,
ui_locale=record.get("ui_locale"),
email_locale=persist_locale or record.get("email_locale"),
device_fingerprint=record.get("device_fingerprint"),
log_prefix="login-code",
)
# ------------------------------------------------------- shared sign-in steps
@dataclass
class SignInCheck:
"""Outcome of :func:`_sign_in_checks`; ``refusal`` is None when sign-in may go ahead."""
refusal: Optional[str] # disposable | blocked_email | blocked_ip | signup_limited | tenant_disallowed
existing_user: object = None
APPLE_RELAY_DOMAIN = "privaterelay.appleid.com"
def _sign_in_checks(
email: str,
http_request: Request,
user_service: UserService,
*,
device_fingerprint: Optional[str],
tenant_config,
log_prefix: str,
allow_apple_relay: bool = False,
) -> SignInCheck:
"""Abuse / eligibility checks shared by every sign-in that can create an account
(email code, Google, Apple), mirroring ``POST /auth/magic-link``: disposable
domain (unless a tenant explicitly allowlisted the address, or it's an Apple
"Hide My Email" relay on the Apple route), blocked email, blocked IP, new-user
per-IP/fingerprint signup cap, tenant email-domain restriction. Blocking
Firestore / DeBounce calls: run from sync routes only.
"""
email_validation = get_email_validation_service()
vouched = bool(tenant_config and tenant_config.is_explicitly_allowlisted(email))
if allow_apple_relay and email.rsplit("@", 1)[-1] == APPLE_RELAY_DOMAIN:
vouched = True # Apple-issued forwarding address tied to a real Apple ID
if not vouched and email_validation.is_disposable_domain(email):
logger.warning(f"{log_prefix}: blocked disposable email {_mask_email(email)}")
return SignInCheck("disposable")
if email_validation.is_email_blocked(email):
logger.warning(f"{log_prefix}: blocked email {_mask_email(email)}")
return SignInCheck("blocked_email")
ip_address = get_client_ip(http_request)
if ip_address and email_validation.is_ip_blocked(ip_address):
logger.warning(f"{log_prefix}: blocked IP {ip_address}")
return SignInCheck("blocked_ip")
existing_user = user_service.get_user(email)
if existing_user is None and user_service.is_signup_rate_limited(
ip_address=ip_address, device_fingerprint=device_fingerprint,
):
logger.warning(f"{log_prefix}: signup rate limit hit: IP={ip_address} for {_mask_email(email)}")
return SignInCheck("signup_limited")
if tenant_config and not tenant_config.is_email_allowed(email):
return SignInCheck("tenant_disallowed")
return SignInCheck(None, existing_user)
def finish_verified_sign_in(
user,
http_request: Request,
*,
user_service: UserService,
email_service: EmailService,
tenant_id: Optional[str],
precomputed_eval: Optional[dict],
referral_code: Optional[str],
persist_locale: Optional[str],
ui_locale: Optional[str],
email_locale: Optional[str],
device_fingerprint: Optional[str],
log_prefix: str,
) -> VerifyMagicLinkResponse:
"""After an identity proof (email code, Google or Apple ID token): mark the
user verified, then the shared ``complete_verified_login`` (welcome credit,
referral, locale, session, welcome email) and the ``VerifyMagicLinkResponse``.
``user`` must already exist. Tenant portal sign-ins never touch consumer
credits (as in magic-link verify).
"""
email = user.email
# Same first-login definition as magic-link verify (checked before last_login_at is set).
is_first_login = user.total_jobs_created == 0 and not user.last_login_at
user = user_service.update_user(
email, email_verified=True, last_login_at=datetime.utcnow(),
) or user
email_locale = email_locale or user.locale or "en"
result = complete_verified_login(
user,
user_service=user_service,
email_service=email_service,
is_first_login=is_first_login,
precomputed_eval=precomputed_eval,
grant_welcome_credit=not tenant_id,
referral_code=referral_code if not tenant_id else None,
locale=persist_locale,
ui_locale=ui_locale,
ip_address=get_client_ip(http_request),
user_agent=http_request.headers.get("user-agent"),
tenant_id=tenant_id,
device_fingerprint=device_fingerprint,
email_locale=email_locale,
)
logger.info(f"{log_prefix}: verified {_mask_email(email)} (credit_status={result.credit_status})")
tenant_subdomain = None
if tenant_id:
from backend.services.tenant_service import get_tenant_service
tenant_cfg = get_tenant_service().get_tenant_config(tenant_id)
if tenant_cfg:
tenant_subdomain = tenant_cfg.subdomain
return VerifyMagicLinkResponse(
status="success",
session_token=result.session.token,
user=_build_user_public(result.user),
message=t(email_locale, "users.magicLinkSuccess"),
tenant_subdomain=tenant_subdomain,
credits_granted=result.credits_granted,
credit_status=result.credit_status,
)
# ------------------------------------------------------------- web handoff
# App → web handoff links carry a real sign-in token in the URL, so they live
# only long enough for the app to open the browser, and are single-use (the
# magic-link verify transaction marks them used).
HANDOFF_TOKEN_TTL_SECONDS = 120
class HandoffTokenRequest(BaseModel):
# Allowlist of web destinations; Phase 1 needs only the job's lyrics review.
target: Literal["job_review"]
job_id: str = Field(min_length=1, max_length=64, pattern=r"^[A-Za-z0-9_-]+$")
locale: Optional[str] = None
class HandoffTokenResponse(BaseModel):
url: str
expires_in_seconds: int
def _bearer_token(authorization: Optional[str]) -> str:
scheme, _, token = (authorization or "").partition(" ")
token = token.strip()
if scheme.lower() != "bearer" or not token:
raise HTTPException(status_code=401, detail="invalid_session")
return token
@router.post("/auth/handoff-token", response_model=HandoffTokenResponse)
def create_handoff_token(
body: HandoffTokenRequest,
user_service: UserService = Depends(get_user_service),
authorization: Optional[str] = Header(default=None),
):
"""Signed-in web link for the app's own job review (the mobile "Review on web").
The URL is ``{FRONTEND_URL}/{locale}/auth/verify?token=…&handoff=1``: a
``job_review:<id>`` magic-link token valid for ``HANDOFF_TOKEN_TTL_SECONDS``
and usable once. ``handoff=1`` makes the verify page sign in without the
email-scanner "Complete Sign-In" tap; a used or expired link shows the
normal "link expired" state. 401 ``invalid_session``, 404 ``not_found``
(missing or not the caller's job), 409 ``not_in_review``.
"""
from backend.services.job_notification_service import build_review_login_url
token = _bearer_token(authorization)
# Any of the web app's 33 locales (a French app user lands on /fr/…/review).
locale = body.locale if body.locale in WEB_LOCALES else "en"
# Sync route (threadpool): session check, job read and mint are blocking Firestore calls.
job, _status = get_owned_review_job(user_service, token, body.job_id)
url = build_review_login_url(
job.job_id, job.user_email, locale=locale, handoff_seconds=HANDOFF_TOKEN_TTL_SECONDS,
)
if not url:
raise HTTPException(status_code=502, detail="link_unavailable")
return HandoffTokenResponse(url=url, expires_in_seconds=HANDOFF_TOKEN_TTL_SECONDS)
# ------------------------------------------------------- Google / Apple sign-in
class SocialSignInRequest(BaseModel):
# The ID token from the native Google / Apple SDK (or ``fake:<email>`` on the local stack).
id_token: str = Field(min_length=1, max_length=8192)
locale: Optional[str] = None
device_fingerprint: Optional[str] = Field(default=None, max_length=256)
referral_code: Optional[str] = Field(default=None, max_length=64)
attribution: Optional[
Dict[Annotated[str, StringConstraints(max_length=32)], Optional[Annotated[str, StringConstraints(max_length=500)]]]
] = Field(default=None, max_length=16)
def _verify_identity(provider: str, id_token: str) -> "social_identity.VerifiedIdentity":
"""ID token → verified identity, or 401 ``invalid_token`` / 503 ``google_not_configured``."""
settings = get_settings()
try:
if id_token.startswith(social_identity.FAKE_TOKEN_PREFIX):
# Local stack only (FAKE_WORKERS, refused in production); never a network call.
if not fake_workers_enabled():
raise social_identity.InvalidIdentityToken("fake token outside fake mode")
return social_identity.fake_identity(id_token, provider)
if provider == "google":
audiences = social_identity.parse_audiences(getattr(settings, "google_oauth_client_ids", ""))
if not audiences:
raise HTTPException(status_code=503, detail="google_not_configured")
return social_identity.verify_google_id_token(id_token, audiences)
audiences = social_identity.parse_audiences(getattr(settings, "apple_signin_audiences", ""))
return social_identity.verify_apple_id_token(id_token, audiences)
except social_identity.InvalidIdentityToken as e:
logger.warning(f"{provider}-signin: {e}")
raise HTTPException(status_code=401, detail="invalid_token")
except social_identity.IdentityProviderUnavailable as e:
logger.error(f"{provider}-signin: {e}")
raise HTTPException(status_code=503, detail="identity_provider_unavailable")
def _social_sign_in(
provider: str,
body: SocialSignInRequest,
http_request: Request,
user_service: UserService,
email_service: EmailService,
) -> VerifyMagicLinkResponse:
"""Verify the ID token, apply the same abuse checks as send-code, find or
create the user BY EMAIL, then the same session path as verify-code."""
from backend.middleware.tenant import get_tenant_config_from_request, get_tenant_from_request
log_prefix = f"{provider}-signin"
identity = _verify_identity(provider, body.id_token)
email = identity.email
email_locale, ui_locale = _split_locale(body.locale)
request_locale = get_locale_from_request(http_request)
has_accept_language = bool(http_request.headers.get("accept-language"))
persist_locale = email_locale or (request_locale if has_accept_language else None)
ui_locale = ui_locale or get_full_locale_from_request(http_request)
message_locale = email_locale or request_locale
tenant_id = get_tenant_from_request(http_request)
tenant_config = get_tenant_config_from_request(http_request)
check = _sign_in_checks(
email, http_request, user_service,
device_fingerprint=body.device_fingerprint, tenant_config=tenant_config,
log_prefix=log_prefix, allow_apple_relay=provider == "apple",
)
# Unlike send-code there's no email to hide behind, so refusals are explicit.
if check.refusal == "disposable":
raise HTTPException(status_code=422, detail=t(message_locale, "users.disposableEmailBlocked"))
if check.refusal == "signup_limited":
raise HTTPException(status_code=429, detail="signup_rate_limited")
if check.refusal: # blocked email / IP, tenant email restriction
raise HTTPException(status_code=422, detail="sign_in_not_allowed")
user = check.existing_user or user_service.get_or_create_user_with_attribution(
email,
tenant_id=tenant_id,
signup_ip=get_client_ip(http_request),
device_fingerprint=body.device_fingerprint,
attribution=body.attribution,
)
return finish_verified_sign_in(
user,
http_request,
user_service=user_service,
email_service=email_service,
tenant_id=tenant_id,
precomputed_eval=None,
referral_code=http_request.headers.get("x-referral-code") or body.referral_code,
persist_locale=persist_locale,
ui_locale=ui_locale,
email_locale=persist_locale,
device_fingerprint=body.device_fingerprint,
log_prefix=log_prefix,
)
@router.post("/auth/google", response_model=VerifyMagicLinkResponse)
def google_sign_in(
body: SocialSignInRequest,
http_request: Request,
user_service: UserService = Depends(get_user_service),
email_service: EmailService = Depends(get_email_service),
):
"""Exchange a Google ID token for a gen session (same response as verify-code).
Audience = ``GOOGLE_OAUTH_CLIENT_IDS``. 401 ``invalid_token`` (bad signature,
issuer, audience, expiry, or unverified email), 503 ``google_not_configured``,
503 ``identity_provider_unavailable`` (couldn't fetch the provider's keys;
retryable), 422 disposable / blocked, 429 ``signup_rate_limited`` (new accounts only).
"""
return _social_sign_in("google", body, http_request, user_service, email_service)
@router.post("/auth/apple", response_model=VerifyMagicLinkResponse)
def apple_sign_in(
body: SocialSignInRequest,
http_request: Request,
user_service: UserService = Depends(get_user_service),
email_service: EmailService = Depends(get_email_service),
):
"""Exchange an Apple ID token for a gen session (same response as verify-code).
Audience = ``APPLE_SIGNIN_AUDIENCES``. "Hide My Email" relay addresses
(``@privaterelay.appleid.com``) are accepted as their own account. Errors as
for ``/auth/google`` (no 503: the Apple audience has a default).
"""
return _social_sign_in("apple", body, http_request, user_service, email_service)
-52
View File
@@ -1,52 +0,0 @@
"""POST /api/parse-karaoke-titles — batch karaoke-filename → artist/title.
Internal admin endpoint used by kjbox to canonicalise downloaded-file names.
Reuses the Vertex Gemini parser; never blocks the caller (kjbox degrades to a
deterministic guess when this is unavailable).
"""
import logging
from typing import Optional
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from backend.api.dependencies import require_admin
from backend.services.auth_service import AuthResult
from backend.services.parse_titles import parse_titles
logger = logging.getLogger(__name__)
router = APIRouter(tags=["parse-titles"])
class ParseItem(BaseModel):
id: str
filename: str
channel: Optional[str] = None
source: Optional[str] = None
class ParseRequest(BaseModel):
items: list[ParseItem]
class ParseResult(BaseModel):
id: str
artist: str
title: str
confidence: float
class ParseResponse(BaseModel):
results: list[ParseResult]
@router.post("/parse-karaoke-titles", response_model=ParseResponse)
async def parse_karaoke_titles(
body: ParseRequest,
auth_result: AuthResult = Depends(require_admin),
):
items = [i.model_dump() for i in body.items]
results = await parse_titles(items)
logger.info("parse-karaoke-titles: %d items", len(items))
return {"results": results}
+21 -87
View File
@@ -5,23 +5,19 @@ Provides endpoints for managing Web Push notification subscriptions:
- GET /api/push/vapid-public-key: Get VAPID public key for client-side subscription
- POST /api/push/subscribe: Register a push subscription
- POST /api/push/unsubscribe: Remove a push subscription
- POST/DELETE /api/push/expo-token: Register/remove an Expo push token (mobile app)
- GET /api/push/subscriptions: List user's subscriptions
- POST /api/push/test: Send a test notification (admin only)
"""
import logging
import asyncio
from typing import Literal, Optional, Dict, List
from typing import Optional, Dict, List
from fastapi import APIRouter, Depends, HTTPException, Request, Response
from pydantic import BaseModel, Field
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from backend.config import get_settings
from backend.api.dependencies import require_auth, require_admin
from backend.services.auth_service import AuthResult
from backend.services.push_notification_service import get_push_notification_service
from backend.services.expo_push_service import get_expo_push_service
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
@@ -41,7 +37,6 @@ class SubscribeRequest(BaseModel):
endpoint: str
keys: Dict[str, str] # p256dh and auth
device_name: Optional[str] = None
tenant_id: Optional[str] = None # Tenant scope for notification isolation
class SubscribeResponse(BaseModel):
@@ -88,21 +83,10 @@ class TestNotificationResponse(BaseModel):
message: str
class ExpoTokenRequest(BaseModel):
"""Register an Expo push token for one device of the mobile app."""
token: str = Field(pattern=r"^Expo(nent)?PushToken\[[^\]\s]+\]$")
platform: Literal["ios", "android"]
device_id: str = Field(min_length=1, max_length=200)
class ExpoTokenDeleteRequest(BaseModel):
token: str
# Routes
@router.get("/vapid-public-key", response_model=VapidPublicKeyResponse)
def get_vapid_public_key():
async def get_vapid_public_key():
"""
Get the VAPID public key for push subscription.
@@ -127,8 +111,7 @@ def get_vapid_public_key():
@router.post("/subscribe", response_model=SubscribeResponse)
async def subscribe_push(
subscribe_request: SubscribeRequest,
http_request: Request,
request: SubscribeRequest,
auth_result: AuthResult = Depends(require_auth)
):
"""
@@ -136,53 +119,39 @@ async def subscribe_push(
Requires authentication. Users can have up to 5 subscriptions
(configurable via MAX_PUSH_SUBSCRIPTIONS_PER_USER).
Subscriptions are scoped by tenant_id to prevent cross-tenant
notification leakage. The tenant is resolved from:
1. Request body tenant_id field (explicit from frontend)
2. Tenant middleware (X-Tenant-ID header or subdomain detection)
"""
locale = get_locale_from_request(http_request)
settings = get_settings()
if not settings.enable_push_notifications:
raise HTTPException(status_code=503, detail=t(locale, "push.disabled"))
raise HTTPException(status_code=503, detail="Push notifications are not enabled")
if not auth_result.user_email:
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
raise HTTPException(status_code=401, detail="User email not available")
push_service = get_push_notification_service()
# Validate keys
if "p256dh" not in subscribe_request.keys or "auth" not in subscribe_request.keys:
raise HTTPException(status_code=400, detail=t(locale, "push.missingEncryptionKeys"))
# Resolve tenant_id: prefer explicit from body, fall back to middleware
tenant_id = subscribe_request.tenant_id
if tenant_id is None:
tenant_id = getattr(http_request.state, "tenant_id", None)
if "p256dh" not in request.keys or "auth" not in request.keys:
raise HTTPException(status_code=400, detail="Missing required keys (p256dh, auth)")
success = await push_service.add_subscription(
user_email=auth_result.user_email,
endpoint=subscribe_request.endpoint,
keys=subscribe_request.keys,
device_name=subscribe_request.device_name,
tenant_id=tenant_id or None,
endpoint=request.endpoint,
keys=request.keys,
device_name=request.device_name
)
if not success:
raise HTTPException(status_code=500, detail=t(locale, "push.subscriptionSaveFailed"))
raise HTTPException(status_code=500, detail="Failed to save subscription")
return SubscribeResponse(
status="success",
message=t(locale, "push.subscriptionRegistered")
message="Push subscription registered successfully"
)
@router.post("/unsubscribe", response_model=UnsubscribeResponse)
async def unsubscribe_push(
request: UnsubscribeRequest,
http_request: Request,
auth_result: AuthResult = Depends(require_auth)
):
"""
@@ -190,10 +159,8 @@ async def unsubscribe_push(
Requires authentication. Users can only remove their own subscriptions.
"""
locale = get_locale_from_request(http_request)
if not auth_result.user_email:
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
raise HTTPException(status_code=401, detail="User email not available")
push_service = get_push_notification_service()
@@ -206,45 +173,17 @@ async def unsubscribe_push(
# Don't error if subscription wasn't found - might already be removed
return UnsubscribeResponse(
status="success",
message=t(locale, "push.subscriptionNotFound")
message="Subscription removed (or was not found)"
)
return UnsubscribeResponse(
status="success",
message=t(locale, "push.subscriptionRemoved")
message="Push subscription removed successfully"
)
@router.post("/expo-token", status_code=204)
async def register_expo_token(
body: ExpoTokenRequest,
auth_result: AuthResult = Depends(require_auth),
):
"""Register this device's Expo token. Moves it off any other account that held it."""
if not auth_result.user_email:
raise HTTPException(status_code=401, detail="User email not available")
await asyncio.to_thread(
get_expo_push_service().register_token,
auth_result.user_email, body.token, body.platform, body.device_id,
)
return Response(status_code=204)
@router.delete("/expo-token", status_code=204)
async def delete_expo_token(
body: ExpoTokenDeleteRequest,
auth_result: AuthResult = Depends(require_auth),
):
"""Remove one of the caller's Expo tokens (the app calls this before sign-out)."""
if not auth_result.user_email:
raise HTTPException(status_code=401, detail="User email not available")
await asyncio.to_thread(get_expo_push_service().remove_tokens, auth_result.user_email, [body.token])
return Response(status_code=204)
@router.get("/subscriptions", response_model=SubscriptionsListResponse)
async def list_subscriptions(
request: Request,
auth_result: AuthResult = Depends(require_auth)
):
"""
@@ -252,10 +191,8 @@ async def list_subscriptions(
Requires authentication.
"""
locale = get_locale_from_request(request)
if not auth_result.user_email:
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
raise HTTPException(status_code=401, detail="User email not available")
push_service = get_push_notification_service()
@@ -270,7 +207,6 @@ async def list_subscriptions(
@router.post("/test", response_model=TestNotificationResponse)
async def send_test_notification(
request: TestNotificationRequest,
http_request: Request,
auth_result: AuthResult = Depends(require_admin)
):
"""
@@ -278,14 +214,12 @@ async def send_test_notification(
Admin only. Useful for testing push notification setup.
"""
locale = get_locale_from_request(http_request)
settings = get_settings()
if not settings.enable_push_notifications:
raise HTTPException(status_code=503, detail=t(locale, "push.disabled"))
raise HTTPException(status_code=503, detail="Push notifications are not enabled")
if not auth_result.user_email:
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
raise HTTPException(status_code=401, detail="User email not available")
push_service = get_push_notification_service()
@@ -300,5 +234,5 @@ async def send_test_notification(
return TestNotificationResponse(
status="success",
sent_count=sent_count,
message=t(locale, "push.testNotificationSent", sent_count=sent_count)
message=f"Test notification sent to {sent_count} device(s)"
)
+249 -167
View File
@@ -1,22 +1,24 @@
"""
Admin API routes for blocklist and YouTube queue management.
Admin API routes for rate limit management.
Handles:
- Rate limit statistics and monitoring
- Blocklist management (disposable domains, blocked emails, blocked IPs)
- YouTube upload queue management
- User override management (whitelist/bypass permissions)
"""
import logging
from datetime import datetime
from datetime import datetime, timezone
from typing import List, Optional
from fastapi import APIRouter, Depends, HTTPException, Request
from pydantic import BaseModel
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, EmailStr
from backend.api.dependencies import require_admin
from backend.services.auth_service import AuthResult
from backend.services.rate_limit_service import get_rate_limit_service, RateLimitService
from backend.services.email_validation_service import get_email_validation_service, EmailValidationService
from backend.i18n import t, get_locale_from_request
from backend.config import settings
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/admin/rate-limits", tags=["admin", "rate-limits"])
@@ -26,40 +28,45 @@ router = APIRouter(prefix="/admin/rate-limits", tags=["admin", "rate-limits"])
# Response Models
# =============================================================================
class YouTubeQueueEntry(BaseModel):
"""A queued YouTube upload entry."""
job_id: str
status: str
reason: Optional[str] = None
user_email: Optional[str] = None
artist: Optional[str] = None
title: Optional[str] = None
brand_code: Optional[str] = None
queued_at: Optional[datetime] = None
attempts: int = 0
max_attempts: int = 5
last_error: Optional[str] = None
youtube_url: Optional[str] = None
notification_sent: bool = False
class RateLimitStatsResponse(BaseModel):
"""Current rate limit statistics."""
# Configuration
jobs_per_day_limit: int
youtube_uploads_per_day_limit: int
beta_ip_per_day_limit: int
rate_limiting_enabled: bool
# Current usage
youtube_uploads_today: int
youtube_uploads_remaining: int
# Blocklist stats
disposable_domains_count: int
blocked_emails_count: int
blocked_ips_count: int
# User override stats
total_overrides: int
class YouTubeQueueListResponse(BaseModel):
"""List of YouTube upload queue entries."""
entries: List[YouTubeQueueEntry]
stats: dict
class UserRateLimitStatusResponse(BaseModel):
"""Rate limit status for a specific user."""
email: str
jobs_today: int
jobs_limit: int
jobs_remaining: int
has_bypass: bool
custom_limit: Optional[int]
bypass_reason: Optional[str]
class BlocklistsResponse(BaseModel):
"""All blocklist data with domain source separation."""
external_domains: List[str]
manual_domains: List[str]
allowlisted_domains: List[str]
"""All blocklist data."""
disposable_domains: List[str]
blocked_emails: List[str]
blocked_ips: List[str]
last_sync_at: Optional[datetime] = None
last_sync_count: Optional[int] = None
updated_at: Optional[datetime] = None
updated_by: Optional[str] = None
updated_at: Optional[datetime]
updated_by: Optional[str]
class DomainRequest(BaseModel):
@@ -77,28 +84,159 @@ class IPRequest(BaseModel):
ip_address: str
class UserOverride(BaseModel):
"""User override configuration."""
email: str
bypass_job_limit: bool
custom_daily_job_limit: Optional[int]
reason: str
created_by: str
created_at: datetime
class UserOverrideRequest(BaseModel):
"""Request to set/update a user override."""
bypass_job_limit: bool = False
custom_daily_job_limit: Optional[int] = None
reason: str
class UserOverridesListResponse(BaseModel):
"""List of all user overrides."""
overrides: List[UserOverride]
total: int
class SuccessResponse(BaseModel):
"""Generic success response."""
success: bool
message: str
# =============================================================================
# Statistics Endpoints
# =============================================================================
@router.get("/stats", response_model=RateLimitStatsResponse)
async def get_rate_limit_stats(
auth_result: AuthResult = Depends(require_admin),
):
"""
Get current rate limit statistics.
Returns configuration values, current usage counts, and blocklist stats.
"""
rate_limit_service = get_rate_limit_service()
email_validation = get_email_validation_service()
# Get YouTube uploads today
youtube_today = rate_limit_service.get_youtube_uploads_today()
youtube_limit = settings.rate_limit_youtube_uploads_per_day
youtube_remaining = max(0, youtube_limit - youtube_today) if youtube_limit > 0 else -1
# Get blocklist stats
blocklist_stats = email_validation.get_blocklist_stats()
# Get override count
overrides = rate_limit_service.get_all_overrides()
return RateLimitStatsResponse(
jobs_per_day_limit=settings.rate_limit_jobs_per_day,
youtube_uploads_per_day_limit=settings.rate_limit_youtube_uploads_per_day,
beta_ip_per_day_limit=settings.rate_limit_beta_ip_per_day,
rate_limiting_enabled=settings.enable_rate_limiting,
youtube_uploads_today=youtube_today,
youtube_uploads_remaining=youtube_remaining,
disposable_domains_count=blocklist_stats["disposable_domains_count"],
blocked_emails_count=blocklist_stats["blocked_emails_count"],
blocked_ips_count=blocklist_stats["blocked_ips_count"],
total_overrides=len(overrides),
)
@router.get("/users/{email}", response_model=UserRateLimitStatusResponse)
async def get_user_rate_limit_status(
email: str,
auth_result: AuthResult = Depends(require_admin),
):
"""
Get rate limit status for a specific user.
Returns their current usage and any override settings.
"""
rate_limit_service = get_rate_limit_service()
jobs_today = rate_limit_service.get_user_job_count_today(email)
override = rate_limit_service.get_user_override(email)
if override:
has_bypass = override.get("bypass_job_limit", False)
custom_limit = override.get("custom_daily_job_limit")
bypass_reason = override.get("reason")
jobs_limit = custom_limit if custom_limit is not None else settings.rate_limit_jobs_per_day
else:
has_bypass = False
custom_limit = None
bypass_reason = None
jobs_limit = settings.rate_limit_jobs_per_day
jobs_remaining = max(0, jobs_limit - jobs_today) if not has_bypass else -1
return UserRateLimitStatusResponse(
email=email,
jobs_today=jobs_today,
jobs_limit=jobs_limit,
jobs_remaining=jobs_remaining,
has_bypass=has_bypass,
custom_limit=custom_limit,
bypass_reason=bypass_reason,
)
# =============================================================================
# Blocklist Management Endpoints
# =============================================================================
@router.get("/blocklists", response_model=BlocklistsResponse)
def get_blocklists(
async def get_blocklists(
auth_result: AuthResult = Depends(require_admin),
):
"""Get all blocklist data with domain source separation."""
"""
Get all blocklist data.
Returns disposable domains, blocked emails, and blocked IPs.
"""
email_validation = get_email_validation_service()
raw_data = email_validation.get_blocklist_raw_data()
return BlocklistsResponse(**raw_data)
# Force refresh to get latest data
config = email_validation.get_blocklist_config(force_refresh=True)
# Get the raw document for metadata
from google.cloud import firestore
from backend.services.firestore_service import get_firestore_client
from backend.services.email_validation_service import BLOCKLISTS_COLLECTION, BLOCKLIST_CONFIG_DOC
db = get_firestore_client()
doc = db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC).get()
updated_at = None
updated_by = None
if doc.exists:
data = doc.to_dict()
updated_at = data.get("updated_at")
updated_by = data.get("updated_by")
return BlocklistsResponse(
disposable_domains=sorted(list(config["disposable_domains"])),
blocked_emails=sorted(list(config["blocked_emails"])),
blocked_ips=sorted(list(config["blocked_ips"])),
updated_at=updated_at,
updated_by=updated_by,
)
@router.post("/blocklists/disposable-domains", response_model=SuccessResponse)
def add_disposable_domain(
async def add_disposable_domain(
request: DomainRequest,
auth_result: AuthResult = Depends(require_admin),
):
@@ -107,18 +245,18 @@ def add_disposable_domain(
domain = request.domain.lower().strip()
if not domain or "." not in domain:
raise HTTPException(status_code=400, detail=t("en", "admin.invalidDomainFormat"))
raise HTTPException(status_code=400, detail="Invalid domain format")
email_validation.add_disposable_domain(domain, auth_result.user_email)
return SuccessResponse(
success=True,
message=t("en", "admin.domainAddedToBl", domain=domain)
message=f"Domain '{domain}' added to disposable domains blocklist"
)
@router.delete("/blocklists/disposable-domains/{domain}", response_model=SuccessResponse)
def remove_disposable_domain(
async def remove_disposable_domain(
domain: str,
auth_result: AuthResult = Depends(require_admin),
):
@@ -127,67 +265,16 @@ def remove_disposable_domain(
domain = domain.lower().strip()
if not email_validation.remove_disposable_domain(domain, auth_result.user_email):
raise HTTPException(status_code=404, detail=t("en", "admin.domainNotFoundInBl"))
raise HTTPException(status_code=404, detail="Domain not found in blocklist")
return SuccessResponse(
success=True,
message=t("en", "admin.domainRemovedFromBl", domain=domain)
)
@router.post("/blocklists/allowlisted-domains", response_model=SuccessResponse)
def add_allowlisted_domain(
request: DomainRequest,
auth_result: AuthResult = Depends(require_admin),
):
"""Add a domain to the allowlist (overrides external blocklist)."""
email_validation = get_email_validation_service()
domain = request.domain.lower().strip()
if not domain or "." not in domain:
raise HTTPException(status_code=400, detail=t("en", "admin.invalidDomainFormat"))
email_validation.add_allowlisted_domain(domain, auth_result.user_email)
return SuccessResponse(success=True, message=t("en", "admin.domainAddedToAllowlist", domain=domain))
@router.delete("/blocklists/allowlisted-domains/{domain}", response_model=SuccessResponse)
def remove_allowlisted_domain(
domain: str,
auth_result: AuthResult = Depends(require_admin),
):
"""Remove a domain from the allowlist."""
email_validation = get_email_validation_service()
domain = domain.lower().strip()
if not email_validation.remove_allowlisted_domain(domain, auth_result.user_email):
raise HTTPException(status_code=404, detail=t("en", "admin.domainNotFoundInAllowlist"))
return SuccessResponse(success=True, message=t("en", "admin.domainRemovedFromAllowlist", domain=domain))
@router.post("/blocklists/sync", response_model=SuccessResponse)
async def trigger_sync(
auth_result: AuthResult = Depends(require_admin),
):
"""Manually trigger a sync of the external disposable domain blocklist."""
import asyncio
from backend.services.disposable_domain_sync_service import (
fetch_external_blocklist, sync_disposable_domains
)
from backend.services.firestore_service import get_firestore_client
domains = await fetch_external_blocklist()
db = get_firestore_client()
result = await asyncio.to_thread(sync_disposable_domains, db, domains)
# Invalidate cache
EmailValidationService._blocklist_cache = None
return SuccessResponse(
success=True,
message=t("en", "admin.syncComplete", external_count=result['external_count'], added=result['added'], removed=result['removed'])
message=f"Domain '{domain}' removed from disposable domains blocklist"
)
@router.post("/blocklists/blocked-emails", response_model=SuccessResponse)
def add_blocked_email(
async def add_blocked_email(
request: EmailRequest,
auth_result: AuthResult = Depends(require_admin),
):
@@ -196,18 +283,18 @@ def add_blocked_email(
email = request.email.lower().strip()
if not email or "@" not in email:
raise HTTPException(status_code=400, detail=t("en", "admin.invalidEmailFormat"))
raise HTTPException(status_code=400, detail="Invalid email format")
email_validation.add_blocked_email(email, auth_result.user_email)
return SuccessResponse(
success=True,
message=t("en", "admin.emailAddedToBl", email=email)
message=f"Email '{email}' added to blocked emails list"
)
@router.delete("/blocklists/blocked-emails/{email}", response_model=SuccessResponse)
def remove_blocked_email(
async def remove_blocked_email(
email: str,
auth_result: AuthResult = Depends(require_admin),
):
@@ -216,16 +303,16 @@ def remove_blocked_email(
email = email.lower().strip()
if not email_validation.remove_blocked_email(email, auth_result.user_email):
raise HTTPException(status_code=404, detail=t("en", "admin.emailNotFoundInBl"))
raise HTTPException(status_code=404, detail="Email not found in blocklist")
return SuccessResponse(
success=True,
message=t("en", "admin.emailRemovedFromBl", email=email)
message=f"Email '{email}' removed from blocked emails list"
)
@router.post("/blocklists/blocked-ips", response_model=SuccessResponse)
def add_blocked_ip(
async def add_blocked_ip(
request: IPRequest,
auth_result: AuthResult = Depends(require_admin),
):
@@ -234,18 +321,18 @@ def add_blocked_ip(
ip_address = request.ip_address.strip()
if not ip_address:
raise HTTPException(status_code=400, detail=t("en", "admin.invalidIPAddress"))
raise HTTPException(status_code=400, detail="Invalid IP address")
email_validation.add_blocked_ip(ip_address, auth_result.user_email)
return SuccessResponse(
success=True,
message=t("en", "admin.ipAddedToBl", ip_address=ip_address)
message=f"IP '{ip_address}' added to blocked IPs list"
)
@router.delete("/blocklists/blocked-ips/{ip_address}", response_model=SuccessResponse)
def remove_blocked_ip(
async def remove_blocked_ip(
ip_address: str,
auth_result: AuthResult = Depends(require_admin),
):
@@ -254,93 +341,88 @@ def remove_blocked_ip(
ip_address = ip_address.strip()
if not email_validation.remove_blocked_ip(ip_address, auth_result.user_email):
raise HTTPException(status_code=404, detail=t("en", "admin.ipNotFoundInBl"))
raise HTTPException(status_code=404, detail="IP not found in blocklist")
return SuccessResponse(
success=True,
message=t("en", "admin.ipRemovedFromBl", ip_address=ip_address)
message=f"IP '{ip_address}' removed from blocked IPs list"
)
# =============================================================================
# YouTube Upload Queue Management Endpoints
# User Override Management Endpoints
# =============================================================================
@router.get("/youtube-queue", response_model=YouTubeQueueListResponse)
def get_youtube_queue(
@router.get("/overrides", response_model=UserOverridesListResponse)
async def get_all_overrides(
auth_result: AuthResult = Depends(require_admin),
):
"""
Get YouTube upload queue entries and stats.
"""Get all user overrides."""
rate_limit_service = get_rate_limit_service()
Returns all queue entries (any status) and aggregate statistics.
"""
from backend.services.youtube_upload_queue_service import get_youtube_upload_queue_service
overrides_data = rate_limit_service.get_all_overrides()
queue_service = get_youtube_upload_queue_service()
entries = queue_service.get_all_queue_entries(limit=50)
stats = queue_service.get_queue_stats()
return YouTubeQueueListResponse(
entries=[YouTubeQueueEntry(**entry) for entry in entries],
stats=stats,
)
@router.post("/youtube-queue/{job_id}/retry", response_model=SuccessResponse)
def retry_youtube_upload(
job_id: str,
auth_result: AuthResult = Depends(require_admin),
):
"""
Manually retry a failed YouTube upload.
Resets the entry back to 'queued' status with 0 attempts.
"""
from backend.services.youtube_upload_queue_service import get_youtube_upload_queue_service
queue_service = get_youtube_upload_queue_service()
if not queue_service.retry_upload(job_id):
raise HTTPException(
status_code=404,
detail=t("en", "admin.youtubeUploadNotFound", job_id=job_id)
overrides = [
UserOverride(
email=email,
bypass_job_limit=data.get("bypass_job_limit", False),
custom_daily_job_limit=data.get("custom_daily_job_limit"),
reason=data.get("reason", ""),
created_by=data.get("created_by", "unknown"),
created_at=data.get("created_at", datetime.now(timezone.utc)),
)
for email, data in overrides_data.items()
]
return SuccessResponse(
success=True,
message=t("en", "admin.youtubeUploadQueuedForRetry", job_id=job_id)
return UserOverridesListResponse(
overrides=overrides,
total=len(overrides),
)
@router.post("/youtube-queue/process", response_model=SuccessResponse)
async def trigger_youtube_queue_processing(
@router.put("/overrides/{email}", response_model=SuccessResponse)
async def set_user_override(
email: str,
request: UserOverrideRequest,
auth_result: AuthResult = Depends(require_admin),
):
"""
Manually trigger YouTube upload queue processing.
"""Set or update a user override."""
rate_limit_service = get_rate_limit_service()
Same as the scheduled processor but triggered from the admin dashboard.
Runs in the background and returns immediately.
"""
from backend.workers.youtube_queue_processor import process_youtube_upload_queue
import asyncio
email = email.lower().strip()
if not email or "@" not in email:
raise HTTPException(status_code=400, detail="Invalid email format")
async def _process():
try:
result = await process_youtube_upload_queue()
logger.info(f"Manual YouTube queue processing complete: {result}")
except Exception as e:
logger.exception(f"Manual YouTube queue processing failed: {e}")
if not request.reason or len(request.reason.strip()) < 3:
raise HTTPException(status_code=400, detail="Reason is required (min 3 characters)")
# Fire and forget
try:
loop = asyncio.get_running_loop()
loop.create_task(_process())
except RuntimeError:
pass
rate_limit_service.set_user_override(
user_email=email,
bypass_job_limit=request.bypass_job_limit,
custom_daily_job_limit=request.custom_daily_job_limit,
reason=request.reason,
admin_email=auth_result.user_email,
)
return SuccessResponse(
success=True,
message=t("en", "admin.youtubeQueueProcessingStarted")
message=f"Override set for user '{email}'"
)
@router.delete("/overrides/{email}", response_model=SuccessResponse)
async def remove_user_override(
email: str,
auth_result: AuthResult = Depends(require_admin),
):
"""Remove a user override."""
rate_limit_service = get_rate_limit_service()
email = email.lower().strip()
if not rate_limit_service.remove_user_override(email, auth_result.user_email):
raise HTTPException(status_code=404, detail="Override not found for user")
return SuccessResponse(
success=True,
message=f"Override removed for user '{email}'"
)
-461
View File
@@ -1,461 +0,0 @@
"""
Referral API routes.
Handles:
- Public referral interstitial (click tracking)
- Authenticated referral dashboard
- Stripe Connect onboarding
- Admin vanity link creation and link management
"""
import logging
from typing import Optional
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi.responses import Response
from pydantic import BaseModel, Field
from backend.api.dependencies import require_auth, require_admin
from backend.models.referral import (
CreateReferralLinkRequest,
UpdateReferralLinkRequest,
ReferralInterstitialResponse,
VanityRequestResponse,
)
from backend.services.flyer_service import get_flyer_service, FlyerError
from backend.services.referral_service import get_referral_service
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/referrals", tags=["referrals"])
class GenerateFlyerRequest(BaseModel):
theme: str = Field(..., pattern="^(light|dark)$")
qr_data_url: str = Field(..., max_length=500_000)
# ============================================================================
# Public endpoints
# ============================================================================
@router.get("/r/{code}", response_model=ReferralInterstitialResponse)
def get_referral_interstitial(code: str, no_track: bool = False):
"""
Public endpoint: get interstitial data for a referral link.
Increments click count unless no_track=true. Returns valid=False if code not found.
"""
service = get_referral_service()
link = service.get_link_by_code(code)
if not link:
return ReferralInterstitialResponse(valid=False)
# Track the click (skip when fetching for display purposes)
if not no_track:
try:
service.increment_clicks(code)
except Exception:
logger.warning("Failed to increment clicks for code %s", code, exc_info=True)
return ReferralInterstitialResponse(
valid=True,
referral_code=link.code,
referrer_display_name=link.display_name,
discount_percent=link.discount_percent,
discount_duration_days=link.discount_duration_days,
custom_message=link.custom_message,
)
# ============================================================================
# Authenticated user endpoints
# ============================================================================
@router.get("/me")
def get_my_dashboard(auth=Depends(require_auth)):
"""Get referral dashboard data for the authenticated user."""
service = get_referral_service()
return service.get_dashboard_data(auth.user_email)
@router.put("/me")
def update_my_link(
updates: UpdateReferralLinkRequest,
auth=Depends(require_auth),
):
"""Update display_name and custom_message on the user's own referral link."""
service = get_referral_service()
link = service.get_or_create_link(auth.user_email)
update_fields = {}
if updates.display_name is not None:
update_fields["display_name"] = updates.display_name
if updates.custom_message is not None:
update_fields["custom_message"] = updates.custom_message
if not update_fields:
return {"ok": True, "message": "No changes"}
success, message = service.update_link(link.code, **update_fields)
if not success:
raise HTTPException(status_code=400, detail=message)
return {"ok": True, "message": message}
@router.post("/me/connect")
def start_stripe_connect(auth=Depends(require_auth)):
"""Start Stripe Connect onboarding for the authenticated user."""
# Check if user already has a Connect account
from backend.services.user_service import get_user_service
user_service = get_user_service()
user = user_service.get_user(auth.user_email)
if user and user.stripe_connect_account_id:
raise HTTPException(status_code=400, detail="Stripe Connect account already configured")
service = get_referral_service()
account_id, onboarding_url = service.create_connect_account(auth.user_email)
if not account_id:
raise HTTPException(status_code=500, detail="Failed to create Stripe Connect account")
user_service.update_user(auth.user_email, stripe_connect_account_id=account_id)
return {"account_id": account_id, "onboarding_url": onboarding_url}
@router.post("/me/connect/dashboard-link")
def get_connect_dashboard_link(auth=Depends(require_auth)):
"""Get a login link to the Stripe Express dashboard for managing payout settings."""
from backend.services.user_service import get_user_service
user = get_user_service().get_user(auth.user_email)
if not user or not user.stripe_connect_account_id:
raise HTTPException(status_code=400, detail="No Stripe Connect account configured")
service = get_referral_service()
url = service.create_connect_login_link(user.stripe_connect_account_id)
if not url:
raise HTTPException(status_code=500, detail="Failed to create dashboard link")
return {"url": url}
@router.post("/me/connect/update-link")
def get_connect_update_link(auth=Depends(require_auth)):
"""Get a link to update Stripe Connect account info (re-onboarding)."""
from backend.services.user_service import get_user_service
user = get_user_service().get_user(auth.user_email)
if not user or not user.stripe_connect_account_id:
raise HTTPException(status_code=400, detail="No Stripe Connect account configured")
service = get_referral_service()
url = service.create_connect_update_link(user.stripe_connect_account_id)
if not url:
raise HTTPException(status_code=500, detail="Failed to create update link")
return {"url": url}
class VanityRequest(BaseModel):
desired_code: str = Field(..., min_length=2, max_length=30, pattern="^[a-z0-9-]+$")
@router.post("/me/vanity-request")
def request_vanity_url(
request: VanityRequest,
auth=Depends(require_auth),
):
"""Request a custom vanity referral URL. Sends email to admin for review."""
from backend.services.email_service import get_email_service
service = get_referral_service()
# Validate up-front against the same rules approval enforces (reserved words,
# vanity pattern) so a request that would later be auto-rejected never lands
# in the admin queue. Do this before get_or_create_link so a rejected request
# from a brand-new user doesn't leave an orphaned auto-generated link behind.
valid, msg = service._validate_vanity_code(request.desired_code)
if not valid:
raise HTTPException(status_code=400, detail=msg)
# Check if code is already taken
existing = service.get_link_by_code(request.desired_code)
if existing:
raise HTTPException(status_code=409, detail="That code is already taken. Please try another.")
link = service.get_or_create_link(auth.user_email)
# Persist the request so it shows up as a pending item in the admin dashboard.
service.create_vanity_request(
owner_email=auth.user_email,
current_code=link.code,
desired_code=request.desired_code,
)
email_service = get_email_service()
email_service.provider.send_email(
to_email="andrew@nomadkaraoke.com",
subject=f"Vanity referral URL request: {request.desired_code}",
html_content=f"""
<h2>Vanity Referral URL Request</h2>
<p><strong>User:</strong> {auth.user_email}</p>
<p><strong>Current code:</strong> {link.code}</p>
<p><strong>Desired vanity code:</strong> {request.desired_code}</p>
<p>Approve or deny it under "Pending Vanity Requests" in the <a href="https://gen.nomadkaraoke.com/admin/referrals">admin dashboard</a>.</p>
""",
text_content=f"Vanity URL request from {auth.user_email}: {request.desired_code} (current: {link.code})",
)
logger.info("Vanity URL request from %s: %s", auth.user_email, request.desired_code)
return {"ok": True, "message": "Request sent"}
@router.post("/me/flyer")
def generate_flyer(
request: GenerateFlyerRequest,
auth=Depends(require_auth),
):
"""Generate a personalized referral flyer PDF."""
service = get_referral_service()
link = service.get_or_create_link(auth.user_email)
try:
flyer_service = get_flyer_service()
pdf_bytes = flyer_service.generate_pdf(
theme=request.theme,
referral_code=link.code,
discount_percent=link.discount_percent,
qr_data_url=request.qr_data_url,
)
except FlyerError as e:
raise HTTPException(status_code=400, detail=str(e))
return Response(
content=pdf_bytes,
media_type="application/pdf",
headers={
"Content-Disposition": 'attachment; filename="nomad-karaoke-referral-flyer.pdf"',
},
)
# ============================================================================
# Admin endpoints
# ============================================================================
@router.post("/admin/vanity")
def create_vanity_link(
request: CreateReferralLinkRequest,
auth=Depends(require_admin),
):
"""Admin: create a vanity referral link."""
if not request.vanity_code:
raise HTTPException(status_code=400, detail="vanity_code is required")
service = get_referral_service()
success, link, message = service.create_vanity_link(
code=request.vanity_code,
owner_email=request.owner_email,
display_name=request.display_name,
custom_message=request.custom_message,
discount_percent=request.discount_percent,
kickback_percent=request.kickback_percent,
discount_duration_days=request.discount_duration_days,
earning_duration_days=request.earning_duration_days,
)
if not success:
raise HTTPException(status_code=400, detail=message)
return {"ok": True, "code": link.code, "message": message}
@router.get("/admin/vanity-requests")
def list_vanity_requests(
status: str = Query("pending", pattern="^(pending|approved|denied)$"),
auth=Depends(require_admin),
):
"""Admin: list vanity code requests (pending by default)."""
service = get_referral_service()
requests = service.list_vanity_requests(status=status)
return {
"requests": [
VanityRequestResponse(**r.model_dump()).model_dump(mode="json")
for r in requests
],
"count": len(requests),
}
@router.post("/admin/vanity-requests/{request_id}/approve")
def approve_vanity_request(
request_id: str,
auth=Depends(require_admin),
):
"""Admin: approve a vanity request by renaming the owner's link, then notify them."""
from backend.services.email_service import get_email_service
service = get_referral_service()
req = service.get_vanity_request(request_id)
if not req:
raise HTTPException(status_code=404, detail="Request not found")
success, link, message = service.approve_vanity_request(
request_id, resolved_by=auth.user_email
)
if not success:
raise HTTPException(status_code=400, detail=message)
# Notify the requester that their vanity URL is live.
try:
referral_url = f"https://nomadkaraoke.com/r/{link.code}"
get_email_service().provider.send_email(
to_email=req.owner_email,
subject="Your custom referral link is live 🎉",
html_content=f"""
<h2>Your custom referral link is ready</h2>
<p>Good news — your requested referral code <strong>{link.code}</strong> is now active.</p>
<p>Share it anywhere: <a href="{referral_url}">{referral_url}</a></p>
<p>All your existing clicks, signups, and earnings carried over automatically.</p>
""",
text_content=f"Your custom referral link is live: {referral_url}",
)
except Exception:
logger.warning("Failed to send vanity-approved email to %s", req.owner_email, exc_info=True)
return {"ok": True, "code": link.code, "message": message}
@router.post("/admin/vanity-requests/{request_id}/deny")
def deny_vanity_request(
request_id: str,
auth=Depends(require_admin),
):
"""Admin: deny a vanity request."""
service = get_referral_service()
success, message = service.deny_vanity_request(request_id, resolved_by=auth.user_email)
if not success:
raise HTTPException(status_code=400, detail=message)
return {"ok": True, "message": message}
@router.get("/admin/links")
def list_links(
limit: int = Query(50, ge=1, le=200),
offset: int = Query(0, ge=0),
auth=Depends(require_admin),
):
"""Admin: list all referral links."""
service = get_referral_service()
links = service.list_links(limit=limit, offset=offset)
return {
"links": [link.model_dump(mode="json") for link in links],
"count": len(links),
}
@router.post("/admin/apply-discount")
async def admin_apply_discount(
request: Request,
auth=Depends(require_admin),
):
"""Admin: apply a referral discount to any user, even without a referral link."""
from backend.services.user_service import get_user_service
body = await request.json()
user_email = body.get("email", "").lower()
discount_percent = body.get("discount_percent", 10)
duration_days = body.get("duration_days", 30)
referral_code = body.get("referral_code", "admin-grant")
if not user_email:
raise HTTPException(status_code=400, detail="Email is required")
user_service = get_user_service()
user = user_service.get_user(user_email)
if not user:
raise HTTPException(status_code=404, detail="User not found")
from datetime import datetime, timedelta
now = datetime.utcnow()
user_service.update_user(
user_email,
referred_by_code=referral_code,
referred_at=now,
referral_discount_expires_at=now + timedelta(days=duration_days),
)
# Create/ensure the Stripe coupon exists for this discount percent
service = get_referral_service()
service.get_or_create_stripe_coupon(discount_percent)
logger.info(f"Admin applied {discount_percent}% discount for {duration_days} days to {user_email}")
return {
"ok": True,
"message": f"Applied {discount_percent}% discount for {duration_days} days",
"discount_expires_at": (now + timedelta(days=duration_days)).isoformat(),
}
@router.put("/admin/links/{code}")
def update_link(
code: str,
updates: UpdateReferralLinkRequest,
auth=Depends(require_admin),
):
"""Admin: update any referral link's settings."""
service = get_referral_service()
update_fields = {
k: v for k, v in updates.model_dump().items() if v is not None
}
if not update_fields:
return {"ok": True, "message": "No changes"}
success, message = service.update_link(code, **update_fields)
if not success:
raise HTTPException(status_code=404, detail=message)
return {"ok": True, "message": message}
@router.post("/admin/links/{code}/flyer")
def admin_generate_flyer(
code: str,
request: GenerateFlyerRequest,
auth=Depends(require_admin),
):
"""Admin: generate a personalized referral flyer PDF for any code."""
service = get_referral_service()
link = service.get_link_by_code(code)
if not link:
raise HTTPException(status_code=404, detail=f"Referral link '{code}' not found")
try:
flyer_service = get_flyer_service()
pdf_bytes = flyer_service.generate_pdf(
theme=request.theme,
referral_code=link.code,
discount_percent=link.discount_percent,
qr_data_url=request.qr_data_url,
)
except FlyerError as e:
raise HTTPException(status_code=400, detail=str(e))
return Response(
content=pdf_bytes,
media_type="application/pdf",
headers={
"Content-Disposition": f'attachment; filename="nomad-karaoke-flyer-{link.code}.pdf"',
},
)
-208
View File
@@ -1,208 +0,0 @@
"""
Public song-request voting board API (requests.nomadkaraoke.com).
Anyone signed in with an email magic link can submit a song request (auto-corrected
artist/title) and cast one vote per calendar day. The board itself (GET /requests) is
readable without auth so it can be linked from YouTube descriptions and browsed freely.
Phase 2 (daily auto-picker, free-credit grant, ownership handoff, voter emails) is not
here — see docs/archive/2026-09-02-requests-voting-board-plan.md.
"""
import logging
from typing import Optional
from fastapi import APIRouter, Depends, HTTPException, Request
from fastapi.security import HTTPAuthorizationCredentials
from backend.api.dependencies import require_auth, security, get_token_from_request
from backend.services.auth_service import AuthResult, get_auth_service
from backend.services.match_judge.classifier import normalize_for_match
from backend.models.song_request import (
BoardResponse,
DailyVoteStatus,
SongRequest,
SongRequestPublic,
SubmitRequestBody,
SubmitResponse,
VoteBody,
)
from backend.services.song_request_service import (
InvalidSubmitter,
RequestNotFound,
SubmissionRateLimited,
VoteAlreadyUsed,
WouldDeleteRequest,
get_song_request_service,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/requests-board", tags=["requests-board"])
async def optional_user_email(
request: Request,
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
) -> Optional[str]:
"""Resolve the caller's email if a valid token is present, else None.
Unlike require_auth this never raises — the board is publicly viewable, we just
annotate the viewer's own vote when they happen to be signed in.
"""
token = await get_token_from_request(request, credentials, None)
if not token:
return None
try:
result = get_auth_service().validate_token_full(token)
return result.user_email if result.is_valid else None
except Exception:
return None
def _was_corrected(req: SongRequest) -> bool:
return (
normalize_for_match(req.artist_raw) != normalize_for_match(req.artist)
or normalize_for_match(req.title_raw) != normalize_for_match(req.title)
)
def _to_public(req: SongRequest, your_vote: Optional[int] = None) -> SongRequestPublic:
return SongRequestPublic(
id=req.id,
artist=req.artist,
title=req.title,
status=req.status,
source=req.source,
vote_count=req.vote_count,
created_at=req.created_at.isoformat(),
youtube_url=req.youtube_url,
was_corrected=_was_corrected(req),
your_vote=your_vote,
)
@router.get("/requests", response_model=BoardResponse)
def list_requests(
viewer_email: Optional[str] = Depends(optional_user_email),
):
"""The ranked board: active requests + recently published, with the caller's vote."""
service = get_song_request_service()
active = service.list_active()
published = service.list_published()
daily_vote = service.get_daily_vote(viewer_email) if viewer_email else None
voted_request_id = daily_vote.request_id if daily_vote else None
voted_value = daily_vote.value if daily_vote else None
return BoardResponse(
requests=[
_to_public(r, your_vote=voted_value if r.id == voted_request_id else None)
for r in active
],
published=[_to_public(r) for r in published],
voted_today=bool(daily_vote) if viewer_email is not None else None,
your_vote_request_id=voted_request_id,
)
@router.post("/requests", response_model=SubmitResponse)
async def submit_request(
body: SubmitRequestBody,
auth_result: AuthResult = Depends(require_auth),
):
"""Submit a song request (requires sign-in). Auto-corrects artist/title and dedupes."""
if not auth_result.user_email:
raise HTTPException(status_code=403, detail="A signed-in email is required to submit.")
service = get_song_request_service()
try:
request, already_existed, canonical_artist, canonical_title = await service.submit_request(
auth_result.user_email, body.artist, body.title
)
except SubmissionRateLimited:
raise HTTPException(
status_code=429,
detail={
"code": "submission_limit",
"message": "You can request one song per day — please come back tomorrow.",
},
)
except VoteAlreadyUsed:
raise HTTPException(
status_code=409,
detail={
"code": "vote_already_used",
"message": "Requesting a song uses your one vote for today, and you've already "
"voted on another request. Remove that vote first, or come back tomorrow.",
},
)
except InvalidSubmitter:
raise HTTPException(status_code=403, detail="A signed-in email is required to submit.")
daily = service.get_daily_vote(auth_result.user_email)
your_vote = daily.value if daily and daily.request_id == request.id else None
return SubmitResponse(
status="already_exists" if already_existed else "created",
request=_to_public(request, your_vote=your_vote),
canonical_artist=canonical_artist,
canonical_title=canonical_title,
was_corrected=_was_corrected(request),
)
@router.post("/requests/{request_id}/vote", response_model=SongRequestPublic)
def vote(
request_id: str,
body: VoteBody,
auth_result: AuthResult = Depends(require_auth),
):
"""Cast/move/undo your single daily vote on a request."""
if not auth_result.user_email:
raise HTTPException(status_code=403, detail="A signed-in email is required to vote.")
service = get_song_request_service()
try:
service.cast_vote(
auth_result.user_email, request_id, body.direction, confirm_delete=body.confirm_delete
)
except RequestNotFound:
raise HTTPException(status_code=404, detail="That request no longer exists.")
except WouldDeleteRequest as exc:
raise HTTPException(
status_code=409,
detail={
"code": "would_delete_request",
# The request that would be deleted (not necessarily the one being voted on:
# moving your only up-vote away deletes the OLD request).
"request_id": exc.request_id,
"message": "Removing your vote will delete this request from the board "
"(nobody else is backing it). Continue?",
},
)
except InvalidSubmitter:
raise HTTPException(status_code=403, detail="A signed-in email is required to vote.")
updated = service.get_request(request_id)
if not updated:
# Its last up-vote was withdrawn (confirmed), so the request was deleted.
return SongRequestPublic(
id=request_id, artist="", title="", status="open", source="human",
vote_count=0, created_at="", request_deleted=True,
)
daily = service.get_daily_vote(auth_result.user_email)
your_vote = daily.value if daily and daily.request_id == request_id else None
return _to_public(updated, your_vote=your_vote)
@router.get("/me", response_model=DailyVoteStatus)
def my_daily_status(
auth_result: AuthResult = Depends(require_auth),
):
"""The caller's daily-vote status (has today's vote been used, and on what)."""
service = get_song_request_service()
daily = service.get_daily_vote(auth_result.user_email) if auth_result.user_email else None
return DailyVoteStatus(
voted_today=bool(daily),
request_id=daily.request_id if daily else None,
value=daily.value if daily else None,
)
+324 -3018
View File
File diff suppressed because it is too large. Load diff
+4 -5
View File
@@ -12,7 +12,6 @@ from fastapi import APIRouter, Query, Request
from backend.models.tenant import TenantConfigResponse, TenantPublicConfig
from backend.services.tenant_service import get_tenant_service
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
@@ -20,7 +19,7 @@ router = APIRouter(prefix="/api/tenant", tags=["tenant"])
@router.get("/config", response_model=TenantConfigResponse)
def get_tenant_config(
async def get_tenant_config(
request: Request,
tenant: Optional[str] = Query(
None,
@@ -81,7 +80,7 @@ def get_tenant_config(
@router.get("/config/{tenant_id}", response_model=TenantConfigResponse)
def get_tenant_config_by_id(tenant_id: str):
async def get_tenant_config_by_id(tenant_id: str):
"""
Get tenant configuration by explicit tenant ID.
@@ -102,7 +101,7 @@ def get_tenant_config_by_id(tenant_id: str):
@router.get("/asset/{tenant_id}/{asset_name}")
def get_tenant_asset(tenant_id: str, asset_name: str):
async def get_tenant_asset(tenant_id: str, asset_name: str):
"""
Get a signed URL for a tenant asset (logo, favicon, etc.).
@@ -116,6 +115,6 @@ def get_tenant_asset(tenant_id: str, asset_name: str):
if not url:
from fastapi import HTTPException
raise HTTPException(status_code=404, detail=t("en", "tenant.assetNotFound"))
raise HTTPException(status_code=404, detail="Asset not found")
return RedirectResponse(url=url)
-375
View File
@@ -1,375 +0,0 @@
"""
Admin API routes for provisioning white-label tenants.
These endpoints let an admin mint a tenant (branded portal + locked theme) from
the UI, replacing the hand-run ``scripts/setup-*-tenant.py`` recipe. Once created,
the tenant can be driven immediately via the admin preview path
``?preview_tenant=<id>`` — including the tenant bulk-upload flow (folder of tracks,
auto-paired mixed/instrumental, resumable uploads, locked theme on every job).
"""
import json
import logging
from typing import Any, Dict, List, Optional, Tuple
from fastapi import APIRouter, Depends, File, Form, HTTPException, UploadFile
from fastapi.concurrency import run_in_threadpool
from pydantic import BaseModel, ValidationError
from backend.api.dependencies import require_admin
from backend.models.tenant import TenantPublicConfig
from backend.models.theme import ColorOverrides
from backend.services.auth_service import AuthResult
from backend.services.tenant_admin_service import (
IMAGE_CONTENT_TYPES,
TenantConflictError,
TenantNotFoundError,
TenantProvisioningError,
TenantValidationError,
create_tenant,
delete_tenant,
get_default_style_params,
get_tenant_detail,
get_tenant_domain_status,
list_tenants,
provision_tenant_domain,
update_tenant,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/admin/tenants", tags=["admin", "tenant"])
# Public app origin used to build the admin-preview link.
APP_ORIGIN = "https://gen.nomadkaraoke.com"
# Reject oversized image uploads (backgrounds/logo).
MAX_IMAGE_BYTES = 15 * 1024 * 1024
class TenantSummary(BaseModel):
id: Optional[str] = None
name: Optional[str] = None
subdomain: Optional[str] = None
is_active: bool = True
locked_theme: Optional[str] = None
theme_id: Optional[str] = None
dropbox_path: Optional[str] = None
brand_prefix: Optional[str] = None
created_at: Optional[str] = None
class TenantListResponse(BaseModel):
tenants: List[TenantSummary]
class TenantCreateResponse(BaseModel):
tenant: TenantPublicConfig
preview_url: str
subdomain_url: str
# Cloudflare state of the portal subdomain: {hostname, state, dns_ok, pages_status}.
# state is "active" once the certificate is issued (usually within ~1 min).
domain: Optional[Dict[str, Any]] = None
class TenantDetailResponse(BaseModel):
tenant: Dict[str, Any]
theme_id: str
style_params: Dict[str, Any]
assets: List[str]
preview_url: str
domain: Optional[Dict[str, Any]] = None
class TenantDomainResponse(BaseModel):
domain: Dict[str, Any]
class TenantTemplateResponse(BaseModel):
style_params: Dict[str, Any]
class TenantUpdateResponse(BaseModel):
tenant: TenantPublicConfig
def _clean(value: Optional[str]) -> Optional[str]:
"""Normalize empty/whitespace form values to None."""
if value is None:
return None
value = value.strip()
return value or None
def _preview_url(tenant_id: str) -> str:
return f"{APP_ORIGIN}/en/app?preview_tenant={tenant_id}"
def _parse_json_field(raw: Optional[str], label: str) -> Optional[Any]:
"""Parse an optional JSON string form field, mapping errors to 400."""
if raw is None or not raw.strip():
return None
try:
return json.loads(raw)
except (ValueError, TypeError) as exc:
raise HTTPException(status_code=400, detail=f"{label}: invalid JSON ({exc}).")
def _parse_domains(raw: Optional[str]) -> List[str]:
if not raw:
return []
parts = raw.replace("\n", ",").replace(" ", ",").split(",")
return [p.strip().lower() for p in parts if p.strip()]
async def _read_image(upload: Optional[UploadFile], label: str) -> Optional[Tuple[bytes, str]]:
"""Read+validate an uploaded image, returning (bytes, ext) or None."""
if upload is None or not upload.filename:
return None
ext = upload.filename.rsplit(".", 1)[-1].lower() if "." in upload.filename else ""
if ext not in IMAGE_CONTENT_TYPES:
raise HTTPException(
status_code=400,
detail=f"{label}: unsupported image type '.{ext}'. Use PNG, JPG, GIF, or WEBP.",
)
data = await upload.read()
if not data:
return None
if len(data) > MAX_IMAGE_BYTES:
raise HTTPException(
status_code=400,
detail=f"{label}: image is too large ({len(data) // (1024 * 1024)}MB). Max 15MB.",
)
return data, ext
# Assets replaceable in the manage flow: images plus fonts.
ASSET_EXTENSIONS = set(IMAGE_CONTENT_TYPES) | {"ttf", "otf", "woff", "woff2"}
async def _read_asset(upload: Optional[UploadFile], label: str) -> Optional[Tuple[bytes, str]]:
"""Read+validate an uploaded theme asset (image or font)."""
if upload is None or not upload.filename:
return None
ext = upload.filename.rsplit(".", 1)[-1].lower() if "." in upload.filename else ""
if ext not in ASSET_EXTENSIONS:
raise HTTPException(
status_code=400,
detail=f"{label}: unsupported asset type '.{ext}'. Use an image or font file.",
)
data = await upload.read()
if not data:
return None
if len(data) > MAX_IMAGE_BYTES:
raise HTTPException(
status_code=400,
detail=f"{label}: file is too large ({len(data) // (1024 * 1024)}MB). Max 15MB.",
)
return data, ext
@router.get("", response_model=TenantListResponse)
def admin_list_tenants(auth_data: AuthResult = Depends(require_admin)):
"""List all white-label tenants."""
return TenantListResponse(tenants=[TenantSummary(**t) for t in list_tenants()])
@router.get("/_template", response_model=TenantTemplateResponse)
def admin_tenant_theme_template(auth_data: AuthResult = Depends(require_admin)):
"""Return the default Nomad theme's full style_params as an editing starting point."""
try:
return TenantTemplateResponse(style_params=get_default_style_params())
except ValueError as exc:
raise HTTPException(status_code=500, detail=str(exc))
# Sync handlers: FastAPI runs them in its threadpool, so the blocking GCS and
# Cloudflare calls don't stall the event loop (the admin UI polls this).
@router.get("/{tenant_id}", response_model=TenantDetailResponse)
def admin_get_tenant(tenant_id: str, auth_data: AuthResult = Depends(require_admin)):
"""Return a tenant's full config, theme style_params, and asset list for editing."""
try:
detail = get_tenant_detail(tenant_id)
except TenantNotFoundError as exc:
raise HTTPException(status_code=404, detail=str(exc))
return TenantDetailResponse(
preview_url=_preview_url(tenant_id),
domain=get_tenant_domain_status(tenant_id),
**detail,
)
@router.post("/{tenant_id}/domain", response_model=TenantDomainResponse)
def admin_provision_tenant_domain(tenant_id: str, auth_data: AuthResult = Depends(require_admin)):
"""(Re)provision the tenant's {id}.nomadkaraoke.com subdomain (idempotent)."""
try:
domain = provision_tenant_domain(tenant_id)
except TenantNotFoundError as exc:
raise HTTPException(status_code=404, detail=str(exc))
except TenantConflictError as exc:
raise HTTPException(status_code=409, detail=str(exc))
except TenantProvisioningError as exc:
raise HTTPException(status_code=502, detail=str(exc))
logger.info(f"Admin {auth_data.user_email or 'admin:unknown'} provisioned domain for tenant '{tenant_id}'")
return TenantDomainResponse(domain=domain)
@router.delete("/{tenant_id}", status_code=204)
def admin_delete_tenant(tenant_id: str, auth_data: AuthResult = Depends(require_admin)):
"""Delete a tenant: subdomain (Cloudflare), theme, and config. Jobs are kept."""
try:
delete_tenant(tenant_id)
except TenantNotFoundError as exc:
raise HTTPException(status_code=404, detail=str(exc))
except TenantProvisioningError as exc:
raise HTTPException(status_code=502, detail=str(exc))
logger.info(f"Admin {auth_data.user_email or 'admin:unknown'} deleted tenant '{tenant_id}'")
@router.put("/{tenant_id}", response_model=TenantUpdateResponse)
async def admin_update_tenant(
tenant_id: str,
auth_data: AuthResult = Depends(require_admin),
config: Optional[str] = Form(None),
style_params: Optional[str] = Form(None),
logo: Optional[UploadFile] = File(None),
assets: Optional[List[UploadFile]] = File(None),
):
"""
Update a tenant: merge config fields (JSON), replace the full theme
style_params (JSON), and/or add/replace theme assets (uploaded files keyed
by their filename). Re-render jobs to apply an updated theme.
"""
config_updates = _parse_json_field(config, "config")
if config_updates is not None and not isinstance(config_updates, dict):
raise HTTPException(status_code=400, detail="config must be a JSON object.")
style = _parse_json_field(style_params, "style_params")
asset_map: Dict[str, Tuple[bytes, str]] = {}
for upload in assets or []:
asset = await _read_asset(upload, f"asset '{upload.filename}'")
if asset and upload.filename:
asset_map[upload.filename] = asset
logo_image = await _read_image(logo, "logo")
try:
updated = await run_in_threadpool(
update_tenant,
tenant_id,
config_updates=config_updates,
style_params=style,
assets=asset_map or None,
logo=logo_image,
)
except TenantNotFoundError as exc:
raise HTTPException(status_code=404, detail=str(exc))
except TenantConflictError as exc:
raise HTTPException(status_code=409, detail=str(exc))
except TenantValidationError as exc:
raise HTTPException(status_code=400, detail=str(exc))
except TenantProvisioningError as exc:
raise HTTPException(status_code=502, detail=str(exc))
except ValidationError as exc:
raise HTTPException(status_code=400, detail=f"Invalid config: {exc}")
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc))
except Exception as exc: # pragma: no cover - defensive
logger.exception("Failed to update tenant")
raise HTTPException(status_code=500, detail=f"Failed to update tenant: {exc}")
admin_id = auth_data.user_email or "admin:unknown"
logger.info(f"Admin {admin_id} updated tenant '{tenant_id}'")
return TenantUpdateResponse(tenant=TenantPublicConfig.from_config(updated))
@router.post("", response_model=TenantCreateResponse, status_code=201)
async def admin_create_tenant(
auth_data: AuthResult = Depends(require_admin),
name: str = Form(...),
tenant_id: Optional[str] = Form(None),
subdomain: Optional[str] = Form(None),
allowed_email_domains: Optional[str] = Form(None),
allowed_emails: Optional[str] = Form(None),
artist_color: Optional[str] = Form(None),
title_color: Optional[str] = Form(None),
sung_lyrics_color: Optional[str] = Form(None),
unsung_lyrics_color: Optional[str] = Form(None),
tagline: Optional[str] = Form(None),
distribution_mode: str = Form("download_only"),
dropbox_path: Optional[str] = Form(None),
brand_prefix: Optional[str] = Form(None),
style_params: Optional[str] = Form(None),
karaoke_background: Optional[UploadFile] = File(None),
intro_background: Optional[UploadFile] = File(None),
end_background: Optional[UploadFile] = File(None),
logo: Optional[UploadFile] = File(None),
):
"""Provision a new white-label tenant (theme + config) from admin form input."""
try:
colors = ColorOverrides(
artist_color=_clean(artist_color),
title_color=_clean(title_color),
sung_lyrics_color=_clean(sung_lyrics_color),
unsung_lyrics_color=_clean(unsung_lyrics_color),
)
except ValidationError:
raise HTTPException(
status_code=400, detail="Colors must be hex values like #ff5bb8."
)
backgrounds = {}
for field, upload in (
("karaoke_background", karaoke_background),
("intro_background", intro_background),
("end_background", end_background),
):
image = await _read_image(upload, field)
if image:
backgrounds[field] = image
logo_image = await _read_image(logo, "logo")
if distribution_mode not in ("all", "download_only", "cloud_only"):
raise HTTPException(status_code=400, detail="Invalid distribution_mode.")
style_override = _parse_json_field(style_params, "style_params")
if style_override is not None and not isinstance(style_override, dict):
raise HTTPException(status_code=400, detail="style_params must be a JSON object.")
try:
config = await run_in_threadpool(
create_tenant,
name=name,
tenant_id=_clean(tenant_id),
subdomain=_clean(subdomain),
allowed_email_domains=_parse_domains(allowed_email_domains),
allowed_emails=_parse_domains(allowed_emails),
colors=colors,
style_params_override=style_override,
tagline=_clean(tagline),
distribution_mode=distribution_mode,
dropbox_path=_clean(dropbox_path),
brand_prefix=_clean(brand_prefix),
backgrounds=backgrounds,
logo=logo_image,
)
except TenantConflictError as exc:
raise HTTPException(status_code=409, detail=str(exc))
except TenantValidationError as exc:
raise HTTPException(status_code=400, detail=str(exc))
except TenantProvisioningError as exc:
raise HTTPException(status_code=502, detail=str(exc))
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc))
except Exception as exc: # pragma: no cover - defensive
logger.exception("Failed to create tenant")
raise HTTPException(status_code=500, detail=f"Failed to create tenant: {exc}")
admin_id = auth_data.user_email or "admin:unknown"
logger.info(f"Admin {admin_id} created tenant '{config.id}'")
return TenantCreateResponse(
tenant=TenantPublicConfig.from_config(config),
preview_url=_preview_url(config.id),
subdomain_url=config.get_frontend_url(),
domain=await run_in_threadpool(get_tenant_domain_status, config.id),
)
-89
View File
@@ -1,89 +0,0 @@
"""Authenticated tenant bulk-upload endpoints.
Separate from ``tenant.py`` (which is public config only): these require a
tenant session and gate on the ``bulk_upload`` feature flag.
"""
import asyncio
import logging
from typing import List
from fastapi import APIRouter, Depends, HTTPException, Request
from pydantic import BaseModel, Field
from backend.api.dependencies import require_auth
from backend.middleware.tenant import get_tenant_config_from_request
from backend.services.auth_service import AuthResult
from pathlib import Path
from backend.services.tenant_bulk import analyze_filenames
from backend.services.tenant_bulk.analyze import AUDIO_EXTENSIONS, default_generate
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/tenant/bulk", tags=["tenant-bulk"])
# Mirror consumer Bulk Mode's cap to bound LLM cost and upload load. Counts
# audio files (candidate tracks), not incidental non-audio files a folder pick
# may include (cover art, etc.), so an image-heavy folder isn't falsely blocked.
MAX_AUDIO_FILES = 100
# Absolute payload guard so a single request can't send an unbounded list.
MAX_FILENAMES = 2000
class BulkAnalyzeRequest(BaseModel):
filenames: List[str] = Field(
..., description="Filenames from the operator's folder pick (no paths needed)"
)
@router.post("/analyze")
async def analyze_bulk_filenames(
request: Request,
body: BulkAnalyzeRequest,
auth_result: AuthResult = Depends(require_auth),
):
"""Analyse a list of filenames into proposed Mixed/Instrumental job rows.
Pure analysis: filenames only, no uploads, no audio, no state written.
Returns ``{rows, unpaired, ignored}`` for an editable review table.
"""
tenant_config = get_tenant_config_from_request(request)
if not tenant_config:
raise HTTPException(
status_code=403, detail="Bulk upload is only available on tenant portals"
)
if not getattr(tenant_config.features, "bulk_upload", False):
raise HTTPException(
status_code=403, detail="Bulk upload is not enabled for this portal"
)
filenames = [f for f in (body.filenames or []) if f and f.strip()]
if not filenames:
raise HTTPException(status_code=400, detail="No filenames provided")
if len(filenames) > MAX_FILENAMES:
raise HTTPException(status_code=400, detail="Too many files in request")
audio_count = sum(
1 for f in filenames if Path(f).suffix.lower() in AUDIO_EXTENSIONS
)
if audio_count > MAX_AUDIO_FILES:
raise HTTPException(
status_code=400,
detail=f"Too many tracks ({audio_count}). Please submit at most {MAX_AUDIO_FILES} at a time.",
)
# The LLM pass is a blocking Vertex call; run it off the event loop. It only
# fires for files the regex could not pair, and analyze_filenames falls back
# to the regex result if the model errors — so this never hard-fails.
def _run() -> dict:
return analyze_filenames(filenames, generate=default_generate).to_dict()
analysis = await asyncio.to_thread(_run)
logger.info(
"tenant-bulk analyze: tenant=%s files=%d rows=%d unpaired=%d ignored=%d",
tenant_config.id,
len(filenames),
len(analysis["rows"]),
len(analysis["unpaired"]),
len(analysis["ignored"]),
)
return analysis
-298
View File
@@ -1,298 +0,0 @@
"""
Tenant self-service theme editor API (``/api/tenant/theme``).
Any user allowed on a tenant's portal can view, preview and save that tenant's
theme. Tenant context comes from the request (subdomain or the client-controlled
``X-Tenant-ID`` header), so every endpoint re-checks membership against the
tenant's allowlist (admins always pass) — a header alone grants nothing.
"""
import asyncio
import logging
from concurrent.futures import ThreadPoolExecutor
from typing import Any, Dict, List, Literal, Optional
from fastapi import APIRouter, Depends, File, Form, HTTPException, Request, UploadFile
from fastapi.concurrency import run_in_threadpool
from pydantic import BaseModel, Field, constr
from backend.api.dependencies import require_auth
from backend.middleware.tenant import get_tenant_config_from_request
from backend.models.tenant import TenantConfig
from backend.services.auth_service import AuthResult
from backend.services.tenant_admin_service import TenantValidationError
from backend.services.tenant_theme_service import (
ThemeNotEditableError,
ThemeNotFoundError,
get_theme_for_editor,
prepare_preview_styles,
save_tenant_theme,
store_uploaded_asset,
)
from backend.services.theme_change_service import outdated_jobs, refresh_inflight_jobs
from backend.services.theme_preview_service import ThemePreviewError, render_theme_preview
from backend.services.tenant_admin_service import _theme_id_for
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/tenant/theme", tags=["tenant-theme"])
MAX_UPLOAD_BYTES = 15 * 1024 * 1024
# Previews run ffmpeg/PIL on the API instance — bound concurrency like the review
# preview path so a burst of edits can't starve request handling.
_PREVIEW_SEMAPHORE = asyncio.Semaphore(2)
_PREVIEW_EXECUTOR = ThreadPoolExecutor(max_workers=2, thread_name_prefix="theme-preview")
PREVIEW_TIMEOUT_S = 45
def require_tenant_member(
request: Request, auth_result: AuthResult = Depends(require_auth)
) -> TenantConfig:
"""The request's tenant, iff the authenticated user is allowed on it."""
config = get_tenant_config_from_request(request)
if config is None or not config.is_active:
raise HTTPException(status_code=404, detail="Theme editing is only available on a tenant portal.")
if auth_result.tenant_id and auth_result.tenant_id != config.id:
raise HTTPException(status_code=403, detail="Your sign-in belongs to a different portal.")
if not auth_result.user_email or not config.is_email_allowed(auth_result.user_email):
raise HTTPException(status_code=403, detail="You don't have access to this portal's theme.")
return config
def _raise_http(exc: TenantValidationError) -> None:
if isinstance(exc, ThemeNotFoundError):
raise HTTPException(status_code=404, detail=str(exc))
if isinstance(exc, ThemeNotEditableError):
raise HTTPException(status_code=403, detail=str(exc))
raise HTTPException(status_code=400, detail=str(exc))
class ThemeResponse(BaseModel):
theme_id: str
style_params: Dict[str, Any]
images: List[str]
fonts: List[str]
# The images that fit the 300x216 CD+G screen (offered on the CD+G tab).
cdg_images: List[str] = Field(default_factory=list)
# Built-in CD+G reveal transitions (greyscale draw-order masks).
cdg_transitions: List[str] = Field(default_factory=list)
class PreviewSample(BaseModel):
artist: str = Field("Artist Name", max_length=80)
title: str = Field("Song Title", max_length=80)
lyrics: Optional[List[constr(max_length=120)]] = Field(None, max_length=4)
class PreviewRequest(BaseModel):
style_params: Dict[str, Any]
sample: PreviewSample = Field(default_factory=PreviewSample)
# Also render the CD+G up card + a lyrics page (slower; the editor asks on its CD+G tab).
include_cdg: bool = False
# MP4 screens to render (default all): the editor asks only for the tab's screen.
kinds: Optional[List[Literal["title_card", "karaoke_frame", "end_screen"]]] = None
class PreviewResponse(BaseModel):
# data:image/jpeg;base64,... — null for a screen the theme leaves out of videos.
title_card: Optional[str] = None
karaoke_frame: Optional[str] = None
end_screen: Optional[str] = None
# data:image/png;base64,... — only when include_cdg was requested.
cdg_title_card: Optional[str] = None
cdg_lyrics: Optional[str] = None
cdg_instrumental: Optional[str] = None
cdg_outro: Optional[str] = None
class SaveRequest(BaseModel):
style_params: Dict[str, Any]
class AssetResponse(BaseModel):
name: str
@router.get("", response_model=ThemeResponse)
def get_theme(config: TenantConfig = Depends(require_tenant_member)):
try:
return ThemeResponse(**get_theme_for_editor(config))
except TenantValidationError as exc:
_raise_http(exc)
@router.post("/assets", response_model=AssetResponse)
async def upload_asset(
file: UploadFile = File(...),
purpose: Optional[str] = Form(None),
config: TenantConfig = Depends(require_tenant_member),
):
if purpose not in (None, "", "cdg"):
raise HTTPException(status_code=400, detail="Unknown upload purpose.")
# Read in chunks and stop as soon as the limit is exceeded, so an oversized
# upload never gets materialised in memory in full.
chunks, size = [], 0
while True:
chunk = await file.read(1024 * 1024)
if not chunk:
break
size += len(chunk)
if size > MAX_UPLOAD_BYTES:
raise HTTPException(status_code=400, detail="The file is too large (max 15 MB).")
chunks.append(chunk)
data = b"".join(chunks)
if not data:
raise HTTPException(status_code=400, detail="The file is empty.")
try:
name = await run_in_threadpool(
lambda: store_uploaded_asset(config, file.filename or "asset", data, purpose=purpose or None)
)
except TenantValidationError as exc:
_raise_http(exc)
return AssetResponse(name=name)
@router.post("/preview", response_model=PreviewResponse)
async def preview_theme(body: PreviewRequest, config: TenantConfig = Depends(require_tenant_member)):
try:
styles = await run_in_threadpool(prepare_preview_styles, config, body.style_params)
except TenantValidationError as exc:
_raise_http(exc)
loop = asyncio.get_running_loop()
await _PREVIEW_SEMAPHORE.acquire()
future = loop.run_in_executor(
_PREVIEW_EXECUTOR,
lambda: render_theme_preview(
_theme_id_for(config),
styles,
artist=body.sample.artist,
title=body.sample.title,
lyrics=body.sample.lyrics,
include_cdg=body.include_cdg,
kinds=body.kinds,
),
)
# The slot is released only when the render really finishes — a timed-out
# render keeps running in its thread, so releasing early would let a burst
# queue unbounded work behind it.
future.add_done_callback(lambda _f: _PREVIEW_SEMAPHORE.release())
try:
images = await asyncio.wait_for(asyncio.shield(future), timeout=PREVIEW_TIMEOUT_S)
except ThemePreviewError as exc:
raise HTTPException(status_code=422, detail=str(exc))
except asyncio.TimeoutError:
raise HTTPException(status_code=504, detail="The preview took too long to render. Try again.")
return PreviewResponse(**images.as_data_urls())
class SaveResponse(ThemeResponse):
# In-progress tracks switched to the new theme by this save.
refreshed_jobs: int = 0
# The caller's finished tracks made with an older theme (re-render to update).
outdated_job_ids: List[str] = Field(default_factory=list)
class OutdatedJobsResponse(BaseModel):
theme_updated_at: Optional[str] = None
job_ids: List[str]
class RerenderOutdatedResponse(BaseModel):
started: List[str]
failed: Dict[str, str]
# Outdated tracks beyond this call's cap (ask again to start them).
remaining: List[str] = Field(default_factory=list)
# Tracks started per bulk call: bounds the request time and the burst of
# render/encode work it queues. The UI offers the remainder again.
MAX_BULK_RERENDER = 20
def _owner_scope(auth_result: AuthResult) -> Optional[str]:
"""Every caller here is a portal member (require_tenant_member), and tenant
portals are shared workspaces (backend/services/job_access.py), so theme
refresh/rerender covers all of the portal's tracks, whoever submitted them."""
return None
@router.put("", response_model=SaveResponse)
def save_theme(
body: SaveRequest,
config: TenantConfig = Depends(require_tenant_member),
auth_result: AuthResult = Depends(require_auth),
):
try:
save_tenant_theme(config, body.style_params)
theme = get_theme_for_editor(config)
except TenantValidationError as exc:
_raise_http(exc)
theme_id = _theme_id_for(config)
# Never fail the save over the follow-up work: the theme is already stored.
refreshed = 0
try:
refreshed = refresh_inflight_jobs(config.id, theme_id)["updated"]
except Exception:
logger.exception(f"Tenant '{config.id}': refreshing in-progress tracks after theme save failed")
outdated: List[str] = []
try:
outdated = outdated_jobs(config.id, theme_id, _owner_scope(auth_result))["job_ids"]
except Exception:
logger.exception(f"Tenant '{config.id}': listing outdated tracks after theme save failed")
return SaveResponse(**theme, refreshed_jobs=refreshed, outdated_job_ids=outdated)
@router.get("/outdated-jobs", response_model=OutdatedJobsResponse)
def get_outdated_jobs(
config: TenantConfig = Depends(require_tenant_member),
auth_result: AuthResult = Depends(require_auth),
):
"""Finished tracks made with an older version of the theme (re-render to update)."""
return OutdatedJobsResponse(**outdated_jobs(config.id, _theme_id_for(config), _owner_scope(auth_result)))
@router.post("/rerender-outdated", response_model=RerenderOutdatedResponse)
async def rerender_outdated(
config: TenantConfig = Depends(require_tenant_member),
auth_result: AuthResult = Depends(require_auth),
):
"""Re-render every outdated finished track with the current theme.
Quiet: no per-track "your video is ready" email/push (one per track would
flood the inbox); the portal shows progress on each track.
"""
from backend.services.job_manager import JobManager
from backend.services.theme_rerender_service import RerenderError, ThemeRerenderService
theme_id = _theme_id_for(config)
job_ids = (await run_in_threadpool(
outdated_jobs, config.id, theme_id, _owner_scope(auth_result)
))["job_ids"]
batch, remaining = job_ids[:MAX_BULK_RERENDER], job_ids[MAX_BULK_RERENDER:]
job_manager = JobManager()
service = ThemeRerenderService(job_manager)
started: List[str] = []
failed: Dict[str, str] = {}
for job_id in batch:
job = await run_in_threadpool(job_manager.get_job, job_id)
if job is None:
continue
try:
await service.start(
job, theme_id=theme_id,
requested_by=auth_result.user_email or "unknown",
notify_customer=False,
)
started.append(job_id)
except RerenderError as e:
failed[job_id] = str(e)
except Exception as e:
logger.exception(f"[job:{job_id}] Bulk theme re-render failed to start")
failed[job_id] = "Couldn't start the re-render."
logger.info(
f"Tenant '{config.id}': bulk re-render started {len(started)}, failed {len(failed)}, "
f"remaining {len(remaining)}"
)
return RerenderOutdatedResponse(started=started, failed=failed, remaining=remaining)
-315
View File
@@ -1,315 +0,0 @@
"""Test-only API for the local E2E stack (``/api/test/*``).
Mounted ONLY when FAKE_WORKERS is on (``backend/main.py``), which startup
refuses in production, and unauthenticated by design: the stack runs on
emulators on a developer machine or CI runner. Web Playwright and mobile Maestro
tests use it to reset state, seed scenarios, tune the fake workers, read
captured email and complete fake Stripe checkouts.
"""
from __future__ import annotations
import html
import logging
import os
from typing import Any, Dict, Optional
import httpx
from fastapi import APIRouter, HTTPException
from fastapi.responses import HTMLResponse, RedirectResponse
from pydantic import BaseModel
from backend.config import get_settings
from backend.services import fake_workers
from backend.services.fake_workers import checkout as fake_checkout
from backend.services.fake_workers.config import clear_fake_config, get_fake_config, set_fake_config
from backend.services.fake_workers.outbox import clear_pushes, list_emails, list_pushes
from backend.services.fake_workers.scenarios import SCENARIOS, ScenarioError, run_scenario
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/test", tags=["test-support"])
# Seeded by scripts/seed-local-data.py; kept across resets.
PRESERVED_BLOB_PREFIXES = ("themes/",)
class ScenarioRequest(BaseModel):
name: str
email: Optional[str] = None
credits: Optional[int] = None
artist: Optional[str] = None
title: Optional[str] = None
@router.get("/status")
def status() -> Dict[str, Any]:
return {
"fake_workers": True,
"active_workers": fake_workers.active_worker_count(),
"config": get_fake_config().to_dict(),
"scenarios": SCENARIOS,
}
@router.post("/reset")
def reset() -> Dict[str, Any]:
"""Wipe every Firestore document and every bucket object except the seed (themes)."""
emulator = os.getenv("FIRESTORE_EMULATOR_HOST")
if not emulator:
raise HTTPException(status_code=409, detail="reset only runs against the Firestore emulator")
project = get_settings().google_cloud_project
resp = httpx.delete(
f"http://{emulator}/emulator/v1/projects/{project}/databases/(default)/documents", timeout=30,
)
if resp.status_code >= 300:
raise HTTPException(status_code=502, detail=f"emulator reset failed: {resp.status_code} {resp.text}")
from backend.services.storage_service import StorageService
bucket = StorageService().bucket
deleted = 0
for blob in list(bucket.list_blobs()):
if not blob.name.startswith(PRESERVED_BLOB_PREFIXES):
blob.delete()
deleted += 1
clear_fake_config()
clear_pushes()
return {"status": "reset", "blobs_deleted": deleted}
@router.get("/fake-config")
def read_config(email: Optional[str] = None) -> Dict[str, Any]:
return get_fake_config(email=email).to_dict()
@router.put("/fake-config")
def update_config(updates: Dict[str, Any], email: Optional[str] = None) -> Dict[str, Any]:
"""Merge ``updates`` into the global knobs, or into ``?email=``'s per-user overrides."""
try:
return set_fake_config(updates, email=email).to_dict()
except (ValueError, TypeError) as e:
raise HTTPException(status_code=422, detail=str(e))
@router.post("/scenario")
def scenario(body: ScenarioRequest) -> Dict[str, Any]:
try:
return run_scenario(body.name, email=body.email, credits=body.credits, artist=body.artist, title=body.title)
except ScenarioError as e:
raise HTTPException(status_code=422, detail=str(e))
@router.get("/jobs/{job_id}/review-result")
def review_result(job_id: str) -> Dict[str, Any]:
"""What the review submitted: the saved corrected lines plus the instrumental / duet choice.
Lets Maestro assert on a review without going through the review auth.
``lines`` is empty until a review was completed (corrections_updated.json exists).
"""
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
job = JobManager().get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="unknown job")
state = job.state_data or {}
path = ((job.file_urls or {}).get("lyrics") or {}).get("corrections_updated") \
or f"jobs/{job_id}/lyrics/corrections_updated.json"
storage = StorageService()
lines = []
if storage.file_exists(path):
for seg in storage.download_json(path).get("corrected_segments") or []:
lines.append({
"text": seg.get("text"),
"singer": seg.get("singer"),
"words": [
{"text": w.get("text"), "start_time": w.get("start_time"), "end_time": w.get("end_time")}
for w in seg.get("words") or []
],
})
return {
"status": getattr(job.status, "value", job.status),
"instrumental_selection": state.get("instrumental_selection"),
"is_duet": bool(state.get("is_duet", False)),
"lines": lines,
}
@router.get("/emails")
def emails(to: Optional[str] = None, limit: int = 20) -> Dict[str, Any]:
return {"emails": list_emails(to, limit=limit)}
@router.get("/push-outbox")
def push_outbox(to: Optional[str] = None, limit: int = 50) -> Dict[str, Any]:
"""Expo pushes captured instead of sent (newest first)."""
return {"pushes": list_pushes(to, limit=limit)}
@router.delete("/push-outbox")
def clear_push_outbox() -> Dict[str, Any]:
clear_pushes()
return {"status": "cleared"}
# ------------------------------------------------------------------ fake Stripe
_CHECKOUT_PAGE = """<!doctype html>
<html><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>Fake Checkout</title>
<style>body{{font-family:system-ui,sans-serif;max-width:420px;margin:48px auto;padding:0 16px}}
button{{font-size:18px;padding:12px 20px;margin:8px 8px 0 0;border-radius:8px;border:0}}
#pay{{background:#635bff;color:#fff}}#cancel{{background:#eee}}</style></head>
<body><h1>Fake Checkout</h1>
<p data-testid="fake-checkout-product">{product}</p>
<p data-testid="fake-checkout-amount">${amount}</p>
<p>{email}</p>
<form method="post" action="{base}/api/test/fake-checkout/{sid}/pay" style="display:inline">
<button id="pay" data-testid="fake-checkout-pay" type="submit">Pay</button></form>
<form method="post" action="{base}/api/test/fake-checkout/{sid}/cancel" style="display:inline">
<button id="cancel" data-testid="fake-checkout-cancel" type="submit">Cancel</button></form>
</body></html>"""
def _checkout_or_404(session_id: str) -> Dict[str, Any]:
record = fake_checkout.get_checkout(session_id)
if not record:
raise HTTPException(status_code=404, detail="unknown fake checkout session")
return record
@router.get("/fake-checkout/{session_id}", response_class=HTMLResponse)
def checkout_page(session_id: str) -> HTMLResponse:
record = _checkout_or_404(session_id)
return HTMLResponse(_CHECKOUT_PAGE.format(
product=html.escape(record.get("product_name") or ""),
amount=f"{(record.get('amount_cents') or 0) / 100:.2f}",
email=html.escape(record.get("customer_email") or ""),
base=fake_checkout.public_api_url(),
sid=html.escape(session_id),
))
@router.post("/fake-checkout/{session_id}/pay")
async def checkout_pay(session_id: str):
"""Complete the session through the same handler as the real Stripe webhook."""
from backend.api.routes.internal import TestWebhookRequest, test_webhook
record = _checkout_or_404(session_id)
result = await test_webhook(
TestWebhookRequest(
event_type="checkout.session.completed",
session_id=session_id,
customer_email=record.get("customer_email") or "",
metadata=record.get("metadata") or {},
),
auth_data=None,
)
if result.status not in ("processed", "already_processed"):
raise HTTPException(status_code=500, detail=f"fake checkout completion failed: {result.message}")
fake_checkout.mark_checkout(session_id, "complete")
return RedirectResponse(fake_checkout.resolve_url(record.get("success_url"), session_id), status_code=303)
@router.post("/fake-checkout/{session_id}/cancel")
def checkout_cancel(session_id: str):
record = _checkout_or_404(session_id)
fake_checkout.mark_checkout(session_id, "cancelled")
return RedirectResponse(fake_checkout.resolve_url(record.get("cancel_url"), session_id), status_code=303)
# ------------------------------------------------------------------ fake in-app purchases
class FakeIapPurchaseRequest(BaseModel):
"""A store purchase, as RevenueCat would report it. Give ``email`` or ``app_user_id``."""
product_id: str
email: Optional[str] = None
app_user_id: Optional[str] = None
transaction_id: Optional[str] = None
store: str = "APP_STORE"
price: Optional[float] = None
takehome_percentage: Optional[float] = None
country_code: str = "US"
class FakeIapRefundRequest(BaseModel):
"""Refunds debit the account the purchase granted, so no user is needed."""
transaction_id: str
product_id: str
store: str = "APP_STORE"
def _fake_app_user_id(email: Optional[str], app_user_id: Optional[str]) -> str:
from backend.services.user_service import get_user_service
if app_user_id:
return app_user_id
if not email:
raise HTTPException(status_code=422, detail="email or app_user_id is required")
user_service = get_user_service()
user_service.get_or_create_user(email)
return user_service.ensure_app_user_id(email)
def _fake_iap_result(event: Dict[str, Any]) -> Dict[str, Any]:
from backend.services.iap_service import get_iap_service
result = get_iap_service().handle_event(event)
return {
"action": result.action,
"credits": result.credits,
"store": event["store"],
"transaction_id": event["transaction_id"],
}
# US store prices of the credit packs (spec: $9.99 / $23.99 / $34.99 / $59.99).
_FAKE_IAP_PRICES = {
"nomad_credits_1": 9.99,
"nomad_credits_3": 23.99,
"nomad_credits_5": 34.99,
"nomad_credits_10": 59.99,
}
@router.post("/fake-iap/purchase")
def fake_iap_purchase(body: FakeIapPurchaseRequest) -> Dict[str, Any]:
"""Grant a store credit pack through the same IapService as the RevenueCat webhook."""
import uuid
price = body.price if body.price is not None else _FAKE_IAP_PRICES.get(body.product_id, 0.0)
event: Dict[str, Any] = {
"type": "NON_RENEWING_PURCHASE",
"app_user_id": _fake_app_user_id(body.email, body.app_user_id),
"product_id": body.product_id,
"transaction_id": body.transaction_id or f"fake-{uuid.uuid4().hex}",
"store": body.store,
"price": price,
"price_in_purchased_currency": price,
"currency": "USD",
"country_code": body.country_code,
"environment": "SANDBOX",
}
if body.takehome_percentage is not None:
event["takehome_percentage"] = body.takehome_percentage
return _fake_iap_result(event)
@router.post("/fake-iap/refund")
def fake_iap_refund(body: FakeIapRefundRequest) -> Dict[str, Any]:
"""Refund a fake store purchase (RevenueCat CANCELLATION / CUSTOMER_SUPPORT)."""
price = -_FAKE_IAP_PRICES.get(body.product_id, 0.0)
event: Dict[str, Any] = {
"type": "CANCELLATION",
"cancel_reason": "CUSTOMER_SUPPORT",
"app_user_id": "",
"product_id": body.product_id,
"transaction_id": body.transaction_id,
"store": body.store,
"price": price,
"price_in_purchased_currency": price,
"currency": "USD",
"environment": "SANDBOX",
}
return _fake_iap_result(event)
+13 -48
View File
@@ -6,15 +6,13 @@ These endpoints are public (no authentication required) to allow theme selection
before job creation.
"""
import asyncio
import logging
from typing import Optional
from fastapi import APIRouter, HTTPException, Request
from fastapi import APIRouter, HTTPException
from backend.models.theme import ThemeDetailResponse, ThemesListResponse
from backend.services.theme_service import get_theme_service
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
@@ -22,7 +20,7 @@ router = APIRouter(prefix="/themes", tags=["themes"])
@router.get("", response_model=ThemesListResponse)
def list_themes(request: Request) -> ThemesListResponse:
async def list_themes() -> ThemesListResponse:
"""
List all available themes.
@@ -31,7 +29,6 @@ def list_themes(request: Request) -> ThemesListResponse:
This endpoint is public and does not require authentication.
"""
locale = get_locale_from_request(request)
try:
theme_service = get_theme_service()
themes = theme_service.list_themes()
@@ -40,41 +37,12 @@ def list_themes(request: Request) -> ThemesListResponse:
logger.error(f"Error listing themes: {e}")
raise HTTPException(
status_code=500,
detail=t(locale, "themes.themesLoadError")
detail="Failed to load themes"
)
@router.get("/translation-preview")
async def get_translation_preview(language: str, request: Request) -> dict:
"""
Preview frame of a translated-lyrics karaoke video in ``language``.
Renders the default theme's karaoke frame with sample lyrics and their
(pre-generated) translation beneath each line, so users can see the option
before enabling it at job creation. Public, like the other theme endpoints.
Returns:
Object with ``image`` (JPEG data URL)
Raises:
400: Unsupported language
"""
from backend.services.lyrics_translation import normalize_language
from backend.services.translation_preview import render_translation_preview
code = normalize_language(language)
if code is None:
raise HTTPException(status_code=400, detail=f"Unsupported language: {language}")
try:
image = await asyncio.to_thread(render_translation_preview, code)
except Exception as e:
logger.error(f"Error rendering translation preview for {code}: {e}", exc_info=True)
raise HTTPException(status_code=500, detail="Could not render the translation preview")
return {"image": image}
@router.get("/{theme_id}", response_model=ThemeDetailResponse)
def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
async def get_theme(theme_id: str) -> ThemeDetailResponse:
"""
Get detailed information about a specific theme.
@@ -90,7 +58,6 @@ def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
Raises:
404: Theme not found
"""
locale = get_locale_from_request(request)
try:
theme_service = get_theme_service()
theme = theme_service.get_theme(theme_id)
@@ -98,7 +65,7 @@ def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
if theme is None:
raise HTTPException(
status_code=404,
detail=t(locale, "themes.themeNotFound", theme_id=theme_id)
detail=f"Theme not found: {theme_id}"
)
return ThemeDetailResponse(theme=theme)
@@ -108,12 +75,12 @@ def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
logger.error(f"Error getting theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail=t(locale, "themes.themeLoadError")
detail="Failed to load theme"
)
@router.get("/{theme_id}/preview")
def get_theme_preview(theme_id: str, request: Request) -> dict:
async def get_theme_preview(theme_id: str) -> dict:
"""
Get a signed URL for the theme's preview image.
@@ -129,21 +96,20 @@ def get_theme_preview(theme_id: str, request: Request) -> dict:
Raises:
404: Theme not found or no preview available
"""
locale = get_locale_from_request(request)
try:
theme_service = get_theme_service()
if not theme_service.theme_exists(theme_id):
raise HTTPException(
status_code=404,
detail=t(locale, "themes.themeNotFound", theme_id=theme_id)
detail=f"Theme not found: {theme_id}"
)
theme = theme_service.get_theme(theme_id)
if theme is None or theme.preview_url is None:
raise HTTPException(
status_code=404,
detail=t(locale, "themes.previewNotAvailable", theme_id=theme_id)
detail=f"No preview available for theme: {theme_id}"
)
return {"preview_url": theme.preview_url}
@@ -153,12 +119,12 @@ def get_theme_preview(theme_id: str, request: Request) -> dict:
logger.error(f"Error getting preview for theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail=t(locale, "themes.previewLoadError")
detail="Failed to load theme preview"
)
@router.get("/{theme_id}/youtube-description")
def get_theme_youtube_description(theme_id: str, request: Request) -> dict:
async def get_theme_youtube_description(theme_id: str) -> dict:
"""
Get the YouTube description template for a theme.
@@ -175,14 +141,13 @@ def get_theme_youtube_description(theme_id: str, request: Request) -> dict:
Raises:
404: Theme not found
"""
locale = get_locale_from_request(request)
try:
theme_service = get_theme_service()
if not theme_service.theme_exists(theme_id):
raise HTTPException(
status_code=404,
detail=t(locale, "themes.youtubeDescriptionNotFound", theme_id=theme_id)
detail=f"Theme not found: {theme_id}"
)
description = theme_service.get_youtube_description(theme_id)
@@ -193,5 +158,5 @@ def get_theme_youtube_description(theme_id: str, request: Request) -> dict:
logger.error(f"Error getting YouTube description for theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail=t(locale, "themes.youtubeDescriptionLoadError")
detail="Failed to load YouTube description"
)
+575 -1782
View File
File diff suppressed because it is too large. Load diff
-35
View File
@@ -1,35 +0,0 @@
# Cloud Build config for GPU base image
#
# Builds the CUDA + PyTorch + models base image used by the audio worker.
# This image is large (~10GB) so builds take 15-30 minutes.
#
# Trigger manually:
# gcloud builds submit --config=backend/cloudbuild-gpu-base.yaml --project=nomadkaraoke --timeout=3600s
#
# The app-layer Dockerfile uses this as its BASE_IMAGE for audio worker deploys.
steps:
- name: 'gcr.io/cloud-builders/docker'
args:
- 'build'
- '-f'
- 'backend/Dockerfile.gpu-base'
- '-t'
- 'us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-gpu-base:latest'
- '-t'
- 'us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-gpu-base:$BUILD_ID'
- '--build-arg'
- 'BUILD_DATE=$_BUILD_DATE'
- '--build-arg'
- 'BUILD_ID=$BUILD_ID'
- '.'
images:
- 'us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-gpu-base:latest'
- 'us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-gpu-base:$BUILD_ID'
timeout: '3600s'
options:
machineType: 'E2_HIGHCPU_32'
diskSizeGb: 100
+18 -493
View File
@@ -4,17 +4,9 @@ Configuration management for the karaoke generation backend.
import os
import logging
from typing import Optional, Dict
from dotenv import load_dotenv
from pydantic_settings import BaseSettings
from google.cloud import secretmanager
# google.cloud.secretmanager is imported lazily in get_secret() — importing it
# (and the google.api_core/grpc graph behind it) costs ~1.7s and config is
# imported by every backend entrypoint (cold-start work, 2026-09-22).
# Load .env for local development (no-op when absent, e.g. in Cloud Run).
# Must run before the Settings class body below, which reads os.getenv at
# class-definition time. Real environment variables take precedence.
load_dotenv()
logger = logging.getLogger(__name__)
@@ -28,27 +20,7 @@ class Settings(BaseSettings):
gcs_temp_bucket: str = os.getenv("GCS_TEMP_BUCKET", "karaoke-gen-temp")
gcs_output_bucket: str = os.getenv("GCS_OUTPUT_BUCKET", "karaoke-gen-outputs")
firestore_collection: str = os.getenv("FIRESTORE_COLLECTION", "jobs")
# Nomad master fast-sync: push freshly-published Nomad 720p masters straight to the
# Divebar GCS mirror so kjbox (which mirrors that folder every 5 min) picks them up
# within minutes instead of waiting for the nightly Drive->GCS VM. Nomad-brand only.
divebar_files_bucket: str = os.getenv("DIVEBAR_FILES_BUCKET", "nomadkaraoke-divebar-files")
nomad_master_gcs_prefix: str = os.getenv("NOMAD_MASTER_GCS_PREFIX", "files/Nomad Karaoke/MP4-720p")
nomad_master_fast_sync_enabled: bool = os.getenv("NOMAD_MASTER_FAST_SYNC_ENABLED", "true").lower() == "true"
# Original-vocals guide: a padded isolated-vocals file (silence[intro] + vocals) pushed to
# a sibling prefix in the same Divebar bucket, which kjbox mirrors into NOMAD-vocals-padded/
# to power the "Original Vocals" sing-along slider. Same brand gating as the master push.
nomad_vocals_guide_gcs_prefix: str = os.getenv("NOMAD_VOCALS_GUIDE_GCS_PREFIX", "files/Nomad Karaoke/vocals-padded")
# KaraokeNerds community catalog: our own daily-refreshed copy of KN's community
# (web-only, directly-playable) tracks, written by the authorized `kn-data-sync`
# export job. The community-version check reads THIS, never scrapes karaokenerds.com.
# Source of truth is BigQuery `karaoke_decide.karaokenerds_community`; the export job
# also mirrors it to this gzipped-JSON GCS object, which the backend loads in-process.
kn_community_bucket: str = os.getenv("KN_COMMUNITY_BUCKET", "nomadkaraoke-kn-data")
kn_community_blob: str = os.getenv("KN_COMMUNITY_BLOB", "community/community-data-latest.json.gz")
kn_community_ttl_seconds: int = int(os.getenv("KN_COMMUNITY_TTL_SECONDS", "21600")) # 6h
# Audio Separator API (for GPU processing)
audio_separator_api_url: Optional[str] = os.getenv("AUDIO_SEPARATOR_API_URL")
@@ -60,17 +32,9 @@ class Settings(BaseSettings):
# Authentication
admin_tokens: Optional[str] = os.getenv("ADMIN_TOKENS") # Comma-separated list
scheduler_service_account: str = os.getenv(
"SCHEDULER_SERVICE_ACCOUNT",
"karaoke-backend@nomadkaraoke.iam.gserviceaccount.com"
)
# Application
environment: str = os.getenv("ENVIRONMENT", "development")
# Event-loop stall watchdog (backend/services/loop_watchdog.py): logs the
# blocking stack when the single uvicorn loop stops ticking for >1s, and
# records stalls >=5s to Firestore client_events (production only).
loop_watchdog_enabled: bool = os.getenv("LOOP_WATCHDOG_ENABLED", "true").lower() in ("true", "1", "yes")
log_level: str = os.getenv("LOG_LEVEL", "INFO")
# Processing
@@ -86,176 +50,24 @@ class Settings(BaseSettings):
# Agentic AI Correction (for lyrics correction via LLM)
# Only used when skip_correction=false
# When enabled, uses Gemini (Developer API, key GEMINI_API_KEY) for intelligent lyrics correction
# When enabled, uses Gemini via Vertex AI for intelligent lyrics correction
use_agentic_ai: bool = os.getenv("USE_AGENTIC_AI", "true").lower() in ("true", "1", "yes")
agentic_ai_model: str = os.getenv("AGENTIC_AI_MODEL", "gemini/gemini-3.8-flash")
agentic_ai_model: str = os.getenv("AGENTIC_AI_MODEL", "vertexai/gemini-3-flash-preview")
# Timeout for agentic correction in seconds. If correction takes longer, abort and
# use uncorrected transcription - human review will fix any issues.
agentic_correction_timeout_seconds: int = int(os.getenv("AGENTIC_CORRECTION_TIMEOUT_SECONDS", "180"))
# Lyrics transcription outer (safety-net) timeout, scaled by audio duration.
# Long inputs (e.g. 30+ minute mashups) legitimately take longer to transcribe,
# so the timeout = clamp(floor, duration * per_audio_second, cap). The cap must
# stay under the Cloud Run Job task timeout so we surface a clean RuntimeError
# before the platform hard-kills the task. The lyrics-transcription-job sets the
# task timeout to 3000s and overrides this cap to 2700s via env; the 1700s
# default here is the safe value if that env is ever unset (under an 1800s task).
transcription_timeout_floor_seconds: int = int(os.getenv("TRANSCRIPTION_TIMEOUT_FLOOR_SECONDS", "1200"))
transcription_timeout_cap_seconds: int = int(os.getenv("TRANSCRIPTION_TIMEOUT_CAP_SECONDS", "1700"))
transcription_timeout_per_audio_second: float = float(os.getenv("TRANSCRIPTION_TIMEOUT_PER_AUDIO_SECOND", "1.0"))
# AI Credit Evaluation (anti-abuse gating for free credits)
# When enabled, welcome and feedback credit grants are evaluated by Gemini
# before granting. Fail-CLOSED: if evaluation fails, the decision becomes
# "pending_review" (no auto-grant; an admin is notified to decide).
credit_eval_enabled: bool = os.getenv("CREDIT_EVAL_ENABLED", "true").lower() in ("true", "1", "yes")
credit_eval_model: str = os.getenv("CREDIT_EVAL_MODEL", "gemini-3.8-flash")
# Custom Lyrics (LLM-powered) — used by the Custom Lyrics mode in the
# Edit All Lyrics modal. Stateless service: takes operator-provided
# custom-lyrics text/file and returns transformed lyric lines.
custom_lyrics_model: str = os.getenv("CUSTOM_LYRICS_MODEL", "gemini-3.8-flash")
custom_lyrics_max_file_mb: int = int(os.getenv("CUSTOM_LYRICS_MAX_FILE_MB", "5"))
custom_lyrics_max_input_lines: int = int(os.getenv("CUSTOM_LYRICS_MAX_INPUT_LINES", "500"))
custom_lyrics_max_iterations: int = int(os.getenv("CUSTOM_LYRICS_MAX_ITERATIONS", "4"))
custom_lyrics_default_strictness: str = os.getenv("CUSTOM_LYRICS_DEFAULT_STRICTNESS", "balanced")
custom_lyrics_max_output_lines_multiplier: float = float(os.getenv("CUSTOM_LYRICS_MAX_OUTPUT_LINES_MULTIPLIER", "2.0"))
# Translated lyrics — per-job opt-in (Job.translation_language): the reviewed
# lyrics are translated line-by-line just before the karaoke video renders.
lyrics_translation_model: str = os.getenv("LYRICS_TRANSLATION_MODEL", "gemini-3.8-flash")
# AI auto-correct suggestions — opt-in, user-triggered from the lyrics
# review UI. Stateless: one whole-song LLM call returning word-level
# suggestions the reviewer accepts/rejects individually.
auto_correct_model: str = os.getenv("AUTO_CORRECT_MODEL", "gemini-3.8-flash")
# Models used by the multi-model "compare" mode (semicolon-separated —
# Cloud Run --set-env-vars is comma-delimited). Empty = single-model only.
auto_correct_compare_models: str = os.getenv("AUTO_CORRECT_COMPARE_MODELS", "")
# Proactively generate (and cache) auto-correct suggestions once
# transcription + references are ready, so the review UI has them instantly.
# This flag gates whether the lyrics worker fires the trigger; the actual
# multi-model LLM call + caching runs on the API service (auto_correct_worker),
# which holds the model credentials — the lyrics-transcription-job itself does
# NOT need ANTHROPIC_API_KEY / AUTO_CORRECT_COMPARE_MODELS. Best-effort:
# failures never block or fail the job. Default off.
auto_correct_proactive_enabled: bool = os.getenv(
"AUTO_CORRECT_PROACTIVE_ENABLED", "false"
).lower() in ("true", "1", "yes")
# Auto-approval ENFORCEMENT: when the auto-approval scorer is fully confident
# (see backend/services/auto_approval/), skip the combined review screens and
# go straight to render. Scoring + verdict recording happen on every job
# regardless of this flag; it only gates actually skipping the human.
# Per-job opt-out via Job.review_mode="always_review". Kill switch: false.
auto_approval_enforce_enabled: bool = os.getenv(
"AUTO_APPROVAL_ENFORCE_ENABLED", "true"
).lower() in ("true", "1", "yes")
# Backing decider ENFORCEMENT (Phase 2B): allow a confident WITH_BACKING
# verdict from the 3-stem decider to auto-select the with-backing
# instrumental. Shadow-validated 2026-08-28/29 (20-job corpus + live
# batteries + human review pass: 10/11 pick agreement, the one
# auto-eligible job shipped identically to the human) -> default ON.
# Default lives HERE, not per-environment env vars, because the executor
# runs in BOTH the backend service and the audio-separation Cloud Run Job
# (independent env; a service-only env flip left the audio_worker trigger
# reading false). Kill switch: set false in ci.yml backend deploy env
# AND infrastructure audio-separation-job envs (or revert this default).
auto_approval_backing_keep_enabled: bool = os.getenv(
"AUTO_APPROVAL_BACKING_KEEP_ENABLED", "true"
).lower() in ("true", "1", "yes")
# Timing-plausibility gate: when the text signals say AUTO, check the word
# timing against the lead-vocal stem (backend/services/auto_approval/
# timing_check.py) and demote to review when it fires. Only ever ADDS
# review (never auto-ships more), so it defaults on. Code-level default for
# the same reason as backing_keep above (executor runs in both the backend
# service and the audio-separation job). Kill switch: set false in ci.yml
# backend deploy env AND the infrastructure audio-separation-job envs.
auto_approval_timing_gate_enabled: bool = os.getenv(
"AUTO_APPROVAL_TIMING_GATE_ENABLED", "true"
).lower() in ("true", "1", "yes")
# Review hold: when stems are still separating after lyrics finish, keep
# the job out of review until the audio-worker re-score has run and the
# review is fully prepared (backend/services/auto_approval/review_gate.py). Code-level default for
# the same two-environment reason as above. Kill switch: false.
auto_approval_audio_hold_enabled: bool = os.getenv(
"AUTO_APPROVAL_AUDIO_HOLD_ENABLED", "true"
).lower() in ("true", "1", "yes")
# Match judge — artist/title formatting + match-acceptance for the job
# submission flow. Deterministic + catalog layers always run; the light AI
# layer (Vertex Gemini Flash) fires only when those aren't confident.
match_judge_enabled: bool = os.getenv("MATCH_JUDGE_ENABLED", "true").lower() in (
"true", "1", "yes",
)
match_judge_model: str = os.getenv("MATCH_JUDGE_MODEL", "gemini-3.8-flash")
# Bounds the AI call so a hung model never strands the submission UI. Kept
# generous because Vertex returns 504 DEADLINE_EXCEEDED if generation can't
# finish within this deadline (3s was too tight for gemini-3.5-flash); the
# call still runs off the critical path and degrades to "none" on timeout.
match_judge_timeout_ms: int = int(os.getenv("MATCH_JUDGE_TIMEOUT_MS", "12000"))
# Batch karaoke-filename parser (kjbox download-naming). Reuses the Vertex
# Gemini stack. Larger timeout than match_judge because it batches ~200 items.
parse_titles_enabled: bool = os.getenv("PARSE_TITLES_ENABLED", "true").lower() in (
"1", "true", "yes",
)
parse_titles_model: str = os.getenv("PARSE_TITLES_MODEL", "gemini-3.8-flash")
parse_titles_timeout_ms: int = int(os.getenv("PARSE_TITLES_TIMEOUT_MS", "20000"))
parse_titles_max_items: int = int(os.getenv("PARSE_TITLES_MAX_ITEMS", "200"))
# Items per Gemini call: large client batches are split into concurrent
# chunks of this size so one call never outgrows parse_titles_timeout_ms
# (a 100-item single call did, blanking the whole batch).
parse_titles_chunk_size: int = int(os.getenv("PARSE_TITLES_CHUNK_SIZE", "10"))
# Cloud Tasks (for scalable worker coordination)
# When enabled, workers are triggered via Cloud Tasks for guaranteed delivery
# When disabled (default), workers are triggered via direct HTTP (for development)
enable_cloud_tasks: bool = os.getenv("ENABLE_CLOUD_TASKS", "false").lower() in ("true", "1", "yes")
gcp_region: str = os.getenv("GCP_REGION", "us-central1")
# ALL Cloud Tasks queues live in us-central1, but workers running in other
# regions (the GPU audio-separation-job sets GCP_REGION to its GPU region)
# must still enqueue to the real queue location — building queue paths from
# gcp_region 404s from those workers ("Queue does not exist").
cloud_tasks_region: str = os.getenv("CLOUD_TASKS_REGION", "us-central1")
# Region of the latency-critical CPU Cloud Run Jobs (audio-download-job,
# lyrics-transcription-job, bulk-search-job, video-encoding-job). Cloud Run
# Jobs in us-central1 queue 2-5 min before the container starts (measured
# 2026-09-28 with a tiny sample image: us-central1 4-5 min vs us-east4 8-19s),
# independent of image size or CPU. video-encoding-job reads ~0.3 GB/run from
# the us-central1 bucket cross-region (~$1.60/mo, 2026-09-29 model). Rollback:
# recreate the jobs in us-central1 via Pulumi, then set CPU_JOBS_REGION on the
# service AND the jobs (see docs/TROUBLESHOOTING.md). Deliberately separate from gcp_region,
# which these workers use for other resources (queues, encoding VM zones).
cpu_jobs_region: str = os.getenv("CPU_JOBS_REGION", "us-east4")
# Cloud Run Jobs (for long-running video encoding)
# When enabled AND enable_cloud_tasks is true, video worker uses Cloud Run Jobs
# instead of Cloud Tasks. This supports encoding times >30 minutes (up to 24 hours).
# Default is false - Cloud Tasks is sufficient for most videos (15-20 min).
# Inside a Cloud Run Job (Cloud Run sets CLOUD_RUN_JOB) both flags default to
# true: a worker job that triggers the next stage (audio worker resuming a
# parked encode, auto-approval starting a render) must never fall back to
# Cloud Tasks -> a BackgroundTask on the API service. That runs the whole
# video worker on the API's event loop (2026-10-19: 52s + 81s freezes,
# ~30 requests each). Explicit env values still win.
use_cloud_run_jobs_for_video: bool = os.getenv(
"USE_CLOUD_RUN_JOBS_FOR_VIDEO", "true" if os.getenv("CLOUD_RUN_JOB") else "false"
).lower() in ("true", "1", "yes")
# When enabled AND enable_cloud_tasks is true, the post-review render worker
# also runs as a Cloud Run Job (reusing video-encoding-job with an args
# override). As a BackgroundTask on the service it was killed by every
# deploy rollout that landed mid-render — Cloud Run gives instances only
# 10s after SIGTERM, far less than a render takes, so the job froze at
# rendering_video until the 45-min stuck-job sweep (incident 2026-09-13,
# job 41e06b90; same class as incident 2026-03-08 for the video worker).
use_cloud_run_jobs_for_render: bool = os.getenv(
"USE_CLOUD_RUN_JOBS_FOR_RENDER", "true" if os.getenv("CLOUD_RUN_JOB") else "false"
).lower() in ("true", "1", "yes")
use_cloud_run_jobs_for_video: bool = os.getenv("USE_CLOUD_RUN_JOBS_FOR_VIDEO", "false").lower() in ("true", "1", "yes")
# GCE Encoding Worker (for high-performance video encoding)
# When enabled, video encoding is offloaded to a dedicated C4 GCE instance
@@ -265,23 +77,12 @@ class Settings(BaseSettings):
encoding_worker_url: Optional[str] = os.getenv("ENCODING_WORKER_URL") # e.g., http://136.119.50.148:8080
encoding_worker_api_key: Optional[str] = os.getenv("ENCODING_WORKER_API_KEY")
# Optional capacity-fallback VMs in alternate zones. JSON list of objects:
# '[{"vm":"encoding-worker-fallback-a","zone":"us-central1-a","ip":"34.x.x.x"}]'
# Empty by default. After `pulumi up` provisions the fallback VMs (Phase 3
# of the capacity-resilience plan), populate this var with their public
# IPs to enable multi-zone failover when us-central1-c is exhausted.
encoding_worker_fallback_vms: Optional[str] = os.getenv("ENCODING_WORKER_FALLBACK_VMS")
# GCE Preview Encoding (for faster preview video generation)
# When enabled, preview video encoding during lyrics review is offloaded to the GCE worker.
# This reduces preview generation time from 60+ seconds to ~15-20 seconds.
# Requires use_gce_encoding to be enabled and the GCE worker to support /encode-preview endpoint.
use_gce_preview_encoding: bool = os.getenv("USE_GCE_PREVIEW_ENCODING", "false").lower() in ("true", "1", "yes")
# Karaoke-Decide main API: catalog autocomplete, and the partner call that
# erases a deleted account's decide data (DELETE /api/partner/gen/users)
karaoke_decide_api_url: str = os.getenv("KARAOKE_DECIDE_API_URL", "https://decide.nomadkaraoke.com")
# Storage paths
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
@@ -296,28 +97,7 @@ class Settings(BaseSettings):
# This is required for torrent downloads since Cloud Run doesn't support BitTorrent.
flacfetch_api_url: Optional[str] = os.getenv("FLACFETCH_API_URL") # e.g., http://10.0.0.5:8080
flacfetch_api_key: Optional[str] = os.getenv("FLACFETCH_API_KEY")
# TornadoAPI: tried first for YouTube downloads when the key is set, with flacfetch as fallback.
# Set YOUTUBE_DOWNLOAD_PROVIDER=flacfetch to switch it off without a code change.
tornado_api_key: Optional[str] = os.getenv("TORNADO_API_KEY")
youtube_download_provider: str = os.getenv("YOUTUBE_DOWNLOAD_PROVIDER", "tornado")
# Flacfetch client retry settings
# Retries help handle transient network failures when calling the flacfetch service
flacfetch_retry_max_attempts: int = int(os.getenv("FLACFETCH_RETRY_MAX_ATTEMPTS", "9")) # Total attempts (1 initial + 8 retries)
flacfetch_retry_min_wait: float = float(os.getenv("FLACFETCH_RETRY_MIN_WAIT", "10.0")) # seconds
flacfetch_retry_max_wait: float = float(os.getenv("FLACFETCH_RETRY_MAX_WAIT", "60.0")) # seconds
# Search retry settings are much lighter than the download defaults above:
# search is called synchronously from user-facing requests that sit behind
# Cloudflare's ~100s edge timeout, so there's only room for one quick retry
# (worst case: 60s timeout + 1s wait + fast cache-hit retry).
flacfetch_search_retry_max_attempts: int = int(os.getenv("FLACFETCH_SEARCH_RETRY_MAX_ATTEMPTS", "2"))
flacfetch_search_retry_wait: float = float(os.getenv("FLACFETCH_SEARCH_RETRY_WAIT", "1.0")) # seconds
# Total budget across all search attempts, kept under Cloudflare's ~100s edge
# deadline so callers get a clean FlacfetchServiceError instead of an edge 524.
flacfetch_search_total_timeout: float = float(os.getenv("FLACFETCH_SEARCH_TOTAL_TIMEOUT", "95.0")) # seconds
# Default distribution settings (can be overridden per-request)
default_dropbox_path: Optional[str] = os.getenv("DEFAULT_DROPBOX_PATH")
default_gdrive_folder_id: Optional[str] = os.getenv("DEFAULT_GDRIVE_FOLDER_ID")
@@ -330,221 +110,25 @@ class Settings(BaseSettings):
default_enable_youtube_upload: bool = os.getenv("DEFAULT_ENABLE_YOUTUBE_UPLOAD", "false").lower() in ("true", "1", "yes")
default_brand_prefix: Optional[str] = os.getenv("DEFAULT_BRAND_PREFIX")
# Requests voting board — Phase 2 daily free-community-track automation.
# Master kill-switch: when False the daily picker claims the day and records a
# "skipped" lock but never grants credits or creates a job (safe to deploy dark).
community_daily_pick_enabled: bool = os.getenv(
"COMMUNITY_DAILY_PICK_ENABLED", "false"
).lower() in ("true", "1", "yes")
# Max distinct voters the 24h ownership handoff will try before parking a track.
community_handoff_max_attempts: int = int(os.getenv("COMMUNITY_HANDOFF_MAX_ATTEMPTS", "5"))
# Hours an owner has to complete their review before the job is handed off.
community_handoff_hours: int = int(os.getenv("COMMUNITY_HANDOFF_HOURS", "24"))
# When a flagged pick is "kept on board", how long before it can be re-flagged.
community_review_snooze_days: int = int(os.getenv("COMMUNITY_REVIEW_SNOOZE_DAYS", "30"))
# Rate Limiting Configuration
# Enable/disable rate limiting system-wide (useful for development)
enable_rate_limiting: bool = os.getenv("ENABLE_RATE_LIMITING", "true").lower() in ("true", "1", "yes")
# Maximum jobs a user can create per day (0 = unlimited)
rate_limit_jobs_per_day: int = int(os.getenv("RATE_LIMIT_JOBS_PER_DAY", "5"))
# Maximum YouTube uploads system-wide per day (0 = unlimited)
rate_limit_youtube_uploads_per_day: int = int(os.getenv("RATE_LIMIT_YOUTUBE_UPLOADS_PER_DAY", "10"))
# Maximum beta enrollments from same IP per day (0 = unlimited)
rate_limit_beta_ip_per_day: int = int(os.getenv("RATE_LIMIT_BETA_IP_PER_DAY", "1"))
# KaraokeHunt retired-app request interceptor (POST /api/karaokehunt/request).
# Shared secret set by the Cloudflare Worker on create.karaokehunt.com; the
# endpoint 503s while this is empty, so deploys are dark until the secret is
# wired (Secret Manager: karaokehunt-forwarder-secret).
karaokehunt_forwarder_secret: str = os.getenv("KARAOKEHUNT_FORWARDER_SECRET", "")
# Max conversion jobs per UTC day (bounds unattended generation spend);
# over-cap requests fall through to the free community requests board.
karaokehunt_daily_job_cap: int = int(os.getenv("KARAOKEHUNT_DAILY_JOB_CAP", "3"))
# kjbox partner API (/api/kjbox/*): singer email-code sign-in + show credits
# for the karaoke-night "make it" flow. Every endpoint 503s while the secret is
# empty, so deploys are dark until it's wired (Secret Manager: kjbox-partner-secret).
kjbox_partner_secret: str = os.getenv("KJBOX_PARTNER_SECRET", "")
# karaoke-decide's shared secret (header X-Partner-Secret, Secret Manager:
# decide-gen-partner-secret): decide sends it to POST /api/kjbox/catalog/resolve, and
# gen sends it to decide's DELETE /api/partner/gen/users on account deletion.
# Empty = decide is not accepted, and account deletion 503s at the decide step.
decide_partner_secret: str = os.getenv("DECIDE_GEN_PARTNER_SECRET", "")
# Partner-wide cap on NEW gen accounts created via kjbox per rolling 24h (a venue
# shares one IP, so gen's per-IP signup cap is bypassed for kjbox; this bounds it).
kjbox_signup_cap_per_24h: int = int(os.getenv("KJBOX_SIGNUP_CAP_PER_24H", "100"))
# Max emailed sign-in codes per address per rolling hour.
kjbox_codes_per_email_per_hour: int = int(os.getenv("KJBOX_CODES_PER_EMAIL_PER_HOUR", "5"))
# Max "free at the show" credits kjbox may grant one user per rolling 24h.
kjbox_show_credits_per_user_per_24h: int = int(os.getenv("KJBOX_SHOW_CREDITS_PER_USER_PER_24H", "5"))
# gen email-code sign-in (/api/users/auth/send-code + verify-code).
# HMAC pepper for stored codes; when empty it's derived from the first admin
# token (see backend/api/routes/login_code.py:_resolve_pepper).
login_code_pepper: str = os.getenv("LOGIN_CODE_PEPPER", "")
# Max emailed sign-in codes per address / per client IP per rolling hour.
login_codes_per_email_per_hour: int = int(os.getenv("LOGIN_CODES_PER_EMAIL_PER_HOUR", "5"))
login_codes_per_ip_per_hour: int = int(os.getenv("LOGIN_CODES_PER_IP_PER_HOUR", "20"))
# Private (non-published) track distribution settings
default_private_dropbox_path: Optional[str] = os.getenv("DEFAULT_PRIVATE_DROPBOX_PATH")
default_private_brand_prefix: Optional[str] = os.getenv("DEFAULT_PRIVATE_BRAND_PREFIX", "NOMADNP")
# Output files NOT uploaded to Nomad's own Dropbox folders (DEFAULT_DROPBOX_PATH /
# DEFAULT_PRIVATE_DROPBOX_PATH), as "|"-separated filename suffixes. They stay in
# GCS (job-page downloads, YouTube upload, AWS backup) — this only saves the
# multi-GB internet egress + upload time. Tenant Dropbox folders are exempt
# (B2B tenants expect the full set). Default skips only the lossless 4K MP4,
# which no customer-facing text promises in Dropbox. The lossless MKV IS promised
# there (completion email + the Fiverr bot), so add
# " (Final Karaoke Lossless 4k).mkv" only after updating those.
# See docs/archive/2026-09-26-encoding-cost-cuts.md.
dropbox_skip_output_suffixes: str = os.getenv(
"DROPBOX_SKIP_OUTPUT_SUFFIXES", " (Final Karaoke Lossless 4k).mp4"
)
# YouTube Data API v3 Quota Configuration
# Daily quota limit (YouTube grants 10,000 units/day by default)
youtube_quota_daily_limit: int = int(os.getenv("YOUTUBE_QUOTA_DAILY_LIMIT", "10000"))
# Estimated units consumed per upload (videos.insert=100 + search.list=100 + thumbnails.set=50 + buffer)
youtube_quota_upload_cost: int = int(os.getenv("YOUTUBE_QUOTA_UPLOAD_COST", "300"))
# Safety margin to reserve (avoid hitting exact limit)
youtube_quota_safety_margin: int = int(os.getenv("YOUTUBE_QUOTA_SAFETY_MARGIN", "500"))
# E2E test bypass key for rate limiting (set via secret in production)
e2e_bypass_key: str = os.getenv("E2E_BYPASS_KEY", "")
# Local-stack E2E mode (scripts/local-stack.sh): workers are replaced by in-process
# fakes that walk jobs through their real states with canned media, the
# /api/test/* router is mounted, and external lookups (Secret Manager, DeBounce,
# ip-api, encoding VM) are skipped. Startup refuses it in production.
# See backend/services/fake_workers/ and docs/TESTING.md § Local stack.
fake_workers: bool = os.getenv("FAKE_WORKERS", "").lower() in ("1", "true", "yes")
# Mobile app account deletion (DELETE /api/users/me). Off in prod until
# Andrew's manual pass; on in the local stack.
account_deletion_enabled: bool = os.getenv("ACCOUNT_DELETION_ENABLED", "").lower() in ("1", "true", "yes")
# App Review demo login: inert unless this AND the app-review-demo-code secret are set.
app_review_demo_email: str = os.getenv("APP_REVIEW_DEMO_EMAIL", "")
# Secret Manager: app-review-demo-code, mounted as APP_REVIEW_DEMO_CODE.
app_review_demo_code: str = os.getenv("APP_REVIEW_DEMO_CODE", "")
# RevenueCat (mobile in-app credit packs). The webhook (POST
# /api/users/webhooks/revenuecat) compares the Authorization header to this
# value and 503s while it is empty (Secret Manager: revenuecat-webhook-auth).
revenuecat_webhook_auth: str = os.getenv("REVENUECAT_WEBHOOK_AUTH", "")
# RevenueCat secret REST API key (Secret Manager: revenuecat-secret-api-key).
revenuecat_secret_api_key: str = os.getenv("REVENUECAT_SECRET_API_KEY", "")
# RevenueCat project the v2 key belongs to (account deletion's customer delete).
revenuecat_project_id: str = os.getenv("REVENUECAT_PROJECT_ID", "9ea947ce")
# URL scheme of the Nomad Karaoke mobile app. App checkouts return through
# GET /api/users/credits/app-return, which 302s to {scheme}://checkout?status=…
mobile_app_scheme: str = os.getenv("MOBILE_APP_SCHEME", "nomadkaraoke")
# Accepted `aud` values for Google ID tokens (POST /api/users/auth/google),
# comma-separated. OAuth client IDs are public, not secrets. Defaults: the
# "Nomad Karaoke Web" client (the audience of tokens from Android Credential
# Manager / iOS with serverClientId) and the iOS client (iOS SDK tokens can
# carry it). The Android client is device-only and never an audience. See
# docs/ARCHITECTURE.md § Mobile app sign-in. Empty → the endpoint returns 503.
google_oauth_client_ids: str = os.getenv(
"GOOGLE_OAUTH_CLIENT_IDS",
"718638054799-fush0smd1e4icq9pt0immp8n59jjlebh.apps.googleusercontent.com,"
"718638054799-hre535n83p8i85tsqf6ai759nlubhd2o.apps.googleusercontent.com",
)
# Accepted `aud` values for Apple ID tokens (POST /api/users/auth/apple):
# the app's bundle ID (native iOS), plus any Services ID used for web/Android:
# com.nomadkaraoke.sing = Sign in with Apple JS on the kjbox singer page
# (sing.nomadkaraoke.com → POST /api/kjbox/auth/apple).
apple_signin_audiences: str = os.getenv("APPLE_SIGNIN_AUDIENCES", "com.nomadkaraoke.mobile,com.nomadkaraoke.sing")
# Sign in with Apple token revocation (account deletion). The .p8 PEM text
# and key id come from Secret Manager; unset => revocation is skipped.
apple_siwa_key_p8: Optional[str] = os.getenv("APPLE_SIWA_KEY_P8")
apple_siwa_key_id: Optional[str] = os.getenv("APPLE_SIWA_KEY_ID")
apple_team_id: str = os.getenv("APPLE_TEAM_ID", "55B76HN73D")
apple_client_id: str = os.getenv("APPLE_CLIENT_ID", "com.nomadkaraoke.mobile")
# Canonical YouTube description template. Placeholders ({title}, {artist},
# {artist_hashtag}, {brand_code}) are filled by
# backend.services.youtube_description.render_youtube_description, which is the
# single source of truth shared with the bulk-rewrite tool
# (scripts/youtube-descriptions). Keep those two in sync when editing this.
default_youtube_description: str = os.getenv(
"DEFAULT_YOUTUBE_DESCRIPTION",
'🎤 Karaoke version of "{title}" by {artist}: original audio, lead vocals '
"removed, hand-synced lyrics.\n"
"▶️ Make a karaoke video of ANY song: "
"https://nomadkaraoke.com/r/youtube?utm_source=youtube&utm_medium=description&utm_campaign=make_cta_top\n\n"
"⬆️ REQUEST THE NEXT NOMAD RELEASE\n"
"Vote for the songs you want us to make next — the most-upvoted requests get "
"made into karaoke videos and published free, right here on this channel. "
"Nothing to buy, no password needed:\n"
"👉 https://requests.nomadkaraoke.com/?utm_source=youtube&utm_medium=description&utm_campaign=requests\n\n"
"▶️ MAKE A VIDEO LIKE THIS FOR ANY SONG\n"
"Create studio-quality karaoke videos of ANY song, ready in under an hour:\n"
"👉 https://nomadkaraoke.com/r/youtube?utm_source=youtube&utm_medium=description&utm_campaign=make_cta\n"
"Follow that link for 50% off all credit purchases for your first 30 days.\n\n"
"🎶 KARAOKE LOVER LINKS\n"
"• Nomad Karaoke Community Discord: https://discord.nomadkaraoke.com\n"
"• Decide what to sing at karaoke: https://decide.nomadkaraoke.com\n"
"• Search all community karaoke tracks: https://karaokenerds.com\n\n"
"🔔 Subscribe for new karaoke tracks, and comment to show some love!\n\n"
"#karaoke #{artist_hashtag} #instrumental #lyrics #singalong\n"
"—\n"
"COPYRIGHT: I don't own the rights to the original music; all rights belong "
"to the respective copyright holders. If you enjoy the song, please buy the "
"original and support the artist. Under §107 of the U.S. Copyright Act 1976, "
'allowance is made for "fair use" (criticism, comment, teaching, and '
"research).\n\n"
"Brand Code: {brand_code}"
"Karaoke video created with Nomad Karaoke (https://nomadkaraoke.com)\n\n"
"AI-powered vocal separation and synchronized lyrics.\n\n"
"#karaoke #music #singing #instrumental #lyrics"
)
# Offer an opt-in "tip the creator" optional item on gen Stripe Checkout pages
# (credits + Made For You). Kill switch: CHECKOUT_TIP_ENABLED=false.
checkout_tip_enabled: bool = os.getenv("CHECKOUT_TIP_ENABLED", "true").lower() in ("true", "1", "yes")
# Free-to-paid conversion emails (backend/services/conversion_emails.py).
# OFF until Andrew approves the copy: CONVERSION_EMAILS_ENABLED=true sends E1
# ("your next song" + time-limited discount) and E2 (offer ending) on Postmark's
# broadcast stream; REVIEW_LAST_CHANCE_ENABLED=true adds the 42h review nudge.
conversion_emails_enabled: bool = os.getenv("CONVERSION_EMAILS_ENABLED", "false").lower() in ("true", "1", "yes")
review_last_chance_enabled: bool = os.getenv("REVIEW_LAST_CHANCE_ENABLED", "false").lower() in ("true", "1", "yes")
conversion_offer_percent: int = int(os.getenv("CONVERSION_OFFER_PERCENT", "25"))
conversion_offer_hours: int = int(os.getenv("CONVERSION_OFFER_HOURS", "48"))
conversion_e1_delay_hours: float = float(os.getenv("CONVERSION_E1_DELAY_HOURS", "2"))
conversion_e2_hours_before_expiry: float = float(os.getenv("CONVERSION_E2_HOURS_BEFORE_EXPIRY", "8"))
# CAN-SPAM: commercial email must carry a physical postal address.
# Default = the business address already published on nomadkaraoke.com.
commercial_email_postal_address: str = os.getenv(
"COMMERCIAL_EMAIL_POSTAL_ADDRESS", "Nomad Karaoke, 612 Joan St, Columbia, SC 29203, USA"
)
# YouTube description back-catalogue rewrite (scheduled daily drain worker).
# Rewrites existing channel videos to match `default_youtube_description`.
# Reusable: whenever the template changes, the worker re-drains the channel.
youtube_backfill_enabled: bool = os.getenv("YOUTUBE_BACKFILL_ENABLED", "true").lower() in ("true", "1", "yes")
# Hard ceiling on updates per daily run (each costs ~50 quota units).
youtube_backfill_daily_max_updates: int = int(os.getenv("YOUTUBE_BACKFILL_DAILY_MAX_UPDATES", "150"))
# Quota units to leave untouched for the live upload pipeline each day.
youtube_backfill_quota_reserve: int = int(os.getenv("YOUTUBE_BACKFILL_QUOTA_RESERVE", "3000"))
# Refresh tags too (not just description) when rewriting.
youtube_backfill_enrich_tags: bool = os.getenv("YOUTUBE_BACKFILL_ENRICH_TAGS", "true").lower() in ("true", "1", "yes")
# Where the daily progress + completion emails are sent.
youtube_backfill_report_email: str = os.getenv("YOUTUBE_BACKFILL_REPORT_EMAIL", "andrew@nomadkaraoke.com")
# YouTube comment auto-reply (backend/workers/youtube_comment_bot_worker.py):
# replies to song requests in channel comments with a requests-board link.
# OFF by default; go-live only after Andrew signs off a local replay
# (scripts/youtube-comments/replay.py). Dry runs work while off.
youtube_comment_autoreply_enabled: bool = os.getenv("YOUTUBE_COMMENT_AUTOREPLY_ENABLED", "false").lower() in ("true", "1", "yes")
youtube_comment_autoreply_max_replies: int = int(os.getenv("YOUTUBE_COMMENT_AUTOREPLY_MAX_REPLIES", "20"))
youtube_comment_autoreply_min_confidence: float = float(os.getenv("YOUTUBE_COMMENT_AUTOREPLY_MIN_CONFIDENCE", "0.8"))
# ISO date/time: on the FIRST run, comments since this are eligible (a one-time
# backfill of requests nobody answered). Empty = only comments after go-live.
youtube_comment_autoreply_start: str = os.getenv("YOUTUBE_COMMENT_AUTOREPLY_START", "")
# Storage retention (backend/services/storage_retention.py): a daily job purges
# regenerable files (big finals, rendered videos, stems, ...) from jobs
# completed more than `storage_retention_min_age_days` ago. DRY-RUN BY DEFAULT:
# it only writes a report until STORAGE_RETENTION_DRY_RUN=false is set after a
# human has reviewed one.
storage_retention_enabled: bool = os.getenv("STORAGE_RETENTION_ENABLED", "true").lower() in ("true", "1", "yes")
storage_retention_dry_run: bool = os.getenv("STORAGE_RETENTION_DRY_RUN", "true").lower() not in ("false", "0", "no")
storage_retention_min_age_days: int = int(os.getenv("STORAGE_RETENTION_MIN_AGE_DAYS", "30"))
# Real (non-dry) runs purge at most this many jobs per run, resuming from a cursor.
storage_retention_max_jobs_per_run: int = int(os.getenv("STORAGE_RETENTION_MAX_JOBS_PER_RUN", "50"))
# Comma-separated tenant ids whose jobs are never purged.
storage_retention_excluded_tenants: str = os.getenv("STORAGE_RETENTION_EXCLUDED_TENANTS", "")
# Customer "Regenerate video" limits (per job / per user, rolling 24h).
regenerate_max_per_job_per_day: int = int(os.getenv("REGENERATE_MAX_PER_JOB_PER_DAY", "3"))
regenerate_max_per_user_per_day: int = int(os.getenv("REGENERATE_MAX_PER_USER_PER_DAY", "10"))
# Default CDG/TXT generation settings
# When True, CDG and TXT packages are generated by default (when a theme is set)
# These can be overridden per-request via explicit enable_cdg/enable_txt parameters
@@ -591,18 +175,12 @@ class Settings(BaseSettings):
self._secret_cache[secret_id] = env_value
return env_value
# Local stack runs offline on emulators: never reach for Secret Manager.
if self.fake_workers is True and not is_production():
return None
# Try Secret Manager (production mode)
if not self.google_cloud_project:
logger.warning(f"Cannot fetch secret {secret_id}: No GCP project configured")
return None
try:
from google.cloud import secretmanager
client = secretmanager.SecretManagerServiceClient()
name = f"projects/{self.google_cloud_project}/secrets/{secret_id}/versions/latest"
response = client.access_secret_version(request={"name": name})
@@ -627,56 +205,3 @@ def get_settings() -> Settings:
"""Get the global settings instance."""
return settings
def is_production() -> bool:
"""True when running in a deployed/production environment.
Cloud Run automatically sets ``K_SERVICE``, so we treat its presence as
production even if ``ENVIRONMENT`` were somehow unset — a production-safe
default. Local dev/test have no ``K_SERVICE`` and a non-"production"
``ENVIRONMENT``. (Fallback audit 2026-06-09, Theme 7.)
"""
env = (os.environ.get("ENVIRONMENT") or os.environ.get("ENV") or "").lower()
if env == "production":
return True
return bool(os.environ.get("K_SERVICE"))
# Env vars that MUST be explicitly set in production. A missing one used to
# silently fall back to a dev default and produce wrong behaviour with only a
# warning (wrong GCP project for secret lookups, wrong GCS bucket, localhost
# worker URL). We only require vars that infra confirms are set on every service
# (GCS temp/output buckets are intentionally excluded — their defaults are
# load-bearing and not set via env). (Fallback audit 2026-06-09, Theme 7.)
_REQUIRED_PRODUCTION_ENV_VARS = (
"GOOGLE_CLOUD_PROJECT",
"GCS_BUCKET_NAME",
"CLOUD_RUN_SERVICE_URL",
)
def validate_production_config() -> None:
"""Fail fast at startup if required production env vars are missing.
In production each var in ``_REQUIRED_PRODUCTION_ENV_VARS`` must be explicitly
set; otherwise the service would silently run with a dev default (wrong GCP
project / wrong GCS bucket / localhost worker URL) and appear to work while
doing the wrong thing. Raises RuntimeError listing every missing var. No-op
outside production. (Fallback audit 2026-06-09, Theme 7.)
"""
if not is_production():
return
if settings.fake_workers is True:
raise RuntimeError(
"FAKE_WORKERS is set in production. It replaces every worker with a fake "
"and mounts the unauthenticated /api/test/* router. Refusing to start."
)
missing = [v for v in _REQUIRED_PRODUCTION_ENV_VARS if not (os.environ.get(v) or "").strip()]
if missing:
raise RuntimeError(
"Production misconfiguration: required environment variable(s) not set: "
+ ", ".join(missing)
+ ". Refusing to start with silent dev defaults (wrong GCP project / "
"GCS bucket / localhost worker URL)."
)
@@ -1,17 +0,0 @@
# Video IDs to FORCE-rewrite even though the title isn't high-confidence
# "Artist - Title (Karaoke)" — e.g. genuine tracks whose "(Karaoke)" suffix was
# truncated by YouTube's 100-char title limit.
#
# Two forms per line ('#' starts a comment):
# <id> use the tool's best-effort parsed artist/title
# <id> | <Artist> | <Title> override artist/title (for truncated titles)
#
# --- Genuine tracks rescued from the "review candidates" bucket (2026-09-03) ---
# Truncated titles restored to full song titles:
5zSFKuCEIhs | Los Campesinos! | This Is How You Spell "HAHAHA, We Destroyed the Hopes and Dreams of a Generation of Faux-Romantics"
kkT-bZhi9jI | Arctic Monkeys | You Probably Couldn't See for the Lights but You Were Looking Straight at Me
# Long medley titles parsed fine — no override needed:
le347q8nHT4 # Daniel O'Donnell - Medley: Galway Bay, The Mountains Of Mourne, ...
3ihDlPse--k # Daniel O'Donnell - Medley: It's a Long Way to Tipperary / ...
@@ -1,29 +0,0 @@
# Video IDs that should NEVER have their description rewritten by the bulk tool.
# One YouTube video ID per line. Everything after a '#' is ignored.
#
# The "unparseable" videos are already auto-skipped by the classifier; these are
# listed explicitly for the record + belt-and-suspenders (so they stay skipped
# even if a title changes later).
# --- Lyrics videos WITH vocals (NOT instrumental karaoke) ---
SUkl49w9GPA # buildspace, MAYV - the end (Lyrics Video)
jAAhbfmh8Bc # 100 gecs - mememe (Lyrics Video With Vocals)
mcZtK8kkx3E # Fleet Foxes - Montezuma (Lyrics / With Vocals)
# --- Demos / tutorials / screencasts / pitch / channel-explainer (not karaoke) ---
VTR3oaqgn0A # Nomad karaoke-gen hosted demo screencast 2026-05-16
-dI3r7qXo3A # Nomad Karaoke Generator - Full Tutorial
C0Zyy_9V5F8 # Purim Shpiel custom lyrics karaoke tracks production screencast
josVDsYwsFM # Nomad Karaoke Lyrics Review Demo (for AudioShake) Screencast
NZ8j0uxq6tA # YouTube "Reused content" claim appeal - channel explanation
asnwjLyqY4k # Nomad Karaoke production workflow (real time) 2025-02-26
dPWyYfjfHvw # Pitch Night - BIC Energy
63-Fk3mfZ7Q # Nomad Karaoke Creation Workflow (Normal Speed) 2023-12-19
ZsROHgqAVHs # Nomad Karaoke Creation Workflow (MidiCo Tutorial) 2023-12-19
# --- Unparseable-title non-karaoke (already auto-skipped; explicit for record) ---
RLkIEHGwFcU # Nomad Karaoke Generator Quick Demo
GJY60IfIHOQ # Nomad Karaoke Generator Demo 2026
v1Hox6XMYEw # "YouTube Karaoke just suffered a major loss" (short/commentary)
R3eOl1aeq5c # Nomad Karaoke stands with Lemmy Caution (commentary)
wHiMti3xLJE # Nomad Karaoke Creation Tutorial / Demo 1 (MidiCo, July 2023)
View File
Whitespace-only changes.
Whitespace-only changes.
@@ -1,81 +0,0 @@
"""One-shot script to populate the year-5-stars fixture from a real production job.
Run once:
poetry run python -m backend.eval.custom_lyrics._bootstrap_year5
"""
from __future__ import annotations
import json
import shutil
from pathlib import Path
from google.cloud import firestore
JOB_ID = "2cb49a45"
FIXTURE_DIR = Path(__file__).parent / "fixtures" / "year-5-stars-shake-it-off"
CLIENT_INPUT_SRC = Path("/Users/andrew/Projects/nomadkaraoke/year-5-stars-client-custom-lyrics.txt")
def main() -> None:
FIXTURE_DIR.mkdir(parents=True, exist_ok=True)
fs = firestore.Client(project="nomadkaraoke")
job_doc = fs.collection("jobs").document(JOB_ID).get()
if not job_doc.exists:
raise SystemExit(f"Job {JOB_ID} not found")
job = job_doc.to_dict()
# Corrected lyrics are stored inline in state_data (not in GCS for this job)
corrected = (
job.get("state_data", {}).get("corrected_lyrics")
or job.get("corrected_lyrics")
or {}
)
if not corrected:
# Fallback: try GCS path if present
corrected_path = job.get("corrected_json_gcs_path") or job.get("lyrics", {}).get("corrected_path")
if corrected_path:
from google.cloud import storage # noqa: PLC0415
storage_client = storage.Client(project="nomadkaraoke")
bucket_name, *blob_parts = corrected_path.replace("gs://", "").split("/", 1)
blob = storage_client.bucket(bucket_name).blob(blob_parts[0])
corrected = json.loads(blob.download_as_bytes())
else:
raise SystemExit(f"Job {JOB_ID} has no corrected lyrics (checked state_data and GCS path fields)")
segments = corrected.get("corrected_segments") or corrected.get("segments") or []
if not segments:
raise SystemExit("No segments found in corrected.json")
(FIXTURE_DIR / "original_segments.json").write_text(json.dumps(segments, indent=2))
(FIXTURE_DIR / "original_lyrics.txt").write_text(
"\n".join(seg.get("text", "").strip() for seg in segments) + "\n"
)
if CLIENT_INPUT_SRC.exists():
shutil.copy(CLIENT_INPUT_SRC, FIXTURE_DIR / "client_input.txt")
else:
print(f"WARNING: {CLIENT_INPUT_SRC} not found; populate client_input.txt manually")
metadata = {
"id": "year-5-stars-shake-it-off",
"artist": job.get("artist") or "Taylor Swift",
"title": job.get("title") or "Shake It Off",
"source_job_id": JOB_ID,
"difficulty": "hard",
"input_style": "long substantive lines, name-heavy, mismatched syllable budget",
"settings_to_test": [
{"name": "default", "allow_reword": True, "allow_omit": True, "fixed_line_count": True, "strictness": "balanced"},
{"name": "verbatim", "allow_reword": True, "allow_omit": True, "fixed_line_count": True, "strictness": "verbatim"},
{"name": "strict", "allow_reword": True, "allow_omit": True, "fixed_line_count": True, "strictness": "strict"},
],
}
(FIXTURE_DIR / "metadata.json").write_text(json.dumps(metadata, indent=2))
print(f"Fixture populated at {FIXTURE_DIR}")
print(f" - {len(segments)} segments")
if __name__ == "__main__":
main()
-64
View File
@@ -1,64 +0,0 @@
"""Disk-backed cache for LLM responses, keyed by (system, user, model, settings)."""
from __future__ import annotations
import hashlib
import json
from pathlib import Path
from typing import Any, Optional
class LlmCallCache:
def __init__(self, cache_dir: Path, replay_only: bool = False) -> None:
self.cache_dir = cache_dir
self.cache_dir.mkdir(parents=True, exist_ok=True)
self.replay_only = replay_only
def _key(
self,
system_prompt: str,
user_prompt: str,
*,
model: str,
settings_dict: dict[str, Any],
) -> str:
h = hashlib.sha256()
h.update(system_prompt.encode())
h.update(b"\x00")
h.update(user_prompt.encode())
h.update(b"\x00")
h.update(model.encode())
h.update(b"\x00")
h.update(json.dumps(settings_dict, sort_keys=True).encode())
return h.hexdigest()
def _path(self, key: str) -> Path:
return self.cache_dir / f"{key}.json"
def get(
self,
system_prompt: str,
user_prompt: str,
*,
model: str,
settings_dict: dict[str, Any],
) -> Optional[list[str]]:
path = self._path(self._key(system_prompt, user_prompt, model=model, settings_dict=settings_dict))
if not path.exists():
if self.replay_only:
raise RuntimeError(f"cache miss in replay-only mode: {path.name}")
return None
return json.loads(path.read_text())["lines"]
def set(
self,
system_prompt: str,
user_prompt: str,
*,
model: str,
settings_dict: dict[str, Any],
response_lines: list[str],
) -> None:
if self.replay_only:
return
path = self._path(self._key(system_prompt, user_prompt, model=model, settings_dict=settings_dict))
path.write_text(json.dumps({"lines": response_lines}, indent=2))
@@ -1,2 +0,0 @@
*
!.gitignore
@@ -1 +0,0 @@
We're singing this for Alex's birthday! Can you change the lyrics so it says Alex's name instead of "he"? Like "For she's a jolly good fellow" or however it makes sense with Alex. Just make it clear it's for Alex. It's a birthday celebration for our friend Alex.
@@ -1,11 +0,0 @@
{
"id": "jolly-good-fellow-name-replace",
"artist": "Traditional",
"title": "For He's a Jolly Good Fellow",
"difficulty": "easy",
"input_style": "explicit name replacement instruction",
"settings_to_test": [
{"name": "default", "allow_reword": true, "allow_omit": true, "fixed_line_count": true, "strictness": "balanced"},
{"name": "strict", "allow_reword": true, "allow_omit": true, "fixed_line_count": true, "strictness": "strict"}
]
}
@@ -1,4 +0,0 @@
For he's a jolly good fellow
For he's a jolly good fellow
For he's a jolly good fellow
Which nobody can deny
@@ -1,30 +0,0 @@
[
{
"id": "seg-00",
"start_time": 0.0,
"end_time": 2.5,
"text": "For he's a jolly good fellow",
"words": []
},
{
"id": "seg-01",
"start_time": 2.5,
"end_time": 5.0,
"text": "For he's a jolly good fellow",
"words": []
},
{
"id": "seg-02",
"start_time": 5.0,
"end_time": 7.5,
"text": "For he's a jolly good fellow",
"words": []
},
{
"id": "seg-03",
"start_time": 7.5,
"end_time": 10.0,
"text": "Which nobody can deny",
"words": []
}
]
@@ -1 +0,0 @@
We need this song to be about Mike and Sarah's wedding. Mike is a software engineer who loves climbing mountains, hiking on Saturdays with his border collie Max, and brewing his own beer in the basement. Sarah is a pediatric nurse with three sisters — Jenny, Tara, and Rachel — and she runs marathons every fall in the city park near her apartment. They met five years ago at a friend's birthday party in Brooklyn and bought their first house together last summer in Queens. Make the song celebrate their journey together. Highlight Max the dog if you can, and mention the marathons and the brewing.
@@ -1,11 +0,0 @@
{
"id": "row-row-line-surplus",
"artist": "Traditional",
"title": "Row, Row, Row Your Boat",
"difficulty": "medium",
"input_style": "client input far exceeds song's structural capacity (line surplus)",
"settings_to_test": [
{"name": "default", "allow_reword": true, "allow_omit": true, "fixed_line_count": true, "strictness": "balanced"},
{"name": "strict", "allow_reword": true, "allow_omit": true, "fixed_line_count": true, "strictness": "strict"}
]
}
@@ -1,4 +0,0 @@
Row, row, row your boat,
Gently down the stream,
Merrily, merrily, merrily, merrily,
Life is but a dream.
@@ -1,30 +0,0 @@
[
{
"text": "Row, row, row your boat,",
"start_time": 0.0,
"end_time": 1.5,
"id": "rr-0",
"words": []
},
{
"text": "Gently down the stream,",
"start_time": 1.5,
"end_time": 3.0,
"id": "rr-1",
"words": []
},
{
"text": "Merrily, merrily, merrily, merrily,",
"start_time": 3.0,
"end_time": 4.5,
"id": "rr-2",
"words": []
},
{
"text": "Life is but a dream.",
"start_time": 4.5,
"end_time": 6.0,
"id": "rr-3",
"words": []
}
]
@@ -1 +0,0 @@
Hi! Can you change the word "star" to "Sam" throughout the song? Every time "star" appears, I want it to say "Sam" instead. It's for my son Sam's birthday party. Thanks!
@@ -1,11 +0,0 @@
{
"id": "twinkle-name-swap",
"artist": "Traditional",
"title": "Twinkle Twinkle Little Star",
"difficulty": "easy",
"input_style": "simple word substitution, well-defined target",
"settings_to_test": [
{"name": "default", "allow_reword": true, "allow_omit": true, "fixed_line_count": true, "strictness": "balanced"},
{"name": "strict", "allow_reword": true, "allow_omit": true, "fixed_line_count": true, "strictness": "strict"}
]
}
@@ -1,6 +0,0 @@
Twinkle, twinkle, little star
How I wonder what you are
Up above the world so high
Like a diamond in the sky
Twinkle, twinkle, little star
How I wonder what you are
@@ -1,44 +0,0 @@
[
{
"id": "seg-00",
"start_time": 0.0,
"end_time": 2.0,
"text": "Twinkle, twinkle, little star",
"words": []
},
{
"id": "seg-01",
"start_time": 2.0,
"end_time": 4.0,
"text": "How I wonder what you are",
"words": []
},
{
"id": "seg-02",
"start_time": 4.0,
"end_time": 6.0,
"text": "Up above the world so high",
"words": []
},
{
"id": "seg-03",
"start_time": 6.0,
"end_time": 8.0,
"text": "Like a diamond in the sky",
"words": []
},
{
"id": "seg-04",
"start_time": 8.0,
"end_time": 10.0,
"text": "Twinkle, twinkle, little star",
"words": []
},
{
"id": "seg-05",
"start_time": 10.0,
"end_time": 12.0,
"text": "How I wonder what you are",
"words": []
}
]
@@ -1,68 +0,0 @@
We walk in the room, yeah we own this place
Anna hits the floor, watch her lead the way
Grace brings the rhythm, dancing every day
Oona on guitar, getting in the groove
Savannah steps forward, watch her make her move
Eirini singing, Frankie right there
Voices together, filling the air
Step aside now, nobody stands a chance
Mr Moulton jokes, we try not to laugh
But we’re ready now, we’ve done the graft
We’re gonna win, win, win, hear the crowd go loud
Singing all together, yeah we’re making you proud
Clap your hands, feel the beat, come and take a glance
We’re Year 5 strong, nobody stands a chance
Oh oh, here we go
Oh oh, steal the show
Watch us now, take our stance
Nobody stands a chance
Mr Bunney listening, nodding to the beat
Mr Woodward watching, tapping his feet
We’ll impress him with every note we play
Best performance of the whole school day
Mrs Woodbridge smiling, watching from the side
Thinking “Year 6 will be full of pride”
We’re stepping up, this is our dance
Better step back, nobody stands a chance
If we feel nervous, we just smile
We’ve been ready for a while
We’re gonna win, win, win, hear the crowd go loud
Singing all together, yeah we’re making you proud
Clap your hands, feel the beat, come and take a glance
We’re Year 5 strong, nobody stands a chance
Oh oh, here we go
Oh oh, steal the show
Watch us now, take our stance
Nobody stands a chance
Savannah solo
I step to the front, now it’s my time
Hear my voice, watch me shine
All eyes on me, feel that vibe
This is my moment, I come alive
Strum it strong, feel the sound
Let the music spin around
Keep the rhythm, don’t slow down
Now we bring it back around
We’re gonna win, win, win, loud and clear
Best group performance of the year
Sing it out, give it all, take a chance
We’re Year 5 stars, nobody stands a chance
Oh oh, this is it
Oh oh, we won’t quit
Watch us now, take our stance
Nobody stands a chance
@@ -1,31 +0,0 @@
{
"id": "year-5-stars-shake-it-off",
"artist": "Taylor Swift",
"title": "Year 5 Stars",
"source_job_id": "2cb49a45",
"difficulty": "hard",
"input_style": "long substantive lines, name-heavy, mismatched syllable budget",
"settings_to_test": [
{
"name": "default",
"allow_reword": true,
"allow_omit": true,
"fixed_line_count": true,
"strictness": "balanced"
},
{
"name": "verbatim",
"allow_reword": true,
"allow_omit": true,
"fixed_line_count": true,
"strictness": "verbatim"
},
{
"name": "strict",
"allow_reword": true,
"allow_omit": true,
"fixed_line_count": true,
"strictness": "strict"
}
]
}
@@ -1,70 +0,0 @@
We walk in the room
Yeah we own this place
Anna hits the floor, mm- mm
Watch her lead the way, mm- mm
Grace brings the rhythm
Dancing every day
Oona on guitar, mm- mm
Getting in the groove, mm- mm
Eirini singing, Frankie right there
Voices together, filling the air
Nobody stands a chance
'Cause the players gonna play, play, play, play, play
And the haters gonna hate, hate, hate, hate, hate
Baby, I'm just gonna shake, shake, shake, shake, shake
I shake it off, I shake it off
Heartbreakers gonna break, break, break, break, break
And the fakers gonna fake, fake, fake, fake, fake
Baby, I'm just gonna shake, shake, shake, shake, shake
I shake it off, I shake it off
Mr Moulton jokes
We try not to laugh
But we’re ready now, mm- mm
We’ve done the graft, mm- mm
We’re gonna win (hear the crowd go loud)
Singing all together (yeah we’re making you proud)
Clap your hands, feel the beat, mm- mm
Come and take a glance, mm- mm
We’re Year 5 strong
Here we go, steal the show
Watch us now, take our stance
Nobody stands a chance
'Cause the players gonna play, play, play, play, play
And the haters gonna hate, hate, hate, hate, hate
Baby, I'm just gonna shake, shake, shake, shake, shake
I shake it off, I shake it off
Heartbreakers gonna break, break, break, break, break
And the fakers gonna fake, fake, fake, fake, fake
Baby, I'm just gonna shake, shake, shake, shake, shake
I shake it off, I shake it off
Shake it off, I shake it off
I- I- I shake it off, I shake it off
I- I- I shake it off, I shake it off
I- I- I shake it off, I shake it off
Hey, hey, hey
Just think, while you've been gettin' down and out about the liars
And the dirty, dirty cheats of the world
You could've been gettin' down to this sick beat
My ex- man brought his new girlfriend
She's like, "Oh my God"
But I'm just gonna shake
And to the fella over there with the hella good hair
Won't you come on over, baby? We can shake, shake, shake
Yeah, oh
'Cause the players gonna play, play, play, play, play
And the haters gonna hate, hate, hate, hate, hate
I'm just gonna shake, shake, shake, shake, shake
I shake it off, I shake it off
Heartbreakers gonna break, break, break, break, break
And the fakers gonna fake, fake, fake, fake
I'm just gonna shake, shake, shake, shake
I shake it off, I shake it off
Shake it off, I shake it off
I- I- I shake it off, I shake it off
I- I- I shake it off, I shake it off
I- I- I shake it off, I shake it off
Shake it off, I shake it off
I- I- I shake it off, I shake it off
(You got to)
I- I- I shake it off, I shake it off
I- I- I shake it off, I shake it off
-109
View File
@@ -1,109 +0,0 @@
"""Markdown report generation for eval runs."""
from __future__ import annotations
import json
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional
from backend.eval.custom_lyrics.runner import FixtureRunResult
from backend.eval.custom_lyrics.scorer import aggregate_corpus
def write_run_reports(
results: list[FixtureRunResult],
out_dir: Path,
baseline: Optional[dict] = None,
) -> None:
out_dir.mkdir(parents=True, exist_ok=True)
(out_dir / "per_fixture").mkdir(exist_ok=True)
summary_lines = ["# Custom Lyrics Eval Run", ""]
summary_lines.append(f"_Generated: {datetime.now(timezone.utc).isoformat()}_")
summary_lines.append("")
summary_lines.append("## Per-fixture summary")
summary_lines.append("")
summary_lines.append("| Fixture | Settings | pass@±2 | Δ | mean_Δ | iters | stop |")
summary_lines.append("|---------|----------|---------|---|--------|-------|------|")
for r in results:
s = r.score
delta_str = ""
if baseline:
prev = baseline.get(r.fixture_id, {}).get(r.settings_name, {})
if "pct_pass_at_2" in prev:
d = s.pct_pass_at_2 - prev["pct_pass_at_2"]
delta_str = f"{d:+.2f}"
summary_lines.append(
f"| {r.fixture_id} | {r.settings_name} | {s.pct_pass_at_2:.2f} | "
f"{delta_str} | {s.mean_delta:.2f} | {s.iterations_used} | {s.stop_reason} |"
)
summary_lines.append("")
corpus = aggregate_corpus([r.score for r in results])
summary_lines.append("## Corpus aggregate")
summary_lines.append("")
summary_lines.append(f"- Macro pass@±2: {corpus.get('macro_pct_pass_at_2', 0):.2f}")
summary_lines.append(f"- Macro pass@±1: {corpus.get('macro_pct_pass_at_1', 0):.2f}")
summary_lines.append(f"- Macro pass@±0: {corpus.get('macro_pct_pass_at_0', 0):.2f}")
summary_lines.append(f"- Total Gemini calls: {corpus.get('total_gemini_calls', 0)}")
(out_dir / "summary.md").write_text("\n".join(summary_lines))
for r in results:
per_dir = out_dir / "per_fixture" / f"{r.fixture_id}__{r.settings_name}"
per_dir.mkdir(parents=True, exist_ok=True)
(per_dir / "output.json").write_text(json.dumps({
"fixture_id": r.fixture_id,
"settings_name": r.settings_name,
"score": r.score.__dict__,
"lines": r.output_lines,
"line_metadata": r.line_metadata,
}, indent=2))
_write_per_fixture_md(per_dir, r)
def _write_per_fixture_md(out_dir: Path, r: FixtureRunResult) -> None:
lines = [
f"# {r.fixture_id} — {r.settings_name}",
"",
f"Stop reason: **{r.score.stop_reason}** after {r.score.iterations_used} iterations.",
f"Pass@±2: **{r.score.pct_pass_at_2:.0%}** ({sum(1 for v in r.line_metadata if v['min_delta']<=2)}/{r.score.line_count})",
"",
"| # | Original target | Candidate | Δ | Severity |",
"|---|------------------|-----------|---|----------|",
]
for v in r.line_metadata:
lines.append(
f"| {v['line_index']+1} | {_escape(v['target_text'])} | "
f"{_escape(v['candidate_text'])} | {v['min_delta']} | {v['severity']} |"
)
lines += [
"",
"## Your rating",
"",
"Add your qualitative notes here:",
"- [ ] Singable end-to-end",
"- [ ] Names placed naturally",
"- [ ] Stress patterns reasonable",
"- Notes:",
"",
]
(out_dir / "output.md").write_text("\n".join(lines))
def _escape(s: str) -> str:
return s.replace("|", "\\|").replace("\n", " ")
def write_baseline(results: list[FixtureRunResult], path: Path) -> None:
out: dict = {}
for r in results:
bucket = out.setdefault(r.fixture_id, {})
bucket[r.settings_name] = {
"pct_pass_at_2": r.score.pct_pass_at_2,
"pct_pass_at_0": r.score.pct_pass_at_0,
"mean_delta": r.score.mean_delta,
"iterations_used": r.score.iterations_used,
"captured_at": datetime.now(timezone.utc).date().isoformat(),
}
path.write_text(json.dumps(out, indent=2))
@@ -1,2 +0,0 @@
*
!.gitignore
-86
View File
@@ -1,86 +0,0 @@
"""CLI: python -m backend.eval.custom_lyrics.run"""
from __future__ import annotations
import argparse
import json
import sys
from datetime import datetime, timezone
from pathlib import Path
from backend.eval.custom_lyrics.cache import LlmCallCache
from backend.eval.custom_lyrics.report import write_baseline, write_run_reports
from backend.eval.custom_lyrics.runner import run_fixture
from backend.services.custom_lyrics.service import CustomLyricsService
EVAL_ROOT = Path(__file__).parent
FIXTURES_DIR = EVAL_ROOT / "fixtures"
RESULTS_DIR = EVAL_ROOT / "results"
CACHE_DIR = EVAL_ROOT / "cache"
BASELINE_PATH = EVAL_ROOT / "baseline.json"
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--fixtures", default="all", help="fixture id, or comma-separated, or 'all'")
parser.add_argument("--settings", default=None, help="settings name from metadata.settings_to_test")
parser.add_argument("--no-cache", action="store_true")
parser.add_argument("--replay-only", action="store_true")
parser.add_argument("--save-as-baseline", default=None,
help="path to write baseline JSON (e.g., baseline-post.json)")
args = parser.parse_args()
cache = LlmCallCache(
CACHE_DIR,
replay_only=args.replay_only,
) if not args.no_cache else _NullCache()
fixture_dirs = _select_fixtures(args.fixtures)
service = CustomLyricsService()
results = []
for fdir in fixture_dirs:
meta = json.loads((fdir / "metadata.json").read_text())
candidate_settings = meta.get("settings_to_test", [{
"name": "default",
"allow_reword": True,
"allow_omit": True,
"fixed_line_count": True,
"strictness": "balanced",
}])
for s in candidate_settings:
if args.settings and s["name"] != args.settings:
continue
settings_dict = {k: v for k, v in s.items() if k != "name"}
print(f"Running {fdir.name} / {s['name']}...", file=sys.stderr)
results.append(run_fixture(
fdir, s["name"], settings_dict,
cache=cache, service=service,
))
ts = datetime.now(timezone.utc).strftime("%Y-%m-%d-%H%M")
out_dir = RESULTS_DIR / ts
baseline = json.loads(BASELINE_PATH.read_text()) if BASELINE_PATH.exists() else None
write_run_reports(results, out_dir, baseline=baseline)
if args.save_as_baseline:
write_baseline(results, EVAL_ROOT / args.save_as_baseline)
print(f"Baseline saved to {args.save_as_baseline}", file=sys.stderr)
print(f"\nReport: {out_dir / 'summary.md'}", file=sys.stderr)
return 0
def _select_fixtures(spec: str) -> list[Path]:
if spec == "all":
return sorted(p for p in FIXTURES_DIR.iterdir() if p.is_dir() and (p / "metadata.json").exists())
ids = [s.strip() for s in spec.split(",")]
return [FIXTURES_DIR / i for i in ids]
class _NullCache:
def get(self, *args, **kwargs): return None
def set(self, *args, **kwargs): pass
if __name__ == "__main__":
sys.exit(main())
-112
View File
@@ -1,112 +0,0 @@
"""Eval runner: load fixture → call service (cached) → score → return result."""
from __future__ import annotations
import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from unittest.mock import patch
from backend.eval.custom_lyrics.cache import LlmCallCache
from backend.eval.custom_lyrics.scorer import FixtureScore, aggregate_fixture
from backend.services.custom_lyrics.service import CustomLyricsService
from backend.services.custom_lyrics.settings import (
GenerationSettings,
settings_from_dict,
)
@dataclass
class FixtureRunResult:
fixture_id: str
settings_name: str
score: FixtureScore
output_lines: list[str]
line_metadata: list[dict[str, Any]]
fixture_dir: Path
def load_fixture(fixture_dir: Path) -> dict:
metadata = json.loads((fixture_dir / "metadata.json").read_text())
metadata["original_lyrics"] = (fixture_dir / "original_lyrics.txt").read_text().strip().splitlines()
metadata["original_segments"] = json.loads((fixture_dir / "original_segments.json").read_text())
metadata["client_input"] = (fixture_dir / "client_input.txt").read_text()
notes_path = fixture_dir / "notes.txt"
metadata["notes"] = notes_path.read_text() if notes_path.exists() else None
return metadata
def run_fixture(
fixture_dir: Path,
settings_name: str,
settings_dict: dict[str, Any],
*,
cache: LlmCallCache,
service: CustomLyricsService,
) -> FixtureRunResult:
fixture = load_fixture(fixture_dir)
settings = settings_from_dict(settings_dict)
gemini_call_count = 0
real_call = service._call_gemini
def cached_call(*, system_prompt: str, user_prompt: str, pdf_bytes, settings: GenerationSettings):
nonlocal gemini_call_count
cached = cache.get(
system_prompt, user_prompt,
model=service.settings.custom_lyrics_model,
settings_dict=settings.to_dict(),
)
if cached is not None:
return cached
gemini_call_count += 1
result = real_call(
system_prompt=system_prompt, user_prompt=user_prompt,
pdf_bytes=pdf_bytes, settings=settings,
)
cache.set(
system_prompt, user_prompt,
model=service.settings.custom_lyrics_model,
settings_dict=settings.to_dict(),
response_lines=result,
)
return result
with patch.object(service, "_call_gemini", side_effect=cached_call):
result = service.generate(
job_id=fixture["id"],
target_lines=fixture["original_lyrics"],
target_segments=fixture["original_segments"],
artist=fixture.get("artist"),
title=fixture.get("title"),
custom_text=fixture["client_input"],
file_bytes=None, file_mime=None, file_name=None,
notes=fixture.get("notes"),
settings=settings,
)
score = aggregate_fixture(
fixture_id=fixture["id"],
settings_name=settings_name,
metadata=result.line_metadata,
iterations_used=result.iterations_used,
stop_reason=result.stop_reason.value,
duration_ms=result.duration_ms,
gemini_calls=gemini_call_count,
line_count_match=not result.line_count_mismatch,
)
return FixtureRunResult(
fixture_id=fixture["id"],
settings_name=settings_name,
score=score,
output_lines=result.lines,
line_metadata=[{
"line_index": v.line_index,
"target_text": v.target_text,
"candidate_text": v.candidate_text,
"min_delta": v.min_delta,
"severity": v.severity.value,
} for v in result.line_metadata],
fixture_dir=fixture_dir,
)
-113
View File
@@ -1,113 +0,0 @@
"""Pure scoring functions for custom-lyrics eval."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Iterable
from backend.services.custom_lyrics.validator import LineValidation
@dataclass
class PerLineScore:
line_index: int
min_delta: int
pass_at_0: bool
pass_at_1: bool
pass_at_2: bool
pass_at_4: bool
severity: str
@dataclass
class FixtureScore:
fixture_id: str
settings_name: str
line_count: int
pct_pass_at_2: float
pct_pass_at_1: float
pct_pass_at_0: float
pct_pass_at_4: float
mean_delta: float
median_delta: float
max_delta: int
iterations_used: int
stop_reason: str
duration_ms: int
gemini_calls: int
line_count_match: bool
severity_breakdown: dict[str, int]
def score_per_line(val: LineValidation) -> PerLineScore:
return PerLineScore(
line_index=val.line_index,
min_delta=val.min_delta,
pass_at_0=val.min_delta <= 0,
pass_at_1=val.min_delta <= 1,
pass_at_2=val.min_delta <= 2,
pass_at_4=val.min_delta <= 4,
severity=val.severity.value,
)
def aggregate_fixture(
*,
fixture_id: str,
settings_name: str,
metadata: list[LineValidation],
iterations_used: int,
stop_reason: str,
duration_ms: int,
gemini_calls: int,
line_count_match: bool,
) -> FixtureScore:
n = len(metadata)
if n == 0:
return FixtureScore(
fixture_id=fixture_id, settings_name=settings_name,
line_count=0,
pct_pass_at_2=0, pct_pass_at_1=0, pct_pass_at_0=0, pct_pass_at_4=0,
mean_delta=0, median_delta=0, max_delta=0,
iterations_used=iterations_used, stop_reason=stop_reason,
duration_ms=duration_ms, gemini_calls=gemini_calls,
line_count_match=line_count_match,
severity_breakdown={"ok": 0, "minor": 0, "major": 0},
)
deltas = sorted(v.min_delta for v in metadata)
severity_breakdown = {"ok": 0, "minor": 0, "major": 0}
for v in metadata:
severity_breakdown[v.severity.value] += 1
return FixtureScore(
fixture_id=fixture_id,
settings_name=settings_name,
line_count=n,
pct_pass_at_2=sum(1 for d in deltas if d <= 2) / n,
pct_pass_at_1=sum(1 for d in deltas if d <= 1) / n,
pct_pass_at_0=sum(1 for d in deltas if d <= 0) / n,
pct_pass_at_4=sum(1 for d in deltas if d <= 4) / n,
mean_delta=sum(deltas) / n,
median_delta=deltas[n // 2],
max_delta=deltas[-1],
iterations_used=iterations_used,
stop_reason=stop_reason,
duration_ms=duration_ms,
gemini_calls=gemini_calls,
line_count_match=line_count_match,
severity_breakdown=severity_breakdown,
)
def aggregate_corpus(per_fixture: Iterable[FixtureScore]) -> dict:
"""Macro-averaged corpus aggregate."""
fixtures = list(per_fixture)
if not fixtures:
return {}
return {
"fixture_count": len(fixtures),
"macro_pct_pass_at_2": sum(f.pct_pass_at_2 for f in fixtures) / len(fixtures),
"macro_pct_pass_at_1": sum(f.pct_pass_at_1 for f in fixtures) / len(fixtures),
"macro_pct_pass_at_0": sum(f.pct_pass_at_0 for f in fixtures) / len(fixtures),
"macro_pct_pass_at_4": sum(f.pct_pass_at_4 for f in fixtures) / len(fixtures),
"macro_mean_delta": sum(f.mean_delta for f in fixtures) / len(fixtures),
"total_gemini_calls": sum(f.gemini_calls for f in fixtures),
}
Whitespace-only changes.
@@ -1,38 +0,0 @@
"""Tests for the LLM-call disk cache."""
from __future__ import annotations
import tempfile
from pathlib import Path
from backend.eval.custom_lyrics.cache import LlmCallCache
def test_miss_returns_none() -> None:
with tempfile.TemporaryDirectory() as tmp:
c = LlmCallCache(Path(tmp))
assert c.get("system", "user", model="m1", settings_dict={"x": 1}) is None
def test_set_then_get_hits() -> None:
with tempfile.TemporaryDirectory() as tmp:
c = LlmCallCache(Path(tmp))
c.set("system", "user", model="m1", settings_dict={"x": 1}, response_lines=["a", "b"])
assert c.get("system", "user", model="m1", settings_dict={"x": 1}) == ["a", "b"]
def test_different_settings_different_keys() -> None:
with tempfile.TemporaryDirectory() as tmp:
c = LlmCallCache(Path(tmp))
c.set("system", "user", model="m1", settings_dict={"x": 1}, response_lines=["a"])
assert c.get("system", "user", model="m1", settings_dict={"x": 2}) is None
def test_replay_only_raises_on_miss() -> None:
with tempfile.TemporaryDirectory() as tmp:
c = LlmCallCache(Path(tmp), replay_only=True)
try:
c.get("system", "user", model="m1", settings_dict={"x": 1})
except RuntimeError as e:
assert "cache miss" in str(e).lower()
else:
raise AssertionError("expected RuntimeError")
@@ -1,66 +0,0 @@
"""Tests for eval scorer."""
from __future__ import annotations
import pytest
from backend.eval.custom_lyrics.scorer import (
PerLineScore,
FixtureScore,
score_per_line,
aggregate_fixture,
)
from backend.services.custom_lyrics.validator import LineValidation, Severity
def _line_val(idx: int, target: list[int], cand: list[int], passes: bool, severity: Severity, delta: int) -> LineValidation:
return LineValidation(
line_index=idx,
target_text=f"t{idx}",
candidate_text=f"c{idx}",
target_syllables=target,
candidate_syllables=cand,
min_delta=delta,
passes=passes,
severity=severity,
time_budget_seconds=1.0,
)
def test_score_per_line_pass_fail_thresholds() -> None:
val = _line_val(0, [5, 5, 5, 5], [6, 6, 6, 6], passes=False, severity=Severity.MINOR, delta=1)
s = score_per_line(val)
assert s.pass_at_0 is False
assert s.pass_at_1 is True
assert s.pass_at_2 is True
assert s.pass_at_4 is True
assert s.min_delta == 1
def test_score_per_line_major_violation() -> None:
val = _line_val(0, [5, 5, 5, 5], [12, 12, 12, 12], passes=False, severity=Severity.MAJOR, delta=7)
s = score_per_line(val)
assert s.pass_at_0 is False
assert s.pass_at_4 is False
assert s.severity == "major"
def test_aggregate_fixture_basic() -> None:
metadata = [
_line_val(0, [5]*4, [5]*4, passes=True, severity=Severity.OK, delta=0),
_line_val(1, [5]*4, [6]*4, passes=False, severity=Severity.MINOR, delta=1),
_line_val(2, [5]*4, [9]*4, passes=False, severity=Severity.MAJOR, delta=4),
]
fixture = aggregate_fixture(
fixture_id="x",
settings_name="default",
metadata=metadata,
iterations_used=2,
stop_reason="max_iters_reached",
duration_ms=1500,
gemini_calls=3,
line_count_match=True,
)
assert fixture.pct_pass_at_2 == pytest.approx(2 / 3)
assert fixture.pct_pass_at_0 == pytest.approx(1 / 3)
assert fixture.mean_delta == pytest.approx((0 + 1 + 4) / 3)
assert fixture.max_delta == 4
+31 -59
View File
@@ -2,10 +2,40 @@
Custom exceptions for the karaoke generation backend.
These exceptions are used for structured error handling across the application,
particularly for validation errors and credit checks.
particularly for rate limiting and validation errors.
"""
class RateLimitExceededError(Exception):
"""
Raised when a rate limit is exceeded.
Includes information about when the limit will reset for client retry logic.
Attributes:
message: Human-readable error message
limit_type: Type of limit exceeded (e.g., "jobs_per_day", "youtube_uploads")
remaining_seconds: Seconds until the limit resets (for Retry-After header)
current_count: Current usage count
limit_value: The limit that was exceeded
"""
def __init__(
self,
message: str,
limit_type: str = "unknown",
remaining_seconds: int = 0,
current_count: int = 0,
limit_value: int = 0
):
self.message = message
self.limit_type = limit_type
self.remaining_seconds = remaining_seconds
self.current_count = current_count
self.limit_value = limit_value
super().__init__(message)
class EmailValidationError(Exception):
"""
Raised when email validation fails.
@@ -34,61 +64,3 @@ class IPBlockedError(Exception):
self.message = message
self.ip_address = ip_address
super().__init__(message)
class InsufficientCreditsError(Exception):
"""
Raised when a user attempts to create a job without sufficient credits.
Attributes:
message: Human-readable error message
credits_available: Number of credits the user currently has
credits_required: Number of credits required (always 1 for now)
"""
def __init__(
self,
message: str,
credits_available: int = 0,
credits_required: int = 1
):
self.message = message
self.credits_available = credits_available
self.credits_required = credits_required
super().__init__(message)
class InvalidStateTransitionError(Exception):
"""
Raised when an invalid job state transition is attempted.
This exception is raised by JobManager.transition_to_state() when the
requested transition is not allowed by the state machine defined in
STATE_TRANSITIONS.
Example: Trying to transition from PENDING to GENERATING_SCREENS is invalid
because PENDING can only transition to DOWNLOADING, SEARCHING_AUDIO, FAILED,
or CANCELLED.
Attributes:
message: Human-readable error message
job_id: The job ID that failed to transition
from_status: Current status of the job
to_status: Attempted target status
valid_transitions: List of valid transitions from current status
"""
def __init__(
self,
message: str,
job_id: str = "",
from_status: str = "",
to_status: str = "",
valid_transitions: list = None
):
self.message = message
self.job_id = job_id
self.from_status = from_status
self.to_status = to_status
self.valid_transitions = valid_transitions or []
super().__init__(message)
-144
View File
@@ -1,144 +0,0 @@
"""
Lightweight i18n utility for backend translations.
Usage:
from backend.i18n import t, get_locale_from_request, SUPPORTED_LOCALES
# In a route handler:
locale = get_locale_from_request(request)
message = t(locale, "auth.userNotFound")
message = t(locale, "jobs.notFound", job_id="abc123")
"""
import json
from functools import lru_cache
from pathlib import Path
from typing import Any
from starlette.requests import Request
SUPPORTED_LOCALES = ("en", "es", "de")
# Every locale the web app serves (frontend/i18n/routing.ts, frontend/messages/*.json;
# a test keeps them in sync). Use for web URL prefixes; SUPPORTED_LOCALES is the
# backend's own (email / API message) translations.
WEB_LOCALES = (
"en", "es", "de", "pt", "fr", "ja", "ko", "zh", "it", "nl",
"pl", "tr", "ru", "th", "id", "vi", "tl", "hi", "ar", "sv",
"nb", "da", "fi", "cs", "ro", "hu", "el", "he", "ms", "uk",
"hr", "sk", "ca",
)
DEFAULT_LOCALE = "en"
TRANSLATIONS_DIR = Path(__file__).parent / "translations"
@lru_cache(maxsize=len(SUPPORTED_LOCALES))
def _load_translations(locale: str) -> dict[str, Any]:
"""Load and cache translation file for a locale."""
path = TRANSLATIONS_DIR / f"{locale}.json"
if not path.exists():
if locale != DEFAULT_LOCALE:
return _load_translations(DEFAULT_LOCALE)
return {}
with open(path, encoding="utf-8") as f:
result: dict[str, Any] = json.load(f)
return result
def t(locale: str, key: str, **kwargs: Any) -> str:
"""Get a translated string by dotted key path.
Args:
locale: Language code (en, es, de)
key: Dotted path like "auth.userNotFound" or "emails.magicLink.subject"
**kwargs: Interpolation variables (e.g., job_id="abc123")
Returns:
Translated string with variables interpolated, or the key itself if not found.
"""
if locale not in SUPPORTED_LOCALES:
locale = DEFAULT_LOCALE
translations = _load_translations(locale)
# Walk the dotted path
value: Any = translations
for part in key.split("."):
if isinstance(value, dict) and part in value:
value = value[part]
else:
# Key not found — fall back to English
if locale != DEFAULT_LOCALE:
return t(DEFAULT_LOCALE, key, **kwargs)
return key # Return the key itself as last resort
if not isinstance(value, str):
return key
result: str = value
# Interpolate variables
if kwargs:
try:
result = result.format(**kwargs)
except KeyError:
pass # Missing variable — return template as-is
return result
def get_locale_from_request(request: Request) -> str:
"""Extract locale from Accept-Language header.
Returns the first supported locale found, or DEFAULT_LOCALE.
"""
accept_lang = request.headers.get("accept-language", "")
# Simple parsing — Accept-Language can be "de", "de-DE", "en-US,en;q=0.9,de;q=0.8"
for part in accept_lang.split(","):
lang = part.split(";")[0].strip().split("-")[0].lower()
if lang in SUPPORTED_LOCALES:
return lang
return DEFAULT_LOCALE
def get_full_locale_from_request(request: Request) -> str | None:
"""Extract the user's actual UI language tag from Accept-Language.
Unlike :func:`get_locale_from_request`, this does NOT narrow to the 3
email-supported locales (en/es/de). It returns the primary language subtag
of the first entry in the header (e.g. ``pt``, ``ja``, ``zh``) so the admin
UI can show the real language a customer was using the product in.
Returns the lowercased language subtag, or ``None`` if the header is absent
or unparseable.
"""
accept_lang = request.headers.get("accept-language", "")
for part in accept_lang.split(","):
# Each part looks like "en-US;q=0.9" — take the language-region token
# before the quality factor, then the primary subtag before any region.
segments = part.split(";")
token = segments[0].strip()
if not token or token == "*":
continue
# Skip ranges the client explicitly rejects with q=0.
if any(seg.strip().lower().replace(" ", "") in ("q=0", "q=0.0", "q=0.00", "q=0.000")
for seg in segments[1:]):
continue
lang = token.split("-")[0].strip().lower()
if lang.isalpha() and len(lang) >= 2:
return lang
return None
def get_web_locale_prefix(locale: str) -> str:
"""Web URL locale prefix for any of the web app's locales (fallback '/en')."""
return f"/{locale if locale in WEB_LOCALES else DEFAULT_LOCALE}"
def get_locale_prefix(locale: str) -> str:
"""Get URL locale prefix for frontend URLs (e.g., '/en', '/es')."""
if locale not in SUPPORTED_LOCALES:
locale = DEFAULT_LOCALE
return f"/{locale}"
+35 -315
View File
@@ -1,21 +1,13 @@
"""
FastAPI application entry point for karaoke generation backend.
(Trivial comment touch to trigger backend CI on ephemeral runners after
the 2026-05-17 dispatcher e2 fix; safe to remove on next backend edit.)
"""
import asyncio
import logging
import threading
import time
from contextlib import asynccontextmanager
from typing import Any
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from backend.config import settings, validate_production_config
from backend.services.fake_workers import fake_workers_enabled
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, tenant_admin, tenant_bulk, tenant_theme, rate_limits, push, catalog, encoding_worker, client_errors, client_events, bulk, parse_titles
from backend.config import settings
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, rate_limits, push
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
from backend.services.structured_logging import setup_structured_logging
from backend.services.spacy_preloader import preload_spacy_model
@@ -23,7 +15,6 @@ from backend.services.nltk_preloader import preload_all_nltk_resources
from backend.services.langfuse_preloader import preload_langfuse_handler
from backend.middleware.audit_logging import AuditLoggingMiddleware
from backend.middleware.tenant import TenantMiddleware
from backend.middleware.edge_auth import EdgeAuthMiddleware
from backend.workers.registry import worker_registry
@@ -43,7 +34,7 @@ tracing_enabled = setup_tracing(
)
def validate_credentials_on_startup():
async def validate_credentials_on_startup():
"""Validate OAuth credentials on startup and send alerts if needed."""
try:
from backend.services.credential_manager import get_credential_manager, CredentialStatus
@@ -73,122 +64,17 @@ def validate_credentials_on_startup():
logger.error(f"Failed to validate credentials on startup: {e}")
# Modules that route handlers import lazily. On a fresh instance the first such
# import ran ON the event loop: httpx/httpcore (via flacfetch_client /
# worker_service) froze it 3-15s on cold disk reads (EVENT_LOOP_STALL in
# importlib, 2026-10-08). Lifespan startup imports them in a worker thread before
# readiness — a background thread alone would still race the first request.
_LAZY_ROUTE_IMPORTS = (
"httpx",
# Imported lazily by httpx / worker_service on first use; still froze the loop
# after the list above existed (httpcore ~1s; google.cloud.run_v2 14.9s with
# 7 requests in flight, 2026-10-16).
"httpcore",
"google.cloud.run_v2",
"backend.services.worker_service",
"backend.services.flacfetch_client",
"backend.services.catalog_proxy_service",
"backend.services.ip_geolocation_service",
"backend.services.email_validation_service",
"backend.services.musicbrainz_service",
# audio_search's auto_download path: select_best -> flacfetch.core.manager ->
# yt_dlp import froze the loop ~11-12s on fresh instances (2026-10-23, 10-26).
"flacfetch.core.manager",
"flacfetch.downloaders.youtube",
"yt_dlp",
# starlette/middleware/base.py picks the anyio backend on the first request:
# 17.3s on the loop with 18 requests in flight (2026-10-30).
"anyio._backends._asyncio",
)
@asynccontextmanager
async def lifespan(app: FastAPI):
"""Lifespan event handler for startup and shutdown."""
# Startup
logger.info("Starting karaoke generation backend")
logger.info(f"Environment: {settings.environment}")
logger.info(f"GCS Bucket: {settings.gcs_bucket_name}")
logger.info(f"Tracing enabled: {tracing_enabled}")
LAZY_ROUTE_IMPORT_TIMEOUT_S = 30
def _import_lazy_route_modules() -> None:
import importlib
for module in _LAZY_ROUTE_IMPORTS:
try:
importlib.import_module(module)
except Exception as e:
logger.warning(f"Startup import of {module} failed (will import lazily): {e}")
def _warm_route_contexts(application: FastAPI) -> int:
"""Build FastAPI's per-route contexts (path regexes, pydantic schemas) up front.
Newer FastAPI builds them lazily on the first request, and the OpenTelemetry
FastAPI instrumentation walks every route (``_get_route_details`` ->
``iter_route_contexts``) for each request's span name. On a fresh instance that
first walk compiled hundreds of route regexes ON the event loop (2026-10-23: 11s,
3 requests; 5.1s on 10-22). The results are cached (by routes version, under a
lock), so one walk here, before readiness, covers every later request.
Best-effort: a FastAPI version without these internals just skips it.
Returns the number of route contexts built (0 if unsupported).
"""
try:
from fastapi.routing import iter_route_contexts
except ImportError: # older FastAPI compiles routes eagerly at include time
return 0
count = 0
for _ in iter_route_contexts(application.routes):
count += 1
for route in application.routes:
low_priority = getattr(route, "effective_low_priority_routes", None)
if callable(low_priority):
count += len(low_priority())
return count
def _warm_middleware_stack(application: Any) -> None:
"""Send one request through the app, so the first user's request isn't what builds
the middleware stack and runs its first-use imports on the loop.
Blocking: runs its own event loop, so call it from a worker thread. /api/health is
exempt from the edge lock, skips the audit log and touches no storage. Best-effort.
"""
import httpx
async def _request() -> None:
transport = httpx.ASGITransport(app=application)
async with httpx.AsyncClient(transport=transport, base_url="http://localhost") as client:
await client.get("/api/health")
start = time.time()
try:
asyncio.run(_request())
except Exception as e: # noqa: BLE001 - warmup must never block startup
logger.warning(f"Middleware warm request failed (stack builds on first request): {e}")
return
logger.info(f"Warmed middleware stack in {time.time() - start:.2f}s")
def _warm_route_contexts_safely(application: FastAPI) -> None:
start = time.time()
try:
built = _warm_route_contexts(application)
except Exception as e: # noqa: BLE001 - warmup must never block startup
logger.warning(f"Route-context warmup failed (routes build lazily): {e}")
return
if built:
logger.info(f"Warmed {built} route contexts in {time.time() - start:.2f}s")
def _run_background_warmup():
"""Warm caches that used to block startup (runs in a daemon thread).
Historically these preloads ran inline in lifespan startup, which meant
Cloud Run held every routed request for the full ~7.5s they take (spaCy
~1s, NLTK ~2.5s, Langfuse ~3s, credential checks ~1.5s) on top of import
time. None of them are needed to serve HTTP: they exist to make the FIRST
lyrics-processing job fast (see docs/archive/2026-01-08-performance-
investigation.md). Running them in a background thread right after boot
keeps that warm-cache property (they finish within seconds, long before
any real job runs) without gating readiness. Workers that race the warmup
simply fall back to the preloaders' lazy paths.
"""
warmup_start = time.time()
# Preload NLP models and resources to avoid cold start delays
# See docs/archive/2026-01-08-performance-investigation.md for background
# 1. SpaCy model (60+ second delay without preload)
try:
@@ -208,159 +94,27 @@ def _run_background_warmup():
except Exception as e:
logger.warning(f"Langfuse preload failed (will initialize lazily): {e}")
if fake_workers_enabled(settings):
# Local stack: no encoding VM and no OAuth credentials to validate.
logger.info(f"Background warmup complete in {time.time() - warmup_start:.2f}s (fake workers)")
return
# 4. Encoding service singleton — its first construction imports
# google.cloud.compute_v1 (~10s on a fresh instance). Build it here, off the
# event loop, before /api/health/detailed or a preview encode needs it.
# Validate OAuth credentials (non-blocking)
try:
from backend.services.encoding_service import get_encoding_service
get_encoding_service()
except Exception as e:
logger.warning(f"Encoding service warmup failed (will build lazily): {e}")
# Validate OAuth credentials (alerting only, nothing depends on it)
try:
validate_credentials_on_startup()
await validate_credentials_on_startup()
except Exception as e:
logger.error(f"Credential validation failed: {e}")
logger.info(f"Background warmup complete in {time.time() - warmup_start:.2f}s")
@asynccontextmanager
async def lifespan(app: FastAPI):
"""Lifespan event handler for startup and shutdown."""
# Startup
logger.info("Starting karaoke generation backend")
logger.info(f"Environment: {settings.environment}")
logger.info(f"GCS Bucket: {settings.gcs_bucket_name}")
logger.info(f"Tracing enabled: {tracing_enabled}")
# Fail fast if required production config is missing, rather than silently
# running on dev defaults (wrong GCP project / GCS bucket / localhost worker
# URL). No-op outside production. (Fallback audit 2026-06-09, Theme 7.)
validate_production_config()
# Off the loop, but before readiness (Cloud Run holds requests meanwhile).
# Bounded so a hung cold-disk read can't keep the instance from ever going
# ready; the import keeps running in its thread and routes fall back to it.
try:
await asyncio.wait_for(
asyncio.to_thread(_import_lazy_route_modules), timeout=LAZY_ROUTE_IMPORT_TIMEOUT_S
)
except asyncio.TimeoutError:
logger.error(
f"Startup import of lazy route modules exceeded {LAZY_ROUTE_IMPORT_TIMEOUT_S}s; "
"continuing startup (first requests may import them on the event loop)"
)
# Same reason: build the route contexts in a thread, before the first request
# (and the OTel route walk) would build them on the loop.
try:
await asyncio.wait_for(
asyncio.to_thread(_warm_route_contexts_safely, app), timeout=LAZY_ROUTE_IMPORT_TIMEOUT_S
)
except asyncio.TimeoutError:
logger.warning(
f"Route-context warmup exceeded {LAZY_ROUTE_IMPORT_TIMEOUT_S}s; continuing startup"
)
# Same reason, for what only the first request through the middleware touches.
try:
await asyncio.wait_for(
asyncio.to_thread(_warm_middleware_stack, app), timeout=LAZY_ROUTE_IMPORT_TIMEOUT_S
)
except asyncio.TimeoutError:
logger.warning(
f"Middleware warmup exceeded {LAZY_ROUTE_IMPORT_TIMEOUT_S}s; continuing startup"
)
# NLP model / credential warmup runs in the background so it doesn't gate
# readiness — Cloud Run holds all routed requests until lifespan startup
# returns, and these preloads were ~7.5s of the ~16s cold start.
threading.Thread(
target=_run_background_warmup, name="startup-warmup", daemon=True
).start()
# asyncio.to_thread's default executor is min(32, cpu+4) ≈ 6 threads on our
# 2 vCPUs. The screens worker, stale-review cron, auto-correct and many
# review endpoints all offload into it (mostly I/O-bound), so size it up —
# otherwise review-page requests queue behind a few long-running jobs.
import concurrent.futures
asyncio.get_running_loop().set_default_executor(
concurrent.futures.ThreadPoolExecutor(max_workers=48, thread_name_prefix="to-thread")
)
# Event-loop stall watchdog: makes "every request froze for 30s" episodes
# self-diagnosing (logs the blocking stack; records >=5s stalls in prod).
watchdog = None
if settings.loop_watchdog_enabled:
from backend.services.loop_watchdog import (
LoopWatchdog,
firestore_stall_recorder,
request_tracker,
)
watchdog = LoopWatchdog(
recorder=firestore_stall_recorder if settings.environment == "production" else None,
# Stalls nobody waited on (idle, CPU-throttled autoscaled instances)
# stay WARNING so they don't trip the >=10s stall alert.
request_tracker=request_tracker,
)
watchdog.start()
yield
if watchdog is not None:
await watchdog.stop()
# Shutdown - best-effort parking of any still-registered workers.
#
# Reality check (incident 2026-09-13, job 41e06b90): Cloud Run services
# SIGKILL instances ~10 seconds after SIGTERM — NOT the 600s this comment
# previously claimed — and uvicorn only runs this lifespan-shutdown code
# AFTER all BackgroundTasks have completed ("Waiting for background tasks
# to complete"). A long-running worker task therefore keeps uvicorn stuck
# in that wait until SIGKILL, and this hook never executes at all. It can
# NOT be relied on to protect in-flight renders; that protection comes
# from running them as Cloud Run Jobs instead (USE_CLOUD_RUN_JOBS_FOR_RENDER,
# mirroring USE_CLOUD_RUN_JOBS_FOR_VIDEO from incident 2026-03-08).
#
# This hook is kept as a cheap safety net for the legacy flag-off path and
# for the narrow case where workers finish right as SIGTERM lands. The wait
# is capped well inside the 10s kill window so the parking pass below still
# gets a chance to write Firestore state.
# Shutdown - wait for any active workers to complete before terminating
# This prevents Cloud Run from killing workers mid-processing
logger.info("Shutdown requested, checking for active workers...")
if worker_registry.has_active_workers():
active = worker_registry.get_active_workers()
logger.info(f"Active workers found: {active}")
logger.info("Waiting for workers to complete (timeout: 5s)...")
completed = await worker_registry.wait_for_completion(timeout=5)
logger.info("Waiting for workers to complete (timeout: 600s)...")
completed = await worker_registry.wait_for_completion(timeout=600) # 10 min max
if not completed:
logger.error(
"Shutdown timeout - some workers may not have completed cleanly. "
f"Remaining workers: {worker_registry.get_active_workers()}"
)
# Park any still-active render jobs so the auto-retry scheduler can
# recover them. Safe to call unconditionally — it's a no-op if nothing
# render-related is in flight.
try:
from backend.workers.render_video_worker import park_active_render_jobs_for_shutdown
parked = park_active_render_jobs_for_shutdown()
if parked:
logger.warning(
f"Parked {parked} in-flight render job(s) for auto-retry "
f"due to shutdown"
)
except Exception as exc:
logger.error(f"Render-job shutdown park failed: {exc!r}")
else:
logger.info("No active workers, proceeding with shutdown")
@@ -382,8 +136,8 @@ if tracing_enabled:
# Configure CORS
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=False,
allow_origins=["*"], # Configure this properly for production
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
@@ -394,76 +148,42 @@ app.add_middleware(AuditLoggingMiddleware)
# Add tenant detection middleware (extracts tenant from subdomain/headers)
app.add_middleware(TenantMiddleware)
# Edge origin lock (added last = outermost, runs first). Rejects direct-to-origin
# requests that bypassed the Cloudflare edge, before tenant/audit middleware do
# any work. No-op unless EDGE_AUTH_MODE is warn/enforce. See
# backend/middleware/edge_auth.py.
app.add_middleware(EdgeAuthMiddleware)
# Counts in-flight requests for the loop watchdog (outermost, so every request that
# waited on a stalled loop counts — even ones the edge lock rejects).
from backend.services.loop_watchdog import RequestTrackingMiddleware # noqa: E402
app.add_middleware(RequestTrackingMiddleware)
# Include routers
app.include_router(health.router, prefix="/api")
app.include_router(jobs.router, prefix="/api")
app.include_router(file_upload.router, prefix="/api") # File upload endpoint
app.include_router(internal.router, prefix="/api") # Internal worker endpoints
app.include_router(encoding_worker.router, prefix="/api") # Encoding worker lifecycle
app.include_router(review.router, prefix="/api") # Review UI compatibility endpoints
app.include_router(auth.router, prefix="/api") # OAuth credential management
app.include_router(audio_search.router, prefix="/api") # Audio search (artist+title mode)
app.include_router(themes.router, prefix="/api") # Theme selection for styles
app.include_router(users.router, prefix="/api") # User auth, credits, and Stripe webhooks
from backend.api.routes import referrals
app.include_router(referrals.router, prefix="/api") # Referral links and payouts
app.include_router(admin.router, prefix="/api") # Admin dashboard and management
app.include_router(rate_limits.router, prefix="/api") # Rate limits admin management
app.include_router(push.router, prefix="/api") # Push notification subscription management
app.include_router(catalog.router, prefix="/api") # Catalog proxy for song/artist autocomplete
app.include_router(parse_titles.router, prefix="/api") # kjbox karaoke-filename parser
app.include_router(bulk.router, prefix="/api") # Bulk Mode: multi-job submission
from backend.api.routes import requests_board
app.include_router(requests_board.router, prefix="/api") # Public song-request voting board
app.include_router(client_errors.router, prefix="/api") # Frontend crash reports
app.include_router(client_events.router, prefix="/api") # Frontend degradation telemetry (banner/unavailable/waveform events)
from backend.api.routes import karaokehunt
app.include_router(karaokehunt.router, prefix="/api") # Retired KaraokeHunt app request interceptor
from backend.api.routes import kjbox
app.include_router(kjbox.router, prefix="/api") # kjbox partner API: singer email-code sign-in + show credits
from backend.api.routes import login_code
app.include_router(login_code.router, prefix="/api") # gen email-code sign-in (/users/auth/send-code, verify-code)
app.include_router(tenant.router) # Tenant/white-label configuration (no /api prefix, router has it)
app.include_router(tenant_admin.router) # Admin tenant provisioning (router has /api prefix)
app.include_router(tenant_bulk.router) # Tenant bulk-upload filename analysis (router has /api prefix)
app.include_router(tenant_theme.router) # Tenant self-service theme editor (router has /api prefix)
if fake_workers_enabled(settings):
# Local E2E stack only (startup refuses FAKE_WORKERS in production).
from backend.api.routes import test_support
app.include_router(test_support.router, prefix="/api")
# Exception handlers
# Exception handler for rate limiting
from fastapi import Request
from fastapi.responses import JSONResponse
from backend.exceptions import InsufficientCreditsError
from backend.i18n import t, get_locale_from_request
from backend.exceptions import RateLimitExceededError
@app.exception_handler(InsufficientCreditsError)
async def insufficient_credits_exception_handler(request: Request, exc: InsufficientCreditsError):
"""Handle insufficient credits errors with 402 Payment Required status."""
locale = get_locale_from_request(request)
@app.exception_handler(RateLimitExceededError)
async def rate_limit_exception_handler(request: Request, exc: RateLimitExceededError):
"""Handle rate limit exceeded errors with 429 status."""
return JSONResponse(
status_code=402,
status_code=429,
content={
"detail": t(locale, "errors.insufficientCredits"),
"credits_available": exc.credits_available,
"credits_required": exc.credits_required,
"buy_url": t(locale, "errors.insufficientCreditsBuyUrl"),
"detail": exc.message,
"limit_type": exc.limit_type,
"current_count": exc.current_count,
"limit_value": exc.limit_value,
},
headers={
"Retry-After": str(exc.remaining_seconds),
} if exc.remaining_seconds > 0 else None,
)
-115
View File
@@ -1,115 +0,0 @@
"""
Edge origin-lock middleware.
Cloudflare fronts ``api.nomadkaraoke.com`` (WAF + rate limiting). To stop
attackers from bypassing the edge by hitting the Cloud Run origin directly
(``*.run.app`` / ``ghs.googlehosted.com``), Cloudflare injects a shared-secret
header (``X-Edge-Auth``) on every proxied request via a Transform Rule. This
middleware rejects public requests that lack the correct header.
A Cloud Run *domain mapping* can't use ``ingress=internal-and-cloud-load-balancing``
(that needs a load balancer), so this app-layer check is how we lock the origin.
Modes (env ``EDGE_AUTH_MODE``, default ``off``):
off - passthrough, no checks (deploy-dormant; behaviour unchanged).
warn - log requests missing/failing the header, but ALLOW them. Use to
confirm nothing legit is missing the header before enforcing.
enforce - reject requests missing/failing the header with 403.
Secret: env ``EDGE_ORIGIN_SECRET`` (same value Cloudflare injects). If enforce
mode is set but the secret is unconfigured, the middleware FAILS CLOSED (503) —
a security control must not silently allow direct-origin traffic. (Low risk:
Cloud Run won't start a revision whose secret ref is missing, and enforce is
only enabled after staging confirms the secret.) In warn mode it still allows.
Exemptions (always allowed, even in enforce mode):
* Health / root endpoints and Cloud Run probes, which hit the origin
directly (not via Cloudflare) and legitimately lack the header.
"""
import hmac
import logging
import os
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
from starlette.responses import JSONResponse, Response
logger = logging.getLogger(__name__)
# Header Cloudflare injects (must match CloudflareConfig.ORIGIN_AUTH_HEADER in
# infrastructure/modules/edge_security.py).
EDGE_AUTH_HEADER = "X-Edge-Auth"
# Paths always allowed without the edge header. These are non-sensitive and are
# probed directly on the origin by Cloud Run startup/liveness checks and GCP
# uptime monitoring, which don't traverse Cloudflare.
_EXEMPT_EXACT = {"/", "/api/health", "/api/health/"}
_EXEMPT_PREFIXES = ("/api/health/",)
_VALID_MODES = {"off", "warn", "enforce"}
def _mode() -> str:
mode = os.environ.get("EDGE_AUTH_MODE", "off").lower().strip()
return mode if mode in _VALID_MODES else "off"
def _is_exempt(path: str) -> bool:
if path in _EXEMPT_EXACT:
return True
return any(path.startswith(p) for p in _EXEMPT_PREFIXES)
class EdgeAuthMiddleware(BaseHTTPMiddleware):
"""Reject origin requests that didn't come through the Cloudflare edge."""
async def dispatch(self, request: Request, call_next) -> Response:
mode = _mode()
if mode == "off":
return await call_next(request)
if _is_exempt(request.url.path):
return await call_next(request)
secret = os.environ.get("EDGE_ORIGIN_SECRET", "")
if not secret:
# Misconfiguration: enforce is requested but no secret is wired up.
# In enforce mode, fail CLOSED (503) — a security control must not
# silently allow direct-origin traffic when it can't do its job.
# This is low-risk operationally: Cloud Run won't even start a
# revision whose --set-secrets ref is missing, and enforce is only
# turned on after staging confirms the secret. In warn mode we still
# allow (observational only).
logger.error(
"EDGE_AUTH_MODE=%s but EDGE_ORIGIN_SECRET is unset.", mode,
)
if mode == "enforce":
return JSONResponse(
status_code=503,
content={"detail": "Origin authentication unavailable"},
)
return await call_next(request)
provided = request.headers.get(EDGE_AUTH_HEADER, "")
# Constant-time comparison to avoid leaking the secret via timing.
ok = bool(provided) and hmac.compare_digest(provided, secret)
if ok:
return await call_next(request)
client = request.client.host if request.client else "?"
if mode == "warn":
logger.warning(
"edge-auth: request missing/invalid %s header (warn mode, "
"allowing). path=%s ip=%s",
EDGE_AUTH_HEADER, request.url.path, client,
)
return await call_next(request)
# enforce
logger.warning(
"edge-auth: blocked direct-to-origin request. path=%s ip=%s",
request.url.path, client,
)
return JSONResponse(status_code=403, content={"detail": "Forbidden"})
+5 -13
View File
@@ -13,7 +13,6 @@ The middleware is non-blocking - if no tenant is detected, the request
proceeds as a default Nomad Karaoke request (tenant_id = None).
"""
import asyncio
import logging
import os
from typing import Optional
@@ -27,19 +26,13 @@ from backend.models.tenant import TenantConfig
logger = logging.getLogger(__name__)
# Only allow the `?tenant=` query-param override in non-production environments —
# in production it is a tenant-spoofing vector. Production-safe by default: Cloud
# Run always sets K_SERVICE, so we treat its presence as production even if
# ENVIRONMENT were somehow unset (which would otherwise re-enable the override).
# (Fallback audit 2026-06-09, Theme 7.)
# Only allow query param tenant override in non-production environments
IS_PRODUCTION = os.environ.get("ENV", "").lower() == "production" or \
os.environ.get("ENVIRONMENT", "").lower() == "production" or \
bool(os.environ.get("K_SERVICE"))
os.environ.get("ENVIRONMENT", "").lower() == "production"
# Known non-tenant subdomains that should be treated as default Nomad Karaoke.
# Mirror of frontend/lib/edge/nonTenantSubdomains.ts ("requests" = voting-board host).
NON_TENANT_SUBDOMAINS = {"gen", "api", "www", "buy", "admin", "app", "beta", "requests"}
# Known non-tenant subdomains that should be treated as default Nomad Karaoke
NON_TENANT_SUBDOMAINS = {"gen", "api", "www", "buy", "admin", "app", "beta"}
class TenantMiddleware(BaseHTTPMiddleware):
@@ -67,8 +60,7 @@ class TenantMiddleware(BaseHTTPMiddleware):
tenant_config: Optional[TenantConfig] = None
if tenant_id:
tenant_service = get_tenant_service()
# get_tenant_config can hit GCS on a cache miss: keep it off the event loop.
tenant_config = await asyncio.to_thread(tenant_service.get_tenant_config, tenant_id)
tenant_config = tenant_service.get_tenant_config(tenant_id)
if tenant_config and not tenant_config.is_active:
# Tenant exists but is inactive - treat as default
-110
View File
@@ -1,110 +0,0 @@
"""
Audio edit session model for persisting input audio editing progress.
Stored as a Firestore subcollection: jobs/{job_id}/audio_edit_sessions/{session_id}
Full edit data is stored in GCS to avoid Firestore's 1MB document limit.
"""
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Optional, Dict, Any, List
import uuid
@dataclass
class AudioEditSessionSummary:
"""Compact summary of operations in an audio edit session, for list views."""
total_operations: int = 0
operations_breakdown: Dict[str, int] = field(default_factory=dict)
duration_change_seconds: float = 0.0
net_duration_seconds: float = 0.0
def to_dict(self) -> Dict[str, Any]:
return {
"total_operations": self.total_operations,
"operations_breakdown": self.operations_breakdown,
"duration_change_seconds": self.duration_change_seconds,
"net_duration_seconds": self.net_duration_seconds,
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "AudioEditSessionSummary":
return cls(
total_operations=data.get("total_operations", 0),
operations_breakdown=data.get("operations_breakdown", {}),
duration_change_seconds=data.get("duration_change_seconds", 0.0),
net_duration_seconds=data.get("net_duration_seconds", 0.0),
)
@dataclass
class AudioEditSession:
"""
A saved snapshot of an audio editing session.
Stored at: jobs/{job_id}/audio_edit_sessions/{session_id}
Edit data stored at: jobs/{job_id}/audio_edit_sessions/{session_id}.json (GCS)
"""
session_id: str = field(default_factory=lambda: str(uuid.uuid4()))
job_id: str = ""
user_email: str = ""
created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
updated_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
edit_count: int = 0
trigger: str = "auto" # "auto" | "manual" | "submit"
audio_duration_seconds: Optional[float] = None
original_duration_seconds: Optional[float] = None
artist: Optional[str] = None
title: Optional[str] = None
summary: AudioEditSessionSummary = field(default_factory=AudioEditSessionSummary)
edit_data_gcs_path: str = ""
# Hash of the edit data JSON, used to skip duplicate saves
data_hash: str = ""
def to_dict(self) -> Dict[str, Any]:
return {
"session_id": self.session_id,
"job_id": self.job_id,
"user_email": self.user_email,
"created_at": self.created_at,
"updated_at": self.updated_at,
"edit_count": self.edit_count,
"trigger": self.trigger,
"audio_duration_seconds": self.audio_duration_seconds,
"original_duration_seconds": self.original_duration_seconds,
"artist": self.artist,
"title": self.title,
"summary": self.summary.to_dict(),
"edit_data_gcs_path": self.edit_data_gcs_path,
"data_hash": self.data_hash,
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "AudioEditSession":
created_at = data.get("created_at")
if isinstance(created_at, str):
created_at = datetime.fromisoformat(created_at.rstrip("Z")).replace(tzinfo=timezone.utc)
updated_at = data.get("updated_at")
if isinstance(updated_at, str):
updated_at = datetime.fromisoformat(updated_at.rstrip("Z")).replace(tzinfo=timezone.utc)
summary_data = data.get("summary", {})
summary = AudioEditSessionSummary.from_dict(summary_data) if isinstance(summary_data, dict) else AudioEditSessionSummary()
return cls(
session_id=data.get("session_id", str(uuid.uuid4())),
job_id=data.get("job_id", ""),
user_email=data.get("user_email", ""),
created_at=created_at or datetime.now(timezone.utc),
updated_at=updated_at or datetime.now(timezone.utc),
edit_count=data.get("edit_count", 0),
trigger=data.get("trigger", "auto"),
audio_duration_seconds=data.get("audio_duration_seconds"),
original_duration_seconds=data.get("original_duration_seconds"),
artist=data.get("artist"),
title=data.get("title"),
summary=summary,
edit_data_gcs_path=data.get("edit_data_gcs_path", ""),
data_hash=data.get("data_hash", ""),
)
+17 -217
View File
@@ -16,9 +16,8 @@ class JobStatus(str, Enum):
"""
Job status enumeration - Complete state machine.
The workflow has 8 main stages with up to 3 human interaction points:
The workflow has 7 main stages with 2 human interaction points:
1. Input & Setup (may include audio source selection)
1b. Optional Audio Editing (BLOCKING) - trim/cut/mute/join input audio
2. Parallel Processing (audio + lyrics)
3. Title/End Screen Generation + Backing Vocals Analysis
4. Countdown Padding Synchronization
@@ -37,18 +36,9 @@ class JobStatus(str, Enum):
SEARCHING_AUDIO = "searching_audio" # Searching for audio sources via flacfetch
AWAITING_AUDIO_SELECTION = "awaiting_audio_selection" # ⚠️ WAITING FOR USER - select audio source
DOWNLOADING_AUDIO = "downloading_audio" # Downloading selected audio from source
DOWNLOAD_PENDING_RETRY = "download_pending_retry" # Download hit a transient issue (few/no seeders, tracker blip); auto-retrying for up to 24h
DOWNLOADING = "downloading" # Downloading from URL or processing upload
# Optional: Audio editing (before processing begins)
AWAITING_AUDIO_EDIT = "awaiting_audio_edit" # ⚠️ WAITING FOR USER - edit input audio
IN_AUDIO_EDIT = "in_audio_edit" # User is actively editing audio
AUDIO_EDIT_COMPLETE = "audio_edit_complete" # User submitted edited audio
# Optional: duration cost confirmation (BLOCKING) - confirm credits before heavy processing
AWAITING_DURATION_CONFIRM = "awaiting_duration_confirm" # ⚠️ WAITING FOR USER - confirm duration-based cost
# Stage 2a: Audio separation (parallel track 1)
SEPARATING_STAGE1 = "separating_stage1" # Clean instrumental separation (Modal API)
SEPARATING_STAGE2 = "separating_stage2" # Backing vocals separation (Modal API)
@@ -72,7 +62,6 @@ class JobStatus(str, Enum):
# Stage 5.5: Render video with corrected lyrics (post-review)
RENDERING_VIDEO = "rendering_video" # Using OutputGenerator to create with_vocals.mkv
RENDER_PENDING_CAPACITY = "render_pending_capacity" # Waiting for GCE encoding capacity (auto-retry)
# Stage 6: Instrumental already selected (during combined review)
# Note: AWAITING_INSTRUMENTAL_SELECTION kept for DB compatibility with historical jobs only
@@ -104,36 +93,16 @@ class JobStatus(str, Enum):
# Valid state transitions
STATE_TRANSITIONS = {
# PENDING can go to DOWNLOADING (file upload), SEARCHING_AUDIO (artist+title search),
# or DOWNLOADING_AUDIO (create-from-search where audio is pre-selected)
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
# PENDING can go to DOWNLOADING (file upload) or SEARCHING_AUDIO (artist+title search)
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
# Audio search flow (for artist+title search mode)
JobStatus.SEARCHING_AUDIO: [JobStatus.AWAITING_AUDIO_SELECTION, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED],
JobStatus.AWAITING_AUDIO_SELECTION: [JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.DOWNLOADING_AUDIO: [JobStatus.DOWNLOADING, JobStatus.AWAITING_DURATION_CONFIRM, JobStatus.DOWNLOAD_PENDING_RETRY, JobStatus.FAILED],
# DOWNLOAD_PENDING_RETRY = parked when a torrent download hits a transient
# issue (few/no seeders, tracker outage); auto-retried by the recover-stuck
# cron for up to 24h before failing permanently.
JobStatus.DOWNLOAD_PENDING_RETRY: [JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.DOWNLOADING_AUDIO: [JobStatus.DOWNLOADING, JobStatus.FAILED],
# DOWNLOADING allows parallel processing (audio + lyrics) and then screens when both complete
# Can also enter optional audio edit phase if requires_audio_edit flag is set
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.AWAITING_AUDIO_EDIT, JobStatus.AWAITING_DURATION_CONFIRM, JobStatus.FAILED],
# Optional audio edit phase (trim, cut, mute, join input audio before processing)
JobStatus.AWAITING_AUDIO_EDIT: [JobStatus.IN_AUDIO_EDIT, JobStatus.AUDIO_EDIT_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.IN_AUDIO_EDIT: [JobStatus.AWAITING_AUDIO_EDIT, JobStatus.AUDIO_EDIT_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.AUDIO_EDIT_COMPLETE: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.AWAITING_DURATION_CONFIRM, JobStatus.FAILED],
# Duration cost confirmation (BLOCKING) - user confirms before heavy processing begins
JobStatus.AWAITING_DURATION_CONFIRM: [
# DOWNLOADING = confirmed, back to the prep phase (see jobs.confirm_duration).
JobStatus.DOWNLOADING,
JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING,
JobStatus.GENERATING_SCREENS, JobStatus.AWAITING_AUDIO_EDIT,
JobStatus.FAILED, JobStatus.CANCELLED,
],
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
# Audio separation flow
JobStatus.SEPARATING_STAGE1: [JobStatus.SEPARATING_STAGE2, JobStatus.FAILED],
@@ -146,37 +115,17 @@ STATE_TRANSITIONS = {
JobStatus.LYRICS_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
# Post-parallel processing (screens + backing vocals analysis)
# GENERATING_SCREENS -> REVIEW_COMPLETE is the auto-approval path: when the
# scorer is fully confident the review screens are skipped entirely (no
# review token / "ready for review" notification is ever issued).
JobStatus.GENERATING_SCREENS: [JobStatus.APPLYING_PADDING, JobStatus.AWAITING_REVIEW, JobStatus.REVIEW_COMPLETE, JobStatus.FAILED],
JobStatus.GENERATING_SCREENS: [JobStatus.APPLYING_PADDING, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
JobStatus.APPLYING_PADDING: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
# Combined human review flow (lyrics + instrumental selection)
# AWAITING_REVIEW can go directly to REVIEW_COMPLETE (quick review) or to IN_REVIEW (editing)
JobStatus.AWAITING_REVIEW: [JobStatus.IN_REVIEW, JobStatus.REVIEW_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.IN_REVIEW: [JobStatus.REVIEW_COMPLETE, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
JobStatus.REVIEW_COMPLETE: [
JobStatus.RENDERING_VIDEO,
JobStatus.PREP_COMPLETE, # for prep-only jobs
JobStatus.RENDER_PENDING_CAPACITY, # GCE capacity exhausted before render started
JobStatus.FAILED,
],
JobStatus.REVIEW_COMPLETE: [JobStatus.RENDERING_VIDEO, JobStatus.PREP_COMPLETE, JobStatus.FAILED], # PREP_COMPLETE for prep-only jobs
# Video rendering (post-review) - instrumental was already selected during combined review
# RENDER_PENDING_CAPACITY = parked when GCE encoding capacity is exhausted; auto-retried by scheduler.
JobStatus.RENDERING_VIDEO: [
JobStatus.INSTRUMENTAL_SELECTED,
JobStatus.PREP_COMPLETE,
JobStatus.RENDER_PENDING_CAPACITY,
JobStatus.FAILED,
],
JobStatus.RENDER_PENDING_CAPACITY: [
JobStatus.RENDERING_VIDEO, # auto-retry succeeded, picked up again
JobStatus.REVIEW_COMPLETE, # admin / scheduler resets to re-trigger render
JobStatus.FAILED, # exceeded max wait or operator gave up
JobStatus.CANCELLED, # user cancelled while waiting
],
JobStatus.RENDERING_VIDEO: [JobStatus.INSTRUMENTAL_SELECTED, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
# AWAITING_INSTRUMENTAL_SELECTION is LEGACY - kept for historical jobs in DB
# New jobs never enter this state; they go directly to INSTRUMENTAL_SELECTED after render
@@ -192,24 +141,20 @@ STATE_TRANSITIONS = {
JobStatus.UPLOADING: [JobStatus.NOTIFYING, JobStatus.COMPLETE, JobStatus.FAILED],
JobStatus.NOTIFYING: [JobStatus.COMPLETE, JobStatus.FAILED],
# Terminal states - COMPLETE allows edit (reopen review) and visibility change
# Terminal states - COMPLETE, PREP_COMPLETE have no transitions
# FAILED and CANCELLED allow retry transitions to resume from checkpoints
# PREP_COMPLETE allows continuation from combined review
JobStatus.COMPLETE: [JobStatus.AWAITING_REVIEW, JobStatus.LYRICS_COMPLETE], # Edit track or visibility change
JobStatus.COMPLETE: [],
JobStatus.PREP_COMPLETE: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED], # Continue from combined review
JobStatus.FAILED: [
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
JobStatus.DOWNLOADING_AUDIO, # Retry audio download (audio search flow)
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
JobStatus.REVIEW_COMPLETE, # Retry from render stage
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
JobStatus.AWAITING_REVIEW, # Retry from combined review
JobStatus.RENDER_PENDING_CAPACITY, # Operator parks a stale-failed job for auto-retry
JobStatus.AWAITING_AUDIO_SELECTION, # Choose different audio after a failed download
],
JobStatus.CANCELLED: [
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
JobStatus.DOWNLOADING_AUDIO, # Retry audio download (audio search flow)
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
JobStatus.REVIEW_COMPLETE, # Retry from render stage
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
@@ -231,7 +176,6 @@ class TimelineEvent(BaseModel):
timestamp: str
progress: Optional[int] = None
message: Optional[str] = None
metadata: Optional[Dict[str, Any]] = None
class LogEntry(BaseModel):
@@ -261,14 +205,7 @@ class Job(BaseModel):
title: Optional[str] = None
filename: Optional[str] = None # Original uploaded filename
input_media_gcs_path: Optional[str] = None # GCS path to uploaded file
# Audio source tracking (for retry capability)
audio_source_type: Optional[str] = None # Source type: 'audio_search', 'youtube_url', 'file_upload'
source_name: Optional[str] = None # Provider name (RED, OPS, YouTube, Spotify) - for audio_search
source_id: Optional[str] = None # Source-specific ID (torrent ID, video ID, track ID) - for audio_search
target_file: Optional[str] = None # For torrents, specific file to extract - for audio_search
download_url: Optional[str] = None # Direct download URL if available - for audio_search
# User preferences
enable_cdg: bool = False # Generate CDG+MP3 package (requires style config)
enable_txt: bool = False # Generate TXT+MP3 package (requires style config)
@@ -278,39 +215,11 @@ class Job(BaseModel):
user_email: Optional[str] = None # Email for notifications
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
# Review autonomy level. "auto" (default): skip the combined review screens
# whenever the auto-approval scorer is fully confident, falling back to human
# review otherwise. "always_review": every job stops at the review screens.
# (Distinct from legacy non_interactive, which is CLI packaging/upload
# semantics and never gated the cloud review flow.)
review_mode: str = "auto"
# Backing-vocals preference chosen up-front at submission. "auto" (default):
# retain backing vocals wherever the 3-stem decider is confident it's safe,
# else fall through to human review. "clean": always use the clean
# instrumental (never retain backing). "review": always let a human pick the
# instrumental. Consumed by the auto-approval instrumental decision.
backing_preference: str = "auto"
# Translated lyrics: language code (one of the UI locales, e.g. "en", "es") whose
# LLM translation of each reviewed line is drawn in smaller text beneath it in the
# karaoke videos. None = no translation row. See backend/services/lyrics_translation.py.
translation_language: Optional[str] = None
# Multi-tenant support ("" = consumer portal, "vocalstar"/etc = tenant portal)
tenant_id: str = "" # Tenant ID for white-label portal scoping
# UI language the job was submitted in (primary Accept-Language subtag, e.g.
# "pt", "ja", "zh"). Any of the 33 supported UI locales — NOT narrowed to the
# 3 email locales. Admin-only signal for support/intervention. None for jobs
# created before this field existed or where the header was absent.
locale: Optional[str] = None
# Multi-tenant support (None = default Nomad Karaoke)
tenant_id: Optional[str] = None # Tenant ID for white-label portal scoping
# Theme configuration (pre-made themes from GCS)
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
# When the theme was last re-snapshotted onto the job after creation (theme
# edit refresh, re-render, Edit). None = the creation-time snapshot (created_at).
theme_applied_at: Optional[datetime] = None
color_overrides: Dict[str, str] = Field(default_factory=dict)
"""
User color overrides applied on top of theme. Keys:
@@ -383,38 +292,10 @@ class Job(BaseModel):
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
customer_notes: Optional[str] = None # Notes provided by customer with their order
# Private (non-published) track mode
is_private: bool = False # Private tracks: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive
# Creation-time decision params (for observability — what did the user actually choose?)
creation_params: Dict[str, Any] = Field(default_factory=dict)
# Anti-abuse: IP address at job creation time (for cross-account correlation)
creation_ip: Optional[str] = None
# Output deletion tracking (for admin cleanup without deleting job)
outputs_deleted_at: Optional[datetime] = None # Timestamp when outputs were deleted by admin
outputs_deleted_by: Optional[str] = None # Admin email who deleted outputs
# Storage retention (backend/services/storage_retention.py): set when the
# daily purge deleted this job's regenerable renders / separated stems.
# Cleared when they're rebuilt (regenerate / re-render / stems restore).
renders_purged_at: Optional[datetime] = None
stems_purged_at: Optional[datetime] = None
renders_regenerated_at: Optional[datetime] = None # last GCS-only regenerate
storage_purge: Optional[Dict[str, Any]] = None # manifest of the last purge
# Credit refund tracking (prevents double-refund on cancel→delete or fail→delete)
credit_refunded: bool = False
# Edit tracking (how many times a completed track has been re-edited)
edit_count: int = 0
# Whole-track tempo change applied in the audio editor (1.0 / None = original
# speed). When set, job.title carries a "(90% Tempo)" label so every published
# output says so; see backend/services/tempo_label.py.
tempo_factor: Optional[float] = None
# Processing state
track_output_dir: Optional[str] = None # Local output directory (temp)
audio_hash: Optional[str] = None # Hash for deduplication
@@ -505,43 +386,6 @@ class Job(BaseModel):
}
"""
# Processing metadata (immutable provenance data written by workers during processing)
processing_metadata: Dict[str, Any] = Field(default_factory=dict)
"""
Immutable processing provenance data. Written once by each worker during processing.
Never modified after initial write. Used for analytics, debugging, and data export.
Structure:
{
"transcription": {
"provider": "audioshake",
"audioshake_task_id": "...",
"audioshake_asset_id": "...",
"language_detected": "en",
"word_count": 358,
"segment_count": 42,
},
"correction": {
"handlers_applied": [...],
"corrections_made": 12,
"correction_ratio": 0.034,
"reference_sources_found": ["genius", "spotify"],
"agentic_routing": "rule-based",
"duration_seconds": 33.2,
},
"separation": {
"provider": "modal",
"clean_model": "...",
"duration_seconds": 420.5,
},
"timing": {
"lyrics_worker_seconds": 125.3,
"audio_worker_seconds": 420.5,
...
},
}
"""
# Request metadata (captured at job creation for tracking and filtering)
request_metadata: Dict[str, Any] = Field(default_factory=dict)
"""
@@ -598,15 +442,7 @@ class JobCreate(BaseModel):
webhook_url: Optional[str] = None
user_email: Optional[str] = None
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
# "auto" (default) = skip review screens when the auto-approval scorer is
# fully confident; "always_review" = always stop at the review screens.
review_mode: str = "auto"
# "auto" (default) = retain backing vocals where confidently safe; "clean" =
# always strip backing vocals; "review" = always let a human pick the instrumental.
backing_preference: str = "auto"
# Translated lyrics language (see Job.translation_language); None = off
translation_language: Optional[str] = None
# Theme configuration (pre-made themes from GCS)
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
color_overrides: Dict[str, str] = Field(default_factory=dict)
@@ -663,14 +499,6 @@ class JobCreate(BaseModel):
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
customer_notes: Optional[str] = None # Notes provided by customer with their order
# Private (non-published) track mode
is_private: bool = False # Private tracks: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive
# Credit pricing: how many credits to charge for this job at creation time.
# Search and upload jobs pass the default (1) and may top up later.
# URL jobs with a known duration pass ceil(duration/600) credits.
credits: int = 1
# Request metadata (set by API endpoint from request headers)
request_metadata: Dict[str, Any] = Field(default_factory=dict)
"""
@@ -683,12 +511,8 @@ class JobCreate(BaseModel):
- custom_headers: All X-* headers
"""
# Tenant scoping for white-label portals ("" = consumer, "vocalstar"/etc = tenant)
tenant_id: str = "" # Tenant ID for job scoping
# UI language the job was submitted in (primary Accept-Language subtag, e.g.
# "pt", "ja"). Set by the API endpoint from the request header; admin-only.
locale: Optional[str] = None
# Tenant scoping for white-label portals
tenant_id: Optional[str] = None # Tenant ID for job scoping
@validator('url')
def validate_url(cls, v):
@@ -697,30 +521,6 @@ class JobCreate(BaseModel):
raise ValueError("Field cannot be empty string")
return v.strip() if isinstance(v, str) else v
@validator('review_mode')
def _normalize_review_mode(cls, v):
"""Coerce an out-of-set value to the fail-safe default (always_review =
more human oversight), so a typo can never silently skip review. The
executor already treats any non-"auto" value as an enforcement blocker."""
return v if v in ("auto", "always_review") else "always_review"
@validator('translation_language')
def _normalize_translation_language(cls, v):
"""Supported language code; empty or unsupported values turn the feature off
(every creation route passes the client value straight through)."""
if v is None or (isinstance(v, str) and not v.strip()):
return None
from backend.services.lyrics_translation import normalize_language
return normalize_language(v)
@validator('backing_preference')
def _normalize_backing_preference(cls, v):
"""Coerce an out-of-set value to the default. Every creation route passes
the client value straight through, so this is the single choke point that
keeps a typo from reaching the instrumental decision as an unknown value."""
return v if v in ("auto", "clean", "review") else "auto"
@validator('artist', 'title')
def normalize_artist_title(cls, v):
"""Normalize artist/title text to standardize Unicode characters.
Loaded 100 of 2186 files, more files were not shown because too many files have changed in this diff. Show more