Compare commits

...
Author SHA1 Message Date
Andrew BeveridgeandClaude Opus 4.6 3b848f696d docs: add error monitor operational reference
Comprehensive reference doc covering: how it works, monitored resources,
infrastructure, Firestore collections, code structure, slash commands,
common operations, configuration, and troubleshooting.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 18:06:36 -04:00
Andrew BeveridgeandClaude Opus 4.6 e8e68b0261 fix(error-monitor): suppress Discord link preview embeds
Add flags=4 (SUPPRESS_EMBEDS) to webhook POST to prevent Discord from
generating link preview cards for URLs in error messages.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 18:06:25 -04:00
Andrew BeveridgeandClaude Opus 4.6 2d54d0c0b6 fix(error-monitor): Gen2 Cloud Functions log as cloud_run_revision
Gen2 Cloud Functions (gdrive-validator, backup-to-aws, etc.) appear as
cloud_run_revision in Cloud Logging, not cloud_function. Moved their
service names into MONITORED_CLOUD_RUN_SERVICES with correct hyphenated
names matching actual Cloud Run service names.

This fixes the monitor finding 0 errors when there were real errors in
karaoke-backend (Dropbox token expired, invalid state transitions) and
encoding-worker-idle-shutdown (memory limit exceeded every 5 min).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 16:49:39 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 e60ce8e479 fix: flyer dialog layout, SVG gradient export, full-size capture (#709)
## Summary

Follow-up fixes to #708 (referral flyer customisation):

- Widen dialog from `max-w-4xl` to `max-w-6xl`, constrain flyer preview
to 240px
- Replace CSS `background-clip: text` with inline SVG `<text>` +
`<linearGradient>` for export compatibility
- Swap `html2canvas` for `html-to-image` (better CSS support)
- Capture flyer at full 816x1056 size by resetting scale transform
before export

## Testing

- [x] 743/743 tests pass
- [x] PNG export verified — gradient text renders correctly at full
resolution
- [x] Preview matches export output

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 16:26:35 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 d4242c78a8 feat(e2e): daily payment + happy path production test (#715)
@coderabbitai ignore

## Summary

- **New daily E2E test** that purchases credits via **real Stripe
Checkout** ($3/day with 70% referral discount), then uses the credit for
the existing happy path karaoke generation test
- Motivated by the April 10 payment webhook outage — the existing
simulated-webhook test would never have caught the Stripe SDK v14 bug
- Adds Discord + email alerts for both stages

## Changes

- **New:** `frontend/e2e/helpers/stripe-checkout.ts` — Playwright helper
for Stripe's hosted checkout page
- **New:** `frontend/e2e/production/credit-purchase-real.spec.ts` — Real
Stripe payment test
- **New:** `.github/workflows/e2e-daily.yml` — Two-stage daily workflow:
credit purchase → happy path
- **Deleted:** `.github/workflows/e2e-happy-path.yml` — Replaced by
combined workflow
- **Docs:** Design spec + implementation plan

## Verified

- Tested locally 15 iterations to get Stripe Checkout selectors right
- Full payment flow works end-to-end: signup → referral discount →
Stripe → credits → email
- $3 real charge processed successfully each test run

## Test plan

- [x] Local test passes (real $3 Stripe payment)
- [ ] Merge and trigger e2e-daily workflow manually
- [ ] Verify Discord notification
- [ ] Monitor first automated daily run (6 AM UTC)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 10:57:08 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 9a5c9d497a feat: production error monitor — log collection, LLM analysis, Discord alerts (#713)
## Summary

- Add a production error monitoring service that runs as a Cloud Run Job
every 15 minutes
- Queries Cloud Logging across all NK services (Cloud Run, Jobs,
Functions, GCE VMs), normalizes/deduplicates errors, uses Gemini Flash
for incident grouping, sends Discord alerts, and auto-resolves silent
patterns
- Updates all `/prod-*` slash commands to use pre-computed Firestore
patterns as primary data source

## Architecture

Based on the proven Aquarius error bot system, adapted for Nomad
Karaoke's multi-service architecture (20+ resources across 4 resource
types).

**New backend modules** (`backend/services/error_monitor/`):
- `config.py` — constants, monitored services, env var overrides
- `normalizer.py` — regex normalization + SHA-256 pattern hashing
- `known_issues.py` — 8 ignore patterns for infrastructure noise
- `firestore_adapter.py` — CRUD for `error_patterns`, `error_incidents`,
`discord_alerts`
- `discord.py` — webhook client with 5 alert formatters (new pattern,
incident, spike, auto-resolved, daily digest)
- `llm_analysis.py` — Gemini Flash incident grouping + near-duplicate
detection
- `monitor.py` — orchestrator entry point (`python -m
backend.services.error_monitor.monitor`)

**New infrastructure** (`infrastructure/modules/error_monitor.py`):
- Cloud Run Job (`nomad-error-monitor`) sharing karaoke-backend image
- Cloud Scheduler: every 15 min monitor + 08:00 UTC daily digest
- Scheduler service account with `roles/run.invoker`
- Backend SA gets `roles/logging.viewer`

**New CLI scripts:**
- `scripts/query-error-patterns.py` — query patterns from Firestore
- `scripts/resolve-error-pattern.py` — mark patterns as fixed with PR
reference

## Testing

- 310 unit tests across 7 test modules (all passing)
- Covers: config, normalizer, known issues, Firestore adapter, Discord
formatting, LLM analysis, monitor orchestrator
- P0/P1 caller-callee contract tests for monitor integration points
- All tests use mocked external dependencies (no GCP calls)

## Review

- [x] CodeRabbit CLI review completed locally (1 cycle, 3 issues fixed)
- [x] Test review completed (P0/P1 gaps addressed)
- [x] Documentation updated (README, ARCHITECTURE)
- [x] Version bumped (0.168.2 → 0.169.0)

## Design Spec

`docs/archive/2026-04-12-error-monitor-design.md`

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 01:02:08 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 b847dba411 fix: Stripe webhook crash — payments not crediting users (#712)
@coderabbitai ignore

## Summary

- **Critical production fix**: All Stripe payment webhooks have been
returning 500 since ~April 9, meaning users pay but don't receive
credits or confirmation emails
- **Root cause**: Stripe SDK v14+ `Event` objects don't support `.get()`
(unlike older versions that inherited from `dict`). The webhook handler
calls `event.get("type")` at line 1020, which raises `AttributeError:
get`
- **Fix**: Convert Stripe Event to plain dict via
`json.loads(str(event))` in `verify_webhook_signature()`, before the
event reaches the handler
- **Secondary fix**: Narrow `except Exception` to `except AlreadyExists`
in `add_credits` idempotency check — was masking permission/network
errors as "already processed"
- **Pin stripe to `^14.1.0`** (was `>=7.0.0`) to prevent unexpected
major upgrades in Docker builds

## Impact

- **3 payments ($59.50 total) for 2 users** affected since April 9
- Stripe retries failed webhooks for 3 days — all affected payments are
within the retry window and will be processed automatically after deploy
- No manual intervention needed for credits or emails

## Test plan

- [x] All 59 existing payment/credit tests pass
- [x] Verified fix with Stripe v14 Event objects (unicode, empty
metadata, refund events)
- [ ] After deploy: monitor logs for successful webhook processing
- [ ] Verify affected users receive credits via Stripe webhook retries

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 11:18:09 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 f0e9a06c44 fix: brighter lime-green highlight for user-edited words (#711)
## Summary
- Changes user-edited word highlight from subtle violet
(`bg-violet-500/25`) to bright lime green (`bg-lime-400/40`)
- Much more visible and obvious which words have been edited
- Distinct from AI-corrected words which use `bg-green-500/25`

## Test plan
- [x] All 743 tests pass
- [ ] Edit a word in lyrics review → bright green background appears

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 00:16:32 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 ce24df53d9 feat: highlight user-edited words in lyrics review (#710)
## Summary
- Words edited by the user in lyrics review now show a **violet
background** (`bg-violet-500/25`) so it's immediately obvious which
words have been modified
- Compares current word text against initial state (`history[0]`), so
undo/redo correctly updates highlights
- New words added by the user are also highlighted

## Changes
- `Word.tsx` — accepts `isUserEdited` prop, renders violet highlight
- `LyricsAnalyzer.tsx` — computes `editedWordIds` via `useMemo`
comparing current vs initial data
- `TranscriptionView.tsx` / `HighlightedText.tsx` — thread
`editedWordIds` set through to Word component
- `constants.ts` / `types.ts` — add `userEdited` highlight class and
`isUserEdited` prop

## Color priority (highest → lowest)
1. Currently playing (blue solid)
2. Anchor (blue tint)
3. **User-edited (violet tint)** ← new
4. Corrected gap (green tint)
5. Uncorrected gap (orange tint)

## Test plan
- [x] All 743 existing tests pass
- [x] Frontend build succeeds
- [ ] Manually verify: edit a word in lyrics review → violet background
appears
- [ ] Manually verify: undo the edit → violet background disappears
- [ ] Manually verify: add a new word → violet background appears on new
word

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-09 23:49:16 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 3fac565c76 feat: referral flyer customisation with tabbed QR/flyer dialog (#708)
## Summary

- Expand the QR Code Generator dialog into a tabbed **Referral Tools**
dialog with **QR Code** and **Printable Flyer** tabs
- Add a live flyer preview with extensive customisation: colors, text
overrides, section visibility, custom branding, and print layout (1/2/4
per page)
- Client-side export to PDF, HTML, and PNG (replaces server-side flyer
generation)
- Preset themes (Light/Dark) with save/load custom presets via
localStorage

## Changes

- **New components:** `ReferralToolsDialog`, `QRCodeTab`, `FlyerTab`,
`FlyerPreview`, `FlyerControls`, `FlyerExport`, `flyer-presets`
- **Removed:** `QRCodeDialog` (replaced by `ReferralToolsDialog`)
- **Modified:** `ReferralDashboard.tsx`, `admin/referrals/page.tsx`
(import updates)
- **Dependencies:** Added `html2canvas`, `jspdf` for client-side export
- **i18n:** 30+ new translation keys across all 33 locales

## Testing

- [x] 32 referral-specific tests pass (presets, export, dialog
interaction)
- [x] 743/743 total frontend tests pass
- [x] TypeScript compilation clean (no new errors)
- [x] CodeRabbit CLI review completed locally, feedback addressed

## Review

- [x] Local CodeRabbit review completed (10 findings, 6 fixed, 4 skipped
as nitpicks)
- [x] Review feedback addressed in dedicated commit

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-09 19:02:12 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 43900921a5 fix: validate auth token on referral interstitial (#707)
## Summary
- Users with stale/expired tokens in localStorage were shown "Go to App"
instead of the magic link email form on referral interstitial pages
(`/r/<code>`)
- Now validates the token against `/api/users/me` before deciding which
UI to show
- Clears stale tokens so future visits work correctly

## Root cause
The interstitial checked `getAccessToken()` (localStorage only) which
returns non-null even for expired tokens. A user who had previously
visited the app would always see "Go to App" even if their session was
long expired.

## Test plan
- [x] Frontend build passes
- [x] All 722 tests pass
- [ ] Manual: visit `/r/<code>` with a stale token in localStorage →
should see email form
- [ ] Manual: visit `/r/<code>` while genuinely logged in → should see
"Go to App"

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 11:18:23 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 6068ef7a93 fix: only clear auth token on explicit 401, not network errors (#706)
## Summary

- `fetchUser()` was clearing the access token on ANY error (network
failures, timeouts, aborted requests during page redirects)
- This caused users to get logged out when returning from external flows
like Stripe Connect onboarding — the redirect chain triggers transient
fetch errors that wiped the token
- Now only clears the token when the server explicitly returns 401
(session expired/invalid)

This is the root cause of the "logged out after Stripe Connect" issue
reported alongside #702.

## Test plan

- [ ] Manual: complete Stripe Connect flow, verify user stays logged in
on return
- [x] Existing auth tests still pass (no behavior change for actual
401s)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 19:38:48 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 83fdc9e786 fix: hide Manage Payouts when Stripe onboarding incomplete (#705)
## Summary

- Hide "Manage Payouts" button when Connect account hasn't completed
onboarding (`details_submitted` is false)
- Stripe API rejects login link creation for incomplete accounts
(returns 400)
- "Update Bank Account" button remains available to complete setup

Follow-up to #702.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 19:15:54 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 77f64a52d7 feat: mobile-friendly two-row header (#704)
## Summary
- Adds a second row to the app header on mobile (< 640px) so action
buttons show text labels instead of cryptic icons
- Row 1: logo + title + auth/login status
- Row 2: refresh, referrals, help, language switcher, theme toggle — all
with visible text labels
- Desktop layout (≥ 640px) is completely unchanged
- Extracted HelpDropdown and ThemeToggle into small sub-components to
avoid JSX duplication

## Test plan
- [x] All 721 existing tests pass
- [ ] Visual check on mobile viewport (375px, 390px) — two rows visible
with text labels
- [ ] Visual check on desktop — single row, unchanged from before
- [ ] Check dashboard page (with Refresh/Admin children) and referrals
page (no children)
- [ ] Check logged-in vs logged-out states
- [ ] Verify language switcher dropdown positions correctly from row 2

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 19:05:13 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 1c1c17f9ab fix: Stripe Connect redirect and referral payout management (#702)
## Summary

- **Fix broken Stripe Connect return URL** — was redirecting to
`/app?tab=referrals` (the dashboard, which ignores that param) instead
of `/app/referrals`. Users were ending up logged out at `/en/` after
completing Stripe onboarding.
- **Add locale redirect fallback** for `/app/referrals` route so Stripe
can redirect without locale prefix
- **Add Connect account status and management** — new backend endpoints
for account status, Express dashboard login, and re-onboarding. Frontend
shows payouts enabled/pending status, "Manage Payouts" and "Update Bank
Account" buttons.
- **Improve Connect button UX** — loading spinner + note that user will
be redirected to Stripe, error feedback for all actions

## Test plan

- [x] Backend: 48 referral tests + 7 Stripe tests pass
- [x] Frontend: 721 tests pass
- [x] CodeRabbit review: 3 issues found and fixed
- [ ] Manual: verify Connect flow with test account after deploy
- [ ] Manual: verify return redirect lands on referrals page with
success banner

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 18:29:29 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 9283d3c1ae feat: referral discount details UX + translation pipeline fix (#701)
@coderabbitai ignore

## Summary

- **Clickable "discount active" in dropdown**: Users who dismissed the
referral discount banner can click "discount active" in the user
dropdown to re-show it (clears localStorage dismissal, navigates to
dashboard)
- **Richer Buy Credits dialog**: Fetches and displays referrer's display
name and referral code alongside existing discount % and expiration info
- **Backend**: `no_track` query param on `GET /api/referrals/r/{code}`
to skip click tracking for display-only fetches
- **Translation pipeline fix**: `translate.py` dry-run no longer saves
snapshot (was causing subsequent runs to skip all translations), and
added missing-keys mode that compares against existing locale files
instead of re-translating everything when no snapshot exists

## Test plan

- [x] Backend unit test for `no_track=true` param
- [x] Frontend AuthStatus test updated with `next/navigation` mock
- [x] All 721 frontend tests passing
- [x] All backend tests passing
- [ ] Manual: dismiss discount banner → click "discount active" in
dropdown → verify banner re-appears on dashboard
- [ ] Manual: open Buy Credits dialog → verify referrer name and code
shown
- [ ] Manual: run `translate.py --dry-run` → verify snapshot not saved

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 18:19:03 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 3a1fd08c96 fix: apply discount dropdown, credit pricing, checkout state (#700)
## Summary
- **Apply Discount modal**: referral code dropdown was empty —
`loadReferralLinks()` was in `onOpenChange` which doesn't fire for
controlled Dialog open prop changes. Moved to button onClick. Also
replaced `Select` with searchable `Combobox` (Popover+Command) to scale
with many referral codes.
- **Buy Credits dialog**: discounted prices showed contradictory values
(e.g. "$9" and "$8.50/video") because `formatPrice` used `toFixed(0)`
rounding. Now shows cents when price isn't a whole dollar.
- **Buy Credits dialog**: "Redirecting to payment..." persisted after
closing/reopening because `checkoutLoading` wasn't reset on dialog open.

## Test plan
- [x] Build passes
- [x] 721 tests pass
- [x] CodeRabbit: no findings
- [ ] Manually verify Apply Discount modal loads and filters referral
codes
- [ ] Manually verify Buy Credits shows correct discounted prices (e.g.
$8.50 not $9)
- [ ] Manually verify closing/reopening Buy Credits resets checkout
button state

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 14:09:47 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 8ef587e7e3 feat(referral): fix discount checkout bug, show discounted prices, admin UX (#699)
## Summary
- **Fix**: Referral discount silently dropped at Stripe checkout due to
timezone-aware vs naive datetime comparison (same bug previously fixed
in `_build_user_public` but missed in `should_apply_discount`)
- **Admin UX**: Replace free-text referral code input in Apply Discount
dialog with dropdown of active codes (prevents typos/nonexistent codes)
- **Buy Credits modal**: Show discounted prices with strikethrough
originals when user has active referral discount
- **Dashboard banner**: Make referral discount banner dismissable (X
button, localStorage)
- **User menu**: Show "X% discount active" in the dropdown so discount
info persists after banner dismissal

## Test plan
- [ ] Verify referred user sees discounted prices in Buy Credits modal
- [ ] Verify Stripe checkout session includes the coupon (discount
actually applied)
- [ ] Verify discount banner can be dismissed and stays dismissed on
reload
- [ ] Verify user menu shows discount status
- [ ] Verify admin Apply Discount dialog shows dropdown of active
referral codes

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 16:53:06 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 3a9438f3e4 fix(referrals): use nomadkaraoke.com/r/ for QR codes and referral URLs (#698)
## Summary
QR codes and referral link displays were using
`gen.nomadkaraoke.com/r/{code}` which 404s. The `/r/` short URLs live on
`nomadkaraoke.com` (which redirects to gen). Fixes all referral URL
generation in both user dashboard and admin QR dialog.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 11:56:38 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 404caa624a feat(admin): full QR/flyer dialog on admin referrals page (#697)
## Summary
Replace the simple flyer download button on the admin referrals page
with the full QRCodeDialog modal — same one users see on their referrals
page. Supports customizable QR styles, colors, center logos, and
light/dark flyer themes.

- Add `onGenerateFlyer` and `flyerFilename` props to `QRCodeDialog` so
admin can use the admin flyer endpoint
- Wrap dialog in `NextIntlClientProvider` since admin pages don't have
the i18n provider

## Test plan
- [x] Build passes
- [ ] Click QR icon on any referral row → full dialog opens
- [ ] Customize QR, generate flyer → downloads PDF for that code

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 11:42:54 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 0dbd1d3ab0 fix(referrals): vanity URL request via backend API instead of mailto (#696)
## Summary
The `mailto:` approach for vanity URL requests silently fails when users
don't have a mail client configured — no email is ever sent.

- Adds `POST /api/referrals/me/vanity-request` endpoint that sends an
email to admin via SendGrid
- Validates the desired code isn't already taken (returns 409 with
friendly error)
- Frontend shows inline error messages and loading state
- Removes the broken `mailto:` approach

## Test plan
- [x] Build passes
- [x] All 721 tests pass
- [ ] Request a vanity URL → admin receives email
- [ ] Request a taken code → shows "already taken" error

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 11:12:56 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 9e96f7e089 fix: logo invisible — next/image SVG collapses to 0x0 (#695)
## Summary
The logo was rendering at 0x0 pixels. Root cause: `next/image` with
`w-auto h-auto` on an SVG has no intrinsic sizing, so it collapses.

Replaces `TenantLogo` with a plain `<img>` using explicit height classes
(`h-8 sm:h-10`) matching the header. Removes the `next/image` import
entirely since it causes more problems than it solves for logo SVGs.

## Test plan
- [x] Build passes
- [ ] Logo visible on dashboard (`/en/app/`)
- [ ] Logo visible on referrals page (`/en/app/referrals/`)
- [ ] Tenant logos still work (custom branding)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 01:54:19 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 31b02abd8c fix: replace broken external logo URL with local SVG (#694)
## Summary
The old WordPress PNG logo URL (`nomadkaraoke.com/wp-content/...`) is
returning 404. Both `logo.tsx` and `tenant-logo.tsx` still referenced it
as the default.

Switches to `/nomad-karaoke-logo.svg` which is already in `public/` and
used by `AppHeader`.

## Test plan
- [x] Build passes
- [ ] Logo loads on all pages (dashboard, referrals, tenant landing)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 01:42:13 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 04c7197445 feat(i18n): add translation cache, CI validation, and enforcement (#693)
## Summary
@coderabbitai ignore

- Add GCS translation cache (`translation_cache.py`) to `translate.py`
for instant repeated translations
- Add `--no-cache`, `--dry-run`, `--cache-bucket` CLI flags to
translation script
- Add `translation-check` CI job that validates all 33 locale files have
matching keys
- Add `.githooks/pre-commit` hook for auto-translation when `en.json`
changes
- Add i18n section to CLAUDE.md documenting 33-locale support and
enforcement rules

## Test plan
- [ ] `python frontend/scripts/translate.py --messages-dir
frontend/messages --target es --dry-run` shows cache stats
- [ ] `python frontend/scripts/validate-translations.py --messages-dir
frontend/messages` passes
- [ ] CI `translation-check` job runs on PR

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 01:22:39 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 19c85ca5b2 fix(referrals): DRY shared AppHeader matching dashboard style (#692)
## Summary
Follow-up to #691. Fixes issues found in production:

- Rewrites `AppHeader` to match the dashboard header style (local SVG
logo, fixed position, dark theme, consistent controls)
- Dashboard now uses the shared `AppHeader` with page-specific buttons
(refresh, admin toggle) passed as children
- Removes broken `Logo` component (external PNG URL) and dead `/jobs`
link that 404'd
- Adds proper top padding on referrals page for the fixed header

## Test plan
- [x] Build passes (`next build`)
- [x] All 721 tests pass (`make test`)
- [ ] Visual check: dashboard header unchanged after refactor
- [ ] Visual check: referrals page header now matches dashboard
- [ ] Logo loads correctly (local SVG)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 01:13:43 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 60c893391e feat(referrals): improve UX with header, explainer, vanity URL request, and i18n (#691)
## Summary
- Add AppHeader + "Back to Dashboard" nav to referrals page (matches the
rest of the app)
- Add friendly "How It Works" section explaining the 3-step referral
process with dynamic terms
- Add vanity URL request flow (opens pre-filled mailto to admin)
- Translate entire referrals section to all 32 non-English locales via
Gemini pipeline

## Test plan
- [x] Build passes (`next build`)
- [x] All 721 tests pass (`make test`)
- [x] Translation validation — no missing keys across 32 locales
- [ ] Visual check: referrals page has consistent header
- [ ] Visual check: "How It Works" section shows correct terms
- [ ] Visual check: vanity URL request form works (opens email)
- [ ] Visual check: non-English locale loads referrals translations

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 00:46:28 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 7c2d50b199 fix(referral): QR dialog — imageOptions error, emoji logo, button label (#690)
@coderabbitai ignore

## Summary
Fixes 3 issues with the QR code dialog shipped in #688:

1. **Bug fix**: `TypeError: can't access property "hideBackgroundDots",
e.imageOptions is undefined` — Always provide `imageOptions` to
`qr-code-styling` even when no logo selected
2. **Fix broken logos**: Replace Lucide icon center logos (Mic/Music)
with an emoji picker. The `iconNode` internal API doesn't work in
production builds. Users can now type any emoji as their QR center logo.
3. **Better button**: Change dashboard button from icon-only QR icon to
labeled "QR Code / Flyer"

Also includes CodeRabbit feedback: debounce cleanup on unmount,
FileReader error handler, user-facing error message for flyer failures.

## Test plan
- [x] 721/721 frontend tests passing
- [x] Updated center logo test assertions
- [ ] Manual: open QR dialog, verify no console error, try emoji logo,
download QR

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 23:30:27 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 e441b854c6 feat(referral): marketing flyer PDF generator (#688)
@coderabbitai ignore

## Summary
- **Marketing Flyer Generator**: Print-ready PDF flyers (light + dark
themes) with personalized referral code, discount, and styled QR code
- Server-side rendering via headless Chromium (added to Dockerfile)
- Available from QR Code dialog (user) and admin referrals table (admin)

## Changes

### Backend
- **New:** `flyer_service.py` — Template rendering + headless Chromium
PDF generation
- **New:** `POST /api/referrals/me/flyer` — User flyer endpoint
- **New:** `POST /api/referrals/admin/links/{code}/flyer` — Admin flyer
endpoint
- **New:** Light + dark HTML flyer templates (from kjbox printables)
- **Modified:** `Dockerfile.base` — Added `chromium` package

### Frontend
- **Modified:** `QRCodeDialog.tsx` — Theme picker + Generate Flyer
button, error handling
- **Modified:** Admin referrals page — Flyer button per row
- **New:** `generateFlyer()` API client function (user + admin)
- **i18n:** 5 new flyer-related strings

### Infrastructure
- **Dockerfile.base change** — Adds Chromium (~200MB). Base image
rebuild required.

## Test plan
- [x] 8 backend unit tests (template substitution, validation, Chromium
flags)
- [x] 11 frontend tests (QR dialog + flyer controls)
- [x] 721 total frontend tests passing
- [x] Local Chrome PDF verification (user confirmed)
- [x] Docker container PDF verification (Chromium 146, fonts + gradients
correct)
- [x] End-to-end test with real styled QR code (user confirmed
scannable)
- [ ] Manual: QR dialog → pick theme → Generate Flyer → verify PDF
- [ ] Manual: Admin referrals → flyer button → verify PDF

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 22:36:03 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 ab174f4d07 docs: add referral system lessons learned (#686)
5 gotchas from the referral system implementation added to
LESSONS-LEARNED.md.

@coderabbitai ignore

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 01:24:33 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 d471dbb47e fix: f-string syntax error blocking backend deploys (#685)
## Summary
- Fixes `SyntaxError: f-string expression part cannot include a
backslash` in `email_service.py:309`
- This has been blocking all backend Cloud Run deploys since PR #680
- Extracts inline HTML with escaped quotes to a variable before the
f-string

## Root Cause
PR #680 added a referral banner with `\"` escaped quotes inside an
f-string expression. Python 3.11 (Cloud Run runtime) doesn't allow
backslashes in f-string expressions — this is only allowed in Python
3.12+. Local tests pass because they don't exercise the import at module
level the same way uvicorn does.

## Testing
- [x] Verified `from backend.services.email_service import
get_email_service` succeeds
- [x] No logic changes — just moves HTML to a variable

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 20:03:02 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 7bf000eb81 feat(referral): add duration fields to admin and user referral UIs (#684)
## Summary

- Admin edit/create dialogs: now include discount duration and earning
duration fields
- Admin table: shows durations below rates (e.g. "10% / 20%" with "30d /
365d" below)
- User dashboard: new "Link Terms" section showing discount %, discount
window, kickback %, earning window

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 19:55:23 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 30bf8178aa feat(pricing): increase default price per track from $5 to $10 (#683)
## Summary
- Doubles base credit price from $5 to $10 per track
- Scales volume discount tiers proportionally (20%/30%/40% discounts
preserved)
- Updates all 33 i18n locale files, frontend savings calculation, and
E2E test fixtures

## Changes
- **Backend:** `CREDIT_PACKAGES` in `stripe_service.py` — 1cr: $10, 3cr:
$24, 5cr: $35, 10cr: $60
- **Frontend:** Savings calc base in `BuyCreditsDialog.tsx` (500 → 1000)
- **i18n:** `createYourselfPrice` "$5" → "$10" in all 33 locale files
- **Tests:** Updated price fixtures in `landing-page.spec.ts` and
`mobile.spec.ts`

## Testing
- [x] All 710 tests pass (`make test`)
- [x] No new logic — config value change only

## Review
- [x] CodeRabbit CLI review completed locally (no issues with pricing
changes)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 19:30:55 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 54c83e924c feat(referral): QR code generator with customizable styles (#682)
@coderabbitai ignore

## Summary
- Adds a QR code generator dialog to the referral dashboard
(`/app/referrals`)
- Referrers can customize dot style (6 options), corner frame (3),
corner dot (2), foreground/background colors, and center logo (NK logo,
microphone, music note)
- Live preview updates as options change
- Download as PNG or SVG
- Style preferences persist in localStorage

## Changes
- **New:** `frontend/components/referrals/QRCodeDialog.tsx` — QR
generator dialog using `qr-code-styling` library
- **Modified:** `frontend/components/referrals/ReferralDashboard.tsx` —
added QR Code button next to Copy Link
- **New:**
`frontend/components/referrals/__tests__/QRCodeDialog.test.tsx` — 10
unit tests
- **Modified:** `frontend/messages/en.json` — 22 new i18n strings
- **New dep:** `qr-code-styling` (client-side QR generation with style
customization)

## Test plan
- [x] 10 unit tests covering: rendering, style options, localStorage
persistence, download calls
- [x] 720/720 total tests passing
- [x] Next.js build compiles cleanly
- [x] ~70% coverage on new component (uncovered lines are
jsdom-incompatible DOM SVG APIs)
- [ ] Manual: open referral dashboard, click QR button, customize
styles, download PNG/SVG

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 19:17:43 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 817f2aa1bd feat(admin): add Apply Discount to user detail page (#681)
## Summary

Admins can apply a referral discount to any user from the user detail
page — useful for grandfathering existing users when raising prices.

## Changes

### Backend
- New endpoint: `POST /api/referrals/admin/apply-discount` — sets
discount %, duration, and referral code on any user
- `UserDetailResponse` now includes `referred_by_code`,
`referral_discount_expires_at`, `referral_code`

### Frontend
- "Apply Discount" button on user detail page (next to Add Credits)
- Dialog with discount %, duration (days), and referral code inputs
- Current discount status shown on user detail page if active

## Usage
- Go to Admin → Users → click a user → "Apply Discount"
- Set 50% off for 3650 days with code "loyalty" to grandfather a user

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 19:03:06 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 607c52bed3 feat(referral): embed referral code in magic link for cross-device attribution (#680)
## Summary

Referral attribution was failing because it relied solely on a browser
cookie. If the user opened the magic link email on a different device or
browser, the cookie wasn't there and attribution was lost.

## Fix: Triple-redundant referral code delivery

The referral code now travels through three independent paths:

1. **Cookie** (`x-referral-code` header) — same device, same browser
2. **Firestore** (magic link token doc) — stored server-side when magic
link is sent
3. **URL param** (`&ref=CODE` on the magic link) — travels with the
email itself

On verification, the backend checks all three and uses the first one
found.

## Changes

### Backend
- `SendMagicLinkRequest` + `MagicLinkToken` models: added
`referral_code` field
- `user_service.create_magic_link()`: accepts and stores referral_code
- `email_service.send_magic_link()`: appends `&ref=CODE` to URL, adds
discount banner to email
- `verify_magic_link`: reads referral code from header OR token doc OR
URL param

### Frontend
- `api.sendMagicLink()`: passes referral_code in request body
- Auth store `sendMagicLink`: reads referral cookie and passes it
- Verify page: reads `?ref=CODE` from magic link URL and stores cookie
before verification

## Testing
- 2952 backend tests pass
- Frontend build passes

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 17:22:53 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 3f00b08fb7 fix(referral): show display name on interstitial page (#679)
## Summary

The interstitial page wasn't showing the affiliate's display name
because of a field name mismatch — frontend type had `display_name` but
the API returns `referrer_display_name`.

## Changes

- `frontend/lib/types.ts` — `ReferralInterstitial.display_name` →
`referrer_display_name`, `code` → `referral_code`
- `frontend/app/[locale]/r/client.tsx` — Updated references to use
`data.referrer_display_name`

@coderabbitai ignore

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 16:44:45 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 af50b53cd8 feat(admin): add edit dialog for referral links (#678)
## Summary

Adds a pencil icon button on each row in the admin referrals table that
opens an edit dialog to modify display name, custom message, discount %,
and kickback %.

## Changes

- `frontend/app/admin/referrals/page.tsx` — Edit button + dialog with
form fields, uses existing `adminApi.updateReferralLink` endpoint

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 16:29:49 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 1cd747f170 fix(referral): compute and display referral discount status for users (#677)
## Summary

Three referral UX fixes:

1. **Discount visibility** — Backend now computes
`has_active_referral_discount` and returns discount details (percent,
expiry, referral code) in all UserPublic responses. Dashboard shows a
purple banner with discount info. Buy Credits dialog shows actual
percent + expiry.

2. **Existing user attribution** — Removed the `is_first_login` gate.
Any user who logs in with a referral code cookie and hasn't been
referred yet gets the referral attributed. This means existing users who
click a referral link will get the discount.

3. **Interstitial email signup** — Replaced the "Get Started" redirect
with an inline email input form. Users enter their email directly on the
interstitial page to receive a magic link. Shows "Check your email"
confirmation after sending. Already-logged-in users see a "Go to App"
CTA instead.

## Changes

### Backend
- `backend/api/routes/users.py` — New `_build_user_public()` helper
computes discount status. Attribution gate changed from `is_first_login`
to `not user.referred_by_code`.
- `backend/models/user.py` — Added `referral_discount_percent`,
`referral_discount_expires_at`, `referred_by_code` to UserPublic

### Frontend
- `frontend/app/[locale]/r/client.tsx` — Full rewrite: email signup form
with magic link, three states (form/sent/logged-in)
- `frontend/app/[locale]/app/page.tsx` — Referral discount banner on
dashboard
- `frontend/components/credits/BuyCreditsDialog.tsx` — Actual discount
percent + expiry date
- `frontend/lib/auth.ts` — Pass through referral fields
- `frontend/lib/types.ts` — New referral type fields

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 16:22:58 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 686402d643 fix(i18n): admin link 404 when accessed from locale-prefixed pages (#676)
## Summary
- `AuthStatus.tsx` imported `Link` from `@/i18n/routing` which
auto-prefixes hrefs with the current locale (e.g. `/en/admin`)
- Since `/admin` lives outside the `[locale]` route group, `/en/admin`
has no matching static page → 404
- Changed to `NextLink` from `next/link` which preserves the href as-is,
correctly navigating to `/admin`

## Test plan
- [x] `make test` passes (710 tests)
- [x] `next build` succeeds
- [ ] Verify clicking "Admin Dashboard" from `/en/app` navigates to
`/admin` (not `/en/admin`)

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 15:53:34 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 bb86722008 fix(referral): dedicated interstitial page instead of banner overlay (#675)
## Summary

Replace the banner overlay (which looked terrible floating over the
landing page) with a proper full-screen interstitial page.

## Flow

1. `nomadkaraoke.com/r/bevvy` → Worker redirects to
`gen.nomadkaraoke.com/en/r?code=bevvy`
2. Static `/en/r` page loads, reads `?code=` param, fetches referral
data from API
3. Full-screen interstitial: gift icon, referrer name, custom message,
discount details, "Get Started" CTA
4. User clicks "Get Started" → landing page (referral code already
stored in cookie)

## Changes

- New: `frontend/app/[locale]/r/page.tsx` + `client.tsx` — dedicated
interstitial page
- Reverted: `frontend/components/ReferralCapture.tsx` — back to silent
cookie-setter (no banner)
- Cleaned: `frontend/public/_redirects` — removed broken `:splat` rules

Also pushed: `public-website/worker.js` updated to redirect to
`/en/r?code=` instead of `/en/?ref=`

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 22:44:22 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 74eac6f5ee feat(referral): show interstitial banner on landing page for referral links (#674)
## Summary

When a user arrives via a referral link (`nomadkaraoke.com/r/CODE` →
`gen.nomadkaraoke.com/en/?ref=CODE`), the landing page now shows a
dismissible banner with the referrer's name, custom message, and
discount details.

## Changes

- `frontend/components/ReferralCapture.tsx` — Enhanced from silent
cookie-setter to interstitial banner. Fetches referral data from API,
shows referrer name/message/discount in a fixed top banner with slide-in
animation. Dismissible via X button.

## How it works

1. User clicks `nomadkaraoke.com/r/bevvy`
2. Cloudflare Worker redirects to `gen.nomadkaraoke.com/en/?ref=bevvy`
3. `ReferralCapture` component detects `?ref=bevvy`, stores cookie,
fetches interstitial data
4. Banner slides in showing "You've been referred! Referred by [name].
[message]. Enjoy 10% off..."
5. User dismisses or scrolls to sign up — referral code is already
stored for checkout

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 21:42:21 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 d142156138 fix: use correct :splat syntax for referral redirect (#673)
## Summary

The referral redirect `/r/:code → /en/?ref=:code` was passing the
literal string `:code` instead of the actual referral code. Cloudflare
Pages uses `*` + `:splat` syntax, not named parameters.

## Fix

`/r/:code /en/?ref=:code 302` → `/r/* /en/?ref=:splat 302`

@coderabbitai ignore

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 19:03:20 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 0353f5890d fix: use redirect for /r/CODE referral links instead of dynamic route (#672)
## Summary

The previous approach (catch-all `[[...code]]` route) doesn't work with
Next.js static export on Cloudflare Pages — dynamic paths 404 because no
HTML file exists for them.

## Fix

Replace the dynamic route with a simple Cloudflare Pages `_redirects`
rule:
- `/r/CODE` → `/en/?ref=CODE` (302 redirect)
- The `ReferralCapture` component on the landing page already captures
`?ref=CODE` and stores it in a cookie

This means referral links redirect to the landing page with the code as
a query param, which is simpler and fully compatible with static export.

## Tradeoff

The custom interstitial page (with referrer name/message) is removed for
now. The referral still works — the code gets stored and applied at
checkout. The interstitial can be added back later as a component on the
landing page that detects `?ref=` and shows the referral info.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 18:57:51 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 c97a340389 fix: add Cloudflare Pages rewrites for referral /r/CODE routes (#671)
## Summary

Referral interstitial pages at `/r/CODE` return 404 because Cloudflare
Pages doesn't know to serve the catch-all page for dynamic paths.

## Changes

Add `_redirects` rules:
- `/r/*` → `/en/r/:splat` (302) — bare paths get locale prefix
- `/{locale}/r/*` → `/{locale}/r/index.html` (200 rewrite) — all 33
locales get SPA-style fallback

## Testing
- Fixes `gen.nomadkaraoke.com/r/bevvy` (currently 404)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 18:42:24 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 1154adc150 feat(admin): add referral links management page (#670)
## Summary

- Add Referrals page to admin dashboard for managing referral links
- Stats overview: total links, clicks, signups, earnings
- Links table with enable/disable toggle and copy-to-clipboard
- Create Vanity Link dialog with custom discount/kickback rates

## Changes

- `frontend/app/admin/referrals/page.tsx` — New admin referrals page
- `frontend/components/admin/admin-sidebar.tsx` — Added Referrals nav
item
- `frontend/app/admin/layout.tsx` — Added breadcrumb support
- `frontend/lib/api.ts` — Added 3 admin referral API methods

## Testing
- [x] Frontend build passes
- [x] 710 frontend unit tests pass
- [x] No backend changes

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 18:26:32 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 80cb53f3a2 feat: referral link system with discounts and Stripe Connect payouts (#669)
## Summary

- Add a referral system where any user gets a shareable link (`/r/CODE`)
with a personalized interstitial page
- Referred users get a configurable discount (default 10%) on credit
purchases for 30 days
- Referrers earn a cash kickback (default 20% of revenue) paid out via
Stripe Connect at $20 threshold
- Admin can create vanity links with custom rates and manage all
referral links

## Changes

### Backend
- **New models** (`backend/models/referral.py`): ReferralLink,
ReferralEarning, ReferralPayout + API request/response models
- **New service** (`backend/services/referral_service.py`): Link CRUD,
attribution, discount detection, earnings recording, Stripe Connect
onboarding, auto-payout
- **New routes** (`backend/api/routes/referrals.py`): 7 endpoints —
public interstitial, user dashboard, link updates, Connect onboarding,
admin vanity/list/update
- **Stripe integration**: Coupon creation/lookup, Connect Express
accounts, transfers
- **Webhook hook**: Referral earnings recorded on credit purchases
- **Auth hook**: Referral attribution on first magic link verification
via `x-referral-code` header
- **User model**: Added referral fields (referral_code,
referred_by_code, referred_at, discount_expires_at,
stripe_connect_account_id)

### Frontend
- **Interstitial page** (`/r/CODE`): Shows referrer's name, custom
message, discount details
- **Referral dashboard** (`/app/referrals`): Link sharing, stats,
earnings, payout history, Stripe Connect CTA
- **Discount badge**: Shown in Buy Credits dialog during active referral
window
- **Referral capture**: Cookie + localStorage via `/r/CODE` route and
`?ref=CODE` query param
- **Auth integration**: Passes referral code on magic link verification,
clears after use
- **i18n**: 33 new translation keys in `referrals` namespace

### Documentation
- Design spec: `docs/archive/2026-04-04-referral-system-design.md`
- Implementation plan: `docs/archive/2026-04-04-referral-system-plan.md`
- Updated `docs/README.md` status table and `docs/API.md` with referral
endpoints

## Testing
- 68 referral-specific tests (models, service, routes) — all passing
- 2935 total backend tests — zero regressions
- 710 frontend tests — all passing
- Covers: link management, attribution, self-referral blocking, discount
detection, earnings calculation, payout threshold, edge cases (rounding,
zero amounts), all API endpoints

## Review
- [x] CodeRabbit CLI review completed locally (11 findings, 7 fixed)
- [x] Test review completed (P0/P1 gaps identified and filled)
- [x] Docs reviewed and updated

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 17:39:18 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 fb4bcc7737 fix(i18n): locale-prefix review links and protect auth from redirect aborts (#668)
## Summary
- Hash-based review/instrumental/audio-edit links in JobCard were
missing the locale prefix (`/en/`), causing a double-redirect through
`LocaleRedirect` that aborted the module-level `fetchUser()` call and
wiped the user's auth session
- Added `useLocale()` to JobCard so links include the locale prefix,
avoiding the redirect entirely
- Made `fetchUser()` resilient to `AbortError` so navigation-triggered
fetch aborts don't clear auth

## Test plan
- [x] Unit tests: 710 passing (3 new auth abort resilience tests)
- [x] JobCard test expectations updated for locale-prefixed URLs
- [x] CodeRabbit review: no findings
- [ ] Manual: verify clicking "Review Lyrics" on dashboard navigates
directly to review page without logout

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 14:30:27 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 48265d398f fix: edit with metadata change now properly regenerates screens (#667)
## Summary
- Fix invalid state transition when editing a completed track with
artist/title changes
- The edit flow now transitions to `LYRICS_COMPLETE` (instead of
`AWAITING_REVIEW`) when metadata changes, allowing the screens worker to
properly regenerate screens via the valid `LYRICS_COMPLETE →
GENERATING_SCREENS` path
- Clear stale `file_urls.screens` entries to prevent retry logic from
skipping screen generation

## Root cause
When a user edits a completed track and changes artist/title:
1. Edit flow correctly deleted screen files from GCS (they had old text
baked in)
2. Edit flow transitioned to `AWAITING_REVIEW` and triggered screens
worker
3. Screens worker tried `AWAITING_REVIEW → GENERATING_SCREENS` —
**invalid transition**
4. Screens never regenerated, but `file_urls.screens` had stale
references
5. Retry saw stale URLs, skipped screen generation → 404 on `title.png`

## Test plan
- [x] All 16 edit track tests pass (including 2 new regression tests)
- [x] Full test suite passes (707 frontend + backend tests)
- [x] Manually verified fix on job `4117228f` — screens regenerated and
job completed successfully

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 18:04:01 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 ab330a3ed6 feat(i18n): scale to 33 languages with RTL support (#666)
## Summary
- Scale i18n from 3 languages (en/es/de) to 33 languages
- Upgrade translation pipeline: async parallel execution (5 concurrent)
+ incremental delta detection
- Add RTL support (`dir="rtl"`) for Arabic and Hebrew
- Replace LanguageSwitcher `<select>` with grouped scrollable dropdown
(flags + native names)
- RTL CSS audit: convert directional properties to logical equivalents
- 30 new frontend translation files (1062 keys each) + 30 backend files
(274 keys each)

### New languages
Portuguese, French, Japanese, Korean, Chinese, Italian, Dutch, Polish,
Turkish, Russian, Thai, Indonesian, Vietnamese, Filipino, Hindi, Arabic,
Swedish, Norwegian, Danish, Finnish, Czech, Romanian, Hungarian, Greek,
Hebrew, Malay, Ukrainian, Croatian, Slovak, Catalan

## Test plan
- [x] `next build` — 285 static pages generated (11 pages x 33 locales -
some deduped)
- [x] All translation files have 100% key parity
- [ ] Post-deploy: verify Japanese, Arabic (RTL), Portuguese render
correctly
- [ ] Post-deploy: verify LanguageSwitcher shows all 33 options

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 05:08:47 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 84c879a352 fix(i18n): pass locale to cross-app links (#665)
## Summary
- Landing page footer link to nomadkaraoke.com now includes locale
prefix
- Order success "Back to Nomad Karaoke" link now includes locale prefix
- Ensures users stay in their chosen language when navigating between
apps

## Test plan
- [x] Build passes
- [ ] Post-deploy: Spanish user on order success page → "Back to Nomad
Karaoke" → nomadkaraoke.com/es/

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 03:17:20 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 4fd701ad21 fix(i18n): add LanguageSwitcher to landing page and dashboard (#664)
## Summary
- Add `LanguageSwitcher` dropdown to the landing page nav bar (before
theme toggle)
- Add `LanguageSwitcher` to the dashboard header (before AuthStatus)
- Previously it was only in `AppHeader` used on review pages — users on
the landing page and dashboard had no way to switch language

## Test plan
- [x] `next build` succeeds
- [ ] Post-deploy: verify language dropdown visible on landing page
- [ ] Post-deploy: verify language dropdown visible on dashboard when
logged in

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 02:53:14 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 9c1e951fd0 feat(i18n): localize email bodies + non-locale route redirects (#663)
## Summary
- **P1**: Replace 5 duplicate non-locale route pages (`/app`,
`/app/jobs`, `/auth/verify`, `/payment/success`, `/order/success`) with
thin `LocaleRedirect` component — detects locale from
localStorage/navigator and redirects to `/{locale}/...`
- **P2**: Localize all user-facing email bodies (7 inline types + 4
template defaults + shared footer). Adds ~80 translation keys to
en/es/de.json. Template service now tries locale-specific GCS paths
before falling back to localized defaults.
- Fix admin dropdown link using locale-aware Link (should use NextLink
since `/admin` is outside locale routing)

### Email types localized
Magic link, welcome, credits added, review reminder, review expired,
made-for-you order confirmation, credit denied, action reminder alert,
job completion/action needed/YouTube upload template defaults, shared
footer

### Not localized (intentionally)
Admin-only: feedback notification, credit review, admin order
notification

## Test plan
- [x] Backend: 2668 tests pass (139 skipped)
- [x] Frontend: `next build` succeeds with all locale directories
- [x] CodeRabbit: 1 finding fixed (admin dropdown link)
- [ ] Post-deploy: verify Spanish user receives localized email body
(magic link flow)
- [ ] Post-deploy: verify `/app` redirects to `/{locale}/app`

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 02:28:17 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 648a26d462 fix(i18n): locale-aware navigation for internal links (#662)
## Summary
- Replace `next/navigation` and `next/link` imports with
`@/i18n/routing` equivalents in 13 user-facing components and pages so
`router.push()` and `<Link>` automatically include the locale prefix
(`/es/`, `/de/`)
- Without this fix, users on Spanish/German routes were kicked back to
non-locale paths on every internal navigation click
- Admin navigation (`/admin`) correctly excluded from locale routing
(uses `next/link` / `window.location`)
- Fix Jest ESM transform for `next-intl` transitive deps and add global
`@/i18n/routing` mock

## Test plan
- [x] `next build` succeeds — all 3 locale directories generated
(`/en/`, `/es/`, `/de/`)
- [x] 39 Jest test suites / 707 tests pass
- [x] CodeRabbit review: clean, no findings
- [ ] Post-deploy: verify `/es/app` links navigate to `/es/` prefixed
routes
- [ ] Post-deploy: verify admin link from header goes to `/admin` (no
locale prefix)

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 01:34:57 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 b486fba790 fix: sanitize artist/title for CDG/TXT packaging (trailing period) (#661)
## Summary
- Fixes TXT packaging failure when song title ends with a period (e.g.
`Lofi Ver.`)
- Root cause: orchestrator's `sanitize_filename()` strips trailing
periods but CDG generator's internal sanitization does not, causing MP3
filename mismatch
- Now passes sanitized artist/title to CDG generator and reuses the
actual MP3 path for TXT packaging
- Adds 2 regression tests for the trailing-period filename scenario

## Details
Job `4117228f` (`Gió - JanK x Quanvrox「Lofi Ver.`) failed 5 times with
"TXT generation was enabled but failed" because:
1. Orchestrator sanitized title to `Lofi Ver` (no period)
2. CDG generator kept `Lofi Ver.` (with period) in ZIP filenames
3. TXT step looked for `...Lofi Ver (Karaoke).mp3` but file was `...Lofi
Ver. (Karaoke).mp3`

## Test plan
- [x] 67 orchestrator tests pass (including 2 new regression tests)
- [x] `test_cdg_packaging_uses_sanitized_artist_title` - verifies
sanitized values passed to CDG
- [x] `test_txt_uses_mp3_path_from_cdg_generation` - verifies TXT uses
actual CDG MP3 path
- [ ] Retry job 4117228f after deployment

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 01:27:44 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 04b5d16774 feat(i18n): add real Spanish and German translations (Gemini pipeline) (#660)
@coderabbitai ignore

## Summary
- Runs the two-pass Gemini 3.1 Pro translation pipeline on all English
strings
- **Frontend:** 1062/1062 keys translated for Spanish and German
- **Backend:** 186/186 keys translated for Spanish and German
- Uses karaoke-specific glossary (55 terms) for consistency
- 100% key coverage, valid JSON for all locales

## Sample translations
- "Create a Karaoke Video for Any Song" → "Crea un video de karaoke para
cualquier canción" (ES) / "Erstelle ein Karaoke-Video für jeden Song"
(DE)
- "Job not found" → "Trabajo no encontrado" (ES) / "Auftrag nicht
gefunden" (DE)

## Test plan
- [x] Build succeeds with real translations (45 pages generated)
- [ ] Post-deploy: visit /es/ and verify Spanish content
- [ ] Post-deploy: visit /de/ and verify German content
- [ ] Post-deploy: switch language via LanguageSwitcher

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 04:48:37 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 6ca0a58c14 feat(i18n): Phase 3 PR 4 — email localization, URL prefixes, user locale (#659)
@coderabbitai ignore

## Summary
- Final Phase 3 PR — localizes email subjects, adds locale prefix to all
frontend URLs in emails, persists user locale on login
- Adds `locale` parameter (default "en") to all 10 user-facing email
methods
- Job notification service looks up user locale from Firestore before
sending
- Stale review processor passes locale to reminder/expiry emails
- On magic link verification, user's locale is stored to Firestore for
async email use
- 11 new email subject translation keys in
`backend/translations/en.json`

## Test plan
- [x] 2897 backend tests pass (1 pre-existing flaky test)
- [x] 6 test files updated for locale-prefixed URL assertions
- [ ] Post-deploy: verify magic link email has /en/ prefix in verify URL
- [ ] Post-deploy: verify review reminder email has /en/ prefix in
review URL

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 04:08:29 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 9e9c4582a6 feat(i18n): Phase 3 backend — i18n utility, 209 strings, all routes localized (#658)
@coderabbitai ignore

## Summary
- Backend i18n for karaoke-gen (Phase 3, PR 3 of 4)
- Adds `backend/i18n.py` utility with `t()`,
`get_locale_from_request()`, `get_locale_prefix()`
- **209 strings** extracted into `backend/translations/en.json` across
13 namespaces
- All 12 user-facing route files converted to use `t(locale,
"namespace.key")`
- Push notification titles/bodies localized
- `InsufficientCreditsError` handler localized
- User model gains `locale` field for async process locale preference
- 22 new i18n unit tests + 3 existing tests updated

## Changes by file
- **New:** `backend/i18n.py`, `backend/translations/{en,es,de}.json`,
`backend/tests/test_i18n.py`
- **Modified:** 12 route files (auth, jobs, users, review, audio_search,
file_upload, catalog, themes, tenant, push, rate_limits, admin)
- **Modified:** `backend/main.py` (exception handler),
`backend/services/push_notification_service.py`,
`backend/models/user.py`

## Test plan
- [x] 22 new i18n unit tests pass
- [x] Full backend test suite passes (2895+ tests)
- [x] Frontend tests unaffected (707 pass)
- [ ] Post-deploy: verify API returns English strings (Accept-Language:
en)
- [ ] Post-deploy: verify Accept-Language: es returns Spanish when
translations are ready

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:15:52 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 cba0e00bcc fix(i18n): add DefaultIntlProvider for non-locale routes (#657)
@coderabbitai ignore

## Summary
- Hotfix for #656 — shared components using `useTranslations()` crashed
on non-locale routes (`/app`, `/admin`) because they had no
`NextIntlClientProvider`
- Adds `DefaultIntlProvider` (client component) to root layout that
provides English translations as fallback
- The `[locale]/layout.tsx` overrides this for locale-prefixed routes

## Test plan
- [x] All 707 frontend tests pass
- [x] Static build generates all 45 pages
- [ ] Verify `/app` dashboard loads without errors
- [ ] Verify `/admin` pages load without errors
- [ ] Verify `/en/app` still works with locale-specific provider

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 22:16:29 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 8c1dfe5460 feat(i18n): Phase 3 frontend — next-intl infrastructure + 1062 strings (en/es/de) (#656)
@coderabbitai ignore

## Summary
- Phase 3 of the i18n multilingual initiative — makes the karaoke-gen
frontend fully multilingual (English, Spanish, German)
- Installs `next-intl` with `[locale]` path-based routing,
LanguageSwitcher, and `Accept-Language` API header
- Extracts **1,062 strings** from ~70 components into `messages/en.json`
across 31 namespaces
- Wires `useTranslations()` into all user-facing components (landing,
dashboard, auth, job flow, lyrics review, instrumental review, audio
editor, etc.)
- Admin pages (`/admin/*`) stay outside locale routing — intentionally
excluded from i18n scope
- Static export generates `/en/`, `/es/`, `/de/` directories (45 total
pages) for Cloudflare Pages
- `es.json`/`de.json` are English placeholders — translation via Gemini
pipeline pending
- Translation pipeline (`scripts/translate.py`) and glossary (55
karaoke-specific terms) included

## Key architecture decisions
- Root layout keeps ThemeProvider/TenantProvider (needed by admin pages
outside `[locale]`)
- Only NextIntlClientProvider added to `[locale]/layout.tsx`
- `job-status.ts` labels converted to translation keys; consumers
translate via `useTranslations('jobStatus')`
- Jest mock loads actual `en.json` so existing tests validate i18n
wiring without assertion changes

## Test plan
- [x] All 707 frontend tests pass (39 suites)
- [x] Static build generates all 45 pages across 3 locales
- [x] Admin pages render at `/admin/` without locale prefix
- [x] CodeRabbit review: 4 issues fixed, 7 pre-existing noted
- [ ] Post-deploy: verify `/en/` landing page renders correctly
- [ ] Post-deploy: verify LanguageSwitcher redirects to `/es/` and
`/de/`
- [ ] Post-deploy: verify `/admin/` still works without locale context

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 21:50:53 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 105a5b28fd fix: default dashboard to all jobs, fix E2E completion detection (#655)
## Summary
- Default the dashboard status filter from "active" to "all" so
completed jobs remain visible (existing localStorage preferences
preserved)
- Add recovery logic to E2E happy path test: after 10 consecutive "card
unavailable" polls, automatically switch the filter dropdown to "All
statuses"
- Bump version to 0.157.4

**Root cause:** The daily E2E test (run #126) timed out at Step 9
because the "active" filter uses `hide_completed: true` at the API level
— when the job completed, the card vanished from the DOM and the test
retried for ~25 min until timeout. The generator itself was working
fine.

## Test plan
- [x] All 707 tests pass (`make test`)
- [x] CodeRabbit review clean (1 finding fixed)
- [ ] Next scheduled E2E run should pass with these changes

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-03 11:41:48 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 a0e6448e7b fix: make E2E job card polling resilient to transient DOM changes (#654)
## Summary
- E2E happy path test failed because the job card DOM element briefly
disappeared during a status transition (step 9/10 "Packaging files"),
causing an unhandled `TimeoutError` on `jobCard.textContent()`
- Wrapped all `textContent()` calls in polling loops with try/catch and
a 15s explicit timeout so transient DOM disappearances retry instead of
crashing the test
- Applied fix to both `happy-path-real-user.spec.ts` and
`demo-recording.spec.ts`

Fixes:
https://github.com/nomadkaraoke/karaoke-gen/actions/runs/23887007717

## Test plan
- [x] Verified TypeScript has no new errors (all existing errors are
pre-existing Playwright type issues)
- [ ] Next scheduled E2E run will validate the fix against production

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-02 03:11:12 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 fbe3edceec fix: video worker downloads PNG instead of MOV for screens (#650)
## Summary

Follow-up to #647. The video worker was hardcoded to download
`title.mov`/`end.mov` from GCS, but those no longer exist after the
screens_worker change. 404 error on all job retries.

- `video_worker.py`: download PNG (or MOV for legacy), validate with PNG
fallback
- `video_worker_orchestrator.py`: `_screen_ext()` helper for dynamic
file extension

## Test plan

- [x] All video unit tests pass
- [ ] Retry job f0d51a0a after deploy

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 23:39:57 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 ead074505b fix: skip files >3GB in divebar sync to prevent OOM kills (#653)
## Summary

- Skip files larger than 3GB in divebar file sync — the VM has 4GB RAM
and buffers files in memory, so a 4.3GB MKV was causing OOM kills every
run
- Sets `gcs_path = 'skipped:too_large'` so the file won't be retried
nightly (the `WHERE gcs_path IS NULL` query excludes it)
- Only 1 file currently affected: `IMBK-0068 - Zolas - Ancient Mars.mkv`
(4.3 GB)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 23:24:56 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 7e93d62457 fix: thread-safe Drive service in divebar file sync (#651)
## Summary

- Use `threading.local()` for Drive API service — `googleapiclient` is
not thread-safe and sharing across `ThreadPoolExecutor` workers caused
SSL errors, bad file descriptors, and heap corruption (`free():
corrupted unsorted chunks`)
- This is why ~200 files were permanently stuck with `gcs_path=NULL` —
different files failed each run depending on thread timing

**Evidence from VM logs:**
```
ERROR Failed files/NOMAD-1346.zip: 'email'
ERROR Failed files/Shark Attack.zip: [Errno 9] Bad file descriptor
ERROR Failed files/KGAL-0004.zip: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC]
ERROR Failed files/LSWK-0024.zip: [SSL: WRONG_VERSION_NUMBER]
free(): corrupted unsorted chunks
```

## Test plan

- [x] Existing unit tests pass
- [ ] Next nightly file sync run should have 0 failures (was ~200/run)
- [ ] Monitor `gcs_path IS NULL` count — should drop to only genuinely
new files

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 23:04:54 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 1c846ebae7 fix: preserve gcs_path in divebar catalog sync (#649)
## Summary

- **index_builder.py**: replaced `WRITE_TRUNCATE` with staging table +
`MERGE` SQL to preserve `gcs_path` values during daily catalog index
updates
- **divebar_file_sync.py**: added `blob.exists()` check before
downloading from Drive — if a file is already in GCS, just updates
`gcs_path` in BigQuery without re-downloading
- Added unit tests verifying the MERGE approach preserves `gcs_path`
correctly

**Root cause**: The daily 2 AM index sync was atomically replacing the
entire `divebar_catalog` BigQuery table, wiping all `gcs_path` values.
The 3 AM file sync then saw all ~48K files as unsynced and re-downloaded
the entire ~779 GB catalog from Google Drive every night.

## Test plan

- [x] Unit tests for staging+MERGE behavior (5 tests pass)
- [ ] Verify staging table auto-creates on first run (BigQuery creates
on WRITE_TRUNCATE with schema)
- [ ] Monitor next nightly sync — file_sync should report "recovered
from GCS" instead of downloading
- [ ] Confirm gcs_path values persist across consecutive index runs

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 22:31:42 +00:00
Andrew Beveridge ☄️andClaude Sonnet 4.6 c02fdc8d9f Fix GCE cache invalidation and empty corrections handling (#648)
## Summary
- **GCE style asset disk cache** now validates file size against GCS on
cache hit, re-downloading when content has changed at the same URI.
Transient GCS errors gracefully fall back to cached version.
- **Review complete endpoint** no longer writes
`corrections_updated.json` when no corrections data is present (root
cause fix for KeyError in render workers).
- **Both GCE and Cloud Run render workers** validate
`corrections_updated.json` has a `corrections` key before processing,
falling back to base result gracefully (defensive fix).

## Test plan
- [x] Added `test_cache_hit_redownloads_on_size_mismatch` — verifies
stale cache replaced
- [x] Updated `test_cache_hit_skips_download` for size validation flow
- [x] Added `test_gce_worker_skips_empty_corrections` — verifies empty
dict doesn't crash
- [x] Added `TestCompleteReviewSkipsEmptyCorrections` — 4 tests covering
guard conditions
- [x] All 2875 tests pass (8 pre-existing emulator failures unrelated)

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-31 18:03:26 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 3bc1fcb250 fix: move screen video generation to GCE encoding worker (#647)
## Summary

- Screen MOV generation (4K H.264) was silently failing on Cloud Run
after right-sizing to 2Gi (PR #640), producing 36-byte empty shells that
broke all downstream encoding
- `screens_worker` now only generates PNG/JPG images (Pillow, no FFmpeg)
- GCE encoding worker generates MOV from PNG before encoding, with
proper `subprocess.run()` error handling
- Fixed `os.system()` in `video_generator.py` to use `subprocess.run()`
so failures are no longer silent

## Root Cause

PR #640 reduced Cloud Run from 8CPU/16Gi to 2CPU/2Gi. The
`VideoGenerator._create_video_from_image()` used `os.system(ffmpeg ...)`
which OOMs silently for 4K H.264 encoding, producing 36-byte empty MOV
shells. All jobs since March 31 are affected.

## Test plan

- [x] 2875 tests pass (8 pre-existing emulator failures unrelated)
- [x] CodeRabbit: no findings
- [ ] Re-run failed job f0d51a0a after deploy to verify fix
- [ ] Verify admin "Regen screens" button works (tested flow: deletes
old screens → worker generates PNG/JPG only → GCE generates MOV from
PNG)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 21:44:53 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 adae022fbf Fix encoding progress callback causing invalid state transition (#646)
## Summary
- Fix `encoding -> encoding` invalid state transition that caused job
7bbd0d9d to fail
- The GCE encoding `progress_callback` was using `transition_to_state()`
to update progress while already in ENCODING state, which the state
machine correctly rejects as a self-transition
- Changed to `update_job()` which updates progress/message without
validating state transitions

## Test plan
- [x] All 2875 backend tests pass (8 pre-existing emulator failures
unrelated to change)
- [x] State machine integration tests pass
- [x] Encoding interface contract tests pass

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 21:19:47 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 9ccad7c6d3 feat: service health status modal with blue-green deploy details (#645)
## Summary
- Clickable footer opens a modal showing all service health, versions,
deploy times, and PR links
- Progressive disclosure: basic info for everyone, blue-green encoder
details for admins
- New aggregated `/api/health/system-status` endpoint (single fetch
instead of 4 separate calls)
- CI captures git SHA and PR info, passes to frontend and backend builds

## Changes
- **Backend**: New `/api/health/system-status` endpoint with optional
admin auth for blue-green Firestore config
- **Backend**: Build metadata env vars (COMMIT_SHA, PR_NUMBER, PR_TITLE,
STARTUP_TIME) in version module
- **Frontend**: New `SystemStatusModal` component with service cards and
expandable blue-green section
- **Frontend**: Footer version line is now clickable to open the modal
- **Frontend**: Build metadata env vars passed through next.config.mjs
- **CI**: Both frontend and backend deploy jobs capture and pass git SHA
+ PR info

## Testing
- [x] 3 unit tests for system-status endpoint (public, offline, admin
blue-green)
- [x] TypeScript compilation clean (no new errors)
- [x] CodeRabbit CLI reviewed

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 15:42:58 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 8e2a949a29 fix: complete blue-green deploy test payload and status check (#644)
## Summary
- Fix the deploy health check test payload and status value mismatches
that prevented blue-green swaps from ever completing
- Follow-up to #642 which fixed the error handling / rollback conditions

## Changes
- **Missing required fields**: `encode-preview` requires
`audio_gcs_path` and `output_gcs_path` but the health check only sent
`job_id`, `ass_gcs_path`, and `duration_seconds` (caused HTTP 422).
Added a 3-second test audio file and proper output path.
- **Status polling mismatch**: Polling checked for `"completed"` but the
GCE worker returns `"complete"` — would always time out even on success.
- **Submit response mismatch**: Validated against `"completed"` which is
never returned — endpoint returns `"accepted"`, `"cached"`, or
`"in_progress"`.

## Root Cause
The test payload was written based on the plan doc which had an
incomplete request schema. The `audio_gcs_path` and `output_gcs_path`
fields have been required since the endpoint was first created (PR #200,
Jan 2026). The status string mismatch (`"completed"` vs `"complete"`)
was likely a typo that was never caught because the request always
failed before reaching the polling stage.

## Testing
- Previous deploy (#642) confirmed the 422 error and verified rollback
now triggers correctly
- This fix addresses the payload so the encode test can actually succeed
and the swap can complete
- Real validation: next deploy after merge

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 14:07:38 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 0464d3edb3 Fix render_video GCE endpoint import paths (#643)
@coderabbitai ignore

## Summary

- Fix broken import paths in GCE worker's `/render-video` endpoint that
caused `ImportError: No module named 'backend.output_generator'` in
production
- Fix incorrect API signatures for OutputGenerator, CountdownProcessor,
CorrectionResult, load_styles_from_gcs
- Update tests to mock correct module paths

## Root cause

The subagent that implemented Task 2 (PR #641) used invented module
paths (`backend.output_generator`,
`backend.services.countdown_processor`, etc.) instead of the actual
karaoke-gen wheel paths
(`karaoke_gen.lyrics_transcriber.output.generator`, etc.). The tests
also mocked the wrong paths, so they passed locally but the code failed
in production.

## Changes

- `backend/services/gce_encoding/main.py` — Rewrote `run_render_video()`
to use correct imports and API calls matching `render_video_worker.py`
- `tests/unit/test_gce_render_video_endpoint.py` — Rewrote
TestRunRenderVideo to mock correct module paths

## Test plan

- [ ] 18 unit tests pass (verified locally with
`GOOGLE_CLOUD_PROJECT=""` to simulate CI)
- [ ] E2E happy path test after merge validates full pipeline

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 14:01:36 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 5dfaa3f73f fix: blue-green deploy health check and rollback conditions (#642)
## Summary
- Fix blue-green deploy workflow that has been silently failing since
v0.151.0 (March 25)
- Every deploy has fallen through to `ensure_latest_wheel()` hot-update
instead of tested blue-green swap

## Changes
- **Health check curl**: Replace `curl -sf` (which exits immediately on
HTTP errors before setting output) with `curl -s -o file -w
'%{http_code}'` to capture HTTP status without early exit
- **Status polling curl**: Remove `-f` flag and add fallback for failed
status checks
- **Rollback condition**: Change `passed == 'false'` to `passed !=
'true'` to catch unset/skipped cases (defense in depth)

## Root Cause
`curl -sf` with `-f` (fail silently) exits non-zero on HTTP errors.
Combined with GitHub Actions' `set -e`, the script exits before writing
`passed=false` to `$GITHUB_OUTPUT`. Both the rollback (`== 'false'`) and
swap (`== 'true'`) conditions evaluate false against the unset output,
creating a silent no-op.

## Testing
- [x] CI workflow syntax validated
- [x] CodeRabbit CLI review completed locally (no issues in changed
file)
- Real validation: next deploy will exercise the fixed workflow

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 13:46:30 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 1cf6b62398 Route render_video to GCE encoding worker (#641)
@coderabbitai ignore

## Summary

- Route `render_video_worker` (OutputGenerator/ffmpeg lyrics rendering)
to the GCE encoding worker instead of running on Cloud Run
- Fixes OOM crashes after Cloud Run memory was right-sized to 2Gi
- Adds style asset disk cache on GCE worker to avoid repeated downloads
of the same default theme
- Preserves local fallback when GCE is disabled

## Changes

### GCE Worker (`backend/services/gce_encoding/main.py`)
- New `/render-video` endpoint with `RenderVideoRequest` model
- `run_render_video()` — downloads corrections/audio/styles from GCS,
runs OutputGenerator, uploads outputs
- `download_with_cache()` — SHA-256 keyed disk cache for style assets
(fonts, backgrounds)
- `metadata` field added to `JobStatus` model for countdown padding info

### EncodingService (`backend/services/encoding_service.py`)
- `submit_render_video_job()` — POST to `/render-video` with retry logic
- `render_video_on_gce()` — submit + wait convenience method

### render_video_worker (`backend/workers/render_video_worker.py`)
- Conditional GCE routing when `encoding_service.is_enabled`
- Cloud Run does zero downloads in GCE path — constructs GCS paths from
Firestore metadata
- Existing local path preserved unchanged as fallback

## Test plan

- [ ] 18 new unit tests covering cache, endpoint, service methods, and
routing logic (all pass)
- [ ] Full test suite: 1164 passed, 0 failures
- [ ] CodeRabbit review: 1 bug caught and fixed (missing metadata in
cached response)
- [ ] E2E happy path CI validates full pipeline after merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 16:58:00 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 82e9becc7c chore: right-size Cloud Run + fix divebar-sync auto-shutdown (#640)
## Summary
- Right-size karaoke-backend Cloud Run: concurrency 1→80, minScale 4→1,
CPU 8→2, memory 16Gi→2Gi, enable cpu-throttling, maxScale 50→20
- Fix divebar-sync VM: add missing `shutdown -h now` after file sync
completes (VM was staying on 24/7)

## Cost Impact
- Backend right-sizing: ~$1,800/mo savings
- Divebar-sync fix: ~$32/mo savings
- Part of broader GCP cost reduction
(docs/2026-03-31-cost-reduction-design.md)

## Context
The backend is an I/O-bound API orchestrator (Firestore queries + Cloud
Run job dispatch). Heavy processing was moved to Cloud Run Jobs and a
dedicated GCE encoding worker months ago, but the Cloud Run config was
never updated. Concurrency=1 forced one-request-per-instance, and
minScale=4 kept 4 instances of 8CPU/16Gi running 24/7.

Changes have already been applied live via `gcloud run services update`
and verified with health checks.

## Test plan
- [x] All 6 Cloud Run settings updated incrementally with health checks
between each
- [x] `curl https://api.nomadkaraoke.com/api/health` returns 200 after
all changes
- [x] Divebar-sync VM stopped and metadata updated with new startup
script
- [ ] Monitor billing data over next 24-48h to confirm cost reduction

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 10:00:16 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 ca0d5507b2 Business continuity: DR backups, read-only SA, deny policies (#639)
## Summary

- Two-tier disaster recovery: GCP-native protections (PITR, versioning,
deny policies) + cross-cloud AWS S3 backups
- Nightly Cloud Function exports Firestore, BigQuery, and GCS job files
to AWS S3
- Read-only Claude SA (`claude-readonly`) and break-glass SA for
emergency access
- IAM deny policy blocking `buckets.delete` and `datasets.delete` for
all principals
- External backup freshness monitor (AWS Lambda alerts if backups go
stale >48h)

## Changes

**Infrastructure (Pulumi):**
- `modules/iam/claude_readonly_sa.py` — new read-only + break-glass SAs
- `modules/backup.py` — backup SA, staging bucket, Cloud Function, Cloud
Scheduler
- `modules/database.py` — enable Firestore PITR
- `modules/storage.py` — enable GCS versioning + soft delete lifecycle
- `modules/secrets.py` — add `discord-alert-webhook` secret
- `__main__.py` — wire new modules

**Backup Cloud Function:**
- `functions/backup_to_aws/` — Firestore export, BigQuery export
(weekly/monthly), GCS delta sync, S3 streaming upload, Discord alerts

**Documentation:**
- `docs/DISASTER-RECOVERY.md` — restore runbook for accidental deletion
+ catastrophic GCP loss
- `docs/archive/2026-03-27-business-continuity-design.md` — full design
spec
- `docs/archive/2026-03-29-business-continuity-plan.md` — implementation
plan
- `docs/archive/2026-03-29-external-services-config.md` —
OAuth/DNS/external service reference

## Already deployed

All infrastructure has been applied via `pulumi up`. First backup run
completed successfully (Firestore + job files in S3). Nightly backups
running at 1 AM ET.

## Test plan

- [x] Pulumi preview — 30 creates, 2 updates, 0 deletes
- [x] Pulumi up — all resources deployed
- [x] Manual backup function trigger — Firestore export (1.4 GB) landed
in S3
- [x] GCS job files sync — 161 files (2.9 GB) in S3
- [x] AWS Lambda freshness check — returned `ok` (3h age)
- [x] IAM deny policy verified active via `gcloud iam policies get`

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 03:32:01 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 789bf80646 Skip welcome credit AI evaluation for test emails (#638)
## Summary
- E2E test accounts (`@inbox.testmail.app`) were triggering the
Gemini-based credit evaluation on every daily test run, wasting LLM
credits and sending unnecessary denial emails
- Skip both the pre-compute (at magic link send) and inline evaluation
(at verification) for test emails — they get auto-granted directly

## Changes
- `backend/api/routes/users.py`: Skip background pre-compute thread for
test emails
- `backend/services/user_service.py`: Skip AI evaluation in
`grant_welcome_credits_if_eligible()` for test emails, go straight to
granting
- `backend/tests/test_anti_abuse.py`: Add test verifying AI eval is
skipped for testmail addresses

## Testing
- [x] All 2872 unit tests pass (8 pre-existing emulator failures
unrelated)
- [x] New test `test_skips_evaluation_for_test_emails` verifies behavior

## Review
- [x] Local CodeRabbit review completed — no findings

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 14:37:14 -04:00
Andrew Beveridge ☄️ c98da49da9 feat: add jobs search, status filter, and fix loading states (#637)
## Summary
- Add server-side text search to jobs list API (case-insensitive
substring match across artist, title, job_id)
- Replace eye icon toggle with status filter dropdown (All statuses,
Active, Failed, Completed, Cancelled, Awaiting review, Processing)
- Add debounced search input to Recent Jobs header
- Expand limit dropdown options (5, 10, 20, 50, 100, 1000)
- Fix loading state showing "No jobs yet" during cold starts (now shows
spinner)
- Fix API always fetching limit=100 regardless of dropdown selection

## Test plan
- [x] 8 unit tests for `_search_filter_jobs` (artist, title, job_id,
audio search fields, case insensitivity, None handling, empty string,
limit interaction)
- [x] Full backend test suite passes (2870 passed)
- [x] Frontend builds cleanly
- [ ] Manual: verify search filters jobs in real-time on dashboard
- [ ] Manual: verify status dropdown filters correctly
- [ ] Manual: verify loading spinner shows during cold starts instead of
"No jobs yet"

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-03-30 05:03:05 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 022b4e343a fix: backing vocals recommendation showing 0% with contradictory message (#636)
## Summary

- Fixed API defaulting `recommended_selection` to `with_backing` when no
analysis exists, creating contradictory "Significant backing vocals
(0%)" message — now defaults to `clean`
- Fixed analyzer returning `review_needed` (unknown to frontend) instead
of `with_backing` when backing vocals detected
- Fixed race condition where backing vocals analysis ran in
screens_worker before audio separation completed — moved to audio_worker

## Test plan

- [x] All 25 analyzer unit tests pass (3 updated to match new
`WITH_BACKING` return value)
- [x] All 21 frontend instrumental review component tests pass
- [x] Full test suite: 2863 passed (8 pre-existing emulator failures)
- [x] CodeRabbit: no findings
- [ ] Verify fix in production: create new job and check instrumental
review shows correct recommendation

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 00:39:59 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 16fcb77d0f fix: relevance filtering in correction_disabled path + guidance message (#635)
## Summary

Two fixes found during post-deploy testing of #632:

1. **Relevance filter bypassed when SKIP_CORRECTION=true** — The
controller's `create_uncorrected_result()` path builds CorrectionResult
directly without going through LyricsCorrector, so wrong-song references
were never filtered. Now applies `filter_irrelevant_sources()` in this
path too.

2. **Misleading "All lyrics matched" message** — When no reference
sources exist (all filtered or none found), the guidance panel showed
"All lyrics matched the reference — no corrections needed." Now shows
"No reference lyrics were found" with guidance to use the search panel.

## Testing

- 9/9 relevance tests passing
- 22/22 GuidancePanel tests passing (updated + new test for empty state)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 04:21:58 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 14a6b767ce fix: improve cancelled job UX and fix /dashboard redirects (#633)
## Summary
- Allow downloading files from cancelled jobs (backend no longer
requires job to be complete if the file exists in GCS)
- Hide cancelled jobs from default recent jobs view alongside completed
jobs; visible via "Show all jobs" toggle
- Fix stale `/dashboard` routes → `/app` (including impersonation
redirect that 404'd)
- Show accurate total job count and helpful empty state when jobs are
filtered out

## Test plan
- [x] Backend unit tests pass (2837 passed)
- [x] Frontend unit tests pass (717 passed, 39 suites)
- [ ] Verify cancelled job download works in prod after deploy
- [ ] Verify impersonation redirects to /app correctly
- [ ] Verify "Show all jobs" toggle shows/hides cancelled and completed
jobs
- [ ] Verify empty state message shows correct total and eye icon hint

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-29 20:49:28 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 591c050545 feat: Reference lyrics search with relevance filtering (#632)
## Summary

- Add relevance filtering to the correction pipeline — automatically
rejects wrong-song reference lyrics (below 30% word match threshold,
validated against 168 prod source pairs)
- New `POST /api/review/{job_id}/search-lyrics` endpoint lets users
search with alternate artist/title from the review UI, with force-add
override for edge cases
- Redesigned Add Reference Lyrics modal with Search (default) and Paste
tabs, plus inline empty state when no valid sources exist

## Changes

### Backend
- **Relevance filtering** (`correction/relevance.py`): Per-source
scoring based on anchor sequence word coverage. Integrated into
`LyricsCorrector.run()` after anchor finding — sources below threshold
removed before corrections computed
- **Search endpoint** (`routes/review.py`): Queries all providers
(Genius, Spotify, Musixmatch, LRCLIB) with user-supplied artist/title,
auto-filters results, returns added/rejected/not-found breakdown
- **Provider factory** (`operations.py`): `create_lyrics_providers()`
reads env vars, `search_lyrics_sources()` orchestrates
fetch+filter+correct
- **Force override**: `add-lyrics` and `search-lyrics` support `force`
flag to bypass threshold

### Frontend
- **Tabbed modal**: `AddLyricsModal` rewritten with Search (default) and
Paste tabs
- **SearchLyricsTab**: Artist/title inputs, provider search, rejected
source cards with force-add checkboxes
- **ReferenceEmptyState**: Inline search/paste UI when no valid
reference sources exist
- **API client**: New `searchLyrics()` method with
`SearchLyricsResponse` types

### Analysis
- Empirical threshold analysis script
(`scripts/analyze_reference_relevance.py`) + visual review tool
- 168 source pairs from 81 prod jobs: wrong-song max 3.5%, correct-song
min 26.8% → 30% threshold

## Testing
- 35 new backend tests (relevance scoring, filtering, corrector
integration, search operation, API endpoint)
- 717 existing frontend tests passing (no regressions)
- Edge case: all sources filtered out handled gracefully

## Review
- [x] Local CodeRabbit review completed (3 issues fixed: response
contract, force param coercion, null checks)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 00:25:47 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 5a5ee4a780 fix: validate Google OIDC tokens from Cloud Scheduler (#631)
## Summary
- All 4 Cloud Scheduler → backend jobs (stale review processor, stuck
job recovery, YouTube queue, disposable domain sync) were returning 401
because the auth system didn't validate Google OIDC JWT tokens
- Added OIDC token validation to `AuthService.validate_token_full()`
using `google.oauth2.id_token.verify_oauth2_token()`
- Only accepts tokens signed by Google from the configured backend
service account

## Changes
- `backend/services/auth_service.py` — New `_validate_oidc_token()`
method + integration into `validate_token_full()`
- `backend/config.py` — New `SCHEDULER_SERVICE_ACCOUNT` setting
(defaults to `karaoke-backend@nomadkaraoke.iam.gserviceaccount.com`)
- `backend/tests/test_auth_oidc.py` — 10 unit tests covering all code
paths
- `docs/LESSONS-LEARNED.md` — New lesson about Cloud Scheduler OIDC vs
admin token auth
- `pyproject.toml` — Version bump 0.153.6 → 0.153.7

## Testing
- [x] 10 new unit tests (valid token, wrong SA, expired, invalid
signature, network error, passthrough)
- [x] Full backend suite: 2837 passed
- [x] Full frontend suite: 717 passed

## Review
- [x] Local CodeRabbit review completed (no findings)
- [x] Docs reviewed and updated

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-29 20:00:36 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 f6102e11ab fix: correct admin credits API path in e2e test (#630)
## Summary
- Follow-up to #629 — the admin credits endpoint is at
`/api/users/admin/credits` (users router prefix `/users`), not
`/api/admin/credits`
- One-line fix to the URL in the e2e test

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-29 12:59:18 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 dba4ea1912 fix: grant credits via admin API when e2e test user gets 0 credits (#629)
## Summary
- Follow-up to #625 and #628 — after fixing the disposable email
allowlist and credit interstitial handling, the e2e test hits a third
issue: the credit evaluation system denies credits to testmail.app
users, leaving the guided flow inputs disabled
- Now detects 0 credits after signup and uses `E2E_ADMIN_TOKEN` to grant
1 credit via the admin API, then reloads the page

## Changes
- `frontend/e2e/production/happy-path-real-user.spec.ts`: After signup,
check credit balance and grant via admin API if 0

## Test plan
- [ ] E2E happy path test passes after merge (triggered manually)

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-29 12:48:19 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 dbadcc5c17 fix: handle credit interstitial in e2e happy path test (#628)
## Summary
- Follow-up to #625 — after fixing the disposable email allowlist, the
e2e test hit a second failure: the magic link verification page now
shows credit evaluation interstitials for new users instead of
"Successfully signed in!" (which is only for returning users)
- The test now handles all verification outcomes (credits_granted,
credits_denied, credits_pending, success) and clicks through credit
interstitials to reach /app

## Changes
- `frontend/e2e/production/happy-path-real-user.spec.ts`: Use
`Promise.race` to detect which verification state appears, handle credit
interstitials by clicking through

## Test plan
- [ ] E2E happy path test passes after deployment (triggered manually)

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-29 12:38:30 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 257315d632 ci: use poetry.lock for GPU base image hash to avoid needless rebuilds (#627)
## Summary

- Use `poetry.lock` instead of `pyproject.toml` when computing the GPU
base image content hash
- Prevents expensive 15-20 min GPU base image rebuilds on version-only
bumps

## Problem

The GPU base image hash was computed from `cat Dockerfile.gpu-base
pyproject.toml | sha256sum`. Any change to `pyproject.toml` — including
version bumps that don't change dependencies — triggered a full GPU base
image rebuild. This adds ~15-20 min to every deploy that follows a
version bump.

## Fix

`poetry.lock` is the authoritative record of resolved dependencies. It
only changes when actual dependencies are added/removed/updated, not on
version bumps. The GPU base image installs deps via `pip install -e .`
which reads `pyproject.toml` at build time, but the Docker layer cache
handles that separately — the CI hash check just needs to know if deps
changed.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-29 11:30:50 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 92d1c9729d ci: auto-update lyrics-transcription-job on deploy (#626)
## Summary

- Add `lyrics-transcription-job` to the CI deploy step that updates
Cloud Run Jobs
- Previously only `video-encoding-job` and `audio-separation-job` were
updated, requiring manual `gcloud run jobs update` for lyrics

## Context

Discovered during the anchor IndexError fix (#623) — the fix was
deployed to the Cloud Run Service automatically but the Cloud Run Job
(where lyrics transcription actually runs) had to be updated manually.

## Changes

- `.github/workflows/ci.yml`: Add `gcloud run jobs update
lyrics-transcription-job` alongside the existing job updates

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-29 10:52:02 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 cd3076fc58 fix: allowlisted domains bypassed by external disposable email APIs (#625)
## Summary
- The disposable email detection had a bug where the Firestore
`allowlisted_domains` only protected against the Tier 1 static blocklist
check. External API checks (DeBounce, verifymail.io) at Tiers 3-4 ran
regardless and could override admin allowlist decisions.
- This broke the E2E happy path test starting March 21 when
`inbox.testmail.app` got flagged by DeBounce as disposable — the signup
step failed because "Check Your Email" never appeared.
- Adds a "Tier 1.5" check so allowlisted domains skip external API calls
entirely, plus a CI setup step to ensure the domain stays allowlisted.

## Changes
- `backend/services/email_validation_service.py`: Add allowlist check
before external API calls; include `allowlisted_domains` in cached
config
- `backend/tests/test_email_validation_service.py`: Regression test for
allowlisted domain bypass
- `.github/workflows/e2e-happy-path.yml`: CI setup step to ensure
`inbox.testmail.app` is allowlisted before test runs

## Test plan
- [x] All 53 email validation tests pass (52 existing + 1 new regression
test)
- [x] `inbox.testmail.app` added to production Firestore allowlist
- [ ] E2E happy path test passes after deployment

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-29 10:48:24 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 78e4cc7010 fix: bounds check ref_words in anchor gap extension (IndexError b7e8c96d) (#623)
## Summary

- Fix IndexError in `_extend_anchors_into_single_word_gaps` when
`ref_texts_clean` and `ref_words` have different word counts due to
`clean_text()` normalization
- Adds production fixture regression test from crashing job `b7e8c96d`
(Bằng Kiều - Dẫu có lỗi lầm)

## Root Cause

`ref_texts_clean` is built by joining words → `clean_text()` →
`split()`, which can produce a **different word count** than the raw
`ref_words` Word objects list. For Spotify in the crashing job:
`ref_texts_clean` had 599 words, `ref_words` had 528. Anchor
`reference_positions` computed against the 599-word list were used to
index the 528-word list.

## Changes

- `anchor_sequence.py`: Add `< len(ref_words[source])` bounds check in
both `_apply_forward_extension` and `_apply_backward_extension`
- `test_anchor_gap_extension_bounds.py`: 4 regression tests (production
fixture + synthetic bounds tests)
- `anchor_crash_b7e8c96d.json`: Sanitized production fixture (49
anchors, 526 words)
- `LESSONS-LEARNED.md`: Document text normalization word count mismatch
gotcha
- Version bump: 0.153.4 → 0.153.5

## Testing

- [x] 33 anchor gap extension tests pass (29 existing + 4 new)
- [x] Full unit suite: 1146 tests pass
- [ ] After deploy: retry job `b7e8c96d`

## Review

- [x] CodeRabbit CLI review completed locally (1 finding fixed:
sanitized copyrighted lyrics)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 14:23:13 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 dd25a6110f docs: troubleshooting runbooks for job repair and Cloud Run Jobs (#622)
## Summary

- Surgical audio re-separation that preserves lyrics/review data (direct
Firestore + Cloud Run Jobs API)
- How to trigger Cloud Run Jobs directly, bypassing internal API
idempotency checks
- CI gap: `lyrics-transcription-job` image not updated on deploy (manual
workaround + permanent fix pointer)

Docs-only — no code changes, no version bump.

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 20:36:51 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 e3c55672e5 fix: bounds check in anchor sequence gap extension (#621)
## Summary

- Fix IndexError in `_extend_anchors_into_single_word_gaps` when
`gap_position >= len(all_words)`
- This crashes lyrics transcription for songs where the last anchor ends
at the transcription boundary
- Affected job: `b7e8c96d` (Bằng Kiều - Dẫu có lỗi lầm)

## Fix

Add `if gap_position >= len(all_words): continue` before accessing
`all_words[gap_position]`.

## Test plan

- [x] 2 new boundary regression tests in test_anchor_gap_extension.py
- [x] Full unit suite: 1142 passed (29 anchor gap tests)
- [ ] After deploy: retry job b7e8c96d

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 19:51:42 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 bcbbb7865f fix: prevent false 'No audio sources found' on search failure (#620)
## Summary

- Fix misleading "No audio sources found" message that appeared when
audio search failed (timeout, network error, 5xx) instead of showing an
error
- Replace scattered boolean/string state with explicit `SearchStatus`
state machine so the no-results message can only appear after a
confirmed successful search
- Add automatic retry (up to 3 attempts with backoff) for transient
failures

## Changes

- **State machine:** Replace `isSearching`, `error`, `isCreditError`
with single `SearchStatus` discriminated union (`searching | succeeded |
failed`)
- **Retry logic:** Auto-retry network errors, timeouts, and 5xx up to 3
times with 2s/4s backoff delays. 4xx errors (auth, credits) fail
immediately
- **Spinner feedback:** Show "Retry 1 of 2" / "Retry 2 of 2" during
retries
- **Fuzzy accept:** Route through `doSearch` via useEffect instead of
inline API call, so corrected searches also get retry protection
- **Tests:** 5 new tests covering the core bug fix, retry behavior, and
error classification
- **Docs:** Added lesson learned about state conflation pattern

## Testing

- [x] 20/20 unit tests pass (15 existing + 5 new)
- [x] 712/712 full suite tests pass
- [x] TypeScript compiles cleanly
- [x] Production flacfetch verified healthy (v0.19.5, all providers up)

## Review

- [x] Local CodeRabbit review completed (no findings)
- [x] Two-stage review (spec compliance + code quality) during
implementation

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 14:44:27 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 8c7627b7ec fix: prevent video worker race condition and add encoding worker warmup (#619)
## Summary
- Fix race condition where Cloud Run Job `max_retries` + manual retry
launched concurrent video workers that corrupted each other's state
transitions (job 13b8adcf)
- Add encoding worker VM warmup to all paths that trigger video
encoding, not just the frontend happy path
- Defense in depth: frontend warmup → worker dispatch warmup → encoding
retry warmup

## Changes
- **job_health_service.py**: Remove `GENERATING_VIDEO`/`RENDERING_VIDEO`
from valid video_worker startup states — if a job is already in those
states, another worker is processing it
- **video_worker.py**: Make `_validate_prerequisites` a hard failure on
wrong status (was warn-and-continue)
- **worker_service.py**: Add `_warmup_encoding_worker()` called from
both `trigger_video_worker` and `trigger_render_video_worker`
- **encoding_service.py**: Add `_warmup_encoding_worker_fallback()`
called on first connection failure in retry loop
- **docs/LESSONS-LEARNED.md**: Document both findings

## Testing
- [x] 7 new unit tests covering race condition fix, warmup dispatch, and
fallback behavior
- [x] Full backend suite passes (2817 passed)
- [x] Full frontend suite passes (712 passed)

## Review
- [x] Local CodeRabbit review completed (0 code issues)
- [x] Review feedback addressed

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 14:27:29 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 0c996d1c8e fix: rename Stage 1 vocals before Stage 2 to avoid stem tag regex collision (#618)
## Summary

Follow-up to #617 — the stem tag parsing fix alone wasn't enough. The
audio-separator library's ensemble code uses
`re.search(r'_\(([^)]+)\)')` to classify intermediate stems, matching
the **first** `_(Tag)_` pattern. When Stage 2 receives Stage 1 output
like `Song_(Vocals)_preset_instrumental_clean.flac`, the regex matches
the Stage 1 tag instead of the Stage 2 tag, classifying ALL stems as
"Vocals" and producing only 1 output file instead of 2.

## Fix

Copy Stage 1 vocals to a clean filename (`Artist - Song (Vocals).flac`)
before passing to Stage 2, avoiding the regex collision.

## Test plan

- [x] Unit tests updated to verify clean filename is used for Stage 2
input
- [x] Full test suite: 1140 passed, 0 failed
- [ ] After deploy: trigger test job and verify backing vocals +
combined instrumental stems appear

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 14:18:07 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 f58c74e67e fix: ensemble preset stem tag parsing — restore backing vocals separation (#617)
## Summary

- Fix stem filename parsing bug where preset name `instrumental_clean`
caused the Vocals file to be misclassified as instrumental, preventing
Stage 2 (backing vocals separation) from running
- All new GPU-processed jobs were only producing `instrumental_clean` —
no backing vocals, lead vocals, or combined instrumentals
- The instrumental review page showed "No backing vocals detected" for
all jobs

## Root Cause

Ensemble preset output filenames follow
`{base}_(StemTag)_preset_{name}.flac`. The old code did `"instrumental"
in basename` which matched the **preset name** `instrumental_clean` in
the Vocals file, so `result["vocals"]` was never set, and Stage 2 never
ran.

## Fix

Extract the stem tag from between `_(` and `)_` delimiters and match
only that, not the full filename.

## Test plan

- [x] 5 new regression tests for stem tag parsing (both stages)
- [x] 2 pre-existing tests fixed to use realistic ensemble filenames
- [x] Full unit suite: 1140 passed, 0 failed
- [x] CodeRabbit: no findings
- [ ] After deploy: trigger a test job and verify all stems appear on
instrumental review page

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 12:14:32 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 ccc1b7c714 Fix lyrics upload race condition when artist/title changes mid-processing (#616)
## Summary

- Fix race condition in `upload_lyrics_results` where admin edits to job
artist/title during lyrics worker execution caused filename mismatch and
"No corrections JSON found" errors
- Scan lyrics directory by file suffix pattern instead of reconstructing
filenames from mutable Firestore metadata
- Remove unnecessary `get_job()` call and `sanitize_filename` import
from upload path

## Root Cause

Job `fbec37d6` failed because:
1. Lyrics worker started processing with artist = `zmcmillan08`
2. Admin updated artist to `Quay Money` mid-processing
3. `upload_lyrics_results` read the updated artist from Firestore and
looked for `Quay Money - ... (Lyrics Corrections).json`
4. File on disk was `zmcmillan08 - ... (Lyrics Corrections).json` → not
found → job failed

## Test plan

- [x] Updated 3 existing tests to validate directory-scanning behavior
- [x] Added new test: error path when no corrections file exists at all
- [x] All 68 worker tests pass
- [x] CodeRabbit review: no findings

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 11:14:55 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 056ce3ce2d docs: GPU audio separation learnings and cleanup (#615)
## Summary
- Remove deprecated `AUDIO_SEPARATOR_API_URL` from DEVELOPMENT.md env
vars (no longer used — separation runs directly in Cloud Run Job)
- Add GPU-specific lessons to LESSONS-LEARNED.md:
  - Pin CUDA-specific PyTorch versions in Docker images
  - Ensemble presets require explicit `load_model()` before `separate()`
  - Validate with actual Docker builds, not just package combos

Docs-only, no version bump needed.

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 01:32:35 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 dfcd79e5c7 fix: GPU app image — preserve pinned torch cu126 with --no-deps (#614)
## Summary
- Use `--no-deps` in app Dockerfile when building on GPU base image
(detected via `MODEL_DIR` env var)
- Without this, `pip install -e .` upgrades torch from pinned cu126 back
to cu130 (pyproject.toml has `torch>=2.7`) and reinstalls
audio-separator[cpu], breaking both CUDA and GPU ONNX Runtime

This is a follow-up to #613 — the GPU base image had the right packages,
but the app layer's `pip install -e .` undid the pinning.

## Test plan
- [ ] CI passes
- [ ] Deploy, trigger audio separation job
- [ ] Verify Job logs show CUDA available + successful separation

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 23:59:47 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 c4ea0e8fb4 fix: GPU audio separation — pin PyTorch cu126 + missing load_model() (#613)
## Summary
- Pin PyTorch to `cu126` in GPU Dockerfile — Cloud Run L4 GPUs have
driver 570 (CUDA 12.8 max), but `audio-separator[gpu]` pulls PyTorch
`cu130` which fails with "driver too old"
- Add missing `separator.load_model()` before `separator.separate()` for
ensemble presets in `audio_processor.py` — the Separator API requires
this even for presets
- Remove `AUDIO_SEPARATOR_API_URL` from Service env vars (cleanup — Job
has its own `MODEL_DIR`)

## Validation
Tested on GCE L4 GPU VM:
- `torch==2.6.0+cu126` + `audio-separator[gpu]` 0.44.1 → CUDA detected,
GPU acceleration enabled
- Full ensemble separation (`instrumental_clean` preset) completed in
**26.6s** on real audio
- Docker image build validation in progress on separate L4 VM

## Test plan
- [ ] CI passes (GPU image rebuilds due to Dockerfile change)
- [ ] Trigger audio separation job after deploy
- [ ] Verify Cloud Run Job logs show "CUDA is available" and "local GPU"
mode
- [ ] Verify stems are correct

@coderabbitai ignore

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 23:18:32 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 7424398fcf fix: use us-east4 region for audio separation job trigger (#612)
## Summary

The audio-separation-job moved to us-east4 (GPU), but the worker trigger
was still using the default us-central1 region. Added `location`
parameter to `_trigger_worker_cloud_run_job()` and passed `us-east4` for
the audio worker.

Also includes the FFmpeg apt fix from #611.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 17:36:07 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 41f296f70b fix: use apt FFmpeg in GPU Dockerfile (CI timeout fix) (#611)
## Summary

The johnvansickle.com static FFmpeg download times out in CI (curl exit
code 28). The audio worker only uses FFmpeg for format conversion and
padding, not encoding, so the Ubuntu apt package is sufficient.

Fixes the GPU base image build failure in the deploy-backend workflow.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 20:56:36 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 177e70efc5 fix: GPU Cloud Run Job infra fixes (zonal redundancy, deletion protection) (#610)
## Summary

Fixes for the GPU audio worker Cloud Run Job that were discovered during
Pulumi apply:

- `deletion_protection=False` — needed so Pulumi can replace the job
when changing regions
- `gpu_zonal_redundancy_disabled=True` — required for Cloud Run Jobs
with GPU (GCP capacity limitation)

Follow-up to #609 (GPU image CI) where the Pulumi apply failed.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 16:28:17 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 55dd7a30d0 ci: add GPU image build and audio worker deploy to workflow (#609)
## Summary

- Adds GPU base image build to the deploy-backend CI job (same
hash-check pattern as regular base image)
- Builds GPU app image (regular Dockerfile with GPU base as
`BASE_IMAGE`)
- Updates `audio-separation-job` in us-east4 with the GPU image on every
deploy
- Configures Docker auth for the us-east4 Artifact Registry

## Changes

- `.github/workflows/ci.yml` — GPU base image hash check + build, GPU
app image build, audio-separation-job update
- `backend/Dockerfile.gpu-base` — Added `BASE_CONTENT_HASH` label for
hash-based rebuild detection

## How it works

On merge to main, the deploy-backend job now:
1. Builds regular base + app images (unchanged)
2. Hashes `Dockerfile.gpu-base` + `pyproject.toml` → checks if GPU base
needs rebuild
3. If rebuild needed: builds GPU base image (CUDA + PyTorch + models,
~45 min timeout)
4. Builds GPU app image (regular Dockerfile with GPU base)
5. Updates `audio-separation-job` in us-east4 with the new GPU image

The first run will take longer (GPU base image bootstrap). Subsequent
runs skip the GPU base build unless `Dockerfile.gpu-base` or
`pyproject.toml` changes.

## Testing
- [x] YAML syntax verified
- [ ] First CI run will bootstrap the GPU base image (expect ~30-45 min)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 15:47:07 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 92ae3fb28f feat: run audio separation directly on GPU in audio worker (#608)
## Summary

- **Eliminates HTTP hop** between the audio worker Cloud Run Job and the
audio-separator Cloud Run Service by running the `Separator` class
directly with an L4 GPU attached
- **Adds GPU Docker image** (`karaoke-backend-gpu`) with CUDA 12.6,
PyTorch, and pre-baked ensemble preset models
- **Moves audio worker to us-east4** with L4 GPU, 4 CPU, 16Gi memory —
old separator service kept for rollback

## Changes

### Code
- `audio_processor.py` — Ensemble presets wired into local Separator
path (2-stage: `instrumental_clean` + `karaoke`)
- `audio_worker.py` — Dual-mode support: `MODEL_DIR` env var → local
GPU, `AUDIO_SEPARATOR_API_URL` → remote API
- `create_audio_processor()` — New `model_file_dir` parameter to enable
local GPU mode

### Infrastructure (Pulumi)
- `gpu_artifact_registry.py` — New Artifact Registry in us-east4 for GPU
images
- `cloud_run.py` — Audio worker job: GPU (nvidia-l4), us-east4, 16Gi,
`MODEL_DIR=/models`
- `__main__.py` — Wire in GPU registry + Cloud Build IAM

### Docker
- `Dockerfile.gpu-base` — CUDA 12.6 + Python 3.11 + audio-separator[gpu]
+ baked models
- `cloudbuild-gpu-base.yaml` — Cloud Build config (E2_HIGHCPU_32, 100GB
disk, 1h timeout)
- `download_models.py` — Bakes `instrumental_clean` + `karaoke` preset
models at build time

### Tests
- 8 new unit tests covering factory config, mode selection, and preset
wiring
- Updated existing test patch targets for module-level Separator import

## Testing
- [x] 22/22 unit tests pass (8 new + 14 existing)
- [x] Code review completed (subagent-driven + final review)
- [x] CodeRabbit CLI review completed locally
- [ ] GPU image build (requires `gcloud builds submit`)
- [ ] Production verification with `under_pressure_harmonies.flac`

## Rollback Plan
Set `AUDIO_SEPARATOR_API_URL` back on the audio worker job → falls
through to remote API path. Old separator service is still running
(scale-to-zero).

## Design & Plan
- Design spec:
`docs/archive/2026-03-26-direct-gpu-audio-separation-design.md`
- Implementation plan:
`docs/archive/2026-03-26-direct-gpu-audio-separation-plan.md`

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 15:13:51 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 2cc1061a9d Audio separator infra + docs (concurrency, Firestore, GCS) (#607)
## Summary

- Infrastructure for async audio separator: GCS output bucket, Firestore
index, IAM, env vars
- Set concurrency=1 (each GPU job gets own instance for proper
autoscaling)
- Updated all docs: Modal→Cloud Run GPU references, architecture design
spec, lessons learned
- Phase 2 noted: migrate separator to Cloud Run Jobs

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 17:37:59 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 be19e6c7a0 Infra for async audio separator (concurrency, GCS, Firestore) (#606)
## Summary

- Update audio separator Cloud Run service:
`max_instance_request_concurrency` 1→50, `max_instances` 5→10
- Add `OUTPUT_BUCKET` and `GCP_PROJECT` env vars for the separator
service
- Create `nomadkaraoke-audio-separator-outputs` GCS bucket with 1-day
lifecycle policy
- Grant separator service account `roles/datastore.user` for Firestore
and `roles/storage.objectAdmin` for the output bucket
- Add Firestore composite index on `audio_separation_jobs` (status +
updated_at) for cleanup queries

Companion to nomadkaraoke/python-audio-separator#280. Deploy order: this
PR's infra first (`pulumi up`), then the separator service.

## Test plan

- [ ] `pulumi preview` shows expected resource changes
- [ ] `pulumi up` applies cleanly
- [ ] Verify GCS bucket created with lifecycle policy
- [ ] Verify Firestore index created
- [ ] Verify separator service updated with new concurrency/scaling
settings

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 00:41:45 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 8f8fc269e6 fix: retry returns to lyrics review when not completed (#605)
## Summary
- Fixed retry checkpoint detection skipping lyrics review when user
never completed it
- Previously, retry used `corrections.json` + `title.mov` as proxy for
"review done," but both are auto-generated artifacts that exist BEFORE
user review
- Now requires `instrumental_selection` in state_data (only set via
review submit endpoint) to confirm review was actually completed
- Jobs that failed before review completion now correctly return to
`awaiting_review` on retry

## Root Cause
Job `45f9bbc0` hit this: audio separation kept failing, but corrections
+ title screen existed from the lyrics/screens pipeline. Retry jumped to
`REVIEW_COMPLETE`, skipping review entirely. Admin had to manually
reset.

## Test plan
- [x] Updated existing render checkpoint test to require
`instrumental_selection`
- [x] Added regression test for the skip-review scenario
- [x] All 8 retry endpoint tests pass
- [x] Full backend suite passes (2583 passed)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 21:58:58 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 ce4bde6a6d fix: update audio-separator to 0.44.1 (multipart fix) (#603)
## Summary
Update audio-separator from 0.44.0 to 0.44.1 to fix 400 Bad Request when
using `gcs_uri`.

v0.44.0 sent `x-www-form-urlencoded` when no file was uploaded (gcs_uri
mode), but FastAPI requires `multipart/form-data`. v0.44.1 fixes this.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 19:01:25 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 22a500d153 Pre-compute credit evaluation at magic link send time (#602)
## Summary
- Run AI credit evaluation in background thread when sending magic link
(new users only)
- Store result on magic link Firestore doc
- At verify time, use pre-computed result if available, fall back to
inline evaluation if not
- Most users will see instant verification instead of 20s wait

## Test plan
- [x] 53 anti-abuse + credit evaluation tests pass (3 new)
- [x] New test: pre-computed eval skips AI call
- [x] New test: verify passes pre-computed eval from magic link doc
- [x] New test: verify falls back to None when no pre-computed eval
- [ ] Manual test: sign up, verify welcome screen appears quickly

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 22:36:31 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 7c39b2817f Fix Vertex AI region: use global for latest Gemini models (#600)
## Summary
- Preview Gemini models (gemini-3.1-pro-preview) require
`location="global"`, not `us-central1`
- This was causing 404 errors when the credit evaluation tried to call
Gemini for users with abuse signal correlations
- Bump version to 0.150.6

## Test plan
- [x] All 50 anti-abuse + credit evaluation tests pass
- [ ] Manual test: sign up with correlated fingerprint/IP, verify AI
evaluation runs successfully

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 17:20:08 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 c47fe6518c fix: update audio separator URL from Modal to Cloud Run (#601)
## Summary
- The CI deploy script was still setting `AUDIO_SEPARATOR_API_URL` to
the old Modal endpoint
- Update to the Cloud Run GPU service URL
(`audio-separator-718638054799.us-east4.run.app`)
- This was the root cause of the 422 error when retrying job 45f9bbc0

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 16:18:44 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 046ca69a1c fix: pass GCS URI to audio separator, fix WIF for deploys (#599)
## Summary
- Fix 413 Request Entity Too Large for audio separation of large FLAC
files
- Fix Workload Identity Federation for python-audio-separator Cloud Run
deploys

## Changes
- **Infrastructure:** Grant `storage.objectViewer` to audio-separator SA
so it can read from the main GCS bucket
- **Infrastructure:** Add `python-audio-separator` repo to the WIF
binding (was missing, blocking all Cloud Run deploys)
- **Audio worker:** Set `input_gcs_uri` on the processor when the input
file is already in GCS
- **Audio processor:** Stage 1 passes `gcs_uri` to the separator API
instead of uploading file bytes
- **Dependency:** Update `audio-separator` to 0.44.0 (adds `gcs_uri`
support)
- **Version:** 0.151.0 → 0.152.0

## How it works
Before: `audio_worker → download from GCS → HTTP POST (file bytes) →
separator` (fails for >32MB)
After: `audio_worker → pass gs:// URI → separator downloads from GCS
directly` (no size limit)

Stage 2 (backing vocals) still uses file upload since the input is a
Stage 1 output that's already local.

## Test plan
- [x] 2 new unit tests: GCS URI passthrough + fallback to file upload
- [x] All 28 migration tests pass
- [x] Pulumi infra changes applied (GCS IAM + WIF binding)
- [x] Audio separator v0.44.0 deployed to Cloud Run and healthy
- [ ] Re-run failed job 45f9bbc0 to verify fix end-to-end

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 14:36:03 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 efb4aa8351 fix: add google-cloud-compute dependency for encoding worker VM management (#598)
## Summary

- Adds `google-cloud-compute` to pyproject.toml dependencies
- The backend needs this to call the Compute API (start/stop VMs) for
the blue-green encoding worker warmup
- Without it, the warmup/heartbeat endpoints return 500 with
`ImportError: cannot import name 'compute_v1' from 'google.cloud'`

## Test plan

- [x] Verified the import error in Cloud Run logs
- [ ] After merge, warmup endpoint should return 200

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 14:12:15 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 d681e26b02 fix: encoding worker deploy failure should not block Docker deploy (#597)
## Summary

- The blue-green encoding worker health check failure was blocking the
entire CI job, preventing the Docker image from being built and deployed
to Cloud Run
- This meant the new backend code (with warmup/heartbeat endpoints) was
never deployed, causing 404s on the new routes
- Fix: add `continue-on-error: true` to health check and encode test
steps so the Docker deploy always proceeds
- The rollback step now checks step outputs instead of `failure()`
condition

## Root cause

The CI job runs encoding worker deploy → Docker build → Cloud Run deploy
sequentially. When the encoding worker health check failed (missing
`packaging` module), the job stopped before reaching the Docker build
step.

## Test plan

- [x] Encoding worker deploy failure no longer blocks Docker deploy
- [ ] Next CI run on main will deploy the backend with new routes

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 13:53:46 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 2003d4acc8 fix: add packaging to encoding worker Packer image venv (#596)
## Summary

- Adds `packaging` to the pre-installed Python dependencies in the
Packer image venv
- The encoding worker imports `packaging.version.Version` but this
wasn't in the base image
- First blue-green deploy caught this via the health check — encoding
worker crashed on startup with `ModuleNotFoundError: No module named
'packaging'`
- Manually installed on both VMs as a temporary fix; this makes it
permanent for future image builds

## Test plan

- [x] Verified encoding worker starts after manual `pip install
packaging` on both VMs
- [ ] Next Packer image build will include this dependency

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 12:00:05 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 19efa269a8 feat: blue-green encoding worker deployment with deep health checks (#595)
## Summary

- Replaces single encoding worker VM with a blue-green pair (a/b) for
zero-downtime deployments
- VMs auto-start on demand (JIT warmup when user opens lyrics review)
and auto-shutdown after 15 min idle
- CI deploys to secondary VM, runs a real encode test, then swaps the
Firestore pointer atomically
- Deep health check catches issues like PR #587's pip crash before they
reach users
- Cloud Function checks idle VMs every 5 minutes and stops them to
minimize cost (~$10-15/mo idle vs $800/mo always-on)

## Architecture

- **Firestore config doc** (`config/encoding-worker`) is single source
of truth for primary/secondary routing
- **Backend** reads primary IP from Firestore (30s TTL cache), starts VM
on demand via Compute API
- **CI workflow** handles blue-green swap: deploy → health check → real
encode test → atomic Firestore swap → drain & stop old
- **Cloud Function** auto-stops idle VMs based on active jobs, user
activity, and deploy status
- **Frontend** sends warmup signal on lyrics review page load +
heartbeat on interactions

## Changes

### New Files
- `backend/services/encoding_worker_manager.py` — VM lifecycle +
Firestore config management
- `backend/api/routes/encoding_worker.py` — Warmup and heartbeat API
endpoints
- `infrastructure/functions/encoding_worker_idle/` — Idle shutdown Cloud
Function
- `infrastructure/modules/encoding_worker_manager.py` — Pulumi module
for Cloud Function + Scheduler
- `scripts/seed-encoding-worker-config.py` — Firestore config seeding
script
- `infrastructure/encoding-worker/test-assets/test-preview.ass` — Test
file for deploy health check

### Modified Files
- `infrastructure/compute/encoding_worker_vm.py` — Single VM →
blue-green pair
- `infrastructure/__main__.py` — Wire up new VMs, Cloud Function, IAM
- `backend/services/encoding_service.py` — Dynamic URL from Firestore,
warmup-aware retry
- `.github/workflows/ci.yml` — Blue-green deploy flow
- `frontend/` — Warmup signal on lyrics review, heartbeat, warming-up UI
state
- `docs/` — Architecture, development, troubleshooting, lessons learned

## Test plan

- [x] 47 new backend tests pass (EncodingWorkerManager, routes, dynamic
URL resolution)
- [x] 9 Cloud Function unit tests pass
- [x] All 2808 existing backend tests pass (1 pre-existing failure
unrelated)
- [x] All 1116 existing unit tests pass (3 pre-existing failures
unrelated)
- [ ] Pulumi preview shows expected resource changes
- [ ] Pulumi up applied locally before merge
- [ ] Firestore config seeded with VM IPs
- [ ] End-to-end verification after deployment

## Migration Steps (post-merge)

1. `pulumi up` creates 2 VMs, removes old VM
2. Seed Firestore: `python scripts/seed-encoding-worker-config.py <ip-a>
<ip-b>`
3. Start VM A, verify health
4. First CI deploy exercises the blue-green flow

## Design Docs

- Spec:
`docs/superpowers/specs/2026-03-24-blue-green-encoding-worker-design.md`
- Plan:
`docs/superpowers/plans/2026-03-24-blue-green-encoding-worker.md`

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 11:28:45 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 4e23f07d0a Fix credit evaluation: Vertex AI credentials + enriched admin review email (#594)
## Summary
- Fix Gemini API call failing due to missing API key — switch to Vertex
AI with default GCP credentials
- Enrich admin review email with abuse signals (IP, geo, fingerprint,
matching accounts, user agent, signup counts)
- HTML-escape user-controlled values in review email

## Test plan
- [x] All 50 anti-abuse + credit evaluation tests pass
- [x] Full backend test suite passes (2549 pass, 139 skipped)
- [ ] Manual test: sign up with clean email, verify welcome credits
granted
- [ ] Manual test: verify admin review email includes signal details
when evaluation triggers

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 23:48:39 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 7d887a11b0 Fix FLAC uploads to AudioShake by stripping metadata (#593)
## Summary
- Strip metadata from FLAC files before uploading to AudioShake to
prevent 502 errors caused by non-standard tags (MP4/M4A container fields
embedded in FLAC)
- Uses `ffmpeg -map 0:a -c copy -map_metadata -1` to remove metadata
while keeping audio unchanged
- Added unit + integration tests for the new metadata stripping path

## Context
AudioShake confirmed that non-standard metadata tags in FLAC files cause
their processing pipeline to fail during task creation. The audio itself
is valid — only the metadata is problematic. This fix proactively strips
all metadata before upload.

## Test plan
- [x] Unit test: FLAC files route through `_strip_metadata` instead of
direct upload
- [x] Integration test: metadata is actually removed from output FLAC
(verified via ffprobe)
- [x] All 30 AudioShake tests pass
- [x] Full test suite: 1117 passed (3 pre-existing failures unrelated to
this change)

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 23:18:56 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 6577babfea Reduce free credits from 2 to 1 per grant (#592)
## Summary

- Welcome credits: 2 → 1 (granted on first email verification)
- Feedback credits: 2 → 1 (granted for submitting product feedback)
- Total possible free credits per account: 4 → 2

## Changes

**Backend:** `NEW_USER_FREE_CREDITS` constant, feedback credit grant, AI
evaluation prompt, model defaults
**Frontend:** Landing page, dashboard banner, feedback dialog, auth
dropdown menu text
**Tests:** Updated all assertions and mock values (115 backend + 712
frontend tests pass)
**Docs:** AI-CREDIT-GATING.md, API.md, README.md,
PRODUCT-COMMUNICATION-GUIDE.md

## Test plan

- [x] Backend tests pass (115 passed)
- [x] Frontend Jest tests pass (712 passed)
- [ ] CI pipeline passes
- [ ] Verify landing page shows "1 Free Credit" after deploy
- [ ] Verify new user signup grants 1 credit (not 2)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 21:30:38 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 a7e41a09d6 Fix custom background image not applied in guided flow (#590)
@coderabbitai ignore

## Summary

- **Bug:** Custom background images uploaded during guided job creation
were silently ignored — the endpoint returned 500 due to
`AttributeError: 'StorageService' object has no attribute 'bucket_name'`
- **Fix:** `storage_service.bucket_name` → `storage_service.bucket.name`
(the correct attribute path)
- **Test fix:** Updated mock to use real attribute path
(`mock.bucket.name`) so this class of bug is caught

## Root Cause

The `complete_style_uploads` endpoint (line 2489) referenced
`storage_service.bucket_name`, but `StorageService` only exposes
`self.bucket` (a GCS Bucket object). The correct path is
`storage_service.bucket.name`.

The frontend's `uploadStyleAssets` caught the 500 error silently
(`console.error` only), so jobs continued rendering with default theme
backgrounds instead of the user's uploaded custom image.

The test passed because Mock objects auto-create attributes —
`mock.bucket_name = "test-bucket"` worked even though the real service
has no such attribute.

## Test plan

- [x] All 23 style upload tests pass
- [ ] Submit a job with custom background image and verify it renders
correctly

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 21:00:12 -04:00
Andrew Beveridge ☄️andClaude Sonnet 4.6 3e8e3474b2 Fix Pulumi state drift: env var ordering, divebar VM, model bucket (#591)
@coderabbitai ignore

## Summary
- Sort Cloud Run Job env vars alphabetically to match GCP API return
order (prevents phantom diffs on every `pulumi preview`)
- Add divebar-sync VM infrastructure code that was deployed from a
feature branch but never merged to main (prevents accidental deletion of
active infrastructure)
- Remove dead `create_model_bucket()` code and empty GCS bucket (models
baked into Docker image)

## Changes
- **`infrastructure/modules/cloud_run.py`**: Reorder env vars
alphabetically in all 4 Cloud Run Jobs (video-encoding,
audio-separation, lyrics-transcription, audio-download)
- **`infrastructure/compute/divebar_sync_vm.py`** (new): GCE VM
definition for divebar file sync (Drive → GCS)
- **`infrastructure/compute/startup_scripts/divebar_sync.sh`** (new): VM
startup script with google-auth pinning and MTLS workarounds
- **`infrastructure/__main__.py`**: Wire divebar-sync VM, Cloud
Scheduler, and IAM binding
- **`infrastructure/modules/audio_separator_service.py`**: Remove unused
`create_model_bucket()` and `storage` import

## Verification
- `pulumi preview` shows **0 changes** after applying
- `pulumi up` applied successfully (4 Cloud Run Job SDK metadata updates
+ 2 empty bucket deletes)
- Divebar-sync VM continues running undisturbed

## Test plan
- [x] `pulumi preview --diff` confirms zero drift
- [x] `pulumi up` applied cleanly (update #649)
- [x] Divebar-sync VM verified RUNNING in GCP
- [x] No application code changes — infra-only

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-24 20:53:58 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 4f8d2d71ed Fix pip resolution-too-deep in Docker base image build (#589)
## Summary
- Pin all 10 google-cloud-* and google-auth-* packages to exact latest
versions instead of loose lower bounds (e.g., `>=2.14.0` → `2.26.0`)
- Eliminates pip's massive backtracking that caused
`resolution-too-deep` errors during Docker base image builds
- Version bump: 0.150.2 → 0.150.3

## Context
The Docker base image build (`pip install -e .`) was failing because pip
had to search through hundreds of version combinations for
google-cloud-run, google-cloud-monitoring, grpc-google-iam-v1, and
googleapis-common-protos. Exact pins give pip zero backtracking to do.

## Test plan
- [x] `poetry lock` resolves cleanly
- [x] Backend tests pass (297 passed)
- [ ] CI Docker base image build succeeds (the actual fix)

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 20:47:05 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 0666a827dd Fix: migrate from google-generativeai to google-genai SDK (#588)
## Summary
- Remove `google-generativeai` (old SDK) from pyproject.toml — caused
17h encoding worker outage via pip dependency resolution explosion
- Migrate `credit_evaluation_service.py` to use `google.genai.Client`
(new SDK, already available via langchain-google-genai)
- Regenerate poetry.lock

Fixes the encoding worker crash loop from PR #584. See PR #587 for the
immediate `--no-deps` workaround.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 16:00:53 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 0635f800f6 Fix audio separation failure: smart fallback, retry, scale GPU (#586)
@coderabbitai ignore

## Summary

Fixes job 491be627 which failed because the remote audio-separator API
was busy and the code fell back to local `Separator()` which crashed
with `model_file_dir=None` in Cloud Run.

- **audio_processor**: Only fall back to local processing if
`model_file_dir` is configured (preserves CLI fallback, prevents crash
in Cloud Run)
- **audio_worker**: Retry separation up to 5x with exponential backoff
(60s, 120s, 240s, 480s — handles 15+ min busy periods)
- **infrastructure**: Scale audio-separator from 1→5 max instances with
`concurrency=1` so Cloud Run routes concurrent jobs to separate GPU
instances

## Infrastructure changes (already applied via `pulumi up`)

- `audio-separator` Cloud Run service: `maxInstanceCount: 1 → 5`,
`maxInstanceRequestConcurrency: 80 → 1`
- `audio-separator-allow-unauthenticated` IAM member created

## Test plan

- [x] Backend tests pass (2549 passed)
- [x] Pulumi changes applied successfully to prod
- [ ] Verify next audio separation job succeeds

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 15:34:48 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 9142ae5e5d fix: encoding worker crash loop from pip resolution-too-deep (#587)
## Summary
- Encoding worker has been in a crash loop since Mar 24 01:58 UTC (~17
hours) because `pip install --force-reinstall` of the 0.150.0 wheel hits
pip 25.3's `resolution-too-deep` error
- Root cause: `google-generativeai >= 0.8.0` added in #584 creates an
overly complex dependency graph when combined with existing
langchain/CUDA packages
- Fix: add `--no-deps` to pip install in `startup.sh` — the venv already
has runtime deps pre-installed, only the karaoke-gen code needs updating
- **Already deployed**: fixed `startup.sh` uploaded to GCS and encoding
worker restarted successfully

## Test plan
- [x] Reproduced `resolution-too-deep` locally with pip 25.3 in clean
venv
- [x] Verified `--no-deps` install works locally (instant, module loads)
- [x] Tested `--no-deps` install directly on the encoding worker VM with
sudo
- [x] Uploaded fixed startup.sh to GCS and restarted service
- [x] Verified health check passes:
`{"status":"ok","wheel_version":"0.150.0"}`
- [x] Retried failed job `b6e4b044` — now processing

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 15:31:26 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 5ea7a57eac Fix admin sidebar transparent background on mobile and desktop (#585)
## Summary
- Add missing sidebar CSS variables to `app/globals.css` for both dark
and light themes
- Add Tailwind v4 `@theme inline` color mappings for sidebar utilities
(`bg-sidebar`, `text-sidebar-foreground`, etc.)
- The sidebar component (shadcn/ui) was using `bg-sidebar` which
resolved to transparent because the CSS variables were defined in an
unused `styles/globals.css` file but missing from the active
`app/globals.css`

## Test plan
- [x] Verified sidebar has solid opaque background on mobile (375px) —
Sheet drawer no longer shows content behind
- [x] Verified sidebar has solid background on desktop (1280px) — no
transparency on abuse page or other admin screens
- [x] Verified sidebar renders correctly at tablet width (768px)
- [x] Verified all 10 admin pages render correctly on mobile (Dashboard,
Users, Jobs, Payments, Audio Searches, Rate Limits, Edit Reviews, Audio
Edits, Feedback, Abuse Investigation)
- [x] All 712 frontend tests pass

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 15:06:23 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 42a7793681 AI-powered credit gating with Gemini evaluation (#584)
## Summary
- Replace automatic welcome/feedback credit grants with AI-powered abuse
evaluation using Gemini 3.1 Pro
- **Fail-closed**: errors/uncertainty → credits withheld, admin notified
via email with user admin link
- Clean users (no correlations) get instant grant with no AI call
latency
- Frontend verify page shows preparing account →
celebratory/denied/pending interstitial
- Comprehensive docs at `docs/AI-CREDIT-GATING.md`

## Changes
| File | What |
|------|------|
| `backend/services/credit_evaluation_service.py` | **New**: Signal
collection, Gemini prompt, response parsing, Firestore audit |
| `backend/config.py` | `CREDIT_EVAL_ENABLED`, `CREDIT_EVAL_MODEL`
config |
| `backend/services/user_service.py` | Welcome credit flow integration
(returns tuple) |
| `backend/api/routes/users.py` | Feedback credit + verify response
integration |
| `backend/services/email_service.py` | `send_credit_denied_email()`,
`send_credit_review_needed_email()` |
| `backend/models/user.py` | `credit_status` field on verify response |
| `frontend/app/auth/verify/page.tsx` | 5-state interstitial flow |
| `frontend/lib/types.ts` + `auth.ts` | Type + state updates |
| `backend/tests/test_credit_evaluation.py` | **New**: 17 tests |
| `backend/tests/test_anti_abuse.py` | Updated for new return type |
| `docs/AI-CREDIT-GATING.md` | **New**: Comprehensive documentation |
| `pyproject.toml` | `google-generativeai` dep, version bump |

## Test plan
- [x] 50 backend tests pass (17 new + 33 updated)
- [ ] Manual: test with new account signup in prod (see docs for
expected flow)
- [ ] Manual: verify admin review email received for error cases

## Rollback
Set `CREDIT_EVAL_ENABLED=false` in Cloud Run env vars → instant revert
to auto-grant.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 21:57:46 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 21d8b270ae Fix: filter bad IPs from abuse correlation clusters (#583)
## Summary
- Filter known bad IPs (169.254.169.126 GCP load balancer, "unknown",
empty) from IP correlation clusters
- Discovered in production: the load balancer IP was creating a false
22-account cluster

Follow-up to #582.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 17:05:43 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 8e33c4b6aa Rework abuse investigation with cross-account correlations (#582)
## Summary
- **Fix** broken user profile link URL (was 404, now goes to correct
detail page)
- **Add** internal/test account filtering — excludes @nomadkaraoke.com,
@beveridge.uk, @inbox.testmail.app from all abuse views
- **New** Correlations tab (now default) showing fingerprint clusters
(RED flag) and IP clusters (ORANGE flag), with user agent matching
highlighted (YELLOW signal)
- **Demote** old "Suspicious Accounts" to secondary tab

## Motivation
The existing "Suspicious Accounts" view flagged too many false positives
— legitimate users who just used their 2 free credits, internal test
accounts, etc. The real abuse signal is **cross-account correlation**:
multiple accounts sharing the same device fingerprint or signup IP.

## Changes
| File | What |
|------|------|
| `backend/utils/test_data.py` | Add internal email domain filtering |
| `backend/services/user_service.py` | New
`find_account_correlations()`, filter suspicious accounts |
| `backend/api/routes/admin.py` | New `GET
/api/admin/abuse/correlations` endpoint |
| `frontend/lib/api.ts` | New types + API client method |
| `frontend/app/admin/abuse/page.tsx` | Fix link URL, new Correlations
tab, restructure tabs |
| `backend/tests/test_anti_abuse_investigation.py` | 10 new tests |

## Test plan
- [x] 27 anti-abuse investigation tests pass (10 new)
- [x] 2749 backend tests pass
- [x] Frontend dev server starts without errors
- [ ] Manual: verify correlations tab shows real data in prod

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 16:40:07 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 c9e434037b feat: admin feedback visibility — page + email notification (#581)
<!-- @coderabbitai ignore -->

## Summary

- **Admin feedback page** at `/admin/feedback` — browse all user
feedback submissions with star ratings, text fields, search, pagination,
and aggregate stats
- **Email notification** to admin@nomadkaraoke.com whenever a user
submits new feedback (configurable via `ADMIN_NOTIFICATION_EMAIL` env
var)
- **Admin API endpoint** `GET /api/admin/feedback` with pagination,
search, test-data filtering, and average ratings

## Context

User feedback was stored in Firestore (`user_feedback` collection) but
completely invisible to the admin — the admin panel only showed a
boolean "Feedback Submitted: Yes/No" badge per user. This PR closes that
gap.

## Changes

**Backend:**
- `GET /api/admin/feedback` endpoint with pagination, search,
`exclude_test` filter, aggregate rating averages
- `send_feedback_notification()` method on `EmailService` — sends HTML
email with star ratings, all text fields, link to admin page
- Wired into `submit_user_feedback` as fire-and-forget (won't block user
response)

**Frontend:**
- New admin page with aggregate stat cards, searchable/paginated table,
visual star ratings, expandable rows for text detail
- "Feedback" sidebar nav item with MessageSquare icon
- `adminApi.listFeedback()` client method + TypeScript types

**Tests:** 14 tests covering endpoint (pagination, search, filtering,
aggregates, empty state, datetime edge cases), email (content, subject,
custom admin, failure), and caller-callee contract integration

## Test plan

- [x] `make test` — 712 tests pass (39 suites)
- [x] 14 new tests in `test_admin_feedback.py`
- [ ] Manual: verify admin page renders at `/admin/feedback` after
deploy
- [ ] Manual: submit feedback → verify email arrives at
admin@nomadkaraoke.com

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 16:18:32 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 17265a6f28 fix: runner manager idle check misses python-audio-separator jobs (#579)
## Summary

- The `check_github_for_pending_jobs` function in the runner manager
Cloud Function only checked `karaoke-gen` for active workflow runs
- When `python-audio-separator` had queued GPU integration tests, the
idle checker saw no pending jobs and stopped the GPU runners prematurely
- Now checks both `karaoke-gen` and `python-audio-separator` repos

Already deployed via `pulumi up` (infra change — deploy first, merge
second).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 15:57:30 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 5e7ede1cfe feat: show audio-separator version in footer (#580)
## Summary

- Adds audio-separator package version to the version footer display
- Shows as `● Separator: v0.43.1` alongside existing Encoder and
Flacfetch versions
- Uses locally installed package version (`importlib.metadata`) — does
NOT call the remote Cloud Run GPU service, avoiding unnecessary
cold-starts (~$0.07 per wake)

## Test plan

- [x] All 712 tests pass (`make test`)
- [ ] After deploy: verify footer shows separator version on
gen.nomadkaraoke.com

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 19:47:34 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 7294db2f6d fix: ASS subtitle preview/render divergence — embedded newlines drop lyrics (#578)
## Summary

- **Bug:** Lyrics that rendered correctly in preview videos were missing
words in final (with_vocals/final) video output. Discovered on job
`0d4b550c` (The Ataris - All Souls Day).
- **Root cause:** Word objects from the transcription pipeline contain
embedded newline characters (e.g., `"spell\n"`) at segment boundaries.
These corrupt ASS subtitle dialogue events by splitting them across file
lines, causing the video renderer to silently drop words after the
newline.
- **Why preview worked but final didn't:** Two divergent
CorrectionResult construction paths — preview used
`update_correction_result_with_data()` which `.strip()`s word text,
while the render worker used manual dict merge + `Word.from_dict()`
which did not strip.

## Changes

- **`Word.__post_init__`** strips whitespace/newlines from text — single
source of truth for Word invariants
- **Defensive strip in `_create_ass_text()`** — belt-and-suspenders
safety net at the ASS output boundary
- **Render worker refactored** to use the same
`CorrectionOperations.update_correction_result_with_data()` method as
preview generation (DRY, eliminates divergence)
- **Removed redundant `.strip()`** from
`update_correction_result_with_data()` Word text (now handled by
`__post_init__`)
- **Documented lesson** in `docs/LESSONS-LEARNED.md`: "Data Types Must
Enforce Their Own Invariants"
- **8 unit tests** covering Word stripping and ASS output sanitization

## Testing

- [x] 1126 unit tests pass, 0 failures
- [x] Sanity check with real bug data from job `0d4b550c` confirms fix
- [x] CodeRabbit CLI review completed, feedback addressed

## Review

- [x] Local CodeRabbit review completed
- [x] Review feedback addressed (test improved to bypass `__post_init__`
and exercise defensive strip independently)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 23:20:12 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 83c75a9d98 Cutover: audio separation Modal → Cloud Run GPU (#577)
@coderabbitai ignore

## Summary
- Switch `AUDIO_SEPARATOR_API_URL` from Modal to Cloud Run GPU service
- Bump `audio-separator` dependency to >=0.43.0 (preset + CLI support)

The Cloud Run GPU service has been tested and verified:
- Health check: healthy, models ready
- Both ensemble presets (instrumental_clean, karaoke) verified
- Full 2-stage separation tested with real audio files

**Note:** Pulumi apply was done locally. The Cloud Run Job env var is
already live.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 02:53:46 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 38b966f30c Modal → GCP audio separation migration (#571)
@coderabbitai ignore

## Summary
- Comprehensive migration plan: Modal → Cloud Run GPU (L4) for audio
separation
- Replaces old stub plan with full implementation details
- **Architecture**: Cloud Run GPU service (scale-to-zero, ~$0.13/job)
- **Model upgrade**: Ensemble presets as default (instrumental_clean +
karaoke) for +35% quality
- **Pipeline optimization**: Decouple separation from lyrics review
critical path
- Implementation will be added to this PR

## Plan highlights
- Phase 1: Cloud Run GPU server (python-audio-separator repo)
- Phase 2: GCP infrastructure via Pulumi (this repo)
- Phase 3: Pipeline decoupling + preset-based model config
- Phase 4: Cutover, monitor, decommission Modal

See `docs/archive/2026-03-22-modal-to-gcp-migration-plan.md` for full
details.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 22:20:55 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 3391e5c9f5 Fix missing pip in GPU runner Python toolcache (#576)
## Summary

- Python 3.13 on GPU runners was built with `--with-ensurepip=install`
but `pip` wasn't actually in site-packages
- `setup-python@v5` requires a working `pip` module — CI failed with `No
module named pip`
- Added explicit `ensurepip --upgrade` bootstrap after `make install`
- Manually fixed all 3 runners (cleared toolcache + bootstrapped pip)

Follows #573, #574, #575 in the GPU runner fix series.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 20:21:29 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 b0575ba62c Fix Poetry permissions on GPU runners (#575)
## Summary

- Follow-up to #574. GPU runners' CI jobs failed with `EACCES:
permission denied, stat '/usr/local/bin/poetry'` because Poetry was
installed under `/root/.local/` (mode 700)
- The `runner` user can't follow symlinks into root's home directory
- Fix: install Poetry to `/opt/poetry` (shared location) using
`POETRY_HOME` env var

## What happened

1. Startup script runs `curl | python3 -` as root → Poetry installed to
`/root/.local/share/pypoetry/`
2. Symlink created: `/usr/local/bin/poetry` → `/root/.local/bin/poetry`
3. Poetry binary's shebang:
`#!/root/.local/share/pypoetry/venv/bin/python`
4. `runner` user can't access anything under `/root/` (mode 700)
5. `setup-python@v5` cache step tries to stat Poetry → EACCES → job
cancellation

## Test plan

- [x] Manually fixed permissions on all 3 running runners (temporary)
- [x] Applied via `pulumi up` — runners recreated with new startup
script
- [ ] Runners will re-register with GitHub after full startup (~15-20
min)
- [ ] Re-run audio-separator CI to confirm

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 20:10:24 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 8baa7d7db2 Fix GPU startup: upgrade kernel when headers unavailable (#574)
## Summary

- Follow-up to #573. The previous fix handled kernel mismatches on
**existing** VMs but failed on **fresh** VMs created from the
`debian-cloud/debian-12` base image
- The base image ships kernel `6.1.0-43` but Debian repos have moved to
`6.1.0-44`, so `apt-get install linux-headers-6.1.0-43-cloud-amd64`
fails with "Unable to locate package"
- The startup script now detects this and upgrades the kernel + reboots
once before proceeding

## What changed

Added kernel upgrade + reboot logic to `github_runner_gpu.sh`:

1. Check if headers are available for the running kernel via `apt-cache
show`
2. If not: install `linux-image-cloud-amd64` +
`linux-headers-cloud-amd64` (meta-packages tracking latest), set a
reboot flag, and reboot
3. After reboot: running kernel matches available headers, flag is
cleaned up, script continues normally with DKMS build + NVIDIA driver
install

The reboot flag (`/var/lib/gpu-runner-kernel-upgraded`) prevents
infinite reboot loops — if headers are still unavailable after one
upgrade attempt, the script gives up with diagnostics.

## Test plan

- [x] Applied new startup script to all 3 GPU runners via `pulumi up
--target`
- [x] Verified runner 1: kernel upgraded from `6.1.0-43` → `6.1.0-44`,
reboot happened, nvidia-smi shows `595.45.04, Tesla T4, 15360 MiB`
- [x] Verified runner 2: same result
- [ ] Waiting for all 3 runners to complete full startup and register
online with GitHub

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 16:52:25 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 067a66fc26 Fix GPU runner NVIDIA driver failure on kernel updates (#573)
## Summary

- GPU runners (github-gpu-runner-1/2/3) failed with `nvidia-smi` errors
because Debian auto-updated the kernel but the NVIDIA kernel module
wasn't rebuilt for the new version
- The startup script's idempotency check (`command -v nvidia-smi`) was
checking for the *binary*, not the *loaded kernel module*, so it
silently skipped driver setup on restarts
- Fixed the startup script to always ensure kernel headers match the
running kernel, rebuild DKMS modules when the nvidia module isn't
loaded, and clean up stale headers

## What happened

1. Debian updated kernel from `6.1.0-43` → `6.1.0-44-cloud-amd64`
2. NVIDIA kernel module was built for `-43` only
3. On next VM start, `nvidia-smi` binary existed (passes `command -v`)
so driver install was skipped
4. But the kernel module couldn't load (built for wrong kernel) →
`nvidia-smi` failed at runtime
5. All 3 GPU integration test jobs failed on PR #273
(python-audio-separator)

## Fix

**Immediate** (already applied to all 3 runners via SSH):
- Installed `linux-headers-6.1.0-44-cloud-amd64`, ran `dkms
autoinstall`, `modprobe nvidia`
- Re-ran CI run 23408072295 → all 4 jobs passed

**Permanent** (this PR — startup script changes):
- Always install kernel headers for `$(uname -r)` on every boot
- Install `dkms` alongside headers (not just as fallback)
- Check `lsmod | grep nvidia` to detect unloaded module → rebuild via
DKMS
- Clean up stale kernel headers for old kernels (prevents disk bloat)
- Add diagnostic output on failure (dkms status, module listing)
- Reduce retry from 30×10s to 10×5s (DKMS rebuild is deterministic)

## Test plan

- [x] Manually fixed all 3 GPU runners and verified `nvidia-smi` works
- [x] Re-ran failing CI (run 23408072295) — all jobs passed
- [ ] After merge + Pulumi deploy, stop/start a GPU runner to verify the
new startup script works end-to-end

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 15:10:13 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 d1a877c8f4 Disable GDrive validator success notifications (#572)
## Summary
- Set `NOTIFY_ON_SUCCESS` to `false` for the GDrive validator Cloud
Function
- Stops "Karaoke GDrive: All Clear" emails and Pushbullet notifications
on successful daily validation
- Error/issue notifications remain active and unaffected

Already deployed via `pulumi up`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 14:23:44 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 be4acbe915 IP geolocation enrichment for admin UI (#570)
Enrich all admin IP displays with country flag + ISP inline, hover
popover for full details. Backend caches via ip-api.com in Firestore.

@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 03:05:18 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 fef5130519 Add stats action to divebar-lookup Cloud Function (#568)
## Summary
- New `stats` action on the divebar-lookup Cloud Function
- Returns catalog, GCS mirror, format breakdown, cross-reference, and KN
data stats
- Used by KJ Controller's new status panel

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 22:14:43 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 25aa2ef8ef Fix three production job failure modes with robust error handling (#565)
@coderabbitai ignore

## Summary

Fixes three distinct production job failures identified from
investigating jobs 503a4ad3, 7be8d1e5, and 1868f323:

- **0-segment lyrics guard**: Block review submission when transcription
produces no lyrics (e.g. karaoke input with no vocals). Backend
validation + frontend warning with guidance. CDG orchestrator gracefully
skips when LRC has no content.
- **Audio processor stage 2 validation**: Validate backing vocals
separation produced all expected stems (mirrors existing stage 1
validation). Defensive null checks prevent confusing NoneType crashes.
Full tracebacks in error logs.
- **AudioShake retry with exponential backoff**: 5 attempts via
tenacity, ~40 min spread (60s base, 3x multiplier), retries on
5xx/429/connection errors. Applied to all HTTP call points. Fixed file
handle exhaustion bug on upload retry.

Also includes:
- Troubleshooting runbooks for all 3 failure modes
- Lessons learned entry on pipeline robustness patterns
- Seed planning doc for Modal-to-GCP audio separation migration
- Version bump to 0.147.0

## Test plan

- [x] 2696 backend tests passing (54 new/modified)
- [x] 712 frontend tests passing (5 new for ReviewChangesModal)
- [x] HTTP endpoint test verifies 400 response for 0-segment review
submission
- [x] AudioShake retry tests verify 502 retry, 400 no-retry, connection
error retry
- [x] Audio processor tests verify stage 2 validation, null safety,
combined instrumentals
- [x] CDG orchestrator tests verify graceful skip on empty LRC, fail on
valid LRC errors
- [x] LRC timestamp regex tests for millisecond, centisecond, and
no-decimal formats

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 17:58:31 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 de02570514 fix: real client IPs + user detail identity info (#567)
## Summary

Two fixes:

1. **Real client IPs** — All stored IPs were `169.254.169.126` (Cloud
Run LB). Now uses `X-Forwarded-For` header via shared `get_client_ip()`
helper. Fixed in magic link, verify, and job creation endpoints.
Backfill endpoint updated to clear bad IPs and re-fetch.

2. **User detail identity info** — Admin user detail page now shows:
- Identity & Security card: signup IP, device fingerprint, welcome
credits status
   - Active sessions: IP, user agent, fingerprint, last activity
   - Total spent and feedback status

After deploy, re-run backfill to fix the bad IPs:
```bash
curl -X POST .../api/admin/abuse/backfill-ips -H "Authorization: Bearer $TOKEN"
```

@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 21:48:14 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 51b2a170f0 Switch Divebar downloads from Google Drive to GCS mirror (#566)
## Summary
- divebar-lookup Cloud Function now returns public GCS URLs for synced
files
- Falls back to Drive URLs for files not yet in GCS mirror
- Search results include `in_gcs` boolean field
- GCS bucket set to public read (same access as source Drive folder)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 21:27:36 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 2f10d5b54e fix: backfill-ips missing index (#564)
Remove order_by from backfill query — sessions(user_email, created_at)
index doesn't exist.

@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 21:08:07 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 ede4538276 Admin abuse UI: sortable columns + IP backfill (#562)
Sortable table columns + backfill endpoint for existing user IPs.

After deploy, run: `curl -X POST .../api/admin/abuse/backfill-ips -H
"Authorization: Bearer $TOKEN"`

@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 20:40:21 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 9683f198fd Add GCS file mirror for Divebar karaoke collection (~779 GB) (#560)
## Summary
- GCS bucket `nomadkaraoke-divebar-files` for mirroring all actual
karaoke files from the Divebar Google Drive
- ~48,533 files, ~779 GB total (MP4s: 570 GB, ZIPs: 123 GB, CDG+MP3: 33
GB, etc.)
- Sync script reads BigQuery index, downloads from Drive API, uploads to
GCS
- Tracks progress via `gcs_path` column (incremental — only downloads
new/unsynced files)
- Estimated storage cost: ~$16/month (Standard), can lifecycle to
Nearline later

## Test plan
- [x] Pulumi deployed: bucket + IAM + BigQuery schema update
- [ ] Initial bulk sync via `python divebar_file_sync.py --workers 4`
(run from Cloud Shell or GCE VM)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 20:18:47 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 a90017889a Update ARCHITECTURE.md with data pipeline Cloud Functions (#557)
## Summary
- Document 3 new data pipeline Cloud Functions (divebar-mirror,
kn-data-sync, divebar-lookup)
- Add infrastructure details (service accounts, GCS buckets, BigQuery
tables, secrets)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 20:03:22 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 0e721a15f1 Fix Firestore index Pulumi errors: remove stale import declarations (#556)
## Summary
- Remove stale `import_=` declarations from 3 Firestore indexes that
were already imported
- These caused "previously-imported resources may not be replaced"
errors blocking all Pulumi deploys
- Re-imported all 13 indexes into Pulumi state with current GCP values
- Pulumi now shows 186 resources, zero pending changes, zero errors

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 18:36:31 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 a05cc7ce6c Admin UI: Abuse Investigation panel (#555)
## Summary

New admin panel at `/admin/abuse` for investigating potential
free-credit abuse patterns.

## Features

### Suspicious Accounts tab
- Table of accounts with many jobs but no spend (configurable
thresholds)
- Shows: email, jobs, credits, spend, feedback, signup IP, fingerprint,
created date
- Click any row to drill into related accounts

### Lookup tab
- Search by email, IP address, or device fingerprint
- Email search opens the related accounts view
- IP/fingerprint search returns a table of all matching users

### Related Accounts view
- Target user details (IP, fingerprint, credits, jobs, spend)
- Linked accounts grouped by shared IP and shared fingerprint
- Clickable drill-down to investigate any related account
- Link to full user profile page

## Changes
- `frontend/app/admin/abuse/page.tsx` (new) — full page with 3 views
- `frontend/components/admin/admin-sidebar.tsx` — Fingerprint icon + nav
entry
- `frontend/app/admin/layout.tsx` — breadcrumb support
- `frontend/lib/api.ts` — 4 adminApi methods + TypeScript types

## Testing
- [x] Frontend build passes
- [x] All tests pass (707 frontend + all backend)
- [x] No backend changes (uses endpoints from PR #553)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 14:01:01 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 e1edc64a31 Divebar Drive mirror, KN pipeline migration, and cross-reference index (#554)
## Summary
- **Phase 1**: Cloud Function `divebar-mirror` — recursively indexes 84
brand folders in Divebar Google Drive into BigQuery `divebar_catalog`
table. Daily at 2 AM ET.
- **Phase 2**: Cloud Function `kn-data-sync` — replaces legacy
`projectbread-karaokay` pipeline. Fetches full KN catalog (281K songs) +
community tracks (58K with YouTube URLs). Stores to GCS + refreshes
BigQuery. KN API key in Secret Manager.
- **Phase 3**: Cloud Function `divebar-lookup` — public API for search,
KN ID lookup, and cross-reference rebuild. Links KN songs to Divebar
files via artist+title matching.

All three are Pulumi-managed with dedicated service accounts, Cloud
Scheduler triggers, and minimal IAM permissions.

## Infrastructure deployed
- 3 Cloud Functions (Gen2, Python 3.12)
- 3 Service accounts with least-privilege IAM
- 4 GCS buckets (3 source + 1 data)
- 3 BigQuery tables (`divebar_catalog`, `karaokenerds_community`,
`kn_divebar_xref`)
- 5 Cloud Scheduler jobs
- 1 Secret Manager secret (`karaokenerds-api-key`)

## Test plan
- [x] KN full catalog sync: 281,007 songs loaded in 60s
- [x] KN community sync: 58,825 tracks loaded in 17s
- [x] Divebar mirror: deployed, running initial sync (cycle detection
fix applied)
- [ ] Cross-reference rebuild after mirror completes
- [ ] KJ Controller search integration (in kjbox PR)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 16:15:52 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 2d47fa7ab3 Anti-abuse: queryability improvements + investigation endpoints (#553)
## Summary

Follow-up to #552. Makes anti-abuse data queryable for investigation and
adds ongoing tracking beyond signup.

- **Firestore composite indexes** for `signup_ip` and
`device_fingerprint` on `gen_users`
- **Session fingerprint tracking** — device fingerprint propagated from
magic link to session
- **Job creation IP** — client IP stored on every job for cross-account
correlation
- **Admin investigation endpoints** — find related accounts, suspicious
users, lookup by IP/fingerprint

## Changes

### Infrastructure
- 2 new Firestore composite indexes: `gen_users(signup_ip, created_at)`
and `gen_users(device_fingerprint, created_at)`

### Backend
- `Session` model: added `device_fingerprint` field
- `create_session()`: accepts and stores `device_fingerprint`
- Verify endpoint: passes magic link's fingerprint to session
- `Job` model: added `creation_ip` field  
- Both job creation endpoints: store client IP on job document
- New `UserService` methods: `find_users_by_signup_ip()`,
`find_users_by_fingerprint()`, `find_suspicious_accounts()`,
`find_related_accounts()`
- New admin endpoints:
- `GET /api/admin/abuse/related/{email}` — find linked accounts by
shared IP/fingerprint
- `GET /api/admin/abuse/suspicious?min_jobs=2&max_spend=0` — free-credit
abusers
  - `GET /api/admin/abuse/by-ip/{ip}` — all users from an IP
- `GET /api/admin/abuse/by-fingerprint/{fp}` — all users with a
fingerprint

### Tests
- 15 new tests in `test_anti_abuse_investigation.py`
- Full suite passes (707 frontend + all backend)

## Testing
- [x] Unit tests pass
- [x] Integration tests pass
- [x] `make test` passes

## Review
- [x] Local review completed

**Note:** Infrastructure changes (Firestore indexes) deploy
automatically on merge. Run `pulumi up` locally first per CLAUDE.md
guidelines.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 16:07:14 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 376008fc8f Anti-abuse: credits on verification, IP + fingerprint rate limiting (#552)
## Summary

Three-layer defense against free credit abuse (disposable email bypass):

- **Credits on verification**: Welcome credits granted on first magic
link verification, not account creation. Unverified accounts get 0
credits.
- **Per-IP rate limiting**: Max 2 new accounts per IP per 24 hours.
Silent reject (anti-enumeration).
- **Device fingerprinting**: FingerprintJS (open source) sends browser
fingerprint with magic link request. Combined with IP — either signal
hitting 2/day triggers block. Graceful degradation if fingerprinting
fails.

## Changes

### Backend
- `get_or_create_user()` creates users with 0 credits, stores
`signup_ip` and `device_fingerprint`
- New `grant_welcome_credits_if_eligible()` with
`welcome_credits_granted` flag for idempotency
- Verify endpoint grants credits and returns `credits_granted` in
response
- `count_recent_signups_from_ip()` and
`count_recent_signups_from_fingerprint()` query Firestore
- `is_signup_rate_limited()` combines IP + fingerprint signals
- `send_magic_link` endpoint checks rate limit before creating magic
link (new users only)
- New fields: `User.signup_ip`, `User.device_fingerprint`,
`User.welcome_credits_granted`, `MagicLinkToken.device_fingerprint`,
`SendMagicLinkRequest.device_fingerprint`,
`VerifyMagicLinkResponse.credits_granted`

### Frontend
- New `fingerprint.ts` lib (lazy-loaded FingerprintJS)
- `sendMagicLink` sends `device_fingerprint` with request
- Verify page shows "You received N free credits to get started!" on
first login
- `VerifyMagicLinkResponse` type updated with `credits_granted`

### Tests
- 33 unit + integration tests in `test_anti_abuse.py`
- Covers: credit granting, idempotency, rate limiting (IP +
fingerprint), error paths, caller-callee contracts, model fields,
endpoint integration

## Testing
- [x] All backend tests pass (33 new + existing)
- [x] All frontend tests pass (707 tests)
- [x] `make test` passes

## Review
- [x] CodeRabbit CLI review completed locally
- [x] Review feedback addressed (added `welcome_credits_granted` flag
for idempotency)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 01:51:44 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 4e7439e80e feat: tiered disposable email detection via external APIs (#550)
## Summary

- Add tiered disposable email checking: static blocklist → DeBounce API
(free) → verifymail.io API (paid) — catches domains the static list
misses
- Auto-learn: flagged domains persist to Firestore `manual_domains`, so
each domain only needs one API call ever
- Clean domain cache: verified-clean domains skip API calls for 7 days
- Well-known providers (gmail, yahoo, etc.) skip all checks instantly
- Graceful degradation: if external APIs are down, static blocklist
still works

Investigation of 60 recent signups found 10 disposable accounts — the
static list only caught 6, external APIs caught the remaining 4
(including niche `.io.vn` vanity domains and `nghienplus.store`).

## Test plan

- [x] 52 unit tests pass (18 new tests for external API checking)
- [x] All 707 frontend tests pass
- [x] Integration tests verify magic link 422 rejection flow
- [x] `make test` passes clean
- [ ] After deploy: create GCP secret `verifymail-api-key`, then attempt
signup with known-disposable domain not in static list (e.g.,
`test@lxbeta.com`)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 00:04:26 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 a9db298b22 fix: review session cross-job search returns empty (#551)
@coderabbitai ignore

## Summary

- **Fixed missing Firestore collection group index** — the
`review_sessions` subcollection needed a single-field index exemption
with `COLLECTION_GROUP` scope on `updated_at` for the cross-job search
to work. Without it, every `collection_group("review_sessions")` query
silently failed with a 400 error, caught by the try/except and returned
as `[]`.
- **Fixed over-fetch before filter bug** — even with the index, the
search was applying `.limit(20)` at the Firestore level before filtering
by text in Python. Sessions not in the 20 most recent across all jobs
would never match a text search. Now over-fetches (up to 500) when a
query is provided, filters in Python, then trims to the requested limit.
- **Documented the index creation command** in the IaC module since
Pulumi doesn't support single-field index exemptions natively.

## Root cause

The `search_review_sessions` endpoint uses a Firestore collection group
query (`db.collection_group("review_sessions")`) which requires an index
with `COLLECTION_GROUP` query scope. This index was never created, so
every cross-job search silently returned empty results.

## Test plan

- [x] Verified the Firestore index was created and is READY
- [x] Tested `GET /api/review/sessions/search?q=&limit=5` returns
sessions (was empty before)
- [x] Tested `GET /api/review/sessions/search?q=d1695003&limit=5`
returns matching sessions
- [x] All 707 tests pass (`make test`)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 23:54:53 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 124036716a fix: show Edit Audio button for in_audio_edit status (#549)
## Summary
- The "Edit Audio" button on job cards was only shown for
`awaiting_audio_edit` status, but not `in_audio_edit`
- Users who navigated away from the audio editor had no button to return
to it from the job card
- Added `in_audio_edit` to both the interactive status check and the
button render condition

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 23:07:31 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 fe7fdc8ed5 fix: map with_vocals_mp4 in GCE encoding output (#548)
## Summary

- **Fix GCE encoding backend** dropping the `(With Vocals).mp4` file
from output results — the list-to-dict mapper didn't recognize the "with
vocals" filename pattern, so `finals/with_vocals_mp4` was never set in
Firestore
- **Set `with_vocals_mp4_path`** in the `EncodingOutput` returned by
`GCEEncodingBackend.encode()` so the orchestrator can thread it through
to upload
- **Fix local encoding edge case** where input already being MP4 caused
the with-vocals output to be silently skipped instead of copied

PR #535 threaded `with_vocals_mp4` through the pipeline but missed these
two spots in the GCE backend path, which is what production uses.

## Test plan

- [x] 27 encoding interface tests pass (including new regression test)
- [x] 38 local encoding service tests pass
- [x] Full backend suite: 2323+ tests pass
- [x] Full frontend suite: 707 tests pass (38 suites)
- [x] CodeRabbit: no findings
- [x] Manual: fixed job `fa1e2eab` in Firestore — download now serves
39.6 MB MP4 instead of 120.4 MB MKV

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 17:23:03 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 05cb7a9bec feat: auto-sync disposable email blocklist from community repo (#546)
## Summary

- Add daily auto-sync of disposable email domains from the
[disposable-email-domains](https://github.com/disposable-email-domains/disposable-email-domains)
community repo (~4,800 domains vs previous 160 hardcoded)
- Restructure Firestore blocklist into three layers: external
(auto-synced), manual (admin-added), allowlisted (admin overrides)
- Admin UI updated with separate sections for each domain source + "Sync
Now" button

## Changes

### Backend
- New `disposable_domain_sync_service.py` — fetch, parse, and sync
external blocklist with migration support
- Updated `email_validation_service.py` — effective set = (external +
manual) - allowlisted, new allowlist CRUD, backward-compatible
`get_blocklist_config()`
- New API endpoints: `POST /blocklists/sync`, `POST/DELETE
/blocklists/allowlisted-domains`, `POST
/internal/sync-disposable-domains`
- Updated `GET /blocklists` response with `external_domains`,
`manual_domains`, `allowlisted_domains`, `last_sync_at`,
`last_sync_count`

### Frontend
- Admin rate-limits page: three sub-sections (external/manual/allowlist)
with sync status bar
- Updated API client types and methods

### Infrastructure
- Cloud Scheduler job for daily sync at 3:00 AM UTC (Pulumi)

### Migration
- First sync automatically migrates from old `disposable_domains` field
- Hardcoded `DEFAULT_DISPOSABLE_DOMAINS` replaced with empty set
(external sync takes over)

## Testing
- 47 new/updated tests across 3 test files (54 total for email
validation + sync)
- 2584 tests pass in full suite
- CodeRabbit CLI: no findings

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 18:45:51 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 4cdb9884df fix: stale review processor uses list_jobs (not get_jobs) (#547)
## Summary

- Fix `get_jobs` → `list_jobs` — the correct method name on
`FirestoreService`
- Production logs showed: `'FirestoreService' object has no attribute
'get_jobs'`

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 21:24:15 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 6b17a1c748 feat: review reminder emails and auto-expiry for stale jobs (#544)
## Summary

- Hourly cron detects jobs stuck in review (`awaiting_review` /
`in_review`) and takes action
- **24h**: sends gentle reminder email with review link, help offer, and
expiry warning
- **48h**: auto-cancels job, refunds credit, sends expiry notification
with balance

## Changes

- **New:** `backend/workers/stale_review_processor.py` — core processor
logic
- **New:** `EmailService.send_review_reminder()` and
`send_review_expired()` email methods
- **New:** `POST /api/internal/process-stale-reviews` endpoint (Cloud
Scheduler target)
- **New:** Cloud Scheduler job `stale-review-hourly` (hourly, Pulumi)
- **New:** 24 unit tests (14 processor + 10 email)
- **Fix:** `make test` no longer hangs on frontend Playwright E2E tests
- Excludes made-for-you and tenant jobs
- Uses separate `expiry_reminder_sent` flag (independent of existing
idle reminder)
- Version bump: 0.142.4 → 0.143.0

## Testing

- [x] 24 new unit tests, all passing
- [x] Full backend test suite: 2578 passed
- [x] Full frontend unit tests: 704 passed
- [x] `make test` completes without hanging

## Review

- [x] Local CodeRabbit review completed (5 findings, 4 fixed, 1
correctly skipped)
- [x] Spec review completed (critical `reminder_sent` flag collision
caught and fixed)

## Design

See `docs/archive/2026-03-19-review-reminder-autoexpiry.plan.md` for
full spec.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 16:15:01 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 f582236d18 fix: prevent flaky "docker not found" in CI emulator tests (#543)
## Summary

- Fix race condition where GHA runner systemd service auto-starts on VM
boot and accepts CI jobs before Docker is available
- Add defense-in-depth Docker readiness check in the emulator test
workflow

## Changes

- **`infrastructure/compute/startup_scripts/github_runner.sh`** — Stop
the runner service at the very top of the startup script, before any
package installations, preventing premature job acceptance
- **`.github/workflows/ci.yml`** — Add a "Wait for Docker to be
available" step with 60s retry loop in `backend-emulator-tests` job
- **`docs/SELF-HOSTED-RUNNERS.md`** — Document Bug 8 in the Historical
Bugs section

## Root Cause

When runner VMs are stopped/started by the runner_manager, the GHA
runner systemd service (from `svc.sh install` on a previous boot)
auto-starts immediately on boot. The GCE startup script runs
concurrently but only stops/re-registers the runner service after
installing Docker and other dependencies. During this window, the runner
can accept jobs that fail because Docker isn't ready yet.

## Testing

- [x] Infrastructure-only change — no unit tests applicable
- [x] Verified startup script logic is correct (subshell for cd,
idempotent)
- [x] CI workflow change adds graceful retry, not a breaking change

## Review

- [x] Local CodeRabbit review completed (no findings)
- [x] Documentation updated

## Note

The startup script change will trigger Pulumi to detect a metadata
change on runner VMs. Run `pulumi up` locally before merging per project
convention.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 15:29:58 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 b8239dc691 Block disposable email registration with friendly messaging (#542)
## Summary

- Wire existing `EmailValidationService` into the magic link signup flow
to reject disposable email domains (yopmail, tempmail, emailhook.site,
etc.) before creating users
- Return 422 with machine-readable `disposable_email_not_allowed` error
code for disposable domains
- Silent rejection for explicitly blocked emails/IPs (anti-enumeration)
- Frontend shows a warm amber message box encouraging real email use,
promising no spam
- Add `emailhook.site` to default disposable domains list (160+ domains
total)

## Context

Users were signing up with disposable emails like `anything@yopmail.com`
to get unlimited free welcome credits (2 per account). The
`EmailValidationService` existed with a comprehensive blocklist but was
never called during signup. This PR wires it in.

The admin UI for managing the blocklist already exists at
`/admin/rate-limits/` (Blocklists tab) with full CRUD for domains,
emails, and IPs.

## Test plan

- [x] 8 new backend integration tests (disposable rejection, blocked
email/IP, email lowercasing, domain list)
- [x] 2 new frontend tests (friendly message rendering, generic error
fallback)
- [x] All 2562 backend tests pass
- [x] All 16 frontend auth component tests pass
- [x] CodeRabbit review: no findings

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 14:33:41 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 d0efa9722f infra: only start GPU runners for gpu-labeled jobs (#541)
## Summary
- GPU runners (3x n1-standard-4 + T4, ~$1.62/hr combined) were starting
on every CI trigger despite no workflow using the `gpu` label
- Now the runner manager checks job labels and only wakes GPU runners
when `gpu` is in the label set
- Also includes the int→float fix for `IDLE_TIMEOUT_HOURS` and 15min
idle timeout from #540

## Changes
- `start_runners(include_gpu=False)` — new parameter, skips GPU runners
by default
- `GPU_RUNNER_NAMES` env var tells the function which runners are GPU
- Webhook handler passes `include_gpu="gpu" in labels`
- 2 new tests for GPU filtering (26 total, all passing)
- Pulumi wires GPU runner names as separate env var

## Test plan
- [x] 26 unit tests passing
- [x] Pulumi applied locally — function deployed with GPU_RUNNER_NAMES
env var
- [x] Function responding correctly (verified via curl)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 09:26:10 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 951c1d543e infra: reduce runner idle timeout to 15min, fix int->float parsing (#540)
## Summary
- Reduce idle timeout from 1h to 15min (0.25h) — solo dev project, no
need for long idle periods
- Fix `int()` -> `float()` for `IDLE_TIMEOUT_HOURS` env var parsing
(crashed on "0.25")

## Test plan
- [x] Pulumi applied locally, function deployed via gcloud
- [x] All 7 runners successfully stopped (verified TERMINATED status)
- [x] Logs confirm: creationTimestamp fallback, parallel stops, all
runners managed

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 09:06:28 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 3af68a5047 fix: runner manager idle shutdown bugs preventing scale-to-zero (#539)
## Summary
- Fixed 6 bugs in the runner manager Cloud Function that prevented idle
shutdown, causing all 7 runner VMs to run 24/7 (~$1,600/mo wasted GCP
credits)
- Added IAM permission (`iam.serviceAccountUser`) so the runner manager
can set metadata on runner VMs
- All fixes learned from the Aquarius project's identical journey
(documented in docs/SELF-HOSTED-RUNNERS.md)

## Changes
- **infrastructure/functions/runner_manager/main.py** — Complete rewrite
applying all Aquarius Phase 4 fixes: await metadata ops,
creationTimestamp fallback, parallel stops, STOPPING state handling,
per-runner metadata updates
- **infrastructure/modules/runner_manager.py** — Added
`iam.serviceAccountUser` IAM binding
- **infrastructure/__main__.py** — Pass runner SA to runner manager for
IAM binding
- **docs/SELF-HOSTED-RUNNERS.md** — Comprehensive doc (architecture,
cost model, bug history, troubleshooting)
- **docs/LESSONS-LEARNED.md** — 3 new infrastructure lessons
- **docs/GCP-COST-OPTIMIZATION.md** — Corrected status and cost
estimates

## Test plan
- [x] 24 unit tests covering all 6 bug fixes (all passing)
- [x] `pulumi up` applied locally — IAM binding created, function source
updated
- [x] Function deployed and processing webhooks (verified via logs)
- [ ] Monitor Cloud Function logs over next hour to confirm idle
shutdown triggers
- [ ] Verify all runners reach TERMINATED state after 1h idle

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 08:32:37 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 c443f99517 infra: 3x on-demand GPU runners + startup reliability (#538)
## Summary

- Scale GPU runners from 1 to 3 for parallel CI jobs
- Switch from spot to on-demand (spot kept getting preempted during
15-min startup)
- Fix startup script: build Python 3.13 from source (pyenv broken on
Debian 12 + CUDA), DEBIAN_FRONTEND=noninteractive, deferred runner
start, nvidia-smi retry

All 3 runners verified online with clean startup — no manual hacks.

Infrastructure already applied via `pulumi up`.

## Test plan

- [x] All 3 runners booted from clean startup script
- [x] GPU, Python 3.13, tool cache, 45 model files verified on each
- [x] Integration tests passed on GPU runner (PR #265)

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 21:04:33 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 33eaaec4c0 fix: guard Python tool cache in GPU runner startup (#537)
## Summary

Fix broken Python tool cache on GPU runner that caused `setup-python` to
find an empty cache and fail with `python: command not found`.

The pyenv Python 3.13 compilation was failing silently, but the
`.complete` marker was still being created, tricking `setup-python` into
thinking Python was cached.

Now checks that pyenv actually produced a binary before populating the
tool cache, and cleans up the empty dir if it didn't.

## Test plan

- [x] Verified on running GPU runner via SSH that broken cache was
cleaned
- [x] All workflow steps verified manually: GPU, pipx, poetry, system
deps, model cache, `poetry install -E gpu`, torch CUDA=True

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 19:17:20 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 0401434a07 infra: add GPU runner for audio-separator CI (#536)
## Summary

- Add `n1-standard-4` + NVIDIA T4 GPU spot VM to GitHub runner
infrastructure
- GPU startup script with CUDA 12.4 drivers, model pre-caching,
nvidia-smi verification with retry
- Runner service starts only after all caches are warm (CodeRabbit fix)
- Wired into runner_manager Cloud Function for auto-start/stop on CI
demand

## Details

Cost: ~$0.03 per CI run ($3/mo at 100 runs) — actually cheaper per-run
than current CPU runners because GPU tests finish 5× faster.

Infrastructure already applied via `pulumi up` — GPU runner is online
and verified working:
- `nvidia-smi`: Tesla T4, 15.6 GB VRAM, driver 595.45.04
- PyTorch CUDA: True, ONNX Runtime CUDAExecutionProvider enabled
- 45 model files pre-cached at `/opt/audio-separator-models`
- Runner registered with labels: `[self-hosted, Linux, X64, gcp, gpu]`

Companion PR in python-audio-separator updates the workflow to target
`[self-hosted, gpu]`.

## Test plan

- [x] `pulumi up` applied successfully (1 created, 1 updated)
- [x] GPU runner VM booted and startup script completed
- [x] nvidia-smi verified Tesla T4 accessible
- [x] audio-separator separation test ran with CUDA acceleration
- [x] Runner shows as "online" in GitHub Actions with correct labels
- [ ] Integration tests pass on GPU runner (verified after companion PR
merges)

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 18:06:08 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 a918c5a310 fix: serve encoded MP4 for With Vocals download (3.4x smaller) (#535)
## Summary

- **Serve the encoded MP4** (~39MB) for "With Vocals" download instead
of the raw MKV (~134MB) — 3.4x smaller, universally compatible
- **Fix .mov mislabeling** — the MKV from GCS was being saved with a
`.mov` extension in the temp dir, causing Dropbox to receive a
wrongly-named file
- **Thread `with_vocals_mp4`** through the full pipeline:
`EncodingOutput` → `OrchestratorResult` → `_upload_results()` →
Firestore `file_urls.finals.with_vocals_mp4`
- **Backward compatible** — frontend prefers `finals.with_vocals_mp4`
(MP4) but falls back to `videos.with_vocals` (MKV) for older jobs

## Test plan

- [x] 3 new OutputLinks tests: MP4 preference, MKV fallback, MP4-only
scenario (all pass)
- [x] All 20 OutputLinks tests pass
- [x] 2323 backend tests pass (1 pre-existing failure unrelated)
- [x] CodeRabbit review: 3 findings fixed (GCE download mapping, legacy
path, distribution dir extension)
- [ ] Manual: verify new job produces `finals/with_vocals_mp4` in
Firestore after encoding
- [ ] Manual: verify "With Vocals" download from dashboard serves MP4

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 03:31:17 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 d7c46a1567 infra: pre-download audio separator models on runner startup (#534)
## Summary
- Add audio-separator model pre-download to the GitHub runner startup
script
- Pre-downloads 8 ML models + 3 metadata files (~2GB) to
`/opt/audio-separator-models/`
- Models persist across job runs, eliminating repeated downloads during
CI

## What changed
- Added model download section to
`infrastructure/compute/startup_scripts/github_runner.sh`
- Downloads models from TRvlvr/model_repo, Facebook demucs, and
nomadkaraoke fallback repos
- Idempotent: skips files that already exist, uses fallback URLs for
404s
- Sets ownership to `runner:runner` for CI job access

## Context
Companion to nomadkaraoke/python-audio-separator#266 which updates the
integration test workflow to use this persistent path.

Models are already manually pre-loaded on all 3 current runners — this
ensures future runner recreations also have them.

## Test plan
- [x] Manually ran the download script on all 3 runners via SSH —
verified 15 files, 2.1GB each
- [ ] Integration tests in python-audio-separator#266 should pass using
these pre-cached models

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 00:07:52 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 c0fabf9f54 fix: NameError 'settings' in orchestrated video worker distribution warnings (#533)
## Summary
Fix `NameError: name 'settings' is not defined` in two call sites when
video generation completes with distribution warnings.

## Root Cause
Both `generate_video_orchestrated()` and `_handle_native_distribution()`
called `_send_distribution_warning_notification(settings=settings)` but
never defined `settings` in their scope. This only triggers when
`result.distribution_warnings` is truthy (partial distribution upload
failure) — a rare code path that was never tested.

## Fix
Remove `settings=settings` argument from both call sites. The function
already handles `settings=None` by calling `get_settings()` internally.

## Testing
- [x] 2555 backend tests pass
- [x] 4 new regression tests added:
- Contract test: `_send_distribution_warning_notification` works without
settings arg
- Static analysis: `generate_video_orchestrated` has no
`settings=settings` reference
- Static analysis: `_handle_native_distribution` has no
`settings=settings` reference
- Integration: `_handle_native_distribution` with distribution_warnings
doesn't crash

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 23:50:27 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 e9ec343598 fix: sanitize artist/title in encoding output paths for slash characters (#532)
## Summary
- Fix encoding failure when song title contains `/` characters (creates
subdirectories in output paths)

## Root Cause
The GCE encoding worker's `run_encoding()` builds output file paths
using raw `artist` and `title`, causing `/` in titles (e.g., "MEDLEY -
PORTRAIT/WALK/BORN") to be treated as directory separators. The encoding
fails at the remux step because the output directory doesn't exist.

## Fix
Sanitize `artist` and `title` with `re.sub(r'[<>:"/\\|?*]', '_', ...)`
before using them in `base_name` for output paths.

## Testing
- [x] 55 encoding tests pass
- [x] 2551 backend tests pass

## Context
Follow-up to PRs #530 and #531. Job 27054772 now passes CDG generation
(fixed in #531) but fails at encoding due to the same slash-in-title
issue in the GCE encoding worker.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 17:08:52 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 00a6cabc5e fix: sanitize artist/title in CDG packaging paths (#531)
## Summary
- Fix CDG generation failure when song title contains `/` characters
(e.g., "MEDLEY - PORTRAIT/WALK/BORN")

## Root Cause
The orchestrator's `_run_packaging()` built CDG output paths using raw
`artist` and `title`, causing `/` in titles to be treated as directory
separators. This resulted in `FileNotFoundError` when the CDG generator
tried to write the TOML file.

## Fix
Use `sanitize_filename()` for `base_name` in `_run_packaging()`,
matching the pattern already used in `_run_encoding()` and
`create_orchestrator_config_from_job()`.

## Testing
- [x] 2551 backend tests pass (including 57 orchestrator tests)

## Context
Follow-up to PR #530. Discovered during retry of job 27054772 which has
title "MEDLEY - PORTRAIT/WALK/BORN".

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 16:44:56 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 1c0eb8bf52 fix: prevent style_params.json lifecycle deletion, CDG font fallback, video worker state guard (#530)
## Summary
- Fix GCS lifecycle rule silently deleting `style_params.json` after 7
days by moving style files from `uploads/` to `jobs/` prefix
- Improve CDG font fallback chain to try bundled fonts by filename
before failing, preventing opaque `KeyError: 'font'` errors
- Add video worker state guard (matching audio/lyrics/render worker
pattern) to prevent cascading retry errors on failed jobs

## Changes
- **Style path prefix**: Changed all `uploads/{job_id}/style/`
references to `jobs/{job_id}/style/` in `theme_service.py`,
`file_upload.py`, `audio_search.py` and 7 test files
- **CDG font fallback**: Enhanced `_validate_and_setup_font()` with
5-step resolution (exact → bundled full path → bundled by filename →
arial.ttf → None), added None guard in `_create_toml_data()`,
absolutized all resolved paths
- **Video worker guard**: Added `validate_worker_can_run()` to both
orchestrated and legacy video pipelines, added terminal state check
(`failed`, `cancelled`, `complete`, `prep_complete`, `error`) to
`_check_worker_idempotency()`
- **Docs**: Added GCS lifecycle gotcha to LESSONS-LEARNED.md

## Context
Triggered by investigation of two production job failures:
- Job `2d4a7bab` (Mar 5): `style_params.json` 404 after 10 days — GCS
lifecycle rule deleted it
- Job `27054772` (Mar 14): CDG generation failed with `KeyError: 'font'`
when font couldn't be resolved, then Cloud Tasks retries produced
cascading `Invalid state transition: failed -> GENERATING_VIDEO` errors

## Testing
- [x] 2551 backend tests pass
- [x] 139 remote CLI tests pass
- [x] 10 new tests added (5 font fallback, 5 state guard/idempotency)

## Review
- [x] Local CodeRabbit review completed (3 findings, 2 fixed, 1 skipped
as nitpick)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 16:19:59 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 c69b129893 fix: add lyrics_dir to transcribe_lyrics() for metadata storage (#529)
## Summary

- **Bug fix**: `transcribe_lyrics()` didn't include `lyrics_dir` in its
return dict, so `_store_lyrics_processing_metadata()` could never find
`corrections.json`. AudioShake IDs and correction stats were silently
lost in production (v0.142.0) because the metadata storage is wrapped in
try/except.
- **One-line fix**: Add `"lyrics_dir": lyrics_dir` to the
`transcriber_outputs` dict in `lyrics_processor.py`
- **Docs improvement**: Added "Test the Contract Between Caller and
Callee" guidance to `TESTING.md` and `/test-review` skill to prevent
this class of bug in future

## Why testing didn't catch it

The test for `_store_lyrics_processing_metadata` hand-crafted its input:
`transcription_result={"lyrics_dir": str(lyrics_dir)}`. This tested the
function in isolation — proving it works *if given correct input* — but
never verified the actual caller (`transcribe_lyrics()`) produces that
key. The mock was "too helpful."

This is especially dangerous with non-fatal code paths: the function is
designed to never crash (try/except), so the broken input silently did
nothing, and the test still passed.

## Test plan

- [x] 3 new tests: contract test (transcribe_lyrics returns lyrics_dir),
integration test (lyrics_dir points to corrections.json location),
regression test (pre-fix behavior: only duration stored without
lyrics_dir)
- [x] All 27 processing metadata tests pass
- [x] Verify in production: new jobs should show AudioShake IDs in
`processing_metadata.transcription`

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 11:16:20 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 694a7ff04e feat: add audio source details modal to admin panel (#528)
## Summary

- **Fix source detection**: `getJobSource` now checks
`audio_source_type` field first, correctly identifying audio search jobs
even when `audio_search_artist`/`audio_search_title` are null (was
showing upload icon for RED audio search jobs)
- **Add source details modal**: Clicking the source text opens a dialog
showing all available source metadata (type, provider, source ID,
filename, target file, download URL, etc.)
- **Add source fields to API**: Backend now includes
`audio_source_type`, `source_name`, `source_id`, `target_file`,
`download_url` in the summary field projection

## Test plan

- [x] Backend: Regression test asserting all source fields are in
`SUMMARY_FIELD_PATHS` (11 tests pass)
- [x] E2E regression: 3 new tests for source modal (audio search,
YouTube, file upload)
- [x] E2E regression: Updated existing YouTube source test for new modal
behavior
- [x] Full admin-job-detail E2E suite: 24/24 passing

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 02:55:29 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 deb870a980 feat: store processing metadata permanently for all workers (#527)
## Summary

- Add `processing_metadata` field to Job model — immutable provenance
data written by each worker during processing, separate from
`state_data` (mutable workflow state)
- **Critical fix**: Persist AudioShake `task_id` and `asset_id` which
were previously lost after processing, requiring costly log/API recovery
for data exports
- Each worker writes its own section: transcription (AudioShake IDs),
separation (models, timing), rendering, encoding, distribution, audio
source (ffprobe), and worker timing
- All metadata writes are non-fatal (try/except wrapped) — never fails
jobs
- Stores auth context (`auth_method`, `user_type`,
`credits_at_creation`) and search query metadata at job creation
- Uses Firestore dot-notation for atomic section updates without
overwriting other workers' data

## Changes

- **Model**: `processing_metadata` field on Job,
`update_processing_metadata` helper on JobManager
- **Workers**: lyrics, audio, screens, render_video, video — each writes
timing + worker-specific metadata
- **AudioShake**: `asset_id` now stored in `TranscriptionData.metadata`,
flows through `corrections.json`
- **API routes**: `extract_request_metadata()` accepts optional
`auth_result` for auth context; audio search stores search query
- **Firestore**: `processing_metadata` added to `SUMMARY_FIELD_PATHS`
- **Tests**: 24 tests covering model, helpers, all worker functions,
error handling, SUMMARY_FIELD_PATHS regression guard

## Test plan

- [x] 24 new unit tests all passing
- [x] All 2512 existing tests pass
- [x] CodeRabbit review — fixed timing overwrite bug (deep dot-notation)
- [ ] Deploy and verify metadata appears on new jobs via Firestore
console

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 01:16:43 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 010784141c fix: resolve 500 error on admin users search (#526)
## Summary
- Fix 500 error when searching users on admin dashboard caused by
Firestore range filter on `email` conflicting with `order_by` on
`created_at` (requires composite index)
- Move sorting to Python when search is active (docs are already fetched
for in-memory pagination anyway)
- Strip whitespace from search input to handle trailing spaces from URL
encoding (`search=dalton+`)

## Test plan
- [x] All 2489 backend tests pass
- [x] CodeRabbit review: 1 finding fixed (type-safe sort key)
- [ ] Verify search works in production after deploy

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 00:38:53 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 64a95d79c6 perf: pre-generate audio editor assets for instant loading (#525)
## Summary
- Audio editor UI was taking 10-20 seconds to load because waveform
generation + FLAC→OGG transcoding happened on-demand
- Now pre-generates both assets in the download worker before
`awaiting_audio_edit`, matching the pattern `screens_worker` uses for
lyrics/instrumental review
- Editor should now load in ~1s (just reads cached JSON + serves signed
URL for pre-transcoded OGG)

## Changes
- `AudioAnalysisService`: new `cache_waveform_data()` and
`load_cached_waveform()` methods
- `audio_download_worker`: pre-transcodes OGG + caches waveform JSON
before state transition (non-fatal)
- `input-audio-info` endpoint: checks for cached waveform first, falls
back to on-demand
- 10 new unit tests covering cache hit/miss, pre-generation, and
fallback behavior

## Test plan
- [x] 2512 backend tests pass (1 pre-existing failure unrelated)
- [ ] Create job with audio edit enabled, verify editor loads instantly

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 18:04:13 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 4824743a99 fix: update E2E tests for new Visibility step (#524)
## Summary
- The guided job creation flow was expanded from 3→4 steps (added
Visibility step between audio selection and Customize & Create)
- E2E tests were not updated, causing the daily happy path test to fail
([run
#95](https://github.com/nomadkaraoke/karaoke-gen/actions/runs/22840888562))
- Updated 3 test files to interact with the new Visibility step before
proceeding to Customize & Create

## Changes
- **happy-path-real-user.spec.ts**: Click "Keep Private" in Visibility
step, remove obsolete `#guided-private` checkbox
- **guided-flow.spec.ts**: All 4 job-creation tests updated for
Visibility step (search, private+display names, YouTube URL, file
upload)
- **demo-recording.spec.ts**: Added Visibility scene, renumbered
subsequent scenes

## Test plan
- [x] All 701 frontend unit tests pass
- [ ] CI E2E smoke tests pass
- [ ] `e2e-happy-path` label triggers production happy path test —
should now pass with the Visibility step interaction

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 17:54:14 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 b2b2469f6f ui: improve audio edit opt-in with clearer visual hierarchy (#523)
## Summary
- Redesigned the audio edit opt-in screen shown after selecting audio in
the job creation flow
- "Use as-is" (default) gets a green Zap icon and "Recommended" badge —
visually prominent so most users click through quickly
- "Edit audio first" gets a purple Scissors icon with a gold lightbulb
hint explaining when it's useful (live recordings with intros, tracks
with sections to remove)
- Left-aligned layout with icons for better scannability

## Test plan
- [x] All 701 frontend tests pass
- [ ] Visual check in production after deploy

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 16:38:17 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 ba265839cf fix: swap upload_json arguments for audio edit endpoints (#522)
## Summary
- Fixed swapped `upload_json(destination_path, data)` arguments in two
audio edit endpoints
- Both `save_audio_edit_session` and `submit_audio_edit` were passing
`(data, path)` instead of `(path, data)`, causing 500 errors

## Test plan
- [x] Verify audio edit session save works (POST
`/api/review/{job_id}/audio-edit-sessions`)
- [x] Verify audio edit submit works (POST
`/api/review/{job_id}/audio-edit/submit`)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 16:27:10 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 6c120b709c feat: audio edits admin dashboard + reset-to-audio-edit button (#521)
## Summary
- New admin page at `/admin/audio-edits` to view and inspect audio
editing sessions
- "Audio Edit" reset button in admin job actions to reset any job back
to audio edit phase
- Backend API endpoints for listing/viewing audio edit reviews

## Changes

### Admin Dashboard (`/admin/audio-edits`)
- List view: jobs with audio edits, searchable by artist/title/email
- Detail dialog with 3 tabs: Edit Stack, Sessions, Audio playback
- Raw data dialog with JSON copy
- Follows existing edit-reviews page pattern

### Reset to Audio Edit
- New "Audio Edit" button in admin job actions (between Audio and
Review)
- **Critical safety**: restores `input_media_gcs_path` to original when
resetting (the submit endpoint overwrites this with edited audio)
- Clears edit stack, redo stack, uploads, and all downstream processing
flags

### Backend
- `GET /api/admin/audio-edit-reviews` - list jobs with audio edits
- `GET /api/admin/audio-edit-reviews/{job_id}` - full detail with
sessions, signed audio URLs

## Test plan
- [x] All 64 backend tests pass (admin reset + audio edit routes)
- [x] All 701 frontend Jest tests pass (38 suites)
- [x] Reset to `awaiting_audio_edit` properly clears state data keys
- [x] `input_media_gcs_path` restored to original on audio edit reset

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 11:59:04 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 483ba9e11b fix: trigger workers in background after audio edit submit (#520)
## Summary
- The audio edit submit endpoint was `await`ing worker triggers (Cloud
Run Job calls) synchronously
- These calls take 30+ seconds, causing the browser `fetch` to time out
- The frontend never received the success response, so the
success/countdown screen never appeared
- Fix: use `asyncio.create_task()` to fire-and-forget the worker
triggers

## Changes
- `backend/api/routes/review.py`: Wrap worker trigger calls in a
background task so the HTTP response returns immediately

## Test plan
- [x] All 25 audio edit route tests pass
- [x] Submit endpoint still returns correct response shape
- [x] Workers still get triggered (just asynchronously)

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 11:09:03 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 7d4a577dcb fix: worker valid statuses for audio_edit_complete + duration display (#519)
## Summary
- Workers (audio_worker, lyrics_worker) were immediately rejecting jobs
in `audio_edit_complete` status because that status wasn't in their
valid status lists
- `formatTimePrecise()` produced garbled output like "-2:-14.5" for
negative durations due to `Math.floor()` on negative numbers

## Changes
- Added `JobStatus.AUDIO_EDIT_COMPLETE` to `audio_worker` and
`lyrics_worker` valid statuses in `job_health_service.py`
- Fixed `formatTimePrecise()` in `AudioEditor.tsx` to use `Math.abs()`
and prepend sign separately

## Test plan
- [x] All 25 audio edit route tests pass
- [x] All 50 job health service tests pass
- [x] Verified negative duration formatting produces correct output

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 23:48:19 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 10ad8b51a7 fix: Align audio edit apply/undo/redo response with frontend (#518)
## Summary
- `_build_audio_edit_response()` returned nested structure
(`edited_audio.*`, `edit_stack_size` integer) but frontend expected flat
fields (`current_audio_url`, `duration_after`, `edit_stack` array)
- This caused "can't access property 'length', t is undefined" when
trying to apply any edit operation (cut, trim, mute)
- Also added `duration_before`/`duration_after` tracking to edit entries

## Test plan
- [x] All 25 audio edit route tests pass
- [x] Full test suite passes (2490 backend + 189 Playwright)
- [ ] Verify edit operations work in production

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 23:13:15 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 7dab28b0ea fix: re-enable Cloud Run Jobs for video encoding to survive deployments (#517)
## Summary

- Re-enable `USE_CLOUD_RUN_JOBS_FOR_VIDEO=true` — this env var was
dropped during the CI migration from cloudbuild.yaml to GitHub Actions,
silently disabling the deployment-safe encoding path
- Add all required env vars and secrets to the `video-encoding-job`
Cloud Run Job in Pulumi (was missing encoding worker credentials,
distribution defaults, Discord, SendGrid, admin tokens)
- Add CI step to update the Cloud Run Job image on each deploy (was
stuck on Dec 2025 image)
- Add unit test verifying video worker routes to Cloud Run Jobs when
flag is enabled

**Incident:** Job `43c0d519` (Westlife - Medley) stuck at `encoding` for
4+ hours on 2026-03-08. A deployment rollout killed the Cloud Run
instance running the encoding orchestrator as a `BackgroundTask`. The
three-layer defense (PRs #413, #473) only protected the GCE encoding
worker, not the Cloud Run orchestrator lifecycle.

**Root cause:** `BackgroundTask` work runs after the HTTP response
returns, so Cloud Run can kill it during deployment drain. Cloud Run
Jobs run to completion and are immune to Service deployments.

See `docs/archive/2026-03-08-fix-encoding-deploy-survival.md` for full
incident analysis.

## Test plan
- [x] Backend tests pass (103 tests)
- [x] New unit test: `test_trigger_video_worker_routes_to_cloud_run_job`
- [x] Job `43c0d519` re-triggered and completed successfully
- [ ] After deploy: verify video worker dispatches Cloud Run Job (check
Cloud Run Jobs console)
- [ ] After deploy: run a test job end-to-end

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 22:57:03 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 2f19921328 fix: Align audio edit API response with frontend interface (#516)
## Summary
- Backend `input-audio-info` endpoint returned nested structure
(`original_audio.playback_url`) but frontend expected flat fields
(`original_audio_url`, `edit_stack`, etc.)
- Caused "can't access property 'length', t.edit_stack is undefined"
error on the audio edit page
- Added defensive `|| []` fallback for `edit_stack` access in frontend

## Test plan
- [x] Backend unit tests updated and passing (25/25)
- [ ] Verify audio edit page loads in production after deploy

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 22:41:39 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 26297fe7c4 feat: Add Edit Input Audio feature (trim, cut, mute, join) (#514)
## Summary

@coderabbitai ignore

Adds a complete "Edit Input Audio" feature that lets users optionally
trim, cut, mute, or join input audio before processing begins. This is
especially useful for live recordings from YouTube where users need to
skip talking/intros.

**5 phases implemented:**
- **Phase 1 (Backend):** 3 new job states, FFmpeg audio operations
service, 6 API endpoints, edit stack undo/redo
- **Phase 2 (Sessions):** Audio edit session persistence with Firestore
+ GCS, SHA256 deduplication
- **Phase 3 (Creation Flow):** Opt-in toggle in job creation,
requires_audio_edit flag across all 3 creation paths
- **Phase 4 (Editor UI):** Full audio editor with waveform
visualization, selection-based editing, playback controls, submit flow
- **Phase 5 (Restore):** Auto-save hook, session restore dialog, manual
save progress

**Key changes:**
- 3 new job states: `awaiting_audio_edit`, `in_audio_edit`,
`audio_edit_complete`
- Server-side FFmpeg operations (lossless FLAC internally, OGG Opus for
browser)
- Collapsible audio edit toggle in VisibilityStep (defaults to off)
- "Edit Audio" action button in JobCard for awaiting_audio_edit status
- Session auto-save every 3 edits + on page hide + before submit
- Session restore dialog when returning to an interrupted edit

## Test plan

- [x] All 706 frontend tests pass (38 suites)
- [x] All 2491 backend tests pass (1 pre-existing failure in unrelated
test)
- [x] 23 AudioEditor component tests (loading, rendering, undo/redo,
submit, restore, save)
- [x] 25 audio edit route tests
- [x] 18 audio edit service tests
- [x] 13 audio edit session tests
- [x] 5 VisibilityStep toggle tests
- [ ] Manual E2E test with real audio file (user to verify)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 20:31:13 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 2131342fa1 Deduplicate catalog track suggestions (#513)
@coderabbitai ignore

## Summary
- Deduplicate song suggestions in the CatalogTrackList by artist + track
name before rendering
- Spotify returns the same song from multiple albums (deluxe, singles,
etc.), causing identical rows

## Test plan
- [x] Visual: "Etherwood - Weightless" no longer shows 5 identical rows
- [x] Logic: `findFuzzyMatches` already deduplicates for the Did You
Mean banner; this adds the same to the suggestion panel

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 12:13:11 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 93823b8a20 Move song suggestions to Step 2 with fuzzy typo correction (#512)
@coderabbitai ignore

## Summary

- **Simplified Step 1 (Song Info)**: Removed autocomplete — plain text
inputs for artist/title so users aren't confused by autocomplete
behavior
- **Step 2 song suggestions**: Catalog lookup + community version check
fire in parallel with audio search. Results appear as a "Song found in
our database" panel (exact match) or correction suggestion panel while
the user waits
- **Fuzzy "Did you mean?" correction**: When audio search returns poor
results (Tier 3) and catalog has no exact title match, fetches the
artist's discography and fuzzy-matches against the user's title. Shows
amber banner with one-click correction + expandable alternatives list.
Title-based fallback with artist tiebreaker handles garbled artist
fields
- **Visibility step reworked**: Action buttons at top for one-click
select+continue, "change later" note, removed Recommended badge
- **Minor UI polish**: Warning icon on community banner, "Recommended
Audio" heading, YouTube icon on community links, filename above release
in pick card

## Test plan

- [x] 70 new tests added (fuzzy matching, SongInfoStep, AudioSourceStep,
VisibilityStep, CommunityVersionBanner)
- [x] All 677 tests pass
- [x] Manual testing via Playwright: "Fox Stevenson / ruises" typo → Did
you mean Bruises; "fox stevenson bruises / bruises" garbled artist →
correct Fox Stevenson suggestion
- [x] CodeRabbit review completed, issues fixed

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 04:40:35 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 ef0bb20094 Fix review session autosave: use edit-count instead of timer (#511)
## Summary

- The 60s interval timer reset on every edit (React dependency
tracking), so active editing could delay saves indefinitely
- Replace with edit-count-based autosave: saves every 3 edits since last
backup
- Uses refs instead of callback dependencies so the save function stays
stable

## Changes

- `use-review-session-autosave.ts`: Replace `setInterval` with
`useEffect` on `historyLength` that checks `editsSinceBackup >= 3`
- Store data/initialData/historyLength in refs so `doSave` callback only
depends on `apiClient`
- Updated tests to match new behavior (11 tests passing)

## Testing
- [x] All 608 frontend tests pass
- [x] New tests for edit-count-based auto-save behavior

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 03:52:43 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 51cd0eadc5 Extend search session TTL to 7 days and handle expiry gracefully (#510)
## Summary
- Extended search session TTL from 30 minutes to 7 days so users who
leave a tab open mid-flow don't hit "Search expired"
- Added graceful frontend handling when sessions do expire — redirects
to re-search with a friendly message instead of raw error
- Updated API docs and tests to match

## Changes
- `backend/api/routes/audio_search.py`: TTL `timedelta(minutes=30)` →
`timedelta(days=7)`
- `frontend/components/job/GuidedJobFlow.tsx`: Catch 404/"Search
expired" errors, clear stale session, redirect to Step 2 with friendly
message preserving song info
- `backend/tests/test_standalone_search.py`: Updated test helper TTL to
match
- `docs/API.md`: Updated TTL reference from "30-min" to "7-day"

## Testing
- [x] All 31 search-specific tests pass
- [x] Full backend suite passes (2212 passed)

## Review
- [x] Local CodeRabbit review completed (no issues with these changes)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 03:23:46 -04:00
Andrew Beveridge ☄️andClaude Opus 4.6 f1ec70924e Add lyrics review session backup/restore system (#509)
## Summary

- Server-side auto-save of lyrics review sessions (every 60s if dirty,
on preview, on page hide) with SHA-256 deduplication
- Split-pane restore UI with session browser, edit diff preview, and
cross-job search
- Auto-prompts on load if saved sessions exist; falls back to
localStorage for local mode

## Changes

### Backend
- **New model**: `ReviewSession` + `ReviewSessionSummary` dataclasses
with Firestore serialization
- **5 new API endpoints** under `/api/review/`: save, list, get, delete
sessions + cross-job search
- **Firestore**: 8 new methods for review session CRUD using
subcollection pattern (`jobs/{jobId}/review_sessions/{sessionId}`)
- **GCS storage**: Correction data stored in GCS (can exceed Firestore's
1MB doc limit); metadata in Firestore
- **Deduplication**: SHA-256 hash comparison skips saving identical data

### Frontend
- **`useReviewSessionAutoSave` hook**: 60s interval auto-save,
visibilitychange save, exposed `saveSession()` for manual/preview
triggers
- **`SessionRestoreDialog`**: Split-pane modal with session list + edit
diff preview, "This Job"/"All Jobs" tabs, cross-job search, duration
mismatch warning
- **`LyricsAnalyzer` integration**: Auto-checks for saved sessions on
load, "Save Progress" and "Session History" toolbar buttons, triggers
save on preview

### Docs
- API.md: 5 new review session endpoints documented
- README.md: Feature added to Current State table + Recent Changes
- LESSONS-LEARNED.md: GCS + Firestore metadata pattern

### Also included (from #508)
- Job observability improvements: timeline metadata, `log_to_job()`
helper

## Testing
- [x] 18 backend unit tests (model serialization, all API endpoints,
deduplication, auth, 404s)
- [x] 10 auto-save hook tests (triggers, deduplication tracking,
interval, read-only, error handling, summary computation)
- [x] 16 SessionRestoreDialog tests (rendering, restore flow, tab
switching, cross-job search, empty states)
- [x] All 607 frontend tests pass
- [x] All 2409 backend tests pass

## Review
- [x] CodeRabbit CLI review completed locally (1 fix applied:
beforeunload → visibilitychange)
- [x] Test review completed (26 new frontend tests added)
- [x] Docs review completed

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 00:23:42 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 9d8bf3a952 Improve job observability with timeline metadata and log_to_job (#508)
@coderabbitai ignore

## Summary
- Add `metadata` field to `TimelineEvent` for structured output details
(brand codes, YouTube video IDs, Dropbox paths, GDrive file IDs)
- Add `log_to_job()` helper for API endpoints to write to Firestore job
log subcollections
- Plumb `timeline_metadata` through `job_manager.transition_to_state()`
and `firestore_service.update_job_status()`
- Capture output details in completion timeline events for full
traceability
- Log edit/delete cleanup operations to both Cloud Logging and Firestore
job logs
- Snapshot previous outputs before cleanup for audit trail
- Enhance distribution logging with video_id, folder path, and file IDs

## Context
After shipping "edit completed tracks," tracing edited jobs was
difficult because timeline events lacked output details, cleanup
operations weren't logged to Firestore, and previous outputs were
overwritten with no history. These changes make the job's Firestore
document self-documenting — any agent or human can read the timeline and
understand the full lifecycle.

## Test plan
- [x] 7 new unit tests for `TimelineEvent.metadata` and `log_to_job()`
helper
- [x] All 103 backend tests pass
- [ ] Verify in production after deploy: edit a test job and confirm
timeline metadata appears in Firestore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 23:08:17 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 e04337fa5d Fix broken notification and CLI review URLs (#507)
## Summary
- Push notifications were sending users to `/review/{id}` and
`/instrumental/{id}` which 404 on Cloudflare Pages static hosting
- Remote CLI was opening browser to path-based URLs missing the `#/`
hash fragment
- Documentation referenced incorrect URL formats

All URLs now correctly use hash-based routing: `/app/jobs#/{id}/review`
and `/app/jobs#/{id}/instrumental`

## Changes
- **backend/services/push_notification_service.py** — Fixed notification
click URLs
- **backend/api/routes/jobs.py** — Fixed edit track response
`review_url`
- **karaoke_gen/utils/remote_cli.py** — Fixed CLI browser open URLs
- **karaoke_gen/utils/cli_args.py** — Fixed help text
- **Tests** — Updated assertions in push notification and remote CLI
tests
- **Docs** — Fixed URL examples in API.md, README.md, TESTING.md
- **pyproject.toml** — Bump version 0.137.0 → 0.137.1

## Test plan
- [x] Backend push notification tests pass (35 tests)
- [x] Remote CLI tests pass (174 tests)
- [x] Full backend test suite passes (103 tests)
- [x] Frontend unit tests pass (581 tests)

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 21:22:28 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 3f568373fd Add edit completed tracks feature (#506)
## Summary

- Add ability to edit completed karaoke tracks: cleans up existing
outputs (YouTube, Dropbox, GDrive, GCS), recycles brand code, resets
state to AWAITING_REVIEW
- Support optional artist/title metadata updates during edit (triggers
screens regeneration)
- New `POST /api/jobs/{job_id}/edit` endpoint with cleanup tracking and
review URL response
- Edit button in OutputLinks for completed jobs with confirmation modal
explaining implications

## Changes

### Backend
- `backend/api/routes/jobs.py` - New edit endpoint with full output
cleanup (YouTube, Dropbox, GDrive, GCS finals), brand code recycling,
state transition, and optional metadata update
- `backend/models/job.py` - Added COMPLETE → AWAITING_REVIEW state
transition and `edit_count` field
- `backend/tests/test_edit_completed_track.py` - 14 unit tests covering
success, metadata updates, validation, and cleanup

### Frontend
- `frontend/components/job/EditTrackModal.tsx` - Confirmation modal with
optional artist/title editing
- `frontend/components/job/OutputLinks.tsx` - Edit button (amber) for
completed jobs with outputs
- `frontend/lib/api.ts` - `editCompletedTrack()` API method and
`EditTrackResponse` type
- `frontend/components/__tests__/OutputLinks.test.tsx` - 3 new tests for
Edit button visibility

### Docs
- `docs/API.md` - Documented new endpoint
- `docs/README.md` - Added recent changes entry
- `docs/archive/2026-03-06-edit-completed-tracks-plan.md` -
Implementation plan

## Test plan

- [x] 14 backend unit tests for edit endpoint (success, metadata,
validation, cleanup)
- [x] 3 frontend tests for Edit button visibility (complete job, deleted
outputs, non-complete)
- [x] All 576 frontend unit tests passing
- [ ] Manual: Edit a completed track, verify outputs are cleaned up and
review reopens
- [ ] Manual: Edit with metadata change, verify screens regenerate

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 02:23:38 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 4d0e3bd1fb Add job creation observability (creation_params + OTel spans) (#505)
## Summary

- Store user's original choices (`is_private`, `is_admin`,
`created_from`) in a `creation_params` dict on the job Firestore
document for permanent observability
- Add OTel span attributes (`job_id`, `job.source`, `job.is_private`) to
all 7 job creation endpoints for Cloud Trace searchability

## Context

Investigation of job `fdcc82a6` revealed that while the system correctly
recorded `is_private: true`, we had no way to definitively prove what
the frontend sent. Cloud Trace request sizes provided forensic evidence,
but this required manual byte-counting. These changes ensure future
investigations can answer "what did the user choose?" directly.

## Changes
- **`backend/models/job.py`** — Added `creation_params` field to Job
model
- **`backend/services/job_manager.py`** — Populate `creation_params` at
job creation
- **`backend/api/routes/jobs.py`** — Span attributes on URL + guided
search endpoints
- **`backend/api/routes/file_upload.py`** — Span attributes on upload,
signed URL, URL, and finalise endpoints
- **`backend/api/routes/audio_search.py`** — Span attributes on audio
search endpoint
- **`backend/api/routes/users.py`** — Span attributes on made-for-you
endpoint
- **`backend/tests/test_private_tracks.py`** — 2 new tests for
creation_params
- **`docs/LESSONS-LEARNED.md`** — Documented Cloud Trace forensic
technique

## Test plan
- [x] All 103 backend tests pass (including 27 private tracks tests)
- [x] 2 new unit tests verify creation_params population and defaults

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 02:01:53 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 5391ac8465 fix: prepare nomad theme style_params for visibility change (#504)
## Summary

Fixes screens worker failure when changing visibility from private to
public: `style_params_gcs_path is required`.

## Root Cause

`change_to_public()` was setting `style_params_gcs_path: None` when
resetting to the nomad theme. The screens worker requires this path to
generate title/end screens.

## Fix

Call `_prepare_theme_for_job("nomad")` to properly copy the nomad
theme's `style_params.json` to the job folder before updating the job
and triggering the screens worker.

## Testing

- [x] 22/22 backend visibility tests pass
- [x] Test now verifies `style_params_gcs_path` is set (not None)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 00:50:10 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 89ade0462c test: add visibility change button tests and lesson learned (#503)
## Summary

Follow-up to #501 and #502 — adds the frontend tests that would have
caught the summary projection bug, plus a lesson learned entry.

## Changes

- 5 new frontend tests for the Change Visibility button (`is_private`
rendering, tenant exclusion, in-progress guard, non-complete exclusion)
- Lesson learned in `docs/LESSONS-LEARNED.md` about always testing the
summary projection layer when adding new fields

## Testing

- [x] 14/14 OutputLinks tests pass (5 new)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 00:33:23 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 7322825a2e fix: add is_private to summary field projection (#502)
## Summary

Hotfix for #501 — the Change Visibility button always showed "Make
Private" because `is_private` was missing from the dashboard's Firestore
field projection.

## Changes

- Add `is_private` to `SUMMARY_FIELD_PATHS` in `firestore_service.py`
- Add `state_data.visibility_change_in_progress` to both the projection
and prune allowlist
- Add regression tests verifying visibility fields are included

## Root Cause

The dashboard uses `fields=summary` which projects only specific fields
from Firestore. `is_private` was not in the projection list, so it was
always `undefined` on the frontend, causing `!!job.is_private` to always
be `false`.

## Testing

- [x] New regression tests pass
- [x] All existing summary endpoint tests pass (10/10)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 00:16:11 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 c2b2aec5fe feat: add Change Visibility button for completed jobs (#501)
## Summary

- Add "Make Public" / "Make Private" button on completed job cards
allowing users to toggle visibility
- Two distinct backend flows: Public→Private (fast redistribution ~1-2
min) and Private→Public (full re-processing ~15-30 min)
- Confirmation modal with direction-specific explanations of what will
happen

## Changes

### Backend
- **New service:** `visibility_change_service.py` — validation, two
change flows, distributed output cleanup (YouTube/Dropbox/GDrive/GCS),
brand code recycling
- **New endpoint:** `POST /api/jobs/{job_id}/change-visibility` with
proper 404/403/400 status codes
- **Video worker:** Added `redistribute_video()` for fast public→private
path (reuses existing finals)
- **Screens worker:** Added render worker trigger after screen regen for
private→public pipeline
- **State machine:** Added `COMPLETE → LYRICS_COMPLETE` transition for
re-processing flow
- **Version:** 0.135.2 → 0.136.0

### Frontend
- **OutputLinks:** "Make Public"/"Make Private" button with Lock/Globe
icons
- **AlertDialog:** Confirmation modal with direction-specific content
explaining consequences
- **API client:** `changeVisibility()` method and response type

### Tests
- 22 backend tests: validation (11), change_to_private (3),
change_to_public (3), output cleanup (4), state transitions (1)
- Frontend OutputLinks test updated for new visibility button

## Testing
- [x] All 22 new backend tests pass
- [x] 2164/2165 backend tests pass (1 pre-existing failure unrelated)
- [x] 573/573 frontend tests pass (30 suites)

## Review
- [x] CodeRabbit CLI review completed locally
- [x] Fixed: Dropbox folder name sanitization mismatch
- [x] Fixed: HTTP status codes (404 for missing job, 403 for ownership)
- [x] Skipped: Screens worker direct DB update (acceptable for simple
restore)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 19:06:51 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 8694099073 docs: Add sequence gap investigation runbook (#500)
## Summary
- Replaced "False Positive Gaps" section (which advised adding to
KNOWN_GAPS) with a full **Sequence Gap Detected** investigation runbook
in `docs/GDRIVE-VALIDATOR.md`
- Added quick reference section in `docs/TROUBLESHOOTING.md`
- Clarified KNOWN_GAPS is historical only (pre-generator era) — new gaps
are always bugs
- Added NOMAD-1295 incident to history

## Context
NOMAD-1295 gap was caused by a job re-trigger (stale GCE cache fix)
allocating a new brand code without recycling the old one. Recycled the
brand code in Firestore so the next public job fills the gap.

## Test plan
- [x] Documentation-only change, no code logic affected
- [x] Comment added to `main.py` KNOWN_GAPS is informational only

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 17:21:00 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 4d93d5372f Fix stale GCE encoding cache causing missing video files (#499)
## Summary
- When admin deletes outputs and re-triggers the video worker, the GCE
encoding worker returns a stale cached result (409/complete) but the
actual files no longer exist in GCS
- This caused YouTube upload to silently skip and the job to complete in
a half-published state (Dropbox has stems but no video, no YouTube
upload)
- Detected on job `f4a552bf` (Queen Anne - Let's Dance)

## Changes
- **Stale cache detection**: `_download_gce_encoded_files` now raises
`RuntimeError` when zero files are downloaded from GCS
- **Auto-retry with new job ID**: `_run_encoding` catches the stale
cache error and re-submits encoding with a suffixed job ID
(`{job_id}_retry_{uuid}`) so the GCE worker treats it as a fresh job
- **Distribution warning**: `_upload_to_youtube` now adds to
`distribution_warnings` when no video file is available (previously
silently returned)
- 3 new tests covering stale cache detection, retry logic, and YouTube
missing-file warning

## Test plan
- [x] All 57 orchestrator tests pass (54 existing + 3 new)
- [ ] Deploy and re-trigger job `f4a552bf` to verify fix in production

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 03:23:03 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 6d7b6494fb fix: scope push notifications by tenant to prevent cross-tenant leakage (#498)
## Summary

- Push subscriptions were stored per-user with no tenant scoping,
causing notifications to leak across portals (e.g., a consumer job
completion appearing as "from vocalstar.nomadkaraoke.com")
- Added `tenant_id` field to push subscriptions and filter by tenant
when sending notifications
- Legacy subscriptions (pre-existing, no tenant_id) continue working for
consumer portal

## Changes

**Backend:**
- `PushSubscription` model: added `tenant_id` field
- `PushNotificationService.send_push()`: filters subscriptions by tenant
- `send_blocking_notification()` / `send_completion_notification()`:
extract tenant from job
- `add_subscription()`: stores tenant_id on new subscriptions
- Push subscribe API: accepts `tenant_id` from request body or tenant
middleware
- `JobManager._send_push_notification()`: passes job's tenant_id

**Frontend:**
- `subscribePush()`: sends `tenant_id` parameter
- `usePushNotifications` hook: gets tenant context from `useTenant()`
and passes when subscribing

## Test plan

- [x] 73 tests passing (11 new tenant isolation tests + updates to
existing)
- [x] `TestTenantIsolation`: consumer↔tenant filtering, legacy
migration, empty string normalization
- [x] `TestPushNotificationTenantIsolation` in tenant_separation.py:
end-to-end isolation tests
- [x] Route tests verify tenant_id passthrough and normalization
- [x] CodeRabbit: clean review, no findings

<!-- @coderabbitai ignore -->
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 16:58:21 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 d41e0c6d90 chore: trigger deploy for #496 (#497)
Empty commit to trigger CI deploy pipeline. The push event from squash
merge of #496 was not fired by GitHub Actions.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 13:16:09 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 e4b2bf51da feat: move audio downloads to Cloud Run Jobs for reliability (#496)
@coderabbitai ignore

## Summary

- Audio downloads via the guided flow (create-from-search, select audio
source) previously used FastAPI BackgroundTasks, which Cloud Run would
silently kill when scaling down "idle" instances. Jobs got permanently
stuck at `downloading_audio` with no error.
- Moves audio downloads to a dedicated Cloud Run Job
(`audio-download-job`), matching the pattern already used for lyrics and
audio separation workers.
- Adds automatic stuck job detection and recovery via Cloud Scheduler.

## Changes

**Core fix (Part 1):**
- New `backend/workers/audio_download_worker.py` — standalone worker
extracted from inline download logic
- New `audio-download-job` Cloud Run Job with VPC access for flacfetch
- New `trigger_audio_download_worker()` in WorkerService
- Replace all 3 BackgroundTasks download call sites + admin retry
endpoint with Cloud Run Job triggers
- Check return value of trigger at all call sites (CodeRabbit finding)

**Defense in depth (Part 2):**
- `downloading_audio_stuck` health check in JobHealthService (>10 min
detection)
- `POST /api/internal/recover-stuck-jobs` endpoint
- Cloud Scheduler job running every 5 minutes for auto-recovery

**State machine fix (Part 3):**
- Allow `FAILED → DOWNLOADING_AUDIO` transition for admin retry

**Infrastructure:**
- New Cloud Run Job: `audio-download-job` (1 CPU, 1Gi, 10min timeout,
VPC connector)
- New Cloud Scheduler: `recover-stuck-downloads` (every 5 min)

## Test plan

- [x] 17 new unit tests for audio_download_worker and stuck detection
- [x] All 120+ backend tests pass (including updated
test_select_overrides.py)
- [x] CodeRabbit review — 1 finding fixed (missing return value check)
- [ ] After deploy: verify with a test job through guided flow
- [ ] After deploy: run `pulumi up` for new infra resources

## Notes

- Fixes stuck jobs 51b8231d and 89e497b1 from 2026-03-05
- The `_download_audio_and_trigger_workers()` function still exists in
audio_search.py (used by the worker) but is no longer called from HTTP
endpoints
- Infrastructure changes require `pulumi up` before the Cloud Run Jobs
will work in prod

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 13:12:04 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 58f1b51244 fix: URL/upload fallback paths now respect private/published visibility (#495)
## Summary

- **Bug:** When creating a job via YouTube URL or file upload (fallback
paths in Step 2), the job was created immediately — skipping the
Visibility step (Step 3) entirely. This meant `is_private` was always
`false` regardless of user intent.
- **Fix:** All three audio source paths (search, URL, upload) now go
through Steps 3-4 before job creation, ensuring the user's
private/published choice is always respected.
- **Tests:** Added 18 unit tests covering all audio source × visibility
combinations, display override handling, mutual exclusivity, and error
cases.

## Root Cause

`AudioSourceStep` had two code paths:
1. **Search result selected** → proceeded to Step 3 (Visibility) → Step
4 (Customize) → job created with correct `is_private`
2. **URL pasted / file uploaded** → job created immediately in Step 2
with `is_private: false` → jumped to success screen

The fix makes path 2 behave like path 1: store the URL/file, proceed
through Steps 3-4, then create the job in `handleConfirm`.

## Test plan

- [x] 37 Jest tests pass (19 existing + 18 new)
- [x] 573 total frontend tests pass
- [x] CodeRabbit review: no findings
- [x] Verified job `5f3f0b88` had `is_private: False` in Firestore
(confirming the bug)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 01:43:17 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 d762055e1e fix: admin payment history + impersonation UX (#494)
## Summary

- **Fix payment history** on admin user detail page showing "0 payments"
— missing Firestore composite index on `stripe_payments(customer_email,
created_at)` caused the query to silently fail. Added index in Pulumi +
backend fallback for immediate fix.
- **Add "Impersonate User" button** to admin user detail page
- **Redesign impersonation banner** — subtle purple strip instead of
bright amber bar that broke header readability
- **Add "Back to Admin" button** in both banner and app header for
one-click return to admin (previously required logout/login)

## Test plan

- [x] Frontend build passes
- [x] 555 frontend tests pass
- [x] 2100 backend tests pass (21 stripe admin service tests pass)
- [ ] Manual: verify payment history loads for user with payments
- [ ] Manual: verify impersonate button works from user detail page
- [ ] Manual: verify impersonation banner is subtle and readable
- [ ] Manual: verify "Back to Admin" returns to admin dashboard

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 01:34:26 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 3a52f09db5 fix: handle string dates in admin edit-reviews endpoint (#493)
@coderabbitai ignore

## Summary
- Fix `AttributeError: 'str' object has no attribute 'isoformat'` in the
edit-reviews endpoint
- Some Firestore jobs store `created_at`/`updated_at` as strings rather
than datetime objects
- Added safe `_to_iso()` helper that handles both types, plus try/except
around each review to skip bad data

## Context
Follow-up to #492 which introduced the edit-reviews endpoint. The
endpoint crashed on first production call due to string dates in legacy
job documents.

## Test plan
- [x] Backend unit tests pass (7/7)
- [x] Verified root cause from Cloud Run logs: `AttributeError: 'str'
object has no attribute 'isoformat'` at admin.py:3130

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 00:58:57 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 83e353438a feat: admin lyrics review dashboard (#492)
@coderabbitai ignore

## Summary
- Add "Edit Reviews" section to admin dashboard for reviewing user
lyrics corrections
- New backend endpoints: `GET /api/admin/edit-reviews` (list) and `GET
/api/admin/edit-reviews/{job_id}` (detail snapshot)
- Three frontend views: read-only lyrics review, edit log replay with
step-through, raw data JSON viewer
- All endpoints are strictly read-only (no state mutations, no
Firestore/GCS writes)
- 7 backend unit tests covering filtering, search, pagination, test
exclusion, error handling

## Motivation
Users are now making lyrics corrections during the review phase, and
we're collecting edit logs. This dashboard lets admins:
1. See what users experienced (original vs corrected lyrics in read-only
LyricsAnalyzer)
2. Replay user edits step-by-step to evaluate quality
3. Review raw human edits data for potential AudioShake data sharing

## Test plan
- [x] 7 backend unit tests pass (list filtering, search, pagination,
test exclusion, 404, full snapshot, missing files)
- [ ] Manual: navigate to /admin/edit-reviews, verify list loads
- [ ] Manual: click View Review, verify read-only LyricsAnalyzer renders
- [ ] Manual: click View Edit Log, verify step-through works
- [ ] Manual: click View Raw Data, verify JSON tabs render

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 00:40:09 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 19314fff6e fix: admin endpoints crash with AuthResult unpacking error (#491)
## Summary

- Five admin endpoints in `users.py` crashed with `ValueError: too many
values to unpack (expected 3)` because `require_admin` now returns
`AuthResult` (4-value `__iter__`) but these endpoints still used 3-tuple
unpacking
- Fixed by using `AuthResult` properties directly
(`auth_data.user_email`) instead of tuple unpacking
- Audit trail improved: now logs actual admin email instead of truncated
token prefix

**Affected endpoints:** `POST /admin/credits`, `POST
/admin/users/{email}/disable`, `POST /admin/users/{email}/enable`,
`DELETE /admin/users/{email}`, `POST /admin/users/{email}/role`

## Test plan

- [x] 10 new unit tests covering all 5 fixed endpoints (success,
failure, unicode regression)
- [x] All 1118 existing tests still pass
- [x] Reproduced 500 error via curl before fix, confirmed fix resolves
it

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 04:53:40 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 ad64406e3b fix: CJK font lookup + allow screen regen for completed jobs (#488)
## Summary

Follow-up to #482 (CJK character support). Three fixes:

1. **CJK font lookup was returning wrong font** — `fc-match :lang=zh`
returned `NotoSans-Regular.ttf` (no CJK glyphs, renders as boxes). Now
explicitly requests "Noto Sans CJK" family and validates the result
actually contains "CJK" in the path.

2. **Screen regeneration fails for completed jobs** — Admin "Regenerate
Screens" failed with `Invalid state transition: complete →
generating_screens`. Fix: temporarily set status to `lyrics_complete`,
store original status in `regen_restore_status`, and restore after
generation.

3. **Version bump** — 0.133.1 → 0.133.3

## Test plan

- [x] 22 unit tests for CJK font detection and fallback
- [x] 59 worker tests passing
- [ ] Regenerate screens for job f8a36b60 after deploy to verify CJK
rendering

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 23:24:22 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 3f6c42d21a Validate YouTube URL availability before job creation (#490)
## Summary

- Adds early YouTube URL validation via flacfetch's `/check-youtube`
endpoint before creating jobs
- Detects geo-restricted, private, removed, and age-restricted videos
with clear error messages
- Graceful degradation: if the check fails or flacfetch is unavailable,
job creation proceeds normally

Fixes the issue where UK users submitting geo-restricted YouTube URLs
would get failed jobs instead of immediate feedback.

## Changes

- `FlacfetchClient.check_youtube()` - new method calling flacfetch's
`/check-youtube` endpoint
- `YouTubeDownloadService.check_availability()` - service layer with
graceful degradation
- `create_job_from_url()` route - validates YouTube URLs before job
creation, returns 400 with specific error messages
- 9 new unit tests covering happy path, error cases, and graceful
degradation
- Fixed existing regression test to mock the new async method

## Test plan

- [x] 2299 backend tests pass (36 skipped)
- [x] New unit tests for FlacfetchClient.check_youtube() (4 tests)
- [x] New unit tests for YouTubeDownloadService.check_availability() (5
tests)
- [x] Existing regression test updated and passing
- [x] CodeRabbit review: no findings
- [ ] Production: submit a geo-restricted YouTube URL and verify 400
error returned

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 23:16:35 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 d7371520bd chore: add tenant_id Firestore indexes to Pulumi IaC (#489)
## Summary
- Import two Firestore composite indexes into Pulumi to prevent IaC
drift
- Indexes were created manually during tenant separation fix (#484)

## Indexes added
- `jobs: tenant_id + created_at` — tenant portal job lists
- `jobs: user_email + tenant_id + created_at` — tenant user job lists

## Verification
- `pulumi up` ran successfully: 2 imported, 0 created, 135 unchanged

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 23:06:20 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 598e00eb2e Skip instrumental review and audio separation for tenant jobs (#487)
## Summary

- **Bug fix**: GCE encoding worker couldn't find uploaded instrumental
audio (stored under `uploads/` but worker only downloaded from `jobs/`).
Now downloads it separately before encoding.
- **Optimization**: Skip GPU-powered audio separation for jobs with
existing instrumental (saves 5-10 min of Modal processing time per
tenant job).
- **UX improvement**: Skip instrumental review screen when job has
uploaded instrumental — after lyrics review, auto-submits with uploaded
instrumental and shows success screen.

## Changes

| File | What |
|------|------|
| `backend/services/gce_encoding/main.py` | Download existing
instrumental from correct GCS path; search with right patterns |
| `backend/services/job_manager.py` | Skip audio worker when
`existing_instrumental_gcs_path` is set |
| `frontend/components/lyrics-review/LyricsAnalyzer.tsx` | Auto-complete
review with "custom" selection when instrumental exists |
| `frontend/app/app/jobs/[[...slug]]/client.tsx` | Pass
`hasExistingInstrumental` prop |
| `frontend/lib/api.ts` | Add `existing_instrumental_gcs_path` to Job
type |
| `backend/tests/test_gce_encoding_worker.py` | 4 new tests for existing
instrumental file patterns |
| `backend/tests/test_job_manager.py` | 2 new tests for audio worker
skip logic |

## Test plan

- [x] 6 new unit tests added (4 file-finding, 2 worker-skip)
- [x] All 72 backend tests pass
- [ ] Submit a tenant job in production to verify the full flow

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 23:04:52 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 7f64ec7708 fix: persist feedback banner dismiss across reloads (#486)
@coderabbitai ignore

## Summary
- The "Share your feedback on the karaoke creation experience" banner
reappeared after every page reload because dismiss state was only stored
in React state
- Now persists dismiss state to `localStorage` (key:
`nomad-feedback-banner-dismissed`), matching the pattern used by other
persistent UI preferences in the same file

## Test plan
- [x] Dismiss banner, reload page — banner stays dismissed
- [x] Clear localStorage, reload — banner reappears (for eligible users)
- [x] Existing tests pass (177 passed, 10 pre-existing failures
unrelated to this change)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 22:46:02 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 d09f16a2f7 fix: tenant job separation — jobs not showing in tenant portal (#484)
## Summary
- Fix tenant jobs not appearing in tenant portal (e.g. Vocal Star)
Recent Jobs list
- Fix tenant/consumer job separation so tenant jobs never leak into
consumer portal
- Load `tenant_config` when auth token overrides tenant context (was
only setting `tenant_id`)

## Details

**Bug 1 — tenant_config not loaded from token auth:**
When a tenant-scoped auth token authenticates against
`api.nomadkaraoke.com`, `require_auth` updated `request.state.tenant_id`
but not `request.state.tenant_config`. Job creation relied on
`tenant_config` → jobs created with `tenant_id=None`. Job listing used
`tenant_id` → queried `WHERE tenant_id='vocalstar'` → missed the job.

**Bug 2 — no tenant/consumer separation:**
All jobs now have `tenant_id` explicitly set: `""` for consumer portal,
`"vocalstar"`/etc for tenant portals. Firestore queries always filter by
`tenant_id`, ensuring full separation. Admin endpoints pass
`tenant_id=None` to bypass the filter and see all jobs.

**Data migration:** Backfilled `tenant_id=""` on all 140 existing
consumer jobs in Firestore.

## Test plan
- [x] All 2276 backend tests pass (0 failures, 139 skipped)
- [x] New test: `test_auth_result_with_tenant_id_loads_tenant_config`
- [x] Verified tenant jobs appear in Vocal Star portal
- [x] Verified consumer portal filters to `tenant_id=""`
- [ ] Manual: confirm Vocal Star portal shows jobs after deploy
- [ ] Manual: confirm admin dashboard still shows all jobs

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 22:41:46 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 d2d5de1dee Fix contrast ratio for tenant portal themed elements (#483)
@coderabbitai ignore

## Summary
- Tenant portals (Vocalstar yellow `#ffff00`, Singa green `#17E87A`) had
unreadable text on primary-colored buttons and UI elements because
`--primary-foreground` was hardcoded to white
- Added WCAG luminance-based contrast computation to dynamically choose
black or white foreground text
- Applied fix in client-side branding, Cloudflare edge function, and all
tenant-specific components
- Fixed light mode to respect tenant colors instead of falling back to
brand pink

## Test plan
- [x] 6 new unit tests for `getContrastColor()` covering yellow, green,
black, blue, pink, white inputs
- [x] 2 existing branding tests updated to verify `--primary-foreground`
is set correctly
- [x] All 555 frontend tests pass
- [ ] Visual verification on vocalstar.nomadkaraoke.com and
singa.nomadkaraoke.com after deploy

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 22:22:00 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 ac12f2cb19 fix: CJK character support for title cards and downloads (#482)
## Summary

- **Title card renders question marks for CJK characters** → Added font
fallback to Noto Sans CJK (auto-detected via fontconfig) when text
contains Chinese/Japanese/Korean characters. Per-element font selection
so mixed CJK/ASCII titles work correctly.
- **Download links fail with latin-1 encoding error** → Use RFC 5987
`filename*=UTF-8''` encoding in Content-Disposition header for
non-latin-1 filenames. ASCII fallback for older clients.

Fixes job `f8a36b60`.

## Test plan

- [x] 22 unit tests for CJK font detection and fallback logic
- [x] 9 unit tests for Content-Disposition header construction (ASCII,
latin-1, CJK, Cyrillic, RFC 5987 roundtrip)
- [x] 4 integration tests for `_render_all_text` with per-element font
selection
- [x] 1 new test in existing unicode sanitization suite for CJK RFC 5987
encoding
- [x] All 125 related tests passing
- [ ] Deploy and re-test job f8a36b60 download links in production
- [ ] Regenerate title/end screens for job f8a36b60 to verify CJK font
rendering

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 22:14:22 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 ca9f3bd5ba feat: streamlined tenant portal with 4-field upload form (#481)
@coderabbitai ignore

## Summary

- Replace the multi-step consumer wizard with a streamlined 4-field form
(Artist, Title, Mixed Audio, Instrumental Audio) for B2B tenant portals
- All tenant jobs automatically private with tenant-specific
Dropbox/GDrive delivery, locked theme, and brand prefix
- Backend: `_apply_tenant_overrides()` helper forces private, applies
locked theme, overlays tenant distribution settings; private tenant jobs
use their own paths instead of NonPublished override
- Frontend: New `TenantJobFlow` component with drag-and-drop file
uploads; conditional rendering based on tenant detection

## Test plan

- [x] 10 backend unit tests (tenant override logic, distribution
routing, partial config edge cases)
- [x] 9 frontend unit tests (API contract, error handling, form
validation)
- [x] 3 E2E production tests (simplified form verification, track
submission, consumer portal isolation)
- [x] All existing tests pass (2307 backend + 520 frontend Jest + 181
Playwright regression)
- [ ] Run production E2E test after deploy: `KARAOKE_ADMIN_TOKEN=xxx npx
playwright test e2e/production/tenant-job-submission.spec.ts
--config=playwright.production.config.ts`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 21:04:36 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 f017ba7d77 feat: add Visibility step and rework Customize & Create flow (#480)
## Summary

- Add a dedicated **Visibility step** (Step 3) to the guided job
creation flow so users clearly understand public vs private delivery
before creating their video
- Rework the **Customize & Create step** (now Step 4) with conditional
UI based on visibility choice, side-by-side preview canvases, and new
style controls
- Document the **public video style consistency rule** in the brand
guide

## Changes

### New Visibility Step
- Two large selectable cards: Published (recommended) and Private
- Full explanations shown upfront — YouTube channel link, venue
distribution, Dropbox delivery, monetization caveats
- Encouraging copy for Published option

### Reworked Customize Step
- **Public mode**: Simple layout with only display artist/title text
overrides (brand consistency enforced)
- **Private mode**: Full customization with side-by-side preview
canvases
- Background mode toggle per canvas: Default / Image upload / Solid
color
- Color presets including Green Screen (#00b140)
- New lyrics color controls: highlight (sung) and text (unsung) colors
- Image upload thumbnails removed (canvas previews serve this purpose)

### Improved Karaoke Preview
- 4 centered lines matching real video output layout
- Configurable sung/unsung colors with live preview
- Solid color background support

### Documentation
- Added "Public Video Style Consistency Rule" to BRAND-STYLE-GUIDE.md
- Requirements doc in docs/archive/

## Test plan
- [x] All 540 Jest unit tests pass
- [x] CustomizeStep tests rewritten for new layout (14 tests covering
both public and private modes)
- [ ] Visual testing via dev server
- [ ] E2E flow test for both public and private paths

@coderabbitai ignore

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 18:47:08 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 a286559cfd fix: support hash-keyed user docs in all user lookups (#479)
## Summary
- Added `get_user_doc_ref()` helper to `UserService` that finds user
documents regardless of key type (email-keyed or hash-keyed)
- Fixed `get_user()`, `update_user()`, `increment_jobs_completed()` to
fall back to querying by email field when direct doc ID lookup fails
- Fixed webhook `total_spent` update which was using wrong collection
name ("users" instead of "gen_users") and direct doc ID lookup
- Fixed backfill endpoint to use `USERS_COLLECTION` constant instead of
hardcoded "users"

### Context
Firestore has two populations of user docs: 42 email-keyed (doc ID =
email, from current `_save_user`) and 36 hash-keyed (doc ID = SHA256
hash, from older auth). All runtime code used `document(email.lower())`
which silently failed for hash-keyed users, meaning their
`total_jobs_completed` and `total_spent` were never updated.

## Test plan
- [x] All 187 tests pass (backend + frontend)
- [ ] After deploy: run backfill endpoint to recalculate stats for
hash-keyed users
- [ ] Verify admin dashboard shows correct stats for users like
`admin@nomadkaraoke.com`

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 16:39:33 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 0281df345e fix: backfill email case handling (#478)
## Summary
- Backfill endpoint now tries both original and lowercase email variants
when counting completed jobs per user

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 15:48:24 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 f6903911d1 feat: admin dashboard UX improvements (#476)
## Summary
- **Clickable links**: Job detail user email navigates to user detail;
source field shows clickable YouTube URLs / artist-title / filename;
fixed broken job links on user detail page
- **New columns**: Users list now shows Created, Last Login, Spent
columns with relative time display and sortable headers
- **Data fixes**: `total_jobs_completed` counter now incremented when
jobs complete; `total_spent` cached on user documents; payment email
normalization on storage
- **Tooltips & cursors**: Audit across all admin pages; private mode
checkbox warns about output deletion
- **E2E tests**: 15 new tests (8 regression for users page, 4 for job
detail, 3 production tests)

## Post-deploy steps
After merging, run these one-time backfill endpoints:
1. `POST /api/admin/payments/normalize-emails` - Lowercase existing
payment emails
2. `POST /api/admin/backfill/user-stats` - Recalculate
total_jobs_completed and total_spent for all users

## Test plan
- [x] All 189 regression tests pass
- [ ] Verify admin dashboard loads in production
- [ ] Verify clickable email on job detail navigates correctly
- [ ] Verify new columns visible on users list
- [ ] Run backfill endpoints post-deploy
- [ ] Run production E2E tests

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 15:16:28 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 04950345bc Remove legacy per-user job rate limiting (#475)
@coderabbitai ignore

## Summary
- Remove the "5 jobs per day" per-user rate limiting system, which is no
longer needed since credits now control usage
- Delete `rate_limit_service.py` and all associated tests
- Remove `RateLimitExceededError` exception and all references across
backend routes
- Remove rate limit config fields (`enable_rate_limiting`,
`rate_limit_jobs_per_day`)
- Clean up frontend admin page: remove Overview tab (rate limit stats)
and User Overrides tab
- Keep all blocklist management (domain/email/IP blocking) and YouTube
queue features intact
- Update API and README docs to reflect the change

## Changes
- **3 files deleted**: `rate_limit_service.py`,
`test_rate_limit_service.py`, `test_rate_limits_api.py`
- **13 files edited** across backend and frontend
- **Net**: ~56 insertions, ~2,465 deletions

## Test plan
- [x] All backend tests pass (1086 core + unit tests)
- [x] All frontend tests pass (176 tests)
- [x] Verified no remaining references to `RateLimitExceededError` or
`rate_limit_service`
- [x] Blocklist endpoints preserved and tested
- [x] YouTube queue endpoints preserved and tested
- [x] Credit enforcement tests updated (removed rate limit fixtures)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 14:41:18 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 9f85ae4221 feat: credit notifications and refund on job deletion (#474)
<!-- @coderabbitai ignore -->

## Summary

- **Backend**: Refund credits when jobs are deleted (if not already
complete/failed/cancelled)
- **Frontend**: Show toast notifications for credit changes — "Credit
used" on job creation, "Credit refunded" on eligible job deletion
- Refreshes user data from backend after credit-changing operations so
header counter stays accurate

## Changes

| File | What |
|------|------|
| `backend/services/job_manager.py` | Refund credit on job deletion
(non-complete jobs) |
| `frontend/components/job/JobSubmission.tsx` | "Credit used — X
remaining" toast after job creation |
| `frontend/components/job/JobActions.tsx` | "Credit refunded — X
remaining" toast after eligible deletion |
| `frontend/components/__tests__/JobActions.test.tsx` | Updated test
mock for auth module |

## Test plan

- [x] All 178 Playwright + Jest tests pass
- [ ] Create a job → see "Credit used" toast with correct remaining
count
- [ ] Delete a pending/processing job → see "Credit refunded" toast
- [ ] Delete a completed job → see "Job deleted" toast (no refund
mention)
- [ ] Header credit counter updates immediately after both operations

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 14:18:10 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 7eecbe8382 fix: make encoding pipeline resilient to deployment restarts (#473)
## Summary

- Encoding jobs no longer fail when CI deploys restart the GCE encoding
worker mid-job
- Three-layer defense: CI graceful drain + extended client retries +
poll failure tolerance
- Root cause: Job `25173cb3` failed 2026-03-04 because the retry window
(~14s) was too short for a worker restart (~30-90s)

## Changes

### CI Workflow (`.github/workflows/ci.yml`)
- New "Drain active encoding jobs before restart" step polls `/health`
for `active_jobs`
- Waits up to 10 minutes for active jobs to finish before restarting the
worker
- Falls through gracefully if the worker is already down or drain
timeout expires

### Encoding Service (`backend/services/encoding_service.py`)
- **Extended retries:** MAX_RETRIES 3→7, INITIAL_BACKOFF 2s→5s,
MAX_BACKOFF 10s→15s (total window ~90s)
- **Poll failure tolerance:** `wait_for_completion()` now tolerates up
to 5 consecutive poll failures instead of failing immediately on a
single network error
- Follows same pattern as flacfetch status polling (PR #446)

### Tests (`backend/tests/test_encoding_service.py`)
- 8 new tests: retry config validation, poll tolerance, counter reset,
error propagation
- Config guard test mathematically verifies retry window exceeds 60s
minimum

### Documentation
- `docs/LESSONS-LEARNED.md`: New section documenting the three-layer
defense pattern
- `docs/TROUBLESHOOTING.md`: Added prevention note to "Job stuck at
encoding" runbook

## Testing
- [x] All 19 encoding service tests pass
- [x] Full test suite passes (178 passed, 7 skipped)
- [x] CodeRabbit CLI review: no findings

## Review
- [x] Local CodeRabbit review completed
- [x] Documentation updated

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 14:04:13 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 47ecf49712 fix: audio search CORS errors and add support contact info (#472)
## Summary
- Fix CORS config: disable `allow_credentials` (unused by frontend,
caused spec violation with wildcard origin returning
`Access-Control-Allow-Origin: *` + `Access-Control-Allow-Credentials:
true`)
- Add auto-retry (1 attempt) on network errors during audio search — the
spinner stays visible during retry
- Add 45s fetch timeout via AbortController to prevent indefinite hangs
- Show actionable error message with Retry button and support email link
- Add help/support dropdown in dashboard and app headers with email
(andrew@nomadkaraoke.com) and phone (+1 803-636-3267)

## Root Cause
Audio search requests take 17-23s on cold instances. During this time,
network connections can drop (proxy timeouts, unstable connections),
causing browsers to report a CORS error with null status code. The
frontend caught this as a generic TypeError and showed an unhelpful
"Failed to search for audio" message with no way to retry.

## Test plan
- [x] All 178 tests pass (backend + frontend unit + Playwright E2E)
- [x] TypeScript compilation clean
- [ ] Manual: trigger audio search, verify retry works on network
failure
- [ ] Manual: verify help dropdown appears in dashboard header
- [ ] Manual: verify help dropdown appears in app-header (logged-in
pages)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 12:46:16 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 33667a6852 feat: custom video styling for private delivery jobs (#471)
@coderabbitai ignore

## Summary
- Private-delivery users can now customize their karaoke video styling:
upload custom background images (karaoke + title card) and change
artist/title text colors via color pickers with live canvas previews
- Backend: 2 new endpoints (`style-upload-urls`,
`style-uploads-complete`) that validate ownership, cutoff status, and
merge custom assets into the theme's `style_params.json`
- Frontend: `ImageUploadField` component, `KaraokeBackgroundPreview`
canvas, extended `TitleCardPreview` with custom bg/colors,
`CustomizeStep` styling section (visible when private), `GuidedJobFlow`
upload orchestration
- Also: skip `take-screenshots` spec by default (manual helper, not a
real test)

## Test plan
- [x] 15 backend tests covering both new endpoints (auth, validation,
cutoff, asset merging, color overrides)
- [x] 14 CustomizeStep tests (conditional visibility, color pickers,
previews, form interactions)
- [x] 9 ImageUploadField tests (drag-and-drop, validation, file
type/size rejection)
- [x] 2 TitleCardPreview tests (custom colors, custom background URL)
- [x] 6 guided-flow API contract tests (style upload methods)
- [x] All 181 existing Playwright E2E tests pass
- [ ] Manual: create a private job, upload custom backgrounds, verify
colors render in video

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 11:44:23 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 de9053eda6 fix(e2e): scope auth email input to dialog element (#470)
## Summary
- Fix Playwright strict mode violation in E2E happy path test where
`getByPlaceholder('you@example.com')` matched 3 elements on the homepage
(AuthDialog, Pricing section, Made For You section)
- Scope the email input locator to the `role="dialog"` element so it
only targets the AuthDialog's input
- Applied same fix to `full-user-journey.spec.ts` which had the same
issue

## Test plan
- [x] CI runs the happy path test against production (via
`e2e-happy-path` label)
- [x] Verify the strict mode violation is resolved

Fixes:
https://github.com/nomadkaraoke/karaoke-gen/actions/runs/22657418752

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 11:23:10 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 0184237242 feat: simplify YouTube quota — GCP Cloud Monitoring as single source of truth (#467)
## Summary

- Replace dual Firestore self-tracking + GCP monitoring with **GCP Cloud
Monitoring as the single source of truth** for YouTube API quota usage
- Add lightweight Firestore **pending buffer** for the ~7-minute delay
before GCP reflects new uploads
- Simplify callers from 3x `record_operation()` to 1x `record_upload()`
per upload
- Add `youtube_quota_gcp_usage` and `youtube_quota_pending_units` to
admin API response
- Show "GCP: X + Pending: X" breakdown on admin dashboard below progress
bar

## Test plan

- [x] All backend tests pass (quota service, API, queue processor)
- [x] All frontend tests pass (19 rate-limits page tests including new
GCP+Pending breakdown)
- [x] Full test suite passes (`make test` — 182 frontend + backend)
- [ ] After deploy: verify `GET /api/admin/rate-limits/stats` returns
`youtube_quota_gcp_usage` matching GCP Console
- [ ] After deploy: verify dashboard progress bar shows real GCP number

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 05:31:26 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 172816b935 fix: grant monitoring.viewer role for GCP quota tracking (#463)
## Summary
- Adds `roles/monitoring.viewer` IAM binding to the backend service
account
- Required for the YouTube quota GCP Cloud Monitoring integration
deployed in PR #460
- Without this role, the quota monitoring query fails silently and
returns `gcp_quota_available: false`

## Test plan
- [x] Verified locally that the Cloud Monitoring query works and returns
real YouTube API quota data (4,493 units today)
- [ ] After deploy, verify `GET /api/admin/rate-limits/stats` returns
`gcp_quota_available: true` with actual quota data

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 05:09:47 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 b4be5968c2 fix: payment dashboard date handling, chart rendering, and sidebar padding (#466)
@coderabbitai ignore

## Summary
- **Revenue chart showed nothing**: `store_payment()` used
`datetime.utcnow()` for `created_at` instead of the Stripe session's
actual `created` timestamp. All backfilled payments appeared on a single
date. Fixed to use `session["created"]`.
- **Single data point invisible**: Recharts AreaChart doesn't render a
visible area with only one point. Added `dot={chart.length === 1}` so
single points show as dots.
- **Payment detail sidebar had no padding**: `SheetContent` had no
padding class. Added `p-6`.
- **Backfill date fix script**: Added `scripts/fix_backfill_dates.py`
(already run against production to correct 32 records).

## Test plan
- [x] All 32 payment tests pass locally
- [x] Revenue chart verified visually via dev server — shows proper date
distribution (Dec 2025 - Mar 2026)
- [x] Payment detail sidebar verified visually — proper padding on all
sides
- [x] Production API confirmed returning correct date-bucketed data
after fix script

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 04:48:16 +00:00
Andrew Beveridge ☄️ 0d4b50b9e6 fix: switch runners to on-demand, remove Pulumi from CI
- Switch runners from spot to on-demand (no more preemption)
- Remove Pulumi deploy from CI (apply locally only — CI runs on the infra Pulumi manages)
- Force apt to use IPv4 (Cloud NAT doesn't support IPv6)
2026-03-03 23:33:34 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 481e0254eb fix: make runner startup script idempotent and update runner version (#468)
## Summary

- Makes the GitHub Actions runner startup script idempotent so it
survives spot VM preemption/restart
- Updates runner binary from 2.321.0 to 2.332.0 (GitHub now enforces
minimum 2.329.0)
- Adds lessons learned from the outage

## Root Cause

All 3 spot runner VMs were preempted and restarted simultaneously. The
startup script had `set -e` and used non-idempotent commands (e.g., `gpg
--dearmor -o file` fails when file already exists from previous run).
The script died at line 26, never reaching the runner registration
section. Result: zero runners registered, all self-hosted CI jobs queued
indefinitely.

## Changes

| Change | Detail |
|--------|--------|
| Remove `set -e` | Use explicit checks instead of fail-fast |
| Add idempotency guards | Skip Docker, Python, Node, Java, Poetry,
gcloud if already installed |
| `install_gpg_key()` helper | Checks if GPG key file exists before
dearmoring |
| Runner version upgrade | 2.321.0 → 2.332.0 (with version check logic
for future upgrades) |
| Runner re-registration | Always runs — this is the critical section
that must succeed on restart |
| NodeSource fallback | Retry with delay, fallback to apt if NodeSource
is unreachable |
| Tool cache check | Skip Python tool cache copy if already populated |
| Docstring fix | Updated example version in `__init__.py` |
| Lessons learned | Added incident write-up to `docs/LESSONS-LEARNED.md`
|

## Test plan

- [x] Manually re-registered all 3 runners (v2.332.0) — all online and
busy
- [x] Installed missing Poetry and Python tool cache on all runners
- [x] Re-triggered failed CI runs — picking up jobs successfully
- [ ] After merge: next spot preemption should auto-recover

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 22:44:02 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 03df57795c docs: local-first Pulumi deploy guidance (#465)
## Summary
- Document the lesson learned from PR #464's Pulumi deploy being
preempted mid-apply on a spot VM
- **Always run `pulumi up` locally before merging infrastructure PRs** —
CI re-runs as a no-op confirmation
- Add recovery steps for interrupted Pulumi deploys (`cancel` →
`refresh` → `import` → `up`)

## Changes
- **CLAUDE.md**: Updated infrastructure rules to require local apply
first
- **docs/DEVELOPMENT.md**: Replaced "DO NOT run pulumi up locally" with
local-first workflow + recovery steps
- **docs/LESSONS-LEARNED.md**: Added two new lessons (local-first
deploy, spot VM preemption)

## Test plan
- [ ] Docs-only change, no code affected
- [ ] Verify guidance is clear for future agents/developers

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 22:01:21 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 a9f935cc82 feat: add dedicated on-demand build runner for Docker deploys (#464)
## Summary

- Adds a 4th GitHub Actions runner VM (`github-build-runner`) dedicated
to Docker deploy builds
- Uses **on-demand scheduling** (not spot/preemptible) to prevent
mid-build preemption that blocked PR #460's deployment
- Larger machine type (`e2-standard-8`, 8 vCPU/32GB) for faster Docker
builds
- `deploy-backend` CI job now targets the `docker-build` label so only
the build runner picks it up
- General runners (3x spot) remain unchanged for PR test jobs

## Changes

| File | Change |
|------|--------|
| `infrastructure/config.py` | Added `GITHUB_BUILD_RUNNER` machine type
and runner label constants |
| `infrastructure/compute/github_runners.py` | New
`create_build_runner()` function (on-demand VM) |
| `infrastructure/compute/startup_scripts/github_runner.sh` | Dynamic
labels from instance metadata |
| `infrastructure/functions/runner_manager/main.py` | `RUNNER_NAMES` env
var replaces hardcoded prefix/count |
| `infrastructure/modules/runner_manager.py` | Passes runner names to
Cloud Function env vars |
| `infrastructure/__main__.py` | Wires up build runner creation and
runner name passing |
| `.github/workflows/ci.yml` | `deploy-backend` targets `[self-hosted,
linux, gcp, docker-build]` |
| `infrastructure/functions/runner_manager/test_runner_manager.py` | 10
new tests for runner manager |
| `docs/GCP-COST-OPTIMIZATION.md` | Changelog entry |

## Cost Impact

- Build runner: ~$0.27/hr (on-demand `e2-standard-8`), only running
during deploys
- General runners unchanged at ~$0.07/hr (spot `e2-standard-4`)

## Test plan

- [x] 10 unit tests for runner manager (RUNNER_NAMES parsing, instance
iteration, start/stop)
- [x] All 38 infrastructure tests pass
- [ ] `pulumi preview` shows 1 new VM + updated Cloud Function env vars
- [ ] After merge: verify `github-build-runner` VM exists in GCE console
- [ ] Push change to trigger `deploy-backend` — verify it runs on build
runner
- [ ] PR test jobs continue running on `github-runner-{1,2,3}` only

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 21:43:46 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 6a3040dc6f fix: reduce CI runner contention and speed up Docker builds (#462)
## Summary

- Eliminates duplicate workflow runs on PR branches (10 → 4 self-hosted
jobs per push)
- Speeds up backend CI jobs with Poetry caching and moves lint off
self-hosted runners
- Optimizes Docker base image build with better layer ordering and pip
cache mounts
- Adds deploy hardening (build timeout, tiered disk cleanup)

## Changes

### CI Workflow (.github/workflows/ci.yml)
- **Push trigger branch filter**: `push:` → `push: branches: [main]` —
stops duplicate runs (push + pull_request) on PR branches
- **Poetry cache**: Added `actions/cache@v4` for `.venv` to
`backend-unit-tests` and `backend-emulator-tests`
- **backend-lint → ubuntu-latest**: Removed Poetry/deps install, just
runs `py_compile` (~10s vs ~3min)
- **propagate-base-image matrix**: 20 → 3 jobs to match actual runner
count
- **Base image build timeout**: Added `timeout-minutes: 20` for clear
failure signal
- **Tiered disk cleanup**: deploy-backend now cleans images at 50%,
build cache at 70% (was flat 70%)

### Docker Base Image (backend/Dockerfile.base)
- **Layer reordering**: Copy `pyproject.toml` first → create package
stubs → `pip install` → THEN copy source code. This keeps the expensive
pip install layer cached in the registry even when source code changes
between base image rebuilds.
- **BuildKit pip cache mount**:
`--mount=type=cache,target=/root/.cache/pip` preserves downloaded wheels
across builds, so partial dependency updates only download changed
packages.

## Impact

| Metric | Before | After |
|--------|--------|-------|
| Workflow runs per PR push | 2 | 1 |
| Self-hosted jobs per PR run | 5 | 4 |
| Total self-hosted jobs per PR push | 10 | 4 |
| backend-lint duration | ~3 min | ~10 sec |
| Poetry cache on backend jobs | 1/3 | 3/3 |
| Base image rebuild (cached deps) | ~10 min | ~3-5 min |

## Context

During shipping of PRs #460 and #461, CI experienced repeated failures:
backend jobs getting cancelled during `poetry install`, 3-10+ minute
queue times, and stuck jobs. PR #460's merge-to-main deploy also had its
base image build cancelled after 10 minutes (likely spot VM preemption).
See `GHA-RUNNER-ISSUES.md` for the full investigation.

## Testing

- [ ] Push to PR branch triggers only ONE workflow run (not two)
- [ ] `backend-lint` runs on `ubuntu-latest` and completes in < 30s
- [ ] `backend-unit-tests` and `backend-emulator-tests` show cache
restore in logs
- [ ] All jobs pass and `ci-gate` succeeds
- [ ] Base image build (when triggered) uses optimized layer caching

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 02:19:56 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 937f8f7ec1 feat: YouTube quota GCP monitoring + legacy beta cleanup (#460)
## Summary
- Remove the deprecated beta enrollment system (endpoints, models, email
templates, admin UI, rate limits) — ~20 files cleaned
- Remove legacy count-based YouTube upload tracking from
rate_limit_service and dead blocklist methods
- Add GCP Cloud Monitoring integration to cross-reference YouTube API
quota usage with Firestore tracking
- Add drift detection (alerts when GCP reported vs Firestore tracked
differs >10%)
- Display GCP quota data in admin rate-limits dashboard

## Changes

### Backend (15 files)
- **Removed:** 5 beta endpoints, `BetaTesterStatus` enum, beta User
fields, beta email templates, `validate_email_for_beta()`, deprecated
YouTube/beta rate limit methods, dead blocklist CRUD methods
- **Added:** `get_gcp_quota_usage()` to `YouTubeQuotaService` with Cloud
Monitoring API query, 5-min cache, and drift detection
- **Updated:** Rate limits API response model with GCP quota fields
(`gcp_quota_available`, `gcp_quota_units_consumed`, `quota_drift`,
`quota_drift_alert`)

### Frontend (11 files)
- **Deleted:** `frontend/app/admin/beta/page.tsx` (entire beta admin
page)
- **Removed:** Beta sidebar nav, layout breadcrumb, admin stats card,
beta types/API methods, legacy YouTube stats cards
- **Added:** GCP Cloud Monitoring section in rate-limits page showing
reported usage, delay, and drift alerts

### Tests (4 files)
- Removed 8 obsolete tests for deprecated methods
- Created `test_gcp_quota_monitor.py` with 10 tests covering GCP quota
querying, caching, fallbacks, and drift detection
- Updated mock data and assertions across test files

### Docs (3 files)
- Removed beta endpoint docs from API.md, added GCP quota fields
- Updated ARCHITECTURE.md with Cloud Monitoring integration
- Removed beta program section from PRODUCT-VISION.md

## Test plan
- [x] All 2265 backend tests pass (1 pre-existing failure unrelated)
- [x] All 516 frontend Jest tests pass
- [x] New GCP quota monitor tests cover success, caching, fallbacks,
drift alerts
- [ ] Verify admin rate-limits page no longer shows beta/legacy cards
- [ ] Verify GCP quota section shows "unavailable" gracefully when
monitoring data not yet populated

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 20:48:07 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 d91c2c2b4e feat: Stripe admin dashboard with revenue analytics and payment management (#461)
<!-- @coderabbitai ignore -->

## Summary
- Full Stripe payment admin dashboard with revenue analytics, payment
history, refund management, payout tracking, dispute monitoring, and
webhook event audit log
- New `StripeAdminService` backend service for payment data storage and
analytics
- Enhanced webhook handler to store enriched payment data and handle
refunds/disputes
- User payment history integration on admin user detail page
- Backfill script for migrating existing payment data
- 32 new unit/endpoint tests covering all P0 critical paths (refund
logic, payment storage, credit deduction)
- Production E2E tests for payments page and API endpoints
- Bug fix: `auth_data[0]` subscript error in refund endpoint (AuthResult
not subscriptable)

## Changes

### Backend (7 phases)
- **New:** `backend/services/stripe_admin_service.py` - Core payment
admin service (store, list, detail, refund, revenue summary, payouts,
disputes, webhook events)
- **New:** `scripts/backfill_stripe_payments.py` - One-time migration
script for existing payments
- **Modified:** `backend/api/routes/admin.py` - 9 new admin endpoints
with Pydantic models
- **Modified:** `backend/api/routes/users.py` - Enhanced webhook handler
with payment storage, refund/dispute handling, event logging

### Frontend
- **New:** `frontend/app/admin/payments/page.tsx` - 4-tab payments page
(Revenue, Transactions, Payouts, Webhooks)
- **New:** `frontend/components/admin/payment-detail.tsx` - Payment
detail slide-out sheet
- **New:** `frontend/components/admin/refund-dialog.tsx` - Refund dialog
with credit deduction warning
- **Modified:** `frontend/lib/api.ts` - 11 new TypeScript interfaces +
adminApi methods
- **Modified:** `frontend/app/admin/users/detail/page.tsx` - Payment
history section
- **Modified:** `frontend/components/admin/admin-sidebar.tsx` - Payments
nav item
- **Modified:** `frontend/app/admin/page.tsx` - Revenue card on
dashboard

### Tests
- **New:** `backend/tests/test_stripe_admin_service.py` - 21 unit tests
- **New:** `backend/tests/test_admin_payment_endpoints.py` - 11 endpoint
tests
- **New:** `frontend/e2e/production/admin-payments.spec.ts` - Production
E2E tests

## Post-deploy steps
1. Register webhook events in Stripe Dashboard: `charge.refunded`,
`charge.dispute.created`, `charge.dispute.closed`
2. Run backfill: `python scripts/backfill_stripe_payments.py`
3. Create Firestore indexes if prompted

## Test plan
- [x] 32 new tests pass (service + endpoint)
- [x] 181 total tests pass (full suite, 1 pre-existing skip)
- [ ] Visual verification of payments page after deploy
- [ ] Backfill script dry-run against production
- [ ] Verify webhook events are logged after deploy


🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 20:22:32 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 6525bf8e1a fix: deploy tenant portal from frontend/ dir for edge functions (#459)
## Summary

- Fix the tenant portal deploy to run from `frontend/` working directory
so Cloudflare Pages discovers the `functions/` directory and deploys the
edge function alongside static files
- Without this, `window.__TENANT_CONFIG__` injection doesn't work
because the Pages Function isn't deployed

## Test plan

- [x] Verified the CI workflow change is correct per [Cloudflare Pages
Functions
docs](https://developers.cloudflare.com/pages/functions/get-started/)
- [ ] After merge: verify `__TENANT_CONFIG__` appears in HTML response
from vocalstar.nomadkaraoke.com

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 17:30:21 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 90ce8951f5 feat: separate B2B tenant portal with edge config injection (#458)
## Summary

- Separate whitelabel B2B tenant portals into their own Cloudflare Pages
project (`karaoke-gen-tenant`) with edge-injected tenant config, fixing
the magic link auth bug and eliminating flash of consumer content
- Fix magic link emails to use tenant-specific URLs and branding (emails
now link to `vocalstar.nomadkaraoke.com` instead of always
`gen.nomadkaraoke.com`)
- Add Cloudflare Pages Function that injects `window.__TENANT_CONFIG__`
and CSS variables at the edge for instant branding
- Add `NEXT_PUBLIC_TENANT_PORTAL=true` build flag for tree-shaking
consumer content from tenant builds
- Add admin tenant preview via `?preview_tenant=X` query parameter
- Add tenant-scoped access tokens (tokens with `tenant_id` override
hostname-based detection)
- Dual build+deploy in CI (consumer → `karaoke-gen`, tenant →
`karaoke-gen-tenant`)

## Test plan

- [x] 17 backend tests pass (magic link params, tenant URLs, verify
response, tenant-scoped tokens, require_auth override)
- [x] 35 frontend tests pass (tenant store, edge-injected config, admin
preview, feature flags)
- [x] 181/182 E2E tests pass (1 pre-existing screenshot failure
unrelated to changes)
- [x] Both consumer and tenant portal builds succeed (`npm run build`
with and without `NEXT_PUBLIC_TENANT_PORTAL=true`)
- [ ] Manual: verify vocalstar.nomadkaraoke.com loads with tenant
branding (already deployed)
- [ ] Manual: verify magic link from tenant portal links to correct
domain

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 17:15:17 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 6ded014acf fix: use nested file_urls structure in queue processor (#457)
## Summary

- Fix queue processor to read video files from `finals.lossless_4k_mkv`
(nested dict) instead of `final_video_mkv` (flat key)
- Fix thumbnail path to use `screens.title_jpg` instead of flat
`title_jpg`
- Required for queuing the 5 jobs that missed YouTube uploads

## Changes
- `youtube_queue_processor.py` — Updated `_download_video_from_gcs` and
`_download_thumbnail_from_gcs` to use nested `file_urls` structure
- `test_youtube_queue_processor.py` — Updated test fixtures to match
nested structure

## Testing
- [x] 13 queue processor tests pass

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 21:43:28 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 98b6590658 feat: YouTube quota-aware upload system with deferred queue (#456)
## Summary

- Replace hard-coded 10 uploads/day limit with YouTube API quota unit
tracking (10,000 units/day, ~33 uploads)
- Queue uploads when quota is exhausted instead of silently skipping
them
- Process queued uploads hourly via Cloud Scheduler, notify users via
email when complete
- Add admin dashboard for quota monitoring and queue management

## Changes

### Backend - New Services
- `youtube_quota_service.py` — Tracks YouTube API quota units per day in
Firestore (`youtube_quota` collection, date-keyed by Pacific Time)
- `youtube_upload_queue_service.py` — Manages deferred upload queue in
Firestore (`youtube_upload_queue` collection) with atomic claiming,
retry logic, and failure handling
- `youtube_queue_processor.py` — Worker that processes queued uploads
when quota available (downloads from GCS, uploads to YouTube, sends
notification)

### Backend - Modified
- `video_worker_orchestrator.py` — Queue on quota exhaustion instead of
skip; record quota units after each API call
- `rate_limit_service.py` — Deprecated YouTube methods delegate to new
quota service with legacy fallback
- `job_notification_service.py` + `template_service.py` +
`email_service.py` — Follow-up email when deferred upload completes
- `rate_limits.py` — 3 new admin endpoints: GET queue, POST retry, POST
process
- `internal.py` — Internal endpoint for Cloud Scheduler to trigger queue
processing
- `config.py` — New settings: `youtube_quota_daily_limit`,
`youtube_quota_upload_cost`, `youtube_quota_safety_margin`

### Frontend
- `rate-limits/page.tsx` — YouTube Queue tab with quota progress bar,
queued uploads table, retry/process buttons
- `api.ts` — Types and API methods for queue management

### Infrastructure (Pulumi)
- Cloud Tasks queue `youtube-upload-queue`
- Cloud Scheduler job: hourly, Pacific Time, calls internal process
endpoint

### Tests (90 passing)
- `test_youtube_quota_service.py` — 16 tests
- `test_youtube_upload_queue_service.py` — 15 tests
- `test_youtube_queue_processor.py` — 14 tests (async)
- `test_rate_limits_api.py` — 4 new endpoint tests
- `test_rate_limit_service.py` — 5 updated/new delegation tests

### Docs
- Updated ARCHITECTURE.md, API.md, README.md, LESSONS-LEARNED.md

## Motivation

4 jobs on 2026-03-03 had YouTube uploads silently skipped because the
hard-coded rate limit of 10 uploads/day was reached. Users expected
uploads but got none with no notification. The actual YouTube API quota
(10,000 units/day) supports ~33 uploads — the rate limit was overly
conservative.

## Testing
- [x] 90 targeted backend tests pass (4.03s)
- [x] Frontend Jest tests pass (503 tests)
- [x] CodeRabbit CLI review completed, feedback addressed

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 20:19:26 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 a75384fb6f docs: complete Singa tenant documentation (#455)
Follow-up docs for PR #453 (Singa tenant). Adds React hydration lesson,
tenant asset endpoint docs, local tenant testing guide, and archive
entry.

<!-- @coderabbitai ignore -->

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 14:31:26 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 2615261672 feat: add Singa whitelabel tenant and fix tenant infrastructure (#453)
## Summary
- Add Singa as second whitelabel tenant (green/black branding,
file-upload-only, locked theme)
- Fix critical Zustand getter bug where computed properties became stale
after set() calls
- Fix React hydration error #300 (conditional return between hooks)
- Add TenantLandingPage component for minimal tenant portal
- Add Cloudflare Pages cache control headers
- Add gs:// URL conversion fallback in tenant-logo

## Test plan
- 27 tenant unit tests pass (5 new regression tests for Zustand bug)
- 181 E2E tests pass
- Verified singa/vocalstar/gen domains all working correctly

<!-- @coderabbitai ignore -->

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 14:05:50 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 e788187351 docs: catalog API, community detection, and autocomplete (#452)
## Summary
- Documentation for the song lookup autocomplete and community version
detection feature shipped in PR #451

## Changes
- **docs/README.md**: Added feature status rows (song lookup
autocomplete, community version detection) and recent changes entry
- **docs/API.md**: New "Catalog (Song Lookup)" section documenting all 3
endpoints (`/api/catalog/artists`, `/api/catalog/tracks`,
`/api/catalog/community-check`) with request/response examples
- **docs/ARCHITECTURE.md**: Added karaoke-decide and KaraokeNerds to
external services table, added `catalog_proxy_service.py` and
`karaokenerds_service.py` to service modules table
- **docs/LESSONS-LEARNED.md**: Added "Autocomplete UX: Less Is More"
pattern (single autocomplete that overwrites both fields > dual
autocompletes)

## Test plan
- Docs-only change, no code modifications

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 18:15:07 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 186b3dd7a9 feat: song lookup autocomplete and community version detection (#451)
## Summary

- Add track autocomplete on the title field using karaoke-decide catalog
API (MusicBrainz + Spotify data) for canonical artist/title styling
- Add community karaoke version detection via karaokenerds.com scraping
— warns users when existing high-quality versions are available on
YouTube, saving credits
- Both features are optional and non-blocking — users can still type
freely and submit without selecting from autocomplete

## Changes

### Backend
- **`backend/services/catalog_proxy_service.py`** — HTTP proxy to
karaoke-decide API with 5-min in-memory TTL cache
- **`backend/services/karaokenerds_service.py`** — Karaokenerds.com HTML
scraper (ported from kjbox) for community version detection
- **`backend/api/routes/catalog.py`** — 3 new endpoints: `GET
/api/catalog/artists`, `GET /api/catalog/tracks`, `POST
/api/catalog/community-check` with per-user rate limiting (20 req/min)
- **`backend/config.py`** — Added `karaoke_decide_api_url` setting

### Frontend
- **`frontend/components/ui/autocomplete-input.tsx`** — Reusable
autocomplete input component (shadcn Command + Popover)
- **`frontend/components/job/CommunityVersionBanner.tsx`** — Green
dismissible banner showing community karaoke versions with YouTube links
- **`frontend/components/job/steps/SongInfoStep.tsx`** — Title field now
has autocomplete; selecting a track overwrites both artist and title
with canonical names
- **`frontend/lib/api.ts`** — Added catalog API client functions

### Tests
- 9 backend route tests (artist/track search, community check, rate
limiting)
- 9 karaokenerds service tests (HTML parsing, community detection,
YouTube URL cleanup)

## How it works

1. User types artist name in the plain text field
2. User starts typing in the title field → autocomplete dropdown shows
tracks from Spotify/MusicBrainz, filtered by artist
3. Selecting a track overwrites both fields with canonical names (e.g.,
"fleetwood mac" → "Fleetwood Mac")
4. When both fields are filled, a community check fires — if existing
karaoke versions exist on YouTube, a green banner appears with links

## Test plan

- [x] 18 backend tests pass (catalog routes + karaokenerds service)
- [x] Tested locally with dev server against real karaoke-decide API
- [x] Verified autocomplete shows results, selection overwrites both
fields
- [x] Verified community banner appears for known songs (e.g., Fleetwood
Mac - Dreams)
- [x] Verified graceful degradation when API is unavailable

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 11:28:01 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 a60c967cfd fix: Buy Credits link opens checkout dialog instead of broken /#pricing link (#450)
## Summary
- **Fix**: "Buy Credits" links in zero-credit warning banners and 402
error messages now open the `BuyCreditsDialog` (Stripe checkout) instead
of navigating to `/#pricing` (which doesn't work for logged-in users)
- **Fix**: `isLocalMode()` now checks pathname before port, fixing E2E
tests running on worktree-hashed ports (3100-3199)
- **Fix**: Updated all E2E regression tests for GuidedJobFlow UI
(replacing old tab-based JobSubmission selectors) and fixed pre-existing
test failures

## Changes
- Replaced 6 `<Link href="/#pricing">Buy Credits</Link>` with `<button>`
+ `BuyCreditsDialog` in `GuidedJobFlow.tsx`, `JobSubmission.tsx`, and
`AudioSourceStep.tsx`
- Reordered checks in `isLocalMode()` so pathname `/local/` is checked
before port allowlist
- Updated `credit-enforcement.spec.ts` for GuidedJobFlow selectors and
search-standalone API
- Updated `karaoke-generation.spec.ts` to use `search-standalone` API
(not old two-step search+poll)
- Updated `mobile.spec.ts` to use `guided-artist-input` (not old
tab-based selectors)
- Fixed `lyrics-review.spec.ts` to open Stats panel before checking
metrics
- Fixed 3 pre-existing failures: admin-token-auth,
instrumental-review-ux, job-management

## Test plan
- [x] All 182 Playwright E2E tests pass (0 failures)
- [x] Credit enforcement tests verify zero-credit warning, 402 error,
and normal credit behavior
- [x] Buy Credits button opens dialog (verified via test assertions
checking `getByRole('button', { name: 'Buy Credits' })`)

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 03:22:08 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 0d5e9f4534 feat: add tutorial video buttons and update homepage demo (#449)
## Summary

- Replace homepage demo video with new quick demo (`RLkIEHGwFcU`)
- Add full tutorial video embed below "How It Works" section
(`-dI3r7qXo3A`)
- Add "Watch full tutorial" links to lyrics review and instrumental
review guidance panels (amber styling with ExternalLink icon, opens
YouTube in new tab)

## Test plan

- [ ] Homepage: verify new demo video plays correctly
- [ ] Homepage: tutorial embed appears below How It Works grid
- [ ] Lyrics review: "Watch full tutorial" link visible in guidance
tips, opens YouTube in new tab
- [ ] Instrumental review: same link visible in both simple and complex
guidance modes

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 01:27:14 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 e44b5f6543 fix: recent jobs UX - cancel→delete, date sort, hide-completed fix, admin controls (#447)
@coderabbitai ignore

## Summary

- **Cancel → Delete**: The Cancel button now permanently deletes jobs
(Firestore doc + GCS files) via the existing `DELETE /api/jobs/{job_id}`
endpoint. Cancelled jobs no longer exist as a concept in the user-facing
flow.
- **Hide completed fix**: The "Hide completed" toggle now only hides
successful completions (`complete`, `prep_complete`). Failed jobs remain
visible so users always see jobs that need attention.
- **Date-only sorting**: Removed priority-based sorting that buried
failed jobs below completed ones. Jobs now sort purely by `created_at`
descending — newest first, always.
- **Admin controls toggle**: Admins see a Shield toggle in the header
that reveals inline admin action buttons (reset, restart, delete, regen
screens, etc.) on each job card — no need to switch to the admin
dashboard.

## Test plan

- [x] Frontend tests pass (496 tests)
- [x] Backend route tests pass (8 tests)
- [x] `next build` succeeds
- [x] New AdminJobActions component has unit tests
- [x] JobCard admin controls rendering tested
- [x] JobActions cancel→delete behavior tested
- [x] E2E regression test updated for delete flow
- [ ] Visual verification: failed job appears at top of Recent Jobs
- [ ] Visual verification: "Hide completed" keeps failed jobs visible
- [ ] Visual verification: Admin shield toggle shows/hides admin actions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 23:28:41 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 a38096e8cc fix: prevent test hanging from emulator prompts, zombies, port conflicts (#448)
## Summary

- Add `--quiet` flag to `gcloud beta emulators firestore start` to
prevent interactive prompts that hang CI/agent sessions
- Fix `stop-emulators.sh` to kill Java child processes (`pkill -P`) with
port-based fallback for orphans on port 8080
- Generate unique Playwright dev server port per worktree (3100-3199
hash range) to prevent cross-worktree conflicts

## Test plan

- [ ] `./scripts/start-emulators.sh` starts without interactive prompts
- [ ] `./scripts/stop-emulators.sh` kills Java emulator child processes
(no orphans on port 8080)
- [ ] Playwright config generates different ports for different worktree
directories
- [ ] `E2E_PORT` env var still overrides the default port

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 23:18:28 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 3e81bbe9a8 fix: make flacfetch status polling resilient to transient network errors (#446)
## Summary
- Add retry tolerance to `wait_for_download` status polling — a single
network blip no longer fails the entire job. Up to 3 consecutive poll
failures are tolerated before giving up.
- Fix empty error messages: `httpx.RequestError` subclasses often have
empty `str()` representations. New `_format_httpx_error()` helper always
produces descriptive messages including error class name and request
URL.
- Changed `get_download_status` to let network errors bubble up to
`wait_for_download` where retry logic handles them, instead of wrapping
them into an unrecoverable `FlacfetchServiceError`.

**Root cause:** Job `c6b1b00f` failed with `"Audio download failed:
Status request failed: "` (empty) — the download had actually succeeded
and the file was in GCS, but a single transient `ConnectError` during
status polling killed the job.

## Test plan
- [x] 43/43 flacfetch client tests pass (8 new tests added)
- [x] Tests cover: single poll failure tolerated, two consecutive
tolerated, three consecutive fails, counter resets on success, custom
threshold, empty error string scenario, application errors still fail
immediately
- [ ] CI passes

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 23:14:04 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 7b7c0f7ff4 fix: allow PENDING → DOWNLOADING_AUDIO state transition (#445)
## Summary
- Adds `DOWNLOADING_AUDIO` as a valid transition from `PENDING` in the
state machine
- Fixes 500 error on the `create-from-search` endpoint deployed in #444
- The `create-from-search` path creates a job (PENDING) then transitions
directly to DOWNLOADING_AUDIO since audio is pre-selected from the
search session

## Changes
- `backend/models/job.py` — Add DOWNLOADING_AUDIO to PENDING transitions
- `backend/scripts/generate_state_diagram.py` — Keep diagram in sync
- `backend/tests/test_standalone_search.py` — Add state transition test

## Testing
- All 31 standalone search tests pass
- All 158 related tests pass (standalone + models + audio search)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 19:22:32 -05:00
Andrew Beveridge ☄️andClaude Sonnet 4.6 f17bc931d0 feat: decouple audio search from job creation (search sessions) (#444)
## Summary

- Fixes 4 UX problems caused by the guided wizard creating a backend job
at Step 2 (audio search), before the user has set `is_private`, display
overrides, or confirmed they want a job at all
- Adds `POST /api/audio-search/search-standalone` — runs search
synchronously, stores results in a 30-minute Firestore session, returns
`search_session_id + results` (no job created)
- Adds `POST /api/jobs/create-from-search` — creates job at Step 3 with
all correct values from the start; atomically consumes the session to
prevent duplicate job creation

## Problems Solved

**A) Orphan job records** — abandoned searches left job records in
Firestore; back button `deleteJob` cleanup was best-effort
**B) Wrong field values at creation** — job existed briefly with
`is_private: false` and no display overrides
**C) Jobs in Recent Jobs list** — `AWAITING_AUDIO_SELECTION` jobs
appeared in the list before user confirmed
**D) Notification chime fired** — state transitions on the temporary job
triggered the action-required chime during search

## Changes

- `backend/routers/audio_search.py` — new `POST
/api/audio-search/search-standalone` endpoint (synchronous, no job)
- `backend/routers/jobs.py` — new `POST /api/jobs/create-from-search`
endpoint with validate-then-consume session pattern
- `backend/services/firestore_service.py` — `create_search_session`,
`get_search_session`, `consume_search_session` (atomic transaction),
`delete_search_session`
- `infrastructure/modules/database.py` — Firestore TTL policy for
`search_sessions` collection (30-min auto-expiry)
- `frontend/lib/api.ts` — `searchStandalone`, `createJobFromSearch`
methods
- `frontend/components/job/steps/AudioSourceStep.tsx` — uses standalone
search; no polling; no job state
- `frontend/components/job/GuidedJobFlow.tsx` — defers job creation to
Step 3 confirm; no `deleteJob` on back

**Unchanged:** old `POST /api/audio-search/search` (CLI backward
compat), upload/URL paths, `AWAITING_AUDIO_SELECTION` status.

## Testing

- [x] 26 new backend unit tests
(`backend/tests/test_standalone_search.py`) covering session CRUD, both
new endpoints, ownership/tenant/TTL/index validation, atomic consumption
- [x] 13 new frontend Jest tests
(`frontend/__tests__/guided-flow.test.tsx`) covering API contract,
component exports, behavioral invariants
- [x] Full test suite passes: 2167 backend + 507 frontend

## Review

- [x] CodeRabbit CLI review completed (3 cycles, 0 remaining findings)
- [x] Atomic session consumption via Firestore transaction (prevents
duplicate jobs on concurrent requests)
- [x] Validate-before-consume pattern (4xx errors leave session intact
for user retry)
- [x] Tenant consistency check added

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 18:54:38 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 324b7a3240 fix: guided flow private mode + delayed GDrive validation (#443)
## Summary

Two fixes for the NOMAD-1276 incident where the GDrive validator
reported "no issues" while NOMAD-1276 was actually missing:

- **Fix 1: Private mode in guided flow** — The `is_private` checkbox in
Step 3 (Customize) had no effect because the job was created in Step 2
with `is_private=false`. Now the select endpoint accepts optional
overrides (`is_private`, `display_artist`, `display_title`) and applies
them before the pipeline starts.
- **Fix 2: Delayed GDrive validation** — Post-job validator fired
immediately after upload, before E2E test cleanup could delete test
files, causing false "all clear" reports. Now uses Cloud Tasks with a
5-minute delay, with fallback to immediate trigger if scheduling fails.

Also includes cross-folder gap detection improvements in the Cloud
Function (uses global max across all folders instead of per-folder max).

## Changes

### Backend
- `AudioSelectRequest` model accepts optional `is_private`,
`display_artist`, `display_title` overrides
- New `/internal/trigger-gdrive-validation` endpoint for delayed Cloud
Tasks invocation
- `WorkerService.schedule_gdrive_validation()` method with 5-minute
delay
- Orchestrator uses delayed scheduling with immediate fallback

### Frontend
- `selectAudioResult()` passes overrides from Step 3 state
- E2E test verifies overrides reach the backend

### Infrastructure
- New `gdrive-validation-queue` Cloud Tasks queue (Pulumi)
- Cloud Function cross-folder gap detection fix + 28 unit tests

## Test plan

- [x] 19 new backend tests (6 select overrides + 13 delayed validation)
- [x] 2 new frontend API tests for override behavior
- [x] 28 new Cloud Function unit tests
- [x] Updated E2E test to verify overrides reach backend
- [x] Full backend suite: 2139 passed
- [x] Full frontend suite: 485 passed
- [x] CodeRabbit: no findings

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 16:48:40 -05:00
Andrew Beveridge ☄️andClaude 7e3e247858 feat: improve audio upload UX + signed URL flow for large files (#442)
@coderabbitai ignore

## Summary

- **Better no-results UX**: Replace bare "No audio sources found" text
with a contextual `NoResultsSection` card explaining why results might
be missing (unreleased tracks, independent music, unusual characters)
and what to try
- **Softer timeout styling**: Timeout errors now show amber
(`text-amber-400 bg-amber-500/10`) instead of red — it's a delay, not a
failure. Credit/API errors remain red
- **Larger fallback buttons** when no results: `h-12` height and `w-4
h-4` icons (vs `h-9`/`w-3.5 h-3.5`) since upload/URL are the primary
actions
- **Fix file input text overlap**: Replace `opacity-0` absolute
`<Input>` with `<label>` + `<input className="sr-only">` — eliminates
browser "Browse..." bleed-through, adds "Click to browse or drag a file"
helper text
- **Signed URL upload flow** for large files (≥25MB): Bypasses Cloud
Run's 32MB body limit; files <25MB still use the existing direct upload
- **Upload progress bar**: Phase labels (Preparing / Uploading X/Y MB /
Finalizing) + animated progress bar during signed URL uploads
- **Large file warning** (>100MB): Amber amber notice about expected
upload time
- **GCS CORS config**: Add Pulumi CORS rule for `PUT` from
`gen.nomadkaraoke.com` and `localhost:3000` — required for
browser-to-GCS signed URL uploads to work

⚠️ **Deploy order matters**: The GCS CORS config (infrastructure change)
must be deployed before the signed URL upload path will work in
production. Both changes are in this PR so they deploy together on
merge.

## Test plan

- [x] 492 Jest tests passing (9 new tests for `createJobWithUploadUrls`,
`uploadToSignedUrl`, `completeJobUpload`, `uploadJobSmart`)
- [x] `npx next build` succeeds
- [x] CodeRabbit CLI: no findings
- [ ] **Manual**: Search for a nonexistent track → verify
`NoResultsSection` appears with guidance text and larger buttons
- [ ] **Manual**: Trigger a timeout (slow network) → verify amber banner
(not red)
- [ ] **Manual**: Open upload form → verify no "Browse..." text
bleed-through, see "Click to browse or drag a file"
- [ ] **Manual** (post-deploy): Upload a file ≥25MB → verify progress
bar appears and upload completes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-03-02 15:40:05 -05:00
Andrew Beveridge ☄️andClaude Sonnet 4.6 10ba0e4b4a fix: show upload/YouTube options when audio search returns no results (#441)
## Summary

- Fixes a bug where the YouTube URL and file upload options were hidden
when audio search returned zero results
- Adds a guidance note on Step 1 for users with unreleased songs or
their own audio files

## Changes

- `AudioSourceStep.tsx`: Fixed condition on bottom `FallbackSection` to
also render when `results.length === 0` (previously it was gated on
`confidence.tier !== 3`, but empty results produce tier 3 — so the
fallback never showed)
- `SongInfoStep.tsx`: Added an info note below the custom lyrics note,
explaining that users with unreleased songs should enter artist/title
anyway, upload on step 2, and mark private on step 3 if needed
- `package.json`: Added `@testing-library/dom` dev dependency (was
missing, caused test suite failures)

## Testing

- [x] `audio-search-utils` unit tests pass (76 tests — cover the tier 3
/ empty results logic)
- [x] All 483 frontend unit tests pass
- [x] CodeRabbit CLI: no findings

## Review

- [x] Local CodeRabbit review completed — no findings
- [x] Logic verified: `getSearchConfidence([], ...)` returns `tier: 3`;
condition fix ensures `FallbackSection` renders for that case

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 13:33:30 -05:00
Andrew Beveridge ☄️andClaude Sonnet 4.6 7ee59e3a2d ci: auto-refresh E2E session token via impersonation API (#440)
## Summary

- Eliminates the static `E2E_TEST_TOKEN` GitHub secret that expires
every 30 days and silently breaks the daily E2E test
- In `ci.yml` (PR-triggered runs), a new step calls the admin
impersonation API to mint a fresh session token before each test run
- In `e2e-happy-path.yml`, the same auto-refresh runs when
`use_test_token=true` (manual dispatch); scheduled daily runs continue
to exercise the full magic link signup flow (no change)

## Changes

- **`.github/workflows/ci.yml`**: Insert "Refresh E2E session token"
step that POSTs to
`/api/admin/users/e2e-test-runner@nomadkaraoke.com/impersonate` with
`E2E_ADMIN_TOKEN`; captures the returned `session_token` as a masked
step output; passes it as `E2E_TEST_TOKEN` to the test run
- **`.github/workflows/e2e-happy-path.yml`**: Same pattern with `if:
inputs.use_test_token == true` guard so it only runs for manual
fast-path dispatch, not scheduled runs

## Post-merge Action

The `E2E_TEST_TOKEN` GitHub Actions secret is no longer needed and can
be deleted from the repo secrets page.

## Testing

- [x] All backend + frontend tests pass locally (`make test`)
- [x] CodeRabbit CLI review: no findings

## Review

- [x] CodeRabbit CLI review completed locally (no findings)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 13:03:15 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 6a0aaa440c fix: E2E Step 4 timeout + CI Gate enforces e2e-happy-path label (#439)
## Summary

- **Fix Step 4 timeout**: Added explicit
`expect(artistInput).toBeVisible()` wait before filling the guided
artist input. The app page has loading states (auth spinner →
WarmingUpLoader → GuidedJobFlow) that must complete before the form is
interactable.
- **CI Gate enforcement**: When a PR has the `e2e-happy-path` label, the
CI Gate now **requires** the E2E Happy Path job to pass before the PR
can be merged. Previously, the E2E happy path ran but wasn't gated,
allowing premature merges.

## Changes

1. `frontend/e2e/production/happy-path-real-user.spec.ts` — Wait for
`guided-artist-input` to be visible with 30s timeout before attempting
to fill it
2. `.github/workflows/ci.yml` — Add `frontend-e2e-happy-path` to CI Gate
`needs` array and check its result when the label is present

## Test plan

- [ ] PR has `e2e-happy-path` label → CI Gate must wait for E2E Happy
Path to complete
- [ ] E2E Happy Path Step 4 should no longer timeout on
guided-artist-input
- [ ] PRs WITHOUT the label should still pass CI Gate normally (E2E
happy path shows as "skipped")

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 12:31:52 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 388b239941 fix: Update E2E tests for magic link signup flow (#438)
## Summary
- E2E happy path test (run #87) failing because landing page beta
enrollment was removed
- Updated both production E2E tests (`happy-path-real-user.spec.ts`,
`full-user-journey.spec.ts`) to use the new AuthDialog magic link signup
flow
- Updated `E2E-HAPPY-PATH-TEST-SPEC.md` to match

## Changes
The beta program was removed from the landing page in #430 but the E2E
tests still looked for "Beta Tester Program" text. The new flow:
1. Click "Sign Up Free" → opens AuthDialog
2. Enter email → click "Send Sign-In Link"
3. Wait for magic link email via testmail.app
4. Navigate to verify URL → authenticate → redirect to /app

## Test plan
- [x] Local unit tests pass (483/483)
- [ ] CI checks pass
- [ ] E2E happy path test passes in CI (triggered by `e2e-happy-path`
label)

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 11:32:36 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 0ba5193601 fix: recycle brand codes on delete-outputs and job deletion (#437)
## Summary

- E2E happy path test now uses **private mode** (NOMADNP prefix) so test
jobs never consume public NOMAD brand codes or upload to GDrive/YouTube
- `delete-outputs` admin endpoint now **recycles brand codes** after
successful Dropbox + GDrive cleanup
- `DELETE /api/jobs/{id}` endpoint now **recycles brand codes** before
deleting the job record
- Frontend delete-outputs dialog accurately shows brand code recycling
status (recycled/skipped/failed)

## Context

Test jobs ("piri - dog") consumed NOMAD-1275 and NOMAD-1276 brand codes.
When manually deleted via `delete-outputs`, the brand codes were not
recycled back to the counter. This caused the GDrive validator to detect
sequence gaps at 1275/1276 when NOMAD-1277 was later uploaded.

## Test plan

- [x] 7 new unit tests for brand code recycling in both endpoints
- [x] All 490 existing tests pass (483 frontend + backend)
- [x] Frontend builds successfully (TypeScript types updated)
- [ ] Manual: run E2E test, verify job uses NOMADNP prefix
- [ ] Manual: delete-outputs on a test job, verify brand code recycled
in response

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 11:25:04 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 89c37a37f3 Reduce idle reminder delay from 5 to 2 minutes (#435)
## Summary
- Reduce the `IDLE_REMINDER_DELAY_SECONDS` from 5 minutes to 2 minutes
so users get their review prompt email sooner after a job reaches
ready-for-review status

## Test plan
- [ ] CI passes
- [ ] Deploy and verify next job that hits AWAITING_REVIEW sends email
after ~2 minutes

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 00:42:40 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 d415c3b2f5 Update homepage demo video (#436)
## Summary
- Update the YouTube demo video on the landing page from `uvKtLOUiqz8`
to `GJY60IfIHOQ`

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 05:41:08 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 a8ea9e6d38 feat: in-app Buy Credits dialog and pricing updates (#434)
## Summary
- Fix broken "Buy More Credits" flow for logged-in users — replaces
redirect (which bounced back to `/app`) with an in-app dialog
- Add `BuyCreditsDialog` component following the existing
`FeedbackDialog` pattern
- Make credits count in header clickable to open the buy dialog
- Update "Made For You" pricing from $15 to $50 (frontend labels +
Stripe backend)
- Update hero section: "Create It Yourself ($5)", green "2 free credits"
message, remove redundant line, tighten spacing

## Test plan
- [ ] Log in, click "Buy More Credits" in dropdown — dialog opens with
package grid
- [ ] Click credits count in header — same dialog opens
- [ ] Select package, click "Continue to Payment" — redirects to Stripe
checkout
- [ ] Landing page pricing still works for unauthenticated users
- [ ] Made For You order form shows $50 pricing
- [ ] Hero section shows updated copy and tighter spacing

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 05:04:22 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 9511d03912 Add delete user to admin dashboard (#433)
## Summary
- Adds backend `delete_user` method to UserService that deletes user
document, sessions, and magic links (prevents deleting admins)
- Adds `DELETE /admin/users/{email}` endpoint with admin auth
- Adds `deleteUser` to frontend API client
- Adds Delete User button + AlertDialog confirmation on admin user
detail page

Jobs are preserved as historical records — only user auth data is
removed.

## Test plan
- [ ] Navigate to admin user detail page for a non-admin test user
- [ ] Verify "Delete User" button appears (should not appear for admin
users)
- [ ] Click delete, verify confirmation dialog appears
- [ ] Confirm deletion, verify success toast and redirect to user list
- [ ] Verify user no longer appears in the list

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 04:41:13 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 e11a5976e3 docs: update for feedback-for-credits and beta deprecation (#432)
## Summary
- Update docs to reflect feedback-for-credits feature (PR #431) and beta
program removal (PR #430)
- Add `user_feedback` Firestore collection to architecture docs
- Mark beta program section as deprecated in API reference
- Add archive entry for the feedback-for-credits feature

## Changes
| File | Update |
|------|--------|
| `docs/README.md` | Replace beta with feedback-for-credits in status
table, add recent changes entries |
| `docs/API.md` | Mark beta section deprecated, document
`feedback_eligible` in `/me`, link from credit enforcement |
| `docs/ARCHITECTURE.md` | Add `user_feedback` collection, mark
`beta_feedback` as deprecated |
| `docs/archive/2026-03-01-feedback-for-credits.md` | New archive entry
|

## Test plan
- [x] Docs-only change, no code modified
- [x] Verified all internal doc links are valid

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 23:35:54 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 fc659d0357 feat: feedback-for-credits mechanism for all users (#431)
## Summary

- Users who complete 2+ karaoke videos can submit product feedback to
earn 2 free credits
- Replaces beta-only feedback flow — available to ALL users
- PR 2 of 2: follows #430 which removed the beta program from homepage

## Changes

**Backend:**
- New `POST /api/users/feedback` endpoint with eligibility checks,
validation (>50 char text), credit granting
- New `GET /api/users/feedback/eligibility` endpoint
- `feedback_eligible` field added to `/api/users/me` response
- `UserFeedback` model + `user_feedback` Firestore collection
- Welcome email updated with prominent credit display and feedback
teaser

**Frontend:**
- `FeedbackDialog` component: 4 star ratings, 3 text fields, 2
checkboxes, submit/success states
- Dismissible banner on dashboard when user is eligible
- "Earn 2 Free Credits" menu item in user dropdown (green Gift icon)
- Auto-refreshes user data after submission to hide prompts

**Docs:**
- API.md updated with new endpoints

## Test plan

- [ ] Backend Python syntax verified (ast.parse)
- [ ] Frontend builds successfully (`npx next build`)
- [ ] Version bumped to 0.121.0
- [ ] Manual flow: sign up → create 2 jobs → see banner → submit
feedback → credits granted → banner disappears
- [ ] Edge cases: duplicate submission blocked, ineligible users don't
see prompts

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 23:05:45 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 61e4cec189 Remove beta program, promote free credits on homepage (#430)
## Summary

- Remove the beta tester program section from the homepage — users
already get 2 free credits on signup, making the beta enrollment form
unnecessary friction
- Update hero CTA to open auth dialog with "Free" badge instead of
linking to pricing with "$5"
- Add prominent free credits callout card in the pricing section with
"Sign Up Free" button
- Remove 5 beta enrollment E2E tests, update landing page and smoke
tests for new free credits UI
- Update product communication guide to reflect free credits instead of
beta program

This is PR 1 of 2. Backend beta endpoints and admin pages are unchanged.
PR 2 will add the feedback-for-credits mechanism.

## Test plan

- [x] `npm run build` passes with no errors from removed imports/state
- [x] 4 auth regression tests pass (magic link + token persistence)
- [x] Landing page and smoke tests updated and passing for free credits
UI
- [x] All 161 passing regression tests still pass (21 pre-existing
failures unrelated to this change)

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 21:06:56 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 30105b1ad1 Update homepage screenshots and add demo recording tooling (#429)
## Summary
- Replace all 4 homepage AVIF screenshots with fresh captures from
production E2E test reflecting recent UX improvements
- Update screenshot labels and descriptions in landing page (page.tsx)
- Add demo video recording infrastructure: Playwright config, E2E spec,
and ffmpeg post-processing script
- Refresh all 16 docs/images screenshots from latest E2E happy path run

## Changes
- **Homepage screenshots**: Replaced job-dashboard, lyrics-review,
instrumental-review, and example-output AVIF/PNG files
- **Landing page copy**: Updated alt text, titles, descriptions to match
new UX (e.g., "Create Your Video", "Review Lyrics", "Choose Your
Instrumental", "Download & Share")
- **Demo tooling**: `playwright.demo.config.ts` (1280x720 recording),
`demo-recording.spec.ts` (paced walkthrough), `create-demo-video.mjs`
(ffmpeg post-processor)
- **npm scripts**: Added `test:e2e:demo`, `test:e2e:demo:headed`,
`demo:create-video`

## Test plan
- [x] All 483 unit tests pass
- [x] Next.js build succeeds
- [x] Screenshots verified from production E2E run
- [x] Demo recording spec tested against production (18.8 min run, all
10 scenes captured)

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 18:59:25 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 5bfefd2df9 Improve E2E test robustness and add production screenshots (#428)
## Summary
- Add `gotoWithRetry()` helper for resilient navigation (retries up to
3x with backoff)
- Apply to all production page navigations (landing, /app, review page)
- Construct absolute URL for review page instead of relying on relative
href
- Add content verification after review page load with automatic reload
fallback
- Add 16 production screenshots documenting the complete happy path
journey

## Root Cause
The previous test run failed with `net::ERR_INTERNET_DISCONNECTED` when
opening the review page in a new tab — a transient network error during
the long-running test. The retry logic handles this gracefully.

## Verified
Full end-to-end test passed against production (17.8 minutes, all 12
steps).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 14:43:46 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 6248b2c50d Fix title card preview font loading race condition (#427)
<!-- @coderabbitai ignore -->

## Summary
- Fix canvas title card preview rendering with serif fallback font on
initial page load
- Canvas API doesn't participate in CSS font swap, so it was drawing
text before the custom font (AvenirNext-Bold) finished loading
- Now explicitly checks `document.fonts.check()` and awaits
`document.fonts.load()` before rendering text on the canvas
- Removed the unreliable `document.fonts.ready` effect (resolves
immediately when no fonts are actively loading)

## Test plan
- [x] Added 5 unit tests for TitleCardPreview font loading behavior
- [x] All 483 frontend unit tests pass
- [x] All 1086 backend tests pass
- [x] CodeRabbit review: no findings
- [ ] Verify in production that title card preview renders with correct
font on first load

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 14:40:59 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 4fe0a8f7c8 Improve E2E screenshot quality and timing (#426)
## Summary
- Remove redundant/duplicate screenshots (05b, 07b, 07e/07f/07g, 08d
duplicate, 10)
- Add render waits before screenshots that were capturing too early
(customize step, preview modal, instrumental page)
- Consolidate the preview video screenshot group into a single
meaningful capture

Reduces total screenshots from 23 to ~17 while improving quality of each
capture.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 14:05:28 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 4bf70138b5 Fix notification chime behavior for audio selection and completion (#425)
## Summary
- Stop playing the action-needed chime when a job enters
`awaiting_audio_selection` — the user is already engaged with the inline
audio picker, so the notification is unnecessary
- Add a distinct success chime (rising C major arpeggio: C6→E6→G6) for
job completion, so users can tell "done" apart from "attention needed"
by sound alone
- Keep existing alert chime for `awaiting_review`, `in_review`,
`awaiting_instrumental_selection`, and `failed`

## Changes
- **`frontend/lib/job-status.ts`**: Add `isNotifiableBlockingStatus()` —
delegates to `isBlockingStatus()` but excludes
`awaiting_audio_selection`. UI sorting/badges/cards unchanged.
- **`frontend/hooks/use-notifications.ts`**: Refactor sound system into
`playAlertChime()` and `playSuccessChime()`, route by
`NotificationReason`. Switch notification filtering to
`isNotifiableBlockingStatus()`.
- **`frontend/__tests__/job-status.test.ts`**: 7 new tests for
`isNotifiableBlockingStatus` (true/false cases, case insensitivity,
null/undefined).

## Test plan
- [x] All 478 Jest tests pass (including 7 new tests)
- [x] CodeRabbit review: no findings
- [ ] Manual: create a job via audio search — confirm no chime during
audio selection
- [ ] Manual: let a job reach review — confirm alert chime plays
- [ ] Manual: let a job complete — confirm distinct success chime plays

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 13:50:13 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 e3645b54d4 Fix happy path E2E test for guided job creation flow (#424)
## Summary
- Update Steps 4-5 of the happy path E2E test to use the new 3-step
guided flow
- Old flow: Search tab → dialog → select → job card
- New flow: Song Info → Choose Audio (inline search) → Customize &
Create → Job Created

## Test plan
- [x] Run happy path test against production
- [ ] All CI checks pass

@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 13:16:45 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 221f26b0d8 UX overhaul: guidance & simplification across all 4 review stages (#421)
## Summary

Complete UX overhaul of the karaoke video generation review flow,
spanning 4 phases:

- **Phase 1 - Job Creation**: Guided audio source selection with 3-tier
confidence system, responsive stepper, renamed "Find Audio" to "Choose
Audio"
- **Phase 2 - Audio Selection**: Production E2E tests for guided job
creation flow
- **Phase 3 - Lyrics Review**: Guidance panel with workflow steps, gap
navigation scoped to visible lines, simplified review UI with edit
tracking and low-friction feedback
- **Phase 4 - Instrumental Review**: Responsive layout (waveform no
longer disappears on short viewports), canvas-rendered time ruler,
hover-to-highlight waveform regions, dedicated Alternate Instrumental
upload section with duration explanation, dismissable guidance panel

### Key changes
- Root layout switches from viewport-locked (`h-screen overflow-hidden`)
to scrollable (`min-h-screen`) — fixes waveform disappearing on short
viewports
- Bottom section stacks vertically on mobile/tablet, side-by-side on
desktop
- Time ruler rendered directly on canvas with auto-scaling tick
intervals
- Hovering mute region or segment buttons highlights the corresponding
waveform range
- Upload button moved from toolbar to dedicated section with context and
duration requirement
- 21 unit tests added for instrumental review components
- Guidance panels with dismissable tips across lyrics and instrumental
review

## Test plan
- [x] `npx next build` — no errors
- [x] `npx jest --testPathPattern="instrumental-review/__tests__"` —
21/21 pass
- [x] Visual testing at 4 viewports: desktop (1280x800), tablet
(768x1024), mobile (375x667), short (1280x400)
- [x] Hover highlight verified with Playwright MCP
- [x] Time ruler scales correctly at 1x and 2x zoom

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 02:58:50 -05:00
Andrew Beveridge ☄️ ef4439b6cc Added docs/archive/2026-02-28-gdrive-validator-email-alerts-plan.md (#423) 2026-02-28 21:43:18 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 e102a2bf36 Fix event loop blocking in audio search endpoints (#422)
@coderabbitai ignore

## Summary
- Fix `/api/audio-search/{job_id}/select` taking 20-36s to respond
(should be <1s) and `/api/audio-search/search` sometimes taking 25-47s
- Root cause: async endpoints called sync methods that create new event
loops via `loop.run_until_complete()`, blocking the FastAPI event loop
- RED/OPS and Spotify download paths now call FlacfetchClient async
methods directly (matching YouTubeDownloadService pattern)
- Generic downloads use `asyncio.to_thread()` as safe fallback
- Search endpoint uses `search_async()` instead of sync `search()`
- Fixed `InsufficientCreditsError` occasionally returning 500 instead of
402
- Added fail-fast guard for missing `remote_search_id` in torrent
download fallback

## Test plan
- [x] 14 new tests covering async download paths, error handling, and
exception propagation
- [x] All 2099 tests pass (1 pre-existing unrelated failure in test
ordering)
- [x] CodeRabbit review: 1 finding fixed (missing remote_search_id
guard)
- [ ] Deploy to prod and verify `/select` latency drops from 20-36s to
<1s
- [ ] Verify InsufficientCreditsError returns 402 not 500

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 16:41:59 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 18fa1aa03f Fix gdrive-validator scheduler IAM + API key whitespace (#420)
@coderabbitai ignore

## Summary

- Add `roles/run.invoker` for the validator's own service account on the
Cloud Run service — Gen2 functions need this for Cloud Scheduler's OIDC
token to be accepted
- Strip whitespace from Pushbullet and SendGrid API key env vars
(handles trailing newlines in Secret Manager values)

This codifies the manual gcloud fix from #412 into Pulumi IaC and closes
that PR.

## Already deployed

- Pulumi applied: IAM binding created
- Cloud function redeployed with `.strip()` fix

## Test plan

- [x] `pulumi preview` — clean, 1 create
- [x] `pulumi up` — applied successfully
- [x] Cloud function redeployed

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 01:14:23 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 e0c8645fea Add email alerts, manual trigger, and post-job check to GDrive validator (#419)
@coderabbitai ignore

## Summary

- **Email notifications via SendGrid** — replaces Pushbullet as the
primary notification channel for the GDrive validator cloud function.
Pushbullet kept as secondary fallback.
- **Post-job trigger** — the backend orchestrator now triggers GDrive
validation after each job that uploads to the public share, catching
duplicates within minutes instead of up to 24 hours.
- **Manual CLI** — `python scripts/check_public_share.py` for on-demand
validation checks with formatted output and `--json` flag.
- **Comprehensive docs** — `docs/GDRIVE-VALIDATOR.md` with architecture,
configuration, troubleshooting, and operational reference.

Motivated by the NOMAD-1271 duplicate brand code incident where
daily-only checks left a duplicate undetected for ~24 hours.

## Changes

| File | Change |
|------|--------|
| `infrastructure/functions/gdrive_validator/main.py` | Add
`send_email_notification()` via SendGrid, unified `send_notification()`
sends both email + Pushbullet |
| `infrastructure/functions/gdrive_validator/requirements.txt` | Add
`sendgrid>=6.0.0` |
| `infrastructure/functions/gdrive_validator/deploy.sh` | Add SendGrid
secret + email env vars |
| `infrastructure/__main__.py` | Add SendGrid secret to function,
backend SA invoker IAM (Cloud Functions + Cloud Run) |
| `infrastructure/modules/secrets.py` | Add `gdrive-validator-url`
secret |
| `.github/workflows/ci.yml` | Add `GDRIVE_VALIDATOR_URL` to Cloud Run
deploy secrets |
| `backend/services/gdrive_validator_client.py` | **New**: HTTP client
with OIDC auth, fire-and-forget pattern |
| `backend/workers/video_worker_orchestrator.py` | Add Stage 6:
post-distribution GDrive validation (only when `gdrive_folder_id` set) |
| `scripts/check_public_share.py` | **New**: CLI tool for manual
validation checks |
| `docs/GDRIVE-VALIDATOR.md` | **New**: Comprehensive validator
documentation |
| `pyproject.toml` | Version bump to 0.119.9 |

## Already deployed

- Pulumi infrastructure applied (IAM bindings, secrets, function config)
- Cloud Function redeployed with SendGrid support
- `gdrive-validator-url` secret set in Secret Manager
- Tested via `check_public_share.py` — validated 3,608 files, email sent
successfully

## Test plan

- [x] `make test` — 170 passed, 3 skipped (1 pre-existing failure in
credit-enforcement unrelated to this PR)
- [x] `pulumi preview` — clean, 3 creates + 1 update
- [x] `pulumi up` — applied successfully
- [x] Cloud Function deployed and tested end-to-end
- [x] `python scripts/check_public_share.py` — formatted output works
- [x] `python scripts/check_public_share.py --json` — JSON output works
- [x] Email received at gen@nomadkaraoke.com (notification_sent: true)
- [ ] Post-job trigger: will verify on next job completion (backend
needs redeploy via CI merge)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 01:05:16 -05:00
Andrew Beveridge ☄️andClaude c7b7f4bee5 fix: gate brand code recycling on full GDrive cleanup confirmation (#418)
## Summary

- Brand code `NOMAD-1271` was reused for a real song (Hot Mulligan - Pop
Shuvit) while old E2E test job files still existed in Google Drive,
creating a duplicate in the public share
- Root cause: `cleanup-distribution` endpoint had two bugs that allowed
brand codes to be recycled before GDrive was confirmed clean

## Root Causes

1. **Empty dict was falsy**: `if gdrive_files:` skips cleanup when
`gdrive_files = {}` (empty dict). This happened when GDrive uploads
silently failed (pre-PR #417), so test job files accumulated in GDrive
without ever being cleaned
2. **Brand code recycled too early**: Brand code was recycled
immediately after Dropbox cleanup succeeded, *without* confirming GDrive
was also clean — a new job could claim the recycled code while old
GDrive files still existed

## Changes

- **`gdrive_service.py`**: Added `find_files_by_brand_code()` — searches
CDG/, MP4/, MP4-720p/ subfolders by brand code prefix as a fallback when
tracked file IDs aren't available (old jobs, or jobs where upload failed
silently). Raises `RuntimeError` if any subfolder search fails so
callers can't misread an incomplete search as "GDrive is clean"
- **`jobs.py` (`cleanup-distribution`)**: Now uses tracked file IDs
(fast path) OR falls back to brand_code search; brand code is only
recycled after **both** Dropbox AND GDrive are confirmed clean. Fixed
`method` key being lost when results dict was replaced (use local
variable instead of interim mutation)
- **`scripts/cleanup_test_job_gdrive.py`**: Remediation script to delete
existing stranded test job files from GDrive

## Testing

- 41 tests passing, 94% coverage on `gdrive_service.py`
- New `TestDeleteFile` class (5 tests for `delete_file`/`delete_files` —
previously uncovered)
- New `test_cleanup_distribution.py` (9 tests covering fast path,
fallback, and brand code gating logic)
- Updated error handling test to verify `RuntimeError` is raised (not
swallowed) on subfolder search failure
- [x] CodeRabbit CLI review completed, 2 findings fixed

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-02-27 17:35:50 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 0f5ddb0092 Fix silent Google Drive upload failures with retry + alerting (#417)
## Summary
- Added tenacity retry with connection reset to Google Drive API calls
(`get_or_create_folder`, `upload_file`, `delete_file`) to handle stale
HTTP connections on idle Cloud Run containers
- Added `distribution_warnings` field to `OrchestratorResult` and
`_handle_native_distribution()` to track non-fatal upload failures
(YouTube, Dropbox, GDrive)
- Added Pushbullet notification when distribution uploads fail, so
operator is alerted immediately
- Added backfill script (`scripts/backfill_gdrive_uploads.py`) for
affected jobs - already run successfully for 12 affected jobs
- Updated docs: LESSONS-LEARNED.md and TROUBLESHOOTING.md with GDrive
stale connection pattern

## Context
Every GDrive upload silently failed for ~1 month (64 errors, 12+
affected jobs). The root cause was the singleton `GoogleDriveService`
caching its HTTP connection, which went stale when Cloud Run containers
sat idle between jobs. The orchestrator caught all exceptions and
continued, so jobs completed with `gdrive_files: {}` and no alert was
sent.

## Test plan
- [x] 5 new retry behavior tests in `test_gdrive_service.py`
(BrokenPipeError, SSLError, give up after 3, service reset, no retry on
non-transient)
- [x] 3 new distribution warnings tests in
`test_video_worker_orchestrator.py`
- [x] All 76 affected tests pass
- [x] Full backend test suite passes
- [x] Backfill script already run successfully for all 12 affected jobs

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 19:47:30 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 23a8b0fb98 Fix missing CDG ZIP packages: fail-fast validation + remediation (#416)
## Summary

- **Systemic fix**: Orchestrator now validates CDG/TXT outputs after
packaging, before encoding. If `enable_cdg=True` but no CDG ZIP was
produced, the job fails immediately with a clear error instead of
silently completing without the package.
- **Remediation**: Created `scripts/regenerate_cdg.py` to regenerate and
distribute CDG/TXT packages for affected jobs (5b6aba25, 5161b069).
Packages uploaded to GCS, GDrive, and Dropbox.
- **Root cause**: Before b671b83f, `instrumental_selection: custom` had
no case in the orchestrator config builder — fell through to "Backing"
suffix, CDG generator looked for wrong filename, `FileNotFoundError` was
silently caught.

## Changes

| File | Change |
|------|--------|
| `backend/workers/video_worker_orchestrator.py` | Post-packaging
validation + LRC-missing check update |
| `backend/tests/test_video_worker_orchestrator.py` | 8 new tests in
`TestPackagingValidation` |
| `scripts/regenerate_cdg.py` | One-time remediation script (GCS +
Firestore + GDrive + Dropbox) |
| `docs/LESSONS-LEARNED.md` | Pipeline stage validation pattern |
| `docs/TROUBLESHOOTING.md` | CDG missing recovery runbook |
| `docs/README.md` | Updated recent changes |
| `pyproject.toml` | Version 0.119.6 → 0.119.7 |

## Test plan

- [x] 8 new tests in `TestPackagingValidation` (CDG fails, TXT fails,
combined CDG+TXT fails, CDG succeeds, CDG disabled, LRC missing raises
with CDG, LRC missing raises with TXT, LRC missing skips without CDG)
- [x] All 2062+ backend tests pass
- [x] Remediation script run successfully for both affected jobs
- [x] Verified CDG/TXT ZIPs in GCS, Firestore, GDrive, and Dropbox

> **CodeRabbit note**: 1 finding (unreachable `else` branch in
`_run_packaging` LRC check) — kept as defensive code since the test
`test_lrc_missing_skips_when_cdg_disabled` exercises this path directly.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 19:13:22 +00:00
Andrew Beveridge ☄️ 028e9f40d4 docs: add TROUBLESHOOTING.md with encoding recovery runbook (#415)
Follows up on #414 — that PR got merged before the force-push landed, so
TROUBLESHOOTING.md was missing.

- Creates `docs/TROUBLESHOOTING.md` with the encoding recovery runbook
- Trims the verbose block from CLAUDE.md down to a one-line pointer
- Adds TROUBLESHOOTING.md to the quick reference table in CLAUDE.md

Docs-only, no code changes.

<!-- @coderabbitai ignore -->
2026-02-27 12:53:45 -05:00
Andrew Beveridge ☄️ c04397d1d1 docs: add stuck encoding job recovery procedure (#414)
Adds a clear recovery procedure to CLAUDE.md for the known scenario
where a Cloud Run deployment kills a poller mid-encoding, leaving a job
stuck at `encoding` status.

**What's added:**
- Diagnosis steps (health endpoint, GCE worker status check)
- Recovery command (re-trigger video worker)
- Note that SSH restart is no longer needed since PR #413 made `/encode`
idempotent

This is purely documentation — no code changes.

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-02-27 12:39:03 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 f404a25328 Make GCE encoding pipeline resilient to deployment interruptions (#413)
## Summary

- Make `/encode` endpoint idempotent on GCE worker — return
cached/in_progress instead of hard 409 reject (mirrors existing
`/encode-preview` pattern)
- Handle cached/in_progress/409 responses in `encoding_service.py` —
`submit_encoding_job()` falls back to status check on 409,
`encode_videos()` returns immediately on cached results
- Detect jobs stuck in `encoding` status >50 min in health service
(encoding timeout is 1 hour)

**Context:** A Cloud Run deployment (PR #411) killed an instance that
was polling the GCE encoding worker for job `a03ad39c`. The encoding
completed on GCE but nobody received the result — the job got
permanently stuck. Recovery required SSH to restart the worker because
`/encode` rejected re-submission with a hard 409.

## Test plan

- [x] 11 new tests for encoding_service.py (cached/in_progress
passthrough, 409 fallback with 5 sub-scenarios, encode_videos
short-circuit)
- [x] 5 new tests for encoding-stuck detection (boundary timing,
non-encoding status, naive datetime handling)
- [x] All 42 new + existing tests pass
- [x] `make test` passes (2055 backend tests, 170 frontend tests)
- [ ] After deploy: restart encoding worker (`sudo systemctl restart
encoding-worker`) to pick up new wheel
- [ ] Verify idempotent behavior: submit same job_id twice to `/encode`

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 21:04:01 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 56fbfd8910 Fix brand code race condition with Firestore atomic counter (#411)
## Summary

- Replace Dropbox folder-scanning brand code allocation with Firestore
atomic counter (`BrandCodeService`)
- Two concurrent jobs received the same brand code (`NOMADNP-0012`)
because `get_next_brand_code()` scanned Dropbox state — a classic TOCTOU
race condition (jobs processed 0.86s apart)
- New service uses `@firestore.transactional` for atomic allocation,
consistent with existing patterns in `rate_limit_service.py`
- E2E cleanup recycles brand codes via `recycle_brand_code()` for reuse
- Counter initializes from Dropbox scan on first use (one-time
migration)

## Changes

- **NEW** `backend/services/brand_code_service.py` — Firestore atomic
counter with allocate, recycle, and parse methods
- **NEW** `backend/tests/test_brand_code_service.py` — 26 unit tests
covering allocation, recycling, initialization, validation
- `backend/workers/video_worker.py` — Use
`brand_code_service.allocate_brand_code()` instead of
`dropbox.get_next_brand_code()`
- `backend/workers/video_worker_orchestrator.py` — Same replacement in
`_run_organization()`
- `backend/api/routes/jobs.py` — Recycle brand code in
`cleanup_distribution()` after Dropbox folder deletion
- `backend/services/dropbox_service.py` — Docstring note that
`get_next_brand_code()` is now only used for initialization
- `backend/tests/test_video_worker_orchestrator.py` — Updated test to
mock new service
- `docs/LESSONS-LEARNED.md` — Added entry about the race condition and
fix

## Test plan

- [x] 26 new brand code service tests pass (allocation, recycling,
initialization, validation, parsing)
- [x] Updated orchestrator test verifies new service is called with
correct args
- [x] Full backend test suite passes (2003 passed)
- [ ] After deploy: process a test job and verify unique brand code
allocated
- [ ] After deploy: check Firestore `brand_code_counters` collection is
created on first allocation

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 14:59:58 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 b671b83f19 Fix custom instrumental ignored during encoding (#410)
## Summary
- Fix critical bug: uploaded custom instrumentals were silently replaced
with clean instrumental during encoding
- GCE encoding worker had no `"custom"` case — fell through to clean
instrumental patterns
- Orchestrator config constructed wrong path (`Backing` instead of
`Custom`) for custom selection
- Upload endpoint now converts all formats to FLAC for pipeline
consistency

## Root Cause
When `instrumental_selection == "custom"`, two downstream components
didn't handle it:
1. **GCE encoding worker** (`gce_encoding/main.py`): only had
`with_backing` and default (clean) cases
2. **Orchestrator** (`video_worker_orchestrator.py`): `"custom" !=
"clean"` evaluated to `"Backing"` suffix

The uploaded `.mp3` file also wouldn't match `*.flac` glob patterns even
with the wildcard fallback.

## Changes
- **backend/services/gce_encoding/main.py**: Add `"custom"` case to
instrumental file search
- **backend/workers/video_worker_orchestrator.py**: Add `"custom"` case
for path construction
- **backend/api/routes/jobs.py**: Convert uploaded audio to FLAC before
storing
- **backend/tests/test_custom_instrumental_route.py**: 10 new regression
tests (43 total)
- **pyproject.toml**: Bump 0.119.3 → 0.119.4

## Test plan
- [x] 43 tests passing (10 new: 5 GCE resolution + 5 orchestrator path)
- [x] Regression test: custom selection does NOT fall back to clean
- [x] Regression test: custom selection does NOT produce Backing path
- [x] Non-FLAC upload triggers FLAC conversion
- [ ] Manual: re-test upload + render for affected jobs

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 01:02:05 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 d7e12e3443 Add custom instrumental upload endpoint with duration validation (#409)
## Summary
- Add missing `POST /api/jobs/{job_id}/upload-instrumental` endpoint
(was returning 404 in production)
- Validate uploaded audio duration matches original ±0.5s using ffprobe
on GCS signed URLs
- Fix frontend bug: map "uploaded" → "custom" selection type at submit
time (API only accepts "custom")

## Changes
- **backend/api/routes/jobs.py**: New upload endpoint +
`_get_audio_duration_ffprobe_signed` helper
- **frontend/components/instrumental-review/InstrumentalSelector.tsx**:
Selection type mapping fix
- **backend/tests/test_custom_instrumental_route.py**: 22 new tests
(upload endpoint, ffprobe helper, contract)
- **docs/API.md**: Document new endpoint
- **docs/README.md**: Add to recent changes
- **pyproject.toml**: Bump version 0.119.2 → 0.119.3

## Test plan
- [x] 22 new tests covering success, auth, state validation, duration
mismatch, boundary cases
- [x] All 33 existing tests continue to pass
- [ ] Manual verification: upload custom instrumental in review UI

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 23:59:59 -05:00
Andrew Beveridge ☄️andClaude Sonnet 4.6 4c55e61cb8 docs: add lessons from private tracks production bugs (#408)
## Summary
- Documents two bugs found during production verification of the private
tracks feature
- **Field mapping gaps**: `is_private` omitted from manual `Job()`
constructor in `create_job()`
- **Return value contract mismatches**: void `update_job()` treated as
boolean, causing all admin PATCH updates to 500

Docs-only change.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 15:04:21 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 6d44089be0 Fix: admin job update always returning 500 (#407)
@coderabbitai ignore

## Summary

Fix admin PATCH endpoint always returning 500 error. Found during
production verification of private tracks feature.

## Root cause

`job_manager.update_job()` returns `None` (no return value), but the
admin PATCH handler checked:
```python
success = job_manager.update_job(job_id, updates)
if not success:  # None is falsy, so this always triggers
    raise HTTPException(status_code=500, ...)
```

## Fix

Changed to `try/except` which properly handles actual errors from the
Firestore update.

## Test plan

- [x] All 25 private tracks tests pass
- [ ] After deploy: verify admin PATCH works for is_private toggle

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 14:22:49 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 b8a107228b Fix: persist is_private flag in job creation (#405)
@coderabbitai ignore

## Summary

- Fix bug where `is_private` was accepted by the API but not persisted
to Firestore
- The `job_manager.create_job()` method was missing
`is_private=job_create.is_private` when constructing the `Job` object

## Root cause

The `create_job` method in `job_manager.py` manually maps fields from
`JobCreate` to `Job`. The `is_private` field was added to both models in
PR #403 but was not added to this mapping, so all jobs were created with
`is_private=False` regardless of the request.

## Changes

- `backend/services/job_manager.py`: Add
`is_private=job_create.is_private` to `Job()` constructor
- `backend/tests/test_private_tracks.py`: Add 2 regression tests
verifying `create_job` persists `is_private`
- `pyproject.toml`: Bump version 0.119.0 → 0.119.1

## Test plan

- [x] New unit tests pass: `test_create_job_persists_is_private`,
`test_create_job_defaults_is_private_false`
- [x] All 25 private tracks tests pass
- [ ] After deploy: create private job via API, verify `is_private=True`
in Firestore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 13:51:13 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 374ce78951 Fix 402 error propagation for credit enforcement (#406)
## Summary

- Fix InsufficientCreditsError being swallowed by catch-all `except
Exception` blocks in route handlers
- Routes were converting 402 errors to 500 because the global exception
handler never saw them

## Root Cause

Route handlers like `search_audio()` wrap `create_job()` in `try/except
Exception` blocks that convert all exceptions to HTTP 500. Our
`InsufficientCreditsError` inherits from `Exception`, so it was caught
and converted before reaching the global
`@app.exception_handler(InsufficientCreditsError)`.

## Fix

Add explicit re-raise for `InsufficientCreditsError` and
`RateLimitExceededError` in all route handlers that call `create_job()`:
- `audio_search.py` - audio search endpoint
- `jobs.py` - direct job creation
- `file_upload.py` - file upload, upload URLs, URL import, finalise-only
(4 endpoints)

## Verified in Production

- Created test user → got 2 welcome credits (confirmed)
- Created job → credit deducted (confirmed)
- Zero credits → got 500 instead of 402 (this bug)
- After this fix → should return 402

## Test plan

- [x] All 13 credit enforcement tests pass
- [ ] Full CI suite
- [ ] Retest 402 in production after deploy

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 13:41:04 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 20453a877c Enforce credit system for karaoke video generation (#404)
## Summary

- Wire up the existing credits system to actually gate job creation -
previously, credits were tracked but never enforced
- Add credit check before job creation, atomic deduction after, and
auto-refund on job failure
- New users now receive 2 welcome credits (was 1)
- Frontend shows zero-credit warning banner, disables submit, and
handles 402 errors with "Buy Credits" link
- Clean up dead credit deduction code from auth_service

## Changes

**Backend:**
- `InsufficientCreditsError` exception with 402 handler
- Credit check + deduction in `JobManager.create_job()` (admin bypass,
atomic deduction, rollback on failure)
- Auto-refund in `JobManager.fail_job()` (skipped for admins,
error-isolated)
- Welcome credits bumped 1 → 2
- Removed dead credit deduction path from
`auth_service.increment_token_usage()`

**Frontend:**
- Zero-credit warning banner at top of job submission form
- Submit buttons disabled when credits = 0
- 402 error handling with "Buy Credits" link
- Boolean state `isCreditError` (not string matching) for credit error
display

**Tests:**
- 13 new backend unit tests covering enforcement, refunds, admin bypass,
atomicity
- 3 new Playwright E2E tests for frontend credit UX
- Tests for 402 exception handler and auth_service cleanup

## Test plan

- [x] Backend unit tests: `pytest tests/test_credit_enforcement.py -v`
(13 tests)
- [x] 402 handler test in `test_main.py`
- [x] Auth service test in `test_auth_firestore.py`
- [x] Frontend E2E tests: `npx playwright test
e2e/regression/credit-enforcement.spec.ts`
- [x] Full suite: `make test` (1928 tests passed)
- [ ] Manual: Create test user with 0 credits → verify 402
- [ ] Manual: Create test user with 1 credit → verify deduction
- [ ] Manual: Fail a job → verify credit refund

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 01:00:06 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 fa9bc8eeeb Add private (non-published) tracks option (#403)
@coderabbitai ignore

## Summary

- Add `is_private` flag to jobs that changes distribution behavior for
non-published tracks
- Private jobs upload to Dropbox `Tracks-NonPublished` folder with
`NOMADNP` brand prefix
- Private jobs skip YouTube and Google Drive uploads entirely
- Completion emails omit YouTube section for private jobs
- Available to all users at job creation; admin can toggle on existing
jobs
- When admin toggles completed job to private, existing outputs are
auto-deleted

## Changes

### Backend
- **Job model**: Added `is_private: bool = False` to `Job` and
`JobCreate`
- **Config**: Added `DEFAULT_PRIVATE_DROPBOX_PATH` and
`DEFAULT_PRIVATE_BRAND_PREFIX` settings
- **Distribution helper**: New `get_effective_distribution_for_job()` in
`job_defaults_service.py` applies private overrides at runtime
- **Orchestrator + native paths**: Both distribution code paths use the
helper (per "Fix Both Sides" lesson)
- **Email template**: YouTube section conditionally removed when
`is_private=True`
- **Credential pre-validation**: Skip YouTube/GDrive credential checks
for private jobs
- **API endpoints**: All 4 job creation endpoints accept `is_private`;
admin can PATCH it
- **Admin auto-delete**: Toggling completed job to private triggers
output deletion

### Frontend
- **Job creation form**: "Private (Non-Published)" checkbox on all 3
tabs (upload, URL, search)
- **Admin dashboard**: Private toggle in job configuration section
- **API interfaces**: Updated TypeScript interfaces and API methods

### Tests
- 23 unit tests covering model, distribution overrides, email template,
notification service, request models, admin toggle, and orchestrator
config
- Fixed MagicMock truthiness issue in existing tests

## Test plan

- [x] All 23 private tracks tests pass
- [x] All 1794 existing backend tests pass (1 flaky unrelated test)
- [x] CodeRabbit review: credential pre-validation issue found and fixed
- [ ] Manual: Create private job end-to-end, verify Dropbox folder
- [ ] Manual: Verify completion email has Dropbox link but no YouTube
section
- [ ] Manual: Admin toggle on completed job deletes outputs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 18:07:57 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 0cee5b8ae3 feat: add beveradb/cryptotrader to WIF and SA IAM binding (#402)
## Summary
- Add `beveradb` org to Workload Identity Federation provider attribute
condition
- Add `beveradb/cryptotrader` repo to the service account IAM binding
for `roles/iam.workloadIdentityUser`
- Syncs Pulumi IaC with GCP drift (changes already applied in GCP)

## Test plan
- [ ] `pulumi preview` shows no drift after merge
- [ ] GitHub Actions from beveradb/cryptotrader can authenticate to GCP

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 19:20:38 +00:00
Andrew Beveridge ☄️andClaude Opus 4.6 9726192de0 fix: persist uploaded audio, fix custom instrumental selection (#401)
## Summary
- **Uploaded audio persistence**: Files uploaded to `uploads/` are
auto-deleted by a 7-day GCS lifecycle rule. After processing, copies are
now saved to `jobs/{job_id}/input/` (mirroring URL job behavior) so
review playback works indefinitely.
- **Defensive cache fallback**: When the transcoding service can't find
the source file, it now checks for an existing OGG cache from earlier
eager transcoding before falling back to FLAC.
- **Custom instrumental selection**: The review completion endpoint
rejected `"custom"` instrumental selection for mute-region edits because
it only checked for an uploaded instrumental file. Now also checks for
the `custom_instrumental` stem created by the mute-region editor.
- **Video worker custom stem**: When `"custom"` is selected and there's
no uploaded instrumental, the video worker now downloads the
`custom_instrumental` stem instead of incorrectly falling through to
`instrumental_with_backing`.

## Test plan
- [x] Unit tests for upload persistence (happy path, no-op for jobs/
prefix, JobManager fallback, persistence failure doesn't block download)
- [x] Unit tests for defensive cache fallback (cache exists when source
deleted, file_exists throws in fallback)
- [x] Unit tests for custom instrumental validation (allowed with stem,
allowed with upload, rejected without either)
- [x] Tightened existing test assertions for new code paths
- [x] All 30 new/modified tests passing

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 02:43:11 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 d22f1893f8 fix: add missing create-custom-instrumental endpoint and prevent oversized Firestore docs (#400)
## Summary
- Add missing `POST /api/jobs/{id}/create-custom-instrumental` endpoint
that was lost during the unified review UI migration. The endpoint
creates a custom instrumental by muting selected backing vocal regions
and returns a signed audio URL for playback.
- Strip `reference_lyrics` from corrections data before saving to
Firestore `state_data`, preventing 1MB document size limit errors when
Genius fetches non-lyrics content (e.g. screenplays). Full data is
preserved in GCS.
- Update frontend `InstrumentalSelector` to use the signed audio URL
returned from the create endpoint instead of a non-existent stream
endpoint.

## Test plan
- [x] 11 new backend tests covering both fixes (happy path, auth,
validation, error handling, Firestore stripping, GCS preservation)
- [x] Full test suite passes (1912 backend, 168 frontend)
- [ ] Manual test: create custom instrumental on a job in review
- [ ] Manual test: submit corrections for a job with oversized Genius
lyrics

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 01:25:02 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 1a56ae09c5 feat: change default jobs list filter from 5 to 10 (#399)
## Summary
- Change the default job display limit from 5 to 10 for new users (or
users who haven't customized the setting)
- Users with a saved preference in localStorage are unaffected

## Test plan
- [x] Frontend unit tests pass (323/323)
- [x] Backend unit tests pass (1920/1920)
- [ ] Verify in production: new/incognito browser shows 10 jobs by
default

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 11:19:38 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 f29fa8b5cd fix: filter exclude_statuses in Python for hide_completed (#398)
## Summary
- Fix `hide_completed=true` query param failing with Firestore composite
index error
- Move `exclude_statuses` filtering from Firestore `not-in` to
Python-side filtering
- Documents are tiny after `select()` projection, so Python filtering is
fast

## Context
Follow-up to #397. The `not-in` filter on `status` combined with
`order_by(created_at)` requires a composite index that doesn't exist.
This moves the filtering to Python after the projected query returns.

## Test plan
- [x] All 33 backend route tests pass
- [ ] Verify `hide_completed=true` works in production (no Firestore
index error)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 09:32:34 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 3ea48e487f feat: server-side jobs filtering with field projection (#397)
## Summary
- Add `fields=summary` query param to `GET /api/jobs` — uses Firestore
`select()` to return only ~12 dashboard-required fields instead of 50+
full Job fields, reducing payload from ~16MB to <500KB
- Add `hide_completed=true` query param for server-side filtering of
terminal statuses (complete, prep_complete, failed, cancelled)
- Safety-net prune helpers strip extra keys from `state_data` and
`file_urls` nested dicts
- Backward compatible: without `fields=summary`, endpoint returns full
`List[Job]` as before (admin pages unaffected)

## Test plan
- [x] 8 backend unit tests covering prune helpers, hide_completed,
exclude_test with dicts, backward compatibility
- [x] 5 frontend Jest tests covering query param serialization and
partial summary response handling
- [x] All 2412 existing tests pass (`make test`)
- [ ] Deploy to staging, verify network tab shows reduced payload size
- [ ] Verify JobCard renders correctly with summary data (title, status,
progress, download links, error)
- [ ] Toggle hide-completed — verify server re-fetch with
`hide_completed=true`
- [ ] Verify admin jobs page (`/admin/jobs`) still works (doesn't send
`fields=summary`)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 09:06:23 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 4fe960481b feat: show all incomplete jobs first + hide-completed toggle (#393)
## Summary
- **Priority sorting**: Blocking + processing jobs always appear first
in Recent Jobs, regardless of creation date
- **Hide completed toggle**: Eye/EyeOff button filters out completed and
failed jobs, showing only incomplete ones
- **Persistent preferences**: Both the display limit dropdown and
hide-completed toggle persist in localStorage across page reloads
- **Oversized lyrics truncation fix**: Corrected screen generation for
songs with unusually long lyric lines

## Changes
- `frontend/lib/job-status.ts`: Added `hideCompleted` parameter to
`getDisplayJobs()` — filters by job priority before applying display
limit
- `frontend/app/app/page.tsx`: Added Eye/EyeOff toggle with tooltip,
localStorage persistence for both `jobLimit` and `hideCompleted`
- `frontend/__tests__/job-status.test.ts`: 5 new tests covering
hideCompleted filtering behavior

## Test plan
- [x] All 168 tests pass (`make test`)
- [x] 85 job-status unit tests pass including 5 new hideCompleted tests
- [ ] Visual verification: toggle button appears next to limit dropdown,
state persists on reload

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 16:22:37 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 fe51369d30 Fix test infrastructure: timeouts, IPv6 hang, and 21 stale E2E tests (#396)
## Summary
- Fix `make test` hanging indefinitely by adding Makefile timeouts and
resolving Playwright IPv6 binding issue
- Update 21 stale Playwright E2E regression tests to match current UI
(buttons renamed, auth mock structure, navigation flow changes)
- Remove dead code: unused admin endpoints, audio transcoding service,
screens worker clear functionality

## Changes

### Test Infrastructure
- **Makefile**: Added 10-min timeout for backend tests, 15-min for
frontend, with proper cleanup on failure
- **Playwright config**: Force IPv4 binding (`127.0.0.1:3000`) to
prevent IPv6 connection hang on macOS

### E2E Test Fixes (21 tests across 4 files)
- **admin-job-detail.spec.ts**: Updated toolbar button selectors
(`Start`→`Audio`, `Lyrics`→`Review`), removed 3 obsolete Clear Workers
tests
- **cloud-mode-review.spec.ts**: Fixed auth mock to match
`UserProfileResponse` nested structure (`{user: {...}, has_session:
true}`), added instrumental-analysis and waveform-data mocks
- **combined-review-corrections.spec.ts**: Fixed auth mock structure,
string path for mock routes, missing auth token parameter
- **lyrics-review.spec.ts**: Updated button text (`Complete
Review`→`Proceed to Instrumental Review`), added `.first()` for
multi-element label selectors, rewrote submit tests to verify navigation
behavior

### Component/Library Fixes
- **TimeInput.tsx**: Added `htmlFor`/`id` association for accessibility
and Playwright `getByLabel()` support
- **local-mode.ts**: Broadened hash check (`hash.startsWith('#/')`) so
invalid hash routes correctly detect cloud mode

### Dead Code Removal
- Removed unused `clear_workers` admin endpoint and related worker code
- Removed unused `audio_transcoding_service.py` and its 353-line test
file

## Testing
- [x] All 168 Playwright tests pass (3 intentionally skipped)
- [x] All backend tests pass
- [x] `make test` completes end-to-end without hanging
- [x] CI passed

## Review
- [x] CodeRabbit CLI review completed locally - clean pass

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 15:19:14 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 0cecc86b79 feat: eager audio transcoding for review UI (#395)
## Summary
- Add `AudioTranscodingService` that transcodes FLAC audio to OGG Opus
128kbps (~3 MB vs 35+ MB), cached in GCS at `review-audio/*.ogg`
- Eager transcoding in `screens_worker` before `AWAITING_REVIEW` so
files are cached before the user opens the review UI
- All 3 review audio paths (`_stream_audio`, `get_correction_data`,
`get_instrumental_analysis`) now serve transcoded OGG URLs with FLAC
fallback
- Admin endpoint `POST /admin/jobs/{job_id}/prepare-review-audio` for
backfilling existing jobs

## Test plan
- [x] 21 unit tests at 100% coverage on `AudioTranscodingService`
- [x] 229 existing review tests pass with 0 regressions
- [ ] Deploy and use admin endpoint to backfill job `4bed6f80`
- [ ] Open review UI — verify audio loads fast, Network tab shows ~3 MB
OGG
- [ ] Check GCS for cached files at `jobs/4bed6f80/review-audio/*.ogg`

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-20 13:24:59 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 87e696556d fix: use signed URL redirect for review audio to bypass 32 MiB limit (#394)
## Summary
- Review audio endpoint was proxying files through Cloud Run via
`FileResponse`, hitting the Google Frontend load balancer's 32 MiB
response body size limit
- Files >32 MiB (like the 35 MiB FLAC for job `4bed6f80`) caused 500
errors — app logged 200 internally but LB rejected the response
- Switched to a 302 redirect to a GCS signed URL, matching the pattern
already used for instrumental stems in the same file

## Test plan
- [x] All 28 review route tests pass
- [ ] Verify audio loads in review UI:
https://gen.nomadkaraoke.com/app/jobs/#/4bed6f80/review

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 22:37:37 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 7ac0e07c95 fix: migrate AudioShake from legacy /upload/ to /assets endpoint (#392)
@coderabbitai ignore

## Summary
- AudioShake removed their legacy `/upload/` endpoint, causing all jobs
to fail with `401 Unauthorized`
- Migrated to the new Assets API (`POST /assets`) which returns an
`assetId`
- Updated task creation to use `assetId` instead of `url` per new API
contract

## Test plan
- [x] Verified new `/assets` endpoint works with existing API token
(returns 200 + asset ID)
- [x] Backend unit tests pass (1081 passed, 8 skipped)
- [ ] CI will run full test suite
- [ ] After merge, verify jobs complete lyrics transcription
successfully in production

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 14:26:56 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 6f08a684d1 fix: close audio search dialog immediately after selection (#390)
## Summary
- The `/api/audio-search/{job_id}/select` endpoint was downloading audio
synchronously before returning, keeping the dialog open for 10-60+
seconds
- Refactored into `_validate_and_prepare_selection()` (returns
instantly) and `_download_audio_and_trigger_workers()` (runs as FastAPI
background task)
- Dialog now closes immediately; download proceeds in background with
job status updates visible in the job card

## Test plan
- [x] All 57 backend tests pass (6 new tests added for the refactored
functions)
- [x] Error paths tested: invalid index, job not found, no results,
download failures
- [x] Background task gracefully handles missing jobs and generic
exceptions
- [x] Worker triggering verified after successful download
- [ ] Manual: select audio in popup, verify it closes immediately and
job transitions to "Downloading audio..."

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-17 12:00:16 -05:00
Andrew Beveridge ☄️andClaude Opus 4.6 2f15cfc985 feat(lyrics-review): Add Replace Segment Lyrics mode (#389)
## Summary

- Adds a third mode ("Replace Segment Lyrics") to the Edit All Lyrics
modal, enabling users to swap custom lyrics line-by-line while
preserving existing segment timing
- Extracts `createWordsWithDistributedTiming` shared utility from
EditModal for reuse
- Fixes Makefile `build-frontend` to clean stale output before copying
(was causing old JS chunks to persist)
- Restores accidentally deleted `nextjs_frontend/__init__.py`

## Test plan

- [x] Unit tests for `createWordsWithDistributedTiming` (8 cases)
- [x] Component tests for Replace Segments mode (17 cases)
- [x] E2E Playwright tests (8 cases) 
- [x] All 306 Jest tests pass
- [x] Manual local testing with real audio file via `karaoke-gen` CLI

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-12 15:39:22 -05:00
Andrew Beveridge ☄️andClaude Sonnet 4.5 9cca4db147 fix(lyrics-review): Enable and improve mode selector buttons (#388)
## Summary
- Fixed non-functional highlight and delete mode buttons in lyrics
review
- Improved mode selector UX with better spacing, icons, and active state
visibility
- Added test infrastructure support for Radix UI components

## Changes
- **ModeSelector.tsx**: Removed `disabled` attributes and fixed onChange
handler to support all modes
- **ModeSelector.tsx**: Enhanced visual design with proper active states
(blue background on selected mode)
- **ModeSelector.tsx**: Improved spacing consistency and changed Edit
icon to Pencil for better visual balance
- **ModeSelector.test.tsx**: Added comprehensive unit tests (100%
coverage)
- **jest.setup.js**: Added ResizeObserver mock for Radix UI ToggleGroup
component
- **pyproject.toml**: Bumped version to 0.116.3

## Testing
- [x] All tests pass (281 frontend + 103 backend tests)
- [x] Unit test coverage: 100% on ModeSelector component
- [x] Manual testing completed via Chrome DevTools MCP
- [x] E2E regression tests pass (existing lyrics review tests)

## Review
- [x] Manual code review completed (CodeRabbit CLI couldn't run due to
447 built files)
- [x] No security, bug, or quality issues found
- [x] Documentation reviewed (no updates needed)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-11 21:41:49 -05:00
Andrew Beveridge ☄️andClaude Sonnet 4.5 1852df9273 fix(gce-encoding): Fix wheel version sorting regression (0.99.9 > 0.116.1) (#387)
## Summary

Critical regression fix from PR #384: Production encoder was running
v0.99.9 instead of v0.116.1 due to alphabetical sorting instead of
semantic version comparison.

**Impact:** GCE encoding worker was using an extremely old version
(0.99.9 from months ago) instead of the latest release (0.116.1),
potentially causing encoding issues or missing recent bug fixes.

## Root Cause

PR #384 fixed invalid wheel filtering but exposed a latent version
sorting bug:
- `sorted(wheels)[-1]` uses string comparison, not semantic versioning
- String comparison: `"0.99.9"` > `"0.116.1"` (because '9' > '1' in
position 2)
- Result: Old version selected over new version

## Changes

**Code (14 lines):**
- Added imports: `re`, `packaging.version.Version`
- Implemented `extract_version()` helper to parse version from filename
- Changed sorting from `sorted(wheels)[-1]` to semantic version sort
- **File:** `backend/services/gce_encoding/main.py`

**Tests (159 lines, 10 new tests):**
- Test exact regression case (0.99.9 vs 0.116.1)
- Demonstrate alphabetical sorting failure
- Test version extraction and edge cases
- Test complex version comparisons (major/minor/patch)
- **File:** `backend/tests/test_gce_encoding_worker.py`

**Documentation:**
- Updated `docs/README.md` with Recent Changes entry
- Added lesson to `docs/LESSONS-LEARNED.md` about semantic version
sorting
- Pattern documented for future reference

**Version:**
- Bumped to v0.116.2 (patch release for bug fix)

## Testing

- [x] All tests pass (24 total, 10 new)
- [x] Test-to-code ratio: 11.4:1
- [x] Edge cases covered (empty list, invalid filenames, complex
versions)
- [x] CodeRabbit review: 0 issues found

## Review

- [x] Local CodeRabbit CLI review completed
- [x] Documentation updated
- [x] All tests passing

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-11 12:38:44 -05:00
Andrew Beveridge ☄️ 30e55a0031 docs: Add GCP cost optimization analysis (#386)
Comprehensive cost analysis document covering:

- Current spend breakdown (~k/month) 
- Implemented GitHub runner auto-scaling (,390/month saved)
- Additional optimization opportunities (Cloud Run, encoding worker)
- Actionable next steps

Document URL will be: `docs/GCP-COST-OPTIMIZATION.md`

<!-- @coderabbitai ignore -->
2026-02-11 17:28:55 +00:00
Andrew Beveridge ☄️andClaude Sonnet 4.5 8b88f43a96 fix(gce-encoding): Fix wheel installation crash by filtering invalid filenames (#384)
## Summary

Fixes GCE encoding worker crash caused by attempting to install
`karaoke_gen-current.whl`, which is not a valid PEP 427 wheel filename.

## Problem

Job 749141f8 failed with "Cannot connect to host" errors. Investigation
revealed:
- GCE encoding worker was crashing and restarting
- Root cause: pip rejected `karaoke_gen-current.whl` with "Invalid wheel
filename (wrong number of parts)"
- The `ensure_latest_wheel()` function downloaded ALL wheels including
the `-current` convenience file
- This file lacks required PEP 427 naming (version, python, abi,
platform tags)

## Solution

Filter out wheels containing '-current' before selecting which to
install:
```python
wheels = glob.glob("/tmp/karaoke_gen-*.whl")
# Filter out karaoke_gen-current.whl (not a valid PEP 427 wheel name)
wheels = [w for w in wheels if '-current' not in w]
```

## Changes

- **Fix** (`backend/services/gce_encoding/main.py`): Add 3-line filter
for valid wheel names
- **Tests** (`backend/tests/test_gce_encoding_worker.py`): Add 5
comprehensive unit tests
  - Current wheel exclusion
  - Empty result handling
  - Version sorting after filtering
  - Pass-through when no current wheel
  - Path-agnostic filtering
- **Docs** (`docs/README.md`): Add to Recent Changes
- **Docs** (`docs/LESSONS-LEARNED.md`): Add pip wheel filename
validation lesson
- **Version**: Bump to 0.116.1 (patch release)

## Testing

- ✅ All 14 GCE encoding worker tests pass (9 existing + 5 new)
- ✅ Backend tests pass
- ✅ Test-to-code ratio: 26:1 (78 test lines for 3 code lines)
- ✅ Tests cover all edge cases and are isolated/fast
- ⚠️ Frontend E2E tests have pre-existing failures (unrelated to backend
changes)

## Review

- ✅ CodeRabbit CLI review completed locally (zero issues found)
- ✅ Code follows project conventions
- ✅ Comprehensive test coverage
- ✅ Documentation updated

## Impact

- Fixes job 749141f8 and prevents future encoding failures
- No breaking changes
- No API changes
- Production-critical bug fix

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-10 16:48:25 -05:00
Andrew Beveridge ☄️ f313fc4208 fix(infra): Update runner manager memory to 512M in config (#385)
Aligns code with deployed infrastructure. Function was seeing OOM with
256M.

<!-- @coderabbitai ignore -->
2026-02-10 16:41:24 -05:00
Andrew Beveridge ☄️ 15d5ed8db1 chore(infra): Trigger deployment for runner autoscaling (#383)
Trigger infrastructure deployment with Cloud Functions admin permission
fixed.

<!-- @coderabbitai ignore -->
2026-02-10 20:52:58 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 fb40b4ac05 feat(infra): Add auto-scaling GitHub runners with Spot VMs (#376)
## Summary

Implement cost optimization for GitHub Actions self-hosted runners:

- Reduce runners from 20 → 3 (auto-scaling handles demand)
- Switch to Spot/Preemptible instances (60-91% cheaper)
- Add Cloud NAT for outbound internet (no external IPs needed)
- Add runner_manager Cloud Function for auto-start/stop

## Cost Impact

| Item | Before | After | Savings |
|------|--------|-------|---------|
| GitHub Runners | 20 VMs × $72/mo = $1,440 | 3 Spot VMs × ~$25 × ~20%
util = ~$50-100 | **~$1,300-1,400/mo** |
| External IPs | 20 × $3 = $60 | $0 (Cloud NAT) | $50/mo |

## How It Works

1. **GitHub sends webhook** → `workflow_job.queued` event
2. **Cloud Function** receives webhook, starts any TERMINATED runner VMs
3. **VMs boot**, register with GitHub, pick up jobs
4. **Cloud Scheduler** triggers idle check every 15 minutes
5. **If no pending jobs and idle > 1 hour** → VMs are stopped

## Manual Setup Required (Post-Deploy)

1. **Create webhook secret:**
```bash
python -c "import secrets; print(secrets.token_hex(32))" | \
  gcloud secrets versions add github-webhook-secret --data-file=- --project=nomadkaraoke
```

2. **Configure GitHub org webhook:**
   - Go to: https://github.com/organizations/nomadkaraoke/settings/hooks
   - Payload URL: `pulumi stack output runner_manager_function_url`
   - Content type: `application/json`
   - Secret: (value from step 1)
   - Events: Select "Workflow jobs"

## Test Plan

- [ ] Pulumi preview succeeds
- [ ] Deploy to prod
- [ ] Configure webhook secret
- [ ] Configure GitHub webhook
- [ ] Trigger a CI workflow, verify runners start
- [ ] Wait 1+ hour with no activity, verify runners stop

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-10 15:43:30 -05:00
Andrew Beveridge ☄️andClaude Sonnet 4.5 716936c8a9 Add retry logic for flacfetch download failures and audio search recovery (#382)
## Summary

Adds comprehensive retry logic to handle transient flacfetch network
failures, preventing permanent job failures from temporary service
issues. Includes state persistence for retry capability, enhanced retry
endpoint for audio search jobs, and user-facing error feedback.

## Key Changes

**Retry Logic (backend/services/flacfetch_client.py):**
- Manual exponential backoff: 9 attempts over ~6 minutes (10s → 20s →
40s → 60s capped)
- Selective retry: retries on connection errors and 5xx, immediate fail
on 4xx
- Runtime-configurable via environment variables
(FLACFETCH_RETRY_MAX_ATTEMPTS, etc.)

**State Persistence (backend/api/routes/audio_search.py):**
- Save download parameters (audio_source_type, source_name, source_id,
target_file, download_url) BEFORE attempting download
- Enables retry even if download fails completely
- Added 5 new fields to Job model (backend/models/job.py)

**Enhanced Retry Endpoint (backend/api/routes/jobs.py):**
- Added ~150 lines handling audio search retry cases
- Re-downloads using saved parameters for YouTube, RED, OPS, Spotify
sources
- Clears error state and transitions to DOWNLOADING before triggering
workers

**User Feedback (frontend/components/job/JobActions.tsx):**
- Error toast shows actual API error message on retry failure
- Success toast confirms retry started
- No more silent failures

**Configuration (backend/config.py):**
- Added retry config: max_attempts (9), min_wait (10s), max_wait (60s)

## Testing

- [x] Added 5 comprehensive retry tests in
`backend/tests/test_flacfetch_client.py`
  - Network error retry with exponential backoff
  - No retry on 4xx client errors
  - Retry on 5xx server errors
  - Max retries exhausted
  - Wrapper consistency
- [x] All backend tests pass: 103 passed
- [x] Frontend E2E tests: 142 passed (21 pre-existing failures unrelated
to changes)

## Review

- [x] CodeRabbit CLI review completed locally
- [x] Fixed unused import identified in review
- [x] Documentation updated (README.md Recent Changes,
LESSONS-LEARNED.md retry patterns)
- [x] Implementation plan archived to
docs/archive/2026-02-10-flacfetch-retry-logic.md

## Root Cause

Job 749141f8 failed because flacfetch network call failed during audio
search, and no source info was saved, making retry impossible. This PR
fixes all four related issues identified during investigation.

## Version

Bumped 0.115.5 → 0.116.0 (minor version for new feature)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-10 14:21:14 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 89f49ffd48 Fix CDG sync issue: strip countdown padding for CDG output (#380)
## Summary

- Fix CDG+MP3 package sync issue where lyrics appeared 3 seconds early
- The CDG was using padded LRC timestamps with unpadded instrumental
audio
- CDGGenerator now strips countdown segment and adjusts timestamps when
needed

## The Bug

The MP4 video has a 3-second countdown ("3... 2... 1...") prepended, and
LRC timestamps are shifted forward by 3 seconds to match. However, the
CDG+MP3 package uses the original instrumental audio (no countdown), but
was using the shifted timestamps.

Result: CDG lyrics appeared 3 seconds too early.

## The Fix

1. Added `lrc_has_countdown_padding` flag to `OrchestratorConfig`
2. Flag is read from job's `lyrics_metadata.has_countdown_padding`
3. Flag is passed through `PackagingService` to `CDGGenerator`
4. When True, `CDGGenerator` strips the countdown segment from LRC and
shifts all timestamps back by 3 seconds

## Test Plan

- [x] Added 11 unit tests for countdown removal logic
- [x] Added packaging service integration test
- [x] Added orchestrator countdown pipeline tests
- [x] All 103 emulator tests pass
- [x] Verified manually with playable CDG+MP3 output

## Files Changed

- `karaoke_gen/lyrics_transcriber/output/cdg.py` - Core fix
- `backend/workers/video_worker_orchestrator.py` - Pipeline changes
- `backend/services/packaging_service.py` - Param passing
- `backend/tests/test_cdg_countdown.py` - New test file
- `backend/tests/test_video_worker_orchestrator.py` - Pipeline tests
- `docs/LESSONS-LEARNED.md` - Documented the pattern
- `docs/README.md` - Added to known issues

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

@coderabbitai ignore

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-05 12:18:23 -08:00
Andrew Beveridge ☄️andClaude Opus 4.5 b7c2ae185f Allow admin restart from in_review state (#381)
## Summary

Fixes 400 error when attempting to restart a job that's in `in_review`
state.

## Problem

Job 1eab1172 failed to restart with error:
```
400: Cannot restart job in 'in_review' state. 
Allowed states: awaiting_review, ...
```

The job was in `in_review` state (user had opened the review UI), but
the restart endpoint only allowed `awaiting_review` (waiting for review
to start).

## Root Cause

`RESTART_ALLOWED_STATES` in `backend/api/routes/admin.py:2036` didn't
include `in_review` state.

**State distinction:**
- `awaiting_review` - Job waiting for user to open review UI
- `in_review` - User has opened review UI and is actively reviewing

## Solution

Add `in_review` to `RESTART_ALLOWED_STATES`:

```python
RESTART_ALLOWED_STATES = {
    "pending",
    "complete",
    "failed",
    "awaiting_review",
    "in_review",  # ← Added
    "awaiting_audio_selection",
    "awaiting_instrumental_selection",
    "instrumental_selected",
    "prep_complete",
}
```

## Changes

- `backend/api/routes/admin.py` - Add `in_review` to allowed states
- `pyproject.toml` - Bump to 0.115.5

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-05 12:17:32 -08:00
Andrew Beveridge ☄️andClaude Opus 4.5 af4b3a7c29 Enable AudioShake language auto-detection for international lyrics (#379)
## Summary

Fixes lyrics translation issue where non-English songs (like Russian)
were being translated to English instead of transcribed in their
original language.

## Problem

Job 1eab1172 with Russian lyrics "я куплю тебе дом" was transcribed by
AudioShake, but the output was in English instead of Russian. The
transcription API was translating the lyrics rather than preserving the
original language.

## Root Cause

`karaoke_gen/lyrics_transcriber/transcribers/audioshake.py:152` was
hardcoding `"language": "en"` in the AudioShake API request, which
forced English output regardless of the audio's actual language.

## Solution

Remove the hardcoded language parameter to enable AudioShake's built-in
language auto-detection:

```python
# Before (forced English):
"targets": [{
    "model": "alignment",
    "formats": ["json"],
    "language": "en"  # ← Forced English
}]

# After (auto-detect):
"targets": [{
    "model": "alignment",
    "formats": ["json"]
    # Language omitted - AudioShake auto-detects from 100+ languages
}]
```

## Testing

Verified with direct AudioShake API test:
- Created task WITHOUT language parameter → HTTP 200 ✓
- Response shows `language: null` → Auto-detection enabled ✓
- AudioShake supports 100+ languages including Russian, Chinese,
Spanish, Arabic, etc.

## Impact

- ✅ Russian, Chinese, and other non-English songs will now be
transcribed correctly
- ✅ No configuration needed - works globally with all languages
- ✅ Existing English songs continue to work (auto-detected as English)

## Changes

- `karaoke_gen/lyrics_transcriber/transcribers/audioshake.py` - Remove
hardcoded language parameter
- `docs/LESSONS-LEARNED.md` - Document pattern for API integration
- `pyproject.toml` - Bump to 0.115.4

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-05 11:45:23 -08:00
Andrew Beveridge ☄️andClaude Sonnet 4.5 67cd6494b4 Fix admin restart state transition for preserve_audio_stems mode (#377)
## Summary

Fixes a state machine validation error when using the admin "Full
Restart" button with preserve_audio_stems mode.

## Problem

When restarting job 1eab1172 after the Unicode fix (PR #375), the
restart failed with:
```
Invalid state transition for job 1eab1172: transcribing -> JobStatus.GENERATING_SCREENS. 
Valid transitions: ['correcting', 'failed']
```

## Root Cause

The restart endpoint was setting the job status to `transcribing`, then
triggering the screens worker. The screens worker immediately tries to
transition to `GENERATING_SCREENS`, but `transcribing ->
generating_screens` is not a valid transition according to the state
machine in `backend/models/job.py`.

## Fix

Changed the restart endpoint to set status to `downloading` instead of
`transcribing`. According to STATE_TRANSITIONS:
- ✅ `downloading` → `generating_screens` (valid)
- ✅ `audio_complete` → `generating_screens` (valid)
- ✅ `lyrics_complete` → `generating_screens` (valid)
- ❌ `transcribing` → `generating_screens` (invalid)

Using `downloading` is the safest choice as it's the most flexible entry
state.

## Changes

- `backend/api/routes/admin.py:2161` - Change status from "transcribing"
to "downloading"
- `docs/LESSONS-LEARNED.md` - Added lesson about validating state
transitions in admin operations
- `pyproject.toml` - Bump version to 0.115.3

## Testing

- Existing tests pass (8 pre-existing emulator test failures unrelated
to this change)
- Will verify in production by restarting job 1eab1172 after deployment

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-05 18:32:46 +00:00
Andrew Beveridge ☄️andClaude Sonnet 4.5 d9bb8b3181 fix: sanitize Unicode in temp file paths (FFmpeg exit 183) (#375)
## Summary
Fixes job 1eab1172 which failed with FFmpeg exit code 183 due to
Cyrillic characters in temp subtitle file paths. Python's
`str.isalnum()` returns True for ALL Unicode alphanumeric characters
(Cyrillic, Arabic, Chinese, emoji, etc.), not just ASCII.

## Root Cause
- Job `1eab1172` with Russian title "я куплю тебе дом" failed
- Temp subtitle path contained Cyrillic:
`temp_subtitles_zivert___я_куплю_тебе_дом_1770150285852.ass`
- FFmpeg cannot handle Unicode in filter argument paths → exit code 183

## Changes
- **karaoke_gen/lyrics_transcriber/output/video.py** (lines 222, 282):
- Changed `c.isalnum()` to `c.isascii() and c.isalnum()` for temp
subtitle paths
  - Affects both `generate_video()` and `generate_preview_video()`

- **karaoke_gen/utils/remote_cli.py** (line 1970):
- Changed `c.isalnum()` to `c.isascii() and c.isalnum()` for output
folder names
- Added fallback to `job_{job_id}` if folder name is empty or contains
no alphanumeric chars

- **Tests added** (21 comprehensive tests):
- `test_video_generator_unicode.py` (10 tests): Cyrillic, Chinese,
Arabic, emoji, accented chars
- `test_remote_cli_unicode_folders.py` (11 tests): Folder sanitization,
edge cases, real failure case

- **Documentation**:
- Updated `docs/LESSONS-LEARNED.md` with "Unicode in File Paths (Feb
2026)" section
  - Updated `docs/README.md` with recent changes
  - Bumped version 0.115.1 → 0.115.2

## Testing
- [x] All 1088 unit tests pass (68.91% coverage, target 69%)
- [x] 21 new comprehensive Unicode tests covering:
  - Cyrillic, Chinese, Arabic, emoji, accented characters
  - Empty folder name edge case
  - Separator-only folder name edge case
  - Exact real-world failure case from job 1eab1172
- [x] Verified temp file paths are ASCII-only
- [x] Verified folder names safe for all filesystems

## Review
- [x] CodeRabbit CLI review completed locally (2 cycles)
- [x] All identified issues addressed:
  - Empty folder name fallback added
  - Separator-only folder name guard enhanced
- [x] Test coverage maintained at target

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-05 09:13:33 -08:00
Andrew Beveridge ☄️andClaude Sonnet 4.5 2b1e4ead00 fix(admin): perform audio search in override-audio-source endpoint (#374)
## Summary

Fixes the override-audio-source endpoint which was transitioning jobs to
`awaiting_audio_selection` without actually performing the search,
leaving jobs stuck with no results to select from.

## Changes

- Now performs actual audio search after clearing state
- Stores search results in state_data before transitioning
- Updated response model to include `search_results_count` and `error`
fields
- Updated API docs to reflect new behavior

## Test plan

- [x] Ran test_admin_job_reset.py - all 39 tests pass
- [x] Manually tested on job 284ed6a6 (populated search results)

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-04 22:22:41 -08:00
Andrew Beveridge ☄️andClaude Sonnet 4.5 b7b5d5c814 feat(admin): add MFY restart and override admin tools (#373)
## Summary

Add three new admin endpoints for better Made-For-You order management,
plus clean up the admin toolbar to remove confusing low-level buttons.

### New Endpoints

1. **`POST /api/admin/jobs/{job_id}/regenerate-screens`**
   - Regenerates title/end screens with current artist/title metadata
- Use after editing artist/title to update screens without full
reprocessing

2. **`POST /api/admin/jobs/{job_id}/restart`**
- Full restart that actually triggers workers (fixes the "stuck in
pending" issue)
   - Option to preserve audio stems for faster restart
   - Handles YouTube, audio search, and file upload jobs appropriately

3. **`POST /api/admin/jobs/{job_id}/override-audio-source`**
   - Switch from YouTube URL to audio search mode
   - Allows finding higher quality audio for MFY orders

### Frontend Changes

- Add new toolbar buttons: **Regen Screens**, **Full Restart**, **Audio
Search**
- Remove confusing low-level buttons that could leave jobs in broken
states:
  - "Reset to Start" (replaced by "Full Restart")
  - "Clear Workers" (low-level escape hatch)
  - "Trigger" (low-level escape hatch)
- Add confirmation dialogs with clear explanations

### Use Case

When a customer submits a Made-For-You order with:
- Wrong artist/title → Edit fields, click "Regen Screens"
- Low-quality YouTube URL → Click "Audio Search" to switch to audio
search mode
- Job stuck after reset → Click "Full Restart" to actually trigger
workers

## Test plan

- [ ] Verify new endpoints return proper responses
- [ ] Verify "Regen Screens" regenerates screens with updated metadata
- [ ] Verify "Full Restart" with preserve_audio_stems=true triggers
screens worker
- [ ] Verify "Full Restart" with preserve_audio_stems=false re-downloads
YouTube audio
- [ ] Verify "Audio Search" transitions job to awaiting_audio_selection
- [ ] Verify old confusing buttons are removed from toolbar

🤖 Generated with [Claude Code](https://claude.ai/claude-code)

@coderabbitai ignore

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-04 18:57:17 -08:00
Andrew Beveridge ☄️andClaude Opus 4.5 747f7cf079 test: fix mock return values to match real API contract (#372)
## Summary
- Fixed 22 test mocks that incorrectly returned `True` instead of `None`
for `update_job()`
- Added emulator integration tests proving the real API contract
- Added test using real production job 984da08b as fixture to validate
the fix
- Documented lesson learned and added mock guidance to TESTING.md

## Context
PR #371 fixed an admin reset endpoint that always returned 500. The root
cause was that tests mocked `update_job()` to return `True`, but the
real method returns `None` on success. Code checking `if not success:`
was always `True` with `None`.

This PR ensures our tests would have caught the bug before it shipped
by:
1. Fixing all mock return values to match the real API contract
2. Adding integration tests that prove `update_job()` returns `None`
3. Documenting this pattern to prevent future occurrences

## Test plan
- [x] All 38 admin reset unit tests pass
- [x] New emulator integration tests verify API contract
- [x] Real production job fixture test validates the fix

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 16:37:10 -08:00
Andrew Beveridge ☄️andClaude Opus 4.5 2f46b3754d fix: admin reset endpoint always returned 500 (#371)
## Summary

The admin job reset endpoint (`POST /api/admin/jobs/{job_id}/reset`)
always returned a 500 error due to a bug introduced in PR #370.

**Root cause:** The code checked `if not success:` but `update_job()`
returns `None` (not a boolean). Since `not None` is `True`, this always
raised the HTTPException.

**Fix:** Removed the erroneous check. `update_job()` raises exceptions
on failure, so checking the return value is unnecessary.

## Test plan

- [x] All 38 admin reset tests pass locally
- [x] All 100 emulator integration tests pass locally
- [ ] Verify reset button works in production admin dashboard

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- @coderabbitai ignore -->

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 15:42:26 -08:00
Andrew Beveridge ☄️andClaude Opus 4.5 5901137724 feat: make state machine transitions robust with explicit failure handling (#370)
## Summary

Jobs 06cfea29 and 984da08b got stuck at `pending` status because state
transitions failed silently. This comprehensive fix makes the state
machine more robust through several improvements:

- **Explicit failure handling**: `transition_to_state()` now raises
`InvalidStateTransitionError` by default instead of returning False
silently
- **Centralized processing pattern**: New `start_job_processing()`
helper ensures atomic transition + worker trigger across all handlers
- **Worker validation**: All workers now validate job status before
processing via `validate_worker_can_run()`
- **Runtime consistency checks**: New `job_health_service.py` detects
stuck jobs and state inconsistencies
- **Health endpoint**: `/health/job-consistency` endpoint for admin
dashboard integration
- **Admin button fixes**: Reset to pending now clears parallel
processing flags; removed deprecated states

## Changes

### Core Services
- `services/job_manager.py`: Added `InvalidStateTransitionError`,
`start_job_processing()`, and updated `transition_to_state()` to raise
by default
- `services/job_health_service.py`: New service for consistency checks
and worker validation
- `exceptions.py`: Added `InvalidStateTransitionError` exception class

### API Routes  
- `routes/health.py`: Added `/health/job-consistency` endpoint
- `routes/admin.py`: Updated STATE_DATA_CLEAR_KEYS and
ALLOWED_RESET_STATES
- `routes/audio_search.py`: Fixed to properly transition from
DOWNLOADING_AUDIO to DOWNLOADING
- `routes/users.py`: Added error handling for state transition failures
- `routes/file_upload.py`: Uses centralized `start_job_processing()`

### Workers
- All workers (audio, lyrics, screens, video, render_video): Added
status validation via `validate_worker_can_run()`

### Frontend
- `admin/jobs/page.tsx`: Replaced "Lyrics" + "Inst." buttons with
combined "Review" button

### Tests
- 26 tests for job_health_service
- 11 integration tests for state machine flow
- 4 tests for job-consistency endpoint
- Updated existing tests for new exception behavior

### Documentation
- `docs/LESSONS-LEARNED.md`: Added "Make State Transitions Loud, Not
Silent" entry
- `docs/ARCHITECTURE.md`: Added state machine robustness note
- `docs/archive/2026-02-02-state-machine-robustness-plan.md`: Complete
implementation plan

## Test plan

- [x] All 1850+ unit tests pass
- [x] CodeRabbit review issues addressed
- [ ] Manual verification of admin reset buttons in staging
- [ ] Verify stuck jobs are properly detected by
`/health/job-consistency`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- @coderabbitai ignore -->

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-02 22:48:57 -08:00
Andrew Beveridge ☄️andClaude Opus 4.5 b0cbd28c8d fix(infrastructure): Import existing Firestore index into Pulumi state (#369)
## Summary
- Adds `import_` directive to the `firestore-index-jobs-customer-mfy`
resource to import the existing index in GCP into Pulumi state
- The index was created before Pulumi managed it, causing a 409 conflict
error on deploy

## Context
PR #367 added this index definition, but the index already existed in
GCP (created manually). This caused the infrastructure deployment to
fail with:
```
Error creating Index: googleapi: Error 409: index already exists
```

The `import_` directive tells Pulumi to adopt the existing resource
rather than trying to create a new one.

## Test plan
- [ ] CI passes
- [ ] Pulumi successfully imports the existing index on deploy

> [!NOTE]
> After this PR deploys successfully, the `import_` line can be removed
in a follow-up PR since the resource will already be in Pulumi state.

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-29 22:33:26 -08:00
Andrew Beveridge ☄️andClaude Opus 4.5 c8d3fa46c4 fix: update E2E tests for combined lyrics/instrumental review UI (#368)
## Summary
- Updates production E2E tests to work with the combined
lyrics/instrumental review UI
- The "Complete Review" button was changed to "Proceed to Instrumental
Review" in a previous update
- Instrumental selection now happens on the same page via hash
navigation instead of a separate page

## Changes
- **happy-path-real-user.spec.ts**: 
- Step 7 now clicks "Proceed to Instrumental Review" instead of
"Complete Review"
- Step 8 handles instrumental selection on the same page (via hash
navigation)
- Removed the old flow that waited for job status change and opened a
new page
- **lyrics-review-only.spec.ts**: Updated to use new button name
- **test-existing-job.spec.ts**: Updated to use new button name

## Test plan
- [x] Verified test passes locally (6.2 minutes)
- [x] Full flow tested: signup → job creation → audio selection → lyrics
review → instrumental selection → completion → cleanup

## Notes
This fixes the daily scheduled E2E test that has been failing since the
UI was combined.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

@coderabbitai ignore

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-29 22:21:06 -08:00
Andrew Beveridge ☄️andClaude Opus 4.5 6ebf076576 fix: E2E test issues - add Firestore index and fix job endpoint URL (#367)
## Summary
- Add missing Firestore composite index for `customer_email +
made_for_you + created_at` queries (required for test-webhook
made-for-you order processing)
- Fix `getJobDetails` helper to use `/api/jobs/{job_id}` instead of
`/api/admin/jobs/{job_id}` (the admin route doesn't have a GET method)

## Test plan
- [x] All production E2E tests pass:
  - `stripe-webhook-security.spec.ts` (9 passed)
  - `stripe-credit-purchase.spec.ts` (3 passed)
  - `stripe-made-for-you-search.spec.ts` (2 passed, 1 skipped)
  - `stripe-made-for-you-youtube.spec.ts` (3 passed)

Note: The Firestore index was already created manually via gcloud. This
PR adds it to Pulumi IaC so it's tracked in infrastructure as code.

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-29 21:54:35 -08:00
1255 changed files with 231361 additions and 12516 deletions

No files matched your search

@@ -0,0 +1,109 @@
# Plan: Admin Feedback Visibility
**Created:** 2026-03-23
**Branch:** feat/sess-20260323-1454-feedback-email-debug
**Status:** Draft
## Overview
User feedback submissions are stored in Firestore (`user_feedback` collection) but are completely invisible to the admin. The admin panel only shows a boolean "Feedback Submitted: Yes/No" badge per user. This plan adds:
1. **Admin UI page** to browse and review all submitted feedback
2. **Email notification** to admin@nomadkaraoke.com on each new submission
## Requirements
- [ ] Admin page at `/admin/feedback` listing all feedback with ratings, text, timestamps, and user info
- [ ] Sortable by date (newest first by default) and filterable by search (email)
- [ ] Star ratings rendered visually (not just numbers)
- [ ] Email sent to admin@nomadkaraoke.com immediately when feedback is submitted
- [ ] Email includes all feedback content (ratings, text fields, user email)
- [ ] Test data filtering works (exclude_test toggle)
- [ ] Sidebar navigation includes new "Feedback" item
## Technical Approach
### Admin API Endpoint
Add `GET /api/admin/feedback` to `backend/api/routes/admin.py`. Follows the standard admin endpoint pattern: `require_admin` dependency, pagination with `limit`/`offset`/`search`/`exclude_test`, returns Pydantic response model. Queries `user_feedback` Firestore collection.
### Admin UI Page
New page at `frontend/app/admin/feedback/page.tsx`. Follows existing admin page patterns (Card, Table, Badge, pagination, search input, loading states). Uses `adminApi.listFeedback()` to fetch data.
### Email Notification
Add a new method to `EmailService` (or call `send_email` directly) from the `submit_user_feedback` endpoint in `users.py`. Uses the existing SendGrid infrastructure — no new template service templates needed since this is an internal admin notification (simple HTML table with the feedback data).
## Implementation Steps
### Step 1: Backend — Admin feedback list endpoint
1. [ ] Add `AdminFeedbackItem` and `AdminFeedbackListResponse` Pydantic models to `admin.py`
2. [ ] Add `GET /api/admin/feedback` endpoint with:
- Pagination: `limit=50`, `offset=0`
- Search by user email
- `exclude_test` filter (reuse `is_test_email`)
- Sort by `created_at` descending
- Return: items, total, offset, limit, has_more
3. [ ] Add aggregate stats in response: average ratings, total count
### Step 2: Backend — Email notification on submission
4. [ ] Add `send_feedback_notification` method to `EmailService` that sends an HTML email to `admin@nomadkaraoke.com` containing:
- User email
- All 4 star ratings
- All text fields
- Would recommend / would use again
- Submitted timestamp
- Link to admin feedback page
5. [ ] Call `send_feedback_notification` from `submit_user_feedback` in `users.py` (fire-and-forget, don't block the user response on email success)
### Step 3: Frontend — Admin API client
6. [ ] Add `listFeedback` method to `adminApi` in `frontend/lib/api.ts`
7. [ ] Add TypeScript types for `AdminFeedbackItem` and `AdminFeedbackListResponse`
### Step 4: Frontend — Admin feedback page
8. [ ] Create `frontend/app/admin/feedback/page.tsx` with:
- Table: user email, date, overall rating (stars), ease of use, lyrics accuracy, correction experience, would recommend, would use again
- Expandable rows or click-to-detail for text fields (what went well, what could improve, additional comments)
- Search by email
- Pagination (50 per page)
- Refresh button
- Test data toggle (via `useAdminSettings`)
9. [ ] Add "Feedback" item to admin sidebar in `frontend/components/admin/admin-sidebar.tsx` with `MessageSquare` icon
### Step 5: Testing
10. [ ] Backend unit test for `GET /api/admin/feedback` endpoint (pagination, search, exclude_test)
11. [ ] Backend unit test for `send_feedback_notification` email method
12. [ ] Frontend component test for feedback page (render, empty state)
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/api/routes/admin.py` | Modify | Add feedback list endpoint + response models |
| `backend/api/routes/users.py` | Modify | Add email notification call in `submit_user_feedback` |
| `backend/services/email_service.py` | Modify | Add `send_feedback_notification` method |
| `frontend/lib/api.ts` | Modify | Add `adminApi.listFeedback()` method + types |
| `frontend/app/admin/feedback/page.tsx` | Create | Admin feedback review page |
| `frontend/components/admin/admin-sidebar.tsx` | Modify | Add "Feedback" nav item |
| `backend/tests/test_admin_feedback.py` | Create | Tests for admin feedback endpoint |
| `backend/tests/test_feedback_email.py` | Create | Tests for feedback email notification |
## Testing Strategy
- **Unit tests**: Admin endpoint (pagination, filtering, sorting), email method (content formatting, send call)
- **Integration**: Submit feedback → verify email service called
- **Manual verification**: Check email arrives in admin@nomadkaraoke.com inbox after submission
## Open Questions
- [ ] Should the admin page show feedback from the old `beta_feedback` collection too, or just `user_feedback`? (Recommendation: just `user_feedback` — the beta collection is legacy)
- [ ] Should there be any spam/quality detection flagging? (Some submissions appear to game the system for credits)
## Rollback Plan
- Backend: Remove endpoint from admin.py, revert email call in users.py
- Frontend: Delete feedback page, remove sidebar item
- No data migration needed — Firestore collection unchanged
+120
View File
@@ -0,0 +1,120 @@
# Plan: Admin Lyrics Review Dashboard
**Created:** 2026-03-05
**Branch:** feat/sess-20260305-0002-admin-lyrics-review
**Status:** Draft
## Overview
Add an "Edit Reviews" section to the admin dashboard for reviewing completed lyrics reviews. Three goals:
1. **View what users saw** - Jump into a read-only lyrics review UI showing original vs corrected lyrics
2. **Replay user edits** - Step through edit log entries to see exactly what each user changed
3. **Review raw data** - View human edits data to evaluate quality for AudioShake data sharing
## Requirements
- [ ] New admin sidebar item "Edit Reviews" linking to `/admin/edit-reviews`
- [ ] Backend endpoint to list recent jobs that have completed lyrics review (with edit logs)
- [ ] Backend endpoint to serve immutable correction data for a job (read-only, cannot modify job)
- [ ] Admin page showing list of recently reviewed jobs with key metrics
- [ ] Ability to open the existing LyricsAnalyzer in read-only mode with snapshot data
- [ ] Edit log replay/stepper UI to walk through user changes one by one
- [ ] Raw data viewer for edit logs, annotations, and correction diffs
## Technical Approach
### Backend: 2 new admin endpoints
1. **`GET /api/admin/edit-reviews`** - List jobs with completed reviews
- Query Firestore for jobs with `state_data.last_edit_log_path` set (indicating review was performed)
- Filter by status (completed jobs, or optionally all post-review statuses)
- Return summary: job_id, artist, title, user_email, review_completed_at, edit_count, feedback_count
- Support pagination, search, exclude_test filtering
2. **`GET /api/admin/edit-reviews/{job_id}`** - Get full review snapshot
- Returns immutable copy of:
- Original corrections (`corrections.json` from GCS)
- Updated corrections (`corrections_updated.json` from GCS)
- Edit log (`edit_log_{session_id}.json` from GCS)
- Annotations (`annotations.json` from GCS)
- Job metadata (artist, title, user_email, timestamps)
- Audio URL (signed, for playback in review UI)
- This is a **read-only** endpoint - no job state mutations
- Uses `require_admin` dependency (not `require_review_auth`)
- Does NOT transition job status or modify any state
### Frontend: New admin page + Read-only review mode
1. **`/admin/edit-reviews/page.tsx`** - List page
- Table of recently reviewed jobs
- Columns: Artist/Title, User, Review Date, Edits Made, Feedback Given, Actions
- "View Review" button → opens read-only lyrics review
- "View Edit Log" button → opens edit replay view
- "View Raw Data" button → shows raw JSON
2. **Read-only review viewer** - Reuse LyricsAnalyzer
- Load data from the new admin endpoint (not the regular review endpoint)
- Pass `isReadOnly={true}` and `apiClient={null}` to LyricsAnalyzer
- Header shows "Admin Review - Read Only" indicator
- No submit/save buttons, no state transitions
- Audio playback still works (via signed URL)
3. **Edit log replay** - New component
- Timeline/stepper showing each edit entry
- For each entry: operation type, text before → text after, timestamp
- Step forward/backward through edits
- Show feedback reason if user provided one
- Summary stats: total edits, edits with feedback, correction types breakdown
4. **Raw data viewer** - JSON viewer tabs
- Tab 1: Edit Log (formatted JSON with syntax highlighting)
- Tab 2: Annotations
- Tab 3: Corrections diff (original vs updated)
### Key safety guarantee
The admin review endpoint serves a **snapshot** of the data. It reads from GCS files that were already written. The LyricsAnalyzer receives `apiClient={null}` so it physically cannot call any mutation endpoints. Even if code changes in the future, the admin endpoint itself never writes to Firestore or GCS.
## Implementation Steps
1. [ ] **Backend: Add `GET /api/admin/edit-reviews` endpoint** - List reviewed jobs with edit log metadata
2. [ ] **Backend: Add `GET /api/admin/edit-reviews/{job_id}` endpoint** - Serve full immutable review snapshot
3. [ ] **Frontend: Add admin API client methods** - `adminApi.listEditReviews()`, `adminApi.getEditReview(jobId)`
4. [ ] **Frontend: Create `/admin/edit-reviews/page.tsx`** - List page with table
5. [ ] **Frontend: Add sidebar nav item** - "Edit Reviews" with BookOpen icon
6. [ ] **Frontend: Create read-only review viewer** - Load snapshot data into LyricsAnalyzer with isReadOnly
7. [ ] **Frontend: Create edit log replay component** - Step-through UI for edit entries
8. [ ] **Frontend: Create raw data viewer** - JSON viewer with tabs
9. [ ] **Tests: Backend unit tests** for new admin endpoints
10. [ ] **Tests: Frontend component tests** for new page
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/api/routes/admin.py` | Modify | Add edit-reviews endpoints |
| `frontend/lib/api.ts` | Modify | Add adminApi methods for edit reviews |
| `frontend/app/admin/edit-reviews/page.tsx` | Create | Main edit reviews list page |
| `frontend/components/admin/edit-review-viewer.tsx` | Create | Read-only review wrapper using LyricsAnalyzer |
| `frontend/components/admin/edit-log-replay.tsx` | Create | Edit log stepper/replay component |
| `frontend/components/admin/raw-data-viewer.tsx` | Create | JSON viewer with tabs |
| `frontend/components/admin/admin-sidebar.tsx` | Modify | Add Edit Reviews nav item |
| `backend/tests/test_admin_edit_reviews.py` | Create | Backend tests |
## Testing Strategy
- Unit tests for backend endpoints (mock Firestore + GCS)
- Verify admin auth is required on all new endpoints
- Verify no write operations occur on read-only endpoints
- Frontend component tests for key interactions
## Open Questions
- [ ] Should we show ALL jobs that went through review, or only ones with edit logs? (Recommendation: only those with edit logs, since those are the interesting ones)
- [ ] Do we need audio playback in the admin review viewer? (Recommendation: yes, it helps understand what the user heard)
- [ ] Should the raw data be downloadable? (Nice-to-have, can add later)
## Rollback Plan
All changes are additive (new endpoints, new pages). Removing the sidebar link effectively hides the feature. No existing behavior is modified.
+151
View File
@@ -0,0 +1,151 @@
# Plan: AI-Powered Credit Gating
**Created:** 2026-03-23
**Branch:** feat/sess-20260323-1716-ai-credit-gating
**Status:** Implemented
## Overview
Replace the automatic welcome/feedback credit grants with an AI-powered evaluation step. When a user triggers a credit grant (email verification or feedback submission), instead of immediately granting credits, we collect abuse signals and ask Gemini to make a grant/deny decision. Suspected abusers get a friendly rejection email (CC'd to Andrew) instead of free credits.
## Requirements
- [ ] Welcome credit grants go through AI evaluation before granting
- [ ] Feedback credit grants go through AI evaluation before granting
- [ ] AI evaluates: fingerprint correlations, IP correlations, IP geo info, user agent, signup recency patterns
- [ ] Approved users get credits as before (seamless experience)
- [ ] Rejected users get a friendly-but-firm email explaining the situation, CC'd to andrew@beveridge.uk
- [ ] Rejection email invites them to reply if it's a mistake
- [ ] Decision is logged to Firestore for audit trail
- [ ] Fail-open: if AI evaluation fails, grant credits (don't punish users for our errors)
- [ ] Admin can override decisions via existing admin credit grant endpoint
## Technical Approach
### Architecture
```
User verifies email / submits feedback
↓
Collect abuse signals (sync, fast):
- User's fingerprint + IP
- Other accounts with same fingerprint (from correlations)
- Other accounts with same IP
- IP geolocation info
- Recent signup count from same IP/fingerprint (last 24h)
- User agent string
- For feedback: the feedback content itself
↓
Call Gemini (vertexai/gemini-2.5-pro-preview) with structured prompt:
- Present all signals
- Ask for JSON response: {decision: "grant" | "deny", reasoning: string, confidence: number}
↓
If "grant": grant credits normally, log decision
If "deny": skip credit grant, send rejection email, log decision
If AI call fails: grant credits (fail-open), log error
```
### Key Design Decisions
1. **Two-phase verification**: The magic link verify endpoint returns immediately with session + a `credit_status: "evaluating"` field. The frontend shows a "preparing your account" loading screen, then polls/waits for the evaluation result via a new lightweight endpoint. This keeps the auth flow snappy while the AI evaluation runs in the background of the same request or via a short follow-up call. The frontend then shows either a celebratory "2 free credits!" interstitial or a "sorry, couldn't grant credits" interstitial before navigating to the dashboard.
2. **Fail-open**: If Gemini is down or returns an error, we grant credits. Better to let an occasional abuser through than block legitimate users.
3. **Use google-genai directly**: The project already has `langchain-google-genai` installed. We can use the simpler `google.generativeai` library directly for a focused prompt/response, or use the existing LangChain setup. We'll use `google.generativeai` for simplicity since this is a single-turn evaluation, not an agentic workflow.
4. **Structured JSON output**: Ask Gemini for structured output with decision, reasoning, and confidence to make decisions parseable and auditable.
5. **Log all decisions**: Store every evaluation in a `credit_evaluations` Firestore collection for audit trail and to tune the prompt over time.
## Implementation Steps
### Step 1: Add google-generativeai dependency
- Add `google-generativeai` to pyproject.toml (lightweight, direct Gemini access)
- Verify it works with Vertex AI auth in the nomadkaraoke project
### Step 2: Create credit evaluation service
- New file: `backend/services/credit_evaluation_service.py`
- `CreditEvaluationService` class with:
- `evaluate_credit_grant(email, grant_type, feedback_content=None) -> CreditEvaluation`
- Collects all abuse signals by calling existing services (user_service, ip_geolocation_service)
- Builds prompt with signals
- Calls Gemini for decision
- Logs decision to Firestore
- Returns structured result
### Step 3: Build the Gemini prompt
- System prompt explaining the role: "You are an anti-abuse evaluator for a karaoke video service..."
- Present all signals in a structured format
- Request JSON response: `{decision: "grant"|"deny", reasoning: "...", confidence: 0.0-1.0}`
- Include clear guidelines: same fingerprint across accounts = very suspicious, same IP = somewhat suspicious, etc.
### Step 4: Create rejection email template
- Add `send_credit_denied_email(email, reasoning)` to EmailService
- Friendly-but-firm tone
- CC andrew@beveridge.uk
- Invite reply if it's a mistake
- Use existing email template infrastructure (branding, styling)
### Step 5: Integrate into welcome credit flow
- Modify `grant_welcome_credits_if_eligible()` in user_service.py
- Before granting: call `CreditEvaluationService.evaluate_credit_grant(email, "welcome")`
- If denied: send rejection email, return False
- If error: log warning, grant anyway (fail-open)
### Step 6: Integrate into feedback credit flow
- Modify feedback endpoint in users.py
- Before granting credits: evaluate with feedback content included
- If denied: return different response message, send rejection email
- If error: grant anyway
### Step 7: Add Firestore collection for audit trail
- Collection: `credit_evaluations`
- Document fields: email, grant_type, decision, reasoning, confidence, signals_snapshot, model_used, evaluated_at
- TTL: keep indefinitely (small volume, high audit value)
### Step 8: Tests
- Unit tests for signal collection
- Unit tests for prompt building
- Unit tests for decision parsing (grant/deny/error cases)
- Integration test for the evaluation service with mocked Gemini
- Integration test for welcome credit flow with evaluation
- Integration test for feedback credit flow with evaluation
- Test fail-open behavior
### Step 9: Configuration
- Add env vars: `CREDIT_EVAL_MODEL` (default: "gemini-2.5-pro-preview"), `CREDIT_EVAL_ENABLED` (default: true)
- Add to config.py
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `pyproject.toml` | Modify | Add google-generativeai dependency |
| `backend/config.py` | Modify | Add credit evaluation config vars |
| `backend/services/credit_evaluation_service.py` | Create | New service: collect signals, call Gemini, log decision |
| `backend/services/user_service.py` | Modify | Integrate evaluation into welcome credit grant |
| `backend/api/routes/users.py` | Modify | Integrate evaluation into feedback credit grant |
| `backend/services/email_service.py` | Modify | Add credit denied email template |
| `backend/tests/test_credit_evaluation.py` | Create | Tests for evaluation service |
| `backend/tests/test_anti_abuse.py` | Modify | Update welcome credit tests for evaluation |
| `frontend/app/auth/verify/page.tsx` | Modify | Add "preparing account" loading + credit result interstitial |
| `frontend/lib/api.ts` | Modify | Add credit evaluation status endpoint |
## Testing Strategy
- **Unit tests**: Signal collection, prompt building, response parsing, fail-open behavior
- **Integration tests**: Full evaluation flow with mocked Gemini responses
- **E2E tests**: Manual verification in production with test accounts
- **Edge cases**: Gemini timeout, malformed response, missing signals, first-ever user (no correlations)
## Open Questions
1. **Gemini model**: Using `gemini-3.1-pro-preview` as specified — already available in the nomadkaraoke GCP project.
2. **Latency UX**: Frontend shows a friendly "preparing your account" loading screen while evaluation runs. Then transitions to either a celebratory "2 free credits!" message (with gentle reminder not to abuse) or a "sorry, couldn't grant free credits" message with a buy credits CTA. User clicks through to dashboard from either.
3. **Should we also gate the existing rate limiter?** Currently `is_signup_rate_limited()` silently blocks at 2 accounts per IP/fingerprint per 24h. The AI evaluation is complementary — rate limiter blocks obvious rapid abuse, AI evaluation catches more subtle patterns.
## Rollback Plan
Set `CREDIT_EVAL_ENABLED=false` in Cloud Run env vars → immediate rollback to auto-grant behavior. No data model changes needed. The evaluation service is purely additive.
@@ -0,0 +1,150 @@
# Plan: Block Disposable Email Registration
**Created:** 2026-03-19
**Branch:** feat/sess-20260319-1332-disposable-email-abuse
**Status:** Draft
## Overview
Users are signing up with disposable email addresses (yopmail.com, emailhook.site, etc.) to get unlimited free credits (2 per account). The `EmailValidationService` already exists with 160+ disposable domains but is **never called during signup**. We need to wire it into the magic link flow and show a friendly rejection message.
## Context
- The `EmailValidationService` (`backend/services/email_validation_service.py`) is fully implemented with disposable domain detection, email/IP blocklists, and admin management APIs
- The admin blocklist management routes exist (`backend/api/routes/rate_limits.py`) and work
- The magic link endpoint (`POST /api/users/auth/magic-link` in `backend/api/routes/users.py:133`) **never calls** the validation service
- Currently 1 yopmail user and 1 emailhook.site user in `gen_users`
- Each new account gets 2 free welcome credits automatically
## Requirements
- [x] Reject disposable email domains at the magic link endpoint (backend)
- [x] Return a **specific, distinguishable error** so the frontend can show a friendly message (not the generic "email enumeration" response)
- [x] Show a warm, friendly message in the AuthDialog explaining why and encouraging real email use
- [x] Add `emailhook.site` to the default disposable domains list
- [x] Add backend unit tests for the integration
- [x] Add frontend test coverage for the new error state
## Design Decision: Explicit Rejection vs Silent Failure
The current magic link endpoint returns a generic success message to prevent email enumeration. For disposable emails, we will **explicitly reject** with a 422 status and a specific error message. This is the right trade-off because:
1. We *want* the user to know their disposable email won't work (so they can use a real one)
2. Disposable domain detection is not sensitive information — the blocklist is public knowledge
3. The friendly message serves our goal of converting them to real email users
## Implementation Steps
### Step 1: Backend — Add disposable email check to magic link endpoint
**File:** `backend/api/routes/users.py`
Add the `EmailValidationService` check **after** email lowercasing (line 158) and **before** tenant validation (line 164). Return a 422 with a friendly message.
```python
# Check for disposable email domains
email_validation = get_email_validation_service()
if email_validation.is_disposable_domain(email):
logger.warning(f"Blocked disposable email signup attempt: {_mask_email(email)}")
raise HTTPException(
status_code=422,
detail="disposable_email_not_allowed"
)
```
Use a machine-readable error code (`disposable_email_not_allowed`) so the frontend can match on it and show the custom friendly message. This keeps the backend response lean and puts the UX copy in the frontend where it belongs.
### Step 2: Backend — Also check email and IP blocklists
In the same location, add checks for explicitly blocked emails and IPs:
```python
if email_validation.is_email_blocked(email):
logger.warning(f"Blocked email signup attempt: {_mask_email(email)}")
# Silent reject — return generic success to prevent enumeration
return SendMagicLinkResponse(
status="success",
message="If this email is registered, you will receive a sign-in link shortly."
)
ip_address = http_request.client.host if http_request.client else None
if ip_address and email_validation.is_ip_blocked(ip_address):
logger.warning(f"Blocked IP signup attempt: {ip_address}")
return SendMagicLinkResponse(
status="success",
message="If this email is registered, you will receive a sign-in link shortly."
)
```
Note: Blocked emails/IPs get silent rejection (anti-enumeration). Disposable domains get explicit rejection (we want the user to switch to a real email).
### Step 3: Backend — Add `emailhook.site` to default disposable domains
**File:** `backend/services/email_validation_service.py`
Add `emailhook.site` to `DEFAULT_DISPOSABLE_DOMAINS` set.
### Step 4: Frontend — Handle the disposable email error in AuthDialog
**File:** `frontend/components/auth/AuthDialog.tsx`
The `sendMagicLink` call in `auth.ts` already catches errors and sets `error` from the API response message. The `handleResponse` in `api.ts` throws an `ApiError` with the `detail` string for non-200 responses.
Update `handleSendMagicLink` to detect the `disposable_email_not_allowed` error code and show a friendly multi-line message instead of the terse error string:
```tsx
const success = await sendMagicLink(email.trim().toLowerCase())
if (success) {
setStep("sent")
}
// If error contains our specific code, override with friendly message
// (the error state is already set by the auth store)
```
We need to detect the specific error. The cleanest approach: check the error string in the `displayError` rendering. If it matches `disposable_email_not_allowed`, render a custom friendly block instead of the red error text.
The friendly message:
> **We don't support disposable email addresses**
>
> We'd love to help you create karaoke videos! Please use your regular email address so we can send you a notification when your video is ready.
>
> We promise — no spam, no marketing emails, ever. We only email you about your karaoke videos.
### Step 5: Backend tests
**File:** `backend/tests/test_email_validation_integration.py` (new)
Test cases:
1. Magic link with disposable email returns 422 with `disposable_email_not_allowed`
2. Magic link with legitimate email proceeds normally
3. Magic link with blocked email returns silent success (no actual email sent)
4. Magic link with blocked IP returns silent success
### Step 6: Frontend tests
**File:** `frontend/components/__tests__/AuthDialog.test.tsx` (new or extend existing)
Test cases:
1. Submitting a disposable email shows the friendly rejection message
2. The friendly message contains the key copy (no spam promise, use real email)
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/api/routes/users.py` | Modify | Add email validation checks to magic link endpoint |
| `backend/services/email_validation_service.py` | Modify | Add `emailhook.site` to default domains |
| `frontend/components/auth/AuthDialog.tsx` | Modify | Handle disposable email error with friendly message |
| `backend/tests/test_email_validation_integration.py` | Create | Integration tests for the validation in signup flow |
## Testing Strategy
- **Unit tests:** Validate `is_disposable_domain` catches yopmail, emailhook.site (already covered in existing tests)
- **Integration tests:** New test file verifying the magic link endpoint rejects disposable emails with 422
- **Frontend tests:** Verify the AuthDialog shows friendly message on disposable email error
- **Manual verification:** Try signing up with `test@yopmail.com` on dev server to see the UX
## Rollback Plan
Remove the validation check from `send_magic_link()` in `users.py`. The `EmailValidationService` itself remains unchanged (it's already deployed and used by admin routes).
@@ -0,0 +1,111 @@
# Plan: Credit Refund on Job Deletion and Cancellation
**Created:** 2026-03-04
**Branch:** feat/sess-20260304-1308-investigate-user-credits
**Status:** Implemented
## Overview
When a user deletes or cancels a job, their credit is not refunded. This was discovered when user john2011@sky.com deleted a job 16 seconds after creation (accidental duplicate) and lost the credit. Job *failures* already auto-refund via `mark_job_failed()` → `refund_credit()`, but deletions and cancellations do not.
## Requirements
- [x] Job failures already refund credits (existing behavior, no change needed)
- [ ] Job cancellation refunds 1 credit
- [ ] Job deletion refunds 1 credit (if not already completed)
- [ ] No refund for completed jobs (user got their video)
- [ ] No refund for admin-owned jobs (consistent with failure refund logic)
- [ ] Idempotency: prevent double-refund if a job is cancelled then deleted, or fails then deleted
- [ ] Credit transaction records the reason clearly ("job_cancelled", "job_deleted")
- [ ] Refund failure must not block the cancel/delete operation
## Technical Approach
### Key Insight: Idempotency via Job Field
The current `refund_credit()` has no guard against double-refund for the same job. Since a job can transition through cancelled → deleted, or failed → deleted, we need a way to track that a refund was already issued.
**Approach**: Add a `credit_refunded` boolean field to the Job model. Check it before refunding. This is simpler and more reliable than scanning the user's transaction history.
### What Should and Shouldn't Be Refunded
| Job Status at Time of Action | Delete | Cancel | Notes |
|------------------------------|--------|--------|-------|
| PENDING through pre-COMPLETE | Refund | Refund | Job didn't deliver value |
| COMPLETE / PREP_COMPLETE | No refund | N/A (can't cancel terminal) | User got their output |
| FAILED | No refund (already refunded) | N/A (can't cancel terminal) | Already refunded by fail_job |
| CANCELLED | No refund (already refunded) | N/A (already cancelled) | Already refunded by cancel |
For deletion: we need to read the job status *before* deleting it to decide whether to refund. The delete endpoint already fetches the job for ownership check, so this is straightforward.
### Refund Helper
Extract the refund logic (currently in `mark_job_failed`) into a reusable private method `_refund_credit_for_job()` that:
1. Checks `job.credit_refunded` — skip if already refunded
2. Checks `job.user_email` — skip if no user
3. Checks if admin — skip if admin job
4. Calls `user_service.refund_credit()` with appropriate reason
5. Sets `credit_refunded = True` on the job (for cancel; for delete the job is about to be removed but we set it anyway for the brief window)
6. Wraps everything in try/except so failures don't block the parent operation
## Implementation Steps
1. [ ] **Add `credit_refunded` field to Job model** (`backend/models/job.py`)
- Add `credit_refunded: bool = False` to the Job model
- No migration needed — Firestore is schemaless, defaults to False for existing docs
2. [ ] **Extract refund helper in JobManager** (`backend/services/job_manager.py`)
- Create `_refund_credit_for_job(self, job_id: str, job: Job, reason: str) -> bool`
- Move the refund logic from `mark_job_failed` (lines 821-839) into this helper
- Add `credit_refunded` check at the top
- Update `mark_job_failed` to call the helper with `reason="job_failed"`
3. [ ] **Add refund to `cancel_job()`** (`backend/services/job_manager.py`)
- After setting status to CANCELLED, call `_refund_credit_for_job(job_id, job, reason="job_cancelled")`
- The job object is already fetched at line 1017
4. [ ] **Add refund to `delete_job()`** (`backend/services/job_manager.py`)
- Need to fetch the job (already done when `delete_files=True`, but not always)
- Before deletion, check if job status is NOT in `[COMPLETE, PREP_COMPLETE, FAILED, CANCELLED]`
- If eligible, call `_refund_credit_for_job(job_id, job, reason="job_deleted")`
- For FAILED/CANCELLED: skip because refund was already issued during that transition
- For COMPLETE/PREP_COMPLETE: skip because user received their output
5. [ ] **Update `mark_job_failed` to set `credit_refunded`**
- The helper handles this, but also update the Firestore doc
6. [ ] **Add unit tests** (`backend/tests/test_credit_enforcement.py`)
- Test: cancel_job refunds credit for non-admin user
- Test: cancel_job skips refund for admin jobs
- Test: delete_job refunds credit for non-complete, non-terminal jobs
- Test: delete_job does NOT refund for completed jobs
- Test: delete_job does NOT double-refund for already-failed jobs
- Test: delete_job does NOT double-refund for already-cancelled jobs
- Test: refund failure doesn't block cancel/delete
- Test: credit_refunded flag prevents double refund
7. [ ] **Backfill: Refund john2011@sky.com** (manual one-time)
- Grant 1 credit for the deleted job 40dcca89 via admin API or direct Firestore update
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Add `credit_refunded: bool = False` field |
| `backend/services/job_manager.py` | Modify | Extract `_refund_credit_for_job()` helper, add refund to `cancel_job()` and `delete_job()` |
| `backend/tests/test_credit_enforcement.py` | Modify | Add tests for cancel/delete refund scenarios |
## Testing Strategy
- **Unit tests**: Extend `test_credit_enforcement.py` with new test classes `TestCreditRefundOnJobCancellation` and `TestCreditRefundOnJobDeletion`, following the exact pattern of existing `TestCreditRefundOnJobFailure`
- **Manual verification**: After deploy, cancel and delete test jobs and verify credit balance updates
## Open Questions
- [ ] Should we notify the user (email/in-app) when a credit is refunded? (Suggest: no, keep it simple for now — the credit just appears back in their balance)
## Rollback Plan
- The `credit_refunded` field is additive (defaults to False for old jobs)
- If issues arise, revert the PR — old behavior (no refund on cancel/delete) is restored
- Any refunds already issued are just bonus credits to users, no harm done
+407
View File
@@ -0,0 +1,407 @@
# Plan: Add Retry Logic for Flacfetch Download Failures
**Created:** 2026-02-10
**Branch:** feat/sess-20260210-1158-investigate-job-749141f8
**Status:** Draft
## Overview
Job 749141f8 failed with error: "Audio download failed: Remote download by ID failed: Download by ID request failed:"
Investigation revealed **multiple related issues** when flacfetch network calls fail:
1. **No retry logic** - A single network hiccup causes immediate job failure
2. **Missing source info** - When download fails, source_id/source_name aren't saved to job
3. **Broken retry button** - Can't retry because no source info was persisted
4. **Silent UI failures** - Frontend doesn't show retry error messages to user
### Root Cause
Audio search flow saves selection info to `state_data`, but doesn't save download parameters (source_id, source_name, etc.) to the job document until AFTER download succeeds. If the flacfetch call fails (network error), the job fails with no way to retry.
### Solution
This plan fixes all four issues:
- Add retry logic to flacfetch client (network resilience)
- Persist download info before attempting download (enables retry)
- Update retry endpoint to handle audio search jobs
- Show error messages in frontend when retry fails
## Requirements
### Functional Requirements - Flacfetch Retry Logic
- [x] Retry `download_by_id()` on `httpx.RequestError` (network failures)
- [x] Use exponential backoff spread over ~5 minutes
- [x] Configurable max retries via environment variable
- [x] Log each retry attempt with context
- [x] Do NOT retry on HTTP 4xx errors (client errors - immediate failure)
- [x] Do retry on HTTP 5xx errors (server errors - transient)
### Functional Requirements - Persist Download Info
- [x] Save source_id, source_name, target_file, download_url to job document
- [x] Save BEFORE attempting download (not after)
- [x] Store in top-level fields for easy retry access
- [x] Preserve existing state_data fields (selected_audio_index, etc.)
### Functional Requirements - Retry Endpoint
- [x] Add case for audio search jobs in `/jobs/{id}/retry` endpoint
- [x] Check for saved download params (source_id, source_name)
- [x] Re-download using saved params
- [x] Clear error state before retrying
- [x] Trigger appropriate workers after successful download
### Functional Requirements - Frontend Error Handling
- [x] Show toast notification when retry fails
- [x] Display actual error message from API
- [x] Don't just console.error - make it visible to user
### Non-Functional Requirements
- [x] Minimal code changes (use existing patterns)
- [x] Clear error messages that include retry context
- [x] Backwards compatible (no breaking changes)
- [x] Should not significantly increase total timeout duration
## Technical Approach
### Architecture Decisions
**Option 1: Tenacity library (Recommended)**
- ✅ Industry standard, well-tested
- ✅ Declarative retry configuration
- ✅ Built-in exponential backoff
- ✅ Already used in project (check `poetry.lock`)
- ❌ New dependency if not already present
**Option 2: Manual retry loop**
- ✅ No new dependencies
- ✅ Full control over logic
- ❌ More code to maintain
- ❌ Easy to get wrong (edge cases)
**Decision:** Use **tenacity** if already in dependencies, otherwise implement manual retry loop.
### Retry Strategy
```python
# Retry on network errors only (RequestError)
# Do NOT retry on:
# - HTTP 4xx (client errors like 400, 401, 404)
# - Already completed downloads
# Do retry on:
# - httpx.RequestError (connection timeout, refused, DNS failure)
# - HTTP 5xx (server errors - service temporarily down)
Max retries: 8 (total 9 attempts)
Backoff: Exponential with cap, multiplier=2, min=10s, max=60s
- Attempt 1: immediate
- Attempt 2: wait 10s
- Attempt 3: wait 20s
- Attempt 4: wait 40s
- Attempt 5: wait 60s (capped)
- Attempt 6: wait 60s (capped)
- Attempt 7: wait 60s (capped)
- Attempt 8: wait 60s (capped)
- Attempt 9: wait 60s (capped)
Total max time: ~370s (6+ minutes) of backoff + request timeouts
This gives the flacfetch service time to recover from restarts,
network issues, or temporary VM problems.
```
### Configuration
Add to `backend/config.py`:
```python
# Flacfetch client retry settings
flacfetch_retry_max_attempts: int = int(os.getenv("FLACFETCH_RETRY_MAX_ATTEMPTS", "9")) # Total attempts (1 initial + 8 retries)
flacfetch_retry_min_wait: float = float(os.getenv("FLACFETCH_RETRY_MIN_WAIT", "10.0")) # seconds
flacfetch_retry_max_wait: float = float(os.getenv("FLACFETCH_RETRY_MAX_WAIT", "60.0")) # seconds
```
### Error Message Improvements
Change from:
```
Download by ID request failed:
```
To:
```
Download by ID request failed after 4 attempts: [ConnectTimeout] Connection to http://10.0.0.5:8080 timed out
```
## Implementation Steps
### Phase 1: Check Dependencies
1. [ ] Check if `tenacity` is in `pyproject.toml` dependencies
2. [ ] If not present, add `tenacity = "^8.2.0"` to dependencies
3. [ ] Run `poetry lock` if dependencies changed
### Phase 2: Add Configuration
4. [ ] Add retry configuration to `backend/config.py`
- `flacfetch_retry_max_attempts = 9`
- `flacfetch_retry_min_wait = 10.0`
- `flacfetch_retry_max_wait = 60.0`
### Phase 3: Implement Flacfetch Retry Logic
5. [ ] Import tenacity in `backend/services/flacfetch_client.py`
6. [ ] Create retry decorator with:
- Retry on `httpx.RequestError`
- Retry on `httpx.HTTPStatusError` with 5xx status
- Stop on `httpx.HTTPStatusError` with 4xx status
- Exponential backoff with cap at 60s
- Before-sleep logging callback
7. [ ] Apply decorator to `download_by_id()` method
8. [ ] Improve error messages to include:
- Number of attempts made
- Original exception type and message
### Phase 4: Persist Download Info Before Download
9. [ ] Update `audio_search.py:_download_and_start_processing()`
10. [ ] After getting source info from selected result (line 307-311)
11. [ ] **Save to job document BEFORE download attempt:**
```python
job_manager.update_job(job_id, {
'audio_source_type': 'audio_search',
'source_name': source_name,
'source_id': source_id,
'target_file': target_file,
'download_url': download_url,
})
```
12. [ ] Then proceed with download (existing code)
13. [ ] Add to Job model if needed (check if fields exist)
### Phase 5: Update Retry Endpoint for Audio Search
14. [ ] Update `backend/api/routes/jobs.py:retry_job()`
15. [ ] Add new case before "no input audio" error (line ~1088)
16. [ ] Check if job has saved download params:
```python
elif job.audio_source_type == 'audio_search' and job.source_id:
# Has audio search download info - retry download
```
17. [ ] Re-call audio search download logic with saved params
18. [ ] Or refactor download into reusable function
### Phase 6: Frontend Error Handling
19. [ ] Update `frontend/components/job/JobActions.tsx`
20. [ ] Add toast notification library (check if exists)
21. [ ] In `handleRetry()` catch block, show toast:
```typescript
catch (error) {
const message = error.response?.data?.detail || "Failed to retry job"
toast.error(message)
console.error("Failed to retry job:", error)
}
```
### Phase 7: Testing - Flacfetch Retry
22. [ ] Add unit tests to `backend/tests/test_flacfetch_client.py`
23. [ ] Test retry on RequestError (mock fails N times then succeeds)
24. [ ] Test no retry on 4xx errors
25. [ ] Test retry on 5xx errors
26. [ ] Test max retries exhausted
27. [ ] Verify logging output includes attempt numbers
### Phase 8: Testing - End-to-End
28. [ ] Test audio search job with flacfetch down
29. [ ] Verify source info is saved even when download fails
30. [ ] Verify retry button works after source info saved
31. [ ] Test retry succeeds when flacfetch comes back up
32. [ ] Verify error toast appears when retry fails
### Phase 9: Documentation
33. [ ] Update `backend/services/flacfetch_client.py` docstring
34. [ ] Add to `docs/LESSONS-LEARNED.md`:
- "Always add retries for external service calls"
- "Save operation parameters before attempting operations"
- "Persist enough state to enable retry from any failure point"
35. [ ] Update `docs/ARCHITECTURE.md` if needed
36. [ ] Document the new job fields in API.md
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/config.py` | Modify | Add retry configuration variables |
| `backend/services/flacfetch_client.py` | Modify | Add retry logic to `download_by_id()` |
| `backend/api/routes/audio_search.py` | Modify | Save download info before attempting download |
| `backend/api/routes/jobs.py` | Modify | Add audio search case to retry endpoint |
| `backend/models/job.py` | Modify (maybe) | Add fields: source_name, source_id, target_file, download_url |
| `frontend/components/job/JobActions.tsx` | Modify | Show error toast when retry fails |
| `backend/tests/test_flacfetch_client.py` | Modify | Add retry tests |
| `backend/tests/test_audio_search.py` | Modify | Test download info persistence |
| `backend/tests/test_jobs.py` | Modify | Test retry with audio search jobs |
| `pyproject.toml` | Modify (maybe) | Add tenacity if not present |
| `docs/LESSONS-LEARNED.md` | Modify | Document patterns |
| `docs/API.md` | Modify | Document new job fields |
## Testing Strategy
### Unit Tests (`backend/tests/test_flacfetch_client.py`)
1. **Test retry on RequestError**
```python
@pytest.mark.asyncio
async def test_download_by_id_retries_on_network_error():
"""Should retry 3 times on connection errors."""
client = FlacfetchClient(base_url="http://test", api_key="key")
with patch("httpx.AsyncClient.post") as mock_post:
# Fail 2 times, then succeed
mock_post.side_effect = [
httpx.RequestError("Connection timeout"),
httpx.RequestError("Connection refused"),
Mock(json=lambda: {"download_id": "test123"}, status_code=200),
]
result = await client.download_by_id("YouTube", "abc123")
assert result == "test123"
assert mock_post.call_count == 3
```
2. **Test no retry on 4xx errors**
```python
@pytest.mark.asyncio
async def test_download_by_id_no_retry_on_client_error():
"""Should NOT retry on 400/401/404 errors."""
client = FlacfetchClient(base_url="http://test", api_key="key")
with patch("httpx.AsyncClient.post") as mock_post:
response = Mock(status_code=404, text="Not found")
response.raise_for_status.side_effect = httpx.HTTPStatusError(
"Not found", request=Mock(), response=response
)
mock_post.return_value = response
with pytest.raises(FlacfetchServiceError, match="404"):
await client.download_by_id("YouTube", "abc123")
assert mock_post.call_count == 1 # No retries
```
3. **Test retry on 5xx errors**
```python
@pytest.mark.asyncio
async def test_download_by_id_retries_on_server_error():
"""Should retry on 500/503 errors."""
# Similar to test 1 but with HTTP 503 errors
```
4. **Test max retries exhausted**
```python
@pytest.mark.asyncio
async def test_download_by_id_fails_after_max_retries():
"""Should fail with clear message after max retries."""
# All attempts fail, verify error message includes attempt count
```
### Integration Tests
5. **Test with actual flacfetch service** (if available in test environment)
- Verify retry doesn't break normal operation
- Test with deliberate network issues
### Manual Testing
6. **Simulate network failure**
```bash
# Temporarily stop flacfetch VM
gcloud compute instances stop flacfetch-service --zone=us-central1-a
# Submit a test job
# Observe logs showing retry attempts
# Restart VM during retries
gcloud compute instances start flacfetch-service --zone=us-central1-a
# Job should succeed
```
7. **Test with intentional delays**
- Add sleep in flacfetch API handler
- Verify retries don't cause issues
## Open Questions
- [x] ~~Is tenacity already in dependencies?~~ (Will check during implementation)
- [x] ~~What's the appropriate total timeout budget?~~ - 9 attempts over ~6 minutes
- [x] ~~Should we fix the Retry button UX in this PR?~~ - YES, it's the same root cause
- [x] ~~Why did job 749141f8 fail before saving audio source info?~~ - Download info saved AFTER download, not before
- [ ] Should we also add retries to `wait_for_download()`?
- Probably NO - that already has its own polling/timeout logic
- [ ] Should we add circuit breaker pattern?
- Probably NO for now - keep it simple
- Could add in future if we see cascading failures
- [ ] Should retry logic be applied to other flacfetch methods?
- `start_download()` - probably YES (same pattern)
- `search()` - maybe YES (search can also have network issues)
- `get_download_status()` - probably NO (polling already handles this)
- **Decision**: Apply to `start_download()` too, skip `search()` for now
- [ ] Which toast library should frontend use?
- Check if already exists (react-hot-toast, sonner, etc.)
- If not, add lightweight library
- [ ] Should we save download params to state_data or top-level fields?
- Top-level fields more visible and easier to query
- **Decision**: Top-level fields (source_id, source_name, etc.)
## Rollback Plan
If this causes issues:
1. **Quick rollback**: Set env var `FLACFETCH_RETRY_MAX_ATTEMPTS=1` in Cloud Run
- Disables retries without code change
- Can deploy in ~2 minutes
2. **Full rollback**: Revert the PR
- Git revert the merge commit
- Deploy previous version
3. **Monitoring**: Watch for:
- Increased request duration to flacfetch
- Jobs timing out at higher layer (Cloud Tasks, Cloud Run)
- Duplicate download attempts causing issues
## Success Criteria
- [ ] Jobs with transient flacfetch failures succeed instead of failing
- [ ] Error messages are more informative (show attempts and root cause)
- [ ] No increase in false positives (jobs that should fail, now hang)
- [ ] All tests pass
- [ ] No impact on successful jobs (retry overhead is minimal)
## Related Issues
### Job 749141f8 - The Incident
**What happened:**
- User searched for audio via audio search
- Selected "Elizabeth Moen - Songbird"
- Backend tried to download via flacfetch
- Flacfetch call failed with network error (connection timeout/refused)
- Job immediately failed
- No source info saved → Retry button broken
- User sees "Retry" button but clicking does nothing (silent 400 error)
**How this fix helps:**
1. ✅ Flacfetch retry logic → Download would have succeeded after retry
2. ✅ Persist download info → Even if all retries fail, user can retry later
3. ✅ Retry endpoint fix → Retry button will actually work
4. ✅ Frontend toast → User sees helpful error if retry still fails
**Note:** Job 749141f8 cannot be fixed retroactively (no source info saved). But future jobs with same issue will be fixable via retry button.
### Future Work (Not in This PR)
Similar pattern should be applied to other external service calls:
- AudioShake API (lyrics transcription)
- Modal API (audio separation)
- YouTube direct API calls (not via flacfetch)
- Email sending (SendGrid)
## References
- [Tenacity documentation](https://tenacity.readthedocs.io/)
- [LESSONS-LEARNED.md](docs/LESSONS-LEARNED.md) - External service response format mismatches
- [flacfetch README](../../flacfetch/README.md) - Understanding the service
@@ -0,0 +1,149 @@
# Plan: Improve Job Observability
**Created:** 2026-03-07
**Branch:** feat/sess-20260307-2216-improve-job-observability
**Status:** Draft
## Overview
After shipping the "edit completed tracks" feature, we found it very difficult to trace what actually happened to edited jobs. Timeline events are too sparse (no output details), cleanup operations aren't logged to Cloud Logging or Firestore, state_data gets fully overwritten during re-processing (losing history of previous outputs), and API endpoints don't write to the job's Firestore log subcollection.
The goal is to make it trivially easy for future agents and humans to reconstruct the complete lifecycle of any job by reading its timeline and Firestore logs.
## Problem Analysis
### What we found investigating two edited jobs:
1. **Timeline events only have `status`, `timestamp`, `progress`, `message`** — no output details like YouTube video IDs, brand codes, Dropbox links, or GDrive file IDs
2. **Edit endpoint cleanup results never logged** — the `cleanup_results` dict is built and returned in the HTTP response but never logged to Cloud Logging or the job's Firestore log subcollection
3. **Distribution details incomplete in logs** — YouTube logs the URL but not video_id separately; Dropbox doesn't log folder path; GDrive doesn't log file IDs
4. **`state_data` fully overwritten on re-processing** — `video_worker.py:342-354` does `{**job.state_data, ...}`, so any `DELETE_FIELD` operations from the edit endpoint get lost when old keys are re-merged from the in-memory copy
5. **`outputs_deleted_at` cleared by video worker** — intentionally, but loses audit trail
6. **No Firestore job logs from API endpoints** — edit/admin endpoints only log to Cloud Run stdout, not to `jobs/{job_id}/logs/`
## Requirements
- [ ] Add `metadata` dict to `TimelineEvent` for structured output details
- [ ] Log all cleanup operations (success AND failure) to both Cloud Logging and Firestore job logs
- [ ] Log distribution output details (YouTube video ID, Dropbox path, brand code, GDrive file IDs) in timeline events and Firestore logs
- [ ] Preserve previous output details in timeline when a job is edited (snapshot before cleanup)
- [ ] Add helper to write Firestore job logs from API endpoints (not just workers)
- [ ] Ensure edit/admin delete endpoints write to Firestore job logs
- [ ] No breaking changes to existing timeline format (additive only)
## Technical Approach
### 1. Add `metadata` field to `TimelineEvent`
The `TimelineEvent` model gains an optional `metadata: Optional[Dict[str, Any]]` field. This allows any timeline event to carry structured data without changing the schema. The field is additive — existing events without metadata continue to work.
Key events that gain metadata:
- **`complete`**: `{youtube_url, youtube_video_id, dropbox_link, brand_code, gdrive_file_ids}`
- **Edit initiated**: `{previous_outputs: {youtube_url, dropbox_link, brand_code, gdrive_files}, cleanup_results: {...}}`
- **Outputs deleted**: `{deleted_outputs: {...}, cleanup_results: {...}}`
### 2. Write Firestore job logs from API endpoints
Create a simple helper function that API route handlers can call to append logs to the job's Firestore subcollection. Workers already have `JobLogAdapter` — we just need a lightweight equivalent for API routes:
```python
def log_to_job(job_id: str, worker: str, level: str, message: str, metadata: dict = None):
"""Write a log entry to the job's Firestore log subcollection."""
entry = WorkerLogEntry.create(job_id, worker, level, message, metadata)
FirestoreService().append_log_to_subcollection(job_id, entry)
```
### 3. Enhance distribution logging in video_worker_orchestrator
After each distribution upload succeeds, log the details with structured metadata:
- YouTube: video_id, video_url, channel
- Dropbox: folder_path, sharing_link
- GDrive: file_ids dict
- Brand code: code, prefix, allocated vs recycled
### 4. Snapshot previous outputs on edit
Before the edit endpoint clears state_data distribution keys, capture the current values in the timeline event's metadata. This preserves the history permanently in the timeline array (which is append-only via `ArrayUnion`).
### 5. Log cleanup results from edit/admin endpoints
After cleanup runs, log each step's result to both Cloud Logging and the Firestore job log subcollection with structured metadata.
## Implementation Steps
### Step 1: Add `metadata` to `TimelineEvent` model
- [ ] Add `metadata: Optional[Dict[str, Any]] = None` to `TimelineEvent` in `backend/models/job.py`
- [ ] No migration needed — Firestore is schemaless, existing events just won't have the field
### Step 2: Create `log_to_job` helper
- [ ] Add `log_to_job()` function in `backend/services/firestore_service.py` (or a small utility module)
- [ ] Takes `job_id`, `worker`, `level`, `message`, optional `metadata`
- [ ] Wraps `WorkerLogEntry.create()` + `append_log_to_subcollection()`
- [ ] Catches and suppresses errors (logging should never break the caller)
### Step 3: Update `transition_to_state` to accept metadata
- [ ] Add optional `timeline_metadata: Optional[Dict[str, Any]] = None` parameter to `job_manager.transition_to_state()`
- [ ] Pass through to `update_job_status()` → `TimelineEvent` construction
- [ ] Update `firestore_service.update_job_status()` to accept and pass `timeline_metadata`
### Step 4: Enhance edit endpoint logging
- [ ] Before cleanup: snapshot current distribution outputs (`youtube_url`, `dropbox_link`, `brand_code`, `gdrive_files`) into `previous_outputs` dict
- [ ] After each cleanup step: log result to Cloud Logging AND call `log_to_job()`
- [ ] After all cleanup: include `previous_outputs` and `cleanup_results` in the timeline event metadata
- [ ] The "Track edit initiated" timeline event gains metadata: `{initiated_by, previous_outputs, cleanup_results}`
### Step 5: Enhance admin delete-outputs endpoint logging
- [ ] Same pattern as edit endpoint: snapshot outputs, log each cleanup step, include in timeline metadata
- [ ] The "Outputs deleted by admin" timeline event gains metadata: `{deleted_by, deleted_outputs, cleanup_results}`
### Step 6: Enhance distribution logging in video_worker_orchestrator
- [ ] YouTube upload success: log video_id and video_url with metadata `{video_id, video_url}`
- [ ] Dropbox upload success: log folder_path and sharing_link with metadata `{folder_path, sharing_link}`
- [ ] GDrive upload success: log file_ids with metadata `{file_ids: {...}}`
- [ ] Brand code allocation: log with metadata `{brand_code, source: "allocated"|"recycled"}`
### Step 7: Add output details to `complete` timeline event
- [ ] When `video_worker.py` calls `transition_to_state(COMPLETE)`, pass `timeline_metadata` with distribution output details
- [ ] Include: `youtube_url`, `youtube_video_id`, `dropbox_link`, `brand_code`, `gdrive_file_ids`, `edit_count`
### Step 8: Tests
- [ ] Unit test: `TimelineEvent` with metadata serializes correctly
- [ ] Unit test: `log_to_job` writes to subcollection
- [ ] Unit test: edit endpoint includes `previous_outputs` and `cleanup_results` in timeline event
- [ ] Unit test: `complete` transition includes output metadata
- [ ] Verify existing tests still pass
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Add `metadata` field to `TimelineEvent` |
| `backend/services/firestore_service.py` | Modify | Add `log_to_job()` helper, update `update_job_status()` to accept timeline metadata |
| `backend/services/job_manager.py` | Modify | Add `timeline_metadata` param to `transition_to_state()` |
| `backend/api/routes/jobs.py` | Modify | Enhance edit endpoint with cleanup logging, output snapshots, timeline metadata |
| `backend/api/routes/admin.py` | Modify | Enhance delete-outputs endpoint similarly |
| `backend/workers/video_worker_orchestrator.py` | Modify | Add structured metadata to distribution log messages |
| `backend/workers/video_worker.py` | Modify | Pass output metadata in `complete` transition (both orchestrator and legacy paths) |
| `backend/tests/test_edit_completed_track.py` | Modify | Verify timeline metadata in edit tests |
| `backend/tests/test_observability.py` | Create | Tests for TimelineEvent metadata and log_to_job |
## Testing Strategy
- **Unit tests**: TimelineEvent metadata serialization, log_to_job helper, edit endpoint timeline metadata assertions
- **Existing test verification**: `make test` must pass — changes are additive so existing tests should be unaffected
- **No E2E tests needed**: This is purely backend logging/observability — no UI changes
## What This Does NOT Include
- **Dedicated admin UI for viewing job logs** — existing admin page already shows timeline
- **Log aggregation/search UI** — Cloud Logging and Firestore viewer already serve this purpose
- **Changing state_data overwrite behavior** — the `{**job.state_data, ...}` pattern in video_worker.py is intentional; instead we preserve history in timeline events
- **Changing outputs_deleted_at clearing** — video worker clearing it after re-processing is correct behavior; the edit history lives in timeline metadata instead
## Rollback Plan
All changes are additive (new optional field on TimelineEvent, additional log writes). Rolling back just means deploying the previous version — no data migration needed.
## Open Questions
None — the approach is straightforward and low-risk.
+279
View File
@@ -0,0 +1,279 @@
# Plan: Private (Non-Published) Tracks
**Created:** 2026-02-24
**Branch:** feat/sess-20260224-1149-private-tracks
**Status:** Implemented
## Overview
Add a `is_private` flag to jobs that changes distribution behavior:
- **Dropbox**: Uses `Tracks-NonPublished` folder (not `Tracks-Organized`) with brand code `NOMADNP` (not `NOMAD`)
- **YouTube**: Skipped entirely
- **Google Drive**: Skipped entirely
- **Email**: Completion email includes Dropbox link but no YouTube section
- **Discord**: Still notified (for internal tracking)
Available to all users at job creation and toggleable by admin on existing jobs.
## Requirements
- [ ] All users can mark a job as "Private (Non-Published)" when creating it
- [ ] Admin can toggle `is_private` on any existing job via admin dashboard
- [ ] Private jobs upload to Dropbox at `Tracks-NonPublished` with `NOMADNP-XXXX` brand codes
- [ ] Private jobs skip YouTube and Google Drive uploads entirely
- [ ] Completion emails omit YouTube section when no YouTube URL (private jobs + any other edge case)
- [ ] When admin toggles a completed job to private, existing outputs are auto-deleted
- [ ] Discord notifications still fire for private jobs
- [ ] Existing (non-private) jobs are completely unaffected (backward compatible)
## Technical Approach
### Core Design: Single Boolean Flag with Runtime Overrides
Store `is_private: bool = False` on the job model. At distribution time, a helper function returns effective distribution parameters based on this flag. This means:
- **Toggling is_private automatically changes all distribution behavior** - no need to manually update dropbox_path, brand_prefix, enable_youtube_upload, gdrive_folder_id
- **Job fields still store the "normal" defaults** - the override happens at the config-to-orchestrator boundary
- **Admin can flip one field** and everything adapts
### Key Helper Function
```python
# In job_defaults_service.py
def get_effective_distribution_for_job(job) -> EffectiveDistributionSettings:
"""Return distribution settings, applying private overrides if is_private=True."""
if getattr(job, 'is_private', False):
settings = get_settings()
return EffectiveDistributionSettings(
dropbox_path=settings.default_private_dropbox_path,
brand_prefix=settings.default_private_brand_prefix,
enable_youtube_upload=False,
gdrive_folder_id=None,
# Keep other settings from job (discord, youtube_description, etc.)
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=None,
)
# Non-private: return job's own settings
return EffectiveDistributionSettings(
dropbox_path=getattr(job, 'dropbox_path', None),
brand_prefix=getattr(job, 'brand_prefix', None),
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=getattr(job, 'youtube_description_template', None),
)
```
This helper is used in both distribution paths (orchestrator and native fallback).
### Distribution Paths That Need Changes
Two code paths handle distribution (see LESSONS-LEARNED: "Fix Both Sides of Dual Code Paths"):
1. **Orchestrator path** (`video_worker_orchestrator.py`): Primary path. `create_orchestrator_config_from_job()` uses the helper to build config.
2. **Native distribution path** (`video_worker.py` → `_handle_native_distribution()`): Fallback path. Also uses the helper instead of reading job fields directly.
### Email Template Change
Add conditional YouTube section removal in `render_template()`, matching the existing pattern for `feedback_url`:
```python
# Remove YouTube section if no URL provided
if not variables.get("youtube_url"):
result = re.sub(
r"\n*Here's the link for the karaoke video published to YouTube:\n\{youtube_url\}\n*",
'\n',
result,
flags=re.DOTALL
)
variables["youtube_url"] = ""
```
And change `render_job_completion()` to pass `None` instead of `"[YouTube URL not available]"` when youtube_url is empty.
### Admin Toggle Auto-Delete
When admin PATCH updates `is_private` from False to True on a job that has existing outputs, reuse the existing delete-outputs logic to remove YouTube + GDrive + Dropbox outputs. Admin can then "Reprocess" to re-distribute to the private Dropbox path.
## Implementation Steps
### Phase 1: Backend Model & Config (foundation)
1. [ ] **Add `is_private` to Job model** (`backend/models/job.py`)
- Add `is_private: bool = False` field after `made_for_you` field
- Add to `JobCreate` model as well
2. [ ] **Add private distribution config** (`backend/config.py`)
- Add `default_private_dropbox_path: Optional[str] = os.getenv("DEFAULT_PRIVATE_DROPBOX_PATH")`
- Add `default_private_brand_prefix: Optional[str] = os.getenv("DEFAULT_PRIVATE_BRAND_PREFIX", "NOMADNP")`
3. [ ] **Add helper function** (`backend/services/job_defaults_service.py`)
- Add `get_effective_distribution_for_job(job)` function that returns distribution settings with private overrides applied
### Phase 2: Backend API (job creation + admin)
4. [ ] **Add `is_private` to request models** (`backend/models/requests.py`)
- Add `is_private: Optional[bool] = None` to `URLSubmissionRequest`
- Add `is_private: Optional[bool] = None` to `AudioSearchRequest`
5. [ ] **Update URL creation endpoint** (`backend/api/routes/jobs.py`)
- Accept and pass through `is_private` field to `JobCreate`
6. [ ] **Update file upload endpoint** (`backend/api/routes/file_upload.py`)
- Accept `is_private` Form parameter
- Pass through to `JobCreate`
7. [ ] **Update audio search endpoint** (`backend/api/routes/audio_search.py`)
- Accept `is_private` in `AudioSearchRequest`
- Pass through to `JobCreate`
8. [ ] **Update admin endpoints** (`backend/api/routes/admin.py`)
- Add `is_private: Optional[bool] = None` to `JobUpdateRequest`
- Add `"is_private"` to `EDITABLE_JOB_FIELDS`
- In the PATCH handler: detect when `is_private` changes from False→True on a completed job, auto-delete existing outputs (call existing delete-outputs service logic)
### Phase 3: Distribution Pipeline
9. [ ] **Update orchestrator config creation** (`backend/workers/video_worker_orchestrator.py`)
- In `create_orchestrator_config_from_job()`: Use `get_effective_distribution_for_job()` helper to determine dropbox_path, brand_prefix, enable_youtube_upload, gdrive_folder_id instead of reading directly from job
10. [ ] **Update native distribution fallback** (`backend/workers/video_worker.py`)
- In `_handle_native_distribution()`: Use `get_effective_distribution_for_job()` helper instead of reading directly from job fields
### Phase 4: Email Template
11. [ ] **Add YouTube section conditional removal** (`backend/services/template_service.py`)
- In `render_template()`: Add regex to remove YouTube paragraph when `youtube_url` is empty (matching existing `feedback_url` pattern)
- In `render_job_completion()`: Pass `None` for youtube_url when empty (instead of `"[YouTube URL not available]"`)
### Phase 5: Frontend
12. [ ] **Add private checkbox to job creation form** (`frontend/components/job/JobSubmission.tsx`)
- Add "Private (Non-Published)" checkbox below the existing "Skip lyrics review" checkbox
- Show for all users (not just admin)
- Include brief description text: "Private tracks go to Dropbox only, not YouTube or Google Drive"
13. [ ] **Update API interfaces** (`frontend/lib/api.ts`)
- Add `is_private?: boolean` to all job creation option interfaces
- Add `is_private` to `JobUpdateRequest` interface
- Pass through in `uploadJob()`, `createJobFromUrl()`, `searchAudio()` methods
14. [ ] **Add private toggle to admin job detail** (`frontend/app/admin/jobs/page.tsx`)
- Add `is_private` as an inline-editable field in the job overview section (use Switch component)
- Show current state with visual indicator (e.g., lock icon for private jobs)
- When toggling to private on a completed job, show confirmation dialog about auto-deleting outputs
### Phase 6: Environment Configuration
15. [ ] **Update CI/deployment config** (`.github/workflows/ci.yml` or Cloud Run env)
- Add `DEFAULT_PRIVATE_DROPBOX_PATH` environment variable (e.g., `/Karaoke/Tracks-NonPublished`)
- Add `DEFAULT_PRIVATE_BRAND_PREFIX` environment variable (e.g., `NOMADNP`)
- These should be configured in Cloud Run service and CI test environment
### Phase 7: Testing
16. [ ] **Unit tests for private distribution logic** (`backend/tests/test_private_tracks.py`)
- Test `get_effective_distribution_for_job()` with private and non-private jobs
- Test `create_orchestrator_config_from_job()` applies private overrides correctly
- Test `_run_distribution()` skips YouTube and GDrive when private config applied
- Test `_upload_to_dropbox()` uses private path and brand prefix
- Test brand code generation with NOMADNP prefix in Tracks-NonPublished
- Test `_handle_native_distribution()` respects private overrides
- Test non-private jobs are completely unaffected (regression tests)
17. [ ] **Unit tests for email changes** (add to `backend/tests/test_email_service.py` or `test_template_service.py`)
- Test YouTube section removed from email when youtube_url is None
- Test YouTube section present when youtube_url is provided
- Test Dropbox section still present for private jobs
- Test full email rendering for private job scenario
18. [ ] **Unit tests for admin toggle** (add to `backend/tests/test_admin.py` or new file)
- Test admin can update is_private field
- Test toggling is_private to True on completed job triggers output deletion
- Test toggling is_private to True on non-completed job does NOT delete outputs
- Test toggling is_private to False does nothing special
19. [ ] **Integration test** (`backend/tests/emulator/` or `test_distribution_services.py`)
- Test full distribution pipeline with is_private=True: verify Dropbox path, brand prefix, YouTube skip, GDrive skip
20. [ ] **Frontend E2E regression test** (`frontend/e2e/regression/`)
- Test private checkbox appears in job creation form
- Test private toggle in admin dashboard
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Add `is_private: bool = False` to Job and JobCreate |
| `backend/config.py` | Modify | Add `default_private_dropbox_path` and `default_private_brand_prefix` |
| `backend/services/job_defaults_service.py` | Modify | Add `get_effective_distribution_for_job()` helper |
| `backend/models/requests.py` | Modify | Add `is_private` to URL and AudioSearch request models |
| `backend/api/routes/jobs.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/file_upload.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/audio_search.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/admin.py` | Modify | Add `is_private` to editable fields + auto-delete logic |
| `backend/workers/video_worker_orchestrator.py` | Modify | Use helper in `create_orchestrator_config_from_job()` |
| `backend/workers/video_worker.py` | Modify | Use helper in `_handle_native_distribution()` |
| `backend/services/template_service.py` | Modify | Conditional YouTube section removal |
| `frontend/components/job/JobSubmission.tsx` | Modify | Add private checkbox |
| `frontend/lib/api.ts` | Modify | Add `is_private` to interfaces and API calls |
| `frontend/app/admin/jobs/page.tsx` | Modify | Add private toggle to admin detail |
| `backend/tests/test_private_tracks.py` | Create | Comprehensive unit tests for private track logic |
## Testing Strategy
### Unit Tests (~25-30 tests in `test_private_tracks.py`)
- `get_effective_distribution_for_job()` with private=True and False
- `create_orchestrator_config_from_job()` private overrides (YouTube=False, GDrive=None, private Dropbox path, NOMADNP brand)
- Distribution stage skips: YouTube not called, GDrive not called, Dropbox IS called with correct path
- Brand code generation uses NOMADNP prefix and Tracks-NonPublished path
- Native distribution fallback applies same overrides
- **Regression tests**: Non-private jobs have unchanged behavior (same paths, same brand code, YouTube+GDrive+Dropbox all fire)
### Email Template Tests (~5 tests)
- YouTube section stripped when youtube_url is None
- YouTube section preserved when youtube_url has value
- Dropbox section always present
- Full private job email rendering (Dropbox + no YouTube)
- Full non-private job email rendering (both present) - regression
### Admin Toggle Tests (~5 tests)
- is_private editable via PATCH
- Toggle to private on completed job deletes outputs
- Toggle to private on pending/in-progress job does NOT delete
- Toggle to non-private does nothing special
### Frontend Tests
- E2E: Private checkbox visible and functional in job creation
- E2E: Admin toggle visible and includes confirmation dialog
### Manual Verification
- Create a private job end-to-end, verify Dropbox folder in Tracks-NonPublished with NOMADNP code
- Verify completion email has Dropbox link but no YouTube section
- Toggle existing completed job to private, verify outputs deleted
## Regression Risk Analysis
| Risk | Mitigation |
|------|-----------|
| Non-private jobs affected | `is_private` defaults to `False`; helper returns unchanged values for non-private. Explicit regression tests. |
| Dual code paths (orchestrator vs native) | Both paths use same helper function. Tests cover both paths. (Lesson: "Fix Both Sides of Dual Code Paths") |
| Email template regex breaks other content | Regex targets specific YouTube paragraph text. Tests verify non-YouTube content untouched. |
| Brand code namespace collision | NOMADNP codes are in a completely separate Dropbox folder (Tracks-NonPublished), no overlap with NOMAD codes in Tracks-Organized |
| Admin auto-delete on wrong job | Only triggers when `is_private` changes from False→True AND job has terminal status (complete). Confirmation dialog in frontend. |
| Made-for-you orders | MFY webhook doesn't set is_private (defaults False). Admin can toggle after. No change to MFY flow. |
| Existing Firestore documents | Missing `is_private` field defaults to `False` via model default. No migration needed. |
## Open Questions
- [x] Who sees private toggle? → **Everyone** (all users at creation, admin on existing)
- [x] Auto-delete on toggle? → **Yes**, auto-delete all outputs when toggling to private on completed job
- [x] Email format? → **Remove YouTube section** when no YouTube URL (improves all edge cases)
- [x] Discord? → **Still notify** (useful for internal tracking)
## Rollback Plan
1. **Feature flag**: The `is_private` field defaults to `False`, so setting `DEFAULT_PRIVATE_DROPBOX_PATH` to empty/None effectively disables private distribution
2. **No schema migration needed**: Missing `is_private` field on existing Firestore documents defaults to `False`
3. **Email change is backward-compatible**: The YouTube section removal only happens when youtube_url is None, which is already an edge case showing "[YouTube URL not available]"
4. **Frontend**: Private checkbox is additive UI, removing it just means users can't create new private jobs
@@ -0,0 +1,239 @@
# Plan: Song Lookup & Autocomplete + Community Version Detection
**Created:** 2026-03-02
**Branch:** feat/sess-20260302-2324-song-lookup-autocomplete
**Status:** Implementation Complete
## Overview
Add two features to the karaoke-gen job creation flow:
1. **Artist/title autocomplete** using karaoke-decide's MusicBrainz + Spotify catalog data, so users get properly-styled artist/title text (e.g. "Fleetwood Mac" instead of "fleetwood mac")
2. **Community karaoke version detection** that warns users when a high-quality community karaoke version already exists on YouTube (via karaokenerds data), saving them credits and time
Both features are **optional and non-blocking** — users can still type whatever they want and proceed even if a community version exists.
## Requirements
- [ ] Artist field autocomplete across all 3 job creation tabs (Search, Upload, URL)
- [ ] Title field autocomplete (optionally filtered by selected artist) across all 3 tabs
- [ ] Autocomplete uses canonical MusicBrainz/Spotify text styling
- [ ] Autocomplete is optional — users can ignore suggestions and type freely
- [ ] Community version check triggers when both artist and title are filled
- [ ] If community karaoke version exists, show a dismissible banner with YouTube link(s)
- [ ] Community check does not block job submission — just informs the user
- [ ] Graceful degradation if karaoke-decide API or karaokenerds.com is down
## Architecture Decisions
### How to access karaoke-decide catalog data
**Decision: Proxy through karaoke-gen backend**
Both services share the same GCP project (`nomadkaraoke`), but the frontend at `gen.nomadkaraoke.com` can't directly call `decide.nomadkaraoke.com/api/` due to CORS (different origins). Options considered:
| Option | Pros | Cons |
|--------|------|------|
| **Proxy via karaoke-gen backend** | No CORS issues, no changes to karaoke-decide, frontend already calls api.nomadkaraoke.com | Added latency (extra hop), karaoke-decide dependency |
| Direct frontend calls + CORS | Simpler, no proxy code | Requires karaoke-decide CORS changes, cross-project coupling |
| Query BigQuery directly from karaoke-gen | No dependency on karaoke-decide service | Duplicates query logic, BigQuery setup needed, higher latency |
The proxy approach is cleanest: karaoke-gen frontend → karaoke-gen backend → karaoke-decide API. Response caching in the proxy minimizes latency impact.
### How to detect community karaoke versions
**Decision: Scrape karaokenerds.com from karaoke-gen backend** (same approach as kjbox)
The `karaokenerds_raw` BigQuery table has the catalog but NOT community status. Community status is determined by an `<img class="check">` badge in karaokenerds.com's HTML, only available via web scraping. The kjbox codebase has proven, well-tested scraping logic we can port directly.
Search URL: `https://karaokenerds.com/Search?query={artist}+{title}&webFilter=OnlyWeb`
Returns: songs with tracks, each track having `brand_name`, `brand_code`, `youtube_url`, `is_community`.
### Autocomplete UI component
**Decision: Use existing shadcn Command (cmdk) + Popover components**
The frontend already has `command.tsx` and `popover.tsx` from shadcn/ui. The standard combobox pattern (Popover + Command) provides:
- Keyboard navigation (arrow keys, enter to select)
- Debounced search-as-you-type
- Clean dropdown with loading states
- Works inline within forms (not modal)
## Technical Approach
### Data flow
```
User types in artist field
→ debounced (300ms) API call to GET /api/catalog/artists?q=fle
→ karaoke-gen backend proxies to decide.nomadkaraoke.com/api/catalog/artists?q=fle
→ returns [{ name: "Fleetwood Mac", mbid: "...", popularity: 85 }, ...]
→ user selects "Fleetwood Mac" → field populated with canonical name
User types in title field
→ debounced (300ms) API call to GET /api/catalog/tracks?q=dreams&artist=Fleetwood+Mac
→ returns [{ track_name: "Dreams", artist_name: "Fleetwood Mac", popularity: 82 }, ...]
→ user selects "Dreams" → field populated with canonical title
Both fields filled → community check fires
→ POST /api/catalog/community-check { artist: "Fleetwood Mac", title: "Dreams" }
→ backend scrapes karaokenerds.com for "Fleetwood Mac Dreams" with webFilter=OnlyWeb
→ if community tracks found with YouTube URLs, return them
→ frontend shows banner: "A community karaoke version already exists! [Watch on YouTube →]"
```
### Caching strategy
| Endpoint | Cache TTL | Scope |
|----------|-----------|-------|
| Artist search proxy | 5 min | In-memory (backend) |
| Track search proxy | 5 min | In-memory (backend) |
| Community check | 1 hour | In-memory (backend) |
### karaoke-decide API endpoints used
These are all **public, no auth required**:
- `GET /api/catalog/artists?q={query}&limit=10` — MBID-first artist search
- Returns: `[{ mbid, name, disambiguation, artist_type, tags, spotify_id, popularity, genres }]`
- `GET /api/catalog/tracks?q={query}&limit=10` — Track search (Spotify data)
- Returns: `[{ track_id, track_name, artist_name, artist_id, popularity, duration_ms, explicit }]`
## Implementation Steps
### Phase 1: Backend — Catalog proxy + community check endpoints
1. [ ] **Add catalog proxy service** — `backend/services/catalog_proxy_service.py`
- HTTP client (httpx async) calling karaoke-decide API
- In-memory TTL cache for responses (5 min)
- Configurable karaoke-decide base URL
- Graceful error handling (return empty results on failure)
2. [ ] **Port karaokenerds scraping logic** — `backend/services/karaokenerds_service.py`
- Port `search()`, `parse_results()`, `_parse_tracks()`, `_parse_single_track()`, `_clean_youtube_url()` from `kjbox/kj-controller/karaoke_nerds.py`
- Add async support (httpx instead of requests)
- In-memory TTL cache (1 hour)
- Return only community tracks with YouTube URLs
3. [ ] **Add catalog routes** — `backend/api/routes/catalog.py`
- `GET /api/catalog/artists?q=&limit=10` — proxy to karaoke-decide
- `GET /api/catalog/tracks?q=&artist=&limit=10` — proxy to karaoke-decide
- `POST /api/catalog/community-check` — scrape karaokenerds.com
- All endpoints require auth (same as job creation)
- Pydantic request/response models
4. [ ] **Register routes in main.py** — add `catalog.router` to app
5. [ ] **Add config** — `backend/config.py`
- `karaoke_decide_api_url` setting (default: `https://decide.nomadkaraoke.com`)
6. [ ] **Add `beautifulsoup4` + `httpx` dependencies** to `pyproject.toml` (if not already present)
### Phase 2: Frontend — Autocomplete component
7. [ ] **Create `AutocompleteInput` component** — `frontend/components/ui/autocomplete-input.tsx`
- Wraps an Input field with a Popover + Command dropdown
- Props: `value`, `onChange`, `onSelect`, `fetchSuggestions(query) => Promise<Suggestion[]>`, `placeholder`, `disabled`, `renderItem`
- Debounced search (300ms) triggered on input change
- Shows loading spinner while fetching
- Shows "No results" when search returns empty
- Keyboard navigation (up/down arrows, enter to select, escape to close)
- Clicking outside closes dropdown
- When user selects a suggestion, calls `onSelect(suggestion)` and populates input
- When user types without selecting, uses their raw text (autocomplete is optional)
8. [ ] **Add catalog API functions** — `frontend/lib/api.ts`
- `searchArtists(query: string, limit?: number)` → `GET /api/catalog/artists`
- `searchTracks(query: string, artist?: string, limit?: number)` → `GET /api/catalog/tracks`
- `checkCommunityVersions(artist: string, title: string)` → `POST /api/catalog/community-check`
### Phase 3: Frontend — Integrate autocomplete into job creation
9. [ ] **Replace artist Input fields with AutocompleteInput** in `JobSubmission.tsx`
- All 3 tabs: `searchArtist`, `uploadArtist`, `youtubeArtist`
- Fetch suggestions from `api.searchArtists(query)`
- On select: populate artist field with canonical name
- Show artist name + optional disambiguation in dropdown items
10. [ ] **Replace title Input fields with AutocompleteInput** in `JobSubmission.tsx`
- All 3 tabs: `searchTitle`, `uploadTitle`, `youtubeTitle`
- Fetch suggestions from `api.searchTracks(query, selectedArtist)`
- On select: populate title field with canonical name
- Show track name + artist name + duration in dropdown items
### Phase 4: Frontend — Community version detection
11. [ ] **Create `CommunityVersionBanner` component** — `frontend/components/job/CommunityVersionBanner.tsx`
- Shows when community karaoke version(s) found
- Displays: "A karaoke version of this song already exists!"
- Lists community tracks with brand name and YouTube link
- "Watch on YouTube" button for top community version
- Dismissible (X button or "Continue anyway" link)
- Styled with green border (matching kjbox community track styling)
12. [ ] **Add community check logic to `JobSubmission.tsx`** (Search tab only)
- Debounced check (500ms) when both artist AND title fields have values
- Only triggers when values change (not on every keystroke)
- Shows `CommunityVersionBanner` between the form fields and the submit button
- Does NOT block form submission
- Clears when artist or title changes
### Phase 5: Testing
13. [ ] **Backend unit tests** — `backend/tests/unit/test_catalog_routes.py`
- Test proxy endpoints with mocked karaoke-decide responses
- Test community check with mocked karaokenerds HTML
- Test error handling (karaoke-decide down, karaokenerds down)
- Test caching behavior
14. [ ] **Backend unit tests** — `backend/tests/unit/test_karaokenerds_service.py`
- Port relevant tests from `kjbox/kj-controller/tests/unit/test_karaoke_nerds.py`
- Test HTML parsing, community detection, YouTube URL cleanup
15. [ ] **Frontend component tests** — `frontend/components/ui/__tests__/autocomplete-input.test.tsx`
- Test suggestion fetching and display
- Test keyboard navigation
- Test selection behavior
- Test free-text input (no selection)
16. [ ] **E2E tests** — `frontend/e2e/job-creation-autocomplete.spec.ts`
- Test autocomplete appears on typing
- Test selecting from autocomplete populates field
- Test community banner appears for known songs
- Test form submission works with and without autocomplete selection
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/services/catalog_proxy_service.py` | Create | HTTP client for karaoke-decide API with caching |
| `backend/services/karaokenerds_service.py` | Create | Karaokenerds.com scraper (ported from kjbox) |
| `backend/api/routes/catalog.py` | Create | Proxy endpoints for artist/track search + community check |
| `backend/main.py` | Modify | Register catalog router |
| `backend/config.py` | Modify | Add karaoke_decide_api_url setting |
| `pyproject.toml` | Modify | Add beautifulsoup4 dependency (if missing) |
| `frontend/components/ui/autocomplete-input.tsx` | Create | Reusable autocomplete input component |
| `frontend/components/job/CommunityVersionBanner.tsx` | Create | Community version warning banner |
| `frontend/components/job/JobSubmission.tsx` | Modify | Replace Input with AutocompleteInput, add community check |
| `frontend/lib/api.ts` | Modify | Add catalog API functions |
| `backend/tests/unit/test_catalog_routes.py` | Create | Backend route tests |
| `backend/tests/unit/test_karaokenerds_service.py` | Create | Scraper unit tests |
| `frontend/components/ui/__tests__/autocomplete-input.test.tsx` | Create | Component tests |
| `frontend/e2e/job-creation-autocomplete.spec.ts` | Create | E2E tests |
## Decisions (Resolved)
- **Track search scope**: All Spotify tracks (256M) — better coverage for new/obscure songs
- **Community check scope**: Search tab only (Upload/URL users already have their audio)
- **Rate limiting**: 20 requests/min per user on catalog proxy endpoints (prevents abuse of karaokenerds scraping). All endpoints require auth.
- **"Display As" fields**: No autocomplete — they're for deliberate overrides like soundtrack names
- **CORS**: Proxy approach (no changes needed to karaoke-decide)
## Rollback Plan
- Feature is entirely additive — no existing functionality is modified
- Autocomplete fields fall back to plain text Input if API calls fail
- Community check is non-blocking — if it fails, form works normally
- All new endpoints are behind the existing auth middleware
- Can disable by reverting the frontend changes (backend proxy is harmless if unused)
+37
View File
@@ -0,0 +1,37 @@
#!/bin/bash
# Auto-translate when English translation files change.
# Uses GCS cache so this is near-instant for small changes.
# Skip with: git commit --no-verify
set -e
STAGED_EN=$(git diff --cached --name-only --diff-filter=ACM | grep -E '(messages|translations)/en\.json$' || true)
if [ -z "$STAGED_EN" ]; then
exit 0
fi
echo "i18n: English translations changed — running translation pipeline..."
for en_file in $STAGED_EN; do
dir=$(dirname "$en_file")
if [[ "$dir" == frontend/messages ]]; then
script="frontend/scripts/translate.py"
elif [[ "$dir" == backend/translations ]]; then
script="frontend/scripts/translate.py"
else
echo " Warning: Unknown messages dir $dir, skipping"
continue
fi
echo " Translating $dir..."
if python "$script" --messages-dir "$dir" --target all --skip-review 2>&1 | tail -5; then
git add "$dir"/*.json
echo " Done: $dir"
else
echo " Warning: Translation failed for $dir (GCP auth may be missing). CI will catch this."
fi
done
echo "i18n: Translation complete."
+576 -215
View File
File diff suppressed because it is too large. Load diff
+337
View File
@@ -0,0 +1,337 @@
name: E2E Daily Test (Payment + Happy Path)
# Full production E2E: buy credits via real Stripe, then use them for karaoke generation.
# Replaces the previous e2e-happy-path.yml with an additional payment stage.
on:
schedule:
- cron: '0 6 * * *'
workflow_dispatch:
inputs:
debug_mode:
description: 'Enable debug mode (slower, more screenshots)'
required: false
default: 'false'
type: boolean
browser:
description: 'Browser to test with'
required: false
default: 'chromium'
type: choice
options:
- chromium
- firefox
- webkit
skip_payment:
description: 'Skip Stage 1 (payment) — use admin-granted credits for happy path only'
required: false
default: 'false'
type: boolean
use_test_token:
description: 'Skip signup entirely — use E2E_TEST_TOKEN (for quick iterations)'
required: false
default: 'false'
type: boolean
permissions:
contents: read
jobs:
# =========================================================================
# Stage 1: Purchase credits via real Stripe Checkout
# =========================================================================
e2e-credit-purchase:
name: "Stage 1: Credit Purchase"
runs-on: ubuntu-latest
timeout-minutes: 15
if: ${{ github.event.inputs.skip_payment != 'true' && github.event.inputs.use_test_token != 'true' }}
outputs:
session_token: ${{ steps.extract_token.outputs.token }}
result: ${{ steps.test_result.outputs.result }}
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: npx playwright install --with-deps ${{ inputs.browser || 'chromium' }}
- name: Create test-results directory
working-directory: frontend
run: mkdir -p test-results
- name: Ensure testmail.app domain is allowlisted
run: |
curl -sf -X POST \
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
-H "Content-Type: application/json" \
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
- name: Run Credit Purchase Test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
E2E_STRIPE_CARD_NUMBER: ${{ secrets.E2E_STRIPE_CARD_NUMBER }}
E2E_STRIPE_CARD_EXPIRY: ${{ secrets.E2E_STRIPE_CARD_EXPIRY }}
E2E_STRIPE_CARD_CVC: ${{ secrets.E2E_STRIPE_CARD_CVC }}
E2E_STRIPE_CARDHOLDER_NAME: ${{ secrets.E2E_STRIPE_CARDHOLDER_NAME }}
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
run: |
echo "=== Stage 1: Credit Purchase via Real Stripe ==="
echo "Time: $(date)"
npx playwright test credit-purchase-real.spec.ts \
--config=playwright.production.config.ts \
--project=${{ inputs.browser || 'chromium' }} \
--reporter=list \
--timeout=600000 \
2>&1 | tee test-output-payment.log
continue-on-error: true
- name: Check test result
id: test_result
working-directory: frontend
run: |
if grep -q "CREDIT PURCHASE TEST PASSED" test-output-payment.log; then
echo "result=success" >> $GITHUB_OUTPUT
else
echo "result=failure" >> $GITHUB_OUTPUT
fi
- name: Extract session token
id: extract_token
if: steps.test_result.outputs.result == 'success'
working-directory: frontend
run: |
if [ -f test-results/e2e-session-token.txt ]; then
TOKEN=$(cat test-results/e2e-session-token.txt)
echo "::add-mask::$TOKEN"
echo "token=$TOKEN" >> $GITHUB_OUTPUT
echo "Session token extracted for Stage 2"
else
echo "No session token file found"
echo "token=" >> $GITHUB_OUTPUT
fi
- name: Upload artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-payment-results-${{ github.run_number }}
path: |
frontend/test-results/
frontend/test-output-payment.log
retention-days: 30
- name: Discord notification (failure)
if: steps.test_result.outputs.result == 'failure'
run: |
curl -s -H "Content-Type: application/json" \
-d "{\"embeds\": [{\"title\": \"🚨 E2E Daily Test FAILED — Stage 1: Credit Purchase\", \"description\": \"Payment flow is broken. Customers may not be receiving credits.\n\n**Run:** #${{ github.run_number }}\n**Details:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": 15158332}]}" \
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
- name: Fail if test failed
if: steps.test_result.outputs.result == 'failure'
run: exit 1
# =========================================================================
# Stage 2: Happy Path (uses credit from Stage 1)
# =========================================================================
e2e-happy-path:
name: "Stage 2: Happy Path"
runs-on: ubuntu-latest
timeout-minutes: 90
needs: e2e-credit-purchase
if: |
always() && (
needs.e2e-credit-purchase.result == 'success' ||
github.event.inputs.skip_payment == 'true' ||
github.event.inputs.use_test_token == 'true'
)
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: npx playwright install --with-deps ${{ inputs.browser || 'chromium' }}
- name: Create test-results directory
working-directory: frontend
run: mkdir -p test-results
- name: Ensure testmail.app domain is allowlisted
run: |
curl -sf -X POST \
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
-H "Content-Type: application/json" \
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
- name: Refresh E2E session token (when use_test_token enabled)
id: refresh_token
if: github.event.inputs.use_test_token == 'true'
run: |
RESPONSE=$(curl -sf -X POST \
"https://api.nomadkaraoke.com/api/admin/users/e2e-test-runner@nomadkaraoke.com/impersonate" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
-H "Content-Type: application/json")
TOKEN=$(echo "$RESPONSE" | python3 -c 'import json,sys; print(json.load(sys.stdin)["session_token"])')
echo "::add-mask::$TOKEN"
echo "token=$TOKEN" >> $GITHUB_OUTPUT
- name: Run Happy Path Test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_TEST_TOKEN: ${{ steps.refresh_token.outputs.token || needs.e2e-credit-purchase.outputs.session_token || '' }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
run: |
echo "=== Stage 2: Happy Path ==="
echo "Time: $(date)"
TOKEN_SOURCE="none"
if [ -n "${{ steps.refresh_token.outputs.token }}" ]; then
TOKEN_SOURCE="use_test_token (impersonation)"
elif [ -n "${{ needs.e2e-credit-purchase.outputs.session_token }}" ]; then
TOKEN_SOURCE="Stage 1 (credit purchase)"
fi
echo "Token source: $TOKEN_SOURCE"
npx playwright test happy-path-real-user.spec.ts \
--config=playwright.production.config.ts \
--project=${{ inputs.browser || 'chromium' }} \
--reporter=list \
--timeout=3600000 \
2>&1 | tee test-output-happy-path.log
continue-on-error: true
- name: Check test result
id: test_result
working-directory: frontend
run: |
if grep -q "passed" test-output-happy-path.log && ! grep -q "failed" test-output-happy-path.log; then
echo "result=success" >> $GITHUB_OUTPUT
else
echo "result=failure" >> $GITHUB_OUTPUT
fi
- name: Upload test artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-results-${{ github.run_number }}
path: |
frontend/test-results/
frontend/playwright-report/
frontend/test-output-happy-path.log
retention-days: 30
- name: Upload video recordings
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-videos-${{ github.run_number }}
path: frontend/test-results/**/*.webm
retention-days: 30
if-no-files-found: ignore
- name: Create summary
if: always()
run: |
STAGE1="${{ needs.e2e-credit-purchase.result || 'skipped' }}"
STAGE2="${{ steps.test_result.outputs.result || 'unknown' }}"
echo "## E2E Daily Test Results" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "| Stage | Result |" >> $GITHUB_STEP_SUMMARY
echo "|-------|--------|" >> $GITHUB_STEP_SUMMARY
echo "| Stage 1: Credit Purchase | ${STAGE1} |" >> $GITHUB_STEP_SUMMARY
echo "| Stage 2: Happy Path | ${STAGE2} |" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
- name: Discord notification (failure)
if: steps.test_result.outputs.result == 'failure'
run: |
curl -s -H "Content-Type: application/json" \
-d "{\"embeds\": [{\"title\": \"🚨 E2E Daily Test FAILED — Stage 2: Happy Path\", \"description\": \"Karaoke generation flow is broken.\n\n**Run:** #${{ github.run_number }}\n**Details:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": 15158332}]}" \
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
- name: Fail if test failed
if: steps.test_result.outputs.result == 'failure'
run: exit 1
# =========================================================================
# Final: Send notifications
# =========================================================================
notify:
name: "Notify"
runs-on: ubuntu-latest
needs: [e2e-credit-purchase, e2e-happy-path]
if: always()
steps:
- name: Discord notification (success)
if: needs.e2e-happy-path.result == 'success'
run: |
STAGE1="${{ needs.e2e-credit-purchase.result || 'skipped' }}"
curl -s -H "Content-Type: application/json" \
-d "{\"embeds\": [{\"title\": \"✅ E2E Daily Test PASSED\", \"description\": \"Payment: ${STAGE1} | Happy Path: success\n\n**Run:** #${{ github.run_number }}\", \"color\": 3066993}]}" \
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
- name: Send email notification
if: github.event_name == 'schedule'
uses: dawidd6/action-send-mail@v3
with:
server_address: smtp.sendgrid.net
server_port: 587
username: apikey
password: ${{ secrets.SENDGRID_API_KEY }}
subject: "E2E Daily Test ${{ needs.e2e-happy-path.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
to: andrew@nomadkaraoke.com
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
body: |
E2E Daily Test Results
======================
Stage 1 (Credit Purchase): ${{ needs.e2e-credit-purchase.result || 'skipped' }}
Stage 2 (Happy Path): ${{ needs.e2e-happy-path.result || 'skipped' }}
Run: #${{ github.run_number }}
Trigger: ${{ github.event_name }}
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
-174
View File
@@ -1,174 +0,0 @@
name: E2E Happy Path Test
# Comprehensive end-to-end test that validates the full karaoke generation journey
# using ONLY browser interactions (no API shortcuts)
#
# See docs/E2E-HAPPY-PATH-TEST-SPEC.md for full requirements
on:
# Daily run at 6 AM UTC
schedule:
- cron: '0 6 * * *'
# Manual trigger with optional parameters
workflow_dispatch:
inputs:
debug_mode:
description: 'Enable debug mode (slower, more screenshots)'
required: false
default: 'false'
type: boolean
browser:
description: 'Browser to test with'
required: false
default: 'chromium'
type: choice
options:
- chromium
- firefox
- webkit
use_test_token:
description: 'Skip signup and use E2E_TEST_TOKEN (for iterations)'
required: false
default: 'false'
type: boolean
permissions:
contents: read
jobs:
e2e-happy-path:
name: "Full Happy Path (Real User)"
runs-on: ubuntu-latest
timeout-minutes: 90 # 1.5 hours max (test itself is 60 min)
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: |
BROWSER=${{ inputs.browser || 'chromium' }}
npx playwright install --with-deps $BROWSER
- name: Create test-results directory
working-directory: frontend
run: mkdir -p test-results
- name: Run E2E Happy Path Test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_TEST_TOKEN: ${{ inputs.use_test_token && secrets.E2E_TEST_TOKEN || '' }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
run: |
echo "Starting E2E Happy Path Test"
echo "Browser: ${{ inputs.browser || 'chromium' }}"
echo "Debug mode: ${{ inputs.debug_mode || 'false' }}"
echo "Use test token: ${{ inputs.use_test_token || 'false' }}"
echo "Time: $(date)"
echo ""
# Run the real user happy path test
npx playwright test happy-path-real-user.spec.ts \
--config=playwright.production.config.ts \
--project=${{ inputs.browser || 'chromium' }} \
--reporter=list \
--timeout=3600000 \
2>&1 | tee test-output.log
continue-on-error: true
- name: Check test result
id: test_result
working-directory: frontend
run: |
if grep -q "passed" test-output.log && ! grep -q "failed" test-output.log; then
echo "result=success" >> $GITHUB_OUTPUT
echo "Test PASSED"
else
echo "result=failure" >> $GITHUB_OUTPUT
echo "Test FAILED - see artifacts for details"
fi
- name: Upload test artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-results-${{ github.run_number }}
path: |
frontend/test-results/
frontend/playwright-report/
frontend/test-output.log
retention-days: 30
- name: Upload video recordings
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-videos-${{ github.run_number }}
path: frontend/test-results/**/*.webm
retention-days: 30
if-no-files-found: ignore
- name: Create summary
if: always()
run: |
echo "## E2E Happy Path Test Results" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
echo "| Result | ${{ steps.test_result.outputs.result || 'unknown' }} |" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
if [ -f frontend/test-output.log ]; then
echo "### Test Output (last 100 lines)" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
tail -100 frontend/test-output.log >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
fi
- name: Send email notification
if: always() && github.event_name == 'schedule'
uses: dawidd6/action-send-mail@v3
with:
server_address: smtp.sendgrid.net
server_port: 587
username: apikey
password: ${{ secrets.SENDGRID_API_KEY }}
subject: "E2E Happy Path Test ${{ steps.test_result.outputs.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
to: andrew@nomadkaraoke.com
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
body: |
E2E Happy Path Test Results
===========================
Status: ${{ steps.test_result.outputs.result == 'success' && 'PASSED ✅' || 'FAILED ❌' }}
Run: #${{ github.run_number }}
Trigger: ${{ github.event_name }}
Time: ${{ github.event.repository.updated_at }}
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
This is an automated notification from the daily E2E happy path test.
- name: Fail if test failed
if: steps.test_result.outputs.result == 'failure'
run: exit 1
+1
View File
@@ -232,3 +232,4 @@ flacfetch
# Test output directories
Rick Astley*/
.superpowers/
+1
View File
@@ -0,0 +1 @@
[{"name":"generate-buildid","duration":272,"timestamp":244518298649,"id":4,"parentId":1,"tags":{},"startTime":1775262331361,"traceId":"b3ef0d179e8acedf"},{"name":"load-custom-routes","duration":314,"timestamp":244518299029,"id":5,"parentId":1,"tags":{},"startTime":1775262331361,"traceId":"b3ef0d179e8acedf"},{"name":"create-dist-dir","duration":333,"timestamp":244518299363,"id":6,"parentId":1,"tags":{},"startTime":1775262331362,"traceId":"b3ef0d179e8acedf"},{"name":"clean","duration":358,"timestamp":244518300558,"id":7,"parentId":1,"tags":{},"startTime":1775262331363,"traceId":"b3ef0d179e8acedf"},{"name":"next-build","duration":1670851,"timestamp":244516630180,"id":1,"tags":{"buildMode":"default","version":"16.2.2","bundler":"turbopack","failed":true},"startTime":1775262329693,"traceId":"b3ef0d179e8acedf"}]
+1
View File
@@ -0,0 +1 @@
[{"name":"next-build","duration":1670851,"timestamp":244516630180,"id":1,"tags":{"buildMode":"default","version":"16.2.2","bundler":"turbopack","failed":true},"startTime":1775262329693,"traceId":"b3ef0d179e8acedf"}]
+25 -7
View File
@@ -19,6 +19,8 @@
| **Testing & code quality** | `docs/TESTING.md` |
| API reference | `docs/API.md` |
| Past learnings | `docs/LESSONS-LEARNED.md` |
| **Operational runbooks** | `docs/TROUBLESHOOTING.md` |
| GDrive validator & gaps | `docs/GDRIVE-VALIDATOR.md` |
| **Product communication** | `docs/PRODUCT-COMMUNICATION-GUIDE.md` |
| Brand style guide | `docs/BRAND-STYLE-GUIDE.md` |
@@ -26,7 +28,7 @@
- **Backend**: FastAPI on Cloud Run, Firestore, GCS, Secret Manager
- **Frontend**: Next.js on Cloudflare Pages
- **Processing**: Modal (GPU audio separation), AudioShake/Whisper (transcription)
- **Processing**: Cloud Run GPU (L4 audio separation), AudioShake/Whisper (transcription)
- **Infra**: Pulumi IaC, GitHub Actions CI/CD
## Essential Rules
@@ -90,11 +92,25 @@ See `docs/TESTING.md` § "Ad-Hoc Production Debugging" for full details.
### Infrastructure
- **All GCP changes via Pulumi PRs** - changes in `infrastructure/` deploy automatically on merge to main
- **Always run `pulumi up` locally BEFORE merging PRs** - CI deploys run on spot VMs that can be preempted mid-apply, causing partial state. Apply locally first, then merge (CI re-runs as no-op)
- Never modify GCP resources directly via console or `gcloud` CLI
- `gcloud` CLI for reading/debugging only (e.g., checking logs, SSH to VMs)
- Local `pulumi preview` works but `pulumi up` requires broader permissions than CI
- Stop and notify user on auth issues
### Internationalization (i18n)
- All user-facing strings live in `frontend/messages/{locale}.json` (33 locales), NOT in components
- Backend strings live in `backend/translations/{locale}.json` (currently 3 locales: en, es, de)
- Components use `useTranslations('namespace')` from next-intl
- Pages are under `frontend/src/app/[locale]/` — locale-aware routing
- Admin pages (under `/admin/`, `/app/`) use DefaultIntlProvider (English-only, excluded from i18n)
- Internal links use `Link` from `@/i18n/routing` (locale-aware)
- After adding/changing English strings: `python frontend/scripts/translate.py --messages-dir frontend/messages --target all`
- Translation uses Gemini 3.1 Pro via Vertex AI with GCS cache for efficiency
- CI validates all locale files have matching keys — **PR will fail if translations are missing**
- Pre-commit hook auto-translates when en.json changes (enable: `git config core.hooksPath .githooks`)
- Don't hardcode user-facing strings — add to `messages/en.json` and use `t('key')`
## API Authentication (for Agents)
When you need to call backend APIs programmatically:
@@ -128,15 +144,17 @@ EOF
### GCE Encoding Worker
```bash
# SSH to restart service (clears in-memory job queue)
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
--command="sudo systemctl restart encoding-worker"
# Check health
# Check health (includes wheel_version)
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
--command="curl -s http://localhost:8080/health"
# Restart service (pick up new wheel after a deploy)
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
--command="sudo systemctl restart encoding-worker"
```
For job stuck at `encoding` status, see [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md).
### What Doesn't Work
- `gcloud auth print-identity-token` - Wrong token type for internal endpoints
- Cloud Tasks without proper service account - Gets 401 from Cloud Run
+34 -11
View File
@@ -1,18 +1,25 @@
# Makefile for karaoke-gen project
# Run `make help` to see available commands
.PHONY: help install install-backend install-frontend build-frontend test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop
.PHONY: help install install-backend install-frontend build-frontend dev-install test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop
# Per-stage timeout in seconds (default: 10 minutes)
# Prevents hangs from stale dev servers, broken emulators, etc.
# Override with: make test TEST_TIMEOUT=300
TEST_TIMEOUT ?= 600
# Default target
help:
@echo "Available commands:"
@echo " make test - Run ALL tests (backend + frontend, installs deps automatically)"
@echo " make test-backend - Run backend tests only (unit + emulator)"
@echo " make test-frontend - Run frontend tests only (unit + E2E)"
@echo " make test-frontend - Run frontend unit tests (Jest)"
@echo " make test-frontend-e2e - Run frontend E2E tests (Playwright, requires dev server)"
@echo " make test-unit - Run unit tests only (karaoke_gen package)"
@echo " make test-e2e - Run emulator tests with auto-start/stop"
@echo " make install - Install all dependencies (backend + frontend)"
@echo " make build-frontend - Build frontend and copy to Python package (for local testing)"
@echo " make build-frontend - Build frontend and copy to Python package"
@echo " make dev-install - Build frontend + pip install -e . (for local CLI testing)"
@echo " make lint - Run linter checks"
@echo " make emulators-start - Start GCP emulators for local development"
@echo " make emulators-stop - Stop GCP emulators"
@@ -34,23 +41,34 @@ install-frontend:
install: install-backend install-frontend
# Build frontend and copy to Python package (for local CLI testing)
# Build frontend and copy to Python package
build-frontend: install-frontend
@echo "=== Building frontend ==="
cd frontend && npm run build
@echo "=== Copying build to Python package ==="
rm -rf karaoke_gen/nextjs_frontend/out
mkdir -p karaoke_gen/nextjs_frontend/out
cp -r frontend/out/* karaoke_gen/nextjs_frontend/out/
@echo "✅ Frontend built and ready for local testing"
@echo "✅ Frontend built and copied to karaoke_gen/nextjs_frontend/out/"
# Build frontend + install package in editable mode for local CLI testing
# After running this, `karaoke-gen` CLI will use the freshly built frontend.
dev-install: build-frontend
@echo "=== Installing package in editable mode ==="
pip install -e .
@echo ""
@echo "✅ Ready! You can now run karaoke-gen with the updated frontend."
@echo " Example: karaoke-gen input.flac \"Artist\" \"Title\""
# Run unit tests for karaoke_gen package
test-unit: install-backend
@echo "=== Running karaoke_gen unit tests ==="
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
timeout $(TEST_TIMEOUT) poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
# Run backend unit tests (excludes emulator tests)
test-backend-unit: install-backend
@echo "=== Running backend unit tests ==="
poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
timeout $(TEST_TIMEOUT) poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
# Run E2E integration tests with emulators
test-e2e: install-backend
@@ -62,7 +80,7 @@ test-e2e: install-backend
export GCS_BUCKET_NAME=test-bucket && \
export ADMIN_TOKENS=test-admin-token && \
export ENVIRONMENT=test && \
poetry run pytest backend/tests/emulator/ -v; \
timeout $(TEST_TIMEOUT) poetry run pytest backend/tests/emulator/ -v; \
TEST_RESULT=$$?; \
./scripts/stop-emulators.sh; \
exit $$TEST_RESULT
@@ -72,10 +90,15 @@ test-backend: test-unit test-backend-unit test-e2e
@echo ""
@echo "✅ Backend tests passed!"
# Run frontend tests (unit + E2E)
# Run frontend unit tests (Jest only - E2E requires manual setup)
test-frontend: install-frontend
@echo "=== Running frontend tests ==="
cd frontend && npm run test:all
@echo "=== Running frontend unit tests ==="
cd frontend && timeout $(TEST_TIMEOUT) npm run test:unit
# Run frontend E2E tests (requires dev server - run manually, not in make test)
test-frontend-e2e: install-frontend
@echo "=== Running frontend E2E tests ==="
cd frontend && timeout $(TEST_TIMEOUT) npm run test:e2e
# Run ALL tests (backend + frontend) - use this before committing!
test: test-backend test-frontend
+2
View File
@@ -29,6 +29,8 @@ tmp/
# Google Cloud
.gcloud/
*.json
# Allow i18n translation files
!translations/*.json
# Testing
.pytest_cache/
+9 -1
View File
@@ -26,7 +26,15 @@ COPY backend /app/backend
# Note: lyrics_transcriber is now a subpackage of karaoke_gen
COPY karaoke_gen /app/karaoke_gen
COPY pyproject.toml README.md LICENSE /app/
RUN pip install --no-cache-dir -e .
# GPU images (MODEL_DIR set in base): use --no-deps to preserve pinned torch cu126
# and audio-separator[gpu]. Without this, pip upgrades torch to cu130 (breaks CUDA
# on Cloud Run L4 drivers) and reinstalls audio-separator[cpu] (breaks GPU ONNX).
# CPU images: normal install resolves all deps from pyproject.toml.
RUN if [ -n "$MODEL_DIR" ]; then \
pip install --no-cache-dir --no-deps -e .; \
else \
pip install --no-cache-dir -e .; \
fi
# Set environment variables
ENV PYTHONUNBUFFERED=1
+26 -7
View File
@@ -1,8 +1,15 @@
# Base image with all Python dependencies pre-installed
# This image is rebuilt only when poetry.lock changes
#
# This image is rebuilt only when poetry.lock or this Dockerfile changes
#
# Build with: gcloud builds submit --config=cloudbuild-base.yaml
# Or manually: docker build -f backend/Dockerfile.base -t karaoke-backend-base .
#
# Layer caching strategy:
# 1. System deps + fonts + ffmpeg (rarely change, always cached)
# 2. pyproject.toml + package stubs (changes with dep updates)
# 3. pip install (the slow part - cached unless pyproject.toml changes)
# 4. Source code copy (changes every rebuild, but AFTER deps are installed)
# 5. spaCy model download (cached unless pip install layer changes)
FROM python:3.11-slim
@@ -16,6 +23,7 @@ RUN apt-get update && apt-get install -y \
build-essential \
curl \
xz-utils \
chromium \
&& rm -rf /var/lib/apt/lists/*
# Install comprehensive font support for international lyrics and symbols
@@ -50,16 +58,27 @@ RUN curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-stati
# Set working directory
WORKDIR /app
# Copy dependency files and source code needed for pip install
# Copy ONLY dependency specification files first (changes less often than source code)
COPY pyproject.toml README.md LICENSE /app/
# Create minimal package stubs so pip can resolve the editable install
# pip install -e . needs package directories to exist but doesn't validate content
RUN mkdir -p /app/karaoke_gen /app/lyrics_transcriber_temp /app/backend && \
touch /app/karaoke_gen/__init__.py /app/lyrics_transcriber_temp/__init__.py /app/backend/__init__.py
# Install all Python dependencies (the slow part)
# BuildKit cache mount preserves downloaded wheels across builds, so partial
# dependency updates only download changed packages instead of everything
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --upgrade pip && \
pip install -e .
# Now copy actual source code (AFTER deps installed, so code changes don't
# invalidate the expensive pip install layer in registry cache)
COPY karaoke_gen /app/karaoke_gen
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
COPY backend /app/backend
# Install all Python dependencies (the slow part - cached in base image)
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir -e .
# Download spaCy language model for lyrics transcription
# This must be done at build time, not runtime (Cloud Run network restrictions)
RUN python -m spacy download en_core_web_sm
+136
View File
@@ -0,0 +1,136 @@
# GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models
# This image is rebuilt only when deps or models change (infrequently).
#
# Build with: gcloud builds submit --config=backend/cloudbuild-gpu-base.yaml --project=nomadkaraoke --timeout=3600s
# Or manually: docker build -f backend/Dockerfile.gpu-base -t karaoke-backend-gpu-base .
#
# Layer caching strategy:
# 1. System deps + fonts + ffmpeg (rarely change, always cached)
# 2. CUDA library path configuration
# 3. pyproject.toml + package stubs (changes with dep updates)
# 4. pip install karaoke-gen deps + audio-separator[gpu] (slow, cached unless pyproject.toml changes)
# 5. Source code copy (changes every rebuild, but AFTER deps are installed)
# 6. spaCy model download (cached unless pip install layer changes)
# 7. Pre-downloaded ensemble preset models (~1-1.5 GB, cached unless pip layer changes)
FROM nvidia/cuda:12.6.3-runtime-ubuntu22.04
# Prevent interactive prompts during package installation
ENV DEBIAN_FRONTEND=noninteractive
# Install Python 3.11 from deadsnakes PPA and system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
software-properties-common \
&& add-apt-repository -y ppa:deadsnakes/ppa \
&& apt-get update && apt-get install -y --no-install-recommends \
# Python 3.11
python3.11 \
python3.11-dev \
python3.11-venv \
# Audio libraries
libsndfile1 \
libsox-dev \
sox \
# Build tools
build-essential \
# Utilities
curl \
xz-utils \
&& rm -rf /var/lib/apt/lists/* \
&& python3.11 --version
# Install comprehensive font support for international lyrics and symbols
# Noto fonts provide full Unicode coverage including:
# - Latin, Greek, Cyrillic (core)
# - CJK: Chinese, Japanese, Korean (~150MB)
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
# - Color emoji (limited support in libass, but available)
# - Musical symbols (♪ ♫ etc.)
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
RUN apt-get update && apt-get install -y --no-install-recommends \
fonts-noto-core \
fonts-noto-cjk \
fonts-noto-extra \
fonts-noto-color-emoji \
fontconfig \
&& fc-cache -f \
&& rm -rf /var/lib/apt/lists/*
# Install FFmpeg from Ubuntu package
# The audio worker only uses FFmpeg for format conversion and padding, not
# encoding-intensive tasks, so the Debian package is sufficient. The static
# build from johnvansickle.com is used for the encoding worker instead.
RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*
# Set Python 3.11 as default and install pip
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1 \
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.11 1 \
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11 \
&& python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel
# Set up CUDA library paths so GPU-accelerated packages can find CUDA libs at runtime
RUN echo '/usr/local/cuda/lib64' >> /etc/ld.so.conf.d/cuda.conf && ldconfig
# Environment configuration for CUDA and Python
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH \
PATH=/usr/local/cuda/bin:$PATH \
PYTHONUNBUFFERED=1 \
MODEL_DIR=/models
# Set working directory
WORKDIR /app
# Copy ONLY dependency specification files first (changes less often than source code)
COPY pyproject.toml README.md LICENSE /app/
# Create minimal package stubs so pip can resolve the editable install
# pip install -e . needs package directories to exist but doesn't validate content
RUN mkdir -p /app/karaoke_gen /app/lyrics_transcriber_temp /app/backend && \
touch /app/karaoke_gen/__init__.py /app/lyrics_transcriber_temp/__init__.py /app/backend/__init__.py
# Install all karaoke-gen Python dependencies (the slow part)
# BuildKit cache mount preserves downloaded wheels across builds
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --upgrade pip && \
pip install -e .
# Install audio-separator with GPU support (PyTorch + CUDA + onnxruntime-gpu)
# This is separate from pip install -e . to keep layer caching granular
#
# IMPORTANT: Install PyTorch cu126 FIRST, then audio-separator[gpu].
# The default audio-separator[gpu] pulls PyTorch cu130, but Cloud Run L4 GPUs
# have NVIDIA driver 570 which only supports up to CUDA 12.8. PyTorch cu130
# fails with "NVIDIA driver is too old". PyTorch cu126 works with 12.8 drivers.
# Installing torch first means audio-separator[gpu] sees it as already satisfied.
RUN --mount=type=cache,target=/root/.cache/pip \
pip install torch==2.6.0+cu126 torchvision==0.21.0+cu126 --index-url https://download.pytorch.org/whl/cu126 && \
pip install "audio-separator[gpu]"
# Now copy actual source code (AFTER deps installed, so code changes don't
# invalidate the expensive pip install layer in registry cache)
COPY karaoke_gen /app/karaoke_gen
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
COPY backend /app/backend
# Download spaCy language model for lyrics transcription
# Must be done at build time, not runtime (Cloud Run network restrictions)
RUN python -m spacy download en_core_web_sm
# Create models directory and pre-download ensemble preset models
# Baking models into the image eliminates cold-start model download latency.
# Total model size: ~1-1.5 GB for instrumental_clean + karaoke presets.
RUN mkdir -p /models
COPY backend/scripts/download_models.py /tmp/download_models.py
RUN python /tmp/download_models.py /models && \
rm /tmp/download_models.py && \
ls -lh /models/
# Label with build info
ARG BUILD_DATE
ARG BUILD_ID
ARG BASE_CONTENT_HASH
LABEL org.opencontainers.image.created="${BUILD_DATE}"
LABEL org.opencontainers.image.revision="${BUILD_ID}"
LABEL org.opencontainers.image.title="karaoke-backend-gpu-base"
LABEL org.opencontainers.image.description="GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models for karaoke-backend audio worker"
LABEL base.content.hash="${BASE_CONTENT_HASH}"
+14
View File
@@ -113,6 +113,19 @@ async def require_auth(
detail=f"Authentication failed: {auth_result.message}"
)
# If token has a tenant_id, override the hostname-based tenant detection.
# This allows tenant-scoped tokens to enforce tenant context regardless of domain.
# We must also load tenant_config so downstream handlers (e.g. job creation)
# that rely on tenant_config for setting tenant_id on jobs work correctly.
if auth_result.tenant_id:
request.state.tenant_id = auth_result.tenant_id
if not getattr(request.state, "tenant_config", None):
from backend.services.tenant_service import get_tenant_service
tenant_service = get_tenant_service()
tenant_config = tenant_service.get_tenant_config(auth_result.tenant_id)
if tenant_config and tenant_config.is_active:
request.state.tenant_config = tenant_config
# Log successful auth with request_id for correlation with request audit
logger.info(
"auth_success",
@@ -122,6 +135,7 @@ async def require_auth(
"user_type": auth_result.user_type.value if auth_result.user_type else None,
"is_admin": auth_result.is_admin,
"remaining_uses": auth_result.remaining_uses,
"tenant_id": auth_result.tenant_id,
"audit_type": "auth_event",
}
)
File diff suppressed because it is too large. Load diff
+381 -99
View File
@@ -17,6 +17,8 @@ import mimetypes
import os
import shutil
import tempfile
import uuid
from datetime import datetime, timedelta
from typing import Optional, List, Dict, Any
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request, Depends
@@ -35,6 +37,7 @@ from backend.services.audio_search_service import (
NoResultsError,
DownloadError,
)
from backend.services.flacfetch_client import get_flacfetch_client, FlacfetchServiceError
from backend.services.youtube_download_service import (
get_youtube_download_service,
YouTubeDownloadError,
@@ -46,7 +49,11 @@ from backend.version import VERSION
from backend.api.dependencies import require_auth
from backend.services.auth_service import UserType, AuthResult
from backend.middleware.tenant import get_tenant_config_from_request
from backend.exceptions import InsufficientCreditsError
from backend.services.tracing import add_span_attribute
from backend.services.firestore_service import FirestoreService
from pathlib import Path
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
@@ -134,6 +141,9 @@ class AudioSearchRequest(BaseModel):
# Non-interactive mode
non_interactive: bool = Field(False, description="Skip interactive steps (lyrics review, instrumental selection)")
# Private (non-published) track mode
is_private: bool = Field(False, description="Private track: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive")
@validator('artist', 'title', 'lyrics_artist', 'lyrics_title', 'display_artist', 'display_title')
def normalize_text_fields(cls, v):
"""Normalize text fields to standardize Unicode characters."""
@@ -192,9 +202,32 @@ class AudioSearchResponse(BaseModel):
style_upload_urls: Optional[List[StyleUploadUrl]] = None
class StandaloneSearchRequest(BaseModel):
"""Request for standalone audio search (no job created)."""
artist: str = Field(..., description="Artist name to search for")
title: str = Field(..., description="Song title to search for")
@validator('artist', 'title')
def normalize_text_fields(cls, v):
if v is not None and isinstance(v, str):
return normalize_text(v)
return v
class StandaloneSearchResponse(BaseModel):
"""Response from standalone audio search."""
search_session_id: str
results: List[AudioSearchResultResponse]
results_count: int
class AudioSelectRequest(BaseModel):
"""Request to select an audio source."""
selection_index: int = Field(..., description="Index of the selected audio source from search results")
# Optional overrides — sent from the guided flow's Customize step (Step 3)
is_private: Optional[bool] = Field(None, description="Override privacy flag (guided flow Step 3)")
display_artist: Optional[str] = Field(None, description="Override display artist name")
display_title: Optional[str] = Field(None, description="Override display title")
class AudioSelectResponse(BaseModel):
@@ -208,24 +241,24 @@ class AudioSelectResponse(BaseModel):
selected_provider: str
def extract_request_metadata(request: Request, created_from: str = "audio_search") -> Dict[str, Any]:
def extract_request_metadata(request: Request, created_from: str = "audio_search", auth_result=None) -> Dict[str, Any]:
"""Extract metadata from request for job tracking."""
headers = dict(request.headers)
client_ip = headers.get('x-forwarded-for', '').split(',')[0].strip()
if not client_ip and request.client:
client_ip = request.client.host
user_agent = headers.get('user-agent', '')
environment = headers.get('x-environment', '')
client_id = headers.get('x-client-id', '')
custom_headers = {}
for key, value in headers.items():
if key.lower().startswith('x-') and key.lower() not in ('x-forwarded-for', 'x-forwarded-proto', 'x-forwarded-host'):
custom_headers[key] = value
return {
metadata = {
'client_ip': client_ip,
'user_agent': user_agent,
'environment': environment,
@@ -235,13 +268,16 @@ def extract_request_metadata(request: Request, created_from: str = "audio_search
'custom_headers': custom_headers,
}
# Add auth context for abuse investigation and analytics
if auth_result is not None:
metadata['auth_method'] = getattr(auth_result, 'auth_method', None)
user_type = getattr(auth_result, 'user_type', None)
metadata['user_type'] = user_type.value if hasattr(user_type, 'value') else str(user_type) if user_type else None
remaining = getattr(auth_result, 'remaining_uses', None)
if remaining is not None:
metadata['credits_at_creation'] = remaining
async def _trigger_workers_parallel(job_id: str) -> None:
"""Trigger both audio and lyrics workers in parallel."""
await asyncio.gather(
worker_service.trigger_audio_worker(job_id),
worker_service.trigger_lyrics_worker(job_id)
)
return metadata
def _extract_gcs_path(filepath: str) -> str:
@@ -262,43 +298,33 @@ def _extract_gcs_path(filepath: str) -> str:
return filepath
async def _download_and_start_processing(
def _validate_and_prepare_selection(
job_id: str,
selection_index: int,
audio_search_service,
background_tasks: BackgroundTasks,
) -> Dict[str, Any]:
"""
Download selected audio and start job processing.
This is called either:
- Immediately after search if auto_download=True
- When user calls the select endpoint
For remote flacfetch downloads (torrent sources), the file is downloaded
on the flacfetch VM and uploaded directly to GCS. For local downloads
(YouTube), the file is downloaded locally and then uploaded to GCS.
Validate selection and transition job to DOWNLOADING_AUDIO state.
Returns selection info dict with the selected result details.
Raises HTTPException on validation errors.
"""
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
raise HTTPException(status_code=404, detail=t("en", "audioSearch.jobNotFound", job_id=job_id))
# Get search results from state_data
search_results = job.state_data.get('audio_search_results', [])
if not search_results:
raise HTTPException(status_code=400, detail="No search results available")
raise HTTPException(status_code=400, detail=t("en", "audioSearch.noSearchResults"))
if selection_index < 0 or selection_index >= len(search_results):
raise HTTPException(
status_code=400,
detail=f"Invalid selection index {selection_index}. Valid range: 0-{len(search_results)-1}"
detail=t("en", "audioSearch.invalidSelectionIndex", selection_index=selection_index, max=len(search_results)-1)
)
selected = search_results[selection_index]
# Get remote_search_id from state_data (stored during initial search)
remote_search_id = job.state_data.get('remote_search_id')
# Transition to downloading state
job_manager.transition_to_state(
job_id=job_id,
@@ -310,8 +336,59 @@ async def _download_and_start_processing(
'selected_audio_provider': selected['provider'],
}
)
# Save download params to job BEFORE attempting download
# This enables retry even if download fails
source_name = selected.get('provider')
source_id = selected.get('source_id')
target_file = selected.get('target_file')
download_url = selected.get('url')
job_manager.update_job(job_id, {
'audio_source_type': 'audio_search',
'source_name': source_name,
'source_id': source_id,
'target_file': target_file,
'download_url': download_url,
})
logger.info(
f"Job {job_id}: Saved download params before attempt: "
f"source_name={source_name}, source_id={source_id}"
)
return {
'selected_index': selection_index,
'selected_title': selected['title'],
'selected_artist': selected['artist'],
'selected_provider': selected['provider'],
}
async def _download_audio_and_trigger_workers(
job_id: str,
selection_index: int,
audio_search_service,
) -> None:
"""
Download audio and trigger processing workers.
This runs as a background task after the select endpoint returns.
Also called synchronously from the auto_download path during search.
For remote flacfetch downloads (torrent sources), the file is downloaded
on the flacfetch VM and uploaded directly to GCS. For local downloads
(YouTube), the file is downloaded locally and then uploaded to GCS.
"""
try:
job = job_manager.get_job(job_id)
if not job:
logger.error(f"Job {job_id} not found during background download")
return
search_results = job.state_data.get('audio_search_results', [])
selected = search_results[selection_index]
remote_search_id = job.state_data.get('remote_search_id')
# Get source info from selected result
source_name = selected.get('provider')
source_id = selected.get('source_id')
@@ -345,8 +422,9 @@ async def _download_and_start_processing(
raise DownloadError(f"YouTube download failed: {e}")
elif source_name in ['RED', 'OPS']:
# Torrent sources - must use remote flacfetch
if not audio_search_service.is_remote_enabled():
# Torrent sources - must use remote flacfetch (async, no event loop blocking)
flacfetch_client = get_flacfetch_client()
if not flacfetch_client:
raise DownloadError(
f"Cannot download from {source_name} without remote flacfetch service. "
"Configure FLACFETCH_API_URL."
@@ -356,31 +434,48 @@ async def _download_and_start_processing(
if source_id:
logger.info(f"Torrent download via download_by_id: {source_name} ID={source_id}")
result = audio_search_service.download_by_id(
download_id = await flacfetch_client.download_by_id(
source_name=source_name,
source_id=source_id,
output_dir="",
target_file=target_file,
download_url=download_url,
gcs_path=gcs_destination,
)
else:
if not remote_search_id:
raise DownloadError(
f"No source_id or remote_search_id available for {source_name} download"
)
logger.info(f"Torrent download via search-based download")
result = audio_search_service.download(
download_id = await flacfetch_client.download(
search_id=remote_search_id,
result_index=selection_index,
output_dir="",
gcs_path=gcs_destination,
remote_search_id=remote_search_id,
)
# Extract GCS path from result
audio_gcs_path = _extract_gcs_path(result.filepath)
filename = os.path.basename(result.filepath)
def log_progress(status):
progress = status.get("progress", 0)
speed = status.get("download_speed_kbps", 0)
logger.debug(f"Download progress: {progress:.1f}% ({speed:.1f} KB/s)")
final_status = await flacfetch_client.wait_for_download(
download_id,
timeout=600,
progress_callback=log_progress,
)
filepath = final_status.get("gcs_path") or final_status.get("output_path")
if not filepath:
raise DownloadError("Remote download completed but no file path returned")
audio_gcs_path = _extract_gcs_path(filepath)
filename = os.path.basename(filepath)
logger.info(f"Torrent download complete: {audio_gcs_path}")
elif source_name == 'Spotify':
# Spotify downloads - use audio_search_service
if not audio_search_service.is_remote_enabled():
# Spotify downloads - use async FlacfetchClient directly (no event loop blocking)
flacfetch_client = get_flacfetch_client()
if not flacfetch_client:
raise DownloadError(
f"Cannot download from Spotify without remote flacfetch service. "
"Configure FLACFETCH_API_URL."
@@ -389,25 +484,40 @@ async def _download_and_start_processing(
gcs_destination = f"uploads/{job_id}/audio/"
logger.info(f"Spotify download: source_id={source_id}")
result = audio_search_service.download_by_id(
download_id = await flacfetch_client.download_by_id(
source_name=source_name,
source_id=source_id,
output_dir="",
download_url=download_url,
gcs_path=gcs_destination,
)
audio_gcs_path = _extract_gcs_path(result.filepath)
filename = os.path.basename(result.filepath)
def log_progress(status):
progress = status.get("progress", 0)
speed = status.get("download_speed_kbps", 0)
logger.debug(f"Download progress: {progress:.1f}% ({speed:.1f} KB/s)")
final_status = await flacfetch_client.wait_for_download(
download_id,
timeout=600,
progress_callback=log_progress,
)
filepath = final_status.get("gcs_path") or final_status.get("output_path")
if not filepath:
raise DownloadError("Remote download completed but no file path returned")
audio_gcs_path = _extract_gcs_path(filepath)
filename = os.path.basename(filepath)
logger.info(f"Spotify download complete: {audio_gcs_path}")
else:
# Unknown source - try generic download
# Unknown source - try generic download (run sync code in thread to avoid blocking)
logger.warning(f"Unknown source type: {source_name}, attempting generic download")
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
try:
result = audio_search_service.download(
result = await asyncio.to_thread(
audio_search_service.download,
result_index=selection_index,
output_dir=temp_dir,
remote_search_id=remote_search_id,
@@ -433,36 +543,36 @@ async def _download_and_start_processing(
shutil.rmtree(temp_dir)
except Exception as e:
logger.warning(f"Failed to clean up temp files: {e}")
# Update job with GCS path and transition to DOWNLOADING
# Update job with GCS path
job_manager.update_job(job_id, {
'input_media_gcs_path': audio_gcs_path,
'filename': filename,
})
# Transition from DOWNLOADING_AUDIO to DOWNLOADING and trigger workers
# Note: Can't use start_job_processing() here because that expects PENDING status,
# but we're in DOWNLOADING_AUDIO state from the audio search flow.
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.DOWNLOADING,
progress=15,
message="Audio downloaded, starting processing"
)
# Trigger workers
background_tasks.add_task(_trigger_workers_parallel, job_id)
return {
'selected_index': selection_index,
'selected_title': selected['title'],
'selected_artist': selected['artist'],
'selected_provider': selected['provider'],
}
except DownloadError as e:
# Trigger workers directly
worker_service = get_worker_service()
await asyncio.gather(
worker_service.trigger_audio_worker(job_id),
worker_service.trigger_lyrics_worker(job_id)
)
except (DownloadError, FlacfetchServiceError) as e:
logger.error(f"Download failed for job {job_id}: {e}")
job_manager.fail_job(job_id, f"Audio download failed: {e}")
raise HTTPException(status_code=500, detail=f"Download failed: {e}")
except Exception as e:
logger.error(f"Download failed for job {job_id}: {e}", exc_info=True)
job_manager.fail_job(job_id, f"Audio download failed: {e}")
raise HTTPException(status_code=500, detail=f"Download failed: {e}")
@router.post("/audio-search/search", response_model=AudioSearchResponse)
@@ -485,12 +595,14 @@ async def search_audio(
- Interactive mode (default): Returns results, user calls /select endpoint
- Auto mode (auto_download=True): Automatically selects and downloads best
"""
locale = get_locale_from_request(request)
# Check tenant feature flag
tenant_config = get_tenant_config_from_request(request)
if tenant_config and not tenant_config.features.audio_search:
raise HTTPException(
status_code=403,
detail="Audio search is not available for this portal"
detail=t(locale, "audioSearch.notEnabled")
)
try:
@@ -535,8 +647,9 @@ async def search_audio(
}
)
# Extract request metadata
request_metadata = extract_request_metadata(request, created_from="audio_search")
# Extract request metadata (includes auth context + search query info)
request_metadata = extract_request_metadata(request, created_from="audio_search", auth_result=auth_result)
request_metadata['search_query'] = {'artist': body.artist, 'title': body.title}
# Apply default theme if none specified
# This ensures all karaoke videos use the Nomad theme by default
@@ -588,12 +701,18 @@ async def search_audio(
auto_download=body.auto_download,
request_metadata=request_metadata,
non_interactive=body.non_interactive,
is_private=body.is_private,
# Tenant scoping
tenant_id=tenant_config.id if tenant_config else None,
tenant_id=tenant_config.id if tenant_config else "",
)
job = job_manager.create_job(job_create, is_admin=auth_result.is_admin)
job_id = job.job_id
# Trace attributes for observability
add_span_attribute("job_id", job_id)
add_span_attribute("job.source", "audio_search")
add_span_attribute("job.is_private", body.is_private)
logger.info(f"Created job {job_id} for audio search: {body.artist} - {body.title}")
# Update job with audio search fields
@@ -650,11 +769,11 @@ async def search_audio(
# Generate GCS path
if file_info.file_type == 'style_params':
gcs_path = f"uploads/{job_id}/style/style_params.json"
gcs_path = f"jobs/{job_id}/style/style_params.json"
else:
# style_intro_background -> intro_background, etc.
asset_key = file_info.file_type.replace('style_', '')
gcs_path = f"uploads/{job_id}/style/{asset_key}{ext}"
gcs_path = f"jobs/{job_id}/style/{asset_key}{ext}"
# Generate signed upload URL
signed_url = storage_service.generate_signed_upload_url(
@@ -696,7 +815,7 @@ async def search_audio(
audio_search_service = get_audio_search_service()
try:
search_results = audio_search_service.search(body.artist, body.title)
search_results = await audio_search_service.search_async(body.artist, body.title)
except NoResultsError as e:
job_manager.fail_job(job_id, f"No audio sources found for: {body.artist} - {body.title}")
raise HTTPException(
@@ -727,16 +846,21 @@ async def search_audio(
# If auto_download, select best and start processing
if body.auto_download:
best_index = audio_search_service.select_best(search_results)
logger.info(f"Auto-download enabled, selecting result {best_index}")
selection_info = await _download_and_start_processing(
selection_info = _validate_and_prepare_selection(
job_id=job_id,
selection_index=best_index,
audio_search_service=audio_search_service,
background_tasks=background_tasks,
)
# Trigger audio download as a Cloud Run Job (survives instance shutdown)
worker_service = get_worker_service()
triggered = await worker_service.trigger_audio_download_worker(job_id)
if not triggered:
job_manager.fail_job(job_id, "Failed to trigger audio download worker")
raise HTTPException(status_code=500, detail="Failed to trigger audio download worker")
return AudioSearchResponse(
status="success",
job_id=job_id,
@@ -824,10 +948,149 @@ async def search_audio(
except HTTPException:
raise
except Exception as e:
if isinstance(e, InsufficientCreditsError):
raise
logger.error(f"Error in audio search: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e))
@router.post("/audio-search/search-standalone", response_model=StandaloneSearchResponse)
async def search_audio_standalone(
request: Request,
body: StandaloneSearchRequest,
auth_result: AuthResult = Depends(require_auth)
):
"""
Search for audio by artist and title WITHOUT creating a job.
Returns a search session ID and results. Call POST /api/jobs/create-from-search
with the session ID and selected result index to create the actual job.
Credits are checked here (early feedback) but NOT deducted — deduction
happens when the job is created via create-from-search.
"""
locale = get_locale_from_request(request)
# Check tenant feature flag
tenant_config = get_tenant_config_from_request(request)
if tenant_config and not tenant_config.features.audio_search:
raise HTTPException(
status_code=403,
detail=t(locale, "audioSearch.notEnabled")
)
try:
user_email = auth_result.user_email
if not user_email:
raise HTTPException(status_code=401, detail="Authentication required")
# Credit check — give early feedback but do NOT deduct here
if not auth_result.is_admin:
from backend.services.user_service import get_user_service
user_service = get_user_service()
if not user_service.has_credits(user_email):
credits_available = user_service.check_credits(user_email)
raise InsufficientCreditsError(
message="You're out of credits. Buy more to continue creating karaoke videos.",
credits_available=credits_available,
credits_required=1,
)
# Perform the search (runs in thread to avoid blocking the event loop)
audio_search_service = get_audio_search_service()
try:
search_results = await audio_search_service.search_async(body.artist, body.title)
except NoResultsError:
# No results is a normal outcome — return empty list, not an error
search_results = []
except AudioSearchError as e:
raise HTTPException(status_code=500, detail=f"Search failed: {e}")
# Build result response objects
result_responses: List[AudioSearchResultResponse] = []
results_dicts: List[Dict[str, Any]] = []
for r in search_results:
raw_dict: Dict[str, Any] = {}
if r.raw_result:
if isinstance(r.raw_result, dict):
raw_dict = r.raw_result
else:
try:
raw_dict = r.raw_result.to_dict()
except AttributeError:
pass
result_responses.append(
AudioSearchResultResponse(
index=r.index,
title=r.title,
artist=r.artist,
provider=r.provider,
url=r.url,
duration=r.duration,
quality=r.quality,
source_id=r.source_id,
seeders=raw_dict.get('seeders') or r.seeders,
target_file=raw_dict.get('target_file') or r.target_file,
year=raw_dict.get('year'),
label=raw_dict.get('label'),
edition_info=raw_dict.get('edition_info'),
release_type=raw_dict.get('release_type'),
channel=raw_dict.get('channel'),
view_count=raw_dict.get('view_count'),
size_bytes=raw_dict.get('size_bytes'),
target_file_size=raw_dict.get('target_file_size'),
track_pattern=raw_dict.get('track_pattern'),
match_score=raw_dict.get('match_score'),
formatted_size=raw_dict.get('formatted_size'),
formatted_duration=raw_dict.get('formatted_duration'),
formatted_views=raw_dict.get('formatted_views'),
is_lossless=raw_dict.get('is_lossless'),
quality_str=raw_dict.get('quality_str'),
quality_data=raw_dict.get('quality_data') or raw_dict.get('quality'),
)
)
results_dicts.append(r.to_dict())
# Store session in Firestore (TTL: 7 days)
# Long TTL so users who leave a tab open mid-flow don't hit "Search expired".
# Sessions are tiny docs (~2 KB) and cleaned up by Firestore TTL policy.
session_id = str(uuid.uuid4())
now = datetime.utcnow()
session_data = {
'session_id': session_id,
'user_email': user_email,
'tenant_id': tenant_config.id if tenant_config else None,
'artist': body.artist,
'title': body.title,
'results': results_dicts,
'remote_search_id': audio_search_service.last_remote_search_id,
'created_at': now.isoformat(),
'ttl_expiry': now + timedelta(days=7),
}
firestore_service = FirestoreService()
firestore_service.create_search_session(session_data)
logger.info(
f"Standalone search complete for {body.artist} - {body.title}: "
f"{len(search_results)} results, session={session_id}"
)
return StandaloneSearchResponse(
search_session_id=session_id,
results=result_responses,
results_count=len(result_responses),
)
except HTTPException:
raise
except InsufficientCreditsError:
raise
except Exception as e:
logger.error(f"Error in standalone audio search: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e))
@router.get("/audio-search/{job_id}/results")
async def get_audio_search_results(
job_id: str,
@@ -840,14 +1103,14 @@ async def get_audio_search_results(
"""
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
raise HTTPException(status_code=404, detail=t("en", "audioSearch.jobNotFound", job_id=job_id))
search_results = job.state_data.get('audio_search_results', [])
if not search_results:
raise HTTPException(
status_code=400,
detail="No search results available for this job"
detail=t("en", "audioSearch.noSearchResults")
)
return {
@@ -870,45 +1133,64 @@ async def select_audio_source(
):
"""
Select an audio source and start job processing.
This endpoint:
1. Validates the job is awaiting selection
2. Downloads the selected audio
3. Uploads to GCS
4. Triggers audio and lyrics workers
This endpoint returns immediately after validating the selection and
transitioning the job to DOWNLOADING_AUDIO state. The actual download
runs as a background task so the UI can close the selection dialog
without waiting for the download to complete.
"""
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
raise HTTPException(status_code=404, detail=t("en", "audioSearch.jobNotFound", job_id=job_id))
# Verify job is awaiting selection
if job.status != JobStatus.AWAITING_AUDIO_SELECTION:
raise HTTPException(
status_code=400,
detail=f"Job is not awaiting audio selection (status: {job.status})"
)
# Get search service instance
# Note: With download_by_id, we no longer need to re-search to populate the cache.
# The source_id stored in job.state_data['audio_search_results'] is sufficient.
audio_search_service = get_audio_search_service()
# Apply optional overrides from guided flow Step 3 (Customize)
# These fields are set AFTER the job is created (Step 2), so they must be
# patched here before the pipeline starts.
overrides = {}
if body.is_private is not None:
overrides["is_private"] = body.is_private
if body.display_artist is not None:
overrides["display_artist"] = body.display_artist
if body.display_title is not None:
overrides["display_title"] = body.display_title
if overrides:
job_manager.update_job(job_id, overrides)
logger.info(f"[job:{job_id}] Applied selection overrides: {list(overrides.keys())}")
# Validate search results exist in job state_data
search_results = job.state_data.get('audio_search_results', [])
if not search_results:
raise HTTPException(status_code=400, detail="No search results cached for this job")
selection_info = await _download_and_start_processing(
# Validate selection and transition to DOWNLOADING_AUDIO (returns immediately)
selection_info = _validate_and_prepare_selection(
job_id=job_id,
selection_index=body.selection_index,
audio_search_service=audio_search_service,
background_tasks=background_tasks,
)
# Trigger audio download as a Cloud Run Job (survives instance shutdown)
worker_service = get_worker_service()
triggered = await worker_service.trigger_audio_download_worker(job_id)
if not triggered:
job_manager.fail_job(job_id, "Failed to trigger audio download worker")
raise HTTPException(status_code=500, detail="Failed to trigger audio download worker")
return AudioSelectResponse(
status="success",
job_id=job_id,
message="Audio selected and download started",
message="Audio selected, download starting",
selected_index=selection_info['selected_index'],
selected_title=selection_info['selected_title'],
selected_artist=selection_info['selected_artist'],
+41 -35
View File
@@ -9,7 +9,7 @@ Provides endpoints for:
import logging
from typing import Optional
from datetime import datetime
from fastapi import APIRouter, HTTPException, BackgroundTasks
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request
from pydantic import BaseModel
from backend.services.credential_manager import (
@@ -18,6 +18,7 @@ from backend.services.credential_manager import (
CredentialCheckResult,
)
from backend.config import get_settings
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/auth", tags=["auth"])
@@ -125,15 +126,16 @@ async def get_credentials_status():
@router.get("/status/{service}", response_model=CredentialStatusResponse)
async def get_service_credential_status(service: str):
async def get_service_credential_status(service: str, request: Request):
"""
Get the status of a specific service's OAuth credentials.
Args:
service: One of "youtube", "gdrive", "dropbox"
"""
locale = get_locale_from_request(request)
manager = get_credential_manager()
if service == "youtube":
result = manager.check_youtube_credentials()
elif service == "gdrive":
@@ -141,7 +143,7 @@ async def get_service_credential_status(service: str):
elif service == "dropbox":
result = manager.check_dropbox_credentials()
else:
raise HTTPException(status_code=400, detail=f"Unknown service: {service}")
raise HTTPException(status_code=400, detail=t(locale, "auth.unknownService", service=service))
return CredentialStatusResponse(
service=result.service,
@@ -153,42 +155,43 @@ async def get_service_credential_status(service: str):
@router.post("/validate", response_model=CredentialValidationResponse)
async def validate_credentials(request: CredentialValidationRequest):
async def validate_credentials(request: CredentialValidationRequest, http_request: Request):
"""
Validate that credentials are available for requested services.
Use this before submitting a job to ensure all required
credentials are valid.
"""
locale = get_locale_from_request(http_request)
manager = get_credential_manager()
invalid_services = []
if request.youtube:
result = manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append("youtube")
if request.gdrive:
result = manager.check_gdrive_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append("gdrive")
if request.dropbox:
result = manager.check_dropbox_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append("dropbox")
if invalid_services:
return CredentialValidationResponse(
valid=False,
invalid_services=invalid_services,
message=f"The following services need re-authorization: {', '.join(invalid_services)}"
message=t(locale, "auth.invalidCredentials", services=", ".join(invalid_services))
)
return CredentialValidationResponse(
valid=True,
invalid_services=[],
message="All requested credentials are valid"
message=t(locale, "auth.validCredentials")
)
@@ -197,40 +200,41 @@ async def validate_credentials(request: CredentialValidationRequest):
# ===========================================================================
@router.post("/youtube/device", response_model=DeviceAuthStartResponse)
async def start_youtube_device_auth(request: Optional[DeviceAuthStartRequest] = None):
async def start_youtube_device_auth(request: Optional[DeviceAuthStartRequest] = None, http_request: Request = None):
"""
Start YouTube device authorization flow.
This initiates a device auth flow that allows authorization
from any device. The user must visit the verification URL
and enter the user code.
Client credentials are loaded from Secret Manager ('youtube-client-credentials')
unless explicitly provided in the request body.
After starting, poll /auth/youtube/device/{device_code} to
check for completion.
"""
locale = get_locale_from_request(http_request) if http_request else "en"
manager = get_credential_manager()
try:
device_info = manager.start_youtube_device_auth(
client_id=request.client_id if request else None,
client_secret=request.client_secret if request else None
)
return DeviceAuthStartResponse(
device_code=device_info.device_code,
user_code=device_info.user_code,
verification_url=device_info.verification_url,
expires_in=device_info.expires_in,
interval=device_info.interval,
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
instructions=t(locale, "auth.youtubeAuthInstructions", verification_url=device_info.verification_url, user_code=device_info.user_code)
)
except Exception as e:
logger.error(f"Failed to start YouTube device auth: {e}")
raise HTTPException(status_code=500, detail=str(e))
raise HTTPException(status_code=500, detail=t(locale, "auth.youtubeAuthStartFailed", error=str(e)))
@router.get("/youtube/device/{device_code}", response_model=DeviceAuthPollResponse)
@@ -259,35 +263,36 @@ async def poll_youtube_device_auth(device_code: str):
@router.post("/gdrive/device", response_model=DeviceAuthStartResponse)
async def start_gdrive_device_auth(request: Optional[DeviceAuthStartRequest] = None):
async def start_gdrive_device_auth(request: Optional[DeviceAuthStartRequest] = None, http_request: Request = None):
"""
Start Google Drive device authorization flow.
Client credentials are loaded from Secret Manager ('gdrive-client-credentials')
unless explicitly provided in the request body.
After starting, poll /auth/gdrive/device/{device_code} to check for completion.
"""
locale = get_locale_from_request(http_request) if http_request else "en"
manager = get_credential_manager()
try:
device_info = manager.start_gdrive_device_auth(
client_id=request.client_id if request else None,
client_secret=request.client_secret if request else None
)
return DeviceAuthStartResponse(
device_code=device_info.device_code,
user_code=device_info.user_code,
verification_url=device_info.verification_url,
expires_in=device_info.expires_in,
interval=device_info.interval,
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
instructions=t(locale, "auth.gdriveLinkInstructions", verification_url=device_info.verification_url, user_code=device_info.user_code)
)
except Exception as e:
logger.error(f"Failed to start Google Drive device auth: {e}")
raise HTTPException(status_code=500, detail=str(e))
raise HTTPException(status_code=500, detail=t(locale, "auth.gdriveAuthStartFailed", error=str(e)))
@router.get("/gdrive/device/{device_code}", response_model=DeviceAuthPollResponse)
@@ -311,19 +316,20 @@ async def poll_gdrive_device_auth(device_code: str):
# ===========================================================================
@router.post("/test-alert")
async def test_credential_alert(background_tasks: BackgroundTasks):
async def test_credential_alert(background_tasks: BackgroundTasks, request: Request):
"""
Test the credential alert mechanism.
Sends a test alert to Discord if configured.
"""
locale = get_locale_from_request(request)
settings = get_settings()
discord_url = settings.get_secret("discord-alert-webhook")
if not discord_url:
raise HTTPException(
status_code=400,
detail="Discord alert webhook not configured"
detail=t(locale, "auth.discordAlertNotConfigured")
)
manager = get_credential_manager()
+163
View File
@@ -0,0 +1,163 @@
"""
Catalog API routes for song/artist lookup and community version detection.
Provides autocomplete data from karaoke-decide's MusicBrainz + Spotify catalog,
and checks karaokenerds.com for existing community karaoke versions.
"""
import logging
import time
from typing import Optional
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from pydantic import BaseModel, Field
from backend.api.dependencies import require_auth
from backend.services.auth_service import AuthResult
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/catalog", tags=["catalog"])
# Simple per-user rate limiting: 20 requests/minute
# { user_email: [timestamp, timestamp, ...] }
_rate_limit_windows: dict[str, list[float]] = {}
RATE_LIMIT_MAX_REQUESTS = 20
RATE_LIMIT_WINDOW_SECONDS = 60
def _check_rate_limit(user_email: str, locale: str = "en") -> None:
"""Check per-user rate limit. Raises 429 if exceeded."""
now = time.monotonic()
window_start = now - RATE_LIMIT_WINDOW_SECONDS
# Get or create window for this user
if user_email not in _rate_limit_windows:
_rate_limit_windows[user_email] = []
# Remove expired entries
timestamps = _rate_limit_windows[user_email]
_rate_limit_windows[user_email] = [ts for ts in timestamps if ts > window_start]
if len(_rate_limit_windows[user_email]) >= RATE_LIMIT_MAX_REQUESTS:
raise HTTPException(
status_code=429,
detail=t(locale, "catalog.rateLimitExceeded", max_requests=RATE_LIMIT_MAX_REQUESTS),
)
_rate_limit_windows[user_email].append(now)
# --- Response Models ---
class ArtistResult(BaseModel):
"""Artist search result from MusicBrainz/Spotify catalog."""
name: str
mbid: Optional[str] = None
disambiguation: Optional[str] = None
artist_type: Optional[str] = None
spotify_id: Optional[str] = None
popularity: Optional[int] = None
genres: Optional[list[str]] = None
tags: Optional[list[str]] = None
class TrackResult(BaseModel):
"""Track search result from Spotify catalog."""
track_name: str
artist_name: str
track_id: Optional[str] = None
artist_id: Optional[str] = None
popularity: Optional[int] = None
duration_ms: Optional[int] = None
explicit: Optional[bool] = None
class CommunityTrack(BaseModel):
"""A community-approved karaoke track from karaokenerds."""
brand_name: str
brand_code: str
youtube_url: str
is_community: bool = True
class CommunityCheckSong(BaseModel):
"""A song with community karaoke tracks."""
title: str
artist: str
community_tracks: list[CommunityTrack]
class CommunityCheckRequest(BaseModel):
"""Request body for community version check."""
artist: str = Field(..., min_length=1)
title: str = Field(..., min_length=1)
class CommunityCheckResponse(BaseModel):
"""Response for community version check."""
has_community: bool
songs: list[CommunityCheckSong]
best_youtube_url: Optional[str] = None
# --- Routes ---
@router.get("/artists", response_model=list[ArtistResult])
async def search_artists(
request: Request,
q: str = Query(..., min_length=2, description="Artist name search query"),
limit: int = Query(10, ge=1, le=50, description="Maximum results"),
auth_result: AuthResult = Depends(require_auth),
):
"""
Search for artists by name. Returns canonical MusicBrainz/Spotify artist data
for autocomplete in job creation forms.
"""
locale = get_locale_from_request(request)
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
from backend.services.catalog_proxy_service import search_artists
results = await search_artists(q, limit)
return results
@router.get("/tracks", response_model=list[TrackResult])
async def search_tracks(
request: Request,
q: str = Query(..., min_length=2, description="Track name search query"),
artist: Optional[str] = Query(None, description="Filter by artist name"),
limit: int = Query(10, ge=1, le=50, description="Maximum results"),
auth_result: AuthResult = Depends(require_auth),
):
"""
Search for tracks by name, optionally filtered by artist.
Returns canonical Spotify track data for autocomplete in job creation forms.
"""
locale = get_locale_from_request(request)
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
from backend.services.catalog_proxy_service import search_tracks
results = await search_tracks(q, artist, limit)
return results
@router.post("/community-check", response_model=CommunityCheckResponse)
async def check_community_versions(
request: Request,
body: CommunityCheckRequest,
auth_result: AuthResult = Depends(require_auth),
):
"""
Check if a song already has community-approved karaoke versions on YouTube
via karaokenerds.com. Non-blocking — just informs the user.
"""
locale = get_locale_from_request(request)
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
from backend.services.karaokenerds_service import check_community_versions
result = await check_community_versions(body.artist, body.title)
return result
+65
View File
@@ -0,0 +1,65 @@
"""
Encoding worker lifecycle endpoints.
Provides warmup and heartbeat endpoints for the blue-green
encoding worker VMs. Called by the frontend to ensure the
primary VM is running before encoding requests.
"""
import logging
from fastapi import APIRouter, Depends
from backend.api.dependencies import require_admin
from backend.services.encoding_worker_manager import EncodingWorkerManager
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/internal/encoding-worker", tags=["encoding-worker"])
# Lazy-loaded singleton
_manager = None
def get_worker_manager():
global _manager
if _manager is None:
from google.cloud import compute_v1, firestore
from backend.config import get_settings
settings = get_settings()
db = firestore.Client(project=settings.google_cloud_project)
compute_client = compute_v1.InstancesClient()
_manager = EncodingWorkerManager(
db=db,
compute_client=compute_client,
project_id=settings.google_cloud_project,
)
return _manager
@router.post("/warmup")
async def warmup_encoding_worker(
_admin=Depends(require_admin),
manager=Depends(get_worker_manager),
):
"""Start the primary encoding worker VM if it's stopped."""
try:
result = manager.ensure_primary_running()
if result["started"]:
logger.info(f"Started encoding worker VM: {result['vm_name']}")
return result
except Exception as e:
logger.error(f"Failed to warm up encoding worker: {e}")
return {"started": False, "error": str(e)}
@router.post("/heartbeat")
async def heartbeat_encoding_worker(
_admin=Depends(require_admin),
manager=Depends(get_worker_manager),
):
"""Update activity timestamp to prevent idle shutdown."""
try:
manager.update_activity()
return {"status": "ok"}
except Exception as e:
logger.error(f"Failed to update encoding worker heartbeat: {e}")
return {"status": "error", "error": str(e)}
File diff suppressed because it is too large. Load diff
+240 -3
View File
@@ -5,10 +5,12 @@ import os
import logging
import subprocess
import platform
from fastapi import APIRouter
from typing import Dict, Any, Optional
from fastapi import APIRouter, Depends
from typing import Dict, Any, Optional, Tuple
from backend.version import VERSION
from backend.version import VERSION, COMMIT_SHA, PR_NUMBER, PR_TITLE, STARTUP_TIME
from backend.api.dependencies import optional_auth
from backend.services.auth_service import UserType
from backend.services.flacfetch_client import get_flacfetch_client
from backend.services.email_service import get_email_service
from backend.services.stripe_service import get_stripe_service
@@ -16,6 +18,9 @@ from backend.services.encoding_service import get_encoding_service
from backend.services.spacy_preloader import get_preloaded_model, is_model_preloaded
from backend.services.nltk_preloader import get_preloaded_cmudict, is_cmudict_preloaded
from backend.services.langfuse_preloader import get_preloaded_langfuse_handler, is_langfuse_preloaded, is_langfuse_configured
from backend.services.job_health_service import check_job_consistency_detailed
from backend.services.job_manager import JobManager
from backend.models.job import JobStatus
router = APIRouter()
logger = logging.getLogger(__name__)
@@ -305,6 +310,175 @@ async def flacfetch_health() -> Dict[str, Any]:
}
def check_audio_separator_status() -> Dict[str, Any]:
"""Check audio separator availability without calling the remote GPU service."""
from backend.config import get_settings
settings = get_settings()
if not settings.audio_separator_api_url:
return {
"available": False,
"status": "not_configured",
}
try:
from importlib.metadata import version
pkg_version = version("audio-separator")
except Exception:
pkg_version = None
return {
"available": True,
"status": "ok",
"version": pkg_version,
}
@router.get("/health/audio-separator")
async def audio_separator_health() -> Dict[str, Any]:
"""
Return the installed audio-separator package version.
Does NOT call the remote GPU service — that would cold-start an expensive
Cloud Run GPU instance (~$0.07 per wake). Instead reports the locally
installed package version, which matches what the audio worker uses.
No authentication required.
"""
from backend.config import get_settings
settings = get_settings()
if not settings.audio_separator_api_url:
return {
"available": False,
"status": "not_configured",
}
try:
from importlib.metadata import version
pkg_version = version("audio-separator")
except Exception:
pkg_version = None
return {
"available": True,
"status": "ok",
"version": pkg_version,
}
def _get_encoding_worker_manager():
"""Get the EncodingWorkerManager instance. Separated for testability."""
try:
encoding_service = get_encoding_service()
if hasattr(encoding_service, '_worker_manager') and encoding_service._worker_manager:
return encoding_service._worker_manager
except Exception:
pass
return None
@router.get("/health/system-status")
async def system_status(
auth: Optional[Tuple] = Depends(optional_auth),
) -> Dict[str, Any]:
"""
Aggregated system status for all services.
Returns basic info for everyone. Admin users get additional
blue-green deployment details for the encoder.
"""
is_admin = auth is not None and len(auth) >= 2 and auth[1] == UserType.ADMIN
# Fetch all health statuses concurrently
import asyncio
encoding_task = check_encoding_worker_status()
flacfetch_task = check_flacfetch_service_status()
encoding_status, flacfetch_status = await asyncio.gather(
encoding_task, flacfetch_task
)
separator_status = check_audio_separator_status()
# Build frontend service info (from env vars baked at build time — returned as-is)
frontend_svc = {
"status": "ok",
"version": VERSION, # Frontend and backend share the same version from pyproject.toml
}
# Build backend service info
backend_svc = {
"status": "ok",
"version": VERSION,
"deployed_at": STARTUP_TIME,
}
if COMMIT_SHA:
backend_svc["commit_sha"] = COMMIT_SHA
if PR_NUMBER:
backend_svc["pr_number"] = PR_NUMBER
if PR_TITLE:
backend_svc["pr_title"] = PR_TITLE
# Build encoder service info
encoder_svc = {
"status": "ok" if encoding_status.get("available") else "offline",
"version": encoding_status.get("wheel_version"),
"active_jobs": encoding_status.get("active_jobs", 0),
}
# Build flacfetch service info
flacfetch_svc = {
"status": "ok" if flacfetch_status.get("available") else "offline",
"version": flacfetch_status.get("version"),
}
# Build separator service info
separator_svc = {
"status": "ok" if separator_status.get("available") else "offline",
"version": separator_status.get("version"),
}
# Admin-only: blue-green encoder details from Firestore
if is_admin:
manager = _get_encoding_worker_manager()
if manager:
try:
config = manager.get_config()
encoder_svc["admin_details"] = {
"primary_vm": config.primary_vm,
"primary_ip": config.primary_ip,
"primary_version": config.primary_version,
"primary_deployed_at": config.primary_deployed_at,
"secondary_vm": config.secondary_vm,
"secondary_ip": config.secondary_ip,
"secondary_version": config.secondary_version,
"secondary_deployed_at": config.secondary_deployed_at,
"last_swap_at": config.last_swap_at,
"deploy_in_progress": config.deploy_in_progress,
"active_jobs": encoding_status.get("active_jobs", 0),
"queue_length": encoding_status.get("queue_length", 0),
}
except Exception as e:
logger.warning(f"Failed to get blue-green config: {e}")
# Add error details for offline services
if not flacfetch_status.get("available"):
flacfetch_svc["admin_details"] = {"error": flacfetch_status.get("error")}
if not separator_status.get("available"):
separator_svc["admin_details"] = {"error": separator_status.get("error")}
if not encoding_status.get("available") and "admin_details" not in encoder_svc:
encoder_svc["admin_details"] = {"error": encoding_status.get("error")}
return {
"services": {
"frontend": frontend_svc,
"backend": backend_svc,
"encoder": encoder_svc,
"flacfetch": flacfetch_svc,
"separator": separator_svc,
}
}
@router.get("/health/detailed")
async def detailed_health_check() -> Dict[str, Any]:
"""
@@ -430,6 +604,69 @@ async def preload_status() -> Dict[str, Any]:
}
@router.get("/health/job-consistency")
async def job_consistency_check(
status: Optional[str] = None,
limit: int = 50
) -> Dict[str, Any]:
"""
Check job consistency across the system.
This endpoint identifies jobs with state inconsistencies that may indicate
bugs in the state machine or trigger logic. Use this to find stuck jobs
or jobs in invalid states.
Args:
status: Optional filter by job status (e.g., "pending", "downloading")
limit: Maximum number of jobs to check (default 50)
Returns:
Dictionary with:
- total_checked: Number of jobs checked
- inconsistent_count: Number of jobs with issues
- inconsistent_jobs: List of job details with their issues
- summary: Summary of issue types found
"""
job_manager = JobManager()
# Query jobs to check
query_params = {}
if status:
try:
job_status = JobStatus(status)
query_params['status'] = job_status
except ValueError:
return {
"error": f"Invalid status: {status}",
"valid_statuses": [s.value for s in JobStatus]
}
# Get jobs from Firestore
jobs = job_manager.list_jobs(limit=limit, **query_params)
# Check each job for consistency
inconsistent_jobs = []
issue_summary = {}
for job in jobs:
result = check_job_consistency_detailed(job)
if not result['is_healthy']:
inconsistent_jobs.append(result)
# Count issue types
for issue in result['issues']:
issue_type = issue.split(':')[0] if ':' in issue else issue
issue_summary[issue_type] = issue_summary.get(issue_type, 0) + 1
return {
"total_checked": len(jobs),
"inconsistent_count": len(inconsistent_jobs),
"inconsistent_jobs": inconsistent_jobs,
"summary": issue_summary,
"status_filter": status,
"limit": limit,
}
@router.get("/readiness")
async def readiness_check() -> Dict[str, str]:
"""Readiness check endpoint for Cloud Run."""
+207 -1
View File
@@ -25,7 +25,7 @@ from backend.workers.screens_worker import generate_screens
from backend.workers.video_worker import generate_video
from backend.workers.render_video_worker import process_render_video
from backend.api.dependencies import require_admin
from backend.services.auth_service import UserType
from backend.services.auth_service import AuthResult, UserType
from backend.services.job_manager import JobManager
from backend.services.tracing import (
extract_trace_context,
@@ -76,6 +76,18 @@ def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerR
message=f"Job {job_id} not found"
)
# Reject workers for jobs in terminal states (failed, cancelled, complete)
# This prevents Cloud Tasks retries from triggering workers on dead jobs
terminal_statuses = {'failed', 'cancelled', 'complete', 'prep_complete', 'error'}
job_status = job.status.value if hasattr(job.status, 'value') else str(job.status)
if job_status in terminal_statuses:
logger.info(f"[job:{job_id}] Job in terminal state '{job_status}', skipping {worker_name} worker")
return WorkerResponse(
status="skipped",
job_id=job_id,
message=f"Job is in terminal state '{job_status}', {worker_name} worker not needed"
)
# Check worker-specific progress in state_data
progress_key = f"{worker_name}_progress"
worker_progress = job.state_data.get(progress_key, {})
@@ -102,6 +114,31 @@ def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerR
return None
@router.post("/sync-disposable-domains", response_model=WorkerResponse)
async def sync_disposable_domains_endpoint(
request: Request,
auth_result: AuthResult = Depends(require_admin),
):
"""Sync external disposable domain blocklist. Called by Cloud Scheduler daily."""
import asyncio
from backend.services.disposable_domain_sync_service import (
fetch_external_blocklist, sync_disposable_domains
)
from backend.services.firestore_service import get_firestore_client
from backend.services.email_validation_service import EmailValidationService
domains = await fetch_external_blocklist()
db = get_firestore_client()
result = await asyncio.to_thread(sync_disposable_domains, db, domains)
EmailValidationService._blocklist_cache = None
return WorkerResponse(
status="completed",
job_id="sync-disposable-domains",
message=f"Synced {result['external_count']} domains"
)
@router.post("/workers/audio", response_model=WorkerResponse)
async def trigger_audio_worker(
request: WorkerRequest,
@@ -428,6 +465,175 @@ async def check_idle_reminder(
return {"status": "error", "job_id": job_id, "message": str(e)}
@router.post("/youtube-queue/process")
async def process_youtube_upload_queue(
http_request: Request,
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Process queued YouTube uploads.
Called by Cloud Scheduler (hourly) to retry uploads that were deferred
due to YouTube API quota exhaustion. Can also be triggered manually
from the admin dashboard.
Processes queued uploads one at a time, stopping if quota is exhausted.
"""
from backend.workers.youtube_queue_processor import process_youtube_upload_queue as process_queue
trace_context = extract_trace_context(dict(http_request.headers))
logger.info("YOUTUBE_QUEUE_PROCESS starting")
add_span_attribute("operation", "youtube_queue_process")
# Run in background so the HTTP response returns quickly
# (Cloud Scheduler has a 30min timeout but we don't want to block)
async def _process():
try:
result = await process_queue()
logger.info(f"YOUTUBE_QUEUE_PROCESS complete: {result}")
except Exception as e:
logger.exception(f"YOUTUBE_QUEUE_PROCESS failed: {e}")
background_tasks.add_task(_process)
add_span_event("queue_processing_started")
return {
"status": "started",
"message": "YouTube upload queue processing started in background"
}
@router.post("/process-stale-reviews")
async def process_stale_reviews_endpoint(
http_request: Request,
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Process stale review jobs — send reminders and expire old ones.
Called by Cloud Scheduler (hourly). Can also be triggered manually
from the admin dashboard.
- Jobs in review for >= 24h get a reminder email
- Jobs in review for >= 48h are auto-cancelled with credit refund
"""
from backend.workers.stale_review_processor import process_stale_reviews
trace_context = extract_trace_context(dict(http_request.headers))
logger.info("STALE_REVIEW_PROCESS starting")
add_span_attribute("operation", "stale_review_process")
async def _process():
try:
result = await process_stale_reviews()
logger.info(f"STALE_REVIEW_PROCESS complete: {result}")
except Exception as e:
logger.exception(f"STALE_REVIEW_PROCESS failed: {e}")
background_tasks.add_task(_process)
add_span_event("stale_review_processing_started")
return {
"status": "started",
"message": "Stale review processing started in background"
}
@router.post("/trigger-gdrive-validation")
async def trigger_gdrive_validation_endpoint(
http_request: Request,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Trigger GDrive validation via the Cloud Function.
Called by a Cloud Tasks delayed task after a job completes and uploads
to the public GDrive share. The 5-minute delay ensures E2E test cleanup
finishes before the validator checks for sequence gaps.
"""
from backend.services.gdrive_validator_client import trigger_gdrive_validation
trace_context = extract_trace_context(dict(http_request.headers))
logger.info("GDRIVE_VALIDATION_TRIGGER starting (delayed post-job)")
add_span_attribute("operation", "gdrive_validation_trigger")
try:
result = trigger_gdrive_validation()
if result is None:
return {"status": "skipped", "message": "GDRIVE_VALIDATOR_URL not configured"}
status = result.get("status", "unknown")
add_span_event("validation_complete", {"result_status": status})
return {"status": status, "message": "GDrive validation completed", "result": result}
except Exception as e:
logger.exception(f"GDrive validation trigger failed: {e}")
add_span_event("error", {"error": str(e)})
return {"status": "error", "message": str(e)}
@router.post("/recover-stuck-jobs")
async def recover_stuck_jobs(
http_request: Request,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Detect and recover jobs stuck in downloading_audio status.
Called by Cloud Scheduler (every 5 minutes) or manually from admin.
Finds jobs stuck in DOWNLOADING_AUDIO for >10 minutes and fails them
so they can be retried.
"""
from backend.services.job_health_service import check_job_consistency
trace_context = extract_trace_context(dict(http_request.headers))
logger.info("RECOVER_STUCK_JOBS starting")
add_span_attribute("operation", "recover_stuck_jobs")
job_manager = JobManager()
# Query jobs in downloading_audio status
from google.cloud.firestore_v1 import FieldFilter
jobs_ref = job_manager.firestore.db.collection("jobs")
stuck_query = jobs_ref.where(
filter=FieldFilter("status", "==", "downloading_audio")
).stream()
recovered = []
for doc in stuck_query:
job_data = doc.to_dict()
job_id = job_data.get("job_id", doc.id)
# Use the health service to check if actually stuck
job = job_manager.get_job(job_id)
if not job:
continue
issues = check_job_consistency(job)
stuck_issues = [i for i in issues if "downloading_audio_stuck" in i]
if stuck_issues:
logger.warning(f"[job:{job_id}] Recovering stuck download: {stuck_issues[0]}")
job_manager.fail_job(
job_id,
"Audio download timed out (stuck >10 minutes). Use retry to re-attempt."
)
recovered.append(job_id)
logger.info(f"RECOVER_STUCK_JOBS complete: recovered={len(recovered)} jobs")
add_span_event("recovery_complete", {"recovered_count": len(recovered)})
return {
"status": "success",
"recovered_jobs": recovered,
"recovered_count": len(recovered),
}
@router.get("/health")
async def internal_health(
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
+1318 -100
View File
File diff suppressed because it is too large. Load diff
+43 -18
View File
@@ -11,13 +11,14 @@ Provides endpoints for managing Web Push notification subscriptions:
import logging
from typing import Optional, Dict, List
from fastapi import APIRouter, Depends, HTTPException
from fastapi import APIRouter, Depends, HTTPException, Request
from pydantic import BaseModel
from backend.config import get_settings
from backend.api.dependencies import require_auth, require_admin
from backend.services.auth_service import AuthResult
from backend.services.push_notification_service import get_push_notification_service
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
@@ -37,6 +38,7 @@ class SubscribeRequest(BaseModel):
endpoint: str
keys: Dict[str, str] # p256dh and auth
device_name: Optional[str] = None
tenant_id: Optional[str] = None # Tenant scope for notification isolation
class SubscribeResponse(BaseModel):
@@ -111,7 +113,8 @@ async def get_vapid_public_key():
@router.post("/subscribe", response_model=SubscribeResponse)
async def subscribe_push(
request: SubscribeRequest,
subscribe_request: SubscribeRequest,
http_request: Request,
auth_result: AuthResult = Depends(require_auth)
):
"""
@@ -119,39 +122,53 @@ async def subscribe_push(
Requires authentication. Users can have up to 5 subscriptions
(configurable via MAX_PUSH_SUBSCRIPTIONS_PER_USER).
Subscriptions are scoped by tenant_id to prevent cross-tenant
notification leakage. The tenant is resolved from:
1. Request body tenant_id field (explicit from frontend)
2. Tenant middleware (X-Tenant-ID header or subdomain detection)
"""
locale = get_locale_from_request(http_request)
settings = get_settings()
if not settings.enable_push_notifications:
raise HTTPException(status_code=503, detail="Push notifications are not enabled")
raise HTTPException(status_code=503, detail=t(locale, "push.disabled"))
if not auth_result.user_email:
raise HTTPException(status_code=401, detail="User email not available")
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
push_service = get_push_notification_service()
# Validate keys
if "p256dh" not in request.keys or "auth" not in request.keys:
raise HTTPException(status_code=400, detail="Missing required keys (p256dh, auth)")
if "p256dh" not in subscribe_request.keys or "auth" not in subscribe_request.keys:
raise HTTPException(status_code=400, detail=t(locale, "push.missingEncryptionKeys"))
# Resolve tenant_id: prefer explicit from body, fall back to middleware
tenant_id = subscribe_request.tenant_id
if tenant_id is None:
tenant_id = getattr(http_request.state, "tenant_id", None)
success = await push_service.add_subscription(
user_email=auth_result.user_email,
endpoint=request.endpoint,
keys=request.keys,
device_name=request.device_name
endpoint=subscribe_request.endpoint,
keys=subscribe_request.keys,
device_name=subscribe_request.device_name,
tenant_id=tenant_id or None,
)
if not success:
raise HTTPException(status_code=500, detail="Failed to save subscription")
raise HTTPException(status_code=500, detail=t(locale, "push.subscriptionSaveFailed"))
return SubscribeResponse(
status="success",
message="Push subscription registered successfully"
message=t(locale, "push.subscriptionRegistered")
)
@router.post("/unsubscribe", response_model=UnsubscribeResponse)
async def unsubscribe_push(
request: UnsubscribeRequest,
http_request: Request,
auth_result: AuthResult = Depends(require_auth)
):
"""
@@ -159,8 +176,10 @@ async def unsubscribe_push(
Requires authentication. Users can only remove their own subscriptions.
"""
locale = get_locale_from_request(http_request)
if not auth_result.user_email:
raise HTTPException(status_code=401, detail="User email not available")
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
push_service = get_push_notification_service()
@@ -173,17 +192,18 @@ async def unsubscribe_push(
# Don't error if subscription wasn't found - might already be removed
return UnsubscribeResponse(
status="success",
message="Subscription removed (or was not found)"
message=t(locale, "push.subscriptionNotFound")
)
return UnsubscribeResponse(
status="success",
message="Push subscription removed successfully"
message=t(locale, "push.subscriptionRemoved")
)
@router.get("/subscriptions", response_model=SubscriptionsListResponse)
async def list_subscriptions(
request: Request,
auth_result: AuthResult = Depends(require_auth)
):
"""
@@ -191,8 +211,10 @@ async def list_subscriptions(
Requires authentication.
"""
locale = get_locale_from_request(request)
if not auth_result.user_email:
raise HTTPException(status_code=401, detail="User email not available")
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
push_service = get_push_notification_service()
@@ -207,6 +229,7 @@ async def list_subscriptions(
@router.post("/test", response_model=TestNotificationResponse)
async def send_test_notification(
request: TestNotificationRequest,
http_request: Request,
auth_result: AuthResult = Depends(require_admin)
):
"""
@@ -214,12 +237,14 @@ async def send_test_notification(
Admin only. Useful for testing push notification setup.
"""
locale = get_locale_from_request(http_request)
settings = get_settings()
if not settings.enable_push_notifications:
raise HTTPException(status_code=503, detail="Push notifications are not enabled")
raise HTTPException(status_code=503, detail=t(locale, "push.disabled"))
if not auth_result.user_email:
raise HTTPException(status_code=401, detail="User email not available")
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
push_service = get_push_notification_service()
@@ -234,5 +259,5 @@ async def send_test_notification(
return TestNotificationResponse(
status="success",
sent_count=sent_count,
message=f"Test notification sent to {sent_count} device(s)"
message=t(locale, "push.testNotificationSent", sent_count=sent_count)
)
+160 -242
View File
@@ -1,24 +1,22 @@
"""
Admin API routes for rate limit management.
Admin API routes for blocklist and YouTube queue management.
Handles:
- Rate limit statistics and monitoring
- Blocklist management (disposable domains, blocked emails, blocked IPs)
- User override management (whitelist/bypass permissions)
- YouTube upload queue management
"""
import logging
from datetime import datetime, timezone
from datetime import datetime
from typing import List, Optional
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, EmailStr
from fastapi import APIRouter, Depends, HTTPException, Request
from pydantic import BaseModel
from backend.api.dependencies import require_admin
from backend.services.auth_service import AuthResult
from backend.services.rate_limit_service import get_rate_limit_service, RateLimitService
from backend.services.email_validation_service import get_email_validation_service, EmailValidationService
from backend.config import settings
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/admin/rate-limits", tags=["admin", "rate-limits"])
@@ -28,45 +26,40 @@ router = APIRouter(prefix="/admin/rate-limits", tags=["admin", "rate-limits"])
# Response Models
# =============================================================================
class RateLimitStatsResponse(BaseModel):
"""Current rate limit statistics."""
# Configuration
jobs_per_day_limit: int
youtube_uploads_per_day_limit: int
beta_ip_per_day_limit: int
rate_limiting_enabled: bool
# Current usage
youtube_uploads_today: int
youtube_uploads_remaining: int
# Blocklist stats
disposable_domains_count: int
blocked_emails_count: int
blocked_ips_count: int
# User override stats
total_overrides: int
class YouTubeQueueEntry(BaseModel):
"""A queued YouTube upload entry."""
job_id: str
status: str
reason: Optional[str] = None
user_email: Optional[str] = None
artist: Optional[str] = None
title: Optional[str] = None
brand_code: Optional[str] = None
queued_at: Optional[datetime] = None
attempts: int = 0
max_attempts: int = 5
last_error: Optional[str] = None
youtube_url: Optional[str] = None
notification_sent: bool = False
class UserRateLimitStatusResponse(BaseModel):
"""Rate limit status for a specific user."""
email: str
jobs_today: int
jobs_limit: int
jobs_remaining: int
has_bypass: bool
custom_limit: Optional[int]
bypass_reason: Optional[str]
class YouTubeQueueListResponse(BaseModel):
"""List of YouTube upload queue entries."""
entries: List[YouTubeQueueEntry]
stats: dict
class BlocklistsResponse(BaseModel):
"""All blocklist data."""
disposable_domains: List[str]
"""All blocklist data with domain source separation."""
external_domains: List[str]
manual_domains: List[str]
allowlisted_domains: List[str]
blocked_emails: List[str]
blocked_ips: List[str]
updated_at: Optional[datetime]
updated_by: Optional[str]
last_sync_at: Optional[datetime] = None
last_sync_count: Optional[int] = None
updated_at: Optional[datetime] = None
updated_by: Optional[str] = None
class DomainRequest(BaseModel):
@@ -84,115 +77,12 @@ class IPRequest(BaseModel):
ip_address: str
class UserOverride(BaseModel):
"""User override configuration."""
email: str
bypass_job_limit: bool
custom_daily_job_limit: Optional[int]
reason: str
created_by: str
created_at: datetime
class UserOverrideRequest(BaseModel):
"""Request to set/update a user override."""
bypass_job_limit: bool = False
custom_daily_job_limit: Optional[int] = None
reason: str
class UserOverridesListResponse(BaseModel):
"""List of all user overrides."""
overrides: List[UserOverride]
total: int
class SuccessResponse(BaseModel):
"""Generic success response."""
success: bool
message: str
# =============================================================================
# Statistics Endpoints
# =============================================================================
@router.get("/stats", response_model=RateLimitStatsResponse)
async def get_rate_limit_stats(
auth_result: AuthResult = Depends(require_admin),
):
"""
Get current rate limit statistics.
Returns configuration values, current usage counts, and blocklist stats.
"""
rate_limit_service = get_rate_limit_service()
email_validation = get_email_validation_service()
# Get YouTube uploads today
youtube_today = rate_limit_service.get_youtube_uploads_today()
youtube_limit = settings.rate_limit_youtube_uploads_per_day
youtube_remaining = max(0, youtube_limit - youtube_today) if youtube_limit > 0 else -1
# Get blocklist stats
blocklist_stats = email_validation.get_blocklist_stats()
# Get override count
overrides = rate_limit_service.get_all_overrides()
return RateLimitStatsResponse(
jobs_per_day_limit=settings.rate_limit_jobs_per_day,
youtube_uploads_per_day_limit=settings.rate_limit_youtube_uploads_per_day,
beta_ip_per_day_limit=settings.rate_limit_beta_ip_per_day,
rate_limiting_enabled=settings.enable_rate_limiting,
youtube_uploads_today=youtube_today,
youtube_uploads_remaining=youtube_remaining,
disposable_domains_count=blocklist_stats["disposable_domains_count"],
blocked_emails_count=blocklist_stats["blocked_emails_count"],
blocked_ips_count=blocklist_stats["blocked_ips_count"],
total_overrides=len(overrides),
)
@router.get("/users/{email}", response_model=UserRateLimitStatusResponse)
async def get_user_rate_limit_status(
email: str,
auth_result: AuthResult = Depends(require_admin),
):
"""
Get rate limit status for a specific user.
Returns their current usage and any override settings.
"""
rate_limit_service = get_rate_limit_service()
jobs_today = rate_limit_service.get_user_job_count_today(email)
override = rate_limit_service.get_user_override(email)
if override:
has_bypass = override.get("bypass_job_limit", False)
custom_limit = override.get("custom_daily_job_limit")
bypass_reason = override.get("reason")
jobs_limit = custom_limit if custom_limit is not None else settings.rate_limit_jobs_per_day
else:
has_bypass = False
custom_limit = None
bypass_reason = None
jobs_limit = settings.rate_limit_jobs_per_day
jobs_remaining = max(0, jobs_limit - jobs_today) if not has_bypass else -1
return UserRateLimitStatusResponse(
email=email,
jobs_today=jobs_today,
jobs_limit=jobs_limit,
jobs_remaining=jobs_remaining,
has_bypass=has_bypass,
custom_limit=custom_limit,
bypass_reason=bypass_reason,
)
# =============================================================================
# Blocklist Management Endpoints
# =============================================================================
@@ -201,38 +91,10 @@ async def get_user_rate_limit_status(
async def get_blocklists(
auth_result: AuthResult = Depends(require_admin),
):
"""
Get all blocklist data.
Returns disposable domains, blocked emails, and blocked IPs.
"""
"""Get all blocklist data with domain source separation."""
email_validation = get_email_validation_service()
# Force refresh to get latest data
config = email_validation.get_blocklist_config(force_refresh=True)
# Get the raw document for metadata
from google.cloud import firestore
from backend.services.firestore_service import get_firestore_client
from backend.services.email_validation_service import BLOCKLISTS_COLLECTION, BLOCKLIST_CONFIG_DOC
db = get_firestore_client()
doc = db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC).get()
updated_at = None
updated_by = None
if doc.exists:
data = doc.to_dict()
updated_at = data.get("updated_at")
updated_by = data.get("updated_by")
return BlocklistsResponse(
disposable_domains=sorted(list(config["disposable_domains"])),
blocked_emails=sorted(list(config["blocked_emails"])),
blocked_ips=sorted(list(config["blocked_ips"])),
updated_at=updated_at,
updated_by=updated_by,
)
raw_data = email_validation.get_blocklist_raw_data()
return BlocklistsResponse(**raw_data)
@router.post("/blocklists/disposable-domains", response_model=SuccessResponse)
@@ -245,13 +107,13 @@ async def add_disposable_domain(
domain = request.domain.lower().strip()
if not domain or "." not in domain:
raise HTTPException(status_code=400, detail="Invalid domain format")
raise HTTPException(status_code=400, detail=t("en", "admin.invalidDomainFormat"))
email_validation.add_disposable_domain(domain, auth_result.user_email)
return SuccessResponse(
success=True,
message=f"Domain '{domain}' added to disposable domains blocklist"
message=t("en", "admin.domainAddedToBl", domain=domain)
)
@@ -265,11 +127,62 @@ async def remove_disposable_domain(
domain = domain.lower().strip()
if not email_validation.remove_disposable_domain(domain, auth_result.user_email):
raise HTTPException(status_code=404, detail="Domain not found in blocklist")
raise HTTPException(status_code=404, detail=t("en", "admin.domainNotFoundInBl"))
return SuccessResponse(
success=True,
message=f"Domain '{domain}' removed from disposable domains blocklist"
message=t("en", "admin.domainRemovedFromBl", domain=domain)
)
@router.post("/blocklists/allowlisted-domains", response_model=SuccessResponse)
async def add_allowlisted_domain(
request: DomainRequest,
auth_result: AuthResult = Depends(require_admin),
):
"""Add a domain to the allowlist (overrides external blocklist)."""
email_validation = get_email_validation_service()
domain = request.domain.lower().strip()
if not domain or "." not in domain:
raise HTTPException(status_code=400, detail=t("en", "admin.invalidDomainFormat"))
email_validation.add_allowlisted_domain(domain, auth_result.user_email)
return SuccessResponse(success=True, message=t("en", "admin.domainAddedToAllowlist", domain=domain))
@router.delete("/blocklists/allowlisted-domains/{domain}", response_model=SuccessResponse)
async def remove_allowlisted_domain(
domain: str,
auth_result: AuthResult = Depends(require_admin),
):
"""Remove a domain from the allowlist."""
email_validation = get_email_validation_service()
domain = domain.lower().strip()
if not email_validation.remove_allowlisted_domain(domain, auth_result.user_email):
raise HTTPException(status_code=404, detail=t("en", "admin.domainNotFoundInAllowlist"))
return SuccessResponse(success=True, message=t("en", "admin.domainRemovedFromAllowlist", domain=domain))
@router.post("/blocklists/sync", response_model=SuccessResponse)
async def trigger_sync(
auth_result: AuthResult = Depends(require_admin),
):
"""Manually trigger a sync of the external disposable domain blocklist."""
import asyncio
from backend.services.disposable_domain_sync_service import (
fetch_external_blocklist, sync_disposable_domains
)
from backend.services.firestore_service import get_firestore_client
domains = await fetch_external_blocklist()
db = get_firestore_client()
result = await asyncio.to_thread(sync_disposable_domains, db, domains)
# Invalidate cache
EmailValidationService._blocklist_cache = None
return SuccessResponse(
success=True,
message=t("en", "admin.syncComplete", external_count=result['external_count'], added=result['added'], removed=result['removed'])
)
@@ -283,13 +196,13 @@ async def add_blocked_email(
email = request.email.lower().strip()
if not email or "@" not in email:
raise HTTPException(status_code=400, detail="Invalid email format")
raise HTTPException(status_code=400, detail=t("en", "admin.invalidEmailFormat"))
email_validation.add_blocked_email(email, auth_result.user_email)
return SuccessResponse(
success=True,
message=f"Email '{email}' added to blocked emails list"
message=t("en", "admin.emailAddedToBl", email=email)
)
@@ -303,11 +216,11 @@ async def remove_blocked_email(
email = email.lower().strip()
if not email_validation.remove_blocked_email(email, auth_result.user_email):
raise HTTPException(status_code=404, detail="Email not found in blocklist")
raise HTTPException(status_code=404, detail=t("en", "admin.emailNotFoundInBl"))
return SuccessResponse(
success=True,
message=f"Email '{email}' removed from blocked emails list"
message=t("en", "admin.emailRemovedFromBl", email=email)
)
@@ -321,13 +234,13 @@ async def add_blocked_ip(
ip_address = request.ip_address.strip()
if not ip_address:
raise HTTPException(status_code=400, detail="Invalid IP address")
raise HTTPException(status_code=400, detail=t("en", "admin.invalidIPAddress"))
email_validation.add_blocked_ip(ip_address, auth_result.user_email)
return SuccessResponse(
success=True,
message=f"IP '{ip_address}' added to blocked IPs list"
message=t("en", "admin.ipAddedToBl", ip_address=ip_address)
)
@@ -341,88 +254,93 @@ async def remove_blocked_ip(
ip_address = ip_address.strip()
if not email_validation.remove_blocked_ip(ip_address, auth_result.user_email):
raise HTTPException(status_code=404, detail="IP not found in blocklist")
raise HTTPException(status_code=404, detail=t("en", "admin.ipNotFoundInBl"))
return SuccessResponse(
success=True,
message=f"IP '{ip_address}' removed from blocked IPs list"
message=t("en", "admin.ipRemovedFromBl", ip_address=ip_address)
)
# =============================================================================
# User Override Management Endpoints
# YouTube Upload Queue Management Endpoints
# =============================================================================
@router.get("/overrides", response_model=UserOverridesListResponse)
async def get_all_overrides(
@router.get("/youtube-queue", response_model=YouTubeQueueListResponse)
async def get_youtube_queue(
auth_result: AuthResult = Depends(require_admin),
):
"""Get all user overrides."""
rate_limit_service = get_rate_limit_service()
"""
Get YouTube upload queue entries and stats.
overrides_data = rate_limit_service.get_all_overrides()
Returns all queue entries (any status) and aggregate statistics.
"""
from backend.services.youtube_upload_queue_service import get_youtube_upload_queue_service
overrides = [
UserOverride(
email=email,
bypass_job_limit=data.get("bypass_job_limit", False),
custom_daily_job_limit=data.get("custom_daily_job_limit"),
reason=data.get("reason", ""),
created_by=data.get("created_by", "unknown"),
created_at=data.get("created_at", datetime.now(timezone.utc)),
queue_service = get_youtube_upload_queue_service()
entries = queue_service.get_all_queue_entries(limit=50)
stats = queue_service.get_queue_stats()
return YouTubeQueueListResponse(
entries=[YouTubeQueueEntry(**entry) for entry in entries],
stats=stats,
)
@router.post("/youtube-queue/{job_id}/retry", response_model=SuccessResponse)
async def retry_youtube_upload(
job_id: str,
auth_result: AuthResult = Depends(require_admin),
):
"""
Manually retry a failed YouTube upload.
Resets the entry back to 'queued' status with 0 attempts.
"""
from backend.services.youtube_upload_queue_service import get_youtube_upload_queue_service
queue_service = get_youtube_upload_queue_service()
if not queue_service.retry_upload(job_id):
raise HTTPException(
status_code=404,
detail=t("en", "admin.youtubeUploadNotFound", job_id=job_id)
)
for email, data in overrides_data.items()
]
return UserOverridesListResponse(
overrides=overrides,
total=len(overrides),
)
@router.put("/overrides/{email}", response_model=SuccessResponse)
async def set_user_override(
email: str,
request: UserOverrideRequest,
auth_result: AuthResult = Depends(require_admin),
):
"""Set or update a user override."""
rate_limit_service = get_rate_limit_service()
email = email.lower().strip()
if not email or "@" not in email:
raise HTTPException(status_code=400, detail="Invalid email format")
if not request.reason or len(request.reason.strip()) < 3:
raise HTTPException(status_code=400, detail="Reason is required (min 3 characters)")
rate_limit_service.set_user_override(
user_email=email,
bypass_job_limit=request.bypass_job_limit,
custom_daily_job_limit=request.custom_daily_job_limit,
reason=request.reason,
admin_email=auth_result.user_email,
)
return SuccessResponse(
success=True,
message=f"Override set for user '{email}'"
message=t("en", "admin.youtubeUploadQueuedForRetry", job_id=job_id)
)
@router.delete("/overrides/{email}", response_model=SuccessResponse)
async def remove_user_override(
email: str,
@router.post("/youtube-queue/process", response_model=SuccessResponse)
async def trigger_youtube_queue_processing(
auth_result: AuthResult = Depends(require_admin),
):
"""Remove a user override."""
rate_limit_service = get_rate_limit_service()
"""
Manually trigger YouTube upload queue processing.
email = email.lower().strip()
if not rate_limit_service.remove_user_override(email, auth_result.user_email):
raise HTTPException(status_code=404, detail="Override not found for user")
Same as the scheduled processor but triggered from the admin dashboard.
Runs in the background and returns immediately.
"""
from backend.workers.youtube_queue_processor import process_youtube_upload_queue
import asyncio
async def _process():
try:
result = await process_youtube_upload_queue()
logger.info(f"Manual YouTube queue processing complete: {result}")
except Exception as e:
logger.exception(f"Manual YouTube queue processing failed: {e}")
# Fire and forget
try:
loop = asyncio.get_running_loop()
loop.create_task(_process())
except RuntimeError:
pass
return SuccessResponse(
success=True,
message=f"Override removed for user '{email}'"
message=t("en", "admin.youtubeQueueProcessingStarted")
)
+374
View File
@@ -0,0 +1,374 @@
"""
Referral API routes.
Handles:
- Public referral interstitial (click tracking)
- Authenticated referral dashboard
- Stripe Connect onboarding
- Admin vanity link creation and link management
"""
import logging
from typing import Optional
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi.responses import Response
from pydantic import BaseModel, Field
from backend.api.dependencies import require_auth, require_admin
from backend.models.referral import (
CreateReferralLinkRequest,
UpdateReferralLinkRequest,
ReferralInterstitialResponse,
)
from backend.services.flyer_service import get_flyer_service, FlyerError
from backend.services.referral_service import get_referral_service
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/referrals", tags=["referrals"])
class GenerateFlyerRequest(BaseModel):
theme: str = Field(..., pattern="^(light|dark)$")
qr_data_url: str = Field(..., max_length=500_000)
# ============================================================================
# Public endpoints
# ============================================================================
@router.get("/r/{code}", response_model=ReferralInterstitialResponse)
async def get_referral_interstitial(code: str, no_track: bool = False):
"""
Public endpoint: get interstitial data for a referral link.
Increments click count unless no_track=true. Returns valid=False if code not found.
"""
service = get_referral_service()
link = service.get_link_by_code(code)
if not link:
return ReferralInterstitialResponse(valid=False)
# Track the click (skip when fetching for display purposes)
if not no_track:
try:
service.increment_clicks(code)
except Exception:
logger.warning("Failed to increment clicks for code %s", code, exc_info=True)
return ReferralInterstitialResponse(
valid=True,
referral_code=link.code,
referrer_display_name=link.display_name,
discount_percent=link.discount_percent,
discount_duration_days=link.discount_duration_days,
custom_message=link.custom_message,
)
# ============================================================================
# Authenticated user endpoints
# ============================================================================
@router.get("/me")
async def get_my_dashboard(auth=Depends(require_auth)):
"""Get referral dashboard data for the authenticated user."""
service = get_referral_service()
return service.get_dashboard_data(auth.user_email)
@router.put("/me")
async def update_my_link(
updates: UpdateReferralLinkRequest,
auth=Depends(require_auth),
):
"""Update display_name and custom_message on the user's own referral link."""
service = get_referral_service()
link = service.get_or_create_link(auth.user_email)
update_fields = {}
if updates.display_name is not None:
update_fields["display_name"] = updates.display_name
if updates.custom_message is not None:
update_fields["custom_message"] = updates.custom_message
if not update_fields:
return {"ok": True, "message": "No changes"}
success, message = service.update_link(link.code, **update_fields)
if not success:
raise HTTPException(status_code=400, detail=message)
return {"ok": True, "message": message}
@router.post("/me/connect")
async def start_stripe_connect(auth=Depends(require_auth)):
"""Start Stripe Connect onboarding for the authenticated user."""
# Check if user already has a Connect account
from backend.services.user_service import get_user_service
user_service = get_user_service()
user = user_service.get_user(auth.user_email)
if user and user.stripe_connect_account_id:
raise HTTPException(status_code=400, detail="Stripe Connect account already configured")
service = get_referral_service()
account_id, onboarding_url = service.create_connect_account(auth.user_email)
if not account_id:
raise HTTPException(status_code=500, detail="Failed to create Stripe Connect account")
user_service.update_user(auth.user_email, stripe_connect_account_id=account_id)
return {"account_id": account_id, "onboarding_url": onboarding_url}
@router.post("/me/connect/dashboard-link")
async def get_connect_dashboard_link(auth=Depends(require_auth)):
"""Get a login link to the Stripe Express dashboard for managing payout settings."""
from backend.services.user_service import get_user_service
user = get_user_service().get_user(auth.user_email)
if not user or not user.stripe_connect_account_id:
raise HTTPException(status_code=400, detail="No Stripe Connect account configured")
service = get_referral_service()
url = service.create_connect_login_link(user.stripe_connect_account_id)
if not url:
raise HTTPException(status_code=500, detail="Failed to create dashboard link")
return {"url": url}
@router.post("/me/connect/update-link")
async def get_connect_update_link(auth=Depends(require_auth)):
"""Get a link to update Stripe Connect account info (re-onboarding)."""
from backend.services.user_service import get_user_service
user = get_user_service().get_user(auth.user_email)
if not user or not user.stripe_connect_account_id:
raise HTTPException(status_code=400, detail="No Stripe Connect account configured")
service = get_referral_service()
url = service.create_connect_update_link(user.stripe_connect_account_id)
if not url:
raise HTTPException(status_code=500, detail="Failed to create update link")
return {"url": url}
class VanityRequest(BaseModel):
desired_code: str = Field(..., min_length=2, max_length=30, pattern="^[a-z0-9-]+$")
@router.post("/me/vanity-request")
async def request_vanity_url(
request: VanityRequest,
auth=Depends(require_auth),
):
"""Request a custom vanity referral URL. Sends email to admin for review."""
from backend.services.email_service import get_email_service
service = get_referral_service()
link = service.get_or_create_link(auth.user_email)
# Check if code is already taken
existing = service.get_link_by_code(request.desired_code)
if existing:
raise HTTPException(status_code=409, detail="That code is already taken. Please try another.")
email_service = get_email_service()
email_service.provider.send_email(
to_email="andrew@nomadkaraoke.com",
subject=f"Vanity referral URL request: {request.desired_code}",
html_content=f"""
<h2>Vanity Referral URL Request</h2>
<p><strong>User:</strong> {auth.user_email}</p>
<p><strong>Current code:</strong> {link.code}</p>
<p><strong>Desired vanity code:</strong> {request.desired_code}</p>
<p>To approve, create the vanity link in the <a href="https://gen.nomadkaraoke.com/admin/referrals">admin dashboard</a>.</p>
""",
text_content=f"Vanity URL request from {auth.user_email}: {request.desired_code} (current: {link.code})",
)
logger.info("Vanity URL request from %s: %s", auth.user_email, request.desired_code)
return {"ok": True, "message": "Request sent"}
@router.post("/me/flyer")
async def generate_flyer(
request: GenerateFlyerRequest,
auth=Depends(require_auth),
):
"""Generate a personalized referral flyer PDF."""
service = get_referral_service()
link = service.get_or_create_link(auth.user_email)
try:
flyer_service = get_flyer_service()
pdf_bytes = flyer_service.generate_pdf(
theme=request.theme,
referral_code=link.code,
discount_percent=link.discount_percent,
qr_data_url=request.qr_data_url,
)
except FlyerError as e:
raise HTTPException(status_code=400, detail=str(e))
return Response(
content=pdf_bytes,
media_type="application/pdf",
headers={
"Content-Disposition": 'attachment; filename="nomad-karaoke-referral-flyer.pdf"',
},
)
# ============================================================================
# Admin endpoints
# ============================================================================
@router.post("/admin/vanity")
async def create_vanity_link(
request: CreateReferralLinkRequest,
auth=Depends(require_admin),
):
"""Admin: create a vanity referral link."""
if not request.vanity_code:
raise HTTPException(status_code=400, detail="vanity_code is required")
service = get_referral_service()
success, link, message = service.create_vanity_link(
code=request.vanity_code,
owner_email=request.owner_email,
display_name=request.display_name,
custom_message=request.custom_message,
discount_percent=request.discount_percent,
kickback_percent=request.kickback_percent,
discount_duration_days=request.discount_duration_days,
earning_duration_days=request.earning_duration_days,
)
if not success:
raise HTTPException(status_code=400, detail=message)
return {"ok": True, "code": link.code, "message": message}
@router.get("/admin/links")
async def list_links(
limit: int = Query(50, ge=1, le=200),
offset: int = Query(0, ge=0),
auth=Depends(require_admin),
):
"""Admin: list all referral links."""
service = get_referral_service()
links = service.list_links(limit=limit, offset=offset)
return {
"links": [link.model_dump(mode="json") for link in links],
"count": len(links),
}
@router.post("/admin/apply-discount")
async def admin_apply_discount(
request: Request,
auth=Depends(require_admin),
):
"""Admin: apply a referral discount to any user, even without a referral link."""
from backend.services.user_service import get_user_service
body = await request.json()
user_email = body.get("email", "").lower()
discount_percent = body.get("discount_percent", 10)
duration_days = body.get("duration_days", 30)
referral_code = body.get("referral_code", "admin-grant")
if not user_email:
raise HTTPException(status_code=400, detail="Email is required")
user_service = get_user_service()
user = user_service.get_user(user_email)
if not user:
raise HTTPException(status_code=404, detail="User not found")
from datetime import datetime, timedelta
now = datetime.utcnow()
user_service.update_user(
user_email,
referred_by_code=referral_code,
referred_at=now,
referral_discount_expires_at=now + timedelta(days=duration_days),
)
# Create/ensure the Stripe coupon exists for this discount percent
service = get_referral_service()
service.get_or_create_stripe_coupon(discount_percent)
logger.info(f"Admin applied {discount_percent}% discount for {duration_days} days to {user_email}")
return {
"ok": True,
"message": f"Applied {discount_percent}% discount for {duration_days} days",
"discount_expires_at": (now + timedelta(days=duration_days)).isoformat(),
}
@router.put("/admin/links/{code}")
async def update_link(
code: str,
updates: UpdateReferralLinkRequest,
auth=Depends(require_admin),
):
"""Admin: update any referral link's settings."""
service = get_referral_service()
update_fields = {
k: v for k, v in updates.model_dump().items() if v is not None
}
if not update_fields:
return {"ok": True, "message": "No changes"}
success, message = service.update_link(code, **update_fields)
if not success:
raise HTTPException(status_code=404, detail=message)
return {"ok": True, "message": message}
@router.post("/admin/links/{code}/flyer")
async def admin_generate_flyer(
code: str,
request: GenerateFlyerRequest,
auth=Depends(require_admin),
):
"""Admin: generate a personalized referral flyer PDF for any code."""
service = get_referral_service()
link = service.get_link_by_code(code)
if not link:
raise HTTPException(status_code=404, detail=f"Referral link '{code}' not found")
try:
flyer_service = get_flyer_service()
pdf_bytes = flyer_service.generate_pdf(
theme=request.theme,
referral_code=link.code,
discount_percent=link.discount_percent,
qr_data_url=request.qr_data_url,
)
except FlyerError as e:
raise HTTPException(status_code=400, detail=str(e))
return Response(
content=pdf_bytes,
media_type="application/pdf",
headers={
"Content-Disposition": f'attachment; filename="nomad-karaoke-flyer-{link.code}.pdf"',
},
)
+1174 -100
View File
File diff suppressed because it is too large. Load diff
+2 -1
View File
@@ -12,6 +12,7 @@ from fastapi import APIRouter, Query, Request
from backend.models.tenant import TenantConfigResponse, TenantPublicConfig
from backend.services.tenant_service import get_tenant_service
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
@@ -115,6 +116,6 @@ async def get_tenant_asset(tenant_id: str, asset_name: str):
if not url:
from fastapi import HTTPException
raise HTTPException(status_code=404, detail="Asset not found")
raise HTTPException(status_code=404, detail=t("en", "tenant.assetNotFound"))
return RedirectResponse(url=url)
+18 -13
View File
@@ -9,10 +9,11 @@ before job creation.
import logging
from typing import Optional
from fastapi import APIRouter, HTTPException
from fastapi import APIRouter, HTTPException, Request
from backend.models.theme import ThemeDetailResponse, ThemesListResponse
from backend.services.theme_service import get_theme_service
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
@@ -20,7 +21,7 @@ router = APIRouter(prefix="/themes", tags=["themes"])
@router.get("", response_model=ThemesListResponse)
async def list_themes() -> ThemesListResponse:
async def list_themes(request: Request) -> ThemesListResponse:
"""
List all available themes.
@@ -29,6 +30,7 @@ async def list_themes() -> ThemesListResponse:
This endpoint is public and does not require authentication.
"""
locale = get_locale_from_request(request)
try:
theme_service = get_theme_service()
themes = theme_service.list_themes()
@@ -37,12 +39,12 @@ async def list_themes() -> ThemesListResponse:
logger.error(f"Error listing themes: {e}")
raise HTTPException(
status_code=500,
detail="Failed to load themes"
detail=t(locale, "themes.themesLoadError")
)
@router.get("/{theme_id}", response_model=ThemeDetailResponse)
async def get_theme(theme_id: str) -> ThemeDetailResponse:
async def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
"""
Get detailed information about a specific theme.
@@ -58,6 +60,7 @@ async def get_theme(theme_id: str) -> ThemeDetailResponse:
Raises:
404: Theme not found
"""
locale = get_locale_from_request(request)
try:
theme_service = get_theme_service()
theme = theme_service.get_theme(theme_id)
@@ -65,7 +68,7 @@ async def get_theme(theme_id: str) -> ThemeDetailResponse:
if theme is None:
raise HTTPException(
status_code=404,
detail=f"Theme not found: {theme_id}"
detail=t(locale, "themes.themeNotFound", theme_id=theme_id)
)
return ThemeDetailResponse(theme=theme)
@@ -75,12 +78,12 @@ async def get_theme(theme_id: str) -> ThemeDetailResponse:
logger.error(f"Error getting theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail="Failed to load theme"
detail=t(locale, "themes.themeLoadError")
)
@router.get("/{theme_id}/preview")
async def get_theme_preview(theme_id: str) -> dict:
async def get_theme_preview(theme_id: str, request: Request) -> dict:
"""
Get a signed URL for the theme's preview image.
@@ -96,20 +99,21 @@ async def get_theme_preview(theme_id: str) -> dict:
Raises:
404: Theme not found or no preview available
"""
locale = get_locale_from_request(request)
try:
theme_service = get_theme_service()
if not theme_service.theme_exists(theme_id):
raise HTTPException(
status_code=404,
detail=f"Theme not found: {theme_id}"
detail=t(locale, "themes.themeNotFound", theme_id=theme_id)
)
theme = theme_service.get_theme(theme_id)
if theme is None or theme.preview_url is None:
raise HTTPException(
status_code=404,
detail=f"No preview available for theme: {theme_id}"
detail=t(locale, "themes.previewNotAvailable", theme_id=theme_id)
)
return {"preview_url": theme.preview_url}
@@ -119,12 +123,12 @@ async def get_theme_preview(theme_id: str) -> dict:
logger.error(f"Error getting preview for theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail="Failed to load theme preview"
detail=t(locale, "themes.previewLoadError")
)
@router.get("/{theme_id}/youtube-description")
async def get_theme_youtube_description(theme_id: str) -> dict:
async def get_theme_youtube_description(theme_id: str, request: Request) -> dict:
"""
Get the YouTube description template for a theme.
@@ -141,13 +145,14 @@ async def get_theme_youtube_description(theme_id: str) -> dict:
Raises:
404: Theme not found
"""
locale = get_locale_from_request(request)
try:
theme_service = get_theme_service()
if not theme_service.theme_exists(theme_id):
raise HTTPException(
status_code=404,
detail=f"Theme not found: {theme_id}"
detail=t(locale, "themes.youtubeDescriptionNotFound", theme_id=theme_id)
)
description = theme_service.get_youtube_description(theme_id)
@@ -158,5 +163,5 @@ async def get_theme_youtube_description(theme_id: str) -> dict:
logger.error(f"Error getting YouTube description for theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail="Failed to load YouTube description"
detail=t(locale, "themes.youtubeDescriptionLoadError")
)
File diff suppressed because it is too large. Load diff
+35
View File
@@ -0,0 +1,35 @@
# Cloud Build config for GPU base image
#
# Builds the CUDA + PyTorch + models base image used by the audio worker.
# This image is large (~10GB) so builds take 15-30 minutes.
#
# Trigger manually:
# gcloud builds submit --config=backend/cloudbuild-gpu-base.yaml --project=nomadkaraoke --timeout=3600s
#
# The app-layer Dockerfile uses this as its BASE_IMAGE for audio worker deploys.
steps:
- name: 'gcr.io/cloud-builders/docker'
args:
- 'build'
- '-f'
- 'backend/Dockerfile.gpu-base'
- '-t'
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:latest'
- '-t'
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:$BUILD_ID'
- '--build-arg'
- 'BUILD_DATE=$_BUILD_DATE'
- '--build-arg'
- 'BUILD_ID=$BUILD_ID'
- '.'
images:
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:latest'
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:$BUILD_ID'
timeout: '3600s'
options:
machineType: 'E2_HIGHCPU_32'
diskSizeGb: 100
+30 -10
View File
@@ -32,6 +32,10 @@ class Settings(BaseSettings):
# Authentication
admin_tokens: Optional[str] = os.getenv("ADMIN_TOKENS") # Comma-separated list
scheduler_service_account: str = os.getenv(
"SCHEDULER_SERVICE_ACCOUNT",
"karaoke-backend@nomadkaraoke.iam.gserviceaccount.com"
)
# Application
environment: str = os.getenv("ENVIRONMENT", "development")
@@ -57,6 +61,12 @@ class Settings(BaseSettings):
# use uncorrected transcription - human review will fix any issues.
agentic_correction_timeout_seconds: int = int(os.getenv("AGENTIC_CORRECTION_TIMEOUT_SECONDS", "180"))
# AI Credit Evaluation (anti-abuse gating for free credits)
# When enabled, welcome and feedback credit grants are evaluated by Gemini
# before granting. Fail-open: if evaluation fails, credits are granted anyway.
credit_eval_enabled: bool = os.getenv("CREDIT_EVAL_ENABLED", "true").lower() in ("true", "1", "yes")
credit_eval_model: str = os.getenv("CREDIT_EVAL_MODEL", "gemini-3.1-pro-preview")
# Cloud Tasks (for scalable worker coordination)
# When enabled, workers are triggered via Cloud Tasks for guaranteed delivery
# When disabled (default), workers are triggered via direct HTTP (for development)
@@ -83,6 +93,9 @@ class Settings(BaseSettings):
# Requires use_gce_encoding to be enabled and the GCE worker to support /encode-preview endpoint.
use_gce_preview_encoding: bool = os.getenv("USE_GCE_PREVIEW_ENCODING", "false").lower() in ("true", "1", "yes")
# Karaoke-Decide catalog API (for song/artist autocomplete)
karaoke_decide_api_url: str = os.getenv("KARAOKE_DECIDE_API_URL", "https://decide.nomadkaraoke.com")
# Storage paths
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
@@ -97,7 +110,13 @@ class Settings(BaseSettings):
# This is required for torrent downloads since Cloud Run doesn't support BitTorrent.
flacfetch_api_url: Optional[str] = os.getenv("FLACFETCH_API_URL") # e.g., http://10.0.0.5:8080
flacfetch_api_key: Optional[str] = os.getenv("FLACFETCH_API_KEY")
# Flacfetch client retry settings
# Retries help handle transient network failures when calling the flacfetch service
flacfetch_retry_max_attempts: int = int(os.getenv("FLACFETCH_RETRY_MAX_ATTEMPTS", "9")) # Total attempts (1 initial + 8 retries)
flacfetch_retry_min_wait: float = float(os.getenv("FLACFETCH_RETRY_MIN_WAIT", "10.0")) # seconds
flacfetch_retry_max_wait: float = float(os.getenv("FLACFETCH_RETRY_MAX_WAIT", "60.0")) # seconds
# Default distribution settings (can be overridden per-request)
default_dropbox_path: Optional[str] = os.getenv("DEFAULT_DROPBOX_PATH")
default_gdrive_folder_id: Optional[str] = os.getenv("DEFAULT_GDRIVE_FOLDER_ID")
@@ -110,16 +129,17 @@ class Settings(BaseSettings):
default_enable_youtube_upload: bool = os.getenv("DEFAULT_ENABLE_YOUTUBE_UPLOAD", "false").lower() in ("true", "1", "yes")
default_brand_prefix: Optional[str] = os.getenv("DEFAULT_BRAND_PREFIX")
# Rate Limiting Configuration
# Enable/disable rate limiting system-wide (useful for development)
enable_rate_limiting: bool = os.getenv("ENABLE_RATE_LIMITING", "true").lower() in ("true", "1", "yes")
# Maximum jobs a user can create per day (0 = unlimited)
rate_limit_jobs_per_day: int = int(os.getenv("RATE_LIMIT_JOBS_PER_DAY", "5"))
# Maximum YouTube uploads system-wide per day (0 = unlimited)
rate_limit_youtube_uploads_per_day: int = int(os.getenv("RATE_LIMIT_YOUTUBE_UPLOADS_PER_DAY", "10"))
# Maximum beta enrollments from same IP per day (0 = unlimited)
rate_limit_beta_ip_per_day: int = int(os.getenv("RATE_LIMIT_BETA_IP_PER_DAY", "1"))
# Private (non-published) track distribution settings
default_private_dropbox_path: Optional[str] = os.getenv("DEFAULT_PRIVATE_DROPBOX_PATH")
default_private_brand_prefix: Optional[str] = os.getenv("DEFAULT_PRIVATE_BRAND_PREFIX", "NOMADNP")
# YouTube Data API v3 Quota Configuration
# Daily quota limit (YouTube grants 10,000 units/day by default)
youtube_quota_daily_limit: int = int(os.getenv("YOUTUBE_QUOTA_DAILY_LIMIT", "10000"))
# Estimated units consumed per upload (videos.insert=100 + search.list=100 + thumbnails.set=50 + buffer)
youtube_quota_upload_cost: int = int(os.getenv("YOUTUBE_QUOTA_UPLOAD_COST", "300"))
# Safety margin to reserve (avoid hitting exact limit)
youtube_quota_safety_margin: int = int(os.getenv("YOUTUBE_QUOTA_SAFETY_MARGIN", "500"))
# E2E test bypass key for rate limiting (set via secret in production)
e2e_bypass_key: str = os.getenv("E2E_BYPASS_KEY", "")
default_youtube_description: str = os.getenv(
+59 -31
View File
@@ -2,40 +2,10 @@
Custom exceptions for the karaoke generation backend.
These exceptions are used for structured error handling across the application,
particularly for rate limiting and validation errors.
particularly for validation errors and credit checks.
"""
class RateLimitExceededError(Exception):
"""
Raised when a rate limit is exceeded.
Includes information about when the limit will reset for client retry logic.
Attributes:
message: Human-readable error message
limit_type: Type of limit exceeded (e.g., "jobs_per_day", "youtube_uploads")
remaining_seconds: Seconds until the limit resets (for Retry-After header)
current_count: Current usage count
limit_value: The limit that was exceeded
"""
def __init__(
self,
message: str,
limit_type: str = "unknown",
remaining_seconds: int = 0,
current_count: int = 0,
limit_value: int = 0
):
self.message = message
self.limit_type = limit_type
self.remaining_seconds = remaining_seconds
self.current_count = current_count
self.limit_value = limit_value
super().__init__(message)
class EmailValidationError(Exception):
"""
Raised when email validation fails.
@@ -64,3 +34,61 @@ class IPBlockedError(Exception):
self.message = message
self.ip_address = ip_address
super().__init__(message)
class InsufficientCreditsError(Exception):
"""
Raised when a user attempts to create a job without sufficient credits.
Attributes:
message: Human-readable error message
credits_available: Number of credits the user currently has
credits_required: Number of credits required (always 1 for now)
"""
def __init__(
self,
message: str,
credits_available: int = 0,
credits_required: int = 1
):
self.message = message
self.credits_available = credits_available
self.credits_required = credits_required
super().__init__(message)
class InvalidStateTransitionError(Exception):
"""
Raised when an invalid job state transition is attempted.
This exception is raised by JobManager.transition_to_state() when the
requested transition is not allowed by the state machine defined in
STATE_TRANSITIONS.
Example: Trying to transition from PENDING to GENERATING_SCREENS is invalid
because PENDING can only transition to DOWNLOADING, SEARCHING_AUDIO, FAILED,
or CANCELLED.
Attributes:
message: Human-readable error message
job_id: The job ID that failed to transition
from_status: Current status of the job
to_status: Attempted target status
valid_transitions: List of valid transitions from current status
"""
def __init__(
self,
message: str,
job_id: str = "",
from_status: str = "",
to_status: str = "",
valid_transitions: list = None
):
self.message = message
self.job_id = job_id
self.from_status = from_status
self.to_status = to_status
self.valid_transitions = valid_transitions or []
super().__init__(message)
+99
View File
@@ -0,0 +1,99 @@
"""
Lightweight i18n utility for backend translations.
Usage:
from backend.i18n import t, get_locale_from_request, SUPPORTED_LOCALES
# In a route handler:
locale = get_locale_from_request(request)
message = t(locale, "auth.userNotFound")
message = t(locale, "jobs.notFound", job_id="abc123")
"""
import json
from functools import lru_cache
from pathlib import Path
from typing import Any
from starlette.requests import Request
SUPPORTED_LOCALES = ("en", "es", "de")
DEFAULT_LOCALE = "en"
TRANSLATIONS_DIR = Path(__file__).parent / "translations"
@lru_cache(maxsize=len(SUPPORTED_LOCALES))
def _load_translations(locale: str) -> dict[str, Any]:
"""Load and cache translation file for a locale."""
path = TRANSLATIONS_DIR / f"{locale}.json"
if not path.exists():
if locale != DEFAULT_LOCALE:
return _load_translations(DEFAULT_LOCALE)
return {}
with open(path, encoding="utf-8") as f:
result: dict[str, Any] = json.load(f)
return result
def t(locale: str, key: str, **kwargs: Any) -> str:
"""Get a translated string by dotted key path.
Args:
locale: Language code (en, es, de)
key: Dotted path like "auth.userNotFound" or "emails.magicLink.subject"
**kwargs: Interpolation variables (e.g., job_id="abc123")
Returns:
Translated string with variables interpolated, or the key itself if not found.
"""
if locale not in SUPPORTED_LOCALES:
locale = DEFAULT_LOCALE
translations = _load_translations(locale)
# Walk the dotted path
value: Any = translations
for part in key.split("."):
if isinstance(value, dict) and part in value:
value = value[part]
else:
# Key not found — fall back to English
if locale != DEFAULT_LOCALE:
return t(DEFAULT_LOCALE, key, **kwargs)
return key # Return the key itself as last resort
if not isinstance(value, str):
return key
result: str = value
# Interpolate variables
if kwargs:
try:
result = result.format(**kwargs)
except KeyError:
pass # Missing variable — return template as-is
return result
def get_locale_from_request(request: Request) -> str:
"""Extract locale from Accept-Language header.
Returns the first supported locale found, or DEFAULT_LOCALE.
"""
accept_lang = request.headers.get("accept-language", "")
# Simple parsing — Accept-Language can be "de", "de-DE", "en-US,en;q=0.9,de;q=0.8"
for part in accept_lang.split(","):
lang = part.split(";")[0].strip().split("-")[0].lower()
if lang in SUPPORTED_LOCALES:
return lang
return DEFAULT_LOCALE
def get_locale_prefix(locale: str) -> str:
"""Get URL locale prefix for frontend URLs (e.g., '/en', '/es')."""
if locale not in SUPPORTED_LOCALES:
locale = DEFAULT_LOCALE
return f"/{locale}"
+19 -16
View File
@@ -7,7 +7,7 @@ from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from backend.config import settings
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, rate_limits, push
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, rate_limits, push, catalog, encoding_worker
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
from backend.services.structured_logging import setup_structured_logging
from backend.services.spacy_preloader import preload_spacy_model
@@ -136,8 +136,8 @@ if tracing_enabled:
# Configure CORS
app.add_middleware(
CORSMiddleware,
allow_origins=["*"], # Configure this properly for production
allow_credentials=True,
allow_origins=["*"],
allow_credentials=False,
allow_methods=["*"],
allow_headers=["*"],
)
@@ -153,37 +153,40 @@ app.include_router(health.router, prefix="/api")
app.include_router(jobs.router, prefix="/api")
app.include_router(file_upload.router, prefix="/api") # File upload endpoint
app.include_router(internal.router, prefix="/api") # Internal worker endpoints
app.include_router(encoding_worker.router, prefix="/api") # Encoding worker lifecycle
app.include_router(review.router, prefix="/api") # Review UI compatibility endpoints
app.include_router(auth.router, prefix="/api") # OAuth credential management
app.include_router(audio_search.router, prefix="/api") # Audio search (artist+title mode)
app.include_router(themes.router, prefix="/api") # Theme selection for styles
app.include_router(users.router, prefix="/api") # User auth, credits, and Stripe webhooks
from backend.api.routes import referrals
app.include_router(referrals.router, prefix="/api") # Referral links and payouts
app.include_router(admin.router, prefix="/api") # Admin dashboard and management
app.include_router(rate_limits.router, prefix="/api") # Rate limits admin management
app.include_router(push.router, prefix="/api") # Push notification subscription management
app.include_router(catalog.router, prefix="/api") # Catalog proxy for song/artist autocomplete
app.include_router(tenant.router) # Tenant/white-label configuration (no /api prefix, router has it)
# Exception handler for rate limiting
# Exception handlers
from fastapi import Request
from fastapi.responses import JSONResponse
from backend.exceptions import RateLimitExceededError
from backend.exceptions import InsufficientCreditsError
from backend.i18n import t, get_locale_from_request
@app.exception_handler(RateLimitExceededError)
async def rate_limit_exception_handler(request: Request, exc: RateLimitExceededError):
"""Handle rate limit exceeded errors with 429 status."""
@app.exception_handler(InsufficientCreditsError)
async def insufficient_credits_exception_handler(request: Request, exc: InsufficientCreditsError):
"""Handle insufficient credits errors with 402 Payment Required status."""
locale = get_locale_from_request(request)
return JSONResponse(
status_code=429,
status_code=402,
content={
"detail": exc.message,
"limit_type": exc.limit_type,
"current_count": exc.current_count,
"limit_value": exc.limit_value,
"detail": t(locale, "errors.insufficientCredits"),
"credits_available": exc.credits_available,
"credits_required": exc.credits_required,
"buy_url": t(locale, "errors.insufficientCreditsBuyUrl"),
},
headers={
"Retry-After": str(exc.remaining_seconds),
} if exc.remaining_seconds > 0 else None,
)
+110
View File
@@ -0,0 +1,110 @@
"""
Audio edit session model for persisting input audio editing progress.
Stored as a Firestore subcollection: jobs/{job_id}/audio_edit_sessions/{session_id}
Full edit data is stored in GCS to avoid Firestore's 1MB document limit.
"""
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Optional, Dict, Any, List
import uuid
@dataclass
class AudioEditSessionSummary:
"""Compact summary of operations in an audio edit session, for list views."""
total_operations: int = 0
operations_breakdown: Dict[str, int] = field(default_factory=dict)
duration_change_seconds: float = 0.0
net_duration_seconds: float = 0.0
def to_dict(self) -> Dict[str, Any]:
return {
"total_operations": self.total_operations,
"operations_breakdown": self.operations_breakdown,
"duration_change_seconds": self.duration_change_seconds,
"net_duration_seconds": self.net_duration_seconds,
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "AudioEditSessionSummary":
return cls(
total_operations=data.get("total_operations", 0),
operations_breakdown=data.get("operations_breakdown", {}),
duration_change_seconds=data.get("duration_change_seconds", 0.0),
net_duration_seconds=data.get("net_duration_seconds", 0.0),
)
@dataclass
class AudioEditSession:
"""
A saved snapshot of an audio editing session.
Stored at: jobs/{job_id}/audio_edit_sessions/{session_id}
Edit data stored at: jobs/{job_id}/audio_edit_sessions/{session_id}.json (GCS)
"""
session_id: str = field(default_factory=lambda: str(uuid.uuid4()))
job_id: str = ""
user_email: str = ""
created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
updated_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
edit_count: int = 0
trigger: str = "auto" # "auto" | "manual" | "submit"
audio_duration_seconds: Optional[float] = None
original_duration_seconds: Optional[float] = None
artist: Optional[str] = None
title: Optional[str] = None
summary: AudioEditSessionSummary = field(default_factory=AudioEditSessionSummary)
edit_data_gcs_path: str = ""
# Hash of the edit data JSON, used to skip duplicate saves
data_hash: str = ""
def to_dict(self) -> Dict[str, Any]:
return {
"session_id": self.session_id,
"job_id": self.job_id,
"user_email": self.user_email,
"created_at": self.created_at,
"updated_at": self.updated_at,
"edit_count": self.edit_count,
"trigger": self.trigger,
"audio_duration_seconds": self.audio_duration_seconds,
"original_duration_seconds": self.original_duration_seconds,
"artist": self.artist,
"title": self.title,
"summary": self.summary.to_dict(),
"edit_data_gcs_path": self.edit_data_gcs_path,
"data_hash": self.data_hash,
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "AudioEditSession":
created_at = data.get("created_at")
if isinstance(created_at, str):
created_at = datetime.fromisoformat(created_at.rstrip("Z")).replace(tzinfo=timezone.utc)
updated_at = data.get("updated_at")
if isinstance(updated_at, str):
updated_at = datetime.fromisoformat(updated_at.rstrip("Z")).replace(tzinfo=timezone.utc)
summary_data = data.get("summary", {})
summary = AudioEditSessionSummary.from_dict(summary_data) if isinstance(summary_data, dict) else AudioEditSessionSummary()
return cls(
session_id=data.get("session_id", str(uuid.uuid4())),
job_id=data.get("job_id", ""),
user_email=data.get("user_email", ""),
created_at=created_at or datetime.now(timezone.utc),
updated_at=updated_at or datetime.now(timezone.utc),
edit_count=data.get("edit_count", 0),
trigger=data.get("trigger", "auto"),
audio_duration_seconds=data.get("audio_duration_seconds"),
original_duration_seconds=data.get("original_duration_seconds"),
artist=data.get("artist"),
title=data.get("title"),
summary=summary,
edit_data_gcs_path=data.get("edit_data_gcs_path", ""),
data_hash=data.get("data_hash", ""),
)
+89 -11
View File
@@ -16,8 +16,9 @@ class JobStatus(str, Enum):
"""
Job status enumeration - Complete state machine.
The workflow has 7 main stages with 2 human interaction points:
The workflow has 8 main stages with up to 3 human interaction points:
1. Input & Setup (may include audio source selection)
1b. Optional Audio Editing (BLOCKING) - trim/cut/mute/join input audio
2. Parallel Processing (audio + lyrics)
3. Title/End Screen Generation + Backing Vocals Analysis
4. Countdown Padding Synchronization
@@ -39,6 +40,11 @@ class JobStatus(str, Enum):
DOWNLOADING = "downloading" # Downloading from URL or processing upload
# Optional: Audio editing (before processing begins)
AWAITING_AUDIO_EDIT = "awaiting_audio_edit" # ⚠️ WAITING FOR USER - edit input audio
IN_AUDIO_EDIT = "in_audio_edit" # User is actively editing audio
AUDIO_EDIT_COMPLETE = "audio_edit_complete" # User submitted edited audio
# Stage 2a: Audio separation (parallel track 1)
SEPARATING_STAGE1 = "separating_stage1" # Clean instrumental separation (Modal API)
SEPARATING_STAGE2 = "separating_stage2" # Backing vocals separation (Modal API)
@@ -93,8 +99,9 @@ class JobStatus(str, Enum):
# Valid state transitions
STATE_TRANSITIONS = {
# PENDING can go to DOWNLOADING (file upload) or SEARCHING_AUDIO (artist+title search)
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
# PENDING can go to DOWNLOADING (file upload), SEARCHING_AUDIO (artist+title search),
# or DOWNLOADING_AUDIO (create-from-search where audio is pre-selected)
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
# Audio search flow (for artist+title search mode)
JobStatus.SEARCHING_AUDIO: [JobStatus.AWAITING_AUDIO_SELECTION, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED],
@@ -102,7 +109,13 @@ STATE_TRANSITIONS = {
JobStatus.DOWNLOADING_AUDIO: [JobStatus.DOWNLOADING, JobStatus.FAILED],
# DOWNLOADING allows parallel processing (audio + lyrics) and then screens when both complete
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
# Can also enter optional audio edit phase if requires_audio_edit flag is set
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.AWAITING_AUDIO_EDIT, JobStatus.FAILED],
# Optional audio edit phase (trim, cut, mute, join input audio before processing)
JobStatus.AWAITING_AUDIO_EDIT: [JobStatus.IN_AUDIO_EDIT, JobStatus.AUDIO_EDIT_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.IN_AUDIO_EDIT: [JobStatus.AWAITING_AUDIO_EDIT, JobStatus.AUDIO_EDIT_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.AUDIO_EDIT_COMPLETE: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
# Audio separation flow
JobStatus.SEPARATING_STAGE1: [JobStatus.SEPARATING_STAGE2, JobStatus.FAILED],
@@ -141,13 +154,14 @@ STATE_TRANSITIONS = {
JobStatus.UPLOADING: [JobStatus.NOTIFYING, JobStatus.COMPLETE, JobStatus.FAILED],
JobStatus.NOTIFYING: [JobStatus.COMPLETE, JobStatus.FAILED],
# Terminal states - COMPLETE, PREP_COMPLETE have no transitions
# Terminal states - COMPLETE allows edit (reopen review) and visibility change
# FAILED and CANCELLED allow retry transitions to resume from checkpoints
# PREP_COMPLETE allows continuation from combined review
JobStatus.COMPLETE: [],
JobStatus.COMPLETE: [JobStatus.AWAITING_REVIEW, JobStatus.LYRICS_COMPLETE], # Edit track or visibility change
JobStatus.PREP_COMPLETE: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED], # Continue from combined review
JobStatus.FAILED: [
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
JobStatus.DOWNLOADING_AUDIO, # Retry audio download (audio search flow)
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
JobStatus.REVIEW_COMPLETE, # Retry from render stage
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
@@ -155,6 +169,7 @@ STATE_TRANSITIONS = {
],
JobStatus.CANCELLED: [
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
JobStatus.DOWNLOADING_AUDIO, # Retry audio download (audio search flow)
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
JobStatus.REVIEW_COMPLETE, # Retry from render stage
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
@@ -176,6 +191,7 @@ class TimelineEvent(BaseModel):
timestamp: str
progress: Optional[int] = None
message: Optional[str] = None
metadata: Optional[Dict[str, Any]] = None
class LogEntry(BaseModel):
@@ -205,7 +221,14 @@ class Job(BaseModel):
title: Optional[str] = None
filename: Optional[str] = None # Original uploaded filename
input_media_gcs_path: Optional[str] = None # GCS path to uploaded file
# Audio source tracking (for retry capability)
audio_source_type: Optional[str] = None # Source type: 'audio_search', 'youtube_url', 'file_upload'
source_name: Optional[str] = None # Provider name (RED, OPS, YouTube, Spotify) - for audio_search
source_id: Optional[str] = None # Source-specific ID (torrent ID, video ID, track ID) - for audio_search
target_file: Optional[str] = None # For torrents, specific file to extract - for audio_search
download_url: Optional[str] = None # Direct download URL if available - for audio_search
# User preferences
enable_cdg: bool = False # Generate CDG+MP3 package (requires style config)
enable_txt: bool = False # Generate TXT+MP3 package (requires style config)
@@ -215,8 +238,8 @@ class Job(BaseModel):
user_email: Optional[str] = None # Email for notifications
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
# Multi-tenant support (None = default Nomad Karaoke)
tenant_id: Optional[str] = None # Tenant ID for white-label portal scoping
# Multi-tenant support ("" = consumer portal, "vocalstar"/etc = tenant portal)
tenant_id: str = "" # Tenant ID for white-label portal scoping
# Theme configuration (pre-made themes from GCS)
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
@@ -292,10 +315,25 @@ class Job(BaseModel):
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
customer_notes: Optional[str] = None # Notes provided by customer with their order
# Private (non-published) track mode
is_private: bool = False # Private tracks: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive
# Creation-time decision params (for observability — what did the user actually choose?)
creation_params: Dict[str, Any] = Field(default_factory=dict)
# Anti-abuse: IP address at job creation time (for cross-account correlation)
creation_ip: Optional[str] = None
# Output deletion tracking (for admin cleanup without deleting job)
outputs_deleted_at: Optional[datetime] = None # Timestamp when outputs were deleted by admin
outputs_deleted_by: Optional[str] = None # Admin email who deleted outputs
# Credit refund tracking (prevents double-refund on cancel→delete or fail→delete)
credit_refunded: bool = False
# Edit tracking (how many times a completed track has been re-edited)
edit_count: int = 0
# Processing state
track_output_dir: Optional[str] = None # Local output directory (temp)
audio_hash: Optional[str] = None # Hash for deduplication
@@ -386,6 +424,43 @@ class Job(BaseModel):
}
"""
# Processing metadata (immutable provenance data written by workers during processing)
processing_metadata: Dict[str, Any] = Field(default_factory=dict)
"""
Immutable processing provenance data. Written once by each worker during processing.
Never modified after initial write. Used for analytics, debugging, and data export.
Structure:
{
"transcription": {
"provider": "audioshake",
"audioshake_task_id": "...",
"audioshake_asset_id": "...",
"language_detected": "en",
"word_count": 358,
"segment_count": 42,
},
"correction": {
"handlers_applied": [...],
"corrections_made": 12,
"correction_ratio": 0.034,
"reference_sources_found": ["genius", "spotify"],
"agentic_routing": "rule-based",
"duration_seconds": 33.2,
},
"separation": {
"provider": "modal",
"clean_model": "...",
"duration_seconds": 420.5,
},
"timing": {
"lyrics_worker_seconds": 125.3,
"audio_worker_seconds": 420.5,
...
},
}
"""
# Request metadata (captured at job creation for tracking and filtering)
request_metadata: Dict[str, Any] = Field(default_factory=dict)
"""
@@ -499,6 +574,9 @@ class JobCreate(BaseModel):
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
customer_notes: Optional[str] = None # Notes provided by customer with their order
# Private (non-published) track mode
is_private: bool = False # Private tracks: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive
# Request metadata (set by API endpoint from request headers)
request_metadata: Dict[str, Any] = Field(default_factory=dict)
"""
@@ -511,8 +589,8 @@ class JobCreate(BaseModel):
- custom_headers: All X-* headers
"""
# Tenant scoping for white-label portals
tenant_id: Optional[str] = None # Tenant ID for job scoping
# Tenant scoping for white-label portals ("" = consumer, "vocalstar"/etc = tenant)
tenant_id: str = "" # Tenant ID for job scoping
@validator('url')
def validate_url(cls, v):
+178
View File
@@ -0,0 +1,178 @@
"""
Referral system models.
Supports:
- Referral link creation and management
- Earnings tracking from referred purchases
- Payout processing via Stripe Connect
- API request/response shapes for referral endpoints
"""
from datetime import datetime
from typing import Optional, List
from pydantic import BaseModel, Field
# ============================================================================
# Firestore Document Models
# ============================================================================
class ReferralLinkStats(BaseModel):
"""Aggregated statistics for a referral link."""
clicks: int = 0
signups: int = 0
purchases: int = 0
total_earned_cents: int = 0
class ReferralLink(BaseModel):
"""
Referral link stored in Firestore.
Each user can have one referral link. Vanity links use a custom code
(e.g., "karaoke-king") while standard links use a generated code.
"""
code: str # Unique referral code (URL slug)
owner_email: str # Email of the referrer
display_name: Optional[str] = None # Public name shown on interstitial
custom_message: Optional[str] = None # Optional message on interstitial
# Reward configuration
discount_percent: int = 10 # Discount for referred user
kickback_percent: int = 20 # Earnings for referrer
discount_duration_days: int = 30 # How long the discount lasts
earning_duration_days: int = 365 # How long referrer earns from this link
# Stripe
stripe_coupon_id: Optional[str] = None # Stripe coupon for this link
# Flags
is_vanity: bool = False # Custom vanity code vs generated
enabled: bool = True # Can be disabled without deleting
# Timestamps
created_at: datetime = Field(default_factory=datetime.utcnow)
updated_at: datetime = Field(default_factory=datetime.utcnow)
# Aggregated stats
stats: ReferralLinkStats = Field(default_factory=ReferralLinkStats)
class ReferralEarning(BaseModel):
"""
Record of a single earning from a referred purchase.
Created when a referred user completes a Stripe checkout.
"""
id: str
referrer_email: str
referred_email: str
referral_code: str
stripe_session_id: str
purchase_amount_cents: int
earning_amount_cents: int
status: str = "pending" # pending, paid, cancelled
created_at: datetime = Field(default_factory=datetime.utcnow)
paid_at: Optional[datetime] = None
payout_id: Optional[str] = None
class ReferralPayout(BaseModel):
"""
Record of a payout to a referrer via Stripe Connect.
Groups multiple earnings into a single transfer.
"""
id: str
referrer_email: str
stripe_transfer_id: str
amount_cents: int
earnings_included: List[str] # List of ReferralEarning IDs
status: str = "processing" # processing, completed, failed
created_at: datetime = Field(default_factory=datetime.utcnow)
# ============================================================================
# API Request Models
# ============================================================================
class CreateReferralLinkRequest(BaseModel):
"""Admin request to create a vanity referral link."""
vanity_code: Optional[str] = None
owner_email: str
display_name: Optional[str] = None
custom_message: Optional[str] = None
discount_percent: int = 10
kickback_percent: int = 20
discount_duration_days: int = 30
earning_duration_days: int = 365
class UpdateReferralLinkRequest(BaseModel):
"""Request to update a referral link (user: name/message only, admin: all fields)."""
display_name: Optional[str] = None
custom_message: Optional[str] = None
discount_percent: Optional[int] = None
kickback_percent: Optional[int] = None
discount_duration_days: Optional[int] = None
earning_duration_days: Optional[int] = None
enabled: Optional[bool] = None
# ============================================================================
# API Response Models
# ============================================================================
class ReferralLinkResponse(BaseModel):
"""Public referral link information."""
code: str
owner_email: str
display_name: Optional[str] = None
custom_message: Optional[str] = None
discount_percent: int
kickback_percent: int
is_vanity: bool
enabled: bool
stats: ReferralLinkStats
created_at: datetime
class ReferralEarningResponse(BaseModel):
"""Public earning information (for referrer dashboard)."""
id: str
referred_email: str
referral_code: str
purchase_amount_cents: int
earning_amount_cents: int
status: str
created_at: datetime
paid_at: Optional[datetime] = None
class ReferralPayoutResponse(BaseModel):
"""Public payout information (for referrer dashboard)."""
id: str
amount_cents: int
earnings_count: int
status: str
created_at: datetime
class ReferralDashboardResponse(BaseModel):
"""Full referral dashboard data for the referrer."""
referral_link: ReferralLinkResponse
recent_earnings: List[ReferralEarningResponse]
recent_payouts: List[ReferralPayoutResponse]
total_earned_cents: int
total_paid_cents: int
pending_balance_cents: int
stripe_connect_onboarded: bool
class ReferralInterstitialResponse(BaseModel):
"""Data for the referral landing/interstitial page."""
valid: bool = True
referral_code: str = ""
referrer_display_name: Optional[str] = None
discount_percent: int = 0
discount_duration_days: int = 0
custom_message: Optional[str] = None
+23 -1
View File
@@ -10,7 +10,7 @@ class URLSubmissionRequest(BaseModel):
url: HttpUrl
artist: Optional[str] = None # Auto-detected if not provided
title: Optional[str] = None # Auto-detected if not provided
# Optional preferences
enable_cdg: bool = False # Requires style config
enable_txt: bool = False # Requires style config
@@ -18,6 +18,7 @@ class URLSubmissionRequest(BaseModel):
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
is_private: Optional[bool] = None # Private (non-published) track mode
class UploadSubmissionRequest(BaseModel):
@@ -141,3 +142,24 @@ class RetryJobRequest(BaseModel):
"""Request to retry a failed job."""
from_stage: Optional[str] = None # Optional: restart from specific stage
class ChangeVisibilityRequest(BaseModel):
"""Request to change job visibility (public/private)."""
target_visibility: str # "public" or "private"
@validator('target_visibility')
def validate_target(cls, v):
if v not in ('public', 'private'):
raise ValueError("target_visibility must be 'public' or 'private'")
return v
class ChangeVisibilityResponse(BaseModel):
"""Response from change visibility endpoint."""
status: str # "success", "processing", "error"
job_id: str
message: str
previous_visibility: str # "public" or "private"
new_visibility: str # "public" or "private"
reprocessing_required: bool
+107
View File
@@ -0,0 +1,107 @@
"""
Review session model for persisting lyrics review progress.
Stored as a Firestore subcollection: jobs/{job_id}/review_sessions/{session_id}
Full correction data is stored in GCS to avoid Firestore's 1MB document limit.
"""
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Optional, Dict, Any, List
import uuid
@dataclass
class ReviewSessionSummary:
"""Compact summary of edits in a review session, for list views."""
total_segments: int = 0
total_words: int = 0
corrections_made: int = 0
changed_words: List[Dict[str, Any]] = field(default_factory=list)
def to_dict(self) -> Dict[str, Any]:
return {
"total_segments": self.total_segments,
"total_words": self.total_words,
"corrections_made": self.corrections_made,
"changed_words": self.changed_words,
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "ReviewSessionSummary":
return cls(
total_segments=data.get("total_segments", 0),
total_words=data.get("total_words", 0),
corrections_made=data.get("corrections_made", 0),
changed_words=data.get("changed_words", []),
)
@dataclass
class ReviewSession:
"""
A saved snapshot of a lyrics review editing session.
Stored at: jobs/{job_id}/review_sessions/{session_id}
Correction data stored at: jobs/{job_id}/review_sessions/{session_id}.json (GCS)
"""
session_id: str = field(default_factory=lambda: str(uuid.uuid4()))
job_id: str = ""
user_email: str = ""
created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
updated_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
edit_count: int = 0
trigger: str = "auto" # "auto" | "preview" | "manual"
audio_duration_seconds: Optional[float] = None
artist: Optional[str] = None
title: Optional[str] = None
summary: ReviewSessionSummary = field(default_factory=ReviewSessionSummary)
correction_data_gcs_path: str = ""
# Hash of the correction data JSON, used to skip duplicate saves
data_hash: str = ""
def to_dict(self) -> Dict[str, Any]:
return {
"session_id": self.session_id,
"job_id": self.job_id,
"user_email": self.user_email,
"created_at": self.created_at,
"updated_at": self.updated_at,
"edit_count": self.edit_count,
"trigger": self.trigger,
"audio_duration_seconds": self.audio_duration_seconds,
"artist": self.artist,
"title": self.title,
"summary": self.summary.to_dict(),
"correction_data_gcs_path": self.correction_data_gcs_path,
"data_hash": self.data_hash,
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "ReviewSession":
created_at = data.get("created_at")
if isinstance(created_at, str):
created_at = datetime.fromisoformat(created_at.rstrip("Z")).replace(tzinfo=timezone.utc)
updated_at = data.get("updated_at")
if isinstance(updated_at, str):
updated_at = datetime.fromisoformat(updated_at.rstrip("Z")).replace(tzinfo=timezone.utc)
summary_data = data.get("summary", {})
summary = ReviewSessionSummary.from_dict(summary_data) if isinstance(summary_data, dict) else ReviewSessionSummary()
return cls(
session_id=data.get("session_id", str(uuid.uuid4())),
job_id=data.get("job_id", ""),
user_email=data.get("user_email", ""),
created_at=created_at or datetime.now(timezone.utc),
updated_at=updated_at or datetime.now(timezone.utc),
edit_count=data.get("edit_count", 0),
trigger=data.get("trigger", "auto"),
audio_duration_seconds=data.get("audio_duration_seconds"),
artist=data.get("artist"),
title=data.get("title"),
summary=summary,
correction_data_gcs_path=data.get("correction_data_gcs_path", ""),
data_hash=data.get("data_hash", ""),
)
+10
View File
@@ -95,6 +95,12 @@ class TenantDefaults(BaseModel):
brand_prefix: Optional[str] = Field(
None, description="Prefix for output filenames (e.g., 'VSTAR')"
)
dropbox_path: Optional[str] = Field(
None, description="Tenant-specific Dropbox folder path (e.g., '/Karaoke/Vocal-Star')"
)
gdrive_folder_id: Optional[str] = Field(
None, description="Tenant-specific Google Drive folder ID for output uploads"
)
youtube_description_template: Optional[str] = Field(
None, description="Default YouTube description template"
)
@@ -139,6 +145,10 @@ class TenantConfig(BaseModel):
created_at: Optional[datetime] = None
updated_at: Optional[datetime] = None
def get_frontend_url(self) -> str:
"""Get the frontend URL for this tenant's portal."""
return f"https://{self.subdomain}"
def get_sender_email(self) -> str:
"""Get the email sender address for this tenant."""
if self.auth.sender_email:
+71 -50
View File
@@ -6,7 +6,6 @@ Supports:
- Credit system for karaoke generation
- Role-based access control (user/admin)
- Stripe integration for payments
- Beta tester program with feedback collection
"""
from datetime import datetime, timezone
from enum import Enum
@@ -20,14 +19,6 @@ class UserRole(str, Enum):
ADMIN = "admin"
class BetaTesterStatus(str, Enum):
"""Status of beta tester participation."""
ACTIVE = "active" # Enrolled, free credits available
PENDING_FEEDBACK = "pending_feedback" # Job completed, awaiting feedback
COMPLETED = "completed" # Feedback submitted
EXPIRED = "expired" # 24hr deadline passed without feedback
class CreditTransaction(BaseModel):
"""Record of a credit transaction."""
id: str
@@ -45,10 +36,15 @@ class PushSubscription(BaseModel):
Stores the push subscription endpoint and encryption keys needed
to send push notifications to the user's browser/device.
Subscriptions are scoped by tenant_id to prevent cross-tenant
notification leakage (e.g., a consumer job notification appearing
on a tenant portal's service worker).
"""
endpoint: str # Push service endpoint URL
keys: Dict[str, str] # p256dh and auth keys for encryption
device_name: Optional[str] = None # e.g., "iPhone", "Chrome on Windows"
tenant_id: Optional[str] = None # Tenant scope (None = consumer/default portal)
created_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
last_used_at: Optional[datetime] = None # Last time a notification was sent
@@ -89,21 +85,36 @@ class User(BaseModel):
total_jobs_created: int = 0
total_jobs_completed: int = 0
# Payment tracking (cached from stripe_payments, in cents)
total_spent: int = 0
# Optional profile fields for future use
display_name: Optional[str] = None
# Beta tester program
is_beta_tester: bool = False
beta_tester_status: Optional[BetaTesterStatus] = None
beta_enrolled_at: Optional[datetime] = None
beta_promise_text: Optional[str] = None # User's promise statement
beta_feedback_due_at: Optional[datetime] = None # 24hr after job completion
beta_feedback_email_sent: bool = False
# i18n locale preference (set from Accept-Language on login, used for emails/push)
locale: Optional[str] = None
# Push notification subscriptions (Web Push API)
# Users can subscribe from multiple devices/browsers
push_subscriptions: List[PushSubscription] = Field(default_factory=list)
# Feedback-for-credits program
has_submitted_feedback: bool = False
# Anti-abuse: IP address and device fingerprint used during signup
signup_ip: Optional[str] = None
device_fingerprint: Optional[str] = None
# Flag to prevent duplicate welcome credit grants (idempotency)
welcome_credits_granted: bool = False
# Referral system
referral_code: Optional[str] = None # User's own referral code
referred_by_code: Optional[str] = None # Code used when this user signed up
referred_at: Optional[datetime] = None # When the referral was applied
referral_discount_expires_at: Optional[datetime] = None # When referral discount ends
stripe_connect_account_id: Optional[str] = None # For receiving referral payouts
class MagicLinkToken(BaseModel):
"""
@@ -123,10 +134,19 @@ class MagicLinkToken(BaseModel):
used_at: Optional[datetime] = None
ip_address: Optional[str] = None
user_agent: Optional[str] = None
device_fingerprint: Optional[str] = None
# Multi-tenant support (None = default Nomad Karaoke)
tenant_id: Optional[str] = None
# Referral attribution (travels with magic link for cross-device support)
referral_code: Optional[str] = None
# Pre-computed credit evaluation (set asynchronously after magic link is sent)
credit_eval_decision: Optional[str] = None # "grant", "deny", "pending_review"
credit_eval_reasoning: Optional[str] = None
credit_eval_error: Optional[str] = None
class Session(BaseModel):
"""
@@ -144,6 +164,7 @@ class Session(BaseModel):
last_activity_at: datetime = Field(default_factory=datetime.utcnow)
ip_address: Optional[str] = None
user_agent: Optional[str] = None
device_fingerprint: Optional[str] = None
is_active: bool = True
# Multi-tenant support (None = default Nomad Karaoke)
@@ -155,6 +176,8 @@ class Session(BaseModel):
class SendMagicLinkRequest(BaseModel):
"""Request to send a magic link email."""
email: str
device_fingerprint: Optional[str] = None
referral_code: Optional[str] = None # From referral interstitial
class SendMagicLinkResponse(BaseModel):
@@ -174,6 +197,9 @@ class VerifyMagicLinkResponse(BaseModel):
session_token: str
user: "UserPublic"
message: str
tenant_subdomain: Optional[str] = None
credits_granted: int = 0
credit_status: str = "not_applicable" # "granted", "denied", "pending_review", "already_granted", "not_applicable"
class UserPublic(BaseModel):
@@ -185,6 +211,15 @@ class UserPublic(BaseModel):
total_jobs_created: int = 0
total_jobs_completed: int = 0
tenant_id: Optional[str] = None
feedback_eligible: bool = False
total_spent: int = 0
referral_code: Optional[str] = None
has_active_referral_discount: bool = False
referral_discount_percent: Optional[int] = None
referral_discount_expires_at: Optional[str] = None # ISO string
referred_by_code: Optional[str] = None
created_at: Optional[str] = None
last_login_at: Optional[str] = None
class AddCreditsRequest(BaseModel):
@@ -210,18 +245,18 @@ class UserListResponse(BaseModel):
# ============================================================================
# Beta Tester Program Models
# User Feedback-for-Credits Models
# ============================================================================
class BetaTesterFeedback(BaseModel):
class UserFeedback(BaseModel):
"""
Feedback from a beta tester after using the service.
Feedback from any user (not just beta testers).
Stored in Firestore 'beta_feedback' collection.
Stored in Firestore 'user_feedback' collection.
Users earn 1 free credit for submitting detailed feedback.
"""
id: str # Unique feedback ID
id: str
user_email: str
job_id: Optional[str] = None # The job they're providing feedback on
# Ratings (1-5 scale)
overall_rating: int = Field(ge=1, le=5)
@@ -240,33 +275,11 @@ class BetaTesterFeedback(BaseModel):
# Metadata
created_at: datetime = Field(default_factory=datetime.utcnow)
submitted_via: str = "web" # 'web' or 'email'
submitted_via: str = "web"
class BetaTesterEnrollRequest(BaseModel):
"""Request to enroll as a beta tester."""
email: str
promise_text: str = Field(
min_length=10,
description="User's promise to provide feedback (min 10 chars)"
)
accept_corrections_work: bool = Field(
description="User accepts they may need to review/correct lyrics"
)
class BetaTesterEnrollResponse(BaseModel):
"""Response after enrolling as beta tester."""
status: str
message: str
credits_granted: int
session_token: Optional[str] = None # If new user, provide session
class BetaFeedbackRequest(BaseModel):
"""Request to submit beta tester feedback."""
job_id: Optional[str] = None
class UserFeedbackRequest(BaseModel):
"""Request to submit user feedback for credits."""
# Ratings (1-5 scale)
overall_rating: int = Field(ge=1, le=5)
ease_of_use_rating: int = Field(ge=1, le=5)
@@ -283,8 +296,16 @@ class BetaFeedbackRequest(BaseModel):
would_use_again: bool = True
class BetaFeedbackResponse(BaseModel):
"""Response after submitting feedback."""
class UserFeedbackResponse(BaseModel):
"""Response after submitting user feedback."""
status: str
message: str
bonus_credits: int = 0 # Bonus credits for great feedback
credits_granted: int = 0
class FeedbackEligibilityResponse(BaseModel):
"""Response for feedback eligibility check."""
eligible: bool
has_submitted: bool
jobs_completed: int
credits_reward: int = 1
+48
View File
@@ -0,0 +1,48 @@
#!/usr/bin/env python3
"""
Download ensemble preset models at Docker build time.
Bakes models into the image so they're available immediately at runtime,
avoiding cold-start model download latency in Cloud Run Jobs.
Usage: python backend/scripts/download_models.py /models
"""
import json
import sys
import logging
import importlib.resources as resources
from audio_separator.separator import Separator
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
# Presets to bake into the GPU image
PRESETS_TO_DOWNLOAD = ["instrumental_clean", "karaoke"]
def download_preset_models(model_dir: str) -> None:
"""Download all models for the configured presets individually.
Ensemble presets resolve to multiple model filenames. We must download
each model individually via load_model(model_filename) — calling
load_model() with no args on an ensemble preset doesn't trigger downloads.
"""
with resources.open_text("audio_separator", "ensemble_presets.json") as f:
presets = json.load(f)["presets"]
models_to_download = set()
for preset_name in PRESETS_TO_DOWNLOAD:
models_to_download.update(presets[preset_name]["models"])
logger.info(f"Downloading {len(models_to_download)} models for presets: {PRESETS_TO_DOWNLOAD}")
for model in sorted(models_to_download):
logger.info(f" Downloading: {model}")
sep = Separator(model_file_dir=model_dir)
sep.load_model(model)
logger.info(f" Done: {model}")
if __name__ == "__main__":
model_dir = sys.argv[1] if len(sys.argv) > 1 else "/models"
download_preset_models(model_dir)
logger.info(f"All preset models downloaded to {model_dir}")
+265
View File
@@ -0,0 +1,265 @@
#!/usr/bin/env python3
"""
Generate a Mermaid state diagram from the job state machine.
This script reads the STATE_TRANSITIONS dictionary from models/job.py
and generates a Mermaid diagram that can be embedded in documentation.
Usage:
python scripts/generate_state_diagram.py > docs/state-machine.mmd
# Or to update ARCHITECTURE.md directly:
python scripts/generate_state_diagram.py --update-docs
Created as part of state machine robustness improvements (2026-02-02).
"""
import sys
import argparse
from pathlib import Path
# Add backend to path for imports
backend_dir = Path(__file__).parent.parent
sys.path.insert(0, str(backend_dir))
# Import the state machine data
# Note: Must be run from an environment where karaoke_gen is installed
try:
from models.job import STATE_TRANSITIONS, JobStatus
except ImportError:
# Fallback: define the state machine inline for documentation purposes
from enum import Enum
class JobStatus(str, Enum):
PENDING = "pending"
SEARCHING_AUDIO = "searching_audio"
AWAITING_AUDIO_SELECTION = "awaiting_audio_selection"
DOWNLOADING_AUDIO = "downloading_audio"
DOWNLOADING = "downloading"
SEPARATING_STAGE1 = "separating_stage1"
SEPARATING_STAGE2 = "separating_stage2"
AUDIO_COMPLETE = "audio_complete"
TRANSCRIBING = "transcribing"
CORRECTING = "correcting"
LYRICS_COMPLETE = "lyrics_complete"
GENERATING_SCREENS = "generating_screens"
APPLYING_PADDING = "applying_padding"
AWAITING_REVIEW = "awaiting_review"
IN_REVIEW = "in_review"
REVIEW_COMPLETE = "review_complete"
RENDERING_VIDEO = "rendering_video"
AWAITING_INSTRUMENTAL_SELECTION = "awaiting_instrumental_selection"
INSTRUMENTAL_SELECTED = "instrumental_selected"
GENERATING_VIDEO = "generating_video"
ENCODING = "encoding"
PACKAGING = "packaging"
UPLOADING = "uploading"
NOTIFYING = "notifying"
COMPLETE = "complete"
PREP_COMPLETE = "prep_complete"
FAILED = "failed"
CANCELLED = "cancelled"
QUEUED = "queued"
PROCESSING = "processing"
ERROR = "error"
READY_FOR_FINALIZATION = "ready_for_finalization"
FINALIZING = "finalizing"
STATE_TRANSITIONS = {
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.SEARCHING_AUDIO: [JobStatus.AWAITING_AUDIO_SELECTION, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED],
JobStatus.AWAITING_AUDIO_SELECTION: [JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.DOWNLOADING_AUDIO: [JobStatus.DOWNLOADING, JobStatus.FAILED],
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
JobStatus.SEPARATING_STAGE1: [JobStatus.SEPARATING_STAGE2, JobStatus.FAILED],
JobStatus.SEPARATING_STAGE2: [JobStatus.AUDIO_COMPLETE, JobStatus.FAILED],
JobStatus.AUDIO_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
JobStatus.TRANSCRIBING: [JobStatus.CORRECTING, JobStatus.FAILED],
JobStatus.CORRECTING: [JobStatus.LYRICS_COMPLETE, JobStatus.FAILED],
JobStatus.LYRICS_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
JobStatus.GENERATING_SCREENS: [JobStatus.APPLYING_PADDING, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
JobStatus.APPLYING_PADDING: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
JobStatus.AWAITING_REVIEW: [JobStatus.IN_REVIEW, JobStatus.REVIEW_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.IN_REVIEW: [JobStatus.REVIEW_COMPLETE, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
JobStatus.REVIEW_COMPLETE: [JobStatus.RENDERING_VIDEO, JobStatus.FAILED],
JobStatus.RENDERING_VIDEO: [JobStatus.INSTRUMENTAL_SELECTED, JobStatus.FAILED],
JobStatus.INSTRUMENTAL_SELECTED: [JobStatus.GENERATING_VIDEO, JobStatus.FAILED],
JobStatus.GENERATING_VIDEO: [JobStatus.ENCODING, JobStatus.FAILED],
JobStatus.ENCODING: [JobStatus.PACKAGING, JobStatus.COMPLETE, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
JobStatus.PACKAGING: [JobStatus.UPLOADING, JobStatus.COMPLETE, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
JobStatus.UPLOADING: [JobStatus.NOTIFYING, JobStatus.COMPLETE, JobStatus.FAILED],
JobStatus.NOTIFYING: [JobStatus.COMPLETE, JobStatus.FAILED],
}
def generate_mermaid() -> str:
"""
Generate a Mermaid state diagram from STATE_TRANSITIONS.
Returns:
Mermaid diagram as a string
"""
lines = ["stateDiagram-v2"]
lines.append(" direction TB")
lines.append("")
# Group states by category for better visualization
categories = {
"Initial": [JobStatus.PENDING],
"Audio Search": [
JobStatus.SEARCHING_AUDIO,
JobStatus.AWAITING_AUDIO_SELECTION,
JobStatus.DOWNLOADING_AUDIO,
],
"Processing": [
JobStatus.DOWNLOADING,
JobStatus.SEPARATING_STAGE1,
JobStatus.SEPARATING_STAGE2,
JobStatus.AUDIO_COMPLETE,
JobStatus.TRANSCRIBING,
JobStatus.CORRECTING,
JobStatus.LYRICS_COMPLETE,
],
"Screens": [
JobStatus.GENERATING_SCREENS,
JobStatus.APPLYING_PADDING,
],
"Review": [
JobStatus.AWAITING_REVIEW,
JobStatus.IN_REVIEW,
JobStatus.REVIEW_COMPLETE,
],
"Video": [
JobStatus.RENDERING_VIDEO,
JobStatus.INSTRUMENTAL_SELECTED,
JobStatus.GENERATING_VIDEO,
JobStatus.ENCODING,
JobStatus.PACKAGING,
],
"Distribution": [
JobStatus.UPLOADING,
JobStatus.NOTIFYING,
],
"Terminal": [
JobStatus.COMPLETE,
JobStatus.PREP_COMPLETE,
JobStatus.FAILED,
JobStatus.CANCELLED,
],
}
# Add state definitions with notes for key states
lines.append(" %% State definitions")
key_states = {
JobStatus.PENDING: "Job created",
JobStatus.AWAITING_AUDIO_SELECTION: "⚠️ USER ACTION",
JobStatus.AWAITING_REVIEW: "⚠️ USER ACTION",
JobStatus.COMPLETE: "✅ Success",
JobStatus.FAILED: "❌ Error",
}
for status, note in key_states.items():
lines.append(f" {status.value}: {status.value}")
lines.append(f" note right of {status.value}: {note}")
lines.append("")
lines.append(" %% Transitions")
# Generate transitions
for from_status, to_statuses in STATE_TRANSITIONS.items():
for to_status in to_statuses:
# Skip deprecated statuses
if to_status in [JobStatus.QUEUED, JobStatus.PROCESSING, JobStatus.ERROR,
JobStatus.READY_FOR_FINALIZATION, JobStatus.FINALIZING]:
continue
if from_status in [JobStatus.QUEUED, JobStatus.PROCESSING, JobStatus.ERROR,
JobStatus.READY_FOR_FINALIZATION, JobStatus.FINALIZING]:
continue
lines.append(f" {from_status.value} --> {to_status.value}")
# Add start and end markers
lines.append("")
lines.append(" %% Start and end")
lines.append(" [*] --> pending")
lines.append(" complete --> [*]")
lines.append(" prep_complete --> [*]")
lines.append(" failed --> [*]")
lines.append(" cancelled --> [*]")
return "\n".join(lines)
def generate_markdown_section() -> str:
"""
Generate a markdown section with the state diagram.
Returns:
Markdown string with the diagram
"""
diagram = generate_mermaid()
return f"""## Job State Machine
The following diagram shows all valid state transitions for a karaoke generation job.
Key states:
- **PENDING**: Initial state when job is created
- **AWAITING_AUDIO_SELECTION**: Human selects audio source (for artist+title search)
- **AWAITING_REVIEW**: Human reviews lyrics and selects instrumental
- **COMPLETE**: All processing finished successfully
```mermaid
{diagram}
```
### Parallel Processing
Jobs use parallel processing for audio separation and lyrics transcription:
1. **DOWNLOADING** triggers both:
- Audio worker → SEPARATING_STAGE1 → SEPARATING_STAGE2 → AUDIO_COMPLETE
- Lyrics worker → TRANSCRIBING → CORRECTING → LYRICS_COMPLETE
2. When BOTH complete, the screens worker is triggered automatically.
### Human Review Points
Jobs wait for human input at these points:
- **AWAITING_AUDIO_SELECTION**: User selects audio source from search results
- **AWAITING_REVIEW**: User reviews/corrects lyrics AND selects instrumental
### Error Handling
- Any state can transition to **FAILED** on unrecoverable errors
- **CANCELLED** is set when user cancels the job
"""
def main():
parser = argparse.ArgumentParser(description="Generate state machine diagram")
parser.add_argument(
"--update-docs",
action="store_true",
help="Update ARCHITECTURE.md with the diagram"
)
parser.add_argument(
"--markdown",
action="store_true",
help="Output markdown section instead of raw Mermaid"
)
args = parser.parse_args()
if args.update_docs:
# TODO: Implement auto-update of ARCHITECTURE.md
print("Auto-update of ARCHITECTURE.md not yet implemented")
print("Please copy the output below to the appropriate section:")
print()
print(generate_markdown_section())
elif args.markdown:
print(generate_markdown_section())
else:
print(generate_mermaid())
if __name__ == "__main__":
main()
+73 -1
View File
@@ -5,6 +5,7 @@ This service wraps the shared karaoke_gen.instrumental_review module
to provide GCS-integrated audio analysis capabilities.
"""
import json
import logging
import os
import tempfile
@@ -187,7 +188,78 @@ class AudioAnalysisService:
)
return amplitudes, duration
def cache_waveform_data(
self,
gcs_audio_path: str,
job_id: str,
cache_gcs_path: str,
num_points: int = 1000,
) -> tuple[list[float], float]:
"""
Generate waveform data and cache it as JSON in GCS.
Args:
gcs_audio_path: Path to the audio file in GCS
job_id: Job ID for logging
cache_gcs_path: GCS path to store the cached waveform JSON
num_points: Number of data points to generate
Returns:
Tuple of (amplitude_values, duration_seconds)
"""
logger.info(f"[{job_id}] Generating and caching waveform data: {gcs_audio_path}")
amplitudes, duration = self.get_waveform_data(
gcs_audio_path=gcs_audio_path,
job_id=job_id,
num_points=num_points,
)
cache_data = {
"amplitudes": list(amplitudes),
"duration_seconds": duration,
"num_points": num_points,
"source_gcs_path": gcs_audio_path,
}
self.storage_service.upload_json(cache_gcs_path, cache_data)
logger.info(f"[{job_id}] Waveform data cached at {cache_gcs_path}")
return amplitudes, duration
def load_cached_waveform(
self,
cache_gcs_path: str,
) -> Optional[tuple[list[float], float]]:
"""
Load cached waveform data from GCS.
Args:
cache_gcs_path: GCS path to the cached waveform JSON
Returns:
Tuple of (amplitude_values, duration_seconds) or None if not found
"""
try:
if not self.storage_service.file_exists(cache_gcs_path):
return None
with tempfile.TemporaryDirectory() as temp_dir:
local_path = os.path.join(temp_dir, "waveform.json")
self.storage_service.download_file(cache_gcs_path, local_path)
with open(local_path) as f:
data = json.load(f)
amplitudes = data.get("amplitudes", [])
duration = data.get("duration_seconds", 0.0)
if not amplitudes:
return None
return amplitudes, duration
except Exception as e:
logger.warning(f"Failed to load cached waveform from {cache_gcs_path}: {e}")
return None
def generate_waveform_with_mutes(
self,
gcs_audio_path: str,
+180
View File
@@ -0,0 +1,180 @@
"""
Audio editing service — server-side FFmpeg operations for trim, cut, mute, join.
All operations are lossless (FLAC in, FLAC out). Playback URLs use OGG Opus
via the existing AudioTranscodingService.
"""
import logging
import os
import json
import subprocess
import tempfile
from dataclasses import dataclass, asdict
from backend.services.storage_service import StorageService
logger = logging.getLogger(__name__)
@dataclass
class AudioMetadata:
duration_seconds: float
sample_rate: int
channels: int
format: str
file_size_bytes: int
class AudioEditService:
"""Server-side audio editing via FFmpeg."""
def __init__(self, storage_service: StorageService | None = None):
self.storage = storage_service or StorageService()
def get_metadata(self, audio_path: str) -> AudioMetadata:
"""Get audio metadata from a local file using ffprobe."""
cmd = [
"ffprobe", "-v", "quiet", "-print_format", "json",
"-show_format", "-show_streams", audio_path,
]
result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
if result.returncode != 0:
raise RuntimeError(f"ffprobe failed: {result.stderr}")
info = json.loads(result.stdout)
fmt = info.get("format", {})
streams = info.get("streams", [])
audio_stream = next((s for s in streams if s.get("codec_type") == "audio"), {})
return AudioMetadata(
duration_seconds=float(fmt.get("duration", 0)),
sample_rate=int(audio_stream.get("sample_rate", 44100)),
channels=int(audio_stream.get("channels", 2)),
format=fmt.get("format_name", "unknown"),
file_size_bytes=int(fmt.get("size", 0)),
)
def get_metadata_from_gcs(self, gcs_path: str) -> AudioMetadata:
"""Download from GCS and get metadata."""
with tempfile.TemporaryDirectory() as temp_dir:
local_path = os.path.join(temp_dir, "audio")
self.storage.download_file(gcs_path, local_path)
return self.get_metadata(local_path)
def trim_start(self, input_path: str, end_seconds: float, output_path: str) -> AudioMetadata:
"""Remove audio from 0 to end_seconds (skip the first N seconds)."""
self._run_ffmpeg([
"-ss", str(end_seconds),
"-i", input_path,
"-c:a", "flac",
output_path,
])
return self.get_metadata(output_path)
def trim_end(self, input_path: str, start_seconds: float, output_path: str) -> AudioMetadata:
"""Keep audio from 0 to start_seconds (remove from start_seconds to end)."""
self._run_ffmpeg([
"-i", input_path,
"-t", str(start_seconds),
"-c:a", "flac",
output_path,
])
return self.get_metadata(output_path)
def cut_region(self, input_path: str, start: float, end: float, output_path: str) -> AudioMetadata:
"""Remove a region from start to end, joining the remaining parts."""
self._run_ffmpeg([
"-i", input_path,
"-filter_complex",
f"[0]atrim=0:{start},asetpts=PTS-STARTPTS[a];"
f"[0]atrim={end},asetpts=PTS-STARTPTS[b];"
f"[a][b]concat=n=2:v=0:a=1[out]",
"-map", "[out]",
"-c:a", "flac",
output_path,
])
return self.get_metadata(output_path)
def mute_region(self, input_path: str, start: float, end: float, output_path: str) -> AudioMetadata:
"""Silence a region (preserve duration)."""
self._run_ffmpeg([
"-i", input_path,
"-af", f"volume=enable='between(t,{start},{end})':volume=0",
"-c:a", "flac",
output_path,
])
return self.get_metadata(output_path)
def join_audio(self, input_path: str, other_path: str, position: str, output_path: str) -> AudioMetadata:
"""Join two audio files. position: 'start' (prepend other) or 'end' (append other)."""
if position == "start":
first, second = other_path, input_path
else:
first, second = input_path, other_path
self._run_ffmpeg([
"-i", first,
"-i", second,
"-filter_complex", "[0:a][1:a]concat=n=2:v=0:a=1[out]",
"-map", "[out]",
"-c:a", "flac",
output_path,
])
return self.get_metadata(output_path)
def apply_edit(
self,
input_gcs_path: str,
operation: str,
params: dict,
output_gcs_path: str,
job_id: str,
) -> tuple[AudioMetadata, str]:
"""
Apply an edit operation to an audio file in GCS.
Returns (metadata, output_gcs_path).
"""
with tempfile.TemporaryDirectory() as temp_dir:
local_input = os.path.join(temp_dir, "input.flac")
local_output = os.path.join(temp_dir, "output.flac")
# Download current audio
self.storage.download_file(input_gcs_path, local_input)
# Apply the operation
if operation == "trim_start":
metadata = self.trim_start(local_input, params["end_seconds"], local_output)
elif operation == "trim_end":
metadata = self.trim_end(local_input, params["start_seconds"], local_output)
elif operation == "cut":
metadata = self.cut_region(
local_input, params["start_seconds"], params["end_seconds"], local_output
)
elif operation == "mute":
metadata = self.mute_region(
local_input, params["start_seconds"], params["end_seconds"], local_output
)
elif operation in ("join_start", "join_end"):
# Download the upload file
upload_gcs_path = params["upload_gcs_path"]
local_other = os.path.join(temp_dir, "other.flac")
self.storage.download_file(upload_gcs_path, local_other)
position = "start" if operation == "join_start" else "end"
metadata = self.join_audio(local_input, local_other, position, local_output)
else:
raise ValueError(f"Unknown operation: {operation}")
# Upload edited audio
self.storage.upload_file(local_output, output_gcs_path)
logger.info(f"[{job_id}] Applied {operation}, uploaded to {output_gcs_path}")
return metadata, output_gcs_path
def _run_ffmpeg(self, args: list[str]) -> None:
"""Run an ffmpeg command with standard options."""
cmd = ["ffmpeg", "-hide_banner", "-loglevel", "error", "-y"] + args
result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
if result.returncode != 0:
raise RuntimeError(f"ffmpeg failed (exit {result.returncode}): {result.stderr}")
+12 -9
View File
@@ -612,13 +612,13 @@ class AudioSearchService:
async def search_async(self, artist: str, title: str) -> List[AudioSearchResult]:
"""
Async version of search for use in async routes.
Note: Currently wraps sync search in executor. Future optimization
could make this fully async.
Runs the sync search in a thread executor to avoid blocking
the event loop.
"""
loop = asyncio.get_event_loop()
loop = asyncio.get_running_loop()
return await loop.run_in_executor(None, self.search, artist, title)
async def download_async(
self,
result_index: int,
@@ -630,10 +630,10 @@ class AudioSearchService:
"""
Async version of download for use in async routes.
Note: Currently wraps sync download in executor. Future optimization
could make this fully async.
Runs the sync download in a thread executor to avoid blocking
the event loop.
"""
loop = asyncio.get_event_loop()
loop = asyncio.get_running_loop()
return await loop.run_in_executor(
None,
lambda: self.download(result_index, output_dir, output_filename, gcs_path, remote_search_id)
@@ -651,8 +651,11 @@ class AudioSearchService:
) -> AudioDownloadResult:
"""
Async version of download_by_id for use in async routes.
Runs the sync download_by_id in a thread executor to avoid blocking
the event loop.
"""
loop = asyncio.get_event_loop()
loop = asyncio.get_running_loop()
return await loop.run_in_executor(
None,
lambda: self.download_by_id(
@@ -0,0 +1,181 @@
"""
Audio transcoding service for review UI optimization.
Transcodes FLAC audio files to OGG Opus for efficient browser playback.
Original FLAC files are never modified — transcoded versions are cached
in GCS at jobs/{job_id}/review-audio/{filename}.ogg.
Eager transcoding: called from screens_worker before AWAITING_REVIEW
so all files are cached before the user opens the review UI.
Falls back to original FLAC signed URLs if transcoded version is missing.
"""
import asyncio
import hashlib
import logging
import os
import subprocess
import tempfile
from pathlib import Path
from typing import Optional
from backend.services.storage_service import StorageService
logger = logging.getLogger(__name__)
class AudioTranscodingService:
"""
Transcodes audio files to OGG Opus and caches them in GCS.
Used by the review UI to serve ~3 MB OGG files instead of 35+ MB FLACs.
"""
def __init__(self, storage_service: Optional[StorageService] = None):
self.storage = storage_service or StorageService()
def _get_cache_path(self, source_gcs_path: str) -> str:
"""
Derive the cache path for a transcoded file.
Examples:
jobs/abc123/input/song.flac -> jobs/abc123/review-audio/song.ogg
jobs/abc123/stems/instrumental_clean.flac -> jobs/abc123/review-audio/instrumental_clean.ogg
"""
parts = Path(source_gcs_path).parts
# Find "jobs/{job_id}" prefix
try:
jobs_idx = list(parts).index("jobs")
job_id = parts[jobs_idx + 1]
except (ValueError, IndexError):
# Fallback: use a hash of the full path to avoid collisions
job_id = "_" + hashlib.sha256(source_gcs_path.encode()).hexdigest()[:12]
filename = Path(source_gcs_path).stem + ".ogg"
return f"jobs/{job_id}/review-audio/{filename}"
def _transcode_and_upload(self, source_gcs_path: str, cache_path: str) -> str:
"""
Download FLAC from GCS, transcode to OGG Opus, upload to cache path.
Returns the cache GCS path on success.
Raises on failure.
"""
with tempfile.TemporaryDirectory() as temp_dir:
local_input = os.path.join(temp_dir, "input.flac")
local_output = os.path.join(temp_dir, "output.ogg")
# Download source
self.storage.download_file(source_gcs_path, local_input)
# Transcode: FLAC -> OGG Opus 128kbps
cmd = [
"ffmpeg", "-hide_banner", "-loglevel", "error",
"-y",
"-i", local_input,
"-c:a", "libopus",
"-b:a", "128k",
"-vn",
local_output,
]
result = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
if result.returncode != 0:
raise RuntimeError(
f"ffmpeg transcoding failed (exit {result.returncode}): {result.stderr}"
)
# Upload transcoded file
self.storage.upload_file(local_output, cache_path)
return cache_path
def transcode_if_needed(self, source_gcs_path: str) -> str:
"""
Return the GCS path of the transcoded OGG file, transcoding if not cached.
Returns the cache path (always an OGG file in review-audio/).
Raises if transcoding fails.
"""
cache_path = self._get_cache_path(source_gcs_path)
if self.storage.file_exists(cache_path):
logger.debug(f"Cache hit: {cache_path}")
return cache_path
logger.info(f"Cache miss, transcoding: {source_gcs_path} -> {cache_path}")
return self._transcode_and_upload(source_gcs_path, cache_path)
def get_review_audio_url(
self, source_gcs_path: str, expiration_minutes: int = 120
) -> str:
"""
Get a signed URL for the transcoded audio, falling back to original FLAC.
"""
try:
cache_path = self.transcode_if_needed(source_gcs_path)
return self.storage.generate_signed_url(cache_path, expiration_minutes)
except Exception as e:
# Source may be deleted but cache from earlier transcode persists
try:
cache_path = self._get_cache_path(source_gcs_path)
if self.storage.file_exists(cache_path):
logger.info(f"Source gone but cache exists, serving: {cache_path}")
return self.storage.generate_signed_url(cache_path, expiration_minutes)
except Exception:
pass # Fall through to FLAC fallback below
logger.warning(
f"Transcoding failed for {source_gcs_path}, falling back to FLAC: {e}"
)
return self.storage.generate_signed_url(source_gcs_path, expiration_minutes)
async def get_review_audio_url_async(
self, source_gcs_path: str, expiration_minutes: int = 120
) -> str:
"""Async wrapper around get_review_audio_url via asyncio.to_thread."""
return await asyncio.to_thread(
self.get_review_audio_url, source_gcs_path, expiration_minutes
)
def prepare_review_audio_for_job(self, job) -> list[str]:
"""
Transcode all review audio files for a job (eager transcoding).
Called from screens_worker before AWAITING_REVIEW transition
and from the admin backfill endpoint.
Returns list of cache paths that were transcoded.
"""
job_id = job.job_id
transcoded = []
# Collect all audio paths to transcode
paths_to_transcode = []
# Main input audio
if job.input_media_gcs_path:
paths_to_transcode.append(("input", job.input_media_gcs_path))
# Stems
stems = job.file_urls.get("stems", {})
for stem_key in ["instrumental_clean", "instrumental_with_backing", "backing_vocals"]:
path = stems.get(stem_key)
if path:
paths_to_transcode.append((stem_key, path))
for label, source_path in paths_to_transcode:
try:
cache_path = self.transcode_if_needed(source_path)
transcoded.append(cache_path)
logger.info(f"[{job_id}] Transcoded {label}: {cache_path}")
except Exception as e:
# Non-fatal: review UI will fall back to FLAC
logger.warning(
f"[{job_id}] Failed to transcode {label} ({source_path}): {e}"
)
logger.info(
f"[{job_id}] Review audio preparation complete: "
f"{len(transcoded)}/{len(paths_to_transcode)} files transcoded"
)
return transcoded
+92 -38
View File
@@ -3,6 +3,7 @@ Authentication service for karaoke backend.
Provides token-based authentication with support for:
- Admin tokens (hardcoded in environment)
- Google OIDC tokens (from Cloud Scheduler / Cloud Tasks service accounts)
- Admin by email domain (@nomadkaraoke.com)
- User tokens (stored in Firestore auth_tokens collection)
- Session tokens (stored in Firestore sessions collection, from magic link auth)
@@ -19,6 +20,9 @@ from enum import Enum
from typing import Optional, Tuple, Dict, Any, List
from datetime import datetime
from google.auth.transport import requests as google_auth_requests
from google.oauth2 import id_token as google_id_token
from backend.services.firestore_service import FirestoreService
from backend.config import get_settings
from backend.models.user import UserRole
@@ -49,6 +53,7 @@ class AuthResult:
user_email: Optional[str] = None # Email if authenticated via session/API key
is_admin: bool = False # True if admin token or admin email domain
api_key_id: Optional[str] = None # API key ID if authenticated via API key
tenant_id: Optional[str] = None # Tenant scope from token or session
def __iter__(self):
"""Allow unpacking as tuple for backward compatibility."""
@@ -71,6 +76,7 @@ class AuthService:
self.firestore = FirestoreService()
self.settings = get_settings()
self._init_admin_tokens()
self._scheduler_service_account = self.settings.scheduler_service_account
def _init_admin_tokens(self):
"""Load admin tokens from environment variables."""
@@ -82,6 +88,56 @@ class AuthService:
logger.info(f"Loaded {len(self.admin_tokens)} admin token(s)")
else:
logger.warning("No admin tokens configured! Set ADMIN_TOKENS environment variable.")
def _validate_oidc_token(self, token: str) -> Optional[AuthResult]:
"""
Validate a Google OIDC token (from Cloud Scheduler or Cloud Tasks).
Returns AuthResult if token is a valid Google OIDC JWT from an allowed
service account, None if the token is not a JWT or validation fails.
"""
# Quick check: JWTs have 3 dot-separated parts
if token.count('.') != 2:
return None
try:
# Verify the token signature and claims with Google
# audience=None skips audience check — Cloud Scheduler sets audience
# to the endpoint URL, which varies per endpoint
claims = google_id_token.verify_oauth2_token(
token,
google_auth_requests.Request(),
audience=None,
)
token_email = claims.get("email", "")
# Only accept tokens from the configured service account
if token_email != self._scheduler_service_account:
logger.warning(
f"OIDC token from unexpected service account: {token_email} "
f"(expected {self._scheduler_service_account})"
)
return None
logger.info(f"OIDC token validated for service account: {token_email}")
return AuthResult(
is_valid=True,
user_type=UserType.ADMIN,
remaining_uses=-1,
message="OIDC service account access granted",
user_email=token_email,
is_admin=True,
)
except ValueError as e:
# verify_oauth2_token raises ValueError for invalid/expired tokens
logger.debug(f"OIDC token validation failed: {e}")
return None
except Exception as e:
# Network errors fetching Google certs, etc.
logger.warning(f"OIDC token verification error: {e}")
return None
def validate_token(self, token: str) -> Tuple[bool, UserType, int, str]:
"""
@@ -212,6 +268,12 @@ class AuthService:
is_admin=True
)
# Check for Google OIDC tokens (Cloud Scheduler / Cloud Tasks)
# Done before Firestore lookup to avoid wasting a read on JWT strings
oidc_result = self._validate_oidc_token(token)
if oidc_result is not None:
return oidc_result
# Check stored tokens in Firestore (auth_tokens collection)
token_data = self.firestore.get_token(token)
@@ -284,6 +346,7 @@ class AuthService:
# All auth_tokens must have an associated user_email for job ownership
token_user_email = token_data.get("user_email")
api_key_id = token_data.get("api_key_id")
token_tenant_id = token_data.get("tenant_id")
# Require user_email on all auth_tokens (no anonymous token auth)
if not token_user_email:
@@ -307,7 +370,8 @@ class AuthService:
message="Unlimited access granted",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id
api_key_id=api_key_id,
tenant_id=token_tenant_id
)
# LIMITED tokens: check usage count
@@ -320,7 +384,8 @@ class AuthService:
message="Limited token with unlimited uses",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id
api_key_id=api_key_id,
tenant_id=token_tenant_id
)
current_uses = token_data.get("usage_count", 0)
@@ -333,7 +398,8 @@ class AuthService:
remaining_uses=0,
message="Token usage limit exceeded",
user_email=token_user_email,
api_key_id=api_key_id
api_key_id=api_key_id,
tenant_id=token_tenant_id
)
return AuthResult(
@@ -343,7 +409,8 @@ class AuthService:
message=f"Limited token: {remaining} uses remaining",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id
api_key_id=api_key_id,
tenant_id=token_tenant_id
)
# STRIPE tokens: check expiration and usage
@@ -357,7 +424,8 @@ class AuthService:
remaining_uses=0,
message="Token has expired",
user_email=token_user_email,
api_key_id=api_key_id
api_key_id=api_key_id,
tenant_id=token_tenant_id
)
if max_uses > 0:
@@ -371,7 +439,8 @@ class AuthService:
remaining_uses=0,
message="Token usage limit exceeded",
user_email=token_user_email,
api_key_id=api_key_id
api_key_id=api_key_id,
tenant_id=token_tenant_id
)
return AuthResult(
@@ -381,7 +450,8 @@ class AuthService:
message=f"Stripe token: {remaining} uses remaining",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id
api_key_id=api_key_id,
tenant_id=token_tenant_id
)
return AuthResult(
@@ -391,7 +461,8 @@ class AuthService:
message="Stripe access granted",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id
api_key_id=api_key_id,
tenant_id=token_tenant_id
)
# API_KEY tokens
@@ -406,7 +477,8 @@ class AuthService:
remaining_uses=0,
message="API key usage limit exceeded",
user_email=token_user_email,
api_key_id=api_key_id
api_key_id=api_key_id,
tenant_id=token_tenant_id
)
return AuthResult(
is_valid=True,
@@ -415,7 +487,8 @@ class AuthService:
message=f"API key valid: {remaining} uses remaining",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id
api_key_id=api_key_id,
tenant_id=token_tenant_id
)
return AuthResult(
@@ -425,7 +498,8 @@ class AuthService:
message="API key access granted",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id
api_key_id=api_key_id,
tenant_id=token_tenant_id
)
return AuthResult(
@@ -437,10 +511,11 @@ class AuthService:
def increment_token_usage(self, token: str, job_id: str) -> bool:
"""
Increment usage count for a token and track the job.
Increment usage count for an auth_token and track the job.
For session tokens (magic link auth), this deducts a credit from the user.
For auth_tokens, this increments the usage count.
Note: Credit deduction for session-based users is handled in
JobManager.create_job(), not here. This method only tracks usage
for legacy auth_tokens stored in Firestore.
Args:
token: The access token
@@ -465,32 +540,11 @@ class AuthService:
if token in self.admin_tokens:
return True
# Check if this is a session token (STRIPE type from validate_token means it's a session)
# Session tokens are not in auth_tokens collection
# Only increment usage for auth_tokens in Firestore
# Session tokens (magic link auth) have credits managed by JobManager.create_job()
token_data = self.firestore.get_token(token)
if not token_data:
# This is a session token - deduct credit from user
from backend.services.user_service import get_user_service
user_service = get_user_service()
# Get the user email from the session
is_valid, user, _ = user_service.validate_session(token)
if not is_valid or not user:
logger.error(f"Session token validation failed during usage increment")
return False
# Deduct one credit
success, new_balance, deduct_message = user_service.deduct_credit(
user.email, job_id, reason="job_creation"
)
if success:
logger.info(f"Deducted credit for user {user.email} (remaining: {new_balance})")
return True
else:
logger.error(f"Failed to deduct credit for user {user.email}: {deduct_message}")
return False
return True
# Regular auth_token - increment usage in Firestore
try:
+225
View File
@@ -0,0 +1,225 @@
"""
Brand code allocation service using Firestore atomic counters.
Replaces the previous Dropbox folder-scanning approach which was susceptible to
TOCTOU race conditions when concurrent jobs read the same folder state.
Uses Firestore transactions to atomically allocate unique brand codes.
Supports recycling numbers when E2E cleanup deletes test brand codes.
"""
import logging
import re
from datetime import datetime, timezone
from typing import Optional
from google.cloud import firestore
from backend.config import settings
logger = logging.getLogger(__name__)
BRAND_CODE_COUNTERS_COLLECTION = "brand_code_counters"
# Singleton instance
_brand_code_service: Optional["BrandCodeService"] = None
class BrandCodeService:
"""Atomic brand code allocation using Firestore counters."""
def __init__(self, db: Optional[firestore.Client] = None):
if db is None:
self.db = firestore.Client(project=settings.google_cloud_project)
else:
self.db = db
def allocate_brand_code(self, prefix: str, dropbox_path: str) -> str:
"""
Atomically allocate the next brand code for a given prefix.
Uses a Firestore transaction to ensure no two concurrent callers
receive the same number. If recycled numbers are available (from
E2E cleanup), the smallest recycled number is reused first.
On first use for a prefix, initializes the counter by scanning
Dropbox to find the current max number.
Args:
prefix: Brand prefix (e.g., "NOMAD" or "NOMADNP")
dropbox_path: Dropbox path for one-time initialization scan
Returns:
Brand code string (e.g., "NOMAD-1264")
"""
doc_ref = self.db.collection(BRAND_CODE_COUNTERS_COLLECTION).document(prefix)
@firestore.transactional
def allocate_in_transaction(transaction):
doc = doc_ref.get(transaction=transaction)
if not doc.exists:
# First use — initialize from Dropbox scan
next_number = self._get_initial_next_number(prefix, dropbox_path)
allocated = next_number
transaction.set(doc_ref, {
"prefix": prefix,
"next_number": next_number + 1,
"recycled": [],
"initialized_at": datetime.now(timezone.utc),
"last_updated": datetime.now(timezone.utc),
})
return allocated
data = doc.to_dict()
recycled = data.get("recycled", [])
if recycled:
# Reuse smallest recycled number
recycled.sort()
allocated = recycled.pop(0)
transaction.update(doc_ref, {
"recycled": recycled,
"last_updated": datetime.now(timezone.utc),
})
else:
# Allocate next sequential number
allocated = data["next_number"]
transaction.update(doc_ref, {
"next_number": allocated + 1,
"last_updated": datetime.now(timezone.utc),
})
return allocated
transaction = self.db.transaction()
number = allocate_in_transaction(transaction)
brand_code = f"{prefix}-{number:04d}"
logger.info(f"Allocated brand code: {brand_code}")
return brand_code
def recycle_brand_code(self, prefix: str, number: int) -> None:
"""
Return a brand code number to the recycled pool.
Called when E2E cleanup deletes a brand code's distribution.
The number will be reused on the next allocation for this prefix.
Args:
prefix: Brand prefix (e.g., "NOMAD")
number: The numeric portion to recycle (e.g., 1234)
"""
doc_ref = self.db.collection(BRAND_CODE_COUNTERS_COLLECTION).document(prefix)
@firestore.transactional
def recycle_in_transaction(transaction):
doc = doc_ref.get(transaction=transaction)
if not doc.exists:
# Counter doesn't exist yet — nothing to recycle into.
# The number will naturally be below next_number when
# the counter is initialized, so it would create a gap.
# Just log and skip.
logger.warning(
f"Cannot recycle {prefix}-{number:04d}: counter doc doesn't exist"
)
return
data = doc.to_dict()
next_number = data.get("next_number", 0)
if number <= 0 or number >= next_number:
logger.warning(
f"Cannot recycle {prefix}-{number:04d}: "
f"invalid (next_number={next_number})"
)
return
recycled = data.get("recycled", [])
if number not in recycled:
recycled.append(number)
transaction.update(doc_ref, {
"recycled": recycled,
"last_updated": datetime.now(timezone.utc),
})
logger.info(f"Recycled brand code number: {prefix}-{number:04d}")
else:
logger.warning(f"Number {number} already in recycled pool for {prefix}")
transaction = self.db.transaction()
recycle_in_transaction(transaction)
def _get_initial_next_number(self, prefix: str, dropbox_path: str) -> int:
"""
Scan Dropbox once to determine the starting next_number.
This is only called when the counter document doesn't exist yet
(one-time migration from the old Dropbox-scanning approach).
Args:
prefix: Brand prefix to scan for
dropbox_path: Dropbox path containing organized folders
Returns:
The next number to allocate (max existing + 1, minimum 1)
"""
try:
from backend.services.dropbox_service import get_dropbox_service
dropbox = get_dropbox_service()
if not dropbox.is_configured:
logger.warning(
f"Dropbox not configured for initialization of {prefix} counter, starting at 1"
)
return 1
folders = dropbox.list_folders(dropbox_path)
pattern = re.compile(rf"^{re.escape(prefix)}-(\d{{4}})")
existing_nums: set[int] = set()
for folder in folders:
match = pattern.match(folder)
if match:
existing_nums.add(int(match.group(1)))
if existing_nums:
next_num = max(existing_nums) + 1
logger.info(
f"Initialized {prefix} counter from Dropbox: "
f"max existing={max(existing_nums)}, next_number={next_num}"
)
return next_num
logger.info(f"No existing {prefix} codes found in Dropbox, starting at 1")
return 1
except Exception as e:
logger.error(f"Failed to initialize {prefix} counter from Dropbox: {e}")
raise
@staticmethod
def parse_brand_code(brand_code: str) -> tuple[str, int]:
"""
Parse a brand code string into prefix and number.
Args:
brand_code: e.g., "NOMAD-1234" or "NOMADNP-0012"
Returns:
Tuple of (prefix, number), e.g., ("NOMAD", 1234)
Raises:
ValueError: If brand_code doesn't match expected format
"""
match = re.match(r"^([A-Z]+)-(\d+)$", brand_code)
if not match:
raise ValueError(f"Invalid brand code format: {brand_code}")
return match.group(1), int(match.group(2))
def get_brand_code_service() -> BrandCodeService:
"""Get or create the singleton BrandCodeService instance."""
global _brand_code_service
if _brand_code_service is None:
_brand_code_service = BrandCodeService()
return _brand_code_service
+105
View File
@@ -0,0 +1,105 @@
"""
Proxy service for karaoke-decide catalog API.
Forwards artist and track search requests to the karaoke-decide backend,
providing autocomplete data from MusicBrainz + Spotify catalogs.
"""
import logging
import time
from typing import Any
import httpx
from backend.config import settings
logger = logging.getLogger(__name__)
# In-memory TTL cache: { cache_key: (expiry_timestamp, data) }
_cache: dict[str, tuple[float, Any]] = {}
CACHE_TTL_SECONDS = 300 # 5 minutes
KARAOKE_DECIDE_BASE_URL = "https://decide.nomadkaraoke.com"
REQUEST_TIMEOUT = 10 # seconds
def _get_base_url() -> str:
"""Get the karaoke-decide API base URL from config or default."""
return getattr(settings, "karaoke_decide_api_url", None) or KARAOKE_DECIDE_BASE_URL
def _cache_get(key: str) -> Any | None:
"""Get a value from cache if it exists and hasn't expired."""
if key in _cache:
expiry, data = _cache[key]
if time.monotonic() < expiry:
return data
del _cache[key]
return None
def _cache_set(key: str, data: Any, ttl: float = CACHE_TTL_SECONDS) -> None:
"""Store a value in cache with TTL."""
_cache[key] = (time.monotonic() + ttl, data)
async def search_artists(query: str, limit: int = 10) -> list[dict]:
"""
Search for artists via karaoke-decide catalog API.
Returns list of artist dicts with canonical MusicBrainz names.
Falls back to empty list on any error.
"""
cache_key = f"artists:{query.lower()}:{limit}"
cached = _cache_get(cache_key)
if cached is not None:
return cached
base_url = _get_base_url()
url = f"{base_url}/api/catalog/artists"
params = {"q": query, "limit": limit}
try:
async with httpx.AsyncClient(timeout=REQUEST_TIMEOUT) as client:
resp = await client.get(url, params=params)
resp.raise_for_status()
data = resp.json()
# karaoke-decide wraps results: {"artists": [...], "total": N}
results = data.get("artists", data) if isinstance(data, dict) else data
_cache_set(cache_key, results)
return results
except Exception as e:
logger.warning(f"Failed to search artists from karaoke-decide: {e}")
return []
async def search_tracks(query: str, artist: str | None = None, limit: int = 10) -> list[dict]:
"""
Search for tracks via karaoke-decide catalog API.
Returns list of track dicts with canonical Spotify/MusicBrainz names.
Falls back to empty list on any error.
"""
cache_key = f"tracks:{query.lower()}:{(artist or '').lower()}:{limit}"
cached = _cache_get(cache_key)
if cached is not None:
return cached
base_url = _get_base_url()
url = f"{base_url}/api/catalog/tracks"
params: dict[str, Any] = {"q": query, "limit": limit}
if artist:
params["artist"] = artist
try:
async with httpx.AsyncClient(timeout=REQUEST_TIMEOUT) as client:
resp = await client.get(url, params=params)
resp.raise_for_status()
data = resp.json()
# karaoke-decide wraps results: {"tracks": [...], "total": N}
results = data.get("tracks", data) if isinstance(data, dict) else data
_cache_set(cache_key, results)
return results
except Exception as e:
logger.warning(f"Failed to search tracks from karaoke-decide: {e}")
return []
@@ -0,0 +1,441 @@
"""
AI-powered credit evaluation service.
Evaluates whether to grant free credits (welcome or feedback) by collecting
abuse signals and asking Gemini for a grant/deny decision. Fail-open: if
the AI call fails for any reason, credits are granted.
"""
import json
import logging
import uuid
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Optional
from backend.config import get_settings
logger = logging.getLogger(__name__)
CREDIT_EVALUATIONS_COLLECTION = "credit_evaluations"
@dataclass
class CreditEvaluation:
"""Result of an AI credit evaluation."""
decision: str # "grant" or "deny"
reasoning: str
confidence: float
error: Optional[str] = None # Set if evaluation failed (decision defaults to "grant")
signals: Optional[dict] = None # Collected abuse signals (for admin review emails)
SYSTEM_PROMPT = """You are an anti-abuse evaluator for Nomad Karaoke, a web service that creates professional karaoke videos. Each job costs the service real money in API credits and cloud compute.
New users get 1 free welcome credit to try the service. Users who complete 2 jobs and submit feedback get 1 more credit. Some users abuse this by creating multiple accounts to get unlimited free karaoke generation.
Your job: Given the signals below, decide whether to GRANT or DENY free credits to this user.
## Decision Guidelines
**DENY if any of these are true:**
- The user's device fingerprint matches other accounts (strong signal of multi-accounting)
- The user's signup IP matches other accounts AND those accounts also have suspicious patterns (e.g., no spend, same user agent)
- There's a clear pattern of account creation from the same device/network to farm free credits
**GRANT if:**
- This appears to be a genuine new user with no suspicious correlations
- The IP match is explainable (e.g., different fingerprints suggest different actual users on a shared network like a school or office)
- There's insufficient evidence to confidently flag abuse
**When in doubt, GRANT.** False positives (blocking real users) are worse than false negatives (letting an occasional abuser through). We'd rather lose a few dollars than alienate a real customer.
## Response Format
Respond with ONLY a JSON object, no other text:
{"decision": "grant" or "deny", "reasoning": "Brief explanation", "confidence": 0.0 to 1.0}
"""
def _build_user_prompt(
email: str,
grant_type: str,
user_data: dict,
fingerprint_matches: list,
ip_matches: list,
ip_geo: Optional[dict],
recent_signups_ip: int,
recent_signups_fp: int,
user_agent: Optional[str],
feedback_content: Optional[dict] = None,
) -> str:
"""Build the user prompt with all abuse signals."""
lines = [
f"## Credit Grant Request",
f"- **Email:** {email}",
f"- **Grant type:** {grant_type} credits",
f"- **User's device fingerprint:** {user_data.get('device_fingerprint', 'unknown')}",
f"- **User's signup IP:** {user_data.get('signup_ip', 'unknown')}",
f"- **User agent:** {user_agent or 'unknown'}",
f"- **Account created:** {user_data.get('created_at', 'unknown')}",
f"- **Jobs created:** {user_data.get('total_jobs_created', 0)}",
f"- **Jobs completed:** {user_data.get('total_jobs_completed', 0)}",
f"- **Total spent (real money):** ${user_data.get('total_spent', 0) / 100:.2f}",
f"- **Current credits:** {user_data.get('credits', 0)}",
"",
]
# IP geolocation
if ip_geo and ip_geo.get("status") == "success":
lines.extend([
f"## IP Geolocation",
f"- **Country:** {ip_geo.get('country', 'unknown')} ({ip_geo.get('country_code', '')})",
f"- **City/Region:** {ip_geo.get('city', '')}, {ip_geo.get('region', '')}",
f"- **ISP:** {ip_geo.get('isp', 'unknown')}",
f"- **Organization:** {ip_geo.get('org', 'unknown')}",
f"- **ASN:** {ip_geo.get('as_number', '')} {ip_geo.get('as_name', '')}",
"",
])
# Recent signup patterns
lines.extend([
f"## Recent Signup Patterns (last 24 hours)",
f"- **New accounts from same IP:** {recent_signups_ip}",
f"- **New accounts from same fingerprint:** {recent_signups_fp}",
"",
])
# Fingerprint correlations
if fingerprint_matches:
lines.append("## Other Accounts with Same Device Fingerprint (RED FLAG)")
for match in fingerprint_matches:
lines.append(
f"- {match.get('email', '?')} | "
f"Jobs: {match.get('total_jobs_created', 0)} | "
f"Spent: ${match.get('total_spent', 0) / 100:.2f} | "
f"Created: {match.get('created_at', '?')}"
)
lines.append("")
else:
lines.extend(["## Other Accounts with Same Device Fingerprint", "None found.", ""])
# IP correlations
if ip_matches:
lines.append("## Other Accounts from Same Signup IP")
for match in ip_matches:
lines.append(
f"- {match.get('email', '?')} | "
f"Fingerprint: {match.get('device_fingerprint', 'unknown')} | "
f"Jobs: {match.get('total_jobs_created', 0)} | "
f"Spent: ${match.get('total_spent', 0) / 100:.2f} | "
f"Created: {match.get('created_at', '?')}"
)
lines.append("")
else:
lines.extend(["## Other Accounts from Same Signup IP", "None found.", ""])
# Feedback content (if applicable)
if feedback_content:
lines.extend([
"## Feedback Submitted",
f"- **Overall rating:** {feedback_content.get('overall_rating', '?')}/5",
f"- **What went well:** {feedback_content.get('what_went_well', 'N/A')[:200]}",
f"- **What could improve:** {feedback_content.get('what_could_improve', 'N/A')[:200]}",
f"- **Additional comments:** {feedback_content.get('additional_comments', 'N/A')[:200]}",
"",
])
lines.append("Based on these signals, should we GRANT or DENY free credits to this user?")
return "\n".join(lines)
def _parse_gemini_response(response_text: str) -> CreditEvaluation:
"""Parse Gemini's JSON response into a CreditEvaluation."""
try:
# Strip markdown code fences if present
text = response_text.strip()
if text.startswith("```"):
text = text.split("\n", 1)[1] if "\n" in text else text[3:]
if text.endswith("```"):
text = text.rsplit("```", 1)[0]
text = text.strip()
if text.startswith("json"):
text = text[4:].strip()
data = json.loads(text)
decision = data.get("decision", "").lower()
if decision not in ("grant", "deny"):
decision = "pending_review" # fail-closed: unknown decision → manual review
return CreditEvaluation(
decision=decision,
reasoning=str(data.get("reasoning", "No reasoning provided")),
confidence=float(data.get("confidence", 0.5)),
)
except (json.JSONDecodeError, ValueError, TypeError) as e:
logger.warning(f"Failed to parse Gemini response: {e}. Response: {response_text[:200]}")
return CreditEvaluation(
decision="pending_review",
reasoning="Failed to parse AI response — pending manual review",
confidence=0.0,
error=f"Parse error: {e}",
)
class CreditEvaluationService:
"""Evaluates credit grant requests using Gemini AI."""
def __init__(self):
self.settings = get_settings()
def _collect_signals(self, email: str) -> dict:
"""Collect all abuse signals for evaluation."""
from backend.services.user_service import get_user_service
from backend.services.ip_geolocation_service import get_ip_geolocation_service
user_service = get_user_service()
user = user_service.get_user(email)
if not user:
return {
"user_data": {},
"fingerprint_matches": [],
"ip_matches": [],
"ip_geo": None,
"recent_signups_ip": 0,
"recent_signups_fp": 0,
"user_agent": None,
}
user_data = {
"email": user.email,
"device_fingerprint": user.device_fingerprint,
"signup_ip": user.signup_ip,
"credits": user.credits,
"total_jobs_created": user.total_jobs_created,
"total_jobs_completed": user.total_jobs_completed,
"total_spent": user.total_spent,
"created_at": str(user.created_at),
}
# Find other accounts with same fingerprint
fingerprint_matches = []
if user.device_fingerprint:
fp_users = user_service.find_users_by_fingerprint(user.device_fingerprint)
fingerprint_matches = [
{
"email": u.email,
"total_jobs_created": u.total_jobs_created,
"total_spent": u.total_spent,
"created_at": str(u.created_at),
}
for u in fp_users if u.email != email
]
# Find other accounts from same IP
ip_matches = []
if user.signup_ip:
ip_users = user_service.find_users_by_signup_ip(user.signup_ip)
ip_matches = [
{
"email": u.email,
"device_fingerprint": u.device_fingerprint,
"total_jobs_created": u.total_jobs_created,
"total_spent": u.total_spent,
"created_at": str(u.created_at),
}
for u in ip_users if u.email != email
]
# IP geolocation
ip_geo = None
if user.signup_ip:
try:
geo_service = get_ip_geolocation_service()
ip_geo = geo_service.lookup_ip(user.signup_ip)
except Exception:
pass
# Recent signup counts
recent_signups_ip = 0
recent_signups_fp = 0
if user.signup_ip:
recent_signups_ip = user_service.count_recent_signups_from_ip(user.signup_ip)
if user.device_fingerprint:
recent_signups_fp = user_service.count_recent_signups_from_fingerprint(user.device_fingerprint)
# User agent from latest session
user_agent = None
try:
from google.cloud.firestore_v1 import FieldFilter
from google.cloud import firestore as fs
sessions = (
user_service.db.collection("sessions")
.where(filter=FieldFilter("user_email", "==", email))
.order_by("created_at", direction=fs.Query.DESCENDING)
.limit(1)
.stream()
)
for session_doc in sessions:
user_agent = session_doc.to_dict().get("user_agent")
except Exception:
pass
return {
"user_data": user_data,
"fingerprint_matches": fingerprint_matches,
"ip_matches": ip_matches,
"ip_geo": ip_geo,
"recent_signups_ip": recent_signups_ip,
"recent_signups_fp": recent_signups_fp,
"user_agent": user_agent,
}
def _call_gemini(self, prompt: str) -> str:
"""Call Gemini and return the response text."""
from google import genai
client = genai.Client(
vertexai=True,
project=self.settings.google_cloud_project,
location="global",
)
response = client.models.generate_content(
model=self.settings.credit_eval_model,
contents=prompt,
config=genai.types.GenerateContentConfig(
system_instruction=SYSTEM_PROMPT,
),
)
return response.text
def _log_evaluation(
self,
email: str,
grant_type: str,
evaluation: CreditEvaluation,
signals: dict,
):
"""Log the evaluation decision to Firestore for audit trail."""
try:
from google.cloud import firestore as fs
db = fs.Client(project=self.settings.google_cloud_project)
doc_id = str(uuid.uuid4())
db.collection(CREDIT_EVALUATIONS_COLLECTION).document(doc_id).set({
"id": doc_id,
"email": email,
"grant_type": grant_type,
"decision": evaluation.decision,
"reasoning": evaluation.reasoning,
"confidence": evaluation.confidence,
"error": evaluation.error,
"model": self.settings.credit_eval_model,
"signals_snapshot": {
"fingerprint_match_count": len(signals.get("fingerprint_matches", [])),
"ip_match_count": len(signals.get("ip_matches", [])),
"recent_signups_ip": signals.get("recent_signups_ip", 0),
"recent_signups_fp": signals.get("recent_signups_fp", 0),
"user_ip": signals.get("user_data", {}).get("signup_ip"),
"user_fingerprint": signals.get("user_data", {}).get("device_fingerprint"),
},
"evaluated_at": datetime.now(timezone.utc).isoformat(),
})
except Exception:
logger.exception(f"Failed to log credit evaluation for {email}")
def evaluate(
self,
email: str,
grant_type: str,
feedback_content: Optional[dict] = None,
) -> CreditEvaluation:
"""
Evaluate whether to grant free credits.
Args:
email: User's email
grant_type: "welcome" or "feedback"
feedback_content: Feedback form data (for feedback grants only)
Returns:
CreditEvaluation with decision, reasoning, confidence.
Fail-closed: returns "pending_review" on any error (admin notified).
"""
if not self.settings.credit_eval_enabled:
return CreditEvaluation(
decision="grant",
reasoning="Credit evaluation disabled — auto-granting",
confidence=1.0,
)
try:
# Collect signals
signals = self._collect_signals(email)
# Quick-grant: if no correlations found at all, skip AI call
if (
not signals["fingerprint_matches"]
and not signals["ip_matches"]
and signals["recent_signups_ip"] <= 1
and signals["recent_signups_fp"] <= 1
):
evaluation = CreditEvaluation(
decision="grant",
reasoning="No suspicious correlations found — clean user",
confidence=1.0,
signals=signals,
)
self._log_evaluation(email, grant_type, evaluation, signals)
return evaluation
# Build prompt and call Gemini
prompt = _build_user_prompt(
email=email,
grant_type=grant_type,
user_data=signals["user_data"],
fingerprint_matches=signals["fingerprint_matches"],
ip_matches=signals["ip_matches"],
ip_geo=signals["ip_geo"],
recent_signups_ip=signals["recent_signups_ip"],
recent_signups_fp=signals["recent_signups_fp"],
user_agent=signals["user_agent"],
feedback_content=feedback_content,
)
response_text = self._call_gemini(prompt)
evaluation = _parse_gemini_response(response_text)
evaluation.signals = signals
self._log_evaluation(email, grant_type, evaluation, signals)
logger.info(
f"Credit evaluation for {email} ({grant_type}): "
f"{evaluation.decision} (confidence={evaluation.confidence:.2f})"
)
return evaluation
except Exception as e:
logger.exception(f"Credit evaluation failed for {email} — pending manual review (fail-closed)")
# Include whatever signals were collected before the failure
collected_signals = locals().get("signals")
evaluation = CreditEvaluation(
decision="pending_review",
reasoning="Evaluation failed — pending manual review",
confidence=0.0,
error=str(e),
signals=collected_signals,
)
try:
self._log_evaluation(email, grant_type, evaluation, collected_signals or {})
except Exception:
pass
return evaluation
_service_instance: Optional[CreditEvaluationService] = None
def get_credit_evaluation_service() -> CreditEvaluationService:
"""Get or create the singleton credit evaluation service."""
global _service_instance
if _service_instance is None:
_service_instance = CreditEvaluationService()
return _service_instance
@@ -0,0 +1,129 @@
"""
Disposable Domain Sync Service.
Fetches and syncs disposable email domain blocklists from external sources.
"""
import logging
from datetime import datetime, timezone
from typing import Set
import httpx
from backend.services.email_validation_service import (
BLOCKLISTS_COLLECTION,
BLOCKLIST_CONFIG_DOC,
DEFAULT_DISPOSABLE_DOMAINS,
)
logger = logging.getLogger(__name__)
EXTERNAL_BLOCKLIST_URL = (
"https://raw.githubusercontent.com/disposable-email-domains/"
"disposable-email-domains/refs/heads/main/disposable_email_blocklist.conf"
)
FETCH_TIMEOUT_SECONDS = 30
MAX_RESPONSE_BYTES = 2 * 1024 * 1024 # 2 MB
MAX_DOMAIN_COUNT = 50_000
def parse_blocklist_text(text: str) -> Set[str]:
"""Parse newline-delimited blocklist text into a set of lowercase domains."""
domains = set()
for line in text.splitlines():
domain = line.strip().lower()
if domain:
domains.add(domain)
return domains
async def fetch_external_blocklist() -> Set[str]:
"""Fetch the external disposable domain blocklist from GitHub.
Checks Content-Length header and response body size against safety limits.
"""
async with httpx.AsyncClient(timeout=FETCH_TIMEOUT_SECONDS) as client:
response = await client.get(EXTERNAL_BLOCKLIST_URL)
response.raise_for_status()
content_length = response.headers.get("content-length")
if content_length and int(content_length) > MAX_RESPONSE_BYTES:
raise ValueError(
f"Response size {content_length} exceeds maximum {MAX_RESPONSE_BYTES}"
)
text = response.text
if len(text.encode()) > MAX_RESPONSE_BYTES:
raise ValueError(
f"Response body size exceeds maximum {MAX_RESPONSE_BYTES}"
)
domains = parse_blocklist_text(text)
if len(domains) > MAX_DOMAIN_COUNT:
raise ValueError(
f"Domain count {len(domains)} exceeds maximum {MAX_DOMAIN_COUNT}"
)
logger.info(f"Fetched {len(domains)} domains from external blocklist")
return domains
def sync_disposable_domains(db, external_domains: Set[str]) -> dict:
"""
Sync external domains into Firestore.
Handles first-run migration from old disposable_domains field,
and cleans up redundant manual domains on subsequent runs.
Note: This is a synchronous function (Firestore SDK is sync).
Call from async code with `await asyncio.to_thread(sync_disposable_domains, db, domains)`.
Returns a summary dict with counts.
"""
doc_ref = db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
doc = doc_ref.get()
data = doc.to_dict() if doc.exists else {}
result = {"external_count": len(external_domains), "added": 0, "removed": 0, "migrated_to_manual": []}
is_migration = "external_domains" not in data
if is_migration:
old_domains = set(data.get("disposable_domains", [])) | DEFAULT_DISPOSABLE_DOMAINS
manual_only = old_domains - external_domains
result["migrated_to_manual"] = sorted(manual_only)
new_data = {
**data, # preserve all existing fields
"external_domains": sorted(external_domains),
"manual_domains": sorted(manual_only),
"allowlisted_domains": data.get("allowlisted_domains", []),
"last_sync_at": datetime.now(timezone.utc),
"last_sync_count": len(external_domains),
}
new_data.pop("disposable_domains", None) # remove legacy field
doc_ref.set(new_data)
else:
old_external = set(data.get("external_domains", []))
manual_domains = set(data.get("manual_domains", []))
result["added"] = len(external_domains - old_external)
result["removed"] = len(old_external - external_domains)
cleaned_manual = manual_domains - external_domains
new_data = {
**data,
"external_domains": sorted(external_domains),
"manual_domains": sorted(cleaned_manual),
"last_sync_at": datetime.now(timezone.utc),
"last_sync_count": len(external_domains),
}
doc_ref.set(new_data)
logger.info(
f"Sync complete: {result['external_count']} external domains, "
f"{result['added']} added, {result['removed']} removed"
)
return result
+4
View File
@@ -111,6 +111,10 @@ class DropboxService:
"""
Calculate next brand code from existing folders, filling gaps in the sequence.
NOTE: This method is only used for one-time initialization of the Firestore
counter (see brand_code_service.py). All runtime brand code allocation now
goes through BrandCodeService.allocate_brand_code() for atomic safety.
Scans the folder for existing brand codes like "NOMAD-0001" and
returns the first available code, filling any gaps left by deleted outputs.
File diff suppressed because it is too large. Load diff
+410 -200
View File
@@ -2,21 +2,43 @@
Email Validation Service.
Provides email validation, normalization, and blocklist checking
to prevent abuse during beta enrollment and other flows.
to prevent abuse during enrollment and other flows.
Uses a tiered disposable domain detection strategy:
1. Static blocklist (instant, from Firestore)
2. Verified-clean domain cache (skip API calls for 7 days)
3. DeBounce API (free, <50ms)
4. verifymail.io API (paid, only if DeBounce says clean)
Auto-learns: flagged domains are persisted to manual_domains,
clean domains are cached for 7 days in verified_clean_domains.
"""
import hashlib
import logging
import re
from datetime import datetime, timezone
from datetime import datetime, timedelta, timezone
from typing import Optional, Tuple, Set
import httpx
from google.cloud import firestore
from backend.config import settings
logger = logging.getLogger(__name__)
# Module-level HTTP client for connection pooling (external API checks)
_http_client: Optional[httpx.Client] = None
_HTTP_TIMEOUT = 2.0 # seconds per external API call
def _get_http_client() -> httpx.Client:
"""Get or create the module-level HTTP client."""
global _http_client
if _http_client is None or _http_client.is_closed:
_http_client = httpx.Client(timeout=_HTTP_TIMEOUT)
return _http_client
def _mask_email(email: str) -> str:
"""Mask an email address for privacy-safe logging."""
@@ -34,164 +56,12 @@ def _mask_email(email: str) -> str:
BLOCKLISTS_COLLECTION = "blocklists"
BLOCKLIST_CONFIG_DOC = "config"
# Default disposable email domains (can be extended via admin UI)
DEFAULT_DISPOSABLE_DOMAINS = {
# Popular temporary email services
"10minutemail.com",
"10minutemail.net",
"guerrillamail.com",
"guerrillamail.org",
"guerrillamail.net",
"guerrillamail.biz",
"guerrillamailblock.com",
"tempmail.com",
"tempmail.net",
"temp-mail.org",
"temp-mail.io",
"throwawaymail.com",
"mailinator.com",
"mailinator.net",
"mailinator.org",
"mailinator.info",
"maildrop.cc",
"yopmail.com",
"yopmail.fr",
"yopmail.net",
"fakeinbox.com",
"fakemailgenerator.com",
"dispostable.com",
"getairmail.com",
"getnada.com",
"mohmal.com",
"trashmail.com",
"trashmail.net",
"trashmail.org",
"sharklasers.com",
"grr.la",
"guerrillamail.de",
"pokemail.net",
"spam4.me",
"spamgourmet.com",
"mytrashmail.com",
"mailnesia.com",
"mailcatch.com",
"mintemail.com",
"tempr.email",
"tempail.com",
"emailondeck.com",
"incognitomail.com",
"inboxalias.com",
"33mail.com",
"spamex.com",
"spamfree24.org",
"spamspot.com",
"mailnull.com",
"mailsac.com",
"emailfake.com",
"fakemail.fr",
"tempinbox.com",
"throwaway.email",
"burnermail.io",
"anonbox.net",
"anonymbox.com",
"discard.email",
"discardmail.com",
"discardmail.de",
"mailexpire.com",
"mailforspam.com",
"meltmail.com",
"mt2009.com",
"mt2014.com",
"nospam.ze.tc",
"nospamfor.us",
"nowmymail.com",
"receiveee.com",
"safe-mail.net",
"spamavert.com",
"spambob.com",
"spambog.com",
"spambox.us",
"spamcannon.com",
"spamcannon.net",
"spamcon.org",
"spamcorptastic.com",
"spamday.com",
"spamfree.eu",
"spamherelots.com",
"spamhereplease.com",
"spamhole.com",
"spamify.com",
"spaminator.de",
"spamkill.info",
"spaml.com",
"spaml.de",
"spamoff.de",
"spamobox.com",
"spamslicer.com",
"spamstack.net",
"spamthis.co.uk",
"spamthisplease.com",
"supergreatmail.com",
"suremail.info",
"teleworm.us",
"tempemail.co.za",
"tempemail.net",
"tempmailaddress.com",
"tempmailo.com",
"thankyou2010.com",
"thisisnotmyrealemail.com",
"tm.slsrs.ru",
"tmpeml.info",
"trash-mail.at",
"trash-mail.de",
"trash2009.com",
"trashemail.de",
"trashmail.at",
"trashmailer.com",
"wegwerfmail.de",
"wegwerfmail.net",
"wegwerfmail.org",
"wh4f.org",
"willhackforfood.biz",
"willselfdestruct.com",
"xmaily.com",
"xyzfree.net",
"yep.it",
"yogamaven.com",
"yuurok.com",
"zehnminutenmail.de",
"zippymail.info",
# Additional common ones
"mailnator.com",
"bugmenot.com",
"dodgeit.com",
"dodgit.com",
"e4ward.com",
"emailsensei.com",
"hushmail.com",
"jetable.org",
"kasmail.com",
"mailblock.net",
"mailcatch.com",
"mymailoasis.com",
"nervmich.net",
"nervtmansen.de",
"oneoffemail.com",
"pookmail.com",
"shortmail.net",
"sneakemail.com",
"sogetthis.com",
"tempemailaddress.com",
"tempomail.fr",
"temporaryemail.net",
"temporaryforwarding.com",
"temporaryinbox.com",
"thankyou2010.com",
"tyldd.com",
"uggsrock.com",
"veryrealemail.com",
"yourewronghereswhy.com",
}
# Legacy default set — kept as empty set for import compatibility.
# The external sync (disposable_domain_sync_service) replaces this with ~4,800
# domains from the community-curated disposable-email-domains repo.
# The migration path in sync_disposable_domains() references this set to ensure
# no domains are lost during the transition.
DEFAULT_DISPOSABLE_DOMAINS: Set[str] = set()
# Gmail-like domains that support alias normalization
GMAIL_LIKE_DOMAINS = {
@@ -199,6 +69,47 @@ GMAIL_LIKE_DOMAINS = {
"googlemail.com",
}
# Well-known email providers that are never disposable.
# These skip all external API checks for zero-latency validation.
WELL_KNOWN_PROVIDERS = {
"gmail.com",
"googlemail.com",
"yahoo.com",
"yahoo.co.uk",
"yahoo.co.jp",
"yahoo.fr",
"hotmail.com",
"hotmail.co.uk",
"outlook.com",
"outlook.fr",
"live.com",
"live.co.uk",
"msn.com",
"icloud.com",
"me.com",
"mac.com",
"aol.com",
"protonmail.com",
"proton.me",
"zoho.com",
"yandex.com",
"yandex.ru",
"mail.com",
"gmx.com",
"gmx.de",
"fastmail.com",
"hey.com",
"tutanota.com",
"tuta.io",
}
# How long a verified-clean domain result is trusted before re-checking
VERIFIED_CLEAN_TTL = timedelta(days=7)
# External API URLs
DEBOUNCE_API_URL = "https://disposable.debounce.io/"
VERIFYMAIL_API_URL = "https://verifymail.io/api"
class EmailValidationService:
"""
@@ -293,17 +204,30 @@ class EmailValidationService:
if doc.exists:
data = doc.to_dict()
if "external_domains" in data:
# New data model: effective = (external + manual) - allowlisted
external = set(data.get("external_domains", []))
manual = set(data.get("manual_domains", []))
allowlisted = set(data.get("allowlisted_domains", []))
effective_domains = (external | manual) - allowlisted
else:
# Old data model (pre-migration): use disposable_domains + defaults
allowlisted = set()
effective_domains = set(data.get("disposable_domains", [])) | DEFAULT_DISPOSABLE_DOMAINS
config = {
"disposable_domains": set(data.get("disposable_domains", [])) | DEFAULT_DISPOSABLE_DOMAINS,
"disposable_domains": effective_domains,
"allowlisted_domains": allowlisted,
"blocked_emails": set(data.get("blocked_emails", [])),
"blocked_ips": set(data.get("blocked_ips", [])),
"verified_clean_domains": data.get("verified_clean_domains", {}),
}
else:
# Use defaults if no config exists
config = {
"disposable_domains": DEFAULT_DISPOSABLE_DOMAINS.copy(),
"allowlisted_domains": set(),
"blocked_emails": set(),
"blocked_ips": set(),
"verified_clean_domains": {},
}
# Update cache
@@ -316,6 +240,16 @@ class EmailValidationService:
"""
Check if an email uses a disposable domain.
Uses a tiered checking strategy:
0. Well-known providers (gmail, yahoo, etc.) → instant False
1. Static blocklist from Firestore → instant
2. Verified-clean cache (7-day TTL) → skip API calls
3. DeBounce API (free) → fast external check
4. verifymail.io API (paid) → only if DeBounce says clean
Flagged domains are auto-learned to manual_domains.
Clean domains are cached for 7 days.
Args:
email: Email address to check
@@ -326,8 +260,194 @@ class EmailValidationService:
return False
domain = email.lower().split("@")[-1]
# Tier 0: Well-known providers — skip everything
if domain in WELL_KNOWN_PROVIDERS:
return False
# Tier 1: Static blocklist (instant, always available)
config = self.get_blocklist_config()
return domain in config["disposable_domains"]
if domain in config["disposable_domains"]:
return True
# Tier 1.5: Explicitly allowlisted domains skip external API checks.
# Without this, external APIs can override admin allowlist decisions
# (e.g. inbox.testmail.app is a legitimate testing domain we allow).
if domain in config.get("allowlisted_domains", set()):
return False
# Tier 2: Recently verified clean? Skip API calls
if self._is_domain_verified_clean(domain, config):
return False
# Tier 3: DeBounce API (free, fast)
debounce_result = self._check_debounce(email)
if debounce_result is True:
self._auto_learn_domain(domain)
return True
# Tier 4: verifymail.io (paid, only if DeBounce gave definitive clean)
if debounce_result is False:
verifymail_result = self._check_verifymail(email)
if verifymail_result is True:
self._auto_learn_domain(domain)
return True
# Both APIs say clean (or errored) — cache clean result
if debounce_result is False:
self._mark_domain_clean(domain)
return False
def _check_debounce(self, email: str) -> Optional[bool]:
"""
Check email against DeBounce disposable email API.
Returns:
True if disposable, False if clean, None if error/timeout.
"""
try:
client = _get_http_client()
response = client.get(DEBOUNCE_API_URL, params={"email": email})
response.raise_for_status()
data = response.json()
result = str(data.get("disposable", "")).lower() == "true"
logger.info(
f"DeBounce check for {_mask_email(email)}: "
f"disposable={result}"
)
return result
except httpx.TimeoutException:
logger.warning(f"DeBounce API timeout for {_mask_email(email)}")
return None
except Exception as e:
logger.warning(f"DeBounce API error for {_mask_email(email)}: {e}")
return None
def _check_verifymail(self, email: str) -> Optional[bool]:
"""
Check email against verifymail.io API.
Only called when DeBounce says clean (to minimize paid API usage).
Returns:
True if disposable/blocked, False if clean, None if error/timeout
or API key not configured.
"""
api_key = settings.get_secret("verifymail-api-key")
if not api_key:
logger.debug("verifymail.io API key not configured, skipping")
return None
try:
client = _get_http_client()
response = client.get(
f"{VERIFYMAIL_API_URL}/{email}",
params={"key": api_key},
)
response.raise_for_status()
data = response.json()
# verifymail.io returns both "disposable" and "block" flags
is_disposable = data.get("disposable") is True
is_blocked = data.get("block") is True
result = is_disposable or is_blocked
logger.info(
f"verifymail.io check for {_mask_email(email)}: "
f"disposable={is_disposable}, block={is_blocked}"
)
return result
except httpx.TimeoutException:
logger.warning(f"verifymail.io API timeout for {_mask_email(email)}")
return None
except Exception as e:
logger.warning(f"verifymail.io API error for {_mask_email(email)}: {e}")
return None
def _is_domain_verified_clean(self, domain: str, config: dict) -> bool:
"""Check if a domain was verified clean within the TTL period."""
verified = config.get("verified_clean_domains", {})
entry = verified.get(domain)
if not entry:
return False
checked_at = entry.get("checked_at")
if not checked_at:
return False
# Handle both datetime objects and ISO strings
if isinstance(checked_at, str):
try:
checked_at = datetime.fromisoformat(checked_at)
except (ValueError, TypeError):
return False
# Make timezone-aware if naive
if checked_at.tzinfo is None:
checked_at = checked_at.replace(tzinfo=timezone.utc)
return (datetime.now(timezone.utc) - checked_at) < VERIFIED_CLEAN_TTL
def _auto_learn_domain(self, domain: str) -> None:
"""
Add a domain flagged by external API to manual_domains in Firestore.
This persists the result so future checks don't need the API call.
Fire-and-forget: errors are logged but don't affect the caller.
"""
try:
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(
BLOCKLIST_CONFIG_DOC
)
doc = doc_ref.get()
if doc.exists:
data = doc.to_dict()
manual = set(data.get("manual_domains", []))
else:
manual = set()
if domain not in manual:
manual.add(domain)
doc_ref.set(
{
"manual_domains": list(manual),
"updated_at": datetime.now(timezone.utc).isoformat(),
"updated_by": "auto-learn",
},
merge=True,
)
logger.info(f"Auto-learned disposable domain: {domain}")
# Invalidate cache so the domain is blocked immediately
EmailValidationService._blocklist_cache = None
except Exception as e:
logger.error(f"Failed to auto-learn domain {domain}: {e}")
def _mark_domain_clean(self, domain: str) -> None:
"""
Record that a domain was verified clean by external APIs.
Stored in verified_clean_domains dict keyed by domain.
Future checks within VERIFIED_CLEAN_TTL will skip API calls.
Note: Uses read-modify-write instead of dot notation because
domain names contain dots which Firestore interprets as nested paths.
"""
try:
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(
BLOCKLIST_CONFIG_DOC
)
doc = doc_ref.get()
verified = doc.to_dict().get("verified_clean_domains", {}) if doc.exists else {}
verified[domain] = {
"checked_at": datetime.now(timezone.utc).isoformat(),
}
doc_ref.set({"verified_clean_domains": verified}, merge=True)
logger.debug(f"Marked domain as verified clean: {domain}")
# Invalidate cache so new data is picked up
EmailValidationService._blocklist_cache = None
except Exception as e:
logger.error(f"Failed to mark domain clean {domain}: {e}")
def is_email_blocked(self, email: str) -> bool:
"""
@@ -369,36 +489,6 @@ class EmailValidationService:
config = self.get_blocklist_config()
return ip_address in config["blocked_ips"]
def validate_email_for_beta(self, email: str) -> Tuple[bool, str]:
"""
Validate an email for beta enrollment.
Performs all validation checks:
1. Basic format validation
2. Disposable domain check
3. Blocked email check
Args:
email: Email address to validate
Returns:
Tuple of (is_valid, error_message)
"""
if not email or "@" not in email:
return False, "Invalid email format"
# Check disposable domain
if self.is_disposable_domain(email):
logger.warning(f"Beta enrollment blocked - disposable domain: {_mask_email(email)}")
return False, "Disposable email addresses are not allowed"
# Check blocked email
if self.is_email_blocked(email):
logger.warning(f"Beta enrollment blocked - email blocked: {_mask_email(email)}")
return False, "This email address is not allowed"
return True, ""
def hash_ip(self, ip_address: str) -> str:
"""
Hash an IP address for privacy-preserving storage.
@@ -428,13 +518,13 @@ class EmailValidationService:
doc = doc_ref.get(transaction=transaction)
if doc.exists:
data = doc.to_dict()
domains = set(data.get("disposable_domains", []))
domains = set(data.get("manual_domains", []))
else:
data = {}
domains = set()
domains.add(domain)
data["disposable_domains"] = list(domains)
data["manual_domains"] = list(domains)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
@@ -462,14 +552,22 @@ class EmailValidationService:
return
data = doc.to_dict()
domains = set(data.get("disposable_domains", []))
if domain not in domains:
manual = set(data.get("manual_domains", []))
external = set(data.get("external_domains", []))
if domain in manual:
result["found"] = True
manual.discard(domain)
data["manual_domains"] = list(manual)
elif domain in external:
result["found"] = True
allowlisted = set(data.get("allowlisted_domains", []))
allowlisted.add(domain)
data["allowlisted_domains"] = list(allowlisted)
else:
return
result["found"] = True
domains.discard(domain)
data["disposable_domains"] = list(domains)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
@@ -628,14 +726,126 @@ class EmailValidationService:
logger.info(f"Removed blocked IP: {ip_address} by {admin_email}")
return True
def add_allowlisted_domain(self, domain: str, admin_email: str) -> bool:
"""Add a domain to the allowlisted domains list."""
domain = domain.lower().strip()
if not domain:
return False
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
@firestore.transactional
def update_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if doc.exists:
data = doc.to_dict()
domains = set(data.get("allowlisted_domains", []))
else:
data = {}
domains = set()
domains.add(domain)
data["allowlisted_domains"] = list(domains)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
transaction = self.db.transaction()
update_in_transaction(transaction, doc_ref)
# Invalidate cache
EmailValidationService._blocklist_cache = None
logger.info(f"Added allowlisted domain: {domain} by {admin_email}")
return True
def remove_allowlisted_domain(self, domain: str, admin_email: str) -> bool:
"""Remove a domain from the allowlisted domains list."""
domain = domain.lower().strip()
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
result = {"found": False}
@firestore.transactional
def remove_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if not doc.exists:
return
data = doc.to_dict()
domains = set(data.get("allowlisted_domains", []))
if domain not in domains:
return
result["found"] = True
domains.discard(domain)
data["allowlisted_domains"] = list(domains)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
transaction = self.db.transaction()
remove_in_transaction(transaction, doc_ref)
if not result["found"]:
return False
# Invalidate cache
EmailValidationService._blocklist_cache = None
logger.info(f"Removed allowlisted domain: {domain} by {admin_email}")
return True
def get_blocklist_raw_data(self) -> dict:
"""Get raw blocklist data for admin display (not the effective set)."""
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
doc = doc_ref.get()
if not doc.exists:
return {
"external_domains": [], "manual_domains": [], "allowlisted_domains": [],
"blocked_emails": [], "blocked_ips": [],
"last_sync_at": None, "last_sync_count": None,
"updated_at": None, "updated_by": None,
}
data = doc.to_dict()
return {
"external_domains": sorted(data.get("external_domains", [])),
"manual_domains": sorted(data.get("manual_domains", [])),
"allowlisted_domains": sorted(data.get("allowlisted_domains", [])),
"blocked_emails": sorted(data.get("blocked_emails", [])),
"blocked_ips": sorted(data.get("blocked_ips", [])),
"last_sync_at": data.get("last_sync_at"),
"last_sync_count": data.get("last_sync_count"),
"updated_at": data.get("updated_at"),
"updated_by": data.get("updated_by"),
}
def get_blocklist_stats(self) -> dict:
"""Get statistics about current blocklists."""
config = self.get_blocklist_config(force_refresh=True)
# Also get raw data for detailed counts
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
doc = doc_ref.get()
if doc.exists:
data = doc.to_dict()
external_count = len(data.get("external_domains", []))
manual_count = len(data.get("manual_domains", []))
allowlisted_count = len(data.get("allowlisted_domains", []))
else:
external_count = 0
manual_count = 0
allowlisted_count = 0
return {
"disposable_domains_count": len(config["disposable_domains"]),
"blocked_emails_count": len(config["blocked_emails"]),
"blocked_ips_count": len(config["blocked_ips"]),
"default_disposable_domains_count": len(DEFAULT_DISPOSABLE_DOMAINS),
"external_domains_count": external_count,
"manual_domains_count": manual_count,
"allowlisted_domains_count": allowlisted_count,
}
+3
View File
@@ -383,6 +383,8 @@ class GCEEncodingBackend(EncodingBackend):
output_files["mp4_4k_lossy"] = path
elif "720p" in filename_lower:
output_files["mp4_720p"] = path
elif "with vocals" in filename_lower and filename.endswith(".mp4"):
output_files["with_vocals_mp4"] = path
self.logger.info(f"Converted output_files list to dict: {output_files}")
else:
output_files = raw_output_files if isinstance(raw_output_files, dict) else {}
@@ -393,6 +395,7 @@ class GCEEncodingBackend(EncodingBackend):
lossy_4k_mp4_path=output_files.get("mp4_4k_lossy"),
lossless_mkv_path=output_files.get("mkv_4k"),
lossy_720p_mp4_path=output_files.get("mp4_720p"),
with_vocals_mp4_path=output_files.get("with_vocals_mp4"),
output_files=output_files,
encoding_time_seconds=encoding_time,
encoding_backend=self.name
+258 -13
View File
@@ -17,6 +17,7 @@ Usage:
import asyncio
import logging
import time
from typing import Optional, Dict, Any
import aiohttp
@@ -25,10 +26,23 @@ from backend.config import get_settings
logger = logging.getLogger(__name__)
# Retry configuration for handling transient failures (e.g., worker restarts)
MAX_RETRIES = 3
INITIAL_BACKOFF_SECONDS = 2.0
MAX_BACKOFF_SECONDS = 10.0
# Retry configuration for handling transient failures (e.g., worker restarts during deployments).
#
# During a CI deployment, the encoding worker is restarted via systemctl. The restart
# involves downloading a new wheel from GCS, installing it, and starting uvicorn —
# which can take 30-90 seconds. The retry window must exceed this to avoid failing
# jobs that happen to hit the encoding stage during a deploy.
#
# With 7 retries and exponential backoff (5s, 10s, 15s, 15s, 15s, 15s, 15s),
# total retry window is ~90s, sufficient to survive a full worker restart.
MAX_RETRIES = 7
INITIAL_BACKOFF_SECONDS = 5.0
MAX_BACKOFF_SECONDS = 15.0
# Poll failure tolerance: number of consecutive poll failures allowed before giving up.
# This prevents a single transient network blip during status polling from killing a
# long-running encoding job. Similar pattern to flacfetch status polling (PR #446).
MAX_CONSECUTIVE_POLL_FAILURES = 5
class EncodingService:
@@ -39,6 +53,34 @@ class EncodingService:
self._url = None
self._api_key = None
self._initialized = False
self._worker_manager = None
self._cached_url = None
self._url_cached_at = 0
self._URL_CACHE_TTL = 30 # seconds
def set_worker_manager(self, manager):
"""Set the worker manager for dynamic URL resolution from Firestore."""
self._worker_manager = manager
def _get_worker_url(self) -> str:
"""Get the current primary worker URL from Firestore (with TTL cache).
Falls back to static URL from config if worker_manager is not set.
"""
now = time.time()
if self._cached_url and (now - self._url_cached_at) < self._URL_CACHE_TTL:
return self._cached_url
if self._worker_manager:
config = self._worker_manager.get_config()
self._cached_url = config.primary_url
self._url_cached_at = now
return self._cached_url
# Fallback to static URL
if not self._initialized:
self._load_credentials()
return self._url
def _load_credentials(self):
"""Load encoding worker URL and API key from config/secrets."""
@@ -73,6 +115,28 @@ class EncodingService:
"""Check if GCE preview encoding is enabled and configured."""
return self.settings.use_gce_preview_encoding and self.is_configured
def _warmup_encoding_worker_fallback(self, job_id: str) -> None:
"""Safety net: try to start the encoding worker VM on first connection failure.
If the worker_manager is available, calls ensure_primary_running() so the VM
starts booting during the retry window. If worker_manager is not set (e.g. dev
mode with static URL), this is a no-op.
"""
if not self._worker_manager:
return
try:
result = self._worker_manager.ensure_primary_running()
if result["started"]:
logger.warning(
f"[job:{job_id}] Encoding worker unreachable — started VM {result['vm_name']} as fallback"
)
else:
logger.info(
f"[job:{job_id}] Encoding worker unreachable — VM {result['vm_name']} already running/starting"
)
except Exception as e:
logger.warning(f"[job:{job_id}] Encoding worker warmup fallback failed (non-fatal): {e}")
async def _request_with_retry(
self,
method: str,
@@ -134,6 +198,8 @@ class EncodingService:
}
except (aiohttp.ClientConnectorError, aiohttp.ServerDisconnectedError, asyncio.TimeoutError) as e:
last_exception = e
if attempt == 0:
self._warmup_encoding_worker_fallback(job_id)
if attempt < MAX_RETRIES:
logger.warning(
f"[job:{job_id}] GCE worker connection failed (attempt {attempt + 1}/{MAX_RETRIES + 1}): {e}. "
@@ -175,7 +241,7 @@ class EncodingService:
if not self.is_configured:
raise RuntimeError("Encoding service not configured")
url = f"{self._url}/encode"
url = f"{self._get_worker_url()}/encode"
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
payload = {
"job_id": job_id,
@@ -198,7 +264,24 @@ class EncodingService:
if resp["status"] == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp["status"] == 409:
raise RuntimeError(f"Encoding job {job_id} already exists")
# Job already exists on worker — check if it already completed
logger.warning(f"[job:{job_id}] GCE worker returned 409, checking job status")
try:
status = await self.get_job_status(job_id)
job_status = status.get("status", "unknown")
if job_status == "complete":
logger.info(f"[job:{job_id}] Job already complete on GCE worker, returning cached result")
return {"status": "cached", "job_id": job_id, "output_files": status.get("output_files")}
elif job_status in ("pending", "running"):
logger.info(f"[job:{job_id}] Job still in progress on GCE worker")
return {"status": "in_progress", "job_id": job_id}
else:
raise RuntimeError(f"Encoding job {job_id} already exists with status: {job_status}")
except RuntimeError as e:
if "not found" in str(e).lower():
# Job was in worker memory but got cleared (restart) — safe to raise original error
raise RuntimeError(f"Encoding job {job_id} conflict: 409 but job not found on status check")
raise
if resp["status"] != 200:
raise RuntimeError(f"Failed to submit encoding job: {resp['status']} - {resp['text']}")
@@ -219,7 +302,7 @@ class EncodingService:
if not self.is_configured:
raise RuntimeError("Encoding service not configured")
url = f"{self._url}/status/{job_id}"
url = f"{self._get_worker_url()}/status/{job_id}"
headers = {"X-API-Key": self._api_key}
resp = await self._request_with_retry(
@@ -247,7 +330,12 @@ class EncodingService:
progress_callback=None,
) -> Dict[str, Any]:
"""
Poll for encoding job completion.
Poll for encoding job completion with tolerance for transient failures.
During a deployment, the encoding worker may restart while we're polling.
Rather than failing the entire job on a single poll error, we tolerate up to
MAX_CONSECUTIVE_POLL_FAILURES consecutive failures before giving up. This is
the same pattern used in flacfetch status polling (PR #446).
Args:
job_id: Job identifier
@@ -260,19 +348,40 @@ class EncodingService:
Raises:
TimeoutError: If job doesn't complete within timeout
RuntimeError: If job fails
RuntimeError: If job fails or too many consecutive poll failures
"""
logger.info(f"[job:{job_id}] Waiting for GCE encoding to complete...")
start_time = asyncio.get_event_loop().time()
last_progress = 0
consecutive_failures = 0
while True:
elapsed = asyncio.get_event_loop().time() - start_time
if elapsed > timeout:
raise TimeoutError(f"Encoding job {job_id} timed out after {timeout}s")
status = await self.get_job_status(job_id)
try:
status = await self.get_job_status(job_id)
# Reset failure counter on successful poll
consecutive_failures = 0
except (aiohttp.ClientConnectorError, aiohttp.ServerDisconnectedError,
asyncio.TimeoutError, RuntimeError) as e:
consecutive_failures += 1
if consecutive_failures >= MAX_CONSECUTIVE_POLL_FAILURES:
logger.error(
f"[job:{job_id}] Status polling failed {consecutive_failures} consecutive times, giving up: {e}"
)
raise RuntimeError(
f"Encoding job {job_id} lost contact with worker after "
f"{consecutive_failures} consecutive poll failures: {e}"
)
logger.warning(
f"[job:{job_id}] Status poll failed ({consecutive_failures}/{MAX_CONSECUTIVE_POLL_FAILURES}): {e}. "
f"Worker may be restarting, will retry..."
)
await asyncio.sleep(poll_interval)
continue
# Handle case where GCE worker returns a list instead of dict
if isinstance(status, list):
@@ -331,7 +440,17 @@ class EncodingService:
config = encoding_config or {"formats": ["mp4_4k", "mp4_720p"]}
# Submit the job
await self.submit_encoding_job(job_id, input_gcs_path, output_gcs_path, config)
submit_result = await self.submit_encoding_job(job_id, input_gcs_path, output_gcs_path, config)
# If cached, return immediately — encoding already done
submit_status = submit_result.get("status")
if submit_status == "cached":
logger.info(f"[job:{job_id}] Encoding already cached, returning immediately")
return {"status": "complete", "output_files": submit_result.get("output_files")}
# If in_progress, another request is encoding it — just wait for that
if submit_status == "in_progress":
logger.info(f"[job:{job_id}] Encoding already in progress, joining poll")
# Wait for completion
return await self.wait_for_completion(
@@ -371,7 +490,7 @@ class EncodingService:
if not self.is_configured:
raise RuntimeError("Encoding service not configured")
url = f"{self._url}/encode-preview"
url = f"{self._get_worker_url()}/encode-preview"
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
payload = {
"job_id": job_id,
@@ -465,6 +584,116 @@ class EncodingService:
timeout=timeout,
)
async def submit_render_video_job(
self,
job_id: str,
render_config: Dict[str, Any],
) -> Dict[str, Any]:
"""
Submit a render-video job to the GCE worker.
Args:
job_id: Unique job identifier
render_config: Configuration dict containing original_corrections_gcs_path,
audio_gcs_path, output_gcs_prefix, artist, title, and any other render params.
Returns:
Response from the encoding worker
Raises:
RuntimeError: If submission fails
"""
self._load_credentials()
if not self.is_configured:
raise RuntimeError("Encoding service not configured")
url = f"{self._get_worker_url()}/render-video"
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
payload = {"job_id": job_id, **render_config}
logger.info(f"[job:{job_id}] Submitting render-video job to GCE worker: {url}")
resp = await self._request_with_retry(
method="POST",
url=url,
headers=headers,
json_payload=payload,
timeout=30.0,
job_id=job_id,
)
if resp["status"] == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp["status"] == 409:
# Job already exists on worker — check if it already completed
logger.warning(f"[job:{job_id}] GCE worker returned 409, checking job status")
try:
status = await self.get_job_status(job_id)
job_status = status.get("status", "unknown")
if job_status == "complete":
logger.info(f"[job:{job_id}] Render-video job already complete on GCE worker, returning cached result")
return {
"status": "cached",
"job_id": job_id,
"output_files": status.get("output_files"),
"metadata": status.get("metadata"),
}
elif job_status in ("pending", "running"):
logger.info(f"[job:{job_id}] Render-video job still in progress on GCE worker")
return {"status": "in_progress", "job_id": job_id}
else:
raise RuntimeError(f"Render-video job {job_id} already exists with status: {job_status}")
except RuntimeError as e:
if "not found" in str(e).lower():
raise RuntimeError(f"Render-video job {job_id} conflict: 409 but job not found on status check")
raise
if resp["status"] != 200:
raise RuntimeError(f"Failed to submit render-video job: {resp['status']} - {resp['text']}")
return resp["json"]
async def render_video_on_gce(
self,
job_id: str,
render_config: Dict[str, Any],
progress_callback=None,
) -> Dict[str, Any]:
"""
Submit render-video job and wait for completion.
This is a convenience method that combines submit + wait.
Args:
job_id: Unique job identifier
render_config: Configuration dict for the render-video worker endpoint
progress_callback: Optional callback for progress updates
Returns:
Final job status with output files and metadata
"""
# Submit the job
submit_result = await self.submit_render_video_job(job_id, render_config)
# If cached, return immediately — rendering already done
submit_status = submit_result.get("status")
if submit_status == "cached":
logger.info(f"[job:{job_id}] Render-video already cached, returning immediately")
return {
"status": "complete",
"output_files": submit_result.get("output_files"),
"metadata": submit_result.get("metadata"),
}
# If in_progress, another request is rendering it — just wait for that
if submit_status == "in_progress":
logger.info(f"[job:{job_id}] Render-video already in progress, joining poll")
# Wait for completion
return await self.wait_for_completion(
job_id, progress_callback=progress_callback
)
async def health_check(self) -> Dict[str, Any]:
"""
Check the health of the encoding worker.
@@ -477,7 +706,7 @@ class EncodingService:
if not self.is_configured:
return {"status": "not_configured"}
url = f"{self._url}/health"
url = f"{self._get_worker_url()}/health"
headers = {"X-API-Key": self._api_key}
try:
@@ -499,4 +728,20 @@ def get_encoding_service() -> EncodingService:
global _encoding_service
if _encoding_service is None:
_encoding_service = EncodingService()
# Wire up worker manager for dynamic URL resolution
try:
from backend.services.encoding_worker_manager import EncodingWorkerManager
from google.cloud import compute_v1, firestore
settings = get_settings()
db = firestore.Client(project=settings.google_cloud_project)
compute_client = compute_v1.InstancesClient()
manager = EncodingWorkerManager(
db=db,
compute_client=compute_client,
project_id=settings.google_cloud_project,
)
_encoding_service.set_worker_manager(manager)
except Exception:
# Fallback to static URL if worker manager setup fails
pass
return _encoding_service
+239
View File
@@ -0,0 +1,239 @@
"""
Blue-Green Encoding Worker Manager.
Manages the lifecycle and role assignment of two GCE encoding worker VMs
(blue/green) via a Firestore config document as the single source of truth.
The config document at `config/encoding-worker` tracks which VM is primary
(actively serving encoding requests) and which is secondary (idle or being
deployed to). This enables zero-downtime deployments by swapping roles
after deploying new code to the secondary.
Usage:
from google.cloud import firestore
from google.cloud.compute_v1 import InstancesClient
db = firestore.Client()
compute = InstancesClient()
manager = EncodingWorkerManager(db, compute)
# Get current config
config = manager.get_config()
print(f"Primary: {config.primary_vm} at {config.primary_url}")
# Ensure primary VM is running before dispatching work
result = manager.ensure_primary_running()
"""
import logging
from dataclasses import dataclass
from datetime import datetime, UTC
from typing import Optional
from google.cloud import firestore
logger = logging.getLogger(__name__)
CONFIG_COLLECTION = "config"
CONFIG_DOCUMENT = "encoding-worker"
ENCODING_WORKER_PORT = 8080
@dataclass
class EncodingWorkerConfig:
"""Represents the Firestore config/encoding-worker document."""
primary_vm: str
primary_ip: str
primary_version: str
primary_deployed_at: Optional[str]
secondary_vm: str
secondary_ip: str
secondary_version: Optional[str]
secondary_deployed_at: Optional[str]
last_swap_at: Optional[str]
last_activity_at: Optional[str]
deploy_in_progress: bool
deploy_in_progress_since: Optional[str]
@property
def primary_url(self) -> str:
return f"http://{self.primary_ip}:{ENCODING_WORKER_PORT}"
@property
def secondary_url(self) -> str:
return f"http://{self.secondary_ip}:{ENCODING_WORKER_PORT}"
class EncodingWorkerManager:
"""Manages blue-green encoding worker VMs via Firestore config and Compute API."""
def __init__(self, db, compute_client=None, project_id="nomadkaraoke", zone="us-central1-c"):
self._db = db
self._compute = compute_client
self._project_id = project_id
self._zone = zone
def _doc_ref(self):
"""Return a reference to the config/encoding-worker document."""
return self._db.collection(CONFIG_COLLECTION).document(CONFIG_DOCUMENT)
# ------------------------------------------------------------------
# Firestore config operations (Task 1)
# ------------------------------------------------------------------
def get_config(self) -> EncodingWorkerConfig:
"""Read config/encoding-worker from Firestore and return as dataclass."""
snapshot = self._doc_ref().get()
if not snapshot.exists:
raise ValueError(
f"Encoding worker config document not found at "
f"{CONFIG_COLLECTION}/{CONFIG_DOCUMENT}"
)
data = snapshot.to_dict()
return EncodingWorkerConfig(
primary_vm=data["primary_vm"],
primary_ip=data["primary_ip"],
primary_version=data["primary_version"],
primary_deployed_at=data.get("primary_deployed_at"),
secondary_vm=data["secondary_vm"],
secondary_ip=data["secondary_ip"],
secondary_version=data.get("secondary_version"),
secondary_deployed_at=data.get("secondary_deployed_at"),
last_swap_at=data.get("last_swap_at"),
last_activity_at=data.get("last_activity_at"),
deploy_in_progress=data.get("deploy_in_progress", False),
deploy_in_progress_since=data.get("deploy_in_progress_since"),
)
def update_activity(self) -> None:
"""Update last_activity_at timestamp to record recent usage."""
now = datetime.now(UTC).isoformat()
self._doc_ref().update({"last_activity_at": now})
logger.info("Updated encoding worker last_activity_at to %s", now)
def swap_roles(
self,
new_primary_vm: str,
new_primary_ip: str,
new_primary_version: str,
old_primary_vm: str,
old_primary_ip: str,
old_primary_version: str,
) -> None:
"""Atomically swap primary and secondary roles using a Firestore transaction.
Args:
new_primary_vm: VM name that will become the new primary.
new_primary_ip: Internal IP of the new primary.
new_primary_version: Version deployed on the new primary.
old_primary_vm: Current primary VM name (becomes secondary).
old_primary_ip: Internal IP of the old primary.
old_primary_version: Version on the old primary.
"""
transaction = self._db.transaction()
doc_ref = self._doc_ref()
now = datetime.now(UTC).isoformat()
@firestore.transactional
def _swap_in_transaction(txn, ref):
txn.update(ref, {
"primary_vm": new_primary_vm,
"primary_ip": new_primary_ip,
"primary_version": new_primary_version,
"primary_deployed_at": now,
"secondary_vm": old_primary_vm,
"secondary_ip": old_primary_ip,
"secondary_version": old_primary_version,
"secondary_deployed_at": None,
"last_swap_at": now,
"deploy_in_progress": False,
"deploy_in_progress_since": None,
})
_swap_in_transaction(transaction, doc_ref)
logger.info(
"Swapped encoding worker roles: %s -> primary, %s -> secondary",
new_primary_vm, old_primary_vm,
)
# ------------------------------------------------------------------
# VM lifecycle operations (Task 2)
# ------------------------------------------------------------------
def get_vm_status(self, vm_name: str) -> str:
"""Get the current status of a GCE VM (e.g. RUNNING, TERMINATED, STAGING)."""
instance = self._compute.get(
project=self._project_id,
zone=self._zone,
instance=vm_name,
)
return instance.status
def start_vm(self, vm_name: str) -> None:
"""Start a VM if it is not already running or starting.
Fire-and-forget: returns immediately. Caller should poll get_vm_status
if they need to wait for the VM to be fully RUNNING.
"""
status = self.get_vm_status(vm_name)
if status in ("RUNNING", "STAGING"):
logger.info("VM %s is already %s, skipping start", vm_name, status)
return
logger.info("Starting VM %s (current status: %s)", vm_name, status)
self._compute.start(
project=self._project_id,
zone=self._zone,
instance=vm_name,
)
def stop_vm(self, vm_name: str) -> None:
"""Stop a VM if it is not already stopped or stopping."""
status = self.get_vm_status(vm_name)
if status in ("TERMINATED", "STOPPING"):
logger.info("VM %s is already %s, skipping stop", vm_name, status)
return
logger.info("Stopping VM %s (current status: %s)", vm_name, status)
self._compute.stop(
project=self._project_id,
zone=self._zone,
instance=vm_name,
)
def ensure_primary_running(self) -> dict:
"""Ensure the primary encoding worker VM is running.
Reads config, starts primary VM if stopped, and updates activity timestamp.
Returns:
dict with keys:
- started (bool): True if VM was started, False if already running
- vm_name (str): Name of the primary VM
- primary_url (str): URL of the primary encoding worker
"""
config = self.get_config()
vm_name = config.primary_vm
status = self.get_vm_status(vm_name)
started = False
if status not in ("RUNNING", "STAGING"):
logger.info("Primary VM %s is %s, starting it", vm_name, status)
self._compute.start(
project=self._project_id,
zone=self._zone,
instance=vm_name,
)
started = True
else:
logger.info("Primary VM %s is already %s", vm_name, status)
self.update_activity()
return {
"started": started,
"vm_name": vm_name,
"primary_url": config.primary_url,
}
@@ -0,0 +1 @@
"""Production error monitoring service for Nomad Karaoke."""
+177
View File
@@ -0,0 +1,177 @@
"""Configuration for the production error monitoring service.
All constants, service lists, and environment-driven configuration
functions used across the error monitor modules.
"""
import os
# ---------------------------------------------------------------------------
# GCP project settings
# ---------------------------------------------------------------------------
GCP_PROJECT: str = os.environ.get("GCP_PROJECT", "nomadkaraoke")
GCP_REGION: str = os.environ.get("GCP_REGION", "us-central1")
# ---------------------------------------------------------------------------
# Log querying settings
# ---------------------------------------------------------------------------
#: How far back to look for errors on each run (minutes).
LOOKBACK_MINUTES: int = 15
#: Maximum number of log entries to fetch per service per run.
MAX_LOG_ENTRIES: int = 500
# ---------------------------------------------------------------------------
# Monitored services
# ---------------------------------------------------------------------------
MONITORED_CLOUD_RUN_SERVICES: list[str] = [
"karaoke-backend",
"karaoke-decide",
"audio-separator",
# Gen2 Cloud Functions appear as Cloud Run services in logs
"gdrive-validator",
"github-runner-manager",
"backup-to-aws",
"divebar-mirror",
"kn-data-sync",
"divebar-lookup",
"encoding-worker-idle-shutdown",
]
MONITORED_CLOUD_RUN_JOBS: list[str] = [
"video-encoding-job",
"lyrics-transcription-job",
"audio-separation-job",
"audio-download-job",
]
# Gen2 Cloud Functions log as cloud_run_revision, not cloud_function.
# Their names are in MONITORED_CLOUD_RUN_SERVICES above.
MONITORED_CLOUD_FUNCTIONS: list[str] = []
MONITORED_GCE_INSTANCES: list[str] = [
"encoding-worker-a",
"encoding-worker-b",
"flacfetch-vm",
"divebar-sync-vm",
]
# ---------------------------------------------------------------------------
# Service dependency map
# Human-readable description of which services depend on which workers.
# Used as LLM context when generating root-cause analysis.
# ---------------------------------------------------------------------------
SERVICE_DEPENDENCY_MAP: str = """\
karaoke-backend (main API)
→ video-encoding-job (Cloud Run Job, triggered per order)
→ lyrics-transcription-job (Cloud Run Job, triggered after audio-sep)
→ audio-separation-job (Cloud Run Job, runs on Cloud Run GPU)
→ audio-download-job (Cloud Run Job, uses flacfetch-vm)
→ gdrive-validator (Cloud Function, validates output files)
→ encoding-worker-a/b (GCE VMs, long-running encoding workers)
karaoke-decide (catalog / discovery API)
→ kn_data_sync (Cloud Function, syncs KaraokeNerds catalog)
→ divebar_mirror (Cloud Function, mirrors Divebar catalog)
→ divebar_lookup (Cloud Function, single-song lookup)
audio-separator (Cloud Run service, GPU stem separation)
→ audio-separation-job (Cloud Run Job, launched by audio-separator)
flacfetch-vm (GCE VM)
→ audio-download-job (initiated by karaoke-backend)
divebar-sync-vm (GCE VM)
→ divebar_mirror (Cloud Function, reads from this VM)
Shared infrastructure:
→ runner_manager (Cloud Function, manages encoding worker lifecycle)
→ encoding_worker_idle (Cloud Function, shuts down idle encoding workers)
→ backup_to_aws (Cloud Function, daily GCS → S3 backup)
"""
# ---------------------------------------------------------------------------
# LLM analysis settings
# ---------------------------------------------------------------------------
LLM_ANALYSIS_MODEL: str = os.environ.get("LLM_ANALYSIS_MODEL", "gemini-2.0-flash-001")
LLM_VERTEX_LOCATION: str = os.environ.get("LLM_VERTEX_LOCATION", "us-central1")
#: Minimum number of active error patterns required before triggering LLM analysis.
MIN_PATTERNS_FOR_ANALYSIS: int = 3
# ---------------------------------------------------------------------------
# Spike detection
# ---------------------------------------------------------------------------
#: An error count is a spike if it exceeds SPIKE_MULTIPLIER × baseline average.
SPIKE_MULTIPLIER: float = 5.0
#: Minimum absolute error count before spike detection fires (avoids noise).
SPIKE_MIN_COUNT: int = 5
# ---------------------------------------------------------------------------
# Auto-resolution settings
# ---------------------------------------------------------------------------
#: Multiply LOOKBACK_MINUTES by this to get auto-resolve window in minutes.
AUTO_RESOLVE_MULTIPLIER: int = 8
#: Minimum hours before a pattern can be auto-resolved.
AUTO_RESOLVE_MIN_HOURS: int = 6
#: Maximum hours before a pattern is force-resolved even if still active.
AUTO_RESOLVE_MAX_HOURS: int = 168 # 1 week
#: Fallback resolve window (hours) if baseline data is unavailable.
AUTO_RESOLVE_FALLBACK_HOURS: int = 48
# ---------------------------------------------------------------------------
# Discord settings
# ---------------------------------------------------------------------------
#: Maximum Discord messages to send per monitor run (rate-limit guard).
MAX_DISCORD_MESSAGES_PER_RUN: int = 10
#: Discord API hard limit on message length.
DISCORD_MAX_MESSAGE_LENGTH: int = 2000
# ---------------------------------------------------------------------------
# Pattern tracking settings
# ---------------------------------------------------------------------------
#: Maximum number of active error patterns to track in Firestore.
MAX_ACTIVE_PATTERNS: int = 500
#: Rolling window used when computing error baselines.
ROLLING_WINDOW_DAYS: int = 7
#: Maximum length of a normalized error message stored in Firestore.
MAX_NORMALIZED_MESSAGE_LENGTH: int = 200
# ---------------------------------------------------------------------------
# Helper functions
# ---------------------------------------------------------------------------
def get_llm_enabled() -> bool:
"""Return True if LLM analysis is enabled via the LLM_ANALYSIS_ENABLED env var."""
return os.environ.get("LLM_ANALYSIS_ENABLED", "false").strip().lower() == "true"
def get_discord_webhook_secret_name() -> str:
"""Return the Secret Manager secret name for the Discord webhook URL."""
return os.environ.get("DISCORD_WEBHOOK_SECRET_NAME", "error-monitor-discord-webhook")
def get_digest_mode() -> bool:
"""Return True if the monitor should run in digest mode (batch alerts instead of immediate).
In digest mode all alerts are bundled into a single Discord message sent at
the end of the run, rather than sending one message per error pattern.
"""
return os.environ.get("DIGEST_MODE", "false").strip().lower() == "true"
+323
View File
@@ -0,0 +1,323 @@
"""Discord alert client for the production error monitoring service.
Provides formatting functions for each alert type, and an ErrorMonitorDiscord
webhook client that rate-limits outbound messages to MAX_DISCORD_MESSAGES_PER_RUN
per monitor run.
Usage:
from backend.services.error_monitor.discord import (
ErrorMonitorDiscord,
format_new_pattern_alert,
format_incident_alert,
format_auto_resolved_alert,
format_spike_alert,
format_daily_digest,
)
discord = ErrorMonitorDiscord(webhook_url="https://discord.com/api/webhooks/...")
msg = format_new_pattern_alert(...)
discord.send_message(msg)
"""
import logging
from datetime import datetime
from zoneinfo import ZoneInfo
import requests
from backend.services.error_monitor.config import (
DISCORD_MAX_MESSAGE_LENGTH,
MAX_DISCORD_MESSAGES_PER_RUN,
)
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Constants
# ---------------------------------------------------------------------------
SEVERITY_EMOJI: dict[str, str] = {
"P0": "🚨",
"P1": "🔴",
"P2": "⚠️",
"P3": "🟡",
}
_EASTERN = ZoneInfo("America/New_York")
# ---------------------------------------------------------------------------
# Private helpers
# ---------------------------------------------------------------------------
def _now_eastern() -> str:
"""Return the current Eastern time as 'HH:MM AM/PM ET' (DST-aware)."""
now = datetime.now(tz=_EASTERN)
return now.strftime("%I:%M %p ET").lstrip("0")
def _truncate(msg: str) -> str:
"""Truncate *msg* to DISCORD_MAX_MESSAGE_LENGTH characters."""
if len(msg) <= DISCORD_MAX_MESSAGE_LENGTH:
return msg
suffix = "…"
return msg[: DISCORD_MAX_MESSAGE_LENGTH - len(suffix)] + suffix
# ---------------------------------------------------------------------------
# Formatting functions
# ---------------------------------------------------------------------------
def format_new_pattern_alert(
service: str,
resource_type: str,
normalized_message: str,
sample_message: str,
count: int,
first_seen: str,
) -> str:
"""Format a Discord alert for a newly-detected error pattern.
Args:
service: Name of the service emitting the error.
resource_type: GCP resource type (e.g. 'cloud_run_service', 'gce_instance').
normalized_message: Normalised error pattern string.
sample_message: A representative raw log message.
count: Number of occurrences seen so far.
first_seen: ISO 8601 timestamp when the pattern was first seen.
Returns:
Formatted Discord message string, truncated to 2000 chars.
"""
lines = [
f"🔴 **New Error Pattern Detected**",
f"",
f"**Service:** {service}",
f"**Resource type:** {resource_type}",
f"**Pattern:** {normalized_message}",
f"**Sample:** {sample_message}",
f"**Count:** {count}",
f"**First seen:** {first_seen}",
f"**Alerted at:** {_now_eastern()}",
f"",
f"To investigate: `/prod-investigate {service} \"{normalized_message}\"`",
]
return _truncate("\n".join(lines))
def format_incident_alert(
title: str,
severity: str,
root_cause: str,
suggested_fix: str,
primary_service: str,
patterns: list[dict],
) -> str:
"""Format a Discord alert for an analysed incident (LLM-generated).
Args:
title: Short incident title.
severity: Severity level — one of P0, P1, P2, P3.
root_cause: Human-readable root-cause analysis.
suggested_fix: Recommended remediation steps.
primary_service: The main service involved in the incident.
patterns: List of dicts, each with keys: service, normalized_message, count.
Only the first 5 are shown; normalized_message is truncated to 80 chars.
Returns:
Formatted Discord message string, truncated to 2000 chars.
"""
emoji = SEVERITY_EMOJI.get(severity, "⚠️")
lines = [
f"{emoji} **[{severity}] {title}**",
f"",
f"**Primary service:** {primary_service}",
f"**Root cause:** {root_cause}",
f"**Suggested fix:** {suggested_fix}",
]
shown_patterns = patterns[:5]
if shown_patterns:
lines.append("")
lines.append(f"**Patterns ({len(shown_patterns)} of {len(patterns)}):**")
for p in shown_patterns:
svc = p.get("service", "unknown")
msg = p.get("normalized_message", "")[:80]
cnt = p.get("count", 0)
lines.append(f" • `{svc}` — {msg} ({cnt}x)")
lines.append("")
lines.append(f"**Alerted at:** {_now_eastern()}")
return _truncate("\n".join(lines))
def format_auto_resolved_alert(resolved_patterns: list[dict]) -> str:
"""Format a consolidated auto-resolved alert.
Args:
resolved_patterns: List of dicts, each with keys: service, message, hours_silent.
Only the first 8 are shown; message is truncated to 60 chars.
Returns:
Formatted Discord message string, truncated to 2000 chars.
"""
lines = [
f"✅ **Patterns Auto-Resolved**",
f"",
]
shown = resolved_patterns[:8]
if shown:
for p in shown:
svc = p.get("service", "unknown")
msg = p.get("message", "")[:60]
hours = p.get("hours_silent", 0)
lines.append(f" • `{svc}` — {msg} (silent {hours}h)")
else:
lines.append(" *(no patterns)*")
if len(resolved_patterns) > 8:
lines.append(f" … and {len(resolved_patterns) - 8} more")
lines.append("")
lines.append(f"**Resolved at:** {_now_eastern()}")
return _truncate("\n".join(lines))
def format_spike_alert(
service: str,
normalized_message: str,
current_count: int,
rolling_average: float,
) -> str:
"""Format a Discord alert for an error rate spike.
Args:
service: Name of the service.
normalized_message: Normalised error pattern string.
current_count: Error count in the current window.
rolling_average: Baseline rolling average count.
Returns:
Formatted Discord message string, truncated to 2000 chars.
"""
if rolling_average > 0:
multiplier = current_count / rolling_average
multiplier_str = f"{multiplier:.1f}x"
else:
multiplier_str = "∞x"
lines = [
f"⚠️ **Error Rate Spike Detected**",
f"",
f"**Service:** {service}",
f"**Pattern:** {normalized_message}",
f"**Current count:** {current_count}",
f"**Normal (rolling avg):** {rolling_average}",
f"**Multiplier:** {multiplier_str} above normal",
f"",
f"**Alerted at:** {_now_eastern()}",
]
return _truncate("\n".join(lines))
def format_daily_digest(
total_errors: int,
new_patterns: int,
resolved_patterns: int,
active_patterns: int,
per_service_counts: dict,
) -> str:
"""Format the daily 24h error digest.
Args:
total_errors: Total error count in the last 24 hours.
new_patterns: Number of new error patterns detected.
resolved_patterns: Number of patterns auto-resolved.
active_patterns: Total number of currently active patterns.
per_service_counts: Mapping of service name → error count.
Displayed sorted by count descending.
Returns:
Formatted Discord message string, truncated to 2000 chars.
"""
lines = [
f"📊 **Daily Error Digest (24h)**",
f"",
f"**Total errors:** {total_errors}",
f"**New patterns:** {new_patterns}",
f"**Resolved patterns:** {resolved_patterns}",
f"**Active patterns:** {active_patterns}",
]
if per_service_counts:
lines.append("")
lines.append("**Per-service breakdown:**")
sorted_services = sorted(per_service_counts.items(), key=lambda kv: kv[1], reverse=True)
for svc, cnt in sorted_services:
lines.append(f" • `{svc}`: {cnt}")
lines.append("")
lines.append(f"**Generated at:** {_now_eastern()}")
return _truncate("\n".join(lines))
# ---------------------------------------------------------------------------
# Discord webhook client
# ---------------------------------------------------------------------------
class ErrorMonitorDiscord:
"""Webhook client for sending error monitor alerts to Discord.
Enforces a per-run rate limit of MAX_DISCORD_MESSAGES_PER_RUN to avoid
flooding the channel during high-volume incident windows.
Attributes:
webhook_url: The Discord incoming webhook URL.
messages_sent: Counter of messages attempted this run (success or failure).
"""
def __init__(self, webhook_url: str) -> None:
self.webhook_url = webhook_url
self.messages_sent: int = 0
def send_message(self, content: str) -> bool:
"""Send a message to the configured Discord webhook.
Rate-limits: if messages_sent >= MAX_DISCORD_MESSAGES_PER_RUN, the
message is silently skipped and False is returned without incrementing
the counter.
Args:
content: The message text to send (should be pre-formatted and
pre-truncated to ≤2000 chars).
Returns:
True if the message was sent successfully, False otherwise.
"""
if self.messages_sent >= MAX_DISCORD_MESSAGES_PER_RUN:
logger.warning(
"Discord rate limit reached (%d messages sent this run); skipping message.",
self.messages_sent,
)
return False
try:
response = requests.post(
self.webhook_url,
json={"content": content, "flags": 4}, # SUPPRESS_EMBEDS
timeout=30,
)
response.raise_for_status()
logger.debug("Discord message sent successfully.")
return True
except Exception as exc:
logger.error("Failed to send Discord message: %s", exc)
return False
finally:
self.messages_sent += 1
@@ -0,0 +1,489 @@
"""Firestore adapter for the production error monitoring service.
Handles all CRUD for three Firestore collections:
- error_patterns: normalized error patterns (pattern_id as doc ID)
- error_incidents: groups of related patterns (incident_id as doc ID)
- discord_alerts: audit trail for Discord messages (alert_id as doc ID)
"""
from __future__ import annotations
import secrets
from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
from typing import Any
from backend.services.error_monitor.config import (
AUTO_RESOLVE_FALLBACK_HOURS,
AUTO_RESOLVE_MAX_HOURS,
AUTO_RESOLVE_MIN_HOURS,
AUTO_RESOLVE_MULTIPLIER,
ROLLING_WINDOW_DAYS,
)
# ---------------------------------------------------------------------------
# Data classes
# ---------------------------------------------------------------------------
@dataclass
class PatternData:
"""Input data for creating or updating an error pattern."""
pattern_id: str
service: str
resource_type: str
normalized_message: str
sample_message: str
count: int
timestamp: datetime
@dataclass
class UpsertResult:
"""Result of an upsert_pattern operation."""
pattern_id: str
is_new: bool
previous_status: str | None = None
# ---------------------------------------------------------------------------
# Collection names
# ---------------------------------------------------------------------------
_COL_PATTERNS = "error_patterns"
_COL_INCIDENTS = "error_incidents"
_COL_ALERTS = "discord_alerts"
# Statuses that can be queried as "active"
_ACTIVE_STATUSES = ["new", "acknowledged", "known"]
# Statuses from which a pattern is reactivated when the error recurs
_REACTIVATION_STATUSES = {"auto_resolved", "fixed"}
# Statuses eligible for auto-resolve checking
_AUTO_RESOLVE_ELIGIBLE = {"new", "acknowledged"}
def _utcnow() -> datetime:
"""Return current UTC datetime."""
return datetime.now(tz=timezone.utc)
def _iso(dt: datetime) -> str:
"""Serialise a datetime to ISO-8601 string."""
return dt.isoformat()
def _generate_id(prefix: str, fmt: str) -> str:
"""Generate a time-stamped ID with 8 hex random suffix.
Args:
prefix: e.g. "inc" or "alert"
fmt: strftime format string applied to utcnow(), e.g. "%Y%m%d_%H%M"
"""
ts = _utcnow().strftime(fmt)
suffix = secrets.token_hex(4) # 8 hex chars
return f"{prefix}_{ts}_{suffix}"
# ---------------------------------------------------------------------------
# Adapter
# ---------------------------------------------------------------------------
class ErrorPatternsAdapter:
"""Firestore CRUD adapter for error monitoring collections.
Args:
db: Optional Firestore client. If omitted, a default client for the
``nomadkaraoke`` GCP project is constructed on first use.
"""
def __init__(self, db=None) -> None:
if db is None:
from google.cloud import firestore # type: ignore[import]
db = firestore.Client(project="nomadkaraoke")
self._db = db
# ------------------------------------------------------------------
# Patterns — read
# ------------------------------------------------------------------
def get_pattern(self, pattern_id: str) -> dict | None:
"""Return the pattern document as a dict, or None if it doesn't exist.
Args:
pattern_id: Firestore document ID in the ``error_patterns`` collection.
"""
snap = self._db.collection(_COL_PATTERNS).document(pattern_id).get()
if not snap.exists:
return None
return snap.to_dict()
def get_active_patterns(self) -> list[dict]:
"""Return all patterns with status in [new, acknowledged, known]."""
query = self._db.collection(_COL_PATTERNS).where(
"status", "in", _ACTIVE_STATUSES
)
return [snap.to_dict() for snap in query.stream()]
def get_patterns_for_auto_resolve(self) -> list[dict]:
"""Return active patterns eligible for auto-resolution (new or acknowledged)."""
query = self._db.collection(_COL_PATTERNS).where(
"status", "in", list(_AUTO_RESOLVE_ELIGIBLE)
)
return [snap.to_dict() for snap in query.stream()]
# ------------------------------------------------------------------
# Patterns — upsert
# ------------------------------------------------------------------
def upsert_pattern(self, data: PatternData) -> UpsertResult:
"""Create a new pattern or update an existing one.
New pattern:
- status="new", severity="P3"
- Initialize ``rolling_counts`` with one entry for this run.
Existing pattern:
- Increment ``total_count`` and append to ``rolling_counts``.
- If status was ``auto_resolved`` or ``fixed`` → reactivate (set status="new").
- Prune ``rolling_counts`` entries older than ROLLING_WINDOW_DAYS.
Returns:
UpsertResult with ``is_new=True`` for brand-new patterns *and* for
reactivated patterns (auto_resolved → new, fixed → new).
"""
doc_ref = self._db.collection(_COL_PATTERNS).document(data.pattern_id)
snap = doc_ref.get()
ts_iso = _iso(data.timestamp)
new_rolling_entry = {"ts": ts_iso, "count": data.count}
if not snap.exists:
# ── brand new pattern ──────────────────────────────────────────
doc_ref.set(
{
"pattern_id": data.pattern_id,
"service": data.service,
"resource_type": data.resource_type,
"normalized_message": data.normalized_message,
"sample_message": data.sample_message,
"status": "new",
"severity": "P3",
"total_count": data.count,
"rolling_counts": [new_rolling_entry],
"first_seen": ts_iso,
"last_seen": ts_iso,
"alerted_at": None,
"created_at": ts_iso,
"updated_at": ts_iso,
}
)
return UpsertResult(pattern_id=data.pattern_id, is_new=True)
# ── existing pattern ───────────────────────────────────────────────
existing = snap.to_dict()
prev_status = existing.get("status", "new")
rolling_counts: list[dict] = list(existing.get("rolling_counts") or [])
rolling_counts.append(new_rolling_entry)
rolling_counts = self._prune_rolling_counts(rolling_counts)
update: dict[str, Any] = {
"total_count": existing.get("total_count", 0) + data.count,
"rolling_counts": rolling_counts,
"last_seen": ts_iso,
"sample_message": data.sample_message,
"updated_at": ts_iso,
}
# Reactivation: pattern was considered resolved/fixed but error recurred
is_new = False
if prev_status in _REACTIVATION_STATUSES:
update["status"] = "new"
update["alerted_at"] = None
is_new = True
doc_ref.update(update)
return UpsertResult(
pattern_id=data.pattern_id,
is_new=is_new,
previous_status=prev_status,
)
# ------------------------------------------------------------------
# Patterns — status changes
# ------------------------------------------------------------------
def auto_resolve_pattern(self, pattern_id: str, hours_silent: float) -> None:
"""Mark a pattern as auto-resolved.
Args:
pattern_id: Firestore document ID.
hours_silent: How many hours the pattern has been silent.
"""
self._db.collection(_COL_PATTERNS).document(pattern_id).update(
{
"status": "auto_resolved",
"auto_resolve_hours_silent": hours_silent,
"auto_resolved_at": _iso(_utcnow()),
"updated_at": _iso(_utcnow()),
}
)
def resolve_pattern(
self,
pattern_id: str,
pr_url: str | None = None,
note: str = "",
) -> None:
"""Mark a pattern as fixed (human-resolved).
Args:
pattern_id: Firestore document ID.
pr_url: Optional URL of the PR that fixed this pattern.
note: Optional human-readable note about the fix.
"""
now_iso = _iso(_utcnow())
self._db.collection(_COL_PATTERNS).document(pattern_id).update(
{
"status": "fixed",
"fixed_by": {
"pr_url": pr_url,
"note": note,
"fixed_at": now_iso,
},
"updated_at": now_iso,
}
)
def update_pattern_alerted(self, pattern_id: str) -> None:
"""Set the ``alerted_at`` timestamp on a pattern.
Called after a Discord alert has been successfully sent for the pattern.
"""
self._db.collection(_COL_PATTERNS).document(pattern_id).update(
{
"alerted_at": _iso(_utcnow()),
"updated_at": _iso(_utcnow()),
}
)
def merge_pattern(self, source_id: str, target_id: str, reason: str) -> None:
"""Merge *source* into *target* by muting the source pattern.
Args:
source_id: Pattern to mute.
target_id: Pattern that the source is being merged into.
reason: Human-readable rationale for the merge.
"""
self._db.collection(_COL_PATTERNS).document(source_id).update(
{
"status": "muted",
"merged_into": target_id,
"merge_reason": reason,
"updated_at": _iso(_utcnow()),
}
)
# ------------------------------------------------------------------
# Auto-resolve logic
# ------------------------------------------------------------------
def check_auto_resolve(self, pattern: dict) -> float | None:
"""Determine whether *pattern* should be auto-resolved.
Computes the frequency-aware silence threshold and compares it against
how long the pattern has been silent.
Args:
pattern: A pattern document dict (as returned by get_pattern).
Returns:
The number of hours the pattern has been silent if the threshold is
exceeded, or ``None`` if the pattern should not yet be resolved.
"""
status = pattern.get("status", "")
if status not in _AUTO_RESOLVE_ELIGIBLE:
return None
last_seen_str = pattern.get("last_seen")
if not last_seen_str:
return None
last_seen = datetime.fromisoformat(last_seen_str)
now = datetime.now(tz=timezone.utc)
# Ensure both datetimes are tz-aware for subtraction
if last_seen.tzinfo is None:
last_seen = last_seen.replace(tzinfo=timezone.utc)
hours_silent = (now - last_seen).total_seconds() / 3600.0
rolling_counts = pattern.get("rolling_counts") or []
threshold = self._compute_resolve_threshold(rolling_counts)
if hours_silent >= threshold:
return hours_silent
return None
def _compute_resolve_threshold(self, rolling_counts: list[dict]) -> float:
"""Compute the frequency-aware auto-resolve threshold in hours.
Algorithm:
1. If fewer than 3 data points → return AUTO_RESOLVE_FALLBACK_HOURS (48).
2. Compute mean interval between consecutive timestamps.
3. threshold = mean_interval_hours * AUTO_RESOLVE_MULTIPLIER.
4. Clamp result to [AUTO_RESOLVE_MIN_HOURS, AUTO_RESOLVE_MAX_HOURS].
Args:
rolling_counts: List of ``{"ts": ISO-string, "count": int}`` dicts.
Returns:
Threshold in hours (float).
"""
if len(rolling_counts) < 3:
return float(AUTO_RESOLVE_FALLBACK_HOURS)
# Parse timestamps and sort ascending
timestamps: list[datetime] = []
for entry in rolling_counts:
ts = datetime.fromisoformat(entry["ts"])
if ts.tzinfo is None:
ts = ts.replace(tzinfo=timezone.utc)
timestamps.append(ts)
timestamps.sort()
# Compute pairwise intervals
intervals_hours: list[float] = []
for i in range(1, len(timestamps)):
delta = (timestamps[i] - timestamps[i - 1]).total_seconds() / 3600.0
intervals_hours.append(delta)
if not intervals_hours:
return float(AUTO_RESOLVE_FALLBACK_HOURS)
mean_interval = sum(intervals_hours) / len(intervals_hours)
threshold = mean_interval * AUTO_RESOLVE_MULTIPLIER
# Clamp to configured bounds
threshold = max(float(AUTO_RESOLVE_MIN_HOURS), threshold)
threshold = min(float(AUTO_RESOLVE_MAX_HOURS), threshold)
return threshold
def _prune_rolling_counts(self, rolling_counts: list[dict]) -> list[dict]:
"""Remove rolling_count entries older than ROLLING_WINDOW_DAYS.
Args:
rolling_counts: List of ``{"ts": ISO-string, "count": int}`` dicts.
Returns:
Filtered list with only recent entries.
"""
if not rolling_counts:
return []
cutoff = _utcnow() - timedelta(days=ROLLING_WINDOW_DAYS)
result = []
for entry in rolling_counts:
ts = datetime.fromisoformat(entry["ts"])
if ts.tzinfo is None:
ts = ts.replace(tzinfo=timezone.utc)
if ts >= cutoff:
result.append(entry)
return result
# ------------------------------------------------------------------
# Incidents
# ------------------------------------------------------------------
def create_incident(
self,
title: str,
root_cause: str,
severity: str,
suggested_fix: str,
primary_service: str,
pattern_ids: list[str],
used_llm: bool,
) -> str:
"""Create a new incident document and link associated patterns.
Args:
title: Short human-readable title.
root_cause: LLM-generated or manual root-cause description.
severity: Severity string (e.g. "P1", "P2", "P3").
suggested_fix: Suggested remediation steps.
primary_service: The primary affected service.
pattern_ids: List of pattern_ids that belong to this incident.
used_llm: Whether LLM analysis was used to generate this incident.
Returns:
The generated incident_id string.
"""
incident_id = _generate_id("inc", "%Y%m%d_%H%M")
now_iso = _iso(_utcnow())
self._db.collection(_COL_INCIDENTS).document(incident_id).set(
{
"incident_id": incident_id,
"title": title,
"root_cause": root_cause,
"severity": severity,
"suggested_fix": suggested_fix,
"primary_service": primary_service,
"pattern_ids": pattern_ids,
"used_llm": used_llm,
"created_at": now_iso,
"updated_at": now_iso,
"status": "open",
}
)
# Link each pattern back to this incident
for pattern_id in pattern_ids:
self._db.collection(_COL_PATTERNS).document(pattern_id).update(
{
"incident_id": incident_id,
"updated_at": now_iso,
}
)
return incident_id
# ------------------------------------------------------------------
# Discord audit trail
# ------------------------------------------------------------------
def log_discord_alert(
self,
alert_type: str,
content: str,
success: bool,
metadata: dict | None = None,
) -> None:
"""Write an audit record to the ``discord_alerts`` collection.
Args:
alert_type: Category of alert (e.g. "new_pattern", "digest").
content: The message body that was (or would have been) sent.
success: Whether the Discord webhook call succeeded.
metadata: Optional additional data (pattern_ids, incident_id, etc.).
"""
alert_id = _generate_id("alert", "%Y%m%d_%H%M%S")
now_iso = _iso(_utcnow())
self._db.collection(_COL_ALERTS).document(alert_id).set(
{
"alert_id": alert_id,
"alert_type": alert_type,
"content": content,
"success": success,
"metadata": metadata or {},
"created_at": now_iso,
}
)
@@ -0,0 +1,106 @@
"""Known infrastructure noise patterns for the production error monitoring service.
Filters out transient, expected log messages so the error monitor is not flooded
with Cloud Run cold-start probes, deploy events, spot VM preemptions, and similar
operational noise.
Usage:
from backend.services.error_monitor.known_issues import should_ignore
reason = should_ignore(service, message)
if reason:
# skip this log entry — it's known infrastructure noise
...
"""
import re
from dataclasses import dataclass
@dataclass
class IgnoreReason:
"""Describes why a log entry should be suppressed.
Attributes:
pattern_name: Machine-readable name of the matching ignore pattern.
reason: Human-readable explanation of why this is infrastructure noise.
"""
pattern_name: str
reason: str
# ---------------------------------------------------------------------------
# Compiled ignore patterns (applied in order)
# All patterns use re.IGNORECASE for case-insensitive matching.
# ---------------------------------------------------------------------------
_IGNORE_PATTERNS: list[tuple[str, re.Pattern, str]] = [
(
"startup_probe",
re.compile(r"startup probe failed", re.IGNORECASE),
"Cloud Run cold start — startup probe failure is transient and expected",
),
(
"ready_condition",
re.compile(r"ready condition status changed", re.IGNORECASE),
"Cloud Run deploy event — readiness condition change is expected on deploy",
),
(
"spot_preemption",
re.compile(r"instance was preempted", re.IGNORECASE),
"Spot/preemptible VM preemption — expected behaviour for spot instances",
),
(
"idle_shutdown",
re.compile(r"stopping idle encoding worker", re.IGNORECASE),
"Encoding worker idle shutdown — expected behaviour when no jobs are queued",
),
(
"health_check_404",
re.compile(
r"(?:GET|HEAD)\b[^\n]*?/health[^\n]*?\b(?:404|not found)\b",
re.IGNORECASE,
),
"Load balancer health check returning 404 — transient during cold start",
),
(
"scheduler_retry",
re.compile(r"cloud scheduler.*retry", re.IGNORECASE),
"Cloud Scheduler transient retry — expected on intermittent job failures",
),
(
"runner_startup",
re.compile(r"github\.runner\..*(?:starting|stopping|idle)", re.IGNORECASE),
"GitHub Actions runner lifecycle event — expected during runner start/stop",
),
(
"container_shutdown",
re.compile(r"container called exit\(0\)", re.IGNORECASE),
"Container clean exit — exit code 0 indicates a graceful, expected shutdown",
),
]
def should_ignore(service: str, message: str) -> IgnoreReason | None:
"""Check whether a log entry matches a known infrastructure noise pattern.
The check is purely message-based; ``service`` is accepted for API symmetry
and future service-specific overrides but does not gate any pattern today.
Args:
service: The originating service name (e.g. ``"karaoke-backend"``).
message: The raw log message string to evaluate.
Returns:
An :class:`IgnoreReason` if the message matches a known noise pattern,
or ``None`` if the message should be processed normally.
"""
if not message:
return None
for pattern_name, pattern, reason in _IGNORE_PATTERNS:
if pattern.search(message):
return IgnoreReason(pattern_name=pattern_name, reason=reason)
return None
@@ -0,0 +1,429 @@
"""LLM-powered incident analysis and duplicate detection for the error monitor.
Uses Gemini Flash via the google-generativeai library to:
1. Group related error patterns into incidents with root-cause analysis.
2. Identify near-duplicate patterns that the regex normalizer missed.
Both functions fall back gracefully when the LLM is unavailable or returns
unparseable output.
"""
from __future__ import annotations
import json
import logging
import re
import time
from collections import defaultdict
from dataclasses import dataclass, field
from google import genai
from google.genai import types as genai_types
from backend.services.error_monitor.config import (
GCP_PROJECT,
LLM_ANALYSIS_MODEL,
LLM_VERTEX_LOCATION,
MIN_PATTERNS_FOR_ANALYSIS,
SERVICE_DEPENDENCY_MAP,
)
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Dataclasses
# ---------------------------------------------------------------------------
@dataclass
class Incident:
"""A grouped incident produced by LLM analysis or the fallback grouper."""
title: str
root_cause: str | None
severity: str # P0 / P1 / P2 / P3
suggested_fix: str | None
primary_service: str
pattern_indices: list[int]
@dataclass
class IncidentAnalysis:
"""The result of an incident grouping pass."""
incidents: list[Incident]
used_llm: bool = True
@dataclass
class DuplicateGroup:
"""A canonical pattern plus the new patterns that duplicate it."""
canonical_index: int
duplicate_indices: list[int]
reason: str
# ---------------------------------------------------------------------------
# LLM call helper
# ---------------------------------------------------------------------------
_RETRY_ATTEMPTS = 3
_RETRY_BASE_DELAY = 2.0 # seconds
def _call_llm(
system_prompt: str,
user_prompt: str,
temperature: float = 0.2,
) -> str:
"""Call Gemini and return the response text.
Retries up to _RETRY_ATTEMPTS times with exponential back-off on failure.
Args:
system_prompt: System instruction for the model.
user_prompt: User-facing prompt content.
temperature: Sampling temperature (lower = more deterministic).
Returns:
The model's response text.
Raises:
Exception: Re-raises the last exception after all retries are exhausted.
"""
last_exc: Exception | None = None
for attempt in range(_RETRY_ATTEMPTS):
try:
client = genai.Client(
vertexai=True,
project=GCP_PROJECT,
location=LLM_VERTEX_LOCATION,
)
response = client.models.generate_content(
model=LLM_ANALYSIS_MODEL,
contents=user_prompt,
config=genai_types.GenerateContentConfig(
system_instruction=system_prompt,
temperature=temperature,
),
)
return response.text
except Exception as exc: # noqa: BLE001
last_exc = exc
delay = _RETRY_BASE_DELAY * (2**attempt)
logger.warning(
"LLM call failed (attempt %d/%d): %s — retrying in %.1fs",
attempt + 1,
_RETRY_ATTEMPTS,
exc,
delay,
)
time.sleep(delay)
raise last_exc # type: ignore[misc]
# ---------------------------------------------------------------------------
# Response parsing
# ---------------------------------------------------------------------------
_MD_CODE_BLOCK_RE = re.compile(r"```(?:json)?\s*([\s\S]*?)```", re.IGNORECASE)
def _strip_markdown_code_block(text: str) -> str:
"""Remove surrounding ```json ... ``` or ``` ... ``` fences if present."""
match = _MD_CODE_BLOCK_RE.search(text)
if match:
return match.group(1).strip()
return text.strip()
def _parse_llm_response(text: str) -> IncidentAnalysis | None:
"""Parse the LLM response text into an IncidentAnalysis.
Args:
text: Raw LLM output, possibly wrapped in a markdown code block.
Returns:
IncidentAnalysis on success, None on any parse failure.
"""
if not text:
return None
cleaned = _strip_markdown_code_block(text)
try:
data = json.loads(cleaned)
except json.JSONDecodeError:
logger.warning("LLM returned non-JSON response (first 200 chars): %.200s", text)
return None
if "incidents" not in data:
logger.warning("LLM response missing 'incidents' key: %s", list(data.keys()))
return None
incidents: list[Incident] = []
for raw in data["incidents"]:
try:
incidents.append(
Incident(
title=raw.get("title", "Unknown incident"),
root_cause=raw.get("root_cause"),
severity=raw.get("severity", "P2"),
suggested_fix=raw.get("suggested_fix"),
primary_service=raw.get("primary_service", "unknown"),
pattern_indices=raw.get("pattern_indices", []),
)
)
except Exception as exc: # noqa: BLE001
logger.warning("Skipping malformed incident entry: %s — %s", raw, exc)
return IncidentAnalysis(incidents=incidents, used_llm=True)
# ---------------------------------------------------------------------------
# Fallback grouper
# ---------------------------------------------------------------------------
def _fallback_group_by_service(patterns: list[dict]) -> IncidentAnalysis:
"""Group patterns by service name when the LLM is unavailable.
Each unique service becomes one P2 incident containing all of that
service's pattern indices.
Args:
patterns: List of error pattern dicts (must include 'service' key).
Returns:
IncidentAnalysis with used_llm=False.
"""
service_to_indices: dict[str, list[int]] = defaultdict(list)
for idx, pattern in enumerate(patterns):
service = pattern.get("service", "unknown")
service_to_indices[service].append(idx)
incidents = [
Incident(
title=f"Errors in {service}",
root_cause=None,
severity="P2",
suggested_fix=None,
primary_service=service,
pattern_indices=indices,
)
for service, indices in service_to_indices.items()
]
return IncidentAnalysis(incidents=incidents, used_llm=False)
# ---------------------------------------------------------------------------
# Main analysis functions
# ---------------------------------------------------------------------------
_ANALYZE_SYSTEM_PROMPT = f"""\
You are an expert site-reliability engineer for Nomad Karaoke, a karaoke video
generation platform running on Google Cloud Platform.
Your task is to analyse a list of active error patterns detected across
production services, group them into meaningful incidents, and identify root
causes.
## Service dependency map
{SERVICE_DEPENDENCY_MAP}
## Severity definitions
- P0: Service is completely down / total outage
- P1: Major pipeline broken — orders cannot complete
- P2: Degraded performance — some orders affected
- P3: Minor / cosmetic issue with no direct user impact
## Output format
Respond with a single JSON object (no markdown fencing) with this schema:
{{
"incidents": [
{{
"title": "<short human-readable title>",
"root_cause": "<root cause or null>",
"severity": "P0|P1|P2|P3",
"suggested_fix": "<actionable suggestion or null>",
"primary_service": "<service name>",
"pattern_indices": [<list of 0-based indices from the input list>]
}}
]
}}
Group related patterns together into a single incident where they share a
common root cause (e.g., a downstream dependency failure). Unrelated patterns
should be in separate incidents. Every input pattern must appear in exactly
one incident.
"""
_DEDUP_SYSTEM_PROMPT = """\
You are an expert at identifying near-duplicate error patterns in production
monitoring systems.
Your task is to compare a list of NEW error patterns against a list of
EXISTING (canonical) patterns and identify which new patterns are essentially
duplicates of existing ones — i.e., they describe the same underlying error
but with slightly different wording due to variation in log messages.
## Output format
Respond with a single JSON object (no markdown fencing) with this schema:
{
"duplicates": [
{
"canonical_index": <0-based index into the EXISTING patterns list>,
"duplicate_indices": [<0-based indices into the NEW patterns list>],
"reason": "<brief explanation>"
}
]
}
If no duplicates are found, return {"duplicates": []}.
"""
def analyze_patterns(patterns: list[dict]) -> IncidentAnalysis | None:
"""Group error patterns into incidents using Gemini LLM analysis.
Returns None if there are fewer than MIN_PATTERNS_FOR_ANALYSIS patterns.
Falls back to service-based grouping when the LLM fails or returns an
unparseable response.
Args:
patterns: List of error pattern dicts to analyse.
Returns:
IncidentAnalysis, or None if insufficient patterns.
"""
if len(patterns) < MIN_PATTERNS_FOR_ANALYSIS:
return None
user_prompt = _build_analyze_user_prompt(patterns)
try:
raw_response = _call_llm(_ANALYZE_SYSTEM_PROMPT, user_prompt, temperature=0.2)
result = _parse_llm_response(raw_response)
if result is not None:
return result
logger.warning("LLM returned unparseable response; falling back to service grouping")
except Exception as exc: # noqa: BLE001
logger.warning("LLM analysis failed: %s — falling back to service grouping", exc)
return _fallback_group_by_service(patterns)
def find_duplicate_patterns(
new_patterns: list[dict],
existing_patterns: list[dict],
) -> list[DuplicateGroup]:
"""Identify near-duplicate patterns that the regex normalizer missed.
Returns an empty list if either input list is empty, or if the LLM call
fails for any reason.
Args:
new_patterns: Newly detected error patterns to check.
existing_patterns: Currently tracked (canonical) error patterns.
Returns:
List of DuplicateGroup objects (may be empty).
"""
if not new_patterns or not existing_patterns:
return []
user_prompt = _build_dedup_user_prompt(new_patterns, existing_patterns)
try:
raw_response = _call_llm(_DEDUP_SYSTEM_PROMPT, user_prompt, temperature=0.1)
return _parse_duplicate_response(raw_response)
except Exception as exc: # noqa: BLE001
logger.warning("Duplicate detection LLM call failed: %s", exc)
return []
# ---------------------------------------------------------------------------
# Prompt builders
# ---------------------------------------------------------------------------
def _build_analyze_user_prompt(patterns: list[dict]) -> str:
"""Build the user prompt listing patterns for incident grouping."""
lines = ["Error patterns to analyse:\n"]
for idx, pattern in enumerate(patterns):
service = pattern.get("service", "unknown")
message = pattern.get("normalized_message", "")
count = pattern.get("count", 0)
lines.append(f"[{idx}] service={service!r} count={count} message={message!r}")
return "\n".join(lines)
def _build_dedup_user_prompt(
new_patterns: list[dict],
existing_patterns: list[dict],
) -> str:
"""Build the user prompt listing existing and new patterns for dedup."""
lines = ["EXISTING (canonical) patterns:\n"]
for idx, pattern in enumerate(existing_patterns):
service = pattern.get("service", "unknown")
message = pattern.get("normalized_message", "")
lines.append(f"[{idx}] service={service!r} message={message!r}")
lines.append("\nNEW patterns to check:\n")
for idx, pattern in enumerate(new_patterns):
service = pattern.get("service", "unknown")
message = pattern.get("normalized_message", "")
lines.append(f"[{idx}] service={service!r} message={message!r}")
return "\n".join(lines)
def _parse_duplicate_response(text: str) -> list[DuplicateGroup]:
"""Parse the LLM response for duplicate detection.
Args:
text: Raw LLM output.
Returns:
List of DuplicateGroup objects, or [] on any failure.
"""
if not text:
return []
cleaned = _strip_markdown_code_block(text)
try:
data = json.loads(cleaned)
except json.JSONDecodeError:
logger.warning("Dedup LLM returned non-JSON: %.200s", text)
return []
if "duplicates" not in data:
logger.warning("Dedup LLM response missing 'duplicates' key")
return []
groups: list[DuplicateGroup] = []
for raw in data["duplicates"]:
try:
groups.append(
DuplicateGroup(
canonical_index=int(raw["canonical_index"]),
duplicate_indices=list(raw.get("duplicate_indices", [])),
reason=raw.get("reason", ""),
)
)
except Exception as exc: # noqa: BLE001
logger.warning("Skipping malformed duplicate entry: %s — %s", raw, exc)
return groups
+641
View File
@@ -0,0 +1,641 @@
"""Monitor orchestrator for the production error monitoring service.
This is the main entry point. Run as::
python -m backend.services.error_monitor.monitor
The ``ErrorMonitor.run()`` method executes the full pipeline:
* Monitor cycle (default) — query Cloud Logging, detect new patterns, send
Discord alerts, check auto-resolution.
* Daily digest mode (``DIGEST_MODE=true``) — bundle 24h stats into one message.
"""
from __future__ import annotations
import logging
import os
from datetime import datetime, timedelta, timezone
from typing import Any
from google.cloud import logging as cloud_logging
from google.cloud import secretmanager
from backend.services.error_monitor.config import (
GCP_PROJECT,
LOOKBACK_MINUTES,
MAX_LOG_ENTRIES,
MONITORED_CLOUD_FUNCTIONS,
MONITORED_CLOUD_RUN_JOBS,
MONITORED_CLOUD_RUN_SERVICES,
MONITORED_GCE_INSTANCES,
SPIKE_MIN_COUNT,
SPIKE_MULTIPLIER,
get_digest_mode,
get_discord_webhook_secret_name,
get_llm_enabled,
)
from backend.services.error_monitor.discord import (
ErrorMonitorDiscord,
format_auto_resolved_alert,
format_daily_digest,
format_incident_alert,
format_new_pattern_alert,
format_spike_alert,
)
from backend.services.error_monitor.firestore_adapter import (
ErrorPatternsAdapter,
PatternData,
UpsertResult,
)
from backend.services.error_monitor.known_issues import should_ignore
from backend.services.error_monitor.llm_analysis import analyze_patterns, find_duplicate_patterns
from backend.services.error_monitor.normalizer import compute_pattern_hash, normalize_message
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Module-level constants
# ---------------------------------------------------------------------------
#: Maps log resource type → (label_key, service_name_list)
RESOURCE_TYPE_FILTERS: dict[str, tuple[str, list[str]]] = {
"cloud_run_revision": ("resource.labels.service_name", MONITORED_CLOUD_RUN_SERVICES),
"cloud_run_job": ("resource.labels.job_name", MONITORED_CLOUD_RUN_JOBS),
"cloud_function": ("resource.labels.function_name", MONITORED_CLOUD_FUNCTIONS),
"gce_instance": ('labels."compute.googleapis.com/resource_name"', MONITORED_GCE_INSTANCES),
}
#: Maps log resource type → canonical resource_type string stored in Firestore.
RESOURCE_TYPE_MAP: dict[str, str] = {
"cloud_run_revision": "cloud_run_service",
"cloud_run_job": "cloud_run_job",
"cloud_function": "cloud_function",
"gce_instance": "gce_instance",
}
# ---------------------------------------------------------------------------
# Free functions
# ---------------------------------------------------------------------------
def _build_log_filter(
resource_type: str,
resource_names: list[str],
lookback_minutes: int,
) -> str:
"""Build a Cloud Logging filter string.
Args:
resource_type: GCP resource type (e.g. ``"cloud_run_revision"``).
resource_names: List of service/job/function/instance names to include.
lookback_minutes: How many minutes back to query.
Returns:
A filter string suitable for ``logging_client.list_entries(filter_=...)``.
"""
label_key = RESOURCE_TYPE_FILTERS[resource_type][0]
cutoff = datetime.now(tz=timezone.utc) - timedelta(minutes=lookback_minutes)
cutoff_iso = cutoff.strftime("%Y-%m-%dT%H:%M:%SZ")
name_clause = " OR ".join(f'{label_key}="{name}"' for name in resource_names)
if len(resource_names) > 1:
name_clause = f"({name_clause})"
return (
f'resource.type="{resource_type}" '
f"severity>=ERROR "
f'timestamp>="{cutoff_iso}" '
f"{name_clause}"
)
def _extract_message(entry: Any) -> str | None:
"""Extract the log message text from a Cloud Logging entry.
Handles several payload shapes:
* ``str`` payload — returned directly.
* ``dict`` payload — tries ``"message"``, ``"textPayload"``, ``"error"``,
``"msg"`` keys in that order.
* ``None`` payload — returns ``None``.
Args:
entry: A Cloud Logging ``LogEntry`` object.
Returns:
The message string, or ``None`` if not extractable.
"""
payload = entry.payload
if payload is None:
return None
if isinstance(payload, str):
return payload if payload else None
if isinstance(payload, dict):
for key in ("message", "textPayload", "error", "msg"):
value = payload.get(key)
if value:
return str(value)
return None
# Fallback: convert to string representation
text = str(payload)
return text if text else None
def _extract_service_name(entry: Any, resource_type: str) -> str:
"""Extract the service/job/function/instance name from a log entry.
Args:
entry: A Cloud Logging ``LogEntry`` object.
resource_type: GCP log resource type string.
Returns:
The service name string (falls back to ``"unknown"`` if not found).
"""
labels = getattr(entry.resource, "labels", {}) or {}
entry_labels = getattr(entry, "labels", {}) or {}
if resource_type == "cloud_run_revision":
return labels.get("service_name") or "unknown"
if resource_type == "cloud_run_job":
return labels.get("job_name") or "unknown"
if resource_type == "cloud_function":
return labels.get("function_name") or "unknown"
if resource_type == "gce_instance":
# Prefer the human-readable resource name label
name = entry_labels.get("compute.googleapis.com/resource_name")
if name:
return name
return labels.get("instance_id") or "unknown"
return "unknown"
def _classify_resource_type(log_resource_type: str) -> str:
"""Map a Cloud Logging resource type to the canonical type stored in Firestore.
Args:
log_resource_type: Raw GCP resource type (e.g. ``"cloud_run_revision"``).
Returns:
Canonical resource type string (falls back to the input if not in map).
"""
return RESOURCE_TYPE_MAP.get(log_resource_type, log_resource_type)
def _group_by_pattern(entries: list[dict]) -> dict[str, dict]:
"""Group normalised log entries by their pattern hash.
Args:
entries: List of dicts with ``service``, ``normalized``, and ``message`` keys.
Returns:
Dict mapping ``pattern_id`` → group dict containing ``service``,
``resource_type``, ``normalized``, ``sample_message``, ``count``.
"""
groups: dict[str, dict] = {}
for entry in entries:
service = entry["service"]
normalized = entry["normalized"]
pattern_id = compute_pattern_hash(service, normalized)
if pattern_id not in groups:
groups[pattern_id] = {
"pattern_id": pattern_id,
"service": service,
"resource_type": entry.get("resource_type", "unknown"),
"normalized": normalized,
"sample_message": entry["message"],
"count": 0,
}
groups[pattern_id]["count"] += 1
return groups
def _rolling_average(rolling_counts: list[dict]) -> float:
"""Compute the mean count from a list of rolling_count dicts.
Args:
rolling_counts: List of ``{"count": int, ...}`` dicts.
Returns:
The arithmetic mean as a float, or ``0.0`` for an empty list.
"""
if not rolling_counts:
return 0.0
total = sum(entry.get("count", 0) for entry in rolling_counts)
return total / len(rolling_counts)
def _is_spike(current_count: int, rolling_counts: list[dict]) -> bool:
"""Determine whether ``current_count`` represents an error rate spike.
A spike is detected when:
- ``current_count`` >= ``SPIKE_MIN_COUNT`` (avoids noise from rare errors), AND
- ``current_count`` > ``rolling_average * SPIKE_MULTIPLIER``
Args:
current_count: Error count in the current monitoring window.
rolling_counts: Historical rolling counts from Firestore.
Returns:
``True`` if a spike is detected, ``False`` otherwise.
"""
if not rolling_counts:
return False
if current_count < SPIKE_MIN_COUNT:
return False
avg = _rolling_average(rolling_counts)
if avg == 0.0:
return False
return current_count > avg * SPIKE_MULTIPLIER
# ---------------------------------------------------------------------------
# ErrorMonitor class
# ---------------------------------------------------------------------------
class ErrorMonitor:
"""Orchestrates the full production error monitoring pipeline.
On each run (typically every 15 minutes via Cloud Scheduler), it:
1. Queries Cloud Logging for recent errors across all monitored services.
2. Groups errors into patterns using normalisation + hashing.
3. Upserts patterns to Firestore, tracking new vs. recurrent.
4. Optionally uses an LLM to deduplicate patterns and create incidents.
5. Sends Discord alerts for new patterns, incidents, and spikes.
6. Auto-resolves patterns that have been silent long enough.
"""
def __init__(self) -> None:
self.logging_client = cloud_logging.Client(project=GCP_PROJECT)
self.firestore_adapter = ErrorPatternsAdapter()
self.discord = ErrorMonitorDiscord(webhook_url=self._get_discord_webhook())
def _get_discord_webhook(self) -> str:
"""Retrieve the Discord webhook URL.
Checks ``DISCORD_WEBHOOK_URL`` environment variable first; falls back
to Secret Manager.
Returns:
Discord webhook URL string.
"""
url = os.environ.get("DISCORD_WEBHOOK_URL")
if url:
return url
secret_name = get_discord_webhook_secret_name()
client = secretmanager.SecretManagerServiceClient()
secret_path = f"projects/{GCP_PROJECT}/secrets/{secret_name}/versions/latest"
response = client.access_secret_version(name=secret_path)
return response.payload.data.decode("utf-8").strip()
# ------------------------------------------------------------------
# Main entry point
# ------------------------------------------------------------------
def run(self) -> None:
"""Execute the monitor pipeline (digest or cycle mode)."""
if get_digest_mode():
self._run_daily_digest()
else:
self._run_monitor_cycle()
# ------------------------------------------------------------------
# Monitor cycle
# ------------------------------------------------------------------
def _run_monitor_cycle(self) -> None:
"""Execute one monitor cycle: collect → group → upsert → alert → auto-resolve."""
now = datetime.now(tz=timezone.utc)
now_iso = now.isoformat()
# ── Step 1: Collect log entries ────────────────────────────────────
raw_entries: list[dict] = []
for resource_type, (_, resource_names) in RESOURCE_TYPE_FILTERS.items():
log_filter = _build_log_filter(resource_type, resource_names, LOOKBACK_MINUTES)
logger.info("Querying %s with filter: %s", resource_type, log_filter)
try:
entries = self.logging_client.list_entries(
filter_=log_filter,
max_results=MAX_LOG_ENTRIES,
)
for entry in entries:
message = _extract_message(entry)
if not message:
continue
service = _extract_service_name(entry, resource_type)
ignore_reason = should_ignore(service, message)
if ignore_reason:
logger.debug(
"Ignoring entry from %s (%s): %s",
service,
ignore_reason.pattern_name,
ignore_reason.reason,
)
continue
normalized = normalize_message(message)
if not normalized:
continue
raw_entries.append(
{
"service": service,
"resource_type": _classify_resource_type(resource_type),
"normalized": normalized,
"message": message,
}
)
except Exception as exc: # noqa: BLE001
logger.error("Failed to query %s logs: %s", resource_type, exc)
logger.info("Collected %d log entries after filtering", len(raw_entries))
# ── Step 2: Group by pattern ───────────────────────────────────────
groups = _group_by_pattern(raw_entries)
logger.info("Grouped into %d distinct patterns", len(groups))
if not groups:
logger.info("No new error entries found this cycle; checking auto-resolve.")
self._check_auto_resolve()
return
# ── Step 3: Upsert to Firestore ────────────────────────────────────
new_pattern_ids: list[str] = []
spike_patterns: list[dict] = []
for pattern_id, group in groups.items():
data = PatternData(
pattern_id=pattern_id,
service=group["service"],
resource_type=group["resource_type"],
normalized_message=group["normalized"],
sample_message=group["sample_message"],
count=group["count"],
timestamp=now,
)
result: UpsertResult = self.firestore_adapter.upsert_pattern(data)
if result.is_new:
new_pattern_ids.append(pattern_id)
logger.info("New pattern detected: %s (%s)", pattern_id, group["service"])
# Check for spike on existing patterns
existing = self.firestore_adapter.get_pattern(pattern_id)
if existing and not result.is_new:
rolling = existing.get("rolling_counts") or []
if _is_spike(group["count"], rolling):
spike_patterns.append(
{
"service": group["service"],
"normalized_message": group["normalized"],
"current_count": group["count"],
"rolling_average": _rolling_average(rolling),
}
)
logger.info("Spike detected for pattern %s", pattern_id)
# ── Step 4: LLM dedup ─────────────────────────────────────────────
if new_pattern_ids and get_llm_enabled():
new_patterns_data = [
{
"service": groups[pid]["service"],
"normalized_message": groups[pid]["normalized"],
"count": groups[pid]["count"],
}
for pid in new_pattern_ids
]
existing_patterns = self.firestore_adapter.get_active_patterns()
# Exclude newly added patterns from the canonical list
canonical = [p for p in existing_patterns if p.get("pattern_id") not in new_pattern_ids]
if canonical:
duplicate_groups = find_duplicate_patterns(new_patterns_data, canonical)
for dup_group in duplicate_groups:
# Merge each duplicate new pattern into the canonical
canonical_pattern = canonical[dup_group.canonical_index]
canonical_id = canonical_pattern.get("pattern_id", "")
for dup_idx in dup_group.duplicate_indices:
if dup_idx < len(new_pattern_ids):
dup_id = new_pattern_ids[dup_idx]
logger.info(
"Merging duplicate pattern %s into %s: %s",
dup_id,
canonical_id,
dup_group.reason,
)
self.firestore_adapter.merge_pattern(dup_id, canonical_id, dup_group.reason)
# Remove from new_pattern_ids since it was merged
new_pattern_ids = [pid for pid in new_pattern_ids if pid != dup_id]
# ── Step 5: LLM incident analysis ─────────────────────────────────
incident_pattern_ids: set[str] = set()
if len(new_pattern_ids) >= 2 and get_llm_enabled():
patterns_for_analysis = [
{
"service": groups[pid]["service"],
"normalized_message": groups[pid]["normalized"],
"count": groups[pid]["count"],
}
for pid in new_pattern_ids
if pid in groups
]
analysis = analyze_patterns(patterns_for_analysis)
if analysis and analysis.incidents:
for incident in analysis.incidents:
incident_pids = [
new_pattern_ids[idx]
for idx in incident.pattern_indices
if idx < len(new_pattern_ids)
]
if not incident_pids:
continue
incident_id = self.firestore_adapter.create_incident(
title=incident.title,
root_cause=incident.root_cause or "",
severity=incident.severity,
suggested_fix=incident.suggested_fix or "",
primary_service=incident.primary_service,
pattern_ids=incident_pids,
used_llm=analysis.used_llm,
)
logger.info(
"Created incident %s (%s) with %d patterns",
incident_id,
incident.severity,
len(incident_pids),
)
# Build pattern list for the Discord alert
pattern_dicts = [
{
"service": groups[pid]["service"],
"normalized_message": groups[pid]["normalized"],
"count": groups[pid]["count"],
}
for pid in incident_pids
if pid in groups
]
msg = format_incident_alert(
title=incident.title,
severity=incident.severity,
root_cause=incident.root_cause or "Unknown",
suggested_fix=incident.suggested_fix or "Investigate logs",
primary_service=incident.primary_service,
patterns=pattern_dicts,
)
if self.discord.send_message(msg):
for pid in incident_pids:
try:
self.firestore_adapter.update_pattern_alerted(pid)
except Exception as exc: # noqa: BLE001
logger.warning("Could not update alerted_at for %s: %s", pid, exc)
incident_pattern_ids.update(incident_pids)
# ── Step 6: Individual alerts for new patterns not in incidents ────
for pattern_id in new_pattern_ids:
if pattern_id in incident_pattern_ids:
continue
if pattern_id not in groups:
continue
group = groups[pattern_id]
# Fetch fresh copy to get first_seen timestamp
existing = self.firestore_adapter.get_pattern(pattern_id)
first_seen = existing.get("first_seen", now_iso) if existing else now_iso
msg = format_new_pattern_alert(
service=group["service"],
resource_type=group["resource_type"],
normalized_message=group["normalized"],
sample_message=group["sample_message"],
count=group["count"],
first_seen=first_seen,
)
if self.discord.send_message(msg):
try:
self.firestore_adapter.update_pattern_alerted(pattern_id)
except Exception as exc: # noqa: BLE001
logger.warning("Could not update alerted_at for %s: %s", pattern_id, exc)
# ── Step 7: Spike alerts ───────────────────────────────────────────
for spike in spike_patterns:
msg = format_spike_alert(
service=spike["service"],
normalized_message=spike["normalized_message"],
current_count=spike["current_count"],
rolling_average=spike["rolling_average"],
)
self.discord.send_message(msg)
# ── Step 8: Auto-resolve check ─────────────────────────────────────
self._check_auto_resolve()
# ------------------------------------------------------------------
# Auto-resolve helper
# ------------------------------------------------------------------
def _check_auto_resolve(self) -> None:
"""Check all active patterns for auto-resolution and send a consolidated alert."""
patterns = self.firestore_adapter.get_patterns_for_auto_resolve()
resolved: list[dict] = []
for pattern in patterns:
pattern_id = pattern.get("pattern_id", "")
hours_silent = self.firestore_adapter.check_auto_resolve(pattern)
if hours_silent is not None:
logger.info(
"Auto-resolving pattern %s (silent %.1f hours)", pattern_id, hours_silent
)
self.firestore_adapter.auto_resolve_pattern(pattern_id, round(hours_silent, 1))
resolved.append(
{
"service": pattern.get("service", "unknown"),
"message": pattern.get("normalized_message", ""),
"hours_silent": round(hours_silent, 1),
}
)
if resolved:
msg = format_auto_resolved_alert(resolved)
self.discord.send_message(msg)
# ------------------------------------------------------------------
# Daily digest
# ------------------------------------------------------------------
def _run_daily_digest(self) -> None:
"""Gather 24h stats and send a daily digest to Discord."""
# Optional dedup sweep (best-effort; LLM errors are non-fatal)
if get_llm_enabled():
try:
active = self.firestore_adapter.get_active_patterns()
if len(active) >= 2:
mid = len(active) // 2
duplicates = find_duplicate_patterns(active[:mid], active[mid:])
for dup in duplicates:
for idx in dup.duplicate_indices:
target_half = active[mid:]
source_half = active[:mid]
if idx < len(target_half) and dup.canonical_index < len(source_half):
self.firestore_adapter.merge_pattern(
target_half[idx]["pattern_id"],
source_half[dup.canonical_index]["pattern_id"],
dup.reason,
)
except Exception as exc: # noqa: BLE001
logger.warning("Digest dedup sweep failed (non-fatal): %s", exc)
# Gather stats from active patterns
active_patterns = self.firestore_adapter.get_active_patterns()
per_service_counts: dict[str, int] = {}
new_count = 0
total_errors = 0
for pattern in active_patterns:
service = pattern.get("service", "unknown")
count = pattern.get("total_count", 0)
per_service_counts[service] = per_service_counts.get(service, 0) + count
total_errors += count
if pattern.get("status") == "new":
new_count += 1
# Count recently auto-resolved patterns
resolved_patterns = self.firestore_adapter.get_patterns_for_auto_resolve()
resolved_count = sum(
1
for p in resolved_patterns
if self.firestore_adapter.check_auto_resolve(p) is not None
)
msg = format_daily_digest(
total_errors=total_errors,
new_patterns=new_count,
resolved_patterns=resolved_count,
active_patterns=len(active_patterns),
per_service_counts=per_service_counts,
)
self.discord.send_message(msg)
# ---------------------------------------------------------------------------
# Entry point
# ---------------------------------------------------------------------------
def main() -> None:
"""Configure logging and run the error monitor."""
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
)
monitor = ErrorMonitor()
monitor.run()
if __name__ == "__main__":
main()
@@ -0,0 +1,137 @@
"""Message normalizer for the production error monitoring service.
Replaces variable parts of error messages (IDs, timestamps, paths, etc.)
with stable placeholders so the same logical error always produces the same
pattern hash, regardless of which specific resource or timestamp was involved.
Normalizer rules are applied in order — more specific patterns first.
"""
import hashlib
import re
from backend.services.error_monitor.config import MAX_NORMALIZED_MESSAGE_LENGTH
# ---------------------------------------------------------------------------
# Compiled regex patterns (applied in order — more specific first)
# ---------------------------------------------------------------------------
_NORMALIZERS: list[tuple[re.Pattern, str]] = [
# 1. GCS paths: gs://bucket/path/to/object → gs://<BUCKET>/<PATH>
# Must come before Firestore and URL normalization.
(
re.compile(r"gs://([^/\s]+)/([^\s]+)"),
r"gs://<BUCKET>/<PATH>",
),
# 1b. GCS bucket root (no object path): gs://bucket/ or gs://bucket
# Uses a lookahead so the slash/end-of-string is not consumed, and uses
# a character class that excludes '<' to avoid re-matching placeholders.
(
re.compile(r"gs://([^/<\s]+)(?=[/\s]|$)"),
r"gs://<BUCKET>",
),
# 2. Firestore document paths (collection/doc/…/doc — at least 2 segments)
# Negative lookbehind for '.' prevents matching inside domain names (URLs).
# Negative lookbehind for '/' prevents matching inside already-replaced paths.
(
re.compile(
r"(?<![./])\b([a-zA-Z_][a-zA-Z0-9_-]*/[a-zA-Z0-9_-]+(?:/[a-zA-Z0-9_-]+){1,})\b"
),
r"<DOC_PATH>",
),
# 3. UUIDs (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
(
re.compile(
r"\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\b"
),
"<ID>",
),
# 4. ISO 8601 timestamps (2024-01-15T10:30:45Z / …+05:30 / ….123Z)
(
re.compile(
r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:\d{2})"
),
"<TS>",
),
# 5. Epoch timestamps: 10+ digit number optionally followed by fractional seconds
(
re.compile(r"\b\d{10,}(?:\.\d+)?\b"),
"<EPOCH>",
),
# 6. Email addresses
(
re.compile(r"\b[A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}\b"),
"<EMAIL>",
),
# 7. IP addresses (IPv4) — must come before large-number replacement
(
re.compile(r"\b(?:\d{1,3}\.){3}\d{1,3}\b"),
"<IP>",
),
# 8. URLs — strip to domain only (https://host/path → https://host)
# Handles http and https; preserves the scheme+domain for context.
(
re.compile(r"https?://([A-Za-z0-9.\-]+(:\d+)?)/[^\s]*"),
r"https://\1",
),
# 9. Job IDs in paths (/jobs/<alphanumeric>)
(
re.compile(r"/jobs/[A-Za-z0-9]+"),
"/jobs/<ID>",
),
# 10. Firebase UIDs (20–28 mixed-case alphanumeric characters)
# Firebase Auth UIDs are typically 28 chars; allow 20-28 for flexibility.
# Must contain both letters and digits (to avoid replacing plain words).
(
re.compile(r"\b(?=[A-Za-z0-9]{20,28}\b)(?=[A-Za-z0-9]*[A-Z])(?=[A-Za-z0-9]*[a-z])(?=[A-Za-z0-9]*\d)[A-Za-z0-9]{20,28}\b"),
"<UID>",
),
# 11. Hex/alphanumeric IDs: 8+ chars containing both letters and digits
# (e.g. git SHAs, Cloud Run revision IDs, job IDs)
# Uses a lookahead to require at least one digit AND one letter.
(
re.compile(r"\b(?=[a-fA-F0-9]*[a-fA-F][a-fA-F0-9]*\d|[a-fA-F0-9]*\d[a-fA-F0-9]*[a-fA-F])[a-fA-F0-9]{8,}\b"),
"<ID>",
),
# 12. Large numbers (4+ digits) — must come after epoch/IP/UUID replacements
(
re.compile(r"\b\d{4,}\b"),
"<NUM>",
),
]
def normalize_message(message: str) -> str:
"""Normalize a raw log message by replacing variable parts with placeholders.
Applies each normalizer rule in order (more specific patterns first) and
truncates the result to MAX_NORMALIZED_MESSAGE_LENGTH characters.
Args:
message: Raw log message string.
Returns:
Normalized message string, at most MAX_NORMALIZED_MESSAGE_LENGTH chars.
"""
if not message:
return ""
result = message
for pattern, replacement in _NORMALIZERS:
result = pattern.sub(replacement, result)
return result[:MAX_NORMALIZED_MESSAGE_LENGTH]
def compute_pattern_hash(service: str, normalized_message: str) -> str:
"""Compute a stable SHA-256 fingerprint for an error pattern.
Args:
service: The originating service name (e.g. "karaoke-backend").
normalized_message: The output of normalize_message().
Returns:
64-character lowercase hex SHA-256 digest.
"""
payload = f"{service}:{normalized_message}"
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
+540 -3
View File
@@ -10,6 +10,8 @@ from google.cloud.firestore_v1 import FieldFilter
from backend.config import settings
from backend.models.job import Job, JobStatus, TimelineEvent
from backend.models.worker_log import WorkerLogEntry
from backend.models.review_session import ReviewSession
from backend.models.audio_edit_session import AudioEditSession
logger = logging.getLogger(__name__)
@@ -75,18 +77,20 @@ class FirestoreService:
status: JobStatus,
progress: Optional[int] = None,
message: Optional[str] = None,
timeline_metadata: Optional[Dict[str, Any]] = None,
**additional_fields
) -> None:
"""Update job status and add timeline event."""
try:
doc_ref = self.db.collection(self.collection).document(job_id)
# Create timeline event
timeline_event = TimelineEvent(
status=status.value,
timestamp=datetime.utcnow().isoformat(),
progress=progress,
message=message
message=message,
metadata=timeline_metadata,
)
# Prepare updates
@@ -153,7 +157,8 @@ class FirestoreService:
query = query.where(filter=FieldFilter('user_email', '==', user_email.lower()))
# Filter by tenant_id (white-label portal scoping)
if tenant_id:
# Always filter: "" for consumer jobs, "vocalstar"/etc for tenant jobs
if tenant_id is not None:
query = query.where(filter=FieldFilter('tenant_id', '==', tenant_id))
# Date range filters
@@ -173,6 +178,86 @@ class FirestoreService:
logger.error(f"Error listing jobs: {e}")
raise
# Fields needed by the dashboard summary view
SUMMARY_FIELD_PATHS = [
'job_id', 'status', 'progress', 'created_at', 'artist', 'title',
'error_message', 'non_interactive', 'outputs_deleted_at', 'user_email', 'is_private',
'url', 'filename', 'audio_search_artist', 'audio_search_title',
'audio_source_type', 'source_name', 'source_id', 'target_file', 'download_url',
'state_data.brand_code', 'state_data.youtube_url', 'state_data.dropbox_link',
'state_data.audio_progress', 'state_data.lyrics_progress',
'state_data.audio_complete', 'state_data.lyrics_complete',
'state_data.backing_vocals_analysis', 'state_data.visibility_change_in_progress',
'file_urls.finals', 'file_urls.videos', 'file_urls.packages',
'processing_metadata',
]
def list_jobs_summary(
self,
status: Optional[JobStatus] = None,
exclude_statuses: Optional[List[str]] = None,
environment: Optional[str] = None,
client_id: Optional[str] = None,
created_after: Optional[datetime] = None,
created_before: Optional[datetime] = None,
user_email: Optional[str] = None,
tenant_id: Optional[str] = None,
limit: int = 100,
) -> List[Dict[str, Any]]:
"""
List jobs with Firestore field projection for reduced payload.
Uses ``query.select()`` so Firestore only reads the requested fields,
drastically cutting bandwidth and deserialization cost.
Returns raw dicts (not Job models) because projected documents would
fail Pydantic validation for missing required fields.
"""
try:
query = self.db.collection(self.collection)
if status:
query = query.where(filter=FieldFilter('status', '==', status.value))
# NOTE: exclude_statuses is filtered in Python after the query
# because Firestore's not-in filter combined with order_by(created_at)
# requires a composite index. Since documents are tiny after select(),
# Python filtering is fast and avoids the index requirement.
if environment:
query = query.where(filter=FieldFilter('request_metadata.environment', '==', environment))
if client_id:
query = query.where(filter=FieldFilter('request_metadata.client_id', '==', client_id))
if user_email:
query = query.where(filter=FieldFilter('user_email', '==', user_email.lower()))
# Always filter: "" for consumer jobs, "vocalstar"/etc for tenant jobs
if tenant_id is not None:
query = query.where(filter=FieldFilter('tenant_id', '==', tenant_id))
if created_after:
query = query.where(filter=FieldFilter('created_at', '>=', created_after))
if created_before:
query = query.where(filter=FieldFilter('created_at', '<=', created_before))
query = query.order_by('created_at', direction=firestore.Query.DESCENDING).limit(limit)
query = query.select(self.SUMMARY_FIELD_PATHS)
docs = query.stream()
results = [doc.to_dict() for doc in docs]
# Apply exclude_statuses filter in Python (see note above)
if exclude_statuses:
results = [r for r in results if r.get('status') not in exclude_statuses]
return results
except Exception as e:
logger.error(f"Error listing jobs summary: {e}")
raise
def delete_jobs_by_filter(
self,
environment: Optional[str] = None,
@@ -516,6 +601,426 @@ class FirestoreService:
logger.error(f"Error listing tokens: {e}")
return []
# ============================================
# Search Session Methods
# ============================================
def create_search_session(self, session_data: dict) -> str:
"""Store a search session for the guided job creation flow.
Sessions are short-lived (TTL enforced via Firestore TTL policy) and
represent a completed audio search that hasn't yet been converted into
a job. The document is consumed (deleted) when create-from-search runs.
Args:
session_data: Dict with session_id, user_email, tenant_id, artist,
title, results, remote_search_id, ttl_expiry.
Returns:
session_id from session_data.
"""
try:
session_id = session_data['session_id']
doc_ref = self.db.collection('search_sessions').document(session_id)
doc_ref.set(session_data)
logger.info(f"Created search session {session_id}")
return session_id
except Exception as e:
logger.error(f"Error creating search session: {e}")
raise
def get_search_session(self, session_id: str) -> Optional[dict]:
"""Retrieve a search session by ID.
Returns None only if the document does not exist.
Raises on Firestore errors so callers can distinguish a missing session
from a transient datastore failure.
TTL-expired documents may still appear briefly; callers should check
ttl_expiry and treat expired sessions as missing.
Args:
session_id: UUID string identifying the session.
Returns:
Session dict or None.
"""
doc_ref = self.db.collection('search_sessions').document(session_id)
doc = doc_ref.get()
if not doc.exists:
return None
return doc.to_dict()
def consume_search_session(self, session_id: str) -> Optional[dict]:
"""Atomically read and delete a search session.
Uses a Firestore transaction to prevent two concurrent job-creation
requests from both reading the same session and creating duplicate jobs.
Returns the session dict if it existed, None if it did not exist.
Raises on Firestore errors.
Args:
session_id: UUID string identifying the session.
Returns:
Session dict or None.
"""
doc_ref = self.db.collection('search_sessions').document(session_id)
@firestore.transactional
def _read_and_delete(transaction, ref):
doc = ref.get(transaction=transaction)
if not doc.exists:
return None
session_data = doc.to_dict()
transaction.delete(ref)
return session_data
transaction = self.db.transaction()
return _read_and_delete(transaction, doc_ref)
# ============================================
# Review Session Subcollection Methods
# ============================================
# These methods store review session snapshots in a subcollection
# (jobs/{job_id}/review_sessions) for lyrics review backup/restore.
def save_review_session(self, job_id: str, session: ReviewSession) -> str:
"""
Save a review session to the subcollection.
Stores at: jobs/{job_id}/review_sessions/{session_id}
Args:
job_id: Job ID
session: ReviewSession instance
Returns:
session_id
"""
try:
session.job_id = job_id
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
doc_ref = sessions_ref.document(session.session_id)
doc_ref.set(session.to_dict())
logger.info(f"Saved review session {session.session_id} for job {job_id}")
return session.session_id
except Exception as e:
logger.error(f"Error saving review session for job {job_id}: {e}")
raise
def list_review_sessions(self, job_id: str) -> List[ReviewSession]:
"""
List all review sessions for a job, ordered by updated_at descending.
Args:
job_id: Job ID
Returns:
List of ReviewSession instances (metadata only, no correction_data)
"""
try:
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING)
docs = query.stream()
return [ReviewSession.from_dict(doc.to_dict()) for doc in docs]
except Exception as e:
logger.error(f"Error listing review sessions for job {job_id}: {e}")
return []
def get_review_session(self, job_id: str, session_id: str) -> Optional[ReviewSession]:
"""
Get a single review session by ID.
Args:
job_id: Job ID
session_id: Session ID
Returns:
ReviewSession or None
"""
try:
doc_ref = (
self.db.collection(self.collection)
.document(job_id)
.collection("review_sessions")
.document(session_id)
)
doc = doc_ref.get()
if not doc.exists:
return None
return ReviewSession.from_dict(doc.to_dict())
except Exception as e:
logger.error(f"Error getting review session {session_id} for job {job_id}: {e}")
raise
def delete_review_session(self, job_id: str, session_id: str) -> None:
"""
Delete a review session.
Args:
job_id: Job ID
session_id: Session ID
"""
try:
doc_ref = (
self.db.collection(self.collection)
.document(job_id)
.collection("review_sessions")
.document(session_id)
)
doc_ref.delete()
logger.info(f"Deleted review session {session_id} for job {job_id}")
except Exception as e:
logger.error(f"Error deleting review session {session_id} for job {job_id}: {e}")
raise
def delete_review_sessions_subcollection(self, job_id: str, batch_size: int = 500) -> int:
"""
Delete all review sessions for a job (used when deleting a job).
Args:
job_id: Job ID
batch_size: Number of documents to delete per batch
Returns:
Number of sessions deleted
"""
try:
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
deleted_count = 0
while True:
docs = sessions_ref.limit(batch_size).stream()
deleted_in_batch = 0
batch = self.db.batch()
for doc in docs:
batch.delete(doc.reference)
deleted_in_batch += 1
if deleted_in_batch == 0:
break
batch.commit()
deleted_count += deleted_in_batch
if deleted_in_batch < batch_size:
break
if deleted_count > 0:
logger.info(f"Deleted {deleted_count} review sessions for job {job_id}")
return deleted_count
except Exception as e:
logger.error(f"Error deleting review sessions subcollection for job {job_id}: {e}")
return 0
def get_latest_review_session_hash(self, job_id: str) -> Optional[str]:
"""
Get the data_hash of the most recent review session for deduplication.
Args:
job_id: Job ID
Returns:
data_hash string or None if no sessions exist
"""
try:
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
query = (
sessions_ref
.order_by("updated_at", direction=firestore.Query.DESCENDING)
.limit(1)
.select(["data_hash"])
)
docs = list(query.stream())
if docs:
return docs[0].to_dict().get("data_hash", "")
return None
except Exception as e:
logger.error(f"Error getting latest review session hash for job {job_id}: {e}")
return None
def search_review_sessions(
self,
query_text: str = "",
job_id: Optional[str] = None,
limit: int = 20,
) -> List[Dict[str, Any]]:
"""
Search review sessions across all jobs using collection group query.
Args:
query_text: Search text (prefix-matched against artist_lower or title_lower)
job_id: If provided, search only within this job
limit: Maximum results
Returns:
List of session dicts with job context
"""
try:
if job_id:
# Search within a specific job
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING).limit(limit)
docs = query.stream()
return [doc.to_dict() for doc in docs]
# Collection group query across all jobs
sessions_ref = self.db.collection_group("review_sessions")
if query_text:
# Over-fetch since we filter in Python (Firestore doesn't support LIKE/contains)
fetch_limit = max(limit * 25, 500)
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING).limit(fetch_limit)
docs = query.stream()
results = [doc.to_dict() for doc in docs]
q_lower = query_text.lower()
results = [
r for r in results
if q_lower in (r.get("artist") or "").lower()
or q_lower in (r.get("title") or "").lower()
or q_lower in (r.get("job_id") or "").lower()
]
return results[:limit]
else:
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING).limit(limit)
docs = query.stream()
return [doc.to_dict() for doc in docs]
except Exception as e:
logger.error(f"Error searching review sessions: {e}")
return []
# ---- Audio Edit Sessions ----
def save_audio_edit_session(self, job_id: str, session: AudioEditSession) -> str:
"""Save an audio edit session to the subcollection."""
try:
session.job_id = job_id
sessions_ref = self.db.collection(self.collection).document(job_id).collection("audio_edit_sessions")
doc_ref = sessions_ref.document(session.session_id)
doc_ref.set(session.to_dict())
logger.info(f"Saved audio edit session {session.session_id} for job {job_id}")
return session.session_id
except Exception as e:
logger.error(f"Error saving audio edit session for job {job_id}: {e}")
raise
def list_audio_edit_sessions(self, job_id: str) -> List[AudioEditSession]:
"""List all audio edit sessions for a job, ordered by updated_at descending."""
try:
sessions_ref = self.db.collection(self.collection).document(job_id).collection("audio_edit_sessions")
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING)
docs = query.stream()
return [AudioEditSession.from_dict(doc.to_dict()) for doc in docs]
except Exception as e:
logger.error(f"Error listing audio edit sessions for job {job_id}: {e}")
return []
def get_audio_edit_session(self, job_id: str, session_id: str) -> Optional[AudioEditSession]:
"""Get a single audio edit session by ID."""
try:
doc_ref = (
self.db.collection(self.collection)
.document(job_id)
.collection("audio_edit_sessions")
.document(session_id)
)
doc = doc_ref.get()
if not doc.exists:
return None
return AudioEditSession.from_dict(doc.to_dict())
except Exception as e:
logger.error(f"Error getting audio edit session {session_id} for job {job_id}: {e}")
raise
def delete_audio_edit_session(self, job_id: str, session_id: str) -> None:
"""Delete a single audio edit session."""
try:
doc_ref = (
self.db.collection(self.collection)
.document(job_id)
.collection("audio_edit_sessions")
.document(session_id)
)
doc_ref.delete()
logger.info(f"Deleted audio edit session {session_id} for job {job_id}")
except Exception as e:
logger.error(f"Error deleting audio edit session {session_id} for job {job_id}: {e}")
raise
def delete_audio_edit_sessions_subcollection(self, job_id: str, batch_size: int = 500) -> int:
"""Delete all audio edit sessions for a job (used when deleting a job)."""
try:
sessions_ref = self.db.collection(self.collection).document(job_id).collection("audio_edit_sessions")
deleted_count = 0
while True:
docs = sessions_ref.limit(batch_size).stream()
deleted_in_batch = 0
batch = self.db.batch()
for doc in docs:
batch.delete(doc.reference)
deleted_in_batch += 1
if deleted_in_batch == 0:
break
batch.commit()
deleted_count += deleted_in_batch
if deleted_in_batch < batch_size:
break
if deleted_count > 0:
logger.info(f"Deleted {deleted_count} audio edit sessions for job {job_id}")
return deleted_count
except Exception as e:
logger.error(f"Error deleting audio edit sessions subcollection for job {job_id}: {e}")
return 0
def get_latest_audio_edit_session_hash(self, job_id: str) -> Optional[str]:
"""Get the data_hash of the most recent audio edit session for deduplication."""
try:
sessions_ref = self.db.collection(self.collection).document(job_id).collection("audio_edit_sessions")
query = (
sessions_ref
.order_by("updated_at", direction=firestore.Query.DESCENDING)
.limit(1)
.select(["data_hash"])
)
docs = list(query.stream())
if docs:
return docs[0].to_dict().get("data_hash", "")
return None
except Exception as e:
logger.error(f"Error getting latest audio edit session hash for job {job_id}: {e}")
return None
def delete_search_session(self, session_id: str) -> None:
"""Delete a search session after it has been consumed by job creation.
Errors are logged but not re-raised — a missing or already-deleted
session is a no-op.
Args:
session_id: UUID string identifying the session.
"""
try:
doc_ref = self.db.collection('search_sessions').document(session_id)
doc_ref.delete()
logger.debug(f"Deleted search session {session_id}")
except Exception as e:
logger.error(f"Error deleting search session {session_id}: {e}")
# Singleton client instance
_firestore_client: Optional[firestore.Client] = None
@@ -537,3 +1042,35 @@ def get_firestore_client() -> firestore.Client:
_firestore_client = firestore.Client(project=settings.google_cloud_project)
return _firestore_client
_log_service: Optional["FirestoreService"] = None
def _get_log_service() -> "FirestoreService":
"""Return a cached FirestoreService instance for log_to_job."""
global _log_service
if _log_service is None:
_log_service = FirestoreService()
return _log_service
def log_to_job(job_id: str, worker: str, level: str, message: str, metadata: Optional[Dict[str, Any]] = None) -> None:
"""
Write a log entry to a job's Firestore log subcollection.
This is a convenience function for API endpoints that need to log
to a job's log history (workers already have JobLogAdapter for this).
Args:
job_id: Job ID to log to
worker: Worker/source name (e.g., "api", "edit", "admin")
level: Log level (INFO, WARNING, ERROR)
message: Log message
metadata: Optional structured metadata dict
"""
try:
entry = WorkerLogEntry.create(job_id, worker, level, message, metadata)
_get_log_service().append_log_to_subcollection(job_id, entry)
except Exception as e:
logger.debug(f"Failed to write job log for {job_id}: {e}")
+283 -84
View File
@@ -34,6 +34,8 @@ from typing import Any, Dict, List, Optional
import httpx
from backend.config import get_settings
logger = logging.getLogger(__name__)
@@ -42,6 +44,104 @@ class FlacfetchServiceError(Exception):
pass
def _format_httpx_error(e: Exception) -> str:
"""
Format an httpx exception into a descriptive error string.
httpx.RequestError subclasses (ConnectError, ReadError, etc.) often have
empty str() representations, which produces useless messages like
"Status request failed: ". This helper always returns something meaningful.
"""
error_str = str(e)
error_type = type(e).__name__
if error_str:
return f"{error_type}: {error_str}"
# str(e) is empty - try to include request URL for context
if isinstance(e, httpx.RequestError):
try:
request = e.request
if request:
return f"{error_type} for {request.method} {request.url}"
except RuntimeError:
pass # .request not set
return error_type
def should_retry_flacfetch_error(exception: Exception) -> bool:
"""
Determine if a flacfetch exception should be retried.
Retry on:
- httpx.RequestError (network failures: timeout, connection refused, etc.)
- httpx.HTTPStatusError with 5xx status (server errors)
Don't retry on:
- httpx.HTTPStatusError with 4xx status (client errors: bad request, auth, not found)
"""
if isinstance(exception, httpx.RequestError):
# Network-level failures - always retry
return True
if isinstance(exception, httpx.HTTPStatusError):
# HTTP errors - only retry on 5xx (server errors)
return exception.response.status_code >= 500
# Don't retry on other exceptions (e.g., FlacfetchServiceError after wrapping)
return False
async def with_retry(func, *args, **kwargs):
"""
Execute an async function with retry logic using exponential backoff.
This implementation uses manual retry logic instead of tenacity decorators
for better testability and runtime configuration.
"""
settings = get_settings()
max_attempts = settings.flacfetch_retry_max_attempts
min_wait = settings.flacfetch_retry_min_wait
max_wait = settings.flacfetch_retry_max_wait
last_exception = None
for attempt in range(1, max_attempts + 1):
try:
return await func(*args, **kwargs)
except Exception as e:
last_exception = e
# Check if we should retry this exception
if not should_retry_flacfetch_error(e):
# Don't retry - raise immediately
raise
# Check if we have more attempts left
if attempt >= max_attempts:
# No more attempts - raise the last exception
logger.warning(
f"Flacfetch request failed after {max_attempts} attempts: "
f"{_format_httpx_error(e) if isinstance(e, httpx.RequestError) else e}"
)
raise
# Calculate wait time with exponential backoff
wait_time = min(min_wait * (2 ** (attempt - 1)), max_wait)
logger.warning(
f"Flacfetch request failed (attempt {attempt}/{max_attempts}): "
f"{_format_httpx_error(e) if isinstance(e, httpx.RequestError) else e}. "
f"Retrying in {wait_time:.1f}s..."
)
# Wait before next attempt
await asyncio.sleep(wait_time)
# Should never reach here, but just in case
raise last_exception
class FlacfetchClient:
"""
Client for remote flacfetch HTTP API.
@@ -100,7 +200,7 @@ class FlacfetchClient:
return data
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Cannot reach flacfetch service: {e}")
raise FlacfetchServiceError(f"Cannot reach flacfetch service: {_format_httpx_error(e)}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Health check failed: {e.response.status_code}")
@@ -145,10 +245,10 @@ class FlacfetchClient:
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Search request failed: {e}")
raise FlacfetchServiceError(f"Search request failed: {_format_httpx_error(e)}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Search failed: {e.response.status_code} - {e.response.text}")
async def download(
self,
search_id: str,
@@ -158,47 +258,59 @@ class FlacfetchClient:
) -> str:
"""
Start downloading an audio file.
Retries automatically on network errors and server errors (5xx) using
exponential backoff. Configured via FLACFETCH_RETRY_* environment variables.
Args:
search_id: Search ID from previous search
result_index: Index of result to download
output_filename: Optional custom filename (without extension)
gcs_path: GCS path for upload (e.g., "uploads/job123/audio/")
Returns:
Download ID for tracking progress
Raises:
FlacfetchServiceError: On download start failure
FlacfetchServiceError: On download start failure (after all retries exhausted)
"""
try:
async with httpx.AsyncClient() as client:
payload = {
"search_id": search_id,
"result_index": result_index,
}
if output_filename:
payload["output_filename"] = output_filename
if gcs_path:
payload["upload_to_gcs"] = True
payload["gcs_path"] = gcs_path
resp = await client.post(
f"{self.base_url}/download",
headers=self._headers(),
json=payload,
timeout=self.timeout,
)
resp.raise_for_status()
data = resp.json()
return data["download_id"]
return await with_retry(self._download_impl, search_id, result_index, output_filename, gcs_path)
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Download request failed: {e}")
raise FlacfetchServiceError(f"Download request failed: {_format_httpx_error(e)}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Download start failed: {e.response.status_code} - {e.response.text}")
async def _download_impl(
self,
search_id: str,
result_index: int,
output_filename: Optional[str] = None,
gcs_path: Optional[str] = None,
) -> str:
"""Implementation of download without error wrapping - lets httpx exceptions bubble up."""
async with httpx.AsyncClient() as client:
payload = {
"search_id": search_id,
"result_index": result_index,
}
if output_filename:
payload["output_filename"] = output_filename
if gcs_path:
payload["upload_to_gcs"] = True
payload["gcs_path"] = gcs_path
resp = await client.post(
f"{self.base_url}/download",
headers=self._headers(),
json=payload,
timeout=self.timeout,
)
resp.raise_for_status()
data = resp.json()
return data["download_id"]
async def download_by_id(
self,
source_name: str,
@@ -214,6 +326,9 @@ class FlacfetchClient:
This is useful when you have stored the source_id from a previous search
and want to download later without re-searching.
Retries automatically on network errors and server errors (5xx) using
exponential backoff. Configured via FLACFETCH_RETRY_* environment variables.
Args:
source_name: Provider name (RED, OPS, YouTube, Spotify)
source_id: Source-specific ID (torrent ID, video ID, track ID)
@@ -226,52 +341,68 @@ class FlacfetchClient:
Download ID for tracking progress
Raises:
FlacfetchServiceError: On download start failure
FlacfetchServiceError: On download start failure (after all retries exhausted)
"""
try:
async with httpx.AsyncClient() as client:
payload = {
"source_name": source_name,
"source_id": source_id,
}
if output_filename:
payload["output_filename"] = output_filename
if target_file:
payload["target_file"] = target_file
if download_url:
payload["download_url"] = download_url
if gcs_path:
payload["upload_to_gcs"] = True
payload["gcs_path"] = gcs_path
resp = await client.post(
f"{self.base_url}/download-by-id",
headers=self._headers(),
json=payload,
timeout=self.timeout,
)
resp.raise_for_status()
data = resp.json()
return data["download_id"]
return await with_retry(
self._download_by_id_impl,
source_name, source_id, output_filename, target_file, download_url, gcs_path
)
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Download by ID request failed: {e}")
raise FlacfetchServiceError(f"Download by ID request failed: {_format_httpx_error(e)}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Download by ID failed: {e.response.status_code} - {e.response.text}")
async def _download_by_id_impl(
self,
source_name: str,
source_id: str,
output_filename: Optional[str] = None,
target_file: Optional[str] = None,
download_url: Optional[str] = None,
gcs_path: Optional[str] = None,
) -> str:
"""Implementation of download_by_id without error wrapping - lets httpx exceptions bubble up."""
async with httpx.AsyncClient() as client:
payload = {
"source_name": source_name,
"source_id": source_id,
}
if output_filename:
payload["output_filename"] = output_filename
if target_file:
payload["target_file"] = target_file
if download_url:
payload["download_url"] = download_url
if gcs_path:
payload["upload_to_gcs"] = True
payload["gcs_path"] = gcs_path
resp = await client.post(
f"{self.base_url}/download-by-id",
headers=self._headers(),
json=payload,
timeout=self.timeout,
)
resp.raise_for_status()
data = resp.json()
return data["download_id"]
async def get_download_status(self, download_id: str) -> Dict[str, Any]:
"""
Get the current status of a download.
Args:
download_id: Download ID from start_download
Returns:
Status dict with progress, speed, path info
Raises:
FlacfetchServiceError: On status check failure
httpx.RequestError: On network-level failures (connection refused, timeout, etc.)
These are NOT wrapped so wait_for_download can handle retries.
FlacfetchServiceError: On application-level failures (404, other HTTP errors)
"""
try:
async with httpx.AsyncClient() as client:
@@ -282,9 +413,10 @@ class FlacfetchClient:
)
resp.raise_for_status()
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Status request failed: {e}")
except httpx.RequestError:
# Let network errors bubble up so wait_for_download can handle retries
raise
except httpx.HTTPStatusError as e:
if e.response.status_code == 404:
raise FlacfetchServiceError(f"Download not found: {download_id}")
@@ -296,35 +428,59 @@ class FlacfetchClient:
timeout: int = 600,
poll_interval: float = 2.0,
progress_callback: Optional[callable] = None,
max_consecutive_errors: int = 3,
) -> Dict[str, Any]:
"""
Wait for a download to complete.
Tolerates transient network errors during status polling. A single
connection hiccup won't fail the entire download -- only N consecutive
poll failures will.
Args:
download_id: Download ID to wait for
timeout: Maximum wait time in seconds
poll_interval: Time between status checks
progress_callback: Optional callback(status_dict) for progress updates
max_consecutive_errors: Number of consecutive poll failures before giving up
Returns:
Final status dict with output_path or gcs_path
Raises:
FlacfetchServiceError: On download failure or timeout
FlacfetchServiceError: On download failure, timeout, or too many consecutive poll errors
"""
elapsed = 0.0
consecutive_errors = 0
while elapsed < timeout:
status = await self.get_download_status(download_id)
try:
status = await self.get_download_status(download_id)
consecutive_errors = 0 # Reset on success
except httpx.RequestError as e:
consecutive_errors += 1
error_desc = _format_httpx_error(e)
logger.warning(
f"Status poll failed for download {download_id} "
f"({consecutive_errors}/{max_consecutive_errors}): {error_desc}"
)
if consecutive_errors >= max_consecutive_errors:
raise FlacfetchServiceError(
f"Status polling failed after {consecutive_errors} consecutive "
f"network errors (last: {error_desc})"
)
await asyncio.sleep(poll_interval)
elapsed += poll_interval
continue
if progress_callback:
try:
progress_callback(status)
except Exception as e:
logger.warning(f"Progress callback error: {e}")
download_status = status.get("status")
if download_status in ["complete", "seeding"]:
logger.info(f"Download {download_id} complete: {status.get('gcs_path') or status.get('output_path')}")
return status
@@ -333,7 +489,7 @@ class FlacfetchClient:
raise FlacfetchServiceError(f"Download failed: {error}")
elif download_status == "cancelled":
raise FlacfetchServiceError("Download was cancelled")
# Log progress
progress = status.get("progress", 0)
speed = status.get("download_speed_kbps", 0)
@@ -342,10 +498,10 @@ class FlacfetchClient:
f"Download {download_id}: {progress:.1f}% "
f"({speed:.1f} KB/s, {peers} peers)"
)
await asyncio.sleep(poll_interval)
elapsed += poll_interval
raise FlacfetchServiceError(f"Download timed out after {timeout}s")
async def list_torrents(self) -> Dict[str, Any]:
@@ -366,7 +522,7 @@ class FlacfetchClient:
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"List torrents failed: {e}")
raise FlacfetchServiceError(f"List torrents failed: {_format_httpx_error(e)}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"List torrents failed: {e.response.status_code}")
@@ -397,7 +553,7 @@ class FlacfetchClient:
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Delete torrent failed: {e}")
raise FlacfetchServiceError(f"Delete torrent failed: {_format_httpx_error(e)}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Delete torrent failed: {e.response.status_code}")
@@ -431,7 +587,7 @@ class FlacfetchClient:
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Cleanup failed: {e}")
raise FlacfetchServiceError(f"Cleanup failed: {_format_httpx_error(e)}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Cleanup failed: {e.response.status_code}")
@@ -472,7 +628,7 @@ class FlacfetchClient:
return deleted
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Clear search cache request failed: {e}")
raise FlacfetchServiceError(f"Clear search cache request failed: {_format_httpx_error(e)}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"Clear search cache failed: {e.response.status_code} - {e.response.text}"
@@ -502,12 +658,55 @@ class FlacfetchClient:
return deleted_count
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Clear all cache request failed: {e}")
raise FlacfetchServiceError(f"Clear all cache request failed: {_format_httpx_error(e)}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"Clear all cache failed: {e.response.status_code} - {e.response.text}"
)
# =========================================================================
# YouTube Availability Check
# =========================================================================
async def check_youtube(self, url: str) -> Dict[str, Any]:
"""
Check if a YouTube video is available for download from this server's location.
Calls the flacfetch /check-youtube endpoint which uses yt-dlp to detect
geo-restrictions, private/removed videos, and other unavailability reasons.
Args:
url: YouTube video URL or video ID
Returns:
Dict with keys: available, video_id, title, error,
is_geo_restricted, is_age_restricted, is_private, is_removed
Raises:
FlacfetchServiceError: On request failure
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.post(
f"{self.base_url}/check-youtube",
headers=self._headers(),
json={"url": url},
timeout=15,
)
resp.raise_for_status()
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"YouTube availability check failed: {_format_httpx_error(e)}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"YouTube availability check failed: {e.response.status_code} - {e.response.text}"
)
# =========================================================================
# Cache Management
# =========================================================================
async def get_cache_stats(self) -> Dict[str, Any]:
"""
Get statistics about the cache.
@@ -529,7 +728,7 @@ class FlacfetchClient:
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Get cache stats request failed: {e}")
raise FlacfetchServiceError(f"Get cache stats request failed: {_format_httpx_error(e)}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"Get cache stats failed: {e.response.status_code} - {e.response.text}"
+141
View File
@@ -0,0 +1,141 @@
"""Service for generating personalized referral flyer PDFs."""
import logging
import os
import shutil
import subprocess
import tempfile
logger = logging.getLogger(__name__)
TEMPLATE_DIR = os.path.join(os.path.dirname(__file__), "..", "templates", "printables")
CHROMIUM_PATHS = [
"/usr/bin/chromium",
"/usr/bin/chromium-browser",
"/usr/bin/google-chrome",
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
]
VALID_THEMES = {"light", "dark"}
MAX_QR_DATA_URL_BYTES = 500_000
class FlyerError(Exception):
"""Error during flyer generation."""
class FlyerService:
"""Generates personalized referral flyer PDFs."""
def __init__(self):
self._chromium_path: str | None = None
def _get_chromium(self) -> str:
"""Lazily find Chromium binary. Only needed when generating PDFs."""
if self._chromium_path:
return self._chromium_path
for path in CHROMIUM_PATHS:
if os.path.exists(path):
self._chromium_path = path
return path
for name in ["chromium", "chromium-browser", "google-chrome"]:
found = shutil.which(name)
if found:
self._chromium_path = found
return found
raise FlyerError("Chromium not found. Install chromium to generate flyer PDFs.")
def _validate_qr_data_url(self, qr_data_url: str) -> None:
if not qr_data_url.startswith("data:image/"):
raise FlyerError("Invalid QR data URL: must be a data:image/ URL")
if len(qr_data_url) > MAX_QR_DATA_URL_BYTES:
raise FlyerError(f"QR data URL too large (>{MAX_QR_DATA_URL_BYTES} bytes)")
def _render_template(
self,
theme: str,
referral_code: str,
discount_percent: int,
qr_data_url: str,
) -> str:
if theme not in VALID_THEMES:
raise FlyerError(f"Invalid theme: {theme}. Must be one of: {VALID_THEMES}")
suffix = "-dark" if theme == "dark" else ""
template_path = os.path.join(TEMPLATE_DIR, f"website-referral-flyer{suffix}.html")
with open(template_path) as f:
html = f.read()
html = html.replace("{{REFERRAL_CODE}}", referral_code.upper())
html = html.replace("{{REFERRAL_CODE_LOWER}}", referral_code.lower())
html = html.replace("{{DISCOUNT_PERCENT}}", str(discount_percent))
html = html.replace("{{QR_DATA_URL}}", qr_data_url)
return html
def generate_pdf(
self,
theme: str,
referral_code: str,
discount_percent: int,
qr_data_url: str,
) -> bytes:
"""Generate a personalized flyer PDF. Returns raw PDF bytes."""
self._validate_qr_data_url(qr_data_url)
html = self._render_template(theme, referral_code, discount_percent, qr_data_url)
with tempfile.NamedTemporaryFile(
suffix=".html", delete=False, mode="w", encoding="utf-8"
) as tmp_html:
tmp_html.write(html)
tmp_html_path = tmp_html.name
tmp_pdf_path = tmp_html_path.replace(".html", ".pdf")
try:
cmd = [
self._get_chromium(),
"--headless",
f"--print-to-pdf={tmp_pdf_path}",
"--no-margins",
"--print-background",
"--no-pdf-header-footer",
"--virtual-time-budget=5000",
"--paper-width=8.5",
"--paper-height=11",
"--disable-gpu",
"--no-sandbox",
tmp_html_path,
]
result = subprocess.run(
cmd, capture_output=True, text=True, timeout=30
)
if result.returncode != 0:
logger.error("Chromium PDF generation failed: %s", result.stderr)
raise FlyerError(f"PDF generation failed: {result.stderr[:200]}")
if not os.path.exists(tmp_pdf_path):
raise FlyerError("PDF generation failed: no output file created")
with open(tmp_pdf_path, "rb") as f:
return f.read()
finally:
for path in [tmp_html_path, tmp_pdf_path]:
try:
os.unlink(path)
except OSError:
pass
_flyer_service = None
def get_flyer_service() -> FlyerService:
global _flyer_service
if _flyer_service is None:
_flyer_service = FlyerService()
return _flyer_service
+440 -6
View File
@@ -1,8 +1,11 @@
import asyncio
import glob as glob_module
import hashlib
import json
import logging
import os
import re
import shutil
import subprocess
import sys
import tempfile
@@ -13,6 +16,7 @@ from typing import Optional
from fastapi import FastAPI, BackgroundTasks, HTTPException, Header, Depends
from google.cloud import storage
from packaging.version import Version
from pydantic import BaseModel
logging.basicConfig(level=logging.INFO)
@@ -57,12 +61,27 @@ class EncodePreviewRequest(BaseModel):
font_gcs_path: Optional[str] = None # gs://bucket/path/to/custom-font.ttf
class RenderVideoRequest(BaseModel):
job_id: str
original_corrections_gcs_path: str
updated_corrections_gcs_path: Optional[str] = None
audio_gcs_path: str
style_params_gcs_path: Optional[str] = None
style_assets: Optional[dict] = None
output_gcs_prefix: str
artist: str
title: str
subtitle_offset_ms: int = 0
video_resolution: str = "4k"
class JobStatus(BaseModel):
job_id: str
status: str # pending, running, complete, failed
progress: int # 0-100
error: Optional[str] = None
output_files: Optional[list[str]] = None
metadata: Optional[dict] = None
def download_from_gcs(gcs_uri: str, local_path: Path):
@@ -134,6 +153,59 @@ def upload_single_file_to_gcs(local_path: Path, gcs_uri: str):
logger.info(f"Uploaded: {local_path} -> {gcs_uri}")
# Style asset disk cache directory
# Persists across jobs to avoid re-downloading the same theme assets (fonts, backgrounds)
STYLE_CACHE_DIR = Path("/var/cache/karaoke-gen/styles")
def download_with_cache(gcs_uri: str, local_path: Path, cache_dir: Optional[Path] = STYLE_CACHE_DIR):
"""Download a file from GCS, using a local disk cache to skip repeated downloads.
Cache key is SHA-256 of the GCS URI. Cache hits are validated by comparing
the cached file size against the GCS object size (via a lightweight metadata
call). On size mismatch the stale entry is replaced.
When cache_dir is None, downloads directly (no caching).
"""
if cache_dir is None:
download_single_file_from_gcs(gcs_uri, local_path)
return
cache_dir.mkdir(parents=True, exist_ok=True)
cache_key = hashlib.sha256(gcs_uri.encode()).hexdigest()
cached_path = cache_dir / cache_key
if cached_path.exists():
# Validate cached file against GCS object size
try:
parts = gcs_uri.replace("gs://", "").split("/", 1)
bucket_name = parts[0]
blob_name = parts[1] if len(parts) > 1 else ""
blob = storage_client.bucket(bucket_name).blob(blob_name)
blob.reload() # Fetch metadata (lightweight HEAD request)
gcs_size = blob.size
cached_size = cached_path.stat().st_size
if gcs_size is not None and cached_size != gcs_size:
logger.warning(
f"Cache stale for {gcs_uri}: cached={cached_size}B, GCS={gcs_size}B. Re-downloading."
)
cached_path.unlink()
else:
logger.info(f"Cache hit for {gcs_uri} -> {cached_path}")
shutil.copy2(str(cached_path), str(local_path))
return
except Exception as e:
# On transient GCS errors, serve from cache rather than failing the job
logger.warning(f"Cache validation failed for {gcs_uri}: {e}. Serving from cache.")
shutil.copy2(str(cached_path), str(local_path))
return
logger.info(f"Cache miss for {gcs_uri}, downloading...")
download_single_file_from_gcs(gcs_uri, local_path)
# Store in cache
shutil.copy2(str(local_path), str(cached_path))
def run_preview_encoding(job_id: str, work_dir: Path, request: "EncodePreviewRequest"):
# Run FFmpeg encoding for preview video (480x270, fast settings)
jobs[job_id]["status"] = "running"
@@ -237,6 +309,192 @@ def run_preview_encoding(job_id: str, work_dir: Path, request: "EncodePreviewReq
raise
def run_render_video(job_id: str, work_dir: Path, request: "RenderVideoRequest"):
"""Run video rendering using OutputGenerator from the karaoke-gen wheel.
Downloads corrections, audio, and style assets from GCS, runs OutputGenerator
to produce with_vocals.mkv and subtitle files, then uploads results.
Style assets use disk cache to avoid repeated downloads of the same theme.
This mirrors the logic in render_video_worker.py but runs on the GCE VM
instead of Cloud Run, providing more memory and CPU for ffmpeg/libass.
"""
jobs[job_id]["status"] = "running"
jobs[job_id]["progress"] = 5
try:
# Import from installed karaoke-gen wheel
from karaoke_gen.lyrics_transcriber.output.generator import OutputGenerator
from karaoke_gen.lyrics_transcriber.output.countdown_processor import CountdownProcessor
from karaoke_gen.lyrics_transcriber.types import CorrectionResult
from karaoke_gen.lyrics_transcriber.correction.operations import CorrectionOperations
from karaoke_gen.lyrics_transcriber.core.config import OutputConfig
from karaoke_gen.style_loader import load_styles_from_gcs
from karaoke_gen.utils import sanitize_filename
# 1. Download original corrections
corrections_path = work_dir / "corrections.json"
download_single_file_from_gcs(request.original_corrections_gcs_path, corrections_path)
jobs[job_id]["progress"] = 15
with open(corrections_path, 'r', encoding='utf-8') as f:
original_data = json.load(f)
base_result = CorrectionResult.from_dict(original_data)
# 2. Apply user corrections if available
if request.updated_corrections_gcs_path:
updated_path = work_dir / "corrections_updated.json"
download_single_file_from_gcs(request.updated_corrections_gcs_path, updated_path)
with open(updated_path, 'r', encoding='utf-8') as f:
updated_data = json.load(f)
if isinstance(updated_data, dict) and "corrections" in updated_data:
correction_result = CorrectionOperations.update_correction_result_with_data(
base_result, updated_data
)
logger.info(f"[job:{job_id}] Applied user corrections")
else:
logger.warning(f"[job:{job_id}] corrections_updated.json exists but has no 'corrections' key, using base result")
correction_result = base_result
else:
correction_result = base_result
jobs[job_id]["progress"] = 25
# 3. Download audio
audio_path = work_dir / "audio.flac"
download_single_file_from_gcs(request.audio_gcs_path, audio_path)
jobs[job_id]["progress"] = 35
# 4. Process countdown intro
countdown_processor = CountdownProcessor(cache_dir=str(work_dir), logger=logger)
correction_result, audio_path_str, padding_added, padding_seconds = countdown_processor.process(
correction_result=correction_result,
audio_filepath=str(audio_path),
)
audio_path = Path(audio_path_str)
jobs[job_id]["progress"] = 40
# 5. Download style assets (with disk cache for repeated themes)
def cached_download(gcs_path, local_path):
download_with_cache(gcs_path, Path(local_path), STYLE_CACHE_DIR)
styles_path, style_data = load_styles_from_gcs(
style_params_gcs_path=request.style_params_gcs_path,
style_assets=request.style_assets,
temp_dir=str(work_dir),
download_func=cached_download,
logger=logger,
)
# Register any downloaded fonts with fontconfig
for asset_key, gcs_path in (request.style_assets or {}).items():
if 'font' in asset_key.lower() and gcs_path.endswith(('.ttf', '.otf', '.woff', '.woff2')):
fonts_dir = Path("/usr/local/share/fonts/custom")
fonts_dir.mkdir(parents=True, exist_ok=True)
font_filename = gcs_path.split("/")[-1]
font_dest = fonts_dir / font_filename
if not font_dest.exists():
ext = os.path.splitext(gcs_path)[1]
style_font = work_dir / "style" / f"{asset_key}{ext}"
if style_font.exists():
shutil.copy2(str(style_font), str(font_dest))
subprocess.run(["fc-cache", "-fv"], capture_output=True)
logger.info(f"Registered font: {font_filename}")
jobs[job_id]["progress"] = 50
# 6. Configure and run OutputGenerator
output_dir = work_dir / "output"
cache_dir = work_dir / "cache"
output_dir.mkdir(exist_ok=True)
cache_dir.mkdir(exist_ok=True)
config = OutputConfig(
output_dir=str(output_dir),
cache_dir=str(cache_dir),
output_styles_json=styles_path,
render_video=True,
generate_cdg=False,
generate_plain_text=True,
generate_lrc=True,
video_resolution=request.video_resolution,
subtitle_offset_ms=request.subtitle_offset_ms,
)
output_generator = OutputGenerator(config, logger)
safe_artist = sanitize_filename(request.artist) if request.artist else "Unknown"
safe_title = sanitize_filename(request.title) if request.title else "Unknown"
output_prefix = f"{safe_artist} - {safe_title}"
logger.info(f"[job:{job_id}] Generating outputs with prefix '{output_prefix}'")
jobs[job_id]["progress"] = 55
outputs = output_generator.generate_outputs(
transcription_corrected=correction_result,
lyrics_results={},
output_prefix=output_prefix,
audio_filepath=str(audio_path),
artist=request.artist,
title=request.title,
)
jobs[job_id]["progress"] = 85
# 7. Upload outputs to GCS
output_prefix_gcs = request.output_gcs_prefix.rstrip("/")
output_files = []
if outputs.video and os.path.exists(outputs.video):
gcs_path = f"{output_prefix_gcs}/videos/with_vocals.mkv"
upload_single_file_to_gcs(Path(outputs.video), gcs_path)
output_files.append(gcs_path)
logger.info(f"[job:{job_id}] Uploaded with_vocals.mkv ({os.path.getsize(outputs.video)} bytes)")
else:
raise RuntimeError("Video generation failed - no output file produced")
if outputs.ass and os.path.exists(outputs.ass):
gcs_path = f"{output_prefix_gcs}/lyrics/karaoke.ass"
upload_single_file_to_gcs(Path(outputs.ass), gcs_path)
output_files.append(gcs_path)
if outputs.lrc and os.path.exists(outputs.lrc):
gcs_path = f"{output_prefix_gcs}/lyrics/karaoke.lrc"
upload_single_file_to_gcs(Path(outputs.lrc), gcs_path)
output_files.append(gcs_path)
if outputs.corrected_txt and os.path.exists(outputs.corrected_txt):
gcs_path = f"{output_prefix_gcs}/lyrics/corrected.txt"
upload_single_file_to_gcs(Path(outputs.corrected_txt), gcs_path)
output_files.append(gcs_path)
jobs[job_id]["progress"] = 95
jobs[job_id]["output_files"] = output_files
jobs[job_id]["metadata"] = {
"countdown_padding_added": padding_added,
"countdown_padding_seconds": padding_seconds if padding_added else 0,
}
logger.info(f"[job:{job_id}] Render video complete. Output files: {output_files}")
except ImportError as e:
error_msg = (
f"OutputGenerator not available: {e}. "
"The karaoke-gen wheel must be installed. "
"Check that ensure_latest_wheel() succeeded."
)
logger.error(error_msg)
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = error_msg
raise RuntimeError(error_msg) from e
except Exception as e:
logger.error(f"[job:{job_id}] Render video failed: {e}")
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = str(e)
raise
def ensure_latest_wheel():
'''Download and install latest karaoke-gen wheel from GCS.
@@ -254,12 +512,23 @@ def ensure_latest_wheel():
# Find the downloaded wheel (get the latest by version sorting)
wheels = glob_module.glob("/tmp/karaoke_gen-*.whl")
# Filter out karaoke_gen-current.whl (not a valid PEP 427 wheel name)
wheels = [w for w in wheels if '-current' not in w]
if not wheels:
logger.warning("No wheel found in GCS, using fallback encoding logic")
return False
# Sort to get latest version
wheel_path = sorted(wheels)[-1]
# Sort to get latest version (by semantic version, not alphabetically)
def extract_version(wheel_path):
"""Extract version from wheel filename like karaoke_gen-0.116.1-py3-none-any.whl"""
match = re.search(r'karaoke_gen-([0-9.]+)-', wheel_path)
if match:
return Version(match.group(1))
return Version("0.0.0") # Fallback for unparseable filenames
wheels.sort(key=extract_version, reverse=True)
wheel_path = wheels[0]
logger.info(f"Installing wheel: {wheel_path}")
# Install (or upgrade) the wheel
@@ -294,6 +563,43 @@ def find_file(work_dir: Path, *patterns):
return None
def generate_mov_from_png(png_path: Path, mov_path: Path, duration: int = 5) -> Path:
"""Generate a MOV video from a static PNG image using FFmpeg.
This runs on the GCE encoding worker (not Cloud Run) because 4K H.264
encoding requires more memory than the right-sized Cloud Run container
provides (2Gi). See PR #640 / #647.
Args:
png_path: Path to input PNG image
mov_path: Path for output MOV video
duration: Video duration in seconds (default 5)
Returns:
Path to generated MOV file
Raises:
RuntimeError: If FFmpeg fails
"""
command = [
"ffmpeg", "-hide_banner", "-loglevel", "error", "-nostats", "-y",
"-loop", "1", "-framerate", "30", "-i", str(png_path),
"-f", "lavfi", "-i", "anullsrc",
"-c:v", "libx264", "-r", "30", "-t", str(duration),
"-pix_fmt", "yuv420p", "-vf", "scale=3840:2160",
"-c:a", "aac", "-shortest",
str(mov_path),
]
logger.info(f"Generating MOV from PNG: {png_path.name} -> {mov_path.name}")
result = subprocess.run(command, capture_output=True, text=True, timeout=120)
if result.returncode != 0:
raise RuntimeError(f"FFmpeg failed to generate {mov_path.name}: {result.stderr}")
if not mov_path.exists() or mov_path.stat().st_size < 1024:
raise RuntimeError(f"FFmpeg produced invalid output for {mov_path.name}")
logger.info(f"Generated MOV: {mov_path.name} ({mov_path.stat().st_size} bytes)")
return mov_path
def run_encoding(job_id: str, work_dir: Path, config: dict):
'''Run encoding using LocalEncodingService (single source of truth).
@@ -316,14 +622,43 @@ def run_encoding(job_id: str, work_dir: Path, config: dict):
# Get artist/title from config for proper naming
artist = config.get("artist", "Unknown Artist")
title = config.get("title", "Unknown Title")
base_name = f"{artist} - {title}"
# Sanitize for use in file paths (e.g., title "PORTRAIT/WALK/BORN" → "PORTRAIT_WALK_BORN")
safe_artist = re.sub(r'[<>:"/\\|?*]', '_', artist)
safe_title = re.sub(r'[<>:"/\\|?*]', '_', title)
base_name = f"{safe_artist} - {safe_title}"
logger.info(f"Encoding for: {base_name}")
# Find input files in work_dir
# Title/end screens are in screens/ subdirectory
# Title/end screens: look for MOV first, fall back to generating from PNG.
# Since PR #640 right-sized Cloud Run to 2Gi, the screens_worker only generates
# PNG/JPG images — MOV video generation now happens here on the GCE worker.
title_video = find_file(work_dir, "screens/title.mov", "*Title*.mov", "*title*.mov")
end_video = find_file(work_dir, "screens/end.mov", "*End*.mov", "*end*.mov")
# Get intro/end video durations from style config (default 5s)
intro_duration = config.get("intro_video_duration", 5)
end_duration = config.get("end_video_duration", 5)
# Generate title MOV from PNG if missing or invalid (< 1KB = empty shell)
if not title_video or title_video.stat().st_size < 1024:
title_png = find_file(work_dir, "screens/title.png", "*Title*.png", "*title*.png")
if title_png:
title_mov_path = title_png.parent / "title.mov"
logger.info(f"Title MOV missing/invalid, generating from PNG: {title_png}")
title_video = generate_mov_from_png(title_png, title_mov_path, duration=intro_duration)
elif title_video:
logger.warning(f"Title MOV is only {title_video.stat().st_size} bytes and no PNG found")
# Generate end MOV from PNG if missing or invalid
if not end_video or (end_video and end_video.stat().st_size < 1024):
end_png = find_file(work_dir, "screens/end.png", "*End*.png", "*end*.png")
if end_png:
end_mov_path = end_png.parent / "end.mov"
logger.info(f"End MOV missing/invalid, generating from PNG: {end_png}")
end_video = generate_mov_from_png(end_png, end_mov_path, duration=end_duration)
elif end_video:
logger.warning(f"End MOV is only {end_video.stat().st_size} bytes and no PNG found")
# Karaoke video - search for With Vocals or main karaoke video
# IMPORTANT: Search order matters! Search specific paths first to avoid picking up
# old encoding outputs from finals/ directory when re-encoding after reset
@@ -357,9 +692,26 @@ def run_encoding(job_id: str, work_dir: Path, config: dict):
# Instrumental audio - respect user's selection from encoding config
instrumental_selection = config.get("instrumental_selection", "clean")
existing_instrumental = config.get("existing_instrumental")
logger.info(f"Instrumental selection from config: {instrumental_selection}")
if existing_instrumental:
logger.info(f"Existing instrumental from config: {existing_instrumental}")
if instrumental_selection == "with_backing":
if existing_instrumental:
# User-provided instrumental uploaded at job creation
# Downloaded to work_dir by process_job before run_encoding is called
instrumental = find_file(
work_dir,
"*existing_instrumental*", "*Instrumental User*",
)
elif instrumental_selection == "custom":
# Custom instrumental created via mute-region editing in review UI
instrumental = find_file(
work_dir,
"*custom_instrumental*.flac", "*Instrumental Custom*.flac",
"*custom_instrumental*.mp3",
)
elif instrumental_selection == "with_backing":
# User selected instrumental with backing vocals
instrumental = find_file(
work_dir,
@@ -472,6 +824,17 @@ async def process_job(job_id: str, request: EncodeRequest):
logger.info(f"Downloading from {request.input_gcs_path}")
download_from_gcs(request.input_gcs_path, work_dir)
# Download existing instrumental if present (stored under uploads/, not jobs/)
existing_instrumental = request.encoding_config.get("existing_instrumental") if request.encoding_config else None
if existing_instrumental:
# Extract bucket name from input_gcs_path (gs://bucket/jobs/...)
bucket_name = request.input_gcs_path.replace("gs://", "").split("/", 1)[0]
ext = Path(existing_instrumental).suffix.lower()
dest = work_dir / f"existing_instrumental{ext}"
gcs_uri = f"gs://{bucket_name}/{existing_instrumental}"
logger.info(f"Downloading existing instrumental: {gcs_uri} -> {dest}")
download_single_file_from_gcs(gcs_uri, dest)
# Run encoding in thread pool (CPU-bound)
loop = asyncio.get_event_loop()
output_dir = await loop.run_in_executor(
@@ -540,6 +903,36 @@ async def process_preview_job(job_id: str, request: EncodePreviewRequest):
jobs[job_id]["error"] = str(e)
async def process_render_video_job(job_id: str, request: RenderVideoRequest):
# Process a render-video job asynchronously
try:
# Download and install latest wheel at job start (allows hot updates without restart)
ensure_latest_wheel()
with tempfile.TemporaryDirectory() as temp_dir:
work_dir = Path(temp_dir) / "work"
work_dir.mkdir()
# Run render-video in thread pool (CPU-bound)
loop = asyncio.get_event_loop()
await loop.run_in_executor(
executor,
run_render_video,
job_id,
work_dir,
request
)
jobs[job_id]["status"] = "complete"
jobs[job_id]["progress"] = 100
logger.info(f"Render video job {job_id} complete")
except Exception as e:
logger.error(f"Render video job {job_id} failed: {e}")
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = str(e)
@app.post("/encode-preview")
async def submit_preview_encode_job(request: EncodePreviewRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
# Submit a preview encoding job
@@ -571,13 +964,54 @@ async def submit_preview_encode_job(request: EncodePreviewRequest, background_ta
return {"status": "accepted", "job_id": job_id}
@app.post("/render-video")
async def submit_render_video_job(request: RenderVideoRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
# Submit a render-video job
job_id = request.job_id
# If job already exists, return cached result or current status
if job_id in jobs:
existing_job = jobs[job_id]
if existing_job["status"] == "complete":
return {"status": "cached", "job_id": job_id, "output_files": existing_job.get("output_files"), "metadata": existing_job.get("metadata")}
elif existing_job["status"] == "failed":
# Previous attempt failed, allow retry by replacing the job
pass
else:
# Job is still in progress
return {"status": "in_progress", "job_id": job_id}
jobs[job_id] = {
"job_id": job_id,
"status": "pending",
"progress": 0,
"error": None,
"output_files": None,
"metadata": None,
}
background_tasks.add_task(process_render_video_job, job_id, request)
return {"status": "accepted", "job_id": job_id}
@app.post("/encode")
async def submit_encode_job(request: EncodeRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
# Submit an encoding job
job_id = request.job_id
# If job already exists, return cached result or current status
if job_id in jobs:
raise HTTPException(status_code=409, detail=f"Job {job_id} already exists")
existing_job = jobs[job_id]
if existing_job["status"] == "complete":
# Return cached result - encoding already done
return {"status": "cached", "job_id": job_id, "output_files": existing_job.get("output_files")}
elif existing_job["status"] == "failed":
# Previous attempt failed, allow retry by replacing the job
pass
else:
# Job is still in progress
return {"status": "in_progress", "job_id": job_id}
jobs[job_id] = {
"job_id": job_id,
+189 -66
View File
@@ -12,13 +12,19 @@ shared with YouTube credentials if scopes include drive.file.
import json
import logging
import os
import ssl
from typing import Any, Dict, Optional
from tenacity import Retrying, stop_after_attempt, wait_exponential, retry_if_exception_type
from backend.config import get_settings
from karaoke_gen.utils import sanitize_filename
logger = logging.getLogger(__name__)
# Transient errors that indicate a stale HTTP connection (Cloud Run idle containers)
TRANSIENT_ERRORS = (BrokenPipeError, ConnectionResetError, ConnectionError, ssl.SSLError)
class GoogleDriveService:
"""Google Drive operations using native API."""
@@ -33,6 +39,11 @@ class GoogleDriveService:
self._credentials_data: Optional[Dict[str, Any]] = None
self._loaded = False
def _reset_service(self):
"""Reset the cached Drive service to force a fresh connection on next use."""
logger.info("Resetting Google Drive service connection")
self._service = None
def _load_credentials(self) -> Optional[Dict[str, Any]]:
"""Load OAuth credentials from Secret Manager."""
if self._loaded:
@@ -127,6 +138,9 @@ class GoogleDriveService:
"""
Get existing folder or create new one, return folder ID.
Retries on transient connection errors (BrokenPipeError, SSLError)
that occur when Cloud Run containers sit idle between jobs.
Args:
parent_id: Parent folder ID
folder_name: Name of folder to find or create
@@ -134,33 +148,41 @@ class GoogleDriveService:
Returns:
Folder ID
"""
logger.info(f"Looking for folder '{folder_name}' in parent {parent_id}")
for attempt in Retrying(
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=2, max=8),
retry=retry_if_exception_type(TRANSIENT_ERRORS),
before_sleep=lambda retry_state: self._reset_service(),
reraise=True,
):
with attempt:
logger.info(f"Looking for folder '{folder_name}' in parent {parent_id}")
# Search for existing folder
# Escape single quotes in folder name for Google Drive API query syntax
escaped_folder_name = folder_name.replace("'", "\\'")
query = (
f"name='{escaped_folder_name}' and '{parent_id}' in parents "
f"and mimeType='application/vnd.google-apps.folder' and trashed=false"
)
results = self.service.files().list(q=query, fields="files(id, name)").execute()
# Search for existing folder
# Escape single quotes in folder name for Google Drive API query syntax
escaped_folder_name = folder_name.replace("'", "\\'")
query = (
f"name='{escaped_folder_name}' and '{parent_id}' in parents "
f"and mimeType='application/vnd.google-apps.folder' and trashed=false"
)
results = self.service.files().list(q=query, fields="files(id, name)").execute()
if results.get("files"):
folder_id = results["files"][0]["id"]
logger.info(f"Found existing folder '{folder_name}': {folder_id}")
return folder_id
if results.get("files"):
folder_id = results["files"][0]["id"]
logger.info(f"Found existing folder '{folder_name}': {folder_id}")
return folder_id
# Create folder
logger.info(f"Creating new folder '{folder_name}'")
metadata = {
"name": folder_name,
"mimeType": "application/vnd.google-apps.folder",
"parents": [parent_id],
}
folder = self.service.files().create(body=metadata, fields="id").execute()
folder_id = folder["id"]
logger.info(f"Created folder '{folder_name}': {folder_id}")
return folder_id
# Create folder
logger.info(f"Creating new folder '{folder_name}'")
metadata = {
"name": folder_name,
"mimeType": "application/vnd.google-apps.folder",
"parents": [parent_id],
}
folder = self.service.files().create(body=metadata, fields="id").execute()
folder_id = folder["id"]
logger.info(f"Created folder '{folder_name}': {folder_id}")
return folder_id
def upload_file(
self,
@@ -172,6 +194,9 @@ class GoogleDriveService:
"""
Upload a file to a specific Drive folder.
Retries on transient connection errors (BrokenPipeError, SSLError)
that occur when Cloud Run containers sit idle between jobs.
Args:
local_path: Local file path
parent_id: Parent folder ID in Google Drive
@@ -183,49 +208,57 @@ class GoogleDriveService:
"""
from googleapiclient.http import MediaFileUpload
file_size = os.path.getsize(local_path)
logger.info(
f"Uploading {local_path} ({file_size / 1024 / 1024:.1f} MB) "
f"as '{filename}' to folder {parent_id}"
)
for attempt in Retrying(
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=2, max=8),
retry=retry_if_exception_type(TRANSIENT_ERRORS),
before_sleep=lambda retry_state: self._reset_service(),
reraise=True,
):
with attempt:
file_size = os.path.getsize(local_path)
logger.info(
f"Uploading {local_path} ({file_size / 1024 / 1024:.1f} MB) "
f"as '{filename}' to folder {parent_id}"
)
# Determine MIME type
ext = os.path.splitext(local_path)[1].lower()
mime_types = {
".mp4": "video/mp4",
".mkv": "video/x-matroska",
".zip": "application/zip",
".flac": "audio/flac",
".mp3": "audio/mpeg",
".wav": "audio/wav",
".png": "image/png",
".jpg": "image/jpeg",
".jpeg": "image/jpeg",
}
mime_type = mime_types.get(ext, "application/octet-stream")
# Determine MIME type
ext = os.path.splitext(local_path)[1].lower()
mime_types = {
".mp4": "video/mp4",
".mkv": "video/x-matroska",
".zip": "application/zip",
".flac": "audio/flac",
".mp3": "audio/mpeg",
".wav": "audio/wav",
".png": "image/png",
".jpg": "image/jpeg",
".jpeg": "image/jpeg",
}
mime_type = mime_types.get(ext, "application/octet-stream")
# Check for existing file with same name
if replace_existing:
# Escape single quotes in filename for Google Drive API query syntax
escaped_filename = filename.replace("'", "\\'")
query = (
f"name='{escaped_filename}' and '{parent_id}' in parents and trashed=false"
)
results = self.service.files().list(q=query, fields="files(id)").execute()
for existing_file in results.get("files", []):
logger.info(f"Deleting existing file: {existing_file['id']}")
self.service.files().delete(fileId=existing_file["id"]).execute()
# Check for existing file with same name
if replace_existing:
# Escape single quotes in filename for Google Drive API query syntax
escaped_filename = filename.replace("'", "\\'")
query = (
f"name='{escaped_filename}' and '{parent_id}' in parents and trashed=false"
)
results = self.service.files().list(q=query, fields="files(id)").execute()
for existing_file in results.get("files", []):
logger.info(f"Deleting existing file: {existing_file['id']}")
self.service.files().delete(fileId=existing_file["id"]).execute()
# Upload file
metadata = {"name": filename, "parents": [parent_id]}
media = MediaFileUpload(local_path, mimetype=mime_type, resumable=True)
# Upload file
metadata = {"name": filename, "parents": [parent_id]}
media = MediaFileUpload(local_path, mimetype=mime_type, resumable=True)
file_result = (
self.service.files().create(body=metadata, media_body=media, fields="id").execute()
)
file_id = file_result["id"]
logger.info(f"Successfully uploaded '{filename}': {file_id}")
return file_id
file_result = (
self.service.files().create(body=metadata, media_body=media, fields="id").execute()
)
file_id = file_result["id"]
logger.info(f"Successfully uploaded '{filename}': {file_id}")
return file_id
def upload_to_public_share(
self,
@@ -308,6 +341,9 @@ class GoogleDriveService:
"""
Delete a file from Google Drive.
Retries on transient connection errors (BrokenPipeError, SSLError)
that occur when Cloud Run containers sit idle between jobs.
Args:
file_id: Google Drive file ID to delete
@@ -317,9 +353,17 @@ class GoogleDriveService:
logger.info(f"Deleting Google Drive file: {file_id}")
try:
self.service.files().delete(fileId=file_id).execute()
logger.info(f"Successfully deleted file: {file_id}")
return True
for attempt in Retrying(
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=2, max=8),
retry=retry_if_exception_type(TRANSIENT_ERRORS),
before_sleep=lambda retry_state: self._reset_service(),
reraise=True,
):
with attempt:
self.service.files().delete(fileId=file_id).execute()
logger.info(f"Successfully deleted file: {file_id}")
return True
except Exception as e:
# Check if it's a 404 (already deleted)
if hasattr(e, 'resp') and e.resp.status == 404:
@@ -343,6 +387,85 @@ class GoogleDriveService:
results[file_id] = self.delete_file(file_id)
return results
def find_files_by_brand_code(self, root_folder_id: str, brand_code: str) -> list[str]:
"""
Search for files in public share subfolders matching a brand code prefix.
Used as a fallback when gdrive_files is not tracked in state_data (e.g., old
jobs uploaded before file ID tracking was implemented, or jobs where GDrive
upload succeeded but file IDs were not saved due to a bug).
Searches CDG/, MP4/, MP4-720p/ subfolders for files whose name starts with
'{brand_code} - '.
Args:
root_folder_id: Root Google Drive folder ID for the public share
brand_code: Brand code prefix to search for (e.g., "NOMAD-1271")
Returns:
List of file IDs found matching the brand code
"""
found_ids = []
subfolders = ["CDG", "MP4", "MP4-720p"]
search_errors: list[Exception] = []
for subfolder in subfolders:
try:
# Find the subfolder
escaped_subfolder = subfolder.replace("'", "\\'")
query = (
f"name='{escaped_subfolder}' and '{root_folder_id}' in parents "
f"and mimeType='application/vnd.google-apps.folder' and trashed=false"
)
results = self.service.files().list(
q=query, fields="files(id, name)", supportsAllDrives=True,
includeItemsFromAllDrives=True
).execute()
if not results.get("files"):
logger.debug(f"Subfolder '{subfolder}' not found in {root_folder_id}")
continue
subfolder_id = results["files"][0]["id"]
# Search for files whose name starts with the brand code
# Use contains for efficiency, then filter by exact prefix
escaped_brand = brand_code.replace("'", "\\'")
file_query = (
f"name contains '{escaped_brand}' and '{subfolder_id}' in parents "
f"and trashed=false"
)
file_results = self.service.files().list(
q=file_query, fields="files(id, name)", supportsAllDrives=True,
includeItemsFromAllDrives=True
).execute()
for f in file_results.get("files", []):
if f["name"].startswith(f"{brand_code} - "):
found_ids.append(f["id"])
logger.info(
f"Found GDrive file to clean up in {subfolder}/: "
f"{f['name']} ({f['id']})"
)
except Exception as e:
logger.error(
f"Error searching for brand_code '{brand_code}' in '{subfolder}': {e}",
exc_info=True,
)
search_errors.append(e)
if search_errors:
# Raise so the caller knows the search was incomplete.
# This prevents the cleanup endpoint from treating an uncertain result
# as "GDrive is clean" and incorrectly recycling the brand code.
raise RuntimeError(
f"GDrive brand_code search incomplete: "
f"{len(search_errors)}/{len(subfolders)} subfolder(s) failed"
)
return found_ids
# Singleton instance
_gdrive_service: Optional[GoogleDriveService] = None
@@ -0,0 +1,62 @@
"""
GDrive Validator Cloud Function client.
Lightweight HTTP client for invoking the gdrive-validator Cloud Function
from the backend (e.g., after job completion). Uses OIDC authentication
via the backend's service account.
Fire-and-forget pattern: never blocks or fails the caller.
"""
import logging
import os
import requests
import google.auth.transport.requests
import google.oauth2.id_token
logger = logging.getLogger(__name__)
# Cloud Function URL - set via environment variable
GDRIVE_VALIDATOR_URL = os.environ.get("GDRIVE_VALIDATOR_URL", "")
def trigger_gdrive_validation() -> dict | None:
"""
Invoke the GDrive validator Cloud Function.
Uses the backend service account's OIDC token for authentication.
Returns:
Parsed JSON response from the function, or None on error.
"""
if not GDRIVE_VALIDATOR_URL:
logger.debug("GDRIVE_VALIDATOR_URL not set, skipping GDrive validation trigger")
return None
try:
# Get OIDC token for the Cloud Function URL
auth_request = google.auth.transport.requests.Request()
token = google.oauth2.id_token.fetch_id_token(auth_request, GDRIVE_VALIDATOR_URL)
response = requests.post(
GDRIVE_VALIDATOR_URL,
headers={"Authorization": f"Bearer {token}"},
timeout=60,
)
response.raise_for_status()
result = response.json()
status = result.get("status", "unknown")
if status == "issues_found":
logger.warning("GDrive validation found issues (notification sent)")
elif status == "ok":
logger.info("GDrive validation: all clear")
else:
logger.info(f"GDrive validation returned status: {status}")
return result
except Exception:
logger.exception("Failed to trigger GDrive validation (non-fatal)")
return None
+228
View File
@@ -0,0 +1,228 @@
"""
IP geolocation service with Firestore caching.
Looks up IP addresses via ip-api.com (free, no key needed) and caches
results in Firestore to avoid repeated lookups. Used by admin UI to
enrich IP displays with country, city, ISP, and ASN info.
"""
import ipaddress
import logging
import time
from datetime import datetime, timedelta
from typing import Optional, Dict, List
import httpx
from google.cloud import firestore
from backend.config import get_settings
logger = logging.getLogger(__name__)
IP_GEOLOCATION_COLLECTION = "ip_geolocation"
CACHE_TTL_DAYS = 90 # Cache results for 90 days
IP_API_URL = "http://ip-api.com/json"
IP_API_BATCH_URL = "http://ip-api.com/batch"
IP_API_FIELDS = "status,country,countryCode,regionName,city,isp,org,as,timezone,query"
RATE_LIMIT_PER_MINUTE = 45
def _is_private_ip(ip: str) -> bool:
"""Check if an IP is private/reserved (not routable on the internet)."""
try:
addr = ipaddress.ip_address(ip)
return addr.is_private or addr.is_reserved or addr.is_loopback or addr.is_link_local
except ValueError:
return True # Invalid IP
def _parse_ip_api_response(data: dict) -> dict:
"""Parse ip-api.com response into our standard format."""
if data.get("status") != "success":
return {
"status": "fail",
"ip": data.get("query", ""),
"cached_at": datetime.utcnow().isoformat(),
}
# Parse AS field: "AS14615 Comporium Inc" -> number + name
as_field = data.get("as", "")
as_number = ""
as_name = ""
if as_field:
parts = as_field.split(" ", 1)
as_number = parts[0] if parts else ""
as_name = parts[1] if len(parts) > 1 else ""
return {
"status": "success",
"ip": data.get("query", ""),
"country": data.get("country", ""),
"country_code": data.get("countryCode", ""),
"region": data.get("regionName", ""),
"city": data.get("city", ""),
"isp": data.get("isp", ""),
"org": data.get("org", ""),
"as_number": as_number,
"as_name": as_name,
"timezone": data.get("timezone", ""),
"cached_at": datetime.utcnow().isoformat(),
}
class IpGeolocationService:
"""IP geolocation lookup with Firestore caching."""
def __init__(self):
settings = get_settings()
self.db = firestore.Client(project=settings.google_cloud_project)
self._last_request_times: List[float] = []
def _check_rate_limit(self):
"""Simple rate limiter — sleep if approaching 45 req/min."""
now = time.time()
# Remove entries older than 60 seconds
self._last_request_times = [
t for t in self._last_request_times if now - t < 60
]
if len(self._last_request_times) >= RATE_LIMIT_PER_MINUTE - 5:
# Getting close to limit, wait
sleep_time = 60 - (now - self._last_request_times[0])
if sleep_time > 0:
logger.info(f"IP geolocation rate limit: sleeping {sleep_time:.1f}s")
time.sleep(sleep_time)
self._last_request_times.append(time.time())
def _get_cached(self, ip: str) -> Optional[dict]:
"""Get cached geolocation data for an IP."""
try:
# Firestore doc IDs can't contain '/' so IPv6 with :: is fine,
# but we need to sanitize for doc ID
doc_id = ip.replace("/", "_")
doc = self.db.collection(IP_GEOLOCATION_COLLECTION).document(doc_id).get()
if not doc.exists:
return None
data = doc.to_dict()
# Check TTL
cached_at = data.get("cached_at", "")
if cached_at:
try:
cached_dt = datetime.fromisoformat(cached_at)
if datetime.utcnow() - cached_dt > timedelta(days=CACHE_TTL_DAYS):
return None # Expired
except (ValueError, TypeError):
pass
return data
except Exception:
logger.exception(f"Error reading IP geo cache for {ip}")
return None
def _cache_result(self, ip: str, data: dict):
"""Store geolocation result in Firestore cache."""
try:
doc_id = ip.replace("/", "_")
self.db.collection(IP_GEOLOCATION_COLLECTION).document(doc_id).set(data)
except Exception:
logger.exception(f"Error caching IP geo for {ip}")
def lookup_ip(self, ip: str) -> dict:
"""
Look up geolocation for a single IP address.
Returns cached result if available, otherwise queries ip-api.com.
"""
if not ip or _is_private_ip(ip):
return {"status": "private", "ip": ip or ""}
# Check cache
cached = self._get_cached(ip)
if cached:
return cached
# Query ip-api.com
try:
self._check_rate_limit()
with httpx.Client(timeout=10) as client:
response = client.get(
f"{IP_API_URL}/{ip}",
params={"fields": IP_API_FIELDS},
)
response.raise_for_status()
data = response.json()
result = _parse_ip_api_response(data)
self._cache_result(ip, result)
return result
except Exception:
logger.exception(f"Error looking up IP {ip}")
return {"status": "error", "ip": ip}
def lookup_ips_batch(self, ips: List[str]) -> Dict[str, dict]:
"""
Look up geolocation for multiple IPs.
Returns dict of ip -> geo data. Uses cache for known IPs,
batches uncached lookups to ip-api.com.
"""
results = {}
uncached = []
for ip in set(ips): # Deduplicate
if not ip or _is_private_ip(ip):
results[ip] = {"status": "private", "ip": ip or ""}
continue
cached = self._get_cached(ip)
if cached:
results[ip] = cached
else:
uncached.append(ip)
if not uncached:
return results
# ip-api.com batch endpoint accepts up to 100 IPs per request
for i in range(0, len(uncached), 100):
batch = uncached[i:i + 100]
try:
self._check_rate_limit()
# Batch endpoint takes a JSON array of IPs
payload = [
{"query": ip, "fields": IP_API_FIELDS}
for ip in batch
]
with httpx.Client(timeout=15) as client:
response = client.post(IP_API_BATCH_URL, json=payload)
response.raise_for_status()
batch_results = response.json()
for data in batch_results:
result = _parse_ip_api_response(data)
ip = result.get("ip", data.get("query", ""))
if ip:
results[ip] = result
self._cache_result(ip, result)
except Exception:
logger.exception(f"Error in batch IP lookup ({len(batch)} IPs)")
# Mark failed IPs
for ip in batch:
if ip not in results:
results[ip] = {"status": "error", "ip": ip}
return results
# Global instance
_service = None
def get_ip_geolocation_service() -> IpGeolocationService:
"""Get the global IP geolocation service instance."""
global _service
if _service is None:
_service = IpGeolocationService()
return _service
+50
View File
@@ -61,6 +61,56 @@ def get_effective_distribution_settings(
)
def get_effective_distribution_for_job(job) -> EffectiveDistributionSettings:
"""
Get effective distribution settings for a job, applying private overrides if is_private=True.
For private (non-published) jobs, this overrides:
- dropbox_path → private Dropbox path (Tracks-NonPublished)
- brand_prefix → private brand prefix (NOMADNP)
- enable_youtube_upload → False (no YouTube for private)
- gdrive_folder_id → None (no Google Drive for private)
For non-private jobs, returns the job's own distribution settings unchanged.
Args:
job: Job object (or any object with distribution fields)
Returns:
EffectiveDistributionSettings with private overrides applied if needed
"""
if getattr(job, 'is_private', False):
# Tenant jobs: use the job's own distribution fields (set at creation from tenant config)
# Non-tenant private jobs: override to NonPublished paths
if getattr(job, 'tenant_id', None):
return EffectiveDistributionSettings(
dropbox_path=getattr(job, 'dropbox_path', None),
brand_prefix=getattr(job, 'brand_prefix', None),
enable_youtube_upload=False,
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=None,
)
settings = get_settings()
return EffectiveDistributionSettings(
dropbox_path=settings.default_private_dropbox_path,
brand_prefix=settings.default_private_brand_prefix,
enable_youtube_upload=False,
gdrive_folder_id=None,
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=None,
)
# Non-private: return job's own settings
return EffectiveDistributionSettings(
dropbox_path=getattr(job, 'dropbox_path', None),
brand_prefix=getattr(job, 'brand_prefix', None),
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=getattr(job, 'youtube_description_template', None),
)
def resolve_cdg_txt_defaults(
theme_id: Optional[str],
enable_cdg: Optional[bool] = None,
+262
View File
@@ -0,0 +1,262 @@
"""
Job health and consistency checking service.
Provides functions to detect state inconsistencies in jobs that might indicate
bugs in the state machine or trigger logic.
Created as part of state machine robustness improvements (2026-02-02).
"""
import logging
from datetime import datetime, timezone, timedelta
from typing import List, Dict, Any, Optional
from backend.models.job import Job, JobStatus
logger = logging.getLogger(__name__)
# Jobs in these statuses should NOT have audio_complete flag set
# (unless they're actively processing or beyond)
STATUSES_BEFORE_AUDIO_START = {
JobStatus.PENDING,
JobStatus.SEARCHING_AUDIO,
JobStatus.AWAITING_AUDIO_SELECTION,
JobStatus.AWAITING_AUDIO_EDIT,
JobStatus.IN_AUDIO_EDIT,
JobStatus.AUDIO_EDIT_COMPLETE,
}
# Jobs in these statuses should NOT have lyrics_complete flag set
STATUSES_BEFORE_LYRICS_START = {
JobStatus.PENDING,
JobStatus.SEARCHING_AUDIO,
JobStatus.AWAITING_AUDIO_SELECTION,
JobStatus.AWAITING_AUDIO_EDIT,
JobStatus.IN_AUDIO_EDIT,
JobStatus.AUDIO_EDIT_COMPLETE,
}
# Statuses where screens_progress.stage='complete' is valid
STATUSES_AFTER_SCREENS_COMPLETE = {
JobStatus.AWAITING_REVIEW,
JobStatus.IN_REVIEW,
JobStatus.REVIEW_COMPLETE,
JobStatus.RENDERING_VIDEO,
JobStatus.AWAITING_INSTRUMENTAL_SELECTION, # Legacy
JobStatus.INSTRUMENTAL_SELECTED,
JobStatus.GENERATING_VIDEO,
JobStatus.ENCODING,
JobStatus.PACKAGING,
JobStatus.UPLOADING,
JobStatus.NOTIFYING,
JobStatus.COMPLETE,
JobStatus.PREP_COMPLETE,
}
def check_job_consistency(job: Job) -> List[str]:
"""
Check for state inconsistencies in a job.
This function detects situations where state_data flags indicate
progress that doesn't match the job status - which usually indicates
a bug in the state machine or trigger logic.
Args:
job: Job object to check
Returns:
List of issue descriptions found. Empty list means no issues.
"""
issues = []
state_data = job.state_data or {}
# Normalize status to JobStatus enum if it's a string
status = job.status
if isinstance(status, str):
try:
status = JobStatus(status)
except ValueError:
issues.append(f"invalid_status: '{job.status}' is not a valid JobStatus")
return issues
# Check: audio_complete=True but status still in pre-audio states
if state_data.get('audio_complete') and status in STATUSES_BEFORE_AUDIO_START:
issues.append(
f"audio_complete_status_mismatch: audio_complete=True but status={status.value}"
)
# Check: lyrics_complete=True but status still in pre-lyrics states
if state_data.get('lyrics_complete') and status in STATUSES_BEFORE_LYRICS_START:
issues.append(
f"lyrics_complete_status_mismatch: lyrics_complete=True but status={status.value}"
)
# Check: screens_progress.stage='complete' but status not in post-screens states
screens_progress = state_data.get('screens_progress', {})
if isinstance(screens_progress, dict) and screens_progress.get('stage') == 'complete':
if status not in STATUSES_AFTER_SCREENS_COMPLETE:
issues.append(
f"screens_complete_status_mismatch: screens_progress.stage=complete but status={status.value}"
)
# Check: job has input_media_gcs_path but is stuck at pending
# This was the exact bug that got jobs 06cfea29 and 984da08b stuck
if job.input_media_gcs_path and status == JobStatus.PENDING:
# This could be legitimate if job was just created and workers haven't started
# But if it's been more than a few minutes, it's likely stuck
# We flag it as a warning, not an error
issues.append(
f"pending_with_input_media: has input_media_gcs_path but status=pending (may be stuck)"
)
# Check: job stuck in encoding status beyond 50 minutes
# Encoding timeout is 1 hour; if updated_at hasn't advanced in 50 min,
# the poller likely died (e.g., Cloud Run deployment killed the instance)
if status == JobStatus.ENCODING and job.updated_at:
now = datetime.now(timezone.utc)
updated_at = job.updated_at
if updated_at.tzinfo is None:
updated_at = updated_at.replace(tzinfo=timezone.utc)
encoding_age = now - updated_at
if encoding_age > timedelta(minutes=50):
minutes = int(encoding_age.total_seconds() / 60)
issues.append(
f"encoding_stuck: status=encoding for {minutes} min without update (updated_at={updated_at.isoformat()})"
)
# Check: job stuck in downloading_audio status beyond 10 minutes
# Audio downloads typically complete in 30s-5min. If stuck longer,
# the Cloud Run Job likely failed silently.
if status == JobStatus.DOWNLOADING_AUDIO and job.updated_at:
now = datetime.now(timezone.utc)
updated_at = job.updated_at
if updated_at.tzinfo is None:
updated_at = updated_at.replace(tzinfo=timezone.utc)
download_age = now - updated_at
if download_age > timedelta(minutes=10):
minutes = int(download_age.total_seconds() / 60)
issues.append(
f"downloading_audio_stuck: status=downloading_audio for {minutes} min without update (updated_at={updated_at.isoformat()})"
)
return issues
def check_job_consistency_detailed(job: Job) -> Dict[str, Any]:
"""
Check job consistency with detailed information for debugging.
Returns a dictionary with:
- job_id: The job ID
- status: Current job status
- issues: List of issue descriptions
- state_data_summary: Summary of relevant state_data fields
- is_healthy: Boolean indicating if no issues were found
Args:
job: Job object to check
Returns:
Dictionary with detailed consistency information
"""
issues = check_job_consistency(job)
state_data = job.state_data or {}
return {
"job_id": job.job_id,
"status": job.status.value if hasattr(job.status, 'value') else str(job.status),
"issues": issues,
"state_data_summary": {
"audio_complete": state_data.get('audio_complete', False),
"lyrics_complete": state_data.get('lyrics_complete', False),
"screens_progress_stage": state_data.get('screens_progress', {}).get('stage'),
"has_input_media_gcs_path": bool(job.input_media_gcs_path),
},
"is_healthy": len(issues) == 0,
}
def get_worker_valid_statuses() -> Dict[str, List[JobStatus]]:
"""
Get mapping of worker names to their valid job statuses.
This is used by workers to validate they should run for a given job.
Returns:
Dictionary mapping worker name to list of valid JobStatus values
"""
return {
"audio_worker": [
JobStatus.DOWNLOADING,
JobStatus.DOWNLOADING_AUDIO,
JobStatus.AUDIO_EDIT_COMPLETE,
JobStatus.SEPARATING_STAGE1,
JobStatus.SEPARATING_STAGE2,
],
"lyrics_worker": [
JobStatus.DOWNLOADING,
JobStatus.AUDIO_EDIT_COMPLETE,
JobStatus.TRANSCRIBING,
JobStatus.CORRECTING,
],
"screens_worker": [
# Screens worker can be triggered when both audio+lyrics complete
# which can happen in any of these states
JobStatus.DOWNLOADING,
JobStatus.AUDIO_COMPLETE,
JobStatus.LYRICS_COMPLETE,
# Also valid if re-running after a failure
JobStatus.GENERATING_SCREENS,
],
"video_worker": [
JobStatus.INSTRUMENTAL_SELECTED,
JobStatus.REVIEW_COMPLETE,
# Note: RENDERING_VIDEO and GENERATING_VIDEO are intentionally excluded.
# If the job is already in those states, another worker is actively processing it.
# Including them caused a race condition where Cloud Run Job retries + manual retries
# could launch concurrent workers that corrupt each other's state transitions.
],
"render_video_worker": [
JobStatus.REVIEW_COMPLETE,
JobStatus.RENDERING_VIDEO,
],
}
def validate_worker_can_run(worker_name: str, job: Job) -> Optional[str]:
"""
Validate that a worker should run for the given job.
Args:
worker_name: Name of the worker (e.g., "screens_worker")
job: Job object to validate
Returns:
None if valid, or an error message string if invalid
"""
valid_statuses_map = get_worker_valid_statuses()
if worker_name not in valid_statuses_map:
# Unknown worker, can't validate
return None
valid_statuses = valid_statuses_map[worker_name]
# Normalize status
status = job.status
if isinstance(status, str):
try:
status = JobStatus(status)
except ValueError:
return f"Invalid job status: {job.status}"
if status not in valid_statuses:
valid_names = [s.value for s in valid_statuses]
return (
f"{worker_name} called but job status is {status.value}, "
f"expected one of {valid_names}. This may indicate a bug in the trigger logic."
)
return None
+348 -98
View File
@@ -14,7 +14,7 @@ from datetime import datetime
from typing import Optional, Dict, Any, List
from backend.config import settings
from backend.exceptions import RateLimitExceededError
from backend.exceptions import InvalidStateTransitionError, InsufficientCreditsError
from backend.models.job import Job, JobStatus, JobCreate, STATE_TRANSITIONS
from backend.models.worker_log import WorkerLogEntry
from backend.services.firestore_service import FirestoreService
@@ -51,33 +51,21 @@ class JobManager:
Args:
job_create: Job creation parameters
is_admin: Whether the requesting user is an admin (bypasses rate limits)
is_admin: Whether the requesting user is an admin (bypasses credit checks)
Raises:
ValueError: If theme_id is not provided (all jobs require a theme)
RateLimitExceededError: If user has exceeded their daily job limit
"""
# Check rate limit FIRST (before any other validation)
# This prevents wasted work if user is rate limited
if job_create.user_email:
from backend.services.rate_limit_service import get_rate_limit_service
rate_limit_service = get_rate_limit_service()
allowed, remaining, message = rate_limit_service.check_user_job_limit(
user_email=job_create.user_email,
is_admin=is_admin
)
if not allowed:
from backend.services.rate_limit_service import _seconds_until_midnight_utc
# Get actual current count - remaining is clamped to 0 which loses info
current_count = rate_limit_service.get_user_job_count_today(job_create.user_email)
raise RateLimitExceededError(
message=message,
limit_type="jobs_per_day",
remaining_seconds=_seconds_until_midnight_utc(),
current_count=current_count,
limit_value=settings.rate_limit_jobs_per_day
# Check credits (skip for admins)
if job_create.user_email and not is_admin:
from backend.services.user_service import get_user_service
user_service = get_user_service()
if not user_service.has_credits(job_create.user_email):
credits_available = user_service.check_credits(job_create.user_email)
raise InsufficientCreditsError(
message="You're out of credits. Buy more to continue creating karaoke videos.",
credits_available=credits_available,
credits_required=1,
)
# Enforce theme requirement - all jobs must have a theme
@@ -130,20 +118,38 @@ class JobManager:
made_for_you=job_create.made_for_you,
customer_email=job_create.customer_email,
customer_notes=job_create.customer_notes,
# Private (non-published) track mode
is_private=job_create.is_private,
# Capture creation-time decision params for observability
creation_params={
"is_private": job_create.is_private,
"is_admin": is_admin,
"created_from": job_create.request_metadata.get("created_from", "unknown"),
},
)
self.firestore.create_job(job)
logger.info(f"Created new job {job_id} with status PENDING")
# Record job creation for rate limiting (after successful persistence)
if job_create.user_email:
try:
from backend.services.rate_limit_service import get_rate_limit_service
rate_limit_service = get_rate_limit_service()
rate_limit_service.record_job_creation(job_create.user_email, job_id)
except Exception as e:
# Don't fail job creation if rate limit recording fails
logger.warning(f"Failed to record job creation for rate limiting: {e}")
# Deduct credit atomically (after job is persisted so we have job_id for transaction record)
if job_create.user_email and not is_admin:
from backend.services.user_service import get_user_service
user_service = get_user_service()
success, _remaining, deduct_msg = user_service.deduct_credit(
job_create.user_email, job_id, reason="job_creation"
)
if not success:
# Deduction failed (e.g., race condition) - delete the job and raise error
logger.error(f"Credit deduction failed for job {job_id}: {deduct_msg}")
try:
self.firestore.delete_job(job_id)
except Exception:
logger.exception(f"Failed to delete job {job_id} after credit deduction failure")
raise InsufficientCreditsError(
message="Failed to deduct credit. Please try again.",
credits_available=0,
credits_required=1,
)
return job
@@ -210,6 +216,31 @@ class JobManager:
limit=limit
)
def list_jobs_summary(
self,
status: Optional[JobStatus] = None,
exclude_statuses: Optional[List[str]] = None,
environment: Optional[str] = None,
client_id: Optional[str] = None,
created_after: Optional[datetime] = None,
created_before: Optional[datetime] = None,
user_email: Optional[str] = None,
tenant_id: Optional[str] = None,
limit: int = 100,
) -> List[Dict[str, Any]]:
"""List jobs with field projection (summary mode)."""
return self.firestore.list_jobs_summary(
status=status,
exclude_statuses=exclude_statuses,
environment=environment,
client_id=client_id,
created_after=created_after,
created_before=created_before,
user_email=user_email,
tenant_id=tenant_id,
limit=limit,
)
def delete_jobs_by_filter(
self,
environment: Optional[str] = None,
@@ -296,14 +327,20 @@ class JobManager:
def delete_job(self, job_id: str, delete_files: bool = True) -> None:
"""Delete a job, its files, and its logs subcollection."""
if delete_files:
job = self.get_job(job_id)
if job and job.output_files:
for gcs_path in job.output_files.values():
try:
self.storage.delete_file(gcs_path)
except Exception as e:
logger.error(f"Error deleting file {gcs_path}: {e}")
job = self.get_job(job_id)
# Refund credit if job never completed (and wasn't already refunded via fail/cancel)
if job:
no_refund_statuses = [JobStatus.COMPLETE, JobStatus.PREP_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED]
if job.status not in no_refund_statuses:
self._refund_credit_for_job(job_id, job, reason="job_deleted")
if delete_files and job and job.output_files:
for gcs_path in job.output_files.values():
try:
self.storage.delete_file(gcs_path)
except Exception as e:
logger.error(f"Error deleting file {gcs_path}: {e}")
# Delete logs subcollection first (must be done before deleting parent doc)
try:
@@ -313,82 +350,126 @@ class JobManager:
except Exception as e:
logger.warning(f"Error deleting logs subcollection for job {job_id}: {e}")
# Delete audio edit sessions subcollection
try:
deleted_sessions = self.firestore.delete_audio_edit_sessions_subcollection(job_id)
if deleted_sessions > 0:
logger.info(f"Deleted {deleted_sessions} audio edit sessions for job {job_id}")
except Exception as e:
logger.warning(f"Error deleting audio edit sessions for job {job_id}: {e}")
self.firestore.delete_job(job_id)
logger.info(f"Deleted job {job_id}")
def validate_state_transition(self, job_id: str, new_status: JobStatus) -> bool:
def validate_state_transition(
self,
job_id: str,
new_status: JobStatus,
raise_on_invalid: bool = True
) -> bool:
"""
Validate that a state transition is legal.
Args:
job_id: Job ID
new_status: Target state
raise_on_invalid: If True, raise InvalidStateTransitionError on invalid
transitions. If False, return False (legacy behavior).
Returns:
True if transition is valid, False otherwise
True if transition is valid
Raises:
InvalidStateTransitionError: If transition is invalid and raise_on_invalid=True
ValueError: If job not found and raise_on_invalid=True
"""
job = self.get_job(job_id)
if not job:
logger.error(f"Job {job_id} not found")
error_msg = f"Job {job_id} not found"
logger.error(error_msg)
if raise_on_invalid:
raise ValueError(error_msg)
return False
current_status = job.status
valid_transitions = STATE_TRANSITIONS.get(current_status, [])
if new_status not in valid_transitions:
logger.error(
error_msg = (
f"Invalid state transition for job {job_id}: "
f"{current_status} -> {new_status}. "
f"Valid transitions: {valid_transitions}"
f"Valid transitions: {[s.value for s in valid_transitions]}"
)
logger.error(error_msg)
if raise_on_invalid:
raise InvalidStateTransitionError(
message=error_msg,
job_id=job_id,
from_status=current_status,
to_status=new_status.value if hasattr(new_status, 'value') else str(new_status),
valid_transitions=[s.value for s in valid_transitions]
)
return False
return True
def transition_to_state(
self,
job_id: str,
new_status: JobStatus,
progress: Optional[int] = None,
message: Optional[str] = None,
state_data_updates: Optional[Dict[str, Any]] = None
state_data_updates: Optional[Dict[str, Any]] = None,
raise_on_invalid: bool = True,
timeline_metadata: Optional[Dict[str, Any]] = None,
) -> bool:
"""
Transition job to new state with validation.
Args:
job_id: Job ID
new_status: Target state
progress: Progress percentage (0-100)
message: Timeline message
state_data_updates: Updates to state_data field
raise_on_invalid: If True (default), raise InvalidStateTransitionError
on invalid transitions. If False, return False silently.
timeline_metadata: Optional structured metadata to attach to the timeline event
Returns:
True if transition succeeded, False otherwise
True if transition succeeded, False if failed (only when raise_on_invalid=False)
Raises:
InvalidStateTransitionError: If transition is invalid and raise_on_invalid=True
"""
if not self.validate_state_transition(job_id, new_status):
if not self.validate_state_transition(job_id, new_status, raise_on_invalid=raise_on_invalid):
return False
updates = {
'status': new_status,
'updated_at': datetime.utcnow()
}
if progress is not None:
updates['progress'] = progress
# Generate review token when entering AWAITING_REVIEW state
# Generate review token when entering blocking states that need user action
# Tokens don't expire - they're job-scoped so low risk, and natural expiry happens when job completes
if new_status == JobStatus.AWAITING_REVIEW:
if new_status in (JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_AUDIO_EDIT):
from backend.api.dependencies import generate_review_token
review_token = generate_review_token()
updates['review_token'] = review_token
updates['review_token_expires_at'] = None # No expiry - token is job-scoped
logger.info(f"Generated review token for job {job_id} (no expiry)")
# If we have state_data_updates, merge them with existing state_data
merged_state_data = None
if state_data_updates:
job = self.get_job(job_id)
if job:
merged_state_data = {**job.state_data, **state_data_updates}
# Update job status (includes timeline event), passing state_data if present
if merged_state_data is not None:
self.update_job_status(
@@ -396,6 +477,7 @@ class JobManager:
status=new_status,
progress=progress,
message=message,
timeline_metadata=timeline_metadata,
state_data=merged_state_data
)
else:
@@ -403,7 +485,8 @@ class JobManager:
job_id=job_id,
status=new_status,
progress=progress,
message=message
message=message,
timeline_metadata=timeline_metadata,
)
# Apply review token update separately if generated
@@ -415,6 +498,17 @@ class JobManager:
logger.info(f"Job {job_id} transitioned to {new_status}")
# Increment user's completed jobs counter on terminal success states
if new_status in (JobStatus.COMPLETE, JobStatus.PREP_COMPLETE):
try:
job = self.get_job(job_id)
if job and job.user_email:
from backend.services.user_service import get_user_service
user_service = get_user_service()
user_service.increment_jobs_completed(job.user_email)
except Exception:
logger.exception(f"Failed to increment jobs_completed for job {job_id}")
# Trigger notifications asynchronously (fire-and-forget)
self._trigger_state_notifications(job_id, new_status)
@@ -445,10 +539,15 @@ class JobManager:
self._send_push_notification(job, "complete")
# Idle reminder scheduling for blocking states
elif new_status in [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION]:
elif new_status in [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.AWAITING_AUDIO_EDIT]:
self._schedule_idle_reminder(job, new_status)
# Send push notification for blocking states
action_type = "lyrics" if new_status == JobStatus.AWAITING_REVIEW else "instrumental"
if new_status == JobStatus.AWAITING_AUDIO_EDIT:
action_type = "audio_edit"
elif new_status == JobStatus.AWAITING_REVIEW:
action_type = "lyrics"
else:
action_type = "instrumental"
self._send_push_notification(job, action_type)
except Exception as e:
@@ -475,6 +574,7 @@ class JobManager:
youtube_url = state_data.get('youtube_url')
dropbox_url = state_data.get('dropbox_link')
brand_code = state_data.get('brand_code')
youtube_queued = state_data.get('youtube_upload_queued', False)
# For made-for-you orders, send to customer and transfer ownership
recipient_email = job.user_email
@@ -498,6 +598,8 @@ class JobManager:
youtube_url=youtube_url,
dropbox_url=dropbox_url,
brand_code=brand_code,
is_private=getattr(job, 'is_private', False),
youtube_queued=youtube_queued,
)
# Try to get existing event loop, create new one if none exists
@@ -532,7 +634,12 @@ class JobManager:
# Record when we entered blocking state (for idle detection)
blocking_entered_at = datetime.utcnow().isoformat()
action_type = "lyrics" if new_status == JobStatus.AWAITING_REVIEW else "instrumental"
if new_status == JobStatus.AWAITING_AUDIO_EDIT:
action_type = "audio_edit"
elif new_status == JobStatus.AWAITING_REVIEW:
action_type = "lyrics"
else:
action_type = "instrumental"
# Update state_data with blocking state info (handle None state_data)
existing_state_data = job.state_data or {}
@@ -599,6 +706,7 @@ class JobManager:
"user_email": job.user_email,
"artist": job.artist,
"title": job.title,
"tenant_id": job.tenant_id or None,
}
async def send_notification():
@@ -640,6 +748,31 @@ class JobManager:
self.update_job(job_id, {'state_data': state_data})
logger.debug(f"Job {job_id} state_data updated: {key} = {value}")
def update_processing_metadata(self, job_id: str, section: str, data) -> None:
"""
Write data into processing_metadata[section] using Firestore dot-notation.
Each worker writes its own section once. This uses Firestore's nested
field update to avoid overwriting other workers' sections.
The section can use dot-notation for deeper nesting, e.g.
'timing.lyrics_worker_seconds' writes to processing_metadata.timing.lyrics_worker_seconds
without overwriting other timing entries.
Args:
job_id: Job ID
section: Metadata section path (e.g., 'transcription', 'timing.audio_worker_seconds')
data: Value to store (dict for sections, scalar for individual fields)
"""
try:
self.firestore.update_job(job_id, {
f"processing_metadata.{section}": data,
})
logger.debug(f"Job {job_id} processing_metadata.{section} updated")
except Exception as e:
# Log but don't raise - metadata storage should never fail a job
logger.warning(f"Job {job_id}: Failed to update processing_metadata.{section}: {e}")
def delete_state_data_key(self, job_id: str, key: str) -> bool:
"""
Delete a specific key from the job's state_data field.
@@ -694,6 +827,46 @@ class JobManager:
logger.error(f"Job {job_id}: Failed to delete state_data keys {keys}: {e}")
return []
def _refund_credit_for_job(self, job_id: str, job: Job, reason: str) -> bool:
"""
Refund one credit for a job if eligible.
Skips refund if: already refunded, no user email, or admin-owned job.
Never raises — refund failure must not block the calling operation.
Returns:
True if credit was refunded, False otherwise.
"""
try:
if job.credit_refunded:
logger.info(f"Credit already refunded for job {job_id}, skipping")
return False
if not job.user_email:
return False
from backend.services.auth_service import is_admin_email
from backend.services.user_service import get_user_service
if is_admin_email(job.user_email):
return False
user_service = get_user_service()
refund_ok, new_balance, refund_msg = user_service.refund_credit(
job.user_email, job_id, reason=reason
)
if refund_ok:
# Mark job as refunded to prevent double-refund
self.update_job(job_id, {'credit_refunded': True})
logger.info(f"Refunded credit for {reason} job {job_id} to {_mask_email(job.user_email)} (balance: {new_balance})")
return True
else:
logger.warning(f"Credit refund failed for job {job_id}: {refund_msg}")
return False
except Exception as e:
logger.warning(f"Error refunding credit for job {job_id}: {e}")
return False
def fail_job(self, job_id: str, error_message: str, error_details: Optional[Dict[str, Any]] = None) -> bool:
"""
Mark a job as failed with error information.
@@ -721,11 +894,17 @@ class JobManager:
)
logger.error(f"Job {job_id} failed: {error_message}")
# Auto-refund credit on failure
job = self.get_job(job_id)
if job:
self._refund_credit_for_job(job_id, job, reason="job_failed")
return True
except Exception as e:
logger.error(f"Error marking job {job_id} as failed: {e}")
return False
def update_file_url(self, job_id: str, category: str, file_type: str, url: str) -> None:
"""
Update a file URL in the job's file_urls structure.
@@ -752,49 +931,116 @@ class JobManager:
def check_parallel_processing_complete(self, job_id: str) -> bool:
"""
Check if both parallel tracks (audio + lyrics) are complete.
This is called after audio_complete or lyrics_complete to determine
if we can proceed to screen generation.
Check if lyrics processing is complete (sufficient for screen generation).
Audio separation is decoupled from the critical path — users can start
reviewing lyrics while separation continues in the background. The
instrumental review page handles the case where separation isn't done yet.
Returns:
True if both tracks complete, False otherwise
True if lyrics are complete, False otherwise
"""
job = self.get_job(job_id)
if not job:
return False
audio_complete = job.state_data.get('audio_complete', False)
lyrics_complete = job.state_data.get('lyrics_complete', False)
return audio_complete and lyrics_complete
return lyrics_complete
async def start_job_processing(
self,
job_id: str,
progress: int = 15,
message: str = "Starting audio and lyrics processing"
) -> None:
"""
Transition job to DOWNLOADING and trigger audio + lyrics workers.
This is the single entry point for starting job processing after audio
is available. All handlers (file_upload, webhook, audio_search) should
use this method to ensure consistent state transitions.
The method:
1. Validates job exists and has input_media_gcs_path
2. Transitions to DOWNLOADING status (raises if invalid)
3. Triggers audio and lyrics workers in parallel
Args:
job_id: Job ID to start processing
progress: Progress percentage to set (default 15)
message: Timeline message (default "Starting audio and lyrics processing")
Raises:
InvalidStateTransitionError: If job can't transition to DOWNLOADING
ValueError: If job not found or missing input_media_gcs_path
Example:
# After downloading audio to GCS:
job_manager.update_job(job_id, {'input_media_gcs_path': audio_path})
await job_manager.start_job_processing(job_id)
"""
import asyncio
from backend.services.worker_service import get_worker_service
job = self.get_job(job_id)
if not job:
raise ValueError(f"Job {job_id} not found")
if not job.input_media_gcs_path:
raise ValueError(
f"Job {job_id} missing input_media_gcs_path. "
"Audio must be downloaded to GCS before starting processing."
)
# Transition to DOWNLOADING - this will raise if invalid
self.transition_to_state(
job_id=job_id,
new_status=JobStatus.DOWNLOADING,
progress=progress,
message=message
)
# Check if audio separation can be skipped
# Jobs with an existing instrumental (e.g. tenant uploads) don't need
# GPU-powered audio separation — saves 5-10 minutes of processing time
worker_service = get_worker_service()
if job.existing_instrumental_gcs_path:
logger.info(f"Job {job_id}: Skipping audio separation (existing instrumental provided)")
# Mark audio as immediately complete and only trigger lyrics worker
self.update_state_data(job_id, 'audio_complete', True)
await worker_service.trigger_lyrics_worker(job_id)
logger.info(f"Job {job_id}: Started processing (lyrics worker only, audio separation skipped)")
else:
# Trigger both workers in parallel
await asyncio.gather(
worker_service.trigger_audio_worker(job_id),
worker_service.trigger_lyrics_worker(job_id)
)
logger.info(f"Job {job_id}: Started processing (audio + lyrics workers triggered)")
def mark_audio_complete(self, job_id: str) -> None:
"""
Mark audio processing as complete and check if can proceed.
If lyrics are also complete, automatically triggers screens worker.
Mark audio processing as complete.
Audio completion is decoupled from the lyrics review path — it no longer
triggers screens. The instrumental review page polls for this flag.
"""
self.update_state_data(job_id, 'audio_complete', True)
if self.check_parallel_processing_complete(job_id):
logger.info(f"Job {job_id}: Both audio and lyrics complete, triggering screens worker")
# Transition happens in screens worker
# We just trigger it here
self._trigger_screens_worker(job_id)
logger.info(f"Job {job_id}: Audio processing complete (separation finished)")
def mark_lyrics_complete(self, job_id: str) -> None:
"""
Mark lyrics processing as complete and check if can proceed.
If audio is also complete, automatically triggers screens worker.
Mark lyrics processing as complete and trigger screens if ready.
Lyrics completion is the sole gate for screen generation and review.
Audio separation runs independently in the background.
"""
self.update_state_data(job_id, 'lyrics_complete', True)
if self.check_parallel_processing_complete(job_id):
logger.info(f"Job {job_id}: Both audio and lyrics complete, triggering screens worker")
# Transition happens in screens worker
# We just trigger it here
logger.info(f"Job {job_id}: Lyrics complete, triggering screens worker")
self._trigger_screens_worker(job_id)
def _trigger_screens_worker(self, job_id: str) -> None:
@@ -853,8 +1099,12 @@ class JobManager:
)
logger.info(f"Job {job_id} cancelled")
# Refund credit for cancelled job
self._refund_credit_for_job(job_id, job, reason="job_cancelled")
return True
def mark_job_failed(
self,
job_id: str,
+119 -6
View File
@@ -12,8 +12,10 @@ import logging
import os
from typing import Optional, Dict, Any
from backend.i18n import get_locale_prefix
from backend.services.email_service import get_email_service
from backend.services.template_service import get_template_service
from backend.services.user_service import get_user_service
logger = logging.getLogger(__name__)
@@ -51,15 +53,30 @@ class JobNotificationService:
self.frontend_url = os.getenv("FRONTEND_URL", "https://gen.nomadkaraoke.com")
self.backend_url = os.getenv("BACKEND_URL", "https://api.nomadkaraoke.com")
def _build_review_url(self, job_id: str, audio_hash: Optional[str] = None, review_token: Optional[str] = None) -> str:
def _get_user_locale(self, user_email: str) -> str:
"""Look up user's locale preference from Firestore, defaulting to 'en'."""
try:
user_service = get_user_service()
user = user_service.get_user(user_email)
if user and user.locale:
return user.locale
except Exception:
pass # Fall back to English
return "en"
def _build_review_url(self, job_id: str, audio_hash: Optional[str] = None, review_token: Optional[str] = None, locale: str = "en") -> str:
"""Build the lyrics review URL for a job."""
# Use hash-based routing for static hosting compatibility
return f"{self.frontend_url}/app/jobs#/{job_id}/review"
return f"{self.frontend_url}{get_locale_prefix(locale)}/app/jobs#/{job_id}/review"
def _build_instrumental_url(self, job_id: str, instrumental_token: Optional[str] = None) -> str:
def _build_instrumental_url(self, job_id: str, instrumental_token: Optional[str] = None, locale: str = "en") -> str:
"""Build the instrumental selection URL for a job."""
# Use hash-based routing for static hosting compatibility
return f"{self.frontend_url}/app/jobs#/{job_id}/instrumental"
return f"{self.frontend_url}{get_locale_prefix(locale)}/app/jobs#/{job_id}/instrumental"
def _build_audio_edit_url(self, job_id: str, locale: str = "en") -> str:
"""Build the audio edit URL for a job."""
return f"{self.frontend_url}{get_locale_prefix(locale)}/app/jobs#/{job_id}/audio-edit"
async def send_job_completion_email(
self,
@@ -71,6 +88,8 @@ class JobNotificationService:
youtube_url: Optional[str] = None,
dropbox_url: Optional[str] = None,
brand_code: Optional[str] = None,
is_private: bool = False,
youtube_queued: bool = False,
) -> bool:
"""
Send job completion email to user.
@@ -84,6 +103,8 @@ class JobNotificationService:
youtube_url: YouTube video URL
dropbox_url: Dropbox folder URL
brand_code: Release ID (e.g., "NOMAD-1178")
is_private: If True, omit YouTube section (private/non-published tracks)
youtube_queued: If True, YouTube upload was deferred due to quota
Returns:
True if email was sent successfully
@@ -97,6 +118,9 @@ class JobNotificationService:
return False
try:
# Get user locale for email
user_locale = self._get_user_locale(user_email)
# Render the completion message using template service
message_content = self.template_service.render_job_completion(
name=user_name,
@@ -106,6 +130,9 @@ class JobNotificationService:
title=title,
job_id=job_id,
feedback_url=FEEDBACK_FORM_URL,
is_private=is_private,
youtube_queued=youtube_queued,
locale=user_locale,
)
# Send the email with CC to admin
@@ -116,6 +143,7 @@ class JobNotificationService:
title=title,
brand_code=brand_code,
cc_admin=True,
locale=user_locale,
)
if success:
@@ -129,6 +157,68 @@ class JobNotificationService:
logger.exception(f"Error sending completion email for job {job_id}: {e}")
return False
async def send_youtube_upload_complete_email(
self,
job_id: str,
user_email: str,
artist: Optional[str] = None,
title: Optional[str] = None,
youtube_url: Optional[str] = None,
brand_code: Optional[str] = None,
) -> bool:
"""
Send follow-up email when a deferred YouTube upload completes.
Args:
job_id: Job ID
user_email: User's email address
artist: Artist name
title: Song title
youtube_url: YouTube video URL
brand_code: Release ID
Returns:
True if email was sent successfully
"""
if not ENABLE_AUTO_EMAILS:
logger.info(f"Auto emails disabled, skipping YouTube notification for job {job_id}")
return False
if not user_email:
logger.warning(f"No user email for job {job_id}, skipping YouTube notification")
return False
try:
# Get user locale for email
user_locale = self._get_user_locale(user_email)
message_content = self.template_service.render_youtube_upload_complete(
artist=artist,
title=title,
youtube_url=youtube_url,
locale=user_locale,
)
success = self.email_service.send_youtube_upload_complete(
to_email=user_email,
message_content=message_content,
artist=artist,
title=title,
brand_code=brand_code,
locale=user_locale,
)
if success:
logger.info(f"Sent YouTube upload notification for job {job_id} to {_mask_email(user_email)}")
else:
logger.error(f"Failed to send YouTube upload notification for job {job_id}")
return success
except Exception as e:
logger.exception(f"Error sending YouTube upload notification for job {job_id}: {e}")
return False
async def send_action_reminder_email(
self,
job_id: str,
@@ -150,6 +240,7 @@ class JobNotificationService:
action_type: Type of action needed:
- "lyrics": Combined review (lyrics + instrumental selection) for normal jobs
- "instrumental": Instrumental-only selection for finalise-only jobs
- "audio_edit": Audio editing before processing begins
user_name: User's display name
artist: Artist name
title: Song title
@@ -169,22 +260,37 @@ class JobNotificationService:
return False
try:
# Get user locale for email
user_locale = self._get_user_locale(user_email)
# Render the appropriate template
if action_type == "lyrics":
review_url = self._build_review_url(job_id, audio_hash, review_token)
review_url = self._build_review_url(job_id, audio_hash, review_token, locale=user_locale)
message_content = self.template_service.render_action_needed_lyrics(
name=user_name,
artist=artist,
title=title,
review_url=review_url,
locale=user_locale,
)
elif action_type == "instrumental":
instrumental_url = self._build_instrumental_url(job_id, instrumental_token)
instrumental_url = self._build_instrumental_url(job_id, instrumental_token, locale=user_locale)
message_content = self.template_service.render_action_needed_instrumental(
name=user_name,
artist=artist,
title=title,
instrumental_url=instrumental_url,
locale=user_locale,
)
elif action_type == "audio_edit":
audio_edit_url = self._build_audio_edit_url(job_id, locale=user_locale)
# Reuse the lyrics template but with the audio edit URL and different subject
message_content = self.template_service.render_action_needed_lyrics(
name=user_name,
artist=artist,
title=title,
review_url=audio_edit_url,
locale=user_locale,
)
else:
logger.error(f"Unknown action type: {action_type}")
@@ -197,6 +303,7 @@ class JobNotificationService:
action_type=action_type,
artist=artist,
title=title,
locale=user_locale,
)
if success:
@@ -218,6 +325,8 @@ class JobNotificationService:
title: Optional[str] = None,
youtube_url: Optional[str] = None,
dropbox_url: Optional[str] = None,
is_private: bool = False,
youtube_queued: bool = False,
) -> str:
"""
Get the rendered completion message for a job (for admin copy functionality).
@@ -229,6 +338,8 @@ class JobNotificationService:
title: Song title
youtube_url: YouTube video URL
dropbox_url: Dropbox folder URL
is_private: If True, omit YouTube section (private/non-published tracks)
youtube_queued: If True, YouTube upload was deferred
Returns:
Rendered message content as plain text
@@ -241,6 +352,8 @@ class JobNotificationService:
title=title,
job_id=job_id,
feedback_url=FEEDBACK_FORM_URL,
is_private=is_private,
youtube_queued=youtube_queued,
)
+203
View File
@@ -0,0 +1,203 @@
"""
KaraokeNerds community version detection service.
Scrapes karaokenerds.com to check if a song already has community-approved
karaoke versions available on YouTube. Ported from kjbox/kj-controller/karaoke_nerds.py.
"""
import logging
import re
import time
from typing import Any
from urllib.parse import urlencode
import httpx
from bs4 import BeautifulSoup
logger = logging.getLogger(__name__)
SEARCH_URL = "https://karaokenerds.com/Search"
REQUEST_TIMEOUT = 8
USER_AGENT = "NomadKaraokeGen/1.0"
# In-memory TTL cache: { cache_key: (expiry_timestamp, data) }
_cache: dict[str, tuple[float, Any]] = {}
CACHE_TTL_SECONDS = 3600 # 1 hour
def _cache_get(key: str) -> Any | None:
"""Get a value from cache if it exists and hasn't expired."""
if key in _cache:
expiry, data = _cache[key]
if time.monotonic() < expiry:
return data
del _cache[key]
return None
def _cache_set(key: str, data: Any, ttl: float = CACHE_TTL_SECONDS) -> None:
"""Store a value in cache with TTL."""
_cache[key] = (time.monotonic() + ttl, data)
def _clean_youtube_url(url: str) -> str:
"""Strip playlist params from YouTube URLs, keep just the video URL."""
return re.sub(r"&list=[^&]*", "", url)
def _parse_single_track(li) -> dict | None:
"""Parse a single track <li> element from karaokenerds search results."""
# Brand name: first <a> in the li
brand_link = li.find("a")
brand_name = brand_link.get_text(strip=True) if brand_link else ""
# Brand code: text inside .badge span
badge = li.find("span", class_="badge")
brand_code = ""
if badge:
brand_code = badge.get_text(strip=True)
# YouTube URL: link containing youtube.com
youtube_url = None
for a in li.find_all("a", href=True):
href = a["href"]
if "youtube.com" in href:
youtube_url = _clean_youtube_url(href)
break
# Community: presence of img.check
is_community = bool(li.find("img", class_="check"))
if not youtube_url:
return None
return {
"brand_name": brand_name,
"brand_code": brand_code,
"youtube_url": youtube_url,
"is_community": is_community,
}
def _parse_tracks(details_row) -> list[dict]:
"""Parse track list items from a details row."""
tracks = []
for li in details_row.find_all("li", class_="track"):
track = _parse_single_track(li)
if track:
tracks.append(track)
return tracks
def parse_results(html: str) -> list[dict]:
"""Parse karaokenerds.com search results HTML into structured data."""
soup = BeautifulSoup(html, "html.parser")
table = soup.find("table")
if not table:
return []
tbody = table.find("tbody")
if not tbody:
return []
songs = []
rows = tbody.find_all("tr", recursive=False)
i = 0
while i < len(rows):
row = rows[i]
# Song rows have class "group"
if "group" not in row.get("class", []):
i += 1
continue
# Extract title and artist from the song row
cells = row.find_all("td")
if len(cells) < 3:
i += 1
continue
title_link = cells[0].find("a")
artist_link = cells[1].find("a")
title = title_link.get_text(strip=True) if title_link else ""
artist = artist_link.get_text(strip=True) if artist_link else ""
# The next row should be the details row with tracks
tracks = []
if i + 1 < len(rows):
details_row = rows[i + 1]
if "details" in details_row.get("class", []):
tracks = _parse_tracks(details_row)
i += 2
else:
i += 1
else:
i += 1
if title:
songs.append({
"title": title,
"artist": artist,
"tracks": tracks,
})
return songs
async def check_community_versions(artist: str, title: str) -> dict:
"""
Check if a song has community-approved karaoke versions on karaokenerds.
Returns a dict with:
- has_community: bool
- songs: list of matched songs with community tracks
- best_youtube_url: URL of the top community version (if any)
"""
query = f"{artist} {title}"
cache_key = f"community:{query.lower().strip()}"
cached = _cache_get(cache_key)
if cached is not None:
return cached
params = urlencode({"query": query, "webFilter": "OnlyWeb"})
url = f"{SEARCH_URL}?{params}"
try:
async with httpx.AsyncClient(timeout=REQUEST_TIMEOUT) as client:
resp = await client.get(
url,
headers={"User-Agent": USER_AGENT},
follow_redirects=True,
)
resp.raise_for_status()
except Exception as e:
logger.warning(f"KaraokeNerds search failed for '{query}': {e}")
result = {"has_community": False, "songs": [], "best_youtube_url": None}
_cache_set(cache_key, result)
return result
songs = parse_results(resp.text)
# Filter to only songs that have community tracks with YouTube URLs
community_songs = []
best_youtube_url = None
for song in songs:
community_tracks = [t for t in song["tracks"] if t["is_community"]]
if community_tracks:
community_songs.append({
"title": song["title"],
"artist": song["artist"],
"community_tracks": community_tracks,
})
if best_youtube_url is None:
best_youtube_url = community_tracks[0]["youtube_url"]
result = {
"has_community": len(community_songs) > 0,
"songs": community_songs,
"best_youtube_url": best_youtube_url,
}
_cache_set(cache_key, result)
return result
+4 -1
View File
@@ -545,7 +545,10 @@ class LocalEncodingService:
)
output_files["with_vocals_mp4"] = config.output_with_vocals_mp4
else:
self.logger.info("[Step 2/6] Skipped - video already MP4")
self.logger.info("[Step 2/6] Skipped - video already MP4, copying as-is")
import shutil
shutil.copy2(config.karaoke_video, config.output_with_vocals_mp4)
output_files["with_vocals_mp4"] = config.output_with_vocals_mp4
# Step 3: Encode lossless 4K MP4
if config.output_lossless_4k_mp4:
+7
View File
@@ -57,6 +57,8 @@ class PackagingService:
title: str,
output_mp3_path: Optional[str] = None,
output_cdg_path: Optional[str] = None,
lrc_has_countdown_padding: bool = False,
countdown_padding_seconds: float = 3.0,
) -> Tuple[str, Optional[str], Optional[str]]:
"""
Create a CDG package (ZIP containing CDG and MP3 files).
@@ -69,6 +71,9 @@ class PackagingService:
title: Song title for metadata
output_mp3_path: Optional path for the extracted MP3 file
output_cdg_path: Optional path for the extracted CDG file
lrc_has_countdown_padding: If True, LRC contains countdown-shifted timestamps
that need adjustment since CDG uses instrumental audio without padding
countdown_padding_seconds: Duration of countdown padding to compensate for
Returns:
Tuple of (zip_path, mp3_path, cdg_path)
@@ -131,6 +136,8 @@ class PackagingService:
title=title,
artist=artist,
cdg_styles=self.cdg_styles,
lrc_has_countdown_padding=lrc_has_countdown_padding,
countdown_padding_seconds=countdown_padding_seconds,
)
# Rename ZIP to expected output path if different
+56 -17
View File
@@ -12,6 +12,7 @@ from pywebpush import webpush, WebPushException
from backend.config import get_settings
from backend.models.user import User, PushSubscription
from backend.i18n import t
logger = logging.getLogger(__name__)
@@ -85,10 +86,14 @@ class PushNotificationService:
title: str,
body: str,
url: str = "/app/",
tag: Optional[str] = None
tag: Optional[str] = None,
tenant_id: Optional[str] = None
) -> int:
"""
Send a push notification to all of a user's subscribed devices.
Send a push notification to a user's subscribed devices for a specific tenant.
Only sends to subscriptions matching the given tenant_id to prevent
cross-tenant notification leakage.
Args:
user_email: The user's email address
@@ -96,6 +101,8 @@ class PushNotificationService:
body: Notification body text
url: URL to open when notification is clicked
tag: Optional tag for notification grouping (replaces notifications with same tag)
tenant_id: Tenant scope - only send to subscriptions for this tenant
(None or "" = consumer/default portal)
Returns:
Number of notifications successfully sent
@@ -111,7 +118,30 @@ class PushNotificationService:
return 0
user_data = user_doc.to_dict()
subscriptions = user_data.get("push_subscriptions", [])
all_subscriptions = user_data.get("push_subscriptions", [])
# Filter subscriptions to only those matching the target tenant
# Normalize: None and "" both mean "consumer/default portal"
target_tenant = tenant_id or None
subscriptions = [
sub for sub in all_subscriptions
if (sub.get("tenant_id") or None) == target_tenant
]
if not subscriptions and all_subscriptions:
# Migration path: if no subscriptions match and none have tenant_id set,
# it means they're legacy subscriptions created before tenant scoping.
# Send to legacy subscriptions (those without tenant_id) only if
# the target is the consumer portal (no tenant).
has_any_tenant_scoped = any(sub.get("tenant_id") for sub in all_subscriptions)
if not has_any_tenant_scoped and target_tenant is None:
subscriptions = all_subscriptions
logger.debug(f"Using legacy unscoped subscriptions for {user_email}")
else:
logger.debug(
f"No matching subscriptions for tenant={target_tenant} "
f"(user has {len(all_subscriptions)} total subscriptions)"
)
if not subscriptions:
logger.debug(f"User {user_email} has no push subscriptions")
@@ -196,7 +226,7 @@ class PushNotificationService:
Send notification when a job enters a blocking state requiring user action.
Args:
job: Job dictionary with job_id, user_email, artist, title
job: Job dictionary with job_id, user_email, artist, title, tenant_id
action_type: Type of action needed ("lyrics" or "instrumental")
Returns:
@@ -209,16 +239,18 @@ class PushNotificationService:
job_id = job.get("job_id", "unknown")
artist = job.get("artist", "Unknown Artist")
title = job.get("title", "Unknown Title")
tenant_id = job.get("tenant_id") or None
locale = job.get("locale", "en")
if action_type == "lyrics":
notif_title = "Review Lyrics"
notif_body = f'"{title}" by {artist} needs lyrics review'
url = f"/review/{job_id}"
notif_title = t(locale, "pushNotifications.reviewLyricsTitle")
notif_body = t(locale, "pushNotifications.reviewLyricsBody", title=title, artist=artist)
url = f"/app/jobs#/{job_id}/review"
tag = f"lyrics-{job_id}"
else: # instrumental
notif_title = "Select Instrumental"
notif_body = f'"{title}" by {artist} needs instrumental selection'
url = f"/instrumental/{job_id}"
notif_title = t(locale, "pushNotifications.selectInstrumentalTitle")
notif_body = t(locale, "pushNotifications.selectInstrumentalBody", title=title, artist=artist)
url = f"/app/jobs#/{job_id}/instrumental"
tag = f"instrumental-{job_id}"
return await self.send_push(
@@ -226,7 +258,8 @@ class PushNotificationService:
title=notif_title,
body=notif_body,
url=url,
tag=tag
tag=tag,
tenant_id=tenant_id,
)
async def send_completion_notification(self, job: dict) -> int:
@@ -234,7 +267,7 @@ class PushNotificationService:
Send notification when a job completes successfully.
Args:
job: Job dictionary with job_id, user_email, artist, title
job: Job dictionary with job_id, user_email, artist, title, tenant_id
Returns:
Number of notifications sent
@@ -246,13 +279,16 @@ class PushNotificationService:
job_id = job.get("job_id", "unknown")
artist = job.get("artist", "Unknown Artist")
title = job.get("title", "Unknown Title")
tenant_id = job.get("tenant_id") or None
locale = job.get("locale", "en")
return await self.send_push(
user_email=user_email,
title="Video Ready!",
body=f'Your karaoke video for "{title}" by {artist} is ready to download',
title=t(locale, "pushNotifications.videoReadyTitle"),
body=t(locale, "pushNotifications.videoReadyBody", title=title, artist=artist),
url=f"/app/?job={job_id}",
tag=f"complete-{job_id}"
tag=f"complete-{job_id}",
tenant_id=tenant_id,
)
async def add_subscription(
@@ -260,10 +296,11 @@ class PushNotificationService:
user_email: str,
endpoint: str,
keys: dict,
device_name: Optional[str] = None
device_name: Optional[str] = None,
tenant_id: Optional[str] = None
) -> bool:
"""
Add a push subscription for a user.
Add a push subscription for a user, scoped to a tenant.
Enforces max subscriptions per user - oldest removed if limit exceeded.
@@ -272,6 +309,7 @@ class PushNotificationService:
endpoint: Push service endpoint URL
keys: Encryption keys (p256dh, auth)
device_name: Optional device identifier
tenant_id: Tenant scope (None = consumer/default portal)
Returns:
True if subscription was added successfully
@@ -298,6 +336,7 @@ class PushNotificationService:
"endpoint": endpoint,
"keys": keys,
"device_name": device_name,
"tenant_id": tenant_id or None,
"created_at": datetime.now(timezone.utc).isoformat(),
"last_used_at": None
}
-641
View File
@@ -1,641 +0,0 @@
"""
Rate limiting service for job creation, YouTube uploads, and beta enrollment.
Uses Firestore for distributed rate limit tracking with date-based document IDs
that automatically reset at UTC midnight.
"""
import hashlib
import logging
from datetime import datetime, timezone
from typing import Tuple, Optional, List, Dict, Any
from google.cloud import firestore
from backend.config import settings
logger = logging.getLogger(__name__)
# Firestore collection names
RATE_LIMITS_COLLECTION = "rate_limits"
BLOCKLISTS_COLLECTION = "blocklists"
OVERRIDES_COLLECTION = "overrides"
def _get_today_date_str() -> str:
"""Get today's date in UTC as YYYY-MM-DD string."""
return datetime.now(timezone.utc).strftime("%Y-%m-%d")
def _seconds_until_midnight_utc() -> int:
"""Calculate seconds until UTC midnight (when rate limits reset)."""
now = datetime.now(timezone.utc)
midnight = now.replace(hour=0, minute=0, second=0, microsecond=0)
# Add 1 day to get next midnight
from datetime import timedelta
next_midnight = midnight + timedelta(days=1)
return int((next_midnight - now).total_seconds())
def _hash_ip(ip_address: str) -> str:
"""Hash an IP address for storage (privacy-preserving)."""
return hashlib.sha256(ip_address.encode()).hexdigest()[:16]
class RateLimitService:
"""
Service for managing rate limits across the platform.
Rate limits use Firestore for distributed tracking with automatic
reset at UTC midnight via date-based document IDs.
"""
def __init__(self, db: Optional[firestore.Client] = None):
"""Initialize rate limit service with Firestore client."""
if db is None:
self.db = firestore.Client(project=settings.google_cloud_project)
else:
self.db = db
self._blocklist_cache: Optional[Dict[str, Any]] = None
self._blocklist_cache_time: Optional[datetime] = None
self._cache_ttl_seconds = 300 # 5 minute cache
# -------------------------------------------------------------------------
# Job Rate Limiting
# -------------------------------------------------------------------------
def check_user_job_limit(self, user_email: str, is_admin: bool = False) -> Tuple[bool, int, str]:
"""
Check if a user can create a new job.
Args:
user_email: User's email address
is_admin: Whether the user is an admin (admins bypass limits)
Returns:
Tuple of (allowed, remaining, message)
- allowed: True if user can create job
- remaining: Number of jobs remaining today
- message: Human-readable status message
"""
if not settings.enable_rate_limiting:
return True, -1, "Rate limiting disabled"
if is_admin:
return True, -1, "Admin users bypass rate limits"
limit = settings.rate_limit_jobs_per_day
if limit == 0:
return True, -1, "No job limit configured"
# Check for user override
override = self.get_user_override(user_email)
if override and override.get("bypass_job_limit"):
return True, -1, "User has rate limit bypass"
if override and override.get("custom_daily_job_limit"):
limit = override["custom_daily_job_limit"]
# Get current count
current_count = self.get_user_job_count_today(user_email)
remaining = max(0, limit - current_count)
if current_count >= limit:
seconds = _seconds_until_midnight_utc()
logger.warning(
f"Rate limit exceeded for {user_email}: {current_count}/{limit} jobs today"
)
return (
False,
0,
f"Daily job limit reached ({limit} jobs per day). Resets in {seconds // 3600}h {(seconds % 3600) // 60}m."
)
return True, remaining, f"{remaining} jobs remaining today"
def get_user_job_count_today(self, user_email: str) -> int:
"""Get the number of jobs created by a user today."""
date_str = _get_today_date_str()
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
f"user_jobs_{user_email}_{date_str}"
)
doc = doc_ref.get()
if not doc.exists:
return 0
return doc.to_dict().get("count", 0)
def record_job_creation(self, user_email: str, job_id: str) -> None:
"""
Record a job creation for rate limiting.
Uses Firestore transactions for atomic increment.
"""
if not settings.enable_rate_limiting:
return
date_str = _get_today_date_str()
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
f"user_jobs_{user_email}_{date_str}"
)
@firestore.transactional
def update_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if doc.exists:
data = doc.to_dict()
count = data.get("count", 0) + 1
job_ids = data.get("job_ids", [])
job_ids.append(job_id)
else:
count = 1
job_ids = [job_id]
transaction.set(doc_ref, {
"user_email": user_email,
"date": date_str,
"count": count,
"job_ids": job_ids,
"updated_at": datetime.now(timezone.utc),
})
transaction = self.db.transaction()
update_in_transaction(transaction, doc_ref)
logger.info(f"Recorded job {job_id} for user {user_email} rate limiting")
# -------------------------------------------------------------------------
# YouTube Upload Rate Limiting
# -------------------------------------------------------------------------
def check_youtube_upload_limit(self) -> Tuple[bool, int, str]:
"""
Check if the system can perform a YouTube upload.
YouTube uploads are limited system-wide due to API quota constraints.
Returns:
Tuple of (allowed, remaining, message)
"""
if not settings.enable_rate_limiting:
return True, -1, "Rate limiting disabled"
limit = settings.rate_limit_youtube_uploads_per_day
if limit == 0:
return True, -1, "No YouTube upload limit configured"
current_count = self.get_youtube_uploads_today()
remaining = max(0, limit - current_count)
if current_count >= limit:
seconds = _seconds_until_midnight_utc()
logger.warning(
f"YouTube upload limit exceeded: {current_count}/{limit} uploads today"
)
return (
False,
0,
f"Daily YouTube upload limit reached ({limit} uploads per day). Resets in {seconds // 3600}h {(seconds % 3600) // 60}m."
)
return True, remaining, f"{remaining} YouTube uploads remaining today"
def get_youtube_uploads_today(self) -> int:
"""Get the number of YouTube uploads performed today (system-wide)."""
date_str = _get_today_date_str()
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
f"youtube_uploads_{date_str}"
)
doc = doc_ref.get()
if not doc.exists:
return 0
return doc.to_dict().get("count", 0)
def record_youtube_upload(self, job_id: str, user_email: str) -> None:
"""
Record a YouTube upload for rate limiting.
Uses Firestore transactions for atomic increment.
"""
if not settings.enable_rate_limiting:
return
date_str = _get_today_date_str()
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
f"youtube_uploads_{date_str}"
)
@firestore.transactional
def update_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if doc.exists:
data = doc.to_dict()
count = data.get("count", 0) + 1
uploads = data.get("uploads", [])
else:
count = 1
uploads = []
uploads.append({
"job_id": job_id,
"user_email": user_email,
"timestamp": datetime.now(timezone.utc),
})
transaction.set(doc_ref, {
"date": date_str,
"count": count,
"uploads": uploads,
"updated_at": datetime.now(timezone.utc),
})
transaction = self.db.transaction()
update_in_transaction(transaction, doc_ref)
logger.info(f"Recorded YouTube upload for job {job_id}")
# -------------------------------------------------------------------------
# Beta Enrollment IP Rate Limiting
# -------------------------------------------------------------------------
def check_beta_ip_limit(self, ip_address: str) -> Tuple[bool, int, str]:
"""
Check if an IP address can enroll in the beta program.
Args:
ip_address: Client IP address
Returns:
Tuple of (allowed, remaining, message)
"""
if not settings.enable_rate_limiting:
return True, -1, "Rate limiting disabled"
limit = settings.rate_limit_beta_ip_per_day
if limit == 0:
return True, -1, "No beta IP limit configured"
# Check today's enrollment count for this IP
ip_hash = _hash_ip(ip_address)
date_str = _get_today_date_str()
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
f"beta_ip_{ip_hash}_{date_str}"
)
doc = doc_ref.get()
current_count = 0
if doc.exists:
current_count = doc.to_dict().get("count", 0)
remaining = max(0, limit - current_count)
if current_count >= limit:
seconds = _seconds_until_midnight_utc()
logger.warning(
f"Beta enrollment IP limit exceeded for {ip_hash}: {current_count}/{limit} enrollments today"
)
return (
False,
0,
f"Too many beta enrollments from this network today. Please try again tomorrow."
)
return True, remaining, f"{remaining} beta enrollments remaining from this IP today"
def record_beta_enrollment(self, ip_address: str, email: str) -> None:
"""
Record a beta enrollment for IP rate limiting.
Uses Firestore transactions for atomic increment.
"""
if not settings.enable_rate_limiting:
return
ip_hash = _hash_ip(ip_address)
date_str = _get_today_date_str()
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
f"beta_ip_{ip_hash}_{date_str}"
)
@firestore.transactional
def update_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if doc.exists:
data = doc.to_dict()
count = data.get("count", 0) + 1
enrollments = data.get("enrollments", [])
else:
count = 1
enrollments = []
enrollments.append({
"email": email,
"timestamp": datetime.now(timezone.utc),
})
transaction.set(doc_ref, {
"ip_hash": ip_hash,
"date": date_str,
"count": count,
"enrollments": enrollments,
"updated_at": datetime.now(timezone.utc),
})
transaction = self.db.transaction()
update_in_transaction(transaction, doc_ref)
logger.info(f"Recorded beta enrollment from IP {ip_hash} for {email}")
# -------------------------------------------------------------------------
# User Overrides (Whitelist)
# -------------------------------------------------------------------------
def get_user_override(self, user_email: str) -> Optional[Dict[str, Any]]:
"""
Get rate limit override settings for a user.
Returns:
Override settings dict or None if no override exists
"""
doc_ref = self.db.collection(OVERRIDES_COLLECTION).document(user_email.lower())
doc = doc_ref.get()
if not doc.exists:
return None
return doc.to_dict()
def set_user_override(
self,
user_email: str,
bypass_job_limit: bool = False,
custom_daily_job_limit: Optional[int] = None,
reason: str = "",
admin_email: str = ""
) -> None:
"""
Set rate limit override for a user.
Args:
user_email: User to override
bypass_job_limit: If True, user bypasses all job limits
custom_daily_job_limit: Custom limit (None = use default)
reason: Reason for override
admin_email: Admin who set the override
"""
doc_ref = self.db.collection(OVERRIDES_COLLECTION).document(user_email.lower())
doc_ref.set({
"email": user_email.lower(),
"bypass_job_limit": bypass_job_limit,
"custom_daily_job_limit": custom_daily_job_limit,
"reason": reason,
"created_by": admin_email,
"created_at": datetime.now(timezone.utc),
})
logger.info(f"Set rate limit override for {user_email} by {admin_email}")
def remove_user_override(self, user_email: str, admin_email: str = "") -> bool:
"""Remove rate limit override for a user."""
doc_ref = self.db.collection(OVERRIDES_COLLECTION).document(user_email.lower())
doc = doc_ref.get()
if not doc.exists:
return False
doc_ref.delete()
logger.info(f"Removed rate limit override for {user_email} by {admin_email}")
return True
def list_user_overrides(self) -> List[Dict[str, Any]]:
"""List all user rate limit overrides."""
docs = self.db.collection(OVERRIDES_COLLECTION).stream()
return [doc.to_dict() for doc in docs]
def get_all_overrides(self) -> Dict[str, Dict[str, Any]]:
"""Get all user rate limit overrides as a dict keyed by email."""
docs = self.db.collection(OVERRIDES_COLLECTION).stream()
return {doc.id: doc.to_dict() for doc in docs}
# -------------------------------------------------------------------------
# Blocklist Management
# -------------------------------------------------------------------------
def _load_blocklist(self, force_refresh: bool = False) -> Dict[str, Any]:
"""
Load blocklist from Firestore with caching.
Returns:
Dict with keys: disposable_domains, blocked_emails, blocked_ips
"""
now = datetime.now(timezone.utc)
# Check cache
if not force_refresh and self._blocklist_cache and self._blocklist_cache_time:
cache_age = (now - self._blocklist_cache_time).total_seconds()
if cache_age < self._cache_ttl_seconds:
return self._blocklist_cache
# Load from Firestore
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc = doc_ref.get()
if not doc.exists:
# Initialize with empty blocklist
self._blocklist_cache = {
"disposable_domains": [],
"blocked_emails": [],
"blocked_ips": [],
}
else:
self._blocklist_cache = doc.to_dict()
self._blocklist_cache_time = now
return self._blocklist_cache
def is_disposable_domain(self, domain: str) -> bool:
"""Check if a domain is in the disposable domains blocklist."""
blocklist = self._load_blocklist()
return domain.lower() in [d.lower() for d in blocklist.get("disposable_domains", [])]
def is_blocked_email(self, email: str) -> bool:
"""Check if an email is explicitly blocked."""
blocklist = self._load_blocklist()
return email.lower() in [e.lower() for e in blocklist.get("blocked_emails", [])]
def is_blocked_ip(self, ip_address: str) -> bool:
"""Check if an IP address is blocked."""
blocklist = self._load_blocklist()
return ip_address in blocklist.get("blocked_ips", [])
def get_blocklist(self) -> Dict[str, Any]:
"""Get all blocklist data."""
return self._load_blocklist(force_refresh=True)
def add_disposable_domain(self, domain: str, admin_email: str) -> None:
"""Add a domain to the disposable domains blocklist."""
blocklist = self._load_blocklist(force_refresh=True)
domains = set(blocklist.get("disposable_domains", []))
domains.add(domain.lower())
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc_ref.set({
**blocklist,
"disposable_domains": list(domains),
"updated_at": datetime.now(timezone.utc),
"updated_by": admin_email,
}, merge=True)
self._blocklist_cache = None # Invalidate cache
logger.info(f"Added disposable domain {domain} by {admin_email}")
def remove_disposable_domain(self, domain: str) -> bool:
"""Remove a domain from the disposable domains blocklist."""
blocklist = self._load_blocklist(force_refresh=True)
domains = set(blocklist.get("disposable_domains", []))
if domain.lower() not in [d.lower() for d in domains]:
return False
domains = {d for d in domains if d.lower() != domain.lower()}
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc_ref.set({
**blocklist,
"disposable_domains": list(domains),
"updated_at": datetime.now(timezone.utc),
}, merge=True)
self._blocklist_cache = None
logger.info(f"Removed disposable domain {domain}")
return True
def add_blocked_email(self, email: str, admin_email: str) -> None:
"""Add an email to the blocked emails list."""
blocklist = self._load_blocklist(force_refresh=True)
emails = set(blocklist.get("blocked_emails", []))
emails.add(email.lower())
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc_ref.set({
**blocklist,
"blocked_emails": list(emails),
"updated_at": datetime.now(timezone.utc),
"updated_by": admin_email,
}, merge=True)
self._blocklist_cache = None
logger.info(f"Added blocked email {email} by {admin_email}")
def remove_blocked_email(self, email: str) -> bool:
"""Remove an email from the blocked emails list."""
blocklist = self._load_blocklist(force_refresh=True)
emails = set(blocklist.get("blocked_emails", []))
if email.lower() not in [e.lower() for e in emails]:
return False
emails = {e for e in emails if e.lower() != email.lower()}
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc_ref.set({
**blocklist,
"blocked_emails": list(emails),
"updated_at": datetime.now(timezone.utc),
}, merge=True)
self._blocklist_cache = None
logger.info(f"Removed blocked email {email}")
return True
def add_blocked_ip(self, ip_address: str, admin_email: str) -> None:
"""Add an IP address to the blocked IPs list."""
blocklist = self._load_blocklist(force_refresh=True)
ips = set(blocklist.get("blocked_ips", []))
ips.add(ip_address)
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc_ref.set({
**blocklist,
"blocked_ips": list(ips),
"updated_at": datetime.now(timezone.utc),
"updated_by": admin_email,
}, merge=True)
self._blocklist_cache = None
logger.info(f"Added blocked IP {ip_address} by {admin_email}")
def remove_blocked_ip(self, ip_address: str) -> bool:
"""Remove an IP address from the blocked IPs list."""
blocklist = self._load_blocklist(force_refresh=True)
ips = set(blocklist.get("blocked_ips", []))
if ip_address not in ips:
return False
ips.discard(ip_address)
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc_ref.set({
**blocklist,
"blocked_ips": list(ips),
"updated_at": datetime.now(timezone.utc),
}, merge=True)
self._blocklist_cache = None
logger.info(f"Removed blocked IP {ip_address}")
return True
# -------------------------------------------------------------------------
# Stats
# -------------------------------------------------------------------------
def get_rate_limit_stats(self) -> Dict[str, Any]:
"""
Get current rate limit statistics.
Returns:
Dict with current usage stats
"""
date_str = _get_today_date_str()
# Get YouTube upload count
youtube_count = self.get_youtube_uploads_today()
# Count unique users with jobs today
users_with_jobs = set()
total_jobs_today = 0
# Query all user job documents for today
docs = self.db.collection(RATE_LIMITS_COLLECTION).where(
"date", "==", date_str
).stream()
for doc in docs:
data = doc.to_dict()
if data.get("user_email"):
users_with_jobs.add(data["user_email"])
total_jobs_today += data.get("count", 0)
return {
"date": date_str,
"youtube_uploads_today": youtube_count,
"youtube_uploads_limit": settings.rate_limit_youtube_uploads_per_day,
"total_jobs_today": total_jobs_today,
"users_with_jobs_today": len(users_with_jobs),
"job_limit_per_user": settings.rate_limit_jobs_per_day,
"beta_ip_limit_per_day": settings.rate_limit_beta_ip_per_day,
"rate_limiting_enabled": settings.enable_rate_limiting,
"seconds_until_reset": _seconds_until_midnight_utc(),
}
# Singleton instance
_rate_limit_service: Optional[RateLimitService] = None
def get_rate_limit_service() -> RateLimitService:
"""Get the singleton RateLimitService instance."""
global _rate_limit_service
if _rate_limit_service is None:
_rate_limit_service = RateLimitService()
return _rate_limit_service
+526
View File
@@ -0,0 +1,526 @@
"""
Referral service for link management, earnings tracking, and payouts.
Handles:
- Referral link creation (generated and vanity codes)
- Link lookup and validation
- Click tracking
- Admin listing
"""
import logging
import re
import secrets
import string
from datetime import datetime, timedelta
from typing import List, Optional, Tuple
from google.cloud import firestore
from backend.models.referral import (
ReferralLink,
ReferralLinkStats,
ReferralEarning,
ReferralPayout,
)
logger = logging.getLogger(__name__)
# ============================================================================
# Constants
# ============================================================================
REFERRAL_LINKS_COLLECTION = "referral_links"
REFERRAL_EARNINGS_COLLECTION = "referral_earnings"
REFERRAL_PAYOUTS_COLLECTION = "referral_payouts"
RESERVED_CODES = frozenset({
"admin", "app", "api", "r", "pricing", "order", "payment",
"login", "auth", "webhook", "webhooks", "internal", "health",
"static", "assets", "public", "private",
})
PAYOUT_THRESHOLD_CENTS = 2000
VANITY_CODE_PATTERN = re.compile(r"^[a-z0-9][a-z0-9\-]{1,28}[a-z0-9]$")
# Characters for generated codes
_CODE_ALPHABET = string.ascii_lowercase + string.digits
# ============================================================================
# Service
# ============================================================================
class ReferralService:
"""Service for referral link management and earnings."""
def __init__(self, db=None, stripe_service=None):
if db is None:
db = firestore.Client()
self.db = db
self.stripe_service = stripe_service
# ========================================================================
# Code Generation & Validation
# ========================================================================
def _generate_code(self) -> str:
"""Generate an 8-character lowercase alphanumeric referral code."""
return "".join(secrets.choice(_CODE_ALPHABET) for _ in range(8))
def _validate_vanity_code(self, code: str) -> Tuple[bool, str]:
"""Validate a vanity code against reserved words and pattern rules.
Returns (is_valid, message).
"""
if code.lower() in RESERVED_CODES:
return False, f"'{code}' is a reserved code"
if not VANITY_CODE_PATTERN.match(code.lower()):
return False, (
"Code must be 3-30 characters, start/end with alphanumeric, "
"and contain only lowercase letters, digits, and hyphens"
)
return True, "Valid"
# ========================================================================
# Link CRUD
# ========================================================================
def get_link_by_code(self, code: str) -> Optional[ReferralLink]:
"""Get a referral link by its code. Returns None if not found or disabled."""
doc = self.db.collection(REFERRAL_LINKS_COLLECTION).document(code).get()
if not doc.exists:
return None
data = doc.to_dict()
if not data.get("enabled", True):
return None
return ReferralLink(**data)
def get_or_create_link(self, owner_email: str) -> ReferralLink:
"""Get existing referral link for owner, or create a new one.
Retries up to 10 times if generated code collides with existing doc.
"""
# Check for existing link
existing = (
self.db.collection(REFERRAL_LINKS_COLLECTION)
.where("owner_email", "==", owner_email)
.limit(1)
.stream()
)
for doc in existing:
return ReferralLink(**doc.to_dict())
# Create new link with retry for code collision
for attempt in range(10):
code = self._generate_code()
doc_ref = self.db.collection(REFERRAL_LINKS_COLLECTION).document(code)
doc_snapshot = doc_ref.get()
if doc_snapshot.exists:
logger.warning("Code collision on attempt %d: %s", attempt + 1, code)
continue
now = datetime.utcnow()
link = ReferralLink(
code=code,
owner_email=owner_email,
created_at=now,
updated_at=now,
)
doc_ref.set(link.model_dump())
return link
raise RuntimeError("Failed to generate unique referral code after 10 attempts")
def create_vanity_link(
self,
code: str,
owner_email: str,
display_name: Optional[str] = None,
custom_message: Optional[str] = None,
discount_percent: int = 10,
kickback_percent: int = 20,
discount_duration_days: int = 30,
earning_duration_days: int = 365,
) -> Tuple[bool, Optional[ReferralLink], str]:
"""Create a vanity referral link.
Returns (success, link_or_none, message).
"""
valid, msg = self._validate_vanity_code(code)
if not valid:
return False, None, msg
normalized = code.lower()
doc_ref = self.db.collection(REFERRAL_LINKS_COLLECTION).document(normalized)
doc_snapshot = doc_ref.get()
if doc_snapshot.exists:
return False, None, f"Code '{normalized}' is already taken"
now = datetime.utcnow()
link = ReferralLink(
code=normalized,
owner_email=owner_email,
display_name=display_name,
custom_message=custom_message,
discount_percent=discount_percent,
kickback_percent=kickback_percent,
discount_duration_days=discount_duration_days,
earning_duration_days=earning_duration_days,
is_vanity=True,
created_at=now,
updated_at=now,
)
doc_ref.set(link.model_dump())
return True, link, "Vanity link created"
def update_link(self, code: str, **updates) -> Tuple[bool, str]:
"""Update fields on an existing referral link.
Returns (success, message).
"""
doc_ref = self.db.collection(REFERRAL_LINKS_COLLECTION).document(code)
doc_snapshot = doc_ref.get()
if not doc_snapshot.exists:
return False, f"Link '{code}' not found"
updates["updated_at"] = datetime.utcnow()
doc_ref.update(updates)
return True, "Link updated"
def increment_clicks(self, code: str) -> None:
"""Increment the click counter for a referral link."""
doc_ref = self.db.collection(REFERRAL_LINKS_COLLECTION).document(code)
doc_ref.update({"stats.clicks": firestore.Increment(1)})
# ========================================================================
# Attribution
# ========================================================================
def attribute_referral(self, referred_email: str, referral_code: str) -> Tuple[bool, str]:
"""Attribute a referral to a user. Called during first login/signup."""
referred_email = referred_email.lower()
referral_code = referral_code.lower()
link = self.get_link_by_code(referral_code)
if not link:
return False, "Invalid or disabled referral code"
if link.owner_email == referred_email:
return False, "Cannot use your own referral code"
# Increment signup count
self.db.collection(REFERRAL_LINKS_COLLECTION).document(referral_code).update({
"stats.signups": firestore.Increment(1),
})
return True, "Referral attributed"
def get_attribution_data(self, referral_code: str) -> Optional[dict]:
"""Get the data to set on user doc for referral attribution."""
link = self.get_link_by_code(referral_code)
if not link:
return None
now = datetime.utcnow()
return {
"referred_by_code": referral_code.lower(),
"referred_at": now,
"referral_discount_expires_at": now + timedelta(days=link.discount_duration_days),
}
def get_or_create_stripe_coupon(self, discount_percent: int) -> Optional[str]:
if not self.stripe_service:
return None
return self.stripe_service.get_or_create_referral_coupon(discount_percent)
def should_apply_discount(self, user_data: dict) -> bool:
if not user_data.get("referred_by_code"):
return False
expires_at = user_data.get("referral_discount_expires_at")
if not expires_at:
return False
if isinstance(expires_at, datetime):
now = datetime.utcnow()
# Firestore returns timezone-aware datetimes; ensure comparison is compatible
if expires_at.tzinfo is not None:
now = now.replace(tzinfo=expires_at.tzinfo)
return expires_at > now
return False
def get_discount_for_checkout(self, user_email: str) -> Optional[dict]:
from backend.services.user_service import get_user_service
user_service = get_user_service()
user = user_service.get_user(user_email)
if not user or not user.referred_by_code:
return None
user_data = {
"referred_by_code": user.referred_by_code,
"referral_discount_expires_at": user.referral_discount_expires_at,
}
if not self.should_apply_discount(user_data):
return None
link = self.get_link_by_code(user.referred_by_code)
if not link:
return None
coupon_id = self.get_or_create_stripe_coupon(link.discount_percent)
if not coupon_id:
return None
return {"coupon_id": coupon_id, "discount_percent": link.discount_percent}
# ========================================================================
# Earnings
# ========================================================================
def record_earning(self, referred_email: str, stripe_session_id: str, purchase_amount_cents: int) -> Optional[dict]:
"""Record a referral earning after a credit purchase."""
referred_email = referred_email.lower()
# Look up the referred user
user_doc = self.db.collection("gen_users").document(referred_email).get()
if not user_doc.exists:
return None
user_data = user_doc.to_dict()
referral_code = user_data.get("referred_by_code")
if not referral_code:
return None
referred_at = user_data.get("referred_at")
if not referred_at:
return None
link = self.get_link_by_code(referral_code)
if not link:
return None
# Check earning window
earning_expires = referred_at + timedelta(days=link.earning_duration_days)
if datetime.utcnow() > earning_expires:
return None
# Calculate earning
earning_amount = int(purchase_amount_cents * link.kickback_percent / 100)
if earning_amount <= 0:
return None
import uuid
earning_id = str(uuid.uuid4())
earning = ReferralEarning(
id=earning_id,
referrer_email=link.owner_email,
referred_email=referred_email,
referral_code=referral_code,
stripe_session_id=stripe_session_id,
purchase_amount_cents=purchase_amount_cents,
earning_amount_cents=earning_amount,
)
self.db.collection(REFERRAL_EARNINGS_COLLECTION).document(earning_id).set(
earning.model_dump(mode="json")
)
self.db.collection(REFERRAL_LINKS_COLLECTION).document(referral_code).update({
"stats.purchases": firestore.Increment(1),
"stats.total_earned_cents": firestore.Increment(earning_amount),
})
logger.info(f"Referral earning recorded: ${earning_amount / 100:.2f} for {link.owner_email}")
return {"earning_id": earning_id, "referrer_email": link.owner_email, "earning_amount_cents": earning_amount}
# ========================================================================
# Stripe Connect & Payouts
# ========================================================================
def create_connect_account(self, email: str) -> Tuple[Optional[str], Optional[str]]:
"""Create Stripe Connect account for a referrer."""
if not self.stripe_service:
return None, None
return self.stripe_service.create_connect_account(email)
def get_connect_account_status(self, account_id: str) -> Optional[dict]:
"""Get Stripe Connect account status."""
if not self.stripe_service:
return None
return self.stripe_service.get_connect_account_status(account_id)
def create_connect_login_link(self, account_id: str) -> Optional[str]:
"""Create login link for Stripe Express dashboard."""
if not self.stripe_service:
return None
return self.stripe_service.create_connect_login_link(account_id)
def create_connect_update_link(self, account_id: str) -> Optional[str]:
"""Create account link for updating Connect info."""
if not self.stripe_service:
return None
return self.stripe_service.create_connect_update_link(account_id)
def get_pending_earnings(self, referrer_email: str) -> List[dict]:
"""Get all pending earnings for a referrer."""
query = (
self.db.collection(REFERRAL_EARNINGS_COLLECTION)
.where("referrer_email", "==", referrer_email.lower())
.where("status", "==", "pending")
)
return [
{**doc.to_dict(), "_ref": doc.reference}
for doc in query.get()
]
def check_and_trigger_payout(self, referrer_email: str, stripe_connect_account_id: str) -> bool:
"""Check if pending earnings meet threshold and trigger payout."""
pending = self.get_pending_earnings(referrer_email)
total_pending = sum(e.get("earning_amount_cents", 0) for e in pending)
if total_pending < PAYOUT_THRESHOLD_CENTS:
return False
if not self.stripe_service:
return False
transfer_id = self.stripe_service.create_transfer(
amount_cents=total_pending,
destination_account_id=stripe_connect_account_id,
)
if not transfer_id:
return False
import uuid
payout_id = str(uuid.uuid4())
earning_ids = [e["id"] for e in pending]
payout = ReferralPayout(
id=payout_id,
referrer_email=referrer_email.lower(),
stripe_transfer_id=transfer_id,
amount_cents=total_pending,
earnings_included=earning_ids,
)
self.db.collection(REFERRAL_PAYOUTS_COLLECTION).document(payout_id).set(
payout.model_dump(mode="json")
)
for earning in pending:
if "_ref" in earning:
earning["_ref"].update({
"status": "paid",
"paid_at": datetime.utcnow(),
"payout_id": payout_id,
})
logger.info(f"Payout triggered: ${total_pending / 100:.2f} to {referrer_email} (transfer: {transfer_id})")
return True
def get_dashboard_data(self, user_email: str) -> dict:
"""Get full referral dashboard data for a user."""
user_email = user_email.lower()
link = self.get_or_create_link(user_email)
# Get recent earnings (last 20)
earnings_query = (
self.db.collection(REFERRAL_EARNINGS_COLLECTION)
.where("referrer_email", "==", user_email)
.order_by("created_at", direction=firestore.Query.DESCENDING)
.limit(20)
)
earnings = [doc.to_dict() for doc in earnings_query.stream()]
# Get recent payouts (last 10)
payouts_query = (
self.db.collection(REFERRAL_PAYOUTS_COLLECTION)
.where("referrer_email", "==", user_email)
.order_by("created_at", direction=firestore.Query.DESCENDING)
.limit(10)
)
payouts = [doc.to_dict() for doc in payouts_query.stream()]
# Compute pending balance from ALL pending earnings (not limited to recent)
pending_earnings = self.get_pending_earnings(user_email)
pending_balance = sum(e.get("earning_amount_cents", 0) for e in pending_earnings)
total_earned = link.stats.total_earned_cents
# total_paid = total_earned - pending (since total_earned tracks all time)
total_paid = total_earned - pending_balance
# Check Connect status and fetch account details
from backend.services.user_service import get_user_service
user = get_user_service().get_user(user_email)
has_connect = bool(user and user.stripe_connect_account_id)
connect_account = None
if has_connect and self.stripe_service:
connect_account = self.stripe_service.get_connect_account_status(
user.stripe_connect_account_id
)
return {
"link": {
"code": link.code,
"display_name": link.display_name,
"custom_message": link.custom_message,
"discount_percent": link.discount_percent,
"kickback_percent": link.kickback_percent,
"discount_duration_days": link.discount_duration_days,
"earning_duration_days": link.earning_duration_days,
"stats": link.stats.model_dump(),
"enabled": link.enabled,
"is_vanity": link.is_vanity,
},
"pending_balance_cents": pending_balance,
"total_earned_cents": total_earned,
"total_paid_cents": total_paid,
"recent_earnings": [
{
"id": e["id"],
"referred_email": e.get("referred_email", ""),
"purchase_amount_cents": e.get("purchase_amount_cents", 0),
"earning_amount_cents": e.get("earning_amount_cents", 0),
"status": e.get("status", "pending"),
"created_at": str(e.get("created_at", "")),
}
for e in earnings
],
"recent_payouts": [
{
"id": p["id"],
"amount_cents": p.get("amount_cents", 0),
"status": p.get("status", "processing"),
"created_at": str(p.get("created_at", "")),
}
for p in payouts
],
"stripe_connect_configured": has_connect,
"stripe_connect_account": connect_account,
}
def list_links(self, limit: int = 50, offset: int = 0) -> list[ReferralLink]:
"""List referral links for admin view."""
query = (
self.db.collection(REFERRAL_LINKS_COLLECTION)
.order_by("created_at", direction=firestore.Query.DESCENDING)
.offset(offset)
.limit(limit)
)
links = []
for doc in query.stream():
try:
links.append(ReferralLink(**doc.to_dict()))
except Exception:
logger.warning("Skipping malformed referral link doc: %s", doc.id)
return links
# ============================================================================
# Singleton accessor
# ============================================================================
_referral_service = None
def get_referral_service() -> ReferralService:
global _referral_service
if _referral_service is None:
from backend.services.stripe_service import get_stripe_service
_referral_service = ReferralService(stripe_service=get_stripe_service())
return _referral_service
+759
View File
@@ -0,0 +1,759 @@
"""
Stripe admin service for payment analytics and management.
Handles:
- Storing enriched payment data from webhooks
- Revenue analytics and reporting
- Payment listing and detail views
- Stripe balance and payout queries
- Refund processing
- Dispute monitoring
- Webhook event logging
"""
import logging
import time
from datetime import datetime, timedelta
from typing import Optional, Dict, Any, List, Tuple
import stripe
from google.cloud import firestore
from backend.config import get_settings
logger = logging.getLogger(__name__)
STRIPE_PAYMENTS_COLLECTION = "stripe_payments"
STRIPE_WEBHOOK_EVENTS_COLLECTION = "stripe_webhook_events"
class StripeAdminService:
"""Service for Stripe admin analytics and payment management."""
def __init__(self):
self.settings = get_settings()
self.db = firestore.Client()
self._revenue_cache: Optional[Dict[str, Any]] = None
self._revenue_cache_time: float = 0
self._revenue_cache_ttl: float = 300 # 5 minutes
# =========================================================================
# Payment Storage
# =========================================================================
def store_payment(self, session: Dict[str, Any]) -> bool:
"""
Store enriched payment data from a completed Checkout Session.
Args:
session: Stripe Checkout Session object (expanded with payment_intent,
charge, and balance_transaction)
Returns:
True if stored successfully
"""
try:
session_id = session.get("id", "")
metadata = session.get("metadata", {})
# Extract payment intent and charge details
payment_intent = session.get("payment_intent", {})
if isinstance(payment_intent, str):
# Not expanded - just store the ID
payment_intent_id = payment_intent
charge_data = {}
balance_txn = {}
else:
payment_intent_id = payment_intent.get("id", "")
charges = payment_intent.get("latest_charge", {})
if isinstance(charges, str):
charge_data = {}
balance_txn = {}
else:
charge_data = charges or {}
bt = charge_data.get("balance_transaction", {})
balance_txn = bt if isinstance(bt, dict) else {}
# Extract payment method details
payment_method_details = charge_data.get("payment_method_details", {})
pm_type = payment_method_details.get("type", "unknown")
card_info = payment_method_details.get("card", {})
# Determine order type
order_type = metadata.get("order_type", "credit_purchase")
package_id = metadata.get("package_id")
credits = 0
try:
credits = int(metadata.get("credits", 0))
except (ValueError, TypeError):
pass
# Build product description
if order_type == "made_for_you":
artist = metadata.get("artist", "")
title = metadata.get("title", "")
product_description = f"Made For You: {artist} - {title}"
else:
from backend.services.stripe_service import CREDIT_PACKAGES
pkg = CREDIT_PACKAGES.get(package_id, {})
product_description = pkg.get("name", f"{credits} Credits")
# Extract discount info
total_details = session.get("total_details", {})
discount_amount = 0
promotion_code = None
if total_details:
discount_obj = total_details.get("amount_discount", 0)
discount_amount = discount_obj if isinstance(discount_obj, int) else 0
# Determine if test payment
is_test = not (session.get("livemode", False))
payment_data = {
"session_id": session_id,
"payment_intent_id": payment_intent_id,
"charge_id": charge_data.get("id", ""),
# Financial (cents)
"amount_total": session.get("amount_total", 0),
"currency": session.get("currency", "usd"),
"stripe_fee": balance_txn.get("fee", 0),
"net_amount": balance_txn.get("net", 0),
# Customer
"customer_email": (
metadata.get("user_email")
or metadata.get("customer_email")
or session.get("customer_email", "")
).lower().strip(),
"customer_name": session.get("customer_details", {}).get("name", ""),
"stripe_customer_id": session.get("customer", ""),
# Payment method
"payment_method_type": pm_type,
"card_brand": card_info.get("brand", ""),
"card_last4": card_info.get("last4", ""),
# Order details
"order_type": order_type,
"package_id": package_id,
"credits_granted": credits,
"product_description": product_description,
# Made-for-you fields
"artist": metadata.get("artist"),
"title": metadata.get("title"),
"job_id": metadata.get("job_id"),
# Status
"status": "succeeded",
"refund_amount": 0,
"refund_id": None,
"refunded_at": None,
"refund_reason": None,
# Timestamps
"created_at": datetime.utcfromtimestamp(session["created"]) if session.get("created") else datetime.utcnow(),
"processed_at": datetime.utcnow(),
# Metadata
"is_test": is_test,
"promotion_code": promotion_code,
"discount_amount": discount_amount,
}
doc_ref = self.db.collection(STRIPE_PAYMENTS_COLLECTION).document(session_id)
doc_ref.set(payment_data)
# Invalidate revenue cache
self._revenue_cache = None
logger.info(
f"Stored payment data for session {session_id}: "
f"${payment_data['amount_total'] / 100:.2f} from {payment_data['customer_email']}"
)
return True
except Exception:
logger.exception(f"Error storing payment data for session {session.get('id', 'unknown')}")
return False
def update_payment_status(self, session_id: str, updates: Dict[str, Any]) -> bool:
"""Update payment status fields (for refunds/disputes)."""
try:
doc_ref = self.db.collection(STRIPE_PAYMENTS_COLLECTION).document(session_id)
doc = doc_ref.get()
if not doc.exists:
logger.warning(f"Payment {session_id} not found for status update")
return False
doc_ref.update(updates)
self._revenue_cache = None
return True
except Exception:
logger.exception(f"Error updating payment status for {session_id}")
return False
# =========================================================================
# Revenue Analytics
# =========================================================================
def get_revenue_summary(
self, days: int = 30, exclude_test: bool = True
) -> Dict[str, Any]:
"""
Get revenue summary metrics.
Returns total gross, fees, net, refunds, count, AOV, revenue by product type.
Uses in-memory cache with 5-minute TTL.
"""
cache_key = f"{days}_{exclude_test}"
now = time.time()
if (
self._revenue_cache
and self._revenue_cache.get("_key") == cache_key
and (now - self._revenue_cache_time) < self._revenue_cache_ttl
):
return self._revenue_cache
try:
collection = self.db.collection(STRIPE_PAYMENTS_COLLECTION)
query = collection
if days > 0:
cutoff = datetime.utcnow() - timedelta(days=days)
query = query.where("created_at", ">=", cutoff)
total_gross = 0
total_fees = 0
total_net = 0
total_refunds = 0
count = 0
by_type: Dict[str, int] = {}
for doc in query.stream():
data = doc.to_dict()
if exclude_test and data.get("is_test"):
continue
amount = data.get("amount_total", 0)
total_gross += amount
total_fees += data.get("stripe_fee", 0)
total_net += data.get("net_amount", 0)
total_refunds += data.get("refund_amount", 0)
count += 1
order_type = data.get("order_type", "credit_purchase")
by_type[order_type] = by_type.get(order_type, 0) + amount
aov = total_gross / count if count > 0 else 0
result = {
"_key": cache_key,
"total_gross": total_gross,
"total_fees": total_fees,
"total_net": total_net,
"total_refunds": total_refunds,
"transaction_count": count,
"average_order_value": round(aov),
"revenue_by_type": by_type,
}
self._revenue_cache = result
self._revenue_cache_time = now
return result
except Exception:
logger.exception("Error calculating revenue summary")
return {
"total_gross": 0,
"total_fees": 0,
"total_net": 0,
"total_refunds": 0,
"transaction_count": 0,
"average_order_value": 0,
"revenue_by_type": {},
}
def get_revenue_by_period(
self,
days: int = 30,
group_by: str = "day",
exclude_test: bool = True,
) -> List[Dict[str, Any]]:
"""
Get revenue grouped by time period for charting.
Args:
days: Number of days to look back
group_by: 'day' or 'week' or 'month'
exclude_test: Exclude test transactions
Returns:
List of {date, gross, net, fees, count} dicts
"""
try:
collection = self.db.collection(STRIPE_PAYMENTS_COLLECTION)
cutoff = datetime.utcnow() - timedelta(days=days)
query = collection.where("created_at", ">=", cutoff)
buckets: Dict[str, Dict[str, Any]] = {}
for doc in query.stream():
data = doc.to_dict()
if exclude_test and data.get("is_test"):
continue
created = data.get("created_at")
if not created:
continue
if isinstance(created, str):
try:
created = datetime.fromisoformat(created.replace("Z", "+00:00"))
except Exception:
continue
if group_by == "week":
# Start of week (Monday)
start = created - timedelta(days=created.weekday())
key = start.strftime("%Y-%m-%d")
elif group_by == "month":
key = created.strftime("%Y-%m-01")
else:
key = created.strftime("%Y-%m-%d")
if key not in buckets:
buckets[key] = {"date": key, "gross": 0, "net": 0, "fees": 0, "count": 0}
buckets[key]["gross"] += data.get("amount_total", 0)
buckets[key]["net"] += data.get("net_amount", 0)
buckets[key]["fees"] += data.get("stripe_fee", 0)
buckets[key]["count"] += 1
result = sorted(buckets.values(), key=lambda x: x["date"])
return result
except Exception:
logger.exception("Error getting revenue by period")
return []
# =========================================================================
# Payment Listing & Details
# =========================================================================
def list_payments(
self,
limit: int = 50,
offset: int = 0,
order_type: Optional[str] = None,
status: Optional[str] = None,
email: Optional[str] = None,
exclude_test: bool = True,
) -> Dict[str, Any]:
"""
List payments with pagination and filters.
Returns:
{payments: [...], total: int, has_more: bool}
"""
try:
collection = self.db.collection(STRIPE_PAYMENTS_COLLECTION)
query = collection.order_by("created_at", direction=firestore.Query.DESCENDING)
if order_type:
query = collection.where("order_type", "==", order_type).order_by(
"created_at", direction=firestore.Query.DESCENDING
)
if status:
query = collection.where("status", "==", status).order_by(
"created_at", direction=firestore.Query.DESCENDING
)
if email:
query = collection.where("customer_email", "==", email.lower()).order_by(
"created_at", direction=firestore.Query.DESCENDING
)
# Stream and filter (Firestore doesn't support multiple inequality filters)
payments = []
skipped = 0
total = 0
for doc in query.limit(1000).stream():
data = doc.to_dict()
data["session_id"] = doc.id
if exclude_test and data.get("is_test"):
continue
# Apply filters that couldn't be done in query
if order_type and data.get("order_type") != order_type:
continue
if status and data.get("status") != status:
continue
if email and data.get("customer_email", "").lower() != email.lower():
continue
total += 1
if skipped < offset:
skipped += 1
continue
if len(payments) < limit:
# Convert datetime objects for JSON serialization
for key in ["created_at", "processed_at", "refunded_at"]:
if isinstance(data.get(key), datetime):
data[key] = data[key].isoformat()
payments.append(data)
return {
"payments": payments,
"total": total,
"has_more": total > offset + limit,
}
except Exception:
logger.exception("Error listing payments")
return {"payments": [], "total": 0, "has_more": False}
def get_payment_detail(self, session_id: str) -> Optional[Dict[str, Any]]:
"""
Get full payment details for a single payment.
Combines Firestore data with on-demand Stripe API call for receipt_url.
"""
try:
doc_ref = self.db.collection(STRIPE_PAYMENTS_COLLECTION).document(session_id)
doc = doc_ref.get()
if not doc.exists:
return None
data = doc.to_dict()
data["session_id"] = doc.id
# Convert datetime objects
for key in ["created_at", "processed_at", "refunded_at"]:
if isinstance(data.get(key), datetime):
data[key] = data[key].isoformat()
# On-demand: fetch receipt URL from Stripe
charge_id = data.get("charge_id")
if charge_id:
try:
charge = stripe.Charge.retrieve(charge_id)
data["receipt_url"] = charge.get("receipt_url")
data["stripe_dashboard_url"] = (
f"https://dashboard.stripe.com/payments/{data.get('payment_intent_id', '')}"
)
except Exception as e:
logger.warning(f"Could not fetch charge {charge_id}: {e}")
data["receipt_url"] = None
return data
except Exception:
logger.exception(f"Error getting payment detail for {session_id}")
return None
def get_user_payment_history(self, email: str) -> Dict[str, Any]:
"""Get all payments for a user with total_spent."""
try:
collection = self.db.collection(STRIPE_PAYMENTS_COLLECTION)
# Try with order_by first (requires composite index on customer_email + created_at)
try:
query = (
collection.where("customer_email", "==", email.lower())
.order_by("created_at", direction=firestore.Query.DESCENDING)
)
docs = list(query.stream())
except Exception:
# Fallback: query without order_by if composite index doesn't exist yet
logger.warning(
f"Composite index missing for stripe_payments(customer_email, created_at). "
f"Falling back to unordered query for {email}"
)
query = collection.where("customer_email", "==", email.lower())
docs = list(query.stream())
payments = []
total_spent = 0
total_refunded = 0
for doc in docs:
data = doc.to_dict()
data["session_id"] = doc.id
total_spent += data.get("amount_total", 0)
total_refunded += data.get("refund_amount", 0)
for key in ["created_at", "processed_at", "refunded_at"]:
if isinstance(data.get(key), datetime):
data[key] = data[key].isoformat()
payments.append(data)
# Sort by created_at descending (handles both indexed and fallback paths)
payments.sort(key=lambda p: p.get("created_at", ""), reverse=True)
first_payment = payments[-1]["created_at"] if payments else None
last_payment = payments[0]["created_at"] if payments else None
return {
"email": email,
"payments": payments,
"total_spent": total_spent,
"total_refunded": total_refunded,
"net_spent": total_spent - total_refunded,
"payment_count": len(payments),
"first_payment_at": first_payment,
"last_payment_at": last_payment,
}
except Exception:
logger.exception(f"Error getting payment history for {email}")
return {
"email": email,
"payments": [],
"total_spent": 0,
"total_refunded": 0,
"net_spent": 0,
"payment_count": 0,
"first_payment_at": None,
"last_payment_at": None,
}
# =========================================================================
# Stripe Balance & Payouts
# =========================================================================
def get_stripe_balance(self) -> Dict[str, Any]:
"""Get current Stripe account balance (available + pending)."""
try:
balance = stripe.Balance.retrieve()
available = sum(b.get("amount", 0) for b in balance.get("available", []))
pending = sum(b.get("amount", 0) for b in balance.get("pending", []))
return {
"available": available,
"pending": pending,
"currency": "usd",
}
except Exception:
logger.exception("Error fetching Stripe balance")
return {"available": 0, "pending": 0, "currency": "usd"}
def get_recent_payouts(self, limit: int = 10) -> List[Dict[str, Any]]:
"""Get recent payouts from Stripe."""
try:
payouts = stripe.Payout.list(limit=limit)
result = []
for p in payouts.get("data", []):
result.append({
"id": p.get("id"),
"amount": p.get("amount"),
"currency": p.get("currency"),
"status": p.get("status"),
"arrival_date": p.get("arrival_date"),
"created": p.get("created"),
"description": p.get("description"),
"method": p.get("method"),
})
return result
except Exception:
logger.exception("Error fetching payouts")
return []
# =========================================================================
# Disputes
# =========================================================================
def get_recent_disputes(self) -> List[Dict[str, Any]]:
"""Get open and recent disputes from Stripe."""
try:
disputes = stripe.Dispute.list(limit=20)
result = []
for d in disputes.get("data", []):
result.append({
"id": d.get("id"),
"amount": d.get("amount"),
"currency": d.get("currency"),
"status": d.get("status"),
"reason": d.get("reason"),
"charge_id": d.get("charge"),
"created": d.get("created"),
"evidence_due_by": d.get("evidence_details", {}).get("due_by"),
"payment_intent_id": d.get("payment_intent"),
})
return result
except Exception:
logger.exception("Error fetching disputes")
return []
# =========================================================================
# Refund Processing
# =========================================================================
def process_refund(
self,
session_id: str,
amount: Optional[int] = None,
reason: str = "requested_by_customer",
) -> Tuple[bool, str]:
"""
Process a refund for a payment.
Args:
session_id: Checkout session ID
amount: Refund amount in cents (None for full refund)
reason: Refund reason
Returns:
(success, message)
"""
try:
# Get payment record
doc_ref = self.db.collection(STRIPE_PAYMENTS_COLLECTION).document(session_id)
doc = doc_ref.get()
if not doc.exists:
return False, "Payment not found"
data = doc.to_dict()
payment_intent_id = data.get("payment_intent_id")
if not payment_intent_id:
return False, "No payment intent ID found"
current_refund = data.get("refund_amount", 0)
original_amount = data.get("amount_total", 0)
refund_amount = amount or (original_amount - current_refund)
if refund_amount <= 0:
return False, "Invalid refund amount"
if current_refund + refund_amount > original_amount:
return False, f"Refund would exceed original amount (${original_amount / 100:.2f})"
# Create Stripe refund
refund_params: Dict[str, Any] = {
"payment_intent": payment_intent_id,
"amount": refund_amount,
"reason": reason,
}
refund = stripe.Refund.create(**refund_params)
# Update Firestore record
new_total_refund = current_refund + refund_amount
new_status = "refunded" if new_total_refund >= original_amount else "partially_refunded"
doc_ref.update({
"status": new_status,
"refund_amount": new_total_refund,
"refund_id": refund.get("id"),
"refunded_at": datetime.utcnow(),
"refund_reason": reason,
})
self._revenue_cache = None
logger.info(
f"Processed refund for {session_id}: "
f"${refund_amount / 100:.2f} ({new_status})"
)
return True, f"Refund of ${refund_amount / 100:.2f} processed successfully"
except stripe.error.StripeError as e:
logger.error(f"Stripe refund error for {session_id}: {e}")
return False, f"Stripe error: {str(e)}"
except Exception:
logger.exception(f"Error processing refund for {session_id}")
return False, "Unexpected error processing refund"
# =========================================================================
# Webhook Event Logging
# =========================================================================
def log_webhook_event(
self,
event_id: str,
event_type: str,
status: str = "processing",
session_id: Optional[str] = None,
customer_email: Optional[str] = None,
summary: Optional[str] = None,
error_message: Optional[str] = None,
) -> None:
"""Log a webhook event for audit trail."""
try:
doc_ref = self.db.collection(STRIPE_WEBHOOK_EVENTS_COLLECTION).document(event_id)
data = {
"event_id": event_id,
"event_type": event_type,
"created_at": datetime.utcnow(),
"status": status,
"session_id": session_id,
"customer_email": customer_email,
"summary": summary,
"error_message": error_message,
}
if status in ("success", "error"):
data["processed_at"] = datetime.utcnow()
doc_ref.set(data, merge=True)
except Exception:
logger.exception(f"Error logging webhook event {event_id}")
def list_webhook_events(
self,
limit: int = 50,
event_type: Optional[str] = None,
status: Optional[str] = None,
) -> List[Dict[str, Any]]:
"""List recent webhook events."""
try:
collection = self.db.collection(STRIPE_WEBHOOK_EVENTS_COLLECTION)
query = collection.order_by("created_at", direction=firestore.Query.DESCENDING)
if event_type:
query = collection.where("event_type", "==", event_type).order_by(
"created_at", direction=firestore.Query.DESCENDING
)
events = []
for doc in query.limit(limit).stream():
data = doc.to_dict()
if status and data.get("status") != status:
continue
for key in ["created_at", "processed_at"]:
if isinstance(data.get(key), datetime):
data[key] = data[key].isoformat()
events.append(data)
return events
except Exception:
logger.exception("Error listing webhook events")
return []
def normalize_customer_emails(self) -> Dict[str, Any]:
"""Backfill: lowercase all customer_email values in stripe_payments."""
try:
collection = self.db.collection(STRIPE_PAYMENTS_COLLECTION)
updated = 0
skipped = 0
for doc in collection.stream():
data = doc.to_dict()
email = data.get("customer_email", "")
normalized = email.lower().strip()
if email != normalized:
doc.reference.update({"customer_email": normalized})
updated += 1
else:
skipped += 1
return {
"status": "success",
"updated": updated,
"skipped": skipped,
"total": updated + skipped,
}
except Exception:
logger.exception("Error normalizing customer emails")
raise
# Global instance
_stripe_admin_service = None
def get_stripe_admin_service() -> StripeAdminService:
"""Get the global Stripe admin service instance."""
global _stripe_admin_service
if _stripe_admin_service is None:
_stripe_admin_service = StripeAdminService()
return _stripe_admin_service
+122 -6
View File
@@ -6,6 +6,7 @@ Handles:
- Processing webhook events
- Managing customer records
"""
import json
import logging
import os
from typing import Optional, Dict, Any, Tuple
@@ -22,25 +23,25 @@ logger = logging.getLogger(__name__)
CREDIT_PACKAGES = {
"1_credit": {
"credits": 1,
"price_cents": 500, # $5.00
"price_cents": 1000, # $10.00
"name": "1 Karaoke Credit",
"description": "Create 1 professional karaoke video",
},
"3_credits": {
"credits": 3,
"price_cents": 1200, # $12.00 (20% discount)
"price_cents": 2400, # $24.00 (20% discount)
"name": "3 Karaoke Credits",
"description": "Create 3 professional karaoke videos (Save 20%)",
},
"5_credits": {
"credits": 5,
"price_cents": 1750, # $17.50 (30% discount)
"price_cents": 3500, # $35.00 (30% discount)
"name": "5 Karaoke Credits",
"description": "Create 5 professional karaoke videos (Save 30%)",
},
"10_credits": {
"credits": 10,
"price_cents": 3000, # $30.00 (40% discount)
"price_cents": 6000, # $60.00 (40% discount)
"name": "10 Karaoke Credits",
"description": "Create 10 professional karaoke videos (Save 40%)",
},
@@ -48,7 +49,7 @@ CREDIT_PACKAGES = {
# Made-for-you service package (not a credit package - creates a job directly)
MADE_FOR_YOU_PACKAGE = {
"price_cents": 1500, # $15.00
"price_cents": 5000, # $50.00
"name": "Made For You Karaoke Video",
"description": "Professional 4K karaoke video with perfectly synced lyrics, delivered to your email within 24 hours",
"images": ["https://gen.nomadkaraoke.com/nomad-logo.png"],
@@ -96,6 +97,7 @@ class StripeService:
user_email: str,
success_url: Optional[str] = None,
cancel_url: Optional[str] = None,
coupon_id: Optional[str] = None,
) -> Tuple[bool, Optional[str], str]:
"""
Create a Stripe Checkout session for purchasing credits.
@@ -151,6 +153,11 @@ class StripeService:
'allow_promotion_codes': True,
}
# If a referral coupon is applied, use discounts instead of allow_promotion_codes
if coupon_id:
session_params.pop('allow_promotion_codes', None)
session_params['discounts'] = [{'coupon': coupon_id}]
# Add payment method configuration if set (enables Google Pay, Link, etc.)
if self.payment_method_config:
session_params['payment_method_configuration'] = self.payment_method_config
@@ -284,7 +291,10 @@ class StripeService:
event = stripe.Webhook.construct_event(
payload, signature, self.webhook_secret
)
return True, event, "Webhook verified"
# Convert Stripe Event object to plain dict — Stripe SDK v14+
# returns StripeObject instances that don't support .get()
event_dict = json.loads(str(event))
return True, event_dict, "Webhook verified"
except stripe.error.SignatureVerificationError as e:
logger.error(f"Invalid webhook signature: {e}")
return False, None, "Invalid signature"
@@ -351,6 +361,112 @@ class StripeService:
logger.error(f"Error retrieving session {session_id}: {e}")
return None
def get_or_create_referral_coupon(self, discount_percent: int) -> Optional[str]:
"""Get or create a Stripe coupon for a referral discount percentage."""
if not self.is_configured():
return None
coupon_id = f"referral-{discount_percent}pct"
try:
stripe.Coupon.retrieve(coupon_id)
return coupon_id
except stripe.error.InvalidRequestError:
try:
stripe.Coupon.create(
id=coupon_id,
percent_off=discount_percent,
duration="once",
name=f"Referral {discount_percent}% off",
)
logger.info(f"Created Stripe coupon: {coupon_id}")
return coupon_id
except stripe.error.StripeError as e:
logger.error(f"Failed to create coupon {coupon_id}: {e}")
return None
def create_connect_account(self, email: str) -> Tuple[Optional[str], Optional[str]]:
"""Create a Stripe Connect Express account. Returns (account_id, onboarding_url)."""
if not self.is_configured():
return None, None
try:
account = stripe.Account.create(
type="express",
email=email,
capabilities={"transfers": {"requested": True}},
metadata={"source": "nomad_karaoke_referrals"},
)
account_link = stripe.AccountLink.create(
account=account.id,
refresh_url=f"{self.frontend_url}/app/referrals?connect=refresh",
return_url=f"{self.frontend_url}/app/referrals?connect=complete",
type="account_onboarding",
)
logger.info(f"Created Connect account {account.id} for {email}")
return account.id, account_link.url
except stripe.error.StripeError as e:
logger.error(f"Failed to create Connect account for {email}: {e}")
return None, None
def get_connect_account_status(self, account_id: str) -> Optional[dict]:
"""Retrieve Stripe Connect account status. Returns dict with account details."""
if not self.is_configured():
return None
try:
account = stripe.Account.retrieve(account_id)
return {
"account_id": account.id,
"charges_enabled": account.charges_enabled,
"payouts_enabled": account.payouts_enabled,
"details_submitted": account.details_submitted,
"email": account.email,
}
except stripe.error.StripeError as e:
logger.error(f"Failed to retrieve Connect account {account_id}: {e}")
return None
def create_connect_login_link(self, account_id: str) -> Optional[str]:
"""Create a login link to the Stripe Express dashboard. Returns URL."""
if not self.is_configured():
return None
try:
link = stripe.Account.create_login_link(account_id)
return link.url
except stripe.error.StripeError as e:
logger.error(f"Failed to create login link for {account_id}: {e}")
return None
def create_connect_update_link(self, account_id: str) -> Optional[str]:
"""Create an account link for updating Connect account info. Returns URL."""
if not self.is_configured():
return None
try:
account_link = stripe.AccountLink.create(
account=account_id,
refresh_url=f"{self.frontend_url}/app/referrals?connect=refresh",
return_url=f"{self.frontend_url}/app/referrals?connect=complete",
type="account_onboarding",
)
return account_link.url
except stripe.error.StripeError as e:
logger.error(f"Failed to create update link for {account_id}: {e}")
return None
def create_transfer(self, amount_cents: int, destination_account_id: str, description: str = "Nomad Karaoke referral payout") -> Optional[str]:
"""Create a transfer to a Connect account. Returns transfer ID."""
if not self.is_configured():
return None
try:
transfer = stripe.Transfer.create(
amount=amount_cents,
currency="usd",
destination=destination_account_id,
description=description,
)
logger.info(f"Created transfer {transfer.id}: ${amount_cents / 100:.2f} to {destination_account_id}")
return transfer.id
except stripe.error.StripeError as e:
logger.error(f"Failed to create transfer to {destination_account_id}: {e}")
return None
def create_customer(self, email: str, name: Optional[str] = None) -> Optional[str]:
"""
Create or get a Stripe customer for a user.
+186 -15
View File
@@ -14,6 +14,7 @@ import re
from typing import Optional, Dict, Any
from backend.config import get_settings
from backend.i18n import t
logger = logging.getLogger(__name__)
@@ -79,6 +80,18 @@ Thanks!
"""
DEFAULT_YOUTUBE_UPLOAD_COMPLETE_TEMPLATE = """Hi {name},
Great news! Your karaoke video for "{artist} - {title}" has been uploaded to YouTube:
{youtube_url}
This upload was briefly delayed while our systems processed other uploads, but everything is all set now.
Thanks!
-Andrew
"""
class TemplateService:
"""
Service for fetching and rendering email templates from GCS.
@@ -142,29 +155,48 @@ class TemplateService:
logger.error(f"Failed to fetch template {blob_path}: {e}")
return None
def get_job_completion_template(self) -> str:
def get_job_completion_template(self, locale: str = "en") -> str:
"""Get the job completion email template."""
# Try locale-specific GCS template first, then default GCS, then localized default
if locale != "en":
template = self._fetch_template_from_gcs(f"{locale}/job-completion.txt")
if template is not None:
return template
template = self._fetch_template_from_gcs("job-completion.txt")
if template is None:
logger.info("Using default job completion template")
if template is not None:
return template
logger.info("Using default job completion template")
if locale == "en":
return DEFAULT_JOB_COMPLETION_TEMPLATE
return template
return self._build_localized_job_completion_template(locale)
def get_action_needed_lyrics_template(self) -> str:
def get_action_needed_lyrics_template(self, locale: str = "en") -> str:
"""Get the lyrics review reminder template."""
if locale != "en":
template = self._fetch_template_from_gcs(f"{locale}/action-needed-lyrics.txt")
if template is not None:
return template
template = self._fetch_template_from_gcs("action-needed-lyrics.txt")
if template is None:
logger.info("Using default lyrics reminder template")
if template is not None:
return template
logger.info("Using default lyrics reminder template")
if locale == "en":
return DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE
return template
return self._build_localized_action_lyrics_template(locale)
def get_action_needed_instrumental_template(self) -> str:
def get_action_needed_instrumental_template(self, locale: str = "en") -> str:
"""Get the instrumental selection reminder template."""
if locale != "en":
template = self._fetch_template_from_gcs(f"{locale}/action-needed-instrumental.txt")
if template is not None:
return template
template = self._fetch_template_from_gcs("action-needed-instrumental.txt")
if template is None:
logger.info("Using default instrumental reminder template")
if template is not None:
return template
logger.info("Using default instrumental reminder template")
if locale == "en":
return DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE
return template
return self._build_localized_action_instrumental_template(locale)
def render_template(self, template: str, variables: Dict[str, Any]) -> str:
"""
@@ -203,6 +235,74 @@ class TemplateService:
return result.strip()
def _build_localized_job_completion_template(self, locale: str) -> str:
"""Build a localized job completion template from translation keys."""
ns = "emails.templates.jobCompletion"
return f"""{t(locale, f"{ns}.greeting")}
{t(locale, f"{ns}.thanks")}
{t(locale, f"{ns}.youtubeLink")}
{{youtube_url}}
{t(locale, f"{ns}.dropboxIntro")}
{t(locale, f"{ns}.fileList")}
{{dropbox_url}}
{t(locale, f"{ns}.fixOffer")}
{t(locale, f"{ns}.feedbackRequest")}
{{feedback_url}}
{t(locale, f"{ns}.signOff")}
"""
def _build_localized_action_lyrics_template(self, locale: str) -> str:
"""Build a localized action-needed lyrics template."""
ns = "emails.templates.actionLyrics"
return f"""{t(locale, f"{ns}.greeting")}
{t(locale, f"{ns}.ready")}
{t(locale, f"{ns}.instructions")}
{t(locale, f"{ns}.reviewPrompt")}
{{review_url}}
{t(locale, f"{ns}.duration")}
{t(locale, f"{ns}.signOff")}
"""
def _build_localized_action_instrumental_template(self, locale: str) -> str:
"""Build a localized action-needed instrumental template."""
ns = "emails.templates.actionInstrumental"
return f"""{t(locale, f"{ns}.greeting")}
{t(locale, f"{ns}.almostDone")}
{t(locale, f"{ns}.instructions")}
{t(locale, f"{ns}.selectPrompt")}
{{instrumental_url}}
{t(locale, f"{ns}.signOff")}
"""
def _build_localized_youtube_complete_template(self, locale: str) -> str:
"""Build a localized YouTube upload complete template."""
ns = "emails.templates.youtubeComplete"
return f"""{t(locale, f"{ns}.greeting")}
{t(locale, f"{ns}.greatNews")}
{{youtube_url}}
{t(locale, f"{ns}.delayNote")}
{t(locale, f"{ns}.signOff")}
"""
def render_job_completion(
self,
name: Optional[str] = None,
@@ -212,6 +312,9 @@ class TemplateService:
title: Optional[str] = None,
job_id: Optional[str] = None,
feedback_url: Optional[str] = None,
is_private: bool = False,
youtube_queued: bool = False,
locale: str = "en",
) -> str:
"""
Render the job completion email template.
@@ -224,11 +327,36 @@ class TemplateService:
title: Song title
job_id: Job ID
feedback_url: Feedback form URL (optional)
is_private: If True, remove YouTube section entirely (private/non-published tracks)
youtube_queued: If True, YouTube upload was deferred - show pending message
Returns:
Rendered email content
"""
template = self.get_job_completion_template()
template = self.get_job_completion_template(locale)
# For private tracks, remove the YouTube section entirely
if is_private:
template = re.sub(
r"\n*Here's the link for the karaoke video published to YouTube:\n\{youtube_url\}\n*",
'\n',
template,
)
elif youtube_queued and not youtube_url:
# Replace YouTube URL with pending message
queued_msg = t(locale, "emails.templates.jobCompletion.youtubeQueued")
youtube_link_text = t(locale, "emails.templates.jobCompletion.youtubeLink")
template = template.replace(
f"{youtube_link_text}\n{{youtube_url}}",
queued_msg
)
# Also handle the English default template pattern
if "{youtube_url}" in template:
template = template.replace(
"Here's the link for the karaoke video published to YouTube:\n{youtube_url}",
queued_msg
)
variables = {
"name": name or "there",
"youtube_url": youtube_url or "[YouTube URL not available]",
@@ -246,6 +374,7 @@ class TemplateService:
artist: Optional[str] = None,
title: Optional[str] = None,
review_url: str = "",
locale: str = "en",
) -> str:
"""
Render the combined review reminder template.
@@ -262,7 +391,7 @@ class TemplateService:
Returns:
Rendered email content
"""
template = self.get_action_needed_lyrics_template()
template = self.get_action_needed_lyrics_template(locale)
variables = {
"name": name or "there",
"artist": artist or "Unknown Artist",
@@ -277,6 +406,7 @@ class TemplateService:
artist: Optional[str] = None,
title: Optional[str] = None,
instrumental_url: str = "",
locale: str = "en",
) -> str:
"""
Render the instrumental selection reminder template.
@@ -294,7 +424,7 @@ class TemplateService:
Returns:
Rendered email content
"""
template = self.get_action_needed_instrumental_template()
template = self.get_action_needed_instrumental_template(locale)
variables = {
"name": name or "there",
"artist": artist or "Unknown Artist",
@@ -303,6 +433,47 @@ class TemplateService:
}
return self.render_template(template, variables)
def render_youtube_upload_complete(
self,
name: Optional[str] = None,
artist: Optional[str] = None,
title: Optional[str] = None,
youtube_url: Optional[str] = None,
locale: str = "en",
) -> str:
"""
Render the YouTube upload complete follow-up email template.
Sent when a deferred YouTube upload completes after quota becomes available.
Args:
name: User's display name
artist: Artist name
title: Song title
youtube_url: YouTube video URL
Returns:
Rendered email content
"""
if locale != "en":
template = self._fetch_template_from_gcs(f"{locale}/youtube-upload-complete.txt")
if template is not None:
pass # Use locale-specific GCS template
else:
template = self._fetch_template_from_gcs("youtube-upload-complete.txt")
else:
template = self._fetch_template_from_gcs("youtube-upload-complete.txt")
if template is None:
template = DEFAULT_YOUTUBE_UPLOAD_COMPLETE_TEMPLATE if locale == "en" else self._build_localized_youtube_complete_template(locale)
variables = {
"name": name or "there",
"artist": artist or "Unknown Artist",
"title": title or "Unknown Title",
"youtube_url": youtube_url or "[YouTube URL not available]",
}
return self.render_template(template, variables)
def upload_template(self, template_name: str, content: str) -> bool:
"""
Upload a template to GCS.
+2 -1
View File
@@ -334,7 +334,8 @@ class ThemeService:
style_params = self._update_asset_paths_in_style(theme_id, style_params)
# Upload modified style_params.json to job's style folder
job_style_path = f"uploads/{job_id}/style/style_params.json"
# Uses jobs/ prefix (not uploads/) to avoid GCS lifecycle deletion rule
job_style_path = f"jobs/{job_id}/style/style_params.json"
self.storage.upload_json(job_style_path, style_params)
logger.info(f"Prepared job {job_id} style from theme {theme_id}")
+527 -26
View File
@@ -13,6 +13,7 @@ import uuid
from datetime import datetime, timedelta, timezone
from typing import Optional, List, Tuple
from google.api_core import exceptions as google_exceptions
from google.cloud import firestore
from google.cloud.firestore_v1 import FieldFilter
from google.cloud.firestore_v1 import Increment
@@ -59,29 +60,71 @@ class UserService:
# User CRUD Operations
# =========================================================================
def get_user_doc_ref(self, email: str):
"""Get the Firestore document reference for a user by email.
Handles both email-keyed docs (doc ID = email) and hash-keyed docs
(doc ID = hash, email stored as field). Returns None if not found.
"""
try:
# First try direct lookup by email as doc ID
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
doc = doc_ref.get()
if doc.exists:
return doc_ref
# Fallback: query by email field (for hash-keyed docs)
query = self.db.collection(USERS_COLLECTION).where(
filter=FieldFilter('email', '==', email.lower())
).limit(1)
results = list(query.stream())
if results:
return results[0].reference
return None
except Exception:
logger.exception(f"Error finding user doc ref for {email}")
return None
def get_user(self, email: str) -> Optional[User]:
"""Get a user by email."""
try:
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
doc = doc_ref.get()
if not doc.exists:
return None
if doc.exists:
return User(**doc.to_dict())
return User(**doc.to_dict())
# Fallback: query by email field (for hash-keyed docs)
query = self.db.collection(USERS_COLLECTION).where(
filter=FieldFilter('email', '==', email.lower())
).limit(1)
results = list(query.stream())
if results:
return User(**results[0].to_dict())
return None
except Exception:
logger.exception(f"Error getting user {email}")
return None
def get_or_create_user(self, email: str, tenant_id: Optional[str] = None) -> User:
def get_or_create_user(
self,
email: str,
tenant_id: Optional[str] = None,
signup_ip: Optional[str] = None,
device_fingerprint: Optional[str] = None,
) -> User:
"""
Get existing user or create a new one.
New users receive a welcome credit to try the service.
Welcome credits are granted on first magic link verification, not here.
Args:
email: User's email address
tenant_id: Tenant ID for white-label portals (None = default Nomad Karaoke)
signup_ip: Client IP address at signup (for rate limiting)
device_fingerprint: Browser fingerprint at signup (for rate limiting)
Note: If user exists but has a different tenant_id, the existing user is returned.
Users are uniquely identified by email, not email+tenant.
@@ -95,22 +138,17 @@ class UserService:
# (though features may be restricted per-tenant)
return user
# Create new user with welcome credit
welcome_credit = self.NEW_USER_FREE_CREDITS
welcome_transaction = CreditTransaction(
id=str(uuid.uuid4()),
amount=welcome_credit,
reason="welcome_credit",
)
# Create new user with zero credits (welcome credits granted on first verification)
user = User(
email=email,
credits=welcome_credit,
credit_transactions=[welcome_transaction],
credits=0,
credit_transactions=[],
tenant_id=tenant_id, # Associate with tenant on creation
signup_ip=signup_ip,
device_fingerprint=device_fingerprint,
)
self._save_user(user)
logger.info(f"Created new user: {email} with {welcome_credit} welcome credit(s) (tenant: {tenant_id or 'default'})")
logger.info(f"Created new user: {email} (tenant: {tenant_id or 'default'}) — credits pending verification")
return user
def _save_user(self, user: User) -> None:
@@ -127,7 +165,10 @@ class UserService:
"""Update user fields."""
try:
updates['updated_at'] = datetime.utcnow()
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
doc_ref = self.get_user_doc_ref(email)
if not doc_ref:
logger.warning(f"Cannot update user {email}: doc not found")
return None
doc_ref.update(updates)
return self.get_user(email)
except Exception:
@@ -176,7 +217,9 @@ class UserService:
email: str,
ip_address: Optional[str] = None,
user_agent: Optional[str] = None,
tenant_id: Optional[str] = None
tenant_id: Optional[str] = None,
device_fingerprint: Optional[str] = None,
referral_code: Optional[str] = None,
) -> MagicLinkToken:
"""
Create a magic link token for email authentication.
@@ -189,11 +232,17 @@ class UserService:
ip_address: Client IP for auditing
user_agent: Client user agent for auditing
tenant_id: Tenant ID for white-label portals (None = default Nomad Karaoke)
device_fingerprint: Browser fingerprint for anti-abuse
"""
email = email.lower()
# Ensure user exists (with tenant association)
self.get_or_create_user(email, tenant_id=tenant_id)
# Ensure user exists (with tenant association, signup IP, and fingerprint)
self.get_or_create_user(
email,
tenant_id=tenant_id,
signup_ip=ip_address,
device_fingerprint=device_fingerprint,
)
# Generate secure token
token = secrets.token_urlsafe(32)
@@ -205,6 +254,8 @@ class UserService:
ip_address=ip_address,
user_agent=user_agent,
tenant_id=tenant_id,
device_fingerprint=device_fingerprint,
referral_code=referral_code,
)
# Save to Firestore
@@ -322,6 +373,406 @@ class UserService:
logger.exception("Error verifying magic link")
return False, None, "An error occurred during verification"
def grant_welcome_credits_if_eligible(
self,
email: str,
precomputed_eval: Optional[dict] = None,
) -> Tuple[bool, str]:
"""
Grant welcome credits on first magic link verification.
Uses AI evaluation to check for abuse before granting. Fail-open:
if evaluation fails, credits are granted anyway.
Uses a dedicated welcome_credits_granted flag as primary idempotency
check (survives credit_transactions list trimming). Falls back to
checking for welcome_credit transaction as secondary defense.
Args:
email: User's email address.
precomputed_eval: Optional pre-computed evaluation from magic link doc.
Keys: credit_eval_decision, credit_eval_reasoning, credit_eval_error.
If provided and has a decision, skips the AI call.
Returns:
Tuple of (credits_granted: bool, status: str).
status is one of: "granted", "denied", "already_granted", "not_found", "error"
"""
try:
email = email.lower()
user = self.get_user(email)
if not user:
return False, "not_found"
# Primary check: dedicated flag (survives transaction list trimming)
if user.welcome_credits_granted:
return False, "already_granted"
# Secondary check: look for existing welcome_credit transaction
for txn in user.credit_transactions:
if txn.reason == "welcome_credit":
return False, "already_granted"
# Skip AI evaluation for test emails (e.g. E2E test accounts)
# to avoid wasting LLM credits and sending unnecessary denial emails
from backend.utils.test_data import is_test_email
if not is_test_email(email):
# AI evaluation: check for abuse signals before granting
# Use pre-computed result from magic link if available
try:
from backend.services.credit_evaluation_service import (
get_credit_evaluation_service,
CreditEvaluation,
)
precomputed_decision = (precomputed_eval or {}).get("credit_eval_decision")
if precomputed_decision:
logger.info(f"Using pre-computed credit eval for {email}: {precomputed_decision}")
evaluation = CreditEvaluation(
decision=precomputed_decision,
reasoning=(precomputed_eval or {}).get("credit_eval_reasoning", ""),
confidence=1.0,
error=(precomputed_eval or {}).get("credit_eval_error"),
)
else:
eval_service = get_credit_evaluation_service()
evaluation = eval_service.evaluate(email, "welcome")
if evaluation.decision == "deny":
# Mark as evaluated but not granted so we don't re-evaluate
self.update_user(email, welcome_credits_granted=True)
logger.info(
f"Denied welcome credits to {email}: {evaluation.reasoning}"
)
try:
from backend.services.email_service import get_email_service
get_email_service().send_credit_denied_email(email, "welcome")
except Exception:
logger.exception(f"Failed to send credit denied email to {email}")
return False, "denied"
if evaluation.decision == "pending_review":
# Don't mark welcome_credits_granted — allow retry after manual review
logger.info(
f"Welcome credits pending review for {email}: {evaluation.reasoning}"
)
try:
from backend.services.email_service import get_email_service
email_service = get_email_service()
email_service.send_credit_review_needed_email(
email, "welcome", evaluation.reasoning, signals=evaluation.signals,
)
except Exception:
logger.exception(f"Failed to send review needed email for {email}")
return False, "pending_review"
except Exception:
logger.exception(f"Credit evaluation failed for {email} — pending review (fail-closed)")
try:
from backend.services.email_service import get_email_service
email_service = get_email_service()
email_service.send_credit_review_needed_email(
email, "welcome", f"Evaluation error: {email}",
)
except Exception:
pass
return False, "pending_review"
else:
logger.info(f"Skipping credit evaluation for test email {email}")
# Grant welcome credits
welcome_credit = self.NEW_USER_FREE_CREDITS
transaction = CreditTransaction(
id=str(uuid.uuid4()),
amount=welcome_credit,
reason="welcome_credit",
)
transactions = user.credit_transactions[-MAX_CREDIT_TRANSACTIONS + 1:]
transactions.append(transaction)
new_balance = user.credits + welcome_credit
self.update_user(
email,
credits=new_balance,
credit_transactions=[t.model_dump(mode='json') for t in transactions],
welcome_credits_granted=True,
)
logger.info(f"Granted {welcome_credit} welcome credits to {email}")
return True, "granted"
except Exception:
logger.exception(f"Error granting welcome credits to {email}")
return False, "error"
# =========================================================================
# Anti-Abuse Rate Limiting
# =========================================================================
# Maximum new signups allowed per IP address in the rate limit window
MAX_SIGNUPS_PER_IP = 2
SIGNUP_RATE_LIMIT_HOURS = 24
def count_recent_signups_from_ip(self, ip_address: str, hours: int = 24) -> int:
"""
Count new user accounts created from this IP in the last N hours.
Queries gen_users where signup_ip matches and created_at is recent.
Used to rate-limit new account creation per IP address.
Args:
ip_address: Client IP to check
hours: Lookback window in hours (default: 24)
Returns:
Number of recent signups from this IP
"""
try:
cutoff = datetime.utcnow() - timedelta(hours=hours)
query = (
self.db.collection(USERS_COLLECTION)
.where(filter=FieldFilter("signup_ip", "==", ip_address))
.where(filter=FieldFilter("created_at", ">=", cutoff))
)
count = 0
for _ in query.stream():
count += 1
return count
except Exception:
logger.exception(f"Error counting signups from IP {ip_address}")
# On error, don't block — return 0 to allow the request
return 0
def count_recent_signups_from_fingerprint(
self, device_fingerprint: str, hours: int = 24
) -> int:
"""
Count new user accounts created with this device fingerprint in the last N hours.
Args:
device_fingerprint: Browser fingerprint to check
hours: Lookback window in hours (default: 24)
Returns:
Number of recent signups from this fingerprint
"""
try:
cutoff = datetime.utcnow() - timedelta(hours=hours)
query = (
self.db.collection(USERS_COLLECTION)
.where(filter=FieldFilter("device_fingerprint", "==", device_fingerprint))
.where(filter=FieldFilter("created_at", ">=", cutoff))
)
count = 0
for _ in query.stream():
count += 1
return count
except Exception:
logger.exception(f"Error counting signups from fingerprint")
return 0
def is_signup_rate_limited(
self, ip_address: Optional[str] = None, device_fingerprint: Optional[str] = None
) -> bool:
"""
Check if a signup should be rate limited based on IP and/or fingerprint.
Either signal hitting the limit triggers a block.
"""
if ip_address:
ip_count = self.count_recent_signups_from_ip(
ip_address, hours=self.SIGNUP_RATE_LIMIT_HOURS
)
if ip_count >= self.MAX_SIGNUPS_PER_IP:
return True
if device_fingerprint:
fp_count = self.count_recent_signups_from_fingerprint(
device_fingerprint, hours=self.SIGNUP_RATE_LIMIT_HOURS
)
if fp_count >= self.MAX_SIGNUPS_PER_IP:
return True
return False
# =========================================================================
# Anti-Abuse Investigation
# =========================================================================
def find_users_by_signup_ip(self, ip_address: str) -> List[User]:
"""Find all users who signed up from this IP address."""
try:
query = (
self.db.collection(USERS_COLLECTION)
.where(filter=FieldFilter("signup_ip", "==", ip_address))
.order_by("created_at", direction=firestore.Query.DESCENDING)
.limit(50)
)
return [User(**doc.to_dict()) for doc in query.stream()]
except Exception:
logger.exception(f"Error finding users by IP {ip_address}")
return []
def find_users_by_fingerprint(self, device_fingerprint: str) -> List[User]:
"""Find all users who signed up with this device fingerprint."""
try:
query = (
self.db.collection(USERS_COLLECTION)
.where(filter=FieldFilter("device_fingerprint", "==", device_fingerprint))
.order_by("created_at", direction=firestore.Query.DESCENDING)
.limit(50)
)
return [User(**doc.to_dict()) for doc in query.stream()]
except Exception:
logger.exception(f"Error finding users by fingerprint")
return []
def find_suspicious_accounts(self, min_jobs: int = 2, max_spend: int = 0) -> List[User]:
"""
Find accounts with many jobs but no spend (potential free-credit abusers).
Args:
min_jobs: Minimum total_jobs_created to flag (default: 2)
max_spend: Maximum total_spent in cents to include (default: 0 = free only)
"""
from backend.utils.test_data import is_internal_or_test_email
try:
query = (
self.db.collection(USERS_COLLECTION)
.where(filter=FieldFilter("total_jobs_created", ">=", min_jobs))
.order_by("total_jobs_created", direction=firestore.Query.DESCENDING)
.limit(100)
)
users = []
for doc in query.stream():
user = User(**doc.to_dict())
if user.total_spent <= max_spend and not is_internal_or_test_email(user.email):
users.append(user)
return users
except Exception:
logger.exception("Error finding suspicious accounts")
return []
def find_related_accounts(self, email: str) -> dict:
"""
Given a user email, find all accounts that share the same signup IP
or device fingerprint. Returns a dict with grouped results.
"""
user = self.get_user(email)
if not user:
return {"user": None, "by_ip": [], "by_fingerprint": []}
by_ip = []
if user.signup_ip:
by_ip = [
u for u in self.find_users_by_signup_ip(user.signup_ip)
if u.email != user.email
]
by_fingerprint = []
if user.device_fingerprint:
by_fingerprint = [
u for u in self.find_users_by_fingerprint(user.device_fingerprint)
if u.email != user.email
]
return {
"user": user,
"by_ip": by_ip,
"by_fingerprint": by_fingerprint,
}
def find_account_correlations(self) -> dict:
"""
Find clusters of accounts sharing the same fingerprint or signup IP.
Returns groups of 2+ accounts that share the same identifier,
sorted by group size (largest first). Excludes internal/test accounts.
"""
from collections import defaultdict
from backend.utils.test_data import is_internal_or_test_email
# Known bad IPs (load balancer, metadata, placeholder) that don't represent real client IPs
BAD_IPS = {"169.254.169.126", "unknown", ""}
try:
all_users = []
for doc in self.db.collection(USERS_COLLECTION).stream():
user = User(**doc.to_dict())
if not is_internal_or_test_email(user.email):
all_users.append(user)
# Group by fingerprint
fp_groups: dict[str, list[User]] = defaultdict(list)
for user in all_users:
if user.device_fingerprint:
fp_groups[user.device_fingerprint].append(user)
# Group by IP (exclude known bad IPs)
ip_groups: dict[str, list[User]] = defaultdict(list)
for user in all_users:
if user.signup_ip and user.signup_ip not in BAD_IPS:
ip_groups[user.signup_ip].append(user)
# Collect user agents from latest sessions for users in clusters
cluster_emails = set()
for fp, users in fp_groups.items():
if len(users) >= 2:
cluster_emails.update(u.email for u in users)
for ip, users in ip_groups.items():
if len(users) >= 2:
cluster_emails.update(u.email for u in users)
user_agents: dict[str, str] = {}
for email in cluster_emails:
try:
sessions = (
self.db.collection(SESSIONS_COLLECTION)
.where(filter=FieldFilter("user_email", "==", email))
.order_by("created_at", direction=firestore.Query.DESCENDING)
.limit(1)
.stream()
)
for session_doc in sessions:
session_data = session_doc.to_dict()
ua = session_data.get("user_agent")
if ua:
user_agents[email] = ua
except Exception:
pass # Skip if session lookup fails
# Build response - only clusters with 2+ users
fingerprint_clusters = []
for fp, users in fp_groups.items():
if len(users) >= 2:
fingerprint_clusters.append({
"fingerprint": fp,
"users": sorted(users, key=lambda u: u.created_at or "", reverse=True),
})
fingerprint_clusters.sort(key=lambda c: len(c["users"]), reverse=True)
ip_clusters = []
for ip, users in ip_groups.items():
if len(users) >= 2:
ip_clusters.append({
"ip": ip,
"users": sorted(users, key=lambda u: u.created_at or "", reverse=True),
})
ip_clusters.sort(key=lambda c: len(c["users"]), reverse=True)
return {
"fingerprint_clusters": fingerprint_clusters,
"ip_clusters": ip_clusters,
"user_agents": user_agents,
}
except Exception:
logger.exception("Error finding account correlations")
return {"fingerprint_clusters": [], "ip_clusters": [], "user_agents": {}}
# =========================================================================
# Session Management
# =========================================================================
@@ -331,7 +782,8 @@ class UserService:
user_email: str,
ip_address: Optional[str] = None,
user_agent: Optional[str] = None,
tenant_id: Optional[str] = None
tenant_id: Optional[str] = None,
device_fingerprint: Optional[str] = None,
) -> Session:
"""
Create a new session for an authenticated user.
@@ -341,6 +793,7 @@ class UserService:
ip_address: Client IP for auditing
user_agent: Client user agent for auditing
tenant_id: Tenant ID for white-label portals (None = default Nomad Karaoke)
device_fingerprint: Browser fingerprint for cross-account correlation
"""
token = secrets.token_urlsafe(32)
token_prefix = token[:12]
@@ -352,6 +805,7 @@ class UserService:
ip_address=ip_address,
user_agent=user_agent,
tenant_id=tenant_id,
device_fingerprint=device_fingerprint,
)
# Serialize and write to Firestore
@@ -546,7 +1000,7 @@ class UserService:
'amount': amount,
'processed_at': datetime.utcnow()
})
except Exception:
except google_exceptions.AlreadyExists:
# Document already exists - this session was already processed
logger.info(f"Stripe session {stripe_session_id} already processed (idempotent skip)")
return False, 0, "Session already processed"
@@ -768,15 +1222,62 @@ class UserService:
logger.exception("Error enabling user")
return False
def delete_user(self, email: str, admin_email: str) -> bool:
"""
Permanently delete a user and all their associated data.
Deletes: user document, sessions, magic links.
Does NOT delete jobs (they remain as historical records).
Args:
email: Email of the user to delete
admin_email: Email/ID of the admin performing the action
Returns:
True if user was deleted, False if user not found
Raises:
ValueError: If trying to delete an admin user
"""
try:
email = email.lower()
user = self.get_user(email)
if not user:
return False
if user.role == UserRole.ADMIN:
raise ValueError("Cannot delete admin users")
# Revoke all active sessions
self.revoke_all_sessions(email)
# Delete magic links for the user
magic_links = self.db.collection(MAGIC_LINKS_COLLECTION).where(
filter=FieldFilter('email', '==', email)
).stream()
for doc in magic_links:
doc.reference.delete()
# Delete the user document
self.db.collection(USERS_COLLECTION).document(email).delete()
logger.info(f"Admin {admin_email} permanently deleted user {email}")
return True
except ValueError:
raise
except Exception:
logger.exception(f"Error deleting user {email}")
return False
def increment_jobs_completed(self, email: str) -> bool:
"""Increment the completed jobs counter for a user using atomic increment."""
try:
# Check user exists first
if not self.get_user(email):
doc_ref = self.get_user_doc_ref(email)
if not doc_ref:
return False
# Use atomic increment to prevent race conditions
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
doc_ref.update({
'total_jobs_completed': Increment(1),
'updated_at': datetime.utcnow()
@@ -0,0 +1,300 @@
"""
Visibility change service for completed jobs.
Handles changing a job's visibility between public and private:
- Public -> Private: redistribute existing finals to private destination (fast, ~1-2 min)
- Private -> Public: reset styles, regenerate screens, re-render, re-encode (slow, ~15-30 min)
"""
import logging
from datetime import datetime, timezone
from typing import Optional
from google.cloud.firestore_v1 import DELETE_FIELD, ArrayUnion
from backend.models.job import JobStatus
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
logger = logging.getLogger(__name__)
class VisibilityChangeService:
"""Service for changing job visibility after completion."""
def __init__(self, job_manager: Optional[JobManager] = None):
self.job_manager = job_manager or JobManager()
def validate_change(self, job, user_email: str, is_admin: bool, target_visibility: str) -> Optional[str]:
"""Validate a visibility change request. Returns error message or None if valid."""
if not job:
return "Job not found"
if job.status != JobStatus.COMPLETE.value and job.status != "complete":
return f"Can only change visibility on completed jobs. Current status: {job.status}"
if getattr(job, 'tenant_id', None):
return "Cannot change visibility on tenant jobs"
current_is_private = getattr(job, 'is_private', False)
target_is_private = target_visibility == "private"
if current_is_private == target_is_private:
current = "private" if current_is_private else "public"
return f"Job is already {current}"
# Check user owns the job or is admin
if not is_admin and job.user_email != user_email:
return "You can only change visibility on your own jobs"
# Check for in-progress visibility change
state_data = job.state_data or {}
if state_data.get('visibility_change_in_progress'):
return "A visibility change is already in progress for this job"
return None
async def change_to_private(self, job_id: str, job, user_email: str) -> dict:
"""
Change a public job to private (fast path).
Deletes distributed outputs (YouTube/Dropbox/GDrive) but keeps GCS finals,
then redistributes to private destination.
"""
logger.info(f"[job:{job_id}] Starting visibility change: public -> private (by {user_email})")
db = self.job_manager.firestore.db
job_ref = db.collection("jobs").document(job_id)
# Set guard flag
job_ref.update({
"state_data.visibility_change_in_progress": True,
"updated_at": datetime.now(timezone.utc),
})
try:
# Step 1: Delete distributed outputs (keep GCS finals)
await self._delete_distributed_outputs(job_id, job, keep_gcs_finals=True)
# Step 2: Update job to private
job_ref.update({
"is_private": True,
"updated_at": datetime.now(timezone.utc),
"timeline": ArrayUnion([{
"status": "complete",
"timestamp": datetime.now(timezone.utc).isoformat(),
"message": f"Visibility changed to private by {user_email}",
}]),
})
# Step 3: Trigger redistribution via video worker
from backend.workers.video_worker import redistribute_video
success = await redistribute_video(job_id)
if not success:
raise RuntimeError("Redistribution failed - check worker logs")
# Clear guard flag (redistribution updates state_data with new brand_code etc.)
job_ref.update({
"state_data.visibility_change_in_progress": DELETE_FIELD,
})
logger.info(f"[job:{job_id}] Visibility change to private complete")
return {
"status": "success",
"message": "Job changed to private. Outputs redistributed to private destination.",
"reprocessing_required": False,
}
except Exception as e:
logger.error(f"[job:{job_id}] Visibility change to private failed: {e}", exc_info=True)
# Clear guard flag on failure
job_ref.update({
"state_data.visibility_change_in_progress": DELETE_FIELD,
})
raise
async def change_to_public(self, job_id: str, job, user_email: str) -> dict:
"""
Change a private job to public (slow path).
Resets custom styles, deletes all outputs/screens/rendered video,
and triggers full re-processing from screens worker onward.
"""
logger.info(f"[job:{job_id}] Starting visibility change: private -> public (by {user_email})")
db = self.job_manager.firestore.db
job_ref = db.collection("jobs").document(job_id)
storage = StorageService()
# Step 1: Delete distributed outputs + GCS finals
await self._delete_distributed_outputs(job_id, job, keep_gcs_finals=False)
# Step 2: Delete screens and rendered video from GCS
for path in [
f"jobs/{job_id}/screens/title.mov",
f"jobs/{job_id}/screens/title.jpg",
f"jobs/{job_id}/screens/title.png",
f"jobs/{job_id}/screens/end.mov",
f"jobs/{job_id}/screens/end.jpg",
f"jobs/{job_id}/screens/end.png",
f"jobs/{job_id}/videos/with_vocals.mkv",
f"jobs/{job_id}/videos/with_vocals.mov",
]:
try:
storage.delete_file(path)
except Exception:
pass
# Delete custom style files from GCS
try:
style_prefix = f"jobs/{job_id}/style/"
storage.delete_folder(style_prefix)
except Exception:
pass
# Step 3: Prepare nomad theme style files for the job
from backend.api.routes.file_upload import _prepare_theme_for_job
style_params_path, style_assets_dict, _youtube_desc = _prepare_theme_for_job(
job_id=job_id, theme_id="nomad"
)
# Step 4: Reset styles and update job fields
# Step 5: Set status to LYRICS_COMPLETE with regen_restore_status
# Step 6: Clear progress keys and set guard flag
update_payload = {
"is_private": False,
"theme_id": "nomad",
"color_overrides": {},
"style_assets": style_assets_dict,
"style_params_gcs_path": style_params_path,
"status": JobStatus.LYRICS_COMPLETE.value,
"state_data.visibility_change_in_progress": True,
"state_data.regen_restore_status": "review_complete",
"state_data.audio_complete": True,
"state_data.lyrics_complete": True,
"state_data.screens_progress": DELETE_FIELD,
"state_data.render_progress": DELETE_FIELD,
"state_data.video_progress": DELETE_FIELD,
"state_data.encoding_progress": DELETE_FIELD,
"outputs_deleted_at": None,
"outputs_deleted_by": None,
"updated_at": datetime.now(timezone.utc),
"timeline": ArrayUnion([{
"status": "lyrics_complete",
"timestamp": datetime.now(timezone.utc).isoformat(),
"message": f"Visibility change to public initiated by {user_email}. "
"Re-processing from screens generation.",
}]),
}
job_ref.update(update_payload)
# Step 6: Trigger screens worker
from backend.services.worker_service import get_worker_service
worker_service = get_worker_service()
triggered = await worker_service.trigger_screens_worker(job_id)
if not triggered:
logger.error(f"[job:{job_id}] Failed to trigger screens worker for visibility change")
# Restore to complete status on failure
job_ref.update({
"status": "complete",
"is_private": True,
"state_data.visibility_change_in_progress": DELETE_FIELD,
"state_data.regen_restore_status": DELETE_FIELD,
})
raise RuntimeError("Failed to trigger screens worker")
logger.info(f"[job:{job_id}] Visibility change to public initiated, screens worker triggered")
return {
"status": "processing",
"message": "Visibility change to public started. The video will be re-rendered "
"with default branding and published publicly. This takes ~15-30 minutes.",
"reprocessing_required": True,
}
async def _delete_distributed_outputs(self, job_id: str, job, keep_gcs_finals: bool = False):
"""Delete distributed outputs (YouTube, Dropbox, GDrive) and recycle brand code."""
import re
state_data = job.state_data or {}
# Delete YouTube video
youtube_url = state_data.get('youtube_url')
if youtube_url:
try:
video_id_match = re.search(r'(?:youtu\.be/|youtube\.com/watch\?v=)([^&\s]+)', youtube_url)
if video_id_match:
video_id = video_id_match.group(1)
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
from backend.services.youtube_service import get_youtube_service
youtube_service = get_youtube_service()
if youtube_service.is_configured:
finalise = KaraokeFinalise(
dry_run=False, non_interactive=True,
user_youtube_credentials=youtube_service.get_credentials_dict()
)
finalise.delete_youtube_video(video_id)
logger.info(f"[job:{job_id}] Deleted YouTube video {video_id}")
except Exception as e:
logger.warning(f"[job:{job_id}] Failed to delete YouTube video: {e}")
# Delete Dropbox folder
brand_code = state_data.get('brand_code')
dropbox_path = getattr(job, 'dropbox_path', None)
if brand_code and dropbox_path:
try:
from backend.services.dropbox_service import get_dropbox_service
dropbox = get_dropbox_service()
if dropbox.is_configured:
from karaoke_gen.utils import sanitize_filename
safe_artist = sanitize_filename(job.artist) if job.artist else "Unknown"
safe_title = sanitize_filename(job.title) if job.title else "Unknown"
folder_name = f"{brand_code} - {safe_artist} - {safe_title}"
full_path = f"{dropbox_path}/{folder_name}"
dropbox.delete_folder(full_path)
logger.info(f"[job:{job_id}] Deleted Dropbox folder {full_path}")
except Exception as e:
logger.warning(f"[job:{job_id}] Failed to delete Dropbox folder: {e}")
# Delete Google Drive files
gdrive_files = state_data.get('gdrive_files')
if gdrive_files:
try:
from backend.services.gdrive_service import get_gdrive_service
gdrive = get_gdrive_service()
if gdrive.is_configured:
file_ids = list(gdrive_files.values()) if isinstance(gdrive_files, dict) else []
gdrive.delete_files(file_ids)
logger.info(f"[job:{job_id}] Deleted Google Drive files")
except Exception as e:
logger.warning(f"[job:{job_id}] Failed to delete Google Drive files: {e}")
# Delete GCS finals if requested
if not keep_gcs_finals:
try:
storage = StorageService()
deleted_count = storage.delete_folder(f"jobs/{job_id}/finals/")
if deleted_count > 0:
logger.info(f"[job:{job_id}] Deleted {deleted_count} GCS finals files")
except Exception as e:
logger.warning(f"[job:{job_id}] Failed to delete GCS finals: {e}")
# Recycle brand code
if brand_code:
try:
from backend.services.brand_code_service import BrandCodeService, get_brand_code_service
prefix, number = BrandCodeService.parse_brand_code(brand_code)
get_brand_code_service().recycle_brand_code(prefix, number)
logger.info(f"[job:{job_id}] Recycled brand code {brand_code}")
except Exception as e:
logger.warning(f"[job:{job_id}] Failed to recycle brand code: {e}")
# Clear distribution state_data
db = self.job_manager.firestore.db
job_ref = db.collection("jobs").document(job_id)
update = {}
for key in ['youtube_url', 'brand_code', 'dropbox_link', 'gdrive_files']:
if key in state_data:
update[f"state_data.{key}"] = DELETE_FIELD
if update:
job_ref.update(update)
+145 -7
View File
@@ -38,6 +38,7 @@ WORKER_QUEUES = {
"render-video": "render-worker-queue",
"video": "video-worker-queue",
"idle-reminder": "idle-reminder-queue", # For delayed idle reminder checks
"gdrive-validation": "gdrive-validation-queue", # For delayed post-job validation
}
# Dispatch deadlines for each worker type (in seconds)
@@ -51,10 +52,15 @@ WORKER_DISPATCH_DEADLINES = {
"render-video": 1800, # 30 min - Video encoding can be slow
"video": 1800, # 30 min - Video encoding can be slow
"idle-reminder": 60, # 1 min - Quick check and potential email send
"gdrive-validation": 120, # 2 min - Calls Cloud Function which validates GDrive
}
# Default delay for idle reminders (seconds)
IDLE_REMINDER_DELAY_SECONDS = 5 * 60 # 5 minutes
IDLE_REMINDER_DELAY_SECONDS = 2 * 60 # 2 minutes
# Default delay for post-job GDrive validation (seconds)
# Gives E2E tests time to verify output and clean up before validator runs
GDRIVE_VALIDATION_DELAY_SECONDS = 5 * 60 # 5 minutes
class WorkerService:
@@ -355,19 +361,35 @@ class WorkerService:
# Convenience methods for specific workers
# These provide a cleaner API and better IDE autocomplete
async def trigger_audio_download_worker(self, job_id: str) -> bool:
"""
Trigger audio download worker.
Downloads audio from the selected source (Spotify, YouTube, RED, OPS)
and then triggers audio separation + lyrics workers.
Uses Cloud Run Jobs to avoid instance termination during long-running
downloads. This replaces the BackgroundTasks approach that caused
jobs to get stuck when Cloud Run scaled down instances.
"""
return await self._trigger_worker_cloud_run_job(
job_id=job_id,
cloud_run_job_name="audio-download-job",
worker_module="audio_download_worker",
)
async def trigger_audio_worker(self, job_id: str) -> bool:
"""
Trigger audio separation worker.
Always uses Cloud Run Jobs to avoid instance termination during long-running
Modal API calls. The BackgroundTasks approach caused Cloud Run to terminate
instances mid-processing when no HTTP requests were active.
Uses Cloud Run Jobs with L4 GPU in us-east4 for direct audio separation.
"""
return await self._trigger_worker_cloud_run_job(
job_id=job_id,
cloud_run_job_name="audio-separation-job",
worker_module="audio_worker",
location="us-east4", # GPU job runs in us-east4 (L4 quota)
)
async def trigger_lyrics_worker(self, job_id: str) -> bool:
@@ -389,14 +411,40 @@ class WorkerService:
"""Trigger screen generation worker."""
return await self.trigger_worker("screens", job_id)
def _warmup_encoding_worker(self, job_id: str) -> None:
"""Fire-and-forget warmup of the encoding worker VM.
Called before dispatching video/render workers so the GCE encoding
VM starts booting while the worker downloads files and prepares.
Failures are logged but never block worker dispatch.
"""
try:
from backend.services.encoding_worker_manager import EncodingWorkerManager
from google.cloud import compute_v1, firestore
db = firestore.Client(project=self.settings.google_cloud_project)
compute_client = compute_v1.InstancesClient()
manager = EncodingWorkerManager(
db=db,
compute_client=compute_client,
project_id=self.settings.google_cloud_project,
)
result = manager.ensure_primary_running()
if result["started"]:
logger.info(f"[job:{job_id}] Encoding worker warmup: started VM {result['vm_name']}")
else:
logger.info(f"[job:{job_id}] Encoding worker warmup: VM {result['vm_name']} already running")
except Exception as e:
logger.warning(f"[job:{job_id}] Encoding worker warmup failed (non-fatal): {e}")
async def trigger_video_worker(self, job_id: str) -> bool:
"""
Trigger video generation worker.
When USE_CLOUD_RUN_JOBS_FOR_VIDEO=true and ENABLE_CLOUD_TASKS=true,
uses Cloud Run Jobs for execution (supports >30 min encoding).
Otherwise, uses Cloud Tasks or direct HTTP.
"""
self._warmup_encoding_worker(job_id)
if self._use_cloud_tasks and self.settings.use_cloud_run_jobs_for_video:
return await self._trigger_cloud_run_job(job_id)
return await self.trigger_worker("video", job_id)
@@ -425,6 +473,7 @@ class WorkerService:
job_id: str,
cloud_run_job_name: str,
worker_module: str,
location: str | None = None,
) -> bool:
"""
Trigger a worker as a Cloud Run Job.
@@ -437,6 +486,7 @@ class WorkerService:
job_id: Job ID to process
cloud_run_job_name: Name of the Cloud Run Job (e.g., "lyrics-transcription-job")
worker_module: Worker module name (e.g., "lyrics_worker")
location: GCP region for the job (defaults to settings.gcp_region)
Returns:
True if job was triggered successfully, False otherwise
@@ -449,7 +499,7 @@ class WorkerService:
logger.error("GOOGLE_CLOUD_PROJECT not set, cannot trigger Cloud Run Job")
return False
location = self.settings.gcp_region
location = location or self.settings.gcp_region
job_name = f"projects/{project}/locations/{location}/jobs/{cloud_run_job_name}"
# Create Cloud Run Jobs client
@@ -487,6 +537,7 @@ class WorkerService:
async def trigger_render_video_worker(self, job_id: str) -> bool:
"""Trigger render video worker (post-review)."""
self._warmup_encoding_worker(job_id)
return await self.trigger_worker("render-video", job_id)
async def schedule_idle_reminder(
@@ -582,6 +633,93 @@ class WorkerService:
)
return False
async def schedule_gdrive_validation(
self,
delay_seconds: int = GDRIVE_VALIDATION_DELAY_SECONDS
) -> bool:
"""
Schedule a delayed GDrive validation check.
Fires after the specified delay so that E2E test cleanup completes
before the validator runs, avoiding false "all clear" reports.
Args:
delay_seconds: Delay before executing the validation (default: 5 minutes)
Returns:
True if task was scheduled successfully, False otherwise
"""
if not self._use_cloud_tasks:
logger.info(
f"GDrive validation not scheduled (Cloud Tasks disabled). "
f"Would have fired in {delay_seconds}s."
)
return True
try:
from google.cloud import tasks_v2
from google.protobuf import timestamp_pb2
import time
queue_name = WORKER_QUEUES.get("gdrive-validation")
if not queue_name:
logger.error("GDrive validation queue not configured")
return False
project = self.settings.google_cloud_project
if not project:
logger.error("GOOGLE_CLOUD_PROJECT not set")
return False
location = self.settings.gcp_region
# Build queue path
parent = self.tasks_client.queue_path(project, location, queue_name)
# Build headers
headers = {
"Content-Type": "application/json",
}
if self._admin_token:
headers["X-Admin-Token"] = self._admin_token
headers = inject_trace_context(headers)
# Calculate schedule time
schedule_time = timestamp_pb2.Timestamp()
schedule_time.FromSeconds(int(time.time()) + delay_seconds)
dispatch_deadline_seconds = WORKER_DISPATCH_DEADLINES.get("gdrive-validation", 120)
# Build task payload
task = {
"http_request": {
"http_method": tasks_v2.HttpMethod.POST,
"url": f"{self._base_url}/api/internal/trigger-gdrive-validation",
"headers": headers,
"body": b"{}",
"oidc_token": {
"service_account_email": f"karaoke-backend@{project}.iam.gserviceaccount.com",
},
},
"schedule_time": schedule_time,
"dispatch_deadline": duration_pb2.Duration(seconds=dispatch_deadline_seconds),
}
response = self.tasks_client.create_task(parent=parent, task=task)
logger.info(
f"Scheduled GDrive validation in {delay_seconds}s: {response.name}"
)
return True
except Exception as e:
logger.error(
f"Failed to schedule GDrive validation: {e}",
exc_info=True
)
return False
# Global worker service instance
_worker_service: Optional[WorkerService] = None
Loaded 100 of 1255 files, more files were not shown because too many files have changed in this diff. Show more