Files
karaoke-gen/docs/ARCHITECTURE.md
T
Andrew Beveridge ☄️andClaude 560c3e4904 feat: Add email notification system with admin controls (#220)
## Summary

- Add automated email notification system for job completion and action
reminders
- Admin UI buttons to copy completion message or send email directly
- Comprehensive test suite (81 new tests)
- **Email subjects include release ID and song info** for easy
identification

## Changes

### Backend
- **Email Service** (`email_service.py`): Extended with SendGrid CC
support, `send_job_completion()` and `send_action_reminder()` methods
- **Job Notification Service** (`job_notification_service.py`): New
service orchestrating completion emails and action reminder scheduling
- **Template Service** (`template_service.py`): GCS-backed HTML email
templates with fallback defaults
- **Worker Service** (`worker_service.py`): Cloud Tasks integration for
delayed idle reminders (5-min delay)
- **Job Manager** (`job_manager.py`): State transition triggers for
automated notifications
- **Admin Routes** (`admin.py`): New endpoints `GET
/api/admin/jobs/{id}/completion-message` and `POST
/api/admin/jobs/{id}/send-completion-email`
- **Internal Routes** (`internal.py`): Idle reminder check endpoint for
Cloud Tasks

### Email Subject Format
Email subjects now include brand code and song info for easy
identification:
- With brand code: `NOMAD-1178: Seether - Tonight (Your karaoke video is
ready!)`
- Without brand code: `Seether - Tonight (Your karaoke video is ready!)`
- Without song info: `Your karaoke video is ready!`

### Frontend
- **OutputLinks Component**: Admin-only "Copy Message" and "Send Email"
buttons with dialog UI
- **E2E Helpers** (`email-testing.ts`): `waitForCompletionEmail()` and
`isCompletionEmail()` methods for testmail.app integration
- **Happy Path Test**: Step 9.5 verifies completion email arrives after
job completion

### Documentation
- Updated API.md with new endpoints
- Updated ARCHITECTURE.md with notification system architecture
- Updated README.md Recent Changes

## Testing

- [x] 81 new unit tests covering:
- `test_email_service.py` (27 tests) - providers, CC support, job
completion/reminder methods
- `test_job_notification_service.py` (21 tests) - URL building, message
generation
- `test_job_manager_notifications.py` (15 tests) - state transition
triggers
  - `test_admin_email_endpoints.py` (12 tests) - admin API endpoints
- `test_template_service.py` (6 tests) - GCS template loading, fallbacks
- [x] E2E test extended to verify completion email delivery
- [x] All backend tests pass (`make test`)
- [x] TypeScript compiles

## Feature Flags

- `ENABLE_AUTO_EMAILS` (default: `false`) - Controls automatic email
sending on job state changes

## Review

- [x] Local CodeRabbit review completed (3 cycles)
- [x] Review feedback addressed:
  - Fixed potential None state_data in internal.py
  - Masked user emails in logs (PII protection)
  - Fixed React setTimeout memory leak
  - Changed placeholder URL default to empty
  - Added CC email deduplication

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-06 11:53:11 -05:00

11 KiB

Architecture

System Overview

┌─────────────────────────────────────────────────────────────────┐
│  Frontend (Cloudflare Pages)                                    │
│  Next.js + TypeScript                                           │
│  https://gen.nomadkaraoke.com                                   │
└───────────────────────────┬─────────────────────────────────────┘
                            │ HTTPS
                            ▼
┌─────────────────────────────────────────────────────────────────┐
│  Backend API (Cloud Run)                                        │
│  FastAPI + Python 3.12                                          │
│  https://api.nomadkaraoke.com                                   │
│                                                                 │
│  Workers:                                                       │
│  • Audio Worker    - Stem separation via Modal API              │
│  • Lyrics Worker   - Transcription via AudioShake               │
│  • Screens Worker  - Title/end screen generation                │
│  • Render Worker   - Karaoke video with lyrics                  │
│  • Video Worker    - Final assembly and encoding                │
└───────────────────────────┬─────────────────────────────────────┘
                            │
              ┌─────────────┼─────────────┬───────────────┐
              ▼             ▼             ▼               ▼
        ┌──────────┐  ┌──────────┐  ┌──────────────┐ ┌──────────────┐
        │ Firestore│  │   GCS    │  │Secret Manager│ │GCE Encoding  │
        │ (jobs)   │  │ (files)  │  │  (API keys)  │ │  Worker (*)  │
        └──────────┘  └──────────┘  └──────────────┘ └──────────────┘

(*) GCE Encoding Worker: C4-standard-8 VM with Intel Granite Rapids for
    high-performance FFmpeg encoding. Used for both final video encoding and
    preview video generation. Uses LocalEncodingService via GCS wheel deployment.

Processing Pipeline

1. UPLOAD
   POST /api/jobs/upload
   • Save audio to GCS
   • Create job in Firestore
   • Trigger parallel workers

2. PARALLEL PROCESSING
   ┌─────────────────┐    ┌─────────────────┐
   │  Audio Worker   │    │  Lyrics Worker  │
   │  Modal API      │    │  AudioShake     │
   │  2-stage sep    │    │  Auto-correct   │
   └────────┬────────┘    └────────┬────────┘
            └──────────┬───────────┘
                       ▼
3. SCREENS GENERATION
   • Title screen (5s video)
   • End screen (5s video)
                       ▼
4. HUMAN REVIEW (AWAITING_REVIEW)
   • React UI for lyrics correction
   • Preview video generation (optionally via GCE for speed)
   • Submit corrections
                       ▼
5. VIDEO RENDERING
   • Merge corrections with original data
   • Generate karaoke video with lyrics
                       ▼
6. INSTRUMENTAL SELECTION
   • User chooses: clean or with_backing
                       ▼
7. FINAL VIDEO
   • Remux with selected instrumental
   • Concatenate: title + karaoke + end
   • Encode to 4 formats
                       ▼
8. COMPLETE
   • User downloads via signed URLs

Job States

State Description Human Action
pending Job created -
downloading Processing audio -
separating_stage1 Audio separation stage 1 -
separating_stage2 Audio separation stage 2 -
transcribing Lyrics transcription -
generating_screens Creating title/end screens -
awaiting_review Waiting for human Review lyrics
in_review Human actively reviewing Edit lyrics
review_complete Review submitted -
rendering_video Generating karaoke video -
awaiting_instrumental_selection Waiting for human Choose audio
instrumental_selected Selection made -
generating_video Final encoding -
complete All done Download files
failed Error occurred -

GCS File Structure

jobs/{job_id}/
├── input/
│   └── original.flac              # Uploaded audio
├── stems/
│   ├── vocals.flac
│   ├── instrumental_clean.flac
│   ├── instrumental_with_backing.flac
│   └── ...
├── lyrics/
│   ├── corrections.json           # Auto-corrected
│   ├── corrections_updated.json   # After human review
│   └── karaoke.ass
├── screens/
│   ├── title.mov
│   └── end.mov
├── videos/
│   └── with_vocals.mkv            # Karaoke video
└── finals/
    ├── lossless_4k.mp4
    ├── lossless_4k.mkv
    ├── lossy_4k.mp4
    └── lossy_720p.mp4

LyricsTranscriber Integration

Key Design Decision: We use LyricsTranscriber as a library, not a server.

CLI Mode (Original):              Cloud Mode (Our Approach):
────────────────────              ──────────────────────────
LyricsTranscriber                 LyricsTranscriber
├── transcribe()                  ├── transcribe()    ✅
├── correct()                     ├── correct()       ✅
├── ReviewServer.start()          │
│   └── BLOCKS (waits)            │  (Skip server, save JSON)
│                                 │  ... ASYNC GAP ...
│                                 │  (Human reviews via React)
└── OutputGenerator               │
                                  └── OutputGenerator  ✅
                                      (called after review)

Use: LyricsTranscriber, CorrectionResult, OutputGenerator, OutputConfig Don't use: ReviewServer, blocking waits

External Services

Service Purpose Required
Modal GPU audio separation Yes
AudioShake Lyrics transcription Yes
Vertex AI Agentic AI correction (Gemini 3 Flash) Default on
Genius Reference lyrics Yes
YouTube API Video upload Optional
SendGrid Email notifications Optional
Cloud Tasks Delayed task scheduling (idle reminders) Optional

Firestore Collections

karaoke-gen shares a GCP project (nomadkaraoke) with karaoke-decide, but uses separate Firestore collections:

Collection Purpose Key Fields
gen_users karaoke-gen user accounts email, credits, role, is_active
jobs Karaoke generation jobs job_id, user_email, status, state_data
jobs/{job_id}/logs Worker log entries (subcollection) timestamp, level, worker, message, ttl_expiry
sessions Magic link auth sessions user_email, token, expires_at
magic_links Passwordless auth tokens email, token, expires_at, used
beta_feedback Beta program feedback user_email, ratings, comments

Note: The users collection in the same Firestore instance belongs to karaoke-decide (different schema: user_id, is_guest, quiz_* fields). Don't use it for karaoke-gen.

Worker Logs: Stored in subcollection (jobs/{job_id}/logs) instead of embedded array to avoid Firestore's 1MB document size limit. TTL policy auto-deletes logs after 30 days. Feature flag: USE_LOG_SUBCOLLECTION (default: true). See LESSONS-LEARNED.md.

Video Worker Orchestrator

The Video Worker uses an orchestrator pattern to ensure all features work regardless of encoding backend (local Cloud Run or GCE VM).

┌─────────────────────────────────────────────────────────────┐
│                 VideoWorkerOrchestrator                      │
│                                                             │
│  Stage 1: PACKAGING        Stage 2: ENCODING                │
│  ├─ CDG file generation    ├─ Local (FFmpeg)               │
│  └─ TXT lyrics export      └─ GCE (remote VM)              │
│                                                             │
│  Stage 3: ORGANIZATION     Stage 4: DISTRIBUTION           │
│  ├─ Final path setup       ├─ Dropbox upload               │
│  └─ File organization      ├─ Google Drive upload          │
│                            └─ YouTube upload               │
│                                                             │
│  Stage 5: NOTIFICATIONS                                     │
│  ├─ Discord webhooks                                        │
│  └─ Email notifications (completion, reminders)             │
└─────────────────────────────────────────────────────────────┘

Key Design Decision: The orchestrator coordinates stages rather than embedding logic in monolithic functions. This:

  • Ensures all features work with both local and GCE encoding
  • Makes each stage independently testable (139 new tests)
  • Allows feature flag rollback via USE_NEW_ORCHESTRATOR env var

Service Modules:

Module Purpose
youtube_upload_service.py YouTube OAuth flow and upload
discord_service.py Discord webhook notifications
packaging_service.py CDG/TXT package generation
local_encoding_service.py FFmpeg encoding (Cloud Run)
encoding_interface.py Abstract interface + GCE backend
video_worker_orchestrator.py Stage coordination
email_service.py SendGrid email delivery with CC support
template_service.py GCS-backed email templates
job_notification_service.py Email orchestration (completion, reminders)

Tech Stack

  • Backend: FastAPI, Python 3.12, Cloud Run
  • Frontend: Next.js, TypeScript, Cloudflare Pages
  • Database: Firestore
  • Storage: Google Cloud Storage
  • Secrets: Google Secret Manager
  • Task Queues: Cloud Tasks (idle reminders)
  • Email: SendGrid
  • IaC: Pulumi
  • CI/CD: GitHub Actions