## Summary
- Add automated email notification system for job completion and action
reminders
- Admin UI buttons to copy completion message or send email directly
- Comprehensive test suite (81 new tests)
- **Email subjects include release ID and song info** for easy
identification
## Changes
### Backend
- **Email Service** (`email_service.py`): Extended with SendGrid CC
support, `send_job_completion()` and `send_action_reminder()` methods
- **Job Notification Service** (`job_notification_service.py`): New
service orchestrating completion emails and action reminder scheduling
- **Template Service** (`template_service.py`): GCS-backed HTML email
templates with fallback defaults
- **Worker Service** (`worker_service.py`): Cloud Tasks integration for
delayed idle reminders (5-min delay)
- **Job Manager** (`job_manager.py`): State transition triggers for
automated notifications
- **Admin Routes** (`admin.py`): New endpoints `GET
/api/admin/jobs/{id}/completion-message` and `POST
/api/admin/jobs/{id}/send-completion-email`
- **Internal Routes** (`internal.py`): Idle reminder check endpoint for
Cloud Tasks
### Email Subject Format
Email subjects now include brand code and song info for easy
identification:
- With brand code: `NOMAD-1178: Seether - Tonight (Your karaoke video is
ready!)`
- Without brand code: `Seether - Tonight (Your karaoke video is ready!)`
- Without song info: `Your karaoke video is ready!`
### Frontend
- **OutputLinks Component**: Admin-only "Copy Message" and "Send Email"
buttons with dialog UI
- **E2E Helpers** (`email-testing.ts`): `waitForCompletionEmail()` and
`isCompletionEmail()` methods for testmail.app integration
- **Happy Path Test**: Step 9.5 verifies completion email arrives after
job completion
### Documentation
- Updated API.md with new endpoints
- Updated ARCHITECTURE.md with notification system architecture
- Updated README.md Recent Changes
## Testing
- [x] 81 new unit tests covering:
- `test_email_service.py` (27 tests) - providers, CC support, job
completion/reminder methods
- `test_job_notification_service.py` (21 tests) - URL building, message
generation
- `test_job_manager_notifications.py` (15 tests) - state transition
triggers
- `test_admin_email_endpoints.py` (12 tests) - admin API endpoints
- `test_template_service.py` (6 tests) - GCS template loading, fallbacks
- [x] E2E test extended to verify completion email delivery
- [x] All backend tests pass (`make test`)
- [x] TypeScript compiles
## Feature Flags
- `ENABLE_AUTO_EMAILS` (default: `false`) - Controls automatic email
sending on job state changes
## Review
- [x] Local CodeRabbit review completed (3 cycles)
- [x] Review feedback addressed:
- Fixed potential None state_data in internal.py
- Masked user emails in logs (PII protection)
- Fixed React setTimeout memory leak
- Changed placeholder URL default to empty
- Added CC email deduplication
@coderabbitai ignore
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
11 KiB
Architecture
System Overview
┌─────────────────────────────────────────────────────────────────┐
│ Frontend (Cloudflare Pages) │
│ Next.js + TypeScript │
│ https://gen.nomadkaraoke.com │
└───────────────────────────┬─────────────────────────────────────┘
│ HTTPS
▼
┌─────────────────────────────────────────────────────────────────┐
│ Backend API (Cloud Run) │
│ FastAPI + Python 3.12 │
│ https://api.nomadkaraoke.com │
│ │
│ Workers: │
│ • Audio Worker - Stem separation via Modal API │
│ • Lyrics Worker - Transcription via AudioShake │
│ • Screens Worker - Title/end screen generation │
│ • Render Worker - Karaoke video with lyrics │
│ • Video Worker - Final assembly and encoding │
└───────────────────────────┬─────────────────────────────────────┘
│
┌─────────────┼─────────────┬───────────────┐
▼ ▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────────┐ ┌──────────────┐
│ Firestore│ │ GCS │ │Secret Manager│ │GCE Encoding │
│ (jobs) │ │ (files) │ │ (API keys) │ │ Worker (*) │
└──────────┘ └──────────┘ └──────────────┘ └──────────────┘
(*) GCE Encoding Worker: C4-standard-8 VM with Intel Granite Rapids for
high-performance FFmpeg encoding. Used for both final video encoding and
preview video generation. Uses LocalEncodingService via GCS wheel deployment.
Processing Pipeline
1. UPLOAD
POST /api/jobs/upload
• Save audio to GCS
• Create job in Firestore
• Trigger parallel workers
2. PARALLEL PROCESSING
┌─────────────────┐ ┌─────────────────┐
│ Audio Worker │ │ Lyrics Worker │
│ Modal API │ │ AudioShake │
│ 2-stage sep │ │ Auto-correct │
└────────┬────────┘ └────────┬────────┘
└──────────┬───────────┘
▼
3. SCREENS GENERATION
• Title screen (5s video)
• End screen (5s video)
▼
4. HUMAN REVIEW (AWAITING_REVIEW)
• React UI for lyrics correction
• Preview video generation (optionally via GCE for speed)
• Submit corrections
▼
5. VIDEO RENDERING
• Merge corrections with original data
• Generate karaoke video with lyrics
▼
6. INSTRUMENTAL SELECTION
• User chooses: clean or with_backing
▼
7. FINAL VIDEO
• Remux with selected instrumental
• Concatenate: title + karaoke + end
• Encode to 4 formats
▼
8. COMPLETE
• User downloads via signed URLs
Job States
| State | Description | Human Action |
|---|---|---|
pending |
Job created | - |
downloading |
Processing audio | - |
separating_stage1 |
Audio separation stage 1 | - |
separating_stage2 |
Audio separation stage 2 | - |
transcribing |
Lyrics transcription | - |
generating_screens |
Creating title/end screens | - |
awaiting_review |
Waiting for human | Review lyrics |
in_review |
Human actively reviewing | Edit lyrics |
review_complete |
Review submitted | - |
rendering_video |
Generating karaoke video | - |
awaiting_instrumental_selection |
Waiting for human | Choose audio |
instrumental_selected |
Selection made | - |
generating_video |
Final encoding | - |
complete |
All done | Download files |
failed |
Error occurred | - |
GCS File Structure
jobs/{job_id}/
├── input/
│ └── original.flac # Uploaded audio
├── stems/
│ ├── vocals.flac
│ ├── instrumental_clean.flac
│ ├── instrumental_with_backing.flac
│ └── ...
├── lyrics/
│ ├── corrections.json # Auto-corrected
│ ├── corrections_updated.json # After human review
│ └── karaoke.ass
├── screens/
│ ├── title.mov
│ └── end.mov
├── videos/
│ └── with_vocals.mkv # Karaoke video
└── finals/
├── lossless_4k.mp4
├── lossless_4k.mkv
├── lossy_4k.mp4
└── lossy_720p.mp4
LyricsTranscriber Integration
Key Design Decision: We use LyricsTranscriber as a library, not a server.
CLI Mode (Original): Cloud Mode (Our Approach):
──────────────────── ──────────────────────────
LyricsTranscriber LyricsTranscriber
├── transcribe() ├── transcribe() ✅
├── correct() ├── correct() ✅
├── ReviewServer.start() │
│ └── BLOCKS (waits) │ (Skip server, save JSON)
│ │ ... ASYNC GAP ...
│ │ (Human reviews via React)
└── OutputGenerator │
└── OutputGenerator ✅
(called after review)
Use: LyricsTranscriber, CorrectionResult, OutputGenerator, OutputConfig
Don't use: ReviewServer, blocking waits
External Services
| Service | Purpose | Required |
|---|---|---|
| Modal | GPU audio separation | Yes |
| AudioShake | Lyrics transcription | Yes |
| Vertex AI | Agentic AI correction (Gemini 3 Flash) | Default on |
| Genius | Reference lyrics | Yes |
| YouTube API | Video upload | Optional |
| SendGrid | Email notifications | Optional |
| Cloud Tasks | Delayed task scheduling (idle reminders) | Optional |
Firestore Collections
karaoke-gen shares a GCP project (nomadkaraoke) with karaoke-decide, but uses separate Firestore collections:
| Collection | Purpose | Key Fields |
|---|---|---|
gen_users |
karaoke-gen user accounts | email, credits, role, is_active |
jobs |
Karaoke generation jobs | job_id, user_email, status, state_data |
jobs/{job_id}/logs |
Worker log entries (subcollection) | timestamp, level, worker, message, ttl_expiry |
sessions |
Magic link auth sessions | user_email, token, expires_at |
magic_links |
Passwordless auth tokens | email, token, expires_at, used |
beta_feedback |
Beta program feedback | user_email, ratings, comments |
Note: The users collection in the same Firestore instance belongs to karaoke-decide (different schema: user_id, is_guest, quiz_* fields). Don't use it for karaoke-gen.
Worker Logs: Stored in subcollection (jobs/{job_id}/logs) instead of embedded array to avoid Firestore's 1MB document size limit. TTL policy auto-deletes logs after 30 days. Feature flag: USE_LOG_SUBCOLLECTION (default: true). See LESSONS-LEARNED.md.
Video Worker Orchestrator
The Video Worker uses an orchestrator pattern to ensure all features work regardless of encoding backend (local Cloud Run or GCE VM).
┌─────────────────────────────────────────────────────────────┐
│ VideoWorkerOrchestrator │
│ │
│ Stage 1: PACKAGING Stage 2: ENCODING │
│ ├─ CDG file generation ├─ Local (FFmpeg) │
│ └─ TXT lyrics export └─ GCE (remote VM) │
│ │
│ Stage 3: ORGANIZATION Stage 4: DISTRIBUTION │
│ ├─ Final path setup ├─ Dropbox upload │
│ └─ File organization ├─ Google Drive upload │
│ └─ YouTube upload │
│ │
│ Stage 5: NOTIFICATIONS │
│ ├─ Discord webhooks │
│ └─ Email notifications (completion, reminders) │
└─────────────────────────────────────────────────────────────┘
Key Design Decision: The orchestrator coordinates stages rather than embedding logic in monolithic functions. This:
- Ensures all features work with both local and GCE encoding
- Makes each stage independently testable (139 new tests)
- Allows feature flag rollback via
USE_NEW_ORCHESTRATORenv var
Service Modules:
| Module | Purpose |
|---|---|
youtube_upload_service.py |
YouTube OAuth flow and upload |
discord_service.py |
Discord webhook notifications |
packaging_service.py |
CDG/TXT package generation |
local_encoding_service.py |
FFmpeg encoding (Cloud Run) |
encoding_interface.py |
Abstract interface + GCE backend |
video_worker_orchestrator.py |
Stage coordination |
email_service.py |
SendGrid email delivery with CC support |
template_service.py |
GCS-backed email templates |
job_notification_service.py |
Email orchestration (completion, reminders) |
Tech Stack
- Backend: FastAPI, Python 3.12, Cloud Run
- Frontend: Next.js, TypeScript, Cloudflare Pages
- Database: Firestore
- Storage: Google Cloud Storage
- Secrets: Google Secret Manager
- Task Queues: Cloud Tasks (idle reminders)
- Email: SendGrid
- IaC: Pulumi
- CI/CD: GitHub Actions