## Summary Follow-up to #1077. `video-encoding-job` runs both the post-review render (`render_video_worker`) and the final video (`video_worker`). In us-central1 each run waited **2–5 min** for Cloud Run Jobs to start it, so every customer job paid that twice. This moves it to us-east4 alongside the download, lyrics and bulk-search jobs. ## Cost (measured, not estimated) - Over the last 30 days the job read **156 GB** from Google services, mostly the us-central1 bucket, across 494 runs (Cloud Monitoring `container/network/received_bytes_count`, kind=google). That's about 0.32 GB per run. - Pricing from the Billing Catalog: GCS inter-region within North America is **free for the first 100 GB/month, then $0.02/GB**. Cloud Run writes to Google APIs are **$0**. - Internet egress (224 GB/month, Dropbox etc.) and Cloud Run CPU cost the same in either region. - Existing cross-region reads are about 24 GB/month (GPU job and lyrics), which the billing export shows at about $0. - **Net: about $1.60/month**, or about $3/month worst case. Time saved: roughly 4–8 min per customer job. ## Changes - **backend**: `trigger_video_worker` (via `_trigger_cloud_run_job`) and `trigger_render_video_worker` now use `cpu_jobs_region`. `GCP_REGION` stays us-central1; encoding VM and zone logic are unchanged. - **infra (Pulumi, already applied)**: new us-east4 `video-encoding-job`, with the legacy us-central1 copy kept for rollback and its deletion protection turned off. A preview after applying showed 290 unchanged. - **ci**: the video job image update now targets us-east4, using the us-east4 CPU image copy. - **Also updated**: rollback.sh, tests (video and render routing), TROUBLESHOOTING/ARCHITECTURE/LESSONS-LEARNED. ## Testing - `make test` passes: backend tests plus 1443 frontend tests. - Prod smoke test of the new us-east4 job: it imports `video_worker` and `render_video_worker` and calls the encoding VM's `/health` on its public IP (port 8080 is open to 0.0.0.0/0 with API-key auth). **Started in 25 s, exit 0.** 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Updates** - Video encoding and render jobs now run in the `us-east4` region, alongside other CPU jobs. - Video encoding continues to access storage and encoding resources in `us-central1`; cross-region transfer may incur costs. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
49 KiB
Architecture
System Overview
┌─────────────────────────────────────────────────────────────────┐
│ Frontend (Cloudflare Pages) │
│ Next.js + TypeScript │
│ https://gen.nomadkaraoke.com │
└───────────────────────────┬─────────────────────────────────────┘
│ HTTPS
▼
┌─────────────────────────────────────────────────────────────────┐
│ Cloudflare edge (api.nomadkaraoke.com proxied — orange-cloud) │
│ WAF (exploit-path block) + rate limit + injects X-Edge-Auth │
│ header. See § Edge Security (Cloudflare). │
└───────────────────────────┬─────────────────────────────────────┘
│ HTTPS (SSL "full")
▼
┌─────────────────────────────────────────────────────────────────┐
│ Backend API (Cloud Run) │
│ FastAPI + Python 3.12 │
│ https://api.nomadkaraoke.com (origin lock: rejects direct- │
│ to-origin requests lacking X-Edge-Auth, health/root exempt) │
│ │
│ Workers: │
│ • Audio Worker - Stem separation via Cloud Run GPU (L4) │
│ • Lyrics Worker - Transcription via AudioShake │
│ • Screens Worker - Title/end screen generation │
│ • Render Worker - Karaoke video with lyrics │
│ • Video Worker - Final assembly and encoding │
└───────────────────────────┬─────────────────────────────────────┘
│
┌─────────────┼─────────────┬───────────────┐
▼ ▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────────┐ ┌──────────────┐
│ Firestore│ │ GCS │ │Secret Manager│ │GCE Encoding │
│ (jobs) │ │ (files) │ │ (API keys) │ │ Worker (*) │
└──────────┘ └──────────┘ └──────────────┘ └──────────────┘
(*) GCE Encoding Worker: a Spot c4d-highcpu-16 blue-green pair (AMD EPYC
Turin, the preferred pick) plus ONE on-demand c2d-highcpu-16 fallback in a
different zone for Spot/c4d stockouts (3 VMs since 2026-09-26 — see
docs/archive/2026-09-26-encoding-cost-cuts.md). Candidates are selected
fastest-first with a capacity cooldown.
Used for both final video encoding and preview video generation.
Uses immutable deployment pattern - see infrastructure/encoding-worker/README.md
and the "Encoding Worker" section below.
Error Monitor
Cloud Logging (all services) ──→ Error Monitor (Cloud Run Job, hourly)
Frontend clients ───────────────→ POST /api/client-errors ──→ Firestore
│
├─ Normalize & deduplicate
├─ LLM incident grouping (Gemini Flash)
├─ Discord alerts (new patterns, spikes, daily digest)
├─ Firestore: error_patterns, error_incidents
└─ Auto-resolve silent patterns
Entry point: python -m backend.services.error_monitor.monitor
Design spec: docs/archive/2026-04-12-error-monitor-design.md
Frontend crash reporting: docs/archive/2026-04-19-frontend-crash-reporting-plan.md
Frontend crashes are captured via window.onerror, unhandledrejection, Next.js
error.tsx, and a CrashReportBoundary around the review UIs. They POST to
/api/client-errors, which normalises and upserts into the same error_patterns
collection with service="frontend" so they flow through the existing
Discord-alerting pipeline. ChunkLoadError after deploy triggers an automatic
hard reload; the CrashReport card also checks /version.json and shows an
"Update now" CTA when the running bundle is stale.
Noise is dropped before alerting: benign messages (media AbortError, ResizeObserver
loop) and opaque cross-origin Script error. events (no error object — thrown by
third-party/extension scripts) are skipped client-side, and /api/client-errors
accepts-but-ignores reports from bot user agents and Script error. window errors
with a blank filename/lineno
(covers older cached bundles).
Edge Security (Cloudflare)
api.nomadkaraoke.com is proxied through Cloudflare (orange-cloud) — the same
zone that already fronts the marketing site, gen, decide, and the tenant
portals. Cut over 2026-07-20 after a vuln-scanner hammered the previously
directly-exposed Cloud Run origin. Managed as IaC in
infrastructure/modules/edge_security.py (pulumi_cloudflare).
Request path: client → Cloudflare edge → Cloud Run. At the edge:
- WAF (
http_request_firewall_custom) blocks exploit/secret-exposure paths (/.env,/.git/,/.ssh/,wp-config,/etc/passwd, …) — uses thecontainsoperator (regexmatchesneeds a Business plan; zone is Free). - Rate limiting (
http_ratelimit) — per-IP flood control (Free-plan limits:period=10s, characteristics must includecf.colo.id). Excludes/api/internal/*(scheduler/Cloud-Tasks cron). - Origin lock — a Transform Rule injects a secret
X-Edge-Authheader (http_request_late_transform); the backend'sEdgeAuthMiddleware(backend/middleware/edge_auth.py,EDGE_AUTH_MODE=enforce) rejects public requests lacking it, so hitting the raw*.run.apporigin directly is blocked (health/root exempt for Cloud Run probes). Internal callers reach the backend viaapi.nomadkaraoke.com(Cloud Tasks useCLOUD_RUN_SERVICE_URL), so they traverse the edge and carry the header.
SSL: zone TLS mode is "full" (not strict). The Cloud Run domain-mapping
managed cert cannot provision behind Cloudflare (ACME challenge not visible), but
"full" mode doesn't validate the origin cert, so the edge works and is
renewal-safe — no run.app Origin Rule or load balancer needed. Never switch
the zone to "full (strict)". Full rationale + cutover runbook:
docs/archive/2026-07-20-edge-security-hardening-plan.md and
docs/archive/2026-07-20-edge-security-cutover-runbook.md.
Processing Pipeline
1. UPLOAD
POST /api/jobs/upload
• Save audio to GCS
• Create job in Firestore
• Trigger parallel workers
2. PARALLEL PROCESSING (with Worker Registry coordination)
┌─────────────────┐ ┌─────────────────┐
│ Audio Worker │ │ Lyrics Worker │
│ Cloud Run GPU │ │ AudioShake │
│ 2-stage sep │ │ Auto-correct │
│ ↓ register() │ │ ↓ register() │
│ ↓ unregister() │ │ ↓ unregister() │
└────────┬────────┘ └────────┬────────┘
└──────────┬───────────┘
▼
3. SCREENS GENERATION + BACKING VOCALS ANALYSIS + AUDIO TRANSCODING
• Title screen (5s video)
• End screen (5s video)
• Analyze backing vocals track (for instrumental selection)
• Transcode audio to OGG Opus for review UI (~3 MB vs 35 MB FLAC)
▼
4. COMBINED HUMAN REVIEW (AWAITING_REVIEW)
• React UI for lyrics correction
• Preview video generation (optionally via GCE for speed)
• Instrumental track selection (clean vs with_backing)
• Single submit saves both lyrics + instrumental choice
▼
5. VIDEO RENDERING (optionally via GCE for speed + memory)
• Merge corrections with original data
• Generate karaoke video with lyrics (OutputGenerator + ffmpeg/libass)
• Style assets cached on GCE worker disk between jobs
▼
6. FINAL VIDEO (INSTRUMENTAL_SELECTED, via GCE)
• Remux with selected instrumental
• Concatenate: title + karaoke + end
• Encode to 4 formats
▼
7. COMPLETE
• User downloads via signed URLs
Note: As of 2026-01, instrumental selection is combined with lyrics review. Users complete both in a single session rather than in separate steps.
Job States
| State | Description | Human Action |
|---|---|---|
pending |
Job created | - |
downloading |
Processing audio | - |
awaiting_duration_confirm |
Waiting for human — duration-based credit cost confirmation before separation/transcription (see Duration-Based Pricing) | Confirm credit charge (or buy more credits) |
separating_stage1 |
Audio separation stage 1 | - |
separating_stage2 |
Audio separation stage 2 | - |
transcribing |
Lyrics transcription | - |
generating_screens |
Creating title/end screens + backing vocals analysis | - |
awaiting_review |
Waiting for human | Review lyrics + select instrumental |
in_review |
Human actively reviewing | Edit lyrics + select instrumental |
review_complete |
Review submitted with instrumental selection | - |
rendering_video |
Generating karaoke video | - |
instrumental_selected |
Instrumental confirmed, ready for final video | - |
generating_video |
Final encoding | - |
complete |
All done | Download files |
failed |
Error occurred | - |
Note: awaiting_instrumental_selection exists for backwards compatibility with historical jobs but is no longer used. Instrumental selection is now part of the combined review (awaiting_review → in_review → review_complete).
awaiting_duration_confirm details: This state is a blocking human checkpoint inserted before
separating_stage1. It is entered when the actual measured duration costs more than was already
charged (stored in state_data.duration_confirm_reason as "reconcile"). This applies to all
job types (URL, upload, search): 1 credit is charged at creation; ffprobe measures the actual audio
via a signed URL just before workers start; if the true cost exceeds the hold,
state_data.pending_additional_credits records the delta owed and the job pauses here.
POST /api/jobs/{id}/confirm-duration deducts the owed credits and triggers workers.
If the actual duration exceeds 60 min all charged credits are refunded and the job is cancelled.
If it costs less than charged an auto-refund is issued silently. Admin/made-for-you jobs bypass
this state entirely. See docs/archive/2026-06-04-long-duration-input-handling-design.md.
State Machine Robustness (Feb 2026): State transitions are enforced by STATE_TRANSITIONS in models/job.py. Invalid transitions raise InvalidStateTransitionError by default. Use scripts/generate_state_diagram.py to generate a Mermaid diagram of valid transitions. The /health/job-consistency endpoint detects jobs stuck in invalid states. See docs/archive/2026-02-02-state-machine-robustness-plan.md for implementation details.
GCS File Structure
jobs/{job_id}/
├── input/
│ └── original.flac # Uploaded audio
├── stems/
│ ├── vocals.flac
│ ├── instrumental_clean.flac
│ ├── instrumental_with_backing.flac
│ └── ...
├── lyrics/
│ ├── corrections.json # Auto-corrected
│ ├── corrections_updated.json # After human review
│ ├── edit_log_{session}.json # User edit log with feedback
│ ├── annotations.json # Correction annotations
│ └── karaoke.ass
├── screens/
│ ├── title.mov
│ └── end.mov
├── videos/
│ └── with_vocals.mkv # Raw karaoke video (FLAC audio, large)
└── finals/
├── with_vocals_mp4.mp4 # Encoded with-vocals (H264/AAC, ~3x smaller)
├── lossless_4k.mp4
├── lossless_4k.mkv
├── lossy_4k.mp4
└── lossy_720p.mp4
LyricsTranscriber Integration
Key Design Decision: We use LyricsTranscriber as a library, not a server.
CLI Mode (Original): Cloud Mode (Our Approach):
──────────────────── ──────────────────────────
LyricsTranscriber LyricsTranscriber
├── transcribe() ├── transcribe() ✅
├── correct() ├── correct() ✅
├── ReviewServer.start() │
│ └── BLOCKS (waits) │ (Skip server, save JSON)
│ │ ... ASYNC GAP ...
│ │ (Human reviews via React)
└── OutputGenerator │
└── OutputGenerator ✅
(called after review)
Use: LyricsTranscriber, CorrectionResult, OutputGenerator, OutputConfig
Don't use: ReviewServer, blocking waits
External Services
| Service | Purpose | Required |
|---|---|---|
| Cloud Run GPU Job (L4, us-east4) | Audio stem separation — Separator runs directly in the audio worker Job (no HTTP hop) |
Yes |
| AudioShake | Lyrics transcription | Yes |
| Vertex AI | Agentic AI correction (Gemini 3.8 Flash) | Default off (SKIP_CORRECTION=false to enable) |
| Genius | Reference lyrics | Yes |
| Flacfetch | Audio downloads (YouTube, torrents) | Recommended* |
| YouTube API | Video upload | Optional |
| Postmark | Email notifications | Optional |
| Cloud Tasks | Delayed task scheduling (idle reminders, YouTube queue) | Optional |
| Cloud Scheduler | Hourly YouTube upload queue processing | Optional |
| Cloud Monitoring | Cross-reference YouTube API quota consumption from GCP metrics | Optional |
| karaoke-decide | Song catalog (MusicBrainz + Spotify) for autocomplete | Optional |
| KaraokeNerds | Community karaoke version detection | Optional |
*Flacfetch runs on a dedicated server (Netcup VPS, reached over public HTTPS) with YouTube cookies and tracker access. Without it, YouTube downloads will fail due to bot detection on Cloud Run IPs.
Data Pipeline Cloud Functions
Three Cloud Functions manage the community karaoke data pipeline, running daily on automated schedules:
| Function | Schedule | Purpose |
|---|---|---|
divebar-mirror |
Daily 2:00 AM ET | Indexes 48K+ files from diveBar Karaoke Google Drive into BigQuery divebar_catalog |
kn-data-sync |
Daily 3:05 AM + 4:30 AM ET | Fetches KaraokeNerds catalog (281K songs) and community tracks (58K with YouTube URLs) to BigQuery + GCS. full mode then exports the kjbox song-identification index (≈6.6M rows: MusicBrainz songs scored/kept by Spotify track popularity and ListenBrainz listeners (lb_* tables from karaoke-decide's lb-refresh), the Spotify snapshot ≥ 30, and all KN; ~8 GB scanned, ~25 s; EXPORT DATA → gs://nomadkaraoke-kn-data/song-id/<run>/songs-*.tsv.gz, manifest song-id/latest.json, last 3 runs kept), which the NomadPC downloads. Design: kjbox docs/SONG-IDENTIFICATION.md |
divebar-lookup |
Daily 6:00 AM ET + on-demand API | Public search/lookup API for KJ Controller; rebuilds KN↔Divebar cross-reference index |
Infrastructure (Pulumi-managed in infrastructure/):
- Service accounts:
divebar-mirror@,kn-data-sync@,divebar-lookup@(least-privilege IAM) - GCS:
nomadkaraoke-kn-data(raw exports), 3 source buckets for function code - BigQuery tables:
divebar_catalog,karaokenerds_community,kn_divebar_xref(inkaraoke_decidedataset) - Secret:
karaokenerds-api-key(KN API access)
Source code: infrastructure/functions/{divebar_mirror,kn_data_sync,divebar_lookup}/
Firestore Collections
karaoke-gen shares a GCP project (nomadkaraoke) with karaoke-decide, but uses separate Firestore collections:
| Collection | Purpose | Key Fields |
|---|---|---|
gen_users |
karaoke-gen user accounts | email, credits, role, is_active |
jobs |
Karaoke generation jobs | job_id, user_email, status, state_data, processing_metadata |
jobs/{job_id}/logs |
Worker log entries (subcollection) | timestamp, level, worker, message, ttl_expiry |
sessions |
Magic link auth sessions | user_email, token, expires_at |
magic_links |
Passwordless auth tokens | email, token, expires_at, used |
user_feedback |
User feedback for credits (all users) | user_email, ratings, comments, created_at |
beta_feedback |
Beta program feedback (deprecated) | user_email, ratings, comments |
youtube_quota_pending |
Pending YouTube API quota buffer (~7min GCP delay) | pending_units, pending_uploads[], updated_at |
youtube_upload_queue |
Deferred YouTube uploads | job_id, status, user_email, queued_at, youtube_url |
Note: The users collection in the same Firestore instance belongs to karaoke-decide (different schema: user_id, is_guest, quiz_* fields). Don't use it for karaoke-gen.
Worker Logs: Stored in subcollection (jobs/{job_id}/logs) instead of embedded array to avoid Firestore's 1MB document size limit. TTL policy auto-deletes logs after 30 days. Feature flag: USE_LOG_SUBCOLLECTION (default: true). See LESSONS-LEARNED.md.
Encoding Worker: blue-green primary pair + multi-family fallback pool
The GCE encoding worker is a blue-green primary pair (encoding-worker-a/-b,
both Spot c4d-highcpu-16 in us-central1-c) plus one on-demand fallback
(encoding-worker-fallback-c2df, c2d-highcpu-16, us-central1-f). 3 VMs total
(cut from 10 on 2026-09-26 — docs/archive/2026-09-26-encoding-cost-cuts.md), all
TERMINATED by default, started on demand, and idle-stopped after 5 min.
Spot preemption: the pair uses instance_termination_action=STOP, so a
preempted worker is just a TERMINATED VM that the next request restarts. An
encode in flight when the VM is preempted is detected by
EncodingService.wait_for_completion (the pinned worker VM is no longer RUNNING)
and raised as EncodingJobLostError, which run_with_lost_job_resubmit resubmits
— the resubmit's connection failure then starts the Spot VM again, or the
on-demand fallback if Spot capacity is gone.
Architecture
Frontend → Backend (Cloud Run) → Firestore config → active worker (primary, or a
↑ capacity-fallback "override")
Cloud Scheduler (2 min) → Cloud Function → checks idle (5 min) + stops VMs
↑
CI (GitHub Actions) → selects a fresh "green" (ranked pool) → health + encode test
→ promote (swap primary OR set active_override) → stop old
Multi-instance-type pool (stockout resilience, v0.195.0)
A single machine family can hit a region-wide ZONE_RESOURCE_POOL_EXHAUSTED
stockout across every zone at once. To survive that, the fallback lives in a
different family AND zone from the c4d pair: encoding-worker-fallback-c2df
(c2d-highcpu-16, on-demand, us-central1-f). The pool was 8 fallbacks across 6
families (v0.195.0) until the 2026-09-26 cost cut; the selection machinery below
still handles any number of fallbacks — re-add entries to
EncodingWorkerConfig.FALLBACKS + the encoding-worker-fallback-vms secret if
stockouts become a problem again.
Candidate ordering is one shared pure module,
backend/services/encoding_worker_preference.py::ordered_candidates(pool, capacity_state),
used by BOTH runtime selection and deploy green selection so they can't drift:
- Fastest-first by
SPEED_RANK(c4d preferred whenever it can start). - Availability-aware: a family that recently hit a capacity/stockout error is
recorded in the Firestore
capacity_statemap and demoted for a 15-min cooldown, then re-probed — so we stop wasting ~2 min probing a dead family but snap back to c4d the moment capacity returns.
The fallback fleet is defined in infrastructure/config.py::EncodingWorkerConfig.FALLBACKS
and published to the backend/idle-fn via the encoding-worker-fallback-vms secret
(entries: {vm, zone, ip, machine_type}).
Key Components
Primary pair + fallback fleet: All VMs TERMINATED by default. The primary pair
(a/b) does blue-green deploys; the 8 fallbacks absorb capacity stockouts. Runtime
warmup (encoding_worker_manager.ensure_any_running) tries the ranked candidates
in order, falling through on stockout and recording an active_override when it
lands on a fallback.
Firestore config doc (config/encoding-worker): source of truth for routing.
Fields: primary_vm/primary_ip/primary_version, secondary_vm/…, the
capacity-fallback pointer active_override_vm/_ip/_zone/_version (set when a
fallback is serving), deploy_in_progress, last_activity_at, and the
capacity_state map ({"<machine_type>@<zone>": last_stockout_iso}). active_url
= override IP if set, else primary IP.
Backend routing (encoding_interface.py): reads the active worker IP from the
config doc with a 30-second TTL cache.
JIT startup: entering the lyrics-review page starts the worker on demand (~60 s natural warmup), avoiding 24/7 cost and cold-start latency.
Idle auto-shutdown: a Cloud Function (every 5 min) stops idle VMs (primary, secondary, and every fallback in the secret) based on Firestore activity — external to the VM so a sick worker still gets stopped.
Capacity-aware deploys: CI selects a FRESH "green" from the ranked pool via
deploy_promote.select_green_candidates (the same preference module) — the c4d
secondary if it can start, else a fallback family, never the currently-serving
override. After a real encode test on the green, it promotes: a c4d-secondary green
does the classic primary swap (and clears any stale override); a fallback green
becomes the active_override. The retired VM is drained + stopped; a failed health
check never promotes. Override routing keys off an explicit is_primary flag, not
list position. See infrastructure/encoding-worker/deploy_promote.py.
Self-sufficient image: the Packer image (infrastructure/packer/scripts/provision.sh)
bakes the full karaoke-gen dependency tree with CPU-only torch, so a fresh VM of
any family boots ready to encode (no first-job dependency install).
Seeding the Config
Initial setup or reset:
python scripts/seed-encoding-worker-config.py <ip-a> <ip-b>
Deploy Flow (capacity-aware, v0.194.2+)
1. CI builds new karaoke-gen wheel + pushes to GCS
2. CI picks a FRESH "green" from the ranked pool (select_green_candidates →
ordered_candidates): the c4d secondary if it can start, else the fastest
available fallback family — never the currently-serving active_override.
3. CI starts the green, waits for /health, runs a real encode test against it.
4. If the test FAILS: CI stops the green; primary/override unchanged (no promote).
5. If the test PASSES: CI promotes the green:
- green is the c4d secondary → swap it to `primary_vm` (classic blue-green)
AND clear any stale `active_override_*` so
traffic returns to the fresh c4d primary.
- green is a fallback family → set it as `active_override_*` (the serving
worker); leave primary/secondary so c4d
resumes as primary when its capacity returns.
6. CI drains (waits active_jobs==0, bounded) + stops the retired VM — never the
promoted green.
7. Last resort (no separate green can start, e.g. deep stockout): in-place restart
of the current serving override — a brief blip, covered by client auto-resubmit.
Override routing keys off an explicit is_primary flag (not list position), so a
fallback ranked ahead of a stocked-out primary is still routed correctly. Pure
decision logic + unit tests: infrastructure/encoding-worker/deploy_promote.py.
Video Worker Orchestrator
The Video Worker uses an orchestrator pattern to ensure all features work regardless of encoding backend (local Cloud Run or GCE VM).
┌─────────────────────────────────────────────────────────────┐
│ VideoWorkerOrchestrator │
│ │
│ Stage 1: PACKAGING Stage 2: ENCODING │
│ ├─ CDG file generation ├─ Local (FFmpeg) │
│ └─ TXT lyrics export └─ GCE (remote VM) │
│ │
│ Stage 3: ORGANIZATION Stage 4: DISTRIBUTION │
│ ├─ Final path setup ├─ Dropbox upload │
│ └─ File organization ├─ Google Drive upload │
│ └─ YouTube upload │
│ │
│ Stage 5: NOTIFICATIONS │
│ ├─ Discord webhooks │
│ └─ Email notifications (completion, reminders) │
└─────────────────────────────────────────────────────────────┘
Key Design Decision: The orchestrator coordinates stages rather than embedding logic in monolithic functions. This:
- Ensures all features work with both local and GCE encoding
- Makes each stage independently testable (139 new tests)
- Allows feature flag rollback via
USE_NEW_ORCHESTRATORenv var
Service Modules:
| Module | Purpose |
|---|---|
youtube_download_service.py |
YouTube audio downloads (via remote flacfetch) |
flacfetch_client.py |
Client for remote flacfetch HTTP API |
audio_search_service.py |
Audio source search (YouTube, RED, OPS, Spotify) |
youtube_upload_service.py |
YouTube OAuth flow and upload |
discord_service.py |
Discord webhook notifications |
packaging_service.py |
CDG/TXT package generation |
local_encoding_service.py |
FFmpeg encoding (Cloud Run) |
encoding_interface.py |
Abstract interface + GCE backend |
video_worker_orchestrator.py |
Stage coordination |
email_service.py |
Postmark email delivery with CC support |
template_service.py |
GCS-backed email templates |
job_notification_service.py |
Email orchestration (completion, reminders) |
audio_transcoding_service.py |
Transcode FLAC → OGG Opus for review UI playback |
youtube_quota_service.py |
Track YouTube API quota (GCP Cloud Monitoring + Firestore pending buffer) |
email_validation_service.py |
Email validation, tiered disposable domain detection (static list + DeBounce + verifymail.io APIs), blocklist management |
youtube_upload_queue_service.py |
Deferred YouTube upload queue management |
catalog_proxy_service.py |
Proxy to karaoke-decide catalog API (artist/track search) |
karaokenerds_service.py |
Scrape karaokenerds.com for community karaoke versions |
Audio Source Download Flow
Audio can come from file upload or remote search (YouTube, torrents). All YouTube downloads route through YouTubeDownloadService to ensure consistent handling:
┌──────────────────────────────────────────────────────────────────────┐
│ Audio Source Entry Points │
│ │
│ 1. File Upload (/api/jobs/upload) │
│ └─ Direct to GCS → triggers workers │
│ │
│ 2. YouTube URL (/api/jobs/create-from-url) │
│ └─ YouTubeDownloadService → GCS → triggers workers │
│ │
│ 3. Audio Search + Select (/api/audio-search/...) │
│ ├─ YouTube → YouTubeDownloadService → GCS → triggers workers │
│ ├─ RED/OPS → FlacfetchClient (torrent) → GCS → triggers workers │
│ └─ Spotify → FlacfetchClient (spotify) → GCS → triggers workers │
└──────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────┐
│ YouTubeDownloadService │
│ │
│ • Single entry point for ALL YouTube downloads in cloud │
│ • Uses remote flacfetch when FLACFETCH_API_URL is set (recommended) │
│ • Falls back to local yt_dlp with warning (usually fails on CR) │
│ • Handles all URL formats (watch, youtu.be, shorts, embed) │
└──────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────┐
│ Flacfetch server (Netcup VPS, off-GCP) │
│ │
│ • Has YouTube cookies (avoids bot detection) │
│ • Has tracker credentials (RED, OPS private trackers) │
│ • Runs BitTorrent client for seeding │
│ • Uploads directly to GCS (no data transfer through Cloud Run) │
└──────────────────────────────────────────────────────────────────────┘
Key environment variables:
FLACFETCH_API_URL- public HTTPS URL of the flacfetch server (secretflacfetch-api-url). The serverless VPC connector that once let Cloud Run reach the old GCE VM's private IP was deleted 2026-09-26; the backend andaudio-download-job/bulk-search-jobhave no VPC egress, so every dependency they call must be reachable on a public address.FLACFETCH_API_KEY- API key for authentication
Worker Execution
Workers run via two mechanisms depending on their processing duration:
Cloud Run Jobs (Long-Running Workers)
Audio and lyrics workers run as Cloud Run Jobs - standalone batch containers that run to completion without HTTP request lifecycle concerns. This eliminates the instance termination issue where Cloud Run would shut down instances mid-processing.
┌─────────────────────────────────────────────────────────────────┐
│ Cloud Run Jobs (via WorkerService) │
│ │
│ audio-download-job - 30s-5 min (flacfetch/YouTube, us-east4) │
│ lyrics-transcription-job - 5-15 min (AudioShake + correction, us-east4)│
│ bulk-search-job - batch audio search (us-east4) │
│ audio-separation-job - 10-20 min (L4 GPU, us-east4, direct) │
│ video-encoding-job - up to 60 min (us-east4) │
│ │
│ Triggered via: google.cloud.run_v2.JobsClient.run_job() │
│ Job ID passed as: --job-id argument │
│ (bulk-search-job takes --batch-id instead) │
└─────────────────────────────────────────────────────────────────┘
Job regions: the CPU jobs a customer waits on (download, lyrics, bulk search, video/render) run in
us-east4 (CPU_JOBS_REGION), because Cloud Run Jobs in us-central1 wait 2-5 min before the
container starts. The bucket and GCE encoding VMs stay in us-central1. See TROUBLESHOOTING.md, "Cloud Run Jobs
take 2-5 minutes to start".
Why Cloud Run Jobs: When using FastAPI BackgroundTasks, Cloud Run would terminate instances when HTTP requests completed, even if background work was still running. Cloud Run Jobs solve this by running workers as standalone processes that complete naturally. This applies to audio downloads (jobs 51b8231d, 89e497b1 were killed mid-download), lyrics transcription (job c94cc9d6 killed mid-processing), and audio separation.
CLI Entry Points: Each worker has a main() function for Cloud Run Job execution:
python -m backend.workers.audio_download_worker --job-id abc123python -m backend.workers.lyrics_worker --job-id abc123python -m backend.workers.audio_worker --job-id abc123python -m backend.workers.video_worker --job-id abc123
HTTP Workers (Fast Workers)
Screens and render workers run via internal HTTP endpoints with BackgroundTasks, as they complete quickly (~30 seconds). They still use the WorkerRegistry for coordination:
┌─────────────────────────────────────────────────────────────────┐
│ WorkerRegistry │
│ │
│ register(job_id, "screens") unregister(job_id, "screens") │
│ register(job_id, "render") unregister(job_id, "render") │
│ │
│ has_active_workers() → bool │
│ wait_for_completion(timeout) → bool │
└─────────────────────────────────────────────────────────────────┘
Shutdown Handler (in main.py lifespan):
- On shutdown signal, checks
worker_registry.has_active_workers() - If workers active, calls
wait_for_completion(timeout=600)(10 min max) - Logs which workers are still running for debugging
Audio Separation Architecture
Audio stem separation runs directly inside the audio worker Cloud Run Job — the Separator class from audio-separator is imported and called in-process. There is no separate GPU microservice.
┌──────────────────────────────────────────────────────────────────┐
│ audio-separation-job (Cloud Run Job, L4 GPU, us-east4) │
│ │
│ AudioWorker │
│ ├── download audio from GCS │
│ ├── create_audio_processor() ← Separator(model_file_dir=...) │
│ ├── Stage 1: instrumental_clean preset │
│ │ └── produces: vocals.flac, instrumental_clean.flac │
│ ├── Stage 2: karaoke preset │
│ │ └── produces: instrumental_with_backing.flac │
│ └── upload stems to GCS │
└──────────────────────────────────────────────────────────────────┘
Key decisions:
- Direct GPU — No HTTP hop to a separate service. Jobs can run up to 24h; no timeout concerns.
- Two-stage ensemble preset —
instrumental_clean(stage 1) +karaoke(stage 2). - Model cache — Models are downloaded to
model_file_dir(GCS-backed or local) and reused across executions.
The audio-separator Cloud Run GPU service (us-east4, scale-to-zero) is no longer used by
prod gen jobs, but it still serves external remote-separation clients (audio-separator-remote /
local karaoke-gen CLI with AUDIO_SEPARATOR_API_URL; ~40 /separate calls/month as of
2026-09). Keep it at min-instances 0; don't remove it without checking its request logs first.
Quick version (kjbox make-it jobs)
For jobs created by kjbox on behalf of a singer (request_metadata.client_id starts with
kjbox), the audio worker also produces a rough scrolling-lyrics draft video within a few
minutes of the audio landing, so the singer can sing now instead of waiting ~30 min for the
full NOMAD version. Code: backend/services/quick_version/ (a port of kjbox fastgen/).
- Before the ensemble, one fast single-model pass (
UVR-MDX-NET-Inst_HQ_4.onnx, fastgen's model; baked bydownload_models.pyEXTRA_MODELS, downloaded on load until the next GPU base rebuild; falls back to the always-bakedmel_band_roformer_instrumental_fv7z_gabox.ckpt, which took 14 s load + 124 s for a 3:15 song on the L4; override withQUICK_VERSION_MODEL) → quick instrumental + vocals. - A background thread renders the video while the GPU runs the ensemble: LRCLIB lyrics
(line-synced → time-anchored scroll; plain → constant crawl; none → title card), PIL-rasterised
crawl PNG + one ffmpeg
overlaypass, 854×480, vocals mixed back in at 30% as a guide. Every ffmpeg input and the output are bounded with-t <duration>: the GPU image's ffmpeg 4.4 never ends on-shortestalone onceamixis involved (first prod job ran into the 300 s join timeout). - Uploads
jobs/{id}/quick/quick.mp4→file_urls.quick.video_mp4, and recordsstate_data.quick_version = {status: separating|rendering|ready|failed, lyrics_tier, ready_at, …}. - The worker joins the render (≤300 s) before cleaning up its temp dir — even when the ensemble
fails. Quick-version failures never fail the job. Skipped when
quick_version.status == ready(job retries), in remote-API separation mode, or withKJBOX_QUICK_VERSION_ENABLED=false.
kjbox's GenPoller reads state_data.quick_version from GET /api/jobs/{id} and downloads the file
via /api/jobs/{id}/download/quick/video_mp4.
Multitenancy
The platform supports white-label B2B portals where business customers get their own branded karaoke generation experience.
Architecture
┌─────────────────────────────────────────────────────────────────┐
│ Tenant Detection Flow │
│ │
│ Request arrives at vocalstar.nomadkaraoke.com │
│ │ │
│ ▼ │
│ Frontend: detectTenantFromUrl() │
│ - Extracts subdomain from hostname │
│ - Calls GET /api/tenant/config?tenant=vocalstar │
│ │ │
│ ▼ │
│ Backend: TenantMiddleware │
│ 1. X-Tenant-ID header (from frontend) │
│ 2. Query param (dev only, disabled in production) │
│ 3. Host header subdomain detection │
│ │ │
│ ▼ │
│ TenantService.get_tenant_config() │
│ - Loads from GCS: tenants/{tenant_id}/config.json │
│ - Caches in memory (5 min TTL) │
│ │ │
│ ▼ │
│ Request proceeds with tenant context │
│ - request.state.tenant_id │
│ - request.state.tenant_config │
└─────────────────────────────────────────────────────────────────┘
GCS Storage Layout
tenants/{tenant_id}/
├── config.json # TenantConfig with branding, features, defaults
└── logo.jpg # Tenant logo (optional)
themes/{theme_id}/
├── style_params.json # Full theme configuration
└── assets/
├── intro_background.png
├── karaoke_background.jpg
├── end_background.png
├── font.ttf
└── cdg_*.gif
Tenant Config Schema
TenantConfig:
id: str # e.g., "vocalstar"
name: str # Display name: "Vocal Star"
subdomain: str # e.g., "vocalstar.nomadkaraoke.com"
is_active: bool # Enable/disable tenant
branding:
logo_url: str | None # GCS path to logo
logo_height: int # Pixels
primary_color: str # Hex color
secondary_color: str
accent_color: str | None
background_color: str | None
favicon_url: str | None
site_title: str
tagline: str | None
features:
audio_search: bool # Enable audio search (vs file upload only)
file_upload: bool
youtube_url: bool
youtube_upload: bool # YouTube distribution
dropbox_upload: bool
gdrive_upload: bool
theme_selection: bool # Allow user to pick theme
color_overrides: bool
enable_cdg: bool
enable_4k: bool
admin_access: bool
defaults:
theme_id: str | None # Default theme if not locked
locked_theme: str | None # Force this theme (user cannot change)
distribution_mode: str # "all", "download_only", etc.
auth:
allowed_email_domains: list[str] # e.g., ["vocal-star.com"]
require_email_domain: bool
sender_email: str | None # Override email sender
Feature Enforcement
Frontend: The useTenant() hook provides features object. UI components conditionally render based on enabled features:
const { features } = useTenant()
if (!features.audio_search) {
// Hide audio search tab
}
Backend: Routes check tenant config from request state:
tenant_config = get_tenant_config_from_request(request)
if tenant_config and not tenant_config.features.audio_search:
raise HTTPException(403, "Audio search not enabled for this tenant")
Active Tenants
Vocal Star (vocalstar.nomadkaraoke.com)
- Features: File upload only (no audio search, no YouTube URL), streamlined 4-field form (Artist, Title, Mixed Audio, Instrumental Audio)
- Distribution: Cloud only (Dropbox + Google Drive to tenant-specific folders, no YouTube)
- Theme: Locked to "vocalstar" theme (yellow/blue)
- Auth: Restricted to
@vocal-star.comand@vocalstarmusic.comemails - All jobs automatically private with VSTAR brand prefix
- Setup:
python scripts/setup-vocalstar-tenant.py
Singa (singa.nomadkaraoke.com)
- Features: File upload only (no audio search, no YouTube URL)
- Distribution: Download only
- Theme: Locked to "singa" theme (green/black)
- Auth: Restricted to
@singa.comemails - Setup:
python scripts/setup-singa-tenant.py
Tenant Portal Frontend (B2B Deployment)
B2B tenants are served from a separate Cloudflare Pages project (karaoke-gen-tenant) built with NEXT_PUBLIC_TENANT_PORTAL=true. This build flag enables tree-shaking of consumer-only content (landing page, pricing) at build time, keeping the bundle lean for white-label deployments.
Runtime tenant injection: A Cloudflare Pages edge function intercepts HTML responses and injects tenant-specific configuration before the page reaches the browser:
// Injected by edge function into <head>
window.__TENANT_CONFIG__ = { id: "vocalstar", branding: { ... }, features: { ... } }
// CSS variables for dynamic theming
--tenant-primary-color: #ffd700;
--tenant-background-color: #1a1a2e;
The frontend's TenantProvider reads window.__TENANT_CONFIG__ on startup (no API round-trip needed for initial render). This approach works for both custom domains (vocalstar.nomadkaraoke.com) and the shared tenant portal domain.
Magic link emails: When a tenant user requests a magic link, the email uses the tenant-specific URL (e.g., https://vocalstar.nomadkaraoke.com/auth/verify?token=...) rather than the main app URL. The job_notification_service.py resolves the correct base URL from the tenant config's subdomain field.
Admin preview: Admins can preview any tenant's branding in the main app by appending ?preview_tenant=<tenant_id> to any URL. This bypasses subdomain detection and loads the specified tenant config, useful for verifying branding before DNS cutover.
Tenant subdomains are automatic (v0.246.0): /admin/tenants create provisions {id}.nomadkaraoke.com via the Cloudflare API (services/tenant_domain_service.py, secret cloudflare-tenant-domains-token = account-owned token scoped to Pages Read/Write + DNS Read/Write on the nomadkaraoke.com zone only), and delete removes it. Cloudflare Pages has no wildcard custom domains, so each tenant gets: (1) a Pages custom domain on karaoke-gen-tenant, (2) a proxied CNAME <id> → karaoke-gen-tenant.pages.dev. Existing non-tenant DNS records are never taken over or deleted. Token blast radius (accepted risk): Cloudflare can't scope Pages permissions to one project, so this token can modify any Pages project in the account (incl. the consumer karaoke-gen site) — it lives only in Secret Manager, readable by the backend SA; rotate it if the backend is ever compromised. DNS access is limited to the nomadkaraoke.com zone. No GCS CORS change is needed (bucket CORS allows any origin — safe because the bucket is private and signed URLs are the auth). Tenants created before this can be backfilled with Manage → Set up domain (POST /api/admin/tenants/{id}/domain). The subdomain is immutable (derived from the id). After a delete, other backend instances may still serve the tenant's cached config for up to 5 minutes (per-instance TenantService TTL).
Theme/tenant edits take effect immediately: while the storage bucket was publicly readable (until 2026-09-28), GCS served objects with the default Cache-Control: public, max-age=3600 and even the backend's own reads could return a stale copy for up to an hour after an overwrite. The bucket is now private (public_access_prevention=enforced), and as defence in depth StorageService.upload_json writes no-store, theme asset/logo uploads pass cache_control=no-store, and download_json resolves the live generation before downloading (bypassing any cached copy of objects written before this).
Tech Stack
- Backend: FastAPI, Python 3.12, Cloud Run
- Frontend: Next.js, TypeScript, Cloudflare Pages
- Database: Firestore
- Storage: Google Cloud Storage
- Secrets: Google Secret Manager
- Task Queues: Cloud Tasks (idle reminders, YouTube upload queue)
- Scheduling: Cloud Scheduler (hourly YouTube queue processing)
- Email: Postmark
- IaC: Pulumi
- CI/CD: GitHub Actions