## Summary Moves **every** Gemini call in gen off **Vertex AI** (billed to the `nomadkaraoke` GCP project, ~$85/mo) onto the **Gemini Developer API** using the `gemini-api-key` Secret Manager secret (an AI Studio key billed outside this project). Once deployed and verified, `aiplatform.googleapis.com` / `generativelanguage.googleapis.com` get disabled in nomadkaraoke. ### Key injection - `backend/services/gemini_client.get_genai_client(timeout_ms)` — key from `GEMINI_API_KEY`, else Secret Manager `gemini-api-key` via `settings.get_secret` (runtime SA already has project-level `secretAccessor`). No `project`/`location`. - Service: CI `--set-secrets ... GEMINI_API_KEY=gemini-api-key:latest`. - Jobs: Pulumi secret env on `lyrics-transcription-job`, `video-encoding-job` (renders + translations), `nomad-error-monitor` — **already applied** (targeted `pulumi up`, only that env added). Other jobs fall back to Secret Manager. - Local scripts (`frontend/scripts/translate.py`, `scripts/measure_autocorrect_cost.py`, `scripts/test_agentic_correction.py`): env, else `gcloud secrets versions access` (in-process only). - **Rotation:** add a new secret version; new instances / job executions pick it up. No code change. ### Degradation (quota / billing / invalid key; per-minute 429s stay retryable) | Caller | Behaviour | |---|---| | auto-correct | non-retryable 503 → compare mode drops the Gemini leg, keeps Opus | | match-judge / free-text / parse-titles / tenant bulk | existing fallbacks (no suggestion / blanks / regex) | | credit eval | `pending_review` (fail-closed, unchanged) | | lyrics translation | no retry; render proceeds without translation, status recorded | | custom lyrics | clean 503 "temporarily unavailable…" (+ 5-min timeout so it can't hang) | | error-monitor LLM | no retry; alerts sent without AI summary | | translate.py | exit 2 "Gemini quota/credit exhausted — top up AI Studio or rotate gemini-api-key" | `note_gemini_failure()` also sends one Discord ops alert per 6h (Firestore-throttled across instances via new `ops_alerts.send_throttled_ops_alert`). ### Infra - Pulumi: removes `roles/aiplatform.user` from `karaoke-backend` — **will be applied after this deploys and Vertex traffic is confirmed at zero**. - Agentic correction default model `gemini/gemini-3.8-flash` (`vertexai/` prefix still accepted, routed to the Developer API). - Version 0.261.1 → 0.262.0. Docs: CLAUDE.md, LESSONS-LEARNED, README, ARCHITECTURE, API, etc. ## Testing - New: `backend/tests/services/test_backend_gemini_client.py` (classifier, key resolution, no-Vertex client kwargs, alert throttle, per-caller degradation), `tests/unit/lyrics_transcriber/test_agentic_gemini_provider.py`, `frontend/scripts/test_gemini_client.py`, lyrics-translation quota test. - `pytest backend/tests --ignore=backend/tests/emulator` — all pass; `pytest tests/unit` — 2021 passed. - Verified `gemini-3.8-flash` is listed on the Developer API and answered a 5-token test call with the key. - `translate.py --dry-run` on `frontend/messages` and `backend/translations` OK. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Updates** * AI-powered lyric translation, custom lyrics, auto-correction, title parsing, match judging, and related features now use the Gemini Developer API. * AI features use the configured Gemini API key. Quota, billing, or key issues may show an unavailable message or use an existing fallback instead of repeated attempts. * **Documentation** * Updated product, API, setup, and recovery guidance to reflect Gemini API key configuration and usage. * Updated guidance for handling AI service availability and quota issues. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
10 KiB
Product Vision
What is Nomad Karaoke?
Nomad Karaoke is a karaoke video generation platform that creates professional-quality karaoke videos from any song in under 10 minutes. Founded by Andrew Beveridge and based in South Carolina, it combines state-of-the-art AI for audio separation and lyrics synchronization with human review to produce accurate, polished results.
Mission: Make any song singable by automating the complex, time-consuming process of creating karaoke videos.
The Problem
Creating high-quality karaoke is surprisingly difficult:
-
Vocal Removal: Separating vocals from instrumentals without artifacts requires sophisticated AI models and significant compute power.
-
Lyrics Accuracy: Getting the right lyrics, with correct spelling, formatting, and line breaks, is error-prone when scraping from various sources.
-
Timestamp Synchronization: Aligning each word or syllable to the exact moment in the audio is the hardest part. No existing tool does this reliably for all songs.
-
Multiple Formats: Professional karaoke needs multiple output formats (MP4, MKV, CDG) at various resolutions, with proper encoding.
-
Distribution: Uploading to YouTube with proper metadata, sharing to cloud storage, and notifying users adds complexity.
Existing solutions either produce poor quality (DIY tools), cost too much time ($50-100/track from manual services), or don't exist at all for many songs.
Target Audiences
1. Individual Karaoke Enthusiasts
People who want custom karaoke for songs not available in commercial libraries - for home parties, personal practice, or sharing with friends.
2. Professional KJs and Venues
Karaoke DJs ("KJs") and bars who need to fulfill on-demand song requests. When a patron asks for a song that isn't in the library, generating it in under 10 minutes means it can be ready within 1-2 performances.
3. Commercial Karaoke Producers (B2B)
Companies that produce karaoke tracks at scale for distribution. They have their own instrumentals, branding, and distribution channels - they need automated generation infrastructure that can handle 5,000+ tracks with custom theming.
Technical Approach
The system uses a multi-stage pipeline:
Input (audio + metadata)
│
├──────────────────────────────────────┐
│ │
v v
┌───────────────────┐ ┌───────────────────┐
│ Audio Separation │ │ Lyrics Processing │
│ (Cloud Run GPU L4)│ │ (AudioShake API) │
│ │ │ │
│ • Clean vocal │ │ • Fetch reference │
│ removal │ │ lyrics │
│ • Backing vocal │ │ • Transcribe with │
│ options │ │ timestamps │
└─────────┬─────────┘ │ • AI correction │
│ │ • HUMAN REVIEW │
│ └─────────┬─────────┘
│ │
└──────────────┬───────────────────┘
v
┌───────────────────┐
│ Video Generation │
│ • Title screen │
│ • Synced lyrics │
│ • End screen │
└─────────┬─────────┘
│
v
┌───────────────────┐
│ Output & Distro │
│ • Multiple formats│
│ • YouTube upload │
│ • Cloud storage │
└───────────────────┘
Why Human Review?
The hardest unsolved problem in karaoke generation is reliable lyrics transcription with accurate timestamps. Current approaches:
- Whisper (open source): Free, but inconsistent for sung audio. Works well ~60% of the time, poorly the rest.
- AudioShake (paid API): Proprietary lyrics transcription model specifically trained for music. Much more accurate/consistent than Whisper.
- Agentic AI Correction: LLM-based system that compares transcriptions against reference lyrics, fixing errors and improving sync. Costs tokens but catches many issues.
Even with the best AI, some songs have tricky timing, unusual pronunciations, or ambiguous lyrics. Human review catches these edge cases and ensures quality. The review UI lets users:
- Correct misspelled words
- Adjust word/line timing
- Fix line breaks and formatting
- Preview changes in real-time
Open Source Philosophy
All code is open source (MIT license) because:
- Community: Others can learn from, contribute to, and build upon the work
- Transparency: Users can see exactly how their data is processed
- Self-hosting option: Technical users can run their own instance
However, reliable fully-automated generation requires:
- AudioShake API: ~$0.50-1.00 per track for high-quality transcription
- LLM tokens: ~$0.50-1.00 for agentic correction workflow
- GPU compute: ~$0.10-0.30 for audio separation on Cloud Run GPU (L4)
- Infrastructure: Cloud Run, Firestore, GCS, etc.
Cost per track: ~$2-3 for fully-automated processing, making a paid service necessary to sustain development.
The open source CLI (karaoke-gen) works with local processing and free APIs for users willing to accept lower reliability and do more manual correction. The hosted service (gen.nomadkaraoke.com) provides the premium, reliable experience.
Business Model
Credit-Based Pricing
Users purchase credits to generate karaoke videos:
| Credits | Price | Per Track | Savings |
|---|---|---|---|
| 1 | $5 | $5.00 | - |
| 3 | $12 | $4.00 | 20% |
| 5 | $18 | $3.50 | 30% |
| 10 | $30 | $3.00 | 40% |
This is significantly cheaper than:
- Manual services: $10-50 per track
- Hiring a studio: $100+ per track
- DIY tools + your time: Hours of frustration
B2B / White-Label
Commercial producers pay for API access or custom integration to generate tracks at scale with their own branding, instrumentals, and distribution.
Current State (December 2025)
What Works
- Audio upload and separation (Cloud Run GPU)
- Lyrics transcription (AudioShake API)
- Agentic AI correction (Gemini 3 Flash via the Gemini Developer API)
- Human lyrics review (React UI)
- Preview video generation
- Instrumental selection (clean vs. with backing vocals)
- Multi-format encoding (4K, 720p, lossless, lossy)
- Token-based authentication
- Magic link authentication
- Payment flow (Stripe)
- YouTube upload integration
- Dropbox/Google Drive distribution
Known Limitations
- CDG format needs additional style configuration
- Very long audio (>10 min) may timeout
- Some edge cases in lyrics timing still need manual correction
Product Ecosystem
Active Products
- gen.nomadkaraoke.com - Web application for generating karaoke videos (includes landing page, pricing, and payment)
- karaoke-gen CLI - Local command-line tool for power users
- karaoke-gen-remote CLI - CLI that uses the cloud backend
- Nomad Karaoke YouTube Channel - Hosts generated videos for users
Future Products
-
KaraokeHunt (revival planned) - Song discovery app that analyzes streaming history to suggest songs you'd enjoy singing. Previously launched as a separate product; will be rebranded and integrated with the karaoke generator.
-
New nomadkaraoke.com Homepage - Replace the legacy static site with a modern landing page showcasing the generator (currently deployed from
nomadkaraoke/public-websiterepo).
Differentiators
vs. DIY Tools (Whisper + Demucs + manual editing)
- Quality: AudioShake's proprietary model beats open source transcription
- Speed: 10 minutes vs. hours of manual work
- Convenience: One-click generation with human review UI
vs. Manual Services ($10-50/track)
- Price: $3-5 per track
- Speed: Minutes instead of days
- Scale: Generate as many as you need, when you need them
vs. Other Automated Tools
- Accuracy: Human review catches AI errors
- Output quality: 4K video, multiple formats, proper encoding
- Distribution: YouTube upload, cloud storage, notifications built in
- Open source: Transparency and community contributions
Technical Stack
- Backend: FastAPI on Google Cloud Run
- Frontend: Next.js on Cloudflare Pages
- Database: Firestore
- Storage: Google Cloud Storage
- Secrets: Google Secret Manager
- Audio Separation: Cloud Run GPU (L4, us-east4)
- Lyrics Transcription: AudioShake API
- AI Correction: Gemini Developer API (Gemini 3 Flash)
- Payments: Stripe
- Email: Postmark
- IaC: Pulumi
- CI/CD: GitHub Actions (self-hosted runner on GCP)
Roadmap
Near-term
- Public launch of gen.nomadkaraoke.com
- New marketing homepage at nomadkaraoke.com
- B2B API for commercial producer integration
- Bulk generation features for high-volume clients
Medium-term
- KaraokeHunt revival and integration
- Mobile-optimized experience
- Real-time generation status notifications
- Additional output format support
Long-term
- Karaoke Eternal integration (community request)
- On-premises deployment option for venues
- Live karaoke event features
Links
- Web App: https://gen.nomadkaraoke.com
- API: https://api.nomadkaraoke.com
- GitHub: https://github.com/nomadkaraoke/karaoke-gen
- YouTube Channel: https://www.youtube.com/@nomadkaraoke
- Updates Signup: https://nomadkaraoke.kit.com/
- Contact: https://cal.com/beveradb
About the Founder
Andrew Beveridge is a software engineer based in South Carolina who runs a weekly karaoke night and built Nomad Karaoke to solve his own problem: wanting to sing songs that don't exist as karaoke. What started as a personal tool evolved into a platform that can help anyone create professional karaoke for any song.