Section 1 of the 2026-11-03 #alerts triage handoff: a stale push on a
finished song got a 500, and the error monitor could stay silent on a
new crash.
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980744448843886
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980746394996879
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980747905073224
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565697725280948294
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565803187334815775
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565818418072916110
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566225593354420226
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566256141074169987
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567207141994135654
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252495535841386
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252497620537415
## What changed
**1a. start-review:** a job not in review gets 409
(`jobs.jobNotInReviewState`) with no transition attempted. Losing the
auto-approver race is a 409 too. `correction-data` has the same race,
and its `except Exception` turned it into a 500; it now returns 409
(`except HTTPException: raise` added). Its status gate still returns
400, as before.
**1b. Crash patterns:** `Exception in ASGI application` and
`request_audit_error` become `Unhandled exception: <Type: message>`,
read from the entry's `exception` field. The handoff's suspicion is
confirmed in `upsert_pattern`: an existing `new` pattern returns
`is_new=False`, so a second, different crash stayed silent. Both log
lines of one crash now share one pattern. The path and method go in the
sample only.
I read the line after the last frame rather than the traceback's last
line. pydantic's last line is a docs link shared by every failure, and
group tracebacks end in a separator.
**1c. One alert per failure:**
- YouTube ids are normalized out of signatures, which also fixes the
false "first time" line in `ops_alerts`.
- `Job <id> failed` (both call sites in `job_manager`) logs WARNING,
since the status write sends the job-failed alert.
- `URL download failed:` is ignored in the monitor, not demoted, because
`file_upload.py` belongs to the section 4 PR.
- `[job:<id>] Download failed` logs WARNING for the expected
`DownloadError`/`FlacfetchServiceError` types. This wasn't named in the
handoff, but it is the same duplicate on the audio-search path.
Unexpected exceptions there stay ERROR with a traceback.
**1d. Suppressions and demotions:**
- Zalo's `zaloJSV2 is not defined`: dropped at `/api/client-errors`,
only with a Zalo UA.
- `python -c` experiments on `audio-separation-job`: ignored, using the
handoff's regex.
- Spotify 403 on `/lyrics`: WARNING with `response.text[:200]`. 401 on
`/lyrics` and 401/403 on `/search` stay ERROR.
- Dropbox token probe: WARNING. A rejected refresh used to log nothing,
so it now logs ERROR, as does having no refresh credentials.
- decide-search health aborts: this was a small change. `should_ignore`
takes an optional `request_url` and matches the exact path
`/api/search/health`.
**1e. Cold start:** `anyio._backends._asyncio` is added to
`_LAZY_ROUTE_IMPORTS`. The prod stack from 10-30 confirms the stall was
in that import. Lifespan also sends one `/api/health` request through
the app in a worker thread, bounded by the existing 30s timeout and
best-effort.
## Testing
- TDD: new tests failed first, then passed.
- Affected suites: 826 passed.
- `backend/tests` (excluding `backend/tests/emulator`): 7088 passed, 8
failed, 29 skipped. All 8 are in `test_emulator_integration.py`, and
that file also shows 8 failures at the base commit a8f9618a. I did not
compare the 8 test names one by one.
- `tests/unit` outside the listed files was not run.
`test_audio_fetcher.py` and `test_remote_cli.py` fail to collect in my
environment; I didn't change either file.
- Read-only replay of real prod entries through the new code: the 11-03
crash gave one specific pattern with `POST
/api/jobs/7f89f884/start-review` in the sample, the decide-search health
aborts and the experiment traceback were ignored.
- Local CodeRabbit review completed with 0 findings.
## Not verified
- Nothing is live, so the alert volume and the cold-start stall haven't
been measured. The warm request has only run in tests, not on a real
Cloud Run cold start.
- A request's first-use imports beyond the anyio backend aren't
individually checked.
- My test environment has no `flacfetch`, so its lazy imports only log a
warning here.
- Zalo: the trigger is one report (11-03).
## Known leftovers
- `validate_state_transition` still logs ERROR before raising, so the
auto-approver race (and the existing in_review race) still emits one
`Invalid state transition` ERROR even though the client now gets a clean
response. The stale-push case no longer reaches it. I left this because
other callers rely on that log.
- A crash that also logs its own ERROR first still gives two patterns,
the ERROR and `Unhandled exception`.
- The first bot block after deploy shows "first time" once, because the
normalized signature is new.
- Old `Exception in ASGI application` and `request_audit_error` patterns
stop recurring and will auto-resolve.
- If the section 4 PR rewords `URL download failed:`, the ignore entry
in `known_issues.py` needs the same wording.
## For Andrew (infra and cost, not changed here)
- Cloud Run startup probe and min-instances for `karaoke-backend`, if
cold-start stalls continue after this ships.
- decide-search's min-0 is why its health pings abort. The suppression
hides those, but real search paths will still alert on the same abort.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Karaoke Generator 🎶 🎥 🚀
Generate professional karaoke videos with instrumental audio and synchronized lyrics. Available as a local CLI (karaoke-gen) or cloud-based CLI (karaoke-gen-remote) that offloads processing to Google Cloud.
✨ Two Ways to Generate Karaoke
1. Local CLI (karaoke-gen)
Run all processing locally on your machine. Requires GPU for optimal audio separation performance.
karaoke-gen "ABBA" "Waterloo"
2. Remote CLI (karaoke-gen-remote)
Offload all processing to a cloud backend. No GPU required - just authenticate and submit jobs.
karaoke-gen-remote ./song.flac "ABBA" "Waterloo"
Both CLIs produce identical outputs: 4K karaoke videos, CDG+MP3 packages, audio stems, and more.
🎯 Features
Core Pipeline
- Audio Separation: AI-powered vocal/instrumental separation using MDX and Demucs models
- Lyrics Transcription: Word-level timestamps via AudioShake API
- Lyrics Correction: Match transcription against online lyrics (Genius, Spotify, Musixmatch)
- Human Review: Interactive UI for correcting lyrics before final render
- Video Rendering: High-quality 4K karaoke videos with customizable styles
- Multiple Outputs: MP4 (4K lossless/lossy, 720p), MKV, CDG+MP3, TXT+MP3
Distribution Features
- YouTube Upload: Automatic upload to your YouTube channel
- Dropbox Integration: Organize output in brand-coded folders
- Google Drive: Upload to public share folders
- Discord Notifications: Webhook notifications on completion
📦 Installation
pip install karaoke-gen
This installs both karaoke-gen (local) and karaoke-gen-remote (cloud) CLIs.
Requirements
- Python 3.10-3.13
- FFmpeg
- For local processing: CUDA-capable GPU or Apple Silicon CPU recommended
Transcription Provider Setup
Transcription is required for creating karaoke videos with synchronized lyrics. The system needs word-level timing data to display lyrics in sync with the music.
Option 1: AudioShake (Recommended)
Commercial service with high-quality transcription. Best for production use.
export AUDIOSHAKE_API_TOKEN="your_audioshake_token"
Get an API key at https://www.audioshake.ai/ - business only, at time of writing this.
Option 2: Local Whisper (No Cloud Required)
Run Whisper directly on your local machine using whisper-timestamped. Works on CPU, NVIDIA GPU (CUDA), or Apple Silicon.
# Install with local Whisper support
pip install "karaoke-gen[local-whisper]"
# Optional: Configure model size (tiny, base, small, medium, large)
export WHISPER_MODEL_SIZE="medium"
# Optional: Force specific device (cpu, cuda, mps)
export WHISPER_DEVICE="cpu"
Model Size Guide:
| Model | VRAM | Speed | Quality |
|---|---|---|---|
| tiny | ~1GB | Fast | Lower |
| base | ~1GB | Fast | Basic |
| small | ~2GB | Medium | Good |
| medium | ~5GB | Slower | Better |
| large | ~10GB | Slowest | Best |
CPU-Only Installation (no GPU required):
# Pre-install CPU-only PyTorch first
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install "karaoke-gen[local-whisper]"
Local Whisper runs automatically as a fallback when no cloud transcription services are configured.
Option 3: Whisper via RunPod
Cloud-based alternative using OpenAI's Whisper model on RunPod infrastructure.
export RUNPOD_API_KEY="your_runpod_key"
export WHISPER_RUNPOD_ID="your_whisper_endpoint_id"
Set up a Whisper endpoint at https://www.runpod.io/
Without Transcription (Instrumental Only)
If you don't need synchronized lyrics, use the --skip-lyrics flag:
karaoke-gen --skip-lyrics "Artist" "Title"
This creates an instrumental-only karaoke video without lyrics overlay.
Note: See
lyrics_transcriber_temp/README.mdfor detailed transcription provider configuration options.
🖥️ Local CLI (karaoke-gen)
Basic Usage
# Generate from local audio file
karaoke-gen ./song.mp3 "Artist Name" "Song Title"
# Search and download audio automatically
karaoke-gen "Rick Astley" "Never Gonna Give You Up"
# Process from YouTube URL
karaoke-gen "https://www.youtube.com/watch?v=dQw4w9WgXcQ" "Rick Astley" "Never Gonna Give You Up"
Remote Audio Separation (Optional)
Offload just the GPU-intensive audio separation to Modal.com while keeping other processing local:
export AUDIO_SEPARATOR_API_URL="https://USERNAME--audio-separator-api.modal.run"
karaoke-gen "Artist" "Title"
Key Options
# Custom styling
karaoke-gen --style_params_json="./styles.json" "Artist" "Title"
# Generate CDG and TXT packages
karaoke-gen --enable_cdg --enable_txt "Artist" "Title"
# Skip video encoding (CDG/TXT only, faster)
karaoke-gen --no-video --enable_cdg "Artist" "Title"
# YouTube upload
karaoke-gen --enable_youtube_upload --youtube_description_file="./desc.txt" "Artist" "Title"
# Full production run
karaoke-gen \
--style_params_json="./branding.json" \
--enable_cdg \
--enable_txt \
--brand_prefix="BRAND" \
--enable_youtube_upload \
--youtube_description_file="./description.txt" \
"Artist" "Title"
Full Options Reference
karaoke-gen --help
☁️ Remote CLI (karaoke-gen-remote)
The remote CLI submits jobs to a Google Cloud backend that handles all processing. You don't need a GPU or any audio processing libraries installed locally.
Setup
-
Set the backend URL:
export KARAOKE_GEN_URL="https://api.nomadkaraoke.com" # Or your own backend -
Authenticate with Google Cloud:
gcloud auth login
Basic Usage
# Submit a job
karaoke-gen-remote ./song.flac "ABBA" "Waterloo"
# The CLI will:
# 1. Upload your audio file
# 2. Monitor processing progress
# 3. Open lyrics review UI when ready
# 4. Prompt for instrumental selection
# 5. Download all outputs when complete
Job Management
# List all jobs
karaoke-gen-remote --list
# Resume monitoring an existing job
karaoke-gen-remote --resume abc12345
# Cancel a running job
karaoke-gen-remote --cancel abc12345
# Delete a job and its files
karaoke-gen-remote --delete abc12345
Full Production Run
karaoke-gen-remote \
--style_params_json="./karaoke-styles.json" \
--enable_cdg \
--enable_txt \
--brand_prefix=NOMAD \
--enable_youtube_upload \
--youtube_description_file="./youtube-description.txt" \
./song.flac "Artist" "Title"
Environment Variables
| Variable | Description | Default |
|---|---|---|
KARAOKE_GEN_URL |
Backend service URL | Required |
KARAOKE_GEN_AUTH_TOKEN |
Admin auth token (for protected endpoints) | Optional |
REVIEW_UI_URL |
Lyrics review UI URL | https://gen.nomadkaraoke.com/lyrics/ |
POLL_INTERVAL |
Seconds between status polls | 5 |
Note: The REVIEW_UI_URL defaults to the hosted lyrics review UI. For local development, set it to http://localhost:5173 if you're running the frontend dev server.
Authentication
The backend uses token-based authentication for admin operations (bulk delete, internal worker triggers). For basic job submission and monitoring, authentication is optional.
For admin access:
export KARAOKE_GEN_AUTH_TOKEN="your-admin-token"
The token must match one of the tokens configured in the backend's ADMIN_TOKENS environment variable.
Non-Interactive Mode
For automated/CI usage:
karaoke-gen-remote -y ./song.flac "Artist" "Title"
The -y flag auto-accepts default corrections and selects clean instrumental.
🎨 Style Configuration
Create a styles.json file to customize the karaoke video appearance:
{
"intro": {
"video_duration": 5,
"background_image": "/path/to/title-background.png",
"font": "/path/to/Font.ttf",
"artist_color": "#ffdf6b",
"title_color": "#ffffff"
},
"karaoke": {
"background_image": "/path/to/karaoke-background.png",
"font_path": "/path/to/Font.ttf"
},
"end": {
"background_image": "/path/to/end-background.png"
},
"cdg": {
"font_path": "/path/to/Font.ttf",
"instrumental_background": "/path/to/cdg-background.png"
}
}
When using karaoke-gen-remote, all referenced files are automatically uploaded with your job.
📤 Output Files
A completed job produces:
BRAND-1234 - Artist - Title/
├── Artist - Title (Final Karaoke Lossless 4k).mp4 # ProRes 4K
├── Artist - Title (Final Karaoke Lossless 4k).mkv # FLAC audio 4K
├── Artist - Title (Final Karaoke Lossy 4k).mp4 # H.264 4K
├── Artist - Title (Final Karaoke Lossy 720p).mp4 # H.264 720p
├── Artist - Title (Final Karaoke CDG).zip # CDG+MP3 package
├── Artist - Title (Final Karaoke TXT).zip # TXT+MP3 package
├── Artist - Title (Karaoke).cdg # Individual CDG
├── Artist - Title (Karaoke).mp3 # Karaoke audio
├── Artist - Title (Karaoke).lrc # LRC lyrics
├── Artist - Title (Karaoke).ass # ASS subtitles
├── Artist - Title (Title).mov # Title screen video
├── Artist - Title (End).mov # End screen video
├── Artist - Title (Instrumental...).flac # Clean instrumental
├── Artist - Title (Instrumental +BV...).flac # With backing vocals
└── stems/ # All audio stems
├── ...Vocals....flac
├── ...Bass....flac
├── ...Drums....flac
└── ...
🏗️ Deploy Your Own Backend
The cloud backend runs on Google Cloud Platform using:
- Cloud Run: Serverless API hosting
- Firestore: Job state management
- Cloud Storage: File uploads and outputs
- Modal.com: GPU-accelerated audio separation
- AudioShake: Lyrics transcription API
Prerequisites
- Google Cloud account with billing enabled
- Pulumi CLI
- Modal.com account (for audio separation)
- AudioShake API key
Infrastructure Setup
cd infrastructure
# Install dependencies
pip install -r requirements.txt
# Login to Pulumi
pulumi login
# Create a stack
pulumi stack init prod
# Configure GCP project
pulumi config set gcp:project your-project-id
pulumi config set gcp:region us-central1
# Deploy infrastructure
pulumi up
This creates:
- Firestore database
- Cloud Storage bucket
- Artifact Registry
- Service account with IAM roles
- Secret Manager secrets (you add values)
Add Secret Values
# AudioShake API key
echo -n "your-audioshake-key" | gcloud secrets versions add audioshake-api-key --data-file=-
# Genius API key
echo -n "your-genius-key" | gcloud secrets versions add genius-api-key --data-file=-
# Modal API URL
echo -n "https://your-modal-url" | gcloud secrets versions add audio-separator-api-url --data-file=-
# YouTube OAuth credentials (JSON)
gcloud secrets versions add youtube-oauth-credentials --data-file=./youtube-creds.json
# Dropbox OAuth credentials (JSON)
gcloud secrets versions add dropbox-oauth-credentials --data-file=./dropbox-creds.json
# Google Drive service account (JSON)
gcloud secrets versions add gdrive-service-account --data-file=./gdrive-sa.json
Deploy Cloud Run
Deployments happen automatically via GitHub Actions CI when pushing to main.
See .github/workflows/ci.yml for the full deployment workflow.
Point CLI to Your Backend
export KARAOKE_GEN_URL="https://your-backend.run.app"
karaoke-gen-remote ./song.flac "Artist" "Title"
🔌 Backend API Reference
The backend exposes a REST API for job management.
Job Submission
POST /api/jobs/upload
Submit a new karaoke generation job with audio file and options.
curl -X POST "https://api.example.com/api/jobs/upload" \
-F "file=@song.flac" \
-F "artist=ABBA" \
-F "title=Waterloo" \
-F "enable_cdg=true" \
-F "enable_txt=true" \
-F "brand_prefix=NOMAD" \
-F "style_params=@styles.json" \
-F "style_karaoke_background=@background.png"
Job Status
GET /api/jobs/{job_id}
Get job status and details.
curl "https://api.example.com/api/jobs/abc12345"
List Jobs
GET /api/jobs
List all jobs with optional status filter.
curl "https://api.example.com/api/jobs?status=complete&limit=10"
Cancel Job
POST /api/jobs/{job_id}/cancel
Cancel a running job.
curl -X POST "https://api.example.com/api/jobs/abc12345/cancel" \
-H "Content-Type: application/json" \
-d '{"reason": "User cancelled"}'
Delete Job
DELETE /api/jobs/{job_id}
Delete a job and its files.
curl -X DELETE "https://api.example.com/api/jobs/abc12345?delete_files=true"
Lyrics Review
GET /api/review/{job_id}/correction-data
Get correction data for lyrics review.
POST /api/review/{job_id}/complete
Submit corrected lyrics and trigger video rendering.
Instrumental Selection
GET /api/jobs/{job_id}/instrumental-options
Get available instrumental options.
POST /api/jobs/{job_id}/select-instrumental
Submit instrumental selection (clean or with_backing).
curl -X POST "https://api.example.com/api/jobs/abc12345/select-instrumental" \
-H "Content-Type: application/json" \
-d '{"selection": "clean"}'
Download Files
GET /api/jobs/{job_id}/download-urls
Get download URLs for all output files.
GET /api/jobs/{job_id}/download/{category}/{file_key}
Stream download a specific file.
Health Check
GET /api/health
Check backend health status.
🔧 Troubleshooting
"No suitable files found for processing"
This error occurs during the finalisation step when the (With Vocals).mkv file is missing. This file is created during lyrics transcription.
Most common cause: No transcription provider configured.
Quick fix:
-
Check if transcription providers are configured:
echo $AUDIOSHAKE_API_TOKEN echo $RUNPOD_API_KEY -
If both are empty, set up a provider (see Transcription Provider Setup)
-
Or use
--skip-lyricsfor instrumental-only karaoke:karaoke-gen --skip-lyrics "Artist" "Title"
Other causes:
- Invalid API credentials - verify your tokens are correct and active
- API service unavailable - check service status pages
- Network connectivity issues - ensure you can reach the API endpoints
- Transcription timeout - try again or use a different provider
Transcription Fails Silently
If karaoke-gen runs without errors but produces no synchronized lyrics:
-
Check logs - Run with
--log_level debugfor detailed output:karaoke-gen --log_level debug "Artist" "Title" -
Verify environment variables - Ensure API tokens are exported in your shell:
# Check if set printenv | grep -E "(AUDIOSHAKE|RUNPOD|WHISPER)" # Set in current session export AUDIOSHAKE_API_TOKEN="your_token" -
Test API connectivity - Verify you can reach the transcription service
"No lyrics found from any source"
This warning means no reference lyrics were fetched from online sources (Genius, Spotify, Musixmatch). The transcription will still work, but auto-correction may be less accurate.
To fix:
- Set
GENIUS_API_TOKENfor Genius lyrics - Set
SPOTIFY_COOKIE_SP_DCfor Spotify lyrics - Set
RAPIDAPI_KEYfor Musixmatch lyrics - Or provide lyrics manually with
--lyrics_file /path/to/lyrics.txt
Video Quality Issues
If the output video has quality problems:
- Ensure FFmpeg is properly installed:
ffmpeg -version - Check available codecs:
ffmpeg -codecs - For 4K output, ensure sufficient disk space (10GB+ per track)
Local Whisper Issues
GPU Out of Memory
If you get CUDA out of memory errors:
# Use a smaller model
export WHISPER_MODEL_SIZE="small" # or "tiny"
# Or force CPU mode
export WHISPER_DEVICE="cpu"
Slow Transcription on CPU
CPU transcription is significantly slower than GPU. For faster processing:
- Use a smaller model (
tinyorbase) - Consider using cloud transcription (AudioShake or RunPod)
- On Apple Silicon, the
smallmodel offers good speed/quality balance
Model Download Issues
Whisper models are downloaded on first use (~1-3GB depending on size). If downloads fail:
- Check your internet connection
- Set a custom cache directory:
export WHISPER_CACHE_DIR="/path/with/space" - Models are cached in
~/.cache/whisper/by default
whisper-timestamped Not Found
If you get "whisper-timestamped is not installed":
pip install "karaoke-gen[local-whisper]"
# Or install directly:
pip install whisper-timestamped
Disabling Local Whisper
If you want to disable local Whisper (e.g., to force cloud transcription):
export ENABLE_LOCAL_WHISPER="false"
🧪 Development
Running Tests
# Run all tests
pytest tests/ backend/tests/ -v
# Run only unit tests
pytest tests/unit/ -v
# Run with coverage
pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing
Project Structure
karaoke-gen/
├── karaoke_gen/ # Core CLI package
│ ├── utils/
│ │ ├── gen_cli.py # Local CLI (karaoke-gen)
│ │ └── remote_cli.py # Remote CLI (karaoke-gen-remote)
│ ├── karaoke_finalise/ # Video encoding, packaging, distribution
│ └── style_loader.py # Unified style configuration
├── backend/ # Cloud backend (FastAPI)
│ ├── api/routes/ # API endpoints
│ ├── workers/ # Background processing workers
│ └── services/ # Business logic services
├── infrastructure/ # Pulumi IaC for GCP
├── docs/ # Documentation
└── tests/ # Test suite
📄 License
MIT
🤝 Contributing
Contributions are welcome! See CONTRIBUTING.md — the whole backend runs locally against emulators with no Google Cloud credentials or paid API keys required.