Andrew Beveridge ☄️andClaude Opus 5.5 70bdd0e693 fix(alerts): start-review 409, per-cause crash patterns, one alert per job failure (v0.290.1) (#1243)
Section 1 of the 2026-11-03 #alerts triage handoff: a stale push on a
finished song got a 500, and the error monitor could stay silent on a
new crash.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980744448843886
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980746394996879
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980747905073224
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565697725280948294
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565803187334815775
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565818418072916110
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566225593354420226
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566256141074169987
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567207141994135654
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252495535841386
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252497620537415

## What changed

**1a. start-review:** a job not in review gets 409
(`jobs.jobNotInReviewState`) with no transition attempted. Losing the
auto-approver race is a 409 too. `correction-data` has the same race,
and its `except Exception` turned it into a 500; it now returns 409
(`except HTTPException: raise` added). Its status gate still returns
400, as before.

**1b. Crash patterns:** `Exception in ASGI application` and
`request_audit_error` become `Unhandled exception: <Type: message>`,
read from the entry's `exception` field. The handoff's suspicion is
confirmed in `upsert_pattern`: an existing `new` pattern returns
`is_new=False`, so a second, different crash stayed silent. Both log
lines of one crash now share one pattern. The path and method go in the
sample only.

I read the line after the last frame rather than the traceback's last
line. pydantic's last line is a docs link shared by every failure, and
group tracebacks end in a separator.

**1c. One alert per failure:**
- YouTube ids are normalized out of signatures, which also fixes the
false "first time" line in `ops_alerts`.
- `Job <id> failed` (both call sites in `job_manager`) logs WARNING,
since the status write sends the job-failed alert.
- `URL download failed:` is ignored in the monitor, not demoted, because
`file_upload.py` belongs to the section 4 PR.
- `[job:<id>] Download failed` logs WARNING for the expected
`DownloadError`/`FlacfetchServiceError` types. This wasn't named in the
handoff, but it is the same duplicate on the audio-search path.
Unexpected exceptions there stay ERROR with a traceback.

**1d. Suppressions and demotions:**
- Zalo's `zaloJSV2 is not defined`: dropped at `/api/client-errors`,
only with a Zalo UA.
- `python -c` experiments on `audio-separation-job`: ignored, using the
handoff's regex.
- Spotify 403 on `/lyrics`: WARNING with `response.text[:200]`. 401 on
`/lyrics` and 401/403 on `/search` stay ERROR.
- Dropbox token probe: WARNING. A rejected refresh used to log nothing,
so it now logs ERROR, as does having no refresh credentials.
- decide-search health aborts: this was a small change. `should_ignore`
takes an optional `request_url` and matches the exact path
`/api/search/health`.

**1e. Cold start:** `anyio._backends._asyncio` is added to
`_LAZY_ROUTE_IMPORTS`. The prod stack from 10-30 confirms the stall was
in that import. Lifespan also sends one `/api/health` request through
the app in a worker thread, bounded by the existing 30s timeout and
best-effort.

## Testing

- TDD: new tests failed first, then passed.
- Affected suites: 826 passed.
- `backend/tests` (excluding `backend/tests/emulator`): 7088 passed, 8
failed, 29 skipped. All 8 are in `test_emulator_integration.py`, and
that file also shows 8 failures at the base commit a8f9618a. I did not
compare the 8 test names one by one.
- `tests/unit` outside the listed files was not run.
`test_audio_fetcher.py` and `test_remote_cli.py` fail to collect in my
environment; I didn't change either file.
- Read-only replay of real prod entries through the new code: the 11-03
crash gave one specific pattern with `POST
/api/jobs/7f89f884/start-review` in the sample, the decide-search health
aborts and the experiment traceback were ignored.
- Local CodeRabbit review completed with 0 findings.

## Not verified

- Nothing is live, so the alert volume and the cold-start stall haven't
been measured. The warm request has only run in tests, not on a real
Cloud Run cold start.
- A request's first-use imports beyond the anyio backend aren't
individually checked.
- My test environment has no `flacfetch`, so its lazy imports only log a
warning here.
- Zalo: the trigger is one report (11-03).

## Known leftovers

- `validate_state_transition` still logs ERROR before raising, so the
auto-approver race (and the existing in_review race) still emits one
`Invalid state transition` ERROR even though the client now gets a clean
response. The stale-push case no longer reaches it. I left this because
other callers rely on that log.
- A crash that also logs its own ERROR first still gives two patterns,
the ERROR and `Unhandled exception`.
- The first bot block after deploy shows "first time" once, because the
normalized signature is new.
- Old `Exception in ASGI application` and `request_audit_error` patterns
stop recurring and will auto-resolve.
- If the section 4 PR rewords `URL download failed:`, the ignore entry
in `known_issues.py` needs the same wording.

## For Andrew (infra and cost, not changed here)

- Cloud Run startup probe and min-instances for `karaoke-backend`, if
cold-start stalls continue after this ships.
- decide-search's min-0 is why its health pings abort. The suppression
hides those, but real search paths will still alert on the same abort.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 23:04:04 +00:00
2025-12-22 17:40:42 -05:00

Karaoke Generator 🎶 🎥 🚀

PyPI - Version PyPI - Python Version Tests Test Coverage

Generate professional karaoke videos with instrumental audio and synchronized lyrics. Available as a local CLI (karaoke-gen) or cloud-based CLI (karaoke-gen-remote) that offloads processing to Google Cloud.

✨ Two Ways to Generate Karaoke

1. Local CLI (karaoke-gen)

Run all processing locally on your machine. Requires GPU for optimal audio separation performance.

karaoke-gen "ABBA" "Waterloo"

2. Remote CLI (karaoke-gen-remote)

Offload all processing to a cloud backend. No GPU required - just authenticate and submit jobs.

karaoke-gen-remote ./song.flac "ABBA" "Waterloo"

Both CLIs produce identical outputs: 4K karaoke videos, CDG+MP3 packages, audio stems, and more.


🎯 Features

Core Pipeline

  • Audio Separation: AI-powered vocal/instrumental separation using MDX and Demucs models
  • Lyrics Transcription: Word-level timestamps via AudioShake API
  • Lyrics Correction: Match transcription against online lyrics (Genius, Spotify, Musixmatch)
  • Human Review: Interactive UI for correcting lyrics before final render
  • Video Rendering: High-quality 4K karaoke videos with customizable styles
  • Multiple Outputs: MP4 (4K lossless/lossy, 720p), MKV, CDG+MP3, TXT+MP3

Distribution Features

  • YouTube Upload: Automatic upload to your YouTube channel
  • Dropbox Integration: Organize output in brand-coded folders
  • Google Drive: Upload to public share folders
  • Discord Notifications: Webhook notifications on completion

📦 Installation

pip install karaoke-gen

This installs both karaoke-gen (local) and karaoke-gen-remote (cloud) CLIs.

Requirements

  • Python 3.10-3.13
  • FFmpeg
  • For local processing: CUDA-capable GPU or Apple Silicon CPU recommended

Transcription Provider Setup

Transcription is required for creating karaoke videos with synchronized lyrics. The system needs word-level timing data to display lyrics in sync with the music.

Commercial service with high-quality transcription. Best for production use.

export AUDIOSHAKE_API_TOKEN="your_audioshake_token"

Get an API key at https://www.audioshake.ai/ - business only, at time of writing this.

Option 2: Local Whisper (No Cloud Required)

Run Whisper directly on your local machine using whisper-timestamped. Works on CPU, NVIDIA GPU (CUDA), or Apple Silicon.

# Install with local Whisper support
pip install "karaoke-gen[local-whisper]"

# Optional: Configure model size (tiny, base, small, medium, large)
export WHISPER_MODEL_SIZE="medium"

# Optional: Force specific device (cpu, cuda, mps)
export WHISPER_DEVICE="cpu"

Model Size Guide:

Model VRAM Speed Quality
tiny ~1GB Fast Lower
base ~1GB Fast Basic
small ~2GB Medium Good
medium ~5GB Slower Better
large ~10GB Slowest Best

CPU-Only Installation (no GPU required):

# Pre-install CPU-only PyTorch first
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install "karaoke-gen[local-whisper]"

Local Whisper runs automatically as a fallback when no cloud transcription services are configured.

Option 3: Whisper via RunPod

Cloud-based alternative using OpenAI's Whisper model on RunPod infrastructure.

export RUNPOD_API_KEY="your_runpod_key"
export WHISPER_RUNPOD_ID="your_whisper_endpoint_id"

Set up a Whisper endpoint at https://www.runpod.io/

Without Transcription (Instrumental Only)

If you don't need synchronized lyrics, use the --skip-lyrics flag:

karaoke-gen --skip-lyrics "Artist" "Title"

This creates an instrumental-only karaoke video without lyrics overlay.

Note: See lyrics_transcriber_temp/README.md for detailed transcription provider configuration options.


🖥️ Local CLI (karaoke-gen)

Basic Usage

# Generate from local audio file
karaoke-gen ./song.mp3 "Artist Name" "Song Title"

# Search and download audio automatically
karaoke-gen "Rick Astley" "Never Gonna Give You Up"

# Process from YouTube URL
karaoke-gen "https://www.youtube.com/watch?v=dQw4w9WgXcQ" "Rick Astley" "Never Gonna Give You Up"

Remote Audio Separation (Optional)

Offload just the GPU-intensive audio separation to Modal.com while keeping other processing local:

export AUDIO_SEPARATOR_API_URL="https://USERNAME--audio-separator-api.modal.run"
karaoke-gen "Artist" "Title"

Key Options

# Custom styling
karaoke-gen --style_params_json="./styles.json" "Artist" "Title"

# Generate CDG and TXT packages
karaoke-gen --enable_cdg --enable_txt "Artist" "Title"

# Skip video encoding (CDG/TXT only, faster)
karaoke-gen --no-video --enable_cdg "Artist" "Title"

# YouTube upload
karaoke-gen --enable_youtube_upload --youtube_description_file="./desc.txt" "Artist" "Title"

# Full production run
karaoke-gen \
  --style_params_json="./branding.json" \
  --enable_cdg \
  --enable_txt \
  --brand_prefix="BRAND" \
  --enable_youtube_upload \
  --youtube_description_file="./description.txt" \
  "Artist" "Title"

Full Options Reference

karaoke-gen --help

☁️ Remote CLI (karaoke-gen-remote)

The remote CLI submits jobs to a Google Cloud backend that handles all processing. You don't need a GPU or any audio processing libraries installed locally.

Setup

  1. Set the backend URL:

    export KARAOKE_GEN_URL="https://api.nomadkaraoke.com"  # Or your own backend
    
  2. Authenticate with Google Cloud:

    gcloud auth login
    

Basic Usage

# Submit a job
karaoke-gen-remote ./song.flac "ABBA" "Waterloo"

# The CLI will:
# 1. Upload your audio file
# 2. Monitor processing progress
# 3. Open lyrics review UI when ready
# 4. Prompt for instrumental selection
# 5. Download all outputs when complete

Job Management

# List all jobs
karaoke-gen-remote --list

# Resume monitoring an existing job
karaoke-gen-remote --resume abc12345

# Cancel a running job
karaoke-gen-remote --cancel abc12345

# Delete a job and its files
karaoke-gen-remote --delete abc12345

Full Production Run

karaoke-gen-remote \
  --style_params_json="./karaoke-styles.json" \
  --enable_cdg \
  --enable_txt \
  --brand_prefix=NOMAD \
  --enable_youtube_upload \
  --youtube_description_file="./youtube-description.txt" \
  ./song.flac "Artist" "Title"

Environment Variables

Variable Description Default
KARAOKE_GEN_URL Backend service URL Required
KARAOKE_GEN_AUTH_TOKEN Admin auth token (for protected endpoints) Optional
REVIEW_UI_URL Lyrics review UI URL https://gen.nomadkaraoke.com/lyrics/
POLL_INTERVAL Seconds between status polls 5

Note: The REVIEW_UI_URL defaults to the hosted lyrics review UI. For local development, set it to http://localhost:5173 if you're running the frontend dev server.

Authentication

The backend uses token-based authentication for admin operations (bulk delete, internal worker triggers). For basic job submission and monitoring, authentication is optional.

For admin access:

export KARAOKE_GEN_AUTH_TOKEN="your-admin-token"

The token must match one of the tokens configured in the backend's ADMIN_TOKENS environment variable.

Non-Interactive Mode

For automated/CI usage:

karaoke-gen-remote -y ./song.flac "Artist" "Title"

The -y flag auto-accepts default corrections and selects clean instrumental.


🎨 Style Configuration

Create a styles.json file to customize the karaoke video appearance:

{
  "intro": {
    "video_duration": 5,
    "background_image": "/path/to/title-background.png",
    "font": "/path/to/Font.ttf",
    "artist_color": "#ffdf6b",
    "title_color": "#ffffff"
  },
  "karaoke": {
    "background_image": "/path/to/karaoke-background.png",
    "font_path": "/path/to/Font.ttf"
  },
  "end": {
    "background_image": "/path/to/end-background.png"
  },
  "cdg": {
    "font_path": "/path/to/Font.ttf",
    "instrumental_background": "/path/to/cdg-background.png"
  }
}

When using karaoke-gen-remote, all referenced files are automatically uploaded with your job.


📤 Output Files

A completed job produces:

BRAND-1234 - Artist - Title/
├── Artist - Title (Final Karaoke Lossless 4k).mp4    # ProRes 4K
├── Artist - Title (Final Karaoke Lossless 4k).mkv    # FLAC audio 4K
├── Artist - Title (Final Karaoke Lossy 4k).mp4       # H.264 4K
├── Artist - Title (Final Karaoke Lossy 720p).mp4     # H.264 720p
├── Artist - Title (Final Karaoke CDG).zip            # CDG+MP3 package
├── Artist - Title (Final Karaoke TXT).zip            # TXT+MP3 package
├── Artist - Title (Karaoke).cdg                      # Individual CDG
├── Artist - Title (Karaoke).mp3                      # Karaoke audio
├── Artist - Title (Karaoke).lrc                      # LRC lyrics
├── Artist - Title (Karaoke).ass                      # ASS subtitles
├── Artist - Title (Title).mov                        # Title screen video
├── Artist - Title (End).mov                          # End screen video
├── Artist - Title (Instrumental...).flac             # Clean instrumental
├── Artist - Title (Instrumental +BV...).flac         # With backing vocals
└── stems/                                            # All audio stems
    ├── ...Vocals....flac
    ├── ...Bass....flac
    ├── ...Drums....flac
    └── ...

🏗️ Deploy Your Own Backend

The cloud backend runs on Google Cloud Platform using:

  • Cloud Run: Serverless API hosting
  • Firestore: Job state management
  • Cloud Storage: File uploads and outputs
  • Modal.com: GPU-accelerated audio separation
  • AudioShake: Lyrics transcription API

Prerequisites

  • Google Cloud account with billing enabled
  • Pulumi CLI
  • Modal.com account (for audio separation)
  • AudioShake API key

Infrastructure Setup

cd infrastructure

# Install dependencies
pip install -r requirements.txt

# Login to Pulumi
pulumi login

# Create a stack
pulumi stack init prod

# Configure GCP project
pulumi config set gcp:project your-project-id
pulumi config set gcp:region us-central1

# Deploy infrastructure
pulumi up

This creates:

  • Firestore database
  • Cloud Storage bucket
  • Artifact Registry
  • Service account with IAM roles
  • Secret Manager secrets (you add values)

Add Secret Values

# AudioShake API key
echo -n "your-audioshake-key" | gcloud secrets versions add audioshake-api-key --data-file=-

# Genius API key
echo -n "your-genius-key" | gcloud secrets versions add genius-api-key --data-file=-

# Modal API URL
echo -n "https://your-modal-url" | gcloud secrets versions add audio-separator-api-url --data-file=-

# YouTube OAuth credentials (JSON)
gcloud secrets versions add youtube-oauth-credentials --data-file=./youtube-creds.json

# Dropbox OAuth credentials (JSON)
gcloud secrets versions add dropbox-oauth-credentials --data-file=./dropbox-creds.json

# Google Drive service account (JSON)
gcloud secrets versions add gdrive-service-account --data-file=./gdrive-sa.json

Deploy Cloud Run

Deployments happen automatically via GitHub Actions CI when pushing to main. See .github/workflows/ci.yml for the full deployment workflow.

Point CLI to Your Backend

export KARAOKE_GEN_URL="https://your-backend.run.app"
karaoke-gen-remote ./song.flac "Artist" "Title"

🔌 Backend API Reference

The backend exposes a REST API for job management.

Job Submission

POST /api/jobs/upload

Submit a new karaoke generation job with audio file and options.

curl -X POST "https://api.example.com/api/jobs/upload" \
  -F "file=@song.flac" \
  -F "artist=ABBA" \
  -F "title=Waterloo" \
  -F "enable_cdg=true" \
  -F "enable_txt=true" \
  -F "brand_prefix=NOMAD" \
  -F "style_params=@styles.json" \
  -F "style_karaoke_background=@background.png"

Job Status

GET /api/jobs/{job_id}

Get job status and details.

curl "https://api.example.com/api/jobs/abc12345"

List Jobs

GET /api/jobs

List all jobs with optional status filter.

curl "https://api.example.com/api/jobs?status=complete&limit=10"

Cancel Job

POST /api/jobs/{job_id}/cancel

Cancel a running job.

curl -X POST "https://api.example.com/api/jobs/abc12345/cancel" \
  -H "Content-Type: application/json" \
  -d '{"reason": "User cancelled"}'

Delete Job

DELETE /api/jobs/{job_id}

Delete a job and its files.

curl -X DELETE "https://api.example.com/api/jobs/abc12345?delete_files=true"

Lyrics Review

GET /api/review/{job_id}/correction-data

Get correction data for lyrics review.

POST /api/review/{job_id}/complete

Submit corrected lyrics and trigger video rendering.

Instrumental Selection

GET /api/jobs/{job_id}/instrumental-options

Get available instrumental options.

POST /api/jobs/{job_id}/select-instrumental

Submit instrumental selection (clean or with_backing).

curl -X POST "https://api.example.com/api/jobs/abc12345/select-instrumental" \
  -H "Content-Type: application/json" \
  -d '{"selection": "clean"}'

Download Files

GET /api/jobs/{job_id}/download-urls

Get download URLs for all output files.

GET /api/jobs/{job_id}/download/{category}/{file_key}

Stream download a specific file.

Health Check

GET /api/health

Check backend health status.


🔧 Troubleshooting

"No suitable files found for processing"

This error occurs during the finalisation step when the (With Vocals).mkv file is missing. This file is created during lyrics transcription.

Most common cause: No transcription provider configured.

Quick fix:

  1. Check if transcription providers are configured:

    echo $AUDIOSHAKE_API_TOKEN
    echo $RUNPOD_API_KEY
    
  2. If both are empty, set up a provider (see Transcription Provider Setup)

  3. Or use --skip-lyrics for instrumental-only karaoke:

    karaoke-gen --skip-lyrics "Artist" "Title"
    

Other causes:

  • Invalid API credentials - verify your tokens are correct and active
  • API service unavailable - check service status pages
  • Network connectivity issues - ensure you can reach the API endpoints
  • Transcription timeout - try again or use a different provider

Transcription Fails Silently

If karaoke-gen runs without errors but produces no synchronized lyrics:

  1. Check logs - Run with --log_level debug for detailed output:

    karaoke-gen --log_level debug "Artist" "Title"
    
  2. Verify environment variables - Ensure API tokens are exported in your shell:

    # Check if set
    printenv | grep -E "(AUDIOSHAKE|RUNPOD|WHISPER)"
    
    # Set in current session
    export AUDIOSHAKE_API_TOKEN="your_token"
    
  3. Test API connectivity - Verify you can reach the transcription service

"No lyrics found from any source"

This warning means no reference lyrics were fetched from online sources (Genius, Spotify, Musixmatch). The transcription will still work, but auto-correction may be less accurate.

To fix:

  • Set GENIUS_API_TOKEN for Genius lyrics
  • Set SPOTIFY_COOKIE_SP_DC for Spotify lyrics
  • Set RAPIDAPI_KEY for Musixmatch lyrics
  • Or provide lyrics manually with --lyrics_file /path/to/lyrics.txt

Video Quality Issues

If the output video has quality problems:

  • Ensure FFmpeg is properly installed: ffmpeg -version
  • Check available codecs: ffmpeg -codecs
  • For 4K output, ensure sufficient disk space (10GB+ per track)

Local Whisper Issues

GPU Out of Memory

If you get CUDA out of memory errors:

# Use a smaller model
export WHISPER_MODEL_SIZE="small"  # or "tiny"

# Or force CPU mode
export WHISPER_DEVICE="cpu"

Slow Transcription on CPU

CPU transcription is significantly slower than GPU. For faster processing:

  • Use a smaller model (tiny or base)
  • Consider using cloud transcription (AudioShake or RunPod)
  • On Apple Silicon, the small model offers good speed/quality balance

Model Download Issues

Whisper models are downloaded on first use (~1-3GB depending on size). If downloads fail:

  • Check your internet connection
  • Set a custom cache directory: export WHISPER_CACHE_DIR="/path/with/space"
  • Models are cached in ~/.cache/whisper/ by default

whisper-timestamped Not Found

If you get "whisper-timestamped is not installed":

pip install "karaoke-gen[local-whisper]"
# Or install directly:
pip install whisper-timestamped

Disabling Local Whisper

If you want to disable local Whisper (e.g., to force cloud transcription):

export ENABLE_LOCAL_WHISPER="false"

🧪 Development

Running Tests

# Run all tests
pytest tests/ backend/tests/ -v

# Run only unit tests
pytest tests/unit/ -v

# Run with coverage
pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing

Project Structure

karaoke-gen/
├── karaoke_gen/           # Core CLI package
│   ├── utils/
│   │   ├── gen_cli.py     # Local CLI (karaoke-gen)
│   │   └── remote_cli.py  # Remote CLI (karaoke-gen-remote)
│   ├── karaoke_finalise/  # Video encoding, packaging, distribution
│   └── style_loader.py    # Unified style configuration
├── backend/               # Cloud backend (FastAPI)
│   ├── api/routes/        # API endpoints
│   ├── workers/           # Background processing workers
│   └── services/          # Business logic services
├── infrastructure/        # Pulumi IaC for GCP
├── docs/                  # Documentation
└── tests/                 # Test suite

📄 License

MIT


🤝 Contributing

Contributions are welcome! See CONTRIBUTING.md — the whole backend runs locally against emulators with no Google Cloud credentials or paid API keys required.

Languages
Python 74.7%
TypeScript 22.1%
HTML 1.4%
Shell 1.2%
JavaScript 0.4%