Andrew Beveridge ☄️andClaude Opus 5.5 c831bd5e89 feat(uploads): line up mismatched user instrumentals + tenant bulk upload modal (v0.251.0) (#1087)
## Summary
A Randy Vild tenant bulk upload lost 3 of 10 tracks to the 0.5s
mix/instrumental duration check, and the bulk UI showed only
"Submitting…" while uploading. All three rejected tracks were valid
edits of the same song:

| Track | Difference | Now |
|---|---|---|
| Keep Smiling | instrumental has 2.97s extra lead-in | start trimmed →
matches mix exactly |
| Simulation | instrumental 6.5s longer outro | cut at mix end with 2s
fade |
| Faithful and True | instrumental stops 4.45s early | end padded with
silence |

## Changes
**Backend: conform instead of cancel**
(`backend/services/instrumental_conform.py`, `uploads-complete`)
- On a >0.5s mismatch, the start offset is found by FFT
cross-correlation (reusing `derived_vocals.find_offset`) at **25% and
75%** of the song, searching ±15s. The start is then trimmed or padded,
and the end is faded and cut or padded to the mix length. All of this is
one ffmpeg pass on the original file, keeping its sample rate and bit
depth, and written as FLAC.
- Stored at `uploads/{job}/conformed/existing_instrumental.flac`, which
becomes `existing_instrumental_gcs_path`. The original upload is kept,
and details go in `state_data.instrumental_conformed`.
- Still rejected (job cancelled and refunded, 400):
- correlation < 0.3 (wrong file; real pairs score 0.78–0.90, wrong pairs
about 0.01);
- offsets at 25% and 75% differ by more than 20 ms (edited differently
mid-song).
- Conforming runs fully off the event loop, with at most 2 at a time per
instance. GCS or ffmpeg failures return a retryable 503 instead of
cancelling the job.
- The orchestrator now stages the recorded
`existing_instrumental_gcs_path` for the GCE encoder, so the conformed
copy is what gets rendered.

**Frontend**
- Tenant bulk shows `UploadProgressModal` for the whole batch: total
bytes, speed and ETA, tracks submitted and failed, and an offline or
retrying notice. The modal gained optional `title`, `detail` and
`notice` props and uses the tenant's primary colour.
- Bulk retry bug:
- **Before:** a server-cancelled job kept its ID. The retry's 400 ("not
pending") was read as "already processing", so the row showed
**Submitted** for a dead job.
- **Now:** any 400 checks the job's status. A processing job counts as
done. A cancelled or failed job is dropped, so the retry creates a fresh
job. Otherwise the row keeps the job for retry, so no duplicate is
created.
- The single-track tenant form and the own-instrumental field now check
only the size cap in the browser; the server handles length differences.

## Testing
- Backend: 5086 passed (new: synthetic lead-in, long/short outro, late
start, wrong-file and mid-song-edit cases; route and helper guards;
orchestrator staging). Frontend jest: 1459 passed (new: batch modal
progress and offline notice, failed rows, retry and no-duplicate cases).
- Ran `conform_instrumental` on Randy's real files:
  - offsets at 25% and 75% agree to the sample;
  - output is exactly the mix length at 24-bit / 44.1 kHz;
  - after conforming, each file lines up with its mix at 0 offset;
  - a wrong pairing is rejected (correlation 0.02).
- Real frontend (`next dev`, randy-vild tenant, mocked API, slowed
upload server), 8 tracks / 192 MB: the modal showed progress, track
counts, speed and ETA, then closed onto the "All tracks submitted"
screen.
- Review: an independent reviewer agent found 1 must-fix and 4
should-fix issues, all addressed in the last commit. The CodeRabbit CLI
has no seat here, so this PR is left open to the CodeRabbit bot.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Uploaded instrumentals with different durations can now be aligned and
adjusted to match the mix when the audio matches.
* Bulk uploads now display overall progress, completed and failed track
counts, connection updates, and retry guidance.
* Failed bulk-upload tracks can be retried, with clearer handling for
submissions already in progress.

* **Bug Fixes**
* Instrumental duration checks now occur during upload processing
instead of blocking the upload in the browser. Files that cannot be
matched are reported as a mismatch.

* **Documentation**
* Updated upload instructions and related messages across supported
languages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 23:42:08 -04:00
2025-12-22 17:40:42 -05:00

Karaoke Generator 🎶 🎥 🚀

PyPI - Version PyPI - Python Version Tests Test Coverage

Generate professional karaoke videos with instrumental audio and synchronized lyrics. Available as a local CLI (karaoke-gen) or cloud-based CLI (karaoke-gen-remote) that offloads processing to Google Cloud.

✨ Two Ways to Generate Karaoke

1. Local CLI (karaoke-gen)

Run all processing locally on your machine. Requires GPU for optimal audio separation performance.

karaoke-gen "ABBA" "Waterloo"

2. Remote CLI (karaoke-gen-remote)

Offload all processing to a cloud backend. No GPU required - just authenticate and submit jobs.

karaoke-gen-remote ./song.flac "ABBA" "Waterloo"

Both CLIs produce identical outputs: 4K karaoke videos, CDG+MP3 packages, audio stems, and more.


🎯 Features

Core Pipeline

  • Audio Separation: AI-powered vocal/instrumental separation using MDX and Demucs models
  • Lyrics Transcription: Word-level timestamps via AudioShake API
  • Lyrics Correction: Match transcription against online lyrics (Genius, Spotify, Musixmatch)
  • Human Review: Interactive UI for correcting lyrics before final render
  • Video Rendering: High-quality 4K karaoke videos with customizable styles
  • Multiple Outputs: MP4 (4K lossless/lossy, 720p), MKV, CDG+MP3, TXT+MP3

Distribution Features

  • YouTube Upload: Automatic upload to your YouTube channel
  • Dropbox Integration: Organize output in brand-coded folders
  • Google Drive: Upload to public share folders
  • Discord Notifications: Webhook notifications on completion

📦 Installation

pip install karaoke-gen

This installs both karaoke-gen (local) and karaoke-gen-remote (cloud) CLIs.

Requirements

  • Python 3.10-3.13
  • FFmpeg
  • For local processing: CUDA-capable GPU or Apple Silicon CPU recommended

Transcription Provider Setup

Transcription is required for creating karaoke videos with synchronized lyrics. The system needs word-level timing data to display lyrics in sync with the music.

Commercial service with high-quality transcription. Best for production use.

export AUDIOSHAKE_API_TOKEN="your_audioshake_token"

Get an API key at https://www.audioshake.ai/ - business only, at time of writing this.

Option 2: Local Whisper (No Cloud Required)

Run Whisper directly on your local machine using whisper-timestamped. Works on CPU, NVIDIA GPU (CUDA), or Apple Silicon.

# Install with local Whisper support
pip install "karaoke-gen[local-whisper]"

# Optional: Configure model size (tiny, base, small, medium, large)
export WHISPER_MODEL_SIZE="medium"

# Optional: Force specific device (cpu, cuda, mps)
export WHISPER_DEVICE="cpu"

Model Size Guide:

Model VRAM Speed Quality
tiny ~1GB Fast Lower
base ~1GB Fast Basic
small ~2GB Medium Good
medium ~5GB Slower Better
large ~10GB Slowest Best

CPU-Only Installation (no GPU required):

# Pre-install CPU-only PyTorch first
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install "karaoke-gen[local-whisper]"

Local Whisper runs automatically as a fallback when no cloud transcription services are configured.

Option 3: Whisper via RunPod

Cloud-based alternative using OpenAI's Whisper model on RunPod infrastructure.

export RUNPOD_API_KEY="your_runpod_key"
export WHISPER_RUNPOD_ID="your_whisper_endpoint_id"

Set up a Whisper endpoint at https://www.runpod.io/

Without Transcription (Instrumental Only)

If you don't need synchronized lyrics, use the --skip-lyrics flag:

karaoke-gen --skip-lyrics "Artist" "Title"

This creates an instrumental-only karaoke video without lyrics overlay.

Note: See lyrics_transcriber_temp/README.md for detailed transcription provider configuration options.


🖥️ Local CLI (karaoke-gen)

Basic Usage

# Generate from local audio file
karaoke-gen ./song.mp3 "Artist Name" "Song Title"

# Search and download audio automatically
karaoke-gen "Rick Astley" "Never Gonna Give You Up"

# Process from YouTube URL
karaoke-gen "https://www.youtube.com/watch?v=dQw4w9WgXcQ" "Rick Astley" "Never Gonna Give You Up"

Remote Audio Separation (Optional)

Offload just the GPU-intensive audio separation to Modal.com while keeping other processing local:

export AUDIO_SEPARATOR_API_URL="https://USERNAME--audio-separator-api.modal.run"
karaoke-gen "Artist" "Title"

Key Options

# Custom styling
karaoke-gen --style_params_json="./styles.json" "Artist" "Title"

# Generate CDG and TXT packages
karaoke-gen --enable_cdg --enable_txt "Artist" "Title"

# Skip video encoding (CDG/TXT only, faster)
karaoke-gen --no-video --enable_cdg "Artist" "Title"

# YouTube upload
karaoke-gen --enable_youtube_upload --youtube_description_file="./desc.txt" "Artist" "Title"

# Full production run
karaoke-gen \
  --style_params_json="./branding.json" \
  --enable_cdg \
  --enable_txt \
  --brand_prefix="BRAND" \
  --enable_youtube_upload \
  --youtube_description_file="./description.txt" \
  "Artist" "Title"

Full Options Reference

karaoke-gen --help

☁️ Remote CLI (karaoke-gen-remote)

The remote CLI submits jobs to a Google Cloud backend that handles all processing. You don't need a GPU or any audio processing libraries installed locally.

Setup

  1. Set the backend URL:

    export KARAOKE_GEN_URL="https://api.nomadkaraoke.com"  # Or your own backend
    
  2. Authenticate with Google Cloud:

    gcloud auth login
    

Basic Usage

# Submit a job
karaoke-gen-remote ./song.flac "ABBA" "Waterloo"

# The CLI will:
# 1. Upload your audio file
# 2. Monitor processing progress
# 3. Open lyrics review UI when ready
# 4. Prompt for instrumental selection
# 5. Download all outputs when complete

Job Management

# List all jobs
karaoke-gen-remote --list

# Resume monitoring an existing job
karaoke-gen-remote --resume abc12345

# Cancel a running job
karaoke-gen-remote --cancel abc12345

# Delete a job and its files
karaoke-gen-remote --delete abc12345

Full Production Run

karaoke-gen-remote \
  --style_params_json="./karaoke-styles.json" \
  --enable_cdg \
  --enable_txt \
  --brand_prefix=NOMAD \
  --enable_youtube_upload \
  --youtube_description_file="./youtube-description.txt" \
  ./song.flac "Artist" "Title"

Environment Variables

Variable Description Default
KARAOKE_GEN_URL Backend service URL Required
KARAOKE_GEN_AUTH_TOKEN Admin auth token (for protected endpoints) Optional
REVIEW_UI_URL Lyrics review UI URL https://gen.nomadkaraoke.com/lyrics/
POLL_INTERVAL Seconds between status polls 5

Note: The REVIEW_UI_URL defaults to the hosted lyrics review UI. For local development, set it to http://localhost:5173 if you're running the frontend dev server.

Authentication

The backend uses token-based authentication for admin operations (bulk delete, internal worker triggers). For basic job submission and monitoring, authentication is optional.

For admin access:

export KARAOKE_GEN_AUTH_TOKEN="your-admin-token"

The token must match one of the tokens configured in the backend's ADMIN_TOKENS environment variable.

Non-Interactive Mode

For automated/CI usage:

karaoke-gen-remote -y ./song.flac "Artist" "Title"

The -y flag auto-accepts default corrections and selects clean instrumental.


🎨 Style Configuration

Create a styles.json file to customize the karaoke video appearance:

{
  "intro": {
    "video_duration": 5,
    "background_image": "/path/to/title-background.png",
    "font": "/path/to/Font.ttf",
    "artist_color": "#ffdf6b",
    "title_color": "#ffffff"
  },
  "karaoke": {
    "background_image": "/path/to/karaoke-background.png",
    "font_path": "/path/to/Font.ttf"
  },
  "end": {
    "background_image": "/path/to/end-background.png"
  },
  "cdg": {
    "font_path": "/path/to/Font.ttf",
    "instrumental_background": "/path/to/cdg-background.png"
  }
}

When using karaoke-gen-remote, all referenced files are automatically uploaded with your job.


📤 Output Files

A completed job produces:

BRAND-1234 - Artist - Title/
├── Artist - Title (Final Karaoke Lossless 4k).mp4    # ProRes 4K
├── Artist - Title (Final Karaoke Lossless 4k).mkv    # FLAC audio 4K
├── Artist - Title (Final Karaoke Lossy 4k).mp4       # H.264 4K
├── Artist - Title (Final Karaoke Lossy 720p).mp4     # H.264 720p
├── Artist - Title (Final Karaoke CDG).zip            # CDG+MP3 package
├── Artist - Title (Final Karaoke TXT).zip            # TXT+MP3 package
├── Artist - Title (Karaoke).cdg                      # Individual CDG
├── Artist - Title (Karaoke).mp3                      # Karaoke audio
├── Artist - Title (Karaoke).lrc                      # LRC lyrics
├── Artist - Title (Karaoke).ass                      # ASS subtitles
├── Artist - Title (Title).mov                        # Title screen video
├── Artist - Title (End).mov                          # End screen video
├── Artist - Title (Instrumental...).flac             # Clean instrumental
├── Artist - Title (Instrumental +BV...).flac         # With backing vocals
└── stems/                                            # All audio stems
    ├── ...Vocals....flac
    ├── ...Bass....flac
    ├── ...Drums....flac
    └── ...

🏗️ Deploy Your Own Backend

The cloud backend runs on Google Cloud Platform using:

  • Cloud Run: Serverless API hosting
  • Firestore: Job state management
  • Cloud Storage: File uploads and outputs
  • Modal.com: GPU-accelerated audio separation
  • AudioShake: Lyrics transcription API

Prerequisites

  • Google Cloud account with billing enabled
  • Pulumi CLI
  • Modal.com account (for audio separation)
  • AudioShake API key

Infrastructure Setup

cd infrastructure

# Install dependencies
pip install -r requirements.txt

# Login to Pulumi
pulumi login

# Create a stack
pulumi stack init prod

# Configure GCP project
pulumi config set gcp:project your-project-id
pulumi config set gcp:region us-central1

# Deploy infrastructure
pulumi up

This creates:

  • Firestore database
  • Cloud Storage bucket
  • Artifact Registry
  • Service account with IAM roles
  • Secret Manager secrets (you add values)

Add Secret Values

# AudioShake API key
echo -n "your-audioshake-key" | gcloud secrets versions add audioshake-api-key --data-file=-

# Genius API key
echo -n "your-genius-key" | gcloud secrets versions add genius-api-key --data-file=-

# Modal API URL
echo -n "https://your-modal-url" | gcloud secrets versions add audio-separator-api-url --data-file=-

# YouTube OAuth credentials (JSON)
gcloud secrets versions add youtube-oauth-credentials --data-file=./youtube-creds.json

# Dropbox OAuth credentials (JSON)
gcloud secrets versions add dropbox-oauth-credentials --data-file=./dropbox-creds.json

# Google Drive service account (JSON)
gcloud secrets versions add gdrive-service-account --data-file=./gdrive-sa.json

Deploy Cloud Run

Deployments happen automatically via GitHub Actions CI when pushing to main. See .github/workflows/ci.yml for the full deployment workflow.

Point CLI to Your Backend

export KARAOKE_GEN_URL="https://your-backend.run.app"
karaoke-gen-remote ./song.flac "Artist" "Title"

🔌 Backend API Reference

The backend exposes a REST API for job management.

Job Submission

POST /api/jobs/upload

Submit a new karaoke generation job with audio file and options.

curl -X POST "https://api.example.com/api/jobs/upload" \
  -F "file=@song.flac" \
  -F "artist=ABBA" \
  -F "title=Waterloo" \
  -F "enable_cdg=true" \
  -F "enable_txt=true" \
  -F "brand_prefix=NOMAD" \
  -F "style_params=@styles.json" \
  -F "style_karaoke_background=@background.png"

Job Status

GET /api/jobs/{job_id}

Get job status and details.

curl "https://api.example.com/api/jobs/abc12345"

List Jobs

GET /api/jobs

List all jobs with optional status filter.

curl "https://api.example.com/api/jobs?status=complete&limit=10"

Cancel Job

POST /api/jobs/{job_id}/cancel

Cancel a running job.

curl -X POST "https://api.example.com/api/jobs/abc12345/cancel" \
  -H "Content-Type: application/json" \
  -d '{"reason": "User cancelled"}'

Delete Job

DELETE /api/jobs/{job_id}

Delete a job and its files.

curl -X DELETE "https://api.example.com/api/jobs/abc12345?delete_files=true"

Lyrics Review

GET /api/review/{job_id}/correction-data

Get correction data for lyrics review.

POST /api/review/{job_id}/complete

Submit corrected lyrics and trigger video rendering.

Instrumental Selection

GET /api/jobs/{job_id}/instrumental-options

Get available instrumental options.

POST /api/jobs/{job_id}/select-instrumental

Submit instrumental selection (clean or with_backing).

curl -X POST "https://api.example.com/api/jobs/abc12345/select-instrumental" \
  -H "Content-Type: application/json" \
  -d '{"selection": "clean"}'

Download Files

GET /api/jobs/{job_id}/download-urls

Get download URLs for all output files.

GET /api/jobs/{job_id}/download/{category}/{file_key}

Stream download a specific file.

Health Check

GET /api/health

Check backend health status.


🔧 Troubleshooting

"No suitable files found for processing"

This error occurs during the finalisation step when the (With Vocals).mkv file is missing. This file is created during lyrics transcription.

Most common cause: No transcription provider configured.

Quick fix:

  1. Check if transcription providers are configured:

    echo $AUDIOSHAKE_API_TOKEN
    echo $RUNPOD_API_KEY
    
  2. If both are empty, set up a provider (see Transcription Provider Setup)

  3. Or use --skip-lyrics for instrumental-only karaoke:

    karaoke-gen --skip-lyrics "Artist" "Title"
    

Other causes:

  • Invalid API credentials - verify your tokens are correct and active
  • API service unavailable - check service status pages
  • Network connectivity issues - ensure you can reach the API endpoints
  • Transcription timeout - try again or use a different provider

Transcription Fails Silently

If karaoke-gen runs without errors but produces no synchronized lyrics:

  1. Check logs - Run with --log_level debug for detailed output:

    karaoke-gen --log_level debug "Artist" "Title"
    
  2. Verify environment variables - Ensure API tokens are exported in your shell:

    # Check if set
    printenv | grep -E "(AUDIOSHAKE|RUNPOD|WHISPER)"
    
    # Set in current session
    export AUDIOSHAKE_API_TOKEN="your_token"
    
  3. Test API connectivity - Verify you can reach the transcription service

"No lyrics found from any source"

This warning means no reference lyrics were fetched from online sources (Genius, Spotify, Musixmatch). The transcription will still work, but auto-correction may be less accurate.

To fix:

  • Set GENIUS_API_TOKEN for Genius lyrics
  • Set SPOTIFY_COOKIE_SP_DC for Spotify lyrics
  • Set RAPIDAPI_KEY for Musixmatch lyrics
  • Or provide lyrics manually with --lyrics_file /path/to/lyrics.txt

Video Quality Issues

If the output video has quality problems:

  • Ensure FFmpeg is properly installed: ffmpeg -version
  • Check available codecs: ffmpeg -codecs
  • For 4K output, ensure sufficient disk space (10GB+ per track)

Local Whisper Issues

GPU Out of Memory

If you get CUDA out of memory errors:

# Use a smaller model
export WHISPER_MODEL_SIZE="small"  # or "tiny"

# Or force CPU mode
export WHISPER_DEVICE="cpu"

Slow Transcription on CPU

CPU transcription is significantly slower than GPU. For faster processing:

  • Use a smaller model (tiny or base)
  • Consider using cloud transcription (AudioShake or RunPod)
  • On Apple Silicon, the small model offers good speed/quality balance

Model Download Issues

Whisper models are downloaded on first use (~1-3GB depending on size). If downloads fail:

  • Check your internet connection
  • Set a custom cache directory: export WHISPER_CACHE_DIR="/path/with/space"
  • Models are cached in ~/.cache/whisper/ by default

whisper-timestamped Not Found

If you get "whisper-timestamped is not installed":

pip install "karaoke-gen[local-whisper]"
# Or install directly:
pip install whisper-timestamped

Disabling Local Whisper

If you want to disable local Whisper (e.g., to force cloud transcription):

export ENABLE_LOCAL_WHISPER="false"

🧪 Development

Running Tests

# Run all tests
pytest tests/ backend/tests/ -v

# Run only unit tests
pytest tests/unit/ -v

# Run with coverage
pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing

Project Structure

karaoke-gen/
├── karaoke_gen/           # Core CLI package
│   ├── utils/
│   │   ├── gen_cli.py     # Local CLI (karaoke-gen)
│   │   └── remote_cli.py  # Remote CLI (karaoke-gen-remote)
│   ├── karaoke_finalise/  # Video encoding, packaging, distribution
│   └── style_loader.py    # Unified style configuration
├── backend/               # Cloud backend (FastAPI)
│   ├── api/routes/        # API endpoints
│   ├── workers/           # Background processing workers
│   └── services/          # Business logic services
├── infrastructure/        # Pulumi IaC for GCP
├── docs/                  # Documentation
└── tests/                 # Test suite

📄 License

MIT


🤝 Contributing

Contributions are welcome! See CONTRIBUTING.md — the whole backend runs locally against emulators with no Google Cloud credentials or paid API keys required.

Languages
Python 74.7%
TypeScript 22.1%
HTML 1.4%
Shell 1.2%
JavaScript 0.4%