Phase 1 (omnizart-music MVP): - config.py: add omnizart-music/vocal/vocal-contour to TRANSCRIBE_MODES with engine + description fields; add OMNIZART_ENABLED and OMNIZART_SIDECAR_URL env-var config constants - transcriber_service.py: dispatch run_transcription() to basic-pitch or omnizart based on mode prefix; extract _run_basic_pitch() helper - routers/transcribe.py: add 503 guard for omnizart modes when disabled; GET /modes now returns available flag per mode Phase 2 (all three Omnizart modes): - backend/services/omnizart_service.py (new): WAV conversion via ffmpeg with 10 s minimum padding; subprocess invocation with cancellation polling; vocal mode CWD quirk handled by setting cwd=tmp; all three modes supported Phase 3 (frontend polish + sidecar): - types.ts: extend ModeOption with engine, available, description fields - ProjectDetail.tsx: replace static TRANSCRIBE_MODES with live useQuery call to GET /api/transcribe/modes; add TranscribeModeSelect component with <optgroup> grouping by engine, disabled options for unavailable modes, and title= tooltips from description field; handleTranscribeAll auto-selects omnizart-music when available, falls back to polyphonic - omnizart-sidecar/app.py + Dockerfile (new): minimal FastAPI HTTP wrapper around the Omnizart CLI; single POST /transcribe endpoint; GET /health - stemengine.yml: add omnizart-sidecar service (profiles: [omnizart]); add OMNIZART_ENABLED and OMNIZART_SIDECAR_URL env vars to stemengine service
StemEngine - Karaoke Anything?
This is a vibe-coded project based on the work of people who actually know what they're doing. But now I can theoretically karaoke anything and so can you.
Features
- Stem Separation — 40+ models across 10 model families
- MIDI Transcription — basic-pitch with polyphonic and melody modes
- ABC Notation — convert MIDI to ABC notation for sheet music
- Lyrics Sync — forced-alignment to produce ELRC files with word-level timestamps and a live animated preview
- Project Management — persistent project tabs, per-stem reprocessing, stem renaming
- Mix Presets — Instrumental, Static (remove elements), Dynamic (per-stem volume)
- Subsonic Integration — browse and import from a Subsonic-compatible server
- Model Downloader — download model weights on-demand from the Settings UI
Separation Models
| Family | Models |
|---|---|
| Demucs | v4, v4 Fine-tuned, v3 Extra Quantized, v3 |
| BS-RoFormer | 4-stem, 5-stem (Guitar), 5-stem (Piano), 6-stem, Chorus, Anvuew, Karaoke Anvuew, SW 6-stem |
| MelBand RoFormer | 4-stem, Vocals, Vocals Anvuew, Dereverb, Dereverb Anvuew, Dereverb Room, Karaoke, Karaoke Gabox, Deux Becruily |
| MDX-Net | Vocals FT, Inst HQ 1, Inst HQ 2, Karaoke, Dereverb, Drum Sep |
| SCNet | XL (2-stem), XL IHF, Large (4-stem), Large Starrytong |
| MedleyVox | Base, 55/66/88/100 Epoch variants |
| Apollo | Vocals, EDM, EDM Big |
| Aspiration | Full, Light |
| Bandit v2 | Multi-Language, English |
| Ensemble | Vocal Best Quality, 4-Stem, 6-Stem Full |
Model weights are downloaded on first use and cached in /opt/.
Requirements
- Docker with the NVIDIA Container Toolkit (for GPU acceleration)
- NVIDIA GPU with CUDA 12.4 support
- Docker Compose
CPU-only mode is available via CPU_SEPARATION=true but separation will be significantly slower.
Quick Start
-
Copy
.env.exampleto.envand fill in the required values (see Configuration below). -
Build and start:
docker compose up -d --build
- Open
http://localhost:8000in your browser.
Configuration
All configuration is via environment variables. The docker-compose.yml accepts these via a .env file in the same directory.
Required
| Variable | Description |
|---|---|
SEFOLDER |
Host path for persistent runtime data (uploads, jobs, DB, LRC sync) |
ENGINEFOLDER |
Host path for model weight caches |
PROJECTS |
Host path for separated project output |
SECRET_KEY |
Random hex key for encrypting Subsonic credentials at rest. Generate with: python3 -c "import secrets; print(secrets.token_hex(32))" |
Optional
| Variable | Default | Description |
|---|---|---|
CPU_SEPARATION |
false |
Set to true to force CPU inference |
MAX_UPLOAD_MB |
500 |
Maximum upload file size in MB |
TZ |
UTC |
Timezone |
NAVSERVER |
— | Subsonic server URL (can also be set in the UI) |
NAVUSER |
— | Subsonic username |
NAVPASS |
— | Subsonic password |
Model Directories
These are set automatically in docker-compose.yml and point inside the /opt/ volume. Override only if you need a custom layout.
BS_ROFORMER_MODEL_DIR, MEL_BAND_MODEL_DIR, MDXNET_MODEL_DIR, SCNET_MODEL_DIR, MEDLEYVOX_MODEL_DIR, APOLLO_MODEL_DIR, ASPIRATION_MODEL_DIR, BANDIT_MODEL_DIR
Project Folder Structure
Each separation job creates a folder under PROJECTS_DIR:
Artist - Title/
Artist - Title.mp3 # Original source audio (copy)
Artist - Title.elrc # ELRC lyrics with word-level timestamps (if aligned)
Artist - Title.lrc # Plain LRC (if aligned)
Stems/
vocals.mp3
drums.mp3
bass.mp3
other.mp3
Instrumental/ # Only when Instrumental preset selected
Artist - Title - Instrumental.mp3
Static/ # Only when Static preset selected
Artist - Title - No Vocals.mp3
Dynamic/ # Only when Dynamic preset selected
Artist - Title - Dynamic Mix.mp3
Tech Stack
- Frontend — React 18, TypeScript, Vite, TailwindCSS
- Backend — FastAPI, Python 3.11, SQLAlchemy
- Separation — Demucs, BS-RoFormer, MelBand RoFormer, MDX-Net (onnxruntime-gpu), SCNet, MedleyVox, Apollo, Aspiration, Bandit
- Transcription — basic-pitch (ONNX, CUDAExecutionProvider)
- Lyrics Sync — wav2vec2-based forced alignment via custom
lsyncmodule - Runtime — NVIDIA CUDA 12.4 + cuDNN 9, PyTorch 2.6 (cu124 wheels)
- Container —
nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04
GPU Notes
The container requires the NVIDIA Container Toolkit. On the host:
# Verify GPU is visible inside the container
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
PyTorch wheels are self-contained (cu124) and do not require a matching CUDA toolkit on the host — only the driver.
onnxruntime-gpu is used for MDX-Net and basic-pitch ONNX inference; it auto-selects CUDAExecutionProvider when a GPU is available.