translator 5c2152c57c feat: Omnizart MIDI transcription — phases 1-3
Phase 1 (omnizart-music MVP):
- config.py: add omnizart-music/vocal/vocal-contour to TRANSCRIBE_MODES with
  engine + description fields; add OMNIZART_ENABLED and OMNIZART_SIDECAR_URL
  env-var config constants
- transcriber_service.py: dispatch run_transcription() to basic-pitch or
  omnizart based on mode prefix; extract _run_basic_pitch() helper
- routers/transcribe.py: add 503 guard for omnizart modes when disabled;
  GET /modes now returns available flag per mode

Phase 2 (all three Omnizart modes):
- backend/services/omnizart_service.py (new): WAV conversion via ffmpeg with
  10 s minimum padding; subprocess invocation with cancellation polling;
  vocal mode CWD quirk handled by setting cwd=tmp; all three modes supported

Phase 3 (frontend polish + sidecar):
- types.ts: extend ModeOption with engine, available, description fields
- ProjectDetail.tsx: replace static TRANSCRIBE_MODES with live useQuery call
  to GET /api/transcribe/modes; add TranscribeModeSelect component with
  <optgroup> grouping by engine, disabled options for unavailable modes, and
  title= tooltips from description field; handleTranscribeAll auto-selects
  omnizart-music when available, falls back to polyphonic
- omnizart-sidecar/app.py + Dockerfile (new): minimal FastAPI HTTP wrapper
  around the Omnizart CLI; single POST /transcribe endpoint; GET /health
- stemengine.yml: add omnizart-sidecar service (profiles: [omnizart]); add
  OMNIZART_ENABLED and OMNIZART_SIDECAR_URL env vars to stemengine service
2026-03-14 13:00:36 +09:00
2026-03-09 17:55:38 +09:00
2026-03-03 14:52:19 +09:00
2026-03-03 14:52:19 +09:00
2026-03-03 14:52:19 +09:00
2026-03-13 17:55:07 +09:00
2026-03-13 15:32:26 +09:00

StemEngine - Karaoke Anything?

This is a vibe-coded project based on the work of people who actually know what they're doing. But now I can theoretically karaoke anything and so can you.


Features

  • Stem Separation — 40+ models across 10 model families
  • MIDI Transcription — basic-pitch with polyphonic and melody modes
  • ABC Notation — convert MIDI to ABC notation for sheet music
  • Lyrics Sync — forced-alignment to produce ELRC files with word-level timestamps and a live animated preview
  • Project Management — persistent project tabs, per-stem reprocessing, stem renaming
  • Mix Presets — Instrumental, Static (remove elements), Dynamic (per-stem volume)
  • Subsonic Integration — browse and import from a Subsonic-compatible server
  • Model Downloader — download model weights on-demand from the Settings UI

Separation Models

Family Models
Demucs v4, v4 Fine-tuned, v3 Extra Quantized, v3
BS-RoFormer 4-stem, 5-stem (Guitar), 5-stem (Piano), 6-stem, Chorus, Anvuew, Karaoke Anvuew, SW 6-stem
MelBand RoFormer 4-stem, Vocals, Vocals Anvuew, Dereverb, Dereverb Anvuew, Dereverb Room, Karaoke, Karaoke Gabox, Deux Becruily
MDX-Net Vocals FT, Inst HQ 1, Inst HQ 2, Karaoke, Dereverb, Drum Sep
SCNet XL (2-stem), XL IHF, Large (4-stem), Large Starrytong
MedleyVox Base, 55/66/88/100 Epoch variants
Apollo Vocals, EDM, EDM Big
Aspiration Full, Light
Bandit v2 Multi-Language, English
Ensemble Vocal Best Quality, 4-Stem, 6-Stem Full

Model weights are downloaded on first use and cached in /opt/.


Requirements

  • Docker with the NVIDIA Container Toolkit (for GPU acceleration)
  • NVIDIA GPU with CUDA 12.4 support
  • Docker Compose

CPU-only mode is available via CPU_SEPARATION=true but separation will be significantly slower.


Quick Start

  1. Copy .env.example to .env and fill in the required values (see Configuration below).

  2. Build and start:

docker compose up -d --build
  1. Open http://localhost:8000 in your browser.

Configuration

All configuration is via environment variables. The docker-compose.yml accepts these via a .env file in the same directory.

Required

Variable Description
SEFOLDER Host path for persistent runtime data (uploads, jobs, DB, LRC sync)
ENGINEFOLDER Host path for model weight caches
PROJECTS Host path for separated project output
SECRET_KEY Random hex key for encrypting Subsonic credentials at rest. Generate with: python3 -c "import secrets; print(secrets.token_hex(32))"

Optional

Variable Default Description
CPU_SEPARATION false Set to true to force CPU inference
MAX_UPLOAD_MB 500 Maximum upload file size in MB
TZ UTC Timezone
NAVSERVER — Subsonic server URL (can also be set in the UI)
NAVUSER — Subsonic username
NAVPASS — Subsonic password

Model Directories

These are set automatically in docker-compose.yml and point inside the /opt/ volume. Override only if you need a custom layout.

BS_ROFORMER_MODEL_DIR, MEL_BAND_MODEL_DIR, MDXNET_MODEL_DIR, SCNET_MODEL_DIR, MEDLEYVOX_MODEL_DIR, APOLLO_MODEL_DIR, ASPIRATION_MODEL_DIR, BANDIT_MODEL_DIR


Project Folder Structure

Each separation job creates a folder under PROJECTS_DIR:

Artist - Title/
  Artist - Title.mp3        # Original source audio (copy)
  Artist - Title.elrc       # ELRC lyrics with word-level timestamps (if aligned)
  Artist - Title.lrc        # Plain LRC (if aligned)
  Stems/
    vocals.mp3
    drums.mp3
    bass.mp3
    other.mp3
  Instrumental/             # Only when Instrumental preset selected
    Artist - Title - Instrumental.mp3
  Static/                   # Only when Static preset selected
    Artist - Title - No Vocals.mp3
  Dynamic/                  # Only when Dynamic preset selected
    Artist - Title - Dynamic Mix.mp3

Tech Stack

  • Frontend — React 18, TypeScript, Vite, TailwindCSS
  • Backend — FastAPI, Python 3.11, SQLAlchemy
  • Separation — Demucs, BS-RoFormer, MelBand RoFormer, MDX-Net (onnxruntime-gpu), SCNet, MedleyVox, Apollo, Aspiration, Bandit
  • Transcription — basic-pitch (ONNX, CUDAExecutionProvider)
  • Lyrics Sync — wav2vec2-based forced alignment via custom lsync module
  • Runtime — NVIDIA CUDA 12.4 + cuDNN 9, PyTorch 2.6 (cu124 wheels)
  • Container — nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04

GPU Notes

The container requires the NVIDIA Container Toolkit. On the host:

# Verify GPU is visible inside the container
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

PyTorch wheels are self-contained (cu124) and do not require a matching CUDA toolkit on the host — only the driver.

onnxruntime-gpu is used for MDX-Net and basic-pitch ONNX inference; it auto-selects CUDAExecutionProvider when a GPU is available.

S
Description
No description provided
Readme
33 MiB
0 Stars 1 Watchers 0 Forks
Languages
Jupyter Notebook 90.9%
TypeScript 4.7%
Python 3.7%
JavaScript 0.6%