16 KiB
Executable File
StemEngine - Karaoke Anything?
This is a vibe-coded project based on the work of people who actually know what they're doing. But now I can theoretically karaoke anything and so can you.
Documentation
- User docs: docs/USER_GUIDE.md
- Developer docs: docs/DEVELOPER_GUIDE.md
- Documentation plan: docs/DOCUMENTATION_PLAN.md
- API reference: docs/API_REFERENCE.md
- Troubleshooting matrix: docs/TROUBLESHOOTING_MATRIX.md
- Configuration reference: docs/CONFIGURATION_REFERENCE.md
- UVR comparison: docs/UVR_VS_STEMENGINE.md
- Long-track chunking plan: docs/LONG_TRACK_CHUNKING_PLAN.md
- Documentation governance: docs/DOC_GOVERNANCE.md
- Release checklist: docs/RELEASE_CHECKLIST.md
Features
- Stem Separation — 40+ models across 10 model families
- MIDI Transcription — basic-pitch with polyphonic and melody modes
- GPU Safety Limits — shared VRAM reserve for separation, lyric transcription, basic-pitch, and Omnizart sidecar
- ABC Notation — convert MIDI to ABC notation for sheet music
- Lyrics Sync — forced-alignment to produce ELRC files with word-level timestamps and a live animated preview
- Project Management — persistent project tabs, per-stem reprocessing, stem renaming
- Mix Presets — Instrumental, Static (remove elements), Dynamic (per-stem volume)
- Subsonic Integration — browse and import from a Subsonic-compatible server
- Model Downloader — download model weights on-demand from the Settings UI
Separation Models
| Family | Models |
|---|---|
| Demucs | v4, v4 Fine-tuned, v3 Extra Quantized, v3 |
| BS-RoFormer | 4-stem, 5-stem (Guitar), 5-stem (Piano), 6-stem, Chorus, Anvuew, Karaoke Anvuew, Karaoke Frazer+Becruily, Mamba2 Vocals (Adapter), SW 6-stem |
| MelBand RoFormer | 4-stem, Vocals, Vocals Anvuew, Vocals Gabox, Dereverb, Dereverb Anvuew, Dereverb Room, Karaoke, Karaoke Gabox, Karaoke Gabox (Small, opt-in), Karaoke Gabox V2 Experimental (opt-in), Deux Becruily |
| MDX-Net | Vocals FT, Inst HQ 1, Inst HQ 2, Karaoke, Dereverb, Drum Sep |
| SCNet | XL (2-stem), XL IHF, Masked XL IHF, Masked Small, Tran Small, Large (4-stem), Large Starrytong |
| MedleyVox | Base, 55/66/88/100 Epoch variants |
| Apollo | Vocals, EDM, EDM Big |
| Aspiration | Full, Light |
| Bandit v2 | Multi-Language, English |
| Ensemble | Vocal Best Quality, 4-Stem, 6-Stem Full |
Model weights are downloaded on first use and cached in /opt/.
Checkpoint formats (.ckpt vs .safetensors)
StemEngine's current BS-RoFormer / MelBand runtime primarily expects a model-specific config YAML plus a PyTorch checkpoint file such as .ckpt.
What safetensors means:
- it is a safer weight serialization format,
- it does not by itself change the MelBand / BS-RoFormer architecture,
- it does not materially reduce inference VRAM or compute cost,
- it can slightly improve load safety and sometimes startup/load efficiency.
Practical expectation in StemEngine:
- a
safetensorsconversion is usually the same trained model packaged differently, - current built-in model loaders are wired for
.ckpt+ YAML style checkpoints, - direct
safetensorssupport would require loader support forsafetensorsdeserialization and any key-mapping needed by the exported weights.
Requirements
- Docker with the NVIDIA Container Toolkit (for GPU acceleration)
- NVIDIA GPU with CUDA 12.4 support
- Docker Compose
CPU-only mode is available via CPU_SEPARATION=true but separation will be significantly slower.
Quick Start
-
Copy
.env.exampleto.envand fill in the required values (see Configuration below). -
Build and start:
docker compose up -d --build
- Open
http://localhost:8000in your browser.
Optional WebUI (Vite dev server) default host URL is http://localhost:5174.
Configuration
All configuration is via environment variables. The docker-compose.yml accepts these via a .env file in the same directory.
Required
| Variable | Description |
|---|---|
SEFOLDER |
Host path for persistent runtime data (uploads, jobs, DB, LRC sync) |
ENGINEFOLDER |
Host path for model weight caches |
PROJECTS |
Host path for separated project output |
SECRET_KEY |
Random hex key for encrypting Subsonic credentials at rest. Generate with: python3 -c "import secrets; print(secrets.token_hex(32))" |
Optional
| Variable | Default | Description |
|---|---|---|
CPU_SEPARATION |
false |
Set to true to force CPU inference |
MAX_UPLOAD_MB |
500 |
Maximum upload file size in MB |
STEMENGINE_GPU_VRAM_RESERVE_MB |
2048 |
Keeps this many MB free on the GPU (WSL2-safe headroom) |
OMNIZART_GPU_VRAM_RESERVE_MB |
STEMENGINE_GPU_VRAM_RESERVE_MB |
Optional sidecar override for Omnizart TensorFlow GPU cap |
STEMENGINE_UVICORN_RELOAD_FLAG |
empty | Set to --reload only when actively developing backend code; leave empty for long-running separation jobs |
WEBUI_HOST_PORT |
5174 |
Host port bound to the WebUI container (set if 5174 or 5173 is already in use) |
WEBUI_PORT |
5173 |
Internal WebUI container port (normally unchanged) |
MELBAND_STAGE_DIAGNOSTICS |
false |
Emit per-stage MelBand memory snapshots into job progress + backend logs |
GABOX_ENABLE_SMALL_MODELS |
false |
Expose Tier B small Gabox models (for constrained VRAM) |
GABOX_ENABLE_EXPERIMENTAL_MODELS |
false |
Expose Tier C experimental Gabox models |
GABOX_SMALL_MODEL_MAX_DURATION_SECS |
360 |
Guardrail threshold used by Gabox small karaoke model |
GABOX_SMALL_MODEL_PREFERRED_FOR_LOW_VRAM |
false |
Reserved policy flag for low-VRAM auto-preference workflows |
STEMENGINE_LYRIC_SEGMENTER |
auto |
Lyric line splitter strategy: auto, karaoke_processor, legacy |
STEMENGINE_LYRIC_MAX_LINE_LENGTH |
36 |
Max line length target used by karaoke lyric segmentation |
TZ |
UTC |
Timezone |
NAVSERVER |
— | Subsonic server URL (can also be set in the UI) |
NAVUSER |
— | Subsonic username |
NAVPASS |
— | Subsonic password |
Model Directories
These are set automatically in docker-compose.yml and point inside the /opt/ volume. Override only if you need a custom layout.
BS_ROFORMER_MODEL_DIR, MEL_BAND_MODEL_DIR, MDXNET_MODEL_DIR, SCNET_MODEL_DIR, MEDLEYVOX_MODEL_DIR, APOLLO_MODEL_DIR, ASPIRATION_MODEL_DIR, BANDIT_MODEL_DIR
Project Folder Structure
Each separation job creates a folder under PROJECTS_DIR:
Artist - Title/
Artist - Title.mp3 # Original source audio (copy)
Artist - Title.elrc # ELRC lyrics with word-level timestamps (if aligned)
Artist - Title.lrc # Plain LRC (if aligned)
Stems/
vocals.mp3
drums.mp3
bass.mp3
other.mp3
Instrumental/ # Only when Instrumental preset selected
Artist - Title - Instrumental.mp3
Static/ # Only when Static preset selected
Artist - Title - No Vocals.mp3
Dynamic/ # Only when Dynamic preset selected
Artist - Title - Dynamic Mix.mp3
Tech Stack
- Frontend — React 18, TypeScript, Vite, TailwindCSS
- Backend — FastAPI, Python 3.11, SQLAlchemy
- Separation — Demucs, BS-RoFormer, MelBand RoFormer, MDX-Net (onnxruntime-gpu), SCNet, MedleyVox, Apollo, Aspiration, Bandit
- Transcription — basic-pitch (ONNX, CUDAExecutionProvider)
- Lyrics Sync — wav2vec2-based forced alignment via custom
lsyncmodule - Runtime — NVIDIA CUDA 12.4 + cuDNN 9, PyTorch 2.6 (cu124 wheels)
- Container —
nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04
Auto Pipeline + Lyric Engine Notes (2026-03)
- Auto Pipeline now supports:
- per-instrument refinement mapping (
instrument_models) at separation stage - lyric engine selection (
WhisperXor self-hostedParakeet) in lyric steps
- per-instrument refinement mapping (
- Saved auto-pipeline configs are migrated on read:
- legacy
whisperModelis upgraded to engine-awareengine+model - missing
instrumentModelsdefaults to empty mapping
- legacy
Parakeet constraints
Parakeet uses alternate handling and does not currently expose WhisperX-only controls:
diarizelsync_aligntwo_passexplicittranslatetask
If Parakeet is selected and unavailable, verify sidecar health and profile wiring:
- backend must have
PARAKEET_ENABLED=true - backend
PARAKEET_SIDECAR_URLmust point to the sidecar service - sidecar
GET /healthmust return200
GPU Notes
The container requires the NVIDIA Container Toolkit. On the host:
# Verify GPU is visible inside the container
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
PyTorch wheels are self-contained (cu124) and do not require a matching CUDA toolkit on the host — only the driver.
STEMENGINE_GPU_VRAM_RESERVE_MB is applied in backend GPU code paths (separation, lyric transcription, and basic-pitch startup) to preserve display headroom on WSL2.
Omnizart sidecar now applies a TensorFlow memory cap using OMNIZART_GPU_VRAM_RESERVE_MB (or STEMENGINE_GPU_VRAM_RESERVE_MB fallback), so sidecar transcription follows the same reserve policy.
RTX 3060 / Constrained-VRAM Stability
Why MelBand Karaoke can OOM
The karaoke checkpoints (mel_band_karaoke, mel_band_karaoke_gabox) load a ~1.7 GB PyTorch checkpoint and allocate tensors proportional to the full track length × batch_size × num_overlap. On a 6–8 GB GPU with WSL2 overhead, this easily exhausts VRAM on:
performancepreset (batch_size=8, num_overlap=4)- tracks longer than 8 minutes
- warm GPU with cached allocations from prior jobs
MelBand stage diagnostics
Set MELBAND_STAGE_DIAGNOSTICS=true to instrument the expensive setup stages that happen before the first visible separation segment begins.
The backend now reports memory snapshots for these MelBand stages:
- start
- before audio load
- after audio load
- before model load
- after model load
- before demix/preallocation
- after demix
- after cleanup
Each snapshot includes:
- process RSS
- current CUDA allocated memory
- current CUDA reserved memory
- peak CUDA allocated memory
- peak CUDA reserved memory
Example progress message:
[diag] MelBand lead/backing_instrumental: after model load — RSS=1840.2 MiB, CUDA alloc=1723.5 MiB, reserved=1824.0 MiB, peak_alloc=1723.5 MiB, peak_reserved=1824.0 MiB
This makes it much easier to tell whether the main spike came from whole-file audio decode, checkpoint load, or the full-length overlap-add buffers.
OOM ladder
When a MelBand Karaoke job encounters an out-of-memory error the backend automatically retries with progressively safer parameters before giving up:
| Attempt | batch_size | num_overlap | Notes |
|---|---|---|---|
| 1 | preset default | preset default | Normal run |
| 2 | 1 | preset default | batch_size reduced |
| 3 | 1 | 2 | num_overlap also reduced (eco) |
| fallback | — | — | Cross-model to mdxnet_karaoke (ONNX) |
The fallback step is governed by FALLBACK_REQUIRE_LEAD_BACK (see below).
Lead/backing-capable fallback policy
FALLBACK_REQUIRE_LEAD_BACK=true (default) ensures the cross-model fallback only ever uses a model that also produces lead and backing_instrumental stems — never a plain vocals/instrumental model. This prevents silent degradation of karaoke output quality.
Models that qualify as lead/back-capable:
| Model | Notes |
|---|---|
mel_band_karaoke |
MelBand primary — full PyTorch |
mel_band_karaoke_gabox |
MelBand alternate checkpoint |
bs_roformer_karaoke_anvuew |
BS-RoFormer karaoke variant |
bs_roformer_karaoke_frazer_becruily |
BS-RoFormer karaoke variant |
mdxnet_karaoke |
ONNX — lowest VRAM (~200 MB), default fallback target |
If all fallback models also OOM, the job fails with a clear error message listing what was tried and actionable suggestions.
Set FALLBACK_REQUIRE_LEAD_BACK=false to skip the cross-model fallback entirely and surface the OOM error immediately.
Duration guardrail
Tracks longer than LONG_TRACK_DURATION_SECS (default: 480 s = 8 min) have MelBand eco params (batch_size=1, num_overlap=2) forced before inference, regardless of the active power preset. The guardrail fires for both mel_band_karaoke and mel_band_roformer_4stem.
A log message is emitted when the guardrail activates so you can confirm it in the job progress stream.
Recommended settings for RTX 3060 (12 GB WSL2)
| Variable | Recommended | Reason |
|---|---|---|
STEMENGINE_GPU_VRAM_RESERVE_MB |
2048 |
Leaves headroom for Windows DWM/WDDM |
LONG_TRACK_DURATION_SECS |
480 |
Force eco on tracks ≥ 8 min |
FALLBACK_REQUIRE_LEAD_BACK |
true |
Never lose lead/back distinction silently |
| Power preset | standard or eco |
Avoid performance on this GPU |
Troubleshooting sequence
- Check job progress — OOM retries and fallback activations are logged inline in the job progress field.
- Restart the container — CUDA context fragmentation from a previous OOM can persist until the process exits.
- Use eco preset —
batch_size=1, num_overlap=2is the safest configuration. - Split long tracks — tracks > 8 min on this GPU should be split before separation.
- Check VRAM headroom — on WSL2, run
nvidia-smion the host while idle and verify that ≥ 9 GB shows as free before submitting a karaoke job.
RoFormer constructor compatibility note
If you see an error such as:
TypeError: MelBandRoformer.__init__() got an unexpected keyword argument 'freqs_per_bands'
this usually means the model config was authored for a different bs_roformer package constructor signature.
Current backend builds include a compatibility shim that:
- remaps known renamed keys when possible,
- drops unsupported constructor keys with a backend warning,
- applies to MelBand and BS-RoFormer loaders.
If the error persists, verify that the checkpoint and YAML config belong to the same model variant.
If you see an error such as:
Error(s) in loading state_dict for MelBandRoformer: size mismatch for ...
the checkpoint and config likely describe different architectures/revisions. The backend now surfaces checkpoint/config filenames in this failure path to speed triage.
Note: the core mel_band_karaoke_gabox checkpoint is sourced from a bsroformers/ path and is handled with BS-RoFormer architecture loading to avoid MelBand class mismatch.
GaboxR67 model expansion (implemented rollout)
Source scope: https://huggingface.co/GaboxR67 (primarily GaboxR67/MelBandRoformers).
Does this include small models?
Yes. Small models are implemented as an explicit opt-in tier controlled by GABOX_ENABLE_SMALL_MODELS.
Rollout tiers
| Tier | Target | Purpose |
|---|---|---|
| Tier A (core) | Stable Gabox MelBand models (mel_band_vocals_gabox) |
Quality-first default additions |
| Tier B (small) | mel_band_karaoke_gabox_small |
Faster/safer runs on constrained GPUs (opt-in) |
| Tier C (experimental) | mel_band_karaoke_gabox_v2_exp |
Opt-in only; no default fallback usage |
Integration boundaries
- Reused existing MelBand specialized runner architecture.
- Added model IDs + labels to backend model catalog.
- Added downloader mappings for Gabox checkpoints/config files.
- Small/experimental tiers are hidden unless enabled by env controls.
- Small karaoke model has an additional duration guardrail using
GABOX_SMALL_MODEL_MAX_DURATION_SECS.
Current behavior notes
mel_band_vocals_gaboxis always available.mel_band_karaoke_gabox_smallappears only whenGABOX_ENABLE_SMALL_MODELS=true.mel_band_karaoke_gabox_v2_expappears only whenGABOX_ENABLE_EXPERIMENTAL_MODELS=true.- MelBand diagnostics and OOM ladder remain active for Gabox variants.