21 KiB
Executable File
StemEngine - Karaoke Anything?
This is a vibe-coded project based on the work of people who actually know what they're doing. But now I can theoretically karaoke anything and so can you.
Documentation
- User docs: docs/USER_GUIDE.md
- Developer docs: docs/DEVELOPER_GUIDE.md
- Documentation plan: docs/DOCUMENTATION_PLAN.md
- API reference: docs/API_REFERENCE.md
- Troubleshooting matrix: docs/TROUBLESHOOTING_MATRIX.md
- Configuration reference: docs/CONFIGURATION_REFERENCE.md
- UVR comparison: docs/UVR_VS_STEMENGINE.md
- Long-track chunking plan: docs/LONG_TRACK_CHUNKING_PLAN.md
- Documentation governance: docs/DOC_GOVERNANCE.md
- Release checklist: docs/RELEASE_CHECKLIST.md
Features
- Stem Separation — 40+ models across 10 model families
- MIDI Transcription — basic-pitch with polyphonic and melody modes
- GPU Safety Limits — shared VRAM reserve for separation, lyric transcription, basic-pitch, and Omnizart sidecar
- ABC Notation — convert MIDI to ABC notation for sheet music
- Lyrics Sync — forced-alignment to produce ELRC files with word-level timestamps and a live animated preview
- Project Management — persistent project tabs, per-stem reprocessing, stem renaming
- Mix Presets — Instrumental, Static (remove elements), Dynamic (per-stem volume)
- Subsonic Integration — browse and import from a Subsonic-compatible server
- Model Downloader — download model weights on-demand from the Settings UI
Separation Models
| Family | Models |
|---|---|
| Demucs | v4, v4 Fine-tuned, v3 Extra Quantized, v3 |
| BS-RoFormer | 4-stem, 5-stem (Guitar), 5-stem (Piano), 6-stem, Chorus, Anvuew, Karaoke Anvuew, Karaoke Frazer+Becruily, Mamba2 Vocals (Adapter), SW 6-stem |
| MelBand RoFormer | 4-stem, Vocals, Vocals Anvuew, Vocals Gabox, Dereverb, Dereverb Anvuew, Dereverb Room, Karaoke, Karaoke Gabox, Karaoke Gabox (Small, opt-in), Karaoke Gabox V2 Experimental (opt-in), Deux Becruily |
| MDX-Net | Vocals FT, Inst HQ 1, Inst HQ 2, Karaoke, Dereverb, Drum Sep |
| SCNet | XL (2-stem), XL IHF, Masked XL IHF, Masked Small, Tran Small, Large (4-stem), Large Starrytong |
| MedleyVox | Base, 55/66/88/100 Epoch variants |
| Apollo | Vocals, EDM, EDM Big |
| Aspiration | Full, Light |
| Bandit v2 | Multi-Language, English |
| Ensemble | Vocal Best Quality, 4-Stem, 6-Stem Full |
Model weights are downloaded on first use and cached in /opt/.
Checkpoint formats (.ckpt vs .safetensors)
StemEngine's current BS-RoFormer / MelBand runtime primarily expects a model-specific config YAML plus a PyTorch checkpoint file such as .ckpt.
What safetensors means:
- it is a safer weight serialization format,
- it does not by itself change the MelBand / BS-RoFormer architecture,
- it does not materially reduce inference VRAM or compute cost,
- it can slightly improve load safety and sometimes startup/load efficiency.
Practical expectation in StemEngine:
- a
safetensorsconversion is usually the same trained model packaged differently, - current built-in model loaders are wired for
.ckpt+ YAML style checkpoints, - direct
safetensorssupport would require loader support forsafetensorsdeserialization and any key-mapping needed by the exported weights.
Requirements
- Docker with the NVIDIA Container Toolkit (for GPU acceleration)
- NVIDIA GPU with CUDA 12.4 support
- Docker Compose
CPU-only mode is available via CPU_SEPARATION=true but separation will be significantly slower.
Quick Start
-
Copy
.env.exampleto.envand fill in the required values (see Configuration below). -
Build and start:
docker compose up -d --build
- Open
http://localhost:8000in your browser.
Optional WebUI (Vite dev server) default host URL is http://localhost:5174.
Configuration
All configuration is via environment variables. The docker-compose.yml accepts these via a .env file in the same directory.
Required
| Variable | Description |
|---|---|
SEFOLDER |
Host path for persistent runtime data (uploads, jobs, DB, LRC sync) |
ENGINEFOLDER |
Host path for model weight caches |
PROJECTS |
Host path for separated project output |
SECRET_KEY |
Random hex key for encrypting Subsonic credentials at rest. Generate with: python3 -c "import secrets; print(secrets.token_hex(32))" |
Optional
| Variable | Default | Description |
|---|---|---|
CPU_SEPARATION |
false |
Set to true to force CPU inference |
MAX_UPLOAD_MB |
500 |
Maximum upload file size in MB |
STEMENGINE_GPU_VRAM_RESERVE_MB |
2048 |
Keeps this many MB free on the GPU (WSL2-safe headroom) |
OMNIZART_GPU_VRAM_RESERVE_MB |
STEMENGINE_GPU_VRAM_RESERVE_MB |
Optional sidecar override for Omnizart TensorFlow GPU cap |
STEMENGINE_UVICORN_RELOAD_FLAG |
empty | Set to --reload only when actively developing backend code; leave empty for long-running separation jobs |
WEBUI_HOST_PORT |
5174 |
Host port bound to the WebUI container (set if 5174 or 5173 is already in use) |
WEBUI_PORT |
5173 |
Internal WebUI container port (normally unchanged) |
MELBAND_STAGE_DIAGNOSTICS |
false |
Emit per-stage MelBand memory snapshots into job progress + backend logs |
GABOX_ENABLE_SMALL_MODELS |
false |
Expose Tier B small Gabox models (for constrained VRAM) |
GABOX_ENABLE_EXPERIMENTAL_MODELS |
false |
Expose Tier C experimental Gabox models |
GABOX_SMALL_MODEL_MAX_DURATION_SECS |
360 |
Guardrail threshold used by Gabox small karaoke model |
GABOX_SMALL_MODEL_PREFERRED_FOR_LOW_VRAM |
false |
Reserved policy flag for low-VRAM auto-preference workflows |
STEMENGINE_LYRIC_SEGMENTER |
auto |
Lyric line splitter strategy: auto, karaoke_processor, legacy |
STEMENGINE_LYRIC_MAX_LINE_LENGTH |
36 |
Max line length target used by karaoke lyric segmentation |
STEMENGINE_KARAOKE_EXPORT_DIR |
empty | Optional global .sek export directory inside the container. If empty, exports go under each project (Exports/StemEngineKaraoke). |
SEK_EXPORTS |
${SEFOLDER}/sek_exports |
Host path mounted to /app/media/sek_exports for centralized .sek package output. |
TZ |
UTC |
Timezone |
NAVSERVER |
— | Subsonic server URL (can also be set in the UI) |
NAVUSER |
— | Subsonic username |
NAVPASS |
— | Subsonic password |
Model Directories
These are set automatically in docker-compose.yml and point inside the /opt/ volume. Override only if you need a custom layout.
BS_ROFORMER_MODEL_DIR, MEL_BAND_MODEL_DIR, MDXNET_MODEL_DIR, SCNET_MODEL_DIR, MEDLEYVOX_MODEL_DIR, APOLLO_MODEL_DIR, ASPIRATION_MODEL_DIR, BANDIT_MODEL_DIR
Project Folder Structure
Each separation job creates a folder under PROJECTS_DIR:
Artist - Title/
Artist - Title.mp3 # Original source audio (copy)
Artist - Title.elrc # ELRC lyrics with word-level timestamps (if aligned)
Artist - Title.lrc # Plain LRC (if aligned)
Stems/
vocals.mp3
drums.mp3
bass.mp3
other.mp3
Instrumental/ # Only when Instrumental preset selected
Artist - Title - Instrumental.mp3
Static/ # Only when Static preset selected
Artist - Title - No Vocals.mp3
Dynamic/ # Only when Dynamic preset selected
Artist - Title - Dynamic Mix.mp3
Tech Stack
- Frontend — React 18, TypeScript, Vite, TailwindCSS
- Backend — FastAPI, Python 3.11, SQLAlchemy
- Separation — Demucs, BS-RoFormer, MelBand RoFormer, MDX-Net (onnxruntime-gpu), SCNet, MedleyVox, Apollo, Aspiration, Bandit
- Transcription — basic-pitch (ONNX, CUDAExecutionProvider)
- Lyrics Sync — wav2vec2-based forced alignment via custom
lsyncmodule - Runtime — NVIDIA CUDA 12.4 + cuDNN 9, PyTorch 2.6 (cu124 wheels)
- Container —
nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04
Auto Pipeline + Lyric Engine Notes (2026-03)
- Auto Pipeline now supports:
- per-instrument refinement mapping (
instrument_models) at separation stage - lyric engine selection (
WhisperXor self-hostedParakeet) in lyric steps - canonical stem finalization after separation (rename to project-style filenames + automatic removal of silent/no-significant-audio stems)
- per-instrument refinement mapping (
- Saved auto-pipeline configs are migrated on read:
- legacy
whisperModelis upgraded to engine-awareengine+model - missing
instrumentModelsdefaults to empty mapping
- legacy
Parakeet constraints
Parakeet uses alternate handling and does not currently expose WhisperX-only controls:
diarizelsync_aligntwo_passexplicittranslatetask
If Parakeet is selected and unavailable, verify sidecar health and profile wiring:
- backend must have
PARAKEET_ENABLED=true - backend
PARAKEET_SIDECAR_URLmust point to the sidecar service - sidecar
GET /healthmust return200
WebUI routing (Phase 1)
- The WebUI now uses URL-backed navigation for the primary screens:
/→ separator/home/settings→ settings/projects/:projectId→ project detail
- Refreshing on a project page now restores that same project view directly from the URL.
- Sidebar project entries are real links, which is the foundation for browser-native new-tab and deep-link behavior.
WebUI persistence and browser-tab behavior (Phases 2/3)
- The WebUI now persists key client-side shell state across refreshes, including:
- sidebar open/collapsed state,
- sidebar project search text,
- separator form context (input mode, selected model/format/preset, project name, imported file selection when still available).
- Auto Pipeline now saves a lightweight draft for its project name and offers restore/discard when reopened.
- Project completion surfaces now expose route-native links so projects can be opened directly or with normal browser new-tab behavior.
WebUI PWA hardening (Phase 4)
- The WebUI manifest is now hardened with explicit PWA metadata (
id,scope,display_override, categories, and maskable/standard icon declarations). - Service worker caching is versioned and conservative:
- shell assets are precached,
- route navigations are network-first with shell fallback,
- API responses are never cached as durable offline truth.
- The shell now includes:
- install affordance when
beforeinstallpromptis available, - update notification with explicit “Reload to update” flow,
- offline status messaging so disconnected behavior is clear.
- install affordance when
WebUI cross-tab session hygiene (Phase 5)
- Cross-tab project deletion notifications are now propagated via
BroadcastChannelwith localStorage fallback. - Other open tabs now invalidate stale project views when a project is deleted and automatically leave deleted project routes.
- Durable appearance/theme changes now propagate across tabs via storage-based synchronization.
Frontend bundle split hardening
- Route-level lazy loading is now used for major pages.
- Vite chunk warning threshold is tuned for the split-route architecture to avoid false-positive size warnings during normal production builds.
WebUI QoL closure (Phase 6)
- Final acceptance checklist is tracked in docs/WEBUI_QOL_QA_CHECKLIST.md.
- Cross-tab events now cover project deletion, project invalidation, and project-list changes.
- Automated browser smoke tests are available via Playwright (
cd frontend && npm run e2e).
GPU Notes
The container requires the NVIDIA Container Toolkit. On the host:
# Verify GPU is visible inside the container
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
PyTorch wheels are self-contained (cu124) and do not require a matching CUDA toolkit on the host — only the driver.
STEMENGINE_GPU_VRAM_RESERVE_MB is applied in backend GPU code paths (separation, lyric transcription, and basic-pitch startup) to preserve display headroom on WSL2.
Omnizart sidecar now applies a TensorFlow memory cap using OMNIZART_GPU_VRAM_RESERVE_MB (or STEMENGINE_GPU_VRAM_RESERVE_MB fallback), so sidecar transcription follows the same reserve policy.
Targeted backend tests (Findings 3/4/5 + optional hardening)
To run the bounded jobs API + retention/cancel-state hardening regression suite with full backend dependencies, use the compose test profile:
docker compose -f stemengine.yml --profile tests run --rm stemengine-tests
or the Make helper:
make test-findings345
This runs:
backend.tests.test_jobs_apibackend.tests.test_project_cleanupbackend.tests.test_projects_deletebackend.tests.test_findings_3_4_unitbackend.tests.test_ensemble_separatorbackend.tests.test_optional_hardeningbackend.tests.test_stem_finalize
Test-only Python dependencies for this profile are declared in backend/requirements-test.txt.
Use this when validating performance-fix regressions around long-run job history growth, retention pruning of old non-project jobs, in-memory cancellation-state cleanup, ensemble-memory accumulation behavior, and automatic stem finalization.
RTX 3060 / Constrained-VRAM Stability
Why MelBand Karaoke can OOM
The karaoke checkpoints (mel_band_karaoke, mel_band_karaoke_gabox) load a ~1.7 GB PyTorch checkpoint and allocate tensors proportional to the full track length × batch_size × num_overlap. On a 6–8 GB GPU with WSL2 overhead, this easily exhausts VRAM on:
performancepreset (batch_size=8, num_overlap=4)- tracks longer than 8 minutes
- warm GPU with cached allocations from prior jobs
MelBand stage diagnostics
Set MELBAND_STAGE_DIAGNOSTICS=true to instrument the expensive setup stages that happen before the first visible separation segment begins.
The backend now reports memory snapshots for these MelBand stages:
- start
- before audio load
- after audio load
- before model load
- after model load
- before demix/preallocation
- after demix
- after cleanup
Each snapshot includes:
- process RSS
- current CUDA allocated memory
- current CUDA reserved memory
- peak CUDA allocated memory
- peak CUDA reserved memory
Example progress message:
[diag] MelBand lead/backing_instrumental: after model load — RSS=1840.2 MiB, CUDA alloc=1723.5 MiB, reserved=1824.0 MiB, peak_alloc=1723.5 MiB, peak_reserved=1824.0 MiB
This makes it much easier to tell whether the main spike came from whole-file audio decode, checkpoint load, or the full-length overlap-add buffers.
Automatic VRAM headroom recovery
Before each separation job, StemEngine now unloads any cached in-process Whisper models left behind by lyric transcription and flushes idle CUDA allocator blocks. This prevents an otherwise-idle transcription model from quietly reducing the VRAM available to BS-RoFormer / MelBand jobs.
In addition, GPU separation jobs now run inside a short-lived child Python process by default. This keeps allocator fragmentation and CUDA fault state from accumulating indefinitely inside the main backend process after repeated heavy jobs or prior OOM events.
You can disable separator child-process isolation only if needed for debugging:
| Variable | Default | Effect |
|---|---|---|
STEMENGINE_SEPARATION_SUBPROCESS |
true |
When false, run separator jobs in the main backend process instead of a short-lived child process |
OOM ladder
When a MelBand Karaoke job encounters an out-of-memory error the backend automatically retries with progressively safer parameters before giving up:
| Attempt | batch_size | num_overlap | Notes |
|---|---|---|---|
| 1 | preset default | preset default | Normal run |
| 2 | 1 | preset default | batch_size reduced |
| 3 | 1 | 2 | num_overlap also reduced (eco) |
| fallback | — | — | Cross-model to mdxnet_karaoke (ONNX) |
The fallback step is governed by FALLBACK_REQUIRE_LEAD_BACK (see below).
Lead/backing-capable fallback policy
FALLBACK_REQUIRE_LEAD_BACK=true (default) ensures the cross-model fallback only ever uses a model that also produces lead and backing_instrumental stems — never a plain vocals/instrumental model. This prevents silent degradation of karaoke output quality.
Models that qualify as lead/back-capable:
| Model | Notes |
|---|---|
mel_band_karaoke |
MelBand primary — full PyTorch |
mel_band_karaoke_gabox |
MelBand alternate checkpoint |
bs_roformer_karaoke_anvuew |
BS-RoFormer karaoke variant |
bs_roformer_karaoke_frazer_becruily |
BS-RoFormer karaoke variant |
mdxnet_karaoke |
ONNX — lowest VRAM (~200 MB), default fallback target |
If all fallback models also OOM, the job fails with a clear error message listing what was tried and actionable suggestions.
Set FALLBACK_REQUIRE_LEAD_BACK=false to skip the cross-model fallback entirely and surface the OOM error immediately.
Duration guardrail
Tracks longer than LONG_TRACK_DURATION_SECS (default: 480 s = 8 min) have MelBand eco params (batch_size=1, num_overlap=2) forced before inference, regardless of the active power preset. The guardrail fires for both mel_band_karaoke and mel_band_roformer_4stem.
A log message is emitted when the guardrail activates so you can confirm it in the job progress stream.
Recommended settings for RTX 3060 (12 GB WSL2)
| Variable | Recommended | Reason |
|---|---|---|
STEMENGINE_GPU_VRAM_RESERVE_MB |
2048 |
Leaves headroom for Windows DWM/WDDM |
LONG_TRACK_DURATION_SECS |
480 |
Force eco on tracks ≥ 8 min |
FALLBACK_REQUIRE_LEAD_BACK |
true |
Never lose lead/back distinction silently |
| Power preset | standard or eco |
Avoid performance on this GPU |
Troubleshooting sequence
- Check job progress — OOM retries and fallback activations are logged inline in the job progress field.
- Use
/api/system/gpu-resetor Settings → GPU reset — this clears cached transcription models and flushes CUDA state without a full restart. - Restart the container — if the GPU context is truly corrupted, process isolation reduces recurrence but a hard CUDA fault can still require a full restart.
- Use eco preset —
batch_size=1, num_overlap=2is the safest configuration. - Split long tracks — tracks > 8 min on this GPU should be split before separation.
- Check VRAM headroom — on WSL2, run
nvidia-smion the host while idle and verify that ≥ 9 GB shows as free before submitting a karaoke job.
RoFormer constructor compatibility note
If you see an error such as:
TypeError: MelBandRoformer.__init__() got an unexpected keyword argument 'freqs_per_bands'
this usually means the model config was authored for a different bs_roformer package constructor signature.
Current backend builds include a compatibility shim that:
- remaps known renamed keys when possible,
- drops unsupported constructor keys with a backend warning,
- applies to MelBand and BS-RoFormer loaders.
If the error persists, verify that the checkpoint and YAML config belong to the same model variant.
If you see an error such as:
Error(s) in loading state_dict for MelBandRoformer: size mismatch for ...
the checkpoint and config likely describe different architectures/revisions. The backend now surfaces checkpoint/config filenames in this failure path to speed triage.
Note: the core mel_band_karaoke_gabox checkpoint is sourced from a bsroformers/ path and is handled with BS-RoFormer architecture loading to avoid MelBand class mismatch.
GaboxR67 model expansion (implemented rollout)
Source scope: https://huggingface.co/GaboxR67 (primarily GaboxR67/MelBandRoformers).
Does this include small models?
Yes. Small models are implemented as an explicit opt-in tier controlled by GABOX_ENABLE_SMALL_MODELS.
Rollout tiers
| Tier | Target | Purpose |
|---|---|---|
| Tier A (core) | Stable Gabox MelBand models (mel_band_vocals_gabox) |
Quality-first default additions |
| Tier B (small) | mel_band_karaoke_gabox_small |
Faster/safer runs on constrained GPUs (opt-in) |
| Tier C (experimental) | mel_band_karaoke_gabox_v2_exp |
Opt-in only; no default fallback usage |
Integration boundaries
- Reused existing MelBand specialized runner architecture.
- Added model IDs + labels to backend model catalog.
- Added downloader mappings for Gabox checkpoints/config files.
- Small/experimental tiers are hidden unless enabled by env controls.
- Small karaoke model has an additional duration guardrail using
GABOX_SMALL_MODEL_MAX_DURATION_SECS.
Current behavior notes
mel_band_vocals_gaboxis always available.mel_band_karaoke_gabox_smallappears only whenGABOX_ENABLE_SMALL_MODELS=true.mel_band_karaoke_gabox_v2_expappears only whenGABOX_ENABLE_EXPERIMENTAL_MODELS=true.- MelBand diagnostics and OOM ladder remain active for Gabox variants.