# StemEngine - Karaoke Anything? > This is a **vibe-coded** project based on the work of people who actually know what they're doing. But now I can theoretically karaoke anything and so can you. --- ## Documentation - User docs: [docs/USER_GUIDE.md](docs/USER_GUIDE.md) - Developer docs: [docs/DEVELOPER_GUIDE.md](docs/DEVELOPER_GUIDE.md) - Documentation plan: [docs/DOCUMENTATION_PLAN.md](docs/DOCUMENTATION_PLAN.md) - API reference: [docs/API_REFERENCE.md](docs/API_REFERENCE.md) - Troubleshooting matrix: [docs/TROUBLESHOOTING_MATRIX.md](docs/TROUBLESHOOTING_MATRIX.md) - Configuration reference: [docs/CONFIGURATION_REFERENCE.md](docs/CONFIGURATION_REFERENCE.md) - UVR comparison: [docs/UVR_VS_STEMENGINE.md](docs/UVR_VS_STEMENGINE.md) - Long-track chunking plan: [docs/LONG_TRACK_CHUNKING_PLAN.md](docs/LONG_TRACK_CHUNKING_PLAN.md) - Documentation governance: [docs/DOC_GOVERNANCE.md](docs/DOC_GOVERNANCE.md) - Release checklist: [docs/RELEASE_CHECKLIST.md](docs/RELEASE_CHECKLIST.md) --- ## Features - **Stem Separation** — 40+ models across 10 model families - **MIDI Transcription** — basic-pitch with polyphonic and melody modes - **GPU Safety Limits** — shared VRAM reserve for separation, lyric transcription, basic-pitch, and Omnizart sidecar - **ABC Notation** — convert MIDI to ABC notation for sheet music - **Lyrics Sync** — forced-alignment to produce ELRC files with word-level timestamps and a live animated preview - **Project Management** — persistent project tabs, per-stem reprocessing, stem renaming - **Mix Presets** — Instrumental, Static (remove elements), Dynamic (per-stem volume) - **Subsonic Integration** — browse and import from a Subsonic-compatible server - **Model Downloader** — download model weights on-demand from the Settings UI --- ## Separation Models | Family | Models | |---|---| | **Demucs** | v4, v4 Fine-tuned, v3 Extra Quantized, v3 | | **BS-RoFormer** | 4-stem, 5-stem (Guitar), 5-stem (Piano), 6-stem, Chorus, Anvuew, Karaoke Anvuew, Karaoke Frazer+Becruily, Mamba2 Vocals (Adapter), SW 6-stem | | **MelBand RoFormer** | 4-stem, Vocals, Vocals Anvuew, Vocals Gabox, Dereverb, Dereverb Anvuew, Dereverb Room, Karaoke, Karaoke Gabox, Karaoke Gabox (Small, opt-in), Karaoke Gabox V2 Experimental (opt-in), Deux Becruily | | **MDX-Net** | Vocals FT, Inst HQ 1, Inst HQ 2, Karaoke, Dereverb, Drum Sep | | **SCNet** | XL (2-stem), XL IHF, Masked XL IHF, Masked Small, Tran Small, Large (4-stem), Large Starrytong | | **MedleyVox** | Base, 55/66/88/100 Epoch variants | | **Apollo** | Vocals, EDM, EDM Big | | **Aspiration** | Full, Light | | **Bandit v2** | Multi-Language, English | | **Ensemble** | Vocal Best Quality, 4-Stem, 6-Stem Full | Model weights are downloaded on first use and cached in `/opt/`. ### Checkpoint formats (`.ckpt` vs `.safetensors`) StemEngine's current BS-RoFormer / MelBand runtime primarily expects a model-specific config YAML plus a PyTorch checkpoint file such as `.ckpt`. What `safetensors` means: - it is a safer weight serialization format, - it does **not** by itself change the MelBand / BS-RoFormer architecture, - it does **not** materially reduce inference VRAM or compute cost, - it can slightly improve load safety and sometimes startup/load efficiency. Practical expectation in StemEngine: - a `safetensors` conversion is usually the same trained model packaged differently, - current built-in model loaders are wired for `.ckpt` + YAML style checkpoints, - direct `safetensors` support would require loader support for `safetensors` deserialization and any key-mapping needed by the exported weights. --- ## Requirements - Docker with the NVIDIA Container Toolkit (for GPU acceleration) - NVIDIA GPU with CUDA 12.4 support - Docker Compose CPU-only mode is available via `CPU_SEPARATION=true` but separation will be significantly slower. --- ## Quick Start 1. Copy `.env.example` to `.env` and fill in the required values (see Configuration below). 2. Build and start: ```bash docker compose up -d --build ``` 3. Open `http://localhost:8000` in your browser. Optional WebUI (Vite dev server) default host URL is `http://localhost:5174`. --- ## Configuration All configuration is via environment variables. The `docker-compose.yml` accepts these via a `.env` file in the same directory. ### Required | Variable | Description | |---|---| | `SEFOLDER` | Host path for persistent runtime data (uploads, jobs, DB, LRC sync) | | `ENGINEFOLDER` | Host path for model weight caches | | `PROJECTS` | Host path for separated project output | | `SECRET_KEY` | Random hex key for encrypting Subsonic credentials at rest. Generate with: `python3 -c "import secrets; print(secrets.token_hex(32))"` | ### Optional | Variable | Default | Description | |---|---|---| | `CPU_SEPARATION` | `false` | Set to `true` to force CPU inference | | `MAX_UPLOAD_MB` | `500` | Maximum upload file size in MB | | `STEMENGINE_GPU_VRAM_RESERVE_MB` | `2048` | Keeps this many MB free on the GPU (WSL2-safe headroom) | | `OMNIZART_GPU_VRAM_RESERVE_MB` | `STEMENGINE_GPU_VRAM_RESERVE_MB` | Optional sidecar override for Omnizart TensorFlow GPU cap | | `STEMENGINE_UVICORN_RELOAD_FLAG` | empty | Set to `--reload` only when actively developing backend code; leave empty for long-running separation jobs | | `WEBUI_HOST_PORT` | `5174` | Host port bound to the WebUI container (set if `5174` or `5173` is already in use) | | `WEBUI_PORT` | `5173` | Internal WebUI container port (normally unchanged) | | `MELBAND_STAGE_DIAGNOSTICS` | `false` | Emit per-stage MelBand memory snapshots into job progress + backend logs | | `GABOX_ENABLE_SMALL_MODELS` | `false` | Expose Tier B small Gabox models (for constrained VRAM) | | `GABOX_ENABLE_EXPERIMENTAL_MODELS` | `false` | Expose Tier C experimental Gabox models | | `GABOX_SMALL_MODEL_MAX_DURATION_SECS` | `360` | Guardrail threshold used by Gabox small karaoke model | | `GABOX_SMALL_MODEL_PREFERRED_FOR_LOW_VRAM` | `false` | Reserved policy flag for low-VRAM auto-preference workflows | | `STEMENGINE_LYRIC_SEGMENTER` | `auto` | Lyric line splitter strategy: `auto`, `karaoke_processor`, `legacy` | | `STEMENGINE_LYRIC_MAX_LINE_LENGTH` | `36` | Max line length target used by karaoke lyric segmentation | | `STEMENGINE_KARAOKE_EXPORT_DIR` | empty | Optional global `.sek` export directory inside the container. If empty, exports go under each project (`Exports/StemEngineKaraoke`). | | `SEK_EXPORTS` | `${SEFOLDER}/sek_exports` | Host path mounted to `/app/media/sek_exports` for centralized `.sek` package output. | | `TZ` | `UTC` | Timezone | | `NAVSERVER` | — | Subsonic server URL (can also be set in the UI) | | `NAVUSER` | — | Subsonic username | | `NAVPASS` | — | Subsonic password | | `SLOPPAK_SOURCE_ROOT` | `/app/media/uploads` | Converter source root for relative PSARC paths | | `SLOPPAK_OUTPUT_ROOT` | `/app/media/sloppak` | Converter output root for generated `.sloppak` files | | `SLOPPAK_PSARC_UNPACK_CMD` | empty | Optional PSARC unpack command template (`{input}` and `{output}` placeholders) used for deep Rocksmith metadata extraction when python patcher module is unavailable | ### Model Directories These are set automatically in `docker-compose.yml` and point inside the `/opt/` volume. Override only if you need a custom layout. `BS_ROFORMER_MODEL_DIR`, `MEL_BAND_MODEL_DIR`, `MDXNET_MODEL_DIR`, `SCNET_MODEL_DIR`, `MEDLEYVOX_MODEL_DIR`, `APOLLO_MODEL_DIR`, `ASPIRATION_MODEL_DIR`, `BANDIT_MODEL_DIR` --- ## Project Folder Structure Each separation job creates a folder under `PROJECTS_DIR`: ``` --- ## sloppak Converter Deep Extraction The PSARC -> sloppak converter can enrich packages with Rocksmith metadata (arrangements, lyrics, WEM inventory) when a PSARC unpacker is available. Unpacker resolution order: 1. Python `patcher.unpack_psarc` module (if installed) 2. `SLOPPAK_PSARC_UNPACK_CMD` command template from environment Command template requirements: - Must include `{input}` and `{output}` placeholders. - Example: ```bash SLOPPAK_PSARC_UNPACK_CMD="/usr/local/bin/psarc unpack {input} {output}" ``` Runtime verification endpoint: - `GET /api/sloppak/capabilities` - Reports whether a usable unpacker is currently detected. Artist - Title/ Artist - Title.mp3 # Original source audio (copy) Artist - Title.elrc # ELRC lyrics with word-level timestamps (if aligned) Artist - Title.lrc # Plain LRC (if aligned) Stems/ vocals.mp3 drums.mp3 bass.mp3 other.mp3 Instrumental/ # Only when Instrumental preset selected Artist - Title - Instrumental.mp3 Static/ # Only when Static preset selected Artist - Title - No Vocals.mp3 Dynamic/ # Only when Dynamic preset selected Artist - Title - Dynamic Mix.mp3 ``` --- ## Tech Stack - **Frontend** — React 18, TypeScript, Vite, TailwindCSS - **Backend** — FastAPI, Python 3.11, SQLAlchemy - **Separation** — Demucs, BS-RoFormer, MelBand RoFormer, MDX-Net (onnxruntime-gpu), SCNet, MedleyVox, Apollo, Aspiration, Bandit - **Transcription** — basic-pitch (ONNX, CUDAExecutionProvider) - **Lyrics Sync** — wav2vec2-based forced alignment via custom `lsync` module - **Runtime** — NVIDIA CUDA 12.4 + cuDNN 9, PyTorch 2.6 (cu124 wheels) - **Container** — `nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04` --- ## Auto Pipeline + Lyric Engine Notes (2026-03) - Auto Pipeline now supports: - per-instrument refinement mapping (`instrument_models`) at separation stage - lyric engine selection (`WhisperX` or self-hosted `Parakeet`) in lyric steps - canonical stem finalization after separation (rename to project-style filenames + automatic removal of silent/no-significant-audio stems) - Saved auto-pipeline configs are migrated on read: - legacy `whisperModel` is upgraded to engine-aware `engine` + `model` - missing `instrumentModels` defaults to empty mapping ### Parakeet constraints Parakeet uses alternate handling and does **not** currently expose WhisperX-only controls: - `diarize` - `lsync_align` - `two_pass` - `explicit` - `translate` task If Parakeet is selected and unavailable, verify sidecar health and profile wiring: - backend must have `PARAKEET_ENABLED=true` - backend `PARAKEET_SIDECAR_URL` must point to the sidecar service - sidecar `GET /health` must return `200` --- ## WebUI routing (Phase 1) - The WebUI now uses URL-backed navigation for the primary screens: - `/` → separator/home - `/settings` → settings - `/projects/:projectId` → project detail - Refreshing on a project page now restores that same project view directly from the URL. - Sidebar project entries are real links, which is the foundation for browser-native new-tab and deep-link behavior. ## WebUI persistence and browser-tab behavior (Phases 2/3) - The WebUI now persists key client-side shell state across refreshes, including: - sidebar open/collapsed state, - sidebar project search text, - separator form context (input mode, selected model/format/preset, project name, imported file selection when still available). - Auto Pipeline now saves a lightweight draft for its project name and offers restore/discard when reopened. - Project completion surfaces now expose route-native links so projects can be opened directly or with normal browser new-tab behavior. ## WebUI PWA hardening (Phase 4) - The WebUI manifest is now hardened with explicit PWA metadata (`id`, `scope`, `display_override`, categories, and maskable/standard icon declarations). - Service worker caching is versioned and conservative: - shell assets are precached, - route navigations are network-first with shell fallback, - API responses are never cached as durable offline truth. - The shell now includes: - install affordance when `beforeinstallprompt` is available, - update notification with explicit “Reload to update” flow, - offline status messaging so disconnected behavior is clear. ## WebUI cross-tab session hygiene (Phase 5) - Cross-tab project deletion notifications are now propagated via `BroadcastChannel` with localStorage fallback. - Other open tabs now invalidate stale project views when a project is deleted and automatically leave deleted project routes. - Durable appearance/theme changes now propagate across tabs via storage-based synchronization. ## Frontend bundle split hardening - Route-level lazy loading is now used for major pages. - Vite chunk warning threshold is tuned for the split-route architecture to avoid false-positive size warnings during normal production builds. ## WebUI QoL closure (Phase 6) - Final acceptance checklist is tracked in [docs/WEBUI_QOL_QA_CHECKLIST.md](docs/WEBUI_QOL_QA_CHECKLIST.md). - Cross-tab events now cover project deletion, project invalidation, and project-list changes. - Automated browser smoke tests are available via Playwright (`cd frontend && npm run e2e`). --- ## GPU Notes The container requires the NVIDIA Container Toolkit. On the host: ```bash # Verify GPU is visible inside the container docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi ``` PyTorch wheels are self-contained (cu124) and do not require a matching CUDA toolkit on the host — only the driver. `STEMENGINE_GPU_VRAM_RESERVE_MB` is applied in backend GPU code paths (separation, lyric transcription, and basic-pitch startup) to preserve display headroom on WSL2. Omnizart sidecar now applies a TensorFlow memory cap using `OMNIZART_GPU_VRAM_RESERVE_MB` (or `STEMENGINE_GPU_VRAM_RESERVE_MB` fallback), so sidecar transcription follows the same reserve policy. --- ## Targeted backend tests (Findings 3/4/5 + optional hardening) To run the bounded jobs API + retention/cancel-state hardening regression suite with full backend dependencies, use the compose test profile: ```bash docker compose -f stemengine.yml --profile tests run --rm stemengine-tests ``` or the Make helper: ```bash make test-findings345 ``` This runs: - `backend.tests.test_jobs_api` - `backend.tests.test_project_cleanup` - `backend.tests.test_projects_delete` - `backend.tests.test_findings_3_4_unit` - `backend.tests.test_ensemble_separator` - `backend.tests.test_optional_hardening` - `backend.tests.test_stem_finalize` Test-only Python dependencies for this profile are declared in `backend/requirements-test.txt`. Use this when validating performance-fix regressions around long-run job history growth, retention pruning of old non-project jobs, in-memory cancellation-state cleanup, ensemble-memory accumulation behavior, and automatic stem finalization. --- ## RTX 3060 / Constrained-VRAM Stability ### Why MelBand Karaoke can OOM The karaoke checkpoints (`mel_band_karaoke`, `mel_band_karaoke_gabox`) load a ~1.7 GB PyTorch checkpoint and allocate tensors proportional to the **full track length** × `batch_size` × `num_overlap`. On a 6–8 GB GPU with WSL2 overhead, this easily exhausts VRAM on: - `performance` preset (`batch_size=8, num_overlap=4`) - tracks longer than 8 minutes - warm GPU with cached allocations from prior jobs ### MelBand stage diagnostics Set `MELBAND_STAGE_DIAGNOSTICS=true` to instrument the expensive setup stages that happen before the first visible separation segment begins. The backend now reports memory snapshots for these MelBand stages: - start - before audio load - after audio load - before model load - after model load - before demix/preallocation - after demix - after cleanup Each snapshot includes: - process RSS - current CUDA allocated memory - current CUDA reserved memory - peak CUDA allocated memory - peak CUDA reserved memory Example progress message: ```text [diag] MelBand lead/backing_instrumental: after model load — RSS=1840.2 MiB, CUDA alloc=1723.5 MiB, reserved=1824.0 MiB, peak_alloc=1723.5 MiB, peak_reserved=1824.0 MiB ``` This makes it much easier to tell whether the main spike came from whole-file audio decode, checkpoint load, or the full-length overlap-add buffers. ### Automatic VRAM headroom recovery Before each separation job, StemEngine now unloads any cached in-process Whisper models left behind by lyric transcription and flushes idle CUDA allocator blocks. This prevents an otherwise-idle transcription model from quietly reducing the VRAM available to BS-RoFormer / MelBand jobs. In addition, GPU separation jobs now run inside a **short-lived child Python process** by default. This keeps allocator fragmentation and CUDA fault state from accumulating indefinitely inside the main backend process after repeated heavy jobs or prior OOM events. You can disable separator child-process isolation only if needed for debugging: | Variable | Default | Effect | |---|---|---| | `STEMENGINE_SEPARATION_SUBPROCESS` | `true` | When `false`, run separator jobs in the main backend process instead of a short-lived child process | ### OOM ladder When a MelBand Karaoke job encounters an out-of-memory error the backend automatically retries with progressively safer parameters before giving up: | Attempt | batch_size | num_overlap | Notes | |---|---|---|---| | 1 | preset default | preset default | Normal run | | 2 | 1 | preset default | batch_size reduced | | 3 | 1 | 2 | num_overlap also reduced (eco) | | fallback | — | — | Cross-model to `mdxnet_karaoke` (ONNX) | The fallback step is governed by `FALLBACK_REQUIRE_LEAD_BACK` (see below). ### Lead/backing-capable fallback policy `FALLBACK_REQUIRE_LEAD_BACK=true` (default) ensures the cross-model fallback only ever uses a model that **also produces `lead` and `backing_instrumental` stems** — never a plain vocals/instrumental model. This prevents silent degradation of karaoke output quality. Models that qualify as lead/back-capable: | Model | Notes | |---|---| | `mel_band_karaoke` | MelBand primary — full PyTorch | | `mel_band_karaoke_gabox` | MelBand alternate checkpoint | | `bs_roformer_karaoke_anvuew` | BS-RoFormer karaoke variant | | `bs_roformer_karaoke_frazer_becruily` | BS-RoFormer karaoke variant | | `mdxnet_karaoke` | **ONNX — lowest VRAM (~200 MB), default fallback target** | If all fallback models also OOM, the job fails with a clear error message listing what was tried and actionable suggestions. Set `FALLBACK_REQUIRE_LEAD_BACK=false` to skip the cross-model fallback entirely and surface the OOM error immediately. ### Duration guardrail Tracks longer than `LONG_TRACK_DURATION_SECS` (default: **480 s = 8 min**) have MelBand eco params (`batch_size=1, num_overlap=2`) forced **before** inference, regardless of the active power preset. The guardrail fires for both `mel_band_karaoke` and `mel_band_roformer_4stem`. A log message is emitted when the guardrail activates so you can confirm it in the job progress stream. ### Recommended settings for RTX 3060 (12 GB WSL2) | Variable | Recommended | Reason | |---|---|---| | `STEMENGINE_GPU_VRAM_RESERVE_MB` | `2048` | Leaves headroom for Windows DWM/WDDM | | `LONG_TRACK_DURATION_SECS` | `480` | Force eco on tracks ≥ 8 min | | `FALLBACK_REQUIRE_LEAD_BACK` | `true` | Never lose lead/back distinction silently | | Power preset | `standard` or `eco` | Avoid `performance` on this GPU | ### Troubleshooting sequence 1. **Check job progress** — OOM retries and fallback activations are logged inline in the job progress field. 2. **Use `/api/system/gpu-reset` or Settings → GPU reset** — this clears cached transcription models and flushes CUDA state without a full restart. 3. **Restart the container** — if the GPU context is truly corrupted, process isolation reduces recurrence but a hard CUDA fault can still require a full restart. 4. **Use eco preset** — `batch_size=1, num_overlap=2` is the safest configuration. 5. **Split long tracks** — tracks > 8 min on this GPU should be split before separation. 6. **Check VRAM headroom** — on WSL2, run `nvidia-smi` on the host while idle and verify that ≥ 9 GB shows as free before submitting a karaoke job. ### RoFormer constructor compatibility note If you see an error such as: `TypeError: MelBandRoformer.__init__() got an unexpected keyword argument 'freqs_per_bands'` this usually means the model config was authored for a different `bs_roformer` package constructor signature. Current backend builds include a compatibility shim that: - remaps known renamed keys when possible, - drops unsupported constructor keys with a backend warning, - applies to MelBand and BS-RoFormer loaders. If the error persists, verify that the checkpoint and YAML config belong to the same model variant. If you see an error such as: `Error(s) in loading state_dict for MelBandRoformer: size mismatch for ...` the checkpoint and config likely describe different architectures/revisions. The backend now surfaces checkpoint/config filenames in this failure path to speed triage. Note: the core `mel_band_karaoke_gabox` checkpoint is sourced from a `bsroformers/` path and is handled with BS-RoFormer architecture loading to avoid MelBand class mismatch. --- ## GaboxR67 model expansion (implemented rollout) Source scope: `https://huggingface.co/GaboxR67` (primarily `GaboxR67/MelBandRoformers`). ### Does this include small models? Yes. Small models are implemented as an explicit opt-in tier controlled by `GABOX_ENABLE_SMALL_MODELS`. ### Rollout tiers | Tier | Target | Purpose | |---|---|---| | Tier A (core) | Stable Gabox MelBand models (`mel_band_vocals_gabox`) | Quality-first default additions | | Tier B (small) | `mel_band_karaoke_gabox_small` | Faster/safer runs on constrained GPUs (opt-in) | | Tier C (experimental) | `mel_band_karaoke_gabox_v2_exp` | Opt-in only; no default fallback usage | ### Integration boundaries - Reused existing MelBand specialized runner architecture. - Added model IDs + labels to backend model catalog. - Added downloader mappings for Gabox checkpoints/config files. - Small/experimental tiers are hidden unless enabled by env controls. - Small karaoke model has an additional duration guardrail using `GABOX_SMALL_MODEL_MAX_DURATION_SECS`. ### Current behavior notes - `mel_band_vocals_gabox` is always available. - `mel_band_karaoke_gabox_small` appears only when `GABOX_ENABLE_SMALL_MODELS=true`. - `mel_band_karaoke_gabox_v2_exp` appears only when `GABOX_ENABLE_EXPERIMENTAL_MODELS=true`. - MelBand diagnostics and OOM ladder remain active for Gabox variants.