Files
stemengine_old/README.md
T
2026-04-06 14:49:06 +09:00

21 KiB
Executable File
Raw Blame History

StemEngine - Karaoke Anything?

This is a vibe-coded project based on the work of people who actually know what they're doing. But now I can theoretically karaoke anything and so can you.


Documentation


Features

  • Stem Separation — 40+ models across 10 model families
  • MIDI Transcription — basic-pitch with polyphonic and melody modes
  • GPU Safety Limits — shared VRAM reserve for separation, lyric transcription, basic-pitch, and Omnizart sidecar
  • ABC Notation — convert MIDI to ABC notation for sheet music
  • Lyrics Sync — forced-alignment to produce ELRC files with word-level timestamps and a live animated preview
  • Project Management — persistent project tabs, per-stem reprocessing, stem renaming
  • Mix Presets — Instrumental, Static (remove elements), Dynamic (per-stem volume)
  • Subsonic Integration — browse and import from a Subsonic-compatible server
  • Model Downloader — download model weights on-demand from the Settings UI

Separation Models

Family Models
Demucs v4, v4 Fine-tuned, v3 Extra Quantized, v3
BS-RoFormer 4-stem, 5-stem (Guitar), 5-stem (Piano), 6-stem, Chorus, Anvuew, Karaoke Anvuew, Karaoke Frazer+Becruily, Mamba2 Vocals (Adapter), SW 6-stem
MelBand RoFormer 4-stem, Vocals, Vocals Anvuew, Vocals Gabox, Dereverb, Dereverb Anvuew, Dereverb Room, Karaoke, Karaoke Gabox, Karaoke Gabox (Small, opt-in), Karaoke Gabox V2 Experimental (opt-in), Deux Becruily
MDX-Net Vocals FT, Inst HQ 1, Inst HQ 2, Karaoke, Dereverb, Drum Sep
SCNet XL (2-stem), XL IHF, Masked XL IHF, Masked Small, Tran Small, Large (4-stem), Large Starrytong
MedleyVox Base, 55/66/88/100 Epoch variants
Apollo Vocals, EDM, EDM Big
Aspiration Full, Light
Bandit v2 Multi-Language, English
Ensemble Vocal Best Quality, 4-Stem, 6-Stem Full

Model weights are downloaded on first use and cached in /opt/.

Checkpoint formats (.ckpt vs .safetensors)

StemEngine's current BS-RoFormer / MelBand runtime primarily expects a model-specific config YAML plus a PyTorch checkpoint file such as .ckpt.

What safetensors means:

  • it is a safer weight serialization format,
  • it does not by itself change the MelBand / BS-RoFormer architecture,
  • it does not materially reduce inference VRAM or compute cost,
  • it can slightly improve load safety and sometimes startup/load efficiency.

Practical expectation in StemEngine:

  • a safetensors conversion is usually the same trained model packaged differently,
  • current built-in model loaders are wired for .ckpt + YAML style checkpoints,
  • direct safetensors support would require loader support for safetensors deserialization and any key-mapping needed by the exported weights.

Requirements

  • Docker with the NVIDIA Container Toolkit (for GPU acceleration)
  • NVIDIA GPU with CUDA 12.4 support
  • Docker Compose

CPU-only mode is available via CPU_SEPARATION=true but separation will be significantly slower.


Quick Start

  1. Copy .env.example to .env and fill in the required values (see Configuration below).

  2. Build and start:

docker compose up -d --build
  1. Open http://localhost:8000 in your browser.

Optional WebUI (Vite dev server) default host URL is http://localhost:5174.


Configuration

All configuration is via environment variables. The docker-compose.yml accepts these via a .env file in the same directory.

Required

Variable Description
SEFOLDER Host path for persistent runtime data (uploads, jobs, DB, LRC sync)
ENGINEFOLDER Host path for model weight caches
PROJECTS Host path for separated project output
SECRET_KEY Random hex key for encrypting Subsonic credentials at rest. Generate with: python3 -c "import secrets; print(secrets.token_hex(32))"

Optional

Variable Default Description
CPU_SEPARATION false Set to true to force CPU inference
MAX_UPLOAD_MB 500 Maximum upload file size in MB
STEMENGINE_GPU_VRAM_RESERVE_MB 2048 Keeps this many MB free on the GPU (WSL2-safe headroom)
OMNIZART_GPU_VRAM_RESERVE_MB STEMENGINE_GPU_VRAM_RESERVE_MB Optional sidecar override for Omnizart TensorFlow GPU cap
STEMENGINE_UVICORN_RELOAD_FLAG empty Set to --reload only when actively developing backend code; leave empty for long-running separation jobs
WEBUI_HOST_PORT 5174 Host port bound to the WebUI container (set if 5174 or 5173 is already in use)
WEBUI_PORT 5173 Internal WebUI container port (normally unchanged)
MELBAND_STAGE_DIAGNOSTICS false Emit per-stage MelBand memory snapshots into job progress + backend logs
GABOX_ENABLE_SMALL_MODELS false Expose Tier B small Gabox models (for constrained VRAM)
GABOX_ENABLE_EXPERIMENTAL_MODELS false Expose Tier C experimental Gabox models
GABOX_SMALL_MODEL_MAX_DURATION_SECS 360 Guardrail threshold used by Gabox small karaoke model
GABOX_SMALL_MODEL_PREFERRED_FOR_LOW_VRAM false Reserved policy flag for low-VRAM auto-preference workflows
STEMENGINE_LYRIC_SEGMENTER auto Lyric line splitter strategy: auto, karaoke_processor, legacy
STEMENGINE_LYRIC_MAX_LINE_LENGTH 36 Max line length target used by karaoke lyric segmentation
STEMENGINE_KARAOKE_EXPORT_DIR empty Optional global .sek export directory inside the container. If empty, exports go under each project (Exports/StemEngineKaraoke).
SEK_EXPORTS ${SEFOLDER}/sek_exports Host path mounted to /app/media/sek_exports for centralized .sek package output.
TZ UTC Timezone
NAVSERVER — Subsonic server URL (can also be set in the UI)
NAVUSER — Subsonic username
NAVPASS — Subsonic password

Model Directories

These are set automatically in docker-compose.yml and point inside the /opt/ volume. Override only if you need a custom layout.

BS_ROFORMER_MODEL_DIR, MEL_BAND_MODEL_DIR, MDXNET_MODEL_DIR, SCNET_MODEL_DIR, MEDLEYVOX_MODEL_DIR, APOLLO_MODEL_DIR, ASPIRATION_MODEL_DIR, BANDIT_MODEL_DIR


Project Folder Structure

Each separation job creates a folder under PROJECTS_DIR:

Artist - Title/
  Artist - Title.mp3        # Original source audio (copy)
  Artist - Title.elrc       # ELRC lyrics with word-level timestamps (if aligned)
  Artist - Title.lrc        # Plain LRC (if aligned)
  Stems/
    vocals.mp3
    drums.mp3
    bass.mp3
    other.mp3
  Instrumental/             # Only when Instrumental preset selected
    Artist - Title - Instrumental.mp3
  Static/                   # Only when Static preset selected
    Artist - Title - No Vocals.mp3
  Dynamic/                  # Only when Dynamic preset selected
    Artist - Title - Dynamic Mix.mp3

Tech Stack

  • Frontend — React 18, TypeScript, Vite, TailwindCSS
  • Backend — FastAPI, Python 3.11, SQLAlchemy
  • Separation — Demucs, BS-RoFormer, MelBand RoFormer, MDX-Net (onnxruntime-gpu), SCNet, MedleyVox, Apollo, Aspiration, Bandit
  • Transcription — basic-pitch (ONNX, CUDAExecutionProvider)
  • Lyrics Sync — wav2vec2-based forced alignment via custom lsync module
  • Runtime — NVIDIA CUDA 12.4 + cuDNN 9, PyTorch 2.6 (cu124 wheels)
  • Container — nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04

Auto Pipeline + Lyric Engine Notes (2026-03)

  • Auto Pipeline now supports:
    • per-instrument refinement mapping (instrument_models) at separation stage
    • lyric engine selection (WhisperX or self-hosted Parakeet) in lyric steps
    • canonical stem finalization after separation (rename to project-style filenames + automatic removal of silent/no-significant-audio stems)
  • Saved auto-pipeline configs are migrated on read:
    • legacy whisperModel is upgraded to engine-aware engine + model
    • missing instrumentModels defaults to empty mapping

Parakeet constraints

Parakeet uses alternate handling and does not currently expose WhisperX-only controls:

  • diarize
  • lsync_align
  • two_pass
  • explicit
  • translate task

If Parakeet is selected and unavailable, verify sidecar health and profile wiring:

  • backend must have PARAKEET_ENABLED=true
  • backend PARAKEET_SIDECAR_URL must point to the sidecar service
  • sidecar GET /health must return 200

WebUI routing (Phase 1)

  • The WebUI now uses URL-backed navigation for the primary screens:
    • / → separator/home
    • /settings → settings
    • /projects/:projectId → project detail
  • Refreshing on a project page now restores that same project view directly from the URL.
  • Sidebar project entries are real links, which is the foundation for browser-native new-tab and deep-link behavior.

WebUI persistence and browser-tab behavior (Phases 2/3)

  • The WebUI now persists key client-side shell state across refreshes, including:
    • sidebar open/collapsed state,
    • sidebar project search text,
    • separator form context (input mode, selected model/format/preset, project name, imported file selection when still available).
  • Auto Pipeline now saves a lightweight draft for its project name and offers restore/discard when reopened.
  • Project completion surfaces now expose route-native links so projects can be opened directly or with normal browser new-tab behavior.

WebUI PWA hardening (Phase 4)

  • The WebUI manifest is now hardened with explicit PWA metadata (id, scope, display_override, categories, and maskable/standard icon declarations).
  • Service worker caching is versioned and conservative:
    • shell assets are precached,
    • route navigations are network-first with shell fallback,
    • API responses are never cached as durable offline truth.
  • The shell now includes:
    • install affordance when beforeinstallprompt is available,
    • update notification with explicit “Reload to update” flow,
    • offline status messaging so disconnected behavior is clear.

WebUI cross-tab session hygiene (Phase 5)

  • Cross-tab project deletion notifications are now propagated via BroadcastChannel with localStorage fallback.
  • Other open tabs now invalidate stale project views when a project is deleted and automatically leave deleted project routes.
  • Durable appearance/theme changes now propagate across tabs via storage-based synchronization.

Frontend bundle split hardening

  • Route-level lazy loading is now used for major pages.
  • Vite chunk warning threshold is tuned for the split-route architecture to avoid false-positive size warnings during normal production builds.

WebUI QoL closure (Phase 6)

  • Final acceptance checklist is tracked in docs/WEBUI_QOL_QA_CHECKLIST.md.
  • Cross-tab events now cover project deletion, project invalidation, and project-list changes.
  • Automated browser smoke tests are available via Playwright (cd frontend && npm run e2e).

GPU Notes

The container requires the NVIDIA Container Toolkit. On the host:

# Verify GPU is visible inside the container
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

PyTorch wheels are self-contained (cu124) and do not require a matching CUDA toolkit on the host — only the driver.

STEMENGINE_GPU_VRAM_RESERVE_MB is applied in backend GPU code paths (separation, lyric transcription, and basic-pitch startup) to preserve display headroom on WSL2.

Omnizart sidecar now applies a TensorFlow memory cap using OMNIZART_GPU_VRAM_RESERVE_MB (or STEMENGINE_GPU_VRAM_RESERVE_MB fallback), so sidecar transcription follows the same reserve policy.


Targeted backend tests (Findings 3/4/5 + optional hardening)

To run the bounded jobs API + retention/cancel-state hardening regression suite with full backend dependencies, use the compose test profile:

docker compose -f stemengine.yml --profile tests run --rm stemengine-tests

or the Make helper:

make test-findings345

This runs:

  • backend.tests.test_jobs_api
  • backend.tests.test_project_cleanup
  • backend.tests.test_projects_delete
  • backend.tests.test_findings_3_4_unit
  • backend.tests.test_ensemble_separator
  • backend.tests.test_optional_hardening
  • backend.tests.test_stem_finalize

Test-only Python dependencies for this profile are declared in backend/requirements-test.txt.

Use this when validating performance-fix regressions around long-run job history growth, retention pruning of old non-project jobs, in-memory cancellation-state cleanup, ensemble-memory accumulation behavior, and automatic stem finalization.


RTX 3060 / Constrained-VRAM Stability

Why MelBand Karaoke can OOM

The karaoke checkpoints (mel_band_karaoke, mel_band_karaoke_gabox) load a ~1.7 GB PyTorch checkpoint and allocate tensors proportional to the full track length × batch_size × num_overlap. On a 6–8 GB GPU with WSL2 overhead, this easily exhausts VRAM on:

  • performance preset (batch_size=8, num_overlap=4)
  • tracks longer than 8 minutes
  • warm GPU with cached allocations from prior jobs

MelBand stage diagnostics

Set MELBAND_STAGE_DIAGNOSTICS=true to instrument the expensive setup stages that happen before the first visible separation segment begins.

The backend now reports memory snapshots for these MelBand stages:

  • start
  • before audio load
  • after audio load
  • before model load
  • after model load
  • before demix/preallocation
  • after demix
  • after cleanup

Each snapshot includes:

  • process RSS
  • current CUDA allocated memory
  • current CUDA reserved memory
  • peak CUDA allocated memory
  • peak CUDA reserved memory

Example progress message:

[diag] MelBand lead/backing_instrumental: after model load — RSS=1840.2 MiB, CUDA alloc=1723.5 MiB, reserved=1824.0 MiB, peak_alloc=1723.5 MiB, peak_reserved=1824.0 MiB

This makes it much easier to tell whether the main spike came from whole-file audio decode, checkpoint load, or the full-length overlap-add buffers.

Automatic VRAM headroom recovery

Before each separation job, StemEngine now unloads any cached in-process Whisper models left behind by lyric transcription and flushes idle CUDA allocator blocks. This prevents an otherwise-idle transcription model from quietly reducing the VRAM available to BS-RoFormer / MelBand jobs.

In addition, GPU separation jobs now run inside a short-lived child Python process by default. This keeps allocator fragmentation and CUDA fault state from accumulating indefinitely inside the main backend process after repeated heavy jobs or prior OOM events.

You can disable separator child-process isolation only if needed for debugging:

Variable Default Effect
STEMENGINE_SEPARATION_SUBPROCESS true When false, run separator jobs in the main backend process instead of a short-lived child process

OOM ladder

When a MelBand Karaoke job encounters an out-of-memory error the backend automatically retries with progressively safer parameters before giving up:

Attempt batch_size num_overlap Notes
1 preset default preset default Normal run
2 1 preset default batch_size reduced
3 1 2 num_overlap also reduced (eco)
fallback — — Cross-model to mdxnet_karaoke (ONNX)

The fallback step is governed by FALLBACK_REQUIRE_LEAD_BACK (see below).

Lead/backing-capable fallback policy

FALLBACK_REQUIRE_LEAD_BACK=true (default) ensures the cross-model fallback only ever uses a model that also produces lead and backing_instrumental stems — never a plain vocals/instrumental model. This prevents silent degradation of karaoke output quality.

Models that qualify as lead/back-capable:

Model Notes
mel_band_karaoke MelBand primary — full PyTorch
mel_band_karaoke_gabox MelBand alternate checkpoint
bs_roformer_karaoke_anvuew BS-RoFormer karaoke variant
bs_roformer_karaoke_frazer_becruily BS-RoFormer karaoke variant
mdxnet_karaoke ONNX — lowest VRAM (~200 MB), default fallback target

If all fallback models also OOM, the job fails with a clear error message listing what was tried and actionable suggestions.

Set FALLBACK_REQUIRE_LEAD_BACK=false to skip the cross-model fallback entirely and surface the OOM error immediately.

Duration guardrail

Tracks longer than LONG_TRACK_DURATION_SECS (default: 480 s = 8 min) have MelBand eco params (batch_size=1, num_overlap=2) forced before inference, regardless of the active power preset. The guardrail fires for both mel_band_karaoke and mel_band_roformer_4stem.

A log message is emitted when the guardrail activates so you can confirm it in the job progress stream.

Variable Recommended Reason
STEMENGINE_GPU_VRAM_RESERVE_MB 2048 Leaves headroom for Windows DWM/WDDM
LONG_TRACK_DURATION_SECS 480 Force eco on tracks ≥ 8 min
FALLBACK_REQUIRE_LEAD_BACK true Never lose lead/back distinction silently
Power preset standard or eco Avoid performance on this GPU

Troubleshooting sequence

  1. Check job progress — OOM retries and fallback activations are logged inline in the job progress field.
  2. Use /api/system/gpu-reset or Settings → GPU reset — this clears cached transcription models and flushes CUDA state without a full restart.
  3. Restart the container — if the GPU context is truly corrupted, process isolation reduces recurrence but a hard CUDA fault can still require a full restart.
  4. Use eco preset — batch_size=1, num_overlap=2 is the safest configuration.
  5. Split long tracks — tracks > 8 min on this GPU should be split before separation.
  6. Check VRAM headroom — on WSL2, run nvidia-smi on the host while idle and verify that ≥ 9 GB shows as free before submitting a karaoke job.

RoFormer constructor compatibility note

If you see an error such as:

TypeError: MelBandRoformer.__init__() got an unexpected keyword argument 'freqs_per_bands'

this usually means the model config was authored for a different bs_roformer package constructor signature.

Current backend builds include a compatibility shim that:

  • remaps known renamed keys when possible,
  • drops unsupported constructor keys with a backend warning,
  • applies to MelBand and BS-RoFormer loaders.

If the error persists, verify that the checkpoint and YAML config belong to the same model variant.

If you see an error such as:

Error(s) in loading state_dict for MelBandRoformer: size mismatch for ...

the checkpoint and config likely describe different architectures/revisions. The backend now surfaces checkpoint/config filenames in this failure path to speed triage.

Note: the core mel_band_karaoke_gabox checkpoint is sourced from a bsroformers/ path and is handled with BS-RoFormer architecture loading to avoid MelBand class mismatch.


GaboxR67 model expansion (implemented rollout)

Source scope: https://huggingface.co/GaboxR67 (primarily GaboxR67/MelBandRoformers).

Does this include small models?

Yes. Small models are implemented as an explicit opt-in tier controlled by GABOX_ENABLE_SMALL_MODELS.

Rollout tiers

Tier Target Purpose
Tier A (core) Stable Gabox MelBand models (mel_band_vocals_gabox) Quality-first default additions
Tier B (small) mel_band_karaoke_gabox_small Faster/safer runs on constrained GPUs (opt-in)
Tier C (experimental) mel_band_karaoke_gabox_v2_exp Opt-in only; no default fallback usage

Integration boundaries

  • Reused existing MelBand specialized runner architecture.
  • Added model IDs + labels to backend model catalog.
  • Added downloader mappings for Gabox checkpoints/config files.
  • Small/experimental tiers are hidden unless enabled by env controls.
  • Small karaoke model has an additional duration guardrail using GABOX_SMALL_MODEL_MAX_DURATION_SECS.

Current behavior notes

  • mel_band_vocals_gabox is always available.
  • mel_band_karaoke_gabox_small appears only when GABOX_ENABLE_SMALL_MODELS=true.
  • mel_band_karaoke_gabox_v2_exp appears only when GABOX_ENABLE_EXPERIMENTAL_MODELS=true.
  • MelBand diagnostics and OOM ladder remain active for Gabox variants.