Files
2026-07-09 10:58:09 +09:00

496 lines
22 KiB
Markdown
Executable File
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# StemEngine - Karaoke Anything?
> This is a **vibe-coded** project based on the work of people who actually know what they're doing. But now I can theoretically karaoke anything and so can you.
---
## Documentation
- User docs: [docs/USER_GUIDE.md](docs/USER_GUIDE.md)
- Developer docs: [docs/DEVELOPER_GUIDE.md](docs/DEVELOPER_GUIDE.md)
- Documentation plan: [docs/DOCUMENTATION_PLAN.md](docs/DOCUMENTATION_PLAN.md)
- API reference: [docs/API_REFERENCE.md](docs/API_REFERENCE.md)
- Troubleshooting matrix: [docs/TROUBLESHOOTING_MATRIX.md](docs/TROUBLESHOOTING_MATRIX.md)
- Configuration reference: [docs/CONFIGURATION_REFERENCE.md](docs/CONFIGURATION_REFERENCE.md)
- UVR comparison: [docs/UVR_VS_STEMENGINE.md](docs/UVR_VS_STEMENGINE.md)
- Long-track chunking plan: [docs/LONG_TRACK_CHUNKING_PLAN.md](docs/LONG_TRACK_CHUNKING_PLAN.md)
- Documentation governance: [docs/DOC_GOVERNANCE.md](docs/DOC_GOVERNANCE.md)
- Release checklist: [docs/RELEASE_CHECKLIST.md](docs/RELEASE_CHECKLIST.md)
---
## Features
- **Stem Separation** — 40+ models across 10 model families
- **MIDI Transcription** — basic-pitch with polyphonic and melody modes
- **GPU Safety Limits** — shared VRAM reserve for separation, lyric transcription, basic-pitch, and Omnizart sidecar
- **ABC Notation** — convert MIDI to ABC notation for sheet music
- **Lyrics Sync** — forced-alignment to produce ELRC files with word-level timestamps and a live animated preview
- **Project Management** — persistent project tabs, per-stem reprocessing, stem renaming
- **Mix Presets** — Instrumental, Static (remove elements), Dynamic (per-stem volume)
- **Subsonic Integration** — browse and import from a Subsonic-compatible server
- **Model Downloader** — download model weights on-demand from the Settings UI
---
## Separation Models
| Family | Models |
|---|---|
| **Demucs** | v4, v4 Fine-tuned, v3 Extra Quantized, v3 |
| **BS-RoFormer** | 4-stem, 5-stem (Guitar), 5-stem (Piano), 6-stem, Chorus, Anvuew, Karaoke Anvuew, Karaoke Frazer+Becruily, Mamba2 Vocals (Adapter), SW 6-stem |
| **MelBand RoFormer** | 4-stem, Vocals, Vocals Anvuew, Vocals Gabox, Dereverb, Dereverb Anvuew, Dereverb Room, Karaoke, Karaoke Gabox, Karaoke Gabox (Small, opt-in), Karaoke Gabox V2 Experimental (opt-in), Deux Becruily |
| **MDX-Net** | Vocals FT, Inst HQ 1, Inst HQ 2, Karaoke, Dereverb, Drum Sep |
| **SCNet** | XL (2-stem), XL IHF, Masked XL IHF, Masked Small, Tran Small, Large (4-stem), Large Starrytong |
| **MedleyVox** | Base, 55/66/88/100 Epoch variants |
| **Apollo** | Vocals, EDM, EDM Big |
| **Aspiration** | Full, Light |
| **Bandit v2** | Multi-Language, English |
| **Ensemble** | Vocal Best Quality, 4-Stem, 6-Stem Full |
Model weights are downloaded on first use and cached in `/opt/`.
### Checkpoint formats (`.ckpt` vs `.safetensors`)
StemEngine's current BS-RoFormer / MelBand runtime primarily expects a model-specific config YAML plus a PyTorch checkpoint file such as `.ckpt`.
What `safetensors` means:
- it is a safer weight serialization format,
- it does **not** by itself change the MelBand / BS-RoFormer architecture,
- it does **not** materially reduce inference VRAM or compute cost,
- it can slightly improve load safety and sometimes startup/load efficiency.
Practical expectation in StemEngine:
- a `safetensors` conversion is usually the same trained model packaged differently,
- current built-in model loaders are wired for `.ckpt` + YAML style checkpoints,
- direct `safetensors` support would require loader support for `safetensors` deserialization and any key-mapping needed by the exported weights.
---
## Requirements
- Docker with the NVIDIA Container Toolkit (for GPU acceleration)
- NVIDIA GPU with CUDA 12.4 support
- Docker Compose
CPU-only mode is available via `CPU_SEPARATION=true` but separation will be significantly slower.
---
## Quick Start
1. Copy `.env.example` to `.env` and fill in the required values (see Configuration below).
2. Build and start:
```bash
docker compose up -d --build
```
3. Open `http://localhost:8000` in your browser.
Optional WebUI (Vite dev server) default host URL is `http://localhost:5174`.
---
## Configuration
All configuration is via environment variables. The `docker-compose.yml` accepts these via a `.env` file in the same directory.
### Required
| Variable | Description |
|---|---|
| `SEFOLDER` | Host path for persistent runtime data (uploads, jobs, DB, LRC sync) |
| `ENGINEFOLDER` | Host path for model weight caches |
| `PROJECTS` | Host path for separated project output |
| `SECRET_KEY` | Random hex key for encrypting Subsonic credentials at rest. Generate with: `python3 -c "import secrets; print(secrets.token_hex(32))"` |
### Optional
| Variable | Default | Description |
|---|---|---|
| `CPU_SEPARATION` | `false` | Set to `true` to force CPU inference |
| `MAX_UPLOAD_MB` | `500` | Maximum upload file size in MB |
| `STEMENGINE_GPU_VRAM_RESERVE_MB` | `2048` | Keeps this many MB free on the GPU (WSL2-safe headroom) |
| `OMNIZART_GPU_VRAM_RESERVE_MB` | `STEMENGINE_GPU_VRAM_RESERVE_MB` | Optional sidecar override for Omnizart TensorFlow GPU cap |
| `STEMENGINE_UVICORN_RELOAD_FLAG` | empty | Set to `--reload` only when actively developing backend code; leave empty for long-running separation jobs |
| `WEBUI_HOST_PORT` | `5174` | Host port bound to the WebUI container (set if `5174` or `5173` is already in use) |
| `WEBUI_PORT` | `5173` | Internal WebUI container port (normally unchanged) |
| `MELBAND_STAGE_DIAGNOSTICS` | `false` | Emit per-stage MelBand memory snapshots into job progress + backend logs |
| `GABOX_ENABLE_SMALL_MODELS` | `false` | Expose Tier B small Gabox models (for constrained VRAM) |
| `GABOX_ENABLE_EXPERIMENTAL_MODELS` | `false` | Expose Tier C experimental Gabox models |
| `GABOX_SMALL_MODEL_MAX_DURATION_SECS` | `360` | Guardrail threshold used by Gabox small karaoke model |
| `GABOX_SMALL_MODEL_PREFERRED_FOR_LOW_VRAM` | `false` | Reserved policy flag for low-VRAM auto-preference workflows |
| `STEMENGINE_LYRIC_SEGMENTER` | `auto` | Lyric line splitter strategy: `auto`, `karaoke_processor`, `legacy` |
| `STEMENGINE_LYRIC_MAX_LINE_LENGTH` | `36` | Max line length target used by karaoke lyric segmentation |
| `STEMENGINE_KARAOKE_EXPORT_DIR` | empty | Optional global `.sek` export directory inside the container. If empty, exports go under each project (`Exports/StemEngineKaraoke`). |
| `SEK_EXPORTS` | `${SEFOLDER}/sek_exports` | Host path mounted to `/app/media/sek_exports` for centralized `.sek` package output. |
| `TZ` | `UTC` | Timezone |
| `NAVSERVER` | — | Subsonic server URL (can also be set in the UI) |
| `NAVUSER` | — | Subsonic username |
| `NAVPASS` | — | Subsonic password |
| `SLOPPAK_SOURCE_ROOT` | `/app/media/uploads` | Converter source root for relative PSARC paths |
| `SLOPPAK_OUTPUT_ROOT` | `/app/media/sloppak` | Converter output root for generated `.sloppak` files |
| `SLOPPAK_PSARC_UNPACK_CMD` | empty | Optional PSARC unpack command template (`{input}` and `{output}` placeholders) used for deep Rocksmith metadata extraction when python patcher module is unavailable |
### Model Directories
These are set automatically in `docker-compose.yml` and point inside the `/opt/` volume. Override only if you need a custom layout.
`BS_ROFORMER_MODEL_DIR`, `MEL_BAND_MODEL_DIR`, `MDXNET_MODEL_DIR`, `SCNET_MODEL_DIR`, `MEDLEYVOX_MODEL_DIR`, `APOLLO_MODEL_DIR`, `ASPIRATION_MODEL_DIR`, `BANDIT_MODEL_DIR`
---
## Project Folder Structure
Each separation job creates a folder under `PROJECTS_DIR`:
```
---
## sloppak Converter Deep Extraction
The PSARC -> sloppak converter can enrich packages with Rocksmith metadata (arrangements, lyrics, WEM inventory) when a PSARC unpacker is available.
Unpacker resolution order:
1. Python `patcher.unpack_psarc` module (if installed)
2. `SLOPPAK_PSARC_UNPACK_CMD` command template from environment
Command template requirements:
- Must include `{input}` and `{output}` placeholders.
- Example:
```bash
SLOPPAK_PSARC_UNPACK_CMD="/usr/local/bin/psarc unpack {input} {output}"
```
Runtime verification endpoint:
- `GET /api/sloppak/capabilities`
- Reports whether a usable unpacker is currently detected.
Artist - Title/
Artist - Title.mp3 # Original source audio (copy)
Artist - Title.elrc # ELRC lyrics with word-level timestamps (if aligned)
Artist - Title.lrc # Plain LRC (if aligned)
Stems/
vocals.mp3
drums.mp3
bass.mp3
other.mp3
Instrumental/ # Only when Instrumental preset selected
Artist - Title - Instrumental.mp3
Static/ # Only when Static preset selected
Artist - Title - No Vocals.mp3
Dynamic/ # Only when Dynamic preset selected
Artist - Title - Dynamic Mix.mp3
```
---
## Tech Stack
- **Frontend** — React 18, TypeScript, Vite, TailwindCSS
- **Backend** — FastAPI, Python 3.11, SQLAlchemy
- **Separation** — Demucs, BS-RoFormer, MelBand RoFormer, MDX-Net (onnxruntime-gpu), SCNet, MedleyVox, Apollo, Aspiration, Bandit
- **Transcription** — basic-pitch (ONNX, CUDAExecutionProvider)
- **Lyrics Sync** — wav2vec2-based forced alignment via custom `lsync` module
- **Runtime** — NVIDIA CUDA 12.4 + cuDNN 9, PyTorch 2.6 (cu124 wheels)
- **Container** — `nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04`
---
## Auto Pipeline + Lyric Engine Notes (2026-03)
- Auto Pipeline now supports:
- per-instrument refinement mapping (`instrument_models`) at separation stage
- lyric engine selection (`WhisperX` or self-hosted `Parakeet`) in lyric steps
- canonical stem finalization after separation (rename to project-style filenames + automatic removal of silent/no-significant-audio stems)
- Saved auto-pipeline configs are migrated on read:
- legacy `whisperModel` is upgraded to engine-aware `engine` + `model`
- missing `instrumentModels` defaults to empty mapping
### Parakeet constraints
Parakeet uses alternate handling and does **not** currently expose WhisperX-only controls:
- `diarize`
- `lsync_align`
- `two_pass`
- `explicit`
- `translate` task
If Parakeet is selected and unavailable, verify sidecar health and profile wiring:
- backend must have `PARAKEET_ENABLED=true`
- backend `PARAKEET_SIDECAR_URL` must point to the sidecar service
- sidecar `GET /health` must return `200`
---
## WebUI routing (Phase 1)
- The WebUI now uses URL-backed navigation for the primary screens:
- `/` → separator/home
- `/settings` → settings
- `/projects/:projectId` → project detail
- Refreshing on a project page now restores that same project view directly from the URL.
- Sidebar project entries are real links, which is the foundation for browser-native new-tab and deep-link behavior.
## WebUI persistence and browser-tab behavior (Phases 2/3)
- The WebUI now persists key client-side shell state across refreshes, including:
- sidebar open/collapsed state,
- sidebar project search text,
- separator form context (input mode, selected model/format/preset, project name, imported file selection when still available).
- Auto Pipeline now saves a lightweight draft for its project name and offers restore/discard when reopened.
- Project completion surfaces now expose route-native links so projects can be opened directly or with normal browser new-tab behavior.
## WebUI PWA hardening (Phase 4)
- The WebUI manifest is now hardened with explicit PWA metadata (`id`, `scope`, `display_override`, categories, and maskable/standard icon declarations).
- Service worker caching is versioned and conservative:
- shell assets are precached,
- route navigations are network-first with shell fallback,
- API responses are never cached as durable offline truth.
- The shell now includes:
- install affordance when `beforeinstallprompt` is available,
- update notification with explicit “Reload to update” flow,
- offline status messaging so disconnected behavior is clear.
## WebUI cross-tab session hygiene (Phase 5)
- Cross-tab project deletion notifications are now propagated via `BroadcastChannel` with localStorage fallback.
- Other open tabs now invalidate stale project views when a project is deleted and automatically leave deleted project routes.
- Durable appearance/theme changes now propagate across tabs via storage-based synchronization.
## Frontend bundle split hardening
- Route-level lazy loading is now used for major pages.
- Vite chunk warning threshold is tuned for the split-route architecture to avoid false-positive size warnings during normal production builds.
## WebUI QoL closure (Phase 6)
- Final acceptance checklist is tracked in [docs/WEBUI_QOL_QA_CHECKLIST.md](docs/WEBUI_QOL_QA_CHECKLIST.md).
- Cross-tab events now cover project deletion, project invalidation, and project-list changes.
- Automated browser smoke tests are available via Playwright (`cd frontend && npm run e2e`).
---
## GPU Notes
The container requires the NVIDIA Container Toolkit. On the host:
```bash
# Verify GPU is visible inside the container
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
```
PyTorch wheels are self-contained (cu124) and do not require a matching CUDA toolkit on the host — only the driver.
`STEMENGINE_GPU_VRAM_RESERVE_MB` is applied in backend GPU code paths (separation, lyric transcription, and basic-pitch startup) to preserve display headroom on WSL2.
Omnizart sidecar now applies a TensorFlow memory cap using `OMNIZART_GPU_VRAM_RESERVE_MB` (or `STEMENGINE_GPU_VRAM_RESERVE_MB` fallback), so sidecar transcription follows the same reserve policy.
---
## Targeted backend tests (Findings 3/4/5 + optional hardening)
To run the bounded jobs API + retention/cancel-state hardening regression suite with full backend dependencies, use the compose test profile:
```bash
docker compose -f stemengine.yml --profile tests run --rm stemengine-tests
```
or the Make helper:
```bash
make test-findings345
```
This runs:
- `backend.tests.test_jobs_api`
- `backend.tests.test_project_cleanup`
- `backend.tests.test_projects_delete`
- `backend.tests.test_findings_3_4_unit`
- `backend.tests.test_ensemble_separator`
- `backend.tests.test_optional_hardening`
- `backend.tests.test_stem_finalize`
Test-only Python dependencies for this profile are declared in `backend/requirements-test.txt`.
Use this when validating performance-fix regressions around long-run job history growth, retention pruning of old non-project jobs, in-memory cancellation-state cleanup, ensemble-memory accumulation behavior, and automatic stem finalization.
---
## RTX 3060 / Constrained-VRAM Stability
### Why MelBand Karaoke can OOM
The karaoke checkpoints (`mel_band_karaoke`, `mel_band_karaoke_gabox`) load a ~1.7 GB PyTorch checkpoint and allocate tensors proportional to the **full track length** × `batch_size` × `num_overlap`. On a 6–8 GB GPU with WSL2 overhead, this easily exhausts VRAM on:
- `performance` preset (`batch_size=8, num_overlap=4`)
- tracks longer than 8 minutes
- warm GPU with cached allocations from prior jobs
### MelBand stage diagnostics
Set `MELBAND_STAGE_DIAGNOSTICS=true` to instrument the expensive setup stages that happen before the first visible separation segment begins.
The backend now reports memory snapshots for these MelBand stages:
- start
- before audio load
- after audio load
- before model load
- after model load
- before demix/preallocation
- after demix
- after cleanup
Each snapshot includes:
- process RSS
- current CUDA allocated memory
- current CUDA reserved memory
- peak CUDA allocated memory
- peak CUDA reserved memory
Example progress message:
```text
[diag] MelBand lead/backing_instrumental: after model load — RSS=1840.2 MiB, CUDA alloc=1723.5 MiB, reserved=1824.0 MiB, peak_alloc=1723.5 MiB, peak_reserved=1824.0 MiB
```
This makes it much easier to tell whether the main spike came from whole-file audio decode, checkpoint load, or the full-length overlap-add buffers.
### Automatic VRAM headroom recovery
Before each separation job, StemEngine now unloads any cached in-process Whisper models left behind by lyric transcription and flushes idle CUDA allocator blocks. This prevents an otherwise-idle transcription model from quietly reducing the VRAM available to BS-RoFormer / MelBand jobs.
In addition, GPU separation jobs now run inside a **short-lived child Python process** by default. This keeps allocator fragmentation and CUDA fault state from accumulating indefinitely inside the main backend process after repeated heavy jobs or prior OOM events.
You can disable separator child-process isolation only if needed for debugging:
| Variable | Default | Effect |
|---|---|---|
| `STEMENGINE_SEPARATION_SUBPROCESS` | `true` | When `false`, run separator jobs in the main backend process instead of a short-lived child process |
### OOM ladder
When a MelBand Karaoke job encounters an out-of-memory error the backend automatically retries with progressively safer parameters before giving up:
| Attempt | batch_size | num_overlap | Notes |
|---|---|---|---|
| 1 | preset default | preset default | Normal run |
| 2 | 1 | preset default | batch_size reduced |
| 3 | 1 | 2 | num_overlap also reduced (eco) |
| fallback | — | — | Cross-model to `mdxnet_karaoke` (ONNX) |
The fallback step is governed by `FALLBACK_REQUIRE_LEAD_BACK` (see below).
### Lead/backing-capable fallback policy
`FALLBACK_REQUIRE_LEAD_BACK=true` (default) ensures the cross-model fallback only ever uses a model that **also produces `lead` and `backing_instrumental` stems** — never a plain vocals/instrumental model. This prevents silent degradation of karaoke output quality.
Models that qualify as lead/back-capable:
| Model | Notes |
|---|---|
| `mel_band_karaoke` | MelBand primary — full PyTorch |
| `mel_band_karaoke_gabox` | MelBand alternate checkpoint |
| `bs_roformer_karaoke_anvuew` | BS-RoFormer karaoke variant |
| `bs_roformer_karaoke_frazer_becruily` | BS-RoFormer karaoke variant |
| `mdxnet_karaoke` | **ONNX — lowest VRAM (~200 MB), default fallback target** |
If all fallback models also OOM, the job fails with a clear error message listing what was tried and actionable suggestions.
Set `FALLBACK_REQUIRE_LEAD_BACK=false` to skip the cross-model fallback entirely and surface the OOM error immediately.
### Duration guardrail
Tracks longer than `LONG_TRACK_DURATION_SECS` (default: **480 s = 8 min**) have MelBand eco params (`batch_size=1, num_overlap=2`) forced **before** inference, regardless of the active power preset. The guardrail fires for both `mel_band_karaoke` and `mel_band_roformer_4stem`.
A log message is emitted when the guardrail activates so you can confirm it in the job progress stream.
### Recommended settings for RTX 3060 (12 GB WSL2)
| Variable | Recommended | Reason |
|---|---|---|
| `STEMENGINE_GPU_VRAM_RESERVE_MB` | `2048` | Leaves headroom for Windows DWM/WDDM |
| `LONG_TRACK_DURATION_SECS` | `480` | Force eco on tracks ≥ 8 min |
| `FALLBACK_REQUIRE_LEAD_BACK` | `true` | Never lose lead/back distinction silently |
| Power preset | `standard` or `eco` | Avoid `performance` on this GPU |
### Troubleshooting sequence
1. **Check job progress** — OOM retries and fallback activations are logged inline in the job progress field.
2. **Use `/api/system/gpu-reset` or Settings → GPU reset** — this clears cached transcription models and flushes CUDA state without a full restart.
3. **Restart the container** — if the GPU context is truly corrupted, process isolation reduces recurrence but a hard CUDA fault can still require a full restart.
4. **Use eco preset** — `batch_size=1, num_overlap=2` is the safest configuration.
5. **Split long tracks** — tracks > 8 min on this GPU should be split before separation.
6. **Check VRAM headroom** — on WSL2, run `nvidia-smi` on the host while idle and verify that ≥ 9 GB shows as free before submitting a karaoke job.
### RoFormer constructor compatibility note
If you see an error such as:
`TypeError: MelBandRoformer.__init__() got an unexpected keyword argument 'freqs_per_bands'`
this usually means the model config was authored for a different `bs_roformer` package constructor signature.
Current backend builds include a compatibility shim that:
- remaps known renamed keys when possible,
- drops unsupported constructor keys with a backend warning,
- applies to MelBand and BS-RoFormer loaders.
If the error persists, verify that the checkpoint and YAML config belong to the same model variant.
If you see an error such as:
`Error(s) in loading state_dict for MelBandRoformer: size mismatch for ...`
the checkpoint and config likely describe different architectures/revisions. The backend now surfaces checkpoint/config filenames in this failure path to speed triage.
Note: the core `mel_band_karaoke_gabox` checkpoint is sourced from a `bsroformers/` path and is handled with BS-RoFormer architecture loading to avoid MelBand class mismatch.
---
## GaboxR67 model expansion (implemented rollout)
Source scope: `https://huggingface.co/GaboxR67` (primarily `GaboxR67/MelBandRoformers`).
### Does this include small models?
Yes. Small models are implemented as an explicit opt-in tier controlled by `GABOX_ENABLE_SMALL_MODELS`.
### Rollout tiers
| Tier | Target | Purpose |
|---|---|---|
| Tier A (core) | Stable Gabox MelBand models (`mel_band_vocals_gabox`) | Quality-first default additions |
| Tier B (small) | `mel_band_karaoke_gabox_small` | Faster/safer runs on constrained GPUs (opt-in) |
| Tier C (experimental) | `mel_band_karaoke_gabox_v2_exp` | Opt-in only; no default fallback usage |
### Integration boundaries
- Reused existing MelBand specialized runner architecture.
- Added model IDs + labels to backend model catalog.
- Added downloader mappings for Gabox checkpoints/config files.
- Small/experimental tiers are hidden unless enabled by env controls.
- Small karaoke model has an additional duration guardrail using `GABOX_SMALL_MODEL_MAX_DURATION_SECS`.
### Current behavior notes
- `mel_band_vocals_gabox` is always available.
- `mel_band_karaoke_gabox_small` appears only when `GABOX_ENABLE_SMALL_MODELS=true`.
- `mel_band_karaoke_gabox_v2_exp` appears only when `GABOX_ENABLE_EXPERIMENTAL_MODELS=true`.
- MelBand diagnostics and OOM ladder remain active for Gabox variants.