496 lines
22 KiB
Markdown
Executable File
496 lines
22 KiB
Markdown
Executable File
# StemEngine - Karaoke Anything?
|
||
|
||
> This is a **vibe-coded** project based on the work of people who actually know what they're doing. But now I can theoretically karaoke anything and so can you.
|
||
|
||
---
|
||
|
||
## Documentation
|
||
|
||
- User docs: [docs/USER_GUIDE.md](docs/USER_GUIDE.md)
|
||
- Developer docs: [docs/DEVELOPER_GUIDE.md](docs/DEVELOPER_GUIDE.md)
|
||
- Documentation plan: [docs/DOCUMENTATION_PLAN.md](docs/DOCUMENTATION_PLAN.md)
|
||
- API reference: [docs/API_REFERENCE.md](docs/API_REFERENCE.md)
|
||
- Troubleshooting matrix: [docs/TROUBLESHOOTING_MATRIX.md](docs/TROUBLESHOOTING_MATRIX.md)
|
||
- Configuration reference: [docs/CONFIGURATION_REFERENCE.md](docs/CONFIGURATION_REFERENCE.md)
|
||
- UVR comparison: [docs/UVR_VS_STEMENGINE.md](docs/UVR_VS_STEMENGINE.md)
|
||
- Long-track chunking plan: [docs/LONG_TRACK_CHUNKING_PLAN.md](docs/LONG_TRACK_CHUNKING_PLAN.md)
|
||
- Documentation governance: [docs/DOC_GOVERNANCE.md](docs/DOC_GOVERNANCE.md)
|
||
- Release checklist: [docs/RELEASE_CHECKLIST.md](docs/RELEASE_CHECKLIST.md)
|
||
|
||
---
|
||
|
||
## Features
|
||
|
||
- **Stem Separation** — 40+ models across 10 model families
|
||
- **MIDI Transcription** — basic-pitch with polyphonic and melody modes
|
||
- **GPU Safety Limits** — shared VRAM reserve for separation, lyric transcription, basic-pitch, and Omnizart sidecar
|
||
- **ABC Notation** — convert MIDI to ABC notation for sheet music
|
||
- **Lyrics Sync** — forced-alignment to produce ELRC files with word-level timestamps and a live animated preview
|
||
- **Project Management** — persistent project tabs, per-stem reprocessing, stem renaming
|
||
- **Mix Presets** — Instrumental, Static (remove elements), Dynamic (per-stem volume)
|
||
- **Subsonic Integration** — browse and import from a Subsonic-compatible server
|
||
- **Model Downloader** — download model weights on-demand from the Settings UI
|
||
|
||
---
|
||
|
||
## Separation Models
|
||
|
||
| Family | Models |
|
||
|---|---|
|
||
| **Demucs** | v4, v4 Fine-tuned, v3 Extra Quantized, v3 |
|
||
| **BS-RoFormer** | 4-stem, 5-stem (Guitar), 5-stem (Piano), 6-stem, Chorus, Anvuew, Karaoke Anvuew, Karaoke Frazer+Becruily, Mamba2 Vocals (Adapter), SW 6-stem |
|
||
| **MelBand RoFormer** | 4-stem, Vocals, Vocals Anvuew, Vocals Gabox, Dereverb, Dereverb Anvuew, Dereverb Room, Karaoke, Karaoke Gabox, Karaoke Gabox (Small, opt-in), Karaoke Gabox V2 Experimental (opt-in), Deux Becruily |
|
||
| **MDX-Net** | Vocals FT, Inst HQ 1, Inst HQ 2, Karaoke, Dereverb, Drum Sep |
|
||
| **SCNet** | XL (2-stem), XL IHF, Masked XL IHF, Masked Small, Tran Small, Large (4-stem), Large Starrytong |
|
||
| **MedleyVox** | Base, 55/66/88/100 Epoch variants |
|
||
| **Apollo** | Vocals, EDM, EDM Big |
|
||
| **Aspiration** | Full, Light |
|
||
| **Bandit v2** | Multi-Language, English |
|
||
| **Ensemble** | Vocal Best Quality, 4-Stem, 6-Stem Full |
|
||
|
||
Model weights are downloaded on first use and cached in `/opt/`.
|
||
|
||
### Checkpoint formats (`.ckpt` vs `.safetensors`)
|
||
|
||
StemEngine's current BS-RoFormer / MelBand runtime primarily expects a model-specific config YAML plus a PyTorch checkpoint file such as `.ckpt`.
|
||
|
||
What `safetensors` means:
|
||
|
||
- it is a safer weight serialization format,
|
||
- it does **not** by itself change the MelBand / BS-RoFormer architecture,
|
||
- it does **not** materially reduce inference VRAM or compute cost,
|
||
- it can slightly improve load safety and sometimes startup/load efficiency.
|
||
|
||
Practical expectation in StemEngine:
|
||
|
||
- a `safetensors` conversion is usually the same trained model packaged differently,
|
||
- current built-in model loaders are wired for `.ckpt` + YAML style checkpoints,
|
||
- direct `safetensors` support would require loader support for `safetensors` deserialization and any key-mapping needed by the exported weights.
|
||
|
||
---
|
||
|
||
## Requirements
|
||
|
||
- Docker with the NVIDIA Container Toolkit (for GPU acceleration)
|
||
- NVIDIA GPU with CUDA 12.4 support
|
||
- Docker Compose
|
||
|
||
CPU-only mode is available via `CPU_SEPARATION=true` but separation will be significantly slower.
|
||
|
||
---
|
||
|
||
## Quick Start
|
||
|
||
1. Copy `.env.example` to `.env` and fill in the required values (see Configuration below).
|
||
|
||
2. Build and start:
|
||
|
||
```bash
|
||
docker compose up -d --build
|
||
```
|
||
|
||
3. Open `http://localhost:8000` in your browser.
|
||
|
||
Optional WebUI (Vite dev server) default host URL is `http://localhost:5174`.
|
||
|
||
---
|
||
|
||
## Configuration
|
||
|
||
All configuration is via environment variables. The `docker-compose.yml` accepts these via a `.env` file in the same directory.
|
||
|
||
### Required
|
||
|
||
| Variable | Description |
|
||
|---|---|
|
||
| `SEFOLDER` | Host path for persistent runtime data (uploads, jobs, DB, LRC sync) |
|
||
| `ENGINEFOLDER` | Host path for model weight caches |
|
||
| `PROJECTS` | Host path for separated project output |
|
||
| `SECRET_KEY` | Random hex key for encrypting Subsonic credentials at rest. Generate with: `python3 -c "import secrets; print(secrets.token_hex(32))"` |
|
||
|
||
### Optional
|
||
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `CPU_SEPARATION` | `false` | Set to `true` to force CPU inference |
|
||
| `MAX_UPLOAD_MB` | `500` | Maximum upload file size in MB |
|
||
| `STEMENGINE_GPU_VRAM_RESERVE_MB` | `2048` | Keeps this many MB free on the GPU (WSL2-safe headroom) |
|
||
| `OMNIZART_GPU_VRAM_RESERVE_MB` | `STEMENGINE_GPU_VRAM_RESERVE_MB` | Optional sidecar override for Omnizart TensorFlow GPU cap |
|
||
| `STEMENGINE_UVICORN_RELOAD_FLAG` | empty | Set to `--reload` only when actively developing backend code; leave empty for long-running separation jobs |
|
||
| `WEBUI_HOST_PORT` | `5174` | Host port bound to the WebUI container (set if `5174` or `5173` is already in use) |
|
||
| `WEBUI_PORT` | `5173` | Internal WebUI container port (normally unchanged) |
|
||
| `MELBAND_STAGE_DIAGNOSTICS` | `false` | Emit per-stage MelBand memory snapshots into job progress + backend logs |
|
||
| `GABOX_ENABLE_SMALL_MODELS` | `false` | Expose Tier B small Gabox models (for constrained VRAM) |
|
||
| `GABOX_ENABLE_EXPERIMENTAL_MODELS` | `false` | Expose Tier C experimental Gabox models |
|
||
| `GABOX_SMALL_MODEL_MAX_DURATION_SECS` | `360` | Guardrail threshold used by Gabox small karaoke model |
|
||
| `GABOX_SMALL_MODEL_PREFERRED_FOR_LOW_VRAM` | `false` | Reserved policy flag for low-VRAM auto-preference workflows |
|
||
| `STEMENGINE_LYRIC_SEGMENTER` | `auto` | Lyric line splitter strategy: `auto`, `karaoke_processor`, `legacy` |
|
||
| `STEMENGINE_LYRIC_MAX_LINE_LENGTH` | `36` | Max line length target used by karaoke lyric segmentation |
|
||
| `STEMENGINE_KARAOKE_EXPORT_DIR` | empty | Optional global `.sek` export directory inside the container. If empty, exports go under each project (`Exports/StemEngineKaraoke`). |
|
||
| `SEK_EXPORTS` | `${SEFOLDER}/sek_exports` | Host path mounted to `/app/media/sek_exports` for centralized `.sek` package output. |
|
||
| `TZ` | `UTC` | Timezone |
|
||
| `NAVSERVER` | — | Subsonic server URL (can also be set in the UI) |
|
||
| `NAVUSER` | — | Subsonic username |
|
||
| `NAVPASS` | — | Subsonic password |
|
||
| `SLOPPAK_SOURCE_ROOT` | `/app/media/uploads` | Converter source root for relative PSARC paths |
|
||
| `SLOPPAK_OUTPUT_ROOT` | `/app/media/sloppak` | Converter output root for generated `.sloppak` files |
|
||
| `SLOPPAK_PSARC_UNPACK_CMD` | empty | Optional PSARC unpack command template (`{input}` and `{output}` placeholders) used for deep Rocksmith metadata extraction when python patcher module is unavailable |
|
||
|
||
### Model Directories
|
||
|
||
These are set automatically in `docker-compose.yml` and point inside the `/opt/` volume. Override only if you need a custom layout.
|
||
|
||
`BS_ROFORMER_MODEL_DIR`, `MEL_BAND_MODEL_DIR`, `MDXNET_MODEL_DIR`, `SCNET_MODEL_DIR`, `MEDLEYVOX_MODEL_DIR`, `APOLLO_MODEL_DIR`, `ASPIRATION_MODEL_DIR`, `BANDIT_MODEL_DIR`
|
||
|
||
---
|
||
|
||
## Project Folder Structure
|
||
|
||
Each separation job creates a folder under `PROJECTS_DIR`:
|
||
|
||
```
|
||
|
||
---
|
||
|
||
## sloppak Converter Deep Extraction
|
||
|
||
The PSARC -> sloppak converter can enrich packages with Rocksmith metadata (arrangements, lyrics, WEM inventory) when a PSARC unpacker is available.
|
||
|
||
Unpacker resolution order:
|
||
|
||
1. Python `patcher.unpack_psarc` module (if installed)
|
||
2. `SLOPPAK_PSARC_UNPACK_CMD` command template from environment
|
||
|
||
Command template requirements:
|
||
|
||
- Must include `{input}` and `{output}` placeholders.
|
||
- Example:
|
||
|
||
```bash
|
||
SLOPPAK_PSARC_UNPACK_CMD="/usr/local/bin/psarc unpack {input} {output}"
|
||
```
|
||
|
||
Runtime verification endpoint:
|
||
|
||
- `GET /api/sloppak/capabilities`
|
||
- Reports whether a usable unpacker is currently detected.
|
||
Artist - Title/
|
||
Artist - Title.mp3 # Original source audio (copy)
|
||
Artist - Title.elrc # ELRC lyrics with word-level timestamps (if aligned)
|
||
Artist - Title.lrc # Plain LRC (if aligned)
|
||
Stems/
|
||
vocals.mp3
|
||
drums.mp3
|
||
bass.mp3
|
||
other.mp3
|
||
Instrumental/ # Only when Instrumental preset selected
|
||
Artist - Title - Instrumental.mp3
|
||
Static/ # Only when Static preset selected
|
||
Artist - Title - No Vocals.mp3
|
||
Dynamic/ # Only when Dynamic preset selected
|
||
Artist - Title - Dynamic Mix.mp3
|
||
```
|
||
|
||
---
|
||
|
||
## Tech Stack
|
||
|
||
- **Frontend** — React 18, TypeScript, Vite, TailwindCSS
|
||
- **Backend** — FastAPI, Python 3.11, SQLAlchemy
|
||
- **Separation** — Demucs, BS-RoFormer, MelBand RoFormer, MDX-Net (onnxruntime-gpu), SCNet, MedleyVox, Apollo, Aspiration, Bandit
|
||
- **Transcription** — basic-pitch (ONNX, CUDAExecutionProvider)
|
||
- **Lyrics Sync** — wav2vec2-based forced alignment via custom `lsync` module
|
||
- **Runtime** — NVIDIA CUDA 12.4 + cuDNN 9, PyTorch 2.6 (cu124 wheels)
|
||
- **Container** — `nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04`
|
||
|
||
---
|
||
|
||
## Auto Pipeline + Lyric Engine Notes (2026-03)
|
||
|
||
- Auto Pipeline now supports:
|
||
- per-instrument refinement mapping (`instrument_models`) at separation stage
|
||
- lyric engine selection (`WhisperX` or self-hosted `Parakeet`) in lyric steps
|
||
- canonical stem finalization after separation (rename to project-style filenames + automatic removal of silent/no-significant-audio stems)
|
||
- Saved auto-pipeline configs are migrated on read:
|
||
- legacy `whisperModel` is upgraded to engine-aware `engine` + `model`
|
||
- missing `instrumentModels` defaults to empty mapping
|
||
|
||
### Parakeet constraints
|
||
|
||
Parakeet uses alternate handling and does **not** currently expose WhisperX-only controls:
|
||
|
||
- `diarize`
|
||
- `lsync_align`
|
||
- `two_pass`
|
||
- `explicit`
|
||
- `translate` task
|
||
|
||
If Parakeet is selected and unavailable, verify sidecar health and profile wiring:
|
||
|
||
- backend must have `PARAKEET_ENABLED=true`
|
||
- backend `PARAKEET_SIDECAR_URL` must point to the sidecar service
|
||
- sidecar `GET /health` must return `200`
|
||
|
||
---
|
||
|
||
## WebUI routing (Phase 1)
|
||
|
||
- The WebUI now uses URL-backed navigation for the primary screens:
|
||
- `/` → separator/home
|
||
- `/settings` → settings
|
||
- `/projects/:projectId` → project detail
|
||
- Refreshing on a project page now restores that same project view directly from the URL.
|
||
- Sidebar project entries are real links, which is the foundation for browser-native new-tab and deep-link behavior.
|
||
|
||
## WebUI persistence and browser-tab behavior (Phases 2/3)
|
||
|
||
- The WebUI now persists key client-side shell state across refreshes, including:
|
||
- sidebar open/collapsed state,
|
||
- sidebar project search text,
|
||
- separator form context (input mode, selected model/format/preset, project name, imported file selection when still available).
|
||
- Auto Pipeline now saves a lightweight draft for its project name and offers restore/discard when reopened.
|
||
- Project completion surfaces now expose route-native links so projects can be opened directly or with normal browser new-tab behavior.
|
||
|
||
## WebUI PWA hardening (Phase 4)
|
||
|
||
- The WebUI manifest is now hardened with explicit PWA metadata (`id`, `scope`, `display_override`, categories, and maskable/standard icon declarations).
|
||
- Service worker caching is versioned and conservative:
|
||
- shell assets are precached,
|
||
- route navigations are network-first with shell fallback,
|
||
- API responses are never cached as durable offline truth.
|
||
- The shell now includes:
|
||
- install affordance when `beforeinstallprompt` is available,
|
||
- update notification with explicit “Reload to update” flow,
|
||
- offline status messaging so disconnected behavior is clear.
|
||
|
||
## WebUI cross-tab session hygiene (Phase 5)
|
||
|
||
- Cross-tab project deletion notifications are now propagated via `BroadcastChannel` with localStorage fallback.
|
||
- Other open tabs now invalidate stale project views when a project is deleted and automatically leave deleted project routes.
|
||
- Durable appearance/theme changes now propagate across tabs via storage-based synchronization.
|
||
|
||
## Frontend bundle split hardening
|
||
|
||
- Route-level lazy loading is now used for major pages.
|
||
- Vite chunk warning threshold is tuned for the split-route architecture to avoid false-positive size warnings during normal production builds.
|
||
|
||
## WebUI QoL closure (Phase 6)
|
||
|
||
- Final acceptance checklist is tracked in [docs/WEBUI_QOL_QA_CHECKLIST.md](docs/WEBUI_QOL_QA_CHECKLIST.md).
|
||
- Cross-tab events now cover project deletion, project invalidation, and project-list changes.
|
||
- Automated browser smoke tests are available via Playwright (`cd frontend && npm run e2e`).
|
||
|
||
---
|
||
|
||
## GPU Notes
|
||
|
||
The container requires the NVIDIA Container Toolkit. On the host:
|
||
|
||
```bash
|
||
# Verify GPU is visible inside the container
|
||
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
|
||
```
|
||
|
||
PyTorch wheels are self-contained (cu124) and do not require a matching CUDA toolkit on the host — only the driver.
|
||
|
||
`STEMENGINE_GPU_VRAM_RESERVE_MB` is applied in backend GPU code paths (separation, lyric transcription, and basic-pitch startup) to preserve display headroom on WSL2.
|
||
|
||
Omnizart sidecar now applies a TensorFlow memory cap using `OMNIZART_GPU_VRAM_RESERVE_MB` (or `STEMENGINE_GPU_VRAM_RESERVE_MB` fallback), so sidecar transcription follows the same reserve policy.
|
||
|
||
---
|
||
|
||
## Targeted backend tests (Findings 3/4/5 + optional hardening)
|
||
|
||
To run the bounded jobs API + retention/cancel-state hardening regression suite with full backend dependencies, use the compose test profile:
|
||
|
||
```bash
|
||
docker compose -f stemengine.yml --profile tests run --rm stemengine-tests
|
||
```
|
||
|
||
or the Make helper:
|
||
|
||
```bash
|
||
make test-findings345
|
||
```
|
||
|
||
This runs:
|
||
|
||
- `backend.tests.test_jobs_api`
|
||
- `backend.tests.test_project_cleanup`
|
||
- `backend.tests.test_projects_delete`
|
||
- `backend.tests.test_findings_3_4_unit`
|
||
- `backend.tests.test_ensemble_separator`
|
||
- `backend.tests.test_optional_hardening`
|
||
- `backend.tests.test_stem_finalize`
|
||
|
||
Test-only Python dependencies for this profile are declared in `backend/requirements-test.txt`.
|
||
|
||
Use this when validating performance-fix regressions around long-run job history growth, retention pruning of old non-project jobs, in-memory cancellation-state cleanup, ensemble-memory accumulation behavior, and automatic stem finalization.
|
||
|
||
---
|
||
|
||
## RTX 3060 / Constrained-VRAM Stability
|
||
|
||
### Why MelBand Karaoke can OOM
|
||
|
||
The karaoke checkpoints (`mel_band_karaoke`, `mel_band_karaoke_gabox`) load a ~1.7 GB PyTorch checkpoint and allocate tensors proportional to the **full track length** × `batch_size` × `num_overlap`. On a 6–8 GB GPU with WSL2 overhead, this easily exhausts VRAM on:
|
||
|
||
- `performance` preset (`batch_size=8, num_overlap=4`)
|
||
- tracks longer than 8 minutes
|
||
- warm GPU with cached allocations from prior jobs
|
||
|
||
### MelBand stage diagnostics
|
||
|
||
Set `MELBAND_STAGE_DIAGNOSTICS=true` to instrument the expensive setup stages that happen before the first visible separation segment begins.
|
||
|
||
The backend now reports memory snapshots for these MelBand stages:
|
||
|
||
- start
|
||
- before audio load
|
||
- after audio load
|
||
- before model load
|
||
- after model load
|
||
- before demix/preallocation
|
||
- after demix
|
||
- after cleanup
|
||
|
||
Each snapshot includes:
|
||
|
||
- process RSS
|
||
- current CUDA allocated memory
|
||
- current CUDA reserved memory
|
||
- peak CUDA allocated memory
|
||
- peak CUDA reserved memory
|
||
|
||
Example progress message:
|
||
|
||
```text
|
||
[diag] MelBand lead/backing_instrumental: after model load — RSS=1840.2 MiB, CUDA alloc=1723.5 MiB, reserved=1824.0 MiB, peak_alloc=1723.5 MiB, peak_reserved=1824.0 MiB
|
||
```
|
||
|
||
This makes it much easier to tell whether the main spike came from whole-file audio decode, checkpoint load, or the full-length overlap-add buffers.
|
||
|
||
### Automatic VRAM headroom recovery
|
||
|
||
Before each separation job, StemEngine now unloads any cached in-process Whisper models left behind by lyric transcription and flushes idle CUDA allocator blocks. This prevents an otherwise-idle transcription model from quietly reducing the VRAM available to BS-RoFormer / MelBand jobs.
|
||
|
||
In addition, GPU separation jobs now run inside a **short-lived child Python process** by default. This keeps allocator fragmentation and CUDA fault state from accumulating indefinitely inside the main backend process after repeated heavy jobs or prior OOM events.
|
||
|
||
You can disable separator child-process isolation only if needed for debugging:
|
||
|
||
| Variable | Default | Effect |
|
||
|---|---|---|
|
||
| `STEMENGINE_SEPARATION_SUBPROCESS` | `true` | When `false`, run separator jobs in the main backend process instead of a short-lived child process |
|
||
|
||
### OOM ladder
|
||
|
||
When a MelBand Karaoke job encounters an out-of-memory error the backend automatically retries with progressively safer parameters before giving up:
|
||
|
||
| Attempt | batch_size | num_overlap | Notes |
|
||
|---|---|---|---|
|
||
| 1 | preset default | preset default | Normal run |
|
||
| 2 | 1 | preset default | batch_size reduced |
|
||
| 3 | 1 | 2 | num_overlap also reduced (eco) |
|
||
| fallback | — | — | Cross-model to `mdxnet_karaoke` (ONNX) |
|
||
|
||
The fallback step is governed by `FALLBACK_REQUIRE_LEAD_BACK` (see below).
|
||
|
||
### Lead/backing-capable fallback policy
|
||
|
||
`FALLBACK_REQUIRE_LEAD_BACK=true` (default) ensures the cross-model fallback only ever uses a model that **also produces `lead` and `backing_instrumental` stems** — never a plain vocals/instrumental model. This prevents silent degradation of karaoke output quality.
|
||
|
||
Models that qualify as lead/back-capable:
|
||
|
||
| Model | Notes |
|
||
|---|---|
|
||
| `mel_band_karaoke` | MelBand primary — full PyTorch |
|
||
| `mel_band_karaoke_gabox` | MelBand alternate checkpoint |
|
||
| `bs_roformer_karaoke_anvuew` | BS-RoFormer karaoke variant |
|
||
| `bs_roformer_karaoke_frazer_becruily` | BS-RoFormer karaoke variant |
|
||
| `mdxnet_karaoke` | **ONNX — lowest VRAM (~200 MB), default fallback target** |
|
||
|
||
If all fallback models also OOM, the job fails with a clear error message listing what was tried and actionable suggestions.
|
||
|
||
Set `FALLBACK_REQUIRE_LEAD_BACK=false` to skip the cross-model fallback entirely and surface the OOM error immediately.
|
||
|
||
### Duration guardrail
|
||
|
||
Tracks longer than `LONG_TRACK_DURATION_SECS` (default: **480 s = 8 min**) have MelBand eco params (`batch_size=1, num_overlap=2`) forced **before** inference, regardless of the active power preset. The guardrail fires for both `mel_band_karaoke` and `mel_band_roformer_4stem`.
|
||
|
||
A log message is emitted when the guardrail activates so you can confirm it in the job progress stream.
|
||
|
||
### Recommended settings for RTX 3060 (12 GB WSL2)
|
||
|
||
| Variable | Recommended | Reason |
|
||
|---|---|---|
|
||
| `STEMENGINE_GPU_VRAM_RESERVE_MB` | `2048` | Leaves headroom for Windows DWM/WDDM |
|
||
| `LONG_TRACK_DURATION_SECS` | `480` | Force eco on tracks ≥ 8 min |
|
||
| `FALLBACK_REQUIRE_LEAD_BACK` | `true` | Never lose lead/back distinction silently |
|
||
| Power preset | `standard` or `eco` | Avoid `performance` on this GPU |
|
||
|
||
### Troubleshooting sequence
|
||
|
||
1. **Check job progress** — OOM retries and fallback activations are logged inline in the job progress field.
|
||
2. **Use `/api/system/gpu-reset` or Settings → GPU reset** — this clears cached transcription models and flushes CUDA state without a full restart.
|
||
3. **Restart the container** — if the GPU context is truly corrupted, process isolation reduces recurrence but a hard CUDA fault can still require a full restart.
|
||
4. **Use eco preset** — `batch_size=1, num_overlap=2` is the safest configuration.
|
||
5. **Split long tracks** — tracks > 8 min on this GPU should be split before separation.
|
||
6. **Check VRAM headroom** — on WSL2, run `nvidia-smi` on the host while idle and verify that ≥ 9 GB shows as free before submitting a karaoke job.
|
||
|
||
### RoFormer constructor compatibility note
|
||
|
||
If you see an error such as:
|
||
|
||
`TypeError: MelBandRoformer.__init__() got an unexpected keyword argument 'freqs_per_bands'`
|
||
|
||
this usually means the model config was authored for a different `bs_roformer` package constructor signature.
|
||
|
||
Current backend builds include a compatibility shim that:
|
||
|
||
- remaps known renamed keys when possible,
|
||
- drops unsupported constructor keys with a backend warning,
|
||
- applies to MelBand and BS-RoFormer loaders.
|
||
|
||
If the error persists, verify that the checkpoint and YAML config belong to the same model variant.
|
||
|
||
If you see an error such as:
|
||
|
||
`Error(s) in loading state_dict for MelBandRoformer: size mismatch for ...`
|
||
|
||
the checkpoint and config likely describe different architectures/revisions. The backend now surfaces checkpoint/config filenames in this failure path to speed triage.
|
||
|
||
Note: the core `mel_band_karaoke_gabox` checkpoint is sourced from a `bsroformers/` path and is handled with BS-RoFormer architecture loading to avoid MelBand class mismatch.
|
||
|
||
---
|
||
|
||
## GaboxR67 model expansion (implemented rollout)
|
||
|
||
Source scope: `https://huggingface.co/GaboxR67` (primarily `GaboxR67/MelBandRoformers`).
|
||
|
||
### Does this include small models?
|
||
|
||
Yes. Small models are implemented as an explicit opt-in tier controlled by `GABOX_ENABLE_SMALL_MODELS`.
|
||
|
||
### Rollout tiers
|
||
|
||
| Tier | Target | Purpose |
|
||
|---|---|---|
|
||
| Tier A (core) | Stable Gabox MelBand models (`mel_band_vocals_gabox`) | Quality-first default additions |
|
||
| Tier B (small) | `mel_band_karaoke_gabox_small` | Faster/safer runs on constrained GPUs (opt-in) |
|
||
| Tier C (experimental) | `mel_band_karaoke_gabox_v2_exp` | Opt-in only; no default fallback usage |
|
||
|
||
### Integration boundaries
|
||
|
||
- Reused existing MelBand specialized runner architecture.
|
||
- Added model IDs + labels to backend model catalog.
|
||
- Added downloader mappings for Gabox checkpoints/config files.
|
||
- Small/experimental tiers are hidden unless enabled by env controls.
|
||
- Small karaoke model has an additional duration guardrail using `GABOX_SMALL_MODEL_MAX_DURATION_SECS`.
|
||
|
||
### Current behavior notes
|
||
|
||
- `mel_band_vocals_gabox` is always available.
|
||
- `mel_band_karaoke_gabox_small` appears only when `GABOX_ENABLE_SMALL_MODELS=true`.
|
||
- `mel_band_karaoke_gabox_v2_exp` appears only when `GABOX_ENABLE_EXPERIMENTAL_MODELS=true`.
|
||
- MelBand diagnostics and OOM ladder remain active for Gabox variants.
|