11 KiB
STEMxLALA — Implementation Plan
Migrate StemEngine's separation backend to use LALAL.AI API v1 exclusively, replacing all local model inference (Demucs, BS-RoFormer, MelBand, MDX-Net, SCNet, etc.) with cloud API calls.
Background
Current architecture
separator_service.py— monolithic 2000+ line dispatcher that branches on model ID to call local ML model code (Demucs CLI subprocess, BS-RoFormer in-process PyTorch, MDX-Net ONNX, etc.)config.py— defines ~50 model IDs across 13 groupsrouters/separate.py— validates model ID, enqueues viajm.submit_gpu()job_manager.py—submit_gpu()routes to a single-worker GPU executor (_gpu_executor)- All separation runs inside the container on local GPU via CUDA (PyTorch cu124 wheels)
LALAL.AI v1 API
Reference implementation: lalalai/api-v1/python/lalalai_splitter.py
Three separation endpoints:
| Endpoint | Stem | Notes |
|---|---|---|
/api/v1/split/voice_clean/ |
voice |
Noise cancellation + optional dereverb |
/api/v1/split/demuser/ |
music |
Background music removal (speech stays) |
/api/v1/split/stem_separator/ |
vocals, drum, piano, bass, electric_guitar, acoustic_guitar, synthesizer, strings, wind |
deep_extraction or clear_cut, optional dereverb, optional multivocal |
Workflow per file: upload → split (async task) → poll check → download tracks → (optional) delete
Splitter model options: orion, perseus, phoenix, andromeda (or null = auto/latest)
Design Decisions
CUDA stays — for other features, not separation
The Docker image (nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04) and runtime: nvidia in
stemengine.yml must remain. CUDA is still required for:
- WhisperX lyric transcription (runs on GPU via
_transcription_executor) - basic-pitch ONNX inference
- Any future local models
The CPU_SEPARATION env var becomes irrelevant for LALA separation (network replaces GPU),
but the hardware config should not be removed.
API key storage
A new env var LALAL_API_KEY (type: str, required for separation) is stored alongside existing
secrets (SECRET_KEY, HF_TOKEN). It can be set in stemengine.yml or via the Settings UI
once the settings router supports it.
Job executor routing
LALAL API calls are network I/O, not GPU work. They must NOT go through submit_gpu()
(the single-worker GPU queue) — doing so would block all GPU-bound transcription while waiting
for the network. Use submit() (general thread pool) instead, or a new dedicated
_lala_executor = ThreadPoolExecutor(max_workers=3) to cap concurrent uploads.
Preset mixing (instrumental, static, dynamic)
The existing preset_service.py (make_instrumental, make_static, make_dynamic) mixes
stem audio files with pydub. It is model-agnostic — it takes file paths and operates on
whatever stems exist. This is fully reusable with LALA output. No changes needed here.
Project folder structure
Unchanged. LALA stems are downloaded into the same PROJECT_SUBFOLDERS["stems"] directory
that local models produce. The rest of the app (pipeline, mix, transcribe) is unaffected.
Output format
LALAL API delivers stems as downloaded files (filename from Content-Disposition). These may
be WAV or MP3 depending on LALA's output. After download, the existing format-conversion logic
(ffmpeg via pydub) should be applied to normalize to the user's requested output_format.
Implementation Steps
Step 1 — New service: lalalai_separator.py
Create backend/services/lalalai_separator.py as a clean, self-contained module.
Contents:
- Constants:
LALAL_API_BASE = "https://www.lalal.ai/api/v1/" - Functions lifted and adapted from the reference
lalalai_splitter.py:upload_file(file_path, api_key) -> source_idsplit_file(source_id, stem, api_key, *, splitter, extraction_level, dereverb, noise_cancelling, multivocal) -> task_idpoll_until_done(task_id, api_key, cancel_flag) -> result_dictdownload_stems(result_dict, dest_dir) -> list[Path]delete_source(source_id, api_key)
- Top-level
run_lala_separation(job_id, file_path, lala_model_id, output_format, *, preset_type, static_option, dynamic_levels, project_name_override)— mirrors the signature ofrun_separation()so callers are swappable.
Cancel support: poll_until_done accepts the job's threading.Event cancel flag (from
job_manager._cancel_flags) and raises CancelledError if set between polls.
Error handling: Wrap all requests calls; on HTTP error or timeout, call
jm.fail_job(job_id, message) and re-raise so the executor logs it.
Step 2 — New config section: LALA models
In backend/config.py, add:
# LALAL.AI v1 models
LALA_STEM_SEPARATOR_STEMS = [
{"id": "lala_vocals", "label": "LALA Vocals", "group": "LALAL.AI", "lala_stem": "vocals"},
{"id": "lala_drum", "label": "LALA Drums", "group": "LALAL.AI", "lala_stem": "drum"},
{"id": "lala_bass", "label": "LALA Bass", "group": "LALAL.AI", "lala_stem": "bass"},
{"id": "lala_piano", "label": "LALA Piano", "group": "LALAL.AI", "lala_stem": "piano"},
{"id": "lala_electric_guitar", "label": "LALA Electric Guitar", "group": "LALAL.AI", "lala_stem": "electric_guitar"},
{"id": "lala_acoustic_guitar", "label": "LALA Acoustic Guitar", "group": "LALAL.AI", "lala_stem": "acoustic_guitar"},
{"id": "lala_synthesizer", "label": "LALA Synthesizer", "group": "LALAL.AI", "lala_stem": "synthesizer"},
{"id": "lala_strings", "label": "LALA Strings", "group": "LALAL.AI", "lala_stem": "strings"},
{"id": "lala_wind", "label": "LALA Wind", "group": "LALAL.AI", "lala_stem": "wind"},
{"id": "lala_voice_clean", "label": "LALA Voice Clean", "group": "LALAL.AI", "lala_stem": "voice"},
{"id": "lala_demuser", "label": "LALA Demuser (Music)", "group": "LALAL.AI", "lala_stem": "music"},
]
LALA_MODELS = LALA_STEM_SEPARATOR_STEMS
LALA_MODEL_IDS = {m["id"] for m in LALA_MODELS}
Add LALA_MODELS to ALL_MODELS. Add LALA_MODEL_IDS to ALL_MODEL_IDS.
Optional LALA-specific parameters (can be form fields or user settings defaults):
LALA_DEFAULT_SPLITTER = None # auto = latest
LALA_DEFAULT_EXTRACTION_LEVEL = "deep_extraction"
LALA_DEFAULT_DEREVERB = False
LALA_DEFAULT_NOISE_CANCELLING = 1
Step 3 — Router changes: routers/separate.py
-
Add optional form fields to
POST /api/separate:lala_splitter: Optional[str] = Form(None)— one oforion|perseus|phoenix|andromeda|Nonelala_extraction_level: str = Form("deep_extraction")lala_dereverb: bool = Form(False)lala_noise_cancelling: int = Form(1)lala_multivocal: Optional[str] = Form(None)—lead_backor None
-
Routing logic in
separate():if model in LALA_MODEL_IDS: jm.submit_lala(run_lala_separation, job_id, ...) else: jm.submit_gpu(run_separation, job_id, ...) -
Apply same routing to
separate_existing()andseparate_from_path().
Step 4 — Job manager: add submit_lala()
In backend/job_manager.py, add:
_lala_executor = ThreadPoolExecutor(max_workers=3, thread_name_prefix="lala")
def submit_lala(fn: Callable, *args, **kwargs):
"""Submit a LALAL.AI API separation task (network I/O, not GPU)."""
_lala_executor.submit(fn, *args, **kwargs)
This keeps LALA network calls off the GPU queue so local GPU tasks (transcription) are never blocked waiting for remote API responses.
Step 5 — Config: env var + settings
In config.py:
LALAL_API_KEY = os.environ.get("LALAL_API_KEY", "")
In stemengine.yml:
- LALAL_API_KEY=${LALAL_API_KEY}
In the Settings UI (if applicable), surface LALAL_API_KEY as an encrypted credential field
alongside the existing Subsonic and HuggingFace token fields.
The existing SECRET_KEY-based encryption in settings_service.py / the settings router
should handle this with no structural changes — just add the field.
Step 6 — Dockerfile cleanup (optional, deferred)
Once LALA is the sole separator:
- PyTorch, torchaudio, torchvision (CUDA wheels — ~3 GB) can be removed if no other feature needs them. Hold off until WhisperX is also evaluated — it currently depends on PyTorch.
- All model-specific Python packages (open-unmix, demucs, ml_collections, etc.) can be removed.
- Model volume mounts (
/opt/bs_roformer_models, etc.) can be removed fromstemengine.yml. CPU_SEPARATIONenv var becomes vestigial but harmless to leave.
This step is explicitly deferred until the LALA path is confirmed working in production.
Step 7 — Frontend (minimal changes)
The frontend's model picker currently renders ALL_MODELS grouped by "group". Adding
"group": "LALAL.AI" models to ALL_MODELS is sufficient — the group will appear
automatically.
LALA-specific form fields (lala_splitter, lala_extraction_level, etc.) should be surfaced
conditionally when the selected model's group is "LALAL.AI". This is a UI-only change in the
separation form component (likely SeparatePanel or equivalent).
File Change Summary
| File | Action | Notes |
|---|---|---|
backend/services/lalalai_separator.py |
Create | New LALA separation service |
backend/config.py |
Edit | Add LALA_MODELS, LALAL_API_KEY |
backend/routers/separate.py |
Edit | Add LALA form fields, routing branch |
backend/job_manager.py |
Edit | Add _lala_executor, submit_lala() |
stemengine.yml |
Edit | Add LALAL_API_KEY env var |
backend/services/separator_service.py |
No change yet | Keep until LALA confirmed |
backend/services/preset_service.py |
No change | Reused as-is |
Dockerfile |
Deferred | Cleanup after LALA confirmed working |
What is NOT changing
job_manager.pyexecutor for GPU (_gpu_executor) — still used for WhisperX transcriptionpreset_service.py— mix logic is model-agnostic, works on file paths- Project folder structure — stems land in the same place
- Docker CUDA base image — required for WhisperX
nvidiaruntime instemengine.yml— required for WhisperX- All non-separation routers (transcribe, lyric_transcribe, lrc, subsonic, pipeline, etc.)
Risks & Notes
- LALAL.AI API key required at runtime. If
LALAL_API_KEYis empty and a LALA model is selected,lalalai_separator.pyshould fail the job immediately with a clear error message rather than getting an HTTP 401 partway through an upload. - Upload cost. LALAL.AI charges credits per minute of audio. The plan does not implement
source caching/reuse (
source_idreuse across jobs for the same file) in v1 — that is a potential future optimization. - File size limits. LALAL.AI imposes upload limits per license tier. Large files may need to be rejected before upload with an appropriate HTTP 413 or job error.
- Polling interval. The reference uses 5-second polling. This is fine for background jobs but should respect the cancel flag between polls.
- Output filename. LALA returns filenames via
Content-Disposition. The download function must parse this (already implemented in the reference code) and then rename to match the StemEngine stem-naming convention:"Artist - Title (vocals).mp3".