Files
stemengine_old/STEMxLALA.md
T

11 KiB

STEMxLALA — Implementation Plan

Migrate StemEngine's separation backend to use LALAL.AI API v1 exclusively, replacing all local model inference (Demucs, BS-RoFormer, MelBand, MDX-Net, SCNet, etc.) with cloud API calls.


Background

Current architecture

  • separator_service.py — monolithic 2000+ line dispatcher that branches on model ID to call local ML model code (Demucs CLI subprocess, BS-RoFormer in-process PyTorch, MDX-Net ONNX, etc.)
  • config.py — defines ~50 model IDs across 13 groups
  • routers/separate.py — validates model ID, enqueues via jm.submit_gpu()
  • job_manager.py — submit_gpu() routes to a single-worker GPU executor (_gpu_executor)
  • All separation runs inside the container on local GPU via CUDA (PyTorch cu124 wheels)

LALAL.AI v1 API

Reference implementation: lalalai/api-v1/python/lalalai_splitter.py

Three separation endpoints:

Endpoint Stem Notes
/api/v1/split/voice_clean/ voice Noise cancellation + optional dereverb
/api/v1/split/demuser/ music Background music removal (speech stays)
/api/v1/split/stem_separator/ vocals, drum, piano, bass, electric_guitar, acoustic_guitar, synthesizer, strings, wind deep_extraction or clear_cut, optional dereverb, optional multivocal

Workflow per file: upload → split (async task) → poll check → download tracks → (optional) delete

Splitter model options: orion, perseus, phoenix, andromeda (or null = auto/latest)


Design Decisions

CUDA stays — for other features, not separation

The Docker image (nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04) and runtime: nvidia in stemengine.yml must remain. CUDA is still required for:

  • WhisperX lyric transcription (runs on GPU via _transcription_executor)
  • basic-pitch ONNX inference
  • Any future local models

The CPU_SEPARATION env var becomes irrelevant for LALA separation (network replaces GPU), but the hardware config should not be removed.

API key storage

A new env var LALAL_API_KEY (type: str, required for separation) is stored alongside existing secrets (SECRET_KEY, HF_TOKEN). It can be set in stemengine.yml or via the Settings UI once the settings router supports it.

Job executor routing

LALAL API calls are network I/O, not GPU work. They must NOT go through submit_gpu() (the single-worker GPU queue) — doing so would block all GPU-bound transcription while waiting for the network. Use submit() (general thread pool) instead, or a new dedicated _lala_executor = ThreadPoolExecutor(max_workers=3) to cap concurrent uploads.

Preset mixing (instrumental, static, dynamic)

The existing preset_service.py (make_instrumental, make_static, make_dynamic) mixes stem audio files with pydub. It is model-agnostic — it takes file paths and operates on whatever stems exist. This is fully reusable with LALA output. No changes needed here.

Project folder structure

Unchanged. LALA stems are downloaded into the same PROJECT_SUBFOLDERS["stems"] directory that local models produce. The rest of the app (pipeline, mix, transcribe) is unaffected.

Output format

LALAL API delivers stems as downloaded files (filename from Content-Disposition). These may be WAV or MP3 depending on LALA's output. After download, the existing format-conversion logic (ffmpeg via pydub) should be applied to normalize to the user's requested output_format.


Implementation Steps

Step 1 — New service: lalalai_separator.py

Create backend/services/lalalai_separator.py as a clean, self-contained module.

Contents:

  • Constants: LALAL_API_BASE = "https://www.lalal.ai/api/v1/"
  • Functions lifted and adapted from the reference lalalai_splitter.py:
    • upload_file(file_path, api_key) -> source_id
    • split_file(source_id, stem, api_key, *, splitter, extraction_level, dereverb, noise_cancelling, multivocal) -> task_id
    • poll_until_done(task_id, api_key, cancel_flag) -> result_dict
    • download_stems(result_dict, dest_dir) -> list[Path]
    • delete_source(source_id, api_key)
  • Top-level run_lala_separation(job_id, file_path, lala_model_id, output_format, *, preset_type, static_option, dynamic_levels, project_name_override) — mirrors the signature of run_separation() so callers are swappable.

Cancel support: poll_until_done accepts the job's threading.Event cancel flag (from job_manager._cancel_flags) and raises CancelledError if set between polls.

Error handling: Wrap all requests calls; on HTTP error or timeout, call jm.fail_job(job_id, message) and re-raise so the executor logs it.


Step 2 — New config section: LALA models

In backend/config.py, add:

# LALAL.AI v1 models
LALA_STEM_SEPARATOR_STEMS = [
    {"id": "lala_vocals",           "label": "LALA Vocals",           "group": "LALAL.AI", "lala_stem": "vocals"},
    {"id": "lala_drum",             "label": "LALA Drums",            "group": "LALAL.AI", "lala_stem": "drum"},
    {"id": "lala_bass",             "label": "LALA Bass",             "group": "LALAL.AI", "lala_stem": "bass"},
    {"id": "lala_piano",            "label": "LALA Piano",            "group": "LALAL.AI", "lala_stem": "piano"},
    {"id": "lala_electric_guitar",  "label": "LALA Electric Guitar",  "group": "LALAL.AI", "lala_stem": "electric_guitar"},
    {"id": "lala_acoustic_guitar",  "label": "LALA Acoustic Guitar",  "group": "LALAL.AI", "lala_stem": "acoustic_guitar"},
    {"id": "lala_synthesizer",      "label": "LALA Synthesizer",      "group": "LALAL.AI", "lala_stem": "synthesizer"},
    {"id": "lala_strings",          "label": "LALA Strings",          "group": "LALAL.AI", "lala_stem": "strings"},
    {"id": "lala_wind",             "label": "LALA Wind",             "group": "LALAL.AI", "lala_stem": "wind"},
    {"id": "lala_voice_clean",      "label": "LALA Voice Clean",      "group": "LALAL.AI", "lala_stem": "voice"},
    {"id": "lala_demuser",          "label": "LALA Demuser (Music)",  "group": "LALAL.AI", "lala_stem": "music"},
]

LALA_MODELS = LALA_STEM_SEPARATOR_STEMS
LALA_MODEL_IDS = {m["id"] for m in LALA_MODELS}

Add LALA_MODELS to ALL_MODELS. Add LALA_MODEL_IDS to ALL_MODEL_IDS.

Optional LALA-specific parameters (can be form fields or user settings defaults):

LALA_DEFAULT_SPLITTER = None           # auto = latest
LALA_DEFAULT_EXTRACTION_LEVEL = "deep_extraction"
LALA_DEFAULT_DEREVERB = False
LALA_DEFAULT_NOISE_CANCELLING = 1

Step 3 — Router changes: routers/separate.py

  1. Add optional form fields to POST /api/separate:

    • lala_splitter: Optional[str] = Form(None) — one of orion|perseus|phoenix|andromeda|None
    • lala_extraction_level: str = Form("deep_extraction")
    • lala_dereverb: bool = Form(False)
    • lala_noise_cancelling: int = Form(1)
    • lala_multivocal: Optional[str] = Form(None) — lead_back or None
  2. Routing logic in separate():

    if model in LALA_MODEL_IDS:
        jm.submit_lala(run_lala_separation, job_id, ...)
    else:
        jm.submit_gpu(run_separation, job_id, ...)
    
  3. Apply same routing to separate_existing() and separate_from_path().


Step 4 — Job manager: add submit_lala()

In backend/job_manager.py, add:

_lala_executor = ThreadPoolExecutor(max_workers=3, thread_name_prefix="lala")

def submit_lala(fn: Callable, *args, **kwargs):
    """Submit a LALAL.AI API separation task (network I/O, not GPU)."""
    _lala_executor.submit(fn, *args, **kwargs)

This keeps LALA network calls off the GPU queue so local GPU tasks (transcription) are never blocked waiting for remote API responses.


Step 5 — Config: env var + settings

In config.py:

LALAL_API_KEY = os.environ.get("LALAL_API_KEY", "")

In stemengine.yml:

- LALAL_API_KEY=${LALAL_API_KEY}

In the Settings UI (if applicable), surface LALAL_API_KEY as an encrypted credential field alongside the existing Subsonic and HuggingFace token fields.

The existing SECRET_KEY-based encryption in settings_service.py / the settings router should handle this with no structural changes — just add the field.


Step 6 — Dockerfile cleanup (optional, deferred)

Once LALA is the sole separator:

  • PyTorch, torchaudio, torchvision (CUDA wheels — ~3 GB) can be removed if no other feature needs them. Hold off until WhisperX is also evaluated — it currently depends on PyTorch.
  • All model-specific Python packages (open-unmix, demucs, ml_collections, etc.) can be removed.
  • Model volume mounts (/opt/bs_roformer_models, etc.) can be removed from stemengine.yml.
  • CPU_SEPARATION env var becomes vestigial but harmless to leave.

This step is explicitly deferred until the LALA path is confirmed working in production.


Step 7 — Frontend (minimal changes)

The frontend's model picker currently renders ALL_MODELS grouped by "group". Adding "group": "LALAL.AI" models to ALL_MODELS is sufficient — the group will appear automatically.

LALA-specific form fields (lala_splitter, lala_extraction_level, etc.) should be surfaced conditionally when the selected model's group is "LALAL.AI". This is a UI-only change in the separation form component (likely SeparatePanel or equivalent).


File Change Summary

File Action Notes
backend/services/lalalai_separator.py Create New LALA separation service
backend/config.py Edit Add LALA_MODELS, LALAL_API_KEY
backend/routers/separate.py Edit Add LALA form fields, routing branch
backend/job_manager.py Edit Add _lala_executor, submit_lala()
stemengine.yml Edit Add LALAL_API_KEY env var
backend/services/separator_service.py No change yet Keep until LALA confirmed
backend/services/preset_service.py No change Reused as-is
Dockerfile Deferred Cleanup after LALA confirmed working

What is NOT changing

  • job_manager.py executor for GPU (_gpu_executor) — still used for WhisperX transcription
  • preset_service.py — mix logic is model-agnostic, works on file paths
  • Project folder structure — stems land in the same place
  • Docker CUDA base image — required for WhisperX
  • nvidia runtime in stemengine.yml — required for WhisperX
  • All non-separation routers (transcribe, lyric_transcribe, lrc, subsonic, pipeline, etc.)

Risks & Notes

  • LALAL.AI API key required at runtime. If LALAL_API_KEY is empty and a LALA model is selected, lalalai_separator.py should fail the job immediately with a clear error message rather than getting an HTTP 401 partway through an upload.
  • Upload cost. LALAL.AI charges credits per minute of audio. The plan does not implement source caching/reuse (source_id reuse across jobs for the same file) in v1 — that is a potential future optimization.
  • File size limits. LALAL.AI imposes upload limits per license tier. Large files may need to be rejected before upload with an appropriate HTTP 413 or job error.
  • Polling interval. The reference uses 5-second polling. This is fine for background jobs but should respect the cancel flag between polls.
  • Output filename. LALA returns filenames via Content-Disposition. The download function must parse this (already implemented in the reference code) and then rename to match the StemEngine stem-naming convention: "Artist - Title (vocals).mp3".