20 KiB
StemEngine Developer Guide
1) Architecture overview
StemEngine is a full-stack system:
- Frontend: React + TypeScript + Vite
- Backend: FastAPI + Python services
- Sidecars: optional model-specific services (for example Parakeet, Omnizart)
- Persistence: project directories, job output directories, DB-backed metadata
High-level flow:
- UI submits jobs to API.
- Backend routes requests to service layers.
- Job manager tracks async progress and status.
- Outputs are written to configured project/output paths.
2) Repository map
backend/API, job orchestration, services, testsfrontend/React UIdocker/compose and deployment assetslyrics-sync/alignment utilitiesparakeet-sidecar/Parakeet integration runtime
3) Backend API surface (router prefixes)
Key API groups:
/api/separate/api/transcribe/api/lyric-transcribe/api/lrc/api/projects/api/pipeline/api/jobs/api/models/api/system/api/settings/api/subsonic/api/abc-convert
4) Local development workflows
Backend
- Build image:
make build - Run tests:
make test - Smoke tests:
make test-smoke - Migrations:
make alembic-upgrade
Frontend
From frontend/:
- Install:
npm install - Dev server:
npm run dev - Type/lint/build check:
npm run check
5) Coding and change strategy
When adding a feature:
- Define API contract changes first.
- Add/update service layer behavior.
- Extend router validations.
- Update frontend types + UI behavior.
- Add or update tests.
- Update user/developer docs in
docs/.
6) Operational notes
- GPU safety limits are centrally important for stability.
- Whisper/WhisperX/sidecar paths can have different feature support.
- Lyric line segmentation now uses
karaoke-lyrics-processorby default inautomode, with a legacy fallback path controlled bySTEMENGINE_LYRIC_SEGMENTER. - Keep long-running tasks cancellable and progress-reporting through the job manager.
7) Documentation rules for contributors
- Every user-facing feature change must update USER_GUIDE.md.
- Every architecture/API/process change must update this guide.
- Large implementation work should include a dated plan or ADR-style note.
- Every PR must complete documentation-impact checks via ../.github/PULL_REQUEST_TEMPLATE.md.
8) Documentation governance (Phase 3)
- Governance process: DOC_GOVERNANCE.md
- Release verification: RELEASE_CHECKLIST.md
- Feature template: templates/FEATURE_DOC_TEMPLATE.md
- Migration template: templates/MIGRATION_NOTE_TEMPLATE.md
9) Suggested next technical docs
- API endpoint reference with request/response examples (implemented: API_REFERENCE.md)
- Troubleshooting matrix for GPU/runtime/sidecar failures (implemented: TROUBLESHOOTING_MATRIX.md)
- Configuration usage guide (implemented: CONFIGURATION_REFERENCE.md)
- Docs governance + templates (implemented: DOC_GOVERNANCE.md)
- Data model and migration policy
- Job lifecycle/state machine documentation
10) GaboxR67 model expansion (implementation baseline)
This section is the current implementation baseline and rollout contract.
Scope source
- Hugging Face publisher:
GaboxR67 - Primary repository:
GaboxR67/MelBandRoformers - Relevant top-level groups:
melbandroformers/bsroformers/
Current baseline in StemEngine
mel_band_karaoke_gaboxis already integrated.- Existing runtime paths for Gabox are in:
- backend model catalog
- model downloader mappings
- specialized MelBand separator classes
- karaoke routing + fallback capability checks
Implemented objective
Gabox coverage is expanded while preserving stability guarantees:
- no silent stem-layout regressions,
- no karaoke fallback to non-karaoke-capable models,
- small-model support as a first-class opt-in tier.
Rollout tiers
| Tier | Inclusion rule | Runtime role |
|---|---|---|
| Tier A (core) | Stable Gabox models with verified stem layout and repeatable outputs | mel_band_vocals_gabox |
| Tier B (small) | Lower-footprint/short-window Gabox variants (for constrained GPUs) | mel_band_karaoke_gabox_small (opt-in) |
| Tier C (experimental) | experimental/* or unstable checkpoints |
mel_band_karaoke_gabox_v2_exp (opt-in) |
"Small model" policy
Small models are explicitly included in scope. For each candidate, documentation and metadata must include:
- expected VRAM profile,
- intended track duration envelope,
- expected quality tradeoff,
- whether the model is eligible for karaoke fallback.
Required metadata per Gabox model
Each integrated Gabox model must declare, at minimum:
- canonical internal ID,
- public label,
- source repository path,
- checkpoint + config filenames,
- stem layout (
vocals/instrumental,lead/backing_instrumental, etc.), - capability class (
karaoke-capableyes/no), - tier (
core,small,experimental).
Acceptance gates (ongoing)
- Download gate: checkpoint + config download succeeds from configured source.
- Load gate: model initializes on target runtime image.
- Layout gate: produced stems match declared layout and naming contract.
- Fallback gate: karaoke jobs only fall back to karaoke-capable models.
- Stability gate: at least one long-track and one short-track test per tier.
- Docs gate: README + config + troubleshooting docs updated together.
Out-of-scope (current)
- Dynamic quality auto-selection based on audio content.
- Multi-hop fallback optimization by metric score.
- Automatic promotion of experimental models to core tier.
Implementation sequence (planned)
- Catalog + metadata additions.
- Downloader/path wiring.
- Separator routing and stem-layout enforcement.
- Tier labels and UI exposure.
- Validation matrix run + docs finalization.
11) Karaoke pipeline plan (instrumental + lead + backing + ELRC + ASS)
A dedicated design-plan document now defines the target karaoke pipeline that outputs:
instrumentallead_vocalsbacking_vocals.elrc- timed
.ass(based on lead-vocal alignment)
See:
The plan includes:
- stage graph and retry boundaries,
- API contract proposal,
- artifact layout contract,
- fallback policy,
- phased rollout and acceptance criteria.
12) Checkpoint format note (.ckpt vs .safetensors)
StemEngine currently loads RoFormer-family models using model-specific YAML configs plus PyTorch checkpoints (.ckpt and similar) through torch.load(...)-based runtime paths.
Important distinctions:
safetensorsis a checkpoint serialization format, not a separate separator architecture.- A MelBand or BS-RoFormer
safetensorsfile is typically the same trained network weights exported into a safer container format. safetensorscan improve load safety and may slightly reduce load overhead in some runtimes.safetensorsdoes not inherently reduce parameter count, separation VRAM, or forward-pass compute.
Current StemEngine implication:
- RoFormer model integrations should assume
.ckpt+ YAML unless a loader explicitly addssafetensorssupport. - Supporting a
safetensorsrelease requires validating:- weight key naming compatibility,
- checkpoint/config pairing,
- deserialization path,
- any packaging differences from ComfyUI-oriented exports.
13) Long-track chunked separation (implementation)
Chunked separation is fully implemented for the RoFormer family (MelBand Karaoke, MelBand 4-stem, BS-RoFormer) and MDX-Net and SCNet separators. Design rationale is in LONG_TRACK_CHUNKING_PLAN.md.
New module — backend/services/separation/chunking.py
| Symbol | Purpose |
|---|---|
ChunkPlan |
Dataclass: boundaries, sample rate, hop, overlap |
plan_chunks(total, sr, chunk_secs, overlap_secs) |
Returns a ChunkPlan for the given duration |
should_chunk(duration, mode, threshold) |
Activation policy: auto | always | off |
extract_chunk(audio, start, end) |
Zero-padded slice of (channels, samples) array |
overlap_add_reconstruct(chunks, plan, stems) |
Crossfade OLA into full-length stems |
The reconstruction uses a trapezoidal window whose fade width equals
overlap_samples. Adjacent fade-out + fade-in sum exactly to 1.0 — no
amplitude discontinuity at seam boundaries.
New separator method — separate_into_parts_chunked()
RoFormer family (added in Phase 2):
MelBandSeparatorBase(mel_band_specialized_separator.py) — inherited by all karaoke/vocals/dereverb/becruily subclassesMelBandRoformerSeparator(mel_band_roformer_separator.py)BSRoformerSeparator(bs_roformer_separator.py)
ONNX/PyTorch separators (added in Phase 3):
MDXNetSeparator(mdxnet_separator.py)SCNetSeparator(scnet_separator.py)
Signature (RoFormer variants):
def separate_into_parts_chunked(
self, input_path, output_dir, chunk_plan,
batch_size_override=None, num_overlap_override=None,
progress_cb=None,
) -> Dict[str, str]
Signature (MDX-Net):
def separate_into_parts_chunked(
self, input_path, output_dir, chunk_plan, progress_cb=None,
) -> Dict[str, str]
Signature (SCNet):
def separate_into_parts_chunked(
self, input_path, output_dir, chunk_plan,
chunk_size_override=None, num_overlap_override=None, progress_cb=None,
) -> Dict[str, str]
Behaviour: loads audio and model once; iterates plan.chunks; calls _demix() on each chunk; releases CUDA cache between chunks; reconstructs with overlap_add_reconstruct().
Service wiring — separator_service.py
Five inference functions updated with a chunking path that takes priority over the eco-params guardrail:
_run_mel_band_karaoke— chunking path + OOM falls through to_karaoke_oom_fallback_run_mel_band_roformer— chunking path, else guardrail + OOM ladder_run_bsroformer— chunking path, else 2-attempt OOM retry_run_mdxnet— chunking path, else standard separation_run_scnet— chunking path with 2-attempt OOM retry, else standard + OOM retry
Additional adapter-backed families:
_run_apollo— service-level file-chunk adapter when chunking is active_run_aspiration— service-level file-chunk adapter when chunking is active_run_bandit— service-level file-chunk adapter when chunking is active
These families currently expose only file-based separate_into_parts() APIs.
separator_service.py now provides _separate_into_parts_chunked_via_files() to
apply the same chunk-plan policy (plan_chunks + extract_chunk +
overlap_add_reconstruct) and map global segment overrides to effective runtime
chunk duration.
When _use_chunking is True the long-track eco-params guardrail is skipped — per-chunk VRAM is bounded by LONG_TRACK_CHUNK_DURATION_SECS, not track length.
Config env vars (new)
| Variable | Default | Description |
|---|---|---|
LONG_TRACK_CHUNKING_MODE |
auto |
auto | always | off |
LONG_TRACK_CHUNK_DURATION_SECS |
180 |
Chunk size (seconds) |
LONG_TRACK_CHUNK_OVERLAP_SECS |
10 |
Crossfade width (seconds) |
DEMUCS_SEGMENT_OVERRIDE |
null |
Override Demucs segment (seconds); unset = auto-scale |
DEMUCS_OVERLAP_OVERRIDE |
null |
Override Demucs overlap factor (0.0–0.5); unset = 0.25 |
14) Phase 5: Testing, tuning, and troubleshooting
Comprehensive test coverage, GPU-class-specific tuning, and production troubleshooting guidance.
New test module — backend/tests/test_chunking.py
Unittest suite covering chunking infrastructure:
| Test Class | Coverage |
|---|---|
TestChunkPlanning |
Chunk boundary math, edge cases (short audio, exact multiples) |
TestChunkActivationPolicy |
should_chunk() decision logic for all modes |
TestExtractChunk |
Time-domain extraction with padding |
TestOverlapAddReconstruction |
Trapezoidal OLA window, amplitude preservation, multi-chunk |
TestIntegrationChunking |
End-to-end pipeline (plan → extract → process → reconstruct) |
Run tests:
cd backend
python3 -m unittest tests.test_chunking -v
GPU-class tuning defaults
Practical tuning guidance by VRAM class:
| VRAM class | Suggested chunk duration | Suggested overlap | Mode |
|---|---|---|---|
| ≤ 4 GB | 120 s | 5–10 s | auto |
| 6–8 GB | 180 s | 10 s | auto |
| ≥ 12 GB | 180–300 s or off | 10 s | auto or off |
For Demucs long tracks, use segment auto-scaling (see below) or set
DEMUCS_SEGMENT_OVERRIDE explicitly.
Troubleshooting focus areas
Most common issues and first actions:
| Issue | Diagnosis | Remediation |
|---|---|---|
| CUDA OOM | Chunk size too large | Reduce LONG_TRACK_CHUNK_DURATION_SECS |
| Slow/hanging job | Disk I/O bottleneck | Move JOBS_DIR to SSD |
| Seam artifacts (clicks) | Overlap too short | Increase LONG_TRACK_CHUNK_OVERLAP_SECS |
| Demucs segment failure | Segment size exceeds VRAM | Enable auto-scaling or set DEMUCS_SEGMENT_OVERRIDE=4 |
| Inconsistent results | GPU memory fragmentation (rare) | Restart container; verify model checksums |
| File I/O errors | Permissions or disk full | Fix mount, free space, chmod |
| Timing misalignment | Duration estimation wrong | Verify input with ffprobe |
Phase 3 enhancements — Demucs long-track tuning
Updated _run_demucs with duration-aware segment auto-scaling:
# Auto-scale formula:
if duration < 120s: segment = 12 # Quality-biased (default Demucs)
elif duration < 480s: segment = 8 # Balanced
else: segment = 4 # VRAM-biased
New config vars (added to config.py):
DEMUCS_SEGMENT_OVERRIDE: Force specific segment size (disables auto-scaling)DEMUCS_OVERLAP_OVERRIDE: Adjust overlap factor for VRAM vs. quality trade-off
Integrated into _run_demucs with progress reporting.
Related docs
15) Phase 4: Request-level runtime overrides (implemented)
Separation endpoints now support optional per-request runtime overrides so callers can tune inference without changing global env vars.
API surface
Implemented for:
POST /api/separatePOST /api/separate/existingPOST /api/separate/from-path
New optional form fields:
chunking_mode_override(auto|always|off)chunk_duration_secs_override,chunk_overlap_secs_overrideroformer_batch_size_override,roformer_num_overlap_overridedemucs_segment_override,demucs_overlap_overridescnet_chunk_size_override,scnet_num_overlap_override
Service integration
run_separation()now accepts all override fields.- Power preset values are deep-copied then overridden per request.
- Chunking settings are resolved per request (
mode,duration,overlap) and passed through model runners. - Completed jobs include both:
result.diagnostics(effective runtime details)result.request_overrides(echo of explicitly supplied overrides)
Validation rules
- Positive integer checks for batch/chunk/segment values
demucs_overlap_overriderange:0.0to0.95chunking_mode_overrideconstrained toauto,always,off
16) Phase 1: Global power policy schema (implemented)
Phase 1 introduces a normalized global power policy object while keeping existing family-specific runtime controls unchanged.
Backend implementation
- Added
GlobalPowerPolicyschema in:backend/services/separation/power_presets.py
- Added builder:
build_global_power_policy(tier, power_opts)
- Current normalized fields:
tiersegment_secsoverlapbatch_size- source metadata (
segment_source,overlap_source,batch_source)
Runtime integration
run_separation()now computesglobal_power_policyafter preset + request overrides are resolved.- Completed jobs now include:
result.diagnostics.effective_power_policy
Compatibility contract
- Existing model-family tuning remains authoritative for execution.
- Phase 1 is diagnostics/schema-first and backward-compatible.
- No breaking API changes; no removal of existing override fields.
Next phase
Phase 2/3 work should add family adapters and optional global request override
fields (global_segment_secs_override, etc.) while preserving current fields.
17) Phase 2: Global override wiring (implemented)
Phase 2 adds request-level global power-policy overrides and maps them onto existing family knobs where supported.
API/validation
Added optional form fields to all separation endpoints:
global_power_tier_overrideglobal_segment_secs_overrideglobal_overlap_overrideglobal_batch_size_override
Validation rules:
global_power_tier_override:eco/standard/performanceglobal_segment_secs_override:0.1 .. 3600.0global_overlap_override:0.0 .. 16.0global_batch_size_override:>= 1
Runtime mapping (current supported families)
- Demucs:
global_segment_secs_override→demucs.segment - MelBand family:
global_segment_secs_override→ MelBanddim_tconverter (dim_t ≈ seconds * 44100 / 1024 + 1) - SCNet:
global_segment_secs_override→scnet.chunk_size(samples) - SepACap / SepReformer:
global_segment_secs_override→chunk_samples - RoFormer family:
global_batch_size_override→roformer.batch_size - Overlap policy:
- factor-style for Demucs (
demucs.overlap) - count-style for RoFormer/SCNet (
num_overlap)
- factor-style for Demucs (
Precedence contract
- Base tier preset (
sep_power_presetorglobal_power_tier_override) - Global override fields (Phase 2)
- Family-specific override fields (Phase 4)
This keeps existing advanced controls authoritative while enabling a global first-pass policy.
18) Phase 3: Adapter coverage expansion (implemented)
Phase 3 extends global segment mapping beyond the initial families.
Implemented coverage
- MDX-Net adapter:
global_segment_secs_overridemaps to MDX chunk-duration policy.
- MedleyVox adapter:
global_segment_secs_overridemaps tochunk_secs_overrideinMedleyVoxSeparator(internal streaming chunk length).
- Apollo / Aspiration / Bandit adapter path:
- these families now use a service-level file chunk adapter when chunking is
active (
plan_chunks+ per-chunk separation + overlap-add reconstruction). - global segment maps to adapter chunk duration for these families.
- these families now use a service-level file chunk adapter when chunking is
active (
- Chunking fallback bridge:
- if
chunk_duration_secs_overrideis absent, global segment provides the chunk-duration fallback hint.
- if
Diagnostics visibility
effective_power_policy.adapter_targetsreports families receiving adapter-mapped policy values.
19) Phase 4: WebUI global controls exposure (implemented)
WebUI now exposes global override controls in the Separator workflow.
UI implementation
frontend/src/pages/Separator.tsx- Added "Global power overrides" panel.
- Added fields:
global_power_tier_overrideglobal_segment_secs_overrideglobal_overlap_overrideglobal_batch_size_override
- Added client-side range validation aligned with backend validation.
- Persisted values in localStorage for reuse.
API client wiring
frontend/src/api/client.tssubmitSeparate()andsubmitSeparateFromPath()now append global override fields to multipart requests when provided.
Auto pipeline parity
frontend/src/components/AutoPipelineDialog.tsx- Separation stage now forwards stored global override values so auto pipeline behavior matches manual separation policy.