Files
stemengine_old/docs/DEVELOPER_GUIDE.md
T
2026-03-26 15:14:41 +09:00

20 KiB
Raw Blame History

StemEngine Developer Guide

1) Architecture overview

StemEngine is a full-stack system:

  • Frontend: React + TypeScript + Vite
  • Backend: FastAPI + Python services
  • Sidecars: optional model-specific services (for example Parakeet, Omnizart)
  • Persistence: project directories, job output directories, DB-backed metadata

High-level flow:

  1. UI submits jobs to API.
  2. Backend routes requests to service layers.
  3. Job manager tracks async progress and status.
  4. Outputs are written to configured project/output paths.

2) Repository map

  • backend/ API, job orchestration, services, tests
  • frontend/ React UI
  • docker/ compose and deployment assets
  • lyrics-sync/ alignment utilities
  • parakeet-sidecar/ Parakeet integration runtime

3) Backend API surface (router prefixes)

Key API groups:

  • /api/separate
  • /api/transcribe
  • /api/lyric-transcribe
  • /api/lrc
  • /api/projects
  • /api/pipeline
  • /api/jobs
  • /api/models
  • /api/system
  • /api/settings
  • /api/subsonic
  • /api/abc-convert

4) Local development workflows

Backend

  • Build image: make build
  • Run tests: make test
  • Smoke tests: make test-smoke
  • Migrations: make alembic-upgrade

Frontend

From frontend/:

  • Install: npm install
  • Dev server: npm run dev
  • Type/lint/build check: npm run check

5) Coding and change strategy

When adding a feature:

  1. Define API contract changes first.
  2. Add/update service layer behavior.
  3. Extend router validations.
  4. Update frontend types + UI behavior.
  5. Add or update tests.
  6. Update user/developer docs in docs/.

6) Operational notes

  • GPU safety limits are centrally important for stability.
  • Whisper/WhisperX/sidecar paths can have different feature support.
  • Lyric line segmentation now uses karaoke-lyrics-processor by default in auto mode, with a legacy fallback path controlled by STEMENGINE_LYRIC_SEGMENTER.
  • Keep long-running tasks cancellable and progress-reporting through the job manager.

7) Documentation rules for contributors

  • Every user-facing feature change must update USER_GUIDE.md.
  • Every architecture/API/process change must update this guide.
  • Large implementation work should include a dated plan or ADR-style note.
  • Every PR must complete documentation-impact checks via ../.github/PULL_REQUEST_TEMPLATE.md.

8) Documentation governance (Phase 3)

9) Suggested next technical docs

10) GaboxR67 model expansion (implementation baseline)

This section is the current implementation baseline and rollout contract.

Scope source

  • Hugging Face publisher: GaboxR67
  • Primary repository: GaboxR67/MelBandRoformers
  • Relevant top-level groups:
    • melbandroformers/
    • bsroformers/

Current baseline in StemEngine

  • mel_band_karaoke_gabox is already integrated.
  • Existing runtime paths for Gabox are in:
    • backend model catalog
    • model downloader mappings
    • specialized MelBand separator classes
    • karaoke routing + fallback capability checks

Implemented objective

Gabox coverage is expanded while preserving stability guarantees:

  1. no silent stem-layout regressions,
  2. no karaoke fallback to non-karaoke-capable models,
  3. small-model support as a first-class opt-in tier.

Rollout tiers

Tier Inclusion rule Runtime role
Tier A (core) Stable Gabox models with verified stem layout and repeatable outputs mel_band_vocals_gabox
Tier B (small) Lower-footprint/short-window Gabox variants (for constrained GPUs) mel_band_karaoke_gabox_small (opt-in)
Tier C (experimental) experimental/* or unstable checkpoints mel_band_karaoke_gabox_v2_exp (opt-in)

"Small model" policy

Small models are explicitly included in scope. For each candidate, documentation and metadata must include:

  • expected VRAM profile,
  • intended track duration envelope,
  • expected quality tradeoff,
  • whether the model is eligible for karaoke fallback.

Required metadata per Gabox model

Each integrated Gabox model must declare, at minimum:

  • canonical internal ID,
  • public label,
  • source repository path,
  • checkpoint + config filenames,
  • stem layout (vocals/instrumental, lead/backing_instrumental, etc.),
  • capability class (karaoke-capable yes/no),
  • tier (core, small, experimental).

Acceptance gates (ongoing)

  1. Download gate: checkpoint + config download succeeds from configured source.
  2. Load gate: model initializes on target runtime image.
  3. Layout gate: produced stems match declared layout and naming contract.
  4. Fallback gate: karaoke jobs only fall back to karaoke-capable models.
  5. Stability gate: at least one long-track and one short-track test per tier.
  6. Docs gate: README + config + troubleshooting docs updated together.

Out-of-scope (current)

  • Dynamic quality auto-selection based on audio content.
  • Multi-hop fallback optimization by metric score.
  • Automatic promotion of experimental models to core tier.

Implementation sequence (planned)

  1. Catalog + metadata additions.
  2. Downloader/path wiring.
  3. Separator routing and stem-layout enforcement.
  4. Tier labels and UI exposure.
  5. Validation matrix run + docs finalization.

11) Karaoke pipeline plan (instrumental + lead + backing + ELRC + ASS)

A dedicated design-plan document now defines the target karaoke pipeline that outputs:

  • instrumental
  • lead_vocals
  • backing_vocals
  • .elrc
  • timed .ass (based on lead-vocal alignment)

See:

The plan includes:

  • stage graph and retry boundaries,
  • API contract proposal,
  • artifact layout contract,
  • fallback policy,
  • phased rollout and acceptance criteria.

12) Checkpoint format note (.ckpt vs .safetensors)

StemEngine currently loads RoFormer-family models using model-specific YAML configs plus PyTorch checkpoints (.ckpt and similar) through torch.load(...)-based runtime paths.

Important distinctions:

  • safetensors is a checkpoint serialization format, not a separate separator architecture.
  • A MelBand or BS-RoFormer safetensors file is typically the same trained network weights exported into a safer container format.
  • safetensors can improve load safety and may slightly reduce load overhead in some runtimes.
  • safetensors does not inherently reduce parameter count, separation VRAM, or forward-pass compute.

Current StemEngine implication:

  • RoFormer model integrations should assume .ckpt + YAML unless a loader explicitly adds safetensors support.
  • Supporting a safetensors release requires validating:
    • weight key naming compatibility,
    • checkpoint/config pairing,
    • deserialization path,
    • any packaging differences from ComfyUI-oriented exports.

13) Long-track chunked separation (implementation)

Chunked separation is fully implemented for the RoFormer family (MelBand Karaoke, MelBand 4-stem, BS-RoFormer) and MDX-Net and SCNet separators. Design rationale is in LONG_TRACK_CHUNKING_PLAN.md.

New module — backend/services/separation/chunking.py

Symbol Purpose
ChunkPlan Dataclass: boundaries, sample rate, hop, overlap
plan_chunks(total, sr, chunk_secs, overlap_secs) Returns a ChunkPlan for the given duration
should_chunk(duration, mode, threshold) Activation policy: auto | always | off
extract_chunk(audio, start, end) Zero-padded slice of (channels, samples) array
overlap_add_reconstruct(chunks, plan, stems) Crossfade OLA into full-length stems

The reconstruction uses a trapezoidal window whose fade width equals overlap_samples. Adjacent fade-out + fade-in sum exactly to 1.0 — no amplitude discontinuity at seam boundaries.

New separator method — separate_into_parts_chunked()

RoFormer family (added in Phase 2):

  • MelBandSeparatorBase (mel_band_specialized_separator.py) — inherited by all karaoke/vocals/dereverb/becruily subclasses
  • MelBandRoformerSeparator (mel_band_roformer_separator.py)
  • BSRoformerSeparator (bs_roformer_separator.py)

ONNX/PyTorch separators (added in Phase 3):

  • MDXNetSeparator (mdxnet_separator.py)
  • SCNetSeparator (scnet_separator.py)

Signature (RoFormer variants):

def separate_into_parts_chunked(
    self, input_path, output_dir, chunk_plan,
    batch_size_override=None, num_overlap_override=None,
    progress_cb=None,
) -> Dict[str, str]

Signature (MDX-Net):

def separate_into_parts_chunked(
    self, input_path, output_dir, chunk_plan, progress_cb=None,
) -> Dict[str, str]

Signature (SCNet):

def separate_into_parts_chunked(
    self, input_path, output_dir, chunk_plan,
    chunk_size_override=None, num_overlap_override=None, progress_cb=None,
) -> Dict[str, str]

Behaviour: loads audio and model once; iterates plan.chunks; calls _demix() on each chunk; releases CUDA cache between chunks; reconstructs with overlap_add_reconstruct().

Service wiring — separator_service.py

Five inference functions updated with a chunking path that takes priority over the eco-params guardrail:

  • _run_mel_band_karaoke — chunking path + OOM falls through to _karaoke_oom_fallback
  • _run_mel_band_roformer — chunking path, else guardrail + OOM ladder
  • _run_bsroformer — chunking path, else 2-attempt OOM retry
  • _run_mdxnet — chunking path, else standard separation
  • _run_scnet — chunking path with 2-attempt OOM retry, else standard + OOM retry

Additional adapter-backed families:

  • _run_apollo — service-level file-chunk adapter when chunking is active
  • _run_aspiration — service-level file-chunk adapter when chunking is active
  • _run_bandit — service-level file-chunk adapter when chunking is active

These families currently expose only file-based separate_into_parts() APIs. separator_service.py now provides _separate_into_parts_chunked_via_files() to apply the same chunk-plan policy (plan_chunks + extract_chunk + overlap_add_reconstruct) and map global segment overrides to effective runtime chunk duration.

When _use_chunking is True the long-track eco-params guardrail is skipped — per-chunk VRAM is bounded by LONG_TRACK_CHUNK_DURATION_SECS, not track length.

Config env vars (new)

Variable Default Description
LONG_TRACK_CHUNKING_MODE auto auto | always | off
LONG_TRACK_CHUNK_DURATION_SECS 180 Chunk size (seconds)
LONG_TRACK_CHUNK_OVERLAP_SECS 10 Crossfade width (seconds)
DEMUCS_SEGMENT_OVERRIDE null Override Demucs segment (seconds); unset = auto-scale
DEMUCS_OVERLAP_OVERRIDE null Override Demucs overlap factor (0.0–0.5); unset = 0.25

14) Phase 5: Testing, tuning, and troubleshooting

Comprehensive test coverage, GPU-class-specific tuning, and production troubleshooting guidance.

New test module — backend/tests/test_chunking.py

Unittest suite covering chunking infrastructure:

Test Class Coverage
TestChunkPlanning Chunk boundary math, edge cases (short audio, exact multiples)
TestChunkActivationPolicy should_chunk() decision logic for all modes
TestExtractChunk Time-domain extraction with padding
TestOverlapAddReconstruction Trapezoidal OLA window, amplitude preservation, multi-chunk
TestIntegrationChunking End-to-end pipeline (plan → extract → process → reconstruct)

Run tests:

cd backend
python3 -m unittest tests.test_chunking -v

GPU-class tuning defaults

Practical tuning guidance by VRAM class:

VRAM class Suggested chunk duration Suggested overlap Mode
≤ 4 GB 120 s 5–10 s auto
6–8 GB 180 s 10 s auto
≥ 12 GB 180–300 s or off 10 s auto or off

For Demucs long tracks, use segment auto-scaling (see below) or set DEMUCS_SEGMENT_OVERRIDE explicitly.

Troubleshooting focus areas

Most common issues and first actions:

Issue Diagnosis Remediation
CUDA OOM Chunk size too large Reduce LONG_TRACK_CHUNK_DURATION_SECS
Slow/hanging job Disk I/O bottleneck Move JOBS_DIR to SSD
Seam artifacts (clicks) Overlap too short Increase LONG_TRACK_CHUNK_OVERLAP_SECS
Demucs segment failure Segment size exceeds VRAM Enable auto-scaling or set DEMUCS_SEGMENT_OVERRIDE=4
Inconsistent results GPU memory fragmentation (rare) Restart container; verify model checksums
File I/O errors Permissions or disk full Fix mount, free space, chmod
Timing misalignment Duration estimation wrong Verify input with ffprobe

Phase 3 enhancements — Demucs long-track tuning

Updated _run_demucs with duration-aware segment auto-scaling:

# Auto-scale formula:
if duration < 120s:   segment = 12  # Quality-biased (default Demucs)
elif duration < 480s: segment = 8   # Balanced
else:                 segment = 4   # VRAM-biased

New config vars (added to config.py):

  • DEMUCS_SEGMENT_OVERRIDE: Force specific segment size (disables auto-scaling)
  • DEMUCS_OVERLAP_OVERRIDE: Adjust overlap factor for VRAM vs. quality trade-off

Integrated into _run_demucs with progress reporting.


15) Phase 4: Request-level runtime overrides (implemented)

Separation endpoints now support optional per-request runtime overrides so callers can tune inference without changing global env vars.

API surface

Implemented for:

  • POST /api/separate
  • POST /api/separate/existing
  • POST /api/separate/from-path

New optional form fields:

  • chunking_mode_override (auto | always | off)
  • chunk_duration_secs_override, chunk_overlap_secs_override
  • roformer_batch_size_override, roformer_num_overlap_override
  • demucs_segment_override, demucs_overlap_override
  • scnet_chunk_size_override, scnet_num_overlap_override

Service integration

  • run_separation() now accepts all override fields.
  • Power preset values are deep-copied then overridden per request.
  • Chunking settings are resolved per request (mode, duration, overlap) and passed through model runners.
  • Completed jobs include both:
    • result.diagnostics (effective runtime details)
    • result.request_overrides (echo of explicitly supplied overrides)

Validation rules

  • Positive integer checks for batch/chunk/segment values
  • demucs_overlap_override range: 0.0 to 0.95
  • chunking_mode_override constrained to auto, always, off

16) Phase 1: Global power policy schema (implemented)

Phase 1 introduces a normalized global power policy object while keeping existing family-specific runtime controls unchanged.

Backend implementation

  • Added GlobalPowerPolicy schema in:
    • backend/services/separation/power_presets.py
  • Added builder:
    • build_global_power_policy(tier, power_opts)
  • Current normalized fields:
    • tier
    • segment_secs
    • overlap
    • batch_size
    • source metadata (segment_source, overlap_source, batch_source)

Runtime integration

  • run_separation() now computes global_power_policy after preset + request overrides are resolved.
  • Completed jobs now include:
    • result.diagnostics.effective_power_policy

Compatibility contract

  • Existing model-family tuning remains authoritative for execution.
  • Phase 1 is diagnostics/schema-first and backward-compatible.
  • No breaking API changes; no removal of existing override fields.

Next phase

Phase 2/3 work should add family adapters and optional global request override fields (global_segment_secs_override, etc.) while preserving current fields.


17) Phase 2: Global override wiring (implemented)

Phase 2 adds request-level global power-policy overrides and maps them onto existing family knobs where supported.

API/validation

Added optional form fields to all separation endpoints:

  • global_power_tier_override
  • global_segment_secs_override
  • global_overlap_override
  • global_batch_size_override

Validation rules:

  • global_power_tier_override: eco / standard / performance
  • global_segment_secs_override: 0.1 .. 3600.0
  • global_overlap_override: 0.0 .. 16.0
  • global_batch_size_override: >= 1

Runtime mapping (current supported families)

  • Demucs: global_segment_secs_override → demucs.segment
  • MelBand family: global_segment_secs_override → MelBand dim_t converter (dim_t ≈ seconds * 44100 / 1024 + 1)
  • SCNet: global_segment_secs_override → scnet.chunk_size (samples)
  • SepACap / SepReformer: global_segment_secs_override → chunk_samples
  • RoFormer family: global_batch_size_override → roformer.batch_size
  • Overlap policy:
    • factor-style for Demucs (demucs.overlap)
    • count-style for RoFormer/SCNet (num_overlap)

Precedence contract

  1. Base tier preset (sep_power_preset or global_power_tier_override)
  2. Global override fields (Phase 2)
  3. Family-specific override fields (Phase 4)

This keeps existing advanced controls authoritative while enabling a global first-pass policy.


18) Phase 3: Adapter coverage expansion (implemented)

Phase 3 extends global segment mapping beyond the initial families.

Implemented coverage

  • MDX-Net adapter:
    • global_segment_secs_override maps to MDX chunk-duration policy.
  • MedleyVox adapter:
    • global_segment_secs_override maps to chunk_secs_override in MedleyVoxSeparator (internal streaming chunk length).
  • Apollo / Aspiration / Bandit adapter path:
    • these families now use a service-level file chunk adapter when chunking is active (plan_chunks + per-chunk separation + overlap-add reconstruction).
    • global segment maps to adapter chunk duration for these families.
  • Chunking fallback bridge:
    • if chunk_duration_secs_override is absent, global segment provides the chunk-duration fallback hint.

Diagnostics visibility

  • effective_power_policy.adapter_targets reports families receiving adapter-mapped policy values.

19) Phase 4: WebUI global controls exposure (implemented)

WebUI now exposes global override controls in the Separator workflow.

UI implementation

  • frontend/src/pages/Separator.tsx
    • Added "Global power overrides" panel.
    • Added fields:
      • global_power_tier_override
      • global_segment_secs_override
      • global_overlap_override
      • global_batch_size_override
    • Added client-side range validation aligned with backend validation.
    • Persisted values in localStorage for reuse.

API client wiring

  • frontend/src/api/client.ts
    • submitSeparate() and submitSeparateFromPath() now append global override fields to multipart requests when provided.

Auto pipeline parity

  • frontend/src/components/AutoPipelineDialog.tsx
    • Separation stage now forwards stored global override values so auto pipeline behavior matches manual separation policy.