Files
stemengine_old/docs/GPU_TUNING_GUIDE.md
T
2026-03-26 15:14:41 +09:00

6.6 KiB
Raw Blame History

GPU Class Tuning & Recommendations (Phase 5)

This guide provides hardware-specific tuning recommendations for StemEngine separation across consumer, professional, and server-class GPUs.

Overview

StemEngine's chunking system allows dynamic tuning based on GPU memory class. Three strategies work in concert:

  1. Eco-params guardrail — Automatic parameter scaling for RoFormer family on small VRAM
  2. Long-track chunking — Time-domain splitting with configurable chunk/overlap sizes
  3. OOM retry ladder — Automatic fallback with reduced batch/segment sizes

GPU Classes & Recommendations

Entry-level GPUs (≤ 4 GB VRAM)

Examples: RTX 3050 (4GB), RTX 2060 (6GB), GTX 1660 (6GB)

Default Behavior:

  • Eco-params forced on all MelBand separators
  • Chunking mode: auto (activates at 480s)
  • Chunk duration: 120 seconds (vs 180s default)
  • Demucs segment: auto-scales to 4s for long tracks

Configuration:

LONG_TRACK_CHUNKING_MODE=auto
LONG_TRACK_CHUNK_DURATION_SECS=120    # Smaller chunks
LONG_TRACK_CHUNK_OVERLAP_SECS=5       # Smaller overlap
DEMUCS_SEGMENT_OVERRIDE=4              # Aggressive tuning

Expected Performance:

  • Throughput: 0.5–1.0× real-time (10-min track → 10–20 min runtime)
  • Max concurrent jobs: 1
  • Supported models: All except BS-RoFormer 6-stem, MDX-Net large
  • Failure modes: OOM on 4-stem + long track; fallback to Demucs recommended

Mid-range GPUs (6–8 GB VRAM)

Examples: RTX 3070 (8GB), RTX 4070 (12GB), RTX A5000 (24GB)

Default Behavior:

  • MelBand Karaoke 4-stem: no eco-params, good quality
  • Chunking mode: auto (activates at 480s)
  • Chunk duration: 180 seconds
  • Demucs segment: auto-scales to 8s for medium tracks

Configuration:

LONG_TRACK_CHUNKING_MODE=auto
LONG_TRACK_CHUNK_DURATION_SECS=180
LONG_TRACK_CHUNK_OVERLAP_SECS=10
DEMUCS_SEGMENT_OVERRIDE=null         # Use auto-scaling

Expected Performance:

  • Throughput: 1.5–2.5× real-time
  • Max concurrent jobs: 2
  • Supported models: All standard models
  • Ideal workloads: Mixed 2–8 min tracks; good production baseline

High-end GPUs (≥ 12 GB VRAM)

Examples: RTX 3090 (24GB), RTX 4090 (24GB), A100 (40–80GB)

Default Behavior:

  • No eco-params; full quality for all separators
  • Chunking mode: off (not needed for typical tracks)
  • Can run full 10–30 min tracks without chunking
  • Demucs segment: default 12s (standard Demucs)

Configuration:

LONG_TRACK_CHUNKING_MODE=off          # Disable for lowest latency
LONG_TRACK_DURATION_SECS=1200         # Increase threshold
DEMUCS_SEGMENT_OVERRIDE=null          # Standard Demucs
CPU_SEPARATION=false                  # GPU-only mode

Expected Performance:

  • Throughput: 3–5× real-time or better
  • Max concurrent jobs: 2–4 (depending on model)
  • Supported models: All models simultaneously
  • Ideal workloads: Batch processing, live streaming

Model-Specific Tuning

MelBand Karaoke + Long Tracks

RTX 4090 baseline (no chunking needed):

input: 10-minute 44.1 kHz stereo
runtime: ~90–120 seconds
VRAM peak: 16–18 GB

RTX 3070 with chunking:

input: 10-minute 44.1 kHz stereo
config: 180s chunks, 10s overlap
runtime: ~8–10 minutes (OLA overhead)
VRAM peak: ~5 GB per chunk

RTX 3050 with aggressive tuning:

input: 10-minute 44.1 kHz stereo
config: 120s chunks, 5s overlap + eco-params
runtime: ~15–20 minutes
VRAM peak: ~3.5 GB
fallback: Demucs recommended for reliability

Demucs Segment Tuning

Auto-scaling formula (Phase 3):

if duration < 120s:   segment = 12  # Default (best quality)
elif duration < 480s: segment = 8   # Quality-biased
else:                 segment = 4   # VRAM-biased

To override (e.g., for high-end GPU):

DEMUCS_SEGMENT_OVERRIDE=12  # Always use quality mode

BS-RoFormer 6-stem

Only runs on RTX 3090+ or with aggressive chunking:

RTX 4090 (recommended):

LONG_TRACK_CHUNKING_MODE=off
input: 6-minute maximum recommended
runtime: ~1–2 minutes

RTX 3070 (chunked):

LONG_TRACK_CHUNKING_MODE=auto
LONG_TRACK_CHUNK_DURATION_SECS=120
input: 10-minute maximum
runtime: ~10–15 minutes
VRAM peak: ~5 GB per chunk

Tuning Workflow

Step 1: Identify Your GPU

nvidia-smi --query-gpu=name,memory.total --format=csv,noheader

Step 2: Select Base Profile

Match your GPU to the class above; copy the recommended env vars.

Step 3: Test a Benchmark Track

Run a 10-minute test with your most common model:

# Monitor VRAM and runtime
nvidia-smi -l 1  # Refresh every 1 second
# Check job status at /api/jobs/{id}

Step 4: Tune Based on Results

Symptom Adjustment
OOM during separation Reduce LONG_TRACK_CHUNK_DURATION_SECS by 30s, or enable chunking with always mode
VRAM unused (< 50% peak) Increase chunk duration or disable chunking for faster latency
Seam artifacts at chunk boundaries Increase LONG_TRACK_CHUNK_OVERLAP_SECS from 10s to 15s
Demucs slow on long tracks Reduce DEMUCS_SEGMENT_OVERRIDE from 8 to 4 (or use auto-scaling)

Step 5: Validate with Multiple Tracks

Test a variety: short (2 min), medium (5 min), long (15 min). Verify no regressions.

Advanced: Server Deployment

Multi-GPU Setup (A100 × 2)

# Each GPU runs independently; use load balancer to distribute jobs
CUDA_VISIBLE_DEVICES=0   # GPU 0
LONG_TRACK_CHUNKING_MODE=off
MAX_PARALLEL_JOBS=2

# Separate container
CUDA_VISIBLE_DEVICES=1   # GPU 1
LONG_TRACK_CHUNKING_MODE=off
MAX_PARALLEL_JOBS=2

High-Throughput CPU-Only Fallback

CPU_SEPARATION=true
# Slower but runs on any server; useful for backup/low-priority queue

Monitoring & Diagnostics

Enable per-stage diagnostics for MelBand:

MELBAND_STAGE_DIAGNOSTICS=true

This logs:

  • Pre-separation VRAM allocator state
  • Checkpoint load time
  • Inference time per stage
  • Post-separation VRAM release

Useful for identifying bottlenecks before OOM occurs.

Future Tuning (Phase 5+)

Planned improvements:

  • Per-model VRAM profile: Learned from profiling, used to pre-allocate optimally
  • Dynamic chunk-size selection: Based on available VRAM at job submit time
  • Overlap-add quality metric: Measure seam artifacts; auto-adjust overlap if needed
  • GPU utilization dashboard: Real-time VRAM/compute occupancy per job

Quick Reference

GPU Max Track Length (no chunking) Recommended Chunk Size Typical Throughput
RTX 3050 (4GB) 2 min 120s 0.5× real-time
RTX 3070 (8GB) 5 min 180s 1.5× real-time
RTX 4090 (24GB) 30+ min off 3–5× real-time