12 KiB
Benchmarking & Performance Profiling
This document specifies the official methodology, metric definitions, and reporting requirements for performance benchmarks in RLD UVR-Opt.
1. When Benchmarking is Mandatory
Benchmarking is strictly required for any Pull Request that:
- Modifies inference routines in
separate.py(chunking, windowing, STFT/iSTFT). - Alters Execution Provider registration or session options in
engine/tensorrt_rtx/. - Changes precision downcasting policies (
SmartPrecisionPolicy, in-memory FP16). - Modifies buffer allocation or headless runner bridges in
uvr_cli/.
2. Metric Definitions & Formulas
To ensure reproducible comparisons across contributors and hardware:
| Metric | Identifier | Definition & Formula | Unit |
|---|---|---|---|
| Inference Time | total_time_sec |
Total wall-clock time from tensor chunk submission until final stems are written. | Seconds (s) |
| Real-Time Factor | speed_factor_rt |
Processing speed relative to audio playback speed: \text{Speed Factor} = \frac{\text{Audio Duration (s)}}{\text{Inference Time (s)}} |
Multiplier (x\text{ RT}) |
| Throughput | throughput |
Audio duration processed per unit of time: \text{Throughput} = \frac{\text{Audio Duration}}{\text{Total Time}} |
Audio sec/sec |
| Chunk Latency (p50 / p95) | chunk_latency_ms |
Per-chunk inference latency evaluated at median (50th percentile) and tail (95th percentile). | Milliseconds (\text{ms}) |
| Active Tensor VRAM | peak_torch_vram_mb |
Peak memory explicitly allocated to active PyTorch tensors: torch.cuda.max_memory_allocated(). |
Megabytes (\text{MB}) |
| Peak Device VRAM | peak_device_vram_mb |
Total physical GPU VRAM consumed across CUDA contexts and ORT memory arenas, measured via NVML nvmlDeviceGetMemoryInfo(). |
Megabytes (\text{MB}) |
| Peak Process RAM | peak_process_ram_mb |
Maximum resident set size (RSS) of the application process: psutil.Process().memory_info().rss. |
Megabytes (\text{MB}) |
Statistical Reporting Standards
When reporting steady-state metrics, contributors must provide:
- Sample Size (
n): Minimum 9 measured repetitions[PROPOSED]. - Central Tendency: Median or Mean.
- Dispersion: Standard Deviation (
s) or Interquartile Range (\text{IQR} = Q_3 - Q_1). - Single-run submissions are strictly rejected.
3. Engine Build vs. Steady-State Measurement
Accelerated Execution Providers (especially TensorRT) introduce significant latency on the first inference pass due to kernel timing calibration, graph optimization, and engine compilation:
- Cold Run (Overhead):
- The initial pass where ONNX Runtime compiles the execution graph and writes plan caches to
models/MDX_Net_Models/trt_cache/. - Must be reported separately as the cold/warmup run latency and must never be averaged into steady-state numbers.
- The initial pass where ONNX Runtime compiles the execution graph and writes plan caches to
- Warm Run (Steady-State):
- Measured rounds after the engine has been compiled and cached in resident memory.
- At least 2 warmup rounds
[PROPOSED]with dummy tensors or test audio must be completed before recording measurement rounds to eliminate cold-start spikes.
4. Controlled Variables & Benchmark Profile
All standardized benchmarks must be conducted using the following baseline profile:
| Parameter | Standard Value | Description |
|---|---|---|
| Test Model | UVR-MDX-NET-Inst_HQ_4.onnx |
MDX-Net architecture FP32 model. |
| Segment Size | 4000 |
Standard sample window length. |
| Overlap | 0.75 |
Standard high-quality overlap factor. |
| Batch Size | 1 |
Single stream inference. |
| Precision | Ultra Quality (FP32) |
Baseline precision mode. Explicitly pass --precision fp32 in CLI. |
| Test Audio | Standard stereo WAV | Minimum 30–60 seconds duration (e.g. 48.34s test sample). Free-licensed or synthetic only. |
Test Audio Licensing Rules
- Audio samples must come from freely licensed sources (CC0, Creative Commons Attribution, or public domain) or synthetic waveforms (
np.sin). - Never commit copyrighted audio files, commercial music tracks, or raw stems to the repository.
Contributors must also attach their install/install_record.json or include OS version, GPU model, and NVIDIA display driver version.
5. Audio Integrity & Quality Tolerances
Performance optimization must not degrade audio separation quality. When submitting optimization PRs:
- Numerical Residual Check:
- Compare output stem arrays against the baseline FP32 output:
\Delta_{\max} = \max |A_{\text{opt}} - A_{\text{base}}| - In pure FP32 code optimization,
\Delta_{\max} \le 10^{-5}.
- Compare output stem arrays against the baseline FP32 output:
- Signal Degradation Tolerance (FP16 / Quantization):
- For lower-precision paths, compute Signal-to-Distortion Ratio (SDR) or Signal-to-Noise Ratio (SNR):
\text{SNR} = 10 \log_{10} \left( \frac{\sum A_{\text{base}}^2}{\sum (A_{\text{base}} - A_{\text{opt}})^2} \right) - Minimum acceptable threshold: $\text{SNR} \ge 40\text{ dB}$
[PROPOSED].
- For lower-precision paths, compute Signal-to-Distortion Ratio (SDR) or Signal-to-Noise Ratio (SNR):
6. Standard Execution Procedure
Follow this step-by-step procedure when running benchmark rounds:
Step 1: Prepare Environment
- Close all background applications (web browsers, media players, IDEs).
- Verify Windows Power Plan is set to High Performance.
- Check that the GPU is at baseline temperature (
< 50^\circ\text{C}) to avoid thermal throttling.
Step 2: Warm-up Phase
Execute at least 2 warmup passes on each target provider to compile caches:
.\venv\Scripts\python.exe -m uvr_cli bench -a "sample.wav" -m "UVR-MDX-NET-Inst_HQ_4.onnx" --precision fp32 --device cuda:0 --warmup 2 --rounds 1
.\venv\Scripts\python.exe -m uvr_cli bench -a "sample.wav" -m "UVR-MDX-NET-Inst_HQ_4.onnx" --precision fp32 --device trt_rtx:0 --warmup 2 --rounds 1
Step 3: Interleaved Measurement Rounds (Mitigating Thermal Drift)
To avoid thermal bias where the second engine tested runs hotter than the first, execute alternating rounds across configurations:
# Interleaved benchmark execution pattern:
# Alternate between Baseline and Target across rounds:
for ($r = 1; $r -le 9; $r++) {
Write-Host "--- Executing Measurement Round $r ---"
# 1. Baseline Run (CUDA EP)
.\venv\Scripts\python.exe -m uvr_cli bench -a "sample.wav" -m "UVR-MDX-NET-Inst_HQ_4.onnx" --precision fp32 --device cuda:0 --rounds 1 --warmup 0
# 2. Optimized Run (TensorRT-RTX EP)
.\venv\Scripts\python.exe -m uvr_cli bench -a "sample.wav" -m "UVR-MDX-NET-Inst_HQ_4.onnx" --precision fp32 --device trt_rtx:0 --rounds 1 --warmup 0
}
Step 4: Outlier Rejection Policy
- If an individual measurement run suffers an operating system spike (e.g. Windows background indexing, GPU interrupt), it may be excluded if it deviates by more than
1.5 \times \text{IQR}from the median. - All excluded outlier runs must be explicitly listed and justified in the PR submission table.
Telemetry and per-round metrics are automatically saved to benchmark_results/summary.csv and timestamped session JSON files.
7. Submission Templates
Markdown Table Template for PRs
Copy and fill this table in your Pull Request description:
### Benchmark Results
**Test Hardware**: [e.g., NVIDIA GeForce RTX 4070 (Driver 572.16) | AMD Ryzen 7 7800X3D | Windows 11]
**Audio Sample**: [e.g., sample.wav (48.34s stereo, 44.1kHz, CC0)]
**Model**: `UVR-MDX-NET-Inst_HQ_4.onnx` (Overlap 0.75, Segment 4000, FP32)
**Sample Count**: n = 9 measured rounds (after 2 warmups, interleaved order)
| Execution Provider | Warmup Time | Steady-State Time (Median / Mean ± Std) | Chunk Latency (p50 / p95) | Real-Time Factor | Peak Device VRAM | Peak Process RAM |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| **CUDA EP (Baseline)** | 0.00 s | 0.00 s / 0.00 s ± 0.00 s | 0.0 ms / 0.0 ms | 0.00x RT | 0,000 MB | 0,000 MB |
| **Your Optimization** | 0.00 s | 0.00 s / 0.00 s ± 0.00 s | 0.0 ms / 0.0 ms | 0.00x RT | 0,000 MB | 0,000 MB |
| **Delta / Speedup** | — | **~0.00x Faster** | **-00.0% Latency** | **+00.0% Throughput** | **-00.0% VRAM** | **-00.0% RAM** |
*Session IDs: `session_<timestamp>_<id>` (Optimized) vs `session_<timestamp>_<id>` (Baseline).*
Machine-Readable JSON Schema (Draft 2020-12)
Benchmark results can be validated against the formal schema below:
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "RLDUVRBenchmarkSession",
"type": "object",
"required": [
"session_id",
"timestamp_utc",
"model_info",
"summary"
],
"properties": {
"session_id": { "type": "string" },
"timestamp_utc": { "type": "string" },
"model_info": {
"type": "object",
"required": ["model_name", "backend", "device", "precision", "overlap", "segment_size"],
"properties": {
"model_name": { "type": "string" },
"backend": { "type": "string" },
"device": { "type": "string" },
"precision": { "type": "string" },
"overlap": { "type": "number" },
"segment_size": { "type": "integer" }
}
},
"summary": {
"type": "object",
"required": [
"warmup_rounds",
"rounds_measured",
"avg_total_time_sec",
"avg_speed_factor_rt",
"peak_device_vram_mb"
],
"properties": {
"warmup_rounds": { "type": "integer" },
"rounds_measured": { "type": "integer" },
"avg_total_time_sec": { "type": "number" },
"median_total_time_sec": { "type": "number" },
"avg_speed_factor_rt": { "type": "number" },
"median_speed_factor_rt": { "type": "number" },
"min_total_time_sec": { "type": "number" },
"max_total_time_sec": { "type": "number" },
"peak_device_vram_mb": { "type": "number" },
"peak_torch_allocated_vram_mb": { "type": "number" },
"chunk_latency_p50_ms": { "type": "number" },
"chunk_latency_p95_ms": { "type": "number" }
}
}
}
}
Valid Session JSON Example (Placeholders Only)
Note: All values below are illustrative placeholders per project contribution standards.
{
"session_id": "session_<YYYYMMDD_HHMMSS>_<hex>",
"timestamp_utc": "2026-01-01T00:00:00Z",
"model_info": {
"model_name": "UVR-MDX-NET-Inst_HQ_4.onnx",
"backend": "MDX-Net",
"device": "trt_rtx:0",
"precision": "Ultra Quality (FP32)",
"overlap": 0.75,
"segment_size": 4000
},
"summary": {
"warmup_rounds": 2,
"rounds_measured": 9,
"avg_total_time_sec": 0.00,
"median_total_time_sec": 0.00,
"avg_speed_factor_rt": 0.00,
"median_speed_factor_rt": 0.00,
"min_total_time_sec": 0.00,
"max_total_time_sec": 0.00,
"peak_device_vram_mb": 0.0,
"peak_torch_allocated_vram_mb": 0.0,
"chunk_latency_p50_ms": 0.0,
"chunk_latency_p95_ms": 0.0
}
}
8. Common Benchmarking Pitfalls
- Averaging the Cold Run: Counting the initial compilation pass into the average latency drastically inflates steady-state timing.
- Thermal Throttling: Running 20 consecutive passes on an overheated GPU will skew measurements downward. Ensure adequate cooling pauses and use interleaved testing.
- Silent Provider Fallbacks: Always confirm
Active execution providerin terminal logs to ensure your run actually evaluated on the GPU. - Different Test Audio Lengths: Comparing speed factor across tracks of differing audio lengths or sample rates introduces bias.
- Missing Explicit Precision Flag: Forgetting
--precision fp32in the CLI causesuvr_cli benchto run in its default FP16 mode instead of the required FP32 baseline.