# Benchmarking & Performance Profiling This document specifies the official methodology, metric definitions, and reporting requirements for performance benchmarks in **RLD UVR-Opt**. --- ## 1. When Benchmarking is Mandatory Benchmarking is strictly required for any Pull Request that: 1. Modifies inference routines in `separate.py` (chunking, windowing, STFT/iSTFT). 2. Alters Execution Provider registration or session options in `engine/tensorrt_rtx/`. 3. Changes precision downcasting policies (`SmartPrecisionPolicy`, in-memory FP16). 4. Modifies buffer allocation or headless runner bridges in `uvr_cli/`. --- ## 2. Metric Definitions & Formulas To ensure reproducible comparisons across contributors and hardware: | Metric | Identifier | Definition & Formula | Unit | | :--- | :--- | :--- | :--- | | **Inference Time** | `total_time_sec` | Total wall-clock time from tensor chunk submission until final stems are written. | Seconds ($s$) | | **Real-Time Factor** | `speed_factor_rt` | Processing speed relative to audio playback speed: $\text{Speed Factor} = \frac{\text{Audio Duration (s)}}{\text{Inference Time (s)}}$ | Multiplier ($x\text{ RT}$) | | **Throughput** | `throughput` | Audio duration processed per unit of time: $\text{Throughput} = \frac{\text{Audio Duration}}{\text{Total Time}}$ | Audio sec/sec | | **Chunk Latency (p50 / p95)** | `chunk_latency_ms` | Per-chunk inference latency evaluated at median (50th percentile) and tail (95th percentile). | Milliseconds ($\text{ms}$) | | **Active Tensor VRAM** | `peak_torch_vram_mb` | Peak memory explicitly allocated to active PyTorch tensors: `torch.cuda.max_memory_allocated()`. | Megabytes ($\text{MB}$) | | **Peak Device VRAM** | `peak_device_vram_mb` | Total physical GPU VRAM consumed across CUDA contexts and ORT memory arenas, measured via NVML `nvmlDeviceGetMemoryInfo()`. | Megabytes ($\text{MB}$) | | **Peak Process RAM** | `peak_process_ram_mb` | Maximum resident set size (RSS) of the application process: `psutil.Process().memory_info().rss`. | Megabytes ($\text{MB}$) | ### Statistical Reporting Standards When reporting steady-state metrics, contributors must provide: - **Sample Size ($n$)**: Minimum 9 measured repetitions `[PROPOSED]`. - **Central Tendency**: Median or Mean. - **Dispersion**: Standard Deviation ($s$) or Interquartile Range ($\text{IQR} = Q_3 - Q_1$). - **Single-run submissions are strictly rejected.** --- ## 3. Engine Build vs. Steady-State Measurement Accelerated Execution Providers (especially TensorRT) introduce significant latency on the first inference pass due to kernel timing calibration, graph optimization, and engine compilation: 1. **Cold Run (Overhead)**: - The initial pass where ONNX Runtime compiles the execution graph and writes plan caches to `models/MDX_Net_Models/trt_cache/`. - **Must be reported separately** as the cold/warmup run latency and must **never** be averaged into steady-state numbers. 2. **Warm Run (Steady-State)**: - Measured rounds after the engine has been compiled and cached in resident memory. - At least **2 warmup rounds** `[PROPOSED]` with dummy tensors or test audio must be completed before recording measurement rounds to eliminate cold-start spikes. --- ## 4. Controlled Variables & Benchmark Profile All standardized benchmarks must be conducted using the following baseline profile: | Parameter | Standard Value | Description | | :--- | :--- | :--- | | **Test Model** | `UVR-MDX-NET-Inst_HQ_4.onnx` | MDX-Net architecture FP32 model. | | **Segment Size** | `4000` | Standard sample window length. | | **Overlap** | `0.75` | Standard high-quality overlap factor. | | **Batch Size** | `1` | Single stream inference. | | **Precision** | `Ultra Quality (FP32)` | Baseline precision mode. Explicitly pass `--precision fp32` in CLI. | | **Test Audio** | Standard stereo WAV | Minimum 30–60 seconds duration (e.g. 48.34s test sample). Free-licensed or synthetic only. | ### Test Audio Licensing Rules - Audio samples must come from freely licensed sources (CC0, Creative Commons Attribution, or public domain) or synthetic waveforms (`np.sin`). - **Never commit copyrighted audio files, commercial music tracks, or raw stems to the repository.** Contributors must also attach their [install/install_record.json](../../install/install_record.json) or include OS version, GPU model, and NVIDIA display driver version. --- ## 5. Audio Integrity & Quality Tolerances Performance optimization must **not** degrade audio separation quality. When submitting optimization PRs: 1. **Numerical Residual Check**: - Compare output stem arrays against the baseline FP32 output: $$\Delta_{\max} = \max |A_{\text{opt}} - A_{\text{base}}|$$ - In pure FP32 code optimization, $\Delta_{\max} \le 10^{-5}$. 2. **Signal Degradation Tolerance (FP16 / Quantization)**: - For lower-precision paths, compute Signal-to-Distortion Ratio (SDR) or Signal-to-Noise Ratio (SNR): $$\text{SNR} = 10 \log_{10} \left( \frac{\sum A_{\text{base}}^2}{\sum (A_{\text{base}} - A_{\text{opt}})^2} \right)$$ - Minimum acceptable threshold: **$\text{SNR} \ge 40\text{ dB}$** `[PROPOSED]`. --- ## 6. Standard Execution Procedure Follow this step-by-step procedure when running benchmark rounds: ### Step 1: Prepare Environment - Close all background applications (web browsers, media players, IDEs). - Verify Windows Power Plan is set to **High Performance**. - Check that the GPU is at baseline temperature ($< 50^\circ\text{C}$) to avoid thermal throttling. ### Step 2: Warm-up Phase Execute at least 2 warmup passes on each target provider to compile caches: ```powershell .\venv\Scripts\python.exe -m uvr_cli bench -a "sample.wav" -m "UVR-MDX-NET-Inst_HQ_4.onnx" --precision fp32 --device cuda:0 --warmup 2 --rounds 1 .\venv\Scripts\python.exe -m uvr_cli bench -a "sample.wav" -m "UVR-MDX-NET-Inst_HQ_4.onnx" --precision fp32 --device trt_rtx:0 --warmup 2 --rounds 1 ``` ### Step 3: Interleaved Measurement Rounds (Mitigating Thermal Drift) To avoid thermal bias where the second engine tested runs hotter than the first, execute alternating rounds across configurations: ```powershell # Interleaved benchmark execution pattern: # Alternate between Baseline and Target across rounds: for ($r = 1; $r -le 9; $r++) { Write-Host "--- Executing Measurement Round $r ---" # 1. Baseline Run (CUDA EP) .\venv\Scripts\python.exe -m uvr_cli bench -a "sample.wav" -m "UVR-MDX-NET-Inst_HQ_4.onnx" --precision fp32 --device cuda:0 --rounds 1 --warmup 0 # 2. Optimized Run (TensorRT-RTX EP) .\venv\Scripts\python.exe -m uvr_cli bench -a "sample.wav" -m "UVR-MDX-NET-Inst_HQ_4.onnx" --precision fp32 --device trt_rtx:0 --rounds 1 --warmup 0 } ``` ### Step 4: Outlier Rejection Policy - If an individual measurement run suffers an operating system spike (e.g. Windows background indexing, GPU interrupt), it may be excluded if it deviates by more than $1.5 \times \text{IQR}$ from the median. - **All excluded outlier runs must be explicitly listed and justified in the PR submission table.** Telemetry and per-round metrics are automatically saved to `benchmark_results/summary.csv` and timestamped session JSON files. --- ## 7. Submission Templates ### Markdown Table Template for PRs Copy and fill this table in your Pull Request description: ```markdown ### Benchmark Results **Test Hardware**: [e.g., NVIDIA GeForce RTX 4070 (Driver 572.16) | AMD Ryzen 7 7800X3D | Windows 11] **Audio Sample**: [e.g., sample.wav (48.34s stereo, 44.1kHz, CC0)] **Model**: `UVR-MDX-NET-Inst_HQ_4.onnx` (Overlap 0.75, Segment 4000, FP32) **Sample Count**: n = 9 measured rounds (after 2 warmups, interleaved order) | Execution Provider | Warmup Time | Steady-State Time (Median / Mean ± Std) | Chunk Latency (p50 / p95) | Real-Time Factor | Peak Device VRAM | Peak Process RAM | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | **CUDA EP (Baseline)** | 0.00 s | 0.00 s / 0.00 s ± 0.00 s | 0.0 ms / 0.0 ms | 0.00x RT | 0,000 MB | 0,000 MB | | **Your Optimization** | 0.00 s | 0.00 s / 0.00 s ± 0.00 s | 0.0 ms / 0.0 ms | 0.00x RT | 0,000 MB | 0,000 MB | | **Delta / Speedup** | — | **~0.00x Faster** | **-00.0% Latency** | **+00.0% Throughput** | **-00.0% VRAM** | **-00.0% RAM** | *Session IDs: `session__` (Optimized) vs `session__` (Baseline).* ``` --- ### Machine-Readable JSON Schema (Draft 2020-12) Benchmark results can be validated against the formal schema below: ```json { "$schema": "https://json-schema.org/draft/2020-12/schema", "title": "RLDUVRBenchmarkSession", "type": "object", "required": [ "session_id", "timestamp_utc", "model_info", "summary" ], "properties": { "session_id": { "type": "string" }, "timestamp_utc": { "type": "string" }, "model_info": { "type": "object", "required": ["model_name", "backend", "device", "precision", "overlap", "segment_size"], "properties": { "model_name": { "type": "string" }, "backend": { "type": "string" }, "device": { "type": "string" }, "precision": { "type": "string" }, "overlap": { "type": "number" }, "segment_size": { "type": "integer" } } }, "summary": { "type": "object", "required": [ "warmup_rounds", "rounds_measured", "avg_total_time_sec", "avg_speed_factor_rt", "peak_device_vram_mb" ], "properties": { "warmup_rounds": { "type": "integer" }, "rounds_measured": { "type": "integer" }, "avg_total_time_sec": { "type": "number" }, "median_total_time_sec": { "type": "number" }, "avg_speed_factor_rt": { "type": "number" }, "median_speed_factor_rt": { "type": "number" }, "min_total_time_sec": { "type": "number" }, "max_total_time_sec": { "type": "number" }, "peak_device_vram_mb": { "type": "number" }, "peak_torch_allocated_vram_mb": { "type": "number" }, "chunk_latency_p50_ms": { "type": "number" }, "chunk_latency_p95_ms": { "type": "number" } } } } } ``` --- ### Valid Session JSON Example (Placeholders Only) *Note: All values below are illustrative placeholders per project contribution standards.* ```json { "session_id": "session__", "timestamp_utc": "2026-01-01T00:00:00Z", "model_info": { "model_name": "UVR-MDX-NET-Inst_HQ_4.onnx", "backend": "MDX-Net", "device": "trt_rtx:0", "precision": "Ultra Quality (FP32)", "overlap": 0.75, "segment_size": 4000 }, "summary": { "warmup_rounds": 2, "rounds_measured": 9, "avg_total_time_sec": 0.00, "median_total_time_sec": 0.00, "avg_speed_factor_rt": 0.00, "median_speed_factor_rt": 0.00, "min_total_time_sec": 0.00, "max_total_time_sec": 0.00, "peak_device_vram_mb": 0.0, "peak_torch_allocated_vram_mb": 0.0, "chunk_latency_p50_ms": 0.0, "chunk_latency_p95_ms": 0.0 } } ``` --- ## 8. Common Benchmarking Pitfalls 1. **Averaging the Cold Run**: Counting the initial compilation pass into the average latency drastically inflates steady-state timing. 2. **Thermal Throttling**: Running 20 consecutive passes on an overheated GPU will skew measurements downward. Ensure adequate cooling pauses and use interleaved testing. 3. **Silent Provider Fallbacks**: Always confirm `Active execution provider` in terminal logs to ensure your run actually evaluated on the GPU. 4. **Different Test Audio Lengths**: Comparing speed factor across tracks of differing audio lengths or sample rates introduces bias. 5. **Missing Explicit Precision Flag**: Forgetting `--precision fp32` in the CLI causes `uvr_cli bench` to run in its default FP16 mode instead of the required FP32 baseline.