Files
RLD-UVR-Opt/docs/development/testing.md
T

4.5 KiB

Testing Guide

This guide details how to execute and write automated tests for RLD UVR-Opt.


1. Test Suite Overview

The repository maintains automated test suites covering Python inference/CLI components and PowerShell installer pipelines:

Suite Runner Location Purpose
Python Regression Tests unittest tests/ Tests CLI argument parsing, model resolution, in-memory cache eviction, TRT guards, and batch file CRLF line endings.
Installer Tests Pester install/tests/ Tests 15 installer scenarios covering plan generation, hardware gates, driver checks, and atomic state recording.
Headless Smoke Test uvr_cli uvr_cli/ Performs end-to-end inference verification on procedurally generated synthetic audio.

2. Running Python Unit Tests

Run Full Test Suite

To run all 41 repository unit tests:

.\venv\Scripts\python.exe -m unittest discover tests -v

Run Targeted Test Modules

You can execute individual test modules during focused development:

# Test CLI commands and flag parsing
.\venv\Scripts\python.exe -m unittest tests.test_uvr_cli -v

# Test in-memory resident model cache and FP16 lifecycle
.\venv\Scripts\python.exe -m unittest tests.test_in_memory_model_cache -v

# Test TensorRT-RTX provider guards and Ampere check
.\venv\Scripts\python.exe -m unittest tests.test_tensorrt_rtx -v

# Test batch file line endings (verifies CRLF formatting)
.\venv\Scripts\python.exe -m unittest tests.test_windows_batch_files -v

3. Running PowerShell Installer Tests (Pester)

The installer logic is verified through Pester 3.4.0+ (bundled with Windows 10/11):

powershell -NoProfile -ExecutionPolicy Bypass -Command "Invoke-Pester -Path ./install/tests"

What the Pester Suite Covers:

  • Phase 2 (Plan Generation):
    • Migrating CPU venv to GPU (Option A) replaces PyTorch CPU with CUDA wheels.
    • Upgrading Option A to Option B keeps PyTorch and installs TRT-RTX plugin.
    • Resolving conflicting onnxruntime vs onnxruntime-gpu packages.
    • Rerunning identical options produces zero destructive changes (Keep).
  • Phase 3 (Verification & Gate):
    • No GPU or AMD GPU detected → adjusts selection to Option C.
    • NVIDIA Turing (CC 7.5) selecting Option B → adjusts to Option A.
    • NVIDIA driver version too old (< 570) → blocks installation.
    • Insufficient disk space (< 5\text{ GB}) or active file locks → blocks installation.
  • Phase 5 (Atomic Record):
    • Verifies atomic writing of install_record.json without file locking or syntax errors.

4. Running the Headless Smoke Test

The smoke test verifies end-to-end model inference without requiring existing music files. It generates a short 440 Hz synthetic stereo sine wave in memory and separates it:

# Run smoke test on CPU
.\venv\Scripts\python.exe -m uvr_cli smoke-test --device cpu

# Run smoke test on GPU (CUDA)
.\venv\Scripts\python.exe -m uvr_cli smoke-test --device cuda:0

# Run smoke test on TensorRT-RTX
.\venv\Scripts\python.exe -m uvr_cli smoke-test --device trt_rtx:0

5. Testing Without Physical GPUs (Headless & Mocking)

Many contributors or automated CI environments do not possess physical NVIDIA RTX GPUs:

  • Mocking GPU Hardware: In tests/test_tensorrt_rtx.py, GPU architecture checks and ONNX Runtime provider sessions are mocked via unittest.mock.patch:
    with patch("torch.cuda.is_available", return_value=True), \
         patch("torch.cuda.get_device_capability", return_value=(7, 5)):
        # Simulates Turing GPU (CC 7.5) to test fallback
        providers = get_supported_execution_providers(device_id=0)
        self.assertNotIn("NvTensorRTRTXExecutionProvider", providers)
    
  • PRs Touching GPU Paths: While mocked tests allow validating control flow without a GPU, any PR that modifies raw CUDA kernels, TensorRT engine options, or memory layout must include verified test logs or benchmark telemetry from a physical GPU machine.

6. Prohibited Test Artifacts

  • No Model Weights in Git: Never commit binary model weights (*.onnx, *.pth, *.ckpt, *.safetensors). Use synthetic models (e.g. tiny base64 ONNX graphs) for unit testing.
  • No Copyrighted Audio: Never commit commercial or copyrighted music files to input/ or tests/. Unit tests should synthesize audio arrays in memory (np.sin(...)).