Optimized VRAM usage for separation and transcription jobs. Added documentation for GPU limiter on Windows side.
This commit is contained in:
1 parent
3483f4f636
commit
74f632297e
7 files changed
+288
-6
No files matched your search
@@ -647,3 +647,35 @@ Returns `{"added": N, "stems": [...]}`. Idempotent — already-tracked paths are
|
||||
**Fix:** Added `lsyncAlign` variable and `effectiveWordTs = wordTs || lsyncAlign`. When `lsyncAlign=true`, `effectiveWordTs` is forced `true`, ensuring `'elrc'` is always included in `defaultFormats` and `word_timestamps: true` is sent to the backend.
|
||||
|
||||
Also reverted an earlier incorrect change that added `'sylrc'` to formats when `lsyncAlign=true` (user confirmed sylrc is not needed; `phonemeAlign` is no longer used).
|
||||
|
||||
### Fix: Lyrics preview lines overflowing the karaoke preview window
|
||||
|
||||
**Problem:** Non-active lines with CSS `filter` (e.g. `blur()` in blur-style, `drop-shadow()` in glow-style) could bleed visually outside the `containerRef` boundary because `overflow: hidden` alone does not reliably clip CSS filter effects across all browsers. Additionally, lines were hard-cut at the container edge rather than fading smoothly.
|
||||
|
||||
**Fix:** Added two properties to `containerRef` in `LyricsLivePreview.tsx` (`renderLyricsArea()`):
|
||||
- `clipPath: 'inset(0)'` — guarantees filter bleed is hard-clipped to the container box in all browsers, even where `overflow: hidden` fails.
|
||||
- `maskImage` / `WebkitMaskImage: 'linear-gradient(to bottom, transparent 0%, black 18%, black 82%, transparent 100%)'` — fades lines in/out at the top and bottom 18% of the preview window so non-active lines appear to enter and exit smoothly rather than being abruptly cut off.
|
||||
|
||||
---
|
||||
|
||||
## 2026-03-10
|
||||
|
||||
### Fix: CUDA out-of-memory errors after several separation jobs
|
||||
|
||||
**Root cause (primary):** `_gpu_executor` (stem separation) and `_transcription_executor` (Whisper/WhisperX transcription) are independent `ThreadPoolExecutor(max_workers=1)` instances in `job_manager.py`. Each serialises its own job type, but both can run simultaneously. A separator model (3–6 GB VRAM) and Whisper large-v2 (3–4 GB VRAM) loading at the same time easily exhausts available VRAM on cards ≤12 GB, producing CUDA OOM errors that surface only after several jobs (whenever separation and transcription happen to overlap).
|
||||
|
||||
**Root cause (secondary):** `gc.collect()` was called only once in both `separator_service.py`'s `finally` block and `_clear_cuda_cache()`. PyTorch model objects form reference cycles (`_parameters → _modules → parent → …`) that Python's generational GC may not fully collect in a single pass — objects promoted to older generations require additional passes. This could leave model tensors resident in VRAM when the next job started.
|
||||
|
||||
**Fix — `backend/job_manager.py`:**
|
||||
Added `gpu_vram_lock = threading.Lock()` — a shared VRAM coordination lock that both executor paths must hold for the duration of model loading + inference.
|
||||
|
||||
**Fix — `backend/services/separator_service.py`:**
|
||||
- `jm.gpu_vram_lock.acquire()` added before the separation `try:` block; `jm.gpu_vram_lock.release()` added at the end of the `finally:` block (after cleanup).
|
||||
- Triple `gc.collect()` in the `finally` block instead of single.
|
||||
|
||||
**Fix — `backend/services/lyric_transcriber_service.py`:**
|
||||
- `jm.gpu_vram_lock.acquire()` added before the transcription `try:` block in `run_lyric_transcription`.
|
||||
- Added a `finally:` block (previously absent) that calls `_clear_cuda_cache()` and releases the lock on both success and failure paths.
|
||||
- `_clear_cuda_cache()` updated to call `gc.collect()` three times instead of once.
|
||||
|
||||
**Effect:** Separation and transcription jobs now queue against the same lock. Whichever starts first holds VRAM exclusively until its `finally` cleanup completes, then releases the lock so the other can proceed. No model can be double-loaded simultaneously regardless of how jobs are submitted.
|
||||
@@ -0,0 +1,202 @@
|
||||
# Limiting GPU Power on WSL2
|
||||
|
||||
## Background
|
||||
|
||||
GPU-intensive Docker workloads (StemEngine, Ollama) can drive the GPU to full TDP,
|
||||
causing fan noise and heat. On a native Linux host, `nvidia-smi --power-limit` works
|
||||
from a privileged container. On WSL2 it does not — the NVIDIA driver is paravirtualized
|
||||
through the Windows WDDM stack, and power management commands require Windows-level
|
||||
Administrator privileges that no Linux process (including root inside a container) can
|
||||
obtain.
|
||||
|
||||
**The rule:** anything that needs to change GPU power limits must run on the Windows side.
|
||||
|
||||
---
|
||||
|
||||
## What Does Not Work in WSL2
|
||||
|
||||
Both approaches below fail with `Insufficient Permissions` in WSL2, even with
|
||||
`cap_add: [SYS_ADMIN]` and even calling `nvidia-smi.exe` (the Windows binary) via
|
||||
WSL2 interop:
|
||||
|
||||
- `nvidia-smi --power-limit=N` from inside a Docker container
|
||||
- `nvidia-smi.exe --power-limit=N` called from a WSL2 shell (even as root)
|
||||
|
||||
WSL2 interop runs Windows executables with the current Windows *user* token, not an
|
||||
elevated Administrator token, so power limit changes are rejected by the driver.
|
||||
|
||||
---
|
||||
|
||||
## Solution: Windows Task Scheduler
|
||||
|
||||
A scheduled task running as **SYSTEM** (highest privilege) calls `nvidia-smi.exe` at
|
||||
Windows startup before any user session begins. This runs entirely outside WSL2 and
|
||||
has the permissions required.
|
||||
|
||||
### PowerShell script (inline, no file needed)
|
||||
|
||||
The task is registered with the command embedded directly to avoid PowerShell execution
|
||||
policy issues when running as SYSTEM:
|
||||
|
||||
```powershell
|
||||
$action = New-ScheduledTaskAction `
|
||||
-Execute "powershell.exe" `
|
||||
-Argument '-NonInteractive -WindowStyle Hidden -Command "& { $max = [double](& nvidia-smi --query-gpu=power.max_limit --format=csv,noheader,nounits).Trim(); $t = [math]::Round($max * 80 / 100); & nvidia-smi --power-limit=$t }"'
|
||||
|
||||
Register-ScheduledTask -TaskName "GPU Power Limit" `
|
||||
-Action $action `
|
||||
-Trigger (New-ScheduledTaskTrigger -AtStartup) `
|
||||
-Principal (New-ScheduledTaskPrincipal -UserId "SYSTEM" -RunLevel Highest) `
|
||||
-Force
|
||||
```
|
||||
|
||||
Change `80` to the desired percentage of the card's max TDP (e.g. `70` for 70%).
|
||||
|
||||
Run this in an **elevated PowerShell on Windows** (Win+X → Terminal (Admin)).
|
||||
|
||||
### Verify it registered correctly
|
||||
|
||||
```powershell
|
||||
Get-ScheduledTask -TaskName "GPU Power Limit" | Select TaskName, State
|
||||
Get-ScheduledTaskInfo -TaskName "GPU Power Limit" | Select LastRunTime, LastTaskResult, NextRunTime
|
||||
```
|
||||
|
||||
Expected: `State: Ready`, `LastTaskResult: 0` (success).
|
||||
|
||||
### Test without rebooting
|
||||
|
||||
```powershell
|
||||
Start-ScheduledTask -TaskName "GPU Power Limit"
|
||||
Start-Sleep 3
|
||||
nvidia-smi -q -d POWER | Select-String "Power Limit"
|
||||
```
|
||||
|
||||
`Current Power Limit` should reflect the new value (e.g. 136 W for 80% of a 170 W card).
|
||||
|
||||
### Remove or update
|
||||
|
||||
```powershell
|
||||
# Update: just re-run Register-ScheduledTask with -Force (overwrites)
|
||||
# Remove:
|
||||
Unregister-ScheduledTask -TaskName "GPU Power Limit" -Confirm:$false
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Undoing / Removing the Power Limit
|
||||
|
||||
### Remove the Task Scheduler task (Windows)
|
||||
|
||||
In an **elevated PowerShell on Windows**:
|
||||
|
||||
```powershell
|
||||
# Remove the startup task
|
||||
Unregister-ScheduledTask -TaskName "GPU Power Limit" -Confirm:$false
|
||||
|
||||
# Restore the GPU to its default power limit immediately
|
||||
nvidia-smi --power-limit=$(
|
||||
(nvidia-smi --query-gpu=power.default_limit --format=csv,noheader,nounits).Trim()
|
||||
)
|
||||
```
|
||||
|
||||
The second command reads the card's factory default TDP from the driver and restores it
|
||||
without needing to know the value in advance. After the next reboot the driver will
|
||||
reset to default on its own anyway since the task no longer runs.
|
||||
|
||||
Verify the limit is back to default:
|
||||
|
||||
```powershell
|
||||
nvidia-smi -q -d POWER | Select-String "Power Limit"
|
||||
```
|
||||
|
||||
`Current Power Limit` and `Default Power Limit` should match.
|
||||
|
||||
### Remove the VRAM reservation (Ollama)
|
||||
|
||||
Edit `llama.yml` and delete or comment out the `OLLAMA_GPU_OVERHEAD` line, then
|
||||
restart Ollama:
|
||||
|
||||
```bash
|
||||
docker compose -f /home/translator/docker/llama.yml up -d ollama
|
||||
```
|
||||
|
||||
### Remove the docker-compose gpu-init services
|
||||
|
||||
If you no longer want the (harmless but noisy) `gpu-init` containers in the compose
|
||||
logs, remove the `gpu-init` service block and its `depends_on` reference from both
|
||||
`stemengine.yml` and `llama.yml`, then redeploy each stack.
|
||||
|
||||
Alternatively, silence them without removing by setting in `.env`:
|
||||
|
||||
```env
|
||||
GPU_POWER_LIMIT=0
|
||||
```
|
||||
|
||||
The `if [ "$$PL" != "0" ]` guard in the entrypoint will skip the `nvidia-smi` call
|
||||
and the container will exit cleanly with no error output.
|
||||
|
||||
---
|
||||
|
||||
## Behaviour
|
||||
|
||||
| Situation | Power limit |
|
||||
|---|---|
|
||||
| Windows boots (before any user logs in) | Task runs → limit set |
|
||||
| WSL2 starts / Docker containers start | Limit already applied |
|
||||
| Windows reboots | Task runs again automatically |
|
||||
| WSL2 shut down mid-session | Limit stays until next reboot (then resets and task re-applies) |
|
||||
| GPU driver update / reinstall | Limit resets to default; task re-applies on next reboot |
|
||||
|
||||
NVIDIA consumer GPU power limits reset to the card's default on driver reload or
|
||||
system restart, so the task always re-applies the correct limit at boot.
|
||||
|
||||
---
|
||||
|
||||
## docker-compose gpu-init containers
|
||||
|
||||
`stemengine.yml` and `llama.yml` both contain a `gpu-init` service that attempts to
|
||||
call `nvidia-smi --power-limit` before the main container starts. These exist for
|
||||
portability (they work on native Linux hosts with a real NVIDIA driver). On WSL2 they
|
||||
fail harmlessly — the main container still starts because the entrypoint uses `|| true`
|
||||
semantics — but the power limit is not changed.
|
||||
|
||||
Set `GPU_POWER_LIMIT=0` in `.env` to skip the `nvidia-smi` call entirely on WSL2:
|
||||
|
||||
```env
|
||||
GPU_POWER_LIMIT=0
|
||||
```
|
||||
|
||||
The Windows Task Scheduler task is the authoritative power limit mechanism on this host.
|
||||
|
||||
---
|
||||
|
||||
## Ollama VRAM Reservation
|
||||
|
||||
`llama.yml` also sets:
|
||||
|
||||
```yaml
|
||||
OLLAMA_GPU_OVERHEAD=2147483648 # 2 GB
|
||||
```
|
||||
|
||||
This tells Ollama to leave 2 GB of VRAM unallocated, reserving headroom for Windows
|
||||
desktop compositor and other GPU processes. It is independent of the power limit and
|
||||
works correctly in WSL2 — it is enforced by Ollama's own allocator, not the driver.
|
||||
|
||||
Adjust the value (in bytes) if more or less headroom is needed:
|
||||
|
||||
| Headroom | Value |
|
||||
|---|---|
|
||||
| 1 GB | `1073741824` |
|
||||
| 2 GB | `2147483648` |
|
||||
| 3 GB | `3221225472` |
|
||||
| 4 GB | `4294967296` |
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Mechanism | Where it runs | Works in WSL2 | Purpose |
|
||||
|---|---|---|---|
|
||||
| Task Scheduler "GPU Power Limit" | Windows (SYSTEM) | ✅ | Enforces watt cap at boot |
|
||||
| `gpu-init` docker service | WSL2 container | ❌ | Portability shim for native Linux |
|
||||
| `OLLAMA_GPU_OVERHEAD` | Ollama process | ✅ | Reserves VRAM for Windows |
|
||||
@@ -110,6 +110,13 @@ _general_executor = ThreadPoolExecutor(max_workers=4)
|
||||
_gpu_executor = ThreadPoolExecutor(max_workers=1)
|
||||
_transcription_executor = ThreadPoolExecutor(max_workers=1)
|
||||
|
||||
# Shared VRAM coordination lock.
|
||||
# Both _gpu_executor (separation) and _transcription_executor (Whisper/WhisperX)
|
||||
# must hold this lock for the duration of GPU model loading + inference.
|
||||
# Prevents the two independent executor threads from loading models simultaneously
|
||||
# and triggering CUDA out-of-memory errors.
|
||||
gpu_vram_lock = threading.Lock()
|
||||
|
||||
# Queue-depth counter for the transcription executor (includes the running job).
|
||||
_tx_queue_lock = threading.Lock()
|
||||
_tx_queue_depth = 0
|
||||
|
||||
@@ -857,10 +857,18 @@ _lsync_recognizer: dict = {} # device → cached PhonemeRecognizer
|
||||
|
||||
|
||||
def _clear_cuda_cache() -> None:
|
||||
"""Release all unreferenced CUDA tensors and empty the allocator cache."""
|
||||
"""Release all unreferenced CUDA tensors and empty the allocator cache.
|
||||
|
||||
Calls gc.collect() three times: PyTorch model objects frequently form
|
||||
reference cycles (parameters → modules → parent → …) that a single
|
||||
generational GC pass does not fully collect. Three passes ensure objects
|
||||
promoted to older generations are also reclaimed before empty_cache() runs.
|
||||
"""
|
||||
try:
|
||||
import gc, torch
|
||||
gc.collect()
|
||||
gc.collect()
|
||||
gc.collect()
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
except Exception:
|
||||
@@ -1315,6 +1323,11 @@ def run_lyric_transcription(
|
||||
log.debug("[%s] run_lyric_transcription start — model=%s engine=%s phoneme=%s lsync=%s two_pass=%s",
|
||||
job_id, model, engine, phoneme_align, lsync_align, two_pass)
|
||||
|
||||
# Acquire the shared VRAM lock before loading any model.
|
||||
# Prevents a simultaneous stem separation job (running in its own executor
|
||||
# thread) from holding a separator model in VRAM while Whisper loads.
|
||||
jm.gpu_vram_lock.acquire()
|
||||
|
||||
try:
|
||||
if engine == "whisperx":
|
||||
# ── WhisperX path: faster-whisper backend + phoneme-level alignment ──
|
||||
@@ -1843,8 +1856,6 @@ def run_lyric_transcription(
|
||||
else:
|
||||
log.debug("[%s] failed — %s: %s", job_id, type(exc).__name__, exc)
|
||||
jm.update_job(job_id, status=jm.JobStatus.ERROR, error=str(exc), progress="Error.")
|
||||
# Always flush VRAM after a failure so the next queued job starts clean.
|
||||
_clear_cuda_cache()
|
||||
# Restore Silero if a hip-hop patch was applied before the failure.
|
||||
try:
|
||||
if _silero_orig_call is not None:
|
||||
@@ -1852,6 +1863,15 @@ def run_lyric_transcription(
|
||||
_silero_mod.Silero.__call__ = _silero_orig_call
|
||||
except Exception:
|
||||
pass
|
||||
finally:
|
||||
# Always flush VRAM and release the shared lock after the job ends
|
||||
# (success, failure, or cancellation) so the next queued job —
|
||||
# separation or transcription — starts with a clean GPU state.
|
||||
_clear_cuda_cache()
|
||||
try:
|
||||
jm.gpu_vram_lock.release()
|
||||
except RuntimeError:
|
||||
pass # already released (should not happen)
|
||||
|
||||
|
||||
def run_lyric_transcription_with_diarization(
|
||||
|
||||
@@ -320,6 +320,11 @@ def run_separation(
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Acquire the shared VRAM lock before loading any model.
|
||||
# Prevents a simultaneous lyric transcription job (running in its own
|
||||
# executor thread) from loading Whisper while a separator is in VRAM.
|
||||
jm.gpu_vram_lock.acquire()
|
||||
|
||||
# Run separation
|
||||
try:
|
||||
if model.startswith(BSR_PREFIX):
|
||||
@@ -631,15 +636,23 @@ def run_separation(
|
||||
finally:
|
||||
# Always flush PyTorch's VRAM cache after a job ends (success, error, or
|
||||
# cancel) so the next job starts with maximum headroom. The model object
|
||||
# goes out of scope here; gc.collect() ensures its CUDA tensors are
|
||||
# released before empty_cache() returns freed blocks to the driver.
|
||||
# goes out of scope here; gc.collect() x3 ensures its CUDA tensors are
|
||||
# released even when PyTorch model objects form reference cycles that a
|
||||
# single generational GC pass misses.
|
||||
try:
|
||||
import gc, torch
|
||||
gc.collect()
|
||||
gc.collect()
|
||||
gc.collect()
|
||||
if torch.cuda.is_available() and not CPU_SEPARATION:
|
||||
torch.cuda.empty_cache()
|
||||
except Exception:
|
||||
pass
|
||||
# Release VRAM lock so the transcription executor can proceed.
|
||||
try:
|
||||
jm.gpu_vram_lock.release()
|
||||
except RuntimeError:
|
||||
pass # already released (should not happen)
|
||||
|
||||
|
||||
def _get_muted_stems(static_option: Optional[str]) -> List[str]:
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
|
||||
<meta name="apple-mobile-web-app-title" content="StemEngine" />
|
||||
<link rel="apple-touch-icon" href="/icon.svg" />
|
||||
<script type="module" crossorigin src="/assets/index-yrnDLUrD.js"></script>
|
||||
<script type="module" crossorigin src="/assets/index-DbaYmYBz.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/assets/index-02k6pNvN.css">
|
||||
</head>
|
||||
<body>
|
||||
|
||||
@@ -1905,6 +1905,14 @@ export default function LyricsLivePreview({ lrcText, audioUrl, stems, title, sty
|
||||
height: isFullscreen ? undefined : 'clamp(200px, 40vh, 340px)',
|
||||
flex: isFullscreen ? '1 1 0' : undefined,
|
||||
overflow: 'hidden',
|
||||
// clip-path: inset(0) guarantees filter effects (blur, drop-shadow) on child
|
||||
// lines are hard-clipped to the container boundary — overflow:hidden alone
|
||||
// does not reliably clip CSS filter bleed in all browsers.
|
||||
clipPath: 'inset(0)',
|
||||
// Gradient mask fades lines smoothly at the top and bottom edges so
|
||||
// non-active lines appear to enter/exit the window rather than being cut off.
|
||||
maskImage: 'linear-gradient(to bottom, transparent 0%, black 18%, black 82%, transparent 100%)',
|
||||
WebkitMaskImage: 'linear-gradient(to bottom, transparent 0%, black 18%, black 82%, transparent 100%)',
|
||||
position: 'relative',
|
||||
// Stroke and shadow are NOT set on the container — each line div renders them
|
||||
// via its own behindLayer (see lineNeedsLayer / lineStrokeVal / lineShadowVal above).
|
||||
|
||||
Reference in new issue
Block a user