Optimized VRAM usage for separation and transcription jobs. Added documentation for GPU limiter on Windows side.

This commit is contained in:
translator committed 2026-03-10 11:48:54 +09:00
1 parent 3483f4f636
commit 74f632297e
7 files changed
+288 -6

No files matched your search

+32
View File
@@ -647,3 +647,35 @@ Returns `{"added": N, "stems": [...]}`. Idempotent — already-tracked paths are
**Fix:** Added `lsyncAlign` variable and `effectiveWordTs = wordTs || lsyncAlign`. When `lsyncAlign=true`, `effectiveWordTs` is forced `true`, ensuring `'elrc'` is always included in `defaultFormats` and `word_timestamps: true` is sent to the backend.
Also reverted an earlier incorrect change that added `'sylrc'` to formats when `lsyncAlign=true` (user confirmed sylrc is not needed; `phonemeAlign` is no longer used).
### Fix: Lyrics preview lines overflowing the karaoke preview window
**Problem:** Non-active lines with CSS `filter` (e.g. `blur()` in blur-style, `drop-shadow()` in glow-style) could bleed visually outside the `containerRef` boundary because `overflow: hidden` alone does not reliably clip CSS filter effects across all browsers. Additionally, lines were hard-cut at the container edge rather than fading smoothly.
**Fix:** Added two properties to `containerRef` in `LyricsLivePreview.tsx` (`renderLyricsArea()`):
- `clipPath: 'inset(0)'` — guarantees filter bleed is hard-clipped to the container box in all browsers, even where `overflow: hidden` fails.
- `maskImage` / `WebkitMaskImage: 'linear-gradient(to bottom, transparent 0%, black 18%, black 82%, transparent 100%)'` — fades lines in/out at the top and bottom 18% of the preview window so non-active lines appear to enter and exit smoothly rather than being abruptly cut off.
---
## 2026-03-10
### Fix: CUDA out-of-memory errors after several separation jobs
**Root cause (primary):** `_gpu_executor` (stem separation) and `_transcription_executor` (Whisper/WhisperX transcription) are independent `ThreadPoolExecutor(max_workers=1)` instances in `job_manager.py`. Each serialises its own job type, but both can run simultaneously. A separator model (3–6 GB VRAM) and Whisper large-v2 (3–4 GB VRAM) loading at the same time easily exhausts available VRAM on cards ≤12 GB, producing CUDA OOM errors that surface only after several jobs (whenever separation and transcription happen to overlap).
**Root cause (secondary):** `gc.collect()` was called only once in both `separator_service.py`'s `finally` block and `_clear_cuda_cache()`. PyTorch model objects form reference cycles (`_parameters → _modules → parent → …`) that Python's generational GC may not fully collect in a single pass — objects promoted to older generations require additional passes. This could leave model tensors resident in VRAM when the next job started.
**Fix — `backend/job_manager.py`:**
Added `gpu_vram_lock = threading.Lock()` — a shared VRAM coordination lock that both executor paths must hold for the duration of model loading + inference.
**Fix — `backend/services/separator_service.py`:**
- `jm.gpu_vram_lock.acquire()` added before the separation `try:` block; `jm.gpu_vram_lock.release()` added at the end of the `finally:` block (after cleanup).
- Triple `gc.collect()` in the `finally` block instead of single.
**Fix — `backend/services/lyric_transcriber_service.py`:**
- `jm.gpu_vram_lock.acquire()` added before the transcription `try:` block in `run_lyric_transcription`.
- Added a `finally:` block (previously absent) that calls `_clear_cuda_cache()` and releases the lock on both success and failure paths.
- `_clear_cuda_cache()` updated to call `gc.collect()` three times instead of once.
**Effect:** Separation and transcription jobs now queue against the same lock. Whichever starts first holds VRAM exclusively until its `finally` cleanup completes, then releases the lock so the other can proceed. No model can be double-loaded simultaneously regardless of how jobs are submitted.
+202
View File
@@ -0,0 +1,202 @@
# Limiting GPU Power on WSL2
## Background
GPU-intensive Docker workloads (StemEngine, Ollama) can drive the GPU to full TDP,
causing fan noise and heat. On a native Linux host, `nvidia-smi --power-limit` works
from a privileged container. On WSL2 it does not — the NVIDIA driver is paravirtualized
through the Windows WDDM stack, and power management commands require Windows-level
Administrator privileges that no Linux process (including root inside a container) can
obtain.
**The rule:** anything that needs to change GPU power limits must run on the Windows side.
---
## What Does Not Work in WSL2
Both approaches below fail with `Insufficient Permissions` in WSL2, even with
`cap_add: [SYS_ADMIN]` and even calling `nvidia-smi.exe` (the Windows binary) via
WSL2 interop:
- `nvidia-smi --power-limit=N` from inside a Docker container
- `nvidia-smi.exe --power-limit=N` called from a WSL2 shell (even as root)
WSL2 interop runs Windows executables with the current Windows *user* token, not an
elevated Administrator token, so power limit changes are rejected by the driver.
---
## Solution: Windows Task Scheduler
A scheduled task running as **SYSTEM** (highest privilege) calls `nvidia-smi.exe` at
Windows startup before any user session begins. This runs entirely outside WSL2 and
has the permissions required.
### PowerShell script (inline, no file needed)
The task is registered with the command embedded directly to avoid PowerShell execution
policy issues when running as SYSTEM:
```powershell
$action = New-ScheduledTaskAction `
-Execute "powershell.exe" `
-Argument '-NonInteractive -WindowStyle Hidden -Command "& { $max = [double](& nvidia-smi --query-gpu=power.max_limit --format=csv,noheader,nounits).Trim(); $t = [math]::Round($max * 80 / 100); & nvidia-smi --power-limit=$t }"'
Register-ScheduledTask -TaskName "GPU Power Limit" `
-Action $action `
-Trigger (New-ScheduledTaskTrigger -AtStartup) `
-Principal (New-ScheduledTaskPrincipal -UserId "SYSTEM" -RunLevel Highest) `
-Force
```
Change `80` to the desired percentage of the card's max TDP (e.g. `70` for 70%).
Run this in an **elevated PowerShell on Windows** (Win+X → Terminal (Admin)).
### Verify it registered correctly
```powershell
Get-ScheduledTask -TaskName "GPU Power Limit" | Select TaskName, State
Get-ScheduledTaskInfo -TaskName "GPU Power Limit" | Select LastRunTime, LastTaskResult, NextRunTime
```
Expected: `State: Ready`, `LastTaskResult: 0` (success).
### Test without rebooting
```powershell
Start-ScheduledTask -TaskName "GPU Power Limit"
Start-Sleep 3
nvidia-smi -q -d POWER | Select-String "Power Limit"
```
`Current Power Limit` should reflect the new value (e.g. 136 W for 80% of a 170 W card).
### Remove or update
```powershell
# Update: just re-run Register-ScheduledTask with -Force (overwrites)
# Remove:
Unregister-ScheduledTask -TaskName "GPU Power Limit" -Confirm:$false
```
---
## Undoing / Removing the Power Limit
### Remove the Task Scheduler task (Windows)
In an **elevated PowerShell on Windows**:
```powershell
# Remove the startup task
Unregister-ScheduledTask -TaskName "GPU Power Limit" -Confirm:$false
# Restore the GPU to its default power limit immediately
nvidia-smi --power-limit=$(
(nvidia-smi --query-gpu=power.default_limit --format=csv,noheader,nounits).Trim()
)
```
The second command reads the card's factory default TDP from the driver and restores it
without needing to know the value in advance. After the next reboot the driver will
reset to default on its own anyway since the task no longer runs.
Verify the limit is back to default:
```powershell
nvidia-smi -q -d POWER | Select-String "Power Limit"
```
`Current Power Limit` and `Default Power Limit` should match.
### Remove the VRAM reservation (Ollama)
Edit `llama.yml` and delete or comment out the `OLLAMA_GPU_OVERHEAD` line, then
restart Ollama:
```bash
docker compose -f /home/translator/docker/llama.yml up -d ollama
```
### Remove the docker-compose gpu-init services
If you no longer want the (harmless but noisy) `gpu-init` containers in the compose
logs, remove the `gpu-init` service block and its `depends_on` reference from both
`stemengine.yml` and `llama.yml`, then redeploy each stack.
Alternatively, silence them without removing by setting in `.env`:
```env
GPU_POWER_LIMIT=0
```
The `if [ "$$PL" != "0" ]` guard in the entrypoint will skip the `nvidia-smi` call
and the container will exit cleanly with no error output.
---
## Behaviour
| Situation | Power limit |
|---|---|
| Windows boots (before any user logs in) | Task runs → limit set |
| WSL2 starts / Docker containers start | Limit already applied |
| Windows reboots | Task runs again automatically |
| WSL2 shut down mid-session | Limit stays until next reboot (then resets and task re-applies) |
| GPU driver update / reinstall | Limit resets to default; task re-applies on next reboot |
NVIDIA consumer GPU power limits reset to the card's default on driver reload or
system restart, so the task always re-applies the correct limit at boot.
---
## docker-compose gpu-init containers
`stemengine.yml` and `llama.yml` both contain a `gpu-init` service that attempts to
call `nvidia-smi --power-limit` before the main container starts. These exist for
portability (they work on native Linux hosts with a real NVIDIA driver). On WSL2 they
fail harmlessly — the main container still starts because the entrypoint uses `|| true`
semantics — but the power limit is not changed.
Set `GPU_POWER_LIMIT=0` in `.env` to skip the `nvidia-smi` call entirely on WSL2:
```env
GPU_POWER_LIMIT=0
```
The Windows Task Scheduler task is the authoritative power limit mechanism on this host.
---
## Ollama VRAM Reservation
`llama.yml` also sets:
```yaml
OLLAMA_GPU_OVERHEAD=2147483648 # 2 GB
```
This tells Ollama to leave 2 GB of VRAM unallocated, reserving headroom for Windows
desktop compositor and other GPU processes. It is independent of the power limit and
works correctly in WSL2 — it is enforced by Ollama's own allocator, not the driver.
Adjust the value (in bytes) if more or less headroom is needed:
| Headroom | Value |
|---|---|
| 1 GB | `1073741824` |
| 2 GB | `2147483648` |
| 3 GB | `3221225472` |
| 4 GB | `4294967296` |
---
## Summary
| Mechanism | Where it runs | Works in WSL2 | Purpose |
|---|---|---|---|
| Task Scheduler "GPU Power Limit" | Windows (SYSTEM) | ✅ | Enforces watt cap at boot |
| `gpu-init` docker service | WSL2 container | ❌ | Portability shim for native Linux |
| `OLLAMA_GPU_OVERHEAD` | Ollama process | ✅ | Reserves VRAM for Windows |
+7
View File
@@ -110,6 +110,13 @@ _general_executor = ThreadPoolExecutor(max_workers=4)
_gpu_executor = ThreadPoolExecutor(max_workers=1)
_transcription_executor = ThreadPoolExecutor(max_workers=1)
# Shared VRAM coordination lock.
# Both _gpu_executor (separation) and _transcription_executor (Whisper/WhisperX)
# must hold this lock for the duration of GPU model loading + inference.
# Prevents the two independent executor threads from loading models simultaneously
# and triggering CUDA out-of-memory errors.
gpu_vram_lock = threading.Lock()
# Queue-depth counter for the transcription executor (includes the running job).
_tx_queue_lock = threading.Lock()
_tx_queue_depth = 0
+23 -3
View File
@@ -857,10 +857,18 @@ _lsync_recognizer: dict = {} # device → cached PhonemeRecognizer
def _clear_cuda_cache() -> None:
"""Release all unreferenced CUDA tensors and empty the allocator cache."""
"""Release all unreferenced CUDA tensors and empty the allocator cache.
Calls gc.collect() three times: PyTorch model objects frequently form
reference cycles (parameters → modules → parent → …) that a single
generational GC pass does not fully collect. Three passes ensure objects
promoted to older generations are also reclaimed before empty_cache() runs.
"""
try:
import gc, torch
gc.collect()
gc.collect()
gc.collect()
if torch.cuda.is_available():
torch.cuda.empty_cache()
except Exception:
@@ -1315,6 +1323,11 @@ def run_lyric_transcription(
log.debug("[%s] run_lyric_transcription start — model=%s engine=%s phoneme=%s lsync=%s two_pass=%s",
job_id, model, engine, phoneme_align, lsync_align, two_pass)
# Acquire the shared VRAM lock before loading any model.
# Prevents a simultaneous stem separation job (running in its own executor
# thread) from holding a separator model in VRAM while Whisper loads.
jm.gpu_vram_lock.acquire()
try:
if engine == "whisperx":
# ── WhisperX path: faster-whisper backend + phoneme-level alignment ──
@@ -1843,8 +1856,6 @@ def run_lyric_transcription(
else:
log.debug("[%s] failed — %s: %s", job_id, type(exc).__name__, exc)
jm.update_job(job_id, status=jm.JobStatus.ERROR, error=str(exc), progress="Error.")
# Always flush VRAM after a failure so the next queued job starts clean.
_clear_cuda_cache()
# Restore Silero if a hip-hop patch was applied before the failure.
try:
if _silero_orig_call is not None:
@@ -1852,6 +1863,15 @@ def run_lyric_transcription(
_silero_mod.Silero.__call__ = _silero_orig_call
except Exception:
pass
finally:
# Always flush VRAM and release the shared lock after the job ends
# (success, failure, or cancellation) so the next queued job —
# separation or transcription — starts with a clean GPU state.
_clear_cuda_cache()
try:
jm.gpu_vram_lock.release()
except RuntimeError:
pass # already released (should not happen)
def run_lyric_transcription_with_diarization(
+15 -2
View File
@@ -320,6 +320,11 @@ def run_separation(
except Exception:
pass
# Acquire the shared VRAM lock before loading any model.
# Prevents a simultaneous lyric transcription job (running in its own
# executor thread) from loading Whisper while a separator is in VRAM.
jm.gpu_vram_lock.acquire()
# Run separation
try:
if model.startswith(BSR_PREFIX):
@@ -631,15 +636,23 @@ def run_separation(
finally:
# Always flush PyTorch's VRAM cache after a job ends (success, error, or
# cancel) so the next job starts with maximum headroom. The model object
# goes out of scope here; gc.collect() ensures its CUDA tensors are
# released before empty_cache() returns freed blocks to the driver.
# goes out of scope here; gc.collect() x3 ensures its CUDA tensors are
# released even when PyTorch model objects form reference cycles that a
# single generational GC pass misses.
try:
import gc, torch
gc.collect()
gc.collect()
gc.collect()
if torch.cuda.is_available() and not CPU_SEPARATION:
torch.cuda.empty_cache()
except Exception:
pass
# Release VRAM lock so the transcription executor can proceed.
try:
jm.gpu_vram_lock.release()
except RuntimeError:
pass # already released (should not happen)
def _get_muted_stems(static_option: Optional[str]) -> List[str]:
+1 -1
View File
@@ -19,7 +19,7 @@
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
<meta name="apple-mobile-web-app-title" content="StemEngine" />
<link rel="apple-touch-icon" href="/icon.svg" />
<script type="module" crossorigin src="/assets/index-yrnDLUrD.js"></script>
<script type="module" crossorigin src="/assets/index-DbaYmYBz.js"></script>
<link rel="stylesheet" crossorigin href="/assets/index-02k6pNvN.css">
</head>
<body>
@@ -1905,6 +1905,14 @@ export default function LyricsLivePreview({ lrcText, audioUrl, stems, title, sty
height: isFullscreen ? undefined : 'clamp(200px, 40vh, 340px)',
flex: isFullscreen ? '1 1 0' : undefined,
overflow: 'hidden',
// clip-path: inset(0) guarantees filter effects (blur, drop-shadow) on child
// lines are hard-clipped to the container boundary — overflow:hidden alone
// does not reliably clip CSS filter bleed in all browsers.
clipPath: 'inset(0)',
// Gradient mask fades lines smoothly at the top and bottom edges so
// non-active lines appear to enter/exit the window rather than being cut off.
maskImage: 'linear-gradient(to bottom, transparent 0%, black 18%, black 82%, transparent 100%)',
WebkitMaskImage: 'linear-gradient(to bottom, transparent 0%, black 18%, black 82%, transparent 100%)',
position: 'relative',
// Stroke and shadow are NOT set on the container — each line div renders them
// via its own behindLayer (see lineNeedsLayer / lineStrokeVal / lineShadowVal above).