diff --git a/SUMMARIES.md b/SUMMARIES.md index 7658fa9..ca3475d 100644 --- a/SUMMARIES.md +++ b/SUMMARIES.md @@ -647,3 +647,35 @@ Returns `{"added": N, "stems": [...]}`. Idempotent — already-tracked paths are **Fix:** Added `lsyncAlign` variable and `effectiveWordTs = wordTs || lsyncAlign`. When `lsyncAlign=true`, `effectiveWordTs` is forced `true`, ensuring `'elrc'` is always included in `defaultFormats` and `word_timestamps: true` is sent to the backend. Also reverted an earlier incorrect change that added `'sylrc'` to formats when `lsyncAlign=true` (user confirmed sylrc is not needed; `phonemeAlign` is no longer used). + +### Fix: Lyrics preview lines overflowing the karaoke preview window + +**Problem:** Non-active lines with CSS `filter` (e.g. `blur()` in blur-style, `drop-shadow()` in glow-style) could bleed visually outside the `containerRef` boundary because `overflow: hidden` alone does not reliably clip CSS filter effects across all browsers. Additionally, lines were hard-cut at the container edge rather than fading smoothly. + +**Fix:** Added two properties to `containerRef` in `LyricsLivePreview.tsx` (`renderLyricsArea()`): +- `clipPath: 'inset(0)'` — guarantees filter bleed is hard-clipped to the container box in all browsers, even where `overflow: hidden` fails. +- `maskImage` / `WebkitMaskImage: 'linear-gradient(to bottom, transparent 0%, black 18%, black 82%, transparent 100%)'` — fades lines in/out at the top and bottom 18% of the preview window so non-active lines appear to enter and exit smoothly rather than being abruptly cut off. + +--- + +## 2026-03-10 + +### Fix: CUDA out-of-memory errors after several separation jobs + +**Root cause (primary):** `_gpu_executor` (stem separation) and `_transcription_executor` (Whisper/WhisperX transcription) are independent `ThreadPoolExecutor(max_workers=1)` instances in `job_manager.py`. Each serialises its own job type, but both can run simultaneously. A separator model (3–6 GB VRAM) and Whisper large-v2 (3–4 GB VRAM) loading at the same time easily exhausts available VRAM on cards ≤12 GB, producing CUDA OOM errors that surface only after several jobs (whenever separation and transcription happen to overlap). + +**Root cause (secondary):** `gc.collect()` was called only once in both `separator_service.py`'s `finally` block and `_clear_cuda_cache()`. PyTorch model objects form reference cycles (`_parameters → _modules → parent → …`) that Python's generational GC may not fully collect in a single pass — objects promoted to older generations require additional passes. This could leave model tensors resident in VRAM when the next job started. + +**Fix — `backend/job_manager.py`:** +Added `gpu_vram_lock = threading.Lock()` — a shared VRAM coordination lock that both executor paths must hold for the duration of model loading + inference. + +**Fix — `backend/services/separator_service.py`:** +- `jm.gpu_vram_lock.acquire()` added before the separation `try:` block; `jm.gpu_vram_lock.release()` added at the end of the `finally:` block (after cleanup). +- Triple `gc.collect()` in the `finally` block instead of single. + +**Fix — `backend/services/lyric_transcriber_service.py`:** +- `jm.gpu_vram_lock.acquire()` added before the transcription `try:` block in `run_lyric_transcription`. +- Added a `finally:` block (previously absent) that calls `_clear_cuda_cache()` and releases the lock on both success and failure paths. +- `_clear_cuda_cache()` updated to call `gc.collect()` three times instead of once. + +**Effect:** Separation and transcription jobs now queue against the same lock. Whichever starts first holds VRAM exclusively until its `finally` cleanup completes, then releases the lock so the other can proceed. No model can be double-loaded simultaneously regardless of how jobs are submitted. diff --git a/WSL2_Limiting GPU.md b/WSL2_Limiting GPU.md new file mode 100644 index 0000000..d08afb5 --- /dev/null +++ b/WSL2_Limiting GPU.md @@ -0,0 +1,202 @@ +# Limiting GPU Power on WSL2 + +## Background + +GPU-intensive Docker workloads (StemEngine, Ollama) can drive the GPU to full TDP, +causing fan noise and heat. On a native Linux host, `nvidia-smi --power-limit` works +from a privileged container. On WSL2 it does not — the NVIDIA driver is paravirtualized +through the Windows WDDM stack, and power management commands require Windows-level +Administrator privileges that no Linux process (including root inside a container) can +obtain. + +**The rule:** anything that needs to change GPU power limits must run on the Windows side. + +--- + +## What Does Not Work in WSL2 + +Both approaches below fail with `Insufficient Permissions` in WSL2, even with +`cap_add: [SYS_ADMIN]` and even calling `nvidia-smi.exe` (the Windows binary) via +WSL2 interop: + +- `nvidia-smi --power-limit=N` from inside a Docker container +- `nvidia-smi.exe --power-limit=N` called from a WSL2 shell (even as root) + +WSL2 interop runs Windows executables with the current Windows *user* token, not an +elevated Administrator token, so power limit changes are rejected by the driver. + +--- + +## Solution: Windows Task Scheduler + +A scheduled task running as **SYSTEM** (highest privilege) calls `nvidia-smi.exe` at +Windows startup before any user session begins. This runs entirely outside WSL2 and +has the permissions required. + +### PowerShell script (inline, no file needed) + +The task is registered with the command embedded directly to avoid PowerShell execution +policy issues when running as SYSTEM: + +```powershell +$action = New-ScheduledTaskAction ` + -Execute "powershell.exe" ` + -Argument '-NonInteractive -WindowStyle Hidden -Command "& { $max = [double](& nvidia-smi --query-gpu=power.max_limit --format=csv,noheader,nounits).Trim(); $t = [math]::Round($max * 80 / 100); & nvidia-smi --power-limit=$t }"' + +Register-ScheduledTask -TaskName "GPU Power Limit" ` + -Action $action ` + -Trigger (New-ScheduledTaskTrigger -AtStartup) ` + -Principal (New-ScheduledTaskPrincipal -UserId "SYSTEM" -RunLevel Highest) ` + -Force +``` + +Change `80` to the desired percentage of the card's max TDP (e.g. `70` for 70%). + +Run this in an **elevated PowerShell on Windows** (Win+X → Terminal (Admin)). + +### Verify it registered correctly + +```powershell +Get-ScheduledTask -TaskName "GPU Power Limit" | Select TaskName, State +Get-ScheduledTaskInfo -TaskName "GPU Power Limit" | Select LastRunTime, LastTaskResult, NextRunTime +``` + +Expected: `State: Ready`, `LastTaskResult: 0` (success). + +### Test without rebooting + +```powershell +Start-ScheduledTask -TaskName "GPU Power Limit" +Start-Sleep 3 +nvidia-smi -q -d POWER | Select-String "Power Limit" +``` + +`Current Power Limit` should reflect the new value (e.g. 136 W for 80% of a 170 W card). + +### Remove or update + +```powershell +# Update: just re-run Register-ScheduledTask with -Force (overwrites) +# Remove: +Unregister-ScheduledTask -TaskName "GPU Power Limit" -Confirm:$false +``` + +--- + +## Undoing / Removing the Power Limit + +### Remove the Task Scheduler task (Windows) + +In an **elevated PowerShell on Windows**: + +```powershell +# Remove the startup task +Unregister-ScheduledTask -TaskName "GPU Power Limit" -Confirm:$false + +# Restore the GPU to its default power limit immediately +nvidia-smi --power-limit=$( + (nvidia-smi --query-gpu=power.default_limit --format=csv,noheader,nounits).Trim() +) +``` + +The second command reads the card's factory default TDP from the driver and restores it +without needing to know the value in advance. After the next reboot the driver will +reset to default on its own anyway since the task no longer runs. + +Verify the limit is back to default: + +```powershell +nvidia-smi -q -d POWER | Select-String "Power Limit" +``` + +`Current Power Limit` and `Default Power Limit` should match. + +### Remove the VRAM reservation (Ollama) + +Edit `llama.yml` and delete or comment out the `OLLAMA_GPU_OVERHEAD` line, then +restart Ollama: + +```bash +docker compose -f /home/translator/docker/llama.yml up -d ollama +``` + +### Remove the docker-compose gpu-init services + +If you no longer want the (harmless but noisy) `gpu-init` containers in the compose +logs, remove the `gpu-init` service block and its `depends_on` reference from both +`stemengine.yml` and `llama.yml`, then redeploy each stack. + +Alternatively, silence them without removing by setting in `.env`: + +```env +GPU_POWER_LIMIT=0 +``` + +The `if [ "$$PL" != "0" ]` guard in the entrypoint will skip the `nvidia-smi` call +and the container will exit cleanly with no error output. + +--- + +## Behaviour + +| Situation | Power limit | +|---|---| +| Windows boots (before any user logs in) | Task runs → limit set | +| WSL2 starts / Docker containers start | Limit already applied | +| Windows reboots | Task runs again automatically | +| WSL2 shut down mid-session | Limit stays until next reboot (then resets and task re-applies) | +| GPU driver update / reinstall | Limit resets to default; task re-applies on next reboot | + +NVIDIA consumer GPU power limits reset to the card's default on driver reload or +system restart, so the task always re-applies the correct limit at boot. + +--- + +## docker-compose gpu-init containers + +`stemengine.yml` and `llama.yml` both contain a `gpu-init` service that attempts to +call `nvidia-smi --power-limit` before the main container starts. These exist for +portability (they work on native Linux hosts with a real NVIDIA driver). On WSL2 they +fail harmlessly — the main container still starts because the entrypoint uses `|| true` +semantics — but the power limit is not changed. + +Set `GPU_POWER_LIMIT=0` in `.env` to skip the `nvidia-smi` call entirely on WSL2: + +```env +GPU_POWER_LIMIT=0 +``` + +The Windows Task Scheduler task is the authoritative power limit mechanism on this host. + +--- + +## Ollama VRAM Reservation + +`llama.yml` also sets: + +```yaml +OLLAMA_GPU_OVERHEAD=2147483648 # 2 GB +``` + +This tells Ollama to leave 2 GB of VRAM unallocated, reserving headroom for Windows +desktop compositor and other GPU processes. It is independent of the power limit and +works correctly in WSL2 — it is enforced by Ollama's own allocator, not the driver. + +Adjust the value (in bytes) if more or less headroom is needed: + +| Headroom | Value | +|---|---| +| 1 GB | `1073741824` | +| 2 GB | `2147483648` | +| 3 GB | `3221225472` | +| 4 GB | `4294967296` | + +--- + +## Summary + +| Mechanism | Where it runs | Works in WSL2 | Purpose | +|---|---|---|---| +| Task Scheduler "GPU Power Limit" | Windows (SYSTEM) | ✅ | Enforces watt cap at boot | +| `gpu-init` docker service | WSL2 container | ❌ | Portability shim for native Linux | +| `OLLAMA_GPU_OVERHEAD` | Ollama process | ✅ | Reserves VRAM for Windows | diff --git a/backend/job_manager.py b/backend/job_manager.py index 69cd0d6..67c830c 100755 --- a/backend/job_manager.py +++ b/backend/job_manager.py @@ -110,6 +110,13 @@ _general_executor = ThreadPoolExecutor(max_workers=4) _gpu_executor = ThreadPoolExecutor(max_workers=1) _transcription_executor = ThreadPoolExecutor(max_workers=1) +# Shared VRAM coordination lock. +# Both _gpu_executor (separation) and _transcription_executor (Whisper/WhisperX) +# must hold this lock for the duration of GPU model loading + inference. +# Prevents the two independent executor threads from loading models simultaneously +# and triggering CUDA out-of-memory errors. +gpu_vram_lock = threading.Lock() + # Queue-depth counter for the transcription executor (includes the running job). _tx_queue_lock = threading.Lock() _tx_queue_depth = 0 diff --git a/backend/services/lyric_transcriber_service.py b/backend/services/lyric_transcriber_service.py index b80d357..62a5298 100755 --- a/backend/services/lyric_transcriber_service.py +++ b/backend/services/lyric_transcriber_service.py @@ -857,10 +857,18 @@ _lsync_recognizer: dict = {} # device → cached PhonemeRecognizer def _clear_cuda_cache() -> None: - """Release all unreferenced CUDA tensors and empty the allocator cache.""" + """Release all unreferenced CUDA tensors and empty the allocator cache. + + Calls gc.collect() three times: PyTorch model objects frequently form + reference cycles (parameters → modules → parent → …) that a single + generational GC pass does not fully collect. Three passes ensure objects + promoted to older generations are also reclaimed before empty_cache() runs. + """ try: import gc, torch gc.collect() + gc.collect() + gc.collect() if torch.cuda.is_available(): torch.cuda.empty_cache() except Exception: @@ -1315,6 +1323,11 @@ def run_lyric_transcription( log.debug("[%s] run_lyric_transcription start — model=%s engine=%s phoneme=%s lsync=%s two_pass=%s", job_id, model, engine, phoneme_align, lsync_align, two_pass) + # Acquire the shared VRAM lock before loading any model. + # Prevents a simultaneous stem separation job (running in its own executor + # thread) from holding a separator model in VRAM while Whisper loads. + jm.gpu_vram_lock.acquire() + try: if engine == "whisperx": # ── WhisperX path: faster-whisper backend + phoneme-level alignment ── @@ -1843,8 +1856,6 @@ def run_lyric_transcription( else: log.debug("[%s] failed — %s: %s", job_id, type(exc).__name__, exc) jm.update_job(job_id, status=jm.JobStatus.ERROR, error=str(exc), progress="Error.") - # Always flush VRAM after a failure so the next queued job starts clean. - _clear_cuda_cache() # Restore Silero if a hip-hop patch was applied before the failure. try: if _silero_orig_call is not None: @@ -1852,6 +1863,15 @@ def run_lyric_transcription( _silero_mod.Silero.__call__ = _silero_orig_call except Exception: pass + finally: + # Always flush VRAM and release the shared lock after the job ends + # (success, failure, or cancellation) so the next queued job — + # separation or transcription — starts with a clean GPU state. + _clear_cuda_cache() + try: + jm.gpu_vram_lock.release() + except RuntimeError: + pass # already released (should not happen) def run_lyric_transcription_with_diarization( diff --git a/backend/services/separator_service.py b/backend/services/separator_service.py index fce40f1..0c4b633 100755 --- a/backend/services/separator_service.py +++ b/backend/services/separator_service.py @@ -320,6 +320,11 @@ def run_separation( except Exception: pass + # Acquire the shared VRAM lock before loading any model. + # Prevents a simultaneous lyric transcription job (running in its own + # executor thread) from loading Whisper while a separator is in VRAM. + jm.gpu_vram_lock.acquire() + # Run separation try: if model.startswith(BSR_PREFIX): @@ -631,15 +636,23 @@ def run_separation( finally: # Always flush PyTorch's VRAM cache after a job ends (success, error, or # cancel) so the next job starts with maximum headroom. The model object - # goes out of scope here; gc.collect() ensures its CUDA tensors are - # released before empty_cache() returns freed blocks to the driver. + # goes out of scope here; gc.collect() x3 ensures its CUDA tensors are + # released even when PyTorch model objects form reference cycles that a + # single generational GC pass misses. try: import gc, torch gc.collect() + gc.collect() + gc.collect() if torch.cuda.is_available() and not CPU_SEPARATION: torch.cuda.empty_cache() except Exception: pass + # Release VRAM lock so the transcription executor can proceed. + try: + jm.gpu_vram_lock.release() + except RuntimeError: + pass # already released (should not happen) def _get_muted_stems(static_option: Optional[str]) -> List[str]: diff --git a/backend/static/index.html b/backend/static/index.html index 3cf49ea..ed78f1d 100644 --- a/backend/static/index.html +++ b/backend/static/index.html @@ -19,7 +19,7 @@ - + diff --git a/frontend/src/components/LyricsLivePreview.tsx b/frontend/src/components/LyricsLivePreview.tsx index 97faab4..d29f98c 100755 --- a/frontend/src/components/LyricsLivePreview.tsx +++ b/frontend/src/components/LyricsLivePreview.tsx @@ -1905,6 +1905,14 @@ export default function LyricsLivePreview({ lrcText, audioUrl, stems, title, sty height: isFullscreen ? undefined : 'clamp(200px, 40vh, 340px)', flex: isFullscreen ? '1 1 0' : undefined, overflow: 'hidden', + // clip-path: inset(0) guarantees filter effects (blur, drop-shadow) on child + // lines are hard-clipped to the container boundary — overflow:hidden alone + // does not reliably clip CSS filter bleed in all browsers. + clipPath: 'inset(0)', + // Gradient mask fades lines smoothly at the top and bottom edges so + // non-active lines appear to enter/exit the window rather than being cut off. + maskImage: 'linear-gradient(to bottom, transparent 0%, black 18%, black 82%, transparent 100%)', + WebkitMaskImage: 'linear-gradient(to bottom, transparent 0%, black 18%, black 82%, transparent 100%)', position: 'relative', // Stroke and shadow are NOT set on the container — each line div renders them // via its own behindLayer (see lineNeedsLayer / lineStrokeVal / lineShadowVal above).