Myriad improvements.

This commit is contained in:
translator committed 2026-03-17 11:39:17 +09:00
1 parent e74819d428
commit 4982cf8427
22 files changed
+2351 -3990

No files matched your search

+197
View File
@@ -2262,3 +2262,200 @@ Status impact:
- Runtime blocker for Parakeet backend execution is cleared.
- Remaining work is manual UI sign-off coverage and rollout approval documentation.
### 2026-03-17 continuation (LyricsLivePreview follow-up hardening)
Reviewed and continued hardening `frontend/src/components/LyricsLivePreview.tsx` after the
canvas-based renderer migration.
Artifacts added:
- `20260317_LyricsLivePreview_Review.md`
Review findings logged:
- style reset dropped caller-provided defaults
- speaker localStorage state could bleed across `storageKey` changes
- RAF loop continued running while idle
- multi-stem audio resources lacked teardown
- legacy recenter path was still present even though the active renderer is `LyricsCanvas`
Fixes implemented:
- reset now restores merged defaults derived from incoming `styleOptions`
- speaker preferences are rehydrated on `storageKey` change with a hydration guard to avoid stale writes
- paused preview now stops the RAF loop instead of scheduling idle animation frames continuously
- multi-stem preview now disposes removed/unmounted audio elements, metadata listeners,
source nodes, gain nodes, and closes the `AudioContext` on final teardown
- removed the dead recenter button/effect/refs from the canvas-based preview path
Validation:
- file diagnostics clean for `frontend/src/components/LyricsLivePreview.tsx`
- frontend typecheck passes (`npm run typecheck`)
### 2026-03-17 continuation (LyricsLivePreview no longer depends on LyricsCanvas.tsx)
Refactored `frontend/src/components/LyricsLivePreview.tsx` so it no longer imports or depends on
`frontend/src/components/LyricsCanvas.tsx`.
What changed:
- inlined the Two.js/SVG lyric renderer and its scene/update helpers directly into
`LyricsLivePreview.tsx`
- removed the `LyricsCanvas` import from the preview component
- reconciled the inlined renderer with the preview component's existing `StyleState`
shape (`wordHighlight` / `dynBg`)
Validation:
- no remaining `LyricsCanvas` import in `LyricsLivePreview.tsx`
- file diagnostics clean for `frontend/src/components/LyricsLivePreview.tsx`
- frontend typecheck passes (`npm run typecheck`)
### 2026-03-17 continuation (final cleanup: remove obsolete LyricsCanvas.tsx)
Completed the follow-up cleanup after inlining the renderer into
`frontend/src/components/LyricsLivePreview.tsx`:
- removed obsolete file: `frontend/src/components/LyricsCanvas.tsx`
- confirmed no remaining frontend references to `LyricsCanvas` import paths
- re-ran frontend typecheck successfully (`npm run typecheck`)
### 2026-03-17 continuation (reviewed remaining frontend components + fix plan)
Completed a targeted review of frontend components under `frontend/src/components/`
excluding `LyricsLivePreview.tsx`, and documented findings plus a phased remediation plan.
Review artifact created:
- `20260317_Components_Review_and_Fix_Plan.md`
Scope reviewed:
- `AutoPipelineDialog.tsx`
- `DynamicMixPlayer.tsx`
- `FileUpload.tsx`
- `GlobalAudioBar.tsx`
- `JobStatus.tsx`
- `KaraokeEditor.tsx`
- `ResultFiles.tsx`
- `SettingsPanel.tsx`
- `Sidebar.tsx`
Prioritized issues captured in the plan:
- high: DynamicMixPlayer resource lifecycle/disposal gaps
- high: Sidebar project delete flow may update UI as if success on failed delete
- medium: KaraokeEditor word timing lower-bound clamp should enforce previous-word ordering
- medium: AutoPipelineDialog abort-listener cleanup hygiene in retry/poll delays
- medium: DynamicMixPlayer state reconciliation when stem set changes
- low/polish: invalid Tailwind class in SettingsPanel; FileUpload multi-select UX mismatch; Subsonic mutation error feedback
Next-step plan included in artifact:
- Phase 1 stability hotfixes
- Phase 2 reliability cleanup
- Phase 3 UX polish
- explicit validation checklist (`npm run typecheck` + focused manual checks)
### 2026-03-17 continuation (implemented components hardening plan)
Implemented the component fixes from `20260317_Components_Review_and_Fix_Plan.md`.
Updated files:
- `frontend/src/components/DynamicMixPlayer.tsx`
- `frontend/src/components/Sidebar.tsx`
- `frontend/src/components/KaraokeEditor.tsx`
- `frontend/src/components/AutoPipelineDialog.tsx`
- `frontend/src/components/SettingsPanel.tsx`
- `frontend/src/components/FileUpload.tsx`
What was implemented:
- DynamicMixPlayer
- added per-stem disposal helper for `HTMLAudioElement` + listeners + WebAudio nodes
- dispose removed stems when stem-set changes
- full unmount teardown (pause/dispose all stems, close `AudioContext`, unregister)
- reconciled per-stem state maps (`gains`, `loaded`, `mutedStems`, `soloedStems`) when stem-set changes
- applied gain matrix reactively when relevant state changes
- tightened readiness check to require loaded metadata for all stems
- Sidebar
- project deletion now updates UI only after successful backend delete
- added visible failure feedback for project delete and import-delete failures
- KaraokeEditor
- enforced monotonic word timing bounds (respect previous word lower bound + next word/line upper bound)
- applied same constraint path for drag edits and typed edits
- made waveform fetch/decode pipeline abortable on cleanup
- AutoPipelineDialog
- replaced ad hoc abortable timeout blocks with a shared `sleepWithAbort()` helper
- ensured abort listeners are removed on both resolve and abort paths
- SettingsPanel
- fixed invalid Tailwind token (`dark:hover:bg-gray-750` → `dark:hover:bg-gray-700`)
- FileUpload
- added `multiple` prop (default `true`) and wired it to the hidden file input
Validation:
- frontend typecheck passes (`npm run typecheck`)
### 2026-03-17 continuation (backend review: lrc_service + lyric_transcriber_service)
Completed issue review and documented findings for:
- `backend/services/lrc_service.py`
- `backend/services/lyric_transcriber_service.py`
Review artifact created:
- `20260317_lrc_and_lyric_transcriber_review.md`
Top issues flagged in report:
- high: Whisper timeout thread can continue while GPU lock is released (race/resource contention risk)
- high: global `torch.load` monkey patch forcing `weights_only=False`
- high: secondary diarization path diverges from primary GPU lock/cleanup orchestration
- medium: HTML preview duration reads wrong data attribute (`data-start` vs `data-time`)
- medium: HTML generation interpolates unescaped dynamic content
- medium: bare cleanup `except: pass` hides temp-file IO errors
Report includes a phased fix plan and validation checklist.
### 2026-03-17 continuation (implemented remediation: lrc_service + lyric_transcriber_service)
Implemented remediation items from `20260317_lrc_and_lyric_transcriber_review.md`.
Updated files:
- `backend/services/lrc_service.py`
- `backend/services/lyric_transcriber_service.py`
What was fixed:
- `lrc_service.py`
- fixed styled preview duration parser mismatch (`[data-start]` → derived from `data-time`)
- escaped dynamic lyric/title HTML content (`html.escape`)
- added style-value sanitization for font family, color, numeric ranges, and text alignment
- replaced bare cleanup `except: pass` with `except OSError` + debug logging
- `lyric_transcriber_service.py`
- added env-gated guardrail for torch-load compatibility patch (`STEMENGINE_PATCH_TORCH_LOAD`)
- added startup warning when compatibility patch is enabled
- removed detached timeout-thread pattern in Whisper path and run transcription inline under GPU lock
to eliminate lock-release/background-worker race
- added shared GPU lock + cleanup symmetry to `run_lyric_transcription_with_diarization()`
Remediation report artifact:
- `20260317_lrc_lyric_transcriber_remediation.md`
Validation:
- `python3 -m py_compile backend/services/lrc_service.py backend/services/lyric_transcriber_service.py`
completed successfully
### 2026-03-17 continuation (silence-aware lyric fill timing)
**Problem:** Word/syllable highlights in the lyrics preview extended their fill animation across long
gaps and silences before the next token, producing slow-creeping or jarring overly-long transitions.
**Root cause:** Both the DOM RAF path (`wordFillPct`) and the canvas/Two.js path (per-`CanvasWordUnit`
`duration`) used the raw distance to the next word start (or next line start) as the full fill window,
with no awareness of whether that distance represented active speech or silent dead air.
**Fix — `frontend/src/components/LyricsLivePreview.tsx`:**
- Added two module-level constants:
- `FILL_GAP_THRESHOLD_S = 0.6` — gaps wider than 600 ms are treated as silences
- `FILL_MAX_DURATION_S = 0.5` — fill window is capped to 500 ms for silence gaps
- Added `silenceAwareEnd(wordStart, rawEnd)` helper that applies the cap when needed.
For tight inter-word gaps (≤ 600 ms) the raw next-token start is used unchanged;
for silences the effective end is `wordStart + 500 ms`, so the highlight sweeps
quickly and then freezes cleanly until the next word is reached.
- **DOM path** — `wordFillPct()` now computes `rawEnd` first and routes it through
`silenceAwareEnd()` before deriving `dur`.
- **Canvas path** — the `CanvasWordUnit.duration` assignment now computes `rawEnd`
(next word time or `line.endTime` fallback) and routes it through `silenceAwareEnd()`
before subtracting `word.time`, matching the DOM path exactly.
Validation:
- `npm run typecheck` completed with no errors
+138
View File
@@ -0,0 +1,138 @@
# 2026-03-17 — Frontend Components Review (excluding `LyricsLivePreview.tsx`)
## Scope reviewed
- `frontend/src/components/AutoPipelineDialog.tsx`
- `frontend/src/components/DynamicMixPlayer.tsx`
- `frontend/src/components/FileUpload.tsx`
- `frontend/src/components/GlobalAudioBar.tsx`
- `frontend/src/components/JobStatus.tsx`
- `frontend/src/components/KaraokeEditor.tsx`
- `frontend/src/components/ResultFiles.tsx`
- `frontend/src/components/SettingsPanel.tsx`
- `frontend/src/components/Sidebar.tsx`
---
## Executive summary
- **High priority fixes:** 2
- **Medium priority fixes:** 4
- **Low priority fixes / polish:** 3
- **No immediate issues found:** `GlobalAudioBar.tsx`, `ResultFiles.tsx`, `JobStatus.tsx` (functional review only)
---
## Findings
### High
1) **DynamicMixPlayer does not dispose stem audio graph/resources on unmount or stem-set changes**
- **File:** `frontend/src/components/DynamicMixPlayer.tsx`
- **Evidence:** lines ~50-54, 63-91
- **Why this matters:** `HTMLAudioElement`, `MediaElementAudioSourceNode`, and `GainNode` are created but never disconnected/removed per stem. This can leak nodes/listeners and cause stale playback state after navigation or stem list mutation.
- **Recommended fix:**
- Add a `disposeStemAudio(name)` helper to pause audio, remove listeners, clear `src`, disconnect source/gain nodes, delete refs.
- On `stems` changes, remove refs for missing stems.
- On unmount, dispose all stems and close `AudioContext`.
2) **Project delete path can update UI as if delete succeeded even when API delete fails**
- **File:** `frontend/src/components/Sidebar.tsx`
- **Evidence:** `ProjectItem.handleDelete()` around lines ~123-131
- **Why this matters:** Errors are swallowed, then `onDeleted()` is still called. UI can desync from backend state and wrongly close tabs/remove project views.
- **Recommended fix:**
- Only call `onDeleted()` after successful `deleteProject`.
- Surface error toast or inline message when deletion fails.
### Medium
3) **Word timing edits can violate monotonic order against previous word**
- **File:** `frontend/src/components/KaraokeEditor.tsx`
- **Evidence:** clamping uses `line.time` and next-word cap only (lines ~531, ~547, plus drag path ~780+)
- **Why this matters:** A word can be set earlier than the previous word, creating invalid ELRC timing order.
- **Recommended fix:**
- Clamp lower bound to `max(line.time, prevWord.time + 0.01)`.
- Apply same rule for drag edits and typed edits.
4) **Abort listeners are attached repeatedly without cleanup in retry/poll delay helpers**
- **File:** `frontend/src/components/AutoPipelineDialog.tsx`
- **Evidence:** lines ~69 and ~84
- **Why this matters:** Repeated runs/retries can accumulate listeners unnecessarily.
- **Recommended fix:**
- Use `{ once: true }` for abort listeners and/or store a handler and remove it after timeout resolves.
5) **`DynamicMixPlayer` local state not reinitialized for changed stem sets**
- **File:** `frontend/src/components/DynamicMixPlayer.tsx`
- **Evidence:** `gains` initialized once from initial props (line ~27), no reconciliation effect.
- **Why this matters:** New stems may lack initialized gain/mute/solo state; removed stems may leave stale entries.
- **Recommended fix:**
- Add reconciliation when `stems` changes to initialize missing keys and prune removed keys in `gains`, `loaded`, `mutedStems`, `soloedStems`.
6) **Waveform fetch/decode has no request cancellation**
- **File:** `frontend/src/components/KaraokeEditor.tsx`
- **Evidence:** audio decode effect around lines ~230-248
- **Why this matters:** On rapid source switching/unmount, unnecessary network/decode work continues.
- **Recommended fix:**
- Add `AbortController` to `fetch(audioUrl)`.
- Guard decode pipeline and skip `decodeAudioData` work after abort.
### Low / polish
7) **Invalid Tailwind utility class likely never applies**
- **File:** `frontend/src/components/SettingsPanel.tsx`
- **Evidence:** `dark:hover:bg-gray-750` at line ~82
- **Why this matters:** Style inconsistency and dead class.
- **Recommended fix:** replace with valid token (`dark:hover:bg-gray-700` or configured custom scale).
8) **`FileUpload` supports multi-file callback but input does not expose multi-select**
- **File:** `frontend/src/components/FileUpload.tsx`
- **Evidence:** file input at line ~37 (no `multiple`)
- **Why this matters:** UX mismatch: drop supports many files; picker appears single-file.
- **Recommended fix:** add `multiple` prop (default `true`) and pass to input.
9) **Import/delete actions in Subsonic panel swallow API errors**
- **File:** `frontend/src/components/Sidebar.tsx`
- **Evidence:** `doDeleteImport` and some mutation handlers
- **Why this matters:** silent failure makes troubleshooting difficult.
- **Recommended fix:** show lightweight inline error state/toast on failures.
---
## Component-by-component status snapshot
- **AutoPipelineDialog.tsx** — solid orchestration flow; minor abort-listener hygiene issue.
- **DynamicMixPlayer.tsx** — good UX, but needs lifecycle/resource hardening.
- **FileUpload.tsx** — clean/small; minor UX enhancement (`multiple`).
- **GlobalAudioBar.tsx** — no blocking issues found.
- **JobStatus.tsx** — polling logic acceptable; no blocking issues found.
- **KaraokeEditor.tsx** — strong feature set; needs timing-order clamp + fetch cancellation.
- **ResultFiles.tsx** — no blocking issues found.
- **SettingsPanel.tsx** — one styling token fix.
- **Sidebar.tsx** — robust overall; deletion/error-flow handling needs tightening.
---
## Proposed fix plan
### Phase 1 — Stability hotfixes (same day)
1. `DynamicMixPlayer`: add full stem/audio graph disposal + stem-set reconciliation.
2. `Sidebar` project delete flow: only update UI on confirmed success; display failure feedback.
3. `KaraokeEditor`: enforce previous-word lower-bound clamp for all word timing edits.
### Phase 2 — Reliability/cleanup (next pass)
4. `AutoPipelineDialog`: cleanup abort listener usage (`once` or removeEventListener).
5. `KaraokeEditor`: add abortable waveform fetch/decode pipeline.
6. `Sidebar` import/delete operations: standardized mutation error feedback.
### Phase 3 — UX polish
7. `SettingsPanel`: replace invalid Tailwind class.
8. `FileUpload`: expose configurable multi-select support.
---
## Validation plan after fixes
- Run: `npm run typecheck`
- Run focused manual checks:
- DynamicMixPlayer: mount/unmount cycles, stem swaps, no lingering audio playback.
- Sidebar: failed delete path leaves UI unchanged and surfaces error.
- KaraokeEditor: cannot set word start before previous word.
- SettingsPanel hover styles in dark mode.
- FileUpload click-select supports intended file count behavior.
+36
View File
@@ -0,0 +1,36 @@
# 2026-03-17 — GPU limiter integration for MIDI transcription
## Objective
Ensure MIDI transcription engines are governed by the same GPU safety policy used by separation/lyric flows, while preserving existing queue behavior and compatibility.
## Recommendation
1. Keep queue-level serialization as-is (`jm.submit_gpu`) for MIDI jobs.
2. Explicitly apply backend GPU safety limits before `basic-pitch` inference.
3. Apply a TensorFlow GPU memory cap in `omnizart-sidecar` using the same reserve-MB policy.
4. Expose one optional sidecar override variable, defaulting to the global reserve.
5. Document the behavior in project docs.
## Implementation plan (completed)
- [x] Wire `basic-pitch` path to existing `_apply_cuda_limits()` before model load.
- [x] Add TensorFlow memory-limit setup in `omnizart-sidecar/app.py`.
- [x] Add `OMNIZART_GPU_VRAM_RESERVE_MB` env in compose files with fallback to `STEMENGINE_GPU_VRAM_RESERVE_MB`.
- [x] Update README configuration/GPU notes.
## Files changed
- `backend/services/transcriber_service.py`
- `omnizart-sidecar/app.py`
- `stemengine.yml` (workspace-level)
- `appdata/stemengine/stemengine.yml` (repo-level)
- `README.md`
## Behavioral outcome
- `basic-pitch` now executes after backend GPU limit setup.
- `omnizart-sidecar` now configures TensorFlow GPU usage to preserve VRAM headroom:
- hard cap when total VRAM is detectable via `nvidia-smi`
- memory-growth fallback when total VRAM is not detectable
- Existing single-worker GPU queue semantics remain unchanged.
## Notes
- TensorFlow logical-device configuration must occur before TF runtime initialization.
- Sidecar applies limiter at process startup (module import path) before loading Omnizart.
- Set reserve to `0` to disable cap behavior.
+98
View File
@@ -0,0 +1,98 @@
# LyricsLivePreview Review
Date: 2026-03-17
Target: `frontend/src/components/LyricsLivePreview.tsx`
Supporting context: `frontend/src/components/2020310_LyricsLivePreview_Claude.tsx`
## Summary
The file has no current TypeScript diagnostics, but it contains a few real behavioral and maintainability issues that are worth addressing before further feature work.
## Findings
### 1. Reset style drops caller-provided defaults
Severity: High
`style` is initialized by merging `DEFAULT_STYLE` with incoming `styleOptions`, but `resetStyle()` restores only `DEFAULT_STYLE`.
Impact:
- project- or page-specific default styling is lost after a reset
- reset behavior is inconsistent with initial load behavior
Relevant area:
- `frontend/src/components/LyricsLivePreview.tsx`
### 2. Speaker settings can bleed across `storageKey` changes
Severity: High
`speakerColors` and `speakerNames` are initialized from `storageKey` only once. If the component later receives a different `storageKey`, the old in-memory speaker state remains active and is then persisted under the new key.
Impact:
- speaker names/colors from one song/project can appear on another
- the new key can be overwritten before its stored values are loaded
Relevant area:
- `frontend/src/components/LyricsLivePreview.tsx`
### 3. RAF loop runs continuously while idle
Severity: Medium
The animation loop keeps scheduling `requestAnimationFrame()` even when playback is stopped and there is no active audio progression.
Impact:
- unnecessary CPU usage for a static preview
- avoidable background work while component is mounted
Relevant area:
- `frontend/src/components/LyricsLivePreview.tsx`
### 4. Multi-stem audio setup lacks teardown
Severity: Medium
The multi-stem initialization effect creates `Audio` elements, event listeners, and Web Audio graph nodes, but does not clean them up when stems change or the component unmounts.
Impact:
- stale audio nodes can accumulate
- event listeners remain attached longer than needed
- future stem changes are riskier than they need to be
Relevant area:
- `frontend/src/components/LyricsLivePreview.tsx`
### 5. Recenter path appears dead in the canvas-based version
Severity: Medium
The current production component renders `LyricsCanvas`, while legacy `scrollDivRef` / `activeLineRef`-based recenter logic remains in the file. Those refs no longer appear to be wired into the active render path.
Impact:
- recenter button likely no-ops
- dead code increases confusion during future changes
Relevant area:
- `frontend/src/components/LyricsLivePreview.tsx`
## Validation performed
- File diagnostics check: no errors reported
- Frontend typecheck: passed via `npm run typecheck`
## Recommended next steps
1. Fix `resetStyle()` to restore merged defaults rather than raw `DEFAULT_STYLE`
2. Rehydrate speaker state when `storageKey` changes
3. Stop or gate the RAF loop while idle
4. Add cleanup for multi-stem audio resources
5. Remove or replace dead recenter logic if `LyricsCanvas` fully owns layout now
## Follow-up implemented
Implemented after this review was written:
- `resetStyle()` now restores merged defaults derived from incoming `styleOptions`
- speaker preferences are reloaded when `storageKey` changes, with a hydration guard to avoid stale writes
- the RAF loop now stops while playback is paused instead of running continuously while idle
Still remaining from the review:
- add teardown for multi-stem audio resources
- remove or replace dead recenter logic if the canvas path fully owns layout
@@ -0,0 +1,152 @@
# 2026-03-17 Review: `lrc_service.py` and `lyric_transcriber_service.py`
## Scope
- `backend/services/lrc_service.py`
- `backend/services/lyric_transcriber_service.py`
## Executive summary
- **High-risk issues:** 3
- **Medium-risk issues:** 3
- **Low-risk issues:** 2
The most urgent problem is the timeout behavior in the Whisper fallback path: the worker thread may continue running after timeout while GPU lock/state is released, enabling overlapping GPU-heavy jobs.
---
## Findings
## High
### 1) Timeout path leaves transcription thread running while shared GPU lock is released
**File:** `backend/services/lyric_transcriber_service.py`
**Evidence:** around thread creation/join timeout (`_t.join(timeout=...)`, `_t.is_alive()`) near lines 1987-1999, with lock release in `finally` near lines 2108-2113.
**Why this is risky:**
- On timeout, function returns error but daemon thread continues using Whisper/GPU in background.
- `finally` still runs and releases `jm.gpu_vram_lock`, so another job can start and contend for VRAM with the still-running timed-out thread.
- Inconsistent state: job marked failed, but expensive compute may still be active.
**Recommended fix:**
- Replace ad-hoc thread timeout with process-level timeout/cancellation boundary, or keep lock held until worker exits.
- If thread timeout is retained, do **not** release shared GPU lock until worker confirms exit.
- Add a watchdog flag and explicit “detached background cleanup in progress” state if unavoidable.
---
### 2) Global monkey patch overrides all `torch.load()` calls to `weights_only=False`
**File:** `backend/services/lyric_transcriber_service.py`
**Evidence:** startup patch around lines 30-49.
**Why this is risky:**
- Affects global process behavior, not just WhisperX code paths.
- Disables safer default for all downstream libraries in this process.
- Makes future security posture and debugging harder.
**Recommended fix:**
- Scope the compatibility workaround to specific loader call sites instead of monkey-patching globally.
- If patch must remain, gate behind explicit env flag and emit startup warning.
---
### 3) Secondary diarization entrypoint bypasses GPU coordination/cleanup standards
**File:** `backend/services/lyric_transcriber_service.py`
**Evidence:** `run_lyric_transcription_with_diarization()` near lines 2115+.
**Why this is risky:**
- Does not use the shared `jm.gpu_vram_lock` path used in `run_lyric_transcription()`.
- Has no explicit `_clear_cuda_cache()`/resource-finalization symmetry.
- Can diverge behaviorally and regress memory stability under load.
**Recommended fix:**
- Route diarization workflow through the primary orchestration path or extract shared resource-guard wrappers.
- Ensure lock acquisition/release + cache cleanup is consistent in all GPU-heavy entrypoints.
---
## Medium
### 4) HTML preview duration logic reads wrong data attribute
**File:** `backend/services/lrc_service.py`
**Evidence:** elements are rendered with `data-time` (lines ~550, ~562), but duration scanner queries `[data-start]` (line ~701).
**Impact:**
- `allTimestamps` resolves empty.
- Preview duration falls back to hardcoded default (300s), causing inaccurate seek/play behavior.
**Recommended fix:**
- Query `data-time` consistently, or write explicit numeric seconds attributes for robust parsing.
---
### 5) Styled HTML generation inserts unescaped dynamic content
**File:** `backend/services/lrc_service.py`
**Evidence:** string interpolation of `title`, `lyrics_html`, and style values into full HTML template.
**Impact:**
- If source text/style contains HTML/JS payloads, generated preview can execute script when opened.
- Risk profile depends on trust model, but this is still unsafe-by-default.
**Recommended fix:**
- Escape HTML entities for text payloads.
- Strictly validate/allowlist style values (font, colors, alignment) before interpolation.
- Prefer templating with escaping helpers.
---
### 6) Broad exception swallowing in temp-file cleanup hides real IO issues
**File:** `backend/services/lrc_service.py`
**Evidence:** bare `except: pass` cleanup blocks around lines ~333-334 and ~420-421.
**Impact:**
- Silent failures for filesystem/permission issues.
- Harder incident diagnosis when temp files accumulate.
**Recommended fix:**
- Catch `OSError` explicitly and log at debug/warn with path context.
---
## Low
### 7) `run_lyric_transcription_with_diarization()` appears functionally redundant
**File:** `backend/services/lyric_transcriber_service.py`
**Impact:**
- Duplicate orchestration path increases maintenance burden and drift risk.
**Recommended fix:**
- Deprecate legacy path and consolidate on `run_lyric_transcription()`.
### 8) `_ensure_vocals()` extraction is not cancellation-aware
**File:** `backend/services/lyric_transcriber_service.py`
**Impact:**
- Long Demucs extraction can continue despite job cancellation intent.
**Recommended fix:**
- Poll cancellation before/after extraction and abort subprocess when cancellation is detected.
---
## Prioritized fix plan
## Phase 1 (Immediate)
1. Fix timeout/lock race in Whisper fallback thread path.
2. Add guardrails around global `torch.load` patch (feature flag + warnings) or remove global patch.
3. Make secondary diarization path use shared GPU lock + cleanup (or route through primary flow).
## Phase 2
4. Fix HTML `data-time`/duration attribute mismatch.
5. Replace bare cleanup `except` blocks with typed exception handling + logs.
## Phase 3
6. Harden HTML preview generation with escaping/allowlisting.
7. Consolidate/deprecate redundant diarization entrypoint.
8. Add cancellation-aware subprocess handling in `_ensure_vocals()`.
---
## Suggested validation checklist after fixes
- Timeout test: force Whisper timeout and verify no lingering worker consumes GPU after job failure.
- Concurrency test: start separation + lyric job; confirm lock behavior prevents overlap.
- Preview test: generated HTML duration matches last lyric timestamp.
- Security test: inject HTML-like lyrics/title and confirm preview displays escaped text.
- Cleanup test: simulate temp-file deletion failure and verify diagnostic log appears.
@@ -0,0 +1,51 @@
# 2026-03-17 Remediation Implemented: `lrc_service.py` + `lyric_transcriber_service.py`
## Scope
- `backend/services/lrc_service.py`
- `backend/services/lyric_transcriber_service.py`
## Implemented changes
### 1) `lrc_service.py`
- Fixed HTML preview duration parsing bug:
- JS now derives duration from `.lyric-line` and `.word` elements using `data-time`.
- Removed incorrect `[data-start]` query path.
- Hardened HTML generation:
- Escaped dynamic lyric/title text via `html.escape()` before interpolation.
- Added style sanitizers for:
- colors (`hex/rgb/rgba/hsl/hsla` pattern check),
- numeric ranges (`fontSize`, `lineSpacing`),
- text alignment (`left|center|right`),
- font family conservative allow-list.
- Improved temp-file cleanup diagnostics:
- Replaced bare `except: pass` cleanup blocks with `except OSError as exc` and debug logging.
### 2) `lyric_transcriber_service.py`
- Added guardrail for global `torch.load` compatibility patch:
- Patch is now controlled by env var `STEMENGINE_PATCH_TORCH_LOAD` (default enabled).
- Emits startup warning when active.
- Removed timeout-thread race in Whisper path:
- Replaced detached timeout thread pattern with inline `transcribe_lyrics(...)` execution under shared GPU lock.
- Eliminates case where timed-out background thread keeps GPU active after lock release.
- Aligned legacy diarization entrypoint with GPU lifecycle safeguards:
- `run_lyric_transcription_with_diarization()` now acquires/releases `jm.gpu_vram_lock`.
- Applies CUDA limits/cache cleanup symmetry (`_apply_cuda_limits()`, `_clear_cuda_cache()`).
## Validation
- Syntax validation executed:
- `python3 -m py_compile backend/services/lrc_service.py backend/services/lyric_transcriber_service.py`
- Result: success (no syntax errors).
## Remaining follow-up items
- Optional hardening:
- Make `_ensure_vocals()` subprocess fully cancellation-aware during long Demucs runs.
- Consider replacing global `torch.load` patch with narrower call-site overrides once upstream deps are stable.
- Consolidate/deprecate legacy `run_lyric_transcription_with_diarization()` if primary path covers all needs.
+6 -1
View File
@@ -8,6 +8,7 @@
- **Stem Separation** — 40+ models across 10 model families
- **MIDI Transcription** — basic-pitch with polyphonic and melody modes
- **GPU Safety Limits** — shared VRAM reserve for separation, lyric transcription, basic-pitch, and Omnizart sidecar
- **ABC Notation** — convert MIDI to ABC notation for sheet music
- **Lyrics Sync** — forced-alignment to produce ELRC files with word-level timestamps and a live animated preview
- **Project Management** — persistent project tabs, per-stem reprocessing, stem renaming
@@ -79,6 +80,8 @@ All configuration is via environment variables. The `docker-compose.yml` accepts
|---|---|---|
| `CPU_SEPARATION` | `false` | Set to `true` to force CPU inference |
| `MAX_UPLOAD_MB` | `500` | Maximum upload file size in MB |
| `STEMENGINE_GPU_VRAM_RESERVE_MB` | `2048` | Keeps this many MB free on the GPU (WSL2-safe headroom) |
| `OMNIZART_GPU_VRAM_RESERVE_MB` | `STEMENGINE_GPU_VRAM_RESERVE_MB` | Optional sidecar override for Omnizart TensorFlow GPU cap |
| `TZ` | `UTC` | Timezone |
| `NAVSERVER` | — | Subsonic server URL (can also be set in the UI) |
| `NAVUSER` | — | Subsonic username |
@@ -166,4 +169,6 @@ docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
PyTorch wheels are self-contained (cu124) and do not require a matching CUDA toolkit on the host — only the driver.
`onnxruntime-gpu` is used for MDX-Net and basic-pitch ONNX inference; it auto-selects `CUDAExecutionProvider` when a GPU is available.
`STEMENGINE_GPU_VRAM_RESERVE_MB` is applied in backend GPU code paths (separation, lyric transcription, and basic-pitch startup) to preserve display headroom on WSL2.
Omnizart sidecar now applies a TensorFlow memory cap using `OMNIZART_GPU_VRAM_RESERVE_MB` (or `STEMENGINE_GPU_VRAM_RESERVE_MB` fallback), so sidecar transcription follows the same reserve policy.
+94 -44
View File
@@ -3,6 +3,7 @@ LRC alignment service for creating enhanced ELRC files with word-level timestamp
Uses the lyrics-sync module for vocal extraction and forced alignment.
"""
import logging
import html
import os
import re
import sys
@@ -215,30 +216,44 @@ def run_lrc_alignment(
# Strip section markers ([Chorus], [Verse 1], etc.) and blank lines
if not lyrics_synced:
lyrics_text = clean_lyrics_text(lyrics_text)
# Choose sync class: skip Demucs when audio is already vocals.
# In eco mode, force CPU for the phoneme recogniser to avoid GPU contention
# that causes display glitches on shared-GPU (WSL2) setups.
# In eco mode, force CPU for the phoneme recogniser to avoid GPU
# contention that causes display glitches on shared-GPU (WSL2) setups.
_align_device = "cpu" if lrc_power_preset == "eco" else None
SyncClass = _LyricsSyncNoExtract if is_vocals else LyricsSync
# If lyrics are already synced (LRC format), convert to ELRC with word-level timestamps
if lyrics_synced:
jm.update_job(job_id, progress="Lyrics already synced. Enhancing to word-level...")
_enhance_existing_lrc(
job_id, audio_path, lyrics_text, out_path,
output_filename, SyncClass, style_options, timing_precision,
align_device=_align_device,
)
else:
# Run full alignment pipeline
jm.update_job(job_id, progress="Starting lyrics alignment...")
_run_full_alignment(
job_id, audio_path, lyrics_text, out_path,
output_filename, SyncClass, style_options, timing_precision,
align_device=_align_device,
)
# Flush VRAM and apply the per-process cap before loading any model.
# Acquire the shared GPU lock so this job cannot overlap with a
# simultaneous WhisperX/parakeet transcription job in the other executor.
from .separation.execution import flush_vram_cache
from .lyric_transcriber_service import _apply_cuda_limits
flush_vram_cache()
_apply_cuda_limits()
jm.gpu_vram_lock.acquire()
try:
# If lyrics are already synced (LRC format), convert to ELRC with word-level timestamps
if lyrics_synced:
jm.update_job(job_id, progress="Lyrics already synced. Enhancing to word-level...")
_enhance_existing_lrc(
job_id, audio_path, lyrics_text, out_path,
output_filename, SyncClass, style_options, timing_precision,
align_device=_align_device,
)
else:
# Run full alignment pipeline
jm.update_job(job_id, progress="Starting lyrics alignment...")
_run_full_alignment(
job_id, audio_path, lyrics_text, out_path,
output_filename, SyncClass, style_options, timing_precision,
align_device=_align_device,
)
finally:
flush_vram_cache()
try:
jm.gpu_vram_lock.release()
except RuntimeError:
pass # already released (should not happen)
except Exception as exc:
log.debug("[%s] lrc_alignment failed — %s: %s", job_id, type(exc).__name__, exc)
jm.update_job(job_id, status=jm.JobStatus.ERROR, error=str(exc), progress="Error.")
@@ -330,8 +345,8 @@ def _run_full_alignment(
# Cleanup temp file
try:
os.unlink(lyrics_temp_path)
except:
pass
except OSError as exc:
log.debug("[%s] temp-file cleanup failed (%s): %s", job_id, lyrics_temp_path, exc)
def _enhance_existing_lrc(
@@ -417,8 +432,8 @@ def _enhance_existing_lrc(
finally:
try:
os.unlink(lyrics_temp_path)
except:
pass
except OSError as exc:
log.debug("[%s] temp-file cleanup failed (%s): %s", job_id, lyrics_temp_path, exc)
def _fmt_ts(seconds: float) -> str:
@@ -522,15 +537,44 @@ def _generate_styled_html(elrc_content: str, style_options: dict, title: str) ->
"""
from .metadata_util import parse_elrc
# Extract style options with defaults
font_family = style_options.get('fontFamily', 'system-ui')
font_size = style_options.get('fontSize', 24)
text_color = style_options.get('textColor', '#e2e8f0')
highlight_color = style_options.get('highlightColor', '#3b82f6')
background_color = style_options.get('backgroundColor', '#0f172a')
word_highlight_color = style_options.get('wordHighlightColor', '#60a5fa')
line_spacing = style_options.get('lineSpacing', 1.5)
text_align = style_options.get('textAlign', 'center')
def _safe_float(value, default: float, min_v: float, max_v: float) -> float:
try:
parsed = float(value)
except (TypeError, ValueError):
return default
return max(min_v, min(max_v, parsed))
def _safe_text_align(value: str) -> str:
return value if value in {'left', 'center', 'right'} else 'center'
def _safe_color(value: str, default: str) -> str:
if not isinstance(value, str):
return default
candidate = value.strip()
color_re = re.compile(
r'^(#[0-9a-fA-F]{3,8}|rgb\([^\)]{1,64}\)|rgba\([^\)]{1,64}\)|'
r'hsl\([^\)]{1,64}\)|hsla\([^\)]{1,64}\))$'
)
return candidate if color_re.match(candidate) else default
def _safe_font_family(value: str) -> str:
if not isinstance(value, str):
return 'system-ui, sans-serif'
candidate = value.strip()
# Conservative allow-list for CSS font-family token(s).
if not re.fullmatch(r'[A-Za-z0-9\s,\-"\']{1,120}', candidate):
return 'system-ui, sans-serif'
return candidate
# Extract and sanitize style options with defaults
font_family = _safe_font_family(style_options.get('fontFamily', 'system-ui, sans-serif'))
font_size = _safe_float(style_options.get('fontSize', 24), 24.0, 10.0, 96.0)
text_color = _safe_color(style_options.get('textColor', '#e2e8f0'), '#e2e8f0')
highlight_color = _safe_color(style_options.get('highlightColor', '#3b82f6'), '#3b82f6')
background_color = _safe_color(style_options.get('backgroundColor', '#0f172a'), '#0f172a')
word_highlight_color = _safe_color(style_options.get('wordHighlightColor', '#60a5fa'), '#60a5fa')
line_spacing = _safe_float(style_options.get('lineSpacing', 1.5), 1.5, 1.0, 3.5)
text_align = _safe_text_align(style_options.get('textAlign', 'center'))
# Parse ELRC content
lines = parse_elrc(elrc_content)
@@ -547,11 +591,11 @@ def _generate_styled_html(elrc_content: str, style_options: dict, title: str) ->
millis = int((word_ts % 1) * 1000)
time_attr = f"{minutes:02d}:{seconds:02d}.{millis:03d}"
word_spans.append(
f'<span class="word" data-time="{time_attr}">{word}</span>'
f'<span class="word" data-time="{time_attr}">{html.escape(word)}</span>'
)
line_html = ' '.join(word_spans)
else:
line_html = line_text
line_html = html.escape(line_text)
minutes = int(line_ts // 60)
seconds = int(line_ts % 60)
@@ -564,12 +608,14 @@ def _generate_styled_html(elrc_content: str, style_options: dict, title: str) ->
lyrics_html = '\n'.join(html_lines)
safe_title = html.escape(title)
html_template = f'''<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>{title} - Lyrics</title>
<title>{safe_title} - Lyrics</title>
<style>
* {{
margin: 0;
@@ -676,7 +722,7 @@ def _generate_styled_html(elrc_content: str, style_options: dict, title: str) ->
</head>
<body>
<div class="container">
<h1>{title}</h1>
<h1>{safe_title}</h1>
<div class="lyrics-container">
{lyrics_html}
</div>
@@ -697,18 +743,22 @@ def _generate_styled_html(elrc_content: str, style_options: dict, title: str) ->
let isPlaying = false;
let currentTime = 0;
// Derive duration from the last timestamp in the ELRC content
const allTimestamps = Array.from(document.querySelectorAll('[data-start]'))
.map(el => parseFloat(el.getAttribute('data-start')))
.filter(t => !isNaN(t));
let duration = allTimestamps.length > 0 ? Math.ceil(Math.max(...allTimestamps) + 10) : 300;
let animationId = null;
// Parse time string to seconds
function parseTime(timeStr) {{
const [mins, secs] = timeStr.split(':');
return parseInt(mins) * 60 + parseFloat(secs);
}}
// Derive duration from the last timestamp in the ELRC content
const allTimestamps = [
...Array.from(document.querySelectorAll('.lyric-line')),
...Array.from(document.querySelectorAll('.word')),
]
.map(el => parseTime(el.dataset.time || '00:00.000'))
.filter(t => !isNaN(t));
let duration = allTimestamps.length > 0 ? Math.ceil(Math.max(...allTimestamps) + 10) : 300;
let animationId = null;
// Format seconds to time string
function formatTime(seconds) {{
+99 -69
View File
@@ -7,7 +7,6 @@ This service transcribes vocals to text with precise timing information,
suitable for generating LRC and ELRC files.
"""
import contextlib
import threading
import io
import json
import os
@@ -34,20 +33,25 @@ warnings.filterwarnings(
# not set by caller → False) by monkey-patching torch.load at startup.
# All checkpoint sources used here (HuggingFace, faster-whisper, pyannote)
# are trusted, so the security trade-off is acceptable.
try:
import torch as _torch
_orig_torch_load = _torch.load
def _patched_torch_load(*args, **kwargs):
# Force weights_only=False regardless of what the caller passed.
# transformers explicitly passes weights_only=True (not just as a
# default), so setdefault is insufficient — we must overwrite it.
# All checkpoint sources used here (HuggingFace, faster-whisper,
# pyannote) are trusted, so the security trade-off is acceptable.
kwargs['weights_only'] = False
return _orig_torch_load(*args, **kwargs)
_torch.load = _patched_torch_load
except Exception:
pass
_TORCH_LOAD_PATCH_ACTIVE = False
if os.environ.get("STEMENGINE_PATCH_TORCH_LOAD", "1") != "0":
try:
import torch as _torch
_orig_torch_load = _torch.load
def _patched_torch_load(*args, **kwargs):
# Force weights_only=False regardless of what the caller passed.
# transformers explicitly passes weights_only=True (not just as a
# default), so setdefault is insufficient — we must overwrite it.
# All checkpoint sources used here (HuggingFace, faster-whisper,
# pyannote) are trusted, so the security trade-off is acceptable.
kwargs['weights_only'] = False
return _orig_torch_load(*args, **kwargs)
_torch.load = _patched_torch_load
_TORCH_LOAD_PATCH_ACTIVE = True
except Exception:
pass
import logging
import requests
@@ -66,6 +70,12 @@ from ..config import (
log = logging.getLogger(__name__)
if _TORCH_LOAD_PATCH_ACTIVE:
log.warning(
"torch.load compatibility patch enabled (weights_only=False override). "
"Set STEMENGINE_PATCH_TORCH_LOAD=0 to disable once dependencies support PyTorch >=2.6 defaults."
)
@dataclass
class WordTimestamp:
@@ -1142,11 +1152,24 @@ def _uncensor_segments(segments: list) -> None:
w["word"] = txt
# Fraction of total GPU VRAM this process is allowed to use.
# Keeping this below 1.0 on WSL2 reserves headroom for the Windows display
# compositor (DWM), preventing TDR timeouts that cause screen flicker.
# Override with the STEMENGINE_GPU_MEMORY_FRACTION env var (0.0–1.0).
_GPU_MEMORY_FRACTION: float = float(os.environ.get("STEMENGINE_GPU_MEMORY_FRACTION", "0.80"))
# MB of VRAM to keep free for Windows (DWM, WDDM, etc.) on WSL2.
# Guarantees headroom for Windows display processes regardless of GPU size:
# 8 GB card: (8192 - 2048) / 8192 ≈ 75 %
# 12 GB card: (12288 - 2048) / 12288 ≈ 83 %
# 24 GB card: (24576 - 2048) / 24576 ≈ 92 %
# Override with STEMENGINE_GPU_VRAM_RESERVE_MB (default 2048 = 2 GB).
# Set to 0 to disable the cap entirely.
# Legacy: STEMENGINE_GPU_MEMORY_FRACTION takes precedence if set explicitly.
_GPU_VRAM_RESERVE_MB: int = int(os.environ.get("STEMENGINE_GPU_VRAM_RESERVE_MB", "2048"))
_GPU_MEMORY_FRACTION: float | None = None
_frac_env = os.environ.get("STEMENGINE_GPU_MEMORY_FRACTION")
if _frac_env is not None:
try:
_v = float(_frac_env)
if 0.0 < _v < 1.0:
_GPU_MEMORY_FRACTION = _v
except ValueError:
pass
def _apply_cuda_limits() -> None:
@@ -1155,7 +1178,17 @@ def _apply_cuda_limits() -> None:
Safe to call multiple times — torch guards against re-setting the fraction
on an already-initialised context, and os.nice is a no-op when already niced.
GPU power limit is set once at container startup via start.sh, not here.
VRAM cap is computed dynamically:
fraction = (total_vram - STEMENGINE_GPU_VRAM_RESERVE_MB) / total_vram
This guarantees the Windows display subsystem (DWM/WDDM) always has at
least STEMENGINE_GPU_VRAM_RESERVE_MB (default 2 GB) free on WSL2.
Override with STEMENGINE_GPU_MEMORY_FRACTION (0.0–1.0) for a fixed
fraction instead (legacy; STEMENGINE_GPU_VRAM_RESERVE_MB is preferred).
NOTE: nvidia-smi -pl (watt-based power cap) requires SYS_ADMIN which is
unavailable in WSL2 containers. VRAM-fraction capping is the correct
WSL2-compatible mechanism for protecting Windows display processes.
"""
# Lower CPU scheduling priority so Windows UI processes aren't starved.
try:
@@ -1163,16 +1196,26 @@ def _apply_cuda_limits() -> None:
_os.nice(10)
except Exception:
pass
# Reserve a fraction of VRAM for the Windows display driver / other processes.
# Reserve VRAM headroom for the Windows display driver / other processes.
try:
import torch as _t
if _t.cuda.is_available() and 0.0 < _GPU_MEMORY_FRACTION < 1.0:
_t.cuda.set_per_process_memory_fraction(_GPU_MEMORY_FRACTION)
# Allow TF32 for matmuls — imperceptible quality difference for audio
# separation but reduces compute intensity and power draw on Ampere+ GPUs.
_t.backends.cuda.matmul.allow_tf32 = True
_t.backends.cudnn.allow_tf32 = True
_t.set_float32_matmul_precision('high')
if _t.cuda.is_available():
if _GPU_MEMORY_FRACTION is not None:
# Legacy: explicit fixed-fraction override takes precedence.
frac = _GPU_MEMORY_FRACTION
elif _GPU_VRAM_RESERVE_MB > 0:
total_bytes = _t.cuda.get_device_properties(0).total_memory
reserve_bytes = _GPU_VRAM_RESERVE_MB * 1_048_576
frac = max(0.01, min(0.99, (total_bytes - reserve_bytes) / total_bytes))
else:
frac = 0.0 # cap disabled
if 0.0 < frac < 1.0:
_t.cuda.set_per_process_memory_fraction(frac)
# Allow TF32 for matmuls — imperceptible quality difference for audio
# separation but reduces compute intensity and power draw on Ampere+ GPUs.
_t.backends.cuda.matmul.allow_tf32 = True
_t.backends.cudnn.allow_tf32 = True
_t.set_float32_matmul_precision('high')
except Exception:
pass
@@ -1963,46 +2006,20 @@ def run_lyric_transcription(
_log.pop(0)
jm.update_job(job_id, progress='\n'.join(_log))
_TRANSCRIBE_TIMEOUT = 7200 # 2-hour hard cap
_result_box: list = [None]
_exc_box: list = [None]
def _do_transcribe():
try:
_result_box[0] = transcribe_lyrics(
audio_path=audio_path,
model_name=model,
language=language,
task=task, # type: ignore
include_word_timestamps=word_timestamps,
initial_prompt=initial_prompt,
temperature=temperature,
condition_on_previous_text=condition_on_previous_text,
beam_size=beam_size,
line_callback=_on_line,
)
except Exception as _e:
_exc_box[0] = _e
_t = threading.Thread(target=_do_transcribe, daemon=True)
_t.start()
_t.join(timeout=_TRANSCRIBE_TIMEOUT)
if _t.is_alive():
jm.update_job(
job_id,
status=jm.JobStatus.ERROR,
error=f"Transcription timed out after {_TRANSCRIBE_TIMEOUT // 3600}h. "
"The model is still running in the background and will release "
"the GPU when it finishes.",
progress="Timed out.",
)
return
if _exc_box[0] is not None:
raise _exc_box[0]
result = _result_box[0]
# Run transcription inline while holding the shared GPU lock so we
# never detach background GPU work that can race with subsequent jobs.
result = transcribe_lyrics(
audio_path=audio_path,
model_name=model,
language=language,
task=task, # type: ignore
include_word_timestamps=word_timestamps,
initial_prompt=initial_prompt,
temperature=temperature,
condition_on_previous_text=condition_on_previous_text,
beam_size=beam_size,
line_callback=_on_line,
)
if jm.should_cancel(job_id):
jm.update_job(job_id, status=jm.JobStatus.CANCELLED, progress="Cancelled.")
@@ -2149,6 +2166,10 @@ def run_lyric_transcription_with_diarization(
else:
out_path = LRC_SYNC_DIR / job_id
out_path.mkdir(parents=True, exist_ok=True)
lock_acquired = False
jm.gpu_vram_lock.acquire()
lock_acquired = True
try:
import whisperx
@@ -2162,6 +2183,8 @@ def run_lyric_transcription_with_diarization(
# Load WhisperX model
device = "cuda" if jm.import_module("torch").cuda.is_available() else "cpu"
compute_type = "float16" if device == "cuda" else "int8"
_apply_cuda_limits()
_clear_cuda_cache()
model_obj = whisperx.load_model(model, device, compute_type=compute_type)
@@ -2236,6 +2259,13 @@ def run_lyric_transcription_with_diarization(
)
except Exception as exc:
jm.update_job(job_id, status=jm.JobStatus.ERROR, error=str(exc), progress="Error.")
finally:
_clear_cuda_cache()
if lock_acquired:
try:
jm.gpu_vram_lock.release()
except RuntimeError:
pass
def is_whisper_available() -> bool:
+3 -3
View File
@@ -359,9 +359,9 @@ def run_separation(
_power = get_power_opts(sep_power_preset)
# Flush PyTorch's allocator cache BEFORE applying the VRAM cap.
# On WSL2 the system reserves ~2 GB for Windows (DWM, etc.), so the effective
# usable VRAM is total - 2 GB. _apply_cuda_limits() caps the process at 80% of
# the GPU's *reported* total, which leaves headroom for Windows.
# On WSL2 the system reserves headroom for Windows (DWM, WDDM, etc.).
# _apply_cuda_limits() caps the process at (total_vram - STEMENGINE_GPU_VRAM_RESERVE_MB)
# / total_vram — e.g. ≈75 % on an 8 GB card, ensuring 2 GB stays free for Windows.
# The flush must come first: if stale cached allocations are still counted against
# the cap from a prior job, the cap appears tighter than it actually is.
flush_vram_cache()
+9
View File
@@ -43,6 +43,15 @@ def run_transcription(job_id: str, input_path: str, mode: str) -> None:
# ---------------------------------------------------------------------------
def _run_basic_pitch(job_id: str, input_path: str, mode: str) -> None:
# Reuse global GPU safety limits (VRAM reserve + TF32 settings) before
# loading any transcription backend for basic-pitch.
try:
from .lyric_transcriber_service import _apply_cuda_limits
_apply_cuda_limits()
except Exception:
# Keep transcription functional even if limiter setup fails.
pass
from basic_pitch.inference import predict
from basic_pitch import ICASSP_2022_MODEL_PATH
File diff suppressed because it is too large. Load diff
+19 -8
View File
@@ -51,6 +51,23 @@ interface StepStatus {
const MAX_RETRIES = 2
function sleepWithAbort(ms: number, signal?: AbortSignal): Promise<void> {
return new Promise<void>((res, rej) => {
const t = setTimeout(() => {
if (signal) signal.removeEventListener('abort', onAbort)
res()
}, ms)
const onAbort = () => {
clearTimeout(t)
if (signal) signal.removeEventListener('abort', onAbort)
rej(new Error('Cancelled'))
}
if (signal) signal.addEventListener('abort', onAbort, { once: true })
})
}
async function withRetry<T>(
fn: () => Promise<T>,
signal: AbortSignal | undefined,
@@ -64,10 +81,7 @@ async function withRetry<T>(
if (signal?.aborted) throw new Error('Cancelled')
if (attempt >= MAX_RETRIES) throw e
onRetry(attempt + 1)
await new Promise<void>((res, rej) => {
const t = setTimeout(res, 3000)
signal?.addEventListener('abort', () => { clearTimeout(t); rej(new Error('Cancelled')) })
})
await sleepWithAbort(3000, signal)
}
}
}
@@ -79,10 +93,7 @@ async function pollUntilDone(jobId: string, signal?: AbortSignal): Promise<Job>
if (job.status === 'done') return job
if (job.status === 'error') throw new Error(job.error || 'Job failed')
if (job.status === 'cancelled') throw new Error('Job was cancelled')
await new Promise<void>((res, rej) => {
const t = setTimeout(res, 2500)
signal?.addEventListener('abort', () => { clearTimeout(t); rej(new Error('Cancelled')) })
})
await sleepWithAbort(2500, signal)
}
}
+85 -8
View File
@@ -17,6 +17,9 @@ interface Props {
export default function DynamicMixPlayer({ stems, title, onLevelChange }: Props) {
const audioRefs = useRef<Record<string, HTMLAudioElement>>({})
const gainRefs = useRef<Record<string, GainNode>>({})
const sourceRefs = useRef<Record<string, MediaElementAudioSourceNode>>({})
const loadedMetadataListenerRefs = useRef<Record<string, () => void>>({})
const errorListenerRefs = useRef<Record<string, () => void>>({})
const audioCtxRef = useRef<AudioContext | null>(null)
const [isPlaying, setIsPlaying] = useState(false)
@@ -36,6 +39,29 @@ export default function DynamicMixPlayer({ stems, title, onLevelChange }: Props)
// Whether this player is "owning" the global floating bar
const isActiveRef = useRef(false)
const disposeStemAudio = useCallback((name: string) => {
const audio = audioRefs.current[name]
const onLoadedMetadata = loadedMetadataListenerRefs.current[name]
const onError = errorListenerRefs.current[name]
if (audio) {
audio.pause()
if (onLoadedMetadata) audio.removeEventListener('loadedmetadata', onLoadedMetadata)
if (onError) audio.removeEventListener('error', onError)
audio.removeAttribute('src')
audio.load()
}
sourceRefs.current[name]?.disconnect()
gainRefs.current[name]?.disconnect()
delete loadedMetadataListenerRefs.current[name]
delete errorListenerRefs.current[name]
delete sourceRefs.current[name]
delete gainRefs.current[name]
delete audioRefs.current[name]
}, [])
// Stable stop callback for the global audio registry
const stopPlayback = useCallback(() => {
isActiveRef.current = false
@@ -49,8 +75,13 @@ export default function DynamicMixPlayer({ stems, title, onLevelChange }: Props)
// pause the audio elements, so they keep playing after navigation.
useEffect(() => () => {
stopPlayback()
Object.keys(audioRefs.current).forEach(disposeStemAudio)
if (audioCtxRef.current && audioCtxRef.current.state !== 'closed') {
void audioCtxRef.current.close().catch(() => {})
audioCtxRef.current = null
}
unregisterPlayback(stopPlayback)
}, [stopPlayback])
}, [disposeStemAudio, stopPlayback])
// Lazily create the shared AudioContext on first interaction (browser autoplay policy)
function getCtx(): AudioContext {
@@ -60,9 +91,42 @@ export default function DynamicMixPlayer({ stems, title, onLevelChange }: Props)
return audioCtxRef.current
}
useEffect(() => {
const stemNames = new Set(stems.map(stem => stem.name))
setGains(prev => {
const next: Record<string, number> = {}
stems.forEach(stem => { next[stem.name] = prev[stem.name] ?? 1.0 })
return next
})
setLoaded(prev => {
const next: Record<string, boolean> = {}
stems.forEach(stem => { next[stem.name] = prev[stem.name] ?? false })
return next
})
setMutedStems(prev => {
const next: Record<string, boolean> = {}
Object.keys(prev).forEach(name => {
if (stemNames.has(name)) next[name] = prev[name]
})
return next
})
setSoloedStems(prev => {
const next: Record<string, boolean> = {}
Object.keys(prev).forEach(name => {
if (stemNames.has(name)) next[name] = prev[name]
})
return next
})
}, [stems])
useEffect(() => {
const ctx = getCtx()
const activeStemNames = new Set(stems.map(stem => stem.name))
Object.keys(audioRefs.current)
.filter(name => !activeStemNames.has(name))
.forEach(disposeStemAudio)
stems.forEach((stem) => {
if (audioRefs.current[stem.name]) return // already wired up
@@ -70,13 +134,15 @@ export default function DynamicMixPlayer({ stems, title, onLevelChange }: Props)
audio.preload = 'metadata'
audio.crossOrigin = 'anonymous'
audio.addEventListener('loadedmetadata', () => {
const onLoadedMetadata = () => {
setDuration((d) => Math.max(d, audio.duration || 0))
setLoaded((l) => ({ ...l, [stem.name]: true }))
})
audio.addEventListener('error', () => {
}
const onError = () => {
setLoaded((l) => ({ ...l, [stem.name]: false }))
})
}
audio.addEventListener('loadedmetadata', onLoadedMetadata)
audio.addEventListener('error', onError)
// Route through Web Audio — all MediaElementSourceNodes in the same
// AudioContext share one clock, eliminating independent-element drift.
@@ -87,9 +153,16 @@ export default function DynamicMixPlayer({ stems, title, onLevelChange }: Props)
audioRefs.current[stem.name] = audio
gainRefs.current[stem.name] = gain
sourceRefs.current[stem.name] = source
loadedMetadataListenerRefs.current[stem.name] = onLoadedMetadata
errorListenerRefs.current[stem.name] = onError
})
// RAF loop — read playback position from the first element
setDuration(0)
}, [disposeStemAudio, stems])
// RAF loop — read playback position from the first element
useEffect(() => {
let raf: number
function tick() {
const first = Object.values(audioRefs.current)[0]
@@ -99,7 +172,7 @@ export default function DynamicMixPlayer({ stems, title, onLevelChange }: Props)
raf = requestAnimationFrame(tick)
return () => cancelAnimationFrame(raf)
}, [stems])
}, [])
const togglePlay = useCallback(async () => {
const ctx = getCtx()
@@ -184,6 +257,10 @@ export default function DynamicMixPlayer({ stems, title, onLevelChange }: Props)
})
}
useEffect(() => {
applyGains()
}, [gains, isMuted, masterVolume, mutedStems, soloedStems])
function updateGain(name: string, value: number) {
setGains((g) => ({ ...g, [name]: value }))
applyGains({ name, gain: value })
@@ -224,7 +301,7 @@ export default function DynamicMixPlayer({ stems, title, onLevelChange }: Props)
})
}
const allLoaded = stems.every((s) => loaded[s.name] !== false)
const allLoaded = stems.every((s) => loaded[s.name] === true)
return (
<div className="space-y-3">
+3 -2
View File
@@ -4,10 +4,11 @@ import { Upload } from 'lucide-react'
interface Props {
label?: string
accept?: string
multiple?: boolean
onFiles: (files: File[]) => void
}
export default function FileUpload({ label = 'Drop files here', accept, onFiles }: Props) {
export default function FileUpload({ label = 'Drop files here', accept, multiple = true, onFiles }: Props) {
const handleDrop = useCallback(
(e: React.DragEvent) => {
e.preventDefault()
@@ -34,7 +35,7 @@ export default function FileUpload({ label = 'Drop files here', accept, onFiles
>
<Upload className="w-8 h-8 text-slate-400 dark:text-gray-600" />
<span className="text-sm text-slate-600 dark:text-gray-400 text-center">{label}</span>
<input type="file" accept={accept} onChange={handleChange} className="hidden" />
<input type="file" accept={accept} multiple={multiple} onChange={handleChange} className="hidden" />
</label>
)
}
+30 -15
View File
@@ -232,10 +232,14 @@ function WaveformCanvas({
dirtyRef.current = true
if (!audioUrl) return
let cancelled = false
const ctrl = new AbortController()
const actx = new AudioContext()
fetch(audioUrl)
fetch(audioUrl, { signal: ctrl.signal })
.then(r => { if (cancelled) throw new Error('cancelled'); return r.arrayBuffer() })
.then(buf => actx.decodeAudioData(buf))
.then(buf => {
if (cancelled || ctrl.signal.aborted) throw new Error('cancelled')
return actx.decodeAudioData(buf)
})
.then(audioBuf => {
if (cancelled) return
peaksRef.current = downsample(audioBuf.getChannelData(0), BUCKET_COUNT)
@@ -243,7 +247,7 @@ function WaveformCanvas({
})
.catch(() => {})
.finally(() => actx.close())
return () => { cancelled = true; actx.close() }
return () => { cancelled = true; ctrl.abort(); actx.close() }
}, [audioUrl])
// ── Size canvas to CSS box ────────────────────────────────────────────────
@@ -357,6 +361,15 @@ function WaveformCanvas({
return -1
}
function clampWordTime(lineIdx: number, wordIdx: number, candidate: number): number {
const line = linesRef.current[lineIdx]
const prev = line.words[wordIdx - 1]?.time
const next = line.words[wordIdx + 1]?.time ?? (linesRef.current[lineIdx + 1]?.time ?? durationRef.current)
const lower = Math.max(line.time, prev !== undefined ? prev + 0.01 : line.time)
const upper = Math.max(lower + 0.001, next - 0.01)
return Math.max(lower, Math.min(upper, candidate))
}
function onPointerDown(e: PointerEvent) {
if (durationRef.current <= 0) return
const wi = nearestWordIdx(e.clientX)
@@ -374,9 +387,8 @@ function WaveformCanvas({
if (ds?.active) {
ds.hasMoved = true
const lineIdx = selectedLineIdxRef.current!
const line = linesRef.current[lineIdx]
const cap = line.words[ds.wordIdx + 1]?.time ?? (linesRef.current[lineIdx + 1]?.time ?? durationRef.current)
dragPreviewRef.current = { wordIdx: ds.wordIdx, time: Math.max(line.time, Math.min(cap - 0.01, getTimeAt(e.clientX))) }
const clamped = clampWordTime(lineIdx, ds.wordIdx, getTimeAt(e.clientX))
dragPreviewRef.current = { wordIdx: ds.wordIdx, time: clamped }
dirtyRef.current = true
} else {
const near = nearestWordIdx(e.clientX)
@@ -390,9 +402,7 @@ function WaveformCanvas({
if (!ds) return
if (ds.active && ds.hasMoved) {
const lineIdx = selectedLineIdxRef.current!
const line = linesRef.current[lineIdx]
const cap = line.words[ds.wordIdx + 1]?.time ?? (linesRef.current[lineIdx + 1]?.time ?? durationRef.current)
const clamped = Math.max(line.time, Math.min(cap - 0.01, getTimeAt(e.clientX)))
const clamped = clampWordTime(lineIdx, ds.wordIdx, getTimeAt(e.clientX))
dragPreviewRef.current = null
dirtyRef.current = true
onWordTimeDragRef.current(ds.wordIdx, clamped)
@@ -526,9 +536,7 @@ function TimingTable({
const { lineIdx, wordIdx } = wordEditing
const t = parseTs(wordDraft)
if (t === null) { setWordEditing(null); return }
const line = lines[lineIdx]
const cap = line.words[wordIdx + 1]?.time ?? (lines[lineIdx + 1]?.time ?? duration)
onWordTimeChange(lineIdx, wordIdx, Math.max(line.time, Math.min(cap - 0.01, t)))
onWordTimeChange(lineIdx, wordIdx, t)
setWordEditing(null)
}
@@ -543,8 +551,7 @@ function TimingTable({
const line = lines[lineIdx]
const t = parseTs(wordDraft)
if (t !== null) {
const cap = line.words[wordIdx + 1]?.time ?? (lines[lineIdx + 1]?.time ?? duration)
onWordTimeChange(lineIdx, wordIdx, Math.max(line.time, Math.min(cap - 0.01, t)))
onWordTimeChange(lineIdx, wordIdx, t)
}
const nextIdx = e.shiftKey ? wordIdx - 1 : wordIdx + 1
if (nextIdx >= 0 && nextIdx < line.words.length) {
@@ -778,9 +785,17 @@ export default function KaraokeEditor({
const [expandedRow, setExpandedRow] = useState<number | null>(null)
function handleWordTimeChange(lineIdx: number, wordIdx: number, newTime: number) {
const line = lines[lineIdx]
if (!line) return
const prev = line.words[wordIdx - 1]?.time
const next = line.words[wordIdx + 1]?.time ?? (lines[lineIdx + 1]?.time ?? duration)
const lower = Math.max(line.time, prev !== undefined ? prev + 0.01 : line.time)
const upper = Math.max(lower + 0.001, next - 0.01)
const clamped = Math.max(lower, Math.min(upper, newTime))
const newLines = lines.map((l, i) => {
if (i !== lineIdx) return l
const newWords = l.words.map((w, j) => j === wordIdx ? { ...w, time: newTime } : w)
const newWords = l.words.map((w, j) => j === wordIdx ? { ...w, time: clamped } : w)
return { ...l, words: newWords }
})
onLrcChange?.(reconstructLrc(newLines))
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+1 -1
View File
@@ -79,7 +79,7 @@ export default function SettingsPanel({ isOpen, onClose, onOpenFullSettings }: P
<div className="border-t border-slate-200 dark:border-gray-800 pt-4">
<button
onClick={handleOpenFullSettings}
className="w-full flex items-center justify-between p-3 rounded-lg bg-slate-50 dark:bg-gray-800 hover:bg-slate-100 dark:hover:bg-gray-750 transition-colors group"
className="w-full flex items-center justify-between p-3 rounded-lg bg-slate-50 dark:bg-gray-800 hover:bg-slate-100 dark:hover:bg-gray-700 transition-colors group"
>
<div className="flex items-center gap-2">
<Settings className="w-4 h-4 text-slate-500 dark:text-gray-400" />
+11 -3
View File
@@ -120,8 +120,14 @@ function ProjectItem({
e.stopPropagation()
if (!window.confirm(`Delete "${project.name}"?\n\nThis will permanently remove the project and all its files.`)) return
setDeleting(true)
try { await deleteProject(project.id) } catch { /* ignore */ }
finally { setDeleting(false) }
try {
await deleteProject(project.id)
} catch {
window.alert('Failed to delete project. Please try again.')
setDeleting(false)
return
}
setDeleting(false)
onDeleted()
}
@@ -219,7 +225,9 @@ function SubsonicPanel() {
await deleteImport(filename)
refetchImports()
queryClient.invalidateQueries({ queryKey: ['imports'] })
} catch { /* ignore */ }
} catch (e) {
setImportErr(e instanceof Error ? e.message : 'Failed to delete imported file')
}
finally { setDeletingFile(null) }
}
+81 -2
View File
@@ -24,13 +24,11 @@ import os
import shlex
import subprocess
import tempfile
import time
from pathlib import Path
import shutil
from fastapi import FastAPI, File, Form, HTTPException, UploadFile
from fastapi.responses import Response
import omnizart
log = logging.getLogger("omnizart-sidecar")
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
@@ -40,6 +38,87 @@ MIN_DURATION_MS: int = 10_000
OMNIZART_OUTPUT_ROOT: Path = Path(os.environ.get("OMNIZART_OUTPUT_ROOT", "/root/.omnizart"))
OMNIZART_PROJECT_ROOT: Path = OMNIZART_OUTPUT_ROOT / "omnizart"
OMNIZART_CACHE_CHECKPOINT_ROOT: Path = OMNIZART_PROJECT_ROOT / "checkpoints"
# Keep this much VRAM free for other GPU consumers (WSL2 DWM/WDDM, etc.).
# Sidecar-specific override takes precedence; falls back to global reserve.
_OMNIZART_GPU_VRAM_RESERVE_MB: int = int(
os.environ.get(
"OMNIZART_GPU_VRAM_RESERVE_MB",
os.environ.get("STEMENGINE_GPU_VRAM_RESERVE_MB", "2048"),
)
)
def _query_total_vram_mb() -> int | None:
"""Return GPU0 total VRAM in MB using nvidia-smi, or None if unavailable."""
try:
out = subprocess.check_output(
["nvidia-smi", "--query-gpu=memory.total", "--format=csv,noheader,nounits"],
stderr=subprocess.DEVNULL,
text=True,
)
first = (out.strip().splitlines() or [""])[0].strip()
value = int(first)
return value if value > 0 else None
except Exception:
return None
def _apply_tensorflow_gpu_limit() -> None:
"""Cap TensorFlow GPU usage for Omnizart so reserve headroom remains free."""
if _OMNIZART_GPU_VRAM_RESERVE_MB <= 0:
log.info("OMNIZART_GPU_VRAM_RESERVE_MB <= 0, TensorFlow GPU cap disabled")
return
try:
import tensorflow as tf # type: ignore[import]
gpus = tf.config.list_physical_devices("GPU")
if not gpus:
return
total_mb = _query_total_vram_mb()
if total_mb is None:
# Fallback when total VRAM cannot be queried.
for gpu in gpus:
tf.config.experimental.set_memory_growth(gpu, True)
log.info("Omnizart TF limiter fallback: enabled memory growth (total VRAM unknown)")
return
limit_mb = max(256, total_mb - _OMNIZART_GPU_VRAM_RESERVE_MB)
if limit_mb >= total_mb:
for gpu in gpus:
tf.config.experimental.set_memory_growth(gpu, True)
log.info(
"Omnizart TF limiter fallback: reserve too small for hard cap "
"(reserve=%dMB total=%dMB); using memory growth",
_OMNIZART_GPU_VRAM_RESERVE_MB,
total_mb,
)
return
for gpu in gpus:
tf.config.set_logical_device_configuration(
gpu,
[tf.config.LogicalDeviceConfiguration(memory_limit=limit_mb)],
)
log.info(
"Omnizart TF limiter active: total=%dMB reserve=%dMB cap=%dMB",
total_mb,
_OMNIZART_GPU_VRAM_RESERVE_MB,
limit_mb,
)
except RuntimeError as exc:
# Happens if TF runtime already initialized before configuration.
log.warning("Omnizart TF limiter not applied (runtime already initialized): %s", exc)
except Exception as exc:
log.warning("Omnizart TF limiter setup failed: %s", exc)
_apply_tensorflow_gpu_limit()
import omnizart
OMNIZART_BUILTIN_CHECKPOINT_ROOT: Path = Path(omnizart.__file__).resolve().parent / "checkpoints"
SUPPORTED_MODES = {"music", "vocal", "vocal-contour"}
+3
View File
@@ -149,6 +149,9 @@ services:
environment:
- PYTHONUNBUFFERED=1
- OMNIZART_BIN=${OMNIZART_BIN:-omnizart}
# Keep sidecar TensorFlow from consuming full VRAM.
# Falls back to STEMENGINE_GPU_VRAM_RESERVE_MB when unset.
- OMNIZART_GPU_VRAM_RESERVE_MB=${OMNIZART_GPU_VRAM_RESERVE_MB:-${STEMENGINE_GPU_VRAM_RESERVE_MB:-2048}}
# No GPU reservation by default — Omnizart can run on CPU (slower).
# Uncomment the block below to grant GPU access:
deploy: