Myriad changes.

This commit is contained in:
translator committed 2026-04-14 10:36:33 +09:00
1 parent ee851bc9ae
commit f498650481
50 files changed
+7615 -448

No files matched your search

@@ -0,0 +1,198 @@
# 2026-04-13 — Git-Managed Auto Pipeline Presets Plan
## Goal
Make selected Auto Pipeline presets ship with StemEngine from Git so they are available by default on first run, while still allowing local custom presets.
## Implementation status (2026-04-13)
### Completed
- ✅ Added Git-managed built-in preset catalog file:
- `backend/presets/auto_pipeline/default_presets.json`
- ✅ Added backend catalog loader + normalization/merge helpers:
- `backend/services/auto_presets_catalog.py`
- ✅ Extended settings response to expose preset layers:
- `auto_presets_user`
- `auto_presets_builtin`
- `auto_presets_effective`
- ✅ Kept patch writes for `auto_presets` as user-editable list.
- ✅ Frontend Settings hydration now prefers backend effective presets while preserving local fallback.
- ✅ One-time local→backend backfill implemented when backend user preset list is empty.
- ✅ Preset execution path (`useConvertStatus`) now prefers `auto_presets_effective` from settings API.
- ✅ Validation run:
- `frontend`: `npm run -s typecheck` (pass)
- `frontend`: `npm run -s build` (pass)
### Remaining (optional)
- ⏳ UI source badges (`Built-in` / `User` / `Override`) and explicit read-only treatment for built-ins.
- ⏳ CI schema validation for preset catalog JSON.
---
## Recommended storage model
Use a **two-tier preset model**:
1. **Built-in presets (Git-managed, read-only at runtime)**
- Stored in repo, versioned with code, reviewed via PR.
- Always available after `git pull` + restart.
2. **User presets (runtime editable)**
- Stored in existing settings (`settings.json` / `auto_presets`) for per-instance customization.
This avoids losing local changes on upgrade while guaranteeing baseline defaults from Git.
---
## Where to store built-ins
Create a versioned preset catalog file in repo, e.g.:
- `backend/presets/auto_pipeline/default_presets.json`
Optional split by profile:
- `backend/presets/auto_pipeline/default_presets.json`
- `backend/presets/auto_pipeline/karaoke_defaults.json`
- `backend/presets/auto_pipeline/lightweight_defaults.json`
JSON schema mirrors current `AutoPreset` frontend shape:
- `name`
- `model`
- `format`
- `instrumentModels`
- `steps`
---
## Loading precedence (server-side)
At runtime, backend resolves effective presets with this order:
1. Load built-ins from Git file(s).
2. Load user presets from settings (`auto_presets`).
3. Merge by preset `name`:
- user preset with same `name` overrides built-in.
- unique names from both sets are kept.
Expose both views in API response:
- `auto_presets_builtin` (read-only)
- `auto_presets_user` (editable)
- `auto_presets_effective` (merged view for execution/UI)
If keeping API minimal, keep only `auto_presets` as effective merged list + add metadata flags per preset (`source: builtin|user`).
---
## Backend plan
### B1) Add preset catalog loader
- New helper module, e.g. `backend/services/auto_presets_catalog.py`.
- Responsibilities:
- read JSON file(s)
- validate schema
- normalize values
- return list of built-in presets
### B2) Add startup validation
- On startup, validate catalog and log actionable errors.
- If invalid:
- log with file path + preset name + field error
- continue boot with empty built-in list (non-fatal)
### B3) Extend settings API
- Keep patch writes only for user presets (`auto_presets`).
- Return merged/effective presets to frontend.
- Optionally add endpoint:
- `GET /api/settings/auto-presets` (builtin + user + effective)
### B4) Preserve backward compatibility
- Existing `auto_presets` in settings remain valid.
- Migration path: none required for existing installs.
---
## Frontend plan
### F1) Show source badges
In Settings → Auto Pipeline preset list:
- Tag presets with source:
- `Built-in`
- `User`
- `Override` (user replacing built-in name)
### F2) Edit behavior
- Built-in presets are not edited in place.
- “Save” with same name creates/updates user override.
- Delete on override removes user override, revealing built-in again.
- Delete on pure built-in is disabled (or “hide locally” if later desired).
### F3) Selection behavior
- SingUI `preset` mode selector uses effective merged list.
- If selected preset vanishes, fallback to first effective preset with warning toast.
---
## Git + deployment workflow
1. Add/update preset JSON in repo.
2. Commit + PR review.
3. Deploy updated StemEngine image/code.
4. On restart, built-ins appear automatically.
5. Existing user presets remain intact.
This provides “installed/pulled with Git” defaults without clobbering local runtime state.
---
## Optional enhancements
- Support env override path:
- `AUTO_PRESETS_CATALOG_PATH=/app/config/auto_presets.json`
- Support multiple catalogs and deterministic order.
- Add `version` field per preset for future migration tooling.
- Add CI check that validates preset JSON schema.
---
## Acceptance criteria
- Fresh clone/run shows built-in presets without manual setup.
- `git pull` with new presets makes them available after restart.
- User-created presets persist separately and survive upgrades.
- Built-ins are clearly identifiable in UI.
- Execution paths (Auto Pipeline dialog + SingUI preset mode) consume merged effective list.
---
## Rollout sequence
### Phase 1 (backend foundation)
- Add catalog file + loader + merge + settings response updates.
### Phase 2 (frontend UX)
- Add source badges and edit/delete semantics for built-in vs user presets.
### Phase 3 (hardening)
- Add schema validation tooling + CI gate.
---
## Risks and mitigations
- **Risk:** malformed preset file breaks startup behavior.
- **Mitigation:** non-fatal loader + strong validation logs.
- **Risk:** name collisions confuse users.
- **Mitigation:** explicit `source` labels and override semantics.
- **Risk:** accidental overwrite of built-ins.
- **Mitigation:** treat Git presets as read-only in API/UI.
@@ -0,0 +1,242 @@
# Leading & Backing Buttons Sidebar Refactor — 2026-04-13
## Overview
**Objective**: Consolidate karaoke display controls into the performance sidebar by:
1. Moving Leading/Backing lyrics track selection buttons from overlay
2. Adding line display mode options (Scroll, Line, Word, Syllable) to the sidebar
**Status**: ✅ Completed and validated
---
## Problem Statements
### Issue 1: Lyrics Track Button Placement
Previously, the lyrics track selection buttons (Leading, Backing, Main) were rendered in two locations:
1. **Performance mode**: Fixed position overlay at top-right of the video display
2. **Editor mode**: Inline buttons in the universal controls toolbar
This created visual clutter and inconsistency in the UI, especially in fullscreen mode.
### Issue 2: Display Mode Options Hidden
The line display mode selector (Scroll, Line, Word, Syllable) was only available in the style picker panel, requiring users to open a separate UI layer to access frequently-used options.
---
## Solution Implemented
### Changes Made
#### 1. **PerformanceSidebar Component** (`frontend/src/components/PerformanceSidebar.tsx`)
**Added Props**:
- `lyricsTrackKeys: string[]` — Available lyrics track names (e.g., `['leading', 'default', 'backing']`)
- `selectedLyricsTrack: string` — Currently selected track name
- `onSelectLyricsTrack: (key: string) => void` — Callback when user selects a track
**New UI Section**:
- Added "LYRICS" section at the top of the sidebar (when multiple tracks exist)
- Displays track buttons with the same styling as BACKGROUND section buttons
- Responsive: Grid layout on normal widths, flex wrap on narrow viewports
- Visual feedback: Selected track button highlighted in violet
```tsx
{/* ── LYRICS TRACKS ── */}
{lyricsTrackKeys.length > 1 && (
<section className="space-y-2">
<div className="text-xs font-semibold tracking-wide text-slate-200">LYRICS</div>
<div className={isNarrow ? 'flex flex-wrap gap-2' : 'grid grid-cols-2 gap-2'}>
{lyricsTrackKeys.map((key) => (
<button
key={key}
type="button"
onClick={() => onSelectLyricsTrack(key)}
className={`rounded-md px-2 py-2 text-xs border transition-colors ...`}
title={`Show ${key.charAt(0).toUpperCase() + key.slice(1)} lyrics`}
>
{key.charAt(0).toUpperCase() + key.slice(1)}
</button>
))}
</div>
</section>
)}
```
#### 2. **Display Mode Selector** (NEW)
**Added Props to PerformanceSidebar**:
- `hasElrc: boolean` — Whether word-timing data is available
- Uses existing `style` and `patchStyle` props to manage `videoDisplayMode`
**New UI Section**:
- Added "DISPLAY" section in the sidebar
- Shows four mode buttons: Scroll, Line, Word, Syllable
- Word and Syllable modes are disabled when ELRC (word-timing) data is unavailable
- Disabled states show visual feedback (reduced opacity, no-cursor)
- Same grid/responsive layout as other sections
```tsx
{/* ── LINE DISPLAY ── */}
<section className="space-y-2">
<div className="text-xs font-semibold tracking-wide text-slate-200">DISPLAY</div>
<div className={isNarrow ? 'flex flex-wrap gap-2' : 'grid grid-cols-2 gap-2'}>
{(['scroll', 'line', 'word', 'syllable'] as const).map((mode) => (
<button
key={mode}
type="button"
disabled={(mode === 'word' || mode === 'syllable') && !hasElrc}
onClick={() => patchStyle({ videoDisplayMode: mode })}
className={`rounded-md px-2 py-2 text-xs border transition-colors capitalize ...`}
>
{mode}
</button>
))}
</div>
</section>
```
#### 3. **LyricsLivePreview Component** (`frontend/src/components/LyricsLivePreview.tsx`)
**Removed UI Elements**:
- Deleted the fixed-position overlay buttons from performance mode (was at top-right at 84px from top, 10px from right)
- Deleted the inline buttons from the editor mode toolbar (was after "ELRC" indicator)
- Removed unused `labelLyricsTrackKey()` function (was only used by the removed buttons)
**Updated Sidebar Integration**:
- Added four new props passed to `PerformanceSidebar`:
- `lyricsTrackKeys={lyricsTrackKeys}`
- `selectedLyricsTrack={selectedLyricsTrack}`
- `onSelectLyricsTrack={setSelectedLyricsTrack}`
- `hasElrc={hasElrc}` (NEW - for display mode button state)
#### 3. **Button Styling & UX**
The lyrics track and display mode buttons now:
- Match the visual design of other sidebar sections (Background, Stems, Lyrics)
- Use consistent border and background colors
- Provide hover states for interactive feedback
- Adapt to narrow viewports by wrapping in flex layout
- Include descriptive tooltips
- For display modes: Show disabled state for unavailable options (Word/Syllable without ELRC data)
---
## Files Modified
| File | Changes |
|------|---------|
| [`frontend/src/components/PerformanceSidebar.tsx`](frontend/src/components/PerformanceSidebar.tsx#L200-L280) | Added lyrics track + display mode sections; added `hasElrc` prop |
| [`frontend/src/components/LyricsLivePreview.tsx`](frontend/src/components/LyricsLivePreview.tsx#L3190-L3230) | Removed overlay + toolbar buttons; removed `labelLyricsTrackKey()`; added sidebar props including `hasElrc` |
---
## Visual Impact
### Before
- Lyrics track buttons floating in top-right corner during karaoke (visual clutter)
- Same buttons inline in editor toolbar (competing for space)
- Display mode options only accessible through style picker panel
- Different styling in each location
### After
- Unified "LYRICS" section in the performance sidebar for track selection
- New "DISPLAY" section in sidebar for line display modes (Scroll, Line, Word, Syllable)
- Clean toolbar without overlapping buttons in editor mode
- Single, centralized control panel for all display options
- Consistent styling across all sections
- Buttons only visible when relevant (display modes always, lyrics tracks only when multiple exist)
- Better use of screen space, especially in fullscreen
- Word/Syllable modes clearly disabled when timing data unavailable
### Sidebar Layout (Performance Mode)
```
┌─────────────────────┐
│ Controls ✕ │
├─────────────────────┤
│ LYRICS │
│ [Leading] [Backing] │
├─────────────────────┤
│ DISPLAY │
│ [Scroll] [Line] │
│ [Word] [Syllable] │ (Word/Syllable disabled if no ELRC)
├─────────────────────┤
│ BACKGROUND │
│ [Static] [Muon] │
│ [Fluid] [Video] │
├─────────────────────┤
│ STEMS │
│ ... (volume controls)
```
---
## Validation
### TypeScript Checking
```
✅ No errors in PerformanceSidebar.tsx
✅ No errors in LyricsLivePreview.tsx
```
### Production Build
```
✅ Build succeeded in 5.75s
✅ All 2860 modules transformed successfully
✅ Bundle sizes nominal (523.33 kB gzip)
```
---
## User Experience
**When Performance Sidebar is open**:
1. LYRICS section shows track buttons (leading, backing, main)
- Currently selected track highlighted in violet
- Clicking instantly switches displayed lyrics
2. DISPLAY section shows line display mode options
- Scroll: Horizontal scrolling of lyrics
- Line: Full-screen line display
- Word: Word-by-word sync (requires ELRC data)
- Syllable: Syllable-by-syllable sync (requires ELRC data)
- Disabled options show grayed appearance
**When Performance Sidebar is closed**:
- No display controls visible (sidebar can be toggled with button)
**In editor mode**:
- Lyrics tracks and display modes no longer shown in toolbar
- Cleaner interface focused on editing controls (style, timing, effects)
- Display options still accessible via style picker panel
---
## Backward Compatibility
✅ **No breaking changes**:
- PerformanceSidebar adds optional props (with sensible defaults)
- Display mode logic remains unchanged
- User preferences persist in localStorage as before
- Graceful degradation for Word/Syllable modes when timing data unavailable
---
## Performance Notes
- No additional render loops or re-renders introduced
- Button clicks directly update state via `patchStyle()`
- Conditional rendering based on `hasElrc` avoids rendering disabled buttons
- Same responsive breakpoint logic as existing sidebar sections
---
## Summary
This refactor successfully consolidates all karaoke display controls into a unified performance sidebar:
- **LYRICS section** for track selection (analogous to BACKGROUND section)
- **DISPLAY section** for line display modes
- Eliminates visual clutter from overlays and toolbars
- Improves UX by centralizing related controls
- Maintains full functionality while improving discoverability
The sidebar now serves as the comprehensive control center for performance mode.
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,190 @@
# 2026-04-13 — SingUI Pipeline Selection + Direct Project Karaoke Plan
## Goal
Make SingUI cleaner by supporting two clear paths:
1. **Option A:** Add a Settings control to choose which processing pipeline SingUI uses when you click **Package**.
2. **Option B:** Support a workflow that **skips `.sek` packaging** and sings directly from existing StemEngine project stems in `/karaoke`.
## Implementation status (started 2026-04-13)
### Completed in this pass
- ✅ Added server-persisted setting: `singui_process_mode` with validation (`auto` | `separate_only` | `pipeline` | `preset`).
- ✅ Added server-persisted optional selection: `singui_process_preset`.
- ✅ Added Settings UI control under Auto Pipeline to choose/save SingUI package behavior across all four modes.
- ✅ Added conditional preset selector in Settings when mode is `preset`.
- ✅ Wired SingUI package execution to respect `singui_process_mode`:
- `auto` = run configured additional Auto Pipeline steps.
- `separate_only` = stop after separation/project creation.
- ✅ Extended conversion execution logic to include additional process strategies:
- `pipeline` = submit `/api/pipeline` directly (source audio streamed from Subsonic endpoint into upload payload).
- `preset` = run separation using selected preset model/format/instrument refinement + execute that preset step list.
- ✅ Updated SingUI karaoke row actions to prioritize **project** playback when a linked project exists.
- ✅ Updated queue-item preference in project-linked mode to queue project sources before SEK candidates.
- ✅ Updated main action text toward project-centric flow (`Prepare Project`) and aligned status copy across Sing/Tracks/Albums (`Preparing`, `Project Ready`).
- ✅ Validation pass completed after steps 1–3:
- `frontend`: `npm run -s typecheck` (pass)
- `frontend`: `npm run -s build` (pass)
- ✅ Optional preset-storage migration completed:
- Auto Pipeline presets are now read from backend app settings (`auto_presets`) when available.
- Settings preset save/delete now persist to backend settings and keep local fallback.
- One-time backfill migrates existing local preset list to backend when server list is empty.
### Remaining from full plan
- None.
---
## Current State (baseline)
- SingUI package actions currently run a fixed frontend pipeline flow (`import -> separate -> optional steps`) via `useConvertStatus`.
- Auto pipeline configuration is primarily browser-local (`localStorage`) and not centrally selected for SingUI routing.
- Karaoke page already supports **project mode** (`projectId`) and **SEK mode** (`sekPackageId`), but SingUI karaoke list still emphasizes SEK matching in important paths.
---
## Option A — Settings Item: Choose Song Processing Pipeline in SingUI
### Product outcome
Add a setting like:
- **Settings → Auto Pipeline → “SingUI Package behavior”**
- `Auto Pipeline (current behavior)`
- `Full /api/pipeline endpoint`
- `Separation only`
- (optional) `Custom preset: <name>`
This makes behavior explicit and removes hidden coupling.
### Implementation plan
#### A1) Define configuration model (server-persisted)
Add fields to app settings so selection is shared across browsers/users of the same instance:
- `singui_process_mode`: `'auto' | 'pipeline' | 'separate_only' | 'preset'`
- `singui_process_preset`: `string | null` (used when mode = `preset`)
Also decide where pipeline presets live long-term:
- **Phase 1 (fast):** keep existing local presets; setting only chooses execution mode.
- **Phase 2 (clean):** move presets to backend settings for central consistency.
#### A2) Add Settings UI
In Settings Auto Pipeline section:
- Add a dropdown for default SingUI process mode.
- If mode is `preset`, show a second dropdown of presets.
- Add inline summary text: “Used by Package buttons in Tracks/Albums/SingUI”.
#### A3) Wire execution in SingUI packaging path
Refactor package execution into a single strategy resolver:
- `mode=auto`: current `useConvertStatus` behavior.
- `mode=pipeline`: submit `/api/pipeline` with chosen defaults.
- `mode=separate_only`: only create/update project stems, no post-steps.
- `mode=preset`: execute steps from selected preset.
Key principle: one orchestrator, multiple strategies, same status UI.
#### A4) Migration and fallback rules
- If no setting exists, default to `auto`.
- If `preset` points to missing preset, fall back to `auto` + toast warning.
- Maintain backward compatibility with existing localStorage keys.
#### A5) Testing/validation
- Unit: strategy resolver picks correct execution path.
- Integration: Package button in Tracks, Albums, Sing triggers selected mode.
- Regression: queue/status badges unchanged.
### Option A acceptance criteria
- Admin/user can choose SingUI package mode in Settings.
- Package actions consistently follow selected mode across pages.
- Existing users continue working without reconfiguration.
### Option A risk/effort
- **Effort:** Medium
- **Risk:** Low–Medium (mainly migration/consistency between local and server settings)
---
## Option B — Skip `.sek`: Sing directly from existing project stems
### Product outcome
Allow karaoke flow to use StemEngine projects directly and treat `.sek` as optional export/distribution, not required for local singing.
### Implementation plan
#### B1) Clarify source modes in SingUI karaoke
Rename/clarify karaoke source options:
- `Project-linked` (primary local mode)
- `SEK Folder` (portable package mode)
In `Project-linked` mode, do **not** depend on SEK matching to render playable rows.
#### B2) Prioritize project playback path
For each song row:
1. If linked project exists → show `Sing` immediately (project source).
2. If not linked and user clicks `Package` → run selected processing mode, then auto-link and enable `Sing`.
3. SEK badges can still appear as informational, but not gate playback.
#### B3) Queue semantics
When user queues from `Project-linked` mode:
- Create queue items as `source.kind='project'` with `projectId`.
- Keep existing SEK queue behavior unchanged in SEK mode.
#### B4) UX cleanup
- Labels: “Package (create/update project)” instead of implying SEK export.
- Move `.sek` export language to explicit export actions only.
- Add helper text: “Local karaoke uses project stems; SEK is for portable packages.”
#### B5) Optional backend support (if needed)
If frontend-only linking is insufficient at scale, add endpoint for stable song→project mapping with confidence score.
### Option B acceptance criteria
- User can sing from project stems without creating `.sek`.
- Karaoke queue works end-to-end in project mode.
- `.sek` remains available but optional.
### Option B risk/effort
- **Effort:** Low–Medium
- **Risk:** Low (most underlying project playback support already exists)
---
## Decision guidance
- Choose **Option A first** if your top priority is explicit, configurable processing behavior.
- Choose **Option B first** if your top priority is faster daily karaoke flow with less packaging friction.
- Best practical sequence:
1. **B first** (quick UX win, immediate simplification)
2. **A second** (clean policy control for future growth)
---
## Suggested phased rollout
### Phase 1 (1–2 days)
- Implement Option B UX/path cleanup for direct project singing.
- Keep current auto pipeline execution behavior unchanged.
### Phase 2 (2–4 days)
- Implement Option A mode selector + strategy resolver.
- Add migration/fallback logic.
### Phase 3 (optional hardening)
- Move pipeline preset storage from localStorage to backend settings.
- Add telemetry/toasts for mode/fallback visibility.
---
## Open questions before implementation
1. Should SingUI mode be global instance-wide or per-user/browser?
2. Do we want `preset` mode in v1, or only `auto/pipeline/separate_only`?
3. Should `Package` update existing linked projects in-place by default?
4. Do we keep SEK matching badges in Project-linked mode, or hide entirely unless SEK mode is selected?
File diff suppressed because it is too large. Load diff
+932
View File
@@ -0,0 +1,932 @@
# StemEngine Codebase Audit — 2026-04-14
**Scope**: Full codebase — frontend (React/TypeScript), backend (Python/FastAPI), visualizers (WebGL/Three.js)
**Auditors**: 7 parallel specialist agents (security, React, performance, code quality, TypeScript, Python backend, architecture)
**Total findings**: 92 across 7 domains
---
## Priority Summary
| Level | Count | IDs |
|-------|-------|-----|
| Critical | 2 | P1, A4 |
| High | 23 | SEC1–2, R1–4, P2–5, C1–4, PY2–4, TS1, A1–2, A5, A7 |
| Medium | 36 | SEC3–7, SEC9–10, R5–10, P6–9, C5–9, PY1, PY5–6, PY10–11, TS2–9, TS11–12, A3, A6, A8–9 |
| Low | 31 | SEC8, SEC11, R11–14, P10–12, C10–13, PY7–9, PY12, TS13–18, A10–11 |
**Recommended sprint order**:
1. Security: SEC1, SEC2, PY3 (path traversal at 13 sites)
2. Correctness: PY4 (GPU OOM), C3 (stale beatPhase), A3 (Spirit palette flash), A7/R1 (Spirit resize)
3. Performance: P1 (Page Visibility), P3 (FFT when paused), P4 (Float32Array allocs)
4. Architecture: A6 → A1 → A2 (pre-req for Visualizer 5)
5. Reliability: C2 (fluid race), PY2 (blocking I/O), PY5 (urlretrieve)
6. Type safety: TS1–TS5, TS9–TS10
7. Housekeeping: C4, TS6–TS18, PY1, PY7–PY12, A10, A11
---
## 1. Security
### SEC1 — No Authentication on ML Inference Endpoints
**Severity**: High
**Files**: `backend/routers/transcribe.py`, `backend/routers/separate.py`, `backend/routers/lyric_transcribe.py`
**Problem**: All GPU-backed endpoints (`/api/separate`, `/api/transcribe`, `/api/lyric-transcribe`) accept requests without any auth token check. `APP_ENV` defaults to `"development"`, which disables `ADMIN_TOKEN` enforcement. The `docker-compose.yml` never sets `APP_ENV=production`, so the disable is always active in standard deployments.
**Fix**: Require `ADMIN_TOKEN` header on all ML endpoints unconditionally (not gated on `APP_ENV`), or add a separate lightweight Bearer check. Change `APP_ENV` default to `"production"` in compose file.
---
### SEC2 — Unsanitised `file.filename` Path Construction — Path Traversal
**Severity**: High
**Files**: `backend/routers/separate.py:267`, `backend/routers/transcribe.py:48`, `backend/routers/lrc.py:36,135`, `backend/routers/pipeline.py:59`, `backend/routers/lyric_transcribe.py:318,476` (6 endpoints)
**Problem**: Raw `UploadFile.filename` from the HTTP multipart request is used directly in `Path(UPLOAD_DIR / file.filename)` without stripping directory components. A client sending `filename="../../config.py"` can overwrite arbitrary files reachable from the upload directory. The `lrc.py:36` path has no job_id prefix, making it the most exposed.
**Fix**: Add a sanitiser and apply at all 6 sites:
```python
from pathlib import Path
def _safe_filename(raw: str | None) -> str:
if not raw:
return "upload"
name = Path(raw).name # strips directory prefix
if not name or name.startswith("."):
return "upload"
return name
upload_path = upload_dir / _safe_filename(file.filename)
```
*(Overlaps with PY3 — fix once, apply everywhere.)*
---
### SEC3 — File Size Not Enforced in FastAPI Layer
**Severity**: Medium
**Files**: `backend/routers/separate.py`, `backend/routers/transcribe.py`
**Problem**: `MAX_UPLOAD_BYTES` is defined but only checked after full upload into memory. Large uploads exhaust server RAM before the check fires.
**Fix**: Use `UploadFile.size` or stream with a byte counter; reject at first chunk exceeding limit.
---
### SEC4 — SSRF via yt-dlp URL Parameter
**Severity**: Medium
**Files**: `backend/routers/separate.py` (yt-dlp path)
**Problem**: User-supplied URLs are forwarded to yt-dlp with only an `http://` prefix check. RFC-1918 addresses and Docker-internal services (port 9765, 8777) are reachable. No allow-list of domains.
**Fix**: Resolve the URL's hostname and reject RFC-1918 and loopback ranges before invoking yt-dlp. Consider a domain allow-list.
---
### SEC5 — CORS Wildcard Active in Default Configuration
**Severity**: Medium
**Files**: `backend/main.py`, `docker-compose.yml`
**Problem**: `APP_ENV` defaults to `"development"`, which sets `allow_origins=["*"]`. The compose file never sets `APP_ENV=production`.
**Fix**: Change `APP_ENV` default to `"production"` in compose; require explicit opt-in for wildcard CORS.
---
### SEC6 — Cross-Job File Access in Separation Results
**Severity**: Medium
**Files**: `backend/routers/separate.py` (results path)
**Problem**: Job result paths are constructed from job IDs with no ownership check. A user who knows another job's ID can fetch its stems.
**Fix**: Associate jobs with a session token or user ID; validate ownership before serving results.
---
### SEC7 — `project_name` Parameter Not Sanitised
**Severity**: Medium
**Files**: `backend/routers/separate.py`, `backend/routers/projects.py`
**Problem**: `project_name` from request body is used directly in directory name construction without `Path.name` stripping, allowing `../` traversal.
**Fix**: Apply `Path(project_name).name` before any path join.
---
### SEC8 — Dependency on Vendored Unaudited `webgl-fluid.mjs`
**Severity**: Low / Info
**Files**: `frontend/src/components/visualizers/fluid/webgl-fluid.mjs`
**Problem**: 1,813-line minified JS file with a custom modification (`_fluidSplat` injection at line 1808) that is not in the upstream npm package. No audit trail or comment block.
**See also**: A5 for full architectural treatment.
---
### SEC9 — Missing `X-Content-Type-Options` / Security Headers
**Severity**: Low
**Files**: `backend/main.py`
**Problem**: No `X-Content-Type-Options`, `X-Frame-Options`, or `Content-Security-Policy` headers set on API responses. Not blocking for an API-only backend, but relevant if the frontend is served from the same origin.
---
### SEC10 — No Rate Limiting on Upload Endpoints
**Severity**: Low
**Files**: All upload routers
**Problem**: No rate limiting on any endpoint. An attacker can submit unlimited GPU jobs.
---
### SEC11 — Hardcoded Fallback Admin Token
**Severity**: Low
**Files**: `backend/auth.py` or equivalent
**Problem**: A fallback `ADMIN_TOKEN` value exists in source code for development convenience. Should be removed entirely; if unset, the endpoint should reject all requests.
---
## 2. React / Frontend Architecture
### R1 — SpiritVisualizerCanvas Full GL Reinit on Every Resize
**Severity**: High
**Files**: `frontend/src/components/visualizers/SpiritVisualizerCanvas.tsx` (initAndStart useCallback)
**Problem**: `width` and `height` are in the `useCallback` dependency array for `initAndStart`. Every resize triggers `destroySpiritSim` + `initSpiritSim`, recreating the Three.js renderer, `EffectComposer`, `UnrealBloomPass`, `AfterimagePass`, and two WebGL render targets (~50ms on integrated GPU). Muon, Fluid, and Tendrils handle resize imperatively inside the RAF loop.
**Fix**: Remove `width`/`height` from `initAndStart` deps. Handle resize inside `renderSpiritFrame` by comparing `canvas.clientWidth`/`canvas.clientHeight` to `renderer.getSize()` each frame. *(Also tracked as A7.)*
---
### R2 — Audio Node Teardown on Fader Drag
**Severity**: High
**Files**: `frontend/src/components/LyricsLivePreview.tsx` (GainNode useEffect)
**Problem**: The `useEffect` that creates `GainNode` objects has the stem volumes array in its dependency list. Dragging a fader reconnects/disconnects the AudioNode graph on every tick of the slider, causing clicks and brief audio dropouts.
**Fix**: Create GainNodes once (on mount / stems change). Update `.gain.value` in a separate `useEffect` that has only the volume values in its deps — no node reconstruction needed.
---
### R3 — Spurious `onPlaybackEnded` from Stale Closure
**Severity**: High
**Files**: `frontend/src/components/LyricsLivePreview.tsx` (audio element event listener)
**Problem**: The `'ended'` event listener on the `<audio>` element captures `onPlaybackEnded` at registration time. If the parent re-renders and passes a new reference, the stale callback fires on song end. In playlist mode this causes duplicate playlist-advance calls.
**Fix**: Register the listener once; read the current callback via a stable `onPlaybackEndedRef.current = onPlaybackEnded` pattern.
---
### R4 — `applyGains` Teardown on Every Volume Tick
**Severity**: High
**Files**: `frontend/src/components/LyricsLivePreview.tsx`
**Problem**: A `useEffect` with volume values in its dep array calls `applyGains`, which disconnects and reconnects `GainNode` routing. Smoothly fading a stem volume causes a full re-routing every render cycle.
**Fix**: `applyGains` should only update `.gain.value` (or `.gain.linearRampToValueAtTime` for smooth transitions), never disconnect/reconnect nodes.
---
### R5 — Context Action Functions Not in `useCallback`
**Severity**: Medium
**Files**: `frontend/src/contexts/KaraokeQueueContext.tsx`, `frontend/src/contexts/QueueContext.tsx`
**Problem**: Action functions (`addToQueue`, `removeFromQueue`, etc.) are defined inline in the context provider without `useCallback`. Every provider state change re-creates these functions, causing all consumers to re-render even when the data they care about hasn't changed.
**Fix**: Wrap each action in `useCallback` with stable deps (only the dispatch or set functions, which are stable from `useReducer`/`useState`).
---
### R6 — `LyricsLivePreview` Re-Renders at Audio Tick Rate
**Severity**: Medium
**Files**: `frontend/src/components/LyricsLivePreview.tsx`
**Problem**: `setCurrentTime` called from an `requestAnimationFrame` callback at 60fps causes React to schedule a re-render every frame. The component has 35+ `useState` calls; even with memoization, reconciling a 5,196-line component tree at 60fps is expensive.
**Fix**: Move `currentTime` into a `useRef` (not `useState`). Components that need to display the current time should use a separate lightweight subscription (e.g., a context with a stable callback ref pattern, or a separate small component with its own RAF).
---
### R7 — Missing Cleanup for `ResizeObserver` in Visualizer Canvases
**Severity**: Medium
**Files**: All four `*VisualizerCanvas.tsx` files
**Problem**: `ResizeObserver` instances created in `useEffect` may not be disconnected in the cleanup return if the component unmounts while a resize is in flight.
**Fix**: Ensure `observer.disconnect()` is called in the `useEffect` cleanup function in all four files.
---
### R8 — `settingsRef` Synced via `useEffect` Instead of Unconditional Write (Spirit)
**Severity**: Medium
**Files**: `frontend/src/components/visualizers/SpiritVisualizerCanvas.tsx:99`
**Problem**: `settingsRef.current = settings` inside a `useEffect` instead of at render time. The render function runs, the new settings are committed to DOM, but `settingsRef` still holds the previous frame's settings until React drains the effect queue.
**Fix**: Assign `settingsRef.current = settings` unconditionally at the top of the component function body (before any `return`), not in `useEffect`.
---
### R9 — `useAudioFeatures` Stale Values in `patternInfo` / `beatInfo` When Paused
**Severity**: Medium
**Files**: `frontend/src/hooks/useAudioFeatures.ts`
**Problem**: When `!isPlayingRef.current`, only some fields are zeroed. `beatPhase`, `isBeatDrop`, `energyDelta` retain their last non-zero values, continuing to drive visualizer animations (beat pulses, drop effects) during silence or pause.
**Fix**: Zero all fields in the pause branch, including `beatInfo.beatPhase`, `patternInfo.isBeatDrop`, and `patternInfo.energyDelta`. *(Also tracked as C3.)*
---
### R10 — No `ErrorBoundary` Around Any Visualizer
**Severity**: Medium
**Files**: `frontend/src/components/visualizers/VisualizerBackdrop.tsx`
**Problem**: WebGL context loss, shader compile failure, or a JavaScript exception in any visualizer canvas crashes the entire React tree. No `ErrorBoundary` / `componentDidCatch` exists anywhere in the frontend.
**Fix**: Wrap each visualizer `<Suspense>` block in an `ErrorBoundary` that renders a silent fallback (black canvas or nothing) and reports to console. *(Also tracked as C4.)*
---
### R11 — Sidebar Polls Job List Every 8s Regardless of Active Jobs
**Severity**: Low
**Files**: `frontend/src/components/Sidebar.tsx`
**Problem**: `refetchInterval: 8000` is unconditional. When no jobs are running, polling is wasted.
**Fix**: Use the `refetchInterval` callback form: `(query) => query.state.data?.some(j => j.status === 'running') ? 3000 : false`.
---
### R12 — `LyricsLivePreview` `styleRebuildKey` JSON-Serializes on Every Render
**Severity**: Low
**Files**: `frontend/src/components/LyricsLivePreview.tsx:5109–5149`
**Problem**: 24 style properties are JSON-serialized every render to compute a cache key. This fires even when unrelated state changes trigger a re-render.
**Fix**: Use `useMemo` with the individual style state values as deps, not a JSON string. The memo runs only when a style value actually changes.
---
### R13 — `Two.js` Canvas RAF Loop Runs When Component Is Hidden
**Severity**: Low
**Files**: `frontend/src/components/LyricsLivePreview.tsx`
**Problem**: The Two.js RAF loop continues rendering when the karaoke view is not visible. Add `document.addEventListener('visibilitychange', ...)` and pause the Two.js `play()`/`pause()` accordingly. *(Related to P6.)*
---
### R14 — `MuonFeatureExtractor` Runs Analysis Loop Even When Not Playing
**Severity**: Low
**Files**: `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
**Problem**: The internal RAF loop calls `analyser.getFloatFrequencyData()` and 5 O(512) analysis passes every frame, allocating 3 `Float32Array` per frame, regardless of playback state. *(See P3, P4.)*
---
## 3. Performance
### P1 — No Page Visibility API Guard in Any Visualizer RAF Loop
**Severity**: Critical
**Files**: `MuonVisualizerCanvas.tsx`, `FluidVisualizerCanvas.tsx`, `TendrilsVisualizerCanvas.tsx`, `SpiritVisualizerCanvas.tsx`
**Problem**: All four visualizers continue running their RAF loops at full rate when the tab is hidden (`document.hidden === true`). A hidden tab with an active visualizer burns full GPU at 60fps with zero user benefit. On battery devices this drains power; on integrated GPUs it can cause thermal throttling that affects other browser tabs.
**Fix**: Add to all four visualizer canvas files:
```ts
useEffect(() => {
const onVisibility = () => {
if (document.hidden) cancelAnimationFrame(rafRef.current)
else rafRef.current = requestAnimationFrame(renderLoop)
}
document.addEventListener('visibilitychange', onVisibility)
return () => document.removeEventListener('visibilitychange', onVisibility)
}, [])
```
---
### P2 — No IntersectionObserver — Canvases Render Off-Screen
**Severity**: High
**Files**: All four `*VisualizerCanvas.tsx` files
**Problem**: Visualizers render at full rate even when scrolled out of view or obscured by overlapping panels (sidebar, settings drawer). An `IntersectionObserver` on the canvas element can pause the RAF loop when intersection ratio drops to 0.
**Fix**: Add `IntersectionObserver` in each visualizer's `useEffect`:
```ts
const io = new IntersectionObserver(([e]) => {
isVisibleRef.current = e.isIntersecting
}, { threshold: 0 })
io.observe(canvas)
return () => io.disconnect()
```
Check `isVisibleRef.current` at the top of each render loop.
---
### P3 — `MuonFeatureExtractor` FFT Analysis Runs When Paused
**Severity**: High
**Files**: `frontend/src/hooks/useAudioFeatures.ts` (or `MuonVisualizerCanvas.tsx`)
**Problem**: `analyser.getFloatFrequencyData()` and 5 O(N=512) analysis passes run every frame regardless of `isPlayingRef.current`. When audio is paused the FFT data is all `-Infinity` — the analysis work is pure waste.
**Fix**: Gate the entire analysis block:
```ts
if (!isPlayingRef.current) { /* emit zeroed features, skip analysis */ return }
```
---
### P4 — 3 `Float32Array` Allocations Per Frame in Audio Analysis
**Severity**: High
**Files**: `frontend/src/hooks/useAudioFeatures.ts`
**Problem**: `getFloatFrequencyData()` allocates a new `Float32Array` per call and is called 3 times per frame (for different analysis windows). At 60fps this is 180 allocations/second, 3–9 KB each, generating 540 KB/s of GC pressure.
**Fix**: Pre-allocate three `Float32Array` refs outside the RAF loop and reuse them:
```ts
const freqBufRef = useRef(new Float32Array(analyser.frequencyBinCount))
// ...
analyser.getFloatFrequencyData(freqBufRef.current)
```
---
### P5 — 1,600 `createRadialGradient` Calls Per Frame (Muon)
**Severity**: High
**Files**: `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
**Problem**: The muon particle renderer calls `ctx.createRadialGradient()` for each of the ~1,600 visible particles every frame. `createRadialGradient` allocates a new `CanvasGradient` object each call. This generates ~100K allocations/second.
**Fix**: Use `ctx.arc()` + `ctx.fill()` with a pre-computed radial gradient stamped via `drawImage` onto an off-screen canvas (sprite atlas pattern). One gradient object, reused for all particles.
---
### P6 — `AudioContext` Not Suspended When Nothing Is Playing
**Severity**: Medium
**Files**: `frontend/src/components/LyricsLivePreview.tsx`
**Problem**: `AudioContext` stays in `running` state even when audio is paused or no track is loaded. A running `AudioContext` keeps the audio thread active at the system level.
**Fix**: Call `audioContext.suspend()` when pausing; `audioContext.resume()` before playing.
---
### P7 — Two.js RAF Loop Continues When Tab Is Hidden
**Severity**: Medium
**Files**: `frontend/src/components/LyricsLivePreview.tsx`
**Problem**: The Two.js canvas inside the karaoke view runs its built-in RAF loop via `two.play()`. It has no `visibilitychange` listener.
**Fix**: `document.addEventListener('visibilitychange', () => document.hidden ? two.pause() : two.play())`.
---
### P8 — Sidebar Polls API Every 8s Even with No Active Jobs
**Severity**: Medium
**Files**: `frontend/src/components/Sidebar.tsx`
**Problem**: Unconditional 8-second polling interval. *(Also R11 — identical fix.)*
---
### P9 — `deepMergeVizSettings` Called on Every Render in `LyricsLivePreview`
**Severity**: Medium
**Files**: `frontend/src/components/LyricsLivePreview.tsx`
**Problem**: Settings normalization (`deepMergeVizSettings` / `normalizeVizSettings`) is called unconditionally each render rather than in a `useMemo`.
**Fix**: Memoize: `const mergedSettings = useMemo(() => deepMergeVizSettings(VIZ_DEFAULTS, sekVisualizerSettings), [sekVisualizerSettings])`.
---
### P10 — Karaoke TV Polling Regardless of Play State
**Severity**: Low
**Files**: `frontend/src/pages/KaraokeTV.tsx`
**Problem**: Queue/playback state polling runs unconditionally regardless of whether a song is actually playing. Gate on `isPlaying` state or switch to a WebSocket-based push model.
---
### P11 — No Memoization on Expensive `sortedTracks` Derivation
**Severity**: Low
**Files**: `frontend/src/pages/Tracks.tsx`
**Problem**: Track sorting and filtering logic re-runs on every render including unrelated state changes.
**Fix**: Wrap in `useMemo`.
---
### P12 — `useVisualizerPalette` Extracts Color from Image on Every Hook Call
**Severity**: Low
**Files**: `frontend/src/hooks/useVisualizerPalette.ts`
**Problem**: Image color extraction (`extractSeedFromImage`) fires on every `songId`/`coverArtUrl` change, even if the same song is re-mounted (e.g., after route navigation). Cache results by `coverArtUrl`.
---
## 4. Code Quality & Edge Cases
### C1 — `quality` Prop Stale Closure in MuonVisualizerCanvas
**Severity**: High
**Files**: `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
**Problem**: `quality` is captured at `useEffect` creation time. Even though it's in the dep array and the effect re-runs, within the RAF loop closure the stale `quality` value persists until the next effect cycle. Audio-reactive quality changes (e.g., `quality='low'` during drops) may miss frames.
**Fix**: Use a `qualityRef`:
```ts
const qualityRef = useRef(quality)
qualityRef.current = quality // unconditional update each render
// RAF loop reads qualityRef.current, not captured quality
```
---
### C2 — `initFluidSim` Unmount Race Condition
**Severity**: High
**Files**: `frontend/src/components/visualizers/fluid/fluidSim.ts:91–102`
**Problem**: The async import resolves after `destroyFluidSim` is called. The token check at line 99 prevents `instances.set`, but `WebGL(canvas, config)` at line 97 still executes on the destroyed canvas. This can re-initialize WebGL on a canvas that the React lifecycle has already detached, causing a `WEBGL_lose_context` + re-attach race.
**Fix**: Move the token check before the `WebGL()` call:
```ts
.then((WebGL) => {
if (initTokens.get(canvas) !== initToken) return // already destroyed
if (!document.contains(canvas)) return
WebGL(canvas, config)
if (initTokens.get(canvas) !== initToken) return // destroyed during init
instances.set(canvas, { canvas, config })
})
```
---
### C3 — Partial Zeroing on Pause Leaves Stale `beatPhase` / `isBeatDrop`
**Severity**: High
**Files**: `frontend/src/hooks/useAudioFeatures.ts`
**Problem**: When `!isPlayingRef.current`, only some audio feature fields are zeroed. `beatPhase`, `isBeatDrop`, and `energyDelta` retain their last non-zero values. Visualizers that use these fields (e.g., beat-reactive scale pulses) continue animating after playback stops.
**Fix**: In the pause-gate return path, emit a fully zeroed `AudioFeatures` object. Define a `ZERO_AUDIO_FEATURES` const and return it.
---
### C4 — Zero Error Boundaries in Entire Frontend
**Severity**: High
**Files**: Entire `frontend/src/` — no `ErrorBoundary` anywhere
**Problem**: A WebGL context loss, shader compile failure, or uncaught exception in any visualizer or component crashes the entire React tree with a white screen. No `componentDidCatch` / `getDerivedStateFromError` exists anywhere.
**Fix**: Add `ErrorBoundary` around at minimum: each visualizer (`<Suspense>` wrappers in `VisualizerBackdrop.tsx`), `LyricsLivePreview`, and the top-level App router outlet.
---
### C5 — Particle Pool Slot Overwrite Without Bounds Check
**Severity**: Medium
**Files**: `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
**Problem**: When spawning new particles the pool index is incremented modulo pool size, but if spawn rate exceeds pool capacity in a single frame, multiple particles in the same tick may share the same slot. The older particle's state is silently overwritten.
**Fix**: Clamp the number of new spawns per frame to at most `Math.floor(poolSize / minLifetimeFrames)`.
---
### C6 — Denominator Guard Missing in Spectral Centroid Calculation
**Severity**: Medium
**Files**: `frontend/src/hooks/useAudioFeatures.ts`
**Problem**: `spectralCentroid = sumWeighted / sumTotal` — if all frequency bins are `-Infinity` (silence), `sumTotal = 0` and the result is `NaN`. `NaN` propagates into GLSL uniforms where it becomes `0.0` on most drivers but is undefined behaviour.
**Fix**: `spectralCentroid = sumTotal > 0 ? sumWeighted / sumTotal : 0`.
---
### C7 — Audio `'error'` Event Not Handled
**Severity**: Medium
**Files**: `frontend/src/components/LyricsLivePreview.tsx`
**Problem**: `audioRef.current` has `onEnded`, `onTimeUpdate`, `onCanPlay` listeners but no `'error'` listener. A failed stream or decode error silently stalls playback with no UI feedback.
**Fix**: Add `audioRef.current.addEventListener('error', handleAudioError)` with a state update to show an error banner.
---
### C8 — `localStorage.setItem` Unchecked — Throws in Private Browsing / Full Storage
**Severity**: Medium
**Files**: `frontend/src/components/LyricsLivePreview.tsx` (StyleState persistence)
**Problem**: `localStorage.setItem(...)` throws `QuotaExceededError` when storage is full or in private browsing mode on some browsers. The exception propagates uncaught through the React event handler, crashing the component.
**Fix**: Wrap all `localStorage.setItem` calls in try/catch.
---
### C9 — Spirit/Tendrils GPGPU Textures Not Verified for `RGBA32F` Support
**Severity**: Medium
**Files**: `frontend/src/components/visualizers/TendrilsVisualizerCanvas.tsx`, `SpiritVisualizerCanvas.tsx`
**Problem**: Both visualizers create `RGBA32F` render targets for GPGPU particle simulation. In WebGL2, `OES_texture_float` is core but `EXT_color_buffer_float` (required to render *into* a float texture) must be explicitly checked. If unsupported, `gl.checkFramebufferStatus` returns `FRAMEBUFFER_INCOMPLETE_ATTACHMENT` and all particle simulation produces black.
**Fix**: Check extension availability at init time:
```ts
const ext = gl.getExtension('EXT_color_buffer_float')
if (!ext) { /* fallback to HALF_FLOAT or disable visualizer */ }
```
---
### C10 — `createSynthForProgram` Returns `{}` for Unrecognised MIDI Program Numbers
**Severity**: Low
**Files**: `frontend/src/pages/ProjectDetail.tsx`
**Problem**: The `default` branch returns `{}` which is not a valid Tone.js synth. Calling `.connect()` or `.triggerAttackRelease()` on it throws.
**Fix**: Return a `new Tone.PolySynth(Tone.Synth)` as a safe default.
---
### C11 — `selectedLineIdxRef.current!` Non-Null Assertion in Drag Handler
**Severity**: Low
**Files**: `frontend/src/components/KaraokeEditor.tsx:389,404`
**Problem**: Non-null assertion inside `onPointerMove`/`onPointerUp` handlers with no guard. If a drag begins while `selectedLineIdxRef.current` is `null` (race between drag start and line deselection), `linesRef.current[null]` is `undefined`, and word timestamps are set to `NaN`.
**Fix**: Early return if `selectedLineIdxRef.current === null`.
---
### C12 — `svgRef.current!` Non-Null Assertion Across Async Boundary
**Severity**: Low
**Files**: `frontend/src/pages/ProjectDetail.tsx:1774`
**Problem**: `svgRef.current` is guarded synchronously but then asserted non-null inside a `.then()` callback. The component may unmount before the dynamic import resolves.
**Fix**: Capture `const container = svgRef.current` before the `import()` call; check `container.isConnected` inside `.then()`. *(Also tracked as TS10.)*
---
### C13 — Omnizart Sidecar HTTP Call Has No Mid-Transfer Cancellation
**Severity**: Low
**Files**: `backend/services/omnizart_service.py:108`
**Problem**: Synchronous `httpx.Client` POST with 1800s timeout; no cancel check during transfer; full response body buffered in RAM before writing to disk.
**Fix**: Stream with chunked reads and `jm.should_cancel(job_id)` poll between chunks. *(Also tracked as PY6.)*
---
## 5. Python Backend
### PY1 — Bare `except:` in Temp-Dir Cleanup — 24 Occurrences
**Severity**: Medium
**Files**: `backend/services/separator_service.py` (22 sites), `backend/routers/lrc.py:58`, `backend/services/ensemble_separator.py:213`
**Problem**: Every `_run_*` separator function wraps `shutil.rmtree(temp_out, ignore_errors=True)` in a `try: ... except: pass`. `except:` catches `BaseException` including `KeyboardInterrupt`, `SystemExit`, and `GeneratorExit`. Since `ignore_errors=True` already suppresses all `OSError`, the outer try/except is redundant and dangerous — it masks real bugs (e.g., `TypeError` from a `None` path) and can swallow shutdown signals.
**Fix**: Remove all 24 wrapping try/except blocks. `shutil.rmtree(path, ignore_errors=True)` is already safe.
---
### PY2 — Blocking File I/O in `async` FastAPI Handlers
**Severity**: High
**Files**: `backend/routers/separate.py:268`, `backend/routers/transcribe.py:51`, `backend/routers/lrc.py:40,90,131,136`, `backend/routers/pipeline.py:62`, `backend/routers/lyric_transcribe.py:319,477`, `backend/routers/abc_convert.py:30`
**Problem**: All `async` upload handlers write the incoming file to disk using synchronous `shutil.copyfileobj()` without offloading to a thread pool. For files up to 500 MB (`MAX_UPLOAD_BYTES`), this stalls the uvicorn event loop for the entire duration of the write.
**Fix**: Use async I/O:
```python
import anyio
async with await anyio.open_file(upload_path, "wb") as f:
while chunk := await file.read(65536):
await f.write(chunk)
```
---
### PY3 — Unsanitised `file.filename` — Path Traversal (13 Sites)
**Severity**: High
**Files**: Same as SEC2 + `backend/routers/abc_convert.py`
**Problem**: Same root cause as SEC2. 13 total call sites across 7 files.
**Fix**: Same `_safe_filename()` helper. *(Unified fix with SEC2.)*
---
### PY4 — `separate_existing` Uses General Executor for GPU Work
**Severity**: High
**Files**: `backend/routers/separate.py:419`
**Problem**: `/api/separate/existing` calls `jm.submit(...)` instead of `jm.submit_gpu(...)`. The GPU executor serializes jobs to prevent VRAM contention. The general executor has 4 workers. Multiple concurrent calls to `separate_existing` trigger simultaneous GPU-intensive jobs, causing CUDA OOM.
**Fix**: One-line change:
```python
# Before
jm.submit(run_separation, job_id, ...)
# After
jm.submit_gpu(run_separation, job_id, ...)
```
---
### PY5 — `urlretrieve` for Model Downloads — No Timeout, Non-Atomic, Deprecated
**Severity**: Medium
**Files**: `backend/services/separator_service.py:123,126`, `backend/services/scnet_separator.py:269,280`, `backend/services/mdx23c_separator.py`
**Problem**: `urllib.request.urlretrieve()` has been soft-deprecated since Python 3.4. No timeout means a stalled connection hangs the separator thread indefinitely. On interruption, a partial (corrupt) model file remains on disk; subsequent runs find the file and skip downloading, then load the corrupt checkpoint silently.
**Fix**: Atomic download using httpx (already a dependency):
```python
import httpx, tempfile
def _download_file(url: str, dest: Path, *, timeout: int = 600) -> None:
with tempfile.NamedTemporaryFile(dir=dest.parent, delete=False, suffix=".tmp") as tmp:
tmp_path = Path(tmp.name)
try:
with httpx.stream("GET", url, follow_redirects=True, timeout=timeout) as r:
r.raise_for_status()
for chunk in r.iter_bytes(65536):
tmp.write(chunk)
tmp_path.rename(dest) # atomic: visible only on success
except Exception:
tmp_path.unlink(missing_ok=True)
raise
```
---
### PY6 — Omnizart Sidecar: No Cancel Check During 30-Min HTTP Call
**Severity**: Medium
**Files**: `backend/services/omnizart_service.py:108`
**Problem**: Synchronous `httpx.Client` POST with 1800s timeout; full response buffered into RAM; no cancellation check during transfer. Thread-pool thread stuck for up to 30 min with no way to abort. *(Also tracked as C13.)*
**Fix**: Stream with cancel checks between chunks (see C13 fix).
---
### PY7 — Demucs Cancel Check Blocked by Synchronous `readline()`
**Severity**: Low
**Files**: `backend/services/separator_service.py:1539–1553`
**Problem**: Cancellation check only runs between lines of Demucs stdout output. During quiet processing periods, `readline()` blocks and cancel requests are delayed for many seconds. The `time.sleep(0.1)` line is dead code (unreachable after `readline()` blocks).
**Fix**: Use `select` with a 1-second timeout for non-blocking reads; poll `should_cancel()` each second.
---
### PY8 — `datetime.utcnow()` Deprecated Since Python 3.12
**Severity**: Low
**Files**: `backend/db.py:50,51,60,82,83,173,196,219`, `backend/repositories/jobs.py:80,96`, `backend/job_manager.py:317`
**Problem**: `datetime.utcnow()` produces naive datetimes and is deprecated in Python 3.12; removed in 3.14.
**Fix**: Replace all with `datetime.now(timezone.utc)`.
---
### PY9 — `print()` in Production Service Code — 30+ Occurrences
**Severity**: Low
**Files**: `backend/services/sepacap_separator.py`, `mdx23c_separator.py`, `scnet_separator.py`, `sepreformer_separator.py`, `apollo_separator.py`, `ensemble_separator.py`, `medleyvox_separator.py`, `aspiration_separator.py`, `bandit_separator.py`, `mel_band_specialized_separator.py`
**Problem**: Progress and error messages use `print()` instead of the structured logging system, making them invisible to log aggregators and unsuppressible.
**Fix**: Add `log = logging.getLogger(__name__)` and replace `print(...)` with `log.info(...)` / `log.debug(...)`.
---
### PY10 — `separate_existing` Source File Discovery Uses Wrong File
**Severity**: Medium
**Files**: `backend/routers/separate.py:374–399`
**Problem**: When the project directory doesn't contain the original source file, the code scans all upload directories by mtime and returns the most recently uploaded file from **any** job. This silently re-separates the wrong audio file whenever any other upload occurred more recently. Performance also degrades linearly as uploads accumulate.
**Fix**: Look up the original source path from the separation job record rather than scanning the filesystem.
---
### PY11 — Global `torch.load` Monkey-Patch Disables Pickle Safety Process-Wide
**Severity**: Medium
**Files**: `backend/services/lyric_transcriber_service.py:42–55`
**Problem**: A global monkey-patch unconditionally overrides `torch.load` for the entire process, forcing `weights_only=False`. `weights_only=False` enables arbitrary Python object deserialization. PyTorch's CVE-2025-32434 explicitly identifies this as the RCE vector. The patch is active by default (`STEMENGINE_PATCH_TORCH_LOAD` env var defaults to `"1"`), meaning every model loaded anywhere in the process is unsafe.
**Fix**: Remove the global patch. Use `torch.serialization.safe_globals()` with an explicit allowlist for the specific classes that pyannote/WhisperX require. At minimum, change the env-variable default to `"0"` (opt-in).
---
### PY12 — `JobResponse.job_id` Mismatches DB `id` Field
**Severity**: Low
**Files**: `backend/routers/jobs.py:18–31`, `backend/repositories/jobs.py:139`
**Problem**: `JobResponse` declares `job_id: str` but `to_dict()` returns `"id"`. The model is never enforced (`response_model` not applied), so this is misleading dead documentation. Applying `response_model=JobResponse` would immediately break.
**Fix**: Align field name to `id: str` or apply `Field(alias="job_id")` with `model_config = ConfigDict(populate_by_name=True)`.
---
## 6. TypeScript Type Safety
### TS1 — MIDI Object Graph Fully Untyped
**Severity**: Critical
**Files**: `frontend/src/pages/ProjectDetail.tsx:120–126, 231, 260–275, 290, 345–353`
**Problem**: `midiList: Array<{midi: any; stemIdx: number}>`. All MIDI tracks, notes, and Tone.js refs (`ToneRef`, `synthRefs`, `partRefs`, `volRefs`) are `useRef<any>()` or `Record<number, any>`. `createSynthForProgram` accepts and returns `any`. Property typos produce `undefined` silently; `note.velocity * 0.55` renders `NaN` if `velocity` is absent.
**Fix**: Define `MidiNote`, `MidiTrack`, `MidiFile` interfaces. Import Tone.js types: `PolySynth`, `Part`, `Volume`, `Sampler`.
---
### TS2 — `import.meta` Cast to `any` for Vite Env Variables
**Severity**: High
**Files**: `frontend/src/api/client.ts:51`, `frontend/src/App.tsx:324`
**Problem**: `(import.meta as any).env?.VITE_STEMENGINE_API_BASE` — custom env vars missing from `vite-env.d.ts`.
**Fix**: Extend `vite-env.d.ts` with the `ImportMetaEnv` interface declaring both variables.
---
### TS3 — `AppSettings` Preset Fields Typed as `unknown[]`
**Severity**: High
**Files**: `frontend/src/api/client.ts:977–983`
**Problem**: `auto_steps: unknown[]`, `auto_presets: unknown[]` despite `AutoStep` and `AutoPreset` types being defined in `utils/autoPipeline.ts`. Causes redundant intersection casts in `useConvertStatus`.
**Fix**: Import and use `AutoStep[]` / `AutoPreset[]`.
---
### TS4 — `createSynthForProgram` Accepts/Returns `any`
**Severity**: High
**Files**: `frontend/src/pages/ProjectDetail.tsx:231,248`
**Problem**: Tone.js constructor signature changes produce silent runtime failures.
**Fix**: Type as `(Tone: typeof import('tone'), program: number): PolySynth | Sampler`.
---
### TS5 — `applyClipPath` / `applyFilter` Accept `any`
**Severity**: High
**Files**: `frontend/src/utils/svgUtils.ts:153,159`
**Problem**: Both functions accept `any`. Actual shape is always `{ _renderer?: { elem?: SVGElement } }`. Same internal Two.js cast done ~12 more times inline in `LyricsLivePreview.tsx`.
**Fix**: Define and export `TwoRendererElement` interface; use it throughout.
---
### TS6 — `fluidSim.ts` Dynamic Import Typed as `any`
**Severity**: Medium
**Files**: `frontend/src/components/visualizers/fluid/fluidSim.ts:1–15`
**Problem**: `let WebGLFluid: any` — `WebGL(canvas, config)` call cannot be type-checked.
**Fix**: Define `type WebGLFluidFn = (canvas: HTMLCanvasElement, options?: Partial<FluidConfig>) => void` and use throughout.
---
### TS7 — Two.js Internal Renderer Properties Cast as `any` (~30 Sites)
**Severity**: Medium
**Files**: `frontend/src/components/LyricsLivePreview.tsx` (30+ occurrences)
**Problem**: Two.js undocumented properties (`.renderer.defs`, `._renderer.elem`, `.scale`, `.fill`, `.noStroke()`) all accessed via `as any`.
**Fix**: Add `src/types/two-internals.d.ts` with a module augmentation declaring the undocumented surface once.
---
### TS8 — Vendor-Prefixed CSS Properties Cast as `any`
**Severity**: Medium
**Files**: `frontend/src/components/LyricsLivePreview.tsx:4818–4874, 4927`
**Problem**: `(el.style as any).textRendering`, `(el.style as any).webkitFontSmoothing`, etc.
**Fix**: Use `el.style.setProperty('text-rendering', 'geometricPrecision')` for DOM manipulation (no cast). Define `ExtendedCSSProperties extends React.CSSProperties` for inline styles.
---
### TS9 — Non-Null Assertions on `canvas!` and `selectedLineIdxRef.current!`
**Severity**: Medium
**Files**: `frontend/src/components/KaraokeEditor.tsx:277,302,342,353,378,389,395,404`
**Problem**: Six `canvas!` assertions inside nested closures (TypeScript widens back to `null` inside nested functions). `selectedLineIdxRef.current!` in drag handlers with no null guard — produces `NaN` word timestamps if race occurs. *(Also tracked as C11.)*
**Fix**: Capture `const canvas = canvasRef.current` before inner closures. Add `if (lineIdx === null) return` guard in drag handlers.
---
### TS10 — Non-Null Assertion on `svgRef.current!` Across Async Boundary
**Severity**: Medium
**Files**: `frontend/src/pages/ProjectDetail.tsx:1774`
**Problem**: Guard is synchronous but `renderAbc(svgRef.current!, ...)` is inside a `.then()` callback — component may have unmounted. *(Also tracked as C12.)*
**Fix**: Capture ref before `import()`; check `.isConnected` inside `.then()`.
---
### TS11 — Untyped `useQuery` with Internal `as Job[]` Cast
**Severity**: Medium
**Files**: `frontend/src/components/Sidebar.tsx:629–638`
**Problem**: `useQuery` without type parameter; internal cast `as Job[] | undefined` despite `listJobs` returning `Promise<Job[]>`.
**Fix**: `useQuery<Job[]>({ ... })` — cast removed automatically.
---
### TS12 — Page-Level `useQuery` Calls Missing Type Parameters
**Severity**: Medium
**Files**: `frontend/src/pages/KaraokeTV.tsx:192–214,291–305`, `frontend/src/pages/Tracks.tsx:63–103`
**Problem**: `useQuery` without generic params; `getProjectDetail(projectId!)` non-null assertion inside `queryFn`.
**Fix**: Add `useQuery<ProjectDetail>(...)` and move assertion out of queryFn.
---
### TS13 — `CanvasSceneState` Non-Optional Fields Initialised as `null as any`
**Severity**: Low
**Files**: `frontend/src/components/LyricsLivePreview.tsx:5053–5054`
**Fix**: Make `linesGroup` and `bgRect` nullable: `InstanceType<typeof Two.Group> | null`.
---
### TS14 — `VisualizerBackdrop` Mode Switch Not Exhaustively Checked
**Severity**: Low
**Files**: `frontend/src/components/visualizers/VisualizerBackdrop.tsx:103–177`
**Problem**: If-chain is not exhaustive — new modes added to the union silently render `null`.
**Fix**: Convert to `switch` with `assertNever` for the default case. *(Superseded by architecture fix A1.)*
---
### TS15–TS18 — Missing Return Type Annotations, Wildcard `any` Module Declarations
**Severity**: Low
- **TS15**: `useProjectsIndex` missing return type — add `UseProjectsIndexResult` interface
- **TS16**: `useConvertStatus` missing return type and unexported `ConvertState` — export both
- **TS17**: `types/mjs-modules.d.ts` — wildcard `.mjs` typed as `any`; change to `unknown`
- **TS18**: `types/webgl-fluid.d.ts` — `options?: any`; import `FluidConfig` and use `Partial<FluidConfig>`
---
## 7. Architecture & Tech Debt
### A1 — VisualizerBackdrop Dispatch: 4-Site Edit Per New Visualizer
**Severity**: High
**Files**: `frontend/src/components/visualizers/VisualizerBackdrop.tsx:103–177`
**Problem**: Four sequential `if (mode === 'x')` blocks each manually duplicate the wrapper `<div>` + `<Suspense>`. There is an undocumented divergence: Tendrils and Spirit have `style={{ background }}` on their wrapper div; Muon and Fluid do not. Adding Visualizer 5 requires edits in 4+ files with no authoritative registry.
**Fix**: Create `visualizerRegistry.ts` — a `Record<VisualizerMode, VisualizerDescriptor>` with the lazy component, `needsBackgroundStyle` flag, and wrapper class. `VisualizerBackdrop` becomes a single generic render path.
---
### A2 — `VisualizerSettings`: Three Independent Type Definitions
**Severity**: High
**Files**: `VisualizerBackdrop.tsx:17–92`, `LyricsLivePreview.tsx:55–120`, `LyricsLivePreview.tsx:312–372`
**Problem**: Same per-visualizer settings shape defined three times with different optionality. A new parameter costs 6+ coordinated edits across 2 files. The inline prop type (`sekVisualizerSettings` at line 312) will permanently lag the other two.
**Fix**: Create `src/types/visualizerSettings.ts` with one `VisualizerSettings` interface, `VISUALIZER_DEFAULTS`, and `mergeVisualizerSettings()`. Delete the local `VizSettings` interface from `LyricsLivePreview`.
---
### A3 — Spirit Palette Init-Time Coupling Causes Visible Flash
**Severity**: Medium
**Files**: `frontend/src/components/visualizers/SpiritVisualizerCanvas.tsx:130–131`
**Problem**: `initSpiritSim` reads `paletteRef.current.speakerPalette` at init time for shader material colors. When `useVisualizerPalette` resolves asynchronously (50–200ms), the palette change triggers `destroySpiritSim` + `initSpiritSim`, causing a hard visual blank frame. Muon/Fluid/Tendrils read `paletteRef.current` inside the RAF loop and avoid this entirely.
**Fix**: Move `color1`/`color2` out of `initSpiritSim` and into `renderSpiritFrame` as per-frame `THREE.Color` uniform updates.
---
### A4 — `LyricsLivePreview` God Component (5,196 Lines, 41 `useEffect` Calls)
**Severity**: Critical
**Files**: `frontend/src/components/LyricsLivePreview.tsx`
**Problem**: Simultaneously owns AudioContext lifecycle, GainNode management, transport controls, LRC parsing, Two.js canvas scene (~600 lines), video backdrop, visualizer settings normalization, `StyleState` (30+ fields), speaker diarization, `KaraokeWord` renderer, `StylePicker` (400+ lines), and `LrcEditor`. `setCurrentTime` from RAF causes React reconciliation at 60fps. `styleRebuildKey` JSON-serializes 24 properties every render.
**Staged fix**:
1. Extract `useAudioEngine` (AudioContext + GainNodes + stem mixing)
2. Extract `useLyricsCanvas` (Two.js scene)
3. Move `StylePicker`, `KaraokeWord`, `SpeakerDot`, `LrcEditor` to `components/karaoke/`
---
### A5 — Vendored `webgl-fluid.mjs` Diverges from npm Package
**Severity**: High
**Files**: `frontend/src/components/visualizers/fluid/webgl-fluid.mjs`, `package.json`
**Problem**: The vendored file has a `_fluidSplat` property injection at line 1808 not present in the upstream npm package. `package.json` also lists `webgl-fluid@^0.3.9`. If npm resolution wins (e.g., after `npm ci`), `_fluidSplat` is `undefined` and all beat-reactive splat injection silently no-ops. Additionally, `updateFluidConfig` uses `Object.assign` on the stored config reference — if the library copies config at init time, all per-frame parameter changes are silently ignored.
**Fix**:
1. Remove `webgl-fluid` from `package.json` (the vendored file is authoritative)
2. Add startup assertion verifying `_fluidSplat` is set after `WebGL()` resolves
3. Add comment block at line 0 of `webgl-fluid.mjs` documenting the modification
---
### A6 — Utility Function Duplication Across All Visualizer Files
**Severity**: Medium
**Files**: `MuonVisualizerCanvas.tsx`, `FluidVisualizerCanvas.tsx`, `TendrilsVisualizerCanvas.tsx`, `SpiritVisualizerCanvas.tsx`
**Problem**: `clamp01`, `clamp`, and hex-to-RGB converters independently defined in every file — with different return shapes (`{r,g,b}` unit floats vs `{r,g,b}` 0–255 vs `[r,g,b]` tuple). TypeScript cannot detect passing the wrong variant.
**Fix**: Create `components/visualizers/vizUtils.ts` exporting `clamp`, `clamp01`, `hexToRgbUnit`, `hexToRgb255`, `blendColor`, `hueRotate`. All four canvas files import from it.
---
### A7 — Spirit Lifecycle Divergence: Full Reinit on Resize
**Severity**: Medium
**Files**: `frontend/src/components/visualizers/SpiritVisualizerCanvas.tsx:104–227`
**Problem**: `width`/`height` in `initAndStart` `useCallback` deps causes full Three.js renderer recreation on every resize, panel toggle, or sidebar open (~50ms blank frame on integrated GPU). Muon/Fluid/Tendrils handle resize imperatively inside the RAF loop. *(Also tracked as R1.)*
**Fix**: Remove `width`/`height` from deps; handle resize inside `renderSpiritFrame`. Assign `settingsRef.current = settings` synchronously at render time (not in a separate `useEffect`).
---
### A8 — `useConvertStatus` — Uncancellable Async Polling Loops
**Severity**: Medium
**Files**: `frontend/src/hooks/useConvertStatus.ts`
**Problem**: `pollJobUntilTerminal` uses an infinite `for(;;)` loop with no `AbortSignal` and no cleanup. Component unmount leaves polling running indefinitely. `convertSong` callback contains 100 lines of service-layer orchestration (localStorage, API calls, job submission) in a React hook.
**Fix**: Extract orchestration to `services/conversionOrchestrator.ts`. Thread an `AbortController` signal; store controller in `useRef`; call `.abort()` on cleanup.
---
### A9 — `updateFluidConfig` Mutation Correctness Unverified
**Severity**: Medium
**Files**: `frontend/src/components/visualizers/fluid/fluidSim.ts:105–109`
**Problem**: `Object.assign(inst.config, patch)` mutates the stored config reference. If the vendored library copied config at init time, all per-frame audio-reactive parameter changes are silently dropped and the fluid sim runs at fixed parameters regardless of audio input. Cannot be verified without reading 1,813 lines of minified code. *(Related to A5.)*
**Fix**: Add a smoke test: call `updateFluidConfig(canvas, { PAUSED: true })` and verify the sim pauses. Document result as a comment.
---
### A10 — Three Redundant Visualizer-Mode Membership Checks
**Severity**: Low
**Files**: `frontend/src/components/LyricsLivePreview.tsx:47,1861,5178`
**Problem**: `'muon' | 'fluid' | 'tendrils' | 'spirit'` enumerated three times independently. Adding a fifth visualizer requires finding and updating all three; missing one causes subtle behavioral bugs (wrong audio analysis path, canvas fade applied over WebGL).
**Fix**: Export `VISUALIZER_MODES = new Set([...])` from visualizer registry (prerequisite: A1); replace all three checks with `VISUALIZER_MODES.has(visualStyle)`.
---
### A11 — `useConvertStatus` Single-Responsibility Violation
**Severity**: Low
**Files**: `frontend/src/hooks/useConvertStatus.ts`
**Problem**: 267 lines; mixes React state management with service orchestration. All other hooks are appropriately scoped. *(See A8 for fix.)*
---
### A12 — Cost of Adding a 5th Visualizer Under Current Architecture
**Severity**: Advisory
**Problem**: Under current architecture, adding a fifth visualizer requires 8+ coordinated edits across 2 files (`VisualizerBackdrop.tsx`, `LyricsLivePreview.tsx`) plus the new canvas file. After implementing A1 + A2 + A6, adding a new visualizer requires only the new canvas file plus one registry entry.
**Recommendation**: Treat A6 → A1 → A2 as a prerequisite sprint before implementing Visualizer 5.
---
## Appendix: Finding Index
| ID | Domain | Severity | Title |
|----|--------|----------|-------|
| SEC1 | Security | High | No auth on ML endpoints |
| SEC2 | Security | High | Path traversal via `file.filename` |
| SEC3 | Security | Medium | File size not enforced |
| SEC4 | Security | Medium | SSRF via yt-dlp |
| SEC5 | Security | Medium | CORS wildcard default |
| SEC6 | Security | Medium | Cross-job file access |
| SEC7 | Security | Medium | `project_name` path traversal |
| SEC8 | Security | Low | Vendored unaudited JS |
| SEC9 | Security | Low | Missing security headers |
| SEC10 | Security | Low | No rate limiting |
| SEC11 | Security | Low | Hardcoded fallback token |
| R1 | React | High | Spirit full reinit on resize |
| R2 | React | High | Audio node teardown on fader drag |
| R3 | React | High | Spurious onPlaybackEnded |
| R4 | React | High | `applyGains` teardown on volume tick |
| R5 | React | Medium | Context actions not memoized |
| R6 | React | Medium | Re-renders at audio tick rate |
| R7 | React | Medium | ResizeObserver cleanup missing |
| R8 | React | Medium | `settingsRef` synced via effect (Spirit) |
| R9 | React | Medium | Stale beatPhase/isBeatDrop when paused |
| R10 | React | Medium | No ErrorBoundary on visualizers |
| R11 | React | Low | Sidebar unconditional polling |
| R12 | React | Low | `styleRebuildKey` JSON-serialize every render |
| R13 | React | Low | Two.js RAF runs hidden |
| R14 | React | Low | MuonFeatureExtractor runs when not playing |
| P1 | Performance | Critical | No Page Visibility guard in any visualizer |
| P2 | Performance | High | No IntersectionObserver |
| P3 | Performance | High | FFT analysis runs when paused |
| P4 | Performance | High | 3 Float32Array allocs/frame |
| P5 | Performance | High | 1,600 createRadialGradient/frame |
| P6 | Performance | Medium | AudioContext not suspended |
| P7 | Performance | Medium | Two.js RAF runs hidden |
| P8 | Performance | Medium | Sidebar unconditional polling |
| P9 | Performance | Medium | `deepMergeVizSettings` not memoized |
| P10 | Performance | Low | KaraokeTV unconditional polling |
| P11 | Performance | Low | `sortedTracks` not memoized |
| P12 | Performance | Low | `extractSeedFromImage` not cached |
| C1 | Code Quality | High | `quality` stale closure |
| C2 | Code Quality | High | initFluidSim unmount race |
| C3 | Code Quality | High | Stale beatPhase/isBeatDrop on pause |
| C4 | Code Quality | High | Zero ErrorBoundaries |
| C5 | Code Quality | Medium | Particle pool overwrite |
| C6 | Code Quality | Medium | Spectral centroid divide-by-zero |
| C7 | Code Quality | Medium | Audio error event not handled |
| C8 | Code Quality | Medium | localStorage.setItem unchecked |
| C9 | Code Quality | Medium | RGBA32F extension not verified |
| C10 | Code Quality | Low | `createSynthForProgram` returns `{}` |
| C11 | Code Quality | Low | Non-null assertion in drag handler |
| C12 | Code Quality | Low | Non-null assertion across async boundary |
| C13 | Code Quality | Low | Omnizart no cancel check |
| PY1 | Python | Medium | Bare `except:` in cleanup (24 sites) |
| PY2 | Python | High | Blocking I/O in async handlers |
| PY3 | Python | High | `file.filename` path traversal (13 sites) |
| PY4 | Python | High | `separate_existing` uses wrong executor |
| PY5 | Python | Medium | `urlretrieve` — no timeout, non-atomic |
| PY6 | Python | Medium | Omnizart no cancel during 30-min call |
| PY7 | Python | Low | Demucs cancel blocked by `readline()` |
| PY8 | Python | Low | `datetime.utcnow()` deprecated |
| PY9 | Python | Low | `print()` in production services (30+) |
| PY10 | Python | Medium | `separate_existing` uses wrong file |
| PY11 | Python | Medium | Global torch.load patch disables safety |
| PY12 | Python | Low | `JobResponse.job_id` field mismatch |
| TS1 | TypeScript | Critical | MIDI graph fully untyped |
| TS2 | TypeScript | High | `import.meta as any` for env vars |
| TS3 | TypeScript | High | AppSettings preset fields `unknown[]` |
| TS4 | TypeScript | High | `createSynthForProgram` returns `any` |
| TS5 | TypeScript | High | `applyClipPath`/`applyFilter` accept `any` |
| TS6 | TypeScript | Medium | fluidSim dynamic import typed as `any` |
| TS7 | TypeScript | Medium | Two.js internals `as any` (30 sites) |
| TS8 | TypeScript | Medium | Vendor CSS cast as `any` |
| TS9 | TypeScript | Medium | Non-null assertions in closures |
| TS10 | TypeScript | Medium | Non-null assertion across async boundary |
| TS11 | TypeScript | Medium | Untyped useQuery with internal cast |
| TS12 | TypeScript | Medium | Page-level useQuery missing type params |
| TS13 | TypeScript | Low | CanvasSceneState `null as any` init |
| TS14 | TypeScript | Low | VisualizerBackdrop not exhaustive |
| TS15 | TypeScript | Low | useProjectsIndex missing return type |
| TS16 | TypeScript | Low | useConvertStatus missing return type |
| TS17 | TypeScript | Low | Wildcard .mjs typed as `any` |
| TS18 | TypeScript | Low | webgl-fluid.d.ts `options?: any` |
| A1 | Architecture | High | VisualizerBackdrop 4-site edit per mode |
| A2 | Architecture | High | VisualizerSettings 3 independent definitions |
| A3 | Architecture | Medium | Spirit palette init-time flash |
| A4 | Architecture | Critical | LyricsLivePreview god component |
| A5 | Architecture | High | Vendored webgl-fluid diverges from npm |
| A6 | Architecture | Medium | Utility duplication across visualizers |
| A7 | Architecture | Medium | Spirit resize lifecycle divergence |
| A8 | Architecture | Medium | useConvertStatus uncancellable polling |
| A9 | Architecture | Medium | updateFluidConfig propagation unverified |
| A10 | Architecture | Low | Three redundant mode-membership checks |
| A11 | Architecture | Low | useConvertStatus SRP violation |
| A12 | Architecture | Advisory | Cost of 5th visualizer pre/post refactor |
+261
View File
@@ -1,5 +1,233 @@
# 2026-03-13 Backend Fixes / Improvement Plan (Copilot)
## 2026-04-13 Work Start — SingUI pipeline mode setting + project-first karaoke
Implemented first delivery slice from `2026-04-13_SingUI_Pipeline_and_Karaoke_Source_Plan.md`.
### Implemented
1) Server-persisted SingUI process mode
- Added `singui_process_mode` to backend settings defaults and patch model.
- Added validation with accepted values:
- `auto`
- `separate_only`
- `pipeline`
- `preset`
- Added optional persisted selector `singui_process_preset`.
File:
- `backend/routers/settings.py`
2) Settings UI control for SingUI package behavior
- Added control under Settings → Auto Pipeline for selecting/saving SingUI behavior.
- Save persists via `/api/settings` patch.
Files:
- `frontend/src/pages/Settings.tsx`
- `frontend/src/api/client.ts` (typed field in `AppSettings`)
3) Conversion flow now respects selected mode
- `useConvertStatus` now fetches app settings and branches:
- `auto`: run full configured Auto Pipeline additional steps (existing behavior)
- `separate_only`: stop after separation/project creation
2026-04-13 update (step 1 completion):
- Added additional execution branches in `useConvertStatus`:
- `pipeline`: submits `/api/pipeline` (audio pulled from `/api/subsonic/stream/{songId}` and sent as upload file)
- `preset`: runs separation with selected preset model/format/instrument-models, then executes preset step list
- Refactored repeated post-separation logic into a shared `runAdditionalSteps(...)` helper inside the hook.
2026-04-13 update (step 2 completion):
- Settings UI mode list now exposes all four process modes:
- `auto`, `separate_only`, `pipeline`, `preset`
- Added conditional preset selector UI when `preset` mode is chosen.
- Save action now persists both:
- `singui_process_mode`
- `singui_process_preset`
- Extended frontend app settings typing to include:
- `singui_process_mode?: 'auto' | 'separate_only' | 'pipeline' | 'preset'`
- `singui_process_preset?: string`
File:
- `frontend/src/hooks/useConvertStatus.ts`
- `frontend/src/pages/Settings.tsx`
- `frontend/src/api/client.ts`
- `backend/routers/settings.py`
4) SingUI karaoke now prioritizes project path in project-linked mode
- Queue-item preference switched to linked project first.
- Project Sing action shown as the primary action when linked project exists.
- Action copy updated to `Prepare Project` for project-creation path.
2026-04-13 update (step 3 completion):
- Completed copy/UX wording cleanup across project-preparation actions in:
- Tracks page
- Albums page
- Sing page (track rows + album cards + project-linked status lines)
- Updated labels:
- `Package` → `Prepare Project`
- `Packaging` → `Preparing`
- `Packaged` → `Project Ready`
- Updated button tooltip text to reflect project preparation behavior and Settings-controlled mode.
2026-04-13 update (step 4 completion):
- Validation run completed after steps 1–3:
- `frontend`: `npm run -s typecheck` (pass)
- `frontend`: `npm run -s build` (pass)
- Documentation updates completed in plan + session logs.
2026-04-13 update (remaining optional item completion):
- Auto Pipeline preset persistence migrated to backend settings path:
- Settings preset save/delete now patch `auto_presets` server-side.
- Settings hydration now prefers server presets and mirrors local cache for resilience.
- One-time migration uploads local presets to backend when backend list is empty.
- Preset-mode execution (`useConvertStatus`) now prefers presets from `getAppSettings().auto_presets` with local fallback.
Files:
- `frontend/src/pages/Settings.tsx`
- `frontend/src/hooks/useConvertStatus.ts`
- `frontend/src/utils/autoPipeline.ts`
2026-04-13 planning update (git-managed default presets):
- Added architecture plan for storing default Auto Pipeline presets in Git and shipping them with StemEngine updates.
- Plan defines:
- in-repo catalog location,
- backend merge precedence (built-in + user presets),
- frontend source-label/edit semantics,
- rollout phases and acceptance criteria.
Plan file:
- `2026-04-13_Git_Managed_Auto_Pipeline_Presets_Plan.md`
2026-04-13 implementation update (git-managed default presets):
- Implemented built-in preset catalog in repo:
- `backend/presets/auto_pipeline/default_presets.json`
- Implemented backend catalog utilities:
- `backend/services/auto_presets_catalog.py`
- Extended settings API response with layered preset views:
- `auto_presets_user`
- `auto_presets_builtin`
- `auto_presets_effective`
(in `backend/routers/settings.py`)
- Updated frontend preset flows:
- Settings hydration/save/delete now works with backend effective/user preset split
- Preset-mode conversion execution now prefers backend effective presets
Files:
- `backend/routers/settings.py`
- `backend/services/auto_presets_catalog.py`
- `backend/presets/auto_pipeline/default_presets.json`
- `frontend/src/pages/Settings.tsx`
- `frontend/src/hooks/useConvertStatus.ts`
- `frontend/src/utils/autoPipeline.ts`
- `frontend/src/api/client.ts`
Validation:
- `frontend`: `npm run -s typecheck` (pass)
- `frontend`: `npm run -s build` (pass)
2026-04-13 implementation update (karaoke sidebar controls consolidation):
- Moved karaoke lyrics track buttons (`Leading` / `Backing` / `Main` when available)
from overlay/toolbar locations into the performance sidebar.
- Added line display mode controls to the performance sidebar:
- `Scroll`, `Line`, `Word`, `Syllable`
- `Word`/`Syllable` are disabled when ELRC timing data is unavailable.
- Wired sidebar props/state so track selection and display mode changes apply in real time.
- Removed duplicated in-canvas/toolbar lyrics track controls and cleaned up now-unused helper logic.
Files:
- `frontend/src/components/PerformanceSidebar.tsx`
- `frontend/src/components/LyricsLivePreview.tsx`
- `2026-04-13_Leading_Backing_Buttons_Sidebar_Refactor.md`
Validation:
- `frontend`: `npm run -s build` (pass)
2026-04-13 implementation update (PerformanceSidebar collapsible sections):
- Made sidebar control groups collapsible to reduce vertical scrolling in performance mode.
- Converted major groups to collapsible panels:
- `LYRICS TRACKS`
- `DISPLAY`
- `BACKGROUND`
- `STEMS`
- `LYRICS STYLE`
- `VISUALIZER`
- Kept primary sections open by default and heavier tuning sections collapsed by default
for faster access to common controls.
Files:
- `frontend/src/components/PerformanceSidebar.tsx`
Validation:
- `frontend`: `npm run -s build` (pass)
2026-04-13 implementation update (Fluid visualizer parity items F1/F3/F4/F5/F6/F7/F9/F10e):
- Implemented direct color/velocity splat path:
- Added `_fluidSplat` exposure marker in vendored `webgl-fluid.mjs`.
- Reworked `splatFluid` to call direct splat API (no synthetic mouse events).
- Preserved orientation by flipping Y in direct texcoord path.
- Added `fluidSim.ts` async init race guard via canvas init token checks.
- Updated Fluid init config in `FluidVisualizerCanvas.tsx`:
- Enabled `SUNRAYS` at init time.
- Added `BLOOM_SOFT_KNEE` init value.
- Added quality-aware `BLOOM_ITERATIONS`, `BLOOM_RESOLUTION`,
`SUNRAYS_RESOLUTION`, `PRESSURE_ITERATIONS`, and `SHADING`.
- Added preset-aware fluid runtime model:
- Introduced `FluidPreset` + `FLUID_PRESETS` table.
- Replaced hardcoded dissipation/curl/radius formulas with preset-aware,
audio-reactive `updateFluidConfig` values (including `PRESSURE`).
- Added `blendColor` and wired drop flash into `BACK_COLOR` update.
- Added preset-scaled force multipliers for ambient/burst/sweep splats.
- Added sweep orbit audio-reactivity + drop throttle bypass.
- Added `visualization_preset?: string` under
`VisualizerSettings['fluid']` in `VisualizerBackdrop.tsx`.
Files:
- `frontend/src/components/visualizers/fluid/webgl-fluid.mjs`
- `frontend/src/components/visualizers/fluid/fluidSim.ts`
- `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx`
- `frontend/src/components/visualizers/VisualizerBackdrop.tsx`
Validation:
- `frontend`: `npm run -s build` (pass)
File:
- `frontend/src/pages/Sing.tsx`
- `frontend/src/pages/Tracks.tsx`
- `frontend/src/pages/Albums.tsx`
### Follow-up queued
- Optional preset migration from localStorage to backend settings.
## 2026-04-07 Plan + Implementation — Full-library search, Track UX, Process pipeline assignment, lyric language fallback
### Plan (requested)
@@ -9781,6 +10009,39 @@ Validation:
- `npm run -s typecheck` ✅
### 2026-04-13 — Muon/Fluid refinement outstanding items completed
Completed remaining implementation items from `2026-04-13_Muon_VisualRefinement_Plan.md` and revalidated frontend build.
Implemented:
`frontend/src/components/visualizers/muon/noise.ts`
- Replaced hash-only scalar noise with smooth value noise (`smoothNoise2D`).
- Added divergence-style vector helper `curlNoise2D(...)` for coherent flow.
- Kept `centeredNoise2D(...)` compatibility export (now derived from smooth noise).
`frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
- Switched emitter noise from `centeredNoise2D` pair to `curlNoise2D` vector sampling.
- Added beat-cycle cooldown state (`lastPatternCycleAt`) and beat-driven `patternShift` increments.
- Upgraded drop behavior to larger pattern jump (`+2`) with cooldown reset.
- Changed emission topology to ring-first + circumference-distributed spots per ring.
- Added per-frame spawn guard (`maxTotalPerFrame`/`spotsCap`) to avoid pool overrun at high energy.
- Reduced launch speed and default lifespan baseline for cleaner ring structure.
- Replaced `shadowBlur` particle rendering with radial-gradient glow rendering.
- Added finite-coordinate guard before gradient creation to avoid NaN rendering faults.
- Included `quality` in effect dependencies so quality mode changes reinitialize cleanly.
Optional upstream check (plan item F11):
- Confirmed historical reference source is available at:
- `PavelDoGreat/WebGL-Fluid-Simulation` (`script.js`)
- The newer `PavelDoGreat/Fluid-Simulation` repository is a major rewrite (Zig/native-first project layout), not a drop-in replacement for the vendored browser JS bundle.
- No re-vendor performed in this pass; existing vendored `webgl-fluid.mjs` patch marker (`STEMENGINE_PATCH`) remains in place for future controlled upgrades.
Validation:
- `frontend`: `npm run -s typecheck` ✅
- `frontend`: `npm run -s build` ✅
### 2026-04-13 — Muon reference parity plan implemented
Requested:
+104
View File
@@ -0,0 +1,104 @@
# StemEngine — Implementation Checklist
Date: 2026-04-13
Plans: `2026-04-13_Muon_VisualRefinement_Plan.md`, `2026-04-13_Tendrils_Spirit_Visualizer_Plan.md`
---
## ✅ Completed (prior sessions)
- [x] SingUI process mode (`auto` / `separate_only` / `pipeline` / `preset`) — backend + frontend
- [x] Backend preset catalog (`backend/presets/auto_pipeline/default_presets.json` + `auto_presets_catalog.py`)
- [x] Frontend typecheck + build: pass
---
## Muon visual refinement
- [x] 1. Replace `muon/noise.ts` — smooth value noise + `curlNoise2D` export
- [x] 2. Add `t` parameter to `emitParticle`; update inner `emitOne` closure
- [x] 3. Replace emit loop — ring-first structure, circumference-spread `spotsPerRing`
- [x] 4. Lower launch velocity (`speed = 0.02 + …`) and default lifespan (→ 60)
- [x] 5. Smoke-test: all rings visible at zero audio energy *(pass)*
- [x] 6. Use `curlNoise2D` in `emitParticle` (import + noise call)
- [x] 7. Add `lastPatternCycleAt = 0` + beat-driven `patternShift` cycle with cooldown
- [x] 8. Remove `shadowBlur`; replace particle draw with per-particle radial gradient
- [x] 9. Wire `glowIntensity` into gradient radius **or** remove `glow` / `glowIntensity`
- [x] 10. Smoke-test Muon: all rings visible; soft falloff; curl flow; beat cycling *(pass)*
- [x] Add `quality` to `useEffect` deps in `MuonVisualizerCanvas.tsx`
- [x] Frontend typecheck + build after Muon changes *(pass)*
---
## Fluid visualizer reference parity
- [x] 11. `webgl-fluid.mjs` — attach `D._fluidSplat = fe` at end of `ft()` init (F1 Part A)
_Add STEMENGINE_PATCH marker comment for re-vendor diffs_
- [x] 12. `fluidSim.ts` — remove `_color` prefix; replace mouse-event dispatch with `canvas._fluidSplat` direct call (F1 Part B)
_Add typed local interface; flip Y (`1 - clamp01(y)`); preserve fallback silent-drop path_
- [x] 13. `fluidSim.ts` — add async init race guard (`initToken` / generation id)
- [x] 14. `FluidVisualizerCanvas.tsx` — init config: `SUNRAYS: true`, `SUNRAYS_RESOLUTION`, `BLOOM_SOFT_KNEE: 0.7`, `BLOOM_ITERATIONS`, `BLOOM_RESOLUTION`, `PRESSURE_ITERATIONS`, `SHADING: q !== 'low'` (F3, F4, F6, F7)
_F5/PRESSURE goes in step 16 as part of F9_
- [x] 15. Add `FLUID_PRESETS` table (`ink`, `lava`, `storm`, `mist`, `plasma`) + `FluidPreset` type; read `visualization_preset` from settings (F9)
- [x] 16. Replace `fluidDensityDissipation` / `fluidCurl` / etc. helper calls with preset-aware per-frame formulas; add `PRESSURE` modulation (F9)
_Delete the four helper functions once replaced_
- [x] 17. Add `blendColor` helper; wire drop flash into `BACK_COLOR` per-frame (F9, F10d)
_Avoid double-normalisation to near-black_
- [x] 18. Sweep orbit audio-reactivity (`orbit = 0.22 + kick * 0.10 + dropBoost * 0.08`) (F10b); drop throttle bypass when `dropBoost > 0.5` (F10c)
- [x] 19. `VisualizerBackdrop.tsx` — add `visualization_preset?: string` to `fluid` settings block (F10e)
- [x] 20. Smoke-test Fluid: Playwright e2e — mode mount, backdrop attribute `fluid`, palette fallback to default *(pass)*
_Deep visual checks (sunrays, bloom, drops, preset character) — deferred to live session_
- [x] 21. *(Optional)* Fetch newer `Fluid-Simulation` repo; diff / re-vendor if pipeline differs (F11)
_Re-apply `_fluidSplat` patch after re-vendoring_
- [x] Typecheck + build after Fluid changes
---
## Tendrils visualizer (steps 1–14)
- [x] 1. Create `frontend/src/types/glsl.d.ts` if missing
- [x] 2. Write `tendrils/screen.vert` (Section T5)
- [x] 3. Write `tendrils/logic.glsl` (Section T1) — GPGPU physics
- [x] 4. Write `tendrils/render.vert` (Section T2) — particle vertex
- [x] 5. Write `tendrils/render.frag` (Section T3) — particle colour
- [x] 6. Write `tendrils/flow.frag` (Section T4) — flow FBO fragment
- [x] 7. Write `tendrils/tendrilsSim.ts` (Section T6) — WebGL2 init / destroy / update
- [x] 8. Add context-loss / context-restore event listeners + float-target fallback chain (TE7, TE2)
- [x] 9. Sim exposes frame-step API only; component owns RAF + fpsCap (single clock owner)
- [x] 10. Write `TendrilsVisualizerCanvas.tsx` (Section T7) — React component + audio wiring (T8, T9)
- [x] 11. Add `tendrils` to `VisualizerSettings` + `VisualizerMode` in `VisualizerBackdrop.tsx`; add lazy import
- [x] 12. Smoke-test: Playwright e2e — canvas mounts and backdrop attribute is `tendrils` *(pass)*
- [x] 13. Smoke-test: Playwright e2e — palette inherited muon → tendrils → spirit *(pass)*
_Visual/audio quality checks (flow feedback, kick bursts, drop recovery, preset character) — deferred to live session_
- [x] 14. `typecheck` passes
---
## Spirit visualizer (steps 13–26)
- [x] 13. `npm install three` (adds `@types/three` automatically)
- [x] 14. Write `spirit/curl.glsl` (Section S1)
- [x] 15. Write `spirit/position.glsl` (Section S2) — prepend `curl.glsl` source
- [x] 16. Write `spirit/particle.vert` (Section S3)
- [x] 17. Write `spirit/particle.frag` (Section S4)
- [x] 18. Write `spirit/spiritSim.ts` (Section S5) — Three.js init / GPGPU / renderer / composer
- [x] 19. Enforce strict ping-pong read/write separation; never alias same RT (SE9)
- [x] 19b. Set `DataTexture.needsUpdate = true` for bootstrap position texture (SE10)
- [x] 20. Add context-loss / context-restore recovery path for renderer + composer + RTs (SE12)
- [x] 21. Write `SpiritVisualizerCanvas.tsx` — React component + audio wiring (S6, S7)
- [x] 22. Single RAF owner (component-level); no duplicate internal render loops
- [x] 23. Add `spirit` to `VisualizerSettings` + `VisualizerMode` in `VisualizerBackdrop.tsx`; add lazy import
- [x] 24. Smoke-test: Playwright e2e — canvas mounts and backdrop attribute is `spirit` *(pass)*
- [x] 25. Smoke-test: Playwright e2e — palette attributes present + inherited for spirit *(pass)*
_Visual/audio quality checks (bloom, afterimage, aurora streaking, drop burst) — deferred to live session_
- [x] 26. Bundle budget check (`npm run check:bundle`); update `SUMMARIES (Claude).md`
- [x] 27. Fix lint warnings in new canvas components (`react-hooks/exhaustive-deps`, stale-ref cleanup) — 0 errors, 21 warnings (pre-existing in unrelated files)
- [x] 28. Raise bundle budget to 720 KB total gzip (was 520 KB) to accommodate Three.js lazy chunk; re-validate `check:bundle` *(pass)*
---
## Integration wiring (both visualizers)
- [x] I1. `LyricsVisualStyle` union + `PerformanceSidebar.tsx` mode buttons include `tendrils` and `spirit`
- [x] I2. Manifest / settings whitelist normalisation for new `tendrils` and `spirit` settings blocks
- [x] I3. Lazy-loading chunk boundaries: `TendrilsVisualizerCanvas` and `SpiritVisualizerCanvas` behind `React.lazy`
+294
View File
@@ -1,5 +1,126 @@
# StemEngine — Session Summaries
## 2026-04-13
### SingUI pipeline mode + project-first karaoke flow (work started)
Implemented the first pass of the SingUI cleanup plan to support configurable processing behavior and reduce SEK dependency in daily karaoke use.
Delivered in this pass:
- Added backend settings key `singui_process_mode` with validation (`auto` / `separate_only`) in settings router defaults + patch schema.
- Added Settings → Auto Pipeline UI control to choose and save SingUI package behavior.
- Updated conversion orchestration (`useConvertStatus`) to honor `singui_process_mode`:
- `auto`: run configured additional Auto Pipeline steps.
- `separate_only`: perform separation/project creation only.
- Updated SingUI karaoke list behavior in project-linked mode:
- project singing button is prioritized when a linked project exists,
- queue item selection prefers project source over SEK source,
- package action wording shifted to `Prepare Project`.
Step-1 follow-up completed in this session:
- Extended `useConvertStatus` strategy coverage to include `pipeline` and `preset` branches.
- `pipeline` branch now submits `/api/pipeline` by streaming song audio from `/api/subsonic/stream/{songId}` into a `File` upload payload.
- `preset` branch now resolves selected preset (or fallback), applies preset separation model/format/instrument models, then executes preset-defined additional steps.
- Consolidated post-separation step execution via shared helper to avoid duplicated logic.
Step-2 follow-up completed in this session:
- Expanded backend settings validation for `singui_process_mode` to include `pipeline` and `preset`.
- Added persisted backend setting `singui_process_preset`.
- Extended Settings → Auto Pipeline UI mode picker to expose all four modes:
- `auto`, `separate_only`, `pipeline`, `preset`
- Added conditional preset dropdown when `preset` mode is selected, with save payload including `singui_process_preset`.
- Added frontend app-settings typing for `singui_process_mode` + `singui_process_preset`.
Step-3 follow-up completed in this session:
- Per-page wording cleanup for project-first flow in Tracks/Albums/Sing:
- idle/action copy moved from `Package` to `Prepare Project`
- running copy moved from `Packaging` to `Preparing`
- ready copy moved from `Packaged` to `Project Ready`
- Updated process button tooltip copy to describe project preparation behavior and reference Settings mode.
- Kept SEK actions explicitly labeled (`Sing (SEK)` / `Sing (Pick SEK)`) so portable-package flow remains distinct.
Step-4 follow-up completed in this session:
- Ran frontend validation commands after step-3 changes:
- `npm run -s typecheck` (pass)
- `npm run -s build` (pass)
- Updated planning/status docs to record completion and verification evidence.
Optional-item follow-up completed in this session:
- Migrated Auto Pipeline preset storage flow toward backend settings:
- Settings preset save/delete now persist `auto_presets` via `/api/settings`.
- Settings hydration now prefers backend `auto_presets` and mirrors to local fallback cache.
- Added one-time backfill: if backend preset list is empty and local preset list exists, local presets are uploaded to backend.
- Updated preset-mode conversion execution to prefer backend `auto_presets` from app settings, with local fallback.
Planning follow-up completed in this session:
- Added a dedicated Git-managed presets strategy plan covering where to store default Auto Pipeline presets in-repo, merge precedence with user presets, backend/frontend changes, rollout phases, and acceptance criteria.
- Plan file created:
- `2026-04-13_Git_Managed_Auto_Pipeline_Presets_Plan.md`
Implementation follow-up completed in this session:
- Added built-in preset catalog file in Git:
- `backend/presets/auto_pipeline/default_presets.json`
- Added backend loader/merge service:
- `backend/services/auto_presets_catalog.py`
- Extended settings response in `backend/routers/settings.py` with:
- `auto_presets_user`
- `auto_presets_builtin`
- `auto_presets_effective`
- Updated frontend preset handling to prefer backend effective presets and persist user presets separately:
- `frontend/src/pages/Settings.tsx`
- `frontend/src/hooks/useConvertStatus.ts`
- `frontend/src/utils/autoPipeline.ts`
- `frontend/src/api/client.ts`
- Validation completed:
- `frontend`: `npm run -s typecheck` (pass)
- `frontend`: `npm run -s build` (pass)
Primary files touched:
- `backend/routers/settings.py`
- `frontend/src/api/client.ts`
- `frontend/src/pages/Settings.tsx`
- `frontend/src/hooks/useConvertStatus.ts`
- `frontend/src/pages/Sing.tsx`
- `2026-04-13_SingUI_Pipeline_and_Karaoke_Source_Plan.md`
### Muon/Fluid visual refinement outstanding items (completed)
Completed the remaining visualizer refinement work from `2026-04-13_Muon_VisualRefinement_Plan.md`.
Delivered:
- `frontend/src/components/visualizers/muon/noise.ts`
- Replaced hash-only noise with smooth value noise and added `curlNoise2D`.
- Preserved `centeredNoise2D` compatibility export.
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
- Switched emitter sampling to `curlNoise2D`.
- Added beat-driven pattern cycling cooldown and stronger drop pattern jumps.
- Replaced emit loop with ring-first circumference distribution and per-frame pool-cap guard.
- Lowered launch speed and default lifespan baseline for cleaner ring structure.
- Replaced `shadowBlur` rendering with radial-gradient particle glow.
- Added finite-position guard before gradient creation.
- Added `quality` to effect dependencies for consistent runtime quality switching.
Validation:
- `frontend`: `npm run -s typecheck` (pass)
- `frontend`: `npm run -s build` (pass)
Optional upstream audit (F11):
- Verified classic upstream JS source at `PavelDoGreat/WebGL-Fluid-Simulation`.
- Confirmed `PavelDoGreat/Fluid-Simulation` now targets a rewritten architecture and is not a drop-in JS re-vendor source for current `webgl-fluid.mjs` integration.
- No re-vendor performed in this pass.
---
## 2026-04-01
@@ -2875,3 +2996,176 @@ Implemented:
Validation:
- `npm run -s typecheck` ✅
### Fluid plan pre-handoff audit (2026-04-13)
Reviewed `2026-04-13_Muon_VisualRefinement_Plan.md` Fluid section against current source files
before Copilot handoff. Found and fixed four gaps:
1. **`visualization_preset` confirmed missing from fluid interface** — F10e changed from
"verify; add if missing" to a definite instruction: add `visualization_preset?: string` to
the `fluid` block in `VisualizerBackdrop.tsx`.
2. **`bgRgb` / `highlightRgb` undeclared** — F9 integration block used these variables without
declaring them. Added explicit declaration block before the `updateFluidConfig` call.
3. **User settings silently superseded + dead code** — F9 now notes that the four helper
functions (`fluidDensityDissipation`, `fluidVelocityDissipation`, `fluidCurl`,
`fluidSplatRadius`) should be deleted; `s?.background_persistence`, `s?.drag`, `s?.blob_life`
are superseded by preset base values; `s?.decay` and `s?.treble_pulse` are preserved.
4. **F5 / implementation order** — Step 13 now notes F5 (PRESSURE) is handled within F9
step 15, not separately. Step 18 updated to confirm the type field is missing.
### Fluid parity implementation completed (2026-04-13)
Implemented the Fluid section from `2026-04-13_Muon_VisualRefinement_Plan.md`
for the planned parity items and safeguards.
Completed:
- `webgl-fluid.mjs`
- Exposed direct splat hook on canvas (`_fluidSplat`) with patch marker.
- `fluidSim.ts`
- Replaced synthetic mouse dispatch in `splatFluid` with direct `_fluidSplat` call.
- Preserved orientation with Y flip for texcoord convention.
- Added async init token guard to avoid stale async attach races.
- `FluidVisualizerCanvas.tsx`
- Added `FluidPreset` + `FLUID_PRESETS` and preset selection via
`settings?.visualization_preset`.
- Switched to preset-aware + audio-reactive runtime config updates:
`DENSITY_DISSIPATION`, `VELOCITY_DISSIPATION`, `CURL`, `PRESSURE`,
`SPLAT_RADIUS`, `BLOOM_INTENSITY`, `BLOOM_THRESHOLD`, `BLOOM_SOFT_KNEE`,
`SUNRAYS_WEIGHT`, `BACK_COLOR`.
- Added `blendColor` and drop flash into `BACK_COLOR`.
- Added quality-aware init values (`SHADING`, bloom/sunrays resolutions,
pressure iterations) and enabled `SUNRAYS` at init.
- Added sweep orbit audio-reactivity and drop throttle bypass.
- `VisualizerBackdrop.tsx`
- Added `visualization_preset?: string` to `VisualizerSettings['fluid']`.
Validation:
- `npm run -s build` ✅
### Tendrils + Spirit visualizer plan (2026-04-13)
Created `2026-04-13_Tendrils_Spirit_Visualizer_Plan.md`.
**Tendrils** — raw WebGL2, no new dependencies. GPGPU ping-pong (RGBA32F, 512×512 = 262 144
particles at high quality). Three forces: smooth simplex noise, flow-field feedback (particles
reinforce popular paths), and beat-driven target orbit. GL_LINES rendering for ribbon trails.
Self-contained `tendrilsSim.ts` module mirrors `fluidSim.ts` API.
**Spirit** — Three.js (`three` npm dependency, ~170 KB gzip). GPGPU 256×256 = 65 536 particles
in 3D with `[x, y, z, life]` per texel. 3D curl noise (`curl.glsl`) gives divergence-free
swirling motion. Renders as point sprites or triangle meshes. `EffectComposer` with
`UnrealBloomPass` + `AfterimagePass` for glow and motion blur. Drop causes attraction reversal
(particles scatter outward then reform).
Both visualizers follow the `featuresRef` / `paletteRef` / `settingsRef` pattern from Muon and
Fluid: all audio values (`bass`, `kick`, `treble`, `drum`, `beatPhase`, `dropBoost`, `phasePulse`,
`rms`, `idle`) drive shader uniforms each frame; settings hot-reload without restarting the effect
loop. Plan includes full GLSL source for all shaders, TypeScript module exports, 5 presets each,
complete edge case sections, `VisualizerSettings` type additions, and a 24-step implementation order.
### Tendrils + Spirit implementation pass (2026-04-13, follow-up)
Completed implementation from `CHECKLIST.md` for the new visualizer stack and integration wiring.
Delivered:
- Added shader typing and Vite raw-import support:
- `frontend/src/types/glsl.d.ts`
- `frontend/src/vite-env.d.ts`
- Implemented Tendrils (WebGL2):
- GLSL: `screen.vert`, `logic.glsl`, `render.vert`, `render.frag`, `flow.frag`, `fade.frag`
- Runtime: `frontend/src/components/visualizers/tendrilsSim.ts`
- React wrapper: `frontend/src/components/visualizers/TendrilsVisualizerCanvas.tsx`
- Implemented Spirit (Three.js + GPGPU):
- Installed `three` and `@types/three`
- GLSL: `curl.glsl`, `position.glsl`, `particle.vert`, `particle.frag`
- Runtime: `frontend/src/components/visualizers/spiritSim.ts`
- React wrapper: `frontend/src/components/visualizers/SpiritVisualizerCanvas.tsx`
- Updated visualizer integration and settings:
- `VisualizerBackdrop.tsx`: added `tendrils`/`spirit` mode + lazy imports + settings blocks
- `LyricsLivePreview.tsx`: extended `LyricsVisualStyle`, SEK mode/settings typing, settings normalization and mode routing for `tendrils`/`spirit`
- `PerformanceSidebar.tsx`: added `tendrils`/`spirit` modes, settings defaults, and mode-specific controls
- Updated `CHECKLIST.md` with completed items and retained manual smoke-test items.
Validation:
- `frontend`: `npx tsc --noEmit` (pass)
- `frontend`: `npm run -s build` (pass)
- `frontend`: `npm run -s check:bundle` (pass)
### Lint fixes + validation re-run (2026-04-13, follow-up)
Re-ran full static validation suite after discovering the previous `check:bundle` pass
used a stale `dist-local` (pre-Spirit build). All issues resolved.
**Lint fixes (new files, 0 errors after):**
- `SpiritVisualizerCanvas.tsx` — captured `canvasRef.current` into a local variable
before cleanup so the `destroySpiritSim` call doesn't read a stale/null ref
(`react-hooks/exhaustive-deps` stale-ref rule).
- `TendrilsVisualizerCanvas.tsx` — added `widthRef`/`heightRef` mirrors of the
`width`/`height` props; effect reads from refs so prop changes don't trigger
unnecessary WebGL re-initialisation (`react-hooks/exhaustive-deps`).
**Bundle budget:**
- Raised total gzip limit from 520 KB → 720 KB in
`frontend/scripts/check-bundle-budget.mjs` (comment added explaining rationale).
- Three.js lazy chunk (`SpiritVisualizerCanvas`) is ~132 KB gzip; total across
30 JS files is ~680 KB gzip, well within the new 720 KB limit.
Validation:
- `npx tsc --noEmit` ✅ (0 errors)
- `npm run -s lint` ✅ (0 errors, 21 warnings — all pre-existing in unrelated files)
- `npm run -s build:local` ✅ (built in ~6 s)
## 2026-04-14
### Full codebase audit (2026-04-14)
Ran 7 parallel specialist agents against the full StemEngine codebase (frontend TypeScript/React,
backend Python/FastAPI, visualizers WebGL/Three.js). All findings compiled into
`2026-04-14_StemEngine_Audit.md`.
**92 findings total across 7 domains:**
- **Security (11)**: SEC1–SEC2 (High) — no auth on ML endpoints; path traversal via
`file.filename` at 13 sites in 7 upload handlers. SEC4 — SSRF via yt-dlp. SEC5 — CORS wildcard
from `APP_ENV` defaulting to development. SEC7 — `project_name` traversal. SEC11 — hardcoded
fallback token.
- **React (14)**: R1 — Spirit full Three.js reinit on every resize. R2–R4 — audio node teardown
on fader drag and stale-closure playback events. R5 — context action functions not memoized.
R10 — zero ErrorBoundaries across entire frontend.
- **Performance (12)**: P1 (Critical) — no Page Visibility API guard in any of the 4 visualizer
RAF loops (full GPU burn in hidden tabs). P2 — no IntersectionObserver. P3–P5 — FFT analysis
runs when paused; 3 Float32Array allocs/frame; 1,600 createRadialGradient calls/frame.
- **Code Quality (13)**: C1 — `quality` stale closure in Muon. C2 — initFluidSim unmount race
(WebGL() called on destroyed canvas). C3 — partial pause zeroing leaves stale `beatPhase` /
`isBeatDrop`. C9 — `EXT_color_buffer_float` not verified before RGBA32F GPGPU FBOs.
- **Python backend (12)**: PY4 (High) — `separate_existing` submits GPU work to general executor
(CUDA OOM risk). PY2 — blocking file I/O in all async upload handlers. PY3 — same path traversal
as SEC2. PY11 — global `torch.load` monkey-patch forces `weights_only=False` process-wide
(CVE-2025-32434 vector). PY8 — `datetime.utcnow()` deprecated.
- **TypeScript (18)**: TS1 (Critical) — entire MIDI object graph typed as `any`. TS2–TS5 — env
vars, AppSettings presets, Tone.js synths, Two.js helper functions all `any`. TS9–TS10 — non-null
assertions across async boundaries and in drag event handlers.
- **Architecture (12)**: A4 (Critical) — LyricsLivePreview 5,196-line god component with 41
useEffects and 60fps re-renders from `setCurrentTime`. A1–A2 — VisualizerBackdrop requires 4-site
edits per new visualizer; VisualizerSettings defined 3 independent times. A5 — vendored
`webgl-fluid.mjs` with `_fluidSplat` injection diverges from npm package (silent breakage risk
after `npm ci`). A3/A7 — Spirit-specific: palette init-time flash and full reinit on resize.
- `npm run -s check:bundle` ✅ (695928 bytes < 737280 limit)
@@ -0,0 +1,36 @@
[
{
"name": "Default — Stems Only",
"model": "htdemucs",
"format": "mp3",
"instrumentModels": {},
"steps": []
},
{
"name": "Default — Karaoke Prep",
"model": "htdemucs",
"format": "mp3",
"instrumentModels": {},
"steps": [
{
"type": "lyrics",
"engine": "whisperx",
"model": "medium",
"diarize": false,
"lsyncAlign": false,
"twoPass": false,
"vocalStyle": "general",
"explicit": false,
"stem": "vocals"
},
{
"type": "lyricsAlign",
"timingPrecision": "word"
},
{
"type": "sekPackage",
"includeVideo": true
}
]
}
]
+8
View File
@@ -48,6 +48,8 @@ def list_library(folder: str | None = Query(default=None, description="Optional
return list_sek_packages(folder)
except ProjectOperationError as exc:
raise HTTPException(status_code=exc.status_code, detail=exc.detail) from exc
except OSError as exc:
raise HTTPException(status_code=503, detail=f"SEK storage unavailable: {exc}") from exc
@router.post("/match-subsonic")
@@ -57,6 +59,8 @@ def match_subsonic(body: MatchSubsonicRequest):
return match_subsonic_songs(songs=songs, folder=body.folder)
except ProjectOperationError as exc:
raise HTTPException(status_code=exc.status_code, detail=exc.detail) from exc
except OSError as exc:
raise HTTPException(status_code=503, detail=f"SEK storage unavailable: {exc}") from exc
@router.get("/packages/{package_id}")
@@ -65,6 +69,8 @@ def get_manifest(package_id: str, folder: str | None = Query(default=None, descr
return get_sek_manifest(package_id, folder)
except ProjectOperationError as exc:
raise HTTPException(status_code=exc.status_code, detail=exc.detail) from exc
except OSError as exc:
raise HTTPException(status_code=503, detail=f"SEK storage unavailable: {exc}") from exc
@router.get("/packages/{package_id}/asset")
@@ -156,3 +162,5 @@ def get_asset(
)
except ProjectOperationError as exc:
raise HTTPException(status_code=exc.status_code, detail=exc.detail) from exc
except OSError as exc:
raise HTTPException(status_code=503, detail=f"SEK storage unavailable: {exc}") from exc
+30
View File
@@ -17,6 +17,11 @@ from fastapi import APIRouter, HTTPException
from pydantic import BaseModel, field_validator
from ..config import SETTINGS_PATH
from ..services.auto_presets_catalog import (
load_builtin_auto_presets,
merge_auto_presets,
normalize_auto_presets,
)
router = APIRouter(prefix="/api/settings", tags=["settings"])
@@ -34,6 +39,7 @@ _VALID_WHISPER_MODELS = {
"large-v3-turbo", "turbo",
}
_VALID_KARAOKE_SOURCE_MODE = {"projects", "sek-folder"}
_VALID_SINGUI_PROCESS_MODE = {"auto", "separate_only", "pipeline", "preset"}
# Default values for all known preference keys
_DEFAULTS: dict[str, Any] = {
@@ -48,6 +54,8 @@ _DEFAULTS: dict[str, Any] = {
"auto_presets": [],
"karaoke_source_mode": "sek-folder",
"karaoke_sek_folder": "",
"singui_process_mode": "auto",
"singui_process_preset": "",
"pixabay_api_key": "",
"pixabay_cache_enabled": True,
"pixabay_min_clip_duration": 15,
@@ -82,6 +90,10 @@ def get_hf_token() -> Optional[str]:
def get_settings():
"""Return all non-sensitive settings. Redacted keys return a boolean flag."""
data = _load()
user_auto_presets = normalize_auto_presets(data.get("auto_presets"))
builtin_auto_presets = load_builtin_auto_presets()
effective_auto_presets = merge_auto_presets(builtin_auto_presets, user_auto_presets)
result = {}
for k, v in data.items():
if k in _REDACTED:
@@ -94,6 +106,13 @@ def get_settings():
# Check env var fallback
env_val = os.environ.get(k.upper())
result[f"{k}_set"] = bool(env_val)
# Preserve user-editable presets separately while exposing built-in and
# effective merged presets for UI/runtime consumption.
result["auto_presets"] = user_auto_presets
result["auto_presets_user"] = user_auto_presets
result["auto_presets_builtin"] = builtin_auto_presets
result["auto_presets_effective"] = effective_auto_presets
return result
@@ -110,6 +129,8 @@ class SettingsPatch(BaseModel):
auto_presets: Optional[list] = None
karaoke_source_mode: Optional[str] = None
karaoke_sek_folder: Optional[str] = None
singui_process_mode: Optional[str] = None
singui_process_preset: Optional[str] = None
pixabay_api_key: Optional[str] = None
pixabay_cache_enabled: Optional[bool] = None
pixabay_min_clip_duration: Optional[float] = None
@@ -142,6 +163,15 @@ class SettingsPatch(BaseModel):
)
return v
@field_validator("singui_process_mode")
@classmethod
def _validate_singui_process_mode(cls, v: Optional[str]) -> Optional[str]:
if v is not None and v not in _VALID_SINGUI_PROCESS_MODE:
raise ValueError(
f"Unknown singui_process_mode '{v}'. Valid values: {sorted(_VALID_SINGUI_PROCESS_MODE)}"
)
return v
@field_validator("pixabay_min_clip_duration", "pixabay_max_clip_duration")
@classmethod
def _validate_pixabay_clip_duration(cls, v: Optional[float]) -> Optional[float]:
+97
View File
@@ -0,0 +1,97 @@
"""Git-managed Auto Pipeline preset catalog loading and merge helpers."""
from __future__ import annotations
import json
import logging
import os
from pathlib import Path
from typing import Any
log = logging.getLogger(__name__)
_DEFAULT_CATALOG_PATH = (
Path(__file__).resolve().parents[1]
/ "presets"
/ "auto_pipeline"
/ "default_presets.json"
)
def _normalize_preset(raw: Any) -> dict[str, Any] | None:
if not isinstance(raw, dict):
return None
name = str(raw.get("name") or "").strip()
model = str(raw.get("model") or "").strip()
fmt = str(raw.get("format") or "").strip()
if not name or not model or not fmt:
return None
inst_raw = raw.get("instrumentModels")
instrument_models: dict[str, str] = {}
if isinstance(inst_raw, dict):
for k, v in inst_raw.items():
if isinstance(k, str) and isinstance(v, str) and v.strip():
instrument_models[k] = v.strip()
steps_raw = raw.get("steps")
steps: list[dict[str, Any]] = []
if isinstance(steps_raw, list):
for step in steps_raw:
if isinstance(step, dict) and step.get("type"):
steps.append(step)
return {
"name": name,
"model": model,
"format": fmt,
"instrumentModels": instrument_models,
"steps": steps,
}
def normalize_auto_presets(raw: Any) -> list[dict[str, Any]]:
if not isinstance(raw, list):
return []
out: list[dict[str, Any]] = []
for item in raw:
preset = _normalize_preset(item)
if preset:
out.append(preset)
return out
def load_builtin_auto_presets() -> list[dict[str, Any]]:
path_raw = os.environ.get("AUTO_PRESETS_CATALOG_PATH", "").strip()
path = Path(path_raw) if path_raw else _DEFAULT_CATALOG_PATH
if not path.exists():
log.info("Auto preset catalog not found at %s; continuing with no built-ins", path)
return []
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except Exception as exc:
log.warning("Failed to parse auto preset catalog %s: %s", path, exc)
return []
presets = normalize_auto_presets(payload)
if not presets:
log.warning("Auto preset catalog %s loaded but produced no valid presets", path)
return presets
def merge_auto_presets(
builtin_presets: list[dict[str, Any]],
user_presets: list[dict[str, Any]],
) -> list[dict[str, Any]]:
merged: dict[str, dict[str, Any]] = {}
for preset in normalize_auto_presets(builtin_presets):
merged[preset["name"]] = preset
for preset in normalize_auto_presets(user_presets):
merged[preset["name"]] = preset
return list(merged.values())
+46 -14
View File
@@ -66,8 +66,26 @@ def _resolve_folder_path(folder: str | None = None) -> Path:
return _coerce_folder_path(configured)
def _ensure_folder_accessible(base: Path) -> None:
try:
if not base.exists() or not base.is_dir():
raise ProjectOperationError(404, f"SEK folder does not exist: {base}")
except ProjectOperationError:
raise
except OSError as exc:
raise ProjectOperationError(503, f"SEK folder is unavailable: {base} ({exc})") from exc
def _safe_resolved_str(path: Path) -> str:
try:
return str(path.resolve())
except OSError:
return str(path)
def _package_id(path: Path) -> str:
return hashlib.sha1(str(path.resolve()).encode("utf-8")).hexdigest()[:16]
stable = _safe_resolved_str(path)
return hashlib.sha1(stable.encode("utf-8")).hexdigest()[:16]
def _prune_expired_cache(cache: dict[str, tuple[float, Any]], now: float, max_entries: int) -> None:
@@ -104,11 +122,22 @@ def _join_norm(*parts: Any) -> str:
def _iter_sek_paths(base: Path) -> list[Path]:
# Recursive scan so exports organized as <sekfold>/<project>/<name>.sek are discovered.
return sorted(
[p for p in base.rglob("*.sek") if p.is_file()],
key=lambda p: p.stat().st_mtime,
reverse=True,
)
try:
candidates = list(base.rglob("*.sek"))
except OSError as exc:
raise ProjectOperationError(503, f"Failed to scan SEK folder: {base} ({exc})") from exc
with_mtime: list[tuple[Path, float]] = []
for path in candidates:
try:
if path.is_file():
with_mtime.append((path, path.stat().st_mtime))
except OSError:
# Skip transient/unreadable entries instead of failing the whole listing.
continue
with_mtime.sort(key=lambda item: item[1], reverse=True)
return [p for p, _ in with_mtime]
def _extract_package_match_keys(
@@ -244,8 +273,7 @@ def _extract_duration_seconds_from_manifest(manifest: dict[str, Any] | None) ->
def list_sek_packages(folder: str | None = None) -> dict[str, Any]:
started = time.perf_counter()
base = _resolve_folder_path(folder)
if not base.exists() or not base.is_dir():
raise ProjectOperationError(404, f"SEK folder does not exist: {base}")
_ensure_folder_accessible(base)
packages: list[dict[str, Any]] = []
scanned = _iter_sek_paths(base)
@@ -253,6 +281,11 @@ def list_sek_packages(folder: str | None = None) -> dict[str, Any]:
warnings: list[str] = []
manifest: dict[str, Any] | None = None
valid = True
try:
mtime = package_path.stat().st_mtime
except OSError:
mtime = 0.0
warnings.append("Package mtime unavailable")
try:
manifest = _read_manifest_cached(package_path)
@@ -303,7 +336,7 @@ def list_sek_packages(folder: str | None = None) -> dict[str, Any]:
"has_video": bool(video),
"lyrics": lyrics if isinstance(lyrics, str) else "",
"lyrics_track_count": len(lyrics_tracks) if isinstance(lyrics_tracks, dict) else 0,
"mtime": package_path.stat().st_mtime,
"mtime": mtime,
"warnings": warnings,
"meta_title": meta_title,
"meta_artist": meta_artist,
@@ -317,8 +350,8 @@ def list_sek_packages(folder: str | None = None) -> dict[str, Any]:
)
return {
"folder": str(base.resolve()),
"scan_root": str(base.resolve()),
"folder": _safe_resolved_str(base),
"scan_root": _safe_resolved_str(base),
"scan_count": len(scanned),
"scan_ms": round((time.perf_counter() - started) * 1000, 2),
"count": len(packages),
@@ -328,8 +361,7 @@ def list_sek_packages(folder: str | None = None) -> dict[str, Any]:
def _resolve_package_by_id(package_id: str, folder: str | None = None) -> Path:
base = _resolve_folder_path(folder)
if not base.exists() or not base.is_dir():
raise ProjectOperationError(404, f"SEK folder does not exist: {base}")
_ensure_folder_accessible(base)
for package_path in _iter_sek_paths(base):
if _package_id(package_path) == package_id:
@@ -447,7 +479,7 @@ def _build_match_cache_key(folder: str | None, songs: list[dict[str, Any]]) -> s
]
songs_json = json.dumps(songs_payload, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
songs_hash = hashlib.sha1(songs_json.encode("utf-8")).hexdigest()
return f"{base.resolve()}|{songs_hash}"
return f"{_safe_resolved_str(base)}|{songs_hash}"
def match_subsonic_songs(
+2 -2
View File
@@ -19,8 +19,8 @@
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
<meta name="apple-mobile-web-app-title" content="StemEngine" />
<link rel="apple-touch-icon" href="/icon.svg" />
<script type="module" crossorigin src="/assets/index-Cyxc7Uk9.js"></script>
<link rel="stylesheet" crossorigin href="/assets/index-B34zlAct.css">
<script type="module" crossorigin src="/assets/index-DzYnwFRo.js"></script>
<link rel="stylesheet" crossorigin href="/assets/index-BCm86cqC.css">
</head>
<body>
<div id="root"></div>
+56
View File
@@ -368,3 +368,59 @@ test('visualizer frame cadence stays healthy in windowed and fullscreen preview'
await page.locator('button[title="Exit fullscreen"]').click()
})
test('switching to Tendrils mode mounts canvas and sets correct backdrop attribute', async ({ page }) => {
await page.goto('/projects/p1')
await page.getByRole('button', { name: /Live preview \(LRC\)/i }).click()
await page.locator('button[title="Style picker"]').click()
await page.locator('button[title="Use Tendrils visual renderer"]').click()
const backdrop = page.locator('[data-testid="visualizer-backdrop"]')
await expect(backdrop).toHaveAttribute('data-visualizer-mode', 'tendrils')
// Tendrils lazy chunk renders a <canvas> element inside the backdrop
await expect(backdrop.locator('canvas')).toBeVisible()
})
test('switching to Spirit mode mounts canvas and sets correct backdrop attribute', async ({ page }) => {
await page.goto('/projects/p1')
await page.getByRole('button', { name: /Live preview \(LRC\)/i }).click()
await page.locator('button[title="Style picker"]').click()
await page.locator('button[title="Use Spirit visual renderer"]').click()
const backdrop = page.locator('[data-testid="visualizer-backdrop"]')
await expect(backdrop).toHaveAttribute('data-visualizer-mode', 'spirit')
// Spirit lazy chunk renders a <canvas> element inside the backdrop
await expect(backdrop.locator('canvas')).toBeVisible()
})
test('tendrils and spirit inherit palette from artwork and preserve it across mode switches', async ({ page }) => {
await page.context().route('**/api/projects/p1/cover-art', async (route) => {
await route.fulfill({ status: 200, contentType: 'image/svg+xml', body: redSvg })
})
await page.goto('/projects/p1')
await page.getByRole('button', { name: /Live preview \(LRC\)/i }).click()
await page.locator('button[title="Style picker"]').click()
// Open picker once — all mode buttons are visible; switch Muon → Tendrils → Spirit
await page.locator('button[title="Use Muon visual renderer"]').click()
const backdrop = page.locator('[data-testid="visualizer-backdrop"]')
await expect(backdrop).toHaveAttribute('data-palette-source', 'runtime-artwork')
const seedColor = await backdrop.getAttribute('data-palette-seed')
const highlightColor = await backdrop.getAttribute('data-palette-highlight')
expect(seedColor).toBeTruthy()
// Switch to Tendrils — palette must be preserved (picker still open)
await page.locator('button[title="Use Tendrils visual renderer"]').click()
await expect(backdrop).toHaveAttribute('data-visualizer-mode', 'tendrils')
await expect(backdrop).toHaveAttribute('data-palette-seed', seedColor || '')
await expect(backdrop).toHaveAttribute('data-palette-highlight', highlightColor || '')
// Switch to Spirit — palette must still be preserved
await page.locator('button[title="Use Spirit visual renderer"]').click()
await expect(backdrop).toHaveAttribute('data-visualizer-mode', 'spirit')
await expect(backdrop).toHaveAttribute('data-palette-seed', seedColor || '')
await expect(backdrop).toHaveAttribute('data-palette-highlight', highlightColor || '')
})
+73
View File
@@ -25,6 +25,7 @@
"react": "^18.2.0",
"react-dom": "^18.2.0",
"react-router-dom": "^6.30.3",
"three": "^0.183.2",
"tone": "^14.7.0",
"two.js": "^0.8.23",
"webgl-fluid": "^0.3.9"
@@ -34,6 +35,7 @@
"@types/qrcode": "^1.5.6",
"@types/react": "^18.2.37",
"@types/react-dom": "^18.2.15",
"@types/three": "^0.183.1",
"@typescript-eslint/eslint-plugin": "^8.26.1",
"@typescript-eslint/parser": "^8.26.1",
"@vitejs/plugin-react": "^4.2.0",
@@ -331,6 +333,13 @@
"node": ">=6.9.0"
}
},
"node_modules/@dimforge/rapier3d-compat": {
"version": "0.12.0",
"resolved": "https://registry.npmjs.org/@dimforge/rapier3d-compat/-/rapier3d-compat-0.12.0.tgz",
"integrity": "sha512-uekIGetywIgopfD97oDL5PfeezkFpNhwlzlaEYNOA0N6ghdsOvh/HYjSMek5Q2O1PYvRSDFcqFVJl4r4ZBwOow==",
"dev": true,
"license": "Apache-2.0"
},
"node_modules/@esbuild/aix-ppc64": {
"version": "0.21.5",
"resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.21.5.tgz",
@@ -1422,6 +1431,13 @@
"midi-file": "^1.2.2"
}
},
"node_modules/@tweenjs/tween.js": {
"version": "23.1.3",
"resolved": "https://registry.npmjs.org/@tweenjs/tween.js/-/tween.js-23.1.3.tgz",
"integrity": "sha512-vJmvvwFxYuGnF2axRtPYocag6Clbb5YS7kLL+SO/TeVFzHqDIWrNKYtcsPMibjDx9O+bu+psAy9NKfWklassUA==",
"dev": true,
"license": "MIT"
},
"node_modules/@types/babel__core": {
"version": "7.20.5",
"resolved": "https://registry.npmjs.org/@types/babel__core/-/babel__core-7.20.5.tgz",
@@ -1514,6 +1530,36 @@
"@types/react": "^18.0.0"
}
},
"node_modules/@types/stats.js": {
"version": "0.17.4",
"resolved": "https://registry.npmjs.org/@types/stats.js/-/stats.js-0.17.4.tgz",
"integrity": "sha512-jIBvWWShCvlBqBNIZt0KAshWpvSjhkwkEu4ZUcASoAvhmrgAUI2t1dXrjSL4xXVLB4FznPrIsX3nKXFl/Dt4vA==",
"dev": true,
"license": "MIT"
},
"node_modules/@types/three": {
"version": "0.183.1",
"resolved": "https://registry.npmjs.org/@types/three/-/three-0.183.1.tgz",
"integrity": "sha512-f2Pu5Hrepfgavttdye3PsH5RWyY/AvdZQwIVhrc4uNtvF7nOWJacQKcoVJn0S4f0yYbmAE6AR+ve7xDcuYtMGw==",
"dev": true,
"license": "MIT",
"dependencies": {
"@dimforge/rapier3d-compat": "~0.12.0",
"@tweenjs/tween.js": "~23.1.3",
"@types/stats.js": "*",
"@types/webxr": ">=0.5.17",
"@webgpu/types": "*",
"fflate": "~0.8.2",
"meshoptimizer": "~1.0.1"
}
},
"node_modules/@types/webxr": {
"version": "0.5.24",
"resolved": "https://registry.npmjs.org/@types/webxr/-/webxr-0.5.24.tgz",
"integrity": "sha512-h8fgEd/DpoS9CBrjEQXR+dIDraopAEfu4wYVNY2tEPwk60stPWhvZMf4Foo5FakuQ7HFZoa8WceaWFervK2Ovg==",
"dev": true,
"license": "MIT"
},
"node_modules/@typescript-eslint/eslint-plugin": {
"version": "8.57.0",
"resolved": "https://registry.npmjs.org/@typescript-eslint/eslint-plugin/-/eslint-plugin-8.57.0.tgz",
@@ -1787,6 +1833,13 @@
"vite": "^4.2.0 || ^5.0.0 || ^6.0.0 || ^7.0.0"
}
},
"node_modules/@webgpu/types": {
"version": "0.1.69",
"resolved": "https://registry.npmjs.org/@webgpu/types/-/types-0.1.69.tgz",
"integrity": "sha512-RPmm6kgRbI8e98zSD3RVACvnuktIja5+yLgDAkTmxLr90BEwdTXRQWNLF3ETTTyH/8mKhznZuN5AveXYFEsMGQ==",
"dev": true,
"license": "BSD-3-Clause"
},
"node_modules/abcjs": {
"version": "6.6.2",
"resolved": "https://registry.npmjs.org/abcjs/-/abcjs-6.6.2.tgz",
@@ -2633,6 +2686,13 @@
"reusify": "^1.0.4"
}
},
"node_modules/fflate": {
"version": "0.8.2",
"resolved": "https://registry.npmjs.org/fflate/-/fflate-0.8.2.tgz",
"integrity": "sha512-cPJU47OaAoCbg0pBvzsgpTPhmhqI5eJjh/JIu8tPj5q+T7iLvW/JAYUqmE7KOB4R1ZyEhzBaIQpQpardBF5z8A==",
"dev": true,
"license": "MIT"
},
"node_modules/file-entry-cache": {
"version": "6.0.1",
"resolved": "https://registry.npmjs.org/file-entry-cache/-/file-entry-cache-6.0.1.tgz",
@@ -3181,6 +3241,13 @@
"node": ">= 8"
}
},
"node_modules/meshoptimizer": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/meshoptimizer/-/meshoptimizer-1.0.1.tgz",
"integrity": "sha512-Vix+QlA1YYT3FwmBBZ+49cE5y/b+pRrcXKqGpS5ouh33d3lSp2PoTpCw19E0cKDFWalembrHnIaZetf27a+W2g==",
"dev": true,
"license": "MIT"
},
"node_modules/micromatch": {
"version": "4.0.8",
"resolved": "https://registry.npmjs.org/micromatch/-/micromatch-4.0.8.tgz",
@@ -4153,6 +4220,12 @@
"node": ">=0.8"
}
},
"node_modules/three": {
"version": "0.183.2",
"resolved": "https://registry.npmjs.org/three/-/three-0.183.2.tgz",
"integrity": "sha512-di3BsL2FEQ1PA7Hcvn4fyJOlxRRgFYBpMTcyOgkwJIaDOdJMebEFPA+t98EvjuljDx4hNulAGwF6KIjtwI5jgQ==",
"license": "MIT"
},
"node_modules/tinyglobby": {
"version": "0.2.15",
"resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.15.tgz",
+2
View File
@@ -34,6 +34,7 @@
"react": "^18.2.0",
"react-dom": "^18.2.0",
"react-router-dom": "^6.30.3",
"three": "^0.183.2",
"tone": "^14.7.0",
"two.js": "^0.8.23",
"webgl-fluid": "^0.3.9"
@@ -43,6 +44,7 @@
"@types/qrcode": "^1.5.6",
"@types/react": "^18.2.37",
"@types/react-dom": "^18.2.15",
"@types/three": "^0.183.1",
"@typescript-eslint/eslint-plugin": "^8.26.1",
"@typescript-eslint/parser": "^8.26.1",
"@vitejs/plugin-react": "^4.2.0",
+3 -1
View File
@@ -24,7 +24,9 @@ if (jsFiles.length === 0) {
}
const maxSingleGzip = Number(process.env.BUNDLE_MAX_SINGLE_GZIP_BYTES || 180 * 1024)
const maxTotalGzip = Number(process.env.BUNDLE_MAX_TOTAL_GZIP_BYTES || 520 * 1024)
// Budget raised from 520 KB → 720 KB (2026-04-13) to accommodate Three.js
// (~132 KB gzip) added as a lazy-loaded chunk for the Spirit visualizer.
const maxTotalGzip = Number(process.env.BUNDLE_MAX_TOTAL_GZIP_BYTES || 720 * 1024)
const stats = jsFiles.map((name) => {
const abs = join(assetsDir, name)
+5
View File
@@ -978,8 +978,13 @@ export interface AppSettings {
auto_model: string
auto_format: string
auto_presets: unknown[]
auto_presets_user?: unknown[]
auto_presets_builtin?: unknown[]
auto_presets_effective?: unknown[]
karaoke_source_mode?: 'projects' | 'sek-folder'
karaoke_sek_folder?: string
singui_process_mode?: 'auto' | 'separate_only' | 'pipeline' | 'preset'
singui_process_preset?: string
pixabay_api_key_set?: boolean
pixabay_cache_enabled?: boolean
pixabay_min_clip_duration?: number
+150 -86
View File
@@ -45,7 +45,7 @@ interface PreviewPlaybackPrefs {
soloedStems?: Record<string, boolean>
}
export type LyricsVisualStyle = 'scroll' | 'fade' | 'karaoke' | 'glow' | 'blur' | 'video' | 'muon' | 'fluid'
export type LyricsVisualStyle = 'scroll' | 'fade' | 'karaoke' | 'glow' | 'blur' | 'video' | 'muon' | 'fluid' | 'tendrils' | 'spirit'
type VideoStartPhase = 'idle' | 'priming' | 'playing'
type DeepPartial<T> = {
@@ -95,6 +95,29 @@ interface VizSettings {
// GUI Parameter Map additions
color_speed: number
}
tendrils: {
noise_weight: number
noise_scale: number
flow_weight: number
flow_decay: number
force_weight: number
speed_limit: number
damping: number
hue_range: number
mono_color: boolean
visualization_preset: string
}
spirit: {
speed: number
die_speed: number
radius: number
curl_size: number
attraction: number
bloom_intensity: number
bloom_threshold: number
afterimage_damp: number
visualization_preset: string
}
}
const VIZ_DEFAULTS: VizSettings = {
@@ -134,6 +157,29 @@ const VIZ_DEFAULTS: VizSettings = {
blob_life: 1.0,
color_speed: 1.0,
},
tendrils: {
noise_weight: 1.0,
noise_scale: 1.0,
flow_weight: 1.0,
flow_decay: 1.0,
force_weight: 1.0,
speed_limit: 1.0,
damping: 1.0,
hue_range: 1.0,
mono_color: false,
visualization_preset: 'flow',
},
spirit: {
speed: 1.0,
die_speed: 1.0,
radius: 1.5,
curl_size: 0.4,
attraction: 0.08,
bloom_intensity: 1.5,
bloom_threshold: 0.05,
afterimage_damp: 0.88,
visualization_preset: 'spirit',
},
}
const PERF_SIDEBAR_WIDTH = 280
@@ -152,21 +198,19 @@ function deepMergeVizSettings(base: VizSettings, patch?: DeepPartial<VizSettings
if (!patch) {
return {
...base,
muon: { ...base.muon },
fluid: { ...base.fluid },
muon: { ...base.muon },
fluid: { ...base.fluid },
tendrils: { ...base.tendrils },
spirit: { ...base.spirit },
}
}
return {
...base,
...patch,
muon: {
...base.muon,
...(patch.muon ?? {}),
},
fluid: {
...base.fluid,
...(patch.fluid ?? {}),
},
muon: { ...base.muon, ...(patch.muon ?? {}) },
fluid: { ...base.fluid, ...(patch.fluid ?? {}) },
tendrils: { ...base.tendrils, ...(patch.tendrils ?? {}) },
spirit: { ...base.spirit, ...(patch.spirit ?? {}) },
}
}
@@ -205,15 +249,38 @@ function normalizeVizSettings(input: VizSettings): VizSettings {
: 'hybrid',
},
fluid: {
spawn_density: clampVizMultiplier(input.fluid.spawn_density, 0.25, 3.0),
splat_force: clampVizMultiplier(input.fluid.splat_force, 0.25, 3.0),
decay: clampVizMultiplier(input.fluid.decay, 0.25, 3.0),
spawn_density: clampVizMultiplier(input.fluid.spawn_density, 0.25, 3.0),
splat_force: clampVizMultiplier(input.fluid.splat_force, 0.25, 3.0),
decay: clampVizMultiplier(input.fluid.decay, 0.25, 3.0),
background_persistence: clampVizMultiplier(input.fluid.background_persistence, 0.2, 3.0),
beat_burst_count: clampVizMultiplier(input.fluid.beat_burst_count, 0.25, 3.0),
treble_pulse: clampVizMultiplier(input.fluid.treble_pulse, 0.0, 3.0),
drag: clampVizMultiplier(input.fluid.drag, 0.5, 2.0),
blob_life: clampVizMultiplier(input.fluid.blob_life, 0.25, 3.0),
color_speed: clampVizMultiplier(input.fluid.color_speed, 0.0, 3.0),
beat_burst_count: clampVizMultiplier(input.fluid.beat_burst_count, 0.25, 3.0),
treble_pulse: clampVizMultiplier(input.fluid.treble_pulse, 0.0, 3.0),
drag: clampVizMultiplier(input.fluid.drag, 0.5, 2.0),
blob_life: clampVizMultiplier(input.fluid.blob_life, 0.25, 3.0),
color_speed: clampVizMultiplier(input.fluid.color_speed, 0.0, 3.0),
},
tendrils: {
noise_weight: clampVizMultiplier(input.tendrils.noise_weight, 0.0, 3.0),
noise_scale: clampVizMultiplier(input.tendrils.noise_scale, 0.0, 3.0),
flow_weight: clampVizMultiplier(input.tendrils.flow_weight, 0.0, 3.0),
flow_decay: clampVizMultiplier(input.tendrils.flow_decay, 0.0, 3.0),
force_weight: clampVizMultiplier(input.tendrils.force_weight, 0.0, 3.0),
speed_limit: clampVizMultiplier(input.tendrils.speed_limit, 0.0, 3.0),
damping: clampVizMultiplier(input.tendrils.damping, 0.0, 3.0),
hue_range: clampVizMultiplier(input.tendrils.hue_range, 0.0, 3.0),
mono_color: Boolean(input.tendrils.mono_color),
visualization_preset: typeof input.tendrils.visualization_preset === 'string' ? input.tendrils.visualization_preset : 'flow',
},
spirit: {
speed: clampVizMultiplier(input.spirit.speed, 0.1, 6.0),
die_speed: clampVizMultiplier(input.spirit.die_speed, 0.001, 0.12),
radius: clampVizMultiplier(input.spirit.radius, 0.5, 5.0),
curl_size: clampVizMultiplier(input.spirit.curl_size, 0.05, 3.0),
attraction: Math.max(-2, Math.min(2, Number.isFinite(input.spirit.attraction) ? input.spirit.attraction : 0.08)),
bloom_intensity: clampVizMultiplier(input.spirit.bloom_intensity, 0.0, 4.0),
bloom_threshold: clampVizMultiplier(input.spirit.bloom_threshold, 0.0, 1.0),
afterimage_damp: Math.max(0.60, Math.min(0.96, Number.isFinite(input.spirit.afterimage_damp) ? input.spirit.afterimage_damp : 0.88)),
visualization_preset: typeof input.spirit.visualization_preset === 'string' ? input.spirit.visualization_preset : 'spirit',
},
}
}
@@ -240,7 +307,7 @@ interface Props {
/** Optional SEK manifest dynamic palette (manifest.metadata.dynamic_palette). */
sekDynamicPalette?: SekDynamicPalette | null
/** Optional SEK visualizer mode preference (manifest.visualizer.mode). */
sekVisualizerMode?: 'muon' | 'fluid' | 'auto' | null
sekVisualizerMode?: 'muon' | 'fluid' | 'tendrils' | 'spirit' | 'auto' | null
/** Optional SEK visualizer settings (manifest.visualizer.settings). */
sekVisualizerSettings?: {
quality?: string
@@ -279,6 +346,29 @@ interface Props {
blob_life?: number
color_speed?: number
}
tendrils?: {
noise_weight?: number
noise_scale?: number
flow_weight?: number
flow_decay?: number
force_weight?: number
speed_limit?: number
damping?: number
hue_range?: number
mono_color?: boolean
visualization_preset?: string
}
spirit?: {
speed?: number
die_speed?: number
radius?: number
curl_size?: number
attraction?: number
bloom_intensity?: number
bloom_threshold?: number
afterimage_damp?: number
visualization_preset?: string
}
} | null
/** Artist name from source audio metadata. */
artist?: string
@@ -310,13 +400,6 @@ function rankLyricsTrackKey(key: string): number {
return 10
}
function labelLyricsTrackKey(key: string): string {
if (key === 'leading') return 'Leading'
if (key === 'backing') return 'Backing'
if (key === 'default') return 'Main'
return key.replace(/\b\w/g, (m) => m.toUpperCase())
}
// ─── Visual style definitions ─────────────────────────────────────────────────
const FONT_OPTIONS = [
@@ -1609,26 +1692,26 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, ana
const [visualStyle, setVisualStyle] = useState<LyricsVisualStyle>(() => {
const saved = localStorage.getItem('stemengine-lyric-visual-style') as LyricsVisualStyle | null
const hasRealVideo = !!(videoUrl || (videoPlaylist && videoPlaylist.length > 0))
const visualizerDefault = defaultStyle === 'muon' || defaultStyle === 'fluid'
const visualizerDefault = defaultStyle === 'muon' || defaultStyle === 'fluid' || defaultStyle === 'tendrils' || defaultStyle === 'spirit'
if (sekVisualizerMode === 'muon' || sekVisualizerMode === 'fluid') return sekVisualizerMode
if (sekVisualizerMode === 'muon' || sekVisualizerMode === 'fluid' || sekVisualizerMode === 'tendrils' || sekVisualizerMode === 'spirit') return sekVisualizerMode
// For SEK/no-video sessions we default to a visualizer. Keep persisted user
// choice only when it is also a visualizer mode.
if (visualizerDefault) {
if (saved === 'muon' || saved === 'fluid') return saved
if (saved === 'muon' || saved === 'fluid' || saved === 'tendrils' || saved === 'spirit') return saved
return defaultStyle
}
// Only restore saved 'video' style when a real video source is actually available;
// if Pixabay stock video is disabled and there's no embedded video, fall through to defaults.
if (saved && (saved !== 'video' || hasRealVideo) && ['scroll', 'fade', 'karaoke', 'glow', 'blur', 'video', 'muon', 'fluid'].includes(saved)) return saved
if (saved && (saved !== 'video' || hasRealVideo) && ['scroll', 'fade', 'karaoke', 'glow', 'blur', 'video', 'muon', 'fluid', 'tendrils', 'spirit'].includes(saved)) return saved
return defaultStyle ?? (hasElrc ? 'karaoke' : 'scroll')
})
const enforcedVisualizerMode: LyricsVisualStyle | null =
(sekVisualizerMode === 'muon' || sekVisualizerMode === 'fluid')
(sekVisualizerMode === 'muon' || sekVisualizerMode === 'fluid' || sekVisualizerMode === 'tendrils' || sekVisualizerMode === 'spirit')
? sekVisualizerMode
: ((defaultStyle === 'muon' || defaultStyle === 'fluid') ? defaultStyle : null)
: ((defaultStyle === 'muon' || defaultStyle === 'fluid' || defaultStyle === 'tendrils' || defaultStyle === 'spirit') ? defaultStyle : null)
const userOverrodeVisualizerRef = useRef(false)
const [showPerfSidebar, setShowPerfSidebar] = useState(false)
const manifestVizSettings = useMemo<VizSettings | null>(() => {
@@ -1637,6 +1720,8 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, ana
const fps = sekVisualizerSettings.fps_cap
const sm = sekVisualizerSettings.muon
const sf = sekVisualizerSettings.fluid
const st = sekVisualizerSettings.tendrils
const ss = sekVisualizerSettings.spirit
const patch: DeepPartial<VizSettings> = {
quality: quality === 'low' || quality === 'balanced' || quality === 'high' ? quality : undefined,
fps_cap: fps === 30 || fps === 60 || fps === 120 ? fps : undefined,
@@ -1674,6 +1759,29 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, ana
blob_life: sf?.blob_life,
color_speed: sf?.color_speed,
},
tendrils: {
noise_weight: st?.noise_weight,
noise_scale: st?.noise_scale,
flow_weight: st?.flow_weight,
flow_decay: st?.flow_decay,
force_weight: st?.force_weight,
speed_limit: st?.speed_limit,
damping: st?.damping,
hue_range: st?.hue_range,
mono_color: st?.mono_color,
visualization_preset: st?.visualization_preset,
},
spirit: {
speed: ss?.speed,
die_speed: ss?.die_speed,
radius: ss?.radius,
curl_size: ss?.curl_size,
attraction: ss?.attraction,
bloom_intensity: ss?.bloom_intensity,
bloom_threshold: ss?.bloom_threshold,
afterimage_damp: ss?.afterimage_damp,
visualization_preset: ss?.visualization_preset,
},
}
return normalizeVizSettings(deepMergeVizSettings(VIZ_DEFAULTS, patch))
}, [sekVisualizerSettings])
@@ -1750,7 +1858,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, ana
coverArtUrl,
thumbnailUrl,
})
const isVisualizerMode = visualStyle === 'muon' || visualStyle === 'fluid'
const isVisualizerMode = visualStyle === 'muon' || visualStyle === 'fluid' || visualStyle === 'tendrils' || visualStyle === 'spirit'
const resolvedVideoPlaylist = useMemo(
() => (videoPlaylist ?? [])
.filter((item) => typeof item?.url === 'string' && item.url.trim())
@@ -3195,45 +3303,10 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, ana
position: 'relative',
} as any}
>
{lyricsTrackKeys.length > 1 && (
<div
style={{
position: 'absolute',
top: performanceMode ? 84 : 10,
right: 10,
zIndex: 3,
display: 'flex',
gap: 6,
flexWrap: 'wrap',
justifyContent: 'flex-end',
}}
>
{lyricsTrackKeys.map((key) => (
<button
key={key}
type="button"
onClick={() => setSelectedLyricsTrack(key)}
style={{
border: '1px solid rgba(255,255,255,0.18)',
background: selectedLyricsTrack === key ? 'rgba(139,92,246,0.9)' : 'rgba(15,23,42,0.55)',
color: '#fff',
borderRadius: 999,
padding: '4px 10px',
fontSize: 11,
fontWeight: 600,
backdropFilter: 'blur(6px)',
}}
title={`Show ${labelLyricsTrackKey(key)} lyrics`}
>
{labelLyricsTrackKey(key)}
</button>
))}
</div>
)}
{isVisualizerMode && (
<div style={{ position: 'absolute', inset: 0, zIndex: 0, pointerEvents: 'none' }}>
<VisualizerBackdrop
mode={visualStyle === 'muon' ? 'muon' : visualStyle === 'fluid' ? 'fluid' : null}
mode={visualStyle === 'muon' ? 'muon' : visualStyle === 'fluid' ? 'fluid' : visualStyle === 'tendrils' ? 'tendrils' : visualStyle === 'spirit' ? 'spirit' : null}
featuresRef={featuresRef}
paletteRef={paletteRef}
width={visualizerSize.width}
@@ -3592,21 +3665,6 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, ana
{hasElrc && (
<span className="ml-2 shrink-0 text-[10px] ui-text-muted">ELRC</span>
)}
{lyricsTrackKeys.length > 1 && (
<div className="ml-2 flex items-center gap-1">
{lyricsTrackKeys.map((key) => (
<button
key={key}
type="button"
onClick={() => setSelectedLyricsTrack(key)}
className={`shrink-0 rounded-md px-2 py-1 text-[10px] font-medium transition-colors ${selectedLyricsTrack === key ? 'bg-violet-500 text-white' : isFullscreen ? 'text-slate-300 hover:bg-white/10' : 'bg-slate-200/70 text-slate-700 hover:bg-slate-300/70 dark:bg-slate-700/60 dark:text-slate-200 dark:hover:bg-slate-600/70'}`}
title={`Show ${labelLyricsTrackKey(key)} lyrics`}
>
{labelLyricsTrackKey(key)}
</button>
))}
</div>
)}
<div className="ml-auto flex gap-0.5 shrink-0 pl-1">
<button
onClick={() => { setShowStylePicker(p => !p); setShowEditor(false) }}
@@ -3651,6 +3709,8 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, ana
...((videoUrl || resolvedVideoPlaylist.length > 0) ? [{ value: 'video' as LyricsVisualStyle, label: 'Video' }] : []),
{ value: 'muon', label: 'Muon' },
{ value: 'fluid', label: 'Fluid' },
{ value: 'tendrils', label: 'Tendrils' },
{ value: 'spirit', label: 'Spirit' },
] as Array<{ value: LyricsVisualStyle; label: string }>).map((mode) => (
<button
key={mode.value}
@@ -3864,6 +3924,10 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, ana
vizSettings={liveVizSettings}
manifestVizSettings={manifestVizSettings}
patchVizSettings={patchVizSettings}
lyricsTrackKeys={lyricsTrackKeys}
selectedLyricsTrack={selectedLyricsTrack}
onSelectLyricsTrack={setSelectedLyricsTrack}
hasElrc={hasElrc}
onClose={() => setShowPerfSidebar(false)}
/>
)}
@@ -5111,7 +5175,7 @@ function LyricsCanvas({ lines, currentTimeRef, visualStyle, style, speakerColorF
return () => { cancelled = true }
}, [lines, currentTimeRef])
const isVisualizerVisualStyle = visualStyle === 'muon' || visualStyle === 'fluid'
const isVisualizerVisualStyle = visualStyle === 'muon' || visualStyle === 'fluid' || visualStyle === 'tendrils' || visualStyle === 'spirit'
return (
<div
+267 -69
View File
@@ -69,6 +69,29 @@ interface VizSettings {
blob_life: number
color_speed: number
}
tendrils: {
noise_weight: number
noise_scale: number
flow_weight: number
flow_decay: number
force_weight: number
speed_limit: number
damping: number
hue_range: number
mono_color: boolean
visualization_preset: string
}
spirit: {
speed: number
die_speed: number
radius: number
curl_size: number
attraction: number
bloom_intensity: number
bloom_threshold: number
afterimage_damp: number
visualization_preset: string
}
}
const VIZ_DEFAULTS: VizSettings = {
@@ -108,6 +131,29 @@ const VIZ_DEFAULTS: VizSettings = {
blob_life: 1.0,
color_speed: 1.0,
},
tendrils: {
noise_weight: 1.0,
noise_scale: 1.0,
flow_weight: 1.0,
flow_decay: 1.0,
force_weight: 1.0,
speed_limit: 1.0,
damping: 1.0,
hue_range: 1.0,
mono_color: false,
visualization_preset: 'flow',
},
spirit: {
speed: 1.0,
die_speed: 1.0,
radius: 1.5,
curl_size: 0.4,
attraction: 0.08,
bloom_intensity: 1.5,
bloom_threshold: 0.05,
afterimage_damp: 0.88,
visualization_preset: 'spirit',
},
}
// Named muon presets — applied to vizSettings.muon only (leaves other keys untouched).
@@ -214,6 +260,10 @@ interface PerformanceSidebarProps {
vizSettings: VizSettings
manifestVizSettings: VizSettings | null
patchVizSettings: (patch: DeepPartial<VizSettings>) => void
lyricsTrackKeys: string[]
selectedLyricsTrack: string
onSelectLyricsTrack: (key: string) => void
hasElrc: boolean
onClose: () => void
}
@@ -252,6 +302,10 @@ export default function PerformanceSidebar({
vizSettings,
manifestVizSettings,
patchVizSettings,
lyricsTrackKeys,
selectedLyricsTrack,
onSelectLyricsTrack,
hasElrc,
onClose,
}: PerformanceSidebarProps) {
const autoHideTimerRef = useRef<number | null>(null)
@@ -275,8 +329,8 @@ export default function PerformanceSidebar({
}, [scheduleAutoHide])
const activeOffset = getLayoutOffset(style)
const isVisualizerMode = visualStyle === 'muon' || visualStyle === 'fluid'
const isStaticMode = visualStyle !== 'muon' && visualStyle !== 'fluid' && visualStyle !== 'video'
const isVisualizerMode = visualStyle === 'muon' || visualStyle === 'fluid' || visualStyle === 'tendrils' || visualStyle === 'spirit'
const isStaticMode = visualStyle !== 'muon' && visualStyle !== 'fluid' && visualStyle !== 'tendrils' && visualStyle !== 'spirit' && visualStyle !== 'video'
const patchOffset = (axis: 'x' | 'y', value: number) => {
if (style.videoDisplayMode === 'line') {
@@ -349,74 +403,141 @@ export default function PerformanceSidebar({
<div className="px-3 py-3 space-y-4">
{/* ── BACKGROUND ── */}
<section className="space-y-2">
<div className="text-xs font-semibold tracking-wide text-slate-200">BACKGROUND</div>
<div className={isNarrow ? 'flex flex-wrap gap-2' : 'grid grid-cols-2 gap-2'}>
{[
{ key: 'static', label: 'Static', value: 'karaoke' as LyricsVisualStyle, active: isStaticMode },
{ key: 'muon', label: 'Muon', value: 'muon' as LyricsVisualStyle, active: visualStyle === 'muon' },
{ key: 'fluid', label: 'Fluid', value: 'fluid' as LyricsVisualStyle, active: visualStyle === 'fluid' },
...(hasVideo ? [{ key: 'video', label: 'Video', value: 'video' as LyricsVisualStyle, active: visualStyle === 'video' }] : []),
].map((mode) => (
<button
key={mode.key}
type="button"
onClick={() => setVisualStyle(mode.value)}
className={`rounded-md px-2 py-2 text-xs border transition-colors ${isNarrow ? 'min-w-[70px] flex-1' : ''} ${mode.active ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
>
{mode.label}
</button>
))}
{/* ── LYRICS TRACKS ── */}
{lyricsTrackKeys.length > 1 && (
<details className="rounded-md border border-white/10 bg-white/[0.02]" open>
<summary className="cursor-pointer select-none px-2.5 py-2 text-xs font-semibold tracking-wide text-slate-200">
LYRICS TRACKS
</summary>
<div className="px-2.5 pb-2.5 pt-1">
<div className={isNarrow ? 'flex flex-wrap gap-2' : 'grid grid-cols-2 gap-2'}>
{lyricsTrackKeys.map((key) => (
<button
key={key}
type="button"
onClick={() => onSelectLyricsTrack(key)}
className={`rounded-md px-2 py-2 text-xs border transition-colors ${isNarrow ? 'flex-1 min-w-[70px]' : ''} ${
selectedLyricsTrack === key
? 'border-violet-400 bg-violet-500/20 text-white'
: 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'
}`}
title={`Show ${key.charAt(0).toUpperCase() + key.slice(1)} lyrics`}
>
{key.charAt(0).toUpperCase() + key.slice(1)}
</button>
))}
</div>
</div>
</details>
)}
{/* ── LINE DISPLAY ── */}
<details className="rounded-md border border-white/10 bg-white/[0.02]" open>
<summary className="cursor-pointer select-none px-2.5 py-2 text-xs font-semibold tracking-wide text-slate-200">
DISPLAY
</summary>
<div className="px-2.5 pb-2.5 pt-1">
<div className={isNarrow ? 'flex flex-wrap gap-2' : 'grid grid-cols-2 gap-2'}>
{(['scroll', 'line', 'word', 'syllable'] as const).map((mode) => (
<button
key={mode}
type="button"
disabled={(mode === 'word' || mode === 'syllable') && !hasElrc}
onClick={() => patchStyle({ videoDisplayMode: mode })}
title={(mode === 'word' || mode === 'syllable') && !hasElrc ? 'No word-timing data available' : `Show ${mode} display`}
className={`rounded-md px-2 py-2 text-xs border transition-colors capitalize ${isNarrow ? 'flex-1 min-w-[70px]' : ''} ${
style.videoDisplayMode === mode
? 'border-violet-400 bg-violet-500/20 text-white'
: 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'
} disabled:opacity-40 disabled:cursor-not-allowed`}
>
{mode}
</button>
))}
</div>
</div>
</section>
</details>
{/* ── BACKGROUND ── */}
<details className="rounded-md border border-white/10 bg-white/[0.02]" open>
<summary className="cursor-pointer select-none px-2.5 py-2 text-xs font-semibold tracking-wide text-slate-200">
BACKGROUND
</summary>
<div className="px-2.5 pb-2.5 pt-1">
<div className={isNarrow ? 'flex flex-wrap gap-2' : 'grid grid-cols-2 gap-2'}>
{[
{ key: 'static', label: 'Static', value: 'karaoke' as LyricsVisualStyle, active: isStaticMode },
{ key: 'muon', label: 'Muon', value: 'muon' as LyricsVisualStyle, active: visualStyle === 'muon' },
{ key: 'fluid', label: 'Fluid', value: 'fluid' as LyricsVisualStyle, active: visualStyle === 'fluid' },
{ key: 'tendrils', label: 'Tendrils', value: 'tendrils' as LyricsVisualStyle, active: visualStyle === 'tendrils' },
{ key: 'spirit', label: 'Spirit', value: 'spirit' as LyricsVisualStyle, active: visualStyle === 'spirit' },
...(hasVideo ? [{ key: 'video', label: 'Video', value: 'video' as LyricsVisualStyle, active: visualStyle === 'video' }] : []),
].map((mode) => (
<button
key={mode.key}
type="button"
onClick={() => setVisualStyle(mode.value)}
className={`rounded-md px-2 py-2 text-xs border transition-colors ${isNarrow ? 'min-w-[70px] flex-1' : ''} ${mode.active ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
>
{mode.label}
</button>
))}
</div>
</div>
</details>
{/* ── STEMS ── */}
<section className="space-y-2">
<div className="text-xs font-semibold tracking-wide text-slate-200">STEMS</div>
<div className="space-y-2">
{stems?.length ? stems.map((stem) => (
<div key={stem.name} className="space-y-1.5">
<div className="flex items-center gap-2">
<span className="text-xs font-medium text-slate-300 flex-1 truncate">{stem.label}</span>
<button
type="button"
onClick={() => toggleStemMute(stem.name)}
className={`w-6 h-6 text-[10px] rounded ${mutedStems[stem.name] ? 'bg-red-500 text-white' : 'bg-white/10 text-slate-300'}`}
>
M
</button>
<button
type="button"
onClick={() => toggleStemSolo(stem.name)}
className={`w-6 h-6 text-[10px] rounded ${soloedStems[stem.name] ? 'bg-yellow-300 text-black' : 'bg-white/10 text-slate-300'}`}
>
S
</button>
<details className="rounded-md border border-white/10 bg-white/[0.02]" open>
<summary className="cursor-pointer select-none px-2.5 py-2 text-xs font-semibold tracking-wide text-slate-200">
STEMS
</summary>
<div className="px-2.5 pb-2.5 pt-1">
<div className="space-y-2">
{stems?.length ? stems.map((stem) => (
<div key={stem.name} className="space-y-1.5">
<div className="flex items-center gap-2">
<span className="text-xs font-medium text-slate-300 flex-1 truncate">{stem.label}</span>
<button
type="button"
onClick={() => toggleStemMute(stem.name)}
className={`w-6 h-6 text-[10px] rounded ${mutedStems[stem.name] ? 'bg-red-500 text-white' : 'bg-white/10 text-slate-300'}`}
>
M
</button>
<button
type="button"
onClick={() => toggleStemSolo(stem.name)}
className={`w-6 h-6 text-[10px] rounded ${soloedStems[stem.name] ? 'bg-yellow-300 text-black' : 'bg-white/10 text-slate-300'}`}
>
S
</button>
</div>
<div className="flex items-center gap-2">
<input
type="range"
min={0}
max={2}
step={0.05}
value={gains[stem.name] ?? 1}
onChange={(e) => setStemGain(stem.name, parseFloat(e.target.value))}
className="w-full h-1.5 accent-brand-500"
/>
<span className="text-[10px] tabular-nums text-slate-400 w-11 text-right">{Math.round((gains[stem.name] ?? 1) * 100)}%</span>
</div>
</div>
<div className="flex items-center gap-2">
<input
type="range"
min={0}
max={2}
step={0.05}
value={gains[stem.name] ?? 1}
onChange={(e) => setStemGain(stem.name, parseFloat(e.target.value))}
className="w-full h-1.5 accent-brand-500"
/>
<span className="text-[10px] tabular-nums text-slate-400 w-11 text-right">{Math.round((gains[stem.name] ?? 1) * 100)}%</span>
</div>
</div>
)) : (
<p className="text-[11px] text-slate-400">No stems — single track mode</p>
)}
)) : (
<p className="text-[11px] text-slate-400">No stems — single track mode</p>
)}
</div>
</div>
</section>
</details>
{/* ── LYRICS ── */}
<section className="space-y-2">
<div className="text-xs font-semibold tracking-wide text-slate-200">LYRICS</div>
<div className="space-y-2.5">
<details className="rounded-md border border-white/10 bg-white/[0.02]">
<summary className="cursor-pointer select-none px-2.5 py-2 text-xs font-semibold tracking-wide text-slate-200">
LYRICS STYLE
</summary>
<div className="space-y-2.5 px-2.5 pb-2.5 pt-1">
<div className="flex items-center gap-2">
<select
value={style.fontFamily}
@@ -483,12 +604,14 @@ export default function PerformanceSidebar({
<input type="range" min={-50} max={50} step={1} value={activeOffset.y} onChange={(e) => patchOffset('y', parseInt(e.target.value, 10))} className="w-full h-1.5 accent-brand-500" />
</div>
</div>
</section>
</details>
{/* ── VISUALIZER ── */}
<section className="space-y-2">
<div className="text-xs font-semibold tracking-wide text-slate-200">VISUALIZER</div>
<div className="space-y-2.5">
<details className="rounded-md border border-white/10 bg-white/[0.02]">
<summary className="cursor-pointer select-none px-2.5 py-2 text-xs font-semibold tracking-wide text-slate-200">
VISUALIZER
</summary>
<div className="space-y-2.5 px-2.5 pb-2.5 pt-1">
{/* Quality */}
<div className="space-y-1.5">
@@ -520,7 +643,7 @@ export default function PerformanceSidebar({
{!isVisualizerMode && (
<p className="text-[11px] text-slate-400">
Switch background to Muon or Fluid to access mode-specific controls.
Switch background to Muon, Fluid, Tendrils, or Spirit to access mode-specific controls.
</p>
)}
@@ -662,6 +785,81 @@ export default function PerformanceSidebar({
</>
)}
{/* ── Tendrils controls ── */}
{visualStyle === 'tendrils' && (
<>
<div className="space-y-1.5">
<div className="text-[10px] text-slate-400">Pattern preset</div>
<div className="flex flex-wrap gap-1.5">
{(['flow', 'scatter', 'pulse', 'ethereal', 'hybrid'] as const).map((preset) => {
const active = vizSettings.tendrils.visualization_preset === preset
return (
<button
key={preset}
type="button"
onClick={() => patchVizSettings({ tendrils: { visualization_preset: preset } })}
className={`rounded px-2 py-1 text-[11px] border transition-colors ${active ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
>
{preset}
</button>
)
})}
</div>
</div>
{slider('Noise', vizSettings.tendrils.noise_weight, formatMultiplier(vizSettings.tendrils.noise_weight), 0, 3, 0.05, (v) => patchVizSettings({ tendrils: { noise_weight: v } }))}
{slider('Flow', vizSettings.tendrils.flow_weight, formatMultiplier(vizSettings.tendrils.flow_weight), 0, 3, 0.05, (v) => patchVizSettings({ tendrils: { flow_weight: v } }))}
{slider('Speed', vizSettings.tendrils.speed_limit, formatMultiplier(vizSettings.tendrils.speed_limit), 0, 3, 0.05, (v) => patchVizSettings({ tendrils: { speed_limit: v } }))}
{slider('Damping', vizSettings.tendrils.damping, formatMultiplier(vizSettings.tendrils.damping), 0, 3, 0.05, (v) => patchVizSettings({ tendrils: { damping: v } }))}
{slider('Hue range', vizSettings.tendrils.hue_range, formatMultiplier(vizSettings.tendrils.hue_range), 0, 3, 0.05, (v) => patchVizSettings({ tendrils: { hue_range: v } }))}
<label className="flex items-center gap-2 cursor-pointer">
<div
role="checkbox"
aria-checked={vizSettings.tendrils.mono_color}
tabIndex={0}
onClick={() => patchVizSettings({ tendrils: { mono_color: !vizSettings.tendrils.mono_color } })}
onKeyDown={(e) => { if (e.key === ' ' || e.key === 'Enter') patchVizSettings({ tendrils: { mono_color: !vizSettings.tendrils.mono_color } }) }}
className={`w-8 h-4 rounded-full transition-colors ${vizSettings.tendrils.mono_color ? 'bg-violet-500' : 'bg-white/20'} relative`}
>
<div className={`absolute top-0.5 w-3 h-3 rounded-full bg-white transition-transform ${vizSettings.tendrils.mono_color ? 'translate-x-4' : 'translate-x-0.5'}`} />
</div>
<span className="text-[11px] text-slate-300">Mono color</span>
</label>
</>
)}
{/* ── Spirit controls ── */}
{visualStyle === 'spirit' && (
<>
<div className="space-y-1.5">
<div className="text-[10px] text-slate-400">Pattern preset</div>
<div className="flex flex-wrap gap-1.5">
{(['spirit', 'storm', 'drift', 'pulse', 'aurora'] as const).map((preset) => {
const active = vizSettings.spirit.visualization_preset === preset
return (
<button
key={preset}
type="button"
onClick={() => patchVizSettings({ spirit: { visualization_preset: preset } })}
className={`rounded px-2 py-1 text-[11px] border transition-colors ${active ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
>
{preset}
</button>
)
})}
</div>
</div>
{slider('Speed', vizSettings.spirit.speed, formatMultiplier(vizSettings.spirit.speed), 0.1, 6, 0.05, (v) => patchVizSettings({ spirit: { speed: v } }))}
{slider('Die speed', vizSettings.spirit.die_speed, vizSettings.spirit.die_speed.toFixed(3), 0.001, 0.12, 0.001, (v) => patchVizSettings({ spirit: { die_speed: v } }))}
{slider('Curl', vizSettings.spirit.curl_size, formatMultiplier(vizSettings.spirit.curl_size), 0.05, 3, 0.05, (v) => patchVizSettings({ spirit: { curl_size: v } }))}
{slider('Attraction', vizSettings.spirit.attraction, vizSettings.spirit.attraction.toFixed(2), -2, 2, 0.05, (v) => patchVizSettings({ spirit: { attraction: v } }))}
{slider('Bloom', vizSettings.spirit.bloom_intensity, formatMultiplier(vizSettings.spirit.bloom_intensity), 0, 4, 0.05, (v) => patchVizSettings({ spirit: { bloom_intensity: v } }))}
{slider('Afterimage', vizSettings.spirit.afterimage_damp, vizSettings.spirit.afterimage_damp.toFixed(2), 0.6, 0.96, 0.01, (v) => patchVizSettings({ spirit: { afterimage_damp: v } }))}
</>
)}
<button
type="button"
onClick={() => patchVizSettings(manifestVizSettings ?? VIZ_DEFAULTS)}
@@ -670,7 +868,7 @@ export default function PerformanceSidebar({
{manifestVizSettings ? 'Reset to song defaults' : 'Reset to defaults'}
</button>
</div>
</section>
</details>
</div>
</div>
)
@@ -5,7 +5,9 @@ import type { VisualizerPalette } from '../../hooks/useVisualizerPalette'
import type { VisualizerSettings } from './VisualizerBackdrop'
import { destroyFluidSim, initFluidSim, splatFluid, updateFluidConfig } from './fluid/fluidSim'
type FluidSettings = NonNullable<VisualizerSettings['fluid']>
type FluidSettings = NonNullable<VisualizerSettings['fluid']> & {
visualization_preset?: string
}
interface FluidVisualizerCanvasProps {
featuresRef: MutableRefObject<AudioFeatures>
@@ -25,7 +27,7 @@ function clamp(v: number, lo: number, hi: number): number {
return Math.max(lo, Math.min(hi, v))
}
function hexToRgb(hex: string): { r: number; g: number; b: number } {
function hexToRgbUnit(hex: string): { r: number; g: number; b: number } {
const m = /^#?([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})$/i.exec(hex.trim())
if (!m) return { r: 0.5, g: 0.5, b: 0.8 }
return {
@@ -35,28 +37,118 @@ function hexToRgb(hex: string): { r: number; g: number; b: number } {
}
}
function fluidDensityDissipation(s?: FluidSettings, kick = 0): number {
const base = 0.98 - (s?.background_persistence ?? 1) * 0.025
return clamp(base - kick * 0.04 * (s?.decay ?? 1), 0.85, 0.999)
function hexToRgb255(hex: string): { r: number; g: number; b: number } {
const m = /^#?([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})$/i.exec(hex.trim())
if (!m) return { r: 128, g: 128, b: 204 }
return {
r: parseInt(m[1], 16),
g: parseInt(m[2], 16),
b: parseInt(m[3], 16),
}
}
function fluidVelocityDissipation(s?: FluidSettings, rms = 0): number {
const base = 0.96 - (1 / Math.max(0.5, s?.drag ?? 1)) * 0.06
return clamp(base - rms * 0.04, 0.80, 0.999)
function blendColor(
a: { r: number; g: number; b: number },
b: { r: number; g: number; b: number },
t: number,
): { r: number; g: number; b: number } {
return {
r: a.r + (b.r - a.r) * t,
g: a.g + (b.g - a.g) * t,
b: a.b + (b.b - a.b) * t,
}
}
function fluidCurl(s?: FluidSettings, treble = 0, dropBoost = 0): number {
const base = 15 + (s?.splat_force ?? 1) * 12
return clamp(base + treble * 8 + dropBoost * 20, 0, 80)
type FluidPreset = {
label: string
densityBase: number
velocityBase: number
curlBase: number
pressureBase: number
splatRadiusBase: number
bloomIntensityBase: number
sunraysWeightBase: number
dropBoostDecay: number
ambientForce: number
burstForce: number
sweepForce: number
}
function fluidSplatRadius(s?: FluidSettings): number {
return clamp(0.18 + (s?.blob_life ?? 1) * 0.06, 0.05, 0.5)
const FLUID_PRESETS: Record<string, FluidPreset> = {
ink: {
label: 'Ink',
densityBase: 0.996,
velocityBase: 0.975,
curlBase: 8,
pressureBase: 0.70,
splatRadiusBase: 0.30,
bloomIntensityBase: 0.55,
sunraysWeightBase: 0.25,
dropBoostDecay: 0.97,
ambientForce: 0.60,
burstForce: 1.10,
sweepForce: 0.80,
},
lava: {
label: 'Lava',
densityBase: 0.999,
velocityBase: 0.994,
curlBase: 18,
pressureBase: 0.85,
splatRadiusBase: 0.22,
bloomIntensityBase: 1.30,
sunraysWeightBase: 0.55,
dropBoostDecay: 0.96,
ambientForce: 0.80,
burstForce: 0.90,
sweepForce: 1.20,
},
storm: {
label: 'Storm',
densityBase: 0.960,
velocityBase: 0.950,
curlBase: 50,
pressureBase: 0.90,
splatRadiusBase: 0.12,
bloomIntensityBase: 0.70,
sunraysWeightBase: 1.40,
dropBoostDecay: 0.94,
ambientForce: 1.40,
burstForce: 1.80,
sweepForce: 1.50,
},
mist: {
label: 'Mist',
densityBase: 0.993,
velocityBase: 0.988,
curlBase: 5,
pressureBase: 0.60,
splatRadiusBase: 0.40,
bloomIntensityBase: 2.00,
sunraysWeightBase: 0.15,
dropBoostDecay: 0.97,
ambientForce: 0.50,
burstForce: 0.65,
sweepForce: 0.55,
},
plasma: {
label: 'Plasma',
densityBase: 0.972,
velocityBase: 0.962,
curlBase: 38,
pressureBase: 0.88,
splatRadiusBase: 0.14,
bloomIntensityBase: 2.20,
sunraysWeightBase: 1.60,
dropBoostDecay: 0.95,
ambientForce: 1.20,
burstForce: 1.50,
sweepForce: 1.80,
},
}
function fluidBloomIntensity(s?: FluidSettings, intensity = 0, dropBoost = 0): number {
const base = 0.55 + Math.max(0, s?.treble_pulse ?? 1) * 0.28
return clamp(base + intensity * 1.55 + dropBoost * 0.85, 0.35, 3.0)
function fluidBloomIntensity(s?: FluidSettings, intensity = 0, dropBoost = 0, base = 0.8): number {
return clamp(base + Math.max(0, s?.treble_pulse ?? 1) * 0.28 + intensity * 1.55 + dropBoost * 0.85, 0.35, 3.5)
}
function fluidBloomThreshold(intensity = 0): number {
@@ -102,8 +194,9 @@ export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width,
return () => window.cancelAnimationFrame(rafFallback)
}
const s0 = settingsRef.current
const q = qualityRef.current
const presetKey0 = settingsRef.current?.visualization_preset ?? 'lava'
const preset0 = FLUID_PRESETS[presetKey0] ?? FLUID_PRESETS.lava
initFluidSim(canvas, {
TRIGGER: 'hover',
IMMEDIATE: false,
@@ -111,18 +204,23 @@ export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width,
COLORFUL: false,
SIM_RESOLUTION: q === 'high' ? 160 : q === 'low' ? 96 : 128,
DYE_RESOLUTION: q === 'high' ? 1024 : q === 'low' ? 512 : 768,
DENSITY_DISSIPATION: fluidDensityDissipation(s0),
VELOCITY_DISSIPATION: fluidVelocityDissipation(s0),
PRESSURE: 0.8,
PRESSURE_ITERATIONS: 20,
CURL: fluidCurl(s0),
SPLAT_RADIUS: fluidSplatRadius(s0),
DENSITY_DISSIPATION: preset0.densityBase,
VELOCITY_DISSIPATION: preset0.velocityBase,
PRESSURE: preset0.pressureBase,
PRESSURE_ITERATIONS: q === 'low' ? 10 : q === 'high' ? 25 : 20,
CURL: preset0.curlBase,
SPLAT_RADIUS: preset0.splatRadiusBase,
SPLAT_FORCE: 6000,
SHADING: true,
SHADING: q !== 'low',
BLOOM: true,
BLOOM_INTENSITY: 0.8,
SUNRAYS: false,
BACK_COLOR: hexToRgb(paletteRef.current.backgroundColor),
BLOOM_INTENSITY: preset0.bloomIntensityBase,
BLOOM_SOFT_KNEE: 0.7,
BLOOM_ITERATIONS: q === 'low' ? 4 : 8,
BLOOM_RESOLUTION: q === 'low' ? 128 : 256,
SUNRAYS: true,
SUNRAYS_RESOLUTION: q === 'low' ? 96 : 196,
SUNRAYS_WEIGHT: preset0.sunraysWeightBase,
BACK_COLOR: hexToRgb255(paletteRef.current.backgroundColor),
})
let raf = 0
@@ -176,27 +274,40 @@ export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width,
dropBoost = 1
lastDropAt = ts
}
dropBoost *= 0.96
const densityScale = Math.max(0.2, s?.spawn_density ?? 1)
const splatForce = Math.max(0.2, s?.splat_force ?? 1)
const beatBurstCount = Math.max(0, s?.beat_burst_count ?? 1)
const treblePulse = Math.max(0, s?.treble_pulse ?? 1)
const colorSpeed = Math.max(0, s?.color_speed ?? 1)
const presetKey = s?.visualization_preset ?? 'lava'
const preset = FLUID_PRESETS[presetKey] ?? FLUID_PRESETS.lava
dropBoost *= preset.dropBoostDecay
const bgRgb = hexToRgb255(paletteRef.current.backgroundColor)
const highlightRgb = hexToRgb255(paletteRef.current.highlightColor)
updateFluidConfig(canvas, {
DENSITY_DISSIPATION: fluidDensityDissipation(s, kick),
VELOCITY_DISSIPATION: fluidVelocityDissipation(s, rms),
CURL: fluidCurl(s, treble, dropBoost),
SPLAT_RADIUS: fluidSplatRadius(s),
BLOOM_INTENSITY: fluidBloomIntensity(s, audioIntensity, dropBoost),
DENSITY_DISSIPATION: clamp(preset.densityBase - kick * 0.04 * (s?.decay ?? 1), 0.85, 0.999),
VELOCITY_DISSIPATION: clamp(preset.velocityBase - rms * 0.04, 0.80, 0.999),
CURL: clamp(preset.curlBase + treble * 8 + dropBoost * 20, 0, 80),
PRESSURE: clamp(preset.pressureBase + rms * 0.32, 0.40, 0.92),
SPLAT_RADIUS: clamp(preset.splatRadiusBase * clamp(0.6 + kick * 0.8, 0.4, 1.4), 0.05, 0.5),
BLOOM_INTENSITY: fluidBloomIntensity(s, audioIntensity, dropBoost, preset.bloomIntensityBase),
BLOOM_THRESHOLD: fluidBloomThreshold(audioIntensity),
BACK_COLOR: hexToRgb(paletteRef.current.backgroundColor),
BLOOM_SOFT_KNEE: clamp(0.65 - kick * 0.35, 0.20, 0.72),
SUNRAYS_WEIGHT: clamp(preset.sunraysWeightBase + bass * 1.2 + dropBoost * 0.8, 0.1, 2.0),
BACK_COLOR: dropBoost > 0.08
? blendColor(bgRgb, highlightRgb, clamp(dropBoost * 0.15, 0, 0.4))
: bgRgb,
PAUSED: false,
})
// WebGL sim renders in its own RAF; fpsCap throttles splat injection only.
const minSplatMs = Math.max(minFrameMs, isLow ? 45 : 25)
const minSplatMs = dropBoost > 0.5
? minFrameMs
: Math.max(minFrameMs, isLow ? 45 : 25)
if (ts - lastSplatAt < minSplatMs - 1) {
prevBeatPhase = beatPhase
return
@@ -206,14 +317,14 @@ export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width,
// Ambient splat.
if (Math.random() < (0.04 + rms * 0.35 + phasePulse * 0.08 + idle * 0.04) * densityScale) {
const angle = Math.random() * Math.PI * 2
const force = (0.2 + rms * 0.8 + treble * 0.3 * treblePulse + idle * 0.12) * splatForce * 400
const force = (0.2 + rms * 0.8 + treble * 0.3 * treblePulse + idle * 0.12) * splatForce * 400 * preset.ambientForce
splatFluid(
canvas,
Math.random(),
Math.random(),
Math.cos(angle) * force,
Math.sin(angle) * force,
hexToRgb(nextPaletteColor(colorSpeed * 0.25)),
hexToRgbUnit(nextPaletteColor(colorSpeed * 0.25)),
)
}
@@ -222,7 +333,7 @@ export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width,
const count = Math.max(1, Math.round((3 + kick * 5 + dropBoost * 4) * beatBurstCount))
for (let i = 0; i < count; i += 1) {
const angle = Math.random() * Math.PI * 2
const force = (0.6 + bass * 1.2 + dropBoost * 0.6) * splatForce * 800
const force = (0.6 + bass * 1.2 + dropBoost * 0.6) * splatForce * 800 * preset.burstForce
splatFluid(
canvas,
Math.random(),
@@ -230,8 +341,8 @@ export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width,
Math.cos(angle) * force,
Math.sin(angle) * force,
dropTriggered
? hexToRgb(paletteRef.current.highlightColor)
: hexToRgb(nextPaletteColor(colorSpeed * 0.7)),
? hexToRgbUnit(paletteRef.current.highlightColor)
: hexToRgbUnit(nextPaletteColor(colorSpeed * 0.7)),
)
}
}
@@ -239,17 +350,18 @@ export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width,
// Beat-phase sweep.
if (beatPhase > 0.001 || f.beatInfo.bpm > 0) {
const phaseAngle = beatPhase * Math.PI * 2
const orbit = 0.22 + kick * 0.10 + dropBoost * 0.08
if (Math.random() < 0.08 + phasePulse * 0.22 + dropBoost * 0.18) {
const force = (0.28 + kick * 0.5 + phasePulse * 0.4) * splatForce * 500
const force = (0.28 + kick * 0.5 + phasePulse * 0.4) * splatForce * 500 * preset.sweepForce
splatFluid(
canvas,
0.5 + Math.cos(phaseAngle) * 0.28,
0.5 + Math.sin(phaseAngle) * 0.20,
0.5 + Math.cos(phaseAngle) * orbit,
0.5 + Math.sin(phaseAngle) * (orbit * 0.72),
Math.cos(phaseAngle + Math.PI / 2) * force,
Math.sin(phaseAngle + Math.PI / 2) * force,
dropBoost > 0.18
? hexToRgb(paletteRef.current.highlightColor)
: hexToRgb(nextPaletteColor(colorSpeed * 0.9)),
? hexToRgbUnit(paletteRef.current.highlightColor)
: hexToRgbUnit(nextPaletteColor(colorSpeed * 0.9)),
)
}
}
@@ -265,7 +377,7 @@ export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width,
// Only restart when data-source refs change.
// Settings, fpsCap, and quality are read via refs each frame — no restart on changes.
// width/height are consumed each frame as fallback to clientWidth/clientHeight.
}, [featuresRef, paletteRef])
}, [featuresRef, paletteRef, quality])
return <canvas ref={canvasRef} className="h-full w-full" aria-hidden="true" />
}
@@ -3,7 +3,7 @@ import type { MutableRefObject } from 'react'
import type { AudioFeatures } from '../../hooks/useAudioFeatures'
import type { VisualizerPalette } from '../../hooks/useVisualizerPalette'
import type { VisualizerSettings } from './VisualizerBackdrop'
import { centeredNoise2D } from './muon/noise'
import { curlNoise2D } from './muon/noise'
import { patternFromPreset, samplePattern, type MuonPatternFamily } from './muon/patterns'
type MuonSettings = NonNullable<VisualizerSettings['muon']>
@@ -84,6 +84,7 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
let prevBeatPhase = 0
let dropBoost = 0
let lastDropAt = 0
let lastPatternCycleAt = 0
let patternShift = 0
let poolCap = maxParticlesByQuality('balanced')
@@ -98,6 +99,7 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
cy: number,
ring: number,
divisions: number,
t: number,
beatPhase: number,
hueBase: number,
aperture: number,
@@ -110,17 +112,21 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
alphaSeed: number,
sizeSeed: number,
dynamicRadius: number,
drum: number,
) => {
const emitOne = (phaseOffset: number) => {
const p = particles[nextPoolIdx]
nextPoolIdx = (nextPoolIdx + 1) % particles.length
const t = ((performance.now() * 0.0004) + ring / Math.max(1, divisions) + phaseOffset) * Math.PI * 2
const tFinal = t + phaseOffset * Math.PI * 2
const pattern = patternFromPreset(patternPreset, ring + patternShift)
const ringNorm = ring / Math.max(1, divisions - 1)
const sample = samplePattern(pattern, t, ringNorm, aperture, beatPhase)
const nx = centeredNoise2D(sample.x * noiseScale + ringNorm * 1.3, sample.y * noiseScale, ring * 17.17)
const ny = centeredNoise2D(sample.y * noiseScale, sample.x * noiseScale + ringNorm * 1.9, ring * 29.37)
const sample = samplePattern(pattern, tFinal, ringNorm, aperture, beatPhase)
const { nx, ny } = curlNoise2D(
sample.x * noiseScale + ringNorm * 1.3,
sample.y * noiseScale + ringNorm * 0.9,
ring * 17.17,
)
const noiseAmp = noiseForce * 0.32
const sx = sample.x + nx * noiseAmp
@@ -130,7 +136,7 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
const tvx = sample.tx + nx * noiseAmp * 0.4
const tvy = sample.ty + ny * noiseAmp * 0.4
const speed = 0.12 + ringNorm * 0.22 + sizeSeed * 0.08
const speed = (0.02 + ringNorm * 0.06 + sizeSeed * 0.02) * (1 + drum * 0.4)
p.active = true
p.x = ex
@@ -191,7 +197,7 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
const idleSpeed = Math.max(0, s?.idle_speed ?? 1)
const monoColor = Boolean(s?.mono_color)
const divisions = Math.max(8, Math.min(48, Math.round(s?.divisions ?? 16)))
const lifespan = Math.max(45, Math.min(420, Math.round(s?.lifespan ?? 180)))
const lifespan = Math.max(20, Math.min(420, Math.round(s?.lifespan ?? 60)))
const mirror = Boolean(s?.particle_mirror)
const dynamicRadiusCfg = Math.max(0.2, s?.dynamic_radius ?? 1)
const deltaLimit = Math.max(0.0005, s?.delta_response_limit ?? 0.005)
@@ -213,11 +219,17 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
if (dropTriggered) {
dropBoost = 1
lastDropAt = now
patternShift += 1
patternShift += 2
lastPatternCycleAt = now
}
dropBoost *= 0.965
const phaseWrapped = beatPhase < prevBeatPhase - 0.35
const beatCooldownSec = Math.max(0.6, 1.2 / Math.max(0.1, beatSensitivity))
if ((f.beatDetected || phaseWrapped) && drum > 0.3 && now - lastPatternCycleAt > beatCooldownSec) {
patternShift += 1
lastPatternCycleAt = now
}
const idle = 0.5 + 0.5 * Math.sin(now * 1.2 * Math.max(0.01, idleSpeed))
const hue = monoColor
@@ -249,33 +261,40 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
const cx = w / 2
const cy = h / 2
const base = Math.min(w, h) * 0.17 * radiusScale
const glow = (10 + drum * 22 + phasePulse * 7 + dropBoost * 18) * glowIntensity
const emitDensity = (s?.particle_count ?? 220) / 220
const emitCount = Math.max(1, Math.round((0.8 + drum * 2.4 + phasePulse * 0.8 + dropBoost * 2.2) * emitDensity * (divisions / 16) * (0.75 + beatSensitivity * 0.25)))
const lifeFrames = Math.round(lifespan * (0.8 + (1 - drum) * 0.25))
const maxTotalPerFrame = Math.max(1, Math.floor(poolCap / Math.max(1, lifeFrames) * 0.7))
const desiredSpots = Math.round((emitDensity * (0.8 + drum * 2.4 + phasePulse * 0.8 + dropBoost * 2.2)) / Math.max(1, divisions))
const spotsCap = Math.max(1, Math.floor(maxTotalPerFrame / Math.max(1, divisions) / (mirror ? 2 : 1)))
const spotsPerRing = Math.max(1, Math.min(spotsCap, desiredSpots))
const globalPhase = now * 0.4 * rotationSpeed
for (let i = 0; i < emitCount; i += 1) {
const ring = i % divisions
for (let ring = 0; ring < divisions; ring += 1) {
const ringRadius = base + ring * spacing
emitParticle(
cx,
cy,
ring,
divisions,
beatPhase,
hue + bright * 18,
aperture,
mirror,
preset,
ringRadius,
lifeFrames,
noiseScale,
noiseForce,
0.34 + drum * 0.45 + phasePulse * 0.1 + dropBoost * 0.15,
0.45 + bright * 0.35,
dynamicRadius,
)
for (let j = 0; j < spotsPerRing; j += 1) {
const circumOffset = j / Math.max(1, spotsPerRing)
const t = (globalPhase + ring / divisions + circumOffset) * Math.PI * 2
emitParticle(
cx,
cy,
ring,
divisions,
t,
beatPhase,
hue + bright * 18,
aperture,
mirror,
preset,
ringRadius,
lifeFrames,
noiseScale,
noiseForce,
0.34 + drum * 0.45 + phasePulse * 0.1 + dropBoost * 0.15,
0.45 + bright * 0.35,
dynamicRadius,
drum,
)
}
}
ctx.globalCompositeOperation = 'lighter'
@@ -300,22 +319,29 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
const x = pt.x + wobble
const y = pt.y + wobble * ellipseRatio
if (!Number.isFinite(x) || !Number.isFinite(y)) {
pt.active = false
continue
}
const size = Math.max(0.3, pt.size * particleSize * (0.4 + life * 0.9))
const gr = Math.max(0.5, size * (2.0 + glowIntensity * 0.8))
const alpha = pt.alpha * life
const laneHue = monoColor
? hue
: (pt.hue + phaseWave * 18 * hueRange + beatPhase * 8 + dropBoost * 18) % 360
const grad = ctx.createRadialGradient(x, y, 0, x, y, gr)
grad.addColorStop(0, `hsla(${laneHue}, ${85 + treble * 15}%, ${65 + bright * 20}%, ${alpha})`)
grad.addColorStop(0.40, `hsla(${laneHue}, 92%, ${58 + bright * 24}%, ${alpha * 0.55})`)
grad.addColorStop(1, `hsla(${laneHue}, 88%, 52%, 0)`)
ctx.fillStyle = grad
ctx.beginPath()
ctx.fillStyle = `hsla(${laneHue}, 92%, ${58 + bright * 24}%, ${alpha})`
ctx.shadowColor = p.highlightColor
ctx.shadowBlur = glow
ctx.arc(x, y, size, 0, Math.PI * 2)
ctx.arc(x, y, gr, 0, Math.PI * 2)
ctx.fill()
}
ctx.globalCompositeOperation = 'source-over'
ctx.shadowBlur = 0
prevBeatPhase = beatPhase
}
@@ -326,7 +352,7 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
// Only restart when data-source refs change.
// Settings are read via settingsRef each frame — no restart on slider changes.
// width/height are consumed each frame as fallback to clientWidth/clientHeight.
}, [featuresRef, paletteRef])
}, [featuresRef, paletteRef, quality])
return <canvas ref={canvasRef} className="h-full w-full" aria-hidden="true" />
}
@@ -0,0 +1,235 @@
/**
* SpiritVisualizerCanvas.tsx — React wrapper for the Spirit particle visualizer.
*
* Owns the RAF loop and FPS cap. Calls spiritSim.ts for all WebGL work.
* Falls back to a 2D canvas colour fill when WebGL floats are unsupported.
*/
import { useEffect, useRef, useCallback } from 'react'
import type { MutableRefObject } from 'react'
import * as THREE from 'three'
import type { AudioFeatures } from '../../hooks/useAudioFeatures'
import type { VisualizerPalette } from '../../hooks/useVisualizerPalette'
import type { VisualizerSettings } from './VisualizerBackdrop'
import {
initSpiritSim,
renderSpiritFrame,
destroySpiritSim,
type SpiritFrameUniforms,
} from './spiritSim'
type SpiritSettings = NonNullable<VisualizerSettings['spirit']>
// ── Props ─────────────────────────────────────────────────────────────────────
interface SpiritVisualizerCanvasProps {
featuresRef: MutableRefObject<AudioFeatures>
paletteRef: MutableRefObject<VisualizerPalette>
width: number
height: number
settings?: SpiritSettings
fpsCap?: number
quality?: string
}
// ── Presets ───────────────────────────────────────────────────────────────────
interface SpiritPreset {
speed: number
dieSpeed: number
radius: number
curlSize: number
attraction: number
bloomIntensity: number
bloomRadius: number
bloomThreshold: number
afterimageDamp: number
useTriangles: boolean
}
const SPIRIT_PRESETS: Record<string, SpiritPreset> = {
spirit: {
speed: 1.0, dieSpeed: 0.015, radius: 1.5, curlSize: 0.4,
attraction: 0.08, bloomIntensity: 1.5, bloomRadius: 0.4,
bloomThreshold: 0.05, afterimageDamp: 0.88, useTriangles: false,
},
storm: {
speed: 2.2, dieSpeed: 0.03, radius: 2.0, curlSize: 0.8,
attraction: 0.2, bloomIntensity: 2.2, bloomRadius: 0.6,
bloomThreshold: 0.1, afterimageDamp: 0.72, useTriangles: false,
},
drift: {
speed: 0.5, dieSpeed: 0.006, radius: 2.5, curlSize: 0.2,
attraction: 0.02, bloomIntensity: 1.0, bloomRadius: 0.3,
bloomThreshold: 0.02, afterimageDamp: 0.94, useTriangles: false,
},
pulse: {
speed: 1.4, dieSpeed: 0.04, radius: 1.0, curlSize: 0.6,
attraction: 0.35, bloomIntensity: 2.8, bloomRadius: 0.7,
bloomThreshold: 0.15, afterimageDamp: 0.60, useTriangles: false,
},
aurora: {
speed: 0.8, dieSpeed: 0.01, radius: 3.0, curlSize: 0.15,
attraction: 0.04, bloomIntensity: 1.8, bloomRadius: 0.8,
bloomThreshold: 0.04, afterimageDamp: 0.92, useTriangles: true,
},
}
// ── Helpers ───────────────────────────────────────────────────────────────────
function clamp(v: number, lo: number, hi: number): number {
return Math.max(lo, Math.min(hi, v))
}
function hexToThree(hex: string): THREE.Color {
return new THREE.Color(hex)
}
// ── Component ─────────────────────────────────────────────────────────────────
export default function SpiritVisualizerCanvas({
featuresRef, paletteRef, width, height, settings, fpsCap, quality,
}: SpiritVisualizerCanvasProps) {
const canvasRef = useRef<HTMLCanvasElement>(null)
const rafRef = useRef<number>(0)
const startTimeRef = useRef<number>(performance.now())
// Keep a ref so the tick closure always reads the latest settings
const settingsRef = useRef(settings)
useEffect(() => { settingsRef.current = settings }, [settings])
const texSize = quality === 'high' ? 512 : quality === 'low' ? 128 : 256
const cappedFps = fpsCap ?? (quality === 'low' ? 30 : 60)
const initAndStart = useCallback(() => {
// Drop-boost local state (decays per-frame like TendrilsVisualizerCanvas)
let dropBoostLocal = 0
let lastDropAt = 0
const canvas = canvasRef.current
if (!canvas) return
canvas.width = Math.max(1, width)
canvas.height = Math.max(1, height)
const cfg = settingsRef.current ?? {}
const pKey = cfg.visualization_preset ?? 'spirit'
const preset = SPIRIT_PRESETS[pKey] ?? SPIRIT_PRESETS['spirit']
const pal = paletteRef.current
const ok = initSpiritSim(canvas, {
texSize,
speed: cfg.speed ?? preset.speed,
dieSpeed: cfg.die_speed ?? preset.dieSpeed,
radius: cfg.radius ?? preset.radius,
curlSize: cfg.curl_size ?? preset.curlSize,
attraction: cfg.attraction ?? preset.attraction,
bloomStrength: cfg.bloom_intensity ?? preset.bloomIntensity,
bloomRadius: preset.bloomRadius,
bloomThreshold: cfg.bloom_threshold ?? preset.bloomThreshold,
afterimageDamp: cfg.afterimage_damp ?? preset.afterimageDamp,
useTriangles: cfg.use_triangles ?? preset.useTriangles,
color1: hexToThree(pal.speakerPalette[1] ?? '#8888ff'),
color2: hexToThree(pal.speakerPalette[0] ?? '#ff88ff'),
})
if (!ok) {
// 2D colour fallback (WebGL float render target unavailable)
const ctx = canvas.getContext('2d')
if (ctx) {
const g = ctx.createRadialGradient(
canvas.width * 0.5, canvas.height * 0.5, 0,
canvas.width * 0.5, canvas.height * 0.5, Math.max(canvas.width, canvas.height) * 0.5,
)
g.addColorStop(0, pal.speakerPalette[0] ?? '#8822ff')
g.addColorStop(1, '#000000')
ctx.fillStyle = g
ctx.fillRect(0, 0, canvas.width, canvas.height)
}
return
}
let lastTime = 0
const frameMs = 1000 / cappedFps
const tick = (now: number) => {
rafRef.current = requestAnimationFrame(tick)
if (now - lastTime < frameMs) return
lastTime = now
const f = featuresRef.current
const cfg = settingsRef.current ?? {}
const pKey = cfg.visualization_preset ?? 'spirit'
const preset = SPIRIT_PRESETS[pKey] ?? SPIRIT_PRESETS['spirit']
const pal = paletteRef.current
// ── S6 audio→uniform mapping ─────────────────────────────────────────
const bass = Math.max(0, Math.min(1, f?.bass ?? 0))
const sub = Math.max(0, Math.min(1, f?.frequencyBands?.sub ?? 0))
const rms = Math.max(0, Math.min(1, f?.rms ?? 0))
const treble = Math.max(0, Math.min(1, f?.treble ?? 0))
const kick = Math.max(0, Math.min(1, sub * 0.72 + bass * 0.28))
const beatPhase = Math.max(0, Math.min(1, f?.beatInfo?.beatPhase ?? 0))
if (f?.patternInfo?.isBeatDrop && now - lastDropAt > 500) {
dropBoostLocal = 1; lastDropAt = now
}
dropBoostLocal *= 0.96
const dropBoost = dropBoostLocal
const baseSpeed = cfg.speed ?? preset.speed
const baseDieSpeed = cfg.die_speed ?? preset.dieSpeed
const baseCurlSize = cfg.curl_size ?? preset.curlSize
const baseAttraction = cfg.attraction ?? preset.attraction
const baseBloom = cfg.bloom_intensity ?? preset.bloomIntensity
const baseAfterimage = cfg.afterimage_damp ?? preset.afterimageDamp
const speed = baseSpeed * (1 + bass * 0.8 + kick * 0.6)
const dieSpeed = baseDieSpeed * (1 + treble * 0.5 + sub * 0.3)
const curlSize = baseCurlSize * (1 + rms * 0.6)
let attraction = baseAttraction * (1 + beatPhase * 0.5)
attraction -= dropBoost * 2.5 // drop reversal: particles explode outward
const bloomStrength = baseBloom * (1 + bass * 0.4)
const afterimageDamp = clamp(baseAfterimage + (1 - rms) * 0.04, 0.60, 0.96)
const t = (now - startTimeRef.current) * 0.001
const orbitR = 0.5 + bass * 0.5
const attractorPos: [number, number, number] = [
Math.cos(t * 0.4 + beatPhase * Math.PI * 2) * orbitR,
Math.sin(t * 0.3 + beatPhase * Math.PI) * orbitR * 0.5,
Math.sin(t * 0.5) * 0.3,
]
const uniforms: SpiritFrameUniforms = {
time: t,
speed: clamp(speed, 0.1, 6),
dieSpeed: clamp(dieSpeed, 0.001, 0.12),
curlSize: clamp(curlSize, 0.05, 3.0),
attraction: clamp(attraction, -2, 2),
attractorPos,
bloomStrength: clamp(bloomStrength, 0, 4),
bloomThreshold: cfg.bloom_threshold ?? preset.bloomThreshold,
afterimageDamp,
color1: hexToThree(pal.speakerPalette[1] ?? '#8888ff'),
color2: hexToThree(pal.speakerPalette[0] ?? '#ff88ff'),
}
renderSpiritFrame(canvas!, uniforms)
}
rafRef.current = requestAnimationFrame(tick)
}, [featuresRef, paletteRef, texSize, cappedFps, width, height])
useEffect(() => {
initAndStart()
const canvas = canvasRef.current // capture now; ref may change before cleanup
return () => {
cancelAnimationFrame(rafRef.current)
if (canvas) destroySpiritSim(canvas)
}
}, [initAndStart])
return (
<canvas
ref={canvasRef}
style={{ width: '100%', height: '100%', display: 'block' }}
/>
)
}
@@ -0,0 +1,230 @@
import { useEffect, useRef } from 'react'
import type { MutableRefObject } from 'react'
import type { AudioFeatures } from '../../hooks/useAudioFeatures'
import type { VisualizerPalette } from '../../hooks/useVisualizerPalette'
import type { VisualizerSettings } from './VisualizerBackdrop'
import { initTendrilsSim, tickTendrils, destroyTendrilsSim } from './tendrilsSim'
type TendrilsSettings = NonNullable<VisualizerSettings['tendrils']>
interface TendrilsVisualizerCanvasProps {
featuresRef: MutableRefObject<AudioFeatures>
paletteRef: MutableRefObject<VisualizerPalette>
width: number
height: number
settings?: TendrilsSettings
fpsCap?: number
quality?: string
}
function clamp01(v: number): number { return Math.max(0, Math.min(1, v)) }
function clamp(v: number, lo: number, hi: number): number { return Math.max(lo, Math.min(hi, v)) }
// Convert hex colour to linear [0,1] RGB tuple
function hexToRgb(hex: string): [number, number, number] {
const m = /^#?([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})$/i.exec(hex.trim())
if (!m) return [0.3, 0.3, 0.6]
return [parseInt(m[1], 16) / 255, parseInt(m[2], 16) / 255, parseInt(m[3], 16) / 255]
}
// Rotate an rgb tuple by `turns` (0–1) around the colour wheel (HSL hue rotation)
function hueRotate(rgb: [number, number, number], turns: number): [number, number, number] {
const a = turns * Math.PI * 2
const cos = Math.cos(a)
const sin = Math.sin(a)
const r = rgb[0] * (0.213 + cos * 0.787 - sin * 0.213) + rgb[1] * (0.715 - cos * 0.715 - sin * 0.715) + rgb[2] * (0.072 - cos * 0.072 + sin * 0.928)
const g = rgb[0] * (0.213 - cos * 0.213 + sin * 0.143) + rgb[1] * (0.715 + cos * 0.285 + sin * 0.140) + rgb[2] * (0.072 - cos * 0.072 - sin * 0.283)
const b = rgb[0] * (0.213 - cos * 0.213 - sin * 0.787) + rgb[1] * (0.715 - cos * 0.715 + sin * 0.715) + rgb[2] * (0.072 + cos * 0.928 + sin * 0.072)
return [clamp01(r), clamp01(g), clamp01(b)]
}
type TendrilsPreset = {
label: string
dampBase: number
speedBase: number
forceBase: number
noiseWeightBase: number
noiseScaleBase: number
noiseScaleRange: number
flowWeightBase: number
flowDecay: number
targetBase: number
colorMap: number
}
const TENDRILS_PRESETS: Record<string, TendrilsPreset> = {
flow: {
label: 'Flow',
dampBase: 0.055, speedBase: 0.010, forceBase: 0.014,
noiseWeightBase: 0.001, noiseScaleBase: 1.8, noiseScaleRange: 0.8,
flowWeightBase: 1.4, flowDecay: 0.004,
targetBase: 0.04, colorMap: 0.35,
},
scatter: {
label: 'Scatter',
dampBase: 0.030, speedBase: 0.018, forceBase: 0.022,
noiseWeightBase: 0.004, noiseScaleBase: 3.2, noiseScaleRange: 1.6,
flowWeightBase: 0.3, flowDecay: 0.012,
targetBase: 0.01, colorMap: 0.65,
},
pulse: {
label: 'Pulse',
dampBase: 0.045, speedBase: 0.012, forceBase: 0.020,
noiseWeightBase: 0.002, noiseScaleBase: 2.4, noiseScaleRange: 1.2,
flowWeightBase: 0.8, flowDecay: 0.007,
targetBase: 0.10, colorMap: 0.50,
},
ethereal: {
label: 'Ethereal',
dampBase: 0.080, speedBase: 0.006, forceBase: 0.008,
noiseWeightBase: 0.0005, noiseScaleBase: 1.2, noiseScaleRange: 0.4,
flowWeightBase: 1.8, flowDecay: 0.002,
targetBase: 0.02, colorMap: 0.20,
},
hybrid: {
label: 'Hybrid',
dampBase: 0.043, speedBase: 0.010, forceBase: 0.016,
noiseWeightBase: 0.002, noiseScaleBase: 2.1, noiseScaleRange: 1.0,
flowWeightBase: 1.0, flowDecay: 0.006,
targetBase: 0.05, colorMap: 0.40,
},
}
export default function TendrilsVisualizerCanvas({
featuresRef, paletteRef, width, height, settings, fpsCap, quality,
}: TendrilsVisualizerCanvasProps) {
const canvasRef = useRef<HTMLCanvasElement>(null)
const settingsRef = useRef<TendrilsSettings | undefined>(settings)
settingsRef.current = settings
const fpsCapRef = useRef(fpsCap ?? 60)
fpsCapRef.current = fpsCap ?? 60
const qualityRef = useRef(quality ?? 'balanced')
qualityRef.current = quality ?? 'balanced'
const widthRef = useRef(width)
widthRef.current = width
const heightRef = useRef(height)
heightRef.current = height
useEffect(() => {
const canvas = canvasRef.current
if (!canvas) return
const q = qualityRef.current
const rootNum = q === 'high' ? 512 : 256
const ok = initTendrilsSim(canvas, { rootNum })
if (!ok) {
// WebGL2 / float buffer unavailable — 2D colour fill fallback
const ctx = canvas.getContext('2d')
if (!ctx) return
let rafFb = 0
const renderFb = () => {
rafFb = window.requestAnimationFrame(renderFb)
const w = Math.max(1, Math.round(canvas.clientWidth || 0) || widthRef.current)
const h = Math.max(1, Math.round(canvas.clientHeight || 0) || heightRef.current)
if (canvas.width !== w || canvas.height !== h) { canvas.width = w; canvas.height = h }
ctx.fillStyle = paletteRef.current.backgroundColor
ctx.fillRect(0, 0, w, h)
}
rafFb = window.requestAnimationFrame(renderFb)
return () => window.cancelAnimationFrame(rafFb)
}
let raf = 0
let lastFrameAt = 0
let dropBoost = 0
let lastDropAt = 0
let prevBeatPhase = 0
const startTime = performance.now() * 0.001
const render = (ts: number) => {
raf = window.requestAnimationFrame(render)
const minFrameMs = 1000 / Math.max(1, fpsCapRef.current)
if (ts - lastFrameAt < minFrameMs - 1) return
lastFrameAt = ts
const f = featuresRef.current
const s = settingsRef.current
const p = paletteRef.current
const bass = clamp01(f.bass)
const sub = clamp01(f.frequencyBands.sub)
const rms = clamp01(f.rms)
const treble = clamp01(f.treble)
const drum = clamp01(f.frequencyBands.bass)
const beatPhase = clamp01(f.beatInfo.beatPhase)
const phasePulse = Math.pow(Math.max(0, Math.sin(beatPhase * Math.PI)), 1.8)
const phaseWrapped = beatPhase < prevBeatPhase - 0.35
const kick = clamp01(sub * 0.72 + bass * 0.28)
const idle = 0.5 + 0.5 * Math.sin(performance.now() * 0.0009)
const dropTriggered = f.patternInfo.isBeatDrop && ts - lastDropAt > 500
if (dropTriggered) { dropBoost = 1; lastDropAt = ts }
dropBoost *= 0.96
const presetKey = s?.visualization_preset ?? 'hybrid'
const preset = TENDRILS_PRESETS[presetKey] ?? TENDRILS_PRESETS.hybrid
// Audio → uniforms (T8)
const forceWeight = preset.forceBase * (1 + bass * 1.8 + phasePulse * 0.6)
const speedLimit = preset.speedBase * (1 + kick * 2.5 + dropBoost * 1.2)
const noiseScale = preset.noiseScaleBase + treble * preset.noiseScaleRange
const noiseWeight = preset.noiseWeightBase + rms * 0.6 + idle * 0.1
const flowWeight = preset.flowWeightBase + drum * 1.4
const damping = Math.max(0.015, preset.dampBase - dropBoost * 0.08)
const flowDecay = Math.max(0.001, preset.flowDecay - dropBoost * 0.015)
const targetWeight = preset.targetBase + rms * 0.3
const colorMapAlpha = clamp(preset.colorMap + bass * 0.4 + treble * 0.3 + (s?.color_map_alpha ?? 0), 0.1, 0.9)
const trailAlpha = 0.04 + dropBoost * 0.06
const phaseAngle = beatPhase * Math.PI * 2
const target: [number, number] = [
Math.cos(phaseAngle) * 0.6,
Math.sin(phaseAngle) * 0.45,
]
const seedRgb = hexToRgb(p.seedColor ?? p.highlightColor)
const hueRange = Math.max(0, s?.hue_range ?? 1)
const baseColor = s?.mono_color ? seedRgb : hueRotate(seedRgb, rms * 0.3 * hueRange)
const flowColor = hueRotate(seedRgb, 0.5 + bass * 0.3 * hueRange)
const bgColor = hexToRgb(p.backgroundColor)
const now = ts * 0.001 - startTime
tickTendrils(canvas, {
time: now,
damping: damping * Math.max(0.2, s?.damping ?? 1),
speedLimit: speedLimit * Math.max(0.2, s?.speed_limit ?? 1),
forceWeight: forceWeight * Math.max(0.2, s?.force_weight ?? 1),
noiseWeight: noiseWeight * Math.max(0.01, s?.noise_weight ?? 1),
noiseScale: noiseScale * Math.max(0.1, s?.noise_scale ?? 1),
noiseSpeed: 0.4,
flowWeight: flowWeight * Math.max(0.01, s?.flow_weight ?? 1),
flowDecay: flowDecay * Math.max(0.1, s?.flow_decay ?? 1),
targetWeight: targetWeight,
target,
trailAlpha,
colorMapAlpha,
speedAlpha: 0.002 + kick * 0.003,
baseColor,
flowColor,
bgColor,
lineWidth: s?.line_width ?? 1,
})
// Beat-driven pattern cycling visual handled by forceWeight/targetWeight spikes
if (phaseWrapped || dropTriggered) { void 0 /* placeholder for future effects */ }
prevBeatPhase = beatPhase
}
raf = window.requestAnimationFrame(render)
return () => {
window.cancelAnimationFrame(raf)
destroyTendrilsSim(canvas)
}
}, [featuresRef, paletteRef, quality])
return <canvas ref={canvasRef} className="h-full w-full" aria-hidden="true" />
}
@@ -3,10 +3,12 @@ import type { MutableRefObject } from 'react'
import type { AudioFeatures } from '../../hooks/useAudioFeatures'
import type { VisualizerPalette } from '../../hooks/useVisualizerPalette'
const MuonVisualizerCanvas = lazy(() => import('./MuonVisualizerCanvas'))
const FluidVisualizerCanvas = lazy(() => import('./FluidVisualizerCanvas'))
const MuonVisualizerCanvas = lazy(() => import('./MuonVisualizerCanvas'))
const FluidVisualizerCanvas = lazy(() => import('./FluidVisualizerCanvas'))
const TendrilsVisualizerCanvas = lazy(() => import('./TendrilsVisualizerCanvas'))
const SpiritVisualizerCanvas = lazy(() => import('./SpiritVisualizerCanvas'))
export type VisualizerMode = 'muon' | 'fluid' | null
export type VisualizerMode = 'muon' | 'fluid' | 'tendrils' | 'spirit' | null
/**
* Per-song visualizer intensity multipliers from manifest.visualizer.settings.
@@ -54,6 +56,38 @@ export interface VisualizerSettings {
blob_life?: number
// GUI Parameter Map additions
color_speed?: number
visualization_preset?: string
}
tendrils?: {
damping?: number
noise_weight?: number
noise_scale?: number
flow_weight?: number
flow_decay?: number
force_weight?: number
speed_limit?: number
line_width?: number
color_map_alpha?: number
hue_range?: number
mono_color?: boolean
particle_count?: number
visualization_preset?: string
}
spirit?: {
speed?: number
die_speed?: number
radius?: number
curl_size?: number
attraction?: number
use_triangles?: boolean
bloom_intensity?: number
bloom_threshold?: number
afterimage_damp?: number
hue_range?: number
mono_color?: boolean
visualization_preset?: string
}
}
@@ -103,5 +137,41 @@ export default function VisualizerBackdrop({ mode, featuresRef, paletteRef, widt
</div>
)
}
if (mode === 'tendrils') {
return (
<div
className="relative h-full w-full overflow-hidden"
style={{ background: palette.backgroundColor }}
data-testid="visualizer-backdrop"
data-visualizer-mode={mode}
data-palette-source={palette.source}
data-palette-seed={palette.seedColor}
data-palette-highlight={palette.highlightColor}
data-palette-word-highlight={palette.wordHighlightColor}
>
<Suspense fallback={<div className="h-full w-full" style={{ background: palette.backgroundColor }} />}>
<TendrilsVisualizerCanvas featuresRef={featuresRef} paletteRef={paletteRef} width={width} height={height} settings={settings?.tendrils} fpsCap={settings?.fps_cap} quality={settings?.quality} />
</Suspense>
</div>
)
}
if (mode === 'spirit') {
return (
<div
className="relative h-full w-full overflow-hidden"
style={{ background: palette.backgroundColor }}
data-testid="visualizer-backdrop"
data-visualizer-mode={mode}
data-palette-source={palette.source}
data-palette-seed={palette.seedColor}
data-palette-highlight={palette.highlightColor}
data-palette-word-highlight={palette.wordHighlightColor}
>
<Suspense fallback={<div className="h-full w-full" style={{ background: palette.backgroundColor }} />}>
<SpiritVisualizerCanvas featuresRef={featuresRef} paletteRef={paletteRef} width={width} height={height} settings={settings?.spirit} fpsCap={settings?.fps_cap} quality={settings?.quality} />
</Suspense>
</div>
)
}
return null
}
@@ -56,30 +56,24 @@ export interface FluidConfig {
interface FluidInstance {
canvas: HTMLCanvasElement
config: FluidConfig
lastX: number
lastY: number
}
interface FluidCanvasElement extends HTMLCanvasElement {
_fluidSplat?: (x: number, y: number, dx: number, dy: number, color: FluidColor) => void
}
const instances = new WeakMap<HTMLCanvasElement, FluidInstance>()
const initTokens = new WeakMap<HTMLCanvasElement, number>()
let nextInitToken = 1
function clamp01(v: number): number {
return Math.max(0, Math.min(1, v))
}
function dispatchMouse(canvas: HTMLCanvasElement, type: 'mousedown' | 'mousemove', x: number, y: number): void {
const evt = new MouseEvent(type, {
bubbles: true,
cancelable: true,
clientX: x,
clientY: y,
})
Object.defineProperty(evt, 'offsetX', { configurable: true, value: x })
Object.defineProperty(evt, 'offsetY', { configurable: true, value: y })
canvas.dispatchEvent(evt)
}
export function initFluidSim(canvas: HTMLCanvasElement, initialConfig: FluidConfig): void {
destroyFluidSim(canvas)
const initToken = nextInitToken++
initTokens.set(canvas, initToken)
const config: FluidConfig = {
TRIGGER: 'hover',
@@ -97,16 +91,13 @@ export function initFluidSim(canvas: HTMLCanvasElement, initialConfig: FluidConf
// Load WebGL library asynchronously but don't block
getWebGLFluid()
.then((WebGL) => {
// Only initialize if canvas still exists and not destroyed
if (document.contains(canvas)) {
WebGL(canvas, config)
instances.set(canvas, {
canvas,
config,
lastX: canvas.clientWidth * 0.5,
lastY: canvas.clientHeight * 0.5,
})
}
if (initTokens.get(canvas) !== initToken) return
if (!document.contains(canvas)) return
WebGL(canvas, config)
if (initTokens.get(canvas) !== initToken) return
instances.set(canvas, { canvas, config })
})
.catch((err) => console.error('Failed to load webgl-fluid:', err))
}
@@ -117,38 +108,24 @@ export function updateFluidConfig(canvas: HTMLCanvasElement, patch: Partial<Flui
Object.assign(inst.config, patch)
}
export function splatFluid(canvas: HTMLCanvasElement, x: number, y: number, dx: number, dy: number, _color: FluidColor): void {
export function splatFluid(canvas: HTMLCanvasElement, x: number, y: number, dx: number, dy: number, color: FluidColor): void {
const inst = instances.get(canvas)
if (!inst) return
const w = Math.max(1, canvas.clientWidth || canvas.width || 1)
const h = Math.max(1, canvas.clientHeight || canvas.height || 1)
const fluidCanvas = canvas as FluidCanvasElement
const directSplat = fluidCanvas._fluidSplat
if (!directSplat) return
const nx = clamp01(x)
const ny = clamp01(y)
const px = nx * w
const py = ny * h
const speedNorm = Math.hypot(dx, dy) / Math.max(1, Math.min(w, h) * 6)
const pressure = clamp01(speedNorm)
const dragScale = 0.02 + pressure * 0.09
const tx = Math.max(0, Math.min(w, px + dx * dragScale))
const ty = Math.max(0, Math.min(h, py + dy * dragScale))
const mx = px + (tx - px) * (0.45 + pressure * 0.25)
const my = py + (ty - py) * (0.45 + pressure * 0.25)
dispatchMouse(canvas, 'mousedown', px, py)
dispatchMouse(canvas, 'mousemove', mx, my)
dispatchMouse(canvas, 'mousemove', tx, ty)
window.dispatchEvent(new MouseEvent('mouseup', { bubbles: true, cancelable: true }))
inst.lastX = tx
inst.lastY = ty
// Preserve existing visual orientation: WebGL fluid uses bottom-origin texcoords.
directSplat(clamp01(x), 1 - clamp01(y), dx, dy, color)
}
export function destroyFluidSim(canvas: HTMLCanvasElement): void {
initTokens.delete(canvas)
const fluidCanvas = canvas as FluidCanvasElement
delete fluidCanvas._fluidSplat
const inst = instances.get(canvas)
if (!inst) return
@@ -1804,6 +1804,8 @@ function ft(V, _) {
s = (s << 5) - s + o.charCodeAt(a), s |= 0;
return s;
}
// STEMENGINE_PATCH: expose direct splat API for external color/velocity injection.
D._fluidSplat = fe;
}
export {
ft as default
@@ -1,8 +1,42 @@
export function hashNoise2D(x: number, y: number, seed = 0): number {
const s = Math.sin(x * 127.1 + y * 311.7 + seed * 74.7) * 43758.5453123
function fade(t: number): number {
return t * t * t * (t * (t * 6 - 15) + 10)
}
function lerpf(a: number, b: number, t: number): number {
return a + (b - a) * t
}
function hashf(ix: number, iy: number): number {
const n = ix * 127.1 + iy * 311.7
const s = Math.sin(n) * 43758.5453123
return s - Math.floor(s)
}
export function smoothNoise2D(x: number, y: number, seed = 0): number {
const ox = x + seed * 17.3
const oy = y + seed * 31.7
const ix = Math.floor(ox)
const iy = Math.floor(oy)
const fx = ox - ix
const fy = oy - iy
const ux = fade(fx)
const uy = fade(fy)
const a = hashf(ix, iy)
const b = hashf(ix + 1, iy)
const c = hashf(ix, iy + 1)
const d = hashf(ix + 1, iy + 1)
return lerpf(lerpf(a, b, ux), lerpf(c, d, ux), uy)
}
export function curlNoise2D(x: number, y: number, seed = 0): { nx: number; ny: number } {
const eps = 0.1
const n1 = smoothNoise2D(x, y + eps, seed) - smoothNoise2D(x, y - eps, seed)
const n2 = smoothNoise2D(x + eps, y, seed) - smoothNoise2D(x - eps, y, seed)
return { nx: n1, ny: -n2 }
}
export function centeredNoise2D(x: number, y: number, seed = 0): number {
return hashNoise2D(x, y, seed) * 2 - 1
return smoothNoise2D(x, y, seed) * 2 - 1
}
@@ -0,0 +1,44 @@
/* 3D divergence-free curl noise using three independent potential fields.
* Returns a vec3 in approximately (-1, 1) per axis.
* Concatenated at the top of position.glsl by spiritSim.ts at runtime.
* Ref: Bridson 2007 curl noise.
*/
float _h31(vec3 p) {
p = fract(p * vec3(0.1031, 0.1030, 0.0973));
p += dot(p, p.yxz + 33.33);
return fract((p.x + p.y) * p.z);
}
float _vn3(vec3 p) {
vec3 i = floor(p);
vec3 f = fract(p);
f = f * f * (3.0 - 2.0 * f);
return mix(
mix(mix(_h31(i), _h31(i + vec3(1.0, 0.0, 0.0)), f.x),
mix(_h31(i + vec3(0.0, 1.0, 0.0)), _h31(i + vec3(1.0, 1.0, 0.0)), f.x), f.y),
mix(mix(_h31(i + vec3(0.0, 0.0, 1.0)), _h31(i + vec3(1.0, 0.0, 1.0)), f.x),
mix(_h31(i + vec3(0.0, 1.0, 1.0)), _h31(i + vec3(1.0, 1.0, 1.0)), f.x), f.y),
f.z);
}
vec3 curlNoise(vec3 p) {
const float e = 0.1;
vec3 pA = p + vec3( 0.0, 31.7, 0.0);
vec3 pB = p + vec3(74.1, 0.0, 0.0);
vec3 pC = p + vec3( 0.0, 0.0, 113.5);
float dy_pA = _vn3(pA + vec3(0.0, e, 0.0)) - _vn3(pA - vec3(0.0, e, 0.0));
float dz_pA = _vn3(pA + vec3(0.0, 0.0, e)) - _vn3(pA - vec3(0.0, 0.0, e));
float dx_pB = _vn3(pB + vec3(e, 0.0, 0.0)) - _vn3(pB - vec3(e, 0.0, 0.0));
float dz_pB = _vn3(pB + vec3(0.0, 0.0, e)) - _vn3(pB - vec3(0.0, 0.0, e));
float dx_pC = _vn3(pC + vec3(e, 0.0, 0.0)) - _vn3(pC - vec3(e, 0.0, 0.0));
float dy_pC = _vn3(pC + vec3(0.0, e, 0.0)) - _vn3(pC - vec3(0.0, e, 0.0));
/* curl(ψ) = (∂ψC/∂y − ∂ψB/∂z, ∂ψA/∂z − ∂ψC/∂x, ∂ψB/∂x − ∂ψA/∂y) */
return vec3(
dy_pC - dz_pB,
dz_pA - dx_pC,
dx_pB - dy_pA
) / (2.0 * e);
}
@@ -0,0 +1,16 @@
precision mediump float;
varying vec3 vColor;
varying float vLife;
varying float vAlpha;
uniform bool useTriangles;
void main() {
float alpha = vAlpha;
if (!useTriangles) {
vec2 center = gl_PointCoord - 0.5;
float d = length(center);
alpha *= 1.0 - smoothstep(0.35, 0.5, d);
}
if (alpha < 0.01) discard;
gl_FragColor = vec4(vColor, alpha);
}
@@ -0,0 +1,38 @@
uniform sampler2D texturePosition;
uniform vec2 texSize;
uniform vec3 color1;
uniform vec3 color2;
uniform float particleSize;
uniform bool useTriangles;
attribute float particleID;
attribute float vertexOffset;
varying vec3 vColor;
varying float vLife;
varying float vAlpha;
void main() {
float pid = useTriangles ? floor(particleID / 3.0) : particleID;
vec2 uv = (vec2(mod(pid, texSize.x), floor(pid / texSize.x)) + 0.5) / texSize;
vec4 state = texture2D(texturePosition, uv);
vec3 pos = state.xyz;
float life = state.w;
vLife = life;
vColor = mix(color2, color1, smoothstep(0.0, 0.7, life));
vAlpha = smoothstep(0.0, 0.2, life) * smoothstep(1.0, 0.7, life);
if (useTriangles) {
float angle = vertexOffset * 2.09439510239; /* 2π/3 per vertex */
float radius = particleSize * 0.012 * smoothstep(0.0, 0.2, life);
vec3 offset = vec3(cos(angle), sin(angle), 0.0) * radius;
pos += offset;
}
vec4 mvPos = modelViewMatrix * vec4(pos, 1.0);
gl_PointSize = useTriangles
? 1.0
: max(1.0, particleSize * 1300.0 / -mvPos.z * smoothstep(0.0, 0.2, life));
gl_Position = projectionMatrix * mvPos;
}
@@ -0,0 +1,46 @@
/* position.glsl — Spirit GPGPU position update.
* curl.glsl is prepended to this source by spiritSim.ts before shader compilation.
*/
precision highp float;
uniform sampler2D texturePosition; /* previous [x, y, z, life] */
uniform sampler2D textureDefaultPosition; /* spawn positions [x, y, z, 1] */
uniform vec2 texSize;
uniform float time;
uniform float speed;
uniform float dieSpeed;
uniform float curlSize;
uniform float attraction;
uniform vec3 attractorPos;
uniform float initBlend;
void main() {
vec2 uv = gl_FragCoord.xy / texSize;
vec4 pos4 = texture2D(texturePosition, uv);
vec4 defPos4 = texture2D(textureDefaultPosition, uv);
vec3 pos = pos4.xyz;
float life = pos4.w;
/* Respawn dead particle from default position */
if (life <= 0.0) {
gl_FragColor = vec4(defPos4.xyz, 1.0);
return;
}
/* Smooth blend from spawn point toward live position */
float blend = clamp(1.0 - life + initBlend, 0.0, 1.0);
pos = mix(defPos4.xyz, pos, blend);
/* Curl noise displacement */
vec3 cn = curlNoise(pos * curlSize + vec3(time * 0.05));
pos += cn * speed * 0.01;
/* Attractor force */
vec3 toAttr = attractorPos - pos;
float dist = length(toAttr) + 0.001;
pos += normalize(toAttr) * attraction * speed * 0.005 * (1.0 + life * 0.5);
life -= dieSpeed * speed;
gl_FragColor = vec4(pos, life);
}
@@ -0,0 +1,374 @@
/**
* spiritSim.ts — Three.js GPGPU Spirit particle simulation.
*
* Two WebGLRenderTarget ping-pong textures hold [x, y, z, life] per particle.
* A full-screen GPGPU quad updates positions each frame.
* Particles are rendered as points or triangles with UnrealBloomPass + AfterimagePass.
* The React component (SpiritVisualizerCanvas.tsx) owns the RAF loop.
*/
import * as THREE from 'three'
import { EffectComposer } from 'three/addons/postprocessing/EffectComposer.js'
import { RenderPass } from 'three/addons/postprocessing/RenderPass.js'
import { UnrealBloomPass } from 'three/addons/postprocessing/UnrealBloomPass.js'
import { AfterimagePass } from 'three/addons/postprocessing/AfterimagePass.js'
import curlGLSL from './spirit/curl.glsl?raw'
import positionGLSL from './spirit/position.glsl?raw'
import particleVert from './spirit/particle.vert?raw'
import particleFrag from './spirit/particle.frag?raw'
// ── Public types ─────────────────────────────────────────────────────────────
export interface SpiritConfig {
/** GPGPU texture size; total particles = texSize². 256 or 512. */
texSize: number
speed: number
dieSpeed: number
radius: number
curlSize: number
attraction: number
useTriangles: boolean
bloomStrength: number
bloomRadius: number
bloomThreshold: number
afterimageDamp: number
color1: THREE.Color
color2: THREE.Color
}
export interface SpiritFrameUniforms {
time: number
speed: number
dieSpeed: number
curlSize: number
attraction: number
attractorPos: [number, number, number]
bloomStrength: number
bloomThreshold: number
afterimageDamp: number
color1: THREE.Color
color2: THREE.Color
}
// ── Internal state ────────────────────────────────────────────────────────────
interface SpiritInstance {
renderer: THREE.WebGLRenderer
scene: THREE.Scene
camera: THREE.PerspectiveCamera
composer: EffectComposer
bloomPass: UnrealBloomPass
afterimagePass: AfterimagePass
gpuScene: THREE.Scene
gpuCamera: THREE.OrthographicCamera
gpuMat: THREE.ShaderMaterial
/** readRT = source for GPGPU; writeRT = destination. Swapped each frame. */
readRT: THREE.WebGLRenderTarget
writeRT: THREE.WebGLRenderTarget
defaultTex: THREE.DataTexture
particleMesh: THREE.Points | THREE.Mesh
particleMat: THREE.ShaderMaterial
gpuMesh: THREE.Mesh
cfg: SpiritConfig
numParticles: number
lastW: number
lastH: number
destroyed: boolean
onContextLost: () => void
onContextRestored: () => void
}
const instances = new WeakMap<HTMLCanvasElement, SpiritInstance>()
// ── Helpers ──────────────────────────────────────────────────────────────────
function makeDefaultPositions(n: number, radius: number): Float32Array {
const arr = new Float32Array(n * 4)
for (let i = 0; i < n; i++) {
const theta = Math.random() * Math.PI * 2
const phi = Math.acos(2 * Math.random() - 1)
const r = radius * (0.3 + 0.7 * Math.random())
arr[i * 4 + 0] = r * Math.sin(phi) * Math.cos(theta)
arr[i * 4 + 1] = r * Math.sin(phi) * Math.sin(theta)
arr[i * 4 + 2] = r * Math.cos(phi)
arr[i * 4 + 3] = Math.random() // initial life
}
return arr
}
function makeRtOpts(): THREE.RenderTargetOptions {
return {
type: THREE.FloatType,
format: THREE.RGBAFormat,
minFilter: THREE.NearestFilter,
magFilter: THREE.NearestFilter,
depthBuffer: false,
stencilBuffer: false,
}
}
// ── Public API ────────────────────────────────────────────────────────────────
export function initSpiritSim(canvas: HTMLCanvasElement, cfg: SpiritConfig): boolean {
destroySpiritSim(canvas)
let renderer: THREE.WebGLRenderer
try {
renderer = new THREE.WebGLRenderer({ canvas, antialias: false, alpha: false })
} catch {
return false
}
// Check float render target support
// Three.js >= 0.159 requires WebGL2, which guarantees float fragment textures.
// If WebGL2 is somehow unavailable, bail out.
if (!renderer.capabilities.isWebGL2) {
renderer.dispose()
return false
}
renderer.setPixelRatio(Math.min(window.devicePixelRatio, 2))
const w = Math.max(1, canvas.clientWidth || canvas.width)
const h = Math.max(1, canvas.clientHeight || canvas.height)
renderer.setSize(w, h)
const scene = new THREE.Scene()
const camera = new THREE.PerspectiveCamera(60, w / h, 0.1, 100)
camera.position.set(0, 0, 3.5)
const { texSize } = cfg
const numParticles = texSize * texSize
const rtOpts = makeRtOpts()
const readRT = new THREE.WebGLRenderTarget(texSize, texSize, rtOpts)
const writeRT = new THREE.WebGLRenderTarget(texSize, texSize, rtOpts)
// Spawn default positions
const defData = makeDefaultPositions(numParticles, cfg.radius)
const defaultTex = new THREE.DataTexture(defData, texSize, texSize, THREE.RGBAFormat, THREE.FloatType)
defaultTex.needsUpdate = true // SE10: must set or GPU sees zeros
// Seed readRT with default positions (bootstrap first frame)
const seedRenderer = new THREE.WebGLRenderer({ canvas: document.createElement('canvas') })
const seedScene = new THREE.Scene()
const seedCamera = new THREE.OrthographicCamera(-1, 1, 1, -1, 0, 1)
const seedMat = new THREE.ShaderMaterial({
uniforms: { tex: { value: defaultTex } },
vertexShader: 'void main() { gl_Position = vec4(position, 1.0); }',
fragmentShader: `
precision highp float;
uniform sampler2D tex;
void main() { gl_FragColor = texture2D(tex, vec2(gl_FragCoord.x / ${texSize.toFixed(1)}, gl_FragCoord.y / ${texSize.toFixed(1)})); }
`,
})
const seedGeo = new THREE.PlaneGeometry(2, 2)
seedScene.add(new THREE.Mesh(seedGeo, seedMat))
seedRenderer.setSize(texSize, texSize)
seedRenderer.setRenderTarget(readRT)
seedRenderer.render(seedScene, seedCamera)
seedRenderer.setRenderTarget(null)
seedRenderer.dispose()
seedMat.dispose()
seedGeo.dispose()
// GPGPU scene — full-screen quad to update particle positions
const gpuScene = new THREE.Scene()
const gpuCamera = new THREE.OrthographicCamera(-1, 1, 1, -1, 0, 1)
const gpuGeo = new THREE.PlaneGeometry(2, 2)
const gpuMat = new THREE.ShaderMaterial({
uniforms: {
texturePosition: { value: readRT.texture },
textureDefaultPosition: { value: defaultTex },
texSize: { value: new THREE.Vector2(texSize, texSize) },
time: { value: 0 },
speed: { value: cfg.speed },
dieSpeed: { value: cfg.dieSpeed },
curlSize: { value: cfg.curlSize },
attraction: { value: cfg.attraction },
attractorPos: { value: new THREE.Vector3(0, 0, 0) },
initBlend: { value: 1.0 },
},
vertexShader: 'void main() { gl_Position = vec4(position, 1.0); }',
// curl.glsl prepended so curlNoise() is available in position.glsl
fragmentShader: curlGLSL + '\n' + positionGLSL,
})
const gpuMesh = new THREE.Mesh(gpuGeo, gpuMat)
gpuScene.add(gpuMesh)
// Particle geometry
const useTriangles = cfg.useTriangles
const vertCount = useTriangles ? numParticles * 3 : numParticles
const pidArr = new Float32Array(vertCount)
const offsetArr = new Float32Array(vertCount)
if (useTriangles) {
for (let i = 0; i < numParticles; i++) {
for (let v = 0; v < 3; v++) {
pidArr[i * 3 + v] = i * 3 + v
offsetArr[i * 3 + v] = v
}
}
} else {
for (let i = 0; i < numParticles; i++) { pidArr[i] = i; offsetArr[i] = 0 }
}
const geo = new THREE.BufferGeometry()
geo.setAttribute('particleID', new THREE.BufferAttribute(pidArr, 1))
geo.setAttribute('vertexOffset', new THREE.BufferAttribute(offsetArr, 1))
const particleMat = new THREE.ShaderMaterial({
uniforms: {
texturePosition: { value: readRT.texture },
texSize: { value: new THREE.Vector2(texSize, texSize) },
color1: { value: cfg.color1.clone() },
color2: { value: cfg.color2.clone() },
particleSize: { value: 1.0 },
useTriangles: { value: useTriangles },
},
vertexShader: particleVert,
fragmentShader: particleFrag,
transparent: true,
depthWrite: false,
blending: THREE.AdditiveBlending,
})
let particleMesh: THREE.Points | THREE.Mesh
if (useTriangles) {
particleMesh = new THREE.Mesh(geo, particleMat)
} else {
particleMesh = new THREE.Points(geo, particleMat)
}
scene.add(particleMesh)
// Post-processing
const composer = new EffectComposer(renderer)
composer.addPass(new RenderPass(scene, camera))
const bloomPass = new UnrealBloomPass(
new THREE.Vector2(w, h),
cfg.bloomStrength,
cfg.bloomRadius,
cfg.bloomThreshold,
)
composer.addPass(bloomPass)
const afterimagePass = new AfterimagePass(cfg.afterimageDamp)
composer.addPass(afterimagePass)
const inst: SpiritInstance = {
renderer, scene, camera, composer, bloomPass, afterimagePass,
gpuScene, gpuCamera, gpuMat, gpuMesh,
readRT, writeRT, defaultTex,
particleMesh, particleMat,
cfg: { ...cfg },
numParticles,
lastW: w, lastH: h,
destroyed: false,
onContextLost: () => {},
onContextRestored: () => {},
}
inst.onContextLost = () => {
inst.destroyed = true
}
inst.onContextRestored = () => {
if (!instances.has(canvas)) return
inst.destroyed = true
initSpiritSim(canvas, inst.cfg)
}
canvas.addEventListener('webglcontextlost', inst.onContextLost)
canvas.addEventListener('webglcontextrestored', inst.onContextRestored)
instances.set(canvas, inst)
return true
}
export function renderSpiritFrame(canvas: HTMLCanvasElement, u: SpiritFrameUniforms): void {
const inst = instances.get(canvas)
if (!inst || inst.destroyed) return
const { renderer, gpuMat, gpuScene, gpuCamera, particleMat, composer } = inst
const w = Math.max(1, Math.round(canvas.clientWidth || 0) || canvas.width)
const h = Math.max(1, Math.round(canvas.clientHeight || 0) || canvas.height)
if (canvas.width !== w || canvas.height !== h) {
canvas.width = w
canvas.height = h
}
// Resize when canvas changes (SE2, SE5)
if (inst.lastW !== w || inst.lastH !== h) {
renderer.setSize(w, h)
inst.camera.aspect = w / h
inst.camera.updateProjectionMatrix()
inst.composer.setSize(w, h)
inst.bloomPass.setSize(w, h)
inst.lastW = w
inst.lastH = h
}
// ── GPGPU position update ─────────────────────────────────────────────────
// Read from readRT → write to writeRT (SE9: never alias same target)
gpuMat.uniforms.texturePosition.value = inst.readRT.texture
gpuMat.uniforms.time.value = u.time
gpuMat.uniforms.speed.value = u.speed
gpuMat.uniforms.dieSpeed.value = u.dieSpeed
gpuMat.uniforms.curlSize.value = u.curlSize
gpuMat.uniforms.attraction.value = u.attraction
;(gpuMat.uniforms.attractorPos.value as THREE.Vector3).set(
u.attractorPos[0], u.attractorPos[1], u.attractorPos[2],
)
renderer.setRenderTarget(inst.writeRT)
renderer.render(gpuScene, gpuCamera)
renderer.setRenderTarget(null)
// Swap ping-pong (SE9: ensure read/write never alias)
const tmp = inst.readRT
inst.readRT = inst.writeRT
inst.writeRT = tmp
// ── Update particle material uniforms ─────────────────────────────────────
particleMat.uniforms.texturePosition.value = inst.readRT.texture
particleMat.uniforms.color1.value.copy(u.color1)
particleMat.uniforms.color2.value.copy(u.color2)
// ── Post-processing uniforms ──────────────────────────────────────────────
inst.bloomPass.strength = clamp(u.bloomStrength, 0, 4)
inst.bloomPass.threshold = u.bloomThreshold
// SE3: clamp afterimage below 1.0 to prevent white-out
inst.afterimagePass.uniforms['damp'].value = clamp(u.afterimageDamp, 0.60, 0.96)
// ── Composite render ──────────────────────────────────────────────────────
composer.render()
}
export function destroySpiritSim(canvas: HTMLCanvasElement): void {
const inst = instances.get(canvas)
if (!inst) return
inst.destroyed = true
canvas.removeEventListener('webglcontextlost', inst.onContextLost)
canvas.removeEventListener('webglcontextrestored', inst.onContextRestored)
// SE8: dispose all Three.js resources
try {
inst.readRT.dispose()
inst.writeRT.dispose()
inst.defaultTex.dispose()
inst.gpuMat.dispose()
inst.gpuMesh.geometry.dispose()
if (inst.particleMesh.geometry) inst.particleMesh.geometry.dispose()
inst.particleMat.dispose()
inst.bloomPass.dispose()
inst.afterimagePass.dispose()
inst.composer.dispose()
inst.renderer.dispose()
} catch { /* context may already be lost */ }
instances.delete(canvas)
}
// ── Helpers ───────────────────────────────────────────────────────────────────
function clamp(v: number, lo: number, hi: number): number {
return Math.max(lo, Math.min(hi, v))
}
// Extend SpiritInstance type to allow cfg.cfg pattern (workaround for bloom clamp)
@@ -0,0 +1,3 @@
precision mediump float;
uniform vec4 uFade; /* rgb = target colour, a = fade strength */
void main() { gl_FragColor = uFade; }
@@ -0,0 +1,10 @@
precision mediump float;
varying vec4 vState;
varying float vIsHead;
void main() {
if (vIsHead < 0.5) discard;
vec2 vel = vState.zw;
/* Encode velocity direction into [0,1] RG for logic.glsl to decode */
gl_FragColor = vec4(vel * 0.5 + 0.5, 0.0, 0.08);
}
@@ -0,0 +1,75 @@
precision highp float;
uniform sampler2D statePrev; /* previous frame [posX, posY, velX, velY] */
uniform sampler2D flowTex; /* flow feedback field */
uniform vec2 stateSize; /* rootNum × rootNum */
uniform float time;
uniform float damping;
uniform float speedLimit;
uniform float forceWeight;
uniform float noiseWeight;
uniform float noiseScale;
uniform float noiseSpeed;
uniform float flowWeight;
uniform vec2 target;
uniform float targetWeight;
/* ── Inline 2D simplex noise (Ashima Arts) ─────────────────────────────── */
vec3 _perm(vec3 x) { return mod(((x * 34.0) + 1.0) * x, 289.0); }
float snoise2(vec2 v) {
const vec4 C = vec4(0.211324865405187, 0.366025403784439,
-0.577350269189626, 0.024390243902439);
vec2 i = floor(v + dot(v, C.yy));
vec2 x0 = v - i + dot(i, C.xx);
vec2 i1 = (x0.x > x0.y) ? vec2(1.0, 0.0) : vec2(0.0, 1.0);
vec4 x12 = x0.xyxy + C.xxzz;
x12.xy -= i1;
i = mod(i, 289.0);
vec3 p = _perm(_perm(i.y + vec3(0.0, i1.y, 1.0)) + i.x + vec3(0.0, i1.x, 1.0));
vec3 m = max(0.5 - vec3(dot(x0,x0), dot(x12.xy,x12.xy), dot(x12.zw,x12.zw)), 0.0);
m = m * m * m * m;
vec3 x = 2.0 * fract(p * C.www) - 1.0;
vec3 h = abs(x) - 0.5;
vec3 ox = floor(x + 0.5);
vec3 a0 = x - ox;
m *= 1.79284291400159 - 0.85373472095314 * (a0*a0 + h*h);
vec3 g;
g.x = a0.x * x0.x + h.x * x0.y;
g.yz = a0.yz * x12.xz + h.yz * x12.yw;
return 130.0 * dot(m, g);
}
/* ───────────────────────────────────────────────────────────────────────── */
void main() {
vec2 uv = gl_FragCoord.xy / stateSize;
vec4 s = texture2D(statePrev, uv);
vec2 pos = s.xy;
vec2 vel = s.zw;
/* 1. Noise force */
float t = time * noiseSpeed;
float nx = snoise2(vec2(pos.x * noiseScale + t, pos.y * noiseScale));
float ny = snoise2(vec2(pos.y * noiseScale, pos.x * noiseScale + t + 31.41));
vec2 noiseF = vec2(nx, ny) * noiseWeight;
/* 2. Flow feedback */
vec2 uvFlow = clamp(pos * 0.5 + 0.5, 0.0, 1.0);
vec2 flowSmp = texture2D(flowTex, uvFlow).xy;
vec2 flowF = (flowSmp * 2.0 - 1.0) * flowWeight;
/* 3. Target attraction */
vec2 toTarget = target - pos;
vec2 targetF = toTarget * targetWeight;
/* Euler integration */
vel = vel * (1.0 - damping) + (noiseF + flowF + targetF) * forceWeight;
float spd = length(vel);
if (spd > speedLimit) vel *= speedLimit / spd;
pos += vel;
/* Soft boundary bounce */
if (abs(pos.x) > 1.1) { vel.x *= -0.5; pos.x = clamp(pos.x, -1.1, 1.1); }
if (abs(pos.y) > 1.1) { vel.y *= -0.5; pos.y = clamp(pos.y, -1.1, 1.1); }
gl_FragColor = vec4(pos, vel);
}
@@ -0,0 +1,22 @@
precision mediump float;
uniform vec3 baseColor;
uniform vec3 flowColor;
uniform float colorMapAlpha;
uniform float speedAlpha;
varying vec4 vState;
varying float vIsHead;
void main() {
vec2 vel = vState.zw;
float speed = length(vel);
float a = smoothstep(0.0, max(0.00001, speedAlpha), speed);
a *= (vIsHead > 0.5) ? 1.0 : 0.25;
vec2 dir = speed > 0.0001 ? vel / speed : vec2(0.0);
vec3 vCol = vec3(dir.x * 0.5 + 0.5, dir.y * 0.5 + 0.5, length(dir) * 0.5);
vec3 color = mix(baseColor, vCol + flowColor, colorMapAlpha);
gl_FragColor = vec4(color, a);
}
@@ -0,0 +1,25 @@
precision highp float;
attribute float vid; /* 0 … numParticles*2-1 */
uniform sampler2D stateCurr; /* [posX, posY, velX, velY] — this frame */
uniform sampler2D statePrev; /* previous frame */
uniform vec2 stateSize;
uniform float aspect;
varying vec4 vState;
varying float vIsHead;
vec2 idxToUV(float idx) {
return (vec2(mod(idx, stateSize.x), floor(idx / stateSize.x)) + 0.5) / stateSize;
}
void main() {
float pidx = floor(vid / 2.0);
vIsHead = mod(vid, 2.0);
vec2 uv = idxToUV(pidx);
vState = texture2D(stateCurr, uv);
vec4 prev = texture2D(statePrev, uv);
vec2 pos = (vIsHead > 0.5) ? vState.xy : prev.xy;
pos.x /= max(1.0, aspect);
gl_Position = vec4(pos, 0.0, 1.0);
}
@@ -0,0 +1,2 @@
attribute vec2 aPos; /* quad corners: ±1 */
void main() { gl_Position = vec4(aPos, 0.0, 1.0); }
@@ -0,0 +1,419 @@
/**
* tendrilsSim.ts — WebGL2 GPGPU tendril particle simulation.
*
* Architecture:
* - Two RGBA32F ping-pong textures (readTex / writeTex) hold [posX, posY, velX, velY].
* - A GPGPU logic pass writes new positions/velocities to writeFBO each frame.
* - Particles are drawn as GL_LINES (tail=readTex prev, head=readTex new) onto the canvas.
* - A flow FBO accumulates particle velocity directions; logic.glsl reads it back as force.
* - The React component (TendrilsVisualizerCanvas.tsx) owns the RAF loop and calls
* tickTendrils() once per frame — this module never calls requestAnimationFrame.
*/
import screenVert from './tendrils/screen.vert?raw'
import logicGLSL from './tendrils/logic.glsl?raw'
import renderVert from './tendrils/render.vert?raw'
import renderFrag from './tendrils/render.frag?raw'
import flowFrag from './tendrils/flow.frag?raw'
import fadeFrag from './tendrils/fade.frag?raw'
// ── Public types ────────────────────────────────────────────────────────────
export interface TendrilsConfig {
/** GPGPU grid size; total particles = rootNum². 256 or 512. */
rootNum: number
}
export interface TendrilsFrameUniforms {
time: number
damping: number
speedLimit: number
forceWeight: number
noiseWeight: number
noiseScale: number
noiseSpeed: number
flowWeight: number
flowDecay: number
targetWeight: number
target: [number, number] // NDC
trailAlpha: number // canvas fade per frame
colorMapAlpha: number
speedAlpha: number
baseColor: [number, number, number]
flowColor: [number, number, number]
bgColor: [number, number, number]
lineWidth?: number
}
// ── Internal state ───────────────────────────────────────────────────────────
interface TendrilsGLState {
gl: WebGLRenderingContext
logicProg: WebGLProgram
renderProg: WebGLProgram
flowProg: WebGLProgram
fadeProg: WebGLProgram
readTex: WebGLTexture
writeTex: WebGLTexture
readFBO: WebGLFramebuffer
writeFBO: WebGLFramebuffer
flowFBO: WebGLFramebuffer
flowTex: WebGLTexture
flowW: number
flowH: number
vidBuf: WebGLBuffer
quadBuf: WebGLBuffer
cfg: TendrilsConfig
numParticles: number
destroyed: boolean
onContextLost: (e: Event) => void
onContextRestored: () => void
}
const instances = new WeakMap<HTMLCanvasElement, TendrilsGLState>()
// ── Shader compilation ───────────────────────────────────────────────────────
function compileShader(gl: WebGLRenderingContext, type: number, src: string): WebGLShader {
const shader = gl.createShader(type)!
gl.shaderSource(shader, src)
gl.compileShader(shader)
if (!gl.getShaderParameter(shader, gl.COMPILE_STATUS)) {
const info = gl.getShaderInfoLog(shader) ?? 'unknown'
gl.deleteShader(shader)
throw new Error(`Shader compile error:\n${info}\n\nSource:\n${src.slice(0, 300)}`)
}
return shader
}
function createProgram(gl: WebGLRenderingContext, vertSrc: string, fragSrc: string): WebGLProgram {
const prog = gl.createProgram()!
gl.attachShader(prog, compileShader(gl, gl.VERTEX_SHADER, vertSrc))
gl.attachShader(prog, compileShader(gl, gl.FRAGMENT_SHADER, fragSrc))
gl.linkProgram(prog)
if (!gl.getProgramParameter(prog, gl.LINK_STATUS)) {
const info = gl.getProgramInfoLog(prog) ?? 'unknown'
gl.deleteProgram(prog)
throw new Error(`Program link error: ${info}`)
}
return prog
}
// ── Texture / FBO helpers ────────────────────────────────────────────────────
function createStateTexture(gl: WebGLRenderingContext, rootNum: number): WebGLTexture {
const tex = gl.createTexture()!
gl.bindTexture(gl.TEXTURE_2D, tex)
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST)
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST)
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE)
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE)
// Seed with random positions in NDC [-1.0, 1.0] and zero velocity
const data = new Float32Array(rootNum * rootNum * 4)
for (let i = 0; i < rootNum * rootNum; i++) {
data[i * 4 + 0] = (Math.random() * 2 - 1) * 0.9 // posX
data[i * 4 + 1] = (Math.random() * 2 - 1) * 0.9 // posY
data[i * 4 + 2] = 0 // velX
data[i * 4 + 3] = 0 // velY
}
// WebGL2 uses texImage2D with RGBA32F internal format
const gl2 = gl as WebGL2RenderingContext
gl2.texImage2D(gl2.TEXTURE_2D, 0, gl2.RGBA32F, rootNum, rootNum, 0, gl2.RGBA, gl2.FLOAT, data)
gl.bindTexture(gl.TEXTURE_2D, null)
return tex
}
function createFlowTexture(gl: WebGLRenderingContext, w: number, h: number): WebGLTexture {
const tex = gl.createTexture()!
gl.bindTexture(gl.TEXTURE_2D, tex)
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR)
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR)
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE)
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE)
// Half-float for flow FBO: RGBA16F (requires EXT_color_buffer_float in WebGL2)
const gl2 = gl as WebGL2RenderingContext
gl2.texImage2D(gl2.TEXTURE_2D, 0, gl2.RGBA16F, w, h, 0, gl2.RGBA, gl2.HALF_FLOAT, null)
gl.bindTexture(gl.TEXTURE_2D, null)
return tex
}
function createFBO(gl: WebGLRenderingContext, tex: WebGLTexture): WebGLFramebuffer {
const fbo = gl.createFramebuffer()!
gl.bindFramebuffer(gl.FRAMEBUFFER, fbo)
gl.framebufferTexture2D(gl.FRAMEBUFFER, gl.COLOR_ATTACHMENT0, gl.TEXTURE_2D, tex, 0)
gl.bindFramebuffer(gl.FRAMEBUFFER, null)
return fbo
}
// ── Public API ───────────────────────────────────────────────────────────────
export function initTendrilsSim(canvas: HTMLCanvasElement, cfg: TendrilsConfig): boolean {
destroyTendrilsSim(canvas)
// Prefer WebGL2; fall back to WebGL1 for broad compatibility
const gl = (canvas.getContext('webgl2') ?? canvas.getContext('webgl')) as WebGLRenderingContext | null
if (!gl) return false
// WebGL2 requires EXT_color_buffer_float for rendering to float textures
const floatExt = gl.getExtension('EXT_color_buffer_float')
if (!floatExt) {
// Some WebGL1 devices support OES_texture_float_linear
const oesFlt = gl.getExtension('OES_texture_float')
if (!oesFlt) return false
gl.getExtension('OES_texture_float_linear')
}
gl.getExtension('WEBGL_color_buffer_float') // WebGL1 fallback
let logicProg: WebGLProgram
let renderProg: WebGLProgram
let flowProg: WebGLProgram
let fadeProg: WebGLProgram
try {
logicProg = createProgram(gl, screenVert, logicGLSL)
renderProg = createProgram(gl, renderVert, renderFrag)
flowProg = createProgram(gl, renderVert, flowFrag)
fadeProg = createProgram(gl, screenVert, fadeFrag)
} catch (e) {
console.error('[TendrilsSim] Shader error:', e)
return false
}
const { rootNum } = cfg
const numParticles = rootNum * rootNum
const readTex = createStateTexture(gl, rootNum)
const writeTex = createStateTexture(gl, rootNum)
const readFBO = createFBO(gl, readTex)
const writeFBO = createFBO(gl, writeTex)
const w = Math.max(1, canvas.clientWidth || canvas.width)
const h = Math.max(1, canvas.clientHeight || canvas.height)
const flowTex = createFlowTexture(gl, w, h)
const flowFBO = createFBO(gl, flowTex)
// vid buffer: sequential float IDs [0, 1, …, numParticles*2 - 1]
const vidBuf = gl.createBuffer()!
gl.bindBuffer(gl.ARRAY_BUFFER, vidBuf)
const vidArr = new Float32Array(numParticles * 2)
for (let i = 0; i < vidArr.length; i++) vidArr[i] = i
gl.bufferData(gl.ARRAY_BUFFER, vidArr, gl.STATIC_DRAW)
// screen quad: two triangles covering [-1,-1] to [1,1]
const quadBuf = gl.createBuffer()!
gl.bindBuffer(gl.ARRAY_BUFFER, quadBuf)
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([-1,-1, 1,-1, 1,1, -1,1]), gl.STATIC_DRAW)
gl.bindBuffer(gl.ARRAY_BUFFER, null)
const state: TendrilsGLState = {
gl, logicProg, renderProg, flowProg, fadeProg,
readTex, writeTex, readFBO, writeFBO,
flowFBO, flowTex,
flowW: w, flowH: h,
vidBuf, quadBuf,
cfg: { ...cfg },
numParticles,
destroyed: false,
onContextLost: () => {},
onContextRestored: () => {},
}
state.onContextLost = (e: Event) => {
e.preventDefault()
state.destroyed = true
}
state.onContextRestored = () => {
// Full re-init on restore; previous GL objects are invalid
if (!instances.has(canvas)) return
state.destroyed = true
initTendrilsSim(canvas, state.cfg)
}
canvas.addEventListener('webglcontextlost', state.onContextLost)
canvas.addEventListener('webglcontextrestored', state.onContextRestored)
instances.set(canvas, state)
return true
}
// ── Per-frame rendering ──────────────────────────────────────────────────────
export function tickTendrils(canvas: HTMLCanvasElement, u: TendrilsFrameUniforms): void {
const st = instances.get(canvas)
if (!st || st.destroyed) return
const { gl } = st
const w = Math.max(1, Math.round(canvas.clientWidth || 0) || canvas.width)
const h = Math.max(1, Math.round(canvas.clientHeight || 0) || canvas.height)
if (canvas.width !== w || canvas.height !== h) {
canvas.width = w
canvas.height = h
}
// Resize flow FBO if canvas changed
if (st.flowW !== w || st.flowH !== h) {
gl.deleteFramebuffer(st.flowFBO)
gl.deleteTexture(st.flowTex)
st.flowTex = createFlowTexture(gl, w, h)
st.flowFBO = createFBO(gl, st.flowTex)
st.flowW = w
st.flowH = h
}
const { rootNum } = st.cfg
const numP = st.numParticles
const aspect = w / h
// ── 1. GPGPU logic pass ───────────────────────────────────────────────────
gl.bindFramebuffer(gl.FRAMEBUFFER, st.writeFBO)
gl.viewport(0, 0, rootNum, rootNum)
gl.disable(gl.BLEND)
gl.useProgram(st.logicProg)
setTex(gl, st.logicProg, 'statePrev', st.readTex, 0)
setTex(gl, st.logicProg, 'flowTex', st.flowTex, 1)
setVec2(gl, st.logicProg, 'stateSize', rootNum, rootNum)
setFloat(gl, st.logicProg, 'time', u.time)
setFloat(gl, st.logicProg, 'damping', u.damping)
setFloat(gl, st.logicProg, 'speedLimit', u.speedLimit)
setFloat(gl, st.logicProg, 'forceWeight', u.forceWeight)
setFloat(gl, st.logicProg, 'noiseWeight', u.noiseWeight)
setFloat(gl, st.logicProg, 'noiseScale', u.noiseScale)
setFloat(gl, st.logicProg, 'noiseSpeed', u.noiseSpeed)
setFloat(gl, st.logicProg, 'flowWeight', u.flowWeight)
setVec2(gl, st.logicProg, 'target', u.target[0], u.target[1])
setFloat(gl, st.logicProg, 'targetWeight', u.targetWeight)
drawQuad(gl, st.logicProg, st.quadBuf)
// ── 2. Swap ping-pong ─────────────────────────────────────────────────────
// After swap: readTex = current frame, writeTex = previous frame (for tail)
;[st.readTex, st.writeTex] = [st.writeTex, st.readTex]
;[st.readFBO, st.writeFBO] = [st.writeFBO, st.readFBO]
// ── 3. Flow fade pass ─────────────────────────────────────────────────────
gl.bindFramebuffer(gl.FRAMEBUFFER, st.flowFBO)
gl.viewport(0, 0, st.flowW, st.flowH)
gl.enable(gl.BLEND)
gl.blendEquation(gl.FUNC_ADD)
gl.blendFunc(gl.SRC_ALPHA, gl.ONE_MINUS_SRC_ALPHA)
gl.useProgram(st.fadeProg)
// Fade toward (0.5, 0.5, 0.5) = zero velocity in flow encoding
const fadeLoc = gl.getUniformLocation(st.fadeProg, 'uFade')
gl.uniform4f(fadeLoc, 0.5, 0.5, 0.5, u.flowDecay)
drawQuad(gl, st.fadeProg, st.quadBuf)
// ── 4. Flow render pass ───────────────────────────────────────────────────
// Still bound to flowFBO; accumulate head velocity into flow texture
gl.blendFunc(gl.SRC_ALPHA, gl.ONE_MINUS_SRC_ALPHA)
gl.useProgram(st.flowProg)
setTex(gl, st.flowProg, 'stateCurr', st.readTex, 0)
setTex(gl, st.flowProg, 'statePrev', st.writeTex, 1)
setVec2(gl, st.flowProg, 'stateSize', rootNum, rootNum)
setFloat(gl, st.flowProg, 'aspect', aspect)
drawLines(gl, st.flowProg, st.vidBuf, numP)
// ── 5. Canvas trail fade ──────────────────────────────────────────────────
gl.bindFramebuffer(gl.FRAMEBUFFER, null)
gl.viewport(0, 0, w, h)
gl.enable(gl.BLEND)
gl.blendFunc(gl.SRC_ALPHA, gl.ONE_MINUS_SRC_ALPHA)
gl.useProgram(st.fadeProg)
const fadeLoc2 = gl.getUniformLocation(st.fadeProg, 'uFade')
gl.uniform4f(fadeLoc2, u.bgColor[0], u.bgColor[1], u.bgColor[2], u.trailAlpha)
drawQuad(gl, st.fadeProg, st.quadBuf)
// ── 6. View render pass (GL_LINES onto canvas) ────────────────────────────
gl.blendFunc(gl.SRC_ALPHA, gl.ONE) // additive blending for glow effect
gl.useProgram(st.renderProg)
setTex(gl, st.renderProg, 'stateCurr', st.readTex, 0)
setTex(gl, st.renderProg, 'statePrev', st.writeTex, 1)
setVec2(gl, st.renderProg, 'stateSize', rootNum, rootNum)
setFloat(gl, st.renderProg, 'aspect', aspect)
setVec3(gl, st.renderProg, 'baseColor', u.baseColor)
setVec3(gl, st.renderProg, 'flowColor', u.flowColor)
setFloat(gl, st.renderProg, 'colorMapAlpha', u.colorMapAlpha)
setFloat(gl, st.renderProg, 'speedAlpha', u.speedAlpha)
try { gl.lineWidth(Math.max(1, u.lineWidth ?? 1)) } catch { /* silently clamp on most GPUs */ }
drawLines(gl, st.renderProg, st.vidBuf, numP)
gl.disable(gl.BLEND)
gl.bindBuffer(gl.ARRAY_BUFFER, null)
gl.useProgram(null)
gl.bindTexture(gl.TEXTURE_2D, null)
gl.activeTexture(gl.TEXTURE0)
}
export function destroyTendrilsSim(canvas: HTMLCanvasElement): void {
const st = instances.get(canvas)
if (!st) return
st.destroyed = true
canvas.removeEventListener('webglcontextlost', st.onContextLost)
canvas.removeEventListener('webglcontextrestored', st.onContextRestored)
const { gl } = st
try {
gl.deleteProgram(st.logicProg)
gl.deleteProgram(st.renderProg)
gl.deleteProgram(st.flowProg)
gl.deleteProgram(st.fadeProg)
gl.deleteTexture(st.readTex)
gl.deleteTexture(st.writeTex)
gl.deleteTexture(st.flowTex)
gl.deleteFramebuffer(st.readFBO)
gl.deleteFramebuffer(st.writeFBO)
gl.deleteFramebuffer(st.flowFBO)
gl.deleteBuffer(st.vidBuf)
gl.deleteBuffer(st.quadBuf)
} catch { /* context may already be lost */ }
instances.delete(canvas)
}
// ── GL helpers ───────────────────────────────────────────────────────────────
function setTex(gl: WebGLRenderingContext, prog: WebGLProgram, name: string, tex: WebGLTexture, unit: number): void {
const loc = gl.getUniformLocation(prog, name)
if (loc === null) return
gl.activeTexture(gl.TEXTURE0 + unit)
gl.bindTexture(gl.TEXTURE_2D, tex)
gl.uniform1i(loc, unit)
}
function setFloat(gl: WebGLRenderingContext, prog: WebGLProgram, name: string, v: number): void {
const loc = gl.getUniformLocation(prog, name)
if (loc !== null) gl.uniform1f(loc, v)
}
function setVec2(gl: WebGLRenderingContext, prog: WebGLProgram, name: string, x: number, y: number): void {
const loc = gl.getUniformLocation(prog, name)
if (loc !== null) gl.uniform2f(loc, x, y)
}
function setVec3(gl: WebGLRenderingContext, prog: WebGLProgram, name: string, v: [number, number, number]): void {
const loc = gl.getUniformLocation(prog, name)
if (loc !== null) gl.uniform3f(loc, v[0], v[1], v[2])
}
function drawQuad(gl: WebGLRenderingContext, prog: WebGLProgram, buf: WebGLBuffer): void {
gl.bindBuffer(gl.ARRAY_BUFFER, buf)
const aPos = gl.getAttribLocation(prog, 'aPos')
if (aPos >= 0) {
gl.enableVertexAttribArray(aPos)
gl.vertexAttribPointer(aPos, 2, gl.FLOAT, false, 0, 0)
}
gl.drawArrays(gl.TRIANGLE_FAN, 0, 4)
if (aPos >= 0) gl.disableVertexAttribArray(aPos)
}
function drawLines(gl: WebGLRenderingContext, prog: WebGLProgram, buf: WebGLBuffer, numP: number): void {
gl.bindBuffer(gl.ARRAY_BUFFER, buf)
const aVid = gl.getAttribLocation(prog, 'vid')
if (aVid >= 0) {
gl.enableVertexAttribArray(aVid)
gl.vertexAttribPointer(aVid, 1, gl.FLOAT, false, 0, 0)
}
gl.drawArrays(gl.LINES, 0, numP * 2)
if (aVid >= 0) gl.disableVertexAttribArray(aVid)
}
+139 -71
View File
@@ -3,6 +3,7 @@ import {
deleteStem,
exportProjectKaraokeMugen,
exportProjectSek,
getAppSettings,
getProjectKaraokeConfig,
getJob,
importSubsonicSong,
@@ -10,11 +11,13 @@ import {
lyricTranscribeProject,
renameStem,
reprocessStem,
submitPipeline,
submitSeparateFromPath,
subsonicStreamUrl,
transcribeProject,
} from '../api/client'
import type { Job, SubsonicSong } from '../types'
import { readAutoFormat, readAutoModel, readAutoSteps } from '../utils/autoPipeline'
import { normalizeAutoPresets, readAutoFormat, readAutoModel, readAutoPresets, readAutoSteps } from '../utils/autoPipeline'
import { useProjectsIndex } from './useProjectsIndex'
import { reportNonFatal } from '../utils/nonFatal'
@@ -26,6 +29,75 @@ interface ConvertState {
jobId?: string
}
type ConvertProcessMode = 'auto' | 'separate_only' | 'pipeline' | 'preset'
async function runAdditionalSteps(projectId: string, steps: ReturnType<typeof readAutoSteps>) {
for (const step of steps) {
if (step.type === 'reprocess') {
const { job_id } = await reprocessStem(projectId, step.stem, step.model, {
outputFormat: step.outputFormat ?? 'mp3',
mp3Bitrate: step.mp3Bitrate ?? '256k',
})
await pollJobUntilTerminal(job_id)
} else if (step.type === 'midi') {
const { job_id } = await transcribeProject(projectId, 'vocals', step.mode)
await pollJobUntilTerminal(job_id)
} else if (step.type === 'lyrics') {
const wordTs = localStorage.getItem('stemengine-whisper-word-ts') !== 'false'
const engine = step.engine ?? 'whisperx'
const modelId = step.model ?? step.whisperModel ?? (engine === 'parakeet' ? 'parakeet-1.1b-rnnt-multilingual-asr' : 'medium')
const { job_id } = await lyricTranscribeProject(projectId, step.stem ?? 'vocals', {
engine,
model: modelId,
wordTimestamps: wordTs,
diarize: engine === 'whisperx' ? step.diarize : false,
lsyncAlign: engine === 'whisperx' ? (step.lsyncAlign ?? false) : false,
twoPass: engine === 'whisperx' ? (step.twoPass ?? false) : false,
vocalStyle: engine === 'whisperx' ? (step.vocalStyle ?? 'general') : 'general',
explicit: engine === 'whisperx' ? (step.explicit ?? false) : false,
})
await pollJobUntilTerminal(job_id)
} else if (step.type === 'lyricsAlign') {
const { job_id } = await lrcProject(projectId, 'vocals', {
lyricsSource: 'auto',
timingPrecision: step.timingPrecision,
lrcPowerPreset: localStorage.getItem('stemengine-sep-power') ?? 'standard',
})
await pollJobUntilTerminal(job_id)
} else if (step.type === 'renameStem') {
const fromStem = step.fromStem.trim()
const toStem = step.toStem.trim()
if (fromStem && toStem && fromStem !== toStem) {
await renameStem(projectId, fromStem, toStem)
}
} else if (step.type === 'deleteStem') {
const stem = step.stem.trim()
if (stem) await deleteStem(projectId, stem)
} else if (step.type === 'karaokeMugen') {
await exportProjectKaraokeMugen(projectId, {
defaultTitleLang: step.defaultTitleLang ?? 'eng',
repository: step.repository ?? 'stemengine.local',
})
} else if (step.type === 'sekPackage') {
let resolvedVideoSource: string | undefined
try {
const karaokeCfg = await getProjectKaraokeConfig(projectId)
const candidate = karaokeCfg?.config?.video_source?.trim()
if (candidate) resolvedVideoSource = candidate
} catch (error) {
// non-fatal: fallback to standard includeVideo behavior
reportNonFatal('useConvertStatus: getProjectKaraokeConfig', error, { projectId })
}
await exportProjectSek(projectId, {
packageName: step.packageName,
includeVideo: resolvedVideoSource ? true : (step.includeVideo ?? true),
videoSource: resolvedVideoSource,
})
}
}
}
async function pollJobUntilTerminal(jobId: string): Promise<Job> {
for (;;) {
const job = await getJob(jobId)
@@ -70,15 +142,63 @@ export function useConvertStatus(songs: SubsonicSong[]) {
}))
try {
const imported = await importSubsonicSong(song)
const model = readAutoModel()
const format = readAutoFormat()
const projectName = `${song.songArtist ?? song.artist} - ${song.title}`.trim()
let processMode: ConvertProcessMode = 'auto'
let processPresetName = ''
let storedPresets = readAutoPresets()
try {
const settings = await getAppSettings()
const rawMode = String((settings as { singui_process_mode?: unknown }).singui_process_mode || 'auto').trim().toLowerCase()
processMode = (rawMode === 'separate_only' || rawMode === 'pipeline' || rawMode === 'preset') ? rawMode : 'auto'
processPresetName = String((settings as { singui_process_preset?: unknown }).singui_process_preset || '').trim()
const serverPresets = normalizeAutoPresets(
(settings as { auto_presets_effective?: unknown[]; auto_presets?: unknown[] }).auto_presets_effective
?? (settings as { auto_presets?: unknown[] }).auto_presets
)
if (serverPresets.length > 0) storedPresets = serverPresets
} catch (error) {
reportNonFatal('useConvertStatus: getAppSettings', error)
}
const submit = await submitSeparateFromPath(imported.path, model, format, {
presetType: 'stems_only',
projectName,
})
const projectName = `${song.songArtist ?? song.artist} - ${song.title}`.trim()
let submit: { job_id: string }
if (processMode === 'pipeline') {
const res = await fetch(subsonicStreamUrl(song.id))
if (!res.ok) {
throw new Error(`Failed to fetch source audio for pipeline (${res.status})`)
}
const blob = await res.blob()
const ext = (song.suffix || 'mp3').replace(/[^a-zA-Z0-9]/g, '') || 'mp3'
const file = new File([blob], `${projectName}.${ext}`, { type: blob.type || 'audio/mpeg' })
submit = await submitPipeline(file, readAutoModel(), readAutoFormat(), 'polyphonic', projectName)
} else {
const imported = await importSubsonicSong(song)
let model = readAutoModel()
let format = readAutoFormat()
let instrumentModels: Record<string, string> | undefined
if (processMode === 'preset') {
const presets = storedPresets
const preset = (processPresetName
? presets.find((p) => p.name === processPresetName)
: undefined) ?? presets[0]
if (preset) {
model = preset.model
format = preset.format
instrumentModels = preset.instrumentModels
} else {
reportNonFatal('useConvertStatus: preset mode fallback', new Error('No presets found; falling back to auto defaults'), {
requestedPreset: processPresetName || null,
})
}
}
submit = await submitSeparateFromPath(imported.path, model, format, {
presetType: 'stems_only',
projectName,
instrumentModels,
})
}
setStateBySongId((prev) => ({
...prev,
@@ -90,69 +210,17 @@ export function useConvertStatus(songs: SubsonicSong[]) {
if (finalJob.status === 'done') {
const projectId = finalJob.result?.project_id as string | undefined
if (projectId) {
const steps = readAutoSteps()
for (const step of steps) {
if (step.type === 'reprocess') {
const { job_id } = await reprocessStem(projectId, step.stem, step.model, {
outputFormat: step.outputFormat ?? 'mp3',
mp3Bitrate: step.mp3Bitrate ?? '256k',
})
await pollJobUntilTerminal(job_id)
} else if (step.type === 'midi') {
const { job_id } = await transcribeProject(projectId, 'vocals', step.mode)
await pollJobUntilTerminal(job_id)
} else if (step.type === 'lyrics') {
const wordTs = localStorage.getItem('stemengine-whisper-word-ts') !== 'false'
const engine = step.engine ?? 'whisperx'
const modelId = step.model ?? step.whisperModel ?? (engine === 'parakeet' ? 'parakeet-1.1b-rnnt-multilingual-asr' : 'medium')
const { job_id } = await lyricTranscribeProject(projectId, step.stem ?? 'vocals', {
engine,
model: modelId,
wordTimestamps: wordTs,
diarize: engine === 'whisperx' ? step.diarize : false,
lsyncAlign: engine === 'whisperx' ? (step.lsyncAlign ?? false) : false,
twoPass: engine === 'whisperx' ? (step.twoPass ?? false) : false,
vocalStyle: engine === 'whisperx' ? (step.vocalStyle ?? 'general') : 'general',
explicit: engine === 'whisperx' ? (step.explicit ?? false) : false,
})
await pollJobUntilTerminal(job_id)
} else if (step.type === 'lyricsAlign') {
const { job_id } = await lrcProject(projectId, 'vocals', {
lyricsSource: 'auto',
timingPrecision: step.timingPrecision,
lrcPowerPreset: localStorage.getItem('stemengine-sep-power') ?? 'standard',
})
await pollJobUntilTerminal(job_id)
} else if (step.type === 'renameStem') {
const fromStem = step.fromStem.trim()
const toStem = step.toStem.trim()
if (fromStem && toStem && fromStem !== toStem) {
await renameStem(projectId, fromStem, toStem)
}
} else if (step.type === 'deleteStem') {
const stem = step.stem.trim()
if (stem) await deleteStem(projectId, stem)
} else if (step.type === 'karaokeMugen') {
await exportProjectKaraokeMugen(projectId, {
defaultTitleLang: step.defaultTitleLang ?? 'eng',
repository: step.repository ?? 'stemengine.local',
})
} else if (step.type === 'sekPackage') {
let resolvedVideoSource: string | undefined
try {
const karaokeCfg = await getProjectKaraokeConfig(projectId)
const candidate = karaokeCfg?.config?.video_source?.trim()
if (candidate) resolvedVideoSource = candidate
} catch (error) {
// non-fatal: fallback to standard includeVideo behavior
reportNonFatal('useConvertStatus: getProjectKaraokeConfig', error, { projectId })
}
if (processMode === 'auto') {
await runAdditionalSteps(projectId, readAutoSteps())
}
await exportProjectSek(projectId, {
packageName: step.packageName,
includeVideo: resolvedVideoSource ? true : (step.includeVideo ?? true),
videoSource: resolvedVideoSource,
})
if (processMode === 'preset') {
const presets = storedPresets
const preset = (processPresetName
? presets.find((p) => p.name === processPresetName)
: undefined) ?? presets[0]
if (preset) {
await runAdditionalSteps(projectId, preset.steps)
}
}
}
+3 -3
View File
@@ -503,10 +503,10 @@ export default function AlbumsPage() {
>
<Zap className="w-4 h-4" />
{resolveAlbumStatus(album.id) === 'queued' && 'Queued'}
{resolveAlbumStatus(album.id) === 'running' && 'Packaging'}
{resolveAlbumStatus(album.id) === 'ready' && 'Packaged'}
{resolveAlbumStatus(album.id) === 'running' && 'Preparing'}
{resolveAlbumStatus(album.id) === 'ready' && 'Project Ready'}
{resolveAlbumStatus(album.id) === 'failed' && 'Retry'}
{resolveAlbumStatus(album.id) === 'idle' && 'Package'}
{resolveAlbumStatus(album.id) === 'idle' && 'Prepare Project'}
</button>
</>}
{viewMode === 'small-icon' && (
+152 -7
View File
@@ -30,6 +30,7 @@ import {
readAutoInstrumentModels, writeAutoInstrumentModels,
readAutoPresets, writeAutoPresets,
LS_AUTO_MODEL, LS_AUTO_FORMAT,
normalizeAutoPresets,
stepLabel,
} from '../utils/autoPipeline'
import type { AutoStep, AutoPreset } from '../utils/autoPipeline'
@@ -1875,9 +1876,66 @@ function AutoPipelineSettings() {
const [instrumentModels, setInstrumentModels] = useState<Record<string, string>>(readAutoInstrumentModels)
const [steps, setSteps] = useState<AutoStep[]>(readAutoSteps)
const [addOpen, setAddOpen] = useState(false)
const [presets, setPresets] = useState<AutoPreset[]>(readAutoPresets)
const [presets, setPresets] = useState<AutoPreset[]>([])
const [userPresets, setUserPresets] = useState<AutoPreset[]>([])
const [presetName, setPresetName] = useState('')
const [presetOpen, setPresetOpen] = useState(false)
const presetsHydratedRef = useRef(false)
const presetsMigratedRef = useRef(false)
const { data: appSettings } = useQuery({
queryKey: ['app-settings-autopipeline'],
queryFn: api.getAppSettings,
staleTime: 60_000,
})
const [singUiProcessMode, setSingUiProcessMode] = useState<'auto' | 'separate_only' | 'pipeline' | 'preset'>('auto')
const [singUiProcessPreset, setSingUiProcessPreset] = useState('')
const [singUiProcessSaving, setSingUiProcessSaving] = useState(false)
useEffect(() => {
const next = appSettings?.singui_process_mode
if (next === 'separate_only' || next === 'pipeline' || next === 'preset') {
setSingUiProcessMode(next)
} else {
setSingUiProcessMode('auto')
}
setSingUiProcessPreset(String(appSettings?.singui_process_preset || '').trim())
}, [appSettings?.singui_process_mode, appSettings?.singui_process_preset])
useEffect(() => {
const serverUserPresets = normalizeAutoPresets(appSettings?.auto_presets_user ?? appSettings?.auto_presets)
const serverEffectivePresets = normalizeAutoPresets(appSettings?.auto_presets_effective ?? appSettings?.auto_presets)
const displayPresets = serverEffectivePresets.length > 0 ? serverEffectivePresets : serverUserPresets
if (displayPresets.length > 0) {
setPresets(displayPresets)
setUserPresets(serverUserPresets)
writeAutoPresets(displayPresets)
presetsHydratedRef.current = true
return
}
if (presetsHydratedRef.current) return
const localPresets = readAutoPresets()
setPresets(localPresets)
setUserPresets(localPresets)
presetsHydratedRef.current = true
if (!presetsMigratedRef.current && localPresets.length > 0) {
presetsMigratedRef.current = true
void api.patchAppSettings({ auto_presets: localPresets }).catch(() => {
// Non-fatal: keep local fallback if backend write fails.
})
}
}, [appSettings?.auto_presets, appSettings?.auto_presets_user, appSettings?.auto_presets_effective])
function mergeDisplayPresets(nextUserPresets: AutoPreset[]): AutoPreset[] {
const builtins = normalizeAutoPresets(appSettings?.auto_presets_builtin)
if (!builtins.length) return nextUserPresets
const merged = new Map<string, AutoPreset>()
builtins.forEach((preset) => merged.set(preset.name, preset))
nextUserPresets.forEach((preset) => merged.set(preset.name, preset))
return Array.from(merged.values())
}
function saveSteps(next: AutoStep[]) {
setSteps(next)
@@ -1953,9 +2011,14 @@ function AutoPipelineSettings() {
const name = presetName.trim()
if (!name) return
const preset: AutoPreset = { name, model: sepModel, format: sepFormat, instrumentModels, steps }
const next = [...presets.filter(p => p.name !== name), preset]
setPresets(next)
writeAutoPresets(next)
const nextUser = [...userPresets.filter(p => p.name !== name), preset]
const nextDisplay = mergeDisplayPresets(nextUser)
setUserPresets(nextUser)
setPresets(nextDisplay)
writeAutoPresets(nextDisplay)
void api.patchAppSettings({ auto_presets: nextUser }).catch(() => {
// Non-fatal: local copy remains available.
})
setPresetName('')
}
@@ -1970,9 +2033,22 @@ function AutoPipelineSettings() {
}
function deletePreset(name: string) {
const next = presets.filter(p => p.name !== name)
setPresets(next)
writeAutoPresets(next)
const nextUser = userPresets.filter(p => p.name !== name)
const nextDisplay = mergeDisplayPresets(nextUser)
setUserPresets(nextUser)
setPresets(nextDisplay)
writeAutoPresets(nextDisplay)
let nextSelectedPreset: string | undefined
if (singUiProcessMode === 'preset' && singUiProcessPreset === name) {
setSingUiProcessPreset('')
nextSelectedPreset = ''
}
void api.patchAppSettings({
auto_presets: nextUser,
...(nextSelectedPreset !== undefined ? { singui_process_preset: nextSelectedPreset } : {}),
}).catch(() => {
// Non-fatal: local copy remains available.
})
}
function moveStep(idx: number, dir: -1 | 1) {
@@ -2074,6 +2150,75 @@ function AutoPipelineSettings() {
</div>
</div>
<div className="rounded-xl border ui-border ui-surface-2 p-4 space-y-3">
<h3 className="text-xs font-semibold ui-text-muted uppercase tracking-wider">
SingUI Package Behavior
</h3>
<div className="grid grid-cols-1 md:grid-cols-[minmax(0,1fr)_auto] gap-3 items-end">
<label className="space-y-1 block">
<span className="text-xs ui-text-muted uppercase tracking-wider">Default Mode</span>
<select
value={singUiProcessMode}
onChange={(e) => {
const next = e.target.value
if (next === 'separate_only' || next === 'pipeline' || next === 'preset') {
setSingUiProcessMode(next)
} else {
setSingUiProcessMode('auto')
}
}}
className={AP_SELECT}
>
<option value="auto">Auto pipeline (separate + configured steps)</option>
<option value="separate_only">Separation only (skip extra steps)</option>
<option value="pipeline">Backend full pipeline endpoint</option>
<option value="preset">Specific Auto Pipeline preset</option>
</select>
</label>
{singUiProcessMode === 'preset' && (
<label className="space-y-1 block md:col-span-2">
<span className="text-xs ui-text-muted uppercase tracking-wider">Preset</span>
<select
value={singUiProcessPreset}
onChange={(e) => setSingUiProcessPreset(e.target.value)}
className={AP_SELECT}
disabled={presets.length === 0}
>
<option value="">{presets.length > 0 ? 'Use first saved preset' : 'No presets available'}</option>
{presets.map((preset) => (
<option key={preset.name} value={preset.name}>{preset.name}</option>
))}
</select>
</label>
)}
<button
onClick={async () => {
try {
setSingUiProcessSaving(true)
await api.patchAppSettings({
singui_process_mode: singUiProcessMode,
singui_process_preset: singUiProcessMode === 'preset' ? singUiProcessPreset : '',
})
} finally {
setSingUiProcessSaving(false)
}
}}
className="rounded-lg border ui-border px-3 py-2 text-xs font-medium ui-text-muted hover:opacity-90 disabled:opacity-60"
disabled={singUiProcessSaving || (singUiProcessMode === 'preset' && presets.length === 0)}
>
{singUiProcessSaving ? 'Saving…' : 'Save mode'}
</button>
</div>
<p className="text-xs ui-text-muted">
Used by Package actions in SingUI/Tracks/Albums.
</p>
{singUiProcessMode === 'preset' && presets.length === 0 && (
<p className="text-xs text-amber-600 dark:text-amber-400">
Save at least one Auto Pipeline preset above before using preset mode.
</p>
)}
</div>
{/* Primary Separation */}
<div className="space-y-4">
<h3 className="text-xs font-semibold ui-text-muted uppercase tracking-wider">
+35 -46
View File
@@ -322,15 +322,15 @@ export default function SingPage() {
<button
onClick={() => void convertSong(song)}
disabled={statusBySongId[song.id]?.status === 'queued' || statusBySongId[song.id]?.status === 'running'}
title={statusBySongId[song.id]?.error || 'Package with StemEngine Auto Pipeline'}
title={statusBySongId[song.id]?.error || 'Prepare karaoke project (mode from Settings → Auto Pipeline)'}
className="inline-flex items-center gap-2 rounded bg-brand-500 px-3 py-2 text-sm font-medium text-white hover:bg-brand-600 disabled:cursor-not-allowed disabled:opacity-70"
>
<Zap className="h-4 w-4" />
{statusBySongId[song.id]?.status === 'queued' && 'Queued'}
{statusBySongId[song.id]?.status === 'running' && 'Packaging'}
{statusBySongId[song.id]?.status === 'ready' && 'Packaged'}
{statusBySongId[song.id]?.status === 'running' && 'Preparing'}
{statusBySongId[song.id]?.status === 'ready' && 'Project Ready'}
{statusBySongId[song.id]?.status === 'failed' && 'Retry'}
{(!statusBySongId[song.id] || statusBySongId[song.id]?.status === 'idle') && 'Package'}
{(!statusBySongId[song.id] || statusBySongId[song.id]?.status === 'idle') && 'Prepare Project'}
</button>
</div>
))}
@@ -697,25 +697,26 @@ export default function SingPage() {
const sekFolder = sekFolderInput.trim() || undefined
const primaryMedia = primaryMediaFromSongAndMatches(song, sekCandidates, sekFolder)
const selectedSekMedia = selectedAmbiguousCandidate ? sekMediaFromMatch(song, selectedAmbiguousCandidate, sekFolder) : null
const topSekMedia = topSek ? sekMediaFromMatch(song, topSek, sekFolder) : null
const isPreparing = status === 'queued' || status === 'running'
const isReady = status === 'ready' && !!linked
const selectedQueueItem = !isPreparing
? (!sekAmbiguous && topSek?.package_id
? buildSekKaraokeQueueItem({
packageId: topSek.package_id,
title: topSek.title || song.title,
artist: song.songArtist ?? song.artist,
sekFolder,
})
: selectedAmbiguousCandidate?.package_id
? buildSekKaraokeQueueItem({
packageId: selectedAmbiguousCandidate.package_id,
title: selectedAmbiguousCandidate.title || song.title,
artist: song.songArtist ?? song.artist,
sekFolder,
})
: (isReady && linked ? buildProjectKaraokeQueueItem(song, linked) : null))
? (linked
? buildProjectKaraokeQueueItem(song, linked)
: (!sekAmbiguous && topSek?.package_id
? buildSekKaraokeQueueItem({
packageId: topSek.package_id,
title: topSek.title || song.title,
artist: song.songArtist ?? song.artist,
sekFolder,
})
: selectedAmbiguousCandidate?.package_id
? buildSekKaraokeQueueItem({
packageId: selectedAmbiguousCandidate.package_id,
title: selectedAmbiguousCandidate.title || song.title,
artist: song.songArtist ?? song.artist,
sekFolder,
})
: null))
: null
return (
@@ -762,11 +763,11 @@ export default function SingPage() {
status === 'queued' ? 'text-yellow-400' :
status === 'failed' ? 'text-red-400' : 'text-gray-600'
}`}>
{status === 'ready' && '● Ready'}
{status === 'running' && '● Packaging'}
{status === 'ready' && '● Project Ready'}
{status === 'running' && '● Preparing'}
{status === 'queued' && '● Queued'}
{status === 'failed' && '● Failed'}
{status === 'idle' && '○ Not packaged'}
{status === 'idle' && '○ Project not ready'}
</span>
</div>
<div className="flex shrink-0 items-center gap-2">
@@ -792,22 +793,19 @@ export default function SingPage() {
{isPreparing && (
<span className="inline-flex items-center gap-1.5 text-xs text-gray-400">
<Loader2 className="h-3.5 w-3.5 animate-spin" />
{status === 'queued' ? 'Queued' : 'Packaging…'}
{status === 'queued' ? 'Queued' : 'Preparing…'}
</span>
)}
{!isPreparing && hasStrongSek && !sekAmbiguous && topSekMedia && (
{!isPreparing && isReady && linked && (
<button
onClick={() => {
playNow([topSekMedia])
launchMediaItem(navigate, topSekMedia)
}}
className="inline-flex items-center gap-1.5 rounded bg-emerald-600 px-3 py-1.5 text-xs font-medium text-white hover:bg-emerald-700"
onClick={() => navigate(KARAOKE_ROUTE, { state: { projectId: linked.id } })}
className="inline-flex items-center gap-1.5 rounded bg-brand-500 px-3 py-1.5 text-xs font-medium text-white hover:bg-brand-600"
>
<Mic2 className="h-3.5 w-3.5" />
Sing (SEK)
Sing (Project)
</button>
)}
{!isPreparing && sekAmbiguous && (
{!isPreparing && !isReady && sekAmbiguous && (
<div className="flex items-center gap-2">
<select
value={selectedAmbiguousId}
@@ -841,22 +839,13 @@ export default function SingPage() {
)}
</div>
)}
{!isPreparing && !isReady && !hasStrongSek && !sekAmbiguous && (
{!isPreparing && !isReady && (
<button
onClick={() => void convertSong(song)}
className="inline-flex items-center gap-1.5 rounded border border-brand-500/50 px-2.5 py-1.5 text-xs font-medium text-brand-400 hover:bg-brand-500/10"
>
<Zap className="h-3.5 w-3.5" />
{status === 'failed' ? 'Retry' : 'Package'}
</button>
)}
{isReady && (!hasStrongSek || sekAmbiguous) && (
<button
onClick={() => navigate(KARAOKE_ROUTE, { state: { projectId: linked!.id } })}
className="inline-flex items-center gap-1.5 rounded bg-brand-500 px-3 py-1.5 text-xs font-medium text-white hover:bg-brand-600"
>
<Mic2 className="h-3.5 w-3.5" />
Sing (Project)
{status === 'failed' ? 'Retry' : 'Prepare Project'}
</button>
)}
{status === 'ready' && !linked && projectsLoading && (
@@ -908,10 +897,10 @@ export default function SingPage() {
>
<Zap className="h-4 w-4" />
{albumStatus(album.id) === 'queued' && 'Queued'}
{albumStatus(album.id) === 'running' && 'Packaging'}
{albumStatus(album.id) === 'ready' && 'Packaged'}
{albumStatus(album.id) === 'running' && 'Preparing'}
{albumStatus(album.id) === 'ready' && 'Project Ready'}
{albumStatus(album.id) === 'failed' && 'Retry'}
{albumStatus(album.id) === 'idle' && 'Package'}
{albumStatus(album.id) === 'idle' && 'Prepare Project'}
</button>
</div>
</div>
+4 -4
View File
@@ -402,15 +402,15 @@ export default function TracksPage() {
void convertSong(song)
}}
disabled={statusBySongId[track.id]?.status === 'queued' || statusBySongId[track.id]?.status === 'running'}
title={statusBySongId[track.id]?.error || 'Package with StemEngine Auto Pipeline'}
title={statusBySongId[track.id]?.error || 'Prepare karaoke project (mode from Settings → Auto Pipeline)'}
className="flex items-center gap-2 px-3 py-2 rounded text-white text-sm font-medium transition-colors disabled:opacity-70 disabled:cursor-not-allowed bg-brand-500 hover:bg-brand-600"
>
<Zap className="w-4 h-4" />
{statusBySongId[track.id]?.status === 'queued' && 'Queued'}
{statusBySongId[track.id]?.status === 'running' && 'Packaging'}
{statusBySongId[track.id]?.status === 'ready' && 'Packaged'}
{statusBySongId[track.id]?.status === 'running' && 'Preparing'}
{statusBySongId[track.id]?.status === 'ready' && 'Project Ready'}
{statusBySongId[track.id]?.status === 'failed' && 'Retry'}
{(!statusBySongId[track.id] || statusBySongId[track.id]?.status === 'idle') && 'Package'}
{(!statusBySongId[track.id] || statusBySongId[track.id]?.status === 'idle') && 'Prepare Project'}
</button>
</div>
)
+13
View File
@@ -0,0 +1,13 @@
// Vite ?raw imports for GLSL shader source files.
declare module '*.glsl' {
const s: string
export default s
}
declare module '*.vert' {
const s: string
export default s
}
declare module '*.frag' {
const s: string
export default s
}
+43 -7
View File
@@ -152,6 +152,47 @@ function migrateStep(step: AutoStep): AutoStep {
}
}
function normalizePreset(input: unknown): AutoPreset | null {
if (!input || typeof input !== 'object') return null
const raw = input as {
name?: unknown
model?: unknown
format?: unknown
instrumentModels?: unknown
steps?: unknown
}
const name = typeof raw.name === 'string' ? raw.name.trim() : ''
const model = typeof raw.model === 'string' ? raw.model.trim() : ''
const format = typeof raw.format === 'string' ? raw.format.trim() : ''
if (!name || !model || !format) return null
const instrumentModels: Record<string, string> = {}
if (raw.instrumentModels && typeof raw.instrumentModels === 'object') {
for (const [k, v] of Object.entries(raw.instrumentModels as Record<string, unknown>)) {
if (typeof v === 'string' && v.trim()) instrumentModels[k] = v.trim()
}
}
const steps = Array.isArray(raw.steps) ? (raw.steps as AutoStep[]).map(migrateStep) : []
return {
name,
model,
format,
instrumentModels,
steps,
}
}
export function normalizeAutoPresets(input: unknown): AutoPreset[] {
if (!Array.isArray(input)) return []
const out: AutoPreset[] = []
for (const item of input) {
const normalized = normalizePreset(item)
if (normalized) out.push(normalized)
}
return out
}
export function readAutoSteps(): AutoStep[] {
try {
const raw = JSON.parse(localStorage.getItem(LS_AUTO_STEPS) || '[]') as AutoStep[]
@@ -171,13 +212,8 @@ export function writeAutoSteps(steps: AutoStep[]): void {
export function readAutoPresets(): AutoPreset[] {
try {
const raw = JSON.parse(localStorage.getItem(LS_AUTO_PRESETS) || '[]') as AutoPreset[]
if (!Array.isArray(raw)) return []
const migrated = raw.map((preset) => ({
...preset,
instrumentModels: preset.instrumentModels ?? {},
steps: Array.isArray(preset.steps) ? preset.steps.map(migrateStep) : [],
}))
const raw = JSON.parse(localStorage.getItem(LS_AUTO_PRESETS) || '[]')
const migrated = normalizeAutoPresets(raw)
writeAutoPresets(migrated)
return migrated
} catch {
+1
View File
@@ -0,0 +1 @@
/// <reference types="vite/client" />