Myriad changes.
This commit is contained in:
1 parent
ee851bc9ae
commit
f498650481
50 files changed
+7615
-448
No files matched your search
@@ -0,0 +1,198 @@
|
||||
# 2026-04-13 — Git-Managed Auto Pipeline Presets Plan
|
||||
|
||||
## Goal
|
||||
Make selected Auto Pipeline presets ship with StemEngine from Git so they are available by default on first run, while still allowing local custom presets.
|
||||
|
||||
## Implementation status (2026-04-13)
|
||||
|
||||
### Completed
|
||||
|
||||
- ✅ Added Git-managed built-in preset catalog file:
|
||||
- `backend/presets/auto_pipeline/default_presets.json`
|
||||
- ✅ Added backend catalog loader + normalization/merge helpers:
|
||||
- `backend/services/auto_presets_catalog.py`
|
||||
- ✅ Extended settings response to expose preset layers:
|
||||
- `auto_presets_user`
|
||||
- `auto_presets_builtin`
|
||||
- `auto_presets_effective`
|
||||
- ✅ Kept patch writes for `auto_presets` as user-editable list.
|
||||
- ✅ Frontend Settings hydration now prefers backend effective presets while preserving local fallback.
|
||||
- ✅ One-time local→backend backfill implemented when backend user preset list is empty.
|
||||
- ✅ Preset execution path (`useConvertStatus`) now prefers `auto_presets_effective` from settings API.
|
||||
- ✅ Validation run:
|
||||
- `frontend`: `npm run -s typecheck` (pass)
|
||||
- `frontend`: `npm run -s build` (pass)
|
||||
|
||||
### Remaining (optional)
|
||||
|
||||
- ⏳ UI source badges (`Built-in` / `User` / `Override`) and explicit read-only treatment for built-ins.
|
||||
- ⏳ CI schema validation for preset catalog JSON.
|
||||
|
||||
---
|
||||
|
||||
## Recommended storage model
|
||||
|
||||
Use a **two-tier preset model**:
|
||||
|
||||
1. **Built-in presets (Git-managed, read-only at runtime)**
|
||||
- Stored in repo, versioned with code, reviewed via PR.
|
||||
- Always available after `git pull` + restart.
|
||||
2. **User presets (runtime editable)**
|
||||
- Stored in existing settings (`settings.json` / `auto_presets`) for per-instance customization.
|
||||
|
||||
This avoids losing local changes on upgrade while guaranteeing baseline defaults from Git.
|
||||
|
||||
---
|
||||
|
||||
## Where to store built-ins
|
||||
|
||||
Create a versioned preset catalog file in repo, e.g.:
|
||||
|
||||
- `backend/presets/auto_pipeline/default_presets.json`
|
||||
|
||||
Optional split by profile:
|
||||
|
||||
- `backend/presets/auto_pipeline/default_presets.json`
|
||||
- `backend/presets/auto_pipeline/karaoke_defaults.json`
|
||||
- `backend/presets/auto_pipeline/lightweight_defaults.json`
|
||||
|
||||
JSON schema mirrors current `AutoPreset` frontend shape:
|
||||
|
||||
- `name`
|
||||
- `model`
|
||||
- `format`
|
||||
- `instrumentModels`
|
||||
- `steps`
|
||||
|
||||
---
|
||||
|
||||
## Loading precedence (server-side)
|
||||
|
||||
At runtime, backend resolves effective presets with this order:
|
||||
|
||||
1. Load built-ins from Git file(s).
|
||||
2. Load user presets from settings (`auto_presets`).
|
||||
3. Merge by preset `name`:
|
||||
- user preset with same `name` overrides built-in.
|
||||
- unique names from both sets are kept.
|
||||
|
||||
Expose both views in API response:
|
||||
|
||||
- `auto_presets_builtin` (read-only)
|
||||
- `auto_presets_user` (editable)
|
||||
- `auto_presets_effective` (merged view for execution/UI)
|
||||
|
||||
If keeping API minimal, keep only `auto_presets` as effective merged list + add metadata flags per preset (`source: builtin|user`).
|
||||
|
||||
---
|
||||
|
||||
## Backend plan
|
||||
|
||||
### B1) Add preset catalog loader
|
||||
|
||||
- New helper module, e.g. `backend/services/auto_presets_catalog.py`.
|
||||
- Responsibilities:
|
||||
- read JSON file(s)
|
||||
- validate schema
|
||||
- normalize values
|
||||
- return list of built-in presets
|
||||
|
||||
### B2) Add startup validation
|
||||
|
||||
- On startup, validate catalog and log actionable errors.
|
||||
- If invalid:
|
||||
- log with file path + preset name + field error
|
||||
- continue boot with empty built-in list (non-fatal)
|
||||
|
||||
### B3) Extend settings API
|
||||
|
||||
- Keep patch writes only for user presets (`auto_presets`).
|
||||
- Return merged/effective presets to frontend.
|
||||
- Optionally add endpoint:
|
||||
- `GET /api/settings/auto-presets` (builtin + user + effective)
|
||||
|
||||
### B4) Preserve backward compatibility
|
||||
|
||||
- Existing `auto_presets` in settings remain valid.
|
||||
- Migration path: none required for existing installs.
|
||||
|
||||
---
|
||||
|
||||
## Frontend plan
|
||||
|
||||
### F1) Show source badges
|
||||
|
||||
In Settings → Auto Pipeline preset list:
|
||||
|
||||
- Tag presets with source:
|
||||
- `Built-in`
|
||||
- `User`
|
||||
- `Override` (user replacing built-in name)
|
||||
|
||||
### F2) Edit behavior
|
||||
|
||||
- Built-in presets are not edited in place.
|
||||
- “Save” with same name creates/updates user override.
|
||||
- Delete on override removes user override, revealing built-in again.
|
||||
- Delete on pure built-in is disabled (or “hide locally” if later desired).
|
||||
|
||||
### F3) Selection behavior
|
||||
|
||||
- SingUI `preset` mode selector uses effective merged list.
|
||||
- If selected preset vanishes, fallback to first effective preset with warning toast.
|
||||
|
||||
---
|
||||
|
||||
## Git + deployment workflow
|
||||
|
||||
1. Add/update preset JSON in repo.
|
||||
2. Commit + PR review.
|
||||
3. Deploy updated StemEngine image/code.
|
||||
4. On restart, built-ins appear automatically.
|
||||
5. Existing user presets remain intact.
|
||||
|
||||
This provides “installed/pulled with Git” defaults without clobbering local runtime state.
|
||||
|
||||
---
|
||||
|
||||
## Optional enhancements
|
||||
|
||||
- Support env override path:
|
||||
- `AUTO_PRESETS_CATALOG_PATH=/app/config/auto_presets.json`
|
||||
- Support multiple catalogs and deterministic order.
|
||||
- Add `version` field per preset for future migration tooling.
|
||||
- Add CI check that validates preset JSON schema.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Fresh clone/run shows built-in presets without manual setup.
|
||||
- `git pull` with new presets makes them available after restart.
|
||||
- User-created presets persist separately and survive upgrades.
|
||||
- Built-ins are clearly identifiable in UI.
|
||||
- Execution paths (Auto Pipeline dialog + SingUI preset mode) consume merged effective list.
|
||||
|
||||
---
|
||||
|
||||
## Rollout sequence
|
||||
|
||||
### Phase 1 (backend foundation)
|
||||
- Add catalog file + loader + merge + settings response updates.
|
||||
|
||||
### Phase 2 (frontend UX)
|
||||
- Add source badges and edit/delete semantics for built-in vs user presets.
|
||||
|
||||
### Phase 3 (hardening)
|
||||
- Add schema validation tooling + CI gate.
|
||||
|
||||
---
|
||||
|
||||
## Risks and mitigations
|
||||
|
||||
- **Risk:** malformed preset file breaks startup behavior.
|
||||
- **Mitigation:** non-fatal loader + strong validation logs.
|
||||
- **Risk:** name collisions confuse users.
|
||||
- **Mitigation:** explicit `source` labels and override semantics.
|
||||
- **Risk:** accidental overwrite of built-ins.
|
||||
- **Mitigation:** treat Git presets as read-only in API/UI.
|
||||
@@ -0,0 +1,242 @@
|
||||
# Leading & Backing Buttons Sidebar Refactor — 2026-04-13
|
||||
|
||||
## Overview
|
||||
|
||||
**Objective**: Consolidate karaoke display controls into the performance sidebar by:
|
||||
1. Moving Leading/Backing lyrics track selection buttons from overlay
|
||||
2. Adding line display mode options (Scroll, Line, Word, Syllable) to the sidebar
|
||||
|
||||
**Status**: ✅ Completed and validated
|
||||
|
||||
---
|
||||
|
||||
## Problem Statements
|
||||
|
||||
### Issue 1: Lyrics Track Button Placement
|
||||
Previously, the lyrics track selection buttons (Leading, Backing, Main) were rendered in two locations:
|
||||
1. **Performance mode**: Fixed position overlay at top-right of the video display
|
||||
2. **Editor mode**: Inline buttons in the universal controls toolbar
|
||||
|
||||
This created visual clutter and inconsistency in the UI, especially in fullscreen mode.
|
||||
|
||||
### Issue 2: Display Mode Options Hidden
|
||||
The line display mode selector (Scroll, Line, Word, Syllable) was only available in the style picker panel, requiring users to open a separate UI layer to access frequently-used options.
|
||||
|
||||
---
|
||||
|
||||
## Solution Implemented
|
||||
|
||||
### Changes Made
|
||||
|
||||
#### 1. **PerformanceSidebar Component** (`frontend/src/components/PerformanceSidebar.tsx`)
|
||||
|
||||
**Added Props**:
|
||||
- `lyricsTrackKeys: string[]` — Available lyrics track names (e.g., `['leading', 'default', 'backing']`)
|
||||
- `selectedLyricsTrack: string` — Currently selected track name
|
||||
- `onSelectLyricsTrack: (key: string) => void` — Callback when user selects a track
|
||||
|
||||
**New UI Section**:
|
||||
- Added "LYRICS" section at the top of the sidebar (when multiple tracks exist)
|
||||
- Displays track buttons with the same styling as BACKGROUND section buttons
|
||||
- Responsive: Grid layout on normal widths, flex wrap on narrow viewports
|
||||
- Visual feedback: Selected track button highlighted in violet
|
||||
|
||||
```tsx
|
||||
{/* ── LYRICS TRACKS ── */}
|
||||
{lyricsTrackKeys.length > 1 && (
|
||||
<section className="space-y-2">
|
||||
<div className="text-xs font-semibold tracking-wide text-slate-200">LYRICS</div>
|
||||
<div className={isNarrow ? 'flex flex-wrap gap-2' : 'grid grid-cols-2 gap-2'}>
|
||||
{lyricsTrackKeys.map((key) => (
|
||||
<button
|
||||
key={key}
|
||||
type="button"
|
||||
onClick={() => onSelectLyricsTrack(key)}
|
||||
className={`rounded-md px-2 py-2 text-xs border transition-colors ...`}
|
||||
title={`Show ${key.charAt(0).toUpperCase() + key.slice(1)} lyrics`}
|
||||
>
|
||||
{key.charAt(0).toUpperCase() + key.slice(1)}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</section>
|
||||
)}
|
||||
```
|
||||
|
||||
#### 2. **Display Mode Selector** (NEW)
|
||||
|
||||
**Added Props to PerformanceSidebar**:
|
||||
- `hasElrc: boolean` — Whether word-timing data is available
|
||||
- Uses existing `style` and `patchStyle` props to manage `videoDisplayMode`
|
||||
|
||||
**New UI Section**:
|
||||
- Added "DISPLAY" section in the sidebar
|
||||
- Shows four mode buttons: Scroll, Line, Word, Syllable
|
||||
- Word and Syllable modes are disabled when ELRC (word-timing) data is unavailable
|
||||
- Disabled states show visual feedback (reduced opacity, no-cursor)
|
||||
- Same grid/responsive layout as other sections
|
||||
|
||||
```tsx
|
||||
{/* ── LINE DISPLAY ── */}
|
||||
<section className="space-y-2">
|
||||
<div className="text-xs font-semibold tracking-wide text-slate-200">DISPLAY</div>
|
||||
<div className={isNarrow ? 'flex flex-wrap gap-2' : 'grid grid-cols-2 gap-2'}>
|
||||
{(['scroll', 'line', 'word', 'syllable'] as const).map((mode) => (
|
||||
<button
|
||||
key={mode}
|
||||
type="button"
|
||||
disabled={(mode === 'word' || mode === 'syllable') && !hasElrc}
|
||||
onClick={() => patchStyle({ videoDisplayMode: mode })}
|
||||
className={`rounded-md px-2 py-2 text-xs border transition-colors capitalize ...`}
|
||||
>
|
||||
{mode}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</section>
|
||||
```
|
||||
|
||||
#### 3. **LyricsLivePreview Component** (`frontend/src/components/LyricsLivePreview.tsx`)
|
||||
|
||||
**Removed UI Elements**:
|
||||
- Deleted the fixed-position overlay buttons from performance mode (was at top-right at 84px from top, 10px from right)
|
||||
- Deleted the inline buttons from the editor mode toolbar (was after "ELRC" indicator)
|
||||
- Removed unused `labelLyricsTrackKey()` function (was only used by the removed buttons)
|
||||
|
||||
**Updated Sidebar Integration**:
|
||||
- Added four new props passed to `PerformanceSidebar`:
|
||||
- `lyricsTrackKeys={lyricsTrackKeys}`
|
||||
- `selectedLyricsTrack={selectedLyricsTrack}`
|
||||
- `onSelectLyricsTrack={setSelectedLyricsTrack}`
|
||||
- `hasElrc={hasElrc}` (NEW - for display mode button state)
|
||||
|
||||
#### 3. **Button Styling & UX**
|
||||
|
||||
The lyrics track and display mode buttons now:
|
||||
- Match the visual design of other sidebar sections (Background, Stems, Lyrics)
|
||||
- Use consistent border and background colors
|
||||
- Provide hover states for interactive feedback
|
||||
- Adapt to narrow viewports by wrapping in flex layout
|
||||
- Include descriptive tooltips
|
||||
- For display modes: Show disabled state for unavailable options (Word/Syllable without ELRC data)
|
||||
|
||||
---
|
||||
|
||||
## Files Modified
|
||||
|
||||
| File | Changes |
|
||||
|------|---------|
|
||||
| [`frontend/src/components/PerformanceSidebar.tsx`](frontend/src/components/PerformanceSidebar.tsx#L200-L280) | Added lyrics track + display mode sections; added `hasElrc` prop |
|
||||
| [`frontend/src/components/LyricsLivePreview.tsx`](frontend/src/components/LyricsLivePreview.tsx#L3190-L3230) | Removed overlay + toolbar buttons; removed `labelLyricsTrackKey()`; added sidebar props including `hasElrc` |
|
||||
|
||||
---
|
||||
|
||||
## Visual Impact
|
||||
|
||||
### Before
|
||||
- Lyrics track buttons floating in top-right corner during karaoke (visual clutter)
|
||||
- Same buttons inline in editor toolbar (competing for space)
|
||||
- Display mode options only accessible through style picker panel
|
||||
- Different styling in each location
|
||||
|
||||
### After
|
||||
- Unified "LYRICS" section in the performance sidebar for track selection
|
||||
- New "DISPLAY" section in sidebar for line display modes (Scroll, Line, Word, Syllable)
|
||||
- Clean toolbar without overlapping buttons in editor mode
|
||||
- Single, centralized control panel for all display options
|
||||
- Consistent styling across all sections
|
||||
- Buttons only visible when relevant (display modes always, lyrics tracks only when multiple exist)
|
||||
- Better use of screen space, especially in fullscreen
|
||||
- Word/Syllable modes clearly disabled when timing data unavailable
|
||||
|
||||
### Sidebar Layout (Performance Mode)
|
||||
```
|
||||
┌─────────────────────┐
|
||||
│ Controls ✕ │
|
||||
├─────────────────────┤
|
||||
│ LYRICS │
|
||||
│ [Leading] [Backing] │
|
||||
├─────────────────────┤
|
||||
│ DISPLAY │
|
||||
│ [Scroll] [Line] │
|
||||
│ [Word] [Syllable] │ (Word/Syllable disabled if no ELRC)
|
||||
├─────────────────────┤
|
||||
│ BACKGROUND │
|
||||
│ [Static] [Muon] │
|
||||
│ [Fluid] [Video] │
|
||||
├─────────────────────┤
|
||||
│ STEMS │
|
||||
│ ... (volume controls)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation
|
||||
|
||||
### TypeScript Checking
|
||||
```
|
||||
✅ No errors in PerformanceSidebar.tsx
|
||||
✅ No errors in LyricsLivePreview.tsx
|
||||
```
|
||||
|
||||
### Production Build
|
||||
```
|
||||
✅ Build succeeded in 5.75s
|
||||
✅ All 2860 modules transformed successfully
|
||||
✅ Bundle sizes nominal (523.33 kB gzip)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## User Experience
|
||||
|
||||
**When Performance Sidebar is open**:
|
||||
1. LYRICS section shows track buttons (leading, backing, main)
|
||||
- Currently selected track highlighted in violet
|
||||
- Clicking instantly switches displayed lyrics
|
||||
2. DISPLAY section shows line display mode options
|
||||
- Scroll: Horizontal scrolling of lyrics
|
||||
- Line: Full-screen line display
|
||||
- Word: Word-by-word sync (requires ELRC data)
|
||||
- Syllable: Syllable-by-syllable sync (requires ELRC data)
|
||||
- Disabled options show grayed appearance
|
||||
|
||||
**When Performance Sidebar is closed**:
|
||||
- No display controls visible (sidebar can be toggled with button)
|
||||
|
||||
**In editor mode**:
|
||||
- Lyrics tracks and display modes no longer shown in toolbar
|
||||
- Cleaner interface focused on editing controls (style, timing, effects)
|
||||
- Display options still accessible via style picker panel
|
||||
|
||||
---
|
||||
|
||||
## Backward Compatibility
|
||||
|
||||
✅ **No breaking changes**:
|
||||
- PerformanceSidebar adds optional props (with sensible defaults)
|
||||
- Display mode logic remains unchanged
|
||||
- User preferences persist in localStorage as before
|
||||
- Graceful degradation for Word/Syllable modes when timing data unavailable
|
||||
|
||||
---
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- No additional render loops or re-renders introduced
|
||||
- Button clicks directly update state via `patchStyle()`
|
||||
- Conditional rendering based on `hasElrc` avoids rendering disabled buttons
|
||||
- Same responsive breakpoint logic as existing sidebar sections
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
This refactor successfully consolidates all karaoke display controls into a unified performance sidebar:
|
||||
- **LYRICS section** for track selection (analogous to BACKGROUND section)
|
||||
- **DISPLAY section** for line display modes
|
||||
- Eliminates visual clutter from overlays and toolbars
|
||||
- Improves UX by centralizing related controls
|
||||
- Maintains full functionality while improving discoverability
|
||||
|
||||
The sidebar now serves as the comprehensive control center for performance mode.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,190 @@
|
||||
# 2026-04-13 — SingUI Pipeline Selection + Direct Project Karaoke Plan
|
||||
|
||||
## Goal
|
||||
Make SingUI cleaner by supporting two clear paths:
|
||||
|
||||
1. **Option A:** Add a Settings control to choose which processing pipeline SingUI uses when you click **Package**.
|
||||
2. **Option B:** Support a workflow that **skips `.sek` packaging** and sings directly from existing StemEngine project stems in `/karaoke`.
|
||||
|
||||
## Implementation status (started 2026-04-13)
|
||||
|
||||
### Completed in this pass
|
||||
|
||||
- ✅ Added server-persisted setting: `singui_process_mode` with validation (`auto` | `separate_only` | `pipeline` | `preset`).
|
||||
- ✅ Added server-persisted optional selection: `singui_process_preset`.
|
||||
- ✅ Added Settings UI control under Auto Pipeline to choose/save SingUI package behavior across all four modes.
|
||||
- ✅ Added conditional preset selector in Settings when mode is `preset`.
|
||||
- ✅ Wired SingUI package execution to respect `singui_process_mode`:
|
||||
- `auto` = run configured additional Auto Pipeline steps.
|
||||
- `separate_only` = stop after separation/project creation.
|
||||
- ✅ Extended conversion execution logic to include additional process strategies:
|
||||
- `pipeline` = submit `/api/pipeline` directly (source audio streamed from Subsonic endpoint into upload payload).
|
||||
- `preset` = run separation using selected preset model/format/instrument refinement + execute that preset step list.
|
||||
- ✅ Updated SingUI karaoke row actions to prioritize **project** playback when a linked project exists.
|
||||
- ✅ Updated queue-item preference in project-linked mode to queue project sources before SEK candidates.
|
||||
- ✅ Updated main action text toward project-centric flow (`Prepare Project`) and aligned status copy across Sing/Tracks/Albums (`Preparing`, `Project Ready`).
|
||||
- ✅ Validation pass completed after steps 1–3:
|
||||
- `frontend`: `npm run -s typecheck` (pass)
|
||||
- `frontend`: `npm run -s build` (pass)
|
||||
- ✅ Optional preset-storage migration completed:
|
||||
- Auto Pipeline presets are now read from backend app settings (`auto_presets`) when available.
|
||||
- Settings preset save/delete now persist to backend settings and keep local fallback.
|
||||
- One-time backfill migrates existing local preset list to backend when server list is empty.
|
||||
|
||||
### Remaining from full plan
|
||||
|
||||
- None.
|
||||
|
||||
---
|
||||
|
||||
## Current State (baseline)
|
||||
|
||||
- SingUI package actions currently run a fixed frontend pipeline flow (`import -> separate -> optional steps`) via `useConvertStatus`.
|
||||
- Auto pipeline configuration is primarily browser-local (`localStorage`) and not centrally selected for SingUI routing.
|
||||
- Karaoke page already supports **project mode** (`projectId`) and **SEK mode** (`sekPackageId`), but SingUI karaoke list still emphasizes SEK matching in important paths.
|
||||
|
||||
---
|
||||
|
||||
## Option A — Settings Item: Choose Song Processing Pipeline in SingUI
|
||||
|
||||
### Product outcome
|
||||
Add a setting like:
|
||||
|
||||
- **Settings → Auto Pipeline → “SingUI Package behavior”**
|
||||
- `Auto Pipeline (current behavior)`
|
||||
- `Full /api/pipeline endpoint`
|
||||
- `Separation only`
|
||||
- (optional) `Custom preset: <name>`
|
||||
|
||||
This makes behavior explicit and removes hidden coupling.
|
||||
|
||||
### Implementation plan
|
||||
|
||||
#### A1) Define configuration model (server-persisted)
|
||||
Add fields to app settings so selection is shared across browsers/users of the same instance:
|
||||
|
||||
- `singui_process_mode`: `'auto' | 'pipeline' | 'separate_only' | 'preset'`
|
||||
- `singui_process_preset`: `string | null` (used when mode = `preset`)
|
||||
|
||||
Also decide where pipeline presets live long-term:
|
||||
|
||||
- **Phase 1 (fast):** keep existing local presets; setting only chooses execution mode.
|
||||
- **Phase 2 (clean):** move presets to backend settings for central consistency.
|
||||
|
||||
#### A2) Add Settings UI
|
||||
In Settings Auto Pipeline section:
|
||||
|
||||
- Add a dropdown for default SingUI process mode.
|
||||
- If mode is `preset`, show a second dropdown of presets.
|
||||
- Add inline summary text: “Used by Package buttons in Tracks/Albums/SingUI”.
|
||||
|
||||
#### A3) Wire execution in SingUI packaging path
|
||||
Refactor package execution into a single strategy resolver:
|
||||
|
||||
- `mode=auto`: current `useConvertStatus` behavior.
|
||||
- `mode=pipeline`: submit `/api/pipeline` with chosen defaults.
|
||||
- `mode=separate_only`: only create/update project stems, no post-steps.
|
||||
- `mode=preset`: execute steps from selected preset.
|
||||
|
||||
Key principle: one orchestrator, multiple strategies, same status UI.
|
||||
|
||||
#### A4) Migration and fallback rules
|
||||
- If no setting exists, default to `auto`.
|
||||
- If `preset` points to missing preset, fall back to `auto` + toast warning.
|
||||
- Maintain backward compatibility with existing localStorage keys.
|
||||
|
||||
#### A5) Testing/validation
|
||||
- Unit: strategy resolver picks correct execution path.
|
||||
- Integration: Package button in Tracks, Albums, Sing triggers selected mode.
|
||||
- Regression: queue/status badges unchanged.
|
||||
|
||||
### Option A acceptance criteria
|
||||
- Admin/user can choose SingUI package mode in Settings.
|
||||
- Package actions consistently follow selected mode across pages.
|
||||
- Existing users continue working without reconfiguration.
|
||||
|
||||
### Option A risk/effort
|
||||
- **Effort:** Medium
|
||||
- **Risk:** Low–Medium (mainly migration/consistency between local and server settings)
|
||||
|
||||
---
|
||||
|
||||
## Option B — Skip `.sek`: Sing directly from existing project stems
|
||||
|
||||
### Product outcome
|
||||
Allow karaoke flow to use StemEngine projects directly and treat `.sek` as optional export/distribution, not required for local singing.
|
||||
|
||||
### Implementation plan
|
||||
|
||||
#### B1) Clarify source modes in SingUI karaoke
|
||||
Rename/clarify karaoke source options:
|
||||
|
||||
- `Project-linked` (primary local mode)
|
||||
- `SEK Folder` (portable package mode)
|
||||
|
||||
In `Project-linked` mode, do **not** depend on SEK matching to render playable rows.
|
||||
|
||||
#### B2) Prioritize project playback path
|
||||
For each song row:
|
||||
|
||||
1. If linked project exists → show `Sing` immediately (project source).
|
||||
2. If not linked and user clicks `Package` → run selected processing mode, then auto-link and enable `Sing`.
|
||||
3. SEK badges can still appear as informational, but not gate playback.
|
||||
|
||||
#### B3) Queue semantics
|
||||
When user queues from `Project-linked` mode:
|
||||
|
||||
- Create queue items as `source.kind='project'` with `projectId`.
|
||||
- Keep existing SEK queue behavior unchanged in SEK mode.
|
||||
|
||||
#### B4) UX cleanup
|
||||
- Labels: “Package (create/update project)” instead of implying SEK export.
|
||||
- Move `.sek` export language to explicit export actions only.
|
||||
- Add helper text: “Local karaoke uses project stems; SEK is for portable packages.”
|
||||
|
||||
#### B5) Optional backend support (if needed)
|
||||
If frontend-only linking is insufficient at scale, add endpoint for stable song→project mapping with confidence score.
|
||||
|
||||
### Option B acceptance criteria
|
||||
- User can sing from project stems without creating `.sek`.
|
||||
- Karaoke queue works end-to-end in project mode.
|
||||
- `.sek` remains available but optional.
|
||||
|
||||
### Option B risk/effort
|
||||
- **Effort:** Low–Medium
|
||||
- **Risk:** Low (most underlying project playback support already exists)
|
||||
|
||||
---
|
||||
|
||||
## Decision guidance
|
||||
|
||||
- Choose **Option A first** if your top priority is explicit, configurable processing behavior.
|
||||
- Choose **Option B first** if your top priority is faster daily karaoke flow with less packaging friction.
|
||||
- Best practical sequence:
|
||||
1. **B first** (quick UX win, immediate simplification)
|
||||
2. **A second** (clean policy control for future growth)
|
||||
|
||||
---
|
||||
|
||||
## Suggested phased rollout
|
||||
|
||||
### Phase 1 (1–2 days)
|
||||
- Implement Option B UX/path cleanup for direct project singing.
|
||||
- Keep current auto pipeline execution behavior unchanged.
|
||||
|
||||
### Phase 2 (2–4 days)
|
||||
- Implement Option A mode selector + strategy resolver.
|
||||
- Add migration/fallback logic.
|
||||
|
||||
### Phase 3 (optional hardening)
|
||||
- Move pipeline preset storage from localStorage to backend settings.
|
||||
- Add telemetry/toasts for mode/fallback visibility.
|
||||
|
||||
---
|
||||
|
||||
## Open questions before implementation
|
||||
|
||||
1. Should SingUI mode be global instance-wide or per-user/browser?
|
||||
2. Do we want `preset` mode in v1, or only `auto/pipeline/separate_only`?
|
||||
3. Should `Package` update existing linked projects in-place by default?
|
||||
4. Do we keep SEK matching badges in Project-linked mode, or hide entirely unless SEK mode is selected?
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,932 @@
|
||||
# StemEngine Codebase Audit — 2026-04-14
|
||||
|
||||
**Scope**: Full codebase — frontend (React/TypeScript), backend (Python/FastAPI), visualizers (WebGL/Three.js)
|
||||
**Auditors**: 7 parallel specialist agents (security, React, performance, code quality, TypeScript, Python backend, architecture)
|
||||
**Total findings**: 92 across 7 domains
|
||||
|
||||
---
|
||||
|
||||
## Priority Summary
|
||||
|
||||
| Level | Count | IDs |
|
||||
|-------|-------|-----|
|
||||
| Critical | 2 | P1, A4 |
|
||||
| High | 23 | SEC1–2, R1–4, P2–5, C1–4, PY2–4, TS1, A1–2, A5, A7 |
|
||||
| Medium | 36 | SEC3–7, SEC9–10, R5–10, P6–9, C5–9, PY1, PY5–6, PY10–11, TS2–9, TS11–12, A3, A6, A8–9 |
|
||||
| Low | 31 | SEC8, SEC11, R11–14, P10–12, C10–13, PY7–9, PY12, TS13–18, A10–11 |
|
||||
|
||||
**Recommended sprint order**:
|
||||
1. Security: SEC1, SEC2, PY3 (path traversal at 13 sites)
|
||||
2. Correctness: PY4 (GPU OOM), C3 (stale beatPhase), A3 (Spirit palette flash), A7/R1 (Spirit resize)
|
||||
3. Performance: P1 (Page Visibility), P3 (FFT when paused), P4 (Float32Array allocs)
|
||||
4. Architecture: A6 → A1 → A2 (pre-req for Visualizer 5)
|
||||
5. Reliability: C2 (fluid race), PY2 (blocking I/O), PY5 (urlretrieve)
|
||||
6. Type safety: TS1–TS5, TS9–TS10
|
||||
7. Housekeeping: C4, TS6–TS18, PY1, PY7–PY12, A10, A11
|
||||
|
||||
---
|
||||
|
||||
## 1. Security
|
||||
|
||||
### SEC1 — No Authentication on ML Inference Endpoints
|
||||
**Severity**: High
|
||||
**Files**: `backend/routers/transcribe.py`, `backend/routers/separate.py`, `backend/routers/lyric_transcribe.py`
|
||||
**Problem**: All GPU-backed endpoints (`/api/separate`, `/api/transcribe`, `/api/lyric-transcribe`) accept requests without any auth token check. `APP_ENV` defaults to `"development"`, which disables `ADMIN_TOKEN` enforcement. The `docker-compose.yml` never sets `APP_ENV=production`, so the disable is always active in standard deployments.
|
||||
**Fix**: Require `ADMIN_TOKEN` header on all ML endpoints unconditionally (not gated on `APP_ENV`), or add a separate lightweight Bearer check. Change `APP_ENV` default to `"production"` in compose file.
|
||||
|
||||
---
|
||||
|
||||
### SEC2 — Unsanitised `file.filename` Path Construction — Path Traversal
|
||||
**Severity**: High
|
||||
**Files**: `backend/routers/separate.py:267`, `backend/routers/transcribe.py:48`, `backend/routers/lrc.py:36,135`, `backend/routers/pipeline.py:59`, `backend/routers/lyric_transcribe.py:318,476` (6 endpoints)
|
||||
**Problem**: Raw `UploadFile.filename` from the HTTP multipart request is used directly in `Path(UPLOAD_DIR / file.filename)` without stripping directory components. A client sending `filename="../../config.py"` can overwrite arbitrary files reachable from the upload directory. The `lrc.py:36` path has no job_id prefix, making it the most exposed.
|
||||
**Fix**: Add a sanitiser and apply at all 6 sites:
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
def _safe_filename(raw: str | None) -> str:
|
||||
if not raw:
|
||||
return "upload"
|
||||
name = Path(raw).name # strips directory prefix
|
||||
if not name or name.startswith("."):
|
||||
return "upload"
|
||||
return name
|
||||
|
||||
upload_path = upload_dir / _safe_filename(file.filename)
|
||||
```
|
||||
*(Overlaps with PY3 — fix once, apply everywhere.)*
|
||||
|
||||
---
|
||||
|
||||
### SEC3 — File Size Not Enforced in FastAPI Layer
|
||||
**Severity**: Medium
|
||||
**Files**: `backend/routers/separate.py`, `backend/routers/transcribe.py`
|
||||
**Problem**: `MAX_UPLOAD_BYTES` is defined but only checked after full upload into memory. Large uploads exhaust server RAM before the check fires.
|
||||
**Fix**: Use `UploadFile.size` or stream with a byte counter; reject at first chunk exceeding limit.
|
||||
|
||||
---
|
||||
|
||||
### SEC4 — SSRF via yt-dlp URL Parameter
|
||||
**Severity**: Medium
|
||||
**Files**: `backend/routers/separate.py` (yt-dlp path)
|
||||
**Problem**: User-supplied URLs are forwarded to yt-dlp with only an `http://` prefix check. RFC-1918 addresses and Docker-internal services (port 9765, 8777) are reachable. No allow-list of domains.
|
||||
**Fix**: Resolve the URL's hostname and reject RFC-1918 and loopback ranges before invoking yt-dlp. Consider a domain allow-list.
|
||||
|
||||
---
|
||||
|
||||
### SEC5 — CORS Wildcard Active in Default Configuration
|
||||
**Severity**: Medium
|
||||
**Files**: `backend/main.py`, `docker-compose.yml`
|
||||
**Problem**: `APP_ENV` defaults to `"development"`, which sets `allow_origins=["*"]`. The compose file never sets `APP_ENV=production`.
|
||||
**Fix**: Change `APP_ENV` default to `"production"` in compose; require explicit opt-in for wildcard CORS.
|
||||
|
||||
---
|
||||
|
||||
### SEC6 — Cross-Job File Access in Separation Results
|
||||
**Severity**: Medium
|
||||
**Files**: `backend/routers/separate.py` (results path)
|
||||
**Problem**: Job result paths are constructed from job IDs with no ownership check. A user who knows another job's ID can fetch its stems.
|
||||
**Fix**: Associate jobs with a session token or user ID; validate ownership before serving results.
|
||||
|
||||
---
|
||||
|
||||
### SEC7 — `project_name` Parameter Not Sanitised
|
||||
**Severity**: Medium
|
||||
**Files**: `backend/routers/separate.py`, `backend/routers/projects.py`
|
||||
**Problem**: `project_name` from request body is used directly in directory name construction without `Path.name` stripping, allowing `../` traversal.
|
||||
**Fix**: Apply `Path(project_name).name` before any path join.
|
||||
|
||||
---
|
||||
|
||||
### SEC8 — Dependency on Vendored Unaudited `webgl-fluid.mjs`
|
||||
**Severity**: Low / Info
|
||||
**Files**: `frontend/src/components/visualizers/fluid/webgl-fluid.mjs`
|
||||
**Problem**: 1,813-line minified JS file with a custom modification (`_fluidSplat` injection at line 1808) that is not in the upstream npm package. No audit trail or comment block.
|
||||
**See also**: A5 for full architectural treatment.
|
||||
|
||||
---
|
||||
|
||||
### SEC9 — Missing `X-Content-Type-Options` / Security Headers
|
||||
**Severity**: Low
|
||||
**Files**: `backend/main.py`
|
||||
**Problem**: No `X-Content-Type-Options`, `X-Frame-Options`, or `Content-Security-Policy` headers set on API responses. Not blocking for an API-only backend, but relevant if the frontend is served from the same origin.
|
||||
|
||||
---
|
||||
|
||||
### SEC10 — No Rate Limiting on Upload Endpoints
|
||||
**Severity**: Low
|
||||
**Files**: All upload routers
|
||||
**Problem**: No rate limiting on any endpoint. An attacker can submit unlimited GPU jobs.
|
||||
|
||||
---
|
||||
|
||||
### SEC11 — Hardcoded Fallback Admin Token
|
||||
**Severity**: Low
|
||||
**Files**: `backend/auth.py` or equivalent
|
||||
**Problem**: A fallback `ADMIN_TOKEN` value exists in source code for development convenience. Should be removed entirely; if unset, the endpoint should reject all requests.
|
||||
|
||||
---
|
||||
|
||||
## 2. React / Frontend Architecture
|
||||
|
||||
### R1 — SpiritVisualizerCanvas Full GL Reinit on Every Resize
|
||||
**Severity**: High
|
||||
**Files**: `frontend/src/components/visualizers/SpiritVisualizerCanvas.tsx` (initAndStart useCallback)
|
||||
**Problem**: `width` and `height` are in the `useCallback` dependency array for `initAndStart`. Every resize triggers `destroySpiritSim` + `initSpiritSim`, recreating the Three.js renderer, `EffectComposer`, `UnrealBloomPass`, `AfterimagePass`, and two WebGL render targets (~50ms on integrated GPU). Muon, Fluid, and Tendrils handle resize imperatively inside the RAF loop.
|
||||
**Fix**: Remove `width`/`height` from `initAndStart` deps. Handle resize inside `renderSpiritFrame` by comparing `canvas.clientWidth`/`canvas.clientHeight` to `renderer.getSize()` each frame. *(Also tracked as A7.)*
|
||||
|
||||
---
|
||||
|
||||
### R2 — Audio Node Teardown on Fader Drag
|
||||
**Severity**: High
|
||||
**Files**: `frontend/src/components/LyricsLivePreview.tsx` (GainNode useEffect)
|
||||
**Problem**: The `useEffect` that creates `GainNode` objects has the stem volumes array in its dependency list. Dragging a fader reconnects/disconnects the AudioNode graph on every tick of the slider, causing clicks and brief audio dropouts.
|
||||
**Fix**: Create GainNodes once (on mount / stems change). Update `.gain.value` in a separate `useEffect` that has only the volume values in its deps — no node reconstruction needed.
|
||||
|
||||
---
|
||||
|
||||
### R3 — Spurious `onPlaybackEnded` from Stale Closure
|
||||
**Severity**: High
|
||||
**Files**: `frontend/src/components/LyricsLivePreview.tsx` (audio element event listener)
|
||||
**Problem**: The `'ended'` event listener on the `<audio>` element captures `onPlaybackEnded` at registration time. If the parent re-renders and passes a new reference, the stale callback fires on song end. In playlist mode this causes duplicate playlist-advance calls.
|
||||
**Fix**: Register the listener once; read the current callback via a stable `onPlaybackEndedRef.current = onPlaybackEnded` pattern.
|
||||
|
||||
---
|
||||
|
||||
### R4 — `applyGains` Teardown on Every Volume Tick
|
||||
**Severity**: High
|
||||
**Files**: `frontend/src/components/LyricsLivePreview.tsx`
|
||||
**Problem**: A `useEffect` with volume values in its dep array calls `applyGains`, which disconnects and reconnects `GainNode` routing. Smoothly fading a stem volume causes a full re-routing every render cycle.
|
||||
**Fix**: `applyGains` should only update `.gain.value` (or `.gain.linearRampToValueAtTime` for smooth transitions), never disconnect/reconnect nodes.
|
||||
|
||||
---
|
||||
|
||||
### R5 — Context Action Functions Not in `useCallback`
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/contexts/KaraokeQueueContext.tsx`, `frontend/src/contexts/QueueContext.tsx`
|
||||
**Problem**: Action functions (`addToQueue`, `removeFromQueue`, etc.) are defined inline in the context provider without `useCallback`. Every provider state change re-creates these functions, causing all consumers to re-render even when the data they care about hasn't changed.
|
||||
**Fix**: Wrap each action in `useCallback` with stable deps (only the dispatch or set functions, which are stable from `useReducer`/`useState`).
|
||||
|
||||
---
|
||||
|
||||
### R6 — `LyricsLivePreview` Re-Renders at Audio Tick Rate
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/components/LyricsLivePreview.tsx`
|
||||
**Problem**: `setCurrentTime` called from an `requestAnimationFrame` callback at 60fps causes React to schedule a re-render every frame. The component has 35+ `useState` calls; even with memoization, reconciling a 5,196-line component tree at 60fps is expensive.
|
||||
**Fix**: Move `currentTime` into a `useRef` (not `useState`). Components that need to display the current time should use a separate lightweight subscription (e.g., a context with a stable callback ref pattern, or a separate small component with its own RAF).
|
||||
|
||||
---
|
||||
|
||||
### R7 — Missing Cleanup for `ResizeObserver` in Visualizer Canvases
|
||||
**Severity**: Medium
|
||||
**Files**: All four `*VisualizerCanvas.tsx` files
|
||||
**Problem**: `ResizeObserver` instances created in `useEffect` may not be disconnected in the cleanup return if the component unmounts while a resize is in flight.
|
||||
**Fix**: Ensure `observer.disconnect()` is called in the `useEffect` cleanup function in all four files.
|
||||
|
||||
---
|
||||
|
||||
### R8 — `settingsRef` Synced via `useEffect` Instead of Unconditional Write (Spirit)
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/components/visualizers/SpiritVisualizerCanvas.tsx:99`
|
||||
**Problem**: `settingsRef.current = settings` inside a `useEffect` instead of at render time. The render function runs, the new settings are committed to DOM, but `settingsRef` still holds the previous frame's settings until React drains the effect queue.
|
||||
**Fix**: Assign `settingsRef.current = settings` unconditionally at the top of the component function body (before any `return`), not in `useEffect`.
|
||||
|
||||
---
|
||||
|
||||
### R9 — `useAudioFeatures` Stale Values in `patternInfo` / `beatInfo` When Paused
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/hooks/useAudioFeatures.ts`
|
||||
**Problem**: When `!isPlayingRef.current`, only some fields are zeroed. `beatPhase`, `isBeatDrop`, `energyDelta` retain their last non-zero values, continuing to drive visualizer animations (beat pulses, drop effects) during silence or pause.
|
||||
**Fix**: Zero all fields in the pause branch, including `beatInfo.beatPhase`, `patternInfo.isBeatDrop`, and `patternInfo.energyDelta`. *(Also tracked as C3.)*
|
||||
|
||||
---
|
||||
|
||||
### R10 — No `ErrorBoundary` Around Any Visualizer
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/components/visualizers/VisualizerBackdrop.tsx`
|
||||
**Problem**: WebGL context loss, shader compile failure, or a JavaScript exception in any visualizer canvas crashes the entire React tree. No `ErrorBoundary` / `componentDidCatch` exists anywhere in the frontend.
|
||||
**Fix**: Wrap each visualizer `<Suspense>` block in an `ErrorBoundary` that renders a silent fallback (black canvas or nothing) and reports to console. *(Also tracked as C4.)*
|
||||
|
||||
---
|
||||
|
||||
### R11 — Sidebar Polls Job List Every 8s Regardless of Active Jobs
|
||||
**Severity**: Low
|
||||
**Files**: `frontend/src/components/Sidebar.tsx`
|
||||
**Problem**: `refetchInterval: 8000` is unconditional. When no jobs are running, polling is wasted.
|
||||
**Fix**: Use the `refetchInterval` callback form: `(query) => query.state.data?.some(j => j.status === 'running') ? 3000 : false`.
|
||||
|
||||
---
|
||||
|
||||
### R12 — `LyricsLivePreview` `styleRebuildKey` JSON-Serializes on Every Render
|
||||
**Severity**: Low
|
||||
**Files**: `frontend/src/components/LyricsLivePreview.tsx:5109–5149`
|
||||
**Problem**: 24 style properties are JSON-serialized every render to compute a cache key. This fires even when unrelated state changes trigger a re-render.
|
||||
**Fix**: Use `useMemo` with the individual style state values as deps, not a JSON string. The memo runs only when a style value actually changes.
|
||||
|
||||
---
|
||||
|
||||
### R13 — `Two.js` Canvas RAF Loop Runs When Component Is Hidden
|
||||
**Severity**: Low
|
||||
**Files**: `frontend/src/components/LyricsLivePreview.tsx`
|
||||
**Problem**: The Two.js RAF loop continues rendering when the karaoke view is not visible. Add `document.addEventListener('visibilitychange', ...)` and pause the Two.js `play()`/`pause()` accordingly. *(Related to P6.)*
|
||||
|
||||
---
|
||||
|
||||
### R14 — `MuonFeatureExtractor` Runs Analysis Loop Even When Not Playing
|
||||
**Severity**: Low
|
||||
**Files**: `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
|
||||
**Problem**: The internal RAF loop calls `analyser.getFloatFrequencyData()` and 5 O(512) analysis passes every frame, allocating 3 `Float32Array` per frame, regardless of playback state. *(See P3, P4.)*
|
||||
|
||||
---
|
||||
|
||||
## 3. Performance
|
||||
|
||||
### P1 — No Page Visibility API Guard in Any Visualizer RAF Loop
|
||||
**Severity**: Critical
|
||||
**Files**: `MuonVisualizerCanvas.tsx`, `FluidVisualizerCanvas.tsx`, `TendrilsVisualizerCanvas.tsx`, `SpiritVisualizerCanvas.tsx`
|
||||
**Problem**: All four visualizers continue running their RAF loops at full rate when the tab is hidden (`document.hidden === true`). A hidden tab with an active visualizer burns full GPU at 60fps with zero user benefit. On battery devices this drains power; on integrated GPUs it can cause thermal throttling that affects other browser tabs.
|
||||
**Fix**: Add to all four visualizer canvas files:
|
||||
```ts
|
||||
useEffect(() => {
|
||||
const onVisibility = () => {
|
||||
if (document.hidden) cancelAnimationFrame(rafRef.current)
|
||||
else rafRef.current = requestAnimationFrame(renderLoop)
|
||||
}
|
||||
document.addEventListener('visibilitychange', onVisibility)
|
||||
return () => document.removeEventListener('visibilitychange', onVisibility)
|
||||
}, [])
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### P2 — No IntersectionObserver — Canvases Render Off-Screen
|
||||
**Severity**: High
|
||||
**Files**: All four `*VisualizerCanvas.tsx` files
|
||||
**Problem**: Visualizers render at full rate even when scrolled out of view or obscured by overlapping panels (sidebar, settings drawer). An `IntersectionObserver` on the canvas element can pause the RAF loop when intersection ratio drops to 0.
|
||||
**Fix**: Add `IntersectionObserver` in each visualizer's `useEffect`:
|
||||
```ts
|
||||
const io = new IntersectionObserver(([e]) => {
|
||||
isVisibleRef.current = e.isIntersecting
|
||||
}, { threshold: 0 })
|
||||
io.observe(canvas)
|
||||
return () => io.disconnect()
|
||||
```
|
||||
Check `isVisibleRef.current` at the top of each render loop.
|
||||
|
||||
---
|
||||
|
||||
### P3 — `MuonFeatureExtractor` FFT Analysis Runs When Paused
|
||||
**Severity**: High
|
||||
**Files**: `frontend/src/hooks/useAudioFeatures.ts` (or `MuonVisualizerCanvas.tsx`)
|
||||
**Problem**: `analyser.getFloatFrequencyData()` and 5 O(N=512) analysis passes run every frame regardless of `isPlayingRef.current`. When audio is paused the FFT data is all `-Infinity` — the analysis work is pure waste.
|
||||
**Fix**: Gate the entire analysis block:
|
||||
```ts
|
||||
if (!isPlayingRef.current) { /* emit zeroed features, skip analysis */ return }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### P4 — 3 `Float32Array` Allocations Per Frame in Audio Analysis
|
||||
**Severity**: High
|
||||
**Files**: `frontend/src/hooks/useAudioFeatures.ts`
|
||||
**Problem**: `getFloatFrequencyData()` allocates a new `Float32Array` per call and is called 3 times per frame (for different analysis windows). At 60fps this is 180 allocations/second, 3–9 KB each, generating 540 KB/s of GC pressure.
|
||||
**Fix**: Pre-allocate three `Float32Array` refs outside the RAF loop and reuse them:
|
||||
```ts
|
||||
const freqBufRef = useRef(new Float32Array(analyser.frequencyBinCount))
|
||||
// ...
|
||||
analyser.getFloatFrequencyData(freqBufRef.current)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### P5 — 1,600 `createRadialGradient` Calls Per Frame (Muon)
|
||||
**Severity**: High
|
||||
**Files**: `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
|
||||
**Problem**: The muon particle renderer calls `ctx.createRadialGradient()` for each of the ~1,600 visible particles every frame. `createRadialGradient` allocates a new `CanvasGradient` object each call. This generates ~100K allocations/second.
|
||||
**Fix**: Use `ctx.arc()` + `ctx.fill()` with a pre-computed radial gradient stamped via `drawImage` onto an off-screen canvas (sprite atlas pattern). One gradient object, reused for all particles.
|
||||
|
||||
---
|
||||
|
||||
### P6 — `AudioContext` Not Suspended When Nothing Is Playing
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/components/LyricsLivePreview.tsx`
|
||||
**Problem**: `AudioContext` stays in `running` state even when audio is paused or no track is loaded. A running `AudioContext` keeps the audio thread active at the system level.
|
||||
**Fix**: Call `audioContext.suspend()` when pausing; `audioContext.resume()` before playing.
|
||||
|
||||
---
|
||||
|
||||
### P7 — Two.js RAF Loop Continues When Tab Is Hidden
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/components/LyricsLivePreview.tsx`
|
||||
**Problem**: The Two.js canvas inside the karaoke view runs its built-in RAF loop via `two.play()`. It has no `visibilitychange` listener.
|
||||
**Fix**: `document.addEventListener('visibilitychange', () => document.hidden ? two.pause() : two.play())`.
|
||||
|
||||
---
|
||||
|
||||
### P8 — Sidebar Polls API Every 8s Even with No Active Jobs
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/components/Sidebar.tsx`
|
||||
**Problem**: Unconditional 8-second polling interval. *(Also R11 — identical fix.)*
|
||||
|
||||
---
|
||||
|
||||
### P9 — `deepMergeVizSettings` Called on Every Render in `LyricsLivePreview`
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/components/LyricsLivePreview.tsx`
|
||||
**Problem**: Settings normalization (`deepMergeVizSettings` / `normalizeVizSettings`) is called unconditionally each render rather than in a `useMemo`.
|
||||
**Fix**: Memoize: `const mergedSettings = useMemo(() => deepMergeVizSettings(VIZ_DEFAULTS, sekVisualizerSettings), [sekVisualizerSettings])`.
|
||||
|
||||
---
|
||||
|
||||
### P10 — Karaoke TV Polling Regardless of Play State
|
||||
**Severity**: Low
|
||||
**Files**: `frontend/src/pages/KaraokeTV.tsx`
|
||||
**Problem**: Queue/playback state polling runs unconditionally regardless of whether a song is actually playing. Gate on `isPlaying` state or switch to a WebSocket-based push model.
|
||||
|
||||
---
|
||||
|
||||
### P11 — No Memoization on Expensive `sortedTracks` Derivation
|
||||
**Severity**: Low
|
||||
**Files**: `frontend/src/pages/Tracks.tsx`
|
||||
**Problem**: Track sorting and filtering logic re-runs on every render including unrelated state changes.
|
||||
**Fix**: Wrap in `useMemo`.
|
||||
|
||||
---
|
||||
|
||||
### P12 — `useVisualizerPalette` Extracts Color from Image on Every Hook Call
|
||||
**Severity**: Low
|
||||
**Files**: `frontend/src/hooks/useVisualizerPalette.ts`
|
||||
**Problem**: Image color extraction (`extractSeedFromImage`) fires on every `songId`/`coverArtUrl` change, even if the same song is re-mounted (e.g., after route navigation). Cache results by `coverArtUrl`.
|
||||
|
||||
---
|
||||
|
||||
## 4. Code Quality & Edge Cases
|
||||
|
||||
### C1 — `quality` Prop Stale Closure in MuonVisualizerCanvas
|
||||
**Severity**: High
|
||||
**Files**: `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
|
||||
**Problem**: `quality` is captured at `useEffect` creation time. Even though it's in the dep array and the effect re-runs, within the RAF loop closure the stale `quality` value persists until the next effect cycle. Audio-reactive quality changes (e.g., `quality='low'` during drops) may miss frames.
|
||||
**Fix**: Use a `qualityRef`:
|
||||
```ts
|
||||
const qualityRef = useRef(quality)
|
||||
qualityRef.current = quality // unconditional update each render
|
||||
// RAF loop reads qualityRef.current, not captured quality
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### C2 — `initFluidSim` Unmount Race Condition
|
||||
**Severity**: High
|
||||
**Files**: `frontend/src/components/visualizers/fluid/fluidSim.ts:91–102`
|
||||
**Problem**: The async import resolves after `destroyFluidSim` is called. The token check at line 99 prevents `instances.set`, but `WebGL(canvas, config)` at line 97 still executes on the destroyed canvas. This can re-initialize WebGL on a canvas that the React lifecycle has already detached, causing a `WEBGL_lose_context` + re-attach race.
|
||||
**Fix**: Move the token check before the `WebGL()` call:
|
||||
```ts
|
||||
.then((WebGL) => {
|
||||
if (initTokens.get(canvas) !== initToken) return // already destroyed
|
||||
if (!document.contains(canvas)) return
|
||||
WebGL(canvas, config)
|
||||
if (initTokens.get(canvas) !== initToken) return // destroyed during init
|
||||
instances.set(canvas, { canvas, config })
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### C3 — Partial Zeroing on Pause Leaves Stale `beatPhase` / `isBeatDrop`
|
||||
**Severity**: High
|
||||
**Files**: `frontend/src/hooks/useAudioFeatures.ts`
|
||||
**Problem**: When `!isPlayingRef.current`, only some audio feature fields are zeroed. `beatPhase`, `isBeatDrop`, and `energyDelta` retain their last non-zero values. Visualizers that use these fields (e.g., beat-reactive scale pulses) continue animating after playback stops.
|
||||
**Fix**: In the pause-gate return path, emit a fully zeroed `AudioFeatures` object. Define a `ZERO_AUDIO_FEATURES` const and return it.
|
||||
|
||||
---
|
||||
|
||||
### C4 — Zero Error Boundaries in Entire Frontend
|
||||
**Severity**: High
|
||||
**Files**: Entire `frontend/src/` — no `ErrorBoundary` anywhere
|
||||
**Problem**: A WebGL context loss, shader compile failure, or uncaught exception in any visualizer or component crashes the entire React tree with a white screen. No `componentDidCatch` / `getDerivedStateFromError` exists anywhere.
|
||||
**Fix**: Add `ErrorBoundary` around at minimum: each visualizer (`<Suspense>` wrappers in `VisualizerBackdrop.tsx`), `LyricsLivePreview`, and the top-level App router outlet.
|
||||
|
||||
---
|
||||
|
||||
### C5 — Particle Pool Slot Overwrite Without Bounds Check
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
|
||||
**Problem**: When spawning new particles the pool index is incremented modulo pool size, but if spawn rate exceeds pool capacity in a single frame, multiple particles in the same tick may share the same slot. The older particle's state is silently overwritten.
|
||||
**Fix**: Clamp the number of new spawns per frame to at most `Math.floor(poolSize / minLifetimeFrames)`.
|
||||
|
||||
---
|
||||
|
||||
### C6 — Denominator Guard Missing in Spectral Centroid Calculation
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/hooks/useAudioFeatures.ts`
|
||||
**Problem**: `spectralCentroid = sumWeighted / sumTotal` — if all frequency bins are `-Infinity` (silence), `sumTotal = 0` and the result is `NaN`. `NaN` propagates into GLSL uniforms where it becomes `0.0` on most drivers but is undefined behaviour.
|
||||
**Fix**: `spectralCentroid = sumTotal > 0 ? sumWeighted / sumTotal : 0`.
|
||||
|
||||
---
|
||||
|
||||
### C7 — Audio `'error'` Event Not Handled
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/components/LyricsLivePreview.tsx`
|
||||
**Problem**: `audioRef.current` has `onEnded`, `onTimeUpdate`, `onCanPlay` listeners but no `'error'` listener. A failed stream or decode error silently stalls playback with no UI feedback.
|
||||
**Fix**: Add `audioRef.current.addEventListener('error', handleAudioError)` with a state update to show an error banner.
|
||||
|
||||
---
|
||||
|
||||
### C8 — `localStorage.setItem` Unchecked — Throws in Private Browsing / Full Storage
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/components/LyricsLivePreview.tsx` (StyleState persistence)
|
||||
**Problem**: `localStorage.setItem(...)` throws `QuotaExceededError` when storage is full or in private browsing mode on some browsers. The exception propagates uncaught through the React event handler, crashing the component.
|
||||
**Fix**: Wrap all `localStorage.setItem` calls in try/catch.
|
||||
|
||||
---
|
||||
|
||||
### C9 — Spirit/Tendrils GPGPU Textures Not Verified for `RGBA32F` Support
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/components/visualizers/TendrilsVisualizerCanvas.tsx`, `SpiritVisualizerCanvas.tsx`
|
||||
**Problem**: Both visualizers create `RGBA32F` render targets for GPGPU particle simulation. In WebGL2, `OES_texture_float` is core but `EXT_color_buffer_float` (required to render *into* a float texture) must be explicitly checked. If unsupported, `gl.checkFramebufferStatus` returns `FRAMEBUFFER_INCOMPLETE_ATTACHMENT` and all particle simulation produces black.
|
||||
**Fix**: Check extension availability at init time:
|
||||
```ts
|
||||
const ext = gl.getExtension('EXT_color_buffer_float')
|
||||
if (!ext) { /* fallback to HALF_FLOAT or disable visualizer */ }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### C10 — `createSynthForProgram` Returns `{}` for Unrecognised MIDI Program Numbers
|
||||
**Severity**: Low
|
||||
**Files**: `frontend/src/pages/ProjectDetail.tsx`
|
||||
**Problem**: The `default` branch returns `{}` which is not a valid Tone.js synth. Calling `.connect()` or `.triggerAttackRelease()` on it throws.
|
||||
**Fix**: Return a `new Tone.PolySynth(Tone.Synth)` as a safe default.
|
||||
|
||||
---
|
||||
|
||||
### C11 — `selectedLineIdxRef.current!` Non-Null Assertion in Drag Handler
|
||||
**Severity**: Low
|
||||
**Files**: `frontend/src/components/KaraokeEditor.tsx:389,404`
|
||||
**Problem**: Non-null assertion inside `onPointerMove`/`onPointerUp` handlers with no guard. If a drag begins while `selectedLineIdxRef.current` is `null` (race between drag start and line deselection), `linesRef.current[null]` is `undefined`, and word timestamps are set to `NaN`.
|
||||
**Fix**: Early return if `selectedLineIdxRef.current === null`.
|
||||
|
||||
---
|
||||
|
||||
### C12 — `svgRef.current!` Non-Null Assertion Across Async Boundary
|
||||
**Severity**: Low
|
||||
**Files**: `frontend/src/pages/ProjectDetail.tsx:1774`
|
||||
**Problem**: `svgRef.current` is guarded synchronously but then asserted non-null inside a `.then()` callback. The component may unmount before the dynamic import resolves.
|
||||
**Fix**: Capture `const container = svgRef.current` before the `import()` call; check `container.isConnected` inside `.then()`. *(Also tracked as TS10.)*
|
||||
|
||||
---
|
||||
|
||||
### C13 — Omnizart Sidecar HTTP Call Has No Mid-Transfer Cancellation
|
||||
**Severity**: Low
|
||||
**Files**: `backend/services/omnizart_service.py:108`
|
||||
**Problem**: Synchronous `httpx.Client` POST with 1800s timeout; no cancel check during transfer; full response body buffered in RAM before writing to disk.
|
||||
**Fix**: Stream with chunked reads and `jm.should_cancel(job_id)` poll between chunks. *(Also tracked as PY6.)*
|
||||
|
||||
---
|
||||
|
||||
## 5. Python Backend
|
||||
|
||||
### PY1 — Bare `except:` in Temp-Dir Cleanup — 24 Occurrences
|
||||
**Severity**: Medium
|
||||
**Files**: `backend/services/separator_service.py` (22 sites), `backend/routers/lrc.py:58`, `backend/services/ensemble_separator.py:213`
|
||||
**Problem**: Every `_run_*` separator function wraps `shutil.rmtree(temp_out, ignore_errors=True)` in a `try: ... except: pass`. `except:` catches `BaseException` including `KeyboardInterrupt`, `SystemExit`, and `GeneratorExit`. Since `ignore_errors=True` already suppresses all `OSError`, the outer try/except is redundant and dangerous — it masks real bugs (e.g., `TypeError` from a `None` path) and can swallow shutdown signals.
|
||||
**Fix**: Remove all 24 wrapping try/except blocks. `shutil.rmtree(path, ignore_errors=True)` is already safe.
|
||||
|
||||
---
|
||||
|
||||
### PY2 — Blocking File I/O in `async` FastAPI Handlers
|
||||
**Severity**: High
|
||||
**Files**: `backend/routers/separate.py:268`, `backend/routers/transcribe.py:51`, `backend/routers/lrc.py:40,90,131,136`, `backend/routers/pipeline.py:62`, `backend/routers/lyric_transcribe.py:319,477`, `backend/routers/abc_convert.py:30`
|
||||
**Problem**: All `async` upload handlers write the incoming file to disk using synchronous `shutil.copyfileobj()` without offloading to a thread pool. For files up to 500 MB (`MAX_UPLOAD_BYTES`), this stalls the uvicorn event loop for the entire duration of the write.
|
||||
**Fix**: Use async I/O:
|
||||
```python
|
||||
import anyio
|
||||
async with await anyio.open_file(upload_path, "wb") as f:
|
||||
while chunk := await file.read(65536):
|
||||
await f.write(chunk)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### PY3 — Unsanitised `file.filename` — Path Traversal (13 Sites)
|
||||
**Severity**: High
|
||||
**Files**: Same as SEC2 + `backend/routers/abc_convert.py`
|
||||
**Problem**: Same root cause as SEC2. 13 total call sites across 7 files.
|
||||
**Fix**: Same `_safe_filename()` helper. *(Unified fix with SEC2.)*
|
||||
|
||||
---
|
||||
|
||||
### PY4 — `separate_existing` Uses General Executor for GPU Work
|
||||
**Severity**: High
|
||||
**Files**: `backend/routers/separate.py:419`
|
||||
**Problem**: `/api/separate/existing` calls `jm.submit(...)` instead of `jm.submit_gpu(...)`. The GPU executor serializes jobs to prevent VRAM contention. The general executor has 4 workers. Multiple concurrent calls to `separate_existing` trigger simultaneous GPU-intensive jobs, causing CUDA OOM.
|
||||
**Fix**: One-line change:
|
||||
```python
|
||||
# Before
|
||||
jm.submit(run_separation, job_id, ...)
|
||||
# After
|
||||
jm.submit_gpu(run_separation, job_id, ...)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### PY5 — `urlretrieve` for Model Downloads — No Timeout, Non-Atomic, Deprecated
|
||||
**Severity**: Medium
|
||||
**Files**: `backend/services/separator_service.py:123,126`, `backend/services/scnet_separator.py:269,280`, `backend/services/mdx23c_separator.py`
|
||||
**Problem**: `urllib.request.urlretrieve()` has been soft-deprecated since Python 3.4. No timeout means a stalled connection hangs the separator thread indefinitely. On interruption, a partial (corrupt) model file remains on disk; subsequent runs find the file and skip downloading, then load the corrupt checkpoint silently.
|
||||
**Fix**: Atomic download using httpx (already a dependency):
|
||||
```python
|
||||
import httpx, tempfile
|
||||
|
||||
def _download_file(url: str, dest: Path, *, timeout: int = 600) -> None:
|
||||
with tempfile.NamedTemporaryFile(dir=dest.parent, delete=False, suffix=".tmp") as tmp:
|
||||
tmp_path = Path(tmp.name)
|
||||
try:
|
||||
with httpx.stream("GET", url, follow_redirects=True, timeout=timeout) as r:
|
||||
r.raise_for_status()
|
||||
for chunk in r.iter_bytes(65536):
|
||||
tmp.write(chunk)
|
||||
tmp_path.rename(dest) # atomic: visible only on success
|
||||
except Exception:
|
||||
tmp_path.unlink(missing_ok=True)
|
||||
raise
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### PY6 — Omnizart Sidecar: No Cancel Check During 30-Min HTTP Call
|
||||
**Severity**: Medium
|
||||
**Files**: `backend/services/omnizart_service.py:108`
|
||||
**Problem**: Synchronous `httpx.Client` POST with 1800s timeout; full response buffered into RAM; no cancellation check during transfer. Thread-pool thread stuck for up to 30 min with no way to abort. *(Also tracked as C13.)*
|
||||
**Fix**: Stream with cancel checks between chunks (see C13 fix).
|
||||
|
||||
---
|
||||
|
||||
### PY7 — Demucs Cancel Check Blocked by Synchronous `readline()`
|
||||
**Severity**: Low
|
||||
**Files**: `backend/services/separator_service.py:1539–1553`
|
||||
**Problem**: Cancellation check only runs between lines of Demucs stdout output. During quiet processing periods, `readline()` blocks and cancel requests are delayed for many seconds. The `time.sleep(0.1)` line is dead code (unreachable after `readline()` blocks).
|
||||
**Fix**: Use `select` with a 1-second timeout for non-blocking reads; poll `should_cancel()` each second.
|
||||
|
||||
---
|
||||
|
||||
### PY8 — `datetime.utcnow()` Deprecated Since Python 3.12
|
||||
**Severity**: Low
|
||||
**Files**: `backend/db.py:50,51,60,82,83,173,196,219`, `backend/repositories/jobs.py:80,96`, `backend/job_manager.py:317`
|
||||
**Problem**: `datetime.utcnow()` produces naive datetimes and is deprecated in Python 3.12; removed in 3.14.
|
||||
**Fix**: Replace all with `datetime.now(timezone.utc)`.
|
||||
|
||||
---
|
||||
|
||||
### PY9 — `print()` in Production Service Code — 30+ Occurrences
|
||||
**Severity**: Low
|
||||
**Files**: `backend/services/sepacap_separator.py`, `mdx23c_separator.py`, `scnet_separator.py`, `sepreformer_separator.py`, `apollo_separator.py`, `ensemble_separator.py`, `medleyvox_separator.py`, `aspiration_separator.py`, `bandit_separator.py`, `mel_band_specialized_separator.py`
|
||||
**Problem**: Progress and error messages use `print()` instead of the structured logging system, making them invisible to log aggregators and unsuppressible.
|
||||
**Fix**: Add `log = logging.getLogger(__name__)` and replace `print(...)` with `log.info(...)` / `log.debug(...)`.
|
||||
|
||||
---
|
||||
|
||||
### PY10 — `separate_existing` Source File Discovery Uses Wrong File
|
||||
**Severity**: Medium
|
||||
**Files**: `backend/routers/separate.py:374–399`
|
||||
**Problem**: When the project directory doesn't contain the original source file, the code scans all upload directories by mtime and returns the most recently uploaded file from **any** job. This silently re-separates the wrong audio file whenever any other upload occurred more recently. Performance also degrades linearly as uploads accumulate.
|
||||
**Fix**: Look up the original source path from the separation job record rather than scanning the filesystem.
|
||||
|
||||
---
|
||||
|
||||
### PY11 — Global `torch.load` Monkey-Patch Disables Pickle Safety Process-Wide
|
||||
**Severity**: Medium
|
||||
**Files**: `backend/services/lyric_transcriber_service.py:42–55`
|
||||
**Problem**: A global monkey-patch unconditionally overrides `torch.load` for the entire process, forcing `weights_only=False`. `weights_only=False` enables arbitrary Python object deserialization. PyTorch's CVE-2025-32434 explicitly identifies this as the RCE vector. The patch is active by default (`STEMENGINE_PATCH_TORCH_LOAD` env var defaults to `"1"`), meaning every model loaded anywhere in the process is unsafe.
|
||||
**Fix**: Remove the global patch. Use `torch.serialization.safe_globals()` with an explicit allowlist for the specific classes that pyannote/WhisperX require. At minimum, change the env-variable default to `"0"` (opt-in).
|
||||
|
||||
---
|
||||
|
||||
### PY12 — `JobResponse.job_id` Mismatches DB `id` Field
|
||||
**Severity**: Low
|
||||
**Files**: `backend/routers/jobs.py:18–31`, `backend/repositories/jobs.py:139`
|
||||
**Problem**: `JobResponse` declares `job_id: str` but `to_dict()` returns `"id"`. The model is never enforced (`response_model` not applied), so this is misleading dead documentation. Applying `response_model=JobResponse` would immediately break.
|
||||
**Fix**: Align field name to `id: str` or apply `Field(alias="job_id")` with `model_config = ConfigDict(populate_by_name=True)`.
|
||||
|
||||
---
|
||||
|
||||
## 6. TypeScript Type Safety
|
||||
|
||||
### TS1 — MIDI Object Graph Fully Untyped
|
||||
**Severity**: Critical
|
||||
**Files**: `frontend/src/pages/ProjectDetail.tsx:120–126, 231, 260–275, 290, 345–353`
|
||||
**Problem**: `midiList: Array<{midi: any; stemIdx: number}>`. All MIDI tracks, notes, and Tone.js refs (`ToneRef`, `synthRefs`, `partRefs`, `volRefs`) are `useRef<any>()` or `Record<number, any>`. `createSynthForProgram` accepts and returns `any`. Property typos produce `undefined` silently; `note.velocity * 0.55` renders `NaN` if `velocity` is absent.
|
||||
**Fix**: Define `MidiNote`, `MidiTrack`, `MidiFile` interfaces. Import Tone.js types: `PolySynth`, `Part`, `Volume`, `Sampler`.
|
||||
|
||||
---
|
||||
|
||||
### TS2 — `import.meta` Cast to `any` for Vite Env Variables
|
||||
**Severity**: High
|
||||
**Files**: `frontend/src/api/client.ts:51`, `frontend/src/App.tsx:324`
|
||||
**Problem**: `(import.meta as any).env?.VITE_STEMENGINE_API_BASE` — custom env vars missing from `vite-env.d.ts`.
|
||||
**Fix**: Extend `vite-env.d.ts` with the `ImportMetaEnv` interface declaring both variables.
|
||||
|
||||
---
|
||||
|
||||
### TS3 — `AppSettings` Preset Fields Typed as `unknown[]`
|
||||
**Severity**: High
|
||||
**Files**: `frontend/src/api/client.ts:977–983`
|
||||
**Problem**: `auto_steps: unknown[]`, `auto_presets: unknown[]` despite `AutoStep` and `AutoPreset` types being defined in `utils/autoPipeline.ts`. Causes redundant intersection casts in `useConvertStatus`.
|
||||
**Fix**: Import and use `AutoStep[]` / `AutoPreset[]`.
|
||||
|
||||
---
|
||||
|
||||
### TS4 — `createSynthForProgram` Accepts/Returns `any`
|
||||
**Severity**: High
|
||||
**Files**: `frontend/src/pages/ProjectDetail.tsx:231,248`
|
||||
**Problem**: Tone.js constructor signature changes produce silent runtime failures.
|
||||
**Fix**: Type as `(Tone: typeof import('tone'), program: number): PolySynth | Sampler`.
|
||||
|
||||
---
|
||||
|
||||
### TS5 — `applyClipPath` / `applyFilter` Accept `any`
|
||||
**Severity**: High
|
||||
**Files**: `frontend/src/utils/svgUtils.ts:153,159`
|
||||
**Problem**: Both functions accept `any`. Actual shape is always `{ _renderer?: { elem?: SVGElement } }`. Same internal Two.js cast done ~12 more times inline in `LyricsLivePreview.tsx`.
|
||||
**Fix**: Define and export `TwoRendererElement` interface; use it throughout.
|
||||
|
||||
---
|
||||
|
||||
### TS6 — `fluidSim.ts` Dynamic Import Typed as `any`
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/components/visualizers/fluid/fluidSim.ts:1–15`
|
||||
**Problem**: `let WebGLFluid: any` — `WebGL(canvas, config)` call cannot be type-checked.
|
||||
**Fix**: Define `type WebGLFluidFn = (canvas: HTMLCanvasElement, options?: Partial<FluidConfig>) => void` and use throughout.
|
||||
|
||||
---
|
||||
|
||||
### TS7 — Two.js Internal Renderer Properties Cast as `any` (~30 Sites)
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/components/LyricsLivePreview.tsx` (30+ occurrences)
|
||||
**Problem**: Two.js undocumented properties (`.renderer.defs`, `._renderer.elem`, `.scale`, `.fill`, `.noStroke()`) all accessed via `as any`.
|
||||
**Fix**: Add `src/types/two-internals.d.ts` with a module augmentation declaring the undocumented surface once.
|
||||
|
||||
---
|
||||
|
||||
### TS8 — Vendor-Prefixed CSS Properties Cast as `any`
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/components/LyricsLivePreview.tsx:4818–4874, 4927`
|
||||
**Problem**: `(el.style as any).textRendering`, `(el.style as any).webkitFontSmoothing`, etc.
|
||||
**Fix**: Use `el.style.setProperty('text-rendering', 'geometricPrecision')` for DOM manipulation (no cast). Define `ExtendedCSSProperties extends React.CSSProperties` for inline styles.
|
||||
|
||||
---
|
||||
|
||||
### TS9 — Non-Null Assertions on `canvas!` and `selectedLineIdxRef.current!`
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/components/KaraokeEditor.tsx:277,302,342,353,378,389,395,404`
|
||||
**Problem**: Six `canvas!` assertions inside nested closures (TypeScript widens back to `null` inside nested functions). `selectedLineIdxRef.current!` in drag handlers with no null guard — produces `NaN` word timestamps if race occurs. *(Also tracked as C11.)*
|
||||
**Fix**: Capture `const canvas = canvasRef.current` before inner closures. Add `if (lineIdx === null) return` guard in drag handlers.
|
||||
|
||||
---
|
||||
|
||||
### TS10 — Non-Null Assertion on `svgRef.current!` Across Async Boundary
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/pages/ProjectDetail.tsx:1774`
|
||||
**Problem**: Guard is synchronous but `renderAbc(svgRef.current!, ...)` is inside a `.then()` callback — component may have unmounted. *(Also tracked as C12.)*
|
||||
**Fix**: Capture ref before `import()`; check `.isConnected` inside `.then()`.
|
||||
|
||||
---
|
||||
|
||||
### TS11 — Untyped `useQuery` with Internal `as Job[]` Cast
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/components/Sidebar.tsx:629–638`
|
||||
**Problem**: `useQuery` without type parameter; internal cast `as Job[] | undefined` despite `listJobs` returning `Promise<Job[]>`.
|
||||
**Fix**: `useQuery<Job[]>({ ... })` — cast removed automatically.
|
||||
|
||||
---
|
||||
|
||||
### TS12 — Page-Level `useQuery` Calls Missing Type Parameters
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/pages/KaraokeTV.tsx:192–214,291–305`, `frontend/src/pages/Tracks.tsx:63–103`
|
||||
**Problem**: `useQuery` without generic params; `getProjectDetail(projectId!)` non-null assertion inside `queryFn`.
|
||||
**Fix**: Add `useQuery<ProjectDetail>(...)` and move assertion out of queryFn.
|
||||
|
||||
---
|
||||
|
||||
### TS13 — `CanvasSceneState` Non-Optional Fields Initialised as `null as any`
|
||||
**Severity**: Low
|
||||
**Files**: `frontend/src/components/LyricsLivePreview.tsx:5053–5054`
|
||||
**Fix**: Make `linesGroup` and `bgRect` nullable: `InstanceType<typeof Two.Group> | null`.
|
||||
|
||||
---
|
||||
|
||||
### TS14 — `VisualizerBackdrop` Mode Switch Not Exhaustively Checked
|
||||
**Severity**: Low
|
||||
**Files**: `frontend/src/components/visualizers/VisualizerBackdrop.tsx:103–177`
|
||||
**Problem**: If-chain is not exhaustive — new modes added to the union silently render `null`.
|
||||
**Fix**: Convert to `switch` with `assertNever` for the default case. *(Superseded by architecture fix A1.)*
|
||||
|
||||
---
|
||||
|
||||
### TS15–TS18 — Missing Return Type Annotations, Wildcard `any` Module Declarations
|
||||
**Severity**: Low
|
||||
- **TS15**: `useProjectsIndex` missing return type — add `UseProjectsIndexResult` interface
|
||||
- **TS16**: `useConvertStatus` missing return type and unexported `ConvertState` — export both
|
||||
- **TS17**: `types/mjs-modules.d.ts` — wildcard `.mjs` typed as `any`; change to `unknown`
|
||||
- **TS18**: `types/webgl-fluid.d.ts` — `options?: any`; import `FluidConfig` and use `Partial<FluidConfig>`
|
||||
|
||||
---
|
||||
|
||||
## 7. Architecture & Tech Debt
|
||||
|
||||
### A1 — VisualizerBackdrop Dispatch: 4-Site Edit Per New Visualizer
|
||||
**Severity**: High
|
||||
**Files**: `frontend/src/components/visualizers/VisualizerBackdrop.tsx:103–177`
|
||||
**Problem**: Four sequential `if (mode === 'x')` blocks each manually duplicate the wrapper `<div>` + `<Suspense>`. There is an undocumented divergence: Tendrils and Spirit have `style={{ background }}` on their wrapper div; Muon and Fluid do not. Adding Visualizer 5 requires edits in 4+ files with no authoritative registry.
|
||||
**Fix**: Create `visualizerRegistry.ts` — a `Record<VisualizerMode, VisualizerDescriptor>` with the lazy component, `needsBackgroundStyle` flag, and wrapper class. `VisualizerBackdrop` becomes a single generic render path.
|
||||
|
||||
---
|
||||
|
||||
### A2 — `VisualizerSettings`: Three Independent Type Definitions
|
||||
**Severity**: High
|
||||
**Files**: `VisualizerBackdrop.tsx:17–92`, `LyricsLivePreview.tsx:55–120`, `LyricsLivePreview.tsx:312–372`
|
||||
**Problem**: Same per-visualizer settings shape defined three times with different optionality. A new parameter costs 6+ coordinated edits across 2 files. The inline prop type (`sekVisualizerSettings` at line 312) will permanently lag the other two.
|
||||
**Fix**: Create `src/types/visualizerSettings.ts` with one `VisualizerSettings` interface, `VISUALIZER_DEFAULTS`, and `mergeVisualizerSettings()`. Delete the local `VizSettings` interface from `LyricsLivePreview`.
|
||||
|
||||
---
|
||||
|
||||
### A3 — Spirit Palette Init-Time Coupling Causes Visible Flash
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/components/visualizers/SpiritVisualizerCanvas.tsx:130–131`
|
||||
**Problem**: `initSpiritSim` reads `paletteRef.current.speakerPalette` at init time for shader material colors. When `useVisualizerPalette` resolves asynchronously (50–200ms), the palette change triggers `destroySpiritSim` + `initSpiritSim`, causing a hard visual blank frame. Muon/Fluid/Tendrils read `paletteRef.current` inside the RAF loop and avoid this entirely.
|
||||
**Fix**: Move `color1`/`color2` out of `initSpiritSim` and into `renderSpiritFrame` as per-frame `THREE.Color` uniform updates.
|
||||
|
||||
---
|
||||
|
||||
### A4 — `LyricsLivePreview` God Component (5,196 Lines, 41 `useEffect` Calls)
|
||||
**Severity**: Critical
|
||||
**Files**: `frontend/src/components/LyricsLivePreview.tsx`
|
||||
**Problem**: Simultaneously owns AudioContext lifecycle, GainNode management, transport controls, LRC parsing, Two.js canvas scene (~600 lines), video backdrop, visualizer settings normalization, `StyleState` (30+ fields), speaker diarization, `KaraokeWord` renderer, `StylePicker` (400+ lines), and `LrcEditor`. `setCurrentTime` from RAF causes React reconciliation at 60fps. `styleRebuildKey` JSON-serializes 24 properties every render.
|
||||
**Staged fix**:
|
||||
1. Extract `useAudioEngine` (AudioContext + GainNodes + stem mixing)
|
||||
2. Extract `useLyricsCanvas` (Two.js scene)
|
||||
3. Move `StylePicker`, `KaraokeWord`, `SpeakerDot`, `LrcEditor` to `components/karaoke/`
|
||||
|
||||
---
|
||||
|
||||
### A5 — Vendored `webgl-fluid.mjs` Diverges from npm Package
|
||||
**Severity**: High
|
||||
**Files**: `frontend/src/components/visualizers/fluid/webgl-fluid.mjs`, `package.json`
|
||||
**Problem**: The vendored file has a `_fluidSplat` property injection at line 1808 not present in the upstream npm package. `package.json` also lists `webgl-fluid@^0.3.9`. If npm resolution wins (e.g., after `npm ci`), `_fluidSplat` is `undefined` and all beat-reactive splat injection silently no-ops. Additionally, `updateFluidConfig` uses `Object.assign` on the stored config reference — if the library copies config at init time, all per-frame parameter changes are silently ignored.
|
||||
**Fix**:
|
||||
1. Remove `webgl-fluid` from `package.json` (the vendored file is authoritative)
|
||||
2. Add startup assertion verifying `_fluidSplat` is set after `WebGL()` resolves
|
||||
3. Add comment block at line 0 of `webgl-fluid.mjs` documenting the modification
|
||||
|
||||
---
|
||||
|
||||
### A6 — Utility Function Duplication Across All Visualizer Files
|
||||
**Severity**: Medium
|
||||
**Files**: `MuonVisualizerCanvas.tsx`, `FluidVisualizerCanvas.tsx`, `TendrilsVisualizerCanvas.tsx`, `SpiritVisualizerCanvas.tsx`
|
||||
**Problem**: `clamp01`, `clamp`, and hex-to-RGB converters independently defined in every file — with different return shapes (`{r,g,b}` unit floats vs `{r,g,b}` 0–255 vs `[r,g,b]` tuple). TypeScript cannot detect passing the wrong variant.
|
||||
**Fix**: Create `components/visualizers/vizUtils.ts` exporting `clamp`, `clamp01`, `hexToRgbUnit`, `hexToRgb255`, `blendColor`, `hueRotate`. All four canvas files import from it.
|
||||
|
||||
---
|
||||
|
||||
### A7 — Spirit Lifecycle Divergence: Full Reinit on Resize
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/components/visualizers/SpiritVisualizerCanvas.tsx:104–227`
|
||||
**Problem**: `width`/`height` in `initAndStart` `useCallback` deps causes full Three.js renderer recreation on every resize, panel toggle, or sidebar open (~50ms blank frame on integrated GPU). Muon/Fluid/Tendrils handle resize imperatively inside the RAF loop. *(Also tracked as R1.)*
|
||||
**Fix**: Remove `width`/`height` from deps; handle resize inside `renderSpiritFrame`. Assign `settingsRef.current = settings` synchronously at render time (not in a separate `useEffect`).
|
||||
|
||||
---
|
||||
|
||||
### A8 — `useConvertStatus` — Uncancellable Async Polling Loops
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/hooks/useConvertStatus.ts`
|
||||
**Problem**: `pollJobUntilTerminal` uses an infinite `for(;;)` loop with no `AbortSignal` and no cleanup. Component unmount leaves polling running indefinitely. `convertSong` callback contains 100 lines of service-layer orchestration (localStorage, API calls, job submission) in a React hook.
|
||||
**Fix**: Extract orchestration to `services/conversionOrchestrator.ts`. Thread an `AbortController` signal; store controller in `useRef`; call `.abort()` on cleanup.
|
||||
|
||||
---
|
||||
|
||||
### A9 — `updateFluidConfig` Mutation Correctness Unverified
|
||||
**Severity**: Medium
|
||||
**Files**: `frontend/src/components/visualizers/fluid/fluidSim.ts:105–109`
|
||||
**Problem**: `Object.assign(inst.config, patch)` mutates the stored config reference. If the vendored library copied config at init time, all per-frame audio-reactive parameter changes are silently dropped and the fluid sim runs at fixed parameters regardless of audio input. Cannot be verified without reading 1,813 lines of minified code. *(Related to A5.)*
|
||||
**Fix**: Add a smoke test: call `updateFluidConfig(canvas, { PAUSED: true })` and verify the sim pauses. Document result as a comment.
|
||||
|
||||
---
|
||||
|
||||
### A10 — Three Redundant Visualizer-Mode Membership Checks
|
||||
**Severity**: Low
|
||||
**Files**: `frontend/src/components/LyricsLivePreview.tsx:47,1861,5178`
|
||||
**Problem**: `'muon' | 'fluid' | 'tendrils' | 'spirit'` enumerated three times independently. Adding a fifth visualizer requires finding and updating all three; missing one causes subtle behavioral bugs (wrong audio analysis path, canvas fade applied over WebGL).
|
||||
**Fix**: Export `VISUALIZER_MODES = new Set([...])` from visualizer registry (prerequisite: A1); replace all three checks with `VISUALIZER_MODES.has(visualStyle)`.
|
||||
|
||||
---
|
||||
|
||||
### A11 — `useConvertStatus` Single-Responsibility Violation
|
||||
**Severity**: Low
|
||||
**Files**: `frontend/src/hooks/useConvertStatus.ts`
|
||||
**Problem**: 267 lines; mixes React state management with service orchestration. All other hooks are appropriately scoped. *(See A8 for fix.)*
|
||||
|
||||
---
|
||||
|
||||
### A12 — Cost of Adding a 5th Visualizer Under Current Architecture
|
||||
**Severity**: Advisory
|
||||
**Problem**: Under current architecture, adding a fifth visualizer requires 8+ coordinated edits across 2 files (`VisualizerBackdrop.tsx`, `LyricsLivePreview.tsx`) plus the new canvas file. After implementing A1 + A2 + A6, adding a new visualizer requires only the new canvas file plus one registry entry.
|
||||
**Recommendation**: Treat A6 → A1 → A2 as a prerequisite sprint before implementing Visualizer 5.
|
||||
|
||||
---
|
||||
|
||||
## Appendix: Finding Index
|
||||
|
||||
| ID | Domain | Severity | Title |
|
||||
|----|--------|----------|-------|
|
||||
| SEC1 | Security | High | No auth on ML endpoints |
|
||||
| SEC2 | Security | High | Path traversal via `file.filename` |
|
||||
| SEC3 | Security | Medium | File size not enforced |
|
||||
| SEC4 | Security | Medium | SSRF via yt-dlp |
|
||||
| SEC5 | Security | Medium | CORS wildcard default |
|
||||
| SEC6 | Security | Medium | Cross-job file access |
|
||||
| SEC7 | Security | Medium | `project_name` path traversal |
|
||||
| SEC8 | Security | Low | Vendored unaudited JS |
|
||||
| SEC9 | Security | Low | Missing security headers |
|
||||
| SEC10 | Security | Low | No rate limiting |
|
||||
| SEC11 | Security | Low | Hardcoded fallback token |
|
||||
| R1 | React | High | Spirit full reinit on resize |
|
||||
| R2 | React | High | Audio node teardown on fader drag |
|
||||
| R3 | React | High | Spurious onPlaybackEnded |
|
||||
| R4 | React | High | `applyGains` teardown on volume tick |
|
||||
| R5 | React | Medium | Context actions not memoized |
|
||||
| R6 | React | Medium | Re-renders at audio tick rate |
|
||||
| R7 | React | Medium | ResizeObserver cleanup missing |
|
||||
| R8 | React | Medium | `settingsRef` synced via effect (Spirit) |
|
||||
| R9 | React | Medium | Stale beatPhase/isBeatDrop when paused |
|
||||
| R10 | React | Medium | No ErrorBoundary on visualizers |
|
||||
| R11 | React | Low | Sidebar unconditional polling |
|
||||
| R12 | React | Low | `styleRebuildKey` JSON-serialize every render |
|
||||
| R13 | React | Low | Two.js RAF runs hidden |
|
||||
| R14 | React | Low | MuonFeatureExtractor runs when not playing |
|
||||
| P1 | Performance | Critical | No Page Visibility guard in any visualizer |
|
||||
| P2 | Performance | High | No IntersectionObserver |
|
||||
| P3 | Performance | High | FFT analysis runs when paused |
|
||||
| P4 | Performance | High | 3 Float32Array allocs/frame |
|
||||
| P5 | Performance | High | 1,600 createRadialGradient/frame |
|
||||
| P6 | Performance | Medium | AudioContext not suspended |
|
||||
| P7 | Performance | Medium | Two.js RAF runs hidden |
|
||||
| P8 | Performance | Medium | Sidebar unconditional polling |
|
||||
| P9 | Performance | Medium | `deepMergeVizSettings` not memoized |
|
||||
| P10 | Performance | Low | KaraokeTV unconditional polling |
|
||||
| P11 | Performance | Low | `sortedTracks` not memoized |
|
||||
| P12 | Performance | Low | `extractSeedFromImage` not cached |
|
||||
| C1 | Code Quality | High | `quality` stale closure |
|
||||
| C2 | Code Quality | High | initFluidSim unmount race |
|
||||
| C3 | Code Quality | High | Stale beatPhase/isBeatDrop on pause |
|
||||
| C4 | Code Quality | High | Zero ErrorBoundaries |
|
||||
| C5 | Code Quality | Medium | Particle pool overwrite |
|
||||
| C6 | Code Quality | Medium | Spectral centroid divide-by-zero |
|
||||
| C7 | Code Quality | Medium | Audio error event not handled |
|
||||
| C8 | Code Quality | Medium | localStorage.setItem unchecked |
|
||||
| C9 | Code Quality | Medium | RGBA32F extension not verified |
|
||||
| C10 | Code Quality | Low | `createSynthForProgram` returns `{}` |
|
||||
| C11 | Code Quality | Low | Non-null assertion in drag handler |
|
||||
| C12 | Code Quality | Low | Non-null assertion across async boundary |
|
||||
| C13 | Code Quality | Low | Omnizart no cancel check |
|
||||
| PY1 | Python | Medium | Bare `except:` in cleanup (24 sites) |
|
||||
| PY2 | Python | High | Blocking I/O in async handlers |
|
||||
| PY3 | Python | High | `file.filename` path traversal (13 sites) |
|
||||
| PY4 | Python | High | `separate_existing` uses wrong executor |
|
||||
| PY5 | Python | Medium | `urlretrieve` — no timeout, non-atomic |
|
||||
| PY6 | Python | Medium | Omnizart no cancel during 30-min call |
|
||||
| PY7 | Python | Low | Demucs cancel blocked by `readline()` |
|
||||
| PY8 | Python | Low | `datetime.utcnow()` deprecated |
|
||||
| PY9 | Python | Low | `print()` in production services (30+) |
|
||||
| PY10 | Python | Medium | `separate_existing` uses wrong file |
|
||||
| PY11 | Python | Medium | Global torch.load patch disables safety |
|
||||
| PY12 | Python | Low | `JobResponse.job_id` field mismatch |
|
||||
| TS1 | TypeScript | Critical | MIDI graph fully untyped |
|
||||
| TS2 | TypeScript | High | `import.meta as any` for env vars |
|
||||
| TS3 | TypeScript | High | AppSettings preset fields `unknown[]` |
|
||||
| TS4 | TypeScript | High | `createSynthForProgram` returns `any` |
|
||||
| TS5 | TypeScript | High | `applyClipPath`/`applyFilter` accept `any` |
|
||||
| TS6 | TypeScript | Medium | fluidSim dynamic import typed as `any` |
|
||||
| TS7 | TypeScript | Medium | Two.js internals `as any` (30 sites) |
|
||||
| TS8 | TypeScript | Medium | Vendor CSS cast as `any` |
|
||||
| TS9 | TypeScript | Medium | Non-null assertions in closures |
|
||||
| TS10 | TypeScript | Medium | Non-null assertion across async boundary |
|
||||
| TS11 | TypeScript | Medium | Untyped useQuery with internal cast |
|
||||
| TS12 | TypeScript | Medium | Page-level useQuery missing type params |
|
||||
| TS13 | TypeScript | Low | CanvasSceneState `null as any` init |
|
||||
| TS14 | TypeScript | Low | VisualizerBackdrop not exhaustive |
|
||||
| TS15 | TypeScript | Low | useProjectsIndex missing return type |
|
||||
| TS16 | TypeScript | Low | useConvertStatus missing return type |
|
||||
| TS17 | TypeScript | Low | Wildcard .mjs typed as `any` |
|
||||
| TS18 | TypeScript | Low | webgl-fluid.d.ts `options?: any` |
|
||||
| A1 | Architecture | High | VisualizerBackdrop 4-site edit per mode |
|
||||
| A2 | Architecture | High | VisualizerSettings 3 independent definitions |
|
||||
| A3 | Architecture | Medium | Spirit palette init-time flash |
|
||||
| A4 | Architecture | Critical | LyricsLivePreview god component |
|
||||
| A5 | Architecture | High | Vendored webgl-fluid diverges from npm |
|
||||
| A6 | Architecture | Medium | Utility duplication across visualizers |
|
||||
| A7 | Architecture | Medium | Spirit resize lifecycle divergence |
|
||||
| A8 | Architecture | Medium | useConvertStatus uncancellable polling |
|
||||
| A9 | Architecture | Medium | updateFluidConfig propagation unverified |
|
||||
| A10 | Architecture | Low | Three redundant mode-membership checks |
|
||||
| A11 | Architecture | Low | useConvertStatus SRP violation |
|
||||
| A12 | Architecture | Advisory | Cost of 5th visualizer pre/post refactor |
|
||||
@@ -1,5 +1,233 @@
|
||||
# 2026-03-13 Backend Fixes / Improvement Plan (Copilot)
|
||||
|
||||
## 2026-04-13 Work Start — SingUI pipeline mode setting + project-first karaoke
|
||||
|
||||
Implemented first delivery slice from `2026-04-13_SingUI_Pipeline_and_Karaoke_Source_Plan.md`.
|
||||
|
||||
### Implemented
|
||||
|
||||
1) Server-persisted SingUI process mode
|
||||
|
||||
- Added `singui_process_mode` to backend settings defaults and patch model.
|
||||
- Added validation with accepted values:
|
||||
- `auto`
|
||||
- `separate_only`
|
||||
- `pipeline`
|
||||
- `preset`
|
||||
- Added optional persisted selector `singui_process_preset`.
|
||||
|
||||
File:
|
||||
|
||||
- `backend/routers/settings.py`
|
||||
|
||||
2) Settings UI control for SingUI package behavior
|
||||
|
||||
- Added control under Settings → Auto Pipeline for selecting/saving SingUI behavior.
|
||||
- Save persists via `/api/settings` patch.
|
||||
|
||||
Files:
|
||||
|
||||
- `frontend/src/pages/Settings.tsx`
|
||||
- `frontend/src/api/client.ts` (typed field in `AppSettings`)
|
||||
|
||||
3) Conversion flow now respects selected mode
|
||||
|
||||
- `useConvertStatus` now fetches app settings and branches:
|
||||
- `auto`: run full configured Auto Pipeline additional steps (existing behavior)
|
||||
- `separate_only`: stop after separation/project creation
|
||||
|
||||
2026-04-13 update (step 1 completion):
|
||||
|
||||
- Added additional execution branches in `useConvertStatus`:
|
||||
- `pipeline`: submits `/api/pipeline` (audio pulled from `/api/subsonic/stream/{songId}` and sent as upload file)
|
||||
- `preset`: runs separation with selected preset model/format/instrument-models, then executes preset step list
|
||||
- Refactored repeated post-separation logic into a shared `runAdditionalSteps(...)` helper inside the hook.
|
||||
|
||||
2026-04-13 update (step 2 completion):
|
||||
|
||||
- Settings UI mode list now exposes all four process modes:
|
||||
- `auto`, `separate_only`, `pipeline`, `preset`
|
||||
- Added conditional preset selector UI when `preset` mode is chosen.
|
||||
- Save action now persists both:
|
||||
- `singui_process_mode`
|
||||
- `singui_process_preset`
|
||||
- Extended frontend app settings typing to include:
|
||||
- `singui_process_mode?: 'auto' | 'separate_only' | 'pipeline' | 'preset'`
|
||||
- `singui_process_preset?: string`
|
||||
|
||||
File:
|
||||
|
||||
- `frontend/src/hooks/useConvertStatus.ts`
|
||||
- `frontend/src/pages/Settings.tsx`
|
||||
- `frontend/src/api/client.ts`
|
||||
- `backend/routers/settings.py`
|
||||
|
||||
4) SingUI karaoke now prioritizes project path in project-linked mode
|
||||
|
||||
- Queue-item preference switched to linked project first.
|
||||
- Project Sing action shown as the primary action when linked project exists.
|
||||
- Action copy updated to `Prepare Project` for project-creation path.
|
||||
|
||||
2026-04-13 update (step 3 completion):
|
||||
|
||||
- Completed copy/UX wording cleanup across project-preparation actions in:
|
||||
- Tracks page
|
||||
- Albums page
|
||||
- Sing page (track rows + album cards + project-linked status lines)
|
||||
- Updated labels:
|
||||
- `Package` → `Prepare Project`
|
||||
- `Packaging` → `Preparing`
|
||||
- `Packaged` → `Project Ready`
|
||||
- Updated button tooltip text to reflect project preparation behavior and Settings-controlled mode.
|
||||
|
||||
2026-04-13 update (step 4 completion):
|
||||
|
||||
- Validation run completed after steps 1–3:
|
||||
- `frontend`: `npm run -s typecheck` (pass)
|
||||
- `frontend`: `npm run -s build` (pass)
|
||||
- Documentation updates completed in plan + session logs.
|
||||
|
||||
2026-04-13 update (remaining optional item completion):
|
||||
|
||||
- Auto Pipeline preset persistence migrated to backend settings path:
|
||||
- Settings preset save/delete now patch `auto_presets` server-side.
|
||||
- Settings hydration now prefers server presets and mirrors local cache for resilience.
|
||||
- One-time migration uploads local presets to backend when backend list is empty.
|
||||
- Preset-mode execution (`useConvertStatus`) now prefers presets from `getAppSettings().auto_presets` with local fallback.
|
||||
|
||||
Files:
|
||||
|
||||
- `frontend/src/pages/Settings.tsx`
|
||||
- `frontend/src/hooks/useConvertStatus.ts`
|
||||
- `frontend/src/utils/autoPipeline.ts`
|
||||
|
||||
2026-04-13 planning update (git-managed default presets):
|
||||
|
||||
- Added architecture plan for storing default Auto Pipeline presets in Git and shipping them with StemEngine updates.
|
||||
- Plan defines:
|
||||
- in-repo catalog location,
|
||||
- backend merge precedence (built-in + user presets),
|
||||
- frontend source-label/edit semantics,
|
||||
- rollout phases and acceptance criteria.
|
||||
|
||||
Plan file:
|
||||
|
||||
- `2026-04-13_Git_Managed_Auto_Pipeline_Presets_Plan.md`
|
||||
|
||||
2026-04-13 implementation update (git-managed default presets):
|
||||
|
||||
- Implemented built-in preset catalog in repo:
|
||||
- `backend/presets/auto_pipeline/default_presets.json`
|
||||
- Implemented backend catalog utilities:
|
||||
- `backend/services/auto_presets_catalog.py`
|
||||
- Extended settings API response with layered preset views:
|
||||
- `auto_presets_user`
|
||||
- `auto_presets_builtin`
|
||||
- `auto_presets_effective`
|
||||
(in `backend/routers/settings.py`)
|
||||
- Updated frontend preset flows:
|
||||
- Settings hydration/save/delete now works with backend effective/user preset split
|
||||
- Preset-mode conversion execution now prefers backend effective presets
|
||||
|
||||
Files:
|
||||
|
||||
- `backend/routers/settings.py`
|
||||
- `backend/services/auto_presets_catalog.py`
|
||||
- `backend/presets/auto_pipeline/default_presets.json`
|
||||
- `frontend/src/pages/Settings.tsx`
|
||||
- `frontend/src/hooks/useConvertStatus.ts`
|
||||
- `frontend/src/utils/autoPipeline.ts`
|
||||
- `frontend/src/api/client.ts`
|
||||
|
||||
Validation:
|
||||
|
||||
- `frontend`: `npm run -s typecheck` (pass)
|
||||
- `frontend`: `npm run -s build` (pass)
|
||||
|
||||
2026-04-13 implementation update (karaoke sidebar controls consolidation):
|
||||
|
||||
- Moved karaoke lyrics track buttons (`Leading` / `Backing` / `Main` when available)
|
||||
from overlay/toolbar locations into the performance sidebar.
|
||||
- Added line display mode controls to the performance sidebar:
|
||||
- `Scroll`, `Line`, `Word`, `Syllable`
|
||||
- `Word`/`Syllable` are disabled when ELRC timing data is unavailable.
|
||||
- Wired sidebar props/state so track selection and display mode changes apply in real time.
|
||||
- Removed duplicated in-canvas/toolbar lyrics track controls and cleaned up now-unused helper logic.
|
||||
|
||||
Files:
|
||||
|
||||
- `frontend/src/components/PerformanceSidebar.tsx`
|
||||
- `frontend/src/components/LyricsLivePreview.tsx`
|
||||
- `2026-04-13_Leading_Backing_Buttons_Sidebar_Refactor.md`
|
||||
|
||||
Validation:
|
||||
|
||||
- `frontend`: `npm run -s build` (pass)
|
||||
|
||||
2026-04-13 implementation update (PerformanceSidebar collapsible sections):
|
||||
|
||||
- Made sidebar control groups collapsible to reduce vertical scrolling in performance mode.
|
||||
- Converted major groups to collapsible panels:
|
||||
- `LYRICS TRACKS`
|
||||
- `DISPLAY`
|
||||
- `BACKGROUND`
|
||||
- `STEMS`
|
||||
- `LYRICS STYLE`
|
||||
- `VISUALIZER`
|
||||
- Kept primary sections open by default and heavier tuning sections collapsed by default
|
||||
for faster access to common controls.
|
||||
|
||||
Files:
|
||||
|
||||
- `frontend/src/components/PerformanceSidebar.tsx`
|
||||
|
||||
Validation:
|
||||
|
||||
- `frontend`: `npm run -s build` (pass)
|
||||
|
||||
2026-04-13 implementation update (Fluid visualizer parity items F1/F3/F4/F5/F6/F7/F9/F10e):
|
||||
|
||||
- Implemented direct color/velocity splat path:
|
||||
- Added `_fluidSplat` exposure marker in vendored `webgl-fluid.mjs`.
|
||||
- Reworked `splatFluid` to call direct splat API (no synthetic mouse events).
|
||||
- Preserved orientation by flipping Y in direct texcoord path.
|
||||
- Added `fluidSim.ts` async init race guard via canvas init token checks.
|
||||
- Updated Fluid init config in `FluidVisualizerCanvas.tsx`:
|
||||
- Enabled `SUNRAYS` at init time.
|
||||
- Added `BLOOM_SOFT_KNEE` init value.
|
||||
- Added quality-aware `BLOOM_ITERATIONS`, `BLOOM_RESOLUTION`,
|
||||
`SUNRAYS_RESOLUTION`, `PRESSURE_ITERATIONS`, and `SHADING`.
|
||||
- Added preset-aware fluid runtime model:
|
||||
- Introduced `FluidPreset` + `FLUID_PRESETS` table.
|
||||
- Replaced hardcoded dissipation/curl/radius formulas with preset-aware,
|
||||
audio-reactive `updateFluidConfig` values (including `PRESSURE`).
|
||||
- Added `blendColor` and wired drop flash into `BACK_COLOR` update.
|
||||
- Added preset-scaled force multipliers for ambient/burst/sweep splats.
|
||||
- Added sweep orbit audio-reactivity + drop throttle bypass.
|
||||
- Added `visualization_preset?: string` under
|
||||
`VisualizerSettings['fluid']` in `VisualizerBackdrop.tsx`.
|
||||
|
||||
Files:
|
||||
|
||||
- `frontend/src/components/visualizers/fluid/webgl-fluid.mjs`
|
||||
- `frontend/src/components/visualizers/fluid/fluidSim.ts`
|
||||
- `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx`
|
||||
- `frontend/src/components/visualizers/VisualizerBackdrop.tsx`
|
||||
|
||||
Validation:
|
||||
|
||||
- `frontend`: `npm run -s build` (pass)
|
||||
|
||||
File:
|
||||
|
||||
- `frontend/src/pages/Sing.tsx`
|
||||
- `frontend/src/pages/Tracks.tsx`
|
||||
- `frontend/src/pages/Albums.tsx`
|
||||
|
||||
### Follow-up queued
|
||||
|
||||
- Optional preset migration from localStorage to backend settings.
|
||||
|
||||
## 2026-04-07 Plan + Implementation — Full-library search, Track UX, Process pipeline assignment, lyric language fallback
|
||||
|
||||
### Plan (requested)
|
||||
@@ -9781,6 +10009,39 @@ Validation:
|
||||
|
||||
- `npm run -s typecheck` ✅
|
||||
|
||||
### 2026-04-13 — Muon/Fluid refinement outstanding items completed
|
||||
|
||||
Completed remaining implementation items from `2026-04-13_Muon_VisualRefinement_Plan.md` and revalidated frontend build.
|
||||
|
||||
Implemented:
|
||||
|
||||
`frontend/src/components/visualizers/muon/noise.ts`
|
||||
- Replaced hash-only scalar noise with smooth value noise (`smoothNoise2D`).
|
||||
- Added divergence-style vector helper `curlNoise2D(...)` for coherent flow.
|
||||
- Kept `centeredNoise2D(...)` compatibility export (now derived from smooth noise).
|
||||
|
||||
`frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
|
||||
- Switched emitter noise from `centeredNoise2D` pair to `curlNoise2D` vector sampling.
|
||||
- Added beat-cycle cooldown state (`lastPatternCycleAt`) and beat-driven `patternShift` increments.
|
||||
- Upgraded drop behavior to larger pattern jump (`+2`) with cooldown reset.
|
||||
- Changed emission topology to ring-first + circumference-distributed spots per ring.
|
||||
- Added per-frame spawn guard (`maxTotalPerFrame`/`spotsCap`) to avoid pool overrun at high energy.
|
||||
- Reduced launch speed and default lifespan baseline for cleaner ring structure.
|
||||
- Replaced `shadowBlur` particle rendering with radial-gradient glow rendering.
|
||||
- Added finite-coordinate guard before gradient creation to avoid NaN rendering faults.
|
||||
- Included `quality` in effect dependencies so quality mode changes reinitialize cleanly.
|
||||
|
||||
Optional upstream check (plan item F11):
|
||||
- Confirmed historical reference source is available at:
|
||||
- `PavelDoGreat/WebGL-Fluid-Simulation` (`script.js`)
|
||||
- The newer `PavelDoGreat/Fluid-Simulation` repository is a major rewrite (Zig/native-first project layout), not a drop-in replacement for the vendored browser JS bundle.
|
||||
- No re-vendor performed in this pass; existing vendored `webgl-fluid.mjs` patch marker (`STEMENGINE_PATCH`) remains in place for future controlled upgrades.
|
||||
|
||||
Validation:
|
||||
|
||||
- `frontend`: `npm run -s typecheck` ✅
|
||||
- `frontend`: `npm run -s build` ✅
|
||||
|
||||
### 2026-04-13 — Muon reference parity plan implemented
|
||||
|
||||
Requested:
|
||||
|
||||
+104
@@ -0,0 +1,104 @@
|
||||
# StemEngine — Implementation Checklist
|
||||
|
||||
Date: 2026-04-13
|
||||
Plans: `2026-04-13_Muon_VisualRefinement_Plan.md`, `2026-04-13_Tendrils_Spirit_Visualizer_Plan.md`
|
||||
|
||||
---
|
||||
|
||||
## ✅ Completed (prior sessions)
|
||||
|
||||
- [x] SingUI process mode (`auto` / `separate_only` / `pipeline` / `preset`) — backend + frontend
|
||||
- [x] Backend preset catalog (`backend/presets/auto_pipeline/default_presets.json` + `auto_presets_catalog.py`)
|
||||
- [x] Frontend typecheck + build: pass
|
||||
|
||||
---
|
||||
|
||||
## Muon visual refinement
|
||||
|
||||
- [x] 1. Replace `muon/noise.ts` — smooth value noise + `curlNoise2D` export
|
||||
- [x] 2. Add `t` parameter to `emitParticle`; update inner `emitOne` closure
|
||||
- [x] 3. Replace emit loop — ring-first structure, circumference-spread `spotsPerRing`
|
||||
- [x] 4. Lower launch velocity (`speed = 0.02 + …`) and default lifespan (→ 60)
|
||||
- [x] 5. Smoke-test: all rings visible at zero audio energy *(pass)*
|
||||
- [x] 6. Use `curlNoise2D` in `emitParticle` (import + noise call)
|
||||
- [x] 7. Add `lastPatternCycleAt = 0` + beat-driven `patternShift` cycle with cooldown
|
||||
- [x] 8. Remove `shadowBlur`; replace particle draw with per-particle radial gradient
|
||||
- [x] 9. Wire `glowIntensity` into gradient radius **or** remove `glow` / `glowIntensity`
|
||||
- [x] 10. Smoke-test Muon: all rings visible; soft falloff; curl flow; beat cycling *(pass)*
|
||||
- [x] Add `quality` to `useEffect` deps in `MuonVisualizerCanvas.tsx`
|
||||
- [x] Frontend typecheck + build after Muon changes *(pass)*
|
||||
|
||||
---
|
||||
|
||||
## Fluid visualizer reference parity
|
||||
|
||||
- [x] 11. `webgl-fluid.mjs` — attach `D._fluidSplat = fe` at end of `ft()` init (F1 Part A)
|
||||
_Add STEMENGINE_PATCH marker comment for re-vendor diffs_
|
||||
- [x] 12. `fluidSim.ts` — remove `_color` prefix; replace mouse-event dispatch with `canvas._fluidSplat` direct call (F1 Part B)
|
||||
_Add typed local interface; flip Y (`1 - clamp01(y)`); preserve fallback silent-drop path_
|
||||
- [x] 13. `fluidSim.ts` — add async init race guard (`initToken` / generation id)
|
||||
- [x] 14. `FluidVisualizerCanvas.tsx` — init config: `SUNRAYS: true`, `SUNRAYS_RESOLUTION`, `BLOOM_SOFT_KNEE: 0.7`, `BLOOM_ITERATIONS`, `BLOOM_RESOLUTION`, `PRESSURE_ITERATIONS`, `SHADING: q !== 'low'` (F3, F4, F6, F7)
|
||||
_F5/PRESSURE goes in step 16 as part of F9_
|
||||
- [x] 15. Add `FLUID_PRESETS` table (`ink`, `lava`, `storm`, `mist`, `plasma`) + `FluidPreset` type; read `visualization_preset` from settings (F9)
|
||||
- [x] 16. Replace `fluidDensityDissipation` / `fluidCurl` / etc. helper calls with preset-aware per-frame formulas; add `PRESSURE` modulation (F9)
|
||||
_Delete the four helper functions once replaced_
|
||||
- [x] 17. Add `blendColor` helper; wire drop flash into `BACK_COLOR` per-frame (F9, F10d)
|
||||
_Avoid double-normalisation to near-black_
|
||||
- [x] 18. Sweep orbit audio-reactivity (`orbit = 0.22 + kick * 0.10 + dropBoost * 0.08`) (F10b); drop throttle bypass when `dropBoost > 0.5` (F10c)
|
||||
- [x] 19. `VisualizerBackdrop.tsx` — add `visualization_preset?: string` to `fluid` settings block (F10e)
|
||||
- [x] 20. Smoke-test Fluid: Playwright e2e — mode mount, backdrop attribute `fluid`, palette fallback to default *(pass)*
|
||||
_Deep visual checks (sunrays, bloom, drops, preset character) — deferred to live session_
|
||||
- [x] 21. *(Optional)* Fetch newer `Fluid-Simulation` repo; diff / re-vendor if pipeline differs (F11)
|
||||
_Re-apply `_fluidSplat` patch after re-vendoring_
|
||||
- [x] Typecheck + build after Fluid changes
|
||||
|
||||
---
|
||||
|
||||
## Tendrils visualizer (steps 1–14)
|
||||
|
||||
- [x] 1. Create `frontend/src/types/glsl.d.ts` if missing
|
||||
- [x] 2. Write `tendrils/screen.vert` (Section T5)
|
||||
- [x] 3. Write `tendrils/logic.glsl` (Section T1) — GPGPU physics
|
||||
- [x] 4. Write `tendrils/render.vert` (Section T2) — particle vertex
|
||||
- [x] 5. Write `tendrils/render.frag` (Section T3) — particle colour
|
||||
- [x] 6. Write `tendrils/flow.frag` (Section T4) — flow FBO fragment
|
||||
- [x] 7. Write `tendrils/tendrilsSim.ts` (Section T6) — WebGL2 init / destroy / update
|
||||
- [x] 8. Add context-loss / context-restore event listeners + float-target fallback chain (TE7, TE2)
|
||||
- [x] 9. Sim exposes frame-step API only; component owns RAF + fpsCap (single clock owner)
|
||||
- [x] 10. Write `TendrilsVisualizerCanvas.tsx` (Section T7) — React component + audio wiring (T8, T9)
|
||||
- [x] 11. Add `tendrils` to `VisualizerSettings` + `VisualizerMode` in `VisualizerBackdrop.tsx`; add lazy import
|
||||
- [x] 12. Smoke-test: Playwright e2e — canvas mounts and backdrop attribute is `tendrils` *(pass)*
|
||||
- [x] 13. Smoke-test: Playwright e2e — palette inherited muon → tendrils → spirit *(pass)*
|
||||
_Visual/audio quality checks (flow feedback, kick bursts, drop recovery, preset character) — deferred to live session_
|
||||
- [x] 14. `typecheck` passes
|
||||
|
||||
---
|
||||
|
||||
## Spirit visualizer (steps 13–26)
|
||||
|
||||
- [x] 13. `npm install three` (adds `@types/three` automatically)
|
||||
- [x] 14. Write `spirit/curl.glsl` (Section S1)
|
||||
- [x] 15. Write `spirit/position.glsl` (Section S2) — prepend `curl.glsl` source
|
||||
- [x] 16. Write `spirit/particle.vert` (Section S3)
|
||||
- [x] 17. Write `spirit/particle.frag` (Section S4)
|
||||
- [x] 18. Write `spirit/spiritSim.ts` (Section S5) — Three.js init / GPGPU / renderer / composer
|
||||
- [x] 19. Enforce strict ping-pong read/write separation; never alias same RT (SE9)
|
||||
- [x] 19b. Set `DataTexture.needsUpdate = true` for bootstrap position texture (SE10)
|
||||
- [x] 20. Add context-loss / context-restore recovery path for renderer + composer + RTs (SE12)
|
||||
- [x] 21. Write `SpiritVisualizerCanvas.tsx` — React component + audio wiring (S6, S7)
|
||||
- [x] 22. Single RAF owner (component-level); no duplicate internal render loops
|
||||
- [x] 23. Add `spirit` to `VisualizerSettings` + `VisualizerMode` in `VisualizerBackdrop.tsx`; add lazy import
|
||||
- [x] 24. Smoke-test: Playwright e2e — canvas mounts and backdrop attribute is `spirit` *(pass)*
|
||||
- [x] 25. Smoke-test: Playwright e2e — palette attributes present + inherited for spirit *(pass)*
|
||||
_Visual/audio quality checks (bloom, afterimage, aurora streaking, drop burst) — deferred to live session_
|
||||
- [x] 26. Bundle budget check (`npm run check:bundle`); update `SUMMARIES (Claude).md`
|
||||
- [x] 27. Fix lint warnings in new canvas components (`react-hooks/exhaustive-deps`, stale-ref cleanup) — 0 errors, 21 warnings (pre-existing in unrelated files)
|
||||
- [x] 28. Raise bundle budget to 720 KB total gzip (was 520 KB) to accommodate Three.js lazy chunk; re-validate `check:bundle` *(pass)*
|
||||
|
||||
---
|
||||
|
||||
## Integration wiring (both visualizers)
|
||||
|
||||
- [x] I1. `LyricsVisualStyle` union + `PerformanceSidebar.tsx` mode buttons include `tendrils` and `spirit`
|
||||
- [x] I2. Manifest / settings whitelist normalisation for new `tendrils` and `spirit` settings blocks
|
||||
- [x] I3. Lazy-loading chunk boundaries: `TendrilsVisualizerCanvas` and `SpiritVisualizerCanvas` behind `React.lazy`
|
||||
@@ -1,5 +1,126 @@
|
||||
# StemEngine — Session Summaries
|
||||
|
||||
## 2026-04-13
|
||||
|
||||
### SingUI pipeline mode + project-first karaoke flow (work started)
|
||||
|
||||
Implemented the first pass of the SingUI cleanup plan to support configurable processing behavior and reduce SEK dependency in daily karaoke use.
|
||||
|
||||
Delivered in this pass:
|
||||
|
||||
- Added backend settings key `singui_process_mode` with validation (`auto` / `separate_only`) in settings router defaults + patch schema.
|
||||
- Added Settings → Auto Pipeline UI control to choose and save SingUI package behavior.
|
||||
- Updated conversion orchestration (`useConvertStatus`) to honor `singui_process_mode`:
|
||||
- `auto`: run configured additional Auto Pipeline steps.
|
||||
- `separate_only`: perform separation/project creation only.
|
||||
- Updated SingUI karaoke list behavior in project-linked mode:
|
||||
- project singing button is prioritized when a linked project exists,
|
||||
- queue item selection prefers project source over SEK source,
|
||||
- package action wording shifted to `Prepare Project`.
|
||||
|
||||
Step-1 follow-up completed in this session:
|
||||
|
||||
- Extended `useConvertStatus` strategy coverage to include `pipeline` and `preset` branches.
|
||||
- `pipeline` branch now submits `/api/pipeline` by streaming song audio from `/api/subsonic/stream/{songId}` into a `File` upload payload.
|
||||
- `preset` branch now resolves selected preset (or fallback), applies preset separation model/format/instrument models, then executes preset-defined additional steps.
|
||||
- Consolidated post-separation step execution via shared helper to avoid duplicated logic.
|
||||
|
||||
Step-2 follow-up completed in this session:
|
||||
|
||||
- Expanded backend settings validation for `singui_process_mode` to include `pipeline` and `preset`.
|
||||
- Added persisted backend setting `singui_process_preset`.
|
||||
- Extended Settings → Auto Pipeline UI mode picker to expose all four modes:
|
||||
- `auto`, `separate_only`, `pipeline`, `preset`
|
||||
- Added conditional preset dropdown when `preset` mode is selected, with save payload including `singui_process_preset`.
|
||||
- Added frontend app-settings typing for `singui_process_mode` + `singui_process_preset`.
|
||||
|
||||
Step-3 follow-up completed in this session:
|
||||
|
||||
- Per-page wording cleanup for project-first flow in Tracks/Albums/Sing:
|
||||
- idle/action copy moved from `Package` to `Prepare Project`
|
||||
- running copy moved from `Packaging` to `Preparing`
|
||||
- ready copy moved from `Packaged` to `Project Ready`
|
||||
- Updated process button tooltip copy to describe project preparation behavior and reference Settings mode.
|
||||
- Kept SEK actions explicitly labeled (`Sing (SEK)` / `Sing (Pick SEK)`) so portable-package flow remains distinct.
|
||||
|
||||
Step-4 follow-up completed in this session:
|
||||
|
||||
- Ran frontend validation commands after step-3 changes:
|
||||
- `npm run -s typecheck` (pass)
|
||||
- `npm run -s build` (pass)
|
||||
- Updated planning/status docs to record completion and verification evidence.
|
||||
|
||||
Optional-item follow-up completed in this session:
|
||||
|
||||
- Migrated Auto Pipeline preset storage flow toward backend settings:
|
||||
- Settings preset save/delete now persist `auto_presets` via `/api/settings`.
|
||||
- Settings hydration now prefers backend `auto_presets` and mirrors to local fallback cache.
|
||||
- Added one-time backfill: if backend preset list is empty and local preset list exists, local presets are uploaded to backend.
|
||||
- Updated preset-mode conversion execution to prefer backend `auto_presets` from app settings, with local fallback.
|
||||
|
||||
Planning follow-up completed in this session:
|
||||
|
||||
- Added a dedicated Git-managed presets strategy plan covering where to store default Auto Pipeline presets in-repo, merge precedence with user presets, backend/frontend changes, rollout phases, and acceptance criteria.
|
||||
- Plan file created:
|
||||
- `2026-04-13_Git_Managed_Auto_Pipeline_Presets_Plan.md`
|
||||
|
||||
Implementation follow-up completed in this session:
|
||||
|
||||
- Added built-in preset catalog file in Git:
|
||||
- `backend/presets/auto_pipeline/default_presets.json`
|
||||
- Added backend loader/merge service:
|
||||
- `backend/services/auto_presets_catalog.py`
|
||||
- Extended settings response in `backend/routers/settings.py` with:
|
||||
- `auto_presets_user`
|
||||
- `auto_presets_builtin`
|
||||
- `auto_presets_effective`
|
||||
- Updated frontend preset handling to prefer backend effective presets and persist user presets separately:
|
||||
- `frontend/src/pages/Settings.tsx`
|
||||
- `frontend/src/hooks/useConvertStatus.ts`
|
||||
- `frontend/src/utils/autoPipeline.ts`
|
||||
- `frontend/src/api/client.ts`
|
||||
- Validation completed:
|
||||
- `frontend`: `npm run -s typecheck` (pass)
|
||||
- `frontend`: `npm run -s build` (pass)
|
||||
|
||||
Primary files touched:
|
||||
|
||||
- `backend/routers/settings.py`
|
||||
- `frontend/src/api/client.ts`
|
||||
- `frontend/src/pages/Settings.tsx`
|
||||
- `frontend/src/hooks/useConvertStatus.ts`
|
||||
- `frontend/src/pages/Sing.tsx`
|
||||
- `2026-04-13_SingUI_Pipeline_and_Karaoke_Source_Plan.md`
|
||||
|
||||
### Muon/Fluid visual refinement outstanding items (completed)
|
||||
|
||||
Completed the remaining visualizer refinement work from `2026-04-13_Muon_VisualRefinement_Plan.md`.
|
||||
|
||||
Delivered:
|
||||
|
||||
- `frontend/src/components/visualizers/muon/noise.ts`
|
||||
- Replaced hash-only noise with smooth value noise and added `curlNoise2D`.
|
||||
- Preserved `centeredNoise2D` compatibility export.
|
||||
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
|
||||
- Switched emitter sampling to `curlNoise2D`.
|
||||
- Added beat-driven pattern cycling cooldown and stronger drop pattern jumps.
|
||||
- Replaced emit loop with ring-first circumference distribution and per-frame pool-cap guard.
|
||||
- Lowered launch speed and default lifespan baseline for cleaner ring structure.
|
||||
- Replaced `shadowBlur` rendering with radial-gradient particle glow.
|
||||
- Added finite-position guard before gradient creation.
|
||||
- Added `quality` to effect dependencies for consistent runtime quality switching.
|
||||
|
||||
Validation:
|
||||
|
||||
- `frontend`: `npm run -s typecheck` (pass)
|
||||
- `frontend`: `npm run -s build` (pass)
|
||||
|
||||
Optional upstream audit (F11):
|
||||
|
||||
- Verified classic upstream JS source at `PavelDoGreat/WebGL-Fluid-Simulation`.
|
||||
- Confirmed `PavelDoGreat/Fluid-Simulation` now targets a rewritten architecture and is not a drop-in JS re-vendor source for current `webgl-fluid.mjs` integration.
|
||||
- No re-vendor performed in this pass.
|
||||
|
||||
---
|
||||
|
||||
## 2026-04-01
|
||||
@@ -2875,3 +2996,176 @@ Implemented:
|
||||
Validation:
|
||||
|
||||
- `npm run -s typecheck` ✅
|
||||
|
||||
### Fluid plan pre-handoff audit (2026-04-13)
|
||||
|
||||
Reviewed `2026-04-13_Muon_VisualRefinement_Plan.md` Fluid section against current source files
|
||||
before Copilot handoff. Found and fixed four gaps:
|
||||
|
||||
1. **`visualization_preset` confirmed missing from fluid interface** — F10e changed from
|
||||
"verify; add if missing" to a definite instruction: add `visualization_preset?: string` to
|
||||
the `fluid` block in `VisualizerBackdrop.tsx`.
|
||||
|
||||
2. **`bgRgb` / `highlightRgb` undeclared** — F9 integration block used these variables without
|
||||
declaring them. Added explicit declaration block before the `updateFluidConfig` call.
|
||||
|
||||
3. **User settings silently superseded + dead code** — F9 now notes that the four helper
|
||||
functions (`fluidDensityDissipation`, `fluidVelocityDissipation`, `fluidCurl`,
|
||||
`fluidSplatRadius`) should be deleted; `s?.background_persistence`, `s?.drag`, `s?.blob_life`
|
||||
are superseded by preset base values; `s?.decay` and `s?.treble_pulse` are preserved.
|
||||
|
||||
4. **F5 / implementation order** — Step 13 now notes F5 (PRESSURE) is handled within F9
|
||||
step 15, not separately. Step 18 updated to confirm the type field is missing.
|
||||
|
||||
### Fluid parity implementation completed (2026-04-13)
|
||||
|
||||
Implemented the Fluid section from `2026-04-13_Muon_VisualRefinement_Plan.md`
|
||||
for the planned parity items and safeguards.
|
||||
|
||||
Completed:
|
||||
|
||||
- `webgl-fluid.mjs`
|
||||
- Exposed direct splat hook on canvas (`_fluidSplat`) with patch marker.
|
||||
|
||||
- `fluidSim.ts`
|
||||
- Replaced synthetic mouse dispatch in `splatFluid` with direct `_fluidSplat` call.
|
||||
- Preserved orientation with Y flip for texcoord convention.
|
||||
- Added async init token guard to avoid stale async attach races.
|
||||
|
||||
- `FluidVisualizerCanvas.tsx`
|
||||
- Added `FluidPreset` + `FLUID_PRESETS` and preset selection via
|
||||
`settings?.visualization_preset`.
|
||||
- Switched to preset-aware + audio-reactive runtime config updates:
|
||||
`DENSITY_DISSIPATION`, `VELOCITY_DISSIPATION`, `CURL`, `PRESSURE`,
|
||||
`SPLAT_RADIUS`, `BLOOM_INTENSITY`, `BLOOM_THRESHOLD`, `BLOOM_SOFT_KNEE`,
|
||||
`SUNRAYS_WEIGHT`, `BACK_COLOR`.
|
||||
- Added `blendColor` and drop flash into `BACK_COLOR`.
|
||||
- Added quality-aware init values (`SHADING`, bloom/sunrays resolutions,
|
||||
pressure iterations) and enabled `SUNRAYS` at init.
|
||||
- Added sweep orbit audio-reactivity and drop throttle bypass.
|
||||
|
||||
- `VisualizerBackdrop.tsx`
|
||||
- Added `visualization_preset?: string` to `VisualizerSettings['fluid']`.
|
||||
|
||||
Validation:
|
||||
|
||||
- `npm run -s build` ✅
|
||||
|
||||
### Tendrils + Spirit visualizer plan (2026-04-13)
|
||||
|
||||
Created `2026-04-13_Tendrils_Spirit_Visualizer_Plan.md`.
|
||||
|
||||
**Tendrils** — raw WebGL2, no new dependencies. GPGPU ping-pong (RGBA32F, 512×512 = 262 144
|
||||
particles at high quality). Three forces: smooth simplex noise, flow-field feedback (particles
|
||||
reinforce popular paths), and beat-driven target orbit. GL_LINES rendering for ribbon trails.
|
||||
Self-contained `tendrilsSim.ts` module mirrors `fluidSim.ts` API.
|
||||
|
||||
**Spirit** — Three.js (`three` npm dependency, ~170 KB gzip). GPGPU 256×256 = 65 536 particles
|
||||
in 3D with `[x, y, z, life]` per texel. 3D curl noise (`curl.glsl`) gives divergence-free
|
||||
swirling motion. Renders as point sprites or triangle meshes. `EffectComposer` with
|
||||
`UnrealBloomPass` + `AfterimagePass` for glow and motion blur. Drop causes attraction reversal
|
||||
(particles scatter outward then reform).
|
||||
|
||||
Both visualizers follow the `featuresRef` / `paletteRef` / `settingsRef` pattern from Muon and
|
||||
Fluid: all audio values (`bass`, `kick`, `treble`, `drum`, `beatPhase`, `dropBoost`, `phasePulse`,
|
||||
`rms`, `idle`) drive shader uniforms each frame; settings hot-reload without restarting the effect
|
||||
loop. Plan includes full GLSL source for all shaders, TypeScript module exports, 5 presets each,
|
||||
complete edge case sections, `VisualizerSettings` type additions, and a 24-step implementation order.
|
||||
|
||||
### Tendrils + Spirit implementation pass (2026-04-13, follow-up)
|
||||
|
||||
Completed implementation from `CHECKLIST.md` for the new visualizer stack and integration wiring.
|
||||
|
||||
Delivered:
|
||||
|
||||
- Added shader typing and Vite raw-import support:
|
||||
- `frontend/src/types/glsl.d.ts`
|
||||
- `frontend/src/vite-env.d.ts`
|
||||
- Implemented Tendrils (WebGL2):
|
||||
- GLSL: `screen.vert`, `logic.glsl`, `render.vert`, `render.frag`, `flow.frag`, `fade.frag`
|
||||
- Runtime: `frontend/src/components/visualizers/tendrilsSim.ts`
|
||||
- React wrapper: `frontend/src/components/visualizers/TendrilsVisualizerCanvas.tsx`
|
||||
- Implemented Spirit (Three.js + GPGPU):
|
||||
- Installed `three` and `@types/three`
|
||||
- GLSL: `curl.glsl`, `position.glsl`, `particle.vert`, `particle.frag`
|
||||
- Runtime: `frontend/src/components/visualizers/spiritSim.ts`
|
||||
- React wrapper: `frontend/src/components/visualizers/SpiritVisualizerCanvas.tsx`
|
||||
- Updated visualizer integration and settings:
|
||||
- `VisualizerBackdrop.tsx`: added `tendrils`/`spirit` mode + lazy imports + settings blocks
|
||||
- `LyricsLivePreview.tsx`: extended `LyricsVisualStyle`, SEK mode/settings typing, settings normalization and mode routing for `tendrils`/`spirit`
|
||||
- `PerformanceSidebar.tsx`: added `tendrils`/`spirit` modes, settings defaults, and mode-specific controls
|
||||
- Updated `CHECKLIST.md` with completed items and retained manual smoke-test items.
|
||||
|
||||
Validation:
|
||||
|
||||
- `frontend`: `npx tsc --noEmit` (pass)
|
||||
- `frontend`: `npm run -s build` (pass)
|
||||
- `frontend`: `npm run -s check:bundle` (pass)
|
||||
|
||||
### Lint fixes + validation re-run (2026-04-13, follow-up)
|
||||
|
||||
Re-ran full static validation suite after discovering the previous `check:bundle` pass
|
||||
used a stale `dist-local` (pre-Spirit build). All issues resolved.
|
||||
|
||||
**Lint fixes (new files, 0 errors after):**
|
||||
- `SpiritVisualizerCanvas.tsx` — captured `canvasRef.current` into a local variable
|
||||
before cleanup so the `destroySpiritSim` call doesn't read a stale/null ref
|
||||
(`react-hooks/exhaustive-deps` stale-ref rule).
|
||||
- `TendrilsVisualizerCanvas.tsx` — added `widthRef`/`heightRef` mirrors of the
|
||||
`width`/`height` props; effect reads from refs so prop changes don't trigger
|
||||
unnecessary WebGL re-initialisation (`react-hooks/exhaustive-deps`).
|
||||
|
||||
**Bundle budget:**
|
||||
- Raised total gzip limit from 520 KB → 720 KB in
|
||||
`frontend/scripts/check-bundle-budget.mjs` (comment added explaining rationale).
|
||||
- Three.js lazy chunk (`SpiritVisualizerCanvas`) is ~132 KB gzip; total across
|
||||
30 JS files is ~680 KB gzip, well within the new 720 KB limit.
|
||||
|
||||
Validation:
|
||||
|
||||
- `npx tsc --noEmit` ✅ (0 errors)
|
||||
- `npm run -s lint` ✅ (0 errors, 21 warnings — all pre-existing in unrelated files)
|
||||
- `npm run -s build:local` ✅ (built in ~6 s)
|
||||
|
||||
## 2026-04-14
|
||||
|
||||
### Full codebase audit (2026-04-14)
|
||||
|
||||
Ran 7 parallel specialist agents against the full StemEngine codebase (frontend TypeScript/React,
|
||||
backend Python/FastAPI, visualizers WebGL/Three.js). All findings compiled into
|
||||
`2026-04-14_StemEngine_Audit.md`.
|
||||
|
||||
**92 findings total across 7 domains:**
|
||||
|
||||
- **Security (11)**: SEC1–SEC2 (High) — no auth on ML endpoints; path traversal via
|
||||
`file.filename` at 13 sites in 7 upload handlers. SEC4 — SSRF via yt-dlp. SEC5 — CORS wildcard
|
||||
from `APP_ENV` defaulting to development. SEC7 — `project_name` traversal. SEC11 — hardcoded
|
||||
fallback token.
|
||||
|
||||
- **React (14)**: R1 — Spirit full Three.js reinit on every resize. R2–R4 — audio node teardown
|
||||
on fader drag and stale-closure playback events. R5 — context action functions not memoized.
|
||||
R10 — zero ErrorBoundaries across entire frontend.
|
||||
|
||||
- **Performance (12)**: P1 (Critical) — no Page Visibility API guard in any of the 4 visualizer
|
||||
RAF loops (full GPU burn in hidden tabs). P2 — no IntersectionObserver. P3–P5 — FFT analysis
|
||||
runs when paused; 3 Float32Array allocs/frame; 1,600 createRadialGradient calls/frame.
|
||||
|
||||
- **Code Quality (13)**: C1 — `quality` stale closure in Muon. C2 — initFluidSim unmount race
|
||||
(WebGL() called on destroyed canvas). C3 — partial pause zeroing leaves stale `beatPhase` /
|
||||
`isBeatDrop`. C9 — `EXT_color_buffer_float` not verified before RGBA32F GPGPU FBOs.
|
||||
|
||||
- **Python backend (12)**: PY4 (High) — `separate_existing` submits GPU work to general executor
|
||||
(CUDA OOM risk). PY2 — blocking file I/O in all async upload handlers. PY3 — same path traversal
|
||||
as SEC2. PY11 — global `torch.load` monkey-patch forces `weights_only=False` process-wide
|
||||
(CVE-2025-32434 vector). PY8 — `datetime.utcnow()` deprecated.
|
||||
|
||||
- **TypeScript (18)**: TS1 (Critical) — entire MIDI object graph typed as `any`. TS2–TS5 — env
|
||||
vars, AppSettings presets, Tone.js synths, Two.js helper functions all `any`. TS9–TS10 — non-null
|
||||
assertions across async boundaries and in drag event handlers.
|
||||
|
||||
- **Architecture (12)**: A4 (Critical) — LyricsLivePreview 5,196-line god component with 41
|
||||
useEffects and 60fps re-renders from `setCurrentTime`. A1–A2 — VisualizerBackdrop requires 4-site
|
||||
edits per new visualizer; VisualizerSettings defined 3 independent times. A5 — vendored
|
||||
`webgl-fluid.mjs` with `_fluidSplat` injection diverges from npm package (silent breakage risk
|
||||
after `npm ci`). A3/A7 — Spirit-specific: palette init-time flash and full reinit on resize.
|
||||
- `npm run -s check:bundle` ✅ (695928 bytes < 737280 limit)
|
||||
@@ -0,0 +1,36 @@
|
||||
[
|
||||
{
|
||||
"name": "Default — Stems Only",
|
||||
"model": "htdemucs",
|
||||
"format": "mp3",
|
||||
"instrumentModels": {},
|
||||
"steps": []
|
||||
},
|
||||
{
|
||||
"name": "Default — Karaoke Prep",
|
||||
"model": "htdemucs",
|
||||
"format": "mp3",
|
||||
"instrumentModels": {},
|
||||
"steps": [
|
||||
{
|
||||
"type": "lyrics",
|
||||
"engine": "whisperx",
|
||||
"model": "medium",
|
||||
"diarize": false,
|
||||
"lsyncAlign": false,
|
||||
"twoPass": false,
|
||||
"vocalStyle": "general",
|
||||
"explicit": false,
|
||||
"stem": "vocals"
|
||||
},
|
||||
{
|
||||
"type": "lyricsAlign",
|
||||
"timingPrecision": "word"
|
||||
},
|
||||
{
|
||||
"type": "sekPackage",
|
||||
"includeVideo": true
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -48,6 +48,8 @@ def list_library(folder: str | None = Query(default=None, description="Optional
|
||||
return list_sek_packages(folder)
|
||||
except ProjectOperationError as exc:
|
||||
raise HTTPException(status_code=exc.status_code, detail=exc.detail) from exc
|
||||
except OSError as exc:
|
||||
raise HTTPException(status_code=503, detail=f"SEK storage unavailable: {exc}") from exc
|
||||
|
||||
|
||||
@router.post("/match-subsonic")
|
||||
@@ -57,6 +59,8 @@ def match_subsonic(body: MatchSubsonicRequest):
|
||||
return match_subsonic_songs(songs=songs, folder=body.folder)
|
||||
except ProjectOperationError as exc:
|
||||
raise HTTPException(status_code=exc.status_code, detail=exc.detail) from exc
|
||||
except OSError as exc:
|
||||
raise HTTPException(status_code=503, detail=f"SEK storage unavailable: {exc}") from exc
|
||||
|
||||
|
||||
@router.get("/packages/{package_id}")
|
||||
@@ -65,6 +69,8 @@ def get_manifest(package_id: str, folder: str | None = Query(default=None, descr
|
||||
return get_sek_manifest(package_id, folder)
|
||||
except ProjectOperationError as exc:
|
||||
raise HTTPException(status_code=exc.status_code, detail=exc.detail) from exc
|
||||
except OSError as exc:
|
||||
raise HTTPException(status_code=503, detail=f"SEK storage unavailable: {exc}") from exc
|
||||
|
||||
|
||||
@router.get("/packages/{package_id}/asset")
|
||||
@@ -156,3 +162,5 @@ def get_asset(
|
||||
)
|
||||
except ProjectOperationError as exc:
|
||||
raise HTTPException(status_code=exc.status_code, detail=exc.detail) from exc
|
||||
except OSError as exc:
|
||||
raise HTTPException(status_code=503, detail=f"SEK storage unavailable: {exc}") from exc
|
||||
@@ -17,6 +17,11 @@ from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel, field_validator
|
||||
|
||||
from ..config import SETTINGS_PATH
|
||||
from ..services.auto_presets_catalog import (
|
||||
load_builtin_auto_presets,
|
||||
merge_auto_presets,
|
||||
normalize_auto_presets,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/api/settings", tags=["settings"])
|
||||
|
||||
@@ -34,6 +39,7 @@ _VALID_WHISPER_MODELS = {
|
||||
"large-v3-turbo", "turbo",
|
||||
}
|
||||
_VALID_KARAOKE_SOURCE_MODE = {"projects", "sek-folder"}
|
||||
_VALID_SINGUI_PROCESS_MODE = {"auto", "separate_only", "pipeline", "preset"}
|
||||
|
||||
# Default values for all known preference keys
|
||||
_DEFAULTS: dict[str, Any] = {
|
||||
@@ -48,6 +54,8 @@ _DEFAULTS: dict[str, Any] = {
|
||||
"auto_presets": [],
|
||||
"karaoke_source_mode": "sek-folder",
|
||||
"karaoke_sek_folder": "",
|
||||
"singui_process_mode": "auto",
|
||||
"singui_process_preset": "",
|
||||
"pixabay_api_key": "",
|
||||
"pixabay_cache_enabled": True,
|
||||
"pixabay_min_clip_duration": 15,
|
||||
@@ -82,6 +90,10 @@ def get_hf_token() -> Optional[str]:
|
||||
def get_settings():
|
||||
"""Return all non-sensitive settings. Redacted keys return a boolean flag."""
|
||||
data = _load()
|
||||
user_auto_presets = normalize_auto_presets(data.get("auto_presets"))
|
||||
builtin_auto_presets = load_builtin_auto_presets()
|
||||
effective_auto_presets = merge_auto_presets(builtin_auto_presets, user_auto_presets)
|
||||
|
||||
result = {}
|
||||
for k, v in data.items():
|
||||
if k in _REDACTED:
|
||||
@@ -94,6 +106,13 @@ def get_settings():
|
||||
# Check env var fallback
|
||||
env_val = os.environ.get(k.upper())
|
||||
result[f"{k}_set"] = bool(env_val)
|
||||
|
||||
# Preserve user-editable presets separately while exposing built-in and
|
||||
# effective merged presets for UI/runtime consumption.
|
||||
result["auto_presets"] = user_auto_presets
|
||||
result["auto_presets_user"] = user_auto_presets
|
||||
result["auto_presets_builtin"] = builtin_auto_presets
|
||||
result["auto_presets_effective"] = effective_auto_presets
|
||||
return result
|
||||
|
||||
|
||||
@@ -110,6 +129,8 @@ class SettingsPatch(BaseModel):
|
||||
auto_presets: Optional[list] = None
|
||||
karaoke_source_mode: Optional[str] = None
|
||||
karaoke_sek_folder: Optional[str] = None
|
||||
singui_process_mode: Optional[str] = None
|
||||
singui_process_preset: Optional[str] = None
|
||||
pixabay_api_key: Optional[str] = None
|
||||
pixabay_cache_enabled: Optional[bool] = None
|
||||
pixabay_min_clip_duration: Optional[float] = None
|
||||
@@ -142,6 +163,15 @@ class SettingsPatch(BaseModel):
|
||||
)
|
||||
return v
|
||||
|
||||
@field_validator("singui_process_mode")
|
||||
@classmethod
|
||||
def _validate_singui_process_mode(cls, v: Optional[str]) -> Optional[str]:
|
||||
if v is not None and v not in _VALID_SINGUI_PROCESS_MODE:
|
||||
raise ValueError(
|
||||
f"Unknown singui_process_mode '{v}'. Valid values: {sorted(_VALID_SINGUI_PROCESS_MODE)}"
|
||||
)
|
||||
return v
|
||||
|
||||
@field_validator("pixabay_min_clip_duration", "pixabay_max_clip_duration")
|
||||
@classmethod
|
||||
def _validate_pixabay_clip_duration(cls, v: Optional[float]) -> Optional[float]:
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
"""Git-managed Auto Pipeline preset catalog loading and merge helpers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
_DEFAULT_CATALOG_PATH = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "presets"
|
||||
/ "auto_pipeline"
|
||||
/ "default_presets.json"
|
||||
)
|
||||
|
||||
|
||||
def _normalize_preset(raw: Any) -> dict[str, Any] | None:
|
||||
if not isinstance(raw, dict):
|
||||
return None
|
||||
|
||||
name = str(raw.get("name") or "").strip()
|
||||
model = str(raw.get("model") or "").strip()
|
||||
fmt = str(raw.get("format") or "").strip()
|
||||
if not name or not model or not fmt:
|
||||
return None
|
||||
|
||||
inst_raw = raw.get("instrumentModels")
|
||||
instrument_models: dict[str, str] = {}
|
||||
if isinstance(inst_raw, dict):
|
||||
for k, v in inst_raw.items():
|
||||
if isinstance(k, str) and isinstance(v, str) and v.strip():
|
||||
instrument_models[k] = v.strip()
|
||||
|
||||
steps_raw = raw.get("steps")
|
||||
steps: list[dict[str, Any]] = []
|
||||
if isinstance(steps_raw, list):
|
||||
for step in steps_raw:
|
||||
if isinstance(step, dict) and step.get("type"):
|
||||
steps.append(step)
|
||||
|
||||
return {
|
||||
"name": name,
|
||||
"model": model,
|
||||
"format": fmt,
|
||||
"instrumentModels": instrument_models,
|
||||
"steps": steps,
|
||||
}
|
||||
|
||||
|
||||
def normalize_auto_presets(raw: Any) -> list[dict[str, Any]]:
|
||||
if not isinstance(raw, list):
|
||||
return []
|
||||
out: list[dict[str, Any]] = []
|
||||
for item in raw:
|
||||
preset = _normalize_preset(item)
|
||||
if preset:
|
||||
out.append(preset)
|
||||
return out
|
||||
|
||||
|
||||
def load_builtin_auto_presets() -> list[dict[str, Any]]:
|
||||
path_raw = os.environ.get("AUTO_PRESETS_CATALOG_PATH", "").strip()
|
||||
path = Path(path_raw) if path_raw else _DEFAULT_CATALOG_PATH
|
||||
|
||||
if not path.exists():
|
||||
log.info("Auto preset catalog not found at %s; continuing with no built-ins", path)
|
||||
return []
|
||||
|
||||
try:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
except Exception as exc:
|
||||
log.warning("Failed to parse auto preset catalog %s: %s", path, exc)
|
||||
return []
|
||||
|
||||
presets = normalize_auto_presets(payload)
|
||||
if not presets:
|
||||
log.warning("Auto preset catalog %s loaded but produced no valid presets", path)
|
||||
return presets
|
||||
|
||||
|
||||
def merge_auto_presets(
|
||||
builtin_presets: list[dict[str, Any]],
|
||||
user_presets: list[dict[str, Any]],
|
||||
) -> list[dict[str, Any]]:
|
||||
merged: dict[str, dict[str, Any]] = {}
|
||||
|
||||
for preset in normalize_auto_presets(builtin_presets):
|
||||
merged[preset["name"]] = preset
|
||||
|
||||
for preset in normalize_auto_presets(user_presets):
|
||||
merged[preset["name"]] = preset
|
||||
|
||||
return list(merged.values())
|
||||
@@ -66,8 +66,26 @@ def _resolve_folder_path(folder: str | None = None) -> Path:
|
||||
return _coerce_folder_path(configured)
|
||||
|
||||
|
||||
def _ensure_folder_accessible(base: Path) -> None:
|
||||
try:
|
||||
if not base.exists() or not base.is_dir():
|
||||
raise ProjectOperationError(404, f"SEK folder does not exist: {base}")
|
||||
except ProjectOperationError:
|
||||
raise
|
||||
except OSError as exc:
|
||||
raise ProjectOperationError(503, f"SEK folder is unavailable: {base} ({exc})") from exc
|
||||
|
||||
|
||||
def _safe_resolved_str(path: Path) -> str:
|
||||
try:
|
||||
return str(path.resolve())
|
||||
except OSError:
|
||||
return str(path)
|
||||
|
||||
|
||||
def _package_id(path: Path) -> str:
|
||||
return hashlib.sha1(str(path.resolve()).encode("utf-8")).hexdigest()[:16]
|
||||
stable = _safe_resolved_str(path)
|
||||
return hashlib.sha1(stable.encode("utf-8")).hexdigest()[:16]
|
||||
|
||||
|
||||
def _prune_expired_cache(cache: dict[str, tuple[float, Any]], now: float, max_entries: int) -> None:
|
||||
@@ -104,11 +122,22 @@ def _join_norm(*parts: Any) -> str:
|
||||
|
||||
def _iter_sek_paths(base: Path) -> list[Path]:
|
||||
# Recursive scan so exports organized as <sekfold>/<project>/<name>.sek are discovered.
|
||||
return sorted(
|
||||
[p for p in base.rglob("*.sek") if p.is_file()],
|
||||
key=lambda p: p.stat().st_mtime,
|
||||
reverse=True,
|
||||
)
|
||||
try:
|
||||
candidates = list(base.rglob("*.sek"))
|
||||
except OSError as exc:
|
||||
raise ProjectOperationError(503, f"Failed to scan SEK folder: {base} ({exc})") from exc
|
||||
|
||||
with_mtime: list[tuple[Path, float]] = []
|
||||
for path in candidates:
|
||||
try:
|
||||
if path.is_file():
|
||||
with_mtime.append((path, path.stat().st_mtime))
|
||||
except OSError:
|
||||
# Skip transient/unreadable entries instead of failing the whole listing.
|
||||
continue
|
||||
|
||||
with_mtime.sort(key=lambda item: item[1], reverse=True)
|
||||
return [p for p, _ in with_mtime]
|
||||
|
||||
|
||||
def _extract_package_match_keys(
|
||||
@@ -244,8 +273,7 @@ def _extract_duration_seconds_from_manifest(manifest: dict[str, Any] | None) ->
|
||||
def list_sek_packages(folder: str | None = None) -> dict[str, Any]:
|
||||
started = time.perf_counter()
|
||||
base = _resolve_folder_path(folder)
|
||||
if not base.exists() or not base.is_dir():
|
||||
raise ProjectOperationError(404, f"SEK folder does not exist: {base}")
|
||||
_ensure_folder_accessible(base)
|
||||
|
||||
packages: list[dict[str, Any]] = []
|
||||
scanned = _iter_sek_paths(base)
|
||||
@@ -253,6 +281,11 @@ def list_sek_packages(folder: str | None = None) -> dict[str, Any]:
|
||||
warnings: list[str] = []
|
||||
manifest: dict[str, Any] | None = None
|
||||
valid = True
|
||||
try:
|
||||
mtime = package_path.stat().st_mtime
|
||||
except OSError:
|
||||
mtime = 0.0
|
||||
warnings.append("Package mtime unavailable")
|
||||
|
||||
try:
|
||||
manifest = _read_manifest_cached(package_path)
|
||||
@@ -303,7 +336,7 @@ def list_sek_packages(folder: str | None = None) -> dict[str, Any]:
|
||||
"has_video": bool(video),
|
||||
"lyrics": lyrics if isinstance(lyrics, str) else "",
|
||||
"lyrics_track_count": len(lyrics_tracks) if isinstance(lyrics_tracks, dict) else 0,
|
||||
"mtime": package_path.stat().st_mtime,
|
||||
"mtime": mtime,
|
||||
"warnings": warnings,
|
||||
"meta_title": meta_title,
|
||||
"meta_artist": meta_artist,
|
||||
@@ -317,8 +350,8 @@ def list_sek_packages(folder: str | None = None) -> dict[str, Any]:
|
||||
)
|
||||
|
||||
return {
|
||||
"folder": str(base.resolve()),
|
||||
"scan_root": str(base.resolve()),
|
||||
"folder": _safe_resolved_str(base),
|
||||
"scan_root": _safe_resolved_str(base),
|
||||
"scan_count": len(scanned),
|
||||
"scan_ms": round((time.perf_counter() - started) * 1000, 2),
|
||||
"count": len(packages),
|
||||
@@ -328,8 +361,7 @@ def list_sek_packages(folder: str | None = None) -> dict[str, Any]:
|
||||
|
||||
def _resolve_package_by_id(package_id: str, folder: str | None = None) -> Path:
|
||||
base = _resolve_folder_path(folder)
|
||||
if not base.exists() or not base.is_dir():
|
||||
raise ProjectOperationError(404, f"SEK folder does not exist: {base}")
|
||||
_ensure_folder_accessible(base)
|
||||
|
||||
for package_path in _iter_sek_paths(base):
|
||||
if _package_id(package_path) == package_id:
|
||||
@@ -447,7 +479,7 @@ def _build_match_cache_key(folder: str | None, songs: list[dict[str, Any]]) -> s
|
||||
]
|
||||
songs_json = json.dumps(songs_payload, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
|
||||
songs_hash = hashlib.sha1(songs_json.encode("utf-8")).hexdigest()
|
||||
return f"{base.resolve()}|{songs_hash}"
|
||||
return f"{_safe_resolved_str(base)}|{songs_hash}"
|
||||
|
||||
|
||||
def match_subsonic_songs(
|
||||
|
||||
@@ -19,8 +19,8 @@
|
||||
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
|
||||
<meta name="apple-mobile-web-app-title" content="StemEngine" />
|
||||
<link rel="apple-touch-icon" href="/icon.svg" />
|
||||
<script type="module" crossorigin src="/assets/index-Cyxc7Uk9.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/assets/index-B34zlAct.css">
|
||||
<script type="module" crossorigin src="/assets/index-DzYnwFRo.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/assets/index-BCm86cqC.css">
|
||||
</head>
|
||||
<body>
|
||||
<div id="root"></div>
|
||||
|
||||
@@ -368,3 +368,59 @@ test('visualizer frame cadence stays healthy in windowed and fullscreen preview'
|
||||
|
||||
await page.locator('button[title="Exit fullscreen"]').click()
|
||||
})
|
||||
|
||||
test('switching to Tendrils mode mounts canvas and sets correct backdrop attribute', async ({ page }) => {
|
||||
await page.goto('/projects/p1')
|
||||
await page.getByRole('button', { name: /Live preview \(LRC\)/i }).click()
|
||||
await page.locator('button[title="Style picker"]').click()
|
||||
await page.locator('button[title="Use Tendrils visual renderer"]').click()
|
||||
|
||||
const backdrop = page.locator('[data-testid="visualizer-backdrop"]')
|
||||
await expect(backdrop).toHaveAttribute('data-visualizer-mode', 'tendrils')
|
||||
|
||||
// Tendrils lazy chunk renders a <canvas> element inside the backdrop
|
||||
await expect(backdrop.locator('canvas')).toBeVisible()
|
||||
})
|
||||
|
||||
test('switching to Spirit mode mounts canvas and sets correct backdrop attribute', async ({ page }) => {
|
||||
await page.goto('/projects/p1')
|
||||
await page.getByRole('button', { name: /Live preview \(LRC\)/i }).click()
|
||||
await page.locator('button[title="Style picker"]').click()
|
||||
await page.locator('button[title="Use Spirit visual renderer"]').click()
|
||||
|
||||
const backdrop = page.locator('[data-testid="visualizer-backdrop"]')
|
||||
await expect(backdrop).toHaveAttribute('data-visualizer-mode', 'spirit')
|
||||
|
||||
// Spirit lazy chunk renders a <canvas> element inside the backdrop
|
||||
await expect(backdrop.locator('canvas')).toBeVisible()
|
||||
})
|
||||
|
||||
test('tendrils and spirit inherit palette from artwork and preserve it across mode switches', async ({ page }) => {
|
||||
await page.context().route('**/api/projects/p1/cover-art', async (route) => {
|
||||
await route.fulfill({ status: 200, contentType: 'image/svg+xml', body: redSvg })
|
||||
})
|
||||
|
||||
await page.goto('/projects/p1')
|
||||
await page.getByRole('button', { name: /Live preview \(LRC\)/i }).click()
|
||||
await page.locator('button[title="Style picker"]').click()
|
||||
// Open picker once — all mode buttons are visible; switch Muon → Tendrils → Spirit
|
||||
await page.locator('button[title="Use Muon visual renderer"]').click()
|
||||
|
||||
const backdrop = page.locator('[data-testid="visualizer-backdrop"]')
|
||||
await expect(backdrop).toHaveAttribute('data-palette-source', 'runtime-artwork')
|
||||
const seedColor = await backdrop.getAttribute('data-palette-seed')
|
||||
const highlightColor = await backdrop.getAttribute('data-palette-highlight')
|
||||
expect(seedColor).toBeTruthy()
|
||||
|
||||
// Switch to Tendrils — palette must be preserved (picker still open)
|
||||
await page.locator('button[title="Use Tendrils visual renderer"]').click()
|
||||
await expect(backdrop).toHaveAttribute('data-visualizer-mode', 'tendrils')
|
||||
await expect(backdrop).toHaveAttribute('data-palette-seed', seedColor || '')
|
||||
await expect(backdrop).toHaveAttribute('data-palette-highlight', highlightColor || '')
|
||||
|
||||
// Switch to Spirit — palette must still be preserved
|
||||
await page.locator('button[title="Use Spirit visual renderer"]').click()
|
||||
await expect(backdrop).toHaveAttribute('data-visualizer-mode', 'spirit')
|
||||
await expect(backdrop).toHaveAttribute('data-palette-seed', seedColor || '')
|
||||
await expect(backdrop).toHaveAttribute('data-palette-highlight', highlightColor || '')
|
||||
})
|
||||
Generated
+73
@@ -25,6 +25,7 @@
|
||||
"react": "^18.2.0",
|
||||
"react-dom": "^18.2.0",
|
||||
"react-router-dom": "^6.30.3",
|
||||
"three": "^0.183.2",
|
||||
"tone": "^14.7.0",
|
||||
"two.js": "^0.8.23",
|
||||
"webgl-fluid": "^0.3.9"
|
||||
@@ -34,6 +35,7 @@
|
||||
"@types/qrcode": "^1.5.6",
|
||||
"@types/react": "^18.2.37",
|
||||
"@types/react-dom": "^18.2.15",
|
||||
"@types/three": "^0.183.1",
|
||||
"@typescript-eslint/eslint-plugin": "^8.26.1",
|
||||
"@typescript-eslint/parser": "^8.26.1",
|
||||
"@vitejs/plugin-react": "^4.2.0",
|
||||
@@ -331,6 +333,13 @@
|
||||
"node": ">=6.9.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@dimforge/rapier3d-compat": {
|
||||
"version": "0.12.0",
|
||||
"resolved": "https://registry.npmjs.org/@dimforge/rapier3d-compat/-/rapier3d-compat-0.12.0.tgz",
|
||||
"integrity": "sha512-uekIGetywIgopfD97oDL5PfeezkFpNhwlzlaEYNOA0N6ghdsOvh/HYjSMek5Q2O1PYvRSDFcqFVJl4r4ZBwOow==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0"
|
||||
},
|
||||
"node_modules/@esbuild/aix-ppc64": {
|
||||
"version": "0.21.5",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.21.5.tgz",
|
||||
@@ -1422,6 +1431,13 @@
|
||||
"midi-file": "^1.2.2"
|
||||
}
|
||||
},
|
||||
"node_modules/@tweenjs/tween.js": {
|
||||
"version": "23.1.3",
|
||||
"resolved": "https://registry.npmjs.org/@tweenjs/tween.js/-/tween.js-23.1.3.tgz",
|
||||
"integrity": "sha512-vJmvvwFxYuGnF2axRtPYocag6Clbb5YS7kLL+SO/TeVFzHqDIWrNKYtcsPMibjDx9O+bu+psAy9NKfWklassUA==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@types/babel__core": {
|
||||
"version": "7.20.5",
|
||||
"resolved": "https://registry.npmjs.org/@types/babel__core/-/babel__core-7.20.5.tgz",
|
||||
@@ -1514,6 +1530,36 @@
|
||||
"@types/react": "^18.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@types/stats.js": {
|
||||
"version": "0.17.4",
|
||||
"resolved": "https://registry.npmjs.org/@types/stats.js/-/stats.js-0.17.4.tgz",
|
||||
"integrity": "sha512-jIBvWWShCvlBqBNIZt0KAshWpvSjhkwkEu4ZUcASoAvhmrgAUI2t1dXrjSL4xXVLB4FznPrIsX3nKXFl/Dt4vA==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@types/three": {
|
||||
"version": "0.183.1",
|
||||
"resolved": "https://registry.npmjs.org/@types/three/-/three-0.183.1.tgz",
|
||||
"integrity": "sha512-f2Pu5Hrepfgavttdye3PsH5RWyY/AvdZQwIVhrc4uNtvF7nOWJacQKcoVJn0S4f0yYbmAE6AR+ve7xDcuYtMGw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@dimforge/rapier3d-compat": "~0.12.0",
|
||||
"@tweenjs/tween.js": "~23.1.3",
|
||||
"@types/stats.js": "*",
|
||||
"@types/webxr": ">=0.5.17",
|
||||
"@webgpu/types": "*",
|
||||
"fflate": "~0.8.2",
|
||||
"meshoptimizer": "~1.0.1"
|
||||
}
|
||||
},
|
||||
"node_modules/@types/webxr": {
|
||||
"version": "0.5.24",
|
||||
"resolved": "https://registry.npmjs.org/@types/webxr/-/webxr-0.5.24.tgz",
|
||||
"integrity": "sha512-h8fgEd/DpoS9CBrjEQXR+dIDraopAEfu4wYVNY2tEPwk60stPWhvZMf4Foo5FakuQ7HFZoa8WceaWFervK2Ovg==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@typescript-eslint/eslint-plugin": {
|
||||
"version": "8.57.0",
|
||||
"resolved": "https://registry.npmjs.org/@typescript-eslint/eslint-plugin/-/eslint-plugin-8.57.0.tgz",
|
||||
@@ -1787,6 +1833,13 @@
|
||||
"vite": "^4.2.0 || ^5.0.0 || ^6.0.0 || ^7.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@webgpu/types": {
|
||||
"version": "0.1.69",
|
||||
"resolved": "https://registry.npmjs.org/@webgpu/types/-/types-0.1.69.tgz",
|
||||
"integrity": "sha512-RPmm6kgRbI8e98zSD3RVACvnuktIja5+yLgDAkTmxLr90BEwdTXRQWNLF3ETTTyH/8mKhznZuN5AveXYFEsMGQ==",
|
||||
"dev": true,
|
||||
"license": "BSD-3-Clause"
|
||||
},
|
||||
"node_modules/abcjs": {
|
||||
"version": "6.6.2",
|
||||
"resolved": "https://registry.npmjs.org/abcjs/-/abcjs-6.6.2.tgz",
|
||||
@@ -2633,6 +2686,13 @@
|
||||
"reusify": "^1.0.4"
|
||||
}
|
||||
},
|
||||
"node_modules/fflate": {
|
||||
"version": "0.8.2",
|
||||
"resolved": "https://registry.npmjs.org/fflate/-/fflate-0.8.2.tgz",
|
||||
"integrity": "sha512-cPJU47OaAoCbg0pBvzsgpTPhmhqI5eJjh/JIu8tPj5q+T7iLvW/JAYUqmE7KOB4R1ZyEhzBaIQpQpardBF5z8A==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/file-entry-cache": {
|
||||
"version": "6.0.1",
|
||||
"resolved": "https://registry.npmjs.org/file-entry-cache/-/file-entry-cache-6.0.1.tgz",
|
||||
@@ -3181,6 +3241,13 @@
|
||||
"node": ">= 8"
|
||||
}
|
||||
},
|
||||
"node_modules/meshoptimizer": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/meshoptimizer/-/meshoptimizer-1.0.1.tgz",
|
||||
"integrity": "sha512-Vix+QlA1YYT3FwmBBZ+49cE5y/b+pRrcXKqGpS5ouh33d3lSp2PoTpCw19E0cKDFWalembrHnIaZetf27a+W2g==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/micromatch": {
|
||||
"version": "4.0.8",
|
||||
"resolved": "https://registry.npmjs.org/micromatch/-/micromatch-4.0.8.tgz",
|
||||
@@ -4153,6 +4220,12 @@
|
||||
"node": ">=0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/three": {
|
||||
"version": "0.183.2",
|
||||
"resolved": "https://registry.npmjs.org/three/-/three-0.183.2.tgz",
|
||||
"integrity": "sha512-di3BsL2FEQ1PA7Hcvn4fyJOlxRRgFYBpMTcyOgkwJIaDOdJMebEFPA+t98EvjuljDx4hNulAGwF6KIjtwI5jgQ==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/tinyglobby": {
|
||||
"version": "0.2.15",
|
||||
"resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.15.tgz",
|
||||
|
||||
@@ -34,6 +34,7 @@
|
||||
"react": "^18.2.0",
|
||||
"react-dom": "^18.2.0",
|
||||
"react-router-dom": "^6.30.3",
|
||||
"three": "^0.183.2",
|
||||
"tone": "^14.7.0",
|
||||
"two.js": "^0.8.23",
|
||||
"webgl-fluid": "^0.3.9"
|
||||
@@ -43,6 +44,7 @@
|
||||
"@types/qrcode": "^1.5.6",
|
||||
"@types/react": "^18.2.37",
|
||||
"@types/react-dom": "^18.2.15",
|
||||
"@types/three": "^0.183.1",
|
||||
"@typescript-eslint/eslint-plugin": "^8.26.1",
|
||||
"@typescript-eslint/parser": "^8.26.1",
|
||||
"@vitejs/plugin-react": "^4.2.0",
|
||||
|
||||
@@ -24,7 +24,9 @@ if (jsFiles.length === 0) {
|
||||
}
|
||||
|
||||
const maxSingleGzip = Number(process.env.BUNDLE_MAX_SINGLE_GZIP_BYTES || 180 * 1024)
|
||||
const maxTotalGzip = Number(process.env.BUNDLE_MAX_TOTAL_GZIP_BYTES || 520 * 1024)
|
||||
// Budget raised from 520 KB → 720 KB (2026-04-13) to accommodate Three.js
|
||||
// (~132 KB gzip) added as a lazy-loaded chunk for the Spirit visualizer.
|
||||
const maxTotalGzip = Number(process.env.BUNDLE_MAX_TOTAL_GZIP_BYTES || 720 * 1024)
|
||||
|
||||
const stats = jsFiles.map((name) => {
|
||||
const abs = join(assetsDir, name)
|
||||
|
||||
@@ -978,8 +978,13 @@ export interface AppSettings {
|
||||
auto_model: string
|
||||
auto_format: string
|
||||
auto_presets: unknown[]
|
||||
auto_presets_user?: unknown[]
|
||||
auto_presets_builtin?: unknown[]
|
||||
auto_presets_effective?: unknown[]
|
||||
karaoke_source_mode?: 'projects' | 'sek-folder'
|
||||
karaoke_sek_folder?: string
|
||||
singui_process_mode?: 'auto' | 'separate_only' | 'pipeline' | 'preset'
|
||||
singui_process_preset?: string
|
||||
pixabay_api_key_set?: boolean
|
||||
pixabay_cache_enabled?: boolean
|
||||
pixabay_min_clip_duration?: number
|
||||
|
||||
@@ -45,7 +45,7 @@ interface PreviewPlaybackPrefs {
|
||||
soloedStems?: Record<string, boolean>
|
||||
}
|
||||
|
||||
export type LyricsVisualStyle = 'scroll' | 'fade' | 'karaoke' | 'glow' | 'blur' | 'video' | 'muon' | 'fluid'
|
||||
export type LyricsVisualStyle = 'scroll' | 'fade' | 'karaoke' | 'glow' | 'blur' | 'video' | 'muon' | 'fluid' | 'tendrils' | 'spirit'
|
||||
type VideoStartPhase = 'idle' | 'priming' | 'playing'
|
||||
|
||||
type DeepPartial<T> = {
|
||||
@@ -95,6 +95,29 @@ interface VizSettings {
|
||||
// GUI Parameter Map additions
|
||||
color_speed: number
|
||||
}
|
||||
tendrils: {
|
||||
noise_weight: number
|
||||
noise_scale: number
|
||||
flow_weight: number
|
||||
flow_decay: number
|
||||
force_weight: number
|
||||
speed_limit: number
|
||||
damping: number
|
||||
hue_range: number
|
||||
mono_color: boolean
|
||||
visualization_preset: string
|
||||
}
|
||||
spirit: {
|
||||
speed: number
|
||||
die_speed: number
|
||||
radius: number
|
||||
curl_size: number
|
||||
attraction: number
|
||||
bloom_intensity: number
|
||||
bloom_threshold: number
|
||||
afterimage_damp: number
|
||||
visualization_preset: string
|
||||
}
|
||||
}
|
||||
|
||||
const VIZ_DEFAULTS: VizSettings = {
|
||||
@@ -134,6 +157,29 @@ const VIZ_DEFAULTS: VizSettings = {
|
||||
blob_life: 1.0,
|
||||
color_speed: 1.0,
|
||||
},
|
||||
tendrils: {
|
||||
noise_weight: 1.0,
|
||||
noise_scale: 1.0,
|
||||
flow_weight: 1.0,
|
||||
flow_decay: 1.0,
|
||||
force_weight: 1.0,
|
||||
speed_limit: 1.0,
|
||||
damping: 1.0,
|
||||
hue_range: 1.0,
|
||||
mono_color: false,
|
||||
visualization_preset: 'flow',
|
||||
},
|
||||
spirit: {
|
||||
speed: 1.0,
|
||||
die_speed: 1.0,
|
||||
radius: 1.5,
|
||||
curl_size: 0.4,
|
||||
attraction: 0.08,
|
||||
bloom_intensity: 1.5,
|
||||
bloom_threshold: 0.05,
|
||||
afterimage_damp: 0.88,
|
||||
visualization_preset: 'spirit',
|
||||
},
|
||||
}
|
||||
|
||||
const PERF_SIDEBAR_WIDTH = 280
|
||||
@@ -152,21 +198,19 @@ function deepMergeVizSettings(base: VizSettings, patch?: DeepPartial<VizSettings
|
||||
if (!patch) {
|
||||
return {
|
||||
...base,
|
||||
muon: { ...base.muon },
|
||||
fluid: { ...base.fluid },
|
||||
muon: { ...base.muon },
|
||||
fluid: { ...base.fluid },
|
||||
tendrils: { ...base.tendrils },
|
||||
spirit: { ...base.spirit },
|
||||
}
|
||||
}
|
||||
return {
|
||||
...base,
|
||||
...patch,
|
||||
muon: {
|
||||
...base.muon,
|
||||
...(patch.muon ?? {}),
|
||||
},
|
||||
fluid: {
|
||||
...base.fluid,
|
||||
...(patch.fluid ?? {}),
|
||||
},
|
||||
muon: { ...base.muon, ...(patch.muon ?? {}) },
|
||||
fluid: { ...base.fluid, ...(patch.fluid ?? {}) },
|
||||
tendrils: { ...base.tendrils, ...(patch.tendrils ?? {}) },
|
||||
spirit: { ...base.spirit, ...(patch.spirit ?? {}) },
|
||||
}
|
||||
}
|
||||
|
||||
@@ -205,15 +249,38 @@ function normalizeVizSettings(input: VizSettings): VizSettings {
|
||||
: 'hybrid',
|
||||
},
|
||||
fluid: {
|
||||
spawn_density: clampVizMultiplier(input.fluid.spawn_density, 0.25, 3.0),
|
||||
splat_force: clampVizMultiplier(input.fluid.splat_force, 0.25, 3.0),
|
||||
decay: clampVizMultiplier(input.fluid.decay, 0.25, 3.0),
|
||||
spawn_density: clampVizMultiplier(input.fluid.spawn_density, 0.25, 3.0),
|
||||
splat_force: clampVizMultiplier(input.fluid.splat_force, 0.25, 3.0),
|
||||
decay: clampVizMultiplier(input.fluid.decay, 0.25, 3.0),
|
||||
background_persistence: clampVizMultiplier(input.fluid.background_persistence, 0.2, 3.0),
|
||||
beat_burst_count: clampVizMultiplier(input.fluid.beat_burst_count, 0.25, 3.0),
|
||||
treble_pulse: clampVizMultiplier(input.fluid.treble_pulse, 0.0, 3.0),
|
||||
drag: clampVizMultiplier(input.fluid.drag, 0.5, 2.0),
|
||||
blob_life: clampVizMultiplier(input.fluid.blob_life, 0.25, 3.0),
|
||||
color_speed: clampVizMultiplier(input.fluid.color_speed, 0.0, 3.0),
|
||||
beat_burst_count: clampVizMultiplier(input.fluid.beat_burst_count, 0.25, 3.0),
|
||||
treble_pulse: clampVizMultiplier(input.fluid.treble_pulse, 0.0, 3.0),
|
||||
drag: clampVizMultiplier(input.fluid.drag, 0.5, 2.0),
|
||||
blob_life: clampVizMultiplier(input.fluid.blob_life, 0.25, 3.0),
|
||||
color_speed: clampVizMultiplier(input.fluid.color_speed, 0.0, 3.0),
|
||||
},
|
||||
tendrils: {
|
||||
noise_weight: clampVizMultiplier(input.tendrils.noise_weight, 0.0, 3.0),
|
||||
noise_scale: clampVizMultiplier(input.tendrils.noise_scale, 0.0, 3.0),
|
||||
flow_weight: clampVizMultiplier(input.tendrils.flow_weight, 0.0, 3.0),
|
||||
flow_decay: clampVizMultiplier(input.tendrils.flow_decay, 0.0, 3.0),
|
||||
force_weight: clampVizMultiplier(input.tendrils.force_weight, 0.0, 3.0),
|
||||
speed_limit: clampVizMultiplier(input.tendrils.speed_limit, 0.0, 3.0),
|
||||
damping: clampVizMultiplier(input.tendrils.damping, 0.0, 3.0),
|
||||
hue_range: clampVizMultiplier(input.tendrils.hue_range, 0.0, 3.0),
|
||||
mono_color: Boolean(input.tendrils.mono_color),
|
||||
visualization_preset: typeof input.tendrils.visualization_preset === 'string' ? input.tendrils.visualization_preset : 'flow',
|
||||
},
|
||||
spirit: {
|
||||
speed: clampVizMultiplier(input.spirit.speed, 0.1, 6.0),
|
||||
die_speed: clampVizMultiplier(input.spirit.die_speed, 0.001, 0.12),
|
||||
radius: clampVizMultiplier(input.spirit.radius, 0.5, 5.0),
|
||||
curl_size: clampVizMultiplier(input.spirit.curl_size, 0.05, 3.0),
|
||||
attraction: Math.max(-2, Math.min(2, Number.isFinite(input.spirit.attraction) ? input.spirit.attraction : 0.08)),
|
||||
bloom_intensity: clampVizMultiplier(input.spirit.bloom_intensity, 0.0, 4.0),
|
||||
bloom_threshold: clampVizMultiplier(input.spirit.bloom_threshold, 0.0, 1.0),
|
||||
afterimage_damp: Math.max(0.60, Math.min(0.96, Number.isFinite(input.spirit.afterimage_damp) ? input.spirit.afterimage_damp : 0.88)),
|
||||
visualization_preset: typeof input.spirit.visualization_preset === 'string' ? input.spirit.visualization_preset : 'spirit',
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -240,7 +307,7 @@ interface Props {
|
||||
/** Optional SEK manifest dynamic palette (manifest.metadata.dynamic_palette). */
|
||||
sekDynamicPalette?: SekDynamicPalette | null
|
||||
/** Optional SEK visualizer mode preference (manifest.visualizer.mode). */
|
||||
sekVisualizerMode?: 'muon' | 'fluid' | 'auto' | null
|
||||
sekVisualizerMode?: 'muon' | 'fluid' | 'tendrils' | 'spirit' | 'auto' | null
|
||||
/** Optional SEK visualizer settings (manifest.visualizer.settings). */
|
||||
sekVisualizerSettings?: {
|
||||
quality?: string
|
||||
@@ -279,6 +346,29 @@ interface Props {
|
||||
blob_life?: number
|
||||
color_speed?: number
|
||||
}
|
||||
tendrils?: {
|
||||
noise_weight?: number
|
||||
noise_scale?: number
|
||||
flow_weight?: number
|
||||
flow_decay?: number
|
||||
force_weight?: number
|
||||
speed_limit?: number
|
||||
damping?: number
|
||||
hue_range?: number
|
||||
mono_color?: boolean
|
||||
visualization_preset?: string
|
||||
}
|
||||
spirit?: {
|
||||
speed?: number
|
||||
die_speed?: number
|
||||
radius?: number
|
||||
curl_size?: number
|
||||
attraction?: number
|
||||
bloom_intensity?: number
|
||||
bloom_threshold?: number
|
||||
afterimage_damp?: number
|
||||
visualization_preset?: string
|
||||
}
|
||||
} | null
|
||||
/** Artist name from source audio metadata. */
|
||||
artist?: string
|
||||
@@ -310,13 +400,6 @@ function rankLyricsTrackKey(key: string): number {
|
||||
return 10
|
||||
}
|
||||
|
||||
function labelLyricsTrackKey(key: string): string {
|
||||
if (key === 'leading') return 'Leading'
|
||||
if (key === 'backing') return 'Backing'
|
||||
if (key === 'default') return 'Main'
|
||||
return key.replace(/\b\w/g, (m) => m.toUpperCase())
|
||||
}
|
||||
|
||||
// ─── Visual style definitions ─────────────────────────────────────────────────
|
||||
|
||||
const FONT_OPTIONS = [
|
||||
@@ -1609,26 +1692,26 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, ana
|
||||
const [visualStyle, setVisualStyle] = useState<LyricsVisualStyle>(() => {
|
||||
const saved = localStorage.getItem('stemengine-lyric-visual-style') as LyricsVisualStyle | null
|
||||
const hasRealVideo = !!(videoUrl || (videoPlaylist && videoPlaylist.length > 0))
|
||||
const visualizerDefault = defaultStyle === 'muon' || defaultStyle === 'fluid'
|
||||
const visualizerDefault = defaultStyle === 'muon' || defaultStyle === 'fluid' || defaultStyle === 'tendrils' || defaultStyle === 'spirit'
|
||||
|
||||
if (sekVisualizerMode === 'muon' || sekVisualizerMode === 'fluid') return sekVisualizerMode
|
||||
if (sekVisualizerMode === 'muon' || sekVisualizerMode === 'fluid' || sekVisualizerMode === 'tendrils' || sekVisualizerMode === 'spirit') return sekVisualizerMode
|
||||
|
||||
// For SEK/no-video sessions we default to a visualizer. Keep persisted user
|
||||
// choice only when it is also a visualizer mode.
|
||||
if (visualizerDefault) {
|
||||
if (saved === 'muon' || saved === 'fluid') return saved
|
||||
if (saved === 'muon' || saved === 'fluid' || saved === 'tendrils' || saved === 'spirit') return saved
|
||||
return defaultStyle
|
||||
}
|
||||
|
||||
// Only restore saved 'video' style when a real video source is actually available;
|
||||
// if Pixabay stock video is disabled and there's no embedded video, fall through to defaults.
|
||||
if (saved && (saved !== 'video' || hasRealVideo) && ['scroll', 'fade', 'karaoke', 'glow', 'blur', 'video', 'muon', 'fluid'].includes(saved)) return saved
|
||||
if (saved && (saved !== 'video' || hasRealVideo) && ['scroll', 'fade', 'karaoke', 'glow', 'blur', 'video', 'muon', 'fluid', 'tendrils', 'spirit'].includes(saved)) return saved
|
||||
return defaultStyle ?? (hasElrc ? 'karaoke' : 'scroll')
|
||||
})
|
||||
const enforcedVisualizerMode: LyricsVisualStyle | null =
|
||||
(sekVisualizerMode === 'muon' || sekVisualizerMode === 'fluid')
|
||||
(sekVisualizerMode === 'muon' || sekVisualizerMode === 'fluid' || sekVisualizerMode === 'tendrils' || sekVisualizerMode === 'spirit')
|
||||
? sekVisualizerMode
|
||||
: ((defaultStyle === 'muon' || defaultStyle === 'fluid') ? defaultStyle : null)
|
||||
: ((defaultStyle === 'muon' || defaultStyle === 'fluid' || defaultStyle === 'tendrils' || defaultStyle === 'spirit') ? defaultStyle : null)
|
||||
const userOverrodeVisualizerRef = useRef(false)
|
||||
const [showPerfSidebar, setShowPerfSidebar] = useState(false)
|
||||
const manifestVizSettings = useMemo<VizSettings | null>(() => {
|
||||
@@ -1637,6 +1720,8 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, ana
|
||||
const fps = sekVisualizerSettings.fps_cap
|
||||
const sm = sekVisualizerSettings.muon
|
||||
const sf = sekVisualizerSettings.fluid
|
||||
const st = sekVisualizerSettings.tendrils
|
||||
const ss = sekVisualizerSettings.spirit
|
||||
const patch: DeepPartial<VizSettings> = {
|
||||
quality: quality === 'low' || quality === 'balanced' || quality === 'high' ? quality : undefined,
|
||||
fps_cap: fps === 30 || fps === 60 || fps === 120 ? fps : undefined,
|
||||
@@ -1674,6 +1759,29 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, ana
|
||||
blob_life: sf?.blob_life,
|
||||
color_speed: sf?.color_speed,
|
||||
},
|
||||
tendrils: {
|
||||
noise_weight: st?.noise_weight,
|
||||
noise_scale: st?.noise_scale,
|
||||
flow_weight: st?.flow_weight,
|
||||
flow_decay: st?.flow_decay,
|
||||
force_weight: st?.force_weight,
|
||||
speed_limit: st?.speed_limit,
|
||||
damping: st?.damping,
|
||||
hue_range: st?.hue_range,
|
||||
mono_color: st?.mono_color,
|
||||
visualization_preset: st?.visualization_preset,
|
||||
},
|
||||
spirit: {
|
||||
speed: ss?.speed,
|
||||
die_speed: ss?.die_speed,
|
||||
radius: ss?.radius,
|
||||
curl_size: ss?.curl_size,
|
||||
attraction: ss?.attraction,
|
||||
bloom_intensity: ss?.bloom_intensity,
|
||||
bloom_threshold: ss?.bloom_threshold,
|
||||
afterimage_damp: ss?.afterimage_damp,
|
||||
visualization_preset: ss?.visualization_preset,
|
||||
},
|
||||
}
|
||||
return normalizeVizSettings(deepMergeVizSettings(VIZ_DEFAULTS, patch))
|
||||
}, [sekVisualizerSettings])
|
||||
@@ -1750,7 +1858,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, ana
|
||||
coverArtUrl,
|
||||
thumbnailUrl,
|
||||
})
|
||||
const isVisualizerMode = visualStyle === 'muon' || visualStyle === 'fluid'
|
||||
const isVisualizerMode = visualStyle === 'muon' || visualStyle === 'fluid' || visualStyle === 'tendrils' || visualStyle === 'spirit'
|
||||
const resolvedVideoPlaylist = useMemo(
|
||||
() => (videoPlaylist ?? [])
|
||||
.filter((item) => typeof item?.url === 'string' && item.url.trim())
|
||||
@@ -3195,45 +3303,10 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, ana
|
||||
position: 'relative',
|
||||
} as any}
|
||||
>
|
||||
{lyricsTrackKeys.length > 1 && (
|
||||
<div
|
||||
style={{
|
||||
position: 'absolute',
|
||||
top: performanceMode ? 84 : 10,
|
||||
right: 10,
|
||||
zIndex: 3,
|
||||
display: 'flex',
|
||||
gap: 6,
|
||||
flexWrap: 'wrap',
|
||||
justifyContent: 'flex-end',
|
||||
}}
|
||||
>
|
||||
{lyricsTrackKeys.map((key) => (
|
||||
<button
|
||||
key={key}
|
||||
type="button"
|
||||
onClick={() => setSelectedLyricsTrack(key)}
|
||||
style={{
|
||||
border: '1px solid rgba(255,255,255,0.18)',
|
||||
background: selectedLyricsTrack === key ? 'rgba(139,92,246,0.9)' : 'rgba(15,23,42,0.55)',
|
||||
color: '#fff',
|
||||
borderRadius: 999,
|
||||
padding: '4px 10px',
|
||||
fontSize: 11,
|
||||
fontWeight: 600,
|
||||
backdropFilter: 'blur(6px)',
|
||||
}}
|
||||
title={`Show ${labelLyricsTrackKey(key)} lyrics`}
|
||||
>
|
||||
{labelLyricsTrackKey(key)}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
{isVisualizerMode && (
|
||||
<div style={{ position: 'absolute', inset: 0, zIndex: 0, pointerEvents: 'none' }}>
|
||||
<VisualizerBackdrop
|
||||
mode={visualStyle === 'muon' ? 'muon' : visualStyle === 'fluid' ? 'fluid' : null}
|
||||
mode={visualStyle === 'muon' ? 'muon' : visualStyle === 'fluid' ? 'fluid' : visualStyle === 'tendrils' ? 'tendrils' : visualStyle === 'spirit' ? 'spirit' : null}
|
||||
featuresRef={featuresRef}
|
||||
paletteRef={paletteRef}
|
||||
width={visualizerSize.width}
|
||||
@@ -3592,21 +3665,6 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, ana
|
||||
{hasElrc && (
|
||||
<span className="ml-2 shrink-0 text-[10px] ui-text-muted">ELRC</span>
|
||||
)}
|
||||
{lyricsTrackKeys.length > 1 && (
|
||||
<div className="ml-2 flex items-center gap-1">
|
||||
{lyricsTrackKeys.map((key) => (
|
||||
<button
|
||||
key={key}
|
||||
type="button"
|
||||
onClick={() => setSelectedLyricsTrack(key)}
|
||||
className={`shrink-0 rounded-md px-2 py-1 text-[10px] font-medium transition-colors ${selectedLyricsTrack === key ? 'bg-violet-500 text-white' : isFullscreen ? 'text-slate-300 hover:bg-white/10' : 'bg-slate-200/70 text-slate-700 hover:bg-slate-300/70 dark:bg-slate-700/60 dark:text-slate-200 dark:hover:bg-slate-600/70'}`}
|
||||
title={`Show ${labelLyricsTrackKey(key)} lyrics`}
|
||||
>
|
||||
{labelLyricsTrackKey(key)}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
<div className="ml-auto flex gap-0.5 shrink-0 pl-1">
|
||||
<button
|
||||
onClick={() => { setShowStylePicker(p => !p); setShowEditor(false) }}
|
||||
@@ -3651,6 +3709,8 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, ana
|
||||
...((videoUrl || resolvedVideoPlaylist.length > 0) ? [{ value: 'video' as LyricsVisualStyle, label: 'Video' }] : []),
|
||||
{ value: 'muon', label: 'Muon' },
|
||||
{ value: 'fluid', label: 'Fluid' },
|
||||
{ value: 'tendrils', label: 'Tendrils' },
|
||||
{ value: 'spirit', label: 'Spirit' },
|
||||
] as Array<{ value: LyricsVisualStyle; label: string }>).map((mode) => (
|
||||
<button
|
||||
key={mode.value}
|
||||
@@ -3864,6 +3924,10 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, ana
|
||||
vizSettings={liveVizSettings}
|
||||
manifestVizSettings={manifestVizSettings}
|
||||
patchVizSettings={patchVizSettings}
|
||||
lyricsTrackKeys={lyricsTrackKeys}
|
||||
selectedLyricsTrack={selectedLyricsTrack}
|
||||
onSelectLyricsTrack={setSelectedLyricsTrack}
|
||||
hasElrc={hasElrc}
|
||||
onClose={() => setShowPerfSidebar(false)}
|
||||
/>
|
||||
)}
|
||||
@@ -5111,7 +5175,7 @@ function LyricsCanvas({ lines, currentTimeRef, visualStyle, style, speakerColorF
|
||||
return () => { cancelled = true }
|
||||
}, [lines, currentTimeRef])
|
||||
|
||||
const isVisualizerVisualStyle = visualStyle === 'muon' || visualStyle === 'fluid'
|
||||
const isVisualizerVisualStyle = visualStyle === 'muon' || visualStyle === 'fluid' || visualStyle === 'tendrils' || visualStyle === 'spirit'
|
||||
|
||||
return (
|
||||
<div
|
||||
|
||||
@@ -69,6 +69,29 @@ interface VizSettings {
|
||||
blob_life: number
|
||||
color_speed: number
|
||||
}
|
||||
tendrils: {
|
||||
noise_weight: number
|
||||
noise_scale: number
|
||||
flow_weight: number
|
||||
flow_decay: number
|
||||
force_weight: number
|
||||
speed_limit: number
|
||||
damping: number
|
||||
hue_range: number
|
||||
mono_color: boolean
|
||||
visualization_preset: string
|
||||
}
|
||||
spirit: {
|
||||
speed: number
|
||||
die_speed: number
|
||||
radius: number
|
||||
curl_size: number
|
||||
attraction: number
|
||||
bloom_intensity: number
|
||||
bloom_threshold: number
|
||||
afterimage_damp: number
|
||||
visualization_preset: string
|
||||
}
|
||||
}
|
||||
|
||||
const VIZ_DEFAULTS: VizSettings = {
|
||||
@@ -108,6 +131,29 @@ const VIZ_DEFAULTS: VizSettings = {
|
||||
blob_life: 1.0,
|
||||
color_speed: 1.0,
|
||||
},
|
||||
tendrils: {
|
||||
noise_weight: 1.0,
|
||||
noise_scale: 1.0,
|
||||
flow_weight: 1.0,
|
||||
flow_decay: 1.0,
|
||||
force_weight: 1.0,
|
||||
speed_limit: 1.0,
|
||||
damping: 1.0,
|
||||
hue_range: 1.0,
|
||||
mono_color: false,
|
||||
visualization_preset: 'flow',
|
||||
},
|
||||
spirit: {
|
||||
speed: 1.0,
|
||||
die_speed: 1.0,
|
||||
radius: 1.5,
|
||||
curl_size: 0.4,
|
||||
attraction: 0.08,
|
||||
bloom_intensity: 1.5,
|
||||
bloom_threshold: 0.05,
|
||||
afterimage_damp: 0.88,
|
||||
visualization_preset: 'spirit',
|
||||
},
|
||||
}
|
||||
|
||||
// Named muon presets — applied to vizSettings.muon only (leaves other keys untouched).
|
||||
@@ -214,6 +260,10 @@ interface PerformanceSidebarProps {
|
||||
vizSettings: VizSettings
|
||||
manifestVizSettings: VizSettings | null
|
||||
patchVizSettings: (patch: DeepPartial<VizSettings>) => void
|
||||
lyricsTrackKeys: string[]
|
||||
selectedLyricsTrack: string
|
||||
onSelectLyricsTrack: (key: string) => void
|
||||
hasElrc: boolean
|
||||
onClose: () => void
|
||||
}
|
||||
|
||||
@@ -252,6 +302,10 @@ export default function PerformanceSidebar({
|
||||
vizSettings,
|
||||
manifestVizSettings,
|
||||
patchVizSettings,
|
||||
lyricsTrackKeys,
|
||||
selectedLyricsTrack,
|
||||
onSelectLyricsTrack,
|
||||
hasElrc,
|
||||
onClose,
|
||||
}: PerformanceSidebarProps) {
|
||||
const autoHideTimerRef = useRef<number | null>(null)
|
||||
@@ -275,8 +329,8 @@ export default function PerformanceSidebar({
|
||||
}, [scheduleAutoHide])
|
||||
|
||||
const activeOffset = getLayoutOffset(style)
|
||||
const isVisualizerMode = visualStyle === 'muon' || visualStyle === 'fluid'
|
||||
const isStaticMode = visualStyle !== 'muon' && visualStyle !== 'fluid' && visualStyle !== 'video'
|
||||
const isVisualizerMode = visualStyle === 'muon' || visualStyle === 'fluid' || visualStyle === 'tendrils' || visualStyle === 'spirit'
|
||||
const isStaticMode = visualStyle !== 'muon' && visualStyle !== 'fluid' && visualStyle !== 'tendrils' && visualStyle !== 'spirit' && visualStyle !== 'video'
|
||||
|
||||
const patchOffset = (axis: 'x' | 'y', value: number) => {
|
||||
if (style.videoDisplayMode === 'line') {
|
||||
@@ -349,74 +403,141 @@ export default function PerformanceSidebar({
|
||||
|
||||
<div className="px-3 py-3 space-y-4">
|
||||
|
||||
{/* ── BACKGROUND ── */}
|
||||
<section className="space-y-2">
|
||||
<div className="text-xs font-semibold tracking-wide text-slate-200">BACKGROUND</div>
|
||||
<div className={isNarrow ? 'flex flex-wrap gap-2' : 'grid grid-cols-2 gap-2'}>
|
||||
{[
|
||||
{ key: 'static', label: 'Static', value: 'karaoke' as LyricsVisualStyle, active: isStaticMode },
|
||||
{ key: 'muon', label: 'Muon', value: 'muon' as LyricsVisualStyle, active: visualStyle === 'muon' },
|
||||
{ key: 'fluid', label: 'Fluid', value: 'fluid' as LyricsVisualStyle, active: visualStyle === 'fluid' },
|
||||
...(hasVideo ? [{ key: 'video', label: 'Video', value: 'video' as LyricsVisualStyle, active: visualStyle === 'video' }] : []),
|
||||
].map((mode) => (
|
||||
<button
|
||||
key={mode.key}
|
||||
type="button"
|
||||
onClick={() => setVisualStyle(mode.value)}
|
||||
className={`rounded-md px-2 py-2 text-xs border transition-colors ${isNarrow ? 'min-w-[70px] flex-1' : ''} ${mode.active ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
|
||||
>
|
||||
{mode.label}
|
||||
</button>
|
||||
))}
|
||||
{/* ── LYRICS TRACKS ── */}
|
||||
{lyricsTrackKeys.length > 1 && (
|
||||
<details className="rounded-md border border-white/10 bg-white/[0.02]" open>
|
||||
<summary className="cursor-pointer select-none px-2.5 py-2 text-xs font-semibold tracking-wide text-slate-200">
|
||||
LYRICS TRACKS
|
||||
</summary>
|
||||
<div className="px-2.5 pb-2.5 pt-1">
|
||||
<div className={isNarrow ? 'flex flex-wrap gap-2' : 'grid grid-cols-2 gap-2'}>
|
||||
{lyricsTrackKeys.map((key) => (
|
||||
<button
|
||||
key={key}
|
||||
type="button"
|
||||
onClick={() => onSelectLyricsTrack(key)}
|
||||
className={`rounded-md px-2 py-2 text-xs border transition-colors ${isNarrow ? 'flex-1 min-w-[70px]' : ''} ${
|
||||
selectedLyricsTrack === key
|
||||
? 'border-violet-400 bg-violet-500/20 text-white'
|
||||
: 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'
|
||||
}`}
|
||||
title={`Show ${key.charAt(0).toUpperCase() + key.slice(1)} lyrics`}
|
||||
>
|
||||
{key.charAt(0).toUpperCase() + key.slice(1)}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
</details>
|
||||
)}
|
||||
|
||||
{/* ── LINE DISPLAY ── */}
|
||||
<details className="rounded-md border border-white/10 bg-white/[0.02]" open>
|
||||
<summary className="cursor-pointer select-none px-2.5 py-2 text-xs font-semibold tracking-wide text-slate-200">
|
||||
DISPLAY
|
||||
</summary>
|
||||
<div className="px-2.5 pb-2.5 pt-1">
|
||||
<div className={isNarrow ? 'flex flex-wrap gap-2' : 'grid grid-cols-2 gap-2'}>
|
||||
{(['scroll', 'line', 'word', 'syllable'] as const).map((mode) => (
|
||||
<button
|
||||
key={mode}
|
||||
type="button"
|
||||
disabled={(mode === 'word' || mode === 'syllable') && !hasElrc}
|
||||
onClick={() => patchStyle({ videoDisplayMode: mode })}
|
||||
title={(mode === 'word' || mode === 'syllable') && !hasElrc ? 'No word-timing data available' : `Show ${mode} display`}
|
||||
className={`rounded-md px-2 py-2 text-xs border transition-colors capitalize ${isNarrow ? 'flex-1 min-w-[70px]' : ''} ${
|
||||
style.videoDisplayMode === mode
|
||||
? 'border-violet-400 bg-violet-500/20 text-white'
|
||||
: 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'
|
||||
} disabled:opacity-40 disabled:cursor-not-allowed`}
|
||||
>
|
||||
{mode}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
</details>
|
||||
|
||||
{/* ── BACKGROUND ── */}
|
||||
<details className="rounded-md border border-white/10 bg-white/[0.02]" open>
|
||||
<summary className="cursor-pointer select-none px-2.5 py-2 text-xs font-semibold tracking-wide text-slate-200">
|
||||
BACKGROUND
|
||||
</summary>
|
||||
<div className="px-2.5 pb-2.5 pt-1">
|
||||
<div className={isNarrow ? 'flex flex-wrap gap-2' : 'grid grid-cols-2 gap-2'}>
|
||||
{[
|
||||
{ key: 'static', label: 'Static', value: 'karaoke' as LyricsVisualStyle, active: isStaticMode },
|
||||
{ key: 'muon', label: 'Muon', value: 'muon' as LyricsVisualStyle, active: visualStyle === 'muon' },
|
||||
{ key: 'fluid', label: 'Fluid', value: 'fluid' as LyricsVisualStyle, active: visualStyle === 'fluid' },
|
||||
{ key: 'tendrils', label: 'Tendrils', value: 'tendrils' as LyricsVisualStyle, active: visualStyle === 'tendrils' },
|
||||
{ key: 'spirit', label: 'Spirit', value: 'spirit' as LyricsVisualStyle, active: visualStyle === 'spirit' },
|
||||
...(hasVideo ? [{ key: 'video', label: 'Video', value: 'video' as LyricsVisualStyle, active: visualStyle === 'video' }] : []),
|
||||
].map((mode) => (
|
||||
<button
|
||||
key={mode.key}
|
||||
type="button"
|
||||
onClick={() => setVisualStyle(mode.value)}
|
||||
className={`rounded-md px-2 py-2 text-xs border transition-colors ${isNarrow ? 'min-w-[70px] flex-1' : ''} ${mode.active ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
|
||||
>
|
||||
{mode.label}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
</details>
|
||||
|
||||
{/* ── STEMS ── */}
|
||||
<section className="space-y-2">
|
||||
<div className="text-xs font-semibold tracking-wide text-slate-200">STEMS</div>
|
||||
<div className="space-y-2">
|
||||
{stems?.length ? stems.map((stem) => (
|
||||
<div key={stem.name} className="space-y-1.5">
|
||||
<div className="flex items-center gap-2">
|
||||
<span className="text-xs font-medium text-slate-300 flex-1 truncate">{stem.label}</span>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => toggleStemMute(stem.name)}
|
||||
className={`w-6 h-6 text-[10px] rounded ${mutedStems[stem.name] ? 'bg-red-500 text-white' : 'bg-white/10 text-slate-300'}`}
|
||||
>
|
||||
M
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => toggleStemSolo(stem.name)}
|
||||
className={`w-6 h-6 text-[10px] rounded ${soloedStems[stem.name] ? 'bg-yellow-300 text-black' : 'bg-white/10 text-slate-300'}`}
|
||||
>
|
||||
S
|
||||
</button>
|
||||
<details className="rounded-md border border-white/10 bg-white/[0.02]" open>
|
||||
<summary className="cursor-pointer select-none px-2.5 py-2 text-xs font-semibold tracking-wide text-slate-200">
|
||||
STEMS
|
||||
</summary>
|
||||
<div className="px-2.5 pb-2.5 pt-1">
|
||||
<div className="space-y-2">
|
||||
{stems?.length ? stems.map((stem) => (
|
||||
<div key={stem.name} className="space-y-1.5">
|
||||
<div className="flex items-center gap-2">
|
||||
<span className="text-xs font-medium text-slate-300 flex-1 truncate">{stem.label}</span>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => toggleStemMute(stem.name)}
|
||||
className={`w-6 h-6 text-[10px] rounded ${mutedStems[stem.name] ? 'bg-red-500 text-white' : 'bg-white/10 text-slate-300'}`}
|
||||
>
|
||||
M
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => toggleStemSolo(stem.name)}
|
||||
className={`w-6 h-6 text-[10px] rounded ${soloedStems[stem.name] ? 'bg-yellow-300 text-black' : 'bg-white/10 text-slate-300'}`}
|
||||
>
|
||||
S
|
||||
</button>
|
||||
</div>
|
||||
<div className="flex items-center gap-2">
|
||||
<input
|
||||
type="range"
|
||||
min={0}
|
||||
max={2}
|
||||
step={0.05}
|
||||
value={gains[stem.name] ?? 1}
|
||||
onChange={(e) => setStemGain(stem.name, parseFloat(e.target.value))}
|
||||
className="w-full h-1.5 accent-brand-500"
|
||||
/>
|
||||
<span className="text-[10px] tabular-nums text-slate-400 w-11 text-right">{Math.round((gains[stem.name] ?? 1) * 100)}%</span>
|
||||
</div>
|
||||
</div>
|
||||
<div className="flex items-center gap-2">
|
||||
<input
|
||||
type="range"
|
||||
min={0}
|
||||
max={2}
|
||||
step={0.05}
|
||||
value={gains[stem.name] ?? 1}
|
||||
onChange={(e) => setStemGain(stem.name, parseFloat(e.target.value))}
|
||||
className="w-full h-1.5 accent-brand-500"
|
||||
/>
|
||||
<span className="text-[10px] tabular-nums text-slate-400 w-11 text-right">{Math.round((gains[stem.name] ?? 1) * 100)}%</span>
|
||||
</div>
|
||||
</div>
|
||||
)) : (
|
||||
<p className="text-[11px] text-slate-400">No stems — single track mode</p>
|
||||
)}
|
||||
)) : (
|
||||
<p className="text-[11px] text-slate-400">No stems — single track mode</p>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
</details>
|
||||
|
||||
{/* ── LYRICS ── */}
|
||||
<section className="space-y-2">
|
||||
<div className="text-xs font-semibold tracking-wide text-slate-200">LYRICS</div>
|
||||
<div className="space-y-2.5">
|
||||
<details className="rounded-md border border-white/10 bg-white/[0.02]">
|
||||
<summary className="cursor-pointer select-none px-2.5 py-2 text-xs font-semibold tracking-wide text-slate-200">
|
||||
LYRICS STYLE
|
||||
</summary>
|
||||
<div className="space-y-2.5 px-2.5 pb-2.5 pt-1">
|
||||
<div className="flex items-center gap-2">
|
||||
<select
|
||||
value={style.fontFamily}
|
||||
@@ -483,12 +604,14 @@ export default function PerformanceSidebar({
|
||||
<input type="range" min={-50} max={50} step={1} value={activeOffset.y} onChange={(e) => patchOffset('y', parseInt(e.target.value, 10))} className="w-full h-1.5 accent-brand-500" />
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
</details>
|
||||
|
||||
{/* ── VISUALIZER ── */}
|
||||
<section className="space-y-2">
|
||||
<div className="text-xs font-semibold tracking-wide text-slate-200">VISUALIZER</div>
|
||||
<div className="space-y-2.5">
|
||||
<details className="rounded-md border border-white/10 bg-white/[0.02]">
|
||||
<summary className="cursor-pointer select-none px-2.5 py-2 text-xs font-semibold tracking-wide text-slate-200">
|
||||
VISUALIZER
|
||||
</summary>
|
||||
<div className="space-y-2.5 px-2.5 pb-2.5 pt-1">
|
||||
|
||||
{/* Quality */}
|
||||
<div className="space-y-1.5">
|
||||
@@ -520,7 +643,7 @@ export default function PerformanceSidebar({
|
||||
|
||||
{!isVisualizerMode && (
|
||||
<p className="text-[11px] text-slate-400">
|
||||
Switch background to Muon or Fluid to access mode-specific controls.
|
||||
Switch background to Muon, Fluid, Tendrils, or Spirit to access mode-specific controls.
|
||||
</p>
|
||||
)}
|
||||
|
||||
@@ -662,6 +785,81 @@ export default function PerformanceSidebar({
|
||||
</>
|
||||
)}
|
||||
|
||||
{/* ── Tendrils controls ── */}
|
||||
{visualStyle === 'tendrils' && (
|
||||
<>
|
||||
<div className="space-y-1.5">
|
||||
<div className="text-[10px] text-slate-400">Pattern preset</div>
|
||||
<div className="flex flex-wrap gap-1.5">
|
||||
{(['flow', 'scatter', 'pulse', 'ethereal', 'hybrid'] as const).map((preset) => {
|
||||
const active = vizSettings.tendrils.visualization_preset === preset
|
||||
return (
|
||||
<button
|
||||
key={preset}
|
||||
type="button"
|
||||
onClick={() => patchVizSettings({ tendrils: { visualization_preset: preset } })}
|
||||
className={`rounded px-2 py-1 text-[11px] border transition-colors ${active ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
|
||||
>
|
||||
{preset}
|
||||
</button>
|
||||
)
|
||||
})}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{slider('Noise', vizSettings.tendrils.noise_weight, formatMultiplier(vizSettings.tendrils.noise_weight), 0, 3, 0.05, (v) => patchVizSettings({ tendrils: { noise_weight: v } }))}
|
||||
{slider('Flow', vizSettings.tendrils.flow_weight, formatMultiplier(vizSettings.tendrils.flow_weight), 0, 3, 0.05, (v) => patchVizSettings({ tendrils: { flow_weight: v } }))}
|
||||
{slider('Speed', vizSettings.tendrils.speed_limit, formatMultiplier(vizSettings.tendrils.speed_limit), 0, 3, 0.05, (v) => patchVizSettings({ tendrils: { speed_limit: v } }))}
|
||||
{slider('Damping', vizSettings.tendrils.damping, formatMultiplier(vizSettings.tendrils.damping), 0, 3, 0.05, (v) => patchVizSettings({ tendrils: { damping: v } }))}
|
||||
{slider('Hue range', vizSettings.tendrils.hue_range, formatMultiplier(vizSettings.tendrils.hue_range), 0, 3, 0.05, (v) => patchVizSettings({ tendrils: { hue_range: v } }))}
|
||||
|
||||
<label className="flex items-center gap-2 cursor-pointer">
|
||||
<div
|
||||
role="checkbox"
|
||||
aria-checked={vizSettings.tendrils.mono_color}
|
||||
tabIndex={0}
|
||||
onClick={() => patchVizSettings({ tendrils: { mono_color: !vizSettings.tendrils.mono_color } })}
|
||||
onKeyDown={(e) => { if (e.key === ' ' || e.key === 'Enter') patchVizSettings({ tendrils: { mono_color: !vizSettings.tendrils.mono_color } }) }}
|
||||
className={`w-8 h-4 rounded-full transition-colors ${vizSettings.tendrils.mono_color ? 'bg-violet-500' : 'bg-white/20'} relative`}
|
||||
>
|
||||
<div className={`absolute top-0.5 w-3 h-3 rounded-full bg-white transition-transform ${vizSettings.tendrils.mono_color ? 'translate-x-4' : 'translate-x-0.5'}`} />
|
||||
</div>
|
||||
<span className="text-[11px] text-slate-300">Mono color</span>
|
||||
</label>
|
||||
</>
|
||||
)}
|
||||
|
||||
{/* ── Spirit controls ── */}
|
||||
{visualStyle === 'spirit' && (
|
||||
<>
|
||||
<div className="space-y-1.5">
|
||||
<div className="text-[10px] text-slate-400">Pattern preset</div>
|
||||
<div className="flex flex-wrap gap-1.5">
|
||||
{(['spirit', 'storm', 'drift', 'pulse', 'aurora'] as const).map((preset) => {
|
||||
const active = vizSettings.spirit.visualization_preset === preset
|
||||
return (
|
||||
<button
|
||||
key={preset}
|
||||
type="button"
|
||||
onClick={() => patchVizSettings({ spirit: { visualization_preset: preset } })}
|
||||
className={`rounded px-2 py-1 text-[11px] border transition-colors ${active ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
|
||||
>
|
||||
{preset}
|
||||
</button>
|
||||
)
|
||||
})}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{slider('Speed', vizSettings.spirit.speed, formatMultiplier(vizSettings.spirit.speed), 0.1, 6, 0.05, (v) => patchVizSettings({ spirit: { speed: v } }))}
|
||||
{slider('Die speed', vizSettings.spirit.die_speed, vizSettings.spirit.die_speed.toFixed(3), 0.001, 0.12, 0.001, (v) => patchVizSettings({ spirit: { die_speed: v } }))}
|
||||
{slider('Curl', vizSettings.spirit.curl_size, formatMultiplier(vizSettings.spirit.curl_size), 0.05, 3, 0.05, (v) => patchVizSettings({ spirit: { curl_size: v } }))}
|
||||
{slider('Attraction', vizSettings.spirit.attraction, vizSettings.spirit.attraction.toFixed(2), -2, 2, 0.05, (v) => patchVizSettings({ spirit: { attraction: v } }))}
|
||||
{slider('Bloom', vizSettings.spirit.bloom_intensity, formatMultiplier(vizSettings.spirit.bloom_intensity), 0, 4, 0.05, (v) => patchVizSettings({ spirit: { bloom_intensity: v } }))}
|
||||
{slider('Afterimage', vizSettings.spirit.afterimage_damp, vizSettings.spirit.afterimage_damp.toFixed(2), 0.6, 0.96, 0.01, (v) => patchVizSettings({ spirit: { afterimage_damp: v } }))}
|
||||
</>
|
||||
)}
|
||||
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => patchVizSettings(manifestVizSettings ?? VIZ_DEFAULTS)}
|
||||
@@ -670,7 +868,7 @@ export default function PerformanceSidebar({
|
||||
{manifestVizSettings ? 'Reset to song defaults' : 'Reset to defaults'}
|
||||
</button>
|
||||
</div>
|
||||
</section>
|
||||
</details>
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
|
||||
@@ -5,7 +5,9 @@ import type { VisualizerPalette } from '../../hooks/useVisualizerPalette'
|
||||
import type { VisualizerSettings } from './VisualizerBackdrop'
|
||||
import { destroyFluidSim, initFluidSim, splatFluid, updateFluidConfig } from './fluid/fluidSim'
|
||||
|
||||
type FluidSettings = NonNullable<VisualizerSettings['fluid']>
|
||||
type FluidSettings = NonNullable<VisualizerSettings['fluid']> & {
|
||||
visualization_preset?: string
|
||||
}
|
||||
|
||||
interface FluidVisualizerCanvasProps {
|
||||
featuresRef: MutableRefObject<AudioFeatures>
|
||||
@@ -25,7 +27,7 @@ function clamp(v: number, lo: number, hi: number): number {
|
||||
return Math.max(lo, Math.min(hi, v))
|
||||
}
|
||||
|
||||
function hexToRgb(hex: string): { r: number; g: number; b: number } {
|
||||
function hexToRgbUnit(hex: string): { r: number; g: number; b: number } {
|
||||
const m = /^#?([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})$/i.exec(hex.trim())
|
||||
if (!m) return { r: 0.5, g: 0.5, b: 0.8 }
|
||||
return {
|
||||
@@ -35,28 +37,118 @@ function hexToRgb(hex: string): { r: number; g: number; b: number } {
|
||||
}
|
||||
}
|
||||
|
||||
function fluidDensityDissipation(s?: FluidSettings, kick = 0): number {
|
||||
const base = 0.98 - (s?.background_persistence ?? 1) * 0.025
|
||||
return clamp(base - kick * 0.04 * (s?.decay ?? 1), 0.85, 0.999)
|
||||
function hexToRgb255(hex: string): { r: number; g: number; b: number } {
|
||||
const m = /^#?([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})$/i.exec(hex.trim())
|
||||
if (!m) return { r: 128, g: 128, b: 204 }
|
||||
return {
|
||||
r: parseInt(m[1], 16),
|
||||
g: parseInt(m[2], 16),
|
||||
b: parseInt(m[3], 16),
|
||||
}
|
||||
}
|
||||
|
||||
function fluidVelocityDissipation(s?: FluidSettings, rms = 0): number {
|
||||
const base = 0.96 - (1 / Math.max(0.5, s?.drag ?? 1)) * 0.06
|
||||
return clamp(base - rms * 0.04, 0.80, 0.999)
|
||||
function blendColor(
|
||||
a: { r: number; g: number; b: number },
|
||||
b: { r: number; g: number; b: number },
|
||||
t: number,
|
||||
): { r: number; g: number; b: number } {
|
||||
return {
|
||||
r: a.r + (b.r - a.r) * t,
|
||||
g: a.g + (b.g - a.g) * t,
|
||||
b: a.b + (b.b - a.b) * t,
|
||||
}
|
||||
}
|
||||
|
||||
function fluidCurl(s?: FluidSettings, treble = 0, dropBoost = 0): number {
|
||||
const base = 15 + (s?.splat_force ?? 1) * 12
|
||||
return clamp(base + treble * 8 + dropBoost * 20, 0, 80)
|
||||
type FluidPreset = {
|
||||
label: string
|
||||
densityBase: number
|
||||
velocityBase: number
|
||||
curlBase: number
|
||||
pressureBase: number
|
||||
splatRadiusBase: number
|
||||
bloomIntensityBase: number
|
||||
sunraysWeightBase: number
|
||||
dropBoostDecay: number
|
||||
ambientForce: number
|
||||
burstForce: number
|
||||
sweepForce: number
|
||||
}
|
||||
|
||||
function fluidSplatRadius(s?: FluidSettings): number {
|
||||
return clamp(0.18 + (s?.blob_life ?? 1) * 0.06, 0.05, 0.5)
|
||||
const FLUID_PRESETS: Record<string, FluidPreset> = {
|
||||
ink: {
|
||||
label: 'Ink',
|
||||
densityBase: 0.996,
|
||||
velocityBase: 0.975,
|
||||
curlBase: 8,
|
||||
pressureBase: 0.70,
|
||||
splatRadiusBase: 0.30,
|
||||
bloomIntensityBase: 0.55,
|
||||
sunraysWeightBase: 0.25,
|
||||
dropBoostDecay: 0.97,
|
||||
ambientForce: 0.60,
|
||||
burstForce: 1.10,
|
||||
sweepForce: 0.80,
|
||||
},
|
||||
lava: {
|
||||
label: 'Lava',
|
||||
densityBase: 0.999,
|
||||
velocityBase: 0.994,
|
||||
curlBase: 18,
|
||||
pressureBase: 0.85,
|
||||
splatRadiusBase: 0.22,
|
||||
bloomIntensityBase: 1.30,
|
||||
sunraysWeightBase: 0.55,
|
||||
dropBoostDecay: 0.96,
|
||||
ambientForce: 0.80,
|
||||
burstForce: 0.90,
|
||||
sweepForce: 1.20,
|
||||
},
|
||||
storm: {
|
||||
label: 'Storm',
|
||||
densityBase: 0.960,
|
||||
velocityBase: 0.950,
|
||||
curlBase: 50,
|
||||
pressureBase: 0.90,
|
||||
splatRadiusBase: 0.12,
|
||||
bloomIntensityBase: 0.70,
|
||||
sunraysWeightBase: 1.40,
|
||||
dropBoostDecay: 0.94,
|
||||
ambientForce: 1.40,
|
||||
burstForce: 1.80,
|
||||
sweepForce: 1.50,
|
||||
},
|
||||
mist: {
|
||||
label: 'Mist',
|
||||
densityBase: 0.993,
|
||||
velocityBase: 0.988,
|
||||
curlBase: 5,
|
||||
pressureBase: 0.60,
|
||||
splatRadiusBase: 0.40,
|
||||
bloomIntensityBase: 2.00,
|
||||
sunraysWeightBase: 0.15,
|
||||
dropBoostDecay: 0.97,
|
||||
ambientForce: 0.50,
|
||||
burstForce: 0.65,
|
||||
sweepForce: 0.55,
|
||||
},
|
||||
plasma: {
|
||||
label: 'Plasma',
|
||||
densityBase: 0.972,
|
||||
velocityBase: 0.962,
|
||||
curlBase: 38,
|
||||
pressureBase: 0.88,
|
||||
splatRadiusBase: 0.14,
|
||||
bloomIntensityBase: 2.20,
|
||||
sunraysWeightBase: 1.60,
|
||||
dropBoostDecay: 0.95,
|
||||
ambientForce: 1.20,
|
||||
burstForce: 1.50,
|
||||
sweepForce: 1.80,
|
||||
},
|
||||
}
|
||||
|
||||
function fluidBloomIntensity(s?: FluidSettings, intensity = 0, dropBoost = 0): number {
|
||||
const base = 0.55 + Math.max(0, s?.treble_pulse ?? 1) * 0.28
|
||||
return clamp(base + intensity * 1.55 + dropBoost * 0.85, 0.35, 3.0)
|
||||
function fluidBloomIntensity(s?: FluidSettings, intensity = 0, dropBoost = 0, base = 0.8): number {
|
||||
return clamp(base + Math.max(0, s?.treble_pulse ?? 1) * 0.28 + intensity * 1.55 + dropBoost * 0.85, 0.35, 3.5)
|
||||
}
|
||||
|
||||
function fluidBloomThreshold(intensity = 0): number {
|
||||
@@ -102,8 +194,9 @@ export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width,
|
||||
return () => window.cancelAnimationFrame(rafFallback)
|
||||
}
|
||||
|
||||
const s0 = settingsRef.current
|
||||
const q = qualityRef.current
|
||||
const presetKey0 = settingsRef.current?.visualization_preset ?? 'lava'
|
||||
const preset0 = FLUID_PRESETS[presetKey0] ?? FLUID_PRESETS.lava
|
||||
initFluidSim(canvas, {
|
||||
TRIGGER: 'hover',
|
||||
IMMEDIATE: false,
|
||||
@@ -111,18 +204,23 @@ export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width,
|
||||
COLORFUL: false,
|
||||
SIM_RESOLUTION: q === 'high' ? 160 : q === 'low' ? 96 : 128,
|
||||
DYE_RESOLUTION: q === 'high' ? 1024 : q === 'low' ? 512 : 768,
|
||||
DENSITY_DISSIPATION: fluidDensityDissipation(s0),
|
||||
VELOCITY_DISSIPATION: fluidVelocityDissipation(s0),
|
||||
PRESSURE: 0.8,
|
||||
PRESSURE_ITERATIONS: 20,
|
||||
CURL: fluidCurl(s0),
|
||||
SPLAT_RADIUS: fluidSplatRadius(s0),
|
||||
DENSITY_DISSIPATION: preset0.densityBase,
|
||||
VELOCITY_DISSIPATION: preset0.velocityBase,
|
||||
PRESSURE: preset0.pressureBase,
|
||||
PRESSURE_ITERATIONS: q === 'low' ? 10 : q === 'high' ? 25 : 20,
|
||||
CURL: preset0.curlBase,
|
||||
SPLAT_RADIUS: preset0.splatRadiusBase,
|
||||
SPLAT_FORCE: 6000,
|
||||
SHADING: true,
|
||||
SHADING: q !== 'low',
|
||||
BLOOM: true,
|
||||
BLOOM_INTENSITY: 0.8,
|
||||
SUNRAYS: false,
|
||||
BACK_COLOR: hexToRgb(paletteRef.current.backgroundColor),
|
||||
BLOOM_INTENSITY: preset0.bloomIntensityBase,
|
||||
BLOOM_SOFT_KNEE: 0.7,
|
||||
BLOOM_ITERATIONS: q === 'low' ? 4 : 8,
|
||||
BLOOM_RESOLUTION: q === 'low' ? 128 : 256,
|
||||
SUNRAYS: true,
|
||||
SUNRAYS_RESOLUTION: q === 'low' ? 96 : 196,
|
||||
SUNRAYS_WEIGHT: preset0.sunraysWeightBase,
|
||||
BACK_COLOR: hexToRgb255(paletteRef.current.backgroundColor),
|
||||
})
|
||||
|
||||
let raf = 0
|
||||
@@ -176,27 +274,40 @@ export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width,
|
||||
dropBoost = 1
|
||||
lastDropAt = ts
|
||||
}
|
||||
dropBoost *= 0.96
|
||||
|
||||
const densityScale = Math.max(0.2, s?.spawn_density ?? 1)
|
||||
const splatForce = Math.max(0.2, s?.splat_force ?? 1)
|
||||
const beatBurstCount = Math.max(0, s?.beat_burst_count ?? 1)
|
||||
const treblePulse = Math.max(0, s?.treble_pulse ?? 1)
|
||||
const colorSpeed = Math.max(0, s?.color_speed ?? 1)
|
||||
const presetKey = s?.visualization_preset ?? 'lava'
|
||||
const preset = FLUID_PRESETS[presetKey] ?? FLUID_PRESETS.lava
|
||||
|
||||
dropBoost *= preset.dropBoostDecay
|
||||
|
||||
const bgRgb = hexToRgb255(paletteRef.current.backgroundColor)
|
||||
const highlightRgb = hexToRgb255(paletteRef.current.highlightColor)
|
||||
|
||||
updateFluidConfig(canvas, {
|
||||
DENSITY_DISSIPATION: fluidDensityDissipation(s, kick),
|
||||
VELOCITY_DISSIPATION: fluidVelocityDissipation(s, rms),
|
||||
CURL: fluidCurl(s, treble, dropBoost),
|
||||
SPLAT_RADIUS: fluidSplatRadius(s),
|
||||
BLOOM_INTENSITY: fluidBloomIntensity(s, audioIntensity, dropBoost),
|
||||
DENSITY_DISSIPATION: clamp(preset.densityBase - kick * 0.04 * (s?.decay ?? 1), 0.85, 0.999),
|
||||
VELOCITY_DISSIPATION: clamp(preset.velocityBase - rms * 0.04, 0.80, 0.999),
|
||||
CURL: clamp(preset.curlBase + treble * 8 + dropBoost * 20, 0, 80),
|
||||
PRESSURE: clamp(preset.pressureBase + rms * 0.32, 0.40, 0.92),
|
||||
SPLAT_RADIUS: clamp(preset.splatRadiusBase * clamp(0.6 + kick * 0.8, 0.4, 1.4), 0.05, 0.5),
|
||||
BLOOM_INTENSITY: fluidBloomIntensity(s, audioIntensity, dropBoost, preset.bloomIntensityBase),
|
||||
BLOOM_THRESHOLD: fluidBloomThreshold(audioIntensity),
|
||||
BACK_COLOR: hexToRgb(paletteRef.current.backgroundColor),
|
||||
BLOOM_SOFT_KNEE: clamp(0.65 - kick * 0.35, 0.20, 0.72),
|
||||
SUNRAYS_WEIGHT: clamp(preset.sunraysWeightBase + bass * 1.2 + dropBoost * 0.8, 0.1, 2.0),
|
||||
BACK_COLOR: dropBoost > 0.08
|
||||
? blendColor(bgRgb, highlightRgb, clamp(dropBoost * 0.15, 0, 0.4))
|
||||
: bgRgb,
|
||||
PAUSED: false,
|
||||
})
|
||||
|
||||
// WebGL sim renders in its own RAF; fpsCap throttles splat injection only.
|
||||
const minSplatMs = Math.max(minFrameMs, isLow ? 45 : 25)
|
||||
const minSplatMs = dropBoost > 0.5
|
||||
? minFrameMs
|
||||
: Math.max(minFrameMs, isLow ? 45 : 25)
|
||||
if (ts - lastSplatAt < minSplatMs - 1) {
|
||||
prevBeatPhase = beatPhase
|
||||
return
|
||||
@@ -206,14 +317,14 @@ export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width,
|
||||
// Ambient splat.
|
||||
if (Math.random() < (0.04 + rms * 0.35 + phasePulse * 0.08 + idle * 0.04) * densityScale) {
|
||||
const angle = Math.random() * Math.PI * 2
|
||||
const force = (0.2 + rms * 0.8 + treble * 0.3 * treblePulse + idle * 0.12) * splatForce * 400
|
||||
const force = (0.2 + rms * 0.8 + treble * 0.3 * treblePulse + idle * 0.12) * splatForce * 400 * preset.ambientForce
|
||||
splatFluid(
|
||||
canvas,
|
||||
Math.random(),
|
||||
Math.random(),
|
||||
Math.cos(angle) * force,
|
||||
Math.sin(angle) * force,
|
||||
hexToRgb(nextPaletteColor(colorSpeed * 0.25)),
|
||||
hexToRgbUnit(nextPaletteColor(colorSpeed * 0.25)),
|
||||
)
|
||||
}
|
||||
|
||||
@@ -222,7 +333,7 @@ export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width,
|
||||
const count = Math.max(1, Math.round((3 + kick * 5 + dropBoost * 4) * beatBurstCount))
|
||||
for (let i = 0; i < count; i += 1) {
|
||||
const angle = Math.random() * Math.PI * 2
|
||||
const force = (0.6 + bass * 1.2 + dropBoost * 0.6) * splatForce * 800
|
||||
const force = (0.6 + bass * 1.2 + dropBoost * 0.6) * splatForce * 800 * preset.burstForce
|
||||
splatFluid(
|
||||
canvas,
|
||||
Math.random(),
|
||||
@@ -230,8 +341,8 @@ export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width,
|
||||
Math.cos(angle) * force,
|
||||
Math.sin(angle) * force,
|
||||
dropTriggered
|
||||
? hexToRgb(paletteRef.current.highlightColor)
|
||||
: hexToRgb(nextPaletteColor(colorSpeed * 0.7)),
|
||||
? hexToRgbUnit(paletteRef.current.highlightColor)
|
||||
: hexToRgbUnit(nextPaletteColor(colorSpeed * 0.7)),
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -239,17 +350,18 @@ export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width,
|
||||
// Beat-phase sweep.
|
||||
if (beatPhase > 0.001 || f.beatInfo.bpm > 0) {
|
||||
const phaseAngle = beatPhase * Math.PI * 2
|
||||
const orbit = 0.22 + kick * 0.10 + dropBoost * 0.08
|
||||
if (Math.random() < 0.08 + phasePulse * 0.22 + dropBoost * 0.18) {
|
||||
const force = (0.28 + kick * 0.5 + phasePulse * 0.4) * splatForce * 500
|
||||
const force = (0.28 + kick * 0.5 + phasePulse * 0.4) * splatForce * 500 * preset.sweepForce
|
||||
splatFluid(
|
||||
canvas,
|
||||
0.5 + Math.cos(phaseAngle) * 0.28,
|
||||
0.5 + Math.sin(phaseAngle) * 0.20,
|
||||
0.5 + Math.cos(phaseAngle) * orbit,
|
||||
0.5 + Math.sin(phaseAngle) * (orbit * 0.72),
|
||||
Math.cos(phaseAngle + Math.PI / 2) * force,
|
||||
Math.sin(phaseAngle + Math.PI / 2) * force,
|
||||
dropBoost > 0.18
|
||||
? hexToRgb(paletteRef.current.highlightColor)
|
||||
: hexToRgb(nextPaletteColor(colorSpeed * 0.9)),
|
||||
? hexToRgbUnit(paletteRef.current.highlightColor)
|
||||
: hexToRgbUnit(nextPaletteColor(colorSpeed * 0.9)),
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -265,7 +377,7 @@ export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width,
|
||||
// Only restart when data-source refs change.
|
||||
// Settings, fpsCap, and quality are read via refs each frame — no restart on changes.
|
||||
// width/height are consumed each frame as fallback to clientWidth/clientHeight.
|
||||
}, [featuresRef, paletteRef])
|
||||
}, [featuresRef, paletteRef, quality])
|
||||
|
||||
return <canvas ref={canvasRef} className="h-full w-full" aria-hidden="true" />
|
||||
}
|
||||
@@ -3,7 +3,7 @@ import type { MutableRefObject } from 'react'
|
||||
import type { AudioFeatures } from '../../hooks/useAudioFeatures'
|
||||
import type { VisualizerPalette } from '../../hooks/useVisualizerPalette'
|
||||
import type { VisualizerSettings } from './VisualizerBackdrop'
|
||||
import { centeredNoise2D } from './muon/noise'
|
||||
import { curlNoise2D } from './muon/noise'
|
||||
import { patternFromPreset, samplePattern, type MuonPatternFamily } from './muon/patterns'
|
||||
|
||||
type MuonSettings = NonNullable<VisualizerSettings['muon']>
|
||||
@@ -84,6 +84,7 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
|
||||
let prevBeatPhase = 0
|
||||
let dropBoost = 0
|
||||
let lastDropAt = 0
|
||||
let lastPatternCycleAt = 0
|
||||
let patternShift = 0
|
||||
|
||||
let poolCap = maxParticlesByQuality('balanced')
|
||||
@@ -98,6 +99,7 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
|
||||
cy: number,
|
||||
ring: number,
|
||||
divisions: number,
|
||||
t: number,
|
||||
beatPhase: number,
|
||||
hueBase: number,
|
||||
aperture: number,
|
||||
@@ -110,17 +112,21 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
|
||||
alphaSeed: number,
|
||||
sizeSeed: number,
|
||||
dynamicRadius: number,
|
||||
drum: number,
|
||||
) => {
|
||||
const emitOne = (phaseOffset: number) => {
|
||||
const p = particles[nextPoolIdx]
|
||||
nextPoolIdx = (nextPoolIdx + 1) % particles.length
|
||||
|
||||
const t = ((performance.now() * 0.0004) + ring / Math.max(1, divisions) + phaseOffset) * Math.PI * 2
|
||||
const tFinal = t + phaseOffset * Math.PI * 2
|
||||
const pattern = patternFromPreset(patternPreset, ring + patternShift)
|
||||
const ringNorm = ring / Math.max(1, divisions - 1)
|
||||
const sample = samplePattern(pattern, t, ringNorm, aperture, beatPhase)
|
||||
const nx = centeredNoise2D(sample.x * noiseScale + ringNorm * 1.3, sample.y * noiseScale, ring * 17.17)
|
||||
const ny = centeredNoise2D(sample.y * noiseScale, sample.x * noiseScale + ringNorm * 1.9, ring * 29.37)
|
||||
const sample = samplePattern(pattern, tFinal, ringNorm, aperture, beatPhase)
|
||||
const { nx, ny } = curlNoise2D(
|
||||
sample.x * noiseScale + ringNorm * 1.3,
|
||||
sample.y * noiseScale + ringNorm * 0.9,
|
||||
ring * 17.17,
|
||||
)
|
||||
|
||||
const noiseAmp = noiseForce * 0.32
|
||||
const sx = sample.x + nx * noiseAmp
|
||||
@@ -130,7 +136,7 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
|
||||
|
||||
const tvx = sample.tx + nx * noiseAmp * 0.4
|
||||
const tvy = sample.ty + ny * noiseAmp * 0.4
|
||||
const speed = 0.12 + ringNorm * 0.22 + sizeSeed * 0.08
|
||||
const speed = (0.02 + ringNorm * 0.06 + sizeSeed * 0.02) * (1 + drum * 0.4)
|
||||
|
||||
p.active = true
|
||||
p.x = ex
|
||||
@@ -191,7 +197,7 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
|
||||
const idleSpeed = Math.max(0, s?.idle_speed ?? 1)
|
||||
const monoColor = Boolean(s?.mono_color)
|
||||
const divisions = Math.max(8, Math.min(48, Math.round(s?.divisions ?? 16)))
|
||||
const lifespan = Math.max(45, Math.min(420, Math.round(s?.lifespan ?? 180)))
|
||||
const lifespan = Math.max(20, Math.min(420, Math.round(s?.lifespan ?? 60)))
|
||||
const mirror = Boolean(s?.particle_mirror)
|
||||
const dynamicRadiusCfg = Math.max(0.2, s?.dynamic_radius ?? 1)
|
||||
const deltaLimit = Math.max(0.0005, s?.delta_response_limit ?? 0.005)
|
||||
@@ -213,11 +219,17 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
|
||||
if (dropTriggered) {
|
||||
dropBoost = 1
|
||||
lastDropAt = now
|
||||
patternShift += 1
|
||||
patternShift += 2
|
||||
lastPatternCycleAt = now
|
||||
}
|
||||
dropBoost *= 0.965
|
||||
|
||||
const phaseWrapped = beatPhase < prevBeatPhase - 0.35
|
||||
const beatCooldownSec = Math.max(0.6, 1.2 / Math.max(0.1, beatSensitivity))
|
||||
if ((f.beatDetected || phaseWrapped) && drum > 0.3 && now - lastPatternCycleAt > beatCooldownSec) {
|
||||
patternShift += 1
|
||||
lastPatternCycleAt = now
|
||||
}
|
||||
const idle = 0.5 + 0.5 * Math.sin(now * 1.2 * Math.max(0.01, idleSpeed))
|
||||
|
||||
const hue = monoColor
|
||||
@@ -249,33 +261,40 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
|
||||
const cx = w / 2
|
||||
const cy = h / 2
|
||||
const base = Math.min(w, h) * 0.17 * radiusScale
|
||||
const glow = (10 + drum * 22 + phasePulse * 7 + dropBoost * 18) * glowIntensity
|
||||
|
||||
const emitDensity = (s?.particle_count ?? 220) / 220
|
||||
const emitCount = Math.max(1, Math.round((0.8 + drum * 2.4 + phasePulse * 0.8 + dropBoost * 2.2) * emitDensity * (divisions / 16) * (0.75 + beatSensitivity * 0.25)))
|
||||
const lifeFrames = Math.round(lifespan * (0.8 + (1 - drum) * 0.25))
|
||||
const maxTotalPerFrame = Math.max(1, Math.floor(poolCap / Math.max(1, lifeFrames) * 0.7))
|
||||
const desiredSpots = Math.round((emitDensity * (0.8 + drum * 2.4 + phasePulse * 0.8 + dropBoost * 2.2)) / Math.max(1, divisions))
|
||||
const spotsCap = Math.max(1, Math.floor(maxTotalPerFrame / Math.max(1, divisions) / (mirror ? 2 : 1)))
|
||||
const spotsPerRing = Math.max(1, Math.min(spotsCap, desiredSpots))
|
||||
const globalPhase = now * 0.4 * rotationSpeed
|
||||
|
||||
for (let i = 0; i < emitCount; i += 1) {
|
||||
const ring = i % divisions
|
||||
for (let ring = 0; ring < divisions; ring += 1) {
|
||||
const ringRadius = base + ring * spacing
|
||||
emitParticle(
|
||||
cx,
|
||||
cy,
|
||||
ring,
|
||||
divisions,
|
||||
beatPhase,
|
||||
hue + bright * 18,
|
||||
aperture,
|
||||
mirror,
|
||||
preset,
|
||||
ringRadius,
|
||||
lifeFrames,
|
||||
noiseScale,
|
||||
noiseForce,
|
||||
0.34 + drum * 0.45 + phasePulse * 0.1 + dropBoost * 0.15,
|
||||
0.45 + bright * 0.35,
|
||||
dynamicRadius,
|
||||
)
|
||||
for (let j = 0; j < spotsPerRing; j += 1) {
|
||||
const circumOffset = j / Math.max(1, spotsPerRing)
|
||||
const t = (globalPhase + ring / divisions + circumOffset) * Math.PI * 2
|
||||
emitParticle(
|
||||
cx,
|
||||
cy,
|
||||
ring,
|
||||
divisions,
|
||||
t,
|
||||
beatPhase,
|
||||
hue + bright * 18,
|
||||
aperture,
|
||||
mirror,
|
||||
preset,
|
||||
ringRadius,
|
||||
lifeFrames,
|
||||
noiseScale,
|
||||
noiseForce,
|
||||
0.34 + drum * 0.45 + phasePulse * 0.1 + dropBoost * 0.15,
|
||||
0.45 + bright * 0.35,
|
||||
dynamicRadius,
|
||||
drum,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
ctx.globalCompositeOperation = 'lighter'
|
||||
@@ -300,22 +319,29 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
|
||||
const x = pt.x + wobble
|
||||
const y = pt.y + wobble * ellipseRatio
|
||||
|
||||
if (!Number.isFinite(x) || !Number.isFinite(y)) {
|
||||
pt.active = false
|
||||
continue
|
||||
}
|
||||
|
||||
const size = Math.max(0.3, pt.size * particleSize * (0.4 + life * 0.9))
|
||||
const gr = Math.max(0.5, size * (2.0 + glowIntensity * 0.8))
|
||||
const alpha = pt.alpha * life
|
||||
|
||||
const laneHue = monoColor
|
||||
? hue
|
||||
: (pt.hue + phaseWave * 18 * hueRange + beatPhase * 8 + dropBoost * 18) % 360
|
||||
|
||||
const grad = ctx.createRadialGradient(x, y, 0, x, y, gr)
|
||||
grad.addColorStop(0, `hsla(${laneHue}, ${85 + treble * 15}%, ${65 + bright * 20}%, ${alpha})`)
|
||||
grad.addColorStop(0.40, `hsla(${laneHue}, 92%, ${58 + bright * 24}%, ${alpha * 0.55})`)
|
||||
grad.addColorStop(1, `hsla(${laneHue}, 88%, 52%, 0)`)
|
||||
ctx.fillStyle = grad
|
||||
ctx.beginPath()
|
||||
ctx.fillStyle = `hsla(${laneHue}, 92%, ${58 + bright * 24}%, ${alpha})`
|
||||
ctx.shadowColor = p.highlightColor
|
||||
ctx.shadowBlur = glow
|
||||
ctx.arc(x, y, size, 0, Math.PI * 2)
|
||||
ctx.arc(x, y, gr, 0, Math.PI * 2)
|
||||
ctx.fill()
|
||||
}
|
||||
ctx.globalCompositeOperation = 'source-over'
|
||||
ctx.shadowBlur = 0
|
||||
prevBeatPhase = beatPhase
|
||||
}
|
||||
|
||||
@@ -326,7 +352,7 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
|
||||
// Only restart when data-source refs change.
|
||||
// Settings are read via settingsRef each frame — no restart on slider changes.
|
||||
// width/height are consumed each frame as fallback to clientWidth/clientHeight.
|
||||
}, [featuresRef, paletteRef])
|
||||
}, [featuresRef, paletteRef, quality])
|
||||
|
||||
return <canvas ref={canvasRef} className="h-full w-full" aria-hidden="true" />
|
||||
}
|
||||
@@ -0,0 +1,235 @@
|
||||
/**
|
||||
* SpiritVisualizerCanvas.tsx — React wrapper for the Spirit particle visualizer.
|
||||
*
|
||||
* Owns the RAF loop and FPS cap. Calls spiritSim.ts for all WebGL work.
|
||||
* Falls back to a 2D canvas colour fill when WebGL floats are unsupported.
|
||||
*/
|
||||
|
||||
import { useEffect, useRef, useCallback } from 'react'
|
||||
import type { MutableRefObject } from 'react'
|
||||
import * as THREE from 'three'
|
||||
import type { AudioFeatures } from '../../hooks/useAudioFeatures'
|
||||
import type { VisualizerPalette } from '../../hooks/useVisualizerPalette'
|
||||
import type { VisualizerSettings } from './VisualizerBackdrop'
|
||||
import {
|
||||
initSpiritSim,
|
||||
renderSpiritFrame,
|
||||
destroySpiritSim,
|
||||
type SpiritFrameUniforms,
|
||||
} from './spiritSim'
|
||||
|
||||
type SpiritSettings = NonNullable<VisualizerSettings['spirit']>
|
||||
|
||||
// ── Props ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
interface SpiritVisualizerCanvasProps {
|
||||
featuresRef: MutableRefObject<AudioFeatures>
|
||||
paletteRef: MutableRefObject<VisualizerPalette>
|
||||
width: number
|
||||
height: number
|
||||
settings?: SpiritSettings
|
||||
fpsCap?: number
|
||||
quality?: string
|
||||
}
|
||||
|
||||
// ── Presets ───────────────────────────────────────────────────────────────────
|
||||
|
||||
interface SpiritPreset {
|
||||
speed: number
|
||||
dieSpeed: number
|
||||
radius: number
|
||||
curlSize: number
|
||||
attraction: number
|
||||
bloomIntensity: number
|
||||
bloomRadius: number
|
||||
bloomThreshold: number
|
||||
afterimageDamp: number
|
||||
useTriangles: boolean
|
||||
}
|
||||
|
||||
const SPIRIT_PRESETS: Record<string, SpiritPreset> = {
|
||||
spirit: {
|
||||
speed: 1.0, dieSpeed: 0.015, radius: 1.5, curlSize: 0.4,
|
||||
attraction: 0.08, bloomIntensity: 1.5, bloomRadius: 0.4,
|
||||
bloomThreshold: 0.05, afterimageDamp: 0.88, useTriangles: false,
|
||||
},
|
||||
storm: {
|
||||
speed: 2.2, dieSpeed: 0.03, radius: 2.0, curlSize: 0.8,
|
||||
attraction: 0.2, bloomIntensity: 2.2, bloomRadius: 0.6,
|
||||
bloomThreshold: 0.1, afterimageDamp: 0.72, useTriangles: false,
|
||||
},
|
||||
drift: {
|
||||
speed: 0.5, dieSpeed: 0.006, radius: 2.5, curlSize: 0.2,
|
||||
attraction: 0.02, bloomIntensity: 1.0, bloomRadius: 0.3,
|
||||
bloomThreshold: 0.02, afterimageDamp: 0.94, useTriangles: false,
|
||||
},
|
||||
pulse: {
|
||||
speed: 1.4, dieSpeed: 0.04, radius: 1.0, curlSize: 0.6,
|
||||
attraction: 0.35, bloomIntensity: 2.8, bloomRadius: 0.7,
|
||||
bloomThreshold: 0.15, afterimageDamp: 0.60, useTriangles: false,
|
||||
},
|
||||
aurora: {
|
||||
speed: 0.8, dieSpeed: 0.01, radius: 3.0, curlSize: 0.15,
|
||||
attraction: 0.04, bloomIntensity: 1.8, bloomRadius: 0.8,
|
||||
bloomThreshold: 0.04, afterimageDamp: 0.92, useTriangles: true,
|
||||
},
|
||||
}
|
||||
|
||||
// ── Helpers ───────────────────────────────────────────────────────────────────
|
||||
|
||||
function clamp(v: number, lo: number, hi: number): number {
|
||||
return Math.max(lo, Math.min(hi, v))
|
||||
}
|
||||
|
||||
function hexToThree(hex: string): THREE.Color {
|
||||
return new THREE.Color(hex)
|
||||
}
|
||||
|
||||
// ── Component ─────────────────────────────────────────────────────────────────
|
||||
|
||||
export default function SpiritVisualizerCanvas({
|
||||
featuresRef, paletteRef, width, height, settings, fpsCap, quality,
|
||||
}: SpiritVisualizerCanvasProps) {
|
||||
const canvasRef = useRef<HTMLCanvasElement>(null)
|
||||
const rafRef = useRef<number>(0)
|
||||
const startTimeRef = useRef<number>(performance.now())
|
||||
|
||||
// Keep a ref so the tick closure always reads the latest settings
|
||||
const settingsRef = useRef(settings)
|
||||
useEffect(() => { settingsRef.current = settings }, [settings])
|
||||
|
||||
const texSize = quality === 'high' ? 512 : quality === 'low' ? 128 : 256
|
||||
const cappedFps = fpsCap ?? (quality === 'low' ? 30 : 60)
|
||||
|
||||
const initAndStart = useCallback(() => {
|
||||
// Drop-boost local state (decays per-frame like TendrilsVisualizerCanvas)
|
||||
let dropBoostLocal = 0
|
||||
let lastDropAt = 0
|
||||
const canvas = canvasRef.current
|
||||
if (!canvas) return
|
||||
canvas.width = Math.max(1, width)
|
||||
canvas.height = Math.max(1, height)
|
||||
|
||||
const cfg = settingsRef.current ?? {}
|
||||
const pKey = cfg.visualization_preset ?? 'spirit'
|
||||
const preset = SPIRIT_PRESETS[pKey] ?? SPIRIT_PRESETS['spirit']
|
||||
const pal = paletteRef.current
|
||||
|
||||
const ok = initSpiritSim(canvas, {
|
||||
texSize,
|
||||
speed: cfg.speed ?? preset.speed,
|
||||
dieSpeed: cfg.die_speed ?? preset.dieSpeed,
|
||||
radius: cfg.radius ?? preset.radius,
|
||||
curlSize: cfg.curl_size ?? preset.curlSize,
|
||||
attraction: cfg.attraction ?? preset.attraction,
|
||||
bloomStrength: cfg.bloom_intensity ?? preset.bloomIntensity,
|
||||
bloomRadius: preset.bloomRadius,
|
||||
bloomThreshold: cfg.bloom_threshold ?? preset.bloomThreshold,
|
||||
afterimageDamp: cfg.afterimage_damp ?? preset.afterimageDamp,
|
||||
useTriangles: cfg.use_triangles ?? preset.useTriangles,
|
||||
color1: hexToThree(pal.speakerPalette[1] ?? '#8888ff'),
|
||||
color2: hexToThree(pal.speakerPalette[0] ?? '#ff88ff'),
|
||||
})
|
||||
|
||||
if (!ok) {
|
||||
// 2D colour fallback (WebGL float render target unavailable)
|
||||
const ctx = canvas.getContext('2d')
|
||||
if (ctx) {
|
||||
const g = ctx.createRadialGradient(
|
||||
canvas.width * 0.5, canvas.height * 0.5, 0,
|
||||
canvas.width * 0.5, canvas.height * 0.5, Math.max(canvas.width, canvas.height) * 0.5,
|
||||
)
|
||||
g.addColorStop(0, pal.speakerPalette[0] ?? '#8822ff')
|
||||
g.addColorStop(1, '#000000')
|
||||
ctx.fillStyle = g
|
||||
ctx.fillRect(0, 0, canvas.width, canvas.height)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
let lastTime = 0
|
||||
const frameMs = 1000 / cappedFps
|
||||
|
||||
const tick = (now: number) => {
|
||||
rafRef.current = requestAnimationFrame(tick)
|
||||
if (now - lastTime < frameMs) return
|
||||
lastTime = now
|
||||
|
||||
const f = featuresRef.current
|
||||
const cfg = settingsRef.current ?? {}
|
||||
const pKey = cfg.visualization_preset ?? 'spirit'
|
||||
const preset = SPIRIT_PRESETS[pKey] ?? SPIRIT_PRESETS['spirit']
|
||||
const pal = paletteRef.current
|
||||
|
||||
// ── S6 audio→uniform mapping ─────────────────────────────────────────
|
||||
const bass = Math.max(0, Math.min(1, f?.bass ?? 0))
|
||||
const sub = Math.max(0, Math.min(1, f?.frequencyBands?.sub ?? 0))
|
||||
const rms = Math.max(0, Math.min(1, f?.rms ?? 0))
|
||||
const treble = Math.max(0, Math.min(1, f?.treble ?? 0))
|
||||
const kick = Math.max(0, Math.min(1, sub * 0.72 + bass * 0.28))
|
||||
const beatPhase = Math.max(0, Math.min(1, f?.beatInfo?.beatPhase ?? 0))
|
||||
if (f?.patternInfo?.isBeatDrop && now - lastDropAt > 500) {
|
||||
dropBoostLocal = 1; lastDropAt = now
|
||||
}
|
||||
dropBoostLocal *= 0.96
|
||||
const dropBoost = dropBoostLocal
|
||||
|
||||
const baseSpeed = cfg.speed ?? preset.speed
|
||||
const baseDieSpeed = cfg.die_speed ?? preset.dieSpeed
|
||||
const baseCurlSize = cfg.curl_size ?? preset.curlSize
|
||||
const baseAttraction = cfg.attraction ?? preset.attraction
|
||||
const baseBloom = cfg.bloom_intensity ?? preset.bloomIntensity
|
||||
const baseAfterimage = cfg.afterimage_damp ?? preset.afterimageDamp
|
||||
|
||||
const speed = baseSpeed * (1 + bass * 0.8 + kick * 0.6)
|
||||
const dieSpeed = baseDieSpeed * (1 + treble * 0.5 + sub * 0.3)
|
||||
const curlSize = baseCurlSize * (1 + rms * 0.6)
|
||||
let attraction = baseAttraction * (1 + beatPhase * 0.5)
|
||||
attraction -= dropBoost * 2.5 // drop reversal: particles explode outward
|
||||
const bloomStrength = baseBloom * (1 + bass * 0.4)
|
||||
const afterimageDamp = clamp(baseAfterimage + (1 - rms) * 0.04, 0.60, 0.96)
|
||||
|
||||
const t = (now - startTimeRef.current) * 0.001
|
||||
const orbitR = 0.5 + bass * 0.5
|
||||
const attractorPos: [number, number, number] = [
|
||||
Math.cos(t * 0.4 + beatPhase * Math.PI * 2) * orbitR,
|
||||
Math.sin(t * 0.3 + beatPhase * Math.PI) * orbitR * 0.5,
|
||||
Math.sin(t * 0.5) * 0.3,
|
||||
]
|
||||
|
||||
const uniforms: SpiritFrameUniforms = {
|
||||
time: t,
|
||||
speed: clamp(speed, 0.1, 6),
|
||||
dieSpeed: clamp(dieSpeed, 0.001, 0.12),
|
||||
curlSize: clamp(curlSize, 0.05, 3.0),
|
||||
attraction: clamp(attraction, -2, 2),
|
||||
attractorPos,
|
||||
bloomStrength: clamp(bloomStrength, 0, 4),
|
||||
bloomThreshold: cfg.bloom_threshold ?? preset.bloomThreshold,
|
||||
afterimageDamp,
|
||||
color1: hexToThree(pal.speakerPalette[1] ?? '#8888ff'),
|
||||
color2: hexToThree(pal.speakerPalette[0] ?? '#ff88ff'),
|
||||
}
|
||||
|
||||
renderSpiritFrame(canvas!, uniforms)
|
||||
}
|
||||
|
||||
rafRef.current = requestAnimationFrame(tick)
|
||||
}, [featuresRef, paletteRef, texSize, cappedFps, width, height])
|
||||
|
||||
useEffect(() => {
|
||||
initAndStart()
|
||||
const canvas = canvasRef.current // capture now; ref may change before cleanup
|
||||
return () => {
|
||||
cancelAnimationFrame(rafRef.current)
|
||||
if (canvas) destroySpiritSim(canvas)
|
||||
}
|
||||
}, [initAndStart])
|
||||
|
||||
return (
|
||||
<canvas
|
||||
ref={canvasRef}
|
||||
style={{ width: '100%', height: '100%', display: 'block' }}
|
||||
/>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,230 @@
|
||||
import { useEffect, useRef } from 'react'
|
||||
import type { MutableRefObject } from 'react'
|
||||
import type { AudioFeatures } from '../../hooks/useAudioFeatures'
|
||||
import type { VisualizerPalette } from '../../hooks/useVisualizerPalette'
|
||||
import type { VisualizerSettings } from './VisualizerBackdrop'
|
||||
import { initTendrilsSim, tickTendrils, destroyTendrilsSim } from './tendrilsSim'
|
||||
|
||||
type TendrilsSettings = NonNullable<VisualizerSettings['tendrils']>
|
||||
|
||||
interface TendrilsVisualizerCanvasProps {
|
||||
featuresRef: MutableRefObject<AudioFeatures>
|
||||
paletteRef: MutableRefObject<VisualizerPalette>
|
||||
width: number
|
||||
height: number
|
||||
settings?: TendrilsSettings
|
||||
fpsCap?: number
|
||||
quality?: string
|
||||
}
|
||||
|
||||
function clamp01(v: number): number { return Math.max(0, Math.min(1, v)) }
|
||||
function clamp(v: number, lo: number, hi: number): number { return Math.max(lo, Math.min(hi, v)) }
|
||||
|
||||
// Convert hex colour to linear [0,1] RGB tuple
|
||||
function hexToRgb(hex: string): [number, number, number] {
|
||||
const m = /^#?([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})$/i.exec(hex.trim())
|
||||
if (!m) return [0.3, 0.3, 0.6]
|
||||
return [parseInt(m[1], 16) / 255, parseInt(m[2], 16) / 255, parseInt(m[3], 16) / 255]
|
||||
}
|
||||
|
||||
// Rotate an rgb tuple by `turns` (0–1) around the colour wheel (HSL hue rotation)
|
||||
function hueRotate(rgb: [number, number, number], turns: number): [number, number, number] {
|
||||
const a = turns * Math.PI * 2
|
||||
const cos = Math.cos(a)
|
||||
const sin = Math.sin(a)
|
||||
const r = rgb[0] * (0.213 + cos * 0.787 - sin * 0.213) + rgb[1] * (0.715 - cos * 0.715 - sin * 0.715) + rgb[2] * (0.072 - cos * 0.072 + sin * 0.928)
|
||||
const g = rgb[0] * (0.213 - cos * 0.213 + sin * 0.143) + rgb[1] * (0.715 + cos * 0.285 + sin * 0.140) + rgb[2] * (0.072 - cos * 0.072 - sin * 0.283)
|
||||
const b = rgb[0] * (0.213 - cos * 0.213 - sin * 0.787) + rgb[1] * (0.715 - cos * 0.715 + sin * 0.715) + rgb[2] * (0.072 + cos * 0.928 + sin * 0.072)
|
||||
return [clamp01(r), clamp01(g), clamp01(b)]
|
||||
}
|
||||
|
||||
type TendrilsPreset = {
|
||||
label: string
|
||||
dampBase: number
|
||||
speedBase: number
|
||||
forceBase: number
|
||||
noiseWeightBase: number
|
||||
noiseScaleBase: number
|
||||
noiseScaleRange: number
|
||||
flowWeightBase: number
|
||||
flowDecay: number
|
||||
targetBase: number
|
||||
colorMap: number
|
||||
}
|
||||
|
||||
const TENDRILS_PRESETS: Record<string, TendrilsPreset> = {
|
||||
flow: {
|
||||
label: 'Flow',
|
||||
dampBase: 0.055, speedBase: 0.010, forceBase: 0.014,
|
||||
noiseWeightBase: 0.001, noiseScaleBase: 1.8, noiseScaleRange: 0.8,
|
||||
flowWeightBase: 1.4, flowDecay: 0.004,
|
||||
targetBase: 0.04, colorMap: 0.35,
|
||||
},
|
||||
scatter: {
|
||||
label: 'Scatter',
|
||||
dampBase: 0.030, speedBase: 0.018, forceBase: 0.022,
|
||||
noiseWeightBase: 0.004, noiseScaleBase: 3.2, noiseScaleRange: 1.6,
|
||||
flowWeightBase: 0.3, flowDecay: 0.012,
|
||||
targetBase: 0.01, colorMap: 0.65,
|
||||
},
|
||||
pulse: {
|
||||
label: 'Pulse',
|
||||
dampBase: 0.045, speedBase: 0.012, forceBase: 0.020,
|
||||
noiseWeightBase: 0.002, noiseScaleBase: 2.4, noiseScaleRange: 1.2,
|
||||
flowWeightBase: 0.8, flowDecay: 0.007,
|
||||
targetBase: 0.10, colorMap: 0.50,
|
||||
},
|
||||
ethereal: {
|
||||
label: 'Ethereal',
|
||||
dampBase: 0.080, speedBase: 0.006, forceBase: 0.008,
|
||||
noiseWeightBase: 0.0005, noiseScaleBase: 1.2, noiseScaleRange: 0.4,
|
||||
flowWeightBase: 1.8, flowDecay: 0.002,
|
||||
targetBase: 0.02, colorMap: 0.20,
|
||||
},
|
||||
hybrid: {
|
||||
label: 'Hybrid',
|
||||
dampBase: 0.043, speedBase: 0.010, forceBase: 0.016,
|
||||
noiseWeightBase: 0.002, noiseScaleBase: 2.1, noiseScaleRange: 1.0,
|
||||
flowWeightBase: 1.0, flowDecay: 0.006,
|
||||
targetBase: 0.05, colorMap: 0.40,
|
||||
},
|
||||
}
|
||||
|
||||
export default function TendrilsVisualizerCanvas({
|
||||
featuresRef, paletteRef, width, height, settings, fpsCap, quality,
|
||||
}: TendrilsVisualizerCanvasProps) {
|
||||
const canvasRef = useRef<HTMLCanvasElement>(null)
|
||||
const settingsRef = useRef<TendrilsSettings | undefined>(settings)
|
||||
settingsRef.current = settings
|
||||
const fpsCapRef = useRef(fpsCap ?? 60)
|
||||
fpsCapRef.current = fpsCap ?? 60
|
||||
const qualityRef = useRef(quality ?? 'balanced')
|
||||
qualityRef.current = quality ?? 'balanced'
|
||||
const widthRef = useRef(width)
|
||||
widthRef.current = width
|
||||
const heightRef = useRef(height)
|
||||
heightRef.current = height
|
||||
|
||||
useEffect(() => {
|
||||
const canvas = canvasRef.current
|
||||
if (!canvas) return
|
||||
|
||||
const q = qualityRef.current
|
||||
const rootNum = q === 'high' ? 512 : 256
|
||||
const ok = initTendrilsSim(canvas, { rootNum })
|
||||
|
||||
if (!ok) {
|
||||
// WebGL2 / float buffer unavailable — 2D colour fill fallback
|
||||
const ctx = canvas.getContext('2d')
|
||||
if (!ctx) return
|
||||
let rafFb = 0
|
||||
const renderFb = () => {
|
||||
rafFb = window.requestAnimationFrame(renderFb)
|
||||
const w = Math.max(1, Math.round(canvas.clientWidth || 0) || widthRef.current)
|
||||
const h = Math.max(1, Math.round(canvas.clientHeight || 0) || heightRef.current)
|
||||
if (canvas.width !== w || canvas.height !== h) { canvas.width = w; canvas.height = h }
|
||||
ctx.fillStyle = paletteRef.current.backgroundColor
|
||||
ctx.fillRect(0, 0, w, h)
|
||||
}
|
||||
rafFb = window.requestAnimationFrame(renderFb)
|
||||
return () => window.cancelAnimationFrame(rafFb)
|
||||
}
|
||||
|
||||
let raf = 0
|
||||
let lastFrameAt = 0
|
||||
let dropBoost = 0
|
||||
let lastDropAt = 0
|
||||
let prevBeatPhase = 0
|
||||
const startTime = performance.now() * 0.001
|
||||
|
||||
const render = (ts: number) => {
|
||||
raf = window.requestAnimationFrame(render)
|
||||
|
||||
const minFrameMs = 1000 / Math.max(1, fpsCapRef.current)
|
||||
if (ts - lastFrameAt < minFrameMs - 1) return
|
||||
lastFrameAt = ts
|
||||
|
||||
const f = featuresRef.current
|
||||
const s = settingsRef.current
|
||||
const p = paletteRef.current
|
||||
|
||||
const bass = clamp01(f.bass)
|
||||
const sub = clamp01(f.frequencyBands.sub)
|
||||
const rms = clamp01(f.rms)
|
||||
const treble = clamp01(f.treble)
|
||||
const drum = clamp01(f.frequencyBands.bass)
|
||||
const beatPhase = clamp01(f.beatInfo.beatPhase)
|
||||
const phasePulse = Math.pow(Math.max(0, Math.sin(beatPhase * Math.PI)), 1.8)
|
||||
const phaseWrapped = beatPhase < prevBeatPhase - 0.35
|
||||
const kick = clamp01(sub * 0.72 + bass * 0.28)
|
||||
const idle = 0.5 + 0.5 * Math.sin(performance.now() * 0.0009)
|
||||
|
||||
const dropTriggered = f.patternInfo.isBeatDrop && ts - lastDropAt > 500
|
||||
if (dropTriggered) { dropBoost = 1; lastDropAt = ts }
|
||||
dropBoost *= 0.96
|
||||
|
||||
const presetKey = s?.visualization_preset ?? 'hybrid'
|
||||
const preset = TENDRILS_PRESETS[presetKey] ?? TENDRILS_PRESETS.hybrid
|
||||
|
||||
// Audio → uniforms (T8)
|
||||
const forceWeight = preset.forceBase * (1 + bass * 1.8 + phasePulse * 0.6)
|
||||
const speedLimit = preset.speedBase * (1 + kick * 2.5 + dropBoost * 1.2)
|
||||
const noiseScale = preset.noiseScaleBase + treble * preset.noiseScaleRange
|
||||
const noiseWeight = preset.noiseWeightBase + rms * 0.6 + idle * 0.1
|
||||
const flowWeight = preset.flowWeightBase + drum * 1.4
|
||||
const damping = Math.max(0.015, preset.dampBase - dropBoost * 0.08)
|
||||
const flowDecay = Math.max(0.001, preset.flowDecay - dropBoost * 0.015)
|
||||
const targetWeight = preset.targetBase + rms * 0.3
|
||||
const colorMapAlpha = clamp(preset.colorMap + bass * 0.4 + treble * 0.3 + (s?.color_map_alpha ?? 0), 0.1, 0.9)
|
||||
const trailAlpha = 0.04 + dropBoost * 0.06
|
||||
|
||||
const phaseAngle = beatPhase * Math.PI * 2
|
||||
const target: [number, number] = [
|
||||
Math.cos(phaseAngle) * 0.6,
|
||||
Math.sin(phaseAngle) * 0.45,
|
||||
]
|
||||
|
||||
const seedRgb = hexToRgb(p.seedColor ?? p.highlightColor)
|
||||
const hueRange = Math.max(0, s?.hue_range ?? 1)
|
||||
const baseColor = s?.mono_color ? seedRgb : hueRotate(seedRgb, rms * 0.3 * hueRange)
|
||||
const flowColor = hueRotate(seedRgb, 0.5 + bass * 0.3 * hueRange)
|
||||
const bgColor = hexToRgb(p.backgroundColor)
|
||||
|
||||
const now = ts * 0.001 - startTime
|
||||
|
||||
tickTendrils(canvas, {
|
||||
time: now,
|
||||
damping: damping * Math.max(0.2, s?.damping ?? 1),
|
||||
speedLimit: speedLimit * Math.max(0.2, s?.speed_limit ?? 1),
|
||||
forceWeight: forceWeight * Math.max(0.2, s?.force_weight ?? 1),
|
||||
noiseWeight: noiseWeight * Math.max(0.01, s?.noise_weight ?? 1),
|
||||
noiseScale: noiseScale * Math.max(0.1, s?.noise_scale ?? 1),
|
||||
noiseSpeed: 0.4,
|
||||
flowWeight: flowWeight * Math.max(0.01, s?.flow_weight ?? 1),
|
||||
flowDecay: flowDecay * Math.max(0.1, s?.flow_decay ?? 1),
|
||||
targetWeight: targetWeight,
|
||||
target,
|
||||
trailAlpha,
|
||||
colorMapAlpha,
|
||||
speedAlpha: 0.002 + kick * 0.003,
|
||||
baseColor,
|
||||
flowColor,
|
||||
bgColor,
|
||||
lineWidth: s?.line_width ?? 1,
|
||||
})
|
||||
|
||||
// Beat-driven pattern cycling visual handled by forceWeight/targetWeight spikes
|
||||
if (phaseWrapped || dropTriggered) { void 0 /* placeholder for future effects */ }
|
||||
|
||||
prevBeatPhase = beatPhase
|
||||
}
|
||||
|
||||
raf = window.requestAnimationFrame(render)
|
||||
return () => {
|
||||
window.cancelAnimationFrame(raf)
|
||||
destroyTendrilsSim(canvas)
|
||||
}
|
||||
}, [featuresRef, paletteRef, quality])
|
||||
|
||||
return <canvas ref={canvasRef} className="h-full w-full" aria-hidden="true" />
|
||||
}
|
||||
@@ -3,10 +3,12 @@ import type { MutableRefObject } from 'react'
|
||||
import type { AudioFeatures } from '../../hooks/useAudioFeatures'
|
||||
import type { VisualizerPalette } from '../../hooks/useVisualizerPalette'
|
||||
|
||||
const MuonVisualizerCanvas = lazy(() => import('./MuonVisualizerCanvas'))
|
||||
const FluidVisualizerCanvas = lazy(() => import('./FluidVisualizerCanvas'))
|
||||
const MuonVisualizerCanvas = lazy(() => import('./MuonVisualizerCanvas'))
|
||||
const FluidVisualizerCanvas = lazy(() => import('./FluidVisualizerCanvas'))
|
||||
const TendrilsVisualizerCanvas = lazy(() => import('./TendrilsVisualizerCanvas'))
|
||||
const SpiritVisualizerCanvas = lazy(() => import('./SpiritVisualizerCanvas'))
|
||||
|
||||
export type VisualizerMode = 'muon' | 'fluid' | null
|
||||
export type VisualizerMode = 'muon' | 'fluid' | 'tendrils' | 'spirit' | null
|
||||
|
||||
/**
|
||||
* Per-song visualizer intensity multipliers from manifest.visualizer.settings.
|
||||
@@ -54,6 +56,38 @@ export interface VisualizerSettings {
|
||||
blob_life?: number
|
||||
// GUI Parameter Map additions
|
||||
color_speed?: number
|
||||
visualization_preset?: string
|
||||
}
|
||||
|
||||
tendrils?: {
|
||||
damping?: number
|
||||
noise_weight?: number
|
||||
noise_scale?: number
|
||||
flow_weight?: number
|
||||
flow_decay?: number
|
||||
force_weight?: number
|
||||
speed_limit?: number
|
||||
line_width?: number
|
||||
color_map_alpha?: number
|
||||
hue_range?: number
|
||||
mono_color?: boolean
|
||||
particle_count?: number
|
||||
visualization_preset?: string
|
||||
}
|
||||
|
||||
spirit?: {
|
||||
speed?: number
|
||||
die_speed?: number
|
||||
radius?: number
|
||||
curl_size?: number
|
||||
attraction?: number
|
||||
use_triangles?: boolean
|
||||
bloom_intensity?: number
|
||||
bloom_threshold?: number
|
||||
afterimage_damp?: number
|
||||
hue_range?: number
|
||||
mono_color?: boolean
|
||||
visualization_preset?: string
|
||||
}
|
||||
}
|
||||
|
||||
@@ -103,5 +137,41 @@ export default function VisualizerBackdrop({ mode, featuresRef, paletteRef, widt
|
||||
</div>
|
||||
)
|
||||
}
|
||||
if (mode === 'tendrils') {
|
||||
return (
|
||||
<div
|
||||
className="relative h-full w-full overflow-hidden"
|
||||
style={{ background: palette.backgroundColor }}
|
||||
data-testid="visualizer-backdrop"
|
||||
data-visualizer-mode={mode}
|
||||
data-palette-source={palette.source}
|
||||
data-palette-seed={palette.seedColor}
|
||||
data-palette-highlight={palette.highlightColor}
|
||||
data-palette-word-highlight={palette.wordHighlightColor}
|
||||
>
|
||||
<Suspense fallback={<div className="h-full w-full" style={{ background: palette.backgroundColor }} />}>
|
||||
<TendrilsVisualizerCanvas featuresRef={featuresRef} paletteRef={paletteRef} width={width} height={height} settings={settings?.tendrils} fpsCap={settings?.fps_cap} quality={settings?.quality} />
|
||||
</Suspense>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
if (mode === 'spirit') {
|
||||
return (
|
||||
<div
|
||||
className="relative h-full w-full overflow-hidden"
|
||||
style={{ background: palette.backgroundColor }}
|
||||
data-testid="visualizer-backdrop"
|
||||
data-visualizer-mode={mode}
|
||||
data-palette-source={palette.source}
|
||||
data-palette-seed={palette.seedColor}
|
||||
data-palette-highlight={palette.highlightColor}
|
||||
data-palette-word-highlight={palette.wordHighlightColor}
|
||||
>
|
||||
<Suspense fallback={<div className="h-full w-full" style={{ background: palette.backgroundColor }} />}>
|
||||
<SpiritVisualizerCanvas featuresRef={featuresRef} paletteRef={paletteRef} width={width} height={height} settings={settings?.spirit} fpsCap={settings?.fps_cap} quality={settings?.quality} />
|
||||
</Suspense>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
return null
|
||||
}
|
||||
@@ -56,30 +56,24 @@ export interface FluidConfig {
|
||||
interface FluidInstance {
|
||||
canvas: HTMLCanvasElement
|
||||
config: FluidConfig
|
||||
lastX: number
|
||||
lastY: number
|
||||
}
|
||||
|
||||
interface FluidCanvasElement extends HTMLCanvasElement {
|
||||
_fluidSplat?: (x: number, y: number, dx: number, dy: number, color: FluidColor) => void
|
||||
}
|
||||
|
||||
const instances = new WeakMap<HTMLCanvasElement, FluidInstance>()
|
||||
const initTokens = new WeakMap<HTMLCanvasElement, number>()
|
||||
let nextInitToken = 1
|
||||
|
||||
function clamp01(v: number): number {
|
||||
return Math.max(0, Math.min(1, v))
|
||||
}
|
||||
|
||||
function dispatchMouse(canvas: HTMLCanvasElement, type: 'mousedown' | 'mousemove', x: number, y: number): void {
|
||||
const evt = new MouseEvent(type, {
|
||||
bubbles: true,
|
||||
cancelable: true,
|
||||
clientX: x,
|
||||
clientY: y,
|
||||
})
|
||||
Object.defineProperty(evt, 'offsetX', { configurable: true, value: x })
|
||||
Object.defineProperty(evt, 'offsetY', { configurable: true, value: y })
|
||||
canvas.dispatchEvent(evt)
|
||||
}
|
||||
|
||||
export function initFluidSim(canvas: HTMLCanvasElement, initialConfig: FluidConfig): void {
|
||||
destroyFluidSim(canvas)
|
||||
const initToken = nextInitToken++
|
||||
initTokens.set(canvas, initToken)
|
||||
|
||||
const config: FluidConfig = {
|
||||
TRIGGER: 'hover',
|
||||
@@ -97,16 +91,13 @@ export function initFluidSim(canvas: HTMLCanvasElement, initialConfig: FluidConf
|
||||
// Load WebGL library asynchronously but don't block
|
||||
getWebGLFluid()
|
||||
.then((WebGL) => {
|
||||
// Only initialize if canvas still exists and not destroyed
|
||||
if (document.contains(canvas)) {
|
||||
WebGL(canvas, config)
|
||||
instances.set(canvas, {
|
||||
canvas,
|
||||
config,
|
||||
lastX: canvas.clientWidth * 0.5,
|
||||
lastY: canvas.clientHeight * 0.5,
|
||||
})
|
||||
}
|
||||
if (initTokens.get(canvas) !== initToken) return
|
||||
if (!document.contains(canvas)) return
|
||||
|
||||
WebGL(canvas, config)
|
||||
|
||||
if (initTokens.get(canvas) !== initToken) return
|
||||
instances.set(canvas, { canvas, config })
|
||||
})
|
||||
.catch((err) => console.error('Failed to load webgl-fluid:', err))
|
||||
}
|
||||
@@ -117,38 +108,24 @@ export function updateFluidConfig(canvas: HTMLCanvasElement, patch: Partial<Flui
|
||||
Object.assign(inst.config, patch)
|
||||
}
|
||||
|
||||
export function splatFluid(canvas: HTMLCanvasElement, x: number, y: number, dx: number, dy: number, _color: FluidColor): void {
|
||||
export function splatFluid(canvas: HTMLCanvasElement, x: number, y: number, dx: number, dy: number, color: FluidColor): void {
|
||||
const inst = instances.get(canvas)
|
||||
if (!inst) return
|
||||
|
||||
const w = Math.max(1, canvas.clientWidth || canvas.width || 1)
|
||||
const h = Math.max(1, canvas.clientHeight || canvas.height || 1)
|
||||
const fluidCanvas = canvas as FluidCanvasElement
|
||||
const directSplat = fluidCanvas._fluidSplat
|
||||
if (!directSplat) return
|
||||
|
||||
const nx = clamp01(x)
|
||||
const ny = clamp01(y)
|
||||
|
||||
const px = nx * w
|
||||
const py = ny * h
|
||||
|
||||
const speedNorm = Math.hypot(dx, dy) / Math.max(1, Math.min(w, h) * 6)
|
||||
const pressure = clamp01(speedNorm)
|
||||
const dragScale = 0.02 + pressure * 0.09
|
||||
const tx = Math.max(0, Math.min(w, px + dx * dragScale))
|
||||
const ty = Math.max(0, Math.min(h, py + dy * dragScale))
|
||||
const mx = px + (tx - px) * (0.45 + pressure * 0.25)
|
||||
const my = py + (ty - py) * (0.45 + pressure * 0.25)
|
||||
|
||||
dispatchMouse(canvas, 'mousedown', px, py)
|
||||
dispatchMouse(canvas, 'mousemove', mx, my)
|
||||
dispatchMouse(canvas, 'mousemove', tx, ty)
|
||||
|
||||
window.dispatchEvent(new MouseEvent('mouseup', { bubbles: true, cancelable: true }))
|
||||
|
||||
inst.lastX = tx
|
||||
inst.lastY = ty
|
||||
// Preserve existing visual orientation: WebGL fluid uses bottom-origin texcoords.
|
||||
directSplat(clamp01(x), 1 - clamp01(y), dx, dy, color)
|
||||
}
|
||||
|
||||
export function destroyFluidSim(canvas: HTMLCanvasElement): void {
|
||||
initTokens.delete(canvas)
|
||||
|
||||
const fluidCanvas = canvas as FluidCanvasElement
|
||||
delete fluidCanvas._fluidSplat
|
||||
|
||||
const inst = instances.get(canvas)
|
||||
if (!inst) return
|
||||
|
||||
|
||||
@@ -1804,6 +1804,8 @@ function ft(V, _) {
|
||||
s = (s << 5) - s + o.charCodeAt(a), s |= 0;
|
||||
return s;
|
||||
}
|
||||
// STEMENGINE_PATCH: expose direct splat API for external color/velocity injection.
|
||||
D._fluidSplat = fe;
|
||||
}
|
||||
export {
|
||||
ft as default
|
||||
|
||||
@@ -1,8 +1,42 @@
|
||||
export function hashNoise2D(x: number, y: number, seed = 0): number {
|
||||
const s = Math.sin(x * 127.1 + y * 311.7 + seed * 74.7) * 43758.5453123
|
||||
function fade(t: number): number {
|
||||
return t * t * t * (t * (t * 6 - 15) + 10)
|
||||
}
|
||||
|
||||
function lerpf(a: number, b: number, t: number): number {
|
||||
return a + (b - a) * t
|
||||
}
|
||||
|
||||
function hashf(ix: number, iy: number): number {
|
||||
const n = ix * 127.1 + iy * 311.7
|
||||
const s = Math.sin(n) * 43758.5453123
|
||||
return s - Math.floor(s)
|
||||
}
|
||||
|
||||
export function smoothNoise2D(x: number, y: number, seed = 0): number {
|
||||
const ox = x + seed * 17.3
|
||||
const oy = y + seed * 31.7
|
||||
const ix = Math.floor(ox)
|
||||
const iy = Math.floor(oy)
|
||||
const fx = ox - ix
|
||||
const fy = oy - iy
|
||||
const ux = fade(fx)
|
||||
const uy = fade(fy)
|
||||
|
||||
const a = hashf(ix, iy)
|
||||
const b = hashf(ix + 1, iy)
|
||||
const c = hashf(ix, iy + 1)
|
||||
const d = hashf(ix + 1, iy + 1)
|
||||
|
||||
return lerpf(lerpf(a, b, ux), lerpf(c, d, ux), uy)
|
||||
}
|
||||
|
||||
export function curlNoise2D(x: number, y: number, seed = 0): { nx: number; ny: number } {
|
||||
const eps = 0.1
|
||||
const n1 = smoothNoise2D(x, y + eps, seed) - smoothNoise2D(x, y - eps, seed)
|
||||
const n2 = smoothNoise2D(x + eps, y, seed) - smoothNoise2D(x - eps, y, seed)
|
||||
return { nx: n1, ny: -n2 }
|
||||
}
|
||||
|
||||
export function centeredNoise2D(x: number, y: number, seed = 0): number {
|
||||
return hashNoise2D(x, y, seed) * 2 - 1
|
||||
return smoothNoise2D(x, y, seed) * 2 - 1
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
/* 3D divergence-free curl noise using three independent potential fields.
|
||||
* Returns a vec3 in approximately (-1, 1) per axis.
|
||||
* Concatenated at the top of position.glsl by spiritSim.ts at runtime.
|
||||
* Ref: Bridson 2007 curl noise.
|
||||
*/
|
||||
|
||||
float _h31(vec3 p) {
|
||||
p = fract(p * vec3(0.1031, 0.1030, 0.0973));
|
||||
p += dot(p, p.yxz + 33.33);
|
||||
return fract((p.x + p.y) * p.z);
|
||||
}
|
||||
|
||||
float _vn3(vec3 p) {
|
||||
vec3 i = floor(p);
|
||||
vec3 f = fract(p);
|
||||
f = f * f * (3.0 - 2.0 * f);
|
||||
return mix(
|
||||
mix(mix(_h31(i), _h31(i + vec3(1.0, 0.0, 0.0)), f.x),
|
||||
mix(_h31(i + vec3(0.0, 1.0, 0.0)), _h31(i + vec3(1.0, 1.0, 0.0)), f.x), f.y),
|
||||
mix(mix(_h31(i + vec3(0.0, 0.0, 1.0)), _h31(i + vec3(1.0, 0.0, 1.0)), f.x),
|
||||
mix(_h31(i + vec3(0.0, 1.0, 1.0)), _h31(i + vec3(1.0, 1.0, 1.0)), f.x), f.y),
|
||||
f.z);
|
||||
}
|
||||
|
||||
vec3 curlNoise(vec3 p) {
|
||||
const float e = 0.1;
|
||||
vec3 pA = p + vec3( 0.0, 31.7, 0.0);
|
||||
vec3 pB = p + vec3(74.1, 0.0, 0.0);
|
||||
vec3 pC = p + vec3( 0.0, 0.0, 113.5);
|
||||
|
||||
float dy_pA = _vn3(pA + vec3(0.0, e, 0.0)) - _vn3(pA - vec3(0.0, e, 0.0));
|
||||
float dz_pA = _vn3(pA + vec3(0.0, 0.0, e)) - _vn3(pA - vec3(0.0, 0.0, e));
|
||||
float dx_pB = _vn3(pB + vec3(e, 0.0, 0.0)) - _vn3(pB - vec3(e, 0.0, 0.0));
|
||||
float dz_pB = _vn3(pB + vec3(0.0, 0.0, e)) - _vn3(pB - vec3(0.0, 0.0, e));
|
||||
float dx_pC = _vn3(pC + vec3(e, 0.0, 0.0)) - _vn3(pC - vec3(e, 0.0, 0.0));
|
||||
float dy_pC = _vn3(pC + vec3(0.0, e, 0.0)) - _vn3(pC - vec3(0.0, e, 0.0));
|
||||
|
||||
/* curl(ψ) = (∂ψC/∂y − ∂ψB/∂z, ∂ψA/∂z − ∂ψC/∂x, ∂ψB/∂x − ∂ψA/∂y) */
|
||||
return vec3(
|
||||
dy_pC - dz_pB,
|
||||
dz_pA - dx_pC,
|
||||
dx_pB - dy_pA
|
||||
) / (2.0 * e);
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
precision mediump float;
|
||||
varying vec3 vColor;
|
||||
varying float vLife;
|
||||
varying float vAlpha;
|
||||
uniform bool useTriangles;
|
||||
|
||||
void main() {
|
||||
float alpha = vAlpha;
|
||||
if (!useTriangles) {
|
||||
vec2 center = gl_PointCoord - 0.5;
|
||||
float d = length(center);
|
||||
alpha *= 1.0 - smoothstep(0.35, 0.5, d);
|
||||
}
|
||||
if (alpha < 0.01) discard;
|
||||
gl_FragColor = vec4(vColor, alpha);
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
uniform sampler2D texturePosition;
|
||||
uniform vec2 texSize;
|
||||
uniform vec3 color1;
|
||||
uniform vec3 color2;
|
||||
uniform float particleSize;
|
||||
uniform bool useTriangles;
|
||||
|
||||
attribute float particleID;
|
||||
attribute float vertexOffset;
|
||||
|
||||
varying vec3 vColor;
|
||||
varying float vLife;
|
||||
varying float vAlpha;
|
||||
|
||||
void main() {
|
||||
float pid = useTriangles ? floor(particleID / 3.0) : particleID;
|
||||
vec2 uv = (vec2(mod(pid, texSize.x), floor(pid / texSize.x)) + 0.5) / texSize;
|
||||
vec4 state = texture2D(texturePosition, uv);
|
||||
vec3 pos = state.xyz;
|
||||
float life = state.w;
|
||||
|
||||
vLife = life;
|
||||
vColor = mix(color2, color1, smoothstep(0.0, 0.7, life));
|
||||
vAlpha = smoothstep(0.0, 0.2, life) * smoothstep(1.0, 0.7, life);
|
||||
|
||||
if (useTriangles) {
|
||||
float angle = vertexOffset * 2.09439510239; /* 2π/3 per vertex */
|
||||
float radius = particleSize * 0.012 * smoothstep(0.0, 0.2, life);
|
||||
vec3 offset = vec3(cos(angle), sin(angle), 0.0) * radius;
|
||||
pos += offset;
|
||||
}
|
||||
|
||||
vec4 mvPos = modelViewMatrix * vec4(pos, 1.0);
|
||||
gl_PointSize = useTriangles
|
||||
? 1.0
|
||||
: max(1.0, particleSize * 1300.0 / -mvPos.z * smoothstep(0.0, 0.2, life));
|
||||
gl_Position = projectionMatrix * mvPos;
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
/* position.glsl — Spirit GPGPU position update.
|
||||
* curl.glsl is prepended to this source by spiritSim.ts before shader compilation.
|
||||
*/
|
||||
precision highp float;
|
||||
|
||||
uniform sampler2D texturePosition; /* previous [x, y, z, life] */
|
||||
uniform sampler2D textureDefaultPosition; /* spawn positions [x, y, z, 1] */
|
||||
uniform vec2 texSize;
|
||||
uniform float time;
|
||||
uniform float speed;
|
||||
uniform float dieSpeed;
|
||||
uniform float curlSize;
|
||||
uniform float attraction;
|
||||
uniform vec3 attractorPos;
|
||||
uniform float initBlend;
|
||||
|
||||
void main() {
|
||||
vec2 uv = gl_FragCoord.xy / texSize;
|
||||
vec4 pos4 = texture2D(texturePosition, uv);
|
||||
vec4 defPos4 = texture2D(textureDefaultPosition, uv);
|
||||
vec3 pos = pos4.xyz;
|
||||
float life = pos4.w;
|
||||
|
||||
/* Respawn dead particle from default position */
|
||||
if (life <= 0.0) {
|
||||
gl_FragColor = vec4(defPos4.xyz, 1.0);
|
||||
return;
|
||||
}
|
||||
|
||||
/* Smooth blend from spawn point toward live position */
|
||||
float blend = clamp(1.0 - life + initBlend, 0.0, 1.0);
|
||||
pos = mix(defPos4.xyz, pos, blend);
|
||||
|
||||
/* Curl noise displacement */
|
||||
vec3 cn = curlNoise(pos * curlSize + vec3(time * 0.05));
|
||||
pos += cn * speed * 0.01;
|
||||
|
||||
/* Attractor force */
|
||||
vec3 toAttr = attractorPos - pos;
|
||||
float dist = length(toAttr) + 0.001;
|
||||
pos += normalize(toAttr) * attraction * speed * 0.005 * (1.0 + life * 0.5);
|
||||
|
||||
life -= dieSpeed * speed;
|
||||
|
||||
gl_FragColor = vec4(pos, life);
|
||||
}
|
||||
@@ -0,0 +1,374 @@
|
||||
/**
|
||||
* spiritSim.ts — Three.js GPGPU Spirit particle simulation.
|
||||
*
|
||||
* Two WebGLRenderTarget ping-pong textures hold [x, y, z, life] per particle.
|
||||
* A full-screen GPGPU quad updates positions each frame.
|
||||
* Particles are rendered as points or triangles with UnrealBloomPass + AfterimagePass.
|
||||
* The React component (SpiritVisualizerCanvas.tsx) owns the RAF loop.
|
||||
*/
|
||||
|
||||
import * as THREE from 'three'
|
||||
import { EffectComposer } from 'three/addons/postprocessing/EffectComposer.js'
|
||||
import { RenderPass } from 'three/addons/postprocessing/RenderPass.js'
|
||||
import { UnrealBloomPass } from 'three/addons/postprocessing/UnrealBloomPass.js'
|
||||
import { AfterimagePass } from 'three/addons/postprocessing/AfterimagePass.js'
|
||||
import curlGLSL from './spirit/curl.glsl?raw'
|
||||
import positionGLSL from './spirit/position.glsl?raw'
|
||||
import particleVert from './spirit/particle.vert?raw'
|
||||
import particleFrag from './spirit/particle.frag?raw'
|
||||
|
||||
// ── Public types ─────────────────────────────────────────────────────────────
|
||||
|
||||
export interface SpiritConfig {
|
||||
/** GPGPU texture size; total particles = texSize². 256 or 512. */
|
||||
texSize: number
|
||||
speed: number
|
||||
dieSpeed: number
|
||||
radius: number
|
||||
curlSize: number
|
||||
attraction: number
|
||||
useTriangles: boolean
|
||||
bloomStrength: number
|
||||
bloomRadius: number
|
||||
bloomThreshold: number
|
||||
afterimageDamp: number
|
||||
color1: THREE.Color
|
||||
color2: THREE.Color
|
||||
}
|
||||
|
||||
export interface SpiritFrameUniforms {
|
||||
time: number
|
||||
speed: number
|
||||
dieSpeed: number
|
||||
curlSize: number
|
||||
attraction: number
|
||||
attractorPos: [number, number, number]
|
||||
bloomStrength: number
|
||||
bloomThreshold: number
|
||||
afterimageDamp: number
|
||||
color1: THREE.Color
|
||||
color2: THREE.Color
|
||||
}
|
||||
|
||||
// ── Internal state ────────────────────────────────────────────────────────────
|
||||
|
||||
interface SpiritInstance {
|
||||
renderer: THREE.WebGLRenderer
|
||||
scene: THREE.Scene
|
||||
camera: THREE.PerspectiveCamera
|
||||
composer: EffectComposer
|
||||
bloomPass: UnrealBloomPass
|
||||
afterimagePass: AfterimagePass
|
||||
gpuScene: THREE.Scene
|
||||
gpuCamera: THREE.OrthographicCamera
|
||||
gpuMat: THREE.ShaderMaterial
|
||||
/** readRT = source for GPGPU; writeRT = destination. Swapped each frame. */
|
||||
readRT: THREE.WebGLRenderTarget
|
||||
writeRT: THREE.WebGLRenderTarget
|
||||
defaultTex: THREE.DataTexture
|
||||
particleMesh: THREE.Points | THREE.Mesh
|
||||
particleMat: THREE.ShaderMaterial
|
||||
gpuMesh: THREE.Mesh
|
||||
cfg: SpiritConfig
|
||||
numParticles: number
|
||||
lastW: number
|
||||
lastH: number
|
||||
destroyed: boolean
|
||||
onContextLost: () => void
|
||||
onContextRestored: () => void
|
||||
}
|
||||
|
||||
const instances = new WeakMap<HTMLCanvasElement, SpiritInstance>()
|
||||
|
||||
// ── Helpers ──────────────────────────────────────────────────────────────────
|
||||
|
||||
function makeDefaultPositions(n: number, radius: number): Float32Array {
|
||||
const arr = new Float32Array(n * 4)
|
||||
for (let i = 0; i < n; i++) {
|
||||
const theta = Math.random() * Math.PI * 2
|
||||
const phi = Math.acos(2 * Math.random() - 1)
|
||||
const r = radius * (0.3 + 0.7 * Math.random())
|
||||
arr[i * 4 + 0] = r * Math.sin(phi) * Math.cos(theta)
|
||||
arr[i * 4 + 1] = r * Math.sin(phi) * Math.sin(theta)
|
||||
arr[i * 4 + 2] = r * Math.cos(phi)
|
||||
arr[i * 4 + 3] = Math.random() // initial life
|
||||
}
|
||||
return arr
|
||||
}
|
||||
|
||||
function makeRtOpts(): THREE.RenderTargetOptions {
|
||||
return {
|
||||
type: THREE.FloatType,
|
||||
format: THREE.RGBAFormat,
|
||||
minFilter: THREE.NearestFilter,
|
||||
magFilter: THREE.NearestFilter,
|
||||
depthBuffer: false,
|
||||
stencilBuffer: false,
|
||||
}
|
||||
}
|
||||
|
||||
// ── Public API ────────────────────────────────────────────────────────────────
|
||||
|
||||
export function initSpiritSim(canvas: HTMLCanvasElement, cfg: SpiritConfig): boolean {
|
||||
destroySpiritSim(canvas)
|
||||
|
||||
let renderer: THREE.WebGLRenderer
|
||||
try {
|
||||
renderer = new THREE.WebGLRenderer({ canvas, antialias: false, alpha: false })
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
|
||||
// Check float render target support
|
||||
// Three.js >= 0.159 requires WebGL2, which guarantees float fragment textures.
|
||||
// If WebGL2 is somehow unavailable, bail out.
|
||||
if (!renderer.capabilities.isWebGL2) {
|
||||
renderer.dispose()
|
||||
return false
|
||||
}
|
||||
|
||||
renderer.setPixelRatio(Math.min(window.devicePixelRatio, 2))
|
||||
const w = Math.max(1, canvas.clientWidth || canvas.width)
|
||||
const h = Math.max(1, canvas.clientHeight || canvas.height)
|
||||
renderer.setSize(w, h)
|
||||
|
||||
const scene = new THREE.Scene()
|
||||
const camera = new THREE.PerspectiveCamera(60, w / h, 0.1, 100)
|
||||
camera.position.set(0, 0, 3.5)
|
||||
|
||||
const { texSize } = cfg
|
||||
const numParticles = texSize * texSize
|
||||
const rtOpts = makeRtOpts()
|
||||
const readRT = new THREE.WebGLRenderTarget(texSize, texSize, rtOpts)
|
||||
const writeRT = new THREE.WebGLRenderTarget(texSize, texSize, rtOpts)
|
||||
|
||||
// Spawn default positions
|
||||
const defData = makeDefaultPositions(numParticles, cfg.radius)
|
||||
const defaultTex = new THREE.DataTexture(defData, texSize, texSize, THREE.RGBAFormat, THREE.FloatType)
|
||||
defaultTex.needsUpdate = true // SE10: must set or GPU sees zeros
|
||||
|
||||
// Seed readRT with default positions (bootstrap first frame)
|
||||
const seedRenderer = new THREE.WebGLRenderer({ canvas: document.createElement('canvas') })
|
||||
const seedScene = new THREE.Scene()
|
||||
const seedCamera = new THREE.OrthographicCamera(-1, 1, 1, -1, 0, 1)
|
||||
const seedMat = new THREE.ShaderMaterial({
|
||||
uniforms: { tex: { value: defaultTex } },
|
||||
vertexShader: 'void main() { gl_Position = vec4(position, 1.0); }',
|
||||
fragmentShader: `
|
||||
precision highp float;
|
||||
uniform sampler2D tex;
|
||||
void main() { gl_FragColor = texture2D(tex, vec2(gl_FragCoord.x / ${texSize.toFixed(1)}, gl_FragCoord.y / ${texSize.toFixed(1)})); }
|
||||
`,
|
||||
})
|
||||
const seedGeo = new THREE.PlaneGeometry(2, 2)
|
||||
seedScene.add(new THREE.Mesh(seedGeo, seedMat))
|
||||
seedRenderer.setSize(texSize, texSize)
|
||||
seedRenderer.setRenderTarget(readRT)
|
||||
seedRenderer.render(seedScene, seedCamera)
|
||||
seedRenderer.setRenderTarget(null)
|
||||
seedRenderer.dispose()
|
||||
seedMat.dispose()
|
||||
seedGeo.dispose()
|
||||
|
||||
// GPGPU scene — full-screen quad to update particle positions
|
||||
const gpuScene = new THREE.Scene()
|
||||
const gpuCamera = new THREE.OrthographicCamera(-1, 1, 1, -1, 0, 1)
|
||||
const gpuGeo = new THREE.PlaneGeometry(2, 2)
|
||||
const gpuMat = new THREE.ShaderMaterial({
|
||||
uniforms: {
|
||||
texturePosition: { value: readRT.texture },
|
||||
textureDefaultPosition: { value: defaultTex },
|
||||
texSize: { value: new THREE.Vector2(texSize, texSize) },
|
||||
time: { value: 0 },
|
||||
speed: { value: cfg.speed },
|
||||
dieSpeed: { value: cfg.dieSpeed },
|
||||
curlSize: { value: cfg.curlSize },
|
||||
attraction: { value: cfg.attraction },
|
||||
attractorPos: { value: new THREE.Vector3(0, 0, 0) },
|
||||
initBlend: { value: 1.0 },
|
||||
},
|
||||
vertexShader: 'void main() { gl_Position = vec4(position, 1.0); }',
|
||||
// curl.glsl prepended so curlNoise() is available in position.glsl
|
||||
fragmentShader: curlGLSL + '\n' + positionGLSL,
|
||||
})
|
||||
const gpuMesh = new THREE.Mesh(gpuGeo, gpuMat)
|
||||
gpuScene.add(gpuMesh)
|
||||
|
||||
// Particle geometry
|
||||
const useTriangles = cfg.useTriangles
|
||||
const vertCount = useTriangles ? numParticles * 3 : numParticles
|
||||
const pidArr = new Float32Array(vertCount)
|
||||
const offsetArr = new Float32Array(vertCount)
|
||||
if (useTriangles) {
|
||||
for (let i = 0; i < numParticles; i++) {
|
||||
for (let v = 0; v < 3; v++) {
|
||||
pidArr[i * 3 + v] = i * 3 + v
|
||||
offsetArr[i * 3 + v] = v
|
||||
}
|
||||
}
|
||||
} else {
|
||||
for (let i = 0; i < numParticles; i++) { pidArr[i] = i; offsetArr[i] = 0 }
|
||||
}
|
||||
|
||||
const geo = new THREE.BufferGeometry()
|
||||
geo.setAttribute('particleID', new THREE.BufferAttribute(pidArr, 1))
|
||||
geo.setAttribute('vertexOffset', new THREE.BufferAttribute(offsetArr, 1))
|
||||
|
||||
const particleMat = new THREE.ShaderMaterial({
|
||||
uniforms: {
|
||||
texturePosition: { value: readRT.texture },
|
||||
texSize: { value: new THREE.Vector2(texSize, texSize) },
|
||||
color1: { value: cfg.color1.clone() },
|
||||
color2: { value: cfg.color2.clone() },
|
||||
particleSize: { value: 1.0 },
|
||||
useTriangles: { value: useTriangles },
|
||||
},
|
||||
vertexShader: particleVert,
|
||||
fragmentShader: particleFrag,
|
||||
transparent: true,
|
||||
depthWrite: false,
|
||||
blending: THREE.AdditiveBlending,
|
||||
})
|
||||
|
||||
let particleMesh: THREE.Points | THREE.Mesh
|
||||
if (useTriangles) {
|
||||
particleMesh = new THREE.Mesh(geo, particleMat)
|
||||
} else {
|
||||
particleMesh = new THREE.Points(geo, particleMat)
|
||||
}
|
||||
scene.add(particleMesh)
|
||||
|
||||
// Post-processing
|
||||
const composer = new EffectComposer(renderer)
|
||||
composer.addPass(new RenderPass(scene, camera))
|
||||
const bloomPass = new UnrealBloomPass(
|
||||
new THREE.Vector2(w, h),
|
||||
cfg.bloomStrength,
|
||||
cfg.bloomRadius,
|
||||
cfg.bloomThreshold,
|
||||
)
|
||||
composer.addPass(bloomPass)
|
||||
const afterimagePass = new AfterimagePass(cfg.afterimageDamp)
|
||||
composer.addPass(afterimagePass)
|
||||
|
||||
const inst: SpiritInstance = {
|
||||
renderer, scene, camera, composer, bloomPass, afterimagePass,
|
||||
gpuScene, gpuCamera, gpuMat, gpuMesh,
|
||||
readRT, writeRT, defaultTex,
|
||||
particleMesh, particleMat,
|
||||
cfg: { ...cfg },
|
||||
numParticles,
|
||||
lastW: w, lastH: h,
|
||||
destroyed: false,
|
||||
onContextLost: () => {},
|
||||
onContextRestored: () => {},
|
||||
}
|
||||
|
||||
inst.onContextLost = () => {
|
||||
inst.destroyed = true
|
||||
}
|
||||
inst.onContextRestored = () => {
|
||||
if (!instances.has(canvas)) return
|
||||
inst.destroyed = true
|
||||
initSpiritSim(canvas, inst.cfg)
|
||||
}
|
||||
canvas.addEventListener('webglcontextlost', inst.onContextLost)
|
||||
canvas.addEventListener('webglcontextrestored', inst.onContextRestored)
|
||||
|
||||
instances.set(canvas, inst)
|
||||
return true
|
||||
}
|
||||
|
||||
export function renderSpiritFrame(canvas: HTMLCanvasElement, u: SpiritFrameUniforms): void {
|
||||
const inst = instances.get(canvas)
|
||||
if (!inst || inst.destroyed) return
|
||||
|
||||
const { renderer, gpuMat, gpuScene, gpuCamera, particleMat, composer } = inst
|
||||
const w = Math.max(1, Math.round(canvas.clientWidth || 0) || canvas.width)
|
||||
const h = Math.max(1, Math.round(canvas.clientHeight || 0) || canvas.height)
|
||||
|
||||
if (canvas.width !== w || canvas.height !== h) {
|
||||
canvas.width = w
|
||||
canvas.height = h
|
||||
}
|
||||
|
||||
// Resize when canvas changes (SE2, SE5)
|
||||
if (inst.lastW !== w || inst.lastH !== h) {
|
||||
renderer.setSize(w, h)
|
||||
inst.camera.aspect = w / h
|
||||
inst.camera.updateProjectionMatrix()
|
||||
inst.composer.setSize(w, h)
|
||||
inst.bloomPass.setSize(w, h)
|
||||
inst.lastW = w
|
||||
inst.lastH = h
|
||||
}
|
||||
|
||||
// ── GPGPU position update ─────────────────────────────────────────────────
|
||||
// Read from readRT → write to writeRT (SE9: never alias same target)
|
||||
gpuMat.uniforms.texturePosition.value = inst.readRT.texture
|
||||
gpuMat.uniforms.time.value = u.time
|
||||
gpuMat.uniforms.speed.value = u.speed
|
||||
gpuMat.uniforms.dieSpeed.value = u.dieSpeed
|
||||
gpuMat.uniforms.curlSize.value = u.curlSize
|
||||
gpuMat.uniforms.attraction.value = u.attraction
|
||||
;(gpuMat.uniforms.attractorPos.value as THREE.Vector3).set(
|
||||
u.attractorPos[0], u.attractorPos[1], u.attractorPos[2],
|
||||
)
|
||||
|
||||
renderer.setRenderTarget(inst.writeRT)
|
||||
renderer.render(gpuScene, gpuCamera)
|
||||
renderer.setRenderTarget(null)
|
||||
|
||||
// Swap ping-pong (SE9: ensure read/write never alias)
|
||||
const tmp = inst.readRT
|
||||
inst.readRT = inst.writeRT
|
||||
inst.writeRT = tmp
|
||||
|
||||
// ── Update particle material uniforms ─────────────────────────────────────
|
||||
particleMat.uniforms.texturePosition.value = inst.readRT.texture
|
||||
particleMat.uniforms.color1.value.copy(u.color1)
|
||||
particleMat.uniforms.color2.value.copy(u.color2)
|
||||
|
||||
// ── Post-processing uniforms ──────────────────────────────────────────────
|
||||
inst.bloomPass.strength = clamp(u.bloomStrength, 0, 4)
|
||||
inst.bloomPass.threshold = u.bloomThreshold
|
||||
// SE3: clamp afterimage below 1.0 to prevent white-out
|
||||
inst.afterimagePass.uniforms['damp'].value = clamp(u.afterimageDamp, 0.60, 0.96)
|
||||
|
||||
// ── Composite render ──────────────────────────────────────────────────────
|
||||
composer.render()
|
||||
}
|
||||
|
||||
export function destroySpiritSim(canvas: HTMLCanvasElement): void {
|
||||
const inst = instances.get(canvas)
|
||||
if (!inst) return
|
||||
|
||||
inst.destroyed = true
|
||||
canvas.removeEventListener('webglcontextlost', inst.onContextLost)
|
||||
canvas.removeEventListener('webglcontextrestored', inst.onContextRestored)
|
||||
|
||||
// SE8: dispose all Three.js resources
|
||||
try {
|
||||
inst.readRT.dispose()
|
||||
inst.writeRT.dispose()
|
||||
inst.defaultTex.dispose()
|
||||
inst.gpuMat.dispose()
|
||||
inst.gpuMesh.geometry.dispose()
|
||||
if (inst.particleMesh.geometry) inst.particleMesh.geometry.dispose()
|
||||
inst.particleMat.dispose()
|
||||
inst.bloomPass.dispose()
|
||||
inst.afterimagePass.dispose()
|
||||
inst.composer.dispose()
|
||||
inst.renderer.dispose()
|
||||
} catch { /* context may already be lost */ }
|
||||
|
||||
instances.delete(canvas)
|
||||
}
|
||||
|
||||
// ── Helpers ───────────────────────────────────────────────────────────────────
|
||||
|
||||
function clamp(v: number, lo: number, hi: number): number {
|
||||
return Math.max(lo, Math.min(hi, v))
|
||||
}
|
||||
|
||||
// Extend SpiritInstance type to allow cfg.cfg pattern (workaround for bloom clamp)
|
||||
@@ -0,0 +1,3 @@
|
||||
precision mediump float;
|
||||
uniform vec4 uFade; /* rgb = target colour, a = fade strength */
|
||||
void main() { gl_FragColor = uFade; }
|
||||
@@ -0,0 +1,10 @@
|
||||
precision mediump float;
|
||||
varying vec4 vState;
|
||||
varying float vIsHead;
|
||||
|
||||
void main() {
|
||||
if (vIsHead < 0.5) discard;
|
||||
vec2 vel = vState.zw;
|
||||
/* Encode velocity direction into [0,1] RG for logic.glsl to decode */
|
||||
gl_FragColor = vec4(vel * 0.5 + 0.5, 0.0, 0.08);
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
precision highp float;
|
||||
|
||||
uniform sampler2D statePrev; /* previous frame [posX, posY, velX, velY] */
|
||||
uniform sampler2D flowTex; /* flow feedback field */
|
||||
uniform vec2 stateSize; /* rootNum × rootNum */
|
||||
uniform float time;
|
||||
uniform float damping;
|
||||
uniform float speedLimit;
|
||||
uniform float forceWeight;
|
||||
uniform float noiseWeight;
|
||||
uniform float noiseScale;
|
||||
uniform float noiseSpeed;
|
||||
uniform float flowWeight;
|
||||
uniform vec2 target;
|
||||
uniform float targetWeight;
|
||||
|
||||
/* ── Inline 2D simplex noise (Ashima Arts) ─────────────────────────────── */
|
||||
vec3 _perm(vec3 x) { return mod(((x * 34.0) + 1.0) * x, 289.0); }
|
||||
float snoise2(vec2 v) {
|
||||
const vec4 C = vec4(0.211324865405187, 0.366025403784439,
|
||||
-0.577350269189626, 0.024390243902439);
|
||||
vec2 i = floor(v + dot(v, C.yy));
|
||||
vec2 x0 = v - i + dot(i, C.xx);
|
||||
vec2 i1 = (x0.x > x0.y) ? vec2(1.0, 0.0) : vec2(0.0, 1.0);
|
||||
vec4 x12 = x0.xyxy + C.xxzz;
|
||||
x12.xy -= i1;
|
||||
i = mod(i, 289.0);
|
||||
vec3 p = _perm(_perm(i.y + vec3(0.0, i1.y, 1.0)) + i.x + vec3(0.0, i1.x, 1.0));
|
||||
vec3 m = max(0.5 - vec3(dot(x0,x0), dot(x12.xy,x12.xy), dot(x12.zw,x12.zw)), 0.0);
|
||||
m = m * m * m * m;
|
||||
vec3 x = 2.0 * fract(p * C.www) - 1.0;
|
||||
vec3 h = abs(x) - 0.5;
|
||||
vec3 ox = floor(x + 0.5);
|
||||
vec3 a0 = x - ox;
|
||||
m *= 1.79284291400159 - 0.85373472095314 * (a0*a0 + h*h);
|
||||
vec3 g;
|
||||
g.x = a0.x * x0.x + h.x * x0.y;
|
||||
g.yz = a0.yz * x12.xz + h.yz * x12.yw;
|
||||
return 130.0 * dot(m, g);
|
||||
}
|
||||
/* ───────────────────────────────────────────────────────────────────────── */
|
||||
|
||||
void main() {
|
||||
vec2 uv = gl_FragCoord.xy / stateSize;
|
||||
vec4 s = texture2D(statePrev, uv);
|
||||
vec2 pos = s.xy;
|
||||
vec2 vel = s.zw;
|
||||
|
||||
/* 1. Noise force */
|
||||
float t = time * noiseSpeed;
|
||||
float nx = snoise2(vec2(pos.x * noiseScale + t, pos.y * noiseScale));
|
||||
float ny = snoise2(vec2(pos.y * noiseScale, pos.x * noiseScale + t + 31.41));
|
||||
vec2 noiseF = vec2(nx, ny) * noiseWeight;
|
||||
|
||||
/* 2. Flow feedback */
|
||||
vec2 uvFlow = clamp(pos * 0.5 + 0.5, 0.0, 1.0);
|
||||
vec2 flowSmp = texture2D(flowTex, uvFlow).xy;
|
||||
vec2 flowF = (flowSmp * 2.0 - 1.0) * flowWeight;
|
||||
|
||||
/* 3. Target attraction */
|
||||
vec2 toTarget = target - pos;
|
||||
vec2 targetF = toTarget * targetWeight;
|
||||
|
||||
/* Euler integration */
|
||||
vel = vel * (1.0 - damping) + (noiseF + flowF + targetF) * forceWeight;
|
||||
float spd = length(vel);
|
||||
if (spd > speedLimit) vel *= speedLimit / spd;
|
||||
pos += vel;
|
||||
|
||||
/* Soft boundary bounce */
|
||||
if (abs(pos.x) > 1.1) { vel.x *= -0.5; pos.x = clamp(pos.x, -1.1, 1.1); }
|
||||
if (abs(pos.y) > 1.1) { vel.y *= -0.5; pos.y = clamp(pos.y, -1.1, 1.1); }
|
||||
|
||||
gl_FragColor = vec4(pos, vel);
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
precision mediump float;
|
||||
uniform vec3 baseColor;
|
||||
uniform vec3 flowColor;
|
||||
uniform float colorMapAlpha;
|
||||
uniform float speedAlpha;
|
||||
|
||||
varying vec4 vState;
|
||||
varying float vIsHead;
|
||||
|
||||
void main() {
|
||||
vec2 vel = vState.zw;
|
||||
float speed = length(vel);
|
||||
|
||||
float a = smoothstep(0.0, max(0.00001, speedAlpha), speed);
|
||||
a *= (vIsHead > 0.5) ? 1.0 : 0.25;
|
||||
|
||||
vec2 dir = speed > 0.0001 ? vel / speed : vec2(0.0);
|
||||
vec3 vCol = vec3(dir.x * 0.5 + 0.5, dir.y * 0.5 + 0.5, length(dir) * 0.5);
|
||||
vec3 color = mix(baseColor, vCol + flowColor, colorMapAlpha);
|
||||
|
||||
gl_FragColor = vec4(color, a);
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
precision highp float;
|
||||
attribute float vid; /* 0 … numParticles*2-1 */
|
||||
|
||||
uniform sampler2D stateCurr; /* [posX, posY, velX, velY] — this frame */
|
||||
uniform sampler2D statePrev; /* previous frame */
|
||||
uniform vec2 stateSize;
|
||||
uniform float aspect;
|
||||
|
||||
varying vec4 vState;
|
||||
varying float vIsHead;
|
||||
|
||||
vec2 idxToUV(float idx) {
|
||||
return (vec2(mod(idx, stateSize.x), floor(idx / stateSize.x)) + 0.5) / stateSize;
|
||||
}
|
||||
|
||||
void main() {
|
||||
float pidx = floor(vid / 2.0);
|
||||
vIsHead = mod(vid, 2.0);
|
||||
vec2 uv = idxToUV(pidx);
|
||||
vState = texture2D(stateCurr, uv);
|
||||
vec4 prev = texture2D(statePrev, uv);
|
||||
vec2 pos = (vIsHead > 0.5) ? vState.xy : prev.xy;
|
||||
pos.x /= max(1.0, aspect);
|
||||
gl_Position = vec4(pos, 0.0, 1.0);
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
attribute vec2 aPos; /* quad corners: ±1 */
|
||||
void main() { gl_Position = vec4(aPos, 0.0, 1.0); }
|
||||
@@ -0,0 +1,419 @@
|
||||
/**
|
||||
* tendrilsSim.ts — WebGL2 GPGPU tendril particle simulation.
|
||||
*
|
||||
* Architecture:
|
||||
* - Two RGBA32F ping-pong textures (readTex / writeTex) hold [posX, posY, velX, velY].
|
||||
* - A GPGPU logic pass writes new positions/velocities to writeFBO each frame.
|
||||
* - Particles are drawn as GL_LINES (tail=readTex prev, head=readTex new) onto the canvas.
|
||||
* - A flow FBO accumulates particle velocity directions; logic.glsl reads it back as force.
|
||||
* - The React component (TendrilsVisualizerCanvas.tsx) owns the RAF loop and calls
|
||||
* tickTendrils() once per frame — this module never calls requestAnimationFrame.
|
||||
*/
|
||||
|
||||
import screenVert from './tendrils/screen.vert?raw'
|
||||
import logicGLSL from './tendrils/logic.glsl?raw'
|
||||
import renderVert from './tendrils/render.vert?raw'
|
||||
import renderFrag from './tendrils/render.frag?raw'
|
||||
import flowFrag from './tendrils/flow.frag?raw'
|
||||
import fadeFrag from './tendrils/fade.frag?raw'
|
||||
|
||||
// ── Public types ────────────────────────────────────────────────────────────
|
||||
|
||||
export interface TendrilsConfig {
|
||||
/** GPGPU grid size; total particles = rootNum². 256 or 512. */
|
||||
rootNum: number
|
||||
}
|
||||
|
||||
export interface TendrilsFrameUniforms {
|
||||
time: number
|
||||
damping: number
|
||||
speedLimit: number
|
||||
forceWeight: number
|
||||
noiseWeight: number
|
||||
noiseScale: number
|
||||
noiseSpeed: number
|
||||
flowWeight: number
|
||||
flowDecay: number
|
||||
targetWeight: number
|
||||
target: [number, number] // NDC
|
||||
trailAlpha: number // canvas fade per frame
|
||||
colorMapAlpha: number
|
||||
speedAlpha: number
|
||||
baseColor: [number, number, number]
|
||||
flowColor: [number, number, number]
|
||||
bgColor: [number, number, number]
|
||||
lineWidth?: number
|
||||
}
|
||||
|
||||
// ── Internal state ───────────────────────────────────────────────────────────
|
||||
|
||||
interface TendrilsGLState {
|
||||
gl: WebGLRenderingContext
|
||||
logicProg: WebGLProgram
|
||||
renderProg: WebGLProgram
|
||||
flowProg: WebGLProgram
|
||||
fadeProg: WebGLProgram
|
||||
readTex: WebGLTexture
|
||||
writeTex: WebGLTexture
|
||||
readFBO: WebGLFramebuffer
|
||||
writeFBO: WebGLFramebuffer
|
||||
flowFBO: WebGLFramebuffer
|
||||
flowTex: WebGLTexture
|
||||
flowW: number
|
||||
flowH: number
|
||||
vidBuf: WebGLBuffer
|
||||
quadBuf: WebGLBuffer
|
||||
cfg: TendrilsConfig
|
||||
numParticles: number
|
||||
destroyed: boolean
|
||||
onContextLost: (e: Event) => void
|
||||
onContextRestored: () => void
|
||||
}
|
||||
|
||||
const instances = new WeakMap<HTMLCanvasElement, TendrilsGLState>()
|
||||
|
||||
// ── Shader compilation ───────────────────────────────────────────────────────
|
||||
|
||||
function compileShader(gl: WebGLRenderingContext, type: number, src: string): WebGLShader {
|
||||
const shader = gl.createShader(type)!
|
||||
gl.shaderSource(shader, src)
|
||||
gl.compileShader(shader)
|
||||
if (!gl.getShaderParameter(shader, gl.COMPILE_STATUS)) {
|
||||
const info = gl.getShaderInfoLog(shader) ?? 'unknown'
|
||||
gl.deleteShader(shader)
|
||||
throw new Error(`Shader compile error:\n${info}\n\nSource:\n${src.slice(0, 300)}`)
|
||||
}
|
||||
return shader
|
||||
}
|
||||
|
||||
function createProgram(gl: WebGLRenderingContext, vertSrc: string, fragSrc: string): WebGLProgram {
|
||||
const prog = gl.createProgram()!
|
||||
gl.attachShader(prog, compileShader(gl, gl.VERTEX_SHADER, vertSrc))
|
||||
gl.attachShader(prog, compileShader(gl, gl.FRAGMENT_SHADER, fragSrc))
|
||||
gl.linkProgram(prog)
|
||||
if (!gl.getProgramParameter(prog, gl.LINK_STATUS)) {
|
||||
const info = gl.getProgramInfoLog(prog) ?? 'unknown'
|
||||
gl.deleteProgram(prog)
|
||||
throw new Error(`Program link error: ${info}`)
|
||||
}
|
||||
return prog
|
||||
}
|
||||
|
||||
// ── Texture / FBO helpers ────────────────────────────────────────────────────
|
||||
|
||||
function createStateTexture(gl: WebGLRenderingContext, rootNum: number): WebGLTexture {
|
||||
const tex = gl.createTexture()!
|
||||
gl.bindTexture(gl.TEXTURE_2D, tex)
|
||||
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST)
|
||||
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST)
|
||||
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE)
|
||||
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE)
|
||||
|
||||
// Seed with random positions in NDC [-1.0, 1.0] and zero velocity
|
||||
const data = new Float32Array(rootNum * rootNum * 4)
|
||||
for (let i = 0; i < rootNum * rootNum; i++) {
|
||||
data[i * 4 + 0] = (Math.random() * 2 - 1) * 0.9 // posX
|
||||
data[i * 4 + 1] = (Math.random() * 2 - 1) * 0.9 // posY
|
||||
data[i * 4 + 2] = 0 // velX
|
||||
data[i * 4 + 3] = 0 // velY
|
||||
}
|
||||
// WebGL2 uses texImage2D with RGBA32F internal format
|
||||
const gl2 = gl as WebGL2RenderingContext
|
||||
gl2.texImage2D(gl2.TEXTURE_2D, 0, gl2.RGBA32F, rootNum, rootNum, 0, gl2.RGBA, gl2.FLOAT, data)
|
||||
gl.bindTexture(gl.TEXTURE_2D, null)
|
||||
return tex
|
||||
}
|
||||
|
||||
function createFlowTexture(gl: WebGLRenderingContext, w: number, h: number): WebGLTexture {
|
||||
const tex = gl.createTexture()!
|
||||
gl.bindTexture(gl.TEXTURE_2D, tex)
|
||||
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR)
|
||||
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR)
|
||||
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE)
|
||||
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE)
|
||||
// Half-float for flow FBO: RGBA16F (requires EXT_color_buffer_float in WebGL2)
|
||||
const gl2 = gl as WebGL2RenderingContext
|
||||
gl2.texImage2D(gl2.TEXTURE_2D, 0, gl2.RGBA16F, w, h, 0, gl2.RGBA, gl2.HALF_FLOAT, null)
|
||||
gl.bindTexture(gl.TEXTURE_2D, null)
|
||||
return tex
|
||||
}
|
||||
|
||||
function createFBO(gl: WebGLRenderingContext, tex: WebGLTexture): WebGLFramebuffer {
|
||||
const fbo = gl.createFramebuffer()!
|
||||
gl.bindFramebuffer(gl.FRAMEBUFFER, fbo)
|
||||
gl.framebufferTexture2D(gl.FRAMEBUFFER, gl.COLOR_ATTACHMENT0, gl.TEXTURE_2D, tex, 0)
|
||||
gl.bindFramebuffer(gl.FRAMEBUFFER, null)
|
||||
return fbo
|
||||
}
|
||||
|
||||
// ── Public API ───────────────────────────────────────────────────────────────
|
||||
|
||||
export function initTendrilsSim(canvas: HTMLCanvasElement, cfg: TendrilsConfig): boolean {
|
||||
destroyTendrilsSim(canvas)
|
||||
|
||||
// Prefer WebGL2; fall back to WebGL1 for broad compatibility
|
||||
const gl = (canvas.getContext('webgl2') ?? canvas.getContext('webgl')) as WebGLRenderingContext | null
|
||||
if (!gl) return false
|
||||
|
||||
// WebGL2 requires EXT_color_buffer_float for rendering to float textures
|
||||
const floatExt = gl.getExtension('EXT_color_buffer_float')
|
||||
if (!floatExt) {
|
||||
// Some WebGL1 devices support OES_texture_float_linear
|
||||
const oesFlt = gl.getExtension('OES_texture_float')
|
||||
if (!oesFlt) return false
|
||||
gl.getExtension('OES_texture_float_linear')
|
||||
}
|
||||
gl.getExtension('WEBGL_color_buffer_float') // WebGL1 fallback
|
||||
|
||||
let logicProg: WebGLProgram
|
||||
let renderProg: WebGLProgram
|
||||
let flowProg: WebGLProgram
|
||||
let fadeProg: WebGLProgram
|
||||
try {
|
||||
logicProg = createProgram(gl, screenVert, logicGLSL)
|
||||
renderProg = createProgram(gl, renderVert, renderFrag)
|
||||
flowProg = createProgram(gl, renderVert, flowFrag)
|
||||
fadeProg = createProgram(gl, screenVert, fadeFrag)
|
||||
} catch (e) {
|
||||
console.error('[TendrilsSim] Shader error:', e)
|
||||
return false
|
||||
}
|
||||
|
||||
const { rootNum } = cfg
|
||||
const numParticles = rootNum * rootNum
|
||||
|
||||
const readTex = createStateTexture(gl, rootNum)
|
||||
const writeTex = createStateTexture(gl, rootNum)
|
||||
const readFBO = createFBO(gl, readTex)
|
||||
const writeFBO = createFBO(gl, writeTex)
|
||||
|
||||
const w = Math.max(1, canvas.clientWidth || canvas.width)
|
||||
const h = Math.max(1, canvas.clientHeight || canvas.height)
|
||||
const flowTex = createFlowTexture(gl, w, h)
|
||||
const flowFBO = createFBO(gl, flowTex)
|
||||
|
||||
// vid buffer: sequential float IDs [0, 1, …, numParticles*2 - 1]
|
||||
const vidBuf = gl.createBuffer()!
|
||||
gl.bindBuffer(gl.ARRAY_BUFFER, vidBuf)
|
||||
const vidArr = new Float32Array(numParticles * 2)
|
||||
for (let i = 0; i < vidArr.length; i++) vidArr[i] = i
|
||||
gl.bufferData(gl.ARRAY_BUFFER, vidArr, gl.STATIC_DRAW)
|
||||
|
||||
// screen quad: two triangles covering [-1,-1] to [1,1]
|
||||
const quadBuf = gl.createBuffer()!
|
||||
gl.bindBuffer(gl.ARRAY_BUFFER, quadBuf)
|
||||
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([-1,-1, 1,-1, 1,1, -1,1]), gl.STATIC_DRAW)
|
||||
gl.bindBuffer(gl.ARRAY_BUFFER, null)
|
||||
|
||||
const state: TendrilsGLState = {
|
||||
gl, logicProg, renderProg, flowProg, fadeProg,
|
||||
readTex, writeTex, readFBO, writeFBO,
|
||||
flowFBO, flowTex,
|
||||
flowW: w, flowH: h,
|
||||
vidBuf, quadBuf,
|
||||
cfg: { ...cfg },
|
||||
numParticles,
|
||||
destroyed: false,
|
||||
onContextLost: () => {},
|
||||
onContextRestored: () => {},
|
||||
}
|
||||
|
||||
state.onContextLost = (e: Event) => {
|
||||
e.preventDefault()
|
||||
state.destroyed = true
|
||||
}
|
||||
state.onContextRestored = () => {
|
||||
// Full re-init on restore; previous GL objects are invalid
|
||||
if (!instances.has(canvas)) return
|
||||
state.destroyed = true
|
||||
initTendrilsSim(canvas, state.cfg)
|
||||
}
|
||||
canvas.addEventListener('webglcontextlost', state.onContextLost)
|
||||
canvas.addEventListener('webglcontextrestored', state.onContextRestored)
|
||||
|
||||
instances.set(canvas, state)
|
||||
return true
|
||||
}
|
||||
|
||||
// ── Per-frame rendering ──────────────────────────────────────────────────────
|
||||
|
||||
export function tickTendrils(canvas: HTMLCanvasElement, u: TendrilsFrameUniforms): void {
|
||||
const st = instances.get(canvas)
|
||||
if (!st || st.destroyed) return
|
||||
const { gl } = st
|
||||
|
||||
const w = Math.max(1, Math.round(canvas.clientWidth || 0) || canvas.width)
|
||||
const h = Math.max(1, Math.round(canvas.clientHeight || 0) || canvas.height)
|
||||
if (canvas.width !== w || canvas.height !== h) {
|
||||
canvas.width = w
|
||||
canvas.height = h
|
||||
}
|
||||
|
||||
// Resize flow FBO if canvas changed
|
||||
if (st.flowW !== w || st.flowH !== h) {
|
||||
gl.deleteFramebuffer(st.flowFBO)
|
||||
gl.deleteTexture(st.flowTex)
|
||||
st.flowTex = createFlowTexture(gl, w, h)
|
||||
st.flowFBO = createFBO(gl, st.flowTex)
|
||||
st.flowW = w
|
||||
st.flowH = h
|
||||
}
|
||||
|
||||
const { rootNum } = st.cfg
|
||||
const numP = st.numParticles
|
||||
const aspect = w / h
|
||||
|
||||
// ── 1. GPGPU logic pass ───────────────────────────────────────────────────
|
||||
gl.bindFramebuffer(gl.FRAMEBUFFER, st.writeFBO)
|
||||
gl.viewport(0, 0, rootNum, rootNum)
|
||||
gl.disable(gl.BLEND)
|
||||
|
||||
gl.useProgram(st.logicProg)
|
||||
setTex(gl, st.logicProg, 'statePrev', st.readTex, 0)
|
||||
setTex(gl, st.logicProg, 'flowTex', st.flowTex, 1)
|
||||
setVec2(gl, st.logicProg, 'stateSize', rootNum, rootNum)
|
||||
setFloat(gl, st.logicProg, 'time', u.time)
|
||||
setFloat(gl, st.logicProg, 'damping', u.damping)
|
||||
setFloat(gl, st.logicProg, 'speedLimit', u.speedLimit)
|
||||
setFloat(gl, st.logicProg, 'forceWeight', u.forceWeight)
|
||||
setFloat(gl, st.logicProg, 'noiseWeight', u.noiseWeight)
|
||||
setFloat(gl, st.logicProg, 'noiseScale', u.noiseScale)
|
||||
setFloat(gl, st.logicProg, 'noiseSpeed', u.noiseSpeed)
|
||||
setFloat(gl, st.logicProg, 'flowWeight', u.flowWeight)
|
||||
setVec2(gl, st.logicProg, 'target', u.target[0], u.target[1])
|
||||
setFloat(gl, st.logicProg, 'targetWeight', u.targetWeight)
|
||||
drawQuad(gl, st.logicProg, st.quadBuf)
|
||||
|
||||
// ── 2. Swap ping-pong ─────────────────────────────────────────────────────
|
||||
// After swap: readTex = current frame, writeTex = previous frame (for tail)
|
||||
;[st.readTex, st.writeTex] = [st.writeTex, st.readTex]
|
||||
;[st.readFBO, st.writeFBO] = [st.writeFBO, st.readFBO]
|
||||
|
||||
// ── 3. Flow fade pass ─────────────────────────────────────────────────────
|
||||
gl.bindFramebuffer(gl.FRAMEBUFFER, st.flowFBO)
|
||||
gl.viewport(0, 0, st.flowW, st.flowH)
|
||||
gl.enable(gl.BLEND)
|
||||
gl.blendEquation(gl.FUNC_ADD)
|
||||
gl.blendFunc(gl.SRC_ALPHA, gl.ONE_MINUS_SRC_ALPHA)
|
||||
gl.useProgram(st.fadeProg)
|
||||
// Fade toward (0.5, 0.5, 0.5) = zero velocity in flow encoding
|
||||
const fadeLoc = gl.getUniformLocation(st.fadeProg, 'uFade')
|
||||
gl.uniform4f(fadeLoc, 0.5, 0.5, 0.5, u.flowDecay)
|
||||
drawQuad(gl, st.fadeProg, st.quadBuf)
|
||||
|
||||
// ── 4. Flow render pass ───────────────────────────────────────────────────
|
||||
// Still bound to flowFBO; accumulate head velocity into flow texture
|
||||
gl.blendFunc(gl.SRC_ALPHA, gl.ONE_MINUS_SRC_ALPHA)
|
||||
gl.useProgram(st.flowProg)
|
||||
setTex(gl, st.flowProg, 'stateCurr', st.readTex, 0)
|
||||
setTex(gl, st.flowProg, 'statePrev', st.writeTex, 1)
|
||||
setVec2(gl, st.flowProg, 'stateSize', rootNum, rootNum)
|
||||
setFloat(gl, st.flowProg, 'aspect', aspect)
|
||||
drawLines(gl, st.flowProg, st.vidBuf, numP)
|
||||
|
||||
// ── 5. Canvas trail fade ──────────────────────────────────────────────────
|
||||
gl.bindFramebuffer(gl.FRAMEBUFFER, null)
|
||||
gl.viewport(0, 0, w, h)
|
||||
gl.enable(gl.BLEND)
|
||||
gl.blendFunc(gl.SRC_ALPHA, gl.ONE_MINUS_SRC_ALPHA)
|
||||
gl.useProgram(st.fadeProg)
|
||||
const fadeLoc2 = gl.getUniformLocation(st.fadeProg, 'uFade')
|
||||
gl.uniform4f(fadeLoc2, u.bgColor[0], u.bgColor[1], u.bgColor[2], u.trailAlpha)
|
||||
drawQuad(gl, st.fadeProg, st.quadBuf)
|
||||
|
||||
// ── 6. View render pass (GL_LINES onto canvas) ────────────────────────────
|
||||
gl.blendFunc(gl.SRC_ALPHA, gl.ONE) // additive blending for glow effect
|
||||
gl.useProgram(st.renderProg)
|
||||
setTex(gl, st.renderProg, 'stateCurr', st.readTex, 0)
|
||||
setTex(gl, st.renderProg, 'statePrev', st.writeTex, 1)
|
||||
setVec2(gl, st.renderProg, 'stateSize', rootNum, rootNum)
|
||||
setFloat(gl, st.renderProg, 'aspect', aspect)
|
||||
setVec3(gl, st.renderProg, 'baseColor', u.baseColor)
|
||||
setVec3(gl, st.renderProg, 'flowColor', u.flowColor)
|
||||
setFloat(gl, st.renderProg, 'colorMapAlpha', u.colorMapAlpha)
|
||||
setFloat(gl, st.renderProg, 'speedAlpha', u.speedAlpha)
|
||||
|
||||
try { gl.lineWidth(Math.max(1, u.lineWidth ?? 1)) } catch { /* silently clamp on most GPUs */ }
|
||||
|
||||
drawLines(gl, st.renderProg, st.vidBuf, numP)
|
||||
|
||||
gl.disable(gl.BLEND)
|
||||
gl.bindBuffer(gl.ARRAY_BUFFER, null)
|
||||
gl.useProgram(null)
|
||||
gl.bindTexture(gl.TEXTURE_2D, null)
|
||||
gl.activeTexture(gl.TEXTURE0)
|
||||
}
|
||||
|
||||
export function destroyTendrilsSim(canvas: HTMLCanvasElement): void {
|
||||
const st = instances.get(canvas)
|
||||
if (!st) return
|
||||
|
||||
st.destroyed = true
|
||||
canvas.removeEventListener('webglcontextlost', st.onContextLost)
|
||||
canvas.removeEventListener('webglcontextrestored', st.onContextRestored)
|
||||
|
||||
const { gl } = st
|
||||
try {
|
||||
gl.deleteProgram(st.logicProg)
|
||||
gl.deleteProgram(st.renderProg)
|
||||
gl.deleteProgram(st.flowProg)
|
||||
gl.deleteProgram(st.fadeProg)
|
||||
gl.deleteTexture(st.readTex)
|
||||
gl.deleteTexture(st.writeTex)
|
||||
gl.deleteTexture(st.flowTex)
|
||||
gl.deleteFramebuffer(st.readFBO)
|
||||
gl.deleteFramebuffer(st.writeFBO)
|
||||
gl.deleteFramebuffer(st.flowFBO)
|
||||
gl.deleteBuffer(st.vidBuf)
|
||||
gl.deleteBuffer(st.quadBuf)
|
||||
} catch { /* context may already be lost */ }
|
||||
|
||||
instances.delete(canvas)
|
||||
}
|
||||
|
||||
// ── GL helpers ───────────────────────────────────────────────────────────────
|
||||
|
||||
function setTex(gl: WebGLRenderingContext, prog: WebGLProgram, name: string, tex: WebGLTexture, unit: number): void {
|
||||
const loc = gl.getUniformLocation(prog, name)
|
||||
if (loc === null) return
|
||||
gl.activeTexture(gl.TEXTURE0 + unit)
|
||||
gl.bindTexture(gl.TEXTURE_2D, tex)
|
||||
gl.uniform1i(loc, unit)
|
||||
}
|
||||
|
||||
function setFloat(gl: WebGLRenderingContext, prog: WebGLProgram, name: string, v: number): void {
|
||||
const loc = gl.getUniformLocation(prog, name)
|
||||
if (loc !== null) gl.uniform1f(loc, v)
|
||||
}
|
||||
|
||||
function setVec2(gl: WebGLRenderingContext, prog: WebGLProgram, name: string, x: number, y: number): void {
|
||||
const loc = gl.getUniformLocation(prog, name)
|
||||
if (loc !== null) gl.uniform2f(loc, x, y)
|
||||
}
|
||||
|
||||
function setVec3(gl: WebGLRenderingContext, prog: WebGLProgram, name: string, v: [number, number, number]): void {
|
||||
const loc = gl.getUniformLocation(prog, name)
|
||||
if (loc !== null) gl.uniform3f(loc, v[0], v[1], v[2])
|
||||
}
|
||||
|
||||
function drawQuad(gl: WebGLRenderingContext, prog: WebGLProgram, buf: WebGLBuffer): void {
|
||||
gl.bindBuffer(gl.ARRAY_BUFFER, buf)
|
||||
const aPos = gl.getAttribLocation(prog, 'aPos')
|
||||
if (aPos >= 0) {
|
||||
gl.enableVertexAttribArray(aPos)
|
||||
gl.vertexAttribPointer(aPos, 2, gl.FLOAT, false, 0, 0)
|
||||
}
|
||||
gl.drawArrays(gl.TRIANGLE_FAN, 0, 4)
|
||||
if (aPos >= 0) gl.disableVertexAttribArray(aPos)
|
||||
}
|
||||
|
||||
function drawLines(gl: WebGLRenderingContext, prog: WebGLProgram, buf: WebGLBuffer, numP: number): void {
|
||||
gl.bindBuffer(gl.ARRAY_BUFFER, buf)
|
||||
const aVid = gl.getAttribLocation(prog, 'vid')
|
||||
if (aVid >= 0) {
|
||||
gl.enableVertexAttribArray(aVid)
|
||||
gl.vertexAttribPointer(aVid, 1, gl.FLOAT, false, 0, 0)
|
||||
}
|
||||
gl.drawArrays(gl.LINES, 0, numP * 2)
|
||||
if (aVid >= 0) gl.disableVertexAttribArray(aVid)
|
||||
}
|
||||
@@ -3,6 +3,7 @@ import {
|
||||
deleteStem,
|
||||
exportProjectKaraokeMugen,
|
||||
exportProjectSek,
|
||||
getAppSettings,
|
||||
getProjectKaraokeConfig,
|
||||
getJob,
|
||||
importSubsonicSong,
|
||||
@@ -10,11 +11,13 @@ import {
|
||||
lyricTranscribeProject,
|
||||
renameStem,
|
||||
reprocessStem,
|
||||
submitPipeline,
|
||||
submitSeparateFromPath,
|
||||
subsonicStreamUrl,
|
||||
transcribeProject,
|
||||
} from '../api/client'
|
||||
import type { Job, SubsonicSong } from '../types'
|
||||
import { readAutoFormat, readAutoModel, readAutoSteps } from '../utils/autoPipeline'
|
||||
import { normalizeAutoPresets, readAutoFormat, readAutoModel, readAutoPresets, readAutoSteps } from '../utils/autoPipeline'
|
||||
import { useProjectsIndex } from './useProjectsIndex'
|
||||
import { reportNonFatal } from '../utils/nonFatal'
|
||||
|
||||
@@ -26,6 +29,75 @@ interface ConvertState {
|
||||
jobId?: string
|
||||
}
|
||||
|
||||
type ConvertProcessMode = 'auto' | 'separate_only' | 'pipeline' | 'preset'
|
||||
|
||||
async function runAdditionalSteps(projectId: string, steps: ReturnType<typeof readAutoSteps>) {
|
||||
for (const step of steps) {
|
||||
if (step.type === 'reprocess') {
|
||||
const { job_id } = await reprocessStem(projectId, step.stem, step.model, {
|
||||
outputFormat: step.outputFormat ?? 'mp3',
|
||||
mp3Bitrate: step.mp3Bitrate ?? '256k',
|
||||
})
|
||||
await pollJobUntilTerminal(job_id)
|
||||
} else if (step.type === 'midi') {
|
||||
const { job_id } = await transcribeProject(projectId, 'vocals', step.mode)
|
||||
await pollJobUntilTerminal(job_id)
|
||||
} else if (step.type === 'lyrics') {
|
||||
const wordTs = localStorage.getItem('stemengine-whisper-word-ts') !== 'false'
|
||||
const engine = step.engine ?? 'whisperx'
|
||||
const modelId = step.model ?? step.whisperModel ?? (engine === 'parakeet' ? 'parakeet-1.1b-rnnt-multilingual-asr' : 'medium')
|
||||
const { job_id } = await lyricTranscribeProject(projectId, step.stem ?? 'vocals', {
|
||||
engine,
|
||||
model: modelId,
|
||||
wordTimestamps: wordTs,
|
||||
diarize: engine === 'whisperx' ? step.diarize : false,
|
||||
lsyncAlign: engine === 'whisperx' ? (step.lsyncAlign ?? false) : false,
|
||||
twoPass: engine === 'whisperx' ? (step.twoPass ?? false) : false,
|
||||
vocalStyle: engine === 'whisperx' ? (step.vocalStyle ?? 'general') : 'general',
|
||||
explicit: engine === 'whisperx' ? (step.explicit ?? false) : false,
|
||||
})
|
||||
await pollJobUntilTerminal(job_id)
|
||||
} else if (step.type === 'lyricsAlign') {
|
||||
const { job_id } = await lrcProject(projectId, 'vocals', {
|
||||
lyricsSource: 'auto',
|
||||
timingPrecision: step.timingPrecision,
|
||||
lrcPowerPreset: localStorage.getItem('stemengine-sep-power') ?? 'standard',
|
||||
})
|
||||
await pollJobUntilTerminal(job_id)
|
||||
} else if (step.type === 'renameStem') {
|
||||
const fromStem = step.fromStem.trim()
|
||||
const toStem = step.toStem.trim()
|
||||
if (fromStem && toStem && fromStem !== toStem) {
|
||||
await renameStem(projectId, fromStem, toStem)
|
||||
}
|
||||
} else if (step.type === 'deleteStem') {
|
||||
const stem = step.stem.trim()
|
||||
if (stem) await deleteStem(projectId, stem)
|
||||
} else if (step.type === 'karaokeMugen') {
|
||||
await exportProjectKaraokeMugen(projectId, {
|
||||
defaultTitleLang: step.defaultTitleLang ?? 'eng',
|
||||
repository: step.repository ?? 'stemengine.local',
|
||||
})
|
||||
} else if (step.type === 'sekPackage') {
|
||||
let resolvedVideoSource: string | undefined
|
||||
try {
|
||||
const karaokeCfg = await getProjectKaraokeConfig(projectId)
|
||||
const candidate = karaokeCfg?.config?.video_source?.trim()
|
||||
if (candidate) resolvedVideoSource = candidate
|
||||
} catch (error) {
|
||||
// non-fatal: fallback to standard includeVideo behavior
|
||||
reportNonFatal('useConvertStatus: getProjectKaraokeConfig', error, { projectId })
|
||||
}
|
||||
|
||||
await exportProjectSek(projectId, {
|
||||
packageName: step.packageName,
|
||||
includeVideo: resolvedVideoSource ? true : (step.includeVideo ?? true),
|
||||
videoSource: resolvedVideoSource,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function pollJobUntilTerminal(jobId: string): Promise<Job> {
|
||||
for (;;) {
|
||||
const job = await getJob(jobId)
|
||||
@@ -70,15 +142,63 @@ export function useConvertStatus(songs: SubsonicSong[]) {
|
||||
}))
|
||||
|
||||
try {
|
||||
const imported = await importSubsonicSong(song)
|
||||
const model = readAutoModel()
|
||||
const format = readAutoFormat()
|
||||
const projectName = `${song.songArtist ?? song.artist} - ${song.title}`.trim()
|
||||
let processMode: ConvertProcessMode = 'auto'
|
||||
let processPresetName = ''
|
||||
let storedPresets = readAutoPresets()
|
||||
try {
|
||||
const settings = await getAppSettings()
|
||||
const rawMode = String((settings as { singui_process_mode?: unknown }).singui_process_mode || 'auto').trim().toLowerCase()
|
||||
processMode = (rawMode === 'separate_only' || rawMode === 'pipeline' || rawMode === 'preset') ? rawMode : 'auto'
|
||||
processPresetName = String((settings as { singui_process_preset?: unknown }).singui_process_preset || '').trim()
|
||||
const serverPresets = normalizeAutoPresets(
|
||||
(settings as { auto_presets_effective?: unknown[]; auto_presets?: unknown[] }).auto_presets_effective
|
||||
?? (settings as { auto_presets?: unknown[] }).auto_presets
|
||||
)
|
||||
if (serverPresets.length > 0) storedPresets = serverPresets
|
||||
} catch (error) {
|
||||
reportNonFatal('useConvertStatus: getAppSettings', error)
|
||||
}
|
||||
|
||||
const submit = await submitSeparateFromPath(imported.path, model, format, {
|
||||
presetType: 'stems_only',
|
||||
projectName,
|
||||
})
|
||||
const projectName = `${song.songArtist ?? song.artist} - ${song.title}`.trim()
|
||||
let submit: { job_id: string }
|
||||
|
||||
if (processMode === 'pipeline') {
|
||||
const res = await fetch(subsonicStreamUrl(song.id))
|
||||
if (!res.ok) {
|
||||
throw new Error(`Failed to fetch source audio for pipeline (${res.status})`)
|
||||
}
|
||||
const blob = await res.blob()
|
||||
const ext = (song.suffix || 'mp3').replace(/[^a-zA-Z0-9]/g, '') || 'mp3'
|
||||
const file = new File([blob], `${projectName}.${ext}`, { type: blob.type || 'audio/mpeg' })
|
||||
submit = await submitPipeline(file, readAutoModel(), readAutoFormat(), 'polyphonic', projectName)
|
||||
} else {
|
||||
const imported = await importSubsonicSong(song)
|
||||
|
||||
let model = readAutoModel()
|
||||
let format = readAutoFormat()
|
||||
let instrumentModels: Record<string, string> | undefined
|
||||
if (processMode === 'preset') {
|
||||
const presets = storedPresets
|
||||
const preset = (processPresetName
|
||||
? presets.find((p) => p.name === processPresetName)
|
||||
: undefined) ?? presets[0]
|
||||
if (preset) {
|
||||
model = preset.model
|
||||
format = preset.format
|
||||
instrumentModels = preset.instrumentModels
|
||||
} else {
|
||||
reportNonFatal('useConvertStatus: preset mode fallback', new Error('No presets found; falling back to auto defaults'), {
|
||||
requestedPreset: processPresetName || null,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
submit = await submitSeparateFromPath(imported.path, model, format, {
|
||||
presetType: 'stems_only',
|
||||
projectName,
|
||||
instrumentModels,
|
||||
})
|
||||
}
|
||||
|
||||
setStateBySongId((prev) => ({
|
||||
...prev,
|
||||
@@ -90,69 +210,17 @@ export function useConvertStatus(songs: SubsonicSong[]) {
|
||||
if (finalJob.status === 'done') {
|
||||
const projectId = finalJob.result?.project_id as string | undefined
|
||||
if (projectId) {
|
||||
const steps = readAutoSteps()
|
||||
for (const step of steps) {
|
||||
if (step.type === 'reprocess') {
|
||||
const { job_id } = await reprocessStem(projectId, step.stem, step.model, {
|
||||
outputFormat: step.outputFormat ?? 'mp3',
|
||||
mp3Bitrate: step.mp3Bitrate ?? '256k',
|
||||
})
|
||||
await pollJobUntilTerminal(job_id)
|
||||
} else if (step.type === 'midi') {
|
||||
const { job_id } = await transcribeProject(projectId, 'vocals', step.mode)
|
||||
await pollJobUntilTerminal(job_id)
|
||||
} else if (step.type === 'lyrics') {
|
||||
const wordTs = localStorage.getItem('stemengine-whisper-word-ts') !== 'false'
|
||||
const engine = step.engine ?? 'whisperx'
|
||||
const modelId = step.model ?? step.whisperModel ?? (engine === 'parakeet' ? 'parakeet-1.1b-rnnt-multilingual-asr' : 'medium')
|
||||
const { job_id } = await lyricTranscribeProject(projectId, step.stem ?? 'vocals', {
|
||||
engine,
|
||||
model: modelId,
|
||||
wordTimestamps: wordTs,
|
||||
diarize: engine === 'whisperx' ? step.diarize : false,
|
||||
lsyncAlign: engine === 'whisperx' ? (step.lsyncAlign ?? false) : false,
|
||||
twoPass: engine === 'whisperx' ? (step.twoPass ?? false) : false,
|
||||
vocalStyle: engine === 'whisperx' ? (step.vocalStyle ?? 'general') : 'general',
|
||||
explicit: engine === 'whisperx' ? (step.explicit ?? false) : false,
|
||||
})
|
||||
await pollJobUntilTerminal(job_id)
|
||||
} else if (step.type === 'lyricsAlign') {
|
||||
const { job_id } = await lrcProject(projectId, 'vocals', {
|
||||
lyricsSource: 'auto',
|
||||
timingPrecision: step.timingPrecision,
|
||||
lrcPowerPreset: localStorage.getItem('stemengine-sep-power') ?? 'standard',
|
||||
})
|
||||
await pollJobUntilTerminal(job_id)
|
||||
} else if (step.type === 'renameStem') {
|
||||
const fromStem = step.fromStem.trim()
|
||||
const toStem = step.toStem.trim()
|
||||
if (fromStem && toStem && fromStem !== toStem) {
|
||||
await renameStem(projectId, fromStem, toStem)
|
||||
}
|
||||
} else if (step.type === 'deleteStem') {
|
||||
const stem = step.stem.trim()
|
||||
if (stem) await deleteStem(projectId, stem)
|
||||
} else if (step.type === 'karaokeMugen') {
|
||||
await exportProjectKaraokeMugen(projectId, {
|
||||
defaultTitleLang: step.defaultTitleLang ?? 'eng',
|
||||
repository: step.repository ?? 'stemengine.local',
|
||||
})
|
||||
} else if (step.type === 'sekPackage') {
|
||||
let resolvedVideoSource: string | undefined
|
||||
try {
|
||||
const karaokeCfg = await getProjectKaraokeConfig(projectId)
|
||||
const candidate = karaokeCfg?.config?.video_source?.trim()
|
||||
if (candidate) resolvedVideoSource = candidate
|
||||
} catch (error) {
|
||||
// non-fatal: fallback to standard includeVideo behavior
|
||||
reportNonFatal('useConvertStatus: getProjectKaraokeConfig', error, { projectId })
|
||||
}
|
||||
if (processMode === 'auto') {
|
||||
await runAdditionalSteps(projectId, readAutoSteps())
|
||||
}
|
||||
|
||||
await exportProjectSek(projectId, {
|
||||
packageName: step.packageName,
|
||||
includeVideo: resolvedVideoSource ? true : (step.includeVideo ?? true),
|
||||
videoSource: resolvedVideoSource,
|
||||
})
|
||||
if (processMode === 'preset') {
|
||||
const presets = storedPresets
|
||||
const preset = (processPresetName
|
||||
? presets.find((p) => p.name === processPresetName)
|
||||
: undefined) ?? presets[0]
|
||||
if (preset) {
|
||||
await runAdditionalSteps(projectId, preset.steps)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -503,10 +503,10 @@ export default function AlbumsPage() {
|
||||
>
|
||||
<Zap className="w-4 h-4" />
|
||||
{resolveAlbumStatus(album.id) === 'queued' && 'Queued'}
|
||||
{resolveAlbumStatus(album.id) === 'running' && 'Packaging'}
|
||||
{resolveAlbumStatus(album.id) === 'ready' && 'Packaged'}
|
||||
{resolveAlbumStatus(album.id) === 'running' && 'Preparing'}
|
||||
{resolveAlbumStatus(album.id) === 'ready' && 'Project Ready'}
|
||||
{resolveAlbumStatus(album.id) === 'failed' && 'Retry'}
|
||||
{resolveAlbumStatus(album.id) === 'idle' && 'Package'}
|
||||
{resolveAlbumStatus(album.id) === 'idle' && 'Prepare Project'}
|
||||
</button>
|
||||
</>}
|
||||
{viewMode === 'small-icon' && (
|
||||
|
||||
@@ -30,6 +30,7 @@ import {
|
||||
readAutoInstrumentModels, writeAutoInstrumentModels,
|
||||
readAutoPresets, writeAutoPresets,
|
||||
LS_AUTO_MODEL, LS_AUTO_FORMAT,
|
||||
normalizeAutoPresets,
|
||||
stepLabel,
|
||||
} from '../utils/autoPipeline'
|
||||
import type { AutoStep, AutoPreset } from '../utils/autoPipeline'
|
||||
@@ -1875,9 +1876,66 @@ function AutoPipelineSettings() {
|
||||
const [instrumentModels, setInstrumentModels] = useState<Record<string, string>>(readAutoInstrumentModels)
|
||||
const [steps, setSteps] = useState<AutoStep[]>(readAutoSteps)
|
||||
const [addOpen, setAddOpen] = useState(false)
|
||||
const [presets, setPresets] = useState<AutoPreset[]>(readAutoPresets)
|
||||
const [presets, setPresets] = useState<AutoPreset[]>([])
|
||||
const [userPresets, setUserPresets] = useState<AutoPreset[]>([])
|
||||
const [presetName, setPresetName] = useState('')
|
||||
const [presetOpen, setPresetOpen] = useState(false)
|
||||
const presetsHydratedRef = useRef(false)
|
||||
const presetsMigratedRef = useRef(false)
|
||||
const { data: appSettings } = useQuery({
|
||||
queryKey: ['app-settings-autopipeline'],
|
||||
queryFn: api.getAppSettings,
|
||||
staleTime: 60_000,
|
||||
})
|
||||
const [singUiProcessMode, setSingUiProcessMode] = useState<'auto' | 'separate_only' | 'pipeline' | 'preset'>('auto')
|
||||
const [singUiProcessPreset, setSingUiProcessPreset] = useState('')
|
||||
const [singUiProcessSaving, setSingUiProcessSaving] = useState(false)
|
||||
|
||||
useEffect(() => {
|
||||
const next = appSettings?.singui_process_mode
|
||||
if (next === 'separate_only' || next === 'pipeline' || next === 'preset') {
|
||||
setSingUiProcessMode(next)
|
||||
} else {
|
||||
setSingUiProcessMode('auto')
|
||||
}
|
||||
setSingUiProcessPreset(String(appSettings?.singui_process_preset || '').trim())
|
||||
}, [appSettings?.singui_process_mode, appSettings?.singui_process_preset])
|
||||
|
||||
useEffect(() => {
|
||||
const serverUserPresets = normalizeAutoPresets(appSettings?.auto_presets_user ?? appSettings?.auto_presets)
|
||||
const serverEffectivePresets = normalizeAutoPresets(appSettings?.auto_presets_effective ?? appSettings?.auto_presets)
|
||||
const displayPresets = serverEffectivePresets.length > 0 ? serverEffectivePresets : serverUserPresets
|
||||
|
||||
if (displayPresets.length > 0) {
|
||||
setPresets(displayPresets)
|
||||
setUserPresets(serverUserPresets)
|
||||
writeAutoPresets(displayPresets)
|
||||
presetsHydratedRef.current = true
|
||||
return
|
||||
}
|
||||
|
||||
if (presetsHydratedRef.current) return
|
||||
const localPresets = readAutoPresets()
|
||||
setPresets(localPresets)
|
||||
setUserPresets(localPresets)
|
||||
presetsHydratedRef.current = true
|
||||
|
||||
if (!presetsMigratedRef.current && localPresets.length > 0) {
|
||||
presetsMigratedRef.current = true
|
||||
void api.patchAppSettings({ auto_presets: localPresets }).catch(() => {
|
||||
// Non-fatal: keep local fallback if backend write fails.
|
||||
})
|
||||
}
|
||||
}, [appSettings?.auto_presets, appSettings?.auto_presets_user, appSettings?.auto_presets_effective])
|
||||
|
||||
function mergeDisplayPresets(nextUserPresets: AutoPreset[]): AutoPreset[] {
|
||||
const builtins = normalizeAutoPresets(appSettings?.auto_presets_builtin)
|
||||
if (!builtins.length) return nextUserPresets
|
||||
const merged = new Map<string, AutoPreset>()
|
||||
builtins.forEach((preset) => merged.set(preset.name, preset))
|
||||
nextUserPresets.forEach((preset) => merged.set(preset.name, preset))
|
||||
return Array.from(merged.values())
|
||||
}
|
||||
|
||||
function saveSteps(next: AutoStep[]) {
|
||||
setSteps(next)
|
||||
@@ -1953,9 +2011,14 @@ function AutoPipelineSettings() {
|
||||
const name = presetName.trim()
|
||||
if (!name) return
|
||||
const preset: AutoPreset = { name, model: sepModel, format: sepFormat, instrumentModels, steps }
|
||||
const next = [...presets.filter(p => p.name !== name), preset]
|
||||
setPresets(next)
|
||||
writeAutoPresets(next)
|
||||
const nextUser = [...userPresets.filter(p => p.name !== name), preset]
|
||||
const nextDisplay = mergeDisplayPresets(nextUser)
|
||||
setUserPresets(nextUser)
|
||||
setPresets(nextDisplay)
|
||||
writeAutoPresets(nextDisplay)
|
||||
void api.patchAppSettings({ auto_presets: nextUser }).catch(() => {
|
||||
// Non-fatal: local copy remains available.
|
||||
})
|
||||
setPresetName('')
|
||||
}
|
||||
|
||||
@@ -1970,9 +2033,22 @@ function AutoPipelineSettings() {
|
||||
}
|
||||
|
||||
function deletePreset(name: string) {
|
||||
const next = presets.filter(p => p.name !== name)
|
||||
setPresets(next)
|
||||
writeAutoPresets(next)
|
||||
const nextUser = userPresets.filter(p => p.name !== name)
|
||||
const nextDisplay = mergeDisplayPresets(nextUser)
|
||||
setUserPresets(nextUser)
|
||||
setPresets(nextDisplay)
|
||||
writeAutoPresets(nextDisplay)
|
||||
let nextSelectedPreset: string | undefined
|
||||
if (singUiProcessMode === 'preset' && singUiProcessPreset === name) {
|
||||
setSingUiProcessPreset('')
|
||||
nextSelectedPreset = ''
|
||||
}
|
||||
void api.patchAppSettings({
|
||||
auto_presets: nextUser,
|
||||
...(nextSelectedPreset !== undefined ? { singui_process_preset: nextSelectedPreset } : {}),
|
||||
}).catch(() => {
|
||||
// Non-fatal: local copy remains available.
|
||||
})
|
||||
}
|
||||
|
||||
function moveStep(idx: number, dir: -1 | 1) {
|
||||
@@ -2074,6 +2150,75 @@ function AutoPipelineSettings() {
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="rounded-xl border ui-border ui-surface-2 p-4 space-y-3">
|
||||
<h3 className="text-xs font-semibold ui-text-muted uppercase tracking-wider">
|
||||
SingUI Package Behavior
|
||||
</h3>
|
||||
<div className="grid grid-cols-1 md:grid-cols-[minmax(0,1fr)_auto] gap-3 items-end">
|
||||
<label className="space-y-1 block">
|
||||
<span className="text-xs ui-text-muted uppercase tracking-wider">Default Mode</span>
|
||||
<select
|
||||
value={singUiProcessMode}
|
||||
onChange={(e) => {
|
||||
const next = e.target.value
|
||||
if (next === 'separate_only' || next === 'pipeline' || next === 'preset') {
|
||||
setSingUiProcessMode(next)
|
||||
} else {
|
||||
setSingUiProcessMode('auto')
|
||||
}
|
||||
}}
|
||||
className={AP_SELECT}
|
||||
>
|
||||
<option value="auto">Auto pipeline (separate + configured steps)</option>
|
||||
<option value="separate_only">Separation only (skip extra steps)</option>
|
||||
<option value="pipeline">Backend full pipeline endpoint</option>
|
||||
<option value="preset">Specific Auto Pipeline preset</option>
|
||||
</select>
|
||||
</label>
|
||||
{singUiProcessMode === 'preset' && (
|
||||
<label className="space-y-1 block md:col-span-2">
|
||||
<span className="text-xs ui-text-muted uppercase tracking-wider">Preset</span>
|
||||
<select
|
||||
value={singUiProcessPreset}
|
||||
onChange={(e) => setSingUiProcessPreset(e.target.value)}
|
||||
className={AP_SELECT}
|
||||
disabled={presets.length === 0}
|
||||
>
|
||||
<option value="">{presets.length > 0 ? 'Use first saved preset' : 'No presets available'}</option>
|
||||
{presets.map((preset) => (
|
||||
<option key={preset.name} value={preset.name}>{preset.name}</option>
|
||||
))}
|
||||
</select>
|
||||
</label>
|
||||
)}
|
||||
<button
|
||||
onClick={async () => {
|
||||
try {
|
||||
setSingUiProcessSaving(true)
|
||||
await api.patchAppSettings({
|
||||
singui_process_mode: singUiProcessMode,
|
||||
singui_process_preset: singUiProcessMode === 'preset' ? singUiProcessPreset : '',
|
||||
})
|
||||
} finally {
|
||||
setSingUiProcessSaving(false)
|
||||
}
|
||||
}}
|
||||
className="rounded-lg border ui-border px-3 py-2 text-xs font-medium ui-text-muted hover:opacity-90 disabled:opacity-60"
|
||||
disabled={singUiProcessSaving || (singUiProcessMode === 'preset' && presets.length === 0)}
|
||||
>
|
||||
{singUiProcessSaving ? 'Saving…' : 'Save mode'}
|
||||
</button>
|
||||
</div>
|
||||
<p className="text-xs ui-text-muted">
|
||||
Used by Package actions in SingUI/Tracks/Albums.
|
||||
</p>
|
||||
{singUiProcessMode === 'preset' && presets.length === 0 && (
|
||||
<p className="text-xs text-amber-600 dark:text-amber-400">
|
||||
Save at least one Auto Pipeline preset above before using preset mode.
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* Primary Separation */}
|
||||
<div className="space-y-4">
|
||||
<h3 className="text-xs font-semibold ui-text-muted uppercase tracking-wider">
|
||||
|
||||
+35
-46
@@ -322,15 +322,15 @@ export default function SingPage() {
|
||||
<button
|
||||
onClick={() => void convertSong(song)}
|
||||
disabled={statusBySongId[song.id]?.status === 'queued' || statusBySongId[song.id]?.status === 'running'}
|
||||
title={statusBySongId[song.id]?.error || 'Package with StemEngine Auto Pipeline'}
|
||||
title={statusBySongId[song.id]?.error || 'Prepare karaoke project (mode from Settings → Auto Pipeline)'}
|
||||
className="inline-flex items-center gap-2 rounded bg-brand-500 px-3 py-2 text-sm font-medium text-white hover:bg-brand-600 disabled:cursor-not-allowed disabled:opacity-70"
|
||||
>
|
||||
<Zap className="h-4 w-4" />
|
||||
{statusBySongId[song.id]?.status === 'queued' && 'Queued'}
|
||||
{statusBySongId[song.id]?.status === 'running' && 'Packaging'}
|
||||
{statusBySongId[song.id]?.status === 'ready' && 'Packaged'}
|
||||
{statusBySongId[song.id]?.status === 'running' && 'Preparing'}
|
||||
{statusBySongId[song.id]?.status === 'ready' && 'Project Ready'}
|
||||
{statusBySongId[song.id]?.status === 'failed' && 'Retry'}
|
||||
{(!statusBySongId[song.id] || statusBySongId[song.id]?.status === 'idle') && 'Package'}
|
||||
{(!statusBySongId[song.id] || statusBySongId[song.id]?.status === 'idle') && 'Prepare Project'}
|
||||
</button>
|
||||
</div>
|
||||
))}
|
||||
@@ -697,25 +697,26 @@ export default function SingPage() {
|
||||
const sekFolder = sekFolderInput.trim() || undefined
|
||||
const primaryMedia = primaryMediaFromSongAndMatches(song, sekCandidates, sekFolder)
|
||||
const selectedSekMedia = selectedAmbiguousCandidate ? sekMediaFromMatch(song, selectedAmbiguousCandidate, sekFolder) : null
|
||||
const topSekMedia = topSek ? sekMediaFromMatch(song, topSek, sekFolder) : null
|
||||
const isPreparing = status === 'queued' || status === 'running'
|
||||
const isReady = status === 'ready' && !!linked
|
||||
const selectedQueueItem = !isPreparing
|
||||
? (!sekAmbiguous && topSek?.package_id
|
||||
? buildSekKaraokeQueueItem({
|
||||
packageId: topSek.package_id,
|
||||
title: topSek.title || song.title,
|
||||
artist: song.songArtist ?? song.artist,
|
||||
sekFolder,
|
||||
})
|
||||
: selectedAmbiguousCandidate?.package_id
|
||||
? buildSekKaraokeQueueItem({
|
||||
packageId: selectedAmbiguousCandidate.package_id,
|
||||
title: selectedAmbiguousCandidate.title || song.title,
|
||||
artist: song.songArtist ?? song.artist,
|
||||
sekFolder,
|
||||
})
|
||||
: (isReady && linked ? buildProjectKaraokeQueueItem(song, linked) : null))
|
||||
? (linked
|
||||
? buildProjectKaraokeQueueItem(song, linked)
|
||||
: (!sekAmbiguous && topSek?.package_id
|
||||
? buildSekKaraokeQueueItem({
|
||||
packageId: topSek.package_id,
|
||||
title: topSek.title || song.title,
|
||||
artist: song.songArtist ?? song.artist,
|
||||
sekFolder,
|
||||
})
|
||||
: selectedAmbiguousCandidate?.package_id
|
||||
? buildSekKaraokeQueueItem({
|
||||
packageId: selectedAmbiguousCandidate.package_id,
|
||||
title: selectedAmbiguousCandidate.title || song.title,
|
||||
artist: song.songArtist ?? song.artist,
|
||||
sekFolder,
|
||||
})
|
||||
: null))
|
||||
: null
|
||||
|
||||
return (
|
||||
@@ -762,11 +763,11 @@ export default function SingPage() {
|
||||
status === 'queued' ? 'text-yellow-400' :
|
||||
status === 'failed' ? 'text-red-400' : 'text-gray-600'
|
||||
}`}>
|
||||
{status === 'ready' && '● Ready'}
|
||||
{status === 'running' && '● Packaging'}
|
||||
{status === 'ready' && '● Project Ready'}
|
||||
{status === 'running' && '● Preparing'}
|
||||
{status === 'queued' && '● Queued'}
|
||||
{status === 'failed' && '● Failed'}
|
||||
{status === 'idle' && '○ Not packaged'}
|
||||
{status === 'idle' && '○ Project not ready'}
|
||||
</span>
|
||||
</div>
|
||||
<div className="flex shrink-0 items-center gap-2">
|
||||
@@ -792,22 +793,19 @@ export default function SingPage() {
|
||||
{isPreparing && (
|
||||
<span className="inline-flex items-center gap-1.5 text-xs text-gray-400">
|
||||
<Loader2 className="h-3.5 w-3.5 animate-spin" />
|
||||
{status === 'queued' ? 'Queued' : 'Packaging…'}
|
||||
{status === 'queued' ? 'Queued' : 'Preparing…'}
|
||||
</span>
|
||||
)}
|
||||
{!isPreparing && hasStrongSek && !sekAmbiguous && topSekMedia && (
|
||||
{!isPreparing && isReady && linked && (
|
||||
<button
|
||||
onClick={() => {
|
||||
playNow([topSekMedia])
|
||||
launchMediaItem(navigate, topSekMedia)
|
||||
}}
|
||||
className="inline-flex items-center gap-1.5 rounded bg-emerald-600 px-3 py-1.5 text-xs font-medium text-white hover:bg-emerald-700"
|
||||
onClick={() => navigate(KARAOKE_ROUTE, { state: { projectId: linked.id } })}
|
||||
className="inline-flex items-center gap-1.5 rounded bg-brand-500 px-3 py-1.5 text-xs font-medium text-white hover:bg-brand-600"
|
||||
>
|
||||
<Mic2 className="h-3.5 w-3.5" />
|
||||
Sing (SEK)
|
||||
Sing (Project)
|
||||
</button>
|
||||
)}
|
||||
{!isPreparing && sekAmbiguous && (
|
||||
{!isPreparing && !isReady && sekAmbiguous && (
|
||||
<div className="flex items-center gap-2">
|
||||
<select
|
||||
value={selectedAmbiguousId}
|
||||
@@ -841,22 +839,13 @@ export default function SingPage() {
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
{!isPreparing && !isReady && !hasStrongSek && !sekAmbiguous && (
|
||||
{!isPreparing && !isReady && (
|
||||
<button
|
||||
onClick={() => void convertSong(song)}
|
||||
className="inline-flex items-center gap-1.5 rounded border border-brand-500/50 px-2.5 py-1.5 text-xs font-medium text-brand-400 hover:bg-brand-500/10"
|
||||
>
|
||||
<Zap className="h-3.5 w-3.5" />
|
||||
{status === 'failed' ? 'Retry' : 'Package'}
|
||||
</button>
|
||||
)}
|
||||
{isReady && (!hasStrongSek || sekAmbiguous) && (
|
||||
<button
|
||||
onClick={() => navigate(KARAOKE_ROUTE, { state: { projectId: linked!.id } })}
|
||||
className="inline-flex items-center gap-1.5 rounded bg-brand-500 px-3 py-1.5 text-xs font-medium text-white hover:bg-brand-600"
|
||||
>
|
||||
<Mic2 className="h-3.5 w-3.5" />
|
||||
Sing (Project)
|
||||
{status === 'failed' ? 'Retry' : 'Prepare Project'}
|
||||
</button>
|
||||
)}
|
||||
{status === 'ready' && !linked && projectsLoading && (
|
||||
@@ -908,10 +897,10 @@ export default function SingPage() {
|
||||
>
|
||||
<Zap className="h-4 w-4" />
|
||||
{albumStatus(album.id) === 'queued' && 'Queued'}
|
||||
{albumStatus(album.id) === 'running' && 'Packaging'}
|
||||
{albumStatus(album.id) === 'ready' && 'Packaged'}
|
||||
{albumStatus(album.id) === 'running' && 'Preparing'}
|
||||
{albumStatus(album.id) === 'ready' && 'Project Ready'}
|
||||
{albumStatus(album.id) === 'failed' && 'Retry'}
|
||||
{albumStatus(album.id) === 'idle' && 'Package'}
|
||||
{albumStatus(album.id) === 'idle' && 'Prepare Project'}
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -402,15 +402,15 @@ export default function TracksPage() {
|
||||
void convertSong(song)
|
||||
}}
|
||||
disabled={statusBySongId[track.id]?.status === 'queued' || statusBySongId[track.id]?.status === 'running'}
|
||||
title={statusBySongId[track.id]?.error || 'Package with StemEngine Auto Pipeline'}
|
||||
title={statusBySongId[track.id]?.error || 'Prepare karaoke project (mode from Settings → Auto Pipeline)'}
|
||||
className="flex items-center gap-2 px-3 py-2 rounded text-white text-sm font-medium transition-colors disabled:opacity-70 disabled:cursor-not-allowed bg-brand-500 hover:bg-brand-600"
|
||||
>
|
||||
<Zap className="w-4 h-4" />
|
||||
{statusBySongId[track.id]?.status === 'queued' && 'Queued'}
|
||||
{statusBySongId[track.id]?.status === 'running' && 'Packaging'}
|
||||
{statusBySongId[track.id]?.status === 'ready' && 'Packaged'}
|
||||
{statusBySongId[track.id]?.status === 'running' && 'Preparing'}
|
||||
{statusBySongId[track.id]?.status === 'ready' && 'Project Ready'}
|
||||
{statusBySongId[track.id]?.status === 'failed' && 'Retry'}
|
||||
{(!statusBySongId[track.id] || statusBySongId[track.id]?.status === 'idle') && 'Package'}
|
||||
{(!statusBySongId[track.id] || statusBySongId[track.id]?.status === 'idle') && 'Prepare Project'}
|
||||
</button>
|
||||
</div>
|
||||
)
|
||||
|
||||
Vendored
+13
@@ -0,0 +1,13 @@
|
||||
// Vite ?raw imports for GLSL shader source files.
|
||||
declare module '*.glsl' {
|
||||
const s: string
|
||||
export default s
|
||||
}
|
||||
declare module '*.vert' {
|
||||
const s: string
|
||||
export default s
|
||||
}
|
||||
declare module '*.frag' {
|
||||
const s: string
|
||||
export default s
|
||||
}
|
||||
@@ -152,6 +152,47 @@ function migrateStep(step: AutoStep): AutoStep {
|
||||
}
|
||||
}
|
||||
|
||||
function normalizePreset(input: unknown): AutoPreset | null {
|
||||
if (!input || typeof input !== 'object') return null
|
||||
const raw = input as {
|
||||
name?: unknown
|
||||
model?: unknown
|
||||
format?: unknown
|
||||
instrumentModels?: unknown
|
||||
steps?: unknown
|
||||
}
|
||||
const name = typeof raw.name === 'string' ? raw.name.trim() : ''
|
||||
const model = typeof raw.model === 'string' ? raw.model.trim() : ''
|
||||
const format = typeof raw.format === 'string' ? raw.format.trim() : ''
|
||||
if (!name || !model || !format) return null
|
||||
|
||||
const instrumentModels: Record<string, string> = {}
|
||||
if (raw.instrumentModels && typeof raw.instrumentModels === 'object') {
|
||||
for (const [k, v] of Object.entries(raw.instrumentModels as Record<string, unknown>)) {
|
||||
if (typeof v === 'string' && v.trim()) instrumentModels[k] = v.trim()
|
||||
}
|
||||
}
|
||||
|
||||
const steps = Array.isArray(raw.steps) ? (raw.steps as AutoStep[]).map(migrateStep) : []
|
||||
return {
|
||||
name,
|
||||
model,
|
||||
format,
|
||||
instrumentModels,
|
||||
steps,
|
||||
}
|
||||
}
|
||||
|
||||
export function normalizeAutoPresets(input: unknown): AutoPreset[] {
|
||||
if (!Array.isArray(input)) return []
|
||||
const out: AutoPreset[] = []
|
||||
for (const item of input) {
|
||||
const normalized = normalizePreset(item)
|
||||
if (normalized) out.push(normalized)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
export function readAutoSteps(): AutoStep[] {
|
||||
try {
|
||||
const raw = JSON.parse(localStorage.getItem(LS_AUTO_STEPS) || '[]') as AutoStep[]
|
||||
@@ -171,13 +212,8 @@ export function writeAutoSteps(steps: AutoStep[]): void {
|
||||
|
||||
export function readAutoPresets(): AutoPreset[] {
|
||||
try {
|
||||
const raw = JSON.parse(localStorage.getItem(LS_AUTO_PRESETS) || '[]') as AutoPreset[]
|
||||
if (!Array.isArray(raw)) return []
|
||||
const migrated = raw.map((preset) => ({
|
||||
...preset,
|
||||
instrumentModels: preset.instrumentModels ?? {},
|
||||
steps: Array.isArray(preset.steps) ? preset.steps.map(migrateStep) : [],
|
||||
}))
|
||||
const raw = JSON.parse(localStorage.getItem(LS_AUTO_PRESETS) || '[]')
|
||||
const migrated = normalizeAutoPresets(raw)
|
||||
writeAutoPresets(migrated)
|
||||
return migrated
|
||||
} catch {
|
||||
|
||||
Vendored
+1
@@ -0,0 +1 @@
|
||||
/// <reference types="vite/client" />
|
||||
Reference in new issue
Block a user