Myriad changes.

This commit is contained in:
translator committed 2026-04-13 14:03:46 +09:00
1 parent cbe946925f
commit ee851bc9ae
32 files changed
+7859 -888

No files matched your search

@@ -0,0 +1,534 @@
# Muon Audio Analyzer Integration Plan
Date: 2026-04-13
Scope: Replace the current single-stem/Meyda feature extraction in `useAudioFeatures` with the
Muon Audio Analysis `FeatureExtractor` approach, tapping a hidden analysis `<audio>` element
fed by `analysisAudioUrl ?? audioUrl` (full-mix source when available; stem fallback in SEK).
Reference repo: https://github.com/najafmohammed/Muon-Audio-Analysis
Status: private, not published to npm — the `FeatureExtractor` class must be inlined.
---
## Integrated decisions from related plans (2026-04-13)
This plan incorporates constraints already established in:
- `2026-04-13_Visualizer_DiagnosisAndFixes_Plan.md`
- `2026-04-13_TrackDetail_VisualizerPreview_Plan.md`
- `2026-04-13_SEKAsset_RangeRequest_Plan.md`
Net effect:
1. Analysis source is **not** `audioUrl`-only anymore; it is `analysisAudioUrl ?? audioUrl`.
2. `analysisAudioUrl` must be threaded through `KaraokePerformanceMode` into
`LyricsLivePreview` and included in `analysisSourceKey`.
3. In SEK mode, where no explicit full-mix source exists in the manifest, analysis may use a
ranked stem fallback (initially first/leading/drums stem strategy).
4. Hidden analysis seek sync depends on working HTTP range support for SEK assets
(`/api/karaoke-sek/packages/{id}/asset`) so `currentTime` jumps are reliable.
---
## Current problems
1. **Wrong source** — `useAudioFeatures` taps only the `drums` stem gain node (or first stem as
fallback), so the visualizer is deaf to vocals, guitar, piano, etc.
2. **Mismatched algorithm** — the current extraction is a patchwork of Meyda (optional, for
rms/energy/spectralCentroid) and a manual FFT slice (for bass/treble/beat). This is not the
Muon approach.
3. **Low FFT resolution** — current `fftSize = 512` gives 256 bins; Muon uses `fftSize = 1024`
(512 bins) for better low-frequency resolution.
4. **Limited feature set** — only 7 output fields. Muon outputs 7 frequency bands, BPM, beat
phase, pattern detection (drop/buildup/silence), spectral rolloff, flatness, ZCR.
5. **Latent audio-muting bug** — the single-audio fallback path calls
`createMediaElementSource(audioEl)` (which reroutes the element through Web Audio) but only
connects it onward through `silentGain(0) → destination`, muting the audio.
---
## Why single-source (original song file) rather than all-stem mix
| Concern | Single source | All stems mixed |
|---|---|---|
| Tap nodes | 1 always | N (one per stem, grows with separator) |
| Effect cleanup | Disconnect 1 node | Track and disconnect N nodes |
| Effect deps | Can drop `stemGainRefs` | Must keep `stemGainRefs` in deps |
| Stems-load-late race | No race — audio element always exists | Effect must restart when stem graph is ready |
| Beat detection quality | Full mix = richest low-freq signal | Same once all stems connected |
| Mute/solo sensitivity | Visualizer ignores user muting | Visualizer responds to audible mix |
The last row is the only meaningful trade-off. Mute/solo affecting the visualizer is arguably
wrong behavior anyway — the visual should reflect the song's energy, not the listener's temporary
mixing choices.
---
## Target state
```
analysisAudioRef (<audio>, hidden, src = analysisAudioUrl ?? audioUrl) ──► MediaElementAudioSource
└──► AnalyserNode (fftSize 1024)
└──► MuonFeatureExtractor.extractLoop()
│
└──► AudioFeatures (expanded interface)
├── bass, treble, rms, drumIntensity, beatDetected (kept, remapped)
├── frequencyBands { sub, bass, lowMid, mid, highMid, presence, brilliance }
├── beatInfo { isBeat, bpm, confidence, timeSinceLastBeat, beatPhase }
└── patternInfo { isBeatDrop, isBuildUp, isSilence, intensity, energyDelta }
```
In multi-stem mode, stems continue to drive audible output through their GainNodes as before.
In single-audio mode, the visible player audio element remains the audible source.
`analysisAudioRef` is analysis-only and is **never connected to `ctx.destination`**.
`analysisAudioRef` is synchronized to the active transport clock (single-track audio, stem anchor,
or video clock) so analysis stays phase-aligned with what the listener hears.
---
## Files affected
| File | Change |
|---|---|
| `frontend/src/audio/MuonFeatureExtractor.ts` | **New** — adapted FeatureExtractor class |
| `frontend/src/hooks/useAudioFeatures.ts` | **Modified** — single-source tap + new extractor |
| `frontend/src/components/LyricsLivePreview.tsx` | **Modified** — add hidden always-mounted full-mix analysis audio + sync strategy |
| `frontend/src/components/KaraokePerformanceMode.tsx` | **Modified** — add/forward `analysisAudioUrl` |
| `frontend/src/pages/KaraokeTV.tsx` | **Modified** — derive/pass `analysisAudioUrl` |
| `frontend/src/pages/Karaoke.tsx` | **Modified** — derive/pass `analysisAudioUrl` |
| `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx` | No immediate change (backward-compatible fields preserved) |
| `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx` | No immediate change |
| `SUMMARIES (Claude).md` | Append entry after implementation |
Meyda optional import can be removed once the new extractor is in place.
---
## Step 1 — New file: `frontend/src/audio/MuonFeatureExtractor.ts`
Inline the `FeatureExtractor` class from the Muon Audio Analysis repo, stripped of all Zustand
store dependencies and adapted to return features via a callback instead of writing to a store.
### API
```typescript
export interface MuonFrequencyBands {
sub: number // 20–60 Hz
bass: number // 60–250 Hz
lowMid: number // 250–500 Hz
mid: number // 500–2 000 Hz
highMid: number // 2 000–4 000 Hz
presence: number // 4 000–6 000 Hz
brilliance: number // 6 000–20 000 Hz
}
export interface MuonBeatInfo {
isBeat: boolean
bpm: number
confidence: number
timeSinceLastBeat: number
beatPhase: number // 0–1 position in current beat cycle
}
export interface MuonPatternInfo {
isBeatDrop: boolean
isBuildUp: boolean
isSilence: boolean
intensity: number // 0–1 overall
energyDelta: number // rate of change
}
export interface MuonFrame {
rms: number
energy: number
spectralCentroid: number
spectralRolloff: number
spectralFlatness: number
zcr: number
frequencyBands: MuonFrequencyBands // raw linear energy values (not yet normalized)
beatInfo: MuonBeatInfo
patternInfo: MuonPatternInfo
floatFreqData: Float32Array // raw dB values from analyser (copy, not aliased)
floatTimeData: Float32Array // raw waveform (copy)
}
export class MuonFeatureExtractor {
private isRunning = false
private animationFrameId: number | null = null
private frequencyData!: Float32Array
private timeData!: Float32Array
private energyHistory: number[] = []
private lastBeatTime = 0
private beatThreshold = 1.5
private bpm = 0
private beatTimes: number[] = []
start(analyser: AnalyserNode, ctx: AudioContext, onFrame: (f: MuonFrame) => void): void
stop(): void
setBeatThreshold(threshold: number): void // 0.0–3.0, default 1.5
reset(): void
}
```
### Implementation notes
- Copy `extractFeatures()`, `extractFrequencyBands()`, `detectBeat()`, `detectPatterns()`,
`calculateLocalAverageAndVariance()` verbatim from the Muon repo source
(`src/audio/FeatureExtractor.ts`).
- Remove all `useAudioStore.getState()` calls; replace with a single `onFrame(frame)` call at
the end of `extractLoop`.
- The RAF loop is `requestAnimationFrame`-based — no change needed.
- Define `AUDIO_CONFIG` constants (FFT_SIZE 1024, SMOOTHING 0.8, MIN_DECIBELS -100,
MAX_DECIBELS -30) and `FREQUENCY_BANDS` Hz ranges as local module-level constants — no import
from the Muon repo's `@core/constants`.
- `ENERGY_HISTORY_SIZE = 43` (≈1 second at 60 fps, same as Muon source).
- Return `floatFreqData` and `floatTimeData` as **copies** (`new Float32Array(this.frequencyData)`)
so the caller can store them without aliasing issues.
---
## Step 2 — Modified: `frontend/src/hooks/useAudioFeatures.ts`
### 2a — Expand the `AudioFeatures` interface
```typescript
export interface AudioFeatures {
// ── Backward-compatible convenience fields (remapped from Muon data) ──
bass: number // frequencyBands.bass normalized 0–1
treble: number // (frequencyBands.presence + frequencyBands.brilliance) / 2, normalized 0–1
rms: number // 0–1
energy: number // 0–1
loudness: number // kept for API compat; set equal to rms
spectralCentroid: number
drumIntensity: number // derived from sub + bass bands — see §2b
beatDetected: boolean // beatInfo.isBeat
// ── Muon full feature set ──
frequencyBands: {
sub: number // normalized 0–1
bass: number
lowMid: number
mid: number
highMid: number
presence: number
brilliance: number
}
beatInfo: {
isBeat: boolean
bpm: number
confidence: number
timeSinceLastBeat: number
beatPhase: number
}
patternInfo: {
isBeatDrop: boolean
isBuildUp: boolean
isSilence: boolean
intensity: number
energyDelta: number
}
spectralRolloff: number
spectralFlatness: number
zcr: number
// ── Raw buffers ──
freqData: Float32Array // dB values (was Uint8Array — see §2f)
updatedAt: number
}
```
### 2b — `drumIntensity` re-derivation
The existing formula used the drums stem signal to isolate kick energy. With the full-mix
analyser, `frequencyBands.sub` (20–60 Hz) is the cleanest kick/sub proxy. Replace:
```typescript
function deriveDrumIntensity(sub: number, bass: number, rms: number): number {
// sub and bass are already normalized 0–1 (auto-gain applied)
const expSub = 1.001 * (1 + sub * 2) - 1.001
const expBass = 1.001 * (1 + bass * 2) - 1.001
const raw = (expSub * 0.50 + expBass * 0.38 + rms * 0.12) / 2.0
return clamp01(Math.pow(clamp01(raw), 0.82))
}
```
`sub` takes 0.50 weight (up from `treble`'s 0.08 in the old formula) because sub-bass is where
kick drums live.
### 2c — Band normalization (auto-gain)
`MuonFeatureExtractor.extractFrequencyBands()` returns raw RMS linear energy values, not 0–1.
Normalize with a **running peak + slow decay**:
```typescript
// Lives outside the useEffect (or inside as a closure) — persists across frames.
const bandPeaks = {
sub: 0.001, bass: 0.001, lowMid: 0.001, mid: 0.001,
highMid: 0.001, presence: 0.001, brilliance: 0.001,
}
const PEAK_DECAY = 0.9998 // per-frame (~0.01 dB/s at 60 fps — very slow)
// Inside onFrame, per band:
bandPeaks.bass = Math.max(bandPeaks.bass * PEAK_DECAY, rawBands.bass)
const normalizedBass = clamp01(rawBands.bass / bandPeaks.bass)
```
Peaks start at `0.001` to prevent divide-by-zero on startup. On the first loud frame they snap
to the true peak, then decay slowly — self-calibrating to the mix level within ~2 seconds.
### 2d — Single-source tap via hidden analysis audio (replaces all current tap logic)
Remove the entire drum/fallback/single-audio branch. Replace with:
```typescript
const audioEl = analysisAudioRef.current
if (!audioEl) return
// createMediaElementSource can only be called once per element.
// Re-use the cached node if the effect re-runs (e.g., on enabled toggle).
try {
mediaSourceRef.current = mediaSourceRef.current ?? ctx.createMediaElementSource(audioEl)
} catch {
return // AudioContext is closed or element already owned by another context
}
const mediaSource = mediaSourceRef.current
const analyser = ctx.createAnalyser()
analyser.fftSize = 1024
analyser.smoothingTimeConstant = 0.8
analyser.minDecibels = -100
analyser.maxDecibels = -30
// Analysis-only path: never connect this element to destination.
mediaSource.connect(analyser)
```
**§2d-note — Hidden analysis element + sync strategy (`LyricsLivePreview.tsx`):**
Add a dedicated ref that is mounted in both single and multi-stem sessions:
```tsx
const analysisAudioRef = useRef<HTMLAudioElement>(null)
```
Render a hidden analysis element whenever `analysisAudioUrl || audioUrl` exists:
```tsx
{(analysisAudioUrl || audioUrl) && (
<audio
ref={analysisAudioRef}
src={analysisAudioUrl ?? audioUrl}
preload="auto"
crossOrigin="anonymous"
muted
style={{ display: 'none' }}
/>
)}
```
Sync policy:
- Mirror `currentTime` from active transport (`audioRef`, stem anchor, or `videoRef`).
- Mirror `playbackRate`.
- On play/resume: set time, then `analysisAudioRef.current?.play()`.
- On pause/stop/end: `analysisAudioRef.current?.pause()`.
- On seek: immediately set `analysisAudioRef.current.currentTime = seekTarget`.
- In RAF sync loop, if drift exceeds ~40 ms, snap analysis time to transport time.
This guarantees one stable analysis source without audible doubling.
**Source policy integration (`KaraokeTV.tsx` / `Karaoke.tsx`):**
- Project mode: prefer source/original recording URL as `analysisAudioUrl`.
- SEK mode: until true full-mix is available, derive `analysisAudioUrl` from a ranked stem
fallback (leading/drums preferred; first stem acceptable initial implementation).
- Always pass `analysisAudioUrl` through `KaraokePerformanceMode` into `LyricsLivePreview`.
**SEK seek prerequisite:**
For SEK fallback analysis to stay sync-accurate on seeks, the SEK asset endpoint must support
HTTP byte ranges (`Accept-Ranges: bytes`, valid `206` responses, `416` handling for invalid
ranges). This is covered by `2026-04-13_SEKAsset_RangeRequest_Plan.md` and is a dependency for
reliable `analysisAudioRef.currentTime` snapping.
```
### 2e — Cleanup
```typescript
return () => {
mounted = false
extractor.stop()
try { mediaSource.disconnect(analyser) } catch { /* noop */ }
try { analyser.disconnect() } catch { /* noop */ }
}
```
The `mediaSourceRef` is preserved across effect re-runs (the `??=` creation guard above). Since
the analysis element is never connected to destination, cleanup only detaches analyser links.
### 2f — Replace Meyda + tick
Remove the dynamic `import(/* @vite-ignore */ 'meyda')` block and the entire RAF `tick()`
function. The `MuonFeatureExtractor` subsumes Meyda's role and runs its own RAF loop.
```typescript
const extractor = new MuonFeatureExtractor()
extractor.start(analyser, ctx, (frame) => {
if (!mounted) return
if (!isPlayingRef.current) {
// Zero out reactive fields while paused; leave beatInfo/patternInfo/freqData as-is.
featuresRef.current.bass = 0
featuresRef.current.treble = 0
featuresRef.current.rms = 0
featuresRef.current.drumIntensity = 0
featuresRef.current.beatDetected = false
return
}
// Normalize each band
const b = frame.frequencyBands
const keys = ['sub','bass','lowMid','mid','highMid','presence','brilliance'] as const
const norm: Record<typeof keys[number], number> = {} as any
for (const k of keys) {
bandPeaks[k] = Math.max(bandPeaks[k] * PEAK_DECAY, b[k])
norm[k] = clamp01(b[k] / bandPeaks[k])
}
const rms = clamp01(frame.rms)
const treble = clamp01((norm.presence + norm.brilliance) / 2)
featuresRef.current = {
bass: norm.bass,
treble,
rms,
energy: clamp01(frame.energy),
loudness: rms,
spectralCentroid: frame.spectralCentroid,
drumIntensity: deriveDrumIntensity(norm.sub, norm.bass, rms),
beatDetected: frame.beatInfo.isBeat,
frequencyBands: norm,
beatInfo: frame.beatInfo,
patternInfo: frame.patternInfo,
spectralRolloff: frame.spectralRolloff,
spectralFlatness: frame.spectralFlatness,
zcr: frame.zcr,
freqData: frame.floatFreqData,
updatedAt: Date.now(),
}
})
```
### 2g — useEffect dependency array
Remove `stemGainRefs` from deps — the tap no longer depends on them. Keep
`analysisSourceKey` in deps so source switches recreate analyser wiring:
```typescript
}, [audioCtxRef, analysisAudioRef, isPlayingRef, enabled, analysisSourceKey])
```
`stemGainRefs` can be removed from `UseAudioFeaturesArgs` entirely if it has no other use in the
hook after the tap logic is replaced.
---
## Step 3 — Default values for new interface fields
```typescript
const EMPTY_FREQ = new Float32Array(0) // was Uint8Array
const EMPTY_BANDS = { sub: 0, bass: 0, lowMid: 0, mid: 0, highMid: 0, presence: 0, brilliance: 0 }
const EMPTY_BEAT = { isBeat: false, bpm: 0, confidence: 0, timeSinceLastBeat: 0, beatPhase: 0 }
const EMPTY_PAT = { isBeatDrop: false, isBuildUp: false, isSilence: true, intensity: 0, energyDelta: 0 }
const featuresRef = useRef<AudioFeatures>({
bass: 0, treble: 0, rms: 0, energy: 0, loudness: 0,
spectralCentroid: 0, spectralRolloff: 0, spectralFlatness: 0, zcr: 0,
drumIntensity: 0, beatDetected: false,
frequencyBands: EMPTY_BANDS,
beatInfo: EMPTY_BEAT,
patternInfo: EMPTY_PAT,
freqData: EMPTY_FREQ,
updatedAt: 0,
})
```
---
## Step 4 — Canvas files (backward-compatible; selected Muon enhancements already landed)
`MuonVisualizerCanvas` and `FluidVisualizerCanvas` read: `f.bass`, `f.treble`, `f.rms`,
`f.drumIntensity`, `f.beatDetected`. All five are backward-compatible remapped fields in the
expanded interface — no changes needed in either canvas file.
Current visualizer integration is backward-compatible and already consumes compatibility fields.
Additionally, targeted Muon enhancements now exist in canvas logic (sub-band weighting,
`beatPhase` shaping, and drop-trigger accenting), so these are no longer purely “future”.
Still optional for later parity work:
- tighter phase-locked geometry states derived from `beatInfo.beatPhase`
- richer transition choreography keyed on `patternInfo` sequences (build-up → drop)
- broader usage of normalized per-band vectors across particle subsystems
---
## Step 5 — Remove Meyda (cleanup after verification)
```bash
grep -r "meyda" frontend/src/ --include="*.ts" --include="*.tsx"
```
If the only remaining reference was the deleted dynamic import, run:
```bash
cd frontend && npm uninstall meyda
```
Also remove stale type-only leftovers if unused:
- `frontend/src/types/meyda.d.ts`
Keep it only if another package still imports Meyda types.
---
## Implementation order
1. Write `frontend/src/audio/MuonFeatureExtractor.ts`
2. Add `analysisAudioRef` to `UseAudioFeaturesArgs`; update call-site in `LyricsLivePreview.tsx`
3. Expand `AudioFeatures` interface and initial ref value
4. Add hidden always-mounted full-mix analysis `<audio>` in `LyricsLivePreview.tsx`
5. Implement analysis sync strategy (play/pause/seek/time drift correction)
6. Replace tap logic with single-source analysis routing (`mediaSource → analyser` only)
7. Add `bandPeaks` / `PEAK_DECAY` normalization state
8. Replace Meyda import + `tick()` with `extractor.start()` + `onFrame` callback
9. Update `deriveDrumIntensity` signature
10. Add/propagate `analysisAudioUrl` (`KaraokeTV`/`Karaoke` → `KaraokePerformanceMode` → `LyricsLivePreview`)
11. Ensure `analysisSourceKey` tracks `analysisAudioUrl ?? audioUrl ?? null`
12. Trim `stemGainRefs` from hook args if no longer needed
13. Run `tsc --noEmit`
14. Verify SEK seeking over `/karaoke-sek/packages/{id}/asset` still returns proper `206` on range probes
15. Remove Meyda if clean
16. Update `SUMMARIES (Claude).md`
---
## Risks and mitigations
| Risk | Mitigation |
|---|---|
| Audio doubling in multi-stem mode | Analysis element is never connected to `ctx.destination` (`mediaSource → analyser` only). |
| `createMediaElementSource` throws if audio element already owned | Wrap in try/catch; bail early. The `mediaSourceRef` guard (`??=`) prevents double-creation across effect re-runs. |
| Analysis and audible transport drift apart | Mirror `playbackRate`, sync on play/seek, and snap when drift exceeds ~40 ms. |
| Analysis source missing in stem sessions (`audioUrl` undefined) | Use `analysisAudioUrl ?? audioUrl`; derive `analysisAudioUrl` in page-level containers for both project and SEK modes. |
| Band normalization near-zero for first frames | Peaks start at 0.001; clamp01 never goes negative. Self-calibrates within ~2 seconds of loud audio. |
| `freqData` type change (`Uint8Array` → `Float32Array`) breaks a consumer | Audit: `grep -r "freqData" frontend/src/`. Values are now dB floats (-100 to 0) rather than 0–255 integers — update any display code. |
| Extractor RAF loop burns CPU while paused | `onFrame` early-returns if `!isPlayingRef.current`, zeroing reactive fields. The extractor RAF still ticks but the callback is almost free when returning immediately. Alternatively call `extractor.stop()` on pause and `extractor.start()` on resume. |
| `analysisAudioRef` is null on first render | `if (!audioEl) return` guard in useEffect; hidden element mounts when `analysisAudioUrl || audioUrl` becomes available. |
| Hidden analysis playback rejected (`play()` Promise reject) | Keep transport state authoritative; catch and ignore reject, retry on explicit user gesture/next resume edge. |
| SEK analysis seek desync on scrub | Ensure backend supports byte ranges for SEK assets; without `206` support, browser seek and `currentTime` sync are unreliable. |
| Cross-origin media blocks analysis | Keep `crossOrigin="anonymous"`; ensure source routes deliver compatible CORS/auth behavior (proxy-backed routes preferred). |
+264
View File
@@ -0,0 +1,264 @@
# Muon Visualizer Reference-Parity Plan
Date: 2026-04-13
Scope: Bring StemEngine’s `muon` visualizer behavior and look closer to the reference at
https://muon-vis.netlify.app/ (multi-ring pattern diversity, emission lifecycle, mirror/division
controls, and stronger pattern variation under audio).
---
## Goal
Current StemEngine Muon is a simplified concentric-dot renderer. This plan upgrades it toward
reference parity by adding:
1. Variable ring divisions and mirror mode
2. Multiple per-ring pattern families (not one shared sinusoid)
3. Particle emission lifecycle (`lifespan`, trail persistence)
4. Noise deformation (`noise_scale`, `noise_force`) actually used in rendering
5. Dynamic radius behavior tied to beat/energy deltas (reference-like responsiveness)
This is parity-oriented, not bit-for-bit visual cloning.
---
## Current gaps (vs reference)
In `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx` today:
- Rings are fixed to 8 lanes (`lane: i % 8`)
- Per-ring shape is one shared formula (`sin(angle * aperture)`-style)
- No true emitter lifecycle (`lifespan`, decay, trail populations)
- `noise_scale` and `noise_force` are configured but not applied
- No mirror mode or alternate geometry profile switching
Result: consistent concentric dots, limited ring pattern variation.
---
## Design target
### A) Ring topology + mirror
Add configurable ring topology:
- `divisions` (e.g. 8–48): ring count / angular partitioning
- `particle_mirror` (boolean): mirror opposite-side emissions/positions
### B) Pattern families
Each ring chooses one of several pattern kernels:
- `orbital`
- `flower`
- `ripple`
- `spiro`
- `strobe-grid`
Pattern assignment rotates over rings and can be preset-driven.
### C) Emission + lifespan
Replace direct stateless point plotting with emitted particles:
- `lifespan` in frames/ms
- `birth`, `age`, `alpha`, `size`, `velocity`
- decays and optional trail persistence
### D) Audio response model
Use richer Muon features already available in `AudioFeatures`:
- kick/sub: `frequencyBands.sub`
- brightness: `presence + brilliance`
- beat phase: `beatInfo.beatPhase`
- drops/build-ups: `patternInfo.isBeatDrop` / `isBuildUp`
- energy delta gate for `dynamic_radius`
### E) Noise deformation
Use `noise_scale` + `noise_force` to deform particle radius/angle per ring.
---
## Data model additions
Update muon visualizer settings schema (non-breaking defaults):
```ts
muon: {
// existing keys remain
divisions?: number // default 16
lifespan?: number // default 180
particle_mirror?: boolean // default false
dynamic_radius?: number // default 1.0
delta_response_limit?: number // default 0.005
visualization_preset?: 'default'|'orbital'|'flower'|'spiro'|'hybrid'
}
```
Keep all current keys (`radius_multiplier`, `aperture`, `spacing`, `particle_count`, etc.) intact.
---
## Implementation phases
## Phase 1 — Renderer architecture upgrade
### Files
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
### Tasks
1. Introduce particle pool/lifecycle structures:
- `type MuonParticle = { x,y,vx,vy,ring,age,maxAge,size,hue,alpha,pattern }`
2. Add emitter step and update step
3. Add ring map from `divisions`
4. Add mirror emission if enabled
5. Keep old rendering as fallback branch behind a debug flag during migration (temporary)
### Acceptance
- Visual output shows obvious multi-pattern ring differentiation even on idle playback
- No FPS collapse on default settings (60 fps target on desktop)
---
## Phase 2 — Pattern kernels + noise
### Files
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
- `frontend/src/components/visualizers/muon/patterns.ts` (new)
- `frontend/src/components/visualizers/muon/noise.ts` (new, lightweight noise util)
### Tasks
1. Implement pattern kernels:
- `computeOrbital()`, `computeFlower()`, `computeRipple()`, `computeSpiro()`
2. Implement ring-level pattern assignment strategy
3. Apply noise deformation using `noise_scale` / `noise_force`
4. Blend phase-lock behavior via `beatPhase`
### Acceptance
- At least 3 visibly distinct concentric ring behaviors simultaneously
- `noise_scale` and `noise_force` slider changes have immediate visual effect
---
## Phase 3 — Audio-reactive dynamics parity
### Files
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
### Tasks
1. Add dynamic radius gate:
- reacts only when `abs(energyDelta) >= delta_response_limit`
2. Use `dynamic_radius` multiplier to smooth/boost radial expansion
3. Add drop behavior presets:
- ring preset swap/intensity boost on `isBeatDrop`
4. Improve per-ring hue strategy:
- shared drift + ring offset + beat-phase perturbation
### Acceptance
- Visual response tracks beat/drop transitions more like reference behavior
- No permanent over-saturation after repeated drops
---
## Phase 4 — Controls, presets, persistence
### Files
- `frontend/src/components/PerformanceSidebar.tsx`
- `frontend/src/components/visualizers/VisualizerBackdrop.tsx`
- `frontend/src/components/LyricsLivePreview.tsx`
- `frontend/src/api/client.ts` (if typing expansion required)
### Tasks
1. Add UI sliders/toggles for new Muon fields:
- `divisions`, `lifespan`, `particle_mirror`, `dynamic_radius`, `delta_response_limit`
2. Add `visualization_preset` selector and map to kernel mixes
3. Ensure normalization defaults and clamping in `LyricsLivePreview` config normalization path
4. Keep manifest compatibility: missing fields default safely
### Acceptance
- New controls visible and functional in sidebar
- Presets produce clearly different multi-ring signatures
---
## Phase 5 — QA + tuning
### Tasks
1. Build tuning matrix:
- low/medium/high BPM
- bass-heavy / treble-heavy / speech-like tracks
2. Performance checks:
- ensure particle count + lifespan don’t degrade below acceptable frame rate
3. Add hard caps:
- max active particles by quality tier
### Acceptance
- `npm run -s typecheck` passes
- `npm run -s build` passes
- No runtime errors when switching styles rapidly (`muon`↔`fluid`↔`video`)
---
## Compatibility and migration notes
- Existing saved manifests must continue to render with defaults.
- New keys are optional and clamped.
- Keep current scalar compatibility fields from `AudioFeatures`; consume richer fields where present.
---
## Risks and mitigations
1. **Performance regressions from lifecycle particles**
Mitigation: pool particles + cap by quality tier (`low`/`balanced`/`high`).
2. **Too many controls causing UX complexity**
Mitigation: advanced section collapsed by default + presets as primary entrypoint.
3. **Visual drift from reference despite upgrades**
Mitigation: define 3 visual parity checkpoints (ring diversity, phase motion, drop transitions) and tune iteratively.
4. **Manifest schema drift**
Mitigation: all new fields optional; normalize with defaults before use.
---
## File change map (planned)
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx` (major rewrite)
- `frontend/src/components/visualizers/muon/patterns.ts` (new)
- `frontend/src/components/visualizers/muon/noise.ts` (new)
- `frontend/src/components/PerformanceSidebar.tsx` (new controls + preset selector)
- `frontend/src/components/visualizers/VisualizerBackdrop.tsx` (settings type expansion)
- `frontend/src/components/LyricsLivePreview.tsx` (settings defaults + normalization expansion)
- `SUMMARIES (Claude).md` (implementation entry)
- `20260313_Fixes (Copilot).md` (implementation entry)
---
## Implementation order
1. Add pattern + noise helper modules
2. Rewrite `MuonVisualizerCanvas` with lifecycle + pattern kernels + mirror/divisions
3. Wire audio-reactive dynamic radius and thresholding
4. Expand settings type/defaults/normalization
5. Add sidebar controls + presets
6. Tune and cap performance by quality tier
7. Validate (`typecheck`, `build`)
8. Document
+310 -1
View File
@@ -9723,4 +9723,313 @@ Fixes:
Validation:
- `npm run typecheck` ✅
- `npm run typecheck` ✅
---
### 2026-04-13 — Muon audio analyzer integration (hidden full-mix analysis audio)
Requested follow-up:
- Implement the Muon analyzer migration plan.
- Use a hidden always-mounted full-mix `<audio>` element for analysis source.
- Update project documentation logs.
Implemented:
`frontend/src/audio/MuonFeatureExtractor.ts` (new)
- Added `MuonFeatureExtractor` with RAF extraction loop.
- Configured analyser to Muon-style defaults (`fftSize=1024`, smoothing `0.8`, dB range `-100..-30`).
- Added feature extraction outputs:
- `rms`, `energy`
- `spectralCentroid`, `spectralRolloff`, `spectralFlatness`, `zcr`
- 7-band energy map (`sub`, `bass`, `lowMid`, `mid`, `highMid`, `presence`, `brilliance`)
- beat info (`isBeat`, `bpm`, `confidence`, `timeSinceLastBeat`, `beatPhase`)
- pattern info (`isBeatDrop`, `isBuildUp`, `isSilence`, `intensity`, `energyDelta`)
- Returns `floatFreqData`/`floatTimeData` as copied buffers per frame.
`frontend/src/hooks/useAudioFeatures.ts`
- Removed prior Meyda + manual byte FFT path.
- Migrated hook to `MuonFeatureExtractor` frame callback path.
- Expanded `AudioFeatures` to include full Muon fields while preserving compatibility fields used by existing canvases.
- Changed `freqData` type from `Uint8Array` to `Float32Array`.
- Switched hook source args to analysis audio strategy:
- removed `stemGainRefs` / `singleAudioRef`
- added `analysisAudioRef`
- added `analysisSourceKey` to force safe remount when audio source changes.
- Implemented running-peak band normalization (`PEAK_DECAY=0.9998`).
- Updated `drumIntensity` derivation to use `sub + bass + rms`.
`frontend/src/components/LyricsLivePreview.tsx`
- Added hidden always-mounted full-mix analysis element:
- `analysisAudioRef`
- rendered whenever `audioUrl` exists
- `muted`, hidden, `crossOrigin="anonymous"`, `preload="auto"`.
- Updated `useAudioFeatures()` call site to pass `analysisAudioRef` and `analysisSourceKey`.
- Added `syncAnalysisAudio()` transport sync helper.
- Integrated sync into:
- seek/apply position
- play/resume and pause flows
- reset/finalize/stop flows
- periodic drift correction + playback-rate mirroring while playing.
Dependency cleanup:
- Removed `meyda` from frontend dependencies (`frontend/package.json` + lockfile updated).
Validation:
- `npm run -s typecheck` ✅
### 2026-04-13 — Muon reference parity plan implemented
Requested:
- Implement and document `2026-04-13_Muon_Reference_Parity_Plan.md`.
Implemented:
1) Renderer architecture / lifecycle
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
- Reworked to pooled lifecycle particles (`age`, `maxAge`, velocity decay, alpha fade).
- Added per-frame emission/update pipeline with trail persistence.
- Added ring `divisions`, `particle_mirror`, and quality-tier active-particle caps.
2) Pattern kernels + noise deformation
- `frontend/src/components/visualizers/muon/patterns.ts` (new)
- Added `orbital` / `flower` / `ripple` / `spiro` / `strobe-grid` sampling helpers.
- Added `patternFromPreset()` mapping for preset-driven ring behavior.
- `frontend/src/components/visualizers/muon/noise.ts` (new)
- Added deterministic noise helpers used by Muon deformation.
3) Audio-reactive parity controls
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
- Added dynamic radius gate using `dynamic_radius` + `delta_response_limit`.
- Added drop-triggered pattern shift/boost and richer hue drift behavior.
- Applied `noise_scale` / `noise_force` in renderer path (no longer reserved/no-op).
4) Settings + UI wiring
- `frontend/src/components/visualizers/VisualizerBackdrop.tsx`
- Expanded Muon settings type for new parity fields.
- `frontend/src/components/LyricsLivePreview.tsx`
- Expanded Muon defaults, normalization, and SEK manifest settings mapping.
- `frontend/src/components/PerformanceSidebar.tsx`
- Added controls for: `divisions`, `lifespan`, `particle_mirror`,
`dynamic_radius`, `delta_response_limit`, `visualization_preset`.
- Extended Muon presets with parity fields and added pattern preset selector.
Validation:
- `npm run -s typecheck` ✅
- `npm run -s build` ✅
### 2026-04-13 — Muon plan remainder implemented
Requested:
- Implement remaining tasks from
`docs/plans/2026-04-13_MuonAudioAnalyzer_Integration_Plan.md` and document.
Implemented:
1) SEK analysis fallback ranking
- `frontend/src/pages/KaraokeTV.tsx`
- `frontend/src/pages/Karaoke.tsx`
- Added `pickSekAnalysisStem()` + `rankSekAnalysisStem()` helpers to select a more
representative fallback source in SEK mode:
- priority: drums/kick/percussion → leading/vocals → backing/instrumental → others.
2) Meyda cleanup artifact
- `frontend/src/types/meyda.d.ts`
- Removed stale `declare module 'meyda'` typings content (runtime already migrated to
`MuonFeatureExtractor`).
Validation:
- `npm run -s typecheck` ✅
- `npm run -s build` ✅
### 2026-04-13 — Fluid brightening by audio intensity + plan mirror
Done:
1) Mirrored plan edits
- Synced root plan copy with integrated `docs/plans` updates:
- `2026-04-13_MuonAudioAnalyzer_Integration_Plan.md`
2) Fluid brightening response
- `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx`
- Added dynamic bloom modulation from audio intensity:
- `BLOOM_INTENSITY` now scales with blended `kick + rms + treble` and drop boost.
- `BLOOM_THRESHOLD` now lowers with intensity so louder moments bloom more.
- `frontend/src/components/visualizers/fluid/fluidSim.ts`
- Increased synthetic drag/pressure scaling from splat velocity.
- Added intermediate pointer move sample to strengthen pressure-like splat behavior.
Validation:
- `npm run -s build` ✅
Follow-up runtime fix:
- Fixed Vite import-analysis failure for `webgl-fluid` in `fluidSim.ts`.
- Vendored `webgl-fluid.mjs` under `frontend/src/components/visualizers/fluid/` and switched the wrapper to `import('./webgl-fluid.mjs')`.
- Added `frontend/src/types/mjs-modules.d.ts` so TypeScript accepts vendored `.mjs` imports.
- Validation: `npm run -s typecheck` ✅, `npm run -s build` ✅
### 2026-04-13 — Visualizer diagnosis plan fully implemented
Completed full implementation from `2026-04-13_Visualizer_DiagnosisAndFixes_Plan.md`.
Shipped changes:
1) Analysis source fixes (A + A1)
- Added `analysisAudioUrl` propagation and usage:
- `frontend/src/components/LyricsLivePreview.tsx`
- `frontend/src/components/KaraokePerformanceMode.tsx`
- `frontend/src/pages/KaraokeTV.tsx`
- `frontend/src/pages/Karaoke.tsx`
- Analysis hook now keys by `analysisAudioUrl ?? audioUrl ?? null`.
2) Muon color variation fix (B)
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
- Per-lane hue offset when `mono_color` is disabled.
3) Fluid migration to WebGL path (E)
- `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx`
- Rewritten to WebGL fluid wrapper + audio-driven splat injection.
- Includes WebGL absence fallback and cleanup with context-loss attempt.
- `frontend/src/components/visualizers/fluid/fluidSim.ts` (new)
- Wrapper over `webgl-fluid` exposing `init/splat/update/destroy` helpers.
- `frontend/src/types/webgl-fluid.d.ts` (new)
- Ambient module typing.
- `frontend/package.json` + lockfile
- Added `webgl-fluid` dependency.
4) Fluid preset retune
- `frontend/src/components/PerformanceSidebar.tsx`
- Updated `FLUID_PRESETS` to new mapping values (Default/Ink/Foam/Lava/Mist/Storm).
5) Plan/document alignment
- Updated plan file `2026-04-13_Visualizer_DiagnosisAndFixes_Plan.md` to reflect implemented
wrapper approach and files/order updates.
- Updated summary logs.
Validation:
- `npm run -s typecheck` ✅
### 2026-04-13 — SEK asset byte-range support + TrackDetail visualizer preview
Requested:
- Review/fix both new plans, then implement and document:
- `2026-04-13_SEKAsset_RangeRequest_Plan.md`
- `2026-04-13_TrackDetail_VisualizerPreview_Plan.md`
Plan corrections applied:
- SEK plan: corrected suffix-range behavior and clarified `Accept-Ranges` wording.
- TrackDetail preview plan: updated to current Muon-era hook API (`analysisAudioRef`, `isPlayingRef`) and direct-media-only Video tab guidance.
Implemented backend:
`backend/routers/karaoke_sek.py`
- Added `Request` param to asset route.
- Added `Accept-Ranges: bytes` + `Content-Length` on full responses.
- Added single-range parsing for `Range: bytes=...`.
- Returns `206` with `Content-Range` for satisfiable ranges.
- Returns `416` with `Content-Range: bytes */total` for invalid ranges.
- Added proper suffix-range (`bytes=-N`) support.
Implemented frontend:
`frontend/src/components/VisualizerPreviewWidget.tsx` (new)
- Added compact 16:9 preview panel with tabs: Muon / Fluid / Video (Video shown only for direct media URLs).
- Added local `<audio>` transport with explicit play/pause handling and `AudioContext` ownership.
- Wired `useAudioFeatures` using current args (`audioCtxRef`, `analysisAudioRef`, `isPlayingRef`, `analysisSourceKey`).
- Added `<video>` play/pause and current-time syncing in Video mode.
- Added visibility resume behavior for suspended contexts.
- Added global audio registry handoff (`registerPlayback`, `unregisterPlayback`) to prevent overlapping playback.
`frontend/src/pages/TrackDetail.tsx`
- Replaced static cover-art tile with `VisualizerPreviewWidget`.
- Added `audioStreamUrl` memo using `subsonicStreamUrl(song.id)`.
`frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
`frontend/src/components/visualizers/FluidVisualizerCanvas.tsx`
- Removed `width`/`height` from effect deps to avoid resize-triggered canvas/RAF restarts.
Validation:
- `npm run -s typecheck` ✅
- `python3 -m py_compile backend/routers/karaoke_sek.py` ✅
### 2026-04-13 — Visualizer diagnosis plan patched + Phase 1 fixes implemented
Requested:
- Patch/incorporate findings from `2026-04-13_Visualizer_DiagnosisAndFixes_Plan.md`.
- Implement and document the practical fixes.
Plan corrections incorporated:
- Included `Karaoke.tsx` alongside `KaraokeTV.tsx` for analysis-source gap closure.
- Updated implementation order to phased rollout (Phase 1 immediate fixes, Phase 2 optional WebGL migration).
- Corrected WebGL context-limit wording in the fallback section.
Implemented (Phase 1):
`frontend/src/components/LyricsLivePreview.tsx`
- Added `analysisAudioUrl?: string` prop.
- Hidden analysis audio now sources from `analysisAudioUrl ?? audioUrl`.
- Updated analysis hook key to `analysisSourceKey: analysisAudioUrl ?? audioUrl ?? null`.
`frontend/src/components/KaraokePerformanceMode.tsx`
- Added `analysisAudioUrl?: string` prop and forwarded it to `LyricsLivePreview`.
`frontend/src/pages/KaraokeTV.tsx`
`frontend/src/pages/Karaoke.tsx`
- Added `analysisAudioUrl` derivation + pass-through:
- project mode uses source media URL,
- SEK mode uses first valid stem URL as fallback analysis source.
`frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
- Added per-lane hue spread (when `mono_color` is false) to resolve frame-level monocolor look.
`frontend/src/components/visualizers/FluidVisualizerCanvas.tsx`
- Fixed `spawnBlob` to take live canvas dimensions (`w`, `h`) from RAF loop.
- Inverted `background_persistence` alpha mapping so higher persistence produces longer trails.
Validation:
- `npm run -s typecheck` ✅
Documentation updates:
- Appended full implementation summary to `SUMMARIES (Claude).md` under 2026-04-13.
### 2026-04-13 — Optional Muon visualizer enhancements (implemented)
Requested follow-up:
- Implement all optional future enhancements from the Muon integration plan.
Implemented:
`frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
- Added kick-focused low-end driver using `f.frequencyBands.sub` blended with legacy bass/drum fields.
- Added smooth beat-phase modulation via `f.beatInfo.beatPhase` (phase pulse/wave) to radius, aperture, glow, and particle motion.
- Added beat-phase wrap detection to supplement beat trigger events.
- Added dramatic beat-drop response via `f.patternInfo.isBeatDrop` with temporary hue/radius-preset escalation.
`frontend/src/components/visualizers/FluidVisualizerCanvas.tsx`
- Added kick-focused burst/decay response using `f.frequencyBands.sub`.
- Added phase-locked sweep emitter driven by `f.beatInfo.beatPhase` for smoother in-cycle motion.
- Added beat-phase wrap as supplemental burst trigger.
- Added dramatic beat-drop response (`f.patternInfo.isBeatDrop`) with temporary density boost and highlight tint overlay.
Validation:
- `npm run -s typecheck` ✅
+310
View File
@@ -2565,3 +2565,313 @@ The `enforcedVisualizerMode` effect includes `visualStyle` in its dep array. Eve
**Bug 2 — Seek slider snaps to 0 after dragging (`LyricsLivePreview.tsx`):**
`endSeekDrag` sets `isSeekingRef.current = false` synchronously after calling `applySeekPosition`. The RAF loop fires before the browser's audio `seeked` event, reads `audio.currentTime` (still returning the pre-seek value because browser seek is async), calls `setCurrentTime(0)`, then the `setTimeout(0)` clears `seekDragTime` — leaving the slider at 0. Fix: defer `isSeekingRef.current = false` until the audio element fires `seeked` (with a 500 ms safety fallback via `seekLockFallbackRef`). `beginSeekDrag` cancels any pending fallback timer from a prior drag.
### Visualizer parameter expansion — implemented
Expanded all visualizer parameters across four files. TypeScript check passes clean (exit 0).
**`LyricsLivePreview.tsx`** — `VizSettings` interface, `VIZ_DEFAULTS`, `normalizeVizSettings`, and `manifestVizSettings` useMemo expanded with all new muon and fluid fields (particle_count, rotation_speed, glow_intensity, particle_size, drum_reactivity, hue_range, beat_sensitivity, ellipse_ratio, idle_speed, noise_scale, noise_force, mono_color; background_persistence, beat_burst_count, treble_pulse, drag, blob_life, color_speed).
**`VisualizerBackdrop.tsx`** — `VisualizerSettings` interface expanded to match all new muon and fluid keys with inline purpose comments.
**`MuonVisualizerCanvas.tsx`** — Rewritten with: settings ref pattern (`settingsRef.current = settings` at render level; read inside RAF loop), `fpsCapRef`, `buildParticles()` helper, per-frame particle count rebuild detection, FPS cap inside RAF. New params applied: `drum_reactivity` as a pre-multiplier on the drum signal, `mono_color` locks hue to `p.seedHue`, `beat_sensitivity` divides beat cooldown, `ellipse_ratio` replaces hardcoded 0.62, `rotationSpeed`/`glowIntensity`/`particleSize`/`idleSpeed` all multiply their respective expressions. useEffect deps: `[featuresRef, paletteRef, width, height]` only.
**`FluidVisualizerCanvas.tsx`** — Rewritten with: settings ref pattern, `fpsCapRef`, `qualityRef`. New params applied: `background_persistence` multiplies `(1 - bass) * 0.16` fill alpha, `beat_burst_count` scales per-beat blob count, `treble_pulse` scales blob radius swell, `drag` divides velocity retention factor, `blob_life` multiplies initial blob life, `color_speed` controls `colorOffset` advance rate (fractional colorIndex). useEffect deps: `[featuresRef, paletteRef, width, height]` only.
**`PerformanceSidebar.tsx`** — `VizSettings` and `VIZ_DEFAULTS` expanded to match. Added `MUON_PRESETS` (Default/Calm/Energetic/Dreamy/Minimal/Cosmic) and `FLUID_PRESETS` (Default/Ink/Foam/Lava/Mist/Storm) as typed constant arrays. `muonPresetMatch`/`fluidPresetMatch` helpers detect active preset by field comparison with ±0.015 tolerance. Section headers simplified to plain divs (removed unused `openRef`/`useState` open/close). Muon section: preset pills row + 12 sliders + mono_color toggle. Fluid section: preset pills row + 9 sliders. Custom amber pill shown when no preset matches.
### Muon audio analyzer integration (implemented, 2026-04-13)
Implemented the full migration from the old stem/Meyda hybrid extractor to a Muon-style
extractor path with a hidden always-mounted full-mix analysis audio element.
Files changed:
- `frontend/src/audio/MuonFeatureExtractor.ts` (new)
- `frontend/src/hooks/useAudioFeatures.ts`
- `frontend/src/components/LyricsLivePreview.tsx`
- `frontend/package.json` / `frontend/package-lock.json`
What shipped:
- Added `MuonFeatureExtractor` class with RAF extraction loop, 1024 FFT config,
frequency-band extraction (`sub`..`brilliance`), beat analysis (`bpm`, confidence,
phase), pattern flags (`drop`, `buildup`, `silence`), and spectral metrics
(`rolloff`, `flatness`, `zcr`).
- Replaced `useAudioFeatures` Meyda + manual byte-FFT pipeline with Muon frame ingestion.
- Expanded `AudioFeatures` interface to include:
- backward-compatible fields (`bass`, `treble`, `rms`, `drumIntensity`, `beatDetected`)
- full Muon fields (`frequencyBands`, `beatInfo`, `patternInfo`, `spectralRolloff`,
`spectralFlatness`, `zcr`)
- `freqData: Float32Array` (analyser dB-domain data).
- Added running-peak/slow-decay band normalization and updated `drumIntensity` derivation
from `sub + bass + rms`.
- Added hidden always-mounted `analysisAudioRef` element in `LyricsLivePreview` and switched
the hook source to that element in all playback modes.
- Implemented analysis sync policy:
- sync on seek/play/pause/reset/finalize
- mirror playback rate
- periodic drift snap when offset exceeds ~40ms.
- Removed `meyda` package from frontend dependencies.
Validation:
- `npm run -s typecheck` passes after integration and cleanup.
### Muon/Fluid optional future enhancements (implemented, 2026-04-13)
Implemented all previously listed optional enhancements in both canvas visualizers.
Files changed:
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
- `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx`
What shipped:
- `frequencyBands.sub` is now used as a kick-focused driver (combined with legacy bass/drum fields).
- `beatInfo.beatPhase` now drives smooth phase-locked animation motion/pulsing.
- `patternInfo.isBeatDrop` now triggers dramatic temporary visual escalation.
`MuonVisualizerCanvas` details:
- Added sub-band kick blend for cleaner low-end reaction.
- Added phase-wave/phase-pulse modulation to radius, aperture, glow, and particle size/alpha.
- Added beat-phase wrap detection as a supplemental beat trigger.
- Added beat-drop boost that shifts hue and jumps radius presets for drop moments.
`FluidVisualizerCanvas` details:
- Added sub-band kick blend for burst sizing/count and decay behavior.
- Added beat-phase orbit/sweep emitter for smoother in-beat motion.
- Added beat-phase wrap as supplemental burst trigger.
- Added beat-drop boost for denser spawns, highlight tint overlay, and larger pulse response.
Validation:
- `npm run -s typecheck` passes.
### SEK asset range support + TrackDetail visualizer preview (implemented, 2026-04-13)
Completed requested review corrections and implementation for both new plans:
- `2026-04-13_SEKAsset_RangeRequest_Plan.md`
- `2026-04-13_TrackDetail_VisualizerPreview_Plan.md`
Plan review corrections applied:
- Updated SEK plan language around `Accept-Ranges` behavior and corrected suffix-range handling notes.
- Updated TrackDetail preview plan to current `useAudioFeatures` API (`analysisAudioRef` + `isPlayingRef`) and direct-media video preview constraints.
Backend implementation:
- `backend/routers/karaoke_sek.py`
- Added `Request` param to `get_asset`.
- Added `Accept-Ranges: bytes` + explicit `Content-Length` on full responses.
- Implemented `Range` handling with `206 Partial Content`.
- Implemented `416 Range Not Satisfiable` for invalid ranges.
- Added proper suffix-range support (`bytes=-N`).
Frontend implementation:
- `frontend/src/components/VisualizerPreviewWidget.tsx` (new)
- Added compact preview widget with Muon/Fluid tabs and optional Video tab.
- Uses local `<audio>` playback + `useAudioFeatures` (analysis from same audio element).
- Added independent `AudioContext` lifecycle and resume-on-visibility handling.
- Added direct-media URL check for Video tab (mp4/webm/m4v/ogg/ogv).
- Added video play/pause + currentTime sync with preview audio.
- Added global playback registry integration (`registerPlayback`/`unregisterPlayback`) to avoid overlap with active players.
- `frontend/src/pages/TrackDetail.tsx`
- Integrated `VisualizerPreviewWidget` in place of static cover thumbnail.
- Added `audioStreamUrl` via `subsonicStreamUrl(song.id)`.
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
- `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx`
- Removed `width`/`height` from effect dependency arrays to prevent resize-triggered RAF resets.
Validation:
- Frontend typecheck: `npm run -s typecheck` ✅
- Backend syntax check: `python3 -m py_compile backend/routers/karaoke_sek.py` ✅
### Visualizer diagnosis plan incorporation + Phase 1 fixes (implemented, 2026-04-13)
Reviewed and incorporated corrections from `2026-04-13_Visualizer_DiagnosisAndFixes_Plan.md`,
then implemented the immediate/safe fix set (A + A1 + B + C + D).
Plan updates applied:
- Corrected implementation scope to include both karaoke pages (`KaraokeTV.tsx` and `Karaoke.tsx`) for analysis source threading.
- Updated rollout to two phases:
- Phase 1: analysis-source gap + Muon hue + Fluid canvas correctness fixes.
- Phase 2: optional high-risk Fluid WebGL migration.
- Corrected WebGL context-limit wording in fallback notes.
Implemented frontend fixes:
- `frontend/src/components/LyricsLivePreview.tsx`
- Added `analysisAudioUrl?: string` prop.
- Hidden analysis element now uses `analysisAudioUrl ?? audioUrl`.
- `useAudioFeatures` now keys by `analysisSourceKey: analysisAudioUrl ?? audioUrl ?? null`.
- `frontend/src/components/KaraokePerformanceMode.tsx`
- Added/forwarded `analysisAudioUrl` prop to `LyricsLivePreview`.
- `frontend/src/pages/KaraokeTV.tsx`
- `frontend/src/pages/Karaoke.tsx`
- Added `analysisAudioUrl` memo derivation:
- Project mode: source file URL.
- SEK mode: first valid stem URL as analysis fallback.
- Passed `analysisAudioUrl` into `KaraokePerformanceMode`.
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
- Added per-lane hue spread when `mono_color` is off:
- `laneHue = (hue + pt.lane * (hueRange * 45)) % 360`.
- `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx`
- Fixed `spawnBlob` to use live canvas dimensions passed from RAF loop (`w`, `h`) instead of captured props.
- Fixed `background_persistence` semantics by inverting alpha scaling:
- higher persistence now yields lower clear alpha (longer trails).
Validation:
- `npm run -s typecheck` ✅
### Muon reference parity plan implemented (2026-04-13)
Implemented from `2026-04-13_Muon_Reference_Parity_Plan.md`:
- New Muon helper modules
- `frontend/src/components/visualizers/muon/patterns.ts`
- Added pattern families: `orbital`, `flower`, `ripple`, `spiro`, `strobe-grid`.
- Added preset-to-pattern mapping via `patternFromPreset(...)`.
- `frontend/src/components/visualizers/muon/noise.ts`
- Added lightweight deterministic 2D noise helpers used for deformation.
- Major renderer upgrade
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
- Replaced fixed concentric particle loop with pooled lifecycle particles.
- Added configurable ring `divisions` and `particle_mirror` emission.
- Added per-particle lifespan/age decay and trail-style persistence.
- Added pattern-driven emission with ring-level variation.
- Applied `noise_scale` / `noise_force` to geometry deformation.
- Added dynamic radius gating with `dynamic_radius` + `delta_response_limit`.
- Added drop-trigger pattern shifting and intensity boost.
- Added quality-tier particle caps for performance safety.
- Settings schema + normalization expansion
- `frontend/src/components/visualizers/VisualizerBackdrop.tsx`
- Expanded Muon settings type with parity keys.
- `frontend/src/components/LyricsLivePreview.tsx`
- Expanded `VizSettings.muon`, defaults, normalization, and SEK manifest mapping for:
`divisions`, `lifespan`, `particle_mirror`, `dynamic_radius`,
`delta_response_limit`, `visualization_preset`.
- Sidebar controls + preset wiring
- `frontend/src/components/PerformanceSidebar.tsx`
- Added UI controls for new Muon keys.
- Added pattern preset selector (`default`, `orbital`, `flower`, `spiro`, `hybrid`).
- Extended Muon named presets to include parity settings.
Validation:
- `npm run -s typecheck` ✅
- `npm run -s build` ✅
### Muon integration plan — remainder implementation (2026-04-13)
Implemented final remaining items from
`docs/plans/2026-04-13_MuonAudioAnalyzer_Integration_Plan.md`:
- SEK analysis-source fallback improvement
- `frontend/src/pages/KaraokeTV.tsx`
- `frontend/src/pages/Karaoke.tsx`
- Replaced plain “first stem” fallback with ranked selection helper:
- prefers `drums`/`kick`/`percussion`, then `leading`/`vocals`, then backing/instrumental.
- Meyda cleanup follow-through
- `frontend/src/types/meyda.d.ts`
- Removed obsolete Meyda module declarations and converted file to cleanup marker comment
after migration to `MuonFeatureExtractor`.
Validation:
- `npm run -s typecheck` ✅
- `npm run -s build` ✅
### Fluid audio-intensity brightening + mirrored plan sync (2026-04-13)
Implemented:
- Mirrored Muon analyzer plan updates to root copy:
- `2026-04-13_MuonAudioAnalyzer_Integration_Plan.md`
- synced with the `docs/plans` version (analysis source policy, dependency updates,
SEK range prerequisite notes, risk table additions).
- Added audio-intensity-driven brightening to WebGL Fluid:
- `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx`
- dynamic `BLOOM_INTENSITY` based on live audio intensity (`kick`, `rms`, `treble`) with
drop boost.
- dynamic `BLOOM_THRESHOLD` lowering on louder frames so more dye contributes to bloom.
- `frontend/src/components/visualizers/fluid/fluidSim.ts`
- strengthened synthetic pointer pressure from splat velocity.
- added intermediate drag sample to better emulate stronger press/drag splats.
Validation:
- `npm run -s build` ✅
Follow-up fix:
- Resolved Vite dev overlay failure for Fluid visualizer import analysis.
- Root cause: package-name import resolution for `webgl-fluid` was failing in the active frontend runtime.
- Fix: vendored `webgl-fluid.mjs` into `src/components/visualizers/fluid/`, changed `fluidSim.ts` to dynamically import the local file, and added `src/types/mjs-modules.d.ts` for TypeScript.
- Revalidated with both `npm run -s typecheck` and `npm run -s build`.
### Visualizer diagnosis plan — full implementation (2026-04-13)
Completed the full implementation scope from
`2026-04-13_Visualizer_DiagnosisAndFixes_Plan.md` and updated the plan doc to match shipped
approach.
Implemented:
- Analysis-source gap closure (Issue A + A1)
- Added `analysisAudioUrl` threading:
- `LyricsLivePreview.tsx`
- `KaraokePerformanceMode.tsx`
- `KaraokeTV.tsx`
- `Karaoke.tsx`
- Analysis audio now mounts from `analysisAudioUrl ?? audioUrl`; Muon feature hook key tracks
whichever source is active.
- Muon lane color fix (Issue B)
- Added per-lane hue spread in `MuonVisualizerCanvas.tsx` while preserving `mono_color` lock.
- Fluid renderer migration (Issue E + edge notes)
- Replaced canvas-blob fluid implementation with WebGL fluid wrapper path in
`FluidVisualizerCanvas.tsx`.
- Added wrapper module `frontend/src/components/visualizers/fluid/fluidSim.ts` around
`webgl-fluid` package, exposing `initFluidSim`, `splatFluid`, `updateFluidConfig`,
`destroyFluidSim`.
- Added fallback rendering path when WebGL is unavailable.
- Added cleanup via `destroyFluidSim` and `WEBGL_lose_context` attempt.
- Added runtime config modulation + audio-driven splat injection using:
- `frequencyBands.sub`, `beatInfo.beatPhase`, `patternInfo.isBeatDrop`, `treble_pulse`.
- Documented behavior: `fpsCap` throttles injection cadence; sim render loop remains internal.
- Fluid presets retuned (Issue E Step 4)
- Updated `FLUID_PRESETS` in `PerformanceSidebar.tsx` to the new parameter mapping table.
- Dependency/type support
- Added `webgl-fluid` dependency in frontend package manifest/lockfile.
- Added ambient module declaration: `frontend/src/types/webgl-fluid.d.ts`.
- Plan file alignment
- Updated `2026-04-13_Visualizer_DiagnosisAndFixes_Plan.md` to reflect implemented wrapper
approach (`webgl-fluid` package + `fluidSim.ts`) and updated files/order list.
Validation:
- `npm run -s typecheck` ✅
+74 -3
View File
@@ -2,7 +2,7 @@ from __future__ import annotations
from urllib.parse import quote
from fastapi import APIRouter, HTTPException, Query
from fastapi import APIRouter, HTTPException, Query, Request
from fastapi.responses import Response
from pydantic import BaseModel
@@ -69,18 +69,89 @@ def get_manifest(package_id: str, folder: str | None = Query(default=None, descr
@router.get("/packages/{package_id}/asset")
def get_asset(
request: Request,
package_id: str,
path: str = Query(..., description="Inner package path, e.g. stems/vocals.mp3"),
folder: str | None = Query(default=None, description="Optional SEK folder override"),
):
try:
data, mime, filename = get_sek_asset(package_id, path, folder)
total = len(data)
base_headers = {
"Accept-Ranges": "bytes",
"Content-Disposition": _safe_content_disposition(filename),
"Cache-Control": "public, max-age=3600",
}
range_header = request.headers.get("range")
if range_header:
try:
units, _, ranges_spec = range_header.partition("=")
if units.strip().lower() == "bytes":
first_range = ranges_spec.split(",", 1)[0].strip()
if "-" not in first_range:
raise ValueError("Malformed range")
start_str, end_str = first_range.split("-", 1)
start_str = start_str.strip()
end_str = end_str.strip()
if start_str == "":
# Suffix range: bytes=-N -> last N bytes
suffix_len = int(end_str)
if suffix_len <= 0:
raise ValueError("Invalid suffix length")
if total <= 0:
return Response(
status_code=416,
headers={
"Content-Range": "bytes */0",
"Content-Length": "0",
**base_headers,
},
)
start = max(total - suffix_len, 0)
end = total - 1
else:
start = int(start_str)
if start < 0:
raise ValueError("Invalid start")
end = int(end_str) if end_str else (total - 1)
if end < 0:
raise ValueError("Invalid end")
if total <= 0 or start >= total or start > end:
return Response(
status_code=416,
headers={
"Content-Range": f"bytes */{total}",
"Content-Length": "0",
**base_headers,
},
)
end = min(end, total - 1)
chunk = data[start : end + 1]
return Response(
content=chunk,
status_code=206,
media_type=mime,
headers={
"Content-Range": f"bytes {start}-{end}/{total}",
"Content-Length": str(len(chunk)),
**base_headers,
},
)
except (TypeError, ValueError):
# Malformed range headers fall back to full response.
pass
return Response(
content=data,
media_type=mime,
headers={
"Content-Disposition": _safe_content_disposition(filename),
"Cache-Control": "public, max-age=3600",
"Content-Length": str(total),
**base_headers,
},
)
except ProjectOperationError as exc:
+2 -2
View File
@@ -19,8 +19,8 @@
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
<meta name="apple-mobile-web-app-title" content="StemEngine" />
<link rel="apple-touch-icon" href="/icon.svg" />
<script type="module" crossorigin src="/assets/index-CqSmBAZ1.js"></script>
<link rel="stylesheet" crossorigin href="/assets/index-CIr2AVKp.css">
<script type="module" crossorigin src="/assets/index-Cyxc7Uk9.js"></script>
<link rel="stylesheet" crossorigin href="/assets/index-B34zlAct.css">
</head>
<body>
<div id="root"></div>
@@ -0,0 +1,534 @@
# Muon Audio Analyzer Integration Plan
Date: 2026-04-13
Scope: Replace the current single-stem/Meyda feature extraction in `useAudioFeatures` with the
Muon Audio Analysis `FeatureExtractor` approach, tapping a hidden analysis `<audio>` element
fed by `analysisAudioUrl ?? audioUrl` (full-mix source when available; stem fallback in SEK).
Reference repo: https://github.com/najafmohammed/Muon-Audio-Analysis
Status: private, not published to npm — the `FeatureExtractor` class must be inlined.
---
## Integrated decisions from related plans (2026-04-13)
This plan incorporates constraints already established in:
- `2026-04-13_Visualizer_DiagnosisAndFixes_Plan.md`
- `2026-04-13_TrackDetail_VisualizerPreview_Plan.md`
- `2026-04-13_SEKAsset_RangeRequest_Plan.md`
Net effect:
1. Analysis source is **not** `audioUrl`-only anymore; it is `analysisAudioUrl ?? audioUrl`.
2. `analysisAudioUrl` must be threaded through `KaraokePerformanceMode` into
`LyricsLivePreview` and included in `analysisSourceKey`.
3. In SEK mode, where no explicit full-mix source exists in the manifest, analysis may use a
ranked stem fallback (initially first/leading/drums stem strategy).
4. Hidden analysis seek sync depends on working HTTP range support for SEK assets
(`/api/karaoke-sek/packages/{id}/asset`) so `currentTime` jumps are reliable.
---
## Current problems
1. **Wrong source** — `useAudioFeatures` taps only the `drums` stem gain node (or first stem as
fallback), so the visualizer is deaf to vocals, guitar, piano, etc.
2. **Mismatched algorithm** — the current extraction is a patchwork of Meyda (optional, for
rms/energy/spectralCentroid) and a manual FFT slice (for bass/treble/beat). This is not the
Muon approach.
3. **Low FFT resolution** — current `fftSize = 512` gives 256 bins; Muon uses `fftSize = 1024`
(512 bins) for better low-frequency resolution.
4. **Limited feature set** — only 7 output fields. Muon outputs 7 frequency bands, BPM, beat
phase, pattern detection (drop/buildup/silence), spectral rolloff, flatness, ZCR.
5. **Latent audio-muting bug** — the single-audio fallback path calls
`createMediaElementSource(audioEl)` (which reroutes the element through Web Audio) but only
connects it onward through `silentGain(0) → destination`, muting the audio.
---
## Why single-source (original song file) rather than all-stem mix
| Concern | Single source | All stems mixed |
|---|---|---|
| Tap nodes | 1 always | N (one per stem, grows with separator) |
| Effect cleanup | Disconnect 1 node | Track and disconnect N nodes |
| Effect deps | Can drop `stemGainRefs` | Must keep `stemGainRefs` in deps |
| Stems-load-late race | No race — audio element always exists | Effect must restart when stem graph is ready |
| Beat detection quality | Full mix = richest low-freq signal | Same once all stems connected |
| Mute/solo sensitivity | Visualizer ignores user muting | Visualizer responds to audible mix |
The last row is the only meaningful trade-off. Mute/solo affecting the visualizer is arguably
wrong behavior anyway — the visual should reflect the song's energy, not the listener's temporary
mixing choices.
---
## Target state
```
analysisAudioRef (<audio>, hidden, src = analysisAudioUrl ?? audioUrl) ──► MediaElementAudioSource
└──► AnalyserNode (fftSize 1024)
└──► MuonFeatureExtractor.extractLoop()
│
└──► AudioFeatures (expanded interface)
├── bass, treble, rms, drumIntensity, beatDetected (kept, remapped)
├── frequencyBands { sub, bass, lowMid, mid, highMid, presence, brilliance }
├── beatInfo { isBeat, bpm, confidence, timeSinceLastBeat, beatPhase }
└── patternInfo { isBeatDrop, isBuildUp, isSilence, intensity, energyDelta }
```
In multi-stem mode, stems continue to drive audible output through their GainNodes as before.
In single-audio mode, the visible player audio element remains the audible source.
`analysisAudioRef` is analysis-only and is **never connected to `ctx.destination`**.
`analysisAudioRef` is synchronized to the active transport clock (single-track audio, stem anchor,
or video clock) so analysis stays phase-aligned with what the listener hears.
---
## Files affected
| File | Change |
|---|---|
| `frontend/src/audio/MuonFeatureExtractor.ts` | **New** — adapted FeatureExtractor class |
| `frontend/src/hooks/useAudioFeatures.ts` | **Modified** — single-source tap + new extractor |
| `frontend/src/components/LyricsLivePreview.tsx` | **Modified** — add hidden always-mounted full-mix analysis audio + sync strategy |
| `frontend/src/components/KaraokePerformanceMode.tsx` | **Modified** — add/forward `analysisAudioUrl` |
| `frontend/src/pages/KaraokeTV.tsx` | **Modified** — derive/pass `analysisAudioUrl` |
| `frontend/src/pages/Karaoke.tsx` | **Modified** — derive/pass `analysisAudioUrl` |
| `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx` | No immediate change (backward-compatible fields preserved) |
| `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx` | No immediate change |
| `SUMMARIES (Claude).md` | Append entry after implementation |
Meyda optional import can be removed once the new extractor is in place.
---
## Step 1 — New file: `frontend/src/audio/MuonFeatureExtractor.ts`
Inline the `FeatureExtractor` class from the Muon Audio Analysis repo, stripped of all Zustand
store dependencies and adapted to return features via a callback instead of writing to a store.
### API
```typescript
export interface MuonFrequencyBands {
sub: number // 20–60 Hz
bass: number // 60–250 Hz
lowMid: number // 250–500 Hz
mid: number // 500–2 000 Hz
highMid: number // 2 000–4 000 Hz
presence: number // 4 000–6 000 Hz
brilliance: number // 6 000–20 000 Hz
}
export interface MuonBeatInfo {
isBeat: boolean
bpm: number
confidence: number
timeSinceLastBeat: number
beatPhase: number // 0–1 position in current beat cycle
}
export interface MuonPatternInfo {
isBeatDrop: boolean
isBuildUp: boolean
isSilence: boolean
intensity: number // 0–1 overall
energyDelta: number // rate of change
}
export interface MuonFrame {
rms: number
energy: number
spectralCentroid: number
spectralRolloff: number
spectralFlatness: number
zcr: number
frequencyBands: MuonFrequencyBands // raw linear energy values (not yet normalized)
beatInfo: MuonBeatInfo
patternInfo: MuonPatternInfo
floatFreqData: Float32Array // raw dB values from analyser (copy, not aliased)
floatTimeData: Float32Array // raw waveform (copy)
}
export class MuonFeatureExtractor {
private isRunning = false
private animationFrameId: number | null = null
private frequencyData!: Float32Array
private timeData!: Float32Array
private energyHistory: number[] = []
private lastBeatTime = 0
private beatThreshold = 1.5
private bpm = 0
private beatTimes: number[] = []
start(analyser: AnalyserNode, ctx: AudioContext, onFrame: (f: MuonFrame) => void): void
stop(): void
setBeatThreshold(threshold: number): void // 0.0–3.0, default 1.5
reset(): void
}
```
### Implementation notes
- Copy `extractFeatures()`, `extractFrequencyBands()`, `detectBeat()`, `detectPatterns()`,
`calculateLocalAverageAndVariance()` verbatim from the Muon repo source
(`src/audio/FeatureExtractor.ts`).
- Remove all `useAudioStore.getState()` calls; replace with a single `onFrame(frame)` call at
the end of `extractLoop`.
- The RAF loop is `requestAnimationFrame`-based — no change needed.
- Define `AUDIO_CONFIG` constants (FFT_SIZE 1024, SMOOTHING 0.8, MIN_DECIBELS -100,
MAX_DECIBELS -30) and `FREQUENCY_BANDS` Hz ranges as local module-level constants — no import
from the Muon repo's `@core/constants`.
- `ENERGY_HISTORY_SIZE = 43` (≈1 second at 60 fps, same as Muon source).
- Return `floatFreqData` and `floatTimeData` as **copies** (`new Float32Array(this.frequencyData)`)
so the caller can store them without aliasing issues.
---
## Step 2 — Modified: `frontend/src/hooks/useAudioFeatures.ts`
### 2a — Expand the `AudioFeatures` interface
```typescript
export interface AudioFeatures {
// ── Backward-compatible convenience fields (remapped from Muon data) ──
bass: number // frequencyBands.bass normalized 0–1
treble: number // (frequencyBands.presence + frequencyBands.brilliance) / 2, normalized 0–1
rms: number // 0–1
energy: number // 0–1
loudness: number // kept for API compat; set equal to rms
spectralCentroid: number
drumIntensity: number // derived from sub + bass bands — see §2b
beatDetected: boolean // beatInfo.isBeat
// ── Muon full feature set ──
frequencyBands: {
sub: number // normalized 0–1
bass: number
lowMid: number
mid: number
highMid: number
presence: number
brilliance: number
}
beatInfo: {
isBeat: boolean
bpm: number
confidence: number
timeSinceLastBeat: number
beatPhase: number
}
patternInfo: {
isBeatDrop: boolean
isBuildUp: boolean
isSilence: boolean
intensity: number
energyDelta: number
}
spectralRolloff: number
spectralFlatness: number
zcr: number
// ── Raw buffers ──
freqData: Float32Array // dB values (was Uint8Array — see §2f)
updatedAt: number
}
```
### 2b — `drumIntensity` re-derivation
The existing formula used the drums stem signal to isolate kick energy. With the full-mix
analyser, `frequencyBands.sub` (20–60 Hz) is the cleanest kick/sub proxy. Replace:
```typescript
function deriveDrumIntensity(sub: number, bass: number, rms: number): number {
// sub and bass are already normalized 0–1 (auto-gain applied)
const expSub = 1.001 * (1 + sub * 2) - 1.001
const expBass = 1.001 * (1 + bass * 2) - 1.001
const raw = (expSub * 0.50 + expBass * 0.38 + rms * 0.12) / 2.0
return clamp01(Math.pow(clamp01(raw), 0.82))
}
```
`sub` takes 0.50 weight (up from `treble`'s 0.08 in the old formula) because sub-bass is where
kick drums live.
### 2c — Band normalization (auto-gain)
`MuonFeatureExtractor.extractFrequencyBands()` returns raw RMS linear energy values, not 0–1.
Normalize with a **running peak + slow decay**:
```typescript
// Lives outside the useEffect (or inside as a closure) — persists across frames.
const bandPeaks = {
sub: 0.001, bass: 0.001, lowMid: 0.001, mid: 0.001,
highMid: 0.001, presence: 0.001, brilliance: 0.001,
}
const PEAK_DECAY = 0.9998 // per-frame (~0.01 dB/s at 60 fps — very slow)
// Inside onFrame, per band:
bandPeaks.bass = Math.max(bandPeaks.bass * PEAK_DECAY, rawBands.bass)
const normalizedBass = clamp01(rawBands.bass / bandPeaks.bass)
```
Peaks start at `0.001` to prevent divide-by-zero on startup. On the first loud frame they snap
to the true peak, then decay slowly — self-calibrating to the mix level within ~2 seconds.
### 2d — Single-source tap via hidden analysis audio (replaces all current tap logic)
Remove the entire drum/fallback/single-audio branch. Replace with:
```typescript
const audioEl = analysisAudioRef.current
if (!audioEl) return
// createMediaElementSource can only be called once per element.
// Re-use the cached node if the effect re-runs (e.g., on enabled toggle).
try {
mediaSourceRef.current = mediaSourceRef.current ?? ctx.createMediaElementSource(audioEl)
} catch {
return // AudioContext is closed or element already owned by another context
}
const mediaSource = mediaSourceRef.current
const analyser = ctx.createAnalyser()
analyser.fftSize = 1024
analyser.smoothingTimeConstant = 0.8
analyser.minDecibels = -100
analyser.maxDecibels = -30
// Analysis-only path: never connect this element to destination.
mediaSource.connect(analyser)
```
**§2d-note — Hidden analysis element + sync strategy (`LyricsLivePreview.tsx`):**
Add a dedicated ref that is mounted in both single and multi-stem sessions:
```tsx
const analysisAudioRef = useRef<HTMLAudioElement>(null)
```
Render a hidden analysis element whenever `analysisAudioUrl || audioUrl` exists:
```tsx
{(analysisAudioUrl || audioUrl) && (
<audio
ref={analysisAudioRef}
src={analysisAudioUrl ?? audioUrl}
preload="auto"
crossOrigin="anonymous"
muted
style={{ display: 'none' }}
/>
)}
```
Sync policy:
- Mirror `currentTime` from active transport (`audioRef`, stem anchor, or `videoRef`).
- Mirror `playbackRate`.
- On play/resume: set time, then `analysisAudioRef.current?.play()`.
- On pause/stop/end: `analysisAudioRef.current?.pause()`.
- On seek: immediately set `analysisAudioRef.current.currentTime = seekTarget`.
- In RAF sync loop, if drift exceeds ~40 ms, snap analysis time to transport time.
This guarantees one stable analysis source without audible doubling.
**Source policy integration (`KaraokeTV.tsx` / `Karaoke.tsx`):**
- Project mode: prefer source/original recording URL as `analysisAudioUrl`.
- SEK mode: until true full-mix is available, derive `analysisAudioUrl` from a ranked stem
fallback (leading/drums preferred; first stem acceptable initial implementation).
- Always pass `analysisAudioUrl` through `KaraokePerformanceMode` into `LyricsLivePreview`.
**SEK seek prerequisite:**
For SEK fallback analysis to stay sync-accurate on seeks, the SEK asset endpoint must support
HTTP byte ranges (`Accept-Ranges: bytes`, valid `206` responses, `416` handling for invalid
ranges). This is covered by `2026-04-13_SEKAsset_RangeRequest_Plan.md` and is a dependency for
reliable `analysisAudioRef.currentTime` snapping.
```
### 2e — Cleanup
```typescript
return () => {
mounted = false
extractor.stop()
try { mediaSource.disconnect(analyser) } catch { /* noop */ }
try { analyser.disconnect() } catch { /* noop */ }
}
```
The `mediaSourceRef` is preserved across effect re-runs (the `??=` creation guard above). Since
the analysis element is never connected to destination, cleanup only detaches analyser links.
### 2f — Replace Meyda + tick
Remove the dynamic `import(/* @vite-ignore */ 'meyda')` block and the entire RAF `tick()`
function. The `MuonFeatureExtractor` subsumes Meyda's role and runs its own RAF loop.
```typescript
const extractor = new MuonFeatureExtractor()
extractor.start(analyser, ctx, (frame) => {
if (!mounted) return
if (!isPlayingRef.current) {
// Zero out reactive fields while paused; leave beatInfo/patternInfo/freqData as-is.
featuresRef.current.bass = 0
featuresRef.current.treble = 0
featuresRef.current.rms = 0
featuresRef.current.drumIntensity = 0
featuresRef.current.beatDetected = false
return
}
// Normalize each band
const b = frame.frequencyBands
const keys = ['sub','bass','lowMid','mid','highMid','presence','brilliance'] as const
const norm: Record<typeof keys[number], number> = {} as any
for (const k of keys) {
bandPeaks[k] = Math.max(bandPeaks[k] * PEAK_DECAY, b[k])
norm[k] = clamp01(b[k] / bandPeaks[k])
}
const rms = clamp01(frame.rms)
const treble = clamp01((norm.presence + norm.brilliance) / 2)
featuresRef.current = {
bass: norm.bass,
treble,
rms,
energy: clamp01(frame.energy),
loudness: rms,
spectralCentroid: frame.spectralCentroid,
drumIntensity: deriveDrumIntensity(norm.sub, norm.bass, rms),
beatDetected: frame.beatInfo.isBeat,
frequencyBands: norm,
beatInfo: frame.beatInfo,
patternInfo: frame.patternInfo,
spectralRolloff: frame.spectralRolloff,
spectralFlatness: frame.spectralFlatness,
zcr: frame.zcr,
freqData: frame.floatFreqData,
updatedAt: Date.now(),
}
})
```
### 2g — useEffect dependency array
Remove `stemGainRefs` from deps — the tap no longer depends on them. Keep
`analysisSourceKey` in deps so source switches recreate analyser wiring:
```typescript
}, [audioCtxRef, analysisAudioRef, isPlayingRef, enabled, analysisSourceKey])
```
`stemGainRefs` can be removed from `UseAudioFeaturesArgs` entirely if it has no other use in the
hook after the tap logic is replaced.
---
## Step 3 — Default values for new interface fields
```typescript
const EMPTY_FREQ = new Float32Array(0) // was Uint8Array
const EMPTY_BANDS = { sub: 0, bass: 0, lowMid: 0, mid: 0, highMid: 0, presence: 0, brilliance: 0 }
const EMPTY_BEAT = { isBeat: false, bpm: 0, confidence: 0, timeSinceLastBeat: 0, beatPhase: 0 }
const EMPTY_PAT = { isBeatDrop: false, isBuildUp: false, isSilence: true, intensity: 0, energyDelta: 0 }
const featuresRef = useRef<AudioFeatures>({
bass: 0, treble: 0, rms: 0, energy: 0, loudness: 0,
spectralCentroid: 0, spectralRolloff: 0, spectralFlatness: 0, zcr: 0,
drumIntensity: 0, beatDetected: false,
frequencyBands: EMPTY_BANDS,
beatInfo: EMPTY_BEAT,
patternInfo: EMPTY_PAT,
freqData: EMPTY_FREQ,
updatedAt: 0,
})
```
---
## Step 4 — Canvas files (backward-compatible; selected Muon enhancements already landed)
`MuonVisualizerCanvas` and `FluidVisualizerCanvas` read: `f.bass`, `f.treble`, `f.rms`,
`f.drumIntensity`, `f.beatDetected`. All five are backward-compatible remapped fields in the
expanded interface — no changes needed in either canvas file.
Current visualizer integration is backward-compatible and already consumes compatibility fields.
Additionally, targeted Muon enhancements now exist in canvas logic (sub-band weighting,
`beatPhase` shaping, and drop-trigger accenting), so these are no longer purely “future”.
Still optional for later parity work:
- tighter phase-locked geometry states derived from `beatInfo.beatPhase`
- richer transition choreography keyed on `patternInfo` sequences (build-up → drop)
- broader usage of normalized per-band vectors across particle subsystems
---
## Step 5 — Remove Meyda (cleanup after verification)
```bash
grep -r "meyda" frontend/src/ --include="*.ts" --include="*.tsx"
```
If the only remaining reference was the deleted dynamic import, run:
```bash
cd frontend && npm uninstall meyda
```
Also remove stale type-only leftovers if unused:
- `frontend/src/types/meyda.d.ts`
Keep it only if another package still imports Meyda types.
---
## Implementation order
1. Write `frontend/src/audio/MuonFeatureExtractor.ts`
2. Add `analysisAudioRef` to `UseAudioFeaturesArgs`; update call-site in `LyricsLivePreview.tsx`
3. Expand `AudioFeatures` interface and initial ref value
4. Add hidden always-mounted full-mix analysis `<audio>` in `LyricsLivePreview.tsx`
5. Implement analysis sync strategy (play/pause/seek/time drift correction)
6. Replace tap logic with single-source analysis routing (`mediaSource → analyser` only)
7. Add `bandPeaks` / `PEAK_DECAY` normalization state
8. Replace Meyda import + `tick()` with `extractor.start()` + `onFrame` callback
9. Update `deriveDrumIntensity` signature
10. Add/propagate `analysisAudioUrl` (`KaraokeTV`/`Karaoke` → `KaraokePerformanceMode` → `LyricsLivePreview`)
11. Ensure `analysisSourceKey` tracks `analysisAudioUrl ?? audioUrl ?? null`
12. Trim `stemGainRefs` from hook args if no longer needed
13. Run `tsc --noEmit`
14. Verify SEK seeking over `/karaoke-sek/packages/{id}/asset` still returns proper `206` on range probes
15. Remove Meyda if clean
16. Update `SUMMARIES (Claude).md`
---
## Risks and mitigations
| Risk | Mitigation |
|---|---|
| Audio doubling in multi-stem mode | Analysis element is never connected to `ctx.destination` (`mediaSource → analyser` only). |
| `createMediaElementSource` throws if audio element already owned | Wrap in try/catch; bail early. The `mediaSourceRef` guard (`??=`) prevents double-creation across effect re-runs. |
| Analysis and audible transport drift apart | Mirror `playbackRate`, sync on play/seek, and snap when drift exceeds ~40 ms. |
| Analysis source missing in stem sessions (`audioUrl` undefined) | Use `analysisAudioUrl ?? audioUrl`; derive `analysisAudioUrl` in page-level containers for both project and SEK modes. |
| Band normalization near-zero for first frames | Peaks start at 0.001; clamp01 never goes negative. Self-calibrates within ~2 seconds of loud audio. |
| `freqData` type change (`Uint8Array` → `Float32Array`) breaks a consumer | Audit: `grep -r "freqData" frontend/src/`. Values are now dB floats (-100 to 0) rather than 0–255 integers — update any display code. |
| Extractor RAF loop burns CPU while paused | `onFrame` early-returns if `!isPlayingRef.current`, zeroing reactive fields. The extractor RAF still ticks but the callback is almost free when returning immediately. Alternatively call `extractor.stop()` on pause and `extractor.start()` on resume. |
| `analysisAudioRef` is null on first render | `if (!audioEl) return` guard in useEffect; hidden element mounts when `analysisAudioUrl || audioUrl` becomes available. |
| Hidden analysis playback rejected (`play()` Promise reject) | Keep transport state authoritative; catch and ignore reject, retry on explicit user gesture/next resume edge. |
| SEK analysis seek desync on scrub | Ensure backend supports byte ranges for SEK assets; without `206` support, browser seek and `currentTime` sync are unreliable. |
| Cross-origin media blocks analysis | Keep `crossOrigin="anonymous"`; ensure source routes deliver compatible CORS/auth behavior (proxy-backed routes preferred). |
@@ -0,0 +1,205 @@
SEK Asset Endpoint — HTTP Range Request Support
Date: 2026-04-13
Scope: Fix seeking in the SEK file karaoke player (`/karaoke`, port 3001). Stems and source
media served from `/api/karaoke-sek/packages/{id}/asset` do not support HTTP byte-range
requests, so `audio.currentTime = t` silently fails in the browser.
---
## Root cause
`GET /api/karaoke-sek/packages/{package_id}/asset` in
`backend/routers/karaoke_sek.py` (lines 70–87):
```python
def get_asset(...):
data, mime, filename = get_sek_asset(package_id, path, folder)
return Response(
content=data, # entire file, always
media_type=mime,
headers={
"Content-Disposition": ...,
"Cache-Control": "public, max-age=3600",
},
)
```
The problems:
1. **No `Accept-Ranges: bytes` header** — browsers and media stacks may still probe with
`Range`, but seek behavior is less reliable/non-standard without explicit range signaling.
2. **No `Range` header parsing** — even if a browser sends `Range: bytes=X-Y`, the server
ignores it and returns the full file with `200 OK`, which the browser interprets as "range
requests are not supported."
When `audio.currentTime = t` is set on an audio element backed by a non-seekable URL:
- The browser may reset `currentTime` to 0 silently, or
- The seek appears to succeed in the UI (the scrubber moves) but playback resumes from the
old position once buffering continues.
Non-SEK project files served via `/api/jobs/{id}/download/{filename}` (FastAPI `FileResponse`)
do support range requests automatically — so seeking works there. Only SEK assets are broken.
---
## Fix — add range request support to the asset endpoint
### File: `backend/routers/karaoke_sek.py`
Change the `get_asset` route to:
1. Always add `Accept-Ranges: bytes` and `Content-Length` to the response.
2. Parse the `Range: bytes=start-end` header when present.
3. Respond with `206 Partial Content` for valid range requests.
4. Respond with `416 Range Not Satisfiable` for invalid ranges.
```python
from fastapi import APIRouter, HTTPException, Query, Request
from fastapi.responses import Response
@router.get("/packages/{package_id}/asset")
def get_asset(
request: Request,
package_id: str,
path: str = Query(..., description="Inner package path, e.g. stems/vocals.mp3"),
folder: str | None = Query(default=None, description="Optional SEK folder override"),
):
try:
data, mime, filename = get_sek_asset(package_id, path, folder)
except ProjectOperationError as exc:
raise HTTPException(status_code=exc.status_code, detail=exc.detail) from exc
total = len(data)
base_headers = {
"Accept-Ranges": "bytes",
"Content-Disposition": _safe_content_disposition(filename),
"Cache-Control": "public, max-age=3600",
}
range_header = request.headers.get("range")
if range_header:
try:
units, ranges_str = range_header.split("=", 1)
if units.strip().lower() != "bytes":
raise ValueError("unsupported range unit")
start_str, _, end_str = ranges_str.partition("-")
if not start_str.strip():
# bytes=-N (suffix)
suffix_len = int(end_str)
if suffix_len <= 0:
raise ValueError("invalid suffix")
start = max(total - suffix_len, 0)
end = total - 1
else:
start = int(start_str)
end = int(end_str) if end_str.strip() else total - 1
end = min(end, total - 1)
if start > end or start >= total:
return Response(
status_code=416,
headers={"Content-Range": f"bytes */{total}", **base_headers},
)
chunk = data[start : end + 1]
return Response(
content=chunk,
status_code=206,
media_type=mime,
headers={
"Content-Range": f"bytes {start}-{end}/{total}",
"Content-Length": str(len(chunk)),
**base_headers,
},
)
except (ValueError, AttributeError):
pass # malformed Range header — fall through to full response
return Response(
content=data,
media_type=mime,
headers={
"Content-Length": str(total),
**base_headers,
},
)
```
### Changes from original
| Before | After |
|---|---|
| `get_asset(package_id, path, folder)` params only | Add `request: Request` param |
| No range parsing | Parse `Range: bytes=X-Y` header |
| Always `200 OK` with full body | `206 Partial Content` for range requests; `200` for full |
| No `Accept-Ranges` header | `Accept-Ranges: bytes` always present |
| No `Content-Length` header | `Content-Length` on all responses |
| No `416` path | Return `416` for out-of-bounds or unsatisfiable ranges |
---
## Why reads-to-bytes is acceptable here
SEK packages are karaoke bundles designed to be portable and self-contained. Individual stem
files inside them are typically 5–30 MB compressed audio. Reading the whole member into memory
before slicing is acceptable for this use case — the files are served from a local NAS, not a
CDN.
A streaming solution (extracting to temp file + `FileResponse`) would handle very large files
more efficiently but adds temp-file lifecycle complexity. The slice approach is simpler and
correct for typical SEK sizes.
If a file is unusually large (e.g., an uncompressed WAV stem > 200 MB), the full read still
happens before the 206 slice is sent, which is wasteful. A future optimization could use
`zipfile.ZipFile.open()` to get a streaming handle and seek within it — but that requires
threading (ZipFile stream seeks aren't async-compatible). Not needed now.
---
## Edge cases
| Case | Handling |
|---|---|
| `Range: bytes=0-` (open-ended, read to EOF) | `end_str` is empty → `end = total - 1`; correct |
| `Range: bytes=-500` (last 500 bytes) | Implemented as a true suffix range: returns `bytes total-500 ... total-1` (clamped when file is smaller than 500 bytes). |
| `Range` header present but malformed (e.g., `Range: bytes`) | `ValueError` in parse → falls through to full `200` response |
| `Range: bytes=999999-0` (start > end) | Caught by `start > end` check → `416` |
| `Range` start ≥ `total` | Caught by `start >= total` check → `416` |
| Multiple ranges (`Range: bytes=0-100, 200-300`) | Only the first range set is considered; multipart/byteranges responses are not implemented (acceptable for browser media seek path, which uses single ranges). |
---
## Testing
```bash
# Full response — expect 200 + Accept-Ranges: bytes header
curl -I "http://work:3001/api/karaoke-sek/packages/<id>/asset?path=stems/vocals.mp3"
# Range request — expect 206 + Content-Range header
curl -I -H "Range: bytes=0-65535" \
"http://work:3001/api/karaoke-sek/packages/<id>/asset?path=stems/vocals.mp3"
# Browser test: open the SEK player, play a song, drag the seek bar midway through
# Playback should resume from the dragged position, not from the beginning
```
---
## Files changed
| File | Change |
|---|---|
| `backend/routers/karaoke_sek.py` | Add `Request` import; rewrite `get_asset` to parse `Range` header and return `206`/`416` as appropriate |
No changes to the frontend — the browser's native range-request mechanism handles everything
once the server responds correctly.
---
## Implementation order
1. Add `Request` to the FastAPI imports at the top of `karaoke_sek.py`
2. Replace the `get_asset` function body with the new implementation above
3. Run `curl -I` tests to confirm `Accept-Ranges: bytes` appears in response
4. Run `curl -H "Range: bytes=0-65535"` to confirm `206` response
5. Open the karaoke TV page with a SEK file; play and seek — confirm playback resumes from the
seeked position
6. Update `SUMMARIES (Claude).md`
@@ -0,0 +1,491 @@
# TrackDetail Visualizer Preview Plan
Date: 2026-04-13
Scope: Add a visualizer preview widget to SingUI's `/tracks/:trackId` detail page, letting the
user switch between Muon, Fluid, and Video (when a video URL exists) before launching a full
karaoke session.
---
## Current state of TrackDetail
`/home/translator/docker/appdata/stemengine/frontend/src/pages/TrackDetail.tsx`
The page already has:
- `song.id` → `subsonicStreamUrl(song.id)` for the full-mix audio stream
- `coverArtUrl` → `subsonicCoverArtUrl(song.coverArt, 800)` for palette
- `videoUrlInput` (state) → fetched from `getProjectKaraokeConfig(matchedProject.id)` for video
- `matchedProject` → project link (used to decide if Video tab is available)
- A 96×96 cover art `<img>` in the hero row that can be replaced by the preview widget
The page does **not** have: LyricsLivePreview, audio playback, or any visualizer.
---
## Design
### Layout change
Replace the static 96×96 cover art thumbnail with the `VisualizerPreviewWidget` component.
The widget is a self-contained panel:
```
┌─────────────────────────────────────────────┐
│ │ ← 16:9, max-w-xs (≈288×162px) on mobile
│ [Muon / Fluid / Video canvas] │ max-w-sm (≈384×216px) on desktop
│ │
├──────────┬──────────┬──────────┬────────────┤
│ ▶ / ⏸ │ Muon │ Fluid │ Video │ ← tab row (Video hidden when no URL)
└──────────┴──────────┴──────────┴────────────┘
```
The play/pause button starts paused. The visualizers run in **idle mode** (the existing
`sin(performance.now() * 0.0011)` idle sway) without audio. Pressing play streams the Subsonic
audio through the hook; the visualizers become audio-reactive.
The existing metadata row (title, artist, back link) sits beside the widget on desktop exactly
as it does beside the cover art today, just with the widget instead of the `<img>`.
---
## New component: `VisualizerPreviewWidget`
**File:** `frontend/src/components/VisualizerPreviewWidget.tsx`
### Props
```typescript
interface VisualizerPreviewWidgetProps {
audioUrl: string // subsonicStreamUrl(song.id)
coverArtUrl?: string // for palette extraction
videoUrl?: string // from karaoke config; if absent, Video tab is hidden
songTitle?: string // displayed as overlay when idle
className?: string
}
```
### Internal structure
```
VisualizerPreviewWidget
├── <audio ref={audioRef} src={audioUrl} crossOrigin="anonymous" preload="metadata" />
├── useVisualizerPalette({ coverArtUrl }) → paletteRef
├── useAudioFeatures({ audioCtxRef, analysisAudioRef: audioRef, isPlayingRef, analysisSourceKey }) → featuresRef
├── useState: mode: 'muon' | 'fluid' | 'video' (default: 'muon')
├── useState: isPlaying: boolean (default: false)
├── VisualizerBackdrop (when mode !== 'video')
│ mode={mode}
│ featuresRef={featuresRef}
│ paletteRef={paletteRef}
│ width={canvasW} height={canvasH}
├── <video> element (when mode === 'video')
└── tab row + play/pause overlay button
```
### Audio playback
The widget manages its own `<audio>` element. With the current Muon extractor integration,
`useAudioFeatures` already consumes `analysisAudioRef`. In this widget there are no stems, so
the same `<audio>` element is used for both playback and analysis (`analysisAudioRef: audioRef`).
No separate hidden analysis element is needed.
**Play/pause — with error handling:**
```typescript
const togglePlay = async () => {
const audio = audioRef.current
if (!audio) return
if (isPlaying) {
audio.pause()
setIsPlaying(false)
} else {
const ctx = ensureCtx()
if (ctx.state === 'suspended') await ctx.resume()
try {
await audio.play()
setIsPlaying(true)
} catch {
// Rejected by browser policy or network error; stay paused.
}
}
}
```
`audio.play()` returns a Promise that can reject (browser autoplay policy, network failure,
unsupported codec). `setIsPlaying(true)` must only be called after it resolves.
**`audio.ended` — registered in `useEffect` with cleanup:**
```typescript
useEffect(() => {
const audio = audioRef.current
if (!audio) return
const onEnded = () => setIsPlaying(false)
audio.addEventListener('ended', onEnded)
return () => audio.removeEventListener('ended', onEnded)
}, [])
```
Using a JSX `onEnded` prop is equivalent and preferred if the element is rendered in JSX.
**`audioUrl` change while playing — reset state:**
React Router may reuse the same `VisualizerPreviewWidget` instance when navigating from
`/tracks/1` to `/tracks/2`. The `<audio>` `src` updates to the new URL (browser pauses and
resets), but `isPlaying` stays `true`, leaving the UI in an inconsistent state.
```typescript
useEffect(() => {
setIsPlaying(false)
audioRef.current?.pause()
}, [audioUrl])
```
**AudioContext suspension on tab switch:**
Browsers suspend AudioContext when the page is hidden. On tab restore, audio plays again but
analysis stays silent because the context is suspended. Handle with:
```typescript
useEffect(() => {
const onVisible = () => {
if (isPlaying && audioCtxRef.current?.state === 'suspended') {
audioCtxRef.current.resume().catch(() => {})
}
}
document.addEventListener('visibilitychange', onVisible)
return () => document.removeEventListener('visibilitychange', onVisible)
}, [isPlaying])
```
### Video mode
In Video mode the canvas is hidden and a `<video>` element is shown instead.
Only direct media URLs (e.g. `.mp4`, `.webm`, `.m4v`, `.ogg`) should enable the Video tab.
Provider page URLs (e.g. standard YouTube watch links) are not valid `<video src>` resources.
**Two edge cases to avoid:**
1. **Double audio** — `muted={!isPlaying}` would unmute the `<video>` while `<audio>` is also
playing, producing two audio tracks. The `<video>` must always be `muted`; audio comes
exclusively from the `<audio>` element.
2. **`autoPlay={isPlaying}` is unreliable** — React's `autoPlay` only sets the HTML attribute at
mount time. It won't start or stop the video when `isPlaying` or `mode` changes after mount.
Use a `videoRef` and call `.play()`/`.pause()` imperatively.
```tsx
const videoRef = useRef<HTMLVideoElement>(null)
// Sync video play/pause when isPlaying or mode changes
useEffect(() => {
const video = videoRef.current
if (!video) return
if (mode === 'video' && isPlaying) {
video.play().catch(() => {})
} else {
video.pause()
}
}, [isPlaying, mode])
{mode === 'video' && videoUrl && (
<video
ref={videoRef}
src={videoUrl}
className="absolute inset-0 h-full w-full object-cover"
loop
muted // always muted — audio comes from the <audio> element
playsInline
/>
)}
```
Video and audio stay in sync passively through the play/pause button — both toggle together.
This is a preview so frame-perfect sync is not required.
### Tab row
Include ARIA attributes so the tab row is screen-reader accessible:
```tsx
const tabs = [
{ id: 'muon', label: 'Muon' },
{ id: 'fluid', label: 'Fluid' },
...(videoUrl ? [{ id: 'video', label: 'Video' }] : []),
] as const
<div className="flex border-t border-white/10" role="tablist" aria-label="Visualizer mode">
<button
onClick={togglePlay}
aria-label={isPlaying ? 'Pause' : 'Play'}
className="px-3 py-1.5 text-sm ..."
>
{isPlaying ? '⏸' : '▶'}
</button>
{tabs.map((tab) => (
<button
key={tab.id}
role="tab"
aria-selected={mode === tab.id}
onClick={() => setMode(tab.id)}
className={`flex-1 py-1.5 text-xs ... ${mode === tab.id ? 'active styles' : ''}`}
>
{tab.label}
</button>
))}
</div>
```
### Canvas sizing
`VisualizerBackdrop` needs explicit `width` / `height` in pixels. Use a `ResizeObserver` on the
container div, initialized in a `useLayoutEffect` so the first measurement happens before paint:
```typescript
const containerRef = useRef<HTMLDivElement>(null)
const [dims, setDims] = useState<{ w: number; h: number } | null>(null)
useLayoutEffect(() => {
const el = containerRef.current
if (!el) return
const measure = () => {
const { width } = el.getBoundingClientRect()
if (width > 0) setDims({ w: Math.round(width), h: Math.round(width * (9 / 16)) })
}
measure()
if (typeof ResizeObserver === 'undefined') {
window.addEventListener('resize', measure)
return () => window.removeEventListener('resize', measure)
}
const ro = new ResizeObserver(measure)
ro.observe(el)
return () => ro.disconnect()
}, [])
```
`VisualizerBackdrop` is only mounted once `dims` is non-null, avoiding a render with 0×0 that
would trigger an unnecessary canvas reset. While `dims` is null (first frame before
`useLayoutEffect` fires), show a placeholder so the container doesn't collapse:
```tsx
<div ref={containerRef} style={{ aspectRatio: '16/9' }} className="relative w-full bg-gray-900">
{dims ? (
<VisualizerBackdrop ... width={dims.w} height={dims.h} />
) : (
<div className="absolute inset-0 flex items-center justify-center">
<Disc3 className="h-8 w-8 text-gray-600 animate-spin-slow" />
</div>
)}
</div>
```
In practice `useLayoutEffect` fires synchronously before paint so the placeholder is never
visible. It guards against the zero-height flash that would occur without it.
---
## Canvas scaling — shared fix for `LyricsLivePreview` and `VisualizerPreviewWidget`
### Problem
Both `MuonVisualizerCanvas` and `FluidVisualizerCanvas` list `width` and `height` in their
`useEffect` dependency arrays:
```typescript
// current — in both canvas files
}, [featuresRef, paletteRef, width, height])
```
This means every window resize causes a **full canvas restart**: the RAF loop is cancelled and
rebuilt, particles are re-allocated (`buildParticles`), all state is reset. The result is a
visible flash on every resize.
This restart is unnecessary. Both canvases already self-adapt to their container each frame:
```typescript
// inside the RAF loop — both canvases already do this
const w = Math.max(1, Math.round(canvas.clientWidth || 0), Math.round(width || 0))
const h = Math.max(1, Math.round(canvas.clientHeight || 0), Math.round(height || 0))
if (canvas.width !== w || canvas.height !== h) {
canvas.width = w // updates pixel buffer on next frame
canvas.height = h
}
```
`canvas.clientWidth`/`canvas.clientHeight` reflect the CSS-rendered size, which already changes
when the parent container resizes. The `width`/`height` props are only needed as a non-zero
fallback if `clientWidth`/`clientHeight` haven't settled yet.
### Fix — remove `width` and `height` from canvas effect deps
**`MuonVisualizerCanvas.tsx`** and **`FluidVisualizerCanvas.tsx`**, change:
```typescript
// before
}, [featuresRef, paletteRef, width, height])
// after
}, [featuresRef, paletteRef])
// width and height are read from canvas.clientWidth/clientHeight each frame — no restart needed
```
This change applies to both canvas files and eliminates all resize-triggered restarts regardless
of whether the canvas is hosted inside `LyricsLivePreview` or `VisualizerPreviewWidget`.
### `LyricsLivePreview` — existing scaling mechanism
`LyricsLivePreview` already has a `ResizeObserver` on `containerRef` (line 2544) that updates
`visualizerSize` and passes it to `VisualizerBackdrop` as `width`/`height`. After removing these
from the canvas deps, `visualizerSize` only affects the prop value (used as the `clientWidth`
fallback) — it no longer triggers canvas restarts. The `ResizeObserver` in LyricsLivePreview can
remain as-is: its updates are now free (no DOM side-effects from the canvas side).
One remaining issue: `visualizerSize` is initialized as `{ width: 960, height: 540 }` (hardcoded),
which is replaced after the first `ResizeObserver` callback. After the dep array fix this no longer
causes a canvas restart, so the hardcoded initial value is harmless.
### Files changed by this fix
| File | Change |
|---|---|
| `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx` | Remove `width`, `height` from `useEffect` dep array |
| `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx` | Remove `width`, `height` from `useEffect` dep array |
---
## AudioContext ownership
`VisualizerPreviewWidget` owns its own `AudioContext` via a ref:
```typescript
const audioCtxRef = useRef<AudioContext | null>(null)
const ensureCtx = () => {
if (!audioCtxRef.current || audioCtxRef.current.state === 'closed') {
audioCtxRef.current = new AudioContext()
}
return audioCtxRef.current
}
```
Disposed in a cleanup `useEffect`:
```typescript
useEffect(() => () => { audioCtxRef.current?.close() }, [])
```
This is intentionally separate from the queue's AudioContext — the preview is an independent
player and must not share or interfere with the SingUI queue playback bridge.
---
## Integration into `TrackDetailPage`
### 1. Import
```typescript
import VisualizerPreviewWidget from '../components/VisualizerPreviewWidget'
```
No lazy load required — the component uses `VisualizerBackdrop` which is canvas-based, no
heavy dynamic imports.
### 2. Compute audio URL
```typescript
import { subsonicStreamUrl } from '../api/client'
const audioStreamUrl = useMemo(
() => (song ? subsonicStreamUrl(song.id) : ''),
[song?.id]
)
```
`TrackDetail.tsx` renders `null` early when `!song`, but `useMemo` runs unconditionally (hooks
cannot be conditional). Guard `song.id` access inside the memo with `song ?` to avoid a
runtime crash if the memo ever evaluates before the song has loaded.
Already available: `coverArtUrl`, `videoUrlInput` (the project video URL).
### 3. Replace the cover art block
Current (lines 157–163 of TrackDetail.tsx):
```tsx
<div className="h-24 w-24 shrink-0 overflow-hidden rounded-lg bg-gray-800 flex items-center justify-center">
{coverArtUrl ? (
<img src={coverArtUrl} alt={`${song.album} cover art`} className="h-full w-full object-cover" />
) : (
<Disc3 className="h-10 w-10 text-gray-500" />
)}
</div>
```
Replace with:
```tsx
<VisualizerPreviewWidget
audioUrl={audioStreamUrl}
coverArtUrl={coverArtUrl || undefined}
videoUrl={videoUrlInput.trim() || undefined}
songTitle={song.title}
className="w-48 shrink-0 rounded-lg overflow-hidden"
/>
```
Width `w-48` (192px) → height auto at 9:16 = 108px thumbnail. Adjust to taste.
---
## Relationship to the Muon Audio Analyzer plan
Current state (already implemented in codebase):
- `useAudioFeatures` uses `MuonFeatureExtractor` and accepts `analysisAudioRef`.
- In `VisualizerPreviewWidget` there are **no stems**, so playback and analysis share one
`<audio>` ref.
- Rich `frequencyBands`, `beatInfo`, and `patternInfo` fields are available for visual response.
The preview widget works correctly before and after the Muon analyzer upgrade, as the
backward-compatible `bass`, `treble`, `rms`, `drumIntensity`, `beatDetected` fields are preserved
through the migration.
---
## Files changed
| File | Change |
|---|---|
| `frontend/src/components/VisualizerPreviewWidget.tsx` | **New** |
| `frontend/src/pages/TrackDetail.tsx` | Replace cover art block; add `audioStreamUrl` memo |
| `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx` | Remove `width`, `height` from `useEffect` dep array |
| `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx` | Remove `width`, `height` from `useEffect` dep array |
| `SUMMARIES (Claude).md` | Append entry after implementation |
No changes required to `LyricsLivePreview` or `useAudioFeatures`.
---
## Implementation order
1. Remove `width`, `height` from `useEffect` deps in `MuonVisualizerCanvas.tsx` and `FluidVisualizerCanvas.tsx`
2. Write `VisualizerPreviewWidget.tsx` — container, `useLayoutEffect` canvas sizing, tab row, play/pause
3. Wire `VisualizerBackdrop` with real `paletteRef` from `useVisualizerPalette`
4. Add `AudioContext` management
5. Add `useAudioFeatures` with `audioCtxRef`, `analysisAudioRef`, and `isPlayingRef`
6. Add Video mode `<video>` element
7. Integrate into `TrackDetail.tsx`
8. Run `tsc --noEmit`
9. Update `SUMMARIES (Claude).md`
---
## Risks and mitigations
| Risk | Mitigation |
|---|---|
| Browser blocks autoplay on page load | Widget starts paused; play only on explicit user interaction. `AudioContext` created on first play click to satisfy policy. |
| Widget audio conflicts with SingUI queue playback | Register preview playback with the existing global audio registry (`registerPlayback`/`unregisterPlayback`) so starting preview pauses any currently active player first. |
| `videoUrlInput` may not be loaded when widget first renders | `videoUrlInput` starts as `''`; the Video tab simply won't appear until the config fetch completes (async `useEffect` in TrackDetailPage). No special handling needed. |
| `subsonicStreamUrl` requires auth or CORS headers | The URL goes through the backend proxy (`/api/...`), same as all other media requests. No special CORS handling needed. |
| `ResizeObserver` not available (old Safari) | Wrap in `typeof ResizeObserver !== 'undefined'` check; fall back to fixed 384×216. |
| Video `<video>` and `<audio>` both produce sound | `<video>` must always be `muted`; sound comes exclusively from the `<audio>` element. Using `muted={!isPlaying}` would produce double audio. |
| `autoPlay` prop doesn't start video after mount | React's `autoPlay` only sets the HTML attribute at mount. Use `videoRef` + `useEffect([isPlaying, mode])` calling `video.play()`/`video.pause()` imperatively. |
| `song.id` access before song is loaded | `useMemo` runs unconditionally before the `!song` guard. Guard with `song ? subsonicStreamUrl(song.id) : ''` inside the memo. |
| Container collapse before `dims` is measured | Show a `<Disc3>` placeholder when `dims === null`. In practice invisible (fires before paint), but prevents zero-height flash. |
@@ -0,0 +1,952 @@
# Visualizer Diagnosis and Fixes Plan
Date: 2026-04-13
Scope: Two-visualizer audit (Muon and Fluid) identifying root causes of the monocolor/idle-only
symptoms and the `spawnBlob` sizing bug. Fixes are documented with exact code changes.
---
## Symptom summary
- Muon visualizer: monocolor, "singular pattern of dots", no audio reactivity
- Fluid visualizer: sparse idle blobs, limited visual variety when multi-stem content plays
---
## Issue A — Analysis audio disconnected in multi-stem mode
### Root cause (shared by both visualizers)
`KaraokeTVPage` passes `audioUrl` to `KaraokePerformanceMode` → `LyricsLivePreview` only when
there are **no stems**:
```tsx
// KaraokeTV.tsx
audioUrl={safeStems.length === 0 && sourceMedia ? sourceMedia.url : undefined}
```
When stems are playing (always in SEK mode; also in project mode when a sep job has completed),
`audioUrl` is `undefined`. `LyricsLivePreview` gates the hidden analysis element on this prop:
```tsx
{/* Hidden full-mix analysis audio element (always mounted when source exists) */}
{audioUrl && (
<audio ref={analysisAudioRef} src={audioUrl} ... muted />
)}
```
With `audioUrl` absent, `analysisAudioRef.current` is `null`. `useAudioFeatures` returns early:
```typescript
const audioEl = analysisAudioRef.current
if (!ctx || !audioEl) return // ← always exits here
```
`featuresRef` stays at its initial zero state forever:
```typescript
{
frequencyBands: { sub: 0, bass: 0, lowMid: 0, mid: 0, highMid: 0, presence: 0, brilliance: 0 },
beatInfo: { isBeat: false, bpm: 0, beatPhase: 0, ... },
patternInfo: { isBeatDrop: false, isBuildUp: false, isSilence: true, intensity: 0, ... },
bass: 0, treble: 0, rms: 0, drumIntensity: 0, beatDetected: false,
}
```
Both canvases then run in pure idle mode — no beat response, no hue shift from audio, no drop
bursts. For Muon: slowly orbiting dots at the idle sway frequency. For Fluid: sparse blobs
driven only by the 4% baseline spawn rate.
### Fix — add `analysisAudioUrl` prop to `LyricsLivePreview` and `KaraokePerformanceMode`
#### 1. `LyricsLivePreview.tsx` — new prop
Add `analysisAudioUrl?: string` to `LyricsLivePreviewProps`. When present, it overrides
`audioUrl` as the source for the hidden analysis element:
```tsx
// Current
{audioUrl && (
<audio ref={analysisAudioRef} src={audioUrl} preload="auto" crossOrigin="anonymous" muted />
)}
// After
{(analysisAudioUrl || audioUrl) && (
<audio
ref={analysisAudioRef}
src={analysisAudioUrl ?? audioUrl}
preload="auto"
crossOrigin="anonymous"
muted
/>
)}
```
Also update the `analysisSourceKey` passed to `useAudioFeatures` to track whichever URL is
used, so the effect re-runs when the source changes:
```typescript
// Current
analysisSourceKey: audioUrl ?? null,
// After
analysisSourceKey: analysisAudioUrl ?? audioUrl ?? null,
```
#### 2. `KaraokePerformanceMode.tsx` — thread the prop through
Add `analysisAudioUrl?: string` to `KaraokePerformanceModeProps` and forward it:
```tsx
<LyricsLivePreview
...
analysisAudioUrl={analysisAudioUrl}
/>
```
#### 3. `KaraokeTV.tsx` — derive and pass the analysis URL
**Project mode** — the source file (original recording) is always available as `sourceMedia.url`
and is the ideal full-mix analysis source:
```tsx
const analysisAudioUrl = useMemo(() => {
if (isSekMode) {
// SEK: use first stem as proxy analysis source (no full-mix file in manifest)
const firstStem = sekManifest?.manifest?.contents?.stems?.[0]
const firstStemPath = firstStem ? normalizeSekInnerPath(firstStem.file) : ''
return firstStemPath && sekManifest
? sekAssetUrl(sekManifest.id, firstStemPath, sekFolder || undefined)
: undefined
}
// Project: use the source file (original recording)
if (!detail) return undefined
const sourceFile = detail.sep_job.files.find((f) => f.type === 'source')
return sourceFile ? downloadUrl(detail.sep_job.id, sourceFile.name) : undefined
}, [isSekMode, sekManifest, sekFolder, detail])
```
Then pass it:
```tsx
<KaraokePerformanceMode
...
analysisAudioUrl={analysisAudioUrl}
/>
```
**Notes on the SEK fallback:**
- SEK packages do not contain a `contents.audio` or `source` field; only `stems`, `lyrics`, and
optionally `video`.
- Using the first stem gives partial frequency data (one instrument's spectrum). It will be
better than zero but will not capture the full mix. A future improvement could mix all stem
`AnalyserNode` outputs, but that requires connecting all stems to a single analyser.
- The "leading" or "drums" stem (if present) is more representative than a random first stem.
The `normalizeLyricsTrackKey` logic already knows how to rank stems — a `pickAnalysisStem`
helper could use similar ranking. For now, first stem is acceptable.
---
## Issue B — Muon: all particles share one hue per frame
### Root cause
In `MuonVisualizerCanvas.tsx`, one `hue` is derived from the audio state each frame:
```typescript
const hue = monoColor
? p.seedHue
: drum > 0.01
? ((p.seedHue + drum * 20 * hueRange + treble * 15 * hueRange + ...) + 360) % 360
: ((p.seedHue + Math.sin(now * 0.00018) * 15 * hueRange) + 360) % 360
```
Every particle in every lane then receives the same fill:
```typescript
ctx.fillStyle = `hsla(${hue}, 92%, 64%, ${alpha})`
```
`hue_range` only controls how much the frame-level hue shifts with audio — it does not spread
colors across particles. The result is a single color at any instant regardless of how many
lanes are visible.
### Fix — per-lane hue offset proportional to `hue_range`
Replace the per-particle fill with a lane-offset hue. 8 lanes × 45° at `hue_range = 1.0`
produces a full 360° rainbow spread. At `hue_range = 0`, all lanes collapse to `seedHue`
(monocolor), making `mono_color = true` still useful for a color-locked look.
```typescript
// Before (inside particle loop)
ctx.fillStyle = `hsla(${hue}, 92%, 64%, ${alpha})`
// After
const laneHue = (hue + pt.lane * (hueRange * 45)) % 360
ctx.fillStyle = `hsla(${laneHue}, 92%, 64%, ${alpha})`
```
When `mono_color` is true, `hueRange` should be ignored so the lane offset is zero. Guard:
```typescript
const laneHue = monoColor
? hue
: (hue + pt.lane * (hueRange * 45)) % 360
ctx.fillStyle = `hsla(${laneHue}, 92%, 64%, ${alpha})`
```
The glow shadow color can also vary per lane for extra depth:
```typescript
ctx.shadowColor = monoColor
? p.highlightColor
: `hsla(${(laneHue + 30) % 360}, 100%, 80%, 1)`
```
Or leave `shadowColor = p.highlightColor` unchanged if the extra glow variation is distracting.
**Effect at defaults (`hue_range = 1.0`, 8 lanes):**
- Lane 0: seedHue + 0°
- Lane 1: seedHue + 45°
- Lane 2: seedHue + 90°
- ...
- Lane 7: seedHue + 315°
This creates concentric rainbow ellipses that react to audio together (shared frame-level hue
drift) but remain visually distinct from each other.
---
## Issue C — Fluid: `spawnBlob` uses captured props instead of live canvas dimensions
### Root cause
`spawnBlob` is a closure defined inside `useEffect`. It captures `width` and `height` from the
component props at the time the effect first runs:
```typescript
const spawnBlob = (
strength: number,
blobLifeScale: number,
splatForce: number,
origin?: { ... },
) => {
const w = Math.max(1, width) // ← captured prop, stale after resize
const h = Math.max(1, height)
...
x: origin?.x ?? (Math.random() * w),
y: origin?.y ?? (Math.random() * h),
```
The `render` function computes live `w`/`h` from `canvas.clientWidth`/`canvas.clientHeight`
each frame, but `spawnBlob` doesn't receive these. After a resize, beat-burst blobs and the
ambient `spawnBlob` calls spawn within the old canvas area, leaving a dead band on the
newly-revealed edges.
Ambient blobs spawned inline in the `render` body **do** use live `w`/`h` — the inconsistency
is only in `spawnBlob`.
### Fix — pass live `w`/`h` to `spawnBlob`
Add `w` and `h` parameters to `spawnBlob` and use them instead of the captured props:
```typescript
// Before
const spawnBlob = (
strength: number,
blobLifeScale: number,
splatForce: number,
origin?: { x?: number; y?: number; angle?: number; color?: string },
) => {
const w = Math.max(1, width)
const h = Math.max(1, height)
...
}
// After
const spawnBlob = (
strength: number,
blobLifeScale: number,
splatForce: number,
w: number,
h: number,
origin?: { x?: number; y?: number; angle?: number; color?: string },
) => {
// w and h received from caller — always live canvas dimensions
...
}
```
Update all call sites inside `render` to pass `w, h`:
```typescript
// Beat burst calls
for (let i = 0; i < count; i += 1) spawnBlob(0.6 + bass * 0.8, blobLife, splatForce, w, h)
// Phase-orbit call
spawnBlob(
0.28 + kick * 0.45 + phasePulse * 0.3 + dropBoost * 0.35,
blobLife,
splatForce,
w,
h,
{ x: px, y: py, angle: phaseAngle + Math.PI * 0.5, color: ... },
)
```
---
## Issue D — Fluid: `background_persistence` formula is inverted
### Root cause
The background fill alpha is computed as:
```typescript
ctx.globalAlpha = 0.20 + (1 - bass) * 0.16 * bgPersistence
```
When `bgPersistence` increases, alpha increases → background clears faster → **less**
persistence. But the parameter is named `background_persistence` which intuitively implies
higher = more persistent trails.
At defaults (`bgPersistence = 1.0`, `bass = 0`): alpha = 0.36.
At `bgPersistence = 3.0`: alpha = 0.68 — trails clear quickly.
At `bgPersistence = 0.0`: alpha = 0.20 — trails persist longest.
The PerformanceSidebar slider direction is also confusing: dragging right increases
"persistence" but visually reduces trail length.
### Fix — two options
**Option A (rename to `background_clearing`):**
Rename the parameter in `VisualizerSettings`, `VizSettings`, all defaults, and the sidebar
label to `background_clearing`. No formula change. Semantically correct: higher value → faster
clearing.
**Option B (invert the formula):**
Keep the name `background_persistence` but invert how the multiplier is applied:
```typescript
// Before
ctx.globalAlpha = 0.20 + (1 - bass) * 0.16 * bgPersistence
// After — higher bgPersistence → lower alpha → more persistent trails
ctx.globalAlpha = 0.20 + (1 - bass) * 0.16 / Math.max(0.1, bgPersistence)
```
At `bgPersistence = 1.0`: alpha = 0.36 (same default behavior).
At `bgPersistence = 3.0`: alpha = `0.20 + 0.053` = 0.25 (long trails).
At `bgPersistence = 0.1`: alpha = `0.20 + 1.6` → clamped to 1.0 by `globalAlpha` (rapid clear).
Option B is less disruptive (no rename ripple) and makes the slider direction intuitive.
**Recommended: Option B.**
---
## Fluid visualizer — what is NOT broken
- **Color variety** — `speakerPalette` always has 8 colors: from the SEK/project manifest's
`dynamic_palette` speaker entries when available, otherwise from the cover art seed hue,
otherwise the DEFAULT_PALETTE 8-color array. Blobs cycle through these via `colorOffset`,
so color variety is present in all modes.
- **Idle baseline** — the 4% ambient spawn rate and `idle = 0.5 + 0.5 * Math.sin(...)` ensure
the canvas is never completely empty, even with zero audio features.
---
## Issue E — Fluid: wrong rendering backend (canvas blobs, not WebGL fluid sim)
### Root cause — the original plan was never implemented
`docs/2026-04-08_Visualizer_Integration_Plan.md` (Phase 3) and `SUMMARIES (Claude).md` both
specify the Fluid visualizer as a wrapper around **`PavelDoGreat/WebGL-Fluid-Simulation`** —
a WebGL shader-based fluid dynamics simulation with pressure solving, velocity advection, and
vorticity. The current `FluidVisualizerCanvas.tsx` is a Canvas 2D blob renderer written as a
placeholder. The SUMMARIES file explicitly notes: *"Most Fluid WebGL params (bloom, sunrays,
pressure, shading) are not applicable to the current canvas blob renderer."*
### Why this causes single-mode confetti
Canvas 2D radial gradients with simple Euler-integration velocity are physically limited.
Regardless of how the parameters (spawn density, drag, blob life, etc.) are tuned, the result
is always the same visual category: small colored discs flying outward and fading. The presets
(Ink, Foam, Lava, Mist, Storm) were designed with WebGL fluid parameters in mind:
| Preset | Intended WebGL behavior | What canvas blobs produce |
|---|---|---|
| **Ink** | Low dissipation, high curl → thick, slow-moving ribbons of color | Many medium blobs, same physics |
| **Foam** | High dissipation, tiny splats, many per beat → bubbly texture | Smaller, faster-fading blobs |
| **Lava** | Very low dissipation, high viscosity → slow swelling masses | Large slow blobs |
| **Mist** | High velocity dissipation, near-invisible forces → translucent wisps | Faint fast blobs |
| **Storm** | High curl (60+), max force → violent rotational turbulence | Dense fast blobs |
All five look like confetti variants because the canvas renderer doesn't have:
- **Velocity field advection** — fluid carries its own motion; canvas blobs travel independently
- **Vorticity / curl** — the fluid swirls; blobs just drift straight
- **Pressure solving** — fluid fills space and pushes back; blobs don't interact
### Fix — replace `FluidVisualizerCanvas.tsx` with a WebGL fluid sim wrapper
#### Step 1 — Acquire the sim
Implementation note (this repository): **Option B was used** with one additional runtime-safety
step: the package's `dist/webgl-fluid.mjs` is vendored into
`frontend/src/components/visualizers/fluid/webgl-fluid.mjs`, and `fluidSim.ts` dynamically
imports that local file. This avoids environment-specific Vite package-resolution failures while
still using the package runtime.
**Option A (recommended): vendor `script.js` directly**
Copy `PavelDoGreat/WebGL-Fluid-Simulation/script.js` into
`frontend/src/components/visualizers/fluid/fluidSim.js`. Strip the 4 event handler blocks
(`canvas.addEventListener('mousedown'|'mousemove'|'touchstart'|'touchmove')`). Expose two
module exports at the bottom:
```js
// Add at the bottom of fluidSim.js
export function initFluidSim(canvas, initialConfig) { /* wraps existing canvas init */ }
export function splatFluid(x, y, dx, dy, color) { /* wraps existing splat() */ }
export function updateFluidConfig(patch) { Object.assign(config, patch) }
```
No npm dependency needed. The sim is ~1000 lines of self-contained WebGL — no build step,
no transitive deps. Its WebGL context attaches directly to the `<canvas>` element passed in.
**Option B: npm package**
```bash
npm install webgl-fluid
```
Exposes `createFluid(canvas)` → `{ splat, config }`. Simpler but the package is a community
wrapper and may lag behind upstream bug fixes.
**Recommended: Option A.** It gives direct control over config fields and is MIT-licensed.
#### Step 2 — Rewrite `FluidVisualizerCanvas.tsx`
```tsx
import { useEffect, useRef } from 'react'
import type { MutableRefObject } from 'react'
import type { AudioFeatures } from '../../hooks/useAudioFeatures'
import type { VisualizerPalette } from '../../hooks/useVisualizerPalette'
import type { VisualizerSettings } from './VisualizerBackdrop'
import { initFluidSim, splatFluid, updateFluidConfig } from './fluid/fluidSim'
type FluidSettings = NonNullable<VisualizerSettings['fluid']>
export default function FluidVisualizerCanvas({
featuresRef, paletteRef, width, height, settings, fpsCap,
}: FluidVisualizerCanvasProps) {
const canvasRef = useRef<HTMLCanvasElement>(null)
const settingsRef = useRef(settings)
settingsRef.current = settings
useEffect(() => {
const canvas = canvasRef.current
if (!canvas) return
// Initialise fluid sim with defaults derived from initial settings
const s = settingsRef.current
initFluidSim(canvas, {
SIM_RESOLUTION: 128,
DYE_RESOLUTION: 1024,
DENSITY_DISSIPATION: fluidDensityDissipation(s),
VELOCITY_DISSIPATION: fluidVelocityDissipation(s),
PRESSURE: 0.8,
CURL: fluidCurl(s),
SPLAT_RADIUS: fluidSplatRadius(s),
COLORFUL: false, // palette-locked; we supply colors from paletteRef
BLOOM: true,
BLOOM_INTENSITY: 0.8,
SUNRAYS: false,
BACK_COLOR: { r: 0, g: 0, b: 0 },
})
let colorOffset = 0
const nextColor = () => {
const palette = paletteRef.current.speakerPalette
if (!palette.length) return hexToRgb(paletteRef.current.highlightColor)
colorOffset = (colorOffset + Math.max(0, settingsRef.current?.color_speed ?? 1) * 0.5) % palette.length
return hexToRgb(palette[Math.floor(colorOffset)])
}
let prevBeatPhase = 0
let dropBoost = 0
let lastDropAt = 0
let lastFrameAt = 0
let raf = 0
const loop = (ts: number) => {
raf = requestAnimationFrame(loop)
const minFrameMs = 1000 / Math.max(1, fpsCap ?? 60)
if (ts - lastFrameAt < minFrameMs - 1) return
lastFrameAt = ts
const f = featuresRef.current
const s = settingsRef.current
const bass = clamp01(f.bass)
const sub = clamp01(f.frequencyBands.sub)
const rms = clamp01(f.rms)
const treble = clamp01(f.treble)
const beatPhase = clamp01(f.beatInfo.beatPhase)
const kick = clamp01(sub * 0.72 + bass * 0.28)
const phasePulse = Math.pow(Math.max(0, Math.sin(beatPhase * Math.PI)), 1.8)
const phaseWrapped = beatPhase < prevBeatPhase - 0.35
const idle = 0.5 + 0.5 * Math.sin(ts * 0.0011)
const dropTriggered = f.patternInfo.isBeatDrop && ts - lastDropAt > 500
if (dropTriggered) { dropBoost = 1; lastDropAt = ts }
dropBoost *= 0.96
// Live config modulation — update fluid sim parameters each frame
updateFluidConfig({
DENSITY_DISSIPATION: fluidDensityDissipation(s, kick),
VELOCITY_DISSIPATION: fluidVelocityDissipation(s, rms),
CURL: fluidCurl(s, treble, dropBoost),
})
const densityScale = Math.max(0.2, s?.spawn_density ?? 1)
const splatForce = Math.max(0.2, s?.splat_force ?? 1)
const beatBurstCount = Math.max(0, s?.beat_burst_count ?? 1)
// Ambient splat — continuous low-level motion proportional to rms
const ambientRate = (0.04 + rms * 0.35 + phasePulse * 0.08 + idle * 0.04) * densityScale
if (Math.random() < ambientRate) {
const angle = Math.random() * Math.PI * 2
const force = (0.2 + rms * 0.8 + idle * 0.12) * splatForce * 400
splatFluid(
Math.random(), Math.random(),
Math.cos(angle) * force,
Math.sin(angle) * force,
nextColor(),
)
}
// Beat burst
if (f.beatDetected || phaseWrapped || dropTriggered) {
const count = Math.max(1, Math.round((3 + kick * 5 + dropBoost * 4) * beatBurstCount))
for (let i = 0; i < count; i++) {
const angle = Math.random() * Math.PI * 2
const force = (0.6 + bass * 1.2 + dropBoost * 0.6) * splatForce * 800
splatFluid(
Math.random(), Math.random(),
Math.cos(angle) * force, Math.sin(angle) * force,
dropTriggered ? hexToRgb(paletteRef.current.highlightColor) : nextColor(),
)
}
}
// Beat-phase orbit — sweep splat source around center following beat phase
if (beatPhase > 0.001 || f.beatInfo.bpm > 0) {
const phaseAngle = beatPhase * Math.PI * 2
if (Math.random() < 0.08 + phasePulse * 0.22 + dropBoost * 0.18) {
const force = (0.28 + kick * 0.5 + phasePulse * 0.4) * splatForce * 500
splatFluid(
0.5 + Math.cos(phaseAngle) * 0.28,
0.5 + Math.sin(phaseAngle) * 0.20,
Math.cos(phaseAngle + Math.PI / 2) * force,
Math.sin(phaseAngle + Math.PI / 2) * force,
dropBoost > 0.18 ? hexToRgb(paletteRef.current.highlightColor) : nextColor(),
)
}
}
prevBeatPhase = beatPhase
}
raf = requestAnimationFrame(loop)
return () => cancelAnimationFrame(raf)
}, [featuresRef, paletteRef])
return <canvas ref={canvasRef} className="h-full w-full" aria-hidden="true" />
}
```
#### Step 3 — Config-mapping helpers
These translate `VizSettings['fluid']` into WebGL sim parameters. The naming is now accurate:
```typescript
// All three helpers accept optional live audio values for per-frame modulation.
function fluidDensityDissipation(s?: FluidSettings, kick = 0): number {
// background_persistence: higher → more persistent (lower dissipation)
// kick: loud kick → longer trails (lower dissipation)
const base = 0.98 - (s?.background_persistence ?? 1) * 0.025
return clamp(base - kick * 0.04 * (s?.decay ?? 1), 0.85, 0.999)
}
function fluidVelocityDissipation(s?: FluidSettings, rms = 0): number {
// drag: higher → slower fluid (higher dissipation)
const base = 0.96 - (1 / Math.max(0.5, s?.drag ?? 1)) * 0.06
return clamp(base - rms * 0.04, 0.80, 0.999)
}
function fluidCurl(s?: FluidSettings, treble = 0, dropBoost = 0): number {
// splat_force mapping: higher force → more turbulence → higher curl
const base = 15 + (s?.splat_force ?? 1) * 12
return clamp(base + treble * 8 + dropBoost * 20, 0, 80)
}
function fluidSplatRadius(s?: FluidSettings): number {
// blob_life as splat radius proxy: larger life → larger radius
return clamp(0.18 + (s?.blob_life ?? 1) * 0.06, 0.05, 0.5)
}
function hexToRgb(hex: string): { r: number; g: number; b: number } {
const m = /^#?([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})$/i.exec(hex)
if (!m) return { r: 0.5, g: 0.5, b: 0.8 }
return { r: parseInt(m[1], 16) / 255, g: parseInt(m[2], 16) / 255, b: parseInt(m[3], 16) / 255 }
}
```
#### Step 4 — Update `FLUID_PRESETS` in `PerformanceSidebar.tsx`
The preset values need to be re-tuned so the mapping functions produce meaningfully different
WebGL configurations:
| Preset | `background_persistence` | `splat_force` | `drag` | `decay` | `blob_life` | `spawn_density` | `beat_burst_count` | `treble_pulse` | `color_speed` |
|---|---|---|---|---|---|---|---|---|---|
| Default | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 |
| Ink | 3.0 | 0.5 | 2.5 | 0.3 | 2.5 | 0.5 | 1.5 | 0.3 | 0.2 |
| Foam | 0.2 | 0.3 | 0.4 | 3.0 | 0.3 | 3.0 | 3.0 | 2.5 | 3.0 |
| Lava | 3.5 | 2.0 | 3.0 | 0.2 | 3.0 | 0.3 | 0.5 | 0.2 | 0.2 |
| Mist | 0.5 | 0.2 | 0.5 | 2.0 | 0.5 | 1.5 | 0.5 | 1.5 | 0.8 |
| Storm | 1.0 | 3.0 | 0.4 | 1.0 | 1.0 | 2.5 | 3.0 | 2.5 | 2.5 |
**Resulting WebGL parameters per preset:**
| Preset | `DENSITY_DISSIPATION` | `VELOCITY_DISSIPATION` | `CURL` | `SPLAT_RADIUS` | Visual character |
|---|---|---|---|---|---|
| Default | ~0.955 | ~0.96 | ~27 | ~0.24 | Balanced color motion |
| Ink | ~0.905 | ~0.916 | ~21 | ~0.33 | Thick slow ribbons, long trails |
| Foam | ~0.975 | ~0.99 | ~18.6 | ~0.18 | Fast-fading tiny bubbles, frantic beat |
| Lava | ~0.893 | ~0.94 | ~39 | ~0.36 | Slow massive blobs, high vortex |
| Mist | ~0.968 | ~0.98 | ~20.6 | ~0.21 | Translucent wisps, gentle movement |
| Storm | ~0.955 | ~0.99 | ~51 | ~0.24 | Violent turbulence on every beat |
#### Step 5 — Update `VisualizerSettings` interface
The canvas-blob-specific fields can be retained (they now map to WebGL params via the helper
functions) or replaced with more semantically accurate WebGL param names. Retain the existing
field names to avoid breaking SEK manifest compatibility. No schema change needed.
Remove the Note from `VisualizerBackdrop.tsx` that says WebGL params "are not applicable to
the canvas blob renderer" — they now are applicable.
---
## Issue E — note on Issues C and D supersession
Issues C (`spawnBlob` dimensions) and D (`background_persistence` formula) exist only in the
canvas blob renderer. After Issue E is implemented, `FluidVisualizerCanvas.tsx` is fully
rewritten. Issues C and D are automatically resolved by the replacement and do not need to be
fixed in the canvas blob code.
---
## Edge cases and additional fixes
### A1 — `Karaoke.tsx` has the same analysis audio gap
The plan's Issue A fix scopes to `KaraokeTV.tsx`. `Karaoke.tsx` (the operator-facing page at
`/karaoke`) uses **identical logic** (lines 463–464):
```tsx
// Karaoke.tsx — exact same conditional as KaraokeTV.tsx
stems={safeStems.length > 0 ? safeStems : undefined}
audioUrl={safeStems.length === 0 && sourceMedia ? sourceMedia.url : undefined}
```
The `analysisAudioUrl` memo and prop must be added to `Karaoke.tsx` as well. The derivation
logic is identical to `KaraokeTV.tsx` (source file for project, first stem for SEK).
Add `Karaoke.tsx` to the files-changed table.
---
### E1 — WebGL context failure: need fallback
`canvas.getContext('webgl2') ?? canvas.getContext('webgl')` returns `null` on devices that
don't support WebGL (old Android WebViews, some iframe sandboxes). The vendored sim will crash
silently or throw.
**Fix:** Detect WebGL support before calling `initFluidSim`. On failure, fall back to the
canvas blob renderer (keep the old blob logic in a separate file or inline) or show a static
dark background:
```typescript
useEffect(() => {
const canvas = canvasRef.current
if (!canvas) return
const supportsWebGL = !!(
canvas.getContext('webgl2') ?? canvas.getContext('webgl')
)
if (!supportsWebGL) {
// Degrade gracefully: render static palette background
const ctx2d = canvas.getContext('2d')
if (ctx2d) {
ctx2d.fillStyle = paletteRef.current.backgroundColor
ctx2d.fillRect(0, 0, canvas.width, canvas.height)
}
return
}
initFluidSim(canvas, { ... })
...
}, [featuresRef, paletteRef])
```
The context test creates a throwaway context; `initFluidSim` will create the real one. WebGL
contexts are still a limited browser resource, so production code should avoid repeated
probe/create cycles and should always release contexts on cleanup (`WEBGL_lose_context`).
---
### E2 — WebGL context not freed on cleanup (context limit)
The cleanup function currently only cancels the outer RAF:
```typescript
return () => cancelAnimationFrame(raf)
```
The WebGL context persists until garbage collection. Browsers enforce a hard per-page limit
(typically 8–16 contexts). Navigating between songs while the fluid visualizer is mounted will
exhaust this limit, causing subsequent `getContext('webgl')` calls to return `null` and the sim
to silently stop rendering.
**Fix:** Export a `destroyFluidSim()` from `fluidSim.js` that:
1. Cancels the sim's internal RAF loop
2. Calls `gl.getExtension('WEBGL_lose_context')?.loseContext()`
3. Nulls internal state
Call it in cleanup:
```typescript
return () => {
cancelAnimationFrame(raf)
destroyFluidSim()
}
```
This must be added during the vendoring step (Step 1) along with the other exported functions.
---
### E3 — `fluidSim.js` is not an ES module; TypeScript has no types
`PavelDoGreat/WebGL-Fluid-Simulation/script.js` is not an ES module — it has no `export`
statements and is structured as top-level imperative code. Vite cannot `import` from it
directly; it will throw `SyntaxError: The requested module does not provide an export`.
**Fix during vendoring:** Wrap the sim in a factory function and export via ES module syntax:
```js
// fluidSim.js — top of file
let _splat, _config, _internalRaf
// ... (existing sim code, modified so `canvas`, `gl`, `config`, `splat` are
// assigned to module-level variables instead of implicit globals)
export function initFluidSim(canvas, overrides) {
// call existing canvas/gl initialisation with the provided canvas
// merge overrides into config
}
export function splatFluid(x, y, dx, dy, color) { _splat(x, y, dx, dy, color) }
export function updateFluidConfig(patch) { Object.assign(_config, patch) }
export function destroyFluidSim() {
cancelAnimationFrame(_internalRaf)
const ext = _gl?.getExtension('WEBGL_lose_context')
ext?.loseContext()
_gl = null
}
```
Add `frontend/src/components/visualizers/fluid/fluidSim.d.ts`:
```typescript
export interface FluidColor { r: number; g: number; b: number }
export interface FluidConfig {
SIM_RESOLUTION?: number; DYE_RESOLUTION?: number
DENSITY_DISSIPATION?: number; VELOCITY_DISSIPATION?: number
PRESSURE?: number; CURL?: number; SPLAT_RADIUS?: number
COLORFUL?: boolean; BLOOM?: boolean; BLOOM_INTENSITY?: number
SUNRAYS?: boolean; BACK_COLOR?: FluidColor
}
export function initFluidSim(canvas: HTMLCanvasElement, config?: FluidConfig): void
export function splatFluid(x: number, y: number, dx: number, dy: number, color: FluidColor): void
export function updateFluidConfig(patch: Partial<FluidConfig>): void
export function destroyFluidSim(): void
```
---
### E4 — `clamp` helper not defined in config-mapping helpers
The config helpers (`fluidDensityDissipation`, `fluidVelocityDissipation`, `fluidCurl`,
`fluidSplatRadius`) all call `clamp(value, min, max)` but this function is not defined in the
plan's code or in the existing canvas files (which use `clamp01`).
**Fix:** Define at module level in `FluidVisualizerCanvas.tsx`:
```typescript
function clamp(v: number, lo: number, hi: number): number {
return Math.max(lo, Math.min(hi, v))
}
```
---
### E5 — `fpsCap` captured in closure; settings not live-updated
The plan's `FluidVisualizerCanvas` reads `fpsCap ?? 60` directly from the closure:
```typescript
const minFrameMs = 1000 / Math.max(1, fpsCap ?? 60)
```
Since `fpsCap` is a prop and not in the `useEffect` dep array, changes to it after mount are
ignored (the effect doesn't re-run). This is inconsistent with how both `MuonVisualizerCanvas`
and the old `FluidVisualizerCanvas` handle it.
**Fix:** Use the same stable ref pattern as the Muon canvas:
```typescript
const fpsCapRef = useRef(fpsCap ?? 60)
fpsCapRef.current = fpsCap ?? 60 // updated at render level, before effect reads it
// Inside the RAF loop:
const minFrameMs = 1000 / Math.max(1, fpsCapRef.current)
```
---
### E6 — `BACK_COLOR` hardcoded to black; doesn't track palette
The plan initialises the sim with `BACK_COLOR: { r: 0, g: 0, b: 0 }` (black), but
`paletteRef.current.backgroundColor` may be a dark blue or other palette-derived color. The
fluid sim's background won't match the surrounding UI.
Additionally, `paletteRef.current` can change after mount (cover art loads asynchronously). The
`BACK_COLOR` won't update until the next config call.
**Fix:** Add `BACK_COLOR` to the per-frame `updateFluidConfig` call, reading from `paletteRef`:
```typescript
updateFluidConfig({
DENSITY_DISSIPATION: fluidDensityDissipation(s, kick),
VELOCITY_DISSIPATION: fluidVelocityDissipation(s, rms),
CURL: fluidCurl(s, treble, dropBoost),
BACK_COLOR: hexToRgb(paletteRef.current.backgroundColor),
})
```
---
### E7 — `treble_pulse` not used in WebGL path
The canvas blob renderer applied `treble_pulse` as a radius-swell multiplier. The WebGL
wrapper's per-frame loop doesn't read `settingsRef.current?.treble_pulse` anywhere, so the
slider has no effect in Fluid mode.
**Fix:** Map `treble_pulse` to ambient splat force intensity. Higher `treble_pulse` makes high
frequencies inject more force into the fluid:
```typescript
const treblePulse = Math.max(0, s?.treble_pulse ?? 1)
// In ambient splat:
const force = (0.2 + rms * 0.8 + treble * 0.3 * treblePulse + idle * 0.12) * splatForce * 400
```
`treble_pulse = 0` suppresses treble-driven force. `treble_pulse > 1` amplifies it. This is
semantically consistent with the original canvas intent ("treble inflates blob radius").
---
### E8 — Two RAF loops; `fpsCap` only throttles splat injection
The WebGL fluid sim runs its **own internal RAF loop** for pressure solving, advection, and
rendering. The component's outer RAF loop only injects splats based on audio features.
This means:
- The sim always renders at the browser's native frame rate (typically 60 Hz)
- `fpsCap` only throttles the frequency of splat injection, not visual rendering
- At `fpsCap = 30`, splats are injected at 30 Hz but the fluid animates at 60 Hz
This is acceptable behavior (cheap splat injection, smooth visual rendering) and is consistent
with how the original PavelDoGreat sim is designed to be extended. However, it should be
documented in code comments to avoid future confusion.
---
### E9 — Verify `COLORFUL: false` respects per-splat color parameter
PavelDoGreat's sim supports two color modes:
- `COLORFUL: true` — sim auto-generates HSL rainbow colors per splat, ignores the `color` arg
- `COLORFUL: false` — sim uses the `color` `{ r, g, b }` argument passed to `splat()`
If `COLORFUL: false` actually ignores the color arg (a known inconsistency in some sim forks),
the palette cycling via `nextColor()` will be silently broken and all fluid will render as
white.
**Mitigation during vendoring:** Inspect the `multipleSplats` / `splat` function in the
vendored `script.js`. Confirm the `color` parameter is applied to the dye texture. If not,
patch it:
```js
// In the vendored splat() function, ensure:
gl.uniform3f(splatProgram.uniforms.color, color.r, color.g, color.b)
// Not:
gl.uniform3f(splatProgram.uniforms.color, Math.random(), Math.random(), Math.random())
```
Add this check as a required verification step before shipping.
---
## Files changed
| File | Issue | Change |
|---|---|---|
| `frontend/src/components/LyricsLivePreview.tsx` | A | Add `analysisAudioUrl?: string` prop; use it as the hidden audio `src` with `audioUrl` as fallback; update `analysisSourceKey` |
| `frontend/src/components/KaraokePerformanceMode.tsx` | A | Add `analysisAudioUrl?: string` prop; forward to `LyricsLivePreview` |
| `frontend/src/pages/KaraokeTV.tsx` | A | Derive `analysisAudioUrl` memo (source file for project, first stem for SEK); pass to `KaraokePerformanceMode` |
| `frontend/src/pages/Karaoke.tsx` | A1 | Apply the same `analysisAudioUrl` derivation + pass-through as KaraokeTV |
| `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx` | B | Per-lane hue offset: `laneHue = (hue + pt.lane * hueRange * 45) % 360` |
| `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx` | C, D | Pass live `w/h` into `spawnBlob`; invert `background_persistence` alpha mapping |
| `frontend/src/components/visualizers/fluid/fluidSim.ts` | E | **New** — wrapper around `webgl-fluid` package exposing `initFluidSim`, `splatFluid`, `updateFluidConfig`, `destroyFluidSim` |
| `frontend/src/types/webgl-fluid.d.ts` | E3 | **New** — ambient module declaration for `webgl-fluid` |
| `frontend/package.json` + lockfile | E | Add `webgl-fluid` dependency |
| `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx` | E | **Full rewrite** — WebGL wrapper with config-mapping helpers; removes canvas blob renderer |
| `frontend/src/components/PerformanceSidebar.tsx` | E | Update `FLUID_PRESETS` values to match new WebGL parameter mapping |
| `SUMMARIES (Claude).md` | — | Append entry after implementation |
Recommended delivery split:
- **Phase 1 (safe/immediate):** A + A1 + B + C + D
- **Phase 2 (high-risk migration):** E + E1..E9
If Phase 2 is deferred, C/D remain required fixes in the existing canvas renderer.
---
## Implementation order
### Phase 1 (ship now)
1. Add `analysisAudioUrl` prop to `LyricsLivePreview` — update hidden analysis element + `analysisSourceKey`
2. Add `analysisAudioUrl` prop to `KaraokePerformanceMode` — forward-only change
3. Derive and pass `analysisAudioUrl` in both `KaraokeTVPage` and `KaraokePage`
4. Fix Muon per-lane hue offset in `MuonVisualizerCanvas.tsx`
5. Fix Fluid canvas issues C/D (`spawnBlob` live dimensions + `background_persistence` inversion)
6. Run `tsc --noEmit`
### Phase 2 (optional migration)
7. Add `webgl-fluid` dependency and implement `fluidSim.ts` wrapper APIs (`init`/`splat`/`update`/`destroy`)
8. Rewrite `FluidVisualizerCanvas.tsx` around WebGL wrapper with graceful fallback
9. Update `FLUID_PRESETS` in `PerformanceSidebar.tsx`
10. Run `tsc --noEmit`
11. Update `SUMMARIES (Claude).md`
+44 -250
View File
@@ -21,13 +21,13 @@
"@tonejs/midi": "^2.0.28",
"abcjs": "^6.6.2",
"lucide-react": "^0.294.0",
"meyda": "^5.6.3",
"qrcode": "^1.5.4",
"react": "^18.2.0",
"react-dom": "^18.2.0",
"react-router-dom": "^6.30.3",
"tone": "^14.7.0",
"two.js": "^0.8.23"
"two.js": "^0.8.23",
"webgl-fluid": "^0.3.9"
},
"devDependencies": {
"@playwright/test": "^1.53.1",
@@ -1064,64 +1064,6 @@
"integrity": "sha512-+d0F4MKMCbeVUJwG96uQ4SgAznZNSq93I3V+9NHA4OpvqG8mRCpGdKmK8l/dl02h2CCDHwW2FqilnTyDcAnqjA==",
"dev": true
},
"node_modules/@rollup/plugin-node-resolve": {
"version": "15.3.1",
"resolved": "https://registry.npmjs.org/@rollup/plugin-node-resolve/-/plugin-node-resolve-15.3.1.tgz",
"integrity": "sha512-tgg6b91pAybXHJQMAAwW9VuWBO6Thi+q7BCNARLwSqlmsHz0XYURtGvh/AuwSADXSI4h/2uHbs7s4FzlZDGSGA==",
"license": "MIT",
"dependencies": {
"@rollup/pluginutils": "^5.0.1",
"@types/resolve": "1.20.2",
"deepmerge": "^4.2.2",
"is-module": "^1.0.0",
"resolve": "^1.22.1"
},
"engines": {
"node": ">=14.0.0"
},
"peerDependencies": {
"rollup": "^2.78.0||^3.0.0||^4.0.0"
},
"peerDependenciesMeta": {
"rollup": {
"optional": true
}
}
},
"node_modules/@rollup/pluginutils": {
"version": "5.3.0",
"resolved": "https://registry.npmjs.org/@rollup/pluginutils/-/pluginutils-5.3.0.tgz",
"integrity": "sha512-5EdhGZtnu3V88ces7s53hhfK5KSASnJZv8Lulpc04cWO3REESroJXg73DFsOmgbU2BhwV0E20bu2IDZb3VKW4Q==",
"license": "MIT",
"dependencies": {
"@types/estree": "^1.0.0",
"estree-walker": "^2.0.2",
"picomatch": "^4.0.2"
},
"engines": {
"node": ">=14.0.0"
},
"peerDependencies": {
"rollup": "^1.20.0||^2.0.0||^3.0.0||^4.0.0"
},
"peerDependenciesMeta": {
"rollup": {
"optional": true
}
}
},
"node_modules/@rollup/pluginutils/node_modules/picomatch": {
"version": "4.0.4",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.4.tgz",
"integrity": "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==",
"license": "MIT",
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://github.com/sponsors/jonschlinkert"
}
},
"node_modules/@rollup/rollup-android-arm-eabi": {
"version": "4.59.0",
"resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm-eabi/-/rollup-android-arm-eabi-4.59.0.tgz",
@@ -1129,6 +1071,7 @@
"cpu": [
"arm"
],
"dev": true,
"optional": true,
"os": [
"android"
@@ -1141,6 +1084,7 @@
"cpu": [
"arm64"
],
"dev": true,
"optional": true,
"os": [
"android"
@@ -1153,6 +1097,7 @@
"cpu": [
"arm64"
],
"dev": true,
"optional": true,
"os": [
"darwin"
@@ -1165,6 +1110,7 @@
"cpu": [
"x64"
],
"dev": true,
"optional": true,
"os": [
"darwin"
@@ -1177,6 +1123,7 @@
"cpu": [
"arm64"
],
"dev": true,
"optional": true,
"os": [
"freebsd"
@@ -1189,6 +1136,7 @@
"cpu": [
"x64"
],
"dev": true,
"optional": true,
"os": [
"freebsd"
@@ -1201,6 +1149,7 @@
"cpu": [
"arm"
],
"dev": true,
"optional": true,
"os": [
"linux"
@@ -1213,6 +1162,7 @@
"cpu": [
"arm"
],
"dev": true,
"optional": true,
"os": [
"linux"
@@ -1225,6 +1175,7 @@
"cpu": [
"arm64"
],
"dev": true,
"optional": true,
"os": [
"linux"
@@ -1237,6 +1188,7 @@
"cpu": [
"arm64"
],
"dev": true,
"optional": true,
"os": [
"linux"
@@ -1249,6 +1201,7 @@
"cpu": [
"loong64"
],
"dev": true,
"optional": true,
"os": [
"linux"
@@ -1261,6 +1214,7 @@
"cpu": [
"loong64"
],
"dev": true,
"optional": true,
"os": [
"linux"
@@ -1273,6 +1227,7 @@
"cpu": [
"ppc64"
],
"dev": true,
"optional": true,
"os": [
"linux"
@@ -1285,6 +1240,7 @@
"cpu": [
"ppc64"
],
"dev": true,
"optional": true,
"os": [
"linux"
@@ -1297,6 +1253,7 @@
"cpu": [
"riscv64"
],
"dev": true,
"optional": true,
"os": [
"linux"
@@ -1309,6 +1266,7 @@
"cpu": [
"riscv64"
],
"dev": true,
"optional": true,
"os": [
"linux"
@@ -1321,6 +1279,7 @@
"cpu": [
"s390x"
],
"dev": true,
"optional": true,
"os": [
"linux"
@@ -1333,6 +1292,7 @@
"cpu": [
"x64"
],
"dev": true,
"optional": true,
"os": [
"linux"
@@ -1345,6 +1305,7 @@
"cpu": [
"x64"
],
"dev": true,
"optional": true,
"os": [
"linux"
@@ -1357,6 +1318,7 @@
"cpu": [
"x64"
],
"dev": true,
"optional": true,
"os": [
"openbsd"
@@ -1369,6 +1331,7 @@
"cpu": [
"arm64"
],
"dev": true,
"optional": true,
"os": [
"openharmony"
@@ -1381,6 +1344,7 @@
"cpu": [
"arm64"
],
"dev": true,
"optional": true,
"os": [
"win32"
@@ -1393,6 +1357,7 @@
"cpu": [
"ia32"
],
"dev": true,
"optional": true,
"os": [
"win32"
@@ -1405,6 +1370,7 @@
"cpu": [
"x64"
],
"dev": true,
"optional": true,
"os": [
"win32"
@@ -1417,6 +1383,7 @@
"cpu": [
"x64"
],
"dev": true,
"optional": true,
"os": [
"win32"
@@ -1499,7 +1466,8 @@
"node_modules/@types/estree": {
"version": "1.0.8",
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.8.tgz",
"integrity": "sha512-dWHzHa2WqEXI/O1E9OjrocMTKJl2mSrEolh1Iomrv6U+JuNwaHXsXx9bLu5gG7BUWFIN0skIQJQ/L1rIex4X6w=="
"integrity": "sha512-dWHzHa2WqEXI/O1E9OjrocMTKJl2mSrEolh1Iomrv6U+JuNwaHXsXx9bLu5gG7BUWFIN0skIQJQ/L1rIex4X6w==",
"dev": true
},
"node_modules/@types/node": {
"version": "25.5.2",
@@ -1546,12 +1514,6 @@
"@types/react": "^18.0.0"
}
},
"node_modules/@types/resolve": {
"version": "1.20.2",
"resolved": "https://registry.npmjs.org/@types/resolve/-/resolve-1.20.2.tgz",
"integrity": "sha512-60BCwRFOZCQhDncwQdxxeOEEkbc5dIMccYLwbxsS4TUNeVECQ/pBJ0j09mrHOl/JJvpRPGwO9SvE4nR2Nb/a4Q==",
"license": "MIT"
},
"node_modules/@typescript-eslint/eslint-plugin": {
"version": "8.57.0",
"resolved": "https://registry.npmjs.org/@typescript-eslint/eslint-plugin/-/eslint-plugin-8.57.0.tgz",
@@ -1985,12 +1947,6 @@
"postcss": "^8.1.0"
}
},
"node_modules/babel-plugin-add-module-exports": {
"version": "0.2.1",
"resolved": "https://registry.npmjs.org/babel-plugin-add-module-exports/-/babel-plugin-add-module-exports-0.2.1.tgz",
"integrity": "sha512-3AN/9V/rKuv90NG65m4tTHsI04XrCKsWbztIcW7a8H5iIN7WlvWucRtVV0V/rT4QvtA11n5Vmp20fLwfMWqp6g==",
"license": "MIT"
},
"node_modules/balanced-match": {
"version": "4.0.4",
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-4.0.4.tgz",
@@ -2083,34 +2039,6 @@
"node": "^6 || ^7 || ^8 || ^9 || ^10 || ^11 || ^12 || >=13.7"
}
},
"node_modules/buffer-alloc": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/buffer-alloc/-/buffer-alloc-1.2.0.tgz",
"integrity": "sha512-CFsHQgjtW1UChdXgbyJGtnm+O/uLQeZdtbDo8mfUgYXCHSM1wgrVxXm6bSyrUuErEb+4sYVGCzASBRot7zyrow==",
"license": "MIT",
"dependencies": {
"buffer-alloc-unsafe": "^1.1.0",
"buffer-fill": "^1.0.0"
}
},
"node_modules/buffer-alloc-unsafe": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/buffer-alloc-unsafe/-/buffer-alloc-unsafe-1.1.0.tgz",
"integrity": "sha512-TEM2iMIEQdJ2yjPJoSIsldnleVaAk1oW3DBVUykyOLsEsFmEc9kn+SFFPz+gl54KQNxlDnAwCXosOS9Okx2xAg==",
"license": "MIT"
},
"node_modules/buffer-fill": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/buffer-fill/-/buffer-fill-1.0.0.tgz",
"integrity": "sha512-T7zexNBwiiaCOGDg9xNX9PBmjrubblRkENuptryuI64URkXDFum9il/JGL8Lm8wYfAXpredVXXZz7eMHilimiQ==",
"license": "MIT"
},
"node_modules/buffer-from": {
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/buffer-from/-/buffer-from-1.1.2.tgz",
"integrity": "sha512-E+XQCRwSbaaiChtv6k6Dwgc+bx+Bs6vuKJHHl5kox/BaKbhiXzqQOwK4cO22yElGp2OCmjwVhT3HmxgyPGnJfQ==",
"license": "MIT"
},
"node_modules/callsites": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/callsites/-/callsites-3.1.0.tgz",
@@ -2263,12 +2191,6 @@
"integrity": "sha512-Kvp459HrV2FEJ1CAsi1Ku+MY3kasH19TFykTz2xWmMeq6bk2NU3XXvfJ+Q61m0xktWwt+1HSYf3JZsTms3aRJg==",
"dev": true
},
"node_modules/core-util-is": {
"version": "1.0.3",
"resolved": "https://registry.npmjs.org/core-util-is/-/core-util-is-1.0.3.tgz",
"integrity": "sha512-ZQBvi1DcpJ4GDqanjucZ2Hj3wEO5pZDS89BWbkcrvdxksJorwUDDZamX9ldFkp9aw2lmBDLgkObEA4DWNJ9FYQ==",
"license": "MIT"
},
"node_modules/cross-spawn": {
"version": "7.0.6",
"resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz",
@@ -2302,15 +2224,6 @@
"integrity": "sha512-z1HGKcYy2xA8AGQfwrn0PAy+PB7X/GSj3UVJW9qKyn43xWa+gl5nXmU4qqLMRzWVLFC8KusUX8T/0kCiOYpAIQ==",
"dev": true
},
"node_modules/dct": {
"version": "0.1.0",
"resolved": "https://registry.npmjs.org/dct/-/dct-0.1.0.tgz",
"integrity": "sha512-/uUtEniuMq1aUxvLAoDtAduyl12oM1zhA/le2f83UFN/9+4KDHXFB6znEfoj5SDDLiTpUTr26NpxC7t8IFOYhQ==",
"license": "MIT",
"engines": {
"node": ">=0.12.0"
}
},
"node_modules/debug": {
"version": "4.4.3",
"resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz",
@@ -2344,15 +2257,6 @@
"dev": true,
"license": "MIT"
},
"node_modules/deepmerge": {
"version": "4.3.1",
"resolved": "https://registry.npmjs.org/deepmerge/-/deepmerge-4.3.1.tgz",
"integrity": "sha512-3sUqbMEc77XqpdNO7FRyRog+eW3ph+GYCbj+rK+uYyRMuwsVy0rMiVtPn+QJlKFvWP/1PYpapqYn0Me2knFn+A==",
"license": "MIT",
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/didyoumean": {
"version": "1.2.2",
"resolved": "https://registry.npmjs.org/didyoumean/-/didyoumean-1.2.2.tgz",
@@ -2661,12 +2565,6 @@
"node": ">=4.0"
}
},
"node_modules/estree-walker": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/estree-walker/-/estree-walker-2.0.2.tgz",
"integrity": "sha512-Rfkk/Mp/DL7JVje3u18FxFujQlTNR2q6QfMSMB7AvCBx91NGj/ba3kCfza0f6dVDbw7YlRf/nDrn7pQrCCyQ/w==",
"license": "MIT"
},
"node_modules/esutils": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/esutils/-/esutils-2.0.3.tgz",
@@ -2735,15 +2633,6 @@
"reusify": "^1.0.4"
}
},
"node_modules/fftjs": {
"version": "0.0.4",
"resolved": "https://registry.npmjs.org/fftjs/-/fftjs-0.0.4.tgz",
"integrity": "sha512-nIWxQyth1LVD6NH8a+YZUv+McjzbOY6dMe4wv6Pq5cGfP+c8Rd1T8Dsd50DCWlNgzSqA3y9lOkpD6dZD3qHa1A==",
"license": "MIT",
"dependencies": {
"babel-plugin-add-module-exports": "^0.2.1"
}
},
"node_modules/file-entry-cache": {
"version": "6.0.1",
"resolved": "https://registry.npmjs.org/file-entry-cache/-/file-entry-cache-6.0.1.tgz",
@@ -2832,6 +2721,7 @@
"version": "2.3.3",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz",
"integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==",
"dev": true,
"hasInstallScript": true,
"optional": true,
"os": [
@@ -2845,6 +2735,7 @@
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz",
"integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==",
"dev": true,
"funding": {
"url": "https://github.com/sponsors/ljharb"
}
@@ -2969,6 +2860,7 @@
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.2.tgz",
"integrity": "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==",
"dev": true,
"dependencies": {
"function-bind": "^1.1.2"
},
@@ -3029,6 +2921,7 @@
"version": "2.0.4",
"resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
"integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==",
"dev": true,
"license": "ISC"
},
"node_modules/is-binary-path": {
@@ -3047,6 +2940,7 @@
"version": "2.16.1",
"resolved": "https://registry.npmjs.org/is-core-module/-/is-core-module-2.16.1.tgz",
"integrity": "sha512-UfoeMA6fIJ8wTYFEUjelnaGI67v6+N7qXJEvQuIGa99l4xsCruSYOVSQ0uPANn4dAzm8lkYPaKLrrijLq7x23w==",
"dev": true,
"dependencies": {
"hasown": "^2.0.2"
},
@@ -3087,12 +2981,6 @@
"node": ">=0.10.0"
}
},
"node_modules/is-module": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/is-module/-/is-module-1.0.0.tgz",
"integrity": "sha512-51ypPSPCoTEIN9dy5Oy+h4pShgJmPCygKfyRCISBI+JoWT/2oJvK8QPxmwv7b/p239jXrm9M1mlQbyKJ5A152g==",
"license": "MIT"
},
"node_modules/is-number": {
"version": "7.0.0",
"resolved": "https://registry.npmjs.org/is-number/-/is-number-7.0.0.tgz",
@@ -3112,12 +3000,6 @@
"node": ">=8"
}
},
"node_modules/isarray": {
"version": "0.0.1",
"resolved": "https://registry.npmjs.org/isarray/-/isarray-0.0.1.tgz",
"integrity": "sha512-D2S+3GLxWH+uhrNEcoh/fnmYeP8E8/zHl644d/jdA0g2uyXvy3sb0qxotE+ne0LtccHknQzWwZEzhak7oJ0COQ==",
"license": "MIT"
},
"node_modules/isexe": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz",
@@ -3299,25 +3181,6 @@
"node": ">= 8"
}
},
"node_modules/meyda": {
"version": "5.6.3",
"resolved": "https://registry.npmjs.org/meyda/-/meyda-5.6.3.tgz",
"integrity": "sha512-fAdwfzIi1WDoL0idUQvCD7dZ7EN74FYH83G+jZQO3Nr9yOEBtzFvcMg2KLdLlu6psSP8XFlO0kYynG5o/E681Q==",
"license": "MIT",
"workspaces": [
"docs"
],
"dependencies": {
"@rollup/plugin-node-resolve": "^15.2.3",
"dct": "0.1.0",
"fftjs": "0.0.4",
"node-getopt": "^0.3.2",
"wav": "^1.0.2"
},
"bin": {
"meyda": "bin/cli.js"
}
},
"node_modules/micromatch": {
"version": "4.0.8",
"resolved": "https://registry.npmjs.org/micromatch/-/micromatch-4.0.8.tgz",
@@ -3394,15 +3257,6 @@
"dev": true,
"license": "MIT"
},
"node_modules/node-getopt": {
"version": "0.3.2",
"resolved": "https://registry.npmjs.org/node-getopt/-/node-getopt-0.3.2.tgz",
"integrity": "sha512-yqkmYrMbK1wPrfz7mgeYvA4tBperLg9FQ4S3Sau3nSAkpOA0x0zC8nQ1siBwozy1f4SE8vq2n1WKv99r+PCa1Q==",
"license": "MIT",
"engines": {
"node": ">= 0.6.0"
}
},
"node_modules/node-releases": {
"version": "2.0.27",
"resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.27.tgz",
@@ -3550,7 +3404,8 @@
"node_modules/path-parse": {
"version": "1.0.7",
"resolved": "https://registry.npmjs.org/path-parse/-/path-parse-1.0.7.tgz",
"integrity": "sha512-LDJzPVEEEPR+y48z93A0Ed0yXb8pAByGWo/k5YYdYgpY2/2EsOsksJrq7lOHxryrVOn1ejG6oAp8ahvOIQD8sw=="
"integrity": "sha512-LDJzPVEEEPR+y48z93A0Ed0yXb8pAByGWo/k5YYdYgpY2/2EsOsksJrq7lOHxryrVOn1ejG6oAp8ahvOIQD8sw==",
"dev": true
},
"node_modules/picocolors": {
"version": "1.1.1",
@@ -3930,18 +3785,6 @@
"pify": "^2.3.0"
}
},
"node_modules/readable-stream": {
"version": "1.1.14",
"resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-1.1.14.tgz",
"integrity": "sha512-+MeVjFf4L44XUkhM1eYbD8fyEsxcV81pqMSR5gblfcLCHfZvbrqy4/qYHE+/R5HoBUT11WV5O08Cr1n3YXkWVQ==",
"license": "MIT",
"dependencies": {
"core-util-is": "~1.0.0",
"inherits": "~2.0.1",
"isarray": "0.0.1",
"string_decoder": "~0.10.x"
}
},
"node_modules/readdirp": {
"version": "3.6.0",
"resolved": "https://registry.npmjs.org/readdirp/-/readdirp-3.6.0.tgz",
@@ -3973,6 +3816,7 @@
"version": "1.22.11",
"resolved": "https://registry.npmjs.org/resolve/-/resolve-1.22.11.tgz",
"integrity": "sha512-RfqAvLnMl313r7c9oclB1HhUEAezcpLjz95wFH4LVuhk9JF/r22qmVP9AMmOU4vMX7Q8pN8jwNg/CSpdFnMjTQ==",
"dev": true,
"dependencies": {
"is-core-module": "^2.16.1",
"path-parse": "^1.0.7",
@@ -4029,7 +3873,7 @@
"version": "4.59.0",
"resolved": "https://registry.npmjs.org/rollup/-/rollup-4.59.0.tgz",
"integrity": "sha512-2oMpl67a3zCH9H79LeMcbDhXW/UmWG/y2zuqnF2jQq5uq9TbM9TVyXvA4+t+ne2IIkBdrLpAaRQAvo7YI/Yyeg==",
"devOptional": true,
"dev": true,
"dependencies": {
"@types/estree": "1.0.8"
},
@@ -4158,36 +4002,6 @@
"tslib": "^2.7.0"
}
},
"node_modules/stream-parser": {
"version": "0.3.1",
"resolved": "https://registry.npmjs.org/stream-parser/-/stream-parser-0.3.1.tgz",
"integrity": "sha512-bJ/HgKq41nlKvlhccD5kaCr/P+Hu0wPNKPJOH7en+YrJu/9EgqUF+88w5Jb6KNcjOFMhfX4B2asfeAtIGuHObQ==",
"license": "MIT",
"dependencies": {
"debug": "2"
}
},
"node_modules/stream-parser/node_modules/debug": {
"version": "2.6.9",
"resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz",
"integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==",
"license": "MIT",
"dependencies": {
"ms": "2.0.0"
}
},
"node_modules/stream-parser/node_modules/ms": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
"integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A==",
"license": "MIT"
},
"node_modules/string_decoder": {
"version": "0.10.31",
"resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-0.10.31.tgz",
"integrity": "sha512-ev2QzSzWPYmy9GuqfIVildA4OdcGLeFZQrq5ys6RtiuF+RQQiZWr8TZNyAcuVXyQRYfEO+MsoB/1BuQVhOJuoQ==",
"license": "MIT"
},
"node_modules/string-width": {
"version": "4.2.3",
"resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz",
@@ -4266,6 +4080,7 @@
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/supports-preserve-symlinks-flag/-/supports-preserve-symlinks-flag-1.0.0.tgz",
"integrity": "sha512-ot0WnXS9fgdkgIcePe6RHNk1WA8+muPa6cSjeR3V8K27q9BB1rTE3R1p7Hv0z1ZyAc8s6Vvv8DIyWf681MAt0w==",
"dev": true,
"engines": {
"node": ">= 0.4"
},
@@ -4587,32 +4402,11 @@
}
}
},
"node_modules/wav": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/wav/-/wav-1.0.2.tgz",
"integrity": "sha512-viHtz3cDd/Tcr/HbNqzQCofKdF6kWUymH9LGDdskfWFoIy/HJ+RTihgjEcHfnsy1PO4e9B+y4HwgTwMrByquhg==",
"license": "MIT",
"dependencies": {
"buffer-alloc": "^1.1.0",
"buffer-from": "^1.0.0",
"debug": "^2.2.0",
"readable-stream": "^1.1.14",
"stream-parser": "^0.3.1"
}
},
"node_modules/wav/node_modules/debug": {
"version": "2.6.9",
"resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz",
"integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==",
"license": "MIT",
"dependencies": {
"ms": "2.0.0"
}
},
"node_modules/wav/node_modules/ms": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
"integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A==",
"node_modules/webgl-fluid": {
"version": "0.3.9",
"resolved": "https://registry.npmjs.org/webgl-fluid/-/webgl-fluid-0.3.9.tgz",
"integrity": "sha512-dQpoe6JB3ZdDoOk4v1ZV7haNCAESu/2/XAQ9zK6/qbjssznZFNWx2MLWEPLdCQtWWGwmiZnDN+n461qTZ9+1nw==",
"hasInstallScript": true,
"license": "MIT"
},
"node_modules/which": {
+2 -2
View File
@@ -30,13 +30,13 @@
"@tonejs/midi": "^2.0.28",
"abcjs": "^6.6.2",
"lucide-react": "^0.294.0",
"meyda": "^5.6.3",
"qrcode": "^1.5.4",
"react": "^18.2.0",
"react-dom": "^18.2.0",
"react-router-dom": "^6.30.3",
"tone": "^14.7.0",
"two.js": "^0.8.23"
"two.js": "^0.8.23",
"webgl-fluid": "^0.3.9"
},
"devDependencies": {
"@playwright/test": "^1.53.1",
+314
View File
@@ -0,0 +1,314 @@
export interface MuonFrequencyBands {
sub: number
bass: number
lowMid: number
mid: number
highMid: number
presence: number
brilliance: number
}
export interface MuonBeatInfo {
isBeat: boolean
bpm: number
confidence: number
timeSinceLastBeat: number
beatPhase: number
}
export interface MuonPatternInfo {
isBeatDrop: boolean
isBuildUp: boolean
isSilence: boolean
intensity: number
energyDelta: number
}
export interface MuonFrame {
rms: number
energy: number
spectralCentroid: number
spectralRolloff: number
spectralFlatness: number
zcr: number
frequencyBands: MuonFrequencyBands
beatInfo: MuonBeatInfo
patternInfo: MuonPatternInfo
floatFreqData: Float32Array
floatTimeData: Float32Array
}
type BandKey = keyof MuonFrequencyBands
const AUDIO_CONFIG = {
FFT_SIZE: 1024,
SMOOTHING: 0.8,
MIN_DECIBELS: -100,
MAX_DECIBELS: -30,
} as const
const ENERGY_HISTORY_SIZE = 43
const BEAT_TIME_HISTORY_SIZE = 24
const FREQUENCY_BANDS: Record<BandKey, [number, number]> = {
sub: [20, 60],
bass: [60, 250],
lowMid: [250, 500],
mid: [500, 2000],
highMid: [2000, 4000],
presence: [4000, 6000],
brilliance: [6000, 20000],
}
function clamp01(value: number): number {
return Math.max(0, Math.min(1, value))
}
export class MuonFeatureExtractor {
private isRunning = false
private animationFrameId: number | null = null
private frequencyData!: Float32Array
private timeData!: Float32Array
private energyHistory: number[] = []
private lastBeatTime = 0
private beatThreshold = 1.5
private bpm = 0
private beatTimes: number[] = []
private previousEnergy = 0
start(analyser: AnalyserNode, ctx: AudioContext, onFrame: (f: MuonFrame) => void): void {
this.stop()
analyser.fftSize = AUDIO_CONFIG.FFT_SIZE
analyser.smoothingTimeConstant = AUDIO_CONFIG.SMOOTHING
analyser.minDecibels = AUDIO_CONFIG.MIN_DECIBELS
analyser.maxDecibels = AUDIO_CONFIG.MAX_DECIBELS
this.frequencyData = new Float32Array(analyser.frequencyBinCount)
this.timeData = new Float32Array(analyser.fftSize)
this.isRunning = true
const extractLoop = () => {
if (!this.isRunning) return
analyser.getFloatFrequencyData(this.frequencyData as unknown as Float32Array<ArrayBuffer>)
analyser.getFloatTimeDomainData(this.timeData as unknown as Float32Array<ArrayBuffer>)
const frame = this.extractFeatures(ctx.sampleRate)
onFrame(frame)
this.animationFrameId = window.requestAnimationFrame(extractLoop)
}
this.animationFrameId = window.requestAnimationFrame(extractLoop)
}
stop(): void {
this.isRunning = false
if (this.animationFrameId !== null) {
window.cancelAnimationFrame(this.animationFrameId)
this.animationFrameId = null
}
}
setBeatThreshold(threshold: number): void {
this.beatThreshold = Math.max(0, Math.min(3, threshold))
}
reset(): void {
this.energyHistory = []
this.lastBeatTime = 0
this.bpm = 0
this.beatTimes = []
this.previousEnergy = 0
}
private extractFeatures(sampleRate: number): MuonFrame {
const bins = this.frequencyData.length || 1
const nyquist = sampleRate / 2
const magnitude = new Float32Array(this.frequencyData.length)
for (let i = 0; i < this.frequencyData.length; i += 1) {
const db = Number.isFinite(this.frequencyData[i]) ? this.frequencyData[i] : AUDIO_CONFIG.MIN_DECIBELS
magnitude[i] = Math.pow(10, db / 20)
}
let powerSum = 0
let magSum = 0
let weightedBinSum = 0
let totalEnergy = 0
for (let i = 0; i < magnitude.length; i += 1) {
const m = magnitude[i]
const p = m * m
powerSum += p
magSum += m
weightedBinSum += i * m
totalEnergy += m
}
const rms = Math.sqrt(powerSum / bins)
const energy = totalEnergy / bins
const centroidBin = magSum > 0 ? weightedBinSum / magSum : 0
const spectralCentroid = (centroidBin / bins) * nyquist
const rolloffTarget = powerSum * 0.85
let rolloffPower = 0
let rolloffBin = 0
for (let i = 0; i < magnitude.length; i += 1) {
rolloffPower += magnitude[i] * magnitude[i]
if (rolloffPower >= rolloffTarget) {
rolloffBin = i
break
}
}
const spectralRolloff = (rolloffBin / bins) * nyquist
let geoLogSum = 0
for (let i = 0; i < magnitude.length; i += 1) {
geoLogSum += Math.log(Math.max(1e-12, magnitude[i]))
}
const geometricMean = Math.exp(geoLogSum / bins)
const arithmeticMean = magSum / bins
const spectralFlatness = arithmeticMean > 0 ? geometricMean / arithmeticMean : 0
let zc = 0
for (let i = 1; i < this.timeData.length; i += 1) {
const prev = this.timeData[i - 1]
const cur = this.timeData[i]
if ((prev >= 0 && cur < 0) || (prev < 0 && cur >= 0)) zc += 1
}
const zcr = this.timeData.length > 1 ? zc / (this.timeData.length - 1) : 0
const frequencyBands = this.extractFrequencyBands(sampleRate, magnitude)
const beatInfo = this.detectBeat(energy)
const patternInfo = this.detectPatterns(energy, beatInfo.isBeat)
return {
rms: clamp01(rms),
energy: clamp01(energy),
spectralCentroid,
spectralRolloff,
spectralFlatness: clamp01(spectralFlatness),
zcr: clamp01(zcr),
frequencyBands,
beatInfo,
patternInfo,
floatFreqData: new Float32Array(this.frequencyData),
floatTimeData: new Float32Array(this.timeData),
}
}
private extractFrequencyBands(sampleRate: number, magnitude: Float32Array): MuonFrequencyBands {
const binCount = magnitude.length
const nyquist = sampleRate / 2
const hzPerBin = binCount > 0 ? nyquist / binCount : 1
const out: MuonFrequencyBands = {
sub: 0,
bass: 0,
lowMid: 0,
mid: 0,
highMid: 0,
presence: 0,
brilliance: 0,
}
const keys: BandKey[] = ['sub', 'bass', 'lowMid', 'mid', 'highMid', 'presence', 'brilliance']
for (const key of keys) {
const [lowHz, highHz] = FREQUENCY_BANDS[key]
const start = Math.max(0, Math.floor(lowHz / hzPerBin))
const end = Math.min(binCount, Math.ceil(highHz / hzPerBin))
const span = Math.max(1, end - start)
let sumSquares = 0
for (let i = start; i < end; i += 1) {
const m = magnitude[i]
sumSquares += m * m
}
out[key] = Math.sqrt(sumSquares / span)
}
return out
}
private detectBeat(energy: number): MuonBeatInfo {
const now = performance.now() / 1000
this.energyHistory.push(energy)
if (this.energyHistory.length > ENERGY_HISTORY_SIZE) this.energyHistory.shift()
const { average, variance } = this.calculateLocalAverageAndVariance(this.energyHistory)
const stdDev = Math.sqrt(Math.max(variance, 1e-9))
const zScore = stdDev > 0 ? (energy - average) / stdDev : 0
const minInterval = 0.18
const isCandidate = energy > average * this.beatThreshold && zScore > 0.35
const isBeat = isCandidate && (now - this.lastBeatTime) > minInterval
if (isBeat) {
this.lastBeatTime = now
this.beatTimes.push(now)
if (this.beatTimes.length > BEAT_TIME_HISTORY_SIZE) this.beatTimes.shift()
this.bpm = this.estimateBpm()
}
const beatPeriod = this.bpm > 0 ? 60 / this.bpm : 0
const timeSinceLastBeat = this.lastBeatTime > 0 ? now - this.lastBeatTime : 0
const beatPhase = beatPeriod > 0 ? (timeSinceLastBeat % beatPeriod) / beatPeriod : 0
const confidence = clamp01((zScore / 3) * 0.6 + ((energy / Math.max(average, 1e-6)) - 1) * 0.4)
return {
isBeat,
bpm: this.bpm,
confidence,
timeSinceLastBeat,
beatPhase: clamp01(beatPhase),
}
}
private detectPatterns(energy: number, isBeat: boolean): MuonPatternInfo {
const energyDelta = energy - this.previousEnergy
this.previousEnergy = energy
const tail = this.energyHistory.slice(-12)
const tailAvg = tail.length ? tail.reduce((sum, v) => sum + v, 0) / tail.length : energy
const silenceFloor = 0.008
const isSilence = energy < silenceFloor
const isBuildUp = !isSilence && energyDelta > 0.01 && energy > tailAvg * 1.02
const isBeatDrop = isBeat && !isSilence && energy > tailAvg * 1.35
return {
isBeatDrop,
isBuildUp,
isSilence,
intensity: clamp01(energy * 8),
energyDelta,
}
}
private calculateLocalAverageAndVariance(values: number[]): { average: number; variance: number } {
if (!values.length) return { average: 0, variance: 0 }
const average = values.reduce((sum, value) => sum + value, 0) / values.length
const variance = values.reduce((sum, value) => {
const d = value - average
return sum + d * d
}, 0) / values.length
return { average, variance }
}
private estimateBpm(): number {
if (this.beatTimes.length < 2) return this.bpm
const intervals: number[] = []
for (let i = 1; i < this.beatTimes.length; i += 1) {
const dt = this.beatTimes[i] - this.beatTimes[i - 1]
if (dt > 0.2 && dt < 1.5) intervals.push(dt)
}
if (!intervals.length) return this.bpm
const avgInterval = intervals.reduce((sum, v) => sum + v, 0) / intervals.length
const bpm = 60 / avgInterval
return Math.max(40, Math.min(220, bpm))
}
}
@@ -11,6 +11,7 @@ interface KaraokePerformanceModeProps {
lyricsTracks?: Record<string, string>
stems?: Array<{ name: string; label: string; url: string }>
audioUrl?: string
analysisAudioUrl?: string
videoUrl?: string
videoPlaylist?: Array<{ url: string; duration?: number }>
videoPlaybackMode?: 'synced' | 'playlist'
@@ -54,6 +55,7 @@ export default function KaraokePerformanceMode({
lyricsTracks,
stems,
audioUrl,
analysisAudioUrl,
videoUrl,
videoPlaylist,
videoPlaybackMode,
@@ -124,6 +126,7 @@ export default function KaraokePerformanceMode({
lyricsTracks={lyricsTracks}
stems={stems}
audioUrl={audioUrl}
analysisAudioUrl={analysisAudioUrl}
videoUrl={videoUrl}
videoPlaylist={videoPlaylist}
videoPlaybackMode={videoPlaybackMode}
+238 -28
View File
@@ -56,22 +56,84 @@ interface VizSettings {
quality: 'low' | 'balanced' | 'high'
fps_cap: 30 | 60 | 120
muon: {
// Original multipliers
radius_multiplier: number
aperture: number
spacing: number
// Controls Audit additions
particle_count: number // direct int 40–500
rotation_speed: number // ×
glow_intensity: number // ×
particle_size: number // ×
drum_reactivity: number // ×
hue_range: number // ×
beat_sensitivity: number // ×
ellipse_ratio: number // direct 0.30–1.0
// GUI Parameter Map additions
idle_speed: number // ×
noise_scale: number // × (reserved for Three.js port)
noise_force: number // × (reserved for Three.js port)
mono_color: boolean
divisions: number
lifespan: number
particle_mirror: boolean
dynamic_radius: number
delta_response_limit: number
visualization_preset: 'default' | 'orbital' | 'flower' | 'spiro' | 'hybrid'
}
fluid: {
// Original multipliers
spawn_density: number
splat_force: number
decay: number
// Controls Audit additions
background_persistence: number
beat_burst_count: number
treble_pulse: number
drag: number
blob_life: number
// GUI Parameter Map additions
color_speed: number
}
}
const VIZ_DEFAULTS: VizSettings = {
quality: 'balanced',
fps_cap: 60,
muon: { radius_multiplier: 1.0, aperture: 1.0, spacing: 1.0 },
fluid: { spawn_density: 1.0, splat_force: 1.0, decay: 1.0 },
muon: {
radius_multiplier: 1.0,
aperture: 1.0,
spacing: 1.0,
particle_count: 220,
rotation_speed: 1.0,
glow_intensity: 1.0,
particle_size: 1.0,
drum_reactivity: 1.0,
hue_range: 1.0,
beat_sensitivity: 1.0,
ellipse_ratio: 0.62,
idle_speed: 1.0,
noise_scale: 1.0,
noise_force: 1.0,
mono_color: false,
divisions: 16,
lifespan: 180,
particle_mirror: false,
dynamic_radius: 1.0,
delta_response_limit: 0.005,
visualization_preset: 'hybrid',
},
fluid: {
spawn_density: 1.0,
splat_force: 1.0,
decay: 1.0,
background_persistence: 1.0,
beat_burst_count: 1.0,
treble_pulse: 1.0,
drag: 1.0,
blob_life: 1.0,
color_speed: 1.0,
},
}
const PERF_SIDEBAR_WIDTH = 280
@@ -81,9 +143,9 @@ function isLeadingVocalStem(name: string): boolean {
return LEADING_VOCAL_PATTERN.test((name || '').trim())
}
function clampVizMultiplier(value: number): number {
function clampVizMultiplier(value: number, min = 0.0, max = 3.0): number {
if (!Number.isFinite(value)) return 1
return Math.max(0.25, Math.min(2, value))
return Math.max(min, Math.min(max, value))
}
function deepMergeVizSettings(base: VizSettings, patch?: DeepPartial<VizSettings> | null): VizSettings {
@@ -113,14 +175,45 @@ function normalizeVizSettings(input: VizSettings): VizSettings {
quality: input.quality === 'low' || input.quality === 'high' ? input.quality : 'balanced',
fps_cap: input.fps_cap === 30 || input.fps_cap === 120 ? input.fps_cap : 60,
muon: {
radius_multiplier: clampVizMultiplier(input.muon.radius_multiplier),
aperture: clampVizMultiplier(input.muon.aperture),
spacing: clampVizMultiplier(input.muon.spacing),
radius_multiplier: clampVizMultiplier(input.muon.radius_multiplier, 0.25, 2.0),
aperture: clampVizMultiplier(input.muon.aperture, 0.25, 2.0),
spacing: clampVizMultiplier(input.muon.spacing, 0.25, 2.0),
particle_count: Math.max(40, Math.min(500, Math.round(Number.isFinite(input.muon.particle_count) ? input.muon.particle_count : 220))),
rotation_speed: clampVizMultiplier(input.muon.rotation_speed, 0.2, 4.0),
glow_intensity: clampVizMultiplier(input.muon.glow_intensity, 0.0, 3.0),
particle_size: clampVizMultiplier(input.muon.particle_size, 0.25, 3.0),
drum_reactivity: clampVizMultiplier(input.muon.drum_reactivity, 0.0, 2.0),
hue_range: clampVizMultiplier(input.muon.hue_range, 0.0, 3.0),
beat_sensitivity: clampVizMultiplier(input.muon.beat_sensitivity, 0.2, 3.0),
ellipse_ratio: clampVizMultiplier(input.muon.ellipse_ratio, 0.30, 1.0),
idle_speed: clampVizMultiplier(input.muon.idle_speed, 0.0, 2.0),
noise_scale: clampVizMultiplier(input.muon.noise_scale, 0.0, 3.0),
noise_force: clampVizMultiplier(input.muon.noise_force, 0.0, 3.0),
mono_color: Boolean(input.muon.mono_color),
divisions: Math.max(8, Math.min(48, Math.round(Number.isFinite(input.muon.divisions) ? input.muon.divisions : 16))),
lifespan: Math.max(45, Math.min(420, Math.round(Number.isFinite(input.muon.lifespan) ? input.muon.lifespan : 180))),
particle_mirror: Boolean(input.muon.particle_mirror),
dynamic_radius: clampVizMultiplier(Number.isFinite(input.muon.dynamic_radius) ? input.muon.dynamic_radius : 1.0, 0.2, 3.0),
delta_response_limit: clampVizMultiplier(Number.isFinite(input.muon.delta_response_limit) ? input.muon.delta_response_limit : 0.005, 0.0005, 0.05),
visualization_preset:
input.muon.visualization_preset === 'default' ||
input.muon.visualization_preset === 'orbital' ||
input.muon.visualization_preset === 'flower' ||
input.muon.visualization_preset === 'spiro' ||
input.muon.visualization_preset === 'hybrid'
? input.muon.visualization_preset
: 'hybrid',
},
fluid: {
spawn_density: clampVizMultiplier(input.fluid.spawn_density),
splat_force: clampVizMultiplier(input.fluid.splat_force),
decay: clampVizMultiplier(input.fluid.decay),
spawn_density: clampVizMultiplier(input.fluid.spawn_density, 0.25, 3.0),
splat_force: clampVizMultiplier(input.fluid.splat_force, 0.25, 3.0),
decay: clampVizMultiplier(input.fluid.decay, 0.25, 3.0),
background_persistence: clampVizMultiplier(input.fluid.background_persistence, 0.2, 3.0),
beat_burst_count: clampVizMultiplier(input.fluid.beat_burst_count, 0.25, 3.0),
treble_pulse: clampVizMultiplier(input.fluid.treble_pulse, 0.0, 3.0),
drag: clampVizMultiplier(input.fluid.drag, 0.5, 2.0),
blob_life: clampVizMultiplier(input.fluid.blob_life, 0.25, 3.0),
color_speed: clampVizMultiplier(input.fluid.color_speed, 0.0, 3.0),
},
}
}
@@ -129,6 +222,7 @@ interface Props {
lrcText: string
lyricsTracks?: Record<string, string>
audioUrl?: string
analysisAudioUrl?: string
videoUrl?: string
videoPlaylist?: Array<{ url: string; duration?: number }>
videoPlaybackMode?: 'synced' | 'playlist'
@@ -155,11 +249,35 @@ interface Props {
radius_multiplier?: number
aperture?: number
spacing?: number
particle_count?: number
rotation_speed?: number
glow_intensity?: number
particle_size?: number
drum_reactivity?: number
hue_range?: number
beat_sensitivity?: number
ellipse_ratio?: number
idle_speed?: number
noise_scale?: number
noise_force?: number
mono_color?: boolean
divisions?: number
lifespan?: number
particle_mirror?: boolean
dynamic_radius?: number
delta_response_limit?: number
visualization_preset?: 'default' | 'orbital' | 'flower' | 'spiro' | 'hybrid'
}
fluid?: {
spawn_density?: number
splat_force?: number
decay?: number
background_persistence?: number
beat_burst_count?: number
treble_pulse?: number
drag?: number
blob_life?: number
color_speed?: number
}
} | null
/** Artist name from source audio metadata. */
@@ -1429,7 +1547,7 @@ function LrcEditor({
// ─── Main component ───────────────────────────────────────────────────────────
export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, videoUrl, videoPlaylist, videoPlaybackMode, stems, title, styleOptions, defaultStyle, onLrcChange, storageKey, coverArtUrl, thumbnailUrl, sekDynamicPalette, sekVisualizerMode, sekVisualizerSettings, artist, songTitle, performanceMode = false, autoPlay = false, disableGlobalPlaybackRegistry = false, onPlaybackEnded }: Props) {
export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, analysisAudioUrl, videoUrl, videoPlaylist, videoPlaybackMode, stems, title, styleOptions, defaultStyle, onLrcChange, storageKey, coverArtUrl, thumbnailUrl, sekDynamicPalette, sekVisualizerMode, sekVisualizerSettings, artist, songTitle, performanceMode = false, autoPlay = false, disableGlobalPlaybackRegistry = false, onPlaybackEnded }: Props) {
const resolvedLyricsTracks = useMemo(() => {
const normalized: Record<string, string> = {}
for (const [rawKey, rawText] of Object.entries(lyricsTracks || {})) {
@@ -1517,18 +1635,44 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
if (!sekVisualizerSettings) return null
const quality = sekVisualizerSettings.quality
const fps = sekVisualizerSettings.fps_cap
const sm = sekVisualizerSettings.muon
const sf = sekVisualizerSettings.fluid
const patch: DeepPartial<VizSettings> = {
quality: quality === 'low' || quality === 'balanced' || quality === 'high' ? quality : undefined,
fps_cap: fps === 30 || fps === 60 || fps === 120 ? fps : undefined,
muon: {
radius_multiplier: sekVisualizerSettings.muon?.radius_multiplier,
aperture: sekVisualizerSettings.muon?.aperture,
spacing: sekVisualizerSettings.muon?.spacing,
radius_multiplier: sm?.radius_multiplier,
aperture: sm?.aperture,
spacing: sm?.spacing,
particle_count: sm?.particle_count,
rotation_speed: sm?.rotation_speed,
glow_intensity: sm?.glow_intensity,
particle_size: sm?.particle_size,
drum_reactivity: sm?.drum_reactivity,
hue_range: sm?.hue_range,
beat_sensitivity: sm?.beat_sensitivity,
ellipse_ratio: sm?.ellipse_ratio,
idle_speed: sm?.idle_speed,
noise_scale: sm?.noise_scale,
noise_force: sm?.noise_force,
mono_color: sm?.mono_color,
divisions: sm?.divisions,
lifespan: sm?.lifespan,
particle_mirror: sm?.particle_mirror,
dynamic_radius: sm?.dynamic_radius,
delta_response_limit: sm?.delta_response_limit,
visualization_preset: sm?.visualization_preset,
},
fluid: {
spawn_density: sekVisualizerSettings.fluid?.spawn_density,
splat_force: sekVisualizerSettings.fluid?.splat_force,
decay: sekVisualizerSettings.fluid?.decay,
spawn_density: sf?.spawn_density,
splat_force: sf?.splat_force,
decay: sf?.decay,
background_persistence: sf?.background_persistence,
beat_burst_count: sf?.beat_burst_count,
treble_pulse: sf?.treble_pulse,
drag: sf?.drag,
blob_life: sf?.blob_life,
color_speed: sf?.color_speed,
},
}
return normalizeVizSettings(deepMergeVizSettings(VIZ_DEFAULTS, patch))
@@ -1767,6 +1911,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
// Single-track audio ref
const audioRef = useRef<HTMLAudioElement>(null)
const analysisAudioRef = useRef<HTMLAudioElement>(null)
const videoRef = useRef<HTMLVideoElement>(null)
// Multi-stem audio refs
const stemAudioRefs = useRef<Record<string, HTMLAudioElement>>({})
@@ -1778,19 +1923,12 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
const gainRefs = useRef<Record<string, GainNode>>({})
const metadataListenerRefs = useRef<Record<string, () => void>>({})
// Important: on first mount in multi-stem mode, `useAudioFeatures` can run
// before the stem graph is created (`gainRefs.current` still empty). In that
// case the hook exits early and would not reconnect until one of its deps
// changes, causing static/non-reactive visualizers. Gate it until tap nodes
// exist so the first enabled run always has a valid audio source.
const audioFeatureTapReady = !isMultiStem || Object.keys(gainRefs.current).length > 0
const featuresRef = useAudioFeatures({
audioCtxRef,
stemGainRefs: gainRefs,
singleAudioRef: audioRef,
analysisAudioRef,
isPlayingRef,
enabled: isVisualizerMode && audioFeatureTapReady,
enabled: isVisualizerMode,
analysisSourceKey: analysisAudioUrl ?? audioUrl ?? null,
})
const containerRef = useRef<HTMLDivElement>(null)
@@ -1832,6 +1970,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
audioRef.current.pause()
}
if (videoRef.current) videoRef.current.pause()
syncAnalysisAudio(currentTimeRef.current, false)
syncAnchorStemRef.current = null
lastAnchorTimeRef.current = null
lastRafTime.current = null
@@ -1861,6 +2000,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
syncAnchorStemRef.current = null
lastAnchorTimeRef.current = null
const finalTime = typeof endedAt === 'number' ? endedAt : currentTimeRef.current
syncAnalysisAudio(finalTime, false)
manualTimeRef.current = finalTime
currentTimeRef.current = finalTime
lastRafTime.current = null
@@ -2312,6 +2452,33 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
return () => window.clearInterval(timer)
}, [isMultiStem, isPlaying, useVideoAsTimeSource])
useEffect(() => {
if (!isPlaying) return
const timer = window.setInterval(() => {
const analysis = analysisAudioRef.current
if (!analysis) return
const transportRate = isMultiStem
? (pickStemClockSource(syncAnchorStemRef.current)?.playbackRate ?? 1)
: (audioRef.current?.playbackRate ?? videoRef.current?.playbackRate ?? 1)
if (analysis.playbackRate !== transportRate) {
analysis.playbackRate = transportRate
}
const transportTime = currentTimeRef.current
if (Math.abs(analysis.currentTime - transportTime) > 0.04) {
analysis.currentTime = transportTime
}
if (analysis.paused) {
analysis.play().catch(() => {})
}
}, 120)
return () => window.clearInterval(timer)
}, [isPlaying, isMultiStem])
// End multi-stem playback as soon as every active stem reaches end.
useEffect(() => {
if (!isMultiStem || !isPlaying) return
@@ -2457,6 +2624,26 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
)
}
function syncAnalysisAudio(targetTime: number, shouldPlay: boolean) {
const analysis = analysisAudioRef.current
if (!analysis) return
const transportRate = isMultiStemRef.current
? (pickStemClockSource(syncAnchorStemRef.current)?.playbackRate ?? 1)
: (audioRef.current?.playbackRate ?? videoRef.current?.playbackRate ?? 1)
analysis.playbackRate = transportRate
if (Math.abs(analysis.currentTime - targetTime) > 0.04) {
analysis.currentTime = targetTime
}
if (shouldPlay) {
analysis.play().catch(() => {})
} else {
analysis.pause()
}
}
function mapTransportTimeToVideoSegment(targetTime: number): { index: number; localTime: number } {
if (!isPlaylistBackdropMode || !videoPlaylistRef.current.length) {
return { index: 0, localTime: targetTime }
@@ -2524,6 +2711,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
currentTimeRef.current = t
lastRafTime.current = null
setCurrentTime(t)
syncAnalysisAudio(t, isPlayingRef.current)
return t
}
@@ -2578,6 +2766,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
const audios = Object.values(stemAudioRefs.current)
if (currentlyPlaying) {
audios.forEach(a => a.pause())
syncAnalysisAudio(currentTimeRef.current, false)
setVideoStartPhase('idle')
// Don't unregister — keep bar visible showing paused state
setPlaybackState(false)
@@ -2608,12 +2797,14 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
videoRef.current.play().catch((err) => console.warn('[LyricsLivePreview] video.play() failed (multi-stem)', err))
}
try { await ctx.resume() } catch { /* AudioContext suspended; audio starts on next user gesture */ }
syncAnalysisAudio(startTime, true)
setPlaybackState(true)
if (hasVideoBackdrop) setVideoStartPhase('playing')
}
} else if (audioRef.current) {
if (currentlyPlaying) {
audioRef.current.pause()
syncAnalysisAudio(currentTimeRef.current, false)
setVideoStartPhase('idle')
// Don't unregister — keep bar visible showing paused state
setPlaybackState(false)
@@ -2628,6 +2819,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
}
audioRef.current.currentTime = startTime
audioRef.current.play().catch(() => {})
syncAnalysisAudio(startTime, true)
// Publish immediately so the bar appears at once rather than waiting
// for the next effect cycle (avoids the clearMeta→publishMeta gap).
if (!disableGlobalPlaybackRegistry) {
@@ -2642,6 +2834,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
} else if (videoRef.current) {
if (currentlyPlaying) {
videoRef.current.pause()
syncAnalysisAudio(currentTimeRef.current, false)
setVideoStartPhase('idle')
setPlaybackState(false)
} else {
@@ -2654,6 +2847,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
if (isPlaylistBackdropMode) syncPlaylistVideoToTransport(startTime)
else videoRef.current.currentTime = startTime
videoRef.current.play().catch(() => {})
syncAnalysisAudio(startTime, true)
if (!disableGlobalPlaybackRegistry) {
publishMeta(
{ title: title ?? 'Live Preview', isPlaying: true, currentTime: startTime, duration, volume: isMuted ? 0 : volume },
@@ -2676,6 +2870,9 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
{ toggle: () => togglePlayRef.current(), seek: seekTime, setVolume: (v) => setVolumeFromBar.current(v) }
)
}
syncAnalysisAudio(startTime, true)
} else {
syncAnalysisAudio(currentTimeRef.current, false)
}
// Don't unregister on pause — keep bar visible
lastRafTime.current = null
@@ -2777,6 +2974,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
videoRef.current.currentTime = 0
videoRef.current.pause()
}
syncAnalysisAudio(0, false)
syncAnchorStemRef.current = null
lastAnchorTimeRef.current = null
isActiveRef.current = false
@@ -3670,7 +3868,19 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
/>
)}
{/* Hidden single-track audio element */}
{/* Hidden analysis audio element (always mounted when analysis source exists) */}
{(analysisAudioUrl || audioUrl) && (
<audio
ref={analysisAudioRef}
src={analysisAudioUrl ?? audioUrl}
preload="auto"
crossOrigin="anonymous"
muted
style={{ display: 'none' }}
/>
)}
{/* Hidden single-track transport audio element */}
{!isMultiStem && audioUrl && (
<audio
ref={audioRef}
+452 -220
View File
@@ -1,4 +1,4 @@
import { useCallback, useEffect, useRef, useState } from 'react'
import { useCallback, useEffect, useRef } from 'react'
import type { LyricsVisualStyle } from './LyricsLivePreview'
interface Stem {
@@ -39,19 +39,149 @@ interface VizSettings {
radius_multiplier: number
aperture: number
spacing: number
particle_count: number
rotation_speed: number
glow_intensity: number
particle_size: number
drum_reactivity: number
hue_range: number
beat_sensitivity: number
ellipse_ratio: number
idle_speed: number
noise_scale: number
noise_force: number
mono_color: boolean
divisions: number
lifespan: number
particle_mirror: boolean
dynamic_radius: number
delta_response_limit: number
visualization_preset: 'default' | 'orbital' | 'flower' | 'spiro' | 'hybrid'
}
fluid: {
spawn_density: number
splat_force: number
decay: number
background_persistence: number
beat_burst_count: number
treble_pulse: number
drag: number
blob_life: number
color_speed: number
}
}
const VIZ_DEFAULTS: VizSettings = {
quality: 'balanced',
fps_cap: 60,
muon: { radius_multiplier: 1, aperture: 1, spacing: 1 },
fluid: { spawn_density: 1, splat_force: 1, decay: 1 },
muon: {
radius_multiplier: 1.0,
aperture: 1.0,
spacing: 1.0,
particle_count: 220,
rotation_speed: 1.0,
glow_intensity: 1.0,
particle_size: 1.0,
drum_reactivity: 1.0,
hue_range: 1.0,
beat_sensitivity: 1.0,
ellipse_ratio: 0.62,
idle_speed: 1.0,
noise_scale: 1.0,
noise_force: 1.0,
mono_color: false,
divisions: 16,
lifespan: 180,
particle_mirror: false,
dynamic_radius: 1.0,
delta_response_limit: 0.005,
visualization_preset: 'hybrid',
},
fluid: {
spawn_density: 1.0,
splat_force: 1.0,
decay: 1.0,
background_persistence: 1.0,
beat_burst_count: 1.0,
treble_pulse: 1.0,
drag: 1.0,
blob_life: 1.0,
color_speed: 1.0,
},
}
// Named muon presets — applied to vizSettings.muon only (leaves other keys untouched).
const MUON_PRESETS: Array<{ label: string; values: VizSettings['muon'] }> = [
{
label: 'Default',
values: { radius_multiplier: 1.0, aperture: 1.0, spacing: 1.0, particle_count: 220, rotation_speed: 1.0, glow_intensity: 1.0, particle_size: 1.0, drum_reactivity: 1.0, hue_range: 1.0, beat_sensitivity: 1.0, ellipse_ratio: 0.62, idle_speed: 1.0, noise_scale: 1.0, noise_force: 1.0, mono_color: false, divisions: 16, lifespan: 180, particle_mirror: false, dynamic_radius: 1.0, delta_response_limit: 0.005, visualization_preset: 'hybrid' },
},
{
label: 'Calm',
values: { radius_multiplier: 1.0, aperture: 0.8, spacing: 1.2, particle_count: 120, rotation_speed: 0.5, glow_intensity: 0.4, particle_size: 0.7, drum_reactivity: 0.5, hue_range: 0.4, beat_sensitivity: 0.5, ellipse_ratio: 0.55, idle_speed: 0.6, noise_scale: 0.7, noise_force: 0.6, mono_color: false, divisions: 12, lifespan: 220, particle_mirror: false, dynamic_radius: 0.7, delta_response_limit: 0.007, visualization_preset: 'orbital' },
},
{
label: 'Energetic',
values: { radius_multiplier: 1.1, aperture: 1.3, spacing: 0.9, particle_count: 350, rotation_speed: 2.0, glow_intensity: 2.0, particle_size: 1.4, drum_reactivity: 1.8, hue_range: 1.8, beat_sensitivity: 2.0, ellipse_ratio: 0.68, idle_speed: 1.5, noise_scale: 1.4, noise_force: 1.5, mono_color: false, divisions: 24, lifespan: 160, particle_mirror: true, dynamic_radius: 1.6, delta_response_limit: 0.003, visualization_preset: 'hybrid' },
},
{
label: 'Dreamy',
values: { radius_multiplier: 1.2, aperture: 0.7, spacing: 1.5, particle_count: 280, rotation_speed: 0.6, glow_intensity: 2.5, particle_size: 1.1, drum_reactivity: 0.6, hue_range: 2.5, beat_sensitivity: 0.4, ellipse_ratio: 0.45, idle_speed: 0.8, noise_scale: 1.7, noise_force: 1.2, mono_color: false, divisions: 20, lifespan: 260, particle_mirror: false, dynamic_radius: 0.8, delta_response_limit: 0.008, visualization_preset: 'flower' },
},
{
label: 'Minimal',
values: { radius_multiplier: 0.8, aperture: 1.0, spacing: 2.0, particle_count: 60, rotation_speed: 0.8, glow_intensity: 0.0, particle_size: 0.6, drum_reactivity: 1.2, hue_range: 0.0, beat_sensitivity: 1.0, ellipse_ratio: 1.0, idle_speed: 0.5, noise_scale: 0.4, noise_force: 0.3, mono_color: true, divisions: 8, lifespan: 120, particle_mirror: false, dynamic_radius: 0.7, delta_response_limit: 0.012, visualization_preset: 'default' },
},
{
label: 'Cosmic',
values: { radius_multiplier: 1.3, aperture: 1.2, spacing: 0.7, particle_count: 400, rotation_speed: 1.5, glow_intensity: 1.8, particle_size: 1.8, drum_reactivity: 1.4, hue_range: 2.0, beat_sensitivity: 1.5, ellipse_ratio: 0.38, idle_speed: 1.2, noise_scale: 1.8, noise_force: 1.6, mono_color: false, divisions: 28, lifespan: 190, particle_mirror: true, dynamic_radius: 1.4, delta_response_limit: 0.004, visualization_preset: 'spiro' },
},
]
// Named fluid presets — applied to vizSettings.fluid only.
const FLUID_PRESETS: Array<{ label: string; values: VizSettings['fluid'] }> = [
{
label: 'Default',
values: { spawn_density: 1.0, splat_force: 1.0, decay: 1.0, background_persistence: 1.0, beat_burst_count: 1.0, treble_pulse: 1.0, drag: 1.0, blob_life: 1.0, color_speed: 1.0 },
},
{
label: 'Ink',
values: { spawn_density: 0.5, splat_force: 0.5, decay: 0.3, background_persistence: 3.0, beat_burst_count: 1.5, treble_pulse: 0.3, drag: 2.5, blob_life: 2.5, color_speed: 0.2 },
},
{
label: 'Foam',
values: { spawn_density: 3.0, splat_force: 0.3, decay: 3.0, background_persistence: 0.2, beat_burst_count: 3.0, treble_pulse: 2.5, drag: 0.4, blob_life: 0.3, color_speed: 3.0 },
},
{
label: 'Lava',
values: { spawn_density: 0.3, splat_force: 2.0, decay: 0.2, background_persistence: 3.5, beat_burst_count: 0.5, treble_pulse: 0.2, drag: 3.0, blob_life: 3.0, color_speed: 0.2 },
},
{
label: 'Mist',
values: { spawn_density: 1.5, splat_force: 0.2, decay: 2.0, background_persistence: 0.5, beat_burst_count: 0.5, treble_pulse: 1.5, drag: 0.5, blob_life: 0.5, color_speed: 0.8 },
},
{
label: 'Storm',
values: { spawn_density: 2.5, splat_force: 3.0, decay: 1.0, background_persistence: 1.0, beat_burst_count: 3.0, treble_pulse: 2.5, drag: 0.4, blob_life: 1.0, color_speed: 2.5 },
},
]
const EPS = 0.015
function muonPresetMatch(current: VizSettings['muon'], preset: VizSettings['muon']): boolean {
const keys = Object.keys(preset) as Array<keyof VizSettings['muon']>
return keys.every((k) => {
const a = current[k]
const b = preset[k]
if (typeof a === 'boolean') return a === b
if (typeof a === 'string') return a === b
return Math.abs((a as number) - (b as number)) < EPS
})
}
function fluidPresetMatch(current: VizSettings['fluid'], preset: VizSettings['fluid']): boolean {
const keys = Object.keys(preset) as Array<keyof VizSettings['fluid']>
return keys.every((k) => Math.abs(current[k] - preset[k]) < EPS)
}
const FONT_OPTIONS = [
@@ -124,12 +254,6 @@ export default function PerformanceSidebar({
patchVizSettings,
onClose,
}: PerformanceSidebarProps) {
const [open, setOpen] = useState({
background: true,
stems: true,
lyrics: true,
visualizer: true,
})
const autoHideTimerRef = useRef<number | null>(null)
const scheduleAutoHide = useCallback(() => {
@@ -170,24 +294,40 @@ export default function PerformanceSidebar({
patchStyle(axis === 'x' ? { layoutOffsetXScroll: value } : { layoutOffsetYScroll: value })
}
const sectionHeader = (key: keyof typeof open, label: string, hidden = false) => {
if (hidden) return null
return (
<button
type="button"
onClick={() => setOpen((prev) => ({ ...prev, [key]: !prev[key] }))}
className="w-full flex items-center justify-between text-xs font-semibold tracking-wide text-slate-200"
>
<span>{label}</span>
<span className="text-slate-400">{open[key] ? '▾' : '▸'}</span>
</button>
)
}
// Preset detection
const activeMuonPreset = MUON_PRESETS.find((p) => muonPresetMatch(vizSettings.muon, p.values))
const activeFluidPreset = FLUID_PRESETS.find((p) => fluidPresetMatch(vizSettings.fluid, p.values))
const panelClass = isNarrow
? 'absolute inset-x-0 bottom-0 h-[62vh] rounded-t-2xl border-t border-white/15'
: 'absolute right-0 top-0 h-full border-l border-white/15'
const slider = (
label: string,
value: number,
displayValue: string,
min: number,
max: number,
step: number,
onChange: (v: number) => void,
) => (
<div className="space-y-1">
<div className="flex items-center justify-between text-[10px] text-slate-400">
<span>{label}</span>
<span>{displayValue}</span>
</div>
<input
type="range"
min={min}
max={max}
step={step}
value={value}
onChange={(e) => onChange(parseFloat(e.target.value))}
className="w-full h-1.5 accent-brand-500"
/>
</div>
)
return (
<div
className={`${panelClass} z-[9] bg-black/70 backdrop-blur-md text-slate-100 overflow-y-auto`}
@@ -208,236 +348,328 @@ export default function PerformanceSidebar({
</div>
<div className="px-3 py-3 space-y-4">
{/* ── BACKGROUND ── */}
<section className="space-y-2">
{sectionHeader('background', 'BACKGROUND')}
{open.background && (
<div className={isNarrow ? 'flex flex-wrap gap-2' : 'grid grid-cols-2 gap-2'}>
{[
{ key: 'static', label: 'Static', value: 'karaoke' as LyricsVisualStyle, active: isStaticMode },
{ key: 'muon', label: 'Muon', value: 'muon' as LyricsVisualStyle, active: visualStyle === 'muon' },
{ key: 'fluid', label: 'Fluid', value: 'fluid' as LyricsVisualStyle, active: visualStyle === 'fluid' },
...(hasVideo ? [{ key: 'video', label: 'Video', value: 'video' as LyricsVisualStyle, active: visualStyle === 'video' }] : []),
].map((mode) => (
<button
key={mode.key}
type="button"
onClick={() => setVisualStyle(mode.value)}
className={`rounded-md px-2 py-2 text-xs border transition-colors ${isNarrow ? 'min-w-[70px] flex-1' : ''} ${mode.active ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
<div className="text-xs font-semibold tracking-wide text-slate-200">BACKGROUND</div>
<div className={isNarrow ? 'flex flex-wrap gap-2' : 'grid grid-cols-2 gap-2'}>
{[
{ key: 'static', label: 'Static', value: 'karaoke' as LyricsVisualStyle, active: isStaticMode },
{ key: 'muon', label: 'Muon', value: 'muon' as LyricsVisualStyle, active: visualStyle === 'muon' },
{ key: 'fluid', label: 'Fluid', value: 'fluid' as LyricsVisualStyle, active: visualStyle === 'fluid' },
...(hasVideo ? [{ key: 'video', label: 'Video', value: 'video' as LyricsVisualStyle, active: visualStyle === 'video' }] : []),
].map((mode) => (
<button
key={mode.key}
type="button"
onClick={() => setVisualStyle(mode.value)}
className={`rounded-md px-2 py-2 text-xs border transition-colors ${isNarrow ? 'min-w-[70px] flex-1' : ''} ${mode.active ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
>
{mode.label}
</button>
))}
</div>
</section>
{/* ── STEMS ── */}
<section className="space-y-2">
<div className="text-xs font-semibold tracking-wide text-slate-200">STEMS</div>
<div className="space-y-2">
{stems?.length ? stems.map((stem) => (
<div key={stem.name} className="space-y-1.5">
<div className="flex items-center gap-2">
<span className="text-xs font-medium text-slate-300 flex-1 truncate">{stem.label}</span>
<button
type="button"
onClick={() => toggleStemMute(stem.name)}
className={`w-6 h-6 text-[10px] rounded ${mutedStems[stem.name] ? 'bg-red-500 text-white' : 'bg-white/10 text-slate-300'}`}
>
M
</button>
<button
type="button"
onClick={() => toggleStemSolo(stem.name)}
className={`w-6 h-6 text-[10px] rounded ${soloedStems[stem.name] ? 'bg-yellow-300 text-black' : 'bg-white/10 text-slate-300'}`}
>
S
</button>
</div>
<div className="flex items-center gap-2">
<input
type="range"
min={0}
max={2}
step={0.05}
value={gains[stem.name] ?? 1}
onChange={(e) => setStemGain(stem.name, parseFloat(e.target.value))}
className="w-full h-1.5 accent-brand-500"
/>
<span className="text-[10px] tabular-nums text-slate-400 w-11 text-right">{Math.round((gains[stem.name] ?? 1) * 100)}%</span>
</div>
</div>
)) : (
<p className="text-[11px] text-slate-400">No stems — single track mode</p>
)}
</div>
</section>
{/* ── LYRICS ── */}
<section className="space-y-2">
<div className="text-xs font-semibold tracking-wide text-slate-200">LYRICS</div>
<div className="space-y-2.5">
<div className="flex items-center gap-2">
<select
value={style.fontFamily}
onChange={(e) => patchStyle({ fontFamily: e.target.value })}
className="flex-1 text-xs bg-white/10 border border-white/15 rounded px-2 py-1"
>
{FONT_OPTIONS.map((font) => (
<option key={font.value} value={font.value} style={{ background: '#0f172a', color: '#e2e8f0', fontFamily: font.value }}>
{font.label}
</option>
))}
</select>
<button
type="button"
onClick={() => patchStyle({ fontBold: !style.fontBold })}
className={`px-2 py-1 rounded text-xs font-semibold ${style.fontBold ? 'bg-white/25 text-white' : 'bg-white/10 text-slate-300'}`}
>
B
</button>
</div>
<div className="space-y-1">
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Size</span><span>{style.fontSize}</span></div>
<input type="range" min={12} max={56} step={1} value={style.fontSize} onChange={(e) => patchStyle({ fontSize: parseInt(e.target.value, 10) })} className="w-full h-1.5 accent-brand-500" />
</div>
<div className="flex items-center gap-2">
{(['left', 'center', 'right'] as const).map((align) => (
<button key={align} type="button" onClick={() => patchStyle({ textAlign: align })}
className={`flex-1 rounded px-2 py-1 text-xs ${style.textAlign === align ? 'bg-white/20 text-white' : 'bg-white/10 text-slate-300'}`}
>
{mode.label}
{align === 'left' ? 'L' : align === 'center' ? 'C' : 'R'}
</button>
))}
</div>
)}
</section>
<section className="space-y-2">
{sectionHeader('stems', 'STEMS')}
{open.stems && (
<div className="space-y-2">
{stems?.length ? stems.map((stem) => (
<div key={stem.name} className="space-y-1.5">
<div className="flex items-center gap-2">
<span className="text-xs font-medium text-slate-300 flex-1 truncate">{stem.label}</span>
<button
type="button"
onClick={() => toggleStemMute(stem.name)}
className={`w-6 h-6 text-[10px] rounded ${mutedStems[stem.name] ? 'bg-red-500 text-white' : 'bg-white/10 text-slate-300'}`}
>
M
</button>
<button
type="button"
onClick={() => toggleStemSolo(stem.name)}
className={`w-6 h-6 text-[10px] rounded ${soloedStems[stem.name] ? 'bg-yellow-300 text-black' : 'bg-white/10 text-slate-300'}`}
>
S
</button>
</div>
<div className="flex items-center gap-2">
<input
type="range"
min={0}
max={2}
step={0.05}
value={gains[stem.name] ?? 1}
onChange={(event) => setStemGain(stem.name, parseFloat(event.target.value))}
className="w-full h-1.5 accent-brand-500"
/>
<span className="text-[10px] tabular-nums text-slate-400 w-11 text-right">{Math.round((gains[stem.name] ?? 1) * 100)}%</span>
</div>
</div>
)) : (
<p className="text-[11px] text-slate-400">No stems — single track mode</p>
)}
</div>
)}
</section>
<section className="space-y-2">
{sectionHeader('lyrics', 'LYRICS')}
{open.lyrics && (
<div className="space-y-2.5">
<div className="flex items-center gap-2">
<select
value={style.fontFamily}
onChange={(event) => patchStyle({ fontFamily: event.target.value })}
className="flex-1 text-xs bg-white/10 border border-white/15 rounded px-2 py-1"
<div className="flex items-center gap-2">
{(['top', 'center', 'bottom'] as const).map((pos) => (
<button key={pos} type="button" onClick={() => patchStyle({ activeLineVerticalAlign: pos })}
className={`flex-1 rounded px-2 py-1 text-xs ${style.activeLineVerticalAlign === pos ? 'bg-white/20 text-white' : 'bg-white/10 text-slate-300'}`}
>
{FONT_OPTIONS.map((font) => (
<option key={font.value} value={font.value} style={{ background: '#0f172a', color: '#e2e8f0', fontFamily: font.value }}>
{font.label}
</option>
))}
</select>
<button
type="button"
onClick={() => patchStyle({ fontBold: !style.fontBold })}
className={`px-2 py-1 rounded text-xs font-semibold ${style.fontBold ? 'bg-white/25 text-white' : 'bg-white/10 text-slate-300'}`}
>
B
{pos === 'top' ? 'T' : pos === 'center' ? 'C' : 'B'}
</button>
</div>
))}
</div>
<div className="space-y-1">
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Size</span><span>{style.fontSize}</span></div>
<input type="range" min={12} max={56} step={1} value={style.fontSize} onChange={(event) => patchStyle({ fontSize: parseInt(event.target.value, 10) })} className="w-full h-1.5 accent-brand-500" />
</div>
<div className="space-y-1">
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Line spacing</span><span>{style.lineSpacing.toFixed(2)}</span></div>
<input type="range" min={1} max={2.5} step={0.05} value={style.lineSpacing} onChange={(e) => patchStyle({ lineSpacing: parseFloat(e.target.value) })} className="w-full h-1.5 accent-brand-500" />
</div>
<div className="flex items-center gap-2">
{(['left', 'center', 'right'] as const).map((align) => (
<button
key={align}
type="button"
onClick={() => patchStyle({ textAlign: align })}
className={`flex-1 rounded px-2 py-1 text-xs ${style.textAlign === align ? 'bg-white/20 text-white' : 'bg-white/10 text-slate-300'}`}
<div className="grid grid-cols-3 gap-2 text-[10px] text-slate-300">
<label className="space-y-1"><span className="block">Text</span><input type="color" value={style.textColor} onChange={(e) => patchStyle({ textColor: e.target.value })} className="w-full h-7 rounded border border-white/15 bg-transparent" /></label>
<label className="space-y-1"><span className="block">Highlight</span><input type="color" value={style.highlightColor} onChange={(e) => patchStyle({ highlightColor: e.target.value })} className="w-full h-7 rounded border border-white/15 bg-transparent" /></label>
<label className="space-y-1"><span className="block">BG</span><input type="color" value={style.bgColor} onChange={(e) => patchStyle({ bgColor: e.target.value })} className="w-full h-7 rounded border border-white/15 bg-transparent" /></label>
</div>
<div className="space-y-1">
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>X offset</span><span>{activeOffset.x}</span></div>
<input type="range" min={-50} max={50} step={1} value={activeOffset.x} onChange={(e) => patchOffset('x', parseInt(e.target.value, 10))} className="w-full h-1.5 accent-brand-500" />
</div>
<div className="space-y-1">
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Y offset</span><span>{activeOffset.y}</span></div>
<input type="range" min={-50} max={50} step={1} value={activeOffset.y} onChange={(e) => patchOffset('y', parseInt(e.target.value, 10))} className="w-full h-1.5 accent-brand-500" />
</div>
</div>
</section>
{/* ── VISUALIZER ── */}
<section className="space-y-2">
<div className="text-xs font-semibold tracking-wide text-slate-200">VISUALIZER</div>
<div className="space-y-2.5">
{/* Quality */}
<div className="space-y-1.5">
<div className="text-[10px] text-slate-400">Quality</div>
<div className="grid grid-cols-3 gap-1.5">
{(['low', 'balanced', 'high'] as const).map((q) => (
<button key={q} type="button" onClick={() => patchVizSettings({ quality: q })}
className={`rounded px-2 py-1 text-[11px] border ${vizSettings.quality === q ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
>
{align === 'left' ? 'L' : align === 'center' ? 'C' : 'R'}
{q === 'balanced' ? 'Balanced' : q === 'low' ? 'Low' : 'High'}
</button>
))}
</div>
</div>
<div className="flex items-center gap-2">
{(['top', 'center', 'bottom'] as const).map((pos) => (
<button
key={pos}
type="button"
onClick={() => patchStyle({ activeLineVerticalAlign: pos })}
className={`flex-1 rounded px-2 py-1 text-xs ${style.activeLineVerticalAlign === pos ? 'bg-white/20 text-white' : 'bg-white/10 text-slate-300'}`}
{/* FPS cap */}
<div className="space-y-1.5">
<div className="text-[10px] text-slate-400">FPS cap</div>
<div className="grid grid-cols-3 gap-1.5">
{([30, 60, 120] as const).map((fps) => (
<button key={fps} type="button" onClick={() => patchVizSettings({ fps_cap: fps })}
className={`rounded px-2 py-1 text-[11px] border ${vizSettings.fps_cap === fps ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
>
{pos === 'top' ? 'T' : pos === 'center' ? 'C' : 'B'}
{fps}
</button>
))}
</div>
<div className="space-y-1">
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Line spacing</span><span>{style.lineSpacing.toFixed(2)}</span></div>
<input type="range" min={1} max={2.5} step={0.05} value={style.lineSpacing} onChange={(event) => patchStyle({ lineSpacing: parseFloat(event.target.value) })} className="w-full h-1.5 accent-brand-500" />
</div>
<div className="grid grid-cols-3 gap-2 text-[10px] text-slate-300">
<label className="space-y-1"><span className="block">Text</span><input type="color" value={style.textColor} onChange={(event) => patchStyle({ textColor: event.target.value })} className="w-full h-7 rounded border border-white/15 bg-transparent" /></label>
<label className="space-y-1"><span className="block">Highlight</span><input type="color" value={style.highlightColor} onChange={(event) => patchStyle({ highlightColor: event.target.value })} className="w-full h-7 rounded border border-white/15 bg-transparent" /></label>
<label className="space-y-1"><span className="block">BG</span><input type="color" value={style.bgColor} onChange={(event) => patchStyle({ bgColor: event.target.value })} className="w-full h-7 rounded border border-white/15 bg-transparent" /></label>
</div>
<div className="space-y-1">
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>X offset</span><span>{activeOffset.x}</span></div>
<input type="range" min={-50} max={50} step={1} value={activeOffset.x} onChange={(event) => patchOffset('x', parseInt(event.target.value, 10))} className="w-full h-1.5 accent-brand-500" />
</div>
<div className="space-y-1">
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Y offset</span><span>{activeOffset.y}</span></div>
<input type="range" min={-50} max={50} step={1} value={activeOffset.y} onChange={(event) => patchOffset('y', parseInt(event.target.value, 10))} className="w-full h-1.5 accent-brand-500" />
</div>
</div>
)}
</section>
<section className="space-y-2">
{sectionHeader('visualizer', 'VISUALIZER')}
{open.visualizer && (
<div className="space-y-2.5">
{!isVisualizerMode && (
<p className="text-[11px] text-slate-400">
Switch background to Muon or Fluid to access mode-specific controls.
</p>
)}
{/* ── Muon controls ── */}
{visualStyle === 'muon' && (
<>
{/* Preset pills */}
<div className="space-y-1.5">
<div className="text-[10px] text-slate-400">Quality</div>
<div className="grid grid-cols-3 gap-1.5">
{(['low', 'balanced', 'high'] as const).map((quality) => (
<button
key={quality}
type="button"
onClick={() => patchVizSettings({ quality })}
className={`rounded px-2 py-1 text-[11px] border ${vizSettings.quality === quality ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
>
{quality === 'balanced' ? 'Balanced' : quality === 'low' ? 'Low' : 'High'}
</button>
))}
<div className="text-[10px] text-slate-400">Preset</div>
<div className="flex flex-wrap gap-1.5">
{MUON_PRESETS.map((preset) => {
const isActive = activeMuonPreset?.label === preset.label
return (
<button
key={preset.label}
type="button"
onClick={() => patchVizSettings({ muon: preset.values })}
className={`rounded px-2 py-1 text-[11px] border transition-colors ${isActive ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
>
{preset.label}
</button>
)
})}
{!activeMuonPreset && (
<span className="rounded px-2 py-1 text-[11px] border border-amber-400/60 bg-amber-500/10 text-amber-300">
Custom
</span>
)}
</div>
</div>
{/* Sliders */}
{slider('Radius', vizSettings.muon.radius_multiplier, formatMultiplier(vizSettings.muon.radius_multiplier), 0.25, 2, 0.05, (v) => patchVizSettings({ muon: { radius_multiplier: v } }))}
{slider('Aperture', vizSettings.muon.aperture, formatMultiplier(vizSettings.muon.aperture), 0.25, 2, 0.05, (v) => patchVizSettings({ muon: { aperture: v } }))}
{slider('Spacing', vizSettings.muon.spacing, formatMultiplier(vizSettings.muon.spacing), 0.25, 2, 0.05, (v) => patchVizSettings({ muon: { spacing: v } }))}
{slider('Particles', vizSettings.muon.particle_count, String(vizSettings.muon.particle_count), 40, 500, 10, (v) => patchVizSettings({ muon: { particle_count: Math.round(v) } }))}
{slider('Rotation', vizSettings.muon.rotation_speed, formatMultiplier(vizSettings.muon.rotation_speed), 0.2, 4, 0.1, (v) => patchVizSettings({ muon: { rotation_speed: v } }))}
{slider('Glow', vizSettings.muon.glow_intensity, formatMultiplier(vizSettings.muon.glow_intensity), 0, 3, 0.1, (v) => patchVizSettings({ muon: { glow_intensity: v } }))}
{slider('Size', vizSettings.muon.particle_size, formatMultiplier(vizSettings.muon.particle_size), 0.25, 3, 0.05, (v) => patchVizSettings({ muon: { particle_size: v } }))}
{slider('Reactivity', vizSettings.muon.drum_reactivity, formatMultiplier(vizSettings.muon.drum_reactivity), 0, 2, 0.1, (v) => patchVizSettings({ muon: { drum_reactivity: v } }))}
{slider('Hue range', vizSettings.muon.hue_range, formatMultiplier(vizSettings.muon.hue_range), 0, 3, 0.1, (v) => patchVizSettings({ muon: { hue_range: v } }))}
{slider('Beat sensitivity', vizSettings.muon.beat_sensitivity, formatMultiplier(vizSettings.muon.beat_sensitivity), 0.2, 3, 0.1, (v) => patchVizSettings({ muon: { beat_sensitivity: v } }))}
{slider('Orbit shape', vizSettings.muon.ellipse_ratio, vizSettings.muon.ellipse_ratio.toFixed(2), 0.3, 1, 0.02, (v) => patchVizSettings({ muon: { ellipse_ratio: v } }))}
{slider('Idle speed', vizSettings.muon.idle_speed, formatMultiplier(vizSettings.muon.idle_speed), 0, 2, 0.1, (v) => patchVizSettings({ muon: { idle_speed: v } }))}
{slider('Divisions', vizSettings.muon.divisions, String(vizSettings.muon.divisions), 8, 48, 1, (v) => patchVizSettings({ muon: { divisions: Math.round(v) } }))}
{slider('Lifespan', vizSettings.muon.lifespan, String(vizSettings.muon.lifespan), 45, 420, 5, (v) => patchVizSettings({ muon: { lifespan: Math.round(v) } }))}
{slider('Dynamic radius', vizSettings.muon.dynamic_radius, formatMultiplier(vizSettings.muon.dynamic_radius), 0.2, 3, 0.05, (v) => patchVizSettings({ muon: { dynamic_radius: v } }))}
{slider('Delta gate', vizSettings.muon.delta_response_limit, vizSettings.muon.delta_response_limit.toFixed(3), 0.001, 0.05, 0.001, (v) => patchVizSettings({ muon: { delta_response_limit: v } }))}
<div className="space-y-1.5">
<div className="text-[10px] text-slate-400">FPS cap</div>
<div className="grid grid-cols-3 gap-1.5">
{([30, 60, 120] as const).map((fps) => (
<button
key={fps}
type="button"
onClick={() => patchVizSettings({ fps_cap: fps })}
className={`rounded px-2 py-1 text-[11px] border ${vizSettings.fps_cap === fps ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
>
{fps}
</button>
))}
<div className="text-[10px] text-slate-400">Pattern preset</div>
<div className="flex flex-wrap gap-1.5">
{(['default', 'orbital', 'flower', 'spiro', 'hybrid'] as const).map((preset) => {
const active = vizSettings.muon.visualization_preset === preset
return (
<button
key={preset}
type="button"
onClick={() => patchVizSettings({ muon: { visualization_preset: preset } })}
className={`rounded px-2 py-1 text-[11px] border transition-colors ${active ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
>
{preset}
</button>
)
})}
</div>
</div>
{!isVisualizerMode && (
<p className="text-[11px] text-slate-400">
Switch background to Muon or Fluid to access mode-specific controls.
</p>
)}
{/* Mono color toggle */}
<label className="flex items-center gap-2 cursor-pointer">
<div
role="checkbox"
aria-checked={vizSettings.muon.mono_color}
tabIndex={0}
onClick={() => patchVizSettings({ muon: { mono_color: !vizSettings.muon.mono_color } })}
onKeyDown={(e) => { if (e.key === ' ' || e.key === 'Enter') patchVizSettings({ muon: { mono_color: !vizSettings.muon.mono_color } }) }}
className={`w-8 h-4 rounded-full transition-colors ${vizSettings.muon.mono_color ? 'bg-violet-500' : 'bg-white/20'} relative`}
>
<div className={`absolute top-0.5 w-3 h-3 rounded-full bg-white transition-transform ${vizSettings.muon.mono_color ? 'translate-x-4' : 'translate-x-0.5'}`} />
</div>
<span className="text-[11px] text-slate-300">Mono color</span>
</label>
{visualStyle === 'muon' && (
<>
<div className="space-y-1">
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Radius</span><span>{formatMultiplier(vizSettings.muon.radius_multiplier)}</span></div>
<input type="range" min={0.25} max={2} step={0.05} value={vizSettings.muon.radius_multiplier} onChange={(event) => patchVizSettings({ muon: { radius_multiplier: parseFloat(event.target.value) } })} className="w-full h-1.5 accent-brand-500" />
</div>
<div className="space-y-1">
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Aperture</span><span>{formatMultiplier(vizSettings.muon.aperture)}</span></div>
<input type="range" min={0.25} max={2} step={0.05} value={vizSettings.muon.aperture} onChange={(event) => patchVizSettings({ muon: { aperture: parseFloat(event.target.value) } })} className="w-full h-1.5 accent-brand-500" />
</div>
<div className="space-y-1">
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Spacing</span><span>{formatMultiplier(vizSettings.muon.spacing)}</span></div>
<input type="range" min={0.25} max={2} step={0.05} value={vizSettings.muon.spacing} onChange={(event) => patchVizSettings({ muon: { spacing: parseFloat(event.target.value) } })} className="w-full h-1.5 accent-brand-500" />
</div>
</>
)}
{/* Mirror toggle */}
<label className="flex items-center gap-2 cursor-pointer">
<div
role="checkbox"
aria-checked={vizSettings.muon.particle_mirror}
tabIndex={0}
onClick={() => patchVizSettings({ muon: { particle_mirror: !vizSettings.muon.particle_mirror } })}
onKeyDown={(e) => { if (e.key === ' ' || e.key === 'Enter') patchVizSettings({ muon: { particle_mirror: !vizSettings.muon.particle_mirror } }) }}
className={`w-8 h-4 rounded-full transition-colors ${vizSettings.muon.particle_mirror ? 'bg-violet-500' : 'bg-white/20'} relative`}
>
<div className={`absolute top-0.5 w-3 h-3 rounded-full bg-white transition-transform ${vizSettings.muon.particle_mirror ? 'translate-x-4' : 'translate-x-0.5'}`} />
</div>
<span className="text-[11px] text-slate-300">Mirror particles</span>
</label>
</>
)}
{visualStyle === 'fluid' && (
<>
<div className="space-y-1">
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Density</span><span>{formatMultiplier(vizSettings.fluid.spawn_density)}</span></div>
<input type="range" min={0.25} max={2} step={0.05} value={vizSettings.fluid.spawn_density} onChange={(event) => patchVizSettings({ fluid: { spawn_density: parseFloat(event.target.value) } })} className="w-full h-1.5 accent-brand-500" />
</div>
<div className="space-y-1">
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Force</span><span>{formatMultiplier(vizSettings.fluid.splat_force)}</span></div>
<input type="range" min={0.25} max={2} step={0.05} value={vizSettings.fluid.splat_force} onChange={(event) => patchVizSettings({ fluid: { splat_force: parseFloat(event.target.value) } })} className="w-full h-1.5 accent-brand-500" />
</div>
<div className="space-y-1">
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Trail decay</span><span>{formatMultiplier(vizSettings.fluid.decay)}</span></div>
<input type="range" min={0.25} max={2} step={0.05} value={vizSettings.fluid.decay} onChange={(event) => patchVizSettings({ fluid: { decay: parseFloat(event.target.value) } })} className="w-full h-1.5 accent-brand-500" />
</div>
</>
)}
{/* ── Fluid controls ── */}
{visualStyle === 'fluid' && (
<>
{/* Preset pills */}
<div className="space-y-1.5">
<div className="text-[10px] text-slate-400">Preset</div>
<div className="flex flex-wrap gap-1.5">
{FLUID_PRESETS.map((preset) => {
const isActive = activeFluidPreset?.label === preset.label
return (
<button
key={preset.label}
type="button"
onClick={() => patchVizSettings({ fluid: preset.values })}
className={`rounded px-2 py-1 text-[11px] border transition-colors ${isActive ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
>
{preset.label}
</button>
)
})}
{!activeFluidPreset && (
<span className="rounded px-2 py-1 text-[11px] border border-amber-400/60 bg-amber-500/10 text-amber-300">
Custom
</span>
)}
</div>
</div>
<button
type="button"
onClick={() => patchVizSettings(manifestVizSettings ?? VIZ_DEFAULTS)}
className="w-full rounded-md px-2 py-1.5 text-xs bg-white/10 hover:bg-white/20 text-slate-200"
>
{manifestVizSettings ? 'Reset to song defaults' : 'Reset to defaults'}
</button>
</div>
)}
{/* Sliders */}
{slider('Density', vizSettings.fluid.spawn_density, formatMultiplier(vizSettings.fluid.spawn_density), 0.25, 3, 0.05, (v) => patchVizSettings({ fluid: { spawn_density: v } }))}
{slider('Force', vizSettings.fluid.splat_force, formatMultiplier(vizSettings.fluid.splat_force), 0.25, 3, 0.05, (v) => patchVizSettings({ fluid: { splat_force: v } }))}
{slider('Trail decay', vizSettings.fluid.decay, formatMultiplier(vizSettings.fluid.decay), 0.25, 3, 0.05, (v) => patchVizSettings({ fluid: { decay: v } }))}
{slider('Trail length', vizSettings.fluid.background_persistence, formatMultiplier(vizSettings.fluid.background_persistence), 0.2, 3, 0.1, (v) => patchVizSettings({ fluid: { background_persistence: v } }))}
{slider('Beat bursts', vizSettings.fluid.beat_burst_count, formatMultiplier(vizSettings.fluid.beat_burst_count), 0.25, 3, 0.05, (v) => patchVizSettings({ fluid: { beat_burst_count: v } }))}
{slider('Treble pulse', vizSettings.fluid.treble_pulse, formatMultiplier(vizSettings.fluid.treble_pulse), 0, 3, 0.1, (v) => patchVizSettings({ fluid: { treble_pulse: v } }))}
{slider('Drag', vizSettings.fluid.drag, formatMultiplier(vizSettings.fluid.drag), 0.5, 2, 0.05, (v) => patchVizSettings({ fluid: { drag: v } }))}
{slider('Blob life', vizSettings.fluid.blob_life, formatMultiplier(vizSettings.fluid.blob_life), 0.25, 3, 0.05, (v) => patchVizSettings({ fluid: { blob_life: v } }))}
{slider('Color speed', vizSettings.fluid.color_speed, formatMultiplier(vizSettings.fluid.color_speed), 0, 3, 0.1, (v) => patchVizSettings({ fluid: { color_speed: v } }))}
</>
)}
<button
type="button"
onClick={() => patchVizSettings(manifestVizSettings ?? VIZ_DEFAULTS)}
className="w-full rounded-md px-2 py-1.5 text-xs bg-white/10 hover:bg-white/20 text-slate-200"
>
{manifestVizSettings ? 'Reset to song defaults' : 'Reset to defaults'}
</button>
</div>
</section>
</div>
</div>
@@ -0,0 +1,304 @@
import { useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react'
import { Disc3 } from 'lucide-react'
import { useAudioFeatures } from '../hooks/useAudioFeatures'
import { useVisualizerPalette } from '../hooks/useVisualizerPalette'
import VisualizerBackdrop, { type VisualizerMode } from './visualizers/VisualizerBackdrop'
import { registerPlayback, unregisterPlayback } from '../utils/audioRegistry'
interface VisualizerPreviewWidgetProps {
audioUrl: string
coverArtUrl?: string
videoUrl?: string
songTitle?: string
className?: string
}
type PreviewMode = 'muon' | 'fluid' | 'video'
function isDirectVideoUrl(input: string): boolean {
return /^(https?:\/\/).+\.(mp4|webm|m4v|ogg|ogv)(?:[?#].*)?$/i.test(input)
}
export default function VisualizerPreviewWidget({
audioUrl,
coverArtUrl,
videoUrl,
songTitle,
className,
}: VisualizerPreviewWidgetProps) {
const containerRef = useRef<HTMLDivElement>(null)
const audioRef = useRef<HTMLAudioElement>(null)
const videoRef = useRef<HTMLVideoElement>(null)
const audioCtxRef = useRef<AudioContext | null>(null)
const isPlayingRef = useRef(false)
const [dims, setDims] = useState<{ w: number; h: number } | null>(null)
const [mode, setMode] = useState<PreviewMode>('muon')
const [isPlaying, setIsPlaying] = useState(false)
const [ctxVersion, setCtxVersion] = useState(0)
const normalizedVideoUrl = useMemo(() => String(videoUrl || '').trim(), [videoUrl])
const canPreviewVideo = useMemo(() => isDirectVideoUrl(normalizedVideoUrl), [normalizedVideoUrl])
const paletteRef = useVisualizerPalette({ coverArtUrl })
const featuresRef = useAudioFeatures({
audioCtxRef,
analysisAudioRef: audioRef,
isPlayingRef,
enabled: true,
analysisSourceKey: `${audioUrl}|${ctxVersion}`,
})
const tabs = useMemo(
() => [
{ id: 'muon' as const, label: 'Muon' },
{ id: 'fluid' as const, label: 'Fluid' },
...(canPreviewVideo ? [{ id: 'video' as const, label: 'Video' }] : []),
],
[canPreviewVideo],
)
const ensureCtx = () => {
if (!audioCtxRef.current || audioCtxRef.current.state === 'closed') {
audioCtxRef.current = new AudioContext()
setCtxVersion((prev) => prev + 1)
}
return audioCtxRef.current
}
const stopPlayback = useCallback(() => {
const audio = audioRef.current
if (!audio) return
audio.pause()
setIsPlaying(false)
}, [])
useEffect(() => {
isPlayingRef.current = isPlaying
}, [isPlaying])
useLayoutEffect(() => {
const el = containerRef.current
if (!el) return
const measure = () => {
const { width } = el.getBoundingClientRect()
if (width <= 0) return
const next = {
w: Math.round(width),
h: Math.round(width * (9 / 16)),
}
setDims((prev) => (prev && prev.w === next.w && prev.h === next.h ? prev : next))
}
measure()
if (typeof ResizeObserver === 'undefined') {
window.addEventListener('resize', measure)
return () => window.removeEventListener('resize', measure)
}
const ro = new ResizeObserver(measure)
ro.observe(el)
return () => ro.disconnect()
}, [])
useEffect(() => {
const audio = audioRef.current
if (!audio) return
const onPlay = () => setIsPlaying(true)
const onPause = () => setIsPlaying(false)
const onEnded = () => setIsPlaying(false)
audio.addEventListener('play', onPlay)
audio.addEventListener('pause', onPause)
audio.addEventListener('ended', onEnded)
return () => {
audio.removeEventListener('play', onPlay)
audio.removeEventListener('pause', onPause)
audio.removeEventListener('ended', onEnded)
}
}, [])
useEffect(() => {
setIsPlaying(false)
isPlayingRef.current = false
audioRef.current?.pause()
const video = videoRef.current
if (video) {
video.pause()
try {
video.currentTime = 0
} catch {
// ignore
}
}
}, [audioUrl])
useEffect(() => {
return () => {
unregisterPlayback(stopPlayback)
}
}, [stopPlayback])
useEffect(() => {
if (mode === 'video' && !canPreviewVideo) setMode('muon')
}, [mode, canPreviewVideo])
useEffect(() => {
const onVisible = () => {
if (document.hidden) return
if (isPlayingRef.current && audioCtxRef.current?.state === 'suspended') {
void audioCtxRef.current.resume().catch(() => {})
}
}
document.addEventListener('visibilitychange', onVisible)
return () => document.removeEventListener('visibilitychange', onVisible)
}, [])
useEffect(() => {
const video = videoRef.current
const audio = audioRef.current
if (!video) return
if (mode !== 'video') {
video.pause()
return
}
if (audio && Math.abs(video.currentTime - audio.currentTime) > 0.15) {
try {
video.currentTime = audio.currentTime
} catch {
// ignore
}
}
if (isPlaying) {
void video.play().catch(() => {})
} else {
video.pause()
}
}, [isPlaying, mode])
useEffect(() => {
if (mode !== 'video' || !isPlaying) return
const timer = window.setInterval(() => {
const video = videoRef.current
const audio = audioRef.current
if (!video || !audio) return
if (Math.abs(video.currentTime - audio.currentTime) > 0.25) {
try {
video.currentTime = audio.currentTime
} catch {
// ignore
}
}
}, 500)
return () => window.clearInterval(timer)
}, [isPlaying, mode])
useEffect(() => {
return () => {
const ctx = audioCtxRef.current
if (ctx && ctx.state !== 'closed') {
void ctx.close().catch(() => {})
}
audioCtxRef.current = null
}
}, [])
const togglePlay = async () => {
const audio = audioRef.current
if (!audio) return
if (isPlayingRef.current) {
audio.pause()
return
}
const ctx = ensureCtx()
if (ctx.state === 'suspended') {
await ctx.resume().catch(() => {})
}
try {
registerPlayback(stopPlayback)
await audio.play()
setIsPlaying(true)
} catch {
setIsPlaying(false)
}
}
const visualizerMode: VisualizerMode = mode === 'video' ? 'muon' : mode
return (
<div className={className}>
<audio ref={audioRef} src={audioUrl} crossOrigin="anonymous" preload="metadata" />
<div ref={containerRef} className="relative w-full overflow-hidden rounded-t-lg bg-gray-900" style={{ aspectRatio: '16 / 9' }}>
{mode === 'video' && canPreviewVideo ? (
<video
ref={videoRef}
src={normalizedVideoUrl}
className="absolute inset-0 h-full w-full object-cover"
loop
muted
playsInline
/>
) : dims ? (
<VisualizerBackdrop
mode={visualizerMode}
featuresRef={featuresRef}
paletteRef={paletteRef}
width={dims.w}
height={dims.h}
/>
) : (
<div className="absolute inset-0 flex items-center justify-center">
<Disc3 className="h-8 w-8 animate-spin text-gray-600" />
</div>
)}
{!isPlaying && songTitle ? (
<div className="pointer-events-none absolute inset-0 flex items-end p-2">
<div className="max-w-full truncate rounded bg-black/45 px-2 py-1 text-xs text-gray-100">{songTitle}</div>
</div>
) : null}
</div>
<div className="flex items-center border-t border-white/10 bg-gray-950/90" role="tablist" aria-label="Visualizer mode">
<button
type="button"
onClick={() => void togglePlay()}
aria-label={isPlaying ? 'Pause preview' : 'Play preview'}
className="px-3 py-1.5 text-sm text-gray-200 hover:bg-white/10"
>
{isPlaying ? '⏸' : '▶'}
</button>
{tabs.map((tab) => (
<button
key={tab.id}
type="button"
role="tab"
aria-selected={mode === tab.id}
onClick={() => setMode(tab.id)}
className={`flex-1 py-1.5 text-xs ${mode === tab.id ? 'bg-brand-500/30 text-white' : 'text-gray-300 hover:bg-white/10'}`}
>
{tab.label}
</button>
))}
</div>
{normalizedVideoUrl && !canPreviewVideo ? (
<p className="mt-1 text-[10px] text-amber-300">Video preview is available only for direct media URLs (mp4/webm/m4v/ogg).</p>
) : null}
</div>
)
}
@@ -2,79 +2,151 @@ import { useEffect, useRef } from 'react'
import type { MutableRefObject } from 'react'
import type { AudioFeatures } from '../../hooks/useAudioFeatures'
import type { VisualizerPalette } from '../../hooks/useVisualizerPalette'
import type { VisualizerSettings } from './VisualizerBackdrop'
import { destroyFluidSim, initFluidSim, splatFluid, updateFluidConfig } from './fluid/fluidSim'
type FluidSettings = NonNullable<VisualizerSettings['fluid']>
interface FluidVisualizerCanvasProps {
featuresRef: MutableRefObject<AudioFeatures>
paletteRef: MutableRefObject<VisualizerPalette>
width: number
height: number
fpsCap?: 30 | 60 | 120
quality?: 'low' | 'balanced' | 'high'
settings?: {
spawn_density?: number
splat_force?: number
decay?: number
}
}
type Blob = {
x: number
y: number
dx: number
dy: number
radius: number
life: number
color: string
settings?: FluidSettings
fpsCap?: number
quality?: string
}
function clamp01(v: number): number {
return Math.max(0, Math.min(1, v))
}
export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width, height, fpsCap = 60, quality = 'balanced', settings }: FluidVisualizerCanvasProps) {
function clamp(v: number, lo: number, hi: number): number {
return Math.max(lo, Math.min(hi, v))
}
function hexToRgb(hex: string): { r: number; g: number; b: number } {
const m = /^#?([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})$/i.exec(hex.trim())
if (!m) return { r: 0.5, g: 0.5, b: 0.8 }
return {
r: parseInt(m[1], 16) / 255,
g: parseInt(m[2], 16) / 255,
b: parseInt(m[3], 16) / 255,
}
}
function fluidDensityDissipation(s?: FluidSettings, kick = 0): number {
const base = 0.98 - (s?.background_persistence ?? 1) * 0.025
return clamp(base - kick * 0.04 * (s?.decay ?? 1), 0.85, 0.999)
}
function fluidVelocityDissipation(s?: FluidSettings, rms = 0): number {
const base = 0.96 - (1 / Math.max(0.5, s?.drag ?? 1)) * 0.06
return clamp(base - rms * 0.04, 0.80, 0.999)
}
function fluidCurl(s?: FluidSettings, treble = 0, dropBoost = 0): number {
const base = 15 + (s?.splat_force ?? 1) * 12
return clamp(base + treble * 8 + dropBoost * 20, 0, 80)
}
function fluidSplatRadius(s?: FluidSettings): number {
return clamp(0.18 + (s?.blob_life ?? 1) * 0.06, 0.05, 0.5)
}
function fluidBloomIntensity(s?: FluidSettings, intensity = 0, dropBoost = 0): number {
const base = 0.55 + Math.max(0, s?.treble_pulse ?? 1) * 0.28
return clamp(base + intensity * 1.55 + dropBoost * 0.85, 0.35, 3.0)
}
function fluidBloomThreshold(intensity = 0): number {
// Louder frames lower threshold so more dye contributes to bloom.
return clamp(0.68 - intensity * 0.28, 0.30, 0.80)
}
export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width, height, settings, fpsCap, quality }: FluidVisualizerCanvasProps) {
const canvasRef = useRef<HTMLCanvasElement>(null)
// Stable ref so the render loop always reads the latest settings without restarting.
const settingsRef = useRef<FluidSettings | undefined>(settings)
settingsRef.current = settings
const fpsCapRef = useRef(fpsCap ?? 60)
fpsCapRef.current = fpsCap ?? 60
const qualityRef = useRef(quality ?? 'balanced')
qualityRef.current = quality ?? 'balanced'
useEffect(() => {
const canvas = canvasRef.current
if (!canvas) return
const ctx = canvas.getContext('2d')
if (!ctx) return
const supportsWebGL = !!(canvas.getContext('webgl2') || canvas.getContext('webgl'))
if (!supportsWebGL) {
const ctx = canvas.getContext('2d')
if (!ctx) return
let rafFallback = 0
const renderFallback = () => {
rafFallback = window.requestAnimationFrame(renderFallback)
const w = Math.max(1, Math.round(canvas.clientWidth || 0), Math.round(width || 0))
const h = Math.max(1, Math.round(canvas.clientHeight || 0), Math.round(height || 0))
if (canvas.width !== w || canvas.height !== h) {
canvas.width = w
canvas.height = h
}
ctx.fillStyle = paletteRef.current.backgroundColor
ctx.fillRect(0, 0, w, h)
}
rafFallback = window.requestAnimationFrame(renderFallback)
return () => window.cancelAnimationFrame(rafFallback)
}
const s0 = settingsRef.current
const q = qualityRef.current
initFluidSim(canvas, {
TRIGGER: 'hover',
IMMEDIATE: false,
AUTO: false,
COLORFUL: false,
SIM_RESOLUTION: q === 'high' ? 160 : q === 'low' ? 96 : 128,
DYE_RESOLUTION: q === 'high' ? 1024 : q === 'low' ? 512 : 768,
DENSITY_DISSIPATION: fluidDensityDissipation(s0),
VELOCITY_DISSIPATION: fluidVelocityDissipation(s0),
PRESSURE: 0.8,
PRESSURE_ITERATIONS: 20,
CURL: fluidCurl(s0),
SPLAT_RADIUS: fluidSplatRadius(s0),
SPLAT_FORCE: 6000,
SHADING: true,
BLOOM: true,
BLOOM_INTENSITY: 0.8,
SUNRAYS: false,
BACK_COLOR: hexToRgb(paletteRef.current.backgroundColor),
})
const blobs: Blob[] = []
let raf = 0
let colorIndex = 0
let colorOffset = 0
let lastSplatAt = 0
let lastFrameAt = 0
const cap = fpsCap === 30 || fpsCap === 120 ? fpsCap : 60
const minFrameMs = 1000 / cap
const nextPaletteColor = () => {
const nextPaletteColor = (speed: number) => {
const palette = paletteRef.current.speakerPalette
if (palette.length === 0) return paletteRef.current.highlightColor
const c = palette[colorIndex % palette.length]
colorIndex += 1
return c
colorOffset += Math.max(0, speed)
return palette[Math.floor(colorOffset) % palette.length]
}
const spawnBlob = (strength: number) => {
const w = Math.max(1, width)
const h = Math.max(1, height)
const forceScale = Math.max(0.2, Number(settings?.splat_force ?? 1))
const angle = Math.random() * Math.PI * 2
const speed = (0.2 + strength * 2.2) * forceScale
blobs.push({
x: Math.random() * w,
y: Math.random() * h,
dx: Math.cos(angle) * speed,
dy: Math.sin(angle) * speed,
radius: (8 + strength * 38) * Math.max(0.25, Math.sqrt(forceScale)),
life: 0.7 + strength * 0.6,
color: nextPaletteColor(),
})
}
let prevBeatPhase = 0
let dropBoost = 0
let lastDropAt = 0
const render = (ts: number) => {
raf = window.requestAnimationFrame(render)
if (lastFrameAt && ts - lastFrameAt < minFrameMs) return
// FPS cap
const minFrameMs = 1000 / Math.max(1, fpsCapRef.current)
if (ts - lastFrameAt < minFrameMs - 1) return
lastFrameAt = ts
const w = Math.max(1, Math.round(canvas.clientWidth || 0), Math.round(width || 0))
@@ -85,70 +157,115 @@ export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width,
}
const f = featuresRef.current
const p = paletteRef.current
const bass = clamp01(f.bass)
const rms = clamp01(f.rms)
const s = settingsRef.current
const isLow = qualityRef.current === 'low'
const bass = clamp01(f.bass)
const sub = clamp01(f.frequencyBands.sub)
const rms = clamp01(f.rms)
const treble = clamp01(f.treble)
const idle = 0.5 + 0.5 * Math.sin(performance.now() * 0.0011)
const densityScale = Math.max(0.2, Number(settings?.spawn_density ?? 1))
const decayScale = Math.max(0.2, Number(settings?.decay ?? 1))
const idle = 0.5 + 0.5 * Math.sin(performance.now() * 0.0011)
const beatPhase = clamp01(f.beatInfo.beatPhase)
const phasePulse = Math.pow(Math.max(0, Math.sin(beatPhase * Math.PI)), 1.8)
const phaseWrapped = beatPhase < prevBeatPhase - 0.35
const kick = clamp01(sub * 0.72 + bass * 0.28)
const audioIntensity = clamp01(kick * 0.46 + rms * 0.34 + treble * 0.20)
// Ambient motion from RMS + small idle baseline so the renderer is never visually flat.
const ambientSpawnScale = quality === 'low' ? 0.7 : 1
if (Math.random() < (0.04 + rms * 0.31) * densityScale * ambientSpawnScale) spawnBlob(0.22 + rms * 0.58 + idle * 0.08)
const dropTriggered = f.patternInfo.isBeatDrop && ts - lastDropAt > 500
if (dropTriggered) {
dropBoost = 1
lastDropAt = ts
}
dropBoost *= 0.96
// Beat burst from bass peaks.
if (f.beatDetected) {
const baseCount = 3 + Math.floor(bass * 4)
const count = quality === 'low' ? Math.max(1, baseCount - 1) : baseCount
for (let i = 0; i < count; i += 1) spawnBlob(0.6 + bass * 0.8)
const densityScale = Math.max(0.2, s?.spawn_density ?? 1)
const splatForce = Math.max(0.2, s?.splat_force ?? 1)
const beatBurstCount = Math.max(0, s?.beat_burst_count ?? 1)
const treblePulse = Math.max(0, s?.treble_pulse ?? 1)
const colorSpeed = Math.max(0, s?.color_speed ?? 1)
updateFluidConfig(canvas, {
DENSITY_DISSIPATION: fluidDensityDissipation(s, kick),
VELOCITY_DISSIPATION: fluidVelocityDissipation(s, rms),
CURL: fluidCurl(s, treble, dropBoost),
SPLAT_RADIUS: fluidSplatRadius(s),
BLOOM_INTENSITY: fluidBloomIntensity(s, audioIntensity, dropBoost),
BLOOM_THRESHOLD: fluidBloomThreshold(audioIntensity),
BACK_COLOR: hexToRgb(paletteRef.current.backgroundColor),
PAUSED: false,
})
// WebGL sim renders in its own RAF; fpsCap throttles splat injection only.
const minSplatMs = Math.max(minFrameMs, isLow ? 45 : 25)
if (ts - lastSplatAt < minSplatMs - 1) {
prevBeatPhase = beatPhase
return
}
lastSplatAt = ts
// Ambient splat.
if (Math.random() < (0.04 + rms * 0.35 + phasePulse * 0.08 + idle * 0.04) * densityScale) {
const angle = Math.random() * Math.PI * 2
const force = (0.2 + rms * 0.8 + treble * 0.3 * treblePulse + idle * 0.12) * splatForce * 400
splatFluid(
canvas,
Math.random(),
Math.random(),
Math.cos(angle) * force,
Math.sin(angle) * force,
hexToRgb(nextPaletteColor(colorSpeed * 0.25)),
)
}
ctx.fillStyle = p.backgroundColor
ctx.globalAlpha = 0.20 + (1 - bass) * 0.16
ctx.fillRect(0, 0, w, h)
ctx.globalAlpha = 1
// Correction C fix: trail lifetime (decay) is driven by bass — loud kick/bass →
// slower decay → longer-lived blobs (more persistent ink on loud beats).
// Velocity drag is driven by rms — overall loudness → more momentum retained.
// Original Integration Plan had these driven by rms uniformly, which made
// trail length insensitive to bass and meant quiet treble-heavy passages
// produced the same persistence as loud bass hits.
const decay = (0.009 + (1 - bass) * 0.012) * decayScale // loud bass → 0.009 (long trail)
const drag = 0.984 - rms * 0.010 // loud rms → more momentum
for (let i = blobs.length - 1; i >= 0; i -= 1) {
const b = blobs[i]
b.x += b.dx
b.y += b.dy
b.dx *= drag
b.dy *= drag
b.life -= decay
if (b.life <= 0) {
blobs.splice(i, 1)
continue
// Beat burst.
if (f.beatDetected || phaseWrapped || dropTriggered) {
const count = Math.max(1, Math.round((3 + kick * 5 + dropBoost * 4) * beatBurstCount))
for (let i = 0; i < count; i += 1) {
const angle = Math.random() * Math.PI * 2
const force = (0.6 + bass * 1.2 + dropBoost * 0.6) * splatForce * 800
splatFluid(
canvas,
Math.random(),
Math.random(),
Math.cos(angle) * force,
Math.sin(angle) * force,
dropTriggered
? hexToRgb(paletteRef.current.highlightColor)
: hexToRgb(nextPaletteColor(colorSpeed * 0.7)),
)
}
const pulse = 1 + treble * 0.25
const r = b.radius * pulse * Math.max(0.15, b.life)
const grad = ctx.createRadialGradient(b.x, b.y, r * 0.1, b.x, b.y, r)
grad.addColorStop(0, `${b.color}cc`)
grad.addColorStop(1, `${b.color}00`)
ctx.fillStyle = grad
ctx.beginPath()
ctx.arc(b.x, b.y, r, 0, Math.PI * 2)
ctx.fill()
}
// Beat-phase sweep.
if (beatPhase > 0.001 || f.beatInfo.bpm > 0) {
const phaseAngle = beatPhase * Math.PI * 2
if (Math.random() < 0.08 + phasePulse * 0.22 + dropBoost * 0.18) {
const force = (0.28 + kick * 0.5 + phasePulse * 0.4) * splatForce * 500
splatFluid(
canvas,
0.5 + Math.cos(phaseAngle) * 0.28,
0.5 + Math.sin(phaseAngle) * 0.20,
Math.cos(phaseAngle + Math.PI / 2) * force,
Math.sin(phaseAngle + Math.PI / 2) * force,
dropBoost > 0.18
? hexToRgb(paletteRef.current.highlightColor)
: hexToRgb(nextPaletteColor(colorSpeed * 0.9)),
)
}
}
prevBeatPhase = beatPhase
}
raf = window.requestAnimationFrame(render)
return () => {
window.cancelAnimationFrame(raf)
destroyFluidSim(canvas)
}
}, [featuresRef, paletteRef, width, height, fpsCap, quality, settings])
// Only restart when data-source refs change.
// Settings, fpsCap, and quality are read via refs each frame — no restart on changes.
// width/height are consumed each frame as fallback to clientWidth/clientHeight.
}, [featuresRef, paletteRef])
return <canvas ref={canvasRef} className="h-full w-full" aria-hidden="true" />
}
@@ -2,31 +2,76 @@ import { useEffect, useRef } from 'react'
import type { MutableRefObject } from 'react'
import type { AudioFeatures } from '../../hooks/useAudioFeatures'
import type { VisualizerPalette } from '../../hooks/useVisualizerPalette'
import type { VisualizerSettings } from './VisualizerBackdrop'
import { centeredNoise2D } from './muon/noise'
import { patternFromPreset, samplePattern, type MuonPatternFamily } from './muon/patterns'
type MuonSettings = NonNullable<VisualizerSettings['muon']>
interface MuonVisualizerCanvasProps {
featuresRef: MutableRefObject<AudioFeatures>
paletteRef: MutableRefObject<VisualizerPalette>
width: number
height: number
fpsCap?: 30 | 60 | 120
quality?: 'low' | 'balanced' | 'high'
settings?: {
radius_multiplier?: number
aperture?: number
spacing?: number
}
settings?: MuonSettings
fpsCap?: number
quality?: string
}
type Particle = { angle: number; radius: number; speed: number; lane: number }
const PARTICLE_COUNT = 220
type MuonParticle = {
active: boolean
x: number
y: number
vx: number
vy: number
ring: number
age: number
maxAge: number
size: number
hue: number
alpha: number
pattern: MuonPatternFamily
}
function clamp01(v: number): number {
return Math.max(0, Math.min(1, v))
}
export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, height, fpsCap = 60, quality = 'balanced', settings }: MuonVisualizerCanvasProps) {
function clamp(v: number, lo: number, hi: number): number {
return Math.max(lo, Math.min(hi, v))
}
function maxParticlesByQuality(quality?: string): number {
if (quality === 'low') return 900
if (quality === 'high') return 2400
return 1600
}
function createParticlePool(size: number): MuonParticle[] {
return Array.from({ length: size }, () => ({
active: false,
x: 0,
y: 0,
vx: 0,
vy: 0,
ring: 0,
age: 0,
maxAge: 1,
size: 1,
hue: 0,
alpha: 1,
pattern: 'orbital',
}))
}
export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, height, settings, fpsCap, quality }: MuonVisualizerCanvasProps) {
const canvasRef = useRef<HTMLCanvasElement>(null)
// Stable ref so the render loop always reads the latest settings without restarting.
const settingsRef = useRef<MuonSettings | undefined>(settings)
settingsRef.current = settings
const fpsCapRef = useRef(fpsCap ?? 60)
fpsCapRef.current = fpsCap ?? 60
useEffect(() => {
const canvas = canvasRef.current
@@ -34,32 +79,94 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
const ctx = canvas.getContext('2d')
if (!ctx) return
const particles: Particle[] = Array.from({ length: PARTICLE_COUNT }, (_, i) => ({
angle: (i / PARTICLE_COUNT) * Math.PI * 2,
radius: 0.2 + (i % 12) * 0.08,
speed: 0.002 + (i % 9) * 0.00045,
lane: i % 8,
}))
let radiusCurrent = 1
let radiusTarget = 1
let prevBeatPhase = 0
let dropBoost = 0
let lastDropAt = 0
let patternShift = 0
let poolCap = maxParticlesByQuality('balanced')
let particles = createParticlePool(poolCap)
let nextPoolIdx = 0
let raf = 0
// Correction B fix: radiusMultiplier cycles through presets on beats rather than
// being mapped directly from drum intensity each frame. We lerp toward a preset
// target, snapping to the next preset on each detected beat — equivalent to
// wavePresetController triggering a GSAP transition in the full Three.js port.
const RADIUS_PRESETS = [0.90, 0.95, 1.00, 1.05, 1.10, 0.85, 1.15, 1.02]
let radiusTarget = 1.0
let radiusCurrent = 1.0
let lastPresetAt = 0
let presetIdx = 0
let lastFrameAt = 0
const cap = fpsCap === 30 || fpsCap === 120 ? fpsCap : 60
const minFrameMs = 1000 / cap
const emitParticle = (
cx: number,
cy: number,
ring: number,
divisions: number,
beatPhase: number,
hueBase: number,
aperture: number,
mirror: boolean,
patternPreset: string,
ringRadius: number,
lifeFrames: number,
noiseScale: number,
noiseForce: number,
alphaSeed: number,
sizeSeed: number,
dynamicRadius: number,
) => {
const emitOne = (phaseOffset: number) => {
const p = particles[nextPoolIdx]
nextPoolIdx = (nextPoolIdx + 1) % particles.length
const t = ((performance.now() * 0.0004) + ring / Math.max(1, divisions) + phaseOffset) * Math.PI * 2
const pattern = patternFromPreset(patternPreset, ring + patternShift)
const ringNorm = ring / Math.max(1, divisions - 1)
const sample = samplePattern(pattern, t, ringNorm, aperture, beatPhase)
const nx = centeredNoise2D(sample.x * noiseScale + ringNorm * 1.3, sample.y * noiseScale, ring * 17.17)
const ny = centeredNoise2D(sample.y * noiseScale, sample.x * noiseScale + ringNorm * 1.9, ring * 29.37)
const noiseAmp = noiseForce * 0.32
const sx = sample.x + nx * noiseAmp
const sy = sample.y + ny * noiseAmp
const ex = cx + sx * ringRadius * dynamicRadius
const ey = cy + sy * ringRadius * 0.86 * dynamicRadius
const tvx = sample.tx + nx * noiseAmp * 0.4
const tvy = sample.ty + ny * noiseAmp * 0.4
const speed = 0.12 + ringNorm * 0.22 + sizeSeed * 0.08
p.active = true
p.x = ex
p.y = ey
p.vx = tvx * speed
p.vy = tvy * speed
p.ring = ring
p.age = 0
p.maxAge = lifeFrames
p.size = 0.9 + sizeSeed * 2.2
p.hue = (hueBase + ring * 360 / Math.max(1, divisions)) % 360
p.alpha = clamp(alphaSeed, 0.2, 1)
p.pattern = pattern
}
emitOne(0)
if (mirror) emitOne(0.5)
}
const render = (ts: number) => {
raf = window.requestAnimationFrame(render)
if (lastFrameAt && ts - lastFrameAt < minFrameMs) return
// FPS cap
const minFrameMs = 1000 / Math.max(1, fpsCapRef.current)
if (ts - lastFrameAt < minFrameMs - 1) return
const dt = Math.max(0.5, Math.min(2, (ts - lastFrameAt) / 16.6667))
lastFrameAt = ts
const s = settingsRef.current
const qualityCap = maxParticlesByQuality(quality)
if (qualityCap !== poolCap) {
poolCap = qualityCap
particles = createParticlePool(poolCap)
nextPoolIdx = 0
}
const w = Math.max(1, Math.round(canvas.clientWidth || 0), Math.round(width || 0))
const h = Math.max(1, Math.round(canvas.clientHeight || 0), Math.round(height || 0))
if (canvas.width !== w || canvas.height !== h) {
@@ -69,76 +176,157 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
const f = featuresRef.current
const p = paletteRef.current
const drum = clamp01(f.drumIntensity)
const now = performance.now() * 0.001
const radiusScale = Math.max(0.2, s?.radius_multiplier ?? 1)
const apertureScale = Math.max(0.2, s?.aperture ?? 1)
const spacingScale = Math.max(0.2, s?.spacing ?? 1)
const rotationSpeed = Math.max(0.1, s?.rotation_speed ?? 1)
const glowIntensity = Math.max(0, s?.glow_intensity ?? 1)
const particleSize = Math.max(0.1, s?.particle_size ?? 1)
const drumReactivity = Math.max(0, s?.drum_reactivity ?? 1)
const hueRange = Math.max(0, s?.hue_range ?? 1)
const beatSensitivity = Math.max(0.1, s?.beat_sensitivity ?? 1)
const ellipseRatio = Math.max(0.1, Math.min(1, s?.ellipse_ratio ?? 0.62))
const idleSpeed = Math.max(0, s?.idle_speed ?? 1)
const monoColor = Boolean(s?.mono_color)
const divisions = Math.max(8, Math.min(48, Math.round(s?.divisions ?? 16)))
const lifespan = Math.max(45, Math.min(420, Math.round(s?.lifespan ?? 180)))
const mirror = Boolean(s?.particle_mirror)
const dynamicRadiusCfg = Math.max(0.2, s?.dynamic_radius ?? 1)
const deltaLimit = Math.max(0.0005, s?.delta_response_limit ?? 0.005)
const preset = s?.visualization_preset ?? 'hybrid'
const noiseScale = Math.max(0.05, s?.noise_scale ?? 1)
const noiseForce = Math.max(0, s?.noise_force ?? 1)
const sub = clamp01(f.frequencyBands.sub)
const kick = clamp01(sub * 0.72 + f.bass * 0.28)
const drum = clamp01((kick * 0.75 + f.drumIntensity * 0.25) * drumReactivity)
const treble = clamp01(f.treble)
const now = ts
const idle = 0.5 + 0.5 * Math.sin(now * 0.0012)
const radiusScale = Math.max(0.2, Number(settings?.radius_multiplier ?? 1))
const apertureScale = Math.max(0.2, Number(settings?.aperture ?? 1))
const spacingScale = Math.max(0.2, Number(settings?.spacing ?? 1))
const bright = clamp01((f.frequencyBands.presence + f.frequencyBands.brilliance) * 0.5)
const beatPhase = clamp01(f.beatInfo.beatPhase)
const phaseWave = 0.5 + 0.5 * Math.sin(beatPhase * Math.PI * 2)
const phasePulse = Math.pow(Math.max(0, Math.sin(beatPhase * Math.PI)), 1.6)
const energyDeltaAbs = Math.abs(f.patternInfo.energyDelta)
// Bug 1 fix: hue animates via drum intensity when audio is playing, and falls
// back to a slow time-based drift when silent so the visual never freezes on a
// single colour (original hueControl bug: hue was always 0 during playback).
// Correction A fix: treble drift is capped at ±15° to stay within the manifest
// palette colour family (original Integration Plan used treble * 40 / 120 which
// rotated far outside the intended hue family).
const hue = drum > 0.01
? ((p.seedHue + drum * 20 + treble * 15) + 360) % 360
: ((p.seedHue + Math.sin(now * 0.00018) * 15) + 360) % 360
// Correction B fix: advance the preset index on each beat and lerp toward the
// new target — mirrors wavePresetController / GSAP transition in the Three.js port.
if (f.beatDetected && now - lastPresetAt > 200) {
presetIdx = (presetIdx + 1) % RADIUS_PRESETS.length
radiusTarget = RADIUS_PRESETS[presetIdx] + drum * 0.25
lastPresetAt = now
const dropTriggered = f.patternInfo.isBeatDrop && now - lastDropAt > 0.5
if (dropTriggered) {
dropBoost = 1
lastDropAt = now
patternShift += 1
}
radiusCurrent += (radiusTarget - radiusCurrent) * 0.07
const radiusMultiplier = (radiusCurrent + idle * 0.12) * radiusScale
dropBoost *= 0.965
const aperture = (1.55 + drum * 1.9 + idle * 0.24) * apertureScale
const spacing = (0.015 + (1 - drum) * 0.014) * spacingScale
const phaseWrapped = beatPhase < prevBeatPhase - 0.35
const idle = 0.5 + 0.5 * Math.sin(now * 1.2 * Math.max(0.01, idleSpeed))
ctx.clearRect(0, 0, w, h)
const hue = monoColor
? p.seedHue
: drum > 0.01
? ((p.seedHue + drum * 20 * hueRange + treble * 15 * hueRange + phaseWave * 10 * hueRange + dropBoost * 55 * hueRange) + 360) % 360
: ((p.seedHue + Math.sin(now * 0.22) * 15 * hueRange) + 360) % 360
const deltaGate = energyDeltaAbs >= deltaLimit ? 1 : 0
radiusTarget = 1
+ drum * 0.18
+ phasePulse * 0.12
+ idle * 0.05
+ dropBoost * 0.3
+ (phaseWrapped ? 0.06 : 0)
+ deltaGate * Math.min(0.5, energyDeltaAbs * dynamicRadiusCfg * 15)
radiusCurrent += (radiusTarget - radiusCurrent) * 0.065
const dynamicRadius = radiusCurrent * dynamicRadiusCfg
const aperture = (1.2 + drum * 1.8 + phaseWave * 0.25 + dropBoost * 0.35) * apertureScale
const spacing = (Math.min(w, h) * (0.013 + (1 - drum) * 0.012)) * spacingScale
const trailAlpha = clamp(0.16 + (1 - clamp01(lifespan / 420)) * 0.36 - dropBoost * 0.05, 0.08, 0.55)
ctx.fillStyle = p.backgroundColor
ctx.globalAlpha = trailAlpha
ctx.fillRect(0, 0, w, h)
ctx.globalAlpha = 1
const cx = w / 2
const cy = h / 2
const base = Math.min(w, h) * 0.2
const glow = 14 + drum * 28
const particleStride = quality === 'low' ? 2 : 1
const cx = w / 2
const cy = h / 2
const base = Math.min(w, h) * 0.17 * radiusScale
const glow = (10 + drum * 22 + phasePulse * 7 + dropBoost * 18) * glowIntensity
const emitDensity = (s?.particle_count ?? 220) / 220
const emitCount = Math.max(1, Math.round((0.8 + drum * 2.4 + phasePulse * 0.8 + dropBoost * 2.2) * emitDensity * (divisions / 16) * (0.75 + beatSensitivity * 0.25)))
const lifeFrames = Math.round(lifespan * (0.8 + (1 - drum) * 0.25))
for (let i = 0; i < emitCount; i += 1) {
const ring = i % divisions
const ringRadius = base + ring * spacing
emitParticle(
cx,
cy,
ring,
divisions,
beatPhase,
hue + bright * 18,
aperture,
mirror,
preset,
ringRadius,
lifeFrames,
noiseScale,
noiseForce,
0.34 + drum * 0.45 + phasePulse * 0.1 + dropBoost * 0.15,
0.45 + bright * 0.35,
dynamicRadius,
)
}
ctx.globalCompositeOperation = 'lighter'
for (let i = 0; i < particles.length; i += particleStride) {
for (let i = 0; i < particles.length; i += 1) {
const pt = particles[i]
pt.angle += pt.speed + drum * 0.02
if (!pt.active) continue
const ring = base + pt.lane * (spacing * Math.min(w, h) * 5)
const r = ring * radiusMultiplier * (1 + Math.sin(pt.angle * aperture) * 0.16)
const x = cx + Math.cos(pt.angle) * r
const y = cy + Math.sin(pt.angle * aperture) * (r * 0.62)
pt.age += dt
if (pt.age >= pt.maxAge) {
pt.active = false
continue
}
const size = 1.75 + drum * 3.0 + (i % 4) * 0.35
const alpha = 0.34 + drum * 0.42
const life = 1 - pt.age / Math.max(1, pt.maxAge)
const drag = clamp(0.94 + (1 - drum) * 0.03, 0.86, 0.985)
pt.vx *= Math.pow(drag, dt)
pt.vy *= Math.pow(drag, dt)
pt.x += pt.vx * rotationSpeed * dt
pt.y += pt.vy * rotationSpeed * dt
const wobble = Math.sin((now * 1.2 + pt.ring * 0.37 + pt.age * 0.03) * aperture) * (0.6 + noiseForce * 0.4)
const x = pt.x + wobble
const y = pt.y + wobble * ellipseRatio
const size = Math.max(0.3, pt.size * particleSize * (0.4 + life * 0.9))
const alpha = pt.alpha * life
const laneHue = monoColor
? hue
: (pt.hue + phaseWave * 18 * hueRange + beatPhase * 8 + dropBoost * 18) % 360
ctx.beginPath()
ctx.fillStyle = `hsla(${hue}, 92%, 64%, ${alpha})`
ctx.fillStyle = `hsla(${laneHue}, 92%, ${58 + bright * 24}%, ${alpha})`
ctx.shadowColor = p.highlightColor
ctx.shadowBlur = glow
ctx.shadowBlur = glow
ctx.arc(x, y, size, 0, Math.PI * 2)
ctx.fill()
}
ctx.globalCompositeOperation = 'source-over'
ctx.shadowBlur = 0
prevBeatPhase = beatPhase
}
raf = window.requestAnimationFrame(render)
return () => {
window.cancelAnimationFrame(raf)
}
}, [featuresRef, paletteRef, width, height, fpsCap, quality, settings])
// Only restart when data-source refs change.
// Settings are read via settingsRef each frame — no restart on slider changes.
// width/height are consumed each frame as fallback to clientWidth/clientHeight.
}, [featuresRef, paletteRef])
return <canvas ref={canvasRef} className="h-full w-full" aria-hidden="true" />
}
@@ -10,26 +10,50 @@ export type VisualizerMode = 'muon' | 'fluid' | null
/**
* Per-song visualizer intensity multipliers from manifest.visualizer.settings.
* All values default to 1.0 (no change from audio-driven behaviour).
* All numeric values are multipliers (1.0 = default audio-driven behaviour) unless noted.
*/
export interface VisualizerSettings {
quality?: 'low' | 'balanced' | 'high'
fps_cap?: 30 | 60 | 120
muon?: {
/** Scales the preset-cycling radius range. Default 1.0. */
// Original
radius_multiplier?: number
/** Scales the spiral aperture range. Default 1.0. */
aperture?: number
/** Scales inter-particle spacing. Default 1.0. */
spacing?: number
// Controls Audit additions
particle_count?: number // direct int 40–500
rotation_speed?: number
glow_intensity?: number
particle_size?: number
drum_reactivity?: number
hue_range?: number
beat_sensitivity?: number
ellipse_ratio?: number // direct 0.30–1.0
// GUI Parameter Map additions
idle_speed?: number
noise_scale?: number
noise_force?: number
mono_color?: boolean
divisions?: number
lifespan?: number
particle_mirror?: boolean
dynamic_radius?: number
delta_response_limit?: number
visualization_preset?: 'default' | 'orbital' | 'flower' | 'spiro' | 'hybrid'
}
fluid?: {
/** Multiplies ambient blob spawn probability per frame. Default 1.0. */
// Original
spawn_density?: number
/** Multiplies blob velocity and size on spawn. Default 1.0. */
splat_force?: number
/** Multiplies per-frame blob lifetime drain (higher = shorter trails). Default 1.0. */
decay?: number
// Controls Audit additions
background_persistence?: number
beat_burst_count?: number
treble_pulse?: number
drag?: number
blob_life?: number
// GUI Parameter Map additions
color_speed?: number
}
}
@@ -0,0 +1,163 @@
let WebGLFluid: any = null
let fluidLoadPromise: Promise<any> | null = null
// Load vendored local module dynamically so Vite resolves a stable in-repo path.
const getWebGLFluid = async (): Promise<any> => {
if (WebGLFluid) return WebGLFluid
if (fluidLoadPromise) return fluidLoadPromise
fluidLoadPromise = (async () => {
const module = await import('./webgl-fluid.mjs')
WebGLFluid = module.default || module
return WebGLFluid
})()
return fluidLoadPromise
}
export interface FluidColor {
r: number
g: number
b: number
}
export interface FluidConfig {
TRIGGER?: 'hover' | 'click'
IMMEDIATE?: boolean
AUTO?: boolean
INTERVAL?: number
SIM_RESOLUTION?: number
DYE_RESOLUTION?: number
DENSITY_DISSIPATION?: number
VELOCITY_DISSIPATION?: number
PRESSURE?: number
PRESSURE_ITERATIONS?: number
CURL?: number
SPLAT_RADIUS?: number
SPLAT_FORCE?: number
SPLAT_COUNT?: number
SHADING?: boolean
COLORFUL?: boolean
COLOR_UPDATE_SPEED?: number
PAUSED?: boolean
BACK_COLOR?: FluidColor
TRANSPARENT?: boolean
BLOOM?: boolean
BLOOM_ITERATIONS?: number
BLOOM_RESOLUTION?: number
BLOOM_INTENSITY?: number
BLOOM_THRESHOLD?: number
BLOOM_SOFT_KNEE?: number
SUNRAYS?: boolean
SUNRAYS_RESOLUTION?: number
SUNRAYS_WEIGHT?: number
}
interface FluidInstance {
canvas: HTMLCanvasElement
config: FluidConfig
lastX: number
lastY: number
}
const instances = new WeakMap<HTMLCanvasElement, FluidInstance>()
function clamp01(v: number): number {
return Math.max(0, Math.min(1, v))
}
function dispatchMouse(canvas: HTMLCanvasElement, type: 'mousedown' | 'mousemove', x: number, y: number): void {
const evt = new MouseEvent(type, {
bubbles: true,
cancelable: true,
clientX: x,
clientY: y,
})
Object.defineProperty(evt, 'offsetX', { configurable: true, value: x })
Object.defineProperty(evt, 'offsetY', { configurable: true, value: y })
canvas.dispatchEvent(evt)
}
export function initFluidSim(canvas: HTMLCanvasElement, initialConfig: FluidConfig): void {
destroyFluidSim(canvas)
const config: FluidConfig = {
TRIGGER: 'hover',
IMMEDIATE: false,
AUTO: false,
INTERVAL: 3000,
COLORFUL: false,
SHADING: true,
BLOOM: true,
SUNRAYS: false,
PAUSED: false,
...initialConfig,
}
// Load WebGL library asynchronously but don't block
getWebGLFluid()
.then((WebGL) => {
// Only initialize if canvas still exists and not destroyed
if (document.contains(canvas)) {
WebGL(canvas, config)
instances.set(canvas, {
canvas,
config,
lastX: canvas.clientWidth * 0.5,
lastY: canvas.clientHeight * 0.5,
})
}
})
.catch((err) => console.error('Failed to load webgl-fluid:', err))
}
export function updateFluidConfig(canvas: HTMLCanvasElement, patch: Partial<FluidConfig>): void {
const inst = instances.get(canvas)
if (!inst) return
Object.assign(inst.config, patch)
}
export function splatFluid(canvas: HTMLCanvasElement, x: number, y: number, dx: number, dy: number, _color: FluidColor): void {
const inst = instances.get(canvas)
if (!inst) return
const w = Math.max(1, canvas.clientWidth || canvas.width || 1)
const h = Math.max(1, canvas.clientHeight || canvas.height || 1)
const nx = clamp01(x)
const ny = clamp01(y)
const px = nx * w
const py = ny * h
const speedNorm = Math.hypot(dx, dy) / Math.max(1, Math.min(w, h) * 6)
const pressure = clamp01(speedNorm)
const dragScale = 0.02 + pressure * 0.09
const tx = Math.max(0, Math.min(w, px + dx * dragScale))
const ty = Math.max(0, Math.min(h, py + dy * dragScale))
const mx = px + (tx - px) * (0.45 + pressure * 0.25)
const my = py + (ty - py) * (0.45 + pressure * 0.25)
dispatchMouse(canvas, 'mousedown', px, py)
dispatchMouse(canvas, 'mousemove', mx, my)
dispatchMouse(canvas, 'mousemove', tx, ty)
window.dispatchEvent(new MouseEvent('mouseup', { bubbles: true, cancelable: true }))
inst.lastX = tx
inst.lastY = ty
}
export function destroyFluidSim(canvas: HTMLCanvasElement): void {
const inst = instances.get(canvas)
if (!inst) return
inst.config.PAUSED = true
inst.config.AUTO = false
const gl = (canvas.getContext('webgl2') || canvas.getContext('webgl')) as WebGLRenderingContext | null
const lose = gl?.getExtension('WEBGL_lose_context')
lose?.loseContext()
instances.delete(canvas)
}
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,2 @@
declare const WebGLFluid: (canvas: HTMLCanvasElement, options?: any) => void
export default WebGLFluid
@@ -0,0 +1,8 @@
export function hashNoise2D(x: number, y: number, seed = 0): number {
const s = Math.sin(x * 127.1 + y * 311.7 + seed * 74.7) * 43758.5453123
return s - Math.floor(s)
}
export function centeredNoise2D(x: number, y: number, seed = 0): number {
return hashNoise2D(x, y, seed) * 2 - 1
}
@@ -0,0 +1,92 @@
export type MuonPatternFamily = 'orbital' | 'flower' | 'ripple' | 'spiro' | 'strobe-grid'
export interface PatternSample {
x: number
y: number
tx: number
ty: number
}
function clamp01(v: number): number {
return Math.max(0, Math.min(1, v))
}
function sampleOrbital(t: number, ring: number, aperture: number): PatternSample {
const r = 1 + Math.sin(t * (1.2 + ring * 0.9) * aperture) * 0.16
const x = Math.cos(t) * r
const y = Math.sin(t) * (0.85 + ring * 0.22)
return { x, y, tx: -Math.sin(t), ty: Math.cos(t) }
}
function sampleFlower(t: number, ring: number, aperture: number, beatPhase: number): PatternSample {
const petals = 3 + (ring % 6)
const pulse = 0.82 + Math.sin(beatPhase * Math.PI * 2) * 0.08
const r = 0.78 + Math.cos(t * petals * aperture) * 0.28 * pulse
const x = Math.cos(t) * r
const y = Math.sin(t) * r
return { x, y, tx: -Math.sin(t) * (1 + petals * 0.04), ty: Math.cos(t) * (1 + petals * 0.04) }
}
function sampleRipple(t: number, ring: number, aperture: number, beatPhase: number): PatternSample {
const ripple = Math.sin(t * (2.4 + ring * 1.3) * aperture + beatPhase * Math.PI * 2)
const r = 0.86 + ripple * 0.22
const x = Math.cos(t) * r
const y = Math.sin(t) * (0.7 + Math.abs(ripple) * 0.5)
return { x, y, tx: -Math.sin(t), ty: Math.cos(t) }
}
function sampleSpiro(t: number, ring: number, aperture: number): PatternSample {
const a = 1
const b = 0.35 + (ring % 5) * 0.07
const k = a - b
const wt = t * (1.2 + aperture * 0.6)
const x = k * Math.cos(wt) + b * Math.cos((k / b) * wt)
const y = k * Math.sin(wt) - b * Math.sin((k / b) * wt)
return { x, y, tx: -Math.sin(wt), ty: Math.cos(wt) }
}
function sampleStrobeGrid(t: number, ring: number, beatPhase: number): PatternSample {
const segments = 8 + (ring % 8)
const q = Math.round((t / (Math.PI * 2)) * segments) / segments
const a = q * Math.PI * 2
const pulse = 0.84 + clamp01(Math.sin(beatPhase * Math.PI * 2) * 0.5 + 0.5) * 0.22
const x = Math.cos(a) * pulse
const y = Math.sin(a) * pulse
return { x, y, tx: -Math.sin(a), ty: Math.cos(a) }
}
export function samplePattern(
family: MuonPatternFamily,
t: number,
ringNorm: number,
aperture: number,
beatPhase: number,
): PatternSample {
switch (family) {
case 'flower':
return sampleFlower(t, ringNorm, aperture, beatPhase)
case 'ripple':
return sampleRipple(t, ringNorm, aperture, beatPhase)
case 'spiro':
return sampleSpiro(t, ringNorm, aperture)
case 'strobe-grid':
return sampleStrobeGrid(t, ringNorm, beatPhase)
case 'orbital':
default:
return sampleOrbital(t, ringNorm, aperture)
}
}
const PRESET_MAP: Record<string, MuonPatternFamily[]> = {
default: ['orbital', 'ripple', 'flower'],
orbital: ['orbital'],
flower: ['flower', 'ripple'],
spiro: ['spiro', 'orbital'],
hybrid: ['orbital', 'flower', 'ripple', 'spiro', 'strobe-grid'],
}
export function patternFromPreset(preset: string | undefined, ring: number): MuonPatternFamily {
const key = preset && PRESET_MAP[preset] ? preset : 'hybrid'
const list = PRESET_MAP[key]
return list[ring % list.length]
}
+153 -169
View File
@@ -1,4 +1,8 @@
import { useEffect, useRef, type MutableRefObject, type RefObject } from 'react'
import {
MuonFeatureExtractor,
type MuonFrequencyBands,
} from '../audio/MuonFeatureExtractor'
export interface AudioFeatures {
bass: number
@@ -9,76 +13,88 @@ export interface AudioFeatures {
spectralCentroid: number
drumIntensity: number
beatDetected: boolean
freqData: Uint8Array
frequencyBands: {
sub: number
bass: number
lowMid: number
mid: number
highMid: number
presence: number
brilliance: number
}
beatInfo: {
isBeat: boolean
bpm: number
confidence: number
timeSinceLastBeat: number
beatPhase: number
}
patternInfo: {
isBeatDrop: boolean
isBuildUp: boolean
isSilence: boolean
intensity: number
energyDelta: number
}
spectralRolloff: number
spectralFlatness: number
zcr: number
freqData: Float32Array
updatedAt: number
}
interface MeydaFeatureFrame {
rms?: number
energy?: number
loudness?: number | { total?: number }
spectralCentroid?: number
}
interface MeydaAnalyzerLike {
start?: () => void
stop?: () => void
}
interface MeydaLike {
createMeydaAnalyzer: (opts: {
audioContext: AudioContext
source: AudioNode
bufferSize: number
featureExtractors: string[]
callback: (features: MeydaFeatureFrame) => void
}) => MeydaAnalyzerLike
}
interface UseAudioFeaturesArgs {
audioCtxRef: RefObject<AudioContext | null>
stemGainRefs: RefObject<Record<string, GainNode>>
singleAudioRef: RefObject<HTMLAudioElement | null>
analysisAudioRef: RefObject<HTMLAudioElement | null>
isPlayingRef: RefObject<boolean>
enabled?: boolean
analysisSourceKey?: string | null
}
const EMPTY_FREQ = new Uint8Array(0)
const EMPTY_FREQ = new Float32Array(0)
const EMPTY_BANDS = {
sub: 0,
bass: 0,
lowMid: 0,
mid: 0,
highMid: 0,
presence: 0,
brilliance: 0,
}
const EMPTY_BEAT = {
isBeat: false,
bpm: 0,
confidence: 0,
timeSinceLastBeat: 0,
beatPhase: 0,
}
const EMPTY_PAT = {
isBeatDrop: false,
isBuildUp: false,
isSilence: true,
intensity: 0,
energyDelta: 0,
}
const PEAK_DECAY = 0.9998
const BAND_KEYS: Array<keyof MuonFrequencyBands> = ['sub', 'bass', 'lowMid', 'mid', 'highMid', 'presence', 'brilliance']
function clamp01(value: number): number {
return Math.max(0, Math.min(1, value))
}
function avgSlice(data: Uint8Array, start: number, end: number): number {
const s = Math.max(0, Math.min(data.length, start))
const e = Math.max(s, Math.min(data.length, end))
const span = e - s
if (span <= 0) return 0
let sum = 0
for (let i = s; i < e; i += 1) sum += data[i]
return sum / span
}
function deriveDrumIntensity(bass: number, treble: number, rms: number): number {
// Exponential scalers matching the corrected muon source audio.js pattern
// (Bug 2 fix). Input range 0–1; outputs range 0–2.002 (normalised back below).
// Original formula: 1.001 * (1 + avgFr * 2) - 1.001
// This curve is near-zero when quiet and amplifies progressively when loud,
// giving a more musical response than a plain linear mix.
const expBass = 1.001 * (1 + bass * 2) - 1.001
const expTreble = 1.001 * (1 + treble * 2) - 1.001
// Coefficient ratio mirrors wave.js beatScaler (bass ~2× treble).
// Divide by 2.0 to normalise the 0–2.002 range back to 0–1 before clamping.
const raw = (expBass * 0.68 + rms * 0.24 + expTreble * 0.08) / 2.0
function deriveDrumIntensity(sub: number, bass: number, rms: number): number {
const expSub = 1.001 * (1 + sub * 2) - 1.001
const expBass = 1.001 * (1 + bass * 2) - 1.001
const raw = (expSub * 0.5 + expBass * 0.38 + rms * 0.12) / 2.0
return clamp01(Math.pow(clamp01(raw), 0.82))
}
export function useAudioFeatures({
audioCtxRef,
stemGainRefs,
singleAudioRef,
analysisAudioRef,
isPlayingRef,
enabled = true,
analysisSourceKey,
}: UseAudioFeaturesArgs): MutableRefObject<AudioFeatures> {
const featuresRef = useRef<AudioFeatures>({
bass: 0,
@@ -89,155 +105,128 @@ export function useAudioFeatures({
spectralCentroid: 0,
drumIntensity: 0,
beatDetected: false,
frequencyBands: EMPTY_BANDS,
beatInfo: EMPTY_BEAT,
patternInfo: EMPTY_PAT,
spectralRolloff: 0,
spectralFlatness: 0,
zcr: 0,
freqData: EMPTY_FREQ,
updatedAt: 0,
})
const mediaSourceRef = useRef<MediaElementAudioSourceNode | null>(null)
const sourceElementRef = useRef<HTMLAudioElement | null>(null)
useEffect(() => {
if (!enabled) return
const ctx = audioCtxRef.current
if (!ctx) return
const audioEl = analysisAudioRef.current
if (!ctx || !audioEl) return
const gains = stemGainRefs.current || {}
const drumGain = gains.drums
const fallbackGain = Object.values(gains)[0]
let tapNode: AudioNode | null = drumGain || fallbackGain || null
if (!tapNode) {
const audioEl = singleAudioRef.current
if (audioEl) {
try {
mediaSourceRef.current = mediaSourceRef.current ?? ctx.createMediaElementSource(audioEl)
tapNode = mediaSourceRef.current
} catch {
tapNode = null
}
if (sourceElementRef.current && sourceElementRef.current !== audioEl) {
try {
mediaSourceRef.current?.disconnect()
} catch {
// noop
}
mediaSourceRef.current = null
sourceElementRef.current = null
}
if (!tapNode) return
try {
if (!mediaSourceRef.current) {
mediaSourceRef.current = ctx.createMediaElementSource(audioEl)
sourceElementRef.current = audioEl
}
} catch {
return
}
const mediaSource = mediaSourceRef.current
if (!mediaSource) return
const analyser = ctx.createAnalyser()
analyser.fftSize = 512
analyser.smoothingTimeConstant = 0.82
analyser.fftSize = 1024
analyser.smoothingTimeConstant = 0.8
analyser.minDecibels = -100
analyser.maxDecibels = -30
const silentGain = ctx.createGain()
silentGain.gain.value = 0
mediaSource.connect(analyser)
// Non-intrusive tap: this does not rewire the existing audible output path.
tapNode.connect(analyser)
analyser.connect(silentGain)
silentGain.connect(ctx.destination)
const byteFreq = new Uint8Array(analyser.frequencyBinCount)
featuresRef.current.freqData = byteFreq
let rafId: number | null = null
let mounted = true
let prevEnergy = 0
let lastBeatAt = 0
const applyMeydaFrame = (frame: MeydaFeatureFrame) => {
if (!mounted) return
if (typeof frame.rms === 'number' && Number.isFinite(frame.rms)) {
featuresRef.current.rms = clamp01(frame.rms)
}
if (typeof frame.energy === 'number' && Number.isFinite(frame.energy)) {
const e = clamp01(frame.energy)
featuresRef.current.energy = e
prevEnergy = e
}
if (typeof frame.spectralCentroid === 'number' && Number.isFinite(frame.spectralCentroid)) {
featuresRef.current.spectralCentroid = Math.max(0, frame.spectralCentroid)
}
if (typeof frame.loudness === 'number' && Number.isFinite(frame.loudness)) {
featuresRef.current.loudness = Math.max(0, frame.loudness)
} else if (
frame.loudness &&
typeof frame.loudness !== 'number' &&
typeof frame.loudness.total === 'number' &&
Number.isFinite(frame.loudness.total)
) {
featuresRef.current.loudness = Math.max(0, frame.loudness.total)
}
const extractor = new MuonFeatureExtractor()
const bandPeaks: Record<keyof MuonFrequencyBands, number> = {
sub: 0.001,
bass: 0.001,
lowMid: 0.001,
mid: 0.001,
highMid: 0.001,
presence: 0.001,
brilliance: 0.001,
}
let meydaAnalyzer: MeydaAnalyzerLike | null = null
let mounted = true
// Keep Meyda optional. Using vite-ignore prevents hard dev-server failure
// when a runtime image is missing node_modules/meyda.
const meydaModuleId = 'meyda'
void import(/* @vite-ignore */ meydaModuleId)
.then((mod) => {
if (!mounted) return
const imported = mod as unknown as {
default?: MeydaLike
createMeydaAnalyzer?: MeydaLike['createMeydaAnalyzer']
}
const meyda: MeydaLike | undefined = imported.default
?? (typeof imported.createMeydaAnalyzer === 'function' ? (imported as MeydaLike) : undefined)
if (!meyda || typeof meyda.createMeydaAnalyzer !== 'function') return
meydaAnalyzer = meyda.createMeydaAnalyzer({
audioContext: ctx,
source: tapNode as AudioNode,
bufferSize: 512,
featureExtractors: ['rms', 'energy', 'loudness', 'spectralCentroid'],
callback: applyMeydaFrame,
})
meydaAnalyzer?.start?.()
})
.catch(() => {
// Optional dependency path: FFT mode still works if Meyda is unavailable.
})
const tick = () => {
extractor.start(analyser, ctx, (frame) => {
if (!mounted) return
rafId = window.requestAnimationFrame(tick)
if (!isPlayingRef.current) {
featuresRef.current.bass = 0
featuresRef.current.treble = 0
featuresRef.current.rms = 0
featuresRef.current.energy = 0
featuresRef.current.loudness = 0
featuresRef.current.drumIntensity = 0
featuresRef.current.beatDetected = false
featuresRef.current.updatedAt = Date.now()
return
}
analyser.getByteFrequencyData(byteFreq)
const rawBands = frame.frequencyBands
const norm = {
sub: 0,
bass: 0,
lowMid: 0,
mid: 0,
highMid: 0,
presence: 0,
brilliance: 0,
}
// Bug 2 fix: low bins (0–9) → bass, high bins (last 56 of 256) → treble.
// Original muon source had these reversed (upperHalfArray → baseFr, etc.).
const bass = avgSlice(byteFreq, 0, 10) / 255
const treble = avgSlice(byteFreq, Math.max(0, byteFreq.length - 56), byteFreq.length) / 255
const energy = byteFreq.length ? (avgSlice(byteFreq, 0, byteFreq.length) / 255) : 0
const rms = featuresRef.current.rms > 0 ? featuresRef.current.rms : energy
for (const key of BAND_KEYS) {
bandPeaks[key] = Math.max(bandPeaks[key] * PEAK_DECAY, rawBands[key])
norm[key] = clamp01(rawBands[key] / bandPeaks[key])
}
featuresRef.current.bass = clamp01(bass)
featuresRef.current.treble = clamp01(treble)
featuresRef.current.energy = clamp01(energy)
featuresRef.current.rms = clamp01(rms)
featuresRef.current.drumIntensity = deriveDrumIntensity(featuresRef.current.bass, featuresRef.current.treble, featuresRef.current.rms)
const rms = clamp01(frame.rms)
const treble = clamp01((norm.presence + norm.brilliance) / 2)
const now = performance.now()
const deltaEnergy = featuresRef.current.energy - prevEnergy
const cooldownMs = 150
const isBeat = deltaEnergy > 0.08 && featuresRef.current.bass > 0.25 && now - lastBeatAt > cooldownMs
featuresRef.current.beatDetected = isBeat
if (isBeat) lastBeatAt = now
prevEnergy = featuresRef.current.energy
featuresRef.current.updatedAt = Date.now()
}
rafId = window.requestAnimationFrame(tick)
featuresRef.current = {
bass: norm.bass,
treble,
rms,
energy: clamp01(frame.energy),
loudness: rms,
spectralCentroid: frame.spectralCentroid,
drumIntensity: deriveDrumIntensity(norm.sub, norm.bass, rms),
beatDetected: frame.beatInfo.isBeat,
frequencyBands: norm,
beatInfo: frame.beatInfo,
patternInfo: frame.patternInfo,
spectralRolloff: frame.spectralRolloff,
spectralFlatness: frame.spectralFlatness,
zcr: frame.zcr,
freqData: frame.floatFreqData,
updatedAt: Date.now(),
}
})
return () => {
mounted = false
if (rafId !== null) window.cancelAnimationFrame(rafId)
meydaAnalyzer?.stop?.()
extractor.stop()
try {
tapNode?.disconnect(analyser)
mediaSource.disconnect(analyser)
} catch {
// noop
}
@@ -246,13 +235,8 @@ export function useAudioFeatures({
} catch {
// noop
}
try {
silentGain.disconnect()
} catch {
// noop
}
}
}, [audioCtxRef, stemGainRefs, singleAudioRef, isPlayingRef, enabled])
}, [audioCtxRef, analysisAudioRef, isPlayingRef, enabled, analysisSourceKey])
return featuresRef
}
+36
View File
@@ -102,6 +102,27 @@ function normalizeSekInnerPath(value?: string | null): string {
.replace(/^\/+/, '')
}
function rankSekAnalysisStem(name?: string): number {
const key = (name || '').trim().toLowerCase()
if (!key) return 10
if (/\b(drums?|kick|percussion)\b/.test(key)) return 0
if (/\b(leading|lead|vocals?|vox|voice|main vocal)\b/.test(key)) return 1
if (/\b(backing|music|instrumental|mix)\b/.test(key)) return 2
return 10
}
function pickSekAnalysisStem(stems?: Array<{ name?: string; file?: string }>) {
if (!stems?.length) return undefined
return stems
.filter((stem) => !!normalizeSekInnerPath(stem.file))
.sort((a, b) => {
const ra = rankSekAnalysisStem(a.name)
const rb = rankSekAnalysisStem(b.name)
if (ra !== rb) return ra - rb
return (a.name || '').localeCompare(b.name || '')
})[0]
}
export default function KaraokePage() {
const location = useLocation()
const navigate = useNavigate()
@@ -214,6 +235,20 @@ export default function KaraokePage() {
}
}, [detail, isSekMode, sekManifest, sekFolder])
const analysisAudioUrl = useMemo(() => {
if (isSekMode) {
const analysisStem = pickSekAnalysisStem(sekManifest?.manifest?.contents?.stems)
const stemPath = normalizeSekInnerPath(analysisStem?.file)
if (!sekManifest || !stemPath) return undefined
return sekAssetUrl(sekManifest.id, stemPath, sekFolder || undefined)
}
if (!detail) return undefined
const sourceFile = detail.sep_job.files.find((f) => f.type === 'source')
if (!sourceFile) return undefined
return downloadUrl(detail.sep_job.id, sourceFile.name)
}, [detail, isSekMode, sekManifest, sekFolder])
// ── Pixabay stock video fallback ───────────────────────────────────────────
// DISABLED: visualizers are now the default backdrop for SEK packages without
// an embedded video. Set needsStockVideo to the expression below to re-enable:
@@ -462,6 +497,7 @@ export default function KaraokePage() {
lyricsTracks={lyricsTracks}
stems={safeStems.length > 0 ? safeStems : undefined}
audioUrl={safeStems.length === 0 && sourceMedia ? sourceMedia.url : undefined}
analysisAudioUrl={analysisAudioUrl}
videoUrl={sourceMedia?.isVideo ? sourceMedia.url : undefined}
videoPlaylist={stockVideoPlaylist.length > 0 ? stockVideoPlaylist : undefined}
videoPlaybackMode={sourceMedia?.isVideo ? 'synced' : 'playlist'}
+36
View File
@@ -107,6 +107,27 @@ function normalizeSekInnerPath(value?: string | null): string {
.replace(/^\/+/, '')
}
function rankSekAnalysisStem(name?: string): number {
const key = (name || '').trim().toLowerCase()
if (!key) return 10
if (/\b(drums?|kick|percussion)\b/.test(key)) return 0
if (/\b(leading|lead|vocals?|vox|voice|main vocal)\b/.test(key)) return 1
if (/\b(backing|music|instrumental|mix)\b/.test(key)) return 2
return 10
}
function pickSekAnalysisStem(stems?: Array<{ name?: string; file?: string }>) {
if (!stems?.length) return undefined
return stems
.filter((stem) => !!normalizeSekInnerPath(stem.file))
.sort((a, b) => {
const ra = rankSekAnalysisStem(a.name)
const rb = rankSekAnalysisStem(b.name)
if (ra !== rb) return ra - rb
return (a.name || '').localeCompare(b.name || '')
})[0]
}
export default function KaraokeTVPage() {
const navigate = useNavigate()
const {
@@ -235,6 +256,20 @@ export default function KaraokeTVPage() {
}
}, [currentItem, detail, isSekMode, sekManifest, sekFolder])
const analysisAudioUrl = useMemo(() => {
if (isSekMode) {
const analysisStem = pickSekAnalysisStem(sekManifest?.manifest?.contents?.stems)
const stemPath = normalizeSekInnerPath(analysisStem?.file)
if (!sekManifest || !stemPath) return undefined
return sekAssetUrl(sekManifest.id, stemPath, sekFolder || undefined)
}
if (!detail) return undefined
const sourceFile = detail.sep_job.files.find((f) => f.type === 'source')
if (!sourceFile) return undefined
return downloadUrl(detail.sep_job.id, sourceFile.name)
}, [detail, isSekMode, sekManifest, sekFolder])
// ── Pixabay stock video fallback ───────────────────────────────────
// DISABLED: visualizers are now the default backdrop for SEK packages without
// an embedded video. Set needsStockVideo to the expression below to re-enable:
@@ -531,6 +566,7 @@ export default function KaraokeTVPage() {
lyricsTracks={lyricsTracks}
stems={safeStems.length > 0 ? safeStems : undefined}
audioUrl={safeStems.length === 0 && sourceMedia ? sourceMedia.url : undefined}
analysisAudioUrl={analysisAudioUrl}
videoUrl={sourceMedia?.isVideo ? sourceMedia.url : undefined}
videoPlaylist={stockVideoPlaylist.length > 0 ? stockVideoPlaylist : undefined}
videoPlaybackMode={sourceMedia?.isVideo ? 'synced' : 'playlist'}
+11 -9
View File
@@ -1,5 +1,5 @@
import { useEffect, useMemo, useRef, useState } from 'react'
import { Disc3, Heart, Loader2, Zap } from 'lucide-react'
import { Heart, Loader2, Zap } from 'lucide-react'
import { Link, useLocation, useNavigate, useParams } from 'react-router-dom'
import { useQuery } from '@tanstack/react-query'
import { useConvertStatus } from '../hooks/useConvertStatus'
@@ -9,7 +9,8 @@ import { useQueue } from '../context/QueueContext'
import { trackMediaFromSong } from '../utils/mediaAdapters'
import { launchMediaItem } from '../utils/queueLauncher'
import MediaQueueActions from '../components/MediaQueueActions'
import { getProjectKaraokeConfig, getSubsonicSong, patchProjectKaraokeConfig, subsonicCoverArtUrl } from '../api/client'
import VisualizerPreviewWidget from '../components/VisualizerPreviewWidget'
import { getProjectKaraokeConfig, getSubsonicSong, patchProjectKaraokeConfig, subsonicCoverArtUrl, subsonicStreamUrl } from '../api/client'
import type { SubsonicSong } from '../types'
import { reportNonFatal } from '../utils/nonFatal'
@@ -65,6 +66,7 @@ export default function TrackDetailPage() {
const { statusBySongId, convertSong } = useConvertStatus(song ? [song] : [])
const { playNow, playNext, addToQueue } = useQueue()
const coverArtUrl = song?.coverArt ? subsonicCoverArtUrl(song.coverArt, 800) : ''
const audioStreamUrl = useMemo(() => (song ? subsonicStreamUrl(song.id) : ''), [song?.id])
const matchedProject = song ? findProjectForSong(song) : null
@@ -154,13 +156,13 @@ export default function TrackDetailPage() {
return (
<div className="p-6 space-y-6">
<div className="flex items-start gap-4">
<div className="h-24 w-24 shrink-0 overflow-hidden rounded-lg bg-gray-800 flex items-center justify-center">
{coverArtUrl ? (
<img src={coverArtUrl} alt={`${song.album} cover art`} className="h-full w-full object-cover" />
) : (
<Disc3 className="h-10 w-10 text-gray-500" />
)}
</div>
<VisualizerPreviewWidget
audioUrl={audioStreamUrl}
coverArtUrl={coverArtUrl || undefined}
videoUrl={videoUrlInput.trim() || undefined}
songTitle={song.title}
className="w-48 shrink-0 overflow-hidden rounded-lg"
/>
<div className="min-w-0">
<Link to={TRACKS_ROUTE} className="text-xs text-gray-400 hover:text-gray-300">← Back to tracks</Link>
<h1 className="text-2xl font-bold mt-1 truncate">{song.title}</h1>
+2 -28
View File
@@ -1,28 +1,2 @@
declare module 'meyda' {
export interface MeydaFeatures {
rms?: number
energy?: number
loudness?: number | { total?: number }
spectralCentroid?: number
}
export interface MeydaAnalyzer {
start?: () => void
stop?: () => void
}
export interface MeydaCreateOptions {
audioContext: AudioContext
source: AudioNode
bufferSize: number
featureExtractors: string[]
callback: (features: MeydaFeatures) => void
}
export interface MeydaApi {
createMeydaAnalyzer: (opts: MeydaCreateOptions) => MeydaAnalyzer
}
const Meyda: MeydaApi
export default Meyda
}
// Meyda runtime usage was removed in favor of MuonFeatureExtractor.
// File intentionally left without module declarations.
+4
View File
@@ -0,0 +1,4 @@
declare module '*.mjs' {
const mod: any
export default mod
}
+3
View File
@@ -0,0 +1,3 @@
declare module 'webgl-fluid' {
export default function WebGLFluid(canvas: HTMLCanvasElement, options?: any): void
}