Myriad changes.
This commit is contained in:
1 parent
cbe946925f
commit
ee851bc9ae
32 files changed
+7859
-888
No files matched your search
@@ -0,0 +1,534 @@
|
||||
# Muon Audio Analyzer Integration Plan
|
||||
|
||||
Date: 2026-04-13
|
||||
Scope: Replace the current single-stem/Meyda feature extraction in `useAudioFeatures` with the
|
||||
Muon Audio Analysis `FeatureExtractor` approach, tapping a hidden analysis `<audio>` element
|
||||
fed by `analysisAudioUrl ?? audioUrl` (full-mix source when available; stem fallback in SEK).
|
||||
|
||||
Reference repo: https://github.com/najafmohammed/Muon-Audio-Analysis
|
||||
Status: private, not published to npm — the `FeatureExtractor` class must be inlined.
|
||||
|
||||
---
|
||||
|
||||
## Integrated decisions from related plans (2026-04-13)
|
||||
|
||||
This plan incorporates constraints already established in:
|
||||
|
||||
- `2026-04-13_Visualizer_DiagnosisAndFixes_Plan.md`
|
||||
- `2026-04-13_TrackDetail_VisualizerPreview_Plan.md`
|
||||
- `2026-04-13_SEKAsset_RangeRequest_Plan.md`
|
||||
|
||||
Net effect:
|
||||
|
||||
1. Analysis source is **not** `audioUrl`-only anymore; it is `analysisAudioUrl ?? audioUrl`.
|
||||
2. `analysisAudioUrl` must be threaded through `KaraokePerformanceMode` into
|
||||
`LyricsLivePreview` and included in `analysisSourceKey`.
|
||||
3. In SEK mode, where no explicit full-mix source exists in the manifest, analysis may use a
|
||||
ranked stem fallback (initially first/leading/drums stem strategy).
|
||||
4. Hidden analysis seek sync depends on working HTTP range support for SEK assets
|
||||
(`/api/karaoke-sek/packages/{id}/asset`) so `currentTime` jumps are reliable.
|
||||
|
||||
---
|
||||
|
||||
## Current problems
|
||||
|
||||
1. **Wrong source** — `useAudioFeatures` taps only the `drums` stem gain node (or first stem as
|
||||
fallback), so the visualizer is deaf to vocals, guitar, piano, etc.
|
||||
|
||||
2. **Mismatched algorithm** — the current extraction is a patchwork of Meyda (optional, for
|
||||
rms/energy/spectralCentroid) and a manual FFT slice (for bass/treble/beat). This is not the
|
||||
Muon approach.
|
||||
|
||||
3. **Low FFT resolution** — current `fftSize = 512` gives 256 bins; Muon uses `fftSize = 1024`
|
||||
(512 bins) for better low-frequency resolution.
|
||||
|
||||
4. **Limited feature set** — only 7 output fields. Muon outputs 7 frequency bands, BPM, beat
|
||||
phase, pattern detection (drop/buildup/silence), spectral rolloff, flatness, ZCR.
|
||||
|
||||
5. **Latent audio-muting bug** — the single-audio fallback path calls
|
||||
`createMediaElementSource(audioEl)` (which reroutes the element through Web Audio) but only
|
||||
connects it onward through `silentGain(0) → destination`, muting the audio.
|
||||
|
||||
---
|
||||
|
||||
## Why single-source (original song file) rather than all-stem mix
|
||||
|
||||
| Concern | Single source | All stems mixed |
|
||||
|---|---|---|
|
||||
| Tap nodes | 1 always | N (one per stem, grows with separator) |
|
||||
| Effect cleanup | Disconnect 1 node | Track and disconnect N nodes |
|
||||
| Effect deps | Can drop `stemGainRefs` | Must keep `stemGainRefs` in deps |
|
||||
| Stems-load-late race | No race — audio element always exists | Effect must restart when stem graph is ready |
|
||||
| Beat detection quality | Full mix = richest low-freq signal | Same once all stems connected |
|
||||
| Mute/solo sensitivity | Visualizer ignores user muting | Visualizer responds to audible mix |
|
||||
|
||||
The last row is the only meaningful trade-off. Mute/solo affecting the visualizer is arguably
|
||||
wrong behavior anyway — the visual should reflect the song's energy, not the listener's temporary
|
||||
mixing choices.
|
||||
|
||||
---
|
||||
|
||||
## Target state
|
||||
|
||||
```
|
||||
analysisAudioRef (<audio>, hidden, src = analysisAudioUrl ?? audioUrl) ──► MediaElementAudioSource
|
||||
└──► AnalyserNode (fftSize 1024)
|
||||
└──► MuonFeatureExtractor.extractLoop()
|
||||
│
|
||||
└──► AudioFeatures (expanded interface)
|
||||
├── bass, treble, rms, drumIntensity, beatDetected (kept, remapped)
|
||||
├── frequencyBands { sub, bass, lowMid, mid, highMid, presence, brilliance }
|
||||
├── beatInfo { isBeat, bpm, confidence, timeSinceLastBeat, beatPhase }
|
||||
└── patternInfo { isBeatDrop, isBuildUp, isSilence, intensity, energyDelta }
|
||||
```
|
||||
|
||||
In multi-stem mode, stems continue to drive audible output through their GainNodes as before.
|
||||
In single-audio mode, the visible player audio element remains the audible source.
|
||||
`analysisAudioRef` is analysis-only and is **never connected to `ctx.destination`**.
|
||||
|
||||
`analysisAudioRef` is synchronized to the active transport clock (single-track audio, stem anchor,
|
||||
or video clock) so analysis stays phase-aligned with what the listener hears.
|
||||
|
||||
---
|
||||
|
||||
## Files affected
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `frontend/src/audio/MuonFeatureExtractor.ts` | **New** — adapted FeatureExtractor class |
|
||||
| `frontend/src/hooks/useAudioFeatures.ts` | **Modified** — single-source tap + new extractor |
|
||||
| `frontend/src/components/LyricsLivePreview.tsx` | **Modified** — add hidden always-mounted full-mix analysis audio + sync strategy |
|
||||
| `frontend/src/components/KaraokePerformanceMode.tsx` | **Modified** — add/forward `analysisAudioUrl` |
|
||||
| `frontend/src/pages/KaraokeTV.tsx` | **Modified** — derive/pass `analysisAudioUrl` |
|
||||
| `frontend/src/pages/Karaoke.tsx` | **Modified** — derive/pass `analysisAudioUrl` |
|
||||
| `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx` | No immediate change (backward-compatible fields preserved) |
|
||||
| `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx` | No immediate change |
|
||||
| `SUMMARIES (Claude).md` | Append entry after implementation |
|
||||
|
||||
Meyda optional import can be removed once the new extractor is in place.
|
||||
|
||||
---
|
||||
|
||||
## Step 1 — New file: `frontend/src/audio/MuonFeatureExtractor.ts`
|
||||
|
||||
Inline the `FeatureExtractor` class from the Muon Audio Analysis repo, stripped of all Zustand
|
||||
store dependencies and adapted to return features via a callback instead of writing to a store.
|
||||
|
||||
### API
|
||||
|
||||
```typescript
|
||||
export interface MuonFrequencyBands {
|
||||
sub: number // 20–60 Hz
|
||||
bass: number // 60–250 Hz
|
||||
lowMid: number // 250–500 Hz
|
||||
mid: number // 500–2 000 Hz
|
||||
highMid: number // 2 000–4 000 Hz
|
||||
presence: number // 4 000–6 000 Hz
|
||||
brilliance: number // 6 000–20 000 Hz
|
||||
}
|
||||
|
||||
export interface MuonBeatInfo {
|
||||
isBeat: boolean
|
||||
bpm: number
|
||||
confidence: number
|
||||
timeSinceLastBeat: number
|
||||
beatPhase: number // 0–1 position in current beat cycle
|
||||
}
|
||||
|
||||
export interface MuonPatternInfo {
|
||||
isBeatDrop: boolean
|
||||
isBuildUp: boolean
|
||||
isSilence: boolean
|
||||
intensity: number // 0–1 overall
|
||||
energyDelta: number // rate of change
|
||||
}
|
||||
|
||||
export interface MuonFrame {
|
||||
rms: number
|
||||
energy: number
|
||||
spectralCentroid: number
|
||||
spectralRolloff: number
|
||||
spectralFlatness: number
|
||||
zcr: number
|
||||
frequencyBands: MuonFrequencyBands // raw linear energy values (not yet normalized)
|
||||
beatInfo: MuonBeatInfo
|
||||
patternInfo: MuonPatternInfo
|
||||
floatFreqData: Float32Array // raw dB values from analyser (copy, not aliased)
|
||||
floatTimeData: Float32Array // raw waveform (copy)
|
||||
}
|
||||
|
||||
export class MuonFeatureExtractor {
|
||||
private isRunning = false
|
||||
private animationFrameId: number | null = null
|
||||
private frequencyData!: Float32Array
|
||||
private timeData!: Float32Array
|
||||
private energyHistory: number[] = []
|
||||
private lastBeatTime = 0
|
||||
private beatThreshold = 1.5
|
||||
private bpm = 0
|
||||
private beatTimes: number[] = []
|
||||
|
||||
start(analyser: AnalyserNode, ctx: AudioContext, onFrame: (f: MuonFrame) => void): void
|
||||
stop(): void
|
||||
setBeatThreshold(threshold: number): void // 0.0–3.0, default 1.5
|
||||
reset(): void
|
||||
}
|
||||
```
|
||||
|
||||
### Implementation notes
|
||||
|
||||
- Copy `extractFeatures()`, `extractFrequencyBands()`, `detectBeat()`, `detectPatterns()`,
|
||||
`calculateLocalAverageAndVariance()` verbatim from the Muon repo source
|
||||
(`src/audio/FeatureExtractor.ts`).
|
||||
- Remove all `useAudioStore.getState()` calls; replace with a single `onFrame(frame)` call at
|
||||
the end of `extractLoop`.
|
||||
- The RAF loop is `requestAnimationFrame`-based — no change needed.
|
||||
- Define `AUDIO_CONFIG` constants (FFT_SIZE 1024, SMOOTHING 0.8, MIN_DECIBELS -100,
|
||||
MAX_DECIBELS -30) and `FREQUENCY_BANDS` Hz ranges as local module-level constants — no import
|
||||
from the Muon repo's `@core/constants`.
|
||||
- `ENERGY_HISTORY_SIZE = 43` (≈1 second at 60 fps, same as Muon source).
|
||||
- Return `floatFreqData` and `floatTimeData` as **copies** (`new Float32Array(this.frequencyData)`)
|
||||
so the caller can store them without aliasing issues.
|
||||
|
||||
---
|
||||
|
||||
## Step 2 — Modified: `frontend/src/hooks/useAudioFeatures.ts`
|
||||
|
||||
### 2a — Expand the `AudioFeatures` interface
|
||||
|
||||
```typescript
|
||||
export interface AudioFeatures {
|
||||
// ── Backward-compatible convenience fields (remapped from Muon data) ──
|
||||
bass: number // frequencyBands.bass normalized 0–1
|
||||
treble: number // (frequencyBands.presence + frequencyBands.brilliance) / 2, normalized 0–1
|
||||
rms: number // 0–1
|
||||
energy: number // 0–1
|
||||
loudness: number // kept for API compat; set equal to rms
|
||||
spectralCentroid: number
|
||||
drumIntensity: number // derived from sub + bass bands — see §2b
|
||||
beatDetected: boolean // beatInfo.isBeat
|
||||
|
||||
// ── Muon full feature set ──
|
||||
frequencyBands: {
|
||||
sub: number // normalized 0–1
|
||||
bass: number
|
||||
lowMid: number
|
||||
mid: number
|
||||
highMid: number
|
||||
presence: number
|
||||
brilliance: number
|
||||
}
|
||||
beatInfo: {
|
||||
isBeat: boolean
|
||||
bpm: number
|
||||
confidence: number
|
||||
timeSinceLastBeat: number
|
||||
beatPhase: number
|
||||
}
|
||||
patternInfo: {
|
||||
isBeatDrop: boolean
|
||||
isBuildUp: boolean
|
||||
isSilence: boolean
|
||||
intensity: number
|
||||
energyDelta: number
|
||||
}
|
||||
spectralRolloff: number
|
||||
spectralFlatness: number
|
||||
zcr: number
|
||||
|
||||
// ── Raw buffers ──
|
||||
freqData: Float32Array // dB values (was Uint8Array — see §2f)
|
||||
updatedAt: number
|
||||
}
|
||||
```
|
||||
|
||||
### 2b — `drumIntensity` re-derivation
|
||||
|
||||
The existing formula used the drums stem signal to isolate kick energy. With the full-mix
|
||||
analyser, `frequencyBands.sub` (20–60 Hz) is the cleanest kick/sub proxy. Replace:
|
||||
|
||||
```typescript
|
||||
function deriveDrumIntensity(sub: number, bass: number, rms: number): number {
|
||||
// sub and bass are already normalized 0–1 (auto-gain applied)
|
||||
const expSub = 1.001 * (1 + sub * 2) - 1.001
|
||||
const expBass = 1.001 * (1 + bass * 2) - 1.001
|
||||
const raw = (expSub * 0.50 + expBass * 0.38 + rms * 0.12) / 2.0
|
||||
return clamp01(Math.pow(clamp01(raw), 0.82))
|
||||
}
|
||||
```
|
||||
|
||||
`sub` takes 0.50 weight (up from `treble`'s 0.08 in the old formula) because sub-bass is where
|
||||
kick drums live.
|
||||
|
||||
### 2c — Band normalization (auto-gain)
|
||||
|
||||
`MuonFeatureExtractor.extractFrequencyBands()` returns raw RMS linear energy values, not 0–1.
|
||||
Normalize with a **running peak + slow decay**:
|
||||
|
||||
```typescript
|
||||
// Lives outside the useEffect (or inside as a closure) — persists across frames.
|
||||
const bandPeaks = {
|
||||
sub: 0.001, bass: 0.001, lowMid: 0.001, mid: 0.001,
|
||||
highMid: 0.001, presence: 0.001, brilliance: 0.001,
|
||||
}
|
||||
const PEAK_DECAY = 0.9998 // per-frame (~0.01 dB/s at 60 fps — very slow)
|
||||
|
||||
// Inside onFrame, per band:
|
||||
bandPeaks.bass = Math.max(bandPeaks.bass * PEAK_DECAY, rawBands.bass)
|
||||
const normalizedBass = clamp01(rawBands.bass / bandPeaks.bass)
|
||||
```
|
||||
|
||||
Peaks start at `0.001` to prevent divide-by-zero on startup. On the first loud frame they snap
|
||||
to the true peak, then decay slowly — self-calibrating to the mix level within ~2 seconds.
|
||||
|
||||
### 2d — Single-source tap via hidden analysis audio (replaces all current tap logic)
|
||||
|
||||
Remove the entire drum/fallback/single-audio branch. Replace with:
|
||||
|
||||
```typescript
|
||||
const audioEl = analysisAudioRef.current
|
||||
if (!audioEl) return
|
||||
|
||||
// createMediaElementSource can only be called once per element.
|
||||
// Re-use the cached node if the effect re-runs (e.g., on enabled toggle).
|
||||
try {
|
||||
mediaSourceRef.current = mediaSourceRef.current ?? ctx.createMediaElementSource(audioEl)
|
||||
} catch {
|
||||
return // AudioContext is closed or element already owned by another context
|
||||
}
|
||||
|
||||
const mediaSource = mediaSourceRef.current
|
||||
|
||||
const analyser = ctx.createAnalyser()
|
||||
analyser.fftSize = 1024
|
||||
analyser.smoothingTimeConstant = 0.8
|
||||
analyser.minDecibels = -100
|
||||
analyser.maxDecibels = -30
|
||||
|
||||
// Analysis-only path: never connect this element to destination.
|
||||
mediaSource.connect(analyser)
|
||||
```
|
||||
|
||||
**§2d-note — Hidden analysis element + sync strategy (`LyricsLivePreview.tsx`):**
|
||||
|
||||
Add a dedicated ref that is mounted in both single and multi-stem sessions:
|
||||
|
||||
```tsx
|
||||
const analysisAudioRef = useRef<HTMLAudioElement>(null)
|
||||
```
|
||||
|
||||
Render a hidden analysis element whenever `analysisAudioUrl || audioUrl` exists:
|
||||
|
||||
```tsx
|
||||
{(analysisAudioUrl || audioUrl) && (
|
||||
<audio
|
||||
ref={analysisAudioRef}
|
||||
src={analysisAudioUrl ?? audioUrl}
|
||||
preload="auto"
|
||||
crossOrigin="anonymous"
|
||||
muted
|
||||
style={{ display: 'none' }}
|
||||
/>
|
||||
)}
|
||||
```
|
||||
|
||||
Sync policy:
|
||||
- Mirror `currentTime` from active transport (`audioRef`, stem anchor, or `videoRef`).
|
||||
- Mirror `playbackRate`.
|
||||
- On play/resume: set time, then `analysisAudioRef.current?.play()`.
|
||||
- On pause/stop/end: `analysisAudioRef.current?.pause()`.
|
||||
- On seek: immediately set `analysisAudioRef.current.currentTime = seekTarget`.
|
||||
- In RAF sync loop, if drift exceeds ~40 ms, snap analysis time to transport time.
|
||||
|
||||
This guarantees one stable analysis source without audible doubling.
|
||||
|
||||
**Source policy integration (`KaraokeTV.tsx` / `Karaoke.tsx`):**
|
||||
|
||||
- Project mode: prefer source/original recording URL as `analysisAudioUrl`.
|
||||
- SEK mode: until true full-mix is available, derive `analysisAudioUrl` from a ranked stem
|
||||
fallback (leading/drums preferred; first stem acceptable initial implementation).
|
||||
- Always pass `analysisAudioUrl` through `KaraokePerformanceMode` into `LyricsLivePreview`.
|
||||
|
||||
**SEK seek prerequisite:**
|
||||
|
||||
For SEK fallback analysis to stay sync-accurate on seeks, the SEK asset endpoint must support
|
||||
HTTP byte ranges (`Accept-Ranges: bytes`, valid `206` responses, `416` handling for invalid
|
||||
ranges). This is covered by `2026-04-13_SEKAsset_RangeRequest_Plan.md` and is a dependency for
|
||||
reliable `analysisAudioRef.currentTime` snapping.
|
||||
```
|
||||
|
||||
### 2e — Cleanup
|
||||
|
||||
```typescript
|
||||
return () => {
|
||||
mounted = false
|
||||
extractor.stop()
|
||||
try { mediaSource.disconnect(analyser) } catch { /* noop */ }
|
||||
try { analyser.disconnect() } catch { /* noop */ }
|
||||
}
|
||||
```
|
||||
|
||||
The `mediaSourceRef` is preserved across effect re-runs (the `??=` creation guard above). Since
|
||||
the analysis element is never connected to destination, cleanup only detaches analyser links.
|
||||
|
||||
### 2f — Replace Meyda + tick
|
||||
|
||||
Remove the dynamic `import(/* @vite-ignore */ 'meyda')` block and the entire RAF `tick()`
|
||||
function. The `MuonFeatureExtractor` subsumes Meyda's role and runs its own RAF loop.
|
||||
|
||||
```typescript
|
||||
const extractor = new MuonFeatureExtractor()
|
||||
extractor.start(analyser, ctx, (frame) => {
|
||||
if (!mounted) return
|
||||
if (!isPlayingRef.current) {
|
||||
// Zero out reactive fields while paused; leave beatInfo/patternInfo/freqData as-is.
|
||||
featuresRef.current.bass = 0
|
||||
featuresRef.current.treble = 0
|
||||
featuresRef.current.rms = 0
|
||||
featuresRef.current.drumIntensity = 0
|
||||
featuresRef.current.beatDetected = false
|
||||
return
|
||||
}
|
||||
|
||||
// Normalize each band
|
||||
const b = frame.frequencyBands
|
||||
const keys = ['sub','bass','lowMid','mid','highMid','presence','brilliance'] as const
|
||||
const norm: Record<typeof keys[number], number> = {} as any
|
||||
for (const k of keys) {
|
||||
bandPeaks[k] = Math.max(bandPeaks[k] * PEAK_DECAY, b[k])
|
||||
norm[k] = clamp01(b[k] / bandPeaks[k])
|
||||
}
|
||||
|
||||
const rms = clamp01(frame.rms)
|
||||
const treble = clamp01((norm.presence + norm.brilliance) / 2)
|
||||
|
||||
featuresRef.current = {
|
||||
bass: norm.bass,
|
||||
treble,
|
||||
rms,
|
||||
energy: clamp01(frame.energy),
|
||||
loudness: rms,
|
||||
spectralCentroid: frame.spectralCentroid,
|
||||
drumIntensity: deriveDrumIntensity(norm.sub, norm.bass, rms),
|
||||
beatDetected: frame.beatInfo.isBeat,
|
||||
frequencyBands: norm,
|
||||
beatInfo: frame.beatInfo,
|
||||
patternInfo: frame.patternInfo,
|
||||
spectralRolloff: frame.spectralRolloff,
|
||||
spectralFlatness: frame.spectralFlatness,
|
||||
zcr: frame.zcr,
|
||||
freqData: frame.floatFreqData,
|
||||
updatedAt: Date.now(),
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### 2g — useEffect dependency array
|
||||
|
||||
Remove `stemGainRefs` from deps — the tap no longer depends on them. Keep
|
||||
`analysisSourceKey` in deps so source switches recreate analyser wiring:
|
||||
|
||||
```typescript
|
||||
}, [audioCtxRef, analysisAudioRef, isPlayingRef, enabled, analysisSourceKey])
|
||||
```
|
||||
|
||||
`stemGainRefs` can be removed from `UseAudioFeaturesArgs` entirely if it has no other use in the
|
||||
hook after the tap logic is replaced.
|
||||
|
||||
---
|
||||
|
||||
## Step 3 — Default values for new interface fields
|
||||
|
||||
```typescript
|
||||
const EMPTY_FREQ = new Float32Array(0) // was Uint8Array
|
||||
const EMPTY_BANDS = { sub: 0, bass: 0, lowMid: 0, mid: 0, highMid: 0, presence: 0, brilliance: 0 }
|
||||
const EMPTY_BEAT = { isBeat: false, bpm: 0, confidence: 0, timeSinceLastBeat: 0, beatPhase: 0 }
|
||||
const EMPTY_PAT = { isBeatDrop: false, isBuildUp: false, isSilence: true, intensity: 0, energyDelta: 0 }
|
||||
|
||||
const featuresRef = useRef<AudioFeatures>({
|
||||
bass: 0, treble: 0, rms: 0, energy: 0, loudness: 0,
|
||||
spectralCentroid: 0, spectralRolloff: 0, spectralFlatness: 0, zcr: 0,
|
||||
drumIntensity: 0, beatDetected: false,
|
||||
frequencyBands: EMPTY_BANDS,
|
||||
beatInfo: EMPTY_BEAT,
|
||||
patternInfo: EMPTY_PAT,
|
||||
freqData: EMPTY_FREQ,
|
||||
updatedAt: 0,
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 4 — Canvas files (backward-compatible; selected Muon enhancements already landed)
|
||||
|
||||
`MuonVisualizerCanvas` and `FluidVisualizerCanvas` read: `f.bass`, `f.treble`, `f.rms`,
|
||||
`f.drumIntensity`, `f.beatDetected`. All five are backward-compatible remapped fields in the
|
||||
expanded interface — no changes needed in either canvas file.
|
||||
|
||||
Current visualizer integration is backward-compatible and already consumes compatibility fields.
|
||||
Additionally, targeted Muon enhancements now exist in canvas logic (sub-band weighting,
|
||||
`beatPhase` shaping, and drop-trigger accenting), so these are no longer purely “future”.
|
||||
|
||||
Still optional for later parity work:
|
||||
- tighter phase-locked geometry states derived from `beatInfo.beatPhase`
|
||||
- richer transition choreography keyed on `patternInfo` sequences (build-up → drop)
|
||||
- broader usage of normalized per-band vectors across particle subsystems
|
||||
|
||||
---
|
||||
|
||||
## Step 5 — Remove Meyda (cleanup after verification)
|
||||
|
||||
```bash
|
||||
grep -r "meyda" frontend/src/ --include="*.ts" --include="*.tsx"
|
||||
```
|
||||
|
||||
If the only remaining reference was the deleted dynamic import, run:
|
||||
|
||||
```bash
|
||||
cd frontend && npm uninstall meyda
|
||||
```
|
||||
|
||||
Also remove stale type-only leftovers if unused:
|
||||
|
||||
- `frontend/src/types/meyda.d.ts`
|
||||
|
||||
Keep it only if another package still imports Meyda types.
|
||||
|
||||
---
|
||||
|
||||
## Implementation order
|
||||
|
||||
1. Write `frontend/src/audio/MuonFeatureExtractor.ts`
|
||||
2. Add `analysisAudioRef` to `UseAudioFeaturesArgs`; update call-site in `LyricsLivePreview.tsx`
|
||||
3. Expand `AudioFeatures` interface and initial ref value
|
||||
4. Add hidden always-mounted full-mix analysis `<audio>` in `LyricsLivePreview.tsx`
|
||||
5. Implement analysis sync strategy (play/pause/seek/time drift correction)
|
||||
6. Replace tap logic with single-source analysis routing (`mediaSource → analyser` only)
|
||||
7. Add `bandPeaks` / `PEAK_DECAY` normalization state
|
||||
8. Replace Meyda import + `tick()` with `extractor.start()` + `onFrame` callback
|
||||
9. Update `deriveDrumIntensity` signature
|
||||
10. Add/propagate `analysisAudioUrl` (`KaraokeTV`/`Karaoke` → `KaraokePerformanceMode` → `LyricsLivePreview`)
|
||||
11. Ensure `analysisSourceKey` tracks `analysisAudioUrl ?? audioUrl ?? null`
|
||||
12. Trim `stemGainRefs` from hook args if no longer needed
|
||||
13. Run `tsc --noEmit`
|
||||
14. Verify SEK seeking over `/karaoke-sek/packages/{id}/asset` still returns proper `206` on range probes
|
||||
15. Remove Meyda if clean
|
||||
16. Update `SUMMARIES (Claude).md`
|
||||
|
||||
---
|
||||
|
||||
## Risks and mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| Audio doubling in multi-stem mode | Analysis element is never connected to `ctx.destination` (`mediaSource → analyser` only). |
|
||||
| `createMediaElementSource` throws if audio element already owned | Wrap in try/catch; bail early. The `mediaSourceRef` guard (`??=`) prevents double-creation across effect re-runs. |
|
||||
| Analysis and audible transport drift apart | Mirror `playbackRate`, sync on play/seek, and snap when drift exceeds ~40 ms. |
|
||||
| Analysis source missing in stem sessions (`audioUrl` undefined) | Use `analysisAudioUrl ?? audioUrl`; derive `analysisAudioUrl` in page-level containers for both project and SEK modes. |
|
||||
| Band normalization near-zero for first frames | Peaks start at 0.001; clamp01 never goes negative. Self-calibrates within ~2 seconds of loud audio. |
|
||||
| `freqData` type change (`Uint8Array` → `Float32Array`) breaks a consumer | Audit: `grep -r "freqData" frontend/src/`. Values are now dB floats (-100 to 0) rather than 0–255 integers — update any display code. |
|
||||
| Extractor RAF loop burns CPU while paused | `onFrame` early-returns if `!isPlayingRef.current`, zeroing reactive fields. The extractor RAF still ticks but the callback is almost free when returning immediately. Alternatively call `extractor.stop()` on pause and `extractor.start()` on resume. |
|
||||
| `analysisAudioRef` is null on first render | `if (!audioEl) return` guard in useEffect; hidden element mounts when `analysisAudioUrl || audioUrl` becomes available. |
|
||||
| Hidden analysis playback rejected (`play()` Promise reject) | Keep transport state authoritative; catch and ignore reject, retry on explicit user gesture/next resume edge. |
|
||||
| SEK analysis seek desync on scrub | Ensure backend supports byte ranges for SEK assets; without `206` support, browser seek and `currentTime` sync are unreliable. |
|
||||
| Cross-origin media blocks analysis | Keep `crossOrigin="anonymous"`; ensure source routes deliver compatible CORS/auth behavior (proxy-backed routes preferred). |
|
||||
@@ -0,0 +1,264 @@
|
||||
# Muon Visualizer Reference-Parity Plan
|
||||
|
||||
Date: 2026-04-13
|
||||
Scope: Bring StemEngine’s `muon` visualizer behavior and look closer to the reference at
|
||||
https://muon-vis.netlify.app/ (multi-ring pattern diversity, emission lifecycle, mirror/division
|
||||
controls, and stronger pattern variation under audio).
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Current StemEngine Muon is a simplified concentric-dot renderer. This plan upgrades it toward
|
||||
reference parity by adding:
|
||||
|
||||
1. Variable ring divisions and mirror mode
|
||||
2. Multiple per-ring pattern families (not one shared sinusoid)
|
||||
3. Particle emission lifecycle (`lifespan`, trail persistence)
|
||||
4. Noise deformation (`noise_scale`, `noise_force`) actually used in rendering
|
||||
5. Dynamic radius behavior tied to beat/energy deltas (reference-like responsiveness)
|
||||
|
||||
This is parity-oriented, not bit-for-bit visual cloning.
|
||||
|
||||
---
|
||||
|
||||
## Current gaps (vs reference)
|
||||
|
||||
In `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx` today:
|
||||
|
||||
- Rings are fixed to 8 lanes (`lane: i % 8`)
|
||||
- Per-ring shape is one shared formula (`sin(angle * aperture)`-style)
|
||||
- No true emitter lifecycle (`lifespan`, decay, trail populations)
|
||||
- `noise_scale` and `noise_force` are configured but not applied
|
||||
- No mirror mode or alternate geometry profile switching
|
||||
|
||||
Result: consistent concentric dots, limited ring pattern variation.
|
||||
|
||||
---
|
||||
|
||||
## Design target
|
||||
|
||||
### A) Ring topology + mirror
|
||||
|
||||
Add configurable ring topology:
|
||||
|
||||
- `divisions` (e.g. 8–48): ring count / angular partitioning
|
||||
- `particle_mirror` (boolean): mirror opposite-side emissions/positions
|
||||
|
||||
### B) Pattern families
|
||||
|
||||
Each ring chooses one of several pattern kernels:
|
||||
|
||||
- `orbital`
|
||||
- `flower`
|
||||
- `ripple`
|
||||
- `spiro`
|
||||
- `strobe-grid`
|
||||
|
||||
Pattern assignment rotates over rings and can be preset-driven.
|
||||
|
||||
### C) Emission + lifespan
|
||||
|
||||
Replace direct stateless point plotting with emitted particles:
|
||||
|
||||
- `lifespan` in frames/ms
|
||||
- `birth`, `age`, `alpha`, `size`, `velocity`
|
||||
- decays and optional trail persistence
|
||||
|
||||
### D) Audio response model
|
||||
|
||||
Use richer Muon features already available in `AudioFeatures`:
|
||||
|
||||
- kick/sub: `frequencyBands.sub`
|
||||
- brightness: `presence + brilliance`
|
||||
- beat phase: `beatInfo.beatPhase`
|
||||
- drops/build-ups: `patternInfo.isBeatDrop` / `isBuildUp`
|
||||
- energy delta gate for `dynamic_radius`
|
||||
|
||||
### E) Noise deformation
|
||||
|
||||
Use `noise_scale` + `noise_force` to deform particle radius/angle per ring.
|
||||
|
||||
---
|
||||
|
||||
## Data model additions
|
||||
|
||||
Update muon visualizer settings schema (non-breaking defaults):
|
||||
|
||||
```ts
|
||||
muon: {
|
||||
// existing keys remain
|
||||
divisions?: number // default 16
|
||||
lifespan?: number // default 180
|
||||
particle_mirror?: boolean // default false
|
||||
dynamic_radius?: number // default 1.0
|
||||
delta_response_limit?: number // default 0.005
|
||||
visualization_preset?: 'default'|'orbital'|'flower'|'spiro'|'hybrid'
|
||||
}
|
||||
```
|
||||
|
||||
Keep all current keys (`radius_multiplier`, `aperture`, `spacing`, `particle_count`, etc.) intact.
|
||||
|
||||
---
|
||||
|
||||
## Implementation phases
|
||||
|
||||
## Phase 1 — Renderer architecture upgrade
|
||||
|
||||
### Files
|
||||
|
||||
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
|
||||
|
||||
### Tasks
|
||||
|
||||
1. Introduce particle pool/lifecycle structures:
|
||||
- `type MuonParticle = { x,y,vx,vy,ring,age,maxAge,size,hue,alpha,pattern }`
|
||||
2. Add emitter step and update step
|
||||
3. Add ring map from `divisions`
|
||||
4. Add mirror emission if enabled
|
||||
5. Keep old rendering as fallback branch behind a debug flag during migration (temporary)
|
||||
|
||||
### Acceptance
|
||||
|
||||
- Visual output shows obvious multi-pattern ring differentiation even on idle playback
|
||||
- No FPS collapse on default settings (60 fps target on desktop)
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Pattern kernels + noise
|
||||
|
||||
### Files
|
||||
|
||||
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
|
||||
- `frontend/src/components/visualizers/muon/patterns.ts` (new)
|
||||
- `frontend/src/components/visualizers/muon/noise.ts` (new, lightweight noise util)
|
||||
|
||||
### Tasks
|
||||
|
||||
1. Implement pattern kernels:
|
||||
- `computeOrbital()`, `computeFlower()`, `computeRipple()`, `computeSpiro()`
|
||||
2. Implement ring-level pattern assignment strategy
|
||||
3. Apply noise deformation using `noise_scale` / `noise_force`
|
||||
4. Blend phase-lock behavior via `beatPhase`
|
||||
|
||||
### Acceptance
|
||||
|
||||
- At least 3 visibly distinct concentric ring behaviors simultaneously
|
||||
- `noise_scale` and `noise_force` slider changes have immediate visual effect
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Audio-reactive dynamics parity
|
||||
|
||||
### Files
|
||||
|
||||
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
|
||||
|
||||
### Tasks
|
||||
|
||||
1. Add dynamic radius gate:
|
||||
- reacts only when `abs(energyDelta) >= delta_response_limit`
|
||||
2. Use `dynamic_radius` multiplier to smooth/boost radial expansion
|
||||
3. Add drop behavior presets:
|
||||
- ring preset swap/intensity boost on `isBeatDrop`
|
||||
4. Improve per-ring hue strategy:
|
||||
- shared drift + ring offset + beat-phase perturbation
|
||||
|
||||
### Acceptance
|
||||
|
||||
- Visual response tracks beat/drop transitions more like reference behavior
|
||||
- No permanent over-saturation after repeated drops
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Controls, presets, persistence
|
||||
|
||||
### Files
|
||||
|
||||
- `frontend/src/components/PerformanceSidebar.tsx`
|
||||
- `frontend/src/components/visualizers/VisualizerBackdrop.tsx`
|
||||
- `frontend/src/components/LyricsLivePreview.tsx`
|
||||
- `frontend/src/api/client.ts` (if typing expansion required)
|
||||
|
||||
### Tasks
|
||||
|
||||
1. Add UI sliders/toggles for new Muon fields:
|
||||
- `divisions`, `lifespan`, `particle_mirror`, `dynamic_radius`, `delta_response_limit`
|
||||
2. Add `visualization_preset` selector and map to kernel mixes
|
||||
3. Ensure normalization defaults and clamping in `LyricsLivePreview` config normalization path
|
||||
4. Keep manifest compatibility: missing fields default safely
|
||||
|
||||
### Acceptance
|
||||
|
||||
- New controls visible and functional in sidebar
|
||||
- Presets produce clearly different multi-ring signatures
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — QA + tuning
|
||||
|
||||
### Tasks
|
||||
|
||||
1. Build tuning matrix:
|
||||
- low/medium/high BPM
|
||||
- bass-heavy / treble-heavy / speech-like tracks
|
||||
2. Performance checks:
|
||||
- ensure particle count + lifespan don’t degrade below acceptable frame rate
|
||||
3. Add hard caps:
|
||||
- max active particles by quality tier
|
||||
|
||||
### Acceptance
|
||||
|
||||
- `npm run -s typecheck` passes
|
||||
- `npm run -s build` passes
|
||||
- No runtime errors when switching styles rapidly (`muon`↔`fluid`↔`video`)
|
||||
|
||||
---
|
||||
|
||||
## Compatibility and migration notes
|
||||
|
||||
- Existing saved manifests must continue to render with defaults.
|
||||
- New keys are optional and clamped.
|
||||
- Keep current scalar compatibility fields from `AudioFeatures`; consume richer fields where present.
|
||||
|
||||
---
|
||||
|
||||
## Risks and mitigations
|
||||
|
||||
1. **Performance regressions from lifecycle particles**
|
||||
Mitigation: pool particles + cap by quality tier (`low`/`balanced`/`high`).
|
||||
|
||||
2. **Too many controls causing UX complexity**
|
||||
Mitigation: advanced section collapsed by default + presets as primary entrypoint.
|
||||
|
||||
3. **Visual drift from reference despite upgrades**
|
||||
Mitigation: define 3 visual parity checkpoints (ring diversity, phase motion, drop transitions) and tune iteratively.
|
||||
|
||||
4. **Manifest schema drift**
|
||||
Mitigation: all new fields optional; normalize with defaults before use.
|
||||
|
||||
---
|
||||
|
||||
## File change map (planned)
|
||||
|
||||
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx` (major rewrite)
|
||||
- `frontend/src/components/visualizers/muon/patterns.ts` (new)
|
||||
- `frontend/src/components/visualizers/muon/noise.ts` (new)
|
||||
- `frontend/src/components/PerformanceSidebar.tsx` (new controls + preset selector)
|
||||
- `frontend/src/components/visualizers/VisualizerBackdrop.tsx` (settings type expansion)
|
||||
- `frontend/src/components/LyricsLivePreview.tsx` (settings defaults + normalization expansion)
|
||||
- `SUMMARIES (Claude).md` (implementation entry)
|
||||
- `20260313_Fixes (Copilot).md` (implementation entry)
|
||||
|
||||
---
|
||||
|
||||
## Implementation order
|
||||
|
||||
1. Add pattern + noise helper modules
|
||||
2. Rewrite `MuonVisualizerCanvas` with lifecycle + pattern kernels + mirror/divisions
|
||||
3. Wire audio-reactive dynamic radius and thresholding
|
||||
4. Expand settings type/defaults/normalization
|
||||
5. Add sidebar controls + presets
|
||||
6. Tune and cap performance by quality tier
|
||||
7. Validate (`typecheck`, `build`)
|
||||
8. Document
|
||||
+310
-1
@@ -9723,4 +9723,313 @@ Fixes:
|
||||
|
||||
Validation:
|
||||
|
||||
- `npm run typecheck` ✅
|
||||
- `npm run typecheck` ✅
|
||||
|
||||
---
|
||||
|
||||
### 2026-04-13 — Muon audio analyzer integration (hidden full-mix analysis audio)
|
||||
|
||||
Requested follow-up:
|
||||
|
||||
- Implement the Muon analyzer migration plan.
|
||||
- Use a hidden always-mounted full-mix `<audio>` element for analysis source.
|
||||
- Update project documentation logs.
|
||||
|
||||
Implemented:
|
||||
|
||||
`frontend/src/audio/MuonFeatureExtractor.ts` (new)
|
||||
- Added `MuonFeatureExtractor` with RAF extraction loop.
|
||||
- Configured analyser to Muon-style defaults (`fftSize=1024`, smoothing `0.8`, dB range `-100..-30`).
|
||||
- Added feature extraction outputs:
|
||||
- `rms`, `energy`
|
||||
- `spectralCentroid`, `spectralRolloff`, `spectralFlatness`, `zcr`
|
||||
- 7-band energy map (`sub`, `bass`, `lowMid`, `mid`, `highMid`, `presence`, `brilliance`)
|
||||
- beat info (`isBeat`, `bpm`, `confidence`, `timeSinceLastBeat`, `beatPhase`)
|
||||
- pattern info (`isBeatDrop`, `isBuildUp`, `isSilence`, `intensity`, `energyDelta`)
|
||||
- Returns `floatFreqData`/`floatTimeData` as copied buffers per frame.
|
||||
|
||||
`frontend/src/hooks/useAudioFeatures.ts`
|
||||
- Removed prior Meyda + manual byte FFT path.
|
||||
- Migrated hook to `MuonFeatureExtractor` frame callback path.
|
||||
- Expanded `AudioFeatures` to include full Muon fields while preserving compatibility fields used by existing canvases.
|
||||
- Changed `freqData` type from `Uint8Array` to `Float32Array`.
|
||||
- Switched hook source args to analysis audio strategy:
|
||||
- removed `stemGainRefs` / `singleAudioRef`
|
||||
- added `analysisAudioRef`
|
||||
- added `analysisSourceKey` to force safe remount when audio source changes.
|
||||
- Implemented running-peak band normalization (`PEAK_DECAY=0.9998`).
|
||||
- Updated `drumIntensity` derivation to use `sub + bass + rms`.
|
||||
|
||||
`frontend/src/components/LyricsLivePreview.tsx`
|
||||
- Added hidden always-mounted full-mix analysis element:
|
||||
- `analysisAudioRef`
|
||||
- rendered whenever `audioUrl` exists
|
||||
- `muted`, hidden, `crossOrigin="anonymous"`, `preload="auto"`.
|
||||
- Updated `useAudioFeatures()` call site to pass `analysisAudioRef` and `analysisSourceKey`.
|
||||
- Added `syncAnalysisAudio()` transport sync helper.
|
||||
- Integrated sync into:
|
||||
- seek/apply position
|
||||
- play/resume and pause flows
|
||||
- reset/finalize/stop flows
|
||||
- periodic drift correction + playback-rate mirroring while playing.
|
||||
|
||||
Dependency cleanup:
|
||||
|
||||
- Removed `meyda` from frontend dependencies (`frontend/package.json` + lockfile updated).
|
||||
|
||||
Validation:
|
||||
|
||||
- `npm run -s typecheck` ✅
|
||||
|
||||
### 2026-04-13 — Muon reference parity plan implemented
|
||||
|
||||
Requested:
|
||||
|
||||
- Implement and document `2026-04-13_Muon_Reference_Parity_Plan.md`.
|
||||
|
||||
Implemented:
|
||||
|
||||
1) Renderer architecture / lifecycle
|
||||
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
|
||||
- Reworked to pooled lifecycle particles (`age`, `maxAge`, velocity decay, alpha fade).
|
||||
- Added per-frame emission/update pipeline with trail persistence.
|
||||
- Added ring `divisions`, `particle_mirror`, and quality-tier active-particle caps.
|
||||
|
||||
2) Pattern kernels + noise deformation
|
||||
- `frontend/src/components/visualizers/muon/patterns.ts` (new)
|
||||
- Added `orbital` / `flower` / `ripple` / `spiro` / `strobe-grid` sampling helpers.
|
||||
- Added `patternFromPreset()` mapping for preset-driven ring behavior.
|
||||
- `frontend/src/components/visualizers/muon/noise.ts` (new)
|
||||
- Added deterministic noise helpers used by Muon deformation.
|
||||
|
||||
3) Audio-reactive parity controls
|
||||
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
|
||||
- Added dynamic radius gate using `dynamic_radius` + `delta_response_limit`.
|
||||
- Added drop-triggered pattern shift/boost and richer hue drift behavior.
|
||||
- Applied `noise_scale` / `noise_force` in renderer path (no longer reserved/no-op).
|
||||
|
||||
4) Settings + UI wiring
|
||||
- `frontend/src/components/visualizers/VisualizerBackdrop.tsx`
|
||||
- Expanded Muon settings type for new parity fields.
|
||||
- `frontend/src/components/LyricsLivePreview.tsx`
|
||||
- Expanded Muon defaults, normalization, and SEK manifest settings mapping.
|
||||
- `frontend/src/components/PerformanceSidebar.tsx`
|
||||
- Added controls for: `divisions`, `lifespan`, `particle_mirror`,
|
||||
`dynamic_radius`, `delta_response_limit`, `visualization_preset`.
|
||||
- Extended Muon presets with parity fields and added pattern preset selector.
|
||||
|
||||
Validation:
|
||||
|
||||
- `npm run -s typecheck` ✅
|
||||
- `npm run -s build` ✅
|
||||
|
||||
### 2026-04-13 — Muon plan remainder implemented
|
||||
|
||||
Requested:
|
||||
|
||||
- Implement remaining tasks from
|
||||
`docs/plans/2026-04-13_MuonAudioAnalyzer_Integration_Plan.md` and document.
|
||||
|
||||
Implemented:
|
||||
|
||||
1) SEK analysis fallback ranking
|
||||
- `frontend/src/pages/KaraokeTV.tsx`
|
||||
- `frontend/src/pages/Karaoke.tsx`
|
||||
- Added `pickSekAnalysisStem()` + `rankSekAnalysisStem()` helpers to select a more
|
||||
representative fallback source in SEK mode:
|
||||
- priority: drums/kick/percussion → leading/vocals → backing/instrumental → others.
|
||||
|
||||
2) Meyda cleanup artifact
|
||||
- `frontend/src/types/meyda.d.ts`
|
||||
- Removed stale `declare module 'meyda'` typings content (runtime already migrated to
|
||||
`MuonFeatureExtractor`).
|
||||
|
||||
Validation:
|
||||
|
||||
- `npm run -s typecheck` ✅
|
||||
- `npm run -s build` ✅
|
||||
|
||||
### 2026-04-13 — Fluid brightening by audio intensity + plan mirror
|
||||
|
||||
Done:
|
||||
|
||||
1) Mirrored plan edits
|
||||
- Synced root plan copy with integrated `docs/plans` updates:
|
||||
- `2026-04-13_MuonAudioAnalyzer_Integration_Plan.md`
|
||||
|
||||
2) Fluid brightening response
|
||||
- `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx`
|
||||
- Added dynamic bloom modulation from audio intensity:
|
||||
- `BLOOM_INTENSITY` now scales with blended `kick + rms + treble` and drop boost.
|
||||
- `BLOOM_THRESHOLD` now lowers with intensity so louder moments bloom more.
|
||||
|
||||
- `frontend/src/components/visualizers/fluid/fluidSim.ts`
|
||||
- Increased synthetic drag/pressure scaling from splat velocity.
|
||||
- Added intermediate pointer move sample to strengthen pressure-like splat behavior.
|
||||
|
||||
Validation:
|
||||
|
||||
- `npm run -s build` ✅
|
||||
|
||||
Follow-up runtime fix:
|
||||
|
||||
- Fixed Vite import-analysis failure for `webgl-fluid` in `fluidSim.ts`.
|
||||
- Vendored `webgl-fluid.mjs` under `frontend/src/components/visualizers/fluid/` and switched the wrapper to `import('./webgl-fluid.mjs')`.
|
||||
- Added `frontend/src/types/mjs-modules.d.ts` so TypeScript accepts vendored `.mjs` imports.
|
||||
- Validation: `npm run -s typecheck` ✅, `npm run -s build` ✅
|
||||
|
||||
### 2026-04-13 — Visualizer diagnosis plan fully implemented
|
||||
|
||||
Completed full implementation from `2026-04-13_Visualizer_DiagnosisAndFixes_Plan.md`.
|
||||
|
||||
Shipped changes:
|
||||
|
||||
1) Analysis source fixes (A + A1)
|
||||
- Added `analysisAudioUrl` propagation and usage:
|
||||
- `frontend/src/components/LyricsLivePreview.tsx`
|
||||
- `frontend/src/components/KaraokePerformanceMode.tsx`
|
||||
- `frontend/src/pages/KaraokeTV.tsx`
|
||||
- `frontend/src/pages/Karaoke.tsx`
|
||||
- Analysis hook now keys by `analysisAudioUrl ?? audioUrl ?? null`.
|
||||
|
||||
2) Muon color variation fix (B)
|
||||
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
|
||||
- Per-lane hue offset when `mono_color` is disabled.
|
||||
|
||||
3) Fluid migration to WebGL path (E)
|
||||
- `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx`
|
||||
- Rewritten to WebGL fluid wrapper + audio-driven splat injection.
|
||||
- Includes WebGL absence fallback and cleanup with context-loss attempt.
|
||||
- `frontend/src/components/visualizers/fluid/fluidSim.ts` (new)
|
||||
- Wrapper over `webgl-fluid` exposing `init/splat/update/destroy` helpers.
|
||||
- `frontend/src/types/webgl-fluid.d.ts` (new)
|
||||
- Ambient module typing.
|
||||
- `frontend/package.json` + lockfile
|
||||
- Added `webgl-fluid` dependency.
|
||||
|
||||
4) Fluid preset retune
|
||||
- `frontend/src/components/PerformanceSidebar.tsx`
|
||||
- Updated `FLUID_PRESETS` to new mapping values (Default/Ink/Foam/Lava/Mist/Storm).
|
||||
|
||||
5) Plan/document alignment
|
||||
- Updated plan file `2026-04-13_Visualizer_DiagnosisAndFixes_Plan.md` to reflect implemented
|
||||
wrapper approach and files/order updates.
|
||||
- Updated summary logs.
|
||||
|
||||
Validation:
|
||||
|
||||
- `npm run -s typecheck` ✅
|
||||
|
||||
### 2026-04-13 — SEK asset byte-range support + TrackDetail visualizer preview
|
||||
|
||||
Requested:
|
||||
|
||||
- Review/fix both new plans, then implement and document:
|
||||
- `2026-04-13_SEKAsset_RangeRequest_Plan.md`
|
||||
- `2026-04-13_TrackDetail_VisualizerPreview_Plan.md`
|
||||
|
||||
Plan corrections applied:
|
||||
|
||||
- SEK plan: corrected suffix-range behavior and clarified `Accept-Ranges` wording.
|
||||
- TrackDetail preview plan: updated to current Muon-era hook API (`analysisAudioRef`, `isPlayingRef`) and direct-media-only Video tab guidance.
|
||||
|
||||
Implemented backend:
|
||||
|
||||
`backend/routers/karaoke_sek.py`
|
||||
- Added `Request` param to asset route.
|
||||
- Added `Accept-Ranges: bytes` + `Content-Length` on full responses.
|
||||
- Added single-range parsing for `Range: bytes=...`.
|
||||
- Returns `206` with `Content-Range` for satisfiable ranges.
|
||||
- Returns `416` with `Content-Range: bytes */total` for invalid ranges.
|
||||
- Added proper suffix-range (`bytes=-N`) support.
|
||||
|
||||
Implemented frontend:
|
||||
|
||||
`frontend/src/components/VisualizerPreviewWidget.tsx` (new)
|
||||
- Added compact 16:9 preview panel with tabs: Muon / Fluid / Video (Video shown only for direct media URLs).
|
||||
- Added local `<audio>` transport with explicit play/pause handling and `AudioContext` ownership.
|
||||
- Wired `useAudioFeatures` using current args (`audioCtxRef`, `analysisAudioRef`, `isPlayingRef`, `analysisSourceKey`).
|
||||
- Added `<video>` play/pause and current-time syncing in Video mode.
|
||||
- Added visibility resume behavior for suspended contexts.
|
||||
- Added global audio registry handoff (`registerPlayback`, `unregisterPlayback`) to prevent overlapping playback.
|
||||
|
||||
`frontend/src/pages/TrackDetail.tsx`
|
||||
- Replaced static cover-art tile with `VisualizerPreviewWidget`.
|
||||
- Added `audioStreamUrl` memo using `subsonicStreamUrl(song.id)`.
|
||||
|
||||
`frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
|
||||
`frontend/src/components/visualizers/FluidVisualizerCanvas.tsx`
|
||||
- Removed `width`/`height` from effect deps to avoid resize-triggered canvas/RAF restarts.
|
||||
|
||||
Validation:
|
||||
|
||||
- `npm run -s typecheck` ✅
|
||||
- `python3 -m py_compile backend/routers/karaoke_sek.py` ✅
|
||||
|
||||
### 2026-04-13 — Visualizer diagnosis plan patched + Phase 1 fixes implemented
|
||||
|
||||
Requested:
|
||||
|
||||
- Patch/incorporate findings from `2026-04-13_Visualizer_DiagnosisAndFixes_Plan.md`.
|
||||
- Implement and document the practical fixes.
|
||||
|
||||
Plan corrections incorporated:
|
||||
|
||||
- Included `Karaoke.tsx` alongside `KaraokeTV.tsx` for analysis-source gap closure.
|
||||
- Updated implementation order to phased rollout (Phase 1 immediate fixes, Phase 2 optional WebGL migration).
|
||||
- Corrected WebGL context-limit wording in the fallback section.
|
||||
|
||||
Implemented (Phase 1):
|
||||
|
||||
`frontend/src/components/LyricsLivePreview.tsx`
|
||||
- Added `analysisAudioUrl?: string` prop.
|
||||
- Hidden analysis audio now sources from `analysisAudioUrl ?? audioUrl`.
|
||||
- Updated analysis hook key to `analysisSourceKey: analysisAudioUrl ?? audioUrl ?? null`.
|
||||
|
||||
`frontend/src/components/KaraokePerformanceMode.tsx`
|
||||
- Added `analysisAudioUrl?: string` prop and forwarded it to `LyricsLivePreview`.
|
||||
|
||||
`frontend/src/pages/KaraokeTV.tsx`
|
||||
`frontend/src/pages/Karaoke.tsx`
|
||||
- Added `analysisAudioUrl` derivation + pass-through:
|
||||
- project mode uses source media URL,
|
||||
- SEK mode uses first valid stem URL as fallback analysis source.
|
||||
|
||||
`frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
|
||||
- Added per-lane hue spread (when `mono_color` is false) to resolve frame-level monocolor look.
|
||||
|
||||
`frontend/src/components/visualizers/FluidVisualizerCanvas.tsx`
|
||||
- Fixed `spawnBlob` to take live canvas dimensions (`w`, `h`) from RAF loop.
|
||||
- Inverted `background_persistence` alpha mapping so higher persistence produces longer trails.
|
||||
|
||||
Validation:
|
||||
|
||||
- `npm run -s typecheck` ✅
|
||||
|
||||
Documentation updates:
|
||||
|
||||
- Appended full implementation summary to `SUMMARIES (Claude).md` under 2026-04-13.
|
||||
|
||||
### 2026-04-13 — Optional Muon visualizer enhancements (implemented)
|
||||
|
||||
Requested follow-up:
|
||||
|
||||
- Implement all optional future enhancements from the Muon integration plan.
|
||||
|
||||
Implemented:
|
||||
|
||||
`frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
|
||||
- Added kick-focused low-end driver using `f.frequencyBands.sub` blended with legacy bass/drum fields.
|
||||
- Added smooth beat-phase modulation via `f.beatInfo.beatPhase` (phase pulse/wave) to radius, aperture, glow, and particle motion.
|
||||
- Added beat-phase wrap detection to supplement beat trigger events.
|
||||
- Added dramatic beat-drop response via `f.patternInfo.isBeatDrop` with temporary hue/radius-preset escalation.
|
||||
|
||||
`frontend/src/components/visualizers/FluidVisualizerCanvas.tsx`
|
||||
- Added kick-focused burst/decay response using `f.frequencyBands.sub`.
|
||||
- Added phase-locked sweep emitter driven by `f.beatInfo.beatPhase` for smoother in-cycle motion.
|
||||
- Added beat-phase wrap as supplemental burst trigger.
|
||||
- Added dramatic beat-drop response (`f.patternInfo.isBeatDrop`) with temporary density boost and highlight tint overlay.
|
||||
|
||||
Validation:
|
||||
|
||||
- `npm run -s typecheck` ✅
|
||||
@@ -2565,3 +2565,313 @@ The `enforcedVisualizerMode` effect includes `visualStyle` in its dep array. Eve
|
||||
|
||||
**Bug 2 — Seek slider snaps to 0 after dragging (`LyricsLivePreview.tsx`):**
|
||||
`endSeekDrag` sets `isSeekingRef.current = false` synchronously after calling `applySeekPosition`. The RAF loop fires before the browser's audio `seeked` event, reads `audio.currentTime` (still returning the pre-seek value because browser seek is async), calls `setCurrentTime(0)`, then the `setTimeout(0)` clears `seekDragTime` — leaving the slider at 0. Fix: defer `isSeekingRef.current = false` until the audio element fires `seeked` (with a 500 ms safety fallback via `seekLockFallbackRef`). `beginSeekDrag` cancels any pending fallback timer from a prior drag.
|
||||
|
||||
### Visualizer parameter expansion — implemented
|
||||
|
||||
Expanded all visualizer parameters across four files. TypeScript check passes clean (exit 0).
|
||||
|
||||
**`LyricsLivePreview.tsx`** — `VizSettings` interface, `VIZ_DEFAULTS`, `normalizeVizSettings`, and `manifestVizSettings` useMemo expanded with all new muon and fluid fields (particle_count, rotation_speed, glow_intensity, particle_size, drum_reactivity, hue_range, beat_sensitivity, ellipse_ratio, idle_speed, noise_scale, noise_force, mono_color; background_persistence, beat_burst_count, treble_pulse, drag, blob_life, color_speed).
|
||||
|
||||
**`VisualizerBackdrop.tsx`** — `VisualizerSettings` interface expanded to match all new muon and fluid keys with inline purpose comments.
|
||||
|
||||
**`MuonVisualizerCanvas.tsx`** — Rewritten with: settings ref pattern (`settingsRef.current = settings` at render level; read inside RAF loop), `fpsCapRef`, `buildParticles()` helper, per-frame particle count rebuild detection, FPS cap inside RAF. New params applied: `drum_reactivity` as a pre-multiplier on the drum signal, `mono_color` locks hue to `p.seedHue`, `beat_sensitivity` divides beat cooldown, `ellipse_ratio` replaces hardcoded 0.62, `rotationSpeed`/`glowIntensity`/`particleSize`/`idleSpeed` all multiply their respective expressions. useEffect deps: `[featuresRef, paletteRef, width, height]` only.
|
||||
|
||||
**`FluidVisualizerCanvas.tsx`** — Rewritten with: settings ref pattern, `fpsCapRef`, `qualityRef`. New params applied: `background_persistence` multiplies `(1 - bass) * 0.16` fill alpha, `beat_burst_count` scales per-beat blob count, `treble_pulse` scales blob radius swell, `drag` divides velocity retention factor, `blob_life` multiplies initial blob life, `color_speed` controls `colorOffset` advance rate (fractional colorIndex). useEffect deps: `[featuresRef, paletteRef, width, height]` only.
|
||||
|
||||
**`PerformanceSidebar.tsx`** — `VizSettings` and `VIZ_DEFAULTS` expanded to match. Added `MUON_PRESETS` (Default/Calm/Energetic/Dreamy/Minimal/Cosmic) and `FLUID_PRESETS` (Default/Ink/Foam/Lava/Mist/Storm) as typed constant arrays. `muonPresetMatch`/`fluidPresetMatch` helpers detect active preset by field comparison with ±0.015 tolerance. Section headers simplified to plain divs (removed unused `openRef`/`useState` open/close). Muon section: preset pills row + 12 sliders + mono_color toggle. Fluid section: preset pills row + 9 sliders. Custom amber pill shown when no preset matches.
|
||||
|
||||
### Muon audio analyzer integration (implemented, 2026-04-13)
|
||||
|
||||
Implemented the full migration from the old stem/Meyda hybrid extractor to a Muon-style
|
||||
extractor path with a hidden always-mounted full-mix analysis audio element.
|
||||
|
||||
Files changed:
|
||||
|
||||
- `frontend/src/audio/MuonFeatureExtractor.ts` (new)
|
||||
- `frontend/src/hooks/useAudioFeatures.ts`
|
||||
- `frontend/src/components/LyricsLivePreview.tsx`
|
||||
- `frontend/package.json` / `frontend/package-lock.json`
|
||||
|
||||
What shipped:
|
||||
|
||||
- Added `MuonFeatureExtractor` class with RAF extraction loop, 1024 FFT config,
|
||||
frequency-band extraction (`sub`..`brilliance`), beat analysis (`bpm`, confidence,
|
||||
phase), pattern flags (`drop`, `buildup`, `silence`), and spectral metrics
|
||||
(`rolloff`, `flatness`, `zcr`).
|
||||
- Replaced `useAudioFeatures` Meyda + manual byte-FFT pipeline with Muon frame ingestion.
|
||||
- Expanded `AudioFeatures` interface to include:
|
||||
- backward-compatible fields (`bass`, `treble`, `rms`, `drumIntensity`, `beatDetected`)
|
||||
- full Muon fields (`frequencyBands`, `beatInfo`, `patternInfo`, `spectralRolloff`,
|
||||
`spectralFlatness`, `zcr`)
|
||||
- `freqData: Float32Array` (analyser dB-domain data).
|
||||
- Added running-peak/slow-decay band normalization and updated `drumIntensity` derivation
|
||||
from `sub + bass + rms`.
|
||||
- Added hidden always-mounted `analysisAudioRef` element in `LyricsLivePreview` and switched
|
||||
the hook source to that element in all playback modes.
|
||||
- Implemented analysis sync policy:
|
||||
- sync on seek/play/pause/reset/finalize
|
||||
- mirror playback rate
|
||||
- periodic drift snap when offset exceeds ~40ms.
|
||||
- Removed `meyda` package from frontend dependencies.
|
||||
|
||||
Validation:
|
||||
|
||||
- `npm run -s typecheck` passes after integration and cleanup.
|
||||
|
||||
### Muon/Fluid optional future enhancements (implemented, 2026-04-13)
|
||||
|
||||
Implemented all previously listed optional enhancements in both canvas visualizers.
|
||||
|
||||
Files changed:
|
||||
|
||||
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
|
||||
- `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx`
|
||||
|
||||
What shipped:
|
||||
|
||||
- `frequencyBands.sub` is now used as a kick-focused driver (combined with legacy bass/drum fields).
|
||||
- `beatInfo.beatPhase` now drives smooth phase-locked animation motion/pulsing.
|
||||
- `patternInfo.isBeatDrop` now triggers dramatic temporary visual escalation.
|
||||
|
||||
`MuonVisualizerCanvas` details:
|
||||
|
||||
- Added sub-band kick blend for cleaner low-end reaction.
|
||||
- Added phase-wave/phase-pulse modulation to radius, aperture, glow, and particle size/alpha.
|
||||
- Added beat-phase wrap detection as a supplemental beat trigger.
|
||||
- Added beat-drop boost that shifts hue and jumps radius presets for drop moments.
|
||||
|
||||
`FluidVisualizerCanvas` details:
|
||||
|
||||
- Added sub-band kick blend for burst sizing/count and decay behavior.
|
||||
- Added beat-phase orbit/sweep emitter for smoother in-beat motion.
|
||||
- Added beat-phase wrap as supplemental burst trigger.
|
||||
- Added beat-drop boost for denser spawns, highlight tint overlay, and larger pulse response.
|
||||
|
||||
Validation:
|
||||
|
||||
- `npm run -s typecheck` passes.
|
||||
|
||||
### SEK asset range support + TrackDetail visualizer preview (implemented, 2026-04-13)
|
||||
|
||||
Completed requested review corrections and implementation for both new plans:
|
||||
|
||||
- `2026-04-13_SEKAsset_RangeRequest_Plan.md`
|
||||
- `2026-04-13_TrackDetail_VisualizerPreview_Plan.md`
|
||||
|
||||
Plan review corrections applied:
|
||||
|
||||
- Updated SEK plan language around `Accept-Ranges` behavior and corrected suffix-range handling notes.
|
||||
- Updated TrackDetail preview plan to current `useAudioFeatures` API (`analysisAudioRef` + `isPlayingRef`) and direct-media video preview constraints.
|
||||
|
||||
Backend implementation:
|
||||
|
||||
- `backend/routers/karaoke_sek.py`
|
||||
- Added `Request` param to `get_asset`.
|
||||
- Added `Accept-Ranges: bytes` + explicit `Content-Length` on full responses.
|
||||
- Implemented `Range` handling with `206 Partial Content`.
|
||||
- Implemented `416 Range Not Satisfiable` for invalid ranges.
|
||||
- Added proper suffix-range support (`bytes=-N`).
|
||||
|
||||
Frontend implementation:
|
||||
|
||||
- `frontend/src/components/VisualizerPreviewWidget.tsx` (new)
|
||||
- Added compact preview widget with Muon/Fluid tabs and optional Video tab.
|
||||
- Uses local `<audio>` playback + `useAudioFeatures` (analysis from same audio element).
|
||||
- Added independent `AudioContext` lifecycle and resume-on-visibility handling.
|
||||
- Added direct-media URL check for Video tab (mp4/webm/m4v/ogg/ogv).
|
||||
- Added video play/pause + currentTime sync with preview audio.
|
||||
- Added global playback registry integration (`registerPlayback`/`unregisterPlayback`) to avoid overlap with active players.
|
||||
|
||||
- `frontend/src/pages/TrackDetail.tsx`
|
||||
- Integrated `VisualizerPreviewWidget` in place of static cover thumbnail.
|
||||
- Added `audioStreamUrl` via `subsonicStreamUrl(song.id)`.
|
||||
|
||||
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
|
||||
- `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx`
|
||||
- Removed `width`/`height` from effect dependency arrays to prevent resize-triggered RAF resets.
|
||||
|
||||
Validation:
|
||||
|
||||
- Frontend typecheck: `npm run -s typecheck` ✅
|
||||
- Backend syntax check: `python3 -m py_compile backend/routers/karaoke_sek.py` ✅
|
||||
|
||||
### Visualizer diagnosis plan incorporation + Phase 1 fixes (implemented, 2026-04-13)
|
||||
|
||||
Reviewed and incorporated corrections from `2026-04-13_Visualizer_DiagnosisAndFixes_Plan.md`,
|
||||
then implemented the immediate/safe fix set (A + A1 + B + C + D).
|
||||
|
||||
Plan updates applied:
|
||||
|
||||
- Corrected implementation scope to include both karaoke pages (`KaraokeTV.tsx` and `Karaoke.tsx`) for analysis source threading.
|
||||
- Updated rollout to two phases:
|
||||
- Phase 1: analysis-source gap + Muon hue + Fluid canvas correctness fixes.
|
||||
- Phase 2: optional high-risk Fluid WebGL migration.
|
||||
- Corrected WebGL context-limit wording in fallback notes.
|
||||
|
||||
Implemented frontend fixes:
|
||||
|
||||
- `frontend/src/components/LyricsLivePreview.tsx`
|
||||
- Added `analysisAudioUrl?: string` prop.
|
||||
- Hidden analysis element now uses `analysisAudioUrl ?? audioUrl`.
|
||||
- `useAudioFeatures` now keys by `analysisSourceKey: analysisAudioUrl ?? audioUrl ?? null`.
|
||||
|
||||
- `frontend/src/components/KaraokePerformanceMode.tsx`
|
||||
- Added/forwarded `analysisAudioUrl` prop to `LyricsLivePreview`.
|
||||
|
||||
- `frontend/src/pages/KaraokeTV.tsx`
|
||||
- `frontend/src/pages/Karaoke.tsx`
|
||||
- Added `analysisAudioUrl` memo derivation:
|
||||
- Project mode: source file URL.
|
||||
- SEK mode: first valid stem URL as analysis fallback.
|
||||
- Passed `analysisAudioUrl` into `KaraokePerformanceMode`.
|
||||
|
||||
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
|
||||
- Added per-lane hue spread when `mono_color` is off:
|
||||
- `laneHue = (hue + pt.lane * (hueRange * 45)) % 360`.
|
||||
|
||||
- `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx`
|
||||
- Fixed `spawnBlob` to use live canvas dimensions passed from RAF loop (`w`, `h`) instead of captured props.
|
||||
- Fixed `background_persistence` semantics by inverting alpha scaling:
|
||||
- higher persistence now yields lower clear alpha (longer trails).
|
||||
|
||||
Validation:
|
||||
|
||||
- `npm run -s typecheck` ✅
|
||||
|
||||
### Muon reference parity plan implemented (2026-04-13)
|
||||
|
||||
Implemented from `2026-04-13_Muon_Reference_Parity_Plan.md`:
|
||||
|
||||
- New Muon helper modules
|
||||
- `frontend/src/components/visualizers/muon/patterns.ts`
|
||||
- Added pattern families: `orbital`, `flower`, `ripple`, `spiro`, `strobe-grid`.
|
||||
- Added preset-to-pattern mapping via `patternFromPreset(...)`.
|
||||
- `frontend/src/components/visualizers/muon/noise.ts`
|
||||
- Added lightweight deterministic 2D noise helpers used for deformation.
|
||||
|
||||
- Major renderer upgrade
|
||||
- `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx`
|
||||
- Replaced fixed concentric particle loop with pooled lifecycle particles.
|
||||
- Added configurable ring `divisions` and `particle_mirror` emission.
|
||||
- Added per-particle lifespan/age decay and trail-style persistence.
|
||||
- Added pattern-driven emission with ring-level variation.
|
||||
- Applied `noise_scale` / `noise_force` to geometry deformation.
|
||||
- Added dynamic radius gating with `dynamic_radius` + `delta_response_limit`.
|
||||
- Added drop-trigger pattern shifting and intensity boost.
|
||||
- Added quality-tier particle caps for performance safety.
|
||||
|
||||
- Settings schema + normalization expansion
|
||||
- `frontend/src/components/visualizers/VisualizerBackdrop.tsx`
|
||||
- Expanded Muon settings type with parity keys.
|
||||
- `frontend/src/components/LyricsLivePreview.tsx`
|
||||
- Expanded `VizSettings.muon`, defaults, normalization, and SEK manifest mapping for:
|
||||
`divisions`, `lifespan`, `particle_mirror`, `dynamic_radius`,
|
||||
`delta_response_limit`, `visualization_preset`.
|
||||
|
||||
- Sidebar controls + preset wiring
|
||||
- `frontend/src/components/PerformanceSidebar.tsx`
|
||||
- Added UI controls for new Muon keys.
|
||||
- Added pattern preset selector (`default`, `orbital`, `flower`, `spiro`, `hybrid`).
|
||||
- Extended Muon named presets to include parity settings.
|
||||
|
||||
Validation:
|
||||
|
||||
- `npm run -s typecheck` ✅
|
||||
- `npm run -s build` ✅
|
||||
|
||||
### Muon integration plan — remainder implementation (2026-04-13)
|
||||
|
||||
Implemented final remaining items from
|
||||
`docs/plans/2026-04-13_MuonAudioAnalyzer_Integration_Plan.md`:
|
||||
|
||||
- SEK analysis-source fallback improvement
|
||||
- `frontend/src/pages/KaraokeTV.tsx`
|
||||
- `frontend/src/pages/Karaoke.tsx`
|
||||
- Replaced plain “first stem” fallback with ranked selection helper:
|
||||
- prefers `drums`/`kick`/`percussion`, then `leading`/`vocals`, then backing/instrumental.
|
||||
|
||||
- Meyda cleanup follow-through
|
||||
- `frontend/src/types/meyda.d.ts`
|
||||
- Removed obsolete Meyda module declarations and converted file to cleanup marker comment
|
||||
after migration to `MuonFeatureExtractor`.
|
||||
|
||||
Validation:
|
||||
|
||||
- `npm run -s typecheck` ✅
|
||||
- `npm run -s build` ✅
|
||||
|
||||
### Fluid audio-intensity brightening + mirrored plan sync (2026-04-13)
|
||||
|
||||
Implemented:
|
||||
|
||||
- Mirrored Muon analyzer plan updates to root copy:
|
||||
- `2026-04-13_MuonAudioAnalyzer_Integration_Plan.md`
|
||||
- synced with the `docs/plans` version (analysis source policy, dependency updates,
|
||||
SEK range prerequisite notes, risk table additions).
|
||||
|
||||
- Added audio-intensity-driven brightening to WebGL Fluid:
|
||||
- `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx`
|
||||
- dynamic `BLOOM_INTENSITY` based on live audio intensity (`kick`, `rms`, `treble`) with
|
||||
drop boost.
|
||||
- dynamic `BLOOM_THRESHOLD` lowering on louder frames so more dye contributes to bloom.
|
||||
- `frontend/src/components/visualizers/fluid/fluidSim.ts`
|
||||
- strengthened synthetic pointer pressure from splat velocity.
|
||||
- added intermediate drag sample to better emulate stronger press/drag splats.
|
||||
|
||||
Validation:
|
||||
|
||||
- `npm run -s build` ✅
|
||||
|
||||
Follow-up fix:
|
||||
|
||||
- Resolved Vite dev overlay failure for Fluid visualizer import analysis.
|
||||
- Root cause: package-name import resolution for `webgl-fluid` was failing in the active frontend runtime.
|
||||
- Fix: vendored `webgl-fluid.mjs` into `src/components/visualizers/fluid/`, changed `fluidSim.ts` to dynamically import the local file, and added `src/types/mjs-modules.d.ts` for TypeScript.
|
||||
- Revalidated with both `npm run -s typecheck` and `npm run -s build`.
|
||||
|
||||
### Visualizer diagnosis plan — full implementation (2026-04-13)
|
||||
|
||||
Completed the full implementation scope from
|
||||
`2026-04-13_Visualizer_DiagnosisAndFixes_Plan.md` and updated the plan doc to match shipped
|
||||
approach.
|
||||
|
||||
Implemented:
|
||||
|
||||
- Analysis-source gap closure (Issue A + A1)
|
||||
- Added `analysisAudioUrl` threading:
|
||||
- `LyricsLivePreview.tsx`
|
||||
- `KaraokePerformanceMode.tsx`
|
||||
- `KaraokeTV.tsx`
|
||||
- `Karaoke.tsx`
|
||||
- Analysis audio now mounts from `analysisAudioUrl ?? audioUrl`; Muon feature hook key tracks
|
||||
whichever source is active.
|
||||
|
||||
- Muon lane color fix (Issue B)
|
||||
- Added per-lane hue spread in `MuonVisualizerCanvas.tsx` while preserving `mono_color` lock.
|
||||
|
||||
- Fluid renderer migration (Issue E + edge notes)
|
||||
- Replaced canvas-blob fluid implementation with WebGL fluid wrapper path in
|
||||
`FluidVisualizerCanvas.tsx`.
|
||||
- Added wrapper module `frontend/src/components/visualizers/fluid/fluidSim.ts` around
|
||||
`webgl-fluid` package, exposing `initFluidSim`, `splatFluid`, `updateFluidConfig`,
|
||||
`destroyFluidSim`.
|
||||
- Added fallback rendering path when WebGL is unavailable.
|
||||
- Added cleanup via `destroyFluidSim` and `WEBGL_lose_context` attempt.
|
||||
- Added runtime config modulation + audio-driven splat injection using:
|
||||
- `frequencyBands.sub`, `beatInfo.beatPhase`, `patternInfo.isBeatDrop`, `treble_pulse`.
|
||||
- Documented behavior: `fpsCap` throttles injection cadence; sim render loop remains internal.
|
||||
|
||||
- Fluid presets retuned (Issue E Step 4)
|
||||
- Updated `FLUID_PRESETS` in `PerformanceSidebar.tsx` to the new parameter mapping table.
|
||||
|
||||
- Dependency/type support
|
||||
- Added `webgl-fluid` dependency in frontend package manifest/lockfile.
|
||||
- Added ambient module declaration: `frontend/src/types/webgl-fluid.d.ts`.
|
||||
|
||||
- Plan file alignment
|
||||
- Updated `2026-04-13_Visualizer_DiagnosisAndFixes_Plan.md` to reflect implemented wrapper
|
||||
approach (`webgl-fluid` package + `fluidSim.ts`) and updated files/order list.
|
||||
|
||||
Validation:
|
||||
|
||||
- `npm run -s typecheck` ✅
|
||||
@@ -2,7 +2,7 @@ from __future__ import annotations
|
||||
|
||||
from urllib.parse import quote
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from fastapi.responses import Response
|
||||
from pydantic import BaseModel
|
||||
|
||||
@@ -69,18 +69,89 @@ def get_manifest(package_id: str, folder: str | None = Query(default=None, descr
|
||||
|
||||
@router.get("/packages/{package_id}/asset")
|
||||
def get_asset(
|
||||
request: Request,
|
||||
package_id: str,
|
||||
path: str = Query(..., description="Inner package path, e.g. stems/vocals.mp3"),
|
||||
folder: str | None = Query(default=None, description="Optional SEK folder override"),
|
||||
):
|
||||
try:
|
||||
data, mime, filename = get_sek_asset(package_id, path, folder)
|
||||
total = len(data)
|
||||
base_headers = {
|
||||
"Accept-Ranges": "bytes",
|
||||
"Content-Disposition": _safe_content_disposition(filename),
|
||||
"Cache-Control": "public, max-age=3600",
|
||||
}
|
||||
|
||||
range_header = request.headers.get("range")
|
||||
if range_header:
|
||||
try:
|
||||
units, _, ranges_spec = range_header.partition("=")
|
||||
if units.strip().lower() == "bytes":
|
||||
first_range = ranges_spec.split(",", 1)[0].strip()
|
||||
if "-" not in first_range:
|
||||
raise ValueError("Malformed range")
|
||||
|
||||
start_str, end_str = first_range.split("-", 1)
|
||||
start_str = start_str.strip()
|
||||
end_str = end_str.strip()
|
||||
|
||||
if start_str == "":
|
||||
# Suffix range: bytes=-N -> last N bytes
|
||||
suffix_len = int(end_str)
|
||||
if suffix_len <= 0:
|
||||
raise ValueError("Invalid suffix length")
|
||||
if total <= 0:
|
||||
return Response(
|
||||
status_code=416,
|
||||
headers={
|
||||
"Content-Range": "bytes */0",
|
||||
"Content-Length": "0",
|
||||
**base_headers,
|
||||
},
|
||||
)
|
||||
start = max(total - suffix_len, 0)
|
||||
end = total - 1
|
||||
else:
|
||||
start = int(start_str)
|
||||
if start < 0:
|
||||
raise ValueError("Invalid start")
|
||||
end = int(end_str) if end_str else (total - 1)
|
||||
if end < 0:
|
||||
raise ValueError("Invalid end")
|
||||
|
||||
if total <= 0 or start >= total or start > end:
|
||||
return Response(
|
||||
status_code=416,
|
||||
headers={
|
||||
"Content-Range": f"bytes */{total}",
|
||||
"Content-Length": "0",
|
||||
**base_headers,
|
||||
},
|
||||
)
|
||||
|
||||
end = min(end, total - 1)
|
||||
chunk = data[start : end + 1]
|
||||
return Response(
|
||||
content=chunk,
|
||||
status_code=206,
|
||||
media_type=mime,
|
||||
headers={
|
||||
"Content-Range": f"bytes {start}-{end}/{total}",
|
||||
"Content-Length": str(len(chunk)),
|
||||
**base_headers,
|
||||
},
|
||||
)
|
||||
except (TypeError, ValueError):
|
||||
# Malformed range headers fall back to full response.
|
||||
pass
|
||||
|
||||
return Response(
|
||||
content=data,
|
||||
media_type=mime,
|
||||
headers={
|
||||
"Content-Disposition": _safe_content_disposition(filename),
|
||||
"Cache-Control": "public, max-age=3600",
|
||||
"Content-Length": str(total),
|
||||
**base_headers,
|
||||
},
|
||||
)
|
||||
except ProjectOperationError as exc:
|
||||
|
||||
@@ -19,8 +19,8 @@
|
||||
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
|
||||
<meta name="apple-mobile-web-app-title" content="StemEngine" />
|
||||
<link rel="apple-touch-icon" href="/icon.svg" />
|
||||
<script type="module" crossorigin src="/assets/index-CqSmBAZ1.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/assets/index-CIr2AVKp.css">
|
||||
<script type="module" crossorigin src="/assets/index-Cyxc7Uk9.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/assets/index-B34zlAct.css">
|
||||
</head>
|
||||
<body>
|
||||
<div id="root"></div>
|
||||
|
||||
@@ -0,0 +1,534 @@
|
||||
# Muon Audio Analyzer Integration Plan
|
||||
|
||||
Date: 2026-04-13
|
||||
Scope: Replace the current single-stem/Meyda feature extraction in `useAudioFeatures` with the
|
||||
Muon Audio Analysis `FeatureExtractor` approach, tapping a hidden analysis `<audio>` element
|
||||
fed by `analysisAudioUrl ?? audioUrl` (full-mix source when available; stem fallback in SEK).
|
||||
|
||||
Reference repo: https://github.com/najafmohammed/Muon-Audio-Analysis
|
||||
Status: private, not published to npm — the `FeatureExtractor` class must be inlined.
|
||||
|
||||
---
|
||||
|
||||
## Integrated decisions from related plans (2026-04-13)
|
||||
|
||||
This plan incorporates constraints already established in:
|
||||
|
||||
- `2026-04-13_Visualizer_DiagnosisAndFixes_Plan.md`
|
||||
- `2026-04-13_TrackDetail_VisualizerPreview_Plan.md`
|
||||
- `2026-04-13_SEKAsset_RangeRequest_Plan.md`
|
||||
|
||||
Net effect:
|
||||
|
||||
1. Analysis source is **not** `audioUrl`-only anymore; it is `analysisAudioUrl ?? audioUrl`.
|
||||
2. `analysisAudioUrl` must be threaded through `KaraokePerformanceMode` into
|
||||
`LyricsLivePreview` and included in `analysisSourceKey`.
|
||||
3. In SEK mode, where no explicit full-mix source exists in the manifest, analysis may use a
|
||||
ranked stem fallback (initially first/leading/drums stem strategy).
|
||||
4. Hidden analysis seek sync depends on working HTTP range support for SEK assets
|
||||
(`/api/karaoke-sek/packages/{id}/asset`) so `currentTime` jumps are reliable.
|
||||
|
||||
---
|
||||
|
||||
## Current problems
|
||||
|
||||
1. **Wrong source** — `useAudioFeatures` taps only the `drums` stem gain node (or first stem as
|
||||
fallback), so the visualizer is deaf to vocals, guitar, piano, etc.
|
||||
|
||||
2. **Mismatched algorithm** — the current extraction is a patchwork of Meyda (optional, for
|
||||
rms/energy/spectralCentroid) and a manual FFT slice (for bass/treble/beat). This is not the
|
||||
Muon approach.
|
||||
|
||||
3. **Low FFT resolution** — current `fftSize = 512` gives 256 bins; Muon uses `fftSize = 1024`
|
||||
(512 bins) for better low-frequency resolution.
|
||||
|
||||
4. **Limited feature set** — only 7 output fields. Muon outputs 7 frequency bands, BPM, beat
|
||||
phase, pattern detection (drop/buildup/silence), spectral rolloff, flatness, ZCR.
|
||||
|
||||
5. **Latent audio-muting bug** — the single-audio fallback path calls
|
||||
`createMediaElementSource(audioEl)` (which reroutes the element through Web Audio) but only
|
||||
connects it onward through `silentGain(0) → destination`, muting the audio.
|
||||
|
||||
---
|
||||
|
||||
## Why single-source (original song file) rather than all-stem mix
|
||||
|
||||
| Concern | Single source | All stems mixed |
|
||||
|---|---|---|
|
||||
| Tap nodes | 1 always | N (one per stem, grows with separator) |
|
||||
| Effect cleanup | Disconnect 1 node | Track and disconnect N nodes |
|
||||
| Effect deps | Can drop `stemGainRefs` | Must keep `stemGainRefs` in deps |
|
||||
| Stems-load-late race | No race — audio element always exists | Effect must restart when stem graph is ready |
|
||||
| Beat detection quality | Full mix = richest low-freq signal | Same once all stems connected |
|
||||
| Mute/solo sensitivity | Visualizer ignores user muting | Visualizer responds to audible mix |
|
||||
|
||||
The last row is the only meaningful trade-off. Mute/solo affecting the visualizer is arguably
|
||||
wrong behavior anyway — the visual should reflect the song's energy, not the listener's temporary
|
||||
mixing choices.
|
||||
|
||||
---
|
||||
|
||||
## Target state
|
||||
|
||||
```
|
||||
analysisAudioRef (<audio>, hidden, src = analysisAudioUrl ?? audioUrl) ──► MediaElementAudioSource
|
||||
└──► AnalyserNode (fftSize 1024)
|
||||
└──► MuonFeatureExtractor.extractLoop()
|
||||
│
|
||||
└──► AudioFeatures (expanded interface)
|
||||
├── bass, treble, rms, drumIntensity, beatDetected (kept, remapped)
|
||||
├── frequencyBands { sub, bass, lowMid, mid, highMid, presence, brilliance }
|
||||
├── beatInfo { isBeat, bpm, confidence, timeSinceLastBeat, beatPhase }
|
||||
└── patternInfo { isBeatDrop, isBuildUp, isSilence, intensity, energyDelta }
|
||||
```
|
||||
|
||||
In multi-stem mode, stems continue to drive audible output through their GainNodes as before.
|
||||
In single-audio mode, the visible player audio element remains the audible source.
|
||||
`analysisAudioRef` is analysis-only and is **never connected to `ctx.destination`**.
|
||||
|
||||
`analysisAudioRef` is synchronized to the active transport clock (single-track audio, stem anchor,
|
||||
or video clock) so analysis stays phase-aligned with what the listener hears.
|
||||
|
||||
---
|
||||
|
||||
## Files affected
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `frontend/src/audio/MuonFeatureExtractor.ts` | **New** — adapted FeatureExtractor class |
|
||||
| `frontend/src/hooks/useAudioFeatures.ts` | **Modified** — single-source tap + new extractor |
|
||||
| `frontend/src/components/LyricsLivePreview.tsx` | **Modified** — add hidden always-mounted full-mix analysis audio + sync strategy |
|
||||
| `frontend/src/components/KaraokePerformanceMode.tsx` | **Modified** — add/forward `analysisAudioUrl` |
|
||||
| `frontend/src/pages/KaraokeTV.tsx` | **Modified** — derive/pass `analysisAudioUrl` |
|
||||
| `frontend/src/pages/Karaoke.tsx` | **Modified** — derive/pass `analysisAudioUrl` |
|
||||
| `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx` | No immediate change (backward-compatible fields preserved) |
|
||||
| `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx` | No immediate change |
|
||||
| `SUMMARIES (Claude).md` | Append entry after implementation |
|
||||
|
||||
Meyda optional import can be removed once the new extractor is in place.
|
||||
|
||||
---
|
||||
|
||||
## Step 1 — New file: `frontend/src/audio/MuonFeatureExtractor.ts`
|
||||
|
||||
Inline the `FeatureExtractor` class from the Muon Audio Analysis repo, stripped of all Zustand
|
||||
store dependencies and adapted to return features via a callback instead of writing to a store.
|
||||
|
||||
### API
|
||||
|
||||
```typescript
|
||||
export interface MuonFrequencyBands {
|
||||
sub: number // 20–60 Hz
|
||||
bass: number // 60–250 Hz
|
||||
lowMid: number // 250–500 Hz
|
||||
mid: number // 500–2 000 Hz
|
||||
highMid: number // 2 000–4 000 Hz
|
||||
presence: number // 4 000–6 000 Hz
|
||||
brilliance: number // 6 000–20 000 Hz
|
||||
}
|
||||
|
||||
export interface MuonBeatInfo {
|
||||
isBeat: boolean
|
||||
bpm: number
|
||||
confidence: number
|
||||
timeSinceLastBeat: number
|
||||
beatPhase: number // 0–1 position in current beat cycle
|
||||
}
|
||||
|
||||
export interface MuonPatternInfo {
|
||||
isBeatDrop: boolean
|
||||
isBuildUp: boolean
|
||||
isSilence: boolean
|
||||
intensity: number // 0–1 overall
|
||||
energyDelta: number // rate of change
|
||||
}
|
||||
|
||||
export interface MuonFrame {
|
||||
rms: number
|
||||
energy: number
|
||||
spectralCentroid: number
|
||||
spectralRolloff: number
|
||||
spectralFlatness: number
|
||||
zcr: number
|
||||
frequencyBands: MuonFrequencyBands // raw linear energy values (not yet normalized)
|
||||
beatInfo: MuonBeatInfo
|
||||
patternInfo: MuonPatternInfo
|
||||
floatFreqData: Float32Array // raw dB values from analyser (copy, not aliased)
|
||||
floatTimeData: Float32Array // raw waveform (copy)
|
||||
}
|
||||
|
||||
export class MuonFeatureExtractor {
|
||||
private isRunning = false
|
||||
private animationFrameId: number | null = null
|
||||
private frequencyData!: Float32Array
|
||||
private timeData!: Float32Array
|
||||
private energyHistory: number[] = []
|
||||
private lastBeatTime = 0
|
||||
private beatThreshold = 1.5
|
||||
private bpm = 0
|
||||
private beatTimes: number[] = []
|
||||
|
||||
start(analyser: AnalyserNode, ctx: AudioContext, onFrame: (f: MuonFrame) => void): void
|
||||
stop(): void
|
||||
setBeatThreshold(threshold: number): void // 0.0–3.0, default 1.5
|
||||
reset(): void
|
||||
}
|
||||
```
|
||||
|
||||
### Implementation notes
|
||||
|
||||
- Copy `extractFeatures()`, `extractFrequencyBands()`, `detectBeat()`, `detectPatterns()`,
|
||||
`calculateLocalAverageAndVariance()` verbatim from the Muon repo source
|
||||
(`src/audio/FeatureExtractor.ts`).
|
||||
- Remove all `useAudioStore.getState()` calls; replace with a single `onFrame(frame)` call at
|
||||
the end of `extractLoop`.
|
||||
- The RAF loop is `requestAnimationFrame`-based — no change needed.
|
||||
- Define `AUDIO_CONFIG` constants (FFT_SIZE 1024, SMOOTHING 0.8, MIN_DECIBELS -100,
|
||||
MAX_DECIBELS -30) and `FREQUENCY_BANDS` Hz ranges as local module-level constants — no import
|
||||
from the Muon repo's `@core/constants`.
|
||||
- `ENERGY_HISTORY_SIZE = 43` (≈1 second at 60 fps, same as Muon source).
|
||||
- Return `floatFreqData` and `floatTimeData` as **copies** (`new Float32Array(this.frequencyData)`)
|
||||
so the caller can store them without aliasing issues.
|
||||
|
||||
---
|
||||
|
||||
## Step 2 — Modified: `frontend/src/hooks/useAudioFeatures.ts`
|
||||
|
||||
### 2a — Expand the `AudioFeatures` interface
|
||||
|
||||
```typescript
|
||||
export interface AudioFeatures {
|
||||
// ── Backward-compatible convenience fields (remapped from Muon data) ──
|
||||
bass: number // frequencyBands.bass normalized 0–1
|
||||
treble: number // (frequencyBands.presence + frequencyBands.brilliance) / 2, normalized 0–1
|
||||
rms: number // 0–1
|
||||
energy: number // 0–1
|
||||
loudness: number // kept for API compat; set equal to rms
|
||||
spectralCentroid: number
|
||||
drumIntensity: number // derived from sub + bass bands — see §2b
|
||||
beatDetected: boolean // beatInfo.isBeat
|
||||
|
||||
// ── Muon full feature set ──
|
||||
frequencyBands: {
|
||||
sub: number // normalized 0–1
|
||||
bass: number
|
||||
lowMid: number
|
||||
mid: number
|
||||
highMid: number
|
||||
presence: number
|
||||
brilliance: number
|
||||
}
|
||||
beatInfo: {
|
||||
isBeat: boolean
|
||||
bpm: number
|
||||
confidence: number
|
||||
timeSinceLastBeat: number
|
||||
beatPhase: number
|
||||
}
|
||||
patternInfo: {
|
||||
isBeatDrop: boolean
|
||||
isBuildUp: boolean
|
||||
isSilence: boolean
|
||||
intensity: number
|
||||
energyDelta: number
|
||||
}
|
||||
spectralRolloff: number
|
||||
spectralFlatness: number
|
||||
zcr: number
|
||||
|
||||
// ── Raw buffers ──
|
||||
freqData: Float32Array // dB values (was Uint8Array — see §2f)
|
||||
updatedAt: number
|
||||
}
|
||||
```
|
||||
|
||||
### 2b — `drumIntensity` re-derivation
|
||||
|
||||
The existing formula used the drums stem signal to isolate kick energy. With the full-mix
|
||||
analyser, `frequencyBands.sub` (20–60 Hz) is the cleanest kick/sub proxy. Replace:
|
||||
|
||||
```typescript
|
||||
function deriveDrumIntensity(sub: number, bass: number, rms: number): number {
|
||||
// sub and bass are already normalized 0–1 (auto-gain applied)
|
||||
const expSub = 1.001 * (1 + sub * 2) - 1.001
|
||||
const expBass = 1.001 * (1 + bass * 2) - 1.001
|
||||
const raw = (expSub * 0.50 + expBass * 0.38 + rms * 0.12) / 2.0
|
||||
return clamp01(Math.pow(clamp01(raw), 0.82))
|
||||
}
|
||||
```
|
||||
|
||||
`sub` takes 0.50 weight (up from `treble`'s 0.08 in the old formula) because sub-bass is where
|
||||
kick drums live.
|
||||
|
||||
### 2c — Band normalization (auto-gain)
|
||||
|
||||
`MuonFeatureExtractor.extractFrequencyBands()` returns raw RMS linear energy values, not 0–1.
|
||||
Normalize with a **running peak + slow decay**:
|
||||
|
||||
```typescript
|
||||
// Lives outside the useEffect (or inside as a closure) — persists across frames.
|
||||
const bandPeaks = {
|
||||
sub: 0.001, bass: 0.001, lowMid: 0.001, mid: 0.001,
|
||||
highMid: 0.001, presence: 0.001, brilliance: 0.001,
|
||||
}
|
||||
const PEAK_DECAY = 0.9998 // per-frame (~0.01 dB/s at 60 fps — very slow)
|
||||
|
||||
// Inside onFrame, per band:
|
||||
bandPeaks.bass = Math.max(bandPeaks.bass * PEAK_DECAY, rawBands.bass)
|
||||
const normalizedBass = clamp01(rawBands.bass / bandPeaks.bass)
|
||||
```
|
||||
|
||||
Peaks start at `0.001` to prevent divide-by-zero on startup. On the first loud frame they snap
|
||||
to the true peak, then decay slowly — self-calibrating to the mix level within ~2 seconds.
|
||||
|
||||
### 2d — Single-source tap via hidden analysis audio (replaces all current tap logic)
|
||||
|
||||
Remove the entire drum/fallback/single-audio branch. Replace with:
|
||||
|
||||
```typescript
|
||||
const audioEl = analysisAudioRef.current
|
||||
if (!audioEl) return
|
||||
|
||||
// createMediaElementSource can only be called once per element.
|
||||
// Re-use the cached node if the effect re-runs (e.g., on enabled toggle).
|
||||
try {
|
||||
mediaSourceRef.current = mediaSourceRef.current ?? ctx.createMediaElementSource(audioEl)
|
||||
} catch {
|
||||
return // AudioContext is closed or element already owned by another context
|
||||
}
|
||||
|
||||
const mediaSource = mediaSourceRef.current
|
||||
|
||||
const analyser = ctx.createAnalyser()
|
||||
analyser.fftSize = 1024
|
||||
analyser.smoothingTimeConstant = 0.8
|
||||
analyser.minDecibels = -100
|
||||
analyser.maxDecibels = -30
|
||||
|
||||
// Analysis-only path: never connect this element to destination.
|
||||
mediaSource.connect(analyser)
|
||||
```
|
||||
|
||||
**§2d-note — Hidden analysis element + sync strategy (`LyricsLivePreview.tsx`):**
|
||||
|
||||
Add a dedicated ref that is mounted in both single and multi-stem sessions:
|
||||
|
||||
```tsx
|
||||
const analysisAudioRef = useRef<HTMLAudioElement>(null)
|
||||
```
|
||||
|
||||
Render a hidden analysis element whenever `analysisAudioUrl || audioUrl` exists:
|
||||
|
||||
```tsx
|
||||
{(analysisAudioUrl || audioUrl) && (
|
||||
<audio
|
||||
ref={analysisAudioRef}
|
||||
src={analysisAudioUrl ?? audioUrl}
|
||||
preload="auto"
|
||||
crossOrigin="anonymous"
|
||||
muted
|
||||
style={{ display: 'none' }}
|
||||
/>
|
||||
)}
|
||||
```
|
||||
|
||||
Sync policy:
|
||||
- Mirror `currentTime` from active transport (`audioRef`, stem anchor, or `videoRef`).
|
||||
- Mirror `playbackRate`.
|
||||
- On play/resume: set time, then `analysisAudioRef.current?.play()`.
|
||||
- On pause/stop/end: `analysisAudioRef.current?.pause()`.
|
||||
- On seek: immediately set `analysisAudioRef.current.currentTime = seekTarget`.
|
||||
- In RAF sync loop, if drift exceeds ~40 ms, snap analysis time to transport time.
|
||||
|
||||
This guarantees one stable analysis source without audible doubling.
|
||||
|
||||
**Source policy integration (`KaraokeTV.tsx` / `Karaoke.tsx`):**
|
||||
|
||||
- Project mode: prefer source/original recording URL as `analysisAudioUrl`.
|
||||
- SEK mode: until true full-mix is available, derive `analysisAudioUrl` from a ranked stem
|
||||
fallback (leading/drums preferred; first stem acceptable initial implementation).
|
||||
- Always pass `analysisAudioUrl` through `KaraokePerformanceMode` into `LyricsLivePreview`.
|
||||
|
||||
**SEK seek prerequisite:**
|
||||
|
||||
For SEK fallback analysis to stay sync-accurate on seeks, the SEK asset endpoint must support
|
||||
HTTP byte ranges (`Accept-Ranges: bytes`, valid `206` responses, `416` handling for invalid
|
||||
ranges). This is covered by `2026-04-13_SEKAsset_RangeRequest_Plan.md` and is a dependency for
|
||||
reliable `analysisAudioRef.currentTime` snapping.
|
||||
```
|
||||
|
||||
### 2e — Cleanup
|
||||
|
||||
```typescript
|
||||
return () => {
|
||||
mounted = false
|
||||
extractor.stop()
|
||||
try { mediaSource.disconnect(analyser) } catch { /* noop */ }
|
||||
try { analyser.disconnect() } catch { /* noop */ }
|
||||
}
|
||||
```
|
||||
|
||||
The `mediaSourceRef` is preserved across effect re-runs (the `??=` creation guard above). Since
|
||||
the analysis element is never connected to destination, cleanup only detaches analyser links.
|
||||
|
||||
### 2f — Replace Meyda + tick
|
||||
|
||||
Remove the dynamic `import(/* @vite-ignore */ 'meyda')` block and the entire RAF `tick()`
|
||||
function. The `MuonFeatureExtractor` subsumes Meyda's role and runs its own RAF loop.
|
||||
|
||||
```typescript
|
||||
const extractor = new MuonFeatureExtractor()
|
||||
extractor.start(analyser, ctx, (frame) => {
|
||||
if (!mounted) return
|
||||
if (!isPlayingRef.current) {
|
||||
// Zero out reactive fields while paused; leave beatInfo/patternInfo/freqData as-is.
|
||||
featuresRef.current.bass = 0
|
||||
featuresRef.current.treble = 0
|
||||
featuresRef.current.rms = 0
|
||||
featuresRef.current.drumIntensity = 0
|
||||
featuresRef.current.beatDetected = false
|
||||
return
|
||||
}
|
||||
|
||||
// Normalize each band
|
||||
const b = frame.frequencyBands
|
||||
const keys = ['sub','bass','lowMid','mid','highMid','presence','brilliance'] as const
|
||||
const norm: Record<typeof keys[number], number> = {} as any
|
||||
for (const k of keys) {
|
||||
bandPeaks[k] = Math.max(bandPeaks[k] * PEAK_DECAY, b[k])
|
||||
norm[k] = clamp01(b[k] / bandPeaks[k])
|
||||
}
|
||||
|
||||
const rms = clamp01(frame.rms)
|
||||
const treble = clamp01((norm.presence + norm.brilliance) / 2)
|
||||
|
||||
featuresRef.current = {
|
||||
bass: norm.bass,
|
||||
treble,
|
||||
rms,
|
||||
energy: clamp01(frame.energy),
|
||||
loudness: rms,
|
||||
spectralCentroid: frame.spectralCentroid,
|
||||
drumIntensity: deriveDrumIntensity(norm.sub, norm.bass, rms),
|
||||
beatDetected: frame.beatInfo.isBeat,
|
||||
frequencyBands: norm,
|
||||
beatInfo: frame.beatInfo,
|
||||
patternInfo: frame.patternInfo,
|
||||
spectralRolloff: frame.spectralRolloff,
|
||||
spectralFlatness: frame.spectralFlatness,
|
||||
zcr: frame.zcr,
|
||||
freqData: frame.floatFreqData,
|
||||
updatedAt: Date.now(),
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### 2g — useEffect dependency array
|
||||
|
||||
Remove `stemGainRefs` from deps — the tap no longer depends on them. Keep
|
||||
`analysisSourceKey` in deps so source switches recreate analyser wiring:
|
||||
|
||||
```typescript
|
||||
}, [audioCtxRef, analysisAudioRef, isPlayingRef, enabled, analysisSourceKey])
|
||||
```
|
||||
|
||||
`stemGainRefs` can be removed from `UseAudioFeaturesArgs` entirely if it has no other use in the
|
||||
hook after the tap logic is replaced.
|
||||
|
||||
---
|
||||
|
||||
## Step 3 — Default values for new interface fields
|
||||
|
||||
```typescript
|
||||
const EMPTY_FREQ = new Float32Array(0) // was Uint8Array
|
||||
const EMPTY_BANDS = { sub: 0, bass: 0, lowMid: 0, mid: 0, highMid: 0, presence: 0, brilliance: 0 }
|
||||
const EMPTY_BEAT = { isBeat: false, bpm: 0, confidence: 0, timeSinceLastBeat: 0, beatPhase: 0 }
|
||||
const EMPTY_PAT = { isBeatDrop: false, isBuildUp: false, isSilence: true, intensity: 0, energyDelta: 0 }
|
||||
|
||||
const featuresRef = useRef<AudioFeatures>({
|
||||
bass: 0, treble: 0, rms: 0, energy: 0, loudness: 0,
|
||||
spectralCentroid: 0, spectralRolloff: 0, spectralFlatness: 0, zcr: 0,
|
||||
drumIntensity: 0, beatDetected: false,
|
||||
frequencyBands: EMPTY_BANDS,
|
||||
beatInfo: EMPTY_BEAT,
|
||||
patternInfo: EMPTY_PAT,
|
||||
freqData: EMPTY_FREQ,
|
||||
updatedAt: 0,
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 4 — Canvas files (backward-compatible; selected Muon enhancements already landed)
|
||||
|
||||
`MuonVisualizerCanvas` and `FluidVisualizerCanvas` read: `f.bass`, `f.treble`, `f.rms`,
|
||||
`f.drumIntensity`, `f.beatDetected`. All five are backward-compatible remapped fields in the
|
||||
expanded interface — no changes needed in either canvas file.
|
||||
|
||||
Current visualizer integration is backward-compatible and already consumes compatibility fields.
|
||||
Additionally, targeted Muon enhancements now exist in canvas logic (sub-band weighting,
|
||||
`beatPhase` shaping, and drop-trigger accenting), so these are no longer purely “future”.
|
||||
|
||||
Still optional for later parity work:
|
||||
- tighter phase-locked geometry states derived from `beatInfo.beatPhase`
|
||||
- richer transition choreography keyed on `patternInfo` sequences (build-up → drop)
|
||||
- broader usage of normalized per-band vectors across particle subsystems
|
||||
|
||||
---
|
||||
|
||||
## Step 5 — Remove Meyda (cleanup after verification)
|
||||
|
||||
```bash
|
||||
grep -r "meyda" frontend/src/ --include="*.ts" --include="*.tsx"
|
||||
```
|
||||
|
||||
If the only remaining reference was the deleted dynamic import, run:
|
||||
|
||||
```bash
|
||||
cd frontend && npm uninstall meyda
|
||||
```
|
||||
|
||||
Also remove stale type-only leftovers if unused:
|
||||
|
||||
- `frontend/src/types/meyda.d.ts`
|
||||
|
||||
Keep it only if another package still imports Meyda types.
|
||||
|
||||
---
|
||||
|
||||
## Implementation order
|
||||
|
||||
1. Write `frontend/src/audio/MuonFeatureExtractor.ts`
|
||||
2. Add `analysisAudioRef` to `UseAudioFeaturesArgs`; update call-site in `LyricsLivePreview.tsx`
|
||||
3. Expand `AudioFeatures` interface and initial ref value
|
||||
4. Add hidden always-mounted full-mix analysis `<audio>` in `LyricsLivePreview.tsx`
|
||||
5. Implement analysis sync strategy (play/pause/seek/time drift correction)
|
||||
6. Replace tap logic with single-source analysis routing (`mediaSource → analyser` only)
|
||||
7. Add `bandPeaks` / `PEAK_DECAY` normalization state
|
||||
8. Replace Meyda import + `tick()` with `extractor.start()` + `onFrame` callback
|
||||
9. Update `deriveDrumIntensity` signature
|
||||
10. Add/propagate `analysisAudioUrl` (`KaraokeTV`/`Karaoke` → `KaraokePerformanceMode` → `LyricsLivePreview`)
|
||||
11. Ensure `analysisSourceKey` tracks `analysisAudioUrl ?? audioUrl ?? null`
|
||||
12. Trim `stemGainRefs` from hook args if no longer needed
|
||||
13. Run `tsc --noEmit`
|
||||
14. Verify SEK seeking over `/karaoke-sek/packages/{id}/asset` still returns proper `206` on range probes
|
||||
15. Remove Meyda if clean
|
||||
16. Update `SUMMARIES (Claude).md`
|
||||
|
||||
---
|
||||
|
||||
## Risks and mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| Audio doubling in multi-stem mode | Analysis element is never connected to `ctx.destination` (`mediaSource → analyser` only). |
|
||||
| `createMediaElementSource` throws if audio element already owned | Wrap in try/catch; bail early. The `mediaSourceRef` guard (`??=`) prevents double-creation across effect re-runs. |
|
||||
| Analysis and audible transport drift apart | Mirror `playbackRate`, sync on play/seek, and snap when drift exceeds ~40 ms. |
|
||||
| Analysis source missing in stem sessions (`audioUrl` undefined) | Use `analysisAudioUrl ?? audioUrl`; derive `analysisAudioUrl` in page-level containers for both project and SEK modes. |
|
||||
| Band normalization near-zero for first frames | Peaks start at 0.001; clamp01 never goes negative. Self-calibrates within ~2 seconds of loud audio. |
|
||||
| `freqData` type change (`Uint8Array` → `Float32Array`) breaks a consumer | Audit: `grep -r "freqData" frontend/src/`. Values are now dB floats (-100 to 0) rather than 0–255 integers — update any display code. |
|
||||
| Extractor RAF loop burns CPU while paused | `onFrame` early-returns if `!isPlayingRef.current`, zeroing reactive fields. The extractor RAF still ticks but the callback is almost free when returning immediately. Alternatively call `extractor.stop()` on pause and `extractor.start()` on resume. |
|
||||
| `analysisAudioRef` is null on first render | `if (!audioEl) return` guard in useEffect; hidden element mounts when `analysisAudioUrl || audioUrl` becomes available. |
|
||||
| Hidden analysis playback rejected (`play()` Promise reject) | Keep transport state authoritative; catch and ignore reject, retry on explicit user gesture/next resume edge. |
|
||||
| SEK analysis seek desync on scrub | Ensure backend supports byte ranges for SEK assets; without `206` support, browser seek and `currentTime` sync are unreliable. |
|
||||
| Cross-origin media blocks analysis | Keep `crossOrigin="anonymous"`; ensure source routes deliver compatible CORS/auth behavior (proxy-backed routes preferred). |
|
||||
@@ -0,0 +1,205 @@
|
||||
SEK Asset Endpoint — HTTP Range Request Support
|
||||
|
||||
Date: 2026-04-13
|
||||
Scope: Fix seeking in the SEK file karaoke player (`/karaoke`, port 3001). Stems and source
|
||||
media served from `/api/karaoke-sek/packages/{id}/asset` do not support HTTP byte-range
|
||||
requests, so `audio.currentTime = t` silently fails in the browser.
|
||||
|
||||
---
|
||||
|
||||
## Root cause
|
||||
|
||||
`GET /api/karaoke-sek/packages/{package_id}/asset` in
|
||||
`backend/routers/karaoke_sek.py` (lines 70–87):
|
||||
|
||||
```python
|
||||
def get_asset(...):
|
||||
data, mime, filename = get_sek_asset(package_id, path, folder)
|
||||
return Response(
|
||||
content=data, # entire file, always
|
||||
media_type=mime,
|
||||
headers={
|
||||
"Content-Disposition": ...,
|
||||
"Cache-Control": "public, max-age=3600",
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
The problems:
|
||||
1. **No `Accept-Ranges: bytes` header** — browsers and media stacks may still probe with
|
||||
`Range`, but seek behavior is less reliable/non-standard without explicit range signaling.
|
||||
2. **No `Range` header parsing** — even if a browser sends `Range: bytes=X-Y`, the server
|
||||
ignores it and returns the full file with `200 OK`, which the browser interprets as "range
|
||||
requests are not supported."
|
||||
|
||||
When `audio.currentTime = t` is set on an audio element backed by a non-seekable URL:
|
||||
- The browser may reset `currentTime` to 0 silently, or
|
||||
- The seek appears to succeed in the UI (the scrubber moves) but playback resumes from the
|
||||
old position once buffering continues.
|
||||
|
||||
Non-SEK project files served via `/api/jobs/{id}/download/{filename}` (FastAPI `FileResponse`)
|
||||
do support range requests automatically — so seeking works there. Only SEK assets are broken.
|
||||
|
||||
---
|
||||
|
||||
## Fix — add range request support to the asset endpoint
|
||||
|
||||
### File: `backend/routers/karaoke_sek.py`
|
||||
|
||||
Change the `get_asset` route to:
|
||||
|
||||
1. Always add `Accept-Ranges: bytes` and `Content-Length` to the response.
|
||||
2. Parse the `Range: bytes=start-end` header when present.
|
||||
3. Respond with `206 Partial Content` for valid range requests.
|
||||
4. Respond with `416 Range Not Satisfiable` for invalid ranges.
|
||||
|
||||
```python
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from fastapi.responses import Response
|
||||
|
||||
@router.get("/packages/{package_id}/asset")
|
||||
def get_asset(
|
||||
request: Request,
|
||||
package_id: str,
|
||||
path: str = Query(..., description="Inner package path, e.g. stems/vocals.mp3"),
|
||||
folder: str | None = Query(default=None, description="Optional SEK folder override"),
|
||||
):
|
||||
try:
|
||||
data, mime, filename = get_sek_asset(package_id, path, folder)
|
||||
except ProjectOperationError as exc:
|
||||
raise HTTPException(status_code=exc.status_code, detail=exc.detail) from exc
|
||||
|
||||
total = len(data)
|
||||
base_headers = {
|
||||
"Accept-Ranges": "bytes",
|
||||
"Content-Disposition": _safe_content_disposition(filename),
|
||||
"Cache-Control": "public, max-age=3600",
|
||||
}
|
||||
|
||||
range_header = request.headers.get("range")
|
||||
if range_header:
|
||||
try:
|
||||
units, ranges_str = range_header.split("=", 1)
|
||||
if units.strip().lower() != "bytes":
|
||||
raise ValueError("unsupported range unit")
|
||||
start_str, _, end_str = ranges_str.partition("-")
|
||||
if not start_str.strip():
|
||||
# bytes=-N (suffix)
|
||||
suffix_len = int(end_str)
|
||||
if suffix_len <= 0:
|
||||
raise ValueError("invalid suffix")
|
||||
start = max(total - suffix_len, 0)
|
||||
end = total - 1
|
||||
else:
|
||||
start = int(start_str)
|
||||
end = int(end_str) if end_str.strip() else total - 1
|
||||
end = min(end, total - 1)
|
||||
if start > end or start >= total:
|
||||
return Response(
|
||||
status_code=416,
|
||||
headers={"Content-Range": f"bytes */{total}", **base_headers},
|
||||
)
|
||||
chunk = data[start : end + 1]
|
||||
return Response(
|
||||
content=chunk,
|
||||
status_code=206,
|
||||
media_type=mime,
|
||||
headers={
|
||||
"Content-Range": f"bytes {start}-{end}/{total}",
|
||||
"Content-Length": str(len(chunk)),
|
||||
**base_headers,
|
||||
},
|
||||
)
|
||||
except (ValueError, AttributeError):
|
||||
pass # malformed Range header — fall through to full response
|
||||
|
||||
return Response(
|
||||
content=data,
|
||||
media_type=mime,
|
||||
headers={
|
||||
"Content-Length": str(total),
|
||||
**base_headers,
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
### Changes from original
|
||||
|
||||
| Before | After |
|
||||
|---|---|
|
||||
| `get_asset(package_id, path, folder)` params only | Add `request: Request` param |
|
||||
| No range parsing | Parse `Range: bytes=X-Y` header |
|
||||
| Always `200 OK` with full body | `206 Partial Content` for range requests; `200` for full |
|
||||
| No `Accept-Ranges` header | `Accept-Ranges: bytes` always present |
|
||||
| No `Content-Length` header | `Content-Length` on all responses |
|
||||
| No `416` path | Return `416` for out-of-bounds or unsatisfiable ranges |
|
||||
|
||||
---
|
||||
|
||||
## Why reads-to-bytes is acceptable here
|
||||
|
||||
SEK packages are karaoke bundles designed to be portable and self-contained. Individual stem
|
||||
files inside them are typically 5–30 MB compressed audio. Reading the whole member into memory
|
||||
before slicing is acceptable for this use case — the files are served from a local NAS, not a
|
||||
CDN.
|
||||
|
||||
A streaming solution (extracting to temp file + `FileResponse`) would handle very large files
|
||||
more efficiently but adds temp-file lifecycle complexity. The slice approach is simpler and
|
||||
correct for typical SEK sizes.
|
||||
|
||||
If a file is unusually large (e.g., an uncompressed WAV stem > 200 MB), the full read still
|
||||
happens before the 206 slice is sent, which is wasteful. A future optimization could use
|
||||
`zipfile.ZipFile.open()` to get a streaming handle and seek within it — but that requires
|
||||
threading (ZipFile stream seeks aren't async-compatible). Not needed now.
|
||||
|
||||
---
|
||||
|
||||
## Edge cases
|
||||
|
||||
| Case | Handling |
|
||||
|---|---|
|
||||
| `Range: bytes=0-` (open-ended, read to EOF) | `end_str` is empty → `end = total - 1`; correct |
|
||||
| `Range: bytes=-500` (last 500 bytes) | Implemented as a true suffix range: returns `bytes total-500 ... total-1` (clamped when file is smaller than 500 bytes). |
|
||||
| `Range` header present but malformed (e.g., `Range: bytes`) | `ValueError` in parse → falls through to full `200` response |
|
||||
| `Range: bytes=999999-0` (start > end) | Caught by `start > end` check → `416` |
|
||||
| `Range` start ≥ `total` | Caught by `start >= total` check → `416` |
|
||||
| Multiple ranges (`Range: bytes=0-100, 200-300`) | Only the first range set is considered; multipart/byteranges responses are not implemented (acceptable for browser media seek path, which uses single ranges). |
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
# Full response — expect 200 + Accept-Ranges: bytes header
|
||||
curl -I "http://work:3001/api/karaoke-sek/packages/<id>/asset?path=stems/vocals.mp3"
|
||||
|
||||
# Range request — expect 206 + Content-Range header
|
||||
curl -I -H "Range: bytes=0-65535" \
|
||||
"http://work:3001/api/karaoke-sek/packages/<id>/asset?path=stems/vocals.mp3"
|
||||
|
||||
# Browser test: open the SEK player, play a song, drag the seek bar midway through
|
||||
# Playback should resume from the dragged position, not from the beginning
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Files changed
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `backend/routers/karaoke_sek.py` | Add `Request` import; rewrite `get_asset` to parse `Range` header and return `206`/`416` as appropriate |
|
||||
|
||||
No changes to the frontend — the browser's native range-request mechanism handles everything
|
||||
once the server responds correctly.
|
||||
|
||||
---
|
||||
|
||||
## Implementation order
|
||||
|
||||
1. Add `Request` to the FastAPI imports at the top of `karaoke_sek.py`
|
||||
2. Replace the `get_asset` function body with the new implementation above
|
||||
3. Run `curl -I` tests to confirm `Accept-Ranges: bytes` appears in response
|
||||
4. Run `curl -H "Range: bytes=0-65535"` to confirm `206` response
|
||||
5. Open the karaoke TV page with a SEK file; play and seek — confirm playback resumes from the
|
||||
seeked position
|
||||
6. Update `SUMMARIES (Claude).md`
|
||||
@@ -0,0 +1,491 @@
|
||||
# TrackDetail Visualizer Preview Plan
|
||||
|
||||
Date: 2026-04-13
|
||||
Scope: Add a visualizer preview widget to SingUI's `/tracks/:trackId` detail page, letting the
|
||||
user switch between Muon, Fluid, and Video (when a video URL exists) before launching a full
|
||||
karaoke session.
|
||||
|
||||
---
|
||||
|
||||
## Current state of TrackDetail
|
||||
|
||||
`/home/translator/docker/appdata/stemengine/frontend/src/pages/TrackDetail.tsx`
|
||||
|
||||
The page already has:
|
||||
- `song.id` → `subsonicStreamUrl(song.id)` for the full-mix audio stream
|
||||
- `coverArtUrl` → `subsonicCoverArtUrl(song.coverArt, 800)` for palette
|
||||
- `videoUrlInput` (state) → fetched from `getProjectKaraokeConfig(matchedProject.id)` for video
|
||||
- `matchedProject` → project link (used to decide if Video tab is available)
|
||||
- A 96×96 cover art `<img>` in the hero row that can be replaced by the preview widget
|
||||
|
||||
The page does **not** have: LyricsLivePreview, audio playback, or any visualizer.
|
||||
|
||||
---
|
||||
|
||||
## Design
|
||||
|
||||
### Layout change
|
||||
|
||||
Replace the static 96×96 cover art thumbnail with the `VisualizerPreviewWidget` component.
|
||||
The widget is a self-contained panel:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ │ ← 16:9, max-w-xs (≈288×162px) on mobile
|
||||
│ [Muon / Fluid / Video canvas] │ max-w-sm (≈384×216px) on desktop
|
||||
│ │
|
||||
├──────────┬──────────┬──────────┬────────────┤
|
||||
│ ▶ / ⏸ │ Muon │ Fluid │ Video │ ← tab row (Video hidden when no URL)
|
||||
└──────────┴──────────┴──────────┴────────────┘
|
||||
```
|
||||
|
||||
The play/pause button starts paused. The visualizers run in **idle mode** (the existing
|
||||
`sin(performance.now() * 0.0011)` idle sway) without audio. Pressing play streams the Subsonic
|
||||
audio through the hook; the visualizers become audio-reactive.
|
||||
|
||||
The existing metadata row (title, artist, back link) sits beside the widget on desktop exactly
|
||||
as it does beside the cover art today, just with the widget instead of the `<img>`.
|
||||
|
||||
---
|
||||
|
||||
## New component: `VisualizerPreviewWidget`
|
||||
|
||||
**File:** `frontend/src/components/VisualizerPreviewWidget.tsx`
|
||||
|
||||
### Props
|
||||
|
||||
```typescript
|
||||
interface VisualizerPreviewWidgetProps {
|
||||
audioUrl: string // subsonicStreamUrl(song.id)
|
||||
coverArtUrl?: string // for palette extraction
|
||||
videoUrl?: string // from karaoke config; if absent, Video tab is hidden
|
||||
songTitle?: string // displayed as overlay when idle
|
||||
className?: string
|
||||
}
|
||||
```
|
||||
|
||||
### Internal structure
|
||||
|
||||
```
|
||||
VisualizerPreviewWidget
|
||||
├── <audio ref={audioRef} src={audioUrl} crossOrigin="anonymous" preload="metadata" />
|
||||
├── useVisualizerPalette({ coverArtUrl }) → paletteRef
|
||||
├── useAudioFeatures({ audioCtxRef, analysisAudioRef: audioRef, isPlayingRef, analysisSourceKey }) → featuresRef
|
||||
├── useState: mode: 'muon' | 'fluid' | 'video' (default: 'muon')
|
||||
├── useState: isPlaying: boolean (default: false)
|
||||
├── VisualizerBackdrop (when mode !== 'video')
|
||||
│ mode={mode}
|
||||
│ featuresRef={featuresRef}
|
||||
│ paletteRef={paletteRef}
|
||||
│ width={canvasW} height={canvasH}
|
||||
├── <video> element (when mode === 'video')
|
||||
└── tab row + play/pause overlay button
|
||||
```
|
||||
|
||||
### Audio playback
|
||||
|
||||
The widget manages its own `<audio>` element. With the current Muon extractor integration,
|
||||
`useAudioFeatures` already consumes `analysisAudioRef`. In this widget there are no stems, so
|
||||
the same `<audio>` element is used for both playback and analysis (`analysisAudioRef: audioRef`).
|
||||
No separate hidden analysis element is needed.
|
||||
|
||||
**Play/pause — with error handling:**
|
||||
```typescript
|
||||
const togglePlay = async () => {
|
||||
const audio = audioRef.current
|
||||
if (!audio) return
|
||||
if (isPlaying) {
|
||||
audio.pause()
|
||||
setIsPlaying(false)
|
||||
} else {
|
||||
const ctx = ensureCtx()
|
||||
if (ctx.state === 'suspended') await ctx.resume()
|
||||
try {
|
||||
await audio.play()
|
||||
setIsPlaying(true)
|
||||
} catch {
|
||||
// Rejected by browser policy or network error; stay paused.
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`audio.play()` returns a Promise that can reject (browser autoplay policy, network failure,
|
||||
unsupported codec). `setIsPlaying(true)` must only be called after it resolves.
|
||||
|
||||
**`audio.ended` — registered in `useEffect` with cleanup:**
|
||||
```typescript
|
||||
useEffect(() => {
|
||||
const audio = audioRef.current
|
||||
if (!audio) return
|
||||
const onEnded = () => setIsPlaying(false)
|
||||
audio.addEventListener('ended', onEnded)
|
||||
return () => audio.removeEventListener('ended', onEnded)
|
||||
}, [])
|
||||
```
|
||||
|
||||
Using a JSX `onEnded` prop is equivalent and preferred if the element is rendered in JSX.
|
||||
|
||||
**`audioUrl` change while playing — reset state:**
|
||||
|
||||
React Router may reuse the same `VisualizerPreviewWidget` instance when navigating from
|
||||
`/tracks/1` to `/tracks/2`. The `<audio>` `src` updates to the new URL (browser pauses and
|
||||
resets), but `isPlaying` stays `true`, leaving the UI in an inconsistent state.
|
||||
|
||||
```typescript
|
||||
useEffect(() => {
|
||||
setIsPlaying(false)
|
||||
audioRef.current?.pause()
|
||||
}, [audioUrl])
|
||||
```
|
||||
|
||||
**AudioContext suspension on tab switch:**
|
||||
|
||||
Browsers suspend AudioContext when the page is hidden. On tab restore, audio plays again but
|
||||
analysis stays silent because the context is suspended. Handle with:
|
||||
|
||||
```typescript
|
||||
useEffect(() => {
|
||||
const onVisible = () => {
|
||||
if (isPlaying && audioCtxRef.current?.state === 'suspended') {
|
||||
audioCtxRef.current.resume().catch(() => {})
|
||||
}
|
||||
}
|
||||
document.addEventListener('visibilitychange', onVisible)
|
||||
return () => document.removeEventListener('visibilitychange', onVisible)
|
||||
}, [isPlaying])
|
||||
```
|
||||
|
||||
### Video mode
|
||||
|
||||
In Video mode the canvas is hidden and a `<video>` element is shown instead.
|
||||
|
||||
Only direct media URLs (e.g. `.mp4`, `.webm`, `.m4v`, `.ogg`) should enable the Video tab.
|
||||
Provider page URLs (e.g. standard YouTube watch links) are not valid `<video src>` resources.
|
||||
|
||||
**Two edge cases to avoid:**
|
||||
|
||||
1. **Double audio** — `muted={!isPlaying}` would unmute the `<video>` while `<audio>` is also
|
||||
playing, producing two audio tracks. The `<video>` must always be `muted`; audio comes
|
||||
exclusively from the `<audio>` element.
|
||||
|
||||
2. **`autoPlay={isPlaying}` is unreliable** — React's `autoPlay` only sets the HTML attribute at
|
||||
mount time. It won't start or stop the video when `isPlaying` or `mode` changes after mount.
|
||||
Use a `videoRef` and call `.play()`/`.pause()` imperatively.
|
||||
|
||||
```tsx
|
||||
const videoRef = useRef<HTMLVideoElement>(null)
|
||||
|
||||
// Sync video play/pause when isPlaying or mode changes
|
||||
useEffect(() => {
|
||||
const video = videoRef.current
|
||||
if (!video) return
|
||||
if (mode === 'video' && isPlaying) {
|
||||
video.play().catch(() => {})
|
||||
} else {
|
||||
video.pause()
|
||||
}
|
||||
}, [isPlaying, mode])
|
||||
|
||||
{mode === 'video' && videoUrl && (
|
||||
<video
|
||||
ref={videoRef}
|
||||
src={videoUrl}
|
||||
className="absolute inset-0 h-full w-full object-cover"
|
||||
loop
|
||||
muted // always muted — audio comes from the <audio> element
|
||||
playsInline
|
||||
/>
|
||||
)}
|
||||
```
|
||||
|
||||
Video and audio stay in sync passively through the play/pause button — both toggle together.
|
||||
This is a preview so frame-perfect sync is not required.
|
||||
|
||||
### Tab row
|
||||
|
||||
Include ARIA attributes so the tab row is screen-reader accessible:
|
||||
|
||||
```tsx
|
||||
const tabs = [
|
||||
{ id: 'muon', label: 'Muon' },
|
||||
{ id: 'fluid', label: 'Fluid' },
|
||||
...(videoUrl ? [{ id: 'video', label: 'Video' }] : []),
|
||||
] as const
|
||||
|
||||
<div className="flex border-t border-white/10" role="tablist" aria-label="Visualizer mode">
|
||||
<button
|
||||
onClick={togglePlay}
|
||||
aria-label={isPlaying ? 'Pause' : 'Play'}
|
||||
className="px-3 py-1.5 text-sm ..."
|
||||
>
|
||||
{isPlaying ? '⏸' : '▶'}
|
||||
</button>
|
||||
{tabs.map((tab) => (
|
||||
<button
|
||||
key={tab.id}
|
||||
role="tab"
|
||||
aria-selected={mode === tab.id}
|
||||
onClick={() => setMode(tab.id)}
|
||||
className={`flex-1 py-1.5 text-xs ... ${mode === tab.id ? 'active styles' : ''}`}
|
||||
>
|
||||
{tab.label}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
```
|
||||
|
||||
### Canvas sizing
|
||||
|
||||
`VisualizerBackdrop` needs explicit `width` / `height` in pixels. Use a `ResizeObserver` on the
|
||||
container div, initialized in a `useLayoutEffect` so the first measurement happens before paint:
|
||||
|
||||
```typescript
|
||||
const containerRef = useRef<HTMLDivElement>(null)
|
||||
const [dims, setDims] = useState<{ w: number; h: number } | null>(null)
|
||||
|
||||
useLayoutEffect(() => {
|
||||
const el = containerRef.current
|
||||
if (!el) return
|
||||
const measure = () => {
|
||||
const { width } = el.getBoundingClientRect()
|
||||
if (width > 0) setDims({ w: Math.round(width), h: Math.round(width * (9 / 16)) })
|
||||
}
|
||||
measure()
|
||||
if (typeof ResizeObserver === 'undefined') {
|
||||
window.addEventListener('resize', measure)
|
||||
return () => window.removeEventListener('resize', measure)
|
||||
}
|
||||
const ro = new ResizeObserver(measure)
|
||||
ro.observe(el)
|
||||
return () => ro.disconnect()
|
||||
}, [])
|
||||
```
|
||||
|
||||
`VisualizerBackdrop` is only mounted once `dims` is non-null, avoiding a render with 0×0 that
|
||||
would trigger an unnecessary canvas reset. While `dims` is null (first frame before
|
||||
`useLayoutEffect` fires), show a placeholder so the container doesn't collapse:
|
||||
|
||||
```tsx
|
||||
<div ref={containerRef} style={{ aspectRatio: '16/9' }} className="relative w-full bg-gray-900">
|
||||
{dims ? (
|
||||
<VisualizerBackdrop ... width={dims.w} height={dims.h} />
|
||||
) : (
|
||||
<div className="absolute inset-0 flex items-center justify-center">
|
||||
<Disc3 className="h-8 w-8 text-gray-600 animate-spin-slow" />
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
```
|
||||
|
||||
In practice `useLayoutEffect` fires synchronously before paint so the placeholder is never
|
||||
visible. It guards against the zero-height flash that would occur without it.
|
||||
|
||||
---
|
||||
|
||||
## Canvas scaling — shared fix for `LyricsLivePreview` and `VisualizerPreviewWidget`
|
||||
|
||||
### Problem
|
||||
|
||||
Both `MuonVisualizerCanvas` and `FluidVisualizerCanvas` list `width` and `height` in their
|
||||
`useEffect` dependency arrays:
|
||||
|
||||
```typescript
|
||||
// current — in both canvas files
|
||||
}, [featuresRef, paletteRef, width, height])
|
||||
```
|
||||
|
||||
This means every window resize causes a **full canvas restart**: the RAF loop is cancelled and
|
||||
rebuilt, particles are re-allocated (`buildParticles`), all state is reset. The result is a
|
||||
visible flash on every resize.
|
||||
|
||||
This restart is unnecessary. Both canvases already self-adapt to their container each frame:
|
||||
|
||||
```typescript
|
||||
// inside the RAF loop — both canvases already do this
|
||||
const w = Math.max(1, Math.round(canvas.clientWidth || 0), Math.round(width || 0))
|
||||
const h = Math.max(1, Math.round(canvas.clientHeight || 0), Math.round(height || 0))
|
||||
if (canvas.width !== w || canvas.height !== h) {
|
||||
canvas.width = w // updates pixel buffer on next frame
|
||||
canvas.height = h
|
||||
}
|
||||
```
|
||||
|
||||
`canvas.clientWidth`/`canvas.clientHeight` reflect the CSS-rendered size, which already changes
|
||||
when the parent container resizes. The `width`/`height` props are only needed as a non-zero
|
||||
fallback if `clientWidth`/`clientHeight` haven't settled yet.
|
||||
|
||||
### Fix — remove `width` and `height` from canvas effect deps
|
||||
|
||||
**`MuonVisualizerCanvas.tsx`** and **`FluidVisualizerCanvas.tsx`**, change:
|
||||
|
||||
```typescript
|
||||
// before
|
||||
}, [featuresRef, paletteRef, width, height])
|
||||
|
||||
// after
|
||||
}, [featuresRef, paletteRef])
|
||||
// width and height are read from canvas.clientWidth/clientHeight each frame — no restart needed
|
||||
```
|
||||
|
||||
This change applies to both canvas files and eliminates all resize-triggered restarts regardless
|
||||
of whether the canvas is hosted inside `LyricsLivePreview` or `VisualizerPreviewWidget`.
|
||||
|
||||
### `LyricsLivePreview` — existing scaling mechanism
|
||||
|
||||
`LyricsLivePreview` already has a `ResizeObserver` on `containerRef` (line 2544) that updates
|
||||
`visualizerSize` and passes it to `VisualizerBackdrop` as `width`/`height`. After removing these
|
||||
from the canvas deps, `visualizerSize` only affects the prop value (used as the `clientWidth`
|
||||
fallback) — it no longer triggers canvas restarts. The `ResizeObserver` in LyricsLivePreview can
|
||||
remain as-is: its updates are now free (no DOM side-effects from the canvas side).
|
||||
|
||||
One remaining issue: `visualizerSize` is initialized as `{ width: 960, height: 540 }` (hardcoded),
|
||||
which is replaced after the first `ResizeObserver` callback. After the dep array fix this no longer
|
||||
causes a canvas restart, so the hardcoded initial value is harmless.
|
||||
|
||||
### Files changed by this fix
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx` | Remove `width`, `height` from `useEffect` dep array |
|
||||
| `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx` | Remove `width`, `height` from `useEffect` dep array |
|
||||
|
||||
---
|
||||
|
||||
## AudioContext ownership
|
||||
|
||||
`VisualizerPreviewWidget` owns its own `AudioContext` via a ref:
|
||||
|
||||
```typescript
|
||||
const audioCtxRef = useRef<AudioContext | null>(null)
|
||||
|
||||
const ensureCtx = () => {
|
||||
if (!audioCtxRef.current || audioCtxRef.current.state === 'closed') {
|
||||
audioCtxRef.current = new AudioContext()
|
||||
}
|
||||
return audioCtxRef.current
|
||||
}
|
||||
```
|
||||
|
||||
Disposed in a cleanup `useEffect`:
|
||||
```typescript
|
||||
useEffect(() => () => { audioCtxRef.current?.close() }, [])
|
||||
```
|
||||
|
||||
This is intentionally separate from the queue's AudioContext — the preview is an independent
|
||||
player and must not share or interfere with the SingUI queue playback bridge.
|
||||
|
||||
---
|
||||
|
||||
## Integration into `TrackDetailPage`
|
||||
|
||||
### 1. Import
|
||||
|
||||
```typescript
|
||||
import VisualizerPreviewWidget from '../components/VisualizerPreviewWidget'
|
||||
```
|
||||
|
||||
No lazy load required — the component uses `VisualizerBackdrop` which is canvas-based, no
|
||||
heavy dynamic imports.
|
||||
|
||||
### 2. Compute audio URL
|
||||
|
||||
```typescript
|
||||
import { subsonicStreamUrl } from '../api/client'
|
||||
|
||||
const audioStreamUrl = useMemo(
|
||||
() => (song ? subsonicStreamUrl(song.id) : ''),
|
||||
[song?.id]
|
||||
)
|
||||
```
|
||||
|
||||
`TrackDetail.tsx` renders `null` early when `!song`, but `useMemo` runs unconditionally (hooks
|
||||
cannot be conditional). Guard `song.id` access inside the memo with `song ?` to avoid a
|
||||
runtime crash if the memo ever evaluates before the song has loaded.
|
||||
|
||||
Already available: `coverArtUrl`, `videoUrlInput` (the project video URL).
|
||||
|
||||
### 3. Replace the cover art block
|
||||
|
||||
Current (lines 157–163 of TrackDetail.tsx):
|
||||
```tsx
|
||||
<div className="h-24 w-24 shrink-0 overflow-hidden rounded-lg bg-gray-800 flex items-center justify-center">
|
||||
{coverArtUrl ? (
|
||||
<img src={coverArtUrl} alt={`${song.album} cover art`} className="h-full w-full object-cover" />
|
||||
) : (
|
||||
<Disc3 className="h-10 w-10 text-gray-500" />
|
||||
)}
|
||||
</div>
|
||||
```
|
||||
|
||||
Replace with:
|
||||
```tsx
|
||||
<VisualizerPreviewWidget
|
||||
audioUrl={audioStreamUrl}
|
||||
coverArtUrl={coverArtUrl || undefined}
|
||||
videoUrl={videoUrlInput.trim() || undefined}
|
||||
songTitle={song.title}
|
||||
className="w-48 shrink-0 rounded-lg overflow-hidden"
|
||||
/>
|
||||
```
|
||||
|
||||
Width `w-48` (192px) → height auto at 9:16 = 108px thumbnail. Adjust to taste.
|
||||
|
||||
---
|
||||
|
||||
## Relationship to the Muon Audio Analyzer plan
|
||||
|
||||
Current state (already implemented in codebase):
|
||||
|
||||
- `useAudioFeatures` uses `MuonFeatureExtractor` and accepts `analysisAudioRef`.
|
||||
- In `VisualizerPreviewWidget` there are **no stems**, so playback and analysis share one
|
||||
`<audio>` ref.
|
||||
- Rich `frequencyBands`, `beatInfo`, and `patternInfo` fields are available for visual response.
|
||||
|
||||
The preview widget works correctly before and after the Muon analyzer upgrade, as the
|
||||
backward-compatible `bass`, `treble`, `rms`, `drumIntensity`, `beatDetected` fields are preserved
|
||||
through the migration.
|
||||
|
||||
---
|
||||
|
||||
## Files changed
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `frontend/src/components/VisualizerPreviewWidget.tsx` | **New** |
|
||||
| `frontend/src/pages/TrackDetail.tsx` | Replace cover art block; add `audioStreamUrl` memo |
|
||||
| `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx` | Remove `width`, `height` from `useEffect` dep array |
|
||||
| `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx` | Remove `width`, `height` from `useEffect` dep array |
|
||||
| `SUMMARIES (Claude).md` | Append entry after implementation |
|
||||
|
||||
No changes required to `LyricsLivePreview` or `useAudioFeatures`.
|
||||
|
||||
---
|
||||
|
||||
## Implementation order
|
||||
|
||||
1. Remove `width`, `height` from `useEffect` deps in `MuonVisualizerCanvas.tsx` and `FluidVisualizerCanvas.tsx`
|
||||
2. Write `VisualizerPreviewWidget.tsx` — container, `useLayoutEffect` canvas sizing, tab row, play/pause
|
||||
3. Wire `VisualizerBackdrop` with real `paletteRef` from `useVisualizerPalette`
|
||||
4. Add `AudioContext` management
|
||||
5. Add `useAudioFeatures` with `audioCtxRef`, `analysisAudioRef`, and `isPlayingRef`
|
||||
6. Add Video mode `<video>` element
|
||||
7. Integrate into `TrackDetail.tsx`
|
||||
8. Run `tsc --noEmit`
|
||||
9. Update `SUMMARIES (Claude).md`
|
||||
|
||||
---
|
||||
|
||||
## Risks and mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| Browser blocks autoplay on page load | Widget starts paused; play only on explicit user interaction. `AudioContext` created on first play click to satisfy policy. |
|
||||
| Widget audio conflicts with SingUI queue playback | Register preview playback with the existing global audio registry (`registerPlayback`/`unregisterPlayback`) so starting preview pauses any currently active player first. |
|
||||
| `videoUrlInput` may not be loaded when widget first renders | `videoUrlInput` starts as `''`; the Video tab simply won't appear until the config fetch completes (async `useEffect` in TrackDetailPage). No special handling needed. |
|
||||
| `subsonicStreamUrl` requires auth or CORS headers | The URL goes through the backend proxy (`/api/...`), same as all other media requests. No special CORS handling needed. |
|
||||
| `ResizeObserver` not available (old Safari) | Wrap in `typeof ResizeObserver !== 'undefined'` check; fall back to fixed 384×216. |
|
||||
| Video `<video>` and `<audio>` both produce sound | `<video>` must always be `muted`; sound comes exclusively from the `<audio>` element. Using `muted={!isPlaying}` would produce double audio. |
|
||||
| `autoPlay` prop doesn't start video after mount | React's `autoPlay` only sets the HTML attribute at mount. Use `videoRef` + `useEffect([isPlaying, mode])` calling `video.play()`/`video.pause()` imperatively. |
|
||||
| `song.id` access before song is loaded | `useMemo` runs unconditionally before the `!song` guard. Guard with `song ? subsonicStreamUrl(song.id) : ''` inside the memo. |
|
||||
| Container collapse before `dims` is measured | Show a `<Disc3>` placeholder when `dims === null`. In practice invisible (fires before paint), but prevents zero-height flash. |
|
||||
@@ -0,0 +1,952 @@
|
||||
# Visualizer Diagnosis and Fixes Plan
|
||||
|
||||
Date: 2026-04-13
|
||||
Scope: Two-visualizer audit (Muon and Fluid) identifying root causes of the monocolor/idle-only
|
||||
symptoms and the `spawnBlob` sizing bug. Fixes are documented with exact code changes.
|
||||
|
||||
---
|
||||
|
||||
## Symptom summary
|
||||
|
||||
- Muon visualizer: monocolor, "singular pattern of dots", no audio reactivity
|
||||
- Fluid visualizer: sparse idle blobs, limited visual variety when multi-stem content plays
|
||||
|
||||
---
|
||||
|
||||
## Issue A — Analysis audio disconnected in multi-stem mode
|
||||
|
||||
### Root cause (shared by both visualizers)
|
||||
|
||||
`KaraokeTVPage` passes `audioUrl` to `KaraokePerformanceMode` → `LyricsLivePreview` only when
|
||||
there are **no stems**:
|
||||
|
||||
```tsx
|
||||
// KaraokeTV.tsx
|
||||
audioUrl={safeStems.length === 0 && sourceMedia ? sourceMedia.url : undefined}
|
||||
```
|
||||
|
||||
When stems are playing (always in SEK mode; also in project mode when a sep job has completed),
|
||||
`audioUrl` is `undefined`. `LyricsLivePreview` gates the hidden analysis element on this prop:
|
||||
|
||||
```tsx
|
||||
{/* Hidden full-mix analysis audio element (always mounted when source exists) */}
|
||||
{audioUrl && (
|
||||
<audio ref={analysisAudioRef} src={audioUrl} ... muted />
|
||||
)}
|
||||
```
|
||||
|
||||
With `audioUrl` absent, `analysisAudioRef.current` is `null`. `useAudioFeatures` returns early:
|
||||
|
||||
```typescript
|
||||
const audioEl = analysisAudioRef.current
|
||||
if (!ctx || !audioEl) return // ← always exits here
|
||||
```
|
||||
|
||||
`featuresRef` stays at its initial zero state forever:
|
||||
|
||||
```typescript
|
||||
{
|
||||
frequencyBands: { sub: 0, bass: 0, lowMid: 0, mid: 0, highMid: 0, presence: 0, brilliance: 0 },
|
||||
beatInfo: { isBeat: false, bpm: 0, beatPhase: 0, ... },
|
||||
patternInfo: { isBeatDrop: false, isBuildUp: false, isSilence: true, intensity: 0, ... },
|
||||
bass: 0, treble: 0, rms: 0, drumIntensity: 0, beatDetected: false,
|
||||
}
|
||||
```
|
||||
|
||||
Both canvases then run in pure idle mode — no beat response, no hue shift from audio, no drop
|
||||
bursts. For Muon: slowly orbiting dots at the idle sway frequency. For Fluid: sparse blobs
|
||||
driven only by the 4% baseline spawn rate.
|
||||
|
||||
### Fix — add `analysisAudioUrl` prop to `LyricsLivePreview` and `KaraokePerformanceMode`
|
||||
|
||||
#### 1. `LyricsLivePreview.tsx` — new prop
|
||||
|
||||
Add `analysisAudioUrl?: string` to `LyricsLivePreviewProps`. When present, it overrides
|
||||
`audioUrl` as the source for the hidden analysis element:
|
||||
|
||||
```tsx
|
||||
// Current
|
||||
{audioUrl && (
|
||||
<audio ref={analysisAudioRef} src={audioUrl} preload="auto" crossOrigin="anonymous" muted />
|
||||
)}
|
||||
|
||||
// After
|
||||
{(analysisAudioUrl || audioUrl) && (
|
||||
<audio
|
||||
ref={analysisAudioRef}
|
||||
src={analysisAudioUrl ?? audioUrl}
|
||||
preload="auto"
|
||||
crossOrigin="anonymous"
|
||||
muted
|
||||
/>
|
||||
)}
|
||||
```
|
||||
|
||||
Also update the `analysisSourceKey` passed to `useAudioFeatures` to track whichever URL is
|
||||
used, so the effect re-runs when the source changes:
|
||||
|
||||
```typescript
|
||||
// Current
|
||||
analysisSourceKey: audioUrl ?? null,
|
||||
|
||||
// After
|
||||
analysisSourceKey: analysisAudioUrl ?? audioUrl ?? null,
|
||||
```
|
||||
|
||||
#### 2. `KaraokePerformanceMode.tsx` — thread the prop through
|
||||
|
||||
Add `analysisAudioUrl?: string` to `KaraokePerformanceModeProps` and forward it:
|
||||
|
||||
```tsx
|
||||
<LyricsLivePreview
|
||||
...
|
||||
analysisAudioUrl={analysisAudioUrl}
|
||||
/>
|
||||
```
|
||||
|
||||
#### 3. `KaraokeTV.tsx` — derive and pass the analysis URL
|
||||
|
||||
**Project mode** — the source file (original recording) is always available as `sourceMedia.url`
|
||||
and is the ideal full-mix analysis source:
|
||||
|
||||
```tsx
|
||||
const analysisAudioUrl = useMemo(() => {
|
||||
if (isSekMode) {
|
||||
// SEK: use first stem as proxy analysis source (no full-mix file in manifest)
|
||||
const firstStem = sekManifest?.manifest?.contents?.stems?.[0]
|
||||
const firstStemPath = firstStem ? normalizeSekInnerPath(firstStem.file) : ''
|
||||
return firstStemPath && sekManifest
|
||||
? sekAssetUrl(sekManifest.id, firstStemPath, sekFolder || undefined)
|
||||
: undefined
|
||||
}
|
||||
// Project: use the source file (original recording)
|
||||
if (!detail) return undefined
|
||||
const sourceFile = detail.sep_job.files.find((f) => f.type === 'source')
|
||||
return sourceFile ? downloadUrl(detail.sep_job.id, sourceFile.name) : undefined
|
||||
}, [isSekMode, sekManifest, sekFolder, detail])
|
||||
```
|
||||
|
||||
Then pass it:
|
||||
|
||||
```tsx
|
||||
<KaraokePerformanceMode
|
||||
...
|
||||
analysisAudioUrl={analysisAudioUrl}
|
||||
/>
|
||||
```
|
||||
|
||||
**Notes on the SEK fallback:**
|
||||
- SEK packages do not contain a `contents.audio` or `source` field; only `stems`, `lyrics`, and
|
||||
optionally `video`.
|
||||
- Using the first stem gives partial frequency data (one instrument's spectrum). It will be
|
||||
better than zero but will not capture the full mix. A future improvement could mix all stem
|
||||
`AnalyserNode` outputs, but that requires connecting all stems to a single analyser.
|
||||
- The "leading" or "drums" stem (if present) is more representative than a random first stem.
|
||||
The `normalizeLyricsTrackKey` logic already knows how to rank stems — a `pickAnalysisStem`
|
||||
helper could use similar ranking. For now, first stem is acceptable.
|
||||
|
||||
---
|
||||
|
||||
## Issue B — Muon: all particles share one hue per frame
|
||||
|
||||
### Root cause
|
||||
|
||||
In `MuonVisualizerCanvas.tsx`, one `hue` is derived from the audio state each frame:
|
||||
|
||||
```typescript
|
||||
const hue = monoColor
|
||||
? p.seedHue
|
||||
: drum > 0.01
|
||||
? ((p.seedHue + drum * 20 * hueRange + treble * 15 * hueRange + ...) + 360) % 360
|
||||
: ((p.seedHue + Math.sin(now * 0.00018) * 15 * hueRange) + 360) % 360
|
||||
```
|
||||
|
||||
Every particle in every lane then receives the same fill:
|
||||
|
||||
```typescript
|
||||
ctx.fillStyle = `hsla(${hue}, 92%, 64%, ${alpha})`
|
||||
```
|
||||
|
||||
`hue_range` only controls how much the frame-level hue shifts with audio — it does not spread
|
||||
colors across particles. The result is a single color at any instant regardless of how many
|
||||
lanes are visible.
|
||||
|
||||
### Fix — per-lane hue offset proportional to `hue_range`
|
||||
|
||||
Replace the per-particle fill with a lane-offset hue. 8 lanes × 45° at `hue_range = 1.0`
|
||||
produces a full 360° rainbow spread. At `hue_range = 0`, all lanes collapse to `seedHue`
|
||||
(monocolor), making `mono_color = true` still useful for a color-locked look.
|
||||
|
||||
```typescript
|
||||
// Before (inside particle loop)
|
||||
ctx.fillStyle = `hsla(${hue}, 92%, 64%, ${alpha})`
|
||||
|
||||
// After
|
||||
const laneHue = (hue + pt.lane * (hueRange * 45)) % 360
|
||||
ctx.fillStyle = `hsla(${laneHue}, 92%, 64%, ${alpha})`
|
||||
```
|
||||
|
||||
When `mono_color` is true, `hueRange` should be ignored so the lane offset is zero. Guard:
|
||||
|
||||
```typescript
|
||||
const laneHue = monoColor
|
||||
? hue
|
||||
: (hue + pt.lane * (hueRange * 45)) % 360
|
||||
ctx.fillStyle = `hsla(${laneHue}, 92%, 64%, ${alpha})`
|
||||
```
|
||||
|
||||
The glow shadow color can also vary per lane for extra depth:
|
||||
|
||||
```typescript
|
||||
ctx.shadowColor = monoColor
|
||||
? p.highlightColor
|
||||
: `hsla(${(laneHue + 30) % 360}, 100%, 80%, 1)`
|
||||
```
|
||||
|
||||
Or leave `shadowColor = p.highlightColor` unchanged if the extra glow variation is distracting.
|
||||
|
||||
**Effect at defaults (`hue_range = 1.0`, 8 lanes):**
|
||||
- Lane 0: seedHue + 0°
|
||||
- Lane 1: seedHue + 45°
|
||||
- Lane 2: seedHue + 90°
|
||||
- ...
|
||||
- Lane 7: seedHue + 315°
|
||||
|
||||
This creates concentric rainbow ellipses that react to audio together (shared frame-level hue
|
||||
drift) but remain visually distinct from each other.
|
||||
|
||||
---
|
||||
|
||||
## Issue C — Fluid: `spawnBlob` uses captured props instead of live canvas dimensions
|
||||
|
||||
### Root cause
|
||||
|
||||
`spawnBlob` is a closure defined inside `useEffect`. It captures `width` and `height` from the
|
||||
component props at the time the effect first runs:
|
||||
|
||||
```typescript
|
||||
const spawnBlob = (
|
||||
strength: number,
|
||||
blobLifeScale: number,
|
||||
splatForce: number,
|
||||
origin?: { ... },
|
||||
) => {
|
||||
const w = Math.max(1, width) // ← captured prop, stale after resize
|
||||
const h = Math.max(1, height)
|
||||
...
|
||||
x: origin?.x ?? (Math.random() * w),
|
||||
y: origin?.y ?? (Math.random() * h),
|
||||
```
|
||||
|
||||
The `render` function computes live `w`/`h` from `canvas.clientWidth`/`canvas.clientHeight`
|
||||
each frame, but `spawnBlob` doesn't receive these. After a resize, beat-burst blobs and the
|
||||
ambient `spawnBlob` calls spawn within the old canvas area, leaving a dead band on the
|
||||
newly-revealed edges.
|
||||
|
||||
Ambient blobs spawned inline in the `render` body **do** use live `w`/`h` — the inconsistency
|
||||
is only in `spawnBlob`.
|
||||
|
||||
### Fix — pass live `w`/`h` to `spawnBlob`
|
||||
|
||||
Add `w` and `h` parameters to `spawnBlob` and use them instead of the captured props:
|
||||
|
||||
```typescript
|
||||
// Before
|
||||
const spawnBlob = (
|
||||
strength: number,
|
||||
blobLifeScale: number,
|
||||
splatForce: number,
|
||||
origin?: { x?: number; y?: number; angle?: number; color?: string },
|
||||
) => {
|
||||
const w = Math.max(1, width)
|
||||
const h = Math.max(1, height)
|
||||
...
|
||||
}
|
||||
|
||||
// After
|
||||
const spawnBlob = (
|
||||
strength: number,
|
||||
blobLifeScale: number,
|
||||
splatForce: number,
|
||||
w: number,
|
||||
h: number,
|
||||
origin?: { x?: number; y?: number; angle?: number; color?: string },
|
||||
) => {
|
||||
// w and h received from caller — always live canvas dimensions
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
Update all call sites inside `render` to pass `w, h`:
|
||||
|
||||
```typescript
|
||||
// Beat burst calls
|
||||
for (let i = 0; i < count; i += 1) spawnBlob(0.6 + bass * 0.8, blobLife, splatForce, w, h)
|
||||
|
||||
// Phase-orbit call
|
||||
spawnBlob(
|
||||
0.28 + kick * 0.45 + phasePulse * 0.3 + dropBoost * 0.35,
|
||||
blobLife,
|
||||
splatForce,
|
||||
w,
|
||||
h,
|
||||
{ x: px, y: py, angle: phaseAngle + Math.PI * 0.5, color: ... },
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Issue D — Fluid: `background_persistence` formula is inverted
|
||||
|
||||
### Root cause
|
||||
|
||||
The background fill alpha is computed as:
|
||||
|
||||
```typescript
|
||||
ctx.globalAlpha = 0.20 + (1 - bass) * 0.16 * bgPersistence
|
||||
```
|
||||
|
||||
When `bgPersistence` increases, alpha increases → background clears faster → **less**
|
||||
persistence. But the parameter is named `background_persistence` which intuitively implies
|
||||
higher = more persistent trails.
|
||||
|
||||
At defaults (`bgPersistence = 1.0`, `bass = 0`): alpha = 0.36.
|
||||
At `bgPersistence = 3.0`: alpha = 0.68 — trails clear quickly.
|
||||
At `bgPersistence = 0.0`: alpha = 0.20 — trails persist longest.
|
||||
|
||||
The PerformanceSidebar slider direction is also confusing: dragging right increases
|
||||
"persistence" but visually reduces trail length.
|
||||
|
||||
### Fix — two options
|
||||
|
||||
**Option A (rename to `background_clearing`):**
|
||||
Rename the parameter in `VisualizerSettings`, `VizSettings`, all defaults, and the sidebar
|
||||
label to `background_clearing`. No formula change. Semantically correct: higher value → faster
|
||||
clearing.
|
||||
|
||||
**Option B (invert the formula):**
|
||||
Keep the name `background_persistence` but invert how the multiplier is applied:
|
||||
|
||||
```typescript
|
||||
// Before
|
||||
ctx.globalAlpha = 0.20 + (1 - bass) * 0.16 * bgPersistence
|
||||
|
||||
// After — higher bgPersistence → lower alpha → more persistent trails
|
||||
ctx.globalAlpha = 0.20 + (1 - bass) * 0.16 / Math.max(0.1, bgPersistence)
|
||||
```
|
||||
|
||||
At `bgPersistence = 1.0`: alpha = 0.36 (same default behavior).
|
||||
At `bgPersistence = 3.0`: alpha = `0.20 + 0.053` = 0.25 (long trails).
|
||||
At `bgPersistence = 0.1`: alpha = `0.20 + 1.6` → clamped to 1.0 by `globalAlpha` (rapid clear).
|
||||
|
||||
Option B is less disruptive (no rename ripple) and makes the slider direction intuitive.
|
||||
**Recommended: Option B.**
|
||||
|
||||
---
|
||||
|
||||
## Fluid visualizer — what is NOT broken
|
||||
|
||||
- **Color variety** — `speakerPalette` always has 8 colors: from the SEK/project manifest's
|
||||
`dynamic_palette` speaker entries when available, otherwise from the cover art seed hue,
|
||||
otherwise the DEFAULT_PALETTE 8-color array. Blobs cycle through these via `colorOffset`,
|
||||
so color variety is present in all modes.
|
||||
- **Idle baseline** — the 4% ambient spawn rate and `idle = 0.5 + 0.5 * Math.sin(...)` ensure
|
||||
the canvas is never completely empty, even with zero audio features.
|
||||
|
||||
---
|
||||
|
||||
## Issue E — Fluid: wrong rendering backend (canvas blobs, not WebGL fluid sim)
|
||||
|
||||
### Root cause — the original plan was never implemented
|
||||
|
||||
`docs/2026-04-08_Visualizer_Integration_Plan.md` (Phase 3) and `SUMMARIES (Claude).md` both
|
||||
specify the Fluid visualizer as a wrapper around **`PavelDoGreat/WebGL-Fluid-Simulation`** —
|
||||
a WebGL shader-based fluid dynamics simulation with pressure solving, velocity advection, and
|
||||
vorticity. The current `FluidVisualizerCanvas.tsx` is a Canvas 2D blob renderer written as a
|
||||
placeholder. The SUMMARIES file explicitly notes: *"Most Fluid WebGL params (bloom, sunrays,
|
||||
pressure, shading) are not applicable to the current canvas blob renderer."*
|
||||
|
||||
### Why this causes single-mode confetti
|
||||
|
||||
Canvas 2D radial gradients with simple Euler-integration velocity are physically limited.
|
||||
Regardless of how the parameters (spawn density, drag, blob life, etc.) are tuned, the result
|
||||
is always the same visual category: small colored discs flying outward and fading. The presets
|
||||
(Ink, Foam, Lava, Mist, Storm) were designed with WebGL fluid parameters in mind:
|
||||
|
||||
| Preset | Intended WebGL behavior | What canvas blobs produce |
|
||||
|---|---|---|
|
||||
| **Ink** | Low dissipation, high curl → thick, slow-moving ribbons of color | Many medium blobs, same physics |
|
||||
| **Foam** | High dissipation, tiny splats, many per beat → bubbly texture | Smaller, faster-fading blobs |
|
||||
| **Lava** | Very low dissipation, high viscosity → slow swelling masses | Large slow blobs |
|
||||
| **Mist** | High velocity dissipation, near-invisible forces → translucent wisps | Faint fast blobs |
|
||||
| **Storm** | High curl (60+), max force → violent rotational turbulence | Dense fast blobs |
|
||||
|
||||
All five look like confetti variants because the canvas renderer doesn't have:
|
||||
- **Velocity field advection** — fluid carries its own motion; canvas blobs travel independently
|
||||
- **Vorticity / curl** — the fluid swirls; blobs just drift straight
|
||||
- **Pressure solving** — fluid fills space and pushes back; blobs don't interact
|
||||
|
||||
### Fix — replace `FluidVisualizerCanvas.tsx` with a WebGL fluid sim wrapper
|
||||
|
||||
#### Step 1 — Acquire the sim
|
||||
|
||||
Implementation note (this repository): **Option B was used** with one additional runtime-safety
|
||||
step: the package's `dist/webgl-fluid.mjs` is vendored into
|
||||
`frontend/src/components/visualizers/fluid/webgl-fluid.mjs`, and `fluidSim.ts` dynamically
|
||||
imports that local file. This avoids environment-specific Vite package-resolution failures while
|
||||
still using the package runtime.
|
||||
|
||||
**Option A (recommended): vendor `script.js` directly**
|
||||
|
||||
Copy `PavelDoGreat/WebGL-Fluid-Simulation/script.js` into
|
||||
`frontend/src/components/visualizers/fluid/fluidSim.js`. Strip the 4 event handler blocks
|
||||
(`canvas.addEventListener('mousedown'|'mousemove'|'touchstart'|'touchmove')`). Expose two
|
||||
module exports at the bottom:
|
||||
|
||||
```js
|
||||
// Add at the bottom of fluidSim.js
|
||||
export function initFluidSim(canvas, initialConfig) { /* wraps existing canvas init */ }
|
||||
export function splatFluid(x, y, dx, dy, color) { /* wraps existing splat() */ }
|
||||
export function updateFluidConfig(patch) { Object.assign(config, patch) }
|
||||
```
|
||||
|
||||
No npm dependency needed. The sim is ~1000 lines of self-contained WebGL — no build step,
|
||||
no transitive deps. Its WebGL context attaches directly to the `<canvas>` element passed in.
|
||||
|
||||
**Option B: npm package**
|
||||
|
||||
```bash
|
||||
npm install webgl-fluid
|
||||
```
|
||||
|
||||
Exposes `createFluid(canvas)` → `{ splat, config }`. Simpler but the package is a community
|
||||
wrapper and may lag behind upstream bug fixes.
|
||||
|
||||
**Recommended: Option A.** It gives direct control over config fields and is MIT-licensed.
|
||||
|
||||
#### Step 2 — Rewrite `FluidVisualizerCanvas.tsx`
|
||||
|
||||
```tsx
|
||||
import { useEffect, useRef } from 'react'
|
||||
import type { MutableRefObject } from 'react'
|
||||
import type { AudioFeatures } from '../../hooks/useAudioFeatures'
|
||||
import type { VisualizerPalette } from '../../hooks/useVisualizerPalette'
|
||||
import type { VisualizerSettings } from './VisualizerBackdrop'
|
||||
import { initFluidSim, splatFluid, updateFluidConfig } from './fluid/fluidSim'
|
||||
|
||||
type FluidSettings = NonNullable<VisualizerSettings['fluid']>
|
||||
|
||||
export default function FluidVisualizerCanvas({
|
||||
featuresRef, paletteRef, width, height, settings, fpsCap,
|
||||
}: FluidVisualizerCanvasProps) {
|
||||
const canvasRef = useRef<HTMLCanvasElement>(null)
|
||||
const settingsRef = useRef(settings)
|
||||
settingsRef.current = settings
|
||||
|
||||
useEffect(() => {
|
||||
const canvas = canvasRef.current
|
||||
if (!canvas) return
|
||||
|
||||
// Initialise fluid sim with defaults derived from initial settings
|
||||
const s = settingsRef.current
|
||||
initFluidSim(canvas, {
|
||||
SIM_RESOLUTION: 128,
|
||||
DYE_RESOLUTION: 1024,
|
||||
DENSITY_DISSIPATION: fluidDensityDissipation(s),
|
||||
VELOCITY_DISSIPATION: fluidVelocityDissipation(s),
|
||||
PRESSURE: 0.8,
|
||||
CURL: fluidCurl(s),
|
||||
SPLAT_RADIUS: fluidSplatRadius(s),
|
||||
COLORFUL: false, // palette-locked; we supply colors from paletteRef
|
||||
BLOOM: true,
|
||||
BLOOM_INTENSITY: 0.8,
|
||||
SUNRAYS: false,
|
||||
BACK_COLOR: { r: 0, g: 0, b: 0 },
|
||||
})
|
||||
|
||||
let colorOffset = 0
|
||||
const nextColor = () => {
|
||||
const palette = paletteRef.current.speakerPalette
|
||||
if (!palette.length) return hexToRgb(paletteRef.current.highlightColor)
|
||||
colorOffset = (colorOffset + Math.max(0, settingsRef.current?.color_speed ?? 1) * 0.5) % palette.length
|
||||
return hexToRgb(palette[Math.floor(colorOffset)])
|
||||
}
|
||||
|
||||
let prevBeatPhase = 0
|
||||
let dropBoost = 0
|
||||
let lastDropAt = 0
|
||||
let lastFrameAt = 0
|
||||
let raf = 0
|
||||
|
||||
const loop = (ts: number) => {
|
||||
raf = requestAnimationFrame(loop)
|
||||
const minFrameMs = 1000 / Math.max(1, fpsCap ?? 60)
|
||||
if (ts - lastFrameAt < minFrameMs - 1) return
|
||||
lastFrameAt = ts
|
||||
|
||||
const f = featuresRef.current
|
||||
const s = settingsRef.current
|
||||
const bass = clamp01(f.bass)
|
||||
const sub = clamp01(f.frequencyBands.sub)
|
||||
const rms = clamp01(f.rms)
|
||||
const treble = clamp01(f.treble)
|
||||
const beatPhase = clamp01(f.beatInfo.beatPhase)
|
||||
const kick = clamp01(sub * 0.72 + bass * 0.28)
|
||||
const phasePulse = Math.pow(Math.max(0, Math.sin(beatPhase * Math.PI)), 1.8)
|
||||
const phaseWrapped = beatPhase < prevBeatPhase - 0.35
|
||||
const idle = 0.5 + 0.5 * Math.sin(ts * 0.0011)
|
||||
|
||||
const dropTriggered = f.patternInfo.isBeatDrop && ts - lastDropAt > 500
|
||||
if (dropTriggered) { dropBoost = 1; lastDropAt = ts }
|
||||
dropBoost *= 0.96
|
||||
|
||||
// Live config modulation — update fluid sim parameters each frame
|
||||
updateFluidConfig({
|
||||
DENSITY_DISSIPATION: fluidDensityDissipation(s, kick),
|
||||
VELOCITY_DISSIPATION: fluidVelocityDissipation(s, rms),
|
||||
CURL: fluidCurl(s, treble, dropBoost),
|
||||
})
|
||||
|
||||
const densityScale = Math.max(0.2, s?.spawn_density ?? 1)
|
||||
const splatForce = Math.max(0.2, s?.splat_force ?? 1)
|
||||
const beatBurstCount = Math.max(0, s?.beat_burst_count ?? 1)
|
||||
|
||||
// Ambient splat — continuous low-level motion proportional to rms
|
||||
const ambientRate = (0.04 + rms * 0.35 + phasePulse * 0.08 + idle * 0.04) * densityScale
|
||||
if (Math.random() < ambientRate) {
|
||||
const angle = Math.random() * Math.PI * 2
|
||||
const force = (0.2 + rms * 0.8 + idle * 0.12) * splatForce * 400
|
||||
splatFluid(
|
||||
Math.random(), Math.random(),
|
||||
Math.cos(angle) * force,
|
||||
Math.sin(angle) * force,
|
||||
nextColor(),
|
||||
)
|
||||
}
|
||||
|
||||
// Beat burst
|
||||
if (f.beatDetected || phaseWrapped || dropTriggered) {
|
||||
const count = Math.max(1, Math.round((3 + kick * 5 + dropBoost * 4) * beatBurstCount))
|
||||
for (let i = 0; i < count; i++) {
|
||||
const angle = Math.random() * Math.PI * 2
|
||||
const force = (0.6 + bass * 1.2 + dropBoost * 0.6) * splatForce * 800
|
||||
splatFluid(
|
||||
Math.random(), Math.random(),
|
||||
Math.cos(angle) * force, Math.sin(angle) * force,
|
||||
dropTriggered ? hexToRgb(paletteRef.current.highlightColor) : nextColor(),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// Beat-phase orbit — sweep splat source around center following beat phase
|
||||
if (beatPhase > 0.001 || f.beatInfo.bpm > 0) {
|
||||
const phaseAngle = beatPhase * Math.PI * 2
|
||||
if (Math.random() < 0.08 + phasePulse * 0.22 + dropBoost * 0.18) {
|
||||
const force = (0.28 + kick * 0.5 + phasePulse * 0.4) * splatForce * 500
|
||||
splatFluid(
|
||||
0.5 + Math.cos(phaseAngle) * 0.28,
|
||||
0.5 + Math.sin(phaseAngle) * 0.20,
|
||||
Math.cos(phaseAngle + Math.PI / 2) * force,
|
||||
Math.sin(phaseAngle + Math.PI / 2) * force,
|
||||
dropBoost > 0.18 ? hexToRgb(paletteRef.current.highlightColor) : nextColor(),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
prevBeatPhase = beatPhase
|
||||
}
|
||||
|
||||
raf = requestAnimationFrame(loop)
|
||||
return () => cancelAnimationFrame(raf)
|
||||
}, [featuresRef, paletteRef])
|
||||
|
||||
return <canvas ref={canvasRef} className="h-full w-full" aria-hidden="true" />
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 3 — Config-mapping helpers
|
||||
|
||||
These translate `VizSettings['fluid']` into WebGL sim parameters. The naming is now accurate:
|
||||
|
||||
```typescript
|
||||
// All three helpers accept optional live audio values for per-frame modulation.
|
||||
|
||||
function fluidDensityDissipation(s?: FluidSettings, kick = 0): number {
|
||||
// background_persistence: higher → more persistent (lower dissipation)
|
||||
// kick: loud kick → longer trails (lower dissipation)
|
||||
const base = 0.98 - (s?.background_persistence ?? 1) * 0.025
|
||||
return clamp(base - kick * 0.04 * (s?.decay ?? 1), 0.85, 0.999)
|
||||
}
|
||||
|
||||
function fluidVelocityDissipation(s?: FluidSettings, rms = 0): number {
|
||||
// drag: higher → slower fluid (higher dissipation)
|
||||
const base = 0.96 - (1 / Math.max(0.5, s?.drag ?? 1)) * 0.06
|
||||
return clamp(base - rms * 0.04, 0.80, 0.999)
|
||||
}
|
||||
|
||||
function fluidCurl(s?: FluidSettings, treble = 0, dropBoost = 0): number {
|
||||
// splat_force mapping: higher force → more turbulence → higher curl
|
||||
const base = 15 + (s?.splat_force ?? 1) * 12
|
||||
return clamp(base + treble * 8 + dropBoost * 20, 0, 80)
|
||||
}
|
||||
|
||||
function fluidSplatRadius(s?: FluidSettings): number {
|
||||
// blob_life as splat radius proxy: larger life → larger radius
|
||||
return clamp(0.18 + (s?.blob_life ?? 1) * 0.06, 0.05, 0.5)
|
||||
}
|
||||
|
||||
function hexToRgb(hex: string): { r: number; g: number; b: number } {
|
||||
const m = /^#?([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})$/i.exec(hex)
|
||||
if (!m) return { r: 0.5, g: 0.5, b: 0.8 }
|
||||
return { r: parseInt(m[1], 16) / 255, g: parseInt(m[2], 16) / 255, b: parseInt(m[3], 16) / 255 }
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 4 — Update `FLUID_PRESETS` in `PerformanceSidebar.tsx`
|
||||
|
||||
The preset values need to be re-tuned so the mapping functions produce meaningfully different
|
||||
WebGL configurations:
|
||||
|
||||
| Preset | `background_persistence` | `splat_force` | `drag` | `decay` | `blob_life` | `spawn_density` | `beat_burst_count` | `treble_pulse` | `color_speed` |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| Default | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 |
|
||||
| Ink | 3.0 | 0.5 | 2.5 | 0.3 | 2.5 | 0.5 | 1.5 | 0.3 | 0.2 |
|
||||
| Foam | 0.2 | 0.3 | 0.4 | 3.0 | 0.3 | 3.0 | 3.0 | 2.5 | 3.0 |
|
||||
| Lava | 3.5 | 2.0 | 3.0 | 0.2 | 3.0 | 0.3 | 0.5 | 0.2 | 0.2 |
|
||||
| Mist | 0.5 | 0.2 | 0.5 | 2.0 | 0.5 | 1.5 | 0.5 | 1.5 | 0.8 |
|
||||
| Storm | 1.0 | 3.0 | 0.4 | 1.0 | 1.0 | 2.5 | 3.0 | 2.5 | 2.5 |
|
||||
|
||||
**Resulting WebGL parameters per preset:**
|
||||
|
||||
| Preset | `DENSITY_DISSIPATION` | `VELOCITY_DISSIPATION` | `CURL` | `SPLAT_RADIUS` | Visual character |
|
||||
|---|---|---|---|---|---|
|
||||
| Default | ~0.955 | ~0.96 | ~27 | ~0.24 | Balanced color motion |
|
||||
| Ink | ~0.905 | ~0.916 | ~21 | ~0.33 | Thick slow ribbons, long trails |
|
||||
| Foam | ~0.975 | ~0.99 | ~18.6 | ~0.18 | Fast-fading tiny bubbles, frantic beat |
|
||||
| Lava | ~0.893 | ~0.94 | ~39 | ~0.36 | Slow massive blobs, high vortex |
|
||||
| Mist | ~0.968 | ~0.98 | ~20.6 | ~0.21 | Translucent wisps, gentle movement |
|
||||
| Storm | ~0.955 | ~0.99 | ~51 | ~0.24 | Violent turbulence on every beat |
|
||||
|
||||
#### Step 5 — Update `VisualizerSettings` interface
|
||||
|
||||
The canvas-blob-specific fields can be retained (they now map to WebGL params via the helper
|
||||
functions) or replaced with more semantically accurate WebGL param names. Retain the existing
|
||||
field names to avoid breaking SEK manifest compatibility. No schema change needed.
|
||||
|
||||
Remove the Note from `VisualizerBackdrop.tsx` that says WebGL params "are not applicable to
|
||||
the canvas blob renderer" — they now are applicable.
|
||||
|
||||
---
|
||||
|
||||
## Issue E — note on Issues C and D supersession
|
||||
|
||||
Issues C (`spawnBlob` dimensions) and D (`background_persistence` formula) exist only in the
|
||||
canvas blob renderer. After Issue E is implemented, `FluidVisualizerCanvas.tsx` is fully
|
||||
rewritten. Issues C and D are automatically resolved by the replacement and do not need to be
|
||||
fixed in the canvas blob code.
|
||||
|
||||
---
|
||||
|
||||
## Edge cases and additional fixes
|
||||
|
||||
### A1 — `Karaoke.tsx` has the same analysis audio gap
|
||||
|
||||
The plan's Issue A fix scopes to `KaraokeTV.tsx`. `Karaoke.tsx` (the operator-facing page at
|
||||
`/karaoke`) uses **identical logic** (lines 463–464):
|
||||
|
||||
```tsx
|
||||
// Karaoke.tsx — exact same conditional as KaraokeTV.tsx
|
||||
stems={safeStems.length > 0 ? safeStems : undefined}
|
||||
audioUrl={safeStems.length === 0 && sourceMedia ? sourceMedia.url : undefined}
|
||||
```
|
||||
|
||||
The `analysisAudioUrl` memo and prop must be added to `Karaoke.tsx` as well. The derivation
|
||||
logic is identical to `KaraokeTV.tsx` (source file for project, first stem for SEK).
|
||||
Add `Karaoke.tsx` to the files-changed table.
|
||||
|
||||
---
|
||||
|
||||
### E1 — WebGL context failure: need fallback
|
||||
|
||||
`canvas.getContext('webgl2') ?? canvas.getContext('webgl')` returns `null` on devices that
|
||||
don't support WebGL (old Android WebViews, some iframe sandboxes). The vendored sim will crash
|
||||
silently or throw.
|
||||
|
||||
**Fix:** Detect WebGL support before calling `initFluidSim`. On failure, fall back to the
|
||||
canvas blob renderer (keep the old blob logic in a separate file or inline) or show a static
|
||||
dark background:
|
||||
|
||||
```typescript
|
||||
useEffect(() => {
|
||||
const canvas = canvasRef.current
|
||||
if (!canvas) return
|
||||
const supportsWebGL = !!(
|
||||
canvas.getContext('webgl2') ?? canvas.getContext('webgl')
|
||||
)
|
||||
if (!supportsWebGL) {
|
||||
// Degrade gracefully: render static palette background
|
||||
const ctx2d = canvas.getContext('2d')
|
||||
if (ctx2d) {
|
||||
ctx2d.fillStyle = paletteRef.current.backgroundColor
|
||||
ctx2d.fillRect(0, 0, canvas.width, canvas.height)
|
||||
}
|
||||
return
|
||||
}
|
||||
initFluidSim(canvas, { ... })
|
||||
...
|
||||
}, [featuresRef, paletteRef])
|
||||
```
|
||||
|
||||
The context test creates a throwaway context; `initFluidSim` will create the real one. WebGL
|
||||
contexts are still a limited browser resource, so production code should avoid repeated
|
||||
probe/create cycles and should always release contexts on cleanup (`WEBGL_lose_context`).
|
||||
|
||||
---
|
||||
|
||||
### E2 — WebGL context not freed on cleanup (context limit)
|
||||
|
||||
The cleanup function currently only cancels the outer RAF:
|
||||
|
||||
```typescript
|
||||
return () => cancelAnimationFrame(raf)
|
||||
```
|
||||
|
||||
The WebGL context persists until garbage collection. Browsers enforce a hard per-page limit
|
||||
(typically 8–16 contexts). Navigating between songs while the fluid visualizer is mounted will
|
||||
exhaust this limit, causing subsequent `getContext('webgl')` calls to return `null` and the sim
|
||||
to silently stop rendering.
|
||||
|
||||
**Fix:** Export a `destroyFluidSim()` from `fluidSim.js` that:
|
||||
1. Cancels the sim's internal RAF loop
|
||||
2. Calls `gl.getExtension('WEBGL_lose_context')?.loseContext()`
|
||||
3. Nulls internal state
|
||||
|
||||
Call it in cleanup:
|
||||
|
||||
```typescript
|
||||
return () => {
|
||||
cancelAnimationFrame(raf)
|
||||
destroyFluidSim()
|
||||
}
|
||||
```
|
||||
|
||||
This must be added during the vendoring step (Step 1) along with the other exported functions.
|
||||
|
||||
---
|
||||
|
||||
### E3 — `fluidSim.js` is not an ES module; TypeScript has no types
|
||||
|
||||
`PavelDoGreat/WebGL-Fluid-Simulation/script.js` is not an ES module — it has no `export`
|
||||
statements and is structured as top-level imperative code. Vite cannot `import` from it
|
||||
directly; it will throw `SyntaxError: The requested module does not provide an export`.
|
||||
|
||||
**Fix during vendoring:** Wrap the sim in a factory function and export via ES module syntax:
|
||||
|
||||
```js
|
||||
// fluidSim.js — top of file
|
||||
let _splat, _config, _internalRaf
|
||||
|
||||
// ... (existing sim code, modified so `canvas`, `gl`, `config`, `splat` are
|
||||
// assigned to module-level variables instead of implicit globals)
|
||||
|
||||
export function initFluidSim(canvas, overrides) {
|
||||
// call existing canvas/gl initialisation with the provided canvas
|
||||
// merge overrides into config
|
||||
}
|
||||
export function splatFluid(x, y, dx, dy, color) { _splat(x, y, dx, dy, color) }
|
||||
export function updateFluidConfig(patch) { Object.assign(_config, patch) }
|
||||
export function destroyFluidSim() {
|
||||
cancelAnimationFrame(_internalRaf)
|
||||
const ext = _gl?.getExtension('WEBGL_lose_context')
|
||||
ext?.loseContext()
|
||||
_gl = null
|
||||
}
|
||||
```
|
||||
|
||||
Add `frontend/src/components/visualizers/fluid/fluidSim.d.ts`:
|
||||
|
||||
```typescript
|
||||
export interface FluidColor { r: number; g: number; b: number }
|
||||
export interface FluidConfig {
|
||||
SIM_RESOLUTION?: number; DYE_RESOLUTION?: number
|
||||
DENSITY_DISSIPATION?: number; VELOCITY_DISSIPATION?: number
|
||||
PRESSURE?: number; CURL?: number; SPLAT_RADIUS?: number
|
||||
COLORFUL?: boolean; BLOOM?: boolean; BLOOM_INTENSITY?: number
|
||||
SUNRAYS?: boolean; BACK_COLOR?: FluidColor
|
||||
}
|
||||
export function initFluidSim(canvas: HTMLCanvasElement, config?: FluidConfig): void
|
||||
export function splatFluid(x: number, y: number, dx: number, dy: number, color: FluidColor): void
|
||||
export function updateFluidConfig(patch: Partial<FluidConfig>): void
|
||||
export function destroyFluidSim(): void
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### E4 — `clamp` helper not defined in config-mapping helpers
|
||||
|
||||
The config helpers (`fluidDensityDissipation`, `fluidVelocityDissipation`, `fluidCurl`,
|
||||
`fluidSplatRadius`) all call `clamp(value, min, max)` but this function is not defined in the
|
||||
plan's code or in the existing canvas files (which use `clamp01`).
|
||||
|
||||
**Fix:** Define at module level in `FluidVisualizerCanvas.tsx`:
|
||||
|
||||
```typescript
|
||||
function clamp(v: number, lo: number, hi: number): number {
|
||||
return Math.max(lo, Math.min(hi, v))
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### E5 — `fpsCap` captured in closure; settings not live-updated
|
||||
|
||||
The plan's `FluidVisualizerCanvas` reads `fpsCap ?? 60` directly from the closure:
|
||||
|
||||
```typescript
|
||||
const minFrameMs = 1000 / Math.max(1, fpsCap ?? 60)
|
||||
```
|
||||
|
||||
Since `fpsCap` is a prop and not in the `useEffect` dep array, changes to it after mount are
|
||||
ignored (the effect doesn't re-run). This is inconsistent with how both `MuonVisualizerCanvas`
|
||||
and the old `FluidVisualizerCanvas` handle it.
|
||||
|
||||
**Fix:** Use the same stable ref pattern as the Muon canvas:
|
||||
|
||||
```typescript
|
||||
const fpsCapRef = useRef(fpsCap ?? 60)
|
||||
fpsCapRef.current = fpsCap ?? 60 // updated at render level, before effect reads it
|
||||
|
||||
// Inside the RAF loop:
|
||||
const minFrameMs = 1000 / Math.max(1, fpsCapRef.current)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### E6 — `BACK_COLOR` hardcoded to black; doesn't track palette
|
||||
|
||||
The plan initialises the sim with `BACK_COLOR: { r: 0, g: 0, b: 0 }` (black), but
|
||||
`paletteRef.current.backgroundColor` may be a dark blue or other palette-derived color. The
|
||||
fluid sim's background won't match the surrounding UI.
|
||||
|
||||
Additionally, `paletteRef.current` can change after mount (cover art loads asynchronously). The
|
||||
`BACK_COLOR` won't update until the next config call.
|
||||
|
||||
**Fix:** Add `BACK_COLOR` to the per-frame `updateFluidConfig` call, reading from `paletteRef`:
|
||||
|
||||
```typescript
|
||||
updateFluidConfig({
|
||||
DENSITY_DISSIPATION: fluidDensityDissipation(s, kick),
|
||||
VELOCITY_DISSIPATION: fluidVelocityDissipation(s, rms),
|
||||
CURL: fluidCurl(s, treble, dropBoost),
|
||||
BACK_COLOR: hexToRgb(paletteRef.current.backgroundColor),
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### E7 — `treble_pulse` not used in WebGL path
|
||||
|
||||
The canvas blob renderer applied `treble_pulse` as a radius-swell multiplier. The WebGL
|
||||
wrapper's per-frame loop doesn't read `settingsRef.current?.treble_pulse` anywhere, so the
|
||||
slider has no effect in Fluid mode.
|
||||
|
||||
**Fix:** Map `treble_pulse` to ambient splat force intensity. Higher `treble_pulse` makes high
|
||||
frequencies inject more force into the fluid:
|
||||
|
||||
```typescript
|
||||
const treblePulse = Math.max(0, s?.treble_pulse ?? 1)
|
||||
|
||||
// In ambient splat:
|
||||
const force = (0.2 + rms * 0.8 + treble * 0.3 * treblePulse + idle * 0.12) * splatForce * 400
|
||||
```
|
||||
|
||||
`treble_pulse = 0` suppresses treble-driven force. `treble_pulse > 1` amplifies it. This is
|
||||
semantically consistent with the original canvas intent ("treble inflates blob radius").
|
||||
|
||||
---
|
||||
|
||||
### E8 — Two RAF loops; `fpsCap` only throttles splat injection
|
||||
|
||||
The WebGL fluid sim runs its **own internal RAF loop** for pressure solving, advection, and
|
||||
rendering. The component's outer RAF loop only injects splats based on audio features.
|
||||
|
||||
This means:
|
||||
- The sim always renders at the browser's native frame rate (typically 60 Hz)
|
||||
- `fpsCap` only throttles the frequency of splat injection, not visual rendering
|
||||
- At `fpsCap = 30`, splats are injected at 30 Hz but the fluid animates at 60 Hz
|
||||
|
||||
This is acceptable behavior (cheap splat injection, smooth visual rendering) and is consistent
|
||||
with how the original PavelDoGreat sim is designed to be extended. However, it should be
|
||||
documented in code comments to avoid future confusion.
|
||||
|
||||
---
|
||||
|
||||
### E9 — Verify `COLORFUL: false` respects per-splat color parameter
|
||||
|
||||
PavelDoGreat's sim supports two color modes:
|
||||
- `COLORFUL: true` — sim auto-generates HSL rainbow colors per splat, ignores the `color` arg
|
||||
- `COLORFUL: false` — sim uses the `color` `{ r, g, b }` argument passed to `splat()`
|
||||
|
||||
If `COLORFUL: false` actually ignores the color arg (a known inconsistency in some sim forks),
|
||||
the palette cycling via `nextColor()` will be silently broken and all fluid will render as
|
||||
white.
|
||||
|
||||
**Mitigation during vendoring:** Inspect the `multipleSplats` / `splat` function in the
|
||||
vendored `script.js`. Confirm the `color` parameter is applied to the dye texture. If not,
|
||||
patch it:
|
||||
|
||||
```js
|
||||
// In the vendored splat() function, ensure:
|
||||
gl.uniform3f(splatProgram.uniforms.color, color.r, color.g, color.b)
|
||||
// Not:
|
||||
gl.uniform3f(splatProgram.uniforms.color, Math.random(), Math.random(), Math.random())
|
||||
```
|
||||
|
||||
Add this check as a required verification step before shipping.
|
||||
|
||||
---
|
||||
|
||||
## Files changed
|
||||
|
||||
| File | Issue | Change |
|
||||
|---|---|---|
|
||||
| `frontend/src/components/LyricsLivePreview.tsx` | A | Add `analysisAudioUrl?: string` prop; use it as the hidden audio `src` with `audioUrl` as fallback; update `analysisSourceKey` |
|
||||
| `frontend/src/components/KaraokePerformanceMode.tsx` | A | Add `analysisAudioUrl?: string` prop; forward to `LyricsLivePreview` |
|
||||
| `frontend/src/pages/KaraokeTV.tsx` | A | Derive `analysisAudioUrl` memo (source file for project, first stem for SEK); pass to `KaraokePerformanceMode` |
|
||||
| `frontend/src/pages/Karaoke.tsx` | A1 | Apply the same `analysisAudioUrl` derivation + pass-through as KaraokeTV |
|
||||
| `frontend/src/components/visualizers/MuonVisualizerCanvas.tsx` | B | Per-lane hue offset: `laneHue = (hue + pt.lane * hueRange * 45) % 360` |
|
||||
| `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx` | C, D | Pass live `w/h` into `spawnBlob`; invert `background_persistence` alpha mapping |
|
||||
| `frontend/src/components/visualizers/fluid/fluidSim.ts` | E | **New** — wrapper around `webgl-fluid` package exposing `initFluidSim`, `splatFluid`, `updateFluidConfig`, `destroyFluidSim` |
|
||||
| `frontend/src/types/webgl-fluid.d.ts` | E3 | **New** — ambient module declaration for `webgl-fluid` |
|
||||
| `frontend/package.json` + lockfile | E | Add `webgl-fluid` dependency |
|
||||
| `frontend/src/components/visualizers/FluidVisualizerCanvas.tsx` | E | **Full rewrite** — WebGL wrapper with config-mapping helpers; removes canvas blob renderer |
|
||||
| `frontend/src/components/PerformanceSidebar.tsx` | E | Update `FLUID_PRESETS` values to match new WebGL parameter mapping |
|
||||
| `SUMMARIES (Claude).md` | — | Append entry after implementation |
|
||||
|
||||
Recommended delivery split:
|
||||
|
||||
- **Phase 1 (safe/immediate):** A + A1 + B + C + D
|
||||
- **Phase 2 (high-risk migration):** E + E1..E9
|
||||
|
||||
If Phase 2 is deferred, C/D remain required fixes in the existing canvas renderer.
|
||||
|
||||
---
|
||||
|
||||
## Implementation order
|
||||
|
||||
### Phase 1 (ship now)
|
||||
|
||||
1. Add `analysisAudioUrl` prop to `LyricsLivePreview` — update hidden analysis element + `analysisSourceKey`
|
||||
2. Add `analysisAudioUrl` prop to `KaraokePerformanceMode` — forward-only change
|
||||
3. Derive and pass `analysisAudioUrl` in both `KaraokeTVPage` and `KaraokePage`
|
||||
4. Fix Muon per-lane hue offset in `MuonVisualizerCanvas.tsx`
|
||||
5. Fix Fluid canvas issues C/D (`spawnBlob` live dimensions + `background_persistence` inversion)
|
||||
6. Run `tsc --noEmit`
|
||||
|
||||
### Phase 2 (optional migration)
|
||||
|
||||
7. Add `webgl-fluid` dependency and implement `fluidSim.ts` wrapper APIs (`init`/`splat`/`update`/`destroy`)
|
||||
8. Rewrite `FluidVisualizerCanvas.tsx` around WebGL wrapper with graceful fallback
|
||||
9. Update `FLUID_PRESETS` in `PerformanceSidebar.tsx`
|
||||
10. Run `tsc --noEmit`
|
||||
11. Update `SUMMARIES (Claude).md`
|
||||
Generated
+44
-250
@@ -21,13 +21,13 @@
|
||||
"@tonejs/midi": "^2.0.28",
|
||||
"abcjs": "^6.6.2",
|
||||
"lucide-react": "^0.294.0",
|
||||
"meyda": "^5.6.3",
|
||||
"qrcode": "^1.5.4",
|
||||
"react": "^18.2.0",
|
||||
"react-dom": "^18.2.0",
|
||||
"react-router-dom": "^6.30.3",
|
||||
"tone": "^14.7.0",
|
||||
"two.js": "^0.8.23"
|
||||
"two.js": "^0.8.23",
|
||||
"webgl-fluid": "^0.3.9"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@playwright/test": "^1.53.1",
|
||||
@@ -1064,64 +1064,6 @@
|
||||
"integrity": "sha512-+d0F4MKMCbeVUJwG96uQ4SgAznZNSq93I3V+9NHA4OpvqG8mRCpGdKmK8l/dl02h2CCDHwW2FqilnTyDcAnqjA==",
|
||||
"dev": true
|
||||
},
|
||||
"node_modules/@rollup/plugin-node-resolve": {
|
||||
"version": "15.3.1",
|
||||
"resolved": "https://registry.npmjs.org/@rollup/plugin-node-resolve/-/plugin-node-resolve-15.3.1.tgz",
|
||||
"integrity": "sha512-tgg6b91pAybXHJQMAAwW9VuWBO6Thi+q7BCNARLwSqlmsHz0XYURtGvh/AuwSADXSI4h/2uHbs7s4FzlZDGSGA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@rollup/pluginutils": "^5.0.1",
|
||||
"@types/resolve": "1.20.2",
|
||||
"deepmerge": "^4.2.2",
|
||||
"is-module": "^1.0.0",
|
||||
"resolve": "^1.22.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=14.0.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"rollup": "^2.78.0||^3.0.0||^4.0.0"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"rollup": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/@rollup/pluginutils": {
|
||||
"version": "5.3.0",
|
||||
"resolved": "https://registry.npmjs.org/@rollup/pluginutils/-/pluginutils-5.3.0.tgz",
|
||||
"integrity": "sha512-5EdhGZtnu3V88ces7s53hhfK5KSASnJZv8Lulpc04cWO3REESroJXg73DFsOmgbU2BhwV0E20bu2IDZb3VKW4Q==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@types/estree": "^1.0.0",
|
||||
"estree-walker": "^2.0.2",
|
||||
"picomatch": "^4.0.2"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=14.0.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"rollup": "^1.20.0||^2.0.0||^3.0.0||^4.0.0"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"rollup": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/@rollup/pluginutils/node_modules/picomatch": {
|
||||
"version": "4.0.4",
|
||||
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.4.tgz",
|
||||
"integrity": "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/jonschlinkert"
|
||||
}
|
||||
},
|
||||
"node_modules/@rollup/rollup-android-arm-eabi": {
|
||||
"version": "4.59.0",
|
||||
"resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm-eabi/-/rollup-android-arm-eabi-4.59.0.tgz",
|
||||
@@ -1129,6 +1071,7 @@
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"android"
|
||||
@@ -1141,6 +1084,7 @@
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"android"
|
||||
@@ -1153,6 +1097,7 @@
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
@@ -1165,6 +1110,7 @@
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
@@ -1177,6 +1123,7 @@
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"freebsd"
|
||||
@@ -1189,6 +1136,7 @@
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"freebsd"
|
||||
@@ -1201,6 +1149,7 @@
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
@@ -1213,6 +1162,7 @@
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
@@ -1225,6 +1175,7 @@
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
@@ -1237,6 +1188,7 @@
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
@@ -1249,6 +1201,7 @@
|
||||
"cpu": [
|
||||
"loong64"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
@@ -1261,6 +1214,7 @@
|
||||
"cpu": [
|
||||
"loong64"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
@@ -1273,6 +1227,7 @@
|
||||
"cpu": [
|
||||
"ppc64"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
@@ -1285,6 +1240,7 @@
|
||||
"cpu": [
|
||||
"ppc64"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
@@ -1297,6 +1253,7 @@
|
||||
"cpu": [
|
||||
"riscv64"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
@@ -1309,6 +1266,7 @@
|
||||
"cpu": [
|
||||
"riscv64"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
@@ -1321,6 +1279,7 @@
|
||||
"cpu": [
|
||||
"s390x"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
@@ -1333,6 +1292,7 @@
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
@@ -1345,6 +1305,7 @@
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
@@ -1357,6 +1318,7 @@
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"openbsd"
|
||||
@@ -1369,6 +1331,7 @@
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"openharmony"
|
||||
@@ -1381,6 +1344,7 @@
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
@@ -1393,6 +1357,7 @@
|
||||
"cpu": [
|
||||
"ia32"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
@@ -1405,6 +1370,7 @@
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
@@ -1417,6 +1383,7 @@
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
@@ -1499,7 +1466,8 @@
|
||||
"node_modules/@types/estree": {
|
||||
"version": "1.0.8",
|
||||
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.8.tgz",
|
||||
"integrity": "sha512-dWHzHa2WqEXI/O1E9OjrocMTKJl2mSrEolh1Iomrv6U+JuNwaHXsXx9bLu5gG7BUWFIN0skIQJQ/L1rIex4X6w=="
|
||||
"integrity": "sha512-dWHzHa2WqEXI/O1E9OjrocMTKJl2mSrEolh1Iomrv6U+JuNwaHXsXx9bLu5gG7BUWFIN0skIQJQ/L1rIex4X6w==",
|
||||
"dev": true
|
||||
},
|
||||
"node_modules/@types/node": {
|
||||
"version": "25.5.2",
|
||||
@@ -1546,12 +1514,6 @@
|
||||
"@types/react": "^18.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@types/resolve": {
|
||||
"version": "1.20.2",
|
||||
"resolved": "https://registry.npmjs.org/@types/resolve/-/resolve-1.20.2.tgz",
|
||||
"integrity": "sha512-60BCwRFOZCQhDncwQdxxeOEEkbc5dIMccYLwbxsS4TUNeVECQ/pBJ0j09mrHOl/JJvpRPGwO9SvE4nR2Nb/a4Q==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@typescript-eslint/eslint-plugin": {
|
||||
"version": "8.57.0",
|
||||
"resolved": "https://registry.npmjs.org/@typescript-eslint/eslint-plugin/-/eslint-plugin-8.57.0.tgz",
|
||||
@@ -1985,12 +1947,6 @@
|
||||
"postcss": "^8.1.0"
|
||||
}
|
||||
},
|
||||
"node_modules/babel-plugin-add-module-exports": {
|
||||
"version": "0.2.1",
|
||||
"resolved": "https://registry.npmjs.org/babel-plugin-add-module-exports/-/babel-plugin-add-module-exports-0.2.1.tgz",
|
||||
"integrity": "sha512-3AN/9V/rKuv90NG65m4tTHsI04XrCKsWbztIcW7a8H5iIN7WlvWucRtVV0V/rT4QvtA11n5Vmp20fLwfMWqp6g==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/balanced-match": {
|
||||
"version": "4.0.4",
|
||||
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-4.0.4.tgz",
|
||||
@@ -2083,34 +2039,6 @@
|
||||
"node": "^6 || ^7 || ^8 || ^9 || ^10 || ^11 || ^12 || >=13.7"
|
||||
}
|
||||
},
|
||||
"node_modules/buffer-alloc": {
|
||||
"version": "1.2.0",
|
||||
"resolved": "https://registry.npmjs.org/buffer-alloc/-/buffer-alloc-1.2.0.tgz",
|
||||
"integrity": "sha512-CFsHQgjtW1UChdXgbyJGtnm+O/uLQeZdtbDo8mfUgYXCHSM1wgrVxXm6bSyrUuErEb+4sYVGCzASBRot7zyrow==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"buffer-alloc-unsafe": "^1.1.0",
|
||||
"buffer-fill": "^1.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/buffer-alloc-unsafe": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/buffer-alloc-unsafe/-/buffer-alloc-unsafe-1.1.0.tgz",
|
||||
"integrity": "sha512-TEM2iMIEQdJ2yjPJoSIsldnleVaAk1oW3DBVUykyOLsEsFmEc9kn+SFFPz+gl54KQNxlDnAwCXosOS9Okx2xAg==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/buffer-fill": {
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/buffer-fill/-/buffer-fill-1.0.0.tgz",
|
||||
"integrity": "sha512-T7zexNBwiiaCOGDg9xNX9PBmjrubblRkENuptryuI64URkXDFum9il/JGL8Lm8wYfAXpredVXXZz7eMHilimiQ==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/buffer-from": {
|
||||
"version": "1.1.2",
|
||||
"resolved": "https://registry.npmjs.org/buffer-from/-/buffer-from-1.1.2.tgz",
|
||||
"integrity": "sha512-E+XQCRwSbaaiChtv6k6Dwgc+bx+Bs6vuKJHHl5kox/BaKbhiXzqQOwK4cO22yElGp2OCmjwVhT3HmxgyPGnJfQ==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/callsites": {
|
||||
"version": "3.1.0",
|
||||
"resolved": "https://registry.npmjs.org/callsites/-/callsites-3.1.0.tgz",
|
||||
@@ -2263,12 +2191,6 @@
|
||||
"integrity": "sha512-Kvp459HrV2FEJ1CAsi1Ku+MY3kasH19TFykTz2xWmMeq6bk2NU3XXvfJ+Q61m0xktWwt+1HSYf3JZsTms3aRJg==",
|
||||
"dev": true
|
||||
},
|
||||
"node_modules/core-util-is": {
|
||||
"version": "1.0.3",
|
||||
"resolved": "https://registry.npmjs.org/core-util-is/-/core-util-is-1.0.3.tgz",
|
||||
"integrity": "sha512-ZQBvi1DcpJ4GDqanjucZ2Hj3wEO5pZDS89BWbkcrvdxksJorwUDDZamX9ldFkp9aw2lmBDLgkObEA4DWNJ9FYQ==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/cross-spawn": {
|
||||
"version": "7.0.6",
|
||||
"resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz",
|
||||
@@ -2302,15 +2224,6 @@
|
||||
"integrity": "sha512-z1HGKcYy2xA8AGQfwrn0PAy+PB7X/GSj3UVJW9qKyn43xWa+gl5nXmU4qqLMRzWVLFC8KusUX8T/0kCiOYpAIQ==",
|
||||
"dev": true
|
||||
},
|
||||
"node_modules/dct": {
|
||||
"version": "0.1.0",
|
||||
"resolved": "https://registry.npmjs.org/dct/-/dct-0.1.0.tgz",
|
||||
"integrity": "sha512-/uUtEniuMq1aUxvLAoDtAduyl12oM1zhA/le2f83UFN/9+4KDHXFB6znEfoj5SDDLiTpUTr26NpxC7t8IFOYhQ==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=0.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/debug": {
|
||||
"version": "4.4.3",
|
||||
"resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz",
|
||||
@@ -2344,15 +2257,6 @@
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/deepmerge": {
|
||||
"version": "4.3.1",
|
||||
"resolved": "https://registry.npmjs.org/deepmerge/-/deepmerge-4.3.1.tgz",
|
||||
"integrity": "sha512-3sUqbMEc77XqpdNO7FRyRog+eW3ph+GYCbj+rK+uYyRMuwsVy0rMiVtPn+QJlKFvWP/1PYpapqYn0Me2knFn+A==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/didyoumean": {
|
||||
"version": "1.2.2",
|
||||
"resolved": "https://registry.npmjs.org/didyoumean/-/didyoumean-1.2.2.tgz",
|
||||
@@ -2661,12 +2565,6 @@
|
||||
"node": ">=4.0"
|
||||
}
|
||||
},
|
||||
"node_modules/estree-walker": {
|
||||
"version": "2.0.2",
|
||||
"resolved": "https://registry.npmjs.org/estree-walker/-/estree-walker-2.0.2.tgz",
|
||||
"integrity": "sha512-Rfkk/Mp/DL7JVje3u18FxFujQlTNR2q6QfMSMB7AvCBx91NGj/ba3kCfza0f6dVDbw7YlRf/nDrn7pQrCCyQ/w==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/esutils": {
|
||||
"version": "2.0.3",
|
||||
"resolved": "https://registry.npmjs.org/esutils/-/esutils-2.0.3.tgz",
|
||||
@@ -2735,15 +2633,6 @@
|
||||
"reusify": "^1.0.4"
|
||||
}
|
||||
},
|
||||
"node_modules/fftjs": {
|
||||
"version": "0.0.4",
|
||||
"resolved": "https://registry.npmjs.org/fftjs/-/fftjs-0.0.4.tgz",
|
||||
"integrity": "sha512-nIWxQyth1LVD6NH8a+YZUv+McjzbOY6dMe4wv6Pq5cGfP+c8Rd1T8Dsd50DCWlNgzSqA3y9lOkpD6dZD3qHa1A==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"babel-plugin-add-module-exports": "^0.2.1"
|
||||
}
|
||||
},
|
||||
"node_modules/file-entry-cache": {
|
||||
"version": "6.0.1",
|
||||
"resolved": "https://registry.npmjs.org/file-entry-cache/-/file-entry-cache-6.0.1.tgz",
|
||||
@@ -2832,6 +2721,7 @@
|
||||
"version": "2.3.3",
|
||||
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz",
|
||||
"integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==",
|
||||
"dev": true,
|
||||
"hasInstallScript": true,
|
||||
"optional": true,
|
||||
"os": [
|
||||
@@ -2845,6 +2735,7 @@
|
||||
"version": "1.1.2",
|
||||
"resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz",
|
||||
"integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==",
|
||||
"dev": true,
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/ljharb"
|
||||
}
|
||||
@@ -2969,6 +2860,7 @@
|
||||
"version": "2.0.2",
|
||||
"resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.2.tgz",
|
||||
"integrity": "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==",
|
||||
"dev": true,
|
||||
"dependencies": {
|
||||
"function-bind": "^1.1.2"
|
||||
},
|
||||
@@ -3029,6 +2921,7 @@
|
||||
"version": "2.0.4",
|
||||
"resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
|
||||
"integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==",
|
||||
"dev": true,
|
||||
"license": "ISC"
|
||||
},
|
||||
"node_modules/is-binary-path": {
|
||||
@@ -3047,6 +2940,7 @@
|
||||
"version": "2.16.1",
|
||||
"resolved": "https://registry.npmjs.org/is-core-module/-/is-core-module-2.16.1.tgz",
|
||||
"integrity": "sha512-UfoeMA6fIJ8wTYFEUjelnaGI67v6+N7qXJEvQuIGa99l4xsCruSYOVSQ0uPANn4dAzm8lkYPaKLrrijLq7x23w==",
|
||||
"dev": true,
|
||||
"dependencies": {
|
||||
"hasown": "^2.0.2"
|
||||
},
|
||||
@@ -3087,12 +2981,6 @@
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/is-module": {
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/is-module/-/is-module-1.0.0.tgz",
|
||||
"integrity": "sha512-51ypPSPCoTEIN9dy5Oy+h4pShgJmPCygKfyRCISBI+JoWT/2oJvK8QPxmwv7b/p239jXrm9M1mlQbyKJ5A152g==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/is-number": {
|
||||
"version": "7.0.0",
|
||||
"resolved": "https://registry.npmjs.org/is-number/-/is-number-7.0.0.tgz",
|
||||
@@ -3112,12 +3000,6 @@
|
||||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/isarray": {
|
||||
"version": "0.0.1",
|
||||
"resolved": "https://registry.npmjs.org/isarray/-/isarray-0.0.1.tgz",
|
||||
"integrity": "sha512-D2S+3GLxWH+uhrNEcoh/fnmYeP8E8/zHl644d/jdA0g2uyXvy3sb0qxotE+ne0LtccHknQzWwZEzhak7oJ0COQ==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/isexe": {
|
||||
"version": "2.0.0",
|
||||
"resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz",
|
||||
@@ -3299,25 +3181,6 @@
|
||||
"node": ">= 8"
|
||||
}
|
||||
},
|
||||
"node_modules/meyda": {
|
||||
"version": "5.6.3",
|
||||
"resolved": "https://registry.npmjs.org/meyda/-/meyda-5.6.3.tgz",
|
||||
"integrity": "sha512-fAdwfzIi1WDoL0idUQvCD7dZ7EN74FYH83G+jZQO3Nr9yOEBtzFvcMg2KLdLlu6psSP8XFlO0kYynG5o/E681Q==",
|
||||
"license": "MIT",
|
||||
"workspaces": [
|
||||
"docs"
|
||||
],
|
||||
"dependencies": {
|
||||
"@rollup/plugin-node-resolve": "^15.2.3",
|
||||
"dct": "0.1.0",
|
||||
"fftjs": "0.0.4",
|
||||
"node-getopt": "^0.3.2",
|
||||
"wav": "^1.0.2"
|
||||
},
|
||||
"bin": {
|
||||
"meyda": "bin/cli.js"
|
||||
}
|
||||
},
|
||||
"node_modules/micromatch": {
|
||||
"version": "4.0.8",
|
||||
"resolved": "https://registry.npmjs.org/micromatch/-/micromatch-4.0.8.tgz",
|
||||
@@ -3394,15 +3257,6 @@
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/node-getopt": {
|
||||
"version": "0.3.2",
|
||||
"resolved": "https://registry.npmjs.org/node-getopt/-/node-getopt-0.3.2.tgz",
|
||||
"integrity": "sha512-yqkmYrMbK1wPrfz7mgeYvA4tBperLg9FQ4S3Sau3nSAkpOA0x0zC8nQ1siBwozy1f4SE8vq2n1WKv99r+PCa1Q==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.6.0"
|
||||
}
|
||||
},
|
||||
"node_modules/node-releases": {
|
||||
"version": "2.0.27",
|
||||
"resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.27.tgz",
|
||||
@@ -3550,7 +3404,8 @@
|
||||
"node_modules/path-parse": {
|
||||
"version": "1.0.7",
|
||||
"resolved": "https://registry.npmjs.org/path-parse/-/path-parse-1.0.7.tgz",
|
||||
"integrity": "sha512-LDJzPVEEEPR+y48z93A0Ed0yXb8pAByGWo/k5YYdYgpY2/2EsOsksJrq7lOHxryrVOn1ejG6oAp8ahvOIQD8sw=="
|
||||
"integrity": "sha512-LDJzPVEEEPR+y48z93A0Ed0yXb8pAByGWo/k5YYdYgpY2/2EsOsksJrq7lOHxryrVOn1ejG6oAp8ahvOIQD8sw==",
|
||||
"dev": true
|
||||
},
|
||||
"node_modules/picocolors": {
|
||||
"version": "1.1.1",
|
||||
@@ -3930,18 +3785,6 @@
|
||||
"pify": "^2.3.0"
|
||||
}
|
||||
},
|
||||
"node_modules/readable-stream": {
|
||||
"version": "1.1.14",
|
||||
"resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-1.1.14.tgz",
|
||||
"integrity": "sha512-+MeVjFf4L44XUkhM1eYbD8fyEsxcV81pqMSR5gblfcLCHfZvbrqy4/qYHE+/R5HoBUT11WV5O08Cr1n3YXkWVQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"core-util-is": "~1.0.0",
|
||||
"inherits": "~2.0.1",
|
||||
"isarray": "0.0.1",
|
||||
"string_decoder": "~0.10.x"
|
||||
}
|
||||
},
|
||||
"node_modules/readdirp": {
|
||||
"version": "3.6.0",
|
||||
"resolved": "https://registry.npmjs.org/readdirp/-/readdirp-3.6.0.tgz",
|
||||
@@ -3973,6 +3816,7 @@
|
||||
"version": "1.22.11",
|
||||
"resolved": "https://registry.npmjs.org/resolve/-/resolve-1.22.11.tgz",
|
||||
"integrity": "sha512-RfqAvLnMl313r7c9oclB1HhUEAezcpLjz95wFH4LVuhk9JF/r22qmVP9AMmOU4vMX7Q8pN8jwNg/CSpdFnMjTQ==",
|
||||
"dev": true,
|
||||
"dependencies": {
|
||||
"is-core-module": "^2.16.1",
|
||||
"path-parse": "^1.0.7",
|
||||
@@ -4029,7 +3873,7 @@
|
||||
"version": "4.59.0",
|
||||
"resolved": "https://registry.npmjs.org/rollup/-/rollup-4.59.0.tgz",
|
||||
"integrity": "sha512-2oMpl67a3zCH9H79LeMcbDhXW/UmWG/y2zuqnF2jQq5uq9TbM9TVyXvA4+t+ne2IIkBdrLpAaRQAvo7YI/Yyeg==",
|
||||
"devOptional": true,
|
||||
"dev": true,
|
||||
"dependencies": {
|
||||
"@types/estree": "1.0.8"
|
||||
},
|
||||
@@ -4158,36 +4002,6 @@
|
||||
"tslib": "^2.7.0"
|
||||
}
|
||||
},
|
||||
"node_modules/stream-parser": {
|
||||
"version": "0.3.1",
|
||||
"resolved": "https://registry.npmjs.org/stream-parser/-/stream-parser-0.3.1.tgz",
|
||||
"integrity": "sha512-bJ/HgKq41nlKvlhccD5kaCr/P+Hu0wPNKPJOH7en+YrJu/9EgqUF+88w5Jb6KNcjOFMhfX4B2asfeAtIGuHObQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"debug": "2"
|
||||
}
|
||||
},
|
||||
"node_modules/stream-parser/node_modules/debug": {
|
||||
"version": "2.6.9",
|
||||
"resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz",
|
||||
"integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"ms": "2.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/stream-parser/node_modules/ms": {
|
||||
"version": "2.0.0",
|
||||
"resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
|
||||
"integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/string_decoder": {
|
||||
"version": "0.10.31",
|
||||
"resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-0.10.31.tgz",
|
||||
"integrity": "sha512-ev2QzSzWPYmy9GuqfIVildA4OdcGLeFZQrq5ys6RtiuF+RQQiZWr8TZNyAcuVXyQRYfEO+MsoB/1BuQVhOJuoQ==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/string-width": {
|
||||
"version": "4.2.3",
|
||||
"resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz",
|
||||
@@ -4266,6 +4080,7 @@
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/supports-preserve-symlinks-flag/-/supports-preserve-symlinks-flag-1.0.0.tgz",
|
||||
"integrity": "sha512-ot0WnXS9fgdkgIcePe6RHNk1WA8+muPa6cSjeR3V8K27q9BB1rTE3R1p7Hv0z1ZyAc8s6Vvv8DIyWf681MAt0w==",
|
||||
"dev": true,
|
||||
"engines": {
|
||||
"node": ">= 0.4"
|
||||
},
|
||||
@@ -4587,32 +4402,11 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/wav": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/wav/-/wav-1.0.2.tgz",
|
||||
"integrity": "sha512-viHtz3cDd/Tcr/HbNqzQCofKdF6kWUymH9LGDdskfWFoIy/HJ+RTihgjEcHfnsy1PO4e9B+y4HwgTwMrByquhg==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"buffer-alloc": "^1.1.0",
|
||||
"buffer-from": "^1.0.0",
|
||||
"debug": "^2.2.0",
|
||||
"readable-stream": "^1.1.14",
|
||||
"stream-parser": "^0.3.1"
|
||||
}
|
||||
},
|
||||
"node_modules/wav/node_modules/debug": {
|
||||
"version": "2.6.9",
|
||||
"resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz",
|
||||
"integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"ms": "2.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/wav/node_modules/ms": {
|
||||
"version": "2.0.0",
|
||||
"resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
|
||||
"integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A==",
|
||||
"node_modules/webgl-fluid": {
|
||||
"version": "0.3.9",
|
||||
"resolved": "https://registry.npmjs.org/webgl-fluid/-/webgl-fluid-0.3.9.tgz",
|
||||
"integrity": "sha512-dQpoe6JB3ZdDoOk4v1ZV7haNCAESu/2/XAQ9zK6/qbjssznZFNWx2MLWEPLdCQtWWGwmiZnDN+n461qTZ9+1nw==",
|
||||
"hasInstallScript": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/which": {
|
||||
|
||||
@@ -30,13 +30,13 @@
|
||||
"@tonejs/midi": "^2.0.28",
|
||||
"abcjs": "^6.6.2",
|
||||
"lucide-react": "^0.294.0",
|
||||
"meyda": "^5.6.3",
|
||||
"qrcode": "^1.5.4",
|
||||
"react": "^18.2.0",
|
||||
"react-dom": "^18.2.0",
|
||||
"react-router-dom": "^6.30.3",
|
||||
"tone": "^14.7.0",
|
||||
"two.js": "^0.8.23"
|
||||
"two.js": "^0.8.23",
|
||||
"webgl-fluid": "^0.3.9"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@playwright/test": "^1.53.1",
|
||||
|
||||
@@ -0,0 +1,314 @@
|
||||
export interface MuonFrequencyBands {
|
||||
sub: number
|
||||
bass: number
|
||||
lowMid: number
|
||||
mid: number
|
||||
highMid: number
|
||||
presence: number
|
||||
brilliance: number
|
||||
}
|
||||
|
||||
export interface MuonBeatInfo {
|
||||
isBeat: boolean
|
||||
bpm: number
|
||||
confidence: number
|
||||
timeSinceLastBeat: number
|
||||
beatPhase: number
|
||||
}
|
||||
|
||||
export interface MuonPatternInfo {
|
||||
isBeatDrop: boolean
|
||||
isBuildUp: boolean
|
||||
isSilence: boolean
|
||||
intensity: number
|
||||
energyDelta: number
|
||||
}
|
||||
|
||||
export interface MuonFrame {
|
||||
rms: number
|
||||
energy: number
|
||||
spectralCentroid: number
|
||||
spectralRolloff: number
|
||||
spectralFlatness: number
|
||||
zcr: number
|
||||
frequencyBands: MuonFrequencyBands
|
||||
beatInfo: MuonBeatInfo
|
||||
patternInfo: MuonPatternInfo
|
||||
floatFreqData: Float32Array
|
||||
floatTimeData: Float32Array
|
||||
}
|
||||
|
||||
type BandKey = keyof MuonFrequencyBands
|
||||
|
||||
const AUDIO_CONFIG = {
|
||||
FFT_SIZE: 1024,
|
||||
SMOOTHING: 0.8,
|
||||
MIN_DECIBELS: -100,
|
||||
MAX_DECIBELS: -30,
|
||||
} as const
|
||||
|
||||
const ENERGY_HISTORY_SIZE = 43
|
||||
const BEAT_TIME_HISTORY_SIZE = 24
|
||||
|
||||
const FREQUENCY_BANDS: Record<BandKey, [number, number]> = {
|
||||
sub: [20, 60],
|
||||
bass: [60, 250],
|
||||
lowMid: [250, 500],
|
||||
mid: [500, 2000],
|
||||
highMid: [2000, 4000],
|
||||
presence: [4000, 6000],
|
||||
brilliance: [6000, 20000],
|
||||
}
|
||||
|
||||
function clamp01(value: number): number {
|
||||
return Math.max(0, Math.min(1, value))
|
||||
}
|
||||
|
||||
export class MuonFeatureExtractor {
|
||||
private isRunning = false
|
||||
private animationFrameId: number | null = null
|
||||
private frequencyData!: Float32Array
|
||||
private timeData!: Float32Array
|
||||
private energyHistory: number[] = []
|
||||
private lastBeatTime = 0
|
||||
private beatThreshold = 1.5
|
||||
private bpm = 0
|
||||
private beatTimes: number[] = []
|
||||
private previousEnergy = 0
|
||||
|
||||
start(analyser: AnalyserNode, ctx: AudioContext, onFrame: (f: MuonFrame) => void): void {
|
||||
this.stop()
|
||||
|
||||
analyser.fftSize = AUDIO_CONFIG.FFT_SIZE
|
||||
analyser.smoothingTimeConstant = AUDIO_CONFIG.SMOOTHING
|
||||
analyser.minDecibels = AUDIO_CONFIG.MIN_DECIBELS
|
||||
analyser.maxDecibels = AUDIO_CONFIG.MAX_DECIBELS
|
||||
|
||||
this.frequencyData = new Float32Array(analyser.frequencyBinCount)
|
||||
this.timeData = new Float32Array(analyser.fftSize)
|
||||
this.isRunning = true
|
||||
|
||||
const extractLoop = () => {
|
||||
if (!this.isRunning) return
|
||||
|
||||
analyser.getFloatFrequencyData(this.frequencyData as unknown as Float32Array<ArrayBuffer>)
|
||||
analyser.getFloatTimeDomainData(this.timeData as unknown as Float32Array<ArrayBuffer>)
|
||||
|
||||
const frame = this.extractFeatures(ctx.sampleRate)
|
||||
onFrame(frame)
|
||||
|
||||
this.animationFrameId = window.requestAnimationFrame(extractLoop)
|
||||
}
|
||||
|
||||
this.animationFrameId = window.requestAnimationFrame(extractLoop)
|
||||
}
|
||||
|
||||
stop(): void {
|
||||
this.isRunning = false
|
||||
if (this.animationFrameId !== null) {
|
||||
window.cancelAnimationFrame(this.animationFrameId)
|
||||
this.animationFrameId = null
|
||||
}
|
||||
}
|
||||
|
||||
setBeatThreshold(threshold: number): void {
|
||||
this.beatThreshold = Math.max(0, Math.min(3, threshold))
|
||||
}
|
||||
|
||||
reset(): void {
|
||||
this.energyHistory = []
|
||||
this.lastBeatTime = 0
|
||||
this.bpm = 0
|
||||
this.beatTimes = []
|
||||
this.previousEnergy = 0
|
||||
}
|
||||
|
||||
private extractFeatures(sampleRate: number): MuonFrame {
|
||||
const bins = this.frequencyData.length || 1
|
||||
const nyquist = sampleRate / 2
|
||||
|
||||
const magnitude = new Float32Array(this.frequencyData.length)
|
||||
for (let i = 0; i < this.frequencyData.length; i += 1) {
|
||||
const db = Number.isFinite(this.frequencyData[i]) ? this.frequencyData[i] : AUDIO_CONFIG.MIN_DECIBELS
|
||||
magnitude[i] = Math.pow(10, db / 20)
|
||||
}
|
||||
|
||||
let powerSum = 0
|
||||
let magSum = 0
|
||||
let weightedBinSum = 0
|
||||
let totalEnergy = 0
|
||||
|
||||
for (let i = 0; i < magnitude.length; i += 1) {
|
||||
const m = magnitude[i]
|
||||
const p = m * m
|
||||
powerSum += p
|
||||
magSum += m
|
||||
weightedBinSum += i * m
|
||||
totalEnergy += m
|
||||
}
|
||||
|
||||
const rms = Math.sqrt(powerSum / bins)
|
||||
const energy = totalEnergy / bins
|
||||
|
||||
const centroidBin = magSum > 0 ? weightedBinSum / magSum : 0
|
||||
const spectralCentroid = (centroidBin / bins) * nyquist
|
||||
|
||||
const rolloffTarget = powerSum * 0.85
|
||||
let rolloffPower = 0
|
||||
let rolloffBin = 0
|
||||
for (let i = 0; i < magnitude.length; i += 1) {
|
||||
rolloffPower += magnitude[i] * magnitude[i]
|
||||
if (rolloffPower >= rolloffTarget) {
|
||||
rolloffBin = i
|
||||
break
|
||||
}
|
||||
}
|
||||
const spectralRolloff = (rolloffBin / bins) * nyquist
|
||||
|
||||
let geoLogSum = 0
|
||||
for (let i = 0; i < magnitude.length; i += 1) {
|
||||
geoLogSum += Math.log(Math.max(1e-12, magnitude[i]))
|
||||
}
|
||||
const geometricMean = Math.exp(geoLogSum / bins)
|
||||
const arithmeticMean = magSum / bins
|
||||
const spectralFlatness = arithmeticMean > 0 ? geometricMean / arithmeticMean : 0
|
||||
|
||||
let zc = 0
|
||||
for (let i = 1; i < this.timeData.length; i += 1) {
|
||||
const prev = this.timeData[i - 1]
|
||||
const cur = this.timeData[i]
|
||||
if ((prev >= 0 && cur < 0) || (prev < 0 && cur >= 0)) zc += 1
|
||||
}
|
||||
const zcr = this.timeData.length > 1 ? zc / (this.timeData.length - 1) : 0
|
||||
|
||||
const frequencyBands = this.extractFrequencyBands(sampleRate, magnitude)
|
||||
const beatInfo = this.detectBeat(energy)
|
||||
const patternInfo = this.detectPatterns(energy, beatInfo.isBeat)
|
||||
|
||||
return {
|
||||
rms: clamp01(rms),
|
||||
energy: clamp01(energy),
|
||||
spectralCentroid,
|
||||
spectralRolloff,
|
||||
spectralFlatness: clamp01(spectralFlatness),
|
||||
zcr: clamp01(zcr),
|
||||
frequencyBands,
|
||||
beatInfo,
|
||||
patternInfo,
|
||||
floatFreqData: new Float32Array(this.frequencyData),
|
||||
floatTimeData: new Float32Array(this.timeData),
|
||||
}
|
||||
}
|
||||
|
||||
private extractFrequencyBands(sampleRate: number, magnitude: Float32Array): MuonFrequencyBands {
|
||||
const binCount = magnitude.length
|
||||
const nyquist = sampleRate / 2
|
||||
const hzPerBin = binCount > 0 ? nyquist / binCount : 1
|
||||
|
||||
const out: MuonFrequencyBands = {
|
||||
sub: 0,
|
||||
bass: 0,
|
||||
lowMid: 0,
|
||||
mid: 0,
|
||||
highMid: 0,
|
||||
presence: 0,
|
||||
brilliance: 0,
|
||||
}
|
||||
|
||||
const keys: BandKey[] = ['sub', 'bass', 'lowMid', 'mid', 'highMid', 'presence', 'brilliance']
|
||||
for (const key of keys) {
|
||||
const [lowHz, highHz] = FREQUENCY_BANDS[key]
|
||||
const start = Math.max(0, Math.floor(lowHz / hzPerBin))
|
||||
const end = Math.min(binCount, Math.ceil(highHz / hzPerBin))
|
||||
const span = Math.max(1, end - start)
|
||||
let sumSquares = 0
|
||||
for (let i = start; i < end; i += 1) {
|
||||
const m = magnitude[i]
|
||||
sumSquares += m * m
|
||||
}
|
||||
out[key] = Math.sqrt(sumSquares / span)
|
||||
}
|
||||
|
||||
return out
|
||||
}
|
||||
|
||||
private detectBeat(energy: number): MuonBeatInfo {
|
||||
const now = performance.now() / 1000
|
||||
|
||||
this.energyHistory.push(energy)
|
||||
if (this.energyHistory.length > ENERGY_HISTORY_SIZE) this.energyHistory.shift()
|
||||
|
||||
const { average, variance } = this.calculateLocalAverageAndVariance(this.energyHistory)
|
||||
const stdDev = Math.sqrt(Math.max(variance, 1e-9))
|
||||
const zScore = stdDev > 0 ? (energy - average) / stdDev : 0
|
||||
|
||||
const minInterval = 0.18
|
||||
const isCandidate = energy > average * this.beatThreshold && zScore > 0.35
|
||||
const isBeat = isCandidate && (now - this.lastBeatTime) > minInterval
|
||||
|
||||
if (isBeat) {
|
||||
this.lastBeatTime = now
|
||||
this.beatTimes.push(now)
|
||||
if (this.beatTimes.length > BEAT_TIME_HISTORY_SIZE) this.beatTimes.shift()
|
||||
this.bpm = this.estimateBpm()
|
||||
}
|
||||
|
||||
const beatPeriod = this.bpm > 0 ? 60 / this.bpm : 0
|
||||
const timeSinceLastBeat = this.lastBeatTime > 0 ? now - this.lastBeatTime : 0
|
||||
const beatPhase = beatPeriod > 0 ? (timeSinceLastBeat % beatPeriod) / beatPeriod : 0
|
||||
|
||||
const confidence = clamp01((zScore / 3) * 0.6 + ((energy / Math.max(average, 1e-6)) - 1) * 0.4)
|
||||
|
||||
return {
|
||||
isBeat,
|
||||
bpm: this.bpm,
|
||||
confidence,
|
||||
timeSinceLastBeat,
|
||||
beatPhase: clamp01(beatPhase),
|
||||
}
|
||||
}
|
||||
|
||||
private detectPatterns(energy: number, isBeat: boolean): MuonPatternInfo {
|
||||
const energyDelta = energy - this.previousEnergy
|
||||
this.previousEnergy = energy
|
||||
|
||||
const tail = this.energyHistory.slice(-12)
|
||||
const tailAvg = tail.length ? tail.reduce((sum, v) => sum + v, 0) / tail.length : energy
|
||||
const silenceFloor = 0.008
|
||||
|
||||
const isSilence = energy < silenceFloor
|
||||
const isBuildUp = !isSilence && energyDelta > 0.01 && energy > tailAvg * 1.02
|
||||
const isBeatDrop = isBeat && !isSilence && energy > tailAvg * 1.35
|
||||
|
||||
return {
|
||||
isBeatDrop,
|
||||
isBuildUp,
|
||||
isSilence,
|
||||
intensity: clamp01(energy * 8),
|
||||
energyDelta,
|
||||
}
|
||||
}
|
||||
|
||||
private calculateLocalAverageAndVariance(values: number[]): { average: number; variance: number } {
|
||||
if (!values.length) return { average: 0, variance: 0 }
|
||||
const average = values.reduce((sum, value) => sum + value, 0) / values.length
|
||||
const variance = values.reduce((sum, value) => {
|
||||
const d = value - average
|
||||
return sum + d * d
|
||||
}, 0) / values.length
|
||||
return { average, variance }
|
||||
}
|
||||
|
||||
private estimateBpm(): number {
|
||||
if (this.beatTimes.length < 2) return this.bpm
|
||||
const intervals: number[] = []
|
||||
for (let i = 1; i < this.beatTimes.length; i += 1) {
|
||||
const dt = this.beatTimes[i] - this.beatTimes[i - 1]
|
||||
if (dt > 0.2 && dt < 1.5) intervals.push(dt)
|
||||
}
|
||||
if (!intervals.length) return this.bpm
|
||||
const avgInterval = intervals.reduce((sum, v) => sum + v, 0) / intervals.length
|
||||
const bpm = 60 / avgInterval
|
||||
return Math.max(40, Math.min(220, bpm))
|
||||
}
|
||||
}
|
||||
@@ -11,6 +11,7 @@ interface KaraokePerformanceModeProps {
|
||||
lyricsTracks?: Record<string, string>
|
||||
stems?: Array<{ name: string; label: string; url: string }>
|
||||
audioUrl?: string
|
||||
analysisAudioUrl?: string
|
||||
videoUrl?: string
|
||||
videoPlaylist?: Array<{ url: string; duration?: number }>
|
||||
videoPlaybackMode?: 'synced' | 'playlist'
|
||||
@@ -54,6 +55,7 @@ export default function KaraokePerformanceMode({
|
||||
lyricsTracks,
|
||||
stems,
|
||||
audioUrl,
|
||||
analysisAudioUrl,
|
||||
videoUrl,
|
||||
videoPlaylist,
|
||||
videoPlaybackMode,
|
||||
@@ -124,6 +126,7 @@ export default function KaraokePerformanceMode({
|
||||
lyricsTracks={lyricsTracks}
|
||||
stems={stems}
|
||||
audioUrl={audioUrl}
|
||||
analysisAudioUrl={analysisAudioUrl}
|
||||
videoUrl={videoUrl}
|
||||
videoPlaylist={videoPlaylist}
|
||||
videoPlaybackMode={videoPlaybackMode}
|
||||
|
||||
@@ -56,22 +56,84 @@ interface VizSettings {
|
||||
quality: 'low' | 'balanced' | 'high'
|
||||
fps_cap: 30 | 60 | 120
|
||||
muon: {
|
||||
// Original multipliers
|
||||
radius_multiplier: number
|
||||
aperture: number
|
||||
spacing: number
|
||||
// Controls Audit additions
|
||||
particle_count: number // direct int 40–500
|
||||
rotation_speed: number // ×
|
||||
glow_intensity: number // ×
|
||||
particle_size: number // ×
|
||||
drum_reactivity: number // ×
|
||||
hue_range: number // ×
|
||||
beat_sensitivity: number // ×
|
||||
ellipse_ratio: number // direct 0.30–1.0
|
||||
// GUI Parameter Map additions
|
||||
idle_speed: number // ×
|
||||
noise_scale: number // × (reserved for Three.js port)
|
||||
noise_force: number // × (reserved for Three.js port)
|
||||
mono_color: boolean
|
||||
divisions: number
|
||||
lifespan: number
|
||||
particle_mirror: boolean
|
||||
dynamic_radius: number
|
||||
delta_response_limit: number
|
||||
visualization_preset: 'default' | 'orbital' | 'flower' | 'spiro' | 'hybrid'
|
||||
}
|
||||
fluid: {
|
||||
// Original multipliers
|
||||
spawn_density: number
|
||||
splat_force: number
|
||||
decay: number
|
||||
// Controls Audit additions
|
||||
background_persistence: number
|
||||
beat_burst_count: number
|
||||
treble_pulse: number
|
||||
drag: number
|
||||
blob_life: number
|
||||
// GUI Parameter Map additions
|
||||
color_speed: number
|
||||
}
|
||||
}
|
||||
|
||||
const VIZ_DEFAULTS: VizSettings = {
|
||||
quality: 'balanced',
|
||||
fps_cap: 60,
|
||||
muon: { radius_multiplier: 1.0, aperture: 1.0, spacing: 1.0 },
|
||||
fluid: { spawn_density: 1.0, splat_force: 1.0, decay: 1.0 },
|
||||
muon: {
|
||||
radius_multiplier: 1.0,
|
||||
aperture: 1.0,
|
||||
spacing: 1.0,
|
||||
particle_count: 220,
|
||||
rotation_speed: 1.0,
|
||||
glow_intensity: 1.0,
|
||||
particle_size: 1.0,
|
||||
drum_reactivity: 1.0,
|
||||
hue_range: 1.0,
|
||||
beat_sensitivity: 1.0,
|
||||
ellipse_ratio: 0.62,
|
||||
idle_speed: 1.0,
|
||||
noise_scale: 1.0,
|
||||
noise_force: 1.0,
|
||||
mono_color: false,
|
||||
divisions: 16,
|
||||
lifespan: 180,
|
||||
particle_mirror: false,
|
||||
dynamic_radius: 1.0,
|
||||
delta_response_limit: 0.005,
|
||||
visualization_preset: 'hybrid',
|
||||
},
|
||||
fluid: {
|
||||
spawn_density: 1.0,
|
||||
splat_force: 1.0,
|
||||
decay: 1.0,
|
||||
background_persistence: 1.0,
|
||||
beat_burst_count: 1.0,
|
||||
treble_pulse: 1.0,
|
||||
drag: 1.0,
|
||||
blob_life: 1.0,
|
||||
color_speed: 1.0,
|
||||
},
|
||||
}
|
||||
|
||||
const PERF_SIDEBAR_WIDTH = 280
|
||||
@@ -81,9 +143,9 @@ function isLeadingVocalStem(name: string): boolean {
|
||||
return LEADING_VOCAL_PATTERN.test((name || '').trim())
|
||||
}
|
||||
|
||||
function clampVizMultiplier(value: number): number {
|
||||
function clampVizMultiplier(value: number, min = 0.0, max = 3.0): number {
|
||||
if (!Number.isFinite(value)) return 1
|
||||
return Math.max(0.25, Math.min(2, value))
|
||||
return Math.max(min, Math.min(max, value))
|
||||
}
|
||||
|
||||
function deepMergeVizSettings(base: VizSettings, patch?: DeepPartial<VizSettings> | null): VizSettings {
|
||||
@@ -113,14 +175,45 @@ function normalizeVizSettings(input: VizSettings): VizSettings {
|
||||
quality: input.quality === 'low' || input.quality === 'high' ? input.quality : 'balanced',
|
||||
fps_cap: input.fps_cap === 30 || input.fps_cap === 120 ? input.fps_cap : 60,
|
||||
muon: {
|
||||
radius_multiplier: clampVizMultiplier(input.muon.radius_multiplier),
|
||||
aperture: clampVizMultiplier(input.muon.aperture),
|
||||
spacing: clampVizMultiplier(input.muon.spacing),
|
||||
radius_multiplier: clampVizMultiplier(input.muon.radius_multiplier, 0.25, 2.0),
|
||||
aperture: clampVizMultiplier(input.muon.aperture, 0.25, 2.0),
|
||||
spacing: clampVizMultiplier(input.muon.spacing, 0.25, 2.0),
|
||||
particle_count: Math.max(40, Math.min(500, Math.round(Number.isFinite(input.muon.particle_count) ? input.muon.particle_count : 220))),
|
||||
rotation_speed: clampVizMultiplier(input.muon.rotation_speed, 0.2, 4.0),
|
||||
glow_intensity: clampVizMultiplier(input.muon.glow_intensity, 0.0, 3.0),
|
||||
particle_size: clampVizMultiplier(input.muon.particle_size, 0.25, 3.0),
|
||||
drum_reactivity: clampVizMultiplier(input.muon.drum_reactivity, 0.0, 2.0),
|
||||
hue_range: clampVizMultiplier(input.muon.hue_range, 0.0, 3.0),
|
||||
beat_sensitivity: clampVizMultiplier(input.muon.beat_sensitivity, 0.2, 3.0),
|
||||
ellipse_ratio: clampVizMultiplier(input.muon.ellipse_ratio, 0.30, 1.0),
|
||||
idle_speed: clampVizMultiplier(input.muon.idle_speed, 0.0, 2.0),
|
||||
noise_scale: clampVizMultiplier(input.muon.noise_scale, 0.0, 3.0),
|
||||
noise_force: clampVizMultiplier(input.muon.noise_force, 0.0, 3.0),
|
||||
mono_color: Boolean(input.muon.mono_color),
|
||||
divisions: Math.max(8, Math.min(48, Math.round(Number.isFinite(input.muon.divisions) ? input.muon.divisions : 16))),
|
||||
lifespan: Math.max(45, Math.min(420, Math.round(Number.isFinite(input.muon.lifespan) ? input.muon.lifespan : 180))),
|
||||
particle_mirror: Boolean(input.muon.particle_mirror),
|
||||
dynamic_radius: clampVizMultiplier(Number.isFinite(input.muon.dynamic_radius) ? input.muon.dynamic_radius : 1.0, 0.2, 3.0),
|
||||
delta_response_limit: clampVizMultiplier(Number.isFinite(input.muon.delta_response_limit) ? input.muon.delta_response_limit : 0.005, 0.0005, 0.05),
|
||||
visualization_preset:
|
||||
input.muon.visualization_preset === 'default' ||
|
||||
input.muon.visualization_preset === 'orbital' ||
|
||||
input.muon.visualization_preset === 'flower' ||
|
||||
input.muon.visualization_preset === 'spiro' ||
|
||||
input.muon.visualization_preset === 'hybrid'
|
||||
? input.muon.visualization_preset
|
||||
: 'hybrid',
|
||||
},
|
||||
fluid: {
|
||||
spawn_density: clampVizMultiplier(input.fluid.spawn_density),
|
||||
splat_force: clampVizMultiplier(input.fluid.splat_force),
|
||||
decay: clampVizMultiplier(input.fluid.decay),
|
||||
spawn_density: clampVizMultiplier(input.fluid.spawn_density, 0.25, 3.0),
|
||||
splat_force: clampVizMultiplier(input.fluid.splat_force, 0.25, 3.0),
|
||||
decay: clampVizMultiplier(input.fluid.decay, 0.25, 3.0),
|
||||
background_persistence: clampVizMultiplier(input.fluid.background_persistence, 0.2, 3.0),
|
||||
beat_burst_count: clampVizMultiplier(input.fluid.beat_burst_count, 0.25, 3.0),
|
||||
treble_pulse: clampVizMultiplier(input.fluid.treble_pulse, 0.0, 3.0),
|
||||
drag: clampVizMultiplier(input.fluid.drag, 0.5, 2.0),
|
||||
blob_life: clampVizMultiplier(input.fluid.blob_life, 0.25, 3.0),
|
||||
color_speed: clampVizMultiplier(input.fluid.color_speed, 0.0, 3.0),
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -129,6 +222,7 @@ interface Props {
|
||||
lrcText: string
|
||||
lyricsTracks?: Record<string, string>
|
||||
audioUrl?: string
|
||||
analysisAudioUrl?: string
|
||||
videoUrl?: string
|
||||
videoPlaylist?: Array<{ url: string; duration?: number }>
|
||||
videoPlaybackMode?: 'synced' | 'playlist'
|
||||
@@ -155,11 +249,35 @@ interface Props {
|
||||
radius_multiplier?: number
|
||||
aperture?: number
|
||||
spacing?: number
|
||||
particle_count?: number
|
||||
rotation_speed?: number
|
||||
glow_intensity?: number
|
||||
particle_size?: number
|
||||
drum_reactivity?: number
|
||||
hue_range?: number
|
||||
beat_sensitivity?: number
|
||||
ellipse_ratio?: number
|
||||
idle_speed?: number
|
||||
noise_scale?: number
|
||||
noise_force?: number
|
||||
mono_color?: boolean
|
||||
divisions?: number
|
||||
lifespan?: number
|
||||
particle_mirror?: boolean
|
||||
dynamic_radius?: number
|
||||
delta_response_limit?: number
|
||||
visualization_preset?: 'default' | 'orbital' | 'flower' | 'spiro' | 'hybrid'
|
||||
}
|
||||
fluid?: {
|
||||
spawn_density?: number
|
||||
splat_force?: number
|
||||
decay?: number
|
||||
background_persistence?: number
|
||||
beat_burst_count?: number
|
||||
treble_pulse?: number
|
||||
drag?: number
|
||||
blob_life?: number
|
||||
color_speed?: number
|
||||
}
|
||||
} | null
|
||||
/** Artist name from source audio metadata. */
|
||||
@@ -1429,7 +1547,7 @@ function LrcEditor({
|
||||
|
||||
// ─── Main component ───────────────────────────────────────────────────────────
|
||||
|
||||
export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, videoUrl, videoPlaylist, videoPlaybackMode, stems, title, styleOptions, defaultStyle, onLrcChange, storageKey, coverArtUrl, thumbnailUrl, sekDynamicPalette, sekVisualizerMode, sekVisualizerSettings, artist, songTitle, performanceMode = false, autoPlay = false, disableGlobalPlaybackRegistry = false, onPlaybackEnded }: Props) {
|
||||
export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, analysisAudioUrl, videoUrl, videoPlaylist, videoPlaybackMode, stems, title, styleOptions, defaultStyle, onLrcChange, storageKey, coverArtUrl, thumbnailUrl, sekDynamicPalette, sekVisualizerMode, sekVisualizerSettings, artist, songTitle, performanceMode = false, autoPlay = false, disableGlobalPlaybackRegistry = false, onPlaybackEnded }: Props) {
|
||||
const resolvedLyricsTracks = useMemo(() => {
|
||||
const normalized: Record<string, string> = {}
|
||||
for (const [rawKey, rawText] of Object.entries(lyricsTracks || {})) {
|
||||
@@ -1517,18 +1635,44 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
|
||||
if (!sekVisualizerSettings) return null
|
||||
const quality = sekVisualizerSettings.quality
|
||||
const fps = sekVisualizerSettings.fps_cap
|
||||
const sm = sekVisualizerSettings.muon
|
||||
const sf = sekVisualizerSettings.fluid
|
||||
const patch: DeepPartial<VizSettings> = {
|
||||
quality: quality === 'low' || quality === 'balanced' || quality === 'high' ? quality : undefined,
|
||||
fps_cap: fps === 30 || fps === 60 || fps === 120 ? fps : undefined,
|
||||
muon: {
|
||||
radius_multiplier: sekVisualizerSettings.muon?.radius_multiplier,
|
||||
aperture: sekVisualizerSettings.muon?.aperture,
|
||||
spacing: sekVisualizerSettings.muon?.spacing,
|
||||
radius_multiplier: sm?.radius_multiplier,
|
||||
aperture: sm?.aperture,
|
||||
spacing: sm?.spacing,
|
||||
particle_count: sm?.particle_count,
|
||||
rotation_speed: sm?.rotation_speed,
|
||||
glow_intensity: sm?.glow_intensity,
|
||||
particle_size: sm?.particle_size,
|
||||
drum_reactivity: sm?.drum_reactivity,
|
||||
hue_range: sm?.hue_range,
|
||||
beat_sensitivity: sm?.beat_sensitivity,
|
||||
ellipse_ratio: sm?.ellipse_ratio,
|
||||
idle_speed: sm?.idle_speed,
|
||||
noise_scale: sm?.noise_scale,
|
||||
noise_force: sm?.noise_force,
|
||||
mono_color: sm?.mono_color,
|
||||
divisions: sm?.divisions,
|
||||
lifespan: sm?.lifespan,
|
||||
particle_mirror: sm?.particle_mirror,
|
||||
dynamic_radius: sm?.dynamic_radius,
|
||||
delta_response_limit: sm?.delta_response_limit,
|
||||
visualization_preset: sm?.visualization_preset,
|
||||
},
|
||||
fluid: {
|
||||
spawn_density: sekVisualizerSettings.fluid?.spawn_density,
|
||||
splat_force: sekVisualizerSettings.fluid?.splat_force,
|
||||
decay: sekVisualizerSettings.fluid?.decay,
|
||||
spawn_density: sf?.spawn_density,
|
||||
splat_force: sf?.splat_force,
|
||||
decay: sf?.decay,
|
||||
background_persistence: sf?.background_persistence,
|
||||
beat_burst_count: sf?.beat_burst_count,
|
||||
treble_pulse: sf?.treble_pulse,
|
||||
drag: sf?.drag,
|
||||
blob_life: sf?.blob_life,
|
||||
color_speed: sf?.color_speed,
|
||||
},
|
||||
}
|
||||
return normalizeVizSettings(deepMergeVizSettings(VIZ_DEFAULTS, patch))
|
||||
@@ -1767,6 +1911,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
|
||||
|
||||
// Single-track audio ref
|
||||
const audioRef = useRef<HTMLAudioElement>(null)
|
||||
const analysisAudioRef = useRef<HTMLAudioElement>(null)
|
||||
const videoRef = useRef<HTMLVideoElement>(null)
|
||||
// Multi-stem audio refs
|
||||
const stemAudioRefs = useRef<Record<string, HTMLAudioElement>>({})
|
||||
@@ -1778,19 +1923,12 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
|
||||
const gainRefs = useRef<Record<string, GainNode>>({})
|
||||
const metadataListenerRefs = useRef<Record<string, () => void>>({})
|
||||
|
||||
// Important: on first mount in multi-stem mode, `useAudioFeatures` can run
|
||||
// before the stem graph is created (`gainRefs.current` still empty). In that
|
||||
// case the hook exits early and would not reconnect until one of its deps
|
||||
// changes, causing static/non-reactive visualizers. Gate it until tap nodes
|
||||
// exist so the first enabled run always has a valid audio source.
|
||||
const audioFeatureTapReady = !isMultiStem || Object.keys(gainRefs.current).length > 0
|
||||
|
||||
const featuresRef = useAudioFeatures({
|
||||
audioCtxRef,
|
||||
stemGainRefs: gainRefs,
|
||||
singleAudioRef: audioRef,
|
||||
analysisAudioRef,
|
||||
isPlayingRef,
|
||||
enabled: isVisualizerMode && audioFeatureTapReady,
|
||||
enabled: isVisualizerMode,
|
||||
analysisSourceKey: analysisAudioUrl ?? audioUrl ?? null,
|
||||
})
|
||||
|
||||
const containerRef = useRef<HTMLDivElement>(null)
|
||||
@@ -1832,6 +1970,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
|
||||
audioRef.current.pause()
|
||||
}
|
||||
if (videoRef.current) videoRef.current.pause()
|
||||
syncAnalysisAudio(currentTimeRef.current, false)
|
||||
syncAnchorStemRef.current = null
|
||||
lastAnchorTimeRef.current = null
|
||||
lastRafTime.current = null
|
||||
@@ -1861,6 +2000,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
|
||||
syncAnchorStemRef.current = null
|
||||
lastAnchorTimeRef.current = null
|
||||
const finalTime = typeof endedAt === 'number' ? endedAt : currentTimeRef.current
|
||||
syncAnalysisAudio(finalTime, false)
|
||||
manualTimeRef.current = finalTime
|
||||
currentTimeRef.current = finalTime
|
||||
lastRafTime.current = null
|
||||
@@ -2312,6 +2452,33 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
|
||||
return () => window.clearInterval(timer)
|
||||
}, [isMultiStem, isPlaying, useVideoAsTimeSource])
|
||||
|
||||
useEffect(() => {
|
||||
if (!isPlaying) return
|
||||
const timer = window.setInterval(() => {
|
||||
const analysis = analysisAudioRef.current
|
||||
if (!analysis) return
|
||||
|
||||
const transportRate = isMultiStem
|
||||
? (pickStemClockSource(syncAnchorStemRef.current)?.playbackRate ?? 1)
|
||||
: (audioRef.current?.playbackRate ?? videoRef.current?.playbackRate ?? 1)
|
||||
|
||||
if (analysis.playbackRate !== transportRate) {
|
||||
analysis.playbackRate = transportRate
|
||||
}
|
||||
|
||||
const transportTime = currentTimeRef.current
|
||||
if (Math.abs(analysis.currentTime - transportTime) > 0.04) {
|
||||
analysis.currentTime = transportTime
|
||||
}
|
||||
|
||||
if (analysis.paused) {
|
||||
analysis.play().catch(() => {})
|
||||
}
|
||||
}, 120)
|
||||
|
||||
return () => window.clearInterval(timer)
|
||||
}, [isPlaying, isMultiStem])
|
||||
|
||||
// End multi-stem playback as soon as every active stem reaches end.
|
||||
useEffect(() => {
|
||||
if (!isMultiStem || !isPlaying) return
|
||||
@@ -2457,6 +2624,26 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
|
||||
)
|
||||
}
|
||||
|
||||
function syncAnalysisAudio(targetTime: number, shouldPlay: boolean) {
|
||||
const analysis = analysisAudioRef.current
|
||||
if (!analysis) return
|
||||
|
||||
const transportRate = isMultiStemRef.current
|
||||
? (pickStemClockSource(syncAnchorStemRef.current)?.playbackRate ?? 1)
|
||||
: (audioRef.current?.playbackRate ?? videoRef.current?.playbackRate ?? 1)
|
||||
|
||||
analysis.playbackRate = transportRate
|
||||
if (Math.abs(analysis.currentTime - targetTime) > 0.04) {
|
||||
analysis.currentTime = targetTime
|
||||
}
|
||||
|
||||
if (shouldPlay) {
|
||||
analysis.play().catch(() => {})
|
||||
} else {
|
||||
analysis.pause()
|
||||
}
|
||||
}
|
||||
|
||||
function mapTransportTimeToVideoSegment(targetTime: number): { index: number; localTime: number } {
|
||||
if (!isPlaylistBackdropMode || !videoPlaylistRef.current.length) {
|
||||
return { index: 0, localTime: targetTime }
|
||||
@@ -2524,6 +2711,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
|
||||
currentTimeRef.current = t
|
||||
lastRafTime.current = null
|
||||
setCurrentTime(t)
|
||||
syncAnalysisAudio(t, isPlayingRef.current)
|
||||
return t
|
||||
}
|
||||
|
||||
@@ -2578,6 +2766,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
|
||||
const audios = Object.values(stemAudioRefs.current)
|
||||
if (currentlyPlaying) {
|
||||
audios.forEach(a => a.pause())
|
||||
syncAnalysisAudio(currentTimeRef.current, false)
|
||||
setVideoStartPhase('idle')
|
||||
// Don't unregister — keep bar visible showing paused state
|
||||
setPlaybackState(false)
|
||||
@@ -2608,12 +2797,14 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
|
||||
videoRef.current.play().catch((err) => console.warn('[LyricsLivePreview] video.play() failed (multi-stem)', err))
|
||||
}
|
||||
try { await ctx.resume() } catch { /* AudioContext suspended; audio starts on next user gesture */ }
|
||||
syncAnalysisAudio(startTime, true)
|
||||
setPlaybackState(true)
|
||||
if (hasVideoBackdrop) setVideoStartPhase('playing')
|
||||
}
|
||||
} else if (audioRef.current) {
|
||||
if (currentlyPlaying) {
|
||||
audioRef.current.pause()
|
||||
syncAnalysisAudio(currentTimeRef.current, false)
|
||||
setVideoStartPhase('idle')
|
||||
// Don't unregister — keep bar visible showing paused state
|
||||
setPlaybackState(false)
|
||||
@@ -2628,6 +2819,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
|
||||
}
|
||||
audioRef.current.currentTime = startTime
|
||||
audioRef.current.play().catch(() => {})
|
||||
syncAnalysisAudio(startTime, true)
|
||||
// Publish immediately so the bar appears at once rather than waiting
|
||||
// for the next effect cycle (avoids the clearMeta→publishMeta gap).
|
||||
if (!disableGlobalPlaybackRegistry) {
|
||||
@@ -2642,6 +2834,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
|
||||
} else if (videoRef.current) {
|
||||
if (currentlyPlaying) {
|
||||
videoRef.current.pause()
|
||||
syncAnalysisAudio(currentTimeRef.current, false)
|
||||
setVideoStartPhase('idle')
|
||||
setPlaybackState(false)
|
||||
} else {
|
||||
@@ -2654,6 +2847,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
|
||||
if (isPlaylistBackdropMode) syncPlaylistVideoToTransport(startTime)
|
||||
else videoRef.current.currentTime = startTime
|
||||
videoRef.current.play().catch(() => {})
|
||||
syncAnalysisAudio(startTime, true)
|
||||
if (!disableGlobalPlaybackRegistry) {
|
||||
publishMeta(
|
||||
{ title: title ?? 'Live Preview', isPlaying: true, currentTime: startTime, duration, volume: isMuted ? 0 : volume },
|
||||
@@ -2676,6 +2870,9 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
|
||||
{ toggle: () => togglePlayRef.current(), seek: seekTime, setVolume: (v) => setVolumeFromBar.current(v) }
|
||||
)
|
||||
}
|
||||
syncAnalysisAudio(startTime, true)
|
||||
} else {
|
||||
syncAnalysisAudio(currentTimeRef.current, false)
|
||||
}
|
||||
// Don't unregister on pause — keep bar visible
|
||||
lastRafTime.current = null
|
||||
@@ -2777,6 +2974,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
|
||||
videoRef.current.currentTime = 0
|
||||
videoRef.current.pause()
|
||||
}
|
||||
syncAnalysisAudio(0, false)
|
||||
syncAnchorStemRef.current = null
|
||||
lastAnchorTimeRef.current = null
|
||||
isActiveRef.current = false
|
||||
@@ -3670,7 +3868,19 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, audioUrl, vid
|
||||
/>
|
||||
)}
|
||||
|
||||
{/* Hidden single-track audio element */}
|
||||
{/* Hidden analysis audio element (always mounted when analysis source exists) */}
|
||||
{(analysisAudioUrl || audioUrl) && (
|
||||
<audio
|
||||
ref={analysisAudioRef}
|
||||
src={analysisAudioUrl ?? audioUrl}
|
||||
preload="auto"
|
||||
crossOrigin="anonymous"
|
||||
muted
|
||||
style={{ display: 'none' }}
|
||||
/>
|
||||
)}
|
||||
|
||||
{/* Hidden single-track transport audio element */}
|
||||
{!isMultiStem && audioUrl && (
|
||||
<audio
|
||||
ref={audioRef}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { useCallback, useEffect, useRef, useState } from 'react'
|
||||
import { useCallback, useEffect, useRef } from 'react'
|
||||
import type { LyricsVisualStyle } from './LyricsLivePreview'
|
||||
|
||||
interface Stem {
|
||||
@@ -39,19 +39,149 @@ interface VizSettings {
|
||||
radius_multiplier: number
|
||||
aperture: number
|
||||
spacing: number
|
||||
particle_count: number
|
||||
rotation_speed: number
|
||||
glow_intensity: number
|
||||
particle_size: number
|
||||
drum_reactivity: number
|
||||
hue_range: number
|
||||
beat_sensitivity: number
|
||||
ellipse_ratio: number
|
||||
idle_speed: number
|
||||
noise_scale: number
|
||||
noise_force: number
|
||||
mono_color: boolean
|
||||
divisions: number
|
||||
lifespan: number
|
||||
particle_mirror: boolean
|
||||
dynamic_radius: number
|
||||
delta_response_limit: number
|
||||
visualization_preset: 'default' | 'orbital' | 'flower' | 'spiro' | 'hybrid'
|
||||
}
|
||||
fluid: {
|
||||
spawn_density: number
|
||||
splat_force: number
|
||||
decay: number
|
||||
background_persistence: number
|
||||
beat_burst_count: number
|
||||
treble_pulse: number
|
||||
drag: number
|
||||
blob_life: number
|
||||
color_speed: number
|
||||
}
|
||||
}
|
||||
|
||||
const VIZ_DEFAULTS: VizSettings = {
|
||||
quality: 'balanced',
|
||||
fps_cap: 60,
|
||||
muon: { radius_multiplier: 1, aperture: 1, spacing: 1 },
|
||||
fluid: { spawn_density: 1, splat_force: 1, decay: 1 },
|
||||
muon: {
|
||||
radius_multiplier: 1.0,
|
||||
aperture: 1.0,
|
||||
spacing: 1.0,
|
||||
particle_count: 220,
|
||||
rotation_speed: 1.0,
|
||||
glow_intensity: 1.0,
|
||||
particle_size: 1.0,
|
||||
drum_reactivity: 1.0,
|
||||
hue_range: 1.0,
|
||||
beat_sensitivity: 1.0,
|
||||
ellipse_ratio: 0.62,
|
||||
idle_speed: 1.0,
|
||||
noise_scale: 1.0,
|
||||
noise_force: 1.0,
|
||||
mono_color: false,
|
||||
divisions: 16,
|
||||
lifespan: 180,
|
||||
particle_mirror: false,
|
||||
dynamic_radius: 1.0,
|
||||
delta_response_limit: 0.005,
|
||||
visualization_preset: 'hybrid',
|
||||
},
|
||||
fluid: {
|
||||
spawn_density: 1.0,
|
||||
splat_force: 1.0,
|
||||
decay: 1.0,
|
||||
background_persistence: 1.0,
|
||||
beat_burst_count: 1.0,
|
||||
treble_pulse: 1.0,
|
||||
drag: 1.0,
|
||||
blob_life: 1.0,
|
||||
color_speed: 1.0,
|
||||
},
|
||||
}
|
||||
|
||||
// Named muon presets — applied to vizSettings.muon only (leaves other keys untouched).
|
||||
const MUON_PRESETS: Array<{ label: string; values: VizSettings['muon'] }> = [
|
||||
{
|
||||
label: 'Default',
|
||||
values: { radius_multiplier: 1.0, aperture: 1.0, spacing: 1.0, particle_count: 220, rotation_speed: 1.0, glow_intensity: 1.0, particle_size: 1.0, drum_reactivity: 1.0, hue_range: 1.0, beat_sensitivity: 1.0, ellipse_ratio: 0.62, idle_speed: 1.0, noise_scale: 1.0, noise_force: 1.0, mono_color: false, divisions: 16, lifespan: 180, particle_mirror: false, dynamic_radius: 1.0, delta_response_limit: 0.005, visualization_preset: 'hybrid' },
|
||||
},
|
||||
{
|
||||
label: 'Calm',
|
||||
values: { radius_multiplier: 1.0, aperture: 0.8, spacing: 1.2, particle_count: 120, rotation_speed: 0.5, glow_intensity: 0.4, particle_size: 0.7, drum_reactivity: 0.5, hue_range: 0.4, beat_sensitivity: 0.5, ellipse_ratio: 0.55, idle_speed: 0.6, noise_scale: 0.7, noise_force: 0.6, mono_color: false, divisions: 12, lifespan: 220, particle_mirror: false, dynamic_radius: 0.7, delta_response_limit: 0.007, visualization_preset: 'orbital' },
|
||||
},
|
||||
{
|
||||
label: 'Energetic',
|
||||
values: { radius_multiplier: 1.1, aperture: 1.3, spacing: 0.9, particle_count: 350, rotation_speed: 2.0, glow_intensity: 2.0, particle_size: 1.4, drum_reactivity: 1.8, hue_range: 1.8, beat_sensitivity: 2.0, ellipse_ratio: 0.68, idle_speed: 1.5, noise_scale: 1.4, noise_force: 1.5, mono_color: false, divisions: 24, lifespan: 160, particle_mirror: true, dynamic_radius: 1.6, delta_response_limit: 0.003, visualization_preset: 'hybrid' },
|
||||
},
|
||||
{
|
||||
label: 'Dreamy',
|
||||
values: { radius_multiplier: 1.2, aperture: 0.7, spacing: 1.5, particle_count: 280, rotation_speed: 0.6, glow_intensity: 2.5, particle_size: 1.1, drum_reactivity: 0.6, hue_range: 2.5, beat_sensitivity: 0.4, ellipse_ratio: 0.45, idle_speed: 0.8, noise_scale: 1.7, noise_force: 1.2, mono_color: false, divisions: 20, lifespan: 260, particle_mirror: false, dynamic_radius: 0.8, delta_response_limit: 0.008, visualization_preset: 'flower' },
|
||||
},
|
||||
{
|
||||
label: 'Minimal',
|
||||
values: { radius_multiplier: 0.8, aperture: 1.0, spacing: 2.0, particle_count: 60, rotation_speed: 0.8, glow_intensity: 0.0, particle_size: 0.6, drum_reactivity: 1.2, hue_range: 0.0, beat_sensitivity: 1.0, ellipse_ratio: 1.0, idle_speed: 0.5, noise_scale: 0.4, noise_force: 0.3, mono_color: true, divisions: 8, lifespan: 120, particle_mirror: false, dynamic_radius: 0.7, delta_response_limit: 0.012, visualization_preset: 'default' },
|
||||
},
|
||||
{
|
||||
label: 'Cosmic',
|
||||
values: { radius_multiplier: 1.3, aperture: 1.2, spacing: 0.7, particle_count: 400, rotation_speed: 1.5, glow_intensity: 1.8, particle_size: 1.8, drum_reactivity: 1.4, hue_range: 2.0, beat_sensitivity: 1.5, ellipse_ratio: 0.38, idle_speed: 1.2, noise_scale: 1.8, noise_force: 1.6, mono_color: false, divisions: 28, lifespan: 190, particle_mirror: true, dynamic_radius: 1.4, delta_response_limit: 0.004, visualization_preset: 'spiro' },
|
||||
},
|
||||
]
|
||||
|
||||
// Named fluid presets — applied to vizSettings.fluid only.
|
||||
const FLUID_PRESETS: Array<{ label: string; values: VizSettings['fluid'] }> = [
|
||||
{
|
||||
label: 'Default',
|
||||
values: { spawn_density: 1.0, splat_force: 1.0, decay: 1.0, background_persistence: 1.0, beat_burst_count: 1.0, treble_pulse: 1.0, drag: 1.0, blob_life: 1.0, color_speed: 1.0 },
|
||||
},
|
||||
{
|
||||
label: 'Ink',
|
||||
values: { spawn_density: 0.5, splat_force: 0.5, decay: 0.3, background_persistence: 3.0, beat_burst_count: 1.5, treble_pulse: 0.3, drag: 2.5, blob_life: 2.5, color_speed: 0.2 },
|
||||
},
|
||||
{
|
||||
label: 'Foam',
|
||||
values: { spawn_density: 3.0, splat_force: 0.3, decay: 3.0, background_persistence: 0.2, beat_burst_count: 3.0, treble_pulse: 2.5, drag: 0.4, blob_life: 0.3, color_speed: 3.0 },
|
||||
},
|
||||
{
|
||||
label: 'Lava',
|
||||
values: { spawn_density: 0.3, splat_force: 2.0, decay: 0.2, background_persistence: 3.5, beat_burst_count: 0.5, treble_pulse: 0.2, drag: 3.0, blob_life: 3.0, color_speed: 0.2 },
|
||||
},
|
||||
{
|
||||
label: 'Mist',
|
||||
values: { spawn_density: 1.5, splat_force: 0.2, decay: 2.0, background_persistence: 0.5, beat_burst_count: 0.5, treble_pulse: 1.5, drag: 0.5, blob_life: 0.5, color_speed: 0.8 },
|
||||
},
|
||||
{
|
||||
label: 'Storm',
|
||||
values: { spawn_density: 2.5, splat_force: 3.0, decay: 1.0, background_persistence: 1.0, beat_burst_count: 3.0, treble_pulse: 2.5, drag: 0.4, blob_life: 1.0, color_speed: 2.5 },
|
||||
},
|
||||
]
|
||||
|
||||
const EPS = 0.015
|
||||
|
||||
function muonPresetMatch(current: VizSettings['muon'], preset: VizSettings['muon']): boolean {
|
||||
const keys = Object.keys(preset) as Array<keyof VizSettings['muon']>
|
||||
return keys.every((k) => {
|
||||
const a = current[k]
|
||||
const b = preset[k]
|
||||
if (typeof a === 'boolean') return a === b
|
||||
if (typeof a === 'string') return a === b
|
||||
return Math.abs((a as number) - (b as number)) < EPS
|
||||
})
|
||||
}
|
||||
|
||||
function fluidPresetMatch(current: VizSettings['fluid'], preset: VizSettings['fluid']): boolean {
|
||||
const keys = Object.keys(preset) as Array<keyof VizSettings['fluid']>
|
||||
return keys.every((k) => Math.abs(current[k] - preset[k]) < EPS)
|
||||
}
|
||||
|
||||
const FONT_OPTIONS = [
|
||||
@@ -124,12 +254,6 @@ export default function PerformanceSidebar({
|
||||
patchVizSettings,
|
||||
onClose,
|
||||
}: PerformanceSidebarProps) {
|
||||
const [open, setOpen] = useState({
|
||||
background: true,
|
||||
stems: true,
|
||||
lyrics: true,
|
||||
visualizer: true,
|
||||
})
|
||||
const autoHideTimerRef = useRef<number | null>(null)
|
||||
|
||||
const scheduleAutoHide = useCallback(() => {
|
||||
@@ -170,24 +294,40 @@ export default function PerformanceSidebar({
|
||||
patchStyle(axis === 'x' ? { layoutOffsetXScroll: value } : { layoutOffsetYScroll: value })
|
||||
}
|
||||
|
||||
const sectionHeader = (key: keyof typeof open, label: string, hidden = false) => {
|
||||
if (hidden) return null
|
||||
return (
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => setOpen((prev) => ({ ...prev, [key]: !prev[key] }))}
|
||||
className="w-full flex items-center justify-between text-xs font-semibold tracking-wide text-slate-200"
|
||||
>
|
||||
<span>{label}</span>
|
||||
<span className="text-slate-400">{open[key] ? '▾' : '▸'}</span>
|
||||
</button>
|
||||
)
|
||||
}
|
||||
// Preset detection
|
||||
const activeMuonPreset = MUON_PRESETS.find((p) => muonPresetMatch(vizSettings.muon, p.values))
|
||||
const activeFluidPreset = FLUID_PRESETS.find((p) => fluidPresetMatch(vizSettings.fluid, p.values))
|
||||
|
||||
const panelClass = isNarrow
|
||||
? 'absolute inset-x-0 bottom-0 h-[62vh] rounded-t-2xl border-t border-white/15'
|
||||
: 'absolute right-0 top-0 h-full border-l border-white/15'
|
||||
|
||||
const slider = (
|
||||
label: string,
|
||||
value: number,
|
||||
displayValue: string,
|
||||
min: number,
|
||||
max: number,
|
||||
step: number,
|
||||
onChange: (v: number) => void,
|
||||
) => (
|
||||
<div className="space-y-1">
|
||||
<div className="flex items-center justify-between text-[10px] text-slate-400">
|
||||
<span>{label}</span>
|
||||
<span>{displayValue}</span>
|
||||
</div>
|
||||
<input
|
||||
type="range"
|
||||
min={min}
|
||||
max={max}
|
||||
step={step}
|
||||
value={value}
|
||||
onChange={(e) => onChange(parseFloat(e.target.value))}
|
||||
className="w-full h-1.5 accent-brand-500"
|
||||
/>
|
||||
</div>
|
||||
)
|
||||
|
||||
return (
|
||||
<div
|
||||
className={`${panelClass} z-[9] bg-black/70 backdrop-blur-md text-slate-100 overflow-y-auto`}
|
||||
@@ -208,236 +348,328 @@ export default function PerformanceSidebar({
|
||||
</div>
|
||||
|
||||
<div className="px-3 py-3 space-y-4">
|
||||
|
||||
{/* ── BACKGROUND ── */}
|
||||
<section className="space-y-2">
|
||||
{sectionHeader('background', 'BACKGROUND')}
|
||||
{open.background && (
|
||||
<div className={isNarrow ? 'flex flex-wrap gap-2' : 'grid grid-cols-2 gap-2'}>
|
||||
{[
|
||||
{ key: 'static', label: 'Static', value: 'karaoke' as LyricsVisualStyle, active: isStaticMode },
|
||||
{ key: 'muon', label: 'Muon', value: 'muon' as LyricsVisualStyle, active: visualStyle === 'muon' },
|
||||
{ key: 'fluid', label: 'Fluid', value: 'fluid' as LyricsVisualStyle, active: visualStyle === 'fluid' },
|
||||
...(hasVideo ? [{ key: 'video', label: 'Video', value: 'video' as LyricsVisualStyle, active: visualStyle === 'video' }] : []),
|
||||
].map((mode) => (
|
||||
<button
|
||||
key={mode.key}
|
||||
type="button"
|
||||
onClick={() => setVisualStyle(mode.value)}
|
||||
className={`rounded-md px-2 py-2 text-xs border transition-colors ${isNarrow ? 'min-w-[70px] flex-1' : ''} ${mode.active ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
|
||||
<div className="text-xs font-semibold tracking-wide text-slate-200">BACKGROUND</div>
|
||||
<div className={isNarrow ? 'flex flex-wrap gap-2' : 'grid grid-cols-2 gap-2'}>
|
||||
{[
|
||||
{ key: 'static', label: 'Static', value: 'karaoke' as LyricsVisualStyle, active: isStaticMode },
|
||||
{ key: 'muon', label: 'Muon', value: 'muon' as LyricsVisualStyle, active: visualStyle === 'muon' },
|
||||
{ key: 'fluid', label: 'Fluid', value: 'fluid' as LyricsVisualStyle, active: visualStyle === 'fluid' },
|
||||
...(hasVideo ? [{ key: 'video', label: 'Video', value: 'video' as LyricsVisualStyle, active: visualStyle === 'video' }] : []),
|
||||
].map((mode) => (
|
||||
<button
|
||||
key={mode.key}
|
||||
type="button"
|
||||
onClick={() => setVisualStyle(mode.value)}
|
||||
className={`rounded-md px-2 py-2 text-xs border transition-colors ${isNarrow ? 'min-w-[70px] flex-1' : ''} ${mode.active ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
|
||||
>
|
||||
{mode.label}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</section>
|
||||
|
||||
{/* ── STEMS ── */}
|
||||
<section className="space-y-2">
|
||||
<div className="text-xs font-semibold tracking-wide text-slate-200">STEMS</div>
|
||||
<div className="space-y-2">
|
||||
{stems?.length ? stems.map((stem) => (
|
||||
<div key={stem.name} className="space-y-1.5">
|
||||
<div className="flex items-center gap-2">
|
||||
<span className="text-xs font-medium text-slate-300 flex-1 truncate">{stem.label}</span>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => toggleStemMute(stem.name)}
|
||||
className={`w-6 h-6 text-[10px] rounded ${mutedStems[stem.name] ? 'bg-red-500 text-white' : 'bg-white/10 text-slate-300'}`}
|
||||
>
|
||||
M
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => toggleStemSolo(stem.name)}
|
||||
className={`w-6 h-6 text-[10px] rounded ${soloedStems[stem.name] ? 'bg-yellow-300 text-black' : 'bg-white/10 text-slate-300'}`}
|
||||
>
|
||||
S
|
||||
</button>
|
||||
</div>
|
||||
<div className="flex items-center gap-2">
|
||||
<input
|
||||
type="range"
|
||||
min={0}
|
||||
max={2}
|
||||
step={0.05}
|
||||
value={gains[stem.name] ?? 1}
|
||||
onChange={(e) => setStemGain(stem.name, parseFloat(e.target.value))}
|
||||
className="w-full h-1.5 accent-brand-500"
|
||||
/>
|
||||
<span className="text-[10px] tabular-nums text-slate-400 w-11 text-right">{Math.round((gains[stem.name] ?? 1) * 100)}%</span>
|
||||
</div>
|
||||
</div>
|
||||
)) : (
|
||||
<p className="text-[11px] text-slate-400">No stems — single track mode</p>
|
||||
)}
|
||||
</div>
|
||||
</section>
|
||||
|
||||
{/* ── LYRICS ── */}
|
||||
<section className="space-y-2">
|
||||
<div className="text-xs font-semibold tracking-wide text-slate-200">LYRICS</div>
|
||||
<div className="space-y-2.5">
|
||||
<div className="flex items-center gap-2">
|
||||
<select
|
||||
value={style.fontFamily}
|
||||
onChange={(e) => patchStyle({ fontFamily: e.target.value })}
|
||||
className="flex-1 text-xs bg-white/10 border border-white/15 rounded px-2 py-1"
|
||||
>
|
||||
{FONT_OPTIONS.map((font) => (
|
||||
<option key={font.value} value={font.value} style={{ background: '#0f172a', color: '#e2e8f0', fontFamily: font.value }}>
|
||||
{font.label}
|
||||
</option>
|
||||
))}
|
||||
</select>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => patchStyle({ fontBold: !style.fontBold })}
|
||||
className={`px-2 py-1 rounded text-xs font-semibold ${style.fontBold ? 'bg-white/25 text-white' : 'bg-white/10 text-slate-300'}`}
|
||||
>
|
||||
B
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div className="space-y-1">
|
||||
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Size</span><span>{style.fontSize}</span></div>
|
||||
<input type="range" min={12} max={56} step={1} value={style.fontSize} onChange={(e) => patchStyle({ fontSize: parseInt(e.target.value, 10) })} className="w-full h-1.5 accent-brand-500" />
|
||||
</div>
|
||||
|
||||
<div className="flex items-center gap-2">
|
||||
{(['left', 'center', 'right'] as const).map((align) => (
|
||||
<button key={align} type="button" onClick={() => patchStyle({ textAlign: align })}
|
||||
className={`flex-1 rounded px-2 py-1 text-xs ${style.textAlign === align ? 'bg-white/20 text-white' : 'bg-white/10 text-slate-300'}`}
|
||||
>
|
||||
{mode.label}
|
||||
{align === 'left' ? 'L' : align === 'center' ? 'C' : 'R'}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</section>
|
||||
|
||||
<section className="space-y-2">
|
||||
{sectionHeader('stems', 'STEMS')}
|
||||
{open.stems && (
|
||||
<div className="space-y-2">
|
||||
{stems?.length ? stems.map((stem) => (
|
||||
<div key={stem.name} className="space-y-1.5">
|
||||
<div className="flex items-center gap-2">
|
||||
<span className="text-xs font-medium text-slate-300 flex-1 truncate">{stem.label}</span>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => toggleStemMute(stem.name)}
|
||||
className={`w-6 h-6 text-[10px] rounded ${mutedStems[stem.name] ? 'bg-red-500 text-white' : 'bg-white/10 text-slate-300'}`}
|
||||
>
|
||||
M
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => toggleStemSolo(stem.name)}
|
||||
className={`w-6 h-6 text-[10px] rounded ${soloedStems[stem.name] ? 'bg-yellow-300 text-black' : 'bg-white/10 text-slate-300'}`}
|
||||
>
|
||||
S
|
||||
</button>
|
||||
</div>
|
||||
<div className="flex items-center gap-2">
|
||||
<input
|
||||
type="range"
|
||||
min={0}
|
||||
max={2}
|
||||
step={0.05}
|
||||
value={gains[stem.name] ?? 1}
|
||||
onChange={(event) => setStemGain(stem.name, parseFloat(event.target.value))}
|
||||
className="w-full h-1.5 accent-brand-500"
|
||||
/>
|
||||
<span className="text-[10px] tabular-nums text-slate-400 w-11 text-right">{Math.round((gains[stem.name] ?? 1) * 100)}%</span>
|
||||
</div>
|
||||
</div>
|
||||
)) : (
|
||||
<p className="text-[11px] text-slate-400">No stems — single track mode</p>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
</section>
|
||||
|
||||
<section className="space-y-2">
|
||||
{sectionHeader('lyrics', 'LYRICS')}
|
||||
{open.lyrics && (
|
||||
<div className="space-y-2.5">
|
||||
<div className="flex items-center gap-2">
|
||||
<select
|
||||
value={style.fontFamily}
|
||||
onChange={(event) => patchStyle({ fontFamily: event.target.value })}
|
||||
className="flex-1 text-xs bg-white/10 border border-white/15 rounded px-2 py-1"
|
||||
<div className="flex items-center gap-2">
|
||||
{(['top', 'center', 'bottom'] as const).map((pos) => (
|
||||
<button key={pos} type="button" onClick={() => patchStyle({ activeLineVerticalAlign: pos })}
|
||||
className={`flex-1 rounded px-2 py-1 text-xs ${style.activeLineVerticalAlign === pos ? 'bg-white/20 text-white' : 'bg-white/10 text-slate-300'}`}
|
||||
>
|
||||
{FONT_OPTIONS.map((font) => (
|
||||
<option key={font.value} value={font.value} style={{ background: '#0f172a', color: '#e2e8f0', fontFamily: font.value }}>
|
||||
{font.label}
|
||||
</option>
|
||||
))}
|
||||
</select>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => patchStyle({ fontBold: !style.fontBold })}
|
||||
className={`px-2 py-1 rounded text-xs font-semibold ${style.fontBold ? 'bg-white/25 text-white' : 'bg-white/10 text-slate-300'}`}
|
||||
>
|
||||
B
|
||||
{pos === 'top' ? 'T' : pos === 'center' ? 'C' : 'B'}
|
||||
</button>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div className="space-y-1">
|
||||
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Size</span><span>{style.fontSize}</span></div>
|
||||
<input type="range" min={12} max={56} step={1} value={style.fontSize} onChange={(event) => patchStyle({ fontSize: parseInt(event.target.value, 10) })} className="w-full h-1.5 accent-brand-500" />
|
||||
</div>
|
||||
<div className="space-y-1">
|
||||
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Line spacing</span><span>{style.lineSpacing.toFixed(2)}</span></div>
|
||||
<input type="range" min={1} max={2.5} step={0.05} value={style.lineSpacing} onChange={(e) => patchStyle({ lineSpacing: parseFloat(e.target.value) })} className="w-full h-1.5 accent-brand-500" />
|
||||
</div>
|
||||
|
||||
<div className="flex items-center gap-2">
|
||||
{(['left', 'center', 'right'] as const).map((align) => (
|
||||
<button
|
||||
key={align}
|
||||
type="button"
|
||||
onClick={() => patchStyle({ textAlign: align })}
|
||||
className={`flex-1 rounded px-2 py-1 text-xs ${style.textAlign === align ? 'bg-white/20 text-white' : 'bg-white/10 text-slate-300'}`}
|
||||
<div className="grid grid-cols-3 gap-2 text-[10px] text-slate-300">
|
||||
<label className="space-y-1"><span className="block">Text</span><input type="color" value={style.textColor} onChange={(e) => patchStyle({ textColor: e.target.value })} className="w-full h-7 rounded border border-white/15 bg-transparent" /></label>
|
||||
<label className="space-y-1"><span className="block">Highlight</span><input type="color" value={style.highlightColor} onChange={(e) => patchStyle({ highlightColor: e.target.value })} className="w-full h-7 rounded border border-white/15 bg-transparent" /></label>
|
||||
<label className="space-y-1"><span className="block">BG</span><input type="color" value={style.bgColor} onChange={(e) => patchStyle({ bgColor: e.target.value })} className="w-full h-7 rounded border border-white/15 bg-transparent" /></label>
|
||||
</div>
|
||||
|
||||
<div className="space-y-1">
|
||||
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>X offset</span><span>{activeOffset.x}</span></div>
|
||||
<input type="range" min={-50} max={50} step={1} value={activeOffset.x} onChange={(e) => patchOffset('x', parseInt(e.target.value, 10))} className="w-full h-1.5 accent-brand-500" />
|
||||
</div>
|
||||
<div className="space-y-1">
|
||||
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Y offset</span><span>{activeOffset.y}</span></div>
|
||||
<input type="range" min={-50} max={50} step={1} value={activeOffset.y} onChange={(e) => patchOffset('y', parseInt(e.target.value, 10))} className="w-full h-1.5 accent-brand-500" />
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
{/* ── VISUALIZER ── */}
|
||||
<section className="space-y-2">
|
||||
<div className="text-xs font-semibold tracking-wide text-slate-200">VISUALIZER</div>
|
||||
<div className="space-y-2.5">
|
||||
|
||||
{/* Quality */}
|
||||
<div className="space-y-1.5">
|
||||
<div className="text-[10px] text-slate-400">Quality</div>
|
||||
<div className="grid grid-cols-3 gap-1.5">
|
||||
{(['low', 'balanced', 'high'] as const).map((q) => (
|
||||
<button key={q} type="button" onClick={() => patchVizSettings({ quality: q })}
|
||||
className={`rounded px-2 py-1 text-[11px] border ${vizSettings.quality === q ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
|
||||
>
|
||||
{align === 'left' ? 'L' : align === 'center' ? 'C' : 'R'}
|
||||
{q === 'balanced' ? 'Balanced' : q === 'low' ? 'Low' : 'High'}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="flex items-center gap-2">
|
||||
{(['top', 'center', 'bottom'] as const).map((pos) => (
|
||||
<button
|
||||
key={pos}
|
||||
type="button"
|
||||
onClick={() => patchStyle({ activeLineVerticalAlign: pos })}
|
||||
className={`flex-1 rounded px-2 py-1 text-xs ${style.activeLineVerticalAlign === pos ? 'bg-white/20 text-white' : 'bg-white/10 text-slate-300'}`}
|
||||
{/* FPS cap */}
|
||||
<div className="space-y-1.5">
|
||||
<div className="text-[10px] text-slate-400">FPS cap</div>
|
||||
<div className="grid grid-cols-3 gap-1.5">
|
||||
{([30, 60, 120] as const).map((fps) => (
|
||||
<button key={fps} type="button" onClick={() => patchVizSettings({ fps_cap: fps })}
|
||||
className={`rounded px-2 py-1 text-[11px] border ${vizSettings.fps_cap === fps ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
|
||||
>
|
||||
{pos === 'top' ? 'T' : pos === 'center' ? 'C' : 'B'}
|
||||
{fps}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
|
||||
<div className="space-y-1">
|
||||
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Line spacing</span><span>{style.lineSpacing.toFixed(2)}</span></div>
|
||||
<input type="range" min={1} max={2.5} step={0.05} value={style.lineSpacing} onChange={(event) => patchStyle({ lineSpacing: parseFloat(event.target.value) })} className="w-full h-1.5 accent-brand-500" />
|
||||
</div>
|
||||
|
||||
<div className="grid grid-cols-3 gap-2 text-[10px] text-slate-300">
|
||||
<label className="space-y-1"><span className="block">Text</span><input type="color" value={style.textColor} onChange={(event) => patchStyle({ textColor: event.target.value })} className="w-full h-7 rounded border border-white/15 bg-transparent" /></label>
|
||||
<label className="space-y-1"><span className="block">Highlight</span><input type="color" value={style.highlightColor} onChange={(event) => patchStyle({ highlightColor: event.target.value })} className="w-full h-7 rounded border border-white/15 bg-transparent" /></label>
|
||||
<label className="space-y-1"><span className="block">BG</span><input type="color" value={style.bgColor} onChange={(event) => patchStyle({ bgColor: event.target.value })} className="w-full h-7 rounded border border-white/15 bg-transparent" /></label>
|
||||
</div>
|
||||
|
||||
<div className="space-y-1">
|
||||
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>X offset</span><span>{activeOffset.x}</span></div>
|
||||
<input type="range" min={-50} max={50} step={1} value={activeOffset.x} onChange={(event) => patchOffset('x', parseInt(event.target.value, 10))} className="w-full h-1.5 accent-brand-500" />
|
||||
</div>
|
||||
<div className="space-y-1">
|
||||
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Y offset</span><span>{activeOffset.y}</span></div>
|
||||
<input type="range" min={-50} max={50} step={1} value={activeOffset.y} onChange={(event) => patchOffset('y', parseInt(event.target.value, 10))} className="w-full h-1.5 accent-brand-500" />
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</section>
|
||||
|
||||
<section className="space-y-2">
|
||||
{sectionHeader('visualizer', 'VISUALIZER')}
|
||||
{open.visualizer && (
|
||||
<div className="space-y-2.5">
|
||||
{!isVisualizerMode && (
|
||||
<p className="text-[11px] text-slate-400">
|
||||
Switch background to Muon or Fluid to access mode-specific controls.
|
||||
</p>
|
||||
)}
|
||||
|
||||
{/* ── Muon controls ── */}
|
||||
{visualStyle === 'muon' && (
|
||||
<>
|
||||
{/* Preset pills */}
|
||||
<div className="space-y-1.5">
|
||||
<div className="text-[10px] text-slate-400">Quality</div>
|
||||
<div className="grid grid-cols-3 gap-1.5">
|
||||
{(['low', 'balanced', 'high'] as const).map((quality) => (
|
||||
<button
|
||||
key={quality}
|
||||
type="button"
|
||||
onClick={() => patchVizSettings({ quality })}
|
||||
className={`rounded px-2 py-1 text-[11px] border ${vizSettings.quality === quality ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
|
||||
>
|
||||
{quality === 'balanced' ? 'Balanced' : quality === 'low' ? 'Low' : 'High'}
|
||||
</button>
|
||||
))}
|
||||
<div className="text-[10px] text-slate-400">Preset</div>
|
||||
<div className="flex flex-wrap gap-1.5">
|
||||
{MUON_PRESETS.map((preset) => {
|
||||
const isActive = activeMuonPreset?.label === preset.label
|
||||
return (
|
||||
<button
|
||||
key={preset.label}
|
||||
type="button"
|
||||
onClick={() => patchVizSettings({ muon: preset.values })}
|
||||
className={`rounded px-2 py-1 text-[11px] border transition-colors ${isActive ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
|
||||
>
|
||||
{preset.label}
|
||||
</button>
|
||||
)
|
||||
})}
|
||||
{!activeMuonPreset && (
|
||||
<span className="rounded px-2 py-1 text-[11px] border border-amber-400/60 bg-amber-500/10 text-amber-300">
|
||||
Custom
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Sliders */}
|
||||
{slider('Radius', vizSettings.muon.radius_multiplier, formatMultiplier(vizSettings.muon.radius_multiplier), 0.25, 2, 0.05, (v) => patchVizSettings({ muon: { radius_multiplier: v } }))}
|
||||
{slider('Aperture', vizSettings.muon.aperture, formatMultiplier(vizSettings.muon.aperture), 0.25, 2, 0.05, (v) => patchVizSettings({ muon: { aperture: v } }))}
|
||||
{slider('Spacing', vizSettings.muon.spacing, formatMultiplier(vizSettings.muon.spacing), 0.25, 2, 0.05, (v) => patchVizSettings({ muon: { spacing: v } }))}
|
||||
{slider('Particles', vizSettings.muon.particle_count, String(vizSettings.muon.particle_count), 40, 500, 10, (v) => patchVizSettings({ muon: { particle_count: Math.round(v) } }))}
|
||||
{slider('Rotation', vizSettings.muon.rotation_speed, formatMultiplier(vizSettings.muon.rotation_speed), 0.2, 4, 0.1, (v) => patchVizSettings({ muon: { rotation_speed: v } }))}
|
||||
{slider('Glow', vizSettings.muon.glow_intensity, formatMultiplier(vizSettings.muon.glow_intensity), 0, 3, 0.1, (v) => patchVizSettings({ muon: { glow_intensity: v } }))}
|
||||
{slider('Size', vizSettings.muon.particle_size, formatMultiplier(vizSettings.muon.particle_size), 0.25, 3, 0.05, (v) => patchVizSettings({ muon: { particle_size: v } }))}
|
||||
{slider('Reactivity', vizSettings.muon.drum_reactivity, formatMultiplier(vizSettings.muon.drum_reactivity), 0, 2, 0.1, (v) => patchVizSettings({ muon: { drum_reactivity: v } }))}
|
||||
{slider('Hue range', vizSettings.muon.hue_range, formatMultiplier(vizSettings.muon.hue_range), 0, 3, 0.1, (v) => patchVizSettings({ muon: { hue_range: v } }))}
|
||||
{slider('Beat sensitivity', vizSettings.muon.beat_sensitivity, formatMultiplier(vizSettings.muon.beat_sensitivity), 0.2, 3, 0.1, (v) => patchVizSettings({ muon: { beat_sensitivity: v } }))}
|
||||
{slider('Orbit shape', vizSettings.muon.ellipse_ratio, vizSettings.muon.ellipse_ratio.toFixed(2), 0.3, 1, 0.02, (v) => patchVizSettings({ muon: { ellipse_ratio: v } }))}
|
||||
{slider('Idle speed', vizSettings.muon.idle_speed, formatMultiplier(vizSettings.muon.idle_speed), 0, 2, 0.1, (v) => patchVizSettings({ muon: { idle_speed: v } }))}
|
||||
{slider('Divisions', vizSettings.muon.divisions, String(vizSettings.muon.divisions), 8, 48, 1, (v) => patchVizSettings({ muon: { divisions: Math.round(v) } }))}
|
||||
{slider('Lifespan', vizSettings.muon.lifespan, String(vizSettings.muon.lifespan), 45, 420, 5, (v) => patchVizSettings({ muon: { lifespan: Math.round(v) } }))}
|
||||
{slider('Dynamic radius', vizSettings.muon.dynamic_radius, formatMultiplier(vizSettings.muon.dynamic_radius), 0.2, 3, 0.05, (v) => patchVizSettings({ muon: { dynamic_radius: v } }))}
|
||||
{slider('Delta gate', vizSettings.muon.delta_response_limit, vizSettings.muon.delta_response_limit.toFixed(3), 0.001, 0.05, 0.001, (v) => patchVizSettings({ muon: { delta_response_limit: v } }))}
|
||||
|
||||
<div className="space-y-1.5">
|
||||
<div className="text-[10px] text-slate-400">FPS cap</div>
|
||||
<div className="grid grid-cols-3 gap-1.5">
|
||||
{([30, 60, 120] as const).map((fps) => (
|
||||
<button
|
||||
key={fps}
|
||||
type="button"
|
||||
onClick={() => patchVizSettings({ fps_cap: fps })}
|
||||
className={`rounded px-2 py-1 text-[11px] border ${vizSettings.fps_cap === fps ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
|
||||
>
|
||||
{fps}
|
||||
</button>
|
||||
))}
|
||||
<div className="text-[10px] text-slate-400">Pattern preset</div>
|
||||
<div className="flex flex-wrap gap-1.5">
|
||||
{(['default', 'orbital', 'flower', 'spiro', 'hybrid'] as const).map((preset) => {
|
||||
const active = vizSettings.muon.visualization_preset === preset
|
||||
return (
|
||||
<button
|
||||
key={preset}
|
||||
type="button"
|
||||
onClick={() => patchVizSettings({ muon: { visualization_preset: preset } })}
|
||||
className={`rounded px-2 py-1 text-[11px] border transition-colors ${active ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
|
||||
>
|
||||
{preset}
|
||||
</button>
|
||||
)
|
||||
})}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{!isVisualizerMode && (
|
||||
<p className="text-[11px] text-slate-400">
|
||||
Switch background to Muon or Fluid to access mode-specific controls.
|
||||
</p>
|
||||
)}
|
||||
{/* Mono color toggle */}
|
||||
<label className="flex items-center gap-2 cursor-pointer">
|
||||
<div
|
||||
role="checkbox"
|
||||
aria-checked={vizSettings.muon.mono_color}
|
||||
tabIndex={0}
|
||||
onClick={() => patchVizSettings({ muon: { mono_color: !vizSettings.muon.mono_color } })}
|
||||
onKeyDown={(e) => { if (e.key === ' ' || e.key === 'Enter') patchVizSettings({ muon: { mono_color: !vizSettings.muon.mono_color } }) }}
|
||||
className={`w-8 h-4 rounded-full transition-colors ${vizSettings.muon.mono_color ? 'bg-violet-500' : 'bg-white/20'} relative`}
|
||||
>
|
||||
<div className={`absolute top-0.5 w-3 h-3 rounded-full bg-white transition-transform ${vizSettings.muon.mono_color ? 'translate-x-4' : 'translate-x-0.5'}`} />
|
||||
</div>
|
||||
<span className="text-[11px] text-slate-300">Mono color</span>
|
||||
</label>
|
||||
|
||||
{visualStyle === 'muon' && (
|
||||
<>
|
||||
<div className="space-y-1">
|
||||
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Radius</span><span>{formatMultiplier(vizSettings.muon.radius_multiplier)}</span></div>
|
||||
<input type="range" min={0.25} max={2} step={0.05} value={vizSettings.muon.radius_multiplier} onChange={(event) => patchVizSettings({ muon: { radius_multiplier: parseFloat(event.target.value) } })} className="w-full h-1.5 accent-brand-500" />
|
||||
</div>
|
||||
<div className="space-y-1">
|
||||
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Aperture</span><span>{formatMultiplier(vizSettings.muon.aperture)}</span></div>
|
||||
<input type="range" min={0.25} max={2} step={0.05} value={vizSettings.muon.aperture} onChange={(event) => patchVizSettings({ muon: { aperture: parseFloat(event.target.value) } })} className="w-full h-1.5 accent-brand-500" />
|
||||
</div>
|
||||
<div className="space-y-1">
|
||||
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Spacing</span><span>{formatMultiplier(vizSettings.muon.spacing)}</span></div>
|
||||
<input type="range" min={0.25} max={2} step={0.05} value={vizSettings.muon.spacing} onChange={(event) => patchVizSettings({ muon: { spacing: parseFloat(event.target.value) } })} className="w-full h-1.5 accent-brand-500" />
|
||||
</div>
|
||||
</>
|
||||
)}
|
||||
{/* Mirror toggle */}
|
||||
<label className="flex items-center gap-2 cursor-pointer">
|
||||
<div
|
||||
role="checkbox"
|
||||
aria-checked={vizSettings.muon.particle_mirror}
|
||||
tabIndex={0}
|
||||
onClick={() => patchVizSettings({ muon: { particle_mirror: !vizSettings.muon.particle_mirror } })}
|
||||
onKeyDown={(e) => { if (e.key === ' ' || e.key === 'Enter') patchVizSettings({ muon: { particle_mirror: !vizSettings.muon.particle_mirror } }) }}
|
||||
className={`w-8 h-4 rounded-full transition-colors ${vizSettings.muon.particle_mirror ? 'bg-violet-500' : 'bg-white/20'} relative`}
|
||||
>
|
||||
<div className={`absolute top-0.5 w-3 h-3 rounded-full bg-white transition-transform ${vizSettings.muon.particle_mirror ? 'translate-x-4' : 'translate-x-0.5'}`} />
|
||||
</div>
|
||||
<span className="text-[11px] text-slate-300">Mirror particles</span>
|
||||
</label>
|
||||
</>
|
||||
)}
|
||||
|
||||
{visualStyle === 'fluid' && (
|
||||
<>
|
||||
<div className="space-y-1">
|
||||
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Density</span><span>{formatMultiplier(vizSettings.fluid.spawn_density)}</span></div>
|
||||
<input type="range" min={0.25} max={2} step={0.05} value={vizSettings.fluid.spawn_density} onChange={(event) => patchVizSettings({ fluid: { spawn_density: parseFloat(event.target.value) } })} className="w-full h-1.5 accent-brand-500" />
|
||||
</div>
|
||||
<div className="space-y-1">
|
||||
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Force</span><span>{formatMultiplier(vizSettings.fluid.splat_force)}</span></div>
|
||||
<input type="range" min={0.25} max={2} step={0.05} value={vizSettings.fluid.splat_force} onChange={(event) => patchVizSettings({ fluid: { splat_force: parseFloat(event.target.value) } })} className="w-full h-1.5 accent-brand-500" />
|
||||
</div>
|
||||
<div className="space-y-1">
|
||||
<div className="flex items-center justify-between text-[10px] text-slate-400"><span>Trail decay</span><span>{formatMultiplier(vizSettings.fluid.decay)}</span></div>
|
||||
<input type="range" min={0.25} max={2} step={0.05} value={vizSettings.fluid.decay} onChange={(event) => patchVizSettings({ fluid: { decay: parseFloat(event.target.value) } })} className="w-full h-1.5 accent-brand-500" />
|
||||
</div>
|
||||
</>
|
||||
)}
|
||||
{/* ── Fluid controls ── */}
|
||||
{visualStyle === 'fluid' && (
|
||||
<>
|
||||
{/* Preset pills */}
|
||||
<div className="space-y-1.5">
|
||||
<div className="text-[10px] text-slate-400">Preset</div>
|
||||
<div className="flex flex-wrap gap-1.5">
|
||||
{FLUID_PRESETS.map((preset) => {
|
||||
const isActive = activeFluidPreset?.label === preset.label
|
||||
return (
|
||||
<button
|
||||
key={preset.label}
|
||||
type="button"
|
||||
onClick={() => patchVizSettings({ fluid: preset.values })}
|
||||
className={`rounded px-2 py-1 text-[11px] border transition-colors ${isActive ? 'border-violet-400 bg-violet-500/20 text-white' : 'border-white/15 bg-white/5 text-slate-300 hover:bg-white/10'}`}
|
||||
>
|
||||
{preset.label}
|
||||
</button>
|
||||
)
|
||||
})}
|
||||
{!activeFluidPreset && (
|
||||
<span className="rounded px-2 py-1 text-[11px] border border-amber-400/60 bg-amber-500/10 text-amber-300">
|
||||
Custom
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => patchVizSettings(manifestVizSettings ?? VIZ_DEFAULTS)}
|
||||
className="w-full rounded-md px-2 py-1.5 text-xs bg-white/10 hover:bg-white/20 text-slate-200"
|
||||
>
|
||||
{manifestVizSettings ? 'Reset to song defaults' : 'Reset to defaults'}
|
||||
</button>
|
||||
</div>
|
||||
)}
|
||||
{/* Sliders */}
|
||||
{slider('Density', vizSettings.fluid.spawn_density, formatMultiplier(vizSettings.fluid.spawn_density), 0.25, 3, 0.05, (v) => patchVizSettings({ fluid: { spawn_density: v } }))}
|
||||
{slider('Force', vizSettings.fluid.splat_force, formatMultiplier(vizSettings.fluid.splat_force), 0.25, 3, 0.05, (v) => patchVizSettings({ fluid: { splat_force: v } }))}
|
||||
{slider('Trail decay', vizSettings.fluid.decay, formatMultiplier(vizSettings.fluid.decay), 0.25, 3, 0.05, (v) => patchVizSettings({ fluid: { decay: v } }))}
|
||||
{slider('Trail length', vizSettings.fluid.background_persistence, formatMultiplier(vizSettings.fluid.background_persistence), 0.2, 3, 0.1, (v) => patchVizSettings({ fluid: { background_persistence: v } }))}
|
||||
{slider('Beat bursts', vizSettings.fluid.beat_burst_count, formatMultiplier(vizSettings.fluid.beat_burst_count), 0.25, 3, 0.05, (v) => patchVizSettings({ fluid: { beat_burst_count: v } }))}
|
||||
{slider('Treble pulse', vizSettings.fluid.treble_pulse, formatMultiplier(vizSettings.fluid.treble_pulse), 0, 3, 0.1, (v) => patchVizSettings({ fluid: { treble_pulse: v } }))}
|
||||
{slider('Drag', vizSettings.fluid.drag, formatMultiplier(vizSettings.fluid.drag), 0.5, 2, 0.05, (v) => patchVizSettings({ fluid: { drag: v } }))}
|
||||
{slider('Blob life', vizSettings.fluid.blob_life, formatMultiplier(vizSettings.fluid.blob_life), 0.25, 3, 0.05, (v) => patchVizSettings({ fluid: { blob_life: v } }))}
|
||||
{slider('Color speed', vizSettings.fluid.color_speed, formatMultiplier(vizSettings.fluid.color_speed), 0, 3, 0.1, (v) => patchVizSettings({ fluid: { color_speed: v } }))}
|
||||
</>
|
||||
)}
|
||||
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => patchVizSettings(manifestVizSettings ?? VIZ_DEFAULTS)}
|
||||
className="w-full rounded-md px-2 py-1.5 text-xs bg-white/10 hover:bg-white/20 text-slate-200"
|
||||
>
|
||||
{manifestVizSettings ? 'Reset to song defaults' : 'Reset to defaults'}
|
||||
</button>
|
||||
</div>
|
||||
</section>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -0,0 +1,304 @@
|
||||
import { useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react'
|
||||
import { Disc3 } from 'lucide-react'
|
||||
import { useAudioFeatures } from '../hooks/useAudioFeatures'
|
||||
import { useVisualizerPalette } from '../hooks/useVisualizerPalette'
|
||||
import VisualizerBackdrop, { type VisualizerMode } from './visualizers/VisualizerBackdrop'
|
||||
import { registerPlayback, unregisterPlayback } from '../utils/audioRegistry'
|
||||
|
||||
interface VisualizerPreviewWidgetProps {
|
||||
audioUrl: string
|
||||
coverArtUrl?: string
|
||||
videoUrl?: string
|
||||
songTitle?: string
|
||||
className?: string
|
||||
}
|
||||
|
||||
type PreviewMode = 'muon' | 'fluid' | 'video'
|
||||
|
||||
function isDirectVideoUrl(input: string): boolean {
|
||||
return /^(https?:\/\/).+\.(mp4|webm|m4v|ogg|ogv)(?:[?#].*)?$/i.test(input)
|
||||
}
|
||||
|
||||
export default function VisualizerPreviewWidget({
|
||||
audioUrl,
|
||||
coverArtUrl,
|
||||
videoUrl,
|
||||
songTitle,
|
||||
className,
|
||||
}: VisualizerPreviewWidgetProps) {
|
||||
const containerRef = useRef<HTMLDivElement>(null)
|
||||
const audioRef = useRef<HTMLAudioElement>(null)
|
||||
const videoRef = useRef<HTMLVideoElement>(null)
|
||||
const audioCtxRef = useRef<AudioContext | null>(null)
|
||||
const isPlayingRef = useRef(false)
|
||||
|
||||
const [dims, setDims] = useState<{ w: number; h: number } | null>(null)
|
||||
const [mode, setMode] = useState<PreviewMode>('muon')
|
||||
const [isPlaying, setIsPlaying] = useState(false)
|
||||
const [ctxVersion, setCtxVersion] = useState(0)
|
||||
|
||||
const normalizedVideoUrl = useMemo(() => String(videoUrl || '').trim(), [videoUrl])
|
||||
const canPreviewVideo = useMemo(() => isDirectVideoUrl(normalizedVideoUrl), [normalizedVideoUrl])
|
||||
|
||||
const paletteRef = useVisualizerPalette({ coverArtUrl })
|
||||
const featuresRef = useAudioFeatures({
|
||||
audioCtxRef,
|
||||
analysisAudioRef: audioRef,
|
||||
isPlayingRef,
|
||||
enabled: true,
|
||||
analysisSourceKey: `${audioUrl}|${ctxVersion}`,
|
||||
})
|
||||
|
||||
const tabs = useMemo(
|
||||
() => [
|
||||
{ id: 'muon' as const, label: 'Muon' },
|
||||
{ id: 'fluid' as const, label: 'Fluid' },
|
||||
...(canPreviewVideo ? [{ id: 'video' as const, label: 'Video' }] : []),
|
||||
],
|
||||
[canPreviewVideo],
|
||||
)
|
||||
|
||||
const ensureCtx = () => {
|
||||
if (!audioCtxRef.current || audioCtxRef.current.state === 'closed') {
|
||||
audioCtxRef.current = new AudioContext()
|
||||
setCtxVersion((prev) => prev + 1)
|
||||
}
|
||||
return audioCtxRef.current
|
||||
}
|
||||
|
||||
const stopPlayback = useCallback(() => {
|
||||
const audio = audioRef.current
|
||||
if (!audio) return
|
||||
audio.pause()
|
||||
setIsPlaying(false)
|
||||
}, [])
|
||||
|
||||
useEffect(() => {
|
||||
isPlayingRef.current = isPlaying
|
||||
}, [isPlaying])
|
||||
|
||||
useLayoutEffect(() => {
|
||||
const el = containerRef.current
|
||||
if (!el) return
|
||||
|
||||
const measure = () => {
|
||||
const { width } = el.getBoundingClientRect()
|
||||
if (width <= 0) return
|
||||
const next = {
|
||||
w: Math.round(width),
|
||||
h: Math.round(width * (9 / 16)),
|
||||
}
|
||||
setDims((prev) => (prev && prev.w === next.w && prev.h === next.h ? prev : next))
|
||||
}
|
||||
|
||||
measure()
|
||||
|
||||
if (typeof ResizeObserver === 'undefined') {
|
||||
window.addEventListener('resize', measure)
|
||||
return () => window.removeEventListener('resize', measure)
|
||||
}
|
||||
|
||||
const ro = new ResizeObserver(measure)
|
||||
ro.observe(el)
|
||||
return () => ro.disconnect()
|
||||
}, [])
|
||||
|
||||
useEffect(() => {
|
||||
const audio = audioRef.current
|
||||
if (!audio) return
|
||||
|
||||
const onPlay = () => setIsPlaying(true)
|
||||
const onPause = () => setIsPlaying(false)
|
||||
const onEnded = () => setIsPlaying(false)
|
||||
|
||||
audio.addEventListener('play', onPlay)
|
||||
audio.addEventListener('pause', onPause)
|
||||
audio.addEventListener('ended', onEnded)
|
||||
|
||||
return () => {
|
||||
audio.removeEventListener('play', onPlay)
|
||||
audio.removeEventListener('pause', onPause)
|
||||
audio.removeEventListener('ended', onEnded)
|
||||
}
|
||||
}, [])
|
||||
|
||||
useEffect(() => {
|
||||
setIsPlaying(false)
|
||||
isPlayingRef.current = false
|
||||
audioRef.current?.pause()
|
||||
const video = videoRef.current
|
||||
if (video) {
|
||||
video.pause()
|
||||
try {
|
||||
video.currentTime = 0
|
||||
} catch {
|
||||
// ignore
|
||||
}
|
||||
}
|
||||
}, [audioUrl])
|
||||
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
unregisterPlayback(stopPlayback)
|
||||
}
|
||||
}, [stopPlayback])
|
||||
|
||||
useEffect(() => {
|
||||
if (mode === 'video' && !canPreviewVideo) setMode('muon')
|
||||
}, [mode, canPreviewVideo])
|
||||
|
||||
useEffect(() => {
|
||||
const onVisible = () => {
|
||||
if (document.hidden) return
|
||||
if (isPlayingRef.current && audioCtxRef.current?.state === 'suspended') {
|
||||
void audioCtxRef.current.resume().catch(() => {})
|
||||
}
|
||||
}
|
||||
document.addEventListener('visibilitychange', onVisible)
|
||||
return () => document.removeEventListener('visibilitychange', onVisible)
|
||||
}, [])
|
||||
|
||||
useEffect(() => {
|
||||
const video = videoRef.current
|
||||
const audio = audioRef.current
|
||||
if (!video) return
|
||||
|
||||
if (mode !== 'video') {
|
||||
video.pause()
|
||||
return
|
||||
}
|
||||
|
||||
if (audio && Math.abs(video.currentTime - audio.currentTime) > 0.15) {
|
||||
try {
|
||||
video.currentTime = audio.currentTime
|
||||
} catch {
|
||||
// ignore
|
||||
}
|
||||
}
|
||||
|
||||
if (isPlaying) {
|
||||
void video.play().catch(() => {})
|
||||
} else {
|
||||
video.pause()
|
||||
}
|
||||
}, [isPlaying, mode])
|
||||
|
||||
useEffect(() => {
|
||||
if (mode !== 'video' || !isPlaying) return
|
||||
|
||||
const timer = window.setInterval(() => {
|
||||
const video = videoRef.current
|
||||
const audio = audioRef.current
|
||||
if (!video || !audio) return
|
||||
if (Math.abs(video.currentTime - audio.currentTime) > 0.25) {
|
||||
try {
|
||||
video.currentTime = audio.currentTime
|
||||
} catch {
|
||||
// ignore
|
||||
}
|
||||
}
|
||||
}, 500)
|
||||
|
||||
return () => window.clearInterval(timer)
|
||||
}, [isPlaying, mode])
|
||||
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
const ctx = audioCtxRef.current
|
||||
if (ctx && ctx.state !== 'closed') {
|
||||
void ctx.close().catch(() => {})
|
||||
}
|
||||
audioCtxRef.current = null
|
||||
}
|
||||
}, [])
|
||||
|
||||
const togglePlay = async () => {
|
||||
const audio = audioRef.current
|
||||
if (!audio) return
|
||||
|
||||
if (isPlayingRef.current) {
|
||||
audio.pause()
|
||||
return
|
||||
}
|
||||
|
||||
const ctx = ensureCtx()
|
||||
if (ctx.state === 'suspended') {
|
||||
await ctx.resume().catch(() => {})
|
||||
}
|
||||
|
||||
try {
|
||||
registerPlayback(stopPlayback)
|
||||
await audio.play()
|
||||
setIsPlaying(true)
|
||||
} catch {
|
||||
setIsPlaying(false)
|
||||
}
|
||||
}
|
||||
|
||||
const visualizerMode: VisualizerMode = mode === 'video' ? 'muon' : mode
|
||||
|
||||
return (
|
||||
<div className={className}>
|
||||
<audio ref={audioRef} src={audioUrl} crossOrigin="anonymous" preload="metadata" />
|
||||
|
||||
<div ref={containerRef} className="relative w-full overflow-hidden rounded-t-lg bg-gray-900" style={{ aspectRatio: '16 / 9' }}>
|
||||
{mode === 'video' && canPreviewVideo ? (
|
||||
<video
|
||||
ref={videoRef}
|
||||
src={normalizedVideoUrl}
|
||||
className="absolute inset-0 h-full w-full object-cover"
|
||||
loop
|
||||
muted
|
||||
playsInline
|
||||
/>
|
||||
) : dims ? (
|
||||
<VisualizerBackdrop
|
||||
mode={visualizerMode}
|
||||
featuresRef={featuresRef}
|
||||
paletteRef={paletteRef}
|
||||
width={dims.w}
|
||||
height={dims.h}
|
||||
/>
|
||||
) : (
|
||||
<div className="absolute inset-0 flex items-center justify-center">
|
||||
<Disc3 className="h-8 w-8 animate-spin text-gray-600" />
|
||||
</div>
|
||||
)}
|
||||
|
||||
{!isPlaying && songTitle ? (
|
||||
<div className="pointer-events-none absolute inset-0 flex items-end p-2">
|
||||
<div className="max-w-full truncate rounded bg-black/45 px-2 py-1 text-xs text-gray-100">{songTitle}</div>
|
||||
</div>
|
||||
) : null}
|
||||
</div>
|
||||
|
||||
<div className="flex items-center border-t border-white/10 bg-gray-950/90" role="tablist" aria-label="Visualizer mode">
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => void togglePlay()}
|
||||
aria-label={isPlaying ? 'Pause preview' : 'Play preview'}
|
||||
className="px-3 py-1.5 text-sm text-gray-200 hover:bg-white/10"
|
||||
>
|
||||
{isPlaying ? '⏸' : '▶'}
|
||||
</button>
|
||||
|
||||
{tabs.map((tab) => (
|
||||
<button
|
||||
key={tab.id}
|
||||
type="button"
|
||||
role="tab"
|
||||
aria-selected={mode === tab.id}
|
||||
onClick={() => setMode(tab.id)}
|
||||
className={`flex-1 py-1.5 text-xs ${mode === tab.id ? 'bg-brand-500/30 text-white' : 'text-gray-300 hover:bg-white/10'}`}
|
||||
>
|
||||
{tab.label}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
|
||||
{normalizedVideoUrl && !canPreviewVideo ? (
|
||||
<p className="mt-1 text-[10px] text-amber-300">Video preview is available only for direct media URLs (mp4/webm/m4v/ogg).</p>
|
||||
) : null}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -2,79 +2,151 @@ import { useEffect, useRef } from 'react'
|
||||
import type { MutableRefObject } from 'react'
|
||||
import type { AudioFeatures } from '../../hooks/useAudioFeatures'
|
||||
import type { VisualizerPalette } from '../../hooks/useVisualizerPalette'
|
||||
import type { VisualizerSettings } from './VisualizerBackdrop'
|
||||
import { destroyFluidSim, initFluidSim, splatFluid, updateFluidConfig } from './fluid/fluidSim'
|
||||
|
||||
type FluidSettings = NonNullable<VisualizerSettings['fluid']>
|
||||
|
||||
interface FluidVisualizerCanvasProps {
|
||||
featuresRef: MutableRefObject<AudioFeatures>
|
||||
paletteRef: MutableRefObject<VisualizerPalette>
|
||||
width: number
|
||||
height: number
|
||||
fpsCap?: 30 | 60 | 120
|
||||
quality?: 'low' | 'balanced' | 'high'
|
||||
settings?: {
|
||||
spawn_density?: number
|
||||
splat_force?: number
|
||||
decay?: number
|
||||
}
|
||||
}
|
||||
|
||||
type Blob = {
|
||||
x: number
|
||||
y: number
|
||||
dx: number
|
||||
dy: number
|
||||
radius: number
|
||||
life: number
|
||||
color: string
|
||||
settings?: FluidSettings
|
||||
fpsCap?: number
|
||||
quality?: string
|
||||
}
|
||||
|
||||
function clamp01(v: number): number {
|
||||
return Math.max(0, Math.min(1, v))
|
||||
}
|
||||
|
||||
export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width, height, fpsCap = 60, quality = 'balanced', settings }: FluidVisualizerCanvasProps) {
|
||||
function clamp(v: number, lo: number, hi: number): number {
|
||||
return Math.max(lo, Math.min(hi, v))
|
||||
}
|
||||
|
||||
function hexToRgb(hex: string): { r: number; g: number; b: number } {
|
||||
const m = /^#?([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})$/i.exec(hex.trim())
|
||||
if (!m) return { r: 0.5, g: 0.5, b: 0.8 }
|
||||
return {
|
||||
r: parseInt(m[1], 16) / 255,
|
||||
g: parseInt(m[2], 16) / 255,
|
||||
b: parseInt(m[3], 16) / 255,
|
||||
}
|
||||
}
|
||||
|
||||
function fluidDensityDissipation(s?: FluidSettings, kick = 0): number {
|
||||
const base = 0.98 - (s?.background_persistence ?? 1) * 0.025
|
||||
return clamp(base - kick * 0.04 * (s?.decay ?? 1), 0.85, 0.999)
|
||||
}
|
||||
|
||||
function fluidVelocityDissipation(s?: FluidSettings, rms = 0): number {
|
||||
const base = 0.96 - (1 / Math.max(0.5, s?.drag ?? 1)) * 0.06
|
||||
return clamp(base - rms * 0.04, 0.80, 0.999)
|
||||
}
|
||||
|
||||
function fluidCurl(s?: FluidSettings, treble = 0, dropBoost = 0): number {
|
||||
const base = 15 + (s?.splat_force ?? 1) * 12
|
||||
return clamp(base + treble * 8 + dropBoost * 20, 0, 80)
|
||||
}
|
||||
|
||||
function fluidSplatRadius(s?: FluidSettings): number {
|
||||
return clamp(0.18 + (s?.blob_life ?? 1) * 0.06, 0.05, 0.5)
|
||||
}
|
||||
|
||||
function fluidBloomIntensity(s?: FluidSettings, intensity = 0, dropBoost = 0): number {
|
||||
const base = 0.55 + Math.max(0, s?.treble_pulse ?? 1) * 0.28
|
||||
return clamp(base + intensity * 1.55 + dropBoost * 0.85, 0.35, 3.0)
|
||||
}
|
||||
|
||||
function fluidBloomThreshold(intensity = 0): number {
|
||||
// Louder frames lower threshold so more dye contributes to bloom.
|
||||
return clamp(0.68 - intensity * 0.28, 0.30, 0.80)
|
||||
}
|
||||
|
||||
export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width, height, settings, fpsCap, quality }: FluidVisualizerCanvasProps) {
|
||||
const canvasRef = useRef<HTMLCanvasElement>(null)
|
||||
// Stable ref so the render loop always reads the latest settings without restarting.
|
||||
const settingsRef = useRef<FluidSettings | undefined>(settings)
|
||||
settingsRef.current = settings
|
||||
|
||||
const fpsCapRef = useRef(fpsCap ?? 60)
|
||||
fpsCapRef.current = fpsCap ?? 60
|
||||
|
||||
const qualityRef = useRef(quality ?? 'balanced')
|
||||
qualityRef.current = quality ?? 'balanced'
|
||||
|
||||
useEffect(() => {
|
||||
const canvas = canvasRef.current
|
||||
if (!canvas) return
|
||||
const ctx = canvas.getContext('2d')
|
||||
if (!ctx) return
|
||||
const supportsWebGL = !!(canvas.getContext('webgl2') || canvas.getContext('webgl'))
|
||||
|
||||
if (!supportsWebGL) {
|
||||
const ctx = canvas.getContext('2d')
|
||||
if (!ctx) return
|
||||
|
||||
let rafFallback = 0
|
||||
const renderFallback = () => {
|
||||
rafFallback = window.requestAnimationFrame(renderFallback)
|
||||
const w = Math.max(1, Math.round(canvas.clientWidth || 0), Math.round(width || 0))
|
||||
const h = Math.max(1, Math.round(canvas.clientHeight || 0), Math.round(height || 0))
|
||||
if (canvas.width !== w || canvas.height !== h) {
|
||||
canvas.width = w
|
||||
canvas.height = h
|
||||
}
|
||||
ctx.fillStyle = paletteRef.current.backgroundColor
|
||||
ctx.fillRect(0, 0, w, h)
|
||||
}
|
||||
|
||||
rafFallback = window.requestAnimationFrame(renderFallback)
|
||||
return () => window.cancelAnimationFrame(rafFallback)
|
||||
}
|
||||
|
||||
const s0 = settingsRef.current
|
||||
const q = qualityRef.current
|
||||
initFluidSim(canvas, {
|
||||
TRIGGER: 'hover',
|
||||
IMMEDIATE: false,
|
||||
AUTO: false,
|
||||
COLORFUL: false,
|
||||
SIM_RESOLUTION: q === 'high' ? 160 : q === 'low' ? 96 : 128,
|
||||
DYE_RESOLUTION: q === 'high' ? 1024 : q === 'low' ? 512 : 768,
|
||||
DENSITY_DISSIPATION: fluidDensityDissipation(s0),
|
||||
VELOCITY_DISSIPATION: fluidVelocityDissipation(s0),
|
||||
PRESSURE: 0.8,
|
||||
PRESSURE_ITERATIONS: 20,
|
||||
CURL: fluidCurl(s0),
|
||||
SPLAT_RADIUS: fluidSplatRadius(s0),
|
||||
SPLAT_FORCE: 6000,
|
||||
SHADING: true,
|
||||
BLOOM: true,
|
||||
BLOOM_INTENSITY: 0.8,
|
||||
SUNRAYS: false,
|
||||
BACK_COLOR: hexToRgb(paletteRef.current.backgroundColor),
|
||||
})
|
||||
|
||||
const blobs: Blob[] = []
|
||||
let raf = 0
|
||||
let colorIndex = 0
|
||||
let colorOffset = 0
|
||||
let lastSplatAt = 0
|
||||
let lastFrameAt = 0
|
||||
const cap = fpsCap === 30 || fpsCap === 120 ? fpsCap : 60
|
||||
const minFrameMs = 1000 / cap
|
||||
|
||||
const nextPaletteColor = () => {
|
||||
const nextPaletteColor = (speed: number) => {
|
||||
const palette = paletteRef.current.speakerPalette
|
||||
if (palette.length === 0) return paletteRef.current.highlightColor
|
||||
const c = palette[colorIndex % palette.length]
|
||||
colorIndex += 1
|
||||
return c
|
||||
colorOffset += Math.max(0, speed)
|
||||
return palette[Math.floor(colorOffset) % palette.length]
|
||||
}
|
||||
|
||||
const spawnBlob = (strength: number) => {
|
||||
const w = Math.max(1, width)
|
||||
const h = Math.max(1, height)
|
||||
const forceScale = Math.max(0.2, Number(settings?.splat_force ?? 1))
|
||||
const angle = Math.random() * Math.PI * 2
|
||||
const speed = (0.2 + strength * 2.2) * forceScale
|
||||
blobs.push({
|
||||
x: Math.random() * w,
|
||||
y: Math.random() * h,
|
||||
dx: Math.cos(angle) * speed,
|
||||
dy: Math.sin(angle) * speed,
|
||||
radius: (8 + strength * 38) * Math.max(0.25, Math.sqrt(forceScale)),
|
||||
life: 0.7 + strength * 0.6,
|
||||
color: nextPaletteColor(),
|
||||
})
|
||||
}
|
||||
let prevBeatPhase = 0
|
||||
let dropBoost = 0
|
||||
let lastDropAt = 0
|
||||
|
||||
const render = (ts: number) => {
|
||||
raf = window.requestAnimationFrame(render)
|
||||
if (lastFrameAt && ts - lastFrameAt < minFrameMs) return
|
||||
|
||||
// FPS cap
|
||||
const minFrameMs = 1000 / Math.max(1, fpsCapRef.current)
|
||||
if (ts - lastFrameAt < minFrameMs - 1) return
|
||||
lastFrameAt = ts
|
||||
|
||||
const w = Math.max(1, Math.round(canvas.clientWidth || 0), Math.round(width || 0))
|
||||
@@ -85,70 +157,115 @@ export default function FluidVisualizerCanvas({ featuresRef, paletteRef, width,
|
||||
}
|
||||
|
||||
const f = featuresRef.current
|
||||
const p = paletteRef.current
|
||||
const bass = clamp01(f.bass)
|
||||
const rms = clamp01(f.rms)
|
||||
const s = settingsRef.current
|
||||
const isLow = qualityRef.current === 'low'
|
||||
|
||||
const bass = clamp01(f.bass)
|
||||
const sub = clamp01(f.frequencyBands.sub)
|
||||
const rms = clamp01(f.rms)
|
||||
const treble = clamp01(f.treble)
|
||||
const idle = 0.5 + 0.5 * Math.sin(performance.now() * 0.0011)
|
||||
const densityScale = Math.max(0.2, Number(settings?.spawn_density ?? 1))
|
||||
const decayScale = Math.max(0.2, Number(settings?.decay ?? 1))
|
||||
const idle = 0.5 + 0.5 * Math.sin(performance.now() * 0.0011)
|
||||
const beatPhase = clamp01(f.beatInfo.beatPhase)
|
||||
const phasePulse = Math.pow(Math.max(0, Math.sin(beatPhase * Math.PI)), 1.8)
|
||||
const phaseWrapped = beatPhase < prevBeatPhase - 0.35
|
||||
const kick = clamp01(sub * 0.72 + bass * 0.28)
|
||||
const audioIntensity = clamp01(kick * 0.46 + rms * 0.34 + treble * 0.20)
|
||||
|
||||
// Ambient motion from RMS + small idle baseline so the renderer is never visually flat.
|
||||
const ambientSpawnScale = quality === 'low' ? 0.7 : 1
|
||||
if (Math.random() < (0.04 + rms * 0.31) * densityScale * ambientSpawnScale) spawnBlob(0.22 + rms * 0.58 + idle * 0.08)
|
||||
const dropTriggered = f.patternInfo.isBeatDrop && ts - lastDropAt > 500
|
||||
if (dropTriggered) {
|
||||
dropBoost = 1
|
||||
lastDropAt = ts
|
||||
}
|
||||
dropBoost *= 0.96
|
||||
|
||||
// Beat burst from bass peaks.
|
||||
if (f.beatDetected) {
|
||||
const baseCount = 3 + Math.floor(bass * 4)
|
||||
const count = quality === 'low' ? Math.max(1, baseCount - 1) : baseCount
|
||||
for (let i = 0; i < count; i += 1) spawnBlob(0.6 + bass * 0.8)
|
||||
const densityScale = Math.max(0.2, s?.spawn_density ?? 1)
|
||||
const splatForce = Math.max(0.2, s?.splat_force ?? 1)
|
||||
const beatBurstCount = Math.max(0, s?.beat_burst_count ?? 1)
|
||||
const treblePulse = Math.max(0, s?.treble_pulse ?? 1)
|
||||
const colorSpeed = Math.max(0, s?.color_speed ?? 1)
|
||||
|
||||
updateFluidConfig(canvas, {
|
||||
DENSITY_DISSIPATION: fluidDensityDissipation(s, kick),
|
||||
VELOCITY_DISSIPATION: fluidVelocityDissipation(s, rms),
|
||||
CURL: fluidCurl(s, treble, dropBoost),
|
||||
SPLAT_RADIUS: fluidSplatRadius(s),
|
||||
BLOOM_INTENSITY: fluidBloomIntensity(s, audioIntensity, dropBoost),
|
||||
BLOOM_THRESHOLD: fluidBloomThreshold(audioIntensity),
|
||||
BACK_COLOR: hexToRgb(paletteRef.current.backgroundColor),
|
||||
PAUSED: false,
|
||||
})
|
||||
|
||||
// WebGL sim renders in its own RAF; fpsCap throttles splat injection only.
|
||||
const minSplatMs = Math.max(minFrameMs, isLow ? 45 : 25)
|
||||
if (ts - lastSplatAt < minSplatMs - 1) {
|
||||
prevBeatPhase = beatPhase
|
||||
return
|
||||
}
|
||||
lastSplatAt = ts
|
||||
|
||||
// Ambient splat.
|
||||
if (Math.random() < (0.04 + rms * 0.35 + phasePulse * 0.08 + idle * 0.04) * densityScale) {
|
||||
const angle = Math.random() * Math.PI * 2
|
||||
const force = (0.2 + rms * 0.8 + treble * 0.3 * treblePulse + idle * 0.12) * splatForce * 400
|
||||
splatFluid(
|
||||
canvas,
|
||||
Math.random(),
|
||||
Math.random(),
|
||||
Math.cos(angle) * force,
|
||||
Math.sin(angle) * force,
|
||||
hexToRgb(nextPaletteColor(colorSpeed * 0.25)),
|
||||
)
|
||||
}
|
||||
|
||||
ctx.fillStyle = p.backgroundColor
|
||||
ctx.globalAlpha = 0.20 + (1 - bass) * 0.16
|
||||
ctx.fillRect(0, 0, w, h)
|
||||
ctx.globalAlpha = 1
|
||||
|
||||
// Correction C fix: trail lifetime (decay) is driven by bass — loud kick/bass →
|
||||
// slower decay → longer-lived blobs (more persistent ink on loud beats).
|
||||
// Velocity drag is driven by rms — overall loudness → more momentum retained.
|
||||
// Original Integration Plan had these driven by rms uniformly, which made
|
||||
// trail length insensitive to bass and meant quiet treble-heavy passages
|
||||
// produced the same persistence as loud bass hits.
|
||||
const decay = (0.009 + (1 - bass) * 0.012) * decayScale // loud bass → 0.009 (long trail)
|
||||
const drag = 0.984 - rms * 0.010 // loud rms → more momentum
|
||||
|
||||
for (let i = blobs.length - 1; i >= 0; i -= 1) {
|
||||
const b = blobs[i]
|
||||
b.x += b.dx
|
||||
b.y += b.dy
|
||||
b.dx *= drag
|
||||
b.dy *= drag
|
||||
b.life -= decay
|
||||
|
||||
if (b.life <= 0) {
|
||||
blobs.splice(i, 1)
|
||||
continue
|
||||
// Beat burst.
|
||||
if (f.beatDetected || phaseWrapped || dropTriggered) {
|
||||
const count = Math.max(1, Math.round((3 + kick * 5 + dropBoost * 4) * beatBurstCount))
|
||||
for (let i = 0; i < count; i += 1) {
|
||||
const angle = Math.random() * Math.PI * 2
|
||||
const force = (0.6 + bass * 1.2 + dropBoost * 0.6) * splatForce * 800
|
||||
splatFluid(
|
||||
canvas,
|
||||
Math.random(),
|
||||
Math.random(),
|
||||
Math.cos(angle) * force,
|
||||
Math.sin(angle) * force,
|
||||
dropTriggered
|
||||
? hexToRgb(paletteRef.current.highlightColor)
|
||||
: hexToRgb(nextPaletteColor(colorSpeed * 0.7)),
|
||||
)
|
||||
}
|
||||
|
||||
const pulse = 1 + treble * 0.25
|
||||
const r = b.radius * pulse * Math.max(0.15, b.life)
|
||||
|
||||
const grad = ctx.createRadialGradient(b.x, b.y, r * 0.1, b.x, b.y, r)
|
||||
grad.addColorStop(0, `${b.color}cc`)
|
||||
grad.addColorStop(1, `${b.color}00`)
|
||||
ctx.fillStyle = grad
|
||||
ctx.beginPath()
|
||||
ctx.arc(b.x, b.y, r, 0, Math.PI * 2)
|
||||
ctx.fill()
|
||||
}
|
||||
|
||||
// Beat-phase sweep.
|
||||
if (beatPhase > 0.001 || f.beatInfo.bpm > 0) {
|
||||
const phaseAngle = beatPhase * Math.PI * 2
|
||||
if (Math.random() < 0.08 + phasePulse * 0.22 + dropBoost * 0.18) {
|
||||
const force = (0.28 + kick * 0.5 + phasePulse * 0.4) * splatForce * 500
|
||||
splatFluid(
|
||||
canvas,
|
||||
0.5 + Math.cos(phaseAngle) * 0.28,
|
||||
0.5 + Math.sin(phaseAngle) * 0.20,
|
||||
Math.cos(phaseAngle + Math.PI / 2) * force,
|
||||
Math.sin(phaseAngle + Math.PI / 2) * force,
|
||||
dropBoost > 0.18
|
||||
? hexToRgb(paletteRef.current.highlightColor)
|
||||
: hexToRgb(nextPaletteColor(colorSpeed * 0.9)),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
prevBeatPhase = beatPhase
|
||||
}
|
||||
|
||||
raf = window.requestAnimationFrame(render)
|
||||
return () => {
|
||||
window.cancelAnimationFrame(raf)
|
||||
destroyFluidSim(canvas)
|
||||
}
|
||||
}, [featuresRef, paletteRef, width, height, fpsCap, quality, settings])
|
||||
// Only restart when data-source refs change.
|
||||
// Settings, fpsCap, and quality are read via refs each frame — no restart on changes.
|
||||
// width/height are consumed each frame as fallback to clientWidth/clientHeight.
|
||||
}, [featuresRef, paletteRef])
|
||||
|
||||
return <canvas ref={canvasRef} className="h-full w-full" aria-hidden="true" />
|
||||
}
|
||||
@@ -2,31 +2,76 @@ import { useEffect, useRef } from 'react'
|
||||
import type { MutableRefObject } from 'react'
|
||||
import type { AudioFeatures } from '../../hooks/useAudioFeatures'
|
||||
import type { VisualizerPalette } from '../../hooks/useVisualizerPalette'
|
||||
import type { VisualizerSettings } from './VisualizerBackdrop'
|
||||
import { centeredNoise2D } from './muon/noise'
|
||||
import { patternFromPreset, samplePattern, type MuonPatternFamily } from './muon/patterns'
|
||||
|
||||
type MuonSettings = NonNullable<VisualizerSettings['muon']>
|
||||
|
||||
interface MuonVisualizerCanvasProps {
|
||||
featuresRef: MutableRefObject<AudioFeatures>
|
||||
paletteRef: MutableRefObject<VisualizerPalette>
|
||||
width: number
|
||||
height: number
|
||||
fpsCap?: 30 | 60 | 120
|
||||
quality?: 'low' | 'balanced' | 'high'
|
||||
settings?: {
|
||||
radius_multiplier?: number
|
||||
aperture?: number
|
||||
spacing?: number
|
||||
}
|
||||
settings?: MuonSettings
|
||||
fpsCap?: number
|
||||
quality?: string
|
||||
}
|
||||
|
||||
type Particle = { angle: number; radius: number; speed: number; lane: number }
|
||||
|
||||
const PARTICLE_COUNT = 220
|
||||
type MuonParticle = {
|
||||
active: boolean
|
||||
x: number
|
||||
y: number
|
||||
vx: number
|
||||
vy: number
|
||||
ring: number
|
||||
age: number
|
||||
maxAge: number
|
||||
size: number
|
||||
hue: number
|
||||
alpha: number
|
||||
pattern: MuonPatternFamily
|
||||
}
|
||||
|
||||
function clamp01(v: number): number {
|
||||
return Math.max(0, Math.min(1, v))
|
||||
}
|
||||
|
||||
export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, height, fpsCap = 60, quality = 'balanced', settings }: MuonVisualizerCanvasProps) {
|
||||
function clamp(v: number, lo: number, hi: number): number {
|
||||
return Math.max(lo, Math.min(hi, v))
|
||||
}
|
||||
|
||||
function maxParticlesByQuality(quality?: string): number {
|
||||
if (quality === 'low') return 900
|
||||
if (quality === 'high') return 2400
|
||||
return 1600
|
||||
}
|
||||
|
||||
function createParticlePool(size: number): MuonParticle[] {
|
||||
return Array.from({ length: size }, () => ({
|
||||
active: false,
|
||||
x: 0,
|
||||
y: 0,
|
||||
vx: 0,
|
||||
vy: 0,
|
||||
ring: 0,
|
||||
age: 0,
|
||||
maxAge: 1,
|
||||
size: 1,
|
||||
hue: 0,
|
||||
alpha: 1,
|
||||
pattern: 'orbital',
|
||||
}))
|
||||
}
|
||||
|
||||
export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, height, settings, fpsCap, quality }: MuonVisualizerCanvasProps) {
|
||||
const canvasRef = useRef<HTMLCanvasElement>(null)
|
||||
// Stable ref so the render loop always reads the latest settings without restarting.
|
||||
const settingsRef = useRef<MuonSettings | undefined>(settings)
|
||||
settingsRef.current = settings
|
||||
|
||||
const fpsCapRef = useRef(fpsCap ?? 60)
|
||||
fpsCapRef.current = fpsCap ?? 60
|
||||
|
||||
useEffect(() => {
|
||||
const canvas = canvasRef.current
|
||||
@@ -34,32 +79,94 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
|
||||
const ctx = canvas.getContext('2d')
|
||||
if (!ctx) return
|
||||
|
||||
const particles: Particle[] = Array.from({ length: PARTICLE_COUNT }, (_, i) => ({
|
||||
angle: (i / PARTICLE_COUNT) * Math.PI * 2,
|
||||
radius: 0.2 + (i % 12) * 0.08,
|
||||
speed: 0.002 + (i % 9) * 0.00045,
|
||||
lane: i % 8,
|
||||
}))
|
||||
let radiusCurrent = 1
|
||||
let radiusTarget = 1
|
||||
let prevBeatPhase = 0
|
||||
let dropBoost = 0
|
||||
let lastDropAt = 0
|
||||
let patternShift = 0
|
||||
|
||||
let poolCap = maxParticlesByQuality('balanced')
|
||||
let particles = createParticlePool(poolCap)
|
||||
let nextPoolIdx = 0
|
||||
|
||||
let raf = 0
|
||||
|
||||
// Correction B fix: radiusMultiplier cycles through presets on beats rather than
|
||||
// being mapped directly from drum intensity each frame. We lerp toward a preset
|
||||
// target, snapping to the next preset on each detected beat — equivalent to
|
||||
// wavePresetController triggering a GSAP transition in the full Three.js port.
|
||||
const RADIUS_PRESETS = [0.90, 0.95, 1.00, 1.05, 1.10, 0.85, 1.15, 1.02]
|
||||
let radiusTarget = 1.0
|
||||
let radiusCurrent = 1.0
|
||||
let lastPresetAt = 0
|
||||
let presetIdx = 0
|
||||
let lastFrameAt = 0
|
||||
const cap = fpsCap === 30 || fpsCap === 120 ? fpsCap : 60
|
||||
const minFrameMs = 1000 / cap
|
||||
|
||||
const emitParticle = (
|
||||
cx: number,
|
||||
cy: number,
|
||||
ring: number,
|
||||
divisions: number,
|
||||
beatPhase: number,
|
||||
hueBase: number,
|
||||
aperture: number,
|
||||
mirror: boolean,
|
||||
patternPreset: string,
|
||||
ringRadius: number,
|
||||
lifeFrames: number,
|
||||
noiseScale: number,
|
||||
noiseForce: number,
|
||||
alphaSeed: number,
|
||||
sizeSeed: number,
|
||||
dynamicRadius: number,
|
||||
) => {
|
||||
const emitOne = (phaseOffset: number) => {
|
||||
const p = particles[nextPoolIdx]
|
||||
nextPoolIdx = (nextPoolIdx + 1) % particles.length
|
||||
|
||||
const t = ((performance.now() * 0.0004) + ring / Math.max(1, divisions) + phaseOffset) * Math.PI * 2
|
||||
const pattern = patternFromPreset(patternPreset, ring + patternShift)
|
||||
const ringNorm = ring / Math.max(1, divisions - 1)
|
||||
const sample = samplePattern(pattern, t, ringNorm, aperture, beatPhase)
|
||||
const nx = centeredNoise2D(sample.x * noiseScale + ringNorm * 1.3, sample.y * noiseScale, ring * 17.17)
|
||||
const ny = centeredNoise2D(sample.y * noiseScale, sample.x * noiseScale + ringNorm * 1.9, ring * 29.37)
|
||||
|
||||
const noiseAmp = noiseForce * 0.32
|
||||
const sx = sample.x + nx * noiseAmp
|
||||
const sy = sample.y + ny * noiseAmp
|
||||
const ex = cx + sx * ringRadius * dynamicRadius
|
||||
const ey = cy + sy * ringRadius * 0.86 * dynamicRadius
|
||||
|
||||
const tvx = sample.tx + nx * noiseAmp * 0.4
|
||||
const tvy = sample.ty + ny * noiseAmp * 0.4
|
||||
const speed = 0.12 + ringNorm * 0.22 + sizeSeed * 0.08
|
||||
|
||||
p.active = true
|
||||
p.x = ex
|
||||
p.y = ey
|
||||
p.vx = tvx * speed
|
||||
p.vy = tvy * speed
|
||||
p.ring = ring
|
||||
p.age = 0
|
||||
p.maxAge = lifeFrames
|
||||
p.size = 0.9 + sizeSeed * 2.2
|
||||
p.hue = (hueBase + ring * 360 / Math.max(1, divisions)) % 360
|
||||
p.alpha = clamp(alphaSeed, 0.2, 1)
|
||||
p.pattern = pattern
|
||||
}
|
||||
|
||||
emitOne(0)
|
||||
if (mirror) emitOne(0.5)
|
||||
}
|
||||
|
||||
const render = (ts: number) => {
|
||||
raf = window.requestAnimationFrame(render)
|
||||
if (lastFrameAt && ts - lastFrameAt < minFrameMs) return
|
||||
|
||||
// FPS cap
|
||||
const minFrameMs = 1000 / Math.max(1, fpsCapRef.current)
|
||||
if (ts - lastFrameAt < minFrameMs - 1) return
|
||||
const dt = Math.max(0.5, Math.min(2, (ts - lastFrameAt) / 16.6667))
|
||||
lastFrameAt = ts
|
||||
|
||||
const s = settingsRef.current
|
||||
const qualityCap = maxParticlesByQuality(quality)
|
||||
if (qualityCap !== poolCap) {
|
||||
poolCap = qualityCap
|
||||
particles = createParticlePool(poolCap)
|
||||
nextPoolIdx = 0
|
||||
}
|
||||
|
||||
const w = Math.max(1, Math.round(canvas.clientWidth || 0), Math.round(width || 0))
|
||||
const h = Math.max(1, Math.round(canvas.clientHeight || 0), Math.round(height || 0))
|
||||
if (canvas.width !== w || canvas.height !== h) {
|
||||
@@ -69,76 +176,157 @@ export default function MuonVisualizerCanvas({ featuresRef, paletteRef, width, h
|
||||
|
||||
const f = featuresRef.current
|
||||
const p = paletteRef.current
|
||||
const drum = clamp01(f.drumIntensity)
|
||||
const now = performance.now() * 0.001
|
||||
|
||||
const radiusScale = Math.max(0.2, s?.radius_multiplier ?? 1)
|
||||
const apertureScale = Math.max(0.2, s?.aperture ?? 1)
|
||||
const spacingScale = Math.max(0.2, s?.spacing ?? 1)
|
||||
const rotationSpeed = Math.max(0.1, s?.rotation_speed ?? 1)
|
||||
const glowIntensity = Math.max(0, s?.glow_intensity ?? 1)
|
||||
const particleSize = Math.max(0.1, s?.particle_size ?? 1)
|
||||
const drumReactivity = Math.max(0, s?.drum_reactivity ?? 1)
|
||||
const hueRange = Math.max(0, s?.hue_range ?? 1)
|
||||
const beatSensitivity = Math.max(0.1, s?.beat_sensitivity ?? 1)
|
||||
const ellipseRatio = Math.max(0.1, Math.min(1, s?.ellipse_ratio ?? 0.62))
|
||||
const idleSpeed = Math.max(0, s?.idle_speed ?? 1)
|
||||
const monoColor = Boolean(s?.mono_color)
|
||||
const divisions = Math.max(8, Math.min(48, Math.round(s?.divisions ?? 16)))
|
||||
const lifespan = Math.max(45, Math.min(420, Math.round(s?.lifespan ?? 180)))
|
||||
const mirror = Boolean(s?.particle_mirror)
|
||||
const dynamicRadiusCfg = Math.max(0.2, s?.dynamic_radius ?? 1)
|
||||
const deltaLimit = Math.max(0.0005, s?.delta_response_limit ?? 0.005)
|
||||
const preset = s?.visualization_preset ?? 'hybrid'
|
||||
const noiseScale = Math.max(0.05, s?.noise_scale ?? 1)
|
||||
const noiseForce = Math.max(0, s?.noise_force ?? 1)
|
||||
|
||||
const sub = clamp01(f.frequencyBands.sub)
|
||||
const kick = clamp01(sub * 0.72 + f.bass * 0.28)
|
||||
const drum = clamp01((kick * 0.75 + f.drumIntensity * 0.25) * drumReactivity)
|
||||
const treble = clamp01(f.treble)
|
||||
const now = ts
|
||||
const idle = 0.5 + 0.5 * Math.sin(now * 0.0012)
|
||||
const radiusScale = Math.max(0.2, Number(settings?.radius_multiplier ?? 1))
|
||||
const apertureScale = Math.max(0.2, Number(settings?.aperture ?? 1))
|
||||
const spacingScale = Math.max(0.2, Number(settings?.spacing ?? 1))
|
||||
const bright = clamp01((f.frequencyBands.presence + f.frequencyBands.brilliance) * 0.5)
|
||||
const beatPhase = clamp01(f.beatInfo.beatPhase)
|
||||
const phaseWave = 0.5 + 0.5 * Math.sin(beatPhase * Math.PI * 2)
|
||||
const phasePulse = Math.pow(Math.max(0, Math.sin(beatPhase * Math.PI)), 1.6)
|
||||
const energyDeltaAbs = Math.abs(f.patternInfo.energyDelta)
|
||||
|
||||
// Bug 1 fix: hue animates via drum intensity when audio is playing, and falls
|
||||
// back to a slow time-based drift when silent so the visual never freezes on a
|
||||
// single colour (original hueControl bug: hue was always 0 during playback).
|
||||
// Correction A fix: treble drift is capped at ±15° to stay within the manifest
|
||||
// palette colour family (original Integration Plan used treble * 40 / 120 which
|
||||
// rotated far outside the intended hue family).
|
||||
const hue = drum > 0.01
|
||||
? ((p.seedHue + drum * 20 + treble * 15) + 360) % 360
|
||||
: ((p.seedHue + Math.sin(now * 0.00018) * 15) + 360) % 360
|
||||
|
||||
// Correction B fix: advance the preset index on each beat and lerp toward the
|
||||
// new target — mirrors wavePresetController / GSAP transition in the Three.js port.
|
||||
if (f.beatDetected && now - lastPresetAt > 200) {
|
||||
presetIdx = (presetIdx + 1) % RADIUS_PRESETS.length
|
||||
radiusTarget = RADIUS_PRESETS[presetIdx] + drum * 0.25
|
||||
lastPresetAt = now
|
||||
const dropTriggered = f.patternInfo.isBeatDrop && now - lastDropAt > 0.5
|
||||
if (dropTriggered) {
|
||||
dropBoost = 1
|
||||
lastDropAt = now
|
||||
patternShift += 1
|
||||
}
|
||||
radiusCurrent += (radiusTarget - radiusCurrent) * 0.07
|
||||
const radiusMultiplier = (radiusCurrent + idle * 0.12) * radiusScale
|
||||
dropBoost *= 0.965
|
||||
|
||||
const aperture = (1.55 + drum * 1.9 + idle * 0.24) * apertureScale
|
||||
const spacing = (0.015 + (1 - drum) * 0.014) * spacingScale
|
||||
const phaseWrapped = beatPhase < prevBeatPhase - 0.35
|
||||
const idle = 0.5 + 0.5 * Math.sin(now * 1.2 * Math.max(0.01, idleSpeed))
|
||||
|
||||
ctx.clearRect(0, 0, w, h)
|
||||
const hue = monoColor
|
||||
? p.seedHue
|
||||
: drum > 0.01
|
||||
? ((p.seedHue + drum * 20 * hueRange + treble * 15 * hueRange + phaseWave * 10 * hueRange + dropBoost * 55 * hueRange) + 360) % 360
|
||||
: ((p.seedHue + Math.sin(now * 0.22) * 15 * hueRange) + 360) % 360
|
||||
|
||||
const deltaGate = energyDeltaAbs >= deltaLimit ? 1 : 0
|
||||
radiusTarget = 1
|
||||
+ drum * 0.18
|
||||
+ phasePulse * 0.12
|
||||
+ idle * 0.05
|
||||
+ dropBoost * 0.3
|
||||
+ (phaseWrapped ? 0.06 : 0)
|
||||
+ deltaGate * Math.min(0.5, energyDeltaAbs * dynamicRadiusCfg * 15)
|
||||
radiusCurrent += (radiusTarget - radiusCurrent) * 0.065
|
||||
const dynamicRadius = radiusCurrent * dynamicRadiusCfg
|
||||
|
||||
const aperture = (1.2 + drum * 1.8 + phaseWave * 0.25 + dropBoost * 0.35) * apertureScale
|
||||
const spacing = (Math.min(w, h) * (0.013 + (1 - drum) * 0.012)) * spacingScale
|
||||
|
||||
const trailAlpha = clamp(0.16 + (1 - clamp01(lifespan / 420)) * 0.36 - dropBoost * 0.05, 0.08, 0.55)
|
||||
ctx.fillStyle = p.backgroundColor
|
||||
ctx.globalAlpha = trailAlpha
|
||||
ctx.fillRect(0, 0, w, h)
|
||||
ctx.globalAlpha = 1
|
||||
|
||||
const cx = w / 2
|
||||
const cy = h / 2
|
||||
const base = Math.min(w, h) * 0.2
|
||||
const glow = 14 + drum * 28
|
||||
const particleStride = quality === 'low' ? 2 : 1
|
||||
const cx = w / 2
|
||||
const cy = h / 2
|
||||
const base = Math.min(w, h) * 0.17 * radiusScale
|
||||
const glow = (10 + drum * 22 + phasePulse * 7 + dropBoost * 18) * glowIntensity
|
||||
|
||||
const emitDensity = (s?.particle_count ?? 220) / 220
|
||||
const emitCount = Math.max(1, Math.round((0.8 + drum * 2.4 + phasePulse * 0.8 + dropBoost * 2.2) * emitDensity * (divisions / 16) * (0.75 + beatSensitivity * 0.25)))
|
||||
const lifeFrames = Math.round(lifespan * (0.8 + (1 - drum) * 0.25))
|
||||
|
||||
for (let i = 0; i < emitCount; i += 1) {
|
||||
const ring = i % divisions
|
||||
const ringRadius = base + ring * spacing
|
||||
emitParticle(
|
||||
cx,
|
||||
cy,
|
||||
ring,
|
||||
divisions,
|
||||
beatPhase,
|
||||
hue + bright * 18,
|
||||
aperture,
|
||||
mirror,
|
||||
preset,
|
||||
ringRadius,
|
||||
lifeFrames,
|
||||
noiseScale,
|
||||
noiseForce,
|
||||
0.34 + drum * 0.45 + phasePulse * 0.1 + dropBoost * 0.15,
|
||||
0.45 + bright * 0.35,
|
||||
dynamicRadius,
|
||||
)
|
||||
}
|
||||
|
||||
ctx.globalCompositeOperation = 'lighter'
|
||||
for (let i = 0; i < particles.length; i += particleStride) {
|
||||
for (let i = 0; i < particles.length; i += 1) {
|
||||
const pt = particles[i]
|
||||
pt.angle += pt.speed + drum * 0.02
|
||||
if (!pt.active) continue
|
||||
|
||||
const ring = base + pt.lane * (spacing * Math.min(w, h) * 5)
|
||||
const r = ring * radiusMultiplier * (1 + Math.sin(pt.angle * aperture) * 0.16)
|
||||
const x = cx + Math.cos(pt.angle) * r
|
||||
const y = cy + Math.sin(pt.angle * aperture) * (r * 0.62)
|
||||
pt.age += dt
|
||||
if (pt.age >= pt.maxAge) {
|
||||
pt.active = false
|
||||
continue
|
||||
}
|
||||
|
||||
const size = 1.75 + drum * 3.0 + (i % 4) * 0.35
|
||||
const alpha = 0.34 + drum * 0.42
|
||||
const life = 1 - pt.age / Math.max(1, pt.maxAge)
|
||||
const drag = clamp(0.94 + (1 - drum) * 0.03, 0.86, 0.985)
|
||||
pt.vx *= Math.pow(drag, dt)
|
||||
pt.vy *= Math.pow(drag, dt)
|
||||
pt.x += pt.vx * rotationSpeed * dt
|
||||
pt.y += pt.vy * rotationSpeed * dt
|
||||
|
||||
const wobble = Math.sin((now * 1.2 + pt.ring * 0.37 + pt.age * 0.03) * aperture) * (0.6 + noiseForce * 0.4)
|
||||
const x = pt.x + wobble
|
||||
const y = pt.y + wobble * ellipseRatio
|
||||
|
||||
const size = Math.max(0.3, pt.size * particleSize * (0.4 + life * 0.9))
|
||||
const alpha = pt.alpha * life
|
||||
|
||||
const laneHue = monoColor
|
||||
? hue
|
||||
: (pt.hue + phaseWave * 18 * hueRange + beatPhase * 8 + dropBoost * 18) % 360
|
||||
|
||||
ctx.beginPath()
|
||||
ctx.fillStyle = `hsla(${hue}, 92%, 64%, ${alpha})`
|
||||
ctx.fillStyle = `hsla(${laneHue}, 92%, ${58 + bright * 24}%, ${alpha})`
|
||||
ctx.shadowColor = p.highlightColor
|
||||
ctx.shadowBlur = glow
|
||||
ctx.shadowBlur = glow
|
||||
ctx.arc(x, y, size, 0, Math.PI * 2)
|
||||
ctx.fill()
|
||||
}
|
||||
ctx.globalCompositeOperation = 'source-over'
|
||||
ctx.shadowBlur = 0
|
||||
prevBeatPhase = beatPhase
|
||||
}
|
||||
|
||||
raf = window.requestAnimationFrame(render)
|
||||
return () => {
|
||||
window.cancelAnimationFrame(raf)
|
||||
}
|
||||
}, [featuresRef, paletteRef, width, height, fpsCap, quality, settings])
|
||||
// Only restart when data-source refs change.
|
||||
// Settings are read via settingsRef each frame — no restart on slider changes.
|
||||
// width/height are consumed each frame as fallback to clientWidth/clientHeight.
|
||||
}, [featuresRef, paletteRef])
|
||||
|
||||
return <canvas ref={canvasRef} className="h-full w-full" aria-hidden="true" />
|
||||
}
|
||||
@@ -10,26 +10,50 @@ export type VisualizerMode = 'muon' | 'fluid' | null
|
||||
|
||||
/**
|
||||
* Per-song visualizer intensity multipliers from manifest.visualizer.settings.
|
||||
* All values default to 1.0 (no change from audio-driven behaviour).
|
||||
* All numeric values are multipliers (1.0 = default audio-driven behaviour) unless noted.
|
||||
*/
|
||||
export interface VisualizerSettings {
|
||||
quality?: 'low' | 'balanced' | 'high'
|
||||
fps_cap?: 30 | 60 | 120
|
||||
muon?: {
|
||||
/** Scales the preset-cycling radius range. Default 1.0. */
|
||||
// Original
|
||||
radius_multiplier?: number
|
||||
/** Scales the spiral aperture range. Default 1.0. */
|
||||
aperture?: number
|
||||
/** Scales inter-particle spacing. Default 1.0. */
|
||||
spacing?: number
|
||||
// Controls Audit additions
|
||||
particle_count?: number // direct int 40–500
|
||||
rotation_speed?: number
|
||||
glow_intensity?: number
|
||||
particle_size?: number
|
||||
drum_reactivity?: number
|
||||
hue_range?: number
|
||||
beat_sensitivity?: number
|
||||
ellipse_ratio?: number // direct 0.30–1.0
|
||||
// GUI Parameter Map additions
|
||||
idle_speed?: number
|
||||
noise_scale?: number
|
||||
noise_force?: number
|
||||
mono_color?: boolean
|
||||
divisions?: number
|
||||
lifespan?: number
|
||||
particle_mirror?: boolean
|
||||
dynamic_radius?: number
|
||||
delta_response_limit?: number
|
||||
visualization_preset?: 'default' | 'orbital' | 'flower' | 'spiro' | 'hybrid'
|
||||
}
|
||||
fluid?: {
|
||||
/** Multiplies ambient blob spawn probability per frame. Default 1.0. */
|
||||
// Original
|
||||
spawn_density?: number
|
||||
/** Multiplies blob velocity and size on spawn. Default 1.0. */
|
||||
splat_force?: number
|
||||
/** Multiplies per-frame blob lifetime drain (higher = shorter trails). Default 1.0. */
|
||||
decay?: number
|
||||
// Controls Audit additions
|
||||
background_persistence?: number
|
||||
beat_burst_count?: number
|
||||
treble_pulse?: number
|
||||
drag?: number
|
||||
blob_life?: number
|
||||
// GUI Parameter Map additions
|
||||
color_speed?: number
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
let WebGLFluid: any = null
|
||||
let fluidLoadPromise: Promise<any> | null = null
|
||||
|
||||
// Load vendored local module dynamically so Vite resolves a stable in-repo path.
|
||||
const getWebGLFluid = async (): Promise<any> => {
|
||||
if (WebGLFluid) return WebGLFluid
|
||||
if (fluidLoadPromise) return fluidLoadPromise
|
||||
|
||||
fluidLoadPromise = (async () => {
|
||||
const module = await import('./webgl-fluid.mjs')
|
||||
WebGLFluid = module.default || module
|
||||
return WebGLFluid
|
||||
})()
|
||||
|
||||
return fluidLoadPromise
|
||||
}
|
||||
|
||||
export interface FluidColor {
|
||||
r: number
|
||||
g: number
|
||||
b: number
|
||||
}
|
||||
|
||||
export interface FluidConfig {
|
||||
TRIGGER?: 'hover' | 'click'
|
||||
IMMEDIATE?: boolean
|
||||
AUTO?: boolean
|
||||
INTERVAL?: number
|
||||
SIM_RESOLUTION?: number
|
||||
DYE_RESOLUTION?: number
|
||||
DENSITY_DISSIPATION?: number
|
||||
VELOCITY_DISSIPATION?: number
|
||||
PRESSURE?: number
|
||||
PRESSURE_ITERATIONS?: number
|
||||
CURL?: number
|
||||
SPLAT_RADIUS?: number
|
||||
SPLAT_FORCE?: number
|
||||
SPLAT_COUNT?: number
|
||||
SHADING?: boolean
|
||||
COLORFUL?: boolean
|
||||
COLOR_UPDATE_SPEED?: number
|
||||
PAUSED?: boolean
|
||||
BACK_COLOR?: FluidColor
|
||||
TRANSPARENT?: boolean
|
||||
BLOOM?: boolean
|
||||
BLOOM_ITERATIONS?: number
|
||||
BLOOM_RESOLUTION?: number
|
||||
BLOOM_INTENSITY?: number
|
||||
BLOOM_THRESHOLD?: number
|
||||
BLOOM_SOFT_KNEE?: number
|
||||
SUNRAYS?: boolean
|
||||
SUNRAYS_RESOLUTION?: number
|
||||
SUNRAYS_WEIGHT?: number
|
||||
}
|
||||
|
||||
interface FluidInstance {
|
||||
canvas: HTMLCanvasElement
|
||||
config: FluidConfig
|
||||
lastX: number
|
||||
lastY: number
|
||||
}
|
||||
|
||||
const instances = new WeakMap<HTMLCanvasElement, FluidInstance>()
|
||||
|
||||
function clamp01(v: number): number {
|
||||
return Math.max(0, Math.min(1, v))
|
||||
}
|
||||
|
||||
function dispatchMouse(canvas: HTMLCanvasElement, type: 'mousedown' | 'mousemove', x: number, y: number): void {
|
||||
const evt = new MouseEvent(type, {
|
||||
bubbles: true,
|
||||
cancelable: true,
|
||||
clientX: x,
|
||||
clientY: y,
|
||||
})
|
||||
Object.defineProperty(evt, 'offsetX', { configurable: true, value: x })
|
||||
Object.defineProperty(evt, 'offsetY', { configurable: true, value: y })
|
||||
canvas.dispatchEvent(evt)
|
||||
}
|
||||
|
||||
export function initFluidSim(canvas: HTMLCanvasElement, initialConfig: FluidConfig): void {
|
||||
destroyFluidSim(canvas)
|
||||
|
||||
const config: FluidConfig = {
|
||||
TRIGGER: 'hover',
|
||||
IMMEDIATE: false,
|
||||
AUTO: false,
|
||||
INTERVAL: 3000,
|
||||
COLORFUL: false,
|
||||
SHADING: true,
|
||||
BLOOM: true,
|
||||
SUNRAYS: false,
|
||||
PAUSED: false,
|
||||
...initialConfig,
|
||||
}
|
||||
|
||||
// Load WebGL library asynchronously but don't block
|
||||
getWebGLFluid()
|
||||
.then((WebGL) => {
|
||||
// Only initialize if canvas still exists and not destroyed
|
||||
if (document.contains(canvas)) {
|
||||
WebGL(canvas, config)
|
||||
instances.set(canvas, {
|
||||
canvas,
|
||||
config,
|
||||
lastX: canvas.clientWidth * 0.5,
|
||||
lastY: canvas.clientHeight * 0.5,
|
||||
})
|
||||
}
|
||||
})
|
||||
.catch((err) => console.error('Failed to load webgl-fluid:', err))
|
||||
}
|
||||
|
||||
export function updateFluidConfig(canvas: HTMLCanvasElement, patch: Partial<FluidConfig>): void {
|
||||
const inst = instances.get(canvas)
|
||||
if (!inst) return
|
||||
Object.assign(inst.config, patch)
|
||||
}
|
||||
|
||||
export function splatFluid(canvas: HTMLCanvasElement, x: number, y: number, dx: number, dy: number, _color: FluidColor): void {
|
||||
const inst = instances.get(canvas)
|
||||
if (!inst) return
|
||||
|
||||
const w = Math.max(1, canvas.clientWidth || canvas.width || 1)
|
||||
const h = Math.max(1, canvas.clientHeight || canvas.height || 1)
|
||||
|
||||
const nx = clamp01(x)
|
||||
const ny = clamp01(y)
|
||||
|
||||
const px = nx * w
|
||||
const py = ny * h
|
||||
|
||||
const speedNorm = Math.hypot(dx, dy) / Math.max(1, Math.min(w, h) * 6)
|
||||
const pressure = clamp01(speedNorm)
|
||||
const dragScale = 0.02 + pressure * 0.09
|
||||
const tx = Math.max(0, Math.min(w, px + dx * dragScale))
|
||||
const ty = Math.max(0, Math.min(h, py + dy * dragScale))
|
||||
const mx = px + (tx - px) * (0.45 + pressure * 0.25)
|
||||
const my = py + (ty - py) * (0.45 + pressure * 0.25)
|
||||
|
||||
dispatchMouse(canvas, 'mousedown', px, py)
|
||||
dispatchMouse(canvas, 'mousemove', mx, my)
|
||||
dispatchMouse(canvas, 'mousemove', tx, ty)
|
||||
|
||||
window.dispatchEvent(new MouseEvent('mouseup', { bubbles: true, cancelable: true }))
|
||||
|
||||
inst.lastX = tx
|
||||
inst.lastY = ty
|
||||
}
|
||||
|
||||
export function destroyFluidSim(canvas: HTMLCanvasElement): void {
|
||||
const inst = instances.get(canvas)
|
||||
if (!inst) return
|
||||
|
||||
inst.config.PAUSED = true
|
||||
inst.config.AUTO = false
|
||||
|
||||
const gl = (canvas.getContext('webgl2') || canvas.getContext('webgl')) as WebGLRenderingContext | null
|
||||
const lose = gl?.getExtension('WEBGL_lose_context')
|
||||
lose?.loseContext()
|
||||
|
||||
instances.delete(canvas)
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,2 @@
|
||||
declare const WebGLFluid: (canvas: HTMLCanvasElement, options?: any) => void
|
||||
export default WebGLFluid
|
||||
@@ -0,0 +1,8 @@
|
||||
export function hashNoise2D(x: number, y: number, seed = 0): number {
|
||||
const s = Math.sin(x * 127.1 + y * 311.7 + seed * 74.7) * 43758.5453123
|
||||
return s - Math.floor(s)
|
||||
}
|
||||
|
||||
export function centeredNoise2D(x: number, y: number, seed = 0): number {
|
||||
return hashNoise2D(x, y, seed) * 2 - 1
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
export type MuonPatternFamily = 'orbital' | 'flower' | 'ripple' | 'spiro' | 'strobe-grid'
|
||||
|
||||
export interface PatternSample {
|
||||
x: number
|
||||
y: number
|
||||
tx: number
|
||||
ty: number
|
||||
}
|
||||
|
||||
function clamp01(v: number): number {
|
||||
return Math.max(0, Math.min(1, v))
|
||||
}
|
||||
|
||||
function sampleOrbital(t: number, ring: number, aperture: number): PatternSample {
|
||||
const r = 1 + Math.sin(t * (1.2 + ring * 0.9) * aperture) * 0.16
|
||||
const x = Math.cos(t) * r
|
||||
const y = Math.sin(t) * (0.85 + ring * 0.22)
|
||||
return { x, y, tx: -Math.sin(t), ty: Math.cos(t) }
|
||||
}
|
||||
|
||||
function sampleFlower(t: number, ring: number, aperture: number, beatPhase: number): PatternSample {
|
||||
const petals = 3 + (ring % 6)
|
||||
const pulse = 0.82 + Math.sin(beatPhase * Math.PI * 2) * 0.08
|
||||
const r = 0.78 + Math.cos(t * petals * aperture) * 0.28 * pulse
|
||||
const x = Math.cos(t) * r
|
||||
const y = Math.sin(t) * r
|
||||
return { x, y, tx: -Math.sin(t) * (1 + petals * 0.04), ty: Math.cos(t) * (1 + petals * 0.04) }
|
||||
}
|
||||
|
||||
function sampleRipple(t: number, ring: number, aperture: number, beatPhase: number): PatternSample {
|
||||
const ripple = Math.sin(t * (2.4 + ring * 1.3) * aperture + beatPhase * Math.PI * 2)
|
||||
const r = 0.86 + ripple * 0.22
|
||||
const x = Math.cos(t) * r
|
||||
const y = Math.sin(t) * (0.7 + Math.abs(ripple) * 0.5)
|
||||
return { x, y, tx: -Math.sin(t), ty: Math.cos(t) }
|
||||
}
|
||||
|
||||
function sampleSpiro(t: number, ring: number, aperture: number): PatternSample {
|
||||
const a = 1
|
||||
const b = 0.35 + (ring % 5) * 0.07
|
||||
const k = a - b
|
||||
const wt = t * (1.2 + aperture * 0.6)
|
||||
const x = k * Math.cos(wt) + b * Math.cos((k / b) * wt)
|
||||
const y = k * Math.sin(wt) - b * Math.sin((k / b) * wt)
|
||||
return { x, y, tx: -Math.sin(wt), ty: Math.cos(wt) }
|
||||
}
|
||||
|
||||
function sampleStrobeGrid(t: number, ring: number, beatPhase: number): PatternSample {
|
||||
const segments = 8 + (ring % 8)
|
||||
const q = Math.round((t / (Math.PI * 2)) * segments) / segments
|
||||
const a = q * Math.PI * 2
|
||||
const pulse = 0.84 + clamp01(Math.sin(beatPhase * Math.PI * 2) * 0.5 + 0.5) * 0.22
|
||||
const x = Math.cos(a) * pulse
|
||||
const y = Math.sin(a) * pulse
|
||||
return { x, y, tx: -Math.sin(a), ty: Math.cos(a) }
|
||||
}
|
||||
|
||||
export function samplePattern(
|
||||
family: MuonPatternFamily,
|
||||
t: number,
|
||||
ringNorm: number,
|
||||
aperture: number,
|
||||
beatPhase: number,
|
||||
): PatternSample {
|
||||
switch (family) {
|
||||
case 'flower':
|
||||
return sampleFlower(t, ringNorm, aperture, beatPhase)
|
||||
case 'ripple':
|
||||
return sampleRipple(t, ringNorm, aperture, beatPhase)
|
||||
case 'spiro':
|
||||
return sampleSpiro(t, ringNorm, aperture)
|
||||
case 'strobe-grid':
|
||||
return sampleStrobeGrid(t, ringNorm, beatPhase)
|
||||
case 'orbital':
|
||||
default:
|
||||
return sampleOrbital(t, ringNorm, aperture)
|
||||
}
|
||||
}
|
||||
|
||||
const PRESET_MAP: Record<string, MuonPatternFamily[]> = {
|
||||
default: ['orbital', 'ripple', 'flower'],
|
||||
orbital: ['orbital'],
|
||||
flower: ['flower', 'ripple'],
|
||||
spiro: ['spiro', 'orbital'],
|
||||
hybrid: ['orbital', 'flower', 'ripple', 'spiro', 'strobe-grid'],
|
||||
}
|
||||
|
||||
export function patternFromPreset(preset: string | undefined, ring: number): MuonPatternFamily {
|
||||
const key = preset && PRESET_MAP[preset] ? preset : 'hybrid'
|
||||
const list = PRESET_MAP[key]
|
||||
return list[ring % list.length]
|
||||
}
|
||||
@@ -1,4 +1,8 @@
|
||||
import { useEffect, useRef, type MutableRefObject, type RefObject } from 'react'
|
||||
import {
|
||||
MuonFeatureExtractor,
|
||||
type MuonFrequencyBands,
|
||||
} from '../audio/MuonFeatureExtractor'
|
||||
|
||||
export interface AudioFeatures {
|
||||
bass: number
|
||||
@@ -9,76 +13,88 @@ export interface AudioFeatures {
|
||||
spectralCentroid: number
|
||||
drumIntensity: number
|
||||
beatDetected: boolean
|
||||
freqData: Uint8Array
|
||||
frequencyBands: {
|
||||
sub: number
|
||||
bass: number
|
||||
lowMid: number
|
||||
mid: number
|
||||
highMid: number
|
||||
presence: number
|
||||
brilliance: number
|
||||
}
|
||||
beatInfo: {
|
||||
isBeat: boolean
|
||||
bpm: number
|
||||
confidence: number
|
||||
timeSinceLastBeat: number
|
||||
beatPhase: number
|
||||
}
|
||||
patternInfo: {
|
||||
isBeatDrop: boolean
|
||||
isBuildUp: boolean
|
||||
isSilence: boolean
|
||||
intensity: number
|
||||
energyDelta: number
|
||||
}
|
||||
spectralRolloff: number
|
||||
spectralFlatness: number
|
||||
zcr: number
|
||||
freqData: Float32Array
|
||||
updatedAt: number
|
||||
}
|
||||
|
||||
interface MeydaFeatureFrame {
|
||||
rms?: number
|
||||
energy?: number
|
||||
loudness?: number | { total?: number }
|
||||
spectralCentroid?: number
|
||||
}
|
||||
|
||||
interface MeydaAnalyzerLike {
|
||||
start?: () => void
|
||||
stop?: () => void
|
||||
}
|
||||
|
||||
interface MeydaLike {
|
||||
createMeydaAnalyzer: (opts: {
|
||||
audioContext: AudioContext
|
||||
source: AudioNode
|
||||
bufferSize: number
|
||||
featureExtractors: string[]
|
||||
callback: (features: MeydaFeatureFrame) => void
|
||||
}) => MeydaAnalyzerLike
|
||||
}
|
||||
|
||||
interface UseAudioFeaturesArgs {
|
||||
audioCtxRef: RefObject<AudioContext | null>
|
||||
stemGainRefs: RefObject<Record<string, GainNode>>
|
||||
singleAudioRef: RefObject<HTMLAudioElement | null>
|
||||
analysisAudioRef: RefObject<HTMLAudioElement | null>
|
||||
isPlayingRef: RefObject<boolean>
|
||||
enabled?: boolean
|
||||
analysisSourceKey?: string | null
|
||||
}
|
||||
|
||||
const EMPTY_FREQ = new Uint8Array(0)
|
||||
const EMPTY_FREQ = new Float32Array(0)
|
||||
const EMPTY_BANDS = {
|
||||
sub: 0,
|
||||
bass: 0,
|
||||
lowMid: 0,
|
||||
mid: 0,
|
||||
highMid: 0,
|
||||
presence: 0,
|
||||
brilliance: 0,
|
||||
}
|
||||
const EMPTY_BEAT = {
|
||||
isBeat: false,
|
||||
bpm: 0,
|
||||
confidence: 0,
|
||||
timeSinceLastBeat: 0,
|
||||
beatPhase: 0,
|
||||
}
|
||||
const EMPTY_PAT = {
|
||||
isBeatDrop: false,
|
||||
isBuildUp: false,
|
||||
isSilence: true,
|
||||
intensity: 0,
|
||||
energyDelta: 0,
|
||||
}
|
||||
const PEAK_DECAY = 0.9998
|
||||
const BAND_KEYS: Array<keyof MuonFrequencyBands> = ['sub', 'bass', 'lowMid', 'mid', 'highMid', 'presence', 'brilliance']
|
||||
|
||||
function clamp01(value: number): number {
|
||||
return Math.max(0, Math.min(1, value))
|
||||
}
|
||||
|
||||
function avgSlice(data: Uint8Array, start: number, end: number): number {
|
||||
const s = Math.max(0, Math.min(data.length, start))
|
||||
const e = Math.max(s, Math.min(data.length, end))
|
||||
const span = e - s
|
||||
if (span <= 0) return 0
|
||||
let sum = 0
|
||||
for (let i = s; i < e; i += 1) sum += data[i]
|
||||
return sum / span
|
||||
}
|
||||
|
||||
function deriveDrumIntensity(bass: number, treble: number, rms: number): number {
|
||||
// Exponential scalers matching the corrected muon source audio.js pattern
|
||||
// (Bug 2 fix). Input range 0–1; outputs range 0–2.002 (normalised back below).
|
||||
// Original formula: 1.001 * (1 + avgFr * 2) - 1.001
|
||||
// This curve is near-zero when quiet and amplifies progressively when loud,
|
||||
// giving a more musical response than a plain linear mix.
|
||||
const expBass = 1.001 * (1 + bass * 2) - 1.001
|
||||
const expTreble = 1.001 * (1 + treble * 2) - 1.001
|
||||
// Coefficient ratio mirrors wave.js beatScaler (bass ~2× treble).
|
||||
// Divide by 2.0 to normalise the 0–2.002 range back to 0–1 before clamping.
|
||||
const raw = (expBass * 0.68 + rms * 0.24 + expTreble * 0.08) / 2.0
|
||||
function deriveDrumIntensity(sub: number, bass: number, rms: number): number {
|
||||
const expSub = 1.001 * (1 + sub * 2) - 1.001
|
||||
const expBass = 1.001 * (1 + bass * 2) - 1.001
|
||||
const raw = (expSub * 0.5 + expBass * 0.38 + rms * 0.12) / 2.0
|
||||
return clamp01(Math.pow(clamp01(raw), 0.82))
|
||||
}
|
||||
|
||||
export function useAudioFeatures({
|
||||
audioCtxRef,
|
||||
stemGainRefs,
|
||||
singleAudioRef,
|
||||
analysisAudioRef,
|
||||
isPlayingRef,
|
||||
enabled = true,
|
||||
analysisSourceKey,
|
||||
}: UseAudioFeaturesArgs): MutableRefObject<AudioFeatures> {
|
||||
const featuresRef = useRef<AudioFeatures>({
|
||||
bass: 0,
|
||||
@@ -89,155 +105,128 @@ export function useAudioFeatures({
|
||||
spectralCentroid: 0,
|
||||
drumIntensity: 0,
|
||||
beatDetected: false,
|
||||
frequencyBands: EMPTY_BANDS,
|
||||
beatInfo: EMPTY_BEAT,
|
||||
patternInfo: EMPTY_PAT,
|
||||
spectralRolloff: 0,
|
||||
spectralFlatness: 0,
|
||||
zcr: 0,
|
||||
freqData: EMPTY_FREQ,
|
||||
updatedAt: 0,
|
||||
})
|
||||
|
||||
const mediaSourceRef = useRef<MediaElementAudioSourceNode | null>(null)
|
||||
const sourceElementRef = useRef<HTMLAudioElement | null>(null)
|
||||
|
||||
useEffect(() => {
|
||||
if (!enabled) return
|
||||
|
||||
const ctx = audioCtxRef.current
|
||||
if (!ctx) return
|
||||
const audioEl = analysisAudioRef.current
|
||||
if (!ctx || !audioEl) return
|
||||
|
||||
const gains = stemGainRefs.current || {}
|
||||
const drumGain = gains.drums
|
||||
const fallbackGain = Object.values(gains)[0]
|
||||
|
||||
let tapNode: AudioNode | null = drumGain || fallbackGain || null
|
||||
|
||||
if (!tapNode) {
|
||||
const audioEl = singleAudioRef.current
|
||||
if (audioEl) {
|
||||
try {
|
||||
mediaSourceRef.current = mediaSourceRef.current ?? ctx.createMediaElementSource(audioEl)
|
||||
tapNode = mediaSourceRef.current
|
||||
} catch {
|
||||
tapNode = null
|
||||
}
|
||||
if (sourceElementRef.current && sourceElementRef.current !== audioEl) {
|
||||
try {
|
||||
mediaSourceRef.current?.disconnect()
|
||||
} catch {
|
||||
// noop
|
||||
}
|
||||
mediaSourceRef.current = null
|
||||
sourceElementRef.current = null
|
||||
}
|
||||
|
||||
if (!tapNode) return
|
||||
try {
|
||||
if (!mediaSourceRef.current) {
|
||||
mediaSourceRef.current = ctx.createMediaElementSource(audioEl)
|
||||
sourceElementRef.current = audioEl
|
||||
}
|
||||
} catch {
|
||||
return
|
||||
}
|
||||
|
||||
const mediaSource = mediaSourceRef.current
|
||||
if (!mediaSource) return
|
||||
|
||||
const analyser = ctx.createAnalyser()
|
||||
analyser.fftSize = 512
|
||||
analyser.smoothingTimeConstant = 0.82
|
||||
analyser.fftSize = 1024
|
||||
analyser.smoothingTimeConstant = 0.8
|
||||
analyser.minDecibels = -100
|
||||
analyser.maxDecibels = -30
|
||||
|
||||
const silentGain = ctx.createGain()
|
||||
silentGain.gain.value = 0
|
||||
mediaSource.connect(analyser)
|
||||
|
||||
// Non-intrusive tap: this does not rewire the existing audible output path.
|
||||
tapNode.connect(analyser)
|
||||
analyser.connect(silentGain)
|
||||
silentGain.connect(ctx.destination)
|
||||
|
||||
const byteFreq = new Uint8Array(analyser.frequencyBinCount)
|
||||
featuresRef.current.freqData = byteFreq
|
||||
|
||||
let rafId: number | null = null
|
||||
let mounted = true
|
||||
let prevEnergy = 0
|
||||
let lastBeatAt = 0
|
||||
|
||||
const applyMeydaFrame = (frame: MeydaFeatureFrame) => {
|
||||
if (!mounted) return
|
||||
if (typeof frame.rms === 'number' && Number.isFinite(frame.rms)) {
|
||||
featuresRef.current.rms = clamp01(frame.rms)
|
||||
}
|
||||
if (typeof frame.energy === 'number' && Number.isFinite(frame.energy)) {
|
||||
const e = clamp01(frame.energy)
|
||||
featuresRef.current.energy = e
|
||||
prevEnergy = e
|
||||
}
|
||||
if (typeof frame.spectralCentroid === 'number' && Number.isFinite(frame.spectralCentroid)) {
|
||||
featuresRef.current.spectralCentroid = Math.max(0, frame.spectralCentroid)
|
||||
}
|
||||
if (typeof frame.loudness === 'number' && Number.isFinite(frame.loudness)) {
|
||||
featuresRef.current.loudness = Math.max(0, frame.loudness)
|
||||
} else if (
|
||||
frame.loudness &&
|
||||
typeof frame.loudness !== 'number' &&
|
||||
typeof frame.loudness.total === 'number' &&
|
||||
Number.isFinite(frame.loudness.total)
|
||||
) {
|
||||
featuresRef.current.loudness = Math.max(0, frame.loudness.total)
|
||||
}
|
||||
const extractor = new MuonFeatureExtractor()
|
||||
const bandPeaks: Record<keyof MuonFrequencyBands, number> = {
|
||||
sub: 0.001,
|
||||
bass: 0.001,
|
||||
lowMid: 0.001,
|
||||
mid: 0.001,
|
||||
highMid: 0.001,
|
||||
presence: 0.001,
|
||||
brilliance: 0.001,
|
||||
}
|
||||
|
||||
let meydaAnalyzer: MeydaAnalyzerLike | null = null
|
||||
let mounted = true
|
||||
|
||||
// Keep Meyda optional. Using vite-ignore prevents hard dev-server failure
|
||||
// when a runtime image is missing node_modules/meyda.
|
||||
const meydaModuleId = 'meyda'
|
||||
void import(/* @vite-ignore */ meydaModuleId)
|
||||
.then((mod) => {
|
||||
if (!mounted) return
|
||||
const imported = mod as unknown as {
|
||||
default?: MeydaLike
|
||||
createMeydaAnalyzer?: MeydaLike['createMeydaAnalyzer']
|
||||
}
|
||||
const meyda: MeydaLike | undefined = imported.default
|
||||
?? (typeof imported.createMeydaAnalyzer === 'function' ? (imported as MeydaLike) : undefined)
|
||||
if (!meyda || typeof meyda.createMeydaAnalyzer !== 'function') return
|
||||
|
||||
meydaAnalyzer = meyda.createMeydaAnalyzer({
|
||||
audioContext: ctx,
|
||||
source: tapNode as AudioNode,
|
||||
bufferSize: 512,
|
||||
featureExtractors: ['rms', 'energy', 'loudness', 'spectralCentroid'],
|
||||
callback: applyMeydaFrame,
|
||||
})
|
||||
meydaAnalyzer?.start?.()
|
||||
})
|
||||
.catch(() => {
|
||||
// Optional dependency path: FFT mode still works if Meyda is unavailable.
|
||||
})
|
||||
|
||||
const tick = () => {
|
||||
extractor.start(analyser, ctx, (frame) => {
|
||||
if (!mounted) return
|
||||
rafId = window.requestAnimationFrame(tick)
|
||||
|
||||
if (!isPlayingRef.current) {
|
||||
featuresRef.current.bass = 0
|
||||
featuresRef.current.treble = 0
|
||||
featuresRef.current.rms = 0
|
||||
featuresRef.current.energy = 0
|
||||
featuresRef.current.loudness = 0
|
||||
featuresRef.current.drumIntensity = 0
|
||||
featuresRef.current.beatDetected = false
|
||||
featuresRef.current.updatedAt = Date.now()
|
||||
return
|
||||
}
|
||||
|
||||
analyser.getByteFrequencyData(byteFreq)
|
||||
const rawBands = frame.frequencyBands
|
||||
const norm = {
|
||||
sub: 0,
|
||||
bass: 0,
|
||||
lowMid: 0,
|
||||
mid: 0,
|
||||
highMid: 0,
|
||||
presence: 0,
|
||||
brilliance: 0,
|
||||
}
|
||||
|
||||
// Bug 2 fix: low bins (0–9) → bass, high bins (last 56 of 256) → treble.
|
||||
// Original muon source had these reversed (upperHalfArray → baseFr, etc.).
|
||||
const bass = avgSlice(byteFreq, 0, 10) / 255
|
||||
const treble = avgSlice(byteFreq, Math.max(0, byteFreq.length - 56), byteFreq.length) / 255
|
||||
const energy = byteFreq.length ? (avgSlice(byteFreq, 0, byteFreq.length) / 255) : 0
|
||||
const rms = featuresRef.current.rms > 0 ? featuresRef.current.rms : energy
|
||||
for (const key of BAND_KEYS) {
|
||||
bandPeaks[key] = Math.max(bandPeaks[key] * PEAK_DECAY, rawBands[key])
|
||||
norm[key] = clamp01(rawBands[key] / bandPeaks[key])
|
||||
}
|
||||
|
||||
featuresRef.current.bass = clamp01(bass)
|
||||
featuresRef.current.treble = clamp01(treble)
|
||||
featuresRef.current.energy = clamp01(energy)
|
||||
featuresRef.current.rms = clamp01(rms)
|
||||
featuresRef.current.drumIntensity = deriveDrumIntensity(featuresRef.current.bass, featuresRef.current.treble, featuresRef.current.rms)
|
||||
const rms = clamp01(frame.rms)
|
||||
const treble = clamp01((norm.presence + norm.brilliance) / 2)
|
||||
|
||||
const now = performance.now()
|
||||
const deltaEnergy = featuresRef.current.energy - prevEnergy
|
||||
const cooldownMs = 150
|
||||
const isBeat = deltaEnergy > 0.08 && featuresRef.current.bass > 0.25 && now - lastBeatAt > cooldownMs
|
||||
featuresRef.current.beatDetected = isBeat
|
||||
if (isBeat) lastBeatAt = now
|
||||
prevEnergy = featuresRef.current.energy
|
||||
|
||||
featuresRef.current.updatedAt = Date.now()
|
||||
}
|
||||
|
||||
rafId = window.requestAnimationFrame(tick)
|
||||
featuresRef.current = {
|
||||
bass: norm.bass,
|
||||
treble,
|
||||
rms,
|
||||
energy: clamp01(frame.energy),
|
||||
loudness: rms,
|
||||
spectralCentroid: frame.spectralCentroid,
|
||||
drumIntensity: deriveDrumIntensity(norm.sub, norm.bass, rms),
|
||||
beatDetected: frame.beatInfo.isBeat,
|
||||
frequencyBands: norm,
|
||||
beatInfo: frame.beatInfo,
|
||||
patternInfo: frame.patternInfo,
|
||||
spectralRolloff: frame.spectralRolloff,
|
||||
spectralFlatness: frame.spectralFlatness,
|
||||
zcr: frame.zcr,
|
||||
freqData: frame.floatFreqData,
|
||||
updatedAt: Date.now(),
|
||||
}
|
||||
})
|
||||
|
||||
return () => {
|
||||
mounted = false
|
||||
if (rafId !== null) window.cancelAnimationFrame(rafId)
|
||||
meydaAnalyzer?.stop?.()
|
||||
extractor.stop()
|
||||
try {
|
||||
tapNode?.disconnect(analyser)
|
||||
mediaSource.disconnect(analyser)
|
||||
} catch {
|
||||
// noop
|
||||
}
|
||||
@@ -246,13 +235,8 @@ export function useAudioFeatures({
|
||||
} catch {
|
||||
// noop
|
||||
}
|
||||
try {
|
||||
silentGain.disconnect()
|
||||
} catch {
|
||||
// noop
|
||||
}
|
||||
}
|
||||
}, [audioCtxRef, stemGainRefs, singleAudioRef, isPlayingRef, enabled])
|
||||
}, [audioCtxRef, analysisAudioRef, isPlayingRef, enabled, analysisSourceKey])
|
||||
|
||||
return featuresRef
|
||||
}
|
||||
@@ -102,6 +102,27 @@ function normalizeSekInnerPath(value?: string | null): string {
|
||||
.replace(/^\/+/, '')
|
||||
}
|
||||
|
||||
function rankSekAnalysisStem(name?: string): number {
|
||||
const key = (name || '').trim().toLowerCase()
|
||||
if (!key) return 10
|
||||
if (/\b(drums?|kick|percussion)\b/.test(key)) return 0
|
||||
if (/\b(leading|lead|vocals?|vox|voice|main vocal)\b/.test(key)) return 1
|
||||
if (/\b(backing|music|instrumental|mix)\b/.test(key)) return 2
|
||||
return 10
|
||||
}
|
||||
|
||||
function pickSekAnalysisStem(stems?: Array<{ name?: string; file?: string }>) {
|
||||
if (!stems?.length) return undefined
|
||||
return stems
|
||||
.filter((stem) => !!normalizeSekInnerPath(stem.file))
|
||||
.sort((a, b) => {
|
||||
const ra = rankSekAnalysisStem(a.name)
|
||||
const rb = rankSekAnalysisStem(b.name)
|
||||
if (ra !== rb) return ra - rb
|
||||
return (a.name || '').localeCompare(b.name || '')
|
||||
})[0]
|
||||
}
|
||||
|
||||
export default function KaraokePage() {
|
||||
const location = useLocation()
|
||||
const navigate = useNavigate()
|
||||
@@ -214,6 +235,20 @@ export default function KaraokePage() {
|
||||
}
|
||||
}, [detail, isSekMode, sekManifest, sekFolder])
|
||||
|
||||
const analysisAudioUrl = useMemo(() => {
|
||||
if (isSekMode) {
|
||||
const analysisStem = pickSekAnalysisStem(sekManifest?.manifest?.contents?.stems)
|
||||
const stemPath = normalizeSekInnerPath(analysisStem?.file)
|
||||
if (!sekManifest || !stemPath) return undefined
|
||||
return sekAssetUrl(sekManifest.id, stemPath, sekFolder || undefined)
|
||||
}
|
||||
|
||||
if (!detail) return undefined
|
||||
const sourceFile = detail.sep_job.files.find((f) => f.type === 'source')
|
||||
if (!sourceFile) return undefined
|
||||
return downloadUrl(detail.sep_job.id, sourceFile.name)
|
||||
}, [detail, isSekMode, sekManifest, sekFolder])
|
||||
|
||||
// ── Pixabay stock video fallback ───────────────────────────────────────────
|
||||
// DISABLED: visualizers are now the default backdrop for SEK packages without
|
||||
// an embedded video. Set needsStockVideo to the expression below to re-enable:
|
||||
@@ -462,6 +497,7 @@ export default function KaraokePage() {
|
||||
lyricsTracks={lyricsTracks}
|
||||
stems={safeStems.length > 0 ? safeStems : undefined}
|
||||
audioUrl={safeStems.length === 0 && sourceMedia ? sourceMedia.url : undefined}
|
||||
analysisAudioUrl={analysisAudioUrl}
|
||||
videoUrl={sourceMedia?.isVideo ? sourceMedia.url : undefined}
|
||||
videoPlaylist={stockVideoPlaylist.length > 0 ? stockVideoPlaylist : undefined}
|
||||
videoPlaybackMode={sourceMedia?.isVideo ? 'synced' : 'playlist'}
|
||||
|
||||
@@ -107,6 +107,27 @@ function normalizeSekInnerPath(value?: string | null): string {
|
||||
.replace(/^\/+/, '')
|
||||
}
|
||||
|
||||
function rankSekAnalysisStem(name?: string): number {
|
||||
const key = (name || '').trim().toLowerCase()
|
||||
if (!key) return 10
|
||||
if (/\b(drums?|kick|percussion)\b/.test(key)) return 0
|
||||
if (/\b(leading|lead|vocals?|vox|voice|main vocal)\b/.test(key)) return 1
|
||||
if (/\b(backing|music|instrumental|mix)\b/.test(key)) return 2
|
||||
return 10
|
||||
}
|
||||
|
||||
function pickSekAnalysisStem(stems?: Array<{ name?: string; file?: string }>) {
|
||||
if (!stems?.length) return undefined
|
||||
return stems
|
||||
.filter((stem) => !!normalizeSekInnerPath(stem.file))
|
||||
.sort((a, b) => {
|
||||
const ra = rankSekAnalysisStem(a.name)
|
||||
const rb = rankSekAnalysisStem(b.name)
|
||||
if (ra !== rb) return ra - rb
|
||||
return (a.name || '').localeCompare(b.name || '')
|
||||
})[0]
|
||||
}
|
||||
|
||||
export default function KaraokeTVPage() {
|
||||
const navigate = useNavigate()
|
||||
const {
|
||||
@@ -235,6 +256,20 @@ export default function KaraokeTVPage() {
|
||||
}
|
||||
}, [currentItem, detail, isSekMode, sekManifest, sekFolder])
|
||||
|
||||
const analysisAudioUrl = useMemo(() => {
|
||||
if (isSekMode) {
|
||||
const analysisStem = pickSekAnalysisStem(sekManifest?.manifest?.contents?.stems)
|
||||
const stemPath = normalizeSekInnerPath(analysisStem?.file)
|
||||
if (!sekManifest || !stemPath) return undefined
|
||||
return sekAssetUrl(sekManifest.id, stemPath, sekFolder || undefined)
|
||||
}
|
||||
|
||||
if (!detail) return undefined
|
||||
const sourceFile = detail.sep_job.files.find((f) => f.type === 'source')
|
||||
if (!sourceFile) return undefined
|
||||
return downloadUrl(detail.sep_job.id, sourceFile.name)
|
||||
}, [detail, isSekMode, sekManifest, sekFolder])
|
||||
|
||||
// ── Pixabay stock video fallback ───────────────────────────────────
|
||||
// DISABLED: visualizers are now the default backdrop for SEK packages without
|
||||
// an embedded video. Set needsStockVideo to the expression below to re-enable:
|
||||
@@ -531,6 +566,7 @@ export default function KaraokeTVPage() {
|
||||
lyricsTracks={lyricsTracks}
|
||||
stems={safeStems.length > 0 ? safeStems : undefined}
|
||||
audioUrl={safeStems.length === 0 && sourceMedia ? sourceMedia.url : undefined}
|
||||
analysisAudioUrl={analysisAudioUrl}
|
||||
videoUrl={sourceMedia?.isVideo ? sourceMedia.url : undefined}
|
||||
videoPlaylist={stockVideoPlaylist.length > 0 ? stockVideoPlaylist : undefined}
|
||||
videoPlaybackMode={sourceMedia?.isVideo ? 'synced' : 'playlist'}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { useEffect, useMemo, useRef, useState } from 'react'
|
||||
import { Disc3, Heart, Loader2, Zap } from 'lucide-react'
|
||||
import { Heart, Loader2, Zap } from 'lucide-react'
|
||||
import { Link, useLocation, useNavigate, useParams } from 'react-router-dom'
|
||||
import { useQuery } from '@tanstack/react-query'
|
||||
import { useConvertStatus } from '../hooks/useConvertStatus'
|
||||
@@ -9,7 +9,8 @@ import { useQueue } from '../context/QueueContext'
|
||||
import { trackMediaFromSong } from '../utils/mediaAdapters'
|
||||
import { launchMediaItem } from '../utils/queueLauncher'
|
||||
import MediaQueueActions from '../components/MediaQueueActions'
|
||||
import { getProjectKaraokeConfig, getSubsonicSong, patchProjectKaraokeConfig, subsonicCoverArtUrl } from '../api/client'
|
||||
import VisualizerPreviewWidget from '../components/VisualizerPreviewWidget'
|
||||
import { getProjectKaraokeConfig, getSubsonicSong, patchProjectKaraokeConfig, subsonicCoverArtUrl, subsonicStreamUrl } from '../api/client'
|
||||
import type { SubsonicSong } from '../types'
|
||||
import { reportNonFatal } from '../utils/nonFatal'
|
||||
|
||||
@@ -65,6 +66,7 @@ export default function TrackDetailPage() {
|
||||
const { statusBySongId, convertSong } = useConvertStatus(song ? [song] : [])
|
||||
const { playNow, playNext, addToQueue } = useQueue()
|
||||
const coverArtUrl = song?.coverArt ? subsonicCoverArtUrl(song.coverArt, 800) : ''
|
||||
const audioStreamUrl = useMemo(() => (song ? subsonicStreamUrl(song.id) : ''), [song?.id])
|
||||
|
||||
const matchedProject = song ? findProjectForSong(song) : null
|
||||
|
||||
@@ -154,13 +156,13 @@ export default function TrackDetailPage() {
|
||||
return (
|
||||
<div className="p-6 space-y-6">
|
||||
<div className="flex items-start gap-4">
|
||||
<div className="h-24 w-24 shrink-0 overflow-hidden rounded-lg bg-gray-800 flex items-center justify-center">
|
||||
{coverArtUrl ? (
|
||||
<img src={coverArtUrl} alt={`${song.album} cover art`} className="h-full w-full object-cover" />
|
||||
) : (
|
||||
<Disc3 className="h-10 w-10 text-gray-500" />
|
||||
)}
|
||||
</div>
|
||||
<VisualizerPreviewWidget
|
||||
audioUrl={audioStreamUrl}
|
||||
coverArtUrl={coverArtUrl || undefined}
|
||||
videoUrl={videoUrlInput.trim() || undefined}
|
||||
songTitle={song.title}
|
||||
className="w-48 shrink-0 overflow-hidden rounded-lg"
|
||||
/>
|
||||
<div className="min-w-0">
|
||||
<Link to={TRACKS_ROUTE} className="text-xs text-gray-400 hover:text-gray-300">← Back to tracks</Link>
|
||||
<h1 className="text-2xl font-bold mt-1 truncate">{song.title}</h1>
|
||||
|
||||
Vendored
+2
-28
@@ -1,28 +1,2 @@
|
||||
declare module 'meyda' {
|
||||
export interface MeydaFeatures {
|
||||
rms?: number
|
||||
energy?: number
|
||||
loudness?: number | { total?: number }
|
||||
spectralCentroid?: number
|
||||
}
|
||||
|
||||
export interface MeydaAnalyzer {
|
||||
start?: () => void
|
||||
stop?: () => void
|
||||
}
|
||||
|
||||
export interface MeydaCreateOptions {
|
||||
audioContext: AudioContext
|
||||
source: AudioNode
|
||||
bufferSize: number
|
||||
featureExtractors: string[]
|
||||
callback: (features: MeydaFeatures) => void
|
||||
}
|
||||
|
||||
export interface MeydaApi {
|
||||
createMeydaAnalyzer: (opts: MeydaCreateOptions) => MeydaAnalyzer
|
||||
}
|
||||
|
||||
const Meyda: MeydaApi
|
||||
export default Meyda
|
||||
}
|
||||
// Meyda runtime usage was removed in favor of MuonFeatureExtractor.
|
||||
// File intentionally left without module declarations.
|
||||
Vendored
+4
@@ -0,0 +1,4 @@
|
||||
declare module '*.mjs' {
|
||||
const mod: any
|
||||
export default mod
|
||||
}
|
||||
Vendored
+3
@@ -0,0 +1,3 @@
|
||||
declare module 'webgl-fluid' {
|
||||
export default function WebGLFluid(canvas: HTMLCanvasElement, options?: any): void
|
||||
}
|
||||
Reference in new issue
Block a user