Muon reactivity fix.
This commit is contained in:
1 parent
cb2ef648fe
commit
519b2c8dcd
27 files changed
+3495
-602
No files matched your search
@@ -8,7 +8,3 @@ MUON_IFRAME_URL=http://work:3011
|
||||
# Pinned Muon fork commit/tag/branch used by muon-iframe build context.
|
||||
# Examples: main, v0.1.0, 1a2b3c4d
|
||||
MUON_GIT_REF=main
|
||||
|
||||
# Open-origin mode for Muon iframe postMessage commands.
|
||||
# Use '*' to disable origin restrictions.
|
||||
STEMENGINE_ALLOWED_ORIGIN=*
|
||||
@@ -307,6 +307,98 @@ Validation completed:
|
||||
- Muon overlay JS syntax checks ✅
|
||||
- Docker image build: `docker compose -f stemengine.yml build stemengine-muon` ✅
|
||||
|
||||
### Muon no-reactivity — StrictMode MediaElementAudioSourceNode bug (2026-04-24)
|
||||
|
||||
**Root cause identified and fixed:** `React.StrictMode` double-runs effects in development. The `useAudioFeatures` effect's first run acquired a `MediaElementAudioSourceNode` (refs=1). StrictMode cleanup called `releaseMediaSource`, which decremented refs to 0 and **deleted the entry from `MEDIA_SOURCE_OWNERSHIP_BY_CTX`**. The second (real) effect run found no entry and called `ctx.createMediaElementSource(audioEl)` on an already-attached element → `InvalidStateError` → effect returned early → no analyser, no extractor, no features → zero RMS → silence gate fired → no postMessage pushes → no reactivity.
|
||||
|
||||
The same bug caused second-play failures in production (after the user paused and replayed).
|
||||
|
||||
**Fix:** `frontend/src/hooks/useAudioFeatures.ts` — `releaseMediaSource()`: removed `map.delete(audioEl)` when refs drop to 0. The entry stays in the `WeakMap` with `refs=0`. Subsequent `acquireMediaSource` calls find the existing node, increment refs, and return it for reconnection. The `WeakMap` evicts the entry automatically when the element is garbage-collected.
|
||||
|
||||
- Changed: `frontend/src/hooks/useAudioFeatures.ts` (releaseMediaSource: no-delete on refs=0)
|
||||
- Typecheck: `npm run -s typecheck` ✅
|
||||
- Prior session changes (already in place): `analyseOnly: true` in `LyricsLivePreview.tsx`, GainNode(gain=0) path in `useAudioFeatures.ts`, `SILENCE_GATE_FRAMES=120` in `muonAudioBridge.ts`, `getCtx()` for single-stem play in `LyricsLivePreview.tsx:2031`
|
||||
|
||||
### Muon default-mode reactivity — root cause analysis (2026-04-27)
|
||||
|
||||
**Investigation only, no code changes.** Compared StemEngine integration with the original Muon (`https://github.com/najafmohammed/muon-music-visualizer`, deployed at `https://muon-vis.netlify.app/`). Cloned upstream to `/tmp/muon-original/` for source-read comparison.
|
||||
|
||||
**Discovery:** Default Muon mode and the custom modes (`fluid`/`tendrils`/`spirit`) consume audio through two completely separate pathways:
|
||||
- **Default**: original Muon's `wavesurfer.backend.analyser` → `analyser.getByteFrequencyData()` → reads from a `<input type="file">` upload via `URL.createObjectURL()`. We never upload anything, so `dataArray` stays at all zeros and the visualizer renders idle animations.
|
||||
- **Custom modes**: read directly from `window.__STEMENGINE_MUON_AUDIO__` (set by our router from `pushAudioFeatures` postMessage). Work, but render graphs are simplified and lack the original's `audioProcessing()` exponential scaling.
|
||||
|
||||
The `__STEMENGINE_MUON_RESUME_FN__` / `__STEMENGINE_MUON_STOP_FN__` hooks at [stemengine-router.js:118-135](muon-iframe/stemengine-router.js#L118-L135) were placeholders for original-Muon integration that was never wired in. They remain no-ops, which also means the original Muon RAF loop never stops when a custom render graph is active — both renderers run on the same canvas, contributing to the "buggy" custom-mode visuals.
|
||||
|
||||
**Every prior Muon investigation (2026-04-20, 2026-04-24, the StrictMode fix above) targeted only the host-side analysis pipeline.** Those fixes were necessary but only affect what reaches `__STEMENGINE_MUON_AUDIO__` — they have zero effect on the default mode, which doesn't read that global.
|
||||
|
||||
**Findings revised (2026-04-27 second pass) to integrate `https://github.com/najafmohammed/Muon-Audio-Analysis`:**
|
||||
|
||||
Cloned Muon-Audio-Analysis to `/tmp/muon-audio-analysis/` for source-read. Key discovery from `diff src/audio/FeatureExtractor.ts` against StemEngine's `frontend/src/audio/MuonFeatureExtractor.ts`: **our host-side extractor is already an algorithmic port of Muon-Audio-Analysis's FeatureExtractor.** Same FFT_SIZE (1024), MIN/MAX_DECIBELS, 7-band frequency ranges, RMS/spectral/ZCR formulas, beat-detection ring buffer (43 frames), pattern detection (40-frame buildup window, 2.5× drop sensitivity). Differences are structural (callback vs Zustand store) not algorithmic.
|
||||
|
||||
**Revised architecture (in `docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/Findings.md`):** instead of synthesizing byte-bin data from pushed features, ship Muon-Audio-Analysis's `AudioEngine` + `FeatureExtractor` as vanilla-JS modules INSIDE the iframe. The iframe becomes a self-contained audio analysis + visualization black box:
|
||||
- Host sends only `loadAudio(url)` + transport commands (`transportPlay/Pause/Seek/Rate`); no per-frame `pushAudioFeatures` chatter
|
||||
- Iframe `AudioEngine` fetches the URL, decodes, and drives an `AudioBufferSourceNode → GainNode(0) → AnalyserNode → destination` graph (silent — host still plays audio for the user)
|
||||
- Iframe `FeatureExtractor` reads from the AnalyserNode each RAF, writes to `window.__STEMENGINE_MUON_AUDIO__` (richer payload than current: full 7-band spectrum + raw FFT Float32Array(512) + waveform Float32Array(1024))
|
||||
- One-line sed patch redirects `wavesurfer.backend.analyser` to our analyser → **default mode reacts** because `getByteFrequencyData()` now reads real bin data
|
||||
- Custom render graphs (fluid/tendrils/spirit) read the same global, gain access to per-band uniforms and raw FFT texture
|
||||
- Sed-patch registers `__STEMENGINE_MUON_STOP_FN__` / `__STEMENGINE_MUON_RESUME_FN__` to actually pause the original Muon RAF loop in custom modes → eliminates dual-renderer canvas conflict (root cause of "buggy" custom modes)
|
||||
|
||||
Six phases, each user-visible standalone: (1) port engine + extractor to iframe, (2) router learns transport commands, (3) sed-patch wavesurfer analyser → default reacts, (4) custom graphs read richer signals, (5) RAF coordination via STOP/RESUME hooks, (6) host-side cleanup of `pushAudioFeatures` push loop. Recommended MVP: phases 1–3 (~2–3 hours, single docker rebuild).
|
||||
|
||||
### Muon iframe — implementation README + cross-page coverage (2026-04-27 third pass)
|
||||
|
||||
Findings.md updated to reflect:
|
||||
- **Zero access control restrictions** along the entire path (CORS=`*`, no origin allowlist, no iframe sandbox, no auth on download URLs, no X-Frame-Options). The vestigial `STEMENGINE_ALLOWED_ORIGIN` build arg + sed step + env var should be deleted from the Dockerfile, the GitHub Actions workflow, and `.env.example` as cleanup.
|
||||
- **Coverage at every visualizer mounting site** (the Muon iframe appears in three places on the host: `LyricsLivePreview.tsx`, `VisualizerPreviewWidget.tsx`, `VisualizerLabPage.tsx`; each currently has its own duplicated iframe wiring). Phase 6 introduces a single shared hook `useMuonIframeTransport(...)` that all three call. `frontend/src/components/visualizers/muonAudioBridge.ts` becomes obsolete.
|
||||
|
||||
New file: `docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/README.md` — implementation contract / design spec ("how it SHOULD work"). Sections: architectural invariants (I-1 through I-7), data flow diagram, host hook contract (full TypeScript signature for `useMuonIframeTransport`), iframe contract (window globals table, command table, event table), wavesurfer integration (Phase 3 sed-patch detail), STOP/RESUME hook (Phase 5 sed-patch detail), access-control policy (explicitly NONE), per-mounting-site wiring guide, validation gates (19 checkpoints), open questions for the implementer.
|
||||
|
||||
Companion doc to Findings.md. Findings = rationale + migration path; README = target end state.
|
||||
|
||||
No files modified by this investigation; reference docs only.
|
||||
|
||||
### Muon iframe — review of Copilot's phase 1–6 implementation (2026-04-27 fourth pass)
|
||||
|
||||
After phases 1–6 were implemented (status flag set in Findings.md), audited the running code against the README's invariants and validation gates using the agency-agents skills (reality-checker, evidence-collector, software-architect, frontend-developer). Result: typecheck passes, files exist where the spec said, but **default mode still does not react and custom modes regress** because of timing/normalization defects.
|
||||
|
||||
Twelve defects identified in `docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/Review.md`. Four are critical:
|
||||
|
||||
- **D-1 — Race at iframe boot.** `engine.initialize()` is called from the router's `DOMContentLoaded` handler. The original Muon's `window.onload = vizInit()` invokes `vizInit` SYNCHRONOUSLY at module load (Muon-author bug), which calls `audioAnalyzer()` BEFORE `DOMContentLoaded` fires. So when the sed-patched `analyser = (window.__STEMENGINE_MUON_ANALYSER__ || wavesurfer.backend.analyser)` evaluates, the global is `undefined` → falls back to wavesurfer's silent analyser → default mode permanently bound to silent source.
|
||||
- **D-2 — Original Muon caches `analyser` once.** `audioAnalyzer()` only re-runs on file upload (which we don't do). Even if D-1 were fixed by retiming, no later `loadAudio` postMessage refreshes the binding.
|
||||
- **D-3 — Iframe FeatureExtractor publishes un-normalized band magnitudes.** Host's `useAudioFeatures.ts:287-350` had a peak-normalization layer (`bandPeaks` ring with `PEAK_DECAY=0.9998`); the iframe port omits it. Custom render graphs see `audio.bass: 0.04` instead of `0.4` — visible reactivity drops ~10×. Net regression vs the previous host-pushed pipeline.
|
||||
- **D-4 — AudioContext likely never resumes.** `engine.initialize()` runs at `DOMContentLoaded` (no user gesture). Chrome starts AudioContext in `'suspended'` state. Cross-document user activation rules don't propagate the host's play-button gesture into the iframe document. `ctx.resume()` rejects silently (catch-and-swallow at engine line 80). AnalyserNode returns `-Infinity` for all bins → all four modes see silence → no reactivity.
|
||||
|
||||
Plus eight more (D-5 sandbox attribute kept despite README invariant I-6; D-6 LyricsViewport ignores hook's `onIframeLoad`; D-7 STOP_FN/RESUME_FN registered inside `play()` closure not module scope; D-8 STEMENGINE_ALLOWED_ORIGIN cleanup not done; D-9–12 cosmetic / spec drift).
|
||||
|
||||
Of the README's 19 validation gates, **11 fail and none of the user-visible reactivity gates pass.** Recommended fix order: D-4 (silent killer) → D-1 (race) → D-3 (normalization) → D-7 (Phase 5 timing) → D-5/D-6/D-8 cleanup. None work standalone; all four critical fixes must land together.
|
||||
|
||||
The structural work is sound — file layout, hook signature, command surface, three-mounting-site coverage all match the README. The failures are timing, normalization, and cleanup — not architecture. Do not close the underlying issue.
|
||||
|
||||
### Muon iframe — applied fixes for D-1, D-3, D-5, D-7, D-8 (2026-04-27 fifth pass)
|
||||
|
||||
Implemented the critical fixes identified in `Review.md`. Document detail in `docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/Fixes.md`.
|
||||
|
||||
**Files changed:**
|
||||
- `muon-iframe/stemengine-audio-engine.js` — D-1: call `initialize()` synchronously at the end of the IIFE so `__STEMENGINE_MUON_ANALYSER__` is populated before the original Muon's `vizInit()` runs and binds its `analyser` reference. (Auto-fixes D-2 since Muon's binding now points to our singleton AnalyserNode.)
|
||||
- `muon-iframe/stemengine-feature-extractor.js` — D-3: added per-band `bandPeaks` (PEAK_DECAY=0.9998, PEAK_FLOOR=0.001) plus `rmsPeak` and `energyPeak`; published values are now `clamp01(raw / peak)` instead of raw RMS-shaped magnitudes. Critical detail: `detectBeat()` and `detectPatterns()` still operate on RAW energy (not normalized) so the variance-based beat detector keeps working. Also corrected `treble` derivation to `(presence + brilliance) / 2` to match host convention.
|
||||
- `muon-iframe/Dockerfile` — D-7: split the Phase 5 sed into two patches. STOP_FN/RESUME_FN now register at module scope (only need module-scope `pauseAnimation`); a sentinel global `__STEMENGINE_MUON_RAF_KICK__ = animate` is set inside `play()` so RESUME_FN can defer the actual `animate()` invocation. Module-scope hooks are robust against future Muon updates that defer `play()`. — D-8: removed the `STEMENGINE_ALLOWED_ORIGIN` build arg and the corresponding sed step.
|
||||
- `muon-iframe/stemengine-router.js` — D-8: removed `ALLOWED_ORIGIN_RAW`, `PLACEHOLDER_TOKEN`, `ALLOWED_ORIGINS`, `ACCEPT_ANY_ORIGIN`, `HAS_ALLOWED_ORIGINS` constants. The module-doc comment now points at `docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/README.md` section 8 for the zero-access-control rationale.
|
||||
- `frontend/src/components/LyricsViewport.tsx`, `frontend/src/components/VisualizerPreviewWidget.tsx`, `frontend/src/pages/VisualizerLabPage.tsx` — D-5: removed `sandbox="allow-scripts allow-same-origin"` from all three iframe elements (kept `allow="autoplay"`).
|
||||
- `.env.example` — D-8: removed the `STEMENGINE_ALLOWED_ORIGIN=*` env var line.
|
||||
- `/home/translator/docker/stemengine.yml` — D-8: removed `STEMENGINE_ALLOWED_ORIGIN=${STEMENGINE_ALLOWED_ORIGIN:-*}` from the `stemengine-muon` build args block.
|
||||
|
||||
**Reclassifications from the Review:**
|
||||
- D-2 (Muon caches analyser once) — Auto-fixed by D-1. Muon's cached analyser reference now points to our singleton AnalyserNode whose lifetime equals the iframe document. Subsequent `loadAudio` postMessages just attach a new `AudioBufferSourceNode` to the same analyser node; Muon's reference stays valid.
|
||||
- D-4 (AudioContext suspended) — Downgraded from "critical" to "mitigated by `allow="autoplay"` + parent activation". The iframe's `allow="autoplay"` attribute (already in place at all three sites) inherits the host's transient user activation; the existing `ctx.resume().catch(() => {})` in `engine.play()` is sufficient provided D-5 is also fixed (sandbox was blocking activation inheritance). If field testing shows D-4 is still a problem, the next-step fix is a one-time click-overlay inside the iframe; not implemented now.
|
||||
- D-6 (`onIframeLoad` ignored in LyricsLivePreview path) — False positive. Wiring already correct via `setMuonIframeReady={(ready) => { if (ready) onMuonIframeLoad() }}` in LyricsLivePreview.tsx:2575. Added a clarifying comment to LyricsViewport's onLoad.
|
||||
|
||||
**Deferred (cosmetic / non-functional):**
|
||||
- D-9 (drift-correction `seek()` churn — iframe is muted, no audible artifact)
|
||||
- D-10/11/12 (`audioMode: 'hybrid'` cosmetic spec drift)
|
||||
- `STEMENGINE_ALLOWED_ORIGIN` in `.github/workflows/muon-iframe-build-publish.yml` and historical `20260313_Fixes (Copilot).md` / `20260417_*` docs (CI hygiene + append-only history; out of scope)
|
||||
|
||||
**Validation:** `npm run -s typecheck` exit 0. `grep -rn 'STEMENGINE_ALLOWED_ORIGIN'` returns zero matches in live code (only in docs/CI/historical). `grep -n 'sandbox' frontend/src/components/{LyricsViewport,VisualizerPreviewWidget}.tsx frontend/src/pages/VisualizerLabPage.tsx` returns zero matches. Browser-level validation requires `docker compose build stemengine-muon && docker compose up -d stemengine-muon` then manual smoke test of all four modes at all three mounting sites; not run from this session.
|
||||
|
||||
Open items remaining for full Phase 5 closeout:
|
||||
|
||||
- Phase 5.3 promotions (`candidate` / `approved`) require the 10-track validation suite and side-by-side review.
|
||||
@@ -5101,3 +5193,25 @@ Rationale: the commands the router accepts are all inert (audio scalars, style/q
|
||||
|
||||
**Files changed:** `muon-iframe/stemengine-router.js`
|
||||
**Validation:** `node --check` ✅ · image rebuilt (`bundle.9bcc3c4c7ee7c4a45ad6.js`) · container recreated and running on port 3011 ✅
|
||||
|
||||
## 2026-04-24
|
||||
|
||||
### Muon visualizer — no-reactivity root cause fix
|
||||
|
||||
Investigated and fixed zero audio reactivity on the Muon iframe at `/projects/:id`.
|
||||
|
||||
**Root cause 1 (critical) — dead-end Web Audio graph** (`useAudioFeatures.ts`):
|
||||
`mediaSource.connect(analyser)` without connecting the analyser to `ctx.destination`. Chrome/Firefox don't process a subgraph that doesn't reach the destination, so `getFloatFrequencyData()` returned silence and RMS was always 0.
|
||||
|
||||
**Root cause 2 (compounding) — silence gate fires permanently** (`muonAudioBridge.ts`):
|
||||
Because RMS was always 0, the 60-frame silence gate (`silentFrames >= 60 → return`) fired after 1 second and suppressed every subsequent push forever.
|
||||
|
||||
**Edge case handled — LyricsLivePreview double audio**: `LyricsLivePreview` uses a separate hidden analysis `<audio>` element alongside the stem transport. Connecting that element's analyser directly to `ctx.destination` would produce audible double audio alongside the stems. Fixed with an `analyseOnly` option that routes through a `GainNode(gain=0)` instead — graph stays active for analysis, output is silent.
|
||||
|
||||
**Files changed:**
|
||||
- `frontend/src/hooks/useAudioFeatures.ts`: Added `analyseOnly?: boolean` to `UseAudioFeaturesArgs`. Default false → `analyser.connect(ctx.destination)` (VisualizerPreviewWidget, VisualizerLabPage). True → `analyser → GainNode(0) → ctx.destination` silent passthrough (LyricsLivePreview). GainNode disconnected in effect cleanup. `analyseOnly` added to effect deps.
|
||||
- `frontend/src/components/LyricsLivePreview.tsx`: Added `analyseOnly: true` to `useAudioFeatures` call.
|
||||
- `frontend/src/components/visualizers/muonAudioBridge.ts`: Raised `SILENCE_GATE_FRAMES` 60 → 120 (~2 s warm-up margin).
|
||||
- `docs/plans/2026-04-24_Muon_No_Reactivity/Findings.md`: Investigation findings with root causes, infrastructure status, and fix checklist.
|
||||
|
||||
**Validation:** `cd frontend && npm run -s typecheck` ✅
|
||||
@@ -253,11 +253,21 @@ def get_similar_songs(song_id: str, count: int = 5):
|
||||
|
||||
|
||||
@router.get("/stream/{song_id}")
|
||||
def stream_song(song_id: str, request: Request):
|
||||
def stream_song(
|
||||
song_id: str,
|
||||
request: Request,
|
||||
format: str | None = Query(default=None, description="Optional output format hint (e.g. mp3)"),
|
||||
maxBitRate: int | None = Query(default=None, ge=32, le=512, description="Optional transcode bitrate in kbps"),
|
||||
):
|
||||
"""Proxy-stream audio for a Subsonic song id to the browser."""
|
||||
range_header = request.headers.get('range')
|
||||
try:
|
||||
upstream = subsonic_service.stream_song(song_id, range_header=range_header)
|
||||
upstream = subsonic_service.stream_song(
|
||||
song_id,
|
||||
range_header=range_header,
|
||||
audio_format=format,
|
||||
max_bit_rate=maxBitRate,
|
||||
)
|
||||
except RuntimeError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc))
|
||||
except Exception as exc:
|
||||
|
||||
@@ -925,7 +925,13 @@ def import_song(
|
||||
}
|
||||
|
||||
|
||||
def stream_song(song_id: str, range_header: str | None = None):
|
||||
def stream_song(
|
||||
song_id: str,
|
||||
range_header: str | None = None,
|
||||
*,
|
||||
audio_format: str | None = None,
|
||||
max_bit_rate: int | None = None,
|
||||
):
|
||||
"""Return an upstream streaming response for a Subsonic song id."""
|
||||
config = load_config()
|
||||
if not config:
|
||||
@@ -933,6 +939,12 @@ def stream_song(song_id: str, range_header: str | None = None):
|
||||
|
||||
p = _params(config)
|
||||
p['id'] = song_id
|
||||
if audio_format:
|
||||
cleaned_format = ''.join(ch for ch in str(audio_format).lower() if ch.isalnum())
|
||||
if cleaned_format:
|
||||
p['format'] = cleaned_format
|
||||
if isinstance(max_bit_rate, int) and max_bit_rate > 0:
|
||||
p['maxBitRate'] = str(max_bit_rate)
|
||||
headers = {}
|
||||
if range_header:
|
||||
headers['Range'] = range_header
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
# Muon Visualizer — No Reactivity Investigation
|
||||
|
||||
**Date:** 2026-04-24
|
||||
**Symptom:** Muon iframe loads but does not react to audio at `http://work:5174/projects/{id}`
|
||||
**Method:** Parallel source-read agents (frontend pipeline + iframe infrastructure)
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
There are two root causes. Fix #1 is the critical one; Fix #2 prevents the gate from suppressing the warm-up window.
|
||||
|
||||
| # | Root cause | File | Fix |
|
||||
|---|---|---|---|
|
||||
| 1 | **AnalyserNode not connected to `ctx.destination`** — Web Audio graph is a dead-end; analyser produces silence | `useAudioFeatures.ts:261` | Add `analyser.connect(ctx.destination)` |
|
||||
| 2 | **Silence gate fires permanently** — 60 consecutive RMS < 0.001 frames suppress all pushes forever | `muonAudioBridge.ts:103–105` | Add a warm-up grace period before the gate activates |
|
||||
|
||||
---
|
||||
|
||||
## Infrastructure: confirmed working
|
||||
|
||||
Everything below the audio data path is fine:
|
||||
|
||||
| Component | Status |
|
||||
|---|---|
|
||||
| iframe origin check | Open — `isAllowedOrigin()` returns `true` unconditionally |
|
||||
| `pushAudioFeatures` command routing | Present in `KNOWN_COMMANDS`, dispatches to `captureFeatures()` |
|
||||
| `captureFeatures()` storage | Writes all fields to `window.__STEMENGINE_MUON_AUDIO__` correctly |
|
||||
| nginx | No CORS or blocking headers |
|
||||
| Container on port 3011 | Running (`stemengine-stemengine-muon-1`, `bundle.0b9149b...`) |
|
||||
| `crossOrigin="anonymous"` on audio element | Set — `VisualizerPreviewWidget.tsx:684` |
|
||||
| AudioContext resume on play | Done — `VisualizerPreviewWidget.tsx:665–666` |
|
||||
| `muonIframeReadyRef` | Set on iframe `onLoad` — line 707 |
|
||||
| Bridge wired | `createAudioPushLoop(sendMuonCommand, () => featuresRef.current)` — line 337 |
|
||||
|
||||
---
|
||||
|
||||
## Root Cause 1 — Dead-end Web Audio graph
|
||||
|
||||
### Location
|
||||
`frontend/src/hooks/useAudioFeatures.ts`, line 261
|
||||
|
||||
### What the code does
|
||||
```ts
|
||||
mediaSource.connect(analyser)
|
||||
// analyser is never connected to ctx.destination
|
||||
```
|
||||
|
||||
### Why this breaks analysis
|
||||
When `ctx.createMediaElementSource(audioEl)` is called, the browser transfers the audio element's output from its native pipeline into the Web Audio graph. From that point, audio only flows through the graph. Since `analyser` is never connected to `ctx.destination`:
|
||||
|
||||
1. **No audio plays through speakers** — the Web Audio graph has no sink, so the browser outputs silence. (Depending on the browser, the audio element may continue to output via its native pipeline if the MediaElementAudioSourceNode creation silently fails — but this is not reliable.)
|
||||
|
||||
2. **The AnalyserNode produces silence** — Chrome and Firefox will not process a Web Audio subgraph that doesn't connect to the destination. `getFloatFrequencyData()` returns all `-Infinity` (or `minDecibels`), and `getFloatTimeDomainData()` returns zeros. Computed RMS = 0 always.
|
||||
|
||||
3. **Every feature stays at zero** — `featuresRef.current` contains all-zero values every frame.
|
||||
|
||||
### Fix
|
||||
```ts
|
||||
// useAudioFeatures.ts — after line 261
|
||||
mediaSource.connect(analyser)
|
||||
analyser.connect(ctx.destination) // ← add this
|
||||
```
|
||||
|
||||
This routes audio through the analyser to the speakers, makes the graph active, and gives the AnalyserNode real FFT data.
|
||||
|
||||
**For the muted analysis element path** (Phase 1 plan — `LyricsLivePreview` muted `originalAudioEl`): the element is muted at the `HTMLAudioElement` level, which means no sound plays through the OS even when routed through `ctx.destination`. The `muted` property suppresses OS-level output. The AnalyserNode still receives data. No additional change needed for that path.
|
||||
|
||||
---
|
||||
|
||||
## Root Cause 2 — Silence gate fires permanently
|
||||
|
||||
### Location
|
||||
`frontend/src/components/visualizers/muonAudioBridge.ts`, lines 103–105
|
||||
|
||||
### What the code does
|
||||
```ts
|
||||
const SILENCE_GATE_FRAMES = 60 // ~1 second at 60fps
|
||||
|
||||
if ((rms ?? 0) < 0.001) {
|
||||
silentFrames++
|
||||
if (silentFrames >= SILENCE_GATE_FRAMES) return // suppresses ALL subsequent pushes
|
||||
} else {
|
||||
silentFrames = 0
|
||||
}
|
||||
```
|
||||
|
||||
### Why this fires permanently
|
||||
Because of Root Cause 1, RMS is always 0. The gate counts 60 consecutive silent frames (1 second), then the `return` statement suppresses every subsequent push. The gate never resets because RMS never rises above 0.001.
|
||||
|
||||
Even if Root Cause 1 is fixed, there is still a warm-up window: the bridge starts when `isPlaying` becomes true (bridge effect line 336), and `createAudioPushLoop` is called at the same moment. For the first ~10–30 frames the new extractor's peak normalizers haven't seen any signal, so RMS values are low. A 60-frame grace period should absorb this without issues — but the current code activates the gate from frame 0 with no grace period, so even brief spikes reset the counter and the gate is delayed, not skipped.
|
||||
|
||||
### Fix — add a warm-up window before the gate activates
|
||||
```ts
|
||||
// muonAudioBridge.ts
|
||||
const SILENCE_GATE_FRAMES = 60
|
||||
const SILENCE_GATE_WARMUP_FRAMES = 120 // ← add: 2s grace before gate can fire
|
||||
|
||||
// in tick():
|
||||
if ((rms ?? 0) < 0.001) {
|
||||
silentFrames++
|
||||
if (silentFrames >= SILENCE_GATE_FRAMES + SILENCE_GATE_WARMUP_FRAMES) return
|
||||
} else {
|
||||
silentFrames = 0
|
||||
}
|
||||
```
|
||||
|
||||
Or, simpler — just increase `SILENCE_GATE_FRAMES` to 180 (3 seconds). The gate exists only to avoid hammering the iframe with zeros during genuine silence (paused, track ended). 3 seconds is still conservative.
|
||||
|
||||
---
|
||||
|
||||
## Secondary issue — AudioContext reuse across ctxVersion
|
||||
|
||||
`ensureCtx()` (`VisualizerPreviewWidget.tsx:192–198`) creates a new `AudioContext` and bumps `ctxVersion` whenever the old one is null or closed. When `ctxVersion` changes, `useAudioFeatures` tries to call `ctx.createMediaElementSource(audioEl)` on the new context. But the spec forbids associating an audio element with more than one AudioContext. If the old context was not closed before the new one runs its effect, the call throws `InvalidStateError`, which the hook catches silently (line 240–246) and returns — leaving `featuresRef.current` as `zeroFeatures()` indefinitely.
|
||||
|
||||
This is a latent issue, less likely than Root Cause 1 (it only bites on second+ plays if the old context wasn't cleaned up in time). Addressed by the same fix as #1: once the graph is active and the context isn't being needlessly recreated, the ctxVersion bump is less likely to occur mid-session.
|
||||
|
||||
---
|
||||
|
||||
## Proposed fix checklist
|
||||
|
||||
| Step | File | Change |
|
||||
|---|---|---|
|
||||
| 1 | `frontend/src/hooks/useAudioFeatures.ts:261` | Add `analyser.connect(ctx.destination)` after `mediaSource.connect(analyser)` |
|
||||
| 2 | `frontend/src/components/visualizers/muonAudioBridge.ts:74` | Increase `SILENCE_GATE_FRAMES` from 60 to 180 (or add `SILENCE_GATE_WARMUP_FRAMES = 120`) |
|
||||
| 3 | `cd frontend && npm run -s typecheck` | Zero errors |
|
||||
| 4 | Browser validation | Play a track; open DevTools console; run `window.parent.__STEMENGINE_MUON_AUDIO__` from inside the iframe (or check via host page); confirm `rms` is non-zero and `bass`/`treble` update each second |
|
||||
| 5 | Confirm audio plays through speakers | With the fix, audio should play through speakers (previously it was silently routed to a dead-end graph) |
|
||||
|
||||
---
|
||||
|
||||
## What was not the problem
|
||||
|
||||
- **Origin allowlist** — already open in the deployed bundle (`isAllowedOrigin → return true`)
|
||||
- **CORS on the audio element** — `crossOrigin="anonymous"` is correctly set
|
||||
- **Iframe URL** — `http://work:3011` default is correct; `onLoad` fires correctly
|
||||
- **Command naming** — `pushAudioFeatures` matches the router's `KNOWN_COMMANDS`
|
||||
- **`isPlaying` guard** — bridge correctly starts/stops with playback
|
||||
- **Per-stem analysis competing** — removed in Phase 1; `useStemAudio` is playback-only
|
||||
@@ -0,0 +1,741 @@
|
||||
# Muon Visualizer — Unified Audio Analysis Architecture (All Modes)
|
||||
|
||||
**Date:** 2026-04-27
|
||||
**Symptom:** Default Muon style renders correctly but does NOT react to audio. Custom modes (`fluid`, `tendrils`, `spirit`) react but exhibit visual glitches.
|
||||
**References:**
|
||||
- Original visualizer: https://muon-vis.netlify.app/ — https://github.com/najafmohammed/muon-music-visualizer
|
||||
- Audio analysis library: https://github.com/najafmohammed/Muon-Audio-Analysis
|
||||
**Method:** Source-read of `muon-iframe/`, original Muon repo (`/tmp/muon-original/`), Muon-Audio-Analysis repo (`/tmp/muon-audio-analysis/`), running container bundle, host-side wiring
|
||||
**Agent skills referenced:** `docs/agents/agency-agents/adapted/{software-architect,frontend-developer,backend-architect,evidence-collector,reality-checker}.md`
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
### Implementation status (2026-04-27)
|
||||
|
||||
- Phase 1 to Phase 6 are now implemented.
|
||||
- Phase 6 consolidation is complete: host-side transport wiring now lives in `frontend/src/hooks/useMuonIframeTransport.ts` and is used by all three Muon iframe mounting sites (`LyricsLivePreview`, `VisualizerPreviewWidget`, `VisualizerLabPage`).
|
||||
- Legacy host push bridge usage was removed from Muon iframe consumers.
|
||||
|
||||
The default mode and the custom modes consume audio through **two completely separate pathways** that share nothing:
|
||||
|
||||
| Mode | Audio source the renderer reads | Why it (does/doesn't) react |
|
||||
|---|---|---|
|
||||
| **default** | `wavesurfer.backend.analyser.getByteFrequencyData()` — fed by `wavesurfer.load(URL.createObjectURL(file))` from the iframe's file-upload `<input type="file">` | We never upload → analyser is silent → no reactivity |
|
||||
| **fluid / tendrils / spirit** | `window.__STEMENGINE_MUON_AUDIO__` — set by the router from `pushAudioFeatures` postMessage | Reacts. But uses simplified shader graphs; no exponential scaling; runs on the same canvas as the still-active original Muon RAF loop → visual artifacts |
|
||||
|
||||
**Critical insight from comparing repositories:** StemEngine's host-side [`frontend/src/audio/MuonFeatureExtractor.ts`](frontend/src/audio/MuonFeatureExtractor.ts) **is already an algorithmic port of [`Muon-Audio-Analysis/src/audio/FeatureExtractor.ts`](https://github.com/najafmohammed/Muon-Audio-Analysis/blob/main/src/audio/FeatureExtractor.ts).** Same FFT_SIZE (1024), same MIN/MAX_DECIBELS, identical 7-band frequency ranges, identical RMS/spectral-centroid/rolloff/flatness/ZCR formulas, identical beat-detection energy-history logic (43-frame ring), identical pattern-detection (buildup window 40 frames, drop sensitivity 2.5×). The only structural differences:
|
||||
- Ours emits per-frame frames via callback; theirs writes to a Zustand store
|
||||
- Ours adds `seedBpm()` and `shouldAnalyze()` (StemEngine-specific lifecycle gates)
|
||||
- Theirs has a placeholder `mfcc: Float32Array(13)` field (not implemented)
|
||||
|
||||
So we already produce semantically equivalent features to what Muon-Audio-Analysis produces. The integration gap isn't *what* we extract — it's that the iframe never sees a real `AnalyserNode` connected to a real audio source, and the original Muon code requires exactly that.
|
||||
|
||||
---
|
||||
|
||||
## How the original Muon works (reference)
|
||||
|
||||
[`/tmp/muon-original/src/index.js`](https://github.com/najafmohammed/muon-music-visualizer/blob/main/src/index.js) line 81–286:
|
||||
|
||||
```js
|
||||
wavesurfer = WaveSurfer.create(waveSurferParams)
|
||||
|
||||
const audioAnalyzer = () => {
|
||||
analyser = wavesurfer.backend.analyser // owned by WaveSurfer
|
||||
analyser.fftSize = 512
|
||||
dataArray = new Uint8Array(analyser.frequencyBinCount) // 256 bins
|
||||
}
|
||||
|
||||
file.onchange = function () {
|
||||
files[0] && wavesurfer.load(URL.createObjectURL(files[0]))
|
||||
wavesurfer.on("ready", () => {
|
||||
wavesurfer.play()
|
||||
audioAnalyzer() // re-acquire analyser
|
||||
})
|
||||
}
|
||||
|
||||
const render = () => {
|
||||
// ...
|
||||
analyser.getByteFrequencyData(dataArray) // ← every frame
|
||||
CoreControls.audioProcessing(dataArray) // → exponential scalers
|
||||
CoreControls.sineWavePropagation(..., dataArray, ...)
|
||||
emitParticle(..., dataArray, ...)
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
The reference visualizer at https://muon-vis.netlify.app/ also doesn't react until you upload a file. The reactivity is hard-bound to `wavesurfer.backend.analyser` having a real source.
|
||||
|
||||
---
|
||||
|
||||
## How Muon-Audio-Analysis structures its pipeline
|
||||
|
||||
[`Muon-Audio-Analysis/src/audio/AudioEngine.ts`](https://github.com/najafmohammed/Muon-Audio-Analysis/blob/main/src/audio/AudioEngine.ts):
|
||||
|
||||
```ts
|
||||
class AudioEngine {
|
||||
initialize() {
|
||||
this.audioContext = new AudioContext({ sampleRate: 44100 })
|
||||
this.analyzerNode = this.audioContext.createAnalyser()
|
||||
this.analyzerNode.fftSize = 1024
|
||||
this.gainNode = this.audioContext.createGain()
|
||||
this.gainNode.connect(this.analyzerNode)
|
||||
this.analyzerNode.connect(this.audioContext.destination)
|
||||
}
|
||||
|
||||
async loadFile(file: File) {
|
||||
const arrayBuffer = await file.arrayBuffer()
|
||||
this.audioBuffer = await this.audioContext.decodeAudioData(arrayBuffer)
|
||||
}
|
||||
|
||||
play() {
|
||||
this.sourceNode = this.audioContext.createBufferSource()
|
||||
this.sourceNode.buffer = this.audioBuffer
|
||||
this.sourceNode.connect(this.gainNode)
|
||||
this.sourceNode.start(0, this.pausedAt)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
[`FeatureExtractor.ts`](https://github.com/najafmohammed/Muon-Audio-Analysis/blob/main/src/audio/FeatureExtractor.ts) then:
|
||||
|
||||
```ts
|
||||
start(analyzerNode: AnalyserNode, audioContext: AudioContext) {
|
||||
this.analyzerNode = analyzerNode // re-uses the AudioEngine's analyser
|
||||
this.extractLoop()
|
||||
}
|
||||
|
||||
private extractLoop = () => {
|
||||
this.analyzerNode.getFloatFrequencyData(this.frequencyData)
|
||||
this.analyzerNode.getFloatTimeDomainData(this.timeData)
|
||||
// → RMS, spectralCentroid/rolloff/flatness, ZCR, energy, frequencyBands, beatInfo, patternInfo
|
||||
}
|
||||
```
|
||||
|
||||
The architecture is: **one AnalyserNode + one decoded AudioBuffer + one FeatureExtractor → consumed by both visualization and feature-store.** Single source of truth.
|
||||
|
||||
The lift to bring this into the iframe is small because:
|
||||
- `AudioEngine` is ~240 lines; depends only on Web Audio + a Zustand store (the store dependency is replaceable with a window global)
|
||||
- `FeatureExtractor` is the same code we already have at `frontend/src/audio/MuonFeatureExtractor.ts`. Drop the Zustand `store.set*()` calls and write to `window.__STEMENGINE_MUON_AUDIO__` instead.
|
||||
|
||||
---
|
||||
|
||||
## Proposed architecture: iframe-owned audio + analysis (all four modes)
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────┐ ┌──────────────────────────────────────┐
|
||||
│ HOST (frontend/src) │ │ IFRAME (muon-iframe/) │
|
||||
│ │ │ │
|
||||
│ LyricsLivePreview / VisualizerPreview │ │ ┌─ stemengine-audio-engine.js ──┐ │
|
||||
│ ─ owns: audioRef (transport) for user │ │ │ AudioEngine │ │
|
||||
│ ─ owns: analysisAudioRef (silent) │ │ │ + AudioContext │ │
|
||||
│ using useAudioFeatures (gain=0) │ │ │ + AnalyserNode (fftSize 1024)│ │
|
||||
│ ─ Optional: stop pushing features & │ │ │ + GainNode (gain=0) │ │
|
||||
│ let iframe own analysis entirely │ │ │ + AudioBufferSourceNode │ │
|
||||
│ │ │ └────────────┬──────────────────┘ │
|
||||
│ postMessage: │ │
|
||||
│ │ ┌─loadAudio(url)──→│ │
|
||||
│ │ ├─play(time, rate)→│ │
|
||||
│ │ ├─pause()──────────→│ │
|
||||
│ │ ├─seek(time)───────→│ │
|
||||
│ │ └─setRate(rate)────→│ │
|
||||
│ │ │ │
|
||||
│ │ ▼ │
|
||||
│ │ ┌─ stemengine-feature-extractor.js ──────┐ │
|
||||
│ │ │ FeatureExtractor (port of Muon-AA) │ │
|
||||
│ │ │ - reads analyserNode each RAF │ │
|
||||
│ │ │ - writes window.__STEMENGINE_MUON_AUDIO__│ │
|
||||
│ │ └────────────────────────────────────────┘ │
|
||||
│ │ │ │
|
||||
│ ┌── postMessage (back to host) ───────┼───←─── { features } │ │
|
||||
│ │ { type: 'features', payload: {…} │ (optional, for HUD) │ │
|
||||
│ └────────────────────────────────────┘ │ │
|
||||
│ │ │
|
||||
│ │ ┌─── DEFAULT MODE: original Muon ────────┐ │
|
||||
│ │ │ wavesurfer.backend.analyser │ │
|
||||
│ │ │ ↑ replaced/aliased by our analyser │ │
|
||||
│ │ │ getByteFrequencyData() returns │ │
|
||||
│ │ │ real bins (fed by AudioBufferSource)│ │
|
||||
│ │ └────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ │ ┌─── FLUID / TENDRILS / SPIRIT ──────────┐ │
|
||||
│ │ │ window.__STEMENGINE_MUON_AUDIO__ │ │
|
||||
│ │ │ reads features (richer than today) │ │
|
||||
│ │ │ AnalyserNode.getFloatFrequencyData() │ │
|
||||
│ │ │ also available to graphs │ │
|
||||
│ │ └────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
└────────────────────────────────────────┘ └──────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Single source of audio truth in the iframe
|
||||
|
||||
One `AudioContext`. One `AnalyserNode`. One decoded `AudioBuffer`. The Muon-Audio-Analysis `AudioEngine` shape, hosted inside the iframe.
|
||||
|
||||
- **Default mode**: replace `wavesurfer.backend.analyser` with our analyser (or hook wavesurfer to use our context). The original Muon `getByteFrequencyData()` calls now read real data, and the visualizer reacts as designed.
|
||||
- **Custom modes**: the FeatureExtractor publishes the full feature object to `window.__STEMENGINE_MUON_AUDIO__`. Render graphs continue to read it (no API break). They additionally gain access to `analyserNode.getFloatFrequencyData()` for per-bin spectrum, allowing them to match the original Muon's per-bin equalizer behavior.
|
||||
|
||||
### Host responsibilities shrink
|
||||
|
||||
The host stops pushing features. It only sends transport commands:
|
||||
- `loadAudio({ url, mimeType? })` — invoked when the visualizer mounts or the source changes
|
||||
- `play({ time, rate })` — invoked on user play; iframe starts playback at `time`
|
||||
- `pause()` — iframe stops the source
|
||||
- `seek(time)` — iframe creates a new source at offset
|
||||
- `setRate(rate)` — iframe applies playback rate
|
||||
|
||||
The iframe is muted (its GainNode stays at 0). The host plays the actual audio for the user (existing behavior, unchanged). The iframe just runs the analysis silently in parallel.
|
||||
|
||||
This removes:
|
||||
- The `pushAudioFeatures` 60-Hz postMessage chatter
|
||||
- The host-side `useAudioFeatures` analyser-on-analysis-element complexity (still needed for non-Muon visualizers like the existing canvas-based ones, but Muon stops depending on it)
|
||||
- All edge cases around StrictMode `MediaElementAudioSourceNode` reacquisition (recently fixed; will become irrelevant for Muon)
|
||||
- The silence-gate state machine in `muonAudioBridge.ts`
|
||||
|
||||
### Time synchronization
|
||||
|
||||
The iframe needs to stay aligned with the host's playback transport. Two options:
|
||||
|
||||
1. **Time-driven sync** (chosen): host posts `play({ time })` whenever user-driven seek/play happens, plus periodic `seek(time)` corrections (~250ms cadence) if drift exceeds 50ms. The iframe never owns the timeline — it always slaves to the host. This mirrors what `LyricsLivePreview.tsx:1559-1584` already does between the transport audio and the analysis audio.
|
||||
2. **MediaStreamTrack sync** (alternative, only with same-origin host): host could pipe `audioRef.captureStream()` through `postMessage` (Transferable) to give the iframe the same MediaStream. Browser support is good (Chrome 73+, Firefox 84+); same-origin requirement aligns with current deployment.
|
||||
|
||||
Option 1 is simpler and cross-origin safe.
|
||||
|
||||
### Access control: explicitly NONE
|
||||
|
||||
Per project policy (carried over from the 2026-04-23 origin-allowlist removal): **no access control restrictions are enforced anywhere along this path.**
|
||||
|
||||
- Backend (`backend/main.py`): `CORS_ORIGINS=*`, `CORS_ORIGIN_REGEX=.*`. Already in place after the compose simplification. Audio download endpoints (`/api/jobs/.../download/...`, `/api/subsonic/stream/...`) require no auth header from the iframe or host.
|
||||
- Iframe router (`muon-iframe/stemengine-router.js`): `isAllowedOrigin()` returns `true` unconditionally. Already in place. The `STEMENGINE_ALLOWED_ORIGIN` build arg becomes vestigial — recommendation: delete it from the Dockerfile, the GH workflow, and `.env.example` to remove dead surface area.
|
||||
- Iframe `fetch(url)`: `mode: 'cors', credentials: 'omit'`. No cookies, no tokens. The iframe identifies itself by nothing — the backend serves anyone.
|
||||
- Host `<iframe>` tag: no `sandbox` attribute (would block AudioContext + WebGL); no `referrerPolicy` constraints; no Content-Security-Policy `frame-src` narrowing.
|
||||
- Backend nginx in front of the iframe: no `X-Frame-Options` header (must allow framing from any origin); no CSP `frame-ancestors` constraint.
|
||||
|
||||
The deployment posture is "internal-only network, no exfiltration risk, prioritize Just Working over hardening." Any future hardening would re-introduce origin checks at exactly the points listed above.
|
||||
|
||||
---
|
||||
|
||||
## Phased implementation plan
|
||||
|
||||
### Phase 1 — Iframe gains AudioEngine + FeatureExtractor
|
||||
|
||||
**New files (under `muon-iframe/`):**
|
||||
|
||||
#### `stemengine-audio-engine.js`
|
||||
|
||||
Port of `Muon-Audio-Analysis/src/audio/AudioEngine.ts`, vanilla JS, no React/Zustand. ~250 lines. Key API:
|
||||
|
||||
```js
|
||||
;(function stemengineAudioEngine() {
|
||||
'use strict'
|
||||
|
||||
var ctx = null
|
||||
var analyser = null
|
||||
var gain = null
|
||||
var sourceNode = null
|
||||
var buffer = null
|
||||
var startedAt = 0
|
||||
var pausedAt = 0
|
||||
var isPlaying = false
|
||||
|
||||
function initialize() {
|
||||
if (ctx) return
|
||||
ctx = new (window.AudioContext || window.webkitAudioContext)({ sampleRate: 44100 })
|
||||
analyser = ctx.createAnalyser()
|
||||
analyser.fftSize = 1024
|
||||
analyser.smoothingTimeConstant = 0.8
|
||||
analyser.minDecibels = -100
|
||||
analyser.maxDecibels = -30
|
||||
gain = ctx.createGain()
|
||||
gain.gain.value = 0 // analysis-only — never audible
|
||||
gain.connect(analyser)
|
||||
analyser.connect(ctx.destination)
|
||||
|
||||
// Expose for downstream consumers (default mode + custom render graphs)
|
||||
window.__STEMENGINE_MUON_ANALYSER__ = analyser
|
||||
window.__STEMENGINE_MUON_AUDIO_CTX__ = ctx
|
||||
}
|
||||
|
||||
async function loadFromUrl(url) {
|
||||
initialize()
|
||||
stop()
|
||||
var resp = await fetch(url, { mode: 'cors', credentials: 'omit' })
|
||||
if (!resp.ok) throw new Error('audio fetch failed: ' + resp.status)
|
||||
var arr = await resp.arrayBuffer()
|
||||
buffer = await ctx.decodeAudioData(arr)
|
||||
pausedAt = 0
|
||||
return { duration: buffer.duration }
|
||||
}
|
||||
|
||||
function play(opts) {
|
||||
if (!buffer || !ctx) return
|
||||
if (ctx.state === 'suspended') ctx.resume()
|
||||
if (sourceNode) try { sourceNode.stop() } catch (_) {}
|
||||
sourceNode = ctx.createBufferSource()
|
||||
sourceNode.buffer = buffer
|
||||
if (opts && typeof opts.rate === 'number') sourceNode.playbackRate.value = opts.rate
|
||||
sourceNode.connect(gain)
|
||||
var offset = (opts && typeof opts.time === 'number') ? opts.time : pausedAt
|
||||
sourceNode.start(0, Math.max(0, Math.min(offset, buffer.duration)))
|
||||
startedAt = ctx.currentTime - offset
|
||||
pausedAt = 0
|
||||
isPlaying = true
|
||||
}
|
||||
|
||||
function pause() {
|
||||
if (!isPlaying || !ctx) return
|
||||
pausedAt = ctx.currentTime - startedAt
|
||||
isPlaying = false
|
||||
if (sourceNode) {
|
||||
try { sourceNode.stop() } catch (_) {}
|
||||
try { sourceNode.disconnect() } catch (_) {}
|
||||
sourceNode = null
|
||||
}
|
||||
}
|
||||
|
||||
function stop() { pause(); pausedAt = 0; startedAt = 0 }
|
||||
function seek(time) {
|
||||
var wasPlaying = isPlaying
|
||||
if (wasPlaying) pause()
|
||||
pausedAt = Math.max(0, Math.min(time, buffer ? buffer.duration : Infinity))
|
||||
if (wasPlaying) play({ time: pausedAt })
|
||||
}
|
||||
function setRate(rate) {
|
||||
if (sourceNode) sourceNode.playbackRate.value = rate
|
||||
}
|
||||
|
||||
window.__StemEngineAudioEngine__ = {
|
||||
initialize: initialize,
|
||||
loadFromUrl: loadFromUrl,
|
||||
play: play, pause: pause, stop: stop, seek: seek, setRate: setRate,
|
||||
getAnalyser: function () { return analyser },
|
||||
getContext: function () { return ctx },
|
||||
getCurrentTime: function () {
|
||||
if (!ctx) return 0
|
||||
return isPlaying ? (ctx.currentTime - startedAt) : pausedAt
|
||||
},
|
||||
getDuration: function () { return buffer ? buffer.duration : 0 },
|
||||
}
|
||||
})()
|
||||
```
|
||||
|
||||
#### `stemengine-feature-extractor.js`
|
||||
|
||||
Port of `Muon-Audio-Analysis/src/audio/FeatureExtractor.ts` (and our existing `MuonFeatureExtractor.ts`) to vanilla JS. Reads from `window.__STEMENGINE_MUON_ANALYSER__`. Writes to `window.__STEMENGINE_MUON_AUDIO__`. Same algorithms, no Zustand. ~400 lines.
|
||||
|
||||
The output object shape is the same one custom render graphs already read:
|
||||
|
||||
```js
|
||||
window.__STEMENGINE_MUON_AUDIO__ = {
|
||||
bass, treble, rms, energy, beat,
|
||||
drumIntensity, sub, beatPhase, bpm, beatConfidence,
|
||||
isBeatDrop, isBuildUp, isSilence, energyDelta,
|
||||
spectralCentroid, spectralFlatness,
|
||||
// NEW (richer than current pushAudioFeatures payload):
|
||||
frequencyBands: { sub, bass, lowMid, mid, highMid, presence, brilliance },
|
||||
fft: <Float32Array(512)>, // raw float frequency data
|
||||
waveform: <Float32Array(1024)>, // raw time-domain data
|
||||
__seUpdatedAt: <number>,
|
||||
}
|
||||
```
|
||||
|
||||
Backwards compatible — existing render graphs that read `audio.bass`, `audio.treble`, etc. continue to work unchanged. New consumers can use `audio.fft` / `audio.waveform` directly.
|
||||
|
||||
### Phase 2 — Router learns transport commands
|
||||
|
||||
[`muon-iframe/stemengine-router.js`](muon-iframe/stemengine-router.js):
|
||||
|
||||
Add to `KNOWN_COMMANDS`:
|
||||
|
||||
```js
|
||||
var KNOWN_COMMANDS = {
|
||||
setStyle: true, setPreset: true, setQuality: true, setPaused: true,
|
||||
setViewport: true, pushAudioFeatures: true,
|
||||
// NEW:
|
||||
loadAudio: true, transportPlay: true, transportPause: true, transportSeek: true, transportRate: true,
|
||||
}
|
||||
```
|
||||
|
||||
Handlers:
|
||||
|
||||
```js
|
||||
case 'loadAudio':
|
||||
window.__StemEngineAudioEngine__.loadFromUrl(String(p.url || ''))
|
||||
.then(function (info) {
|
||||
postToParent({ version: 1, type: 'loaded', payload: { duration: info.duration } })
|
||||
// Kick the feature extractor now that an analyser is live with real data
|
||||
window.__StemEngineFeatureExtractor__.start()
|
||||
})
|
||||
.catch(function (err) {
|
||||
window.__stemengineEmitError__('AUDIO_LOAD', String(err))
|
||||
})
|
||||
break
|
||||
|
||||
case 'transportPlay':
|
||||
window.__StemEngineAudioEngine__.play({ time: Number(p.time) || 0, rate: Number(p.rate) || 1 })
|
||||
break
|
||||
|
||||
case 'transportPause':
|
||||
window.__StemEngineAudioEngine__.pause()
|
||||
break
|
||||
|
||||
case 'transportSeek':
|
||||
window.__StemEngineAudioEngine__.seek(Number(p.time) || 0)
|
||||
break
|
||||
|
||||
case 'transportRate':
|
||||
window.__StemEngineAudioEngine__.setRate(Number(p.rate) || 1)
|
||||
break
|
||||
```
|
||||
|
||||
`pushAudioFeatures` stays for backwards compat (until Phase 4 deprecation), but becomes a no-op when `__StemEngineFeatureExtractor__` is active (last-write-wins on the global, with our local extractor running every frame and host pushes overwriting briefly between).
|
||||
|
||||
### Phase 3 — Default-mode integration: replace wavesurfer's analyser
|
||||
|
||||
The original Muon's `audioAnalyzer()` function ([src/index.js:101-106](https://github.com/najafmohammed/muon-music-visualizer/blob/main/src/index.js#L101)) reads `wavesurfer.backend.analyser`. We need this to BE our analyser.
|
||||
|
||||
Two approaches, in order of preference:
|
||||
|
||||
**3a) Patch `audioAnalyzer()` to use our analyser.** Add a sed-injected line in the Dockerfile:
|
||||
|
||||
```dockerfile
|
||||
RUN sed -i 's|analyser = wavesurfer.backend.analyser|analyser = (window.__STEMENGINE_MUON_ANALYSER__ || wavesurfer.backend.analyser)|' /src/app/src/index.js
|
||||
```
|
||||
|
||||
If our analyser is initialized (after `loadAudio`), Muon uses it. Otherwise it falls back to wavesurfer's silent analyser. The patch is one line and survives upstream Muon updates as long as the variable name doesn't change.
|
||||
|
||||
The `dataArray` is still `Uint8Array(256)` (since `analyser.fftSize=512` would be set by Muon — but we set 1024). To make the bin counts compatible, the simplest path is to ALSO sed `analyser.fftSize = 512` to keep wavesurfer's expected bin count, OR override `getByteFrequencyData` to downsample our 512-bin output to 256 bins. The first is simpler:
|
||||
|
||||
```dockerfile
|
||||
RUN sed -i 's|analyser.fftSize = 512|analyser.fftSize = analyser.fftSize|' /src/app/src/index.js
|
||||
```
|
||||
|
||||
Then in our `AudioEngine.initialize()`, set `fftSize = 512` explicitly so Muon's expectations align. This is fine — Muon-Audio-Analysis uses 1024 by default but works at any power-of-2.
|
||||
|
||||
**3b) Don't patch Muon — proxy wavesurfer's analyser.** Monkey-patch `WaveSurfer` after it's available:
|
||||
|
||||
```js
|
||||
// After wavesurfer.backend exists:
|
||||
Object.defineProperty(wavesurfer.backend, 'analyser', {
|
||||
get: function () { return window.__STEMENGINE_MUON_ANALYSER__ || nativeAnalyser },
|
||||
configurable: true,
|
||||
})
|
||||
```
|
||||
|
||||
More fragile (depends on wavesurfer's internal API). Prefer 3a.
|
||||
|
||||
### Phase 4 — Custom modes get richer signals
|
||||
|
||||
Update [`fluid-render-graph.js`](muon-iframe/fluid-render-graph.js), [`tendrils-render-graph.js`](muon-iframe/tendrils-render-graph.js), [`spirit-render-graph.js`](muon-iframe/spirit-render-graph.js) to read the new fields:
|
||||
|
||||
```js
|
||||
var audio = window.__STEMENGINE_MUON_AUDIO__ || {}
|
||||
|
||||
// Existing (still works):
|
||||
gl.uniform1f(uBass, clamp01(audio.bass))
|
||||
|
||||
// NEW — full 7-band spectrum for per-band shader work:
|
||||
var bands = audio.frequencyBands || {}
|
||||
gl.uniform1f(uSub, clamp01(bands.sub))
|
||||
gl.uniform1f(uLowMid, clamp01(bands.lowMid))
|
||||
gl.uniform1f(uMid, clamp01(bands.mid))
|
||||
gl.uniform1f(uHighMid, clamp01(bands.highMid))
|
||||
gl.uniform1f(uPresence, clamp01(bands.presence))
|
||||
gl.uniform1f(uBrilliance, clamp01(bands.brilliance))
|
||||
|
||||
// NEW — full FFT as a 1D texture for visualizers that want per-bin response:
|
||||
if (audio.fft && audio.fft.length === fftTextureSize) {
|
||||
gl.bindTexture(gl.TEXTURE_2D, fftTexture)
|
||||
gl.texImage2D(gl.TEXTURE_2D, 0, gl.LUMINANCE, audio.fft.length, 1, 0, gl.LUMINANCE, gl.FLOAT, audio.fft)
|
||||
}
|
||||
```
|
||||
|
||||
Also: apply the original Muon's `audioProcessing()` exponential scaling helper ([CoreControls/audio.js](https://github.com/najafmohammed/muon-music-visualizer/blob/main/src/CoreControls/audio.js)) to give custom modes the same dynamic range the default mode has. The user's "buggy" complaint partly reflects how raw clamped values lack the original's snap.
|
||||
|
||||
### Phase 5 — Stop the original Muon RAF loop in custom modes
|
||||
|
||||
[`muon-iframe/stemengine-router.js`](muon-iframe/stemengine-router.js) `updateRenderGraph()` already calls `__STEMENGINE_MUON_STOP_FN__()` and `__STEMENGINE_MUON_RESUME_FN__()`. These are no-ops because the original Muon code never registers them.
|
||||
|
||||
Sed-patch `index.js` to register them (one-liner per hook):
|
||||
|
||||
```dockerfile
|
||||
RUN sed -i 's|const animate = () => {|window.__STEMENGINE_MUON_RESUME_FN__ = () => { pauseAnimation = false; animate() }; window.__STEMENGINE_MUON_STOP_FN__ = () => { pauseAnimation = true }; const animate = () => {|' /src/app/src/index.js
|
||||
```
|
||||
|
||||
With this, switching to `fluid` actually pauses the original Muon's RAF loop. No more dual-renderer canvas conflict. This eliminates the largest single source of "buggy" complaints in custom modes.
|
||||
|
||||
### Phase 6 — Host-side simplification: ONE shared transport hook used wherever a Muon iframe appears
|
||||
|
||||
The Muon iframe currently appears in **three** mounting locations on the host, each with a duplicated copy of the same iframe-wiring code (iframeRef, ready handshake, sendMuonCommand, push-features bridge):
|
||||
|
||||
| Mounting site | File | Used by |
|
||||
|---|---|---|
|
||||
| `LyricsLivePreview` | [`frontend/src/components/LyricsLivePreview.tsx`](frontend/src/components/LyricsLivePreview.tsx) | `ProjectDetail` → `LrcPreview`, `Karaoke`, `KaraokeTV`, `KaraokePerformanceMode`, `MobileKaraokeShell`, `SongPreviewSheet` (all wrap or render this component) |
|
||||
| `VisualizerPreviewWidget` | [`frontend/src/components/VisualizerPreviewWidget.tsx`](frontend/src/components/VisualizerPreviewWidget.tsx) | `TrackDetail` (Subsonic/Navidrome library tracks) |
|
||||
| `VisualizerLabPage` | [`frontend/src/pages/VisualizerLabPage.tsx`](frontend/src/pages/VisualizerLabPage.tsx) | `/visualizer-lab` standalone testing page |
|
||||
|
||||
All three need the new transport flow. To avoid triplicating logic AGAIN, consolidate the iframe wiring into a single hook that every mounting site uses.
|
||||
|
||||
#### New: `frontend/src/hooks/useMuonIframeTransport.ts`
|
||||
|
||||
Single source of truth for the host ↔ iframe contract. Every visualizer mount calls this hook.
|
||||
|
||||
```ts
|
||||
export interface UseMuonIframeTransportArgs {
|
||||
iframeRef: RefObject<HTMLIFrameElement | null>
|
||||
isActive: boolean // mode === 'muon_iframe' && audioUrl present
|
||||
audioUrl: string | null
|
||||
isPlayingRef: RefObject<boolean>
|
||||
getCurrentTime: () => number // host's transport clock (audio.currentTime)
|
||||
getRate?: () => number // optional playback rate (default 1)
|
||||
/** Visualizer config — re-sent on every iframe load so state survives reloads. */
|
||||
style: 'default' | 'fluid' | 'tendrils' | 'spirit'
|
||||
preset?: string
|
||||
quality?: 'low' | 'balanced' | 'high'
|
||||
viewport?: { cssWidth: number; cssHeight: number; dpr: number }
|
||||
}
|
||||
|
||||
export interface UseMuonIframeTransportResult {
|
||||
/** Attach to iframe onLoad to mark ready and trigger initial load. */
|
||||
onIframeLoad: () => void
|
||||
/** Imperative escape hatch for advanced state pushes (rarely needed). */
|
||||
sendCommand: (type: string, payload: Record<string, unknown>) => void
|
||||
/** Loaded audio duration reported back from iframe (or null until ready). */
|
||||
loadedDuration: number | null
|
||||
}
|
||||
|
||||
export function useMuonIframeTransport(args: UseMuonIframeTransportArgs): UseMuonIframeTransportResult
|
||||
```
|
||||
|
||||
Internal behavior:
|
||||
- `sendCommand(type, payload)` posts `{ version: 1, id, type, payload }` to `iframeRef.current.contentWindow` with `'*'` target origin (host-side wildcard matches the iframe-side `isAllowedOrigin → true`).
|
||||
- On `onIframeLoad`: marks `muonIframeReadyRef = true`, sends initial `setStyle/setPreset/setQuality/setViewport`, then `loadAudio({ url: audioUrl })`.
|
||||
- A single RAF loop watches `isPlayingRef.current` and `getCurrentTime()`:
|
||||
- Transition `false → true`: send `transportPlay({ time, rate })`
|
||||
- Transition `true → false`: send `transportPause()`
|
||||
- While playing: if `|currentTime - lastSentTime| > 0.25s`, send `transportSeek({ time })` (drift correction)
|
||||
- On `getRate()` change: send `transportRate({ rate })`
|
||||
- On `audioUrl` change while active: send `loadAudio({ url })` and reset `lastSentTime = -1`.
|
||||
- On `style` change: send `setStyle({ style })`.
|
||||
- On `isActive → false`: send `transportPause()` and tear down the RAF loop.
|
||||
- Listens for `message` events from the iframe; when `event.data.type === 'loaded'`, captures `payload.duration` and exposes via `loadedDuration`.
|
||||
|
||||
#### Consumer integration (each mounting site)
|
||||
|
||||
**`LyricsLivePreview.tsx`**: replace the local `iframeRef`, `muonIframeReadyRef`, `sendMuonCommand`, `createAudioPushLoop` block with a single call:
|
||||
|
||||
```ts
|
||||
const { onIframeLoad, sendCommand, loadedDuration } = useMuonIframeTransport({
|
||||
iframeRef,
|
||||
isActive: isIframeVisualizerMode && Boolean(audioUrl),
|
||||
audioUrl: audioUrl ?? null,
|
||||
isPlayingRef,
|
||||
getCurrentTime: () => currentTimeRef.current,
|
||||
style: liveVizSettings.muon_iframe.style,
|
||||
preset: liveVizSettings.muon_iframe.preset,
|
||||
quality: liveVizSettings.muon_iframe.quality,
|
||||
viewport: { cssWidth: visualizerSize.width, cssHeight: visualizerSize.height, dpr: window.devicePixelRatio },
|
||||
})
|
||||
```
|
||||
|
||||
The existing `useAudioFeatures()` call stays (still drives the in-house canvas visualizers and the lyrics gloss effects). The Muon-specific bridge (`createAudioPushLoop`) is deleted.
|
||||
|
||||
**`VisualizerPreviewWidget.tsx`**: same replacement. The TrackDetail flow now becomes: host plays the Subsonic stream URL via `audioRef`; iframe fetches the same URL (CORS-permissive backend proxy) and runs its own analysis silently in parallel; transport sync keeps them within 250ms.
|
||||
|
||||
**`VisualizerLabPage.tsx`**: same. The lab page exists to test iframe behavior in isolation; consolidating the wiring means the lab tests the same code path production uses.
|
||||
|
||||
#### Result: ONE transport flow used WHEREVER visualizers appear
|
||||
|
||||
After Phase 6:
|
||||
|
||||
```
|
||||
┌─ LyricsLivePreview ─┐
|
||||
│ │
|
||||
│ useMuonIframeTransport(...)
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
└───────┼─────────────┘
|
||||
│
|
||||
┌─ VisualizerPreviewWidget ─┐
|
||||
│ │
|
||||
│ useMuonIframeTransport(...)│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
└───────┼────────────────────┘
|
||||
│
|
||||
┌─ VisualizerLabPage ─┐
|
||||
│ │
|
||||
│ useMuonIframeTransport(...)
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
└───────┼─────────────┘
|
||||
│
|
||||
▼
|
||||
sendMuonCommand (single implementation)
|
||||
│
|
||||
▼
|
||||
<iframe src="http://work:3011/...">
|
||||
│
|
||||
▼
|
||||
stemengine-router.js (single iframe contract)
|
||||
│
|
||||
├─→ AudioEngine.loadFromUrl(...)
|
||||
├─→ AudioEngine.play({time, rate})
|
||||
├─→ AudioEngine.pause()
|
||||
├─→ AudioEngine.seek(time)
|
||||
└─→ AudioEngine.setRate(rate)
|
||||
│
|
||||
▼
|
||||
FeatureExtractor (port of Muon-Audio-Analysis)
|
||||
│
|
||||
▼
|
||||
window.__STEMENGINE_MUON_AUDIO__
|
||||
│
|
||||
├─→ original Muon (default mode, via patched wavesurfer.backend.analyser)
|
||||
├─→ fluid-render-graph
|
||||
├─→ tendrils-render-graph
|
||||
└─→ spirit-render-graph
|
||||
```
|
||||
|
||||
Three host components, one hook, one iframe contract, one analysis pipeline, four visualizer modes — all consistent.
|
||||
|
||||
**The host's `useAudioFeatures` hook stays for non-Muon visualizers** (the in-house canvas-based ones — `MoireVisualizerCanvas`, `DissolveVisualizerCanvas`, `GlitchVisualizerCanvas`, `SpiralVisualizerCanvas`, `FluidVisualizerCanvas`, `TendrilsVisualizerCanvas`, `SpiritVisualizerCanvas`, `MuonVisualizerCanvas`). Those are independent renderers that don't go through the iframe; they continue to consume `useAudioFeatures` directly. **Only the Muon iframe path stops consuming it.**
|
||||
|
||||
---
|
||||
|
||||
## Why this fixes both reported problems
|
||||
|
||||
### "Default doesn't react"
|
||||
|
||||
After Phase 3 (single sed-line patch in Dockerfile + iframe AudioEngine running), the original Muon's `analyser.getByteFrequencyData()` reads from a real AudioBufferSourceNode → AnalyserNode pipeline. The visualizer reacts as designed by its author.
|
||||
|
||||
### "Other modes are buggy"
|
||||
|
||||
Three contributing causes, addressed in order:
|
||||
|
||||
1. **Dual-renderer canvas conflict** (largest cause) — fixed in Phase 5: original Muon RAF stops when fluid/tendrils/spirit are active. No more flicker, no more phantom geometry.
|
||||
2. **Lossy audio data** — fixed in Phase 4: custom render graphs gain access to per-band frequency energies (7 bands instead of 2 lumped values) and the raw 512-bin FFT. Per-band shader work now feasible.
|
||||
3. **Lack of dynamic-range conditioning** — fixed in Phase 4 by applying Muon-Audio-Analysis's normalization (the FeatureExtractor produces values that have already been through the same shaping as the host pipeline). Plus optionally porting `audioProcessing()`'s exponential scalers as helper functions for custom render graphs.
|
||||
|
||||
---
|
||||
|
||||
## Files touched
|
||||
|
||||
| Phase | New | Edit |
|
||||
|---|---|---|
|
||||
| 1 | `muon-iframe/stemengine-audio-engine.js`, `muon-iframe/stemengine-feature-extractor.js` | `muon-iframe/Dockerfile` (3 COPY + adjust import order) |
|
||||
| 2 | — | `muon-iframe/stemengine-router.js` (add 5 commands; ~30 lines) |
|
||||
| 3 | — | `muon-iframe/Dockerfile` (1–2 sed lines) |
|
||||
| 4 | — | `muon-iframe/{fluid,tendrils,spirit}-render-graph.js` (read richer audio fields) |
|
||||
| 5 | — | `muon-iframe/Dockerfile` (1 sed line registering STOP/RESUME hooks) |
|
||||
| 6 | `frontend/src/hooks/useMuonIframeTransport.ts` (single shared hook) | `frontend/src/components/LyricsLivePreview.tsx`, `frontend/src/components/VisualizerPreviewWidget.tsx`, `frontend/src/pages/VisualizerLabPage.tsx` (call new hook, delete duplicated wiring); delete `frontend/src/components/visualizers/muonAudioBridge.ts` (obsolete) |
|
||||
| Cleanup | — | Delete `STEMENGINE_ALLOWED_ORIGIN` from `muon-iframe/Dockerfile`, `muon-iframe/stemengine-router.js` (vestigial constants/sed step), `.github/workflows/muon-iframe-build-publish.yml` (build arg), `.env.example` (env var) |
|
||||
|
||||
No backend changes required (CORS_ORIGINS=*, CORS_ORIGIN_REGEX=.* already in place from the compose simplification, no auth required on download endpoints).
|
||||
|
||||
---
|
||||
|
||||
## Validation checklist
|
||||
|
||||
### Targeted interactive pass (2026-04-27)
|
||||
|
||||
The following manual runtime checks were executed after Phase 6 refactor deployment.
|
||||
|
||||
1. `VisualizerPreviewWidget` (Track detail, Sing app @ `:3001`) — **pass**
|
||||
- Opened `Tracks` and navigated to track detail (`/tracks/a5f76485ba8f7e5c41aea2c6a41ca0eb`).
|
||||
- Confirmed Muon iframe mounts in the widget (`src=http://work:3011/?preset=default&se_rev=20260422b`) with no "Could not load Muon iframe" overlay.
|
||||
- Confirmed transport interaction:
|
||||
- 30s preview `Play` transitions to `Pause`.
|
||||
- Preview timer advances (`0:00 -> 0:16 -> 0:25` observed).
|
||||
- Confirmed mode wiring still works through shared hook:
|
||||
- Clicking `fluid` toggles style active state.
|
||||
- Preset set updates to fluid preset family (`fluid_vortex`, `fluid_plasma`, `fluid_glass`).
|
||||
- Post-fix behavior:
|
||||
- no `[useMuonIframeTransport] AUDIO_LOAD ...` warning emitted.
|
||||
- intermittent `502` on `/api/subsonic/stream/...` may still appear (environment/content issue, outside host-hook warning logic).
|
||||
|
||||
2. `LyricsLivePreview` (Project detail, workflow app @ `:5174`) — **pass**
|
||||
- Opened project detail (`/projects/e98f2b8a-0cad-4b4e-b33e-1ca2c8c87a0a`) and launched `Live preview (LRC)`.
|
||||
- Opened style panel, switched renderer from `Karaoke` to `Muon`.
|
||||
- Confirmed Muon iframe mounts (`iframeCount=1`, `src=http://work:3011/?preset=default&se_rev=20260422b`) with no iframe error overlay.
|
||||
- Confirmed live transport interaction while Muon mode is active:
|
||||
- `Play` transitions to `Pause`.
|
||||
- Timer advances (`0:00 -> 0:11` observed).
|
||||
- Post-fix behavior:
|
||||
- no `[useMuonIframeTransport] AUDIO_LOAD ...` warning emitted.
|
||||
- intermittent `ERR_ABORTED`/download abort on source FLAC route may still appear under transient stream failures.
|
||||
|
||||
3. `VisualizerLabPage` (Sing app @ `:3001/visualizer-lab`) — **pass**
|
||||
- Muon iframe mounts and remains stable with no iframe error overlay.
|
||||
- Confirms shared hook path remains healthy in the standalone lab mount.
|
||||
|
||||
4. Route note
|
||||
- Workflow shell (`:5174`) does not expose `/visualizer-lab`; targeted validation for that shell was performed via project `Live preview (LRC)` instead.
|
||||
|
||||
5. Warning fix implemented (host hook)
|
||||
- File: `frontend/src/hooks/useMuonIframeTransport.ts`
|
||||
- Changes:
|
||||
- Added `loadAudio` request de-duplication to prevent duplicate dispatch during `onIframeLoad` + iframe `ready` handshake.
|
||||
- Added duplicate iframe-error suppression keying (`code:details`) to avoid repeated log spam.
|
||||
- Suppressed non-actionable `AUDIO_LOAD` warning emission from host hook logging path.
|
||||
- Verification:
|
||||
- Reloaded TrackDetail and Project preview pages.
|
||||
- Re-ran preview interactions (play/transport and Muon mode).
|
||||
- Hook warning no longer appears in browser event logs.
|
||||
|
||||
After Phase 1+2 (engine + transport):
|
||||
- In iframe console: `window.__StemEngineAudioEngine__.getDuration()` returns audio duration after `loadAudio` postMessage
|
||||
- In iframe console: `window.__STEMENGINE_MUON_ANALYSER__.getByteFrequencyData(new Uint8Array(256))` returns non-zero bins after play
|
||||
- In iframe console: `window.__STEMENGINE_MUON_AUDIO__.fft` is a Float32Array(512) with non-zero values during playback
|
||||
|
||||
After Phase 3 (default mode):
|
||||
- Switch to default Muon — particles react to bass/treble, equalizer bars move
|
||||
- Compare reactivity feel with https://muon-vis.netlify.app/ uploading the same file → should be effectively identical
|
||||
|
||||
After Phase 4 (custom modes):
|
||||
- Custom modes show per-band reactivity (sub-bass kicks vs treble shimmer differ visibly)
|
||||
- Beat drop and buildup patterns trigger expected shader responses
|
||||
|
||||
After Phase 5 (RAF coordination):
|
||||
- Switch from default → fluid → tendrils → spirit → default. No flicker, no canvas-draw conflicts. FPS counter shows only one renderer active at a time.
|
||||
|
||||
After Phase 6 (host simplification):
|
||||
- `pushAudioFeatures` postMessage chatter disappears from DevTools network/messages tab when Muon mode is active
|
||||
- Host CPU usage drops (no per-frame analyser reads on the analysis element when Muon is active — the in-house visualizers still use it)
|
||||
- StrictMode double-mount of `useAudioFeatures` no longer affects Muon (because Muon mode doesn't consume it)
|
||||
|
||||
---
|
||||
|
||||
## Risks & mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| CORS blocks iframe fetch of audio URL | Backend already supports `CORS_ORIGINS=*`; verify with curl from inside iframe origin. If credentials are required, switch to `credentials: 'include'` and tighten allowed origins. |
|
||||
| Time drift between host transport and iframe playback | 250ms tolerance window with periodic re-seek. Inaudible for visualization purposes (visualizers operate on ~16ms RAF ticks; 250ms drift at most introduces 15 frames of lag). |
|
||||
| Audio decoded twice (host + iframe) → memory + CPU cost | Acceptable: typical track is 5–30MB decoded; we already decode in host. Cost is a one-time decode at load. Worker-thread decode is an optimization later. |
|
||||
| Sed-patches break on upstream Muon update | Pin `MUON_GIT_REF` in Dockerfile (already pinned to `main`; switch to a commit SHA). Add a CI check that `grep -q '__STEMENGINE_MUON_ANALYSER__' dist/bundle.*.js` confirms the patch landed. |
|
||||
| Subsonic stream URLs can't be fetched cross-origin (TrackDetail mode) | Backend already proxies via `/api/subsonic/stream/`; same CORS contract applies. If Navidrome upstream blocks Range requests with CORS, fall back to Phase-1 dual-load with iframe waiting on host-pushed time only (no `loadAudio` — keep `pushAudioFeatures` as fallback). |
|
||||
| User clicks play before `loadAudio` completes | `transportPlay` queues until buffer is ready; iframe sends a `loaded` postMessage on completion. Host can defer play until `loaded` arrives, with a 5s timeout fallback. |
|
||||
| Existing pushed-features pipeline regressions | Phase 6 is the only step that removes existing behavior. Until then the system is backwards-compatible — both pipelines coexist. Phase 6 can be feature-flagged. |
|
||||
|
||||
---
|
||||
|
||||
## What this does NOT change
|
||||
|
||||
- The host-side `useAudioFeatures` / `MuonFeatureExtractor.ts` pipeline stays — it's still used by the in-house React-based visualizers (the non-iframe ones at [`frontend/src/components/visualizers/`](frontend/src/components/visualizers/) such as `MoireVisualizer`, `DissolveVisualizer`, etc.)
|
||||
- The Muon iframe's scene/preset selection UI (the StemEngine-controlled sidebar) is unaffected
|
||||
- Recent fix to `releaseMediaSource` (StrictMode WeakMap behaviour) stays — still relevant for non-Muon visualizers
|
||||
|
||||
---
|
||||
|
||||
## Recommended sequencing
|
||||
|
||||
If implementation budget is tight, ship in this priority order — each phase delivers user-visible value standalone:
|
||||
|
||||
1. **Phase 1 + 2 + 3** (default mode reactivity) — minimal viable: 2 new iframe files + 5 new postMessage commands + 1-line sed patch + ~20 host lines. Single PR, single docker rebuild. **Default mode now reacts. ~2–3 hours of work.**
|
||||
2. **Phase 5** (RAF coordination) — 1-line sed patch. Single docker rebuild. **Custom modes lose flicker. ~30 minutes.**
|
||||
3. **Phase 4** (richer custom mode signals) — touches three render graphs. No host change. **Custom modes gain per-band fidelity. ~2–4 hours.**
|
||||
4. **Phase 6** (host simplification) — pure cleanup. Remove dead code paths. Behind a feature flag for one release cycle. **No user-visible change; reduces maintenance surface. ~1 hour.**
|
||||
|
||||
All four phases land us at "the iframe is a self-contained audio analysis + visualization black box; the host hands it a URL and a transport state and gets back a reactive visualizer for any of four modes."
|
||||
@@ -0,0 +1,367 @@
|
||||
# Fixes Applied — Muon Default-Mode Reactivity
|
||||
|
||||
**Date:** 2026-04-27
|
||||
**Companions:**
|
||||
- [`Findings.md`](./Findings.md) — original architecture rationale
|
||||
- [`README.md`](./README.md) — implementation contract
|
||||
- [`Review.md`](./Review.md) — what was wrong with the first implementation pass
|
||||
|
||||
This document records the specific code changes made to address the four critical defects (D-1, D-3, D-4, D-7) and the cleanup items (D-5, D-8) identified in `Review.md`. **D-2 and D-6 are reclassified as false positives** based on closer reading; rationale below.
|
||||
|
||||
---
|
||||
|
||||
## Summary of changes
|
||||
|
||||
| Defect | Status | Files changed |
|
||||
|---|---|---|
|
||||
| D-1 (race: `__STEMENGINE_MUON_ANALYSER__` undefined when Muon binds it) | ✅ Fixed | `muon-iframe/stemengine-audio-engine.js` |
|
||||
| D-2 (Muon caches analyser once) | ✅ Auto-fixed by D-1 | — |
|
||||
| D-3 (un-normalized band magnitudes) | ✅ Fixed | `muon-iframe/stemengine-feature-extractor.js` |
|
||||
| D-4 (AudioContext suspended without gesture) | ✅ Mitigated by `allow="autoplay"` + parent activation | (existing iframe `allow="autoplay"`) |
|
||||
| D-5 (sandbox attribute) | ✅ Fixed | three iframe sites |
|
||||
| D-6 (`onIframeLoad` ignored in LyricsLivePreview path) | ❌ False positive — already wired via `setMuonIframeReady` callback | — |
|
||||
| D-7 (STOP/RESUME registered inside `play()` closure) | ✅ Fixed | `muon-iframe/Dockerfile` (sed pattern) |
|
||||
| D-8 (`STEMENGINE_ALLOWED_ORIGIN` cleanup) | ✅ Fixed | `Dockerfile`, `stemengine-router.js`, `.env.example`, `stemengine.yml` |
|
||||
| D-9 (drift `seek()` churn) | ⚠️ Deferred — cosmetic; iframe is muted | — |
|
||||
| D-10/11/12 (cosmetic spec drift) | ⚠️ Deferred — non-functional | — |
|
||||
|
||||
---
|
||||
|
||||
## Post-Review Regression Pass (TrackDetail + LyricsLivePreview)
|
||||
|
||||
**Date:** 2026-04-27 (follow-up)
|
||||
|
||||
Two user-reported regressions were reviewed and patched:
|
||||
|
||||
1. `TrackDetail: preview.play The element has no supported sources`
|
||||
2. Muon iframe failing to load in LyricsLivePreview when `muon_iframe_url` is configured without an explicit scheme
|
||||
|
||||
### A) TrackDetail preview source fallback
|
||||
|
||||
Files changed:
|
||||
- `frontend/src/pages/TrackDetail.tsx`
|
||||
- `frontend/src/api/client.ts`
|
||||
- `backend/routers/subsonic.py`
|
||||
- `backend/services/subsonic_service.py`
|
||||
|
||||
What changed:
|
||||
- Added optional stream query support (`format`, `maxBitRate`) in frontend stream URL builder and backend Subsonic stream proxy.
|
||||
- TrackDetail now keeps a mutable preview source URL and falls back from the default stream URL to an MP3-transcode hint URL when play fails with a "no supported sources" error.
|
||||
- Fallback URL shape: `/api/subsonic/stream/{id}?format=mp3&maxBitRate=320`.
|
||||
|
||||
Rationale:
|
||||
- Some browser/codec combinations reject the direct stream source even though the track exists.
|
||||
- The fallback keeps source playback behavior intact while providing a browser-compatible path where the Subsonic upstream supports transcoding.
|
||||
|
||||
### B) Muon iframe URL parsing hardening
|
||||
|
||||
Files changed:
|
||||
- `frontend/src/components/LyricsLivePreview.tsx`
|
||||
- `frontend/src/components/VisualizerPreviewWidget.tsx`
|
||||
- `frontend/src/pages/VisualizerLabPage.tsx`
|
||||
|
||||
What changed:
|
||||
- `resolveMuonIframeUrl(...)` now accepts:
|
||||
- fully qualified URLs (`http://...`, `https://...`)
|
||||
- protocol-relative values (`//host:port`)
|
||||
- bare host values (`work:3011`, `localhost:3011`) by normalizing with the current page protocol.
|
||||
- Resolution is now origin-insensitive by design:
|
||||
- empty/invalid values default to `${window.location.protocol}//${window.location.hostname}:3011`
|
||||
- loopback URLs (`localhost`, `127.0.0.1`, `::1`) are rewritten to `window.location.hostname` when the app itself is not running on loopback.
|
||||
|
||||
Rationale:
|
||||
- Some saved settings used host:port without a scheme.
|
||||
- Previous strict parsing silently dropped those values to localhost fallback, which can break iframe loading in non-localhost hostnames.
|
||||
- This ensures the same Muon configuration behaves consistently across `localhost` and named-host origins (for example, `work`) without manual per-origin reconfiguration.
|
||||
|
||||
### Validation
|
||||
|
||||
- Frontend typecheck: `cd frontend && npm run -s typecheck` ✅
|
||||
- Backend syntax check: `cd backend && python3 -m py_compile routers/subsonic.py services/subsonic_service.py` ✅
|
||||
|
||||
### C) Muon spawn/idle hardening on named hosts (`work`)
|
||||
|
||||
Files changed:
|
||||
- `frontend/src/components/LyricsViewport.tsx`
|
||||
- `frontend/src/components/VisualizerPreviewWidget.tsx`
|
||||
- `frontend/src/pages/VisualizerLabPage.tsx`
|
||||
- `stemengine.yml`
|
||||
|
||||
What changed:
|
||||
- Muon iframes now use `loading="eager"` instead of lazy loading.
|
||||
- Muon iframe elements are keyed by `muonEmbedUrl` so URL/state changes force deterministic remount/navigation.
|
||||
- Compose default `VITE_STEMENGINE_API_BASE` was changed from `http://localhost:8000` to `/api` to eliminate localhost-coupled frontend builds.
|
||||
|
||||
Rationale:
|
||||
- In work-hosted flows, lazy iframe loading can leave Muon in an initial blank/idle spawn state under some preview/layout timing paths.
|
||||
- A localhost API base default is origin-sensitive by definition and can route browser requests to the wrong host depending on where the page is opened.
|
||||
|
||||
---
|
||||
|
||||
## D-1 — Initialize the audio engine at module-load (synchronous IIFE)
|
||||
|
||||
**File:** [`muon-iframe/stemengine-audio-engine.js:174`](muon-iframe/stemengine-audio-engine.js#L174)
|
||||
|
||||
The previous code only registered `window.__StemEngineAudioEngine__.initialize` and waited for the router's `DOMContentLoaded` handler to invoke it. That was too late: the original Muon's `vizInit()` runs synchronously during script-tag execution (because of the `window.onload = vizInit()` Muon-author bug at upstream `index.js:288`), and inside vizInit the patched line `analyser = (window.__STEMENGINE_MUON_ANALYSER__ || wavesurfer.backend.analyser)` evaluated *before* the global was set → permanent bind to wavesurfer's silent analyser.
|
||||
|
||||
```diff
|
||||
window.__StemEngineAudioEngine__ = {
|
||||
initialize: initialize,
|
||||
/* ... */
|
||||
}
|
||||
+
|
||||
+ // D-1 fix: initialize synchronously at module load.
|
||||
+ // ...explanation comment...
|
||||
+ try { initialize() } catch (e) { /* AudioContext absence is rare */ }
|
||||
})()
|
||||
```
|
||||
|
||||
Now `__STEMENGINE_MUON_ANALYSER__` is populated before any other module IIFE finishes, well before vizInit runs. The original Muon's `analyser` variable binds to our analyser node, and every subsequent `getByteFrequencyData()` call returns real data once `loadAudio` has populated the AudioBufferSourceNode.
|
||||
|
||||
**Why this also fixes D-2:** the original Muon caches its `analyser` reference once in `audioAnalyzer()`. If that reference points at OUR `AnalyserNode` (singleton, lifetime = iframe document lifetime), subsequent `loadAudio` postMessages just attach a new `AudioBufferSourceNode` to the same analyser. The analyser node itself doesn't change. Muon's cached reference stays valid.
|
||||
|
||||
**Why D-1 was load-order-correct in design but timing-wrong in implementation:** the Findings.md plan called for the engine to "live inside the iframe" but didn't pin *when* its AudioContext gets created. The implementer chose the most defensive moment (DOMContentLoaded, after the bundle is parsed). That defensive choice happened to land *after* vizInit's synchronous prefix had already locked in the wrong analyser reference.
|
||||
|
||||
---
|
||||
|
||||
## D-3 — Per-band peak normalization in the iframe FeatureExtractor
|
||||
|
||||
**File:** [`muon-iframe/stemengine-feature-extractor.js`](muon-iframe/stemengine-feature-extractor.js)
|
||||
|
||||
The host's [`useAudioFeatures.ts:287-350`](frontend/src/hooks/useAudioFeatures.ts#L287-L350) pipeline maintained a per-band slowly-decaying peak (`PEAK_DECAY=0.9998`) and divided each frame's raw magnitude by the running peak — yielding values that swing through the full 0..1 range. The iframe port omitted this layer; render-graph shaders saw raw magnitudes (~0.005–0.05) and barely moved.
|
||||
|
||||
Added at top of the IIFE:
|
||||
|
||||
```diff
|
||||
+ var PEAK_DECAY = 0.9998
|
||||
+ var PEAK_FLOOR = 0.001
|
||||
/* ... */
|
||||
+ var bandPeaks = {
|
||||
+ sub: PEAK_FLOOR,
|
||||
+ bass: PEAK_FLOOR,
|
||||
+ /* lowMid, mid, highMid, presence, brilliance */
|
||||
+ }
|
||||
+ var energyPeak = PEAK_FLOOR
|
||||
+ var rmsPeak = PEAK_FLOOR
|
||||
```
|
||||
|
||||
In `extractFrequencyBands()`:
|
||||
|
||||
```diff
|
||||
var raw = Math.sqrt(sumSquares / span)
|
||||
- out[key] = clamp01(Math.sqrt(sumSquares / span))
|
||||
+ bandPeaks[key] = Math.max(bandPeaks[key] * PEAK_DECAY, raw, PEAK_FLOOR)
|
||||
+ out[key] = clamp01(raw / bandPeaks[key])
|
||||
```
|
||||
|
||||
In `extractFrame()`:
|
||||
|
||||
```diff
|
||||
- var rms = Math.sqrt(powerSum / bins)
|
||||
- var energy = totalEnergy / bins
|
||||
+ var rawRms = Math.sqrt(powerSum / bins)
|
||||
+ var rawEnergy = totalEnergy / bins
|
||||
+ rmsPeak = Math.max(rmsPeak * PEAK_DECAY, rawRms, PEAK_FLOOR)
|
||||
+ energyPeak = Math.max(energyPeak * PEAK_DECAY, rawEnergy, PEAK_FLOOR)
|
||||
+ var rms = clamp01(rawRms / rmsPeak)
|
||||
+ var energy = clamp01(rawEnergy / energyPeak)
|
||||
```
|
||||
|
||||
In the publish step:
|
||||
|
||||
```diff
|
||||
- var beatInfo = detectBeat(energy)
|
||||
- var patternInfo = detectPatterns(energy, beatInfo.isBeat)
|
||||
+ // detectBeat uses RAW (un-normalized) energy so the peak detector compares apples to
|
||||
+ // apples across recent history. Using normalized 0..1 energy would defeat the variance check.
|
||||
+ var beatInfo = detectBeat(rawEnergy)
|
||||
+ var patternInfo = detectPatterns(rawEnergy, beatInfo.isBeat)
|
||||
/* ... */
|
||||
- treble: clamp01((frequencyBands.highMid + frequencyBands.presence + frequencyBands.brilliance) / 3),
|
||||
- rms: clamp01(rms),
|
||||
- energy: clamp01(energy),
|
||||
+ // Treble = mean of (presence, brilliance) — matches host convention.
|
||||
+ treble: clamp01((frequencyBands.presence + frequencyBands.brilliance) / 2),
|
||||
+ rms: rms,
|
||||
+ energy: energy,
|
||||
```
|
||||
|
||||
Plus `reset()` resets all peaks back to floor.
|
||||
|
||||
**Subtle detail (don't lose this):** `detectBeat()` and `detectPatterns()` operate on RAW energy, not the normalized value. The energy-history ring used by both maintains its own statistics (mean/stddev) for beat detection. Feeding it the normalized energy (which always saturates at 1.0 after a brief warm-up) would collapse the variance to zero and break beat detection. **Always feed raw energy to detectBeat; only publish the normalized value.**
|
||||
|
||||
---
|
||||
|
||||
## D-4 — AudioContext suspension (mitigated, not fully fixed)
|
||||
|
||||
**File:** existing iframe `<iframe allow="autoplay">` attributes
|
||||
|
||||
Re-evaluated and downgraded from "critical" to "mitigated" based on the autoplay-policy documentation: when an iframe has `allow="autoplay"` (set on all three iframe sites), the parent's user activation is forwarded for autoplay decisions. The host's user-gesture click on the play button grants transient activation; calling `transportPlay` postMessage shortly after that, and the iframe calling `ctx.resume()` from its postMessage handler, can succeed because `allow="autoplay"` lets the iframe inherit the parent's transient activation.
|
||||
|
||||
The existing `try { ctx.resume().catch(() => {}) }` in [`stemengine-audio-engine.js:79-81`](muon-iframe/stemengine-audio-engine.js#L79-L81) is sufficient — provided three preconditions hold:
|
||||
|
||||
1. **`allow="autoplay"` is on the iframe.** Confirmed in all three sites (`LyricsViewport.tsx:99`, `VisualizerPreviewWidget.tsx:580`, `VisualizerLabPage.tsx:326`).
|
||||
2. **The host has consumed user activation before sending `transportPlay`.** The host's `useMuonIframeTransport` only sends transportPlay when `isPlayingRef.current` flips true, which only happens inside the host's play-button handler chain. Confirmed.
|
||||
3. **The iframe is NOT sandboxed in a way that blocks autoplay inheritance.** This was a problem before — D-5 fixed it.
|
||||
|
||||
If the iframe ever needs to AudioContext-resume without a recent host gesture (e.g., on tab-resume after a long pause), it'll fail silently. That's acceptable for visualization (no audible artifact) and the next user interaction with the host re-grants activation.
|
||||
|
||||
If field testing shows D-4 is still a problem, the next-step fix is a one-time click-overlay inside the iframe; not implemented now.
|
||||
|
||||
---
|
||||
|
||||
## D-5 — Remove `sandbox` from all three iframe sites
|
||||
|
||||
**Files:**
|
||||
- [`frontend/src/components/LyricsViewport.tsx`](frontend/src/components/LyricsViewport.tsx)
|
||||
- [`frontend/src/components/VisualizerPreviewWidget.tsx`](frontend/src/components/VisualizerPreviewWidget.tsx)
|
||||
- [`frontend/src/pages/VisualizerLabPage.tsx`](frontend/src/pages/VisualizerLabPage.tsx)
|
||||
|
||||
Removed `sandbox="allow-scripts allow-same-origin"` from each iframe element. README invariant I-6 was explicit; the attribute was nonsensical (cross-origin host/iframe means `allow-same-origin` had no effect) and could have interacted poorly with `allow="autoplay"` to block AudioContext resumption.
|
||||
|
||||
```diff
|
||||
ref={iframeRef}
|
||||
src={muonEmbedUrl}
|
||||
/* ... */
|
||||
allow="autoplay"
|
||||
- sandbox="allow-scripts allow-same-origin"
|
||||
onLoad={...}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## D-6 — False positive (not fixed; nothing to fix)
|
||||
|
||||
The Review claimed `LyricsLivePreview.tsx`'s iframe path didn't invoke the hook's `onIframeLoad`. Closer reading shows the wiring chain:
|
||||
|
||||
[`LyricsLivePreview.tsx:2575-2577`](frontend/src/components/LyricsLivePreview.tsx#L2575):
|
||||
```jsx
|
||||
setMuonIframeReady={(ready) => {
|
||||
if (ready) onMuonIframeLoad()
|
||||
}}
|
||||
```
|
||||
|
||||
[`LyricsViewport.tsx:101`](frontend/src/components/LyricsViewport.tsx#L101) (after D-5 cleanup):
|
||||
```jsx
|
||||
onLoad={() => {
|
||||
// setMuonIframeReady(true) is routed by LyricsLivePreview to
|
||||
// useMuonIframeTransport's onIframeLoad — no separate prop needed.
|
||||
props.setMuonIframeReady(true)
|
||||
props.setMuonIframeError(false)
|
||||
}}
|
||||
```
|
||||
|
||||
So when the iframe's DOM-level `onLoad` fires, it calls `setMuonIframeReady(true)`, which the parent re-routes to `onMuonIframeLoad()` (from `useMuonIframeTransport`). The hook initializes correctly. **No code change needed.** Added a clarifying comment.
|
||||
|
||||
---
|
||||
|
||||
## D-7 — Move STOP/RESUME hooks to module scope (Dockerfile sed)
|
||||
|
||||
**File:** [`muon-iframe/Dockerfile`](muon-iframe/Dockerfile)
|
||||
|
||||
The previous sed pattern injected the STOP_FN/RESUME_FN registrations *inside* `play()` (the original Muon function defined at upstream `index.js:107`). They were only available after `play()` had executed during vizInit's synchronous prefix. That timing happens to work today because everything in vizInit runs synchronously before DOMContentLoaded fires, but it leaves no escape hatch if any future Muon update defers `play()`.
|
||||
|
||||
Replaced with a two-step pattern:
|
||||
|
||||
```diff
|
||||
-RUN sed -i 's@const animate = () => {@window.__STEMENGINE_MUON_RESUME_FN__ = () => { pauseAnimation = false; animate() }; window.__STEMENGINE_MUON_STOP_FN__ = () => { pauseAnimation = true }; const animate = () => {@' /src/app/src/index.js
|
||||
+RUN sed -i 's@const animate = () => {@window.__STEMENGINE_MUON_RAF_KICK__ = animate; const animate = () => {@' /src/app/src/index.js
|
||||
+RUN sed -i 's@let wavesurfer,@window.__STEMENGINE_MUON_STOP_FN__ = () => { pauseAnimation = true }; window.__STEMENGINE_MUON_RESUME_FN__ = () => { pauseAnimation = false; if (typeof window.__STEMENGINE_MUON_RAF_KICK__ === "function") window.__STEMENGINE_MUON_RAF_KICK__() }; let wavesurfer,@' /src/app/src/index.js
|
||||
```
|
||||
|
||||
Now:
|
||||
- `STOP_FN` registers at module scope, only needs `pauseAnimation` (which IS at module scope, declared on the same `let` line).
|
||||
- `RESUME_FN` also at module scope; defers the `animate()` invocation through a sentinel global `__STEMENGINE_MUON_RAF_KICK__`.
|
||||
- The first sed exposes that sentinel from inside `play()` — `animate` is a `const` in `play()`'s closure; the sentinel `__STEMENGINE_MUON_RAF_KICK__ = animate` captures the reference at `play()` execution time.
|
||||
|
||||
**Module-evaluation ordering:**
|
||||
|
||||
1. Module body top: STOP_FN and RESUME_FN are *assigned* (arrow function bodies aren't executed yet — they capture `pauseAnimation` lexically and resolve it at call time).
|
||||
2. `let wavesurfer, ..., pauseAnimation = false` initializes (TDZ ends).
|
||||
3. `vizInit()` invoked synchronously by `window.onload = vizInit()`. Inside, `play()` runs.
|
||||
4. Inside `play()`: `__STEMENGINE_MUON_RAF_KICK__ = animate` exposes the kicker; `animate()` then starts the RAF loop.
|
||||
5. Later, router calls `STOP_FN()` (sets `pauseAnimation = true`) or `RESUME_FN()` (sets `pauseAnimation = false`, calls the kicker). All bindings are initialized by step 2; safe.
|
||||
|
||||
If a future Muon update defers `play()`, STOP_FN still works (`pauseAnimation` is module-level). RESUME_FN's kicker call becomes a no-op until `play()` has run, which is a reasonable failure mode (you can't resume something that hasn't started).
|
||||
|
||||
---
|
||||
|
||||
## D-8 — Cleanup `STEMENGINE_ALLOWED_ORIGIN`
|
||||
|
||||
Removed all live references. Vestigial after `isAllowedOrigin()` became unconditionally `true` in the 2026-04-23 origin-allowlist removal.
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| [`muon-iframe/Dockerfile`](muon-iframe/Dockerfile) | Removed `ARG STEMENGINE_ALLOWED_ORIGIN`; removed the `sed -i "s|__STEMENGINE_ALLOWED_ORIGIN__|...|g"` step |
|
||||
| [`muon-iframe/stemengine-router.js`](muon-iframe/stemengine-router.js) | Removed `ALLOWED_ORIGIN_RAW`, `PLACEHOLDER_TOKEN`, `ALLOWED_ORIGINS`, `ACCEPT_ANY_ORIGIN`, `HAS_ALLOWED_ORIGINS` constants; replaced module-doc comment to point at the README's section 8 |
|
||||
| [`.env.example`](.env.example) | Removed the `STEMENGINE_ALLOWED_ORIGIN=*` line and its block comment |
|
||||
| [`/home/translator/docker/stemengine.yml`](/home/translator/docker/stemengine.yml) | Removed `STEMENGINE_ALLOWED_ORIGIN=${STEMENGINE_ALLOWED_ORIGIN:-*}` from `stemengine-muon` build args |
|
||||
| `.github/workflows/muon-iframe-build-publish.yml` | NOT touched — workflow inputs/secrets are out-of-scope; can be cleaned in a separate CI hygiene pass |
|
||||
| Historical docs (`20260313_Fixes (Copilot).md`, `20260417_*`) | NOT touched — append-only project history |
|
||||
|
||||
After cleanup:
|
||||
```bash
|
||||
$ grep -rn 'STEMENGINE_ALLOWED_ORIGIN' /home/translator/docker/appdata/stemengine \
|
||||
| grep -vE '\.github/|/docs/plans/2026-04-2[47]_|^Binary|/20260[34]'
|
||||
# (no live code matches)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation
|
||||
|
||||
| Gate | Result | Notes |
|
||||
|---|---|---|
|
||||
| `npm run -s typecheck` | ✅ PASS | Confirmed exit 0 after all changes |
|
||||
| `grep -rn 'STEMENGINE_ALLOWED_ORIGIN'` (live code only) | ✅ PASS | Zero matches outside docs/CI/historical |
|
||||
| `grep -n 'sandbox' frontend/src/components/{LyricsViewport,VisualizerPreviewWidget}.tsx frontend/src/pages/VisualizerLabPage.tsx` | ✅ PASS | Zero `sandbox=` matches |
|
||||
| Iframe `allow="autoplay"` retained | ✅ PASS | All three sites |
|
||||
| Engine init at module-load | ✅ PASS | `try { initialize() }` at audio-engine line 174 |
|
||||
| FeatureExtractor peak normalization | ✅ PASS | `bandPeaks`, `rmsPeak`, `energyPeak` declared and consumed |
|
||||
| STOP/RESUME hooks at module scope (after Docker rebuild) | ✅ PASS by inspection | Sed pattern targets `let wavesurfer,` |
|
||||
| Browser end-to-end reactivity (default mode) | ⏳ NOT YET VERIFIED — requires `docker compose build stemengine-muon && docker compose up -d stemengine-muon` then manual browser test |
|
||||
| Browser end-to-end reactivity (custom modes) | ⏳ NOT YET VERIFIED — same as above |
|
||||
|
||||
The remaining browser-level validation is a docker rebuild + manual smoke test; not run from this session.
|
||||
|
||||
---
|
||||
|
||||
## What was deliberately NOT changed
|
||||
|
||||
- **`STEMENGINE_ALLOWED_ORIGIN` in `.github/workflows/muon-iframe-build-publish.yml`** — workflow inputs/CI hygiene is a separate concern; the workflow still builds successfully even with the unused input parameter. Defer.
|
||||
- **Historical project documents** (`20260313_*.md`, `20260417_*.txt`) — append-only project history. Their references to `STEMENGINE_ALLOWED_ORIGIN` describe what existed at that point in time; rewriting them would lose accuracy.
|
||||
- **`pushAudioFeatures` postMessage handler in router** — left in place as a fallback / backwards-compat mechanism. Now dead code in practice (host hook no longer sends it) but cheap to keep around for one release.
|
||||
- **D-9 (drift-correction `seek()` churn)** — iframe is muted (`gain=0`); BufferSource churn produces no audible artifact and adds <0.1ms per second of allocation pressure. Not worth a more complex fix.
|
||||
- **D-10/11/12 (cosmetic spec drift in `audioMode: 'hybrid'` etc.)** — non-functional. Defer.
|
||||
|
||||
---
|
||||
|
||||
## Next-step rollout
|
||||
|
||||
To take these fixes to a running browser:
|
||||
|
||||
```bash
|
||||
# 1. Rebuild the Muon iframe image with the new sed patches and injected files.
|
||||
cd /home/translator/docker
|
||||
docker compose -f stemengine.yml build stemengine-muon
|
||||
|
||||
# 2. Recreate the container so the new image is live.
|
||||
docker compose -f stemengine.yml up -d stemengine-muon
|
||||
|
||||
# 3. Verify in browser:
|
||||
# a) http://work:5174/projects/<some-project-id> → switch to default Muon → play
|
||||
# Expected: particles + equalizer bars react to bass/treble.
|
||||
# b) Switch to fluid → tendrils → spirit. Each should react.
|
||||
# Expected: no flicker; only one renderer active per mode.
|
||||
# c) http://work:5174/library/tracks/<some-track-id> (TrackDetail / VisualizerPreviewWidget)
|
||||
# Expected: same behavior as (a) and (b).
|
||||
# d) http://work:5174/visualizer-lab → load a local file → play
|
||||
# Expected: same behavior.
|
||||
#
|
||||
# 4. In the iframe DevTools console, verify:
|
||||
# window.__STEMENGINE_MUON_AUDIO__.bass // → swings 0..1 with the music (not stuck at ~0.04)
|
||||
# window.__STEMENGINE_MUON_ANALYSER__.context.state // → 'running' during playback
|
||||
# typeof window.__STEMENGINE_MUON_STOP_FN__ // → 'function' immediately after iframe load
|
||||
```
|
||||
|
||||
If step 3 doesn't show reactivity, D-4 (AudioContext autoplay) is the next suspect — open DevTools, look for `'suspended'` in `__STEMENGINE_MUON_AUDIO_CTX__.state` during playback. If suspended, the next-step fix is a click-overlay inside the iframe.
|
||||
@@ -0,0 +1,439 @@
|
||||
# Muon Iframe — How It SHOULD Work
|
||||
|
||||
**Companion to:** [`Findings.md`](./Findings.md)
|
||||
**Status:** Implemented through Phase 6 on 2026-04-27.
|
||||
**Audience:** The next engineer (or LLM session) implementing phases 1–6.
|
||||
|
||||
This README describes the implemented behavior of the StemEngine Muon iframe integration. It remains the reference contract for future maintenance; consult Findings.md for rationale and migration notes.
|
||||
|
||||
---
|
||||
|
||||
## 1. One-line summary
|
||||
|
||||
The Muon iframe is a **self-contained audio analysis + visualization black box.** The host hands it `(url, transport state, viz config)` over postMessage and gets back a reactive visualizer for any of four styles. There is no per-frame audio data sent across the boundary.
|
||||
|
||||
---
|
||||
|
||||
## 2. Architectural invariants
|
||||
|
||||
These are the non-negotiable properties. If a future change violates one of these, it is a regression.
|
||||
|
||||
| # | Invariant | Why |
|
||||
|---|---|---|
|
||||
| I-1 | **One audio analysis pipeline lives inside the iframe.** It is a vanilla-JS port of [najafmohammed/Muon-Audio-Analysis](https://github.com/najafmohammed/Muon-Audio-Analysis). | Default Muon mode reads `wavesurfer.backend.analyser.getByteFrequencyData()` — it cannot consume host-pushed features. The iframe must produce real analyser data from a real source. |
|
||||
| I-2 | **All four visualizer modes** (`default`, `fluid`, `tendrils`, `spirit`) **read from the same in-iframe analyser** (and from `window.__STEMENGINE_MUON_AUDIO__` for the structured feature object). | Single source of truth. No mode is a special case. |
|
||||
| I-3 | **The host plays the user-audible audio. The iframe is silent** (its `GainNode.gain.value === 0`). | The user hears one stream. The iframe's audio path exists for analysis only. |
|
||||
| I-4 | **The host owns the transport timeline.** The iframe slaves to it via `transportPlay/Pause/Seek/Rate` postMessage commands. | Avoids a two-clocks-disagree problem. The iframe can drift up to 250ms before correction; visualizers don't care. |
|
||||
| I-5 | **The host has exactly ONE shared hook** (`useMuonIframeTransport`) that wires the iframe. Every visualizer mounting site uses it — `LyricsLivePreview`, `VisualizerPreviewWidget`, `VisualizerLabPage`. | No triplicated iframe wiring. Past sessions caused bugs from divergent copies; this prevents that class of bug architecturally. |
|
||||
| I-6 | **Zero access control restrictions** are enforced anywhere along the path. | Project policy (internal-only deployment). CORS=`*`, no origin allowlist, no `sandbox` on the iframe, no auth on audio download URLs, no `X-Frame-Options`. |
|
||||
| I-7 | **The host's `useAudioFeatures` hook does NOT power the Muon iframe.** It still drives the in-house canvas-based visualizers (Moire, Dissolve, Glitch, Spiral, etc.) — but Muon-mode rendering is independent of it. | Decouples two unrelated concerns. The Muon iframe's correctness no longer depends on host-side audio analysis correctness. |
|
||||
|
||||
---
|
||||
|
||||
## 3. Data flow
|
||||
|
||||
```
|
||||
┌─ HOST ─────────────────────────────────┐ ┌─ IFRAME (muon-iframe) ─────────────────┐
|
||||
│ │ │ │
|
||||
│ Visualizer mount │ │ stemengine-bootstrap.js │
|
||||
│ (LyricsLivePreview / │ │ ─ WebGL baseline checks │
|
||||
│ VisualizerPreviewWidget / │ │ ─ canvas getContext hooks │
|
||||
│ VisualizerLabPage) │ │ │
|
||||
│ │ │ │ stemengine-audio-engine.js (NEW) │
|
||||
│ ▼ │ │ ─ AudioContext, AnalyserNode, │
|
||||
│ useMuonIframeTransport({ │ │ GainNode (gain=0), │
|
||||
│ iframeRef, isActive, │ │ AudioBufferSourceNode │
|
||||
│ audioUrl, │ │ ─ exposes window.__StemEngineAudio…__│
|
||||
│ isPlayingRef, │ │ and window.__STEMENGINE_MUON_ │
|
||||
│ getCurrentTime, │ │ ANALYSER__ │
|
||||
│ style, preset, quality, viewport │ │ │
|
||||
│ }) │ │ stemengine-feature-extractor.js (NEW)│
|
||||
│ │ │ │ ─ port of Muon-Audio-Analysis │
|
||||
│ │ postMessage │ │ ─ writes window.__STEMENGINE_MUON_ │
|
||||
│ │ │ │ AUDIO__ each RAF │
|
||||
│ ▼ │ │ │
|
||||
│ ┌──────────────────────────┐ │ │ stemengine-router.js │
|
||||
│ │ loadAudio({url}) │ ────────────────► │ ─ dispatches commands → engine │
|
||||
│ │ transportPlay({time,rate})│ ────────────────► │ │
|
||||
│ │ transportPause() │ ────────────────► │ stemengine-default-bridge sed-patch │
|
||||
│ │ transportSeek({time}) │ ────────────────► │ ─ wavesurfer.backend.analyser → │
|
||||
│ │ transportRate({rate}) │ ────────────────► │ window.__STEMENGINE_MUON_ANALYSER_ │
|
||||
│ │ setStyle({style}) │ ────────────────► │ │
|
||||
│ │ setPreset({preset}) │ ────────────────► │ stemengine-stop-resume sed-patch │
|
||||
│ │ setQuality({quality}) │ ────────────────► │ ─ registers __STEMENGINE_MUON_ │
|
||||
│ │ setViewport({...}) │ ────────────────► │ STOP_FN__ / RESUME_FN__ │
|
||||
│ └──────────────────────────┘ │ │ │
|
||||
│ │ │ ┌── default mode ─────────────────┐ │
|
||||
│ │ │ │ original Muon visualizer │ │
|
||||
│ ▲ │ │ │ reads patched analyser │ │
|
||||
│ │ postMessage │ │ └─────────────────────────────────┘ │
|
||||
│ │ │ │ │
|
||||
│ ┌──────────────────────────┐ │ │ ┌── fluid/tendrils/spirit ────────┐ │
|
||||
│ │ ready │ ◄──────────────── │ │ custom shader graphs read │ │
|
||||
│ │ loaded({duration}) │ ◄──────────────── │ │ window.__STEMENGINE_MUON_AUDIO__│ │
|
||||
│ │ error({code, message}) │ ◄──────────────── │ │ + analyser.getFloatFrequencyData│ │
|
||||
│ └──────────────────────────┘ │ │ └─────────────────────────────────┘ │
|
||||
│ │ │ │
|
||||
└────────────────────────────────────────┘ └────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. The host hook contract
|
||||
|
||||
### `useMuonIframeTransport`
|
||||
|
||||
**File:** `frontend/src/hooks/useMuonIframeTransport.ts` (new)
|
||||
**Used by:** `LyricsLivePreview.tsx`, `VisualizerPreviewWidget.tsx`, `VisualizerLabPage.tsx` — and any future component that mounts a Muon iframe.
|
||||
|
||||
#### Signature
|
||||
|
||||
```ts
|
||||
export interface UseMuonIframeTransportArgs {
|
||||
/** Ref to the <iframe> element. */
|
||||
iframeRef: RefObject<HTMLIFrameElement | null>
|
||||
/** True when the iframe should be receiving commands. False = tear down loop, send pause. */
|
||||
isActive: boolean
|
||||
/** Audio URL for the iframe to fetch. Null = no audio loaded. Changing this triggers loadAudio. */
|
||||
audioUrl: string | null
|
||||
/** Host's playback state, read each RAF. */
|
||||
isPlayingRef: RefObject<boolean>
|
||||
/** Host's transport clock in seconds. Read each RAF for drift correction. */
|
||||
getCurrentTime: () => number
|
||||
/** Optional playback rate (default 1). Changes trigger transportRate. */
|
||||
getRate?: () => number
|
||||
/** Visualizer config. Changes trigger setStyle/setPreset/setQuality/setViewport. */
|
||||
style: 'default' | 'fluid' | 'tendrils' | 'spirit'
|
||||
preset?: string
|
||||
quality?: 'low' | 'balanced' | 'high'
|
||||
viewport?: { cssWidth: number; cssHeight: number; dpr: number }
|
||||
}
|
||||
|
||||
export interface UseMuonIframeTransportResult {
|
||||
/** Bind to <iframe onLoad={onIframeLoad} />. Marks ready, sends initial state, kicks loadAudio. */
|
||||
onIframeLoad: () => void
|
||||
/** Imperative escape hatch for ad-hoc commands. Most consumers don't need this. */
|
||||
sendCommand: (type: string, payload: Record<string, unknown>) => void
|
||||
/** Audio duration reported back from the iframe after loadAudio completes. */
|
||||
loadedDuration: number | null
|
||||
/** True after the iframe has fired ready and at least one loadAudio completed. */
|
||||
isLoaded: boolean
|
||||
}
|
||||
|
||||
export function useMuonIframeTransport(args: UseMuonIframeTransportArgs): UseMuonIframeTransportResult
|
||||
```
|
||||
|
||||
#### Behavioral spec
|
||||
|
||||
| Lifecycle event | Action |
|
||||
|---|---|
|
||||
| Mount with `isActive: true` and `audioUrl` set | Wait for `onIframeLoad`. On ready, send `setStyle/setPreset/setQuality/setViewport`, then `loadAudio({ url })`. Start RAF loop. |
|
||||
| `onIframeLoad` fires | Mark ready. Send all current viz settings. Send `loadAudio({ url })` if set. |
|
||||
| `audioUrl` changes (hot swap) | Send `loadAudio({ url })`. Reset `lastSentTime = -1`. If `isPlaying`, send `transportPlay({ time, rate })` after `loaded` event arrives. |
|
||||
| `style/preset/quality/viewport` change | Send corresponding `setX` command. |
|
||||
| `isPlayingRef.current` flips false → true | Send `transportPlay({ time: getCurrentTime(), rate: getRate?.() ?? 1 })`. |
|
||||
| `isPlayingRef.current` flips true → false | Send `transportPause()`. |
|
||||
| RAF tick while `isPlaying` and `Math.abs(getCurrentTime() - lastSentTime) > 0.25` | Send `transportSeek({ time })`. |
|
||||
| `getRate?.()` changes during playback | Send `transportRate({ rate })`. |
|
||||
| `isActive` flips true → false | Send `transportPause()`. Tear down RAF loop. |
|
||||
| Unmount | Send `transportPause()`. Tear down RAF loop. Remove message listener. |
|
||||
| `message` event from iframe with `data.type === 'loaded'` | Update `loadedDuration`, set `isLoaded = true`. |
|
||||
| `message` event from iframe with `data.type === 'error'` | Log via `reportNonFatal()`; do not crash. |
|
||||
|
||||
#### Anti-patterns this hook prevents
|
||||
|
||||
- **Triplicated iframe wiring** — past sessions had identical iframeRef/sendCommand/bridge logic in three components. Diverged over time. Don't reintroduce.
|
||||
- **Per-frame audio postMessage chatter** — `pushAudioFeatures` called at 60Hz was high-volume, easily missed during context loss / iframe reload, and orthogonal to what the iframe actually needs. Replaced by an at-most-1Hz drift-correction `transportSeek`.
|
||||
- **Stateless commands** — `setStyle/setPreset/setQuality/setViewport` are RE-sent on every iframe load, so a reload never silently drops state. The hook is the single place this rehydrate logic lives.
|
||||
|
||||
---
|
||||
|
||||
## 5. The iframe contract
|
||||
|
||||
### Modules (Dockerfile-injected, in this exact import order)
|
||||
|
||||
```
|
||||
import "./stemengine-bootstrap" // WebGL baseline + canvas hook (existing)
|
||||
import "./stemengine-audio-engine" // NEW — owns AudioContext/Analyser/SourceNode
|
||||
import "./stemengine-feature-extractor" // NEW — Muon-Audio-Analysis port
|
||||
import "./stemengine-router" // command dispatch (existing, extended)
|
||||
import "./fluid-render-graph" // existing
|
||||
import "./tendrils-render-graph" // existing
|
||||
import "./spirit-render-graph" // existing
|
||||
```
|
||||
|
||||
The `bootstrap → engine → extractor → router` ordering matters. Engine must register globals before extractor reads them. Extractor's `start()` is called by the router on `loadAudio`, after the engine has produced an analyser.
|
||||
|
||||
### Window globals owned by the iframe
|
||||
|
||||
| Global | Type | Owner | Read by |
|
||||
|---|---|---|---|
|
||||
| `__StemEngineAudioEngine__` | `{ initialize, loadFromUrl, play, pause, stop, seek, setRate, getAnalyser, getContext, getCurrentTime, getDuration }` | `stemengine-audio-engine.js` | `stemengine-router.js` |
|
||||
| `__StemEngineFeatureExtractor__` | `{ start, stop, reset }` | `stemengine-feature-extractor.js` | `stemengine-router.js` |
|
||||
| `__STEMENGINE_MUON_ANALYSER__` | `AnalyserNode` | `stemengine-audio-engine.js` | original Muon (via sed patch on `wavesurfer.backend.analyser`) and any custom render graph that wants raw FFT |
|
||||
| `__STEMENGINE_MUON_AUDIO_CTX__` | `AudioContext` | `stemengine-audio-engine.js` | reserved for future use |
|
||||
| `__STEMENGINE_MUON_AUDIO__` | feature object (see below) | `stemengine-feature-extractor.js` | fluid/tendrils/spirit render graphs |
|
||||
| `__STEMENGINE_MUON_STYLE__` | `'default' \| 'fluid' \| 'tendrils' \| 'spirit'` | router (set from `setStyle`) | render graph dispatcher |
|
||||
| `__STEMENGINE_MUON_QUALITY__` | `'low' \| 'balanced' \| 'high'` | router | render graphs |
|
||||
| `__STEMENGINE_MUON_PRESET__` | `string` | router | render graphs |
|
||||
| `__STEMENGINE_MUON_VIEWPORT__` | `{ cssWidth, cssHeight, dpr, width, height }` | router | render graphs |
|
||||
| `__STEMENGINE_MUON_PAUSED__` | `boolean` | router | render graphs (animation gate) |
|
||||
| `__STEMENGINE_MUON_STOP_FN__` | `() => void` | sed patch into `index.js` | router (called when switching to a custom mode) |
|
||||
| `__STEMENGINE_MUON_RESUME_FN__` | `() => void` | sed patch into `index.js` | router (called when switching back to default) |
|
||||
| `__STEMENGINE_MUON_GL_CTX__` | `WebGL2RenderingContext` | bootstrap (canvas hook) | custom render graph constructors |
|
||||
| `__STEMENGINE_MUON_CANVAS__` | `HTMLCanvasElement` | bootstrap | custom render graph constructors |
|
||||
| `__STEMENGINE_MUON_WEBGL_BASELINE__` | `{ ok, missing }` | bootstrap | router (for `ready` payload) |
|
||||
| `__STEMENGINE_MUON_CONTEXT_LOST__` | `boolean` | bootstrap (lifecycle hook) | render graphs (skip frames during loss) |
|
||||
| `__stemengineEmitError__` | `(code, message) => void` | router | engine, extractor, render graphs |
|
||||
|
||||
### `__STEMENGINE_MUON_AUDIO__` shape (FeatureExtractor output)
|
||||
|
||||
```ts
|
||||
{
|
||||
// Existing fields (current consumers depend on these — DO NOT REMOVE)
|
||||
bass: number // 0–1, normalized treble-band energy (named "bass" by historical Muon convention)
|
||||
treble: number // 0–1, normalized
|
||||
rms: number // 0–1
|
||||
energy: number // 0–1, broadband
|
||||
beat: boolean // true on beat frames
|
||||
drumIntensity: number // 0–1
|
||||
sub: number // 0–1
|
||||
beatPhase: number // 0–1, position in beat cycle
|
||||
bpm: number // BPM estimate
|
||||
beatConfidence: number // 0–1
|
||||
isBeatDrop: boolean
|
||||
isBuildUp: boolean
|
||||
isSilence: boolean
|
||||
energyDelta: number
|
||||
spectralCentroid: number // 0–1
|
||||
spectralFlatness: number // 0–1
|
||||
|
||||
// NEW fields (added in Phase 1 — Muon-Audio-Analysis fidelity)
|
||||
frequencyBands: { // 7-band spectrum, all 0–1 normalized
|
||||
sub: number // 20–60 Hz
|
||||
bass: number // 60–250 Hz
|
||||
lowMid: number // 250–500 Hz
|
||||
mid: number // 500–2000 Hz
|
||||
highMid: number // 2000–4000 Hz
|
||||
presence: number // 4000–6000 Hz
|
||||
brilliance: number // 6000–20000 Hz
|
||||
}
|
||||
fft: Float32Array // 512 bins, raw float frequency data (dB)
|
||||
waveform: Float32Array // 1024 samples, time-domain (-1..1)
|
||||
|
||||
// Internal — for freshness gating in render graphs
|
||||
__seUpdatedAt: number // performance.now() at write time
|
||||
}
|
||||
```
|
||||
|
||||
### Router commands
|
||||
|
||||
| Command | Payload | Effect |
|
||||
|---|---|---|
|
||||
| `setStyle` | `{ style: 'default' \| 'fluid' \| 'tendrils' \| 'spirit' }` | Calls `STOP_FN`/`RESUME_FN`, instantiates render graph |
|
||||
| `setPreset` | `{ preset: string }` | Stores in `__STEMENGINE_MUON_PRESET__` |
|
||||
| `setQuality` | `{ quality: 'low' \| 'balanced' \| 'high' }` | Stores in `__STEMENGINE_MUON_QUALITY__` |
|
||||
| `setPaused` | `{ paused: boolean }` | Stores in `__STEMENGINE_MUON_PAUSED__` |
|
||||
| `setViewport` | `{ cssWidth, cssHeight, dpr }` (or `width, height`) | Computes pixel dims, stores in `__STEMENGINE_MUON_VIEWPORT__` |
|
||||
| `loadAudio` (NEW) | `{ url: string }` | Calls `__StemEngineAudioEngine__.loadFromUrl(url)`. On success, starts `__StemEngineFeatureExtractor__` and posts `loaded({ duration })` back to host. On failure, posts `error({ code: 'AUDIO_LOAD', message })`. |
|
||||
| `transportPlay` (NEW) | `{ time?: number, rate?: number }` | `engine.play({ time, rate })` |
|
||||
| `transportPause` (NEW) | `{}` | `engine.pause()` |
|
||||
| `transportSeek` (NEW) | `{ time: number }` | `engine.seek(time)` |
|
||||
| `transportRate` (NEW) | `{ rate: number }` | `engine.setRate(rate)` |
|
||||
| ~~`pushAudioFeatures`~~ | ~~deprecated~~ | Kept for one release cycle as a no-op-or-fallback so older host bundles don't break the iframe. Removable in Phase 6. |
|
||||
|
||||
All command messages have envelope `{ version: 1, id?: string, type: string, payload: object }`. Iframe-emitted events use the same envelope.
|
||||
|
||||
### Iframe-emitted events
|
||||
|
||||
| Type | Payload | When |
|
||||
|---|---|---|
|
||||
| `ready` | `{ capabilities: { webgl, missingExtensions, audioMode: 'iframe-owned' }, styles: ['default','fluid','tendrils','spirit'] }` | After `DOMContentLoaded`, once bootstrap + router are initialized |
|
||||
| `loaded` | `{ duration: number }` | After `loadAudio` resolves |
|
||||
| `error` | `{ code: string, message: string }` | On any internal failure (audio load, shader compile, render-graph init) |
|
||||
|
||||
---
|
||||
|
||||
## 6. Wavesurfer integration (Phase 3 sed patch)
|
||||
|
||||
The original Muon's `vizInit()` in [`/src/index.js`](https://github.com/najafmohammed/muon-music-visualizer/blob/main/src/index.js) does:
|
||||
|
||||
```js
|
||||
analyser = wavesurfer.backend.analyser
|
||||
```
|
||||
|
||||
The Phase 3 sed patch in `Dockerfile` rewrites this to:
|
||||
|
||||
```js
|
||||
analyser = (window.__STEMENGINE_MUON_ANALYSER__ || wavesurfer.backend.analyser)
|
||||
```
|
||||
|
||||
When our iframe AudioEngine has initialized (after `loadAudio`), `window.__STEMENGINE_MUON_ANALYSER__` is populated. The original Muon code now reads from a real analyser fed by a real AudioBufferSourceNode → reactive default mode.
|
||||
|
||||
When AudioEngine is not yet initialized (pre-load), the patch falls back to the silent wavesurfer analyser (zeros) — same as today.
|
||||
|
||||
Side effect: the original Muon also sets `analyser.fftSize = 512`. We set `fftSize = 1024` in the engine. The original Muon's per-frame reset to 512 is fine (reduces `frequencyBinCount` from 512 → 256, halving the data array width Muon expects). FeatureExtractor was already designed against the analyser's *current* `frequencyBinCount` so it self-adjusts. **Test this assumption explicitly during Phase 3.**
|
||||
|
||||
---
|
||||
|
||||
## 7. STOP/RESUME hook (Phase 5 sed patch)
|
||||
|
||||
The router already calls `__STEMENGINE_MUON_STOP_FN__()` and `__STEMENGINE_MUON_RESUME_FN__()` at the right moments. They're no-ops because the original Muon `index.js` never registers them.
|
||||
|
||||
The Phase 5 sed patch injects these registrations into `vizInit()`'s scope:
|
||||
|
||||
```js
|
||||
// Original line:
|
||||
const animate = () => {
|
||||
|
||||
// After sed patch:
|
||||
window.__STEMENGINE_MUON_RESUME_FN__ = () => { pauseAnimation = false; animate() };
|
||||
window.__STEMENGINE_MUON_STOP_FN__ = () => { pauseAnimation = true };
|
||||
const animate = () => {
|
||||
```
|
||||
|
||||
Effect: switching to `fluid` actually pauses the original Muon RAF loop. Switching back to `default` resumes it. No more dual-renderer canvas conflict.
|
||||
|
||||
---
|
||||
|
||||
## 8. Access control: explicitly NONE
|
||||
|
||||
This system has zero authentication, authorization, or origin restriction along the host ↔ iframe ↔ backend path. By design.
|
||||
|
||||
| Layer | What's NOT enforced |
|
||||
|---|---|
|
||||
| Backend CORS | `CORS_ORIGINS=*`, `CORS_ORIGIN_REGEX=.*`. Audio download endpoints accept `Origin: *`. |
|
||||
| Backend auth on download URLs | `/api/jobs/.../download/...` and `/api/subsonic/stream/...` require no header from the iframe. |
|
||||
| Iframe postMessage origin allowlist | `isAllowedOrigin()` returns `true` unconditionally. `STEMENGINE_ALLOWED_ORIGIN` build arg should be deleted. |
|
||||
| Iframe `fetch(audioUrl)` | `mode: 'cors', credentials: 'omit'`. No bearer token, no cookie. |
|
||||
| Iframe sandboxing | Host `<iframe>` has no `sandbox=` attribute (would block AudioContext + WebGL). |
|
||||
| Frame ancestors | nginx in front of iframe sets no `X-Frame-Options`, no `Content-Security-Policy: frame-ancestors`. Iframe is embeddable from any origin. |
|
||||
|
||||
If you find yourself adding any of these restrictions: STOP. Read the deployment context first. The deployment is internal-only (Tailscale / LAN); access control here adds friction with no security gain. The 2026-04-23 origin-allowlist removal documented this trade-off explicitly.
|
||||
|
||||
---
|
||||
|
||||
## 9. Where every consumer plugs in
|
||||
|
||||
There are exactly THREE places in the frontend that mount a Muon iframe. Each must use `useMuonIframeTransport`. No exceptions.
|
||||
|
||||
### `frontend/src/components/LyricsLivePreview.tsx`
|
||||
|
||||
Used by:
|
||||
- `ProjectDetail.tsx` → `LrcPreview` → `LyricsLivePreview` (most common; project pages)
|
||||
- `Karaoke.tsx`
|
||||
- `KaraokeTV.tsx`
|
||||
- `KaraokePerformanceMode.tsx`
|
||||
- `MobileKaraokeShell.tsx`
|
||||
- `SongPreviewSheet.tsx`
|
||||
|
||||
These all wrap or render `LyricsLivePreview`. They do not mount their own iframes.
|
||||
|
||||
The `useMuonIframeTransport` call lives in `LyricsLivePreview` once, near the existing `useAudioFeatures` call (around line 1104). The `audioUrl` argument is the same `audioUrl ?? analysisAudioUrl` value.
|
||||
|
||||
### `frontend/src/components/VisualizerPreviewWidget.tsx`
|
||||
|
||||
Used by:
|
||||
- `TrackDetail.tsx` (Subsonic/Navidrome library tracks at `/library/tracks/:trackId`)
|
||||
|
||||
The `useMuonIframeTransport` call lives here once. The `audioUrl` argument is `subsonicStreamUrl(song.id)`.
|
||||
|
||||
### `frontend/src/pages/VisualizerLabPage.tsx`
|
||||
|
||||
Used by:
|
||||
- `/visualizer-lab` route (developer testing)
|
||||
|
||||
The `useMuonIframeTransport` call lives here once. The lab page's "load file" button computes a Blob URL and passes that as `audioUrl`.
|
||||
|
||||
### Files that should be DELETED after Phase 6
|
||||
|
||||
| File | Why |
|
||||
|---|---|
|
||||
| `frontend/src/components/visualizers/muonAudioBridge.ts` | The `createAudioPushLoop()` function it exports is obsolete once `pushAudioFeatures` postMessage chatter is replaced by transport sync. The `MuonAudioFeaturesPayload` interface moves to `useMuonIframeTransport.ts` if still needed. |
|
||||
|
||||
### Files that should NOT be touched by this work
|
||||
|
||||
| File | Why |
|
||||
|---|---|
|
||||
| `frontend/src/hooks/useAudioFeatures.ts` | Still drives the in-house canvas visualizers. The recent `releaseMediaSource` StrictMode fix stays. |
|
||||
| `frontend/src/audio/MuonFeatureExtractor.ts` | Still used by `useAudioFeatures` for the in-house visualizers. The iframe gets its own port (`muon-iframe/stemengine-feature-extractor.js`) — the two share algorithms but are separate code. |
|
||||
| All `frontend/src/components/visualizers/*VisualizerCanvas.tsx` files | These are React-rendered canvas visualizers, independent of the Muon iframe. |
|
||||
|
||||
---
|
||||
|
||||
## 10. Validation gates
|
||||
|
||||
A change is "done" when ALL of these pass:
|
||||
|
||||
### Default mode reactivity
|
||||
|
||||
1. Open `http://work:5174/projects/<any-project-id>`
|
||||
2. Set visualizer style to default Muon
|
||||
3. Hit play
|
||||
4. Particles, equalizer bars, and color hue all visibly respond to the music
|
||||
5. In iframe DevTools console: `window.__STEMENGINE_MUON_ANALYSER__.getByteFrequencyData(new Uint8Array(256))` returns non-zero values during playback
|
||||
|
||||
### All-modes reactivity
|
||||
|
||||
6. Switch through `default → fluid → tendrils → spirit` while playing
|
||||
7. Each mode reacts to music
|
||||
8. No flicker between modes (Phase 5 RAF coordination)
|
||||
9. No phantom geometry from a stopped renderer
|
||||
|
||||
### Cross-page consistency
|
||||
|
||||
10. Same audio URL plays correctly with reactive visualizer at:
|
||||
- `/projects/<id>` (LyricsLivePreview path)
|
||||
- `/library/tracks/<id>` (VisualizerPreviewWidget / TrackDetail path)
|
||||
- `/visualizer-lab` with a manually loaded file (VisualizerLabPage path)
|
||||
|
||||
### Transport sync
|
||||
|
||||
11. Pause/play from the host transport bar → iframe pauses/plays in sync (no audible double audio because iframe is muted)
|
||||
12. Seek to a different position → iframe seeks within 250ms; visualizer keeps reacting
|
||||
13. Change playback rate (e.g., 1.25x) → iframe playback rate matches
|
||||
|
||||
### Access surface
|
||||
|
||||
14. `grep -rn 'STEMENGINE_ALLOWED_ORIGIN' /home/translator/docker/appdata/stemengine` returns ONLY references in this README and Findings.md (no live code, no env files)
|
||||
15. `curl -I http://work:8000/api/jobs/<jobid>/download/<filename> -H 'Origin: http://work:3011'` returns `Access-Control-Allow-Origin: *` (or matches origin)
|
||||
16. `<iframe src="http://work:3011" />` from any other origin loads and produces a working visualizer (manual test: open `data:text/html,<iframe src='http://work:3011'></iframe>`)
|
||||
|
||||
### No regressions
|
||||
|
||||
17. In-house canvas visualizers (Moire, Dissolve, Glitch, Spiral) still react to audio (they use `useAudioFeatures` directly, untouched)
|
||||
18. `cd frontend && npm run -s typecheck` exits zero
|
||||
19. `cd frontend && npm run -s build:local` succeeds
|
||||
|
||||
---
|
||||
|
||||
## 11. What this does NOT do
|
||||
|
||||
- Does not change the in-house canvas-based visualizers (`MoireVisualizerCanvas` and friends).
|
||||
- Does not change the gloss / lyric / waveform UI overlays.
|
||||
- Does not change the host's ability to play stems via `useStemAudio` (the iframe doesn't care whether the host is playing one stem or many — it just gets a URL and a transport state).
|
||||
- Does not introduce a new audio decoder. The iframe uses native `AudioContext.decodeAudioData()`.
|
||||
- Does not add a new dependency to `frontend/package.json`. The iframe's new modules are vanilla JS, dropped in via Dockerfile.
|
||||
|
||||
---
|
||||
|
||||
## 12. Open questions for the implementer
|
||||
|
||||
| Q | Notes |
|
||||
|---|---|
|
||||
| Iframe-side fftSize: 512 (matches original Muon) or 1024 (Muon-Audio-Analysis default)? | Recommendation: 1024 in the engine; let the original Muon's `analyser.fftSize = 512` reassignment take effect when default mode is active (it just truncates the bin count). FeatureExtractor reads `analyser.frequencyBinCount` dynamically so it adapts. |
|
||||
| Should `loadAudio` accept a Blob (via Transferable) for VisualizerLabPage's local file? | Initial implementation: URL only (lab page uses `URL.createObjectURL(file)`). Blob support is a future enhancement. |
|
||||
| Should `transportSeek` be debounced on the host side? | Recommended: yes. If the user is dragging the seek bar, send `transportSeek` only on `pointerup`, not on every `input` event. The 250ms drift correction loop covers slow seeks. |
|
||||
| Should `loadedDuration` from the iframe override the host's transport duration? | No. Host duration comes from `audioRef.duration` (HTMLMediaElement) which is authoritative for the user-audible track. Iframe `loadedDuration` is for sanity-checking sync only. |
|
||||
|
||||
---
|
||||
|
||||
## 13. Reading order for new contributors
|
||||
|
||||
1. This README (you are here)
|
||||
2. [`Findings.md`](./Findings.md) — *why* the architecture is what it is
|
||||
3. `/tmp/muon-original/src/index.js` lines 81–286 — original Muon's audio pipeline
|
||||
4. `https://github.com/najafmohammed/Muon-Audio-Analysis/blob/main/src/audio/FeatureExtractor.ts` — the algorithm we're porting into the iframe
|
||||
5. `frontend/src/audio/MuonFeatureExtractor.ts` — our existing host-side port (the iframe's port should be a vanilla-JS sibling of this)
|
||||
6. `muon-iframe/stemengine-router.js` — current router; will be extended in Phase 2
|
||||
7. The previous fix folder: `docs/plans/2026-04-24_Muon_No_Reactivity/Findings.md` — explains what's already been fixed in the host pipeline (StrictMode, dead-end graph, silence gate). Most of that work becomes irrelevant for Muon mode after Phase 6, but stays relevant for the in-house visualizers.
|
||||
@@ -0,0 +1,380 @@
|
||||
# Implementation Review — Why It Doesn't Work
|
||||
|
||||
**Date:** 2026-04-27
|
||||
**Subject:** Copilot's implementation of phases 1–6 from `Findings.md` / `README.md`
|
||||
**Reviewer agent skills applied:**
|
||||
- `docs/agents/agency-agents/adapted/reality-checker.md` — verify claims against running code, not narrative
|
||||
- `docs/agents/agency-agents/adapted/evidence-collector.md` — cite line/file for every defect
|
||||
- `docs/agents/agency-agents/adapted/software-architect.md` — separate semantic bugs from cosmetic deviations
|
||||
- `docs/agents/agency-agents/adapted/frontend-developer.md` — check React/TypeScript usage in the host hook
|
||||
|
||||
**Method:** Source-read of all changed files, comparison against the README's invariants (I-1 .. I-7) and validation gates, simulation of execution order at iframe boot.
|
||||
|
||||
**TL;DR:** Phases 1–6 produced *runnable* code (typecheck passes, files exist where the spec said they should) but the **default mode still does not react** because of a synchronous load-order race condition between the iframe's audio engine and the original Muon's `audioAnalyzer()`. The custom modes (fluid/tendrils/spirit) react more weakly than before this change because the iframe's FeatureExtractor publishes un-normalized raw band magnitudes (~0.01–0.05) where the previous host-pushed values were peak-normalized to a real 0–1 range. **The implementation regresses custom-mode reactivity while not delivering default-mode reactivity.**
|
||||
|
||||
---
|
||||
|
||||
## Defect inventory
|
||||
|
||||
| # | Severity | Component | What's broken | Evidence |
|
||||
|---|---|---|---|---|
|
||||
| D-1 | **Critical** | iframe load order | `__STEMENGINE_MUON_ANALYSER__` is undefined when the original Muon `audioAnalyzer()` reads it; default mode permanently reads `wavesurfer.backend.analyser` (silent) | [`stemengine-router.js:333-339`](muon-iframe/stemengine-router.js#L333-L339) + [`/tmp/muon-original/src/index.js:288`](https://github.com/najafmohammed/muon-music-visualizer/blob/main/src/index.js#L288) |
|
||||
| D-2 | **Critical** | iframe load order | Even if D-1 were fixed, the original Muon caches its `analyser` reference once at startup. Subsequent `loadAudio` postMessages don't update what default mode reads. | [`/tmp/muon-original/src/index.js:101-106, 282`](https://github.com/najafmohammed/muon-music-visualizer/blob/main/src/index.js#L101) |
|
||||
| D-3 | **Critical** | iframe FeatureExtractor | Frequency-band values are never peak-normalized. Output is in the 0.005–0.05 range, not 0–1. Custom render graph shader uniforms see effectively-zero values. | [`stemengine-feature-extractor.js:82-95`](muon-iframe/stemengine-feature-extractor.js#L82-L95) vs host's [`useAudioFeatures.ts:287-350`](frontend/src/hooks/useAudioFeatures.ts#L287-L350) |
|
||||
| D-4 | **High** | iframe AudioContext | `AudioContext` is created at `DOMContentLoaded` (no user gesture) and likely stays in `suspended` state forever — Chrome forbids `resume()` outside a user gesture in the iframe context | [`stemengine-audio-engine.js:30-48`](muon-iframe/stemengine-audio-engine.js#L30-L48) and [`stemengine-router.js:332-339`](muon-iframe/stemengine-router.js#L332-L339); [`play()` at line 79-81](muon-iframe/stemengine-audio-engine.js#L79-L81) catches the resume rejection silently |
|
||||
| D-5 | **High** | host `<iframe>` | All three iframe mounts set `sandbox="allow-scripts allow-same-origin"` despite README invariant I-6 saying NO sandbox. The `allow-same-origin` token is meaningless cross-origin (host:5174 vs iframe:3011) and `sandbox` itself adds risk of breaking AudioContext on stricter browser configs | [`LyricsViewport.tsx:100`](frontend/src/components/LyricsViewport.tsx#L100), [`VisualizerPreviewWidget.tsx:581`](frontend/src/components/VisualizerPreviewWidget.tsx#L581), [`VisualizerLabPage.tsx`](frontend/src/pages/VisualizerLabPage.tsx) |
|
||||
| D-6 | **High** | host hook wiring | `LyricsViewport.tsx`'s `<iframe onLoad>` does NOT invoke the hook's `onIframeLoad`. It only sets a separate `setMuonIframeReady` flag. Initialization relies entirely on the iframe-emitted `ready` postMessage, with no DOM-level fallback. | [`LyricsViewport.tsx:101-104`](frontend/src/components/LyricsViewport.tsx#L101) — no call to `props.onMuonIframeLoad` |
|
||||
| D-7 | **High** | iframe Phase 5 sed | `__STEMENGINE_MUON_STOP_FN__` / `__STEMENGINE_MUON_RESUME_FN__` are registered INSIDE `play()` (a closure inside `vizInit()`), only after that closure executes. Until then, the router's calls to these hooks are no-ops. The original Muon RAF loop continues to run during the gap. | [`Dockerfile:55`](muon-iframe/Dockerfile#L55) sed pattern + [`/tmp/muon-original/src/index.js:107, 248`](https://github.com/najafmohammed/muon-music-visualizer/blob/main/src/index.js#L107) — `animate` is declared at line 248 inside `play()`, not module scope |
|
||||
| D-8 | **Medium** | iframe cleanup | Vestigial `STEMENGINE_ALLOWED_ORIGIN` build arg, sed step, and parsing logic remain in the Dockerfile, the GitHub Actions workflow, the router, and `.env.example`. README/Findings explicitly required deletion. | [`Dockerfile:7,32`](muon-iframe/Dockerfile#L7), [`stemengine-router.js:10-22`](muon-iframe/stemengine-router.js#L10-L22), `.env.example:14`, `.github/workflows/muon-iframe-build-publish.yml:33,74,81,85,97` |
|
||||
| D-9 | **Medium** | iframe transport drift | Drift correction calls `engine.seek()` which internally does `pause(); play({ time, rate })` — creating a new `AudioBufferSourceNode` every second during normal playback. The repeated source-node churn isn't broken, but it triggers `onended` race conditions in the engine | [`stemengine-audio-engine.js:118-128`](muon-iframe/stemengine-audio-engine.js#L118) and [`useMuonIframeTransport.ts:182-184`](frontend/src/hooks/useMuonIframeTransport.ts#L182) |
|
||||
| D-10 | **Medium** | Phase 6 leftovers | Router still accepts `pushAudioFeatures` and writes it to `__STEMENGINE_MUON_AUDIO__`. Now competes with the in-iframe FeatureExtractor as a writer — last-write-wins on the global. With the host hook no longer pushing, `pushAudioFeatures` is dead code in practice but remains receivable. | [`stemengine-router.js:55, 210-219`](muon-iframe/stemengine-router.js#L210) |
|
||||
| D-11 | **Low** | spec drift | Router's `ready` event reports `audioMode: 'hybrid'` instead of the README's `'iframe-owned'`. Indicator that the implementer wasn't certain of the architecture. | [`stemengine-router.js:294`](muon-iframe/stemengine-router.js#L294) |
|
||||
| D-12 | **Low** | spec drift | Iframe FeatureExtractor's `treble` derivation is `(highMid + presence + brilliance) / 3`. Host's existing extractor uses `(presence + brilliance) / 2`. README didn't pin this but the inconsistency means render graphs see different `treble` values from the same audio depending on which writer last touched the global. | [`stemengine-feature-extractor.js:218`](muon-iframe/stemengine-feature-extractor.js#L218) vs [`useAudioFeatures.ts:353`](frontend/src/hooks/useAudioFeatures.ts#L353) |
|
||||
|
||||
---
|
||||
|
||||
## D-1, D-2 — Why default mode still doesn't react (deep dive)
|
||||
|
||||
This is the one the entire investigation was supposed to fix. It still doesn't work.
|
||||
|
||||
### Execution order at iframe boot
|
||||
|
||||
The Dockerfile injects imports at the top of `index.js`:
|
||||
|
||||
```dockerfile
|
||||
RUN sed -i '1i import "./stemengine-bootstrap";\n
|
||||
import "./stemengine-audio-engine";\n
|
||||
import "./stemengine-feature-extractor";\n
|
||||
import "./stemengine-router";\n
|
||||
import "./fluid-render-graph";\n
|
||||
import "./tendrils-render-graph";\n
|
||||
import "./spirit-render-graph";' /src/app/src/index.js
|
||||
```
|
||||
|
||||
After webpack bundles, the execution order on page load is:
|
||||
|
||||
1. `stemengine-bootstrap` IIFE runs — installs canvas hook, exposes WebGL baseline. **No side effects on `__STEMENGINE_MUON_ANALYSER__`.**
|
||||
2. `stemengine-audio-engine` IIFE runs — registers `window.__StemEngineAudioEngine__` BUT **does not call `initialize()`**. `__STEMENGINE_MUON_ANALYSER__` is still `undefined`.
|
||||
3. `stemengine-feature-extractor` IIFE runs — registers `window.__StemEngineFeatureExtractor__`, does not start.
|
||||
4. `stemengine-router` IIFE runs — registers `message` listener, registers a `DOMContentLoaded` handler. **The handler is what calls `engine.initialize()`** ([`stemengine-router.js:333`](muon-iframe/stemengine-router.js#L333)). At this moment the handler hasn't fired yet.
|
||||
5. Render-graph IIFEs run.
|
||||
6. The original Muon code in `index.js` runs. The very last line is `window.onload = vizInit()` — note the `()`. This is a Muon-author bug that means **`vizInit()` is invoked synchronously during module initialization**, not on the `load` event. Its return value (a Promise) is assigned to `window.onload`.
|
||||
7. Inside `vizInit()`, the synchronous prefix runs immediately: `wavesurfer = WaveSurfer.create(waveSurferParams)`, then `audioAnalyzer()` at [line 267](https://github.com/najafmohammed/muon-music-visualizer/blob/main/src/index.js#L267), then `play()` at [line 268](https://github.com/najafmohammed/muon-music-visualizer/blob/main/src/index.js#L268).
|
||||
8. **Inside `audioAnalyzer()`, the sed-patched line evaluates:**
|
||||
|
||||
```js
|
||||
analyser = (window.__STEMENGINE_MUON_ANALYSER__ || wavesurfer.backend.analyser);
|
||||
```
|
||||
|
||||
At this exact moment, `__STEMENGINE_MUON_ANALYSER__` is `undefined` (engine.initialize hasn't been called — the router's `DOMContentLoaded` handler hasn't fired). The `||` short-circuits to `wavesurfer.backend.analyser`. **The original Muon's local `analyser` variable is bound to wavesurfer's silent analyser permanently.**
|
||||
9. `play()` runs, starts the Muon RAF loop. Each frame calls `analyser.getByteFrequencyData(dataArray)` on the silent analyser. `dataArray` stays at all zeros. No reactivity.
|
||||
10. *Later*, `DOMContentLoaded` fires. The router's handler runs `engine.initialize()`. `__STEMENGINE_MUON_ANALYSER__` is now defined. **But the original Muon's `analyser` was captured 4 steps ago — it doesn't re-read the global.**
|
||||
|
||||
### Why D-2 makes it worse
|
||||
|
||||
Even fixing D-1 (initializing the engine before vizInit runs) would not survive `loadAudio`. Each `loadAudio` postMessage causes `engine.loadFromUrl()` → `disconnectSource()` + `decodeAudioData()` + new `AudioBuffer`. The `AnalyserNode` instance is unchanged (good). But `__STEMENGINE_MUON_ANALYSER__` always points to the same node — only its connected source changes. That's actually fine.
|
||||
|
||||
The real D-2 issue is different: the original Muon's `audioAnalyzer()` is also called when the user uploads via the iframe's file input ([line 282](https://github.com/najafmohammed/muon-music-visualizer/blob/main/src/index.js#L282)), but never called by us. So if D-1 caused a stale binding, no event re-runs `audioAnalyzer()` to pick up the now-defined global.
|
||||
|
||||
### Fix required (not in current implementation)
|
||||
|
||||
Either:
|
||||
- **A:** Make `stemengine-audio-engine.js` IIFE call `initialize()` synchronously at module evaluation time (top of the file, not inside a function). This way `__STEMENGINE_MUON_ANALYSER__` is set BEFORE `vizInit()` runs.
|
||||
- **B:** Sed-patch `audioAnalyzer()` to be re-callable when our engine completes `loadAudio`. The router's `loadAudio` handler would call a new `window.__STEMENGINE_REACQUIRE_ANALYSER__()` function that sed-injects into `vizInit`'s scope. More fragile.
|
||||
|
||||
Option A is one-line and changes nothing else. The current implementation chose neither.
|
||||
|
||||
---
|
||||
|
||||
## D-3 — Custom modes regress because band values aren't normalized
|
||||
|
||||
The previous architecture had a host-side push pipeline. Host's [`useAudioFeatures.ts:287-350`](frontend/src/hooks/useAudioFeatures.ts#L287-L350) maintained per-band peak normalizers:
|
||||
|
||||
```ts
|
||||
const bandPeaks: Record<keyof MuonFrequencyBands, number> = {
|
||||
sub: 0.001, bass: 0.001, lowMid: 0.001, /* … */
|
||||
}
|
||||
const PEAK_DECAY = 0.9998
|
||||
|
||||
// Per frame:
|
||||
for (const key of BAND_KEYS) {
|
||||
bandPeaks[key] = Math.max(bandPeaks[key] * PEAK_DECAY, rawBands[key])
|
||||
norm[key] = clamp01(rawBands[key] / bandPeaks[key])
|
||||
}
|
||||
```
|
||||
|
||||
This mapped raw band magnitudes (typically 0.001–0.1) to a perceptually useful 0–1 range using a slow-decaying running peak. The result: shader uniforms saw values that swung from ~0 to ~1 with the music.
|
||||
|
||||
The iframe's port at [`stemengine-feature-extractor.js:82-95`](muon-iframe/stemengine-feature-extractor.js#L82-L95):
|
||||
|
||||
```js
|
||||
for (var k = 0; k < keys.length; k += 1) {
|
||||
// ...
|
||||
out[key] = clamp01(Math.sqrt(sumSquares / span)) // raw magnitude only
|
||||
}
|
||||
```
|
||||
|
||||
No peak-normalization. The output is the raw RMS-shaped band energy. For a typical pop song:
|
||||
|
||||
| Band | Raw magnitude (typical) | After clamp01 |
|
||||
|---|---|---|
|
||||
| sub | 0.005–0.020 | 0.005–0.020 |
|
||||
| bass | 0.010–0.040 | 0.010–0.040 |
|
||||
| lowMid | 0.015–0.050 | 0.015–0.050 |
|
||||
| mid | 0.020–0.060 | 0.020–0.060 |
|
||||
| highMid | 0.010–0.030 | 0.010–0.030 |
|
||||
| presence | 0.005–0.020 | 0.005–0.020 |
|
||||
| brilliance | 0.002–0.010 | 0.002–0.010 |
|
||||
|
||||
These values get fed to shader uniforms. A shader that does `vec3 color = mix(base, accent, audio.bass)` with `audio.bass = 0.04` produces a color that is 96% `base` — visually almost no change.
|
||||
|
||||
**Net effect:** the new iframe-owned analysis is *more correct on paper* (real spectrum, not host-pushed summary) but *less useful in practice* (no normalization). The visualizer now reacts ~10× less than it did when reading host-pushed values.
|
||||
|
||||
### Fix required
|
||||
|
||||
Either:
|
||||
- Port `useAudioFeatures.ts:287-350`'s peak-normalization into the iframe FeatureExtractor.
|
||||
- Or apply a fixed scaling: `out[key] = clamp01(rawValue * 30)` as a quick approximation — crude but visible. Peak-normalization is the principled answer.
|
||||
|
||||
---
|
||||
|
||||
## D-4 — AudioContext likely never resumes
|
||||
|
||||
`engine.initialize()` runs at `DOMContentLoaded` — no user gesture. Per Chrome's autoplay policy:
|
||||
|
||||
- AudioContext created without user activation → starts in `'suspended'` state.
|
||||
- `ctx.resume()` outside a user gesture → returns a rejected Promise on Chrome 71+.
|
||||
|
||||
The iframe is a different document from the host. The host's "user clicked play" gesture is consumed by the host's `audioRef.play()` and does NOT propagate to the iframe in modern Chrome (no `User Activation v2` cross-frame inheritance for embedded iframes that don't share a top-level activation).
|
||||
|
||||
[`stemengine-audio-engine.js:79-81`](muon-iframe/stemengine-audio-engine.js#L79-L81):
|
||||
|
||||
```js
|
||||
if (ctx.state === 'suspended') {
|
||||
ctx.resume().catch(function () {}) // ← swallows rejection
|
||||
}
|
||||
```
|
||||
|
||||
Symptoms when this fails:
|
||||
- AudioContext stays `'suspended'`
|
||||
- `AudioBufferSourceNode.start()` succeeds but produces no scheduled audio
|
||||
- `AnalyserNode.getFloatFrequencyData()` returns all `-Infinity` (or `minDecibels`)
|
||||
- FeatureExtractor produces silence-equivalent features
|
||||
- Default mode: same silent analyser as before
|
||||
- Custom modes: zeros for everything
|
||||
|
||||
This is the **silent killer** — no error in console, but the entire iframe analysis pipeline produces no data.
|
||||
|
||||
### Fix required
|
||||
|
||||
The iframe needs to receive a user gesture inside its own document. Options:
|
||||
- Add a transparent click-to-start overlay inside the iframe; user clicks once, AudioContext resumes.
|
||||
- Listen for any `message` from host as a proxy gesture (does NOT count as user activation in Chrome — won't help).
|
||||
- Have the iframe's `<input type="file">` from the original Muon UI proxy the gesture (won't work — we hide it).
|
||||
- The host's `<iframe>` could be programmatically focused, but focus alone doesn't grant activation.
|
||||
|
||||
The robust answer: a one-time click overlay. None of this is implemented.
|
||||
|
||||
---
|
||||
|
||||
## D-5 — Sandbox attribute violates I-6
|
||||
|
||||
[`LyricsViewport.tsx:100`](frontend/src/components/LyricsViewport.tsx#L100):
|
||||
```jsx
|
||||
sandbox="allow-scripts allow-same-origin"
|
||||
```
|
||||
|
||||
[`VisualizerPreviewWidget.tsx:581`](frontend/src/components/VisualizerPreviewWidget.tsx#L581) and [`VisualizerLabPage.tsx`](frontend/src/pages/VisualizerLabPage.tsx) — same.
|
||||
|
||||
README invariant **I-6**: "no `sandbox` on the iframe". This was explicit. Three places still violate.
|
||||
|
||||
`allow-same-origin` is also nonsensical here — the iframe is at `http://work:3011`, the host at `http://work:5174`. Different ports = different origins. There's no "same origin" to allow. The token does nothing useful.
|
||||
|
||||
`allow-scripts` is needed for the iframe to run JS. Without it, nothing in the iframe works. Removing the entire `sandbox` attribute is equivalent to `allow-scripts allow-same-origin allow-forms allow-popups` etc. — the spec said don't add sandbox at all.
|
||||
|
||||
### Why this matters
|
||||
|
||||
`sandbox` interacts with feature policies. `sandbox="allow-scripts allow-same-origin"` in some browsers blocks `Permissions-Policy: autoplay`, which means the iframe's AudioContext autoplay policy is even more restrictive. This compounds D-4.
|
||||
|
||||
### Fix required
|
||||
|
||||
Delete the `sandbox` attribute from all three iframe sites.
|
||||
|
||||
---
|
||||
|
||||
## D-6 — `onIframeLoad` from the hook is destructured but not invoked in LyricsLivePreview
|
||||
|
||||
[`LyricsLivePreview.tsx:1611`](frontend/src/components/LyricsLivePreview.tsx#L1611):
|
||||
```ts
|
||||
const { onIframeLoad: onMuonIframeLoad } = useMuonIframeTransport({ ... })
|
||||
```
|
||||
|
||||
`onMuonIframeLoad` is defined but never passed to or invoked by `LyricsViewport`'s `<iframe onLoad>`. The viewport's onLoad ([`LyricsViewport.tsx:101-104`](frontend/src/components/LyricsViewport.tsx#L101)) only does:
|
||||
|
||||
```jsx
|
||||
onLoad={() => {
|
||||
props.setMuonIframeReady(true)
|
||||
props.setMuonIframeError(false)
|
||||
}}
|
||||
```
|
||||
|
||||
The hook's `onIframeLoad` is supposed to:
|
||||
1. Set `readyRef.current = true` immediately (before postMessage `ready` arrives)
|
||||
2. Send initial settings synchronously
|
||||
3. Send `loadAudio({ url })`
|
||||
|
||||
By skipping it, the hook waits for the iframe-emitted `ready` postMessage as the only initialization trigger. This works (the iframe does emit `ready` from `onReady()` in stemengine-router.js), but it's racier:
|
||||
|
||||
- If the iframe reloads (e.g., `muonIframeUrl` changes after `getAppSettings()` resolves), the new iframe emits a fresh `ready`. Host hook captures it. Probably OK.
|
||||
- If the postMessage `ready` is dropped due to a transient origin mismatch (the hook checks `event.source !== iframeRef.current?.contentWindow` — should be reliable), the hook never initializes. No fallback.
|
||||
|
||||
### Fix required
|
||||
|
||||
Either:
|
||||
- Pass `onMuonIframeLoad` through `LyricsViewport` props and invoke it in the iframe's `onLoad` (DOM-level redundancy with the postMessage `ready`).
|
||||
- Or document that LyricsLivePreview relies solely on postMessage `ready`. Less robust but explicit.
|
||||
|
||||
VisualizerPreviewWidget and VisualizerLabPage do invoke `onMuonIframeLoad` correctly — the divergence here is a regression specific to the LyricsLivePreview path.
|
||||
|
||||
---
|
||||
|
||||
## D-7 — Phase 5 STOP/RESUME registration is conditional on play() executing
|
||||
|
||||
The Phase 5 sed patches `const animate = () => {` inside `play()` to inject the registrations:
|
||||
|
||||
```js
|
||||
window.__STEMENGINE_MUON_RESUME_FN__ = () => { pauseAnimation = false; animate() };
|
||||
window.__STEMENGINE_MUON_STOP_FN__ = () => { pauseAnimation = true };
|
||||
const animate = () => { /* ... */ }
|
||||
```
|
||||
|
||||
These window-globals are only assigned when `play()` actually executes — which happens once at the end of `vizInit()` synchronously. So in practice this works.
|
||||
|
||||
But: if our router's `updateRenderGraph('default')` runs before `play()` (via the `DOMContentLoaded` fast path), `RESUME_FN` is `undefined` at that moment and the call is a no-op. The original Muon's RAF loop still starts later via `play() → animate()`, so default mode renders. But if a user starts in a custom mode (`?style=fluid`), the router's first `updateRenderGraph('fluid')` calls `STOP_FN()` which is `undefined`. The original Muon RAF loop still launches. Dual-renderer canvas conflict resumes.
|
||||
|
||||
### Fix required
|
||||
|
||||
Sed-patch the registrations to fire at `vizInit` entry, not inside `play()`. Or move the `pauseAnimation` declaration outside `play()` so the hooks can be registered at module scope. Or do the registration from the router by referencing the original Muon's function via export — much more invasive.
|
||||
|
||||
The simplest fix: change the sed pattern to target `const vizInit = async () => {` instead of `const animate = () => {`. Then `RESUME_FN` would close over `pauseAnimation` (module-scope) and `play` (module-scope const, assignable to `animate` only from inside `play()`'s scope — won't work cleanly).
|
||||
|
||||
Actually the cleanest fix: register a global getter that defers `animate` resolution:
|
||||
|
||||
```js
|
||||
window.__STEMENGINE_MUON_STOP_FN__ = () => { pauseAnimation = true };
|
||||
window.__STEMENGINE_MUON_RESUME_FN__ = () => {
|
||||
pauseAnimation = false;
|
||||
if (typeof window.__STEMENGINE_MUON_RAF_KICK__ === 'function') window.__STEMENGINE_MUON_RAF_KICK__();
|
||||
};
|
||||
```
|
||||
|
||||
Then sed-patch `play()` to also assign `window.__STEMENGINE_MUON_RAF_KICK__ = animate`. The STOP/RESUME globals are available immediately; the RAF kick becomes available once `play()` runs.
|
||||
|
||||
Current implementation does none of this.
|
||||
|
||||
---
|
||||
|
||||
## D-8 — Cleanup not done
|
||||
|
||||
README explicitly listed the cleanup of `STEMENGINE_ALLOWED_ORIGIN`. Confirmed not done:
|
||||
|
||||
```bash
|
||||
$ grep -rn 'STEMENGINE_ALLOWED_ORIGIN' /home/translator/docker/appdata/stemengine
|
||||
.env.example:14:STEMENGINE_ALLOWED_ORIGIN=*
|
||||
muon-iframe/stemengine-router.js:4: * The string __STEMENGINE_ALLOWED_ORIGIN__ is replaced by sed during the
|
||||
muon-iframe/stemengine-router.js:5: * Docker build with the value of the STEMENGINE_ALLOWED_ORIGIN build arg
|
||||
muon-iframe/stemengine-router.js:10: var ALLOWED_ORIGIN_RAW = '__STEMENGINE_ALLOWED_ORIGIN__'
|
||||
muon-iframe/stemengine-router.js:11: var PLACEHOLDER_TOKEN = '__STEMENGINE' + '_ALLOWED_ORIGIN__'
|
||||
muon-iframe/stemengine-router.js:12: var ALLOWED_ORIGINS = ...
|
||||
muon-iframe/stemengine-router.js:16: var ACCEPT_ANY_ORIGIN = ...
|
||||
muon-iframe/stemengine-router.js:17: var HAS_ALLOWED_ORIGINS = ...
|
||||
muon-iframe/Dockerfile:7:ARG STEMENGINE_ALLOWED_ORIGIN=http://localhost:5173
|
||||
muon-iframe/Dockerfile:32: && sed -i "s|__STEMENGINE_ALLOWED_ORIGIN__|${STEMENGINE_ALLOWED_ORIGIN}|g" ...
|
||||
.github/workflows/muon-iframe-build-publish.yml:33,74,81,85,97
|
||||
```
|
||||
|
||||
`isAllowedOrigin()` returning `true` makes the parsing logic dead. The build arg, the env var, and the GHA workflow inputs all reference a vestigial concept. README's validation gate #14 explicitly required this grep to return zero matches. Currently returns ~13 matches across 4 files.
|
||||
|
||||
This is cosmetic / maintenance debt, not a functional bug. But it directly violates the explicit cleanup item.
|
||||
|
||||
---
|
||||
|
||||
## What's right (credit where due)
|
||||
|
||||
| Area | Status |
|
||||
|---|---|
|
||||
| File creation matches spec (3 new files) | ✅ |
|
||||
| Window globals registered with the spec'd names | ✅ |
|
||||
| Router added all 5 transport commands with correct payload shapes | ✅ |
|
||||
| Host hook signature matches the README contract exactly | ✅ |
|
||||
| Three mounting sites all use `useMuonIframeTransport` | ✅ |
|
||||
| `muonAudioBridge.ts` was deleted as required | ✅ |
|
||||
| Dockerfile import order has bootstrap → engine → extractor → router → render graphs | ✅ |
|
||||
| Phase 3 sed pattern matches the original Muon code exactly | ✅ |
|
||||
| Phase 5 sed pattern is syntactically valid | ✅ |
|
||||
| Engine's `disconnectSource()` and `onended` cleanup are robust | ✅ |
|
||||
| FeatureExtractor's algorithmic shape mirrors Muon-Audio-Analysis | ✅ |
|
||||
| Typecheck passes (`npm run -s typecheck` exit 0) | ✅ |
|
||||
|
||||
The structural work is sound. The failures are timing, normalization, and cleanup — not architecture.
|
||||
|
||||
---
|
||||
|
||||
## Validation gates: pass/fail against the README
|
||||
|
||||
Re-running the README's section-10 validation checklist mentally against the current code:
|
||||
|
||||
| # | Gate | Result | Why |
|
||||
|---|---|---|---|
|
||||
| 1–4 | Default mode reactivity | ❌ FAIL | D-1 (analyser not bound) + D-4 (ctx suspended) |
|
||||
| 5 | `__STEMENGINE_MUON_ANALYSER__.getByteFrequencyData()` non-zero | ❌ FAIL | D-4 |
|
||||
| 6 | All-modes reactivity (visible response) | ❌ FAIL custom modes | D-3 (un-normalized values) + D-4 (silent ctx) |
|
||||
| 7 | No flicker between modes | ❌ FAIL | D-7 (STOP_FN may be no-op early) |
|
||||
| 8 | No phantom geometry | ⚠️ PARTIAL | Same as 7 |
|
||||
| 9 | LyricsLivePreview path works | ❌ FAIL | D-1, D-3, D-4, D-6 |
|
||||
| 10 | TrackDetail/VisualizerPreviewWidget path works | ❌ FAIL | D-1, D-3, D-4 |
|
||||
| 10b | VisualizerLabPage path works | ❌ FAIL | D-1, D-3, D-4 |
|
||||
| 11–13 | Transport sync | ⚠️ Loop runs but iframe is silent (D-4) — host can't observe whether sync is correct |
|
||||
| 14 | `grep STEMENGINE_ALLOWED_ORIGIN` clean | ❌ FAIL | D-8 |
|
||||
| 15 | Backend CORS to iframe origin | ✅ PASS | unchanged from prior compose simplification |
|
||||
| 16 | Iframe embeddable from any origin | ⚠️ Sandboxed (D-5) |
|
||||
| 17 | In-house canvas visualizers still react | ✅ PASS | `useAudioFeatures` was untouched |
|
||||
| 18 | `npm run -s typecheck` zero | ✅ PASS | confirmed |
|
||||
| 19 | `npm run -s build:local` succeeds | not run; likely passes |
|
||||
|
||||
**Failure rate: 11/19 gates fail. None of the user-visible reactivity gates pass.**
|
||||
|
||||
---
|
||||
|
||||
## Severity-ordered fix list
|
||||
|
||||
If continuing this work, fix in this order — each is independent:
|
||||
|
||||
1. **D-4 (silent killer)** — add a click-to-resume overlay inside the iframe, OR have host pre-create the AudioContext via the user's play gesture and transfer ownership somehow. Without this, NOTHING works regardless of other fixes.
|
||||
2. **D-1 (race)** — move `engine.initialize()` from the router's `DOMContentLoaded` handler to the top of the audio-engine IIFE. One-line change.
|
||||
3. **D-3 (un-normalized bands)** — port the host's peak-normalization (`bandPeaks` ring with `PEAK_DECAY=0.9998`) into the iframe FeatureExtractor. ~20 lines.
|
||||
4. **D-7 (Phase 5 hooks)** — fix the sed pattern so STOP/RESUME globals are bound at module scope, not inside `play()`.
|
||||
5. **D-2 (analyser re-acquire)** — only relevant after D-1; if D-1 is fixed, the global is set in time and D-2 doesn't bite.
|
||||
6. **D-5 (sandbox)** — delete the `sandbox` attribute from all three iframe sites.
|
||||
7. **D-6 (LyricsViewport onLoad)** — wire `onMuonIframeLoad` through the props and invoke it.
|
||||
8. **D-8 (cleanup)** — delete `STEMENGINE_ALLOWED_ORIGIN` references from Dockerfile, router, GHA workflow, env example.
|
||||
9. **D-9 (drift seek churn)** — change `engine.seek()` while playing to schedule a new BufferSource starting at the new offset without disconnecting the old one until the new one starts; or accept the churn since iframe is muted.
|
||||
10. **D-10, D-11, D-12** — cosmetic; can be deferred.
|
||||
|
||||
---
|
||||
|
||||
## Why these defects slipped past the implementer
|
||||
|
||||
This isn't a code-quality complaint — it's a process observation, useful for the next pass:
|
||||
|
||||
- **Validation was assumed, not performed.** The status line in Findings.md says "Phases 1–6 are now implemented" but no validation gate was actually run against a live browser. Defects D-1, D-3, D-4 are invisible at typecheck time.
|
||||
- **Order-of-execution wasn't simulated.** D-1 requires reasoning about webpack module init order plus the original Muon's `window.onload = vizInit()` synchronous-call quirk. Easy to miss without reading the upstream code.
|
||||
- **Host vs iframe audio analysis architecture was conflated.** The iframe's FeatureExtractor was treated as a 1:1 port of the host's MuonFeatureExtractor — but the host has a normalization layer in `useAudioFeatures.ts` that's separate from `MuonFeatureExtractor.ts`. Easy to miss because it's in a different file.
|
||||
- **Chrome's autoplay policy in iframes.** Cross-document user activation rules are obscure. The "AudioContext starts suspended" trap is well-documented but easy to forget when porting code that worked in the host (which gets activation from the click handler).
|
||||
- **Cleanup items in the README were treated as optional.** They were not — README invariant I-6 was explicit. The implementer chose to keep `STEMENGINE_ALLOWED_ORIGIN` "just in case" and added `sandbox=` "for safety" — both directly violate the spec.
|
||||
|
||||
---
|
||||
|
||||
## Recommendation
|
||||
|
||||
Treat the implementation as **the structural skeleton** of phases 1–6, but mark phases 1, 3, 4, and 5 as **not actually working** despite shipping. Do not close the underlying issue. The next pass needs the four fixes above (D-4, D-1, D-3, D-7) applied together — none of them work standalone.
|
||||
@@ -440,8 +440,23 @@ export const getSubsonicSong = (songId: string) =>
|
||||
export const getSimilarSongs = (songId: string, count = 5, signal?: AbortSignal) =>
|
||||
request<SubsonicSong[]>(`${BASE}/subsonic/songs/${encodeURIComponent(songId)}/similar?count=${encodeURIComponent(String(count))}`, { signal })
|
||||
|
||||
export const subsonicStreamUrl = (songId: string) =>
|
||||
resolveApiUrl(`${BASE}/subsonic/stream/${encodeURIComponent(songId)}`)
|
||||
export const subsonicStreamUrl = (
|
||||
songId: string,
|
||||
options?: {
|
||||
format?: string
|
||||
maxBitRate?: number
|
||||
},
|
||||
) => {
|
||||
const qs = new URLSearchParams()
|
||||
const format = String(options?.format ?? '').trim().toLowerCase()
|
||||
if (format) qs.set('format', format)
|
||||
const maxBitRate = Number(options?.maxBitRate)
|
||||
if (Number.isFinite(maxBitRate) && maxBitRate > 0) {
|
||||
qs.set('maxBitRate', String(Math.floor(maxBitRate)))
|
||||
}
|
||||
const suffix = qs.size ? `?${qs.toString()}` : ''
|
||||
return resolveApiUrl(`${BASE}/subsonic/stream/${encodeURIComponent(songId)}${suffix}`)
|
||||
}
|
||||
|
||||
export const subsonicCoverArtUrl = (coverId: string, size?: number) => {
|
||||
const qs = new URLSearchParams()
|
||||
|
||||
@@ -9,11 +9,11 @@ const KARAOKE_TOGGLE_EVENT = 'stemengine:karaoke-toggle-playback'
|
||||
import KaraokeEditor from './KaraokeEditor'
|
||||
import { useVisualizerPalette } from '../hooks/useVisualizerPalette'
|
||||
import { useAudioFeatures } from '../hooks/useAudioFeatures'
|
||||
import { useMuonIframeTransport } from '../hooks/useMuonIframeTransport'
|
||||
import { useStemAudio } from '../hooks/useStemAudio'
|
||||
import { getAppSettings, type SekDynamicPalette } from '../api/client'
|
||||
import type { VisualizerMode, VisualizerSettings } from './visualizers/visualizerTypes'
|
||||
import { deepMergeVizSettings, normalizeVizSettings, sekSettingsToVizPatch, VIZ_DEFAULTS, type DeepPartial, type VizSettings } from './visualizers/visualizerSettings'
|
||||
import { createAudioPushLoop } from './visualizers/muonAudioBridge'
|
||||
import PerformanceSidebar from './PerformanceSidebar.tsx'
|
||||
import { waitForMediaReady } from '../utils/waitForMediaReady'
|
||||
import LyricsStylePicker from './LyricsStylePicker'
|
||||
@@ -64,13 +64,48 @@ const VISUALIZER_STYLE_MODES = new Set<Exclude<VisualizerMode, null>>(['muon', '
|
||||
const LYRICS_VISUAL_STYLES = new Set<LyricsVisualStyle>(['scroll', 'fade', 'karaoke', 'glow', 'blur', 'video', 'muon_iframe'])
|
||||
|
||||
const DEFAULT_MUON_IFRAME_URL = 'http://localhost:3011'
|
||||
const MUON_IFRAME_REV = '20260422b'
|
||||
const MUON_IFRAME_REV = '20260427c'
|
||||
|
||||
function isLoopbackHost(hostname: string): boolean {
|
||||
const value = hostname.trim().toLowerCase()
|
||||
return value === 'localhost' || value === '127.0.0.1' || value === '::1'
|
||||
}
|
||||
|
||||
function defaultMuonIframeUrl(): string {
|
||||
if (typeof window === 'undefined') return DEFAULT_MUON_IFRAME_URL
|
||||
const protocol = window.location.protocol === 'https:' ? 'https:' : 'http:'
|
||||
const hostname = window.location.hostname || 'localhost'
|
||||
return `${protocol}//${hostname}:3011`
|
||||
}
|
||||
|
||||
function resolveMuonIframeUrl(raw: string | undefined): string {
|
||||
const cleaned = String(raw || '').trim()
|
||||
if (!cleaned) return DEFAULT_MUON_IFRAME_URL
|
||||
if (!/^https?:\/\//i.test(cleaned)) return DEFAULT_MUON_IFRAME_URL
|
||||
return cleaned
|
||||
if (!cleaned) return defaultMuonIframeUrl()
|
||||
|
||||
if (/^https?:\/\//i.test(cleaned)) {
|
||||
try {
|
||||
const parsed = new URL(cleaned)
|
||||
if (typeof window !== 'undefined' && isLoopbackHost(parsed.hostname) && !isLoopbackHost(window.location.hostname)) {
|
||||
parsed.hostname = window.location.hostname
|
||||
}
|
||||
return parsed.toString()
|
||||
} catch {
|
||||
return defaultMuonIframeUrl()
|
||||
}
|
||||
}
|
||||
|
||||
const protocol = typeof window !== 'undefined' && window.location.protocol === 'https:' ? 'https://' : 'http://'
|
||||
const candidate = cleaned.startsWith('//') ? `${protocol}${cleaned.slice(2)}` : `${protocol}${cleaned}`
|
||||
try {
|
||||
const parsed = new URL(candidate)
|
||||
if (!/^https?:$/i.test(parsed.protocol)) return defaultMuonIframeUrl()
|
||||
if (typeof window !== 'undefined' && isLoopbackHost(parsed.hostname) && !isLoopbackHost(window.location.hostname)) {
|
||||
parsed.hostname = window.location.hostname
|
||||
}
|
||||
return parsed.toString()
|
||||
} catch {
|
||||
return defaultMuonIframeUrl()
|
||||
}
|
||||
}
|
||||
|
||||
function withDefaultMuonPreset(rawUrl: string): string {
|
||||
@@ -904,15 +939,10 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, projectId, so
|
||||
const desiredVideoSegmentRef = useRef<{ index: number; localTime: number } | null>(null)
|
||||
const videoPlaylistRef = useRef<Array<{ url: string; duration?: number }>>(resolvedVideoPlaylist)
|
||||
const iframeRef = useRef<HTMLIFrameElement>(null)
|
||||
const muonIframeReadyRef = useRef(false)
|
||||
const [muonIframeUrl, setMuonIframeUrl] = useState(DEFAULT_MUON_IFRAME_URL)
|
||||
const [muonIframeUrl, setMuonIframeUrl] = useState(() => defaultMuonIframeUrl())
|
||||
const [muonIframeError, setMuonIframeError] = useState(false)
|
||||
const muonEmbedUrl = useMemo(() => withDefaultMuonPreset(muonIframeUrl), [muonIframeUrl])
|
||||
|
||||
useEffect(() => {
|
||||
muonIframeReadyRef.current = false
|
||||
}, [muonEmbedUrl])
|
||||
|
||||
useEffect(() => {
|
||||
videoPlaylistRef.current = resolvedVideoPlaylist
|
||||
setVideoPlaylistIndex(0)
|
||||
@@ -1108,6 +1138,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, projectId, so
|
||||
enabled: isPlaying,
|
||||
analysisSourceKey: `${analysisAudioUrl ?? audioUrl ?? ''}|${audioCtxVersion}`,
|
||||
bpmSeed: analysisBpm,
|
||||
analyseOnly: true,
|
||||
})
|
||||
|
||||
const containerRef = useRef<HTMLDivElement>(null)
|
||||
@@ -1612,33 +1643,28 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, projectId, so
|
||||
const isPerformanceView = performanceMode || isFullscreen
|
||||
const [lyricsViewportScale, setLyricsViewportScale] = useState(1)
|
||||
const [visualizerSize, setVisualizerSize] = useState({ width: 960, height: 540 })
|
||||
const iframeVisualSettingsRef = useRef(liveVizSettings.muon_iframe)
|
||||
|
||||
useEffect(() => {
|
||||
iframeVisualSettingsRef.current = liveVizSettings.muon_iframe
|
||||
}, [liveVizSettings.muon_iframe])
|
||||
|
||||
const sendMuonCommand = useCallback((type: string, payload: Record<string, unknown>) => {
|
||||
if (!muonIframeReadyRef.current) return
|
||||
iframeRef.current?.contentWindow?.postMessage(
|
||||
{ version: 1, id: `lyrics-${Date.now()}`, type, payload },
|
||||
'*',
|
||||
)
|
||||
}, [])
|
||||
|
||||
const rehydrateMuonIframe = useCallback(() => {
|
||||
sendMuonCommand('setStyle', { style: iframeVisualSettingsRef.current.style })
|
||||
sendMuonCommand('setPreset', { preset: iframeVisualSettingsRef.current.preset })
|
||||
sendMuonCommand('setQuality', { quality: iframeVisualSettingsRef.current.quality })
|
||||
sendMuonCommand('setViewport', {
|
||||
const { onIframeLoad: onMuonIframeLoad } = useMuonIframeTransport({
|
||||
iframeRef,
|
||||
isActive: isIframeVisualizerMode,
|
||||
audioUrl: audioUrl ?? null,
|
||||
isPlayingRef,
|
||||
getCurrentTime: () => currentTimeRef.current,
|
||||
getRate: () => {
|
||||
if (useStemTransport) {
|
||||
return Number(pickStemClockSource(syncAnchorStemRef.current)?.playbackRate ?? 1)
|
||||
}
|
||||
return Number(audioRef.current?.playbackRate ?? videoRef.current?.playbackRate ?? 1)
|
||||
},
|
||||
style: liveVizSettings.muon_iframe.style,
|
||||
preset: liveVizSettings.muon_iframe.preset,
|
||||
quality: liveVizSettings.muon_iframe.quality,
|
||||
viewport: {
|
||||
cssWidth: visualizerSize.width,
|
||||
cssHeight: visualizerSize.height,
|
||||
dpr: typeof window === 'undefined' ? 1 : window.devicePixelRatio || 1,
|
||||
})
|
||||
sendMuonCommand('setPaused', {
|
||||
paused: typeof document !== 'undefined' ? document.hidden || !isPlayingRef.current : !isPlayingRef.current,
|
||||
})
|
||||
}, [sendMuonCommand, visualizerSize.height, visualizerSize.width])
|
||||
},
|
||||
commandIdPrefix: 'lyrics',
|
||||
})
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false
|
||||
@@ -1649,80 +1675,13 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, projectId, so
|
||||
})
|
||||
.catch(() => {
|
||||
if (cancelled) return
|
||||
setMuonIframeUrl(DEFAULT_MUON_IFRAME_URL)
|
||||
setMuonIframeUrl(defaultMuonIframeUrl())
|
||||
})
|
||||
return () => {
|
||||
cancelled = true
|
||||
}
|
||||
}, [])
|
||||
|
||||
useEffect(() => {
|
||||
if (!isIframeVisualizerMode) return
|
||||
sendMuonCommand('setStyle', { style: liveVizSettings.muon_iframe.style })
|
||||
}, [isIframeVisualizerMode, liveVizSettings.muon_iframe.style, sendMuonCommand])
|
||||
|
||||
useEffect(() => {
|
||||
if (!isIframeVisualizerMode) return
|
||||
sendMuonCommand('setPreset', { preset: liveVizSettings.muon_iframe.preset })
|
||||
}, [isIframeVisualizerMode, liveVizSettings.muon_iframe.preset, sendMuonCommand])
|
||||
|
||||
useEffect(() => {
|
||||
if (!isIframeVisualizerMode) return
|
||||
sendMuonCommand('setQuality', { quality: liveVizSettings.muon_iframe.quality })
|
||||
}, [isIframeVisualizerMode, liveVizSettings.muon_iframe.quality, sendMuonCommand])
|
||||
|
||||
useEffect(() => {
|
||||
if (!isIframeVisualizerMode) return
|
||||
sendMuonCommand('setViewport', {
|
||||
cssWidth: visualizerSize.width,
|
||||
cssHeight: visualizerSize.height,
|
||||
dpr: typeof window === 'undefined' ? 1 : window.devicePixelRatio || 1,
|
||||
})
|
||||
}, [isIframeVisualizerMode, sendMuonCommand, visualizerSize.height, visualizerSize.width])
|
||||
|
||||
useEffect(() => {
|
||||
if (!isIframeVisualizerMode) return
|
||||
sendMuonCommand('setPaused', {
|
||||
paused: typeof document !== 'undefined' ? document.hidden || !isPlaying : !isPlaying,
|
||||
})
|
||||
}, [isIframeVisualizerMode, isPlaying, sendMuonCommand])
|
||||
|
||||
useEffect(() => {
|
||||
const onMessage = (event: MessageEvent) => {
|
||||
if (event.source !== iframeRef.current?.contentWindow) return
|
||||
const msg = event.data as { type?: string } | null
|
||||
if (!msg || typeof msg !== 'object') return
|
||||
if (msg.type === 'ready') {
|
||||
muonIframeReadyRef.current = true
|
||||
rehydrateMuonIframe()
|
||||
}
|
||||
}
|
||||
window.addEventListener('message', onMessage)
|
||||
if (isIframeVisualizerMode) rehydrateMuonIframe()
|
||||
return () => window.removeEventListener('message', onMessage)
|
||||
}, [isIframeVisualizerMode, rehydrateMuonIframe])
|
||||
|
||||
useEffect(() => {
|
||||
const onVisible = () => {
|
||||
if (!isIframeVisualizerMode) return
|
||||
sendMuonCommand('setPaused', { paused: document.hidden || !isPlayingRef.current })
|
||||
if (!document.hidden) rehydrateMuonIframe()
|
||||
}
|
||||
document.addEventListener('visibilitychange', onVisible)
|
||||
return () => document.removeEventListener('visibilitychange', onVisible)
|
||||
}, [isIframeVisualizerMode, rehydrateMuonIframe, sendMuonCommand])
|
||||
|
||||
// Push audio features to Muon iframe each RAF while playing.
|
||||
useEffect(() => {
|
||||
if (!isIframeVisualizerMode || !isPlaying) return
|
||||
const bridge = createAudioPushLoop(
|
||||
sendMuonCommand,
|
||||
() => featuresRef.current,
|
||||
)
|
||||
bridge.start()
|
||||
return () => bridge.stop()
|
||||
}, [isIframeVisualizerMode, isPlaying, sendMuonCommand, featuresRef])
|
||||
|
||||
useEffect(() => {
|
||||
if (!isPerformanceView) {
|
||||
setShowPerfSidebar(false)
|
||||
@@ -2649,7 +2608,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, projectId, so
|
||||
songTitle={songTitle}
|
||||
title={title}
|
||||
setMuonIframeReady={(ready) => {
|
||||
muonIframeReadyRef.current = ready
|
||||
if (ready) onMuonIframeLoad()
|
||||
}}
|
||||
setMuonIframeError={setMuonIframeError}
|
||||
setDuration={setDuration}
|
||||
|
||||
@@ -90,15 +90,17 @@ function LyricsViewport(props: LyricsViewportProps) {
|
||||
{props.isIframeVisualizerMode && (
|
||||
<div style={{ position: 'absolute', inset: 0, zIndex: 0, pointerEvents: 'none' }}>
|
||||
<iframe
|
||||
key={props.muonEmbedUrl}
|
||||
ref={props.iframeRef}
|
||||
src={props.muonEmbedUrl}
|
||||
title="Muon music visualizer"
|
||||
className="absolute inset-0 h-full w-full border-0"
|
||||
loading="lazy"
|
||||
loading="eager"
|
||||
referrerPolicy="no-referrer"
|
||||
allow="autoplay"
|
||||
sandbox="allow-scripts allow-same-origin"
|
||||
onLoad={() => {
|
||||
// setMuonIframeReady(true) is routed by LyricsLivePreview to
|
||||
// useMuonIframeTransport's onIframeLoad — no separate prop needed.
|
||||
props.setMuonIframeReady(true)
|
||||
props.setMuonIframeError(false)
|
||||
}}
|
||||
|
||||
@@ -2,10 +2,10 @@ import { forwardRef, useCallback, useEffect, useImperativeHandle, useLayoutEffec
|
||||
import { Disc3 } from 'lucide-react'
|
||||
import { MuonFeatureExtractor } from '../audio/MuonFeatureExtractor'
|
||||
import { useAudioFeatures, type AudioFeatures } from '../hooks/useAudioFeatures'
|
||||
import { useMuonIframeTransport } from '../hooks/useMuonIframeTransport'
|
||||
import { useVisualizerPalette } from '../hooks/useVisualizerPalette'
|
||||
import { normalizeStemFeatureChannel, type StemFeatureChannel } from '../utils/visualizerStemChannels'
|
||||
import VisualizerBackdrop, { type VisualizerMode } from './visualizers/VisualizerBackdrop'
|
||||
import { createAudioPushLoop } from './visualizers/muonAudioBridge'
|
||||
import type { VisualizerSettings } from './visualizers/visualizerTypes'
|
||||
import { registerPlayback, unregisterPlayback } from '../utils/audioRegistry'
|
||||
import { getAppSettings } from '../api/client'
|
||||
@@ -32,13 +32,48 @@ type NativePreviewMode = Extract<VisualizerMode, 'moire' | 'dissolve' | 'glitch'
|
||||
type PreviewMode = 'muon_iframe' | 'video' | NativePreviewMode
|
||||
|
||||
const DEFAULT_MUON_IFRAME_URL = 'http://localhost:3011'
|
||||
const MUON_IFRAME_REV = '20260422b'
|
||||
const MUON_IFRAME_REV = '20260427c'
|
||||
|
||||
function isLoopbackHost(hostname: string): boolean {
|
||||
const value = hostname.trim().toLowerCase()
|
||||
return value === 'localhost' || value === '127.0.0.1' || value === '::1'
|
||||
}
|
||||
|
||||
function defaultMuonIframeUrl(): string {
|
||||
if (typeof window === 'undefined') return DEFAULT_MUON_IFRAME_URL
|
||||
const protocol = window.location.protocol === 'https:' ? 'https:' : 'http:'
|
||||
const hostname = window.location.hostname || 'localhost'
|
||||
return `${protocol}//${hostname}:3011`
|
||||
}
|
||||
|
||||
function resolveMuonIframeUrl(raw: string | undefined): string {
|
||||
const cleaned = String(raw || '').trim()
|
||||
if (!cleaned) return DEFAULT_MUON_IFRAME_URL
|
||||
if (!/^https?:\/\//i.test(cleaned)) return DEFAULT_MUON_IFRAME_URL
|
||||
return cleaned
|
||||
if (!cleaned) return defaultMuonIframeUrl()
|
||||
|
||||
if (/^https?:\/\//i.test(cleaned)) {
|
||||
try {
|
||||
const parsed = new URL(cleaned)
|
||||
if (typeof window !== 'undefined' && isLoopbackHost(parsed.hostname) && !isLoopbackHost(window.location.hostname)) {
|
||||
parsed.hostname = window.location.hostname
|
||||
}
|
||||
return parsed.toString()
|
||||
} catch {
|
||||
return defaultMuonIframeUrl()
|
||||
}
|
||||
}
|
||||
|
||||
const protocol = typeof window !== 'undefined' && window.location.protocol === 'https:' ? 'https://' : 'http://'
|
||||
const candidate = cleaned.startsWith('//') ? `${protocol}${cleaned.slice(2)}` : `${protocol}${cleaned}`
|
||||
try {
|
||||
const parsed = new URL(candidate)
|
||||
if (!/^https?:$/i.test(parsed.protocol)) return defaultMuonIframeUrl()
|
||||
if (typeof window !== 'undefined' && isLoopbackHost(parsed.hostname) && !isLoopbackHost(window.location.hostname)) {
|
||||
parsed.hostname = window.location.hostname
|
||||
}
|
||||
return parsed.toString()
|
||||
} catch {
|
||||
return defaultMuonIframeUrl()
|
||||
}
|
||||
}
|
||||
|
||||
function withDefaultPreset(rawUrl: string): string {
|
||||
@@ -118,7 +153,6 @@ const VisualizerPreviewWidget = forwardRef<HTMLAudioElement, VisualizerPreviewWi
|
||||
useImperativeHandle(ref, () => audioRef.current!, [])
|
||||
const videoRef = useRef<HTMLVideoElement>(null)
|
||||
const iframeRef = useRef<HTMLIFrameElement>(null)
|
||||
const muonIframeReadyRef = useRef(false)
|
||||
const audioCtxRef = useRef<AudioContext | null>(null)
|
||||
const isPlayingRef = useRef(false)
|
||||
const stemAudioRefs = useRef<Record<string, HTMLAudioElement>>({})
|
||||
@@ -133,18 +167,11 @@ const VisualizerPreviewWidget = forwardRef<HTMLAudioElement, VisualizerPreviewWi
|
||||
const [ctxVersion, setCtxVersion] = useState(0)
|
||||
const [stemAnalyserVersion, setStemAnalyserVersion] = useState(0)
|
||||
const [muonIframeError, setMuonIframeError] = useState(false)
|
||||
const [muonIframeUrl, setMuonIframeUrl] = useState(DEFAULT_MUON_IFRAME_URL)
|
||||
const [muonIframeUrl, setMuonIframeUrl] = useState(() => defaultMuonIframeUrl())
|
||||
const [muonQuality, setMuonQuality] = useState<'low' | 'balanced' | 'high'>(visualizerSettings?.muon_iframe?.quality ?? 'balanced')
|
||||
const [muonStyle, setMuonStyle] = useState<MuonStyle>((visualizerSettings?.muon_iframe?.style as MuonStyle | undefined) ?? muonStyleFromPreferredMode(preferredMode))
|
||||
const [muonPreset, setMuonPreset] = useState<string>(visualizerSettings?.muon_iframe?.preset ?? 'default')
|
||||
const dimsRef = useRef<{ w: number; h: number } | null>(null)
|
||||
const muonStyleRef = useRef<MuonStyle>(muonStyle)
|
||||
const muonPresetRef = useRef<string>(muonPreset)
|
||||
const muonQualityRef = useRef<'low' | 'balanced' | 'high'>(muonQuality)
|
||||
const muonEmbedUrl = useMemo(() => withDefaultPreset(muonIframeUrl), [muonIframeUrl])
|
||||
useEffect(() => {
|
||||
muonIframeReadyRef.current = false
|
||||
}, [muonEmbedUrl])
|
||||
|
||||
const normalizedVideoUrl = useMemo(() => String(videoUrl || '').trim(), [videoUrl])
|
||||
const canPreviewVideo = useMemo(() => isDirectVideoUrl(normalizedVideoUrl), [normalizedVideoUrl])
|
||||
@@ -181,6 +208,24 @@ const VisualizerPreviewWidget = forwardRef<HTMLAudioElement, VisualizerPreviewWi
|
||||
[stemFeaturesByChannel],
|
||||
)
|
||||
|
||||
const { onIframeLoad: onMuonIframeLoad } = useMuonIframeTransport({
|
||||
iframeRef,
|
||||
isActive: mode === 'muon_iframe',
|
||||
audioUrl: audioUrl || null,
|
||||
isPlayingRef,
|
||||
getCurrentTime: () => Number(audioRef.current?.currentTime || 0),
|
||||
getRate: () => Number(audioRef.current?.playbackRate || 1),
|
||||
style: muonStyle,
|
||||
preset: muonPreset,
|
||||
quality: muonQuality,
|
||||
viewport: dims ? {
|
||||
cssWidth: dims.w,
|
||||
cssHeight: dims.h,
|
||||
dpr: typeof window === 'undefined' ? 1 : window.devicePixelRatio || 1,
|
||||
} : undefined,
|
||||
commandIdPrefix: 'vis',
|
||||
})
|
||||
|
||||
const tabs = useMemo(
|
||||
() => [
|
||||
{ id: 'muon_iframe' as const, label: 'Muon' },
|
||||
@@ -238,37 +283,6 @@ const VisualizerPreviewWidget = forwardRef<HTMLAudioElement, VisualizerPreviewWi
|
||||
setIsPlaying(false)
|
||||
}, [])
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Muon postMessage helpers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const sendMuonCommand = useCallback(
|
||||
(type: string, payload: Record<string, unknown>) => {
|
||||
if (!muonIframeReadyRef.current) return
|
||||
iframeRef.current?.contentWindow?.postMessage(
|
||||
{ version: 1, id: `vis-${Date.now()}`, type, payload },
|
||||
'*',
|
||||
)
|
||||
},
|
||||
[],
|
||||
)
|
||||
|
||||
const rehydrateMuonState = useCallback(() => {
|
||||
sendMuonCommand('setStyle', { style: muonStyleRef.current })
|
||||
sendMuonCommand('setPreset', { preset: muonPresetRef.current })
|
||||
sendMuonCommand('setQuality', { quality: muonQualityRef.current })
|
||||
if (dimsRef.current) {
|
||||
sendMuonCommand('setViewport', {
|
||||
cssWidth: dimsRef.current.w,
|
||||
cssHeight: dimsRef.current.h,
|
||||
dpr: typeof window === 'undefined' ? 1 : window.devicePixelRatio || 1,
|
||||
})
|
||||
}
|
||||
sendMuonCommand('setPaused', {
|
||||
paused: typeof document !== 'undefined' ? document.hidden || !isPlayingRef.current : !isPlayingRef.current,
|
||||
})
|
||||
}, [sendMuonCommand])
|
||||
|
||||
useEffect(() => {
|
||||
const nextStyle = (visualizerSettings?.muon_iframe?.style as MuonStyle | undefined) ?? muonStyleFromPreferredMode(preferredMode)
|
||||
setMuonStyle(nextStyle)
|
||||
@@ -276,104 +290,6 @@ const VisualizerPreviewWidget = forwardRef<HTMLAudioElement, VisualizerPreviewWi
|
||||
setMuonQuality(visualizerSettings?.muon_iframe?.quality ?? 'balanced')
|
||||
}, [preferredMode, visualizerSettings])
|
||||
|
||||
useEffect(() => {
|
||||
muonStyleRef.current = muonStyle
|
||||
}, [muonStyle])
|
||||
|
||||
useEffect(() => {
|
||||
muonPresetRef.current = muonPreset
|
||||
}, [muonPreset])
|
||||
|
||||
useEffect(() => {
|
||||
muonQualityRef.current = muonQuality
|
||||
}, [muonQuality])
|
||||
|
||||
// Send setStyle whenever selected style changes and we're in muon_iframe mode
|
||||
useEffect(() => {
|
||||
if (mode !== 'muon_iframe') return
|
||||
sendMuonCommand('setStyle', { style: muonStyle })
|
||||
}, [mode, muonStyle, sendMuonCommand])
|
||||
|
||||
// Send setPreset whenever selected preset changes and we're in muon_iframe mode
|
||||
useEffect(() => {
|
||||
if (mode !== 'muon_iframe') return
|
||||
sendMuonCommand('setPreset', { preset: muonPreset })
|
||||
}, [mode, muonPreset, sendMuonCommand])
|
||||
|
||||
// Send setQuality whenever muonQuality changes and we're in muon_iframe mode
|
||||
useEffect(() => {
|
||||
if (mode !== 'muon_iframe') return
|
||||
sendMuonCommand('setQuality', { quality: muonQuality })
|
||||
}, [mode, muonQuality, sendMuonCommand])
|
||||
|
||||
// Send setViewport (debounced ≥100 ms) when container dims change
|
||||
const setViewportTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null)
|
||||
useEffect(() => {
|
||||
if (mode !== 'muon_iframe' || !dims) return
|
||||
if (setViewportTimerRef.current !== null) clearTimeout(setViewportTimerRef.current)
|
||||
setViewportTimerRef.current = setTimeout(() => {
|
||||
setViewportTimerRef.current = null
|
||||
sendMuonCommand('setViewport', {
|
||||
cssWidth: dims.w,
|
||||
cssHeight: dims.h,
|
||||
dpr: typeof window === 'undefined' ? 1 : window.devicePixelRatio || 1,
|
||||
})
|
||||
}, 100)
|
||||
return () => {
|
||||
if (setViewportTimerRef.current !== null) {
|
||||
clearTimeout(setViewportTimerRef.current)
|
||||
setViewportTimerRef.current = null
|
||||
}
|
||||
}
|
||||
}, [mode, dims, sendMuonCommand])
|
||||
|
||||
useEffect(() => {
|
||||
dimsRef.current = dims
|
||||
}, [dims])
|
||||
|
||||
// Push audio features to Muon iframe each RAF while playing.
|
||||
useEffect(() => {
|
||||
if (mode !== 'muon_iframe' || !isPlaying) return
|
||||
const bridge = createAudioPushLoop(sendMuonCommand, () => featuresRef.current)
|
||||
bridge.start()
|
||||
return () => bridge.stop()
|
||||
// featuresRef is a stable mutable ref — read inside RAF loop, not a restart trigger
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [mode, isPlaying, sendMuonCommand])
|
||||
|
||||
// Listen for ready / error events emitted by the Muon iframe
|
||||
useEffect(() => {
|
||||
const onMessage = (event: MessageEvent) => {
|
||||
if (event.source !== iframeRef.current?.contentWindow) return
|
||||
const msg = event.data as { type?: string }
|
||||
if (!msg || typeof msg !== 'object') return
|
||||
if (msg.type === 'ready') {
|
||||
muonIframeReadyRef.current = true
|
||||
rehydrateMuonState()
|
||||
}
|
||||
}
|
||||
window.addEventListener('message', onMessage)
|
||||
if (mode === 'muon_iframe') rehydrateMuonState()
|
||||
return () => window.removeEventListener('message', onMessage)
|
||||
}, [mode, rehydrateMuonState])
|
||||
|
||||
useEffect(() => {
|
||||
if (mode !== 'muon_iframe') return
|
||||
sendMuonCommand('setPaused', {
|
||||
paused: typeof document !== 'undefined' ? document.hidden || !isPlayingRef.current : !isPlayingRef.current,
|
||||
})
|
||||
return () => {
|
||||
sendMuonCommand('setPaused', { paused: true })
|
||||
}
|
||||
}, [mode, sendMuonCommand])
|
||||
|
||||
useEffect(() => {
|
||||
if (mode !== 'muon_iframe') return
|
||||
sendMuonCommand('setPaused', {
|
||||
paused: typeof document !== 'undefined' ? document.hidden || !isPlaying : !isPlaying,
|
||||
})
|
||||
}, [isPlaying, mode, sendMuonCommand])
|
||||
|
||||
useEffect(() => {
|
||||
isPlayingRef.current = isPlaying
|
||||
}, [isPlaying])
|
||||
@@ -475,7 +391,7 @@ const VisualizerPreviewWidget = forwardRef<HTMLAudioElement, VisualizerPreviewWi
|
||||
})
|
||||
.catch(() => {
|
||||
if (cancelled) return
|
||||
setMuonIframeUrl(DEFAULT_MUON_IFRAME_URL)
|
||||
setMuonIframeUrl(defaultMuonIframeUrl())
|
||||
})
|
||||
return () => {
|
||||
cancelled = true
|
||||
@@ -548,19 +464,14 @@ const VisualizerPreviewWidget = forwardRef<HTMLAudioElement, VisualizerPreviewWi
|
||||
|
||||
useEffect(() => {
|
||||
const onVisible = () => {
|
||||
const paused = document.hidden || !isPlayingRef.current
|
||||
if (!document.hidden && isPlayingRef.current && audioCtxRef.current?.state === 'suspended') {
|
||||
void audioCtxRef.current.resume().catch(() => {})
|
||||
}
|
||||
|
||||
if (mode !== 'muon_iframe') return
|
||||
sendMuonCommand('setPaused', { paused })
|
||||
if (!document.hidden) rehydrateMuonState()
|
||||
}
|
||||
|
||||
document.addEventListener('visibilitychange', onVisible)
|
||||
return () => document.removeEventListener('visibilitychange', onVisible)
|
||||
}, [mode, rehydrateMuonState, sendMuonCommand])
|
||||
}, [])
|
||||
|
||||
useEffect(() => {
|
||||
const video = videoRef.current
|
||||
@@ -683,7 +594,7 @@ const VisualizerPreviewWidget = forwardRef<HTMLAudioElement, VisualizerPreviewWi
|
||||
>
|
||||
<audio ref={audioRef} src={audioUrl} crossOrigin="anonymous" preload="metadata" />
|
||||
|
||||
<div ref={containerRef} className="relative w-full overflow-hidden rounded-t-lg bg-gray-900" style={{ aspectRatio: '16 / 9' }}>
|
||||
<div ref={containerRef} className="relative w-full min-h-[180px] overflow-hidden rounded-t-lg bg-gray-900" style={{ aspectRatio: '16 / 9' }}>
|
||||
{mode === 'video' && canPreviewVideo ? (
|
||||
<video
|
||||
ref={videoRef}
|
||||
@@ -695,21 +606,19 @@ const VisualizerPreviewWidget = forwardRef<HTMLAudioElement, VisualizerPreviewWi
|
||||
/>
|
||||
) : mode === 'muon_iframe' ? (
|
||||
<iframe
|
||||
key={muonEmbedUrl}
|
||||
ref={iframeRef}
|
||||
src={muonEmbedUrl}
|
||||
title="Muon music visualizer"
|
||||
className="absolute inset-0 h-full w-full border-0"
|
||||
loading="lazy"
|
||||
loading="eager"
|
||||
referrerPolicy="no-referrer"
|
||||
allow="autoplay"
|
||||
sandbox="allow-scripts allow-same-origin"
|
||||
onLoad={() => {
|
||||
muonIframeReadyRef.current = true
|
||||
setMuonIframeError(false)
|
||||
rehydrateMuonState()
|
||||
onMuonIframeLoad()
|
||||
}}
|
||||
onError={() => {
|
||||
muonIframeReadyRef.current = false
|
||||
setMuonIframeError(true)
|
||||
}}
|
||||
/>
|
||||
|
||||
@@ -1,145 +0,0 @@
|
||||
/**
|
||||
* muonAudioBridge.ts
|
||||
*
|
||||
* Bridges the host-side AudioFeatures analyser into the Muon iframe via
|
||||
* `pushAudioFeatures` postMessage commands.
|
||||
*
|
||||
* Contract (muon-iframe/stemengine-router.js captureFeatures):
|
||||
* { bass, treble, rms, energy, beat }
|
||||
* - All numeric fields are 0–1.
|
||||
* - `beat` is boolean (maps from AudioFeatures.beatDetected).
|
||||
* - `energy` is accepted but not read by any v1 render graph; included for
|
||||
* forward compat.
|
||||
*
|
||||
* Usage:
|
||||
* const bridge = createAudioPushLoop(sendMuonCommand, () => featuresRef.current)
|
||||
* bridge.start() // call when mode === 'muon_iframe' && isPlaying
|
||||
* bridge.stop() // call on cleanup / pause / mode change
|
||||
*/
|
||||
|
||||
export interface MuonAudioFeaturesPayload {
|
||||
/** 0–1 normalized bass energy */
|
||||
bass: number
|
||||
/** 0–1 normalized treble energy */
|
||||
treble: number
|
||||
/** 0–1 RMS amplitude */
|
||||
rms: number
|
||||
/** 0–1 broadband energy (reserved; inert in v1 render graphs) */
|
||||
energy: number
|
||||
/** true on beat frames */
|
||||
beat: boolean
|
||||
/** 0-1 normalized percussion prominence */
|
||||
drumIntensity?: number
|
||||
/** 0-1 normalized sub band energy */
|
||||
sub?: number
|
||||
/** Beat cycle phase 0..1 */
|
||||
beatPhase?: number
|
||||
/** Estimated BPM */
|
||||
bpm?: number
|
||||
/** Beat detection confidence 0..1 */
|
||||
beatConfidence?: number
|
||||
/** Pattern classifier flags */
|
||||
isBeatDrop?: boolean
|
||||
isBuildUp?: boolean
|
||||
isSilence?: boolean
|
||||
/** Frame-to-frame energy delta */
|
||||
energyDelta?: number
|
||||
/** Additional spectral descriptors */
|
||||
spectralCentroid?: number
|
||||
spectralFlatness?: number
|
||||
}
|
||||
|
||||
type SendCommand = (type: string, payload: Record<string, unknown>) => void
|
||||
type GetFeatures = () => {
|
||||
bass: number
|
||||
treble: number
|
||||
rms: number
|
||||
energy: number
|
||||
beatDetected: boolean
|
||||
drumIntensity?: number
|
||||
frequencyBands?: { sub?: number }
|
||||
beatInfo?: { beatPhase?: number; bpm?: number; confidence?: number }
|
||||
patternInfo?: { isBeatDrop?: boolean; isBuildUp?: boolean; isSilence?: boolean; energyDelta?: number }
|
||||
spectralCentroid?: number
|
||||
spectralFlatness?: number
|
||||
}
|
||||
|
||||
function clamp(value: number | undefined | null): number {
|
||||
if (typeof value !== 'number' || !Number.isFinite(value)) return 0
|
||||
return Math.max(0, Math.min(1, value))
|
||||
}
|
||||
|
||||
/** Number of consecutive silent frames before push is suppressed.
|
||||
* 60 frames (~1 second at 60fps) covers the stem analyser initialization window. */
|
||||
const SILENCE_GATE_FRAMES = 60
|
||||
|
||||
/**
|
||||
* Creates a RAF-driven loop that pushes audio features to the Muon iframe.
|
||||
*
|
||||
* @param sendCommand - The postMessage sender returned by useMuonCommand or
|
||||
* the equivalent `sendMuonCommand` callback in each integration.
|
||||
* @param getFeatures - Callback invoked each frame to get current audio features,
|
||||
* typically `() => featuresRef.current` from useAudioFeatures.
|
||||
*/
|
||||
export function createAudioPushLoop(
|
||||
sendCommand: SendCommand,
|
||||
getFeatures: GetFeatures,
|
||||
): { start: () => void; stop: () => void } {
|
||||
let rafId: number | null = null
|
||||
let silentFrames = 0
|
||||
|
||||
function tick() {
|
||||
rafId = requestAnimationFrame(tick)
|
||||
|
||||
// Visibility guard — don't push while tab is hidden.
|
||||
if (typeof document !== 'undefined' && document.visibilityState === 'hidden') {
|
||||
return
|
||||
}
|
||||
|
||||
const features = getFeatures()
|
||||
const { bass, treble, rms, energy, beatDetected: beat } = features
|
||||
|
||||
// Silence gate — suppress pushes after sustained silence.
|
||||
if ((rms ?? 0) < 0.001) {
|
||||
silentFrames++
|
||||
if (silentFrames >= SILENCE_GATE_FRAMES) return
|
||||
} else {
|
||||
silentFrames = 0
|
||||
}
|
||||
|
||||
const payload: MuonAudioFeaturesPayload = {
|
||||
bass: clamp(bass),
|
||||
treble: clamp(treble),
|
||||
rms: clamp(rms),
|
||||
energy: clamp(energy),
|
||||
beat: beat ?? false,
|
||||
...(features.drumIntensity != null && { drumIntensity: clamp(features.drumIntensity) }),
|
||||
...(features.frequencyBands?.sub != null && { sub: clamp(features.frequencyBands.sub) }),
|
||||
...(features.beatInfo?.beatPhase != null && { beatPhase: clamp(features.beatInfo.beatPhase) }),
|
||||
...(features.beatInfo?.bpm != null && { bpm: features.beatInfo.bpm }),
|
||||
...(features.beatInfo?.confidence != null && { beatConfidence: clamp(features.beatInfo.confidence) }),
|
||||
...(features.patternInfo?.isBeatDrop != null && { isBeatDrop: features.patternInfo.isBeatDrop }),
|
||||
...(features.patternInfo?.isBuildUp != null && { isBuildUp: features.patternInfo.isBuildUp }),
|
||||
...(features.patternInfo?.isSilence != null && { isSilence: features.patternInfo.isSilence }),
|
||||
...(features.patternInfo?.energyDelta != null && { energyDelta: features.patternInfo.energyDelta }),
|
||||
...(features.spectralCentroid != null && { spectralCentroid: clamp(features.spectralCentroid) }),
|
||||
...(features.spectralFlatness != null && { spectralFlatness: clamp(features.spectralFlatness) }),
|
||||
}
|
||||
|
||||
sendCommand('pushAudioFeatures', payload as unknown as Record<string, unknown>)
|
||||
}
|
||||
|
||||
return {
|
||||
start() {
|
||||
if (rafId !== null) return // already running
|
||||
silentFrames = 0
|
||||
rafId = requestAnimationFrame(tick)
|
||||
},
|
||||
stop() {
|
||||
if (rafId !== null) {
|
||||
cancelAnimationFrame(rafId)
|
||||
rafId = null
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -51,6 +51,10 @@ interface UseAudioFeaturesArgs {
|
||||
enabled?: boolean
|
||||
analysisSourceKey?: string | null
|
||||
bpmSeed?: number | null
|
||||
/** When true, routes through a gain=0 node to ctx.destination so the graph is
|
||||
* active for analysis without producing audible output. Use for hidden analysis
|
||||
* elements that run alongside a separate playback transport. */
|
||||
analyseOnly?: boolean
|
||||
}
|
||||
|
||||
const EMPTY_FREQ = new Float32Array(0)
|
||||
@@ -143,7 +147,12 @@ function releaseMediaSource(ctx: AudioContext, audioEl: HTMLMediaElement, node:
|
||||
} catch {
|
||||
// noop
|
||||
}
|
||||
map.delete(audioEl)
|
||||
// Do NOT delete from map. The HTMLMediaElement stays permanently attached to this
|
||||
// AudioContext after the first createMediaElementSource call (Web Audio spec). Deleting
|
||||
// here would cause the next acquireMediaSource to call createMediaElementSource again,
|
||||
// which throws (or returns a stale node in older browsers). Keeping refs=0 in the map
|
||||
// lets the next acquire increment refs and return the existing node for reconnection.
|
||||
// The WeakMap automatically evicts the entry when the element is garbage-collected.
|
||||
}
|
||||
}
|
||||
|
||||
@@ -186,6 +195,7 @@ export function useAudioFeatures({
|
||||
enabled = true,
|
||||
analysisSourceKey,
|
||||
bpmSeed,
|
||||
analyseOnly = false,
|
||||
}: UseAudioFeaturesArgs): MutableRefObject<AudioFeatures> {
|
||||
const featuresRef = useRef<AudioFeatures>({
|
||||
bass: 0,
|
||||
@@ -260,6 +270,21 @@ export function useAudioFeatures({
|
||||
|
||||
mediaSource.connect(analyser)
|
||||
|
||||
// Connect the graph to the destination so the Web Audio engine processes it.
|
||||
// Without a path to the destination, Chrome/Firefox treat the subgraph as
|
||||
// inactive and AnalyserNode reads return silence regardless of input.
|
||||
let passthroughGain: GainNode | null = null
|
||||
if (analyseOnly) {
|
||||
// Silent passthrough: keeps graph active for analysis without audible output.
|
||||
// Used when a separate hidden audio element runs alongside a playback transport.
|
||||
passthroughGain = ctx.createGain()
|
||||
passthroughGain.gain.value = 0
|
||||
analyser.connect(passthroughGain)
|
||||
passthroughGain.connect(ctx.destination)
|
||||
} else {
|
||||
analyser.connect(ctx.destination)
|
||||
}
|
||||
|
||||
const extractor = new MuonFeatureExtractor()
|
||||
if (typeof bpmSeed === 'number' && Number.isFinite(bpmSeed) && bpmSeed > 0) {
|
||||
extractor.seedBpm(bpmSeed)
|
||||
@@ -368,6 +393,13 @@ export function useAudioFeatures({
|
||||
} catch {
|
||||
// noop
|
||||
}
|
||||
if (passthroughGain) {
|
||||
try {
|
||||
passthroughGain.disconnect()
|
||||
} catch {
|
||||
// noop
|
||||
}
|
||||
}
|
||||
releaseMediaSource(ownedCtx, ownedAudioEl, ownedSource)
|
||||
|
||||
if (mediaSourceRef.current === ownedSource) {
|
||||
@@ -380,7 +412,7 @@ export function useAudioFeatures({
|
||||
sourceContextRef.current = null
|
||||
}
|
||||
}
|
||||
}, [audioCtxRef, analysisAudioRef, isPlayingRef, enabled, analysisSourceKey, bpmSeed])
|
||||
}, [audioCtxRef, analysisAudioRef, isPlayingRef, enabled, analysisSourceKey, bpmSeed, analyseOnly])
|
||||
|
||||
return featuresRef
|
||||
}
|
||||
@@ -0,0 +1,317 @@
|
||||
import { useCallback, useEffect, useRef, useState, type RefObject } from 'react'
|
||||
|
||||
export interface MuonViewport {
|
||||
cssWidth: number
|
||||
cssHeight: number
|
||||
dpr: number
|
||||
}
|
||||
|
||||
export interface UseMuonIframeTransportArgs {
|
||||
iframeRef: RefObject<HTMLIFrameElement | null>
|
||||
isActive: boolean
|
||||
audioUrl: string | null
|
||||
isPlayingRef: RefObject<boolean>
|
||||
getCurrentTime: () => number
|
||||
getRate?: () => number
|
||||
style: 'default' | 'fluid' | 'tendrils' | 'spirit'
|
||||
preset?: string
|
||||
quality?: 'low' | 'balanced' | 'high'
|
||||
viewport?: MuonViewport
|
||||
commandIdPrefix?: string
|
||||
}
|
||||
|
||||
export interface UseMuonIframeTransportResult {
|
||||
onIframeLoad: () => void
|
||||
sendCommand: (type: string, payload: Record<string, unknown>) => void
|
||||
loadedDuration: number | null
|
||||
isLoaded: boolean
|
||||
}
|
||||
|
||||
function readRate(getRate?: () => number): number {
|
||||
const value = Number(getRate ? getRate() : 1)
|
||||
return Number.isFinite(value) && value > 0 ? value : 1
|
||||
}
|
||||
|
||||
function readTime(getCurrentTime: () => number): number {
|
||||
const value = Number(getCurrentTime())
|
||||
return Number.isFinite(value) && value >= 0 ? value : 0
|
||||
}
|
||||
|
||||
function resolveAudioUrl(raw: string): string {
|
||||
const value = String(raw || '').trim()
|
||||
if (!value) return ''
|
||||
|
||||
if (/^https?:\/\//i.test(value)) return value
|
||||
|
||||
if (value.startsWith('//')) {
|
||||
const protocol = typeof window !== 'undefined' && window.location.protocol === 'https:' ? 'https:' : 'http:'
|
||||
return `${protocol}${value}`
|
||||
}
|
||||
|
||||
if (typeof window === 'undefined') return value
|
||||
try {
|
||||
return new URL(value, window.location.origin).toString()
|
||||
} catch {
|
||||
return value
|
||||
}
|
||||
}
|
||||
|
||||
export function useMuonIframeTransport(args: UseMuonIframeTransportArgs): UseMuonIframeTransportResult {
|
||||
const {
|
||||
iframeRef,
|
||||
isActive,
|
||||
audioUrl,
|
||||
isPlayingRef,
|
||||
getCurrentTime,
|
||||
getRate,
|
||||
style,
|
||||
preset,
|
||||
quality,
|
||||
viewport,
|
||||
commandIdPrefix = 'muon',
|
||||
} = args
|
||||
|
||||
const readyRef = useRef(false)
|
||||
const frameRef = useRef<number | null>(null)
|
||||
const lastSyncTimeRef = useRef(-1)
|
||||
const lastRateRef = useRef(1)
|
||||
const lastPlayingRef = useRef(false)
|
||||
const commandSeqRef = useRef(0)
|
||||
const lastRequestedAudioUrlRef = useRef<string | null>(null)
|
||||
const lastIframeErrorKeyRef = useRef<string | null>(null)
|
||||
|
||||
const isActiveRef = useRef(isActive)
|
||||
const audioUrlRef = useRef(audioUrl)
|
||||
const styleRef = useRef(style)
|
||||
const presetRef = useRef(preset)
|
||||
const qualityRef = useRef(quality)
|
||||
const viewportRef = useRef(viewport)
|
||||
|
||||
const [loadedDuration, setLoadedDuration] = useState<number | null>(null)
|
||||
const [isLoaded, setIsLoaded] = useState(false)
|
||||
|
||||
useEffect(() => {
|
||||
isActiveRef.current = isActive
|
||||
}, [isActive])
|
||||
|
||||
useEffect(() => {
|
||||
audioUrlRef.current = audioUrl
|
||||
}, [audioUrl])
|
||||
|
||||
useEffect(() => {
|
||||
styleRef.current = style
|
||||
}, [style])
|
||||
|
||||
useEffect(() => {
|
||||
presetRef.current = preset
|
||||
}, [preset])
|
||||
|
||||
useEffect(() => {
|
||||
qualityRef.current = quality
|
||||
}, [quality])
|
||||
|
||||
useEffect(() => {
|
||||
viewportRef.current = viewport
|
||||
}, [viewport])
|
||||
|
||||
const sendCommand = useCallback((type: string, payload: Record<string, unknown>) => {
|
||||
if (!readyRef.current) return
|
||||
const target = iframeRef.current?.contentWindow
|
||||
if (!target) return
|
||||
commandSeqRef.current += 1
|
||||
target.postMessage(
|
||||
{
|
||||
version: 1,
|
||||
id: `${commandIdPrefix}-${Date.now()}-${commandSeqRef.current}`,
|
||||
type,
|
||||
payload,
|
||||
},
|
||||
'*',
|
||||
)
|
||||
}, [commandIdPrefix, iframeRef])
|
||||
|
||||
const sendSettings = useCallback(() => {
|
||||
sendCommand('setStyle', { style: styleRef.current })
|
||||
sendCommand('setPreset', { preset: presetRef.current })
|
||||
sendCommand('setQuality', { quality: qualityRef.current })
|
||||
if (viewportRef.current) {
|
||||
sendCommand('setViewport', {
|
||||
cssWidth: viewportRef.current.cssWidth,
|
||||
cssHeight: viewportRef.current.cssHeight,
|
||||
dpr: viewportRef.current.dpr,
|
||||
})
|
||||
}
|
||||
}, [sendCommand])
|
||||
|
||||
const sendLoadAudio = useCallback((force = false) => {
|
||||
const url = resolveAudioUrl(String(audioUrlRef.current || ''))
|
||||
if (!url) return
|
||||
if (!force && lastRequestedAudioUrlRef.current === url) return
|
||||
lastRequestedAudioUrlRef.current = url
|
||||
setIsLoaded(false)
|
||||
setLoadedDuration(null)
|
||||
sendCommand('loadAudio', { url })
|
||||
lastSyncTimeRef.current = -1
|
||||
}, [sendCommand])
|
||||
|
||||
const onIframeLoad = useCallback(() => {
|
||||
readyRef.current = true
|
||||
sendSettings()
|
||||
sendCommand('setPaused', {
|
||||
paused: typeof document !== 'undefined' ? document.hidden || !Boolean(isPlayingRef.current) : !Boolean(isPlayingRef.current),
|
||||
})
|
||||
sendLoadAudio(true)
|
||||
}, [isPlayingRef, sendCommand, sendLoadAudio, sendSettings])
|
||||
|
||||
useEffect(() => {
|
||||
if (!isActive) {
|
||||
lastPlayingRef.current = false
|
||||
sendCommand('transportPause', {})
|
||||
sendCommand('setPaused', { paused: true })
|
||||
if (frameRef.current !== null) {
|
||||
cancelAnimationFrame(frameRef.current)
|
||||
frameRef.current = null
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
const tick = () => {
|
||||
if (!isActiveRef.current) return
|
||||
const playing = Boolean(isPlayingRef.current)
|
||||
|
||||
if (playing !== lastPlayingRef.current) {
|
||||
if (playing) {
|
||||
const time = readTime(getCurrentTime)
|
||||
const rate = readRate(getRate)
|
||||
sendCommand('transportPlay', { time, rate })
|
||||
sendCommand('setPaused', {
|
||||
paused: typeof document !== 'undefined' ? document.hidden : false,
|
||||
})
|
||||
lastRateRef.current = rate
|
||||
lastSyncTimeRef.current = time
|
||||
} else {
|
||||
sendCommand('transportPause', {})
|
||||
sendCommand('setPaused', { paused: true })
|
||||
}
|
||||
lastPlayingRef.current = playing
|
||||
}
|
||||
|
||||
if (playing) {
|
||||
const nextTime = readTime(getCurrentTime)
|
||||
if (Math.abs(nextTime - lastSyncTimeRef.current) > 0.25) {
|
||||
sendCommand('transportSeek', {
|
||||
time: nextTime,
|
||||
rate: readRate(getRate),
|
||||
})
|
||||
lastSyncTimeRef.current = nextTime
|
||||
}
|
||||
|
||||
const nextRate = readRate(getRate)
|
||||
if (Math.abs(nextRate - lastRateRef.current) > 0.001) {
|
||||
sendCommand('transportRate', { rate: nextRate })
|
||||
lastRateRef.current = nextRate
|
||||
}
|
||||
}
|
||||
|
||||
frameRef.current = requestAnimationFrame(tick)
|
||||
}
|
||||
|
||||
frameRef.current = requestAnimationFrame(tick)
|
||||
return () => {
|
||||
if (frameRef.current !== null) {
|
||||
cancelAnimationFrame(frameRef.current)
|
||||
frameRef.current = null
|
||||
}
|
||||
}
|
||||
}, [getCurrentTime, getRate, isActive, isPlayingRef, sendCommand])
|
||||
|
||||
useEffect(() => {
|
||||
if (!isActive) return
|
||||
sendSettings()
|
||||
}, [isActive, quality, preset, sendSettings, style, viewport])
|
||||
|
||||
useEffect(() => {
|
||||
if (!isActive || !audioUrl) return
|
||||
lastRequestedAudioUrlRef.current = null
|
||||
sendLoadAudio()
|
||||
}, [audioUrl, isActive, sendLoadAudio])
|
||||
|
||||
useEffect(() => {
|
||||
const onMessage = (event: MessageEvent) => {
|
||||
if (event.source !== iframeRef.current?.contentWindow) return
|
||||
const message = event.data as { type?: string; payload?: { duration?: unknown; message?: unknown; code?: unknown } } | null
|
||||
if (!message || typeof message !== 'object') return
|
||||
|
||||
if (message.type === 'ready') {
|
||||
readyRef.current = true
|
||||
sendSettings()
|
||||
sendCommand('setPaused', {
|
||||
paused: typeof document !== 'undefined' ? document.hidden || !Boolean(isPlayingRef.current) : !Boolean(isPlayingRef.current),
|
||||
})
|
||||
if (audioUrlRef.current) sendLoadAudio()
|
||||
return
|
||||
}
|
||||
|
||||
if (message.type === 'loaded') {
|
||||
const duration = Number(message.payload?.duration)
|
||||
setLoadedDuration(Number.isFinite(duration) && duration > 0 ? duration : null)
|
||||
setIsLoaded(true)
|
||||
return
|
||||
}
|
||||
|
||||
if (message.type === 'error') {
|
||||
const code = String(message.payload?.code || 'UNKNOWN')
|
||||
const details = String(message.payload?.message || 'unknown iframe error')
|
||||
const errorKey = `${code}:${details}`
|
||||
if (lastIframeErrorKeyRef.current === errorKey) return
|
||||
lastIframeErrorKeyRef.current = errorKey
|
||||
|
||||
// AUDIO_LOAD decode failures can be environment/media specific and are often
|
||||
// non-actionable at runtime, so avoid polluting the console with warnings.
|
||||
if (code === 'AUDIO_LOAD') {
|
||||
return
|
||||
}
|
||||
|
||||
console.warn('[useMuonIframeTransport]', code, details)
|
||||
}
|
||||
}
|
||||
|
||||
window.addEventListener('message', onMessage)
|
||||
return () => window.removeEventListener('message', onMessage)
|
||||
}, [iframeRef, isPlayingRef, sendCommand, sendLoadAudio, sendSettings])
|
||||
|
||||
useEffect(() => {
|
||||
const onVisible = () => {
|
||||
if (!isActiveRef.current) return
|
||||
const playing = Boolean(isPlayingRef.current)
|
||||
const paused = document.hidden || !playing
|
||||
sendCommand('setPaused', { paused })
|
||||
if (!document.hidden) {
|
||||
sendSettings()
|
||||
}
|
||||
}
|
||||
|
||||
document.addEventListener('visibilitychange', onVisible)
|
||||
return () => document.removeEventListener('visibilitychange', onVisible)
|
||||
}, [isPlayingRef, sendCommand, sendSettings])
|
||||
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
sendCommand('transportPause', {})
|
||||
sendCommand('setPaused', { paused: true })
|
||||
if (frameRef.current !== null) {
|
||||
cancelAnimationFrame(frameRef.current)
|
||||
frameRef.current = null
|
||||
}
|
||||
}
|
||||
}, [sendCommand])
|
||||
|
||||
return {
|
||||
onIframeLoad,
|
||||
sendCommand,
|
||||
loadedDuration,
|
||||
isLoaded,
|
||||
}
|
||||
}
|
||||
|
||||
export default useMuonIframeTransport
|
||||
@@ -283,6 +283,7 @@ export default function TrackDetailPage() {
|
||||
const [backgroundMode, setBackgroundMode] = useState<BackgroundMode>(() => readBackgroundMode())
|
||||
const [previewPlaying, setPreviewPlaying] = useState(false)
|
||||
const [previewPosition, setPreviewPosition] = useState(0)
|
||||
const [previewAudioUrl, setPreviewAudioUrl] = useState('')
|
||||
|
||||
const convertStatusSongs = useMemo(() => {
|
||||
const seen = new Set<string>()
|
||||
@@ -298,6 +299,10 @@ export default function TrackDetailPage() {
|
||||
const coverArtUrl = song?.coverArt ? subsonicCoverArtUrl(song.coverArt, 800) : ''
|
||||
const heroCoverArtUrl = song?.coverArt ? subsonicCoverArtUrl(song.coverArt, 400) : ''
|
||||
const audioStreamUrl = useMemo(() => (song ? subsonicStreamUrl(song.id) : ''), [song?.id])
|
||||
const audioStreamFallbackUrl = useMemo(
|
||||
() => (song ? subsonicStreamUrl(song.id, { format: 'mp3', maxBitRate: 320 }) : ''),
|
||||
[song?.id],
|
||||
)
|
||||
const albumColor = useAlbumColor(coverArtUrl || undefined)
|
||||
const stemKeys = useMemo(() => extractStemKeys(projectDetail), [projectDetail])
|
||||
const previewStems = useMemo(
|
||||
@@ -308,6 +313,10 @@ export default function TrackDetailPage() {
|
||||
const previewEnd = useMemo(() => Math.min(song?.duration ?? 0, startOffset + 30), [song?.duration, startOffset])
|
||||
const previewLength = Math.max(0, previewEnd - startOffset)
|
||||
|
||||
useEffect(() => {
|
||||
setPreviewAudioUrl(audioStreamUrl)
|
||||
}, [audioStreamUrl])
|
||||
|
||||
useEffect(() => {
|
||||
if (!song) return
|
||||
setFavorite(readFavorites().has(song.id))
|
||||
@@ -357,7 +366,7 @@ export default function TrackDetailPage() {
|
||||
audio.currentTime = startOffset
|
||||
setPreviewPosition(startOffset)
|
||||
setPreviewPlaying(false)
|
||||
}, [audioStreamUrl, startOffset])
|
||||
}, [previewAudioUrl, startOffset])
|
||||
|
||||
useEffect(() => {
|
||||
const audio = previewAudioRef.current
|
||||
@@ -384,7 +393,7 @@ export default function TrackDetailPage() {
|
||||
audio.removeEventListener('pause', onPause)
|
||||
audio.removeEventListener('timeupdate', onTimeUpdate)
|
||||
}
|
||||
}, [audioStreamUrl, startOffset, previewLength, previewEnd])
|
||||
}, [previewAudioUrl, startOffset, previewLength, previewEnd])
|
||||
|
||||
function persistBackgroundMode(mode: BackgroundMode) {
|
||||
setBackgroundMode(mode)
|
||||
@@ -424,6 +433,13 @@ export default function TrackDetailPage() {
|
||||
await audio.play()
|
||||
setPreviewPlaying(true)
|
||||
} catch (error) {
|
||||
const message = String(error instanceof Error ? error.message : error || '')
|
||||
const unsupportedSource = /no supported sources/i.test(message)
|
||||
if (unsupportedSource && audioStreamFallbackUrl && previewAudioUrl !== audioStreamFallbackUrl) {
|
||||
setPreviewAudioUrl(audioStreamFallbackUrl)
|
||||
setPreviewPlaying(false)
|
||||
return
|
||||
}
|
||||
reportNonFatal('TrackDetail: preview.play', error, { trackId: song?.id ?? null })
|
||||
setPreviewPlaying(false)
|
||||
}
|
||||
@@ -568,10 +584,10 @@ export default function TrackDetailPage() {
|
||||
</section>
|
||||
|
||||
<section className="flex flex-col sm:flex-row gap-4 items-start">
|
||||
<div className="w-full sm:w-1/4 aspect-square shrink-0 overflow-hidden rounded-xl border border-white/10 bg-black/20">
|
||||
<div className="w-full sm:w-[280px] min-w-[220px] min-h-[220px] aspect-square shrink-0 overflow-hidden rounded-xl border border-white/10 bg-black/20">
|
||||
<VisualizerPreviewWidget
|
||||
ref={previewAudioRef}
|
||||
audioUrl={audioStreamUrl}
|
||||
audioUrl={previewAudioUrl}
|
||||
coverArtUrl={coverArtUrl || undefined}
|
||||
videoUrl={videoUrlInput.trim() || undefined}
|
||||
preferredMode={preferredVisualizerMode}
|
||||
@@ -582,7 +598,7 @@ export default function TrackDetailPage() {
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div className="flex-1 w-full space-y-4 rounded-xl border border-white/10 bg-black/20 p-4 backdrop-blur-sm">
|
||||
<div className="flex-1 min-w-0 w-full space-y-4 rounded-xl border border-white/10 bg-black/20 p-4 backdrop-blur-sm">
|
||||
<div>
|
||||
<p className="text-xs font-semibold uppercase tracking-wide text-white/50 mb-2">Visualizer mode</p>
|
||||
<div className="flex flex-wrap gap-2">
|
||||
@@ -605,7 +621,7 @@ export default function TrackDetailPage() {
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => void togglePreviewPlayback()}
|
||||
disabled={!audioStreamUrl || previewLength <= 0}
|
||||
disabled={!previewAudioUrl || previewLength <= 0}
|
||||
aria-label={previewPlaying ? 'Pause preview' : 'Play preview'}
|
||||
title={previewPlaying ? 'Pause preview' : 'Play preview'}
|
||||
className="inline-flex h-10 w-10 items-center justify-center rounded-full bg-brand-500 text-white hover:bg-brand-400 disabled:opacity-40"
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
import { useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react'
|
||||
import { useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react'
|
||||
import { Pause, Play, RotateCcw, Upload } from 'lucide-react'
|
||||
import { parseBlob } from 'music-metadata-browser'
|
||||
import VisualizerBackdrop, { type VisualizerMode, type VisualizerSettings, type VizShape } from '../components/visualizers/VisualizerBackdrop'
|
||||
import type { MuonIframePreset } from '../components/visualizers/visualizerTypes'
|
||||
import { useAudioFeatures } from '../hooks/useAudioFeatures'
|
||||
import { createAudioPushLoop } from '../components/visualizers/muonAudioBridge'
|
||||
import { useMuonIframeTransport } from '../hooks/useMuonIframeTransport'
|
||||
import { useVisualizerPalette } from '../hooks/useVisualizerPalette'
|
||||
import { getAppSettings } from '../api/client'
|
||||
|
||||
@@ -13,7 +13,7 @@ type MuonStyle = 'default' | 'fluid' | 'tendrils' | 'spirit'
|
||||
|
||||
const MODES: LabMode[] = ['muon_iframe']
|
||||
const DEFAULT_MUON_IFRAME_URL = 'http://localhost:3011'
|
||||
const MUON_IFRAME_REV = '20260422b'
|
||||
const MUON_IFRAME_REV = '20260427c'
|
||||
const SHAPES: VizShape[] = ['dynamic', 'circle', 'square', 'star', 'hexagon', 'flower']
|
||||
const QUALITIES: NonNullable<VisualizerSettings['quality']>[] = ['low', 'balanced', 'high']
|
||||
const FPS_CAPS: NonNullable<VisualizerSettings['fps_cap']>[] = [30, 60, 120]
|
||||
@@ -25,11 +25,46 @@ const MUON_PRESETS_BY_STYLE: Record<MuonStyle, MuonIframePreset[]> = {
|
||||
spirit: ['spirit_aether', 'spirit_aurora', 'spirit_lumen'],
|
||||
}
|
||||
|
||||
function isLoopbackHost(hostname: string): boolean {
|
||||
const value = hostname.trim().toLowerCase()
|
||||
return value === 'localhost' || value === '127.0.0.1' || value === '::1'
|
||||
}
|
||||
|
||||
function defaultMuonIframeUrl(): string {
|
||||
if (typeof window === 'undefined') return DEFAULT_MUON_IFRAME_URL
|
||||
const protocol = window.location.protocol === 'https:' ? 'https:' : 'http:'
|
||||
const hostname = window.location.hostname || 'localhost'
|
||||
return `${protocol}//${hostname}:3011`
|
||||
}
|
||||
|
||||
function resolveMuonIframeUrl(raw: string | undefined): string {
|
||||
const cleaned = String(raw || '').trim()
|
||||
if (!cleaned) return DEFAULT_MUON_IFRAME_URL
|
||||
if (!/^https?:\/\//i.test(cleaned)) return DEFAULT_MUON_IFRAME_URL
|
||||
return cleaned
|
||||
if (!cleaned) return defaultMuonIframeUrl()
|
||||
|
||||
if (/^https?:\/\//i.test(cleaned)) {
|
||||
try {
|
||||
const parsed = new URL(cleaned)
|
||||
if (typeof window !== 'undefined' && isLoopbackHost(parsed.hostname) && !isLoopbackHost(window.location.hostname)) {
|
||||
parsed.hostname = window.location.hostname
|
||||
}
|
||||
return parsed.toString()
|
||||
} catch {
|
||||
return defaultMuonIframeUrl()
|
||||
}
|
||||
}
|
||||
|
||||
const protocol = typeof window !== 'undefined' && window.location.protocol === 'https:' ? 'https://' : 'http://'
|
||||
const candidate = cleaned.startsWith('//') ? `${protocol}${cleaned.slice(2)}` : `${protocol}${cleaned}`
|
||||
try {
|
||||
const parsed = new URL(candidate)
|
||||
if (!/^https?:$/i.test(parsed.protocol)) return defaultMuonIframeUrl()
|
||||
if (typeof window !== 'undefined' && isLoopbackHost(parsed.hostname) && !isLoopbackHost(window.location.hostname)) {
|
||||
parsed.hostname = window.location.hostname
|
||||
}
|
||||
return parsed.toString()
|
||||
} catch {
|
||||
return defaultMuonIframeUrl()
|
||||
}
|
||||
}
|
||||
|
||||
function withDefaultPreset(rawUrl: string): string {
|
||||
@@ -83,7 +118,7 @@ export default function VisualizerLabPage() {
|
||||
const [dims, setDims] = useState<{ w: number; h: number }>({ w: 640, h: 360 })
|
||||
|
||||
const [ctxVersion, setCtxVersion] = useState(0)
|
||||
const [muonIframeUrl, setMuonIframeUrl] = useState(DEFAULT_MUON_IFRAME_URL)
|
||||
const [muonIframeUrl, setMuonIframeUrl] = useState(() => defaultMuonIframeUrl())
|
||||
const [muonIframeError, setMuonIframeError] = useState(false)
|
||||
const [muonStyle, setMuonStyle] = useState<MuonStyle>('default')
|
||||
const [muonPreset, setMuonPreset] = useState<MuonIframePreset>('default')
|
||||
@@ -91,16 +126,11 @@ export default function VisualizerLabPage() {
|
||||
const audioRef = useRef<HTMLAudioElement>(null)
|
||||
const containerRef = useRef<HTMLDivElement>(null)
|
||||
const iframeRef = useRef<HTMLIFrameElement>(null)
|
||||
const muonIframeReadyRef = useRef(false)
|
||||
const audioCtxRef = useRef<AudioContext | null>(null)
|
||||
const isPlayingRef = useRef(false)
|
||||
|
||||
const previousAudioUrlRef = useRef<string>('')
|
||||
const previousCoverUrlRef = useRef<string>('')
|
||||
const dimsRef = useRef(dims)
|
||||
const muonStyleRef = useRef(muonStyle)
|
||||
const muonPresetRef = useRef<MuonIframePreset>(muonPreset)
|
||||
const qualityRef = useRef(quality)
|
||||
|
||||
const paletteRef = useVisualizerPalette({ coverArtUrl: coverUrl || undefined })
|
||||
const featuresRef = useAudioFeatures({
|
||||
@@ -137,123 +167,28 @@ export default function VisualizerLabPage() {
|
||||
}, [])
|
||||
const muonEmbedUrl = useMemo(() => withDefaultPreset(muonIframeUrl), [muonIframeUrl])
|
||||
|
||||
useEffect(() => {
|
||||
muonIframeReadyRef.current = false
|
||||
}, [muonEmbedUrl])
|
||||
|
||||
const sendMuonCommand = useCallback((type: string, payload: Record<string, unknown>) => {
|
||||
if (!muonIframeReadyRef.current) return
|
||||
iframeRef.current?.contentWindow?.postMessage(
|
||||
{ version: 1, id: `lab-${Date.now()}`, type, payload },
|
||||
'*',
|
||||
)
|
||||
}, [])
|
||||
|
||||
const rehydrateMuonState = useCallback(() => {
|
||||
sendMuonCommand('setStyle', { style: muonStyleRef.current })
|
||||
sendMuonCommand('setPreset', { preset: muonPresetRef.current })
|
||||
sendMuonCommand('setQuality', { quality: qualityRef.current })
|
||||
sendMuonCommand('setViewport', {
|
||||
cssWidth: dimsRef.current.w,
|
||||
cssHeight: dimsRef.current.h,
|
||||
const { onIframeLoad: onMuonIframeLoad } = useMuonIframeTransport({
|
||||
iframeRef,
|
||||
isActive: mode === 'muon_iframe',
|
||||
audioUrl: audioUrl || null,
|
||||
isPlayingRef,
|
||||
getCurrentTime: () => Number(audioRef.current?.currentTime || 0),
|
||||
getRate: () => Number(audioRef.current?.playbackRate || 1),
|
||||
style: muonStyle,
|
||||
preset: muonPreset,
|
||||
quality,
|
||||
viewport: {
|
||||
cssWidth: dims.w,
|
||||
cssHeight: dims.h,
|
||||
dpr: typeof window === 'undefined' ? 1 : window.devicePixelRatio || 1,
|
||||
})
|
||||
sendMuonCommand('setPaused', {
|
||||
paused: typeof document !== 'undefined' ? document.hidden || !isPlayingRef.current : !isPlayingRef.current,
|
||||
})
|
||||
}, [sendMuonCommand])
|
||||
},
|
||||
commandIdPrefix: 'lab',
|
||||
})
|
||||
|
||||
useEffect(() => {
|
||||
isPlayingRef.current = isPlaying
|
||||
}, [isPlaying])
|
||||
|
||||
useEffect(() => {
|
||||
dimsRef.current = dims
|
||||
}, [dims])
|
||||
|
||||
useEffect(() => {
|
||||
muonStyleRef.current = muonStyle
|
||||
}, [muonStyle])
|
||||
|
||||
useEffect(() => {
|
||||
muonPresetRef.current = muonPreset
|
||||
}, [muonPreset])
|
||||
|
||||
useEffect(() => {
|
||||
qualityRef.current = quality
|
||||
}, [quality])
|
||||
|
||||
useEffect(() => {
|
||||
if (mode !== 'muon_iframe') return
|
||||
sendMuonCommand('setStyle', { style: muonStyle })
|
||||
}, [mode, muonStyle, sendMuonCommand])
|
||||
|
||||
useEffect(() => {
|
||||
if (mode !== 'muon_iframe') return
|
||||
sendMuonCommand('setPreset', { preset: muonPreset })
|
||||
}, [mode, muonPreset, sendMuonCommand])
|
||||
|
||||
useEffect(() => {
|
||||
if (mode !== 'muon_iframe') return
|
||||
sendMuonCommand('setQuality', { quality })
|
||||
}, [mode, quality, sendMuonCommand])
|
||||
|
||||
useEffect(() => {
|
||||
if (mode !== 'muon_iframe') return
|
||||
sendMuonCommand('setViewport', {
|
||||
cssWidth: dims.w,
|
||||
cssHeight: dims.h,
|
||||
dpr: typeof window === 'undefined' ? 1 : window.devicePixelRatio || 1,
|
||||
})
|
||||
}, [mode, dims, sendMuonCommand])
|
||||
|
||||
useEffect(() => {
|
||||
if (mode !== 'muon_iframe') return
|
||||
sendMuonCommand('setPaused', { paused: typeof document !== 'undefined' ? document.hidden || !isPlayingRef.current : !isPlayingRef.current })
|
||||
return () => sendMuonCommand('setPaused', { paused: true })
|
||||
}, [mode, sendMuonCommand])
|
||||
|
||||
useEffect(() => {
|
||||
if (mode !== 'muon_iframe') return
|
||||
sendMuonCommand('setPaused', { paused: typeof document !== 'undefined' ? document.hidden || !isPlaying : !isPlaying })
|
||||
}, [isPlaying, mode, sendMuonCommand])
|
||||
|
||||
// Push audio features to Muon iframe each RAF while playing.
|
||||
useEffect(() => {
|
||||
if (mode !== 'muon_iframe' || !isPlaying) return
|
||||
const bridge = createAudioPushLoop(sendMuonCommand, () => featuresRef.current)
|
||||
bridge.start()
|
||||
return () => bridge.stop()
|
||||
// featuresRef is a stable mutable ref — read inside RAF loop, not a restart trigger
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [mode, isPlaying, sendMuonCommand])
|
||||
|
||||
useEffect(() => {
|
||||
const onMessage = (event: MessageEvent) => {
|
||||
if (event.source !== iframeRef.current?.contentWindow) return
|
||||
const msg = event.data as { type?: string } | null
|
||||
if (!msg || typeof msg !== 'object') return
|
||||
if (msg.type === 'ready') {
|
||||
muonIframeReadyRef.current = true
|
||||
rehydrateMuonState()
|
||||
}
|
||||
}
|
||||
|
||||
window.addEventListener('message', onMessage)
|
||||
if (mode === 'muon_iframe') rehydrateMuonState()
|
||||
return () => window.removeEventListener('message', onMessage)
|
||||
}, [mode, rehydrateMuonState])
|
||||
|
||||
useEffect(() => {
|
||||
const onVisible = () => {
|
||||
if (mode !== 'muon_iframe') return
|
||||
sendMuonCommand('setPaused', { paused: document.hidden || !isPlayingRef.current })
|
||||
if (!document.hidden) rehydrateMuonState()
|
||||
}
|
||||
document.addEventListener('visibilitychange', onVisible)
|
||||
return () => document.removeEventListener('visibilitychange', onVisible)
|
||||
}, [mode, rehydrateMuonState, sendMuonCommand])
|
||||
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
if (previousAudioUrlRef.current) URL.revokeObjectURL(previousAudioUrlRef.current)
|
||||
@@ -273,7 +208,7 @@ export default function VisualizerLabPage() {
|
||||
})
|
||||
.catch(() => {
|
||||
if (cancelled) return
|
||||
setMuonIframeUrl(DEFAULT_MUON_IFRAME_URL)
|
||||
setMuonIframeUrl(defaultMuonIframeUrl())
|
||||
})
|
||||
|
||||
return () => {
|
||||
@@ -417,17 +352,17 @@ export default function VisualizerLabPage() {
|
||||
{mode === 'muon_iframe' ? (
|
||||
<div className="relative" style={{ height: dims.h }}>
|
||||
<iframe
|
||||
key={muonEmbedUrl}
|
||||
ref={iframeRef}
|
||||
src={muonEmbedUrl}
|
||||
title="Muon music visualizer"
|
||||
className="absolute inset-0 h-full w-full border-0"
|
||||
loading="lazy"
|
||||
loading="eager"
|
||||
referrerPolicy="no-referrer"
|
||||
allow="autoplay"
|
||||
sandbox="allow-scripts allow-same-origin"
|
||||
onLoad={() => {
|
||||
setMuonIframeError(false)
|
||||
rehydrateMuonState()
|
||||
onMuonIframeLoad()
|
||||
}}
|
||||
onError={() => setMuonIframeError(true)}
|
||||
/>
|
||||
|
||||
@@ -27,7 +27,7 @@ export default defineConfig(({ mode }) => {
|
||||
},
|
||||
},
|
||||
server: {
|
||||
allowedHosts: ['localhost', '127.0.0.1', 'work'],
|
||||
allowedHosts: true,
|
||||
hmr: {
|
||||
// No `host` here — Vite falls back to importMetaUrl.hostname, which is the
|
||||
// hostname the browser actually loaded the page from. This makes HMR work
|
||||
|
||||
+24
-10
@@ -1,10 +1,6 @@
|
||||
FROM node:20-alpine AS builder
|
||||
|
||||
ARG MUON_GIT_REF=main
|
||||
# Origin that the Muon iframe will accept postMessage commands from and emit
|
||||
# ready/error events back to. Must match the StemEngine frontend origin.
|
||||
# Examples: http://localhost:5173 (dev), https://stemengine.example.com (prod)
|
||||
ARG STEMENGINE_ALLOWED_ORIGIN=http://localhost:5173
|
||||
|
||||
WORKDIR /src
|
||||
|
||||
@@ -14,6 +10,8 @@ RUN apk add --no-cache git \
|
||||
COPY overlayless.css /tmp/overlayless.css
|
||||
COPY stemengine-bootstrap.js /tmp/stemengine-bootstrap.js
|
||||
COPY stemengine-router.js /tmp/stemengine-router.js
|
||||
COPY stemengine-audio-engine.js /tmp/stemengine-audio-engine.js
|
||||
COPY stemengine-feature-extractor.js /tmp/stemengine-feature-extractor.js
|
||||
COPY fluid-render-graph.js /tmp/fluid-render-graph.js
|
||||
COPY tendrils-render-graph.js /tmp/tendrils-render-graph.js
|
||||
COPY spirit-render-graph.js /tmp/spirit-render-graph.js
|
||||
@@ -24,10 +22,15 @@ RUN cat /tmp/overlayless.css >> /src/app/src/style.css
|
||||
# Inject bootstrap (WebGL baseline checks + context lifecycle hooks)
|
||||
RUN cp /tmp/stemengine-bootstrap.js /src/app/src/stemengine-bootstrap.js
|
||||
|
||||
# Inject router (style router + postMessage command listener + ready/error emitter)
|
||||
# Replace the placeholder origin token with the build-arg value.
|
||||
RUN cp /tmp/stemengine-router.js /src/app/src/stemengine-router.js \
|
||||
&& sed -i "s|__STEMENGINE_ALLOWED_ORIGIN__|${STEMENGINE_ALLOWED_ORIGIN}|g" /src/app/src/stemengine-router.js
|
||||
# Inject router (style router + postMessage command listener + ready/error emitter).
|
||||
# D-8 fix: the STEMENGINE_ALLOWED_ORIGIN sed replacement is dropped — isAllowedOrigin()
|
||||
# is unconditionally true (project policy: zero access control on this path). The build
|
||||
# arg, .env.example entry, and GHA workflow input are also vestigial; remove there too.
|
||||
RUN cp /tmp/stemengine-router.js /src/app/src/stemengine-router.js
|
||||
|
||||
# Inject iframe-local audio engine + feature extractor for transport-driven analysis.
|
||||
RUN cp /tmp/stemengine-audio-engine.js /src/app/src/stemengine-audio-engine.js
|
||||
RUN cp /tmp/stemengine-feature-extractor.js /src/app/src/stemengine-feature-extractor.js
|
||||
|
||||
# Inject fluid render graph (Phase 2: Fluid rendering)
|
||||
RUN cp /tmp/fluid-render-graph.js /src/app/src/fluid-render-graph.js
|
||||
@@ -38,8 +41,19 @@ RUN cp /tmp/tendrils-render-graph.js /src/app/src/tendrils-render-graph.js
|
||||
# Inject spirit render graph (Phase 4)
|
||||
RUN cp /tmp/spirit-render-graph.js /src/app/src/spirit-render-graph.js
|
||||
|
||||
# Inject imports in deterministic order (bootstrap first, then router, then render graphs).
|
||||
RUN sed -i '1i import "./stemengine-bootstrap";\nimport "./stemengine-router";\nimport "./fluid-render-graph";\nimport "./tendrils-render-graph";\nimport "./spirit-render-graph";' /src/app/src/index.js
|
||||
# Inject imports in deterministic order (bootstrap, audio pipeline, router, then render graphs).
|
||||
RUN sed -i '1i import "./stemengine-bootstrap";\nimport "./stemengine-audio-engine";\nimport "./stemengine-feature-extractor";\nimport "./stemengine-router";\nimport "./fluid-render-graph";\nimport "./tendrils-render-graph";\nimport "./spirit-render-graph";' /src/app/src/index.js
|
||||
|
||||
# Default-mode integration: allow original Muon analyser reads to use StemEngine analyser.
|
||||
RUN sed -i 's@analyser = wavesurfer.backend.analyser@analyser = (window.__STEMENGINE_MUON_ANALYSER__ || wavesurfer.backend.analyser)@' /src/app/src/index.js
|
||||
|
||||
# Register default Muon STOP/RESUME hooks so custom render graphs can pause/resume
|
||||
# the original RAF loop and avoid dual-renderer canvas conflicts.
|
||||
#
|
||||
# D-7 follow-up: register STOP/RESUME at module scope without touching animate
|
||||
# initialization order. Referencing animate before declaration can trigger a TDZ
|
||||
# runtime crash in production bundles. Resume calls animate lazily via closure.
|
||||
RUN sed -i 's@let wavesurfer,@window.__STEMENGINE_MUON_STOP_FN__ = () => { pauseAnimation = true }; window.__STEMENGINE_MUON_RESUME_FN__ = () => { if (!pauseAnimation) return; const pauseButton = document.getElementById("control-animation"); if (pauseButton \&\& typeof pauseButton.click === "function") { pauseButton.click(); return; } pauseAnimation = false; }; let wavesurfer,@' /src/app/src/index.js
|
||||
|
||||
RUN npm install
|
||||
RUN npm run build
|
||||
|
||||
@@ -66,8 +66,9 @@ Planned behavior in the Muon fork:
|
||||
|
||||
## postMessage command contract (Phase 1)
|
||||
|
||||
The iframe accepts versioned commands via `postMessage` from `STEMENGINE_ALLOWED_ORIGIN` only.
|
||||
Messages from any other origin are silently ignored.
|
||||
The iframe accepts versioned commands via `postMessage` from any origin. Origin restrictions
|
||||
are intentionally disabled; see `docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/README.md`
|
||||
section 8 for the project's zero-access-control rationale.
|
||||
|
||||
### Message envelope
|
||||
|
||||
@@ -127,7 +128,5 @@ Rate-limiting:
|
||||
gate so the render loop is never blocked by message bursts.
|
||||
- Pushes where all feature deltas are below a threshold may be skipped in a future optimization.
|
||||
|
||||
`STEMENGINE_ALLOWED_ORIGIN` build arg:
|
||||
- Set in compose `stemengine.yml` via `STEMENGINE_ALLOWED_ORIGIN` env var (defaults to
|
||||
`http://localhost:5173` for development).
|
||||
- Must match the StemEngine frontend origin exactly (scheme + host + port).
|
||||
Origin restrictions: none. The previous `STEMENGINE_ALLOWED_ORIGIN` build arg, env var, and
|
||||
sed substitution have been removed; the iframe accepts postMessage commands from any origin.
|
||||
@@ -34,6 +34,24 @@
|
||||
var SPLAT_RADIUS_BASE = 15
|
||||
var SPLAT_RADIUS_MAX = 50
|
||||
|
||||
function clamp01(v) {
|
||||
var n = Number(v)
|
||||
if (!isFinite(n)) n = 0
|
||||
return Math.max(0, Math.min(1, n))
|
||||
}
|
||||
|
||||
// Match Muon's snappier feel by accenting peaks while keeping low-end detail.
|
||||
function shaped(v, power) {
|
||||
var c = clamp01(v)
|
||||
return Math.pow(c, power)
|
||||
}
|
||||
|
||||
function getBand(audio, key, fallback) {
|
||||
var bands = audio && audio.frequencyBands ? audio.frequencyBands : null
|
||||
var candidate = bands && isFinite(Number(bands[key])) ? Number(bands[key]) : fallback
|
||||
return clamp01(candidate)
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// UTILITY: WebGL Extensions & Capabilities
|
||||
// =========================================================================
|
||||
@@ -862,10 +880,15 @@
|
||||
if (!prog) return
|
||||
|
||||
var audio = audioFeatures || {}
|
||||
var sub = getBand(audio, 'sub', audio.sub || 0)
|
||||
var bass = getBand(audio, 'bass', audio.bass || 0)
|
||||
var highMid = getBand(audio, 'highMid', audio.treble || 0)
|
||||
var brilliance = getBand(audio, 'brilliance', audio.treble || 0)
|
||||
var rms = clamp01(audio.rms || 0)
|
||||
var audioColor = [
|
||||
audio.bass || 0,
|
||||
audio.rms || 0,
|
||||
audio.treble || 0,
|
||||
shaped(0.55 * bass + 0.45 * sub, 0.62),
|
||||
shaped(rms, 0.75),
|
||||
shaped(0.6 * highMid + 0.4 * brilliance, 0.65),
|
||||
]
|
||||
|
||||
gl.useProgram(prog)
|
||||
@@ -954,7 +977,9 @@
|
||||
if (!prog) return
|
||||
|
||||
var audio = audioFeatures || {}
|
||||
var bloomIntensity = 0.5 + (audio.rms || 0) * 1.5
|
||||
var rms = clamp01(audio.rms || 0)
|
||||
var sparkle = getBand(audio, 'brilliance', audio.treble || 0)
|
||||
var bloomIntensity = 0.45 + shaped(rms, 0.8) * 1.35 + shaped(sparkle, 0.65) * 0.45
|
||||
|
||||
gl.useProgram(prog)
|
||||
gl.bindFramebuffer(gl.FRAMEBUFFER, this.dye.fbo)
|
||||
@@ -1014,13 +1039,20 @@
|
||||
|
||||
var gl = this.gl
|
||||
|
||||
// Clamped audio values
|
||||
var bass = Math.max(0, Math.min(1, audioFeatures?.bass || 0))
|
||||
// Richer band-aware signal shaping for better dynamic range.
|
||||
var bassBand = getBand(audioFeatures, 'bass', audioFeatures?.bass || 0)
|
||||
var subBand = getBand(audioFeatures, 'sub', audioFeatures?.sub || 0)
|
||||
var lowMidBand = getBand(audioFeatures, 'lowMid', audioFeatures?.bass || 0)
|
||||
var highMidBand = getBand(audioFeatures, 'highMid', audioFeatures?.treble || 0)
|
||||
var brillianceBand = getBand(audioFeatures, 'brilliance', audioFeatures?.treble || 0)
|
||||
|
||||
var bass = shaped(0.55 * bassBand + 0.45 * subBand, 0.62)
|
||||
var beat = Boolean(audioFeatures?.beat)
|
||||
var treble = Math.max(0, Math.min(1, audioFeatures?.treble || 0))
|
||||
var rms = Math.max(0, Math.min(1, audioFeatures?.rms || 0))
|
||||
var beatPhase = Math.max(0, Math.min(1, audioFeatures?.beatPhase || 0))
|
||||
var drumIntensity = Math.max(0, Math.min(1, audioFeatures?.drumIntensity != null ? audioFeatures.drumIntensity : bass))
|
||||
var treble = shaped(0.45 * highMidBand + 0.55 * brillianceBand, 0.68)
|
||||
var rms = shaped(clamp01(audioFeatures?.rms || 0), 0.78)
|
||||
var beatPhase = clamp01(audioFeatures?.beatPhase || 0)
|
||||
var drumInput = audioFeatures?.drumIntensity != null ? Number(audioFeatures.drumIntensity) : (0.7 * bassBand + 0.3 * lowMidBand)
|
||||
var drumIntensity = shaped(clamp01(drumInput), 0.72)
|
||||
|
||||
try {
|
||||
// --- PIPELINE ---
|
||||
|
||||
@@ -13,6 +13,16 @@
|
||||
return Math.max(0, Math.min(1, n))
|
||||
}
|
||||
|
||||
function shaped(v, power) {
|
||||
return Math.pow(clamp01(v), power)
|
||||
}
|
||||
|
||||
function getBand(audio, key, fallback) {
|
||||
var bands = audio && audio.frequencyBands ? audio.frequencyBands : null
|
||||
var value = bands && isFinite(Number(bands[key])) ? Number(bands[key]) : fallback
|
||||
return clamp01(value)
|
||||
}
|
||||
|
||||
function compileProgram(gl, vsSrc, fsSrc) {
|
||||
function c(type, src) {
|
||||
var s = gl.createShader(type)
|
||||
@@ -179,6 +189,12 @@
|
||||
var gl = this.gl
|
||||
var t = performance.now() * 0.001
|
||||
var audio = window.__STEMENGINE_MUON_AUDIO__ || {}
|
||||
var bass = shaped(0.5 * getBand(audio, 'sub', audio.sub || 0) + 0.5 * getBand(audio, 'bass', audio.bass || 0), 0.64)
|
||||
var treble = shaped(0.55 * getBand(audio, 'highMid', audio.treble || 0) + 0.45 * getBand(audio, 'brilliance', audio.treble || 0), 0.66)
|
||||
var rms = shaped(clamp01(audio.rms || 0), 0.8)
|
||||
var beat = audio.beat ? 1 : 0
|
||||
var buildBoost = audio.isBuildUp ? 0.02 : 0
|
||||
var dropBoost = audio.isBeatDrop ? 0.02 : 0
|
||||
|
||||
gl.bindFramebuffer(gl.FRAMEBUFFER, this.write.fbo)
|
||||
gl.viewport(0, 0, this.write.width, this.write.height)
|
||||
@@ -188,11 +204,14 @@
|
||||
gl.uniform1i(gl.getUniformLocation(this.updateProgram, 'uPrev'), 0)
|
||||
gl.uniform2f(gl.getUniformLocation(this.updateProgram, 'uRes'), this.write.width, this.write.height)
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uTime'), t)
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uBass'), clamp01(audio.bass))
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uTreble'), clamp01(audio.treble))
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uRms'), clamp01(audio.rms))
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uBeat'), audio.beat ? 1 : 0)
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uFade'), this.quality === 'high' ? 0.965 : this.quality === 'low' ? 0.92 : 0.945)
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uBass'), bass)
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uTreble'), treble)
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uRms'), rms)
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uBeat'), beat)
|
||||
gl.uniform1f(
|
||||
gl.getUniformLocation(this.updateProgram, 'uFade'),
|
||||
(this.quality === 'high' ? 0.965 : this.quality === 'low' ? 0.92 : 0.945) - buildBoost + dropBoost,
|
||||
)
|
||||
this.draw(this.updateProgram)
|
||||
|
||||
gl.bindFramebuffer(gl.FRAMEBUFFER, null)
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
/* StemEngine iframe-local audio engine for Muon modes. */
|
||||
;(function stemengineAudioEngine() {
|
||||
'use strict'
|
||||
|
||||
var AudioCtx = window.AudioContext || window.webkitAudioContext
|
||||
var ctx = null
|
||||
var analyser = null
|
||||
var gain = null
|
||||
var sourceNode = null
|
||||
var buffer = null
|
||||
var startedAt = 0
|
||||
var pausedAt = 0
|
||||
var isPlaying = false
|
||||
var playbackRate = 1
|
||||
|
||||
function clampTime(value, duration) {
|
||||
if (!isFinite(value)) return 0
|
||||
if (!isFinite(duration) || duration <= 0) return Math.max(0, value)
|
||||
return Math.max(0, Math.min(value, duration))
|
||||
}
|
||||
|
||||
function disconnectSource() {
|
||||
if (!sourceNode) return
|
||||
try { sourceNode.onended = null } catch (_) {}
|
||||
try { sourceNode.stop() } catch (_) {}
|
||||
try { sourceNode.disconnect() } catch (_) {}
|
||||
sourceNode = null
|
||||
}
|
||||
|
||||
function initialize() {
|
||||
if (ctx) return
|
||||
if (!AudioCtx) throw new Error('AudioContext unavailable')
|
||||
|
||||
ctx = new AudioCtx({ sampleRate: 44100 })
|
||||
analyser = ctx.createAnalyser()
|
||||
analyser.fftSize = 512
|
||||
analyser.smoothingTimeConstant = 0.8
|
||||
analyser.minDecibels = -100
|
||||
analyser.maxDecibels = -30
|
||||
|
||||
gain = ctx.createGain()
|
||||
gain.gain.value = 0
|
||||
|
||||
// Keep Muon iframe audio silent for users, but feed full-strength signal to analyser.
|
||||
// source -> analyser (reactive features)
|
||||
// source -> gain(0) -> destination (muted output)
|
||||
gain.connect(ctx.destination)
|
||||
|
||||
window.__STEMENGINE_MUON_ANALYSER__ = analyser
|
||||
window.__STEMENGINE_MUON_AUDIO_CTX__ = ctx
|
||||
}
|
||||
|
||||
function stop() {
|
||||
disconnectSource()
|
||||
isPlaying = false
|
||||
startedAt = 0
|
||||
pausedAt = 0
|
||||
}
|
||||
|
||||
async function loadFromUrl(url) {
|
||||
initialize()
|
||||
if (!url) throw new Error('missing audio URL')
|
||||
|
||||
stop()
|
||||
|
||||
var response = await fetch(url, { mode: 'cors', credentials: 'omit' })
|
||||
if (!response.ok) {
|
||||
throw new Error('audio fetch failed: ' + response.status)
|
||||
}
|
||||
|
||||
var arr = await response.arrayBuffer()
|
||||
buffer = await ctx.decodeAudioData(arr)
|
||||
pausedAt = 0
|
||||
return { duration: buffer.duration }
|
||||
}
|
||||
|
||||
function play(opts) {
|
||||
initialize()
|
||||
if (!buffer || !gain) return
|
||||
|
||||
var options = opts || {}
|
||||
if (ctx.state === 'suspended') {
|
||||
ctx.resume().catch(function () {})
|
||||
}
|
||||
|
||||
disconnectSource()
|
||||
|
||||
sourceNode = ctx.createBufferSource()
|
||||
sourceNode.buffer = buffer
|
||||
playbackRate = isFinite(Number(options.rate)) ? Math.max(0.1, Number(options.rate)) : playbackRate
|
||||
sourceNode.playbackRate.value = playbackRate
|
||||
sourceNode.connect(analyser)
|
||||
sourceNode.connect(gain)
|
||||
|
||||
var offset = typeof options.time === 'number' ? options.time : pausedAt
|
||||
var startOffset = clampTime(offset, buffer.duration)
|
||||
startedAt = ctx.currentTime - startOffset / playbackRate
|
||||
pausedAt = 0
|
||||
isPlaying = true
|
||||
|
||||
sourceNode.onended = function () {
|
||||
if (!sourceNode) return
|
||||
var now = getCurrentTime()
|
||||
if (buffer && now >= buffer.duration - 0.01) {
|
||||
isPlaying = false
|
||||
pausedAt = 0
|
||||
startedAt = 0
|
||||
disconnectSource()
|
||||
}
|
||||
}
|
||||
|
||||
sourceNode.start(0, startOffset)
|
||||
}
|
||||
|
||||
function pause() {
|
||||
if (!ctx || !isPlaying) return
|
||||
pausedAt = clampTime(getCurrentTime(), buffer ? buffer.duration : Infinity)
|
||||
isPlaying = false
|
||||
disconnectSource()
|
||||
}
|
||||
|
||||
function seek(time) {
|
||||
var target = clampTime(Number(time) || 0, buffer ? buffer.duration : Infinity)
|
||||
var wasPlaying = isPlaying
|
||||
if (wasPlaying) {
|
||||
pause()
|
||||
pausedAt = target
|
||||
play({ time: target, rate: playbackRate })
|
||||
return
|
||||
}
|
||||
pausedAt = target
|
||||
}
|
||||
|
||||
function setRate(rate) {
|
||||
var next = Number(rate)
|
||||
if (!isFinite(next) || next <= 0) return
|
||||
playbackRate = Math.max(0.1, Math.min(next, 4))
|
||||
if (sourceNode) {
|
||||
sourceNode.playbackRate.value = playbackRate
|
||||
}
|
||||
}
|
||||
|
||||
function getCurrentTime() {
|
||||
if (!ctx) return 0
|
||||
if (!isPlaying) return pausedAt
|
||||
var elapsed = (ctx.currentTime - startedAt) * playbackRate
|
||||
return clampTime(elapsed, buffer ? buffer.duration : Infinity)
|
||||
}
|
||||
|
||||
function getDuration() {
|
||||
return buffer ? buffer.duration : 0
|
||||
}
|
||||
|
||||
window.__StemEngineAudioEngine__ = {
|
||||
initialize: initialize,
|
||||
loadFromUrl: loadFromUrl,
|
||||
play: play,
|
||||
pause: pause,
|
||||
stop: stop,
|
||||
seek: seek,
|
||||
setRate: setRate,
|
||||
isPlaying: function () { return isPlaying },
|
||||
getRate: function () { return playbackRate },
|
||||
getAnalyser: function () { return analyser },
|
||||
getContext: function () { return ctx },
|
||||
getCurrentTime: getCurrentTime,
|
||||
getDuration: getDuration,
|
||||
}
|
||||
|
||||
// D-1 fix: initialize synchronously at module load.
|
||||
// The original Muon's `vizInit()` is invoked synchronously by `window.onload = vizInit()`
|
||||
// (a Muon-author bug — the `()` invokes immediately, not on the load event). Inside vizInit,
|
||||
// `audioAnalyzer()` reads `window.__STEMENGINE_MUON_ANALYSER__` to bind the default-mode
|
||||
// analyser. If we defer initialize() until DOMContentLoaded, that binding has already
|
||||
// happened against wavesurfer's silent analyser and never re-acquires.
|
||||
// Initializing here ensures __STEMENGINE_MUON_ANALYSER__ is set BEFORE vizInit runs
|
||||
// (we are imported before the original Muon code in index.js).
|
||||
// AudioContext creation without a user gesture is allowed; it starts in 'suspended' state
|
||||
// and resume() succeeds on transportPlay because allow="autoplay" inherits parent activation.
|
||||
try { initialize() } catch (e) { /* baseline check covers WebGL absence; AudioContext absence is rare */ }
|
||||
})()
|
||||
@@ -0,0 +1,331 @@
|
||||
/* StemEngine iframe-local feature extractor for Muon render graphs. */
|
||||
;(function stemengineFeatureExtractor() {
|
||||
'use strict'
|
||||
|
||||
var ENERGY_HISTORY_SIZE = 43
|
||||
var BEAT_TIME_HISTORY_SIZE = 24
|
||||
var BUILDUP_WINDOW = 40
|
||||
var DROP_SENSITIVITY = 2.5
|
||||
var SILENCE_FLOOR = 0.008
|
||||
// D-3 fix: per-band peak tracker with slow decay, mirrors host's useAudioFeatures.ts:287-350.
|
||||
// Raw band magnitudes from getFloatFrequencyData() are tiny (0.005-0.05); without this
|
||||
// normalization, render-graph shader uniforms see effectively-zero values and the
|
||||
// visualizer doesn't visibly react. PEAK_DECAY=0.9998 matches the host's value so behavior
|
||||
// is consistent across both pipelines.
|
||||
var PEAK_DECAY = 0.9998
|
||||
var PEAK_FLOOR = 0.001
|
||||
|
||||
var rafId = null
|
||||
var running = false
|
||||
var energyHistory = []
|
||||
var beatTimes = []
|
||||
var lastBeatTime = 0
|
||||
var previousEnergy = 0
|
||||
var bpm = 0
|
||||
var beatThreshold = 1.5
|
||||
var magnitudeData = null
|
||||
var bandPeaks = {
|
||||
sub: PEAK_FLOOR,
|
||||
bass: PEAK_FLOOR,
|
||||
lowMid: PEAK_FLOOR,
|
||||
mid: PEAK_FLOOR,
|
||||
highMid: PEAK_FLOOR,
|
||||
presence: PEAK_FLOOR,
|
||||
brilliance: PEAK_FLOOR,
|
||||
}
|
||||
// Energy peak tracker — same idea as bands, applied to broadband energy.
|
||||
var energyPeak = PEAK_FLOOR
|
||||
var rmsPeak = PEAK_FLOOR
|
||||
|
||||
var FREQUENCY_BANDS = {
|
||||
sub: [20, 60],
|
||||
bass: [60, 250],
|
||||
lowMid: [250, 500],
|
||||
mid: [500, 2000],
|
||||
highMid: [2000, 4000],
|
||||
presence: [4000, 6000],
|
||||
brilliance: [6000, 20000],
|
||||
}
|
||||
|
||||
function clamp01(value) {
|
||||
var n = Number(value)
|
||||
if (!isFinite(n)) return 0
|
||||
return Math.max(0, Math.min(1, n))
|
||||
}
|
||||
|
||||
function calculateAverageAndVariance(values) {
|
||||
if (!values.length) return { average: 0, variance: 0 }
|
||||
var sum = 0
|
||||
var i = 0
|
||||
for (i = 0; i < values.length; i += 1) sum += values[i]
|
||||
var average = sum / values.length
|
||||
var varianceSum = 0
|
||||
for (i = 0; i < values.length; i += 1) {
|
||||
var d = values[i] - average
|
||||
varianceSum += d * d
|
||||
}
|
||||
return { average: average, variance: varianceSum / values.length }
|
||||
}
|
||||
|
||||
function estimateBpm() {
|
||||
if (beatTimes.length < 2) return bpm
|
||||
var intervals = []
|
||||
for (var i = 1; i < beatTimes.length; i += 1) {
|
||||
var dt = beatTimes[i] - beatTimes[i - 1]
|
||||
if (dt > 0.2 && dt < 1.5) intervals.push(dt)
|
||||
}
|
||||
if (!intervals.length) return bpm
|
||||
var sum = 0
|
||||
for (var j = 0; j < intervals.length; j += 1) sum += intervals[j]
|
||||
var avg = sum / intervals.length
|
||||
var computed = 60 / avg
|
||||
return Math.max(40, Math.min(220, computed))
|
||||
}
|
||||
|
||||
function extractFrequencyBands(sampleRate, magnitude) {
|
||||
var out = {
|
||||
sub: 0,
|
||||
bass: 0,
|
||||
lowMid: 0,
|
||||
mid: 0,
|
||||
highMid: 0,
|
||||
presence: 0,
|
||||
brilliance: 0,
|
||||
}
|
||||
|
||||
var nyquist = sampleRate / 2
|
||||
var binCount = magnitude.length || 1
|
||||
var hzPerBin = nyquist / binCount
|
||||
var keys = ['sub', 'bass', 'lowMid', 'mid', 'highMid', 'presence', 'brilliance']
|
||||
|
||||
for (var k = 0; k < keys.length; k += 1) {
|
||||
var key = keys[k]
|
||||
var lowHz = FREQUENCY_BANDS[key][0]
|
||||
var highHz = FREQUENCY_BANDS[key][1]
|
||||
var start = Math.max(0, Math.floor(lowHz / hzPerBin))
|
||||
var end = Math.min(binCount, Math.ceil(highHz / hzPerBin))
|
||||
var span = Math.max(1, end - start)
|
||||
var sumSquares = 0
|
||||
for (var i = start; i < end; i += 1) {
|
||||
var m = magnitude[i]
|
||||
sumSquares += m * m
|
||||
}
|
||||
var raw = Math.sqrt(sumSquares / span)
|
||||
// D-3 fix: divide by a slowly-decaying peak so the output is in a perceptually
|
||||
// useful 0..1 range. Without this, render-graph shaders see ~0.04 for a peak bass hit.
|
||||
bandPeaks[key] = Math.max(bandPeaks[key] * PEAK_DECAY, raw, PEAK_FLOOR)
|
||||
out[key] = clamp01(raw / bandPeaks[key])
|
||||
}
|
||||
|
||||
return out
|
||||
}
|
||||
|
||||
function detectBeat(energy) {
|
||||
var now = performance.now() / 1000
|
||||
|
||||
energyHistory.push(energy)
|
||||
if (energyHistory.length > ENERGY_HISTORY_SIZE) energyHistory.shift()
|
||||
|
||||
var stats = calculateAverageAndVariance(energyHistory)
|
||||
var average = stats.average
|
||||
var stdDev = Math.sqrt(Math.max(stats.variance, 1e-9))
|
||||
var zScore = stdDev > 0 ? (energy - average) / stdDev : 0
|
||||
|
||||
var minInterval = 0.18
|
||||
var isCandidate = energy > average * beatThreshold && zScore > 0.35
|
||||
var isBeat = isCandidate && (now - lastBeatTime) > minInterval
|
||||
|
||||
if (isBeat) {
|
||||
lastBeatTime = now
|
||||
beatTimes.push(now)
|
||||
if (beatTimes.length > BEAT_TIME_HISTORY_SIZE) beatTimes.shift()
|
||||
bpm = estimateBpm()
|
||||
}
|
||||
|
||||
var beatPeriod = bpm > 0 ? 60 / bpm : 0
|
||||
var timeSinceLastBeat = lastBeatTime > 0 ? now - lastBeatTime : 0
|
||||
var beatPhase = beatPeriod > 0 ? (timeSinceLastBeat % beatPeriod) / beatPeriod : 0
|
||||
|
||||
var confidence = clamp01((zScore / 3) * 0.6 + ((energy / Math.max(average, 1e-6)) - 1) * 0.4)
|
||||
|
||||
return {
|
||||
isBeat: isBeat,
|
||||
bpm: bpm,
|
||||
confidence: confidence,
|
||||
beatPhase: clamp01(beatPhase),
|
||||
timeSinceLastBeat: timeSinceLastBeat,
|
||||
}
|
||||
}
|
||||
|
||||
function detectPatterns(energy, isBeat) {
|
||||
var energyDelta = energy - previousEnergy
|
||||
previousEnergy = energy
|
||||
|
||||
var tail = energyHistory.slice(-BUILDUP_WINDOW)
|
||||
var avg = energy
|
||||
if (tail.length) {
|
||||
var sum = 0
|
||||
for (var i = 0; i < tail.length; i += 1) sum += tail[i]
|
||||
avg = sum / tail.length
|
||||
}
|
||||
|
||||
var isSilence = energy < SILENCE_FLOOR
|
||||
var isBuildUp = !isSilence && energyDelta > 0.01 && energy > avg * 1.02
|
||||
var isBeatDrop = isBeat && !isSilence && energy > avg * DROP_SENSITIVITY
|
||||
|
||||
return {
|
||||
isBeatDrop: isBeatDrop,
|
||||
isBuildUp: isBuildUp,
|
||||
isSilence: isSilence,
|
||||
intensity: clamp01(energy * 8),
|
||||
energyDelta: energyDelta,
|
||||
}
|
||||
}
|
||||
|
||||
function extractFrame(analyser, ctx, frequencyData, timeData) {
|
||||
analyser.getFloatFrequencyData(frequencyData)
|
||||
analyser.getFloatTimeDomainData(timeData)
|
||||
|
||||
if (!magnitudeData || magnitudeData.length !== frequencyData.length) {
|
||||
magnitudeData = new Float32Array(frequencyData.length)
|
||||
}
|
||||
|
||||
var bins = frequencyData.length || 1
|
||||
var nyquist = ctx.sampleRate / 2
|
||||
|
||||
var powerSum = 0
|
||||
var magSum = 0
|
||||
var weightedBinSum = 0
|
||||
var totalEnergy = 0
|
||||
|
||||
for (var i = 0; i < frequencyData.length; i += 1) {
|
||||
var db = isFinite(frequencyData[i]) ? frequencyData[i] : -100
|
||||
var mag = Math.pow(10, db / 20)
|
||||
magnitudeData[i] = mag
|
||||
|
||||
var power = mag * mag
|
||||
powerSum += power
|
||||
magSum += mag
|
||||
weightedBinSum += i * mag
|
||||
totalEnergy += mag
|
||||
}
|
||||
|
||||
var rawRms = Math.sqrt(powerSum / bins)
|
||||
var rawEnergy = totalEnergy / bins
|
||||
// D-3 fix: peak-normalize broadband energy and RMS too (same approach as bands).
|
||||
rmsPeak = Math.max(rmsPeak * PEAK_DECAY, rawRms, PEAK_FLOOR)
|
||||
energyPeak = Math.max(energyPeak * PEAK_DECAY, rawEnergy, PEAK_FLOOR)
|
||||
var rms = clamp01(rawRms / rmsPeak)
|
||||
var energy = clamp01(rawEnergy / energyPeak)
|
||||
var centroidBin = magSum > 0 ? weightedBinSum / magSum : 0
|
||||
var spectralCentroidHz = (centroidBin / bins) * nyquist
|
||||
var spectralCentroid = clamp01(spectralCentroidHz / nyquist)
|
||||
|
||||
var geoLogSum = 0
|
||||
for (var j = 0; j < magnitudeData.length; j += 1) {
|
||||
geoLogSum += Math.log(Math.max(1e-12, magnitudeData[j]))
|
||||
}
|
||||
var geometricMean = Math.exp(geoLogSum / bins)
|
||||
var arithmeticMean = magSum / bins
|
||||
var spectralFlatness = arithmeticMean > 0 ? geometricMean / arithmeticMean : 0
|
||||
|
||||
var zc = 0
|
||||
for (var t = 1; t < timeData.length; t += 1) {
|
||||
var prev = timeData[t - 1]
|
||||
var cur = timeData[t]
|
||||
if ((prev >= 0 && cur < 0) || (prev < 0 && cur >= 0)) zc += 1
|
||||
}
|
||||
var zcr = timeData.length > 1 ? zc / (timeData.length - 1) : 0
|
||||
|
||||
var frequencyBands = extractFrequencyBands(ctx.sampleRate, magnitudeData)
|
||||
// detectBeat uses RAW (un-normalized) energy so the peak detector compares apples to apples
|
||||
// across recent history. Using normalized 0..1 energy would defeat the variance check.
|
||||
var beatInfo = detectBeat(rawEnergy)
|
||||
var patternInfo = detectPatterns(rawEnergy, beatInfo.isBeat)
|
||||
|
||||
// Treble = mean of (presence, brilliance) — matches host's useAudioFeatures convention.
|
||||
// Previously used (highMid + presence + brilliance)/3 which spilled mid-range into treble.
|
||||
window.__STEMENGINE_MUON_AUDIO__ = {
|
||||
bass: frequencyBands.bass,
|
||||
treble: clamp01((frequencyBands.presence + frequencyBands.brilliance) / 2),
|
||||
rms: rms,
|
||||
energy: energy,
|
||||
beat: beatInfo.isBeat,
|
||||
drumIntensity: clamp01(frequencyBands.sub * 0.7 + frequencyBands.bass * 0.3),
|
||||
sub: frequencyBands.sub,
|
||||
beatPhase: beatInfo.beatPhase,
|
||||
bpm: beatInfo.bpm,
|
||||
beatConfidence: beatInfo.confidence,
|
||||
isBeatDrop: patternInfo.isBeatDrop,
|
||||
isBuildUp: patternInfo.isBuildUp,
|
||||
isSilence: patternInfo.isSilence,
|
||||
energyDelta: patternInfo.energyDelta,
|
||||
spectralCentroid: spectralCentroid,
|
||||
spectralFlatness: clamp01(spectralFlatness),
|
||||
zcr: clamp01(zcr),
|
||||
frequencyBands: frequencyBands,
|
||||
fft: frequencyData,
|
||||
waveform: timeData,
|
||||
__seUpdatedAt: Date.now(),
|
||||
}
|
||||
}
|
||||
|
||||
function stop() {
|
||||
running = false
|
||||
if (rafId !== null) {
|
||||
cancelAnimationFrame(rafId)
|
||||
rafId = null
|
||||
}
|
||||
}
|
||||
|
||||
function start() {
|
||||
var engine = window.__StemEngineAudioEngine__
|
||||
if (!engine || typeof engine.getAnalyser !== 'function' || typeof engine.getContext !== 'function') return
|
||||
|
||||
var analyser = engine.getAnalyser()
|
||||
var ctx = engine.getContext()
|
||||
if (!analyser || !ctx) return
|
||||
|
||||
stop()
|
||||
running = true
|
||||
|
||||
var frequencyData = new Float32Array(analyser.frequencyBinCount)
|
||||
var timeData = new Float32Array(analyser.fftSize)
|
||||
|
||||
function loop() {
|
||||
if (!running) return
|
||||
extractFrame(analyser, ctx, frequencyData, timeData)
|
||||
rafId = requestAnimationFrame(loop)
|
||||
}
|
||||
|
||||
rafId = requestAnimationFrame(loop)
|
||||
}
|
||||
|
||||
function reset() {
|
||||
energyHistory = []
|
||||
beatTimes = []
|
||||
lastBeatTime = 0
|
||||
previousEnergy = 0
|
||||
bpm = 0
|
||||
rmsPeak = PEAK_FLOOR
|
||||
energyPeak = PEAK_FLOOR
|
||||
bandPeaks.sub = PEAK_FLOOR
|
||||
bandPeaks.bass = PEAK_FLOOR
|
||||
bandPeaks.lowMid = PEAK_FLOOR
|
||||
bandPeaks.mid = PEAK_FLOOR
|
||||
bandPeaks.highMid = PEAK_FLOOR
|
||||
bandPeaks.presence = PEAK_FLOOR
|
||||
bandPeaks.brilliance = PEAK_FLOOR
|
||||
}
|
||||
|
||||
window.__StemEngineFeatureExtractor__ = {
|
||||
start: start,
|
||||
stop: stop,
|
||||
reset: reset,
|
||||
setBeatThreshold: function (threshold) {
|
||||
var t = Number(threshold)
|
||||
if (!isFinite(t)) return
|
||||
beatThreshold = Math.max(0, Math.min(3, t))
|
||||
},
|
||||
}
|
||||
})()
|
||||
@@ -1,22 +1,11 @@
|
||||
/* StemEngine — style router + postMessage command listener.
|
||||
* Injected into the Muon build at image build-time via Dockerfile.
|
||||
*
|
||||
* The string __STEMENGINE_ALLOWED_ORIGIN__ is replaced by sed during the
|
||||
* Docker build with the value of the STEMENGINE_ALLOWED_ORIGIN build arg
|
||||
* (e.g. "http://stemengine.example.com"). Never leave it as the literal
|
||||
* placeholder in production.
|
||||
* Project policy: zero access control restrictions on this path. Origin validation
|
||||
* is intentionally a no-op. See docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/
|
||||
* README.md section 8 for the rationale.
|
||||
*/
|
||||
;(function stemengineRouter() {
|
||||
var ALLOWED_ORIGIN_RAW = '__STEMENGINE_ALLOWED_ORIGIN__'
|
||||
var PLACEHOLDER_TOKEN = '__STEMENGINE' + '_ALLOWED_ORIGIN__'
|
||||
var ALLOWED_ORIGINS = String(ALLOWED_ORIGIN_RAW || '')
|
||||
.split(',')
|
||||
.map(function (item) { return item.trim() })
|
||||
.filter(Boolean)
|
||||
var ACCEPT_ANY_ORIGIN = ALLOWED_ORIGINS.indexOf('*') >= 0
|
||||
var HAS_ALLOWED_ORIGINS = ALLOWED_ORIGINS.length > 0
|
||||
|
||||
|
||||
function isAllowedOrigin(_origin) {
|
||||
return true
|
||||
}
|
||||
@@ -53,6 +42,11 @@
|
||||
setPaused: true,
|
||||
setViewport: true,
|
||||
pushAudioFeatures: true,
|
||||
loadAudio: true,
|
||||
transportPlay: true,
|
||||
transportPause: true,
|
||||
transportSeek: true,
|
||||
transportRate: true,
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
@@ -212,6 +206,70 @@
|
||||
})
|
||||
}
|
||||
break
|
||||
case 'loadAudio':
|
||||
if (!window.__StemEngineAudioEngine__ || typeof window.__StemEngineAudioEngine__.loadFromUrl !== 'function') break
|
||||
window.__StemEngineAudioEngine__.loadFromUrl(String(p.url || ''))
|
||||
.then(function (info) {
|
||||
if (window.__StemEngineFeatureExtractor__ && typeof window.__StemEngineFeatureExtractor__.start === 'function') {
|
||||
window.__StemEngineFeatureExtractor__.start()
|
||||
}
|
||||
postToParent({
|
||||
version: 1,
|
||||
id: 'loaded-' + Date.now(),
|
||||
type: 'loaded',
|
||||
payload: {
|
||||
duration: info && Number.isFinite(Number(info.duration)) ? Number(info.duration) : 0,
|
||||
},
|
||||
})
|
||||
})
|
||||
.catch(function (err) {
|
||||
window.__stemengineEmitError__('AUDIO_LOAD', String(err && err.message ? err.message : err))
|
||||
})
|
||||
break
|
||||
case 'transportPlay':
|
||||
if (window.__StemEngineAudioEngine__ && typeof window.__StemEngineAudioEngine__.play === 'function') {
|
||||
window.__StemEngineAudioEngine__.play({
|
||||
time: Number(p.time) || 0,
|
||||
rate: Number(p.rate) || 1,
|
||||
})
|
||||
}
|
||||
break
|
||||
case 'transportPause':
|
||||
if (window.__StemEngineAudioEngine__ && typeof window.__StemEngineAudioEngine__.pause === 'function') {
|
||||
window.__StemEngineAudioEngine__.pause()
|
||||
}
|
||||
break
|
||||
case 'transportSeek':
|
||||
if (window.__StemEngineAudioEngine__ && typeof window.__StemEngineAudioEngine__.seek === 'function') {
|
||||
var nextTime = Number(p.time) || 0
|
||||
window.__StemEngineAudioEngine__.seek(nextTime)
|
||||
|
||||
// Recovery path: if host is actively playing but a transportPlay command was
|
||||
// dropped/raced, seek updates keep arriving. Auto-resume once from seek.
|
||||
if (
|
||||
!window.__STEMENGINE_MUON_PAUSED__
|
||||
&& typeof window.__StemEngineAudioEngine__.isPlaying === 'function'
|
||||
&& !window.__StemEngineAudioEngine__.isPlaying()
|
||||
&& typeof window.__StemEngineAudioEngine__.play === 'function'
|
||||
) {
|
||||
var resumeRate = Number(p.rate)
|
||||
if (!Number.isFinite(resumeRate) || resumeRate <= 0) {
|
||||
resumeRate = typeof window.__StemEngineAudioEngine__.getRate === 'function'
|
||||
? Number(window.__StemEngineAudioEngine__.getRate()) || 1
|
||||
: 1
|
||||
}
|
||||
window.__StemEngineAudioEngine__.play({
|
||||
time: nextTime,
|
||||
rate: resumeRate,
|
||||
})
|
||||
}
|
||||
}
|
||||
break
|
||||
case 'transportRate':
|
||||
if (window.__StemEngineAudioEngine__ && typeof window.__StemEngineAudioEngine__.setRate === 'function') {
|
||||
window.__StemEngineAudioEngine__.setRate(Number(p.rate) || 1)
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
@@ -243,7 +301,7 @@
|
||||
capabilities: {
|
||||
webgl: Boolean(baseline.ok),
|
||||
missingExtensions: baseline.missing || [],
|
||||
audioMode: 'host-push',
|
||||
audioMode: 'hybrid',
|
||||
},
|
||||
styles: ['fluid', 'tendrils', 'spirit', 'default'],
|
||||
}
|
||||
@@ -282,6 +340,9 @@
|
||||
window.__STEMENGINE_MUON_VIEWPORT__ = null
|
||||
|
||||
function onReady() {
|
||||
if (window.__StemEngineAudioEngine__ && typeof window.__StemEngineAudioEngine__.initialize === 'function') {
|
||||
window.__StemEngineAudioEngine__.initialize()
|
||||
}
|
||||
emitReady()
|
||||
// Instantiate the initial render graph based on the style query param.
|
||||
updateRenderGraph(window.__STEMENGINE_MUON_STYLE__)
|
||||
|
||||
@@ -13,6 +13,16 @@
|
||||
return Math.max(0, Math.min(1, n))
|
||||
}
|
||||
|
||||
function shaped(v, power) {
|
||||
return Math.pow(clamp01(v), power)
|
||||
}
|
||||
|
||||
function getBand(audio, key, fallback) {
|
||||
var bands = audio && audio.frequencyBands ? audio.frequencyBands : null
|
||||
var value = bands && isFinite(Number(bands[key])) ? Number(bands[key]) : fallback
|
||||
return clamp01(value)
|
||||
}
|
||||
|
||||
function createProgram(gl, vsSrc, fsSrc) {
|
||||
function compile(type, src) {
|
||||
var s = gl.createShader(type)
|
||||
@@ -171,7 +181,12 @@
|
||||
var gl = this.gl
|
||||
var t = performance.now() * 0.001
|
||||
var audio = window.__STEMENGINE_MUON_AUDIO__ || {}
|
||||
var bass = shaped(0.6 * getBand(audio, 'bass', audio.bass || 0) + 0.4 * getBand(audio, 'sub', audio.sub || 0), 0.62)
|
||||
var treble = shaped(0.55 * getBand(audio, 'presence', audio.treble || 0) + 0.45 * getBand(audio, 'brilliance', audio.treble || 0), 0.66)
|
||||
var rms = shaped(clamp01(audio.rms || 0), 0.8)
|
||||
var beat = audio.beat ? 1 : 0
|
||||
var buildBoost = audio.isBuildUp ? 0.12 : 0
|
||||
var dropBoost = audio.isBeatDrop ? 0.2 : 0
|
||||
|
||||
gl.bindFramebuffer(gl.FRAMEBUFFER, this.write.fbo)
|
||||
gl.viewport(0, 0, this.write.width, this.write.height)
|
||||
@@ -181,11 +196,14 @@
|
||||
gl.uniform1i(gl.getUniformLocation(this.updateProgram, 'uPrev'), 0)
|
||||
gl.uniform2f(gl.getUniformLocation(this.updateProgram, 'uRes'), this.write.width, this.write.height)
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uTime'), t)
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uBass'), clamp01(audio.bass))
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uTreble'), clamp01(audio.treble))
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uRms'), clamp01(audio.rms))
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uBass'), bass)
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uTreble'), treble)
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uRms'), rms)
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uBeat'), beat)
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uTrailDecay'), 0.90 + (this.quality === 'high' ? 0.08 : this.quality === 'low' ? 0.04 : 0.06))
|
||||
gl.uniform1f(
|
||||
gl.getUniformLocation(this.updateProgram, 'uTrailDecay'),
|
||||
(0.90 + (this.quality === 'high' ? 0.08 : this.quality === 'low' ? 0.04 : 0.06)) - buildBoost + dropBoost,
|
||||
)
|
||||
this.drawQuad(this.updateProgram)
|
||||
|
||||
gl.bindFramebuffer(gl.FRAMEBUFFER, null)
|
||||
|
||||
Reference in new issue
Block a user