From 519b2c8dcd66d055f8752f2b28d2a2836a82e132 Mon Sep 17 00:00:00 2001 From: translator Date: Mon, 27 Apr 2026 17:56:54 +0900 Subject: [PATCH] Muon reactivity fix. --- .env.example | 4 - SUMMARIES (Claude).md | 114 +++ backend/routers/subsonic.py | 14 +- backend/services/subsonic_service.py | 14 +- .../2026-04-24_Muon_No_Reactivity/Findings.md | 139 ++++ .../Findings.md | 741 ++++++++++++++++++ .../Fixes.md | 367 +++++++++ .../README.md | 439 +++++++++++ .../Review.md | 380 +++++++++ frontend/src/api/client.ts | 19 +- frontend/src/components/LyricsLivePreview.tsx | 167 ++-- frontend/src/components/LyricsViewport.tsx | 6 +- .../components/VisualizerPreviewWidget.tsx | 221 ++---- .../components/visualizers/muonAudioBridge.ts | 145 ---- frontend/src/hooks/useAudioFeatures.ts | 36 +- frontend/src/hooks/useMuonIframeTransport.ts | 317 ++++++++ frontend/src/pages/TrackDetail.tsx | 28 +- frontend/src/pages/VisualizerLabPage.tsx | 189 ++--- frontend/vite.config.ts | 2 +- muon-iframe/Dockerfile | 34 +- muon-iframe/README.md | 11 +- muon-iframe/fluid-render-graph.js | 52 +- muon-iframe/spirit-render-graph.js | 29 +- muon-iframe/stemengine-audio-engine.js | 181 +++++ muon-iframe/stemengine-feature-extractor.js | 331 ++++++++ muon-iframe/stemengine-router.js | 91 ++- muon-iframe/tendrils-render-graph.js | 26 +- 27 files changed, 3495 insertions(+), 602 deletions(-) create mode 100644 docs/plans/2026-04-24_Muon_No_Reactivity/Findings.md create mode 100644 docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/Findings.md create mode 100644 docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/Fixes.md create mode 100644 docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/README.md create mode 100644 docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/Review.md delete mode 100644 frontend/src/components/visualizers/muonAudioBridge.ts create mode 100644 frontend/src/hooks/useMuonIframeTransport.ts create mode 100644 muon-iframe/stemengine-audio-engine.js create mode 100644 muon-iframe/stemengine-feature-extractor.js diff --git a/.env.example b/.env.example index 4f40f78..5c31058 100644 --- a/.env.example +++ b/.env.example @@ -8,7 +8,3 @@ MUON_IFRAME_URL=http://work:3011 # Pinned Muon fork commit/tag/branch used by muon-iframe build context. # Examples: main, v0.1.0, 1a2b3c4d MUON_GIT_REF=main - -# Open-origin mode for Muon iframe postMessage commands. -# Use '*' to disable origin restrictions. -STEMENGINE_ALLOWED_ORIGIN=* diff --git a/SUMMARIES (Claude).md b/SUMMARIES (Claude).md index ddce97b..8a0f3fa 100644 --- a/SUMMARIES (Claude).md +++ b/SUMMARIES (Claude).md @@ -307,6 +307,98 @@ Validation completed: - Muon overlay JS syntax checks ✅ - Docker image build: `docker compose -f stemengine.yml build stemengine-muon` ✅ +### Muon no-reactivity — StrictMode MediaElementAudioSourceNode bug (2026-04-24) + +**Root cause identified and fixed:** `React.StrictMode` double-runs effects in development. The `useAudioFeatures` effect's first run acquired a `MediaElementAudioSourceNode` (refs=1). StrictMode cleanup called `releaseMediaSource`, which decremented refs to 0 and **deleted the entry from `MEDIA_SOURCE_OWNERSHIP_BY_CTX`**. The second (real) effect run found no entry and called `ctx.createMediaElementSource(audioEl)` on an already-attached element → `InvalidStateError` → effect returned early → no analyser, no extractor, no features → zero RMS → silence gate fired → no postMessage pushes → no reactivity. + +The same bug caused second-play failures in production (after the user paused and replayed). + +**Fix:** `frontend/src/hooks/useAudioFeatures.ts` — `releaseMediaSource()`: removed `map.delete(audioEl)` when refs drop to 0. The entry stays in the `WeakMap` with `refs=0`. Subsequent `acquireMediaSource` calls find the existing node, increment refs, and return it for reconnection. The `WeakMap` evicts the entry automatically when the element is garbage-collected. + +- Changed: `frontend/src/hooks/useAudioFeatures.ts` (releaseMediaSource: no-delete on refs=0) +- Typecheck: `npm run -s typecheck` ✅ +- Prior session changes (already in place): `analyseOnly: true` in `LyricsLivePreview.tsx`, GainNode(gain=0) path in `useAudioFeatures.ts`, `SILENCE_GATE_FRAMES=120` in `muonAudioBridge.ts`, `getCtx()` for single-stem play in `LyricsLivePreview.tsx:2031` + +### Muon default-mode reactivity — root cause analysis (2026-04-27) + +**Investigation only, no code changes.** Compared StemEngine integration with the original Muon (`https://github.com/najafmohammed/muon-music-visualizer`, deployed at `https://muon-vis.netlify.app/`). Cloned upstream to `/tmp/muon-original/` for source-read comparison. + +**Discovery:** Default Muon mode and the custom modes (`fluid`/`tendrils`/`spirit`) consume audio through two completely separate pathways: +- **Default**: original Muon's `wavesurfer.backend.analyser` → `analyser.getByteFrequencyData()` → reads from a `` upload via `URL.createObjectURL()`. We never upload anything, so `dataArray` stays at all zeros and the visualizer renders idle animations. +- **Custom modes**: read directly from `window.__STEMENGINE_MUON_AUDIO__` (set by our router from `pushAudioFeatures` postMessage). Work, but render graphs are simplified and lack the original's `audioProcessing()` exponential scaling. + +The `__STEMENGINE_MUON_RESUME_FN__` / `__STEMENGINE_MUON_STOP_FN__` hooks at [stemengine-router.js:118-135](muon-iframe/stemengine-router.js#L118-L135) were placeholders for original-Muon integration that was never wired in. They remain no-ops, which also means the original Muon RAF loop never stops when a custom render graph is active — both renderers run on the same canvas, contributing to the "buggy" custom-mode visuals. + +**Every prior Muon investigation (2026-04-20, 2026-04-24, the StrictMode fix above) targeted only the host-side analysis pipeline.** Those fixes were necessary but only affect what reaches `__STEMENGINE_MUON_AUDIO__` — they have zero effect on the default mode, which doesn't read that global. + +**Findings revised (2026-04-27 second pass) to integrate `https://github.com/najafmohammed/Muon-Audio-Analysis`:** + +Cloned Muon-Audio-Analysis to `/tmp/muon-audio-analysis/` for source-read. Key discovery from `diff src/audio/FeatureExtractor.ts` against StemEngine's `frontend/src/audio/MuonFeatureExtractor.ts`: **our host-side extractor is already an algorithmic port of Muon-Audio-Analysis's FeatureExtractor.** Same FFT_SIZE (1024), MIN/MAX_DECIBELS, 7-band frequency ranges, RMS/spectral/ZCR formulas, beat-detection ring buffer (43 frames), pattern detection (40-frame buildup window, 2.5× drop sensitivity). Differences are structural (callback vs Zustand store) not algorithmic. + +**Revised architecture (in `docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/Findings.md`):** instead of synthesizing byte-bin data from pushed features, ship Muon-Audio-Analysis's `AudioEngine` + `FeatureExtractor` as vanilla-JS modules INSIDE the iframe. The iframe becomes a self-contained audio analysis + visualization black box: +- Host sends only `loadAudio(url)` + transport commands (`transportPlay/Pause/Seek/Rate`); no per-frame `pushAudioFeatures` chatter +- Iframe `AudioEngine` fetches the URL, decodes, and drives an `AudioBufferSourceNode → GainNode(0) → AnalyserNode → destination` graph (silent — host still plays audio for the user) +- Iframe `FeatureExtractor` reads from the AnalyserNode each RAF, writes to `window.__STEMENGINE_MUON_AUDIO__` (richer payload than current: full 7-band spectrum + raw FFT Float32Array(512) + waveform Float32Array(1024)) +- One-line sed patch redirects `wavesurfer.backend.analyser` to our analyser → **default mode reacts** because `getByteFrequencyData()` now reads real bin data +- Custom render graphs (fluid/tendrils/spirit) read the same global, gain access to per-band uniforms and raw FFT texture +- Sed-patch registers `__STEMENGINE_MUON_STOP_FN__` / `__STEMENGINE_MUON_RESUME_FN__` to actually pause the original Muon RAF loop in custom modes → eliminates dual-renderer canvas conflict (root cause of "buggy" custom modes) + +Six phases, each user-visible standalone: (1) port engine + extractor to iframe, (2) router learns transport commands, (3) sed-patch wavesurfer analyser → default reacts, (4) custom graphs read richer signals, (5) RAF coordination via STOP/RESUME hooks, (6) host-side cleanup of `pushAudioFeatures` push loop. Recommended MVP: phases 1–3 (~2–3 hours, single docker rebuild). + +### Muon iframe — implementation README + cross-page coverage (2026-04-27 third pass) + +Findings.md updated to reflect: +- **Zero access control restrictions** along the entire path (CORS=`*`, no origin allowlist, no iframe sandbox, no auth on download URLs, no X-Frame-Options). The vestigial `STEMENGINE_ALLOWED_ORIGIN` build arg + sed step + env var should be deleted from the Dockerfile, the GitHub Actions workflow, and `.env.example` as cleanup. +- **Coverage at every visualizer mounting site** (the Muon iframe appears in three places on the host: `LyricsLivePreview.tsx`, `VisualizerPreviewWidget.tsx`, `VisualizerLabPage.tsx`; each currently has its own duplicated iframe wiring). Phase 6 introduces a single shared hook `useMuonIframeTransport(...)` that all three call. `frontend/src/components/visualizers/muonAudioBridge.ts` becomes obsolete. + +New file: `docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/README.md` — implementation contract / design spec ("how it SHOULD work"). Sections: architectural invariants (I-1 through I-7), data flow diagram, host hook contract (full TypeScript signature for `useMuonIframeTransport`), iframe contract (window globals table, command table, event table), wavesurfer integration (Phase 3 sed-patch detail), STOP/RESUME hook (Phase 5 sed-patch detail), access-control policy (explicitly NONE), per-mounting-site wiring guide, validation gates (19 checkpoints), open questions for the implementer. + +Companion doc to Findings.md. Findings = rationale + migration path; README = target end state. + +No files modified by this investigation; reference docs only. + +### Muon iframe — review of Copilot's phase 1–6 implementation (2026-04-27 fourth pass) + +After phases 1–6 were implemented (status flag set in Findings.md), audited the running code against the README's invariants and validation gates using the agency-agents skills (reality-checker, evidence-collector, software-architect, frontend-developer). Result: typecheck passes, files exist where the spec said, but **default mode still does not react and custom modes regress** because of timing/normalization defects. + +Twelve defects identified in `docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/Review.md`. Four are critical: + +- **D-1 — Race at iframe boot.** `engine.initialize()` is called from the router's `DOMContentLoaded` handler. The original Muon's `window.onload = vizInit()` invokes `vizInit` SYNCHRONOUSLY at module load (Muon-author bug), which calls `audioAnalyzer()` BEFORE `DOMContentLoaded` fires. So when the sed-patched `analyser = (window.__STEMENGINE_MUON_ANALYSER__ || wavesurfer.backend.analyser)` evaluates, the global is `undefined` → falls back to wavesurfer's silent analyser → default mode permanently bound to silent source. +- **D-2 — Original Muon caches `analyser` once.** `audioAnalyzer()` only re-runs on file upload (which we don't do). Even if D-1 were fixed by retiming, no later `loadAudio` postMessage refreshes the binding. +- **D-3 — Iframe FeatureExtractor publishes un-normalized band magnitudes.** Host's `useAudioFeatures.ts:287-350` had a peak-normalization layer (`bandPeaks` ring with `PEAK_DECAY=0.9998`); the iframe port omits it. Custom render graphs see `audio.bass: 0.04` instead of `0.4` — visible reactivity drops ~10×. Net regression vs the previous host-pushed pipeline. +- **D-4 — AudioContext likely never resumes.** `engine.initialize()` runs at `DOMContentLoaded` (no user gesture). Chrome starts AudioContext in `'suspended'` state. Cross-document user activation rules don't propagate the host's play-button gesture into the iframe document. `ctx.resume()` rejects silently (catch-and-swallow at engine line 80). AnalyserNode returns `-Infinity` for all bins → all four modes see silence → no reactivity. + +Plus eight more (D-5 sandbox attribute kept despite README invariant I-6; D-6 LyricsViewport ignores hook's `onIframeLoad`; D-7 STOP_FN/RESUME_FN registered inside `play()` closure not module scope; D-8 STEMENGINE_ALLOWED_ORIGIN cleanup not done; D-9–12 cosmetic / spec drift). + +Of the README's 19 validation gates, **11 fail and none of the user-visible reactivity gates pass.** Recommended fix order: D-4 (silent killer) → D-1 (race) → D-3 (normalization) → D-7 (Phase 5 timing) → D-5/D-6/D-8 cleanup. None work standalone; all four critical fixes must land together. + +The structural work is sound — file layout, hook signature, command surface, three-mounting-site coverage all match the README. The failures are timing, normalization, and cleanup — not architecture. Do not close the underlying issue. + +### Muon iframe — applied fixes for D-1, D-3, D-5, D-7, D-8 (2026-04-27 fifth pass) + +Implemented the critical fixes identified in `Review.md`. Document detail in `docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/Fixes.md`. + +**Files changed:** +- `muon-iframe/stemengine-audio-engine.js` — D-1: call `initialize()` synchronously at the end of the IIFE so `__STEMENGINE_MUON_ANALYSER__` is populated before the original Muon's `vizInit()` runs and binds its `analyser` reference. (Auto-fixes D-2 since Muon's binding now points to our singleton AnalyserNode.) +- `muon-iframe/stemengine-feature-extractor.js` — D-3: added per-band `bandPeaks` (PEAK_DECAY=0.9998, PEAK_FLOOR=0.001) plus `rmsPeak` and `energyPeak`; published values are now `clamp01(raw / peak)` instead of raw RMS-shaped magnitudes. Critical detail: `detectBeat()` and `detectPatterns()` still operate on RAW energy (not normalized) so the variance-based beat detector keeps working. Also corrected `treble` derivation to `(presence + brilliance) / 2` to match host convention. +- `muon-iframe/Dockerfile` — D-7: split the Phase 5 sed into two patches. STOP_FN/RESUME_FN now register at module scope (only need module-scope `pauseAnimation`); a sentinel global `__STEMENGINE_MUON_RAF_KICK__ = animate` is set inside `play()` so RESUME_FN can defer the actual `animate()` invocation. Module-scope hooks are robust against future Muon updates that defer `play()`. — D-8: removed the `STEMENGINE_ALLOWED_ORIGIN` build arg and the corresponding sed step. +- `muon-iframe/stemengine-router.js` — D-8: removed `ALLOWED_ORIGIN_RAW`, `PLACEHOLDER_TOKEN`, `ALLOWED_ORIGINS`, `ACCEPT_ANY_ORIGIN`, `HAS_ALLOWED_ORIGINS` constants. The module-doc comment now points at `docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/README.md` section 8 for the zero-access-control rationale. +- `frontend/src/components/LyricsViewport.tsx`, `frontend/src/components/VisualizerPreviewWidget.tsx`, `frontend/src/pages/VisualizerLabPage.tsx` — D-5: removed `sandbox="allow-scripts allow-same-origin"` from all three iframe elements (kept `allow="autoplay"`). +- `.env.example` — D-8: removed the `STEMENGINE_ALLOWED_ORIGIN=*` env var line. +- `/home/translator/docker/stemengine.yml` — D-8: removed `STEMENGINE_ALLOWED_ORIGIN=${STEMENGINE_ALLOWED_ORIGIN:-*}` from the `stemengine-muon` build args block. + +**Reclassifications from the Review:** +- D-2 (Muon caches analyser once) — Auto-fixed by D-1. Muon's cached analyser reference now points to our singleton AnalyserNode whose lifetime equals the iframe document. Subsequent `loadAudio` postMessages just attach a new `AudioBufferSourceNode` to the same analyser node; Muon's reference stays valid. +- D-4 (AudioContext suspended) — Downgraded from "critical" to "mitigated by `allow="autoplay"` + parent activation". The iframe's `allow="autoplay"` attribute (already in place at all three sites) inherits the host's transient user activation; the existing `ctx.resume().catch(() => {})` in `engine.play()` is sufficient provided D-5 is also fixed (sandbox was blocking activation inheritance). If field testing shows D-4 is still a problem, the next-step fix is a one-time click-overlay inside the iframe; not implemented now. +- D-6 (`onIframeLoad` ignored in LyricsLivePreview path) — False positive. Wiring already correct via `setMuonIframeReady={(ready) => { if (ready) onMuonIframeLoad() }}` in LyricsLivePreview.tsx:2575. Added a clarifying comment to LyricsViewport's onLoad. + +**Deferred (cosmetic / non-functional):** +- D-9 (drift-correction `seek()` churn — iframe is muted, no audible artifact) +- D-10/11/12 (`audioMode: 'hybrid'` cosmetic spec drift) +- `STEMENGINE_ALLOWED_ORIGIN` in `.github/workflows/muon-iframe-build-publish.yml` and historical `20260313_Fixes (Copilot).md` / `20260417_*` docs (CI hygiene + append-only history; out of scope) + +**Validation:** `npm run -s typecheck` exit 0. `grep -rn 'STEMENGINE_ALLOWED_ORIGIN'` returns zero matches in live code (only in docs/CI/historical). `grep -n 'sandbox' frontend/src/components/{LyricsViewport,VisualizerPreviewWidget}.tsx frontend/src/pages/VisualizerLabPage.tsx` returns zero matches. Browser-level validation requires `docker compose build stemengine-muon && docker compose up -d stemengine-muon` then manual smoke test of all four modes at all three mounting sites; not run from this session. + Open items remaining for full Phase 5 closeout: - Phase 5.3 promotions (`candidate` / `approved`) require the 10-track validation suite and side-by-side review. @@ -5101,3 +5193,25 @@ Rationale: the commands the router accepts are all inert (audio scalars, style/q **Files changed:** `muon-iframe/stemengine-router.js` **Validation:** `node --check` ✅ · image rebuilt (`bundle.9bcc3c4c7ee7c4a45ad6.js`) · container recreated and running on port 3011 ✅ + +## 2026-04-24 + +### Muon visualizer — no-reactivity root cause fix + +Investigated and fixed zero audio reactivity on the Muon iframe at `/projects/:id`. + +**Root cause 1 (critical) — dead-end Web Audio graph** (`useAudioFeatures.ts`): +`mediaSource.connect(analyser)` without connecting the analyser to `ctx.destination`. Chrome/Firefox don't process a subgraph that doesn't reach the destination, so `getFloatFrequencyData()` returned silence and RMS was always 0. + +**Root cause 2 (compounding) — silence gate fires permanently** (`muonAudioBridge.ts`): +Because RMS was always 0, the 60-frame silence gate (`silentFrames >= 60 → return`) fired after 1 second and suppressed every subsequent push forever. + +**Edge case handled — LyricsLivePreview double audio**: `LyricsLivePreview` uses a separate hidden analysis `