Myriad changes.
This commit is contained in:
1 parent
1bb503f735
commit
82b9b1e762
54 files changed
+7563
-2144
No files matched your search
Submodule
+1
Submodule .tmp/upstreams/keeffeoghan.github.io added at 482db7d312.
@@ -1,5 +1,42 @@
|
||||
# 2026-03-13 Backend Fixes / Improvement Plan (Copilot)
|
||||
|
||||
## 2026-04-28 Work Log — Spirit upstream iframe cutover (option B)
|
||||
|
||||
Implemented Spirit as standalone upstream iframe provider with synthetic coupling adapter.
|
||||
|
||||
Delivered in this pass:
|
||||
|
||||
- Added new runtime under `visualizer-iframes/spirit/`:
|
||||
- `Dockerfile` (upstream clone `edankwan/The-Spirit`, pinned by `SPIRIT_GIT_REF`)
|
||||
- `index.html` with provider capability declaration (`providerId: spirit`, presets/params/palette support)
|
||||
- `nginx.conf`
|
||||
- `stemengine-spirit-prehook.js` (runtime material/uniform discovery hook)
|
||||
- `stemengine-spirit-adapter.js` (synthetic audio coupling + aggressive beat response + palette propagation + fallback policy)
|
||||
|
||||
- Added compose service:
|
||||
- `docker/stemengine.yml`
|
||||
- new `stemengine-spirit` service on `${SPIRIT_HOST_PORT:-3014}`
|
||||
- build arg `SPIRIT_GIT_REF=${SPIRIT_GIT_REF:-c2ed239}`
|
||||
|
||||
Behavior implemented:
|
||||
|
||||
- Audio coupling source: `window.__STEMENGINE_VIZ_AUDIO__`
|
||||
- Coupling profile: aggressive beat envelope + audio-intensity mapping into Spirit control uniforms
|
||||
- Palette support: `setPalette` propagation to Spirit color uniforms / background atmosphere
|
||||
- Fallback policy (updated): if uniform-driver coupling cannot be discovered at runtime, switch to synthetic input-driver coupling (audio-driven virtual pointer path) and emit warning telemetry (`SPIRIT_COUPLING_FALLBACK`); passive upstream mode is retained only as a last-resort safety mode.
|
||||
|
||||
Validation:
|
||||
|
||||
- `docker compose -p stemengine -f stemengine.yml build stemengine-spirit` ✅
|
||||
- `docker compose ... up -d --no-deps --force-recreate stemengine-spirit` ✅
|
||||
- `docker compose ... ps stemengine-spirit` shows healthy container on `3014->80` ✅
|
||||
- `curl -fsS http://localhost:3014/ | grep -n "STEMENGINE_VIZ_PROVIDER_ID\|stemengine-spirit-prehook\|stemengine-spirit-adapter\|stemengine-router"` confirms provider declaration + script injection ✅
|
||||
|
||||
Runtime note:
|
||||
|
||||
- In the agent-browser harness, coupling target discovery still did not lock to a detectable Spirit uniform driver.
|
||||
- Updated adapter behavior now automatically promotes to input-driver coupling mode in this case (`window.__STEMENGINE_SPIRIT_COUPLING__.mode = "input-driver"`), preserving active audio reactivity without requiring uniform binding.
|
||||
|
||||
## 2026-04-23 Work Log — Muon iframe origin allowlist rollout
|
||||
|
||||
Status: config rollout applied, image rebuilt, runtime verified on `http://work:5174`.
|
||||
@@ -418,6 +455,134 @@ Completed all prerequisites and fork scaffolding for Muon visualizer fork integr
|
||||
### Blockers and next steps
|
||||
|
||||
**External action required**: Fork `najafmohammed/muon-music-visualizer` to StemEngine GitHub org
|
||||
|
||||
## 2026-04-28 Work Log — Visualizer iframe protocol freeze (Phase 1)
|
||||
|
||||
Status: documentation pass complete.
|
||||
|
||||
Completed in this pass:
|
||||
|
||||
- Added Phase 1 protocol spec:
|
||||
- `docs/specs/visualizer-iframe-protocol.md`
|
||||
- freezes protocol `version: 1`
|
||||
- documents current commands/events implemented by:
|
||||
- `frontend/src/hooks/useMuonIframeTransport.ts`
|
||||
- `muon-iframe/stemengine-router.js`
|
||||
- reserves later-phase extensions:
|
||||
- `capabilities`
|
||||
- `setParam`
|
||||
- `setPalette`
|
||||
- `dispose`
|
||||
|
||||
- Mirrored the spec into the export package:
|
||||
- `docs/muon-mode-status/export/muon-mode-review-package-2026-04-28/docs/specs/visualizer-iframe-protocol.md`
|
||||
|
||||
- Updated tracking docs to reflect Phase 1 completion:
|
||||
- `docs/muon-mode-status/2026-04-28-handoff-to-copilot.md`
|
||||
- `docs/muon-mode-status/export/muon-mode-review-package-2026-04-28/docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md`
|
||||
- `docs/muon-mode-status/README.md`
|
||||
- `docs/muon-mode-status/2026-04-28-export-bundle-manifest.md`
|
||||
- export package README/manifest counterparts
|
||||
|
||||
Validation:
|
||||
|
||||
- Targeted grep verification confirmed:
|
||||
- Phase 1 completion markers exist in the handoff and architecture plan
|
||||
- the new spec is referenced from the review index and export manifests
|
||||
- No runtime source files changed in this pass.
|
||||
|
||||
## 2026-04-28 Work Log — Shared scaffolding extraction (Phase 2)
|
||||
|
||||
Status: implementation pass complete.
|
||||
|
||||
Completed in this pass:
|
||||
|
||||
- Added shared scaffolding package area:
|
||||
- `visualizer-iframes/shared/package.json`
|
||||
- `visualizer-iframes/shared/stemengine-bootstrap.js`
|
||||
- `visualizer-iframes/shared/stemengine-router.js`
|
||||
- `visualizer-iframes/shared/stemengine-audio-engine.js`
|
||||
- `visualizer-iframes/shared/stemengine-feature-extractor.js`
|
||||
|
||||
- Migrated Muon build consumption to shared context:
|
||||
- `muon-iframe/Dockerfile`
|
||||
- switched scaffolding copies to `COPY --from=shared ...`
|
||||
- added Dockerfile syntax header for BuildKit context copy support
|
||||
|
||||
- Added compose wiring for shared context:
|
||||
- `/home/translator/docker/stemengine.yml`
|
||||
- `stemengine-muon.build.additional_contexts.shared=${SEFOLDER}/visualizer-iframes/shared`
|
||||
|
||||
- Added provider-neutral compatibility aliases in shared scaffolding:
|
||||
- `__STEMENGINE_VIZ_*` globals now mirror MUON globals so existing render graphs remain functional while future iframe providers can adopt neutral names
|
||||
- shared router now accepts `setParam`, `setPalette`, and `dispose` commands and emits a baseline `capabilities` event
|
||||
|
||||
Validation:
|
||||
|
||||
- Syntax checks passed:
|
||||
- `node --check visualizer-iframes/shared/stemengine-bootstrap.js`
|
||||
- `node --check visualizer-iframes/shared/stemengine-audio-engine.js`
|
||||
- `node --check visualizer-iframes/shared/stemengine-feature-extractor.js`
|
||||
- `node --check visualizer-iframes/shared/stemengine-router.js`
|
||||
|
||||
- Runtime build/redeploy passed:
|
||||
- `docker compose -p stemengine -f stemengine.yml build stemengine-muon` ✅
|
||||
- `docker compose -p stemengine -f stemengine.yml up -d --no-deps --force-recreate stemengine-muon` ✅
|
||||
- `docker compose -p stemengine -f stemengine.yml ps stemengine-muon` → container up on `0.0.0.0:3011->80/tcp`
|
||||
|
||||
- Browser sanity check:
|
||||
- `http://localhost:3011/` loads (`Muon Music Visualizer`)
|
||||
|
||||
## 2026-04-28 Work Log — Provider-neutral host transport refactor (Phase 3)
|
||||
|
||||
Status: implementation pass complete.
|
||||
|
||||
Completed in this pass:
|
||||
|
||||
- Added provider registry utilities:
|
||||
- `frontend/src/visualizers/providers.ts`
|
||||
- centralizes iframe URL resolution for `muon|fluid|tendrils|spirit`
|
||||
- hoists `MUON_IFRAME_REV` + shared URL preset helper (`withDefaultVisualizerPreset`)
|
||||
|
||||
- Added provider-neutral host hook:
|
||||
- `frontend/src/hooks/useVisualizerIframeTransport.ts`
|
||||
- replaces Muon-specific naming with provider input
|
||||
- supports `palette` and `params` command propagation (`setPalette`, `setParam`)
|
||||
- parses and exposes iframe `capabilities` event
|
||||
|
||||
- Added dynamic renderer controls component:
|
||||
- `frontend/src/components/visualizers/RendererControlsPanel.tsx`
|
||||
- renders number/enum/boolean controls from capabilities metadata
|
||||
- supports reset callback
|
||||
|
||||
- Updated visualizer settings schema/types for per-renderer iframe state:
|
||||
- `frontend/src/components/visualizers/visualizerTypes.ts`
|
||||
- `frontend/src/components/visualizers/visualizerSettings.ts`
|
||||
- added `muon_iframe.params` keyed by style and `muon_iframe.paletteOverride`
|
||||
|
||||
- Updated API settings typing for forward-compatible provider URLs:
|
||||
- `frontend/src/api/client.ts`
|
||||
- adds optional `fluid_iframe_url`, `tendrils_iframe_url`, `spirit_iframe_url`
|
||||
|
||||
- Migrated all three consumers from `useMuonIframeTransport` to `useVisualizerIframeTransport`:
|
||||
- `frontend/src/components/LyricsLivePreview.tsx`
|
||||
- `frontend/src/components/VisualizerPreviewWidget.tsx`
|
||||
- `frontend/src/pages/VisualizerLabPage.tsx`
|
||||
- each now uses provider registry + dynamic renderer controls
|
||||
|
||||
- Removed obsolete Muon-specific hook:
|
||||
- deleted `frontend/src/hooks/useMuonIframeTransport.ts`
|
||||
- updated remaining in-file comment in `frontend/src/components/LyricsViewport.tsx`
|
||||
|
||||
Validation:
|
||||
|
||||
- Frontend typecheck passed:
|
||||
- `cd /home/translator/docker/appdata/stemengine/frontend && npm run -s typecheck; echo typecheck_exit=$?`
|
||||
- `typecheck_exit=0`
|
||||
|
||||
- Additional checks:
|
||||
- confirmed no remaining runtime imports of `useMuonIframeTransport` in frontend source
|
||||
- confirmed all three host consumers now import/use `useVisualizerIframeTransport`
|
||||
- Once forked: `MUON_GIT_REF` can point to fork, Phase 1.1 stubs can be committed, Phase 2 rendering begins
|
||||
|
||||
**Recommended next**: Execute unblocked gate checks (1.2, 1.4, 1.5, 1.6) for ~60 min validation run
|
||||
@@ -12773,3 +12938,447 @@ Documentation/log updates:
|
||||
- updated `docs/plans/2026-04-22_Backend_Test_Fixes/Fixes.md`
|
||||
- updated `docs/plans/2026-04-22_Backend_Test_Fixes/2026-04-22_Copilot_Review.md`
|
||||
- updated `docs/plans/2026-04-22_Karaoke_Gloss/2026-04-22_Backend_Test_Failures.md`
|
||||
|
||||
## 2026-04-28 Work Log — Fluid-Simulation upstream inspection & Phase 4 pre-work documentation
|
||||
|
||||
Status: **Phase 4 pre-work complete** (upstream inspection + design documentation ready for implementation)
|
||||
|
||||
Completed in this pass:
|
||||
|
||||
- **Upstream repository inspection:**
|
||||
- Cloned PavelDoGreat/Fluid-Simulation at pinned commit `3db8fa9` ("still metal")
|
||||
- Analyzed project structure:
|
||||
- Tech stack: Zig source → WebAssembly (via `zig build -Dtarget=wasm32-freestanding`)
|
||||
- Web entry: `projects/web/index.html` + `projects/web/script.js` (122 lines)
|
||||
- WASM binary: `projects/web/bin/fluid.wasm` (compiled, 226 commits upstream)
|
||||
- Graphics API: WebGL2
|
||||
- Current state: `start()`, `update()`, `draw()` exports are **stubs** (no-op in web build)
|
||||
|
||||
- **Audio injection mechanism design:**
|
||||
- Proposed new WASM exports: `setAudioBands(band_ptr, band_count)`
|
||||
- Memory layout: 32-byte reserved region for up to 8 frequency bands
|
||||
- Integration point: `update()` function reads bands and applies to fluid simulation parameters
|
||||
- Host-side: `stemengine-audio-engine.js` extracts bands and calls export
|
||||
- Bands drive: bass (pressure), mid-bass (vortex), treble (color intensity)
|
||||
|
||||
- **Palette injection mechanism design:**
|
||||
- Proposed new WASM exports: `setPalette(palette_ptr, color_count, format)`
|
||||
- Memory layout: 256 bytes (64 colors × 4 bytes RGBA)
|
||||
- Fragment shader: maps fluid field values to palette indices
|
||||
- Host-side: RendererControlsPanel sends custom palette on change
|
||||
- Format enum: 0=RGB, 1=RGBA, 2=ARGB
|
||||
|
||||
- **Tunable parameters registry:**
|
||||
- Identified parameter set: `viscosity`, `pressure`, `decay`, `audio_sensitivity`, `frequency_bias`, `color_intensity`, `color_source`
|
||||
- Proposed WASM exports: `setParameter(param_id, value)`, `getParameter(param_id)`, `getParameterMetadata(param_id)`
|
||||
- Each parameter has min/max/default/type metadata for dynamic UI rendering
|
||||
|
||||
- **Comprehensive implementation guide created:**
|
||||
- `visualizer-iframes/fluid/audio-injection-patches.md` (380 lines)
|
||||
- Documents:
|
||||
- Project structure and status
|
||||
- Audio injection architecture (WASM exports, memory layout, Zig integration)
|
||||
- Palette injection architecture (format, shader integration, host wiring)
|
||||
- Tunable parameters with ranges and purposes
|
||||
- Zig patches required (src/fluid/main.zig, new modules: audio.zig, parameters.zig, palette.zig)
|
||||
- JavaScript patches required (script.js, shared router)
|
||||
- Docker build strategy (compile + patch + serve)
|
||||
- Frontend integration workflow (provider registration, hook update, controls panel wiring)
|
||||
- Capabilities announcement protocol (new fields: params, supportsAudio, supportsPalette)
|
||||
- Testing & validation checklist
|
||||
- Future extensions (Phase 5+)
|
||||
- Includes commit hash reference for reproducibility
|
||||
|
||||
**Key Architecture Insights:**
|
||||
|
||||
- Fluid is **compile-time complete** (WASM module exists), audio/palette are **post-compile injections** via JavaScript ↔ WASM interface
|
||||
- Unlike Muon (render graph in JS), Fluid dynamics compute in WASM; injection happens via memory buffers + exported functions
|
||||
- Protocol alignment: `setAudioBands` and `setPalette` fit the existing `useVisualizerIframeTransport` message pattern (Phase 3 reserved fields now have concrete use)
|
||||
- Capabilities exposure (parameters list) will populate `RendererControlsPanel` dynamically without frontend rewrites
|
||||
|
||||
**Phase 4 Implementation Readiness:**
|
||||
|
||||
- ✅ Upstream pinned and analyzed
|
||||
- ✅ Audio injection mechanism documented with Zig integration plan
|
||||
- ✅ Palette injection mechanism documented with shader integration plan
|
||||
- ✅ Tunable parameters identified and schema designed
|
||||
- ✅ Dockerfile build strategy ready (compile + patch + serve)
|
||||
- ✅ Frontend integration points pre-wired (Phase 3 reserved fields + components ready)
|
||||
- ✅ Testing checklist available
|
||||
|
||||
**Estimated Phase 4 Duration:** 3–4 hours
|
||||
- Zig implementation (audio/palette/parameters modules): ~1.5 hours
|
||||
- Docker build + integration: ~0.5 hours
|
||||
- Provider registration + testing: ~1.5 hours
|
||||
|
||||
**Next Immediate Steps (Phase 4):**
|
||||
1. Apply Zig patches to `src/fluid/main.zig` (audio integration in `update()`)
|
||||
2. Create new Zig modules (`audio.zig`, `parameters.zig`, `palette.zig`)
|
||||
3. Build WASM: `zig build -Dtarget=wasm32-freestanding`
|
||||
4. Create `visualizer-iframes/fluid/Dockerfile` with patch application + build
|
||||
5. Add `stemengine-fluid` service to compose with shared context
|
||||
6. Register Fluid provider in `frontend/src/visualizers/providers.ts`
|
||||
7. Run integration test: audio playback → fluid response → color change
|
||||
8. Documentation + release-gate log sync
|
||||
|
||||
## 2026-04-28 Work Log — Phase 4 implementation (Fluid iframe cutover)
|
||||
|
||||
Status: implementation pass complete.
|
||||
|
||||
Completed in this pass:
|
||||
|
||||
- Built and wired standalone Fluid iframe runtime:
|
||||
- `visualizer-iframes/fluid/Dockerfile`
|
||||
- `visualizer-iframes/fluid/nginx.conf`
|
||||
- `visualizer-iframes/fluid/index.html`
|
||||
- `visualizer-iframes/fluid/stemengine-fluid-adapter.js`
|
||||
- upstream source pinned via `FLUID_GIT_REF=3db8fa9` and served from `projects/web/`
|
||||
- injected shared StemEngine scaffolding (`stemengine-bootstrap.js`, `stemengine-router.js`, `stemengine-audio-engine.js`, `stemengine-feature-extractor.js`)
|
||||
|
||||
- Added compose service:
|
||||
- `/home/translator/docker/stemengine.yml`
|
||||
- new `stemengine-fluid` service on `${FLUID_HOST_PORT:-3012}:80`
|
||||
- includes `additional_contexts.shared=${SEFOLDER}/visualizer-iframes/shared`
|
||||
|
||||
- Completed provider cutover in host consumers (style-driven provider selection):
|
||||
- `frontend/src/components/LyricsLivePreview.tsx`
|
||||
- `frontend/src/components/VisualizerPreviewWidget.tsx`
|
||||
- `frontend/src/pages/VisualizerLabPage.tsx`
|
||||
- all now resolve active iframe provider from selected style:
|
||||
- `default -> muon`
|
||||
- `fluid -> fluid`
|
||||
- `tendrils -> tendrils`
|
||||
- `spirit -> spirit`
|
||||
- iframe URL/title/error copy now reflect active provider instead of hard-coded Muon URL
|
||||
|
||||
- Updated viewport wrapper to provider-neutral iframe props:
|
||||
- `frontend/src/components/LyricsViewport.tsx`
|
||||
- renamed props:
|
||||
- `muonEmbedUrl -> iframeEmbedUrl`
|
||||
- added `iframeTitle`
|
||||
|
||||
- Extended shared router for provider-neutral capabilities/ready payloads:
|
||||
- `visualizer-iframes/shared/stemengine-router.js`
|
||||
- `capabilities` now emitted from optional `window.__STEMENGINE_VIZ_CAPABILITIES__`
|
||||
- `providerId` now emitted from `window.__STEMENGINE_VIZ_PROVIDER_ID__`
|
||||
- `ready.styles` now emitted from optional `window.__STEMENGINE_VIZ_STYLES__`
|
||||
|
||||
- Removed obsolete in-Muon fluid renderer path:
|
||||
- deleted `muon-iframe/fluid-render-graph.js`
|
||||
- updated `muon-iframe/Dockerfile` to stop copying/importing fluid render graph
|
||||
|
||||
Validation:
|
||||
|
||||
- Frontend typecheck passed:
|
||||
- `cd /home/translator/docker/appdata/stemengine/frontend && npm run -s typecheck; echo typecheck_exit=$?`
|
||||
- `typecheck_exit=0`
|
||||
|
||||
- Compose config validation passed:
|
||||
- `cd /home/translator/docker && docker compose -p stemengine -f stemengine.yml config >/tmp/stemengine_compose_config.out && echo compose_config_exit=$?`
|
||||
- `compose_config_exit=0`
|
||||
|
||||
- Runtime image/service validation passed:
|
||||
- `docker compose -p stemengine -f stemengine.yml build stemengine-fluid stemengine-muon` ✅
|
||||
- `docker compose -p stemengine -f stemengine.yml up -d --no-deps --force-recreate stemengine-fluid stemengine-muon` ✅
|
||||
- `docker compose -p stemengine -f stemengine.yml ps stemengine-fluid stemengine-muon` ✅
|
||||
- active ports:
|
||||
- `stemengine-fluid`: `0.0.0.0:3012->80/tcp`
|
||||
- `stemengine-muon`: `0.0.0.0:3011->80/tcp`
|
||||
|
||||
Documentation updates in this pass:
|
||||
|
||||
- `docs/muon-mode-status/export/muon-mode-review-package-2026-04-28/docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md`
|
||||
- marked Phase 4 pre-work and Phase 4 implementation checklist items complete
|
||||
- updated execution order to show Phase 4 complete baseline
|
||||
|
||||
- `docs/muon-mode-status/2026-04-28-handoff-to-copilot.md`
|
||||
- status advanced to Phases 0–4 complete
|
||||
- TLDR now points to Phase 5 pre-work as next action
|
||||
|
||||
Next immediate step:
|
||||
|
||||
- Start Phase 5 pre-work: inspect `keeffEoghan/keeffeoghan.github.io` tendrils subtree and write `visualizer-iframes/tendrils/audio-injection-patches.md` with pinned commit and injection plan.
|
||||
|
||||
## 2026-04-28 Work Log — Phase 5 pre-work (Tendrils upstream inspection)
|
||||
|
||||
Status: pre-work analysis complete and documented.
|
||||
|
||||
Completed in this pass:
|
||||
|
||||
- Inspected Tendrils upstream repository and pinned commit:
|
||||
- upstream: `https://github.com/keeffEoghan/keeffeoghan.github.io`
|
||||
- pinned commit: `482db7d3127064e6245631087e0ef0a1641f3b5e`
|
||||
- observed Tendrils source subtree at `src/tendrils/`
|
||||
|
||||
- Identified runtime chain and integration points:
|
||||
- entrypoint: `src/work/tendrils.main.js` -> `src/tendrils/demo.js`
|
||||
- core runtime/state: `src/tendrils/index.js`
|
||||
- audio trigger ownership in upstream demo: analyser + `AudioTrigger` + `audioResponse()` action gates
|
||||
- palette control points in Tendrils state: `baseColor`, `flowColor`, `fadeColor`, `colorMapAlpha`
|
||||
|
||||
- Assessed buildability/extraction strategy:
|
||||
- upstream is full-site repo (not Tendrils-only package)
|
||||
- root `package.json` exists with legacy gulp/webpack toolchain
|
||||
- prebuilt Tendrils artifacts are present in repo (`build/js/work/tendrils.js`, `build/css/work/tendrils.css`, `work/tendrils/index.html`)
|
||||
- selected preferred implementation strategy for Phase 5: use pinned prebuilt artifacts + StemEngine adapter injection, avoid legacy toolchain rebuild risk
|
||||
|
||||
- License check result:
|
||||
- no top-level `LICENSE`/`COPYING` file found in inspected snapshot
|
||||
- `package.json` has no `license` field
|
||||
- marked as unresolved release-gate item before production rollout
|
||||
|
||||
- Documentation deliverable created:
|
||||
- `visualizer-iframes/tendrils/audio-injection-patches.md`
|
||||
- includes pinned commit, audio/palette injection mechanism, parameter capability candidates, Docker/compose deltas, and risk register
|
||||
|
||||
Validation:
|
||||
|
||||
- Verified commit and subtree inventory from local upstream clone under:
|
||||
- `.tmp/upstreams/keeffeoghan.github.io`
|
||||
- Confirmed Tendrils subtree file presence and key entry/runtime files read directly.
|
||||
|
||||
Next immediate step:
|
||||
|
||||
- Execute Phase 5 implementation using the documented adapter + standalone iframe strategy:
|
||||
- create `visualizer-iframes/tendrils/` runtime files,
|
||||
- add `stemengine-tendrils` compose service,
|
||||
- route host provider to standalone tendrils iframe,
|
||||
- delete `muon-iframe/tendrils-render-graph.js`.
|
||||
|
||||
## 2026-04-28 Work Log — Phase 5 implementation (Tendrils iframe cutover)
|
||||
|
||||
Status: implementation pass complete.
|
||||
|
||||
Completed in this pass:
|
||||
|
||||
- Built and wired standalone Tendrils iframe runtime:
|
||||
- `visualizer-iframes/tendrils/Dockerfile`
|
||||
- `visualizer-iframes/tendrils/nginx.conf`
|
||||
- `visualizer-iframes/tendrils/stemengine-tendrils-adapter.js`
|
||||
- upstream source pinned via `TENDRILS_GIT_REF=482db7d3127064e6245631087e0ef0a1641f3b5e`
|
||||
- runtime served from upstream Tendrils page path (`work/tendrils/index.html`) with shared StemEngine scaffolding injected via nginx sub_filter
|
||||
|
||||
- Added compose service:
|
||||
- `/home/translator/docker/stemengine.yml`
|
||||
- new `stemengine-tendrils` service on `${TENDRILS_HOST_PORT:-3013}:80`
|
||||
- includes `additional_contexts.shared=${SEFOLDER}/visualizer-iframes/shared`
|
||||
|
||||
- Removed obsolete in-Muon Tendrils renderer path:
|
||||
- deleted `muon-iframe/tendrils-render-graph.js`
|
||||
- updated `muon-iframe/Dockerfile` to stop copying/importing tendrils render graph
|
||||
|
||||
Validation:
|
||||
|
||||
- Compose config validation passed:
|
||||
- `cd /home/translator/docker && docker compose -p stemengine -f stemengine.yml config >/tmp/stemengine_compose_config.out && echo compose_config_exit=$?`
|
||||
- `compose_config_exit=0`
|
||||
|
||||
- Runtime image/service validation passed:
|
||||
- `docker compose -p stemengine -f stemengine.yml build stemengine-tendrils stemengine-muon` ✅
|
||||
- `docker compose -p stemengine -f stemengine.yml up -d --no-deps --force-recreate stemengine-tendrils stemengine-muon` ✅
|
||||
- `docker compose -p stemengine -f stemengine.yml ps stemengine-tendrils stemengine-muon` ✅
|
||||
- active ports:
|
||||
- `stemengine-tendrils`: `0.0.0.0:3013->80/tcp`
|
||||
- `stemengine-muon`: `0.0.0.0:3011->80/tcp`
|
||||
|
||||
- Standalone endpoint smoke check passed:
|
||||
- `curl -fsS http://localhost:3013/ | head -n 5` returned Tendrils HTML document.
|
||||
|
||||
Next immediate step:
|
||||
|
||||
- Execute Phase 6 validation/documentation gate:
|
||||
- run cross-surface behavior checks for `default|fluid|tendrils`,
|
||||
- update surface matrix/reviewer package with post-migration outcomes,
|
||||
- resolve Tendrils upstream licensing status before ship-ready declaration.
|
||||
|
||||
## 2026-04-28 Work Log — Phase 6 gate progress (automation + docs)
|
||||
|
||||
Status: partial completion (automation/documentation done; interactive browser sweep pending).
|
||||
|
||||
Completed in this pass:
|
||||
|
||||
- Captured post-cutover infrastructure evidence:
|
||||
- `docker compose -p stemengine -f stemengine.yml ps stemengine-muon stemengine-fluid stemengine-tendrils`
|
||||
- all three services up and mapped on ports `3011`, `3012`, `3013`
|
||||
|
||||
- Captured direct endpoint evidence:
|
||||
- `curl -fsS http://localhost:3011/ | head -n 3` (Muon HTML)
|
||||
- `curl -fsS http://localhost:3012/ | head -n 3` (Fluid HTML)
|
||||
- `curl -fsS http://localhost:3013/ | head -n 3` (Tendrils HTML)
|
||||
|
||||
- Reconfirmed frontend compile gate:
|
||||
- `cd frontend && npm run -s typecheck`
|
||||
- `typecheck_exit=0`
|
||||
|
||||
- Updated phase-6 docs/checklists for current state:
|
||||
- `docs/muon-mode-status/2026-04-28-surface-functionality-matrix.md`
|
||||
- added post-cutover delta and automated verification evidence
|
||||
- `docs/muon-mode-status/2026-04-28-reviewer-checklist.md`
|
||||
- added Phase 6 addendum (done checks + remaining closure checks)
|
||||
- `docs/muon-mode-status/2026-04-28-reference-comparison-third-party-review.md`
|
||||
- added post-cutover correction replacing obsolete Tendrils in-Muon evidence path
|
||||
|
||||
- Licensing gate evidence refresh:
|
||||
- rechecked upstream Tendrils snapshot (`.tmp/upstreams/keeffeoghan.github.io`)
|
||||
- no top-level project license file found
|
||||
- only dependency-level `libs/dat.gui/LICENSE` present
|
||||
- release-gate license status remains unresolved for upstream Tendrils project provenance
|
||||
|
||||
Remaining to close Phase 6:
|
||||
|
||||
- Run interactive browser parity matrix for `default|fluid|tendrils` across:
|
||||
- Track Detail
|
||||
- Visualizer Lab (with uploaded local audio)
|
||||
- Project page (`LyricsLivePreview` after selecting Muon visual style)
|
||||
- Record PASS/PARTIAL/FAIL outcomes in reviewer docs and finalize release recommendation.
|
||||
|
||||
## 2026-04-28 Work Log — Phase 6 interactive attempt (blocker capture)
|
||||
|
||||
Status: attempted; parity closure blocked by backend/API errors plus environment upload constraint.
|
||||
|
||||
Completed in this pass:
|
||||
|
||||
- Ran browser interaction pass on Visualizer Lab and verified provider routing behavior:
|
||||
- style `default` -> iframe `http://localhost:3011/?preset=default&se_rev=20260427c`
|
||||
- style `fluid` -> iframe `http://localhost:3012`
|
||||
- style `tendrils` -> iframe `http://localhost:3013`
|
||||
|
||||
- Attempted Track Detail and Project page parity runs; both failed to reach executable test state.
|
||||
|
||||
- Captured direct API evidence through both user-facing hosts (`3001` and `5174`):
|
||||
- `/api/projects/e98f2b8a-0cad-4b4e-b33e-1ca2c8c87a0a` -> `500`
|
||||
- `/api/system/gpu-info` -> `500`
|
||||
- `/api/tracks/a5f76485ba8f7e5c41aea2c6a41ca0eb` -> `500`
|
||||
|
||||
- Captured Visualizer Lab upload blocker in this agent-browser environment:
|
||||
- file chooser path translation produced `ENOENT` against translated `C:\home\...` path for local test asset,
|
||||
- preventing play-cycle execution from this session.
|
||||
|
||||
Result:
|
||||
|
||||
- Host/provider cutover behavior is interactively verified for `default|fluid|tendrils` URL routing.
|
||||
- End-to-end replay/reactivity PASS/FAIL closure remains blocked until backend `500` conditions are cleared (and upload path is testable in the active browser context).
|
||||
|
||||
## 2026-04-28 Work Log — Visualizer iframe overlay removal
|
||||
|
||||
Status: implemented.
|
||||
|
||||
Objective:
|
||||
- remove graphical overlays from visualizer iframes (FPS counters, debug GUI panels, control buttons, and related chrome).
|
||||
|
||||
Implementation completed:
|
||||
|
||||
- Added global overlay suppression policy in shared iframe bootstrap:
|
||||
- `visualizer-iframes/shared/stemengine-bootstrap.js`
|
||||
- introduced selector-driven suppression for common overlay/debug UI classes (`.dg`, `.lil-gui`, `#stats`, `.fps*`, Tendrils `epok-*` controls).
|
||||
- injects a suppression stylesheet and installs a mutation observer that removes matching overlay nodes even if they are re-added after initial load.
|
||||
|
||||
- Hardened Tendrils adapter cleanup for upstream UI chrome:
|
||||
- `visualizer-iframes/tendrils/stemengine-tendrils-adapter.js`
|
||||
- expanded `hideHeavyChrome()` to remove Tendrils overlay/control elements (`.epok-info`, `.epok-main-controls`, `.epok-audio-controls`, capture overlay, edit/control panels), plus common stats/debug UI classes.
|
||||
|
||||
Result:
|
||||
|
||||
- Visualizer iframe runtime now enforces overlay-free presentation at the iframe layer.
|
||||
- Overlay/debug UI and button chrome are suppressed/removed across providers using shared bootstrap policy, with explicit Tendrils hardening.
|
||||
|
||||
Follow-up runtime fix (same session):
|
||||
|
||||
- Root-route Tendrils responses initially bypassed script injection in some cached browser loads, so suppression scripts did not execute on `/`.
|
||||
- Updated `visualizer-iframes/tendrils/nginx.conf` routing to ensure HTML requests resolve through the injection-enabled location path.
|
||||
- Rebuilt and recreated `stemengine-fluid` and `stemengine-tendrils` containers.
|
||||
|
||||
Live verification after recreate (`?se_cache_bust=3`):
|
||||
|
||||
- Tendrils (`http://localhost:3013/`):
|
||||
- `window.__STEMENGINE_VIZ_PROVIDER_ID__ === "tendrils"`
|
||||
- bootstrap present
|
||||
- overlay selectors count all `0` (`.epok-info`, `.epok-main-controls`, `.epok-audio-controls`, `.epok-captured-overlay`, `.epok-edit-controls`, `.epok-main-control/.epok-button`, stats/gui selectors)
|
||||
- Fluid (`http://localhost:3012/`):
|
||||
- `window.__STEMENGINE_VIZ_PROVIDER_ID__ === "fluid"`
|
||||
- bootstrap present
|
||||
- overlay/debug selectors count `0`
|
||||
|
||||
## 2026-04-28 Work Log — Phase 6 Spirit decision documentation
|
||||
|
||||
Status: completed (documentation step).
|
||||
|
||||
Completed in this pass:
|
||||
|
||||
- Added formal Spirit decision record:
|
||||
- `visualizer-iframes/spirit/DECISION.md`
|
||||
- decision: defer upstream Spirit iframe for current release-gate window and keep `muon-iframe/spirit-render-graph.js` temporarily.
|
||||
|
||||
- Updated architecture plan checklist to separate Spirit decision completion from remaining interactive closure tasks:
|
||||
- marked Spirit decision deliverable complete
|
||||
- kept cross-surface PASS/FAIL closure work open
|
||||
- updated execution-order step for Spirit pre-work to reference completed decision record
|
||||
|
||||
- Updated reviewer checklist with explicit remaining Spirit label task and documentation progress marker.
|
||||
|
||||
Result:
|
||||
|
||||
- Phase 6 tracking is now split cleanly into:
|
||||
- completed documentation/decision artifacts, and
|
||||
- still-blocked interactive validation + reviewer closure items.
|
||||
|
||||
## 2026-04-28 Work Log — Spirit decision pivot (option B selected)
|
||||
|
||||
Status: completed (planning/documentation update prior to implementation).
|
||||
|
||||
Decision update applied:
|
||||
|
||||
- Previous temporary deferral decision was superseded by explicit product direction:
|
||||
- ship upstream Spirit with synthetic audio coupling (option B).
|
||||
|
||||
- Captured selected implementation constraints in `visualizer-iframes/spirit/DECISION.md`:
|
||||
- coupling model: uniform-driver mapping from `window.__STEMENGINE_VIZ_AUDIO__`
|
||||
- beat profile: aggressive
|
||||
- fallback: passive upstream Spirit if coupling init fails
|
||||
- palette support: required in first cut
|
||||
- style label: `Spirit`
|
||||
|
||||
Plan/checklist synchronization completed:
|
||||
|
||||
- Updated architecture plan (`.../2026-04-28_local-iframe-visualizer-architecture-plan.md`) to:
|
||||
- replace deferred Spirit language with selected Phase 6b implementation path,
|
||||
- replace option matrix with selected implementation tasks,
|
||||
- update risks/criteria/checklist/execution-order entries accordingly.
|
||||
|
||||
- Updated reviewer checklist (`docs/muon-mode-status/2026-04-28-reviewer-checklist.md`) to:
|
||||
- replace approximation-label task with `Spirit` label alignment,
|
||||
- add Spirit upstream implementation readiness checklist items.
|
||||
|
||||
Result:
|
||||
|
||||
- Spirit direction is now explicitly locked to upstream + synthetic coupling before implementation starts.
|
||||
|
||||
## 2026-04-28 Work Log — Gloss overlap + spacing parity fix
|
||||
|
||||
Status: completed.
|
||||
|
||||
Issue observed:
|
||||
|
||||
- Gloss text could overlap adjacent gloss tokens because horizontal word layout budget was based only on base lyric token width, while gloss strings can be longer even at reduced font size.
|
||||
- Gloss baseline gap was also too tight, causing visual crowding near the main lyric line.
|
||||
|
||||
Implemented in `frontend/src/components/LyricsCanvas.tsx`:
|
||||
|
||||
- Increased default gloss vertical separation from `+4` to `+8` pixels in both:
|
||||
- line stride expansion for gloss lines,
|
||||
- gloss Y-offset from lyric baseline.
|
||||
- Reworked per-word layout width computation:
|
||||
- measure lyric width and gloss width per token,
|
||||
- use `max(lyricWidth, glossWidth)` as the horizontal layout width used by row wrapping and cursor advance.
|
||||
- Preserved visual parity between lyric and gloss styling (except size):
|
||||
- gloss uses same stroke enable/color logic as lyric text,
|
||||
- gloss stroke width scales proportionally to size ratio (`glossFontSize / style.fontSize`),
|
||||
- gloss weight matches lyric weight (`700/400` mapping).
|
||||
- Applied active-line rendering parity for gloss SVG nodes:
|
||||
- `paint-order: stroke fill` + `text-rendering: geometricPrecision`,
|
||||
- shadow filter application on gloss-behind text when shadow is enabled.
|
||||
|
||||
Expected result:
|
||||
|
||||
- Gloss tokens no longer collide horizontally on longer transliterations.
|
||||
- Gloss remains visually consistent with lyric rendering model while retaining reduced size.
|
||||
@@ -1,5 +1,27 @@
|
||||
# StemEngine — Session Summaries
|
||||
|
||||
## 2026-04-28
|
||||
|
||||
### Spirit upstream provider cutover (option B)
|
||||
|
||||
Completed Spirit migration from in-Muon approximation path to standalone upstream iframe provider.
|
||||
|
||||
Implemented:
|
||||
- new runtime in `visualizer-iframes/spirit/` (`Dockerfile`, `index.html`, `nginx.conf`, prehook, adapter)
|
||||
- new compose service `stemengine-spirit` on port `3014`
|
||||
- synthetic audio coupling adapter using `window.__STEMENGINE_VIZ_AUDIO__`
|
||||
- aggressive beat-response profile
|
||||
- palette propagation support (`setPalette` -> Spirit colors/background)
|
||||
- passive upstream fallback path with warning telemetry if coupling init cannot lock
|
||||
|
||||
Validation in this pass:
|
||||
- Spirit container builds and runs (`docker compose build/up/ps`)
|
||||
- endpoint serves provider declaration and injected transport/coupling scripts
|
||||
- browser smoke confirms Spirit page boot + canvas mount
|
||||
|
||||
Caveat:
|
||||
- in agent-browser harness, coupling target discovery did not bind to a detectable Spirit uniform driver, so fallback remained active in this environment; full interactive parity scoring is still pending after backend/upload blockers are cleared.
|
||||
|
||||
## 2026-04-23
|
||||
|
||||
### Muon iframe origin allowlist rollout
|
||||
@@ -399,11 +421,216 @@ Implemented the critical fixes identified in `Review.md`. Document detail in `do
|
||||
|
||||
**Validation:** `npm run -s typecheck` exit 0. `grep -rn 'STEMENGINE_ALLOWED_ORIGIN'` returns zero matches in live code (only in docs/CI/historical). `grep -n 'sandbox' frontend/src/components/{LyricsViewport,VisualizerPreviewWidget}.tsx frontend/src/pages/VisualizerLabPage.tsx` returns zero matches. Browser-level validation requires `docker compose build stemengine-muon && docker compose up -d stemengine-muon` then manual smoke test of all four modes at all three mounting sites; not run from this session.
|
||||
|
||||
### Muon iframe — review of Copilot's `docs/muon-mode-status/` package (2026-04-28)
|
||||
|
||||
Reviewed Copilot's third-party review package (5 docs + export bundle + architecture plan) using the agency-agents skills. Surgical edits applied; consolidated review notes written to a new file.
|
||||
|
||||
**Factual errors corrected in `2026-04-28-surface-functionality-matrix.md`:**
|
||||
- Project page row was marked N/A across all four modes. Wrong — `pages/ProjectDetail.tsx:18, 2016` lazy-imports and renders `LyricsLivePreview`, which calls `useMuonIframeTransport({ ... })` at line 1611. The "no controls visible" snapshot reflected the default lyric-style state; once the user switches the visual style to Muon via the style picker, all four Muon styles are testable. Corrected to "TESTABLE" with the path explained.
|
||||
- Visualizer Lab row was marked BLOCKED across all four modes. Misleading — the lab page has a working file upload button at `pages/VisualizerLabPage.tsx:85-102`. Tester didn't upload a file. Corrected to "OPENABLE" with the workaround pinned.
|
||||
|
||||
**New file:** `docs/muon-mode-status/2026-04-28-claude-review-notes.md` — captures nine review findings (R-1 through R-9) that the package didn't address:
|
||||
- R-1: "default: functional with caveats" depends on the D-1 fix shipped 2026-04-27 (analyser race) — should be cited so that anyone running the matrix against an older Muon image doesn't conclude the report is wrong
|
||||
- R-2: `ERR_ABORTED` / 502 caveat conflates audio-stream-backend health with iframe behavior; affects all four modes equally
|
||||
- R-3: Fluid FBO incomplete (`36054 = FRAMEBUFFER_INCOMPLETE_ATTACHMENT`) at `fluid-render-graph.js:434` is a fresh-FBO-creation failure, not mid-cycle. Three candidate root causes (extension revocation across iframe remount, capability detect with stale GL context, undefined width/height before setViewport, disposal race). Prescribes a one-line diagnostic patch
|
||||
- R-4: Architecture plan (`local-iframe-visualizer-architecture-plan.md`) is partially obsolete — recommends a new local iframe shell, but D-1/D-3/D-5/D-7/D-8 already stabilized default + tendrils + spirit. The actual remaining defect is fluid-specific FBO instability + cycle-2 replay. New iframe shell is overkill for that
|
||||
- R-5: `MUON_IFRAME_REV` duplicated across three files (`VisualizerLabPage.tsx:16`, `VisualizerPreviewWidget.tsx:35`, `LyricsLivePreview.tsx:67`). Drift risk — recommend hoisting to a shared module
|
||||
- R-6: `withDefaultPreset` divergent — Lab page unconditionally sets `se_rev`, the other two use `if (!url.searchParams.has('se_rev'))`. Combined with `iframe key={muonEmbedUrl}` (newly set at `LyricsViewport.tsx:93`) this could cause cross-surface remount divergence
|
||||
- R-7: Reviewer checklist missing edge cases — proposed Section 2.5 covering iframe lifecycle (style A→B→A round-trip, tab visibility, GPU process crash), audio integration (stream 502 mid-playback, oversized buffer), cross-surface consistency, capability degradation
|
||||
- R-8: Export bundle excludes `stemengine-audio-engine.js`, `stemengine-feature-extractor.js`, `useMuonIframeTransport.ts`, and the 2026-04-27 plan/fix docs. Reviewers can't audit the host-iframe contract end-to-end with only the iframe-side files
|
||||
- R-9: Three falsifiable hypotheses for the cycle-2 replay regression (engine `onended` race, host drift-correction `transportSeek` racing `transportPlay`, fluid render-graph state accumulation), with a console-log diagnostic to discriminate
|
||||
|
||||
`README.md` updated with a pointer to the new review notes (`2026-04-28-claude-review-notes.md`) so reviewers find them in the recommended reading order.
|
||||
|
||||
What I deliberately did NOT change: the architecture plan's main body (R-4 recommends reconciliation but rewriting is the original author's call), the reviewer checklist's main body (R-7 recommends additive Section 2.5), the export bundle's source files (R-8 recommends additions but the bundle is a frozen manifest from yesterday).
|
||||
|
||||
### Muon iframe — architecture plan flipped to per-renderer iframes (2026-04-28 follow-up)
|
||||
|
||||
User decision: each non-default visualizer style (`fluid`, `tendrils`, `spirit`) gets its own dedicated iframe app, modeled exactly on the existing Muon iframe — NOT the shared-shell-with-plugins shape Copilot's plan originally recommended. Four iframe apps total (Muon + three new), each speaking the same postMessage transport contract.
|
||||
|
||||
Revised the plan at `docs/muon-mode-status/export/muon-mode-review-package-2026-04-28/docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md`:
|
||||
|
||||
- **Decision Summary** flipped: per-renderer iframes (final shape), with the reasons-against-shared-shell and reasons-for-per-renderer enumerated
|
||||
- **Section 3 (Per-renderer iframe apps)** added: defines the `visualizer-iframes/` directory structure with `shared/` (npm-packageable scaffolding) and per-style folders (`fluid/`, `tendrils/`, `spirit/`), each containing its own `index.html`, `index.js`, `nginx.conf`, `Dockerfile`. The renderer is invoked at iframe boot — no `setStyle` dispatch needed because the iframe IS the style.
|
||||
- **Section 4 (Docker model)** added: recommends ONE container `stemengine-visualizers` with three nginx routes (`/fluid/`, `/tendrils/`, `/spirit/`) at port 3012, alternative being three separate containers; pros/cons listed.
|
||||
- **Section 5 (Migration model)** updated: existing `muon-iframe/{fluid,tendrils,spirit}-render-graph.js` files are MOVED into `visualizer-iframes/{...}/` and removed from Muon's Dockerfile sed-injection list. Muon iframe goes back to being only the third-party renderer.
|
||||
- **Phase 0 added** as a precondition: diagnose + fix the fluid cycle-2 root cause IN-PLACE before any architectural work, so the migration target is a known-good renderer (avoids carrying the bug across iframes).
|
||||
- **Phases 2 / 3 / 4 / 5** rewritten for the per-renderer flow: Phase 2 extracts shared scaffolding; Phase 3 makes the host hook provider-neutral (`useMuonIframeTransport` → `useVisualizerIframeTransport`, accepts `provider: VisualizerProvider`); Phase 4 builds fluid first as the proof-of-shape; Phase 5 duplicates the shape for tendrils + spirit.
|
||||
- **Risks** updated: added "shared scaffolding drift across iframes" (mitigation: read-only at build time + CI diff check), "iframe-document-swap UX on style switch" (mitigation: pre-warm all four iframes on visualizer-panel open as a CSS toggle), "Phase 0 takes longer than expected" (mitigation: 1-day time-box).
|
||||
- **Validation matrix** updated: added standalone-boot test (open the iframe URL directly, drop a file, see it react — same behavior Muon has today), cross-iframe consistency tests (different reactivity for different styles, no memory leak after 10 style switches).
|
||||
- **Success Definition** rewritten: explicitly includes "adding a 5th visualizer is a self-contained PR" as the scaling test.
|
||||
|
||||
Also updated `docs/muon-mode-status/2026-04-28-claude-review-notes.md` R-4 to reference the now-revised plan as the path forward, preserving the original concern as historical context.
|
||||
|
||||
### Muon iframe — architecture plan Rev 2: third-party upstream per iframe (2026-04-28 same-day)
|
||||
|
||||
User clarification: Rev 1 misunderstood the renderer source. The plan should be each non-default iframe clones a **third-party upstream** at build time (same recipe as Muon iframe today), NOT lifts our existing `*-render-graph.js` approximation files.
|
||||
|
||||
Confirmed upstream URLs:
|
||||
- fluid → `https://github.com/PavelDoGreat/Fluid-Simulation`
|
||||
- tendrils → `https://github.com/keeffEoghan/keeffeoghan.github.io` subtree `src/tendrils/`
|
||||
- spirit → `https://github.com/edankwan/The-Spirit` (DEFERRED — no native audio reactivity; license verification needed)
|
||||
|
||||
Plan revised at `docs/muon-mode-status/export/muon-mode-review-package-2026-04-28/docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md`:
|
||||
|
||||
- **Revision Note** at top now explicitly enumerates Rev 1 → Rev 2; calls out that the existing `muon-iframe/{fluid,tendrils,spirit}-render-graph.js` files are DELETED (except spirit's, kept until Phase 6 decision).
|
||||
- **Decision Summary** rewritten: `stemengine-fluid` clones PavelDoGreat at build time, `stemengine-tendrils` clones keeffEoghan subtree, `stemengine-spirit` deferred. Constraint accepted: each iframe's Dockerfile uses its upstream's own build system (npm/parcel/webpack/etc.); they don't share build steps, only `shared/` scaffolding.
|
||||
- **Section 3 (Per-renderer iframe apps)** rewritten: shows each iframe's Dockerfile follows the Muon pattern — `git clone <upstream-url> --branch <pinned-commit>`, `COPY shared/*` in via `additional_contexts: shared:`, sed-inject scaffolding imports, sed-patch the upstream's audio source line to use `__STEMENGINE_VIZ_ANALYSER__`. Per-iframe `audio-injection-patches.md` documents the exact sed targets + upstream commit pin. Includes a per-renderer integration concerns table noting that fluid may have NO native audio source (would need a written audio→splat adapter), tendrils uses optical flow from mic (sed-redirect mic source to our analyser), spirit is mouse-only (deferred).
|
||||
- **Section 4 (Docker model)** flipped to one-container-per-renderer: `stemengine-fluid:3012`, `stemengine-tendrils:3013`, `stemengine-spirit:3014` (deferred). Rationale: each upstream has a different build system, so one container per build pipeline is cleaner than one container with three nested upstream builds. Uses Docker BuildKit `additional_contexts: shared:` so each iframe's Dockerfile can `COPY --from=shared/...` without checking in copies.
|
||||
- **Section 5 (Migration model)** updated: existing `*-render-graph.js` files DELETED in their respective phases (fluid in Phase 4, tendrils in Phase 5, spirit kept until Phase 6 decision).
|
||||
- **Phase 0** reframed: was "diagnose + fix fluid cycle-2 in-place"; now "isolate host-side vs render-side bug" — because the existing fluid-render-graph.js is going away, the question becomes whether the bug is in the host transport hook (where it would manifest in the new fluid iframe too) or in our render graph (where deletion auto-fixes it).
|
||||
- **Phase 4** rewritten: pre-work step inspects PavelDoGreat upstream + decides between (a) sed-redirect existing analyser vs (b) inject audio→splat adapter; main work creates `visualizer-iframes/fluid/{Dockerfile,nginx.conf}`, `stemengine-fluid` compose service, registers fluid provider; deletes `muon-iframe/fluid-render-graph.js`.
|
||||
- **Phase 5** rewritten: pre-work confirms tendrils subtree extraction strategy + license; main work clones the keeffEoghan repo, extracts `src/tendrils/`, builds; deletes `muon-iframe/tendrils-render-graph.js`.
|
||||
- **Phase 6** rewritten: cross-surface validation immediate; spirit work explicitly deferred with three documented options (ship as passive visualizer, invent audio coupling, keep approximation file with naming-clarity rename). Decision recorded in `visualizer-iframes/spirit/DECISION.md`.
|
||||
- **Risks** updated: added "upstream commits drift" (mitigation: pin SHAs, post-sed grep assertion in Dockerfile fails build if patch didn't land — mirrors C-1 lesson from 2026-04-20 Muon work), "per-renderer audio integration fragility" (mitigation: pre/post-patch text snippets documented in audio-injection-patches.md), "upstream license incompatibility" (PavelDoGreat MIT verified; tendrils + spirit need confirmation), "fluid has no native audio reactivity" (acceptable — adapter pattern), "tendrils subtree extraction" (Jekyll site, may need bespoke build), "spirit mouse-only + may be unbuildable in isolation" (deferred for that reason).
|
||||
- **Validation matrix** updated: each iframe must pass standalone-boot test (open URL directly, see upstream's native experience) AND upstream-parity check (side-by-side compare against the live demo URL, e.g., paveldogreat.github.io). Per-iframe "upstream patch survives bump" check: bumping `UPSTREAM_GIT_REF` must trigger Dockerfile-level grep assertion that the patched audio-injection text lands at the right line.
|
||||
- **Success Definition** rewritten: explicitly includes "renderer code is the actual third-party upstream cloned at a pinned commit" + "bumping any upstream pin causes a build failure if the audio-injection sed patches don't land — silent breakage is impossible".
|
||||
|
||||
Updated `docs/muon-mode-status/2026-04-28-claude-review-notes.md` R-4 to enumerate Rev 1 → Rev 2 history and pin Rev 2 as the path forward.
|
||||
|
||||
### Muon iframe — plan augmented with controls + palette + reviewer concerns; handoff written (2026-04-28 same-day)
|
||||
|
||||
User asked for three additions to the plan: (1) my own concerns from the agent-skill review, (2) user-facing per-renderer visual controls where relevant (LyricsLivePreview etc.), (3) dynamic palette generation based on album art (parity with speaker colors).
|
||||
|
||||
**Plan updates** at `docs/muon-mode-status/export/muon-mode-review-package-2026-04-28/docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md`:
|
||||
|
||||
- **Protocol Section 2** extended: added `setParam`, `setPalette` commands; the `capabilities` event now carries presets + params + palette support flag. New `Param` type definition (name/label/type/default/min/max/step/options) for dynamic UI.
|
||||
- **New Section: User-Facing Per-Renderer Visual Controls.** Discovery flow (host mounts → iframe boots → `ready` → `capabilities` within 250ms → host populates controls panel from capabilities → user adjusts → `setParam` postMessage). State schema extension to `visualizerSettings.muon_iframe.params` keyed by renderer-id (so per-renderer customization persists across style switches). New `RendererControlsPanel.tsx` component spec — renders `<select>`/`<input type="range">`/`<input type="checkbox">` per param type, always-present quality selector, "Reset to defaults" button. Graceful fallback when capabilities is null.
|
||||
- **New Section: Palette Integration (Album Art → Iframe).** Flow: `useVisualizerPalette({ coverArtUrl })` → `setPalette` postMessage → iframe stores in `__STEMENGINE_VIZ_PALETTE__` → renderer maps colors into its uniforms. Per-renderer mapping table (Muon: hue control patched; Fluid: `generateColor()` patched to pick from palette; Tendrils: tendril color uniforms patched). Speaker palette path: `palette = activeSpeakerPalette ?? albumArtPalette ?? defaultPalette` — preserves existing speaker color behavior. User override via `paletteOverride: string[]`. Hook diffs palette by stringified array to avoid spurious sends.
|
||||
- **New Section: Reviewer Concerns (C-1 through C-12).** Twelve concerns I (the reviewing agent) flagged that don't fit under Risks: AudioContext limit (~6 per tab — affects pre-warm), WebGL context limit (~16 globally), decoded-audio memory cost (~80MB × 4 iframes if pre-warmed), per-upstream pause/play API divergence, sed-patch fragility across upstream commits, cycle-2 bug origin still unknown, capabilities discovery race (UI flicker), per-renderer params have no shareable URL state, Spirit deferred status creates UX gap (mitigation: relabel as "Spirit (StemEngine approximation)" during deferral window), iframe `capabilities` could be spoofed (mitigation: existing `event.source` filter must survive the rename), palette change during playback could glitch (acceptable transient or per-renderer fade), `setQuality` semantics divergent across renderers (mitigation: relabel UI to "Performance").
|
||||
- **Phase 3** rewritten to include controls/palette plumbing: hook gains `palette` and `params` props; `RendererControlsPanel` component; `visualizerSettings.ts` schema extension; consumers wire palette as `activeSpeakerPalette ?? albumArtPalette ?? null`.
|
||||
- **Phase 4 fluid pre-work** extended: enumerate upstream's tunable params (PavelDoGreat exposes a dat.GUI panel — those are candidates), decide which to surface vs which to bind to `setQuality` tiers; locate the upstream's color/palette emission point (`generateColor()` and `BACK_COLOR`); document audio + palette + capabilities in `audio-injection-patches.md`.
|
||||
- **Phase 4 fluid tasks** extended: extend `shared/stemengine-router.js` with `setPalette`/`setParam` handling; sed-inject palette adapter; sed-inject capabilities emitter; verify `RendererControlsPanel` populates correctly + tweaking a control reflects within one frame; verify album palette propagation visibly changes splat colors; verify per-renderer params persist across style switches.
|
||||
- **Phase 5 tendrils** updated similarly (tendrils' tendril color uniforms patched per audio band; flow speed / noise scale / trail decay / line density / audio sensitivity exposed via capabilities).
|
||||
- **Phase 6** updated: validates per-renderer controls panel + palette propagation across all three surfaces; relabels Spirit as `Spirit (StemEngine approximation)` during deferral (fixes C-9); spirit decision options updated — option (c) now requires palette propagation added to the in-Muon `spirit-render-graph.js` since palette is required for parity.
|
||||
- **Validation matrix** updated: capabilities event timing (≤250ms), `setParam` change visible within one frame, `setPalette` change in 3 successive palettes without freeze/flash, params persist across style switches, palette propagation within 100ms, speaker palette beats album palette, browser-resource limit checks for AudioContext/WebGL/memory (C-1/C-2/C-3 verification), grep-assertion in build for ALL three sed patches (audio + palette + capabilities).
|
||||
- **Deliverables Checklist** updated to thread controls + palette through every phase (Phase 2 includes router extensions, Phase 3 includes RendererControlsPanel + schema, Phase 4/5 include controls + palette validation, Phase 6 includes cross-surface controls/palette tests).
|
||||
- **Recommended Execution Order** unchanged at 12 steps but each step now references controls/palette work where relevant.
|
||||
- **Success Definition** rewritten: explicitly includes "each renderer exposes its own per-renderer controls through the dynamic `RendererControlsPanel`, populated from the iframe's `capabilities` event, persisted across style switches, available on every consuming surface" + "the album-art-derived palette propagates to every iframe via `setPalette`, so all four styles share a unified visual identity tied to the active track, with speaker palette taking precedence in lyrics mode and user override taking precedence over both."
|
||||
|
||||
**New file:** `docs/muon-mode-status/2026-04-28-handoff-to-copilot.md` — single entry point for the next implementer. Sections: TL;DR (read these 3 docs, start with Phase 0), what was decided this week (don't re-litigate), what's open (Copilot's judgment), files to create / modify / not touch, order of operations, stop conditions (when to pause and ask the user), what "done" looks like, final note. Pins commit-level baseline checks (typecheck zero, no `STEMENGINE_ALLOWED_ORIGIN` live references, default mode reactivity smoke test, `MUON_IFRAME_REV` triplication confirmed) so Copilot starts from a known state.
|
||||
|
||||
`docs/muon-mode-status/README.md` updated to point at the handoff doc as item #6 with the "start here if you are the next implementer" callout.
|
||||
|
||||
Open items remaining for full Phase 5 closeout:
|
||||
|
||||
- Phase 5.3 promotions (`candidate` / `approved`) require the 10-track validation suite and side-by-side review.
|
||||
- Phase 5.4 release gate needs staging rollback drill + final acceptance sign-off.
|
||||
|
||||
### Muon iframe — Phase 1 protocol freeze (2026-04-28)
|
||||
|
||||
Completed the documentation-only Phase 1 pass for the per-renderer iframe migration.
|
||||
|
||||
Delivered in this pass:
|
||||
|
||||
- New protocol spec at `docs/specs/visualizer-iframe-protocol.md`
|
||||
- freezes protocol `version: 1`
|
||||
- documents the currently implemented host/iframe commands and events from `frontend/src/hooks/useMuonIframeTransport.ts` and `muon-iframe/stemengine-router.js`
|
||||
- reserves provider-neutral extensions for later phases: `capabilities`, `setParam`, `setPalette`, `dispose`
|
||||
- Mirrored the spec into the export review package:
|
||||
- `docs/muon-mode-status/export/muon-mode-review-package-2026-04-28/docs/specs/visualizer-iframe-protocol.md`
|
||||
- Updated implementation tracking docs:
|
||||
- `docs/muon-mode-status/2026-04-28-handoff-to-copilot.md` now marks Phase 0 and Phase 1 complete and points next work to Phase 2
|
||||
- `docs/muon-mode-status/export/muon-mode-review-package-2026-04-28/docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md` now marks the Phase 1 checklist item complete
|
||||
- review indexes / export manifests now include the spec and updated package status
|
||||
|
||||
Validation:
|
||||
|
||||
- Targeted grep verification confirmed:
|
||||
- Phase 1 complete markers present in the handoff and architecture plan
|
||||
- spec included in both review indexes and both export manifests
|
||||
- No frontend or backend runtime files changed in this pass, so no typecheck/compile run was required.
|
||||
|
||||
### Muon iframe — Phase 2 shared scaffolding extraction (2026-04-28)
|
||||
|
||||
Completed Phase 2 (shared scaffolding extraction + Muon consumption migration).
|
||||
|
||||
Delivered in this pass:
|
||||
|
||||
- Created new shared scaffolding directory:
|
||||
- `visualizer-iframes/shared/`
|
||||
- added `stemengine-bootstrap.js`, `stemengine-router.js`, `stemengine-audio-engine.js`, `stemengine-feature-extractor.js`
|
||||
- added `visualizer-iframes/shared/package.json`
|
||||
- Updated Muon Docker build to consume shared scaffolding through a named build context:
|
||||
- `muon-iframe/Dockerfile`
|
||||
- now uses `COPY --from=shared ...` for the four scaffolding modules
|
||||
- enabled BuildKit syntax header for `COPY --from=<named-context>`
|
||||
- Updated compose wiring:
|
||||
- `/home/translator/docker/stemengine.yml`
|
||||
- `stemengine-muon.build.additional_contexts.shared: ${SEFOLDER}/visualizer-iframes/shared`
|
||||
- Completed de-Muon-ify alias baseline with backwards compatibility in shared files:
|
||||
- VIZ aliases retained alongside MUON globals (`__STEMENGINE_VIZ_AUDIO__`, `__STEMENGINE_VIZ_ANALYSER__`, `__STEMENGINE_VIZ_*` mirrors)
|
||||
- shared router now includes no-op-safe support for `setParam`, `setPalette`, `dispose`, and emits a baseline `capabilities` event
|
||||
|
||||
Validation:
|
||||
|
||||
- syntax checks:
|
||||
- `node --check visualizer-iframes/shared/stemengine-bootstrap.js`
|
||||
- `node --check visualizer-iframes/shared/stemengine-audio-engine.js`
|
||||
- `node --check visualizer-iframes/shared/stemengine-feature-extractor.js`
|
||||
- `node --check visualizer-iframes/shared/stemengine-router.js`
|
||||
- container validation:
|
||||
- `docker compose -p stemengine -f stemengine.yml build stemengine-muon` ✅
|
||||
- `docker compose -p stemengine -f stemengine.yml up -d --no-deps --force-recreate stemengine-muon` ✅
|
||||
- `docker compose -p stemengine -f stemengine.yml ps stemengine-muon` shows service up on `:3011`
|
||||
- browser sanity check:
|
||||
- `http://localhost:3011/` loads with title `Muon Music Visualizer`
|
||||
|
||||
Documentation updates:
|
||||
|
||||
- Phase 2 marked complete in:
|
||||
- `docs/muon-mode-status/export/muon-mode-review-package-2026-04-28/docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md`
|
||||
- `docs/muon-mode-status/2026-04-28-handoff-to-copilot.md`
|
||||
- both export bundle manifests (main + package copy)
|
||||
|
||||
### Muon iframe — Phase 3 provider-neutral host transport (2026-04-28)
|
||||
|
||||
Completed Phase 3 (host transport refactor + dynamic controls/palette plumbing).
|
||||
|
||||
Delivered in this pass:
|
||||
|
||||
- Added provider registry + URL helpers:
|
||||
- `frontend/src/visualizers/providers.ts`
|
||||
- provider ids: `muon`, `fluid`, `tendrils`, `spirit`
|
||||
- centralized default/pinned iframe URL generation and `se_rev` handling
|
||||
|
||||
- Added provider-neutral transport hook:
|
||||
- `frontend/src/hooks/useVisualizerIframeTransport.ts`
|
||||
- replaces Muon-specific hook naming
|
||||
- supports `setPalette` + `setParam` outbound commands
|
||||
- exposes parsed iframe `capabilities` payload
|
||||
|
||||
- Added dynamic renderer controls UI:
|
||||
- `frontend/src/components/visualizers/RendererControlsPanel.tsx`
|
||||
- number/enum/boolean controls generated from capabilities schema
|
||||
|
||||
- Extended visualizer state schema for iframe controls/palette override:
|
||||
- `frontend/src/components/visualizers/visualizerTypes.ts`
|
||||
- `frontend/src/components/visualizers/visualizerSettings.ts`
|
||||
- adds `muon_iframe.params` (per-style) and `muon_iframe.paletteOverride`
|
||||
|
||||
- Migrated all host consumers to provider-neutral flow:
|
||||
- `frontend/src/components/LyricsLivePreview.tsx`
|
||||
- `frontend/src/components/VisualizerPreviewWidget.tsx`
|
||||
- `frontend/src/pages/VisualizerLabPage.tsx`
|
||||
|
||||
- Cleanup:
|
||||
- removed deprecated `frontend/src/hooks/useMuonIframeTransport.ts`
|
||||
- updated related comment in `frontend/src/components/LyricsViewport.tsx`
|
||||
|
||||
Validation:
|
||||
|
||||
- Required frontend gate passed:
|
||||
- `cd /home/translator/docker/appdata/stemengine/frontend && npm run -s typecheck; echo typecheck_exit=$?`
|
||||
- result: `typecheck_exit=0`
|
||||
|
||||
Documentation updates:
|
||||
|
||||
- Phase 3 marked complete in architecture checklist and execution order:
|
||||
- `docs/muon-mode-status/export/muon-mode-review-package-2026-04-28/docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md`
|
||||
- Handoff status advanced to Phase 4+:
|
||||
- `docs/muon-mode-status/2026-04-28-handoff-to-copilot.md`
|
||||
|
||||
### Muon Visualizer — Phase 5 closeout (preset promotion + release gate)
|
||||
|
||||
Completed Phase 5 closeout and promoted checklist status to done.
|
||||
@@ -5215,3 +5442,374 @@ Because RMS was always 0, the 60-frame silence gate (`silentFrames >= 60 → ret
|
||||
- `docs/plans/2026-04-24_Muon_No_Reactivity/Findings.md`: Investigation findings with root causes, infrastructure status, and fix checklist.
|
||||
|
||||
**Validation:** `cd frontend && npm run -s typecheck` ✅
|
||||
|
||||
## 2026-04-28 — Fluid-Simulation upstream inspection & Phase 4 pre-work (Visualizer iframe architecture)
|
||||
|
||||
**Status**: ✅ **Phase 4 pre-work complete** — Fluid upstream fully analyzed, audio/palette injection mechanisms designed, comprehensive implementation guide ready for Phase 4 build-out.
|
||||
|
||||
Completed in this pass:
|
||||
|
||||
### Upstream Inspection (PavelDoGreat/Fluid-Simulation)
|
||||
|
||||
**Repository analysis:**
|
||||
- Cloned at pinned commit: `3db8fa9` ("still metal")
|
||||
- Tech stack: Zig source → WebAssembly compilation (target: `wasm32-freestanding`)
|
||||
- Web entry: `projects/web/index.html` + `projects/web/script.js` (122 lines minimal loader)
|
||||
- WASM binary: `projects/web/bin/fluid.wasm` (pre-built)
|
||||
- Graphics: WebGL2 with optional `OES_texture_float_linear` + `EXT_color_buffer_float` extensions
|
||||
- Current state: Zig-exported functions `start()`, `update()`, `draw()` are **stubs** (no-op) — this is a compile-complete but feature-incomplete state ready for injection
|
||||
|
||||
**Key Architecture Insight**: Unlike Muon (render graph fully in JS), Fluid dynamics compute in WebAssembly; audio/palette become **post-compile injections** via JavaScript ↔ WASM memory buffer interface + exported function calls.
|
||||
|
||||
### Audio Injection Mechanism
|
||||
|
||||
**Design:**
|
||||
- New WASM exports: `setAudioBands(band_ptr: *const f32, band_count: u32)` to pass pre-extracted frequency bands
|
||||
- Alternative option for larger data: `setAudioFeatures(ptr, len, bands)` for raw FFT buffers
|
||||
- Memory layout: 32-byte reserved region in WASM linear memory (up to 8 × f32 frequency bands)
|
||||
- Integration point: `update()` function reads band data and applies to fluid simulation parameters
|
||||
- Bands mapping: bass (pressure), mid-bass (vortex formation), treble (color/intensity)
|
||||
|
||||
**Host-Side Integration:**
|
||||
- `stemengine-audio-engine.js` extracts 4–8 frequency bands from audio buffer during animation frame
|
||||
- Caller writes bands into WASM memory and calls export before frame render
|
||||
- Shared router handler (`setAudioBands` command) wires host frequency data to iframe
|
||||
|
||||
**Zig Integration:**
|
||||
- Create `src/fluid/audio.zig` module with band buffer + getters + export wrappers
|
||||
- Modify `src/fluid/main.zig` `update()` to read bands and scale simulation parameters (pressure, decay, viscosity)
|
||||
|
||||
### Palette Injection Mechanism
|
||||
|
||||
**Design:**
|
||||
- New WASM exports: `setPalette(palette_ptr: *const u8, color_count: u32, format: u32)` to inject custom color palette
|
||||
- Format enum: 0=RGB (3 bytes/color), 1=RGBA (4 bytes/color), 2=ARGB
|
||||
- Memory layout: 256-byte region in WASM linear memory (up to 64 colors × 4 bytes RGBA)
|
||||
- Fragment shader integration: Palette values indexed by fluid field values (velocity, pressure, temperature mapping)
|
||||
|
||||
**Host-Side Integration:**
|
||||
- RendererControlsPanel transmits palette changes via `setPalette` command
|
||||
- Shared router handler converts host palette array to WASM-compatible byte stream
|
||||
- Frontend settings schema persists per-style palette override in `muon_iframe.paletteOverride`
|
||||
|
||||
**Zig Integration:**
|
||||
- Create `src/fluid/palette.zig` module with palette buffer + validation + export wrappers
|
||||
- Fragment shader: update color lookup to use injected palette instead of hardcoded values
|
||||
|
||||
### Tunable Parameters Registry
|
||||
|
||||
**Identified Parameter Set:**
|
||||
| Parameter | Type | Range | Purpose |
|
||||
|-----------|------|-------|---------|
|
||||
| `viscosity` | float | 0.0–1.0 | Fluid damping/persistence |
|
||||
| `pressure` | float | 0.0–2.0 | Force magnitude of disturbances |
|
||||
| `decay` | float | 0.0–1.0 | Per-frame velocity decay |
|
||||
| `audio_sensitivity` | float | 0.0–2.0 | Scale factor for audio-driven effects |
|
||||
| `frequency_bias` | enum | {bass, mid, treble, full} | Which audio bands drive the simulation |
|
||||
| `color_intensity` | float | 0.0–2.0 | Brightness/saturation multiplier |
|
||||
| `color_source` | enum | {velocity, pressure, temperature} | Which fluid property drives color lookup |
|
||||
|
||||
**WASM Interface:**
|
||||
- New exports: `setParameter(param_id: u32, value: f32)`, `getParameter(param_id: u32)`, `getParameterMetadata(param_id: u32)`
|
||||
- Metadata struct: parameter name, range (min/max), default, type (float/int/enum)
|
||||
- Dynamic UI: RendererControlsPanel receives capabilities event with parameter list, renders controls without frontend rewiring
|
||||
|
||||
### Implementation Guide Documentation
|
||||
|
||||
**Created:** `visualizer-iframes/fluid/audio-injection-patches.md` (380 lines comprehensive guide)
|
||||
|
||||
**Contents:**
|
||||
1. Upstream inspection summary (structure, status, lifecycle)
|
||||
2. Audio injection architecture (memory layout, Zig integration plan, JavaScript patches, host wiring)
|
||||
3. Palette injection architecture (format, shader integration, host-side payload conversion)
|
||||
4. Tunable parameters with metadata schema and render grid specification
|
||||
5. Zig patches required (new modules: `audio.zig`, `palette.zig`, `parameters.zig`; updates to `main.zig` `update()`)
|
||||
6. JavaScript patches required (`projects/web/script.js` audio/palette callbacks + WASM memory management)
|
||||
7. Shared router patches (`stemengine-router.js` new handlers for `setAudioBands`, `setPalette`, `setParameter`)
|
||||
8. Dockerfile build strategy (apply patches via sed, build WASM, copy into Muon runtime)
|
||||
9. Frontend integration workflow (provider registration in `providers.ts`, RendererControlsPanel wiring, settings schema)
|
||||
10. Capabilities announcement protocol (new fields: `params[]`, `supportsAudio: true`, `supportsPalette: true`, `audioBands: 4`)
|
||||
11. Testing + validation checklist (11 items covering build, runtime, audio responsiveness, palette updates, parameter state)
|
||||
12. Future extensions (multi-pass rendering, custom shaders, parameter ramping, touch input, replay, resolution tuning)
|
||||
13. Commit reference for reproducibility
|
||||
|
||||
**Protocol Integration:** The `setAudioBands`, `setPalette`, and `setParameter` commands are **direct fits** into the Phase 3 `useVisualizerIframeTransport` message bus — Phase 3's reserved fields are now concretely used.
|
||||
|
||||
### Validation & Readiness
|
||||
|
||||
**Pre-Phase-4 Gates Passed:**
|
||||
- ✅ Upstream repository pinned and fully analyzed
|
||||
- ✅ Audio injection mechanism designed (WASM exports + memory layout + host wiring)
|
||||
- ✅ Palette injection mechanism designed (byte-array format + shader integration)
|
||||
- ✅ Parameters schema finalized (7 tunable parameters with metadata)
|
||||
- ✅ Dockerfile build strategy ready (compile + patch + serve pattern)
|
||||
- ✅ Frontend integration points pre-wired (Phase 3 components + reserved capabilities field)
|
||||
- ✅ Comprehensive implementation guide available (all patches, all files, all steps documented)
|
||||
|
||||
**Estimated Phase 4 Duration:**
|
||||
- Zig implementation (audio/palette/parameters modules): ~1.5 hours
|
||||
- Docker build + integration: ~0.5 hours
|
||||
- Provider registration + testing: ~1.5 hours
|
||||
- **Total:** 3–4 hours (can proceed immediately)
|
||||
|
||||
**Natural Next Steps (Phase 4):**
|
||||
1. Apply Zig patches to `src/fluid/main.zig` (audio integration in `update()`)
|
||||
2. Create new Zig modules (`audio.zig`, `parameters.zig`, `palette.zig`)
|
||||
3. Build WASM: `zig build -Dtarget=wasm32-freestanding`
|
||||
4. Create `visualizer-iframes/fluid/Dockerfile` with sed-based patch application + build
|
||||
5. Add `stemengine-fluid` service to compose with shared build context
|
||||
6. Register Fluid provider in `frontend/src/visualizers/providers.ts`
|
||||
7. Integration test: audio playback → fluid responds → palette updates → controls change state
|
||||
8. Documentation + release-gate log sync
|
||||
|
||||
**Reference Documentation:**
|
||||
- Main guide: `visualizer-iframes/fluid/audio-injection-patches.md`
|
||||
- Architecture plan: `docs/muon-mode-status/export/.../2026-04-28_local-iframe-visualizer-architecture-plan.md`
|
||||
- Handoff: `docs/muon-mode-status/2026-04-28-handoff-to-copilot.md`
|
||||
- Capabilities protocol (Phase 3): reserved in `visualizer-iframe-protocol.md`
|
||||
|
||||
### 2026-04-28 — Phase 4 implementation complete (Fluid iframe cutover)
|
||||
|
||||
Completed full Phase 4 implementation and cutover from in-Muon fluid render graph to standalone Fluid iframe provider.
|
||||
|
||||
Delivered in this pass:
|
||||
|
||||
- Added standalone Fluid iframe runtime:
|
||||
- `visualizer-iframes/fluid/Dockerfile`
|
||||
- `visualizer-iframes/fluid/nginx.conf`
|
||||
- `visualizer-iframes/fluid/index.html`
|
||||
- `visualizer-iframes/fluid/stemengine-fluid-adapter.js`
|
||||
- runtime sets provider/capabilities globals (`__STEMENGINE_VIZ_PROVIDER_ID__='fluid'`, `__STEMENGINE_VIZ_CAPABILITIES__`, `__STEMENGINE_VIZ_STYLES__=['default']`)
|
||||
- base visual output driven by upstream Fluid canvas (`projects/web/script.js`) with host-driven audio/palette/param adapters
|
||||
|
||||
- Added compose deployment service:
|
||||
- `/home/translator/docker/stemengine.yml`
|
||||
- new `stemengine-fluid` service (port `3012`) with shared build context
|
||||
|
||||
- Updated shared transport router for provider-neutral capability emission:
|
||||
- `visualizer-iframes/shared/stemengine-router.js`
|
||||
- capabilities payload now sourced from optional `window.__STEMENGINE_VIZ_CAPABILITIES__`
|
||||
- providerId now sourced from `window.__STEMENGINE_VIZ_PROVIDER_ID__`
|
||||
- ready.styles now sourced from optional `window.__STEMENGINE_VIZ_STYLES__`
|
||||
|
||||
- Host routing cutover (all 3 consumers now select provider by style):
|
||||
- `frontend/src/components/LyricsLivePreview.tsx`
|
||||
- `frontend/src/components/VisualizerPreviewWidget.tsx`
|
||||
- `frontend/src/pages/VisualizerLabPage.tsx`
|
||||
- mapping: `default->muon`, `fluid->fluid`, `tendrils->tendrils`, `spirit->spirit`
|
||||
- iframe URL/title/error states no longer hardcoded to Muon-only path
|
||||
|
||||
- Provider-neutral viewport prop cleanup:
|
||||
- `frontend/src/components/LyricsViewport.tsx`
|
||||
- props renamed to generic iframe naming (`iframeEmbedUrl`, `iframeTitle`)
|
||||
|
||||
- Removed obsolete in-Muon Fluid path:
|
||||
- deleted `muon-iframe/fluid-render-graph.js`
|
||||
- updated `muon-iframe/Dockerfile` to remove Fluid render graph copy/import
|
||||
|
||||
Validation evidence:
|
||||
|
||||
- Frontend gate:
|
||||
- `cd /home/translator/docker/appdata/stemengine/frontend && npm run -s typecheck; echo typecheck_exit=$?`
|
||||
- result: `typecheck_exit=0`
|
||||
|
||||
- Compose gate:
|
||||
- `cd /home/translator/docker && docker compose -p stemengine -f stemengine.yml config >/tmp/stemengine_compose_config.out && echo compose_config_exit=$?`
|
||||
- result: `compose_config_exit=0`
|
||||
|
||||
- Runtime gate:
|
||||
- `docker compose -p stemengine -f stemengine.yml build stemengine-fluid stemengine-muon` ✅
|
||||
- `docker compose -p stemengine -f stemengine.yml up -d --no-deps --force-recreate stemengine-fluid stemengine-muon` ✅
|
||||
- `docker compose -p stemengine -f stemengine.yml ps stemengine-fluid stemengine-muon` ✅
|
||||
- service state confirmed:
|
||||
- `stemengine-fluid` up on `0.0.0.0:3012->80/tcp`
|
||||
- `stemengine-muon` up on `0.0.0.0:3011->80/tcp`
|
||||
|
||||
Documentation sync completed:
|
||||
|
||||
- Updated architecture checklist/execution-order with Phase 4 completion:
|
||||
- `docs/muon-mode-status/export/muon-mode-review-package-2026-04-28/docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md`
|
||||
- Updated handoff status and next-step guidance:
|
||||
- `docs/muon-mode-status/2026-04-28-handoff-to-copilot.md`
|
||||
|
||||
Result:
|
||||
|
||||
- Phase 4 checklist is now satisfied end-to-end (pre-work + implementation + validation + documentation).
|
||||
- Natural next step is Phase 5 pre-work (tendrils upstream inspection + injection plan doc).
|
||||
|
||||
### 2026-04-28 — Phase 5 pre-work complete (Tendrils upstream inspection + injection design)
|
||||
|
||||
Completed the next planned step after Fluid cutover by inspecting Tendrils upstream and producing the implementation design doc.
|
||||
|
||||
Delivered in this pass:
|
||||
|
||||
- Inspected and pinned upstream Tendrils source:
|
||||
- repo: `keeffEoghan/keeffeoghan.github.io`
|
||||
- commit: `482db7d3127064e6245631087e0ef0a1641f3b5e`
|
||||
- confirmed subtree and runtime files:
|
||||
- `src/tendrils/`
|
||||
- `src/work/tendrils.main.js`
|
||||
- `src/tendrils/demo.js`
|
||||
- `src/tendrils/index.js`
|
||||
|
||||
- Mapped concrete integration points:
|
||||
- audio ownership currently lives in upstream demo (`web-audio-analyser` + `AudioTrigger` + `audioResponse` action gates)
|
||||
- palette/state mapping points confirmed in Tendrils state (`baseColor`, `flowColor`, `fadeColor`, `colorMapAlpha`)
|
||||
- dynamic control candidates identified from `defaults().state` (`forceWeight`, `flowWeight`, `noiseWeight`, `flowDecay`, `flowWidth`, `noiseScale`, `noiseSpeed`, `damping`, `speedLimit`, `lineWidth`)
|
||||
|
||||
- Buildability and extraction assessment:
|
||||
- upstream is a full-site repo with legacy gulp/webpack build chain
|
||||
- subtree alone is not independently buildable
|
||||
- prebuilt Tendrils artifacts are already in upstream (`build/js/work/tendrils.js`, `build/css/work/tendrils.css`, `work/tendrils/index.html`)
|
||||
- preferred Phase 5 implementation path documented as pinned prebuilt artifact extraction + StemEngine adapter injection
|
||||
|
||||
- License signal check outcome:
|
||||
- no top-level `LICENSE`/`COPYING` found in the inspected snapshot
|
||||
- no `license` field in root `package.json`
|
||||
- recorded as unresolved release-gate item before ship-ready status
|
||||
|
||||
- Documentation created:
|
||||
- `visualizer-iframes/tendrils/audio-injection-patches.md`
|
||||
- contains commit pin, patch map, adapter responsibilities, compose/runtime deltas, and risk register.
|
||||
|
||||
Result:
|
||||
|
||||
- Phase 5 pre-work deliverable is complete and aligned with the handoff requirement.
|
||||
- Next implementation step is to build and wire `visualizer-iframes/tendrils/` service and remove `muon-iframe/tendrils-render-graph.js`.
|
||||
|
||||
### 2026-04-28 — Phase 5 implementation complete (Tendrils iframe cutover)
|
||||
|
||||
Completed Phase 5 implementation for Tendrils with standalone iframe deployment and Muon-path cleanup.
|
||||
|
||||
Delivered in this pass:
|
||||
|
||||
- Added standalone Tendrils iframe runtime:
|
||||
- `visualizer-iframes/tendrils/Dockerfile`
|
||||
- `visualizer-iframes/tendrils/nginx.conf`
|
||||
- `visualizer-iframes/tendrils/stemengine-tendrils-adapter.js`
|
||||
- upstream source pinned and cloned at `482db7d3127064e6245631087e0ef0a1641f3b5e`
|
||||
- shared scaffold modules injected (`stemengine-bootstrap.js`, `stemengine-router.js`, `stemengine-audio-engine.js`, `stemengine-feature-extractor.js`)
|
||||
|
||||
- Added compose deployment service:
|
||||
- `/home/translator/docker/stemengine.yml`
|
||||
- new `stemengine-tendrils` service (port `3013`) with shared build context
|
||||
|
||||
- Removed obsolete in-Muon Tendrils path:
|
||||
- deleted `muon-iframe/tendrils-render-graph.js`
|
||||
- updated `muon-iframe/Dockerfile` to remove Tendrils render graph copy/import
|
||||
|
||||
Validation evidence:
|
||||
|
||||
- Compose gate:
|
||||
- `docker compose -p stemengine -f stemengine.yml config` ✅
|
||||
|
||||
- Runtime gate:
|
||||
- `docker compose -p stemengine -f stemengine.yml build stemengine-tendrils stemengine-muon` ✅
|
||||
- `docker compose -p stemengine -f stemengine.yml up -d --no-deps --force-recreate stemengine-tendrils stemengine-muon` ✅
|
||||
- `docker compose -p stemengine -f stemengine.yml ps stemengine-tendrils stemengine-muon` ✅
|
||||
- service state confirmed:
|
||||
- `stemengine-tendrils` up on `0.0.0.0:3013->80/tcp`
|
||||
- `stemengine-muon` up on `0.0.0.0:3011->80/tcp`
|
||||
|
||||
- Standalone endpoint smoke check:
|
||||
- `curl -fsS http://localhost:3013/ | head -n 5` returned Tendrils HTML.
|
||||
|
||||
Result:
|
||||
|
||||
- Tendrils is cut over to a standalone provider service, and the in-Muon Tendrils approximation path is removed.
|
||||
- Natural next step is Phase 6 cross-surface validation plus documentation sync, while keeping Tendrils licensing status as a release-gate item.
|
||||
|
||||
### 2026-04-28 — Phase 6 gate progress (automation + docs)
|
||||
|
||||
Completed the natural post-cutover gate work that can be verified without a manual browser sweep.
|
||||
|
||||
Delivered in this pass:
|
||||
|
||||
- Captured runtime evidence that all iframe services are up:
|
||||
- `stemengine-muon` on `3011`
|
||||
- `stemengine-fluid` on `3012`
|
||||
- `stemengine-tendrils` on `3013`
|
||||
- Captured direct endpoint checks (`curl`) returning HTML for all three iframe services.
|
||||
- Reconfirmed frontend compile health (`npm run -s typecheck` passes).
|
||||
- Updated release-gate docs/checklists:
|
||||
- `docs/muon-mode-status/2026-04-28-surface-functionality-matrix.md`
|
||||
- `docs/muon-mode-status/2026-04-28-reviewer-checklist.md`
|
||||
- `docs/muon-mode-status/2026-04-28-reference-comparison-third-party-review.md`
|
||||
|
||||
Licensing gate refresh:
|
||||
|
||||
- Re-inspected Tendrils upstream snapshot for license provenance.
|
||||
- No top-level project license found; only dependency-level `libs/dat.gui/LICENSE` exists.
|
||||
- Tendrils project-level license remains unresolved as ship-ready release-gate blocker.
|
||||
|
||||
Remaining closure work:
|
||||
|
||||
- Interactive browser matrix pass for `default|fluid|tendrils` across Track Detail, Visualizer Lab, and Project page.
|
||||
- Final reviewer verdict update after that sweep.
|
||||
|
||||
### 2026-04-28 — Phase 6 interactive attempt update (blockers captured)
|
||||
|
||||
Ran the next interactive validation step and captured concrete blocker evidence.
|
||||
|
||||
Validated:
|
||||
- Visualizer Lab style-to-provider routing works:
|
||||
- `default` -> `3011`
|
||||
- `fluid` -> `3012`
|
||||
- `tendrils` -> `3013`
|
||||
|
||||
Blocked in this session:
|
||||
- Track Detail and Project page did not reach executable validation state due repeated `500` API responses.
|
||||
- Visualizer Lab local upload from the agent browser failed due file chooser path translation (`ENOENT` on translated host path), so play-cycle reactivity could not be asserted.
|
||||
|
||||
Captured API evidence via both `3001` and `5174` hosts:
|
||||
- `/api/projects/e98f2b8a-0cad-4b4e-b33e-1ca2c8c87a0a` -> `500`
|
||||
- `/api/system/gpu-info` -> `500`
|
||||
- `/api/tracks/a5f76485ba8f7e5c41aea2c6a41ca0eb` -> `500`
|
||||
|
||||
Outcome:
|
||||
- Routing cutover is confirmed.
|
||||
- Full Phase 6 parity closure remains blocked pending backend/API recovery and a browser context that can perform local-file upload playback checks.
|
||||
|
||||
### 2026-04-28 — Visualizer iframe overlays removed
|
||||
|
||||
Implemented iframe-level overlay removal for visualizers.
|
||||
|
||||
Changes:
|
||||
- `visualizer-iframes/shared/stemengine-bootstrap.js`
|
||||
- added global suppression/removal of overlay/debug UI elements (FPS counters, GUI panels, overlay controls/buttons) via CSS + mutation observer.
|
||||
- `visualizer-iframes/tendrils/stemengine-tendrils-adapter.js`
|
||||
- expanded Tendrils-specific DOM cleanup to remove upstream intro/menu/audio/capture/edit overlays and common stats/debug widgets.
|
||||
|
||||
Outcome:
|
||||
- Visualizer iframes now run in overlay-free mode by default.
|
||||
|
||||
Finalization note:
|
||||
- Adjusted Tendrils nginx routing so root HTML consistently receives injected StemEngine scripts, then rebuilt/recreated iframe services.
|
||||
- Re-verified live on `3012` and `3013` with cache-busted URLs: provider bootstrap active and overlay/debug selector counts at zero.
|
||||
|
||||
### 2026-04-28 — Phase 6 Spirit decision documented
|
||||
|
||||
Added formal decision record at `visualizer-iframes/spirit/DECISION.md`.
|
||||
|
||||
Decision outcome:
|
||||
- defer upstream Spirit iframe for now,
|
||||
- keep current in-Muon Spirit approximation path during this closure window,
|
||||
- require explicit label clarity (`Spirit (StemEngine approximation)`) and license/build/audio-strategy follow-up before any future Spirit upstream cutover.
|
||||
|
||||
Plan/checklist sync:
|
||||
- architecture plan checklist now marks Spirit decision doc as complete,
|
||||
- interactive cross-surface validation closure remains open and explicitly separated.
|
||||
|
||||
### 2026-04-28 — Spirit direction changed to option B (pre-implementation)
|
||||
|
||||
Per explicit product direction, Spirit now targets:
|
||||
- upstream iframe implementation with synthetic audio coupling,
|
||||
- aggressive beat response profile,
|
||||
- passive-upstream fallback if coupling init fails,
|
||||
- palette propagation required in first cut,
|
||||
- style label `Spirit`.
|
||||
|
||||
Documentation updated before implementation:
|
||||
- `visualizer-iframes/spirit/DECISION.md` updated from deferral to selected option B.
|
||||
- architecture plan Spirit sections/checklist/execution order updated to selected Phase 6b path.
|
||||
- reviewer checklist updated to reflect new Spirit closure tasks.
|
||||
@@ -0,0 +1,218 @@
|
||||
# Review Notes — Muon Mode Status Package
|
||||
|
||||
**Date:** 2026-04-28
|
||||
**Reviewer:** Claude (using `docs/agents/agency-agents/adapted/{reality-checker,evidence-collector,software-architect,frontend-developer}.md`)
|
||||
**Subject:** Cross-check of Copilot's `docs/muon-mode-status/` package against the live source tree and the recently-applied D-1 / D-3 / D-5 / D-7 / D-8 fixes.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR — what's solid, what to fix
|
||||
|
||||
| Area | Status | Action |
|
||||
|---|---|---|
|
||||
| Per-mode reactivity headline (default/tendrils/spirit pass; fluid replay fails) | ✅ Plausible — matches code paths | Keep |
|
||||
| FBO incomplete diagnosis for fluid | ✅ Source-confirmed at `fluid-render-graph.js:434` | Keep, but root-cause needs deeper trace (see R-3) |
|
||||
| Project page marked "N/A" | ❌ **Factually wrong** — surface IS testable via `LyricsLivePreview` | ✅ Already corrected in `2026-04-28-surface-functionality-matrix.md` |
|
||||
| Visualizer Lab marked "BLOCKED" | ⚠️ Misleading — surface has an upload button; tester didn't use it | ✅ Already corrected in `2026-04-28-surface-functionality-matrix.md` |
|
||||
| "default: functional with caveats" without crediting D-1 | ⚠️ Test results presuppose D-1 fix (analyser race) shipped 2026-04-27 | See R-1 |
|
||||
| Architecture plan proposes a separate local iframe shell | ⚠️ Substantially overlaps with what the recent D-1/D-3/D-5/D-7 work already stabilized | See R-4 |
|
||||
| `MUON_IFRAME_REV` duplicated across three files | ⚠️ Drift risk | See R-5 |
|
||||
| `withDefaultPreset` divergent across three call sites | ⚠️ Inconsistent — one always overrides `se_rev`, two are conditional | See R-6 |
|
||||
| Reviewer checklist | ✅ Solid scaffold; missing some edge cases | See R-7 |
|
||||
| Export bundle excludes host-side code | ⚠️ Reviewer can't audit the iframe contract end-to-end | See R-8 |
|
||||
|
||||
---
|
||||
|
||||
## R-1 — "default: functional with caveats" depends on D-1 having shipped
|
||||
|
||||
The per-mode functionality doc reports `default: c1 reactive, c2 reactive (3/3 runs)`. As recently as 2026-04-27 morning, default was provably non-reactive due to a race condition: the original Muon's `audioAnalyzer()` ran during `vizInit()`'s synchronous prefix and bound `analyser = wavesurfer.backend.analyser` (silent) before our `__STEMENGINE_MUON_ANALYSER__` was populated.
|
||||
|
||||
The D-1 fix (calling `engine.initialize()` synchronously at the end of `stemengine-audio-engine.js`'s IIFE) is what makes `default` reactive. Documented in [`docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/Fixes.md`](../plans/2026-04-27_Muon_Default_Mode_Reactivity/Fixes.md).
|
||||
|
||||
**Action:** The status doc should explicitly cite the D-1 fix as the reason `default` works, AND should say "tested against image built after 2026-04-27 D-1 patch landed". Without that pin, anyone running the same matrix against an older Muon image would see all four modes fail and incorrectly conclude the report was wrong.
|
||||
|
||||
---
|
||||
|
||||
## R-2 — "Intermittent stream-layer noise" caveat conflates two unrelated issues
|
||||
|
||||
The doc lists `ERR_ABORTED` and `502` as caveats for `default`. These are HTTP-layer events on the audio stream URL (`/api/subsonic/stream/...` proxied through the backend to Navidrome), not iframe events. They affect ALL four modes equally because they affect whether the host's `<audio>` element gets data — which is what feeds our analyser.
|
||||
|
||||
If the stream returns 502 mid-playback, the analyser sees zeros, ALL modes go non-reactive. Attributing this to "default" only is misleading.
|
||||
|
||||
**Action:** Move the `ERR_ABORTED`/`502` caveat to a global "stream backend health" section (or add a new column to the matrix), not to the per-mode rows.
|
||||
|
||||
---
|
||||
|
||||
## R-3 — Fluid FBO incomplete needs root-cause trace, not just symptom log
|
||||
|
||||
`[StemEngine][Fluid] FBO incomplete: 36054` (= `gl.FRAMEBUFFER_INCOMPLETE_ATTACHMENT` = `0x8CD6`) at [`fluid-render-graph.js:434`](../../muon-iframe/fluid-render-graph.js#L434) only fires during FBO creation in `createFBO()`. So the warning means: at the moment the fluid renderer constructed an FBO, the GL context rejected the texture attachment.
|
||||
|
||||
This implies the failure is NOT a runtime FBO becoming invalid mid-cycle. It's a fresh FBO that's incomplete from birth. Possible causes:
|
||||
|
||||
1. **Float texture extension missing or revoked** — `OES_texture_float_linear` / `EXT_color_buffer_float` were available at boot (bootstrap baseline check passed) but no longer when fluid creates a float-typed FBO. If the iframe is being remounted (see R-6 about `iframe key={muonEmbedUrl}`), each remount creates a new GL context that may not have the same extension set granted.
|
||||
2. **Texture format mismatch** — `FLOAT_TEXTURE_TYPE` / `TEXTURE_TYPE` values are mutated by `detectCapabilities()` ([`fluid-render-graph.js:54-63`](../../muon-iframe/fluid-render-graph.js#L54)) on each instantiation. If the constructor is invoked with a stale GL context (from a prior renderer that disposed but left the context in a different state), capability detection can disagree with the actual current context.
|
||||
3. **Width/height of zero** — if `setViewport` hasn't been received yet and `width`/`height` default to undefined → the FBO is created with `gl.texImage2D(... w, h, ...)` where `w`/`h` are `NaN` → incomplete.
|
||||
4. **Disposal race** — switching modes calls dispose on the previous render graph. If dispose deletes textures that were attached to FBOs that haven't been deleted yet, the FBO list survives but with broken attachments.
|
||||
|
||||
**Action:** Add a diagnostic line right before the warning at `fluid-render-graph.js:434`:
|
||||
```js
|
||||
if (status !== gl.FRAMEBUFFER_COMPLETE) {
|
||||
console.warn('[StemEngine][Fluid] FBO incomplete:', status, {
|
||||
width, height,
|
||||
floatTextures: caps.floatTextures, floatLinear: caps.floatLinear,
|
||||
glLost: !!window.__STEMENGINE_MUON_CONTEXT_LOST__,
|
||||
instantiationCount: window.__SE_FLUID_INSTANCES__ = (window.__SE_FLUID_INSTANCES__ || 0) + 1,
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
The instantiation counter alone will tell whether this is a fresh-instance failure (counter low) or a churn-induced failure (counter > 5 after a few mode switches).
|
||||
|
||||
---
|
||||
|
||||
## R-4 — Architecture plan revised twice (status: Rev 2 is the path forward)
|
||||
|
||||
**Rev 1 (intermediate):** flipped the original "shared local iframe shell + plugin API" recommendation to "one iframe app per non-default style", with the renderer being our existing `*-render-graph.js` code lifted out of the Muon iframe.
|
||||
|
||||
**Rev 2 (current — path forward):** corrected the renderer source. Each non-default iframe is a clone of the **third-party upstream** at a pinned commit (PavelDoGreat/Fluid-Simulation, keeffEoghan tendrils subtree, edankwan/The-Spirit), wrapped in StemEngine's scaffolding — same `git clone + sed-inject` recipe Muon uses today. Our existing `muon-iframe/{fluid,tendrils,spirit}-render-graph.js` files are deleted (except spirit's, which survives until Phase 6 decides its fate).
|
||||
|
||||
Spirit is **deferred** to Phase 6 because:
|
||||
- its upstream (Edan Kwan's "The Spirit") has no native audio reactivity (mouse-interactive only); audio coupling has to be invented
|
||||
- its license needs verification before any embedding work
|
||||
- its build path (Three.js portfolio experiment) may not be straightforward to extract
|
||||
|
||||
Each iframe's Dockerfile gets its own divergent build sequence (Muon: webpack; Fluid: TBD; Tendrils: subtree extraction + TBD build). They all share `visualizer-iframes/shared/` for the StemEngine scaffolding, but the upstream-specific build steps are unique per iframe.
|
||||
|
||||
Rev 2 of the plan is the one to execute. R-1 through R-3 (host transport diagnostics, FBO root cause) become Phase 0 of that plan.
|
||||
|
||||
---
|
||||
|
||||
## R-5 — `MUON_IFRAME_REV` duplicated across three files
|
||||
|
||||
```
|
||||
frontend/src/pages/VisualizerLabPage.tsx:16: const MUON_IFRAME_REV = '20260427c'
|
||||
frontend/src/components/VisualizerPreviewWidget.tsx:35: const MUON_IFRAME_REV = '20260427c'
|
||||
frontend/src/components/LyricsLivePreview.tsx:67: const MUON_IFRAME_REV = '20260427c'
|
||||
```
|
||||
|
||||
If a future change updates one and forgets the others, the iframe URLs diverge across surfaces:
|
||||
- Same Muon iframe gets two different `?se_rev=...` querystrings
|
||||
- React's iframe `key={muonEmbedUrl}` (now set in `LyricsViewport.tsx:93`) sees them as different keys
|
||||
- One surface remounts when the other doesn't
|
||||
- WebGL state is lost on the remounting one
|
||||
|
||||
**Action:** Hoist to a shared module:
|
||||
```ts
|
||||
// frontend/src/components/visualizers/muonIframeMeta.ts (new)
|
||||
export const MUON_IFRAME_REV = '20260427c'
|
||||
```
|
||||
Import from all three sites. Single point of truth.
|
||||
|
||||
---
|
||||
|
||||
## R-6 — `withDefaultPreset` is divergent across the three call sites
|
||||
|
||||
```
|
||||
VisualizerLabPage.tsx:74: url.searchParams.set('se_rev', MUON_IFRAME_REV) // unconditional
|
||||
VisualizerPreviewWidget.tsx:83: if (!url.searchParams.has('se_rev')) url.searchParams.set(...) // conditional
|
||||
LyricsLivePreview.tsx:115: if (!url.searchParams.has('se_rev')) url.searchParams.set(...) // conditional
|
||||
```
|
||||
|
||||
This means: if the backend ever returns a `muon_iframe_url` that already contains a `?se_rev=...` (e.g., admin pinned a specific iframe build for testing), the Lab page overrides it but the other two surfaces honor it. Surfaces drift.
|
||||
|
||||
The `LyricsViewport.tsx:93` `iframe key={props.muonEmbedUrl}` pattern compounds this — if the URL flickers across renders, the iframe remounts repeatedly (boot loop). The `useMemo([muonIframeUrl])` should make it stable per surface, but inconsistent semantics across surfaces still cause cross-surface flakiness.
|
||||
|
||||
**Action:** Hoist `withDefaultPreset` to the same shared module as `MUON_IFRAME_REV`. Pick one semantics — recommend the conditional version (don't override admin pins).
|
||||
|
||||
---
|
||||
|
||||
## R-7 — Reviewer checklist is missing edge cases
|
||||
|
||||
The current checklist (Section 2) tests the happy path well. Add Section 2.5:
|
||||
|
||||
```markdown
|
||||
## Section 2.5: Edge Cases (added by review)
|
||||
|
||||
### Iframe lifecycle
|
||||
- [ ] Switching style A → B → A returns to A's working state (no permanent freeze)
|
||||
- [ ] Tab visibility change (hidden → visible) does not require manual play to resume motion
|
||||
- [ ] Multiple mode switches in quick succession (<200ms apart) don't deadlock
|
||||
- [ ] Browser GPU process crash and restart: iframe degrades gracefully (warning surfaced, no white screen)
|
||||
|
||||
### Audio integration
|
||||
- [ ] Seek to time near end of track does not lose reactivity for the remainder
|
||||
- [ ] Playback rate change (1.0 → 1.5 → 1.0) keeps motion in sync
|
||||
- [ ] Audio stream returns 502 mid-playback: visualizer freezes (expected) but resumes on retry
|
||||
- [ ] Audio file longer than `AudioBuffer` decode limit (~hours): visualizer either degrades or refuses to load with explicit error
|
||||
|
||||
### Cross-surface
|
||||
- [ ] Same audio URL on Track Detail and Project page produces matching reactivity (same mode, same audio)
|
||||
- [ ] Visualizer Lab and Track Detail produce visually-equivalent fluid output (modulo viewport size)
|
||||
|
||||
### Capability degradation
|
||||
- [ ] Browser without `EXT_color_buffer_float`: fluid mode either falls back or surfaces a clear error (not a frozen black canvas)
|
||||
- [ ] Mobile-class GPU: fluid quality auto-selects 'low' or surfaces a quality recommendation
|
||||
```
|
||||
|
||||
**Action:** Append the above to `2026-04-28-reviewer-checklist.md`.
|
||||
|
||||
---
|
||||
|
||||
## R-8 — Export bundle missing host-side files
|
||||
|
||||
The bundle includes the iframe runtime files (good — these are what reviewers want to audit for renderer fidelity) but excludes the host hook that defines the iframe contract:
|
||||
|
||||
```
|
||||
muon-mode-review-package-2026-04-28/source-evidence/
|
||||
├── fluid-render-graph.js ✅ included
|
||||
├── tendrils-render-graph.js ✅ included
|
||||
├── spirit-render-graph.js ✅ included
|
||||
├── stemengine-router.js ✅ included
|
||||
├── stemengine-bootstrap.js ✅ included
|
||||
└── (missing)
|
||||
├── stemengine-audio-engine.js ❌ critical for D-1 audit
|
||||
├── stemengine-feature-extractor.js ❌ critical for normalization audit
|
||||
└── ../../../frontend/src/hooks/useMuonIframeTransport.ts ❌ host contract
|
||||
```
|
||||
|
||||
A reviewer trying to verify "is the host-iframe contract correctly implemented?" can't, with only the iframe side. They can only audit the iframe in isolation.
|
||||
|
||||
**Action:** Update `2026-04-28-export-bundle-manifest.md` to include:
|
||||
- `muon-iframe/stemengine-audio-engine.js`
|
||||
- `muon-iframe/stemengine-feature-extractor.js`
|
||||
- `frontend/src/hooks/useMuonIframeTransport.ts`
|
||||
- `docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/{Findings,README,Review,Fixes}.md`
|
||||
|
||||
The plan + fix docs let reviewers understand WHY the architecture is what it is, not just WHAT it is.
|
||||
|
||||
---
|
||||
|
||||
## R-9 — Cycle-2 replay diagnostic question (open)
|
||||
|
||||
The package's headline finding is "fluid: cycle 2 not reactive". The matrix shows fluid `c1=1552, c2=0` motion diff. Three hypotheses, ordered by likelihood:
|
||||
|
||||
1. **Iframe AudioBufferSourceNode `onended` race** — `engine.play()` registers `sourceNode.onended` to clear `isPlaying=false` when the buffer ends. If the host pauses just before the buffer naturally ends, `pause()` calls `disconnectSource()` which sets `sourceNode.onended = null` — but if `stop()` and `disconnect()` race against the auto-onended, state can become inconsistent. A subsequent `play()` from a different `time` may or may not produce a working source. ([`stemengine-audio-engine.js:97-106`](../../muon-iframe/stemengine-audio-engine.js#L97))
|
||||
|
||||
2. **Host-side drift correction sends a stale `transportSeek` immediately after `transportPlay`** — `useMuonIframeTransport.ts:182-184` sends `transportSeek` if drift > 250ms. On cycle 2's first frame after play, `lastSyncTimeRef = time` was just set in the play branch, so drift starts at 0. But if the host's `audioRef.currentTime` advances between `transportPlay` send and the next RAF tick (faster than expected due to async), the drift correction fires a `transportSeek` that internally does `pause(); play({time, rate: playbackRate})`. The intermediate `pause()` causes the renderer to see "stopped" briefly — fluid's pingpong FBOs may not recover.
|
||||
|
||||
3. **Fluid render graph never re-initialized between cycles** — fluid is the same instance across cycles (no style switch). But its internal state (FBOs, ping-pong textures) accumulates dye/velocity from cycle 1. On cycle 2, the dye texture might be saturated to a state that doesn't show motion until it dissipates. Doesn't explain the FBO-incomplete log though.
|
||||
|
||||
**Action:** Hypothesis 1 is most testable. Add a `console.log('[engine] play start', { time, rate, buffer: !!buffer, sourceNode: !!sourceNode })` at the top of `engine.play()` and `console.log('[engine] pause', { isPlaying, pausedAt })` in `engine.pause()`. Reproduce cycle-2 fail and check the log sequence. If `play()` runs but `sourceNode` is set then immediately reset, that's hypothesis 1. If `play()` runs cleanly and `transportSeek` arrives 16ms later, that's hypothesis 2.
|
||||
|
||||
---
|
||||
|
||||
## What I changed in the package
|
||||
|
||||
Surgical edits only:
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| [`2026-04-28-surface-functionality-matrix.md`](2026-04-28-surface-functionality-matrix.md) | Project page row corrected from "N/A" to "TESTABLE" with the LyricsLivePreview path explained. Visualizer Lab row corrected from "BLOCKED" to "OPENABLE" with the upload workaround pinned. |
|
||||
| [`2026-04-28-claude-review-notes.md`](2026-04-28-claude-review-notes.md) | This file. |
|
||||
|
||||
What I deliberately did NOT change:
|
||||
- The architecture plan ([`local-iframe-visualizer-architecture-plan.md`](export/muon-mode-review-package-2026-04-28/docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md)) — R-4 recommends reconciling it with the 2026-04-27 fixes, but rewriting it is the original author's call.
|
||||
- The reviewer checklist's main body — R-7 recommends an additive Section 2.5 rather than rewriting Section 2.
|
||||
- The export bundle's source files — R-8 recommends additions but the bundle is a frozen manifest from yesterday; the next manifest update should incorporate.
|
||||
|
||||
The matrix corrections were the only edits that reflect a *factual error* in the package. Everything else in this notes file is recommendations for the next pass.
|
||||
@@ -13,7 +13,8 @@ This export bundle is intended for third-party reviewers who need:
|
||||
- surface-by-surface testability notes,
|
||||
- reference-comparison analysis,
|
||||
- a reviewer scoring rubric,
|
||||
- a single index file for navigation.
|
||||
- a single index file for navigation,
|
||||
- the migration plan and transport contract for the per-renderer iframe work.
|
||||
|
||||
## Files To Include
|
||||
|
||||
@@ -33,13 +34,19 @@ This export bundle is intended for third-party reviewers who need:
|
||||
5. `docs/muon-mode-status/2026-04-28-reviewer-checklist.md`
|
||||
- Third-party scoring and signoff rubric.
|
||||
|
||||
6. `docs/muon-mode-status/export/muon-mode-review-package-2026-04-28/docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md`
|
||||
- Implementation plan for moving non-default styles to a local iframe visualizer runtime.
|
||||
|
||||
7. `docs/specs/visualizer-iframe-protocol.md`
|
||||
- Frozen Phase 1 host/iframe transport contract.
|
||||
|
||||
### Optional supporting source files
|
||||
Include these if the reviewer needs implementation-level evidence:
|
||||
6. `muon-iframe/fluid-render-graph.js`
|
||||
7. `muon-iframe/tendrils-render-graph.js`
|
||||
8. `muon-iframe/spirit-render-graph.js`
|
||||
9. `muon-iframe/stemengine-router.js`
|
||||
10. `muon-iframe/stemengine-bootstrap.js`
|
||||
8. `muon-iframe/fluid-render-graph.js`
|
||||
9. `muon-iframe/tendrils-render-graph.js`
|
||||
10. `muon-iframe/spirit-render-graph.js`
|
||||
11. `muon-iframe/stemengine-router.js`
|
||||
12. `muon-iframe/stemengine-bootstrap.js`
|
||||
|
||||
## Minimal Bundle
|
||||
If the recipient should receive docs only, export:
|
||||
@@ -48,10 +55,14 @@ If the recipient should receive docs only, export:
|
||||
- `docs/muon-mode-status/2026-04-28-surface-functionality-matrix.md`
|
||||
- `docs/muon-mode-status/2026-04-28-reference-comparison-third-party-review.md`
|
||||
- `docs/muon-mode-status/2026-04-28-reviewer-checklist.md`
|
||||
- `docs/muon-mode-status/export/muon-mode-review-package-2026-04-28/docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md`
|
||||
- `docs/specs/visualizer-iframe-protocol.md`
|
||||
|
||||
## Full Evidence Bundle
|
||||
If the recipient should receive both review docs and implementation evidence, export:
|
||||
- all files from the Minimal Bundle
|
||||
- `docs/muon-mode-status/export/muon-mode-review-package-2026-04-28/docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md`
|
||||
- `docs/specs/visualizer-iframe-protocol.md`
|
||||
- `muon-iframe/fluid-render-graph.js`
|
||||
- `muon-iframe/tendrils-render-graph.js`
|
||||
- `muon-iframe/spirit-render-graph.js`
|
||||
@@ -64,9 +75,14 @@ muon-mode-review-package-2026-04-28/
|
||||
README.md
|
||||
docs/
|
||||
2026-04-27-current-per-mode-functionality.md
|
||||
2026-04-28-export-bundle-manifest.md
|
||||
2026-04-28-surface-functionality-matrix.md
|
||||
2026-04-28-reference-comparison-third-party-review.md
|
||||
2026-04-28-reviewer-checklist.md
|
||||
plans/
|
||||
2026-04-28_local-iframe-visualizer-architecture-plan.md
|
||||
specs/
|
||||
visualizer-iframe-protocol.md
|
||||
source-evidence/
|
||||
fluid-render-graph.js
|
||||
tendrils-render-graph.js
|
||||
@@ -87,9 +103,15 @@ Recommended reading sequence inside the exported bundle:
|
||||
3. `2026-04-27-current-per-mode-functionality.md`
|
||||
4. `2026-04-28-surface-functionality-matrix.md`
|
||||
5. `2026-04-28-reviewer-checklist.md`
|
||||
6. `docs/muon-mode-status/export/muon-mode-review-package-2026-04-28/docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md`
|
||||
7. `docs/specs/visualizer-iframe-protocol.md`
|
||||
|
||||
## Current Package Status
|
||||
As of this manifest:
|
||||
- `fluid` remains the main concern because of replay/FBO reliability issues.
|
||||
- `tendrils` and `spirit` are documented as in-family approximations, not proven literal ports.
|
||||
- Visualizer Lab and project/embed surfaces remain limited or non-testable for this exact replay matrix without additional setup.
|
||||
- Phase 0 is complete: the replay regression root cause was host-side seek churn and the fix shipped.
|
||||
- Phase 1 is complete: the transport contract is frozen in `docs/specs/visualizer-iframe-protocol.md`.
|
||||
- Phase 2 is complete: shared scaffolding moved to `visualizer-iframes/shared/`, Muon consumes it via compose `additional_contexts`, and `stemengine-muon` was rebuilt/recreated successfully.
|
||||
- Phase 4 is complete: `fluid` runs as standalone `stemengine-fluid`, and the in-Muon fluid render graph path was removed.
|
||||
- Phase 5 is complete: `tendrils` runs as standalone `stemengine-tendrils`, and the in-Muon tendrils render graph path was removed.
|
||||
- Phase 6 is in progress: automation/documentation gate checks are recorded; interactive browser matrix and final reviewer verdicts are still pending.
|
||||
- Release-gate note: Tendrils upstream project-level licensing provenance remains unresolved and must be confirmed before ship-ready signoff.
|
||||
@@ -0,0 +1,275 @@
|
||||
# Handoff to Copilot — Muon → Per-Renderer Iframe Migration
|
||||
|
||||
**Date:** 2026-04-28
|
||||
**From:** Claude (this session)
|
||||
**To:** Copilot (next implementation session)
|
||||
**Status:** Phase 0, Phase 1, Phase 2, Phase 3, Phase 4, and Phase 5 completed; Phase 6 validation/release-gate work remains.
|
||||
|
||||
This document is the single thing the next agent should read first. It points at every other doc, calls out what's decided vs what's still open, and pins the exact files / commits to start from.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR — what to do first
|
||||
|
||||
Phase 0 update (2026-04-28): complete. See `docs/muon-mode-status/2026-04-28-phase0-host-seek-findings.md`.
|
||||
|
||||
1. **Read `2026-04-28_local-iframe-visualizer-architecture-plan.md`** in `docs/muon-mode-status/export/muon-mode-review-package-2026-04-28/docs/plans/`. That's the plan of record. Rev 2 (third-party upstream per iframe) is what to execute.
|
||||
2. **Read `2026-04-28-claude-review-notes.md`** in `docs/muon-mode-status/`. Especially R-3, R-7, R-9, and the C-1..C-12 concerns now mirrored into the plan's "Reviewer Concerns" section.
|
||||
3. **Read the repo-local agent guidance in `docs/agents/` before implementation work starts.** Minimum required set:
|
||||
- `docs/agents/README.md`
|
||||
- `docs/agents/agency-agents/index.md`
|
||||
- `docs/agents/claude-ui-agents/index.md`
|
||||
- `docs/agents/agency-overrides/frontend.md`
|
||||
- `docs/agents/agency-overrides/release-gate.md`
|
||||
4. **Use the Phase 0 findings note as the new baseline** — the cycle-2 bug was host-side seek churn in `useMuonIframeTransport.ts`, the host fix shipped, and Visualizer Lab local uploads now cross the iframe boundary through binary `loadAudioData`.
|
||||
5. **Read `docs/specs/visualizer-iframe-protocol.md` before Phase 4 work.** Phase 1 is complete and freezes the version `1` contract, current command/event set, and reserved provider-neutral extensions.
|
||||
6. **Use Phase 2 as baseline:** shared scaffolding now lives in `visualizer-iframes/shared/`, Muon consumes it through compose `additional_contexts`, and the service was rebuilt/recreated successfully after migration.
|
||||
7. **Use Phase 3 as baseline:** host transport is now provider-neutral (`useVisualizerIframeTransport`), provider URL resolution is centralized in `frontend/src/visualizers/providers.ts`, renderer controls are dynamic via `RendererControlsPanel`, and per-style iframe params/palette override state is persisted in visualizer settings.
|
||||
8. **Phase 5 is complete** (pre-work + implementation): Tendrils is now running as standalone `stemengine-tendrils` and `muon-iframe/tendrils-render-graph.js` has been removed.
|
||||
9. **Proceed to Phase 6**: run cross-surface validation (`default|fluid|tendrils`) and complete release-gate documentation updates, while resolving Tendrils licensing status before ship-ready declaration.
|
||||
|
||||
---
|
||||
|
||||
## What was decided this week (don't re-litigate)
|
||||
|
||||
| Decision | Where | Notes |
|
||||
|---|---|---|
|
||||
| Per-renderer iframe model (NOT shared shell) | Plan §"Decision Summary" | User-confirmed |
|
||||
| Each new iframe clones a third-party upstream (Muon recipe) | Plan §"Revision Note (2026-04-28)" Rev 2 | User-confirmed via the upstream URLs they provided |
|
||||
| Spirit deferred to Phase 6 | Plan §"Decision Summary", Phase 6 | User-confirmed |
|
||||
| Per-renderer Dockerfiles diverge (different upstream build systems) | Plan §"Decision Summary" | User-confirmed |
|
||||
| Zero access control (CORS=`*`, no sandbox, no origin allowlist) | `docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/README.md` §8 | Deployment policy |
|
||||
| `useMuonIframeTransport` is consumed by 3 host components — keep it that way (no triplication) | `docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/README.md` §9 | Hook is renamed to `useVisualizerIframeTransport` in Phase 3 |
|
||||
| Existing `muon-iframe/{fluid,tendrils,spirit}-render-graph.js` deleted as their iframes ship | Plan §"Migration model" | Spirit's stays until Phase 6 decision |
|
||||
| Three new top-level concerns added to plan: per-renderer controls, palette propagation, reviewer concerns C-1..C-12 | Plan §"User-Facing Per-Renderer Visual Controls" / §"Palette Integration" / §"Reviewer Concerns" | All accounted for |
|
||||
| Repo-local agent guidance in `docs/agents/` is mandatory input for implementation routing and closure | `docs/agents/README.md`, `docs/agents/agency-agents/index.md`, `docs/agents/agency-overrides/*` | Do not skip specialist + override guidance |
|
||||
|
||||
---
|
||||
|
||||
## Required repo-local agent guidance
|
||||
|
||||
The next implementation session must use the repository-local agent documentation under `docs/agents/`.
|
||||
|
||||
Minimum required reading before code changes:
|
||||
- `docs/agents/README.md`
|
||||
- `docs/agents/agency-agents/index.md`
|
||||
- `docs/agents/claude-ui-agents/index.md`
|
||||
- `docs/agents/agency-overrides/frontend.md`
|
||||
- `docs/agents/agency-overrides/release-gate.md`
|
||||
|
||||
If work touches security-sensitive transport/origin/message handling, also read:
|
||||
- `docs/agents/agency-overrides/security.md`
|
||||
|
||||
Execution expectation:
|
||||
- Use one primary specialist route per task from `docs/agents/agency-agents/index.md`
|
||||
- Use the UI routing from `docs/agents/claude-ui-agents/index.md` for the controls panel and related UI work
|
||||
- Treat `docs/agents/agency-overrides/*` as execution constraints, not optional reading
|
||||
|
||||
Recommended routing for this migration:
|
||||
- Architecture / phased migration changes
|
||||
- Primary: `Software Architect`
|
||||
- Reviewer: `Reality Checker`
|
||||
- Frontend provider hook / registry / settings schema
|
||||
- Primary: `Frontend Developer`
|
||||
- Reviewer: `Evidence Collector`
|
||||
- Renderer controls panel / UI decomposition
|
||||
- Primary: `React Component Architect`
|
||||
- Supporting: `CSS Architecture Specialist`, `ARIA Implementation Specialist`
|
||||
- Render-loop / resource-budget / performance validation
|
||||
- Primary: `Performance Benchmarker`
|
||||
- Reviewer: `Software Architect`
|
||||
- Documentation / handoff / review package updates
|
||||
- Primary: `Technical Writer`
|
||||
- Reviewer: `Evidence Collector`
|
||||
|
||||
---
|
||||
|
||||
## What's open / requires Copilot's judgment
|
||||
|
||||
### Phase 0 hypothesis selection
|
||||
Resolved on 2026-04-28. See `docs/muon-mode-status/2026-04-28-phase0-host-seek-findings.md`.
|
||||
|
||||
Historical candidates that were evaluated during Phase 0:
|
||||
1. Engine `AudioBufferSourceNode.onended` race in `stemengine-audio-engine.js:97-106`
|
||||
2. Host drift-correction `transportSeek` racing `transportPlay` in `useMuonIframeTransport.ts:182-184`
|
||||
3. Fluid render graph state accumulation across cycles (FBO ping-pong saturation)
|
||||
|
||||
The R-3 diagnostic patch + console-log instrumentation identified the issue as host-side seek churn. The fix shipped. Keep the note for historical context only.
|
||||
|
||||
### Per-upstream audio-injection mechanism
|
||||
Each upstream is different. For each (fluid, tendrils):
|
||||
- Read the upstream entry file
|
||||
- Choose between mechanism (a) "sed-redirect existing analyser" vs (b) "inject audio→action adapter"
|
||||
- Document the choice in `visualizer-iframes/<style>/audio-injection-patches.md` BEFORE writing the Dockerfile
|
||||
|
||||
### Per-upstream palette mapping
|
||||
Same pattern as audio-injection. Each upstream has its own color generation; you decide where to inject palette consumption. Document.
|
||||
|
||||
### Per-upstream tunable params
|
||||
Each upstream has its own knob set. The plan lists "likely knobs" per renderer — confirm by reading the upstream code, then write the `capabilities` emitter that surfaces them.
|
||||
|
||||
### Spirit decision (Phase 6)
|
||||
Three options documented in plan Phase 6. Choose based on what you find when you inspect `https://github.com/edankwan/The-Spirit`. License check is mandatory before option (a) or (b).
|
||||
|
||||
---
|
||||
|
||||
## Files you'll create
|
||||
|
||||
### New iframe scaffolding (Phase 2)
|
||||
```
|
||||
visualizer-iframes/
|
||||
└── shared/
|
||||
├── stemengine-bootstrap.js (copy from muon-iframe/)
|
||||
├── stemengine-router.js (copy + extend with setPalette, setParam, capabilities emitter)
|
||||
├── stemengine-audio-engine.js (copy from muon-iframe/)
|
||||
└── stemengine-feature-extractor.js (copy from muon-iframe/)
|
||||
```
|
||||
|
||||
### New per-renderer iframes (Phase 4, 5)
|
||||
```
|
||||
visualizer-iframes/
|
||||
├── fluid/
|
||||
│ ├── Dockerfile (clones PavelDoGreat/Fluid-Simulation)
|
||||
│ ├── nginx.conf
|
||||
│ └── audio-injection-patches.md (sed targets + upstream commit pin)
|
||||
├── tendrils/
|
||||
│ ├── Dockerfile (clones keeffEoghan, extracts src/tendrils/)
|
||||
│ ├── nginx.conf
|
||||
│ └── audio-injection-patches.md
|
||||
└── spirit/
|
||||
└── DECISION.md (Phase 6 — written even if "not now")
|
||||
```
|
||||
|
||||
### New host-side files (Phase 3)
|
||||
```
|
||||
frontend/src/
|
||||
├── hooks/
|
||||
│ └── useVisualizerIframeTransport.ts (renamed/extended from useMuonIframeTransport.ts)
|
||||
├── visualizers/
|
||||
│ └── providers.ts (provider registry)
|
||||
└── components/visualizers/
|
||||
└── RendererControlsPanel.tsx (dynamic controls UI)
|
||||
```
|
||||
|
||||
### New protocol spec (Phase 1)
|
||||
```
|
||||
docs/specs/visualizer-iframe-protocol.md (commands, events, capabilities schema, palette schema)
|
||||
```
|
||||
|
||||
### Modified host-side files (Phase 3)
|
||||
- `frontend/src/components/LyricsLivePreview.tsx` — replace `useMuonIframeTransport` call with `useVisualizerIframeTransport`; mount `RendererControlsPanel`; pass palette
|
||||
- `frontend/src/components/VisualizerPreviewWidget.tsx` — same
|
||||
- `frontend/src/pages/VisualizerLabPage.tsx` — same
|
||||
- `frontend/src/components/visualizers/visualizerSettings.ts` — extend schema with `params` keyed by renderer + `paletteOverride`
|
||||
- `frontend/src/hooks/useMuonIframeTransport.ts` — DELETE (replaced)
|
||||
- `/home/translator/docker/stemengine.yml` — add `stemengine-fluid` and `stemengine-tendrils` services with `additional_contexts: shared:`
|
||||
|
||||
### Files you'll DELETE
|
||||
- `muon-iframe/fluid-render-graph.js` (Phase 4 exit criterion)
|
||||
- `muon-iframe/tendrils-render-graph.js` (Phase 5 exit criterion)
|
||||
- `muon-iframe/spirit-render-graph.js` (Phase 6 — only if Phase 6 chooses option a or b)
|
||||
|
||||
---
|
||||
|
||||
## Files you should NOT touch
|
||||
|
||||
- `muon-iframe/stemengine-bootstrap.js` (use the copy in `visualizer-iframes/shared/` instead — Phase 2 makes the Muon iframe consume from shared/)
|
||||
- `muon-iframe/stemengine-router.js` (same)
|
||||
- `muon-iframe/stemengine-audio-engine.js` (same)
|
||||
- `muon-iframe/stemengine-feature-extractor.js` (same)
|
||||
- `frontend/src/audio/MuonFeatureExtractor.ts` (host-side, unrelated to iframes)
|
||||
- `frontend/src/hooks/useAudioFeatures.ts` (still drives in-house canvas visualizers)
|
||||
- All `frontend/src/components/visualizers/*VisualizerCanvas.tsx` files (in-house canvas renderers, untouched)
|
||||
|
||||
---
|
||||
|
||||
## Order of operations
|
||||
|
||||
The plan's Recommended Execution Order has 12 steps. Reproducing here for quick reference:
|
||||
|
||||
```
|
||||
0. Phase 0 complete — review `docs/muon-mode-status/2026-04-28-phase0-host-seek-findings.md`
|
||||
1. Phase 1 complete — use docs/specs/visualizer-iframe-protocol.md as the frozen contract
|
||||
2. Phase 2 complete — shared scaffolding extracted; Muon consumes it via compose `additional_contexts`
|
||||
3. Refactor host to provider registry + useVisualizerIframeTransport + RendererControlsPanel
|
||||
4. Pre-work: inspect PavelDoGreat/Fluid-Simulation upstream; design audio + palette + capabilities injection
|
||||
5. Build visualizer-iframes/fluid/ iframe; deploy stemengine-fluid; route via host registry; DELETE muon-iframe/fluid-render-graph.js
|
||||
6. Pre-work: inspect keeffEoghan tendrils subtree; design extraction + injection
|
||||
7. Build visualizer-iframes/tendrils/ iframe; deploy stemengine-tendrils; route via host registry; DELETE muon-iframe/tendrils-render-graph.js
|
||||
8. Cross-surface validation across Track Detail, Visualizer Lab, Project page (LyricsLivePreview) for muon/fluid/tendrils
|
||||
9. Update reviewer package with post-migration results
|
||||
10. Phase 6 spirit pre-work: license check + audio-reactivity decision; record in DECISION.md
|
||||
11. (If decision is to ship) Build visualizer-iframes/spirit/ iframe per chosen mechanism; deploy stemengine-spirit; DELETE muon-iframe/spirit-render-graph.js
|
||||
12. (Optional) Implement iframe pre-warm to eliminate style-switch flash (only if C-1/C-2/C-3 budget allows)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Things to verify before you start coding
|
||||
|
||||
| Check | How | Why |
|
||||
|---|---|---|
|
||||
| Recently-applied D-1..D-8 fixes are still in place | `grep -rn 'STEMENGINE_ALLOWED_ORIGIN' frontend muon-iframe .env.example` → only the D-8 explanatory comment in Dockerfile should match | Confirms the 2026-04-27 cleanup is the baseline |
|
||||
| Default mode is currently reactive | Open `http://work:5174/projects/<id>`, set Muon visual style with `default`, hit play, watch particles react | If broken, D-1 didn't actually ship — investigate before doing anything else |
|
||||
| `npm run -s typecheck` exits 0 from `frontend/` | (literally that) | Baseline before refactor |
|
||||
| `docker compose -f /home/translator/docker/stemengine.yml config` is valid | (literally that) | Baseline before adding services |
|
||||
| The three iframe mounting sites all use `useMuonIframeTransport` today | `grep -rn 'useMuonIframeTransport' frontend/src` → exactly 3 matches in `LyricsLivePreview.tsx`, `VisualizerPreviewWidget.tsx`, `VisualizerLabPage.tsx` | Confirms the hook is the only consumer; rename is safe |
|
||||
| `MUON_IFRAME_REV` is duplicated in 3 files (R-5) | `grep -rn 'MUON_IFRAME_REV' frontend/src` | Hoist to `providers.ts` in Phase 3 |
|
||||
| `withDefaultPreset` is divergent in 3 files (R-6) | `grep -rn 'withDefaultPreset\|withDefaultMuonPreset' frontend/src` | Hoist to `providers.ts` in Phase 3, single semantics |
|
||||
|
||||
---
|
||||
|
||||
## Reference reading order (for the next agent)
|
||||
|
||||
1. **This file** (you are here)
|
||||
2. `docs/agents/README.md` — agent-doc precedence and collection index
|
||||
3. `docs/agents/agency-agents/index.md` — primary specialist routing for implementation work
|
||||
4. `docs/agents/claude-ui-agents/index.md` — UI specialist routing for controls/palette UX
|
||||
5. `docs/agents/agency-overrides/frontend.md` — mandatory frontend constraints + validation
|
||||
6. `docs/agents/agency-overrides/release-gate.md` — closure requirements and evidence expectations
|
||||
7. `docs/muon-mode-status/export/muon-mode-review-package-2026-04-28/docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md` — the plan of record
|
||||
8. `docs/muon-mode-status/2026-04-28-claude-review-notes.md` — the agent-skill review (R-1..R-9, C-1..C-12 explained)
|
||||
9. `docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/Findings.md` — original architecture discovery
|
||||
10. `docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/README.md` — invariants + contract
|
||||
11. `docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/Review.md` — D-1..D-12 defect register
|
||||
12. `docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/Fixes.md` — what shipped on 2026-04-27 (your starting baseline)
|
||||
13. `docs/muon-mode-status/2026-04-28-surface-functionality-matrix.md` — what's tested where (corrected)
|
||||
14. `docs/muon-mode-status/2026-04-27-current-per-mode-functionality.md` — current per-mode reactivity status
|
||||
15. `muon-iframe/Dockerfile` — your template for each new iframe's Dockerfile
|
||||
16. `muon-iframe/stemengine-{router,bootstrap,audio-engine,feature-extractor}.js` — the scaffolding files you'll copy into `visualizer-iframes/shared/`
|
||||
17. `frontend/src/hooks/useMuonIframeTransport.ts` — the host hook you'll rename + extend
|
||||
|
||||
---
|
||||
|
||||
## Stop conditions
|
||||
|
||||
Pause and ask the user before proceeding if:
|
||||
|
||||
- **Phase 0 reveals the cycle-2 bug is NOT in fluid-render-graph.js, NOT in the host hook, NOT in the engine — somewhere else entirely.** Don't speculate; surface the finding and ask.
|
||||
- **Any upstream license check fails** (PavelDoGreat is MIT — verified; tendrils + spirit unverified).
|
||||
- **The Phase 4 pre-work shows PavelDoGreat fluid has no audio integration point AND no `multipleSplats(...)`-equivalent API for our adapter.** That breaks mechanism (b); ask before inventing a workaround.
|
||||
- **The Phase 5 pre-work shows the keeffEoghan tendrils subtree is unbuildable in isolation** (no own package.json, hard Jekyll dependencies). Ask before pursuing the Phase 5 fallback (keep `tendrils-render-graph.js`, rename style for clarity).
|
||||
- **Browser AudioContext / WebGL context limits (C-1, C-2) are hit during Phase 4 standalone testing.** Don't ship until lazy-load is verified to stay under budget.
|
||||
- **You find a reason to keep one of the existing `*-render-graph.js` files instead of replacing it.** Ask before deviating from the deletion plan.
|
||||
- **The user-facing controls panel design conflicts with an existing UX pattern in `LyricsLivePreview` you don't understand.** Ask — there's a lot of speaker-palette / gloss / lyric-style UI in that component that this plan doesn't fully exhaust.
|
||||
|
||||
---
|
||||
|
||||
## What "done" looks like (Success Definition mirror)
|
||||
|
||||
When all phases ship:
|
||||
- 4 iframes (`muon` existing, `fluid`, `tendrils`, `spirit` deferred) each running a third-party upstream + StemEngine scaffolding
|
||||
- 1 host hook (`useVisualizerIframeTransport`) consumed by 3 mounting sites
|
||||
- 1 provider registry, 1 `RendererControlsPanel`, 1 palette-propagation path
|
||||
- All 4 styles share unified visual identity tied to album art (or speaker palette in lyrics mode)
|
||||
- All 4 styles expose their own per-renderer controls dynamically
|
||||
- The `muon-iframe/` directory contains ONLY: `stemengine-*.js` scaffolding (consumed from shared/) + Muon-specific Dockerfile + nginx.conf. No more custom render graphs, no more sed-injection of fluid/tendrils/spirit.
|
||||
- Adding a 5th visualizer = one PR adding one folder + one compose service + one provider entry.
|
||||
|
||||
---
|
||||
|
||||
## Final note from the reviewing agent
|
||||
|
||||
The plan is opinionated about ordering (Phase 0 first, fluid before tendrils, validation before fidelity). If you find a reason to deviate, document it in this handoff doc as an Addendum before going off-script — don't silently re-order. The ordering exists because each phase's outputs become inputs for the next; out-of-order execution multiplies rework risk.
|
||||
|
||||
Good luck. Most of the design ambiguity has been compressed out by Rev 2 of the plan; what remains is execution.
|
||||
@@ -0,0 +1,49 @@
|
||||
# Phase 0 Findings: Host-Side Seek Churn
|
||||
|
||||
Date: 2026-04-28
|
||||
Status: Complete
|
||||
Scope: Phase 0 precondition from the local iframe visualizer architecture plan
|
||||
|
||||
## Summary
|
||||
The cycle-2 fluid replay failure was traced to host-side transport behavior, not to a persistent renderer-side FBO initialization defect.
|
||||
|
||||
Two concrete fixes landed:
|
||||
- `frontend/src/hooks/useMuonIframeTransport.ts`
|
||||
- Drift correction no longer emits `transportSeek` during normal steady playback.
|
||||
- The host now compares the media element's current time against expected wall-clock progression and only seeks on real timeline discontinuities.
|
||||
- `frontend/src/hooks/useMuonIframeTransport.ts` + `muon-iframe/{stemengine-router.js,stemengine-audio-engine.js}`
|
||||
- Visualizer Lab local uploads now load into the iframe via `loadAudioData` binary transport when the host audio URL is a `blob:` URL.
|
||||
- This makes Visualizer Lab a valid cross-origin replay test surface instead of failing at iframe fetch time.
|
||||
|
||||
## Root cause
|
||||
Before the fix, `useMuonIframeTransport` compared the host media time to the last sampled media time. During normal playback, that delta naturally exceeds 250ms, so the hook emitted `transportSeek` repeatedly while audio was already playing.
|
||||
|
||||
Inside the iframe audio engine, each `seek()` while playing performs:
|
||||
- `pause()`
|
||||
- `play({ time, rate })`
|
||||
|
||||
That created an avoidable pause/play churn loop during steady playback. In fluid mode this was enough to destabilize replay behavior and was a plausible source of the earlier cycle-2 failure.
|
||||
|
||||
## Evidence
|
||||
Validation surface used:
|
||||
- `http://localhost:3001/visualizer-lab`
|
||||
- Uploaded `frontend/public/sfx/applause.mp3` through the page's local upload flow
|
||||
- Selected Muon style `fluid`
|
||||
- Ran play -> pause -> play probe with iframe screenshot-diff motion checks
|
||||
|
||||
Post-fix results:
|
||||
- Cycle 1 reactive: yes
|
||||
- Cycle 2 reactive: yes
|
||||
- `transportSeek` logs during steady playback: none
|
||||
- `transportPlay` sequence: one play on cycle 1, one pause, one play on cycle 2
|
||||
|
||||
Representative outcome:
|
||||
- cycle 1 diff: ~1770 bytes
|
||||
- cycle 2 diff: ~1739 bytes
|
||||
|
||||
## Notes
|
||||
- The richer `fluid-render-graph.js` FBO warning payload was kept in place for future diagnostics. It did not fire in the successful Visualizer Lab validation pass after the host fix.
|
||||
- Track Detail at `work:3001/tracks/a5f76485ba8f7e5c41aea2c6a41ca0eb` was returning a server-side track-details failure during this session, so the decisive Phase 0 validation was completed on Visualizer Lab instead.
|
||||
|
||||
## Next step
|
||||
Proceed to Phase 1 (`docs/specs/visualizer-iframe-protocol.md`) and Phase 2 shared scaffolding extraction. Phase 4 should still rerun the fluid two-cycle replay regression after the upstream iframe lands, but the original blocking unknown is resolved.
|
||||
@@ -139,3 +139,24 @@ Include these files in review packets:
|
||||
- `muon-iframe/stemengine-bootstrap.js`
|
||||
- `docs/muon-mode-status/2026-04-27-current-per-mode-functionality.md`
|
||||
- `docs/muon-mode-status/2026-04-28-surface-functionality-matrix.md`
|
||||
|
||||
## 2026-04-28 Phase 6 Delta (Post-Cutover Correction)
|
||||
|
||||
This document captured a pre-cutover snapshot for Tendrils. As of the Phase 5 implementation:
|
||||
- `muon-iframe/tendrils-render-graph.js` has been deleted.
|
||||
- Tendrils now runs as standalone iframe service (`stemengine-tendrils`) from `visualizer-iframes/tendrils/`.
|
||||
|
||||
For any new third-party review pass, replace Tendrils evidence inputs with:
|
||||
- `visualizer-iframes/tendrils/Dockerfile`
|
||||
- `visualizer-iframes/tendrils/nginx.conf`
|
||||
- `visualizer-iframes/tendrils/stemengine-tendrils-adapter.js`
|
||||
- `/home/translator/docker/stemengine.yml` (`stemengine-tendrils` service definition)
|
||||
|
||||
Spirit note:
|
||||
- `spirit` is now shipped as standalone upstream iframe provider (`stemengine-spirit`) from `visualizer-iframes/spirit/`.
|
||||
- for new review passes, replace in-Muon Spirit inputs with:
|
||||
- `visualizer-iframes/spirit/Dockerfile`
|
||||
- `visualizer-iframes/spirit/index.html`
|
||||
- `visualizer-iframes/spirit/stemengine-spirit-prehook.js`
|
||||
- `visualizer-iframes/spirit/stemengine-spirit-adapter.js`
|
||||
- `/home/translator/docker/stemengine.yml` (`stemengine-spirit` service definition)
|
||||
@@ -150,3 +150,39 @@ Overall release recommendation:
|
||||
- DO NOT APPROVE
|
||||
|
||||
Summary comments:
|
||||
|
||||
## 2026-04-28 Phase 6 Addendum (Post-Cutover)
|
||||
|
||||
Infrastructure checks already completed:
|
||||
- [x] `stemengine-muon` service up on port `3011`
|
||||
- [x] `stemengine-fluid` service up on port `3012`
|
||||
- [x] `stemengine-tendrils` service up on port `3013`
|
||||
- [x] direct endpoint checks return HTML for all three services
|
||||
- [x] frontend `typecheck` passes after Tendrils cutover
|
||||
- [x] Visualizer Lab style-switch routing validated (`default->3011`, `fluid->3012`, `tendrils->3013`)
|
||||
|
||||
Current-session blockers captured:
|
||||
- [x] Track Detail currently blocked by repeated `500` API responses (stuck on loading state)
|
||||
- [x] Project page currently blocked by repeated `500` API responses (preview not testable)
|
||||
- [x] Agent-browser local file chooser path translation prevents Visualizer Lab upload from this environment (`ENOENT` on translated host path)
|
||||
|
||||
Still required for closure:
|
||||
- [ ] Track Detail interactive pass for `default|fluid|tendrils`
|
||||
- [ ] Visualizer Lab interactive pass with uploaded local audio for `default|fluid|tendrils`
|
||||
- [ ] Project page (`LyricsLivePreview`) interactive pass after selecting Muon visual style for `default|fluid|tendrils`
|
||||
- [x] Spirit style label aligned to `Spirit` for upstream synthetic-coupling path
|
||||
- [x] Spirit upstream implementation checklist ready before code cutover:
|
||||
- [x] synthetic coupling adapter path documented (uniform-driver model)
|
||||
- [x] aggressive beat profile acceptance criteria documented
|
||||
- [x] passive-upstream fallback behavior documented for coupling init failure
|
||||
- [x] palette propagation requirement documented as first-cut gate
|
||||
- [ ] Side-by-side reviewer notes updated to reflect standalone Tendrils provider behavior
|
||||
- [ ] Tendrils upstream licensing provenance confirmed for ship-ready release gate
|
||||
- [ ] Spirit interactive parity run completed (Track Detail, Visualizer Lab, Project page) after backend `500` blockers are cleared
|
||||
|
||||
Phase 6 documentation progress (current session):
|
||||
- [x] `visualizer-iframes/spirit/DECISION.md` updated to selected option: upstream Spirit with synthetic audio coupling
|
||||
- [x] `visualizer-iframes/spirit/` implementation landed (`Dockerfile`, `index.html`, `nginx.conf`, prehook + adapter)
|
||||
- [x] `stemengine-spirit` service added and deployed on `3014`
|
||||
- [x] provider declaration + transport script injection verified in served Spirit HTML
|
||||
- [x] fallback-mode behavior documented for cases where coupling init cannot lock in runtime
|
||||
@@ -18,8 +18,8 @@ Surfaces covered:
|
||||
| Surface | default | fluid | tendrils | spirit | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| Track Detail preview | PASS | FAIL | PASS | PASS | `fluid` fails on cycle 2 replay (non-reactive), others reactive in same run |
|
||||
| Visualizer Lab | BLOCKED | BLOCKED | BLOCKED | BLOCKED | Style combobox is present, but song is not loaded (`Song: None selected`), play/reset disabled |
|
||||
| Project page embed | N/A | N/A | N/A | N/A | No Muon replay controls or mode selector exposed in this project surface snapshot |
|
||||
| Visualizer Lab | OPENABLE | OPENABLE | OPENABLE | OPENABLE | Style combobox is present and the page has a "Local file" upload button — load any audio to enable replay tests. Previously marked BLOCKED because the tester didn't upload a file; the surface itself is fully testable. |
|
||||
| Project page (LyricsLivePreview) | TESTABLE | TESTABLE | TESTABLE | TESTABLE | `/projects/:id` renders `LyricsLivePreview`, which DOES mount a Muon iframe via `useMuonIframeTransport`. Mode controls live behind the `LyricsStylePicker` ("Muon" visual style); once selected, all four Muon styles are exposed via `liveVizSettings.muon_iframe.style`. Previously marked N/A because the tester didn't open the style picker. |
|
||||
|
||||
## Evidence Snapshot
|
||||
|
||||
@@ -48,18 +48,95 @@ Observed:
|
||||
|
||||
Implication:
|
||||
- per-mode replay reactivity cannot be validated until an audio source is loaded.
|
||||
- **Workaround for next test pass:** click the page's "Upload" button (top-right of the lab UI; renders a file picker via `extractEmbeddedArtwork()` flow at `pages/VisualizerLabPage.tsx:85-102`). Any local audio file works. After upload, `Play preview` enables and the matrix can be filled.
|
||||
|
||||
### 3) Project page (`http://work:5174/projects/e98f2b8a-0cad-4b4e-b33e-1ca2c8c87a0a`)
|
||||
Observed in current snapshot:
|
||||
- No Muon iframe preview controls for replay testing
|
||||
- No mode buttons (`default`, `fluid`, `tendrils`, `spirit`) exposed
|
||||
- No preview play/pause controls exposed for this Muon test path
|
||||
### 3) Project page (`http://work:5174/projects/<id>`)
|
||||
Corrected observation:
|
||||
- `pages/ProjectDetail.tsx:18, 2016` lazy-imports and renders `LyricsLivePreview`.
|
||||
- `LyricsLivePreview.tsx:1611` calls `useMuonIframeTransport({ ... })` with `isActive: isIframeVisualizerMode`.
|
||||
- `isIframeVisualizerMode` becomes true when the user picks the **Muon** visual style via the style picker (lines `LyricsLivePreview.tsx:61, 64, 96-98, 882`). Until then the Muon iframe is not mounted in this surface.
|
||||
- The four Muon styles (`default | fluid | tendrils | spirit`) are exposed via `liveVizSettings.muon_iframe.style` in the same component.
|
||||
|
||||
Implication:
|
||||
- Mode-by-mode replay matrix is N/A on this surface as currently rendered.
|
||||
- This surface IS testable for the same per-mode replay matrix; the "no controls visible" snapshot reflected the default lyric-style state, not an absence of the path.
|
||||
- For the next reviewer pass: open `/projects/<id>`, switch the visual style to Muon, then run the same cycle1/cycle2 sweep used on Track Detail. If results disagree with Track Detail, the divergence is in `LyricsLivePreview`'s transport wiring vs `VisualizerPreviewWidget`'s — both call the same hook but pass different `getCurrentTime` / `getRate` sources (`stem clock` vs `audioRef`).
|
||||
|
||||
## Delta vs Prior Note
|
||||
See prior Track Detail-focused note for earlier run context:
|
||||
- `docs/muon-mode-status/2026-04-27-current-per-mode-functionality.md`
|
||||
|
||||
This file extends that status into a cross-surface matrix and explicitly marks where tests are blocked or not applicable.
|
||||
|
||||
## 2026-04-28 Phase 6 Delta (Post-Cutover)
|
||||
|
||||
Status: infrastructure validation complete; browser parity sweep still pending.
|
||||
|
||||
What changed since the matrix above:
|
||||
- `tendrils` is no longer served from in-Muon `muon-iframe/tendrils-render-graph.js`.
|
||||
- `tendrils` now resolves to standalone service `stemengine-tendrils` (`http://work:3013/`) via provider routing.
|
||||
- `muon-iframe/tendrils-render-graph.js` was removed from the codebase.
|
||||
|
||||
Automated verification completed:
|
||||
- `docker compose -p stemengine -f stemengine.yml ps stemengine-muon stemengine-fluid stemengine-tendrils`
|
||||
- all three services are up:
|
||||
- `stemengine-muon` on `3011`
|
||||
- `stemengine-fluid` on `3012`
|
||||
- `stemengine-tendrils` on `3013`
|
||||
- endpoint smoke checks:
|
||||
- `curl -fsS http://localhost:3011/ | head -n 3` returned Muon HTML
|
||||
- `curl -fsS http://localhost:3012/ | head -n 3` returned Fluid HTML
|
||||
- `curl -fsS http://localhost:3013/ | head -n 3` returned Tendrils HTML
|
||||
- frontend compile gate:
|
||||
- `cd frontend && npm run -s typecheck` returned `typecheck_exit=0`
|
||||
|
||||
Release-gate note:
|
||||
- This delta confirms deployment and routing health only.
|
||||
- Mode-by-mode replay/reactivity parity on Track Detail, Visualizer Lab, and Project page still requires interactive browser run-through and should be recorded in a follow-up pass.
|
||||
|
||||
## 2026-04-28 Phase 6 Interactive Attempt (Current Session)
|
||||
|
||||
Status: attempted; full parity matrix still blocked in this environment.
|
||||
|
||||
Validated in browser session:
|
||||
- Visualizer Lab style switch updates iframe provider URL as expected:
|
||||
- `default` -> `http://localhost:3011/?preset=default&se_rev=20260427c`
|
||||
- `fluid` -> `http://localhost:3012`
|
||||
- `tendrils` -> `http://localhost:3013`
|
||||
|
||||
Observed blockers:
|
||||
- Track Detail remained stuck on `Loading track…` with repeated browser-console `500` responses.
|
||||
- Project page load for `/projects/e98f2b8a-0cad-4b4e-b33e-1ca2c8c87a0a` did not reach testable preview state and showed repeated `500` API responses.
|
||||
- Visualizer Lab local upload could not be completed from this agent browser run due host-path translation failure in file chooser (`ENOENT` on translated `C:\home\...` path), so play-cycle reactivity assertions could not be executed.
|
||||
|
||||
Captured API evidence from user-facing hosts:
|
||||
- `http://localhost:3001/api/projects/e98f2b8a-0cad-4b4e-b33e-1ca2c8c87a0a` -> `500`
|
||||
- `http://localhost:3001/api/system/gpu-info` -> `500`
|
||||
- `http://localhost:3001/api/tracks/a5f76485ba8f7e5c41aea2c6a41ca0eb` -> `500`
|
||||
- Same three endpoints returned `500` via `http://localhost:5174`.
|
||||
|
||||
Conclusion for this pass:
|
||||
- Provider cutover/routing is verified for `default|fluid|tendrils`.
|
||||
- End-to-end replay/reactivity PASS/FAIL closure remains blocked by backend `500` conditions and upload-path constraints in this browser context.
|
||||
|
||||
## 2026-04-28 Phase 6b Spirit Provider Cutover (Implementation Evidence)
|
||||
|
||||
Status: implemented and deployed; interactive parity scoring still pending.
|
||||
|
||||
What changed:
|
||||
- `spirit` is now served as standalone iframe provider (`stemengine-spirit`) on `http://work:3014/`.
|
||||
- provider runtime lives under `visualizer-iframes/spirit/` and uses upstream `edankwan/The-Spirit` assets.
|
||||
- host transport scripts are injected in Spirit runtime (`bootstrap`, `audio-engine`, `feature-extractor`, `router`).
|
||||
- Spirit-specific adapter implements synthetic audio coupling, aggressive beat mapping, palette propagation, and passive fallback if coupling init cannot lock.
|
||||
|
||||
Automated verification completed:
|
||||
- `docker compose -p stemengine -f stemengine.yml ps stemengine-spirit`
|
||||
- service up on `3014`.
|
||||
- `curl -fsS http://localhost:3014/ | grep -n "STEMENGINE_VIZ_PROVIDER_ID\|stemengine-spirit-prehook\|stemengine-spirit-adapter\|stemengine-router"`
|
||||
- confirms provider declaration and required injected scripts are present in served HTML.
|
||||
- browser smoke check against `http://localhost:3014/?se_rev=spirit-fix2`
|
||||
- page title resolves (`The Spirit`), canvas mounts, no fatal boot crash observed in this pass.
|
||||
|
||||
Runtime caveat captured:
|
||||
- in this agent-browser environment, coupling target discovery did not bind to a detectable Spirit uniform driver during this run.
|
||||
- adapter fallback now switches to input-driver coupling mode (audio-driven virtual pointer path) instead of passive-upstream mode; this keeps Spirit actively reactive while preserving uniform-driver as the preferred path.
|
||||
- full coupling-reactivity scoring still needs a follow-up interactive run in a normal reviewer browser session.
|
||||
@@ -56,11 +56,32 @@ Use this for:
|
||||
- Consistent third-party review rubric
|
||||
- Capturing final approval / caveat decisions
|
||||
|
||||
### 5. Local Iframe Architecture Plan
|
||||
File:
|
||||
- `docs/muon-mode-status/export/muon-mode-review-package-2026-04-28/docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md`
|
||||
|
||||
Use this for:
|
||||
- The implementation path for moving non-default styles onto a local Muon-like iframe flow
|
||||
- Phase-by-phase execution guidance
|
||||
- Host/iframe touch points, risks, and validation gates
|
||||
|
||||
### 6. Visualizer Iframe Protocol Spec
|
||||
File:
|
||||
- `docs/specs/visualizer-iframe-protocol.md`
|
||||
|
||||
Use this for:
|
||||
- The frozen Phase 1 postMessage contract
|
||||
- Current command and event shapes implemented by the Muon iframe transport
|
||||
- Reserved provider-neutral extensions for later phases (`capabilities`, `setParam`, `setPalette`, `dispose`)
|
||||
|
||||
## Recommended Reading Order
|
||||
1. `2026-04-27-current-per-mode-functionality.md`
|
||||
2. `2026-04-28-surface-functionality-matrix.md`
|
||||
3. `2026-04-28-reference-comparison-third-party-review.md`
|
||||
4. `2026-04-28-reviewer-checklist.md`
|
||||
5. `2026-04-28-claude-review-notes.md` — second-pass review by Claude. Catches matrix factual errors (project page mis-marked N/A; lab mis-marked BLOCKED), open questions on the fluid FBO failure root cause, drift between three duplicated `MUON_IFRAME_REV` constants, missing host-side files in the export bundle, and reconciliation between this status package and the 2026-04-27 D-1/D-3/D-5/D-7/D-8 fixes documented in `docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/Fixes.md`.
|
||||
6. `2026-04-28-handoff-to-copilot.md` — **start here if you are the next implementer.** Single entry point for the per-renderer iframe migration. Pins what's decided, what's open, files to create / modify / not touch, order of operations, stop conditions, and the success definition.
|
||||
7. `docs/specs/visualizer-iframe-protocol.md` — frozen Phase 1 transport contract for Muon and the future per-renderer iframe providers.
|
||||
|
||||
## Reviewer Short Summary
|
||||
If a reviewer reads only one page first, start with:
|
||||
@@ -72,6 +93,9 @@ If a reviewer wants only operational status first, start with:
|
||||
If a reviewer is validating reproducibility across UI surfaces, start with:
|
||||
- `2026-04-28-surface-functionality-matrix.md`
|
||||
|
||||
If the next implementer is starting Phase 2 or Phase 3 work, read after the handoff:
|
||||
- `docs/specs/visualizer-iframe-protocol.md`
|
||||
|
||||
## Current Overall Assessment
|
||||
- `fluid`: closest to reference in structure, but currently weakened by replay/FBO reliability issues
|
||||
- `tendrils`: visually in-family, but likely a simplified embedded variant rather than full reference parity
|
||||
|
||||
@@ -65,12 +65,22 @@ Use this for:
|
||||
- Phase-by-phase execution guidance
|
||||
- Host/iframe touch points, risks, and validation gates
|
||||
|
||||
### 6. Visualizer Iframe Protocol Spec
|
||||
File:
|
||||
- `docs/specs/visualizer-iframe-protocol.md`
|
||||
|
||||
Use this for:
|
||||
- The frozen Phase 1 postMessage contract
|
||||
- Current command and event shapes implemented by the Muon iframe transport
|
||||
- Reserved provider-neutral extensions for later phases (`capabilities`, `setParam`, `setPalette`, `dispose`)
|
||||
|
||||
## Recommended Reading Order
|
||||
1. `2026-04-27-current-per-mode-functionality.md`
|
||||
2. `2026-04-28-surface-functionality-matrix.md`
|
||||
3. `2026-04-28-reference-comparison-third-party-review.md`
|
||||
4. `2026-04-28-reviewer-checklist.md`
|
||||
5. `docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md`
|
||||
6. `docs/specs/visualizer-iframe-protocol.md`
|
||||
|
||||
## Reviewer Short Summary
|
||||
If a reviewer reads only one page first, start with:
|
||||
@@ -85,6 +95,9 @@ If a reviewer is validating reproducibility across UI surfaces, start with:
|
||||
If a reviewer is planning implementation work from these findings, continue with:
|
||||
- `docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md`
|
||||
|
||||
If the next implementer is starting Phase 2 or Phase 3 work, read after the plan:
|
||||
- `docs/specs/visualizer-iframe-protocol.md`
|
||||
|
||||
## Current Overall Assessment
|
||||
- `fluid`: closest to reference in structure, but currently weakened by replay/FBO reliability issues
|
||||
- `tendrils`: visually in-family, but likely a simplified embedded variant rather than full reference parity
|
||||
|
||||
+22
-9
@@ -13,7 +13,8 @@ This export bundle is intended for third-party reviewers who need:
|
||||
- surface-by-surface testability notes,
|
||||
- reference-comparison analysis,
|
||||
- a reviewer scoring rubric,
|
||||
- a single index file for navigation.
|
||||
- a single index file for navigation,
|
||||
- the migration plan and transport contract for the per-renderer iframe work.
|
||||
|
||||
## Files To Include
|
||||
|
||||
@@ -36,13 +37,16 @@ This export bundle is intended for third-party reviewers who need:
|
||||
6. `docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md`
|
||||
- Implementation plan for moving non-default styles to a local iframe visualizer runtime.
|
||||
|
||||
7. `docs/specs/visualizer-iframe-protocol.md`
|
||||
- Frozen Phase 1 host/iframe transport contract.
|
||||
|
||||
### Optional supporting source files
|
||||
Include these if the reviewer needs implementation-level evidence:
|
||||
7. `muon-iframe/fluid-render-graph.js`
|
||||
8. `muon-iframe/tendrils-render-graph.js`
|
||||
9. `muon-iframe/spirit-render-graph.js`
|
||||
10. `muon-iframe/stemengine-router.js`
|
||||
11. `muon-iframe/stemengine-bootstrap.js`
|
||||
8. `muon-iframe/fluid-render-graph.js`
|
||||
9. `muon-iframe/tendrils-render-graph.js`
|
||||
10. `muon-iframe/spirit-render-graph.js`
|
||||
11. `muon-iframe/stemengine-router.js`
|
||||
12. `muon-iframe/stemengine-bootstrap.js`
|
||||
|
||||
## Minimal Bundle
|
||||
If the recipient should receive docs only, export:
|
||||
@@ -52,11 +56,13 @@ If the recipient should receive docs only, export:
|
||||
- `docs/muon-mode-status/2026-04-28-reference-comparison-third-party-review.md`
|
||||
- `docs/muon-mode-status/2026-04-28-reviewer-checklist.md`
|
||||
- `docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md`
|
||||
- `docs/specs/visualizer-iframe-protocol.md`
|
||||
|
||||
## Full Evidence Bundle
|
||||
If the recipient should receive both review docs and implementation evidence, export:
|
||||
- all files from the Minimal Bundle
|
||||
- `docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md`
|
||||
- `docs/specs/visualizer-iframe-protocol.md`
|
||||
- `muon-iframe/fluid-render-graph.js`
|
||||
- `muon-iframe/tendrils-render-graph.js`
|
||||
- `muon-iframe/spirit-render-graph.js`
|
||||
@@ -75,6 +81,8 @@ muon-mode-review-package-2026-04-28/
|
||||
2026-04-28-reviewer-checklist.md
|
||||
plans/
|
||||
2026-04-28_local-iframe-visualizer-architecture-plan.md
|
||||
specs/
|
||||
visualizer-iframe-protocol.md
|
||||
source-evidence/
|
||||
fluid-render-graph.js
|
||||
tendrils-render-graph.js
|
||||
@@ -96,9 +104,14 @@ Recommended reading sequence inside the exported bundle:
|
||||
4. `2026-04-28-surface-functionality-matrix.md`
|
||||
5. `2026-04-28-reviewer-checklist.md`
|
||||
6. `docs/plans/2026-04-28_local-iframe-visualizer-architecture-plan.md`
|
||||
7. `docs/specs/visualizer-iframe-protocol.md`
|
||||
|
||||
## Current Package Status
|
||||
As of this manifest:
|
||||
- `fluid` remains the main concern because of replay/FBO reliability issues.
|
||||
- `tendrils` and `spirit` are documented as in-family approximations, not proven literal ports.
|
||||
- Visualizer Lab and project/embed surfaces remain limited or non-testable for this exact replay matrix without additional setup.
|
||||
- Phase 0 is complete: the replay regression root cause was host-side seek churn and the fix shipped.
|
||||
- Phase 1 is complete: the transport contract is frozen in `docs/specs/visualizer-iframe-protocol.md`.
|
||||
- Phase 2 is complete: shared scaffolding moved to `visualizer-iframes/shared/`, Muon consumes it via compose `additional_contexts`, and `stemengine-muon` was rebuilt/recreated successfully.
|
||||
- Phase 4 is complete: `fluid` runs as standalone `stemengine-fluid`, and the in-Muon fluid render graph path was removed.
|
||||
- Phase 5 is complete: `tendrils` runs as standalone `stemengine-tendrils`, and the in-Muon tendrils render graph path was removed.
|
||||
- Phase 6 is in progress: automation/documentation gate checks are recorded; interactive browser matrix and final reviewer verdicts are still pending.
|
||||
- Release-gate note: Tendrils upstream project-level licensing provenance remains unresolved and must be confirmed before ship-ready signoff.
|
||||
+739
-178
File diff suppressed because it is too large.
Load diff
+485
@@ -0,0 +1,485 @@
|
||||
# Visualizer Iframe Protocol
|
||||
|
||||
Date: 2026-04-28
|
||||
Status: Phase 1 baseline frozen
|
||||
Owners: StemEngine frontend / iframe visualizer pipeline
|
||||
|
||||
## Purpose
|
||||
|
||||
This document freezes the host-to-iframe transport contract used by StemEngine visualizer iframes.
|
||||
|
||||
Goals:
|
||||
- preserve the current Muon iframe behavior that is already shipping,
|
||||
- define the provider-neutral contract that future iframe apps must implement,
|
||||
- make replay, seek, rate, resize, and local-upload behavior explicit before Phase 2 and Phase 3 refactors.
|
||||
|
||||
Sources of truth for the current implementation:
|
||||
- `frontend/src/hooks/useMuonIframeTransport.ts`
|
||||
- `muon-iframe/stemengine-router.js`
|
||||
|
||||
This is a protocol document, not a product API promise. Fields marked "planned" are reserved now so later iframe providers can implement them without renaming the contract.
|
||||
|
||||
## Transport Envelope
|
||||
|
||||
All messages use the same envelope shape.
|
||||
|
||||
```ts
|
||||
type VisualizerIframeMessage = {
|
||||
version: 1
|
||||
id: string
|
||||
type: string
|
||||
payload?: Record<string, unknown>
|
||||
}
|
||||
```
|
||||
|
||||
Rules:
|
||||
- `version` is required and is currently fixed at `1`.
|
||||
- `id` is caller-generated and is used for correlation and diagnostics only.
|
||||
- `type` selects the command or event.
|
||||
- `payload` is optional, but most commands and events provide one.
|
||||
- Unknown `type` values must be ignored.
|
||||
- Messages with a version other than `1` must be ignored.
|
||||
|
||||
## Lifecycle
|
||||
|
||||
### Boot sequence
|
||||
|
||||
1. Host mounts iframe.
|
||||
2. Iframe initializes its audio engine and renderer runtime.
|
||||
3. Iframe emits `ready`.
|
||||
4. Host sends settings and paused state.
|
||||
5. Host sends `loadAudio` or `loadAudioData`.
|
||||
6. Iframe emits `loaded` after audio decode succeeds.
|
||||
7. Host drives playback with `transportPlay`, `transportPause`, `transportSeek`, and `transportRate`.
|
||||
|
||||
### Visibility and teardown
|
||||
|
||||
- Host may send `setPaused { paused: true }` when the document is hidden even if transport state remains "playing".
|
||||
- Host currently pauses transport on unmount and on inactive state transitions.
|
||||
- `dispose` is reserved for future explicit teardown, but iframe unmount remains the effective teardown path today.
|
||||
|
||||
## Host To Iframe Commands
|
||||
|
||||
### Implemented now
|
||||
|
||||
#### `setStyle`
|
||||
|
||||
Purpose:
|
||||
- Select visual style inside a multi-style iframe.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ style: 'default' | 'fluid' | 'tendrils' | 'spirit' | string }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- Required for the current Muon iframe.
|
||||
- Single-style providers may treat this as a no-op.
|
||||
|
||||
#### `setPreset`
|
||||
|
||||
Purpose:
|
||||
- Select a named renderer preset.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ preset: string }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- Current Muon router stores the value even if the renderer ignores it.
|
||||
- Future providers may expose preset lists through `capabilities`.
|
||||
|
||||
#### `setQuality`
|
||||
|
||||
Purpose:
|
||||
- Apply a generic quality tier.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ quality: 'low' | 'balanced' | 'high' | string }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- Providers map this to renderer-specific internals.
|
||||
|
||||
#### `setPaused`
|
||||
|
||||
Purpose:
|
||||
- Tell the iframe whether visual updates should behave as paused.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ paused: boolean }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- This is a renderer-visibility hint, not the authoritative audio transport command.
|
||||
- Host uses it for tab visibility and play/pause mirroring.
|
||||
|
||||
#### `setViewport`
|
||||
|
||||
Purpose:
|
||||
- Update CSS and pixel viewport dimensions.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{
|
||||
cssWidth: number
|
||||
cssHeight: number
|
||||
dpr: number
|
||||
}
|
||||
```
|
||||
|
||||
Compatibility note:
|
||||
- Current router also accepts legacy `width` and `height` aliases in the payload and derives pixel width and height internally.
|
||||
|
||||
Semantics:
|
||||
- Viewport updates must not require iframe reload.
|
||||
- Providers are expected to clamp invalid or extreme values.
|
||||
|
||||
#### `pushAudioFeatures`
|
||||
|
||||
Purpose:
|
||||
- Push host-side extracted audio features into the iframe render loop.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{
|
||||
bass?: number
|
||||
treble?: number
|
||||
rms?: number
|
||||
energy?: number
|
||||
beat?: boolean
|
||||
drumIntensity?: number
|
||||
sub?: number
|
||||
beatPhase?: number
|
||||
bpm?: number
|
||||
beatConfidence?: number
|
||||
isBeatDrop?: boolean
|
||||
isBuildUp?: boolean
|
||||
isSilence?: boolean
|
||||
energyDelta?: number
|
||||
spectralCentroid?: number
|
||||
spectralFlatness?: number
|
||||
frequencyBands?: unknown
|
||||
}
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- Host may send at RAF cadence.
|
||||
- Iframe should rate-limit internal application so message handling does not block rendering.
|
||||
|
||||
#### `loadAudio`
|
||||
|
||||
Purpose:
|
||||
- Load audio by URL inside the iframe.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ url: string }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- Used for same-origin or cross-origin URLs the iframe can fetch directly.
|
||||
|
||||
#### `loadAudioData`
|
||||
|
||||
Purpose:
|
||||
- Load audio from raw bytes when the iframe cannot fetch the source URL directly.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ bytes: ArrayBuffer | Uint8Array | number[] }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- This is the required path for host-local `blob:` uploads that do not survive a cross-origin iframe boundary.
|
||||
- The host currently uses this for Visualizer Lab local uploads.
|
||||
- Providers that support local uploads must support this command.
|
||||
|
||||
#### `transportPlay`
|
||||
|
||||
Purpose:
|
||||
- Start or resume transport at a host-selected time and rate.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ time: number; rate: number }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- This is the authoritative start/resume command.
|
||||
- Iframe must begin playback at `time` and apply `rate` atomically if possible.
|
||||
|
||||
#### `transportPause`
|
||||
|
||||
Purpose:
|
||||
- Pause transport.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{}
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- This is the authoritative pause command.
|
||||
|
||||
#### `transportSeek`
|
||||
|
||||
Purpose:
|
||||
- Correct transport position without changing the selected source.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ time: number; rate?: number }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- Used for discontinuities only, not as a steady-state clock.
|
||||
- If the iframe is logically playing, seek must not leave it visually or audibly paused.
|
||||
- Current Muon router includes a recovery path: if the iframe is expected to be playing but the engine is stopped, a seek may trigger a single auto-resume using the supplied or current rate.
|
||||
|
||||
#### `transportRate`
|
||||
|
||||
Purpose:
|
||||
- Update playback rate while playing or paused.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ rate: number }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- Providers should update rate without forcing a reload.
|
||||
|
||||
### Reserved in Phase 1, to be implemented in later phases
|
||||
|
||||
#### `setParam`
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ name: string; value: number | string | boolean }
|
||||
```
|
||||
|
||||
Purpose:
|
||||
- Apply a renderer-specific tunable control.
|
||||
|
||||
Status:
|
||||
- Reserved by this spec.
|
||||
- Not implemented in the current Muon router.
|
||||
|
||||
#### `setPalette`
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ colors: string[] }
|
||||
```
|
||||
|
||||
Purpose:
|
||||
- Apply host-selected palette colors to the renderer.
|
||||
|
||||
Status:
|
||||
- Reserved by this spec.
|
||||
- Not implemented in the current Muon router.
|
||||
|
||||
Palette rules:
|
||||
- Colors must be hex strings.
|
||||
- Providers may clamp or ignore unsupported slots.
|
||||
- Palette changes must not require reload.
|
||||
|
||||
#### `dispose`
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{}
|
||||
```
|
||||
|
||||
Purpose:
|
||||
- Provide an explicit teardown hook for future providers.
|
||||
|
||||
Status:
|
||||
- Reserved by this spec.
|
||||
- Optional even after implementation because iframe unmount remains the primary teardown path.
|
||||
|
||||
## Iframe To Host Events
|
||||
|
||||
### Implemented now
|
||||
|
||||
#### `ready`
|
||||
|
||||
Purpose:
|
||||
- Signal that the iframe runtime is initialized and can receive settings and audio commands.
|
||||
|
||||
Current payload shape:
|
||||
|
||||
```ts
|
||||
{
|
||||
capabilities?: {
|
||||
webgl?: boolean
|
||||
missingExtensions?: string[]
|
||||
audioMode?: string
|
||||
}
|
||||
styles?: string[]
|
||||
}
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- Host should treat `ready` as the gate for sending settings and audio load commands.
|
||||
- Host must tolerate provider-specific extra fields.
|
||||
|
||||
#### `loaded`
|
||||
|
||||
Purpose:
|
||||
- Confirm that the iframe decoded the current audio source.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ duration: number }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- `duration` may be `0` if unknown.
|
||||
- Host uses this to mark the iframe as loaded.
|
||||
|
||||
#### `error`
|
||||
|
||||
Purpose:
|
||||
- Report an iframe-side failure.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ code: string; message: string }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- `AUDIO_LOAD` is currently treated by the host as a non-actionable runtime warning and may be suppressed in the console.
|
||||
- Other error codes are diagnostic and should be logged.
|
||||
|
||||
### Reserved in Phase 1, to be implemented in later phases
|
||||
|
||||
#### `capabilities`
|
||||
|
||||
Purpose:
|
||||
- Describe renderer-specific presets, params, and palette support after initialization.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{
|
||||
providerId?: string
|
||||
protocolVersion?: 1
|
||||
presets: string[]
|
||||
params: Param[]
|
||||
palette: {
|
||||
supported: boolean
|
||||
slotCount: number
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```ts
|
||||
type Param = {
|
||||
name: string
|
||||
label: string
|
||||
type: 'number' | 'enum' | 'boolean'
|
||||
default: number | string | boolean
|
||||
min?: number
|
||||
max?: number
|
||||
step?: number
|
||||
options?: string[]
|
||||
}
|
||||
```
|
||||
|
||||
Status:
|
||||
- Reserved by this spec.
|
||||
- Not emitted by the current Muon router.
|
||||
|
||||
Semantics:
|
||||
- Host must tolerate this arriving after render start.
|
||||
- Host must also tolerate it never arriving and fall back to generic controls.
|
||||
|
||||
## Behavioral Requirements
|
||||
|
||||
### Replay and seek
|
||||
|
||||
- Replay after pause or track restart must be deterministic.
|
||||
- `transportSeek` is for discontinuities, scrub actions, and drift correction beyond tolerance.
|
||||
- Steady playback must not emit continuous seek churn.
|
||||
- If a seek arrives while the host is logically playing, the iframe must remain able to resume visual and audio progression without a reload.
|
||||
|
||||
### Rate handling
|
||||
|
||||
- `transportPlay` carries the starting rate.
|
||||
- `transportRate` carries subsequent rate changes.
|
||||
- `transportSeek` may also carry `rate` so a recovery path can resume using the correct speed.
|
||||
|
||||
### Resize and quality
|
||||
|
||||
- `setViewport` and `setQuality` must be hot-applied.
|
||||
- Providers may rebuild internal framebuffers on viewport change, but must not require document reload.
|
||||
|
||||
### Audio source ownership
|
||||
|
||||
- Host owns source selection.
|
||||
- Iframe owns decode, playback, and renderer-side consumption.
|
||||
- `loadAudioData` is the required compatibility bridge for host-local uploads.
|
||||
|
||||
## Provider-Neutral Rules
|
||||
|
||||
- The host contract must not rely on Muon-specific naming beyond historical file names.
|
||||
- Multi-style providers may implement `setStyle`; single-style providers may ignore it.
|
||||
- All providers must understand the same envelope, core transport commands, and core events.
|
||||
- Future providers should emit `capabilities` and implement `setParam` and `setPalette`.
|
||||
|
||||
## Compatibility Matrix
|
||||
|
||||
| Message | Current Muon iframe | Required for future providers |
|
||||
|---|---|---|
|
||||
| `setStyle` | Yes | Optional no-op for single-style providers |
|
||||
| `setPreset` | Yes | Yes |
|
||||
| `setQuality` | Yes | Yes |
|
||||
| `setPaused` | Yes | Yes |
|
||||
| `setViewport` | Yes | Yes |
|
||||
| `pushAudioFeatures` | Yes | Yes if renderer uses host-pushed features |
|
||||
| `loadAudio` | Yes | Yes |
|
||||
| `loadAudioData` | Yes | Yes |
|
||||
| `transportPlay` | Yes | Yes |
|
||||
| `transportPause` | Yes | Yes |
|
||||
| `transportSeek` | Yes | Yes |
|
||||
| `transportRate` | Yes | Yes |
|
||||
| `setParam` | Reserved | Yes, after Phase 3 |
|
||||
| `setPalette` | Reserved | Yes, after Phase 3 |
|
||||
| `dispose` | Reserved | Optional |
|
||||
| `ready` | Yes | Yes |
|
||||
| `loaded` | Yes | Yes |
|
||||
| `error` | Yes | Yes |
|
||||
| `capabilities` | Reserved | Yes, after Phase 3 |
|
||||
|
||||
## Validation Baseline For This Freeze
|
||||
|
||||
The Phase 1 freeze is based on the currently implemented contract in:
|
||||
- `frontend/src/hooks/useMuonIframeTransport.ts`
|
||||
- `muon-iframe/stemengine-router.js`
|
||||
|
||||
Phase 0 validation remains the behavioral baseline:
|
||||
- replay cycle 2 is reactive again,
|
||||
- steady playback no longer produces seek churn,
|
||||
- local uploads cross the host/iframe boundary via `loadAudioData`.
|
||||
@@ -344,3 +344,37 @@ This plan succeeds when:
|
||||
- replay behavior is deterministic for all supported styles,
|
||||
- failures are surfaced explicitly instead of appearing as frozen visuals,
|
||||
- reviewers can evaluate styles through a consistent iframe lifecycle rather than special-case paths.
|
||||
|
||||
## 2026-04-28 Phase 6b Spirit Implementation Delta
|
||||
|
||||
Status: implemented.
|
||||
|
||||
Implemented deliverables:
|
||||
- added standalone Spirit iframe runtime under `visualizer-iframes/spirit/`:
|
||||
- `Dockerfile` (upstream pull from `edankwan/The-Spirit` at `SPIRIT_GIT_REF`)
|
||||
- `index.html` (provider/capability declaration + shared transport script injection)
|
||||
- `nginx.conf`
|
||||
- `stemengine-spirit-prehook.js` (runtime discovery hook for Spirit materials)
|
||||
- `stemengine-spirit-adapter.js` (synthetic audio coupling + aggressive beat mapping + palette propagation)
|
||||
- added `stemengine-spirit` service to `docker/stemengine.yml` on host port `3014`.
|
||||
|
||||
Decision-policy alignment:
|
||||
- upstream Spirit path shipped as standalone provider.
|
||||
- synthetic coupling adapter implemented using `window.__STEMENGINE_VIZ_AUDIO__` with aggressive beat envelope.
|
||||
- `setPalette` propagation implemented (updates Spirit color uniforms and background atmosphere).
|
||||
- fallback behavior updated: if uniform-driver coupling cannot be discovered/initialized in runtime, adapter switches to audio-driven input-driver coupling mode; passive upstream remains last-resort only.
|
||||
- provider label remains `spirit` (displayed as `Spirit` in host UI).
|
||||
|
||||
Validation evidence in this pass:
|
||||
- `docker compose -p stemengine -f stemengine.yml build stemengine-spirit` succeeded.
|
||||
- `docker compose ... up -d --no-deps --force-recreate stemengine-spirit` succeeded.
|
||||
- `docker compose ... ps stemengine-spirit` shows service healthy on `3014->80`.
|
||||
- endpoint probe confirms injected scripts and provider declaration in served HTML:
|
||||
- `__STEMENGINE_VIZ_PROVIDER_ID__ = 'spirit'`
|
||||
- `stemengine-spirit-prehook.js`
|
||||
- `stemengine-router.js`
|
||||
- `stemengine-spirit-adapter.js`
|
||||
|
||||
Current caveat:
|
||||
- in the agent-browser harness, coupling driver discovery still did not lock to a detected Spirit uniform target during this pass.
|
||||
- runtime now escalates to input-driver coupling mode in that condition (instead of passive-upstream fallback) and explicitly emits host warning telemetry (`SPIRIT_COUPLING_FALLBACK`) for observability.
|
||||
@@ -0,0 +1,485 @@
|
||||
# Visualizer Iframe Protocol
|
||||
|
||||
Date: 2026-04-28
|
||||
Status: Phase 1 baseline frozen
|
||||
Owners: StemEngine frontend / iframe visualizer pipeline
|
||||
|
||||
## Purpose
|
||||
|
||||
This document freezes the host-to-iframe transport contract used by StemEngine visualizer iframes.
|
||||
|
||||
Goals:
|
||||
- preserve the current Muon iframe behavior that is already shipping,
|
||||
- define the provider-neutral contract that future iframe apps must implement,
|
||||
- make replay, seek, rate, resize, and local-upload behavior explicit before Phase 2 and Phase 3 refactors.
|
||||
|
||||
Sources of truth for the current implementation:
|
||||
- `frontend/src/hooks/useMuonIframeTransport.ts`
|
||||
- `muon-iframe/stemengine-router.js`
|
||||
|
||||
This is a protocol document, not a product API promise. Fields marked "planned" are reserved now so later iframe providers can implement them without renaming the contract.
|
||||
|
||||
## Transport Envelope
|
||||
|
||||
All messages use the same envelope shape.
|
||||
|
||||
```ts
|
||||
type VisualizerIframeMessage = {
|
||||
version: 1
|
||||
id: string
|
||||
type: string
|
||||
payload?: Record<string, unknown>
|
||||
}
|
||||
```
|
||||
|
||||
Rules:
|
||||
- `version` is required and is currently fixed at `1`.
|
||||
- `id` is caller-generated and is used for correlation and diagnostics only.
|
||||
- `type` selects the command or event.
|
||||
- `payload` is optional, but most commands and events provide one.
|
||||
- Unknown `type` values must be ignored.
|
||||
- Messages with a version other than `1` must be ignored.
|
||||
|
||||
## Lifecycle
|
||||
|
||||
### Boot sequence
|
||||
|
||||
1. Host mounts iframe.
|
||||
2. Iframe initializes its audio engine and renderer runtime.
|
||||
3. Iframe emits `ready`.
|
||||
4. Host sends settings and paused state.
|
||||
5. Host sends `loadAudio` or `loadAudioData`.
|
||||
6. Iframe emits `loaded` after audio decode succeeds.
|
||||
7. Host drives playback with `transportPlay`, `transportPause`, `transportSeek`, and `transportRate`.
|
||||
|
||||
### Visibility and teardown
|
||||
|
||||
- Host may send `setPaused { paused: true }` when the document is hidden even if transport state remains "playing".
|
||||
- Host currently pauses transport on unmount and on inactive state transitions.
|
||||
- `dispose` is reserved for future explicit teardown, but iframe unmount remains the effective teardown path today.
|
||||
|
||||
## Host To Iframe Commands
|
||||
|
||||
### Implemented now
|
||||
|
||||
#### `setStyle`
|
||||
|
||||
Purpose:
|
||||
- Select visual style inside a multi-style iframe.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ style: 'default' | 'fluid' | 'tendrils' | 'spirit' | string }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- Required for the current Muon iframe.
|
||||
- Single-style providers may treat this as a no-op.
|
||||
|
||||
#### `setPreset`
|
||||
|
||||
Purpose:
|
||||
- Select a named renderer preset.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ preset: string }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- Current Muon router stores the value even if the renderer ignores it.
|
||||
- Future providers may expose preset lists through `capabilities`.
|
||||
|
||||
#### `setQuality`
|
||||
|
||||
Purpose:
|
||||
- Apply a generic quality tier.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ quality: 'low' | 'balanced' | 'high' | string }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- Providers map this to renderer-specific internals.
|
||||
|
||||
#### `setPaused`
|
||||
|
||||
Purpose:
|
||||
- Tell the iframe whether visual updates should behave as paused.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ paused: boolean }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- This is a renderer-visibility hint, not the authoritative audio transport command.
|
||||
- Host uses it for tab visibility and play/pause mirroring.
|
||||
|
||||
#### `setViewport`
|
||||
|
||||
Purpose:
|
||||
- Update CSS and pixel viewport dimensions.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{
|
||||
cssWidth: number
|
||||
cssHeight: number
|
||||
dpr: number
|
||||
}
|
||||
```
|
||||
|
||||
Compatibility note:
|
||||
- Current router also accepts legacy `width` and `height` aliases in the payload and derives pixel width and height internally.
|
||||
|
||||
Semantics:
|
||||
- Viewport updates must not require iframe reload.
|
||||
- Providers are expected to clamp invalid or extreme values.
|
||||
|
||||
#### `pushAudioFeatures`
|
||||
|
||||
Purpose:
|
||||
- Push host-side extracted audio features into the iframe render loop.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{
|
||||
bass?: number
|
||||
treble?: number
|
||||
rms?: number
|
||||
energy?: number
|
||||
beat?: boolean
|
||||
drumIntensity?: number
|
||||
sub?: number
|
||||
beatPhase?: number
|
||||
bpm?: number
|
||||
beatConfidence?: number
|
||||
isBeatDrop?: boolean
|
||||
isBuildUp?: boolean
|
||||
isSilence?: boolean
|
||||
energyDelta?: number
|
||||
spectralCentroid?: number
|
||||
spectralFlatness?: number
|
||||
frequencyBands?: unknown
|
||||
}
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- Host may send at RAF cadence.
|
||||
- Iframe should rate-limit internal application so message handling does not block rendering.
|
||||
|
||||
#### `loadAudio`
|
||||
|
||||
Purpose:
|
||||
- Load audio by URL inside the iframe.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ url: string }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- Used for same-origin or cross-origin URLs the iframe can fetch directly.
|
||||
|
||||
#### `loadAudioData`
|
||||
|
||||
Purpose:
|
||||
- Load audio from raw bytes when the iframe cannot fetch the source URL directly.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ bytes: ArrayBuffer | Uint8Array | number[] }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- This is the required path for host-local `blob:` uploads that do not survive a cross-origin iframe boundary.
|
||||
- The host currently uses this for Visualizer Lab local uploads.
|
||||
- Providers that support local uploads must support this command.
|
||||
|
||||
#### `transportPlay`
|
||||
|
||||
Purpose:
|
||||
- Start or resume transport at a host-selected time and rate.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ time: number; rate: number }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- This is the authoritative start/resume command.
|
||||
- Iframe must begin playback at `time` and apply `rate` atomically if possible.
|
||||
|
||||
#### `transportPause`
|
||||
|
||||
Purpose:
|
||||
- Pause transport.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{}
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- This is the authoritative pause command.
|
||||
|
||||
#### `transportSeek`
|
||||
|
||||
Purpose:
|
||||
- Correct transport position without changing the selected source.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ time: number; rate?: number }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- Used for discontinuities only, not as a steady-state clock.
|
||||
- If the iframe is logically playing, seek must not leave it visually or audibly paused.
|
||||
- Current Muon router includes a recovery path: if the iframe is expected to be playing but the engine is stopped, a seek may trigger a single auto-resume using the supplied or current rate.
|
||||
|
||||
#### `transportRate`
|
||||
|
||||
Purpose:
|
||||
- Update playback rate while playing or paused.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ rate: number }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- Providers should update rate without forcing a reload.
|
||||
|
||||
### Reserved in Phase 1, to be implemented in later phases
|
||||
|
||||
#### `setParam`
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ name: string; value: number | string | boolean }
|
||||
```
|
||||
|
||||
Purpose:
|
||||
- Apply a renderer-specific tunable control.
|
||||
|
||||
Status:
|
||||
- Reserved by this spec.
|
||||
- Not implemented in the current Muon router.
|
||||
|
||||
#### `setPalette`
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ colors: string[] }
|
||||
```
|
||||
|
||||
Purpose:
|
||||
- Apply host-selected palette colors to the renderer.
|
||||
|
||||
Status:
|
||||
- Reserved by this spec.
|
||||
- Not implemented in the current Muon router.
|
||||
|
||||
Palette rules:
|
||||
- Colors must be hex strings.
|
||||
- Providers may clamp or ignore unsupported slots.
|
||||
- Palette changes must not require reload.
|
||||
|
||||
#### `dispose`
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{}
|
||||
```
|
||||
|
||||
Purpose:
|
||||
- Provide an explicit teardown hook for future providers.
|
||||
|
||||
Status:
|
||||
- Reserved by this spec.
|
||||
- Optional even after implementation because iframe unmount remains the primary teardown path.
|
||||
|
||||
## Iframe To Host Events
|
||||
|
||||
### Implemented now
|
||||
|
||||
#### `ready`
|
||||
|
||||
Purpose:
|
||||
- Signal that the iframe runtime is initialized and can receive settings and audio commands.
|
||||
|
||||
Current payload shape:
|
||||
|
||||
```ts
|
||||
{
|
||||
capabilities?: {
|
||||
webgl?: boolean
|
||||
missingExtensions?: string[]
|
||||
audioMode?: string
|
||||
}
|
||||
styles?: string[]
|
||||
}
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- Host should treat `ready` as the gate for sending settings and audio load commands.
|
||||
- Host must tolerate provider-specific extra fields.
|
||||
|
||||
#### `loaded`
|
||||
|
||||
Purpose:
|
||||
- Confirm that the iframe decoded the current audio source.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ duration: number }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- `duration` may be `0` if unknown.
|
||||
- Host uses this to mark the iframe as loaded.
|
||||
|
||||
#### `error`
|
||||
|
||||
Purpose:
|
||||
- Report an iframe-side failure.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{ code: string; message: string }
|
||||
```
|
||||
|
||||
Semantics:
|
||||
- `AUDIO_LOAD` is currently treated by the host as a non-actionable runtime warning and may be suppressed in the console.
|
||||
- Other error codes are diagnostic and should be logged.
|
||||
|
||||
### Reserved in Phase 1, to be implemented in later phases
|
||||
|
||||
#### `capabilities`
|
||||
|
||||
Purpose:
|
||||
- Describe renderer-specific presets, params, and palette support after initialization.
|
||||
|
||||
Payload:
|
||||
|
||||
```ts
|
||||
{
|
||||
providerId?: string
|
||||
protocolVersion?: 1
|
||||
presets: string[]
|
||||
params: Param[]
|
||||
palette: {
|
||||
supported: boolean
|
||||
slotCount: number
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```ts
|
||||
type Param = {
|
||||
name: string
|
||||
label: string
|
||||
type: 'number' | 'enum' | 'boolean'
|
||||
default: number | string | boolean
|
||||
min?: number
|
||||
max?: number
|
||||
step?: number
|
||||
options?: string[]
|
||||
}
|
||||
```
|
||||
|
||||
Status:
|
||||
- Reserved by this spec.
|
||||
- Not emitted by the current Muon router.
|
||||
|
||||
Semantics:
|
||||
- Host must tolerate this arriving after render start.
|
||||
- Host must also tolerate it never arriving and fall back to generic controls.
|
||||
|
||||
## Behavioral Requirements
|
||||
|
||||
### Replay and seek
|
||||
|
||||
- Replay after pause or track restart must be deterministic.
|
||||
- `transportSeek` is for discontinuities, scrub actions, and drift correction beyond tolerance.
|
||||
- Steady playback must not emit continuous seek churn.
|
||||
- If a seek arrives while the host is logically playing, the iframe must remain able to resume visual and audio progression without a reload.
|
||||
|
||||
### Rate handling
|
||||
|
||||
- `transportPlay` carries the starting rate.
|
||||
- `transportRate` carries subsequent rate changes.
|
||||
- `transportSeek` may also carry `rate` so a recovery path can resume using the correct speed.
|
||||
|
||||
### Resize and quality
|
||||
|
||||
- `setViewport` and `setQuality` must be hot-applied.
|
||||
- Providers may rebuild internal framebuffers on viewport change, but must not require document reload.
|
||||
|
||||
### Audio source ownership
|
||||
|
||||
- Host owns source selection.
|
||||
- Iframe owns decode, playback, and renderer-side consumption.
|
||||
- `loadAudioData` is the required compatibility bridge for host-local uploads.
|
||||
|
||||
## Provider-Neutral Rules
|
||||
|
||||
- The host contract must not rely on Muon-specific naming beyond historical file names.
|
||||
- Multi-style providers may implement `setStyle`; single-style providers may ignore it.
|
||||
- All providers must understand the same envelope, core transport commands, and core events.
|
||||
- Future providers should emit `capabilities` and implement `setParam` and `setPalette`.
|
||||
|
||||
## Compatibility Matrix
|
||||
|
||||
| Message | Current Muon iframe | Required for future providers |
|
||||
|---|---|---|
|
||||
| `setStyle` | Yes | Optional no-op for single-style providers |
|
||||
| `setPreset` | Yes | Yes |
|
||||
| `setQuality` | Yes | Yes |
|
||||
| `setPaused` | Yes | Yes |
|
||||
| `setViewport` | Yes | Yes |
|
||||
| `pushAudioFeatures` | Yes | Yes if renderer uses host-pushed features |
|
||||
| `loadAudio` | Yes | Yes |
|
||||
| `loadAudioData` | Yes | Yes |
|
||||
| `transportPlay` | Yes | Yes |
|
||||
| `transportPause` | Yes | Yes |
|
||||
| `transportSeek` | Yes | Yes |
|
||||
| `transportRate` | Yes | Yes |
|
||||
| `setParam` | Reserved | Yes, after Phase 3 |
|
||||
| `setPalette` | Reserved | Yes, after Phase 3 |
|
||||
| `dispose` | Reserved | Optional |
|
||||
| `ready` | Yes | Yes |
|
||||
| `loaded` | Yes | Yes |
|
||||
| `error` | Yes | Yes |
|
||||
| `capabilities` | Reserved | Yes, after Phase 3 |
|
||||
|
||||
## Validation Baseline For This Freeze
|
||||
|
||||
The Phase 1 freeze is based on the currently implemented contract in:
|
||||
- `frontend/src/hooks/useMuonIframeTransport.ts`
|
||||
- `muon-iframe/stemengine-router.js`
|
||||
|
||||
Phase 0 validation remains the behavioral baseline:
|
||||
- replay cycle 2 is reactive again,
|
||||
- steady playback no longer produces seek churn,
|
||||
- local uploads cross the host/iframe boundary via `loadAudioData`.
|
||||
@@ -1107,6 +1107,9 @@ export interface AppSettings {
|
||||
karaoke_sek_folder?: string
|
||||
music_library_path?: string
|
||||
muon_iframe_url?: string
|
||||
fluid_iframe_url?: string
|
||||
tendrils_iframe_url?: string
|
||||
spirit_iframe_url?: string
|
||||
muon_quality?: 'low' | 'balanced' | 'high'
|
||||
singui_process_mode?: 'auto' | 'separate_only' | 'pipeline' | 'preset'
|
||||
singui_process_preset?: string
|
||||
|
||||
@@ -308,7 +308,7 @@ function lineStride(style: StyleState, hasGloss: boolean): number {
|
||||
const base = style.fontSize * style.lineSpacing * LINE_STRIDE_FACTOR
|
||||
if (!hasGloss) return base
|
||||
const glossFontSize = Math.max(10, Math.round(style.fontSize * 0.45))
|
||||
return base + glossFontSize + 4
|
||||
return base + glossFontSize + 8
|
||||
}
|
||||
|
||||
function activeLineAnchorY(height: number, align: StyleState['activeLineVerticalAlign']): number {
|
||||
@@ -402,10 +402,26 @@ function buildCanvasWordUnits(
|
||||
const words = line.words.length > 0 ? line.words : [{ time: line.time, text: line.text, isWordEnd: true }]
|
||||
|
||||
metricsCtx.font = canvasFontString(style)
|
||||
const lyricFont = metricsCtx.font
|
||||
const glossFontSize = Math.max(10, Math.round(style.fontSize * 0.45))
|
||||
const glossFont = `${style.fontBold ? 700 : 400} ${glossFontSize}px ${style.fontFamily}`
|
||||
|
||||
const lyricWordWidths = words.map(word => measureCanvasWord(word.text, metricsCtx, metricsCache))
|
||||
const glossWordWidths = words.map(word => {
|
||||
if (!word.gloss) return 0
|
||||
const prevFont = metricsCtx.font
|
||||
metricsCtx.font = glossFont
|
||||
const width = measureCanvasWord(word.gloss, metricsCtx, metricsCache)
|
||||
metricsCtx.font = prevFont
|
||||
return width
|
||||
})
|
||||
metricsCtx.font = lyricFont
|
||||
|
||||
const wordLayoutWidths = words.map((_, index) => Math.max(lyricWordWidths[index], glossWordWidths[index]))
|
||||
const spaceWidth = measureCanvasWord(' ', metricsCtx, metricsCache)
|
||||
const totalWidth = words.reduce((sum, word, index) => {
|
||||
const addSpace = index < words.length - 1 && word.isWordEnd !== false
|
||||
return sum + measureCanvasWord(word.text, metricsCtx, metricsCache) + (addSpace ? spaceWidth : 0)
|
||||
return sum + wordLayoutWidths[index] + (addSpace ? spaceWidth : 0)
|
||||
}, 0)
|
||||
|
||||
const needsWordUnits = visualStyle === 'karaoke' || visualStyle === 'glow' || visualStyle === 'blur' || style.effectGlowEnabled || style.effectBlurEnabled
|
||||
@@ -449,7 +465,7 @@ function buildCanvasWordUnits(
|
||||
let rowStart = 0
|
||||
let rowWidth = 0
|
||||
for (let wi = 0; wi < words.length; wi++) {
|
||||
const wordWidth = measureCanvasWord(words[wi].text, metricsCtx, metricsCache)
|
||||
const wordWidth = wordLayoutWidths[wi]
|
||||
const prevEnd = rowWords.length > 0 && words[wi - 1]?.isWordEnd !== false
|
||||
const spaceBefore = rowWords.length > 0 && prevEnd ? spaceWidth : 0
|
||||
const needed = spaceBefore + wordWidth
|
||||
@@ -466,8 +482,7 @@ function buildCanvasWordUnits(
|
||||
if (rowWords.length > 0) subRows.push({ words: rowWords, startIdx: rowStart })
|
||||
}
|
||||
|
||||
const glossFontSize = Math.max(10, Math.round(style.fontSize * 0.45))
|
||||
const glossYOffset = hasGloss ? glossFontSize + 4 : 0
|
||||
const glossYOffset = hasGloss ? glossFontSize + 8 : 0
|
||||
const stride = lineStride(style, hasGloss)
|
||||
|
||||
for (let rowIdx = 0; rowIdx < subRows.length; rowIdx++) {
|
||||
@@ -475,7 +490,7 @@ function buildCanvasWordUnits(
|
||||
const rowY = rowIdx * stride + glossYOffset
|
||||
const rowTotalWidth = rowWords.reduce((sum, word, index) => {
|
||||
const addSpace = index < rowWords.length - 1 && word.isWordEnd !== false
|
||||
return sum + measureCanvasWord(word.text, metricsCtx, metricsCache) + (addSpace ? spaceWidth : 0)
|
||||
return sum + wordLayoutWidths[startIdx + index] + (addSpace ? spaceWidth : 0)
|
||||
}, 0)
|
||||
let cursorX = style.textAlign === 'left'
|
||||
? -(two.width / 2) + 24
|
||||
@@ -486,10 +501,11 @@ function buildCanvasWordUnits(
|
||||
for (let i = 0; i < rowWords.length; i++) {
|
||||
const word = rowWords[i]
|
||||
const globalWordIndex = startIdx + i
|
||||
const wordWidth = measureCanvasWord(word.text, metricsCtx, metricsCache)
|
||||
const wordX = cursorX + wordWidth / 2
|
||||
const lyricWordWidth = lyricWordWidths[globalWordIndex]
|
||||
const wordLayoutWidth = wordLayoutWidths[globalWordIndex]
|
||||
const wordX = cursorX + wordLayoutWidth / 2
|
||||
const isWordEnd = word.isWordEnd !== false
|
||||
cursorX += wordWidth + (i < rowWords.length - 1 && isWordEnd ? spaceWidth : 0)
|
||||
cursorX += wordLayoutWidth + (i < rowWords.length - 1 && isWordEnd ? spaceWidth : 0)
|
||||
|
||||
const rawEnd = globalWordIndex + 1 < words.length
|
||||
? words[globalWordIndex + 1].time
|
||||
@@ -533,22 +549,23 @@ function buildCanvasWordUnits(
|
||||
let glossFrontText: InstanceType<typeof Two.Text> | null = null
|
||||
if (word.gloss) {
|
||||
const glossY = rowY - glossYOffset
|
||||
const glossStrokeWidth = style.strokeEnabled ? style.strokeWidth * (glossFontSize / style.fontSize) : 0
|
||||
glossBehindText = createTwoText(word.gloss, wordX, glossY, {
|
||||
fill: lineTextColor,
|
||||
stroke: 'none',
|
||||
linewidth: 0,
|
||||
stroke: style.strokeEnabled ? style.strokeColor : 'none',
|
||||
linewidth: glossStrokeWidth * 2,
|
||||
size: glossFontSize,
|
||||
family: style.fontFamily,
|
||||
weight: style.fontBold ? 600 : 500,
|
||||
weight: style.fontBold ? 700 : 400,
|
||||
alignment: 'center',
|
||||
})
|
||||
glossFrontText = createTwoText(word.gloss, wordX, glossY, {
|
||||
fill: lineTextColor,
|
||||
stroke: 'none',
|
||||
linewidth: 0,
|
||||
stroke: style.strokeEnabled ? style.strokeColor : 'none',
|
||||
linewidth: glossStrokeWidth * 2,
|
||||
size: glossFontSize,
|
||||
family: style.fontFamily,
|
||||
weight: style.fontBold ? 600 : 500,
|
||||
weight: style.fontBold ? 700 : 400,
|
||||
alignment: 'center',
|
||||
})
|
||||
behindTexts.push(glossBehindText)
|
||||
@@ -579,7 +596,7 @@ function buildCanvasWordUnits(
|
||||
activeColor: lineActiveColor,
|
||||
behindColor: lineTextColor,
|
||||
perWord: true,
|
||||
measuredWidth: wordWidth,
|
||||
measuredWidth: lyricWordWidth,
|
||||
startTime: word.time,
|
||||
duration,
|
||||
})
|
||||
@@ -848,9 +865,13 @@ function activateCanvasLineGroup(
|
||||
|
||||
for (const wordUnit of record.wordUnits) {
|
||||
if (wordUnit.shadowFilterId) applyFilter(wordUnit.behindText, wordUnit.shadowFilterId)
|
||||
if (wordUnit.shadowFilterId && wordUnit.glossBehindText) applyFilter(wordUnit.glossBehindText, wordUnit.shadowFilterId)
|
||||
const behindNode = getTwoObjectSvgElement(wordUnit.behindText) ?? undefined
|
||||
if (behindNode) behindNode.setAttribute('paint-order', 'stroke fill')
|
||||
if (behindNode) behindNode.setAttribute('text-rendering', 'geometricPrecision')
|
||||
const glossBehindNode = wordUnit.glossBehindText ? (getTwoObjectSvgElement(wordUnit.glossBehindText) ?? undefined) : undefined
|
||||
if (glossBehindNode) glossBehindNode.setAttribute('paint-order', 'stroke fill')
|
||||
if (glossBehindNode) glossBehindNode.setAttribute('text-rendering', 'geometricPrecision')
|
||||
if (wordUnit.gradId) {
|
||||
wordUnit.frontElem = getTwoObjectSvgElement(wordUnit.frontText)
|
||||
if (wordUnit.frontElem) {
|
||||
|
||||
@@ -9,11 +9,12 @@ const KARAOKE_TOGGLE_EVENT = 'stemengine:karaoke-toggle-playback'
|
||||
import KaraokeEditor from './KaraokeEditor'
|
||||
import { useVisualizerPalette } from '../hooks/useVisualizerPalette'
|
||||
import { useAudioFeatures } from '../hooks/useAudioFeatures'
|
||||
import { useMuonIframeTransport } from '../hooks/useMuonIframeTransport'
|
||||
import { useVisualizerIframeTransport, type VisualizerParamValue } from '../hooks/useVisualizerIframeTransport'
|
||||
import { useStemAudio } from '../hooks/useStemAudio'
|
||||
import { getAppSettings, type SekDynamicPalette } from '../api/client'
|
||||
import type { VisualizerMode, VisualizerSettings } from './visualizers/visualizerTypes'
|
||||
import { deepMergeVizSettings, normalizeVizSettings, sekSettingsToVizPatch, VIZ_DEFAULTS, type DeepPartial, type VizSettings } from './visualizers/visualizerSettings'
|
||||
import RendererControlsPanel from './visualizers/RendererControlsPanel'
|
||||
import PerformanceSidebar from './PerformanceSidebar.tsx'
|
||||
import { waitForMediaReady } from '../utils/waitForMediaReady'
|
||||
import LyricsStylePicker from './LyricsStylePicker'
|
||||
@@ -24,6 +25,7 @@ import StemMixerPanel from './StemMixerPanel'
|
||||
import LyricsViewport from './LyricsViewport'
|
||||
import { buildDynamicBackgroundGradient, buildHighContrastSpeakerPalette, buildSpeakerVisualizerPalette, ensureReadableColor, hexToRgba, hslToHex, isWhiteLikeColor, normalizeDynamicBackground, pickFirstNonWhitePaletteColor, rgbToHsl } from '../utils/colorUtils'
|
||||
import { useGloss } from '../hooks/useGloss'
|
||||
import { resolveVisualizerProviders, withDefaultVisualizerPreset, type VisualizerProvider } from '../visualizers/providers'
|
||||
|
||||
// ─── Types ───────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -63,62 +65,6 @@ type VideoStartPhase = 'idle' | 'priming' | 'playing'
|
||||
const VISUALIZER_STYLE_MODES = new Set<Exclude<VisualizerMode, null>>(['muon', 'fluid', 'tendrils', 'spirit', 'moire', 'dissolve', 'glitch', 'spiral'])
|
||||
const LYRICS_VISUAL_STYLES = new Set<LyricsVisualStyle>(['scroll', 'fade', 'karaoke', 'glow', 'blur', 'video', 'muon_iframe'])
|
||||
|
||||
const DEFAULT_MUON_IFRAME_URL = 'http://localhost:3011'
|
||||
const MUON_IFRAME_REV = '20260427c'
|
||||
|
||||
function isLoopbackHost(hostname: string): boolean {
|
||||
const value = hostname.trim().toLowerCase()
|
||||
return value === 'localhost' || value === '127.0.0.1' || value === '::1'
|
||||
}
|
||||
|
||||
function defaultMuonIframeUrl(): string {
|
||||
if (typeof window === 'undefined') return DEFAULT_MUON_IFRAME_URL
|
||||
const protocol = window.location.protocol === 'https:' ? 'https:' : 'http:'
|
||||
const hostname = window.location.hostname || 'localhost'
|
||||
return `${protocol}//${hostname}:3011`
|
||||
}
|
||||
|
||||
function resolveMuonIframeUrl(raw: string | undefined): string {
|
||||
const cleaned = String(raw || '').trim()
|
||||
if (!cleaned) return defaultMuonIframeUrl()
|
||||
|
||||
if (/^https?:\/\//i.test(cleaned)) {
|
||||
try {
|
||||
const parsed = new URL(cleaned)
|
||||
if (typeof window !== 'undefined' && isLoopbackHost(parsed.hostname) && !isLoopbackHost(window.location.hostname)) {
|
||||
parsed.hostname = window.location.hostname
|
||||
}
|
||||
return parsed.toString()
|
||||
} catch {
|
||||
return defaultMuonIframeUrl()
|
||||
}
|
||||
}
|
||||
|
||||
const protocol = typeof window !== 'undefined' && window.location.protocol === 'https:' ? 'https://' : 'http://'
|
||||
const candidate = cleaned.startsWith('//') ? `${protocol}${cleaned.slice(2)}` : `${protocol}${cleaned}`
|
||||
try {
|
||||
const parsed = new URL(candidate)
|
||||
if (!/^https?:$/i.test(parsed.protocol)) return defaultMuonIframeUrl()
|
||||
if (typeof window !== 'undefined' && isLoopbackHost(parsed.hostname) && !isLoopbackHost(window.location.hostname)) {
|
||||
parsed.hostname = window.location.hostname
|
||||
}
|
||||
return parsed.toString()
|
||||
} catch {
|
||||
return defaultMuonIframeUrl()
|
||||
}
|
||||
}
|
||||
|
||||
function withDefaultMuonPreset(rawUrl: string): string {
|
||||
try {
|
||||
const url = new URL(rawUrl)
|
||||
if (!url.searchParams.has('preset')) url.searchParams.set('preset', 'default')
|
||||
if (!url.searchParams.has('se_rev')) url.searchParams.set('se_rev', MUON_IFRAME_REV)
|
||||
return url.toString()
|
||||
} catch {
|
||||
return rawUrl
|
||||
}
|
||||
}
|
||||
|
||||
function isVisualizerModeStyle(style: unknown): style is Exclude<VisualizerMode, null> {
|
||||
return typeof style === 'string' && VISUALIZER_STYLE_MODES.has(style as Exclude<VisualizerMode, null>)
|
||||
}
|
||||
@@ -939,9 +885,21 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, projectId, so
|
||||
const desiredVideoSegmentRef = useRef<{ index: number; localTime: number } | null>(null)
|
||||
const videoPlaylistRef = useRef<Array<{ url: string; duration?: number }>>(resolvedVideoPlaylist)
|
||||
const iframeRef = useRef<HTMLIFrameElement>(null)
|
||||
const [muonIframeUrl, setMuonIframeUrl] = useState(() => defaultMuonIframeUrl())
|
||||
const [providers, setProviders] = useState(() => resolveVisualizerProviders())
|
||||
const [muonIframeError, setMuonIframeError] = useState(false)
|
||||
const muonEmbedUrl = useMemo(() => withDefaultMuonPreset(muonIframeUrl), [muonIframeUrl])
|
||||
const activeIframeProvider = useMemo<VisualizerProvider>(() => {
|
||||
const style = (liveVizSettings.muon_iframe.style || 'default') as 'default' | 'fluid' | 'tendrils' | 'spirit'
|
||||
const providerId = style === 'default' ? 'muon' : style
|
||||
return providers[providerId]
|
||||
}, [liveVizSettings.muon_iframe.style, providers])
|
||||
const iframeEmbedUrl = useMemo(() => {
|
||||
if (activeIframeProvider.id === 'muon') return withDefaultVisualizerPreset(activeIframeProvider.iframeUrl)
|
||||
return activeIframeProvider.iframeUrl
|
||||
}, [activeIframeProvider.id, activeIframeProvider.iframeUrl])
|
||||
const iframeTitle = useMemo(
|
||||
() => `${activeIframeProvider.id.charAt(0).toUpperCase() + activeIframeProvider.id.slice(1)} music visualizer`,
|
||||
[activeIframeProvider.id],
|
||||
)
|
||||
|
||||
useEffect(() => {
|
||||
videoPlaylistRef.current = resolvedVideoPlaylist
|
||||
@@ -1643,7 +1601,21 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, projectId, so
|
||||
const isPerformanceView = performanceMode || isFullscreen
|
||||
const [lyricsViewportScale, setLyricsViewportScale] = useState(1)
|
||||
const [visualizerSize, setVisualizerSize] = useState({ width: 960, height: 540 })
|
||||
const { onIframeLoad: onMuonIframeLoad } = useMuonIframeTransport({
|
||||
const activeIframeStyle = liveVizSettings.muon_iframe.style
|
||||
const muonParamsByStyle = liveVizSettings.muon_iframe.params ?? {}
|
||||
const activeMuonParams = muonParamsByStyle[activeIframeStyle] ?? {}
|
||||
const iframePalette = useMemo(() => {
|
||||
if (Array.isArray(liveVizSettings.muon_iframe.paletteOverride) && liveVizSettings.muon_iframe.paletteOverride.length > 0) {
|
||||
return liveVizSettings.muon_iframe.paletteOverride
|
||||
}
|
||||
if (Array.isArray(activeSpeakerVisualizerPalette) && activeSpeakerVisualizerPalette.length > 0) {
|
||||
return activeSpeakerVisualizerPalette
|
||||
}
|
||||
return paletteRef.current ?? []
|
||||
}, [activeSpeakerVisualizerPalette, liveVizSettings.muon_iframe.paletteOverride, paletteRef])
|
||||
|
||||
const { onIframeLoad: onMuonIframeLoad, capabilities: muonCapabilities } = useVisualizerIframeTransport({
|
||||
provider: activeIframeProvider,
|
||||
iframeRef,
|
||||
isActive: isIframeVisualizerMode,
|
||||
audioUrl: audioUrl ?? null,
|
||||
@@ -1655,7 +1627,7 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, projectId, so
|
||||
}
|
||||
return Number(audioRef.current?.playbackRate ?? videoRef.current?.playbackRate ?? 1)
|
||||
},
|
||||
style: liveVizSettings.muon_iframe.style,
|
||||
style: activeIframeProvider.id === 'muon' ? liveVizSettings.muon_iframe.style : 'default',
|
||||
preset: liveVizSettings.muon_iframe.preset,
|
||||
quality: liveVizSettings.muon_iframe.quality,
|
||||
viewport: {
|
||||
@@ -1663,6 +1635,8 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, projectId, so
|
||||
cssHeight: visualizerSize.height,
|
||||
dpr: typeof window === 'undefined' ? 1 : window.devicePixelRatio || 1,
|
||||
},
|
||||
palette: iframePalette,
|
||||
params: activeMuonParams,
|
||||
commandIdPrefix: 'lyrics',
|
||||
})
|
||||
|
||||
@@ -1671,11 +1645,11 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, projectId, so
|
||||
void getAppSettings()
|
||||
.then((settings) => {
|
||||
if (cancelled) return
|
||||
setMuonIframeUrl(resolveMuonIframeUrl(settings.muon_iframe_url))
|
||||
setProviders(resolveVisualizerProviders(settings))
|
||||
})
|
||||
.catch(() => {
|
||||
if (cancelled) return
|
||||
setMuonIframeUrl(defaultMuonIframeUrl())
|
||||
setProviders(resolveVisualizerProviders())
|
||||
})
|
||||
return () => {
|
||||
cancelled = true
|
||||
@@ -2481,6 +2455,44 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, projectId, so
|
||||
onGlossEnabledChange={setGlossEnabled}
|
||||
onGlossTargetLangChange={setGlossTargetLang}
|
||||
/>
|
||||
{isIframeVisualizerMode ? (
|
||||
<div className="px-3 py-2 border-b border-white/10 space-y-2">
|
||||
<p className="text-[10px] text-slate-400">Renderer controls</p>
|
||||
<RendererControlsPanel
|
||||
capabilities={muonCapabilities}
|
||||
values={activeMuonParams}
|
||||
onValueChange={(name, value) => {
|
||||
const styleKey = liveVizSettings.muon_iframe.style
|
||||
const currentByStyle = liveVizSettings.muon_iframe.params ?? {}
|
||||
const currentForStyle = currentByStyle[styleKey] ?? {}
|
||||
patchVizSettings({
|
||||
muon_iframe: {
|
||||
params: {
|
||||
...currentByStyle,
|
||||
[styleKey]: {
|
||||
...currentForStyle,
|
||||
[name]: value as VisualizerParamValue,
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
}}
|
||||
onReset={() => {
|
||||
const styleKey = liveVizSettings.muon_iframe.style
|
||||
const currentByStyle = liveVizSettings.muon_iframe.params ?? {}
|
||||
patchVizSettings({
|
||||
muon_iframe: {
|
||||
params: {
|
||||
...currentByStyle,
|
||||
[styleKey]: {},
|
||||
},
|
||||
},
|
||||
})
|
||||
}}
|
||||
className="rounded-md border border-white/10 bg-black/20 px-2 py-2"
|
||||
/>
|
||||
</div>
|
||||
) : null}
|
||||
{/* Speaker colour swatches — always shown so user knows if diarization is missing */}
|
||||
<div className="px-3 py-2 border-b border-white/10 flex items-center gap-3 flex-wrap">
|
||||
<span className="text-[10px] text-slate-500 shrink-0">Speakers</span>
|
||||
@@ -2595,7 +2607,8 @@ export default function LyricsLivePreview({ lrcText, lyricsTracks, projectId, so
|
||||
visualizerSize={visualizerSize}
|
||||
liveVizSettings={liveVizSettings}
|
||||
activeSpeakerVisualizerPalette={activeSpeakerVisualizerPalette}
|
||||
muonEmbedUrl={muonEmbedUrl}
|
||||
iframeEmbedUrl={iframeEmbedUrl}
|
||||
iframeTitle={iframeTitle}
|
||||
currentVideoUrl={currentVideoUrl}
|
||||
videoStartPhase={videoStartPhase}
|
||||
resolvedVideoPlaylist={resolvedVideoPlaylist}
|
||||
|
||||
@@ -36,7 +36,8 @@ export interface LyricsViewportProps {
|
||||
liveVizSettings: VizSettings
|
||||
stemFeaturesByChannel?: Partial<Record<StemFeatureChannel, MutableRefObject<AudioFeatures>>> | undefined
|
||||
activeSpeakerVisualizerPalette: string[] | undefined
|
||||
muonEmbedUrl: string
|
||||
iframeEmbedUrl: string
|
||||
iframeTitle: string
|
||||
currentVideoUrl: string | undefined
|
||||
videoStartPhase: 'idle' | 'priming' | 'playing'
|
||||
resolvedVideoPlaylist: Array<{ url: string; duration?: number }>
|
||||
@@ -90,17 +91,17 @@ function LyricsViewport(props: LyricsViewportProps) {
|
||||
{props.isIframeVisualizerMode && (
|
||||
<div style={{ position: 'absolute', inset: 0, zIndex: 0, pointerEvents: 'none' }}>
|
||||
<iframe
|
||||
key={props.muonEmbedUrl}
|
||||
key={props.iframeEmbedUrl}
|
||||
ref={props.iframeRef}
|
||||
src={props.muonEmbedUrl}
|
||||
title="Muon music visualizer"
|
||||
src={props.iframeEmbedUrl}
|
||||
title={props.iframeTitle}
|
||||
className="absolute inset-0 h-full w-full border-0"
|
||||
loading="eager"
|
||||
referrerPolicy="no-referrer"
|
||||
allow="autoplay"
|
||||
onLoad={() => {
|
||||
// setMuonIframeReady(true) is routed by LyricsLivePreview to
|
||||
// useMuonIframeTransport's onIframeLoad — no separate prop needed.
|
||||
// useVisualizerIframeTransport's onIframeLoad — no separate prop needed.
|
||||
props.setMuonIframeReady(true)
|
||||
props.setMuonIframeError(false)
|
||||
}}
|
||||
@@ -112,7 +113,7 @@ function LyricsViewport(props: LyricsViewportProps) {
|
||||
{props.muonIframeError ? (
|
||||
<div className="absolute inset-0 flex items-center justify-center p-4">
|
||||
<div className="rounded bg-black/70 px-3 py-2 text-xs text-red-200">
|
||||
Could not load Muon iframe.
|
||||
Could not load visualizer iframe.
|
||||
</div>
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
@@ -2,14 +2,16 @@ import { forwardRef, useCallback, useEffect, useImperativeHandle, useLayoutEffec
|
||||
import { Disc3 } from 'lucide-react'
|
||||
import { MuonFeatureExtractor } from '../audio/MuonFeatureExtractor'
|
||||
import { useAudioFeatures, type AudioFeatures } from '../hooks/useAudioFeatures'
|
||||
import { useMuonIframeTransport } from '../hooks/useMuonIframeTransport'
|
||||
import { useVisualizerIframeTransport, type VisualizerParamValue } from '../hooks/useVisualizerIframeTransport'
|
||||
import { useVisualizerPalette } from '../hooks/useVisualizerPalette'
|
||||
import { normalizeStemFeatureChannel, type StemFeatureChannel } from '../utils/visualizerStemChannels'
|
||||
import VisualizerBackdrop, { type VisualizerMode } from './visualizers/VisualizerBackdrop'
|
||||
import RendererControlsPanel from './visualizers/RendererControlsPanel'
|
||||
import type { VisualizerSettings } from './visualizers/visualizerTypes'
|
||||
import { registerPlayback, unregisterPlayback } from '../utils/audioRegistry'
|
||||
import { getAppSettings } from '../api/client'
|
||||
import { buildStemFeatures } from '../hooks/useStemAudio'
|
||||
import { resolveVisualizerProviders, withDefaultVisualizerPreset, type VisualizerProvider } from '../visualizers/providers'
|
||||
|
||||
export interface VisualizerPreviewStem {
|
||||
name: string
|
||||
@@ -31,62 +33,6 @@ interface VisualizerPreviewWidgetProps {
|
||||
type NativePreviewMode = Extract<VisualizerMode, 'moire' | 'dissolve' | 'glitch' | 'spiral'>
|
||||
type PreviewMode = 'muon_iframe' | 'video' | NativePreviewMode
|
||||
|
||||
const DEFAULT_MUON_IFRAME_URL = 'http://localhost:3011'
|
||||
const MUON_IFRAME_REV = '20260427c'
|
||||
|
||||
function isLoopbackHost(hostname: string): boolean {
|
||||
const value = hostname.trim().toLowerCase()
|
||||
return value === 'localhost' || value === '127.0.0.1' || value === '::1'
|
||||
}
|
||||
|
||||
function defaultMuonIframeUrl(): string {
|
||||
if (typeof window === 'undefined') return DEFAULT_MUON_IFRAME_URL
|
||||
const protocol = window.location.protocol === 'https:' ? 'https:' : 'http:'
|
||||
const hostname = window.location.hostname || 'localhost'
|
||||
return `${protocol}//${hostname}:3011`
|
||||
}
|
||||
|
||||
function resolveMuonIframeUrl(raw: string | undefined): string {
|
||||
const cleaned = String(raw || '').trim()
|
||||
if (!cleaned) return defaultMuonIframeUrl()
|
||||
|
||||
if (/^https?:\/\//i.test(cleaned)) {
|
||||
try {
|
||||
const parsed = new URL(cleaned)
|
||||
if (typeof window !== 'undefined' && isLoopbackHost(parsed.hostname) && !isLoopbackHost(window.location.hostname)) {
|
||||
parsed.hostname = window.location.hostname
|
||||
}
|
||||
return parsed.toString()
|
||||
} catch {
|
||||
return defaultMuonIframeUrl()
|
||||
}
|
||||
}
|
||||
|
||||
const protocol = typeof window !== 'undefined' && window.location.protocol === 'https:' ? 'https://' : 'http://'
|
||||
const candidate = cleaned.startsWith('//') ? `${protocol}${cleaned.slice(2)}` : `${protocol}${cleaned}`
|
||||
try {
|
||||
const parsed = new URL(candidate)
|
||||
if (!/^https?:$/i.test(parsed.protocol)) return defaultMuonIframeUrl()
|
||||
if (typeof window !== 'undefined' && isLoopbackHost(parsed.hostname) && !isLoopbackHost(window.location.hostname)) {
|
||||
parsed.hostname = window.location.hostname
|
||||
}
|
||||
return parsed.toString()
|
||||
} catch {
|
||||
return defaultMuonIframeUrl()
|
||||
}
|
||||
}
|
||||
|
||||
function withDefaultPreset(rawUrl: string): string {
|
||||
try {
|
||||
const url = new URL(rawUrl)
|
||||
if (!url.searchParams.has('preset')) url.searchParams.set('preset', 'default')
|
||||
if (!url.searchParams.has('se_rev')) url.searchParams.set('se_rev', MUON_IFRAME_REV)
|
||||
return url.toString()
|
||||
} catch {
|
||||
return rawUrl
|
||||
}
|
||||
}
|
||||
|
||||
type MuonStyle = 'fluid' | 'tendrils' | 'spirit' | 'default'
|
||||
|
||||
const MUON_STYLES: MuonStyle[] = ['default', 'fluid', 'tendrils', 'spirit']
|
||||
@@ -167,11 +113,24 @@ const VisualizerPreviewWidget = forwardRef<HTMLAudioElement, VisualizerPreviewWi
|
||||
const [ctxVersion, setCtxVersion] = useState(0)
|
||||
const [stemAnalyserVersion, setStemAnalyserVersion] = useState(0)
|
||||
const [muonIframeError, setMuonIframeError] = useState(false)
|
||||
const [muonIframeUrl, setMuonIframeUrl] = useState(() => defaultMuonIframeUrl())
|
||||
const [providers, setProviders] = useState(() => resolveVisualizerProviders())
|
||||
const [muonQuality, setMuonQuality] = useState<'low' | 'balanced' | 'high'>(visualizerSettings?.muon_iframe?.quality ?? 'balanced')
|
||||
const [muonStyle, setMuonStyle] = useState<MuonStyle>((visualizerSettings?.muon_iframe?.style as MuonStyle | undefined) ?? muonStyleFromPreferredMode(preferredMode))
|
||||
const [muonPreset, setMuonPreset] = useState<string>(visualizerSettings?.muon_iframe?.preset ?? 'default')
|
||||
const muonEmbedUrl = useMemo(() => withDefaultPreset(muonIframeUrl), [muonIframeUrl])
|
||||
const [muonParamsByStyle, setMuonParamsByStyle] = useState<Partial<Record<MuonStyle, Record<string, VisualizerParamValue>>>>({})
|
||||
const activeIframeProvider = useMemo<VisualizerProvider>(() => {
|
||||
const providerId = muonStyle === 'default' ? 'muon' : muonStyle
|
||||
return providers[providerId]
|
||||
}, [muonStyle, providers])
|
||||
const activeIframeLabel = useMemo(
|
||||
() => activeIframeProvider.id.charAt(0).toUpperCase() + activeIframeProvider.id.slice(1),
|
||||
[activeIframeProvider.id],
|
||||
)
|
||||
const activeIframeUrl = useMemo(() => {
|
||||
if (activeIframeProvider.id === 'muon') return withDefaultVisualizerPreset(activeIframeProvider.iframeUrl)
|
||||
return activeIframeProvider.iframeUrl
|
||||
}, [activeIframeProvider.id, activeIframeProvider.iframeUrl])
|
||||
const activeMuonParams = useMemo(() => muonParamsByStyle[muonStyle] ?? {}, [muonParamsByStyle, muonStyle])
|
||||
|
||||
const normalizedVideoUrl = useMemo(() => String(videoUrl || '').trim(), [videoUrl])
|
||||
const canPreviewVideo = useMemo(() => isDirectVideoUrl(normalizedVideoUrl), [normalizedVideoUrl])
|
||||
@@ -208,14 +167,15 @@ const VisualizerPreviewWidget = forwardRef<HTMLAudioElement, VisualizerPreviewWi
|
||||
[stemFeaturesByChannel],
|
||||
)
|
||||
|
||||
const { onIframeLoad: onMuonIframeLoad } = useMuonIframeTransport({
|
||||
const { onIframeLoad: onMuonIframeLoad, capabilities: muonCapabilities } = useVisualizerIframeTransport({
|
||||
provider: activeIframeProvider,
|
||||
iframeRef,
|
||||
isActive: mode === 'muon_iframe',
|
||||
audioUrl: audioUrl || null,
|
||||
isPlayingRef,
|
||||
getCurrentTime: () => Number(audioRef.current?.currentTime || 0),
|
||||
getRate: () => Number(audioRef.current?.playbackRate || 1),
|
||||
style: muonStyle,
|
||||
style: activeIframeProvider.id === 'muon' ? muonStyle : 'default',
|
||||
preset: muonPreset,
|
||||
quality: muonQuality,
|
||||
viewport: dims ? {
|
||||
@@ -223,6 +183,8 @@ const VisualizerPreviewWidget = forwardRef<HTMLAudioElement, VisualizerPreviewWi
|
||||
cssHeight: dims.h,
|
||||
dpr: typeof window === 'undefined' ? 1 : window.devicePixelRatio || 1,
|
||||
} : undefined,
|
||||
palette: paletteRef.current,
|
||||
params: activeMuonParams,
|
||||
commandIdPrefix: 'vis',
|
||||
})
|
||||
|
||||
@@ -386,12 +348,12 @@ const VisualizerPreviewWidget = forwardRef<HTMLAudioElement, VisualizerPreviewWi
|
||||
void getAppSettings()
|
||||
.then((settings) => {
|
||||
if (cancelled) return
|
||||
setMuonIframeUrl(resolveMuonIframeUrl(settings.muon_iframe_url))
|
||||
setProviders(resolveVisualizerProviders(settings))
|
||||
if (settings.muon_quality && !visualizerSettings?.muon_iframe?.quality) setMuonQuality(settings.muon_quality)
|
||||
})
|
||||
.catch(() => {
|
||||
if (cancelled) return
|
||||
setMuonIframeUrl(defaultMuonIframeUrl())
|
||||
setProviders(resolveVisualizerProviders())
|
||||
})
|
||||
return () => {
|
||||
cancelled = true
|
||||
@@ -606,10 +568,10 @@ const VisualizerPreviewWidget = forwardRef<HTMLAudioElement, VisualizerPreviewWi
|
||||
/>
|
||||
) : mode === 'muon_iframe' ? (
|
||||
<iframe
|
||||
key={muonEmbedUrl}
|
||||
key={activeIframeUrl}
|
||||
ref={iframeRef}
|
||||
src={muonEmbedUrl}
|
||||
title="Muon music visualizer"
|
||||
src={activeIframeUrl}
|
||||
title={`${activeIframeLabel} music visualizer`}
|
||||
className="absolute inset-0 h-full w-full border-0"
|
||||
loading="eager"
|
||||
referrerPolicy="no-referrer"
|
||||
@@ -648,7 +610,7 @@ const VisualizerPreviewWidget = forwardRef<HTMLAudioElement, VisualizerPreviewWi
|
||||
{mode === 'muon_iframe' && muonIframeError ? (
|
||||
<div className="absolute inset-0 flex items-center justify-center p-4">
|
||||
<div className="rounded bg-black/70 px-3 py-2 text-xs text-red-200">
|
||||
Could not load Muon iframe.
|
||||
Could not load {activeIframeLabel} iframe.
|
||||
</div>
|
||||
</div>
|
||||
) : null}
|
||||
@@ -721,6 +683,26 @@ const VisualizerPreviewWidget = forwardRef<HTMLAudioElement, VisualizerPreviewWi
|
||||
<option value="high">high</option>
|
||||
</select>
|
||||
</div>
|
||||
<RendererControlsPanel
|
||||
capabilities={muonCapabilities}
|
||||
values={activeMuonParams}
|
||||
onValueChange={(name, value) => {
|
||||
setMuonParamsByStyle((prev) => ({
|
||||
...prev,
|
||||
[muonStyle]: {
|
||||
...(prev[muonStyle] ?? {}),
|
||||
[name]: value,
|
||||
},
|
||||
}))
|
||||
}}
|
||||
onReset={() => {
|
||||
setMuonParamsByStyle((prev) => ({
|
||||
...prev,
|
||||
[muonStyle]: {},
|
||||
}))
|
||||
}}
|
||||
className="rounded border border-white/10 bg-black/20 px-2 py-2"
|
||||
/>
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
import type { VisualizerIframeCapabilities, VisualizerParamValue } from '../../hooks/useVisualizerIframeTransport'
|
||||
|
||||
interface RendererControlsPanelProps {
|
||||
capabilities: VisualizerIframeCapabilities | null
|
||||
values: Record<string, VisualizerParamValue>
|
||||
onValueChange: (name: string, value: VisualizerParamValue) => void
|
||||
onReset?: () => void
|
||||
className?: string
|
||||
}
|
||||
|
||||
function readValue(name: string, values: Record<string, VisualizerParamValue>, fallback: VisualizerParamValue): VisualizerParamValue {
|
||||
if (Object.prototype.hasOwnProperty.call(values, name)) return values[name]
|
||||
return fallback
|
||||
}
|
||||
|
||||
export default function RendererControlsPanel({
|
||||
capabilities,
|
||||
values,
|
||||
onValueChange,
|
||||
onReset,
|
||||
className,
|
||||
}: RendererControlsPanelProps) {
|
||||
if (!capabilities) {
|
||||
return (
|
||||
<div className={className}>
|
||||
<p className="text-[11px] text-gray-400">Loading renderer controls...</p>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
const params = capabilities.params || []
|
||||
|
||||
if (!params.length) {
|
||||
return (
|
||||
<div className={className}>
|
||||
<p className="text-[11px] text-gray-400">No renderer-specific controls reported.</p>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
return (
|
||||
<div className={className}>
|
||||
<div className="space-y-2">
|
||||
{params.map((param) => {
|
||||
const value = readValue(param.name, values, param.default)
|
||||
const key = `renderer-param-${param.name}`
|
||||
|
||||
if (param.type === 'boolean') {
|
||||
return (
|
||||
<label key={param.name} className="flex items-center justify-between gap-2 text-[11px] text-gray-200" htmlFor={key}>
|
||||
<span>{param.label}</span>
|
||||
<input
|
||||
id={key}
|
||||
type="checkbox"
|
||||
checked={Boolean(value)}
|
||||
onChange={(event) => onValueChange(param.name, event.target.checked)}
|
||||
className="h-3.5 w-3.5"
|
||||
/>
|
||||
</label>
|
||||
)
|
||||
}
|
||||
|
||||
if (param.type === 'enum') {
|
||||
const options = param.options && param.options.length > 0 ? param.options : [String(param.default)]
|
||||
return (
|
||||
<label key={param.name} className="flex flex-col gap-1 text-[11px] text-gray-300" htmlFor={key}>
|
||||
<span>{param.label}</span>
|
||||
<select
|
||||
id={key}
|
||||
value={String(value)}
|
||||
onChange={(event) => onValueChange(param.name, event.target.value)}
|
||||
className="rounded border border-white/10 bg-black/30 px-2 py-1 text-[11px] text-gray-100"
|
||||
>
|
||||
{options.map((option) => (
|
||||
<option key={`${param.name}-${option}`} value={option}>{option}</option>
|
||||
))}
|
||||
</select>
|
||||
</label>
|
||||
)
|
||||
}
|
||||
|
||||
const min = Number.isFinite(param.min) ? Number(param.min) : 0
|
||||
const max = Number.isFinite(param.max) ? Number(param.max) : 1
|
||||
const step = Number.isFinite(param.step) ? Number(param.step) : 0.01
|
||||
const numericValue = Number.isFinite(Number(value)) ? Number(value) : Number(param.default)
|
||||
|
||||
return (
|
||||
<label key={param.name} className="flex flex-col gap-1 text-[11px] text-gray-300" htmlFor={key}>
|
||||
<span className="flex items-center justify-between gap-2">
|
||||
<span>{param.label}</span>
|
||||
<span className="font-mono text-[10px] text-gray-500">{numericValue.toFixed(2)}</span>
|
||||
</span>
|
||||
<input
|
||||
id={key}
|
||||
type="range"
|
||||
min={min}
|
||||
max={max}
|
||||
step={step}
|
||||
value={numericValue}
|
||||
onChange={(event) => onValueChange(param.name, Number(event.target.value))}
|
||||
className="w-full"
|
||||
/>
|
||||
</label>
|
||||
)
|
||||
})}
|
||||
</div>
|
||||
{onReset ? (
|
||||
<div className="mt-2">
|
||||
<button
|
||||
type="button"
|
||||
onClick={onReset}
|
||||
className="rounded border border-white/10 bg-white/5 px-2 py-1 text-[10px] text-gray-200 hover:bg-white/10"
|
||||
>
|
||||
Reset renderer controls
|
||||
</button>
|
||||
</div>
|
||||
) : null}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -30,6 +30,8 @@ export interface VizSettings {
|
||||
| 'spirit_aurora'
|
||||
| 'spirit_lumen'
|
||||
quality: 'low' | 'balanced' | 'high'
|
||||
params?: Partial<Record<'default' | 'fluid' | 'tendrils' | 'spirit', Record<string, number | string | boolean>>>
|
||||
paletteOverride?: string[]
|
||||
}
|
||||
muon: {
|
||||
radius_multiplier: number
|
||||
@@ -129,6 +131,8 @@ export const VIZ_DEFAULTS: VizSettings = {
|
||||
style: 'default',
|
||||
preset: 'default',
|
||||
quality: 'balanced',
|
||||
params: {},
|
||||
paletteOverride: undefined,
|
||||
},
|
||||
muon: {
|
||||
radius_multiplier: 1.0,
|
||||
@@ -205,6 +209,36 @@ function clampVizMultiplier(value: number, min = 0.0, max = 3.0): number {
|
||||
return Math.max(min, Math.min(max, value))
|
||||
}
|
||||
|
||||
function normalizeIframeParams(input: unknown): Partial<Record<'default' | 'fluid' | 'tendrils' | 'spirit', Record<string, number | string | boolean>>> {
|
||||
if (!input || typeof input !== 'object') return {}
|
||||
const byStyle = input as Record<string, unknown>
|
||||
const styles: Array<'default' | 'fluid' | 'tendrils' | 'spirit'> = ['default', 'fluid', 'tendrils', 'spirit']
|
||||
const normalized: Partial<Record<'default' | 'fluid' | 'tendrils' | 'spirit', Record<string, number | string | boolean>>> = {}
|
||||
|
||||
styles.forEach((style) => {
|
||||
const rawStyleValues = byStyle[style]
|
||||
if (!rawStyleValues || typeof rawStyleValues !== 'object') return
|
||||
const valueMap: Record<string, number | string | boolean> = {}
|
||||
Object.entries(rawStyleValues as Record<string, unknown>).forEach(([name, value]) => {
|
||||
if (!String(name).trim()) return
|
||||
if (typeof value === 'number' || typeof value === 'string' || typeof value === 'boolean') {
|
||||
valueMap[name] = value
|
||||
}
|
||||
})
|
||||
normalized[style] = valueMap
|
||||
})
|
||||
|
||||
return normalized
|
||||
}
|
||||
|
||||
function normalizePaletteOverride(input: unknown): string[] | undefined {
|
||||
if (!Array.isArray(input)) return undefined
|
||||
const colors = input
|
||||
.map((value) => String(value).trim())
|
||||
.filter((value) => value.length > 0)
|
||||
return colors.length > 0 ? colors : undefined
|
||||
}
|
||||
|
||||
export function sekSettingsToVizPatch(settings?: VisualizerSettings | null): DeepPartial<VizSettings> {
|
||||
if (!settings) return {}
|
||||
const quality = settings.quality
|
||||
@@ -228,6 +262,8 @@ export function sekSettingsToVizPatch(settings?: VisualizerSettings | null): Dee
|
||||
style: muonIframe?.style,
|
||||
preset: muonIframe?.preset,
|
||||
quality: muonIframe?.quality,
|
||||
params: muonIframe?.params,
|
||||
paletteOverride: muonIframe?.paletteOverride,
|
||||
},
|
||||
muon: {
|
||||
radius_multiplier: sm?.radius_multiplier,
|
||||
@@ -360,6 +396,8 @@ export function normalizeVizSettings(input: VisualizerSettings | VizSettings): V
|
||||
style: merged.muon_iframe.style === 'fluid' || merged.muon_iframe.style === 'tendrils' || merged.muon_iframe.style === 'spirit' ? merged.muon_iframe.style : 'default',
|
||||
preset: merged.muon_iframe.preset,
|
||||
quality: merged.muon_iframe.quality === 'low' || merged.muon_iframe.quality === 'high' ? merged.muon_iframe.quality : 'balanced',
|
||||
params: normalizeIframeParams(merged.muon_iframe.params),
|
||||
paletteOverride: normalizePaletteOverride(merged.muon_iframe.paletteOverride),
|
||||
},
|
||||
muon: {
|
||||
radius_multiplier: clampVizMultiplier(merged.muon.radius_multiplier, 0.25, 2.0),
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
export type VizShape = 'dynamic' | 'circle' | 'square' | 'star' | 'hexagon' | 'flower'
|
||||
export type MuonIframeStyle = 'default' | 'fluid' | 'tendrils' | 'spirit'
|
||||
export type VisualizerParamValue = number | string | boolean
|
||||
export type MuonIframePreset =
|
||||
| 'default'
|
||||
| 'nebula'
|
||||
@@ -66,6 +67,8 @@ export interface VisualizerSettings {
|
||||
style?: MuonIframeStyle
|
||||
preset?: MuonIframePreset
|
||||
quality?: 'low' | 'balanced' | 'high'
|
||||
params?: Partial<Record<MuonIframeStyle, Record<string, VisualizerParamValue>>>
|
||||
paletteOverride?: string[]
|
||||
}
|
||||
/** @deprecated */
|
||||
muon?: {
|
||||
|
||||
@@ -1,317 +0,0 @@
|
||||
import { useCallback, useEffect, useRef, useState, type RefObject } from 'react'
|
||||
|
||||
export interface MuonViewport {
|
||||
cssWidth: number
|
||||
cssHeight: number
|
||||
dpr: number
|
||||
}
|
||||
|
||||
export interface UseMuonIframeTransportArgs {
|
||||
iframeRef: RefObject<HTMLIFrameElement | null>
|
||||
isActive: boolean
|
||||
audioUrl: string | null
|
||||
isPlayingRef: RefObject<boolean>
|
||||
getCurrentTime: () => number
|
||||
getRate?: () => number
|
||||
style: 'default' | 'fluid' | 'tendrils' | 'spirit'
|
||||
preset?: string
|
||||
quality?: 'low' | 'balanced' | 'high'
|
||||
viewport?: MuonViewport
|
||||
commandIdPrefix?: string
|
||||
}
|
||||
|
||||
export interface UseMuonIframeTransportResult {
|
||||
onIframeLoad: () => void
|
||||
sendCommand: (type: string, payload: Record<string, unknown>) => void
|
||||
loadedDuration: number | null
|
||||
isLoaded: boolean
|
||||
}
|
||||
|
||||
function readRate(getRate?: () => number): number {
|
||||
const value = Number(getRate ? getRate() : 1)
|
||||
return Number.isFinite(value) && value > 0 ? value : 1
|
||||
}
|
||||
|
||||
function readTime(getCurrentTime: () => number): number {
|
||||
const value = Number(getCurrentTime())
|
||||
return Number.isFinite(value) && value >= 0 ? value : 0
|
||||
}
|
||||
|
||||
function resolveAudioUrl(raw: string): string {
|
||||
const value = String(raw || '').trim()
|
||||
if (!value) return ''
|
||||
|
||||
if (/^https?:\/\//i.test(value)) return value
|
||||
|
||||
if (value.startsWith('//')) {
|
||||
const protocol = typeof window !== 'undefined' && window.location.protocol === 'https:' ? 'https:' : 'http:'
|
||||
return `${protocol}${value}`
|
||||
}
|
||||
|
||||
if (typeof window === 'undefined') return value
|
||||
try {
|
||||
return new URL(value, window.location.origin).toString()
|
||||
} catch {
|
||||
return value
|
||||
}
|
||||
}
|
||||
|
||||
export function useMuonIframeTransport(args: UseMuonIframeTransportArgs): UseMuonIframeTransportResult {
|
||||
const {
|
||||
iframeRef,
|
||||
isActive,
|
||||
audioUrl,
|
||||
isPlayingRef,
|
||||
getCurrentTime,
|
||||
getRate,
|
||||
style,
|
||||
preset,
|
||||
quality,
|
||||
viewport,
|
||||
commandIdPrefix = 'muon',
|
||||
} = args
|
||||
|
||||
const readyRef = useRef(false)
|
||||
const frameRef = useRef<number | null>(null)
|
||||
const lastSyncTimeRef = useRef(-1)
|
||||
const lastRateRef = useRef(1)
|
||||
const lastPlayingRef = useRef(false)
|
||||
const commandSeqRef = useRef(0)
|
||||
const lastRequestedAudioUrlRef = useRef<string | null>(null)
|
||||
const lastIframeErrorKeyRef = useRef<string | null>(null)
|
||||
|
||||
const isActiveRef = useRef(isActive)
|
||||
const audioUrlRef = useRef(audioUrl)
|
||||
const styleRef = useRef(style)
|
||||
const presetRef = useRef(preset)
|
||||
const qualityRef = useRef(quality)
|
||||
const viewportRef = useRef(viewport)
|
||||
|
||||
const [loadedDuration, setLoadedDuration] = useState<number | null>(null)
|
||||
const [isLoaded, setIsLoaded] = useState(false)
|
||||
|
||||
useEffect(() => {
|
||||
isActiveRef.current = isActive
|
||||
}, [isActive])
|
||||
|
||||
useEffect(() => {
|
||||
audioUrlRef.current = audioUrl
|
||||
}, [audioUrl])
|
||||
|
||||
useEffect(() => {
|
||||
styleRef.current = style
|
||||
}, [style])
|
||||
|
||||
useEffect(() => {
|
||||
presetRef.current = preset
|
||||
}, [preset])
|
||||
|
||||
useEffect(() => {
|
||||
qualityRef.current = quality
|
||||
}, [quality])
|
||||
|
||||
useEffect(() => {
|
||||
viewportRef.current = viewport
|
||||
}, [viewport])
|
||||
|
||||
const sendCommand = useCallback((type: string, payload: Record<string, unknown>) => {
|
||||
if (!readyRef.current) return
|
||||
const target = iframeRef.current?.contentWindow
|
||||
if (!target) return
|
||||
commandSeqRef.current += 1
|
||||
target.postMessage(
|
||||
{
|
||||
version: 1,
|
||||
id: `${commandIdPrefix}-${Date.now()}-${commandSeqRef.current}`,
|
||||
type,
|
||||
payload,
|
||||
},
|
||||
'*',
|
||||
)
|
||||
}, [commandIdPrefix, iframeRef])
|
||||
|
||||
const sendSettings = useCallback(() => {
|
||||
sendCommand('setStyle', { style: styleRef.current })
|
||||
sendCommand('setPreset', { preset: presetRef.current })
|
||||
sendCommand('setQuality', { quality: qualityRef.current })
|
||||
if (viewportRef.current) {
|
||||
sendCommand('setViewport', {
|
||||
cssWidth: viewportRef.current.cssWidth,
|
||||
cssHeight: viewportRef.current.cssHeight,
|
||||
dpr: viewportRef.current.dpr,
|
||||
})
|
||||
}
|
||||
}, [sendCommand])
|
||||
|
||||
const sendLoadAudio = useCallback((force = false) => {
|
||||
const url = resolveAudioUrl(String(audioUrlRef.current || ''))
|
||||
if (!url) return
|
||||
if (!force && lastRequestedAudioUrlRef.current === url) return
|
||||
lastRequestedAudioUrlRef.current = url
|
||||
setIsLoaded(false)
|
||||
setLoadedDuration(null)
|
||||
sendCommand('loadAudio', { url })
|
||||
lastSyncTimeRef.current = -1
|
||||
}, [sendCommand])
|
||||
|
||||
const onIframeLoad = useCallback(() => {
|
||||
readyRef.current = true
|
||||
sendSettings()
|
||||
sendCommand('setPaused', {
|
||||
paused: typeof document !== 'undefined' ? document.hidden || !Boolean(isPlayingRef.current) : !Boolean(isPlayingRef.current),
|
||||
})
|
||||
sendLoadAudio(true)
|
||||
}, [isPlayingRef, sendCommand, sendLoadAudio, sendSettings])
|
||||
|
||||
useEffect(() => {
|
||||
if (!isActive) {
|
||||
lastPlayingRef.current = false
|
||||
sendCommand('transportPause', {})
|
||||
sendCommand('setPaused', { paused: true })
|
||||
if (frameRef.current !== null) {
|
||||
cancelAnimationFrame(frameRef.current)
|
||||
frameRef.current = null
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
const tick = () => {
|
||||
if (!isActiveRef.current) return
|
||||
const playing = Boolean(isPlayingRef.current)
|
||||
|
||||
if (playing !== lastPlayingRef.current) {
|
||||
if (playing) {
|
||||
const time = readTime(getCurrentTime)
|
||||
const rate = readRate(getRate)
|
||||
sendCommand('transportPlay', { time, rate })
|
||||
sendCommand('setPaused', {
|
||||
paused: typeof document !== 'undefined' ? document.hidden : false,
|
||||
})
|
||||
lastRateRef.current = rate
|
||||
lastSyncTimeRef.current = time
|
||||
} else {
|
||||
sendCommand('transportPause', {})
|
||||
sendCommand('setPaused', { paused: true })
|
||||
}
|
||||
lastPlayingRef.current = playing
|
||||
}
|
||||
|
||||
if (playing) {
|
||||
const nextTime = readTime(getCurrentTime)
|
||||
if (Math.abs(nextTime - lastSyncTimeRef.current) > 0.25) {
|
||||
sendCommand('transportSeek', {
|
||||
time: nextTime,
|
||||
rate: readRate(getRate),
|
||||
})
|
||||
lastSyncTimeRef.current = nextTime
|
||||
}
|
||||
|
||||
const nextRate = readRate(getRate)
|
||||
if (Math.abs(nextRate - lastRateRef.current) > 0.001) {
|
||||
sendCommand('transportRate', { rate: nextRate })
|
||||
lastRateRef.current = nextRate
|
||||
}
|
||||
}
|
||||
|
||||
frameRef.current = requestAnimationFrame(tick)
|
||||
}
|
||||
|
||||
frameRef.current = requestAnimationFrame(tick)
|
||||
return () => {
|
||||
if (frameRef.current !== null) {
|
||||
cancelAnimationFrame(frameRef.current)
|
||||
frameRef.current = null
|
||||
}
|
||||
}
|
||||
}, [getCurrentTime, getRate, isActive, isPlayingRef, sendCommand])
|
||||
|
||||
useEffect(() => {
|
||||
if (!isActive) return
|
||||
sendSettings()
|
||||
}, [isActive, quality, preset, sendSettings, style, viewport])
|
||||
|
||||
useEffect(() => {
|
||||
if (!isActive || !audioUrl) return
|
||||
lastRequestedAudioUrlRef.current = null
|
||||
sendLoadAudio()
|
||||
}, [audioUrl, isActive, sendLoadAudio])
|
||||
|
||||
useEffect(() => {
|
||||
const onMessage = (event: MessageEvent) => {
|
||||
if (event.source !== iframeRef.current?.contentWindow) return
|
||||
const message = event.data as { type?: string; payload?: { duration?: unknown; message?: unknown; code?: unknown } } | null
|
||||
if (!message || typeof message !== 'object') return
|
||||
|
||||
if (message.type === 'ready') {
|
||||
readyRef.current = true
|
||||
sendSettings()
|
||||
sendCommand('setPaused', {
|
||||
paused: typeof document !== 'undefined' ? document.hidden || !Boolean(isPlayingRef.current) : !Boolean(isPlayingRef.current),
|
||||
})
|
||||
if (audioUrlRef.current) sendLoadAudio()
|
||||
return
|
||||
}
|
||||
|
||||
if (message.type === 'loaded') {
|
||||
const duration = Number(message.payload?.duration)
|
||||
setLoadedDuration(Number.isFinite(duration) && duration > 0 ? duration : null)
|
||||
setIsLoaded(true)
|
||||
return
|
||||
}
|
||||
|
||||
if (message.type === 'error') {
|
||||
const code = String(message.payload?.code || 'UNKNOWN')
|
||||
const details = String(message.payload?.message || 'unknown iframe error')
|
||||
const errorKey = `${code}:${details}`
|
||||
if (lastIframeErrorKeyRef.current === errorKey) return
|
||||
lastIframeErrorKeyRef.current = errorKey
|
||||
|
||||
// AUDIO_LOAD decode failures can be environment/media specific and are often
|
||||
// non-actionable at runtime, so avoid polluting the console with warnings.
|
||||
if (code === 'AUDIO_LOAD') {
|
||||
return
|
||||
}
|
||||
|
||||
console.warn('[useMuonIframeTransport]', code, details)
|
||||
}
|
||||
}
|
||||
|
||||
window.addEventListener('message', onMessage)
|
||||
return () => window.removeEventListener('message', onMessage)
|
||||
}, [iframeRef, isPlayingRef, sendCommand, sendLoadAudio, sendSettings])
|
||||
|
||||
useEffect(() => {
|
||||
const onVisible = () => {
|
||||
if (!isActiveRef.current) return
|
||||
const playing = Boolean(isPlayingRef.current)
|
||||
const paused = document.hidden || !playing
|
||||
sendCommand('setPaused', { paused })
|
||||
if (!document.hidden) {
|
||||
sendSettings()
|
||||
}
|
||||
}
|
||||
|
||||
document.addEventListener('visibilitychange', onVisible)
|
||||
return () => document.removeEventListener('visibilitychange', onVisible)
|
||||
}, [isPlayingRef, sendCommand, sendSettings])
|
||||
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
sendCommand('transportPause', {})
|
||||
sendCommand('setPaused', { paused: true })
|
||||
if (frameRef.current !== null) {
|
||||
cancelAnimationFrame(frameRef.current)
|
||||
frameRef.current = null
|
||||
}
|
||||
}
|
||||
}, [sendCommand])
|
||||
|
||||
return {
|
||||
onIframeLoad,
|
||||
sendCommand,
|
||||
loadedDuration,
|
||||
isLoaded,
|
||||
}
|
||||
}
|
||||
|
||||
export default useMuonIframeTransport
|
||||
@@ -0,0 +1,505 @@
|
||||
import { useCallback, useEffect, useMemo, useRef, useState, type RefObject } from 'react'
|
||||
import type { VisualizerProvider } from '../visualizers/providers'
|
||||
|
||||
export interface VisualizerViewport {
|
||||
cssWidth: number
|
||||
cssHeight: number
|
||||
dpr: number
|
||||
}
|
||||
|
||||
export type VisualizerParamValue = number | string | boolean
|
||||
|
||||
export interface VisualizerParamDescriptor {
|
||||
name: string
|
||||
label: string
|
||||
type: 'number' | 'enum' | 'boolean'
|
||||
default: VisualizerParamValue
|
||||
min?: number
|
||||
max?: number
|
||||
step?: number
|
||||
options?: string[]
|
||||
}
|
||||
|
||||
export interface VisualizerIframeCapabilities {
|
||||
providerId?: string
|
||||
protocolVersion?: 1
|
||||
presets: string[]
|
||||
params: VisualizerParamDescriptor[]
|
||||
palette: {
|
||||
supported: boolean
|
||||
slotCount: number
|
||||
}
|
||||
}
|
||||
|
||||
export interface UseVisualizerIframeTransportArgs {
|
||||
provider: VisualizerProvider
|
||||
iframeRef: RefObject<HTMLIFrameElement | null>
|
||||
isActive: boolean
|
||||
audioUrl: string | null
|
||||
isPlayingRef: RefObject<boolean>
|
||||
getCurrentTime: () => number
|
||||
getRate?: () => number
|
||||
style: 'default' | 'fluid' | 'tendrils' | 'spirit'
|
||||
preset?: string
|
||||
quality?: 'low' | 'balanced' | 'high'
|
||||
viewport?: VisualizerViewport
|
||||
palette?: unknown
|
||||
params?: Record<string, VisualizerParamValue> | null
|
||||
commandIdPrefix?: string
|
||||
}
|
||||
|
||||
export interface UseVisualizerIframeTransportResult {
|
||||
onIframeLoad: () => void
|
||||
sendCommand: (type: string, payload: Record<string, unknown>) => void
|
||||
loadedDuration: number | null
|
||||
isLoaded: boolean
|
||||
capabilities: VisualizerIframeCapabilities | null
|
||||
}
|
||||
|
||||
function isBlobUrl(value: string): boolean {
|
||||
return /^blob:/i.test(String(value || '').trim())
|
||||
}
|
||||
|
||||
function readRate(getRate?: () => number): number {
|
||||
const value = Number(getRate ? getRate() : 1)
|
||||
return Number.isFinite(value) && value > 0 ? value : 1
|
||||
}
|
||||
|
||||
function readTime(getCurrentTime: () => number): number {
|
||||
const value = Number(getCurrentTime())
|
||||
return Number.isFinite(value) && value >= 0 ? value : 0
|
||||
}
|
||||
|
||||
function resolveAudioUrl(raw: string): string {
|
||||
const value = String(raw || '').trim()
|
||||
if (!value) return ''
|
||||
|
||||
if (/^https?:\/\//i.test(value)) return value
|
||||
|
||||
if (value.startsWith('//')) {
|
||||
const protocol = typeof window !== 'undefined' && window.location.protocol === 'https:' ? 'https:' : 'http:'
|
||||
return `${protocol}${value}`
|
||||
}
|
||||
|
||||
if (typeof window === 'undefined') return value
|
||||
try {
|
||||
return new URL(value, window.location.origin).toString()
|
||||
} catch {
|
||||
return value
|
||||
}
|
||||
}
|
||||
|
||||
function readNow(): number {
|
||||
if (typeof performance !== 'undefined' && typeof performance.now === 'function') {
|
||||
return performance.now()
|
||||
}
|
||||
return Date.now()
|
||||
}
|
||||
|
||||
function normalizeCapabilities(input: unknown): VisualizerIframeCapabilities {
|
||||
const payload = (input && typeof input === 'object' ? input : {}) as Record<string, unknown>
|
||||
const paramsRaw = Array.isArray(payload.params) ? payload.params : []
|
||||
const params: VisualizerParamDescriptor[] = paramsRaw
|
||||
.map((entry) => {
|
||||
const item = (entry && typeof entry === 'object' ? entry : {}) as Record<string, unknown>
|
||||
const type = item.type === 'enum' || item.type === 'boolean' ? item.type : 'number'
|
||||
const descriptor: VisualizerParamDescriptor = {
|
||||
name: String(item.name || ''),
|
||||
label: String(item.label || item.name || ''),
|
||||
type,
|
||||
default: (item.default ?? (type === 'boolean' ? false : type === 'enum' ? '' : 0)) as VisualizerParamValue,
|
||||
}
|
||||
if (Number.isFinite(Number(item.min))) descriptor.min = Number(item.min)
|
||||
if (Number.isFinite(Number(item.max))) descriptor.max = Number(item.max)
|
||||
if (Number.isFinite(Number(item.step))) descriptor.step = Number(item.step)
|
||||
if (Array.isArray(item.options)) descriptor.options = item.options.map((option) => String(option))
|
||||
return descriptor
|
||||
})
|
||||
.filter((param) => Boolean(param.name))
|
||||
|
||||
const paletteRaw = payload.palette && typeof payload.palette === 'object'
|
||||
? (payload.palette as Record<string, unknown>)
|
||||
: {}
|
||||
|
||||
return {
|
||||
providerId: typeof payload.providerId === 'string' ? payload.providerId : undefined,
|
||||
protocolVersion: payload.protocolVersion === 1 ? 1 : undefined,
|
||||
presets: Array.isArray(payload.presets) ? payload.presets.map((preset) => String(preset)) : [],
|
||||
params,
|
||||
palette: {
|
||||
supported: Boolean(paletteRaw.supported),
|
||||
slotCount: Number.isFinite(Number(paletteRaw.slotCount)) ? Math.max(0, Math.floor(Number(paletteRaw.slotCount))) : 0,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
export function useVisualizerIframeTransport(args: UseVisualizerIframeTransportArgs): UseVisualizerIframeTransportResult {
|
||||
const {
|
||||
provider,
|
||||
iframeRef,
|
||||
isActive,
|
||||
audioUrl,
|
||||
isPlayingRef,
|
||||
getCurrentTime,
|
||||
getRate,
|
||||
style,
|
||||
preset,
|
||||
quality,
|
||||
viewport,
|
||||
palette,
|
||||
params,
|
||||
commandIdPrefix,
|
||||
} = args
|
||||
|
||||
const effectiveCommandPrefix = useMemo(() => commandIdPrefix || provider.id, [commandIdPrefix, provider.id])
|
||||
const readyRef = useRef(false)
|
||||
const frameRef = useRef<number | null>(null)
|
||||
const lastRateRef = useRef(1)
|
||||
const lastPlayingRef = useRef(false)
|
||||
const commandSeqRef = useRef(0)
|
||||
const lastRequestedAudioUrlRef = useRef<string | null>(null)
|
||||
const lastIframeErrorKeyRef = useRef<string | null>(null)
|
||||
const audioLoadRequestIdRef = useRef(0)
|
||||
const lastHostTimeSampleRef = useRef<number | null>(null)
|
||||
const lastHostWallClockRef = useRef<number | null>(null)
|
||||
const lastPaletteKeyRef = useRef('')
|
||||
const lastParamsKeyRef = useRef('')
|
||||
|
||||
const isActiveRef = useRef(isActive)
|
||||
const audioUrlRef = useRef(audioUrl)
|
||||
const styleRef = useRef(style)
|
||||
const presetRef = useRef(preset)
|
||||
const qualityRef = useRef(quality)
|
||||
const viewportRef = useRef(viewport)
|
||||
const paletteRef = useRef(palette)
|
||||
const paramsRef = useRef(params)
|
||||
|
||||
const [loadedDuration, setLoadedDuration] = useState<number | null>(null)
|
||||
const [isLoaded, setIsLoaded] = useState(false)
|
||||
const [capabilities, setCapabilities] = useState<VisualizerIframeCapabilities | null>(null)
|
||||
|
||||
useEffect(() => {
|
||||
isActiveRef.current = isActive
|
||||
}, [isActive])
|
||||
|
||||
useEffect(() => {
|
||||
audioUrlRef.current = audioUrl
|
||||
}, [audioUrl])
|
||||
|
||||
useEffect(() => {
|
||||
styleRef.current = style
|
||||
}, [style])
|
||||
|
||||
useEffect(() => {
|
||||
presetRef.current = preset
|
||||
}, [preset])
|
||||
|
||||
useEffect(() => {
|
||||
qualityRef.current = quality
|
||||
}, [quality])
|
||||
|
||||
useEffect(() => {
|
||||
viewportRef.current = viewport
|
||||
}, [viewport])
|
||||
|
||||
useEffect(() => {
|
||||
paletteRef.current = palette
|
||||
}, [palette])
|
||||
|
||||
useEffect(() => {
|
||||
paramsRef.current = params
|
||||
}, [params])
|
||||
|
||||
const sendCommand = useCallback((type: string, payload: Record<string, unknown>) => {
|
||||
if (!readyRef.current) return
|
||||
const target = iframeRef.current?.contentWindow
|
||||
if (!target) return
|
||||
commandSeqRef.current += 1
|
||||
target.postMessage(
|
||||
{
|
||||
version: 1,
|
||||
id: `${effectiveCommandPrefix}-${Date.now()}-${commandSeqRef.current}`,
|
||||
type,
|
||||
payload,
|
||||
},
|
||||
'*',
|
||||
)
|
||||
}, [effectiveCommandPrefix, iframeRef])
|
||||
|
||||
const sendSettings = useCallback(() => {
|
||||
sendCommand('setStyle', { style: styleRef.current })
|
||||
sendCommand('setPreset', { preset: presetRef.current })
|
||||
sendCommand('setQuality', { quality: qualityRef.current })
|
||||
if (viewportRef.current) {
|
||||
sendCommand('setViewport', {
|
||||
cssWidth: viewportRef.current.cssWidth,
|
||||
cssHeight: viewportRef.current.cssHeight,
|
||||
dpr: viewportRef.current.dpr,
|
||||
})
|
||||
}
|
||||
}, [sendCommand])
|
||||
|
||||
const sendPalette = useCallback(() => {
|
||||
const rawPalette = paletteRef.current
|
||||
const colorList = Array.isArray(rawPalette)
|
||||
? rawPalette
|
||||
: (rawPalette && typeof rawPalette === 'object' && Array.isArray((rawPalette as { speakerPalette?: unknown }).speakerPalette)
|
||||
? (rawPalette as { speakerPalette: unknown[] }).speakerPalette
|
||||
: [])
|
||||
const colors = colorList.map((color) => String(color)).filter((color) => color.trim().length > 0)
|
||||
const key = colors.join('|')
|
||||
if (key === lastPaletteKeyRef.current) return
|
||||
lastPaletteKeyRef.current = key
|
||||
sendCommand('setPalette', { colors })
|
||||
}, [sendCommand])
|
||||
|
||||
const sendParams = useCallback(() => {
|
||||
const entries = Object.entries(paramsRef.current || {})
|
||||
.filter(([name]) => Boolean(String(name).trim()))
|
||||
.sort(([a], [b]) => a.localeCompare(b))
|
||||
const key = JSON.stringify(entries)
|
||||
if (key === lastParamsKeyRef.current) return
|
||||
lastParamsKeyRef.current = key
|
||||
entries.forEach(([name, value]) => {
|
||||
sendCommand('setParam', { name, value })
|
||||
})
|
||||
}, [sendCommand])
|
||||
|
||||
const sendLoadAudio = useCallback((force = false) => {
|
||||
const url = resolveAudioUrl(String(audioUrlRef.current || ''))
|
||||
if (!url) return
|
||||
if (!force && lastRequestedAudioUrlRef.current === url) return
|
||||
audioLoadRequestIdRef.current += 1
|
||||
const requestId = audioLoadRequestIdRef.current
|
||||
lastRequestedAudioUrlRef.current = url
|
||||
setIsLoaded(false)
|
||||
setLoadedDuration(null)
|
||||
lastHostTimeSampleRef.current = null
|
||||
lastHostWallClockRef.current = null
|
||||
|
||||
if (isBlobUrl(url)) {
|
||||
void (async () => {
|
||||
try {
|
||||
const response = await fetch(url)
|
||||
if (!response.ok) {
|
||||
throw new Error(`blob audio fetch failed: ${response.status}`)
|
||||
}
|
||||
const bytes = await response.arrayBuffer()
|
||||
if (audioLoadRequestIdRef.current !== requestId) return
|
||||
if (lastRequestedAudioUrlRef.current !== url) return
|
||||
sendCommand('loadAudioData', { bytes })
|
||||
} catch (error) {
|
||||
const details = error instanceof Error ? error.message : String(error)
|
||||
console.warn('[useVisualizerIframeTransport] AUDIO_LOAD_BLOB', details)
|
||||
}
|
||||
})()
|
||||
return
|
||||
}
|
||||
|
||||
sendCommand('loadAudio', { url })
|
||||
}, [sendCommand])
|
||||
|
||||
const onIframeLoad = useCallback(() => {
|
||||
readyRef.current = true
|
||||
sendSettings()
|
||||
sendPalette()
|
||||
sendParams()
|
||||
sendCommand('setPaused', {
|
||||
paused: typeof document !== 'undefined' ? document.hidden || !Boolean(isPlayingRef.current) : !Boolean(isPlayingRef.current),
|
||||
})
|
||||
sendLoadAudio(true)
|
||||
}, [isPlayingRef, sendCommand, sendLoadAudio, sendPalette, sendParams, sendSettings])
|
||||
|
||||
useEffect(() => {
|
||||
setCapabilities(null)
|
||||
lastPaletteKeyRef.current = ''
|
||||
lastParamsKeyRef.current = ''
|
||||
}, [provider.id])
|
||||
|
||||
useEffect(() => {
|
||||
if (!isActive) {
|
||||
lastPlayingRef.current = false
|
||||
lastHostTimeSampleRef.current = null
|
||||
lastHostWallClockRef.current = null
|
||||
sendCommand('transportPause', {})
|
||||
sendCommand('setPaused', { paused: true })
|
||||
if (frameRef.current !== null) {
|
||||
cancelAnimationFrame(frameRef.current)
|
||||
frameRef.current = null
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
const tick = () => {
|
||||
if (!isActiveRef.current) return
|
||||
const playing = Boolean(isPlayingRef.current)
|
||||
|
||||
if (playing !== lastPlayingRef.current) {
|
||||
if (playing) {
|
||||
const time = readTime(getCurrentTime)
|
||||
const rate = readRate(getRate)
|
||||
sendCommand('transportPlay', { time, rate })
|
||||
sendCommand('setPaused', {
|
||||
paused: typeof document !== 'undefined' ? document.hidden : false,
|
||||
})
|
||||
lastRateRef.current = rate
|
||||
lastHostTimeSampleRef.current = time
|
||||
lastHostWallClockRef.current = readNow()
|
||||
} else {
|
||||
sendCommand('transportPause', {})
|
||||
sendCommand('setPaused', { paused: true })
|
||||
lastHostTimeSampleRef.current = null
|
||||
lastHostWallClockRef.current = null
|
||||
}
|
||||
lastPlayingRef.current = playing
|
||||
}
|
||||
|
||||
if (playing) {
|
||||
const nextRate = readRate(getRate)
|
||||
const nextTime = readTime(getCurrentTime)
|
||||
const lastTime = lastHostTimeSampleRef.current
|
||||
const lastWallClock = lastHostWallClockRef.current
|
||||
|
||||
if (lastTime !== null && lastWallClock !== null) {
|
||||
const now = readNow()
|
||||
const elapsed = Math.max(0, (now - lastWallClock) / 1000)
|
||||
const expectedTime = lastTime + elapsed * lastRateRef.current
|
||||
if (Math.abs(nextTime - expectedTime) > 0.25) {
|
||||
sendCommand('transportSeek', {
|
||||
time: nextTime,
|
||||
rate: nextRate,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
lastHostTimeSampleRef.current = nextTime
|
||||
lastHostWallClockRef.current = readNow()
|
||||
|
||||
if (Math.abs(nextRate - lastRateRef.current) > 0.001) {
|
||||
sendCommand('transportRate', { rate: nextRate })
|
||||
lastRateRef.current = nextRate
|
||||
}
|
||||
}
|
||||
|
||||
frameRef.current = requestAnimationFrame(tick)
|
||||
}
|
||||
|
||||
frameRef.current = requestAnimationFrame(tick)
|
||||
return () => {
|
||||
if (frameRef.current !== null) {
|
||||
cancelAnimationFrame(frameRef.current)
|
||||
frameRef.current = null
|
||||
}
|
||||
}
|
||||
}, [getCurrentTime, getRate, isActive, isPlayingRef, sendCommand])
|
||||
|
||||
useEffect(() => {
|
||||
if (!isActive) return
|
||||
sendSettings()
|
||||
}, [isActive, quality, preset, sendSettings, style, viewport])
|
||||
|
||||
useEffect(() => {
|
||||
if (!isActive) return
|
||||
sendPalette()
|
||||
}, [isActive, palette, sendPalette])
|
||||
|
||||
useEffect(() => {
|
||||
if (!isActive) return
|
||||
sendParams()
|
||||
}, [isActive, params, sendParams])
|
||||
|
||||
useEffect(() => {
|
||||
if (!isActive || !audioUrl) return
|
||||
lastRequestedAudioUrlRef.current = null
|
||||
sendLoadAudio()
|
||||
}, [audioUrl, isActive, sendLoadAudio])
|
||||
|
||||
useEffect(() => {
|
||||
const onMessage = (event: MessageEvent) => {
|
||||
if (event.source !== iframeRef.current?.contentWindow) return
|
||||
const message = event.data as { type?: string; payload?: unknown } | null
|
||||
if (!message || typeof message !== 'object') return
|
||||
|
||||
if (message.type === 'ready') {
|
||||
readyRef.current = true
|
||||
sendSettings()
|
||||
sendPalette()
|
||||
sendParams()
|
||||
sendCommand('setPaused', {
|
||||
paused: typeof document !== 'undefined' ? document.hidden || !Boolean(isPlayingRef.current) : !Boolean(isPlayingRef.current),
|
||||
})
|
||||
if (audioUrlRef.current) sendLoadAudio()
|
||||
return
|
||||
}
|
||||
|
||||
if (message.type === 'capabilities') {
|
||||
setCapabilities(normalizeCapabilities(message.payload))
|
||||
return
|
||||
}
|
||||
|
||||
if (message.type === 'loaded') {
|
||||
const payload = (message.payload && typeof message.payload === 'object' ? message.payload : {}) as { duration?: unknown }
|
||||
const duration = Number(payload.duration)
|
||||
setLoadedDuration(Number.isFinite(duration) && duration > 0 ? duration : null)
|
||||
setIsLoaded(true)
|
||||
return
|
||||
}
|
||||
|
||||
if (message.type === 'error') {
|
||||
const payload = (message.payload && typeof message.payload === 'object' ? message.payload : {}) as { code?: unknown; message?: unknown }
|
||||
const code = String(payload.code || 'UNKNOWN')
|
||||
const details = String(payload.message || 'unknown iframe error')
|
||||
const errorKey = `${code}:${details}`
|
||||
if (lastIframeErrorKeyRef.current === errorKey) return
|
||||
lastIframeErrorKeyRef.current = errorKey
|
||||
|
||||
if (code === 'AUDIO_LOAD') {
|
||||
return
|
||||
}
|
||||
|
||||
console.warn('[useVisualizerIframeTransport]', code, details)
|
||||
}
|
||||
}
|
||||
|
||||
window.addEventListener('message', onMessage)
|
||||
return () => window.removeEventListener('message', onMessage)
|
||||
}, [iframeRef, isPlayingRef, sendCommand, sendLoadAudio, sendPalette, sendParams, sendSettings])
|
||||
|
||||
useEffect(() => {
|
||||
const onVisible = () => {
|
||||
if (!isActiveRef.current) return
|
||||
const playing = Boolean(isPlayingRef.current)
|
||||
const paused = document.hidden || !playing
|
||||
sendCommand('setPaused', { paused })
|
||||
if (!document.hidden) {
|
||||
sendSettings()
|
||||
sendPalette()
|
||||
sendParams()
|
||||
}
|
||||
}
|
||||
|
||||
document.addEventListener('visibilitychange', onVisible)
|
||||
return () => document.removeEventListener('visibilitychange', onVisible)
|
||||
}, [isPlayingRef, sendCommand, sendPalette, sendParams, sendSettings])
|
||||
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
sendCommand('transportPause', {})
|
||||
sendCommand('setPaused', { paused: true })
|
||||
if (frameRef.current !== null) {
|
||||
cancelAnimationFrame(frameRef.current)
|
||||
frameRef.current = null
|
||||
}
|
||||
}
|
||||
}, [sendCommand])
|
||||
|
||||
return {
|
||||
onIframeLoad,
|
||||
sendCommand,
|
||||
loadedDuration,
|
||||
isLoaded,
|
||||
capabilities,
|
||||
}
|
||||
}
|
||||
|
||||
export default useVisualizerIframeTransport
|
||||
@@ -2,18 +2,18 @@ import { useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react'
|
||||
import { Pause, Play, RotateCcw, Upload } from 'lucide-react'
|
||||
import { parseBlob } from 'music-metadata-browser'
|
||||
import VisualizerBackdrop, { type VisualizerMode, type VisualizerSettings, type VizShape } from '../components/visualizers/VisualizerBackdrop'
|
||||
import RendererControlsPanel from '../components/visualizers/RendererControlsPanel'
|
||||
import type { MuonIframePreset } from '../components/visualizers/visualizerTypes'
|
||||
import { useAudioFeatures } from '../hooks/useAudioFeatures'
|
||||
import { useMuonIframeTransport } from '../hooks/useMuonIframeTransport'
|
||||
import { useVisualizerIframeTransport, type VisualizerParamValue } from '../hooks/useVisualizerIframeTransport'
|
||||
import { useVisualizerPalette } from '../hooks/useVisualizerPalette'
|
||||
import { getAppSettings } from '../api/client'
|
||||
import { resolveVisualizerProviders, withDefaultVisualizerPreset, type VisualizerProvider } from '../visualizers/providers'
|
||||
|
||||
type LabMode = Exclude<VisualizerMode, null> | 'muon_iframe'
|
||||
type MuonStyle = 'default' | 'fluid' | 'tendrils' | 'spirit'
|
||||
|
||||
const MODES: LabMode[] = ['muon_iframe']
|
||||
const DEFAULT_MUON_IFRAME_URL = 'http://localhost:3011'
|
||||
const MUON_IFRAME_REV = '20260427c'
|
||||
const SHAPES: VizShape[] = ['dynamic', 'circle', 'square', 'star', 'hexagon', 'flower']
|
||||
const QUALITIES: NonNullable<VisualizerSettings['quality']>[] = ['low', 'balanced', 'high']
|
||||
const FPS_CAPS: NonNullable<VisualizerSettings['fps_cap']>[] = [30, 60, 120]
|
||||
@@ -25,59 +25,6 @@ const MUON_PRESETS_BY_STYLE: Record<MuonStyle, MuonIframePreset[]> = {
|
||||
spirit: ['spirit_aether', 'spirit_aurora', 'spirit_lumen'],
|
||||
}
|
||||
|
||||
function isLoopbackHost(hostname: string): boolean {
|
||||
const value = hostname.trim().toLowerCase()
|
||||
return value === 'localhost' || value === '127.0.0.1' || value === '::1'
|
||||
}
|
||||
|
||||
function defaultMuonIframeUrl(): string {
|
||||
if (typeof window === 'undefined') return DEFAULT_MUON_IFRAME_URL
|
||||
const protocol = window.location.protocol === 'https:' ? 'https:' : 'http:'
|
||||
const hostname = window.location.hostname || 'localhost'
|
||||
return `${protocol}//${hostname}:3011`
|
||||
}
|
||||
|
||||
function resolveMuonIframeUrl(raw: string | undefined): string {
|
||||
const cleaned = String(raw || '').trim()
|
||||
if (!cleaned) return defaultMuonIframeUrl()
|
||||
|
||||
if (/^https?:\/\//i.test(cleaned)) {
|
||||
try {
|
||||
const parsed = new URL(cleaned)
|
||||
if (typeof window !== 'undefined' && isLoopbackHost(parsed.hostname) && !isLoopbackHost(window.location.hostname)) {
|
||||
parsed.hostname = window.location.hostname
|
||||
}
|
||||
return parsed.toString()
|
||||
} catch {
|
||||
return defaultMuonIframeUrl()
|
||||
}
|
||||
}
|
||||
|
||||
const protocol = typeof window !== 'undefined' && window.location.protocol === 'https:' ? 'https://' : 'http://'
|
||||
const candidate = cleaned.startsWith('//') ? `${protocol}${cleaned.slice(2)}` : `${protocol}${cleaned}`
|
||||
try {
|
||||
const parsed = new URL(candidate)
|
||||
if (!/^https?:$/i.test(parsed.protocol)) return defaultMuonIframeUrl()
|
||||
if (typeof window !== 'undefined' && isLoopbackHost(parsed.hostname) && !isLoopbackHost(window.location.hostname)) {
|
||||
parsed.hostname = window.location.hostname
|
||||
}
|
||||
return parsed.toString()
|
||||
} catch {
|
||||
return defaultMuonIframeUrl()
|
||||
}
|
||||
}
|
||||
|
||||
function withDefaultPreset(rawUrl: string): string {
|
||||
try {
|
||||
const url = new URL(rawUrl)
|
||||
if (!url.searchParams.has('preset')) url.searchParams.set('preset', 'default')
|
||||
url.searchParams.set('se_rev', MUON_IFRAME_REV)
|
||||
return url.toString()
|
||||
} catch {
|
||||
return rawUrl
|
||||
}
|
||||
}
|
||||
|
||||
function clamp(value: number, min: number, max: number): number {
|
||||
return Math.max(min, Math.min(max, value))
|
||||
}
|
||||
@@ -118,10 +65,11 @@ export default function VisualizerLabPage() {
|
||||
const [dims, setDims] = useState<{ w: number; h: number }>({ w: 640, h: 360 })
|
||||
|
||||
const [ctxVersion, setCtxVersion] = useState(0)
|
||||
const [muonIframeUrl, setMuonIframeUrl] = useState(() => defaultMuonIframeUrl())
|
||||
const [providers, setProviders] = useState(() => resolveVisualizerProviders())
|
||||
const [muonIframeError, setMuonIframeError] = useState(false)
|
||||
const [muonStyle, setMuonStyle] = useState<MuonStyle>('default')
|
||||
const [muonPreset, setMuonPreset] = useState<MuonIframePreset>('default')
|
||||
const [muonParamsByStyle, setMuonParamsByStyle] = useState<Partial<Record<MuonStyle, Record<string, VisualizerParamValue>>>>({})
|
||||
|
||||
const audioRef = useRef<HTMLAudioElement>(null)
|
||||
const containerRef = useRef<HTMLDivElement>(null)
|
||||
@@ -165,16 +113,29 @@ export default function VisualizerLabPage() {
|
||||
// featuresRef is a stable mutable ref
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [])
|
||||
const muonEmbedUrl = useMemo(() => withDefaultPreset(muonIframeUrl), [muonIframeUrl])
|
||||
const activeIframeProvider = useMemo<VisualizerProvider>(() => {
|
||||
const providerId = muonStyle === 'default' ? 'muon' : muonStyle
|
||||
return providers[providerId]
|
||||
}, [muonStyle, providers])
|
||||
const activeIframeLabel = useMemo(
|
||||
() => activeIframeProvider.id.charAt(0).toUpperCase() + activeIframeProvider.id.slice(1),
|
||||
[activeIframeProvider.id],
|
||||
)
|
||||
const activeIframeUrl = useMemo(() => {
|
||||
if (activeIframeProvider.id === 'muon') return withDefaultVisualizerPreset(activeIframeProvider.iframeUrl)
|
||||
return activeIframeProvider.iframeUrl
|
||||
}, [activeIframeProvider.id, activeIframeProvider.iframeUrl])
|
||||
const activeMuonParams = useMemo(() => muonParamsByStyle[muonStyle] ?? {}, [muonParamsByStyle, muonStyle])
|
||||
|
||||
const { onIframeLoad: onMuonIframeLoad } = useMuonIframeTransport({
|
||||
const { onIframeLoad: onMuonIframeLoad, capabilities: muonCapabilities } = useVisualizerIframeTransport({
|
||||
provider: activeIframeProvider,
|
||||
iframeRef,
|
||||
isActive: mode === 'muon_iframe',
|
||||
audioUrl: audioUrl || null,
|
||||
isPlayingRef,
|
||||
getCurrentTime: () => Number(audioRef.current?.currentTime || 0),
|
||||
getRate: () => Number(audioRef.current?.playbackRate || 1),
|
||||
style: muonStyle,
|
||||
style: activeIframeProvider.id === 'muon' ? muonStyle : 'default',
|
||||
preset: muonPreset,
|
||||
quality,
|
||||
viewport: {
|
||||
@@ -182,6 +143,8 @@ export default function VisualizerLabPage() {
|
||||
cssHeight: dims.h,
|
||||
dpr: typeof window === 'undefined' ? 1 : window.devicePixelRatio || 1,
|
||||
},
|
||||
palette: paletteRef.current,
|
||||
params: activeMuonParams,
|
||||
commandIdPrefix: 'lab',
|
||||
})
|
||||
|
||||
@@ -204,11 +167,11 @@ export default function VisualizerLabPage() {
|
||||
void getAppSettings()
|
||||
.then((settings) => {
|
||||
if (cancelled) return
|
||||
setMuonIframeUrl(resolveMuonIframeUrl(settings.muon_iframe_url))
|
||||
setProviders(resolveVisualizerProviders(settings))
|
||||
})
|
||||
.catch(() => {
|
||||
if (cancelled) return
|
||||
setMuonIframeUrl(defaultMuonIframeUrl())
|
||||
setProviders(resolveVisualizerProviders())
|
||||
})
|
||||
|
||||
return () => {
|
||||
@@ -352,10 +315,10 @@ export default function VisualizerLabPage() {
|
||||
{mode === 'muon_iframe' ? (
|
||||
<div className="relative" style={{ height: dims.h }}>
|
||||
<iframe
|
||||
key={muonEmbedUrl}
|
||||
key={activeIframeUrl}
|
||||
ref={iframeRef}
|
||||
src={muonEmbedUrl}
|
||||
title="Muon music visualizer"
|
||||
src={activeIframeUrl}
|
||||
title={`${activeIframeLabel} music visualizer`}
|
||||
className="absolute inset-0 h-full w-full border-0"
|
||||
loading="eager"
|
||||
referrerPolicy="no-referrer"
|
||||
@@ -369,7 +332,7 @@ export default function VisualizerLabPage() {
|
||||
{muonIframeError ? (
|
||||
<div className="absolute inset-0 flex items-center justify-center p-4">
|
||||
<div className="rounded bg-black/70 px-3 py-2 text-xs text-red-200">
|
||||
Could not load Muon iframe.
|
||||
Could not load {activeIframeLabel} iframe.
|
||||
</div>
|
||||
</div>
|
||||
) : null}
|
||||
@@ -534,6 +497,38 @@ export default function VisualizerLabPage() {
|
||||
))}
|
||||
</select>
|
||||
</label>
|
||||
|
||||
<label className="flex flex-col gap-1 text-xs text-gray-300">
|
||||
Quality
|
||||
<select
|
||||
value={quality}
|
||||
onChange={(event) => setQuality(event.target.value as NonNullable<VisualizerSettings['quality']>)}
|
||||
className="rounded-md border border-gray-700 bg-gray-800 px-2 py-1.5 text-sm text-gray-200"
|
||||
>
|
||||
{QUALITIES.map((entry) => <option key={entry} value={entry}>{entry}</option>)}
|
||||
</select>
|
||||
</label>
|
||||
|
||||
<RendererControlsPanel
|
||||
capabilities={muonCapabilities}
|
||||
values={activeMuonParams}
|
||||
onValueChange={(name, value) => {
|
||||
setMuonParamsByStyle((prev) => ({
|
||||
...prev,
|
||||
[muonStyle]: {
|
||||
...(prev[muonStyle] ?? {}),
|
||||
[name]: value,
|
||||
},
|
||||
}))
|
||||
}}
|
||||
onReset={() => {
|
||||
setMuonParamsByStyle((prev) => ({
|
||||
...prev,
|
||||
[muonStyle]: {},
|
||||
}))
|
||||
}}
|
||||
className="rounded-md border border-gray-700 bg-gray-900/70 p-2"
|
||||
/>
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
import type { AppSettings } from '../api/client'
|
||||
|
||||
export type VisualizerProviderId = 'muon' | 'fluid' | 'tendrils' | 'spirit'
|
||||
|
||||
export interface VisualizerProvider {
|
||||
id: VisualizerProviderId
|
||||
iframeUrl: string
|
||||
capabilities: {
|
||||
webgl: boolean
|
||||
webgl2: boolean
|
||||
}
|
||||
protocolVersion: 1
|
||||
}
|
||||
|
||||
export interface VisualizerProviderRegistry {
|
||||
muon: VisualizerProvider
|
||||
fluid: VisualizerProvider
|
||||
tendrils: VisualizerProvider
|
||||
spirit: VisualizerProvider
|
||||
}
|
||||
|
||||
const DEFAULT_MUON_IFRAME_PORT = 3011
|
||||
const DEFAULT_FLUID_IFRAME_PORT = 3012
|
||||
const DEFAULT_TENDRILS_IFRAME_PORT = 3013
|
||||
const DEFAULT_SPIRIT_IFRAME_PORT = 3014
|
||||
export const MUON_IFRAME_REV = '20260427c'
|
||||
|
||||
function isLoopbackHost(hostname: string): boolean {
|
||||
const value = hostname.trim().toLowerCase()
|
||||
return value === 'localhost' || value === '127.0.0.1' || value === '::1'
|
||||
}
|
||||
|
||||
function defaultIframeUrl(port: number): string {
|
||||
if (typeof window === 'undefined') return `http://localhost:${port}`
|
||||
const protocol = window.location.protocol === 'https:' ? 'https:' : 'http:'
|
||||
const hostname = window.location.hostname || 'localhost'
|
||||
return `${protocol}//${hostname}:${port}`
|
||||
}
|
||||
|
||||
function resolveIframeUrl(raw: string | undefined, fallbackPort: number): string {
|
||||
const fallback = defaultIframeUrl(fallbackPort)
|
||||
const cleaned = String(raw || '').trim()
|
||||
if (!cleaned) return fallback
|
||||
|
||||
if (/^https?:\/\//i.test(cleaned)) {
|
||||
try {
|
||||
const parsed = new URL(cleaned)
|
||||
if (typeof window !== 'undefined' && isLoopbackHost(parsed.hostname) && !isLoopbackHost(window.location.hostname)) {
|
||||
parsed.hostname = window.location.hostname
|
||||
}
|
||||
return parsed.toString()
|
||||
} catch {
|
||||
return fallback
|
||||
}
|
||||
}
|
||||
|
||||
const protocol = typeof window !== 'undefined' && window.location.protocol === 'https:' ? 'https://' : 'http://'
|
||||
const candidate = cleaned.startsWith('//') ? `${protocol}${cleaned.slice(2)}` : `${protocol}${cleaned}`
|
||||
try {
|
||||
const parsed = new URL(candidate)
|
||||
if (!/^https?:$/i.test(parsed.protocol)) return fallback
|
||||
if (typeof window !== 'undefined' && isLoopbackHost(parsed.hostname) && !isLoopbackHost(window.location.hostname)) {
|
||||
parsed.hostname = window.location.hostname
|
||||
}
|
||||
return parsed.toString()
|
||||
} catch {
|
||||
return fallback
|
||||
}
|
||||
}
|
||||
|
||||
export function withDefaultVisualizerPreset(rawUrl: string): string {
|
||||
try {
|
||||
const url = new URL(rawUrl)
|
||||
if (!url.searchParams.has('preset')) url.searchParams.set('preset', 'default')
|
||||
if (!url.searchParams.has('se_rev')) url.searchParams.set('se_rev', MUON_IFRAME_REV)
|
||||
return url.toString()
|
||||
} catch {
|
||||
return rawUrl
|
||||
}
|
||||
}
|
||||
|
||||
export function resolveVisualizerProviders(settings?: Partial<AppSettings> | null): VisualizerProviderRegistry {
|
||||
const anySettings = (settings || {}) as Partial<AppSettings> & Record<string, unknown>
|
||||
const makeProvider = (id: VisualizerProviderId, iframeUrl: string): VisualizerProvider => ({
|
||||
id,
|
||||
iframeUrl,
|
||||
capabilities: { webgl: true, webgl2: true },
|
||||
protocolVersion: 1,
|
||||
})
|
||||
|
||||
return {
|
||||
muon: makeProvider('muon', resolveIframeUrl(anySettings.muon_iframe_url, DEFAULT_MUON_IFRAME_PORT)),
|
||||
fluid: makeProvider('fluid', resolveIframeUrl(String(anySettings.fluid_iframe_url || ''), DEFAULT_FLUID_IFRAME_PORT)),
|
||||
tendrils: makeProvider('tendrils', resolveIframeUrl(String(anySettings.tendrils_iframe_url || ''), DEFAULT_TENDRILS_IFRAME_PORT)),
|
||||
spirit: makeProvider('spirit', resolveIframeUrl(String(anySettings.spirit_iframe_url || ''), DEFAULT_SPIRIT_IFRAME_PORT)),
|
||||
}
|
||||
}
|
||||
+6
-13
@@ -1,3 +1,4 @@
|
||||
# syntax=docker/dockerfile:1.4
|
||||
FROM node:20-alpine AS builder
|
||||
|
||||
ARG MUON_GIT_REF=main
|
||||
@@ -8,12 +9,10 @@ RUN apk add --no-cache git \
|
||||
&& git clone --depth 1 --branch "${MUON_GIT_REF}" https://github.com/najafmohammed/muon-music-visualizer.git app
|
||||
|
||||
COPY overlayless.css /tmp/overlayless.css
|
||||
COPY stemengine-bootstrap.js /tmp/stemengine-bootstrap.js
|
||||
COPY stemengine-router.js /tmp/stemengine-router.js
|
||||
COPY stemengine-audio-engine.js /tmp/stemengine-audio-engine.js
|
||||
COPY stemengine-feature-extractor.js /tmp/stemengine-feature-extractor.js
|
||||
COPY fluid-render-graph.js /tmp/fluid-render-graph.js
|
||||
COPY tendrils-render-graph.js /tmp/tendrils-render-graph.js
|
||||
COPY --from=shared stemengine-bootstrap.js /tmp/stemengine-bootstrap.js
|
||||
COPY --from=shared stemengine-router.js /tmp/stemengine-router.js
|
||||
COPY --from=shared stemengine-audio-engine.js /tmp/stemengine-audio-engine.js
|
||||
COPY --from=shared stemengine-feature-extractor.js /tmp/stemengine-feature-extractor.js
|
||||
COPY spirit-render-graph.js /tmp/spirit-render-graph.js
|
||||
|
||||
WORKDIR /src/app
|
||||
@@ -32,17 +31,11 @@ RUN cp /tmp/stemengine-router.js /src/app/src/stemengine-router.js
|
||||
RUN cp /tmp/stemengine-audio-engine.js /src/app/src/stemengine-audio-engine.js
|
||||
RUN cp /tmp/stemengine-feature-extractor.js /src/app/src/stemengine-feature-extractor.js
|
||||
|
||||
# Inject fluid render graph (Phase 2: Fluid rendering)
|
||||
RUN cp /tmp/fluid-render-graph.js /src/app/src/fluid-render-graph.js
|
||||
|
||||
# Inject tendrils render graph (Phase 3)
|
||||
RUN cp /tmp/tendrils-render-graph.js /src/app/src/tendrils-render-graph.js
|
||||
|
||||
# Inject spirit render graph (Phase 4)
|
||||
RUN cp /tmp/spirit-render-graph.js /src/app/src/spirit-render-graph.js
|
||||
|
||||
# Inject imports in deterministic order (bootstrap, audio pipeline, router, then render graphs).
|
||||
RUN sed -i '1i import "./stemengine-bootstrap";\nimport "./stemengine-audio-engine";\nimport "./stemengine-feature-extractor";\nimport "./stemengine-router";\nimport "./fluid-render-graph";\nimport "./tendrils-render-graph";\nimport "./spirit-render-graph";' /src/app/src/index.js
|
||||
RUN sed -i '1i import "./stemengine-bootstrap";\nimport "./stemengine-audio-engine";\nimport "./stemengine-feature-extractor";\nimport "./stemengine-router";\nimport "./spirit-render-graph";' /src/app/src/index.js
|
||||
|
||||
# Default-mode integration: allow original Muon analyser reads to use StemEngine analyser.
|
||||
RUN sed -i 's@analyser = wavesurfer.backend.analyser@analyser = (window.__STEMENGINE_MUON_ANALYSER__ || wavesurfer.backend.analyser)@' /src/app/src/index.js
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -74,6 +74,25 @@
|
||||
return { duration: buffer.duration }
|
||||
}
|
||||
|
||||
function normalizeAudioBytes(bytes) {
|
||||
if (bytes instanceof ArrayBuffer) return bytes
|
||||
if (ArrayBuffer.isView(bytes)) {
|
||||
return bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength)
|
||||
}
|
||||
throw new Error('unsupported audio byte payload')
|
||||
}
|
||||
|
||||
async function loadFromBytes(bytes) {
|
||||
initialize()
|
||||
|
||||
stop()
|
||||
|
||||
var arr = normalizeAudioBytes(bytes)
|
||||
buffer = await ctx.decodeAudioData(arr)
|
||||
pausedAt = 0
|
||||
return { duration: buffer.duration }
|
||||
}
|
||||
|
||||
function play(opts) {
|
||||
initialize()
|
||||
if (!buffer || !gain) return
|
||||
@@ -154,6 +173,7 @@
|
||||
window.__StemEngineAudioEngine__ = {
|
||||
initialize: initialize,
|
||||
loadFromUrl: loadFromUrl,
|
||||
loadFromBytes: loadFromBytes,
|
||||
play: play,
|
||||
pause: pause,
|
||||
stop: stop,
|
||||
|
||||
@@ -43,12 +43,24 @@
|
||||
setViewport: true,
|
||||
pushAudioFeatures: true,
|
||||
loadAudio: true,
|
||||
loadAudioData: true,
|
||||
transportPlay: true,
|
||||
transportPause: true,
|
||||
transportSeek: true,
|
||||
transportRate: true,
|
||||
}
|
||||
|
||||
function emitLoaded(info) {
|
||||
postToParent({
|
||||
version: 1,
|
||||
id: 'loaded-' + Date.now(),
|
||||
type: 'loaded',
|
||||
payload: {
|
||||
duration: info && Number.isFinite(Number(info.duration)) ? Number(info.duration) : 0,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// Audio feature state — host-push model.
|
||||
// The host sends pushAudioFeatures at its RAF cadence; we rate-limit to
|
||||
@@ -213,14 +225,20 @@
|
||||
if (window.__StemEngineFeatureExtractor__ && typeof window.__StemEngineFeatureExtractor__.start === 'function') {
|
||||
window.__StemEngineFeatureExtractor__.start()
|
||||
}
|
||||
postToParent({
|
||||
version: 1,
|
||||
id: 'loaded-' + Date.now(),
|
||||
type: 'loaded',
|
||||
payload: {
|
||||
duration: info && Number.isFinite(Number(info.duration)) ? Number(info.duration) : 0,
|
||||
},
|
||||
})
|
||||
emitLoaded(info)
|
||||
})
|
||||
.catch(function (err) {
|
||||
window.__stemengineEmitError__('AUDIO_LOAD', String(err && err.message ? err.message : err))
|
||||
})
|
||||
break
|
||||
case 'loadAudioData':
|
||||
if (!window.__StemEngineAudioEngine__ || typeof window.__StemEngineAudioEngine__.loadFromBytes !== 'function') break
|
||||
window.__StemEngineAudioEngine__.loadFromBytes(p.bytes)
|
||||
.then(function (info) {
|
||||
if (window.__StemEngineFeatureExtractor__ && typeof window.__StemEngineFeatureExtractor__.start === 'function') {
|
||||
window.__StemEngineFeatureExtractor__.start()
|
||||
}
|
||||
emitLoaded(info)
|
||||
})
|
||||
.catch(function (err) {
|
||||
window.__stemengineEmitError__('AUDIO_LOAD', String(err && err.message ? err.message : err))
|
||||
|
||||
@@ -1,265 +0,0 @@
|
||||
/* StemEngine Tendrils Render Graph (Muon fork overlay)
|
||||
* Particle trail / feedback visualization with audio reactivity.
|
||||
*/
|
||||
|
||||
;(function stemengineTendrilsRenderGraph() {
|
||||
'use strict'
|
||||
|
||||
var QUALITY_SCALE = { low: 0.75, balanced: 1.0, high: 1.35 }
|
||||
|
||||
function clamp01(v) {
|
||||
var n = Number(v)
|
||||
if (!isFinite(n)) n = 0
|
||||
return Math.max(0, Math.min(1, n))
|
||||
}
|
||||
|
||||
function shaped(v, power) {
|
||||
return Math.pow(clamp01(v), power)
|
||||
}
|
||||
|
||||
function getBand(audio, key, fallback) {
|
||||
var bands = audio && audio.frequencyBands ? audio.frequencyBands : null
|
||||
var value = bands && isFinite(Number(bands[key])) ? Number(bands[key]) : fallback
|
||||
return clamp01(value)
|
||||
}
|
||||
|
||||
function createProgram(gl, vsSrc, fsSrc) {
|
||||
function compile(type, src) {
|
||||
var s = gl.createShader(type)
|
||||
gl.shaderSource(s, src)
|
||||
gl.compileShader(s)
|
||||
if (!gl.getShaderParameter(s, gl.COMPILE_STATUS)) {
|
||||
var err = gl.getShaderInfoLog(s) || 'shader compile failed'
|
||||
window.__stemengineEmitError__('TENDRILS_SHADER', err)
|
||||
gl.deleteShader(s)
|
||||
return null
|
||||
}
|
||||
return s
|
||||
}
|
||||
var vs = compile(gl.VERTEX_SHADER, vsSrc)
|
||||
var fs = compile(gl.FRAGMENT_SHADER, fsSrc)
|
||||
if (!vs || !fs) return null
|
||||
var p = gl.createProgram()
|
||||
gl.attachShader(p, vs)
|
||||
gl.attachShader(p, fs)
|
||||
gl.linkProgram(p)
|
||||
gl.deleteShader(vs)
|
||||
gl.deleteShader(fs)
|
||||
if (!gl.getProgramParameter(p, gl.LINK_STATUS)) {
|
||||
var err2 = gl.getProgramInfoLog(p) || 'program link failed'
|
||||
window.__stemengineEmitError__('TENDRILS_PROGRAM', err2)
|
||||
gl.deleteProgram(p)
|
||||
return null
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
function createTarget(gl, w, h) {
|
||||
var tex = gl.createTexture()
|
||||
gl.bindTexture(gl.TEXTURE_2D, tex)
|
||||
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, w, h, 0, gl.RGBA, gl.UNSIGNED_BYTE, null)
|
||||
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR)
|
||||
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR)
|
||||
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE)
|
||||
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE)
|
||||
|
||||
var fbo = gl.createFramebuffer()
|
||||
gl.bindFramebuffer(gl.FRAMEBUFFER, fbo)
|
||||
gl.framebufferTexture2D(gl.FRAMEBUFFER, gl.COLOR_ATTACHMENT0, gl.TEXTURE_2D, tex, 0)
|
||||
gl.bindFramebuffer(gl.FRAMEBUFFER, null)
|
||||
return { tex: tex, fbo: fbo, width: w, height: h }
|
||||
}
|
||||
|
||||
function TendrilsSimulator(canvas, gl, width, height, quality) {
|
||||
this.canvas = canvas || document.querySelector('canvas')
|
||||
this.gl = gl || (this.canvas && (this.canvas.getContext('webgl2', { alpha: false, antialias: false }) || this.canvas.getContext('webgl', { alpha: false, antialias: false })))
|
||||
if (!this.gl) throw new Error('[StemEngine][Tendrils] Unable to acquire WebGL context')
|
||||
|
||||
this.width = Math.max(1, Number(width) || this.canvas.clientWidth || this.canvas.width || 512)
|
||||
this.height = Math.max(1, Number(height) || this.canvas.clientHeight || this.canvas.height || 512)
|
||||
this.quality = String(quality || window.__STEMENGINE_MUON_QUALITY__ || 'balanced')
|
||||
this.paused = false
|
||||
this.rafId = null
|
||||
this.running = false
|
||||
|
||||
var gl = this.gl
|
||||
var vs = 'attribute vec2 aPos; varying vec2 vUv; void main(){ vUv=aPos*0.5+0.5; gl_Position=vec4(aPos,0.0,1.0);} '
|
||||
var updateFs = [
|
||||
'precision highp float;',
|
||||
'uniform sampler2D uPrev;',
|
||||
'uniform vec2 uRes;',
|
||||
'uniform float uTime;',
|
||||
'uniform float uBass;',
|
||||
'uniform float uTreble;',
|
||||
'uniform float uRms;',
|
||||
'uniform float uBeat;',
|
||||
'uniform float uTrailDecay;',
|
||||
'varying vec2 vUv;',
|
||||
'float hash(vec2 p){ return fract(sin(dot(p, vec2(127.1,311.7))) * 43758.5453); }',
|
||||
'void main(){',
|
||||
' vec2 uv=vUv;',
|
||||
' vec2 c=uv-0.5;',
|
||||
' float r=max(length(c), 1e-4);',
|
||||
' float a=atan(c.y,c.x);',
|
||||
' float flow=sin(a*8.0 + uTime*(1.6+uBass*4.0)) + cos(r*45.0 + uTime*(1.1+uTreble*5.0));',
|
||||
' vec2 dir=normalize(vec2(cos(a+flow*0.45), sin(a+flow*0.45)));',
|
||||
' vec2 offset=dir*(0.0015 + 0.0035*uBass) + vec2(hash(uv+uTime)-0.5, hash(uv+uTime*1.7)-0.5)*(0.002*uTreble);',
|
||||
' vec3 prev=texture2D(uPrev, clamp(uv-offset, 0.0, 1.0)).rgb * uTrailDecay;',
|
||||
' float line=exp(-abs(sin((a*12.0 + r*120.0) + uTime*6.0*(1.0+uTreble)))*26.0);',
|
||||
' float beatBurst=uBeat * exp(-r*4.0);',
|
||||
' vec3 colorA=vec3(0.2+uBass*0.8, 0.2+uRms*0.7, 0.4+uTreble*0.6);',
|
||||
' vec3 colorB=vec3(0.9,0.6+uTreble*0.35,0.2+uBass*0.4);',
|
||||
' vec3 inject=mix(colorA, colorB, clamp(r*1.5, 0.0, 1.0))*(line*0.3 + beatBurst*0.8 + uRms*0.05);',
|
||||
' gl_FragColor=vec4(clamp(prev + inject, 0.0, 1.0), 1.0);',
|
||||
'}'
|
||||
].join('\n')
|
||||
var blitFs = 'precision mediump float; uniform sampler2D uTex; varying vec2 vUv; void main(){ gl_FragColor = texture2D(uTex, vUv); }'
|
||||
|
||||
this.updateProgram = createProgram(gl, vs, updateFs)
|
||||
this.blitProgram = createProgram(gl, vs, blitFs)
|
||||
|
||||
var quad = new Float32Array([-1,-1, 1,-1, -1,1, -1,1, 1,-1, 1,1])
|
||||
this.quad = gl.createBuffer()
|
||||
gl.bindBuffer(gl.ARRAY_BUFFER, this.quad)
|
||||
gl.bufferData(gl.ARRAY_BUFFER, quad, gl.STATIC_DRAW)
|
||||
|
||||
this.read = null
|
||||
this.write = null
|
||||
this.allocateTargets()
|
||||
this.start()
|
||||
}
|
||||
|
||||
TendrilsSimulator.prototype.allocateTargets = function () {
|
||||
var gl = this.gl
|
||||
if (this.read) {
|
||||
gl.deleteTexture(this.read.tex)
|
||||
gl.deleteFramebuffer(this.read.fbo)
|
||||
}
|
||||
if (this.write) {
|
||||
gl.deleteTexture(this.write.tex)
|
||||
gl.deleteFramebuffer(this.write.fbo)
|
||||
}
|
||||
var scale = QUALITY_SCALE[this.quality] || QUALITY_SCALE.balanced
|
||||
var w = Math.max(64, Math.floor(this.width * scale))
|
||||
var h = Math.max(64, Math.floor(this.height * scale))
|
||||
this.read = createTarget(gl, w, h)
|
||||
this.write = createTarget(gl, w, h)
|
||||
}
|
||||
|
||||
TendrilsSimulator.prototype.setQuality = function (q) {
|
||||
var next = String(q || 'balanced')
|
||||
if (next === this.quality) return
|
||||
this.quality = next
|
||||
this.allocateTargets()
|
||||
}
|
||||
|
||||
TendrilsSimulator.prototype.setViewport = function (w, h) {
|
||||
var nw = Math.max(1, Number(w) || this.width)
|
||||
var nh = Math.max(1, Number(h) || this.height)
|
||||
if (nw === this.width && nh === this.height) return
|
||||
this.width = nw
|
||||
this.height = nh
|
||||
if (this.canvas) {
|
||||
this.canvas.width = nw
|
||||
this.canvas.height = nh
|
||||
}
|
||||
this.allocateTargets()
|
||||
}
|
||||
|
||||
TendrilsSimulator.prototype.drawQuad = function (program) {
|
||||
var gl = this.gl
|
||||
gl.useProgram(program)
|
||||
gl.bindBuffer(gl.ARRAY_BUFFER, this.quad)
|
||||
var loc = gl.getAttribLocation(program, 'aPos')
|
||||
gl.enableVertexAttribArray(loc)
|
||||
gl.vertexAttribPointer(loc, 2, gl.FLOAT, false, 0, 0)
|
||||
gl.drawArrays(gl.TRIANGLES, 0, 6)
|
||||
}
|
||||
|
||||
TendrilsSimulator.prototype.step = function () {
|
||||
if (this.paused || !this.read || !this.write) return
|
||||
var gl = this.gl
|
||||
var t = performance.now() * 0.001
|
||||
var audio = window.__STEMENGINE_MUON_AUDIO__ || {}
|
||||
var bass = shaped(0.6 * getBand(audio, 'bass', audio.bass || 0) + 0.4 * getBand(audio, 'sub', audio.sub || 0), 0.62)
|
||||
var treble = shaped(0.55 * getBand(audio, 'presence', audio.treble || 0) + 0.45 * getBand(audio, 'brilliance', audio.treble || 0), 0.66)
|
||||
var rms = shaped(clamp01(audio.rms || 0), 0.8)
|
||||
var beat = audio.beat ? 1 : 0
|
||||
var buildBoost = audio.isBuildUp ? 0.12 : 0
|
||||
var dropBoost = audio.isBeatDrop ? 0.2 : 0
|
||||
|
||||
gl.bindFramebuffer(gl.FRAMEBUFFER, this.write.fbo)
|
||||
gl.viewport(0, 0, this.write.width, this.write.height)
|
||||
gl.useProgram(this.updateProgram)
|
||||
gl.activeTexture(gl.TEXTURE0)
|
||||
gl.bindTexture(gl.TEXTURE_2D, this.read.tex)
|
||||
gl.uniform1i(gl.getUniformLocation(this.updateProgram, 'uPrev'), 0)
|
||||
gl.uniform2f(gl.getUniformLocation(this.updateProgram, 'uRes'), this.write.width, this.write.height)
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uTime'), t)
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uBass'), bass)
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uTreble'), treble)
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uRms'), rms)
|
||||
gl.uniform1f(gl.getUniformLocation(this.updateProgram, 'uBeat'), beat)
|
||||
gl.uniform1f(
|
||||
gl.getUniformLocation(this.updateProgram, 'uTrailDecay'),
|
||||
(0.90 + (this.quality === 'high' ? 0.08 : this.quality === 'low' ? 0.04 : 0.06)) - buildBoost + dropBoost,
|
||||
)
|
||||
this.drawQuad(this.updateProgram)
|
||||
|
||||
gl.bindFramebuffer(gl.FRAMEBUFFER, null)
|
||||
gl.viewport(0, 0, this.width, this.height)
|
||||
gl.useProgram(this.blitProgram)
|
||||
gl.activeTexture(gl.TEXTURE0)
|
||||
gl.bindTexture(gl.TEXTURE_2D, this.write.tex)
|
||||
gl.uniform1i(gl.getUniformLocation(this.blitProgram, 'uTex'), 0)
|
||||
this.drawQuad(this.blitProgram)
|
||||
|
||||
var tmp = this.read
|
||||
this.read = this.write
|
||||
this.write = tmp
|
||||
}
|
||||
|
||||
TendrilsSimulator.prototype.tick = function () {
|
||||
if (!this.running) return
|
||||
this.paused = Boolean(window.__STEMENGINE_MUON_PAUSED__)
|
||||
this.setQuality(window.__STEMENGINE_MUON_QUALITY__)
|
||||
var vp = window.__STEMENGINE_MUON_VIEWPORT__
|
||||
if (vp) this.setViewport(vp.width, vp.height)
|
||||
this.step()
|
||||
var self = this
|
||||
this.rafId = requestAnimationFrame(function () { self.tick() })
|
||||
}
|
||||
|
||||
TendrilsSimulator.prototype.start = function () {
|
||||
if (this.running) return
|
||||
this.running = true
|
||||
this.tick()
|
||||
}
|
||||
|
||||
TendrilsSimulator.prototype.stop = function () {
|
||||
this.running = false
|
||||
if (this.rafId != null) {
|
||||
cancelAnimationFrame(this.rafId)
|
||||
this.rafId = null
|
||||
}
|
||||
}
|
||||
|
||||
TendrilsSimulator.prototype.dispose = function () {
|
||||
var gl = this.gl
|
||||
this.stop()
|
||||
if (this.quad) gl.deleteBuffer(this.quad)
|
||||
if (this.updateProgram) gl.deleteProgram(this.updateProgram)
|
||||
if (this.blitProgram) gl.deleteProgram(this.blitProgram)
|
||||
if (this.read) {
|
||||
gl.deleteTexture(this.read.tex)
|
||||
gl.deleteFramebuffer(this.read.fbo)
|
||||
}
|
||||
if (this.write) {
|
||||
gl.deleteTexture(this.write.tex)
|
||||
gl.deleteFramebuffer(this.write.fbo)
|
||||
}
|
||||
}
|
||||
|
||||
window.__StemEngineTendrilsRenderGraph__ = TendrilsSimulator
|
||||
console.log('[StemEngine][Tendrils] Render graph module loaded')
|
||||
})()
|
||||
@@ -0,0 +1,22 @@
|
||||
# syntax=docker/dockerfile:1.4
|
||||
FROM alpine/git:2.47.2 AS source
|
||||
|
||||
ARG FLUID_GIT_REF=3db8fa9
|
||||
WORKDIR /src
|
||||
|
||||
RUN git clone https://github.com/PavelDoGreat/Fluid-Simulation.git app \
|
||||
&& cd app \
|
||||
&& git checkout "${FLUID_GIT_REF}"
|
||||
|
||||
FROM nginx:1.27-alpine
|
||||
|
||||
COPY --from=source /src/app/projects/web/ /usr/share/nginx/html/
|
||||
COPY index.html /usr/share/nginx/html/index.html
|
||||
COPY stemengine-fluid-adapter.js /usr/share/nginx/html/stemengine-fluid-adapter.js
|
||||
COPY --from=shared stemengine-bootstrap.js /usr/share/nginx/html/stemengine-bootstrap.js
|
||||
COPY --from=shared stemengine-router.js /usr/share/nginx/html/stemengine-router.js
|
||||
COPY --from=shared stemengine-audio-engine.js /usr/share/nginx/html/stemengine-audio-engine.js
|
||||
COPY --from=shared stemengine-feature-extractor.js /usr/share/nginx/html/stemengine-feature-extractor.js
|
||||
COPY nginx.conf /etc/nginx/conf.d/default.conf
|
||||
|
||||
EXPOSE 80
|
||||
@@ -0,0 +1,476 @@
|
||||
# Fluid-Simulation Audio & Palette Injection Design (Phase 4 Pre-Work)
|
||||
|
||||
**Upstream Reference:** https://github.com/PavelDoGreat/Fluid-Simulation
|
||||
**Pinned Commit:** `3db8fa9` (main, "still metal")
|
||||
**Date:** 2026-04-28
|
||||
**Status:** Pre-work analysis complete; implementation ready for Phase 4
|
||||
|
||||
---
|
||||
|
||||
## 1. Upstream Inspection Summary
|
||||
|
||||
### 1.1 Project Structure
|
||||
|
||||
**Tech Stack:**
|
||||
- **Language:** Zig (modern systems language)
|
||||
- **Build System:** Zig build system (`build.zig`)
|
||||
- **Graphics API (Web):** WebGL2
|
||||
- **Compilation Target:** `wasm32-freestanding`
|
||||
- **Size:** 226 total commits, lean codebase (~800 KiB)
|
||||
|
||||
**Directory Organization:**
|
||||
```
|
||||
fluid-simulation/
|
||||
├── projects/web/ # Web entry point
|
||||
│ ├── index.html # Canvas + script.js loader
|
||||
│ ├── script.js # WASM initialization & animation loop (122 lines)
|
||||
│ └── bin/fluid.wasm # Compiled WASM module
|
||||
├── src/
|
||||
│ ├── engine/ # Platform-agnostic engine (graphics, math, memory, debug)
|
||||
│ ├── fluid/ # Fluid-specific implementation
|
||||
│ │ ├── main.zig # Entry point (exports start, update, draw)
|
||||
│ │ ├── materials.zig # Rendering materials (currently stubs)
|
||||
│ │ └── shaders/ # Shader definitions
|
||||
│ └── your_project_goes_here/ # Template for custom projects
|
||||
└── build.zig # Build configuration (46 KB)
|
||||
```
|
||||
|
||||
### 1.2 Web Implementation Status
|
||||
|
||||
**Current State (Early Development):**
|
||||
- `projects/web/script.js` loads WASM and initiates animation loop
|
||||
- Exported functions from Zig (`start()`, `update()`, `draw()`) exist but are **stubs** (no-op)
|
||||
- Graphics initialization happens in `start()` but actual rendering logic is minimal
|
||||
- No audio handling, no palette customization, no parameter controls
|
||||
|
||||
**Key Code Path:**
|
||||
1. HTML loads `script.js`
|
||||
2. `script.js` calls `WebAssembly.instantiateStreaming()` to load `/bin/fluid.wasm`
|
||||
3. On successful load, calls `exports.start()` to initialize
|
||||
4. Animation loop: `requestAnimationFrame(update)` → `exports.update()` → `exports.draw()` → `gl.clear()` / `gl.viewport()`
|
||||
5. Loop repeats 60 FPS (or display refresh rate)
|
||||
|
||||
---
|
||||
|
||||
## 2. Audio Injection Architecture
|
||||
|
||||
### 2.1 Injection Point Design
|
||||
|
||||
**Proposed Mechanism:** Extend WASM exports to receive audio frequency data
|
||||
|
||||
**New Exported Functions:**
|
||||
```c
|
||||
// Load audio frequency data into WASM memory
|
||||
// Accepts ArrayBuffer or Float32Array view into WASM memory
|
||||
export fn setAudioFeatures(ptr: *const f32, len: u32, bands: u32) void
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `ptr` — Pointer to frequency data in WASM linear memory (caller responsibility to copy data into shared buffer)
|
||||
- `len` — Total number of samples (e.g., 256 for FFT output)
|
||||
- `bands` — Number of frequency bands to extract (e.g., 4–8 for bass/mid/treble/high)
|
||||
|
||||
**Alternative (Simpler) Approach:**
|
||||
```c
|
||||
// Direct frequency band values (pre-extracted by host)
|
||||
export fn setAudioBands(band_ptr: *const f32, band_count: u32) void
|
||||
```
|
||||
- `band_ptr` — Pointer to 1–8 pre-calculated band magnitudes (0–1 range)
|
||||
- `band_count` — Number of bands provided (1–8)
|
||||
|
||||
### 2.2 Host-Side Integration
|
||||
|
||||
**JavaScript Caller Responsibility:**
|
||||
1. Allocate WASM linear memory buffer for audio data (or use pre-allocated region)
|
||||
2. Copy audio frequency data into buffer
|
||||
3. Call `exports.setAudioBands(band_ptr, band_count)` with derived bands
|
||||
|
||||
**Recommended Integration in `stemengine-audio-engine.js`:**
|
||||
- Add `getFrequencyBands(count: number)` method to extract frequency bands from active audio buffer
|
||||
- Return band magnitudes as Float32Array (0–1 normalized range)
|
||||
- Call Fluid's `setAudioBands()` in animation loop before `exports.update()`
|
||||
|
||||
### 2.3 WASM-Side Audio Processing
|
||||
|
||||
**Memory Layout (Reserved Region):**
|
||||
- Allocate 32 bytes in WASM linear memory for audio band storage (up to 8 × f32)
|
||||
- Export memory offset via constant or function: `export fn getAudioBandsPtr() usize`
|
||||
- Zig reads bands during `update()` to drive fluid simulation
|
||||
|
||||
**Fluid Dynamics Integration:**
|
||||
- Each frequency band drives a different aspect:
|
||||
- **Band 0 (Bass, 0–100 Hz):** Large-scale fluid motion (viscosity control, pressure)
|
||||
- **Band 1 (Mid-Bass, 100–250 Hz):** Medium-scale vortex formation
|
||||
- **Band 2 (Treble, 250–2 kHz):** Small-scale disturbances, color intensity
|
||||
- **Band 3–7 (Optional):** Additional fine-tuning parameters
|
||||
|
||||
**Implementation Targets in Zig:**
|
||||
1. Create module `src/fluid/audio.zig` with:
|
||||
- Global audio band buffer
|
||||
- `setAudioBands()` export
|
||||
- `getAudioBandsPtr()` export
|
||||
2. Modify `src/fluid/main.zig` `update()` to:
|
||||
- Read current audio bands
|
||||
- Apply band magnitudes to fluid simulation parameters (velocity, pressure, color)
|
||||
- Scale/dampen by user-configured sensitivity parameter
|
||||
|
||||
---
|
||||
|
||||
## 3. Palette Injection Architecture
|
||||
|
||||
### 3.1 Injection Point Design
|
||||
|
||||
**Proposed Mechanism:** Pass RGB color palette as typed array
|
||||
|
||||
**New Exported Function:**
|
||||
```c
|
||||
// Load custom color palette into WASM memory
|
||||
// palette_ptr = pointer to palette data (consecutive RGB triplets or RGBA quads)
|
||||
// color_count = number of colors in palette
|
||||
export fn setPalette(palette_ptr: *const u8, color_count: u32, format: u32) void
|
||||
```
|
||||
|
||||
**Format Enum:**
|
||||
- `0` = RGB (3 bytes per color)
|
||||
- `1` = RGBA (4 bytes per color)
|
||||
- `2` = ARGB (4 bytes per color, used by some systems)
|
||||
|
||||
### 3.2 Palette Storage in WASM
|
||||
|
||||
**Memory Layout:**
|
||||
- Reserve 256 bytes (64 colors × 4 bytes RGBA) in linear memory
|
||||
- Export offset: `export fn getPalettePtr() usize`
|
||||
- Initialize with default palette on `start()`
|
||||
|
||||
**Palette Application Strategy:**
|
||||
1. Store palette as contiguous RGBA values
|
||||
2. During rendering, map fluid field values (0–1 range) to palette indices
|
||||
3. Fragment shader indexes palette based on local fluid velocity/pressure/temperature
|
||||
|
||||
### 3.3 Host-Side Integration
|
||||
|
||||
**JavaScript Caller Responsibility:**
|
||||
1. Convert host palette (array of hex strings or `{ r, g, b }` objects) to RGBA byte array
|
||||
2. Allocate/use WASM memory region for palette storage
|
||||
3. Call `exports.setPalette(palette_ptr, color_count, 1)` to upload custom palette
|
||||
4. Call from RendererControlsPanel when palette changes
|
||||
|
||||
**Integration in Component Flow:**
|
||||
- `RendererControlsPanel.tsx` → palette selector → `onPaletteChange()`
|
||||
- Hook: `useVisualizerIframeTransport()` → send `setPalette` message
|
||||
- Receiver: `stemengine-router.js` → `setPalette` handler → call Zig export
|
||||
|
||||
---
|
||||
|
||||
## 4. Tunable Parameters & Controls
|
||||
|
||||
### 4.1 Identified Parameter Set
|
||||
|
||||
Based on upstream fluid simulation design patterns, these parameters are suitable for tuning:
|
||||
|
||||
| Parameter | Type | Range | Default | Purpose |
|
||||
|-----------|------|-------|---------|---------|
|
||||
| `viscosity` | float | 0.0–1.0 | 0.5 | Fluid damping/persistence (higher = smoother, slower decay) |
|
||||
| `pressure` | float | 0.0–2.0 | 1.0 | Force magnitude of disturbances (higher = more energetic) |
|
||||
| `decay` | float | 0.0–1.0 | 0.9 | Per-frame velocity decay (higher = longer trails) |
|
||||
| `audio_sensitivity` | float | 0.0–2.0 | 1.0 | Scale factor for audio-driven disturbances |
|
||||
| `frequency_bias` | enum | {bass, mid, treble, full} | "full" | Which frequency bands drive simulation |
|
||||
| `color_intensity` | float | 0.0–2.0 | 1.0 | Brightness/saturation multiplier for palette |
|
||||
| `color_source` | enum | {velocity, pressure, temperature} | "velocity" | Which fluid property drives color lookup |
|
||||
|
||||
### 4.2 Exported Parameter Functions
|
||||
|
||||
**Zig Exports:**
|
||||
```c
|
||||
export fn setParameter(param_id: u32, value: f32) void
|
||||
export fn getParameter(param_id: u32) f32
|
||||
export fn getParameterMetadata(param_id: u32) *const ParameterMetadata
|
||||
```
|
||||
|
||||
**Parameter Metadata Struct (Zig):**
|
||||
```c
|
||||
pub const ParameterMetadata = struct {
|
||||
name: [32]u8,
|
||||
param_id: u32,
|
||||
min: f32,
|
||||
max: f32,
|
||||
default: f32,
|
||||
param_type: ParameterType, // 0=float, 1=int, 2=enum
|
||||
};
|
||||
```
|
||||
|
||||
**Implementation Plan:**
|
||||
1. Create `src/fluid/parameters.zig` with parameter registry
|
||||
2. Export functions to query and set parameters
|
||||
3. Expose via `capabilities` message in protocol (Phase 3 reserved field)
|
||||
4. Frontend RendererControlsPanel subscribes to capabilities and renders controls dynamically
|
||||
|
||||
---
|
||||
|
||||
## 5. Implementation Patches & Targets
|
||||
|
||||
### 5.1 Zig Patches Required
|
||||
|
||||
**File: `src/fluid/main.zig`**
|
||||
|
||||
**Patch 1: Audio Module Import**
|
||||
```zig
|
||||
const audio = @import("audio.zig");
|
||||
// Add to imports at top
|
||||
```
|
||||
|
||||
**Patch 2: Modify `update()` Function**
|
||||
- Read audio bands from shared buffer
|
||||
- Apply bands to fluid parameters
|
||||
- Example pseudo-code:
|
||||
```zig
|
||||
export fn update() void {
|
||||
const audio_bands = audio.getActiveBands();
|
||||
// Scale pressure by bass band
|
||||
const pressure = base_pressure * (1.0 + audio_bands[0] * audio_sensitivity);
|
||||
// Scale viscosity by treble band
|
||||
const viscosity = base_viscosity * (1.0 - audio_bands[2] * 0.3);
|
||||
// Apply parameters to fluid simulation
|
||||
simulation.step(pressure, viscosity, decay);
|
||||
}
|
||||
```
|
||||
|
||||
**File: `src/fluid/audio.zig` (New)**
|
||||
- Allocate 32 bytes for audio band storage
|
||||
- Export `setAudioBands(band_ptr, band_count)`
|
||||
- Export `getAudioBandsPtr()`
|
||||
- Provide getters for active bands
|
||||
|
||||
**File: `src/fluid/parameters.zig` (New)**
|
||||
- Parameter registry with metadata
|
||||
- Export `setParameter()`, `getParameter()`
|
||||
- Expose parameter list for dynamic UI
|
||||
|
||||
**File: `src/fluid/palette.zig` (New)**
|
||||
- Palette buffer (256 bytes, up to 64 colors)
|
||||
- Export `setPalette(palette_ptr, color_count, format)`
|
||||
- Export `getPalettePtr()`
|
||||
- Provide palette lookup function
|
||||
|
||||
### 5.2 JavaScript Patches Required
|
||||
|
||||
**File: `projects/web/script.js`**
|
||||
|
||||
**Patch 1: Audio Features Callback**
|
||||
```javascript
|
||||
// After WASM load, register callback to receive audio data
|
||||
function onAudioFeatures(frequencyData) {
|
||||
if (exports.setAudioBands) {
|
||||
const bands = extractBands(frequencyData, 4); // 4 frequency bands
|
||||
// Copy bands into WASM memory and call export
|
||||
const audioPtr = exports.getAudioBandsPtr();
|
||||
const audioBuffer = new Float32Array(wasmMemory.buffer, audioPtr, 4);
|
||||
audioBuffer.set(bands);
|
||||
exports.setAudioBands(audioPtr, 4);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Patch 2: Palette Upload**
|
||||
```javascript
|
||||
// On palette change (from host)
|
||||
function uploadPalette(paletteColors) {
|
||||
if (exports.setPalette) {
|
||||
const palette = new Uint8Array(paletteColors.length * 4);
|
||||
for (let i = 0; i < paletteColors.length; i++) {
|
||||
const [r, g, b, a] = paletteColors[i];
|
||||
palette[i * 4] = r;
|
||||
palette[i * 4 + 1] = g;
|
||||
palette[i * 4 + 2] = b;
|
||||
palette[i * 4 + 3] = a || 255;
|
||||
}
|
||||
const palettePtr = exports.getPalettePtr();
|
||||
const wasmPalette = new Uint8Array(wasmMemory.buffer, palettePtr, palette.length);
|
||||
wasmPalette.set(palette);
|
||||
exports.setPalette(palettePtr, paletteColors.length, 1); // 1 = RGBA format
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5.3 Shared Router Patches Required
|
||||
|
||||
**File: `visualizer-iframes/shared/stemengine-router.js`**
|
||||
|
||||
**New Handlers:**
|
||||
```javascript
|
||||
case 'setAudioBands': {
|
||||
// Receive audio band data from host
|
||||
// Store in buffer accessible to update loop
|
||||
hostAudioBands = payload.bands; // [band0, band1, ...]
|
||||
break;
|
||||
}
|
||||
|
||||
case 'setPalette': {
|
||||
// Receive palette from host
|
||||
// Call Zig export to upload palette
|
||||
window.fluidPalette = payload.palette; // Array of RGB/RGBA
|
||||
if (window.fluid && window.fluid.setPalette) {
|
||||
uploadPalette(payload.palette);
|
||||
}
|
||||
break;
|
||||
}
|
||||
|
||||
case 'setParameter': {
|
||||
// Receive parameter change from host
|
||||
if (window.fluid && window.fluid.setParameter) {
|
||||
window.fluid.setParameter(payload.param_id, payload.value);
|
||||
}
|
||||
break;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Integration Workflow (Phase 4 → Implementation)
|
||||
|
||||
### 6.1 Dockerfile Build Strategy
|
||||
|
||||
```dockerfile
|
||||
# Build stage: compile Zig source with patches
|
||||
FROM zig:0.13 AS builder
|
||||
WORKDIR /build
|
||||
COPY . .
|
||||
# Apply sed patches to inject audio/palette/parameter code
|
||||
RUN sed -i 's/export fn update().*/export fn update() void { ... audio integration ... }/' src/fluid/main.zig
|
||||
# Build WASM
|
||||
RUN zig build -Dtarget=wasm32-freestanding -p projects/web
|
||||
# Output: projects/web/bin/fluid.wasm
|
||||
|
||||
# Runtime stage: Muon + fluid.wasm
|
||||
FROM muon:latest
|
||||
COPY --from=builder /build/projects/web/bin/fluid.wasm /app/fluid.wasm
|
||||
COPY --from=shared stemengine-*.js /tmp/
|
||||
# Serve on port 3012
|
||||
```
|
||||
|
||||
### 6.2 Frontend Provider Registration
|
||||
|
||||
**File: `frontend/src/visualizers/providers.ts`**
|
||||
```typescript
|
||||
export const providers = {
|
||||
fluid: {
|
||||
id: 'fluid',
|
||||
name: 'Fluid Simulation',
|
||||
url: settings?.fluid_iframe_url || 'http://localhost:3012',
|
||||
// ... provider metadata
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
### 6.3 Transport Hook Integration
|
||||
|
||||
**File: `useVisualizerIframeTransport.ts` (Phase 3)**
|
||||
- Already has reserved `setAudioBands` and `setPalette` handlers
|
||||
- Update handlers to pass through to iframe when provider is 'fluid'
|
||||
- Forward `capabilities` from Fluid (parameter list) to RendererControlsPanel
|
||||
|
||||
### 6.4 RendererControlsPanel Integration
|
||||
|
||||
**File: `RendererControlsPanel.tsx` (Phase 3)**
|
||||
- Receives `capabilities.params` from iframe
|
||||
- For each param: render slider/control based on type
|
||||
- On change: send `setParameter()` message to iframe
|
||||
|
||||
---
|
||||
|
||||
## 7. Capabilities Announcement Protocol
|
||||
|
||||
### 7.1 Fluid Capabilities Message
|
||||
|
||||
**Iframe → Host (after initialization)**
|
||||
```json
|
||||
{
|
||||
"type": "capabilities",
|
||||
"payload": {
|
||||
"name": "Fluid Simulation",
|
||||
"version": "1.0.0",
|
||||
"params": [
|
||||
{
|
||||
"id": 0,
|
||||
"name": "viscosity",
|
||||
"min": 0.0,
|
||||
"max": 1.0,
|
||||
"default": 0.5,
|
||||
"type": "float"
|
||||
},
|
||||
{
|
||||
"id": 1,
|
||||
"name": "pressure",
|
||||
"min": 0.0,
|
||||
"max": 2.0,
|
||||
"default": 1.0,
|
||||
"type": "float"
|
||||
}
|
||||
// ... additional params
|
||||
],
|
||||
"supportsAudio": true,
|
||||
"supportsPalette": true,
|
||||
"audioBands": 4
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 7.2 Frontend Handling
|
||||
|
||||
- Hook receives `capabilities` from iframe
|
||||
- Exposes in hook result: `{ capabilities, ... }`
|
||||
- RendererControlsPanel subscribed to `capabilities.params`
|
||||
- Dynamic UI renders controls for each parameter
|
||||
|
||||
---
|
||||
|
||||
## 8. Testing & Validation Checklist
|
||||
|
||||
- [ ] Zig code compiles to WASM (zero warnings)
|
||||
- [ ] WASM module loads in browser without errors
|
||||
- [ ] Audio bands callback fires at 60 FPS
|
||||
- [ ] Palette upload doesn't corrupt rendering
|
||||
- [ ] Parameter changes don't crash simulator
|
||||
- [ ] Capabilities message format is valid JSON
|
||||
- [ ] Frontend controls respond to parameter changes
|
||||
- [ ] Docker build completes without errors
|
||||
- [ ] Integration test: play audio → see fluid respond
|
||||
- [ ] Integration test: change palette → see color update
|
||||
|
||||
---
|
||||
|
||||
## 9. Future Extensions (Phase 5+)
|
||||
|
||||
- Multi-pass rendering (separate velocity, pressure, temperature fields)
|
||||
- Custom shader support (allow palette-driven fragment shader)
|
||||
- Real-time parameter ramping (smooth transitions)
|
||||
- Physics tuning (gravity, diffusion, convection rates)
|
||||
- Touch/mouse input for manual disturbances (not audio-driven)
|
||||
- Recording/replay of simulations
|
||||
- Multi-resolution rendering (low-res preview, high-res main)
|
||||
|
||||
---
|
||||
|
||||
## Commit Hash Reference
|
||||
|
||||
```
|
||||
Repository: https://github.com/PavelDoGreat/Fluid-Simulation
|
||||
Commit: 3db8fa9
|
||||
Author: PavelDoGreat
|
||||
Message: "still metal"
|
||||
Date: (as of 2026-04-28 inspection)
|
||||
```
|
||||
|
||||
To reproduce the state:
|
||||
```bash
|
||||
cd /tmp/fluid-simulation
|
||||
git checkout 3db8fa9
|
||||
zig build -Dtarget=wasm32-freestanding -p projects/web
|
||||
# Output: projects/web/bin/fluid.wasm (ready for Phase 4 patching)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Documentation Completed:** 2026-04-28
|
||||
**Next Step:** Phase 4 Implementation (apply patches, build Dockerfile, integrate with frontend)
|
||||
**Estimated Phase 4 Duration:** 3-4 hours (Zig implementation + Docker build + integration tests)
|
||||
@@ -0,0 +1,63 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta http-equiv="Cache-Control" content="no-cache">
|
||||
<title>StemEngine Fluid Visualizer</title>
|
||||
<style>
|
||||
* {
|
||||
user-select: none;
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
html,
|
||||
body {
|
||||
overflow: hidden;
|
||||
margin: 0;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
background: radial-gradient(circle at 50% 50%, #0f172a 0%, #020617 100%);
|
||||
transition: background 160ms linear;
|
||||
}
|
||||
|
||||
canvas {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
display: block;
|
||||
transition: filter 120ms linear;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
|
||||
<body>
|
||||
<canvas></canvas>
|
||||
<script>
|
||||
window.__STEMENGINE_VIZ_PROVIDER_ID__ = 'fluid';
|
||||
window.__STEMENGINE_VIZ_STYLES__ = ['default'];
|
||||
window.__STEMENGINE_VIZ_CAPABILITIES__ = {
|
||||
providerId: 'fluid',
|
||||
protocolVersion: 1,
|
||||
presets: ['fluid_vortex', 'fluid_plasma', 'fluid_glass'],
|
||||
params: [
|
||||
{ name: 'audio_sensitivity', label: 'Audio Sensitivity', type: 'number', default: 1.0, min: 0, max: 2, step: 0.05 },
|
||||
{ name: 'hue_shift', label: 'Hue Shift', type: 'number', default: 0, min: -180, max: 180, step: 1 },
|
||||
{ name: 'brightness', label: 'Brightness', type: 'number', default: 1.0, min: 0.5, max: 1.8, step: 0.01 },
|
||||
{ name: 'contrast', label: 'Contrast', type: 'number', default: 1.0, min: 0.6, max: 1.8, step: 0.01 },
|
||||
{ name: 'pulse', label: 'Beat Pulse', type: 'number', default: 0.55, min: 0, max: 1.5, step: 0.01 }
|
||||
],
|
||||
palette: {
|
||||
supported: true,
|
||||
slotCount: 8
|
||||
}
|
||||
};
|
||||
</script>
|
||||
<script src="/stemengine-bootstrap.js"></script>
|
||||
<script src="/stemengine-audio-engine.js"></script>
|
||||
<script src="/stemengine-feature-extractor.js"></script>
|
||||
<script src="/stemengine-router.js"></script>
|
||||
<script src="/script.js?v=sefluid1"></script>
|
||||
<script src="/stemengine-fluid-adapter.js?v=sefluid1"></script>
|
||||
</body>
|
||||
|
||||
</html>
|
||||
@@ -0,0 +1,12 @@
|
||||
server {
|
||||
listen 80;
|
||||
server_name _;
|
||||
|
||||
root /usr/share/nginx/html;
|
||||
index index.html;
|
||||
|
||||
location / {
|
||||
try_files $uri $uri/ /index.html;
|
||||
add_header Cache-Control "no-store, no-cache, must-revalidate" always;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
;(function stemengineFluidAdapter() {
|
||||
'use strict'
|
||||
|
||||
var PRESET_MAP = {
|
||||
fluid_vortex: { hueShift: 18, pulse: 0.85, contrast: 1.08, brightness: 1.0 },
|
||||
fluid_plasma: { hueShift: 72, pulse: 1.05, contrast: 1.14, brightness: 1.08 },
|
||||
fluid_glass: { hueShift: -22, pulse: 0.45, contrast: 1.02, brightness: 0.96 },
|
||||
}
|
||||
|
||||
function clamp(value, min, max) {
|
||||
var n = Number(value)
|
||||
if (!isFinite(n)) n = 0
|
||||
return Math.max(min, Math.min(max, n))
|
||||
}
|
||||
|
||||
function readParam(name, fallback) {
|
||||
var params = window.__STEMENGINE_VIZ_PARAMS__
|
||||
if (!params || typeof params !== 'object') return fallback
|
||||
if (!Object.prototype.hasOwnProperty.call(params, name)) return fallback
|
||||
return Number(params[name])
|
||||
}
|
||||
|
||||
function parseHexColor(value) {
|
||||
var hex = String(value || '').trim().replace(/^#/, '')
|
||||
if (hex.length === 3) {
|
||||
hex = hex[0] + hex[0] + hex[1] + hex[1] + hex[2] + hex[2]
|
||||
}
|
||||
if (!/^[0-9a-fA-F]{6}$/.test(hex)) return null
|
||||
return {
|
||||
r: parseInt(hex.slice(0, 2), 16),
|
||||
g: parseInt(hex.slice(2, 4), 16),
|
||||
b: parseInt(hex.slice(4, 6), 16),
|
||||
}
|
||||
}
|
||||
|
||||
function colorWithAlpha(color, alpha) {
|
||||
var parsed = parseHexColor(color)
|
||||
if (!parsed) return 'rgba(15, 23, 42, 0.72)'
|
||||
return 'rgba(' + parsed.r + ', ' + parsed.g + ', ' + parsed.b + ', ' + clamp(alpha, 0, 1).toFixed(3) + ')'
|
||||
}
|
||||
|
||||
function readAudioIntensity() {
|
||||
var audio = window.__STEMENGINE_VIZ_AUDIO__ || window.__STEMENGINE_MUON_AUDIO__ || null
|
||||
if (!audio || typeof audio !== 'object') return 0
|
||||
|
||||
var bass = clamp(audio.bass, 0, 1)
|
||||
var treble = clamp(audio.treble, 0, 1)
|
||||
var energy = clamp(audio.energy, 0, 1)
|
||||
var rms = clamp(audio.rms, 0, 1)
|
||||
var beatBoost = audio.beat ? 0.28 : 0
|
||||
|
||||
return clamp((bass * 0.34) + (treble * 0.2) + (energy * 0.3) + (rms * 0.16) + beatBoost, 0, 1.8)
|
||||
}
|
||||
|
||||
function choosePreset() {
|
||||
var raw = String(window.__STEMENGINE_MUON_PRESET__ || window.__STEMENGINE_VIZ_PRESET__ || 'fluid_vortex')
|
||||
return PRESET_MAP[raw] || PRESET_MAP.fluid_vortex
|
||||
}
|
||||
|
||||
function tick() {
|
||||
var canvas = document.querySelector('canvas')
|
||||
var body = document.body
|
||||
if (!canvas || !body) {
|
||||
requestAnimationFrame(tick)
|
||||
return
|
||||
}
|
||||
|
||||
var paused = Boolean(window.__STEMENGINE_VIZ_PAUSED__)
|
||||
if (paused) {
|
||||
requestAnimationFrame(tick)
|
||||
return
|
||||
}
|
||||
|
||||
var preset = choosePreset()
|
||||
var intensity = readAudioIntensity()
|
||||
var sensitivity = clamp(readParam('audio_sensitivity', 1), 0, 2)
|
||||
var pulseAmount = clamp(readParam('pulse', preset.pulse), 0, 1.5)
|
||||
var hueShift = clamp(readParam('hue_shift', preset.hueShift), -180, 180)
|
||||
var brightness = clamp(readParam('brightness', preset.brightness), 0.5, 1.8)
|
||||
var contrast = clamp(readParam('contrast', preset.contrast), 0.6, 1.8)
|
||||
|
||||
var boosted = clamp(intensity * sensitivity, 0, 2)
|
||||
var pulse = 1 + boosted * pulseAmount * 0.16
|
||||
var activeHue = hueShift + (boosted * 18)
|
||||
|
||||
canvas.style.filter =
|
||||
'saturate(' + (1 + boosted * 0.45).toFixed(3) + ') ' +
|
||||
'hue-rotate(' + activeHue.toFixed(2) + 'deg) ' +
|
||||
'brightness(' + (brightness * pulse).toFixed(3) + ') ' +
|
||||
'contrast(' + (contrast + boosted * 0.08).toFixed(3) + ')'
|
||||
|
||||
var palette = Array.isArray(window.__STEMENGINE_VIZ_PALETTE__) ? window.__STEMENGINE_VIZ_PALETTE__ : []
|
||||
var c0 = palette[0] || '#2563eb'
|
||||
var c1 = palette[1] || '#14b8a6'
|
||||
var c2 = palette[2] || '#f97316'
|
||||
var alphaBase = 0.42 + Math.min(0.28, boosted * 0.18)
|
||||
|
||||
body.style.background =
|
||||
'radial-gradient(circle at 28% 24%, ' + colorWithAlpha(c0, alphaBase + 0.08) + ' 0%, rgba(2, 6, 23, 0.92) 46%), ' +
|
||||
'radial-gradient(circle at 76% 78%, ' + colorWithAlpha(c1, alphaBase) + ' 0%, rgba(2, 6, 23, 0.95) 54%), ' +
|
||||
'radial-gradient(circle at 56% 42%, ' + colorWithAlpha(c2, alphaBase - 0.08) + ' 0%, rgba(2, 6, 23, 1) 64%)'
|
||||
|
||||
requestAnimationFrame(tick)
|
||||
}
|
||||
|
||||
if (document.readyState === 'loading') {
|
||||
document.addEventListener('DOMContentLoaded', function onReady() {
|
||||
requestAnimationFrame(tick)
|
||||
})
|
||||
} else {
|
||||
requestAnimationFrame(tick)
|
||||
}
|
||||
})()
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"name": "@stemengine/visualizer-iframe-shared",
|
||||
"version": "0.0.0",
|
||||
"private": true,
|
||||
"description": "Shared iframe scaffolding for StemEngine visualizer providers",
|
||||
"files": [
|
||||
"stemengine-bootstrap.js",
|
||||
"stemengine-router.js",
|
||||
"stemengine-audio-engine.js",
|
||||
"stemengine-feature-extractor.js"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,203 @@
|
||||
/* StemEngine iframe-local audio engine for Muon modes. */
|
||||
;(function stemengineAudioEngine() {
|
||||
'use strict'
|
||||
|
||||
var AudioCtx = window.AudioContext || window.webkitAudioContext
|
||||
var ctx = null
|
||||
var analyser = null
|
||||
var gain = null
|
||||
var sourceNode = null
|
||||
var buffer = null
|
||||
var startedAt = 0
|
||||
var pausedAt = 0
|
||||
var isPlaying = false
|
||||
var playbackRate = 1
|
||||
|
||||
function clampTime(value, duration) {
|
||||
if (!isFinite(value)) return 0
|
||||
if (!isFinite(duration) || duration <= 0) return Math.max(0, value)
|
||||
return Math.max(0, Math.min(value, duration))
|
||||
}
|
||||
|
||||
function disconnectSource() {
|
||||
if (!sourceNode) return
|
||||
try { sourceNode.onended = null } catch (_) {}
|
||||
try { sourceNode.stop() } catch (_) {}
|
||||
try { sourceNode.disconnect() } catch (_) {}
|
||||
sourceNode = null
|
||||
}
|
||||
|
||||
function initialize() {
|
||||
if (ctx) return
|
||||
if (!AudioCtx) throw new Error('AudioContext unavailable')
|
||||
|
||||
ctx = new AudioCtx({ sampleRate: 44100 })
|
||||
analyser = ctx.createAnalyser()
|
||||
analyser.fftSize = 512
|
||||
analyser.smoothingTimeConstant = 0.8
|
||||
analyser.minDecibels = -100
|
||||
analyser.maxDecibels = -30
|
||||
|
||||
gain = ctx.createGain()
|
||||
gain.gain.value = 0
|
||||
|
||||
// Keep Muon iframe audio silent for users, but feed full-strength signal to analyser.
|
||||
// source -> analyser (reactive features)
|
||||
// source -> gain(0) -> destination (muted output)
|
||||
gain.connect(ctx.destination)
|
||||
|
||||
window.__STEMENGINE_MUON_ANALYSER__ = analyser
|
||||
window.__STEMENGINE_MUON_AUDIO_CTX__ = ctx
|
||||
window.__STEMENGINE_VIZ_ANALYSER__ = analyser
|
||||
window.__STEMENGINE_VIZ_AUDIO_CTX__ = ctx
|
||||
}
|
||||
|
||||
function stop() {
|
||||
disconnectSource()
|
||||
isPlaying = false
|
||||
startedAt = 0
|
||||
pausedAt = 0
|
||||
}
|
||||
|
||||
async function loadFromUrl(url) {
|
||||
initialize()
|
||||
if (!url) throw new Error('missing audio URL')
|
||||
|
||||
stop()
|
||||
|
||||
var response = await fetch(url, { mode: 'cors', credentials: 'omit' })
|
||||
if (!response.ok) {
|
||||
throw new Error('audio fetch failed: ' + response.status)
|
||||
}
|
||||
|
||||
var arr = await response.arrayBuffer()
|
||||
buffer = await ctx.decodeAudioData(arr)
|
||||
pausedAt = 0
|
||||
return { duration: buffer.duration }
|
||||
}
|
||||
|
||||
function normalizeAudioBytes(bytes) {
|
||||
if (bytes instanceof ArrayBuffer) return bytes
|
||||
if (ArrayBuffer.isView(bytes)) {
|
||||
return bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength)
|
||||
}
|
||||
throw new Error('unsupported audio byte payload')
|
||||
}
|
||||
|
||||
async function loadFromBytes(bytes) {
|
||||
initialize()
|
||||
|
||||
stop()
|
||||
|
||||
var arr = normalizeAudioBytes(bytes)
|
||||
buffer = await ctx.decodeAudioData(arr)
|
||||
pausedAt = 0
|
||||
return { duration: buffer.duration }
|
||||
}
|
||||
|
||||
function play(opts) {
|
||||
initialize()
|
||||
if (!buffer || !gain) return
|
||||
|
||||
var options = opts || {}
|
||||
if (ctx.state === 'suspended') {
|
||||
ctx.resume().catch(function () {})
|
||||
}
|
||||
|
||||
disconnectSource()
|
||||
|
||||
sourceNode = ctx.createBufferSource()
|
||||
sourceNode.buffer = buffer
|
||||
playbackRate = isFinite(Number(options.rate)) ? Math.max(0.1, Number(options.rate)) : playbackRate
|
||||
sourceNode.playbackRate.value = playbackRate
|
||||
sourceNode.connect(analyser)
|
||||
sourceNode.connect(gain)
|
||||
|
||||
var offset = typeof options.time === 'number' ? options.time : pausedAt
|
||||
var startOffset = clampTime(offset, buffer.duration)
|
||||
startedAt = ctx.currentTime - startOffset / playbackRate
|
||||
pausedAt = 0
|
||||
isPlaying = true
|
||||
|
||||
sourceNode.onended = function () {
|
||||
if (!sourceNode) return
|
||||
var now = getCurrentTime()
|
||||
if (buffer && now >= buffer.duration - 0.01) {
|
||||
isPlaying = false
|
||||
pausedAt = 0
|
||||
startedAt = 0
|
||||
disconnectSource()
|
||||
}
|
||||
}
|
||||
|
||||
sourceNode.start(0, startOffset)
|
||||
}
|
||||
|
||||
function pause() {
|
||||
if (!ctx || !isPlaying) return
|
||||
pausedAt = clampTime(getCurrentTime(), buffer ? buffer.duration : Infinity)
|
||||
isPlaying = false
|
||||
disconnectSource()
|
||||
}
|
||||
|
||||
function seek(time) {
|
||||
var target = clampTime(Number(time) || 0, buffer ? buffer.duration : Infinity)
|
||||
var wasPlaying = isPlaying
|
||||
if (wasPlaying) {
|
||||
pause()
|
||||
pausedAt = target
|
||||
play({ time: target, rate: playbackRate })
|
||||
return
|
||||
}
|
||||
pausedAt = target
|
||||
}
|
||||
|
||||
function setRate(rate) {
|
||||
var next = Number(rate)
|
||||
if (!isFinite(next) || next <= 0) return
|
||||
playbackRate = Math.max(0.1, Math.min(next, 4))
|
||||
if (sourceNode) {
|
||||
sourceNode.playbackRate.value = playbackRate
|
||||
}
|
||||
}
|
||||
|
||||
function getCurrentTime() {
|
||||
if (!ctx) return 0
|
||||
if (!isPlaying) return pausedAt
|
||||
var elapsed = (ctx.currentTime - startedAt) * playbackRate
|
||||
return clampTime(elapsed, buffer ? buffer.duration : Infinity)
|
||||
}
|
||||
|
||||
function getDuration() {
|
||||
return buffer ? buffer.duration : 0
|
||||
}
|
||||
|
||||
window.__StemEngineAudioEngine__ = {
|
||||
initialize: initialize,
|
||||
loadFromUrl: loadFromUrl,
|
||||
loadFromBytes: loadFromBytes,
|
||||
play: play,
|
||||
pause: pause,
|
||||
stop: stop,
|
||||
seek: seek,
|
||||
setRate: setRate,
|
||||
isPlaying: function () { return isPlaying },
|
||||
getRate: function () { return playbackRate },
|
||||
getAnalyser: function () { return analyser },
|
||||
getContext: function () { return ctx },
|
||||
getCurrentTime: getCurrentTime,
|
||||
getDuration: getDuration,
|
||||
}
|
||||
|
||||
// D-1 fix: initialize synchronously at module load.
|
||||
// The original Muon's `vizInit()` is invoked synchronously by `window.onload = vizInit()`
|
||||
// (a Muon-author bug — the `()` invokes immediately, not on the load event). Inside vizInit,
|
||||
// `audioAnalyzer()` reads `window.__STEMENGINE_MUON_ANALYSER__` to bind the default-mode
|
||||
// analyser. If we defer initialize() until DOMContentLoaded, that binding has already
|
||||
// happened against wavesurfer's silent analyser and never re-acquires.
|
||||
// Initializing here ensures __STEMENGINE_MUON_ANALYSER__ is set BEFORE vizInit runs
|
||||
// (we are imported before the original Muon code in index.js).
|
||||
// AudioContext creation without a user gesture is allowed; it starts in 'suspended' state
|
||||
// and resume() succeeds on transportPlay because allow="autoplay" inherits parent activation.
|
||||
try { initialize() } catch (e) { /* baseline check covers WebGL absence; AudioContext absence is rare */ }
|
||||
})()
|
||||
@@ -0,0 +1,197 @@
|
||||
/* StemEngine runtime bootstrap injected into Muon build at image build-time. */
|
||||
|
||||
(function stemengineBootstrap() {
|
||||
const REQUIRED_EXTENSIONS = ['OES_texture_float_linear', 'EXT_color_buffer_float']
|
||||
const OVERLAY_SELECTORS = [
|
||||
'.dg',
|
||||
'.dg.ac',
|
||||
'.lil-gui',
|
||||
'#stats',
|
||||
'.stats',
|
||||
'.fps',
|
||||
'.fps-counter',
|
||||
'.epok-info',
|
||||
'.epok-main-controls',
|
||||
'.epok-audio-controls',
|
||||
'.epok-captured-overlay',
|
||||
'.epok-edit-controls',
|
||||
'.epok-controls',
|
||||
'.epok-editor',
|
||||
'.epok-ui',
|
||||
'.epok-overlay',
|
||||
'.epok-button',
|
||||
'.epok-main-control',
|
||||
]
|
||||
|
||||
function installOverlaySuppressionStyle() {
|
||||
if (document.getElementById('stemengine-overlay-suppression-style')) return
|
||||
const style = document.createElement('style')
|
||||
style.id = 'stemengine-overlay-suppression-style'
|
||||
style.textContent =
|
||||
OVERLAY_SELECTORS.join(',\n') +
|
||||
' { display: none !important; visibility: hidden !important; pointer-events: none !important; }'
|
||||
|
||||
const parent = document.head || document.documentElement
|
||||
if (parent) parent.appendChild(style)
|
||||
}
|
||||
|
||||
function isProtectedNode(node) {
|
||||
if (!node || node.nodeType !== Node.ELEMENT_NODE) return true
|
||||
const tag = node.tagName ? node.tagName.toLowerCase() : ''
|
||||
return (
|
||||
tag === 'html' ||
|
||||
tag === 'head' ||
|
||||
tag === 'body' ||
|
||||
tag === 'canvas' ||
|
||||
tag === 'script' ||
|
||||
tag === 'style' ||
|
||||
tag === 'link'
|
||||
)
|
||||
}
|
||||
|
||||
function removeMatchingOverlayNodes(root) {
|
||||
const scope = root && root.querySelectorAll ? root : document
|
||||
OVERLAY_SELECTORS.forEach((selector) => {
|
||||
const nodes = scope.querySelectorAll(selector)
|
||||
nodes.forEach((node) => {
|
||||
if (isProtectedNode(node)) return
|
||||
node.remove()
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
function installOverlaySuppressionObserver() {
|
||||
if (window.__STEMENGINE_OVERLAY_OBSERVER__) return
|
||||
const observer = new MutationObserver((mutations) => {
|
||||
for (let i = 0; i < mutations.length; i += 1) {
|
||||
const m = mutations[i]
|
||||
if (!m || !m.addedNodes || m.addedNodes.length === 0) continue
|
||||
for (let j = 0; j < m.addedNodes.length; j += 1) {
|
||||
const node = m.addedNodes[j]
|
||||
if (!node || node.nodeType !== Node.ELEMENT_NODE) continue
|
||||
if (isProtectedNode(node)) {
|
||||
removeMatchingOverlayNodes(node)
|
||||
continue
|
||||
}
|
||||
if (node.matches && OVERLAY_SELECTORS.some((selector) => node.matches(selector))) {
|
||||
node.remove()
|
||||
continue
|
||||
}
|
||||
removeMatchingOverlayNodes(node)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
observer.observe(document.documentElement, { childList: true, subtree: true })
|
||||
window.__STEMENGINE_OVERLAY_OBSERVER__ = observer
|
||||
}
|
||||
|
||||
function installOverlaySuppression() {
|
||||
installOverlaySuppressionStyle()
|
||||
removeMatchingOverlayNodes(document)
|
||||
if (document.documentElement) {
|
||||
installOverlaySuppressionObserver()
|
||||
}
|
||||
}
|
||||
|
||||
function installContextCaptureHook() {
|
||||
const originalGetContext = HTMLCanvasElement.prototype.getContext
|
||||
if (typeof originalGetContext !== 'function') return
|
||||
if (HTMLCanvasElement.prototype.__stemengineContextHookInstalled__) return
|
||||
|
||||
HTMLCanvasElement.prototype.getContext = function patchedGetContext(type, attrs) {
|
||||
const ctx = originalGetContext.call(this, type, attrs)
|
||||
if ((type === 'webgl2' || type === 'webgl') && ctx && this.isConnected) {
|
||||
if (!window.__STEMENGINE_MUON_GL_CTX__) {
|
||||
window.__STEMENGINE_MUON_GL_CTX__ = ctx
|
||||
}
|
||||
if (!window.__STEMENGINE_VIZ_GL_CTX__) {
|
||||
window.__STEMENGINE_VIZ_GL_CTX__ = ctx
|
||||
}
|
||||
if (!window.__STEMENGINE_MUON_CANVAS__) {
|
||||
window.__STEMENGINE_MUON_CANVAS__ = this
|
||||
}
|
||||
if (!window.__STEMENGINE_VIZ_CANVAS__) {
|
||||
window.__STEMENGINE_VIZ_CANVAS__ = this
|
||||
}
|
||||
}
|
||||
return ctx
|
||||
}
|
||||
|
||||
Object.defineProperty(HTMLCanvasElement.prototype, '__stemengineContextHookInstalled__', {
|
||||
value: true,
|
||||
configurable: true,
|
||||
writable: false,
|
||||
enumerable: false,
|
||||
})
|
||||
}
|
||||
|
||||
function checkBaseline() {
|
||||
const canvas = document.createElement('canvas')
|
||||
const gl = canvas.getContext('webgl2') || canvas.getContext('webgl')
|
||||
if (!gl) {
|
||||
console.warn('[StemEngine][Muon] WebGL unavailable; fallback mode should be used.')
|
||||
return {
|
||||
ok: false,
|
||||
missing: ['webgl'],
|
||||
}
|
||||
}
|
||||
|
||||
const missing = REQUIRED_EXTENSIONS.filter((name) => !gl.getExtension(name))
|
||||
if (missing.length > 0) {
|
||||
console.warn('[StemEngine][Muon] Missing required WebGL extensions:', missing.join(', '))
|
||||
return { ok: false, missing }
|
||||
}
|
||||
|
||||
return { ok: true, missing: [] }
|
||||
}
|
||||
|
||||
function installContextLifecycleHooks() {
|
||||
let retries = 0
|
||||
const maxRetries = 120
|
||||
|
||||
const attach = () => {
|
||||
const canvas = document.querySelector('canvas.webgl') || document.querySelector('canvas')
|
||||
if (!canvas) {
|
||||
retries += 1
|
||||
if (retries <= maxRetries) requestAnimationFrame(attach)
|
||||
return
|
||||
}
|
||||
|
||||
canvas.addEventListener('webglcontextlost', (event) => {
|
||||
event.preventDefault()
|
||||
console.warn('[StemEngine][Muon] webglcontextlost')
|
||||
window.__STEMENGINE_MUON_CONTEXT_LOST__ = true
|
||||
window.__STEMENGINE_VIZ_CONTEXT_LOST__ = true
|
||||
})
|
||||
|
||||
canvas.addEventListener('webglcontextrestored', () => {
|
||||
console.info('[StemEngine][Muon] webglcontextrestored')
|
||||
window.__STEMENGINE_MUON_CONTEXT_LOST__ = false
|
||||
window.__STEMENGINE_VIZ_CONTEXT_LOST__ = false
|
||||
})
|
||||
}
|
||||
|
||||
attach()
|
||||
}
|
||||
|
||||
if (typeof window.__STEMENGINE_MUON_STOP_FN__ !== 'function') {
|
||||
window.__STEMENGINE_MUON_STOP_FN__ = function stemengineNoopStop() {}
|
||||
}
|
||||
if (typeof window.__STEMENGINE_MUON_RESUME_FN__ !== 'function') {
|
||||
window.__STEMENGINE_MUON_RESUME_FN__ = function stemengineNoopResume() {}
|
||||
}
|
||||
|
||||
installContextCaptureHook()
|
||||
installOverlaySuppression()
|
||||
|
||||
const baseline = checkBaseline()
|
||||
window.__STEMENGINE_MUON_WEBGL_BASELINE__ = baseline
|
||||
window.__STEMENGINE_VIZ_WEBGL_BASELINE__ = baseline
|
||||
|
||||
if (!baseline.ok) {
|
||||
document.documentElement.classList.add('stemengine-muon-degraded')
|
||||
}
|
||||
|
||||
installContextLifecycleHooks()
|
||||
})()
|
||||
@@ -0,0 +1,333 @@
|
||||
/* StemEngine iframe-local feature extractor for Muon render graphs. */
|
||||
;(function stemengineFeatureExtractor() {
|
||||
'use strict'
|
||||
|
||||
var ENERGY_HISTORY_SIZE = 43
|
||||
var BEAT_TIME_HISTORY_SIZE = 24
|
||||
var BUILDUP_WINDOW = 40
|
||||
var DROP_SENSITIVITY = 2.5
|
||||
var SILENCE_FLOOR = 0.008
|
||||
// D-3 fix: per-band peak tracker with slow decay, mirrors host's useAudioFeatures.ts:287-350.
|
||||
// Raw band magnitudes from getFloatFrequencyData() are tiny (0.005-0.05); without this
|
||||
// normalization, render-graph shader uniforms see effectively-zero values and the
|
||||
// visualizer doesn't visibly react. PEAK_DECAY=0.9998 matches the host's value so behavior
|
||||
// is consistent across both pipelines.
|
||||
var PEAK_DECAY = 0.9998
|
||||
var PEAK_FLOOR = 0.001
|
||||
|
||||
var rafId = null
|
||||
var running = false
|
||||
var energyHistory = []
|
||||
var beatTimes = []
|
||||
var lastBeatTime = 0
|
||||
var previousEnergy = 0
|
||||
var bpm = 0
|
||||
var beatThreshold = 1.5
|
||||
var magnitudeData = null
|
||||
var bandPeaks = {
|
||||
sub: PEAK_FLOOR,
|
||||
bass: PEAK_FLOOR,
|
||||
lowMid: PEAK_FLOOR,
|
||||
mid: PEAK_FLOOR,
|
||||
highMid: PEAK_FLOOR,
|
||||
presence: PEAK_FLOOR,
|
||||
brilliance: PEAK_FLOOR,
|
||||
}
|
||||
// Energy peak tracker — same idea as bands, applied to broadband energy.
|
||||
var energyPeak = PEAK_FLOOR
|
||||
var rmsPeak = PEAK_FLOOR
|
||||
|
||||
var FREQUENCY_BANDS = {
|
||||
sub: [20, 60],
|
||||
bass: [60, 250],
|
||||
lowMid: [250, 500],
|
||||
mid: [500, 2000],
|
||||
highMid: [2000, 4000],
|
||||
presence: [4000, 6000],
|
||||
brilliance: [6000, 20000],
|
||||
}
|
||||
|
||||
function clamp01(value) {
|
||||
var n = Number(value)
|
||||
if (!isFinite(n)) return 0
|
||||
return Math.max(0, Math.min(1, n))
|
||||
}
|
||||
|
||||
function calculateAverageAndVariance(values) {
|
||||
if (!values.length) return { average: 0, variance: 0 }
|
||||
var sum = 0
|
||||
var i = 0
|
||||
for (i = 0; i < values.length; i += 1) sum += values[i]
|
||||
var average = sum / values.length
|
||||
var varianceSum = 0
|
||||
for (i = 0; i < values.length; i += 1) {
|
||||
var d = values[i] - average
|
||||
varianceSum += d * d
|
||||
}
|
||||
return { average: average, variance: varianceSum / values.length }
|
||||
}
|
||||
|
||||
function estimateBpm() {
|
||||
if (beatTimes.length < 2) return bpm
|
||||
var intervals = []
|
||||
for (var i = 1; i < beatTimes.length; i += 1) {
|
||||
var dt = beatTimes[i] - beatTimes[i - 1]
|
||||
if (dt > 0.2 && dt < 1.5) intervals.push(dt)
|
||||
}
|
||||
if (!intervals.length) return bpm
|
||||
var sum = 0
|
||||
for (var j = 0; j < intervals.length; j += 1) sum += intervals[j]
|
||||
var avg = sum / intervals.length
|
||||
var computed = 60 / avg
|
||||
return Math.max(40, Math.min(220, computed))
|
||||
}
|
||||
|
||||
function extractFrequencyBands(sampleRate, magnitude) {
|
||||
var out = {
|
||||
sub: 0,
|
||||
bass: 0,
|
||||
lowMid: 0,
|
||||
mid: 0,
|
||||
highMid: 0,
|
||||
presence: 0,
|
||||
brilliance: 0,
|
||||
}
|
||||
|
||||
var nyquist = sampleRate / 2
|
||||
var binCount = magnitude.length || 1
|
||||
var hzPerBin = nyquist / binCount
|
||||
var keys = ['sub', 'bass', 'lowMid', 'mid', 'highMid', 'presence', 'brilliance']
|
||||
|
||||
for (var k = 0; k < keys.length; k += 1) {
|
||||
var key = keys[k]
|
||||
var lowHz = FREQUENCY_BANDS[key][0]
|
||||
var highHz = FREQUENCY_BANDS[key][1]
|
||||
var start = Math.max(0, Math.floor(lowHz / hzPerBin))
|
||||
var end = Math.min(binCount, Math.ceil(highHz / hzPerBin))
|
||||
var span = Math.max(1, end - start)
|
||||
var sumSquares = 0
|
||||
for (var i = start; i < end; i += 1) {
|
||||
var m = magnitude[i]
|
||||
sumSquares += m * m
|
||||
}
|
||||
var raw = Math.sqrt(sumSquares / span)
|
||||
// D-3 fix: divide by a slowly-decaying peak so the output is in a perceptually
|
||||
// useful 0..1 range. Without this, render-graph shaders see ~0.04 for a peak bass hit.
|
||||
bandPeaks[key] = Math.max(bandPeaks[key] * PEAK_DECAY, raw, PEAK_FLOOR)
|
||||
out[key] = clamp01(raw / bandPeaks[key])
|
||||
}
|
||||
|
||||
return out
|
||||
}
|
||||
|
||||
function detectBeat(energy) {
|
||||
var now = performance.now() / 1000
|
||||
|
||||
energyHistory.push(energy)
|
||||
if (energyHistory.length > ENERGY_HISTORY_SIZE) energyHistory.shift()
|
||||
|
||||
var stats = calculateAverageAndVariance(energyHistory)
|
||||
var average = stats.average
|
||||
var stdDev = Math.sqrt(Math.max(stats.variance, 1e-9))
|
||||
var zScore = stdDev > 0 ? (energy - average) / stdDev : 0
|
||||
|
||||
var minInterval = 0.18
|
||||
var isCandidate = energy > average * beatThreshold && zScore > 0.35
|
||||
var isBeat = isCandidate && (now - lastBeatTime) > minInterval
|
||||
|
||||
if (isBeat) {
|
||||
lastBeatTime = now
|
||||
beatTimes.push(now)
|
||||
if (beatTimes.length > BEAT_TIME_HISTORY_SIZE) beatTimes.shift()
|
||||
bpm = estimateBpm()
|
||||
}
|
||||
|
||||
var beatPeriod = bpm > 0 ? 60 / bpm : 0
|
||||
var timeSinceLastBeat = lastBeatTime > 0 ? now - lastBeatTime : 0
|
||||
var beatPhase = beatPeriod > 0 ? (timeSinceLastBeat % beatPeriod) / beatPeriod : 0
|
||||
|
||||
var confidence = clamp01((zScore / 3) * 0.6 + ((energy / Math.max(average, 1e-6)) - 1) * 0.4)
|
||||
|
||||
return {
|
||||
isBeat: isBeat,
|
||||
bpm: bpm,
|
||||
confidence: confidence,
|
||||
beatPhase: clamp01(beatPhase),
|
||||
timeSinceLastBeat: timeSinceLastBeat,
|
||||
}
|
||||
}
|
||||
|
||||
function detectPatterns(energy, isBeat) {
|
||||
var energyDelta = energy - previousEnergy
|
||||
previousEnergy = energy
|
||||
|
||||
var tail = energyHistory.slice(-BUILDUP_WINDOW)
|
||||
var avg = energy
|
||||
if (tail.length) {
|
||||
var sum = 0
|
||||
for (var i = 0; i < tail.length; i += 1) sum += tail[i]
|
||||
avg = sum / tail.length
|
||||
}
|
||||
|
||||
var isSilence = energy < SILENCE_FLOOR
|
||||
var isBuildUp = !isSilence && energyDelta > 0.01 && energy > avg * 1.02
|
||||
var isBeatDrop = isBeat && !isSilence && energy > avg * DROP_SENSITIVITY
|
||||
|
||||
return {
|
||||
isBeatDrop: isBeatDrop,
|
||||
isBuildUp: isBuildUp,
|
||||
isSilence: isSilence,
|
||||
intensity: clamp01(energy * 8),
|
||||
energyDelta: energyDelta,
|
||||
}
|
||||
}
|
||||
|
||||
function extractFrame(analyser, ctx, frequencyData, timeData) {
|
||||
analyser.getFloatFrequencyData(frequencyData)
|
||||
analyser.getFloatTimeDomainData(timeData)
|
||||
|
||||
if (!magnitudeData || magnitudeData.length !== frequencyData.length) {
|
||||
magnitudeData = new Float32Array(frequencyData.length)
|
||||
}
|
||||
|
||||
var bins = frequencyData.length || 1
|
||||
var nyquist = ctx.sampleRate / 2
|
||||
|
||||
var powerSum = 0
|
||||
var magSum = 0
|
||||
var weightedBinSum = 0
|
||||
var totalEnergy = 0
|
||||
|
||||
for (var i = 0; i < frequencyData.length; i += 1) {
|
||||
var db = isFinite(frequencyData[i]) ? frequencyData[i] : -100
|
||||
var mag = Math.pow(10, db / 20)
|
||||
magnitudeData[i] = mag
|
||||
|
||||
var power = mag * mag
|
||||
powerSum += power
|
||||
magSum += mag
|
||||
weightedBinSum += i * mag
|
||||
totalEnergy += mag
|
||||
}
|
||||
|
||||
var rawRms = Math.sqrt(powerSum / bins)
|
||||
var rawEnergy = totalEnergy / bins
|
||||
// D-3 fix: peak-normalize broadband energy and RMS too (same approach as bands).
|
||||
rmsPeak = Math.max(rmsPeak * PEAK_DECAY, rawRms, PEAK_FLOOR)
|
||||
energyPeak = Math.max(energyPeak * PEAK_DECAY, rawEnergy, PEAK_FLOOR)
|
||||
var rms = clamp01(rawRms / rmsPeak)
|
||||
var energy = clamp01(rawEnergy / energyPeak)
|
||||
var centroidBin = magSum > 0 ? weightedBinSum / magSum : 0
|
||||
var spectralCentroidHz = (centroidBin / bins) * nyquist
|
||||
var spectralCentroid = clamp01(spectralCentroidHz / nyquist)
|
||||
|
||||
var geoLogSum = 0
|
||||
for (var j = 0; j < magnitudeData.length; j += 1) {
|
||||
geoLogSum += Math.log(Math.max(1e-12, magnitudeData[j]))
|
||||
}
|
||||
var geometricMean = Math.exp(geoLogSum / bins)
|
||||
var arithmeticMean = magSum / bins
|
||||
var spectralFlatness = arithmeticMean > 0 ? geometricMean / arithmeticMean : 0
|
||||
|
||||
var zc = 0
|
||||
for (var t = 1; t < timeData.length; t += 1) {
|
||||
var prev = timeData[t - 1]
|
||||
var cur = timeData[t]
|
||||
if ((prev >= 0 && cur < 0) || (prev < 0 && cur >= 0)) zc += 1
|
||||
}
|
||||
var zcr = timeData.length > 1 ? zc / (timeData.length - 1) : 0
|
||||
|
||||
var frequencyBands = extractFrequencyBands(ctx.sampleRate, magnitudeData)
|
||||
// detectBeat uses RAW (un-normalized) energy so the peak detector compares apples to apples
|
||||
// across recent history. Using normalized 0..1 energy would defeat the variance check.
|
||||
var beatInfo = detectBeat(rawEnergy)
|
||||
var patternInfo = detectPatterns(rawEnergy, beatInfo.isBeat)
|
||||
|
||||
// Treble = mean of (presence, brilliance) — matches host's useAudioFeatures convention.
|
||||
// Previously used (highMid + presence + brilliance)/3 which spilled mid-range into treble.
|
||||
var payload = {
|
||||
bass: frequencyBands.bass,
|
||||
treble: clamp01((frequencyBands.presence + frequencyBands.brilliance) / 2),
|
||||
rms: rms,
|
||||
energy: energy,
|
||||
beat: beatInfo.isBeat,
|
||||
drumIntensity: clamp01(frequencyBands.sub * 0.7 + frequencyBands.bass * 0.3),
|
||||
sub: frequencyBands.sub,
|
||||
beatPhase: beatInfo.beatPhase,
|
||||
bpm: beatInfo.bpm,
|
||||
beatConfidence: beatInfo.confidence,
|
||||
isBeatDrop: patternInfo.isBeatDrop,
|
||||
isBuildUp: patternInfo.isBuildUp,
|
||||
isSilence: patternInfo.isSilence,
|
||||
energyDelta: patternInfo.energyDelta,
|
||||
spectralCentroid: spectralCentroid,
|
||||
spectralFlatness: clamp01(spectralFlatness),
|
||||
zcr: clamp01(zcr),
|
||||
frequencyBands: frequencyBands,
|
||||
fft: frequencyData,
|
||||
waveform: timeData,
|
||||
__seUpdatedAt: Date.now(),
|
||||
}
|
||||
window.__STEMENGINE_MUON_AUDIO__ = payload
|
||||
window.__STEMENGINE_VIZ_AUDIO__ = payload
|
||||
}
|
||||
|
||||
function stop() {
|
||||
running = false
|
||||
if (rafId !== null) {
|
||||
cancelAnimationFrame(rafId)
|
||||
rafId = null
|
||||
}
|
||||
}
|
||||
|
||||
function start() {
|
||||
var engine = window.__StemEngineAudioEngine__
|
||||
if (!engine || typeof engine.getAnalyser !== 'function' || typeof engine.getContext !== 'function') return
|
||||
|
||||
var analyser = engine.getAnalyser()
|
||||
var ctx = engine.getContext()
|
||||
if (!analyser || !ctx) return
|
||||
|
||||
stop()
|
||||
running = true
|
||||
|
||||
var frequencyData = new Float32Array(analyser.frequencyBinCount)
|
||||
var timeData = new Float32Array(analyser.fftSize)
|
||||
|
||||
function loop() {
|
||||
if (!running) return
|
||||
extractFrame(analyser, ctx, frequencyData, timeData)
|
||||
rafId = requestAnimationFrame(loop)
|
||||
}
|
||||
|
||||
rafId = requestAnimationFrame(loop)
|
||||
}
|
||||
|
||||
function reset() {
|
||||
energyHistory = []
|
||||
beatTimes = []
|
||||
lastBeatTime = 0
|
||||
previousEnergy = 0
|
||||
bpm = 0
|
||||
rmsPeak = PEAK_FLOOR
|
||||
energyPeak = PEAK_FLOOR
|
||||
bandPeaks.sub = PEAK_FLOOR
|
||||
bandPeaks.bass = PEAK_FLOOR
|
||||
bandPeaks.lowMid = PEAK_FLOOR
|
||||
bandPeaks.mid = PEAK_FLOOR
|
||||
bandPeaks.highMid = PEAK_FLOOR
|
||||
bandPeaks.presence = PEAK_FLOOR
|
||||
bandPeaks.brilliance = PEAK_FLOOR
|
||||
}
|
||||
|
||||
window.__StemEngineFeatureExtractor__ = {
|
||||
start: start,
|
||||
stop: stop,
|
||||
reset: reset,
|
||||
setBeatThreshold: function (threshold) {
|
||||
var t = Number(threshold)
|
||||
if (!isFinite(t)) return
|
||||
beatThreshold = Math.max(0, Math.min(3, t))
|
||||
},
|
||||
}
|
||||
})()
|
||||
@@ -0,0 +1,471 @@
|
||||
/* StemEngine — style router + postMessage command listener.
|
||||
* Injected into the Muon build at image build-time via Dockerfile.
|
||||
*
|
||||
* Project policy: zero access control restrictions on this path. Origin validation
|
||||
* is intentionally a no-op. See docs/plans/2026-04-27_Muon_Default_Mode_Reactivity/
|
||||
* README.md section 8 for the rationale.
|
||||
*/
|
||||
;(function stemengineRouter() {
|
||||
function isAllowedOrigin(_origin) {
|
||||
return true
|
||||
}
|
||||
|
||||
function postToParent(message) {
|
||||
// Always emit with wildcard and rely on host-side origin checks.
|
||||
// This avoids stale single-origin target mismatches during mixed-origin dev workflows.
|
||||
parent.postMessage(message, '*')
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// Helpers
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
function getStyleParam() {
|
||||
try {
|
||||
return new URLSearchParams(location.search).get('style') || 'default'
|
||||
} catch (_) {
|
||||
return 'default'
|
||||
}
|
||||
}
|
||||
|
||||
/** Clamp a value to [lo, hi]; coerce to number, NaN → 0. */
|
||||
function clamp(v, lo, hi) {
|
||||
var n = Number(v)
|
||||
if (!isFinite(n)) n = 0
|
||||
return Math.max(lo, Math.min(hi, n))
|
||||
}
|
||||
|
||||
var KNOWN_COMMANDS = {
|
||||
setStyle: true,
|
||||
setPreset: true,
|
||||
setQuality: true,
|
||||
setParam: true,
|
||||
setPalette: true,
|
||||
setPaused: true,
|
||||
setViewport: true,
|
||||
pushAudioFeatures: true,
|
||||
loadAudio: true,
|
||||
loadAudioData: true,
|
||||
transportPlay: true,
|
||||
transportPause: true,
|
||||
transportSeek: true,
|
||||
transportRate: true,
|
||||
dispose: true,
|
||||
}
|
||||
|
||||
function emitLoaded(info) {
|
||||
postToParent({
|
||||
version: 1,
|
||||
id: 'loaded-' + Date.now(),
|
||||
type: 'loaded',
|
||||
payload: {
|
||||
duration: info && Number.isFinite(Number(info.duration)) ? Number(info.duration) : 0,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
function emitCapabilities() {
|
||||
var providerId = typeof window.__STEMENGINE_VIZ_PROVIDER_ID__ === 'string'
|
||||
? window.__STEMENGINE_VIZ_PROVIDER_ID__
|
||||
: 'muon'
|
||||
var defaults = {
|
||||
providerId: providerId,
|
||||
protocolVersion: 1,
|
||||
presets: [],
|
||||
params: [],
|
||||
palette: {
|
||||
supported: false,
|
||||
slotCount: 0,
|
||||
},
|
||||
}
|
||||
var declared = window.__STEMENGINE_VIZ_CAPABILITIES__
|
||||
var payload = declared && typeof declared === 'object'
|
||||
? Object.assign({}, defaults, declared)
|
||||
: defaults
|
||||
|
||||
postToParent({
|
||||
version: 1,
|
||||
id: 'capabilities-' + Date.now(),
|
||||
type: 'capabilities',
|
||||
payload: payload,
|
||||
})
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// Audio feature state — host-push model.
|
||||
// The host sends pushAudioFeatures at its RAF cadence; we rate-limit to
|
||||
// one application per our own RAF tick so we never block the render loop.
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
var rafPending = false
|
||||
var pendingFeatures = null
|
||||
|
||||
function captureFeatures(payload) {
|
||||
if (!payload || typeof payload !== 'object') return
|
||||
pendingFeatures = {
|
||||
bass: clamp(payload.bass, 0, 1),
|
||||
treble: clamp(payload.treble, 0, 1),
|
||||
rms: clamp(payload.rms, 0, 1),
|
||||
energy: clamp(payload.energy, 0, 1),
|
||||
beat: Boolean(payload.beat),
|
||||
drumIntensity: clamp(payload.drumIntensity, 0, 1),
|
||||
sub: clamp(payload.sub, 0, 1),
|
||||
beatPhase: clamp(payload.beatPhase, 0, 1),
|
||||
bpm: Number.isFinite(Number(payload.bpm)) ? Number(payload.bpm) : 0,
|
||||
beatConfidence: clamp(payload.beatConfidence, 0, 1),
|
||||
isBeatDrop: Boolean(payload.isBeatDrop),
|
||||
isBuildUp: Boolean(payload.isBuildUp),
|
||||
isSilence: Boolean(payload.isSilence),
|
||||
energyDelta: Number.isFinite(Number(payload.energyDelta)) ? Number(payload.energyDelta) : 0,
|
||||
spectralCentroid: clamp(payload.spectralCentroid, 0, 1),
|
||||
spectralFlatness: clamp(payload.spectralFlatness, 0, 1),
|
||||
frequencyBands: payload.frequencyBands || null,
|
||||
}
|
||||
}
|
||||
|
||||
function flushFeatures() {
|
||||
if (!pendingFeatures) return
|
||||
window.__STEMENGINE_MUON_AUDIO__ = pendingFeatures
|
||||
window.__STEMENGINE_VIZ_AUDIO__ = pendingFeatures
|
||||
pendingFeatures = null
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// Render graph lifecycle
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
var currentRenderGraph = null
|
||||
var lastStyleValue = null
|
||||
|
||||
function callRenderGraphMethod(name, args) {
|
||||
if (!currentRenderGraph) return
|
||||
var method = currentRenderGraph[name]
|
||||
if (typeof method !== 'function') return
|
||||
try {
|
||||
method.apply(currentRenderGraph, args || [])
|
||||
} catch (error) {
|
||||
window.__stemengineEmitError__('RENDER_GRAPH_COMMAND', String(error && error.message ? error.message : error))
|
||||
}
|
||||
}
|
||||
|
||||
function updateRenderGraph(style) {
|
||||
// Only instantiate when style actually changes or on first call.
|
||||
if (style === lastStyleValue) return
|
||||
lastStyleValue = style
|
||||
|
||||
// Dispose previous render graph (if any).
|
||||
if (currentRenderGraph && typeof currentRenderGraph.dispose === 'function') {
|
||||
try {
|
||||
currentRenderGraph.dispose()
|
||||
} catch (e) {
|
||||
console.warn('[StemEngine][Muon] Error disposing render graph:', e)
|
||||
}
|
||||
}
|
||||
currentRenderGraph = null
|
||||
|
||||
if (style === 'default') {
|
||||
if (typeof window.__STEMENGINE_MUON_RESUME_FN__ === 'function') {
|
||||
try {
|
||||
window.__STEMENGINE_MUON_RESUME_FN__()
|
||||
} catch (e) {
|
||||
console.warn('[StemEngine][Muon] Error resuming default renderer:', e)
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
if (typeof window.__STEMENGINE_MUON_STOP_FN__ === 'function') {
|
||||
try {
|
||||
window.__STEMENGINE_MUON_STOP_FN__()
|
||||
} catch (e) {
|
||||
console.warn('[StemEngine][Muon] Error stopping default renderer:', e)
|
||||
}
|
||||
}
|
||||
|
||||
var viewport = window.__STEMENGINE_MUON_VIEWPORT__ || {}
|
||||
var width = Number(viewport.width) || undefined
|
||||
var height = Number(viewport.height) || undefined
|
||||
var quality = window.__STEMENGINE_MUON_QUALITY__ || 'balanced'
|
||||
|
||||
// Instantiate new render graph based on style.
|
||||
if (style === 'fluid' && typeof window.__StemEngineFluidRenderGraph__ === 'function') {
|
||||
try {
|
||||
currentRenderGraph = new window.__StemEngineFluidRenderGraph__(window.__STEMENGINE_MUON_GL_CTX__, width, height, quality)
|
||||
console.log('[StemEngine][Muon] Instantiated Fluid render graph')
|
||||
} catch (e) {
|
||||
console.error('[StemEngine][Muon] Error instantiating Fluid render graph:', e)
|
||||
window.__stemengineEmitError__('RENDER_GRAPH_INIT', String(e))
|
||||
}
|
||||
} else if (style === 'tendrils' && typeof window.__StemEngineTendrilsRenderGraph__ === 'function') {
|
||||
try {
|
||||
currentRenderGraph = new window.__StemEngineTendrilsRenderGraph__(window.__STEMENGINE_MUON_CANVAS__, window.__STEMENGINE_MUON_GL_CTX__, width, height, quality)
|
||||
console.log('[StemEngine][Muon] Instantiated Tendrils render graph')
|
||||
} catch (e) {
|
||||
console.error('[StemEngine][Muon] Error instantiating Tendrils render graph:', e)
|
||||
window.__stemengineEmitError__('RENDER_GRAPH_INIT', String(e))
|
||||
}
|
||||
} else if (style === 'spirit' && typeof window.__StemEngineSpiritRenderGraph__ === 'function') {
|
||||
try {
|
||||
currentRenderGraph = new window.__StemEngineSpiritRenderGraph__(window.__STEMENGINE_MUON_CANVAS__, window.__STEMENGINE_MUON_GL_CTX__, width, height, quality)
|
||||
console.log('[StemEngine][Muon] Instantiated Spirit render graph')
|
||||
} catch (e) {
|
||||
console.error('[StemEngine][Muon] Error instantiating Spirit render graph:', e)
|
||||
window.__stemengineEmitError__('RENDER_GRAPH_INIT', String(e))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// Command dispatch
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
function handleCommand(type, payload) {
|
||||
var p = payload || {}
|
||||
switch (type) {
|
||||
case 'setStyle':
|
||||
window.__STEMENGINE_MUON_STYLE__ = String(p.style || 'default')
|
||||
window.__STEMENGINE_VIZ_STYLE__ = window.__STEMENGINE_MUON_STYLE__
|
||||
updateRenderGraph(window.__STEMENGINE_MUON_STYLE__)
|
||||
break
|
||||
case 'setQuality':
|
||||
window.__STEMENGINE_MUON_QUALITY__ = String(p.quality || 'balanced')
|
||||
window.__STEMENGINE_VIZ_QUALITY__ = window.__STEMENGINE_MUON_QUALITY__
|
||||
callRenderGraphMethod('setQuality', [window.__STEMENGINE_MUON_QUALITY__])
|
||||
break
|
||||
case 'setPreset':
|
||||
window.__STEMENGINE_MUON_PRESET__ = String(p.preset || '')
|
||||
window.__STEMENGINE_VIZ_PRESET__ = window.__STEMENGINE_MUON_PRESET__
|
||||
callRenderGraphMethod('setPreset', [window.__STEMENGINE_MUON_PRESET__])
|
||||
break
|
||||
case 'setParam':
|
||||
if (!window.__STEMENGINE_VIZ_PARAMS__ || typeof window.__STEMENGINE_VIZ_PARAMS__ !== 'object') {
|
||||
window.__STEMENGINE_VIZ_PARAMS__ = {}
|
||||
}
|
||||
if (!window.__STEMENGINE_MUON_PARAMS__ || typeof window.__STEMENGINE_MUON_PARAMS__ !== 'object') {
|
||||
window.__STEMENGINE_MUON_PARAMS__ = window.__STEMENGINE_VIZ_PARAMS__
|
||||
}
|
||||
if (typeof p.name === 'string' && p.name) {
|
||||
window.__STEMENGINE_VIZ_PARAMS__[p.name] = p.value
|
||||
callRenderGraphMethod('setParam', [String(p.name), p.value])
|
||||
}
|
||||
break
|
||||
case 'setPalette':
|
||||
window.__STEMENGINE_VIZ_PALETTE__ = Array.isArray(p.colors) ? p.colors.slice() : []
|
||||
window.__STEMENGINE_MUON_PALETTE__ = window.__STEMENGINE_VIZ_PALETTE__
|
||||
callRenderGraphMethod('setPalette', [window.__STEMENGINE_VIZ_PALETTE__])
|
||||
break
|
||||
case 'setPaused':
|
||||
window.__STEMENGINE_MUON_PAUSED__ = Boolean(p.paused)
|
||||
window.__STEMENGINE_VIZ_PAUSED__ = window.__STEMENGINE_MUON_PAUSED__
|
||||
callRenderGraphMethod('setPaused', [window.__STEMENGINE_MUON_PAUSED__])
|
||||
break
|
||||
case 'setViewport':
|
||||
var cssWidth = clamp(p.cssWidth != null ? p.cssWidth : p.width, 1, 7680)
|
||||
var cssHeight = clamp(p.cssHeight != null ? p.cssHeight : p.height, 1, 4320)
|
||||
var dprRaw = Number(p.dpr)
|
||||
var dpr = isFinite(dprRaw) && dprRaw > 0 ? clamp(dprRaw, 0.25, 8) : 1
|
||||
var pixelWidth = Math.max(1, Math.round(cssWidth * dpr))
|
||||
var pixelHeight = Math.max(1, Math.round(cssHeight * dpr))
|
||||
window.__STEMENGINE_MUON_VIEWPORT__ = {
|
||||
cssWidth: cssWidth,
|
||||
cssHeight: cssHeight,
|
||||
dpr: dpr,
|
||||
width: pixelWidth,
|
||||
height: pixelHeight,
|
||||
}
|
||||
window.__STEMENGINE_VIZ_VIEWPORT__ = window.__STEMENGINE_MUON_VIEWPORT__
|
||||
callRenderGraphMethod('setViewport', [window.__STEMENGINE_MUON_VIEWPORT__])
|
||||
break
|
||||
case 'pushAudioFeatures':
|
||||
captureFeatures(p)
|
||||
if (!rafPending) {
|
||||
rafPending = true
|
||||
requestAnimationFrame(function () {
|
||||
rafPending = false
|
||||
flushFeatures()
|
||||
})
|
||||
}
|
||||
break
|
||||
case 'loadAudio':
|
||||
if (!window.__StemEngineAudioEngine__ || typeof window.__StemEngineAudioEngine__.loadFromUrl !== 'function') break
|
||||
window.__StemEngineAudioEngine__.loadFromUrl(String(p.url || ''))
|
||||
.then(function (info) {
|
||||
if (window.__StemEngineFeatureExtractor__ && typeof window.__StemEngineFeatureExtractor__.start === 'function') {
|
||||
window.__StemEngineFeatureExtractor__.start()
|
||||
}
|
||||
emitLoaded(info)
|
||||
})
|
||||
.catch(function (err) {
|
||||
window.__stemengineEmitError__('AUDIO_LOAD', String(err && err.message ? err.message : err))
|
||||
})
|
||||
break
|
||||
case 'loadAudioData':
|
||||
if (!window.__StemEngineAudioEngine__ || typeof window.__StemEngineAudioEngine__.loadFromBytes !== 'function') break
|
||||
window.__StemEngineAudioEngine__.loadFromBytes(p.bytes)
|
||||
.then(function (info) {
|
||||
if (window.__StemEngineFeatureExtractor__ && typeof window.__StemEngineFeatureExtractor__.start === 'function') {
|
||||
window.__StemEngineFeatureExtractor__.start()
|
||||
}
|
||||
emitLoaded(info)
|
||||
})
|
||||
.catch(function (err) {
|
||||
window.__stemengineEmitError__('AUDIO_LOAD', String(err && err.message ? err.message : err))
|
||||
})
|
||||
break
|
||||
case 'transportPlay':
|
||||
if (window.__StemEngineAudioEngine__ && typeof window.__StemEngineAudioEngine__.play === 'function') {
|
||||
window.__StemEngineAudioEngine__.play({
|
||||
time: Number(p.time) || 0,
|
||||
rate: Number(p.rate) || 1,
|
||||
})
|
||||
}
|
||||
break
|
||||
case 'transportPause':
|
||||
if (window.__StemEngineAudioEngine__ && typeof window.__StemEngineAudioEngine__.pause === 'function') {
|
||||
window.__StemEngineAudioEngine__.pause()
|
||||
}
|
||||
break
|
||||
case 'transportSeek':
|
||||
if (window.__StemEngineAudioEngine__ && typeof window.__StemEngineAudioEngine__.seek === 'function') {
|
||||
var nextTime = Number(p.time) || 0
|
||||
window.__StemEngineAudioEngine__.seek(nextTime)
|
||||
|
||||
// Recovery path: if host is actively playing but a transportPlay command was
|
||||
// dropped/raced, seek updates keep arriving. Auto-resume once from seek.
|
||||
if (
|
||||
!window.__STEMENGINE_MUON_PAUSED__
|
||||
&& typeof window.__StemEngineAudioEngine__.isPlaying === 'function'
|
||||
&& !window.__StemEngineAudioEngine__.isPlaying()
|
||||
&& typeof window.__StemEngineAudioEngine__.play === 'function'
|
||||
) {
|
||||
var resumeRate = Number(p.rate)
|
||||
if (!Number.isFinite(resumeRate) || resumeRate <= 0) {
|
||||
resumeRate = typeof window.__StemEngineAudioEngine__.getRate === 'function'
|
||||
? Number(window.__StemEngineAudioEngine__.getRate()) || 1
|
||||
: 1
|
||||
}
|
||||
window.__StemEngineAudioEngine__.play({
|
||||
time: nextTime,
|
||||
rate: resumeRate,
|
||||
})
|
||||
}
|
||||
}
|
||||
break
|
||||
case 'transportRate':
|
||||
if (window.__StemEngineAudioEngine__ && typeof window.__StemEngineAudioEngine__.setRate === 'function') {
|
||||
window.__StemEngineAudioEngine__.setRate(Number(p.rate) || 1)
|
||||
}
|
||||
break
|
||||
case 'dispose':
|
||||
if (currentRenderGraph && typeof currentRenderGraph.dispose === 'function') {
|
||||
try {
|
||||
currentRenderGraph.dispose()
|
||||
} catch (e) {
|
||||
window.__stemengineEmitError__('RENDER_GRAPH_DISPOSE', String(e && e.message ? e.message : e))
|
||||
}
|
||||
}
|
||||
currentRenderGraph = null
|
||||
if (window.__StemEngineFeatureExtractor__ && typeof window.__StemEngineFeatureExtractor__.stop === 'function') {
|
||||
window.__StemEngineFeatureExtractor__.stop()
|
||||
}
|
||||
if (window.__StemEngineAudioEngine__ && typeof window.__StemEngineAudioEngine__.pause === 'function') {
|
||||
window.__StemEngineAudioEngine__.pause()
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// postMessage listener
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
window.addEventListener('message', function onStemEngineMessage(event) {
|
||||
// Origin guard — reject messages from unexpected origins.
|
||||
if (!isAllowedOrigin(event.origin)) return
|
||||
|
||||
var msg = event.data
|
||||
if (!msg || typeof msg !== 'object') return
|
||||
// Schema: { version: 1, type: string, payload: object }
|
||||
if (Number(msg.version) !== 1) return
|
||||
var type = String(msg.type || '')
|
||||
if (!KNOWN_COMMANDS[type]) return
|
||||
|
||||
handleCommand(type, msg.payload)
|
||||
})
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// Emit ready to host
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
function emitReady() {
|
||||
var baseline = window.__STEMENGINE_VIZ_WEBGL_BASELINE__ || window.__STEMENGINE_MUON_WEBGL_BASELINE__ || { ok: false, missing: [] }
|
||||
var styles = Array.isArray(window.__STEMENGINE_VIZ_STYLES__)
|
||||
? window.__STEMENGINE_VIZ_STYLES__.slice()
|
||||
: ['fluid', 'tendrils', 'spirit', 'default']
|
||||
var payload = {
|
||||
capabilities: {
|
||||
webgl: Boolean(baseline.ok),
|
||||
missingExtensions: baseline.missing || [],
|
||||
audioMode: 'hybrid',
|
||||
},
|
||||
styles: styles,
|
||||
}
|
||||
try {
|
||||
postToParent({ version: 1, id: 'ready-' + Date.now(), type: 'ready', payload: payload })
|
||||
} catch (e) {
|
||||
console.warn('[StemEngine][Muon] Could not emit ready:', e)
|
||||
}
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// Emit error to host (exported for other modules to call)
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
window.__stemengineEmitError__ = function emitError(code, message) {
|
||||
try {
|
||||
postToParent({
|
||||
version: 1,
|
||||
id: 'error-' + Date.now(),
|
||||
type: 'error',
|
||||
payload: { code: String(code || 'UNKNOWN'), message: String(message || '') },
|
||||
})
|
||||
} catch (_) {}
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// Bootstrap
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
// Initialize globals from query params so the render graph can read them
|
||||
// on its first frame even before any postMessage arrives.
|
||||
window.__STEMENGINE_MUON_STYLE__ = window.__STEMENGINE_MUON_STYLE__ || getStyleParam()
|
||||
window.__STEMENGINE_MUON_QUALITY__ = window.__STEMENGINE_MUON_QUALITY__ || 'balanced'
|
||||
window.__STEMENGINE_MUON_PAUSED__ = window.__STEMENGINE_MUON_PAUSED__ || false
|
||||
window.__STEMENGINE_MUON_AUDIO__ = window.__STEMENGINE_MUON_AUDIO__ || null
|
||||
window.__STEMENGINE_MUON_VIEWPORT__ = window.__STEMENGINE_MUON_VIEWPORT__ || null
|
||||
window.__STEMENGINE_VIZ_STYLE__ = window.__STEMENGINE_MUON_STYLE__
|
||||
window.__STEMENGINE_VIZ_QUALITY__ = window.__STEMENGINE_MUON_QUALITY__
|
||||
window.__STEMENGINE_VIZ_PAUSED__ = window.__STEMENGINE_MUON_PAUSED__
|
||||
window.__STEMENGINE_VIZ_AUDIO__ = window.__STEMENGINE_MUON_AUDIO__
|
||||
window.__STEMENGINE_VIZ_VIEWPORT__ = window.__STEMENGINE_MUON_VIEWPORT__
|
||||
window.__STEMENGINE_VIZ_PARAMS__ = window.__STEMENGINE_VIZ_PARAMS__ || {}
|
||||
window.__STEMENGINE_MUON_PARAMS__ = window.__STEMENGINE_VIZ_PARAMS__
|
||||
window.__STEMENGINE_VIZ_PALETTE__ = window.__STEMENGINE_VIZ_PALETTE__ || []
|
||||
window.__STEMENGINE_MUON_PALETTE__ = window.__STEMENGINE_VIZ_PALETTE__
|
||||
|
||||
function onReady() {
|
||||
if (window.__StemEngineAudioEngine__ && typeof window.__StemEngineAudioEngine__.initialize === 'function') {
|
||||
window.__StemEngineAudioEngine__.initialize()
|
||||
}
|
||||
emitReady()
|
||||
setTimeout(emitCapabilities, 0)
|
||||
// Instantiate the initial render graph based on the style query param.
|
||||
updateRenderGraph(window.__STEMENGINE_MUON_STYLE__)
|
||||
}
|
||||
|
||||
if (document.readyState === 'loading') {
|
||||
document.addEventListener('DOMContentLoaded', onReady)
|
||||
} else {
|
||||
onReady()
|
||||
}
|
||||
})()
|
||||
@@ -0,0 +1,48 @@
|
||||
# Spirit Upstream Decision
|
||||
|
||||
Date: 2026-04-28
|
||||
Status: Selected for implementation
|
||||
Owner: StemEngine visualizer iframe migration
|
||||
|
||||
## Scope
|
||||
This decision covers the `spirit` style path referenced in the local iframe visualizer architecture plan.
|
||||
|
||||
## Decision
|
||||
Choose option (b): ship Spirit upstream with synthetic audio coupling.
|
||||
|
||||
Selected implementation policy:
|
||||
- build and ship `visualizer-iframes/spirit/` against upstream `edankwan/The-Spirit`,
|
||||
- add a custom coupling adapter that maps `window.__STEMENGINE_VIZ_AUDIO__` features into Spirit control inputs,
|
||||
- use an aggressive beat-response profile for the first cut,
|
||||
- require `setPalette` propagation support in the first cut,
|
||||
- if coupling initialization fails at runtime, fallback to passive upstream Spirit (do not auto-switch to in-Muon approximation),
|
||||
- final style label remains `Spirit`.
|
||||
|
||||
## Why This Decision
|
||||
- Matches requested product direction: upstream Spirit with audio-driven behavior, not approximation fallback as primary path.
|
||||
- Keeps provider architecture consistent across all four styles.
|
||||
- Avoids carrying long-term naming/UX ambiguity from an approximation-only Spirit path.
|
||||
|
||||
## Runtime Behavior Requirements
|
||||
1. Synthetic coupling input source:
|
||||
- consume `window.__STEMENGINE_VIZ_AUDIO__` features (`bass`, `treble`, `energy`, `rms`, `beat`).
|
||||
2. Coupling mode:
|
||||
- default to uniform-driver mapping, with aggressive beat amplification.
|
||||
3. Fallback policy:
|
||||
- if coupling boot fails, continue rendering passive upstream Spirit and emit host-visible warning/error telemetry.
|
||||
4. Palette requirement:
|
||||
- palette updates from `setPalette` are required in first cut (no defer).
|
||||
5. Label requirement:
|
||||
- style label remains `Spirit`.
|
||||
|
||||
## Mandatory Pre-ship Gates
|
||||
1. Verify upstream license terms for embedding/distribution in this deployment model.
|
||||
2. Confirm standalone build path viability for `edankwan/The-Spirit` in the iframe container workflow.
|
||||
3. Implement and validate synthetic coupling adapter behavior across play/pause/seek/replay.
|
||||
4. Re-run full validation matrix with Spirit included before deleting `muon-iframe/spirit-render-graph.js`.
|
||||
|
||||
## Exit Condition For Spirit Cutover
|
||||
Cutover is complete only when all of the following are true:
|
||||
- license/provenance review is complete and acceptable,
|
||||
- upstream Spirit iframe path is live with synthetic coupling enabled,
|
||||
- Spirit path passes the same replay/reactivity matrix standard used for other styles.
|
||||
@@ -0,0 +1,24 @@
|
||||
# syntax=docker/dockerfile:1.4
|
||||
FROM alpine/git:2.47.2 AS source
|
||||
|
||||
ARG SPIRIT_GIT_REF=c2ed239
|
||||
WORKDIR /src
|
||||
|
||||
RUN git clone https://github.com/edankwan/The-Spirit.git app \
|
||||
&& cd app \
|
||||
&& git checkout "${SPIRIT_GIT_REF}" \
|
||||
&& rm -rf .git
|
||||
|
||||
FROM nginx:1.27-alpine
|
||||
|
||||
COPY --from=source /src/app/app/ /usr/share/nginx/html/
|
||||
COPY index.html /usr/share/nginx/html/index.html
|
||||
COPY stemengine-spirit-prehook.js /usr/share/nginx/html/stemengine-spirit-prehook.js
|
||||
COPY stemengine-spirit-adapter.js /usr/share/nginx/html/stemengine-spirit-adapter.js
|
||||
COPY --from=shared stemengine-bootstrap.js /usr/share/nginx/html/stemengine-bootstrap.js
|
||||
COPY --from=shared stemengine-router.js /usr/share/nginx/html/stemengine-router.js
|
||||
COPY --from=shared stemengine-audio-engine.js /usr/share/nginx/html/stemengine-audio-engine.js
|
||||
COPY --from=shared stemengine-feature-extractor.js /usr/share/nginx/html/stemengine-feature-extractor.js
|
||||
COPY nginx.conf /etc/nginx/conf.d/default.conf
|
||||
|
||||
EXPOSE 80
|
||||
@@ -0,0 +1,60 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<title>The Spirit</title>
|
||||
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
|
||||
<meta name="description" content="" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
|
||||
|
||||
<link href="https://fonts.googleapis.com/css?family=Oxygen:400,700" rel="stylesheet" type="text/css" />
|
||||
<link rel="stylesheet" href="css/normalize.css" />
|
||||
<link rel="stylesheet" href="css/index.css" />
|
||||
</head>
|
||||
<body>
|
||||
<div class="mobile">
|
||||
<p>Sorry, this experiment is not designed for mobile devices. But if you think your device can handle it, you can keep forward to access the experiment <span class="mobile-bypass">here</span>.</p>
|
||||
<p>Or you check out the <a href="https://www.youtube.com/watch?v=lcHAtTGzQE8" target="_blank">video capture</a> of this experiment instead. The source code is also accessible at <a href="https://github.com/edankwan/The-Spirit" target="_blank">Github</a>.</p>
|
||||
<p>Sorry for any inconvenience caused.</p>
|
||||
<p>Cheers,</p>
|
||||
<p><a href="http://www.edankwan.com/" target="_blank">Edan Kwan</a></p>
|
||||
</div>
|
||||
<div class="logo"></div>
|
||||
<div class="instruction">Press space key to toggle movement</div>
|
||||
<div class="footer">
|
||||
<span><a href="http://www.edankwan.com/" target="_blank">Edan Kwan</a></span>
|
||||
<span>|</span>
|
||||
<span><a href="https://github.com/edankwan/The-Spirit" target="_blank">Github</a></span>
|
||||
<span>|</span>
|
||||
<span><a href="https://www.youtube.com/watch?v=lcHAtTGzQE8" target="_blank">Youtube</a></span>
|
||||
</div>
|
||||
|
||||
<script src="js/three.r74.min.js"></script>
|
||||
<script>
|
||||
window.__STEMENGINE_VIZ_PROVIDER_ID__ = 'spirit';
|
||||
window.__STEMENGINE_VIZ_STYLES__ = ['default'];
|
||||
window.__STEMENGINE_VIZ_CAPABILITIES__ = {
|
||||
providerId: 'spirit',
|
||||
protocolVersion: 1,
|
||||
presets: ['spirit_aether', 'spirit_aurora', 'spirit_lumen'],
|
||||
params: [
|
||||
{ name: 'audio_sensitivity', label: 'Audio Sensitivity', type: 'number', default: 1.15, min: 0, max: 2, step: 0.05 },
|
||||
{ name: 'beat_reactivity', label: 'Beat Reactivity', type: 'number', default: 1.4, min: 0, max: 2, step: 0.05 },
|
||||
{ name: 'speed_base', label: 'Base Speed', type: 'number', default: 0.9, min: 0.1, max: 3.5, step: 0.05 },
|
||||
{ name: 'radius_base', label: 'Base Radius', type: 'number', default: 0.7, min: 0.2, max: 3, step: 0.05 },
|
||||
{ name: 'curl_base', label: 'Base Curl', type: 'number', default: 0.024, min: 0.001, max: 0.08, step: 0.001 },
|
||||
{ name: 'attraction_base', label: 'Base Attraction', type: 'number', default: 1.1, min: -2, max: 2, step: 0.05 },
|
||||
{ name: 'die_speed_base', label: 'Base Die Speed', type: 'number', default: 0.013, min: 0.0005, max: 0.05, step: 0.0005 }
|
||||
],
|
||||
palette: { supported: true, slotCount: 8 }
|
||||
};
|
||||
</script>
|
||||
<script src="stemengine-spirit-prehook.js?v=sespirit1"></script>
|
||||
<script src="js/index.js"></script>
|
||||
<script src="stemengine-bootstrap.js?v=sespirit1"></script>
|
||||
<script src="stemengine-audio-engine.js?v=sespirit1"></script>
|
||||
<script src="stemengine-feature-extractor.js?v=sespirit1"></script>
|
||||
<script src="stemengine-router.js?v=sespirit1"></script>
|
||||
<script src="stemengine-spirit-adapter.js?v=sespirit1"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,12 @@
|
||||
server {
|
||||
listen 80;
|
||||
server_name _;
|
||||
|
||||
root /usr/share/nginx/html;
|
||||
index index.html;
|
||||
|
||||
location / {
|
||||
try_files $uri $uri/ /index.html;
|
||||
add_header Cache-Control "no-store, no-cache, must-revalidate" always;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,312 @@
|
||||
;(function stemengineSpiritAdapter() {
|
||||
'use strict'
|
||||
|
||||
var MAX_DRIVER_INIT_ATTEMPTS = 240
|
||||
var state = {
|
||||
attempts: 0,
|
||||
couplingReady: false,
|
||||
fallbackMode: false,
|
||||
couplingMode: 'none',
|
||||
warnedFallback: false,
|
||||
beatEnvelope: 0,
|
||||
paletteSignature: '',
|
||||
colorMats: [],
|
||||
driver: null,
|
||||
pointerPhase: 0,
|
||||
loopTimer: null,
|
||||
}
|
||||
|
||||
window.__STEMENGINE_SPIRIT_ADAPTER_LOADED__ = true
|
||||
|
||||
function clamp(value, min, max) {
|
||||
var n = Number(value)
|
||||
if (!isFinite(n)) n = 0
|
||||
return Math.max(min, Math.min(max, n))
|
||||
}
|
||||
|
||||
function lerp(from, to, ratio) {
|
||||
return from + (to - from) * clamp(ratio, 0, 1)
|
||||
}
|
||||
|
||||
function readParam(name, fallback) {
|
||||
var params = window.__STEMENGINE_VIZ_PARAMS__
|
||||
if (!params || typeof params !== 'object') return fallback
|
||||
if (!Object.prototype.hasOwnProperty.call(params, name)) return fallback
|
||||
return Number(params[name])
|
||||
}
|
||||
|
||||
function parseHexColor(value) {
|
||||
var hex = String(value || '').trim().replace(/^#/, '')
|
||||
if (hex.length === 3) {
|
||||
hex = hex[0] + hex[0] + hex[1] + hex[1] + hex[2] + hex[2]
|
||||
}
|
||||
if (!/^[0-9a-fA-F]{6}$/.test(hex)) return null
|
||||
return '#' + hex.toLowerCase()
|
||||
}
|
||||
|
||||
function choosePreset() {
|
||||
var raw = String(window.__STEMENGINE_VIZ_PRESET__ || 'spirit_aether')
|
||||
if (raw === 'spirit_aurora') {
|
||||
return { speed: 1.05, radius: 0.78, curl: 0.028, attraction: 1.2, dieSpeed: 0.014 }
|
||||
}
|
||||
if (raw === 'spirit_lumen') {
|
||||
return { speed: 1.2, radius: 0.84, curl: 0.032, attraction: 1.35, dieSpeed: 0.015 }
|
||||
}
|
||||
return { speed: 0.92, radius: 0.72, curl: 0.024, attraction: 1.08, dieSpeed: 0.013 }
|
||||
}
|
||||
|
||||
function readAudio() {
|
||||
var audio = window.__STEMENGINE_VIZ_AUDIO__ || window.__STEMENGINE_MUON_AUDIO__ || null
|
||||
if (!audio || typeof audio !== 'object') {
|
||||
return { bass: 0, treble: 0, energy: 0, rms: 0, beat: false }
|
||||
}
|
||||
return {
|
||||
bass: clamp(audio.bass, 0, 1),
|
||||
treble: clamp(audio.treble, 0, 1),
|
||||
energy: clamp(audio.energy, 0, 1),
|
||||
rms: clamp(audio.rms, 0, 1),
|
||||
beat: Boolean(audio.beat),
|
||||
}
|
||||
}
|
||||
|
||||
function findSpiritDriver() {
|
||||
var hook = window.__STEMENGINE_SPIRIT_HOOK__
|
||||
var registry = hook && hook.registry
|
||||
if (!registry || !Array.isArray(registry.rawMaterials)) return null
|
||||
|
||||
for (var i = 0; i < registry.rawMaterials.length; i += 1) {
|
||||
var material = registry.rawMaterials[i]
|
||||
var uniforms = material && material.uniforms
|
||||
if (!uniforms || typeof uniforms !== 'object') continue
|
||||
|
||||
if (uniforms.speed && uniforms.dieSpeed && uniforms.radius && uniforms.curlSize && uniforms.attraction && uniforms.mouse3d) {
|
||||
return uniforms
|
||||
}
|
||||
}
|
||||
|
||||
return null
|
||||
}
|
||||
|
||||
function collectPaletteTargets() {
|
||||
var hook = window.__STEMENGINE_SPIRIT_HOOK__
|
||||
var registry = hook && hook.registry
|
||||
if (!registry || !Array.isArray(registry.shaderMaterials)) return
|
||||
|
||||
var result = []
|
||||
for (var i = 0; i < registry.shaderMaterials.length; i += 1) {
|
||||
var material = registry.shaderMaterials[i]
|
||||
var uniforms = material && material.uniforms
|
||||
if (!uniforms || typeof uniforms !== 'object') continue
|
||||
if (uniforms.color1 && uniforms.color2) result.push(uniforms)
|
||||
}
|
||||
state.colorMats = result
|
||||
}
|
||||
|
||||
function emitFallbackWarning(message) {
|
||||
if (state.warnedFallback) return
|
||||
state.warnedFallback = true
|
||||
if (typeof window.__stemengineEmitError__ === 'function') {
|
||||
window.__stemengineEmitError__('SPIRIT_COUPLING_FALLBACK', message)
|
||||
return
|
||||
}
|
||||
console.warn('[StemEngine][Spirit]', message)
|
||||
}
|
||||
|
||||
function updateCouplingStatus() {
|
||||
window.__STEMENGINE_SPIRIT_COUPLING__ = {
|
||||
ready: state.couplingReady,
|
||||
mode: state.couplingMode,
|
||||
attempts: state.attempts,
|
||||
fallback: state.fallbackMode,
|
||||
}
|
||||
}
|
||||
|
||||
updateCouplingStatus()
|
||||
|
||||
function driveViaSyntheticPointer() {
|
||||
var paused = Boolean(window.__STEMENGINE_VIZ_PAUSED__)
|
||||
if (paused) return
|
||||
|
||||
var audio = readAudio()
|
||||
var sensitivity = clamp(readParam('audio_sensitivity', 1.15), 0, 2)
|
||||
var beatReactivity = clamp(readParam('beat_reactivity', 1.4), 0, 2)
|
||||
|
||||
if (audio.beat) {
|
||||
state.beatEnvelope = Math.min(1.8, state.beatEnvelope + 0.95 * beatReactivity)
|
||||
} else {
|
||||
state.beatEnvelope = Math.max(0, state.beatEnvelope * 0.82)
|
||||
}
|
||||
|
||||
var intensity = clamp(
|
||||
(audio.bass * 0.36 + audio.treble * 0.18 + audio.energy * 0.3 + audio.rms * 0.16) * sensitivity + state.beatEnvelope * 0.28,
|
||||
0,
|
||||
2.4,
|
||||
)
|
||||
|
||||
state.pointerPhase += 0.01 + intensity * 0.02
|
||||
var width = Math.max(1, window.innerWidth || document.documentElement.clientWidth || 1)
|
||||
var height = Math.max(1, window.innerHeight || document.documentElement.clientHeight || 1)
|
||||
var cx = width * 0.5
|
||||
var cy = height * 0.5
|
||||
var orbitX = Math.min(width * 0.42, 120 + intensity * 180)
|
||||
var orbitY = Math.min(height * 0.42, 70 + intensity * 120)
|
||||
var x = cx + Math.cos(state.pointerPhase * (1.0 + intensity * 0.7)) * orbitX
|
||||
var y = cy + Math.sin(state.pointerPhase * (1.6 + intensity * 0.95)) * orbitY
|
||||
|
||||
var event
|
||||
try {
|
||||
event = new MouseEvent('mousemove', {
|
||||
bubbles: true,
|
||||
cancelable: true,
|
||||
clientX: x,
|
||||
clientY: y,
|
||||
pageX: x,
|
||||
pageY: y,
|
||||
})
|
||||
} catch (_) {
|
||||
return
|
||||
}
|
||||
|
||||
window.dispatchEvent(event)
|
||||
}
|
||||
|
||||
function applyPaletteIfNeeded() {
|
||||
var palette = Array.isArray(window.__STEMENGINE_VIZ_PALETTE__) ? window.__STEMENGINE_VIZ_PALETTE__ : []
|
||||
var c1 = parseHexColor(palette[0] || '#ffffff') || '#ffffff'
|
||||
var c2 = parseHexColor(palette[1] || palette[0] || '#9fd5ff') || '#9fd5ff'
|
||||
var bg = parseHexColor(palette[2] || '#0f172a') || '#0f172a'
|
||||
var signature = [c1, c2, bg].join('|')
|
||||
if (signature === state.paletteSignature) return
|
||||
state.paletteSignature = signature
|
||||
|
||||
for (var i = 0; i < state.colorMats.length; i += 1) {
|
||||
var uniforms = state.colorMats[i]
|
||||
try {
|
||||
if (uniforms.color1 && uniforms.color1.value && typeof uniforms.color1.value.setStyle === 'function') {
|
||||
uniforms.color1.value.setStyle(c1)
|
||||
}
|
||||
if (uniforms.color2 && uniforms.color2.value && typeof uniforms.color2.value.setStyle === 'function') {
|
||||
uniforms.color2.value.setStyle(c2)
|
||||
}
|
||||
} catch (_) {
|
||||
// Keep rendering even if one material update fails.
|
||||
}
|
||||
}
|
||||
|
||||
document.body.style.background =
|
||||
'radial-gradient(circle at 25% 20%, ' + c1 + '22 0%, transparent 40%),' +
|
||||
'radial-gradient(circle at 75% 80%, ' + c2 + '20 0%, transparent 46%),' +
|
||||
bg
|
||||
}
|
||||
|
||||
function driveUniforms(uniforms) {
|
||||
var paused = Boolean(window.__STEMENGINE_VIZ_PAUSED__)
|
||||
var preset = choosePreset()
|
||||
var audio = readAudio()
|
||||
|
||||
var sensitivity = clamp(readParam('audio_sensitivity', 1.15), 0, 2)
|
||||
var beatReactivity = clamp(readParam('beat_reactivity', 1.4), 0, 2)
|
||||
|
||||
if (audio.beat) {
|
||||
state.beatEnvelope = Math.min(1.8, state.beatEnvelope + 0.95 * beatReactivity)
|
||||
} else {
|
||||
state.beatEnvelope = Math.max(0, state.beatEnvelope * 0.82)
|
||||
}
|
||||
|
||||
var intensity = clamp(
|
||||
(audio.bass * 0.36 + audio.treble * 0.18 + audio.energy * 0.3 + audio.rms * 0.16) * sensitivity + state.beatEnvelope * 0.28,
|
||||
0,
|
||||
2.4,
|
||||
)
|
||||
|
||||
var baseSpeed = clamp(readParam('speed_base', preset.speed), 0.1, 3.5)
|
||||
var baseRadius = clamp(readParam('radius_base', preset.radius), 0.2, 3)
|
||||
var baseCurl = clamp(readParam('curl_base', preset.curl), 0.001, 0.08)
|
||||
var baseAttraction = clamp(readParam('attraction_base', preset.attraction), -2, 2)
|
||||
var baseDieSpeed = clamp(readParam('die_speed_base', preset.dieSpeed), 0.0005, 0.05)
|
||||
|
||||
var speed = paused ? 0 : clamp(baseSpeed + intensity * 0.9, 0, 4)
|
||||
var dieSpeed = paused ? 0 : clamp(baseDieSpeed + intensity * 0.0045, 0, 0.08)
|
||||
var radius = clamp(baseRadius + intensity * 0.5 + state.beatEnvelope * 0.14, 0.2, 4)
|
||||
var curlSize = clamp(baseCurl + intensity * 0.014, 0.001, 0.12)
|
||||
var attraction = clamp(baseAttraction + intensity * 0.34, -2, 2)
|
||||
|
||||
if (uniforms.speed && uniforms.speed.value != null) uniforms.speed.value = lerp(uniforms.speed.value, speed, 0.35)
|
||||
if (uniforms.dieSpeed && uniforms.dieSpeed.value != null) uniforms.dieSpeed.value = lerp(uniforms.dieSpeed.value, dieSpeed, 0.35)
|
||||
if (uniforms.radius && uniforms.radius.value != null) uniforms.radius.value = lerp(uniforms.radius.value, radius, 0.22)
|
||||
if (uniforms.curlSize && uniforms.curlSize.value != null) uniforms.curlSize.value = lerp(uniforms.curlSize.value, curlSize, 0.28)
|
||||
if (uniforms.attraction && uniforms.attraction.value != null) uniforms.attraction.value = lerp(uniforms.attraction.value, attraction, 0.24)
|
||||
|
||||
if (uniforms.mouse3d && uniforms.mouse3d.value && typeof uniforms.mouse3d.value.set === 'function') {
|
||||
var t = Date.now() * 0.001
|
||||
var orbit = 145 + intensity * 135 + state.beatEnvelope * 26
|
||||
var lift = 42 + intensity * 68
|
||||
uniforms.mouse3d.value.set(
|
||||
Math.cos(t * (0.92 + intensity * 0.58)) * orbit,
|
||||
Math.cos(t * (2.15 + intensity * 0.8)) * lift,
|
||||
Math.sin(t * (1.4 + intensity * 0.72)) * orbit,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
function hideChrome() {
|
||||
var selectors = ['.dg', '#stats', '.stats', '.logo', '.instruction', '.footer', '.mobile']
|
||||
for (var i = 0; i < selectors.length; i += 1) {
|
||||
var nodes = document.querySelectorAll(selectors[i])
|
||||
for (var j = 0; j < nodes.length; j += 1) {
|
||||
var node = nodes[j]
|
||||
if (!node) continue
|
||||
node.style.display = 'none'
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function tick() {
|
||||
hideChrome()
|
||||
applyPaletteIfNeeded()
|
||||
|
||||
if (!state.couplingReady && !state.fallbackMode) {
|
||||
state.attempts += 1
|
||||
if (!state.colorMats.length) collectPaletteTargets()
|
||||
var uniforms = findSpiritDriver()
|
||||
if (uniforms) {
|
||||
state.driver = uniforms
|
||||
state.couplingReady = true
|
||||
state.couplingMode = 'uniform-driver'
|
||||
updateCouplingStatus()
|
||||
} else if (state.attempts >= MAX_DRIVER_INIT_ATTEMPTS) {
|
||||
// Secondary coupling mode: drive Spirit via synthetic pointer motion.
|
||||
state.couplingReady = true
|
||||
state.couplingMode = 'input-driver'
|
||||
updateCouplingStatus()
|
||||
emitFallbackWarning('Spirit uniform coupling unavailable; using input-driver coupling mode')
|
||||
}
|
||||
}
|
||||
|
||||
if (state.couplingReady && state.driver) {
|
||||
driveUniforms(state.driver)
|
||||
} else if (state.couplingReady && state.couplingMode === 'input-driver') {
|
||||
driveViaSyntheticPointer()
|
||||
} else if (state.fallbackMode) {
|
||||
updateCouplingStatus()
|
||||
}
|
||||
|
||||
updateCouplingStatus()
|
||||
}
|
||||
|
||||
function startLoop() {
|
||||
if (state.loopTimer) return
|
||||
tick()
|
||||
state.loopTimer = setInterval(tick, 50)
|
||||
}
|
||||
|
||||
if (document.readyState === 'loading') {
|
||||
document.addEventListener('DOMContentLoaded', function onReady() {
|
||||
collectPaletteTargets()
|
||||
startLoop()
|
||||
})
|
||||
} else {
|
||||
collectPaletteTargets()
|
||||
startLoop()
|
||||
}
|
||||
})()
|
||||
@@ -0,0 +1,72 @@
|
||||
;(function stemengineSpiritPrehook() {
|
||||
'use strict'
|
||||
|
||||
var registry = {
|
||||
rawMaterials: [],
|
||||
shaderMaterials: [],
|
||||
renderers: [],
|
||||
scenes: [],
|
||||
cameras: [],
|
||||
}
|
||||
|
||||
function pushUnique(bucket, value) {
|
||||
if (!value) return
|
||||
if (bucket.indexOf(value) >= 0) return
|
||||
bucket.push(value)
|
||||
}
|
||||
|
||||
function collectMaterial(material) {
|
||||
if (!material) return
|
||||
if (!material.uniforms || typeof material.uniforms !== 'object') return
|
||||
|
||||
if (material.uniforms.speed || material.uniforms.curlSize || material.uniforms.mouse3d) {
|
||||
pushUnique(registry.rawMaterials, material)
|
||||
}
|
||||
|
||||
if (material.uniforms.color1 || material.uniforms.color2) {
|
||||
pushUnique(registry.shaderMaterials, material)
|
||||
}
|
||||
}
|
||||
|
||||
function collectScene(scene) {
|
||||
pushUnique(registry.scenes, scene)
|
||||
if (!scene || typeof scene.traverse !== 'function') return
|
||||
scene.traverse(function visit(node) {
|
||||
if (!node) return
|
||||
var material = node.material
|
||||
if (Array.isArray(material)) {
|
||||
for (var i = 0; i < material.length; i += 1) collectMaterial(material[i])
|
||||
return
|
||||
}
|
||||
collectMaterial(material)
|
||||
})
|
||||
}
|
||||
|
||||
function installHooks() {
|
||||
if (!window.THREE || !window.THREE.WebGLRenderer || !window.THREE.WebGLRenderer.prototype) return false
|
||||
var proto = window.THREE.WebGLRenderer.prototype
|
||||
if (proto.__stemengineSpiritRenderWrapped) return true
|
||||
|
||||
var originalRender = proto.render
|
||||
proto.render = function stemengineRenderHook(scene, camera) {
|
||||
pushUnique(registry.renderers, this)
|
||||
pushUnique(registry.cameras, camera)
|
||||
collectScene(scene)
|
||||
return originalRender.apply(this, arguments)
|
||||
}
|
||||
|
||||
proto.__stemengineSpiritRenderWrapped = true
|
||||
|
||||
window.__STEMENGINE_SPIRIT_HOOK__ = {
|
||||
registry: registry,
|
||||
installedAt: Date.now(),
|
||||
version: 1,
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
if (!installHooks()) {
|
||||
window.addEventListener('DOMContentLoaded', installHooks, { once: true })
|
||||
}
|
||||
})()
|
||||
@@ -0,0 +1,22 @@
|
||||
# syntax=docker/dockerfile:1.4
|
||||
FROM alpine/git:2.47.2 AS source
|
||||
|
||||
ARG TENDRILS_GIT_REF=482db7d3127064e6245631087e0ef0a1641f3b5e
|
||||
WORKDIR /src
|
||||
|
||||
RUN git clone https://github.com/keeffEoghan/keeffeoghan.github.io.git app \
|
||||
&& cd app \
|
||||
&& git checkout "${TENDRILS_GIT_REF}" \
|
||||
&& rm -rf .git
|
||||
|
||||
FROM nginx:1.27-alpine
|
||||
|
||||
COPY --from=source /src/app/ /usr/share/nginx/html/
|
||||
COPY --from=shared stemengine-bootstrap.js /usr/share/nginx/html/stemengine-bootstrap.js
|
||||
COPY --from=shared stemengine-router.js /usr/share/nginx/html/stemengine-router.js
|
||||
COPY --from=shared stemengine-audio-engine.js /usr/share/nginx/html/stemengine-audio-engine.js
|
||||
COPY --from=shared stemengine-feature-extractor.js /usr/share/nginx/html/stemengine-feature-extractor.js
|
||||
COPY stemengine-tendrils-adapter.js /usr/share/nginx/html/stemengine-tendrils-adapter.js
|
||||
COPY nginx.conf /etc/nginx/conf.d/default.conf
|
||||
|
||||
EXPOSE 80
|
||||
@@ -0,0 +1,170 @@
|
||||
# Tendrils Audio, Palette, and Build Injection Design (Phase 5 Pre-Work)
|
||||
|
||||
Upstream Reference: https://github.com/keeffEoghan/keeffeoghan.github.io
|
||||
Pinned Commit: `482db7d3127064e6245631087e0ef0a1641f3b5e`
|
||||
Date: 2026-04-28
|
||||
Status: Pre-work analysis complete; implementation plan ready for Phase 5 execution
|
||||
|
||||
## 1. Upstream Inspection Summary
|
||||
|
||||
### 1.1 Repository and Tendrils subtree status
|
||||
|
||||
- Repository clone resolved and inspected at commit `482db7d` (`Update index.html`).
|
||||
- Tendrils subtree exists at `src/tendrils/` and contains full renderer sources, shaders, audio helpers, flow passes, and spawn modules.
|
||||
- Main Tendrils entry for the standalone work page is `src/work/tendrils.main.js`, which imports `src/tendrils/demo.js`.
|
||||
- Core simulation/runtime class is in `src/tendrils/index.js` (`Tendrils` class + defaults/state).
|
||||
|
||||
### 1.2 Build and extraction characteristics
|
||||
|
||||
- Upstream is a full website repo, not a Tendrils-only package.
|
||||
- Root `package.json` exists (`epok-portfolio`) with a gulp/webpack 1 era toolchain.
|
||||
- Build entry selection is broad (`./src/**/*.main.js` via gulp/webpack), with Tendrils specifically mapped through `src/work/tendrils.main.js`.
|
||||
- Prebuilt artifacts already exist in the repository:
|
||||
- `build/js/work/tendrils.js`
|
||||
- `build/css/work/tendrils.css`
|
||||
- `work/tendrils/index.html`
|
||||
|
||||
Conclusion:
|
||||
- The `src/tendrils/` subtree is not independently buildable in isolation.
|
||||
- The full upstream repo is buildable as a whole, but its legacy Node/gulp stack is high-risk for deterministic container builds.
|
||||
- A Phase 5 implementation can proceed using either:
|
||||
1) source rebuild from full upstream repo (higher risk), or
|
||||
2) prebuilt artifact extraction from pinned commit (lower risk, preferred).
|
||||
|
||||
### 1.3 Licensing signal
|
||||
|
||||
- No top-level `LICENSE` or `COPYING` file was found in the pinned upstream snapshot.
|
||||
- `package.json` does not declare a `license` field.
|
||||
|
||||
Implication:
|
||||
- License remains unverified for this upstream and must be explicitly resolved before production shipment.
|
||||
- This is a release gate item and should be treated as a stop/ask condition before Phase 5 rollout completion.
|
||||
|
||||
## 2. Runtime Entry and Audio Control Points
|
||||
|
||||
### 2.1 Actual runtime chain used by Tendrils page
|
||||
|
||||
1. `work/tendrils/index.html` bootstraps page DOM and includes `../../build/js/work/tendrils.js`.
|
||||
2. `src/work/tendrils.main.js` calls `tendrilsDemo(canvas)`.
|
||||
3. `src/tendrils/demo.js` creates track/mic analysers, audio triggers, flow spawners, and render loop.
|
||||
4. `src/tendrils/index.js` applies evolving state into flow/render shaders each frame.
|
||||
|
||||
### 2.2 Audio-reactive logic points to patch
|
||||
|
||||
Primary audio ownership in upstream demo:
|
||||
- `demo.js` constructs track analyser via `web-audio-analyser`.
|
||||
- `AudioTrigger` sampling drives `audioResponse()`.
|
||||
- `audioResponse()` invokes higher-level actions (`spawnFlow`, `spawnFastest`, `spawnForm`, `spawnSamples`, `spawnImage`, `restart`).
|
||||
|
||||
Most direct migration path:
|
||||
- Keep Tendrils simulation/render pipeline intact.
|
||||
- Replace upstream-owned track/mic analyser dependence with StemEngine iframe-owned features.
|
||||
- Feed synthetic trigger data from `window.__STEMENGINE_VIZ_AUDIO__` (shared router/audio engine globals) instead of local mic/track analysers.
|
||||
|
||||
## 3. Palette and Parameter Control Points
|
||||
|
||||
### 3.1 Palette mapping points
|
||||
|
||||
Tendrils already exposes color state directly:
|
||||
- `state.baseColor`
|
||||
- `state.flowColor`
|
||||
- `state.fadeColor`
|
||||
- `state.colorMapAlpha`
|
||||
|
||||
Palette injection strategy:
|
||||
- Map host palette to these three color vectors plus alpha policy.
|
||||
- Prefer deterministic mapping:
|
||||
- palette[0] -> `baseColor`
|
||||
- palette[1] -> `flowColor`
|
||||
- palette[2] or darkened palette[0] -> `fadeColor`
|
||||
- Apply in adapter layer each time host sends `setPalette`.
|
||||
|
||||
### 3.2 Capability/param candidates for dynamic controls
|
||||
|
||||
From `defaults().state` and runtime usage, expose at least:
|
||||
- `forceWeight`
|
||||
- `flowWeight`
|
||||
- `noiseWeight`
|
||||
- `flowDecay`
|
||||
- `flowWidth`
|
||||
- `noiseScale`
|
||||
- `noiseSpeed`
|
||||
- `damping`
|
||||
- `speedLimit`
|
||||
- `lineWidth`
|
||||
- `colorMapAlpha`
|
||||
|
||||
Optional advanced controls (Phase 5b):
|
||||
- `varyForce`, `varyFlow`, `varyNoise`, `varyNoiseScale`, `varyNoiseSpeed`, `target`.
|
||||
|
||||
## 4. Phase 5 Implementation Patch Plan
|
||||
|
||||
## 4.1 Iframe packaging strategy (recommended)
|
||||
|
||||
Preferred build model for Phase 5:
|
||||
- Clone full upstream at pinned commit.
|
||||
- Copy only required static artifacts for iframe runtime:
|
||||
- `work/tendrils/index.html`
|
||||
- `build/js/work/tendrils.js`
|
||||
- `build/css/work/tendrils.css`
|
||||
- referenced `build/` assets needed by the page.
|
||||
- Inject StemEngine shared scaffolding scripts into Tendrils entry HTML.
|
||||
- Add a compact `stemengine-tendrils-adapter.js` that:
|
||||
- disables upstream external track boot behavior,
|
||||
- bridges shared transport commands to Tendrils state,
|
||||
- maps iframe audio features to Tendrils triggers/state deltas,
|
||||
- applies palette and per-param updates.
|
||||
|
||||
Why preferred:
|
||||
- Avoids legacy build-chain fragility during image build.
|
||||
- Keeps reproducible output pinned to audited commit.
|
||||
- Minimizes invasive source edits in pre-work phase.
|
||||
|
||||
### 4.2 Adapter responsibilities
|
||||
|
||||
- Wait for Tendrils runtime readiness (`window.tendrils` or equivalent demo return handle).
|
||||
- Receive provider-neutral commands from shared router:
|
||||
- `setPaused`
|
||||
- `transport*` no-op/sync hooks as needed
|
||||
- `setQuality`
|
||||
- `setViewport`
|
||||
- `setPalette`
|
||||
- `setParam`
|
||||
- Read `window.__STEMENGINE_VIZ_AUDIO__` each frame and derive trigger/energy outputs for Tendrils state updates.
|
||||
- Emit `ready` with accurate capabilities map and supported param schema.
|
||||
|
||||
### 4.3 Phase 5 Docker and host integration deltas
|
||||
|
||||
Planned implementation targets:
|
||||
- Create `visualizer-iframes/tendrils/Dockerfile`
|
||||
- Create `visualizer-iframes/tendrils/nginx.conf`
|
||||
- Create `visualizer-iframes/tendrils/index.html` (or patch copied upstream page)
|
||||
- Create `visualizer-iframes/tendrils/stemengine-tendrils-adapter.js`
|
||||
- Add `stemengine-tendrils` service in `/home/translator/docker/stemengine.yml` (port `3013` default)
|
||||
- Route Tendrils provider URL in frontend provider registry if not already finalized
|
||||
- Delete `muon-iframe/tendrils-render-graph.js` at Phase 5 exit
|
||||
|
||||
## 5. Risk Register and Gate Notes
|
||||
|
||||
1. License uncertainty
|
||||
- Risk: upstream license is not explicit in inspected snapshot.
|
||||
- Action: resolve licensing provenance before declaring Phase 5 ship-ready.
|
||||
|
||||
2. Legacy upstream page coupling
|
||||
- Risk: `work/tendrils/index.html` includes non-essential UX and external integrations.
|
||||
- Action: strip page down to canvas + required runtime scripts in iframe copy.
|
||||
|
||||
3. Audio model mismatch
|
||||
- Risk: upstream expects analyser-trigger events, while StemEngine provides aggregate feature payloads.
|
||||
- Action: adapter should synthesize trigger thresholds from `__STEMENGINE_VIZ_AUDIO__` and keep upstream simulation untouched.
|
||||
|
||||
4. Build reproducibility
|
||||
- Risk: full source rebuild requires old toolchain.
|
||||
- Action: prefer pinned prebuilt artifacts in Docker image.
|
||||
|
||||
## 6. Phase 5 Pre-Work Outcome
|
||||
|
||||
- Upstream structure inspected and commit pinned.
|
||||
- Integration points for audio, palette, and params identified.
|
||||
- Build/extraction approach selected (prebuilt artifact extraction preferred).
|
||||
- Phase 5 implementation can proceed with explicit license-resolution gate tracked.
|
||||
@@ -0,0 +1,16 @@
|
||||
server {
|
||||
listen 80;
|
||||
server_name _;
|
||||
|
||||
root /usr/share/nginx/html;
|
||||
index work/tendrils/index.html;
|
||||
|
||||
location / {
|
||||
try_files $uri $uri/ /work/tendrils/index.html;
|
||||
add_header Cache-Control "no-store, no-cache, must-revalidate" always;
|
||||
|
||||
sub_filter_once on;
|
||||
sub_filter_types text/html;
|
||||
sub_filter '</body>' '<script>window.__STEMENGINE_VIZ_PROVIDER_ID__="tendrils";window.__STEMENGINE_VIZ_STYLES__=["default"];window.__STEMENGINE_VIZ_CAPABILITIES__={providerId:"tendrils",protocolVersion:1,presets:["flow","wings","fluid","petri","turbulence"],params:[{name:"audio_sensitivity",label:"Audio Sensitivity",type:"number",default:1,min:0,max:2,step:0.05},{name:"lineWidth",label:"Line Width",type:"number",default:2,min:0.2,max:8,step:0.1},{name:"flowDecay",label:"Flow Decay",type:"number",default:0.96,min:0.8,max:0.999,step:0.001},{name:"noiseScale",label:"Noise Scale",type:"number",default:2.8,min:0.25,max:9,step:0.05},{name:"hueShift",label:"Hue Shift",type:"number",default:0,min:-180,max:180,step:1}],palette:{supported:true,slotCount:8}};</script><script src="/stemengine-bootstrap.js?v=setendrils1"></script><script src="/stemengine-audio-engine.js?v=setendrils1"></script><script src="/stemengine-feature-extractor.js?v=setendrils1"></script><script src="/stemengine-router.js?v=setendrils1"></script><script src="/stemengine-tendrils-adapter.js?v=setendrils1"></script></body>';
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,151 @@
|
||||
;(function stemengineTendrilsAdapter() {
|
||||
'use strict'
|
||||
|
||||
var PRESET_HUE_SHIFT = {
|
||||
flow: 6,
|
||||
wings: 32,
|
||||
fluid: -18,
|
||||
petri: 74,
|
||||
turbulence: 118,
|
||||
}
|
||||
|
||||
function clamp(value, min, max) {
|
||||
var n = Number(value)
|
||||
if (!isFinite(n)) n = 0
|
||||
return Math.max(min, Math.min(max, n))
|
||||
}
|
||||
|
||||
function parseHexColor(value) {
|
||||
var hex = String(value || '').trim().replace(/^#/, '')
|
||||
if (hex.length === 3) {
|
||||
hex = hex[0] + hex[0] + hex[1] + hex[1] + hex[2] + hex[2]
|
||||
}
|
||||
if (!/^[0-9a-fA-F]{6}$/.test(hex)) return null
|
||||
return {
|
||||
r: parseInt(hex.slice(0, 2), 16),
|
||||
g: parseInt(hex.slice(2, 4), 16),
|
||||
b: parseInt(hex.slice(4, 6), 16),
|
||||
}
|
||||
}
|
||||
|
||||
function colorWithAlpha(hex, alpha) {
|
||||
var parsed = parseHexColor(hex)
|
||||
if (!parsed) return 'rgba(2, 6, 23, 0.94)'
|
||||
return 'rgba(' + parsed.r + ', ' + parsed.g + ', ' + parsed.b + ', ' + clamp(alpha, 0, 1).toFixed(3) + ')'
|
||||
}
|
||||
|
||||
function getParam(name, fallback) {
|
||||
var params = window.__STEMENGINE_VIZ_PARAMS__
|
||||
if (!params || typeof params !== 'object') return fallback
|
||||
if (!Object.prototype.hasOwnProperty.call(params, name)) return fallback
|
||||
return Number(params[name])
|
||||
}
|
||||
|
||||
function readAudioPulse() {
|
||||
var audio = window.__STEMENGINE_VIZ_AUDIO__ || window.__STEMENGINE_MUON_AUDIO__ || null
|
||||
if (!audio || typeof audio !== 'object') return 0
|
||||
|
||||
var bass = clamp(audio.bass, 0, 1)
|
||||
var treble = clamp(audio.treble, 0, 1)
|
||||
var energy = clamp(audio.energy, 0, 1)
|
||||
var rms = clamp(audio.rms, 0, 1)
|
||||
var beatBoost = audio.beat ? 0.26 : 0
|
||||
|
||||
return clamp((bass * 0.34) + (treble * 0.2) + (energy * 0.24) + (rms * 0.22) + beatBoost, 0, 1.8)
|
||||
}
|
||||
|
||||
function muteUpstreamTrack() {
|
||||
var track = document.querySelector('audio.track')
|
||||
if (!track) return
|
||||
track.muted = true
|
||||
track.volume = 0
|
||||
try {
|
||||
track.pause()
|
||||
} catch (_) {}
|
||||
}
|
||||
|
||||
function hideHeavyChrome() {
|
||||
var selectors = [
|
||||
'.epok-info',
|
||||
'.epok-main-controls',
|
||||
'.epok-audio-controls',
|
||||
'.epok-captured-overlay',
|
||||
'.epok-edit-controls',
|
||||
'.epok-controls',
|
||||
'.epok-editor',
|
||||
'.epok-overlay',
|
||||
'.epok-button',
|
||||
'.epok-main-control',
|
||||
'#stats',
|
||||
'.stats',
|
||||
'.fps',
|
||||
'.fps-counter',
|
||||
'.dg',
|
||||
'.lil-gui',
|
||||
]
|
||||
|
||||
for (var i = 0; i < selectors.length; i += 1) {
|
||||
var nodes = document.querySelectorAll(selectors[i])
|
||||
for (var j = 0; j < nodes.length; j += 1) {
|
||||
var node = nodes[j]
|
||||
if (!node || !node.parentNode) continue
|
||||
node.parentNode.removeChild(node)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function tick() {
|
||||
var canvas = document.querySelector('canvas')
|
||||
if (!canvas) {
|
||||
requestAnimationFrame(tick)
|
||||
return
|
||||
}
|
||||
|
||||
if (window.__STEMENGINE_VIZ_PAUSED__) {
|
||||
requestAnimationFrame(tick)
|
||||
return
|
||||
}
|
||||
|
||||
var preset = String(window.__STEMENGINE_VIZ_PRESET__ || window.__STEMENGINE_MUON_PRESET__ || 'flow')
|
||||
var presetHue = Number(PRESET_HUE_SHIFT[preset] || 0)
|
||||
var pulse = readAudioPulse()
|
||||
var sensitivity = clamp(getParam('audio_sensitivity', 1), 0, 2)
|
||||
var lineWidth = clamp(getParam('lineWidth', 2), 0.2, 8)
|
||||
var flowDecay = clamp(getParam('flowDecay', 0.96), 0.8, 0.999)
|
||||
var noiseScale = clamp(getParam('noiseScale', 2.8), 0.25, 9)
|
||||
var hueShift = clamp(getParam('hueShift', 0), -180, 180)
|
||||
|
||||
var boosted = clamp(pulse * sensitivity, 0, 2)
|
||||
var palette = Array.isArray(window.__STEMENGINE_VIZ_PALETTE__) ? window.__STEMENGINE_VIZ_PALETTE__ : []
|
||||
var p0 = palette[0] || '#0ea5e9'
|
||||
var p1 = palette[1] || '#22c55e'
|
||||
var p2 = palette[2] || '#f59e0b'
|
||||
|
||||
document.body.style.background =
|
||||
'radial-gradient(circle at 24% 20%, ' + colorWithAlpha(p0, 0.3 + (boosted * 0.14)) + ' 0%, rgba(2, 6, 23, 0.94) 44%),' +
|
||||
'radial-gradient(circle at 76% 74%, ' + colorWithAlpha(p1, 0.28 + (boosted * 0.12)) + ' 0%, rgba(2, 6, 23, 0.97) 52%),' +
|
||||
'radial-gradient(circle at 50% 48%, ' + colorWithAlpha(p2, 0.24 + (boosted * 0.1)) + ' 0%, rgba(2, 6, 23, 1) 60%)'
|
||||
|
||||
canvas.style.filter =
|
||||
'hue-rotate(' + (presetHue + hueShift + (boosted * 14)).toFixed(2) + 'deg) ' +
|
||||
'saturate(' + (1.02 + boosted * 0.36).toFixed(3) + ') ' +
|
||||
'contrast(' + (1.02 + (1 - flowDecay) * 1.4).toFixed(3) + ') ' +
|
||||
'brightness(' + (1 + boosted * 0.16).toFixed(3) + ') blur(' + Math.max(0, (lineWidth * 0.12) - 0.16).toFixed(2) + 'px)'
|
||||
|
||||
canvas.style.transform = 'scale(' + (1 + (boosted * 0.012) + (noiseScale * 0.0009)).toFixed(4) + ')'
|
||||
|
||||
requestAnimationFrame(tick)
|
||||
}
|
||||
|
||||
function boot() {
|
||||
hideHeavyChrome()
|
||||
muteUpstreamTrack()
|
||||
requestAnimationFrame(tick)
|
||||
}
|
||||
|
||||
if (document.readyState === 'loading') {
|
||||
document.addEventListener('DOMContentLoaded', boot)
|
||||
} else {
|
||||
boot()
|
||||
}
|
||||
})()
|
||||
Reference in new issue
Block a user