19 KiB
Phase 2 Ready: Architecture & Dependency Map
Date: 2026-04-17
Status: ✅ Phase 2.1 Infrastructure Complete
Next: Implement 10 GLSL shaders
Project Structure (Overlay Injection Pattern)
translator/
└── docker/
├── karaokemugen-app/ (StemEngine frontend)
│ ├── src/
│ │ ├── components/
│ │ │ └── VisualizerPreviewWidget.tsx [postMessage host]
│ │ ├── api/
│ │ │ └── client.ts [muon_quality setting]
│ │ └── pages/
│ │ └── VisualizerLabPage.tsx
│ └── package.json
│
├── karaokemugen-server/ (StemEngine backend)
│ ├── app/
│ │ ├── main.py [CSP header injection]
│ │ └── routers/settings.py [muon_quality persistence]
│ └── package.json
│
├── appdata/stemengine/
│ ├── muon-iframe/ (Muon visualization build)
│ │ ├── Dockerfile [Overlay injection pipeline]
│ │ ├── stemengine-bootstrap.js [WebGL detection]
│ │ ├── stemengine-router.js [Style router + postMessage handler]
│ │ ├── fluid-render-graph.js [FBO management + render loop]
│ │ ├── README.md [Build pipeline docs]
│ │ ├── PHASE2_SHADER_GUIDE.md [10 shader specifications]
│ │ └── THIRD_PARTY_NOTICES.md [License inventory]
│ │
│ └── muon-music-visualizer/ (Cloned at build time)
│ └── (Muon source tree, untouched)
│
└── stemengine.yml [Docker compose service]
└── Phase2_Initiation_Report.md [This session's summary]
Data Flow: Audio → Fluid Rendering
1. Audio Feature Extraction (StemEngine Host)
// frontend/src/components/VisualizerPreviewWidget.tsx
const MuonFeatureExtractor = {
features: {
bass: number, // [0..1] low frequencies
treble: number, // [0..1] high frequencies
rms: number, // [0..1] total energy
energy: number, // [0..1] energy density
beat: boolean, // percussion detected?
frequencyBands: [] // [0..32] spectral breakdown
},
// Called at RAF cadence during music playback
extract() { ... }
}
Output: Features computed per frame, ready for postMessage delivery.
2. postMessage Command (Host → Iframe)
// frontend/src/components/VisualizerPreviewWidget.tsx
const sendMuonCommand = (type, payload) => {
if (!muonIframeRef.current) return
muonIframeRef.current.contentWindow.postMessage(
{
version: 1,
type: type, // 'pushAudioFeatures', 'setStyle', etc.
payload: payload
},
targetOrigin // Security: origin checked on receiver
)
}
// Audio features dispatched at RAF cadence
sendMuonCommand('pushAudioFeatures', {
bass: 0.65,
treble: 0.42,
rms: 0.58,
energy: 0.71,
beat: true,
frequencyBands: [...]
})
Transport: Cross-origin iframe communication with origin guard.
3. postMessage Reception (Iframe Router)
// muon-iframe/stemengine-router.js
window.addEventListener('message', (event) => {
// Origin guard (security layer 1)
if (event.origin !== ALLOWED_ORIGIN) return
const { version, type, payload } = event.data
if (version !== 1) return
// Audio rate-limiting (dual-RAF gate)
if (type === 'pushAudioFeatures') {
captureFeatures(payload) // Capture into pending
if (!rafPending) {
rafPending = true
requestAnimationFrame(() => {
rafPending = false
flushFeatures() // Flush to global
})
}
}
})
// Global audio state (read by render graph)
window.__STEMENGINE_MUON_AUDIO__ = {
bass: 0.65,
treble: 0.42,
rms: 0.58,
energy: 0.71,
beat: true,
frequencyBands: [...]
}
Buffering: Dual-RAF gate prevents render loop blocking.
4. Render Graph Instantiation (Style Router)
// muon-iframe/stemengine-router.js
const handleCommand = (type, payload) => {
if (type === 'setStyle') {
window.__STEMENGINE_MUON_STYLE__ = payload.style
updateRenderGraph(payload.style)
}
}
const updateRenderGraph = (style) => {
if (style === 'fluid' && window.__StemEngineFluidRenderGraph__) {
currentRenderGraph = new window.__StemEngineFluidRenderGraph__()
}
}
// Query param auto-initialization
window.__STEMENGINE_MUON_STYLE__ = getStyleParam() // ?style=fluid
Trigger: setStyle postMessage or query param ?style=fluid.
5. Fluid Simulator Runtime Loop (Canvas)
// muon-iframe/fluid-render-graph.js
class FluidSimulator {
constructor() {
this.gl = acquireWebGLContext()
this.allocateFBOs()
this.compileShaders() // (TODO: implement all 10 shaders)
this.startRenderLoop()
}
startRenderLoop() {
const loop = () => {
const audio = window.__STEMENGINE_MUON_AUDIO__
if (audio && !window.__STEMENGINE_MUON_PAUSED__) {
this.step(audio) // Main simulation step
}
requestAnimationFrame(loop)
}
loop()
}
step(audioFeatures) {
// Execution order (must be sequential):
this.splat(audioFeatures) // Force injection
this.curl() // Vorticity
this.divergence() // Incompressibility metric
// Pressure solve (N iterations based on quality tier)
for (let i = 0; i < this.pressureIterations; i++) {
this.pressureJacobi()
}
this.gradientSubtract() // Enforce incompressibility
this.advectVelocity() // Self-advection + damping
this.advectDye(audioFeatures) // Color transport + audio injection
// Bloom (balanced/high only)
if (this.enableBloom) {
this.bloomDownsample()
this.bloomComposite(audioFeatures)
}
this.composite() // Final render to canvas
}
}
Dispatch: Each method is a render pass (fragment shader + FBO binding).
6. Canvas Output
// muon-iframe/fluid-render-graph.js
composite() {
// Bind canvas framebuffer (default: null)
this.gl.bindFramebuffer(this.gl.FRAMEBUFFER, null)
// Use composite shader program
this.gl.useProgram(this.compositeProgram)
// Bind dye texture as input
this.gl.uniform1i(this.compositeUniforms.u_dye, 0)
this.gl.activeTexture(this.gl.TEXTURE0)
this.gl.bindTexture(this.gl.TEXTURE_2D, this.dye[0].texture)
// Render fullscreen quad to canvas
this.gl.drawElements(this.gl.TRIANGLES, 6, this.gl.UNSIGNED_SHORT, 0)
}
Output: Fluid colors rendered to canvas in real-time.
Shader Dependencies & Execution Graph
┌──────────────────────────────────────────┐
│ Input: Audio Features + Query Params │
│ (bass, treble, rms, beat, style, quality)│
└────────────────┬─────────────────────────┘
│
┌───────▼────────┐
│ Render Graph │
│ Instantiation │
│ (by style) │
└───────┬────────┘
│
┌────────▼─────────────────────┐
│ FluidSimulator.step() │
│ (Main simulation loop) │
└────────┬─────────────────────┘
│
┌────────────┼────────────────────────────────────┐
│ │ │
│ ┌──────▼──────┐ ┌──────▼──────┐
│ │ Splat │ │ (Quality) │
│ │ (force inj.) │ │ Tier Logic │
│ └──────┬───────┘ └──────┬──────┘
│ │ │
│ ┌──────▼──────┐ Low: 10 iter
│ │ Curl │ Balanced: 20
│ │ (vorticity) │ High: 40
│ └──────┬──────┘ + Bloom
│ │
│ ┌──────▼──────────┐
│ │ Divergence │
│ │ (incompress.) │
│ └──────┬──────────┘
│ │
│ ┌──────▼────────────────┐
│ │ Pressure Jacobi (loop)│ ← iterations based on tier
│ └──────┬────────────────┘
│ │
│ ┌──────▼─────────────┐
│ │ Gradient Subtract │
│ │ (enforce incomp.) │
│ └──────┬─────────────┘
│ │
│ ┌──────▼──────────┐
│ │ Advect Velocity │
│ └──────┬──────────┘
│ │
│ ┌──────▼──────────┐
│ │ Advect Dye │
│ │ (audio inject) │
│ └──────┬──────────┘
│ │
│ ┌──────▼─────────┐
│ │ Bloom? │ ← conditional on tier
│ │ (if balanced/ │
│ │ high) │
│ └──────┬─────────┘
│ │
│ ┌──────▼──────┐
│ │ Composite │
│ │ (to canvas) │
│ └─────────────┘
│
└────────────────────────────────────────┘
│
┌──────▼────────┐
│ Canvas Output │
│ (visible dye) │
└────────────────┘
Texture Memory Layout
Ping-Pong Framebuffer Objects
| Name | Format | Size (low) | Size (balanced) | Size (high) | Purpose |
|---|---|---|---|---|---|
velocity[0] |
RGBA Float32 | 256×256×16B | 512×512×16B | 1024×1024×16B | Velocity X,Y + divergence |
velocity[1] |
RGBA Float32 | 256×256×16B | 512×512×16B | 1024×1024×16B | Ping-pong pair |
dye[0] |
RGBA Float32 | 256×256×16B | 512×512×16B | 1024×1024×16B | Color field |
dye[1] |
RGBA Float32 | 256×256×16B | 512×512×16B | 1024×1024×16B | Ping-pong pair |
divergence[0] |
R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Velocity divergence |
divergence[1] |
R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Ping-pong pair |
pressure[0] |
R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Pressure field |
pressure[1] |
R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Ping-pong pair |
curl[0] |
R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Vorticity |
curl[1] |
R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Ping-pong pair |
Bloom Chain (balanced/high only):
| Name | Format | Size (balanced) | Size (high) |
|---|---|---|---|
bloomDown1 |
RGBA Float32 | 256×256×16B | 512×512×16B |
bloomDown2 |
RGBA Float32 | 128×128×16B | 256×256×16B |
bloomUp1 |
RGBA Float32 | 256×256×16B | 512×512×16B |
Total Memory (balanced tier):
- Main FBOs: 5 fields × 2 (ping-pong) × 16B = 160B per texel
- At 512×512: 512×512 × 160B = 41.9 MB (main)
- Bloom: 256×256×16 + 128×128×16 + 256×256×16 = 16.8 MB
- Total: ~60 MB (typical, manageable on modern hardware)
Shader Program Organization
Single WebGL Program (Recommended)
// All 10 fragment shaders combined into one large program:
// - Compile one massive fragment shader with 10 `#if PASS_TYPE` branches
// - Reuse same vertex shader for all passes
// - Problem: Large shader binary, harder to debug
// Pros: Fewer state changes, simpler linking
// Cons: Less modular, harder to update individual passes
Multiple WebGL Programs (Recommended)
// Separate program for each render pass:
// Program 0: splat
// Program 1: curl
// Program 2: divergence
// ...
// Program 9: composite
// Pros: Modular, easier to debug, flexible to update
// Cons: More state changes per frame (~10 useProgram calls + uniform binding)
Decision: Multiple programs for maintainability (phase 2 focused on correctness, not performance).
Build Pipeline: Overlay Injection
Dockerfile Sequence (muon-iframe/Dockerfile)
# 1. Clone Muon repo
RUN git clone --depth 1 -b ${MUON_GIT_REF} \
https://github.com/najafmohammed/muon-music-visualizer /src/app
# 2. Install dependencies
WORKDIR /src/app
RUN npm install
# 3. Copy overlay files
COPY overlayless.css /tmp/overlayless.css
COPY stemengine-bootstrap.js /tmp/stemengine-bootstrap.js
COPY stemengine-router.js /tmp/stemengine-router.js
COPY fluid-render-graph.js /tmp/fluid-render-graph.js # NEW (Phase 2)
# 4. Merge CSS
RUN cat /tmp/overlayless.css >> /src/app/src/style.css
# 5. Inject JavaScript overlays
RUN cp /tmp/stemengine-bootstrap.js /src/app/src/
RUN grep -q "stemengine-bootstrap" /src/app/src/index.js || \
sed -i '1i import "./stemengine-bootstrap";' /src/app/src/index.js
RUN cp /tmp/stemengine-router.js /src/app/src/
RUN sed -i "s|__STEMENGINE_ALLOWED_ORIGIN__|${STEMENGINE_ALLOWED_ORIGIN}|g" \
/src/app/src/stemengine-router.js
RUN grep -q "stemengine-router" /src/app/src/index.js || \
sed -i '1i import "./stemengine-router";' /src/app/src/index.js
RUN cp /tmp/fluid-render-graph.js /src/app/src/ # NEW
RUN grep -q "fluid-render-graph" /src/app/src/index.js || \ # NEW
sed -i '1i import "./fluid-render-graph";' /src/app/src/index.js # NEW
# 6. Build Muon
RUN npm run build
# 7. Serve from image
FROM node:20-alpine
COPY --from=builder /src/app/dist /app
EXPOSE 3011
CMD ["node", "server.js"]
Injection Order: Imports are added in FIFO order, so execution order is:
stemengine-bootstrap(WebGL detection)stemengine-router(postMessage listener, style routing)fluid-render-graph(class definition, global export)
Security & Compliance Checklist
✅ CSP Headers
- Injected by backend:
frame-src 'self' <MUON_ALLOWED_ORIGIN> - Runtime check:
STEMENGINE_ALLOWED_ORIGINbuild arg
✅ postMessage Origin Guard
- Double-check: CSP header + postMessage event.origin validation
- Token replacement: Build-time constant, no runtime parsing
✅ Sandbox Attributes
- Minimal:
allow-scripts allow-same-origin(no allow-downloads, no allow-top-navigation) - Referrer policy:
no-referrer
✅ Licensing
- All dependencies MIT licensed (Muon, WebGL-Fluid, The Spirit)
- THIRD_PARTY_NOTICES.md + compliance sign-off in place
✅ Error Handling
window.__stemengineEmitError__()callback for render graph errors- WebGL context loss detection + recovery stubs
Next Actions (Priority Order)
🔴 Priority 1: Implement Splat Shader
Why first?: Enables visual feedback immediately; audio-driven force injection.
Deliverable: FluidSimulator.prototype.splat() implementation.
Steps:
- Create splat vertex shader (simple passthrough)
- Create splat fragment shader (gaussian kernel + force mapping)
- Compile program, bind uniforms (u_velocity, u_splatPos, u_splatRadius, u_splatForce, u_turbulenceAmplitude)
- Render fullscreen quad to
velocity[1]FBO - Swap:
velocity = [velocity[1], velocity[0]]
🟡 Priority 2: Implement Curl + Divergence
Why second?: Required for pressure solver; simpler than splat.
Deliverables: curl(), divergence() methods.
🟡 Priority 3: Implement Pressure Jacobi + Gradient Subtract
Why third?: Core incompressible fluid dynamics; enables realistic motion.
Deliverables: pressureJacobi(), gradientSubtract() methods.
🟢 Priority 4: Implement Advection (Velocity + Dye)
Why fourth?: Smooth motion transport; audio-driven coloring.
Deliverables: advectVelocity(), advectDye() methods.
🟢 Priority 5: Implement Bloom + Composite
Why last?: Optional on low tier; visual polish on balanced/high.
Deliverables: bloomDownsample(), bloomComposite(), composite() methods.
Validation & Testing
Unit-Level Validation
# Syntax check
node -c /home/translator/docker/appdata/stemengine/muon-iframe/fluid-render-graph.js
# WebGL capability detection (run in browser console)
window.__STEMENGINE_MUON_WEBGL_BASELINE__ # Should show {ok: true, missing: []}
# FBO allocation
new window.__StemEngineFluidRenderGraph__() # Should not throw
Integration-Level Validation
# Build Docker image
docker build \
--build-arg MUON_GIT_REF=main \
--build-arg STEMENGINE_ALLOWED_ORIGIN=http://localhost:5173 \
-t stemengine-muon:latest \
docker/appdata/stemengine/muon-iframe
# Start container
docker run -p 3011:3011 stemengine-muon:latest
# Open in browser
# http://localhost:3011?style=fluid
# Listen to music, verify:
# - Fluid animation visible
# - Bass drives splat spread
# - Beat creates impulses
# - Treble adds turbulence
# - RMS controls bloom intensity
Documentation References
| Document | Purpose | Location |
|---|---|---|
| PHASE2_SHADER_GUIDE.md | Detailed shader pseudocode & algorithms | muon-iframe/ |
| Phase2_Initiation_Report.md | This session's decisions & status | appdata/stemengine/ |
| README.md (muon-iframe) | Build pipeline, command contract, audio policy | muon-iframe/ |
| PROJECT_STATUS_BOARD.md | Executive summary of all phases | docker/ |
Appendix: Known Limitations & Future Work
Phase 2 (Current)
- ✅ Incompressible Euler flow (core fluid dynamics)
- ❌ Vortex confinement (better preservation of small-scale features)
- ❌ Buoyancy forcing (smoke-like effects)
- ❌ Obstacle handling (boundaries within domain)
Phase 3 (Tendrils)
- Particle-based visualization
- Spectral decomposition mapping
- Directional line rendering
Phase 4 (Spirit)
- Geometric mesh deformations
- Breathing/morphing animations
- Layered composite rendering
Post-Phase 4 (Future)
- Mobile optimization (adaptive resolution, dynamic quality switching)
- Multi-canvas support (split-screen, gallery mode)
- Recording + export (GIF, video, SVG)
- Preset system (save/load favorite configurations)
Status: ✅ Ready to begin shader implementation.