# Phase 2 Ready: Architecture & Dependency Map **Date**: 2026-04-17 **Status**: ✅ Phase 2.1 Infrastructure Complete **Next**: Implement 10 GLSL shaders --- ## Project Structure (Overlay Injection Pattern) ``` translator/ └── docker/ ├── karaokemugen-app/ (StemEngine frontend) │ ├── src/ │ │ ├── components/ │ │ │ └── VisualizerPreviewWidget.tsx [postMessage host] │ │ ├── api/ │ │ │ └── client.ts [muon_quality setting] │ │ └── pages/ │ │ └── VisualizerLabPage.tsx │ └── package.json │ ├── karaokemugen-server/ (StemEngine backend) │ ├── app/ │ │ ├── main.py [CSP header injection] │ │ └── routers/settings.py [muon_quality persistence] │ └── package.json │ ├── appdata/stemengine/ │ ├── muon-iframe/ (Muon visualization build) │ │ ├── Dockerfile [Overlay injection pipeline] │ │ ├── stemengine-bootstrap.js [WebGL detection] │ │ ├── stemengine-router.js [Style router + postMessage handler] │ │ ├── fluid-render-graph.js [FBO management + render loop] │ │ ├── README.md [Build pipeline docs] │ │ ├── PHASE2_SHADER_GUIDE.md [10 shader specifications] │ │ └── THIRD_PARTY_NOTICES.md [License inventory] │ │ │ └── muon-music-visualizer/ (Cloned at build time) │ └── (Muon source tree, untouched) │ └── stemengine.yml [Docker compose service] └── Phase2_Initiation_Report.md [This session's summary] ``` --- ## Data Flow: Audio → Fluid Rendering ### 1. Audio Feature Extraction (StemEngine Host) ```typescript // frontend/src/components/VisualizerPreviewWidget.tsx const MuonFeatureExtractor = { features: { bass: number, // [0..1] low frequencies treble: number, // [0..1] high frequencies rms: number, // [0..1] total energy energy: number, // [0..1] energy density beat: boolean, // percussion detected? frequencyBands: [] // [0..32] spectral breakdown }, // Called at RAF cadence during music playback extract() { ... } } ``` **Output**: Features computed per frame, ready for postMessage delivery. ### 2. postMessage Command (Host → Iframe) ```javascript // frontend/src/components/VisualizerPreviewWidget.tsx const sendMuonCommand = (type, payload) => { if (!muonIframeRef.current) return muonIframeRef.current.contentWindow.postMessage( { version: 1, type: type, // 'pushAudioFeatures', 'setStyle', etc. payload: payload }, targetOrigin // Security: origin checked on receiver ) } // Audio features dispatched at RAF cadence sendMuonCommand('pushAudioFeatures', { bass: 0.65, treble: 0.42, rms: 0.58, energy: 0.71, beat: true, frequencyBands: [...] }) ``` **Transport**: Cross-origin iframe communication with origin guard. ### 3. postMessage Reception (Iframe Router) ```javascript // muon-iframe/stemengine-router.js window.addEventListener('message', (event) => { // Origin guard (security layer 1) if (event.origin !== ALLOWED_ORIGIN) return const { version, type, payload } = event.data if (version !== 1) return // Audio rate-limiting (dual-RAF gate) if (type === 'pushAudioFeatures') { captureFeatures(payload) // Capture into pending if (!rafPending) { rafPending = true requestAnimationFrame(() => { rafPending = false flushFeatures() // Flush to global }) } } }) // Global audio state (read by render graph) window.__STEMENGINE_MUON_AUDIO__ = { bass: 0.65, treble: 0.42, rms: 0.58, energy: 0.71, beat: true, frequencyBands: [...] } ``` **Buffering**: Dual-RAF gate prevents render loop blocking. ### 4. Render Graph Instantiation (Style Router) ```javascript // muon-iframe/stemengine-router.js const handleCommand = (type, payload) => { if (type === 'setStyle') { window.__STEMENGINE_MUON_STYLE__ = payload.style updateRenderGraph(payload.style) } } const updateRenderGraph = (style) => { if (style === 'fluid' && window.__StemEngineFluidRenderGraph__) { currentRenderGraph = new window.__StemEngineFluidRenderGraph__() } } // Query param auto-initialization window.__STEMENGINE_MUON_STYLE__ = getStyleParam() // ?style=fluid ``` **Trigger**: `setStyle` postMessage or query param ?style=fluid. ### 5. Fluid Simulator Runtime Loop (Canvas) ```javascript // muon-iframe/fluid-render-graph.js class FluidSimulator { constructor() { this.gl = acquireWebGLContext() this.allocateFBOs() this.compileShaders() // (TODO: implement all 10 shaders) this.startRenderLoop() } startRenderLoop() { const loop = () => { const audio = window.__STEMENGINE_MUON_AUDIO__ if (audio && !window.__STEMENGINE_MUON_PAUSED__) { this.step(audio) // Main simulation step } requestAnimationFrame(loop) } loop() } step(audioFeatures) { // Execution order (must be sequential): this.splat(audioFeatures) // Force injection this.curl() // Vorticity this.divergence() // Incompressibility metric // Pressure solve (N iterations based on quality tier) for (let i = 0; i < this.pressureIterations; i++) { this.pressureJacobi() } this.gradientSubtract() // Enforce incompressibility this.advectVelocity() // Self-advection + damping this.advectDye(audioFeatures) // Color transport + audio injection // Bloom (balanced/high only) if (this.enableBloom) { this.bloomDownsample() this.bloomComposite(audioFeatures) } this.composite() // Final render to canvas } } ``` **Dispatch**: Each method is a render pass (fragment shader + FBO binding). ### 6. Canvas Output ```javascript // muon-iframe/fluid-render-graph.js composite() { // Bind canvas framebuffer (default: null) this.gl.bindFramebuffer(this.gl.FRAMEBUFFER, null) // Use composite shader program this.gl.useProgram(this.compositeProgram) // Bind dye texture as input this.gl.uniform1i(this.compositeUniforms.u_dye, 0) this.gl.activeTexture(this.gl.TEXTURE0) this.gl.bindTexture(this.gl.TEXTURE_2D, this.dye[0].texture) // Render fullscreen quad to canvas this.gl.drawElements(this.gl.TRIANGLES, 6, this.gl.UNSIGNED_SHORT, 0) } ``` **Output**: Fluid colors rendered to canvas in real-time. --- ## Shader Dependencies & Execution Graph ``` ┌──────────────────────────────────────────┐ │ Input: Audio Features + Query Params │ │ (bass, treble, rms, beat, style, quality)│ └────────────────┬─────────────────────────┘ │ ┌───────▼────────┐ │ Render Graph │ │ Instantiation │ │ (by style) │ └───────┬────────┘ │ ┌────────▼─────────────────────┐ │ FluidSimulator.step() │ │ (Main simulation loop) │ └────────┬─────────────────────┘ │ ┌────────────┼────────────────────────────────────┐ │ │ │ │ ┌──────▼──────┐ ┌──────▼──────┐ │ │ Splat │ │ (Quality) │ │ │ (force inj.) │ │ Tier Logic │ │ └──────┬───────┘ └──────┬──────┘ │ │ │ │ ┌──────▼──────┐ Low: 10 iter │ │ Curl │ Balanced: 20 │ │ (vorticity) │ High: 40 │ └──────┬──────┘ + Bloom │ │ │ ┌──────▼──────────┐ │ │ Divergence │ │ │ (incompress.) │ │ └──────┬──────────┘ │ │ │ ┌──────▼────────────────┐ │ │ Pressure Jacobi (loop)│ ← iterations based on tier │ └──────┬────────────────┘ │ │ │ ┌──────▼─────────────┐ │ │ Gradient Subtract │ │ │ (enforce incomp.) │ │ └──────┬─────────────┘ │ │ │ ┌──────▼──────────┐ │ │ Advect Velocity │ │ └──────┬──────────┘ │ │ │ ┌──────▼──────────┐ │ │ Advect Dye │ │ │ (audio inject) │ │ └──────┬──────────┘ │ │ │ ┌──────▼─────────┐ │ │ Bloom? │ ← conditional on tier │ │ (if balanced/ │ │ │ high) │ │ └──────┬─────────┘ │ │ │ ┌──────▼──────┐ │ │ Composite │ │ │ (to canvas) │ │ └─────────────┘ │ └────────────────────────────────────────┘ │ ┌──────▼────────┐ │ Canvas Output │ │ (visible dye) │ └────────────────┘ ``` --- ## Texture Memory Layout ### Ping-Pong Framebuffer Objects | Name | Format | Size (low) | Size (balanced) | Size (high) | Purpose | |------|--------|-----------|-----------------|------------|---------| | `velocity[0]` | RGBA Float32 | 256×256×16B | 512×512×16B | 1024×1024×16B | Velocity X,Y + divergence | | `velocity[1]` | RGBA Float32 | 256×256×16B | 512×512×16B | 1024×1024×16B | Ping-pong pair | | `dye[0]` | RGBA Float32 | 256×256×16B | 512×512×16B | 1024×1024×16B | Color field | | `dye[1]` | RGBA Float32 | 256×256×16B | 512×512×16B | 1024×1024×16B | Ping-pong pair | | `divergence[0]` | R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Velocity divergence | | `divergence[1]` | R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Ping-pong pair | | `pressure[0]` | R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Pressure field | | `pressure[1]` | R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Ping-pong pair | | `curl[0]` | R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Vorticity | | `curl[1]` | R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Ping-pong pair | **Bloom Chain** (balanced/high only): | Name | Format | Size (balanced) | Size (high) | |------|--------|-----------------|------------| | `bloomDown1` | RGBA Float32 | 256×256×16B | 512×512×16B | | `bloomDown2` | RGBA Float32 | 128×128×16B | 256×256×16B | | `bloomUp1` | RGBA Float32 | 256×256×16B | 512×512×16B | **Total Memory (balanced tier)**: - Main FBOs: 5 fields × 2 (ping-pong) × 16B = 160B per texel - At 512×512: 512×512 × 160B = **41.9 MB** (main) - Bloom: 256×256×16 + 128×128×16 + 256×256×16 = **16.8 MB** - **Total: ~60 MB** (typical, manageable on modern hardware) --- ## Shader Program Organization ### Single WebGL Program (Recommended) ```glsl // All 10 fragment shaders combined into one large program: // - Compile one massive fragment shader with 10 `#if PASS_TYPE` branches // - Reuse same vertex shader for all passes // - Problem: Large shader binary, harder to debug // Pros: Fewer state changes, simpler linking // Cons: Less modular, harder to update individual passes ``` ### Multiple WebGL Programs (Recommended) ```glsl // Separate program for each render pass: // Program 0: splat // Program 1: curl // Program 2: divergence // ... // Program 9: composite // Pros: Modular, easier to debug, flexible to update // Cons: More state changes per frame (~10 useProgram calls + uniform binding) ``` **Decision**: Multiple programs for maintainability (phase 2 focused on correctness, not performance). --- ## Build Pipeline: Overlay Injection ### Dockerfile Sequence (muon-iframe/Dockerfile) ```dockerfile # 1. Clone Muon repo RUN git clone --depth 1 -b ${MUON_GIT_REF} \ https://github.com/najafmohammed/muon-music-visualizer /src/app # 2. Install dependencies WORKDIR /src/app RUN npm install # 3. Copy overlay files COPY overlayless.css /tmp/overlayless.css COPY stemengine-bootstrap.js /tmp/stemengine-bootstrap.js COPY stemengine-router.js /tmp/stemengine-router.js COPY fluid-render-graph.js /tmp/fluid-render-graph.js # NEW (Phase 2) # 4. Merge CSS RUN cat /tmp/overlayless.css >> /src/app/src/style.css # 5. Inject JavaScript overlays RUN cp /tmp/stemengine-bootstrap.js /src/app/src/ RUN grep -q "stemengine-bootstrap" /src/app/src/index.js || \ sed -i '1i import "./stemengine-bootstrap";' /src/app/src/index.js RUN cp /tmp/stemengine-router.js /src/app/src/ RUN sed -i "s|__STEMENGINE_ALLOWED_ORIGIN__|${STEMENGINE_ALLOWED_ORIGIN}|g" \ /src/app/src/stemengine-router.js RUN grep -q "stemengine-router" /src/app/src/index.js || \ sed -i '1i import "./stemengine-router";' /src/app/src/index.js RUN cp /tmp/fluid-render-graph.js /src/app/src/ # NEW RUN grep -q "fluid-render-graph" /src/app/src/index.js || \ # NEW sed -i '1i import "./fluid-render-graph";' /src/app/src/index.js # NEW # 6. Build Muon RUN npm run build # 7. Serve from image FROM node:20-alpine COPY --from=builder /src/app/dist /app EXPOSE 3011 CMD ["node", "server.js"] ``` **Injection Order**: Imports are added in FIFO order, so execution order is: 1. `stemengine-bootstrap` (WebGL detection) 2. `stemengine-router` (postMessage listener, style routing) 3. `fluid-render-graph` (class definition, global export) --- ## Security & Compliance Checklist ✅ **CSP Headers** - Injected by backend: `frame-src 'self' ` - Runtime check: `STEMENGINE_ALLOWED_ORIGIN` build arg ✅ **postMessage Origin Guard** - Double-check: CSP header + postMessage event.origin validation - Token replacement: Build-time constant, no runtime parsing ✅ **Sandbox Attributes** - Minimal: `allow-scripts allow-same-origin` (no allow-downloads, no allow-top-navigation) - Referrer policy: `no-referrer` ✅ **Licensing** - All dependencies MIT licensed (Muon, WebGL-Fluid, The Spirit) - THIRD_PARTY_NOTICES.md + compliance sign-off in place ✅ **Error Handling** - `window.__stemengineEmitError__()` callback for render graph errors - WebGL context loss detection + recovery stubs --- ## Next Actions (Priority Order) ### 🔴 Priority 1: Implement Splat Shader **Why first?**: Enables visual feedback immediately; audio-driven force injection. **Deliverable**: `FluidSimulator.prototype.splat()` implementation. **Steps**: 1. Create splat vertex shader (simple passthrough) 2. Create splat fragment shader (gaussian kernel + force mapping) 3. Compile program, bind uniforms (u_velocity, u_splatPos, u_splatRadius, u_splatForce, u_turbulenceAmplitude) 4. Render fullscreen quad to `velocity[1]` FBO 5. Swap: `velocity = [velocity[1], velocity[0]]` ### 🟡 Priority 2: Implement Curl + Divergence **Why second?**: Required for pressure solver; simpler than splat. **Deliverables**: `curl()`, `divergence()` methods. ### 🟡 Priority 3: Implement Pressure Jacobi + Gradient Subtract **Why third?**: Core incompressible fluid dynamics; enables realistic motion. **Deliverables**: `pressureJacobi()`, `gradientSubtract()` methods. ### 🟢 Priority 4: Implement Advection (Velocity + Dye) **Why fourth?**: Smooth motion transport; audio-driven coloring. **Deliverables**: `advectVelocity()`, `advectDye()` methods. ### 🟢 Priority 5: Implement Bloom + Composite **Why last?**: Optional on low tier; visual polish on balanced/high. **Deliverables**: `bloomDownsample()`, `bloomComposite()`, `composite()` methods. --- ## Validation & Testing ### Unit-Level Validation ```bash # Syntax check node -c /home/translator/docker/appdata/stemengine/muon-iframe/fluid-render-graph.js # WebGL capability detection (run in browser console) window.__STEMENGINE_MUON_WEBGL_BASELINE__ # Should show {ok: true, missing: []} # FBO allocation new window.__StemEngineFluidRenderGraph__() # Should not throw ``` ### Integration-Level Validation ```bash # Build Docker image docker build \ --build-arg MUON_GIT_REF=main \ --build-arg STEMENGINE_ALLOWED_ORIGIN=http://localhost:5173 \ -t stemengine-muon:latest \ docker/appdata/stemengine/muon-iframe # Start container docker run -p 3011:3011 stemengine-muon:latest # Open in browser # http://localhost:3011?style=fluid # Listen to music, verify: # - Fluid animation visible # - Bass drives splat spread # - Beat creates impulses # - Treble adds turbulence # - RMS controls bloom intensity ``` --- ## Documentation References | Document | Purpose | Location | |----------|---------|----------| | **PHASE2_SHADER_GUIDE.md** | Detailed shader pseudocode & algorithms | `muon-iframe/` | | **Phase2_Initiation_Report.md** | This session's decisions & status | `appdata/stemengine/` | | **README.md** (muon-iframe) | Build pipeline, command contract, audio policy | `muon-iframe/` | | **PROJECT_STATUS_BOARD.md** | Executive summary of all phases | `docker/` | --- ## Appendix: Known Limitations & Future Work ### Phase 2 (Current) - ✅ Incompressible Euler flow (core fluid dynamics) - ❌ Vortex confinement (better preservation of small-scale features) - ❌ Buoyancy forcing (smoke-like effects) - ❌ Obstacle handling (boundaries within domain) ### Phase 3 (Tendrils) - Particle-based visualization - Spectral decomposition mapping - Directional line rendering ### Phase 4 (Spirit) - Geometric mesh deformations - Breathing/morphing animations - Layered composite rendering ### Post-Phase 4 (Future) - Mobile optimization (adaptive resolution, dynamic quality switching) - Multi-canvas support (split-screen, gallery mode) - Recording + export (GIF, video, SVG) - Preset system (save/load favorite configurations) --- **Status**: ✅ Ready to begin shader implementation.