Files
stemengine_old/Phase2_Architecture_And_Dependencies.md
2026-04-21 16:24:11 +09:00

19 KiB
Raw Permalink Blame History

Phase 2 Ready: Architecture & Dependency Map

Date: 2026-04-17
Status: ✅ Phase 2.1 Infrastructure Complete
Next: Implement 10 GLSL shaders


Project Structure (Overlay Injection Pattern)

translator/
└── docker/
    ├── karaokemugen-app/              (StemEngine frontend)
    │   ├── src/
    │   │   ├── components/
    │   │   │   └── VisualizerPreviewWidget.tsx  [postMessage host]
    │   │   ├── api/
    │   │   │   └── client.ts           [muon_quality setting]
    │   │   └── pages/
    │   │       └── VisualizerLabPage.tsx
    │   └── package.json
    │
    ├── karaokemugen-server/           (StemEngine backend)
    │   ├── app/
    │   │   ├── main.py                 [CSP header injection]
    │   │   └── routers/settings.py     [muon_quality persistence]
    │   └── package.json
    │
    ├── appdata/stemengine/
    │   ├── muon-iframe/                (Muon visualization build)
    │   │   ├── Dockerfile              [Overlay injection pipeline]
    │   │   ├── stemengine-bootstrap.js [WebGL detection]
    │   │   ├── stemengine-router.js    [Style router + postMessage handler]
    │   │   ├── fluid-render-graph.js   [FBO management + render loop]
    │   │   ├── README.md               [Build pipeline docs]
    │   │   ├── PHASE2_SHADER_GUIDE.md  [10 shader specifications]
    │   │   └── THIRD_PARTY_NOTICES.md  [License inventory]
    │   │
    │   └── muon-music-visualizer/      (Cloned at build time)
    │       └── (Muon source tree, untouched)
    │
    └── stemengine.yml                  [Docker compose service]
    └── Phase2_Initiation_Report.md     [This session's summary]

Data Flow: Audio → Fluid Rendering

1. Audio Feature Extraction (StemEngine Host)

// frontend/src/components/VisualizerPreviewWidget.tsx
const MuonFeatureExtractor = {
  features: {
    bass: number,         // [0..1] low frequencies
    treble: number,       // [0..1] high frequencies
    rms: number,          // [0..1] total energy
    energy: number,       // [0..1] energy density
    beat: boolean,        // percussion detected?
    frequencyBands: []    // [0..32] spectral breakdown
  },
  
  // Called at RAF cadence during music playback
  extract() { ... }
}

Output: Features computed per frame, ready for postMessage delivery.

2. postMessage Command (Host → Iframe)

// frontend/src/components/VisualizerPreviewWidget.tsx
const sendMuonCommand = (type, payload) => {
  if (!muonIframeRef.current) return
  muonIframeRef.current.contentWindow.postMessage(
    {
      version: 1,
      type: type,      // 'pushAudioFeatures', 'setStyle', etc.
      payload: payload
    },
    targetOrigin      // Security: origin checked on receiver
  )
}

// Audio features dispatched at RAF cadence
sendMuonCommand('pushAudioFeatures', {
  bass: 0.65,
  treble: 0.42,
  rms: 0.58,
  energy: 0.71,
  beat: true,
  frequencyBands: [...]
})

Transport: Cross-origin iframe communication with origin guard.

3. postMessage Reception (Iframe Router)

// muon-iframe/stemengine-router.js
window.addEventListener('message', (event) => {
  // Origin guard (security layer 1)
  if (event.origin !== ALLOWED_ORIGIN) return
  
  const { version, type, payload } = event.data
  if (version !== 1) return
  
  // Audio rate-limiting (dual-RAF gate)
  if (type === 'pushAudioFeatures') {
    captureFeatures(payload)       // Capture into pending
    if (!rafPending) {
      rafPending = true
      requestAnimationFrame(() => {
        rafPending = false
        flushFeatures()             // Flush to global
      })
    }
  }
})

// Global audio state (read by render graph)
window.__STEMENGINE_MUON_AUDIO__ = {
  bass: 0.65,
  treble: 0.42,
  rms: 0.58,
  energy: 0.71,
  beat: true,
  frequencyBands: [...]
}

Buffering: Dual-RAF gate prevents render loop blocking.

4. Render Graph Instantiation (Style Router)

// muon-iframe/stemengine-router.js
const handleCommand = (type, payload) => {
  if (type === 'setStyle') {
    window.__STEMENGINE_MUON_STYLE__ = payload.style
    updateRenderGraph(payload.style)
  }
}

const updateRenderGraph = (style) => {
  if (style === 'fluid' && window.__StemEngineFluidRenderGraph__) {
    currentRenderGraph = new window.__StemEngineFluidRenderGraph__()
  }
}

// Query param auto-initialization
window.__STEMENGINE_MUON_STYLE__ = getStyleParam()  // ?style=fluid

Trigger: setStyle postMessage or query param ?style=fluid.

5. Fluid Simulator Runtime Loop (Canvas)

// muon-iframe/fluid-render-graph.js
class FluidSimulator {
  constructor() {
    this.gl = acquireWebGLContext()
    this.allocateFBOs()
    this.compileShaders()       // (TODO: implement all 10 shaders)
    this.startRenderLoop()
  }
  
  startRenderLoop() {
    const loop = () => {
      const audio = window.__STEMENGINE_MUON_AUDIO__
      if (audio && !window.__STEMENGINE_MUON_PAUSED__) {
        this.step(audio)        // Main simulation step
      }
      requestAnimationFrame(loop)
    }
    loop()
  }
  
  step(audioFeatures) {
    // Execution order (must be sequential):
    this.splat(audioFeatures)        // Force injection
    this.curl()                      // Vorticity
    this.divergence()                // Incompressibility metric
    
    // Pressure solve (N iterations based on quality tier)
    for (let i = 0; i < this.pressureIterations; i++) {
      this.pressureJacobi()
    }
    
    this.gradientSubtract()          // Enforce incompressibility
    this.advectVelocity()            // Self-advection + damping
    this.advectDye(audioFeatures)    // Color transport + audio injection
    
    // Bloom (balanced/high only)
    if (this.enableBloom) {
      this.bloomDownsample()
      this.bloomComposite(audioFeatures)
    }
    
    this.composite()                 // Final render to canvas
  }
}

Dispatch: Each method is a render pass (fragment shader + FBO binding).

6. Canvas Output

// muon-iframe/fluid-render-graph.js
composite() {
  // Bind canvas framebuffer (default: null)
  this.gl.bindFramebuffer(this.gl.FRAMEBUFFER, null)
  
  // Use composite shader program
  this.gl.useProgram(this.compositeProgram)
  
  // Bind dye texture as input
  this.gl.uniform1i(this.compositeUniforms.u_dye, 0)
  this.gl.activeTexture(this.gl.TEXTURE0)
  this.gl.bindTexture(this.gl.TEXTURE_2D, this.dye[0].texture)
  
  // Render fullscreen quad to canvas
  this.gl.drawElements(this.gl.TRIANGLES, 6, this.gl.UNSIGNED_SHORT, 0)
}

Output: Fluid colors rendered to canvas in real-time.


Shader Dependencies & Execution Graph

┌──────────────────────────────────────────┐
│ Input: Audio Features + Query Params      │
│ (bass, treble, rms, beat, style, quality)│
└────────────────┬─────────────────────────┘
                 │
         ┌───────▼────────┐
         │ Render Graph   │
         │ Instantiation  │
         │ (by style)     │
         └───────┬────────┘
                 │
        ┌────────▼─────────────────────┐
        │ FluidSimulator.step()         │
        │ (Main simulation loop)        │
        └────────┬─────────────────────┘
                 │
    ┌────────────┼────────────────────────────────────┐
    │            │                                    │
    │    ┌──────▼──────┐                      ┌──────▼──────┐
    │    │ Splat        │                      │ (Quality)  │
    │    │ (force inj.) │                      │ Tier Logic │
    │    └──────┬───────┘                      └──────┬──────┘
    │           │                                     │
    │    ┌──────▼──────┐                       Low: 10 iter
    │    │ Curl        │                       Balanced: 20
    │    │ (vorticity) │                       High: 40
    │    └──────┬──────┘                       + Bloom
    │           │
    │    ┌──────▼──────────┐
    │    │ Divergence      │
    │    │ (incompress.)   │
    │    └──────┬──────────┘
    │           │
    │    ┌──────▼────────────────┐
    │    │ Pressure Jacobi (loop)│ ← iterations based on tier
    │    └──────┬────────────────┘
    │           │
    │    ┌──────▼─────────────┐
    │    │ Gradient Subtract   │
    │    │ (enforce incomp.)   │
    │    └──────┬─────────────┘
    │           │
    │    ┌──────▼──────────┐
    │    │ Advect Velocity │
    │    └──────┬──────────┘
    │           │
    │    ┌──────▼──────────┐
    │    │ Advect Dye      │
    │    │ (audio inject)  │
    │    └──────┬──────────┘
    │           │
    │    ┌──────▼─────────┐
    │    │ Bloom?          │ ← conditional on tier
    │    │ (if balanced/   │
    │    │  high)          │
    │    └──────┬─────────┘
    │           │
    │    ┌──────▼──────┐
    │    │ Composite   │
    │    │ (to canvas) │
    │    └─────────────┘
    │
    └────────────────────────────────────────┘
                   │
            ┌──────▼────────┐
            │ Canvas Output  │
            │ (visible dye)  │
            └────────────────┘

Texture Memory Layout

Ping-Pong Framebuffer Objects

Name Format Size (low) Size (balanced) Size (high) Purpose
velocity[0] RGBA Float32 256×256×16B 512×512×16B 1024×1024×16B Velocity X,Y + divergence
velocity[1] RGBA Float32 256×256×16B 512×512×16B 1024×1024×16B Ping-pong pair
dye[0] RGBA Float32 256×256×16B 512×512×16B 1024×1024×16B Color field
dye[1] RGBA Float32 256×256×16B 512×512×16B 1024×1024×16B Ping-pong pair
divergence[0] R Float32 256×256×4B 512×512×4B 1024×1024×4B Velocity divergence
divergence[1] R Float32 256×256×4B 512×512×4B 1024×1024×4B Ping-pong pair
pressure[0] R Float32 256×256×4B 512×512×4B 1024×1024×4B Pressure field
pressure[1] R Float32 256×256×4B 512×512×4B 1024×1024×4B Ping-pong pair
curl[0] R Float32 256×256×4B 512×512×4B 1024×1024×4B Vorticity
curl[1] R Float32 256×256×4B 512×512×4B 1024×1024×4B Ping-pong pair

Bloom Chain (balanced/high only):

Name Format Size (balanced) Size (high)
bloomDown1 RGBA Float32 256×256×16B 512×512×16B
bloomDown2 RGBA Float32 128×128×16B 256×256×16B
bloomUp1 RGBA Float32 256×256×16B 512×512×16B

Total Memory (balanced tier):

  • Main FBOs: 5 fields × 2 (ping-pong) × 16B = 160B per texel
  • At 512×512: 512×512 × 160B = 41.9 MB (main)
  • Bloom: 256×256×16 + 128×128×16 + 256×256×16 = 16.8 MB
  • Total: ~60 MB (typical, manageable on modern hardware)

Shader Program Organization

// All 10 fragment shaders combined into one large program:
// - Compile one massive fragment shader with 10 `#if PASS_TYPE` branches
// - Reuse same vertex shader for all passes
// - Problem: Large shader binary, harder to debug

// Pros: Fewer state changes, simpler linking
// Cons: Less modular, harder to update individual passes
// Separate program for each render pass:
// Program 0: splat
// Program 1: curl
// Program 2: divergence
// ...
// Program 9: composite

// Pros: Modular, easier to debug, flexible to update
// Cons: More state changes per frame (~10 useProgram calls + uniform binding)

Decision: Multiple programs for maintainability (phase 2 focused on correctness, not performance).


Build Pipeline: Overlay Injection

Dockerfile Sequence (muon-iframe/Dockerfile)

# 1. Clone Muon repo
RUN git clone --depth 1 -b ${MUON_GIT_REF} \
    https://github.com/najafmohammed/muon-music-visualizer /src/app

# 2. Install dependencies
WORKDIR /src/app
RUN npm install

# 3. Copy overlay files
COPY overlayless.css /tmp/overlayless.css
COPY stemengine-bootstrap.js /tmp/stemengine-bootstrap.js
COPY stemengine-router.js /tmp/stemengine-router.js
COPY fluid-render-graph.js /tmp/fluid-render-graph.js          # NEW (Phase 2)

# 4. Merge CSS
RUN cat /tmp/overlayless.css >> /src/app/src/style.css

# 5. Inject JavaScript overlays
RUN cp /tmp/stemengine-bootstrap.js /src/app/src/
RUN grep -q "stemengine-bootstrap" /src/app/src/index.js || \
    sed -i '1i import "./stemengine-bootstrap";' /src/app/src/index.js

RUN cp /tmp/stemengine-router.js /src/app/src/
RUN sed -i "s|__STEMENGINE_ALLOWED_ORIGIN__|${STEMENGINE_ALLOWED_ORIGIN}|g" \
    /src/app/src/stemengine-router.js
RUN grep -q "stemengine-router" /src/app/src/index.js || \
    sed -i '1i import "./stemengine-router";' /src/app/src/index.js

RUN cp /tmp/fluid-render-graph.js /src/app/src/              # NEW
RUN grep -q "fluid-render-graph" /src/app/src/index.js || \ # NEW
    sed -i '1i import "./fluid-render-graph";' /src/app/src/index.js # NEW

# 6. Build Muon
RUN npm run build

# 7. Serve from image
FROM node:20-alpine
COPY --from=builder /src/app/dist /app
EXPOSE 3011
CMD ["node", "server.js"]

Injection Order: Imports are added in FIFO order, so execution order is:

  1. stemengine-bootstrap (WebGL detection)
  2. stemengine-router (postMessage listener, style routing)
  3. fluid-render-graph (class definition, global export)

Security & Compliance Checklist

✅ CSP Headers

  • Injected by backend: frame-src 'self' <MUON_ALLOWED_ORIGIN>
  • Runtime check: STEMENGINE_ALLOWED_ORIGIN build arg

✅ postMessage Origin Guard

  • Double-check: CSP header + postMessage event.origin validation
  • Token replacement: Build-time constant, no runtime parsing

✅ Sandbox Attributes

  • Minimal: allow-scripts allow-same-origin (no allow-downloads, no allow-top-navigation)
  • Referrer policy: no-referrer

✅ Licensing

  • All dependencies MIT licensed (Muon, WebGL-Fluid, The Spirit)
  • THIRD_PARTY_NOTICES.md + compliance sign-off in place

✅ Error Handling

  • window.__stemengineEmitError__() callback for render graph errors
  • WebGL context loss detection + recovery stubs

Next Actions (Priority Order)

🔴 Priority 1: Implement Splat Shader

Why first?: Enables visual feedback immediately; audio-driven force injection.

Deliverable: FluidSimulator.prototype.splat() implementation.

Steps:

  1. Create splat vertex shader (simple passthrough)
  2. Create splat fragment shader (gaussian kernel + force mapping)
  3. Compile program, bind uniforms (u_velocity, u_splatPos, u_splatRadius, u_splatForce, u_turbulenceAmplitude)
  4. Render fullscreen quad to velocity[1] FBO
  5. Swap: velocity = [velocity[1], velocity[0]]

🟡 Priority 2: Implement Curl + Divergence

Why second?: Required for pressure solver; simpler than splat.

Deliverables: curl(), divergence() methods.

🟡 Priority 3: Implement Pressure Jacobi + Gradient Subtract

Why third?: Core incompressible fluid dynamics; enables realistic motion.

Deliverables: pressureJacobi(), gradientSubtract() methods.

🟢 Priority 4: Implement Advection (Velocity + Dye)

Why fourth?: Smooth motion transport; audio-driven coloring.

Deliverables: advectVelocity(), advectDye() methods.

🟢 Priority 5: Implement Bloom + Composite

Why last?: Optional on low tier; visual polish on balanced/high.

Deliverables: bloomDownsample(), bloomComposite(), composite() methods.


Validation & Testing

Unit-Level Validation

# Syntax check
node -c /home/translator/docker/appdata/stemengine/muon-iframe/fluid-render-graph.js

# WebGL capability detection (run in browser console)
window.__STEMENGINE_MUON_WEBGL_BASELINE__  # Should show {ok: true, missing: []}

# FBO allocation
new window.__StemEngineFluidRenderGraph__()  # Should not throw

Integration-Level Validation

# Build Docker image
docker build \
  --build-arg MUON_GIT_REF=main \
  --build-arg STEMENGINE_ALLOWED_ORIGIN=http://localhost:5173 \
  -t stemengine-muon:latest \
  docker/appdata/stemengine/muon-iframe

# Start container
docker run -p 3011:3011 stemengine-muon:latest

# Open in browser
# http://localhost:3011?style=fluid

# Listen to music, verify:
# - Fluid animation visible
# - Bass drives splat spread
# - Beat creates impulses
# - Treble adds turbulence
# - RMS controls bloom intensity

Documentation References

Document Purpose Location
PHASE2_SHADER_GUIDE.md Detailed shader pseudocode & algorithms muon-iframe/
Phase2_Initiation_Report.md This session's decisions & status appdata/stemengine/
README.md (muon-iframe) Build pipeline, command contract, audio policy muon-iframe/
PROJECT_STATUS_BOARD.md Executive summary of all phases docker/

Appendix: Known Limitations & Future Work

Phase 2 (Current)

  • ✅ Incompressible Euler flow (core fluid dynamics)
  • ❌ Vortex confinement (better preservation of small-scale features)
  • ❌ Buoyancy forcing (smoke-like effects)
  • ❌ Obstacle handling (boundaries within domain)

Phase 3 (Tendrils)

  • Particle-based visualization
  • Spectral decomposition mapping
  • Directional line rendering

Phase 4 (Spirit)

  • Geometric mesh deformations
  • Breathing/morphing animations
  • Layered composite rendering

Post-Phase 4 (Future)

  • Mobile optimization (adaptive resolution, dynamic quality switching)
  • Multi-canvas support (split-screen, gallery mode)
  • Recording + export (GIF, video, SVG)
  • Preset system (save/load favorite configurations)

Status: ✅ Ready to begin shader implementation.