575 lines
19 KiB
Markdown
575 lines
19 KiB
Markdown
# Phase 2 Ready: Architecture & Dependency Map
|
||
|
||
**Date**: 2026-04-17
|
||
**Status**: ✅ Phase 2.1 Infrastructure Complete
|
||
**Next**: Implement 10 GLSL shaders
|
||
|
||
---
|
||
|
||
## Project Structure (Overlay Injection Pattern)
|
||
|
||
```
|
||
translator/
|
||
└── docker/
|
||
├── karaokemugen-app/ (StemEngine frontend)
|
||
│ ├── src/
|
||
│ │ ├── components/
|
||
│ │ │ └── VisualizerPreviewWidget.tsx [postMessage host]
|
||
│ │ ├── api/
|
||
│ │ │ └── client.ts [muon_quality setting]
|
||
│ │ └── pages/
|
||
│ │ └── VisualizerLabPage.tsx
|
||
│ └── package.json
|
||
│
|
||
├── karaokemugen-server/ (StemEngine backend)
|
||
│ ├── app/
|
||
│ │ ├── main.py [CSP header injection]
|
||
│ │ └── routers/settings.py [muon_quality persistence]
|
||
│ └── package.json
|
||
│
|
||
├── appdata/stemengine/
|
||
│ ├── muon-iframe/ (Muon visualization build)
|
||
│ │ ├── Dockerfile [Overlay injection pipeline]
|
||
│ │ ├── stemengine-bootstrap.js [WebGL detection]
|
||
│ │ ├── stemengine-router.js [Style router + postMessage handler]
|
||
│ │ ├── fluid-render-graph.js [FBO management + render loop]
|
||
│ │ ├── README.md [Build pipeline docs]
|
||
│ │ ├── PHASE2_SHADER_GUIDE.md [10 shader specifications]
|
||
│ │ └── THIRD_PARTY_NOTICES.md [License inventory]
|
||
│ │
|
||
│ └── muon-music-visualizer/ (Cloned at build time)
|
||
│ └── (Muon source tree, untouched)
|
||
│
|
||
└── stemengine.yml [Docker compose service]
|
||
└── Phase2_Initiation_Report.md [This session's summary]
|
||
```
|
||
|
||
---
|
||
|
||
## Data Flow: Audio → Fluid Rendering
|
||
|
||
### 1. Audio Feature Extraction (StemEngine Host)
|
||
|
||
```typescript
|
||
// frontend/src/components/VisualizerPreviewWidget.tsx
|
||
const MuonFeatureExtractor = {
|
||
features: {
|
||
bass: number, // [0..1] low frequencies
|
||
treble: number, // [0..1] high frequencies
|
||
rms: number, // [0..1] total energy
|
||
energy: number, // [0..1] energy density
|
||
beat: boolean, // percussion detected?
|
||
frequencyBands: [] // [0..32] spectral breakdown
|
||
},
|
||
|
||
// Called at RAF cadence during music playback
|
||
extract() { ... }
|
||
}
|
||
```
|
||
|
||
**Output**: Features computed per frame, ready for postMessage delivery.
|
||
|
||
### 2. postMessage Command (Host → Iframe)
|
||
|
||
```javascript
|
||
// frontend/src/components/VisualizerPreviewWidget.tsx
|
||
const sendMuonCommand = (type, payload) => {
|
||
if (!muonIframeRef.current) return
|
||
muonIframeRef.current.contentWindow.postMessage(
|
||
{
|
||
version: 1,
|
||
type: type, // 'pushAudioFeatures', 'setStyle', etc.
|
||
payload: payload
|
||
},
|
||
targetOrigin // Security: origin checked on receiver
|
||
)
|
||
}
|
||
|
||
// Audio features dispatched at RAF cadence
|
||
sendMuonCommand('pushAudioFeatures', {
|
||
bass: 0.65,
|
||
treble: 0.42,
|
||
rms: 0.58,
|
||
energy: 0.71,
|
||
beat: true,
|
||
frequencyBands: [...]
|
||
})
|
||
```
|
||
|
||
**Transport**: Cross-origin iframe communication with origin guard.
|
||
|
||
### 3. postMessage Reception (Iframe Router)
|
||
|
||
```javascript
|
||
// muon-iframe/stemengine-router.js
|
||
window.addEventListener('message', (event) => {
|
||
// Origin guard (security layer 1)
|
||
if (event.origin !== ALLOWED_ORIGIN) return
|
||
|
||
const { version, type, payload } = event.data
|
||
if (version !== 1) return
|
||
|
||
// Audio rate-limiting (dual-RAF gate)
|
||
if (type === 'pushAudioFeatures') {
|
||
captureFeatures(payload) // Capture into pending
|
||
if (!rafPending) {
|
||
rafPending = true
|
||
requestAnimationFrame(() => {
|
||
rafPending = false
|
||
flushFeatures() // Flush to global
|
||
})
|
||
}
|
||
}
|
||
})
|
||
|
||
// Global audio state (read by render graph)
|
||
window.__STEMENGINE_MUON_AUDIO__ = {
|
||
bass: 0.65,
|
||
treble: 0.42,
|
||
rms: 0.58,
|
||
energy: 0.71,
|
||
beat: true,
|
||
frequencyBands: [...]
|
||
}
|
||
```
|
||
|
||
**Buffering**: Dual-RAF gate prevents render loop blocking.
|
||
|
||
### 4. Render Graph Instantiation (Style Router)
|
||
|
||
```javascript
|
||
// muon-iframe/stemengine-router.js
|
||
const handleCommand = (type, payload) => {
|
||
if (type === 'setStyle') {
|
||
window.__STEMENGINE_MUON_STYLE__ = payload.style
|
||
updateRenderGraph(payload.style)
|
||
}
|
||
}
|
||
|
||
const updateRenderGraph = (style) => {
|
||
if (style === 'fluid' && window.__StemEngineFluidRenderGraph__) {
|
||
currentRenderGraph = new window.__StemEngineFluidRenderGraph__()
|
||
}
|
||
}
|
||
|
||
// Query param auto-initialization
|
||
window.__STEMENGINE_MUON_STYLE__ = getStyleParam() // ?style=fluid
|
||
```
|
||
|
||
**Trigger**: `setStyle` postMessage or query param ?style=fluid.
|
||
|
||
### 5. Fluid Simulator Runtime Loop (Canvas)
|
||
|
||
```javascript
|
||
// muon-iframe/fluid-render-graph.js
|
||
class FluidSimulator {
|
||
constructor() {
|
||
this.gl = acquireWebGLContext()
|
||
this.allocateFBOs()
|
||
this.compileShaders() // (TODO: implement all 10 shaders)
|
||
this.startRenderLoop()
|
||
}
|
||
|
||
startRenderLoop() {
|
||
const loop = () => {
|
||
const audio = window.__STEMENGINE_MUON_AUDIO__
|
||
if (audio && !window.__STEMENGINE_MUON_PAUSED__) {
|
||
this.step(audio) // Main simulation step
|
||
}
|
||
requestAnimationFrame(loop)
|
||
}
|
||
loop()
|
||
}
|
||
|
||
step(audioFeatures) {
|
||
// Execution order (must be sequential):
|
||
this.splat(audioFeatures) // Force injection
|
||
this.curl() // Vorticity
|
||
this.divergence() // Incompressibility metric
|
||
|
||
// Pressure solve (N iterations based on quality tier)
|
||
for (let i = 0; i < this.pressureIterations; i++) {
|
||
this.pressureJacobi()
|
||
}
|
||
|
||
this.gradientSubtract() // Enforce incompressibility
|
||
this.advectVelocity() // Self-advection + damping
|
||
this.advectDye(audioFeatures) // Color transport + audio injection
|
||
|
||
// Bloom (balanced/high only)
|
||
if (this.enableBloom) {
|
||
this.bloomDownsample()
|
||
this.bloomComposite(audioFeatures)
|
||
}
|
||
|
||
this.composite() // Final render to canvas
|
||
}
|
||
}
|
||
```
|
||
|
||
**Dispatch**: Each method is a render pass (fragment shader + FBO binding).
|
||
|
||
### 6. Canvas Output
|
||
|
||
```javascript
|
||
// muon-iframe/fluid-render-graph.js
|
||
composite() {
|
||
// Bind canvas framebuffer (default: null)
|
||
this.gl.bindFramebuffer(this.gl.FRAMEBUFFER, null)
|
||
|
||
// Use composite shader program
|
||
this.gl.useProgram(this.compositeProgram)
|
||
|
||
// Bind dye texture as input
|
||
this.gl.uniform1i(this.compositeUniforms.u_dye, 0)
|
||
this.gl.activeTexture(this.gl.TEXTURE0)
|
||
this.gl.bindTexture(this.gl.TEXTURE_2D, this.dye[0].texture)
|
||
|
||
// Render fullscreen quad to canvas
|
||
this.gl.drawElements(this.gl.TRIANGLES, 6, this.gl.UNSIGNED_SHORT, 0)
|
||
}
|
||
```
|
||
|
||
**Output**: Fluid colors rendered to canvas in real-time.
|
||
|
||
---
|
||
|
||
## Shader Dependencies & Execution Graph
|
||
|
||
```
|
||
┌──────────────────────────────────────────┐
|
||
│ Input: Audio Features + Query Params │
|
||
│ (bass, treble, rms, beat, style, quality)│
|
||
└────────────────┬─────────────────────────┘
|
||
│
|
||
┌───────▼────────┐
|
||
│ Render Graph │
|
||
│ Instantiation │
|
||
│ (by style) │
|
||
└───────┬────────┘
|
||
│
|
||
┌────────▼─────────────────────┐
|
||
│ FluidSimulator.step() │
|
||
│ (Main simulation loop) │
|
||
└────────┬─────────────────────┘
|
||
│
|
||
┌────────────┼────────────────────────────────────┐
|
||
│ │ │
|
||
│ ┌──────▼──────┐ ┌──────▼──────┐
|
||
│ │ Splat │ │ (Quality) │
|
||
│ │ (force inj.) │ │ Tier Logic │
|
||
│ └──────┬───────┘ └──────┬──────┘
|
||
│ │ │
|
||
│ ┌──────▼──────┐ Low: 10 iter
|
||
│ │ Curl │ Balanced: 20
|
||
│ │ (vorticity) │ High: 40
|
||
│ └──────┬──────┘ + Bloom
|
||
│ │
|
||
│ ┌──────▼──────────┐
|
||
│ │ Divergence │
|
||
│ │ (incompress.) │
|
||
│ └──────┬──────────┘
|
||
│ │
|
||
│ ┌──────▼────────────────┐
|
||
│ │ Pressure Jacobi (loop)│ ← iterations based on tier
|
||
│ └──────┬────────────────┘
|
||
│ │
|
||
│ ┌──────▼─────────────┐
|
||
│ │ Gradient Subtract │
|
||
│ │ (enforce incomp.) │
|
||
│ └──────┬─────────────┘
|
||
│ │
|
||
│ ┌──────▼──────────┐
|
||
│ │ Advect Velocity │
|
||
│ └──────┬──────────┘
|
||
│ │
|
||
│ ┌──────▼──────────┐
|
||
│ │ Advect Dye │
|
||
│ │ (audio inject) │
|
||
│ └──────┬──────────┘
|
||
│ │
|
||
│ ┌──────▼─────────┐
|
||
│ │ Bloom? │ ← conditional on tier
|
||
│ │ (if balanced/ │
|
||
│ │ high) │
|
||
│ └──────┬─────────┘
|
||
│ │
|
||
│ ┌──────▼──────┐
|
||
│ │ Composite │
|
||
│ │ (to canvas) │
|
||
│ └─────────────┘
|
||
│
|
||
└────────────────────────────────────────┘
|
||
│
|
||
┌──────▼────────┐
|
||
│ Canvas Output │
|
||
│ (visible dye) │
|
||
└────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Texture Memory Layout
|
||
|
||
### Ping-Pong Framebuffer Objects
|
||
|
||
| Name | Format | Size (low) | Size (balanced) | Size (high) | Purpose |
|
||
|------|--------|-----------|-----------------|------------|---------|
|
||
| `velocity[0]` | RGBA Float32 | 256×256×16B | 512×512×16B | 1024×1024×16B | Velocity X,Y + divergence |
|
||
| `velocity[1]` | RGBA Float32 | 256×256×16B | 512×512×16B | 1024×1024×16B | Ping-pong pair |
|
||
| `dye[0]` | RGBA Float32 | 256×256×16B | 512×512×16B | 1024×1024×16B | Color field |
|
||
| `dye[1]` | RGBA Float32 | 256×256×16B | 512×512×16B | 1024×1024×16B | Ping-pong pair |
|
||
| `divergence[0]` | R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Velocity divergence |
|
||
| `divergence[1]` | R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Ping-pong pair |
|
||
| `pressure[0]` | R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Pressure field |
|
||
| `pressure[1]` | R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Ping-pong pair |
|
||
| `curl[0]` | R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Vorticity |
|
||
| `curl[1]` | R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Ping-pong pair |
|
||
|
||
**Bloom Chain** (balanced/high only):
|
||
|
||
| Name | Format | Size (balanced) | Size (high) |
|
||
|------|--------|-----------------|------------|
|
||
| `bloomDown1` | RGBA Float32 | 256×256×16B | 512×512×16B |
|
||
| `bloomDown2` | RGBA Float32 | 128×128×16B | 256×256×16B |
|
||
| `bloomUp1` | RGBA Float32 | 256×256×16B | 512×512×16B |
|
||
|
||
**Total Memory (balanced tier)**:
|
||
- Main FBOs: 5 fields × 2 (ping-pong) × 16B = 160B per texel
|
||
- At 512×512: 512×512 × 160B = **41.9 MB** (main)
|
||
- Bloom: 256×256×16 + 128×128×16 + 256×256×16 = **16.8 MB**
|
||
- **Total: ~60 MB** (typical, manageable on modern hardware)
|
||
|
||
---
|
||
|
||
## Shader Program Organization
|
||
|
||
### Single WebGL Program (Recommended)
|
||
|
||
```glsl
|
||
// All 10 fragment shaders combined into one large program:
|
||
// - Compile one massive fragment shader with 10 `#if PASS_TYPE` branches
|
||
// - Reuse same vertex shader for all passes
|
||
// - Problem: Large shader binary, harder to debug
|
||
|
||
// Pros: Fewer state changes, simpler linking
|
||
// Cons: Less modular, harder to update individual passes
|
||
```
|
||
|
||
### Multiple WebGL Programs (Recommended)
|
||
|
||
```glsl
|
||
// Separate program for each render pass:
|
||
// Program 0: splat
|
||
// Program 1: curl
|
||
// Program 2: divergence
|
||
// ...
|
||
// Program 9: composite
|
||
|
||
// Pros: Modular, easier to debug, flexible to update
|
||
// Cons: More state changes per frame (~10 useProgram calls + uniform binding)
|
||
```
|
||
|
||
**Decision**: Multiple programs for maintainability (phase 2 focused on correctness, not performance).
|
||
|
||
---
|
||
|
||
## Build Pipeline: Overlay Injection
|
||
|
||
### Dockerfile Sequence (muon-iframe/Dockerfile)
|
||
|
||
```dockerfile
|
||
# 1. Clone Muon repo
|
||
RUN git clone --depth 1 -b ${MUON_GIT_REF} \
|
||
https://github.com/najafmohammed/muon-music-visualizer /src/app
|
||
|
||
# 2. Install dependencies
|
||
WORKDIR /src/app
|
||
RUN npm install
|
||
|
||
# 3. Copy overlay files
|
||
COPY overlayless.css /tmp/overlayless.css
|
||
COPY stemengine-bootstrap.js /tmp/stemengine-bootstrap.js
|
||
COPY stemengine-router.js /tmp/stemengine-router.js
|
||
COPY fluid-render-graph.js /tmp/fluid-render-graph.js # NEW (Phase 2)
|
||
|
||
# 4. Merge CSS
|
||
RUN cat /tmp/overlayless.css >> /src/app/src/style.css
|
||
|
||
# 5. Inject JavaScript overlays
|
||
RUN cp /tmp/stemengine-bootstrap.js /src/app/src/
|
||
RUN grep -q "stemengine-bootstrap" /src/app/src/index.js || \
|
||
sed -i '1i import "./stemengine-bootstrap";' /src/app/src/index.js
|
||
|
||
RUN cp /tmp/stemengine-router.js /src/app/src/
|
||
RUN sed -i "s|__STEMENGINE_ALLOWED_ORIGIN__|${STEMENGINE_ALLOWED_ORIGIN}|g" \
|
||
/src/app/src/stemengine-router.js
|
||
RUN grep -q "stemengine-router" /src/app/src/index.js || \
|
||
sed -i '1i import "./stemengine-router";' /src/app/src/index.js
|
||
|
||
RUN cp /tmp/fluid-render-graph.js /src/app/src/ # NEW
|
||
RUN grep -q "fluid-render-graph" /src/app/src/index.js || \ # NEW
|
||
sed -i '1i import "./fluid-render-graph";' /src/app/src/index.js # NEW
|
||
|
||
# 6. Build Muon
|
||
RUN npm run build
|
||
|
||
# 7. Serve from image
|
||
FROM node:20-alpine
|
||
COPY --from=builder /src/app/dist /app
|
||
EXPOSE 3011
|
||
CMD ["node", "server.js"]
|
||
```
|
||
|
||
**Injection Order**: Imports are added in FIFO order, so execution order is:
|
||
1. `stemengine-bootstrap` (WebGL detection)
|
||
2. `stemengine-router` (postMessage listener, style routing)
|
||
3. `fluid-render-graph` (class definition, global export)
|
||
|
||
---
|
||
|
||
## Security & Compliance Checklist
|
||
|
||
✅ **CSP Headers**
|
||
- Injected by backend: `frame-src 'self' <MUON_ALLOWED_ORIGIN>`
|
||
- Runtime check: `STEMENGINE_ALLOWED_ORIGIN` build arg
|
||
|
||
✅ **postMessage Origin Guard**
|
||
- Double-check: CSP header + postMessage event.origin validation
|
||
- Token replacement: Build-time constant, no runtime parsing
|
||
|
||
✅ **Sandbox Attributes**
|
||
- Minimal: `allow-scripts allow-same-origin` (no allow-downloads, no allow-top-navigation)
|
||
- Referrer policy: `no-referrer`
|
||
|
||
✅ **Licensing**
|
||
- All dependencies MIT licensed (Muon, WebGL-Fluid, The Spirit)
|
||
- THIRD_PARTY_NOTICES.md + compliance sign-off in place
|
||
|
||
✅ **Error Handling**
|
||
- `window.__stemengineEmitError__()` callback for render graph errors
|
||
- WebGL context loss detection + recovery stubs
|
||
|
||
---
|
||
|
||
## Next Actions (Priority Order)
|
||
|
||
### 🔴 Priority 1: Implement Splat Shader
|
||
|
||
**Why first?**: Enables visual feedback immediately; audio-driven force injection.
|
||
|
||
**Deliverable**: `FluidSimulator.prototype.splat()` implementation.
|
||
|
||
**Steps**:
|
||
1. Create splat vertex shader (simple passthrough)
|
||
2. Create splat fragment shader (gaussian kernel + force mapping)
|
||
3. Compile program, bind uniforms (u_velocity, u_splatPos, u_splatRadius, u_splatForce, u_turbulenceAmplitude)
|
||
4. Render fullscreen quad to `velocity[1]` FBO
|
||
5. Swap: `velocity = [velocity[1], velocity[0]]`
|
||
|
||
### 🟡 Priority 2: Implement Curl + Divergence
|
||
|
||
**Why second?**: Required for pressure solver; simpler than splat.
|
||
|
||
**Deliverables**: `curl()`, `divergence()` methods.
|
||
|
||
### 🟡 Priority 3: Implement Pressure Jacobi + Gradient Subtract
|
||
|
||
**Why third?**: Core incompressible fluid dynamics; enables realistic motion.
|
||
|
||
**Deliverables**: `pressureJacobi()`, `gradientSubtract()` methods.
|
||
|
||
### 🟢 Priority 4: Implement Advection (Velocity + Dye)
|
||
|
||
**Why fourth?**: Smooth motion transport; audio-driven coloring.
|
||
|
||
**Deliverables**: `advectVelocity()`, `advectDye()` methods.
|
||
|
||
### 🟢 Priority 5: Implement Bloom + Composite
|
||
|
||
**Why last?**: Optional on low tier; visual polish on balanced/high.
|
||
|
||
**Deliverables**: `bloomDownsample()`, `bloomComposite()`, `composite()` methods.
|
||
|
||
---
|
||
|
||
## Validation & Testing
|
||
|
||
### Unit-Level Validation
|
||
|
||
```bash
|
||
# Syntax check
|
||
node -c /home/translator/docker/appdata/stemengine/muon-iframe/fluid-render-graph.js
|
||
|
||
# WebGL capability detection (run in browser console)
|
||
window.__STEMENGINE_MUON_WEBGL_BASELINE__ # Should show {ok: true, missing: []}
|
||
|
||
# FBO allocation
|
||
new window.__StemEngineFluidRenderGraph__() # Should not throw
|
||
```
|
||
|
||
### Integration-Level Validation
|
||
|
||
```bash
|
||
# Build Docker image
|
||
docker build \
|
||
--build-arg MUON_GIT_REF=main \
|
||
--build-arg STEMENGINE_ALLOWED_ORIGIN=http://localhost:5173 \
|
||
-t stemengine-muon:latest \
|
||
docker/appdata/stemengine/muon-iframe
|
||
|
||
# Start container
|
||
docker run -p 3011:3011 stemengine-muon:latest
|
||
|
||
# Open in browser
|
||
# http://localhost:3011?style=fluid
|
||
|
||
# Listen to music, verify:
|
||
# - Fluid animation visible
|
||
# - Bass drives splat spread
|
||
# - Beat creates impulses
|
||
# - Treble adds turbulence
|
||
# - RMS controls bloom intensity
|
||
```
|
||
|
||
---
|
||
|
||
## Documentation References
|
||
|
||
| Document | Purpose | Location |
|
||
|----------|---------|----------|
|
||
| **PHASE2_SHADER_GUIDE.md** | Detailed shader pseudocode & algorithms | `muon-iframe/` |
|
||
| **Phase2_Initiation_Report.md** | This session's decisions & status | `appdata/stemengine/` |
|
||
| **README.md** (muon-iframe) | Build pipeline, command contract, audio policy | `muon-iframe/` |
|
||
| **PROJECT_STATUS_BOARD.md** | Executive summary of all phases | `docker/` |
|
||
|
||
---
|
||
|
||
## Appendix: Known Limitations & Future Work
|
||
|
||
### Phase 2 (Current)
|
||
- ✅ Incompressible Euler flow (core fluid dynamics)
|
||
- ❌ Vortex confinement (better preservation of small-scale features)
|
||
- ❌ Buoyancy forcing (smoke-like effects)
|
||
- ❌ Obstacle handling (boundaries within domain)
|
||
|
||
### Phase 3 (Tendrils)
|
||
- Particle-based visualization
|
||
- Spectral decomposition mapping
|
||
- Directional line rendering
|
||
|
||
### Phase 4 (Spirit)
|
||
- Geometric mesh deformations
|
||
- Breathing/morphing animations
|
||
- Layered composite rendering
|
||
|
||
### Post-Phase 4 (Future)
|
||
- Mobile optimization (adaptive resolution, dynamic quality switching)
|
||
- Multi-canvas support (split-screen, gallery mode)
|
||
- Recording + export (GIF, video, SVG)
|
||
- Preset system (save/load favorite configurations)
|
||
|
||
---
|
||
|
||
**Status**: ✅ Ready to begin shader implementation.
|