Files
stemengine_old/Phase2_Architecture_And_Dependencies.md
2026-04-21 16:24:11 +09:00

575 lines
19 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Phase 2 Ready: Architecture & Dependency Map
**Date**: 2026-04-17
**Status**: ✅ Phase 2.1 Infrastructure Complete
**Next**: Implement 10 GLSL shaders
---
## Project Structure (Overlay Injection Pattern)
```
translator/
└── docker/
├── karaokemugen-app/ (StemEngine frontend)
│ ├── src/
│ │ ├── components/
│ │ │ └── VisualizerPreviewWidget.tsx [postMessage host]
│ │ ├── api/
│ │ │ └── client.ts [muon_quality setting]
│ │ └── pages/
│ │ └── VisualizerLabPage.tsx
│ └── package.json
│
├── karaokemugen-server/ (StemEngine backend)
│ ├── app/
│ │ ├── main.py [CSP header injection]
│ │ └── routers/settings.py [muon_quality persistence]
│ └── package.json
│
├── appdata/stemengine/
│ ├── muon-iframe/ (Muon visualization build)
│ │ ├── Dockerfile [Overlay injection pipeline]
│ │ ├── stemengine-bootstrap.js [WebGL detection]
│ │ ├── stemengine-router.js [Style router + postMessage handler]
│ │ ├── fluid-render-graph.js [FBO management + render loop]
│ │ ├── README.md [Build pipeline docs]
│ │ ├── PHASE2_SHADER_GUIDE.md [10 shader specifications]
│ │ └── THIRD_PARTY_NOTICES.md [License inventory]
│ │
│ └── muon-music-visualizer/ (Cloned at build time)
│ └── (Muon source tree, untouched)
│
└── stemengine.yml [Docker compose service]
└── Phase2_Initiation_Report.md [This session's summary]
```
---
## Data Flow: Audio → Fluid Rendering
### 1. Audio Feature Extraction (StemEngine Host)
```typescript
// frontend/src/components/VisualizerPreviewWidget.tsx
const MuonFeatureExtractor = {
features: {
bass: number, // [0..1] low frequencies
treble: number, // [0..1] high frequencies
rms: number, // [0..1] total energy
energy: number, // [0..1] energy density
beat: boolean, // percussion detected?
frequencyBands: [] // [0..32] spectral breakdown
},
// Called at RAF cadence during music playback
extract() { ... }
}
```
**Output**: Features computed per frame, ready for postMessage delivery.
### 2. postMessage Command (Host → Iframe)
```javascript
// frontend/src/components/VisualizerPreviewWidget.tsx
const sendMuonCommand = (type, payload) => {
if (!muonIframeRef.current) return
muonIframeRef.current.contentWindow.postMessage(
{
version: 1,
type: type, // 'pushAudioFeatures', 'setStyle', etc.
payload: payload
},
targetOrigin // Security: origin checked on receiver
)
}
// Audio features dispatched at RAF cadence
sendMuonCommand('pushAudioFeatures', {
bass: 0.65,
treble: 0.42,
rms: 0.58,
energy: 0.71,
beat: true,
frequencyBands: [...]
})
```
**Transport**: Cross-origin iframe communication with origin guard.
### 3. postMessage Reception (Iframe Router)
```javascript
// muon-iframe/stemengine-router.js
window.addEventListener('message', (event) => {
// Origin guard (security layer 1)
if (event.origin !== ALLOWED_ORIGIN) return
const { version, type, payload } = event.data
if (version !== 1) return
// Audio rate-limiting (dual-RAF gate)
if (type === 'pushAudioFeatures') {
captureFeatures(payload) // Capture into pending
if (!rafPending) {
rafPending = true
requestAnimationFrame(() => {
rafPending = false
flushFeatures() // Flush to global
})
}
}
})
// Global audio state (read by render graph)
window.__STEMENGINE_MUON_AUDIO__ = {
bass: 0.65,
treble: 0.42,
rms: 0.58,
energy: 0.71,
beat: true,
frequencyBands: [...]
}
```
**Buffering**: Dual-RAF gate prevents render loop blocking.
### 4. Render Graph Instantiation (Style Router)
```javascript
// muon-iframe/stemengine-router.js
const handleCommand = (type, payload) => {
if (type === 'setStyle') {
window.__STEMENGINE_MUON_STYLE__ = payload.style
updateRenderGraph(payload.style)
}
}
const updateRenderGraph = (style) => {
if (style === 'fluid' && window.__StemEngineFluidRenderGraph__) {
currentRenderGraph = new window.__StemEngineFluidRenderGraph__()
}
}
// Query param auto-initialization
window.__STEMENGINE_MUON_STYLE__ = getStyleParam() // ?style=fluid
```
**Trigger**: `setStyle` postMessage or query param ?style=fluid.
### 5. Fluid Simulator Runtime Loop (Canvas)
```javascript
// muon-iframe/fluid-render-graph.js
class FluidSimulator {
constructor() {
this.gl = acquireWebGLContext()
this.allocateFBOs()
this.compileShaders() // (TODO: implement all 10 shaders)
this.startRenderLoop()
}
startRenderLoop() {
const loop = () => {
const audio = window.__STEMENGINE_MUON_AUDIO__
if (audio && !window.__STEMENGINE_MUON_PAUSED__) {
this.step(audio) // Main simulation step
}
requestAnimationFrame(loop)
}
loop()
}
step(audioFeatures) {
// Execution order (must be sequential):
this.splat(audioFeatures) // Force injection
this.curl() // Vorticity
this.divergence() // Incompressibility metric
// Pressure solve (N iterations based on quality tier)
for (let i = 0; i < this.pressureIterations; i++) {
this.pressureJacobi()
}
this.gradientSubtract() // Enforce incompressibility
this.advectVelocity() // Self-advection + damping
this.advectDye(audioFeatures) // Color transport + audio injection
// Bloom (balanced/high only)
if (this.enableBloom) {
this.bloomDownsample()
this.bloomComposite(audioFeatures)
}
this.composite() // Final render to canvas
}
}
```
**Dispatch**: Each method is a render pass (fragment shader + FBO binding).
### 6. Canvas Output
```javascript
// muon-iframe/fluid-render-graph.js
composite() {
// Bind canvas framebuffer (default: null)
this.gl.bindFramebuffer(this.gl.FRAMEBUFFER, null)
// Use composite shader program
this.gl.useProgram(this.compositeProgram)
// Bind dye texture as input
this.gl.uniform1i(this.compositeUniforms.u_dye, 0)
this.gl.activeTexture(this.gl.TEXTURE0)
this.gl.bindTexture(this.gl.TEXTURE_2D, this.dye[0].texture)
// Render fullscreen quad to canvas
this.gl.drawElements(this.gl.TRIANGLES, 6, this.gl.UNSIGNED_SHORT, 0)
}
```
**Output**: Fluid colors rendered to canvas in real-time.
---
## Shader Dependencies & Execution Graph
```
┌──────────────────────────────────────────┐
│ Input: Audio Features + Query Params │
│ (bass, treble, rms, beat, style, quality)│
└────────────────┬─────────────────────────┘
│
┌───────▼────────┐
│ Render Graph │
│ Instantiation │
│ (by style) │
└───────┬────────┘
│
┌────────▼─────────────────────┐
│ FluidSimulator.step() │
│ (Main simulation loop) │
└────────┬─────────────────────┘
│
┌────────────┼────────────────────────────────────┐
│ │ │
│ ┌──────▼──────┐ ┌──────▼──────┐
│ │ Splat │ │ (Quality) │
│ │ (force inj.) │ │ Tier Logic │
│ └──────┬───────┘ └──────┬──────┘
│ │ │
│ ┌──────▼──────┐ Low: 10 iter
│ │ Curl │ Balanced: 20
│ │ (vorticity) │ High: 40
│ └──────┬──────┘ + Bloom
│ │
│ ┌──────▼──────────┐
│ │ Divergence │
│ │ (incompress.) │
│ └──────┬──────────┘
│ │
│ ┌──────▼────────────────┐
│ │ Pressure Jacobi (loop)│ ← iterations based on tier
│ └──────┬────────────────┘
│ │
│ ┌──────▼─────────────┐
│ │ Gradient Subtract │
│ │ (enforce incomp.) │
│ └──────┬─────────────┘
│ │
│ ┌──────▼──────────┐
│ │ Advect Velocity │
│ └──────┬──────────┘
│ │
│ ┌──────▼──────────┐
│ │ Advect Dye │
│ │ (audio inject) │
│ └──────┬──────────┘
│ │
│ ┌──────▼─────────┐
│ │ Bloom? │ ← conditional on tier
│ │ (if balanced/ │
│ │ high) │
│ └──────┬─────────┘
│ │
│ ┌──────▼──────┐
│ │ Composite │
│ │ (to canvas) │
│ └─────────────┘
│
└────────────────────────────────────────┘
│
┌──────▼────────┐
│ Canvas Output │
│ (visible dye) │
└────────────────┘
```
---
## Texture Memory Layout
### Ping-Pong Framebuffer Objects
| Name | Format | Size (low) | Size (balanced) | Size (high) | Purpose |
|------|--------|-----------|-----------------|------------|---------|
| `velocity[0]` | RGBA Float32 | 256×256×16B | 512×512×16B | 1024×1024×16B | Velocity X,Y + divergence |
| `velocity[1]` | RGBA Float32 | 256×256×16B | 512×512×16B | 1024×1024×16B | Ping-pong pair |
| `dye[0]` | RGBA Float32 | 256×256×16B | 512×512×16B | 1024×1024×16B | Color field |
| `dye[1]` | RGBA Float32 | 256×256×16B | 512×512×16B | 1024×1024×16B | Ping-pong pair |
| `divergence[0]` | R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Velocity divergence |
| `divergence[1]` | R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Ping-pong pair |
| `pressure[0]` | R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Pressure field |
| `pressure[1]` | R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Ping-pong pair |
| `curl[0]` | R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Vorticity |
| `curl[1]` | R Float32 | 256×256×4B | 512×512×4B | 1024×1024×4B | Ping-pong pair |
**Bloom Chain** (balanced/high only):
| Name | Format | Size (balanced) | Size (high) |
|------|--------|-----------------|------------|
| `bloomDown1` | RGBA Float32 | 256×256×16B | 512×512×16B |
| `bloomDown2` | RGBA Float32 | 128×128×16B | 256×256×16B |
| `bloomUp1` | RGBA Float32 | 256×256×16B | 512×512×16B |
**Total Memory (balanced tier)**:
- Main FBOs: 5 fields × 2 (ping-pong) × 16B = 160B per texel
- At 512×512: 512×512 × 160B = **41.9 MB** (main)
- Bloom: 256×256×16 + 128×128×16 + 256×256×16 = **16.8 MB**
- **Total: ~60 MB** (typical, manageable on modern hardware)
---
## Shader Program Organization
### Single WebGL Program (Recommended)
```glsl
// All 10 fragment shaders combined into one large program:
// - Compile one massive fragment shader with 10 `#if PASS_TYPE` branches
// - Reuse same vertex shader for all passes
// - Problem: Large shader binary, harder to debug
// Pros: Fewer state changes, simpler linking
// Cons: Less modular, harder to update individual passes
```
### Multiple WebGL Programs (Recommended)
```glsl
// Separate program for each render pass:
// Program 0: splat
// Program 1: curl
// Program 2: divergence
// ...
// Program 9: composite
// Pros: Modular, easier to debug, flexible to update
// Cons: More state changes per frame (~10 useProgram calls + uniform binding)
```
**Decision**: Multiple programs for maintainability (phase 2 focused on correctness, not performance).
---
## Build Pipeline: Overlay Injection
### Dockerfile Sequence (muon-iframe/Dockerfile)
```dockerfile
# 1. Clone Muon repo
RUN git clone --depth 1 -b ${MUON_GIT_REF} \
https://github.com/najafmohammed/muon-music-visualizer /src/app
# 2. Install dependencies
WORKDIR /src/app
RUN npm install
# 3. Copy overlay files
COPY overlayless.css /tmp/overlayless.css
COPY stemengine-bootstrap.js /tmp/stemengine-bootstrap.js
COPY stemengine-router.js /tmp/stemengine-router.js
COPY fluid-render-graph.js /tmp/fluid-render-graph.js # NEW (Phase 2)
# 4. Merge CSS
RUN cat /tmp/overlayless.css >> /src/app/src/style.css
# 5. Inject JavaScript overlays
RUN cp /tmp/stemengine-bootstrap.js /src/app/src/
RUN grep -q "stemengine-bootstrap" /src/app/src/index.js || \
sed -i '1i import "./stemengine-bootstrap";' /src/app/src/index.js
RUN cp /tmp/stemengine-router.js /src/app/src/
RUN sed -i "s|__STEMENGINE_ALLOWED_ORIGIN__|${STEMENGINE_ALLOWED_ORIGIN}|g" \
/src/app/src/stemengine-router.js
RUN grep -q "stemengine-router" /src/app/src/index.js || \
sed -i '1i import "./stemengine-router";' /src/app/src/index.js
RUN cp /tmp/fluid-render-graph.js /src/app/src/ # NEW
RUN grep -q "fluid-render-graph" /src/app/src/index.js || \ # NEW
sed -i '1i import "./fluid-render-graph";' /src/app/src/index.js # NEW
# 6. Build Muon
RUN npm run build
# 7. Serve from image
FROM node:20-alpine
COPY --from=builder /src/app/dist /app
EXPOSE 3011
CMD ["node", "server.js"]
```
**Injection Order**: Imports are added in FIFO order, so execution order is:
1. `stemengine-bootstrap` (WebGL detection)
2. `stemengine-router` (postMessage listener, style routing)
3. `fluid-render-graph` (class definition, global export)
---
## Security & Compliance Checklist
✅ **CSP Headers**
- Injected by backend: `frame-src 'self' <MUON_ALLOWED_ORIGIN>`
- Runtime check: `STEMENGINE_ALLOWED_ORIGIN` build arg
✅ **postMessage Origin Guard**
- Double-check: CSP header + postMessage event.origin validation
- Token replacement: Build-time constant, no runtime parsing
✅ **Sandbox Attributes**
- Minimal: `allow-scripts allow-same-origin` (no allow-downloads, no allow-top-navigation)
- Referrer policy: `no-referrer`
✅ **Licensing**
- All dependencies MIT licensed (Muon, WebGL-Fluid, The Spirit)
- THIRD_PARTY_NOTICES.md + compliance sign-off in place
✅ **Error Handling**
- `window.__stemengineEmitError__()` callback for render graph errors
- WebGL context loss detection + recovery stubs
---
## Next Actions (Priority Order)
### 🔴 Priority 1: Implement Splat Shader
**Why first?**: Enables visual feedback immediately; audio-driven force injection.
**Deliverable**: `FluidSimulator.prototype.splat()` implementation.
**Steps**:
1. Create splat vertex shader (simple passthrough)
2. Create splat fragment shader (gaussian kernel + force mapping)
3. Compile program, bind uniforms (u_velocity, u_splatPos, u_splatRadius, u_splatForce, u_turbulenceAmplitude)
4. Render fullscreen quad to `velocity[1]` FBO
5. Swap: `velocity = [velocity[1], velocity[0]]`
### 🟡 Priority 2: Implement Curl + Divergence
**Why second?**: Required for pressure solver; simpler than splat.
**Deliverables**: `curl()`, `divergence()` methods.
### 🟡 Priority 3: Implement Pressure Jacobi + Gradient Subtract
**Why third?**: Core incompressible fluid dynamics; enables realistic motion.
**Deliverables**: `pressureJacobi()`, `gradientSubtract()` methods.
### 🟢 Priority 4: Implement Advection (Velocity + Dye)
**Why fourth?**: Smooth motion transport; audio-driven coloring.
**Deliverables**: `advectVelocity()`, `advectDye()` methods.
### 🟢 Priority 5: Implement Bloom + Composite
**Why last?**: Optional on low tier; visual polish on balanced/high.
**Deliverables**: `bloomDownsample()`, `bloomComposite()`, `composite()` methods.
---
## Validation & Testing
### Unit-Level Validation
```bash
# Syntax check
node -c /home/translator/docker/appdata/stemengine/muon-iframe/fluid-render-graph.js
# WebGL capability detection (run in browser console)
window.__STEMENGINE_MUON_WEBGL_BASELINE__ # Should show {ok: true, missing: []}
# FBO allocation
new window.__StemEngineFluidRenderGraph__() # Should not throw
```
### Integration-Level Validation
```bash
# Build Docker image
docker build \
--build-arg MUON_GIT_REF=main \
--build-arg STEMENGINE_ALLOWED_ORIGIN=http://localhost:5173 \
-t stemengine-muon:latest \
docker/appdata/stemengine/muon-iframe
# Start container
docker run -p 3011:3011 stemengine-muon:latest
# Open in browser
# http://localhost:3011?style=fluid
# Listen to music, verify:
# - Fluid animation visible
# - Bass drives splat spread
# - Beat creates impulses
# - Treble adds turbulence
# - RMS controls bloom intensity
```
---
## Documentation References
| Document | Purpose | Location |
|----------|---------|----------|
| **PHASE2_SHADER_GUIDE.md** | Detailed shader pseudocode & algorithms | `muon-iframe/` |
| **Phase2_Initiation_Report.md** | This session's decisions & status | `appdata/stemengine/` |
| **README.md** (muon-iframe) | Build pipeline, command contract, audio policy | `muon-iframe/` |
| **PROJECT_STATUS_BOARD.md** | Executive summary of all phases | `docker/` |
---
## Appendix: Known Limitations & Future Work
### Phase 2 (Current)
- ✅ Incompressible Euler flow (core fluid dynamics)
- ❌ Vortex confinement (better preservation of small-scale features)
- ❌ Buoyancy forcing (smoke-like effects)
- ❌ Obstacle handling (boundaries within domain)
### Phase 3 (Tendrils)
- Particle-based visualization
- Spectral decomposition mapping
- Directional line rendering
### Phase 4 (Spirit)
- Geometric mesh deformations
- Breathing/morphing animations
- Layered composite rendering
### Post-Phase 4 (Future)
- Mobile optimization (adaptive resolution, dynamic quality switching)
- Multi-canvas support (split-screen, gallery mode)
- Recording + export (GIF, video, SVG)
- Preset system (save/load favorite configurations)
---
**Status**: ✅ Ready to begin shader implementation.