Files
stemengine_old/muon-iframe/PHASE2_SHADER_GUIDE.md
2026-04-21 16:24:11 +09:00

423 lines
14 KiB
Markdown

# Phase 2: Fluid Render Graph — Shader Implementation Guide
**Status**: In Progress (Infrastructure ✅, Shaders 🚧)
**Date Started**: 2026-04-17
**Last Updated**: 2026-04-17
## Overview
The `fluid-render-graph.js` overlay provides FBO infrastructure, quality tier configuration, and audio mapping logic. This guide documents the **10 GLSL shader programs** that must be implemented to complete Phase 2 rendering.
## Architecture Review
### Execution Order
Each frame, the `FluidSimulator.step(audioFeatures)` method executes render passes in this sequence:
1. **Splat** — Inject audio-driven forces into velocity field
2. **Curl** — Compute vorticity from velocity
3. **Divergence** — Compute divergence of velocity field
4. **Pressure Jacobi** (loop N times per quality tier)
5. **Gradient Subtract** — Remove pressure gradient from velocity
6. **Advect Velocity** — Self-advection of velocity field
7. **Advect Dye** — Carry colors downstream
8. **(Optional) Bloom Downsample** — 2-tap downsampling (balanced/high only)
9. **(Optional) Bloom Composite** — Upsample + accumulate (balanced/high only)
10. **Composite** — Final render to canvas
### FBO Layout
```
Ping-Pong Pairs (swapped each frame):
- velocity[0], velocity[1] [vec4: vx, vy, divx, divy]
- dye[0], dye[1] [vec4: r, g, b, a]
- divergence[0], divergence[1] [float: div]
- pressure[0], pressure[1] [float: p]
- curl[0], curl[1] [float: ω_z (2D vorticity)]
Bloom Chain (balanced/high only):
- bloomDown1, bloomDown2 [quarter-res downsamples]
- bloomUp1 [upsampled for composite]
```
## Shader Programs
### 1. Splat Shader
**Purpose**: Inject audio-driven forces into velocity field.
**Input Audio Mapping**:
- `bass` → Force magnitude & radius (low frequencies = diffuse spread)
- `beat` → Impulse splat (percussion pulse = concentrated force)
- `treble` → Turbulence injection (high frequencies = chaotic rotation)
**Uniforms**:
```glsl
uniform sampler2D u_velocity; // Ping-pong read (velocity[i-1])
uniform vec2 u_splatPos; // Canvas center or random offset
uniform float u_splatRadius; // Gaussian kernel radius [0.5..2.0] * canvas_scale
uniform float u_splatForce; // Force magnitude [0..10]
uniform bool u_splatIsBeat; // Beat-driven impulse? (concentrated)
uniform float u_turbulenceAmplitude; // Treble-driven vorticity [0..1]
```
**Algorithm**:
```glsl
vec2 pos = gl_FragCoord.xy;
float dist = length(pos - u_splatPos);
float kernel = exp(-dist * dist / (u_splatRadius * u_splatRadius));
vec2 splatVel;
if (u_splatIsBeat) {
// Concentrated impulse for percussion
splatVel = normalize(u_splatPos - pos) * u_splatForce * kernel;
} else {
// Diffuse spread for bass
splatVel = (u_splatPos - pos) * u_splatForce * kernel * 0.1;
}
// Rotate splat velocity by treble-driven angle
float angle = u_turbulenceAmplitude * sin(length(pos) * 0.01) * 3.14159;
splatVel = vec2(
splatVel.x * cos(angle) - splatVel.y * sin(angle),
splatVel.x * sin(angle) + splatVel.y * cos(angle)
);
vec2 vel = texture2D(u_velocity, uv).xy;
gl_FragColor = vec4(vel + splatVel, 0.0, 1.0);
```
**Call Site** (in `FluidSimulator.splat()`):
```javascript
const audioFeatures = window.__STEMENGINE_MUON_AUDIO__
const bass = audioFeatures.bass // [0..1]
const beat = audioFeatures.beat // boolean
const treble = audioFeatures.treble // [0..1]
const splatRadius = 50 + bass * 100 // Spread with bass: [50..150] px
const splatForce = 10 + bass * 20 // Force with bass: [10..30]
const turbulence = treble * 1.5 // Amplitude: [0..1.5]
// Render splat pass with these uniforms
```
### 2. Curl Shader
**Purpose**: Compute 2D vorticity (rotation) from velocity field.
**Formula**: `ω_z = ∂v_y/∂x - ∂v_x/∂y` (central differences)
**Uniforms**:
```glsl
uniform sampler2D u_velocity;
uniform vec2 u_invSize; // 1.0 / vec2(texW, texH)
```
**Algorithm**:
```glsl
float n = texture2D(u_velocity, uv + vec2(0.0, u_invSize.y)).y;
float s = texture2D(u_velocity, uv - vec2(0.0, u_invSize.y)).y;
float e = texture2D(u_velocity, uv + vec2(u_invSize.x, 0.0)).x;
float w = texture2D(u_velocity, uv - vec2(u_invSize.x, 0.0)).x;
float curl = (e - w) - (n - s);
gl_FragColor = vec4(curl, 0.0, 0.0, 1.0);
```
### 3. Divergence Shader
**Purpose**: Compute divergence of velocity field (∇·v).
**Formula**: `div = ∂v_x/∂x + ∂v_y/∂y` (central differences)
**Uniforms**:
```glsl
uniform sampler2D u_velocity;
uniform vec2 u_invSize;
```
**Algorithm**:
```glsl
float n = texture2D(u_velocity, uv + vec2(0.0, u_invSize.y)).y;
float s = texture2D(u_velocity, uv - vec2(0.0, u_invSize.y)).y;
float e = texture2D(u_velocity, uv + vec2(u_invSize.x, 0.0)).x;
float w = texture2D(u_velocity, uv - vec2(u_invSize.x, 0.0)).x;
float div = 0.5 * ((e - w) + (n - s));
gl_FragColor = vec4(div, 0.0, 0.0, 1.0);
```
### 4. Pressure Jacobi Shader
**Purpose**: Iterative pressure solver using Jacobi method.
**Converges to**: ∇²p = -div(v)
**Iterations per tier**: Low=10, Balanced=20, High=40
**Uniforms**:
```glsl
uniform sampler2D u_pressure; // Ping-pong read
uniform sampler2D u_divergence;
uniform float u_alpha; // -1.0 / 4.0
uniform float u_rBeta; // 0.25 (1/4)
uniform vec2 u_invSize;
```
**Algorithm**:
```glsl
float n = texture2D(u_pressure, uv + vec2(0.0, u_invSize.y)).x;
float s = texture2D(u_pressure, uv - vec2(0.0, u_invSize.y)).x;
float e = texture2D(u_pressure, uv + vec2(u_invSize.x, 0.0)).x;
float w = texture2D(u_pressure, uv - vec2(u_invSize.x, 0.0)).x;
float div = texture2D(u_divergence, uv).x;
float p = (n + s + e + w + u_alpha * div) * u_rBeta;
gl_FragColor = vec4(p, 0.0, 0.0, 1.0);
```
### 5. Gradient Subtract Shader
**Purpose**: Subtract pressure gradient from velocity (enforces incompressibility).
**Formula**: `v_new = v - ∇p` (central difference gradient)
**Uniforms**:
```glsl
uniform sampler2D u_velocity;
uniform sampler2D u_pressure;
uniform vec2 u_invSize;
uniform float u_gradientScale; // 1.0 (or scaled for stability)
```
**Algorithm**:
```glsl
float n = texture2D(u_pressure, uv + vec2(0.0, u_invSize.y)).x;
float s = texture2D(u_pressure, uv - vec2(0.0, u_invSize.y)).x;
float e = texture2D(u_pressure, uv + vec2(u_invSize.x, 0.0)).x;
float w = texture2D(u_pressure, uv - vec2(u_invSize.x, 0.0)).x;
vec2 grad = u_gradientScale * vec2(e - w, n - s);
vec2 vel = texture2D(u_velocity, uv).xy;
gl_FragColor = vec4(vel - grad, 0.0, 1.0);
```
### 6. Advect Velocity Shader
**Purpose**: Self-advect velocity field along its own streamlines.
**Formula**: `v(t+dt) = v(t + dt*∇x(t))` (semi-Lagrangian backtracking)
**Uniforms**:
```glsl
uniform sampler2D u_velocity;
uniform vec2 u_invSize;
uniform float u_dt; // Time step [0.016 / 1000] ≈ 0.00001..0.01
uniform float u_decay; // [0.99..0.995] for damping
```
**Algorithm**:
```glsl
vec2 vel = texture2D(u_velocity, uv).xy;
vec2 backPos = uv - u_dt * vel * u_invSize;
// Clamp to valid texture range [0, 1]
backPos = clamp(backPos, u_invSize, 1.0 - u_invSize);
vec2 advectedVel = texture2D(u_velocity, backPos).xy * u_decay;
gl_FragColor = vec4(advectedVel, 0.0, 1.0);
```
### 7. Advect Dye Shader
**Purpose**: Carry color/dye downstream along velocity streamlines.
**Maps audio features to dye color**:
- `bass` → Red channel (low frequencies = warm)
- `treble` → Blue channel (high frequencies = cool)
- `rms` → Green channel (total energy = brightness)
**Uniforms**:
```glsl
uniform sampler2D u_dye;
uniform sampler2D u_velocity;
uniform vec2 u_invSize;
uniform float u_dt;
uniform float u_decay; // [0.98..0.99] for dye persistence
uniform vec3 u_audioColor; // (r, g, b) from bass/treble/rms
```
**Algorithm**:
```glsl
vec2 vel = texture2D(u_velocity, uv).xy;
vec2 backPos = uv - u_dt * vel * u_invSize;
backPos = clamp(backPos, u_invSize, 1.0 - u_invSize);
vec4 dye = texture2D(u_dye, backPos) * u_decay;
// Inject audio-driven color into this fragment
float mixFactor = 0.15; // Injection strength
dye.rgb += u_audioColor * mixFactor;
dye.rgb = clamp(dye.rgb, 0.0, 1.0);
gl_FragColor = dye;
```
**Call Site** (in `FluidSimulator.advectDye()`):
```javascript
const audio = window.__STEMENGINE_MUON_AUDIO__
const audioColor = [
audio.bass, // Red channel
audio.rms, // Green channel
audio.treble, // Blue channel
]
// Pass audioColor as u_audioColor uniform
```
### 8. Bloom Downsample Shader
**Purpose**: 2-tap downsampling for bloom chain.
**Only rendered on balanced/high quality tiers**.
**Uniforms**:
```glsl
uniform sampler2D u_source; // Full-res dye texture
uniform vec2 u_invSize;
```
**Algorithm**:
```glsl
vec2 uv0 = uv - u_invSize * 0.5;
vec2 uv1 = uv + u_invSize * 0.5;
vec4 s0 = texture2D(u_source, vec2(uv0.x, uv0.y));
vec4 s1 = texture2D(u_source, vec2(uv1.x, uv0.y));
vec4 s2 = texture2D(u_source, vec2(uv0.x, uv1.y));
vec4 s3 = texture2D(u_source, vec2(uv1.x, uv1.y));
gl_FragColor = (s0 + s1 + s2 + s3) * 0.25;
```
### 9. Bloom Composite Shader
**Purpose**: Upsample and composite bloom chain back into full-res dye.
**Creates soft glow/halo effect from bright regions**.
**Uniforms**:
```glsl
uniform sampler2D u_fullRes; // Full-res dye
uniform sampler2D u_bloomUp; // Upsampled bloom (from down→up)
uniform float u_bloomIntensity; // [0..2], maps from rms audio
```
**Algorithm**:
```glsl
vec4 full = texture2D(u_fullRes, uv);
vec4 bloom = texture2D(u_bloomUp, uv);
// Additive composite with intensity control
gl_FragColor = full + bloom * u_bloomIntensity;
gl_FragColor.rgb = clamp(gl_FragColor.rgb, 0.0, 1.0);
```
**Call Site** (in `FluidSimulator.bloomComposite()`):
```javascript
const audio = window.__STEMENGINE_MUON_AUDIO__
const bloomIntensity = 0.5 + audio.rms * 1.5 // [0.5..2.0]
// Pass as u_bloomIntensity uniform
```
### 10. Composite Shader
**Purpose**: Final render to canvas.
**Outputs**: Velocity curl as hue, magnitude as value (HSV colorspace for artistic effect).
**Uniforms**:
```glsl
uniform sampler2D u_dye; // Advected color field
uniform sampler2D u_curl; // Vorticity field
uniform sampler2D u_velocity; // Velocity field
uniform bool u_showVelocity; // Debug mode flag (default: false)
uniform bool u_showCurl; // Debug mode flag (default: false)
```
**Algorithm (Production)**:
```glsl
// Normal mode: show dye
vec4 col = texture2D(u_dye, uv);
gl_FragColor = vec4(col.rgb, 1.0);
```
**Algorithm (Debug)**:
```glsl
// Show curl (vorticity) as grayscale
float c = texture2D(u_curl, uv).x;
float col = abs(c) * 10.0; // Amplify for visibility
gl_FragColor = vec4(vec3(col), 1.0);
```
## Implementation Checklist
- [ ] Splat shader compiles, uniforms bind, output visible
- [ ] Curl shader computes correctly (validation: compare against CPU reference)
- [ ] Divergence shader computes correctly
- [ ] Pressure Jacobi shader converges (test: divergence should decrease after N iterations)
- [ ] Gradient Subtract shader enforces incompressibility
- [ ] Advect Velocity shader prevents fluid explosion (decay working)
- [ ] Advect Dye shader shows audio-reactive color injection
- [ ] Bloom Downsample shader outputs quarter-res texture
- [ ] Bloom Composite shader produces visible glow on balanced/high tiers
- [ ] Composite shader renders final image to canvas
- [ ] All 10 shaders linked into one WebGL program or individual programs
- [ ] FBOs allocated, ping-pong swapping works
- [ ] Quality tier iteration counts applied (low=10, balanced=20, high=40)
- [ ] Audio features (bass, beat, treble, rms) correctly injected into splat/advect
- [ ] Live performance: 60 FPS on desktop, graceful degration on mobile
- [ ] Style router instantiates FluidSimulator when style='fluid'
## Integration Points
### In `fluid-render-graph.js`:
Each method stub in `FluidSimulator.prototype` must:
1. Create or reuse a WebGL program with the appropriate shader pair (vertex + fragment)
2. Bind input textures to texture units
3. Set uniforms from `this.width`, `this.height`, `this.quality`, audio features
4. Bind output FBO
5. Render fullscreen quad (use existing VAO from constructor)
6. Ensure ping-pong swapping happens (read from [i-1], write to [i])
### In `stemengine-router.js`:
`handleCommand('setStyle', {style: 'fluid'})` → `updateRenderGraph()` → `new window.__StemEngineFluidRenderGraph__()`
- Constructor detects WebGL context from canvas
- Allocates FBOs and compiles all 10 shader programs
- Hooks into animation loop via `requestAnimationFrame`
### In Dockerfile:
```dockerfile
COPY fluid-render-graph.js /tmp/
RUN cp /tmp/fluid-render-graph.js /src/app/src/fluid-render-graph.js
RUN grep -q "fluid-render-graph" /src/app/src/index.js || sed -i '1i import "./fluid-render-graph";' /src/app/src/index.js
```
## Testing Strategy
### Unit Tests (manual):
1. **Shader compilation**: Verify each shader compiles without errors
2. **Uniform binding**: Check all uniforms receive correct values
3. **Texture unit management**: Verify no texture conflicts (max 8 units, carefully manage)
4. **FBO attachments**: Test each FBO binds correctly and renders to its texture
### Integration Tests:
1. **Full loop**: Run step() for 60 frames, verify no WebGL errors
2. **Audio injection**: Verify bass/beat/treble/rms values propagate to splat
3. **Bloom effectiveness**: Compare balanced/high to low tier visually (bloom should appear on balanced+)
4. **Quality tier switching**: Jacobi iteration counts should change on quality change
### Performance Tests:
1. **Low tier**: 256x256 resolution, 10 pressure iterations → expect 144+ FPS
2. **Balanced tier**: 512x512 resolution, 20 pressure iterations + bloom → expect 60 FPS
3. **High tier**: 1024x1024 resolution, 40 pressure iterations + bloom → expect 30+ FPS
## Debugging Tips
- Use `u_showVelocity` and `u_showCurl` flags in composite shader to visualize intermediate fields
- Check WebGL error log after each render pass: `gl.getError()`
- Verify texture coordinates are in [0, 1] range (invSize calculations often source of bugs)
- Use small splat radius during development (easier to see what's happening)
- Rasterize shader output to canvas temporarily during debugging to inspect intermediate results