423 lines
14 KiB
Markdown
423 lines
14 KiB
Markdown
# Phase 2: Fluid Render Graph — Shader Implementation Guide
|
|
|
|
**Status**: In Progress (Infrastructure ✅, Shaders 🚧)
|
|
**Date Started**: 2026-04-17
|
|
**Last Updated**: 2026-04-17
|
|
|
|
## Overview
|
|
|
|
The `fluid-render-graph.js` overlay provides FBO infrastructure, quality tier configuration, and audio mapping logic. This guide documents the **10 GLSL shader programs** that must be implemented to complete Phase 2 rendering.
|
|
|
|
## Architecture Review
|
|
|
|
### Execution Order
|
|
|
|
Each frame, the `FluidSimulator.step(audioFeatures)` method executes render passes in this sequence:
|
|
|
|
1. **Splat** — Inject audio-driven forces into velocity field
|
|
2. **Curl** — Compute vorticity from velocity
|
|
3. **Divergence** — Compute divergence of velocity field
|
|
4. **Pressure Jacobi** (loop N times per quality tier)
|
|
5. **Gradient Subtract** — Remove pressure gradient from velocity
|
|
6. **Advect Velocity** — Self-advection of velocity field
|
|
7. **Advect Dye** — Carry colors downstream
|
|
8. **(Optional) Bloom Downsample** — 2-tap downsampling (balanced/high only)
|
|
9. **(Optional) Bloom Composite** — Upsample + accumulate (balanced/high only)
|
|
10. **Composite** — Final render to canvas
|
|
|
|
### FBO Layout
|
|
|
|
```
|
|
Ping-Pong Pairs (swapped each frame):
|
|
- velocity[0], velocity[1] [vec4: vx, vy, divx, divy]
|
|
- dye[0], dye[1] [vec4: r, g, b, a]
|
|
- divergence[0], divergence[1] [float: div]
|
|
- pressure[0], pressure[1] [float: p]
|
|
- curl[0], curl[1] [float: ω_z (2D vorticity)]
|
|
|
|
Bloom Chain (balanced/high only):
|
|
- bloomDown1, bloomDown2 [quarter-res downsamples]
|
|
- bloomUp1 [upsampled for composite]
|
|
```
|
|
|
|
## Shader Programs
|
|
|
|
### 1. Splat Shader
|
|
|
|
**Purpose**: Inject audio-driven forces into velocity field.
|
|
**Input Audio Mapping**:
|
|
- `bass` → Force magnitude & radius (low frequencies = diffuse spread)
|
|
- `beat` → Impulse splat (percussion pulse = concentrated force)
|
|
- `treble` → Turbulence injection (high frequencies = chaotic rotation)
|
|
|
|
**Uniforms**:
|
|
```glsl
|
|
uniform sampler2D u_velocity; // Ping-pong read (velocity[i-1])
|
|
uniform vec2 u_splatPos; // Canvas center or random offset
|
|
uniform float u_splatRadius; // Gaussian kernel radius [0.5..2.0] * canvas_scale
|
|
uniform float u_splatForce; // Force magnitude [0..10]
|
|
uniform bool u_splatIsBeat; // Beat-driven impulse? (concentrated)
|
|
uniform float u_turbulenceAmplitude; // Treble-driven vorticity [0..1]
|
|
```
|
|
|
|
**Algorithm**:
|
|
```glsl
|
|
vec2 pos = gl_FragCoord.xy;
|
|
float dist = length(pos - u_splatPos);
|
|
float kernel = exp(-dist * dist / (u_splatRadius * u_splatRadius));
|
|
|
|
vec2 splatVel;
|
|
if (u_splatIsBeat) {
|
|
// Concentrated impulse for percussion
|
|
splatVel = normalize(u_splatPos - pos) * u_splatForce * kernel;
|
|
} else {
|
|
// Diffuse spread for bass
|
|
splatVel = (u_splatPos - pos) * u_splatForce * kernel * 0.1;
|
|
}
|
|
|
|
// Rotate splat velocity by treble-driven angle
|
|
float angle = u_turbulenceAmplitude * sin(length(pos) * 0.01) * 3.14159;
|
|
splatVel = vec2(
|
|
splatVel.x * cos(angle) - splatVel.y * sin(angle),
|
|
splatVel.x * sin(angle) + splatVel.y * cos(angle)
|
|
);
|
|
|
|
vec2 vel = texture2D(u_velocity, uv).xy;
|
|
gl_FragColor = vec4(vel + splatVel, 0.0, 1.0);
|
|
```
|
|
|
|
**Call Site** (in `FluidSimulator.splat()`):
|
|
```javascript
|
|
const audioFeatures = window.__STEMENGINE_MUON_AUDIO__
|
|
const bass = audioFeatures.bass // [0..1]
|
|
const beat = audioFeatures.beat // boolean
|
|
const treble = audioFeatures.treble // [0..1]
|
|
|
|
const splatRadius = 50 + bass * 100 // Spread with bass: [50..150] px
|
|
const splatForce = 10 + bass * 20 // Force with bass: [10..30]
|
|
const turbulence = treble * 1.5 // Amplitude: [0..1.5]
|
|
|
|
// Render splat pass with these uniforms
|
|
```
|
|
|
|
### 2. Curl Shader
|
|
|
|
**Purpose**: Compute 2D vorticity (rotation) from velocity field.
|
|
**Formula**: `ω_z = ∂v_y/∂x - ∂v_x/∂y` (central differences)
|
|
|
|
**Uniforms**:
|
|
```glsl
|
|
uniform sampler2D u_velocity;
|
|
uniform vec2 u_invSize; // 1.0 / vec2(texW, texH)
|
|
```
|
|
|
|
**Algorithm**:
|
|
```glsl
|
|
float n = texture2D(u_velocity, uv + vec2(0.0, u_invSize.y)).y;
|
|
float s = texture2D(u_velocity, uv - vec2(0.0, u_invSize.y)).y;
|
|
float e = texture2D(u_velocity, uv + vec2(u_invSize.x, 0.0)).x;
|
|
float w = texture2D(u_velocity, uv - vec2(u_invSize.x, 0.0)).x;
|
|
|
|
float curl = (e - w) - (n - s);
|
|
gl_FragColor = vec4(curl, 0.0, 0.0, 1.0);
|
|
```
|
|
|
|
### 3. Divergence Shader
|
|
|
|
**Purpose**: Compute divergence of velocity field (∇·v).
|
|
**Formula**: `div = ∂v_x/∂x + ∂v_y/∂y` (central differences)
|
|
|
|
**Uniforms**:
|
|
```glsl
|
|
uniform sampler2D u_velocity;
|
|
uniform vec2 u_invSize;
|
|
```
|
|
|
|
**Algorithm**:
|
|
```glsl
|
|
float n = texture2D(u_velocity, uv + vec2(0.0, u_invSize.y)).y;
|
|
float s = texture2D(u_velocity, uv - vec2(0.0, u_invSize.y)).y;
|
|
float e = texture2D(u_velocity, uv + vec2(u_invSize.x, 0.0)).x;
|
|
float w = texture2D(u_velocity, uv - vec2(u_invSize.x, 0.0)).x;
|
|
|
|
float div = 0.5 * ((e - w) + (n - s));
|
|
gl_FragColor = vec4(div, 0.0, 0.0, 1.0);
|
|
```
|
|
|
|
### 4. Pressure Jacobi Shader
|
|
|
|
**Purpose**: Iterative pressure solver using Jacobi method.
|
|
**Converges to**: ∇²p = -div(v)
|
|
**Iterations per tier**: Low=10, Balanced=20, High=40
|
|
|
|
**Uniforms**:
|
|
```glsl
|
|
uniform sampler2D u_pressure; // Ping-pong read
|
|
uniform sampler2D u_divergence;
|
|
uniform float u_alpha; // -1.0 / 4.0
|
|
uniform float u_rBeta; // 0.25 (1/4)
|
|
uniform vec2 u_invSize;
|
|
```
|
|
|
|
**Algorithm**:
|
|
```glsl
|
|
float n = texture2D(u_pressure, uv + vec2(0.0, u_invSize.y)).x;
|
|
float s = texture2D(u_pressure, uv - vec2(0.0, u_invSize.y)).x;
|
|
float e = texture2D(u_pressure, uv + vec2(u_invSize.x, 0.0)).x;
|
|
float w = texture2D(u_pressure, uv - vec2(u_invSize.x, 0.0)).x;
|
|
|
|
float div = texture2D(u_divergence, uv).x;
|
|
float p = (n + s + e + w + u_alpha * div) * u_rBeta;
|
|
gl_FragColor = vec4(p, 0.0, 0.0, 1.0);
|
|
```
|
|
|
|
### 5. Gradient Subtract Shader
|
|
|
|
**Purpose**: Subtract pressure gradient from velocity (enforces incompressibility).
|
|
**Formula**: `v_new = v - ∇p` (central difference gradient)
|
|
|
|
**Uniforms**:
|
|
```glsl
|
|
uniform sampler2D u_velocity;
|
|
uniform sampler2D u_pressure;
|
|
uniform vec2 u_invSize;
|
|
uniform float u_gradientScale; // 1.0 (or scaled for stability)
|
|
```
|
|
|
|
**Algorithm**:
|
|
```glsl
|
|
float n = texture2D(u_pressure, uv + vec2(0.0, u_invSize.y)).x;
|
|
float s = texture2D(u_pressure, uv - vec2(0.0, u_invSize.y)).x;
|
|
float e = texture2D(u_pressure, uv + vec2(u_invSize.x, 0.0)).x;
|
|
float w = texture2D(u_pressure, uv - vec2(u_invSize.x, 0.0)).x;
|
|
|
|
vec2 grad = u_gradientScale * vec2(e - w, n - s);
|
|
vec2 vel = texture2D(u_velocity, uv).xy;
|
|
gl_FragColor = vec4(vel - grad, 0.0, 1.0);
|
|
```
|
|
|
|
### 6. Advect Velocity Shader
|
|
|
|
**Purpose**: Self-advect velocity field along its own streamlines.
|
|
**Formula**: `v(t+dt) = v(t + dt*∇x(t))` (semi-Lagrangian backtracking)
|
|
|
|
**Uniforms**:
|
|
```glsl
|
|
uniform sampler2D u_velocity;
|
|
uniform vec2 u_invSize;
|
|
uniform float u_dt; // Time step [0.016 / 1000] ≈ 0.00001..0.01
|
|
uniform float u_decay; // [0.99..0.995] for damping
|
|
```
|
|
|
|
**Algorithm**:
|
|
```glsl
|
|
vec2 vel = texture2D(u_velocity, uv).xy;
|
|
vec2 backPos = uv - u_dt * vel * u_invSize;
|
|
|
|
// Clamp to valid texture range [0, 1]
|
|
backPos = clamp(backPos, u_invSize, 1.0 - u_invSize);
|
|
|
|
vec2 advectedVel = texture2D(u_velocity, backPos).xy * u_decay;
|
|
gl_FragColor = vec4(advectedVel, 0.0, 1.0);
|
|
```
|
|
|
|
### 7. Advect Dye Shader
|
|
|
|
**Purpose**: Carry color/dye downstream along velocity streamlines.
|
|
**Maps audio features to dye color**:
|
|
- `bass` → Red channel (low frequencies = warm)
|
|
- `treble` → Blue channel (high frequencies = cool)
|
|
- `rms` → Green channel (total energy = brightness)
|
|
|
|
**Uniforms**:
|
|
```glsl
|
|
uniform sampler2D u_dye;
|
|
uniform sampler2D u_velocity;
|
|
uniform vec2 u_invSize;
|
|
uniform float u_dt;
|
|
uniform float u_decay; // [0.98..0.99] for dye persistence
|
|
uniform vec3 u_audioColor; // (r, g, b) from bass/treble/rms
|
|
```
|
|
|
|
**Algorithm**:
|
|
```glsl
|
|
vec2 vel = texture2D(u_velocity, uv).xy;
|
|
vec2 backPos = uv - u_dt * vel * u_invSize;
|
|
backPos = clamp(backPos, u_invSize, 1.0 - u_invSize);
|
|
|
|
vec4 dye = texture2D(u_dye, backPos) * u_decay;
|
|
|
|
// Inject audio-driven color into this fragment
|
|
float mixFactor = 0.15; // Injection strength
|
|
dye.rgb += u_audioColor * mixFactor;
|
|
dye.rgb = clamp(dye.rgb, 0.0, 1.0);
|
|
|
|
gl_FragColor = dye;
|
|
```
|
|
|
|
**Call Site** (in `FluidSimulator.advectDye()`):
|
|
```javascript
|
|
const audio = window.__STEMENGINE_MUON_AUDIO__
|
|
const audioColor = [
|
|
audio.bass, // Red channel
|
|
audio.rms, // Green channel
|
|
audio.treble, // Blue channel
|
|
]
|
|
// Pass audioColor as u_audioColor uniform
|
|
```
|
|
|
|
### 8. Bloom Downsample Shader
|
|
|
|
**Purpose**: 2-tap downsampling for bloom chain.
|
|
**Only rendered on balanced/high quality tiers**.
|
|
|
|
**Uniforms**:
|
|
```glsl
|
|
uniform sampler2D u_source; // Full-res dye texture
|
|
uniform vec2 u_invSize;
|
|
```
|
|
|
|
**Algorithm**:
|
|
```glsl
|
|
vec2 uv0 = uv - u_invSize * 0.5;
|
|
vec2 uv1 = uv + u_invSize * 0.5;
|
|
|
|
vec4 s0 = texture2D(u_source, vec2(uv0.x, uv0.y));
|
|
vec4 s1 = texture2D(u_source, vec2(uv1.x, uv0.y));
|
|
vec4 s2 = texture2D(u_source, vec2(uv0.x, uv1.y));
|
|
vec4 s3 = texture2D(u_source, vec2(uv1.x, uv1.y));
|
|
|
|
gl_FragColor = (s0 + s1 + s2 + s3) * 0.25;
|
|
```
|
|
|
|
### 9. Bloom Composite Shader
|
|
|
|
**Purpose**: Upsample and composite bloom chain back into full-res dye.
|
|
**Creates soft glow/halo effect from bright regions**.
|
|
|
|
**Uniforms**:
|
|
```glsl
|
|
uniform sampler2D u_fullRes; // Full-res dye
|
|
uniform sampler2D u_bloomUp; // Upsampled bloom (from down→up)
|
|
uniform float u_bloomIntensity; // [0..2], maps from rms audio
|
|
```
|
|
|
|
**Algorithm**:
|
|
```glsl
|
|
vec4 full = texture2D(u_fullRes, uv);
|
|
vec4 bloom = texture2D(u_bloomUp, uv);
|
|
|
|
// Additive composite with intensity control
|
|
gl_FragColor = full + bloom * u_bloomIntensity;
|
|
gl_FragColor.rgb = clamp(gl_FragColor.rgb, 0.0, 1.0);
|
|
```
|
|
|
|
**Call Site** (in `FluidSimulator.bloomComposite()`):
|
|
```javascript
|
|
const audio = window.__STEMENGINE_MUON_AUDIO__
|
|
const bloomIntensity = 0.5 + audio.rms * 1.5 // [0.5..2.0]
|
|
// Pass as u_bloomIntensity uniform
|
|
```
|
|
|
|
### 10. Composite Shader
|
|
|
|
**Purpose**: Final render to canvas.
|
|
**Outputs**: Velocity curl as hue, magnitude as value (HSV colorspace for artistic effect).
|
|
|
|
**Uniforms**:
|
|
```glsl
|
|
uniform sampler2D u_dye; // Advected color field
|
|
uniform sampler2D u_curl; // Vorticity field
|
|
uniform sampler2D u_velocity; // Velocity field
|
|
uniform bool u_showVelocity; // Debug mode flag (default: false)
|
|
uniform bool u_showCurl; // Debug mode flag (default: false)
|
|
```
|
|
|
|
**Algorithm (Production)**:
|
|
```glsl
|
|
// Normal mode: show dye
|
|
vec4 col = texture2D(u_dye, uv);
|
|
gl_FragColor = vec4(col.rgb, 1.0);
|
|
```
|
|
|
|
**Algorithm (Debug)**:
|
|
```glsl
|
|
// Show curl (vorticity) as grayscale
|
|
float c = texture2D(u_curl, uv).x;
|
|
float col = abs(c) * 10.0; // Amplify for visibility
|
|
gl_FragColor = vec4(vec3(col), 1.0);
|
|
```
|
|
|
|
## Implementation Checklist
|
|
|
|
- [ ] Splat shader compiles, uniforms bind, output visible
|
|
- [ ] Curl shader computes correctly (validation: compare against CPU reference)
|
|
- [ ] Divergence shader computes correctly
|
|
- [ ] Pressure Jacobi shader converges (test: divergence should decrease after N iterations)
|
|
- [ ] Gradient Subtract shader enforces incompressibility
|
|
- [ ] Advect Velocity shader prevents fluid explosion (decay working)
|
|
- [ ] Advect Dye shader shows audio-reactive color injection
|
|
- [ ] Bloom Downsample shader outputs quarter-res texture
|
|
- [ ] Bloom Composite shader produces visible glow on balanced/high tiers
|
|
- [ ] Composite shader renders final image to canvas
|
|
- [ ] All 10 shaders linked into one WebGL program or individual programs
|
|
- [ ] FBOs allocated, ping-pong swapping works
|
|
- [ ] Quality tier iteration counts applied (low=10, balanced=20, high=40)
|
|
- [ ] Audio features (bass, beat, treble, rms) correctly injected into splat/advect
|
|
- [ ] Live performance: 60 FPS on desktop, graceful degration on mobile
|
|
- [ ] Style router instantiates FluidSimulator when style='fluid'
|
|
|
|
## Integration Points
|
|
|
|
### In `fluid-render-graph.js`:
|
|
|
|
Each method stub in `FluidSimulator.prototype` must:
|
|
1. Create or reuse a WebGL program with the appropriate shader pair (vertex + fragment)
|
|
2. Bind input textures to texture units
|
|
3. Set uniforms from `this.width`, `this.height`, `this.quality`, audio features
|
|
4. Bind output FBO
|
|
5. Render fullscreen quad (use existing VAO from constructor)
|
|
6. Ensure ping-pong swapping happens (read from [i-1], write to [i])
|
|
|
|
### In `stemengine-router.js`:
|
|
|
|
`handleCommand('setStyle', {style: 'fluid'})` → `updateRenderGraph()` → `new window.__StemEngineFluidRenderGraph__()`
|
|
- Constructor detects WebGL context from canvas
|
|
- Allocates FBOs and compiles all 10 shader programs
|
|
- Hooks into animation loop via `requestAnimationFrame`
|
|
|
|
### In Dockerfile:
|
|
|
|
```dockerfile
|
|
COPY fluid-render-graph.js /tmp/
|
|
RUN cp /tmp/fluid-render-graph.js /src/app/src/fluid-render-graph.js
|
|
RUN grep -q "fluid-render-graph" /src/app/src/index.js || sed -i '1i import "./fluid-render-graph";' /src/app/src/index.js
|
|
```
|
|
|
|
## Testing Strategy
|
|
|
|
### Unit Tests (manual):
|
|
1. **Shader compilation**: Verify each shader compiles without errors
|
|
2. **Uniform binding**: Check all uniforms receive correct values
|
|
3. **Texture unit management**: Verify no texture conflicts (max 8 units, carefully manage)
|
|
4. **FBO attachments**: Test each FBO binds correctly and renders to its texture
|
|
|
|
### Integration Tests:
|
|
1. **Full loop**: Run step() for 60 frames, verify no WebGL errors
|
|
2. **Audio injection**: Verify bass/beat/treble/rms values propagate to splat
|
|
3. **Bloom effectiveness**: Compare balanced/high to low tier visually (bloom should appear on balanced+)
|
|
4. **Quality tier switching**: Jacobi iteration counts should change on quality change
|
|
|
|
### Performance Tests:
|
|
1. **Low tier**: 256x256 resolution, 10 pressure iterations → expect 144+ FPS
|
|
2. **Balanced tier**: 512x512 resolution, 20 pressure iterations + bloom → expect 60 FPS
|
|
3. **High tier**: 1024x1024 resolution, 40 pressure iterations + bloom → expect 30+ FPS
|
|
|
|
## Debugging Tips
|
|
|
|
- Use `u_showVelocity` and `u_showCurl` flags in composite shader to visualize intermediate fields
|
|
- Check WebGL error log after each render pass: `gl.getError()`
|
|
- Verify texture coordinates are in [0, 1] range (invSize calculations often source of bugs)
|
|
- Use small splat radius during development (easier to see what's happening)
|
|
- Rasterize shader output to canvas temporarily during debugging to inspect intermediate results
|