14 KiB
Phase 2: Fluid Render Graph — Shader Implementation Guide
Status: In Progress (Infrastructure ✅, Shaders 🚧)
Date Started: 2026-04-17
Last Updated: 2026-04-17
Overview
The fluid-render-graph.js overlay provides FBO infrastructure, quality tier configuration, and audio mapping logic. This guide documents the 10 GLSL shader programs that must be implemented to complete Phase 2 rendering.
Architecture Review
Execution Order
Each frame, the FluidSimulator.step(audioFeatures) method executes render passes in this sequence:
- Splat — Inject audio-driven forces into velocity field
- Curl — Compute vorticity from velocity
- Divergence — Compute divergence of velocity field
- Pressure Jacobi (loop N times per quality tier)
- Gradient Subtract — Remove pressure gradient from velocity
- Advect Velocity — Self-advection of velocity field
- Advect Dye — Carry colors downstream
- (Optional) Bloom Downsample — 2-tap downsampling (balanced/high only)
- (Optional) Bloom Composite — Upsample + accumulate (balanced/high only)
- Composite — Final render to canvas
FBO Layout
Ping-Pong Pairs (swapped each frame):
- velocity[0], velocity[1] [vec4: vx, vy, divx, divy]
- dye[0], dye[1] [vec4: r, g, b, a]
- divergence[0], divergence[1] [float: div]
- pressure[0], pressure[1] [float: p]
- curl[0], curl[1] [float: ω_z (2D vorticity)]
Bloom Chain (balanced/high only):
- bloomDown1, bloomDown2 [quarter-res downsamples]
- bloomUp1 [upsampled for composite]
Shader Programs
1. Splat Shader
Purpose: Inject audio-driven forces into velocity field.
Input Audio Mapping:
bass→ Force magnitude & radius (low frequencies = diffuse spread)beat→ Impulse splat (percussion pulse = concentrated force)treble→ Turbulence injection (high frequencies = chaotic rotation)
Uniforms:
uniform sampler2D u_velocity; // Ping-pong read (velocity[i-1])
uniform vec2 u_splatPos; // Canvas center or random offset
uniform float u_splatRadius; // Gaussian kernel radius [0.5..2.0] * canvas_scale
uniform float u_splatForce; // Force magnitude [0..10]
uniform bool u_splatIsBeat; // Beat-driven impulse? (concentrated)
uniform float u_turbulenceAmplitude; // Treble-driven vorticity [0..1]
Algorithm:
vec2 pos = gl_FragCoord.xy;
float dist = length(pos - u_splatPos);
float kernel = exp(-dist * dist / (u_splatRadius * u_splatRadius));
vec2 splatVel;
if (u_splatIsBeat) {
// Concentrated impulse for percussion
splatVel = normalize(u_splatPos - pos) * u_splatForce * kernel;
} else {
// Diffuse spread for bass
splatVel = (u_splatPos - pos) * u_splatForce * kernel * 0.1;
}
// Rotate splat velocity by treble-driven angle
float angle = u_turbulenceAmplitude * sin(length(pos) * 0.01) * 3.14159;
splatVel = vec2(
splatVel.x * cos(angle) - splatVel.y * sin(angle),
splatVel.x * sin(angle) + splatVel.y * cos(angle)
);
vec2 vel = texture2D(u_velocity, uv).xy;
gl_FragColor = vec4(vel + splatVel, 0.0, 1.0);
Call Site (in FluidSimulator.splat()):
const audioFeatures = window.__STEMENGINE_MUON_AUDIO__
const bass = audioFeatures.bass // [0..1]
const beat = audioFeatures.beat // boolean
const treble = audioFeatures.treble // [0..1]
const splatRadius = 50 + bass * 100 // Spread with bass: [50..150] px
const splatForce = 10 + bass * 20 // Force with bass: [10..30]
const turbulence = treble * 1.5 // Amplitude: [0..1.5]
// Render splat pass with these uniforms
2. Curl Shader
Purpose: Compute 2D vorticity (rotation) from velocity field.
Formula: ω_z = ∂v_y/∂x - ∂v_x/∂y (central differences)
Uniforms:
uniform sampler2D u_velocity;
uniform vec2 u_invSize; // 1.0 / vec2(texW, texH)
Algorithm:
float n = texture2D(u_velocity, uv + vec2(0.0, u_invSize.y)).y;
float s = texture2D(u_velocity, uv - vec2(0.0, u_invSize.y)).y;
float e = texture2D(u_velocity, uv + vec2(u_invSize.x, 0.0)).x;
float w = texture2D(u_velocity, uv - vec2(u_invSize.x, 0.0)).x;
float curl = (e - w) - (n - s);
gl_FragColor = vec4(curl, 0.0, 0.0, 1.0);
3. Divergence Shader
Purpose: Compute divergence of velocity field (∇·v).
Formula: div = ∂v_x/∂x + ∂v_y/∂y (central differences)
Uniforms:
uniform sampler2D u_velocity;
uniform vec2 u_invSize;
Algorithm:
float n = texture2D(u_velocity, uv + vec2(0.0, u_invSize.y)).y;
float s = texture2D(u_velocity, uv - vec2(0.0, u_invSize.y)).y;
float e = texture2D(u_velocity, uv + vec2(u_invSize.x, 0.0)).x;
float w = texture2D(u_velocity, uv - vec2(u_invSize.x, 0.0)).x;
float div = 0.5 * ((e - w) + (n - s));
gl_FragColor = vec4(div, 0.0, 0.0, 1.0);
4. Pressure Jacobi Shader
Purpose: Iterative pressure solver using Jacobi method.
Converges to: ∇²p = -div(v)
Iterations per tier: Low=10, Balanced=20, High=40
Uniforms:
uniform sampler2D u_pressure; // Ping-pong read
uniform sampler2D u_divergence;
uniform float u_alpha; // -1.0 / 4.0
uniform float u_rBeta; // 0.25 (1/4)
uniform vec2 u_invSize;
Algorithm:
float n = texture2D(u_pressure, uv + vec2(0.0, u_invSize.y)).x;
float s = texture2D(u_pressure, uv - vec2(0.0, u_invSize.y)).x;
float e = texture2D(u_pressure, uv + vec2(u_invSize.x, 0.0)).x;
float w = texture2D(u_pressure, uv - vec2(u_invSize.x, 0.0)).x;
float div = texture2D(u_divergence, uv).x;
float p = (n + s + e + w + u_alpha * div) * u_rBeta;
gl_FragColor = vec4(p, 0.0, 0.0, 1.0);
5. Gradient Subtract Shader
Purpose: Subtract pressure gradient from velocity (enforces incompressibility).
Formula: v_new = v - ∇p (central difference gradient)
Uniforms:
uniform sampler2D u_velocity;
uniform sampler2D u_pressure;
uniform vec2 u_invSize;
uniform float u_gradientScale; // 1.0 (or scaled for stability)
Algorithm:
float n = texture2D(u_pressure, uv + vec2(0.0, u_invSize.y)).x;
float s = texture2D(u_pressure, uv - vec2(0.0, u_invSize.y)).x;
float e = texture2D(u_pressure, uv + vec2(u_invSize.x, 0.0)).x;
float w = texture2D(u_pressure, uv - vec2(u_invSize.x, 0.0)).x;
vec2 grad = u_gradientScale * vec2(e - w, n - s);
vec2 vel = texture2D(u_velocity, uv).xy;
gl_FragColor = vec4(vel - grad, 0.0, 1.0);
6. Advect Velocity Shader
Purpose: Self-advect velocity field along its own streamlines.
Formula: v(t+dt) = v(t + dt*∇x(t)) (semi-Lagrangian backtracking)
Uniforms:
uniform sampler2D u_velocity;
uniform vec2 u_invSize;
uniform float u_dt; // Time step [0.016 / 1000] ≈ 0.00001..0.01
uniform float u_decay; // [0.99..0.995] for damping
Algorithm:
vec2 vel = texture2D(u_velocity, uv).xy;
vec2 backPos = uv - u_dt * vel * u_invSize;
// Clamp to valid texture range [0, 1]
backPos = clamp(backPos, u_invSize, 1.0 - u_invSize);
vec2 advectedVel = texture2D(u_velocity, backPos).xy * u_decay;
gl_FragColor = vec4(advectedVel, 0.0, 1.0);
7. Advect Dye Shader
Purpose: Carry color/dye downstream along velocity streamlines.
Maps audio features to dye color:
bass→ Red channel (low frequencies = warm)treble→ Blue channel (high frequencies = cool)rms→ Green channel (total energy = brightness)
Uniforms:
uniform sampler2D u_dye;
uniform sampler2D u_velocity;
uniform vec2 u_invSize;
uniform float u_dt;
uniform float u_decay; // [0.98..0.99] for dye persistence
uniform vec3 u_audioColor; // (r, g, b) from bass/treble/rms
Algorithm:
vec2 vel = texture2D(u_velocity, uv).xy;
vec2 backPos = uv - u_dt * vel * u_invSize;
backPos = clamp(backPos, u_invSize, 1.0 - u_invSize);
vec4 dye = texture2D(u_dye, backPos) * u_decay;
// Inject audio-driven color into this fragment
float mixFactor = 0.15; // Injection strength
dye.rgb += u_audioColor * mixFactor;
dye.rgb = clamp(dye.rgb, 0.0, 1.0);
gl_FragColor = dye;
Call Site (in FluidSimulator.advectDye()):
const audio = window.__STEMENGINE_MUON_AUDIO__
const audioColor = [
audio.bass, // Red channel
audio.rms, // Green channel
audio.treble, // Blue channel
]
// Pass audioColor as u_audioColor uniform
8. Bloom Downsample Shader
Purpose: 2-tap downsampling for bloom chain.
Only rendered on balanced/high quality tiers.
Uniforms:
uniform sampler2D u_source; // Full-res dye texture
uniform vec2 u_invSize;
Algorithm:
vec2 uv0 = uv - u_invSize * 0.5;
vec2 uv1 = uv + u_invSize * 0.5;
vec4 s0 = texture2D(u_source, vec2(uv0.x, uv0.y));
vec4 s1 = texture2D(u_source, vec2(uv1.x, uv0.y));
vec4 s2 = texture2D(u_source, vec2(uv0.x, uv1.y));
vec4 s3 = texture2D(u_source, vec2(uv1.x, uv1.y));
gl_FragColor = (s0 + s1 + s2 + s3) * 0.25;
9. Bloom Composite Shader
Purpose: Upsample and composite bloom chain back into full-res dye.
Creates soft glow/halo effect from bright regions.
Uniforms:
uniform sampler2D u_fullRes; // Full-res dye
uniform sampler2D u_bloomUp; // Upsampled bloom (from down→up)
uniform float u_bloomIntensity; // [0..2], maps from rms audio
Algorithm:
vec4 full = texture2D(u_fullRes, uv);
vec4 bloom = texture2D(u_bloomUp, uv);
// Additive composite with intensity control
gl_FragColor = full + bloom * u_bloomIntensity;
gl_FragColor.rgb = clamp(gl_FragColor.rgb, 0.0, 1.0);
Call Site (in FluidSimulator.bloomComposite()):
const audio = window.__STEMENGINE_MUON_AUDIO__
const bloomIntensity = 0.5 + audio.rms * 1.5 // [0.5..2.0]
// Pass as u_bloomIntensity uniform
10. Composite Shader
Purpose: Final render to canvas.
Outputs: Velocity curl as hue, magnitude as value (HSV colorspace for artistic effect).
Uniforms:
uniform sampler2D u_dye; // Advected color field
uniform sampler2D u_curl; // Vorticity field
uniform sampler2D u_velocity; // Velocity field
uniform bool u_showVelocity; // Debug mode flag (default: false)
uniform bool u_showCurl; // Debug mode flag (default: false)
Algorithm (Production):
// Normal mode: show dye
vec4 col = texture2D(u_dye, uv);
gl_FragColor = vec4(col.rgb, 1.0);
Algorithm (Debug):
// Show curl (vorticity) as grayscale
float c = texture2D(u_curl, uv).x;
float col = abs(c) * 10.0; // Amplify for visibility
gl_FragColor = vec4(vec3(col), 1.0);
Implementation Checklist
- Splat shader compiles, uniforms bind, output visible
- Curl shader computes correctly (validation: compare against CPU reference)
- Divergence shader computes correctly
- Pressure Jacobi shader converges (test: divergence should decrease after N iterations)
- Gradient Subtract shader enforces incompressibility
- Advect Velocity shader prevents fluid explosion (decay working)
- Advect Dye shader shows audio-reactive color injection
- Bloom Downsample shader outputs quarter-res texture
- Bloom Composite shader produces visible glow on balanced/high tiers
- Composite shader renders final image to canvas
- All 10 shaders linked into one WebGL program or individual programs
- FBOs allocated, ping-pong swapping works
- Quality tier iteration counts applied (low=10, balanced=20, high=40)
- Audio features (bass, beat, treble, rms) correctly injected into splat/advect
- Live performance: 60 FPS on desktop, graceful degration on mobile
- Style router instantiates FluidSimulator when style='fluid'
Integration Points
In fluid-render-graph.js:
Each method stub in FluidSimulator.prototype must:
- Create or reuse a WebGL program with the appropriate shader pair (vertex + fragment)
- Bind input textures to texture units
- Set uniforms from
this.width,this.height,this.quality, audio features - Bind output FBO
- Render fullscreen quad (use existing VAO from constructor)
- Ensure ping-pong swapping happens (read from [i-1], write to [i])
In stemengine-router.js:
handleCommand('setStyle', {style: 'fluid'}) → updateRenderGraph() → new window.__StemEngineFluidRenderGraph__()
- Constructor detects WebGL context from canvas
- Allocates FBOs and compiles all 10 shader programs
- Hooks into animation loop via
requestAnimationFrame
In Dockerfile:
COPY fluid-render-graph.js /tmp/
RUN cp /tmp/fluid-render-graph.js /src/app/src/fluid-render-graph.js
RUN grep -q "fluid-render-graph" /src/app/src/index.js || sed -i '1i import "./fluid-render-graph";' /src/app/src/index.js
Testing Strategy
Unit Tests (manual):
- Shader compilation: Verify each shader compiles without errors
- Uniform binding: Check all uniforms receive correct values
- Texture unit management: Verify no texture conflicts (max 8 units, carefully manage)
- FBO attachments: Test each FBO binds correctly and renders to its texture
Integration Tests:
- Full loop: Run step() for 60 frames, verify no WebGL errors
- Audio injection: Verify bass/beat/treble/rms values propagate to splat
- Bloom effectiveness: Compare balanced/high to low tier visually (bloom should appear on balanced+)
- Quality tier switching: Jacobi iteration counts should change on quality change
Performance Tests:
- Low tier: 256x256 resolution, 10 pressure iterations → expect 144+ FPS
- Balanced tier: 512x512 resolution, 20 pressure iterations + bloom → expect 60 FPS
- High tier: 1024x1024 resolution, 40 pressure iterations + bloom → expect 30+ FPS
Debugging Tips
- Use
u_showVelocityandu_showCurlflags in composite shader to visualize intermediate fields - Check WebGL error log after each render pass:
gl.getError() - Verify texture coordinates are in [0, 1] range (invSize calculations often source of bugs)
- Use small splat radius during development (easier to see what's happening)
- Rasterize shader output to canvas temporarily during debugging to inspect intermediate results