Files
stemengine_old/muon-iframe/PHASE2_SHADER_GUIDE.md
2026-04-21 16:24:11 +09:00

14 KiB

Phase 2: Fluid Render Graph — Shader Implementation Guide

Status: In Progress (Infrastructure ✅, Shaders 🚧)
Date Started: 2026-04-17
Last Updated: 2026-04-17

Overview

The fluid-render-graph.js overlay provides FBO infrastructure, quality tier configuration, and audio mapping logic. This guide documents the 10 GLSL shader programs that must be implemented to complete Phase 2 rendering.

Architecture Review

Execution Order

Each frame, the FluidSimulator.step(audioFeatures) method executes render passes in this sequence:

  1. Splat — Inject audio-driven forces into velocity field
  2. Curl — Compute vorticity from velocity
  3. Divergence — Compute divergence of velocity field
  4. Pressure Jacobi (loop N times per quality tier)
  5. Gradient Subtract — Remove pressure gradient from velocity
  6. Advect Velocity — Self-advection of velocity field
  7. Advect Dye — Carry colors downstream
  8. (Optional) Bloom Downsample — 2-tap downsampling (balanced/high only)
  9. (Optional) Bloom Composite — Upsample + accumulate (balanced/high only)
  10. Composite — Final render to canvas

FBO Layout

Ping-Pong Pairs (swapped each frame):
  - velocity[0], velocity[1]        [vec4: vx, vy, divx, divy]
  - dye[0], dye[1]                  [vec4: r, g, b, a]
  - divergence[0], divergence[1]    [float: div]
  - pressure[0], pressure[1]        [float: p]
  - curl[0], curl[1]                [float: ω_z (2D vorticity)]

Bloom Chain (balanced/high only):
  - bloomDown1, bloomDown2           [quarter-res downsamples]
  - bloomUp1                         [upsampled for composite]

Shader Programs

1. Splat Shader

Purpose: Inject audio-driven forces into velocity field.
Input Audio Mapping:

  • bass → Force magnitude & radius (low frequencies = diffuse spread)
  • beat → Impulse splat (percussion pulse = concentrated force)
  • treble → Turbulence injection (high frequencies = chaotic rotation)

Uniforms:

uniform sampler2D u_velocity;           // Ping-pong read (velocity[i-1])
uniform vec2 u_splatPos;                // Canvas center or random offset
uniform float u_splatRadius;            // Gaussian kernel radius [0.5..2.0] * canvas_scale
uniform float u_splatForce;             // Force magnitude [0..10]
uniform bool u_splatIsBeat;             // Beat-driven impulse? (concentrated)
uniform float u_turbulenceAmplitude;    // Treble-driven vorticity [0..1]

Algorithm:

vec2 pos = gl_FragCoord.xy;
float dist = length(pos - u_splatPos);
float kernel = exp(-dist * dist / (u_splatRadius * u_splatRadius));

vec2 splatVel;
if (u_splatIsBeat) {
  // Concentrated impulse for percussion
  splatVel = normalize(u_splatPos - pos) * u_splatForce * kernel;
} else {
  // Diffuse spread for bass
  splatVel = (u_splatPos - pos) * u_splatForce * kernel * 0.1;
}

// Rotate splat velocity by treble-driven angle
float angle = u_turbulenceAmplitude * sin(length(pos) * 0.01) * 3.14159;
splatVel = vec2(
  splatVel.x * cos(angle) - splatVel.y * sin(angle),
  splatVel.x * sin(angle) + splatVel.y * cos(angle)
);

vec2 vel = texture2D(u_velocity, uv).xy;
gl_FragColor = vec4(vel + splatVel, 0.0, 1.0);

Call Site (in FluidSimulator.splat()):

const audioFeatures = window.__STEMENGINE_MUON_AUDIO__
const bass = audioFeatures.bass       // [0..1]
const beat = audioFeatures.beat       // boolean
const treble = audioFeatures.treble   // [0..1]

const splatRadius = 50 + bass * 100   // Spread with bass: [50..150] px
const splatForce = 10 + bass * 20     // Force with bass: [10..30]
const turbulence = treble * 1.5       // Amplitude: [0..1.5]

// Render splat pass with these uniforms

2. Curl Shader

Purpose: Compute 2D vorticity (rotation) from velocity field.
Formula: ω_z = ∂v_y/∂x - ∂v_x/∂y (central differences)

Uniforms:

uniform sampler2D u_velocity;
uniform vec2 u_invSize;               // 1.0 / vec2(texW, texH)

Algorithm:

float n  = texture2D(u_velocity, uv + vec2(0.0, u_invSize.y)).y;
float s  = texture2D(u_velocity, uv - vec2(0.0, u_invSize.y)).y;
float e  = texture2D(u_velocity, uv + vec2(u_invSize.x, 0.0)).x;
float w  = texture2D(u_velocity, uv - vec2(u_invSize.x, 0.0)).x;

float curl = (e - w) - (n - s);
gl_FragColor = vec4(curl, 0.0, 0.0, 1.0);

3. Divergence Shader

Purpose: Compute divergence of velocity field (∇·v).
Formula: div = ∂v_x/∂x + ∂v_y/∂y (central differences)

Uniforms:

uniform sampler2D u_velocity;
uniform vec2 u_invSize;

Algorithm:

float n  = texture2D(u_velocity, uv + vec2(0.0, u_invSize.y)).y;
float s  = texture2D(u_velocity, uv - vec2(0.0, u_invSize.y)).y;
float e  = texture2D(u_velocity, uv + vec2(u_invSize.x, 0.0)).x;
float w  = texture2D(u_velocity, uv - vec2(u_invSize.x, 0.0)).x;

float div = 0.5 * ((e - w) + (n - s));
gl_FragColor = vec4(div, 0.0, 0.0, 1.0);

4. Pressure Jacobi Shader

Purpose: Iterative pressure solver using Jacobi method.
Converges to: ∇²p = -div(v)
Iterations per tier: Low=10, Balanced=20, High=40

Uniforms:

uniform sampler2D u_pressure;         // Ping-pong read
uniform sampler2D u_divergence;
uniform float u_alpha;                // -1.0 / 4.0
uniform float u_rBeta;                // 0.25 (1/4)
uniform vec2 u_invSize;

Algorithm:

float n  = texture2D(u_pressure, uv + vec2(0.0, u_invSize.y)).x;
float s  = texture2D(u_pressure, uv - vec2(0.0, u_invSize.y)).x;
float e  = texture2D(u_pressure, uv + vec2(u_invSize.x, 0.0)).x;
float w  = texture2D(u_pressure, uv - vec2(u_invSize.x, 0.0)).x;

float div = texture2D(u_divergence, uv).x;
float p = (n + s + e + w + u_alpha * div) * u_rBeta;
gl_FragColor = vec4(p, 0.0, 0.0, 1.0);

5. Gradient Subtract Shader

Purpose: Subtract pressure gradient from velocity (enforces incompressibility).
Formula: v_new = v - ∇p (central difference gradient)

Uniforms:

uniform sampler2D u_velocity;
uniform sampler2D u_pressure;
uniform vec2 u_invSize;
uniform float u_gradientScale;        // 1.0 (or scaled for stability)

Algorithm:

float n  = texture2D(u_pressure, uv + vec2(0.0, u_invSize.y)).x;
float s  = texture2D(u_pressure, uv - vec2(0.0, u_invSize.y)).x;
float e  = texture2D(u_pressure, uv + vec2(u_invSize.x, 0.0)).x;
float w  = texture2D(u_pressure, uv - vec2(u_invSize.x, 0.0)).x;

vec2 grad = u_gradientScale * vec2(e - w, n - s);
vec2 vel = texture2D(u_velocity, uv).xy;
gl_FragColor = vec4(vel - grad, 0.0, 1.0);

6. Advect Velocity Shader

Purpose: Self-advect velocity field along its own streamlines.
Formula: v(t+dt) = v(t + dt*∇x(t)) (semi-Lagrangian backtracking)

Uniforms:

uniform sampler2D u_velocity;
uniform vec2 u_invSize;
uniform float u_dt;                   // Time step [0.016 / 1000] ≈ 0.00001..0.01
uniform float u_decay;                // [0.99..0.995] for damping

Algorithm:

vec2 vel = texture2D(u_velocity, uv).xy;
vec2 backPos = uv - u_dt * vel * u_invSize;

// Clamp to valid texture range [0, 1]
backPos = clamp(backPos, u_invSize, 1.0 - u_invSize);

vec2 advectedVel = texture2D(u_velocity, backPos).xy * u_decay;
gl_FragColor = vec4(advectedVel, 0.0, 1.0);

7. Advect Dye Shader

Purpose: Carry color/dye downstream along velocity streamlines.
Maps audio features to dye color:

  • bass → Red channel (low frequencies = warm)
  • treble → Blue channel (high frequencies = cool)
  • rms → Green channel (total energy = brightness)

Uniforms:

uniform sampler2D u_dye;
uniform sampler2D u_velocity;
uniform vec2 u_invSize;
uniform float u_dt;
uniform float u_decay;                // [0.98..0.99] for dye persistence
uniform vec3 u_audioColor;            // (r, g, b) from bass/treble/rms

Algorithm:

vec2 vel = texture2D(u_velocity, uv).xy;
vec2 backPos = uv - u_dt * vel * u_invSize;
backPos = clamp(backPos, u_invSize, 1.0 - u_invSize);

vec4 dye = texture2D(u_dye, backPos) * u_decay;

// Inject audio-driven color into this fragment
float mixFactor = 0.15;  // Injection strength
dye.rgb += u_audioColor * mixFactor;
dye.rgb = clamp(dye.rgb, 0.0, 1.0);

gl_FragColor = dye;

Call Site (in FluidSimulator.advectDye()):

const audio = window.__STEMENGINE_MUON_AUDIO__
const audioColor = [
  audio.bass,      // Red channel
  audio.rms,       // Green channel
  audio.treble,    // Blue channel
]
// Pass audioColor as u_audioColor uniform

8. Bloom Downsample Shader

Purpose: 2-tap downsampling for bloom chain.
Only rendered on balanced/high quality tiers.

Uniforms:

uniform sampler2D u_source;           // Full-res dye texture
uniform vec2 u_invSize;

Algorithm:

vec2 uv0 = uv - u_invSize * 0.5;
vec2 uv1 = uv + u_invSize * 0.5;

vec4 s0 = texture2D(u_source, vec2(uv0.x, uv0.y));
vec4 s1 = texture2D(u_source, vec2(uv1.x, uv0.y));
vec4 s2 = texture2D(u_source, vec2(uv0.x, uv1.y));
vec4 s3 = texture2D(u_source, vec2(uv1.x, uv1.y));

gl_FragColor = (s0 + s1 + s2 + s3) * 0.25;

9. Bloom Composite Shader

Purpose: Upsample and composite bloom chain back into full-res dye.
Creates soft glow/halo effect from bright regions.

Uniforms:

uniform sampler2D u_fullRes;          // Full-res dye
uniform sampler2D u_bloomUp;          // Upsampled bloom (from down→up)
uniform float u_bloomIntensity;       // [0..2], maps from rms audio

Algorithm:

vec4 full = texture2D(u_fullRes, uv);
vec4 bloom = texture2D(u_bloomUp, uv);

// Additive composite with intensity control
gl_FragColor = full + bloom * u_bloomIntensity;
gl_FragColor.rgb = clamp(gl_FragColor.rgb, 0.0, 1.0);

Call Site (in FluidSimulator.bloomComposite()):

const audio = window.__STEMENGINE_MUON_AUDIO__
const bloomIntensity = 0.5 + audio.rms * 1.5  // [0.5..2.0]
// Pass as u_bloomIntensity uniform

10. Composite Shader

Purpose: Final render to canvas.
Outputs: Velocity curl as hue, magnitude as value (HSV colorspace for artistic effect).

Uniforms:

uniform sampler2D u_dye;              // Advected color field
uniform sampler2D u_curl;             // Vorticity field
uniform sampler2D u_velocity;         // Velocity field
uniform bool u_showVelocity;          // Debug mode flag (default: false)
uniform bool u_showCurl;              // Debug mode flag (default: false)

Algorithm (Production):

// Normal mode: show dye
vec4 col = texture2D(u_dye, uv);
gl_FragColor = vec4(col.rgb, 1.0);

Algorithm (Debug):

// Show curl (vorticity) as grayscale
float c = texture2D(u_curl, uv).x;
float col = abs(c) * 10.0;  // Amplify for visibility
gl_FragColor = vec4(vec3(col), 1.0);

Implementation Checklist

  • Splat shader compiles, uniforms bind, output visible
  • Curl shader computes correctly (validation: compare against CPU reference)
  • Divergence shader computes correctly
  • Pressure Jacobi shader converges (test: divergence should decrease after N iterations)
  • Gradient Subtract shader enforces incompressibility
  • Advect Velocity shader prevents fluid explosion (decay working)
  • Advect Dye shader shows audio-reactive color injection
  • Bloom Downsample shader outputs quarter-res texture
  • Bloom Composite shader produces visible glow on balanced/high tiers
  • Composite shader renders final image to canvas
  • All 10 shaders linked into one WebGL program or individual programs
  • FBOs allocated, ping-pong swapping works
  • Quality tier iteration counts applied (low=10, balanced=20, high=40)
  • Audio features (bass, beat, treble, rms) correctly injected into splat/advect
  • Live performance: 60 FPS on desktop, graceful degration on mobile
  • Style router instantiates FluidSimulator when style='fluid'

Integration Points

In fluid-render-graph.js:

Each method stub in FluidSimulator.prototype must:

  1. Create or reuse a WebGL program with the appropriate shader pair (vertex + fragment)
  2. Bind input textures to texture units
  3. Set uniforms from this.width, this.height, this.quality, audio features
  4. Bind output FBO
  5. Render fullscreen quad (use existing VAO from constructor)
  6. Ensure ping-pong swapping happens (read from [i-1], write to [i])

In stemengine-router.js:

handleCommand('setStyle', {style: 'fluid'}) → updateRenderGraph() → new window.__StemEngineFluidRenderGraph__()

  • Constructor detects WebGL context from canvas
  • Allocates FBOs and compiles all 10 shader programs
  • Hooks into animation loop via requestAnimationFrame

In Dockerfile:

COPY fluid-render-graph.js /tmp/
RUN cp /tmp/fluid-render-graph.js /src/app/src/fluid-render-graph.js
RUN grep -q "fluid-render-graph" /src/app/src/index.js || sed -i '1i import "./fluid-render-graph";' /src/app/src/index.js

Testing Strategy

Unit Tests (manual):

  1. Shader compilation: Verify each shader compiles without errors
  2. Uniform binding: Check all uniforms receive correct values
  3. Texture unit management: Verify no texture conflicts (max 8 units, carefully manage)
  4. FBO attachments: Test each FBO binds correctly and renders to its texture

Integration Tests:

  1. Full loop: Run step() for 60 frames, verify no WebGL errors
  2. Audio injection: Verify bass/beat/treble/rms values propagate to splat
  3. Bloom effectiveness: Compare balanced/high to low tier visually (bloom should appear on balanced+)
  4. Quality tier switching: Jacobi iteration counts should change on quality change

Performance Tests:

  1. Low tier: 256x256 resolution, 10 pressure iterations → expect 144+ FPS
  2. Balanced tier: 512x512 resolution, 20 pressure iterations + bloom → expect 60 FPS
  3. High tier: 1024x1024 resolution, 40 pressure iterations + bloom → expect 30+ FPS

Debugging Tips

  • Use u_showVelocity and u_showCurl flags in composite shader to visualize intermediate fields
  • Check WebGL error log after each render pass: gl.getError()
  • Verify texture coordinates are in [0, 1] range (invSize calculations often source of bugs)
  • Use small splat radius during development (easier to see what's happening)
  • Rasterize shader output to canvas temporarily during debugging to inspect intermediate results