Files
stemengine_old/docs/2026-04-08_Visualizer_Implementation_Checklist.md
T
2026-04-09 08:30:51 +09:00

8.4 KiB

Visualizer Integration — Implementation Checklist

Date: 2026-04-08
Source plan: 2026-04-08_Visualizer_Integration_Plan.md


0) Non-negotiable color policy

  • Use manifest.metadata.dynamic_palette as the visualizer color source of truth. (2026-04-08: useVisualizerPalette prefers manifest palette and is now wired into LyricsLivePreview + karaoke pages.)
  • Do not treat visualizer colors as free/random runtime state when manifest palette is present. (2026-04-08: Muon/Fluid renderers consume paletteRef; Fluid palette sampling replaces random color source.)
  • Apply the same palette policy to both muon and fluid modes.
  • Allow audio to modulate motion/intensity only; keep color drift constrained to palette family. (2026-04-08: Muon hue drift limited to seed-hue offset from treble; geometry/velocity driven by audio.)

Acceptance:

  • Two clients opening the same .sek render matching base visualizer color family. (2026-04-08: SingUI Playwright harness karaoke-sek-parity.spec.ts validates two clients on /karaoke with the same .sek produce matching manifest palette seed/highlight.)

1) Backend manifest + API surface

  • Ensure .sek export writes metadata.dynamic_palette into manifest.json.
  • Ensure metadata.dynamic_palette includes:
    • source
    • seed_color
    • background_color
    • text_color
    • highlight_color
    • word_highlight_color
    • speaker_palette
  • Ensure SEK manifest/list endpoints return manifest.metadata.dynamic_palette. (2026-04-08: list payload now includes meta_dynamic_palette; manifest endpoint already returns full manifest.)

Acceptance:

  • GET manifest payload includes metadata.dynamic_palette for newly exported packages. (Code path validated: export writes palette; manifest endpoint returns full manifest object.)

2) Frontend types and palette resolver

  • Add/verify dynamic_palette typing in SEK manifest response model. (2026-04-08: added shared SekDynamicPalette type; applied to both SekManifestResponse.manifest.metadata.dynamic_palette and SekLibraryItem.meta_dynamic_palette.)
  • Implement useVisualizerPalette hook:
    • Input: sekManifest?.manifest?.metadata?.dynamic_palette, coverArtUrl, thumbnailUrl
    • Output: stable paletteRef
  • Fallback chain:
    1. Manifest palette
    2. Runtime extraction from cover/thumbnail
    3. Static neutral defaults

Acceptance:

  • Missing manifest palette does not break rendering. (hook falls back to runtime extraction/default palette)
  • Runtime fallback only used when manifest palette is absent.

3) Audio feature layer

  • Implement/verify useAudioFeatures with analyser + Meyda. (2026-04-08: hook added with Meyda + FFT fallback path.)
  • Keep feature transport in featuresRef (no 60fps React state updates).
  • Expose at least: bass, treble, rms, beatDetected, drumIntensity. (also exposes energy, loudness, spectralCentroid, freqData)

Acceptance:

  • No double-audio output. (analyser tap is routed through silent gain node)
  • Cleanup releases analyser/Meyda resources. (disconnect + analyzer stop implemented in hook cleanup)

4) Muon visualizer integration

  • Add paletteRef input to MuonVisualizerCanvas.
  • Map drumIntensity to geometry controls (radiusMultiplier, aperture, spacing).
  • Seed color from manifest palette (seed_color / highlight_color).
  • Restrict hue drift to small offset around seed (treble-driven).
  • Keep fallback to defaults if palette unavailable.

Acceptance:

  • Muon keeps package-consistent color identity while reacting to audio.

5) Fluid visualizer integration

  • Add paletteRef input to FluidVisualizerCanvas.
  • Replace random color generation with palette sampling (speaker_palette, highlight_color, word_highlight_color).
  • Keep beat/rms driving splat force and density behavior.

Acceptance:

  • Fluid colors are deterministic per package, not random per session.

6) Backdrop switcher + LyricsLivePreview wiring

  • Add paletteRef to VisualizerBackdrop props and forward to both modes.
  • Add useVisualizerPalette call in LyricsLivePreview. (2026-04-08: hook now invoked in LyricsLivePreview with optional sekDynamicPalette + cover/thumbnail fallbacks.)
  • Ensure backdrop renders under lyrics layer (z-index order preserved).
  • Extend style union with muon | fluid and validate guards. (2026-04-08: LyricsVisualStyle + saved-style allowlist updated in LyricsLivePreview.)
  • Add style-picker options for Muon and Fluid. (2026-04-08: added renderer selector buttons in style panel: Karaoke/Video/Muon/Fluid.)

Acceptance:

  • Switching styles does not reset playback or break lyric rendering. (2026-04-08: Playwright regression lyrics-live-preview.spec.ts verifies playback continues and lyrics remain visible while switching Karaoke → Muon → Fluid.)

7) Visualizer settings persistence

  • Keep manifest.visualizer for mode + geometry/performance settings. (2026-04-08: SEK export now writes normalized manifest.visualizer from karaoke config with mode/quality/fps and muon/fluid geometry/intensity controls.)
  • Keep color data in manifest.metadata.dynamic_palette. (No visual color fields were added to manifest.visualizer; colors remain in metadata.dynamic_palette only.)
  • Avoid duplicate/conflicting color settings in manifest.visualizer.

Acceptance:

  • Package stores one authoritative color definition path.

8) Performance and safety checks

  • Lazy-load heavy visualizer bundles where possible. (2026-04-08: VisualizerBackdrop now lazy-loads Muon/Fluid canvases with React.lazy + Suspense.)
  • Confirm GPU/CPU use remains acceptable in fullscreen and windowed modes. (2026-04-08: Playwright frame-cadence smoke test verifies healthy RAF cadence in both windowed and fullscreen visualizer preview.)
  • Verify graceful behavior when WebGL is unavailable. (Current visualizers are Canvas2D-based and bail out cleanly if context init fails.)

Acceptance:

  • No regressions in non-visualizer lyric modes. (2026-04-08: preview regression starts in standard live-preview flow, then switches renderers while transport/lyrics continue without reset.)

9) QA matrix

  • .sek with full dynamic_palette → Muon colors match manifest. (SingUI Playwright harness: /karaoke asserts data-palette-source=manifest, seed #123456 in Muon mode.)
  • .sek with full dynamic_palette → Fluid colors match manifest. (SingUI Playwright harness: /karaoke asserts data-palette-source=manifest, seed #123456 in Fluid mode.)
  • Legacy .sek (no palette) → runtime fallback path works. (SingUI Playwright harness: legacy-with-cover resolves runtime-artwork; no-cover resolves default.)
  • No-cover/no-thumbnail project → static default fallback works. (2026-04-08: Playwright regression verifies data-palette-source=default and default seed color when cover art is unavailable.)
  • TV/performance mode renders same color policy as standard mode. (SingUI Playwright harness: /karaoke/tv parity assertion matches /karaoke palette source/seed/highlight for the same .sek.)

10) Definition of done

  • Backend emits and serves manifest.metadata.dynamic_palette.
  • Frontend consumes palette via paletteRef in both visualizers.
  • Color behavior is deterministic per package.
  • Audio reactivity remains smooth and independent from color source policy. (Playback continuity test under renderer switching passes; SEK-route parity tests confirm identical color policy behavior independent from audio-driven mode dynamics.)
  • Checklist and implementation notes are committed.

Validation evidence captured:

  • Frontend typecheck passes. (npm run typecheck)
  • Backend regression passes. (python3 -m unittest backend.tests.test_karaoke_sek_manifest_export -v)
  • Focused Playwright regressions pass. (npx playwright test e2e/lyrics-live-preview.spec.ts --grep "switching renderers keeps playback running|artwork palette stays consistent across clients and renderers|visualizer palette falls back to static defaults without artwork")
  • SingUI .sek karaoke/TV parity harness passes. (npx playwright test -c playwright.sing.config.ts e2e/karaoke-sek-parity.spec.ts)