Files
stemengine_old/docs/specs/visualizer-iframe-protocol.md
T
2026-04-30 13:50:11 +09:00

11 KiB

Visualizer Iframe Protocol

Date: 2026-04-28 Status: Phase 6 baseline (updated) Owners: StemEngine frontend / iframe visualizer pipeline

Purpose

This document freezes the host-to-iframe transport contract used by StemEngine visualizer iframes.

Goals:

  • preserve the current Muon iframe behavior that is already shipping,
  • define the provider-neutral contract that future iframe apps must implement,
  • make replay, seek, rate, resize, and local-upload behavior explicit before Phase 2 and Phase 3 refactors.

Sources of truth for the current implementation:

  • frontend/src/hooks/useVisualizerIframeTransport.ts
  • muon-iframe/stemengine-router.js

This is a protocol document, not a product API promise. Fields marked "planned" are reserved now so later iframe providers can implement them without renaming the contract.

Transport Envelope

All messages use the same envelope shape.

type VisualizerIframeMessage = {
  version: 1
  id: string
  type: string
  payload?: Record<string, unknown>
}

Rules:

  • version is required and is currently fixed at 1.
  • id is caller-generated and is used for correlation and diagnostics only.
  • type selects the command or event.
  • payload is optional, but most commands and events provide one.
  • Unknown type values must be ignored.
  • Messages with a version other than 1 must be ignored.

Lifecycle

Boot sequence

  1. Host mounts iframe.
  2. Iframe initializes its audio engine and renderer runtime.
  3. Iframe emits ready.
  4. Host sends settings and paused state.
  5. Host sends loadAudio or loadAudioData.
  6. Iframe emits loaded after audio decode succeeds.
  7. Host drives playback with transportPlay, transportPause, transportSeek, and transportRate.

Visibility and teardown

  • Host may send setPaused { paused: true } when the document is hidden even if transport state remains "playing".
  • Host currently pauses transport on unmount and on inactive state transitions.
  • dispose is reserved for future explicit teardown, but iframe unmount remains the effective teardown path today.

Host To Iframe Commands

Implemented now

setStyle

Purpose:

  • Select visual style inside a multi-style iframe.

Payload:

{ style: 'default' | 'fluid' | 'tendrils' | 'spirit' | string }

Semantics:

  • Required for the current Muon iframe.
  • Single-style providers may treat this as a no-op.

setPreset

Purpose:

  • Select a named renderer preset.

Payload:

{ preset: string }

Semantics:

  • Current Muon router stores the value even if the renderer ignores it.
  • Future providers may expose preset lists through capabilities.

setQuality

Purpose:

  • Apply a generic quality tier.

Payload:

{ quality: 'low' | 'balanced' | 'high' | string }

Semantics:

  • Providers map this to renderer-specific internals.

setPaused

Purpose:

  • Tell the iframe whether visual updates should behave as paused.

Payload:

{ paused: boolean }

Semantics:

  • This is a renderer-visibility hint, not the authoritative audio transport command.
  • Host uses it for tab visibility and play/pause mirroring.

setViewport

Purpose:

  • Update CSS and pixel viewport dimensions.

Payload:

{
  cssWidth: number
  cssHeight: number
  dpr: number
}

Compatibility note:

  • Current router also accepts legacy width and height aliases in the payload and derives pixel width and height internally.

Semantics:

  • Viewport updates must not require iframe reload.
  • Providers are expected to clamp invalid or extreme values.

pushAudioFeatures

Purpose:

  • Push host-side extracted audio features into the iframe render loop.

Payload:

{
  bass?: number
  treble?: number
  rms?: number
  energy?: number
  beat?: boolean
  drumIntensity?: number
  sub?: number
  beatPhase?: number
  bpm?: number
  beatConfidence?: number
  isBeatDrop?: boolean
  isBuildUp?: boolean
  isSilence?: boolean
  energyDelta?: number
  spectralCentroid?: number
  spectralFlatness?: number
  frequencyBands?: unknown
}

Semantics:

  • Host may send at RAF cadence.
  • Iframe should rate-limit internal application so message handling does not block rendering.

loadAudio

Purpose:

  • Load audio by URL inside the iframe.

Payload:

{ url: string }

Semantics:

  • Used for same-origin or cross-origin URLs the iframe can fetch directly.

loadAudioData

Purpose:

  • Load audio from raw bytes when the iframe cannot fetch the source URL directly.

Payload:

{ bytes: ArrayBuffer | Uint8Array | number[] }

Semantics:

  • This is the required path for host-local blob: uploads that do not survive a cross-origin iframe boundary.
  • The host currently uses this for Visualizer Lab local uploads.
  • Providers that support local uploads must support this command.

transportPlay

Purpose:

  • Start or resume transport at a host-selected time and rate.

Payload:

{ time: number; rate: number }

Semantics:

  • This is the authoritative start/resume command.
  • Iframe must begin playback at time and apply rate atomically if possible.

transportPause

Purpose:

  • Pause transport.

Payload:

{}

Semantics:

  • This is the authoritative pause command.

transportSeek

Purpose:

  • Correct transport position without changing the selected source.

Payload:

{ time: number; rate?: number }

Semantics:

  • Used for discontinuities only, not as a steady-state clock.
  • If the iframe is logically playing, seek must not leave it visually or audibly paused.
  • Current Muon router includes a recovery path: if the iframe is expected to be playing but the engine is stopped, a seek may trigger a single auto-resume using the supplied or current rate.

transportRate

Purpose:

  • Update playback rate while playing or paused.

Payload:

{ rate: number }

Semantics:

  • Providers should update rate without forcing a reload.

setParam

Payload:

{ name: string; value: number | string | boolean }

Purpose:

  • Apply a renderer-specific tunable control.

Semantics:

  • Host sends one message per param key/value.
  • Providers may ignore unknown param names.

setPalette

Payload:

{ colors: string[] }

Purpose:

  • Apply host-selected palette colors to the renderer.

Semantics:

  • Colors are passed as hex-like strings from the host.
  • Providers may clamp or ignore unsupported slots.
  • Palette changes must not require reload.

Reserved for future expansion

dispose

Payload:

{}

Purpose:

  • Provide an explicit teardown hook for future providers.

Status:

  • Reserved by this spec.
  • Optional even after implementation because iframe unmount remains the primary teardown path.

Iframe To Host Events

Implemented now

ready

Purpose:

  • Signal that the iframe runtime is initialized and can receive settings and audio commands.

Current payload shape:

{
  capabilities?: {
    webgl?: boolean
    missingExtensions?: string[]
    audioMode?: string
  }
  styles?: string[]
}

Semantics:

  • Host should treat ready as the gate for sending settings and audio load commands.
  • Host must tolerate provider-specific extra fields.

loaded

Purpose:

  • Confirm that the iframe decoded the current audio source.

Payload:

{ duration: number }

Semantics:

  • duration may be 0 if unknown.
  • Host uses this to mark the iframe as loaded.

error

Purpose:

  • Report an iframe-side failure.

Payload:

{ code: string; message: string }

Semantics:

  • AUDIO_LOAD is currently treated by the host as a non-actionable runtime warning and may be suppressed in the console.
  • Other error codes are diagnostic and should be logged.

capabilities

Purpose:

  • Describe renderer-specific presets, params, and palette support after initialization.

Payload:

{
  providerId?: string
  protocolVersion?: 1
  presets: string[]
  params: Param[]
  palette: {
    supported: boolean
    slotCount: number
  }
}
type Param = {
  name: string
  label: string
  type: 'number' | 'enum' | 'boolean'
  default: number | string | boolean
  min?: number
  max?: number
  step?: number
  options?: string[]
}

Semantics:

  • Host must tolerate this arriving after render start.
  • Host must also tolerate it never arriving and fall back to generic controls.

Reserved for future expansion

Behavioral Requirements

Replay and seek

  • Replay after pause or track restart must be deterministic.
  • transportSeek is for discontinuities, scrub actions, and drift correction beyond tolerance.
  • Steady playback must not emit continuous seek churn.
  • If a seek arrives while the host is logically playing, the iframe must remain able to resume visual and audio progression without a reload.

Rate handling

  • transportPlay carries the starting rate.
  • transportRate carries subsequent rate changes.
  • transportSeek may also carry rate so a recovery path can resume using the correct speed.

Resize and quality

  • setViewport and setQuality must be hot-applied.
  • Providers may rebuild internal framebuffers on viewport change, but must not require document reload.

Audio source ownership

  • Host owns source selection.
  • Iframe owns decode, playback, and renderer-side consumption.
  • loadAudioData is the required compatibility bridge for host-local uploads.

Provider-Neutral Rules

  • The host contract must not rely on Muon-specific naming beyond historical file names.
  • Multi-style providers may implement setStyle; single-style providers may ignore it.
  • All providers must understand the same envelope, core transport commands, and core events.
  • Future providers should emit capabilities and implement setParam and setPalette.

Compatibility Matrix

Message Current Muon iframe Required for future providers
setStyle Yes Optional no-op for single-style providers
setPreset Yes Yes
setQuality Yes Yes
setPaused Yes Yes
setViewport Yes Yes
pushAudioFeatures Yes Yes if renderer uses host-pushed features
loadAudio Yes Yes
loadAudioData Yes Yes
transportPlay Yes Yes
transportPause Yes Yes
transportSeek Yes Yes
transportRate Yes Yes
setParam Yes Yes
setPalette Yes Yes
dispose Reserved Optional
ready Yes Yes
loaded Yes Yes
error Yes Yes
capabilities Yes Yes

Validation Baseline

The current baseline is based on the implemented contract in:

  • frontend/src/hooks/useVisualizerIframeTransport.ts
  • muon-iframe/stemengine-router.js

Phase 0 validation remains the behavioral baseline:

  • replay cycle 2 is reactive again,
  • steady playback no longer produces seek churn,
  • local uploads cross the host/iframe boundary via loadAudioData.