11 KiB
Visualizer Iframe Protocol
Date: 2026-04-28 Status: Phase 1 baseline frozen Owners: StemEngine frontend / iframe visualizer pipeline
Purpose
This document freezes the host-to-iframe transport contract used by StemEngine visualizer iframes.
Goals:
- preserve the current Muon iframe behavior that is already shipping,
- define the provider-neutral contract that future iframe apps must implement,
- make replay, seek, rate, resize, and local-upload behavior explicit before Phase 2 and Phase 3 refactors.
Sources of truth for the current implementation:
frontend/src/hooks/useMuonIframeTransport.tsmuon-iframe/stemengine-router.js
This is a protocol document, not a product API promise. Fields marked "planned" are reserved now so later iframe providers can implement them without renaming the contract.
Transport Envelope
All messages use the same envelope shape.
type VisualizerIframeMessage = {
version: 1
id: string
type: string
payload?: Record<string, unknown>
}
Rules:
versionis required and is currently fixed at1.idis caller-generated and is used for correlation and diagnostics only.typeselects the command or event.payloadis optional, but most commands and events provide one.- Unknown
typevalues must be ignored. - Messages with a version other than
1must be ignored.
Lifecycle
Boot sequence
- Host mounts iframe.
- Iframe initializes its audio engine and renderer runtime.
- Iframe emits
ready. - Host sends settings and paused state.
- Host sends
loadAudioorloadAudioData. - Iframe emits
loadedafter audio decode succeeds. - Host drives playback with
transportPlay,transportPause,transportSeek, andtransportRate.
Visibility and teardown
- Host may send
setPaused { paused: true }when the document is hidden even if transport state remains "playing". - Host currently pauses transport on unmount and on inactive state transitions.
disposeis reserved for future explicit teardown, but iframe unmount remains the effective teardown path today.
Host To Iframe Commands
Implemented now
setStyle
Purpose:
- Select visual style inside a multi-style iframe.
Payload:
{ style: 'default' | 'fluid' | 'tendrils' | 'spirit' | string }
Semantics:
- Required for the current Muon iframe.
- Single-style providers may treat this as a no-op.
setPreset
Purpose:
- Select a named renderer preset.
Payload:
{ preset: string }
Semantics:
- Current Muon router stores the value even if the renderer ignores it.
- Future providers may expose preset lists through
capabilities.
setQuality
Purpose:
- Apply a generic quality tier.
Payload:
{ quality: 'low' | 'balanced' | 'high' | string }
Semantics:
- Providers map this to renderer-specific internals.
setPaused
Purpose:
- Tell the iframe whether visual updates should behave as paused.
Payload:
{ paused: boolean }
Semantics:
- This is a renderer-visibility hint, not the authoritative audio transport command.
- Host uses it for tab visibility and play/pause mirroring.
setViewport
Purpose:
- Update CSS and pixel viewport dimensions.
Payload:
{
cssWidth: number
cssHeight: number
dpr: number
}
Compatibility note:
- Current router also accepts legacy
widthandheightaliases in the payload and derives pixel width and height internally.
Semantics:
- Viewport updates must not require iframe reload.
- Providers are expected to clamp invalid or extreme values.
pushAudioFeatures
Purpose:
- Push host-side extracted audio features into the iframe render loop.
Payload:
{
bass?: number
treble?: number
rms?: number
energy?: number
beat?: boolean
drumIntensity?: number
sub?: number
beatPhase?: number
bpm?: number
beatConfidence?: number
isBeatDrop?: boolean
isBuildUp?: boolean
isSilence?: boolean
energyDelta?: number
spectralCentroid?: number
spectralFlatness?: number
frequencyBands?: unknown
}
Semantics:
- Host may send at RAF cadence.
- Iframe should rate-limit internal application so message handling does not block rendering.
loadAudio
Purpose:
- Load audio by URL inside the iframe.
Payload:
{ url: string }
Semantics:
- Used for same-origin or cross-origin URLs the iframe can fetch directly.
loadAudioData
Purpose:
- Load audio from raw bytes when the iframe cannot fetch the source URL directly.
Payload:
{ bytes: ArrayBuffer | Uint8Array | number[] }
Semantics:
- This is the required path for host-local
blob:uploads that do not survive a cross-origin iframe boundary. - The host currently uses this for Visualizer Lab local uploads.
- Providers that support local uploads must support this command.
transportPlay
Purpose:
- Start or resume transport at a host-selected time and rate.
Payload:
{ time: number; rate: number }
Semantics:
- This is the authoritative start/resume command.
- Iframe must begin playback at
timeand applyrateatomically if possible.
transportPause
Purpose:
- Pause transport.
Payload:
{}
Semantics:
- This is the authoritative pause command.
transportSeek
Purpose:
- Correct transport position without changing the selected source.
Payload:
{ time: number; rate?: number }
Semantics:
- Used for discontinuities only, not as a steady-state clock.
- If the iframe is logically playing, seek must not leave it visually or audibly paused.
- Current Muon router includes a recovery path: if the iframe is expected to be playing but the engine is stopped, a seek may trigger a single auto-resume using the supplied or current rate.
transportRate
Purpose:
- Update playback rate while playing or paused.
Payload:
{ rate: number }
Semantics:
- Providers should update rate without forcing a reload.
Reserved in Phase 1, to be implemented in later phases
setParam
Payload:
{ name: string; value: number | string | boolean }
Purpose:
- Apply a renderer-specific tunable control.
Status:
- Reserved by this spec.
- Not implemented in the current Muon router.
setPalette
Payload:
{ colors: string[] }
Purpose:
- Apply host-selected palette colors to the renderer.
Status:
- Reserved by this spec.
- Not implemented in the current Muon router.
Palette rules:
- Colors must be hex strings.
- Providers may clamp or ignore unsupported slots.
- Palette changes must not require reload.
dispose
Payload:
{}
Purpose:
- Provide an explicit teardown hook for future providers.
Status:
- Reserved by this spec.
- Optional even after implementation because iframe unmount remains the primary teardown path.
Iframe To Host Events
Implemented now
ready
Purpose:
- Signal that the iframe runtime is initialized and can receive settings and audio commands.
Current payload shape:
{
capabilities?: {
webgl?: boolean
missingExtensions?: string[]
audioMode?: string
}
styles?: string[]
}
Semantics:
- Host should treat
readyas the gate for sending settings and audio load commands. - Host must tolerate provider-specific extra fields.
loaded
Purpose:
- Confirm that the iframe decoded the current audio source.
Payload:
{ duration: number }
Semantics:
durationmay be0if unknown.- Host uses this to mark the iframe as loaded.
error
Purpose:
- Report an iframe-side failure.
Payload:
{ code: string; message: string }
Semantics:
AUDIO_LOADis currently treated by the host as a non-actionable runtime warning and may be suppressed in the console.- Other error codes are diagnostic and should be logged.
Reserved in Phase 1, to be implemented in later phases
capabilities
Purpose:
- Describe renderer-specific presets, params, and palette support after initialization.
Payload:
{
providerId?: string
protocolVersion?: 1
presets: string[]
params: Param[]
palette: {
supported: boolean
slotCount: number
}
}
type Param = {
name: string
label: string
type: 'number' | 'enum' | 'boolean'
default: number | string | boolean
min?: number
max?: number
step?: number
options?: string[]
}
Status:
- Reserved by this spec.
- Not emitted by the current Muon router.
Semantics:
- Host must tolerate this arriving after render start.
- Host must also tolerate it never arriving and fall back to generic controls.
Behavioral Requirements
Replay and seek
- Replay after pause or track restart must be deterministic.
transportSeekis for discontinuities, scrub actions, and drift correction beyond tolerance.- Steady playback must not emit continuous seek churn.
- If a seek arrives while the host is logically playing, the iframe must remain able to resume visual and audio progression without a reload.
Rate handling
transportPlaycarries the starting rate.transportRatecarries subsequent rate changes.transportSeekmay also carryrateso a recovery path can resume using the correct speed.
Resize and quality
setViewportandsetQualitymust be hot-applied.- Providers may rebuild internal framebuffers on viewport change, but must not require document reload.
Audio source ownership
- Host owns source selection.
- Iframe owns decode, playback, and renderer-side consumption.
loadAudioDatais the required compatibility bridge for host-local uploads.
Provider-Neutral Rules
- The host contract must not rely on Muon-specific naming beyond historical file names.
- Multi-style providers may implement
setStyle; single-style providers may ignore it. - All providers must understand the same envelope, core transport commands, and core events.
- Future providers should emit
capabilitiesand implementsetParamandsetPalette.
Compatibility Matrix
| Message | Current Muon iframe | Required for future providers |
|---|---|---|
setStyle |
Yes | Optional no-op for single-style providers |
setPreset |
Yes | Yes |
setQuality |
Yes | Yes |
setPaused |
Yes | Yes |
setViewport |
Yes | Yes |
pushAudioFeatures |
Yes | Yes if renderer uses host-pushed features |
loadAudio |
Yes | Yes |
loadAudioData |
Yes | Yes |
transportPlay |
Yes | Yes |
transportPause |
Yes | Yes |
transportSeek |
Yes | Yes |
transportRate |
Yes | Yes |
setParam |
Reserved | Yes, after Phase 3 |
setPalette |
Reserved | Yes, after Phase 3 |
dispose |
Reserved | Optional |
ready |
Yes | Yes |
loaded |
Yes | Yes |
error |
Yes | Yes |
capabilities |
Reserved | Yes, after Phase 3 |
Validation Baseline For This Freeze
The Phase 1 freeze is based on the currently implemented contract in:
frontend/src/hooks/useMuonIframeTransport.tsmuon-iframe/stemengine-router.js
Phase 0 validation remains the behavioral baseline:
- replay cycle 2 is reactive again,
- steady playback no longer produces seek churn,
- local uploads cross the host/iframe boundary via
loadAudioData.