# Visualizer Iframe Protocol Date: 2026-04-28 Status: Phase 1 baseline frozen Owners: StemEngine frontend / iframe visualizer pipeline ## Purpose This document freezes the host-to-iframe transport contract used by StemEngine visualizer iframes. Goals: - preserve the current Muon iframe behavior that is already shipping, - define the provider-neutral contract that future iframe apps must implement, - make replay, seek, rate, resize, and local-upload behavior explicit before Phase 2 and Phase 3 refactors. Sources of truth for the current implementation: - `frontend/src/hooks/useMuonIframeTransport.ts` - `muon-iframe/stemengine-router.js` This is a protocol document, not a product API promise. Fields marked "planned" are reserved now so later iframe providers can implement them without renaming the contract. ## Transport Envelope All messages use the same envelope shape. ```ts type VisualizerIframeMessage = { version: 1 id: string type: string payload?: Record } ``` Rules: - `version` is required and is currently fixed at `1`. - `id` is caller-generated and is used for correlation and diagnostics only. - `type` selects the command or event. - `payload` is optional, but most commands and events provide one. - Unknown `type` values must be ignored. - Messages with a version other than `1` must be ignored. ## Lifecycle ### Boot sequence 1. Host mounts iframe. 2. Iframe initializes its audio engine and renderer runtime. 3. Iframe emits `ready`. 4. Host sends settings and paused state. 5. Host sends `loadAudio` or `loadAudioData`. 6. Iframe emits `loaded` after audio decode succeeds. 7. Host drives playback with `transportPlay`, `transportPause`, `transportSeek`, and `transportRate`. ### Visibility and teardown - Host may send `setPaused { paused: true }` when the document is hidden even if transport state remains "playing". - Host currently pauses transport on unmount and on inactive state transitions. - `dispose` is reserved for future explicit teardown, but iframe unmount remains the effective teardown path today. ## Host To Iframe Commands ### Implemented now #### `setStyle` Purpose: - Select visual style inside a multi-style iframe. Payload: ```ts { style: 'default' | 'fluid' | 'tendrils' | 'spirit' | string } ``` Semantics: - Required for the current Muon iframe. - Single-style providers may treat this as a no-op. #### `setPreset` Purpose: - Select a named renderer preset. Payload: ```ts { preset: string } ``` Semantics: - Current Muon router stores the value even if the renderer ignores it. - Future providers may expose preset lists through `capabilities`. #### `setQuality` Purpose: - Apply a generic quality tier. Payload: ```ts { quality: 'low' | 'balanced' | 'high' | string } ``` Semantics: - Providers map this to renderer-specific internals. #### `setPaused` Purpose: - Tell the iframe whether visual updates should behave as paused. Payload: ```ts { paused: boolean } ``` Semantics: - This is a renderer-visibility hint, not the authoritative audio transport command. - Host uses it for tab visibility and play/pause mirroring. #### `setViewport` Purpose: - Update CSS and pixel viewport dimensions. Payload: ```ts { cssWidth: number cssHeight: number dpr: number } ``` Compatibility note: - Current router also accepts legacy `width` and `height` aliases in the payload and derives pixel width and height internally. Semantics: - Viewport updates must not require iframe reload. - Providers are expected to clamp invalid or extreme values. #### `pushAudioFeatures` Purpose: - Push host-side extracted audio features into the iframe render loop. Payload: ```ts { bass?: number treble?: number rms?: number energy?: number beat?: boolean drumIntensity?: number sub?: number beatPhase?: number bpm?: number beatConfidence?: number isBeatDrop?: boolean isBuildUp?: boolean isSilence?: boolean energyDelta?: number spectralCentroid?: number spectralFlatness?: number frequencyBands?: unknown } ``` Semantics: - Host may send at RAF cadence. - Iframe should rate-limit internal application so message handling does not block rendering. #### `loadAudio` Purpose: - Load audio by URL inside the iframe. Payload: ```ts { url: string } ``` Semantics: - Used for same-origin or cross-origin URLs the iframe can fetch directly. #### `loadAudioData` Purpose: - Load audio from raw bytes when the iframe cannot fetch the source URL directly. Payload: ```ts { bytes: ArrayBuffer | Uint8Array | number[] } ``` Semantics: - This is the required path for host-local `blob:` uploads that do not survive a cross-origin iframe boundary. - The host currently uses this for Visualizer Lab local uploads. - Providers that support local uploads must support this command. #### `transportPlay` Purpose: - Start or resume transport at a host-selected time and rate. Payload: ```ts { time: number; rate: number } ``` Semantics: - This is the authoritative start/resume command. - Iframe must begin playback at `time` and apply `rate` atomically if possible. #### `transportPause` Purpose: - Pause transport. Payload: ```ts {} ``` Semantics: - This is the authoritative pause command. #### `transportSeek` Purpose: - Correct transport position without changing the selected source. Payload: ```ts { time: number; rate?: number } ``` Semantics: - Used for discontinuities only, not as a steady-state clock. - If the iframe is logically playing, seek must not leave it visually or audibly paused. - Current Muon router includes a recovery path: if the iframe is expected to be playing but the engine is stopped, a seek may trigger a single auto-resume using the supplied or current rate. #### `transportRate` Purpose: - Update playback rate while playing or paused. Payload: ```ts { rate: number } ``` Semantics: - Providers should update rate without forcing a reload. ### Reserved in Phase 1, to be implemented in later phases #### `setParam` Payload: ```ts { name: string; value: number | string | boolean } ``` Purpose: - Apply a renderer-specific tunable control. Status: - Reserved by this spec. - Not implemented in the current Muon router. #### `setPalette` Payload: ```ts { colors: string[] } ``` Purpose: - Apply host-selected palette colors to the renderer. Status: - Reserved by this spec. - Not implemented in the current Muon router. Palette rules: - Colors must be hex strings. - Providers may clamp or ignore unsupported slots. - Palette changes must not require reload. #### `dispose` Payload: ```ts {} ``` Purpose: - Provide an explicit teardown hook for future providers. Status: - Reserved by this spec. - Optional even after implementation because iframe unmount remains the primary teardown path. ## Iframe To Host Events ### Implemented now #### `ready` Purpose: - Signal that the iframe runtime is initialized and can receive settings and audio commands. Current payload shape: ```ts { capabilities?: { webgl?: boolean missingExtensions?: string[] audioMode?: string } styles?: string[] } ``` Semantics: - Host should treat `ready` as the gate for sending settings and audio load commands. - Host must tolerate provider-specific extra fields. #### `loaded` Purpose: - Confirm that the iframe decoded the current audio source. Payload: ```ts { duration: number } ``` Semantics: - `duration` may be `0` if unknown. - Host uses this to mark the iframe as loaded. #### `error` Purpose: - Report an iframe-side failure. Payload: ```ts { code: string; message: string } ``` Semantics: - `AUDIO_LOAD` is currently treated by the host as a non-actionable runtime warning and may be suppressed in the console. - Other error codes are diagnostic and should be logged. ### Reserved in Phase 1, to be implemented in later phases #### `capabilities` Purpose: - Describe renderer-specific presets, params, and palette support after initialization. Payload: ```ts { providerId?: string protocolVersion?: 1 presets: string[] params: Param[] palette: { supported: boolean slotCount: number } } ``` ```ts type Param = { name: string label: string type: 'number' | 'enum' | 'boolean' default: number | string | boolean min?: number max?: number step?: number options?: string[] } ``` Status: - Reserved by this spec. - Not emitted by the current Muon router. Semantics: - Host must tolerate this arriving after render start. - Host must also tolerate it never arriving and fall back to generic controls. ## Behavioral Requirements ### Replay and seek - Replay after pause or track restart must be deterministic. - `transportSeek` is for discontinuities, scrub actions, and drift correction beyond tolerance. - Steady playback must not emit continuous seek churn. - If a seek arrives while the host is logically playing, the iframe must remain able to resume visual and audio progression without a reload. ### Rate handling - `transportPlay` carries the starting rate. - `transportRate` carries subsequent rate changes. - `transportSeek` may also carry `rate` so a recovery path can resume using the correct speed. ### Resize and quality - `setViewport` and `setQuality` must be hot-applied. - Providers may rebuild internal framebuffers on viewport change, but must not require document reload. ### Audio source ownership - Host owns source selection. - Iframe owns decode, playback, and renderer-side consumption. - `loadAudioData` is the required compatibility bridge for host-local uploads. ## Provider-Neutral Rules - The host contract must not rely on Muon-specific naming beyond historical file names. - Multi-style providers may implement `setStyle`; single-style providers may ignore it. - All providers must understand the same envelope, core transport commands, and core events. - Future providers should emit `capabilities` and implement `setParam` and `setPalette`. ## Compatibility Matrix | Message | Current Muon iframe | Required for future providers | |---|---|---| | `setStyle` | Yes | Optional no-op for single-style providers | | `setPreset` | Yes | Yes | | `setQuality` | Yes | Yes | | `setPaused` | Yes | Yes | | `setViewport` | Yes | Yes | | `pushAudioFeatures` | Yes | Yes if renderer uses host-pushed features | | `loadAudio` | Yes | Yes | | `loadAudioData` | Yes | Yes | | `transportPlay` | Yes | Yes | | `transportPause` | Yes | Yes | | `transportSeek` | Yes | Yes | | `transportRate` | Yes | Yes | | `setParam` | Reserved | Yes, after Phase 3 | | `setPalette` | Reserved | Yes, after Phase 3 | | `dispose` | Reserved | Optional | | `ready` | Yes | Yes | | `loaded` | Yes | Yes | | `error` | Yes | Yes | | `capabilities` | Reserved | Yes, after Phase 3 | ## Validation Baseline For This Freeze The Phase 1 freeze is based on the currently implemented contract in: - `frontend/src/hooks/useMuonIframeTransport.ts` - `muon-iframe/stemengine-router.js` Phase 0 validation remains the behavioral baseline: - replay cycle 2 is reactive again, - steady playback no longer produces seek churn, - local uploads cross the host/iframe boundary via `loadAudioData`.