Files
stemengine_old/docs/specs/visualizer-iframe-protocol.md
T
2026-04-30 13:50:11 +09:00

477 lines
11 KiB
Markdown

# Visualizer Iframe Protocol
Date: 2026-04-28
Status: Phase 6 baseline (updated)
Owners: StemEngine frontend / iframe visualizer pipeline
## Purpose
This document freezes the host-to-iframe transport contract used by StemEngine visualizer iframes.
Goals:
- preserve the current Muon iframe behavior that is already shipping,
- define the provider-neutral contract that future iframe apps must implement,
- make replay, seek, rate, resize, and local-upload behavior explicit before Phase 2 and Phase 3 refactors.
Sources of truth for the current implementation:
- `frontend/src/hooks/useVisualizerIframeTransport.ts`
- `muon-iframe/stemengine-router.js`
This is a protocol document, not a product API promise. Fields marked "planned" are reserved now so later iframe providers can implement them without renaming the contract.
## Transport Envelope
All messages use the same envelope shape.
```ts
type VisualizerIframeMessage = {
version: 1
id: string
type: string
payload?: Record<string, unknown>
}
```
Rules:
- `version` is required and is currently fixed at `1`.
- `id` is caller-generated and is used for correlation and diagnostics only.
- `type` selects the command or event.
- `payload` is optional, but most commands and events provide one.
- Unknown `type` values must be ignored.
- Messages with a version other than `1` must be ignored.
## Lifecycle
### Boot sequence
1. Host mounts iframe.
2. Iframe initializes its audio engine and renderer runtime.
3. Iframe emits `ready`.
4. Host sends settings and paused state.
5. Host sends `loadAudio` or `loadAudioData`.
6. Iframe emits `loaded` after audio decode succeeds.
7. Host drives playback with `transportPlay`, `transportPause`, `transportSeek`, and `transportRate`.
### Visibility and teardown
- Host may send `setPaused { paused: true }` when the document is hidden even if transport state remains "playing".
- Host currently pauses transport on unmount and on inactive state transitions.
- `dispose` is reserved for future explicit teardown, but iframe unmount remains the effective teardown path today.
## Host To Iframe Commands
### Implemented now
#### `setStyle`
Purpose:
- Select visual style inside a multi-style iframe.
Payload:
```ts
{ style: 'default' | 'fluid' | 'tendrils' | 'spirit' | string }
```
Semantics:
- Required for the current Muon iframe.
- Single-style providers may treat this as a no-op.
#### `setPreset`
Purpose:
- Select a named renderer preset.
Payload:
```ts
{ preset: string }
```
Semantics:
- Current Muon router stores the value even if the renderer ignores it.
- Future providers may expose preset lists through `capabilities`.
#### `setQuality`
Purpose:
- Apply a generic quality tier.
Payload:
```ts
{ quality: 'low' | 'balanced' | 'high' | string }
```
Semantics:
- Providers map this to renderer-specific internals.
#### `setPaused`
Purpose:
- Tell the iframe whether visual updates should behave as paused.
Payload:
```ts
{ paused: boolean }
```
Semantics:
- This is a renderer-visibility hint, not the authoritative audio transport command.
- Host uses it for tab visibility and play/pause mirroring.
#### `setViewport`
Purpose:
- Update CSS and pixel viewport dimensions.
Payload:
```ts
{
cssWidth: number
cssHeight: number
dpr: number
}
```
Compatibility note:
- Current router also accepts legacy `width` and `height` aliases in the payload and derives pixel width and height internally.
Semantics:
- Viewport updates must not require iframe reload.
- Providers are expected to clamp invalid or extreme values.
#### `pushAudioFeatures`
Purpose:
- Push host-side extracted audio features into the iframe render loop.
Payload:
```ts
{
bass?: number
treble?: number
rms?: number
energy?: number
beat?: boolean
drumIntensity?: number
sub?: number
beatPhase?: number
bpm?: number
beatConfidence?: number
isBeatDrop?: boolean
isBuildUp?: boolean
isSilence?: boolean
energyDelta?: number
spectralCentroid?: number
spectralFlatness?: number
frequencyBands?: unknown
}
```
Semantics:
- Host may send at RAF cadence.
- Iframe should rate-limit internal application so message handling does not block rendering.
#### `loadAudio`
Purpose:
- Load audio by URL inside the iframe.
Payload:
```ts
{ url: string }
```
Semantics:
- Used for same-origin or cross-origin URLs the iframe can fetch directly.
#### `loadAudioData`
Purpose:
- Load audio from raw bytes when the iframe cannot fetch the source URL directly.
Payload:
```ts
{ bytes: ArrayBuffer | Uint8Array | number[] }
```
Semantics:
- This is the required path for host-local `blob:` uploads that do not survive a cross-origin iframe boundary.
- The host currently uses this for Visualizer Lab local uploads.
- Providers that support local uploads must support this command.
#### `transportPlay`
Purpose:
- Start or resume transport at a host-selected time and rate.
Payload:
```ts
{ time: number; rate: number }
```
Semantics:
- This is the authoritative start/resume command.
- Iframe must begin playback at `time` and apply `rate` atomically if possible.
#### `transportPause`
Purpose:
- Pause transport.
Payload:
```ts
{}
```
Semantics:
- This is the authoritative pause command.
#### `transportSeek`
Purpose:
- Correct transport position without changing the selected source.
Payload:
```ts
{ time: number; rate?: number }
```
Semantics:
- Used for discontinuities only, not as a steady-state clock.
- If the iframe is logically playing, seek must not leave it visually or audibly paused.
- Current Muon router includes a recovery path: if the iframe is expected to be playing but the engine is stopped, a seek may trigger a single auto-resume using the supplied or current rate.
#### `transportRate`
Purpose:
- Update playback rate while playing or paused.
Payload:
```ts
{ rate: number }
```
Semantics:
- Providers should update rate without forcing a reload.
#### `setParam`
Payload:
```ts
{ name: string; value: number | string | boolean }
```
Purpose:
- Apply a renderer-specific tunable control.
Semantics:
- Host sends one message per param key/value.
- Providers may ignore unknown param names.
#### `setPalette`
Payload:
```ts
{ colors: string[] }
```
Purpose:
- Apply host-selected palette colors to the renderer.
Semantics:
- Colors are passed as hex-like strings from the host.
- Providers may clamp or ignore unsupported slots.
- Palette changes must not require reload.
### Reserved for future expansion
#### `dispose`
Payload:
```ts
{}
```
Purpose:
- Provide an explicit teardown hook for future providers.
Status:
- Reserved by this spec.
- Optional even after implementation because iframe unmount remains the primary teardown path.
## Iframe To Host Events
### Implemented now
#### `ready`
Purpose:
- Signal that the iframe runtime is initialized and can receive settings and audio commands.
Current payload shape:
```ts
{
capabilities?: {
webgl?: boolean
missingExtensions?: string[]
audioMode?: string
}
styles?: string[]
}
```
Semantics:
- Host should treat `ready` as the gate for sending settings and audio load commands.
- Host must tolerate provider-specific extra fields.
#### `loaded`
Purpose:
- Confirm that the iframe decoded the current audio source.
Payload:
```ts
{ duration: number }
```
Semantics:
- `duration` may be `0` if unknown.
- Host uses this to mark the iframe as loaded.
#### `error`
Purpose:
- Report an iframe-side failure.
Payload:
```ts
{ code: string; message: string }
```
Semantics:
- `AUDIO_LOAD` is currently treated by the host as a non-actionable runtime warning and may be suppressed in the console.
- Other error codes are diagnostic and should be logged.
#### `capabilities`
Purpose:
- Describe renderer-specific presets, params, and palette support after initialization.
Payload:
```ts
{
providerId?: string
protocolVersion?: 1
presets: string[]
params: Param[]
palette: {
supported: boolean
slotCount: number
}
}
```
```ts
type Param = {
name: string
label: string
type: 'number' | 'enum' | 'boolean'
default: number | string | boolean
min?: number
max?: number
step?: number
options?: string[]
}
```
Semantics:
- Host must tolerate this arriving after render start.
- Host must also tolerate it never arriving and fall back to generic controls.
### Reserved for future expansion
## Behavioral Requirements
### Replay and seek
- Replay after pause or track restart must be deterministic.
- `transportSeek` is for discontinuities, scrub actions, and drift correction beyond tolerance.
- Steady playback must not emit continuous seek churn.
- If a seek arrives while the host is logically playing, the iframe must remain able to resume visual and audio progression without a reload.
### Rate handling
- `transportPlay` carries the starting rate.
- `transportRate` carries subsequent rate changes.
- `transportSeek` may also carry `rate` so a recovery path can resume using the correct speed.
### Resize and quality
- `setViewport` and `setQuality` must be hot-applied.
- Providers may rebuild internal framebuffers on viewport change, but must not require document reload.
### Audio source ownership
- Host owns source selection.
- Iframe owns decode, playback, and renderer-side consumption.
- `loadAudioData` is the required compatibility bridge for host-local uploads.
## Provider-Neutral Rules
- The host contract must not rely on Muon-specific naming beyond historical file names.
- Multi-style providers may implement `setStyle`; single-style providers may ignore it.
- All providers must understand the same envelope, core transport commands, and core events.
- Future providers should emit `capabilities` and implement `setParam` and `setPalette`.
## Compatibility Matrix
| Message | Current Muon iframe | Required for future providers |
|---|---|---|
| `setStyle` | Yes | Optional no-op for single-style providers |
| `setPreset` | Yes | Yes |
| `setQuality` | Yes | Yes |
| `setPaused` | Yes | Yes |
| `setViewport` | Yes | Yes |
| `pushAudioFeatures` | Yes | Yes if renderer uses host-pushed features |
| `loadAudio` | Yes | Yes |
| `loadAudioData` | Yes | Yes |
| `transportPlay` | Yes | Yes |
| `transportPause` | Yes | Yes |
| `transportSeek` | Yes | Yes |
| `transportRate` | Yes | Yes |
| `setParam` | Yes | Yes |
| `setPalette` | Yes | Yes |
| `dispose` | Reserved | Optional |
| `ready` | Yes | Yes |
| `loaded` | Yes | Yes |
| `error` | Yes | Yes |
| `capabilities` | Yes | Yes |
## Validation Baseline
The current baseline is based on the implemented contract in:
- `frontend/src/hooks/useVisualizerIframeTransport.ts`
- `muon-iframe/stemengine-router.js`
Phase 0 validation remains the behavioral baseline:
- replay cycle 2 is reactive again,
- steady playback no longer produces seek churn,
- local uploads cross the host/iframe boundary via `loadAudioData`.