485 lines
11 KiB
Markdown
485 lines
11 KiB
Markdown
# Visualizer Iframe Protocol
|
|
|
|
Date: 2026-04-28
|
|
Status: Phase 1 baseline frozen
|
|
Owners: StemEngine frontend / iframe visualizer pipeline
|
|
|
|
## Purpose
|
|
|
|
This document freezes the host-to-iframe transport contract used by StemEngine visualizer iframes.
|
|
|
|
Goals:
|
|
- preserve the current Muon iframe behavior that is already shipping,
|
|
- define the provider-neutral contract that future iframe apps must implement,
|
|
- make replay, seek, rate, resize, and local-upload behavior explicit before Phase 2 and Phase 3 refactors.
|
|
|
|
Sources of truth for the current implementation:
|
|
- `frontend/src/hooks/useMuonIframeTransport.ts`
|
|
- `muon-iframe/stemengine-router.js`
|
|
|
|
This is a protocol document, not a product API promise. Fields marked "planned" are reserved now so later iframe providers can implement them without renaming the contract.
|
|
|
|
## Transport Envelope
|
|
|
|
All messages use the same envelope shape.
|
|
|
|
```ts
|
|
type VisualizerIframeMessage = {
|
|
version: 1
|
|
id: string
|
|
type: string
|
|
payload?: Record<string, unknown>
|
|
}
|
|
```
|
|
|
|
Rules:
|
|
- `version` is required and is currently fixed at `1`.
|
|
- `id` is caller-generated and is used for correlation and diagnostics only.
|
|
- `type` selects the command or event.
|
|
- `payload` is optional, but most commands and events provide one.
|
|
- Unknown `type` values must be ignored.
|
|
- Messages with a version other than `1` must be ignored.
|
|
|
|
## Lifecycle
|
|
|
|
### Boot sequence
|
|
|
|
1. Host mounts iframe.
|
|
2. Iframe initializes its audio engine and renderer runtime.
|
|
3. Iframe emits `ready`.
|
|
4. Host sends settings and paused state.
|
|
5. Host sends `loadAudio` or `loadAudioData`.
|
|
6. Iframe emits `loaded` after audio decode succeeds.
|
|
7. Host drives playback with `transportPlay`, `transportPause`, `transportSeek`, and `transportRate`.
|
|
|
|
### Visibility and teardown
|
|
|
|
- Host may send `setPaused { paused: true }` when the document is hidden even if transport state remains "playing".
|
|
- Host currently pauses transport on unmount and on inactive state transitions.
|
|
- `dispose` is reserved for future explicit teardown, but iframe unmount remains the effective teardown path today.
|
|
|
|
## Host To Iframe Commands
|
|
|
|
### Implemented now
|
|
|
|
#### `setStyle`
|
|
|
|
Purpose:
|
|
- Select visual style inside a multi-style iframe.
|
|
|
|
Payload:
|
|
|
|
```ts
|
|
{ style: 'default' | 'fluid' | 'tendrils' | 'spirit' | string }
|
|
```
|
|
|
|
Semantics:
|
|
- Required for the current Muon iframe.
|
|
- Single-style providers may treat this as a no-op.
|
|
|
|
#### `setPreset`
|
|
|
|
Purpose:
|
|
- Select a named renderer preset.
|
|
|
|
Payload:
|
|
|
|
```ts
|
|
{ preset: string }
|
|
```
|
|
|
|
Semantics:
|
|
- Current Muon router stores the value even if the renderer ignores it.
|
|
- Future providers may expose preset lists through `capabilities`.
|
|
|
|
#### `setQuality`
|
|
|
|
Purpose:
|
|
- Apply a generic quality tier.
|
|
|
|
Payload:
|
|
|
|
```ts
|
|
{ quality: 'low' | 'balanced' | 'high' | string }
|
|
```
|
|
|
|
Semantics:
|
|
- Providers map this to renderer-specific internals.
|
|
|
|
#### `setPaused`
|
|
|
|
Purpose:
|
|
- Tell the iframe whether visual updates should behave as paused.
|
|
|
|
Payload:
|
|
|
|
```ts
|
|
{ paused: boolean }
|
|
```
|
|
|
|
Semantics:
|
|
- This is a renderer-visibility hint, not the authoritative audio transport command.
|
|
- Host uses it for tab visibility and play/pause mirroring.
|
|
|
|
#### `setViewport`
|
|
|
|
Purpose:
|
|
- Update CSS and pixel viewport dimensions.
|
|
|
|
Payload:
|
|
|
|
```ts
|
|
{
|
|
cssWidth: number
|
|
cssHeight: number
|
|
dpr: number
|
|
}
|
|
```
|
|
|
|
Compatibility note:
|
|
- Current router also accepts legacy `width` and `height` aliases in the payload and derives pixel width and height internally.
|
|
|
|
Semantics:
|
|
- Viewport updates must not require iframe reload.
|
|
- Providers are expected to clamp invalid or extreme values.
|
|
|
|
#### `pushAudioFeatures`
|
|
|
|
Purpose:
|
|
- Push host-side extracted audio features into the iframe render loop.
|
|
|
|
Payload:
|
|
|
|
```ts
|
|
{
|
|
bass?: number
|
|
treble?: number
|
|
rms?: number
|
|
energy?: number
|
|
beat?: boolean
|
|
drumIntensity?: number
|
|
sub?: number
|
|
beatPhase?: number
|
|
bpm?: number
|
|
beatConfidence?: number
|
|
isBeatDrop?: boolean
|
|
isBuildUp?: boolean
|
|
isSilence?: boolean
|
|
energyDelta?: number
|
|
spectralCentroid?: number
|
|
spectralFlatness?: number
|
|
frequencyBands?: unknown
|
|
}
|
|
```
|
|
|
|
Semantics:
|
|
- Host may send at RAF cadence.
|
|
- Iframe should rate-limit internal application so message handling does not block rendering.
|
|
|
|
#### `loadAudio`
|
|
|
|
Purpose:
|
|
- Load audio by URL inside the iframe.
|
|
|
|
Payload:
|
|
|
|
```ts
|
|
{ url: string }
|
|
```
|
|
|
|
Semantics:
|
|
- Used for same-origin or cross-origin URLs the iframe can fetch directly.
|
|
|
|
#### `loadAudioData`
|
|
|
|
Purpose:
|
|
- Load audio from raw bytes when the iframe cannot fetch the source URL directly.
|
|
|
|
Payload:
|
|
|
|
```ts
|
|
{ bytes: ArrayBuffer | Uint8Array | number[] }
|
|
```
|
|
|
|
Semantics:
|
|
- This is the required path for host-local `blob:` uploads that do not survive a cross-origin iframe boundary.
|
|
- The host currently uses this for Visualizer Lab local uploads.
|
|
- Providers that support local uploads must support this command.
|
|
|
|
#### `transportPlay`
|
|
|
|
Purpose:
|
|
- Start or resume transport at a host-selected time and rate.
|
|
|
|
Payload:
|
|
|
|
```ts
|
|
{ time: number; rate: number }
|
|
```
|
|
|
|
Semantics:
|
|
- This is the authoritative start/resume command.
|
|
- Iframe must begin playback at `time` and apply `rate` atomically if possible.
|
|
|
|
#### `transportPause`
|
|
|
|
Purpose:
|
|
- Pause transport.
|
|
|
|
Payload:
|
|
|
|
```ts
|
|
{}
|
|
```
|
|
|
|
Semantics:
|
|
- This is the authoritative pause command.
|
|
|
|
#### `transportSeek`
|
|
|
|
Purpose:
|
|
- Correct transport position without changing the selected source.
|
|
|
|
Payload:
|
|
|
|
```ts
|
|
{ time: number; rate?: number }
|
|
```
|
|
|
|
Semantics:
|
|
- Used for discontinuities only, not as a steady-state clock.
|
|
- If the iframe is logically playing, seek must not leave it visually or audibly paused.
|
|
- Current Muon router includes a recovery path: if the iframe is expected to be playing but the engine is stopped, a seek may trigger a single auto-resume using the supplied or current rate.
|
|
|
|
#### `transportRate`
|
|
|
|
Purpose:
|
|
- Update playback rate while playing or paused.
|
|
|
|
Payload:
|
|
|
|
```ts
|
|
{ rate: number }
|
|
```
|
|
|
|
Semantics:
|
|
- Providers should update rate without forcing a reload.
|
|
|
|
### Reserved in Phase 1, to be implemented in later phases
|
|
|
|
#### `setParam`
|
|
|
|
Payload:
|
|
|
|
```ts
|
|
{ name: string; value: number | string | boolean }
|
|
```
|
|
|
|
Purpose:
|
|
- Apply a renderer-specific tunable control.
|
|
|
|
Status:
|
|
- Reserved by this spec.
|
|
- Not implemented in the current Muon router.
|
|
|
|
#### `setPalette`
|
|
|
|
Payload:
|
|
|
|
```ts
|
|
{ colors: string[] }
|
|
```
|
|
|
|
Purpose:
|
|
- Apply host-selected palette colors to the renderer.
|
|
|
|
Status:
|
|
- Reserved by this spec.
|
|
- Not implemented in the current Muon router.
|
|
|
|
Palette rules:
|
|
- Colors must be hex strings.
|
|
- Providers may clamp or ignore unsupported slots.
|
|
- Palette changes must not require reload.
|
|
|
|
#### `dispose`
|
|
|
|
Payload:
|
|
|
|
```ts
|
|
{}
|
|
```
|
|
|
|
Purpose:
|
|
- Provide an explicit teardown hook for future providers.
|
|
|
|
Status:
|
|
- Reserved by this spec.
|
|
- Optional even after implementation because iframe unmount remains the primary teardown path.
|
|
|
|
## Iframe To Host Events
|
|
|
|
### Implemented now
|
|
|
|
#### `ready`
|
|
|
|
Purpose:
|
|
- Signal that the iframe runtime is initialized and can receive settings and audio commands.
|
|
|
|
Current payload shape:
|
|
|
|
```ts
|
|
{
|
|
capabilities?: {
|
|
webgl?: boolean
|
|
missingExtensions?: string[]
|
|
audioMode?: string
|
|
}
|
|
styles?: string[]
|
|
}
|
|
```
|
|
|
|
Semantics:
|
|
- Host should treat `ready` as the gate for sending settings and audio load commands.
|
|
- Host must tolerate provider-specific extra fields.
|
|
|
|
#### `loaded`
|
|
|
|
Purpose:
|
|
- Confirm that the iframe decoded the current audio source.
|
|
|
|
Payload:
|
|
|
|
```ts
|
|
{ duration: number }
|
|
```
|
|
|
|
Semantics:
|
|
- `duration` may be `0` if unknown.
|
|
- Host uses this to mark the iframe as loaded.
|
|
|
|
#### `error`
|
|
|
|
Purpose:
|
|
- Report an iframe-side failure.
|
|
|
|
Payload:
|
|
|
|
```ts
|
|
{ code: string; message: string }
|
|
```
|
|
|
|
Semantics:
|
|
- `AUDIO_LOAD` is currently treated by the host as a non-actionable runtime warning and may be suppressed in the console.
|
|
- Other error codes are diagnostic and should be logged.
|
|
|
|
### Reserved in Phase 1, to be implemented in later phases
|
|
|
|
#### `capabilities`
|
|
|
|
Purpose:
|
|
- Describe renderer-specific presets, params, and palette support after initialization.
|
|
|
|
Payload:
|
|
|
|
```ts
|
|
{
|
|
providerId?: string
|
|
protocolVersion?: 1
|
|
presets: string[]
|
|
params: Param[]
|
|
palette: {
|
|
supported: boolean
|
|
slotCount: number
|
|
}
|
|
}
|
|
```
|
|
|
|
```ts
|
|
type Param = {
|
|
name: string
|
|
label: string
|
|
type: 'number' | 'enum' | 'boolean'
|
|
default: number | string | boolean
|
|
min?: number
|
|
max?: number
|
|
step?: number
|
|
options?: string[]
|
|
}
|
|
```
|
|
|
|
Status:
|
|
- Reserved by this spec.
|
|
- Not emitted by the current Muon router.
|
|
|
|
Semantics:
|
|
- Host must tolerate this arriving after render start.
|
|
- Host must also tolerate it never arriving and fall back to generic controls.
|
|
|
|
## Behavioral Requirements
|
|
|
|
### Replay and seek
|
|
|
|
- Replay after pause or track restart must be deterministic.
|
|
- `transportSeek` is for discontinuities, scrub actions, and drift correction beyond tolerance.
|
|
- Steady playback must not emit continuous seek churn.
|
|
- If a seek arrives while the host is logically playing, the iframe must remain able to resume visual and audio progression without a reload.
|
|
|
|
### Rate handling
|
|
|
|
- `transportPlay` carries the starting rate.
|
|
- `transportRate` carries subsequent rate changes.
|
|
- `transportSeek` may also carry `rate` so a recovery path can resume using the correct speed.
|
|
|
|
### Resize and quality
|
|
|
|
- `setViewport` and `setQuality` must be hot-applied.
|
|
- Providers may rebuild internal framebuffers on viewport change, but must not require document reload.
|
|
|
|
### Audio source ownership
|
|
|
|
- Host owns source selection.
|
|
- Iframe owns decode, playback, and renderer-side consumption.
|
|
- `loadAudioData` is the required compatibility bridge for host-local uploads.
|
|
|
|
## Provider-Neutral Rules
|
|
|
|
- The host contract must not rely on Muon-specific naming beyond historical file names.
|
|
- Multi-style providers may implement `setStyle`; single-style providers may ignore it.
|
|
- All providers must understand the same envelope, core transport commands, and core events.
|
|
- Future providers should emit `capabilities` and implement `setParam` and `setPalette`.
|
|
|
|
## Compatibility Matrix
|
|
|
|
| Message | Current Muon iframe | Required for future providers |
|
|
|---|---|---|
|
|
| `setStyle` | Yes | Optional no-op for single-style providers |
|
|
| `setPreset` | Yes | Yes |
|
|
| `setQuality` | Yes | Yes |
|
|
| `setPaused` | Yes | Yes |
|
|
| `setViewport` | Yes | Yes |
|
|
| `pushAudioFeatures` | Yes | Yes if renderer uses host-pushed features |
|
|
| `loadAudio` | Yes | Yes |
|
|
| `loadAudioData` | Yes | Yes |
|
|
| `transportPlay` | Yes | Yes |
|
|
| `transportPause` | Yes | Yes |
|
|
| `transportSeek` | Yes | Yes |
|
|
| `transportRate` | Yes | Yes |
|
|
| `setParam` | Reserved | Yes, after Phase 3 |
|
|
| `setPalette` | Reserved | Yes, after Phase 3 |
|
|
| `dispose` | Reserved | Optional |
|
|
| `ready` | Yes | Yes |
|
|
| `loaded` | Yes | Yes |
|
|
| `error` | Yes | Yes |
|
|
| `capabilities` | Reserved | Yes, after Phase 3 |
|
|
|
|
## Validation Baseline For This Freeze
|
|
|
|
The Phase 1 freeze is based on the currently implemented contract in:
|
|
- `frontend/src/hooks/useMuonIframeTransport.ts`
|
|
- `muon-iframe/stemengine-router.js`
|
|
|
|
Phase 0 validation remains the behavioral baseline:
|
|
- replay cycle 2 is reactive again,
|
|
- steady playback no longer produces seek churn,
|
|
- local uploads cross the host/iframe boundary via `loadAudioData`. |