Added documentation for Two.js karaoke renderer and updated summaries.

This commit is contained in:
translator committed 2026-03-11 10:27:34 +09:00
1 parent fa99da7100
commit 84541e3fa3
2 files changed
+478

No files matched your search

+460
View File
@@ -0,0 +1,460 @@
# Karaoke Renderer — Two.js Rewrite Plan
Replace the lyrics display rendering layer of `LyricsLivePreview` with
[Two.js](https://github.com/jonobr1/two.js) (`two.js`), moving from a
DOM/CSS + RAF hybrid to a proper 2D scene graph with renderer-agnostic
drawing primitives.
---
## Scope
**Replaced:** the scrolling/fading/karaoke lyrics canvas — the visible window
where lines and words animate. Specifically the inner `<div>` stack that currently
renders `lineGroup` elements, `KaraokeWord` spans, and their CSS transitions.
**Untouched:**
- Audio engine (HTMLAudioElement / Tone.js gain graph)
- Playback transport controls (play/pause, seek, volume)
- Style picker panel (all the settings UI)
- Speaker management (colour assignment, name editing)
- LRC text editor
- `KaraokeEditor` (waveform + timing table — completely separate component)
- Cover art display
- Fullscreen container wrapper
The Two.js scene drops into the exact same bounding box that the `<div>` stack
currently occupies. Everything outside it stays in React/HTML.
---
## Why Two.js
| Current DOM approach | Two.js scene graph |
|---|---|
| Gradient word fill via `WebkitBackgroundClip: text` — Chrome/Safari only | Clip-rect fill via native SVG `<clipPath>` or Canvas2D save/clip — universal |
| CSS `transition` on `transform`/`opacity` — GPU-accelerated but not sequenceable | `two.bind('update', fn)` RAF + explicit tweens — fully programmable |
| Behind/front layer trick (two `<span>` stacked) for stroke-outside-fill | First-class Z-ordered scene graph: stroke pass below, fill pass above |
| Blur style faked with CSS `filter: blur()` — blurs children too | Per-object blur via SVG `<feGaussianBlur>` filter on individual line groups |
| Glow: CSS `text-shadow` — no per-word control | Per-word glow via SVG `<feDropShadow>` or Canvas2D `shadowBlur` per draw call |
| Word positions computed from DOM layout (getBoundingClientRect) | Word positions computed from explicit text metrics (`ctx.measureText`) — no layout reflow |
| RAF reads `activeWordSpanRef.style.backgroundImage` — one element at a time | RAF updates `clipRect.width` on all in-flight words simultaneously |
| Renderer locked to HTML/CSS | Choose SVG (crisp, DPI-independent), Canvas2D, or WebGL |
---
## Recommended Renderer: SVG
- Text is crisp at any display scale (important for 4K displays / fullscreen).
- `<clipPath>` is the canonical, universally supported way to do karaoke word fill.
- SVG elements remain inspectable in DevTools.
- Two.js SVG renderer outputs a real `<svg>` DOM subtree; raw SVG nodes can be
injected for effects Two.js doesn't expose (e.g. `<feMorphology>` stroke expansion).
- Switch to Canvas2D in `new Two({ type: Two.Types.canvas })` if SVG performance
degrades on very long songs (> 1000 lines visible).
---
## Installation
```bash
npm install two.js
```
No other new dependencies. Two.js is self-contained (zero runtime deps).
---
## Scene Graph Architecture
```
Two instance (SVG renderer, auto-sized to container)
│
└── scene
├── bgRect ← Two.Rectangle, full scene size, bgColor fill
│
├── fadeTop ← Two.Rectangle, top ~18% height, gradient fill (transparent→bg)
├── fadeBottom ← Two.Rectangle, bottom ~18% height, gradient fill (bg→transparent)
│
└── linesGroup ← Two.Group — translated vertically each frame (scroll)
│
├── lineGroup[n-2] ← Two.Group — past line
├── lineGroup[n-1] ← Two.Group — previous line
├── lineGroup[n] ← Two.Group — ACTIVE LINE
├── lineGroup[n+1] ← Two.Group — next line
└── lineGroup[n+2] ← Two.Group — upcoming line
```
### lineGroup (inactive line)
```
lineGroup
├── speakerDot ← Two.Circle, speaker accent colour (visible if speaker changes)
└── lineText ← Two.Text, textColor fill, opacity=0.4, scale=0.9 (or style-dependent)
```
### lineGroup (active line — karaoke/glow style)
```
activeLineGroup
├── speakerDot
└── wordsGroup
├── wordUnit[0]
│ ├── wordText_behind ← Two.Text, stroke+shadow pass, z=0
│ └── wordText_front ← Two.Text, highlight fill, z=1
│ └── clipGroup ← SVG <clipPath><rect/></clipPath> — width grows 0→wordWidth
├── wordUnit[1]
└── ...
```
For **scroll / fade** styles where per-word fill is not needed, `activeLineGroup`
uses a single `Two.Text` per line (no `wordsGroup`), matching the visual weight of
inactive lines but with `opacity=1` and `scale=1`.
---
## Word Fill Technique (SVG renderer)
The current `WebkitBackgroundClip: 'text'` gradient is replaced with a native
SVG clip rectangle — the standard approach used by professional karaoke software.
```
For each word[j] in the active line:
wordText_front has clip-path pointing to clipRect[j]
Per frame (RAF update callback):
fillFrac = clamp((currentTime - word[j].time) / wordDuration, 0, 1)
clipRect[j].width = fillFrac * wordMetrics[j].width
```
Two.js does not expose `<clipPath>` natively, so we inject raw SVG:
```ts
// Once, during scene build for this line:
const svgNS = 'http://www.w3.org/2000/svg'
const defs = two.renderer.domElement.querySelector('defs')
?? two.renderer.domElement.insertBefore(
document.createElementNS(svgNS, 'defs'),
two.renderer.domElement.firstChild)
const clipPath = document.createElementNS(svgNS, 'clipPath')
clipPath.id = `kclip-${lineIdx}-${wordIdx}`
const clipRect = document.createElementNS(svgNS, 'rect')
clipRect.setAttribute('x', String(wordX))
clipRect.setAttribute('y', String(lineY - fontSize))
clipRect.setAttribute('height', String(fontSize * 1.2))
clipRect.setAttribute('width', '0') // grows each frame
clipPath.appendChild(clipRect)
defs.appendChild(clipPath)
// Apply to the Two.Text SVG element:
wordText_front._renderer.elem.setAttribute(
'clip-path', `url(#kclip-${lineIdx}-${wordIdx})`)
```
Each frame in `two.bind('update', fn)`:
```ts
// Running words (started but not finished):
for (const w of activeWords) {
const frac = clamp((time - w.startTime) / w.duration, 0, 1)
w.clipRect.setAttribute('width', String(frac * w.measuredWidth))
}
```
For **Canvas2D renderer** (fallback), the same effect via save/clip:
```ts
// Draw dim base text first (done by Two.js normally)
// Then overlay the filled portion:
ctx.save()
ctx.beginPath()
ctx.rect(wordX, wordY - fontSize, frac * wordWidth, fontSize * 1.2)
ctx.clip()
ctx.fillStyle = highlightColor
ctx.font = `${fontSize}px ${fontFamily}`
ctx.fillText(word.text, wordX, wordY)
ctx.restore()
```
This requires hooking into Two.js's Canvas2D render pipeline after the normal
draw — achieved via a custom `Two.Element` subclass that overrides `_render`.
---
## Word Metrics & Layout
CSS layout (the current approach) automatically positions words via `display:inline`.
Two.js has no text flow engine — word positions must be computed explicitly.
```ts
// Once per active line, after font is set:
const canvas2d = document.createElement('canvas')
const mctx = canvas2d.getContext('2d')!
mctx.font = `${fontWeight} ${fontSize}px ${fontFamily}`
let cursorX = startX
const wordMeta: WordMeta[] = []
for (const word of line.words) {
const text = word.text + (word.isWordEnd ? ' ' : '')
const metrics = mctx.measureText(text)
wordMeta.push({ x: cursorX, width: metrics.width, text })
cursorX += metrics.width
}
// Total line width → center horizontally: startX = (sceneWidth - totalWidth) / 2
```
This measurement must re-run when:
- Font family or size changes
- Scene width changes (resize observer)
- Active line changes (new word set)
---
## Scroll / Layout Logic
The `linesGroup` Y translation replaces the current CSS `scrollY` state.
```ts
two.bind('update', (frameCount, timeDelta) => {
const time = currentTimeRef.current
const activeIdx = findActiveLine(lines, time)
// Target Y: centre the active line vertically
const targetY = sceneH / 2 - activeIdx * lineStride
// Smooth scroll (lerp, same feel as current CSS transition)
linesGroup.translation.y += (targetY - linesGroup.translation.y) * 0.12
// Fade + scale inactive lines
for (let i = 0; i < lineGroups.length; i++) {
const dist = Math.abs(i - activeIdx)
const target = styleConfig.opacityForDist(dist) // style-dependent
lineGroups[i].opacity += (target - lineGroups[i].opacity) * 0.15
// similarly for scale via lineGroups[i].scale
}
// Update word fill clips
updateWordFills(time, activeIdx)
})
```
---
## Style Modes — Two.js Mappings
| Style | Current | Two.js |
|-------|---------|--------|
| **scroll** | CSS `transform: translateY` with `transition` | `linesGroup.translation.y` lerp each frame |
| **fade** | CSS `opacity` + `transition` on each line div | `lineGroup.opacity` lerp per frame |
| **karaoke** | `WebkitBackgroundClip` gradient sweep, CSS scale | SVG clipRect sweep, `lineGroup.scale` lerp |
| **glow** | `text-shadow` + gradient sweep | SVG `<feDropShadow>` filter on active group + clipRect sweep |
| **blur** | CSS `filter: blur()` on inactive divs | SVG `<feGaussianBlur>` filter on inactive lineGroups |
**Glow filter (applied to activeLineGroup):**
```ts
// Injected once into <defs>:
// <filter id="glow"><feDropShadow dx="0" dy="0" stdDeviation="4" flood-color="#60a5fa" /></filter>
activeLineGroup._renderer.elem.setAttribute('filter', 'url(#glow)')
```
**Blur filter (applied to inactive lineGroups):**
```ts
// <filter id="blur-inactive"><feGaussianBlur stdDeviation="2"/></filter>
// Intensity scaled with distance from active line
const stdDev = Math.min(dist * 1.5, 6)
blurFilter.querySelector('feGaussianBlur').setAttribute('stdDeviation', String(stdDev))
```
---
## Stroke + Shadow Layering
Current: two stacked `<span>` (behind = stroke+shadow, front = fill only).
Two.js: explicit Z-order within `wordUnit` group.
```
wordUnit
wordText_behind ← fill='none', stroke=strokeColor, linewidth=strokeWidth*2, opacity=0.8
+ SVG <filter> with <feDropShadow> for shadow
wordText_front ← fill=textColor (or highlightColor when filled), stroke='none'
+ clipPath for fill progress
```
Because `wordText_behind` is a child with lower z-index, Two.js renders it first —
stroke appears outside the letterforms, shadow is behind — identical to the current
behind-layer pattern but expressed as a proper scene hierarchy.
---
## Speaker Dots
```ts
const dot = two.makeCircle(x - dotOffset, lineY, 4)
dot.fill = speakerColor(line.speaker)
dot.noStroke()
lineGroup.add(dot)
dot.visible = speakerChanged(i)
```
---
## React Integration
```tsx
// LyricsCanvas.tsx — new component replacing the <div> scroll stack
import { useRef, useEffect } from 'react'
import Two from 'two.js'
export function LyricsCanvas({ lines, currentTimeRef, styleOptions, ...rest }: Props) {
const containerRef = useRef<HTMLDivElement>(null)
const twoRef = useRef<Two | null>(null)
const sceneRef = useRef<SceneState | null>(null)
// ── Mount Two.js ────────────────────────────────────────────────────────
useEffect(() => {
const el = containerRef.current!
const two = new Two({ type: Two.Types.svg, autostart: true, fitted: true })
two.appendTo(el)
twoRef.current = two
sceneRef.current = buildScene(two, styleOptions)
two.bind('update', () => {
updateScene(sceneRef.current!, currentTimeRef.current, /* latest lines/style via refs */)
})
const ro = new ResizeObserver(() => {
two.width = el.clientWidth
two.height = el.clientHeight
rebuildLayout(sceneRef.current!, two.width, two.height)
})
ro.observe(el)
return () => {
two.pause()
two.unbind('update')
ro.disconnect()
el.removeChild(two.renderer.domElement)
}
}, []) // mount once
// ── Sync style changes (no scene rebuild, just update properties) ───────
useEffect(() => {
if (!sceneRef.current) return
applyStyleOptions(sceneRef.current, styleOptions)
}, [styleOptions])
return <div ref={containerRef} style={{ width: '100%', height: '100%' }} />
}
```
`updateScene` runs inside `two.bind('update')` — equivalent to the current RAF
loop. It reads `currentTimeRef.current` directly (a ref, no React re-render) and
mutates Two.js object properties. React only re-renders when structural changes
happen (line count changes, style mode changes) — identical to the current strategy.
---
## Word Metrics Cache
Computing `measureText` for every word on every render is expensive. Cache by
`${fontFamily}:${fontSize}:${text}`:
```ts
const metricsCache = new Map<string, number>()
function measureWord(text: string, font: string): number {
const key = `${font}:${text}`
if (metricsCache.has(key)) return metricsCache.get(key)!
mctx.font = font
const w = mctx.measureText(text).width
metricsCache.set(key, w)
return w
}
```
Cache is invalidated (cleared) when font family or size changes.
---
## Scene Rebuild vs. Property Update
Two cases require different responses:
| Trigger | Action |
|---------|--------|
| Active line changes (new `activeIdx`) | Update `linesGroup` child visibility + word clip rects for the new active line; set up new `clipRect` nodes for new active words |
| Font / font size changes | Full scene rebuild: re-measure all words, rebuild `wordUnit` groups, reposition all `lineGroup` objects |
| Colour / opacity / style mode changes | Property update only: `two.text.fill = newColor`, `lineGroup.opacity = newOpacity` |
| Scene resize | Re-centre `linesGroup`, recompute `lineStride`, re-measure words (font-size-dependent if `em`-relative) |
| `lines` array changes (LRC edit) | Re-index `lineGroups` array, add/remove groups as needed |
---
## Implementation Phases
### Phase 1 — Static text rendering
- Set up Two.js SVG renderer in a `LyricsCanvas` component.
- Render all lines as `Two.Text` objects, positioned vertically.
- No animation yet — static layout with correct fonts, colours, spacing.
- Verify text rendering quality vs current CSS (especially on HiDPI).
### Phase 2 — Scroll + fade animation
- Implement `linesGroup.translation.y` lerp in the RAF update loop.
- Implement per-line `opacity` and `scale` lerp for **scroll** and **fade** styles.
- Connect `currentTimeRef` → `findActiveLine` → active index.
- Replace the CSS scroll div with `LyricsCanvas` for these two modes.
### Phase 3 — Word fill (karaoke / glow style)
- Compute word metrics for active line.
- Inject `<clipPath>` + `<clipRect>` nodes into Two.js SVG defs.
- Implement per-frame clip rect width update in update loop.
- Implement stroke/shadow layering (behind/front group pattern).
### Phase 4 — Glow + blur styles
- Inject SVG filter defs (`<feDropShadow>`, `<feGaussianBlur>`).
- Apply glow filter to active line group.
- Scale blur `stdDeviation` with distance from active line.
### Phase 5 — Speaker dots + full style parity
- Add speaker dot circles.
- Verify all `LyricsStyleOptions` properties are honoured.
- Replace `KaraokeWord` / CSS gradient path entirely; remove old DOM scroll stack.
### Phase 6 — Polish + fallback
- Benchmark SVG vs Canvas2D on a long test file; switch default renderer if needed.
- Verify fullscreen mode (scene resizes via ResizeObserver + `two.fitted`).
- Verify album art colour extraction (still Canvas2D, unchanged).
- Remove `KaraokeWord` component and all `WebkitBackgroundClip` CSS.
---
## What Does NOT Change
| Item | Reason |
|------|--------|
| `KaraokeEditor.tsx` | Waveform + timing table — completely separate |
| Audio playback engine | `<audio>`, Tone.js, gain graph |
| Transport controls JSX | Play/pause/seek/volume — pure React |
| Style picker panel | All settings UI — pure React |
| Speaker management | Colour assignment, name editing |
| `parseLrcContent()` | Data parsing — unaffected |
| `reconstructLrc()` | Data serialisation — unaffected |
| Album art extraction canvas | Already a separate `<canvas>` — untouched |
| `audioRegistry.ts` | Pub/sub utility — unaffected |
| `ThemeContext` | App-level theming — unaffected |
---
## Risk Notes
| Risk | Mitigation |
|------|-----------|
| **Two.js `<clipPath>` injection** — not part of Two.js public API; accesses `_renderer.elem` | Encapsulate in `svgUtils.ts`; test against Two.js version pinned in `package.json`; if Two.js internals change, only this utility needs updating |
| **Text metrics vs CSS layout** — `measureText` may differ subtly from browser text layout (kerning, hinting) | Measure in a hidden `<canvas>` with the same font stack; accept ~1px variance |
| **Font loading timing** — Two.js renders before custom web fonts load → fallback font metrics used | Use `document.fonts.ready` before calling `buildScene`; re-measure and rebuild layout after fonts are confirmed loaded |
| **HiDPI / devicePixelRatio** — SVG renderer is automatically crisp; Canvas2D renderer needs explicit DPR scaling | If Canvas2D is used: `canvas.width = w * dpr`, `canvas.height = h * dpr`, `ctx.scale(dpr, dpr)` |
| **React StrictMode double-mount** — Two.js `appendTo` called twice in dev mode | The `useEffect` cleanup removes `two.renderer.domElement` from the container; second mount starts fresh |
| **Long song performance** — 500+ line `linesGroup` with many `Two.Text` objects | Only keep ±10 lines around active index in the scene; virtualise the rest (swap objects in/out as the active index moves) |
| **Safari SVG filter performance** — `<feGaussianBlur>` on many elements can be slow | Gate blur style behind a capability check; fall back to `opacity` dimming on Safari |
+18
View File
@@ -811,3 +811,21 @@ system, `data-role` auto-init in dynamic React renders, Vite side-effects config
**Bug fix — waveform not redrawing on expand:** When `expandedRow` changed (line expanded or collapsed), `dirtyRef.current` was never set, so the word markers didn't appear/disappear until the next playhead movement frame. Fixed by adding `useEffect(() => { dirtyRef.current = true }, [selectedLineIdx])` in `WaveformCanvas`.
**Feature — word sub-row highlight on click:** Clicking a word sub-row in the timing table now applies an amber (`rgba(251,191,36,0.12)`) background highlight to that row (direct DOM mutation, no re-render). Clicking a different word row moves the highlight; the highlight clears automatically when the active line changes during playback (RAF loop). Implemented via `wordRowRefsMap` (keyed `${lineIdx}-${wordIdx}`), `activeWordRowRef`, and `highlightWordRow()` helper in `TimingTable`.
### 2026-03-11 — Karaoke renderer Two.js rewrite plan
**Created — `Karaoke Renderer (Two JS).md`:**
Plan to replace the DOM/CSS+RAF lyrics rendering layer of `LyricsLivePreview` with a
Two.js SVG scene graph. Covers: scope (only the lyrics display window; audio, controls,
settings, KaraokeEditor all untouched); why Two.js (native SVG clipPath word fill,
scene-graph Z-ordering for stroke/shadow layering, per-object SVG filters for glow/blur,
renderer-agnostic API); recommended SVG renderer with Canvas2D fallback; full scene graph
structure (bgRect, fadeTop/Bottom gradient rects, linesGroup with per-line lineGroup
containing wordsGroup / wordUnit / wordText_behind / wordText_front + clipPath); SVG
clipRect injection technique (not part of Two.js public API, via `_renderer.elem`);
word metrics computation via hidden canvas `measureText` with font-keyed cache;
style mode mappings (scroll/fade/karaoke/glow/blur); speaker dot circles; React
integration pattern (LyricsCanvas component with `useEffect` mount, `two.bind('update')`
for RAF, ResizeObserver for resize, `useEffect` for style prop changes); 6 implementation
phases; and risk notes (clipPath injection stability, font loading timing, HiDPI,
StrictMode double-mount, long-song virtualisation, Safari feGaussianBlur performance).