From 84541e3fa373979d645d02448c0dc6da117514a7 Mon Sep 17 00:00:00 2001 From: translator Date: Wed, 11 Mar 2026 10:27:34 +0900 Subject: [PATCH] Added documentation for Two.js karaoke renderer and updated summaries. --- Karaoke Renderer (Two JS).md | 460 +++++++++++++++++++++++++++++++++++ SUMMARIES.md | 18 ++ 2 files changed, 478 insertions(+) create mode 100644 Karaoke Renderer (Two JS).md diff --git a/Karaoke Renderer (Two JS).md b/Karaoke Renderer (Two JS).md new file mode 100644 index 0000000..4847b70 --- /dev/null +++ b/Karaoke Renderer (Two JS).md @@ -0,0 +1,460 @@ +# Karaoke Renderer — Two.js Rewrite Plan + +Replace the lyrics display rendering layer of `LyricsLivePreview` with +[Two.js](https://github.com/jonobr1/two.js) (`two.js`), moving from a +DOM/CSS + RAF hybrid to a proper 2D scene graph with renderer-agnostic +drawing primitives. + +--- + +## Scope + +**Replaced:** the scrolling/fading/karaoke lyrics canvas — the visible window +where lines and words animate. Specifically the inner `
` stack that currently +renders `lineGroup` elements, `KaraokeWord` spans, and their CSS transitions. + +**Untouched:** +- Audio engine (HTMLAudioElement / Tone.js gain graph) +- Playback transport controls (play/pause, seek, volume) +- Style picker panel (all the settings UI) +- Speaker management (colour assignment, name editing) +- LRC text editor +- `KaraokeEditor` (waveform + timing table — completely separate component) +- Cover art display +- Fullscreen container wrapper + +The Two.js scene drops into the exact same bounding box that the `
` stack +currently occupies. Everything outside it stays in React/HTML. + +--- + +## Why Two.js + +| Current DOM approach | Two.js scene graph | +|---|---| +| Gradient word fill via `WebkitBackgroundClip: text` — Chrome/Safari only | Clip-rect fill via native SVG `` or Canvas2D save/clip — universal | +| CSS `transition` on `transform`/`opacity` — GPU-accelerated but not sequenceable | `two.bind('update', fn)` RAF + explicit tweens — fully programmable | +| Behind/front layer trick (two `` stacked) for stroke-outside-fill | First-class Z-ordered scene graph: stroke pass below, fill pass above | +| Blur style faked with CSS `filter: blur()` — blurs children too | Per-object blur via SVG `` filter on individual line groups | +| Glow: CSS `text-shadow` — no per-word control | Per-word glow via SVG `` or Canvas2D `shadowBlur` per draw call | +| Word positions computed from DOM layout (getBoundingClientRect) | Word positions computed from explicit text metrics (`ctx.measureText`) — no layout reflow | +| RAF reads `activeWordSpanRef.style.backgroundImage` — one element at a time | RAF updates `clipRect.width` on all in-flight words simultaneously | +| Renderer locked to HTML/CSS | Choose SVG (crisp, DPI-independent), Canvas2D, or WebGL | + +--- + +## Recommended Renderer: SVG + +- Text is crisp at any display scale (important for 4K displays / fullscreen). +- `` is the canonical, universally supported way to do karaoke word fill. +- SVG elements remain inspectable in DevTools. +- Two.js SVG renderer outputs a real `` DOM subtree; raw SVG nodes can be + injected for effects Two.js doesn't expose (e.g. `` stroke expansion). +- Switch to Canvas2D in `new Two({ type: Two.Types.canvas })` if SVG performance + degrades on very long songs (> 1000 lines visible). + +--- + +## Installation + +```bash +npm install two.js +``` + +No other new dependencies. Two.js is self-contained (zero runtime deps). + +--- + +## Scene Graph Architecture + +``` +Two instance (SVG renderer, auto-sized to container) +│ +└── scene + ├── bgRect ← Two.Rectangle, full scene size, bgColor fill + │ + ├── fadeTop ← Two.Rectangle, top ~18% height, gradient fill (transparent→bg) + ├── fadeBottom ← Two.Rectangle, bottom ~18% height, gradient fill (bg→transparent) + │ + └── linesGroup ← Two.Group — translated vertically each frame (scroll) + │ + ├── lineGroup[n-2] ← Two.Group — past line + ├── lineGroup[n-1] ← Two.Group — previous line + ├── lineGroup[n] ← Two.Group — ACTIVE LINE + ├── lineGroup[n+1] ← Two.Group — next line + └── lineGroup[n+2] ← Two.Group — upcoming line +``` + +### lineGroup (inactive line) + +``` +lineGroup +├── speakerDot ← Two.Circle, speaker accent colour (visible if speaker changes) +└── lineText ← Two.Text, textColor fill, opacity=0.4, scale=0.9 (or style-dependent) +``` + +### lineGroup (active line — karaoke/glow style) + +``` +activeLineGroup +├── speakerDot +└── wordsGroup + ├── wordUnit[0] + │ ├── wordText_behind ← Two.Text, stroke+shadow pass, z=0 + │ └── wordText_front ← Two.Text, highlight fill, z=1 + │ └── clipGroup ← SVG — width grows 0→wordWidth + ├── wordUnit[1] + └── ... +``` + +For **scroll / fade** styles where per-word fill is not needed, `activeLineGroup` +uses a single `Two.Text` per line (no `wordsGroup`), matching the visual weight of +inactive lines but with `opacity=1` and `scale=1`. + +--- + +## Word Fill Technique (SVG renderer) + +The current `WebkitBackgroundClip: 'text'` gradient is replaced with a native +SVG clip rectangle — the standard approach used by professional karaoke software. + +``` +For each word[j] in the active line: + wordText_front has clip-path pointing to clipRect[j] + +Per frame (RAF update callback): + fillFrac = clamp((currentTime - word[j].time) / wordDuration, 0, 1) + clipRect[j].width = fillFrac * wordMetrics[j].width +``` + +Two.js does not expose `` natively, so we inject raw SVG: + +```ts +// Once, during scene build for this line: +const svgNS = 'http://www.w3.org/2000/svg' +const defs = two.renderer.domElement.querySelector('defs') + ?? two.renderer.domElement.insertBefore( + document.createElementNS(svgNS, 'defs'), + two.renderer.domElement.firstChild) + +const clipPath = document.createElementNS(svgNS, 'clipPath') +clipPath.id = `kclip-${lineIdx}-${wordIdx}` +const clipRect = document.createElementNS(svgNS, 'rect') +clipRect.setAttribute('x', String(wordX)) +clipRect.setAttribute('y', String(lineY - fontSize)) +clipRect.setAttribute('height', String(fontSize * 1.2)) +clipRect.setAttribute('width', '0') // grows each frame +clipPath.appendChild(clipRect) +defs.appendChild(clipPath) + +// Apply to the Two.Text SVG element: +wordText_front._renderer.elem.setAttribute( + 'clip-path', `url(#kclip-${lineIdx}-${wordIdx})`) +``` + +Each frame in `two.bind('update', fn)`: + +```ts +// Running words (started but not finished): +for (const w of activeWords) { + const frac = clamp((time - w.startTime) / w.duration, 0, 1) + w.clipRect.setAttribute('width', String(frac * w.measuredWidth)) +} +``` + +For **Canvas2D renderer** (fallback), the same effect via save/clip: + +```ts +// Draw dim base text first (done by Two.js normally) +// Then overlay the filled portion: +ctx.save() +ctx.beginPath() +ctx.rect(wordX, wordY - fontSize, frac * wordWidth, fontSize * 1.2) +ctx.clip() +ctx.fillStyle = highlightColor +ctx.font = `${fontSize}px ${fontFamily}` +ctx.fillText(word.text, wordX, wordY) +ctx.restore() +``` + +This requires hooking into Two.js's Canvas2D render pipeline after the normal +draw — achieved via a custom `Two.Element` subclass that overrides `_render`. + +--- + +## Word Metrics & Layout + +CSS layout (the current approach) automatically positions words via `display:inline`. +Two.js has no text flow engine — word positions must be computed explicitly. + +```ts +// Once per active line, after font is set: +const canvas2d = document.createElement('canvas') +const mctx = canvas2d.getContext('2d')! +mctx.font = `${fontWeight} ${fontSize}px ${fontFamily}` + +let cursorX = startX +const wordMeta: WordMeta[] = [] +for (const word of line.words) { + const text = word.text + (word.isWordEnd ? ' ' : '') + const metrics = mctx.measureText(text) + wordMeta.push({ x: cursorX, width: metrics.width, text }) + cursorX += metrics.width +} +// Total line width → center horizontally: startX = (sceneWidth - totalWidth) / 2 +``` + +This measurement must re-run when: +- Font family or size changes +- Scene width changes (resize observer) +- Active line changes (new word set) + +--- + +## Scroll / Layout Logic + +The `linesGroup` Y translation replaces the current CSS `scrollY` state. + +```ts +two.bind('update', (frameCount, timeDelta) => { + const time = currentTimeRef.current + const activeIdx = findActiveLine(lines, time) + + // Target Y: centre the active line vertically + const targetY = sceneH / 2 - activeIdx * lineStride + + // Smooth scroll (lerp, same feel as current CSS transition) + linesGroup.translation.y += (targetY - linesGroup.translation.y) * 0.12 + + // Fade + scale inactive lines + for (let i = 0; i < lineGroups.length; i++) { + const dist = Math.abs(i - activeIdx) + const target = styleConfig.opacityForDist(dist) // style-dependent + lineGroups[i].opacity += (target - lineGroups[i].opacity) * 0.15 + // similarly for scale via lineGroups[i].scale + } + + // Update word fill clips + updateWordFills(time, activeIdx) +}) +``` + +--- + +## Style Modes — Two.js Mappings + +| Style | Current | Two.js | +|-------|---------|--------| +| **scroll** | CSS `transform: translateY` with `transition` | `linesGroup.translation.y` lerp each frame | +| **fade** | CSS `opacity` + `transition` on each line div | `lineGroup.opacity` lerp per frame | +| **karaoke** | `WebkitBackgroundClip` gradient sweep, CSS scale | SVG clipRect sweep, `lineGroup.scale` lerp | +| **glow** | `text-shadow` + gradient sweep | SVG `` filter on active group + clipRect sweep | +| **blur** | CSS `filter: blur()` on inactive divs | SVG `` filter on inactive lineGroups | + +**Glow filter (applied to activeLineGroup):** +```ts +// Injected once into : +// +activeLineGroup._renderer.elem.setAttribute('filter', 'url(#glow)') +``` + +**Blur filter (applied to inactive lineGroups):** +```ts +// +// Intensity scaled with distance from active line +const stdDev = Math.min(dist * 1.5, 6) +blurFilter.querySelector('feGaussianBlur').setAttribute('stdDeviation', String(stdDev)) +``` + +--- + +## Stroke + Shadow Layering + +Current: two stacked `` (behind = stroke+shadow, front = fill only). +Two.js: explicit Z-order within `wordUnit` group. + +``` +wordUnit + wordText_behind ← fill='none', stroke=strokeColor, linewidth=strokeWidth*2, opacity=0.8 + + SVG with for shadow + wordText_front ← fill=textColor (or highlightColor when filled), stroke='none' + + clipPath for fill progress +``` + +Because `wordText_behind` is a child with lower z-index, Two.js renders it first — +stroke appears outside the letterforms, shadow is behind — identical to the current +behind-layer pattern but expressed as a proper scene hierarchy. + +--- + +## Speaker Dots + +```ts +const dot = two.makeCircle(x - dotOffset, lineY, 4) +dot.fill = speakerColor(line.speaker) +dot.noStroke() +lineGroup.add(dot) +dot.visible = speakerChanged(i) +``` + +--- + +## React Integration + +```tsx +// LyricsCanvas.tsx — new component replacing the
scroll stack + +import { useRef, useEffect } from 'react' +import Two from 'two.js' + +export function LyricsCanvas({ lines, currentTimeRef, styleOptions, ...rest }: Props) { + const containerRef = useRef(null) + const twoRef = useRef(null) + const sceneRef = useRef(null) + + // ── Mount Two.js ──────────────────────────────────────────────────────── + useEffect(() => { + const el = containerRef.current! + const two = new Two({ type: Two.Types.svg, autostart: true, fitted: true }) + two.appendTo(el) + twoRef.current = two + sceneRef.current = buildScene(two, styleOptions) + + two.bind('update', () => { + updateScene(sceneRef.current!, currentTimeRef.current, /* latest lines/style via refs */) + }) + + const ro = new ResizeObserver(() => { + two.width = el.clientWidth + two.height = el.clientHeight + rebuildLayout(sceneRef.current!, two.width, two.height) + }) + ro.observe(el) + + return () => { + two.pause() + two.unbind('update') + ro.disconnect() + el.removeChild(two.renderer.domElement) + } + }, []) // mount once + + // ── Sync style changes (no scene rebuild, just update properties) ─────── + useEffect(() => { + if (!sceneRef.current) return + applyStyleOptions(sceneRef.current, styleOptions) + }, [styleOptions]) + + return
+} +``` + +`updateScene` runs inside `two.bind('update')` — equivalent to the current RAF +loop. It reads `currentTimeRef.current` directly (a ref, no React re-render) and +mutates Two.js object properties. React only re-renders when structural changes +happen (line count changes, style mode changes) — identical to the current strategy. + +--- + +## Word Metrics Cache + +Computing `measureText` for every word on every render is expensive. Cache by +`${fontFamily}:${fontSize}:${text}`: + +```ts +const metricsCache = new Map() + +function measureWord(text: string, font: string): number { + const key = `${font}:${text}` + if (metricsCache.has(key)) return metricsCache.get(key)! + mctx.font = font + const w = mctx.measureText(text).width + metricsCache.set(key, w) + return w +} +``` + +Cache is invalidated (cleared) when font family or size changes. + +--- + +## Scene Rebuild vs. Property Update + +Two cases require different responses: + +| Trigger | Action | +|---------|--------| +| Active line changes (new `activeIdx`) | Update `linesGroup` child visibility + word clip rects for the new active line; set up new `clipRect` nodes for new active words | +| Font / font size changes | Full scene rebuild: re-measure all words, rebuild `wordUnit` groups, reposition all `lineGroup` objects | +| Colour / opacity / style mode changes | Property update only: `two.text.fill = newColor`, `lineGroup.opacity = newOpacity` | +| Scene resize | Re-centre `linesGroup`, recompute `lineStride`, re-measure words (font-size-dependent if `em`-relative) | +| `lines` array changes (LRC edit) | Re-index `lineGroups` array, add/remove groups as needed | + +--- + +## Implementation Phases + +### Phase 1 — Static text rendering +- Set up Two.js SVG renderer in a `LyricsCanvas` component. +- Render all lines as `Two.Text` objects, positioned vertically. +- No animation yet — static layout with correct fonts, colours, spacing. +- Verify text rendering quality vs current CSS (especially on HiDPI). + +### Phase 2 — Scroll + fade animation +- Implement `linesGroup.translation.y` lerp in the RAF update loop. +- Implement per-line `opacity` and `scale` lerp for **scroll** and **fade** styles. +- Connect `currentTimeRef` → `findActiveLine` → active index. +- Replace the CSS scroll div with `LyricsCanvas` for these two modes. + +### Phase 3 — Word fill (karaoke / glow style) +- Compute word metrics for active line. +- Inject `` + `` nodes into Two.js SVG defs. +- Implement per-frame clip rect width update in update loop. +- Implement stroke/shadow layering (behind/front group pattern). + +### Phase 4 — Glow + blur styles +- Inject SVG filter defs (``, ``). +- Apply glow filter to active line group. +- Scale blur `stdDeviation` with distance from active line. + +### Phase 5 — Speaker dots + full style parity +- Add speaker dot circles. +- Verify all `LyricsStyleOptions` properties are honoured. +- Replace `KaraokeWord` / CSS gradient path entirely; remove old DOM scroll stack. + +### Phase 6 — Polish + fallback +- Benchmark SVG vs Canvas2D on a long test file; switch default renderer if needed. +- Verify fullscreen mode (scene resizes via ResizeObserver + `two.fitted`). +- Verify album art colour extraction (still Canvas2D, unchanged). +- Remove `KaraokeWord` component and all `WebkitBackgroundClip` CSS. + +--- + +## What Does NOT Change + +| Item | Reason | +|------|--------| +| `KaraokeEditor.tsx` | Waveform + timing table — completely separate | +| Audio playback engine | `