Files
stemengine_old/Lyrics Editor_Plan.md
T

8.2 KiB
Raw Blame History

Aegisub-Style Lyrics Editor: Implementation Plan

What Already Exists (no re-invention needed)

  • LrcLine / LrcWord types with time: number fields in LyricsLivePreview.tsx
  • seekTime(t) already handles both single-track and multi-stem seek
  • currentTimeRef is updated every frame by the existing RAF loop
  • audioRef / stemAudioRefs already manage the <audio> elements
  • findActiveLine() / findActiveWord() binary scans already implemented
  • parseLrcContent() produces sorted LrcLine[] from ELRC/LRC text

Core Recommendation: Raw Web Audio API + Canvas (no WaveSurfer.js)

WaveSurfer.js would conflict with the existing multi-stem audio setup — you cannot create two MediaElementSource nodes from the same HTMLAudioElement (throws InvalidStateError). Instead:

  1. Fetch the audio URL once
  2. Decode with a dedicated AudioContext (separate from the existing stem gain graph)
  3. Downsample PCM data to ~1200 peak values
  4. Pre-render to an offscreen canvas
  5. RAF-driven playhead redraws on top each frame

This is the same pattern the existing component uses for karaoke word fill.


New File: frontend/src/components/KaraokeEditor.tsx

One new component with two sub-sections.

Props Interface

interface KaraokeEditorProps {
  lines: LrcLine[]
  currentTimeRef: React.MutableRefObject<number>
  duration: number
  audioUrl: string | undefined
  onSeek: (t: number) => void       // calls seekTime() in LyricsLivePreview
  onLrcChange?: (newLrc: string) => void
}

WaveformCanvas

  • fetch(audioUrl) → arrayBuffer → dedicated AudioContext.decodeAudioData() → close context after decoding
  • Downsample: chunk into ~1200 buckets, take max(abs) per bucket → Float32Array
  • Pre-render bars to offscreen canvas (two colors: dim = future, bright = played)
  • RAF loop reads currentTimeRef.current → draws 2px playhead at (t / duration) * width
  • onClick(e) → onSeek((x / width) * duration)

TimingTable (Aegisub-style grid)

# Start End (derived) Text Speaker
  • Start time and text cells are inline-editable (<input> on click, mm:ss.xx format)
  • End time = next line's start time, shown read-only
  • Clicking a row calls onSeek(line.time)
  • Active row highlighted via RAF loop reading currentTimeRef — no React re-renders per frame
  • scrollIntoView({ block: 'nearest', behavior: 'smooth' }) on active row index change
  • On edit → reconstruct LRC string → call onLrcChange(newLrc)

Minimal Changes to LyricsLivePreview.tsx

  1. Add an "Aegisub" toggle button in the existing toolbar (next to existing Palette/Pencil buttons)
  2. Render <KaraokeEditor> below the transport bar when toggled
  3. Pass currentTimeRef, duration, audioUrl, lines, seekTime, onLrcChange as props

No changes to the RAF loop, audio element wiring, or any render functions.


Data Flow

audioUrl ──────────────────────────────► WaveformCanvas (decode + draw)
currentTimeRef (RAF, no re-renders) ───► playhead position + active row highlight
onSeek(t) ◄────────────────────────────  click on waveform OR click table row
                                             → calls seekTime() in LyricsLivePreview
onLrcChange(newLrc) ◄──────────────────  inline cell edits → reconstructed LRC string

State Management

  • No new global state. KaraokeEditor is a leaf component.
  • currentTime without re-renders: KaraokeEditor reads currentTimeRef.current in its own internal RAF loop — never subscribes to React currentTime state. Matches existing DOM-mutation pattern.
  • Waveform data: useRef<Float32Array | null> — never in React state (doesn't drive renders, only canvas draws).
  • Editing state: useState<LrcLine[]> local override inside KaraokeEditor for in-progress edits, identical to how editedLrc works in LyricsLivePreview.
  • No context, Zustand, or new stores needed.

Critical Files

File Role
frontend/src/components/LyricsLivePreview.tsx Modify: add toolbar button, render <KaraokeEditor>, expose seekTime + currentTimeRef
frontend/src/components/KaraokeEditor.tsx Create: waveform canvas + timing table
frontend/src/utils/audioRegistry.ts Reference only: follow existing stable-ref patterns; do NOT call registerPlayback from editor
frontend/src/types.ts Reference: check naming before adding any new exported types
frontend/src/pages/ProjectDetail.tsx Reference: caller context for audioUrl, lrcText, onLrcChange prop wiring

Implementation Order

Phase 1 — Waveform canvas (read-only)

  1. Create KaraokeEditor.tsx with WaveformCanvas only
  2. Fetch + decode audio URL with dedicated AudioContext
  3. Downsample → offscreen canvas pre-render
  4. RAF loop: playhead line
  5. Click-to-seek via onSeek
  6. Wire into LyricsLivePreview.tsx: toolbar toggle button, conditional render below transport

Phase 2 — Timing table (read-only)

  1. Add TimingTable inside KaraokeEditor
  2. Render all lines with start/end times, text, speaker columns
  3. Active row highlight + scrollIntoView via shared RAF loop
  4. Row click → onSeek(line.time)

Phase 3 — Inline editing

  1. Editable start-time cells (mm:ss.xx input)
  2. Editable text cells
  3. On change: reconstruct LRC string from modified lines array
  4. Call onLrcChange(newLrc) to propagate upward

Phase 4 — Polish

  • Waveform horizontal zoom + scroll
  • Line-start markers on waveform; arrow-key navigation (←/→ jump lines)

Phase 5 — Syllable / word timing editor

All changes in KaraokeEditor.tsx only.

New constants:

const WORD_MARKER_COLOR        = 'rgba(251,191,36,0.75)'   // amber-400/75
const WORD_MARKER_DRAG_COLOR   = '#fbbf24'                  // amber-400 bright
const WORD_MARKER_HIT_PX       = 8                          // ±px hit zone for drag

Step 5-A — Expandable word sub-rows in TimingTable

  • Lines with words.length > 0 show a ▶/▼ toggle in the # column
  • Clicking expands into word sub-rows (indented, amber left border) directly below the parent line
  • Sub-row columns: indent └ | Start (editable, m:ss.xx) | End (read-only = next word's time) | Text (read-only) | Speaker (blank)
  • Double-click Start on a word sub-row opens the same inline <input> as line start editing
  • Word times clamped: [line.time … nextWord.time − 0.01s]
  • Editing text on a parent line drops words[] and auto-collapses the expanded row
  • New props on TimingTable: expandedRow, onExpandRow, onWordTimeChange
  • expandedRow: number | null state lives in KaraokeEditor
  • handleWordTimeChange(lineIdx, wordIdx, newTime) in KaraokeEditor mutates words, calls reconstructLrc → onLrcChange

Step 5-B — Word markers on waveform (read-only)

  • drawFrame gains two optional trailing params: wordLines?: LrcLineKE[], dragPreview?: { wordIdx: number; time: number } | null
  • When selectedLineIdx is set, draws 2 px × 14 px amber ticks for each word in that line
  • WaveformCanvas gains selectedLineIdx: number | null prop

Step 5-C — Drag-to-retime on waveform

  • Replace onClick with unified pointer-event handlers (onPointerDown/onPointerMove/onPointerUp) + canvas.setPointerCapture
  • onPointerDown within WORD_MARKER_HIT_PX of a word marker → start drag (dragStateRef)
  • onPointerMove → update dragPreviewRef, dirtyRef = true (no React state change)
  • onPointerUp → if moved: commit via onWordTimeDrag('commit', …); else: treat as seek click
  • Cursor: ew-resize when hovering a word marker hit zone, crosshair otherwise
  • dragPreviewRef and setDirty callback passed from KaraokeEditor to WaveformCanvas

Data flow:

expandedRow ──────────────────────► TimingTable (sub-rows) + WaveformCanvas (word markers)
onWordTimeChange/onWordTimeDrag ──► KaraokeEditor mutates lines → reconstructLrc → onLrcChange
dragPreviewRef (RAF, no re-render) ► drawFrame dragPreview param → amber drag-position tick