8.2 KiB
8.2 KiB
Aegisub-Style Lyrics Editor: Implementation Plan
What Already Exists (no re-invention needed)
LrcLine/LrcWordtypes withtime: numberfields inLyricsLivePreview.tsxseekTime(t)already handles both single-track and multi-stem seekcurrentTimeRefis updated every frame by the existing RAF loopaudioRef/stemAudioRefsalready manage the<audio>elementsfindActiveLine()/findActiveWord()binary scans already implementedparseLrcContent()produces sortedLrcLine[]from ELRC/LRC text
Core Recommendation: Raw Web Audio API + Canvas (no WaveSurfer.js)
WaveSurfer.js would conflict with the existing multi-stem audio setup — you cannot create two MediaElementSource nodes from the same HTMLAudioElement (throws InvalidStateError). Instead:
- Fetch the audio URL once
- Decode with a dedicated
AudioContext(separate from the existing stem gain graph) - Downsample PCM data to ~1200 peak values
- Pre-render to an offscreen canvas
- RAF-driven playhead redraws on top each frame
This is the same pattern the existing component uses for karaoke word fill.
New File: frontend/src/components/KaraokeEditor.tsx
One new component with two sub-sections.
Props Interface
interface KaraokeEditorProps {
lines: LrcLine[]
currentTimeRef: React.MutableRefObject<number>
duration: number
audioUrl: string | undefined
onSeek: (t: number) => void // calls seekTime() in LyricsLivePreview
onLrcChange?: (newLrc: string) => void
}
WaveformCanvas
fetch(audioUrl)→arrayBuffer→ dedicatedAudioContext.decodeAudioData()→ close context after decoding- Downsample: chunk into ~1200 buckets, take
max(abs)per bucket →Float32Array - Pre-render bars to offscreen canvas (two colors: dim = future, bright = played)
- RAF loop reads
currentTimeRef.current→ draws 2px playhead at(t / duration) * width onClick(e)→onSeek((x / width) * duration)
TimingTable (Aegisub-style grid)
| # | Start | End (derived) | Text | Speaker |
|---|
- Start time and text cells are inline-editable (
<input>on click,mm:ss.xxformat) - End time = next line's start time, shown read-only
- Clicking a row calls
onSeek(line.time) - Active row highlighted via RAF loop reading
currentTimeRef— no React re-renders per frame scrollIntoView({ block: 'nearest', behavior: 'smooth' })on active row index change- On edit → reconstruct LRC string → call
onLrcChange(newLrc)
Minimal Changes to LyricsLivePreview.tsx
- Add an "Aegisub" toggle button in the existing toolbar (next to existing Palette/Pencil buttons)
- Render
<KaraokeEditor>below the transport bar when toggled - Pass
currentTimeRef,duration,audioUrl,lines,seekTime,onLrcChangeas props
No changes to the RAF loop, audio element wiring, or any render functions.
Data Flow
audioUrl ──────────────────────────────► WaveformCanvas (decode + draw)
currentTimeRef (RAF, no re-renders) ───► playhead position + active row highlight
onSeek(t) ◄──────────────────────────── click on waveform OR click table row
→ calls seekTime() in LyricsLivePreview
onLrcChange(newLrc) ◄────────────────── inline cell edits → reconstructed LRC string
State Management
- No new global state.
KaraokeEditoris a leaf component. currentTimewithout re-renders:KaraokeEditorreadscurrentTimeRef.currentin its own internal RAF loop — never subscribes to ReactcurrentTimestate. Matches existing DOM-mutation pattern.- Waveform data:
useRef<Float32Array | null>— never in React state (doesn't drive renders, only canvas draws). - Editing state:
useState<LrcLine[]>local override insideKaraokeEditorfor in-progress edits, identical to howeditedLrcworks inLyricsLivePreview. - No context, Zustand, or new stores needed.
Critical Files
| File | Role |
|---|---|
frontend/src/components/LyricsLivePreview.tsx |
Modify: add toolbar button, render <KaraokeEditor>, expose seekTime + currentTimeRef |
frontend/src/components/KaraokeEditor.tsx |
Create: waveform canvas + timing table |
frontend/src/utils/audioRegistry.ts |
Reference only: follow existing stable-ref patterns; do NOT call registerPlayback from editor |
frontend/src/types.ts |
Reference: check naming before adding any new exported types |
frontend/src/pages/ProjectDetail.tsx |
Reference: caller context for audioUrl, lrcText, onLrcChange prop wiring |
Implementation Order
Phase 1 — Waveform canvas (read-only)
- Create
KaraokeEditor.tsxwithWaveformCanvasonly - Fetch + decode audio URL with dedicated
AudioContext - Downsample → offscreen canvas pre-render
- RAF loop: playhead line
- Click-to-seek via
onSeek - Wire into
LyricsLivePreview.tsx: toolbar toggle button, conditional render below transport
Phase 2 — Timing table (read-only)
- Add
TimingTableinsideKaraokeEditor - Render all lines with start/end times, text, speaker columns
- Active row highlight +
scrollIntoViewvia shared RAF loop - Row click →
onSeek(line.time)
Phase 3 — Inline editing
- Editable start-time cells (
mm:ss.xxinput) - Editable text cells
- On change: reconstruct LRC string from modified lines array
- Call
onLrcChange(newLrc)to propagate upward
Phase 4 — Polish
- Waveform horizontal zoom + scroll
- Line-start markers on waveform; arrow-key navigation (←/→ jump lines)
Phase 5 — Syllable / word timing editor
All changes in KaraokeEditor.tsx only.
New constants:
const WORD_MARKER_COLOR = 'rgba(251,191,36,0.75)' // amber-400/75
const WORD_MARKER_DRAG_COLOR = '#fbbf24' // amber-400 bright
const WORD_MARKER_HIT_PX = 8 // ±px hit zone for drag
Step 5-A — Expandable word sub-rows in TimingTable
- Lines with
words.length > 0show a▶/▼toggle in the#column - Clicking expands into word sub-rows (indented, amber left border) directly below the parent line
- Sub-row columns: indent
└| Start (editable,m:ss.xx) | End (read-only = next word's time) | Text (read-only) | Speaker (blank) - Double-click Start on a word sub-row opens the same inline
<input>as line start editing - Word times clamped:
[line.time … nextWord.time − 0.01s] - Editing text on a parent line drops
words[]and auto-collapses the expanded row - New props on
TimingTable:expandedRow,onExpandRow,onWordTimeChange expandedRow: number | nullstate lives inKaraokeEditorhandleWordTimeChange(lineIdx, wordIdx, newTime)inKaraokeEditormutates words, callsreconstructLrc → onLrcChange
Step 5-B — Word markers on waveform (read-only)
drawFramegains two optional trailing params:wordLines?: LrcLineKE[],dragPreview?: { wordIdx: number; time: number } | null- When
selectedLineIdxis set, draws 2 px × 14 px amber ticks for each word in that line WaveformCanvasgainsselectedLineIdx: number | nullprop
Step 5-C — Drag-to-retime on waveform
- Replace
onClickwith unified pointer-event handlers (onPointerDown/onPointerMove/onPointerUp) +canvas.setPointerCapture onPointerDownwithinWORD_MARKER_HIT_PXof a word marker → start drag (dragStateRef)onPointerMove→ updatedragPreviewRef,dirtyRef = true(no React state change)onPointerUp→ if moved: commit viaonWordTimeDrag('commit', …); else: treat as seek click- Cursor:
ew-resizewhen hovering a word marker hit zone,crosshairotherwise dragPreviewRefandsetDirtycallback passed fromKaraokeEditortoWaveformCanvas
Data flow:
expandedRow ──────────────────────► TimingTable (sub-rows) + WaveformCanvas (word markers)
onWordTimeChange/onWordTimeDrag ──► KaraokeEditor mutates lines → reconstructLrc → onLrcChange
dragPreviewRef (RAF, no re-render) ► drawFrame dragPreview param → amber drag-position tick