From fa99da7100ccc231d1d640499c3ea6ab6ddcb83c Mon Sep 17 00:00:00 2001 From: translator Date: Wed, 11 Mar 2026 09:44:36 +0900 Subject: [PATCH] Added syllable-level timing adjustment to the karaoke editor. --- Lyrics Editor Overview.md | 179 ++ Lyrics Editor_Plan.md | 177 ++ SUMMARIES.md | 144 +- StemEngine WebUI.md | 396 +++ Syllable Timing Plan.md | 235 ++ backend/services/bs_roformer_sw_separator.py | 3 - backend/static/index.html | 4 +- .../2020310_LyricsLivePreview_Claude.tsx | 2317 +++++++++++++++++ frontend/src/components/KaraokeEditor.tsx | 841 ++++++ frontend/src/components/LyricsLivePreview.tsx | 28 +- msengine.yml | 29 - stemengine.yml | 26 - 12 files changed, 4311 insertions(+), 68 deletions(-) create mode 100644 Lyrics Editor Overview.md create mode 100644 Lyrics Editor_Plan.md create mode 100644 StemEngine WebUI.md create mode 100644 Syllable Timing Plan.md create mode 100755 frontend/src/components/2020310_LyricsLivePreview_Claude.tsx create mode 100644 frontend/src/components/KaraokeEditor.tsx diff --git a/Lyrics Editor Overview.md b/Lyrics Editor Overview.md new file mode 100644 index 0000000..5316d67 --- /dev/null +++ b/Lyrics Editor Overview.md @@ -0,0 +1,179 @@ +# Lyrics Editor Overview + +The Lyrics Editor is an Aegisub-style panel built into the Lyrics Preview component. It gives you a visual waveform, a timing table, and inline editing tools — all synchronized with the audio player. + +--- + +## Opening the Editor + +In the Lyrics Preview toolbar, click the **waveform icon** (Activity/zigzag icon) to toggle the editor panel. It sits between the toolbar and the lyrics display area. + +Clicking the icon again, or clicking the **Style** (palette) or **Edit** (pencil) buttons, closes the panel. + +--- + +## Layout + +``` +┌─────────────────────────────────────────┐ +│ Toolbar [Scroll|Fade|…] [⊙][🎨][✏][⊞] │ +├─────────────────────────────────────────┤ +│ WAVEFORM − 1× + │ +│ ▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ │ +│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ │ ← scrollbar (zoom only) +├─────────────────────────────────────────┤ +│ LINES 42 │ +│ # Start End Text … │ +│ 1 0:00.00 0:03.21 Hello world │ +│ 2 0:03.21 0:05.80 This is a test │ +│ … │ +└─────────────────────────────────────────┘ +``` + +--- + +## Waveform Panel + +### Playback and seeking +- **Click anywhere** on the waveform to seek the audio to that position. +- The **white playhead** moves in real time during playback. +- The portion to the left of the playhead is drawn in bright blue; the unplayed portion is dark blue. + +### Line markers +- Small vertical ticks at the top of the waveform mark the **start time of each lyric line**. +- The **active line marker** (the line currently playing) is taller and brighter blue. + +### Zoom +| Action | Effect | +|--------|--------| +| `Ctrl` + scroll wheel | Zoom in / out, centred on the cursor position | +| `+` button (top-right) | Zoom in one step (doubles the zoom level) | +| `−` button (top-right) | Zoom out one step | +| `✕` button (top-right) | Reset zoom to 1× (full song view) | + +Zoom levels: 1×, 2×, 4×, 8×, 16×, 32×. + +### Panning (when zoomed in) +| Action | Effect | +|--------|--------| +| Scroll wheel (no modifier) | Pan left / right | +| Shift + scroll wheel | Pan left / right (on trackpads that send vertical scroll) | +| Drag the scrollbar below the canvas | Pan to any position | + +> **Tip:** The scrollbar only appears when zoom > 1×. + +--- + +## Timing Table + +The table below the waveform lists every lyric line with the following columns: + +| Column | Description | +|--------|-------------| +| **#** | Line number (read-only) | +| **Start** | Line start time in `m:ss.xx` format — **editable** | +| **End** | Derived end time (next line's start, or total duration for the last line) — read-only | +| **Text** | Lyric text — **editable** | +| **Speaker** | Speaker ID from diarized transcription — read-only, shown only when diarization data is present | + +### Navigation +- **Click any row** to seek the audio to that line's start time. +- **Arrow keys** (`←` / `→`) jump to the previous / next line while the editor is open (active when no input field is focused). +- The **active row** (currently playing line) is highlighted in blue and automatically scrolls into view. + +--- + +## Inline Editing + +### Editing a start time +1. **Double-click** the **Start** cell of any row. +2. An input field appears pre-filled with the current timestamp (`m:ss.xx`). +3. Type the new time. Accepted formats: `m:ss`, `m:ss.x`, `m:ss.xx` (e.g. `1:23.45`). +4. Press **Enter** or click away to commit. The line is re-sorted if the new time changes its order. +5. Press **Escape** to cancel without saving. + +> If you type an invalid timestamp the edit is silently discarded and the original value is kept. + +### Editing text +1. **Double-click** the **Text** cell of any row. +2. An input field appears with the current text. +3. Edit freely and press **Enter** or click away to commit. +4. Press **Escape** to cancel. + +> **Note:** If a line has word-level (ELRC) timing, editing its text will replace the sub-word timestamps with plain text. This is indicated in the cell tooltip. The audio timing for the line itself is preserved; only the per-word karaoke fill is lost. + +### Tab navigation +While an input field is open: +- **Tab** commits the current edit and moves to the next editable cell (Start → Text → next row's Start → …). +- **Shift + Tab** moves in reverse. + +### How edits propagate +Every committed edit immediately: +1. Updates the live lyrics preview (the karaoke/scroll/fade view updates in real time). +2. Saves back to the project's LRC text via the same path as the raw text editor (`onLrcChange`). + +--- + +## Word / Syllable Timing + +Lines that were transcribed with word-level (ELRC) timestamps can have their individual word timings adjusted. + +### Expanding a line + +In the **#** column, lines with word data show a small **▶** indicator next to the row number. Click the **#** cell to expand that line into word sub-rows. Click again to collapse. + +### Word sub-rows + +Each word sub-row shows: + +| Column | Description | +|--------|-------------| +| `└` | Indent marker — identifies this as a word row | +| **Start** | Word start time — **editable** (double-click) | +| **End** | Next word's start time, or line end — read-only | +| **Text** | Word text — read-only | + +- **Click a word sub-row** to seek the audio to that word's start time. +- **Double-click Start** to open an inline `m:ss.xx` input (same flow as line start editing). +- **Tab / Shift+Tab** moves between word start cells within the same line. +- Word times are clamped: cannot go before the parent line's start or after the next word's start. + +### Waveform word markers + +When a line is expanded, **amber vertical ticks** appear on the waveform at each word's start position — distinct from the blue line-start markers. + +### Drag-to-retime + +While a line is expanded, hover the waveform over an amber word marker — the cursor changes to **⇔**. Drag left or right to move that word's start time: + +- The marker follows the cursor in real time (live preview). +- Release to commit. The change is saved immediately and updates the karaoke fill/preview. +- Dragging is clamped to the same bounds as inline editing. +- If you click without dragging, it seeks as normal. + +> **Note:** Word timing is only available when the source transcription includes ELRC word-level timestamps. If a line's text is edited inline, its word timing is replaced with plain text. + +--- + +## Multi-stem mode + +When the project uses separated stems instead of a single audio file, the waveform is decoded from the **first stem track**. All other editor features (timing table, editing, navigation, word timing) work identically regardless of the number of stems. + +> **Note:** In multi-stem mode the waveform is not rendered, so drag-to-retime is unavailable. Word start times can still be edited via the inline double-click input in the table. + +--- + +## Keyboard Reference + +| Key | Action | +|-----|--------| +| `←` | Jump to the previous line | +| `→` | Jump to the next line | +| `Enter` (in cell) | Commit edit | +| `Escape` (in cell) | Cancel edit | +| `Tab` (in cell) | Commit and move to next editable cell | +| `Shift+Tab` (in cell) | Commit and move to previous editable cell | +| `Ctrl+scroll` | Zoom waveform in / out | +| `scroll` | Pan waveform left / right (when zoomed) | +| Click **#** cell | Expand / collapse word sub-rows (lines with word timing only) | +| Drag amber marker | Retime a word on the waveform (when a line is expanded) | diff --git a/Lyrics Editor_Plan.md b/Lyrics Editor_Plan.md new file mode 100644 index 0000000..f970fb1 --- /dev/null +++ b/Lyrics Editor_Plan.md @@ -0,0 +1,177 @@ +# Aegisub-Style Lyrics Editor: Implementation Plan + +## What Already Exists (no re-invention needed) + +- `LrcLine` / `LrcWord` types with `time: number` fields in `LyricsLivePreview.tsx` +- `seekTime(t)` already handles both single-track and multi-stem seek +- `currentTimeRef` is updated every frame by the existing RAF loop +- `audioRef` / `stemAudioRefs` already manage the `