feat(review): hold a word's resize handle at the row edge to keep extending it (v0.252.0) (#1090)

## Summary
In the lyrics review Waveforms rows, the view is the segment ±1s and it
was fixed during a drag (and leaving the row ended the drag). So a word
ending 5s+ too early took several drag-release cycles to fix.

Now, holding a word's resize handle within 24px of the edge it's heading
toward (or past it) keeps growing the word at 1–4 s/s (faster nearer the
edge). The view zooms out so the handle stays under the cursor. It stops
when:
- the pointer leaves the edge zone (then the handle tracks the pointer
normally in the zoomed view),
- the mouse is released (commits once, as before), or
- it reaches the neighbouring word, including the **next line's first
word** even when that's beyond the drawn context padding (new
`computeNeighbourBoundsBySegment`), else the audio end/start.

Works for both handles and also in the Edit Segment modal timeline (same
`TimelineEditor`).

## Changes
- `TimelineEditor.tsx`: absolute pointer→time mapping for resizes (grab
offset), a rAF auto-extend loop, a `dragView` override that widens the
view mid-drag, and window-level mousemove/mouseup (so drags survive
leaving the row, with text selection prevented). Resizes now clamp to
the neighbour instead of freezing short of it on overshoot, and never
grow into the adjacent line (pre-existing overlaps aren't snapped back).
- `contextWords.ts`: `computeNeighbourBoundsBySegment` (prev line end /
next line start per segment), wired through `TranscriptionView` →
`WaveformSegmentRow`.
- Version 0.251.1 → 0.252.0; docs/README.md recent-changes entry.

## Testing
- 7 new `TimelineEditor` tests (fake rAF): extends while held at the
edge, stops at the next line / same-segment neighbour, stops when
leaving the zone, respects the drag threshold, left handle bounded by
the previous line, commits once when released outside the row. Plus 2
new util tests.
- `make test`: all backend + 1476 frontend tests pass.
- Checked in the browser against the local fixture harness: the last
line's "shoes" extended with the handle glued to the edge while the row
zoomed out; "alright" stopped exactly at the next line's "Because".

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* In lyrics review waveforms and the Edit Segment timeline, dragging a
resize handle within 24 pixels of the direction of travel now
automatically extends the word. The timeline zooms to keep the handle
under the cursor as the word grows.
* Resizing stops at neighboring words, adjacent segment boundaries, or
the audio limit. You can continue dragging outside the row, and
releasing completes the resize.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 5.5 authored and GitHub committed 2026-09-30 00:54:00 -04:00
1 parent 69986bb844
commit eaa70e04ea
11 files changed
+416 -82

No files matched your search

+1
View File
@@ -57,6 +57,7 @@
## Recent Changes
- **Hold a word's resize handle at the row edge to keep extending it** (2026-09-30, v0.252.0): in the lyrics review Waveforms rows (and the Edit Segment timeline) the view is the segment ±1s and was fixed during a drag, so a word ending 5s+ too early took several drag-release cycles. Holding a resize handle within 24px of the edge it's heading toward (or past it) now keeps growing the word at 1–4 s/s (faster nearer the edge), zooming the view out so the handle stays under the cursor. It stops when the pointer leaves the zone, on release, or at the neighbouring word, including the next/previous line's first/last word (`computeNeighbourBoundsBySegment`), else the audio bounds. Drags are tracked on `window`, so leaving the row no longer abandons them, and resizes clamp to the neighbour instead of freezing short of it (`TimelineEditor.tsx`).
- **Mismatched user instrumentals are lined up, not rejected + tenant bulk upload modal** (2026-09-29, v0.251.0): A Randy Vild bulk upload lost 3 of 10 tracks to the 0.5s duration check, and all three were valid edits of the same song: an extra 2.97s count-in on the instrumental, a 6.5s longer outro, and an outro 4.45s short. `uploads-complete` now conforms the instrumental to the mix instead of cancelling (`backend/services/instrumental_conform.py`, reusing `derived_vocals.find_offset`). It measures the offset at 25% and 75% of the song, trims/pads the start, and fades-and-cuts or pads the end. It still rejects files that don't correlate (wrong pairing, ~0.01 vs 0.78–0.90 for real pairs) or whose offset changes mid-song. The single-track tenant form and own-instrumental field now only check the size cap in the browser. Tenant bulk: shows `UploadProgressModal` for the whole batch (aggregate bytes, speed/ETA, tracks submitted/failed, offline notice); a job the server rejected is dropped so a retry creates a fresh job (previously a retry's 400 on the cancelled job was read as "already processing" and shown as Submitted).
- **Vocal waveforms for own-instrumental jobs** (2026-09-26, v0.244.1): jobs with a user-supplied instrumental skip GPU separation, so the lyrics review Waveforms mode had no vocals stem (`/vocals-peaks` 404). The lyrics worker now derives approximate vocals on CPU, alongside transcription: it aligns the instrumental to the mix by FFT cross-correlation (±2s), fits a gain by least squares, and subtracts (`backend/services/derived_vocals.py`, numpy + ffmpeg, a few seconds). The result is stored as `stems.vocals_derived` (mono 22.05 kHz FLAC), which `vocals_stem_path` uses only as a last resort, so screens pre-computes the peaks as usual. It's waveform-only: never shipped as a deliverable stem, never downloaded by the remote CLI, and never used for the original-vocals guide. Safeguards: alignment correlates a fixed 60s excerpt (memory doesn't scale with track length); inputs over 20 min are skipped; ffmpeg calls time out; the handoff waits at most 3 min; nothing is registered when the instrumental didn't cancel (residual > 85% of the mix), since a full-mix envelope would mislead timing edits. Best-effort: any failure logs and the job proceeds.
- **User-instrumental renders: WAV/M4A/OGG/AAC no longer fail** (2026-09-26, v0.243.1): the orchestrator render path sent only `instrumental_selection="custom"` to the GCE encoder, whose custom lookup accepted just `*custom_instrumental*.{flac,mp3}`. Tenant MP3s worked, but a `.wav` user instrumental failed with "No instrumental audio found" (job 3ea34552). The orchestrator now passes `existing_instrumental` (the encoder downloads it and matches any extension), and the custom lookup (`resolve_instrumental` in `gce_encoding/main.py`) accepts any audio extension.
@@ -62,6 +62,9 @@ interface EditTimelineSectionProps {
words: Word[]
/** Neighbouring segments' words, drawn greyed/read-only in the timeline padding for context. */
contextWords?: Word[]
/** Nearest word edges in the previous / next lines — resizes (incl. edge auto-extend) stop there. */
prevBoundaryTime?: number | null
nextBoundaryTime?: number | null
startTime: number
endTime: number
currentTime?: number
@@ -80,6 +83,8 @@ interface EditTimelineSectionProps {
export default function EditTimelineSection({
words,
contextWords,
prevBoundaryTime,
nextBoundaryTime,
startTime,
endTime,
currentTime,
@@ -115,6 +120,8 @@ export default function EditTimelineSection({
<TimelineEditor
words={words}
contextWords={contextWords}
prevBoundaryTime={prevBoundaryTime}
nextBoundaryTime={nextBoundaryTime}
startTime={startTime}
endTime={endTime}
onWordUpdate={onWordUpdate}
@@ -68,7 +68,13 @@ import { useAutoCorrect } from '@/hooks/useAutoCorrect'
import type { AiSuggestion } from '@/lib/api/autoCorrect'
import type { ServerSuggestionUndoInfo } from '@/lib/lyrics-review/utils/autoCorrectApply'
import { getWordsFromIds } from '@/lib/lyrics-review/utils/wordUtils'
import { applyOffsetToCorrectionData, applyOffsetToSegment, applyOffsetToWord } from '@/lib/lyrics-review/utils/timingUtils'
import {
applyOffsetToCorrectionData,
applyOffsetToSegment,
applyOffsetToTime,
applyOffsetToWord,
} from '@/lib/lyrics-review/utils/timingUtils'
import { computeNeighbourBoundsBySegment } from '@/lib/lyrics-review/utils/contextWords'
import { resolveInitialViewMode } from '@/lib/lyrics-review/utils/segmentTiming'
import { VocalsAudioDataLoader } from './VocalsAudioDataLoader'
@@ -1504,6 +1510,18 @@ export default function LyricsAnalyzer({
return timingOffsetMs !== 0 ? nearby.map((w) => applyOffsetToWord(w, timingOffsetMs)) : nearby
}, [editModalSegment, data.corrected_segments, timingOffsetMs])
// Nearest word edges in the neighbouring lines (offset-applied), so the modal timeline's resize /
// edge auto-extend can't grow a word into an adjacent line beyond the drawn context window.
const editNeighbourBounds = useMemo(() => {
if (!editModalSegment || editModalSegment.index === null) return null
const bounds = computeNeighbourBoundsBySegment(data.corrected_segments).get(editModalSegment.index)
if (!bounds) return null
return {
prevEnd: applyOffsetToTime(bounds.prevEnd, timingOffsetMs),
nextStart: applyOffsetToTime(bounds.nextStart, timingOffsetMs),
}
}, [editModalSegment, data.corrected_segments, timingOffsetMs])
// Timing offset handlers
const handleOpenTimingOffsetModal = useCallback(() => {
setIsTimingOffsetModalOpen(true)
@@ -1830,6 +1848,8 @@ export default function LyricsAnalyzer({
onPlaySegment={handlePlaySegment}
currentTime={currentAudioTime}
contextWords={editContextWords}
prevBoundaryTime={editNeighbourBounds?.prevEnd ?? null}
nextBoundaryTime={editNeighbourBounds?.nextStart ?? null}
originalTranscribedSegment={
editModalSegment?.segment &&
editModalSegment?.index !== null &&
@@ -1,6 +1,6 @@
'use client'
import { useRef, useState } from 'react'
import { useContext, useEffect, useRef, useState } from 'react'
import { Word } from '@/lib/lyrics-review/types'
import { WordDecoration } from '@/lib/lyrics-review/utils/wordDecorations'
import { cn } from '@/lib/utils'
@@ -42,11 +42,25 @@ interface TimelineEditorProps {
/** Fires when a word bar is Ctrl/Cmd-clicked (read from the event, so no key tracking is
needed). Used by the Edit Segment modal to delete a word without scrolling the word list. */
onWordCtrlDelete?: (index: number) => void
/** End of the nearest timed word in an *earlier* segment. Resizing (incl. edge auto-extend)
stops here so a word can't grow into the previous line. */
prevBoundaryTime?: number | null
/** Start of the nearest timed word in a *later* segment — the auto-extend / resize stop. */
nextBoundaryTime?: number | null
}
// Pointer travel (px) beyond which a press counts as a drag rather than a click.
const DRAG_THRESHOLD_PX = 4
// Edge auto-extend: while a resize handle is held within this many px of the timeline edge it
// is heading toward (or beyond it), the word keeps growing and the view zooms out to follow, so
// a word that ends seconds too early doesn't need a dozen drag-release cycles.
export const AUTO_EXTEND_EDGE_PX = 24
// Growth rate (seconds of word per second held), ramping from the inner edge of the zone to
// the timeline edge so the reviewer can go slow near the target and fast when it's far away.
export const AUTO_EXTEND_MIN_RATE = 1
export const AUTO_EXTEND_MAX_RATE = 4
export default function TimelineEditor({
words,
contextWords,
@@ -64,7 +78,10 @@ export default function TimelineEditor({
onWordClick,
onWordDelete,
onWordCtrlDelete,
prevBoundaryTime,
nextBoundaryTime,
}: TimelineEditorProps) {
const { audioData } = useContext(VocalsAudioDataLoaderContext)
const containerRef = useRef<HTMLDivElement>(null)
// Whether the current press has moved past the drag threshold. Distinguishes a click
// (open the modal) from a drag (move/resize the word timing).
@@ -75,7 +92,15 @@ export default function TimelineEditor({
initialX: number
initialTime: number
word: Word
// Resize only: time offset between the grabbed pointer position and the edge being
// resized, so the edge tracks the pointer absolutely (stays correct as the view zooms).
grabOffset: number
} | null>(null)
// View domain widened while auto-extending a resize past the edge; cleared on release (the
// committed segment bounds then cover the new timing).
const [dragView, setDragView] = useState<{ start: number; end: number } | null>(null)
// Latest pointer x (container-relative) during a drag, read by the auto-extend loop.
const pointerXRef = useRef(0)
const MIN_DURATION = 0.1 // Minimum word duration in seconds
@@ -83,39 +108,13 @@ export default function TimelineEditor({
// dragged/synced earlier and the last word later, and so the surrounding waveform + any word
// blocks that spill just outside the segment stay visible. The whole timeline maps this padded
// "view domain" to 0–100%; the segment itself is the un-shaded band in the middle.
const viewStart = Math.max(0, startTime - TIMELINE_PAD_SECONDS)
const viewEnd = endTime + TIMELINE_PAD_SECONDS
const viewStart = Math.min(Math.max(0, startTime - TIMELINE_PAD_SECONDS), dragView?.start ?? Infinity)
const viewEnd = Math.max(endTime + TIMELINE_PAD_SECONDS, dragView?.end ?? -Infinity)
const viewDuration = viewEnd - viewStart
const checkCollision = (
proposedStart: number,
proposedEnd: number,
currentIndex: number,
isResize: boolean
): boolean => {
if (isResize) {
// Compare against the nearest *timed* neighbours, not just array neighbours:
// unsynchronized words (null timestamps) aren't drawn on the timeline, so an
// adjacent null-timed word must be skipped to reach the real neighbour bar.
const nextWord = words
.slice(currentIndex + 1)
.find((word) => word.start_time !== null && word.end_time !== null)
if (nextWord && nextWord.start_time !== null && proposedEnd > nextWord.start_time) {
return true
}
const previousWord = words
.slice(0, currentIndex)
.reverse()
.find((word) => word.start_time !== null && word.end_time !== null)
if (previousWord && previousWord.end_time !== null && proposedStart < previousWord.end_time) {
return true
}
return false
}
return words.some((word, index) => {
// Overlap test for moving a whole word (resizes clamp against resizeBounds instead).
const checkCollision = (proposedStart: number, proposedEnd: number, currentIndex: number): boolean =>
words.some((word, index) => {
if (index === currentIndex) return false
if (word.start_time === null || word.end_time === null) return false
@@ -125,6 +124,25 @@ export default function TimelineEditor({
(proposedStart <= word.start_time && proposedEnd >= word.end_time)
)
})
// How far a word's edges may be resized: up to the nearest *timed* neighbour in this segment
// (unsynchronized words aren't drawn, so they're skipped), else the neighbouring segment's
// nearest word, else the audio bounds. Neighbouring-segment limits never pull an edge back
// from where the drag started, so pre-existing overlaps don't snap on the first pixel.
const resizeBounds = (index: number, original: Word): { min: number; max: number } => {
const isTimed = (w: Word) => w.start_time !== null && w.end_time !== null
const next = words.slice(index + 1).find(isTimed)
const prev = words.slice(0, index).reverse().find(isTimed)
let max = next?.start_time ?? Infinity
if (!next) {
if (nextBoundaryTime != null) max = Math.max(nextBoundaryTime, original.end_time ?? -Infinity)
else if (audioData?.duration) max = Math.max(audioData.duration, original.end_time ?? -Infinity)
}
let min = prev?.end_time ?? 0
if (!prev && prevBoundaryTime != null) {
min = Math.min(prevBoundaryTime, original.start_time ?? Infinity)
}
return { min, max }
}
const timeToPosition = (time: number): number => {
@@ -182,25 +200,39 @@ export default function TimelineEditor({
return
}
// The drag is tracked on the window (it may leave the row), so stop the browser from
// starting a text selection across the page.
e.preventDefault()
const initialX = e.clientX - rect.left
const initialTime = (initialX / rect.width) * viewDuration
const pointerTime = viewStart + initialTime
const grabOffset =
type === 'resize-right'
? word.end_time - pointerTime
: type === 'resize-left'
? word.start_time - pointerTime
: 0
hasDraggedRef.current = false
pointerXRef.current = initialX
setDragState({
wordIndex,
type,
initialX,
initialTime,
word,
grabOffset,
})
}
const handleMouseMove = (e: React.MouseEvent) => {
const handleMouseMove = (clientX: number) => {
if (!dragState || !containerRef.current) return
const rect = containerRef.current.getBoundingClientRect()
const x = e.clientX - rect.left
const x = clientX - rect.left
const width = rect.width
pointerXRef.current = x
// Ignore sub-threshold jitter so a click (which opens the modal) doesn't nudge the
// timing; once the threshold is crossed the press is a drag for the rest of its life.
@@ -218,42 +250,28 @@ export default function TimelineEditor({
)
return
if (dragState.type === 'resize-right') {
const initialWordDuration = dragState.word.end_time - dragState.word.start_time
const initialWordWidth = (initialWordDuration / viewDuration) * width
const pixelDelta = x - dragState.initialX
const percentageMoved = pixelDelta / initialWordWidth
const timeDelta = initialWordDuration * percentageMoved
if (dragState.type === 'resize-right' || dragState.type === 'resize-left') {
// Absolute mapping: the edge sits under the pointer (plus the grab offset) in the
// *current* view, so it stays under the cursor even after auto-extend zoomed out.
const clampedX = Math.max(0, Math.min(width, x))
const pointerTime = viewStart + (clampedX / width) * viewDuration + dragState.grabOffset
const { min, max } = resizeBounds(dragState.wordIndex, dragState.word)
const proposedEnd = Math.max(
currentWord.start_time + MIN_DURATION,
dragState.word.end_time + timeDelta
)
if (checkCollision(currentWord.start_time, proposedEnd, dragState.wordIndex, true)) return
onWordUpdate(dragState.wordIndex, {
start_time: currentWord.start_time,
end_time: proposedEnd,
})
} else if (dragState.type === 'resize-left') {
const initialWordDuration = dragState.word.end_time - dragState.word.start_time
const initialWordWidth = (initialWordDuration / viewDuration) * width
const pixelDelta = x - dragState.initialX
const percentageMoved = pixelDelta / initialWordWidth
const timeDelta = initialWordDuration * percentageMoved
const proposedStart = Math.min(
currentWord.end_time - MIN_DURATION,
dragState.word.start_time + timeDelta
)
if (checkCollision(proposedStart, currentWord.end_time, dragState.wordIndex, true)) return
onWordUpdate(dragState.wordIndex, {
start_time: proposedStart,
end_time: currentWord.end_time,
})
if (dragState.type === 'resize-right') {
const proposedEnd = Math.min(max, Math.max(currentWord.start_time + MIN_DURATION, pointerTime))
if (proposedEnd === currentWord.end_time) return
onWordUpdate(dragState.wordIndex, {
start_time: currentWord.start_time,
end_time: proposedEnd,
})
} else {
const proposedStart = Math.max(min, Math.min(currentWord.end_time - MIN_DURATION, pointerTime))
if (proposedStart === currentWord.start_time) return
onWordUpdate(dragState.wordIndex, {
start_time: proposedStart,
end_time: currentWord.end_time,
})
}
} else if (dragState.type === 'move') {
const pixelsPerSecond = width / viewDuration
const pixelDelta = x - dragState.initialX
@@ -266,7 +284,7 @@ export default function TimelineEditor({
// Allow dragging a little outside the segment (into the padded view) so the first/last
// word can extend the segment; updateSegment recomputes the segment bounds from the words.
if (proposedStart < viewStart || proposedEnd > viewEnd) return
if (checkCollision(proposedStart, proposedEnd, dragState.wordIndex, false)) return
if (checkCollision(proposedStart, proposedEnd, dragState.wordIndex)) return
onWordUpdate(dragState.wordIndex, {
start_time: proposedStart,
@@ -275,21 +293,102 @@ export default function TimelineEditor({
}
}
const handleMouseUp = (fromLeave = false) => {
const handleMouseUp = () => {
if (dragState) {
if (hasDraggedRef.current) {
// A real drag: persist the new timing.
onCommit?.()
} else if (!fromLeave) {
// A click (no drag): open the Edit Segment modal. Skipped when the pointer merely
// left the row, so leaving without releasing doesn't spuriously open the modal.
} else {
// A click (no drag): open the Edit Segment modal.
onWordClick?.(dragState.wordIndex)
}
}
hasDraggedRef.current = false
setDragState(null)
setDragView(null)
}
// One step of edge auto-extend (called every animation frame during a resize drag): if the
// pointer is in the edge zone the handle is heading toward, grow the word at a rate scaled by
// how deep into the zone it is, and widen the view so the edge stays glued under the cursor.
// Stops by itself at the neighbouring word / audio bound; leaving the zone or releasing ends it.
const autoExtendStep = (dtSeconds: number) => {
if (!dragState || !hasDraggedRef.current || dragState.type === 'move') return
const width = containerRef.current?.getBoundingClientRect().width ?? 0
if (width <= 0) return
const word = words[dragState.wordIndex]
if (word.start_time === null || word.end_time === null) return
const x = pointerXRef.current
const isRight = dragState.type === 'resize-right'
const depth = isRight
? (x - (width - AUTO_EXTEND_EDGE_PX)) / AUTO_EXTEND_EDGE_PX
: (AUTO_EXTEND_EDGE_PX - x) / AUTO_EXTEND_EDGE_PX
if (depth <= 0) return
const rate =
AUTO_EXTEND_MIN_RATE + (AUTO_EXTEND_MAX_RATE - AUTO_EXTEND_MIN_RATE) * Math.min(1, depth)
const step = rate * dtSeconds
const { min, max } = resizeBounds(dragState.wordIndex, dragState.word)
const fraction = Math.max(0, Math.min(width, x)) / width
if (isRight) {
const newEnd = Math.min(max, word.end_time + step)
if (newEnd <= word.end_time) return
onWordUpdate(dragState.wordIndex, { start_time: word.start_time, end_time: newEnd })
// Solve viewStart + fraction * (viewEnd - viewStart) + grabOffset = newEnd for viewEnd.
if (fraction > 0) {
const target = newEnd - dragState.grabOffset
const end = viewStart + (target - viewStart) / fraction
if (end > viewEnd) setDragView({ start: viewStart, end })
}
} else {
const newStart = Math.max(min, word.start_time - step)
if (newStart >= word.start_time) return
onWordUpdate(dragState.wordIndex, { start_time: newStart, end_time: word.end_time })
if (fraction < 1) {
const target = newStart - dragState.grabOffset
const start = Math.max(0, (target - fraction * viewEnd) / (1 - fraction))
if (start < viewStart) setDragView({ start, end: viewEnd })
}
}
}
// The window listeners + rAF loop outlive individual renders, so they call through a ref to
// the latest handlers (which close over the current words/view).
const handlersRef = useRef({ handleMouseMove, handleMouseUp, autoExtendStep })
handlersRef.current = { handleMouseMove, handleMouseUp, autoExtendStep }
// Track the pointer on the window while dragging, so pushing past the row edge (where
// auto-extend is fastest) or releasing outside it doesn't abandon the drag.
const dragging = dragState !== null
const resizing = dragState !== null && dragState.type !== 'move'
useEffect(() => {
if (!dragging) return
const onMove = (e: MouseEvent) => handlersRef.current.handleMouseMove(e.clientX)
const onUp = () => handlersRef.current.handleMouseUp()
window.addEventListener('mousemove', onMove)
window.addEventListener('mouseup', onUp)
return () => {
window.removeEventListener('mousemove', onMove)
window.removeEventListener('mouseup', onUp)
}
}, [dragging])
useEffect(() => {
if (!resizing) return
let frame = 0
let last: number | null = null
const tick = (now: number) => {
// Cap dt so a backgrounded tab doesn't leap the word forward on return.
const dt = last === null ? 0 : Math.min(0.05, (now - last) / 1000)
last = now
if (dt > 0) handlersRef.current.autoExtendStep(dt)
frame = requestAnimationFrame(tick)
}
frame = requestAnimationFrame(tick)
return () => cancelAnimationFrame(frame)
}, [resizing])
const handleContextMenu = (e: React.MouseEvent, wordIndex: number) => {
e.preventDefault()
e.stopPropagation()
@@ -336,9 +435,6 @@ export default function TimelineEditor({
<div
ref={containerRef}
className={cn(compact ? 'relative' : 'relative bg-card rounded border border-border')}
onMouseMove={handleMouseMove}
onMouseUp={() => handleMouseUp(false)}
onMouseLeave={() => handleMouseUp(true)}
>
{/* Out-of-segment padding: greyed bands on each side of the real segment. These sit above
the waveform but below the word bars and are click-through so playback scrubbing still
@@ -9,7 +9,10 @@ import { Play, Trash2, Type, Clock, AudioWaveform } from 'lucide-react'
import { HighlightedText } from './shared/HighlightedText'
import { TranscriptionViewProps, TranscriptionWordPosition } from '@/lib/lyrics-review/types'
import { deleteSegment } from '@/lib/lyrics-review/utils/segmentOperations'
import { computeContextWordsBySegment } from '@/lib/lyrics-review/utils/contextWords'
import {
computeContextWordsBySegment,
computeNeighbourBoundsBySegment,
} from '@/lib/lyrics-review/utils/contextWords'
import { buildSegmentDecorations } from '@/lib/lyrics-review/utils/wordDecorations'
import { TIMELINE_PAD_SECONDS } from './TimelineEditor'
import WaveformSegmentRow from './WaveformSegmentRow'
@@ -71,6 +74,12 @@ export default function TranscriptionView({
: null,
[waveformsMode, data.corrected_segments]
)
// Nearest word edges in the neighbouring lines, so a Waveforms-row resize / edge auto-extend
// stops at the next line's first word even when it's beyond the drawn context padding.
const neighbourBoundsBySegment = useMemo(
() => (waveformsMode ? computeNeighbourBoundsBySegment(data.corrected_segments) : null),
[waveformsMode, data.corrected_segments]
)
// Per-segment bar colours + AI ghost text for Waveforms mode. Memoised so a playback tick
// (currentTime changing every ~100ms) doesn't reclassify every word each frame.
@@ -183,6 +192,8 @@ export default function TranscriptionView({
segment={segment}
segmentIndex={segmentIndex}
contextWords={contextWordsBySegment?.get(segmentIndex) ?? []}
prevBoundaryTime={neighbourBoundsBySegment?.get(segmentIndex)?.prevEnd ?? null}
nextBoundaryTime={neighbourBoundsBySegment?.get(segmentIndex)?.nextStart ?? null}
currentTime={currentTime}
wordDecorations={decorationsBySegment?.get(segment.id) ?? new Map()}
onCommit={(idx, updated) => onCommitSegment?.(idx, updated)}
@@ -15,6 +15,9 @@ interface WaveformSegmentRowProps {
segmentIndex: number
/** Neighbouring segments' words within the padded window, greyed/read-only for context. */
contextWords: Word[]
/** Nearest word edges in the previous / next lines — resizes (incl. edge auto-extend) stop there. */
prevBoundaryTime?: number | null
nextBoundaryTime?: number | null
/** Per-word bar colour + AI-correction ghost text, keyed by word id. */
wordDecorations: Map<string, WordDecoration>
currentTime?: number
@@ -43,6 +46,8 @@ export default function WaveformSegmentRow({
segment,
segmentIndex,
contextWords,
prevBoundaryTime,
nextBoundaryTime,
wordDecorations,
currentTime,
onCommit,
@@ -125,6 +130,8 @@ export default function WaveformSegmentRow({
<TimelineEditor
words={words}
contextWords={contextWords}
prevBoundaryTime={prevBoundaryTime}
nextBoundaryTime={nextBoundaryTime}
startTime={startTime}
endTime={endTime}
onWordUpdate={handleWordUpdate}
@@ -1,4 +1,5 @@
import { fireEvent, render, screen } from '@testing-library/react'
import React from 'react'
import { act, fireEvent, render, screen } from '@testing-library/react'
import TimelineEditor from '../TimelineEditor'
import { VocalsAudioDataLoaderContext } from '../VocalsAudioDataLoader'
import { Word } from '@/lib/lyrics-review/types'
@@ -76,3 +77,128 @@ describe('TimelineEditor Ctrl/Cmd-click delete', () => {
expect(onWordCtrlDelete).toHaveBeenCalledWith(0)
})
})
describe('TimelineEditor resize edge auto-extend', () => {
// Stateful host so the editor re-renders with each update, like WaveformSegmentRow.
function Host(props: {
initial: Word[]
onUpdate: (words: Word[]) => void
onCommit?: () => void
nextBoundaryTime?: number | null
prevBoundaryTime?: number | null
}) {
const [ws, setWs] = React.useState(props.initial)
return (
<TimelineEditor
words={ws}
startTime={11}
endTime={13}
onWordUpdate={(i, u) => {
setWs((prev) => {
const next = prev.map((w, j) => (j === i ? { ...w, ...u } : w))
props.onUpdate(next)
return next
})
}}
onCommit={props.onCommit}
nextBoundaryTime={props.nextBoundaryTime}
prevBoundaryTime={props.prevBoundaryTime}
/>
)
}
let latest: Word[]
function setup(extra: Partial<React.ComponentProps<typeof Host>> = {}) {
latest = words
const onCommit = jest.fn()
const utils = render(
<VocalsAudioDataLoaderContext.Provider
value={{ audioData: { amplitudes: [0.1], duration: 60 } as never }}
>
<Host initial={words} onUpdate={(w) => (latest = w)} onCommit={onCommit} {...extra} />
</VocalsAudioDataLoaderContext.Provider>
)
const container = utils.container.firstElementChild as HTMLElement
container.getBoundingClientRect = () =>
({ left: 0, top: 0, width: 400, height: 100, right: 400, bottom: 100 }) as DOMRect
return { onCommit }
}
const rightHandle = (text: string) => screen.getByText(text).nextElementSibling as HTMLElement
const leftHandle = (text: string) => screen.getByText(text).previousElementSibling as HTMLElement
// Advance fake time in animation-frame-sized steps so the rAF loop runs.
const hold = (ms: number) => {
for (let t = 0; t < ms; t += 16) act(() => jest.advanceTimersByTime(16))
}
beforeEach(() => jest.useFakeTimers())
afterEach(() => jest.useRealTimers())
it('keeps extending the last word while the end handle is held at the right edge', () => {
setup()
// 'you' ends at 12.0s → x=200 in the 10–14s view.
fireEvent.mouseDown(rightHandle('you'), { clientX: 200 })
fireEvent.mouseMove(window, { clientX: 399 })
const afterMove = latest[1].end_time!
expect(afterMove).toBeCloseTo(13.99, 1)
hold(1000)
// ~4 s/s at the very edge, so a second's hold adds several seconds past the view.
expect(latest[1].end_time!).toBeGreaterThan(afterMove + 2.5)
})
it('stops at the next line\'s first word', () => {
setup({ nextBoundaryTime: 15 })
fireEvent.mouseDown(rightHandle('you'), { clientX: 200 })
fireEvent.mouseMove(window, { clientX: 399 })
hold(3000)
expect(latest[1].end_time).toBe(15)
})
it('stops at the next timed word in the same segment', () => {
setup()
fireEvent.mouseDown(rightHandle('wish'), { clientX: 150 })
fireEvent.mouseMove(window, { clientX: 399 })
hold(2000)
expect(latest[0].end_time).toBe(11.6)
})
it('stops growing once the pointer leaves the edge zone, then tracks the pointer in the zoomed view', () => {
setup()
fireEvent.mouseDown(rightHandle('you'), { clientX: 200 })
fireEvent.mouseMove(window, { clientX: 399 })
hold(500)
fireEvent.mouseMove(window, { clientX: 300 })
const held = latest[1].end_time!
// Pulled back inside the (now wider) view: shorter than at the edge, longer than the start.
expect(held).toBeGreaterThan(12)
hold(500)
expect(latest[1].end_time).toBe(held)
})
it('does not auto-extend before the press crosses the drag threshold', () => {
setup()
fireEvent.mouseDown(rightHandle('you'), { clientX: 398 })
hold(500)
expect(latest[1].end_time).toBe(12.0)
})
it('extends the start handle toward the left edge, bounded by the previous line', () => {
setup({ prevBoundaryTime: 9 })
// 'wish' starts at 11.0s → x=100.
fireEvent.mouseDown(leftHandle('wish'), { clientX: 100 })
fireEvent.mouseMove(window, { clientX: 1 })
hold(3000)
expect(latest[0].start_time).toBe(9)
})
it('commits once on release, even when released outside the row', () => {
const { onCommit } = setup()
fireEvent.mouseDown(rightHandle('you'), { clientX: 200 })
fireEvent.mouseMove(window, { clientX: 450 })
hold(200)
fireEvent.mouseUp(window)
expect(onCommit).toHaveBeenCalledTimes(1)
const final = latest[1].end_time
hold(500)
expect(latest[1].end_time).toBe(final)
})
})
@@ -29,6 +29,9 @@ interface EditModalProps {
currentTime?: number
/** Words from neighbouring segments near this one, shown as greyed read-only context in the timeline padding. */
contextWords?: Word[]
/** Nearest word edges in the previous / next lines — timeline resizes stop there. */
prevBoundaryTime?: number | null
nextBoundaryTime?: number | null
onDelete?: (segmentIndex: number) => void
onAddSegment?: (segmentIndex: number) => void
onSplitSegment?: (segmentIndex: number, afterWordIndex: number) => void
@@ -49,6 +52,8 @@ export default function EditModal({
onPlaySegment,
currentTime = 0,
contextWords,
prevBoundaryTime,
nextBoundaryTime,
onDelete,
onAddSegment,
onSplitSegment,
@@ -479,6 +484,8 @@ export default function EditModal({
<EditTimelineSection
words={editedSegment.words}
contextWords={contextWords}
prevBoundaryTime={prevBoundaryTime}
nextBoundaryTime={nextBoundaryTime}
startTime={startTime}
endTime={endTime}
currentTime={currentTime}
@@ -1,4 +1,4 @@
import { computeContextWordsBySegment } from '../contextWords'
import { computeContextWordsBySegment, computeNeighbourBoundsBySegment } from '../contextWords'
import { LyricsSegment, Word } from '../../types'
const word = (id: string, text: string, start: number | null, end: number | null): Word => ({
@@ -73,3 +73,29 @@ describe('computeContextWordsBySegment', () => {
expect(forS0).not.toContain('b')
})
})
describe('computeNeighbourBoundsBySegment', () => {
it('returns the previous line end and next line start per segment', () => {
const segments = [
segment('s0', [word('a', 'let', 1, 2), word('b', 'go', 2, 3)]),
segment('s1', [word('c', 'never', 9, 10)]),
segment('s2', [word('d', 'my', 20, 21)]),
]
const bounds = computeNeighbourBoundsBySegment(segments)
expect(bounds.get(0)).toEqual({ prevEnd: null, nextStart: 9 })
expect(bounds.get(1)).toEqual({ prevEnd: 3, nextStart: 20 })
expect(bounds.get(2)).toEqual({ prevEnd: 10, nextStart: null })
})
it('skips untimed words and untimed segments', () => {
const segments = [
segment('s0', [word('a', 'one', 1, 2), word('b', 'two', null, null)]),
segment('s1', [word('c', 'gap', null, null)]),
segment('s2', [word('d', 'three', null, null), word('e', 'four', 8, 9)]),
]
const bounds = computeNeighbourBoundsBySegment(segments)
expect(bounds.get(0)).toEqual({ prevEnd: null, nextStart: 8 })
expect(bounds.get(1)).toEqual({ prevEnd: 2, nextStart: 8 })
expect(bounds.get(2)).toEqual({ prevEnd: 2, nextStart: null })
})
})
@@ -40,3 +40,36 @@ export function computeContextWordsBySegment(
return result
}
/**
* For each segment, the nearest timed word edges in *other* segments: the latest `end_time`
* among earlier segments (`prevEnd`) and the earliest `start_time` among later segments
* (`nextStart`). The Waveforms inline timeline clamps word resizes (incl. edge auto-extend) to
* these so a word can't be stretched into the neighbouring line — even when that line's words
* are too far away to be drawn as context. `null` when there is no such neighbour.
*/
export function computeNeighbourBoundsBySegment(
segments: LyricsSegment[]
): Map<number, { prevEnd: number | null; nextStart: number | null }> {
const result = new Map<number, { prevEnd: number | null; nextStart: number | null }>()
const timed = (seg: LyricsSegment) =>
seg.words.filter((w) => w.start_time !== null && w.end_time !== null)
let prevEnd: number | null = null
for (let i = 0; i < segments.length; i++) {
result.set(i, { prevEnd, nextStart: null })
for (const w of timed(segments[i])) {
if (prevEnd === null || w.end_time! > prevEnd) prevEnd = w.end_time!
}
}
let nextStart: number | null = null
for (let i = segments.length - 1; i >= 0; i--) {
result.get(i)!.nextStart = nextStart
for (const w of timed(segments[i])) {
if (nextStart === null || w.start_time! < nextStart) nextStart = w.start_time!
}
}
return result
}
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.251.1"
version = "0.252.0"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"