feat: guitar chords in karaoke videos + ChordMini ensemble chord detection (v0.277.0) (#1175)

## Summary
Adds optional guitar chords to karaoke videos, first used for a Fiverr
medley customer who wants chords on their video.

- **Rendering (gen library):**
`OutputGenerator.generate_outputs(chords=[ChordChange(start, end,
label)])`.
- Each chord change during a sung line is drawn above the word it lands
on, in a row reserved above every line (the same slot mechanism as
translated lyrics).
- Chords light up when played, using the same `\k` highlight as lyrics:
gold while upcoming (theme `chord_color`), sung-lyric colour once
played. Quick changes on one word never overlap.
- A top-right "now / next" chord box runs for the whole song, covering
intros and instrumental breaks.
- With no chords supplied, output is unchanged. A test asserts the ASS
is byte-identical.
- **Detection (`scripts/medley/detect_chords.py`):**
- Ensembles both ChordMini checkpoints (BTC + 2E1D, MIT; about 80%
majmin/MIREX, the best open weights we found) over the full mix, gen's
clean instrumental, and an htdemucs_6s harmonic mix (guitar + piano +
bass + other).
- Weighted per-frame vote, then merges short blips, snaps changes to
beats and spells chords for the key (Ab/Db/Eb7, F#m, borrowed
bIII/bVI/bVII as flats).
- **Medley script:** new `chords` step plus a `"chords": true` spec
option. Chords are clipped per section and hand over mid-crossfade, then
shifted for the countdown.
- **Docs:** `docs/MEDLEYS.md` § Guitar chords (setup, method, accuracy
caveats, theme keys) and a docs/README.md entry.

There's no job or UI option yet. The medley pipeline uses this for now;
a product version would need a chords toggle plus detection in a worker.

## Testing
- New: `test_guitar_chords.py` (12: chord timeline helpers, word
placement measured against the font, layout geometry, highlight timing,
now/next box, theme colour, no-chords path byte-identical),
`test_detect_chords.py` (7), and 2 medley chord-shifting tests.
- `tests/unit`: 2087 passed, 65 skipped.
- End to end on the 5 real medley songs: models/inputs agree on 91–99%
of major/minor labels, and the output matches the expected I–IV–V
progressions. Rendered 1080p and 4K medley videos with 146 chord
changes; spot-checked frames (verse, instrumental screen, chorus).
- Local CodeRabbit: 1 finding (doc section reference), fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 5.5 authored and GitHub committed 2026-10-17 00:24:11 -04:00
1 parent 8ec789a9aa
commit 7cf5beede8
12 files changed
+1022 -12

No files matched your search

+109
View File
@@ -134,6 +134,115 @@ Give the customer the audio file they asked for (usually `medley_instrumental_wi
or `.mp3`; offer `instrumental_clean` as an alternative) plus the karaoke video. The per-song
gen jobs are a free extra: four full karaoke tracks.
### Guitar chords (optional)
The video can show guitar chords. Each chord name sits above the word where the chord
changes and lights up when played, and a "now / next" box in the top-right corner shows
the current chord all the time, including intros and instrumental breaks. First delivered
in October 2026 for the country medley (146 chord changes over 8 minutes). Detection and
rendering together took about 15 minutes of machine time once set up.
#### Checklist (from zero)
```bash
# 1. One-time: ChordMini in its own venv (pinned: the commit we evaluated)
git clone https://github.com/ptnghia-j/ChordMini ~/ChordMini
git -C ~/ChordMini checkout aa6e3a8
cd ~/ChordMini && uv venv -p 3.12 .venv && uv pip install -p .venv/bin/python -r requirements.txt
# 2. Per medley (karaoke-gen checkout, env where karaoke_gen imports + audio-separator on PATH)
export PYTHONPATH=$PWD MEDLEY_CHORDMINI=~/ChordMini
export MEDLEY_GCLOUD_ACCOUNT=claude-readonly@nomadkaraoke.iam.gserviceaccount.com
# add "chords": true to the spec, then:
python scripts/medley/make_medley.py fetch spec.json # if not already fetched
python scripts/medley/make_medley.py chords spec.json # ~1-2 min/song: stems + 6 model runs
python scripts/medley/make_medley.py audio spec.json # if not already built
python scripts/medley/make_medley.py video spec.json --resolution 1080p # check frames first
python scripts/medley/make_medley.py video spec.json # 4K final
```
- **Check it before delivering.** Watch the `With Vocals` video for a verse or two of each
song (chords should change with the band), and grab a few frames to check layout: one in a
verse, one on an `INSTRUMENTAL` screen (the corner box should show chords there), one in a
chorus.
- **Fixing a chord.** Edit `work/<job_id>/chords.json` (list of
`{"start", "end", "label"}` in that song's own source seconds) and re-run `video`. If a
customer sends their own chord chart, enter it the same way: their chords replace the
detected ones, and nothing else changes.
- **Deliver both versions** (with and without chords), and say the chords are auto-detected.
#### A single song, not a medley
The same pipeline adds chords to any one gen track: write a one-section spec
(`{"job_id": "...", "start": 0, "end": null}`) and run `fetch`, `chords`, `audio`, `video`.
The output is the job's normal karaoke video plus chords (re-rendered locally from its
reviewed lyrics and theme).
#### How detection works (`scripts/medley/detect_chords.py`)
We compared the open chord-recognition models in October 2026.
[ChordMini](https://github.com/ptnghia-j/ChordMini) (MIT) had the best published accuracy
with downloadable weights: its continual-learning BTC checkpoint scores about 80%
majmin/MIREX on a 120-song test set (arXiv 2602.19778, Table 3). ChordFormer
(arXiv 2502.11840) reports similar numbers but has no released weights; autochord and
Chordino are older and weaker (25-class / ~67%). The script:
1. Runs **both** ChordMini checkpoints (BTC weight 1.1, 2E1D weight 1.0) on **three
inputs**: the full mix, gen's clean instrumental (vocals removed, weight × 1.2) and a
"harmonic" mix of htdemucs_6s guitar + piano + bass + other (drums removed too; cached
as `work/<job>/harmonic.flac`). Removing vocals before chord recognition is a known
accuracy win (APSIPA 2025).
2. Simplifies every label to a guitar chord (major, m, 7, maj7, m7, sus2/4, dim, aug,
m7b5, dim7; no inversions), then takes a weighted vote on a 50ms frame grid.
3. Merges changes shorter than 0.6s into their longer neighbour, and snaps the remaining
changes onto librosa beats when one is within 0.2s.
4. Spells roots for the local key, estimated over a 20s window: A♭ / D♭ / E♭7 in A-flat,
F♯m in D, and borrowed ♭III / ♭VI / ♭VII stay flat in sharp keys.
**Accuracy evidence.** Run on the five country songs, each pair of model/input runs agreed
on 91–99% of major/minor labels (93–99% for the instrumental-vs-harmonic pairs), and the
output was the expected progressions (G–C–D7 then A♭–D♭–E♭7 after the half-step key
change; F–B♭–C with a II chord; D–F♯m7–G–A). That's a consistency check, not ground truth.
**Re-evaluating (if a better model appears).** Run each candidate on the same inputs, write
`.lab` files and compare pairwise with `mir_eval.chord` (`majmin` and `sevenths` weighted
accuracy). Better still, compare against a hand-checked chart for one song. Add a model by
extending `MODELS` in the script.
**Safety.** Load only `btc_model_best.pth` and `2e1d_model_best.pth`. Both load with
PyTorch's safe `weights_only=True`; the third bundled checkpoint
(`btc_model_large_voca.pt`) needs unsafe unpickling and isn't used.
#### How rendering works (in gen)
`karaoke_gen/lyrics_transcriber/output/ass/chords.py`, used via
`OutputGenerator.generate_outputs(chords=[ChordChange(start, end, label)])`. With no chords,
output is byte-identical to before.
- **Layout.** Each lyric slot gets a chord row above the line: the slot grows by 1.05 ×
the chord font size, and lines per screen are capped at 4. It's the same mechanism as
translated lyrics (which add a row below).
- **Which line.** A change belongs to the line being sung, from 0.6s before its first
word to the end of its last word. The chord already ringing when a line starts is
repeated over its first word, so every line shows what to play. Changes between lines
appear only in the corner box.
- **Which word.** A change sits over the word sung at that moment, or the next word if it
comes up to 0.25s early (a pickup strum). Quick changes on one word are pushed right so
they never overlap. Positions use the renderer's libass-matched text measurement.
- **Highlight.** Each chord is a one-syllable karaoke event (`\k` wait, then `\kf`):
upcoming colour, then the sung-lyric colour once played.
- **Medley handover.** In the medley script, each section's chords are clipped to the
section, handing over to the next song halfway through each crossfade, then shifted for
the 3s countdown.
- **Theme keys** (optional, `karaoke` block):
- `chord_color`: upcoming chord colour, default gold `255, 223, 107`.
- `chord_font_size`, or `chord_font_ratio` (default 0.55 × lyric size).
- `chord_max_visible_lines` (default 4).
There's no gen job/UI option for chords yet: it's agent-run through this script. A product
version would need a job flag, detection in a worker (ChordMini is small and CPU-friendly),
and a chord editor in review.
## Making it sound good
- **Cut where the next vocal starts, not where the last one ends.** A section should run until
+1
View File
@@ -60,6 +60,7 @@
## Recent Changes
- **Guitar chords in karaoke videos** (2026-10-16, v0.277.0). `OutputGenerator.generate_outputs(chords=[ChordChange(...)])` draws each chord above the word it lands on (it lights up when played) in a row reserved above every line, plus a "now / next" chord box in the top-right corner that also covers intros and instrumental breaks. Chords come from `scripts/medley/detect_chords.py`, an ensemble of ChordMini models over the mix, instrumental and harmonic stems. It's used by the medley script for now (`"chords": true`); there's no job/UI option yet. Theme keys: `chord_color`, `chord_font_size`/`chord_font_ratio`, `chord_max_visible_lines`. See [MEDLEYS.md § Guitar chords](MEDLEYS.md#guitar-chords-optional).
- **Local E2E stack + email-code sign-in** (2026-10-15, v0.275.0). Plan: [archive/2026-10-15-local-stack-e2e-plan.md](archive/2026-10-15-local-stack-e2e-plan.md).
- **Local stack.** `scripts/local-stack.sh up --frontend` runs the real API on the Firestore + GCS emulators with `FAKE_WORKERS=1`. In-process fake workers (`backend/services/fake_workers/`) move jobs through their real states in seconds with small real media; the search, download, Stripe Checkout, email, Secret Manager and encoding-VM seams are faked or skipped. The `/api/test/*` router (only mounted in that mode; startup refuses it in production) seeds scenarios, tunes the fakes per user, exposes captured email and serves a fake checkout page.
- **New suite.** `frontend/e2e/local-stack` covers sign-up, make, review, instrumental, download, buying credits, the duration-confirm and failure states, and code sign-in through the real UI and API in about 1 minute. It runs in CI as the required `local-stack-e2e` job. Mobile Maestro flows will use the same stack. See [TESTING.md § Local stack](TESTING.md#local-stack-real-backend-fake-workers).
@@ -0,0 +1,118 @@
"""Guitar chord symbols for karaoke videos.
Chords are a timed sequence of ``ChordChange`` (start, end, label) for the whole song.
Each change that happens while a lyric line is being sung is drawn above the word it
lands on, in a row reserved above the line (see ``chord_layout``), and switches colour
when the chord is played. A "now / next" box in the top-right corner shows the current
chord at all times, which covers intros and instrumental breaks where there's no lyric
line to sit a chord above.
"""
from dataclasses import dataclass
from typing import Callable, Dict, List, Optional, Sequence
from karaoke_gen.lyrics_transcriber.types import LyricsSegment
# A chord that changes just before a word (strummed on the pickup) belongs to that word
CHORD_WORD_SNAP_SECONDS = 0.25
# A chord shortly before a line's first word is shown over that first word
CHORD_LINE_LEAD_SECONDS = 0.6
NO_CHORD_LABELS = frozenset({"", "N", "X", "N.C.", "NC"})
@dataclass(frozen=True)
class ChordChange:
start: float
end: float
label: str
@dataclass(frozen=True)
class PlacedChord:
chord: ChordChange
word_index: int
x: int # left edge of the chord label, in pixels
def clean_chords(chords: Sequence[ChordChange]) -> List[ChordChange]:
"""Sorted, without no-chord markers, with repeated labels merged."""
out: List[ChordChange] = []
for c in sorted(chords, key=lambda c: c.start):
if c.label.strip() in NO_CHORD_LABELS or c.end <= c.start:
continue
if out and out[-1].label == c.label and c.start - out[-1].end < 0.05:
out[-1] = ChordChange(out[-1].start, c.end, c.label)
else:
out.append(ChordChange(c.start, c.end, c.label.strip()))
return out
def assign_chords_to_segments(
segments: Sequence[LyricsSegment], chords: Sequence[ChordChange]
) -> Dict[str, List[ChordChange]]:
"""Map segment id -> chord changes to draw above that line.
A change belongs to the line being sung when it happens (from slightly before the
first word to the last word's end). Changes between lines go only in the corner box.
The chord already ringing when a line starts is repeated over its first word, so a
guitarist reading the line always sees what to play.
"""
result: Dict[str, List[ChordChange]] = {}
chords = list(chords)
for seg in segments:
if not seg.words:
continue
line_start, line_end = seg.words[0].start_time, seg.words[-1].end_time
on_line = [c for c in chords if line_start - CHORD_LINE_LEAD_SECONDS <= c.start <= line_end]
ringing = [c for c in chords if c.start < line_start - CHORD_LINE_LEAD_SECONDS and c.end > line_start]
if ringing and (not on_line or on_line[0].start > line_start + CHORD_WORD_SNAP_SECONDS):
on_line.insert(0, ringing[-1])
if on_line:
result[seg.id] = on_line
return result
def word_index_for_time(segment: LyricsSegment, t: float) -> int:
"""Index of the word a chord change at time ``t`` sits above."""
idx = 0
for i, w in enumerate(segment.words):
if w.start_time <= t + CHORD_WORD_SNAP_SECONDS:
idx = i
else:
break
return idx
def place_chords(
segment: LyricsSegment,
chords: Sequence[ChordChange],
word_texts: Sequence[str],
line_left: int,
measure: Callable[[str], int],
chord_measure: Callable[[str], int],
min_spacing: int,
) -> List[PlacedChord]:
"""X positions for chords over a line whose rendered text starts at ``line_left``.
``word_texts`` are the words exactly as rendered (case-transformed) and joined by
single spaces. Two chords over the same word (a quick change) are pushed right so
they never overlap.
"""
placed: List[PlacedChord] = []
next_free = None
for chord in chords:
i = word_index_for_time(segment, chord.start)
prefix = " ".join(word_texts[:i]) + (" " if i else "")
x = line_left + (measure(prefix) if prefix else 0)
if next_free is not None and x < next_free:
x = next_free
placed.append(PlacedChord(chord=chord, word_index=i, x=x))
next_free = x + chord_measure(chord.label) + min_spacing
return placed
def chord_at(chords: Sequence[ChordChange], t: float) -> Optional[ChordChange]:
for c in chords:
if c.start <= t < c.end:
return c
return None
@@ -8,6 +8,12 @@ DEFAULT_TRANSLATION_MAX_VISIBLE_LINES = 3 # lyric lines per screen when transla
TRANSLATION_GAP_RATIO = 0.12 # gap between a lyric line and its translation / lyric font size
TRANSLATION_ROW_RATIO = 1.15 # translation row height / translation font size
# Guitar-chord layout defaults (see chord_layout)
DEFAULT_CHORD_FONT_RATIO = 0.55 # chord symbol font size / lyric font size
DEFAULT_CHORD_MAX_VISIBLE_LINES = 4 # lyric lines per screen when showing chords
CHORD_GAP_RATIO = 0.0 # gap between the chord row and its lyric line / lyric font size
CHORD_ROW_RATIO = 1.05 # chord row height / chord font size
class ScreenConfig:
"""Configuration for screen timing and layout.
@@ -97,6 +103,12 @@ class ScreenConfig:
self.translation_max_rows: int = DEFAULT_TRANSLATION_MAX_ROWS
# Height of the lyric line itself within its (taller) slot
self.lyric_line_height: int = line_height
# Guitar-chord layout: set by SubtitlesGenerator when chords are supplied.
# The chord row sits at the top of each slot and the lyric line is pushed
# down by lyric_y_offset. None = no chord row (classic layout).
self.chord_style = None
self.chord_font_size: int = 0
self.lyric_y_offset: int = 0
def get_lead_in_color_ass_format(self) -> str:
"""Convert RGB lead-in color to ASS format.
@@ -213,6 +225,36 @@ def translation_layout(font_size: float, line_height: int, karaoke_styles: Optio
)
@dataclass
class ChordLayout:
"""Geometry of a lyric slot with a chord-symbol row above the line."""
slot_height: int # chord row + gap + lyric line (the new line_height)
lyric_line_height: int
chord_font_size: int
lyric_y_offset: int # chord row + gap: where the lyric line starts within the slot
max_visible_lines: int
def chord_layout(font_size: float, line_height: int, karaoke_styles: Optional[dict] = None) -> ChordLayout:
"""Slot geometry for guitar chords.
Theme keys (all optional, in the ``karaoke`` block): ``chord_font_size`` (absolute)
or ``chord_font_ratio``, ``chord_max_visible_lines``.
"""
k = karaoke_styles or {}
c_size = k.get("chord_font_size") or font_size * float(k.get("chord_font_ratio", DEFAULT_CHORD_FONT_RATIO))
c_size = max(1, int(round(c_size)))
offset = int(round(c_size * CHORD_ROW_RATIO + font_size * CHORD_GAP_RATIO))
return ChordLayout(
slot_height=line_height + offset,
lyric_line_height=line_height,
chord_font_size=c_size,
lyric_y_offset=offset,
max_visible_lines=int(k.get("chord_max_visible_lines", DEFAULT_CHORD_MAX_VISIBLE_LINES)),
)
@dataclass
class LineTimingInfo:
"""Timing information for a single line."""
@@ -1,4 +1,4 @@
from dataclasses import dataclass
from dataclasses import dataclass, field
import functools
from typing import Optional, Tuple, List
import logging
@@ -10,6 +10,7 @@ from karaoke_gen.lyrics_transcriber.types import LyricsSegment
from karaoke_gen.lyrics_transcriber.output.ass.event import Event
from karaoke_gen.lyrics_transcriber.output.ass.style import Style
from karaoke_gen.lyrics_transcriber.output.ass.config import LineState, ScreenConfig
from karaoke_gen.lyrics_transcriber.output.ass.chords import ChordChange, place_chords
from karaoke_gen.lyrics_transcriber.output.ass.text_direction import is_rtl_text, rtl_karaoke_fill_tags
from karaoke_gen.lyrics_transcriber.output.segment_resizer import (
display_width,
@@ -37,6 +38,8 @@ class LyricsLine:
screen_config: ScreenConfig
logger: Optional[logging.Logger] = None
previous_end_time: Optional[float] = None
# Chord changes drawn above this line (only used when the screen config has a chord row)
chords: List[ChordChange] = field(default_factory=list)
def __post_init__(self):
"""Ensure logger is initialized"""
@@ -173,7 +176,7 @@ class LyricsLine:
start_x_position = 0
# Apply vertical offset if configured
vertical_offset = int(self.screen_config.video_height * (config.lead_in_vert_offset_percent / 100))
final_y_position = state.y_position + main_height + vertical_offset
final_y_position = state.y_position + config.lyric_y_offset + main_height + vertical_offset
self.logger.debug(f"Position calculations:")
self.logger.debug(f" Video dimensions: {self.screen_config.video_width}x{self.screen_config.video_height}")
@@ -271,7 +274,7 @@ class LyricsLine:
# Main lyrics text with positioning and fade
text = (
f"{{\\an8}}{{\\pos({x_pos},{state.y_position})}}"
f"{{\\an8}}{{\\pos({x_pos},{state.y_position + config.lyric_y_offset})}}"
f"{{\\fad({config.fade_in_ms},{config.fade_out_ms})}}"
)
rtl_tags = rtl_karaoke_fill_tags(line_style.Angle) if is_rtl_text(self.segment.text) else ""
@@ -291,6 +294,48 @@ class LyricsLine:
if translation_event:
events.append(translation_event)
events.extend(self._create_chord_events(state, line_style, config))
return events
def _create_chord_events(self, state: LineState, line_style: Style, config: ScreenConfig) -> List[Event]:
"""Chord symbols above the words they land on, lighting up when played.
Each chord is one karaoke syllable: upcoming colour until the chord's time,
then the played colour (same \\k mechanism as the lyric highlight).
"""
chord_style = getattr(config, "chord_style", None)
if chord_style is None or not self.chords or not self.segment.words:
return []
word_texts = [self._apply_case_transform(w.text.replace("\n", "").strip()) for w in self.segment.words]
line_text = " ".join(word_texts)
line_width = self._measure_text(line_text, line_style)[0]
line_left = config.video_width // 2 - line_width // 2
placed = place_chords(
self.segment,
self.chords,
word_texts,
line_left,
measure=lambda text: self._measure_text(text, line_style)[0],
chord_measure=lambda text: self._measure_text(text, chord_style)[0],
min_spacing=max(1, chord_style.Fontsize // 3),
)
events = []
start = state.timing.fade_in_time
for p in placed:
wait_cs = max(0, int(round((p.chord.start - start) * 100)))
event = Event()
event.type = "Dialogue"
event.Layer = 1
event.Style = chord_style
event.Start = start
event.End = state.timing.end_time
event.Text = (
f"{{\\an7}}{{\\pos({p.x},{state.y_position})}}"
f"{{\\fad({config.fade_in_ms},{config.fade_out_ms})}}"
f"{{\\k{wait_cs}}}{{\\kf15}}{p.chord.label}"
)
events.append(event)
return events
# Fraction of the frame width a translation may use before it's wrapped / scaled
@@ -361,7 +406,7 @@ class LyricsLine:
if widest > max_width:
scale = max(1, int(max_width / widest * 100))
y = state.y_position + config.lyric_line_height + config.translation_gap
y = state.y_position + config.lyric_y_offset + config.lyric_line_height + config.translation_gap
tags = (
f"{{\\an8}}{{\\pos({config.video_width // 2},{y})}}"
f"{{\\fad({config.fade_in_ms},{config.fade_out_ms})}}"
@@ -155,6 +155,7 @@ class OutputGenerator:
artist: Optional[str] = None,
title: Optional[str] = None,
ass_only: bool = False,
chords: Optional[list] = None,
) -> OutputPaths:
"""Generate all requested output formats.
@@ -165,6 +166,8 @@ class OutputGenerator:
audio_filepath: Path to audio file
artist: Optional artist name
title: Optional title
chords: Optional guitar chord changes (``ChordChange`` list) to draw above the
lyrics in the ASS/video (no effect on other formats)
ass_only: If True (only in preview_mode), generate only ASS subtitles
without video encoding. Useful when video encoding is offloaded
to an external service.
@@ -188,7 +191,7 @@ class OutputGenerator:
# For preview, we only need to generate ASS and video
if self.preview_mode:
# Generate ASS subtitles for preview
outputs.ass = self.subtitle.generate_ass(transcription_corrected.resized_segments, output_prefix, audio_filepath)
outputs.ass = self.subtitle.generate_ass(transcription_corrected.resized_segments, output_prefix, audio_filepath, chords=chords)
# Generate preview video (unless ass_only mode for GCE encoding)
if not ass_only:
@@ -225,7 +228,7 @@ class OutputGenerator:
# Generate video if requested
if self.config.render_video:
# Generate ASS subtitles
outputs.ass = self.subtitle.generate_ass(resized_segments, output_prefix, audio_filepath)
outputs.ass = self.subtitle.generate_ass(resized_segments, output_prefix, audio_filepath, chords=chords)
outputs.video = self.video.generate_video(outputs.ass, audio_filepath, output_prefix)
return outputs
@@ -13,12 +13,27 @@ from karaoke_gen.lyrics_transcriber.output.ass.style import Style
from karaoke_gen.lyrics_transcriber.output.ass.constants import ALIGN_TOP_CENTER
from karaoke_gen.lyrics_transcriber.output.ass import LyricsScreen
from karaoke_gen.lyrics_transcriber.output.ass.section_detector import SectionDetector
from karaoke_gen.lyrics_transcriber.output.ass.config import ScreenConfig, translation_layout
from karaoke_gen.lyrics_transcriber.output.ass.config import ScreenConfig, chord_layout, translation_layout
from karaoke_gen.lyrics_transcriber.output.ass.chords import ChordChange, assign_chords_to_segments, clean_chords
from karaoke_gen.lyrics_transcriber.output.ass.event import Event
# Translation rows render at this fraction of the lyric colour's opacity, so they read as
# secondary to the sung line (Andrew: identical white made it unclear which to read).
TRANSLATION_OPACITY = 0.7
# Guitar chords: upcoming chord colour (warm gold) unless the theme sets chord_color
DEFAULT_CHORD_COLOR = (255, 223, 107, 255)
# "Now / next" chord box in the top-right corner, as fractions of the frame
CHORD_BOX_RIGHT = 0.955
CHORD_BOX_TOP = 0.045
CHORD_BOX_SIZE_RATIO = 1.25 # box chord size / chord-row size
CHORD_BOX_NEXT_RATIO = 0.55 # "next" chord size / current chord size
def _ass_colour(color) -> str:
"""(r, g, b[, a]) -> ASS inline colour &HBBGGRR&."""
r, g, b = color[:3]
return f"&H{b:02X}{g:02X}{r:02X}&"
def _fade(color, opacity: float):
@@ -58,6 +73,7 @@ class SubtitlesGenerator:
self.styles = styles
self.is_duet = is_duet
self.subtitle_offset_ms = subtitle_offset_ms
self.chords: List[ChordChange] = []
# Create ScreenConfig with potential overrides from styles
karaoke_styles = styles.get("karaoke", {})
@@ -143,8 +159,23 @@ class SubtitlesGenerator:
order = [1, 2, 0]
return [sid for sid in order if sid in found]
def generate_ass(self, segments: List[LyricsSegment], output_prefix: str, audio_filepath: Optional[str]) -> str:
def generate_ass(
self,
segments: List[LyricsSegment],
output_prefix: str,
audio_filepath: Optional[str],
chords: Optional[List[ChordChange]] = None,
) -> str:
"""Write the karaoke ASS file.
``chords``: optional guitar chord changes for the whole song. When given, each
line gets a chord row above it and a "now / next" chord box is shown throughout.
"""
self.logger.info("Generating ASS format subtitles")
self.chords = clean_chords(chords or [])
if self.chords and self.subtitle_offset_ms:
off = self.subtitle_offset_ms / 1000.0
self.chords = [ChordChange(c.start + off, c.end + off, c.label) for c in self.chords]
output_path = self._get_output_path(f"{output_prefix} (Karaoke)", "ass")
try:
@@ -152,7 +183,9 @@ class SubtitlesGenerator:
song_duration = self._get_audio_duration(audio_filepath, segments)
base_config = self.config
if any(seg.translation for seg in segments):
if self.chords:
self.config = self._chord_config(base_config)
elif any(seg.translation for seg in segments):
self.config = self._translation_config(base_config)
try:
screens = self._create_screens(segments, song_duration)
@@ -189,6 +222,62 @@ class SubtitlesGenerator:
)
return config
def _chord_config(self, base: ScreenConfig) -> ScreenConfig:
"""Copy of ``base`` with taller line slots that fit a chord row above each line."""
layout = chord_layout(self.font_size, base.line_height, self.styles.get("karaoke", {}))
config = copy.copy(base)
config.line_height = layout.slot_height
config.lyric_line_height = layout.lyric_line_height
config.lyric_y_offset = layout.lyric_y_offset
config.chord_font_size = layout.chord_font_size
config.max_visible_lines = min(base.max_visible_lines, layout.max_visible_lines)
self.logger.info(
f"Guitar chords: {len(self.chords)} changes, {config.max_visible_lines} lines/screen, "
f"slot {layout.slot_height}px, chord font {layout.chord_font_size}px"
)
return config
def _build_chord_style(self, primary: Style, karaoke_styles: dict, name: str, size: int) -> Style:
"""Chord symbols: upcoming = chord colour (gold), played = the sung lyric colour."""
style = copy.copy(primary)
style.Name = name
style.Fontsize = size
color = karaoke_styles.get("chord_color")
upcoming = tuple(int(x.strip()) for x in str(color).split(",")) if color else DEFAULT_CHORD_COLOR
if len(upcoming) == 3:
upcoming = (*upcoming, 255)
style.SecondaryColour = upcoming
style.PrimaryColour = primary.PrimaryColour
ratio = size / max(1, self.font_size)
if primary.Outline:
style.Outline = max(1, int(round(primary.Outline * ratio)))
style.Alignment = 7 # top-left: chords are positioned by their left edge
return style
def _chord_box_events(self, box_style: Style) -> List[Event]:
"""Top-right "now / next" chord box for the whole song (covers intros and breaks)."""
width, height = self.video_resolution
x, y = int(width * CHORD_BOX_RIGHT), int(height * CHORD_BOX_TOP)
next_size = max(1, int(round(box_style.Fontsize * CHORD_BOX_NEXT_RATIO)))
now_colour = _ass_colour(box_style.SecondaryColour)
next_colour = _ass_colour(_fade(box_style.SecondaryColour, TRANSLATION_OPACITY))
events = []
for i, chord in enumerate(self.chords):
nxt = self.chords[i + 1] if i + 1 < len(self.chords) else None
event = Event()
event.type = "Dialogue"
event.Layer = 2
event.Style = box_style
event.Start = chord.start
# Hold the chord through short no-chord gaps so the box doesn't flicker
event.End = nxt.start if nxt and nxt.start - chord.end < 1.0 else chord.end
text = f"{{\\an9}}{{\\pos({x},{y})}}{{\\1c{now_colour}}}{chord.label}"
if nxt:
text += f"\\N{{\\fs{next_size}}}{{\\1c{next_colour}}}next: {nxt.label}"
event.Text = text
events.append(event)
return events
def _build_translation_style(self, primary: Style, karaoke_styles: dict) -> Style:
"""Static style for translation rows: the unsung lyric colour, faded and smaller.
@@ -284,6 +373,7 @@ class SubtitlesGenerator:
"""Create regular lyric screens, handling instrumental boundaries."""
screens: List[LyricsScreen] = []
current_screen: Optional[LyricsScreen] = None
chords_by_segment = assign_chords_to_segments(segments, getattr(self, "chords", None) or [])
for i, segment in enumerate(segments):
self.logger.debug(f"Processing segment {i}: {segment.start_time:.2f}s - {segment.end_time:.2f}s")
@@ -306,7 +396,12 @@ class SubtitlesGenerator:
self.logger.debug(" Created new screen")
# Add line to current screen
line = LyricsLine(logger=self.logger, segment=segment, screen_config=self.config)
line = LyricsLine(
logger=self.logger,
segment=segment,
screen_config=self.config,
chords=chords_by_segment.get(segment.id, []),
)
current_screen.lines.append(line)
self.logger.debug(f" Added line to screen (now has {len(current_screen.lines)} lines)")
@@ -419,6 +514,16 @@ class SubtitlesGenerator:
a.add_style(translation_style)
self.config.translation_style = translation_style
if self.config.chord_font_size:
chord_style = self._build_chord_style(style_list[0], karaoke_styles, "Karaoke.Chord", self.config.chord_font_size)
a.add_style(chord_style)
self.config.chord_style = chord_style
box_size = int(round(self.config.chord_font_size * CHORD_BOX_SIZE_RATIO))
box_style = self._build_chord_style(style_list[0], karaoke_styles, "Karaoke.ChordBox", box_size)
box_style.Alignment = 9
a.add_style(box_style)
self._chord_box_style = box_style
a.events_format = ["Layer", "Style", "Start", "End", "MarginV", "Text"]
# Primary (fallback) style is the first one (singer 1 for duet, ass_name for solo)
primary_style = style_list[0]
@@ -463,4 +568,8 @@ class SubtitlesGenerator:
for event in events:
ass_file.add(event)
if getattr(self.config, "chord_style", None) is not None:
for event in self._chord_box_events(self._chord_box_style):
ass_file.add(event)
return ass_file
+215
View File
@@ -0,0 +1,215 @@
#!/usr/bin/env python3
"""
Detect guitar chords for one song: an ensemble of ChordMini models over several stem mixes.
python scripts/medley/detect_chords.py --chordmini ~/ChordMini \\
--input mix=song.flac --input inst=instrumental_clean.flac --input harm=harmonic.flac \\
--out chords.json
Run it with ChordMini's own Python (it needs torch + librosa + mir_eval); see
docs/MEDLEYS.md § Guitar chords for setup and why this ensemble.
Pipeline:
1. Both ChordMini checkpoints (BTC and 2E1D, continual-learning versions; the best open
chord models we found, ~80% MIREX/majmin) on every input -> .lab files.
2. Each label simplified to a guitar chord (root + one of: major, m, 7, maj7, m7, sus2,
sus4, dim, aug, m7b5, dim7), then a weighted frame-by-frame vote across all runs.
3. Changes shorter than MIN_CHORD_SECONDS merged away; boundaries snapped to the
nearest beat when one is within BEAT_SNAP_SECONDS.
4. Roots spelled for the local key (Ab/Db/Eb7 in A-flat, not G#/C#/D#7).
Output: JSON list of {"start", "end", "label"} (seconds, no "no chord" entries).
"""
import argparse
import json
import os
import subprocess
import sys
import tempfile
from collections import defaultdict
import numpy as np
FRAME = 0.05
MIN_CHORD_SECONDS = 0.6
BEAT_SNAP_SECONDS = 0.2
KEY_WINDOW_SECONDS = 20.0
MODELS = {"BTC": ("btc_model_best.pth", 1.1), "ChordNet": ("2e1d_model_best.pth", 1.0)}
INPUT_WEIGHTS = {"inst": 1.2} # vocals removed = cleaner harmony; everything else 1.0
PITCH = {"C": 0, "C#": 1, "Db": 1, "D": 2, "D#": 3, "Eb": 3, "E": 4, "Fb": 4, "E#": 5, "F": 5, "F#": 6,
"Gb": 6, "G": 7, "G#": 8, "Ab": 8, "A": 9, "A#": 10, "Bb": 10, "B": 11, "Cb": 11, "B#": 0}
SHARP_NAMES = ["C", "C#", "D", "D#", "E", "F", "F#", "G", "G#", "A", "A#", "B"]
FLAT_NAMES = ["C", "Db", "D", "Eb", "E", "F", "Gb", "G", "Ab", "A", "Bb", "B"]
FLAT_KEYS = {1, 3, 5, 6, 8, 10} # Db Eb F Gb Ab Bb major
QUALITY = {"maj": "", "": "", "min": "m", "7": "7", "maj7": "maj7", "min7": "m7", "sus4": "sus4",
"sus2": "sus2", "dim": "dim", "aug": "aug", "maj6": "", "min6": "m", "dim7": "dim7",
"hdim7": "m7b5", "minmaj7": "m", "9": "7", "maj9": "maj7", "min9": "m7", "11": "7", "13": "7"}
# Scale degree (semitones above the key) -> qualities that are diatonic there in a major key
DIATONIC = {0: {"", "maj7"}, 2: {"m", "m7"}, 4: {"m", "m7"}, 5: {"", "maj7"}, 7: {"", "7"}, 9: {"m", "m7"},
11: {"dim", "m7b5"}}
def simplify(label: str):
"""'G#:7' -> (8, '7'); 'N'/'X' -> None. Drops inversions ('C/E' -> C)."""
label = label.strip()
if label in ("N", "X", ""):
return None
root, _, qual = label.partition(":")
qual = qual.split("/")[0].split("(")[0]
root = root.split("/")[0]
if root not in PITCH:
return None
return PITCH[root], QUALITY.get(qual, "")
def spell(pc: int, qual: str, key: int) -> str:
names = FLAT_NAMES if key in FLAT_KEYS else SHARP_NAMES
name = names[pc]
# Borrowed bIII / bVI / bVII in sharp keys read better as flats (Eb, Ab, Bb in C)
if key not in FLAT_KEYS and pc in ((key + 3) % 12, (key + 8) % 12, (key + 10) % 12) and len(name) > 1:
name = FLAT_NAMES[pc]
return name + qual
def estimate_key(chords, t0: float, t1: float) -> int:
"""Major key whose diatonic chords cover the most time in [t0, t1]."""
scores = np.zeros(12)
for start, end, (pc, qual) in chords:
overlap = min(end, t1) - max(start, t0)
if overlap <= 0:
continue
for k in range(12):
if qual in DIATONIC.get((pc - k) % 12, ()):
scores[k] += overlap * (1.5 if (pc - k) % 12 in (0, 7) else 1.0)
return int(np.argmax(scores))
def read_lab(path):
rows = []
with open(path) as f:
for line in f:
parts = line.split()
if len(parts) >= 3:
rows.append((float(parts[0]), float(parts[1]), " ".join(parts[2:])))
return rows
def vote(labs, duration: float):
"""labs: list of (weight, rows). Returns per-frame winning (pc, qual) or None."""
n = int(np.ceil(duration / FRAME))
tallies = [defaultdict(float) for _ in range(n)]
for weight, rows in labs:
for start, end, label in rows:
chord = simplify(label)
for i in range(int(start / FRAME), min(n, int(np.ceil(end / FRAME)))):
tallies[i][chord] += weight
return [max(t.items(), key=lambda kv: kv[1])[0] if t else None for t in tallies]
def frames_to_segments(frames):
segs, start = [], 0
for i in range(1, len(frames) + 1):
if i == len(frames) or frames[i] != frames[start]:
segs.append([start * FRAME, i * FRAME, frames[start]])
start = i
return segs
def merge_short(segs, min_len: float = MIN_CHORD_SECONDS):
"""Fold segments shorter than min_len into the longer neighbour, until none remain."""
segs = [s[:] for s in segs]
while True:
short = [i for i, s in enumerate(segs) if s[1] - s[0] < min_len]
if not short or len(segs) == 1:
break
i = min(short, key=lambda j: segs[j][1] - segs[j][0])
prev_len = segs[i - 1][1] - segs[i - 1][0] if i > 0 else -1
next_len = segs[i + 1][1] - segs[i + 1][0] if i + 1 < len(segs) else -1
if prev_len >= next_len:
segs[i - 1][1] = segs[i][1]
else:
segs[i + 1][0] = segs[i][0]
del segs[i]
# Re-join neighbours that now carry the same chord
j = 0
while j + 1 < len(segs):
if segs[j][2] == segs[j + 1][2]:
segs[j][1] = segs[j + 1][1]
del segs[j + 1]
else:
j += 1
return segs
def snap_to_beats(segs, beats, tolerance: float = BEAT_SNAP_SECONDS):
if len(beats) == 0:
return segs
beats = np.asarray(beats)
for i in range(1, len(segs)):
b = beats[np.argmin(np.abs(beats - segs[i][0]))]
if abs(b - segs[i][0]) <= tolerance and segs[i - 1][0] < b < segs[i][1]:
segs[i - 1][1] = segs[i][0] = float(b)
return segs
def label_segments(segs):
"""Spell each chord for the key around it; drop no-chord segments."""
chords = [(s, e, c) for s, e, c in segs if c is not None]
out = []
for start, end, (pc, qual) in chords:
mid = (start + end) / 2
key = estimate_key(chords, mid - KEY_WINDOW_SECONDS / 2, mid + KEY_WINDOW_SECONDS / 2)
out.append({"start": round(start, 3), "end": round(end, 3), "label": spell(pc, qual, key)})
return out
def run_chordmini(chordmini: str, audio: str, model: str, out_dir: str) -> str:
ckpt, _ = MODELS[model]
in_dir = tempfile.mkdtemp(prefix="cm_in_")
link = os.path.join(in_dir, "song" + os.path.splitext(audio)[1])
os.symlink(os.path.abspath(audio), link)
args = [sys.executable, "src/evaluation/test.py", "--model_type", model, "--checkpoint", f"checkpoints/{ckpt}",
"--config", "config/ChordMini.yaml", "--audio_dir", in_dir, "--save_dir", out_dir, "--use_overlap",
"--vote_aggregation", "logit", "--min_segment_duration", "0.5", "--smooth_predictions",
"--use_gaussian", "--kernel_size", "9"]
if model == "BTC":
args.append("--smooth_logits")
subprocess.run(args, cwd=chordmini, check=True, capture_output=True)
return os.path.join(out_dir, "song.lab")
def main(argv=None):
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--chordmini", required=True, help="ChordMini checkout (github.com/ptnghia-j/ChordMini)")
p.add_argument("--input", action="append", required=True, help="name=path; e.g. mix=song.flac inst=inst.flac")
p.add_argument("--beats-from", help="audio to beat-track (default: the 'mix' input)")
p.add_argument("--out", required=True)
args = p.parse_args(argv)
import librosa
import scipy.signal
if not hasattr(scipy.signal, "hann"): # librosa 0.10.1 beat tracking on scipy >= 1.13
scipy.signal.hann = scipy.signal.windows.hann
inputs = dict(i.split("=", 1) for i in args.input)
labs, duration = [], 0.0
with tempfile.TemporaryDirectory() as tmp:
for name, path in inputs.items():
duration = max(duration, librosa.get_duration(path=path))
for model, (_, model_weight) in MODELS.items():
lab = run_chordmini(args.chordmini, path, model, os.path.join(tmp, f"{name}_{model}"))
labs.append((model_weight * INPUT_WEIGHTS.get(name, 1.0), read_lab(lab)))
segs = merge_short(frames_to_segments(vote(labs, duration)))
beat_src = args.beats_from or inputs.get("mix") or next(iter(inputs.values()))
y, sr = librosa.load(beat_src, sr=22050, mono=True)
_, beat_frames = librosa.beat.beat_track(y=y, sr=sr)
segs = snap_to_beats(segs, librosa.frames_to_time(beat_frames, sr=sr))
chords = label_segments(segs)
with open(args.out, "w") as f:
json.dump(chords, f, indent=1)
print(f"{len(chords)} chord changes -> {args.out}")
if __name__ == "__main__":
sys.exit(main())
+97 -2
View File
@@ -13,6 +13,8 @@ Usage (from the repo root, in an env where karaoke_gen imports):
python scripts/medley/make_medley.py fetch spec.json # Firestore + GCS -> <workdir>/<job_id>/
python scripts/medley/make_medley.py audio spec.json # medley WAV/MP3 per audio variant
python scripts/medley/make_medley.py video spec.json [--resolution 4k] # karaoke + guide MP4s
python scripts/medley/make_medley.py chords spec.json # optional: detect guitar chords per song
# (needs MEDLEY_CHORDMINI; see docs)
Requires Firestore read access (ADC; the claude-readonly SA is enough), the gcloud CLI for GCS
downloads (set MEDLEY_GCLOUD_ACCOUNT to pick an account, e.g. the read-only SA when your user
@@ -122,6 +124,34 @@ def shift_segments(segments, start: float, end: Optional[float], offset: float,
return out
def shift_chords(chords: list[dict], src_start: float, src_end: float, offset: float,
trim_in: float = 0.0, trim_out: float = 0.0) -> list[dict]:
"""Chords of one section on the medley timeline.
Clipped to [src_start + trim_in, src_end - trim_out] (half of each crossfade, so the
outgoing song's chords hand over to the incoming song's mid-fade) and moved by offset.
"""
lo, hi = src_start + trim_in, src_end - trim_out
out = []
for c in chords:
start, end = max(c["start"], lo), min(c["end"], hi)
if end - start > 0.05:
out.append({"start": start + offset, "end": end + offset, "label": c["label"]})
return out
def medley_chords(sections: list[Section], durations: list[float], offsets: list[float],
chords_by_job: dict) -> list[dict]:
out = []
for i, (sec, dur, off) in enumerate(zip(sections, durations, offsets)):
trim_in = sec.join_duration / 2 if i > 0 and sec.join == "crossfade" else 0.0
nxt = sections[i + 1] if i + 1 < len(sections) else None
trim_out = nxt.join_duration / 2 if nxt is not None and nxt.join == "crossfade" else 0.0
out += shift_chords(chords_by_job.get(sec.job_id, []), sec.start, sec.start + dur, off - sec.start,
trim_in, trim_out)
return out
def build_filtergraph(sections: list[Section], durations: list[float], gains_db: list[float]) -> str:
"""ffmpeg filter_complex joining inputs 0..n-1 (one per section, already trimmed) into [out]."""
chains = []
@@ -266,6 +296,55 @@ def cmd_audio(spec: dict, workdir: str, outdir: str):
"medley_start": o, "medley_end": o + d} for s, d, o in zip(sections, durations, offsets)], f, indent=1)
def cmd_chords(spec: dict, workdir: str):
"""Detect chords for each job with scripts/medley/detect_chords.py (ChordMini ensemble).
MEDLEY_CHORDMINI = a ChordMini checkout with its own .venv (see docs/MEDLEYS.md).
Harmonic stems (guitar+piano+bass+other via htdemucs_6s) are added when the
audio-separator CLI is available; otherwise just the full mix + clean instrumental.
"""
chordmini = os.environ.get("MEDLEY_CHORDMINI")
if not chordmini or not os.path.isdir(chordmini):
raise SystemExit("set MEDLEY_CHORDMINI to a ChordMini checkout (see docs/MEDLEYS.md § Guitar chords)")
python = os.path.join(chordmini, ".venv", "bin", "python")
detect = os.path.join(os.path.dirname(os.path.abspath(__file__)), "detect_chords.py")
for job_id in dict.fromkeys(s["job_id"] for s in spec["sections"]):
d = job_dir(workdir, job_id)
inputs = [f"mix={variant_path(workdir, job_id, GUIDE_VARIANT)}",
f"inst={variant_path(workdir, job_id, 'instrumental_clean')}"]
harm = _harmonic_stem(d, variant_path(workdir, job_id, GUIDE_VARIANT))
if harm:
inputs.append(f"harm={harm}")
args = [python, detect, "--chordmini", chordmini, "--out", os.path.join(d, "chords.json")]
for i in inputs:
args += ["--input", i]
_run(args)
def _harmonic_stem(d: str, source: str) -> Optional[str]:
"""guitar+piano+bass+other from htdemucs_6s (drums and vocals removed), cached per job."""
out = os.path.join(d, "harmonic.flac")
if os.path.exists(out):
return out
if subprocess.run(["which", "audio-separator"], capture_output=True).returncode != 0:
log.info("audio-separator not on PATH: skipping harmonic stems")
return None
stems_dir = os.path.join(d, "stems6")
os.makedirs(stems_dir, exist_ok=True)
_run(["audio-separator", "-m", "htdemucs_6s.yaml", "--output_format", "FLAC", "--output_dir", stems_dir, source],
capture_output=True)
parts = [os.path.join(stems_dir, f) for f in sorted(os.listdir(stems_dir))
if any(f"({n})" in f for n in ("Guitar", "Piano", "Bass", "Other"))]
if len(parts) != 4:
log.warning("unexpected htdemucs_6s output in %s", stems_dir)
return None
args = ["ffmpeg", "-hide_banner", "-loglevel", "error", "-y"]
for part in parts:
args += ["-i", part]
_run(args + ["-filter_complex", "amix=inputs=4:normalize=0", out])
return out
def _load_reviewed_lyrics(workdir: str, job_id: str):
from karaoke_gen.lyrics_transcriber.correction.operations import CorrectionOperations
from karaoke_gen.lyrics_transcriber.types import CorrectionResult
@@ -320,6 +399,20 @@ def cmd_video(spec: dict, workdir: str, outdir: str, resolution: str):
medley, audio, _, pad = CountdownProcessor(cache_dir=cache_dir, logger=log).process(
correction_result=medley, audio_filepath=karaoke_wav)
chords = None
if spec.get("chords"):
from karaoke_gen.lyrics_transcriber.output.ass.chords import ChordChange
by_job = {}
for job_id in dict.fromkeys(s.job_id for s in sections):
path = os.path.join(job_dir(workdir, job_id), "chords.json")
if not os.path.exists(path):
raise SystemExit(f"no chords for {job_id}: run the `chords` step first")
with open(path) as f:
by_job[job_id] = json.load(f)
chords = [ChordChange(c["start"] + pad, c["end"] + pad, c["label"])
for c in medley_chords(sections, durations, offsets, by_job)]
log.info("guitar chords: %d changes", len(chords))
artist, title = spec["artist"], spec["title"]
vg = VideoGenerator(logger=log, ffmpeg_base_command="ffmpeg -hide_banner -loglevel error -nostats -y",
render_bounding_boxes=False, output_png=True, output_jpg=False)
@@ -333,7 +426,7 @@ def cmd_video(spec: dict, workdir: str, outdir: str, resolution: str):
generate_cdg=False, generate_plain_text=True, generate_lrc=True, video_resolution=resolution)
outputs = OutputGenerator(cfg, log).generate_outputs(
transcription_corrected=medley, lyrics_results={}, output_prefix=title, audio_filepath=audio,
artist=artist, title=title)
artist=artist, title=title, chords=chords)
w, h = {"4k": (3840, 2160), "1080p": (1920, 1080), "720p": (1280, 720), "360p": (640, 360)}[resolution]
intro_s, end_s = style["intro"]["video_duration"], style["end"]["video_duration"]
@@ -359,7 +452,7 @@ def cmd_video(spec: dict, workdir: str, outdir: str, resolution: str):
def main(argv=None):
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("command", choices=["fetch", "audio", "video"])
p.add_argument("command", choices=["fetch", "audio", "video", "chords"])
p.add_argument("spec")
p.add_argument("--resolution", default="4k", choices=["4k", "1080p", "720p", "360p"])
args = p.parse_args(argv)
@@ -372,6 +465,8 @@ def main(argv=None):
outdir = os.path.join(base, spec.get("outdir", "out"))
if args.command == "fetch":
cmd_fetch(spec, workdir)
elif args.command == "chords":
cmd_chords(spec, workdir)
elif args.command == "audio":
cmd_audio(spec, workdir, outdir)
else:
@@ -0,0 +1,170 @@
"""Guitar chords: chord symbols above the words they land on + a now/next chord box."""
import os
import re
from PIL import ImageFont
from karaoke_gen.lyrics_transcriber.output.ass.chords import (
ChordChange,
assign_chords_to_segments,
chord_at,
clean_chords,
place_chords,
word_index_for_time,
)
from karaoke_gen.lyrics_transcriber.output.ass.config import chord_layout
from karaoke_gen.lyrics_transcriber.output.subtitles import SubtitlesGenerator
from karaoke_gen.lyrics_transcriber.types import LyricsSegment, Word
from karaoke_gen.style_loader import DEFAULT_KARAOKE_STYLE
FONT_PATH = os.path.abspath(os.path.join(
os.path.dirname(__file__), "..", "..", "..", "..", "karaoke_gen", "resources", "AvenirNext-Bold.ttf"
))
W, H = 1920, 1080
FONT = 120
def seg(text, start, seg_id=None, step=0.5):
words = [
Word(id=f"{seg_id or start}-{i}", text=w, start_time=start + i * step, end_time=start + (i + 1) * step - 0.05)
for i, w in enumerate(text.split())
]
return LyricsSegment(id=seg_id or f"s{start}", text=text, words=words,
start_time=words[0].start_time, end_time=words[-1].end_time)
def generate(tmp_path, segments, chords=None, karaoke_overrides=None):
karaoke = dict(DEFAULT_KARAOKE_STYLE)
karaoke["font_path"] = FONT_PATH
karaoke.update(karaoke_overrides or {})
gen = SubtitlesGenerator(output_dir=str(tmp_path), video_resolution=(W, H), font_size=FONT,
line_height=FONT, styles={"karaoke": karaoke})
path = gen.generate_ass(segments, "test", None, chords=chords)
with open(path, encoding="utf-8") as f:
return f.read()
def dialogues(ass_text, style=None):
lines = [line for line in ass_text.splitlines() if line.startswith("Dialogue:")]
return [d for d in lines if style is None or f",{style}," in d]
def pos(d):
return tuple(int(v) for v in re.search(r"\\pos\((\d+),(\d+)\)", d).groups())
# ---- chord timeline helpers ----
def test_clean_chords_drops_no_chord_and_merges_repeats():
chords = clean_chords([
ChordChange(4, 6, "C"), ChordChange(0, 2, "G"), ChordChange(2, 4, "G"),
ChordChange(6, 7, "N"), ChordChange(7, 9, "D7"),
])
assert [(c.start, c.end, c.label) for c in chords] == [(0, 4, "G"), (4, 6, "C"), (7, 9, "D7")]
def test_word_index_snaps_a_pickup_strum_to_the_next_word():
s = seg("one two three", 10.0) # words at 10.0, 10.5, 11.0
assert word_index_for_time(s, 10.3) == 1 # 0.2s before "two" -> over "two"
assert word_index_for_time(s, 10.2) == 0
assert word_index_for_time(s, 5.0) == 0 # before the line -> first word
assert word_index_for_time(s, 20.0) == 2
def test_assign_puts_changes_on_the_line_being_sung():
a, b = seg("first line here", 10.0, "a"), seg("second line", 20.0, "b")
chords = [ChordChange(9.8, 10.9, "G"), ChordChange(10.9, 15.0, "C"),
ChordChange(15.0, 20.4, "D"), ChordChange(20.4, 30, "G")]
by_seg = assign_chords_to_segments([a, b], chords)
assert [c.label for c in by_seg["a"]] == ["G", "C"]
# D started between the lines and is still ringing when "second" is sung: repeated over it
assert [c.label for c in by_seg["b"]] == ["D", "G"]
def test_assign_skips_the_ringing_chord_when_a_change_lands_on_the_first_word():
a = seg("only line", 10.0, "a")
by_seg = assign_chords_to_segments([a], [ChordChange(2, 10.1, "D"), ChordChange(10.1, 12, "G")])
assert [c.label for c in by_seg["a"]] == ["G"]
def test_place_chords_never_overlaps_quick_changes():
s = seg("hello world", 0.0)
measure = lambda text: 10 * len(text) # 10px per character
placed = place_chords(s, [ChordChange(0.0, 0.1, "G"), ChordChange(0.1, 0.4, "Cmaj7"),
ChordChange(0.5, 1, "D")], ["hello", "world"], 100, measure, measure, 5)
xs = [p.x for p in placed]
assert xs[0] == 100 # over "hello"
assert xs[1] == 100 + 10 + 5 # pushed right of "G" (same word)
# "world" starts at 160, but Cmaj7 runs to 165 (+5 spacing): pushed clear of it
assert [p.word_index for p in placed] == [0, 0, 1]
assert xs[2] == xs[1] + measure("Cmaj7") + 5
def test_chord_at():
chords = [ChordChange(0, 2, "G"), ChordChange(2, 4, "C")]
assert chord_at(chords, 2.5).label == "C"
assert chord_at(chords, 9) is None
def test_chord_layout_reserves_a_row_above_the_line():
layout = chord_layout(120, 120)
assert layout.chord_font_size == 66
assert layout.lyric_y_offset == round(66 * 1.05)
assert layout.slot_height == 120 + layout.lyric_y_offset
assert chord_layout(120, 120, {"chord_font_size": 50, "chord_max_visible_lines": 3}).max_visible_lines == 3
# ---- rendered ASS ----
def test_no_chords_leaves_the_classic_layout_untouched(tmp_path):
segments = [seg("hello there world", 5.0, "a")]
assert generate(tmp_path, segments) == generate(tmp_path, segments, chords=[])
def test_chord_symbols_sit_above_their_words(tmp_path):
segments = [seg("hello there world", 5.0, "a")]
chords = [ChordChange(0, 5.5, "G"), ChordChange(5.5, 9, "C")]
ass = generate(tmp_path, segments, chords)
assert "Style: Karaoke.Chord," in ass and "Style: Karaoke.ChordBox," in ass
lyric = next(d for d in dialogues(ass) if "\\kf" in d and ",Karaoke.Chord" not in d)
chord_rows = dialogues(ass, "Karaoke.Chord")
assert [re.sub(r"\{[^}]*\}", "", d.split(",", 5)[-1]) for d in chord_rows] == ["G", "C"]
lyric_x, lyric_y = pos(lyric)
(g_x, g_y), (c_x, c_y) = pos(chord_rows[0]), pos(chord_rows[1])
assert g_y == c_y < lyric_y # chord row above the line
assert lyric_y - g_y == chord_layout(FONT, FONT).lyric_y_offset
# G sits on the centred line's left edge, C where "there" starts. Compare as a
# fraction of the line width: the renderer scales PIL metrics to match libass.
rendered_width = 2 * (lyric_x - g_x)
font = ImageFont.truetype(FONT_PATH, FONT)
expected = font.getlength("hello ") / font.getlength("hello there world")
assert abs((c_x - g_x) / rendered_width - expected) < 0.01
def test_chord_lights_up_when_played(tmp_path):
segments = [seg("hello there world", 5.0, "a")]
ass = generate(tmp_path, segments, [ChordChange(5.5, 9, "C")])
row = dialogues(ass, "Karaoke.Chord")[0]
start = row.split(",")[2] # H:MM:SS.cc
h, m, sec = start.split(":")
start_s = int(h) * 3600 + int(m) * 60 + float(sec)
wait = int(re.search(r"\\k(\d+)\}\{\\kf", row).group(1))
assert abs(start_s + wait / 100 - 5.5) < 0.02
def test_chord_box_shows_now_and_next(tmp_path):
segments = [seg("hello there world", 5.0, "a")]
ass = generate(tmp_path, segments, [ChordChange(0, 5.5, "G"), ChordChange(5.5, 9, "C")])
box = dialogues(ass, "Karaoke.ChordBox")
assert len(box) == 2
assert box[0].endswith("next: C") and "}G\\N" in box[0]
assert "next:" not in box[1] # last chord
def test_chord_colour_comes_from_theme(tmp_path):
segments = [seg("hello there world", 5.0, "a")]
ass = generate(tmp_path, segments, [ChordChange(5.5, 9, "C")], {"chord_color": "10, 20, 30"})
style = next(line for line in ass.splitlines() if line.startswith("Style: Karaoke.Chord,"))
assert "&H001E140A" in style # secondary (upcoming) colour = theme chord_color (BGR)
+64
View File
@@ -0,0 +1,64 @@
"""Unit tests for the chord-detection ensemble's pure helpers (scripts/medley/detect_chords.py)."""
import importlib.util
import sys
from pathlib import Path
_SCRIPT = Path(__file__).resolve().parents[2] / "scripts" / "medley" / "detect_chords.py"
_spec = importlib.util.spec_from_file_location("detect_chords", _SCRIPT)
dc = importlib.util.module_from_spec(_spec)
sys.modules[_spec.name] = dc
_spec.loader.exec_module(dc)
def test_simplify_maps_to_guitar_chords():
assert dc.simplify("G#:7") == (8, "7")
assert dc.simplify("C:maj/3") == (0, "")
assert dc.simplify("A:min7") == (9, "m7")
assert dc.simplify("D:hdim7") == (2, "m7b5")
assert dc.simplify("Bb:min6") == (10, "m")
assert dc.simplify("N") is None and dc.simplify("X") is None
def test_spelling_follows_the_key():
ab, g, c = 8, 7, 0
assert dc.spell(8, "", ab) == "Ab" and dc.spell(1, "", ab) == "Db" and dc.spell(3, "7", ab) == "Eb7"
assert dc.spell(6, "m", 2) == "F#m" # F#m in D
assert dc.spell(10, "", c) == "Bb" # borrowed bVII in C stays flat
assert dc.spell(10, "", g) == "Bb" and dc.spell(3, "", g) == "Eb" # bIII and bVI in G
assert dc.spell(1, "", 9) == "C#" # C# in A
def test_estimate_key_picks_the_diatonic_home():
i_iv_v_in_ab = [(0, 10, (8, "")), (10, 15, (1, "")), (15, 20, (3, "7"))]
assert dc.estimate_key(i_iv_v_in_ab, 0, 20) == 8
in_d = [(0, 4, (2, "")), (4, 6, (6, "m7")), (6, 8, (7, "")), (8, 10, (9, ""))]
assert dc.estimate_key(in_d, 0, 10) == 2
def test_vote_weights_and_ties():
labs = [
(1.0, [(0.0, 1.0, "G"), (1.0, 2.0, "C")]),
(1.0, [(0.0, 1.0, "G"), (1.0, 2.0, "A:min")]),
(1.5, [(0.0, 2.0, "G")]),
]
frames = dc.vote(labs, 2.0)
assert frames[0] == (7, "")
assert frames[-1] == (7, "") # G 1.5 beats C 1.0 and Am 1.0
def test_merge_short_folds_blips_into_neighbours():
segs = [[0, 4, "G"], [4, 4.2, "C"], [4.2, 8, "G"], [8, 12, "D"]]
assert dc.merge_short(segs) == [[0, 8, "G"], [8, 12, "D"]]
def test_snap_to_beats_moves_boundaries_within_tolerance_only():
segs = [[0, 4.1, "G"], [4.1, 8.5, "C"], [8.5, 12, "D"]]
out = dc.snap_to_beats([s[:] for s in segs], [2.0, 4.0, 6.0, 8.0, 10.0])
assert out[0][1] == out[1][0] == 4.0
assert out[1][1] == 8.5 # 0.5s from the nearest beat: left alone
def test_label_segments_spells_and_drops_no_chord():
segs = [[0, 10, (8, "")], [10, 12, None], [12, 20, (1, "")], [20, 26, (3, "7")]]
assert [c["label"] for c in dc.label_segments(segs)] == ["Ab", "Db", "Eb7"]
+39
View File
@@ -107,3 +107,42 @@ class TestBuildFiltergraph:
assert "afade=t=in:st=0:d=0.500" in graph # pot in on the finale
assert "[x3][s4]concat=n=2:v=0:a=1[x4]" in graph
assert graph.endswith("[x4]loudnorm=I=-14.0:TP=-1.0:LRA=11,aresample=44100[out]")
class TestMedleyChords:
def test_chords_follow_their_section_and_hand_over_mid_crossfade(self):
sections = medley.parse_spec(EXAMPLE)
durations = [54.0, 52.0, 57.0, 82.0, 66.9]
offsets = medley.compute_offsets(sections, durations)
by_job = {
"a": [{"start": 0.0, "end": 60.0, "label": "G"}, {"start": 130.0, "end": 200.0, "label": "Ab"}],
"b": [{"start": 0.0, "end": 52.0, "label": "C"}],
"c": [{"start": 0.0, "end": 57.0, "label": "D"}],
"d": [{"start": 0.0, "end": 82.0, "label": "F"}],
}
out = medley.medley_chords(sections, durations, offsets, by_job)
labels = [(round(c["start"], 2), round(c["end"], 2), c["label"]) for c in out]
# opener G cut where the 2s crossfade is half done (54 - 1), C starts there
assert labels[0] == (0.0, 53.0, "G")
assert labels[1] == (53.0, 103.0, "C")
assert labels[2] == (103.0, 158.0, "D")
# F runs to the end of its section (the finale joins with a gap, not a crossfade)
assert labels[3] == (158.0, 239.0, "F")
# the finale's chords start at the finale (source 132.2 -> medley 239.7)
assert labels[4][0] == pytest.approx(239.7) and labels[4][2] == "Ab"
def test_shift_chords_drops_slivers(self):
out = medley.shift_chords([{"start": 0, "end": 10.02, "label": "G"}, {"start": 10.02, "end": 20, "label": "C"}],
10.0, 20.0, 100.0)
assert [c["label"] for c in out] == ["C"]
class TestSingleSong:
def test_one_section_spec_builds_a_plain_graph_and_full_chords(self):
sections = medley.parse_spec({"sections": [{"job_id": "a", "start": 0, "end": None}]})
assert medley.compute_offsets(sections, [200.0]) == [0.0]
graph = medley.build_filtergraph(sections, [200.0], [1.5])
assert "acrossfade" not in graph and "concat" not in graph
assert graph.endswith("[s0]loudnorm=I=-14.0:TP=-1.0:LRA=11,aresample=44100[out]")
chords = medley.medley_chords(sections, [200.0], [0.0], {"a": [{"start": 0, "end": 200, "label": "G"}]})
assert chords == [{"start": 0.0, "end": 200.0, "label": "G"}]