mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
feat: guitar chords in karaoke videos + ChordMini ensemble chord detection (v0.277.0) (#1175)
## Summary Adds optional guitar chords to karaoke videos, first used for a Fiverr medley customer who wants chords on their video. - **Rendering (gen library):** `OutputGenerator.generate_outputs(chords=[ChordChange(start, end, label)])`. - Each chord change during a sung line is drawn above the word it lands on, in a row reserved above every line (the same slot mechanism as translated lyrics). - Chords light up when played, using the same `\k` highlight as lyrics: gold while upcoming (theme `chord_color`), sung-lyric colour once played. Quick changes on one word never overlap. - A top-right "now / next" chord box runs for the whole song, covering intros and instrumental breaks. - With no chords supplied, output is unchanged. A test asserts the ASS is byte-identical. - **Detection (`scripts/medley/detect_chords.py`):** - Ensembles both ChordMini checkpoints (BTC + 2E1D, MIT; about 80% majmin/MIREX, the best open weights we found) over the full mix, gen's clean instrumental, and an htdemucs_6s harmonic mix (guitar + piano + bass + other). - Weighted per-frame vote, then merges short blips, snaps changes to beats and spells chords for the key (Ab/Db/Eb7, F#m, borrowed bIII/bVI/bVII as flats). - **Medley script:** new `chords` step plus a `"chords": true` spec option. Chords are clipped per section and hand over mid-crossfade, then shifted for the countdown. - **Docs:** `docs/MEDLEYS.md` § Guitar chords (setup, method, accuracy caveats, theme keys) and a docs/README.md entry. There's no job or UI option yet. The medley pipeline uses this for now; a product version would need a chords toggle plus detection in a worker. ## Testing - New: `test_guitar_chords.py` (12: chord timeline helpers, word placement measured against the font, layout geometry, highlight timing, now/next box, theme colour, no-chords path byte-identical), `test_detect_chords.py` (7), and 2 medley chord-shifting tests. - `tests/unit`: 2087 passed, 65 skipped. - End to end on the 5 real medley songs: models/inputs agree on 91–99% of major/minor labels, and the output matches the expected I–IV–V progressions. Rendered 1080p and 4K medley videos with 146 chord changes; spot-checked frames (verse, instrumental screen, chorus). - Local CodeRabbit: 1 finding (doc section reference), fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
8ec789a9aa
commit
7cf5beede8
12 files changed
+1022
-12
No files matched your search
+109
@@ -134,6 +134,115 @@ Give the customer the audio file they asked for (usually `medley_instrumental_wi
|
||||
or `.mp3`; offer `instrumental_clean` as an alternative) plus the karaoke video. The per-song
|
||||
gen jobs are a free extra: four full karaoke tracks.
|
||||
|
||||
### Guitar chords (optional)
|
||||
|
||||
The video can show guitar chords. Each chord name sits above the word where the chord
|
||||
changes and lights up when played, and a "now / next" box in the top-right corner shows
|
||||
the current chord all the time, including intros and instrumental breaks. First delivered
|
||||
in October 2026 for the country medley (146 chord changes over 8 minutes). Detection and
|
||||
rendering together took about 15 minutes of machine time once set up.
|
||||
|
||||
#### Checklist (from zero)
|
||||
|
||||
```bash
|
||||
# 1. One-time: ChordMini in its own venv (pinned: the commit we evaluated)
|
||||
git clone https://github.com/ptnghia-j/ChordMini ~/ChordMini
|
||||
git -C ~/ChordMini checkout aa6e3a8
|
||||
cd ~/ChordMini && uv venv -p 3.12 .venv && uv pip install -p .venv/bin/python -r requirements.txt
|
||||
|
||||
# 2. Per medley (karaoke-gen checkout, env where karaoke_gen imports + audio-separator on PATH)
|
||||
export PYTHONPATH=$PWD MEDLEY_CHORDMINI=~/ChordMini
|
||||
export MEDLEY_GCLOUD_ACCOUNT=claude-readonly@nomadkaraoke.iam.gserviceaccount.com
|
||||
# add "chords": true to the spec, then:
|
||||
python scripts/medley/make_medley.py fetch spec.json # if not already fetched
|
||||
python scripts/medley/make_medley.py chords spec.json # ~1-2 min/song: stems + 6 model runs
|
||||
python scripts/medley/make_medley.py audio spec.json # if not already built
|
||||
python scripts/medley/make_medley.py video spec.json --resolution 1080p # check frames first
|
||||
python scripts/medley/make_medley.py video spec.json # 4K final
|
||||
```
|
||||
|
||||
- **Check it before delivering.** Watch the `With Vocals` video for a verse or two of each
|
||||
song (chords should change with the band), and grab a few frames to check layout: one in a
|
||||
verse, one on an `INSTRUMENTAL` screen (the corner box should show chords there), one in a
|
||||
chorus.
|
||||
- **Fixing a chord.** Edit `work/<job_id>/chords.json` (list of
|
||||
`{"start", "end", "label"}` in that song's own source seconds) and re-run `video`. If a
|
||||
customer sends their own chord chart, enter it the same way: their chords replace the
|
||||
detected ones, and nothing else changes.
|
||||
- **Deliver both versions** (with and without chords), and say the chords are auto-detected.
|
||||
|
||||
#### A single song, not a medley
|
||||
|
||||
The same pipeline adds chords to any one gen track: write a one-section spec
|
||||
(`{"job_id": "...", "start": 0, "end": null}`) and run `fetch`, `chords`, `audio`, `video`.
|
||||
The output is the job's normal karaoke video plus chords (re-rendered locally from its
|
||||
reviewed lyrics and theme).
|
||||
|
||||
#### How detection works (`scripts/medley/detect_chords.py`)
|
||||
|
||||
We compared the open chord-recognition models in October 2026.
|
||||
[ChordMini](https://github.com/ptnghia-j/ChordMini) (MIT) had the best published accuracy
|
||||
with downloadable weights: its continual-learning BTC checkpoint scores about 80%
|
||||
majmin/MIREX on a 120-song test set (arXiv 2602.19778, Table 3). ChordFormer
|
||||
(arXiv 2502.11840) reports similar numbers but has no released weights; autochord and
|
||||
Chordino are older and weaker (25-class / ~67%). The script:
|
||||
|
||||
1. Runs **both** ChordMini checkpoints (BTC weight 1.1, 2E1D weight 1.0) on **three
|
||||
inputs**: the full mix, gen's clean instrumental (vocals removed, weight × 1.2) and a
|
||||
"harmonic" mix of htdemucs_6s guitar + piano + bass + other (drums removed too; cached
|
||||
as `work/<job>/harmonic.flac`). Removing vocals before chord recognition is a known
|
||||
accuracy win (APSIPA 2025).
|
||||
2. Simplifies every label to a guitar chord (major, m, 7, maj7, m7, sus2/4, dim, aug,
|
||||
m7b5, dim7; no inversions), then takes a weighted vote on a 50ms frame grid.
|
||||
3. Merges changes shorter than 0.6s into their longer neighbour, and snaps the remaining
|
||||
changes onto librosa beats when one is within 0.2s.
|
||||
4. Spells roots for the local key, estimated over a 20s window: A♭ / D♭ / E♭7 in A-flat,
|
||||
F♯m in D, and borrowed ♭III / ♭VI / ♭VII stay flat in sharp keys.
|
||||
|
||||
**Accuracy evidence.** Run on the five country songs, each pair of model/input runs agreed
|
||||
on 91–99% of major/minor labels (93–99% for the instrumental-vs-harmonic pairs), and the
|
||||
output was the expected progressions (G–C–D7 then A♭–D♭–E♭7 after the half-step key
|
||||
change; F–B♭–C with a II chord; D–F♯m7–G–A). That's a consistency check, not ground truth.
|
||||
|
||||
**Re-evaluating (if a better model appears).** Run each candidate on the same inputs, write
|
||||
`.lab` files and compare pairwise with `mir_eval.chord` (`majmin` and `sevenths` weighted
|
||||
accuracy). Better still, compare against a hand-checked chart for one song. Add a model by
|
||||
extending `MODELS` in the script.
|
||||
|
||||
**Safety.** Load only `btc_model_best.pth` and `2e1d_model_best.pth`. Both load with
|
||||
PyTorch's safe `weights_only=True`; the third bundled checkpoint
|
||||
(`btc_model_large_voca.pt`) needs unsafe unpickling and isn't used.
|
||||
|
||||
#### How rendering works (in gen)
|
||||
|
||||
`karaoke_gen/lyrics_transcriber/output/ass/chords.py`, used via
|
||||
`OutputGenerator.generate_outputs(chords=[ChordChange(start, end, label)])`. With no chords,
|
||||
output is byte-identical to before.
|
||||
|
||||
- **Layout.** Each lyric slot gets a chord row above the line: the slot grows by 1.05 ×
|
||||
the chord font size, and lines per screen are capped at 4. It's the same mechanism as
|
||||
translated lyrics (which add a row below).
|
||||
- **Which line.** A change belongs to the line being sung, from 0.6s before its first
|
||||
word to the end of its last word. The chord already ringing when a line starts is
|
||||
repeated over its first word, so every line shows what to play. Changes between lines
|
||||
appear only in the corner box.
|
||||
- **Which word.** A change sits over the word sung at that moment, or the next word if it
|
||||
comes up to 0.25s early (a pickup strum). Quick changes on one word are pushed right so
|
||||
they never overlap. Positions use the renderer's libass-matched text measurement.
|
||||
- **Highlight.** Each chord is a one-syllable karaoke event (`\k` wait, then `\kf`):
|
||||
upcoming colour, then the sung-lyric colour once played.
|
||||
- **Medley handover.** In the medley script, each section's chords are clipped to the
|
||||
section, handing over to the next song halfway through each crossfade, then shifted for
|
||||
the 3s countdown.
|
||||
- **Theme keys** (optional, `karaoke` block):
|
||||
- `chord_color`: upcoming chord colour, default gold `255, 223, 107`.
|
||||
- `chord_font_size`, or `chord_font_ratio` (default 0.55 × lyric size).
|
||||
- `chord_max_visible_lines` (default 4).
|
||||
|
||||
There's no gen job/UI option for chords yet: it's agent-run through this script. A product
|
||||
version would need a job flag, detection in a worker (ChordMini is small and CPU-friendly),
|
||||
and a chord editor in review.
|
||||
|
||||
## Making it sound good
|
||||
|
||||
- **Cut where the next vocal starts, not where the last one ends.** A section should run until
|
||||
|
||||
@@ -60,6 +60,7 @@
|
||||
|
||||
## Recent Changes
|
||||
|
||||
- **Guitar chords in karaoke videos** (2026-10-16, v0.277.0). `OutputGenerator.generate_outputs(chords=[ChordChange(...)])` draws each chord above the word it lands on (it lights up when played) in a row reserved above every line, plus a "now / next" chord box in the top-right corner that also covers intros and instrumental breaks. Chords come from `scripts/medley/detect_chords.py`, an ensemble of ChordMini models over the mix, instrumental and harmonic stems. It's used by the medley script for now (`"chords": true`); there's no job/UI option yet. Theme keys: `chord_color`, `chord_font_size`/`chord_font_ratio`, `chord_max_visible_lines`. See [MEDLEYS.md § Guitar chords](MEDLEYS.md#guitar-chords-optional).
|
||||
- **Local E2E stack + email-code sign-in** (2026-10-15, v0.275.0). Plan: [archive/2026-10-15-local-stack-e2e-plan.md](archive/2026-10-15-local-stack-e2e-plan.md).
|
||||
- **Local stack.** `scripts/local-stack.sh up --frontend` runs the real API on the Firestore + GCS emulators with `FAKE_WORKERS=1`. In-process fake workers (`backend/services/fake_workers/`) move jobs through their real states in seconds with small real media; the search, download, Stripe Checkout, email, Secret Manager and encoding-VM seams are faked or skipped. The `/api/test/*` router (only mounted in that mode; startup refuses it in production) seeds scenarios, tunes the fakes per user, exposes captured email and serves a fake checkout page.
|
||||
- **New suite.** `frontend/e2e/local-stack` covers sign-up, make, review, instrumental, download, buying credits, the duration-confirm and failure states, and code sign-in through the real UI and API in about 1 minute. It runs in CI as the required `local-stack-e2e` job. Mobile Maestro flows will use the same stack. See [TESTING.md § Local stack](TESTING.md#local-stack-real-backend-fake-workers).
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
"""Guitar chord symbols for karaoke videos.
|
||||
|
||||
Chords are a timed sequence of ``ChordChange`` (start, end, label) for the whole song.
|
||||
Each change that happens while a lyric line is being sung is drawn above the word it
|
||||
lands on, in a row reserved above the line (see ``chord_layout``), and switches colour
|
||||
when the chord is played. A "now / next" box in the top-right corner shows the current
|
||||
chord at all times, which covers intros and instrumental breaks where there's no lyric
|
||||
line to sit a chord above.
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Callable, Dict, List, Optional, Sequence
|
||||
|
||||
from karaoke_gen.lyrics_transcriber.types import LyricsSegment
|
||||
|
||||
# A chord that changes just before a word (strummed on the pickup) belongs to that word
|
||||
CHORD_WORD_SNAP_SECONDS = 0.25
|
||||
# A chord shortly before a line's first word is shown over that first word
|
||||
CHORD_LINE_LEAD_SECONDS = 0.6
|
||||
NO_CHORD_LABELS = frozenset({"", "N", "X", "N.C.", "NC"})
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ChordChange:
|
||||
start: float
|
||||
end: float
|
||||
label: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PlacedChord:
|
||||
chord: ChordChange
|
||||
word_index: int
|
||||
x: int # left edge of the chord label, in pixels
|
||||
|
||||
|
||||
def clean_chords(chords: Sequence[ChordChange]) -> List[ChordChange]:
|
||||
"""Sorted, without no-chord markers, with repeated labels merged."""
|
||||
out: List[ChordChange] = []
|
||||
for c in sorted(chords, key=lambda c: c.start):
|
||||
if c.label.strip() in NO_CHORD_LABELS or c.end <= c.start:
|
||||
continue
|
||||
if out and out[-1].label == c.label and c.start - out[-1].end < 0.05:
|
||||
out[-1] = ChordChange(out[-1].start, c.end, c.label)
|
||||
else:
|
||||
out.append(ChordChange(c.start, c.end, c.label.strip()))
|
||||
return out
|
||||
|
||||
|
||||
def assign_chords_to_segments(
|
||||
segments: Sequence[LyricsSegment], chords: Sequence[ChordChange]
|
||||
) -> Dict[str, List[ChordChange]]:
|
||||
"""Map segment id -> chord changes to draw above that line.
|
||||
|
||||
A change belongs to the line being sung when it happens (from slightly before the
|
||||
first word to the last word's end). Changes between lines go only in the corner box.
|
||||
The chord already ringing when a line starts is repeated over its first word, so a
|
||||
guitarist reading the line always sees what to play.
|
||||
"""
|
||||
result: Dict[str, List[ChordChange]] = {}
|
||||
chords = list(chords)
|
||||
for seg in segments:
|
||||
if not seg.words:
|
||||
continue
|
||||
line_start, line_end = seg.words[0].start_time, seg.words[-1].end_time
|
||||
on_line = [c for c in chords if line_start - CHORD_LINE_LEAD_SECONDS <= c.start <= line_end]
|
||||
ringing = [c for c in chords if c.start < line_start - CHORD_LINE_LEAD_SECONDS and c.end > line_start]
|
||||
if ringing and (not on_line or on_line[0].start > line_start + CHORD_WORD_SNAP_SECONDS):
|
||||
on_line.insert(0, ringing[-1])
|
||||
if on_line:
|
||||
result[seg.id] = on_line
|
||||
return result
|
||||
|
||||
|
||||
def word_index_for_time(segment: LyricsSegment, t: float) -> int:
|
||||
"""Index of the word a chord change at time ``t`` sits above."""
|
||||
idx = 0
|
||||
for i, w in enumerate(segment.words):
|
||||
if w.start_time <= t + CHORD_WORD_SNAP_SECONDS:
|
||||
idx = i
|
||||
else:
|
||||
break
|
||||
return idx
|
||||
|
||||
|
||||
def place_chords(
|
||||
segment: LyricsSegment,
|
||||
chords: Sequence[ChordChange],
|
||||
word_texts: Sequence[str],
|
||||
line_left: int,
|
||||
measure: Callable[[str], int],
|
||||
chord_measure: Callable[[str], int],
|
||||
min_spacing: int,
|
||||
) -> List[PlacedChord]:
|
||||
"""X positions for chords over a line whose rendered text starts at ``line_left``.
|
||||
|
||||
``word_texts`` are the words exactly as rendered (case-transformed) and joined by
|
||||
single spaces. Two chords over the same word (a quick change) are pushed right so
|
||||
they never overlap.
|
||||
"""
|
||||
placed: List[PlacedChord] = []
|
||||
next_free = None
|
||||
for chord in chords:
|
||||
i = word_index_for_time(segment, chord.start)
|
||||
prefix = " ".join(word_texts[:i]) + (" " if i else "")
|
||||
x = line_left + (measure(prefix) if prefix else 0)
|
||||
if next_free is not None and x < next_free:
|
||||
x = next_free
|
||||
placed.append(PlacedChord(chord=chord, word_index=i, x=x))
|
||||
next_free = x + chord_measure(chord.label) + min_spacing
|
||||
return placed
|
||||
|
||||
|
||||
def chord_at(chords: Sequence[ChordChange], t: float) -> Optional[ChordChange]:
|
||||
for c in chords:
|
||||
if c.start <= t < c.end:
|
||||
return c
|
||||
return None
|
||||
@@ -8,6 +8,12 @@ DEFAULT_TRANSLATION_MAX_VISIBLE_LINES = 3 # lyric lines per screen when transla
|
||||
TRANSLATION_GAP_RATIO = 0.12 # gap between a lyric line and its translation / lyric font size
|
||||
TRANSLATION_ROW_RATIO = 1.15 # translation row height / translation font size
|
||||
|
||||
# Guitar-chord layout defaults (see chord_layout)
|
||||
DEFAULT_CHORD_FONT_RATIO = 0.55 # chord symbol font size / lyric font size
|
||||
DEFAULT_CHORD_MAX_VISIBLE_LINES = 4 # lyric lines per screen when showing chords
|
||||
CHORD_GAP_RATIO = 0.0 # gap between the chord row and its lyric line / lyric font size
|
||||
CHORD_ROW_RATIO = 1.05 # chord row height / chord font size
|
||||
|
||||
|
||||
class ScreenConfig:
|
||||
"""Configuration for screen timing and layout.
|
||||
@@ -97,6 +103,12 @@ class ScreenConfig:
|
||||
self.translation_max_rows: int = DEFAULT_TRANSLATION_MAX_ROWS
|
||||
# Height of the lyric line itself within its (taller) slot
|
||||
self.lyric_line_height: int = line_height
|
||||
# Guitar-chord layout: set by SubtitlesGenerator when chords are supplied.
|
||||
# The chord row sits at the top of each slot and the lyric line is pushed
|
||||
# down by lyric_y_offset. None = no chord row (classic layout).
|
||||
self.chord_style = None
|
||||
self.chord_font_size: int = 0
|
||||
self.lyric_y_offset: int = 0
|
||||
|
||||
def get_lead_in_color_ass_format(self) -> str:
|
||||
"""Convert RGB lead-in color to ASS format.
|
||||
@@ -213,6 +225,36 @@ def translation_layout(font_size: float, line_height: int, karaoke_styles: Optio
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChordLayout:
|
||||
"""Geometry of a lyric slot with a chord-symbol row above the line."""
|
||||
|
||||
slot_height: int # chord row + gap + lyric line (the new line_height)
|
||||
lyric_line_height: int
|
||||
chord_font_size: int
|
||||
lyric_y_offset: int # chord row + gap: where the lyric line starts within the slot
|
||||
max_visible_lines: int
|
||||
|
||||
|
||||
def chord_layout(font_size: float, line_height: int, karaoke_styles: Optional[dict] = None) -> ChordLayout:
|
||||
"""Slot geometry for guitar chords.
|
||||
|
||||
Theme keys (all optional, in the ``karaoke`` block): ``chord_font_size`` (absolute)
|
||||
or ``chord_font_ratio``, ``chord_max_visible_lines``.
|
||||
"""
|
||||
k = karaoke_styles or {}
|
||||
c_size = k.get("chord_font_size") or font_size * float(k.get("chord_font_ratio", DEFAULT_CHORD_FONT_RATIO))
|
||||
c_size = max(1, int(round(c_size)))
|
||||
offset = int(round(c_size * CHORD_ROW_RATIO + font_size * CHORD_GAP_RATIO))
|
||||
return ChordLayout(
|
||||
slot_height=line_height + offset,
|
||||
lyric_line_height=line_height,
|
||||
chord_font_size=c_size,
|
||||
lyric_y_offset=offset,
|
||||
max_visible_lines=int(k.get("chord_max_visible_lines", DEFAULT_CHORD_MAX_VISIBLE_LINES)),
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class LineTimingInfo:
|
||||
"""Timing information for a single line."""
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import dataclass, field
|
||||
import functools
|
||||
from typing import Optional, Tuple, List
|
||||
import logging
|
||||
@@ -10,6 +10,7 @@ from karaoke_gen.lyrics_transcriber.types import LyricsSegment
|
||||
from karaoke_gen.lyrics_transcriber.output.ass.event import Event
|
||||
from karaoke_gen.lyrics_transcriber.output.ass.style import Style
|
||||
from karaoke_gen.lyrics_transcriber.output.ass.config import LineState, ScreenConfig
|
||||
from karaoke_gen.lyrics_transcriber.output.ass.chords import ChordChange, place_chords
|
||||
from karaoke_gen.lyrics_transcriber.output.ass.text_direction import is_rtl_text, rtl_karaoke_fill_tags
|
||||
from karaoke_gen.lyrics_transcriber.output.segment_resizer import (
|
||||
display_width,
|
||||
@@ -37,6 +38,8 @@ class LyricsLine:
|
||||
screen_config: ScreenConfig
|
||||
logger: Optional[logging.Logger] = None
|
||||
previous_end_time: Optional[float] = None
|
||||
# Chord changes drawn above this line (only used when the screen config has a chord row)
|
||||
chords: List[ChordChange] = field(default_factory=list)
|
||||
|
||||
def __post_init__(self):
|
||||
"""Ensure logger is initialized"""
|
||||
@@ -173,7 +176,7 @@ class LyricsLine:
|
||||
start_x_position = 0
|
||||
# Apply vertical offset if configured
|
||||
vertical_offset = int(self.screen_config.video_height * (config.lead_in_vert_offset_percent / 100))
|
||||
final_y_position = state.y_position + main_height + vertical_offset
|
||||
final_y_position = state.y_position + config.lyric_y_offset + main_height + vertical_offset
|
||||
|
||||
self.logger.debug(f"Position calculations:")
|
||||
self.logger.debug(f" Video dimensions: {self.screen_config.video_width}x{self.screen_config.video_height}")
|
||||
@@ -271,7 +274,7 @@ class LyricsLine:
|
||||
|
||||
# Main lyrics text with positioning and fade
|
||||
text = (
|
||||
f"{{\\an8}}{{\\pos({x_pos},{state.y_position})}}"
|
||||
f"{{\\an8}}{{\\pos({x_pos},{state.y_position + config.lyric_y_offset})}}"
|
||||
f"{{\\fad({config.fade_in_ms},{config.fade_out_ms})}}"
|
||||
)
|
||||
rtl_tags = rtl_karaoke_fill_tags(line_style.Angle) if is_rtl_text(self.segment.text) else ""
|
||||
@@ -291,6 +294,48 @@ class LyricsLine:
|
||||
if translation_event:
|
||||
events.append(translation_event)
|
||||
|
||||
events.extend(self._create_chord_events(state, line_style, config))
|
||||
|
||||
return events
|
||||
|
||||
def _create_chord_events(self, state: LineState, line_style: Style, config: ScreenConfig) -> List[Event]:
|
||||
"""Chord symbols above the words they land on, lighting up when played.
|
||||
|
||||
Each chord is one karaoke syllable: upcoming colour until the chord's time,
|
||||
then the played colour (same \\k mechanism as the lyric highlight).
|
||||
"""
|
||||
chord_style = getattr(config, "chord_style", None)
|
||||
if chord_style is None or not self.chords or not self.segment.words:
|
||||
return []
|
||||
word_texts = [self._apply_case_transform(w.text.replace("\n", "").strip()) for w in self.segment.words]
|
||||
line_text = " ".join(word_texts)
|
||||
line_width = self._measure_text(line_text, line_style)[0]
|
||||
line_left = config.video_width // 2 - line_width // 2
|
||||
placed = place_chords(
|
||||
self.segment,
|
||||
self.chords,
|
||||
word_texts,
|
||||
line_left,
|
||||
measure=lambda text: self._measure_text(text, line_style)[0],
|
||||
chord_measure=lambda text: self._measure_text(text, chord_style)[0],
|
||||
min_spacing=max(1, chord_style.Fontsize // 3),
|
||||
)
|
||||
events = []
|
||||
start = state.timing.fade_in_time
|
||||
for p in placed:
|
||||
wait_cs = max(0, int(round((p.chord.start - start) * 100)))
|
||||
event = Event()
|
||||
event.type = "Dialogue"
|
||||
event.Layer = 1
|
||||
event.Style = chord_style
|
||||
event.Start = start
|
||||
event.End = state.timing.end_time
|
||||
event.Text = (
|
||||
f"{{\\an7}}{{\\pos({p.x},{state.y_position})}}"
|
||||
f"{{\\fad({config.fade_in_ms},{config.fade_out_ms})}}"
|
||||
f"{{\\k{wait_cs}}}{{\\kf15}}{p.chord.label}"
|
||||
)
|
||||
events.append(event)
|
||||
return events
|
||||
|
||||
# Fraction of the frame width a translation may use before it's wrapped / scaled
|
||||
@@ -361,7 +406,7 @@ class LyricsLine:
|
||||
if widest > max_width:
|
||||
scale = max(1, int(max_width / widest * 100))
|
||||
|
||||
y = state.y_position + config.lyric_line_height + config.translation_gap
|
||||
y = state.y_position + config.lyric_y_offset + config.lyric_line_height + config.translation_gap
|
||||
tags = (
|
||||
f"{{\\an8}}{{\\pos({config.video_width // 2},{y})}}"
|
||||
f"{{\\fad({config.fade_in_ms},{config.fade_out_ms})}}"
|
||||
|
||||
@@ -155,6 +155,7 @@ class OutputGenerator:
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
ass_only: bool = False,
|
||||
chords: Optional[list] = None,
|
||||
) -> OutputPaths:
|
||||
"""Generate all requested output formats.
|
||||
|
||||
@@ -165,6 +166,8 @@ class OutputGenerator:
|
||||
audio_filepath: Path to audio file
|
||||
artist: Optional artist name
|
||||
title: Optional title
|
||||
chords: Optional guitar chord changes (``ChordChange`` list) to draw above the
|
||||
lyrics in the ASS/video (no effect on other formats)
|
||||
ass_only: If True (only in preview_mode), generate only ASS subtitles
|
||||
without video encoding. Useful when video encoding is offloaded
|
||||
to an external service.
|
||||
@@ -188,7 +191,7 @@ class OutputGenerator:
|
||||
# For preview, we only need to generate ASS and video
|
||||
if self.preview_mode:
|
||||
# Generate ASS subtitles for preview
|
||||
outputs.ass = self.subtitle.generate_ass(transcription_corrected.resized_segments, output_prefix, audio_filepath)
|
||||
outputs.ass = self.subtitle.generate_ass(transcription_corrected.resized_segments, output_prefix, audio_filepath, chords=chords)
|
||||
|
||||
# Generate preview video (unless ass_only mode for GCE encoding)
|
||||
if not ass_only:
|
||||
@@ -225,7 +228,7 @@ class OutputGenerator:
|
||||
# Generate video if requested
|
||||
if self.config.render_video:
|
||||
# Generate ASS subtitles
|
||||
outputs.ass = self.subtitle.generate_ass(resized_segments, output_prefix, audio_filepath)
|
||||
outputs.ass = self.subtitle.generate_ass(resized_segments, output_prefix, audio_filepath, chords=chords)
|
||||
outputs.video = self.video.generate_video(outputs.ass, audio_filepath, output_prefix)
|
||||
|
||||
return outputs
|
||||
|
||||
@@ -13,12 +13,27 @@ from karaoke_gen.lyrics_transcriber.output.ass.style import Style
|
||||
from karaoke_gen.lyrics_transcriber.output.ass.constants import ALIGN_TOP_CENTER
|
||||
from karaoke_gen.lyrics_transcriber.output.ass import LyricsScreen
|
||||
from karaoke_gen.lyrics_transcriber.output.ass.section_detector import SectionDetector
|
||||
from karaoke_gen.lyrics_transcriber.output.ass.config import ScreenConfig, translation_layout
|
||||
from karaoke_gen.lyrics_transcriber.output.ass.config import ScreenConfig, chord_layout, translation_layout
|
||||
from karaoke_gen.lyrics_transcriber.output.ass.chords import ChordChange, assign_chords_to_segments, clean_chords
|
||||
from karaoke_gen.lyrics_transcriber.output.ass.event import Event
|
||||
|
||||
|
||||
# Translation rows render at this fraction of the lyric colour's opacity, so they read as
|
||||
# secondary to the sung line (Andrew: identical white made it unclear which to read).
|
||||
TRANSLATION_OPACITY = 0.7
|
||||
# Guitar chords: upcoming chord colour (warm gold) unless the theme sets chord_color
|
||||
DEFAULT_CHORD_COLOR = (255, 223, 107, 255)
|
||||
# "Now / next" chord box in the top-right corner, as fractions of the frame
|
||||
CHORD_BOX_RIGHT = 0.955
|
||||
CHORD_BOX_TOP = 0.045
|
||||
CHORD_BOX_SIZE_RATIO = 1.25 # box chord size / chord-row size
|
||||
CHORD_BOX_NEXT_RATIO = 0.55 # "next" chord size / current chord size
|
||||
|
||||
|
||||
def _ass_colour(color) -> str:
|
||||
"""(r, g, b[, a]) -> ASS inline colour &HBBGGRR&."""
|
||||
r, g, b = color[:3]
|
||||
return f"&H{b:02X}{g:02X}{r:02X}&"
|
||||
|
||||
|
||||
def _fade(color, opacity: float):
|
||||
@@ -58,6 +73,7 @@ class SubtitlesGenerator:
|
||||
self.styles = styles
|
||||
self.is_duet = is_duet
|
||||
self.subtitle_offset_ms = subtitle_offset_ms
|
||||
self.chords: List[ChordChange] = []
|
||||
|
||||
# Create ScreenConfig with potential overrides from styles
|
||||
karaoke_styles = styles.get("karaoke", {})
|
||||
@@ -143,8 +159,23 @@ class SubtitlesGenerator:
|
||||
order = [1, 2, 0]
|
||||
return [sid for sid in order if sid in found]
|
||||
|
||||
def generate_ass(self, segments: List[LyricsSegment], output_prefix: str, audio_filepath: Optional[str]) -> str:
|
||||
def generate_ass(
|
||||
self,
|
||||
segments: List[LyricsSegment],
|
||||
output_prefix: str,
|
||||
audio_filepath: Optional[str],
|
||||
chords: Optional[List[ChordChange]] = None,
|
||||
) -> str:
|
||||
"""Write the karaoke ASS file.
|
||||
|
||||
``chords``: optional guitar chord changes for the whole song. When given, each
|
||||
line gets a chord row above it and a "now / next" chord box is shown throughout.
|
||||
"""
|
||||
self.logger.info("Generating ASS format subtitles")
|
||||
self.chords = clean_chords(chords or [])
|
||||
if self.chords and self.subtitle_offset_ms:
|
||||
off = self.subtitle_offset_ms / 1000.0
|
||||
self.chords = [ChordChange(c.start + off, c.end + off, c.label) for c in self.chords]
|
||||
output_path = self._get_output_path(f"{output_prefix} (Karaoke)", "ass")
|
||||
|
||||
try:
|
||||
@@ -152,7 +183,9 @@ class SubtitlesGenerator:
|
||||
song_duration = self._get_audio_duration(audio_filepath, segments)
|
||||
|
||||
base_config = self.config
|
||||
if any(seg.translation for seg in segments):
|
||||
if self.chords:
|
||||
self.config = self._chord_config(base_config)
|
||||
elif any(seg.translation for seg in segments):
|
||||
self.config = self._translation_config(base_config)
|
||||
try:
|
||||
screens = self._create_screens(segments, song_duration)
|
||||
@@ -189,6 +222,62 @@ class SubtitlesGenerator:
|
||||
)
|
||||
return config
|
||||
|
||||
def _chord_config(self, base: ScreenConfig) -> ScreenConfig:
|
||||
"""Copy of ``base`` with taller line slots that fit a chord row above each line."""
|
||||
layout = chord_layout(self.font_size, base.line_height, self.styles.get("karaoke", {}))
|
||||
config = copy.copy(base)
|
||||
config.line_height = layout.slot_height
|
||||
config.lyric_line_height = layout.lyric_line_height
|
||||
config.lyric_y_offset = layout.lyric_y_offset
|
||||
config.chord_font_size = layout.chord_font_size
|
||||
config.max_visible_lines = min(base.max_visible_lines, layout.max_visible_lines)
|
||||
self.logger.info(
|
||||
f"Guitar chords: {len(self.chords)} changes, {config.max_visible_lines} lines/screen, "
|
||||
f"slot {layout.slot_height}px, chord font {layout.chord_font_size}px"
|
||||
)
|
||||
return config
|
||||
|
||||
def _build_chord_style(self, primary: Style, karaoke_styles: dict, name: str, size: int) -> Style:
|
||||
"""Chord symbols: upcoming = chord colour (gold), played = the sung lyric colour."""
|
||||
style = copy.copy(primary)
|
||||
style.Name = name
|
||||
style.Fontsize = size
|
||||
color = karaoke_styles.get("chord_color")
|
||||
upcoming = tuple(int(x.strip()) for x in str(color).split(",")) if color else DEFAULT_CHORD_COLOR
|
||||
if len(upcoming) == 3:
|
||||
upcoming = (*upcoming, 255)
|
||||
style.SecondaryColour = upcoming
|
||||
style.PrimaryColour = primary.PrimaryColour
|
||||
ratio = size / max(1, self.font_size)
|
||||
if primary.Outline:
|
||||
style.Outline = max(1, int(round(primary.Outline * ratio)))
|
||||
style.Alignment = 7 # top-left: chords are positioned by their left edge
|
||||
return style
|
||||
|
||||
def _chord_box_events(self, box_style: Style) -> List[Event]:
|
||||
"""Top-right "now / next" chord box for the whole song (covers intros and breaks)."""
|
||||
width, height = self.video_resolution
|
||||
x, y = int(width * CHORD_BOX_RIGHT), int(height * CHORD_BOX_TOP)
|
||||
next_size = max(1, int(round(box_style.Fontsize * CHORD_BOX_NEXT_RATIO)))
|
||||
now_colour = _ass_colour(box_style.SecondaryColour)
|
||||
next_colour = _ass_colour(_fade(box_style.SecondaryColour, TRANSLATION_OPACITY))
|
||||
events = []
|
||||
for i, chord in enumerate(self.chords):
|
||||
nxt = self.chords[i + 1] if i + 1 < len(self.chords) else None
|
||||
event = Event()
|
||||
event.type = "Dialogue"
|
||||
event.Layer = 2
|
||||
event.Style = box_style
|
||||
event.Start = chord.start
|
||||
# Hold the chord through short no-chord gaps so the box doesn't flicker
|
||||
event.End = nxt.start if nxt and nxt.start - chord.end < 1.0 else chord.end
|
||||
text = f"{{\\an9}}{{\\pos({x},{y})}}{{\\1c{now_colour}}}{chord.label}"
|
||||
if nxt:
|
||||
text += f"\\N{{\\fs{next_size}}}{{\\1c{next_colour}}}next: {nxt.label}"
|
||||
event.Text = text
|
||||
events.append(event)
|
||||
return events
|
||||
|
||||
def _build_translation_style(self, primary: Style, karaoke_styles: dict) -> Style:
|
||||
"""Static style for translation rows: the unsung lyric colour, faded and smaller.
|
||||
|
||||
@@ -284,6 +373,7 @@ class SubtitlesGenerator:
|
||||
"""Create regular lyric screens, handling instrumental boundaries."""
|
||||
screens: List[LyricsScreen] = []
|
||||
current_screen: Optional[LyricsScreen] = None
|
||||
chords_by_segment = assign_chords_to_segments(segments, getattr(self, "chords", None) or [])
|
||||
|
||||
for i, segment in enumerate(segments):
|
||||
self.logger.debug(f"Processing segment {i}: {segment.start_time:.2f}s - {segment.end_time:.2f}s")
|
||||
@@ -306,7 +396,12 @@ class SubtitlesGenerator:
|
||||
self.logger.debug(" Created new screen")
|
||||
|
||||
# Add line to current screen
|
||||
line = LyricsLine(logger=self.logger, segment=segment, screen_config=self.config)
|
||||
line = LyricsLine(
|
||||
logger=self.logger,
|
||||
segment=segment,
|
||||
screen_config=self.config,
|
||||
chords=chords_by_segment.get(segment.id, []),
|
||||
)
|
||||
current_screen.lines.append(line)
|
||||
self.logger.debug(f" Added line to screen (now has {len(current_screen.lines)} lines)")
|
||||
|
||||
@@ -419,6 +514,16 @@ class SubtitlesGenerator:
|
||||
a.add_style(translation_style)
|
||||
self.config.translation_style = translation_style
|
||||
|
||||
if self.config.chord_font_size:
|
||||
chord_style = self._build_chord_style(style_list[0], karaoke_styles, "Karaoke.Chord", self.config.chord_font_size)
|
||||
a.add_style(chord_style)
|
||||
self.config.chord_style = chord_style
|
||||
box_size = int(round(self.config.chord_font_size * CHORD_BOX_SIZE_RATIO))
|
||||
box_style = self._build_chord_style(style_list[0], karaoke_styles, "Karaoke.ChordBox", box_size)
|
||||
box_style.Alignment = 9
|
||||
a.add_style(box_style)
|
||||
self._chord_box_style = box_style
|
||||
|
||||
a.events_format = ["Layer", "Style", "Start", "End", "MarginV", "Text"]
|
||||
# Primary (fallback) style is the first one (singer 1 for duet, ass_name for solo)
|
||||
primary_style = style_list[0]
|
||||
@@ -463,4 +568,8 @@ class SubtitlesGenerator:
|
||||
for event in events:
|
||||
ass_file.add(event)
|
||||
|
||||
if getattr(self.config, "chord_style", None) is not None:
|
||||
for event in self._chord_box_events(self._chord_box_style):
|
||||
ass_file.add(event)
|
||||
|
||||
return ass_file
|
||||
@@ -0,0 +1,215 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Detect guitar chords for one song: an ensemble of ChordMini models over several stem mixes.
|
||||
|
||||
python scripts/medley/detect_chords.py --chordmini ~/ChordMini \\
|
||||
--input mix=song.flac --input inst=instrumental_clean.flac --input harm=harmonic.flac \\
|
||||
--out chords.json
|
||||
|
||||
Run it with ChordMini's own Python (it needs torch + librosa + mir_eval); see
|
||||
docs/MEDLEYS.md § Guitar chords for setup and why this ensemble.
|
||||
|
||||
Pipeline:
|
||||
1. Both ChordMini checkpoints (BTC and 2E1D, continual-learning versions; the best open
|
||||
chord models we found, ~80% MIREX/majmin) on every input -> .lab files.
|
||||
2. Each label simplified to a guitar chord (root + one of: major, m, 7, maj7, m7, sus2,
|
||||
sus4, dim, aug, m7b5, dim7), then a weighted frame-by-frame vote across all runs.
|
||||
3. Changes shorter than MIN_CHORD_SECONDS merged away; boundaries snapped to the
|
||||
nearest beat when one is within BEAT_SNAP_SECONDS.
|
||||
4. Roots spelled for the local key (Ab/Db/Eb7 in A-flat, not G#/C#/D#7).
|
||||
|
||||
Output: JSON list of {"start", "end", "label"} (seconds, no "no chord" entries).
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from collections import defaultdict
|
||||
|
||||
import numpy as np
|
||||
|
||||
FRAME = 0.05
|
||||
MIN_CHORD_SECONDS = 0.6
|
||||
BEAT_SNAP_SECONDS = 0.2
|
||||
KEY_WINDOW_SECONDS = 20.0
|
||||
MODELS = {"BTC": ("btc_model_best.pth", 1.1), "ChordNet": ("2e1d_model_best.pth", 1.0)}
|
||||
INPUT_WEIGHTS = {"inst": 1.2} # vocals removed = cleaner harmony; everything else 1.0
|
||||
|
||||
PITCH = {"C": 0, "C#": 1, "Db": 1, "D": 2, "D#": 3, "Eb": 3, "E": 4, "Fb": 4, "E#": 5, "F": 5, "F#": 6,
|
||||
"Gb": 6, "G": 7, "G#": 8, "Ab": 8, "A": 9, "A#": 10, "Bb": 10, "B": 11, "Cb": 11, "B#": 0}
|
||||
SHARP_NAMES = ["C", "C#", "D", "D#", "E", "F", "F#", "G", "G#", "A", "A#", "B"]
|
||||
FLAT_NAMES = ["C", "Db", "D", "Eb", "E", "F", "Gb", "G", "Ab", "A", "Bb", "B"]
|
||||
FLAT_KEYS = {1, 3, 5, 6, 8, 10} # Db Eb F Gb Ab Bb major
|
||||
QUALITY = {"maj": "", "": "", "min": "m", "7": "7", "maj7": "maj7", "min7": "m7", "sus4": "sus4",
|
||||
"sus2": "sus2", "dim": "dim", "aug": "aug", "maj6": "", "min6": "m", "dim7": "dim7",
|
||||
"hdim7": "m7b5", "minmaj7": "m", "9": "7", "maj9": "maj7", "min9": "m7", "11": "7", "13": "7"}
|
||||
# Scale degree (semitones above the key) -> qualities that are diatonic there in a major key
|
||||
DIATONIC = {0: {"", "maj7"}, 2: {"m", "m7"}, 4: {"m", "m7"}, 5: {"", "maj7"}, 7: {"", "7"}, 9: {"m", "m7"},
|
||||
11: {"dim", "m7b5"}}
|
||||
|
||||
|
||||
def simplify(label: str):
|
||||
"""'G#:7' -> (8, '7'); 'N'/'X' -> None. Drops inversions ('C/E' -> C)."""
|
||||
label = label.strip()
|
||||
if label in ("N", "X", ""):
|
||||
return None
|
||||
root, _, qual = label.partition(":")
|
||||
qual = qual.split("/")[0].split("(")[0]
|
||||
root = root.split("/")[0]
|
||||
if root not in PITCH:
|
||||
return None
|
||||
return PITCH[root], QUALITY.get(qual, "")
|
||||
|
||||
|
||||
def spell(pc: int, qual: str, key: int) -> str:
|
||||
names = FLAT_NAMES if key in FLAT_KEYS else SHARP_NAMES
|
||||
name = names[pc]
|
||||
# Borrowed bIII / bVI / bVII in sharp keys read better as flats (Eb, Ab, Bb in C)
|
||||
if key not in FLAT_KEYS and pc in ((key + 3) % 12, (key + 8) % 12, (key + 10) % 12) and len(name) > 1:
|
||||
name = FLAT_NAMES[pc]
|
||||
return name + qual
|
||||
|
||||
|
||||
def estimate_key(chords, t0: float, t1: float) -> int:
|
||||
"""Major key whose diatonic chords cover the most time in [t0, t1]."""
|
||||
scores = np.zeros(12)
|
||||
for start, end, (pc, qual) in chords:
|
||||
overlap = min(end, t1) - max(start, t0)
|
||||
if overlap <= 0:
|
||||
continue
|
||||
for k in range(12):
|
||||
if qual in DIATONIC.get((pc - k) % 12, ()):
|
||||
scores[k] += overlap * (1.5 if (pc - k) % 12 in (0, 7) else 1.0)
|
||||
return int(np.argmax(scores))
|
||||
|
||||
|
||||
def read_lab(path):
|
||||
rows = []
|
||||
with open(path) as f:
|
||||
for line in f:
|
||||
parts = line.split()
|
||||
if len(parts) >= 3:
|
||||
rows.append((float(parts[0]), float(parts[1]), " ".join(parts[2:])))
|
||||
return rows
|
||||
|
||||
|
||||
def vote(labs, duration: float):
|
||||
"""labs: list of (weight, rows). Returns per-frame winning (pc, qual) or None."""
|
||||
n = int(np.ceil(duration / FRAME))
|
||||
tallies = [defaultdict(float) for _ in range(n)]
|
||||
for weight, rows in labs:
|
||||
for start, end, label in rows:
|
||||
chord = simplify(label)
|
||||
for i in range(int(start / FRAME), min(n, int(np.ceil(end / FRAME)))):
|
||||
tallies[i][chord] += weight
|
||||
return [max(t.items(), key=lambda kv: kv[1])[0] if t else None for t in tallies]
|
||||
|
||||
|
||||
def frames_to_segments(frames):
|
||||
segs, start = [], 0
|
||||
for i in range(1, len(frames) + 1):
|
||||
if i == len(frames) or frames[i] != frames[start]:
|
||||
segs.append([start * FRAME, i * FRAME, frames[start]])
|
||||
start = i
|
||||
return segs
|
||||
|
||||
|
||||
def merge_short(segs, min_len: float = MIN_CHORD_SECONDS):
|
||||
"""Fold segments shorter than min_len into the longer neighbour, until none remain."""
|
||||
segs = [s[:] for s in segs]
|
||||
while True:
|
||||
short = [i for i, s in enumerate(segs) if s[1] - s[0] < min_len]
|
||||
if not short or len(segs) == 1:
|
||||
break
|
||||
i = min(short, key=lambda j: segs[j][1] - segs[j][0])
|
||||
prev_len = segs[i - 1][1] - segs[i - 1][0] if i > 0 else -1
|
||||
next_len = segs[i + 1][1] - segs[i + 1][0] if i + 1 < len(segs) else -1
|
||||
if prev_len >= next_len:
|
||||
segs[i - 1][1] = segs[i][1]
|
||||
else:
|
||||
segs[i + 1][0] = segs[i][0]
|
||||
del segs[i]
|
||||
# Re-join neighbours that now carry the same chord
|
||||
j = 0
|
||||
while j + 1 < len(segs):
|
||||
if segs[j][2] == segs[j + 1][2]:
|
||||
segs[j][1] = segs[j + 1][1]
|
||||
del segs[j + 1]
|
||||
else:
|
||||
j += 1
|
||||
return segs
|
||||
|
||||
|
||||
def snap_to_beats(segs, beats, tolerance: float = BEAT_SNAP_SECONDS):
|
||||
if len(beats) == 0:
|
||||
return segs
|
||||
beats = np.asarray(beats)
|
||||
for i in range(1, len(segs)):
|
||||
b = beats[np.argmin(np.abs(beats - segs[i][0]))]
|
||||
if abs(b - segs[i][0]) <= tolerance and segs[i - 1][0] < b < segs[i][1]:
|
||||
segs[i - 1][1] = segs[i][0] = float(b)
|
||||
return segs
|
||||
|
||||
|
||||
def label_segments(segs):
|
||||
"""Spell each chord for the key around it; drop no-chord segments."""
|
||||
chords = [(s, e, c) for s, e, c in segs if c is not None]
|
||||
out = []
|
||||
for start, end, (pc, qual) in chords:
|
||||
mid = (start + end) / 2
|
||||
key = estimate_key(chords, mid - KEY_WINDOW_SECONDS / 2, mid + KEY_WINDOW_SECONDS / 2)
|
||||
out.append({"start": round(start, 3), "end": round(end, 3), "label": spell(pc, qual, key)})
|
||||
return out
|
||||
|
||||
|
||||
def run_chordmini(chordmini: str, audio: str, model: str, out_dir: str) -> str:
|
||||
ckpt, _ = MODELS[model]
|
||||
in_dir = tempfile.mkdtemp(prefix="cm_in_")
|
||||
link = os.path.join(in_dir, "song" + os.path.splitext(audio)[1])
|
||||
os.symlink(os.path.abspath(audio), link)
|
||||
args = [sys.executable, "src/evaluation/test.py", "--model_type", model, "--checkpoint", f"checkpoints/{ckpt}",
|
||||
"--config", "config/ChordMini.yaml", "--audio_dir", in_dir, "--save_dir", out_dir, "--use_overlap",
|
||||
"--vote_aggregation", "logit", "--min_segment_duration", "0.5", "--smooth_predictions",
|
||||
"--use_gaussian", "--kernel_size", "9"]
|
||||
if model == "BTC":
|
||||
args.append("--smooth_logits")
|
||||
subprocess.run(args, cwd=chordmini, check=True, capture_output=True)
|
||||
return os.path.join(out_dir, "song.lab")
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--chordmini", required=True, help="ChordMini checkout (github.com/ptnghia-j/ChordMini)")
|
||||
p.add_argument("--input", action="append", required=True, help="name=path; e.g. mix=song.flac inst=inst.flac")
|
||||
p.add_argument("--beats-from", help="audio to beat-track (default: the 'mix' input)")
|
||||
p.add_argument("--out", required=True)
|
||||
args = p.parse_args(argv)
|
||||
import librosa
|
||||
import scipy.signal
|
||||
|
||||
if not hasattr(scipy.signal, "hann"): # librosa 0.10.1 beat tracking on scipy >= 1.13
|
||||
scipy.signal.hann = scipy.signal.windows.hann
|
||||
|
||||
inputs = dict(i.split("=", 1) for i in args.input)
|
||||
labs, duration = [], 0.0
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
for name, path in inputs.items():
|
||||
duration = max(duration, librosa.get_duration(path=path))
|
||||
for model, (_, model_weight) in MODELS.items():
|
||||
lab = run_chordmini(args.chordmini, path, model, os.path.join(tmp, f"{name}_{model}"))
|
||||
labs.append((model_weight * INPUT_WEIGHTS.get(name, 1.0), read_lab(lab)))
|
||||
segs = merge_short(frames_to_segments(vote(labs, duration)))
|
||||
beat_src = args.beats_from or inputs.get("mix") or next(iter(inputs.values()))
|
||||
y, sr = librosa.load(beat_src, sr=22050, mono=True)
|
||||
_, beat_frames = librosa.beat.beat_track(y=y, sr=sr)
|
||||
segs = snap_to_beats(segs, librosa.frames_to_time(beat_frames, sr=sr))
|
||||
chords = label_segments(segs)
|
||||
with open(args.out, "w") as f:
|
||||
json.dump(chords, f, indent=1)
|
||||
print(f"{len(chords)} chord changes -> {args.out}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -13,6 +13,8 @@ Usage (from the repo root, in an env where karaoke_gen imports):
|
||||
python scripts/medley/make_medley.py fetch spec.json # Firestore + GCS -> <workdir>/<job_id>/
|
||||
python scripts/medley/make_medley.py audio spec.json # medley WAV/MP3 per audio variant
|
||||
python scripts/medley/make_medley.py video spec.json [--resolution 4k] # karaoke + guide MP4s
|
||||
python scripts/medley/make_medley.py chords spec.json # optional: detect guitar chords per song
|
||||
# (needs MEDLEY_CHORDMINI; see docs)
|
||||
|
||||
Requires Firestore read access (ADC; the claude-readonly SA is enough), the gcloud CLI for GCS
|
||||
downloads (set MEDLEY_GCLOUD_ACCOUNT to pick an account, e.g. the read-only SA when your user
|
||||
@@ -122,6 +124,34 @@ def shift_segments(segments, start: float, end: Optional[float], offset: float,
|
||||
return out
|
||||
|
||||
|
||||
def shift_chords(chords: list[dict], src_start: float, src_end: float, offset: float,
|
||||
trim_in: float = 0.0, trim_out: float = 0.0) -> list[dict]:
|
||||
"""Chords of one section on the medley timeline.
|
||||
|
||||
Clipped to [src_start + trim_in, src_end - trim_out] (half of each crossfade, so the
|
||||
outgoing song's chords hand over to the incoming song's mid-fade) and moved by offset.
|
||||
"""
|
||||
lo, hi = src_start + trim_in, src_end - trim_out
|
||||
out = []
|
||||
for c in chords:
|
||||
start, end = max(c["start"], lo), min(c["end"], hi)
|
||||
if end - start > 0.05:
|
||||
out.append({"start": start + offset, "end": end + offset, "label": c["label"]})
|
||||
return out
|
||||
|
||||
|
||||
def medley_chords(sections: list[Section], durations: list[float], offsets: list[float],
|
||||
chords_by_job: dict) -> list[dict]:
|
||||
out = []
|
||||
for i, (sec, dur, off) in enumerate(zip(sections, durations, offsets)):
|
||||
trim_in = sec.join_duration / 2 if i > 0 and sec.join == "crossfade" else 0.0
|
||||
nxt = sections[i + 1] if i + 1 < len(sections) else None
|
||||
trim_out = nxt.join_duration / 2 if nxt is not None and nxt.join == "crossfade" else 0.0
|
||||
out += shift_chords(chords_by_job.get(sec.job_id, []), sec.start, sec.start + dur, off - sec.start,
|
||||
trim_in, trim_out)
|
||||
return out
|
||||
|
||||
|
||||
def build_filtergraph(sections: list[Section], durations: list[float], gains_db: list[float]) -> str:
|
||||
"""ffmpeg filter_complex joining inputs 0..n-1 (one per section, already trimmed) into [out]."""
|
||||
chains = []
|
||||
@@ -266,6 +296,55 @@ def cmd_audio(spec: dict, workdir: str, outdir: str):
|
||||
"medley_start": o, "medley_end": o + d} for s, d, o in zip(sections, durations, offsets)], f, indent=1)
|
||||
|
||||
|
||||
def cmd_chords(spec: dict, workdir: str):
|
||||
"""Detect chords for each job with scripts/medley/detect_chords.py (ChordMini ensemble).
|
||||
|
||||
MEDLEY_CHORDMINI = a ChordMini checkout with its own .venv (see docs/MEDLEYS.md).
|
||||
Harmonic stems (guitar+piano+bass+other via htdemucs_6s) are added when the
|
||||
audio-separator CLI is available; otherwise just the full mix + clean instrumental.
|
||||
"""
|
||||
chordmini = os.environ.get("MEDLEY_CHORDMINI")
|
||||
if not chordmini or not os.path.isdir(chordmini):
|
||||
raise SystemExit("set MEDLEY_CHORDMINI to a ChordMini checkout (see docs/MEDLEYS.md § Guitar chords)")
|
||||
python = os.path.join(chordmini, ".venv", "bin", "python")
|
||||
detect = os.path.join(os.path.dirname(os.path.abspath(__file__)), "detect_chords.py")
|
||||
for job_id in dict.fromkeys(s["job_id"] for s in spec["sections"]):
|
||||
d = job_dir(workdir, job_id)
|
||||
inputs = [f"mix={variant_path(workdir, job_id, GUIDE_VARIANT)}",
|
||||
f"inst={variant_path(workdir, job_id, 'instrumental_clean')}"]
|
||||
harm = _harmonic_stem(d, variant_path(workdir, job_id, GUIDE_VARIANT))
|
||||
if harm:
|
||||
inputs.append(f"harm={harm}")
|
||||
args = [python, detect, "--chordmini", chordmini, "--out", os.path.join(d, "chords.json")]
|
||||
for i in inputs:
|
||||
args += ["--input", i]
|
||||
_run(args)
|
||||
|
||||
|
||||
def _harmonic_stem(d: str, source: str) -> Optional[str]:
|
||||
"""guitar+piano+bass+other from htdemucs_6s (drums and vocals removed), cached per job."""
|
||||
out = os.path.join(d, "harmonic.flac")
|
||||
if os.path.exists(out):
|
||||
return out
|
||||
if subprocess.run(["which", "audio-separator"], capture_output=True).returncode != 0:
|
||||
log.info("audio-separator not on PATH: skipping harmonic stems")
|
||||
return None
|
||||
stems_dir = os.path.join(d, "stems6")
|
||||
os.makedirs(stems_dir, exist_ok=True)
|
||||
_run(["audio-separator", "-m", "htdemucs_6s.yaml", "--output_format", "FLAC", "--output_dir", stems_dir, source],
|
||||
capture_output=True)
|
||||
parts = [os.path.join(stems_dir, f) for f in sorted(os.listdir(stems_dir))
|
||||
if any(f"({n})" in f for n in ("Guitar", "Piano", "Bass", "Other"))]
|
||||
if len(parts) != 4:
|
||||
log.warning("unexpected htdemucs_6s output in %s", stems_dir)
|
||||
return None
|
||||
args = ["ffmpeg", "-hide_banner", "-loglevel", "error", "-y"]
|
||||
for part in parts:
|
||||
args += ["-i", part]
|
||||
_run(args + ["-filter_complex", "amix=inputs=4:normalize=0", out])
|
||||
return out
|
||||
|
||||
|
||||
def _load_reviewed_lyrics(workdir: str, job_id: str):
|
||||
from karaoke_gen.lyrics_transcriber.correction.operations import CorrectionOperations
|
||||
from karaoke_gen.lyrics_transcriber.types import CorrectionResult
|
||||
@@ -320,6 +399,20 @@ def cmd_video(spec: dict, workdir: str, outdir: str, resolution: str):
|
||||
medley, audio, _, pad = CountdownProcessor(cache_dir=cache_dir, logger=log).process(
|
||||
correction_result=medley, audio_filepath=karaoke_wav)
|
||||
|
||||
chords = None
|
||||
if spec.get("chords"):
|
||||
from karaoke_gen.lyrics_transcriber.output.ass.chords import ChordChange
|
||||
by_job = {}
|
||||
for job_id in dict.fromkeys(s.job_id for s in sections):
|
||||
path = os.path.join(job_dir(workdir, job_id), "chords.json")
|
||||
if not os.path.exists(path):
|
||||
raise SystemExit(f"no chords for {job_id}: run the `chords` step first")
|
||||
with open(path) as f:
|
||||
by_job[job_id] = json.load(f)
|
||||
chords = [ChordChange(c["start"] + pad, c["end"] + pad, c["label"])
|
||||
for c in medley_chords(sections, durations, offsets, by_job)]
|
||||
log.info("guitar chords: %d changes", len(chords))
|
||||
|
||||
artist, title = spec["artist"], spec["title"]
|
||||
vg = VideoGenerator(logger=log, ffmpeg_base_command="ffmpeg -hide_banner -loglevel error -nostats -y",
|
||||
render_bounding_boxes=False, output_png=True, output_jpg=False)
|
||||
@@ -333,7 +426,7 @@ def cmd_video(spec: dict, workdir: str, outdir: str, resolution: str):
|
||||
generate_cdg=False, generate_plain_text=True, generate_lrc=True, video_resolution=resolution)
|
||||
outputs = OutputGenerator(cfg, log).generate_outputs(
|
||||
transcription_corrected=medley, lyrics_results={}, output_prefix=title, audio_filepath=audio,
|
||||
artist=artist, title=title)
|
||||
artist=artist, title=title, chords=chords)
|
||||
|
||||
w, h = {"4k": (3840, 2160), "1080p": (1920, 1080), "720p": (1280, 720), "360p": (640, 360)}[resolution]
|
||||
intro_s, end_s = style["intro"]["video_duration"], style["end"]["video_duration"]
|
||||
@@ -359,7 +452,7 @@ def cmd_video(spec: dict, workdir: str, outdir: str, resolution: str):
|
||||
|
||||
def main(argv=None):
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("command", choices=["fetch", "audio", "video"])
|
||||
p.add_argument("command", choices=["fetch", "audio", "video", "chords"])
|
||||
p.add_argument("spec")
|
||||
p.add_argument("--resolution", default="4k", choices=["4k", "1080p", "720p", "360p"])
|
||||
args = p.parse_args(argv)
|
||||
@@ -372,6 +465,8 @@ def main(argv=None):
|
||||
outdir = os.path.join(base, spec.get("outdir", "out"))
|
||||
if args.command == "fetch":
|
||||
cmd_fetch(spec, workdir)
|
||||
elif args.command == "chords":
|
||||
cmd_chords(spec, workdir)
|
||||
elif args.command == "audio":
|
||||
cmd_audio(spec, workdir, outdir)
|
||||
else:
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
"""Guitar chords: chord symbols above the words they land on + a now/next chord box."""
|
||||
import os
|
||||
import re
|
||||
|
||||
from PIL import ImageFont
|
||||
|
||||
from karaoke_gen.lyrics_transcriber.output.ass.chords import (
|
||||
ChordChange,
|
||||
assign_chords_to_segments,
|
||||
chord_at,
|
||||
clean_chords,
|
||||
place_chords,
|
||||
word_index_for_time,
|
||||
)
|
||||
from karaoke_gen.lyrics_transcriber.output.ass.config import chord_layout
|
||||
from karaoke_gen.lyrics_transcriber.output.subtitles import SubtitlesGenerator
|
||||
from karaoke_gen.lyrics_transcriber.types import LyricsSegment, Word
|
||||
from karaoke_gen.style_loader import DEFAULT_KARAOKE_STYLE
|
||||
|
||||
FONT_PATH = os.path.abspath(os.path.join(
|
||||
os.path.dirname(__file__), "..", "..", "..", "..", "karaoke_gen", "resources", "AvenirNext-Bold.ttf"
|
||||
))
|
||||
W, H = 1920, 1080
|
||||
FONT = 120
|
||||
|
||||
|
||||
def seg(text, start, seg_id=None, step=0.5):
|
||||
words = [
|
||||
Word(id=f"{seg_id or start}-{i}", text=w, start_time=start + i * step, end_time=start + (i + 1) * step - 0.05)
|
||||
for i, w in enumerate(text.split())
|
||||
]
|
||||
return LyricsSegment(id=seg_id or f"s{start}", text=text, words=words,
|
||||
start_time=words[0].start_time, end_time=words[-1].end_time)
|
||||
|
||||
|
||||
def generate(tmp_path, segments, chords=None, karaoke_overrides=None):
|
||||
karaoke = dict(DEFAULT_KARAOKE_STYLE)
|
||||
karaoke["font_path"] = FONT_PATH
|
||||
karaoke.update(karaoke_overrides or {})
|
||||
gen = SubtitlesGenerator(output_dir=str(tmp_path), video_resolution=(W, H), font_size=FONT,
|
||||
line_height=FONT, styles={"karaoke": karaoke})
|
||||
path = gen.generate_ass(segments, "test", None, chords=chords)
|
||||
with open(path, encoding="utf-8") as f:
|
||||
return f.read()
|
||||
|
||||
|
||||
def dialogues(ass_text, style=None):
|
||||
lines = [line for line in ass_text.splitlines() if line.startswith("Dialogue:")]
|
||||
return [d for d in lines if style is None or f",{style}," in d]
|
||||
|
||||
|
||||
def pos(d):
|
||||
return tuple(int(v) for v in re.search(r"\\pos\((\d+),(\d+)\)", d).groups())
|
||||
|
||||
|
||||
# ---- chord timeline helpers ----
|
||||
|
||||
|
||||
def test_clean_chords_drops_no_chord_and_merges_repeats():
|
||||
chords = clean_chords([
|
||||
ChordChange(4, 6, "C"), ChordChange(0, 2, "G"), ChordChange(2, 4, "G"),
|
||||
ChordChange(6, 7, "N"), ChordChange(7, 9, "D7"),
|
||||
])
|
||||
assert [(c.start, c.end, c.label) for c in chords] == [(0, 4, "G"), (4, 6, "C"), (7, 9, "D7")]
|
||||
|
||||
|
||||
def test_word_index_snaps_a_pickup_strum_to_the_next_word():
|
||||
s = seg("one two three", 10.0) # words at 10.0, 10.5, 11.0
|
||||
assert word_index_for_time(s, 10.3) == 1 # 0.2s before "two" -> over "two"
|
||||
assert word_index_for_time(s, 10.2) == 0
|
||||
assert word_index_for_time(s, 5.0) == 0 # before the line -> first word
|
||||
assert word_index_for_time(s, 20.0) == 2
|
||||
|
||||
|
||||
def test_assign_puts_changes_on_the_line_being_sung():
|
||||
a, b = seg("first line here", 10.0, "a"), seg("second line", 20.0, "b")
|
||||
chords = [ChordChange(9.8, 10.9, "G"), ChordChange(10.9, 15.0, "C"),
|
||||
ChordChange(15.0, 20.4, "D"), ChordChange(20.4, 30, "G")]
|
||||
by_seg = assign_chords_to_segments([a, b], chords)
|
||||
assert [c.label for c in by_seg["a"]] == ["G", "C"]
|
||||
# D started between the lines and is still ringing when "second" is sung: repeated over it
|
||||
assert [c.label for c in by_seg["b"]] == ["D", "G"]
|
||||
|
||||
|
||||
def test_assign_skips_the_ringing_chord_when_a_change_lands_on_the_first_word():
|
||||
a = seg("only line", 10.0, "a")
|
||||
by_seg = assign_chords_to_segments([a], [ChordChange(2, 10.1, "D"), ChordChange(10.1, 12, "G")])
|
||||
assert [c.label for c in by_seg["a"]] == ["G"]
|
||||
|
||||
|
||||
def test_place_chords_never_overlaps_quick_changes():
|
||||
s = seg("hello world", 0.0)
|
||||
measure = lambda text: 10 * len(text) # 10px per character
|
||||
placed = place_chords(s, [ChordChange(0.0, 0.1, "G"), ChordChange(0.1, 0.4, "Cmaj7"),
|
||||
ChordChange(0.5, 1, "D")], ["hello", "world"], 100, measure, measure, 5)
|
||||
xs = [p.x for p in placed]
|
||||
assert xs[0] == 100 # over "hello"
|
||||
assert xs[1] == 100 + 10 + 5 # pushed right of "G" (same word)
|
||||
# "world" starts at 160, but Cmaj7 runs to 165 (+5 spacing): pushed clear of it
|
||||
assert [p.word_index for p in placed] == [0, 0, 1]
|
||||
assert xs[2] == xs[1] + measure("Cmaj7") + 5
|
||||
|
||||
|
||||
def test_chord_at():
|
||||
chords = [ChordChange(0, 2, "G"), ChordChange(2, 4, "C")]
|
||||
assert chord_at(chords, 2.5).label == "C"
|
||||
assert chord_at(chords, 9) is None
|
||||
|
||||
|
||||
def test_chord_layout_reserves_a_row_above_the_line():
|
||||
layout = chord_layout(120, 120)
|
||||
assert layout.chord_font_size == 66
|
||||
assert layout.lyric_y_offset == round(66 * 1.05)
|
||||
assert layout.slot_height == 120 + layout.lyric_y_offset
|
||||
assert chord_layout(120, 120, {"chord_font_size": 50, "chord_max_visible_lines": 3}).max_visible_lines == 3
|
||||
|
||||
|
||||
# ---- rendered ASS ----
|
||||
|
||||
|
||||
def test_no_chords_leaves_the_classic_layout_untouched(tmp_path):
|
||||
segments = [seg("hello there world", 5.0, "a")]
|
||||
assert generate(tmp_path, segments) == generate(tmp_path, segments, chords=[])
|
||||
|
||||
|
||||
def test_chord_symbols_sit_above_their_words(tmp_path):
|
||||
segments = [seg("hello there world", 5.0, "a")]
|
||||
chords = [ChordChange(0, 5.5, "G"), ChordChange(5.5, 9, "C")]
|
||||
ass = generate(tmp_path, segments, chords)
|
||||
assert "Style: Karaoke.Chord," in ass and "Style: Karaoke.ChordBox," in ass
|
||||
lyric = next(d for d in dialogues(ass) if "\\kf" in d and ",Karaoke.Chord" not in d)
|
||||
chord_rows = dialogues(ass, "Karaoke.Chord")
|
||||
assert [re.sub(r"\{[^}]*\}", "", d.split(",", 5)[-1]) for d in chord_rows] == ["G", "C"]
|
||||
lyric_x, lyric_y = pos(lyric)
|
||||
(g_x, g_y), (c_x, c_y) = pos(chord_rows[0]), pos(chord_rows[1])
|
||||
assert g_y == c_y < lyric_y # chord row above the line
|
||||
assert lyric_y - g_y == chord_layout(FONT, FONT).lyric_y_offset
|
||||
# G sits on the centred line's left edge, C where "there" starts. Compare as a
|
||||
# fraction of the line width: the renderer scales PIL metrics to match libass.
|
||||
rendered_width = 2 * (lyric_x - g_x)
|
||||
font = ImageFont.truetype(FONT_PATH, FONT)
|
||||
expected = font.getlength("hello ") / font.getlength("hello there world")
|
||||
assert abs((c_x - g_x) / rendered_width - expected) < 0.01
|
||||
|
||||
|
||||
def test_chord_lights_up_when_played(tmp_path):
|
||||
segments = [seg("hello there world", 5.0, "a")]
|
||||
ass = generate(tmp_path, segments, [ChordChange(5.5, 9, "C")])
|
||||
row = dialogues(ass, "Karaoke.Chord")[0]
|
||||
start = row.split(",")[2] # H:MM:SS.cc
|
||||
h, m, sec = start.split(":")
|
||||
start_s = int(h) * 3600 + int(m) * 60 + float(sec)
|
||||
wait = int(re.search(r"\\k(\d+)\}\{\\kf", row).group(1))
|
||||
assert abs(start_s + wait / 100 - 5.5) < 0.02
|
||||
|
||||
|
||||
def test_chord_box_shows_now_and_next(tmp_path):
|
||||
segments = [seg("hello there world", 5.0, "a")]
|
||||
ass = generate(tmp_path, segments, [ChordChange(0, 5.5, "G"), ChordChange(5.5, 9, "C")])
|
||||
box = dialogues(ass, "Karaoke.ChordBox")
|
||||
assert len(box) == 2
|
||||
assert box[0].endswith("next: C") and "}G\\N" in box[0]
|
||||
assert "next:" not in box[1] # last chord
|
||||
|
||||
|
||||
def test_chord_colour_comes_from_theme(tmp_path):
|
||||
segments = [seg("hello there world", 5.0, "a")]
|
||||
ass = generate(tmp_path, segments, [ChordChange(5.5, 9, "C")], {"chord_color": "10, 20, 30"})
|
||||
style = next(line for line in ass.splitlines() if line.startswith("Style: Karaoke.Chord,"))
|
||||
assert "&H001E140A" in style # secondary (upcoming) colour = theme chord_color (BGR)
|
||||
@@ -0,0 +1,64 @@
|
||||
"""Unit tests for the chord-detection ensemble's pure helpers (scripts/medley/detect_chords.py)."""
|
||||
|
||||
import importlib.util
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
_SCRIPT = Path(__file__).resolve().parents[2] / "scripts" / "medley" / "detect_chords.py"
|
||||
_spec = importlib.util.spec_from_file_location("detect_chords", _SCRIPT)
|
||||
dc = importlib.util.module_from_spec(_spec)
|
||||
sys.modules[_spec.name] = dc
|
||||
_spec.loader.exec_module(dc)
|
||||
|
||||
|
||||
def test_simplify_maps_to_guitar_chords():
|
||||
assert dc.simplify("G#:7") == (8, "7")
|
||||
assert dc.simplify("C:maj/3") == (0, "")
|
||||
assert dc.simplify("A:min7") == (9, "m7")
|
||||
assert dc.simplify("D:hdim7") == (2, "m7b5")
|
||||
assert dc.simplify("Bb:min6") == (10, "m")
|
||||
assert dc.simplify("N") is None and dc.simplify("X") is None
|
||||
|
||||
|
||||
def test_spelling_follows_the_key():
|
||||
ab, g, c = 8, 7, 0
|
||||
assert dc.spell(8, "", ab) == "Ab" and dc.spell(1, "", ab) == "Db" and dc.spell(3, "7", ab) == "Eb7"
|
||||
assert dc.spell(6, "m", 2) == "F#m" # F#m in D
|
||||
assert dc.spell(10, "", c) == "Bb" # borrowed bVII in C stays flat
|
||||
assert dc.spell(10, "", g) == "Bb" and dc.spell(3, "", g) == "Eb" # bIII and bVI in G
|
||||
assert dc.spell(1, "", 9) == "C#" # C# in A
|
||||
|
||||
|
||||
def test_estimate_key_picks_the_diatonic_home():
|
||||
i_iv_v_in_ab = [(0, 10, (8, "")), (10, 15, (1, "")), (15, 20, (3, "7"))]
|
||||
assert dc.estimate_key(i_iv_v_in_ab, 0, 20) == 8
|
||||
in_d = [(0, 4, (2, "")), (4, 6, (6, "m7")), (6, 8, (7, "")), (8, 10, (9, ""))]
|
||||
assert dc.estimate_key(in_d, 0, 10) == 2
|
||||
|
||||
|
||||
def test_vote_weights_and_ties():
|
||||
labs = [
|
||||
(1.0, [(0.0, 1.0, "G"), (1.0, 2.0, "C")]),
|
||||
(1.0, [(0.0, 1.0, "G"), (1.0, 2.0, "A:min")]),
|
||||
(1.5, [(0.0, 2.0, "G")]),
|
||||
]
|
||||
frames = dc.vote(labs, 2.0)
|
||||
assert frames[0] == (7, "")
|
||||
assert frames[-1] == (7, "") # G 1.5 beats C 1.0 and Am 1.0
|
||||
|
||||
|
||||
def test_merge_short_folds_blips_into_neighbours():
|
||||
segs = [[0, 4, "G"], [4, 4.2, "C"], [4.2, 8, "G"], [8, 12, "D"]]
|
||||
assert dc.merge_short(segs) == [[0, 8, "G"], [8, 12, "D"]]
|
||||
|
||||
|
||||
def test_snap_to_beats_moves_boundaries_within_tolerance_only():
|
||||
segs = [[0, 4.1, "G"], [4.1, 8.5, "C"], [8.5, 12, "D"]]
|
||||
out = dc.snap_to_beats([s[:] for s in segs], [2.0, 4.0, 6.0, 8.0, 10.0])
|
||||
assert out[0][1] == out[1][0] == 4.0
|
||||
assert out[1][1] == 8.5 # 0.5s from the nearest beat: left alone
|
||||
|
||||
|
||||
def test_label_segments_spells_and_drops_no_chord():
|
||||
segs = [[0, 10, (8, "")], [10, 12, None], [12, 20, (1, "")], [20, 26, (3, "7")]]
|
||||
assert [c["label"] for c in dc.label_segments(segs)] == ["Ab", "Db", "Eb7"]
|
||||
@@ -107,3 +107,42 @@ class TestBuildFiltergraph:
|
||||
assert "afade=t=in:st=0:d=0.500" in graph # pot in on the finale
|
||||
assert "[x3][s4]concat=n=2:v=0:a=1[x4]" in graph
|
||||
assert graph.endswith("[x4]loudnorm=I=-14.0:TP=-1.0:LRA=11,aresample=44100[out]")
|
||||
|
||||
|
||||
class TestMedleyChords:
|
||||
def test_chords_follow_their_section_and_hand_over_mid_crossfade(self):
|
||||
sections = medley.parse_spec(EXAMPLE)
|
||||
durations = [54.0, 52.0, 57.0, 82.0, 66.9]
|
||||
offsets = medley.compute_offsets(sections, durations)
|
||||
by_job = {
|
||||
"a": [{"start": 0.0, "end": 60.0, "label": "G"}, {"start": 130.0, "end": 200.0, "label": "Ab"}],
|
||||
"b": [{"start": 0.0, "end": 52.0, "label": "C"}],
|
||||
"c": [{"start": 0.0, "end": 57.0, "label": "D"}],
|
||||
"d": [{"start": 0.0, "end": 82.0, "label": "F"}],
|
||||
}
|
||||
out = medley.medley_chords(sections, durations, offsets, by_job)
|
||||
labels = [(round(c["start"], 2), round(c["end"], 2), c["label"]) for c in out]
|
||||
# opener G cut where the 2s crossfade is half done (54 - 1), C starts there
|
||||
assert labels[0] == (0.0, 53.0, "G")
|
||||
assert labels[1] == (53.0, 103.0, "C")
|
||||
assert labels[2] == (103.0, 158.0, "D")
|
||||
# F runs to the end of its section (the finale joins with a gap, not a crossfade)
|
||||
assert labels[3] == (158.0, 239.0, "F")
|
||||
# the finale's chords start at the finale (source 132.2 -> medley 239.7)
|
||||
assert labels[4][0] == pytest.approx(239.7) and labels[4][2] == "Ab"
|
||||
|
||||
def test_shift_chords_drops_slivers(self):
|
||||
out = medley.shift_chords([{"start": 0, "end": 10.02, "label": "G"}, {"start": 10.02, "end": 20, "label": "C"}],
|
||||
10.0, 20.0, 100.0)
|
||||
assert [c["label"] for c in out] == ["C"]
|
||||
|
||||
|
||||
class TestSingleSong:
|
||||
def test_one_section_spec_builds_a_plain_graph_and_full_chords(self):
|
||||
sections = medley.parse_spec({"sections": [{"job_id": "a", "start": 0, "end": None}]})
|
||||
assert medley.compute_offsets(sections, [200.0]) == [0.0]
|
||||
graph = medley.build_filtergraph(sections, [200.0], [1.5])
|
||||
assert "acrossfade" not in graph and "concat" not in graph
|
||||
assert graph.endswith("[s0]loudnorm=I=-14.0:TP=-1.0:LRA=11,aresample=44100[out]")
|
||||
chords = medley.medley_chords(sections, [200.0], [0.0], {"a": [{"start": 0, "end": 200, "label": "G"}]})
|
||||
assert chords == [{"start": 0.0, "end": 200.0, "label": "G"}]
|
||||
Reference in new issue
Block a user