docs: medley-making guide + spec-driven make_medley script (#1174)

## Summary
Documents the multi-song medley process used for the first
classic-country medley, and turns it into a reusable script so the next
one only needs a spec file.

- `docs/MEDLEYS.md`: the full process. One gen job per original song,
then identical cuts on every audio variant, and the reviewed lyrics
shifted onto the medley timeline and rendered with gen's own
`OutputGenerator`. Also covers how to choose cut points (crossfades,
pot-out/pot-in, keys/tempos, checking the brief against the recording)
and gotchas.
- `scripts/medley/make_medley.py`: `fetch` (Firestore + gcloud CLI) →
`audio` (per-section LUFS gain, crossfade/gap/pot fades, `loudnorm` to
-14 LUFS / -1 dBTP, three variants) → `video` (shifted reviewed lyrics +
countdown + themed title/end screens + karaoke and guide MP4s).
- `scripts/medley/example-spec.json`: the country medley as a worked
example.
- Linked from `CLAUDE.md` Quick Reference and `docs/README.md`.

Ops tooling only. No backend or frontend change, so no version bump.

## Testing
- `tests/unit/test_make_medley.py`: 12 unit tests covering spec
validation, the timeline offset maths, lyric-segment shifting (including
the same song used twice) and the ffmpeg filtergraph.
- End-to-end run against the real prod jobs: `fetch` → `audio`
reproduces the hand-built v1 medley **bit-for-bit** (max sample diff 0.0
on both the instrumental and guide WAVs). `video --resolution 360p`
renders both MP4s and the LRC.
- Local CodeRabbit review: 1 finding (unchecked ffmpeg exit code in
`measure_lufs`), fixed.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 5.5 authored and GitHub committed 2026-10-16 19:44:27 -04:00
1 parent bb55efd3c2
commit 85390fd12b
6 files changed
+688

No files matched your search

+1
View File
@@ -23,6 +23,7 @@
| GDrive validator & gaps | `docs/GDRIVE-VALIDATOR.md` |
| **Product communication** | `docs/PRODUCT-COMMUNICATION-GUIDE.md` |
| Brand style guide | `docs/BRAND-STYLE-GUIDE.md` |
| **Making medleys** (multi-song karaoke) | `docs/MEDLEYS.md` + `scripts/medley/make_medley.py` |
## Tech Stack
+175
View File
@@ -0,0 +1,175 @@
# Making Karaoke Medleys
How to turn several songs into one continuous medley: a backing track plus a synced karaoke
video in the normal gen look. It's a manual, agent-run process for now. A self-serve
version is on the backlog.
First done in October 2026 for a 5-section classic country medley. Total time was about an
hour, mostly waiting on gen jobs. That medley is the worked example throughout this doc and
in `scripts/medley/example-spec.json`.
## The idea in one paragraph
**Don't** build the medley first and push it through gen as a new job. Gen would re-transcribe
it with no reference lyrics to check against, and someone would have to review it from scratch.
Instead, make **one ordinary gen job per original song**. Each job gives us the lossless source,
sample-aligned stems and reviewed word-level lyrics. Then the medley is just two things:
1. **Audio:** apply the same cuts, gains and fades to every variant (instrumental with backing
vocals, clean instrumental, original mix). Stems are sample-aligned with their source, so all
variants line up exactly.
2. **Video:** take each job's *reviewed* lyric lines that fall inside its cut, shift them by that
section's medley offset, and render locally with gen's own `OutputGenerator`. That gives the
same theme, countdown and title/end screens as a normal track.
Bonus: the per-song jobs are full karaoke tracks too. They can be published as community
versions or given to the customer as extras.
## Step by step
### 1. Make a gen job for each original song
One job per distinct song. A song used twice, like an opener that comes back as the finale,
needs only one job.
- Bulk submit works (`POST /api/bulk/submit` with the admin token; see `docs/API.md`
§ Bulk Mode). Expect jobs to stop at `awaiting_audio_selection`, then pick the source by hand
with `GET /api/audio-search/{id}/results` + `POST /api/audio-search/{id}/select`.
- **Pick the original studio recording.** Avoid live versions, duets, remasters and "greatest
hits" sets that might be re-recordings. The customer knows the original arrangement, and the
cut points assume it. Album compilations from the original label (e.g. an "Essential" or
"Songs of" collection) are usually the original masters.
- **Prefer CD FLAC torrent results.** They're lossless. Admin jobs get a
`402 Insufficient credits` on Spotify results, because the admin account has 0 credits and
Spotify results carry a duration-based charge; torrent results don't.
- Review the lyrics properly. These reviewed timings are what the medley video uses. Timing
matters more than usual, because every line will be used.
### 2. Write the spec and fetch the jobs
Copy `scripts/medley/example-spec.json` and edit it. Each section names a job, a source
`start`/`end` (seconds; `end: null` = to the end of the song) and how it joins the previous
section:
| Field | Meaning |
|---|---|
| `join: "crossfade"` + `join_duration` | Overlap the previous section's tail (default 2.0s, equal-power) |
| `join: "gap"` + `join_duration` | Insert this much silence instead |
| `fade_out_start` | "Pot out": fader drops from this source second to the section `end` |
| `fade_in` | "Pot in": fade-in length at the start of the section |
| `theme_job_id` (top level) | Whose theme to render with (default: the first section's job) |
```bash
# from a karaoke-gen checkout, in an env where karaoke_gen imports
export PYTHONPATH=$PWD
export MEDLEY_GCLOUD_ACCOUNT=claude-readonly@nomadkaraoke.iam.gserviceaccount.com # optional
python scripts/medley/make_medley.py fetch path/to/spec.json
```
`fetch` reads each job from Firestore and downloads the source audio, stems, `corrections.json`
and `corrections_updated.json` into `<spec dir>/work/<job_id>/`. It uses the `gcloud` CLI,
because the Python GCS client stalled for about an hour on large FLACs. Stems are deleted 30
days after a job completes (`docs/STORAGE-RETENTION.md`), so build medleys from recent jobs or
regenerate the stems first.
### 3. Find the cut points
This is where the craft is (see [Making it sound good](#making-it-sound-good)). Use data
rather than guesswork:
- **Word timings** from `corrections.json` / the job's `karaoke.lrc`: where lines start and end.
- **Lead-vocal activity** from `stems/lead_vocals.flac`. RMS in 100ms frames shows exactly when
a held note dies away and when the next line starts. Use it to check the lyric timings and to
place fades. In the example, "hand" starts at 80.66s and is held until 82.1s, so the pot-out
runs 80.7 → 82.0.
- **Key and tempo:** `librosa` (`beat.beat_track` + chroma key-profile matching) is good enough
to confirm a brief's BPM/key claims and spot modulations. librosa often reports double tempo,
so halve it when it looks too high.
### 4. Build the audio
```bash
python scripts/medley/make_medley.py audio path/to/spec.json
```
This writes `out/medley_{instrumental_with_backing,instrumental_clean,guide_with_vocals}.{wav,mp3}`
and `out/timeline.json` (each section's source and medley times).
- Each section's gain is set so the section itself measures -18 LUFS. That's measured on the
section, not the whole song, which matters for quiet intros. The full medley is then run
through `loudnorm` to -14 LUFS with peaks no higher than -1 dBTP. In the example, all five
sections ended up within 1.6 LU of each other.
- WAV is 24-bit / 44.1 kHz. MP3 is 320k.
- **Listen to the guide (`guide_with_vocals`) first.** Bad transitions are much easier to hear
with vocals in.
### 5. Render the video
```bash
python scripts/medley/make_medley.py video path/to/spec.json --resolution 360p # quick check
python scripts/medley/make_medley.py video path/to/spec.json # 4K final
```
This writes `out/<title> (Karaoke <res>).mp4` (instrumental with backing vocals),
`out/<title> (With Vocals <res>).mp4` (same picture over the original-vocals medley, for
checking sync) and `out/<title> (Karaoke).lrc`. A 4K render of a 5-minute medley takes a few
minutes on an M-series Mac.
What it does:
- Loads each job's reviewed lyrics the same way the render worker does: `CorrectionResult` +
`CorrectionOperations.update_correction_result_with_data`.
- Keeps the lines that **start** inside each cut and shifts them by
`medley_offset - source_start`. IDs get a section prefix so a song can appear twice.
- Runs `CountdownProcessor`, which adds gen's "3… 2… 1…" plus 3s of padding if the first word
starts within 3s, and `VideoGenerator` title/end PNGs with the spec's title/artist.
- Renders with `OutputGenerator`, then joins title (5s) + karaoke + end (5s) with ffmpeg.
**Check sync:** the vocals in the guide video should sit exactly *title + countdown* seconds
(8.0s in the example) after their position in the medley WAV. Cross-correlating a 10s window
confirms it in seconds, and spot-checking a few frames confirms the right line is on screen.
### 6. Deliver
Give the customer the audio file they asked for (usually `medley_instrumental_with_backing.wav`
or `.mp3`; offer `instrumental_clean` as an alternative) plus the karaoke video. The per-song
gen jobs are a free extra: four full karaoke tracks.
## Making it sound good
- **Cut where the next vocal starts, not where the last one ends.** A section should run until
just before the next line of its song, so the crossfade uses the instrumental tail (the last
chord ringing out) rather than chopping the band mid-phrase.
- **Use the next song's intro as the bridge.** Starting the next section at 0:00 lets its
intro (guitar pickup, fiddle intro) play under the crossfade, and the singer gets a few bars
to find the new key.
- **Different keys and tempos are fine.** Medleys jump keys all the time (the example goes
G → C → D → F → G/A♭). A 2s equal-power crossfade on a sustained chord sounds like a
transition. Don't time-stretch to match tempos. It sounds digital, and briefs often ask you
not to.
- **A pot-out/pot-in is the strongest transition.** A fast fader drop (about 1.3s) on a
held word, a short silence (0.5–1s), then a quick fade-in (about 0.5s) on a distinctive
instrumental phrase. It's the natural way to reset tempo or mood before a finale.
- **Check the brief against the recording.** Customers describe the arrangement from memory
or from a karaoke track. In the example, the brief asked for an "instrumental intro → verse 1
→ chorus 1" on a song whose original starts singing at 0.4s and has no chorus until the third
verse. Measure, then confirm the interpretation with the customer.
- **Gain-match per section, then normalise once.** Don't run `loudnorm` on each section
separately (it pumps); set static per-section gains, then process the whole medley once.
## Gotchas
- Pre-review lyric timings are fine for *choosing* cut points (check them against the
lead-vocal stem). Only the render needs the reviewed `corrections_updated.json`.
- Lines are kept by **start** time. A line that starts just before a cut belongs to the
previous section, so place cuts in gaps between lines.
- The audio file has no title card or countdown; it starts straight into the song. Only the
video gets the 5s title and 3s countdown. Mention this when delivering both.
- Bulk auto-select left every job (8 of 8 across two batches) at `awaiting_audio_selection`,
even with exact lossless matches. Plan to pick sources by hand.
## Future: self-serve medleys
Tracked in the workspace `BACKLOG.md` ("Self-serve medley builder in gen"). The building blocks
already exist: per-song jobs, sample-aligned stems, reviewed lyrics, `OutputGenerator`. A
product version would mainly need a UI for picking sections (a waveform with lyric lines to
snap cut points to) and a backend job type that runs the `audio` + `video` steps.
+3
View File
@@ -11,6 +11,9 @@
(MusicBrainz lookup + KaraokeNerds availability); auto-picks confident
lossless audio, parks the rest in a review queue. See `docs/API.md` § Bulk Mode.
- `/admin` - Admin dashboard (user management, job monitoring)
- **Medleys** (agent-run, local): one gen job per original song, then
`scripts/medley/make_medley.py` cuts the audio and renders a synced karaoke video from the
reviewed lyrics. See [MEDLEYS.md](MEDLEYS.md).
- **Backend API**: https://api.nomadkaraoke.com - FastAPI on Cloud Run
- **CLI Tools**: `karaoke-gen` (local), `karaoke-gen-remote` (cloud)
+18
View File
@@ -0,0 +1,18 @@
{
"title": "Classic Country Medley",
"artist": "George Jones, Kris Kristofferson, George Strait & Randy Travis",
"theme_job_id": "a70794e2",
"workdir": "work",
"outdir": "out",
"sections": [
{"label": "He Stopped Loving Her Today (verse 1)", "job_id": "a70794e2", "start": 0.0, "end": 54.0},
{"label": "Help Me Make It Through the Night (intro, verse 1, chorus)", "job_id": "0ac008b5", "start": 0.0, "end": 52.0,
"join": "crossfade", "join_duration": 2.0},
{"label": "Amarillo by Morning (fiddle intro, verse 1, chorus)", "job_id": "4c90a9a2", "start": 0.0, "end": 57.0,
"join": "crossfade", "join_duration": 2.0},
{"label": "On the Other Hand (verse 1, chorus, pot out on 'hand')", "job_id": "6a9ce5e2", "start": 0.0, "end": 82.0,
"join": "crossfade", "join_duration": 2.0, "fade_out_start": 80.7},
{"label": "He Stopped Loving Her Today (fiddle bridge to end)", "job_id": "a70794e2", "start": 132.2, "end": null,
"join": "gap", "join_duration": 0.7, "fade_in": 0.5}
]
}
+382
View File
@@ -0,0 +1,382 @@
#!/usr/bin/env python3
"""
Build a karaoke medley (audio + synced karaoke video) from completed gen jobs.
See docs/MEDLEYS.md for the full process and the craft of choosing cut points.
Every section of the medley is cut from a gen job for the ORIGINAL song. Gen already gave
us the lossless source, sample-aligned stems and reviewed word-level lyrics for each one,
so the medley is just: the same cuts applied to every audio variant, plus the reviewed
lyric segments shifted onto the medley timeline and rendered with gen's own renderer.
Usage (from the repo root, in an env where karaoke_gen imports):
python scripts/medley/make_medley.py fetch spec.json # Firestore + GCS -> <workdir>/<job_id>/
python scripts/medley/make_medley.py audio spec.json # medley WAV/MP3 per audio variant
python scripts/medley/make_medley.py video spec.json [--resolution 4k] # karaoke + guide MP4s
Requires Firestore read access (ADC; the claude-readonly SA is enough), the gcloud CLI for GCS
downloads (set MEDLEY_GCLOUD_ACCOUNT to pick an account, e.g. the read-only SA when your user
login has expired) and ffmpeg.
See scripts/medley/example-spec.json for the spec format.
"""
import argparse
import dataclasses
import json
import logging
import os
import re
import subprocess
import sys
from dataclasses import dataclass
from typing import Optional
PROJECT = "nomadkaraoke"
BUCKET = "karaoke-gen-storage-nomadkaraoke"
DEFAULT_CROSSFADE = 2.0
SEGMENT_TARGET_LUFS = -18.0
FINAL_LUFS = -14.0
FINAL_TRUE_PEAK = -1.0
# Audio variants built by `audio`: name -> stem file (None = the job's original input audio)
VARIANTS = {
"instrumental_with_backing": "instrumental_with_backing.flac",
"instrumental_clean": "instrumental_clean.flac",
"guide_with_vocals": None,
}
KARAOKE_VARIANT = "instrumental_with_backing"
GUIDE_VARIANT = "guide_with_vocals"
log = logging.getLogger("medley")
# --- Spec + timeline (pure, unit-tested) -------------------------------------------------
@dataclass
class Section:
job_id: str
start: float # source seconds
end: Optional[float] # source seconds; None = to the end of the song
join: str = "crossfade" # how this section joins the PREVIOUS one: crossfade | gap
join_duration: float = DEFAULT_CROSSFADE # crossfade length, or silence length for a gap
fade_in: float = 0.0 # "pot in": fade-in length at the start of this section
fade_out_start: Optional[float] = None # "pot out": source second the fader starts dropping
label: str = ""
def parse_spec(spec: dict) -> list[Section]:
sections = []
for i, s in enumerate(spec["sections"]):
join = s.get("join", "crossfade")
if join not in ("crossfade", "gap"):
raise ValueError(f"section {i}: join must be 'crossfade' or 'gap', got {join!r}")
sec = Section(
job_id=s["job_id"], start=float(s.get("start", 0.0)),
end=None if s.get("end") is None else float(s["end"]),
join=join,
join_duration=float(s.get("join_duration", DEFAULT_CROSSFADE if join == "crossfade" else 0.5)),
fade_in=float(s.get("fade_in", 0.0)),
fade_out_start=None if s.get("fade_out_start") is None else float(s["fade_out_start"]),
label=s.get("label", ""),
)
if sec.end is not None and sec.end <= sec.start:
raise ValueError(f"section {i}: end must be after start")
if sec.fade_out_start is not None and sec.end is not None and not sec.start < sec.fade_out_start < sec.end:
raise ValueError(f"section {i}: fade_out_start must fall inside the section")
sections.append(sec)
if not sections:
raise ValueError("spec has no sections")
return sections
def compute_offsets(sections: list[Section], durations: list[float]) -> list[float]:
"""Medley start time of each section. durations[i] = section length in seconds.
A crossfade overlaps the previous section's tail by join_duration; a gap inserts
join_duration seconds of silence. Offsets are what the lyric shift adds to source times.
"""
offsets, cursor = [], 0.0
for i, (sec, dur) in enumerate(zip(sections, durations)):
if i == 0:
start = 0.0
elif sec.join == "crossfade":
start = cursor - sec.join_duration
else:
start = cursor + sec.join_duration
offsets.append(start)
cursor = start + dur
return offsets
def shift_segments(segments, start: float, end: Optional[float], offset: float, prefix: str, tolerance: float = 0.05):
"""Keep lyric lines that START inside [start, end) and move them by offset.
IDs are prefixed so the same song can appear twice (e.g. opener + finale).
"""
out = []
for seg in segments:
if seg.start_time >= start - tolerance and (end is None or seg.start_time < end):
words = [dataclasses.replace(w, id=f"{prefix}_{w.id}", start_time=w.start_time + offset,
end_time=w.end_time + offset) for w in seg.words]
out.append(dataclasses.replace(seg, id=f"{prefix}_{seg.id}", words=words,
start_time=seg.start_time + offset, end_time=seg.end_time + offset))
return out
def build_filtergraph(sections: list[Section], durations: list[float], gains_db: list[float]) -> str:
"""ffmpeg filter_complex joining inputs 0..n-1 (one per section, already trimmed) into [out]."""
chains = []
for i, (sec, dur, gain) in enumerate(zip(sections, durations, gains_db)):
f = f"[{i}:a]aformat=sample_rates=44100:channel_layouts=stereo,volume={gain:.2f}dB"
if sec.fade_in > 0:
f += f",afade=t=in:st=0:d={sec.fade_in:.3f}:curve=qsin"
if sec.fade_out_start is not None:
st = sec.fade_out_start - sec.start
f += f",afade=t=out:st={st:.3f}:d={dur - st:.3f}:curve=qsin"
nxt = sections[i + 1] if i + 1 < len(sections) else None
if nxt is not None and nxt.join == "gap":
f += f",apad=pad_dur={nxt.join_duration:.3f}"
chains.append(f + f"[s{i}]")
cur = "s0"
for i in range(1, len(sections)):
out = f"x{i}"
if sections[i].join == "crossfade":
chains.append(f"[{cur}][s{i}]acrossfade=d={sections[i].join_duration:.3f}:c1=qsin:c2=qsin[{out}]")
else:
chains.append(f"[{cur}][s{i}]concat=n=2:v=0:a=1[{out}]")
cur = out
chains.append(f"[{cur}]loudnorm=I={FINAL_LUFS}:TP={FINAL_TRUE_PEAK}:LRA=11,aresample=44100[out]")
return ";".join(chains)
# --- I/O helpers ---------------------------------------------------------------------------
def _run(args, **kw):
return subprocess.run(args, check=True, **kw)
def probe_duration(path: str) -> float:
out = _run(["ffprobe", "-v", "error", "-show_entries", "format=duration", "-of", "csv=p=0", path],
capture_output=True, text=True).stdout
return float(out.strip())
def measure_lufs(path: str, start: float, end: Optional[float]) -> float:
args = ["ffmpeg", "-hide_banner", "-nostats", "-ss", str(start)]
if end is not None:
args += ["-to", str(end)]
args += ["-i", path, "-af", "ebur128", "-f", "null", "-"]
res = subprocess.run(args, capture_output=True, text=True)
found = re.findall(r"I:\s+(-?[\d.]+) LUFS", res.stderr)
if res.returncode != 0 or not found:
raise RuntimeError(f"loudness measurement failed for {path}: {res.stderr.strip()[-300:]}")
return float(found[-1])
def gcs_download(src: str, dst: str, required: bool = True) -> bool:
"""Download gs://BUCKET/src via the gcloud CLI (the Python client stalled for ~1h on large FLACs)."""
os.makedirs(os.path.dirname(dst), exist_ok=True)
args = ["gcloud", "storage", "cp", f"gs://{BUCKET}/{src}", dst]
if os.environ.get("MEDLEY_GCLOUD_ACCOUNT"):
args.insert(3, f"--account={os.environ['MEDLEY_GCLOUD_ACCOUNT']}")
result = subprocess.run(args, capture_output=True, text=True)
if result.returncode != 0:
if required:
raise RuntimeError(f"download failed for gs://{BUCKET}/{src}: {result.stderr.strip()[-300:]}")
return False
return True
def job_dir(workdir: str, job_id: str) -> str:
return os.path.join(workdir, job_id)
def load_job_meta(workdir: str, job_id: str) -> dict:
with open(os.path.join(job_dir(workdir, job_id), "job.json")) as f:
return json.load(f)
def variant_path(workdir: str, job_id: str, variant: str) -> str:
d = job_dir(workdir, job_id)
stem = VARIANTS[variant]
if stem is None:
return os.path.join(d, "input", os.path.basename(load_job_meta(workdir, job_id)["input_media_gcs_path"]))
return os.path.join(d, "stems", stem)
def section_durations(sections: list[Section], workdir: str) -> list[float]:
durs = []
for sec in sections:
end = sec.end if sec.end is not None else probe_duration(variant_path(workdir, sec.job_id, GUIDE_VARIANT))
durs.append(end - sec.start)
return durs
# --- Commands ------------------------------------------------------------------------------
def cmd_fetch(spec: dict, workdir: str):
from google.cloud import firestore
db = firestore.Client(project=PROJECT)
for job_id in dict.fromkeys(s["job_id"] for s in spec["sections"]):
doc = db.collection("jobs").document(job_id).get()
if not doc.exists:
raise SystemExit(f"job {job_id} not found")
job = doc.to_dict()
if job.get("status") != "complete":
log.warning("job %s is %s, not complete: lyrics may not be reviewed yet", job_id, job.get("status"))
d = job_dir(workdir, job_id)
os.makedirs(os.path.join(d, "stems"), exist_ok=True)
os.makedirs(os.path.join(d, "input"), exist_ok=True)
meta = {k: job.get(k) for k in ("job_id", "artist", "title", "status", "input_media_gcs_path",
"style_params_gcs_path", "style_assets")}
with open(os.path.join(d, "job.json"), "w") as f:
json.dump(meta, f, indent=1, default=str)
wanted = {job["input_media_gcs_path"]: os.path.join(d, "input", os.path.basename(job["input_media_gcs_path"]))}
for name in ("instrumental_with_backing", "instrumental_clean", "lead_vocals", "backing_vocals"):
wanted[f"jobs/{job_id}/stems/{name}.flac"] = os.path.join(d, "stems", f"{name}.flac")
for name in ("corrections.json", "corrections_updated.json"):
wanted[f"jobs/{job_id}/lyrics/{name}"] = os.path.join(d, name)
for src, dst in wanted.items():
optional = "corrections_updated" in src or src.endswith(("lead_vocals.flac", "backing_vocals.flac"))
if not gcs_download(src, dst, required=not optional) and not optional:
log.warning("missing gs://%s/%s", BUCKET, src)
log.info("fetched %s - %s (%s)", job.get("artist"), job.get("title"), job_id)
def cmd_audio(spec: dict, workdir: str, outdir: str):
sections = parse_spec(spec)
durations = section_durations(sections, workdir)
os.makedirs(outdir, exist_ok=True)
for variant in VARIANTS:
inputs, gains = [], []
for sec in sections:
p = variant_path(workdir, sec.job_id, variant)
gains.append(SEGMENT_TARGET_LUFS - measure_lufs(p, sec.start, sec.end))
inputs += ["-ss", str(sec.start)] + (["-to", str(sec.end)] if sec.end is not None else []) + ["-i", p]
wav = os.path.join(outdir, f"medley_{variant}.wav")
_run(["ffmpeg", "-y", "-hide_banner", "-loglevel", "error"] + inputs +
["-filter_complex", build_filtergraph(sections, durations, gains), "-map", "[out]", "-c:a", "pcm_s24le", wav])
_run(["ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-i", wav, "-c:a", "libmp3lame", "-b:a", "320k",
wav[:-4] + ".mp3"])
log.info("%s: %.1fs", wav, probe_duration(wav))
offsets = compute_offsets(sections, durations)
with open(os.path.join(outdir, "timeline.json"), "w") as f:
json.dump([{"label": s.label, "job_id": s.job_id, "source_start": s.start, "source_end": s.start + d,
"medley_start": o, "medley_end": o + d} for s, d, o in zip(sections, durations, offsets)], f, indent=1)
def _load_reviewed_lyrics(workdir: str, job_id: str):
from karaoke_gen.lyrics_transcriber.correction.operations import CorrectionOperations
from karaoke_gen.lyrics_transcriber.types import CorrectionResult
d = job_dir(workdir, job_id)
with open(os.path.join(d, "corrections.json")) as f:
base = CorrectionResult.from_dict(json.load(f))
updated = os.path.join(d, "corrections_updated.json")
if os.path.exists(updated):
with open(updated) as f:
data = json.load(f)
if isinstance(data, dict) and "corrections" in data:
return CorrectionOperations.update_correction_result_with_data(base, data)
return base
def cmd_video(spec: dict, workdir: str, outdir: str, resolution: str):
from karaoke_gen.lyrics_transcriber.core.config import OutputConfig
from karaoke_gen.lyrics_transcriber.output.countdown_processor import CountdownProcessor
from karaoke_gen.lyrics_transcriber.output.generator import OutputGenerator
from karaoke_gen.style_loader import get_end_format, get_intro_format, load_styles_from_gcs
from karaoke_gen.video_generator import VideoGenerator
sections = parse_spec(spec)
durations = section_durations(sections, workdir)
offsets = compute_offsets(sections, durations)
karaoke_wav = os.path.join(outdir, f"medley_{KARAOKE_VARIANT}.wav")
guide_wav = os.path.join(outdir, f"medley_{GUIDE_VARIANT}.wav")
if not os.path.exists(karaoke_wav):
raise SystemExit("run the `audio` step first")
render_dir = os.path.join(outdir, f"render_{resolution}")
cache_dir = os.path.join(render_dir, "cache")
os.makedirs(cache_dir, exist_ok=True)
lyrics = {j: _load_reviewed_lyrics(workdir, j) for j in dict.fromkeys(s.job_id for s in sections)}
segments = []
for n, (sec, dur, off) in enumerate(zip(sections, durations, offsets)):
segments += shift_segments(lyrics[sec.job_id].corrected_segments, sec.start, sec.start + dur,
off - sec.start, prefix=f"m{n}")
first = lyrics[sections[0].job_id]
medley = dataclasses.replace(first, corrected_segments=segments, resized_segments=[])
for seg in segments:
log.info("%7.2f-%7.2f %s", seg.start_time, seg.end_time, seg.text)
# Theme: the style of the job named in the spec (default: first section's job)
theme_job = load_job_meta(workdir, spec.get("theme_job_id", sections[0].job_id))
styles_path, _ = load_styles_from_gcs(theme_job["style_params_gcs_path"], theme_job["style_assets"],
os.path.join(render_dir, "style"), gcs_download, log)
with open(styles_path) as f:
style = json.load(f)
# Same countdown gen adds when the first word starts within 3s (pads audio + shifts lyrics)
medley, audio, _, pad = CountdownProcessor(cache_dir=cache_dir, logger=log).process(
correction_result=medley, audio_filepath=karaoke_wav)
artist, title = spec["artist"], spec["title"]
vg = VideoGenerator(logger=log, ffmpeg_base_command="ffmpeg -hide_banner -loglevel error -nostats -y",
render_bounding_boxes=False, output_png=True, output_jpg=False)
title_png, end_png = os.path.join(render_dir, "title"), os.path.join(render_dir, "end")
vg.create_title_video(artist=artist, title=title, format=get_intro_format(style), output_image_filepath_noext=title_png,
output_video_filepath=title_png + ".mov", existing_title_image=None, intro_video_duration=0)
vg.create_end_video(artist=artist, title=title, format=get_end_format(style), output_image_filepath_noext=end_png,
output_video_filepath=end_png + ".mov", existing_end_image=None, end_video_duration=0)
cfg = OutputConfig(output_dir=render_dir, cache_dir=cache_dir, output_styles_json=styles_path, render_video=True,
generate_cdg=False, generate_plain_text=True, generate_lrc=True, video_resolution=resolution)
outputs = OutputGenerator(cfg, log).generate_outputs(
transcription_corrected=medley, lyrics_results={}, output_prefix=title, audio_filepath=audio,
artist=artist, title=title)
w, h = {"4k": (3840, 2160), "1080p": (1920, 1080), "720p": (1280, 720), "360p": (640, 360)}[resolution]
intro_s, end_s = style["intro"]["video_duration"], style["end"]["video_duration"]
master = os.path.join(outdir, f"{title} (Karaoke {resolution}).mp4")
scale = f"scale={w}:{h},fps=30,format=yuv420p,setsar=1"
_run(["ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
"-loop", "1", "-t", str(intro_s), "-i", title_png + ".png", "-f", "lavfi", "-t", str(intro_s), "-i", "anullsrc=r=44100:cl=stereo",
"-i", outputs.video,
"-loop", "1", "-t", str(end_s), "-i", end_png + ".png", "-f", "lavfi", "-t", str(end_s), "-i", "anullsrc=r=44100:cl=stereo",
"-filter_complex", f"[0:v]{scale}[t];[2:v]{scale}[k];[3:v]{scale}[e];[t][k][e]concat=n=3:v=1:a=0[v];[1:a][2:a][4:a]concat=n=3:v=0:a=1[a]",
"-map", "[v]", "-map", "[a]", "-c:v", "libx264", "-preset", "slow", "-crf", "18", "-c:a", "aac", "-b:a", "320k",
"-movflags", "+faststart", master])
# Guide: same picture, original-vocals medley delayed by title + countdown (for checking sync)
delay_ms = int((intro_s + pad) * 1000)
guide = os.path.join(outdir, f"{title} (With Vocals {resolution}).mp4")
_run(["ffmpeg", "-hide_banner", "-loglevel", "error", "-y", "-i", master, "-i", guide_wav,
"-filter_complex", f"[1:a]adelay={delay_ms}|{delay_ms},apad[a]", "-map", "0:v", "-map", "[a]", "-shortest",
"-c:v", "copy", "-c:a", "aac", "-b:a", "320k", "-movflags", "+faststart", guide])
if outputs.lrc:
_run(["cp", outputs.lrc, os.path.join(outdir, f"{title} (Karaoke).lrc")])
log.info("karaoke: %s\nguide: %s", master, guide)
def main(argv=None):
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("command", choices=["fetch", "audio", "video"])
p.add_argument("spec")
p.add_argument("--resolution", default="4k", choices=["4k", "1080p", "720p", "360p"])
args = p.parse_args(argv)
logging.basicConfig(level=logging.INFO, format="%(message)s")
with open(args.spec) as f:
spec = json.load(f)
parse_spec(spec) # validate early
base = os.path.dirname(os.path.abspath(args.spec))
workdir = os.path.join(base, spec.get("workdir", "work"))
outdir = os.path.join(base, spec.get("outdir", "out"))
if args.command == "fetch":
cmd_fetch(spec, workdir)
elif args.command == "audio":
cmd_audio(spec, workdir, outdir)
else:
cmd_video(spec, workdir, outdir, args.resolution)
if __name__ == "__main__":
sys.exit(main())
+109
View File
@@ -0,0 +1,109 @@
"""Unit tests for the medley builder's pure helpers (scripts/medley/make_medley.py).
The script isn't part of the installed package, so we load it by path.
"""
import importlib.util
import sys
from pathlib import Path
import pytest
from karaoke_gen.lyrics_transcriber.types import LyricsSegment, Word
_SCRIPT = Path(__file__).resolve().parents[2] / "scripts" / "medley" / "make_medley.py"
_spec = importlib.util.spec_from_file_location("make_medley", _SCRIPT)
medley = importlib.util.module_from_spec(_spec)
sys.modules[_spec.name] = medley
_spec.loader.exec_module(medley)
EXAMPLE = {
"title": "T", "artist": "A",
"sections": [
{"job_id": "a", "start": 0.0, "end": 54.0},
{"job_id": "b", "start": 0.0, "end": 52.0, "join": "crossfade", "join_duration": 2.0},
{"job_id": "c", "start": 0.0, "end": 57.0},
{"job_id": "d", "start": 0.0, "end": 82.0, "fade_out_start": 80.7},
{"job_id": "a", "start": 132.2, "end": None, "join": "gap", "join_duration": 0.7, "fade_in": 0.5},
],
}
def _segment(seg_id: str, start: float, end: float) -> LyricsSegment:
words = [Word(id=f"{seg_id}w1", text="hello", start_time=start, end_time=(start + end) / 2),
Word(id=f"{seg_id}w2", text="there", start_time=(start + end) / 2, end_time=end)]
return LyricsSegment(id=seg_id, text="hello there", words=words, start_time=start, end_time=end)
class TestParseSpec:
def test_defaults_to_two_second_crossfade(self):
sections = medley.parse_spec(EXAMPLE)
assert sections[2].join == "crossfade"
assert sections[2].join_duration == 2.0
def test_gap_and_pot_fields(self):
sections = medley.parse_spec(EXAMPLE)
assert sections[3].fade_out_start == 80.7
assert sections[4].join == "gap" and sections[4].join_duration == 0.7
assert sections[4].fade_in == 0.5 and sections[4].end is None
@pytest.mark.parametrize("bad", [
{"sections": []},
{"sections": [{"job_id": "a", "start": 10, "end": 5}]},
{"sections": [{"job_id": "a", "start": 0, "end": 10, "join": "splice"}]},
{"sections": [{"job_id": "a", "start": 0, "end": 10, "fade_out_start": 12}]},
])
def test_rejects_invalid_specs(self, bad):
with pytest.raises(ValueError):
medley.parse_spec(bad)
class TestComputeOffsets:
def test_matches_the_country_medley_timeline(self):
sections = medley.parse_spec(EXAMPLE)
offsets = medley.compute_offsets(sections, [54.0, 52.0, 57.0, 82.0, 199.133 - 132.2])
assert offsets == pytest.approx([0.0, 52.0, 102.0, 157.0, 239.7])
def test_gap_inserts_silence_and_crossfade_overlaps(self):
sections = medley.parse_spec({"sections": [
{"job_id": "a", "start": 0, "end": 10},
{"job_id": "b", "start": 0, "end": 10, "join": "gap", "join_duration": 1.0},
{"job_id": "c", "start": 0, "end": 10, "join": "crossfade", "join_duration": 3.0},
]})
assert medley.compute_offsets(sections, [10, 10, 10]) == pytest.approx([0, 11, 18])
class TestShiftSegments:
def test_keeps_lines_starting_inside_the_cut_and_shifts_them(self):
segs = [_segment("s1", 1.0, 3.0), _segment("s2", 50.0, 53.0), _segment("s3", 54.2, 57.0)]
out = medley.shift_segments(segs, start=0.0, end=54.0, offset=100.0, prefix="m0")
assert [s.id for s in out] == ["m0_s1", "m0_s2"]
assert out[1].start_time == pytest.approx(150.0)
assert [w.start_time for w in out[1].words] == pytest.approx([150.0, 151.5])
assert out[1].words[0].id == "m0_s2w1"
def test_open_ended_section_keeps_everything_after_start(self):
segs = [_segment("s1", 120.0, 125.0), _segment("s2", 138.0, 140.0), _segment("s3", 180.0, 184.0)]
out = medley.shift_segments(segs, start=132.2, end=None, offset=107.5, prefix="m4")
assert [s.id for s in out] == ["m4_s2", "m4_s3"]
assert out[0].start_time == pytest.approx(245.5)
def test_same_song_twice_gets_unique_ids(self):
segs = [_segment("s1", 1.0, 3.0)]
a = medley.shift_segments(segs, 0.0, 10.0, 0.0, "m0")
b = medley.shift_segments(segs, 0.0, 10.0, 50.0, "m4")
assert a[0].id != b[0].id
assert segs[0].start_time == 1.0 # originals untouched
class TestBuildFiltergraph:
def test_crossfades_gap_and_pot_fades(self):
sections = medley.parse_spec(EXAMPLE)
graph = medley.build_filtergraph(sections, [54.0, 52.0, 57.0, 82.0, 66.9], [0.0, 1.0, -1.0, 0.5, 2.0])
assert graph.count("acrossfade=d=2.000") == 3
assert "afade=t=out:st=80.700:d=1.300" in graph # pot out on section 4
assert "apad=pad_dur=0.700" in graph # silence before the gap join
assert "afade=t=in:st=0:d=0.500" in graph # pot in on the finale
assert "[x3][s4]concat=n=2:v=0:a=1[x4]" in graph
assert graph.endswith("[x4]loudnorm=I=-14.0:TP=-1.0:LRA=11,aresample=44100[out]")