mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
docs: medley-making guide + spec-driven make_medley script (#1174)
## Summary Documents the multi-song medley process used for the first classic-country medley, and turns it into a reusable script so the next one only needs a spec file. - `docs/MEDLEYS.md`: the full process. One gen job per original song, then identical cuts on every audio variant, and the reviewed lyrics shifted onto the medley timeline and rendered with gen's own `OutputGenerator`. Also covers how to choose cut points (crossfades, pot-out/pot-in, keys/tempos, checking the brief against the recording) and gotchas. - `scripts/medley/make_medley.py`: `fetch` (Firestore + gcloud CLI) → `audio` (per-section LUFS gain, crossfade/gap/pot fades, `loudnorm` to -14 LUFS / -1 dBTP, three variants) → `video` (shifted reviewed lyrics + countdown + themed title/end screens + karaoke and guide MP4s). - `scripts/medley/example-spec.json`: the country medley as a worked example. - Linked from `CLAUDE.md` Quick Reference and `docs/README.md`. Ops tooling only. No backend or frontend change, so no version bump. ## Testing - `tests/unit/test_make_medley.py`: 12 unit tests covering spec validation, the timeline offset maths, lyric-segment shifting (including the same song used twice) and the ffmpeg filtergraph. - End-to-end run against the real prod jobs: `fetch` → `audio` reproduces the hand-built v1 medley **bit-for-bit** (max sample diff 0.0 on both the instrumental and guide WAVs). `video --resolution 360p` renders both MP4s and the LRC. - Local CodeRabbit review: 1 finding (unchecked ffmpeg exit code in `measure_lufs`), fixed. @coderabbitai ignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QV2TQwyC7pwVkGPCk98Bqg --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
bb55efd3c2
commit
85390fd12b
6 files changed
+688
No files matched your search
@@ -23,6 +23,7 @@
|
||||
| GDrive validator & gaps | `docs/GDRIVE-VALIDATOR.md` |
|
||||
| **Product communication** | `docs/PRODUCT-COMMUNICATION-GUIDE.md` |
|
||||
| Brand style guide | `docs/BRAND-STYLE-GUIDE.md` |
|
||||
| **Making medleys** (multi-song karaoke) | `docs/MEDLEYS.md` + `scripts/medley/make_medley.py` |
|
||||
|
||||
## Tech Stack
|
||||
|
||||
|
||||
+175
@@ -0,0 +1,175 @@
|
||||
# Making Karaoke Medleys
|
||||
|
||||
How to turn several songs into one continuous medley: a backing track plus a synced karaoke
|
||||
video in the normal gen look. It's a manual, agent-run process for now. A self-serve
|
||||
version is on the backlog.
|
||||
|
||||
First done in October 2026 for a 5-section classic country medley. Total time was about an
|
||||
hour, mostly waiting on gen jobs. That medley is the worked example throughout this doc and
|
||||
in `scripts/medley/example-spec.json`.
|
||||
|
||||
## The idea in one paragraph
|
||||
|
||||
**Don't** build the medley first and push it through gen as a new job. Gen would re-transcribe
|
||||
it with no reference lyrics to check against, and someone would have to review it from scratch.
|
||||
Instead, make **one ordinary gen job per original song**. Each job gives us the lossless source,
|
||||
sample-aligned stems and reviewed word-level lyrics. Then the medley is just two things:
|
||||
|
||||
1. **Audio:** apply the same cuts, gains and fades to every variant (instrumental with backing
|
||||
vocals, clean instrumental, original mix). Stems are sample-aligned with their source, so all
|
||||
variants line up exactly.
|
||||
2. **Video:** take each job's *reviewed* lyric lines that fall inside its cut, shift them by that
|
||||
section's medley offset, and render locally with gen's own `OutputGenerator`. That gives the
|
||||
same theme, countdown and title/end screens as a normal track.
|
||||
|
||||
Bonus: the per-song jobs are full karaoke tracks too. They can be published as community
|
||||
versions or given to the customer as extras.
|
||||
|
||||
## Step by step
|
||||
|
||||
### 1. Make a gen job for each original song
|
||||
|
||||
One job per distinct song. A song used twice, like an opener that comes back as the finale,
|
||||
needs only one job.
|
||||
|
||||
- Bulk submit works (`POST /api/bulk/submit` with the admin token; see `docs/API.md`
|
||||
§ Bulk Mode). Expect jobs to stop at `awaiting_audio_selection`, then pick the source by hand
|
||||
with `GET /api/audio-search/{id}/results` + `POST /api/audio-search/{id}/select`.
|
||||
- **Pick the original studio recording.** Avoid live versions, duets, remasters and "greatest
|
||||
hits" sets that might be re-recordings. The customer knows the original arrangement, and the
|
||||
cut points assume it. Album compilations from the original label (e.g. an "Essential" or
|
||||
"Songs of" collection) are usually the original masters.
|
||||
- **Prefer CD FLAC torrent results.** They're lossless. Admin jobs get a
|
||||
`402 Insufficient credits` on Spotify results, because the admin account has 0 credits and
|
||||
Spotify results carry a duration-based charge; torrent results don't.
|
||||
- Review the lyrics properly. These reviewed timings are what the medley video uses. Timing
|
||||
matters more than usual, because every line will be used.
|
||||
|
||||
### 2. Write the spec and fetch the jobs
|
||||
|
||||
Copy `scripts/medley/example-spec.json` and edit it. Each section names a job, a source
|
||||
`start`/`end` (seconds; `end: null` = to the end of the song) and how it joins the previous
|
||||
section:
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `join: "crossfade"` + `join_duration` | Overlap the previous section's tail (default 2.0s, equal-power) |
|
||||
| `join: "gap"` + `join_duration` | Insert this much silence instead |
|
||||
| `fade_out_start` | "Pot out": fader drops from this source second to the section `end` |
|
||||
| `fade_in` | "Pot in": fade-in length at the start of the section |
|
||||
| `theme_job_id` (top level) | Whose theme to render with (default: the first section's job) |
|
||||
|
||||
```bash
|
||||
# from a karaoke-gen checkout, in an env where karaoke_gen imports
|
||||
export PYTHONPATH=$PWD
|
||||
export MEDLEY_GCLOUD_ACCOUNT=claude-readonly@nomadkaraoke.iam.gserviceaccount.com # optional
|
||||
python scripts/medley/make_medley.py fetch path/to/spec.json
|
||||
```
|
||||
|
||||
`fetch` reads each job from Firestore and downloads the source audio, stems, `corrections.json`
|
||||
and `corrections_updated.json` into `<spec dir>/work/<job_id>/`. It uses the `gcloud` CLI,
|
||||
because the Python GCS client stalled for about an hour on large FLACs. Stems are deleted 30
|
||||
days after a job completes (`docs/STORAGE-RETENTION.md`), so build medleys from recent jobs or
|
||||
regenerate the stems first.
|
||||
|
||||
### 3. Find the cut points
|
||||
|
||||
This is where the craft is (see [Making it sound good](#making-it-sound-good)). Use data
|
||||
rather than guesswork:
|
||||
|
||||
- **Word timings** from `corrections.json` / the job's `karaoke.lrc`: where lines start and end.
|
||||
- **Lead-vocal activity** from `stems/lead_vocals.flac`. RMS in 100ms frames shows exactly when
|
||||
a held note dies away and when the next line starts. Use it to check the lyric timings and to
|
||||
place fades. In the example, "hand" starts at 80.66s and is held until 82.1s, so the pot-out
|
||||
runs 80.7 → 82.0.
|
||||
- **Key and tempo:** `librosa` (`beat.beat_track` + chroma key-profile matching) is good enough
|
||||
to confirm a brief's BPM/key claims and spot modulations. librosa often reports double tempo,
|
||||
so halve it when it looks too high.
|
||||
|
||||
### 4. Build the audio
|
||||
|
||||
```bash
|
||||
python scripts/medley/make_medley.py audio path/to/spec.json
|
||||
```
|
||||
|
||||
This writes `out/medley_{instrumental_with_backing,instrumental_clean,guide_with_vocals}.{wav,mp3}`
|
||||
and `out/timeline.json` (each section's source and medley times).
|
||||
|
||||
- Each section's gain is set so the section itself measures -18 LUFS. That's measured on the
|
||||
section, not the whole song, which matters for quiet intros. The full medley is then run
|
||||
through `loudnorm` to -14 LUFS with peaks no higher than -1 dBTP. In the example, all five
|
||||
sections ended up within 1.6 LU of each other.
|
||||
- WAV is 24-bit / 44.1 kHz. MP3 is 320k.
|
||||
- **Listen to the guide (`guide_with_vocals`) first.** Bad transitions are much easier to hear
|
||||
with vocals in.
|
||||
|
||||
### 5. Render the video
|
||||
|
||||
```bash
|
||||
python scripts/medley/make_medley.py video path/to/spec.json --resolution 360p # quick check
|
||||
python scripts/medley/make_medley.py video path/to/spec.json # 4K final
|
||||
```
|
||||
|
||||
This writes `out/<title> (Karaoke <res>).mp4` (instrumental with backing vocals),
|
||||
`out/<title> (With Vocals <res>).mp4` (same picture over the original-vocals medley, for
|
||||
checking sync) and `out/<title> (Karaoke).lrc`. A 4K render of a 5-minute medley takes a few
|
||||
minutes on an M-series Mac.
|
||||
|
||||
What it does:
|
||||
- Loads each job's reviewed lyrics the same way the render worker does: `CorrectionResult` +
|
||||
`CorrectionOperations.update_correction_result_with_data`.
|
||||
- Keeps the lines that **start** inside each cut and shifts them by
|
||||
`medley_offset - source_start`. IDs get a section prefix so a song can appear twice.
|
||||
- Runs `CountdownProcessor`, which adds gen's "3… 2… 1…" plus 3s of padding if the first word
|
||||
starts within 3s, and `VideoGenerator` title/end PNGs with the spec's title/artist.
|
||||
- Renders with `OutputGenerator`, then joins title (5s) + karaoke + end (5s) with ffmpeg.
|
||||
|
||||
**Check sync:** the vocals in the guide video should sit exactly *title + countdown* seconds
|
||||
(8.0s in the example) after their position in the medley WAV. Cross-correlating a 10s window
|
||||
confirms it in seconds, and spot-checking a few frames confirms the right line is on screen.
|
||||
|
||||
### 6. Deliver
|
||||
|
||||
Give the customer the audio file they asked for (usually `medley_instrumental_with_backing.wav`
|
||||
or `.mp3`; offer `instrumental_clean` as an alternative) plus the karaoke video. The per-song
|
||||
gen jobs are a free extra: four full karaoke tracks.
|
||||
|
||||
## Making it sound good
|
||||
|
||||
- **Cut where the next vocal starts, not where the last one ends.** A section should run until
|
||||
just before the next line of its song, so the crossfade uses the instrumental tail (the last
|
||||
chord ringing out) rather than chopping the band mid-phrase.
|
||||
- **Use the next song's intro as the bridge.** Starting the next section at 0:00 lets its
|
||||
intro (guitar pickup, fiddle intro) play under the crossfade, and the singer gets a few bars
|
||||
to find the new key.
|
||||
- **Different keys and tempos are fine.** Medleys jump keys all the time (the example goes
|
||||
G → C → D → F → G/A♭). A 2s equal-power crossfade on a sustained chord sounds like a
|
||||
transition. Don't time-stretch to match tempos. It sounds digital, and briefs often ask you
|
||||
not to.
|
||||
- **A pot-out/pot-in is the strongest transition.** A fast fader drop (about 1.3s) on a
|
||||
held word, a short silence (0.5–1s), then a quick fade-in (about 0.5s) on a distinctive
|
||||
instrumental phrase. It's the natural way to reset tempo or mood before a finale.
|
||||
- **Check the brief against the recording.** Customers describe the arrangement from memory
|
||||
or from a karaoke track. In the example, the brief asked for an "instrumental intro → verse 1
|
||||
→ chorus 1" on a song whose original starts singing at 0.4s and has no chorus until the third
|
||||
verse. Measure, then confirm the interpretation with the customer.
|
||||
- **Gain-match per section, then normalise once.** Don't run `loudnorm` on each section
|
||||
separately (it pumps); set static per-section gains, then process the whole medley once.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Pre-review lyric timings are fine for *choosing* cut points (check them against the
|
||||
lead-vocal stem). Only the render needs the reviewed `corrections_updated.json`.
|
||||
- Lines are kept by **start** time. A line that starts just before a cut belongs to the
|
||||
previous section, so place cuts in gaps between lines.
|
||||
- The audio file has no title card or countdown; it starts straight into the song. Only the
|
||||
video gets the 5s title and 3s countdown. Mention this when delivering both.
|
||||
- Bulk auto-select left every job (8 of 8 across two batches) at `awaiting_audio_selection`,
|
||||
even with exact lossless matches. Plan to pick sources by hand.
|
||||
|
||||
## Future: self-serve medleys
|
||||
|
||||
Tracked in the workspace `BACKLOG.md` ("Self-serve medley builder in gen"). The building blocks
|
||||
already exist: per-song jobs, sample-aligned stems, reviewed lyrics, `OutputGenerator`. A
|
||||
product version would mainly need a UI for picking sections (a waveform with lyric lines to
|
||||
snap cut points to) and a backend job type that runs the `audio` + `video` steps.
|
||||
@@ -11,6 +11,9 @@
|
||||
(MusicBrainz lookup + KaraokeNerds availability); auto-picks confident
|
||||
lossless audio, parks the rest in a review queue. See `docs/API.md` § Bulk Mode.
|
||||
- `/admin` - Admin dashboard (user management, job monitoring)
|
||||
- **Medleys** (agent-run, local): one gen job per original song, then
|
||||
`scripts/medley/make_medley.py` cuts the audio and renders a synced karaoke video from the
|
||||
reviewed lyrics. See [MEDLEYS.md](MEDLEYS.md).
|
||||
- **Backend API**: https://api.nomadkaraoke.com - FastAPI on Cloud Run
|
||||
- **CLI Tools**: `karaoke-gen` (local), `karaoke-gen-remote` (cloud)
|
||||
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"title": "Classic Country Medley",
|
||||
"artist": "George Jones, Kris Kristofferson, George Strait & Randy Travis",
|
||||
"theme_job_id": "a70794e2",
|
||||
"workdir": "work",
|
||||
"outdir": "out",
|
||||
"sections": [
|
||||
{"label": "He Stopped Loving Her Today (verse 1)", "job_id": "a70794e2", "start": 0.0, "end": 54.0},
|
||||
{"label": "Help Me Make It Through the Night (intro, verse 1, chorus)", "job_id": "0ac008b5", "start": 0.0, "end": 52.0,
|
||||
"join": "crossfade", "join_duration": 2.0},
|
||||
{"label": "Amarillo by Morning (fiddle intro, verse 1, chorus)", "job_id": "4c90a9a2", "start": 0.0, "end": 57.0,
|
||||
"join": "crossfade", "join_duration": 2.0},
|
||||
{"label": "On the Other Hand (verse 1, chorus, pot out on 'hand')", "job_id": "6a9ce5e2", "start": 0.0, "end": 82.0,
|
||||
"join": "crossfade", "join_duration": 2.0, "fade_out_start": 80.7},
|
||||
{"label": "He Stopped Loving Her Today (fiddle bridge to end)", "job_id": "a70794e2", "start": 132.2, "end": null,
|
||||
"join": "gap", "join_duration": 0.7, "fade_in": 0.5}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,382 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Build a karaoke medley (audio + synced karaoke video) from completed gen jobs.
|
||||
|
||||
See docs/MEDLEYS.md for the full process and the craft of choosing cut points.
|
||||
|
||||
Every section of the medley is cut from a gen job for the ORIGINAL song. Gen already gave
|
||||
us the lossless source, sample-aligned stems and reviewed word-level lyrics for each one,
|
||||
so the medley is just: the same cuts applied to every audio variant, plus the reviewed
|
||||
lyric segments shifted onto the medley timeline and rendered with gen's own renderer.
|
||||
|
||||
Usage (from the repo root, in an env where karaoke_gen imports):
|
||||
python scripts/medley/make_medley.py fetch spec.json # Firestore + GCS -> <workdir>/<job_id>/
|
||||
python scripts/medley/make_medley.py audio spec.json # medley WAV/MP3 per audio variant
|
||||
python scripts/medley/make_medley.py video spec.json [--resolution 4k] # karaoke + guide MP4s
|
||||
|
||||
Requires Firestore read access (ADC; the claude-readonly SA is enough), the gcloud CLI for GCS
|
||||
downloads (set MEDLEY_GCLOUD_ACCOUNT to pick an account, e.g. the read-only SA when your user
|
||||
login has expired) and ffmpeg.
|
||||
See scripts/medley/example-spec.json for the spec format.
|
||||
"""
|
||||
import argparse
|
||||
import dataclasses
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
|
||||
PROJECT = "nomadkaraoke"
|
||||
BUCKET = "karaoke-gen-storage-nomadkaraoke"
|
||||
DEFAULT_CROSSFADE = 2.0
|
||||
SEGMENT_TARGET_LUFS = -18.0
|
||||
FINAL_LUFS = -14.0
|
||||
FINAL_TRUE_PEAK = -1.0
|
||||
# Audio variants built by `audio`: name -> stem file (None = the job's original input audio)
|
||||
VARIANTS = {
|
||||
"instrumental_with_backing": "instrumental_with_backing.flac",
|
||||
"instrumental_clean": "instrumental_clean.flac",
|
||||
"guide_with_vocals": None,
|
||||
}
|
||||
KARAOKE_VARIANT = "instrumental_with_backing"
|
||||
GUIDE_VARIANT = "guide_with_vocals"
|
||||
|
||||
log = logging.getLogger("medley")
|
||||
|
||||
|
||||
# --- Spec + timeline (pure, unit-tested) -------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class Section:
|
||||
job_id: str
|
||||
start: float # source seconds
|
||||
end: Optional[float] # source seconds; None = to the end of the song
|
||||
join: str = "crossfade" # how this section joins the PREVIOUS one: crossfade | gap
|
||||
join_duration: float = DEFAULT_CROSSFADE # crossfade length, or silence length for a gap
|
||||
fade_in: float = 0.0 # "pot in": fade-in length at the start of this section
|
||||
fade_out_start: Optional[float] = None # "pot out": source second the fader starts dropping
|
||||
label: str = ""
|
||||
|
||||
|
||||
def parse_spec(spec: dict) -> list[Section]:
|
||||
sections = []
|
||||
for i, s in enumerate(spec["sections"]):
|
||||
join = s.get("join", "crossfade")
|
||||
if join not in ("crossfade", "gap"):
|
||||
raise ValueError(f"section {i}: join must be 'crossfade' or 'gap', got {join!r}")
|
||||
sec = Section(
|
||||
job_id=s["job_id"], start=float(s.get("start", 0.0)),
|
||||
end=None if s.get("end") is None else float(s["end"]),
|
||||
join=join,
|
||||
join_duration=float(s.get("join_duration", DEFAULT_CROSSFADE if join == "crossfade" else 0.5)),
|
||||
fade_in=float(s.get("fade_in", 0.0)),
|
||||
fade_out_start=None if s.get("fade_out_start") is None else float(s["fade_out_start"]),
|
||||
label=s.get("label", ""),
|
||||
)
|
||||
if sec.end is not None and sec.end <= sec.start:
|
||||
raise ValueError(f"section {i}: end must be after start")
|
||||
if sec.fade_out_start is not None and sec.end is not None and not sec.start < sec.fade_out_start < sec.end:
|
||||
raise ValueError(f"section {i}: fade_out_start must fall inside the section")
|
||||
sections.append(sec)
|
||||
if not sections:
|
||||
raise ValueError("spec has no sections")
|
||||
return sections
|
||||
|
||||
|
||||
def compute_offsets(sections: list[Section], durations: list[float]) -> list[float]:
|
||||
"""Medley start time of each section. durations[i] = section length in seconds.
|
||||
|
||||
A crossfade overlaps the previous section's tail by join_duration; a gap inserts
|
||||
join_duration seconds of silence. Offsets are what the lyric shift adds to source times.
|
||||
"""
|
||||
offsets, cursor = [], 0.0
|
||||
for i, (sec, dur) in enumerate(zip(sections, durations)):
|
||||
if i == 0:
|
||||
start = 0.0
|
||||
elif sec.join == "crossfade":
|
||||
start = cursor - sec.join_duration
|
||||
else:
|
||||
start = cursor + sec.join_duration
|
||||
offsets.append(start)
|
||||
cursor = start + dur
|
||||
return offsets
|
||||
|
||||
|
||||
def shift_segments(segments, start: float, end: Optional[float], offset: float, prefix: str, tolerance: float = 0.05):
|
||||
"""Keep lyric lines that START inside [start, end) and move them by offset.
|
||||
|
||||
IDs are prefixed so the same song can appear twice (e.g. opener + finale).
|
||||
"""
|
||||
out = []
|
||||
for seg in segments:
|
||||
if seg.start_time >= start - tolerance and (end is None or seg.start_time < end):
|
||||
words = [dataclasses.replace(w, id=f"{prefix}_{w.id}", start_time=w.start_time + offset,
|
||||
end_time=w.end_time + offset) for w in seg.words]
|
||||
out.append(dataclasses.replace(seg, id=f"{prefix}_{seg.id}", words=words,
|
||||
start_time=seg.start_time + offset, end_time=seg.end_time + offset))
|
||||
return out
|
||||
|
||||
|
||||
def build_filtergraph(sections: list[Section], durations: list[float], gains_db: list[float]) -> str:
|
||||
"""ffmpeg filter_complex joining inputs 0..n-1 (one per section, already trimmed) into [out]."""
|
||||
chains = []
|
||||
for i, (sec, dur, gain) in enumerate(zip(sections, durations, gains_db)):
|
||||
f = f"[{i}:a]aformat=sample_rates=44100:channel_layouts=stereo,volume={gain:.2f}dB"
|
||||
if sec.fade_in > 0:
|
||||
f += f",afade=t=in:st=0:d={sec.fade_in:.3f}:curve=qsin"
|
||||
if sec.fade_out_start is not None:
|
||||
st = sec.fade_out_start - sec.start
|
||||
f += f",afade=t=out:st={st:.3f}:d={dur - st:.3f}:curve=qsin"
|
||||
nxt = sections[i + 1] if i + 1 < len(sections) else None
|
||||
if nxt is not None and nxt.join == "gap":
|
||||
f += f",apad=pad_dur={nxt.join_duration:.3f}"
|
||||
chains.append(f + f"[s{i}]")
|
||||
cur = "s0"
|
||||
for i in range(1, len(sections)):
|
||||
out = f"x{i}"
|
||||
if sections[i].join == "crossfade":
|
||||
chains.append(f"[{cur}][s{i}]acrossfade=d={sections[i].join_duration:.3f}:c1=qsin:c2=qsin[{out}]")
|
||||
else:
|
||||
chains.append(f"[{cur}][s{i}]concat=n=2:v=0:a=1[{out}]")
|
||||
cur = out
|
||||
chains.append(f"[{cur}]loudnorm=I={FINAL_LUFS}:TP={FINAL_TRUE_PEAK}:LRA=11,aresample=44100[out]")
|
||||
return ";".join(chains)
|
||||
|
||||
|
||||
# --- I/O helpers ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _run(args, **kw):
|
||||
return subprocess.run(args, check=True, **kw)
|
||||
|
||||
|
||||
def probe_duration(path: str) -> float:
|
||||
out = _run(["ffprobe", "-v", "error", "-show_entries", "format=duration", "-of", "csv=p=0", path],
|
||||
capture_output=True, text=True).stdout
|
||||
return float(out.strip())
|
||||
|
||||
|
||||
def measure_lufs(path: str, start: float, end: Optional[float]) -> float:
|
||||
args = ["ffmpeg", "-hide_banner", "-nostats", "-ss", str(start)]
|
||||
if end is not None:
|
||||
args += ["-to", str(end)]
|
||||
args += ["-i", path, "-af", "ebur128", "-f", "null", "-"]
|
||||
res = subprocess.run(args, capture_output=True, text=True)
|
||||
found = re.findall(r"I:\s+(-?[\d.]+) LUFS", res.stderr)
|
||||
if res.returncode != 0 or not found:
|
||||
raise RuntimeError(f"loudness measurement failed for {path}: {res.stderr.strip()[-300:]}")
|
||||
return float(found[-1])
|
||||
|
||||
|
||||
def gcs_download(src: str, dst: str, required: bool = True) -> bool:
|
||||
"""Download gs://BUCKET/src via the gcloud CLI (the Python client stalled for ~1h on large FLACs)."""
|
||||
os.makedirs(os.path.dirname(dst), exist_ok=True)
|
||||
args = ["gcloud", "storage", "cp", f"gs://{BUCKET}/{src}", dst]
|
||||
if os.environ.get("MEDLEY_GCLOUD_ACCOUNT"):
|
||||
args.insert(3, f"--account={os.environ['MEDLEY_GCLOUD_ACCOUNT']}")
|
||||
result = subprocess.run(args, capture_output=True, text=True)
|
||||
if result.returncode != 0:
|
||||
if required:
|
||||
raise RuntimeError(f"download failed for gs://{BUCKET}/{src}: {result.stderr.strip()[-300:]}")
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def job_dir(workdir: str, job_id: str) -> str:
|
||||
return os.path.join(workdir, job_id)
|
||||
|
||||
|
||||
def load_job_meta(workdir: str, job_id: str) -> dict:
|
||||
with open(os.path.join(job_dir(workdir, job_id), "job.json")) as f:
|
||||
return json.load(f)
|
||||
|
||||
|
||||
def variant_path(workdir: str, job_id: str, variant: str) -> str:
|
||||
d = job_dir(workdir, job_id)
|
||||
stem = VARIANTS[variant]
|
||||
if stem is None:
|
||||
return os.path.join(d, "input", os.path.basename(load_job_meta(workdir, job_id)["input_media_gcs_path"]))
|
||||
return os.path.join(d, "stems", stem)
|
||||
|
||||
|
||||
def section_durations(sections: list[Section], workdir: str) -> list[float]:
|
||||
durs = []
|
||||
for sec in sections:
|
||||
end = sec.end if sec.end is not None else probe_duration(variant_path(workdir, sec.job_id, GUIDE_VARIANT))
|
||||
durs.append(end - sec.start)
|
||||
return durs
|
||||
|
||||
|
||||
# --- Commands ------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def cmd_fetch(spec: dict, workdir: str):
|
||||
from google.cloud import firestore
|
||||
db = firestore.Client(project=PROJECT)
|
||||
for job_id in dict.fromkeys(s["job_id"] for s in spec["sections"]):
|
||||
doc = db.collection("jobs").document(job_id).get()
|
||||
if not doc.exists:
|
||||
raise SystemExit(f"job {job_id} not found")
|
||||
job = doc.to_dict()
|
||||
if job.get("status") != "complete":
|
||||
log.warning("job %s is %s, not complete: lyrics may not be reviewed yet", job_id, job.get("status"))
|
||||
d = job_dir(workdir, job_id)
|
||||
os.makedirs(os.path.join(d, "stems"), exist_ok=True)
|
||||
os.makedirs(os.path.join(d, "input"), exist_ok=True)
|
||||
meta = {k: job.get(k) for k in ("job_id", "artist", "title", "status", "input_media_gcs_path",
|
||||
"style_params_gcs_path", "style_assets")}
|
||||
with open(os.path.join(d, "job.json"), "w") as f:
|
||||
json.dump(meta, f, indent=1, default=str)
|
||||
wanted = {job["input_media_gcs_path"]: os.path.join(d, "input", os.path.basename(job["input_media_gcs_path"]))}
|
||||
for name in ("instrumental_with_backing", "instrumental_clean", "lead_vocals", "backing_vocals"):
|
||||
wanted[f"jobs/{job_id}/stems/{name}.flac"] = os.path.join(d, "stems", f"{name}.flac")
|
||||
for name in ("corrections.json", "corrections_updated.json"):
|
||||
wanted[f"jobs/{job_id}/lyrics/{name}"] = os.path.join(d, name)
|
||||
for src, dst in wanted.items():
|
||||
optional = "corrections_updated" in src or src.endswith(("lead_vocals.flac", "backing_vocals.flac"))
|
||||
if not gcs_download(src, dst, required=not optional) and not optional:
|
||||
log.warning("missing gs://%s/%s", BUCKET, src)
|
||||
log.info("fetched %s - %s (%s)", job.get("artist"), job.get("title"), job_id)
|
||||
|
||||
|
||||
def cmd_audio(spec: dict, workdir: str, outdir: str):
|
||||
sections = parse_spec(spec)
|
||||
durations = section_durations(sections, workdir)
|
||||
os.makedirs(outdir, exist_ok=True)
|
||||
for variant in VARIANTS:
|
||||
inputs, gains = [], []
|
||||
for sec in sections:
|
||||
p = variant_path(workdir, sec.job_id, variant)
|
||||
gains.append(SEGMENT_TARGET_LUFS - measure_lufs(p, sec.start, sec.end))
|
||||
inputs += ["-ss", str(sec.start)] + (["-to", str(sec.end)] if sec.end is not None else []) + ["-i", p]
|
||||
wav = os.path.join(outdir, f"medley_{variant}.wav")
|
||||
_run(["ffmpeg", "-y", "-hide_banner", "-loglevel", "error"] + inputs +
|
||||
["-filter_complex", build_filtergraph(sections, durations, gains), "-map", "[out]", "-c:a", "pcm_s24le", wav])
|
||||
_run(["ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-i", wav, "-c:a", "libmp3lame", "-b:a", "320k",
|
||||
wav[:-4] + ".mp3"])
|
||||
log.info("%s: %.1fs", wav, probe_duration(wav))
|
||||
offsets = compute_offsets(sections, durations)
|
||||
with open(os.path.join(outdir, "timeline.json"), "w") as f:
|
||||
json.dump([{"label": s.label, "job_id": s.job_id, "source_start": s.start, "source_end": s.start + d,
|
||||
"medley_start": o, "medley_end": o + d} for s, d, o in zip(sections, durations, offsets)], f, indent=1)
|
||||
|
||||
|
||||
def _load_reviewed_lyrics(workdir: str, job_id: str):
|
||||
from karaoke_gen.lyrics_transcriber.correction.operations import CorrectionOperations
|
||||
from karaoke_gen.lyrics_transcriber.types import CorrectionResult
|
||||
d = job_dir(workdir, job_id)
|
||||
with open(os.path.join(d, "corrections.json")) as f:
|
||||
base = CorrectionResult.from_dict(json.load(f))
|
||||
updated = os.path.join(d, "corrections_updated.json")
|
||||
if os.path.exists(updated):
|
||||
with open(updated) as f:
|
||||
data = json.load(f)
|
||||
if isinstance(data, dict) and "corrections" in data:
|
||||
return CorrectionOperations.update_correction_result_with_data(base, data)
|
||||
return base
|
||||
|
||||
|
||||
def cmd_video(spec: dict, workdir: str, outdir: str, resolution: str):
|
||||
from karaoke_gen.lyrics_transcriber.core.config import OutputConfig
|
||||
from karaoke_gen.lyrics_transcriber.output.countdown_processor import CountdownProcessor
|
||||
from karaoke_gen.lyrics_transcriber.output.generator import OutputGenerator
|
||||
from karaoke_gen.style_loader import get_end_format, get_intro_format, load_styles_from_gcs
|
||||
from karaoke_gen.video_generator import VideoGenerator
|
||||
|
||||
sections = parse_spec(spec)
|
||||
durations = section_durations(sections, workdir)
|
||||
offsets = compute_offsets(sections, durations)
|
||||
karaoke_wav = os.path.join(outdir, f"medley_{KARAOKE_VARIANT}.wav")
|
||||
guide_wav = os.path.join(outdir, f"medley_{GUIDE_VARIANT}.wav")
|
||||
if not os.path.exists(karaoke_wav):
|
||||
raise SystemExit("run the `audio` step first")
|
||||
render_dir = os.path.join(outdir, f"render_{resolution}")
|
||||
cache_dir = os.path.join(render_dir, "cache")
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
|
||||
lyrics = {j: _load_reviewed_lyrics(workdir, j) for j in dict.fromkeys(s.job_id for s in sections)}
|
||||
segments = []
|
||||
for n, (sec, dur, off) in enumerate(zip(sections, durations, offsets)):
|
||||
segments += shift_segments(lyrics[sec.job_id].corrected_segments, sec.start, sec.start + dur,
|
||||
off - sec.start, prefix=f"m{n}")
|
||||
first = lyrics[sections[0].job_id]
|
||||
medley = dataclasses.replace(first, corrected_segments=segments, resized_segments=[])
|
||||
for seg in segments:
|
||||
log.info("%7.2f-%7.2f %s", seg.start_time, seg.end_time, seg.text)
|
||||
|
||||
# Theme: the style of the job named in the spec (default: first section's job)
|
||||
theme_job = load_job_meta(workdir, spec.get("theme_job_id", sections[0].job_id))
|
||||
styles_path, _ = load_styles_from_gcs(theme_job["style_params_gcs_path"], theme_job["style_assets"],
|
||||
os.path.join(render_dir, "style"), gcs_download, log)
|
||||
with open(styles_path) as f:
|
||||
style = json.load(f)
|
||||
|
||||
# Same countdown gen adds when the first word starts within 3s (pads audio + shifts lyrics)
|
||||
medley, audio, _, pad = CountdownProcessor(cache_dir=cache_dir, logger=log).process(
|
||||
correction_result=medley, audio_filepath=karaoke_wav)
|
||||
|
||||
artist, title = spec["artist"], spec["title"]
|
||||
vg = VideoGenerator(logger=log, ffmpeg_base_command="ffmpeg -hide_banner -loglevel error -nostats -y",
|
||||
render_bounding_boxes=False, output_png=True, output_jpg=False)
|
||||
title_png, end_png = os.path.join(render_dir, "title"), os.path.join(render_dir, "end")
|
||||
vg.create_title_video(artist=artist, title=title, format=get_intro_format(style), output_image_filepath_noext=title_png,
|
||||
output_video_filepath=title_png + ".mov", existing_title_image=None, intro_video_duration=0)
|
||||
vg.create_end_video(artist=artist, title=title, format=get_end_format(style), output_image_filepath_noext=end_png,
|
||||
output_video_filepath=end_png + ".mov", existing_end_image=None, end_video_duration=0)
|
||||
|
||||
cfg = OutputConfig(output_dir=render_dir, cache_dir=cache_dir, output_styles_json=styles_path, render_video=True,
|
||||
generate_cdg=False, generate_plain_text=True, generate_lrc=True, video_resolution=resolution)
|
||||
outputs = OutputGenerator(cfg, log).generate_outputs(
|
||||
transcription_corrected=medley, lyrics_results={}, output_prefix=title, audio_filepath=audio,
|
||||
artist=artist, title=title)
|
||||
|
||||
w, h = {"4k": (3840, 2160), "1080p": (1920, 1080), "720p": (1280, 720), "360p": (640, 360)}[resolution]
|
||||
intro_s, end_s = style["intro"]["video_duration"], style["end"]["video_duration"]
|
||||
master = os.path.join(outdir, f"{title} (Karaoke {resolution}).mp4")
|
||||
scale = f"scale={w}:{h},fps=30,format=yuv420p,setsar=1"
|
||||
_run(["ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
|
||||
"-loop", "1", "-t", str(intro_s), "-i", title_png + ".png", "-f", "lavfi", "-t", str(intro_s), "-i", "anullsrc=r=44100:cl=stereo",
|
||||
"-i", outputs.video,
|
||||
"-loop", "1", "-t", str(end_s), "-i", end_png + ".png", "-f", "lavfi", "-t", str(end_s), "-i", "anullsrc=r=44100:cl=stereo",
|
||||
"-filter_complex", f"[0:v]{scale}[t];[2:v]{scale}[k];[3:v]{scale}[e];[t][k][e]concat=n=3:v=1:a=0[v];[1:a][2:a][4:a]concat=n=3:v=0:a=1[a]",
|
||||
"-map", "[v]", "-map", "[a]", "-c:v", "libx264", "-preset", "slow", "-crf", "18", "-c:a", "aac", "-b:a", "320k",
|
||||
"-movflags", "+faststart", master])
|
||||
# Guide: same picture, original-vocals medley delayed by title + countdown (for checking sync)
|
||||
delay_ms = int((intro_s + pad) * 1000)
|
||||
guide = os.path.join(outdir, f"{title} (With Vocals {resolution}).mp4")
|
||||
_run(["ffmpeg", "-hide_banner", "-loglevel", "error", "-y", "-i", master, "-i", guide_wav,
|
||||
"-filter_complex", f"[1:a]adelay={delay_ms}|{delay_ms},apad[a]", "-map", "0:v", "-map", "[a]", "-shortest",
|
||||
"-c:v", "copy", "-c:a", "aac", "-b:a", "320k", "-movflags", "+faststart", guide])
|
||||
if outputs.lrc:
|
||||
_run(["cp", outputs.lrc, os.path.join(outdir, f"{title} (Karaoke).lrc")])
|
||||
log.info("karaoke: %s\nguide: %s", master, guide)
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("command", choices=["fetch", "audio", "video"])
|
||||
p.add_argument("spec")
|
||||
p.add_argument("--resolution", default="4k", choices=["4k", "1080p", "720p", "360p"])
|
||||
args = p.parse_args(argv)
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
||||
with open(args.spec) as f:
|
||||
spec = json.load(f)
|
||||
parse_spec(spec) # validate early
|
||||
base = os.path.dirname(os.path.abspath(args.spec))
|
||||
workdir = os.path.join(base, spec.get("workdir", "work"))
|
||||
outdir = os.path.join(base, spec.get("outdir", "out"))
|
||||
if args.command == "fetch":
|
||||
cmd_fetch(spec, workdir)
|
||||
elif args.command == "audio":
|
||||
cmd_audio(spec, workdir, outdir)
|
||||
else:
|
||||
cmd_video(spec, workdir, outdir, args.resolution)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,109 @@
|
||||
"""Unit tests for the medley builder's pure helpers (scripts/medley/make_medley.py).
|
||||
|
||||
The script isn't part of the installed package, so we load it by path.
|
||||
"""
|
||||
|
||||
import importlib.util
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from karaoke_gen.lyrics_transcriber.types import LyricsSegment, Word
|
||||
|
||||
_SCRIPT = Path(__file__).resolve().parents[2] / "scripts" / "medley" / "make_medley.py"
|
||||
_spec = importlib.util.spec_from_file_location("make_medley", _SCRIPT)
|
||||
medley = importlib.util.module_from_spec(_spec)
|
||||
sys.modules[_spec.name] = medley
|
||||
_spec.loader.exec_module(medley)
|
||||
|
||||
|
||||
EXAMPLE = {
|
||||
"title": "T", "artist": "A",
|
||||
"sections": [
|
||||
{"job_id": "a", "start": 0.0, "end": 54.0},
|
||||
{"job_id": "b", "start": 0.0, "end": 52.0, "join": "crossfade", "join_duration": 2.0},
|
||||
{"job_id": "c", "start": 0.0, "end": 57.0},
|
||||
{"job_id": "d", "start": 0.0, "end": 82.0, "fade_out_start": 80.7},
|
||||
{"job_id": "a", "start": 132.2, "end": None, "join": "gap", "join_duration": 0.7, "fade_in": 0.5},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _segment(seg_id: str, start: float, end: float) -> LyricsSegment:
|
||||
words = [Word(id=f"{seg_id}w1", text="hello", start_time=start, end_time=(start + end) / 2),
|
||||
Word(id=f"{seg_id}w2", text="there", start_time=(start + end) / 2, end_time=end)]
|
||||
return LyricsSegment(id=seg_id, text="hello there", words=words, start_time=start, end_time=end)
|
||||
|
||||
|
||||
class TestParseSpec:
|
||||
def test_defaults_to_two_second_crossfade(self):
|
||||
sections = medley.parse_spec(EXAMPLE)
|
||||
assert sections[2].join == "crossfade"
|
||||
assert sections[2].join_duration == 2.0
|
||||
|
||||
def test_gap_and_pot_fields(self):
|
||||
sections = medley.parse_spec(EXAMPLE)
|
||||
assert sections[3].fade_out_start == 80.7
|
||||
assert sections[4].join == "gap" and sections[4].join_duration == 0.7
|
||||
assert sections[4].fade_in == 0.5 and sections[4].end is None
|
||||
|
||||
@pytest.mark.parametrize("bad", [
|
||||
{"sections": []},
|
||||
{"sections": [{"job_id": "a", "start": 10, "end": 5}]},
|
||||
{"sections": [{"job_id": "a", "start": 0, "end": 10, "join": "splice"}]},
|
||||
{"sections": [{"job_id": "a", "start": 0, "end": 10, "fade_out_start": 12}]},
|
||||
])
|
||||
def test_rejects_invalid_specs(self, bad):
|
||||
with pytest.raises(ValueError):
|
||||
medley.parse_spec(bad)
|
||||
|
||||
|
||||
class TestComputeOffsets:
|
||||
def test_matches_the_country_medley_timeline(self):
|
||||
sections = medley.parse_spec(EXAMPLE)
|
||||
offsets = medley.compute_offsets(sections, [54.0, 52.0, 57.0, 82.0, 199.133 - 132.2])
|
||||
assert offsets == pytest.approx([0.0, 52.0, 102.0, 157.0, 239.7])
|
||||
|
||||
def test_gap_inserts_silence_and_crossfade_overlaps(self):
|
||||
sections = medley.parse_spec({"sections": [
|
||||
{"job_id": "a", "start": 0, "end": 10},
|
||||
{"job_id": "b", "start": 0, "end": 10, "join": "gap", "join_duration": 1.0},
|
||||
{"job_id": "c", "start": 0, "end": 10, "join": "crossfade", "join_duration": 3.0},
|
||||
]})
|
||||
assert medley.compute_offsets(sections, [10, 10, 10]) == pytest.approx([0, 11, 18])
|
||||
|
||||
|
||||
class TestShiftSegments:
|
||||
def test_keeps_lines_starting_inside_the_cut_and_shifts_them(self):
|
||||
segs = [_segment("s1", 1.0, 3.0), _segment("s2", 50.0, 53.0), _segment("s3", 54.2, 57.0)]
|
||||
out = medley.shift_segments(segs, start=0.0, end=54.0, offset=100.0, prefix="m0")
|
||||
assert [s.id for s in out] == ["m0_s1", "m0_s2"]
|
||||
assert out[1].start_time == pytest.approx(150.0)
|
||||
assert [w.start_time for w in out[1].words] == pytest.approx([150.0, 151.5])
|
||||
assert out[1].words[0].id == "m0_s2w1"
|
||||
|
||||
def test_open_ended_section_keeps_everything_after_start(self):
|
||||
segs = [_segment("s1", 120.0, 125.0), _segment("s2", 138.0, 140.0), _segment("s3", 180.0, 184.0)]
|
||||
out = medley.shift_segments(segs, start=132.2, end=None, offset=107.5, prefix="m4")
|
||||
assert [s.id for s in out] == ["m4_s2", "m4_s3"]
|
||||
assert out[0].start_time == pytest.approx(245.5)
|
||||
|
||||
def test_same_song_twice_gets_unique_ids(self):
|
||||
segs = [_segment("s1", 1.0, 3.0)]
|
||||
a = medley.shift_segments(segs, 0.0, 10.0, 0.0, "m0")
|
||||
b = medley.shift_segments(segs, 0.0, 10.0, 50.0, "m4")
|
||||
assert a[0].id != b[0].id
|
||||
assert segs[0].start_time == 1.0 # originals untouched
|
||||
|
||||
|
||||
class TestBuildFiltergraph:
|
||||
def test_crossfades_gap_and_pot_fades(self):
|
||||
sections = medley.parse_spec(EXAMPLE)
|
||||
graph = medley.build_filtergraph(sections, [54.0, 52.0, 57.0, 82.0, 66.9], [0.0, 1.0, -1.0, 0.5, 2.0])
|
||||
assert graph.count("acrossfade=d=2.000") == 3
|
||||
assert "afade=t=out:st=80.700:d=1.300" in graph # pot out on section 4
|
||||
assert "apad=pad_dur=0.700" in graph # silence before the gap join
|
||||
assert "afade=t=in:st=0:d=0.500" in graph # pot in on the finale
|
||||
assert "[x3][s4]concat=n=2:v=0:a=1[x4]" in graph
|
||||
assert graph.endswith("[x4]loudnorm=I=-14.0:TP=-1.0:LRA=11,aresample=44100[out]")
|
||||
Reference in new issue
Block a user