diff --git a/CLAUDE.md b/CLAUDE.md index 56b4592d..be1ed058 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -23,6 +23,7 @@ | GDrive validator & gaps | `docs/GDRIVE-VALIDATOR.md` | | **Product communication** | `docs/PRODUCT-COMMUNICATION-GUIDE.md` | | Brand style guide | `docs/BRAND-STYLE-GUIDE.md` | +| **Making medleys** (multi-song karaoke) | `docs/MEDLEYS.md` + `scripts/medley/make_medley.py` | ## Tech Stack diff --git a/docs/MEDLEYS.md b/docs/MEDLEYS.md new file mode 100644 index 00000000..a9effc02 --- /dev/null +++ b/docs/MEDLEYS.md @@ -0,0 +1,175 @@ +# Making Karaoke Medleys + +How to turn several songs into one continuous medley: a backing track plus a synced karaoke +video in the normal gen look. It's a manual, agent-run process for now. A self-serve +version is on the backlog. + +First done in October 2026 for a 5-section classic country medley. Total time was about an +hour, mostly waiting on gen jobs. That medley is the worked example throughout this doc and +in `scripts/medley/example-spec.json`. + +## The idea in one paragraph + +**Don't** build the medley first and push it through gen as a new job. Gen would re-transcribe +it with no reference lyrics to check against, and someone would have to review it from scratch. +Instead, make **one ordinary gen job per original song**. Each job gives us the lossless source, +sample-aligned stems and reviewed word-level lyrics. Then the medley is just two things: + +1. **Audio:** apply the same cuts, gains and fades to every variant (instrumental with backing + vocals, clean instrumental, original mix). Stems are sample-aligned with their source, so all + variants line up exactly. +2. **Video:** take each job's *reviewed* lyric lines that fall inside its cut, shift them by that + section's medley offset, and render locally with gen's own `OutputGenerator`. That gives the + same theme, countdown and title/end screens as a normal track. + +Bonus: the per-song jobs are full karaoke tracks too. They can be published as community +versions or given to the customer as extras. + +## Step by step + +### 1. Make a gen job for each original song + +One job per distinct song. A song used twice, like an opener that comes back as the finale, +needs only one job. + +- Bulk submit works (`POST /api/bulk/submit` with the admin token; see `docs/API.md` + § Bulk Mode). Expect jobs to stop at `awaiting_audio_selection`, then pick the source by hand + with `GET /api/audio-search/{id}/results` + `POST /api/audio-search/{id}/select`. +- **Pick the original studio recording.** Avoid live versions, duets, remasters and "greatest + hits" sets that might be re-recordings. The customer knows the original arrangement, and the + cut points assume it. Album compilations from the original label (e.g. an "Essential" or + "Songs of" collection) are usually the original masters. +- **Prefer CD FLAC torrent results.** They're lossless. Admin jobs get a + `402 Insufficient credits` on Spotify results, because the admin account has 0 credits and + Spotify results carry a duration-based charge; torrent results don't. +- Review the lyrics properly. These reviewed timings are what the medley video uses. Timing + matters more than usual, because every line will be used. + +### 2. Write the spec and fetch the jobs + +Copy `scripts/medley/example-spec.json` and edit it. Each section names a job, a source +`start`/`end` (seconds; `end: null` = to the end of the song) and how it joins the previous +section: + +| Field | Meaning | +|---|---| +| `join: "crossfade"` + `join_duration` | Overlap the previous section's tail (default 2.0s, equal-power) | +| `join: "gap"` + `join_duration` | Insert this much silence instead | +| `fade_out_start` | "Pot out": fader drops from this source second to the section `end` | +| `fade_in` | "Pot in": fade-in length at the start of the section | +| `theme_job_id` (top level) | Whose theme to render with (default: the first section's job) | + +```bash +# from a karaoke-gen checkout, in an env where karaoke_gen imports +export PYTHONPATH=$PWD +export MEDLEY_GCLOUD_ACCOUNT=claude-readonly@nomadkaraoke.iam.gserviceaccount.com # optional +python scripts/medley/make_medley.py fetch path/to/spec.json +``` + +`fetch` reads each job from Firestore and downloads the source audio, stems, `corrections.json` +and `corrections_updated.json` into `/work//`. It uses the `gcloud` CLI, +because the Python GCS client stalled for about an hour on large FLACs. Stems are deleted 30 +days after a job completes (`docs/STORAGE-RETENTION.md`), so build medleys from recent jobs or +regenerate the stems first. + +### 3. Find the cut points + +This is where the craft is (see [Making it sound good](#making-it-sound-good)). Use data +rather than guesswork: + +- **Word timings** from `corrections.json` / the job's `karaoke.lrc`: where lines start and end. +- **Lead-vocal activity** from `stems/lead_vocals.flac`. RMS in 100ms frames shows exactly when + a held note dies away and when the next line starts. Use it to check the lyric timings and to + place fades. In the example, "hand" starts at 80.66s and is held until 82.1s, so the pot-out + runs 80.7 → 82.0. +- **Key and tempo:** `librosa` (`beat.beat_track` + chroma key-profile matching) is good enough + to confirm a brief's BPM/key claims and spot modulations. librosa often reports double tempo, + so halve it when it looks too high. + +### 4. Build the audio + +```bash +python scripts/medley/make_medley.py audio path/to/spec.json +``` + +This writes `out/medley_{instrumental_with_backing,instrumental_clean,guide_with_vocals}.{wav,mp3}` +and `out/timeline.json` (each section's source and medley times). + +- Each section's gain is set so the section itself measures -18 LUFS. That's measured on the + section, not the whole song, which matters for quiet intros. The full medley is then run + through `loudnorm` to -14 LUFS with peaks no higher than -1 dBTP. In the example, all five + sections ended up within 1.6 LU of each other. +- WAV is 24-bit / 44.1 kHz. MP3 is 320k. +- **Listen to the guide (`guide_with_vocals`) first.** Bad transitions are much easier to hear + with vocals in. + +### 5. Render the video + +```bash +python scripts/medley/make_medley.py video path/to/spec.json --resolution 360p # quick check +python scripts/medley/make_medley.py video path/to/spec.json # 4K final +``` + +This writes `out/ (Karaoke <res>).mp4` (instrumental with backing vocals), +`out/<title> (With Vocals <res>).mp4` (same picture over the original-vocals medley, for +checking sync) and `out/<title> (Karaoke).lrc`. A 4K render of a 5-minute medley takes a few +minutes on an M-series Mac. + +What it does: +- Loads each job's reviewed lyrics the same way the render worker does: `CorrectionResult` + + `CorrectionOperations.update_correction_result_with_data`. +- Keeps the lines that **start** inside each cut and shifts them by + `medley_offset - source_start`. IDs get a section prefix so a song can appear twice. +- Runs `CountdownProcessor`, which adds gen's "3… 2… 1…" plus 3s of padding if the first word + starts within 3s, and `VideoGenerator` title/end PNGs with the spec's title/artist. +- Renders with `OutputGenerator`, then joins title (5s) + karaoke + end (5s) with ffmpeg. + +**Check sync:** the vocals in the guide video should sit exactly *title + countdown* seconds +(8.0s in the example) after their position in the medley WAV. Cross-correlating a 10s window +confirms it in seconds, and spot-checking a few frames confirms the right line is on screen. + +### 6. Deliver + +Give the customer the audio file they asked for (usually `medley_instrumental_with_backing.wav` +or `.mp3`; offer `instrumental_clean` as an alternative) plus the karaoke video. The per-song +gen jobs are a free extra: four full karaoke tracks. + +## Making it sound good + +- **Cut where the next vocal starts, not where the last one ends.** A section should run until + just before the next line of its song, so the crossfade uses the instrumental tail (the last + chord ringing out) rather than chopping the band mid-phrase. +- **Use the next song's intro as the bridge.** Starting the next section at 0:00 lets its + intro (guitar pickup, fiddle intro) play under the crossfade, and the singer gets a few bars + to find the new key. +- **Different keys and tempos are fine.** Medleys jump keys all the time (the example goes + G → C → D → F → G/A♭). A 2s equal-power crossfade on a sustained chord sounds like a + transition. Don't time-stretch to match tempos. It sounds digital, and briefs often ask you + not to. +- **A pot-out/pot-in is the strongest transition.** A fast fader drop (about 1.3s) on a + held word, a short silence (0.5–1s), then a quick fade-in (about 0.5s) on a distinctive + instrumental phrase. It's the natural way to reset tempo or mood before a finale. +- **Check the brief against the recording.** Customers describe the arrangement from memory + or from a karaoke track. In the example, the brief asked for an "instrumental intro → verse 1 + → chorus 1" on a song whose original starts singing at 0.4s and has no chorus until the third + verse. Measure, then confirm the interpretation with the customer. +- **Gain-match per section, then normalise once.** Don't run `loudnorm` on each section + separately (it pumps); set static per-section gains, then process the whole medley once. + +## Gotchas + +- Pre-review lyric timings are fine for *choosing* cut points (check them against the + lead-vocal stem). Only the render needs the reviewed `corrections_updated.json`. +- Lines are kept by **start** time. A line that starts just before a cut belongs to the + previous section, so place cuts in gaps between lines. +- The audio file has no title card or countdown; it starts straight into the song. Only the + video gets the 5s title and 3s countdown. Mention this when delivering both. +- Bulk auto-select left every job (8 of 8 across two batches) at `awaiting_audio_selection`, + even with exact lossless matches. Plan to pick sources by hand. + +## Future: self-serve medleys + +Tracked in the workspace `BACKLOG.md` ("Self-serve medley builder in gen"). The building blocks +already exist: per-song jobs, sample-aligned stems, reviewed lyrics, `OutputGenerator`. A +product version would mainly need a UI for picking sections (a waveform with lyric lines to +snap cut points to) and a backend job type that runs the `audio` + `video` steps. diff --git a/docs/README.md b/docs/README.md index 6dccdb01..a3e23729 100644 --- a/docs/README.md +++ b/docs/README.md @@ -11,6 +11,9 @@ (MusicBrainz lookup + KaraokeNerds availability); auto-picks confident lossless audio, parks the rest in a review queue. See `docs/API.md` § Bulk Mode. - `/admin` - Admin dashboard (user management, job monitoring) +- **Medleys** (agent-run, local): one gen job per original song, then + `scripts/medley/make_medley.py` cuts the audio and renders a synced karaoke video from the + reviewed lyrics. See [MEDLEYS.md](MEDLEYS.md). - **Backend API**: https://api.nomadkaraoke.com - FastAPI on Cloud Run - **CLI Tools**: `karaoke-gen` (local), `karaoke-gen-remote` (cloud) diff --git a/scripts/medley/example-spec.json b/scripts/medley/example-spec.json new file mode 100644 index 00000000..8cbf6d90 --- /dev/null +++ b/scripts/medley/example-spec.json @@ -0,0 +1,18 @@ +{ + "title": "Classic Country Medley", + "artist": "George Jones, Kris Kristofferson, George Strait & Randy Travis", + "theme_job_id": "a70794e2", + "workdir": "work", + "outdir": "out", + "sections": [ + {"label": "He Stopped Loving Her Today (verse 1)", "job_id": "a70794e2", "start": 0.0, "end": 54.0}, + {"label": "Help Me Make It Through the Night (intro, verse 1, chorus)", "job_id": "0ac008b5", "start": 0.0, "end": 52.0, + "join": "crossfade", "join_duration": 2.0}, + {"label": "Amarillo by Morning (fiddle intro, verse 1, chorus)", "job_id": "4c90a9a2", "start": 0.0, "end": 57.0, + "join": "crossfade", "join_duration": 2.0}, + {"label": "On the Other Hand (verse 1, chorus, pot out on 'hand')", "job_id": "6a9ce5e2", "start": 0.0, "end": 82.0, + "join": "crossfade", "join_duration": 2.0, "fade_out_start": 80.7}, + {"label": "He Stopped Loving Her Today (fiddle bridge to end)", "job_id": "a70794e2", "start": 132.2, "end": null, + "join": "gap", "join_duration": 0.7, "fade_in": 0.5} + ] +} diff --git a/scripts/medley/make_medley.py b/scripts/medley/make_medley.py new file mode 100644 index 00000000..403ed97a --- /dev/null +++ b/scripts/medley/make_medley.py @@ -0,0 +1,382 @@ +#!/usr/bin/env python3 +""" +Build a karaoke medley (audio + synced karaoke video) from completed gen jobs. + +See docs/MEDLEYS.md for the full process and the craft of choosing cut points. + +Every section of the medley is cut from a gen job for the ORIGINAL song. Gen already gave +us the lossless source, sample-aligned stems and reviewed word-level lyrics for each one, +so the medley is just: the same cuts applied to every audio variant, plus the reviewed +lyric segments shifted onto the medley timeline and rendered with gen's own renderer. + +Usage (from the repo root, in an env where karaoke_gen imports): + python scripts/medley/make_medley.py fetch spec.json # Firestore + GCS -> <workdir>/<job_id>/ + python scripts/medley/make_medley.py audio spec.json # medley WAV/MP3 per audio variant + python scripts/medley/make_medley.py video spec.json [--resolution 4k] # karaoke + guide MP4s + +Requires Firestore read access (ADC; the claude-readonly SA is enough), the gcloud CLI for GCS +downloads (set MEDLEY_GCLOUD_ACCOUNT to pick an account, e.g. the read-only SA when your user +login has expired) and ffmpeg. +See scripts/medley/example-spec.json for the spec format. +""" +import argparse +import dataclasses +import json +import logging +import os +import re +import subprocess +import sys +from dataclasses import dataclass +from typing import Optional + +PROJECT = "nomadkaraoke" +BUCKET = "karaoke-gen-storage-nomadkaraoke" +DEFAULT_CROSSFADE = 2.0 +SEGMENT_TARGET_LUFS = -18.0 +FINAL_LUFS = -14.0 +FINAL_TRUE_PEAK = -1.0 +# Audio variants built by `audio`: name -> stem file (None = the job's original input audio) +VARIANTS = { + "instrumental_with_backing": "instrumental_with_backing.flac", + "instrumental_clean": "instrumental_clean.flac", + "guide_with_vocals": None, +} +KARAOKE_VARIANT = "instrumental_with_backing" +GUIDE_VARIANT = "guide_with_vocals" + +log = logging.getLogger("medley") + + +# --- Spec + timeline (pure, unit-tested) ------------------------------------------------- + + +@dataclass +class Section: + job_id: str + start: float # source seconds + end: Optional[float] # source seconds; None = to the end of the song + join: str = "crossfade" # how this section joins the PREVIOUS one: crossfade | gap + join_duration: float = DEFAULT_CROSSFADE # crossfade length, or silence length for a gap + fade_in: float = 0.0 # "pot in": fade-in length at the start of this section + fade_out_start: Optional[float] = None # "pot out": source second the fader starts dropping + label: str = "" + + +def parse_spec(spec: dict) -> list[Section]: + sections = [] + for i, s in enumerate(spec["sections"]): + join = s.get("join", "crossfade") + if join not in ("crossfade", "gap"): + raise ValueError(f"section {i}: join must be 'crossfade' or 'gap', got {join!r}") + sec = Section( + job_id=s["job_id"], start=float(s.get("start", 0.0)), + end=None if s.get("end") is None else float(s["end"]), + join=join, + join_duration=float(s.get("join_duration", DEFAULT_CROSSFADE if join == "crossfade" else 0.5)), + fade_in=float(s.get("fade_in", 0.0)), + fade_out_start=None if s.get("fade_out_start") is None else float(s["fade_out_start"]), + label=s.get("label", ""), + ) + if sec.end is not None and sec.end <= sec.start: + raise ValueError(f"section {i}: end must be after start") + if sec.fade_out_start is not None and sec.end is not None and not sec.start < sec.fade_out_start < sec.end: + raise ValueError(f"section {i}: fade_out_start must fall inside the section") + sections.append(sec) + if not sections: + raise ValueError("spec has no sections") + return sections + + +def compute_offsets(sections: list[Section], durations: list[float]) -> list[float]: + """Medley start time of each section. durations[i] = section length in seconds. + + A crossfade overlaps the previous section's tail by join_duration; a gap inserts + join_duration seconds of silence. Offsets are what the lyric shift adds to source times. + """ + offsets, cursor = [], 0.0 + for i, (sec, dur) in enumerate(zip(sections, durations)): + if i == 0: + start = 0.0 + elif sec.join == "crossfade": + start = cursor - sec.join_duration + else: + start = cursor + sec.join_duration + offsets.append(start) + cursor = start + dur + return offsets + + +def shift_segments(segments, start: float, end: Optional[float], offset: float, prefix: str, tolerance: float = 0.05): + """Keep lyric lines that START inside [start, end) and move them by offset. + + IDs are prefixed so the same song can appear twice (e.g. opener + finale). + """ + out = [] + for seg in segments: + if seg.start_time >= start - tolerance and (end is None or seg.start_time < end): + words = [dataclasses.replace(w, id=f"{prefix}_{w.id}", start_time=w.start_time + offset, + end_time=w.end_time + offset) for w in seg.words] + out.append(dataclasses.replace(seg, id=f"{prefix}_{seg.id}", words=words, + start_time=seg.start_time + offset, end_time=seg.end_time + offset)) + return out + + +def build_filtergraph(sections: list[Section], durations: list[float], gains_db: list[float]) -> str: + """ffmpeg filter_complex joining inputs 0..n-1 (one per section, already trimmed) into [out].""" + chains = [] + for i, (sec, dur, gain) in enumerate(zip(sections, durations, gains_db)): + f = f"[{i}:a]aformat=sample_rates=44100:channel_layouts=stereo,volume={gain:.2f}dB" + if sec.fade_in > 0: + f += f",afade=t=in:st=0:d={sec.fade_in:.3f}:curve=qsin" + if sec.fade_out_start is not None: + st = sec.fade_out_start - sec.start + f += f",afade=t=out:st={st:.3f}:d={dur - st:.3f}:curve=qsin" + nxt = sections[i + 1] if i + 1 < len(sections) else None + if nxt is not None and nxt.join == "gap": + f += f",apad=pad_dur={nxt.join_duration:.3f}" + chains.append(f + f"[s{i}]") + cur = "s0" + for i in range(1, len(sections)): + out = f"x{i}" + if sections[i].join == "crossfade": + chains.append(f"[{cur}][s{i}]acrossfade=d={sections[i].join_duration:.3f}:c1=qsin:c2=qsin[{out}]") + else: + chains.append(f"[{cur}][s{i}]concat=n=2:v=0:a=1[{out}]") + cur = out + chains.append(f"[{cur}]loudnorm=I={FINAL_LUFS}:TP={FINAL_TRUE_PEAK}:LRA=11,aresample=44100[out]") + return ";".join(chains) + + +# --- I/O helpers --------------------------------------------------------------------------- + + +def _run(args, **kw): + return subprocess.run(args, check=True, **kw) + + +def probe_duration(path: str) -> float: + out = _run(["ffprobe", "-v", "error", "-show_entries", "format=duration", "-of", "csv=p=0", path], + capture_output=True, text=True).stdout + return float(out.strip()) + + +def measure_lufs(path: str, start: float, end: Optional[float]) -> float: + args = ["ffmpeg", "-hide_banner", "-nostats", "-ss", str(start)] + if end is not None: + args += ["-to", str(end)] + args += ["-i", path, "-af", "ebur128", "-f", "null", "-"] + res = subprocess.run(args, capture_output=True, text=True) + found = re.findall(r"I:\s+(-?[\d.]+) LUFS", res.stderr) + if res.returncode != 0 or not found: + raise RuntimeError(f"loudness measurement failed for {path}: {res.stderr.strip()[-300:]}") + return float(found[-1]) + + +def gcs_download(src: str, dst: str, required: bool = True) -> bool: + """Download gs://BUCKET/src via the gcloud CLI (the Python client stalled for ~1h on large FLACs).""" + os.makedirs(os.path.dirname(dst), exist_ok=True) + args = ["gcloud", "storage", "cp", f"gs://{BUCKET}/{src}", dst] + if os.environ.get("MEDLEY_GCLOUD_ACCOUNT"): + args.insert(3, f"--account={os.environ['MEDLEY_GCLOUD_ACCOUNT']}") + result = subprocess.run(args, capture_output=True, text=True) + if result.returncode != 0: + if required: + raise RuntimeError(f"download failed for gs://{BUCKET}/{src}: {result.stderr.strip()[-300:]}") + return False + return True + + +def job_dir(workdir: str, job_id: str) -> str: + return os.path.join(workdir, job_id) + + +def load_job_meta(workdir: str, job_id: str) -> dict: + with open(os.path.join(job_dir(workdir, job_id), "job.json")) as f: + return json.load(f) + + +def variant_path(workdir: str, job_id: str, variant: str) -> str: + d = job_dir(workdir, job_id) + stem = VARIANTS[variant] + if stem is None: + return os.path.join(d, "input", os.path.basename(load_job_meta(workdir, job_id)["input_media_gcs_path"])) + return os.path.join(d, "stems", stem) + + +def section_durations(sections: list[Section], workdir: str) -> list[float]: + durs = [] + for sec in sections: + end = sec.end if sec.end is not None else probe_duration(variant_path(workdir, sec.job_id, GUIDE_VARIANT)) + durs.append(end - sec.start) + return durs + + +# --- Commands ------------------------------------------------------------------------------ + + +def cmd_fetch(spec: dict, workdir: str): + from google.cloud import firestore + db = firestore.Client(project=PROJECT) + for job_id in dict.fromkeys(s["job_id"] for s in spec["sections"]): + doc = db.collection("jobs").document(job_id).get() + if not doc.exists: + raise SystemExit(f"job {job_id} not found") + job = doc.to_dict() + if job.get("status") != "complete": + log.warning("job %s is %s, not complete: lyrics may not be reviewed yet", job_id, job.get("status")) + d = job_dir(workdir, job_id) + os.makedirs(os.path.join(d, "stems"), exist_ok=True) + os.makedirs(os.path.join(d, "input"), exist_ok=True) + meta = {k: job.get(k) for k in ("job_id", "artist", "title", "status", "input_media_gcs_path", + "style_params_gcs_path", "style_assets")} + with open(os.path.join(d, "job.json"), "w") as f: + json.dump(meta, f, indent=1, default=str) + wanted = {job["input_media_gcs_path"]: os.path.join(d, "input", os.path.basename(job["input_media_gcs_path"]))} + for name in ("instrumental_with_backing", "instrumental_clean", "lead_vocals", "backing_vocals"): + wanted[f"jobs/{job_id}/stems/{name}.flac"] = os.path.join(d, "stems", f"{name}.flac") + for name in ("corrections.json", "corrections_updated.json"): + wanted[f"jobs/{job_id}/lyrics/{name}"] = os.path.join(d, name) + for src, dst in wanted.items(): + optional = "corrections_updated" in src or src.endswith(("lead_vocals.flac", "backing_vocals.flac")) + if not gcs_download(src, dst, required=not optional) and not optional: + log.warning("missing gs://%s/%s", BUCKET, src) + log.info("fetched %s - %s (%s)", job.get("artist"), job.get("title"), job_id) + + +def cmd_audio(spec: dict, workdir: str, outdir: str): + sections = parse_spec(spec) + durations = section_durations(sections, workdir) + os.makedirs(outdir, exist_ok=True) + for variant in VARIANTS: + inputs, gains = [], [] + for sec in sections: + p = variant_path(workdir, sec.job_id, variant) + gains.append(SEGMENT_TARGET_LUFS - measure_lufs(p, sec.start, sec.end)) + inputs += ["-ss", str(sec.start)] + (["-to", str(sec.end)] if sec.end is not None else []) + ["-i", p] + wav = os.path.join(outdir, f"medley_{variant}.wav") + _run(["ffmpeg", "-y", "-hide_banner", "-loglevel", "error"] + inputs + + ["-filter_complex", build_filtergraph(sections, durations, gains), "-map", "[out]", "-c:a", "pcm_s24le", wav]) + _run(["ffmpeg", "-y", "-hide_banner", "-loglevel", "error", "-i", wav, "-c:a", "libmp3lame", "-b:a", "320k", + wav[:-4] + ".mp3"]) + log.info("%s: %.1fs", wav, probe_duration(wav)) + offsets = compute_offsets(sections, durations) + with open(os.path.join(outdir, "timeline.json"), "w") as f: + json.dump([{"label": s.label, "job_id": s.job_id, "source_start": s.start, "source_end": s.start + d, + "medley_start": o, "medley_end": o + d} for s, d, o in zip(sections, durations, offsets)], f, indent=1) + + +def _load_reviewed_lyrics(workdir: str, job_id: str): + from karaoke_gen.lyrics_transcriber.correction.operations import CorrectionOperations + from karaoke_gen.lyrics_transcriber.types import CorrectionResult + d = job_dir(workdir, job_id) + with open(os.path.join(d, "corrections.json")) as f: + base = CorrectionResult.from_dict(json.load(f)) + updated = os.path.join(d, "corrections_updated.json") + if os.path.exists(updated): + with open(updated) as f: + data = json.load(f) + if isinstance(data, dict) and "corrections" in data: + return CorrectionOperations.update_correction_result_with_data(base, data) + return base + + +def cmd_video(spec: dict, workdir: str, outdir: str, resolution: str): + from karaoke_gen.lyrics_transcriber.core.config import OutputConfig + from karaoke_gen.lyrics_transcriber.output.countdown_processor import CountdownProcessor + from karaoke_gen.lyrics_transcriber.output.generator import OutputGenerator + from karaoke_gen.style_loader import get_end_format, get_intro_format, load_styles_from_gcs + from karaoke_gen.video_generator import VideoGenerator + + sections = parse_spec(spec) + durations = section_durations(sections, workdir) + offsets = compute_offsets(sections, durations) + karaoke_wav = os.path.join(outdir, f"medley_{KARAOKE_VARIANT}.wav") + guide_wav = os.path.join(outdir, f"medley_{GUIDE_VARIANT}.wav") + if not os.path.exists(karaoke_wav): + raise SystemExit("run the `audio` step first") + render_dir = os.path.join(outdir, f"render_{resolution}") + cache_dir = os.path.join(render_dir, "cache") + os.makedirs(cache_dir, exist_ok=True) + + lyrics = {j: _load_reviewed_lyrics(workdir, j) for j in dict.fromkeys(s.job_id for s in sections)} + segments = [] + for n, (sec, dur, off) in enumerate(zip(sections, durations, offsets)): + segments += shift_segments(lyrics[sec.job_id].corrected_segments, sec.start, sec.start + dur, + off - sec.start, prefix=f"m{n}") + first = lyrics[sections[0].job_id] + medley = dataclasses.replace(first, corrected_segments=segments, resized_segments=[]) + for seg in segments: + log.info("%7.2f-%7.2f %s", seg.start_time, seg.end_time, seg.text) + + # Theme: the style of the job named in the spec (default: first section's job) + theme_job = load_job_meta(workdir, spec.get("theme_job_id", sections[0].job_id)) + styles_path, _ = load_styles_from_gcs(theme_job["style_params_gcs_path"], theme_job["style_assets"], + os.path.join(render_dir, "style"), gcs_download, log) + with open(styles_path) as f: + style = json.load(f) + + # Same countdown gen adds when the first word starts within 3s (pads audio + shifts lyrics) + medley, audio, _, pad = CountdownProcessor(cache_dir=cache_dir, logger=log).process( + correction_result=medley, audio_filepath=karaoke_wav) + + artist, title = spec["artist"], spec["title"] + vg = VideoGenerator(logger=log, ffmpeg_base_command="ffmpeg -hide_banner -loglevel error -nostats -y", + render_bounding_boxes=False, output_png=True, output_jpg=False) + title_png, end_png = os.path.join(render_dir, "title"), os.path.join(render_dir, "end") + vg.create_title_video(artist=artist, title=title, format=get_intro_format(style), output_image_filepath_noext=title_png, + output_video_filepath=title_png + ".mov", existing_title_image=None, intro_video_duration=0) + vg.create_end_video(artist=artist, title=title, format=get_end_format(style), output_image_filepath_noext=end_png, + output_video_filepath=end_png + ".mov", existing_end_image=None, end_video_duration=0) + + cfg = OutputConfig(output_dir=render_dir, cache_dir=cache_dir, output_styles_json=styles_path, render_video=True, + generate_cdg=False, generate_plain_text=True, generate_lrc=True, video_resolution=resolution) + outputs = OutputGenerator(cfg, log).generate_outputs( + transcription_corrected=medley, lyrics_results={}, output_prefix=title, audio_filepath=audio, + artist=artist, title=title) + + w, h = {"4k": (3840, 2160), "1080p": (1920, 1080), "720p": (1280, 720), "360p": (640, 360)}[resolution] + intro_s, end_s = style["intro"]["video_duration"], style["end"]["video_duration"] + master = os.path.join(outdir, f"{title} (Karaoke {resolution}).mp4") + scale = f"scale={w}:{h},fps=30,format=yuv420p,setsar=1" + _run(["ffmpeg", "-hide_banner", "-loglevel", "error", "-y", + "-loop", "1", "-t", str(intro_s), "-i", title_png + ".png", "-f", "lavfi", "-t", str(intro_s), "-i", "anullsrc=r=44100:cl=stereo", + "-i", outputs.video, + "-loop", "1", "-t", str(end_s), "-i", end_png + ".png", "-f", "lavfi", "-t", str(end_s), "-i", "anullsrc=r=44100:cl=stereo", + "-filter_complex", f"[0:v]{scale}[t];[2:v]{scale}[k];[3:v]{scale}[e];[t][k][e]concat=n=3:v=1:a=0[v];[1:a][2:a][4:a]concat=n=3:v=0:a=1[a]", + "-map", "[v]", "-map", "[a]", "-c:v", "libx264", "-preset", "slow", "-crf", "18", "-c:a", "aac", "-b:a", "320k", + "-movflags", "+faststart", master]) + # Guide: same picture, original-vocals medley delayed by title + countdown (for checking sync) + delay_ms = int((intro_s + pad) * 1000) + guide = os.path.join(outdir, f"{title} (With Vocals {resolution}).mp4") + _run(["ffmpeg", "-hide_banner", "-loglevel", "error", "-y", "-i", master, "-i", guide_wav, + "-filter_complex", f"[1:a]adelay={delay_ms}|{delay_ms},apad[a]", "-map", "0:v", "-map", "[a]", "-shortest", + "-c:v", "copy", "-c:a", "aac", "-b:a", "320k", "-movflags", "+faststart", guide]) + if outputs.lrc: + _run(["cp", outputs.lrc, os.path.join(outdir, f"{title} (Karaoke).lrc")]) + log.info("karaoke: %s\nguide: %s", master, guide) + + +def main(argv=None): + p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("command", choices=["fetch", "audio", "video"]) + p.add_argument("spec") + p.add_argument("--resolution", default="4k", choices=["4k", "1080p", "720p", "360p"]) + args = p.parse_args(argv) + logging.basicConfig(level=logging.INFO, format="%(message)s") + with open(args.spec) as f: + spec = json.load(f) + parse_spec(spec) # validate early + base = os.path.dirname(os.path.abspath(args.spec)) + workdir = os.path.join(base, spec.get("workdir", "work")) + outdir = os.path.join(base, spec.get("outdir", "out")) + if args.command == "fetch": + cmd_fetch(spec, workdir) + elif args.command == "audio": + cmd_audio(spec, workdir, outdir) + else: + cmd_video(spec, workdir, outdir, args.resolution) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/unit/test_make_medley.py b/tests/unit/test_make_medley.py new file mode 100644 index 00000000..abee2f2e --- /dev/null +++ b/tests/unit/test_make_medley.py @@ -0,0 +1,109 @@ +"""Unit tests for the medley builder's pure helpers (scripts/medley/make_medley.py). + +The script isn't part of the installed package, so we load it by path. +""" + +import importlib.util +import sys +from pathlib import Path + +import pytest + +from karaoke_gen.lyrics_transcriber.types import LyricsSegment, Word + +_SCRIPT = Path(__file__).resolve().parents[2] / "scripts" / "medley" / "make_medley.py" +_spec = importlib.util.spec_from_file_location("make_medley", _SCRIPT) +medley = importlib.util.module_from_spec(_spec) +sys.modules[_spec.name] = medley +_spec.loader.exec_module(medley) + + +EXAMPLE = { + "title": "T", "artist": "A", + "sections": [ + {"job_id": "a", "start": 0.0, "end": 54.0}, + {"job_id": "b", "start": 0.0, "end": 52.0, "join": "crossfade", "join_duration": 2.0}, + {"job_id": "c", "start": 0.0, "end": 57.0}, + {"job_id": "d", "start": 0.0, "end": 82.0, "fade_out_start": 80.7}, + {"job_id": "a", "start": 132.2, "end": None, "join": "gap", "join_duration": 0.7, "fade_in": 0.5}, + ], +} + + +def _segment(seg_id: str, start: float, end: float) -> LyricsSegment: + words = [Word(id=f"{seg_id}w1", text="hello", start_time=start, end_time=(start + end) / 2), + Word(id=f"{seg_id}w2", text="there", start_time=(start + end) / 2, end_time=end)] + return LyricsSegment(id=seg_id, text="hello there", words=words, start_time=start, end_time=end) + + +class TestParseSpec: + def test_defaults_to_two_second_crossfade(self): + sections = medley.parse_spec(EXAMPLE) + assert sections[2].join == "crossfade" + assert sections[2].join_duration == 2.0 + + def test_gap_and_pot_fields(self): + sections = medley.parse_spec(EXAMPLE) + assert sections[3].fade_out_start == 80.7 + assert sections[4].join == "gap" and sections[4].join_duration == 0.7 + assert sections[4].fade_in == 0.5 and sections[4].end is None + + @pytest.mark.parametrize("bad", [ + {"sections": []}, + {"sections": [{"job_id": "a", "start": 10, "end": 5}]}, + {"sections": [{"job_id": "a", "start": 0, "end": 10, "join": "splice"}]}, + {"sections": [{"job_id": "a", "start": 0, "end": 10, "fade_out_start": 12}]}, + ]) + def test_rejects_invalid_specs(self, bad): + with pytest.raises(ValueError): + medley.parse_spec(bad) + + +class TestComputeOffsets: + def test_matches_the_country_medley_timeline(self): + sections = medley.parse_spec(EXAMPLE) + offsets = medley.compute_offsets(sections, [54.0, 52.0, 57.0, 82.0, 199.133 - 132.2]) + assert offsets == pytest.approx([0.0, 52.0, 102.0, 157.0, 239.7]) + + def test_gap_inserts_silence_and_crossfade_overlaps(self): + sections = medley.parse_spec({"sections": [ + {"job_id": "a", "start": 0, "end": 10}, + {"job_id": "b", "start": 0, "end": 10, "join": "gap", "join_duration": 1.0}, + {"job_id": "c", "start": 0, "end": 10, "join": "crossfade", "join_duration": 3.0}, + ]}) + assert medley.compute_offsets(sections, [10, 10, 10]) == pytest.approx([0, 11, 18]) + + +class TestShiftSegments: + def test_keeps_lines_starting_inside_the_cut_and_shifts_them(self): + segs = [_segment("s1", 1.0, 3.0), _segment("s2", 50.0, 53.0), _segment("s3", 54.2, 57.0)] + out = medley.shift_segments(segs, start=0.0, end=54.0, offset=100.0, prefix="m0") + assert [s.id for s in out] == ["m0_s1", "m0_s2"] + assert out[1].start_time == pytest.approx(150.0) + assert [w.start_time for w in out[1].words] == pytest.approx([150.0, 151.5]) + assert out[1].words[0].id == "m0_s2w1" + + def test_open_ended_section_keeps_everything_after_start(self): + segs = [_segment("s1", 120.0, 125.0), _segment("s2", 138.0, 140.0), _segment("s3", 180.0, 184.0)] + out = medley.shift_segments(segs, start=132.2, end=None, offset=107.5, prefix="m4") + assert [s.id for s in out] == ["m4_s2", "m4_s3"] + assert out[0].start_time == pytest.approx(245.5) + + def test_same_song_twice_gets_unique_ids(self): + segs = [_segment("s1", 1.0, 3.0)] + a = medley.shift_segments(segs, 0.0, 10.0, 0.0, "m0") + b = medley.shift_segments(segs, 0.0, 10.0, 50.0, "m4") + assert a[0].id != b[0].id + assert segs[0].start_time == 1.0 # originals untouched + + +class TestBuildFiltergraph: + def test_crossfades_gap_and_pot_fades(self): + sections = medley.parse_spec(EXAMPLE) + graph = medley.build_filtergraph(sections, [54.0, 52.0, 57.0, 82.0, 66.9], [0.0, 1.0, -1.0, 0.5, 2.0]) + assert graph.count("acrossfade=d=2.000") == 3 + assert "afade=t=out:st=80.700:d=1.300" in graph # pot out on section 4 + assert "apad=pad_dur=0.700" in graph # silence before the gap join + assert "afade=t=in:st=0:d=0.500" in graph # pot in on the finale + assert "[x3][s4]concat=n=2:v=0:a=1[x4]" in graph + assert graph.endswith("[x4]loudnorm=I=-14.0:TP=-1.0:LRA=11,aresample=44100[out]")