mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
fix: rescue repetitive-song reference lyrics the relevance filter undercounts (v0.279.4) (#1201)
@coderabbitai ignore ## Why Andrew: job 5c1d24eb loaded no reference lyrics. LRCLIB and Genius both returned the correct song, but the relevance filter scored them 26% and 25%, below its 30% threshold. Anchor coverage credits a repeated phrase only once, so chorus-heavy songs undercount. The transcription was decent, with a few consistent mishearings. ## What `filter_irrelevant_sources(..., transcribed_text=)` now does a second, **additive** check. A source the anchor filter rejects is kept when its word-trigram overlap with the transcription clears four guards: - **≥ 25%** of reference trigrams matched, repeats counted; - **≥ 20** reference trigrams, so tiny references can't score by chance; - **≥ 10** matches; - **≥ 15 distinct** matched trigrams, so a shared refrain alone can't pass. Trigrams must contain three distinct words, so "na na na" and "oh oh oh" don't count. Tokens are NFC-normalized and keep combining marks. Both call sites pass the transcription: the corrector, which also serves the review UI's lyrics search and add-source flow, and the controller's uncorrected path. Without `transcribed_text`, behaviour is unchanged. The rescue **never rejects** a source the anchor filter keeps. The serialized `rejected_sources` metadata is unchanged. ## Calibration (459 recent production jobs; per-job reference txt vs `(Lyrics Uncorrected).txt`) - **Wrong-song baseline:** 491k cross-job pairs, excluding pairs with the same title. - Worst genuine pair: 9.4%; p99.99: 5.5%. - The only cross-job pairs that pass are one job whose audio is another job's song under a wrong title. Their transcriptions overlap 45%, where unrelated songs score under 10%. - A naive n-gram rule would have admitted a wrong song at 41% through shared "na na" filler. Requiring three distinct words removes it. - **Rescued:** 22 sources that are currently rejected, **all the correct song**, scoring 26–88%. Examples: the 5c1d24eb song, Deacon Blue, UB40, Atomic Kitten, Sezen Aksu, Grateful Dead, Etherwood, Sub Focus, Terry Allen. - **Left rejected by the distinct-trigram guard:** one refrain-only drum & bass track (8 distinct matches; it's the right song but has the risk profile). This was a deliberate, cautious choice. - No lyrics text is committed; the tests use invented lyrics. ## Testing - `tests/correction/test_relevance.py`, 17 passed (8 new): - a repetitive correct song is rescued; - no transcription means unchanged behaviour; - a wrong song is still rejected; - filler doesn't count; - a tiny reference isn't rescued; - a source the anchor filter keeps is never rejected; - a shared refrain alone can't rescue; - combining marks are kept. - `tests/unit`: 2184 passed. Running `tests/unit`, `tests/correction` and the backend review tests together gives the same 12 order-dependent failures on untouched main; they pass separately, as CI runs them. - Local CodeRabbit: 2 cycles. Fixed the combining-marks and distinct-trigram findings. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
b67b273056
commit
e050cb8b3d
6 files changed
+190
-4
No files matched your search
@@ -74,6 +74,7 @@ The "servers unavailable" banner (49 episodes in 2 weeks) was mostly the API's o
|
||||
- the tenant middleware (`get_tenant_config`, a GCS read on a cache miss).
|
||||
|
||||
Job-log writes made on the API loop now go to one background thread (`backend/services/log_write_offload.py`). They stay inline in worker threads and Cloud Run Jobs, where nothing else waits on the loop. The auth dependencies and tenant middleware use `asyncio.to_thread`. The guard now holds every `async def` in `api/dependencies.py` and `middleware/` to `BLOCKING_CALLS`, not just decorated routes. Still on the loop: about 70 direct `get_job`/`update_job` calls in `async def` routes. Each is a single 50-200 ms Firestore round-trip and rarely shows up in the watchdog. Move them when one does.
|
||||
- **The reference-lyrics relevance filter undercounted repetitive songs** (2026-10-19). Anchor coverage credits a repeated phrase once, so correct references for chorus-heavy songs scored 25-30% and were dropped. On 459 jobs that was 25 sources across about a dozen songs (22 rescued; one refrain-only D&B track left out), e.g. Wolfmother "Apple Tree" (job 5c1d24eb). `correction/relevance.py` now rescues a rejected source whose distinct-word trigram overlap with the transcription is at least 25% (with at least 20 reference trigrams, 10 matches and 15 distinct matched trigrams). The worst wrong-song pair across 491k cross-job pairs was 9.4%. It only adds sources and never rejects a kept one. A naive n-gram rule would have let "na na na"-style lyrics through at 41%, so trigrams must have three distinct words. To recalibrate, compare each job's `jobs/{id}/lyrics/*(Lyrics <Source>).txt` with its `(Lyrics Uncorrected).txt`, and use cross-job pairs as the wrong-song baseline.
|
||||
- **Detection:** `backend/services/loop_watchdog.py` logs `EVENT_LOOP_STALL` with the loop thread's stack and a `culprit=file:line`. Grep for it before guessing.
|
||||
- **Alert on impact, not on gaps** (v0.268.5). After the CPU-throttle discount, the stall alert still flapped. Cloud Run autoscaling starts extra instances that serve one request, then sit idle and throttled while the startup warmup thread, which spends 10+ minutes loading SpaCy, competes for slices. Every 10-40s ERROR "stall" over 4 days (11 of them) was on an instance with **zero** requests, and all had an empty culprit. A freeze advances the 1/5/15s sample schedule without sampling, so whatever happens after the thaw has no stack. The watchdog now takes a `RequestTracker`, fed by the outermost pure-ASGI `RequestTrackingMiddleware`. A stall reaches ERROR only if a request was in flight at its start or arrived during it. A request that queued during the stall is counted if it starts within the 0.5s grace window after the loop resumes. Check whether anyone waited before chasing a culprit: join `labels.instanceId` between the stall lines and the request logs.
|
||||
- **Async services with sync bodies** (v0.268.7). `PushNotificationService.send_push` was an `async def` with no real awaits: a Firestore read, a `webpush` POST per device, and a Firestore update, all on the loop. The route guard can't see it because it's a service method. The watchdog caught it at 1.2s with a request waiting. The body is now `send_push_sync`, and `send_push` calls `await asyncio.to_thread(...)`. When `requests_affected>0` lines name a `backend/services/*` frame, that service method is the fix site. Also since v0.268.7: if a process freeze skips every sample point and the loop is still blocked once the process thaws, the watchdog takes one sample then, so the stall gets a culprit.
|
||||
|
||||
@@ -563,6 +563,7 @@ class LyricsTranscriber:
|
||||
from karaoke_gen.lyrics_transcriber.correction.relevance import filter_irrelevant_sources
|
||||
filtered_lyrics, rejected_sources = filter_irrelevant_sources(
|
||||
self.results.lyrics_results, anchor_sequences, logger=self.logger,
|
||||
transcribed_text=transcribed_text,
|
||||
)
|
||||
if len(filtered_lyrics) < len(self.results.lyrics_results):
|
||||
self.logger.info(
|
||||
|
||||
@@ -156,7 +156,7 @@ class LyricsCorrector:
|
||||
|
||||
# Filter irrelevant reference sources based on anchor match quality
|
||||
filtered_lyrics, rejected_sources = filter_irrelevant_sources(
|
||||
lyrics_results, anchor_sequences, logger=self.logger,
|
||||
lyrics_results, anchor_sequences, logger=self.logger, transcribed_text=transcribed_text,
|
||||
)
|
||||
|
||||
# If sources were filtered, re-run anchor finding with only valid sources
|
||||
|
||||
@@ -3,10 +3,20 @@
|
||||
Computes how well each reference source matches the transcription based on
|
||||
anchor sequence coverage. Sources below the relevance threshold are filtered
|
||||
out to prevent wrong-song lyrics from polluting corrections.
|
||||
|
||||
Anchor coverage undercounts songs that repeat the same lines: correct references for
|
||||
Wolfmother "Apple Tree", Deacon Blue, UB40 "Wedding Day" etc. scored 25-30% and were
|
||||
dropped. A source the anchor filter rejects is therefore rescued when its word-trigram
|
||||
overlap with the transcription (repeats counted, filler like "na na na" excluded) is high.
|
||||
Calibrated on 459 production jobs (2026-10-19): wrong-song pairs max 9.4% (p99.99 5.5%,
|
||||
491k cross-job pairs), the 22 sources it rescues were all the correct song (26-88%).
|
||||
The rescue can only add sources; it never rejects one the anchor filter keeps.
|
||||
"""
|
||||
import logging
|
||||
import unicodedata
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass
|
||||
from typing import Dict, List, Tuple
|
||||
from typing import Dict, List, Optional, Tuple
|
||||
|
||||
from karaoke_gen.lyrics_transcriber.types import AnchorSequence, LyricsData
|
||||
|
||||
@@ -14,6 +24,13 @@ from karaoke_gen.lyrics_transcriber.types import AnchorSequence, LyricsData
|
||||
# See scripts/analyze_reference_relevance.py and the design spec for methodology.
|
||||
MIN_REFERENCE_RELEVANCE = 0.30 # Empirical: wrong-song max 3.5%, correct-song min 26.8%
|
||||
|
||||
# N-gram rescue (see module docstring): ~2.7x the worst wrong-song score seen.
|
||||
MIN_NGRAM_RELEVANCE = 0.25
|
||||
MIN_NGRAM_REFERENCE = 20 # distinct-word trigrams in the reference; tiny references score by chance
|
||||
MIN_NGRAM_MATCHES = 10
|
||||
MIN_NGRAM_DISTINCT_MATCHES = 15 # distinct matched trigrams: a shared refrain alone can't pass (rescues had 22+, bar one 8-phrase D&B track left rejected)
|
||||
_NGRAM = 3
|
||||
|
||||
|
||||
@dataclass
|
||||
class SourceRelevanceResult:
|
||||
@@ -24,6 +41,7 @@ class SourceRelevanceResult:
|
||||
total_words: int
|
||||
track_name: str = ""
|
||||
artist_names: str = ""
|
||||
ngram_relevance: Optional[float] = None # set when the n-gram rescue was evaluated
|
||||
|
||||
|
||||
def compute_source_relevance(
|
||||
@@ -68,11 +86,56 @@ def compute_source_relevance(
|
||||
)
|
||||
|
||||
|
||||
def _is_word_char(ch: str) -> bool:
|
||||
# Letters/digits plus combining marks (Devanagari vowel signs etc.), which \w drops.
|
||||
return ch.isalnum() or ch == "'" or unicodedata.category(ch).startswith("M")
|
||||
|
||||
|
||||
def _words(text: str) -> list:
|
||||
text = unicodedata.normalize("NFC", text.lower().replace("\u2019", "'"))
|
||||
words, current = [], []
|
||||
for ch in text:
|
||||
if _is_word_char(ch):
|
||||
current.append(ch)
|
||||
elif current:
|
||||
words.append("".join(current).strip("'"))
|
||||
current = []
|
||||
if current:
|
||||
words.append("".join(current).strip("'"))
|
||||
return [w for w in words if w]
|
||||
|
||||
|
||||
def _trigrams(text: str) -> Counter:
|
||||
"""Word trigrams with three distinct words (so "na na na" / "oh oh oh" don't count)."""
|
||||
words = _words(text)
|
||||
grams = (tuple(words[i:i + _NGRAM]) for i in range(len(words) - _NGRAM + 1))
|
||||
return Counter(g for g in grams if len(set(g)) == _NGRAM)
|
||||
|
||||
|
||||
def compute_ngram_relevance(lyrics_data: LyricsData, transcribed_text: str) -> Tuple[float, int, int, int]:
|
||||
"""(score, matched, total, distinct_matched): share of the reference's trigrams, repeats
|
||||
included, found in the transcription, and how many different trigrams that match covers."""
|
||||
reference_text = " ".join(w.text for seg in lyrics_data.segments for w in seg.words)
|
||||
ref = _trigrams(reference_text)
|
||||
total = sum(ref.values())
|
||||
if not total:
|
||||
return 0.0, 0, 0, 0
|
||||
common = ref & _trigrams(transcribed_text)
|
||||
matched = sum(common.values())
|
||||
return matched / total, matched, total, len(common)
|
||||
|
||||
|
||||
def _ngram_rescues(score: float, matched: int, total: int, distinct_matched: int) -> bool:
|
||||
return (score >= MIN_NGRAM_RELEVANCE and total >= MIN_NGRAM_REFERENCE
|
||||
and matched >= MIN_NGRAM_MATCHES and distinct_matched >= MIN_NGRAM_DISTINCT_MATCHES)
|
||||
|
||||
|
||||
def filter_irrelevant_sources(
|
||||
lyrics_results: Dict[str, LyricsData],
|
||||
anchor_sequences: List[AnchorSequence],
|
||||
min_relevance: float = MIN_REFERENCE_RELEVANCE,
|
||||
logger: logging.Logger = None,
|
||||
transcribed_text: Optional[str] = None,
|
||||
) -> Tuple[Dict[str, LyricsData], Dict[str, SourceRelevanceResult]]:
|
||||
"""Filter out reference sources below the relevance threshold.
|
||||
|
||||
@@ -81,6 +144,8 @@ def filter_irrelevant_sources(
|
||||
anchor_sequences: Anchor sequences from the anchor finder.
|
||||
min_relevance: Minimum relevance score to keep a source.
|
||||
logger: Optional logger.
|
||||
transcribed_text: The transcription. When given, a source below ``min_relevance``
|
||||
is still kept if its trigram overlap passes the n-gram rescue.
|
||||
|
||||
Returns:
|
||||
Tuple of (filtered_lyrics_results, rejected_sources).
|
||||
@@ -103,10 +168,22 @@ def filter_irrelevant_sources(
|
||||
f"{result.relevance:.1%} ({result.matched_words}/{result.total_words} words)"
|
||||
)
|
||||
else:
|
||||
if transcribed_text:
|
||||
score, matched, total, distinct = compute_ngram_relevance(lyrics_data, transcribed_text)
|
||||
result.ngram_relevance = score
|
||||
if _ngram_rescues(score, matched, total, distinct):
|
||||
filtered[source_name] = lyrics_data
|
||||
logger.info(
|
||||
f"Source '{source_name}' kept by n-gram rescue: anchor relevance "
|
||||
f"{result.relevance:.1%} ({result.matched_words}/{result.total_words} words), "
|
||||
f"trigram overlap {score:.1%} ({matched}/{total})"
|
||||
)
|
||||
continue
|
||||
rejected[source_name] = result
|
||||
ngram = f", trigram overlap {result.ngram_relevance:.1%}" if result.ngram_relevance is not None else ""
|
||||
logger.info(
|
||||
f"Source '{source_name}' rejected by relevance filter: "
|
||||
f"{result.relevance:.1%} ({result.matched_words}/{result.total_words} words)"
|
||||
f"{result.relevance:.1%} ({result.matched_words}/{result.total_words} words){ngram}"
|
||||
)
|
||||
|
||||
return filtered, rejected
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "karaoke-gen"
|
||||
version = "0.279.3"
|
||||
version = "0.279.4"
|
||||
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
|
||||
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
|
||||
license = "MIT"
|
||||
|
||||
@@ -255,3 +255,110 @@ class TestCorrectorRelevanceIntegration:
|
||||
assert result.corrections_made == 0
|
||||
# Original segments should be preserved
|
||||
assert len(result.original_segments) == 1
|
||||
|
||||
|
||||
# --- n-gram rescue (repetitive songs) -------------------------------------------------
|
||||
# Invented lyrics with the same shape as the production failure: two lines repeated many
|
||||
# times, transcribed with a few consistent mishearings.
|
||||
|
||||
_CHORUS = [
|
||||
["hello", "neighbour", "will", "you", "wave", "at", "me"],
|
||||
["i", "am", "the", "one", "who", "painted", "the", "garden", "gate"],
|
||||
]
|
||||
_VERSE = [
|
||||
["morning", "comes", "slowly", "over", "the", "hill"],
|
||||
["coffee", "goes", "cold", "while", "the", "radio", "plays"],
|
||||
]
|
||||
|
||||
|
||||
def _song_lines() -> list[list[str]]:
|
||||
return _CHORUS * 4 + _VERSE + _CHORUS * 4 + _VERSE + _CHORUS * 2
|
||||
|
||||
|
||||
def _misheard(lines: list[list[str]]) -> str:
|
||||
swaps = {"neighbour": "nathan", "who": "that", "garden": "guarded", "gate": "grape"}
|
||||
return " ".join(swaps.get(w, w) for line in lines for w in line)
|
||||
|
||||
|
||||
def _repetitive_case():
|
||||
lines = _song_lines()
|
||||
lyrics = _make_lyrics_data(lines, "lrclib")
|
||||
# Anchors that only credit the first occurrence of each phrase in the reference,
|
||||
# the undercount that rejected correct lyrics for repetitive songs in production.
|
||||
first_ids = [w.id for seg in lyrics.segments[:2] for w in seg.words]
|
||||
anchor = _make_anchor(["t1"], {"lrclib": first_ids})
|
||||
return lyrics, anchor, _misheard(lines)
|
||||
|
||||
|
||||
def test_repetitive_correct_song_is_rescued_by_ngrams():
|
||||
from karaoke_gen.lyrics_transcriber.correction.relevance import compute_ngram_relevance
|
||||
|
||||
lyrics, anchor, transcription = _repetitive_case()
|
||||
assert compute_source_relevance("lrclib", lyrics, [anchor]).relevance < 0.30
|
||||
score, matched, total, distinct = compute_ngram_relevance(lyrics, transcription)
|
||||
assert score >= 0.25 and matched >= 10 and total >= 20 and distinct >= 15
|
||||
|
||||
kept, rejected = filter_irrelevant_sources({"lrclib": lyrics}, [anchor], transcribed_text=transcription)
|
||||
assert "lrclib" in kept and not rejected
|
||||
|
||||
|
||||
def test_without_transcribed_text_behaviour_is_unchanged():
|
||||
lyrics, anchor, _ = _repetitive_case()
|
||||
kept, rejected = filter_irrelevant_sources({"lrclib": lyrics}, [anchor])
|
||||
assert not kept and "lrclib" in rejected
|
||||
assert rejected["lrclib"].ngram_relevance is None
|
||||
|
||||
|
||||
def test_wrong_song_is_still_rejected():
|
||||
lyrics, anchor, _ = _repetitive_case()
|
||||
other = "rain falls on the quiet harbour and the boats drift out past the lighthouse " * 6
|
||||
kept, rejected = filter_irrelevant_sources({"lrclib": lyrics}, [anchor], transcribed_text=other)
|
||||
assert not kept
|
||||
assert rejected["lrclib"].ngram_relevance < 0.25
|
||||
|
||||
|
||||
def test_filler_repetition_does_not_count():
|
||||
"""Shared "na na na" / "oh oh oh" stretches must not make a wrong song look relevant."""
|
||||
from karaoke_gen.lyrics_transcriber.correction.relevance import compute_ngram_relevance
|
||||
|
||||
filler = [["na"] * 8, ["oh", "oh", "oh", "yeah"]] * 10 + [["dance", "with", "me", "tonight"]]
|
||||
lyrics = _make_lyrics_data(filler, "genius")
|
||||
transcription = "na na na na na na oh oh oh yeah " * 12 + "different words entirely here"
|
||||
score, matched, total, _ = compute_ngram_relevance(lyrics, transcription)
|
||||
assert total < 20 # only distinct-word trigrams count, so filler can't reach the size guard
|
||||
kept, _ = filter_irrelevant_sources({"genius": lyrics}, [], transcribed_text=transcription)
|
||||
assert not kept
|
||||
|
||||
|
||||
def test_tiny_reference_is_not_rescued():
|
||||
lyrics = _make_lyrics_data([["one", "two", "three", "four"]], "spotify")
|
||||
kept, rejected = filter_irrelevant_sources(
|
||||
{"spotify": lyrics}, [], transcribed_text="one two three four"
|
||||
)
|
||||
assert not kept and rejected["spotify"].ngram_relevance == 1.0
|
||||
|
||||
|
||||
def test_sources_passing_anchor_filter_are_never_rejected_by_rescue():
|
||||
lyrics, anchor, _ = _repetitive_case()
|
||||
all_ids = [w.id for seg in lyrics.segments for w in seg.words]
|
||||
full_anchor = _make_anchor(["t1"], {"lrclib": all_ids})
|
||||
kept, rejected = filter_irrelevant_sources(
|
||||
{"lrclib": lyrics}, [full_anchor], transcribed_text="nothing in common at all here"
|
||||
)
|
||||
assert "lrclib" in kept and not rejected
|
||||
|
||||
|
||||
def test_shared_refrain_alone_cannot_rescue():
|
||||
"""A wrong song sharing only a long repeated three-word refrain stays rejected."""
|
||||
refrain = [["na", "hey", "oh"]] * 30 + [["walking", "down", "the", "avenue", "tonight"]]
|
||||
lyrics = _make_lyrics_data(refrain, "genius")
|
||||
transcription = "na hey oh " * 40 + "something else about the sea and sky"
|
||||
kept, rejected = filter_irrelevant_sources({"genius": lyrics}, [], transcribed_text=transcription)
|
||||
assert not kept and rejected["genius"].ngram_relevance >= 0.25 # high score, but too few distinct trigrams
|
||||
|
||||
|
||||
def test_combining_marks_are_kept_in_words():
|
||||
from karaoke_gen.lyrics_transcriber.correction.relevance import _words
|
||||
|
||||
assert _words("कला बाला रामा") != _words("कली बाली रामी")
|
||||
assert _words("Don’t stop") == ["don't", "stop"]
|
||||
Reference in new issue
Block a user