fix: rescue repetitive-song reference lyrics the relevance filter undercounts (v0.279.4) (#1201)

@coderabbitai ignore

## Why
Andrew: job 5c1d24eb loaded no reference lyrics. LRCLIB and Genius both
returned the correct song, but the relevance filter scored them 26% and
25%, below its 30% threshold. Anchor coverage credits a repeated phrase
only once, so chorus-heavy songs undercount. The transcription was
decent, with a few consistent mishearings.

## What
`filter_irrelevant_sources(..., transcribed_text=)` now does a second,
**additive** check. A source the anchor filter rejects is kept when its
word-trigram overlap with the transcription clears four guards:
- **≥ 25%** of reference trigrams matched, repeats counted;
- **≥ 20** reference trigrams, so tiny references can't score by chance;
- **≥ 10** matches;
- **≥ 15 distinct** matched trigrams, so a shared refrain alone can't
pass.

Trigrams must contain three distinct words, so "na na na" and "oh oh oh"
don't count. Tokens are NFC-normalized and keep combining marks.

Both call sites pass the transcription: the corrector, which also serves
the review UI's lyrics search and add-source flow, and the controller's
uncorrected path. Without `transcribed_text`, behaviour is unchanged.
The rescue **never rejects** a source the anchor filter keeps. The
serialized `rejected_sources` metadata is unchanged.

## Calibration (459 recent production jobs; per-job reference txt vs
`(Lyrics Uncorrected).txt`)
- **Wrong-song baseline:** 491k cross-job pairs, excluding pairs with
the same title.
  - Worst genuine pair: 9.4%; p99.99: 5.5%.
- The only cross-job pairs that pass are one job whose audio is another
job's song under a wrong title. Their transcriptions overlap 45%, where
unrelated songs score under 10%.
- A naive n-gram rule would have admitted a wrong song at 41% through
shared "na na" filler. Requiring three distinct words removes it.
- **Rescued:** 22 sources that are currently rejected, **all the correct
song**, scoring 26–88%. Examples: the 5c1d24eb song, Deacon Blue, UB40,
Atomic Kitten, Sezen Aksu, Grateful Dead, Etherwood, Sub Focus, Terry
Allen.
- **Left rejected by the distinct-trigram guard:** one refrain-only drum
& bass track (8 distinct matches; it's the right song but has the risk
profile). This was a deliberate, cautious choice.
- No lyrics text is committed; the tests use invented lyrics.

## Testing
- `tests/correction/test_relevance.py`, 17 passed (8 new):
  - a repetitive correct song is rescued;
  - no transcription means unchanged behaviour;
  - a wrong song is still rejected;
  - filler doesn't count;
  - a tiny reference isn't rescued;
  - a source the anchor filter keeps is never rejected;
  - a shared refrain alone can't rescue;
  - combining marks are kept.
- `tests/unit`: 2184 passed. Running `tests/unit`, `tests/correction`
and the backend review tests together gives the same 12 order-dependent
failures on untouched main; they pass separately, as CI runs them.
- Local CodeRabbit: 2 cycles. Fixed the combining-marks and
distinct-trigram findings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Jw21syo4HizJJjYw2mg2Vp

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 5.5 authored and GitHub committed 2026-10-20 00:08:38 -04:00
1 parent b67b273056
commit e050cb8b3d
6 files changed
+190 -4

No files matched your search

+1
View File
@@ -74,6 +74,7 @@ The "servers unavailable" banner (49 episodes in 2 weeks) was mostly the API's o
- the tenant middleware (`get_tenant_config`, a GCS read on a cache miss).
Job-log writes made on the API loop now go to one background thread (`backend/services/log_write_offload.py`). They stay inline in worker threads and Cloud Run Jobs, where nothing else waits on the loop. The auth dependencies and tenant middleware use `asyncio.to_thread`. The guard now holds every `async def` in `api/dependencies.py` and `middleware/` to `BLOCKING_CALLS`, not just decorated routes. Still on the loop: about 70 direct `get_job`/`update_job` calls in `async def` routes. Each is a single 50-200 ms Firestore round-trip and rarely shows up in the watchdog. Move them when one does.
- **The reference-lyrics relevance filter undercounted repetitive songs** (2026-10-19). Anchor coverage credits a repeated phrase once, so correct references for chorus-heavy songs scored 25-30% and were dropped. On 459 jobs that was 25 sources across about a dozen songs (22 rescued; one refrain-only D&B track left out), e.g. Wolfmother "Apple Tree" (job 5c1d24eb). `correction/relevance.py` now rescues a rejected source whose distinct-word trigram overlap with the transcription is at least 25% (with at least 20 reference trigrams, 10 matches and 15 distinct matched trigrams). The worst wrong-song pair across 491k cross-job pairs was 9.4%. It only adds sources and never rejects a kept one. A naive n-gram rule would have let "na na na"-style lyrics through at 41%, so trigrams must have three distinct words. To recalibrate, compare each job's `jobs/{id}/lyrics/*(Lyrics <Source>).txt` with its `(Lyrics Uncorrected).txt`, and use cross-job pairs as the wrong-song baseline.
- **Detection:** `backend/services/loop_watchdog.py` logs `EVENT_LOOP_STALL` with the loop thread's stack and a `culprit=file:line`. Grep for it before guessing.
- **Alert on impact, not on gaps** (v0.268.5). After the CPU-throttle discount, the stall alert still flapped. Cloud Run autoscaling starts extra instances that serve one request, then sit idle and throttled while the startup warmup thread, which spends 10+ minutes loading SpaCy, competes for slices. Every 10-40s ERROR "stall" over 4 days (11 of them) was on an instance with **zero** requests, and all had an empty culprit. A freeze advances the 1/5/15s sample schedule without sampling, so whatever happens after the thaw has no stack. The watchdog now takes a `RequestTracker`, fed by the outermost pure-ASGI `RequestTrackingMiddleware`. A stall reaches ERROR only if a request was in flight at its start or arrived during it. A request that queued during the stall is counted if it starts within the 0.5s grace window after the loop resumes. Check whether anyone waited before chasing a culprit: join `labels.instanceId` between the stall lines and the request logs.
- **Async services with sync bodies** (v0.268.7). `PushNotificationService.send_push` was an `async def` with no real awaits: a Firestore read, a `webpush` POST per device, and a Firestore update, all on the loop. The route guard can't see it because it's a service method. The watchdog caught it at 1.2s with a request waiting. The body is now `send_push_sync`, and `send_push` calls `await asyncio.to_thread(...)`. When `requests_affected>0` lines name a `backend/services/*` frame, that service method is the fix site. Also since v0.268.7: if a process freeze skips every sample point and the loop is still blocked once the process thaws, the watchdog takes one sample then, so the stall gets a culprit.
@@ -563,6 +563,7 @@ class LyricsTranscriber:
from karaoke_gen.lyrics_transcriber.correction.relevance import filter_irrelevant_sources
filtered_lyrics, rejected_sources = filter_irrelevant_sources(
self.results.lyrics_results, anchor_sequences, logger=self.logger,
transcribed_text=transcribed_text,
)
if len(filtered_lyrics) < len(self.results.lyrics_results):
self.logger.info(
@@ -156,7 +156,7 @@ class LyricsCorrector:
# Filter irrelevant reference sources based on anchor match quality
filtered_lyrics, rejected_sources = filter_irrelevant_sources(
lyrics_results, anchor_sequences, logger=self.logger,
lyrics_results, anchor_sequences, logger=self.logger, transcribed_text=transcribed_text,
)
# If sources were filtered, re-run anchor finding with only valid sources
@@ -3,10 +3,20 @@
Computes how well each reference source matches the transcription based on
anchor sequence coverage. Sources below the relevance threshold are filtered
out to prevent wrong-song lyrics from polluting corrections.
Anchor coverage undercounts songs that repeat the same lines: correct references for
Wolfmother "Apple Tree", Deacon Blue, UB40 "Wedding Day" etc. scored 25-30% and were
dropped. A source the anchor filter rejects is therefore rescued when its word-trigram
overlap with the transcription (repeats counted, filler like "na na na" excluded) is high.
Calibrated on 459 production jobs (2026-10-19): wrong-song pairs max 9.4% (p99.99 5.5%,
491k cross-job pairs), the 22 sources it rescues were all the correct song (26-88%).
The rescue can only add sources; it never rejects one the anchor filter keeps.
"""
import logging
import unicodedata
from collections import Counter
from dataclasses import dataclass
from typing import Dict, List, Tuple
from typing import Dict, List, Optional, Tuple
from karaoke_gen.lyrics_transcriber.types import AnchorSequence, LyricsData
@@ -14,6 +24,13 @@ from karaoke_gen.lyrics_transcriber.types import AnchorSequence, LyricsData
# See scripts/analyze_reference_relevance.py and the design spec for methodology.
MIN_REFERENCE_RELEVANCE = 0.30 # Empirical: wrong-song max 3.5%, correct-song min 26.8%
# N-gram rescue (see module docstring): ~2.7x the worst wrong-song score seen.
MIN_NGRAM_RELEVANCE = 0.25
MIN_NGRAM_REFERENCE = 20 # distinct-word trigrams in the reference; tiny references score by chance
MIN_NGRAM_MATCHES = 10
MIN_NGRAM_DISTINCT_MATCHES = 15 # distinct matched trigrams: a shared refrain alone can't pass (rescues had 22+, bar one 8-phrase D&B track left rejected)
_NGRAM = 3
@dataclass
class SourceRelevanceResult:
@@ -24,6 +41,7 @@ class SourceRelevanceResult:
total_words: int
track_name: str = ""
artist_names: str = ""
ngram_relevance: Optional[float] = None # set when the n-gram rescue was evaluated
def compute_source_relevance(
@@ -68,11 +86,56 @@ def compute_source_relevance(
)
def _is_word_char(ch: str) -> bool:
# Letters/digits plus combining marks (Devanagari vowel signs etc.), which \w drops.
return ch.isalnum() or ch == "'" or unicodedata.category(ch).startswith("M")
def _words(text: str) -> list:
text = unicodedata.normalize("NFC", text.lower().replace("\u2019", "'"))
words, current = [], []
for ch in text:
if _is_word_char(ch):
current.append(ch)
elif current:
words.append("".join(current).strip("'"))
current = []
if current:
words.append("".join(current).strip("'"))
return [w for w in words if w]
def _trigrams(text: str) -> Counter:
"""Word trigrams with three distinct words (so "na na na" / "oh oh oh" don't count)."""
words = _words(text)
grams = (tuple(words[i:i + _NGRAM]) for i in range(len(words) - _NGRAM + 1))
return Counter(g for g in grams if len(set(g)) == _NGRAM)
def compute_ngram_relevance(lyrics_data: LyricsData, transcribed_text: str) -> Tuple[float, int, int, int]:
"""(score, matched, total, distinct_matched): share of the reference's trigrams, repeats
included, found in the transcription, and how many different trigrams that match covers."""
reference_text = " ".join(w.text for seg in lyrics_data.segments for w in seg.words)
ref = _trigrams(reference_text)
total = sum(ref.values())
if not total:
return 0.0, 0, 0, 0
common = ref & _trigrams(transcribed_text)
matched = sum(common.values())
return matched / total, matched, total, len(common)
def _ngram_rescues(score: float, matched: int, total: int, distinct_matched: int) -> bool:
return (score >= MIN_NGRAM_RELEVANCE and total >= MIN_NGRAM_REFERENCE
and matched >= MIN_NGRAM_MATCHES and distinct_matched >= MIN_NGRAM_DISTINCT_MATCHES)
def filter_irrelevant_sources(
lyrics_results: Dict[str, LyricsData],
anchor_sequences: List[AnchorSequence],
min_relevance: float = MIN_REFERENCE_RELEVANCE,
logger: logging.Logger = None,
transcribed_text: Optional[str] = None,
) -> Tuple[Dict[str, LyricsData], Dict[str, SourceRelevanceResult]]:
"""Filter out reference sources below the relevance threshold.
@@ -81,6 +144,8 @@ def filter_irrelevant_sources(
anchor_sequences: Anchor sequences from the anchor finder.
min_relevance: Minimum relevance score to keep a source.
logger: Optional logger.
transcribed_text: The transcription. When given, a source below ``min_relevance``
is still kept if its trigram overlap passes the n-gram rescue.
Returns:
Tuple of (filtered_lyrics_results, rejected_sources).
@@ -103,10 +168,22 @@ def filter_irrelevant_sources(
f"{result.relevance:.1%} ({result.matched_words}/{result.total_words} words)"
)
else:
if transcribed_text:
score, matched, total, distinct = compute_ngram_relevance(lyrics_data, transcribed_text)
result.ngram_relevance = score
if _ngram_rescues(score, matched, total, distinct):
filtered[source_name] = lyrics_data
logger.info(
f"Source '{source_name}' kept by n-gram rescue: anchor relevance "
f"{result.relevance:.1%} ({result.matched_words}/{result.total_words} words), "
f"trigram overlap {score:.1%} ({matched}/{total})"
)
continue
rejected[source_name] = result
ngram = f", trigram overlap {result.ngram_relevance:.1%}" if result.ngram_relevance is not None else ""
logger.info(
f"Source '{source_name}' rejected by relevance filter: "
f"{result.relevance:.1%} ({result.matched_words}/{result.total_words} words)"
f"{result.relevance:.1%} ({result.matched_words}/{result.total_words} words){ngram}"
)
return filtered, rejected
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.279.3"
version = "0.279.4"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"
+107
View File
@@ -255,3 +255,110 @@ class TestCorrectorRelevanceIntegration:
assert result.corrections_made == 0
# Original segments should be preserved
assert len(result.original_segments) == 1
# --- n-gram rescue (repetitive songs) -------------------------------------------------
# Invented lyrics with the same shape as the production failure: two lines repeated many
# times, transcribed with a few consistent mishearings.
_CHORUS = [
["hello", "neighbour", "will", "you", "wave", "at", "me"],
["i", "am", "the", "one", "who", "painted", "the", "garden", "gate"],
]
_VERSE = [
["morning", "comes", "slowly", "over", "the", "hill"],
["coffee", "goes", "cold", "while", "the", "radio", "plays"],
]
def _song_lines() -> list[list[str]]:
return _CHORUS * 4 + _VERSE + _CHORUS * 4 + _VERSE + _CHORUS * 2
def _misheard(lines: list[list[str]]) -> str:
swaps = {"neighbour": "nathan", "who": "that", "garden": "guarded", "gate": "grape"}
return " ".join(swaps.get(w, w) for line in lines for w in line)
def _repetitive_case():
lines = _song_lines()
lyrics = _make_lyrics_data(lines, "lrclib")
# Anchors that only credit the first occurrence of each phrase in the reference,
# the undercount that rejected correct lyrics for repetitive songs in production.
first_ids = [w.id for seg in lyrics.segments[:2] for w in seg.words]
anchor = _make_anchor(["t1"], {"lrclib": first_ids})
return lyrics, anchor, _misheard(lines)
def test_repetitive_correct_song_is_rescued_by_ngrams():
from karaoke_gen.lyrics_transcriber.correction.relevance import compute_ngram_relevance
lyrics, anchor, transcription = _repetitive_case()
assert compute_source_relevance("lrclib", lyrics, [anchor]).relevance < 0.30
score, matched, total, distinct = compute_ngram_relevance(lyrics, transcription)
assert score >= 0.25 and matched >= 10 and total >= 20 and distinct >= 15
kept, rejected = filter_irrelevant_sources({"lrclib": lyrics}, [anchor], transcribed_text=transcription)
assert "lrclib" in kept and not rejected
def test_without_transcribed_text_behaviour_is_unchanged():
lyrics, anchor, _ = _repetitive_case()
kept, rejected = filter_irrelevant_sources({"lrclib": lyrics}, [anchor])
assert not kept and "lrclib" in rejected
assert rejected["lrclib"].ngram_relevance is None
def test_wrong_song_is_still_rejected():
lyrics, anchor, _ = _repetitive_case()
other = "rain falls on the quiet harbour and the boats drift out past the lighthouse " * 6
kept, rejected = filter_irrelevant_sources({"lrclib": lyrics}, [anchor], transcribed_text=other)
assert not kept
assert rejected["lrclib"].ngram_relevance < 0.25
def test_filler_repetition_does_not_count():
"""Shared "na na na" / "oh oh oh" stretches must not make a wrong song look relevant."""
from karaoke_gen.lyrics_transcriber.correction.relevance import compute_ngram_relevance
filler = [["na"] * 8, ["oh", "oh", "oh", "yeah"]] * 10 + [["dance", "with", "me", "tonight"]]
lyrics = _make_lyrics_data(filler, "genius")
transcription = "na na na na na na oh oh oh yeah " * 12 + "different words entirely here"
score, matched, total, _ = compute_ngram_relevance(lyrics, transcription)
assert total < 20 # only distinct-word trigrams count, so filler can't reach the size guard
kept, _ = filter_irrelevant_sources({"genius": lyrics}, [], transcribed_text=transcription)
assert not kept
def test_tiny_reference_is_not_rescued():
lyrics = _make_lyrics_data([["one", "two", "three", "four"]], "spotify")
kept, rejected = filter_irrelevant_sources(
{"spotify": lyrics}, [], transcribed_text="one two three four"
)
assert not kept and rejected["spotify"].ngram_relevance == 1.0
def test_sources_passing_anchor_filter_are_never_rejected_by_rescue():
lyrics, anchor, _ = _repetitive_case()
all_ids = [w.id for seg in lyrics.segments for w in seg.words]
full_anchor = _make_anchor(["t1"], {"lrclib": all_ids})
kept, rejected = filter_irrelevant_sources(
{"lrclib": lyrics}, [full_anchor], transcribed_text="nothing in common at all here"
)
assert "lrclib" in kept and not rejected
def test_shared_refrain_alone_cannot_rescue():
"""A wrong song sharing only a long repeated three-word refrain stays rejected."""
refrain = [["na", "hey", "oh"]] * 30 + [["walking", "down", "the", "avenue", "tonight"]]
lyrics = _make_lyrics_data(refrain, "genius")
transcription = "na hey oh " * 40 + "something else about the sea and sky"
kept, rejected = filter_irrelevant_sources({"genius": lyrics}, [], transcribed_text=transcription)
assert not kept and rejected["genius"].ngram_relevance >= 0.25 # high score, but too few distinct trigrams
def test_combining_marks_are_kept_in_words():
from karaoke_gen.lyrics_transcriber.correction.relevance import _words
assert _words("कला बाला रामा") != _words("कली बाली रामी")
assert _words("Don’t stop") == ["don't", "stop"]