diff --git a/docs/LESSONS-LEARNED.md b/docs/LESSONS-LEARNED.md index f2af26a6..a61cf502 100644 --- a/docs/LESSONS-LEARNED.md +++ b/docs/LESSONS-LEARNED.md @@ -74,6 +74,7 @@ The "servers unavailable" banner (49 episodes in 2 weeks) was mostly the API's o - the tenant middleware (`get_tenant_config`, a GCS read on a cache miss). Job-log writes made on the API loop now go to one background thread (`backend/services/log_write_offload.py`). They stay inline in worker threads and Cloud Run Jobs, where nothing else waits on the loop. The auth dependencies and tenant middleware use `asyncio.to_thread`. The guard now holds every `async def` in `api/dependencies.py` and `middleware/` to `BLOCKING_CALLS`, not just decorated routes. Still on the loop: about 70 direct `get_job`/`update_job` calls in `async def` routes. Each is a single 50-200 ms Firestore round-trip and rarely shows up in the watchdog. Move them when one does. +- **The reference-lyrics relevance filter undercounted repetitive songs** (2026-10-19). Anchor coverage credits a repeated phrase once, so correct references for chorus-heavy songs scored 25-30% and were dropped. On 459 jobs that was 25 sources across about a dozen songs (22 rescued; one refrain-only D&B track left out), e.g. Wolfmother "Apple Tree" (job 5c1d24eb). `correction/relevance.py` now rescues a rejected source whose distinct-word trigram overlap with the transcription is at least 25% (with at least 20 reference trigrams, 10 matches and 15 distinct matched trigrams). The worst wrong-song pair across 491k cross-job pairs was 9.4%. It only adds sources and never rejects a kept one. A naive n-gram rule would have let "na na na"-style lyrics through at 41%, so trigrams must have three distinct words. To recalibrate, compare each job's `jobs/{id}/lyrics/*(Lyrics ).txt` with its `(Lyrics Uncorrected).txt`, and use cross-job pairs as the wrong-song baseline. - **Detection:** `backend/services/loop_watchdog.py` logs `EVENT_LOOP_STALL` with the loop thread's stack and a `culprit=file:line`. Grep for it before guessing. - **Alert on impact, not on gaps** (v0.268.5). After the CPU-throttle discount, the stall alert still flapped. Cloud Run autoscaling starts extra instances that serve one request, then sit idle and throttled while the startup warmup thread, which spends 10+ minutes loading SpaCy, competes for slices. Every 10-40s ERROR "stall" over 4 days (11 of them) was on an instance with **zero** requests, and all had an empty culprit. A freeze advances the 1/5/15s sample schedule without sampling, so whatever happens after the thaw has no stack. The watchdog now takes a `RequestTracker`, fed by the outermost pure-ASGI `RequestTrackingMiddleware`. A stall reaches ERROR only if a request was in flight at its start or arrived during it. A request that queued during the stall is counted if it starts within the 0.5s grace window after the loop resumes. Check whether anyone waited before chasing a culprit: join `labels.instanceId` between the stall lines and the request logs. - **Async services with sync bodies** (v0.268.7). `PushNotificationService.send_push` was an `async def` with no real awaits: a Firestore read, a `webpush` POST per device, and a Firestore update, all on the loop. The route guard can't see it because it's a service method. The watchdog caught it at 1.2s with a request waiting. The body is now `send_push_sync`, and `send_push` calls `await asyncio.to_thread(...)`. When `requests_affected>0` lines name a `backend/services/*` frame, that service method is the fix site. Also since v0.268.7: if a process freeze skips every sample point and the loop is still blocked once the process thaws, the watchdog takes one sample then, so the stall gets a culprit. diff --git a/karaoke_gen/lyrics_transcriber/core/controller.py b/karaoke_gen/lyrics_transcriber/core/controller.py index 3576a8bd..ffe7a282 100644 --- a/karaoke_gen/lyrics_transcriber/core/controller.py +++ b/karaoke_gen/lyrics_transcriber/core/controller.py @@ -563,6 +563,7 @@ class LyricsTranscriber: from karaoke_gen.lyrics_transcriber.correction.relevance import filter_irrelevant_sources filtered_lyrics, rejected_sources = filter_irrelevant_sources( self.results.lyrics_results, anchor_sequences, logger=self.logger, + transcribed_text=transcribed_text, ) if len(filtered_lyrics) < len(self.results.lyrics_results): self.logger.info( diff --git a/karaoke_gen/lyrics_transcriber/correction/corrector.py b/karaoke_gen/lyrics_transcriber/correction/corrector.py index 0b162448..61a36e01 100644 --- a/karaoke_gen/lyrics_transcriber/correction/corrector.py +++ b/karaoke_gen/lyrics_transcriber/correction/corrector.py @@ -156,7 +156,7 @@ class LyricsCorrector: # Filter irrelevant reference sources based on anchor match quality filtered_lyrics, rejected_sources = filter_irrelevant_sources( - lyrics_results, anchor_sequences, logger=self.logger, + lyrics_results, anchor_sequences, logger=self.logger, transcribed_text=transcribed_text, ) # If sources were filtered, re-run anchor finding with only valid sources diff --git a/karaoke_gen/lyrics_transcriber/correction/relevance.py b/karaoke_gen/lyrics_transcriber/correction/relevance.py index 6d868dda..e6fe2a6f 100644 --- a/karaoke_gen/lyrics_transcriber/correction/relevance.py +++ b/karaoke_gen/lyrics_transcriber/correction/relevance.py @@ -3,10 +3,20 @@ Computes how well each reference source matches the transcription based on anchor sequence coverage. Sources below the relevance threshold are filtered out to prevent wrong-song lyrics from polluting corrections. + +Anchor coverage undercounts songs that repeat the same lines: correct references for +Wolfmother "Apple Tree", Deacon Blue, UB40 "Wedding Day" etc. scored 25-30% and were +dropped. A source the anchor filter rejects is therefore rescued when its word-trigram +overlap with the transcription (repeats counted, filler like "na na na" excluded) is high. +Calibrated on 459 production jobs (2026-10-19): wrong-song pairs max 9.4% (p99.99 5.5%, +491k cross-job pairs), the 22 sources it rescues were all the correct song (26-88%). +The rescue can only add sources; it never rejects one the anchor filter keeps. """ import logging +import unicodedata +from collections import Counter from dataclasses import dataclass -from typing import Dict, List, Tuple +from typing import Dict, List, Optional, Tuple from karaoke_gen.lyrics_transcriber.types import AnchorSequence, LyricsData @@ -14,6 +24,13 @@ from karaoke_gen.lyrics_transcriber.types import AnchorSequence, LyricsData # See scripts/analyze_reference_relevance.py and the design spec for methodology. MIN_REFERENCE_RELEVANCE = 0.30 # Empirical: wrong-song max 3.5%, correct-song min 26.8% +# N-gram rescue (see module docstring): ~2.7x the worst wrong-song score seen. +MIN_NGRAM_RELEVANCE = 0.25 +MIN_NGRAM_REFERENCE = 20 # distinct-word trigrams in the reference; tiny references score by chance +MIN_NGRAM_MATCHES = 10 +MIN_NGRAM_DISTINCT_MATCHES = 15 # distinct matched trigrams: a shared refrain alone can't pass (rescues had 22+, bar one 8-phrase D&B track left rejected) +_NGRAM = 3 + @dataclass class SourceRelevanceResult: @@ -24,6 +41,7 @@ class SourceRelevanceResult: total_words: int track_name: str = "" artist_names: str = "" + ngram_relevance: Optional[float] = None # set when the n-gram rescue was evaluated def compute_source_relevance( @@ -68,11 +86,56 @@ def compute_source_relevance( ) +def _is_word_char(ch: str) -> bool: + # Letters/digits plus combining marks (Devanagari vowel signs etc.), which \w drops. + return ch.isalnum() or ch == "'" or unicodedata.category(ch).startswith("M") + + +def _words(text: str) -> list: + text = unicodedata.normalize("NFC", text.lower().replace("\u2019", "'")) + words, current = [], [] + for ch in text: + if _is_word_char(ch): + current.append(ch) + elif current: + words.append("".join(current).strip("'")) + current = [] + if current: + words.append("".join(current).strip("'")) + return [w for w in words if w] + + +def _trigrams(text: str) -> Counter: + """Word trigrams with three distinct words (so "na na na" / "oh oh oh" don't count).""" + words = _words(text) + grams = (tuple(words[i:i + _NGRAM]) for i in range(len(words) - _NGRAM + 1)) + return Counter(g for g in grams if len(set(g)) == _NGRAM) + + +def compute_ngram_relevance(lyrics_data: LyricsData, transcribed_text: str) -> Tuple[float, int, int, int]: + """(score, matched, total, distinct_matched): share of the reference's trigrams, repeats + included, found in the transcription, and how many different trigrams that match covers.""" + reference_text = " ".join(w.text for seg in lyrics_data.segments for w in seg.words) + ref = _trigrams(reference_text) + total = sum(ref.values()) + if not total: + return 0.0, 0, 0, 0 + common = ref & _trigrams(transcribed_text) + matched = sum(common.values()) + return matched / total, matched, total, len(common) + + +def _ngram_rescues(score: float, matched: int, total: int, distinct_matched: int) -> bool: + return (score >= MIN_NGRAM_RELEVANCE and total >= MIN_NGRAM_REFERENCE + and matched >= MIN_NGRAM_MATCHES and distinct_matched >= MIN_NGRAM_DISTINCT_MATCHES) + + def filter_irrelevant_sources( lyrics_results: Dict[str, LyricsData], anchor_sequences: List[AnchorSequence], min_relevance: float = MIN_REFERENCE_RELEVANCE, logger: logging.Logger = None, + transcribed_text: Optional[str] = None, ) -> Tuple[Dict[str, LyricsData], Dict[str, SourceRelevanceResult]]: """Filter out reference sources below the relevance threshold. @@ -81,6 +144,8 @@ def filter_irrelevant_sources( anchor_sequences: Anchor sequences from the anchor finder. min_relevance: Minimum relevance score to keep a source. logger: Optional logger. + transcribed_text: The transcription. When given, a source below ``min_relevance`` + is still kept if its trigram overlap passes the n-gram rescue. Returns: Tuple of (filtered_lyrics_results, rejected_sources). @@ -103,10 +168,22 @@ def filter_irrelevant_sources( f"{result.relevance:.1%} ({result.matched_words}/{result.total_words} words)" ) else: + if transcribed_text: + score, matched, total, distinct = compute_ngram_relevance(lyrics_data, transcribed_text) + result.ngram_relevance = score + if _ngram_rescues(score, matched, total, distinct): + filtered[source_name] = lyrics_data + logger.info( + f"Source '{source_name}' kept by n-gram rescue: anchor relevance " + f"{result.relevance:.1%} ({result.matched_words}/{result.total_words} words), " + f"trigram overlap {score:.1%} ({matched}/{total})" + ) + continue rejected[source_name] = result + ngram = f", trigram overlap {result.ngram_relevance:.1%}" if result.ngram_relevance is not None else "" logger.info( f"Source '{source_name}' rejected by relevance filter: " - f"{result.relevance:.1%} ({result.matched_words}/{result.total_words} words)" + f"{result.relevance:.1%} ({result.matched_words}/{result.total_words} words){ngram}" ) return filtered, rejected diff --git a/pyproject.toml b/pyproject.toml index d4c7f987..846c4411 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "karaoke-gen" -version = "0.279.3" +version = "0.279.4" description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens." authors = ["Andrew Beveridge "] license = "MIT" diff --git a/tests/correction/test_relevance.py b/tests/correction/test_relevance.py index ba09c666..ffd6d178 100644 --- a/tests/correction/test_relevance.py +++ b/tests/correction/test_relevance.py @@ -255,3 +255,110 @@ class TestCorrectorRelevanceIntegration: assert result.corrections_made == 0 # Original segments should be preserved assert len(result.original_segments) == 1 + + +# --- n-gram rescue (repetitive songs) ------------------------------------------------- +# Invented lyrics with the same shape as the production failure: two lines repeated many +# times, transcribed with a few consistent mishearings. + +_CHORUS = [ + ["hello", "neighbour", "will", "you", "wave", "at", "me"], + ["i", "am", "the", "one", "who", "painted", "the", "garden", "gate"], +] +_VERSE = [ + ["morning", "comes", "slowly", "over", "the", "hill"], + ["coffee", "goes", "cold", "while", "the", "radio", "plays"], +] + + +def _song_lines() -> list[list[str]]: + return _CHORUS * 4 + _VERSE + _CHORUS * 4 + _VERSE + _CHORUS * 2 + + +def _misheard(lines: list[list[str]]) -> str: + swaps = {"neighbour": "nathan", "who": "that", "garden": "guarded", "gate": "grape"} + return " ".join(swaps.get(w, w) for line in lines for w in line) + + +def _repetitive_case(): + lines = _song_lines() + lyrics = _make_lyrics_data(lines, "lrclib") + # Anchors that only credit the first occurrence of each phrase in the reference, + # the undercount that rejected correct lyrics for repetitive songs in production. + first_ids = [w.id for seg in lyrics.segments[:2] for w in seg.words] + anchor = _make_anchor(["t1"], {"lrclib": first_ids}) + return lyrics, anchor, _misheard(lines) + + +def test_repetitive_correct_song_is_rescued_by_ngrams(): + from karaoke_gen.lyrics_transcriber.correction.relevance import compute_ngram_relevance + + lyrics, anchor, transcription = _repetitive_case() + assert compute_source_relevance("lrclib", lyrics, [anchor]).relevance < 0.30 + score, matched, total, distinct = compute_ngram_relevance(lyrics, transcription) + assert score >= 0.25 and matched >= 10 and total >= 20 and distinct >= 15 + + kept, rejected = filter_irrelevant_sources({"lrclib": lyrics}, [anchor], transcribed_text=transcription) + assert "lrclib" in kept and not rejected + + +def test_without_transcribed_text_behaviour_is_unchanged(): + lyrics, anchor, _ = _repetitive_case() + kept, rejected = filter_irrelevant_sources({"lrclib": lyrics}, [anchor]) + assert not kept and "lrclib" in rejected + assert rejected["lrclib"].ngram_relevance is None + + +def test_wrong_song_is_still_rejected(): + lyrics, anchor, _ = _repetitive_case() + other = "rain falls on the quiet harbour and the boats drift out past the lighthouse " * 6 + kept, rejected = filter_irrelevant_sources({"lrclib": lyrics}, [anchor], transcribed_text=other) + assert not kept + assert rejected["lrclib"].ngram_relevance < 0.25 + + +def test_filler_repetition_does_not_count(): + """Shared "na na na" / "oh oh oh" stretches must not make a wrong song look relevant.""" + from karaoke_gen.lyrics_transcriber.correction.relevance import compute_ngram_relevance + + filler = [["na"] * 8, ["oh", "oh", "oh", "yeah"]] * 10 + [["dance", "with", "me", "tonight"]] + lyrics = _make_lyrics_data(filler, "genius") + transcription = "na na na na na na oh oh oh yeah " * 12 + "different words entirely here" + score, matched, total, _ = compute_ngram_relevance(lyrics, transcription) + assert total < 20 # only distinct-word trigrams count, so filler can't reach the size guard + kept, _ = filter_irrelevant_sources({"genius": lyrics}, [], transcribed_text=transcription) + assert not kept + + +def test_tiny_reference_is_not_rescued(): + lyrics = _make_lyrics_data([["one", "two", "three", "four"]], "spotify") + kept, rejected = filter_irrelevant_sources( + {"spotify": lyrics}, [], transcribed_text="one two three four" + ) + assert not kept and rejected["spotify"].ngram_relevance == 1.0 + + +def test_sources_passing_anchor_filter_are_never_rejected_by_rescue(): + lyrics, anchor, _ = _repetitive_case() + all_ids = [w.id for seg in lyrics.segments for w in seg.words] + full_anchor = _make_anchor(["t1"], {"lrclib": all_ids}) + kept, rejected = filter_irrelevant_sources( + {"lrclib": lyrics}, [full_anchor], transcribed_text="nothing in common at all here" + ) + assert "lrclib" in kept and not rejected + + +def test_shared_refrain_alone_cannot_rescue(): + """A wrong song sharing only a long repeated three-word refrain stays rejected.""" + refrain = [["na", "hey", "oh"]] * 30 + [["walking", "down", "the", "avenue", "tonight"]] + lyrics = _make_lyrics_data(refrain, "genius") + transcription = "na hey oh " * 40 + "something else about the sea and sky" + kept, rejected = filter_irrelevant_sources({"genius": lyrics}, [], transcribed_text=transcription) + assert not kept and rejected["genius"].ngram_relevance >= 0.25 # high score, but too few distinct trigrams + + +def test_combining_marks_are_kept_in_words(): + from karaoke_gen.lyrics_transcriber.correction.relevance import _words + + assert _words("कला बाला रामा") != _words("कली बाली रामी") + assert _words("Don’t stop") == ["don't", "stop"]