fix: bounds check ref_words in anchor gap extension (IndexError b7e8c96d) (#623)

## Summary

- Fix IndexError in `_extend_anchors_into_single_word_gaps` when
`ref_texts_clean` and `ref_words` have different word counts due to
`clean_text()` normalization
- Adds production fixture regression test from crashing job `b7e8c96d`
(Bằng Kiều - Dẫu có lỗi lầm)

## Root Cause

`ref_texts_clean` is built by joining words → `clean_text()` →
`split()`, which can produce a **different word count** than the raw
`ref_words` Word objects list. For Spotify in the crashing job:
`ref_texts_clean` had 599 words, `ref_words` had 528. Anchor
`reference_positions` computed against the 599-word list were used to
index the 528-word list.

## Changes

- `anchor_sequence.py`: Add `< len(ref_words[source])` bounds check in
both `_apply_forward_extension` and `_apply_backward_extension`
- `test_anchor_gap_extension_bounds.py`: 4 regression tests (production
fixture + synthetic bounds tests)
- `anchor_crash_b7e8c96d.json`: Sanitized production fixture (49
anchors, 526 words)
- `LESSONS-LEARNED.md`: Document text normalization word count mismatch
gotcha
- Version bump: 0.153.4 → 0.153.5

## Testing

- [x] 33 anchor gap extension tests pass (29 existing + 4 new)
- [x] Full unit suite: 1146 tests pass
- [ ] After deploy: retry job `b7e8c96d`

## Review

- [x] CodeRabbit CLI review completed locally (1 finding fixed:
sanitized copyrighted lyrics)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 4.6 authored and GitHub committed 2026-03-28 14:23:13 -04:00
1 parent dd25a6110f
commit 78e4cc7010
5 files changed
+16979 -3

No files matched your search

+1
View File
@@ -28,6 +28,7 @@ Three patterns from investigating production job failures where errors cascaded
1. **Validate every stage's outputs, not just some** — stage 1 of audio separation had validation for expected stems, stage 2 didn't. The missing validation let a transient Modal API failure propagate through 3 more functions before crashing with a confusing NoneType error.
2. **Block impossible downstream work early** — AudioShake returning 0 lyrics segments was treated as "success", letting the user submit an empty review that only failed at CDG generation. Validate at the boundary: if transcription returns 0 segments, fail with a user-friendly error, not 4 steps later.
3. **File handles in retry loops** — when wrapping HTTP calls with retry (e.g. tenacity), ensure file handles are re-opened on each attempt. A `with open()` wrapping a retried `requests.post(files=...)` exhausts the file handle on the first attempt; subsequent retries send empty data.
4. **Text normalization changes word counts** — `clean_text()` (strip punctuation, normalize whitespace) then `split()` can produce a different word count than the original Word objects list. In anchor sequence correction, `ref_texts_clean` had 599 words for Spotify but `ref_words` only had 528 Word objects. Anchor positions computed against the cleaned list were used to index the Word list, causing IndexError. Always bounds-check when indexing a different data structure than the one used to compute positions.
---
@@ -1468,7 +1468,9 @@ class AnchorSequenceFinder:
ref_start = anchor.reference_positions[source]
ref_end = ref_start + len(word_ids)
if source in ref_words and ref_end < len(ref_words[source]):
# Bounds check: ref_end must be within ref_words (which may be shorter
# than ref_texts_clean due to clean_text() normalizing word boundaries)
if source in ref_words and 0 <= ref_end < len(ref_words[source]):
new_ref_word = ref_words[source][ref_end]
new_reference_word_ids[source] = list(word_ids) + [new_ref_word.id]
else:
@@ -1523,7 +1525,9 @@ class AnchorSequenceFinder:
ref_start = anchor.reference_positions[source]
new_ref_start = ref_start - 1
if source in ref_words and new_ref_start >= 0:
# Bounds check: new_ref_start must be within ref_words (which may be shorter
# than ref_texts_clean due to clean_text() normalizing word boundaries)
if source in ref_words and 0 <= new_ref_start < len(ref_words[source]):
new_ref_word = ref_words[source][new_ref_start]
new_reference_word_ids[source] = [new_ref_word.id] + list(word_ids)
new_reference_positions[source] = new_ref_start
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.153.4"
version = "0.153.5"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,179 @@
"""
Regression test for IndexError in anchor gap extension (job b7e8c96d).
Root cause: ref_texts_clean and ref_words have different lengths because
clean_text() normalizes text (merging/splitting words), but anchor positions
are computed against ref_texts_clean and then used to index ref_words.
When ref_texts_clean[source] has MORE words than ref_words[source],
anchor reference_positions can exceed ref_words bounds.
"""
import json
import logging
import tempfile
from pathlib import Path
import pytest
from karaoke_gen.lyrics_transcriber.correction.anchor_sequence import AnchorSequenceFinder
from karaoke_gen.lyrics_transcriber.types import (
AnchorSequence,
PhraseScore,
PhraseType,
ScoredAnchor,
Word,
)
FIXTURE_PATH = Path(__file__).parent.parent / "fixtures" / "anchor_crash_b7e8c96d.json"
@pytest.fixture
def finder():
with tempfile.TemporaryDirectory() as temp_dir:
yield AnchorSequenceFinder(
cache_dir=temp_dir,
min_sequence_length=3,
min_sources=1,
timeout_seconds=60,
logger=logging.getLogger(__name__),
)
def _load_fixture():
"""Load the production crash fixture data."""
with open(FIXTURE_PATH) as f:
return json.load(f)
def _rebuild_scored_anchors(fixture_data):
"""Rebuild ScoredAnchor objects from fixture JSON."""
scored_anchors = []
for sa_data in fixture_data["filtered_scored"]:
a = sa_data["anchor"]
anchor = AnchorSequence(
id=a["id"],
transcribed_word_ids=a["transcribed_word_ids"],
transcription_position=a["transcription_position"],
reference_positions=a["reference_positions"],
reference_word_ids=a["reference_word_ids"],
confidence=a["confidence"],
_words=a["words"],
)
phrase_score = PhraseScore(
phrase_type=PhraseType(sa_data["phrase_score"]["phrase_type"]),
natural_break_score=sa_data["phrase_score"]["natural_break_score"],
length_score=sa_data["phrase_score"]["length_score"],
)
scored_anchors.append(ScoredAnchor(anchor=anchor, phrase_score=phrase_score))
return scored_anchors
def _rebuild_words(word_dicts):
"""Rebuild Word objects from fixture JSON."""
return [
Word(id=w["id"], text=w["text"], start_time=w["start_time"], end_time=w["end_time"])
for w in word_dicts
]
@pytest.mark.skipif(not FIXTURE_PATH.exists(), reason="Fixture not available")
class TestProductionCrashB7e8c96d:
"""Regression tests using production data from job b7e8c96d."""
def test_gap_extension_does_not_crash(self, finder):
"""_extend_anchors_into_single_word_gaps must not IndexError on mismatched ref lengths."""
data = _load_fixture()
scored_anchors = _rebuild_scored_anchors(data)
all_words = _rebuild_words(data["all_words"])
ref_texts_clean = data["ref_texts_clean"]
ref_words = {src: _rebuild_words(words) for src, words in data["ref_words"].items()}
# This crashed with IndexError before the fix
result = finder._extend_anchors_into_single_word_gaps(
scored_anchors, all_words, ref_texts_clean, ref_words
)
assert len(result) == len(scored_anchors)
def test_ref_texts_and_ref_words_length_mismatch(self, finder):
"""Verify the fixture actually has the length mismatch that causes the bug."""
data = _load_fixture()
mismatched = False
for source in data["ref_texts_clean"]:
clean_len = len(data["ref_texts_clean"][source])
words_len = len(data["ref_words"].get(source, []))
if clean_len != words_len:
mismatched = True
assert mismatched, "Fixture should have ref_texts_clean/ref_words length mismatch"
class TestBoundsCheckInExtension:
"""Unit tests for bounds checking in forward/backward extension."""
def _make_anchor(self, words, trans_pos, ref_positions, ref_word_ids=None):
anchor = AnchorSequence(
words=words,
transcription_position=trans_pos,
reference_positions=ref_positions,
confidence=1.0,
)
if ref_word_ids:
anchor.reference_word_ids = ref_word_ids
return ScoredAnchor(
anchor=anchor,
phrase_score=PhraseScore(
phrase_type=PhraseType.COMPLETE,
natural_break_score=1.0,
length_score=1.0,
),
)
def _make_word(self, idx, text="word"):
return Word(id=f"w{idx}", text=text, start_time=float(idx), end_time=float(idx + 1))
def test_backward_extension_ref_position_exceeds_ref_words(self, finder):
"""Backward extension must not crash when ref position > len(ref_words)."""
# ref_texts_clean has 10 words, but ref_words only has 5
# Anchor at ref_position=8 would crash when doing ref_words[source][8-1]
all_words = [self._make_word(i, f"w{i}") for i in range(10)]
ref_words = {"src": [self._make_word(i, f"r{i}") for i in range(5)]}
ref_texts_clean = {"src": [f"r{i}" for i in range(10)]}
anchor1 = self._make_anchor(
["w5", "w6"], trans_pos=5, ref_positions={"src": 5},
ref_word_ids={"src": ["r5_id", "r6_id"]},
)
anchor2 = self._make_anchor(
["w8", "w9"], trans_pos=8, ref_positions={"src": 8},
ref_word_ids={"src": ["r8_id", "r9_id"]},
)
# gap at position 7, backward extension would try ref_words["src"][7]
# which exceeds len(ref_words["src"]) = 5
result = finder._extend_anchors_into_single_word_gaps(
[anchor1, anchor2], all_words, ref_texts_clean, ref_words
)
assert len(result) == 2
def test_forward_extension_ref_end_exceeds_ref_words(self, finder):
"""Forward extension must not crash when ref_end > len(ref_words)."""
all_words = [self._make_word(i, f"w{i}") for i in range(10)]
ref_words = {"src": [self._make_word(i, f"r{i}") for i in range(5)]}
ref_texts_clean = {"src": [f"r{i}" for i in range(10)]}
anchor1 = self._make_anchor(
["w3", "w4"], trans_pos=3, ref_positions={"src": 3},
ref_word_ids={"src": ["r3_id", "r4_id"]},
)
anchor2 = self._make_anchor(
["w6", "w7"], trans_pos=6, ref_positions={"src": 6},
ref_word_ids={"src": ["r6_id", "r7_id"]},
)
# gap at position 5, forward extension would try ref_words["src"][5]
# which is exactly at the boundary (len=5, index 5 is out of bounds)
result = finder._extend_anchors_into_single_word_gaps(
[anchor1, anchor2], all_words, ref_texts_clean, ref_words
)
assert len(result) == 2