Files
karaoke-gen/tests/unit/test_font_fallback_render.py
Andrew Beveridge ☄️andClaude Opus 5.5 354e43bab2 Render Hebrew/Arabic karaoke right-to-left; fix title-card tofu (v0.255.0) (#1094)
## Summary

A Hebrew KaraokeHunt job (5710831e, Omer Adam – "שני משוגעים") shipped
with its lyrics **laid out and highlighted left-to-right** and a **title
card of "?" boxes**. Diagnosed by measuring rendered pixels (not by
eye), reproduced locally and with prod's static ffmpeg 7.0.2 in Docker.

**Root causes**
- ASS style `Encoding=0` makes libass force LTR paragraphs; the `{\kf}`
before each word splits the line into runs, so each word is shaped
correctly but the words go left→right.
- `\kf` always wipes left→right within a word
([libass#406](https://github.com/libass/libass/issues/406)).
- PIL (title/end cards) has no per-glyph font fallback; the theme font
(Avenir Next) has no Hebrew glyphs.

**Fixes**
- `build_karaoke_styles` forces `Encoding=-1` (libass auto paragraph
direction; LTR unchanged).
- RTL lines get `{\frz(180+angle)\frx180\fry180}` — a visual no-op that
makes libass ≥0.15 sweep fills right→left without splitting words
(Arabic still joins). Theme angle preserved; re-emitted after duet
`{\r}` resets.
- Lead-in indicator mirrored for RTL lines.
- Text measurement mirrors libass: per-glyph fontconfig fallback, size
scaled from each font's metrics (old fixed 0.70 was 20%+ off for
fallback fonts).
- New `karaoke_gen/utils/font_fallback.py` (fontTools cmap +
fc-match/fc-list) used by title/end cards, portrait headers and ASS
measurement.
- Found by the new tests: inter-word gaps ≤0.1s were dropped, so the
highlight ran up to ~0.2s early by line end (all languages). Karaoke
tags now derive from one absolute centisecond timeline.
- Fixed pre-existing `test_render_video_worker_gce` failure on main
(MagicMock generation → `_g<mock>` id).

## Testing
- **Pixel-level render tests** (`test_ass_render_highlight.py` +
`ass_render_harness.py`): real `SubtitlesGenerator` output rendered with
ffmpeg/libass in a pure-colour palette, asserting on frames for English,
Hebrew, Arabic, Chinese, Japanese, Korean + an RTL duet line: highlight
starts at the reading-start edge, sung pixels always precede unsung in
reading order (word order + in-word fill direction), monotonic and in
time with the words, lead-in on the correct side, no missing glyphs, RTL
tags don't move the line (incl. rotated themes).
- 32 fail on pre-fix main; mutation checks ("Encoding only", "no re-emit
after `\r`", "ignore theme angle") each fail exactly the expected cases.
- 123/123 pass on Linux against **both** Debian ffmpeg (CI-equivalent)
and **prod's johnvansickle static ffmpeg** via
`scripts/run-render-tests-linux.sh`.
- CI now installs `fonts-noto-core fonts-noto-cjk`; under `CI=true`
these tests fail rather than skip on missing preconditions. (macOS
libass uses CoreText → font-sensitive checks skip locally.)
- Title-card tests (`test_font_fallback_render.py`): no `.notdef` tofu
for Hebrew/Arabic/CJK titles; Hebrew title reads right-to-left.
- Unit tests: `test_rtl_ass_output.py`, `test_font_fallback.py`.
- `make test`: unit 1972 passed; emulator 150 passed; frontend 1491
passed.
- Full local re-render of job 5710831e reviewed by Andrew: RTL highlight
smooth and correct, title card correct.

## Follow-ups (not in this PR)
- CDG output still renders RTL left-to-right / tofu (separate renderer).
- Separate investigation: transcription dropped 3 sung lines in this
job, which the section detector then labelled INSTRUMENTAL.

No local CodeRabbit review (CLI has no seat for this org), so leaving
this open for the CodeRabbit bot. A local code review ran; its findings
are addressed in the second commit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Karaoke lyrics now render more reliably for Hebrew, Arabic, Chinese,
Japanese, and Korean, with improved reading-direction highlighting,
timing, and lead-in placement.
* Titles and portrait text can use fallback fonts when the selected font
lacks the necessary characters.

* **Bug Fixes**
* Improved lyric centering, word highlighting, and glyph rendering
across supported scripts.

* **Documentation**
  * Added guidance on lyric rendering and multilingual test coverage.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 21:02:53 -04:00

149 lines
5.9 KiB
Python
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""Title/end cards and portrait headers must render non-Latin titles with real glyphs.
Job 5710831e (Omer Adam – "שני משוגעים") shipped a title card of "?" boxes: the theme
font (Avenir Next) has no Hebrew glyphs and PIL, unlike libass, doesn't fall back.
These tests use real fonts (fonts-noto-core / fonts-noto-cjk in CI and prod) and check
rendered pixels, not just which path was chosen.
"""
import os
import shutil
import numpy as np
import pytest
from PIL import Image, ImageDraw, ImageFont, features
from unittest.mock import MagicMock
from karaoke_gen.portrait.background import PortraitBrandConfig, _load_font
from karaoke_gen.utils.font_fallback import missing_codepoints, resolve_font_for_text
from karaoke_gen.video_generator import VideoGenerator
IN_CI = os.environ.get("CI", "").lower() == "true"
THEME_FONT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "karaoke_gen", "resources", "AvenirNext-Bold.ttf"))
TITLES = {
"hebrew": "שני משוגעים",
"arabic": "كيف حالك",
"chinese": "青花瓷",
"japanese": "宇多田ヒカル",
"korean": "방탄소년단",
"mixed": "Omer Adam - שני משוגעים",
}
def skip_unless_ci(reason):
if IN_CI:
pytest.fail(f"precondition failed in CI: {reason}")
pytest.skip(reason)
@pytest.fixture(autouse=True)
def _need_fontconfig():
if not shutil.which("fc-match"):
skip_unless_ci("fontconfig (fc-match) not installed")
def _notdef_mask(font: ImageFont.FreeTypeFont) -> bytes:
# U+10FFFD is a private-use noncharacter no font maps -> renders .notdef
return bytes(font.getmask("\U0010FFFD"))
def _assert_no_tofu(font_path: str, text: str):
font = ImageFont.truetype(font_path, 80)
notdef = _notdef_mask(font)
tofu = [ch for ch in text if not ch.isspace() and bytes(font.getmask(ch)) == notdef]
assert not tofu, f"{os.path.basename(font_path)} draws .notdef boxes for {tofu!r}"
@pytest.mark.parametrize("script", list(TITLES))
def test_fallback_font_covers_title(script):
text = TITLES[script]
assert missing_codepoints(THEME_FONT, text), "theme font should lack these glyphs (test premise)"
chosen = resolve_font_for_text(THEME_FONT, text)
if chosen == THEME_FONT:
skip_unless_ci(f"no installed font covers {script}")
assert not missing_codepoints(chosen, text)
_assert_no_tofu(chosen, text)
def test_latin_title_keeps_theme_font():
assert resolve_font_for_text(THEME_FONT, "Omer Adam") == THEME_FONT
assert resolve_font_for_text(THEME_FONT, "Café del Mar") == THEME_FONT
def _generator():
return VideoGenerator(logger=MagicMock(), ffmpeg_base_command="ffmpeg",
render_bounding_boxes=False, output_png=True, output_jpg=True)
def _render_title(title: str) -> np.ndarray:
"""Render ``title`` through the real title-card path; return the ink mask."""
img = Image.new("RGB", (3840, 2160), "black")
fmt = {
"title_region": "370, 200, 3100, 480", "title_color": "#ffffff", "title_gradient": None,
"artist_region": None, "extra_text": None,
}
gen = _generator()
gen._render_all_text(ImageDraw.Draw(img), THEME_FONT, title, None, fmt, False)
return np.asarray(img.convert("L")) > 128
@pytest.mark.parametrize("script", ["hebrew", "arabic", "korean"])
def test_title_card_uses_a_covering_font(script):
gen = _generator()
used_fonts = []
real_render = gen._render_text_in_region
def spy(draw, text, font_path, *args, **kwargs):
used_fonts.append(font_path)
return real_render(draw, text, font_path, *args, **kwargs)
gen._render_text_in_region = spy
img = Image.new("RGB", (3840, 2160), "black")
fmt = {"title_region": "370, 200, 3100, 480", "title_color": "#ffffff", "title_gradient": None,
"artist_region": None, "extra_text": None}
gen._render_all_text(ImageDraw.Draw(img), THEME_FONT, TITLES[script], None, fmt, False)
assert len(used_fonts) == 1
if used_fonts[0] == THEME_FONT:
skip_unless_ci(f"no installed font covers {script}")
_assert_no_tofu(used_fonts[0], TITLES[script])
def _split_at_widest_gap(mask: np.ndarray):
"""Column extents of the two words either side of the widest ink gap (the space)."""
cols = np.where(mask.any(axis=0))[0]
gaps = np.diff(cols)
i = int(np.argmax(gaps))
return (int(cols[0]), int(cols[i])), (int(cols[i + 1]), int(cols[-1]))
def test_hebrew_title_card_reads_right_to_left():
""""שני משוגעים": the first word (3 letters) must be drawn on the RIGHT of the second
(7 letters). Needs Pillow's raqm layout (libraqm + fribidi), present in CI and prod."""
if not features.check("raqm"):
skip_unless_ci("Pillow without raqm cannot lay out RTL text")
if resolve_font_for_text(THEME_FONT, TITLES["hebrew"]) == THEME_FONT:
skip_unless_ci("no installed Hebrew font")
(l0, l1), (r0, r1) = _split_at_widest_gap(_render_title(TITLES["hebrew"]))
left_width, right_width = l1 - l0, r1 - r0
assert right_width < left_width * 0.6, (
f"first word שני should be the narrow one on the right; widths left={left_width} right={right_width}"
)
@pytest.mark.parametrize("script", ["hebrew", "chinese"])
def test_portrait_header_font_covers_title(script):
cfg = PortraitBrandConfig(font_path=THEME_FONT)
font = _load_font(cfg, 60, TITLES[script])
if font.path == THEME_FONT:
skip_unless_ci(f"no installed font covers {script}")
_assert_no_tofu(font.path, TITLES[script])
def test_latin_only_noto_font_is_not_kept_for_cjk_title(tmp_path):
"""A theme font named Noto*-Bold that is Latin-only must not pass for a CJK font."""
latin_noto = tmp_path / "NotoSans-Bold.ttf"
shutil.copy(THEME_FONT, latin_noto) # Latin-only font under a "noto" name
gen = _generator()
gen._cjk_font_path = "/fonts/NotoSansCJK-Bold.ttc"
assert gen._get_font_path_for_text(str(latin_noto), TITLES["chinese"]) == "/fonts/NotoSansCJK-Bold.ttc"