Files
karaoke-gen/tests/unit/test_remote_cli_unicode_folders.py
T
Andrew Beveridge ☄️andClaude Sonnet 4.5 d9bb8b3181 fix: sanitize Unicode in temp file paths (FFmpeg exit 183) (#375)
## Summary
Fixes job 1eab1172 which failed with FFmpeg exit code 183 due to
Cyrillic characters in temp subtitle file paths. Python's
`str.isalnum()` returns True for ALL Unicode alphanumeric characters
(Cyrillic, Arabic, Chinese, emoji, etc.), not just ASCII.

## Root Cause
- Job `1eab1172` with Russian title "я куплю тебе дом" failed
- Temp subtitle path contained Cyrillic:
`temp_subtitles_zivert___я_куплю_тебе_дом_1770150285852.ass`
- FFmpeg cannot handle Unicode in filter argument paths → exit code 183

## Changes
- **karaoke_gen/lyrics_transcriber/output/video.py** (lines 222, 282):
- Changed `c.isalnum()` to `c.isascii() and c.isalnum()` for temp
subtitle paths
  - Affects both `generate_video()` and `generate_preview_video()`

- **karaoke_gen/utils/remote_cli.py** (line 1970):
- Changed `c.isalnum()` to `c.isascii() and c.isalnum()` for output
folder names
- Added fallback to `job_{job_id}` if folder name is empty or contains
no alphanumeric chars

- **Tests added** (21 comprehensive tests):
- `test_video_generator_unicode.py` (10 tests): Cyrillic, Chinese,
Arabic, emoji, accented chars
- `test_remote_cli_unicode_folders.py` (11 tests): Folder sanitization,
edge cases, real failure case

- **Documentation**:
- Updated `docs/LESSONS-LEARNED.md` with "Unicode in File Paths (Feb
2026)" section
  - Updated `docs/README.md` with recent changes
  - Bumped version 0.115.1 → 0.115.2

## Testing
- [x] All 1088 unit tests pass (68.91% coverage, target 69%)
- [x] 21 new comprehensive Unicode tests covering:
  - Cyrillic, Chinese, Arabic, emoji, accented characters
  - Empty folder name edge case
  - Separator-only folder name edge case
  - Exact real-world failure case from job 1eab1172
- [x] Verified temp file paths are ASCII-only
- [x] Verified folder names safe for all filesystems

## Review
- [x] CodeRabbit CLI review completed locally (2 cycles)
- [x] All identified issues addressed:
  - Empty folder name fallback added
  - Separator-only folder name guard enhanced
- [x] Test coverage maintained at target

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-05 09:13:33 -08:00

134 lines
5.7 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""Unit tests for remote_cli Unicode folder name sanitization.
Tests the fix for job 1eab1172 where Cyrillic characters in folder names
could cause filesystem issues.
"""
import pytest
class TestRemoteCliFolderNameSanitization:
"""Test folder name sanitization logic for Unicode characters."""
@pytest.mark.parametrize("artist,title,expected_has,expected_not_has", [
("zivert", "я куплю тебе дом", ["zivert"], ["я", "куплю", "тебе", "дом", " - ", "_"]), # Cyrillic removed, only artist remains
("artist", "中文标题", ["artist"], ["中", "文", "标", "题", " - ", "_"]), # Chinese removed
("artist", "العربية", ["artist"], ["ا", "ل", "ع", "ر", "ب", " - ", "_"]), # Arabic removed
("artist", "título", ["artist", " - ", "tulo"], ["í"]), # Accented (í removed, t+ulo remain)
("Artist", "Normal Title", ["Artist", " - ", "Normal", "Title"], []), # ASCII preserved
("Test", "Song (2024)", ["Test", " - ", "Song", "2024"], ["(", ")"]), # Parens removed
])
def test_folder_name_sanitization_logic(self, artist, title, expected_has, expected_not_has):
"""Test the folder name sanitization logic directly."""
# This is the exact logic from remote_cli.py:1970
folder_name = f"{artist} - {title}"
folder_name = "".join(c for c in folder_name if (c.isascii() and c.isalnum()) or c in " -_").strip()
# Verify folder name is ASCII only
assert folder_name.isascii(), f"Folder name should be ASCII-only: {folder_name}"
# Verify expected parts are present
for expected in expected_has:
assert expected in folder_name, \
f"Expected '{expected}' to be in folder name: {folder_name}"
# Verify unwanted parts are removed
for not_expected in expected_not_has:
assert not_expected not in folder_name, \
f"Did not expect '{not_expected}' in folder name: {folder_name}"
def test_folder_name_brand_code_logic(self):
"""Test folder name with brand code prefix."""
brand_code = "BRAND-001"
artist = "Artist"
title = "Title"
folder_name = f"{brand_code} - {artist} - {title}"
folder_name = "".join(c for c in folder_name if (c.isascii() and c.isalnum()) or c in " -_").strip()
assert folder_name.startswith("BRAND-001 - ")
assert "Artist" in folder_name
assert "Title" in folder_name
def test_folder_name_strips_whitespace(self):
"""Test that leading/trailing whitespace is stripped."""
artist = " Artist "
title = " Title "
folder_name = f"{artist} - {title}"
folder_name = "".join(c for c in folder_name if (c.isascii() and c.isalnum()) or c in " -_").strip()
assert not folder_name.startswith(" ")
assert not folder_name.endswith(" ")
assert "Artist" in folder_name
assert "Title" in folder_name
def test_cyrillic_real_world_case_job_1eab1172(self):
"""Test the exact case from failed job 1eab1172 - Russian title."""
artist = "zivert"
title = "я куплю тебе дом" # Russian: "I will buy you a house"
folder_name = f"{artist} - {title}"
folder_name = "".join(c for c in folder_name if (c.isascii() and c.isalnum()) or c in " -_").strip()
# Should be ASCII-only
assert folder_name.isascii()
# Should contain the artist name
assert "zivert" in folder_name
# Cyrillic characters should be removed
assert "я" not in folder_name
assert "куплю" not in folder_name
assert "тебе" not in folder_name
assert "дом" not in folder_name
def test_emoji_and_special_unicode(self):
"""Test that emojis and special Unicode characters are removed."""
artist = "Artist 😀"
title = "Song 🎵"
folder_name = f"{artist} - {title}"
folder_name = "".join(c for c in folder_name if (c.isascii() and c.isalnum()) or c in " -_").strip()
assert folder_name.isascii()
assert "😀" not in folder_name
assert "🎵" not in folder_name
assert "Artist" in folder_name
assert "Song" in folder_name
def test_empty_folder_name_fallback(self):
"""Test that if sanitization results in empty or separator-only string, fallback to job_id."""
job_id = "abc123xyz"
# Test 1: Emoji with separator results in "-" after sanitization
artist = "😀😀😀"
title = "🎵🎵🎵"
folder_name = f"{artist} - {title}"
folder_name = "".join(c for c in folder_name if (c.isascii() and c.isalnum()) or c in " -_").strip()
# After sanitization, only "-" remains (spaces/hyphens allowed but emoji removed)
assert folder_name == "-"
# Apply the fallback logic (updated to check for alphanumeric content)
if not folder_name or not any(c.isascii() and c.isalnum() for c in folder_name):
folder_name = f"job_{job_id}"
# Verify fallback is used for separator-only result
assert folder_name == "job_abc123xyz"
# Test 2: Truly empty result (all characters removed, no separator)
artist2 = "😀"
title2 = "🎵"
folder_name2 = f"{artist2}{title2}" # No separator, just emoji
folder_name2 = "".join(c for c in folder_name2 if (c.isascii() and c.isalnum()) or c in " -_").strip()
# Should be empty
assert folder_name2 == ""
# Apply fallback
if not folder_name2 or not any(c.isascii() and c.isalnum() for c in folder_name2):
folder_name2 = f"job_{job_id}"
assert folder_name2 == "job_abc123xyz"