Fix custom instrumental ignored during encoding (#410)

## Summary
- Fix critical bug: uploaded custom instrumentals were silently replaced
with clean instrumental during encoding
- GCE encoding worker had no `"custom"` case — fell through to clean
instrumental patterns
- Orchestrator config constructed wrong path (`Backing` instead of
`Custom`) for custom selection
- Upload endpoint now converts all formats to FLAC for pipeline
consistency

## Root Cause
When `instrumental_selection == "custom"`, two downstream components
didn't handle it:
1. **GCE encoding worker** (`gce_encoding/main.py`): only had
`with_backing` and default (clean) cases
2. **Orchestrator** (`video_worker_orchestrator.py`): `"custom" !=
"clean"` evaluated to `"Backing"` suffix

The uploaded `.mp3` file also wouldn't match `*.flac` glob patterns even
with the wildcard fallback.

## Changes
- **backend/services/gce_encoding/main.py**: Add `"custom"` case to
instrumental file search
- **backend/workers/video_worker_orchestrator.py**: Add `"custom"` case
for path construction
- **backend/api/routes/jobs.py**: Convert uploaded audio to FLAC before
storing
- **backend/tests/test_custom_instrumental_route.py**: 10 new regression
tests (43 total)
- **pyproject.toml**: Bump 0.119.3 → 0.119.4

## Test plan
- [x] 43 tests passing (10 new: 5 GCE resolution + 5 orchestrator path)
- [x] Regression test: custom selection does NOT fall back to clean
- [x] Regression test: custom selection does NOT produce Backing path
- [x] Non-FLAC upload triggers FLAC conversion
- [ ] Manual: re-test upload + render for affected jobs

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 4.6 authored and GitHub committed 2026-02-26 01:02:05 -05:00
1 parent d7e12e3443
commit b671b83f19
6 files changed
+237 -10

No files matched your search

+14 -3
View File
@@ -774,6 +774,7 @@ async def upload_custom_instrumental(
storage = StorageService()
tmp_path = None
flac_path = None
try:
with tempfile.NamedTemporaryFile(delete=False, suffix=suffix) as tmp:
@@ -800,9 +801,17 @@ async def upload_custom_instrumental(
),
)
# Upload to GCS
output_path = f"jobs/{job_id}/stems/custom_instrumental{suffix}"
storage.upload_file(tmp_path, output_path)
# Convert to FLAC for consistency with the rest of the pipeline
# (GCE encoding worker searches for *.flac patterns)
flac_path = tmp_path.rsplit('.', 1)[0] + '.flac' if '.' in tmp_path else tmp_path + '.flac'
if suffix != '.flac':
audio_segment.export(flac_path, format='flac')
else:
flac_path = tmp_path # Already FLAC, no conversion needed
# Upload to GCS (always as .flac)
output_path = f"jobs/{job_id}/stems/custom_instrumental.flac"
storage.upload_file(flac_path, output_path)
except HTTPException:
raise
@@ -812,6 +821,8 @@ async def upload_custom_instrumental(
finally:
if tmp_path and os.path.exists(tmp_path):
os.unlink(tmp_path)
if flac_path and flac_path != tmp_path and os.path.exists(flac_path):
os.unlink(flac_path)
# Record in job file_urls
job_manager.update_file_url(job_id, 'stems', 'custom_instrumental', output_path)
+8 -1
View File
@@ -372,7 +372,14 @@ def run_encoding(job_id: str, work_dir: Path, config: dict):
instrumental_selection = config.get("instrumental_selection", "clean")
logger.info(f"Instrumental selection from config: {instrumental_selection}")
if instrumental_selection == "with_backing":
if instrumental_selection == "custom":
# User uploaded a custom instrumental or created one via mute-region editing
instrumental = find_file(
work_dir,
"*custom_instrumental*.flac", "*Instrumental Custom*.flac",
"*custom_instrumental*.mp3",
)
elif instrumental_selection == "with_backing":
# User selected instrumental with backing vocals
instrumental = find_file(
work_dir,
+210 -5
View File
@@ -467,7 +467,7 @@ class TestUploadInstrumentalEndpoint:
def test_success_returns_duration(
self, review_job, mock_job_manager, mock_worker_service, patched_client, upload_headers
):
"""Happy path: uploads file, gets duration, stores in GCS, returns duration."""
"""Happy path: uploads FLAC file, gets duration, stores in GCS, returns duration."""
mock_job_manager.get_job.return_value = review_job
mock_storage = MagicMock()
@@ -489,6 +489,8 @@ class TestUploadInstrumentalEndpoint:
assert data["status"] == "success"
assert data["duration_seconds"] == 240.0
assert "240.0s" in data["message"]
# FLAC input should not trigger conversion — no export call
mock_audio_segment.export.assert_not_called()
def test_job_not_found_returns_404(
self, mock_job_manager, mock_worker_service, patched_client, upload_headers
@@ -562,10 +564,10 @@ class TestUploadInstrumentalEndpoint:
"jobs/job-abc/stems/custom_instrumental.flac",
)
def test_extension_inferred_from_content_type(
def test_non_flac_upload_converted_to_flac(
self, review_job, mock_job_manager, mock_worker_service, patched_client, upload_headers
):
"""When filename has no extension, derives it from content-type."""
"""Non-FLAC uploads (mp3, m4a, etc.) are converted to FLAC before storing in GCS."""
mock_job_manager.get_job.return_value = review_job
mock_storage = MagicMock()
@@ -578,14 +580,19 @@ class TestUploadInstrumentalEndpoint:
new_callable=AsyncMock, return_value=None):
patched_client.post(
"/api/jobs/job-abc/upload-instrumental",
files={"file": ("noext", b"audio", "audio/mpeg")},
files={"file": ("track.mp3", b"audio", "audio/mpeg")},
headers=upload_headers,
)
# Always stored as .flac regardless of input format
mock_job_manager.update_file_url.assert_called_with(
"job-abc", "stems", "custom_instrumental",
"jobs/job-abc/stems/custom_instrumental.mp3",
"jobs/job-abc/stems/custom_instrumental.flac",
)
# Conversion to FLAC was triggered
mock_audio_segment.export.assert_called_once()
call_args = mock_audio_segment.export.call_args
assert call_args[1]["format"] == "flac" or call_args[0][1] == "flac"
def test_in_review_status_also_accepted(
self, in_review_job, mock_job_manager, mock_worker_service, patched_client, upload_headers
@@ -945,3 +952,201 @@ class TestUploadedVsCustomSelectionContract:
mock_job_manager.update_state_data.assert_any_call(
"job-abc", "instrumental_selection", "custom"
)
# --- Tests: GCE encoding worker instrumental file resolution ---
class TestGceEncodingInstrumentalResolution:
"""
Tests that the GCE encoding worker's find_file logic correctly resolves
custom instrumentals instead of falling back to clean/with_backing.
"""
def test_find_file_returns_first_match(self, tmp_path):
"""find_file returns the first matching pattern."""
from backend.services.gce_encoding.main import find_file
# Create test files
stems = tmp_path / "stems"
stems.mkdir()
(stems / "custom_instrumental.flac").write_bytes(b"custom")
(stems / "instrumental_clean.flac").write_bytes(b"clean")
result = find_file(tmp_path, "*custom_instrumental*.flac")
assert result is not None
assert "custom_instrumental" in str(result)
def test_custom_selection_finds_custom_instrumental(self, tmp_path):
"""When instrumental_selection is 'custom', find_file should match custom_instrumental.flac."""
from backend.services.gce_encoding.main import find_file
stems = tmp_path / "stems"
stems.mkdir()
(stems / "custom_instrumental.flac").write_bytes(b"custom audio")
(stems / "instrumental_clean.flac").write_bytes(b"clean audio")
(stems / "instrumental_with_backing.flac").write_bytes(b"backing audio")
# Reproduce the exact logic from gce_encoding/main.py run_encoding
instrumental_selection = "custom"
if instrumental_selection == "custom":
instrumental = find_file(
tmp_path,
"*custom_instrumental*.flac", "*Instrumental Custom*.flac",
"*custom_instrumental*.mp3",
)
elif instrumental_selection == "with_backing":
instrumental = find_file(
tmp_path,
"*instrumental_with_backing*.flac", "*Instrumental Backing*.flac",
)
else:
instrumental = find_file(
tmp_path,
"*instrumental_clean*.flac", "*Instrumental Clean*.flac",
)
assert instrumental is not None
assert "custom_instrumental" in str(instrumental)
def test_custom_selection_does_not_fallback_to_clean(self, tmp_path):
"""Custom selection must NOT accidentally resolve to clean instrumental."""
from backend.services.gce_encoding.main import find_file
stems = tmp_path / "stems"
stems.mkdir()
# Only clean instrumental exists — custom is missing
(stems / "instrumental_clean.flac").write_bytes(b"clean audio")
(stems / "instrumental_with_backing.flac").write_bytes(b"backing audio")
instrumental_selection = "custom"
if instrumental_selection == "custom":
instrumental = find_file(
tmp_path,
"*custom_instrumental*.flac", "*Instrumental Custom*.flac",
"*custom_instrumental*.mp3",
)
else:
instrumental = find_file(tmp_path, "*instrumental_clean*.flac")
# Should be None — no custom instrumental exists
assert instrumental is None
def test_clean_selection_still_works(self, tmp_path):
"""Clean selection unaffected by the custom case addition."""
from backend.services.gce_encoding.main import find_file
stems = tmp_path / "stems"
stems.mkdir()
(stems / "custom_instrumental.flac").write_bytes(b"custom")
(stems / "instrumental_clean.flac").write_bytes(b"clean")
(stems / "instrumental_with_backing.flac").write_bytes(b"backing")
instrumental_selection = "clean"
if instrumental_selection == "custom":
instrumental = find_file(tmp_path, "*custom_instrumental*.flac")
elif instrumental_selection == "with_backing":
instrumental = find_file(tmp_path, "*instrumental_with_backing*.flac")
else:
instrumental = find_file(tmp_path, "*instrumental_clean*.flac")
assert instrumental is not None
assert "instrumental_clean" in str(instrumental)
def test_with_backing_selection_still_works(self, tmp_path):
"""With-backing selection unaffected by the custom case addition."""
from backend.services.gce_encoding.main import find_file
stems = tmp_path / "stems"
stems.mkdir()
(stems / "instrumental_clean.flac").write_bytes(b"clean")
(stems / "instrumental_with_backing.flac").write_bytes(b"backing")
instrumental_selection = "with_backing"
if instrumental_selection == "custom":
instrumental = find_file(tmp_path, "*custom_instrumental*.flac")
elif instrumental_selection == "with_backing":
instrumental = find_file(
tmp_path,
"*instrumental_with_backing*.flac", "*Instrumental Backing*.flac",
)
else:
instrumental = find_file(tmp_path, "*instrumental_clean*.flac")
assert instrumental is not None
assert "instrumental_with_backing" in str(instrumental)
# --- Tests: Orchestrator config instrumental path resolution ---
class TestOrchestratorInstrumentalPath:
"""
Tests that create_orchestrator_config_from_job correctly constructs the instrumental
file path for all selection types, including 'custom'.
"""
def _make_job(self, instrumental_selection, existing_instrumental=None, custom_stem=False):
"""Create a Job with the given instrumental selection in state_data."""
file_urls = {"stems": {"instrumental_clean": "jobs/j/stems/instrumental_clean.flac"}}
if custom_stem:
file_urls["stems"]["custom_instrumental"] = "jobs/j/stems/custom_instrumental.flac"
job = Job(
job_id="test-orch",
status=JobStatus.RENDERING_VIDEO,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test Artist",
title="Test Song",
state_data={"instrumental_selection": instrumental_selection},
file_urls=file_urls,
existing_instrumental_gcs_path=existing_instrumental,
)
return job
def test_custom_selection_uses_custom_path(self, tmp_path):
"""Custom selection produces '(Instrumental Custom).flac' path."""
from backend.workers.video_worker_orchestrator import create_orchestrator_config_from_job
job = self._make_job("custom", custom_stem=True)
config = create_orchestrator_config_from_job(job, str(tmp_path))
assert "(Instrumental Custom).flac" in config.instrumental_audio_path
def test_clean_selection_uses_clean_path(self, tmp_path):
"""Clean selection produces '(Instrumental Clean).flac' path."""
from backend.workers.video_worker_orchestrator import create_orchestrator_config_from_job
job = self._make_job("clean")
config = create_orchestrator_config_from_job(job, str(tmp_path))
assert "(Instrumental Clean).flac" in config.instrumental_audio_path
def test_with_backing_selection_uses_backing_path(self, tmp_path):
"""With-backing selection produces '(Instrumental Backing).flac' path."""
from backend.workers.video_worker_orchestrator import create_orchestrator_config_from_job
job = self._make_job("with_backing")
config = create_orchestrator_config_from_job(job, str(tmp_path))
assert "(Instrumental Backing).flac" in config.instrumental_audio_path
def test_existing_instrumental_takes_priority(self, tmp_path):
"""existing_instrumental_gcs_path takes priority over selection type."""
from backend.workers.video_worker_orchestrator import create_orchestrator_config_from_job
job = self._make_job("custom", existing_instrumental="jobs/j/input/user_instrumental.wav")
config = create_orchestrator_config_from_job(job, str(tmp_path))
assert "(Instrumental User).wav" in config.instrumental_audio_path
def test_custom_selection_not_confused_with_backing(self, tmp_path):
"""Regression: custom selection must NOT produce Backing path (the original bug)."""
from backend.workers.video_worker_orchestrator import create_orchestrator_config_from_job
job = self._make_job("custom", custom_stem=True)
config = create_orchestrator_config_from_job(job, str(tmp_path))
assert "Backing" not in config.instrumental_audio_path
assert "Custom" in config.instrumental_audio_path
@@ -727,6 +727,8 @@ def create_orchestrator_config_from_job(
if existing_instrumental:
ext = Path(existing_instrumental).suffix.lower()
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental User){ext}")
elif instrumental_selection == 'custom':
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental Custom).flac")
else:
instrumental_suffix = "Clean" if instrumental_selection == 'clean' else "Backing"
instrumental_path = os.path.join(temp_dir, f"{base_name} (Instrumental {instrumental_suffix}).flac")
+2
View File
@@ -52,6 +52,8 @@
- **CDG Sync Fix for Countdown-Padded Songs** (2026-02-05): Fixed CDG files being 3 seconds out of sync with audio when songs have countdown padding applied. **Root cause**: Video rendering adds 3s countdown padding to audio AND shifts LRC timestamps by +3s. CDG generation was using these countdown-shifted LRC timestamps but with instrumental audio (which has no countdown padding), causing lyrics to appear 3s late. **Fix**: CDG generator now receives `lrc_has_countdown_padding` flag; when True, it strips the countdown segment and shifts timestamps back before generating the CDG. Added 11 unit tests and updated orchestrator/packaging pipeline. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#fix-both-sides-of-dual-code-paths).
- **Custom Instrumental Upload Fix** (2026-02-26): Fixed critical bug where uploaded custom instrumentals were ignored during encoding — GCE encoding worker and orchestrator had no `"custom"` case, falling back to clean instrumental instead. Upload endpoint now converts all formats to FLAC for pipeline consistency. Three fixes: (1) GCE encoding worker `find_file` now searches for `custom_instrumental*.flac` when selection is `"custom"`, (2) orchestrator config constructs correct `(Instrumental Custom).flac` path, (3) upload endpoint converts mp3/m4a/wav/etc. to FLAC before storing. Added 10 new regression tests for GCE resolution and orchestrator paths. See [API.md](API.md#upload-custom-instrumental).
- **Custom Instrumental Upload** (2026-02-25): Added `POST /api/jobs/{job_id}/upload-instrumental` endpoint — was missing from backend, causing 404 in production when users tried to upload custom instrumental files during review. Validates uploaded audio duration matches original (±0.5s tolerance) using ffprobe on GCS signed URLs. Fixed frontend bug where "uploaded" selection type was sent to API instead of "custom" (the only value the backend accepts). Added 22 tests covering upload validation, duration checks, and frontend/backend contract. See [API.md](API.md#upload-custom-instrumental).
- **Unicode Filename Fix** (2026-02-04): Fixed FFmpeg failures (exit code 183) when processing songs with non-ASCII characters in titles. Job `1eab1172` with Russian title "я куплю тебе дом" failed because Python's `str.isalnum()` returns True for ALL Unicode alphanumeric characters (Cyrillic, Arabic, Chinese, emoji, etc.), not just ASCII. These Unicode characters in temp subtitle file paths (`temp_subtitles_zivert___я_куплю_тебе_дом_1770150285852.ass`) caused FFmpeg to fail. **Fix**: Changed filename sanitization from `c.isalnum()` to `c.isascii() and c.isalnum()` in three locations: `video.py:222` (generate_video), `video.py:282` (generate_preview_video), and `remote_cli.py:1970` (output folders). Added comprehensive test suite covering Cyrillic, Chinese, Arabic, emoji, and accented characters. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#unicode-in-file-paths-feb-2026).
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.119.3"
version = "0.119.4"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"