fix: ensemble preset stem tag parsing — restore backing vocals separation (#617)

## Summary

- Fix stem filename parsing bug where preset name `instrumental_clean`
caused the Vocals file to be misclassified as instrumental, preventing
Stage 2 (backing vocals separation) from running
- All new GPU-processed jobs were only producing `instrumental_clean` —
no backing vocals, lead vocals, or combined instrumentals
- The instrumental review page showed "No backing vocals detected" for
all jobs

## Root Cause

Ensemble preset output filenames follow
`{base}_(StemTag)_preset_{name}.flac`. The old code did `"instrumental"
in basename` which matched the **preset name** `instrumental_clean` in
the Vocals file, so `result["vocals"]` was never set, and Stage 2 never
ran.

## Fix

Extract the stem tag from between `_(` and `)_` delimiters and match
only that, not the full filename.

## Test plan

- [x] 5 new regression tests for stem tag parsing (both stages)
- [x] 2 pre-existing tests fixed to use realistic ensemble filenames
- [x] Full unit suite: 1140 passed, 0 failed
- [x] CodeRabbit: no findings
- [ ] After deploy: trigger a test job and verify all stems appear on
instrumental review page

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 4.6 authored and GitHub committed 2026-03-27 12:14:32 -04:00
1 parent ccc1b7c714
commit f58c74e67e
3 files changed
+297 -12

No files matched your search

@@ -0,0 +1,60 @@
# Plan: Fix Ensemble Preset Stem Tag Parsing Bug
**Created:** 2026-03-27
**Branch:** feat/sess-20260326-2142-prod-broken-jobs
**Status:** Complete
## Overview
The ensemble preset stem filename parser in `audio_processor.py` uses naive substring matching (`"instrumental" in basename`) to classify output files as vocals vs instrumental. This breaks when the preset name itself contains the word "instrumental" (e.g., `instrumental_clean`), causing the Vocals file to be misclassified. Stage 2 (backing vocals separation) never runs, so users only get `instrumental_clean` — no backing vocals, no combined instrumentals, and the instrumental review page shows "No backing vocals detected."
## Requirements
- [x] Fix Stage 1 stem tag parsing to correctly identify Vocals vs Instrumental files regardless of preset name
- [x] Fix Stage 2 stem tag parsing with the same approach (same bug pattern)
- [ ] Add unit tests for stem tag parsing with various preset names
- [ ] Add regression test specifically for `instrumental_clean` preset name
- [ ] Verify upload function correctly handles the now-populated backing vocals data
## Technical Approach
**Root cause:** Output filenames follow the pattern `{base}_(StemTag)_preset_{name}.flac` where StemTag is `Vocals`, `Instrumental`, or `No Vocal`. The old code matched substrings against the entire filename, so preset name `instrumental_clean` caused the Vocals file to match `"instrumental" in basename`.
**Fix:** Extract the stem tag from between `_(` and `)_` delimiters and match only against that, not the full filename. This is already implemented — just needs tests and verification.
**Fallback:** If the stem tag extraction fails (no `_(...)_` pattern found), log a warning. This handles edge cases where the separator library changes its naming convention.
## Implementation Steps
1. [x] Fix stem tag parsing in Stage 1 (lines 308-331) — **already done**
2. [x] Fix stem tag parsing in Stage 2 (lines 364-380) — **already done**
3. [ ] Write unit tests for `process_audio_separation` ensemble path:
- Test Stage 1 correctly classifies files with `instrumental_clean` preset
- Test Stage 1 correctly classifies files with `karaoke` preset
- Test Stage 2 correctly classifies backing vocal files
- Test unrecognized stem tags produce warnings
- Test fallback when `_(...)_` pattern not present
4. [ ] Run existing tests to verify no regressions
5. [ ] Manual verification: check that a test job produces all stems (instrumental_clean, backing_vocals, lead_vocals, instrumental_with_backing)
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `karaoke_gen/audio_processor.py` | Modified | Fixed stem tag parsing in Stage 1 (L308-331) and Stage 2 (L364-380) |
| `tests/unit/test_audio_processor_local_gpu.py` | Modify | Add tests for ensemble stem tag parsing |
## Testing Strategy
- **Unit tests:** Mock the `Separator` to return filenames with various preset names, verify correct classification into vocals/instrumental/backing_vocals
- **Regression test:** Specifically test `instrumental_clean` preset to ensure it never regresses
- **Existing tests:** Run full test suite to verify no regressions
- **Manual:** After deploy, trigger a test job and verify all stems appear in GCS and on the instrumental review page
## Open Questions
None — root cause is confirmed and fix is verified.
## Rollback Plan
Revert the single commit. The old behavior (only instrumental_clean uploaded) would return as a known limitation. No data loss risk.
+26 -6
View File
@@ -306,18 +306,29 @@ class AudioProcessor:
self.logger.info(f"Stage 1 ensemble produced {len(output_files)} files: {output_files}")
# Find vocals and instrumental from ensemble output
# Output filenames follow: {base}_(Vocals)_preset_{name}.flac
# Check instrumental/no_vocal first to avoid substring false match
# Output filenames follow: {base}_(StemTag)_preset_{name}.flac
# where StemTag is "Vocals", "Instrumental", or "No Vocal"
# Must match the stem tag specifically, not the preset name
# (e.g., preset "instrumental_clean" contains "instrumental")
for f in output_files:
basename = os.path.basename(f).lower()
if "no_vocal" in basename or "instrumental" in basename:
# Extract stem tag: text between _( and )_ in the filename
stem_tag = ""
tag_start = basename.rfind("_(")
tag_end = basename.find(")_", tag_start) if tag_start >= 0 else -1
if tag_start >= 0 and tag_end >= 0:
stem_tag = basename[tag_start + 2:tag_end]
if stem_tag in ("no vocal", "instrumental", "no_vocal"):
# Move instrumental to track_output_dir per convention
final_path = os.path.join(track_output_dir, os.path.basename(f))
if f != final_path:
shutil.move(f, final_path)
result["clean_instrumental"]["instrumental"] = final_path
elif "vocal" in basename:
elif stem_tag in ("vocals", "vocal"):
result["clean_instrumental"]["vocals"] = f
else:
self.logger.warning(f"Stage 1: Unrecognized stem tag '{stem_tag}' in {basename}")
else:
# Legacy mode — individual model calls (existing code)
separator = Separator(
@@ -354,10 +365,19 @@ class AudioProcessor:
result["backing_vocals"][bv_key] = {}
for f in bv_output_files:
basename = os.path.basename(f).lower()
if "instrumental" in basename or "backing" in basename or "no_vocal" in basename:
# Extract stem tag: text between _( and )_ in the filename
stem_tag = ""
tag_start = basename.rfind("_(")
tag_end = basename.find(")_", tag_start) if tag_start >= 0 else -1
if tag_start >= 0 and tag_end >= 0:
stem_tag = basename[tag_start + 2:tag_end]
if stem_tag in ("no vocal", "instrumental", "no_vocal", "backing"):
result["backing_vocals"][bv_key]["backing_vocals"] = f
elif "vocal" in basename:
elif stem_tag in ("vocals", "vocal"):
result["backing_vocals"][bv_key]["lead_vocals"] = f
else:
self.logger.warning(f"Stage 2: Unrecognized stem tag '{stem_tag}' in {basename}")
else:
# Legacy: individual model calls (existing code)
result["backing_vocals"] = self._separate_backing_vocals(
+211 -6
View File
@@ -204,11 +204,17 @@ class TestLocalSeparationWithPresets:
self.processor.karaoke_preset = None
self.processor.backing_vocals_models = []
# Use realistic ensemble preset filenames
stems_dir = "/tmp/stems"
mock_sep = MagicMock()
mock_sep.separate.return_value = ["/tmp/stems/vocals.flac", "/tmp/stems/instrumental.flac"]
mock_sep.separate.return_value = [
f"{stems_dir}/Song_(Vocals)_preset_instrumental_clean.flac",
f"{stems_dir}/Song_(Instrumental)_preset_instrumental_clean.flac",
]
mock_separator_cls.return_value = mock_sep
mock_shutil.move.side_effect = lambda src, dst: dst
with patch.object(self.processor, '_create_stems_directory', return_value="/tmp/stems"):
with patch.object(self.processor, '_create_stems_directory', return_value=stems_dir):
with patch.object(self.processor, '_normalize_audio_files'):
with patch.object(self.processor, '_generate_combined_instrumentals', return_value={}):
self.processor.process_audio_separation(
@@ -227,11 +233,23 @@ class TestLocalSeparationWithPresets:
"""Stage 2 uses karaoke_preset on a fresh Separator for BV separation."""
os.environ.pop("AUDIO_SEPARATOR_API_URL", None)
mock_sep = MagicMock()
mock_sep.separate.return_value = ["/tmp/stems/vocals.flac", "/tmp/stems/instrumental.flac"]
mock_separator_cls.return_value = mock_sep
stems_dir = "/tmp/stems"
with patch.object(self.processor, '_create_stems_directory', return_value="/tmp/stems"):
# Stage 1 output
s1_vocals = f"{stems_dir}/Song_(Vocals)_preset_instrumental_clean.flac"
s1_instrumental = f"{stems_dir}/Song_(Instrumental)_preset_instrumental_clean.flac"
# Stage 2 output
s2_lead = f"{stems_dir}/Song_(Vocals)_preset_karaoke.flac"
s2_backing = f"{stems_dir}/Song_(No Vocal)_preset_karaoke.flac"
mock_sep1 = MagicMock()
mock_sep1.separate.return_value = [s1_vocals, s1_instrumental]
mock_sep2 = MagicMock()
mock_sep2.separate.return_value = [s2_lead, s2_backing]
mock_separator_cls.side_effect = [mock_sep1, mock_sep2]
mock_shutil.move.side_effect = lambda src, dst: dst
with patch.object(self.processor, '_create_stems_directory', return_value=stems_dir):
with patch.object(self.processor, '_normalize_audio_files'):
with patch.object(self.processor, '_generate_combined_instrumentals', return_value={}):
self.processor.process_audio_separation(
@@ -242,3 +260,190 @@ class TestLocalSeparationWithPresets:
assert mock_separator_cls.call_count == 2
stage2_kwargs = mock_separator_cls.call_args_list[1][1]
assert stage2_kwargs.get("ensemble_preset") == "karaoke"
class TestEnsembleStemTagParsing:
"""Tests for stem tag extraction from ensemble preset output filenames.
Ensemble presets produce files named: {base}_(StemTag)_preset_{name}.flac
The parser must identify the StemTag (between _( and )_) and NOT be confused
by the preset name (e.g., "instrumental_clean" contains "instrumental").
"""
def setup_method(self):
self.processor = AudioProcessor(
logger=logging.getLogger("test"),
log_level=logging.DEBUG,
log_formatter=None,
model_file_dir="/models",
lossless_output_format="FLAC",
clean_instrumental_model="model_bs_roformer_ep_317_sdr_12.9755.ckpt",
backing_vocals_models=["mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt"],
other_stems_models=[],
ffmpeg_base_command="ffmpeg",
)
self.processor.instrumental_preset = "instrumental_clean"
self.processor.karaoke_preset = "karaoke"
@patch.dict(os.environ, {}, clear=False)
@patch("karaoke_gen.audio_processor.shutil")
@patch("karaoke_gen.audio_processor.Separator")
def test_stage1_instrumental_clean_preset_classifies_vocals_correctly(
self, mock_separator_cls, mock_shutil
):
"""Regression: preset name 'instrumental_clean' must not cause Vocals file
to be classified as instrumental (the old substring match bug)."""
os.environ.pop("AUDIO_SEPARATOR_API_URL", None)
# Disable stage 2 to isolate stage 1
self.processor.karaoke_preset = None
self.processor.backing_vocals_models = []
# Realistic filenames from instrumental_clean ensemble preset
stems_dir = "/tmp/stems"
vocals_file = f"{stems_dir}/Song_(Vocals)_preset_instrumental_clean.flac"
instrumental_file = f"{stems_dir}/Song_(Instrumental)_preset_instrumental_clean.flac"
mock_sep = MagicMock()
mock_sep.separate.return_value = [vocals_file, instrumental_file]
mock_separator_cls.return_value = mock_sep
# shutil.move returns the destination path
mock_shutil.move.side_effect = lambda src, dst: dst
with patch.object(self.processor, '_create_stems_directory', return_value=stems_dir):
with patch.object(self.processor, '_normalize_audio_files'):
with patch.object(self.processor, '_generate_combined_instrumentals', return_value={}):
result = self.processor.process_audio_separation(
"/tmp/test.wav", "Artist - Song", "/tmp/output"
)
# Vocals must be set (this is what the bug prevented)
assert result["clean_instrumental"]["vocals"] == vocals_file
# Instrumental must also be set
assert result["clean_instrumental"]["instrumental"] is not None
@patch.dict(os.environ, {}, clear=False)
@patch("karaoke_gen.audio_processor.shutil")
@patch("karaoke_gen.audio_processor.Separator")
def test_stage1_vocals_path_enables_stage2(self, mock_separator_cls, mock_shutil):
"""When Stage 1 correctly identifies vocals, Stage 2 runs and produces
backing vocals + lead vocals stems."""
os.environ.pop("AUDIO_SEPARATOR_API_URL", None)
stems_dir = "/tmp/stems"
# Stage 1 output (instrumental_clean preset)
s1_vocals = f"{stems_dir}/Song_(Vocals)_preset_instrumental_clean.flac"
s1_instrumental = f"{stems_dir}/Song_(Instrumental)_preset_instrumental_clean.flac"
# Stage 2 output (karaoke preset)
s2_lead = f"{stems_dir}/Song_(Vocals)_preset_karaoke.flac"
s2_backing = f"{stems_dir}/Song_(No Vocal)_preset_karaoke.flac"
mock_sep1 = MagicMock()
mock_sep1.separate.return_value = [s1_vocals, s1_instrumental]
mock_sep2 = MagicMock()
mock_sep2.separate.return_value = [s2_lead, s2_backing]
mock_separator_cls.side_effect = [mock_sep1, mock_sep2]
mock_shutil.move.side_effect = lambda src, dst: dst
with patch.object(self.processor, '_create_stems_directory', return_value=stems_dir):
with patch.object(self.processor, '_normalize_audio_files'):
with patch.object(self.processor, '_generate_combined_instrumentals', return_value={}):
result = self.processor.process_audio_separation(
"/tmp/test.wav", "Artist - Song", "/tmp/output"
)
# Stage 2 should have run (Separator instantiated twice)
assert mock_separator_cls.call_count == 2
# Backing vocals correctly classified
bv = result["backing_vocals"]["karaoke"]
assert bv["lead_vocals"] == s2_lead
assert bv["backing_vocals"] == s2_backing
@patch.dict(os.environ, {}, clear=False)
@patch("karaoke_gen.audio_processor.shutil")
@patch("karaoke_gen.audio_processor.Separator")
def test_stage1_no_vocal_tag_recognized(self, mock_separator_cls, mock_shutil):
"""Files with '(No Vocal)' stem tag are correctly classified as instrumental."""
os.environ.pop("AUDIO_SEPARATOR_API_URL", None)
self.processor.karaoke_preset = None
self.processor.backing_vocals_models = []
stems_dir = "/tmp/stems"
vocals_file = f"{stems_dir}/Song_(Vocals)_preset_instrumental_clean.flac"
no_vocal_file = f"{stems_dir}/Song_(No Vocal)_preset_instrumental_clean.flac"
mock_sep = MagicMock()
mock_sep.separate.return_value = [vocals_file, no_vocal_file]
mock_separator_cls.return_value = mock_sep
mock_shutil.move.side_effect = lambda src, dst: dst
with patch.object(self.processor, '_create_stems_directory', return_value=stems_dir):
with patch.object(self.processor, '_normalize_audio_files'):
with patch.object(self.processor, '_generate_combined_instrumentals', return_value={}):
result = self.processor.process_audio_separation(
"/tmp/test.wav", "Artist - Song", "/tmp/output"
)
assert result["clean_instrumental"]["vocals"] == vocals_file
assert result["clean_instrumental"]["instrumental"] is not None
@patch.dict(os.environ, {}, clear=False)
@patch("karaoke_gen.audio_processor.shutil")
@patch("karaoke_gen.audio_processor.Separator")
def test_unrecognized_stem_tag_logs_warning(self, mock_separator_cls, mock_shutil):
"""Files with unrecognized stem tags produce a warning log."""
os.environ.pop("AUDIO_SEPARATOR_API_URL", None)
self.processor.karaoke_preset = None
self.processor.backing_vocals_models = []
stems_dir = "/tmp/stems"
weird_file = f"{stems_dir}/Song_(Drums)_preset_instrumental_clean.flac"
mock_sep = MagicMock()
mock_sep.separate.return_value = [weird_file]
mock_separator_cls.return_value = mock_sep
with patch.object(self.processor, '_create_stems_directory', return_value=stems_dir):
with patch.object(self.processor, '_normalize_audio_files'):
with patch.object(self.processor, '_generate_combined_instrumentals', return_value={}):
with patch.object(self.processor.logger, 'warning') as mock_warn:
result = self.processor.process_audio_separation(
"/tmp/test.wav", "Artist - Song", "/tmp/output"
)
mock_warn.assert_called_once()
assert "drums" in mock_warn.call_args[0][0].lower()
@patch.dict(os.environ, {}, clear=False)
@patch("karaoke_gen.audio_processor.shutil")
@patch("karaoke_gen.audio_processor.Separator")
def test_filename_without_stem_tag_pattern_logs_warning(self, mock_separator_cls, mock_shutil):
"""Files missing the _(Tag)_ pattern produce a warning."""
os.environ.pop("AUDIO_SEPARATOR_API_URL", None)
self.processor.karaoke_preset = None
self.processor.backing_vocals_models = []
stems_dir = "/tmp/stems"
plain_file = f"{stems_dir}/vocals.flac"
mock_sep = MagicMock()
mock_sep.separate.return_value = [plain_file]
mock_separator_cls.return_value = mock_sep
with patch.object(self.processor, '_create_stems_directory', return_value=stems_dir):
with patch.object(self.processor, '_normalize_audio_files'):
with patch.object(self.processor, '_generate_combined_instrumentals', return_value={}):
with patch.object(self.processor.logger, 'warning') as mock_warn:
result = self.processor.process_audio_separation(
"/tmp/test.wav", "Artist - Song", "/tmp/output"
)
mock_warn.assert_called_once()
# Should not have classified anything
assert result["clean_instrumental"].get("vocals") is None
assert result["clean_instrumental"].get("instrumental") is None