fix(lyrics-cache): resolve cache_dir at instantiation — stop burning AudioShake credits on every job (#1017)

## Problem

The cross-container AudioShake/lyrics transcription cache has been
**completely non-functional in production**. Every lyrics job logs
`Cache sync from GCS: 0 downloaded, 14 not found` and the GCS
`lyrics-transcriber-cache/` prefix is **empty** — so every job
(including repeat songs and every daily/canary E2E run of the same
track) makes a fresh, credit-spending AudioShake alignment call.

## Root cause

`OutputConfig.cache_dir` used a bare
`os.getenv("LYRICS_TRANSCRIBER_CACHE_DIR", ...)` as its dataclass field
default. Dataclass defaults are evaluated **once at import time**. The
cloud lyrics worker sets `LYRICS_TRANSCRIBER_CACHE_DIR` to a per-job
temp dir at **runtime** (`backend/workers/lyrics_worker.py:316`), long
after the module imported — so it was ignored.

Result: the transcriber wrote its cache to the frozen default
`~/lyrics-transcriber-cache`, while the GCS-sync agent read/wrote
`{temp_dir}/lyrics-cache`. The upload step scanned the sync dir, found
nothing the transcriber had written, and uploaded nothing → GCS stayed
empty forever → every future job missed.

## Fix

Use `field(default_factory=...)` for `cache_dir` (and `output_dir`,
which had the same latent `os.getcwd()`-at-import bug) so both resolve
when an `OutputConfig` is **constructed**, honoring the worker's runtime
env. This mirrors the existing `default_factory` pattern already used
for the local-whisper fields in `TranscriberConfig`.

After this deploys, the first job for a given audio hash populates GCS,
and subsequent jobs with the same audio (all the daily/canary E2E runs,
plus real repeat songs) hit the cache and spend **zero** AudioShake
credits.

## Testing

- New regression tests
(`tests/unit/lyrics_transcriber/test_output_config_cache_dir.py`) assert
`cache_dir` tracks the env set *after* import, is re-evaluated per
instance, respects an explicit override, and that `output_dir` tracks
cwd at construction.
- Existing `tests/unit/test_lyrics_cache_service.py` still passes (21
passed).

## Verification plan (post-deploy)

Watch a lyrics job's logs for `Cache sync to GCS: N uploaded` on the
first run, then `Cache sync from GCS: >0 downloaded` on the next run of
the same audio, and confirm objects appear under
`gs://karaoke-gen-storage-nomadkaraoke/lyrics-transcriber-cache/`.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Cache and output directories now reflect the environment and current
working directory when each job starts.
* Runtime cache directory changes are honored, improving reuse of cached
transcription results.

* **Tests**
* Added coverage for cache path resolution, fallback behavior,
overrides, cache reuse, and transcription pipeline outcomes.
* Added validation for malformed countdown settings and local GPU
processing handoffs.

* **Chores**
  * Updated the project version to 0.229.1.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 4.8 authored and GitHub committed 2026-09-17 15:14:15 -04:00
1 parent 0e3b2f6e08
commit 303ff7e728
6 files changed
+285 -6

No files matched your search

+16 -4
View File
@@ -36,10 +36,22 @@ class OutputConfig:
output_styles_json: str
default_max_line_length: int = 36
styles: Dict[str, Any] = field(default_factory=dict)
output_dir: Optional[str] = os.getcwd()
cache_dir: str = os.getenv(
"LYRICS_TRANSCRIBER_CACHE_DIR",
os.path.join(os.path.expanduser("~"), "lyrics-transcriber-cache")
# output_dir / cache_dir MUST use default_factory so they are resolved at
# instantiation time, not at import time. The cloud lyrics worker sets
# LYRICS_TRANSCRIBER_CACHE_DIR to a per-job temp dir at *runtime* (after this
# module is already imported) so that the transcriber writes its cache where
# the GCS cache-sync agent reads/writes it. A bare `os.getenv(...)` default is
# captured once at import and ignores that runtime value — which silently
# broke the cross-container AudioShake cache (transcriber wrote to
# ~/lyrics-transcriber-cache while the sync used {temp}/lyrics-cache), so the
# GCS cache never populated and every job made a fresh, credit-spending
# AudioShake call. default_factory re-reads the env on each construction.
output_dir: Optional[str] = field(default_factory=os.getcwd)
cache_dir: str = field(
default_factory=lambda: os.getenv(
"LYRICS_TRANSCRIBER_CACHE_DIR",
os.path.join(os.path.expanduser("~"), "lyrics-transcriber-cache"),
)
)
fetch_lyrics: bool = True
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.229.2"
version = "0.229.3"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"
@@ -0,0 +1,120 @@
"""
Tests for BaseTranscriber's on-disk cache round-trip.
This is the mechanism the OutputConfig.cache_dir fix depends on: the transcriber
writes its raw + converted API responses into `cache_dir` (keyed by the audio
file's MD5), and a subsequent transcription of the same audio reads them back
instead of calling the (credit-spending) provider API. The cloud worker's GCS
cache-sync relies on those files landing in the configured directory.
"""
import os
import pytest
from karaoke_gen.lyrics_transcriber.transcribers.base_transcriber import BaseTranscriber
from karaoke_gen.lyrics_transcriber.types import TranscriptionData
class FakeTranscriber(BaseTranscriber):
"""Concrete transcriber that counts how often it hits the 'API'."""
def __init__(self, cache_dir, logger=None):
super().__init__(cache_dir=cache_dir, logger=logger)
self.perform_calls = 0
def get_name(self) -> str:
return "FakeProvider"
def _perform_transcription(self, audio_filepath: str):
self.perform_calls += 1
# Raw API payload shape mirrors what real providers return before
# conversion; here it is already the TranscriptionData dict form.
return {
"segments": [],
"words": [],
"text": "hello world",
"source": "fakeprovider",
"metadata": {"audio": os.path.basename(audio_filepath)},
}
def _convert_result_format(self, raw_data) -> TranscriptionData:
return TranscriptionData(
segments=raw_data.get("segments", []),
words=raw_data.get("words", []),
text=raw_data["text"],
source=raw_data["source"],
metadata=raw_data.get("metadata", {}),
)
@pytest.fixture
def audio_file(tmp_path):
path = tmp_path / "song.wav"
path.write_bytes(b"deterministic fake audio bytes for md5 hashing")
return str(path)
def test_first_call_transcribes_and_writes_cache(tmp_path, audio_file):
cache_dir = tmp_path / "cache"
t = FakeTranscriber(cache_dir=str(cache_dir))
result = t.transcribe(audio_file)
assert result.text == "hello world"
assert t.perform_calls == 1
# Both raw and converted cache files land in the configured cache_dir,
# named "{provider}_{audio_md5}_{suffix}.json".
written = sorted(p.name for p in cache_dir.glob("fakeprovider_*.json"))
assert any(name.endswith("_raw.json") for name in written)
assert any(name.endswith("_converted.json") for name in written)
def test_second_call_hits_converted_cache_without_reperforming(tmp_path, audio_file):
cache_dir = tmp_path / "cache"
t = FakeTranscriber(cache_dir=str(cache_dir))
t.transcribe(audio_file)
assert t.perform_calls == 1
# Same audio -> same MD5 -> converted-cache hit -> no second API call.
result2 = t.transcribe(audio_file)
assert result2.text == "hello world"
assert t.perform_calls == 1
def test_raw_cache_hit_when_only_raw_present(tmp_path, audio_file):
cache_dir = tmp_path / "cache"
t = FakeTranscriber(cache_dir=str(cache_dir))
t.transcribe(audio_file)
# Delete only the converted cache so the raw-cache branch is exercised.
for p in cache_dir.glob("*_converted.json"):
p.unlink()
result = t.transcribe(audio_file)
assert result.text == "hello world"
# Still no new API call — served from the raw cache.
assert t.perform_calls == 1
# The converted cache is rebuilt from raw.
assert list(cache_dir.glob("*_converted.json"))
def test_corrupt_cache_file_is_ignored(tmp_path, audio_file):
cache_dir = tmp_path / "cache"
t = FakeTranscriber(cache_dir=str(cache_dir))
t.transcribe(audio_file)
# Corrupt both cache files -> loader returns None -> re-transcribes.
for p in cache_dir.glob("fakeprovider_*.json"):
p.write_text("{ not valid json")
result = t.transcribe(audio_file)
assert result.text == "hello world"
assert t.perform_calls == 2
def test_missing_audio_file_raises(tmp_path):
t = FakeTranscriber(cache_dir=str(tmp_path / "cache"))
with pytest.raises(FileNotFoundError):
t.transcribe(str(tmp_path / "does-not-exist.wav"))
assert t.perform_calls == 0
@@ -0,0 +1,62 @@
"""
Regression tests for OutputConfig.cache_dir / output_dir resolution timing.
Background: the cloud lyrics worker sets LYRICS_TRANSCRIBER_CACHE_DIR to a
per-job temp dir at *runtime* (after the config module is already imported) so
that the transcriber writes its AudioShake/lyrics cache into the same directory
the GCS cache-sync agent reads/writes. Previously OutputConfig.cache_dir used a
bare `os.getenv(...)` dataclass default, which is evaluated once at import time
and therefore ignored the worker's runtime value. That silently broke the
cross-container cache and made every job spend a fresh AudioShake credit.
These tests lock in that cache_dir (and output_dir) are resolved when an
OutputConfig is *instantiated*, honoring the current environment.
"""
import os
from unittest.mock import patch
from karaoke_gen.lyrics_transcriber.core.config import OutputConfig
STYLES = "/tmp/does-not-need-to-exist-styles.json"
def test_cache_dir_reads_env_set_after_import():
"""cache_dir must reflect LYRICS_TRANSCRIBER_CACHE_DIR set at runtime."""
runtime_dir = "/tmp/job-abc123/lyrics-cache"
with patch.dict(os.environ, {"LYRICS_TRANSCRIBER_CACHE_DIR": runtime_dir}):
config = OutputConfig(output_styles_json=STYLES)
assert config.cache_dir == runtime_dir
def test_cache_dir_falls_back_to_home_when_env_unset():
"""Without the env var, cache_dir falls back to ~/lyrics-transcriber-cache."""
env = {k: v for k, v in os.environ.items() if k != "LYRICS_TRANSCRIBER_CACHE_DIR"}
with patch.dict(os.environ, env, clear=True):
config = OutputConfig(output_styles_json=STYLES)
expected = os.path.join(os.path.expanduser("~"), "lyrics-transcriber-cache")
assert config.cache_dir == expected
def test_cache_dir_is_reevaluated_per_instance():
"""Two instances constructed under different env values must differ."""
with patch.dict(os.environ, {"LYRICS_TRANSCRIBER_CACHE_DIR": "/tmp/first"}):
first = OutputConfig(output_styles_json=STYLES)
with patch.dict(os.environ, {"LYRICS_TRANSCRIBER_CACHE_DIR": "/tmp/second"}):
second = OutputConfig(output_styles_json=STYLES)
assert first.cache_dir == "/tmp/first"
assert second.cache_dir == "/tmp/second"
def test_explicit_cache_dir_argument_wins():
"""An explicitly passed cache_dir overrides the env-derived default."""
with patch.dict(os.environ, {"LYRICS_TRANSCRIBER_CACHE_DIR": "/tmp/env-dir"}):
config = OutputConfig(output_styles_json=STYLES, cache_dir="/tmp/explicit")
assert config.cache_dir == "/tmp/explicit"
def test_output_dir_defaults_to_cwd_at_instantiation(tmp_path, monkeypatch):
"""output_dir default should track the cwd at construction, not import."""
monkeypatch.chdir(tmp_path)
config = OutputConfig(output_styles_json=STYLES)
assert os.path.realpath(config.output_dir) == os.path.realpath(str(tmp_path))
@@ -98,6 +98,11 @@ class TestProcessAudioSeparationMode:
mock_jm = MagicMock()
mock_jm.get_job.return_value = mock_job
# advance_to_screens_if_ready is an async method that the worker awaits
# after a successful separation (durable prep->screens handoff, #1002).
# It must be an AsyncMock, or `await` raises TypeError and the worker
# reports status=error / result=False.
mock_jm.advance_to_screens_if_ready = AsyncMock(return_value=True)
mock_job_manager_cls.return_value = mock_jm
mock_settings_obj = MagicMock()
+81 -1
View File
@@ -210,10 +210,90 @@ class TestTranscriptionStage:
with patch('karaoke_gen.lyrics_processor.LyricsProcessor', side_effect=Exception("Test error")):
result = await stage.execute(context)
assert result.status == StageStatus.FAILED
assert "Test error" in result.error_message
@pytest.mark.asyncio
async def test_execute_happy_path_builds_outputs(self, tmp_path):
"""Full execute: transcribe, collect file outputs, parse countdown padding."""
from karaoke_gen.pipeline.stages.transcription import TranscriptionStage
from karaoke_gen.pipeline.context import PipelineContext
from karaoke_gen.pipeline.base import StageStatus
output_dir = tmp_path / "out"
lyrics_dir = output_dir / "lyrics"
lyrics_dir.mkdir(parents=True)
# A countdown padding file the stage should read back as a float.
(lyrics_dir / "countdown_padding_seconds.txt").write_text("3.0\n")
stage = TranscriptionStage(skip_transcription=False)
# style_params (no style_params_json) -> exercises the temp-file write branch.
context = PipelineContext(
job_id="test",
artist="Artist",
title="Title",
input_audio_path="/audio.flac",
output_dir=str(output_dir),
style_params={"foo": "bar"},
)
mock_processor = MagicMock()
mock_processor.transcribe_lyrics.return_value = {
"lrc_filepath": "/out/lyrics.lrc",
"ass_filepath": "/out/lyrics.ass",
"corrected_txt_path": "/out/corrected.txt",
}
mock_processor.corrections_result = {"corrections": []}
with patch('karaoke_gen.lyrics_processor.LyricsProcessor', return_value=mock_processor):
result = await stage.execute(context)
assert result.status == StageStatus.COMPLETED
assert result.outputs["lrc_filepath"] == "/out/lyrics.lrc"
assert result.outputs["ass_filepath"] == "/out/lyrics.ass"
assert result.outputs["corrected_txt_path"] == "/out/corrected.txt"
assert result.outputs["corrections_result"] == {"corrections": []}
assert result.outputs["countdown_padding_seconds"] == 3.0
# transcribe_lyrics received the context's audio path and artist/title.
_, kwargs = mock_processor.transcribe_lyrics.call_args
assert kwargs["input_audio_wav"] == "/audio.flac"
assert kwargs["artist"] == "Artist"
# style_params (no style_params_json) makes the stage write a temp file and
# register it on the context; clean it up so the test leaves no /tmp files.
context.cleanup_temp_paths()
@pytest.mark.asyncio
async def test_execute_ignores_malformed_countdown_file(self, tmp_path):
"""A non-numeric countdown file is ignored, not fatal."""
from karaoke_gen.pipeline.stages.transcription import TranscriptionStage
from karaoke_gen.pipeline.context import PipelineContext
from karaoke_gen.pipeline.base import StageStatus
output_dir = tmp_path / "out"
lyrics_dir = output_dir / "lyrics"
lyrics_dir.mkdir(parents=True)
(lyrics_dir / "countdown_padding_seconds.txt").write_text("not-a-number")
stage = TranscriptionStage(skip_transcription=False)
context = PipelineContext(
job_id="test",
artist="Artist",
title="Title",
input_audio_path="/audio.flac",
output_dir=str(output_dir),
)
mock_processor = MagicMock()
mock_processor.transcribe_lyrics.return_value = {}
with patch('karaoke_gen.lyrics_processor.LyricsProcessor', return_value=mock_processor):
result = await stage.execute(context)
assert result.status == StageStatus.COMPLETED
assert "countdown_padding_seconds" not in result.outputs
class TestScreensStage:
"""Tests for ScreensStage."""