fix(fallbacks): T8 — fail loud cleanups (docstrings, download errors, admin-token signal) (#805)

## Summary
First (low-risk) batch of the silent-fallback audit — see
`docs/archive/2026-06-09-fallback-audit-plan.md` for the full catalogue
and Andrew's per-theme decisions. This PR is **Theme 8**: fix
lying/stale docstrings, surface specific download-failure reasons, and
make an empty admin-token config a loud signal.

## Changes
- **Fail-open docstring lie → fail-closed**
(`credit_evaluation_service.py`, `config.py`): the AI credit evaluator
is fail-CLOSED (`pending_review`, no auto-grant) but its docstrings
claimed fail-open. Corrected so a future maintainer can't "fix" the code
into a free-credit-abuse hole.
- **Stale yt-dlp docstrings** (`youtube_download_service.py`,
`lyrics_worker.py`): removed text still describing the deleted
local-yt-dlp fallback. flacfetch is the sole downloader; the service
raises when unconfigured.
- **Specific download errors** (`audio_worker.py`, `lyrics_worker.py`):
`download_audio()` now raises a specific `DownloadError` per failure
mode instead of returning `None` → generic "Failed to download audio
file". Workers map it to a clear `Audio download failed: <reason>`
job-failure message.
- **Loud admin-token signal** (`auth_service.py`): empty `ADMIN_TOKENS`
now logs at ERROR (error-monitor → Discord) instead of a quiet warning.
It fails closed (403 everywhere) and looked like an auth bug rather than
a config error.
- Adds the audit plan doc + a `LESSONS-LEARNED` principle ("don't invent
silent fallbacks — ask or fail loud").

## Testing
- [x] Affected backend tests pass locally (85 passed): `test_workers`,
`test_workers_extended`, `test_audio_worker_download`,
`test_auth_admin_tokens`
- [x] Import sanity check (no circular import from new `DownloadError`
imports)
- Updated `test_download_audio_url_without_gcs_path_refuses` to expect
`DownloadError`; added no-source raise tests for both workers +
admin-token tests.

## Review
- [x] CodeRabbit CLI review completed locally (1 minor doc finding,
fixed)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 4.8 authored and GitHub committed 2026-08-22 18:19:30 -04:00
1 parent 3d56dd4108
commit 27a5d8fdff
11 files changed
+482 -46

No files matched your search

+2 -1
View File
@@ -85,7 +85,8 @@ class Settings(BaseSettings):
# AI Credit Evaluation (anti-abuse gating for free credits)
# When enabled, welcome and feedback credit grants are evaluated by Gemini
# before granting. Fail-open: if evaluation fails, credits are granted anyway.
# before granting. Fail-CLOSED: if evaluation fails, the decision becomes
# "pending_review" (no auto-grant; an admin is notified to decide).
credit_eval_enabled: bool = os.getenv("CREDIT_EVAL_ENABLED", "true").lower() in ("true", "1", "yes")
credit_eval_model: str = os.getenv("CREDIT_EVAL_MODEL", "gemini-3.1-pro-preview")
+9 -1
View File
@@ -87,7 +87,15 @@ class AuthService:
if self.admin_tokens:
logger.info(f"Loaded {len(self.admin_tokens)} admin token(s)")
else:
logger.warning("No admin tokens configured! Set ADMIN_TOKENS environment variable.")
# Loud signal: with no admin tokens, every admin/internal-worker
# endpoint fails closed (403). That's safe but looks like an auth bug
# rather than a config error, so log at ERROR (picked up by the error
# monitor → Discord alert) instead of a quiet warning.
logger.error(
"No admin tokens configured! All admin and internal-worker "
"endpoints will reject every request (403). Set the ADMIN_TOKENS "
"secret/env var. This is almost certainly a misconfiguration."
)
def _validate_oidc_token(self, token: str) -> Optional[AuthResult]:
"""
@@ -2,8 +2,9 @@
AI-powered credit evaluation service.
Evaluates whether to grant free credits (welcome or feedback) by collecting
abuse signals and asking Gemini for a grant/deny decision. Fail-open: if
the AI call fails for any reason, credits are granted.
abuse signals and asking Gemini for a grant/deny decision. Fail-CLOSED: if
the AI call fails for any reason, the decision becomes "pending_review" (no
credits auto-granted; an admin is notified to decide manually).
"""
import json
import logging
@@ -25,7 +26,7 @@ class CreditEvaluation:
decision: str # "grant" or "deny"
reasoning: str
confidence: float
error: Optional[str] = None # Set if evaluation failed (decision defaults to "grant")
error: Optional[str] = None # Set if evaluation failed (decision defaults to "pending_review")
signals: Optional[dict] = None # Collected abuse signals (for admin review emails)
+7 -5
View File
@@ -2,13 +2,15 @@
YouTube download service for cloud backend.
This service provides a single entry point for all YouTube downloads in the cloud.
It automatically routes downloads through the remote flacfetch service when configured,
avoiding YouTube bot detection issues on Cloud Run IPs.
flacfetch is the SOLE downloader: downloads are always routed through the remote
flacfetch service, which has YouTube cookies and avoids bot detection on Cloud Run IPs.
The flow is:
1. Check if remote flacfetch is configured (FLACFETCH_API_URL)
2. If yes: Use FlacfetchClient.download_by_id() - downloads on VM, uploads to GCS
3. If no: Use local yt_dlp (with warning about bot detection risk)
3. If no: raise YouTubeDownloadError — there is NO local yt-dlp fallback. We never
run yt-dlp inside the Cloud Run container (it's blocked by bot detection and a
silent fallback previously masked a real download bug — 2026-06-08 incident).
All entry points (audio search selection, direct URL submission) should use this
service for YouTube downloads to ensure consistent behavior.
@@ -34,8 +36,8 @@ class YouTubeDownloadService:
When remote flacfetch is configured (FLACFETCH_API_URL), downloads happen
on the flacfetch VM which has YouTube cookies and avoids bot detection.
When remote is not configured, falls back to local yt_dlp with a warning
that downloads may be blocked on Cloud Run IPs.
When remote is not configured, this raises YouTubeDownloadError. There is no
local yt-dlp fallback — flacfetch is the sole downloader.
Usage:
service = get_youtube_download_service()
+42
View File
@@ -0,0 +1,42 @@
"""
Tests for admin-token initialization in AuthService.
An empty ADMIN_TOKENS configuration silently breaks every admin/internal-worker
endpoint (they fail closed with 403). That's safe but looks like an auth bug
rather than a config error, so it must be a LOUD signal: an ERROR log (picked up
by the error monitor → Discord alert), not a quiet warning.
"""
from unittest.mock import patch, MagicMock
from backend.services.auth_service import AuthService
def _make_service(admin_tokens_value):
"""Construct AuthService with a controlled settings.admin_tokens value."""
with patch("backend.services.auth_service.FirestoreService"), \
patch("backend.services.auth_service.get_settings") as mock_get_settings, \
patch("backend.services.auth_service.logger") as mock_logger:
mock_settings = MagicMock()
mock_settings.admin_tokens = admin_tokens_value
mock_settings.scheduler_service_account = "karaoke-backend@nomadkaraoke.iam.gserviceaccount.com"
mock_get_settings.return_value = mock_settings
service = AuthService()
return service, mock_logger
def test_no_admin_tokens_logs_error_not_warning():
"""Empty ADMIN_TOKENS → ERROR-level loud signal, not a quiet warning."""
service, mock_logger = _make_service("")
assert service.admin_tokens == []
mock_logger.error.assert_called_once()
# It used to be a quiet warning — make sure it isn't anymore.
mock_logger.warning.assert_not_called()
def test_admin_tokens_present_no_error():
"""Configured admin tokens → no error, tokens parsed."""
service, mock_logger = _make_service("tok1, tok2 ,tok3")
assert service.admin_tokens == ["tok1", "tok2", "tok3"]
mock_logger.error.assert_not_called()
+36 -10
View File
@@ -816,20 +816,46 @@ class TestDownloadAudioWithUrl:
@pytest.mark.asyncio
async def test_download_audio_url_without_gcs_path_refuses(self, mock_job_with_url, mock_job_manager, mock_storage):
"""A URL job with no GCS file must fail (no local yt-dlp fallback)."""
"""A URL job with no GCS file must fail loudly (no local yt-dlp fallback)."""
from backend.workers.audio_worker import download_audio
from backend.services.audio_search_service import DownloadError
with tempfile.TemporaryDirectory() as temp_dir:
result = await download_audio(
"test123",
temp_dir,
mock_storage,
mock_job_with_url,
job_manager_instance=mock_job_manager
)
# Fails loudly with a specific DownloadError rather than returning
# None (which used to surface a generic "Failed to download audio file").
with pytest.raises(DownloadError, match="flacfetch service"):
await download_audio(
"test123",
temp_dir,
mock_storage,
mock_job_with_url,
job_manager_instance=mock_job_manager
)
# Refuses to download locally → returns None, never touches storage
assert result is None
# Refuses to download locally → never touches storage
mock_storage.download_file.assert_not_called()
@pytest.mark.asyncio
async def test_download_audio_no_source_raises(self, mock_job_manager, mock_storage):
"""A job with no GCS path, no file_urls and no URL must fail loudly."""
from backend.workers.audio_worker import download_audio
from backend.services.audio_search_service import DownloadError
no_source_job = Job(
job_id="test123",
status=JobStatus.PROCESSING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
artist="Test Artist",
title="Test Song",
)
with tempfile.TemporaryDirectory() as temp_dir:
with pytest.raises(DownloadError, match="No input audio source"):
await download_audio(
"test123", temp_dir, mock_storage, no_source_job,
job_manager_instance=mock_job_manager,
)
mock_storage.download_file.assert_not_called()
@pytest.mark.asyncio
+22
View File
@@ -176,3 +176,25 @@ class TestWorkerFileOperations:
result = await download_audio("test123", temp_dir, mock_storage, mock_job, mock_job_manager)
mock_storage.download_file.assert_called()
@pytest.mark.asyncio
async def test_lyrics_download_audio_no_source_raises(self):
"""Lyrics worker download_audio must fail loudly when there's no source."""
from backend.workers.lyrics_worker import download_audio
from backend.services.audio_search_service import DownloadError
mock_storage = MagicMock()
mock_job_manager = MagicMock()
mock_job = Job(
job_id="test123",
status=JobStatus.TRANSCRIBING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
# No input_media_gcs_path and no url → nothing to download
)
mock_job_manager.get_job.return_value = mock_job
with tempfile.TemporaryDirectory() as temp_dir:
with pytest.raises(DownloadError, match="No input audio available"):
await download_audio("test123", temp_dir, mock_storage, mock_job, mock_job_manager)
mock_storage.download_file.assert_not_called()
+33 -9
View File
@@ -24,6 +24,7 @@ from typing import Optional, Dict, Any
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.audio_search_service import DownloadError
from backend.services.job_health_service import validate_worker_can_run
from backend.config import get_settings
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
@@ -213,9 +214,9 @@ async def process_audio_separation(job_id: str) -> bool:
# Download audio file from GCS or URL
with job_span("download-audio", job_id) as download_span:
job_log.info("Downloading audio file...")
# download_audio raises DownloadError with a specific reason
# on failure (no silent None → generic-error fallback).
audio_path = await download_audio(job_id, temp_dir, storage, job, job_manager_instance=job_manager)
if not audio_path:
raise Exception("Failed to download audio file")
job_log.info(f"Audio downloaded: {os.path.basename(audio_path)}")
download_span.set_attribute("audio_file", os.path.basename(audio_path))
download_span.set_attribute("source", "url" if job.url else "gcs")
@@ -363,6 +364,16 @@ async def process_audio_separation(job_id: str) -> bool:
logger.info(f"[job:{job_id}] WORKER_END worker=audio status=success duration={duration:.1f}s")
return True
except DownloadError as e:
duration = time.time() - start_time
logger.error(f"[job:{job_id}] WORKER_END worker=audio status=error duration={duration:.1f}s error={e}", exc_info=True)
job_manager.mark_job_failed(
job_id=job_id,
error_message=f"Audio download failed: {str(e)}",
error_details={"stage": "download_audio", "error": str(e)}
)
return False
except Exception as e:
duration = time.time() - start_time
logger.error(f"[job:{job_id}] WORKER_END worker=audio status=error duration={duration:.1f}s error={e}", exc_info=True)
@@ -396,7 +407,7 @@ async def download_audio(
storage: StorageService,
job,
job_manager_instance: JobManager = None
) -> Optional[str]:
) -> str:
"""
Fetch the job's audio file from GCS to a local temp directory.
@@ -414,9 +425,14 @@ async def download_audio(
storage: StorageService instance
job: Job object with URL or GCS path
job_manager_instance: Optional JobManager instance for updating job metadata
Returns:
Path to downloaded audio file, or None if failed
Path to the downloaded audio file.
Raises:
DownloadError: with a specific reason if the audio can't be obtained.
We fail loudly with the real cause rather than returning None and
surfacing a generic "Failed to download audio file" to the user.
"""
try:
from karaoke_gen.utils import sanitize_filename
@@ -464,14 +480,22 @@ async def download_audio(
"fetched via the flacfetch service before processing; refusing to "
"download locally."
)
return None
raise DownloadError(
"URL job reached the audio worker without a downloaded input file. "
"URLs must be fetched via the flacfetch service before processing; "
"we never download locally on Cloud Run."
)
logger.error(f"Job {job_id}: No input source found (no GCS path or file_urls)")
return None
raise DownloadError(
"No input audio source found for this job (no GCS path and no file_urls)."
)
except DownloadError:
raise
except Exception as e:
logger.error(f"Job {job_id}: Failed to download audio: {e}", exc_info=True)
return None
raise DownloadError(f"Failed to download audio from storage: {e}") from e
async def upload_separation_results(
+45 -17
View File
@@ -31,6 +31,7 @@ from pathlib import Path
from backend.models.job import JobStatus
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.audio_search_service import DownloadError
from backend.services.job_health_service import validate_worker_can_run
from backend.services.lyrics_cache_service import LyricsCacheService
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
@@ -302,9 +303,9 @@ async def process_lyrics_transcription(job_id: str) -> bool:
# Download audio file from GCS (waits for audio worker if URL job)
with job_span("download-audio", job_id) as download_span:
job_log.info("Downloading audio file from GCS...")
# download_audio raises DownloadError with a specific reason
# on failure (no silent None → generic-error fallback).
audio_path = await download_audio(job_id, temp_dir, storage, job, job_manager)
if not audio_path:
raise Exception("Failed to download audio file")
job_log.info(f"Audio downloaded: {os.path.basename(audio_path)}")
download_span.set_attribute("audio_file", os.path.basename(audio_path))
@@ -552,6 +553,17 @@ async def process_lyrics_transcription(job_id: str) -> bool:
logger.info(f"[job:{job_id}] WORKER_END worker=lyrics status=success duration={duration:.1f}s")
return True
except DownloadError as e:
duration = time.time() - start_time
job_log.error(f"Audio download failed: {str(e)}", exc_info=True)
logger.error(f"[job:{job_id}] WORKER_END worker=lyrics status=error duration={duration:.1f}s error={e}")
job_manager.mark_job_failed(
job_id=job_id,
error_message=f"Audio download failed: {str(e)}",
error_details={"stage": "download_audio", "error": str(e)}
)
return False
except Exception as e:
duration = time.time() - start_time
job_log.error(f"Lyrics transcription failed: {str(e)}", exc_info=True)
@@ -587,13 +599,14 @@ async def download_audio(
job,
job_manager: JobManager,
max_wait_seconds: int = 300
) -> Optional[str]:
) -> str:
"""
Download audio file from GCS to local temp directory.
For URL jobs, the audio worker downloads from YouTube first and uploads to GCS.
We wait for the input_media_gcs_path to be set by the audio worker.
For URL jobs, the audio worker fetches the source via the flacfetch service
first and uploads it to GCS. We wait for the input_media_gcs_path to be set
by the audio worker.
Args:
job_id: Job ID
temp_dir: Temporary directory for download
@@ -601,9 +614,14 @@ async def download_audio(
job: Job object
job_manager: JobManager instance
max_wait_seconds: Maximum time to wait for audio worker to download
Returns:
Path to downloaded audio file, or None if failed
Path to the downloaded audio file.
Raises:
DownloadError: with a specific reason if the audio can't be obtained
(fail loudly with the real cause instead of returning None and
surfacing a generic "Failed to download audio file").
"""
import time
@@ -636,20 +654,30 @@ async def download_audio(
# Check if audio worker failed
if updated_job and updated_job.status == JobStatus.FAILED:
logger.error(f"Job {job_id}: Audio worker failed, cannot proceed with lyrics")
return None
raise DownloadError(
"Audio worker failed before lyrics could run; cannot "
"proceed without the downloaded/separated audio."
)
# Wait before next poll - use asyncio.sleep to not block event loop
await asyncio.sleep(poll_interval)
logger.error(f"Job {job_id}: Timed out waiting for audio download (waited {max_wait_seconds}s)")
return None
raise DownloadError(
f"Timed out after {max_wait_seconds}s waiting for the audio worker "
"to download the input audio."
)
logger.error(f"Job {job_id}: No input_media_gcs_path found and no URL")
return None
raise DownloadError(
"No input audio available (no input_media_gcs_path and no URL)."
)
except DownloadError:
raise
except Exception as e:
logger.error(f"Job {job_id}: Failed to download audio: {e}", exc_info=True)
return None
raise DownloadError(f"Failed to download audio for lyrics: {e}") from e
async def upload_lyrics_results(
+45
View File
@@ -6,6 +6,51 @@ Key insights for future AI agents working on this codebase.
---
## Don't Invent Silent Fallbacks — Ask or Fail Loud (Jun 2026)
When the desired behaviour for an edge case is unclear, **do not** invent a quiet
fallback that lets the system keep going while doing the wrong thing. Either ask
the human, or fail loudly with a clear, specific error. Silent fallbacks hide
bugs: the system "works" while producing wrong output, and the real failure
surfaces much later (or never).
**Motivating incident (2026-06-08):** a Facebook-URL job failed with a generic
"Failed to download audio file". A non-YouTube path had silently fallen back to
running yt-dlp *inside the Cloud Run container* (flacfetch was supposed to be the
sole downloader), which then died in a broken `convert_to_wav`. The download had
actually succeeded — the fallback was both unintended and broken. The fix
**deleted** the fallback rather than patching it.
A full audit of the codebase's fallback paths followed — see
`docs/archive/2026-06-09-fallback-audit-plan.md` for the catalogue and the
per-item decisions.
**Rules of thumb:**
- A function that can't produce its result should **raise a specific exception**,
not `return None` and let a caller surface a generic message. (See
`download_audio()` in `backend/workers/audio_worker.py` / `lyrics_worker.py`:
each failure mode now raises `DownloadError` with the real reason.)
- "If X isn't configured, silently do Y instead" is the most dangerous shape —
especially around download, payment, job-state, and worker-triggering. Prefer
failing closed and visibly. (See `YouTubeDownloadService.download` — no
flacfetch → raises, no local yt-dlp.)
- Don't trust a comment that *says* a path is a safe fallback — verify what the
user actually observes when it triggers.
- Comments/docstrings must not lie about fail-open vs fail-closed. The credit
evaluation service is **fail-closed** (`pending_review` on error); its
docstrings used to claim fail-open, which would have tempted a future
maintainer to "fix" the code into a free-credit-abuse hole.
- A safe-but-silent fallback (e.g. empty `ADMIN_TOKENS` → everything 403s) should
still log at ERROR so the error monitor alerts — a quiet warning reads as an
auth bug, not a config error.
Good loud-failure models already in the tree to copy:
`lyrics_transcriber/output/video.py` (raises on a missing background image) and
`backend/services/visibility_change_service.py` (checks the trigger result,
restores state, and raises rather than silently dropping a worker trigger).
---
## Multi-instance-type encode pool: one shared speed-rank, availability-aware (Aug 2026, v0.195.0)
"c4d primary + a single n2 fallback family" is not resilient — a stockout is per
@@ -0,0 +1,237 @@
# Fallback Logic Audit — karaoke-gen (2026-06-09)
**Driver prompt:** `docs/archive/2026-06-08-fallback-logic-audit-prompt.md`
**Motivating incident:** 2026-06-08 Facebook-URL job → generic "Failed to download
audio file". A non-YouTube path silently fell back to running yt-dlp *inside the
Cloud Run container*, which died in a broken `convert_to_wav`. flacfetch was
supposed to be the sole downloader. The fix **deleted** the fallback rather than
patching it. See `docs/archive/2026-06-08-flacfetch-sole-downloader-plan.md` and
memory `project_flacfetch_sole_downloader`.
**Goal of this audit:** make every silent fallback an *explicit, intentional*
decision — fail loudly with a clear error, OR keep + log/alert, OR something
else. Do **not** change behaviour without Andrew's answer. Some fallbacks are
load-bearing graceful degradation and should stay.
**Classification:**
- **A** = legitimate graceful degradation (intended, fails safe + visible). Keep.
- **B** = silent fallback masking a bug/misconfig (like the yt-dlp case). Delete or fail-fast.
- **C** = ambiguous — needs Andrew's intent.
**Method:** 5 parallel read-only triage agents over (1) audio/lyrics acquisition,
(2) job-state & worker triggering, (3) payment/credits/referral/auth, (4)
karaoke_gen core render, (5) backend routes/services/config. Findings below are
verified against actual behaviour, not comments.
> **DECISION fields capture Andrew's answers verbatim** (interview held 2026-06-09).
---
## THEME 1 — Source downgrade: flacfetch → YouTube-only (the motivating class)
These reconstruct the exact "silently use a worse source" pattern that caused the
incident. flacfetch is supposed to be the sole/lossless downloader.
| # | file:line | Class | What happens when it triggers |
|---|-----------|-------|-------------------------------|
| 1.1 | `backend/services/audio_search_service.py:224-230` | **B** | Remote flacfetch search error → silently re-runs **local** flacfetch (YouTube-only on Cloud Run). Transient remote outage silently downgrades a RED/OPS lossless job to a YouTube rip; user sees only a `logger.warning`. |
| 1.2 | `karaoke_gen/audio_fetcher.py:405-433` | **B** | If `TorrentDownloader` import fails / Transmission down / tracker `*_API_URL` unset → RED/OPS skipped, only `YoutubeProvider` registered. The concrete mechanism behind 1.1. |
| 1.3 | `karaoke_gen/audio_fetcher.py:1818-1823` | **C** | If exactly one of `FLACFETCH_API_URL` / `FLACFETCH_API_KEY` is set → logs "falling back to local mode". A typo'd secret silently disables the remote service. |
| 1.4 | `karaoke_gen/audio_fetcher.py:558-577` | **C** | `select_best` returns result[0] on any ranking failure → may silently auto-pick a lower-quality source. |
**DECISION (Theme 1):** Andrew — **"Fail loudly, no YouTube fallback."** flacfetch
is the sole/lossless downloader; if it can't serve (remote error, RED/OPS
unavailable, half-configured), fail the job with a clear error rather than
silently ripping YouTube. Applies to 1.1–1.4.
---
## THEME 2 — Silent worker-trigger drops → jobs stall forever
A recurring pattern: a downstream worker is triggered fire-and-forget and the
boolean/return is discarded. If the enqueue fails, the job sits in an intermediate
state with **no error and nothing to recover it**. The user is often told "success".
| # | file:line | Class | What happens when it triggers |
|---|-----------|-------|-------------------------------|
| 2.1 | `backend/services/job_manager.py:1126-1137` | **B** | `_trigger_screens_worker` catches `RuntimeError` (no event loop) and just logs `# TODO: use message queue`. Called from `mark_lyrics_complete` in the **lyrics_worker Cloud Run Job** (sync). If it no-ops, `lyrics_complete=True` is written but screens worker never fires → job stalls before AWAITING_REVIEW; `recover_stuck_jobs` doesn't cover this. *(Verify: does a loop exist in that Job?)* |
| 2.2 | `backend/api/routes/review.py:504` | **B** | After REVIEW_COMPLETE, `asyncio.create_task(trigger_render_video_worker)` result discarded; endpoint returns `{"status":"success"}`. Failed enqueue → stuck at REVIEW_COMPLETE, no error, no auto-recovery. |
| 2.3 | `backend/workers/render_video_worker.py:281,546` | **B** | `await trigger_video_worker(job_id)` bool ignored. Failed trigger → job stuck at INSTRUMENTAL_SELECTED with render done but final encode never started. |
| 2.4 | `backend/api/routes/internal.py:725-728` | **C** | Retry scheduler resets parked job to REVIEW_COMPLETE then triggers; on raise the comment assumes it'll be re-picked as RENDER_PENDING_CAPACITY, but the next tick queries `status==RENDER_PENDING_CAPACITY` and won't see it. Escapes both retry query and 24h timeout sweep → stuck indefinitely. |
| 2.5 | `backend/services/encoding_service.py:1018-1020` | **B** | `get_encoding_service()` does `try: set_worker_manager(manager) except Exception: pass`. Any client-init failure → service silently runs with **no worker manager**, falls back to static URL, loses multi-zone capacity failover, no signal. |
| 2.6 | `audio_download_worker.py:291-294`, `job_manager.py:1078-1081`, `jobs.py:70-71`, `review.py:2208-2209` | **B** | `asyncio.gather(trigger_audio, trigger_lyrics)` results discarded across these sites. One failed enqueue → that pipeline branch silently never starts while the other proceeds; job hangs. |
**Model to emulate:** `backend/services/visibility_change_service.py:194-205` does it
right — checks `triggered`, logs error, restores state, raises.
**DECISION (Theme 2):** Andrew — **"Fail loud / re-park & retry."** Check every
trigger result; on failure mark the job failed or re-queue (the
`visibility_change_service` pattern) so a job can never silently stall. Applies
to 2.1–2.6. _(Still verify reachability of 2.1's no-event-loop path during impl,
but the directional call is fail-loud/retry.)_
---
## THEME 3 — Lyrics/transcription silently degraded or missing
| # | file:line | Class | What happens when it triggers |
|---|-----------|-------|-------------------------------|
| 3.1 | `karaoke_gen/lyrics_transcriber/core/controller.py:487-491` | **B** | If every transcriber returns empty (no exception, e.g. AudioShake 200 with no words), code logs a warning and proceeds; `process()` skips output but job is still reported "completed successfully". → karaoke video with **no synced lyrics**, no error. |
| 3.2 | `karaoke_gen/lyrics_processor.py:372-373` + `core/config.py:16` | **B/C** | `enable_local_whisper=True` hardcoded as priority-3 fallback. On a host with `whisper_timestamped` installed, AudioShake failure silently yields much lower-quality Whisper timing with no signal. (No-op on Cloud Run where pkg absent — env-dependent.) |
| 3.3 | `core/controller.py:396-398` | **C** | Failure loading existing corrections JSON is swallowed → silently re-transcribes from scratch, discarding prior **human review** work. |
| 3.4 | `core/controller.py:442-454` | **C** | Per-provider reference-lyric fetch errors (Genius/Spotify/LRCLIB/Musixmatch) swallowed; all-fail → "No lyrics found", correction runs transcription-only (lower accuracy). Masks systemic outages (e.g. expired Spotify cookie). |
**DECISION (Theme 3):** Andrew — **"Fail the job loudly."** No transcription
results = fail with a clear error; lyrics are the core deliverable and a
lyric-less "success" must never ship. Treat 3.1 as a hard fail. _(Apply the same
fail-loud principle to 3.2–3.4 during impl — silent low-quality Whisper and
silently discarding human-review corrections both violate the principle; flag if
any needs a different call.)_
---
## THEME 4 — Wrong font / asset / color on a paid deliverable
Silent substitution of a missing branded asset → a wrong-looking video/CDG the
customer paid for. Note the karaoke-render path at
`lyrics_transcriber/output/video.py:43-44` already **raises** on a missing
background — a good loud-failure model the others diverge from.
| # | file:line | Class | What happens when it triggers |
|---|-----------|-------|-------------------------------|
| 4.1 | `lyrics_transcriber/output/video.py:417-423` | **B** | Final karaoke MP4: if theme font file missing, `:fontsdir` is silently dropped → libass resolves via system fontconfig → whole song in a substitute font, no log above debug. |
| 4.2 | `video_generator.py:120-132,153,174,191` | **B** | Title/end screens: missing font → `ImageFont.load_default()` (~10px bitmap). 4K title/end card renders artist/title in a near-invisible default font. |
| 4.3 | `video_generator.py:368-377` | **B/C** | Title/end screen: missing `background_image` → silently uses flat `background_color` (karaoke path raises instead — inconsistent). |
| 4.4 | `lyrics_transcriber/output/cdg.py:294-306` | **B/C** | CDG: unresolved theme font → bundled `arial.ttf` (warning only). Hardware disc renders all lyrics in Arial. |
| 4.5 | `karaoke_finalise.py:1132-1159` | **B/C** | Duet job: missing/malformed `duet_corrections_json` → renders CDG via LRC path = **solo colors**, loses singer-color distinction (warning only). |
| 4.6 | `lyrics_transcriber/output/cdg.py:551-561` | **B/C** | Duet CDG: lost/length-mismatched line→singer map → defaults every line to singer 1 "to keep the file valid". Silently wrong color attribution. |
| 4.7 | `lyrics_transcriber/output/ass/config.py:115-117,149-151` | **B/C** | Malformed theme color → hardcoded default blue/black. A typo'd theme color renders lyrics in the wrong brand color. |
| 4.8 | `lyrics_transcriber/output/ass/lyrics_line.py:39-45` | **B/C** | Text-measurement font → `load_default()` (different metrics) while libass renders a different font → line-wrap decisions desync from the actual video. Pairs with 4.1. |
**A (keep, verified intended + logged):** GPU→CPU encode fallback
(`karaoke_finalise.py:1803-1881`, visually equivalent libx264, warned);
`style_loader.py:164-192` documented default duet palette;
`subtitles.py:100-107` audio-duration estimate (logged).
**DECISION (Theme 4):** Andrew — **"Fail loudly on missing branded asset."** Match
`video.py:43-44` which already raises. A paid deliverable shouldn't silently
substitute fonts/backgrounds/colors — fail with a clear error so it's caught
before delivery. Applies to 4.1–4.8. _(Keep the confirmed-A items: GPU→CPU
encode, documented duet palette, audio-duration estimate.)_
---
## THEME 5 — Pricing undercharge / ceiling bypass
| # | file:line | Class | What happens when it triggers |
|---|-----------|-------|-------------------------------|
| 5.1 | `backend/api/routes/jobs.py:128-134` + `services/pricing.py:13-22` | **B** | Client-supplied `duration_seconds`; if `None`, `duration_to_credits(None)=1` and `is_blocked(None)=False`. Omitting duration → any-length job for 1 credit **and** skips the 60-min hard ceiling. `acknowledged_credits` guard only fires if client *sends* a mismatch. |
| 5.2 | `backend/api/routes/jobs.py:2446-2466` (`create_job_from_search`) | **B** | Guided/search flow never passes `credits=` → always `JobCreate.credits=1` regardless of length. Inconsistent with URL path; long-input undercharge. |
| 5.3 | `backend/api/routes/users.py:97` | **B** | Referral discount: `except Exception: discount_percent = 10` on Firestore read failure. A 20% link charged at 10% on a transient error. |
**DECISION (Theme 5):** Andrew — **"Keep defaults but always enforce ceiling +
alert."** Keep the 1-credit default, but the 60-min hard ceiling must always be
enforced (never silently bypassed when duration is omitted), and any
undercharge / discount-guess must raise a Discord alert so it's visible.
_Impl note:_ when duration is unknown we still alert + apply the ceiling rather
than reject; charge-by-duration consistency between URL and search flows to be
handled under this (alert if they diverge). Applies to 5.1–5.3.
---
## THEME 6 — Lost paid orders / "Unknown" customer data
| # | file:line | Class | What happens when it triggers |
|---|-----------|-------|-------------------------------|
| 6.1 | `backend/api/routes/users.py:1105-1135` (`_handle_made_for_you_order`) | **B** | Broad `except Exception` swallows fulfilment failure *after Stripe charged*; emails admin "[FAILED]" but does **not** re-raise → webhook returns 200 (no Stripe retry). If the email also fails (line 1134 logs only), a paid $50 order silently vanishes. |
| 6.2 | `backend/api/routes/users.py:858-859` | **B** | Made-for-you order: `metadata.get("artist","Unknown Artist")`/`("title","Unknown Title")` → a paid video titled "Unknown Artist - Unknown Title" instead of rejecting the order. |
**DECISION (Theme 6):** Andrew — **"Re-raise + durable failed-order record."** On
made-for-you fulfilment failure, re-raise so Stripe retries AND/OR write a
durable `needs_manual_fulfilment` record (don't rely on an email an admin might
miss); reject orders missing artist/title rather than shipping "Unknown Artist -
Unknown Title". Applies to 6.1–6.2.
---
## THEME 7 — Env-var / config defaults masking prod misconfiguration
A single unset/rotated env var silently produces wrong behaviour with only a
warning. Env vars are set in Cloud Run today, but the defaults are landmines.
| # | file:line | Class | What happens when it triggers |
|---|-----------|-------|-------------------------------|
| 7.1 | `backend/services/dropbox_service.py:34` | **B** | `GOOGLE_CLOUD_PROJECT` default `"karaoke-gen"` (real project is `nomadkaraoke`). If unset → secret lookup targets a non-existent project; bare `except` swallows → Dropbox distribution silently disabled. |
| 7.2 | `backend/config.py:19-21` | **B** | GCS bucket defaults `karaoke-gen-storage[/-temp/-outputs]` ≠ real `karaoke-gen-storage-nomadkaraoke`. If `GCS_BUCKET_NAME` unset → reads/writes a non-existent bucket, runtime 500s deep in processing instead of boot failure. |
| 7.3 | `backend/services/worker_service.py:160-166` | **B** | If `CLOUD_RUN_SERVICE_URL` unset → worker base URL = `http://localhost:{PORT}` with no env guard. Workers invoked against localhost → jobs created but silently never run. |
| 7.4 | `backend/middleware/tenant.py:30-31,102-105` | **B (security)** | `IS_PRODUCTION` from `ENV/ENVIRONMENT=="production"`, but app default is `"development"`. If prod doesn't explicitly set it → `?tenant=<id>` query override active → **tenant spoofing** (impersonate white-label portal). *(Verify prod sets ENVIRONMENT=production.)* |
| 7.5 | `backend/services/email_service.py:230-245` | **B** | No `POSTMARK_SERVER_TOKEN` → silent `ConsoleEmailProvider`; all sends "succeed". In prod with token unset/rotated, magic-link/completion/receipt emails silently vanish → users can't log in or get downloads. |
**DECISION (Theme 7):** Andrew — **"Fail-fast at startup in prod."** When
`ENVIRONMENT=production`, require these vars and refuse to boot (or go
health-check red) if missing; make the tenant-detection gate production-safe by
default (disable `?tenant=` override unless explicitly development). No silent
wrong-resource defaults. Applies to 7.1–7.5. _Impl note:_ first confirm prod
actually sets `ENVIRONMENT=production` (Pulumi `cloud_run.py`) — the security
gate 7.4 depends on it.
---
## THEME 8 — Lower-risk cleanups (docs lie, opaque errors, auth layers)
| # | file:line | Class | What happens when it triggers |
|---|-----------|-------|-------------------------------|
| 8.1 | `backend/config.py:77-78` & `credit_evaluation_service.py` module/`grant_welcome_credits` docstrings | **C/doc** | Docstrings say "Fail-open: credits granted on error" but code is **fail-CLOSED** (`pending_review`, no grant) — the *safe* behaviour. The lie risks a future maintainer "fixing code to match docs" and re-introducing free-credit abuse. Fix the docs, confirm policy. |
| 8.2 | `youtube_download_service.py:11,37-38`; `lyrics_worker.py:566` | **doc** | Stale docstrings still describe the deleted yt-dlp fallback ("falls back to local yt_dlp"). Misleading post-incident. |
| 8.3 | `backend/workers/audio_worker.py:467-473`, `lyrics_worker.py:611-623` → `audio_worker.py:217-218` | **C** | Distinct download failure causes all collapse to `return None` → generic user-facing "Failed to download audio file" (the opaque message from the incident). Surface specific reasons to the user? |
| 8.4 | `backend/api/routes/users.py:734-762` (`create_checkout`) | **C** | Unauthenticated; email + referral-discount taken from request body. Anyone can compute a discount against an arbitrary email. Direct money risk low (webhook keys off session metadata). Intended pre-login flow? |
| 8.5 | `backend/services/auth_service.py:107-111` | **C** | OIDC verified with `audience=None` (audience check skipped); mitigated by strict `token_email == scheduler SA`. Acceptable or tighten to URL allowlist? |
| 8.6 | `backend/services/auth_service.py:81-90` | **C** | Empty `ADMIN_TOKENS` → warning only; all admin endpoints 403 (fails closed = safe, but looks like an auth bug not a config bug). Louder signal in prod? |
| 8.7 | `backend/services/email_validation_service.py:226` | **C** | Missing Firestore blocklist doc → `DEFAULT_DISPOSABLE_DOMAINS` (the Firestore `.get` itself is unwrapped, so hard failures propagate — good). Acceptable degradation? |
**DECISION (Theme 8):** Andrew — **"Fix docs + specific errors + admin-token
signal."** Correct the lying/stale docstrings (8.1 fail-open→fail-closed, 8.2
yt-dlp), surface specific download-failure reasons to the user instead of the
generic "Failed to download audio file" (8.3), and add a loud prod signal when
`ADMIN_TOKENS` is empty (8.6). Leave 8.4 (unauth checkout), 8.5 (OIDC
audience=None), 8.7 (blocklist default) as acceptable fail-safe.
---
## A — Legitimate graceful degradation confirmed (no change; listed for the record)
- `audio_transcoding_service.py:113-130,171-175` — review-audio transcode fail → serve original FLAC (logged).
- `backend/workers/audio_worker.py:207-211` — audio separation not configured → **raises** (good model).
- `youtube_download_service.py:118-126` — flacfetch unconfigured → **raises** (the post-incident fix).
- `audio_download_worker.py:341-342` — unsupported source → raises `DownloadError` (post-fix state of the incident).
- `video_worker.py:182-185` — `_encode_via_gce` re-raises on GCE failure (no silent local fallback).
- `local_encoding_service.py:241-264` — GPU→CPU encode, warned. Visible perf degradation.
- `stripe_service.py` (multiple) — unconfigured/invalid → fail closed + visible.
- `user_service.py:1003-1017` — `add_credits` idempotent via `PROCESSED_STRIPE_SESSIONS`.
- `job_manager.py:146-164` — credit-deduction failure deletes job + raises `InsufficientCreditsError`.
- `referral_service.py:283-305` — `record_earning` returns None only for invalid/expired referral; wrapped so it never blocks the buyer's credits.
- `catalog_proxy_service.py:71,103` — autocomplete `[]` on karaoke-decide outage (cosmetic).
- `credential_manager` distribution checks — distinct NOT_CONFIGURED/INVALID/ERROR/VALID, blocking 400 on create-from-url.
- `file_upload.py:1827-1833` (create-from-url) — re-raises, calls `fail_job`, 500 with real detail.
---
## Execution roadmap (Phase 4) — one PR per theme, with tests
Ordered safest/clearest → most delicate. Each PR updates `docs/LESSONS-LEARNED.md`
with the principle ("don't invent silent fallbacks — ask or fail loud") + the
notable example(s) it fixes.
| PR | Theme | Decision in one line | Risk |
|----|-------|----------------------|------|
| 1 | **T8** docs/errors | Fix lying fail-open docstrings + stale yt-dlp docstrings; surface specific download-failure reasons; loud prod signal on empty ADMIN_TOKENS | Low |
| 2 | **T1** sourcing | flacfetch sole downloader → fail loud (no silent YouTube downgrade) across 1.1–1.4 | Low/Med |
| 3 | **T4** render assets | Missing branded font/bg/duet-color → raise (match video.py:43-44) across 4.1–4.8 | Med |
| 4 | **T7** env config | Prod fail-fast on missing required vars; tenant gate production-safe by default | Med (verify Pulumi env first) |
| 5 | **T3** lyrics | Empty transcription = fail loud; same principle to whisper/corrections/reference | Med |
| 6 | **T2** worker triggers | Check every trigger result → fail/re-park; never silently stall (2.1–2.6) | High (job state machine) |
| 7 | **T5** pricing | Keep 1-credit default but always enforce 60-min ceiling + alert on undercharge/discount-guess | High (money) |
| 8 | **T6** paid orders | Re-raise + durable needs_manual_fulfilment record; reject missing artist/title | High (money/webhook) |
**Status:** Phases 1–3 complete (discover + triage + interview; all 8 decisions
captured above). Phase 4 (execute) — pending sequencing go-ahead from Andrew.