fix: pass gcs_path for YouTube downloads via remote flacfetch (#320)

## Summary

- Fix YouTube audio downloads failing when remote flacfetch is enabled
- Add dynamic MIME type detection for audio uploads (fixes hardcoded
`audio/flac` for `.webm`/`.opus` files)
- Add documentation example to LESSONS-LEARNED.md

## Problem

When remote flacfetch is enabled and a user selects a YouTube audio
source, the backend was calling `download_by_id()` without the
`gcs_path` parameter. This caused flacfetch to download the file locally
on the VM instead of uploading to GCS. The backend then tried to open
the VM's local path (`/var/lib/transmission-daemon/downloads/...`) which
doesn't exist on Cloud Run, causing the job to fail.

**Production impact:** Job `4c266875` failed with "No such file or
directory"

## Root Cause

This is the same class of bug that was fixed in December 2025 for
RED/OPS torrent sources. However, that fix only addressed the torrent
code path, leaving YouTube downloads broken when remote is enabled.

## Changes

### `backend/api/routes/audio_search.py`
- Add new `elif` branch to handle remote YouTube downloads with
`gcs_path` parameter
- Use `mimetypes.guess_type()` to detect correct content type instead of
hardcoded `audio/flac`

### `backend/tests/test_audio_search.py`
- Add `test_youtube_remote_download_passes_gcs_path` to verify the fix
- Add `test_youtube_local_download_when_remote_disabled` for coverage

### `docs/LESSONS-LEARNED.md`
- Add specific example of the `gcs_path` bug to the "Fix Both Sides of
Dual Code Paths" lesson

## Test plan

- [x] All 50 audio_search tests pass
- [x] New test verifies `gcs_path` is passed to `download_by_id()` for
YouTube sources
- [x] New test verifies local download path when remote is disabled
- [ ] Production verification: retry previously failed job after
deployment

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 4.5 authored and GitHub committed 2026-01-20 18:33:42 -08:00
1 parent b9f1b8b1ba
commit 170acfd508
5 files changed
+314 -55

No files matched your search

@@ -0,0 +1,166 @@
# Plan: Fix YouTube Remote Download Missing GCS Path
**Created:** 2026-01-20
**Branch:** feat/sess-20260120-1756-fix-youtube-remote-download
**Status:** Complete
## Overview
When remote flacfetch is enabled and a user selects a YouTube audio source, the backend calls `download_by_id()` without the `gcs_path` parameter. This causes flacfetch to download the file locally on the VM instead of uploading to GCS. The backend then tries to open the VM's local path, which doesn't exist on Cloud Run, causing the job to fail.
This is the same class of bug that was fixed on 2025-12-22 for RED/OPS torrent sources (see `docs/archive/2025-12-phase1-testing-gaps.md`), but the fix only covered torrent sources, not YouTube.
## Root Cause
In `backend/api/routes/audio_search.py` lines 354-363:
```python
# Use download_by_id if we have source_id and remote is enabled
if source_id and source_name and is_remote_enabled:
logger.info(f"Using download_by_id for local download: {source_name} ID={source_id}")
result = audio_search_service.download_by_id(
source_name=source_name,
source_id=source_id,
output_dir=temp_dir, # <-- This is a local temp dir, but flacfetch runs on VM!
target_file=target_file,
download_url=download_url,
# <-- MISSING: gcs_path parameter
)
```
When `is_remote_enabled=True`, `download_by_id()` routes to the remote flacfetch VM. Without `gcs_path`, the VM downloads locally and returns a local path like `/var/lib/transmission-daemon/downloads/...` which doesn't exist on Cloud Run.
## Requirements
- [x] YouTube downloads with remote flacfetch should upload directly to GCS
- [x] Backend should receive a `gs://` path from flacfetch
- [x] No changes to flacfetch itself (it already supports `upload_to_gcs` and `gcs_destination`)
- [x] Maintain backwards compatibility (local-only deployments should still work)
- [x] Add test coverage for this code path
## Technical Approach
**Option A (chosen):** Pass `gcs_path` to `download_by_id()` for ALL remote downloads, not just torrent sources.
The current code structure has two branches:
1. `if is_torrent_source and is_remote_enabled:` - handles RED/OPS with GCS upload ✅
2. `else:` branch - handles YouTube, but incorrectly assumes "local download" even when remote is enabled
The fix refactors to check `is_remote_enabled` first, then handle torrent vs YouTube within that:
- If remote enabled: always pass `gcs_path` to let flacfetch upload directly
- If remote disabled: download locally and upload to GCS from Cloud Run
## Implementation Steps
1. [x] Refactor the `else` branch in `_download_and_start_processing()` to check for remote-enabled YouTube downloads
2. [x] Pass `gcs_path` when calling `download_by_id()` for remote YouTube downloads
3. [x] Add test case for YouTube with remote enabled to verify `gcs_path` is passed
4. [x] Update logging to clarify the download path being used
5. [x] Bump version in `pyproject.toml`
6. [x] Run tests to verify no regressions
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/api/routes/audio_search.py` | Modify | Add `gcs_path` parameter for remote YouTube downloads |
| `backend/tests/test_audio_search.py` | Modify | Add test case for YouTube remote download with GCS path |
| `pyproject.toml` | Modify | Bump version |
## Detailed Code Changes
### audio_search.py (lines 349-393)
Current structure:
```python
else:
# Local download (YouTube or fallback) <-- WRONG: not always local!
temp_dir = tempfile.mkdtemp(...)
if source_id and source_name and is_remote_enabled:
result = audio_search_service.download_by_id(
...
output_dir=temp_dir, # <-- BUG: unused for remote
# Missing gcs_path!
)
```
Fixed structure:
```python
else:
# Non-torrent download (YouTube or fallback)
if is_remote_enabled and source_id and source_name:
# Remote download via flacfetch VM - upload directly to GCS
gcs_destination = f"uploads/{job_id}/audio/"
logger.info(f"Using remote download_by_id for {source_name} ID={source_id} with GCS upload to: {gcs_destination}")
result = audio_search_service.download_by_id(
source_name=source_name,
source_id=source_id,
output_dir="", # Not used for remote
target_file=target_file,
download_url=download_url,
gcs_path=gcs_destination, # <-- FIX: Add this!
)
# Handle GCS path response (same logic as torrent branch)
if result.filepath.startswith("gs://"):
parts = result.filepath.replace("gs://", "").split("/", 1)
if len(parts) == 2:
audio_gcs_path = parts[1]
else:
audio_gcs_path = result.filepath
filename = os.path.basename(result.filepath)
else:
# Fallback: upload manually
filename = os.path.basename(result.filepath)
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
logger.warning(f"Remote download returned local path: {result.filepath}, uploading manually")
with open(result.filepath, 'rb') as f:
storage_service.upload_fileobj(f, audio_gcs_path, content_type='audio/flac')
logger.info(f"Remote download complete, GCS path: {audio_gcs_path}")
else:
# Local download (remote disabled or no source_id)
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
# ... existing local download logic ...
```
## Testing Strategy
### Unit Tests
Add to `test_audio_search.py` in `TestAudioSearchApiRouteDownload`:
```python
def test_youtube_remote_download_passes_gcs_path(self, mock_job_manager, mock_storage_service):
"""
Test that YouTube downloads via remote flacfetch include GCS path.
This is the same bug pattern as the torrent download bug fixed in 2025-12:
when remote is enabled, we must pass gcs_path so flacfetch uploads to GCS.
"""
# Setup: YouTube source with remote enabled
# Verify: download_by_id called with gcs_path parameter
```
### Manual Testing (post-deploy)
1. Create a job and select a YouTube audio source
2. Verify job completes successfully
3. Check GCS path is properly set in job data
## Open Questions
None - the fix is straightforward and follows the pattern established for torrent sources.
## Rollback Plan
This is a minor code change. If issues occur:
1. Revert the PR
2. The failure mode returns to the current state (YouTube downloads fail when remote is enabled)
## Related Documentation
- `docs/archive/2025-12-phase1-testing-gaps.md` - Original bug fix for torrent sources
- `docs/LESSONS-LEARNED.md` - "Fix Both Sides of Dual Code Paths" pattern
+44 -15
View File
@@ -13,6 +13,7 @@ The flow is:
"""
import asyncio
import logging
import mimetypes
import os
import tempfile
from typing import Optional, List, Dict, Any
@@ -342,26 +343,52 @@ async def _download_and_start_processing(
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
logger.warning(f"Remote download returned local path: {result.filepath}, uploading manually")
content_type, _ = mimetypes.guess_type(result.filepath)
with open(result.filepath, 'rb') as f:
storage_service.upload_fileobj(f, audio_gcs_path, content_type='audio/flac')
storage_service.upload_fileobj(f, audio_gcs_path, content_type=content_type or 'application/octet-stream')
logger.info(f"Remote download complete, GCS path: {audio_gcs_path}")
elif is_remote_enabled and source_id and source_name:
# Non-torrent remote download (e.g., YouTube via flacfetch VM)
# Must pass gcs_path so flacfetch uploads directly to GCS
gcs_destination = f"uploads/{job_id}/audio/"
logger.info(f"Using remote download_by_id for {source_name} ID={source_id} with GCS upload to: {gcs_destination}")
result = audio_search_service.download_by_id(
source_name=source_name,
source_id=source_id,
output_dir="", # Not used for remote
target_file=target_file,
download_url=download_url,
gcs_path=gcs_destination,
)
# Handle GCS path response (same logic as torrent branch)
if result.filepath.startswith("gs://"):
# Extract the path portion after the bucket name
# Format: gs://bucket/uploads/job_id/audio/filename.flac
parts = result.filepath.replace("gs://", "").split("/", 1)
if len(parts) == 2:
audio_gcs_path = parts[1]
else:
audio_gcs_path = result.filepath
filename = os.path.basename(result.filepath)
else:
# Fallback: treat as local path (shouldn't happen for remote)
filename = os.path.basename(result.filepath)
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
logger.warning(f"Remote download returned local path: {result.filepath}, uploading manually")
content_type, _ = mimetypes.guess_type(result.filepath)
with open(result.filepath, 'rb') as f:
storage_service.upload_fileobj(f, audio_gcs_path, content_type=content_type or 'application/octet-stream')
logger.info(f"Remote download complete, GCS path: {audio_gcs_path}")
else:
# Local download (YouTube or fallback)
# Local download (remote disabled or no source_id)
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
# Use download_by_id if we have source_id and remote is enabled
if source_id and source_name and is_remote_enabled:
logger.info(f"Using download_by_id for local download: {source_name} ID={source_id}")
result = audio_search_service.download_by_id(
source_name=source_name,
source_id=source_id,
output_dir=temp_dir,
target_file=target_file,
download_url=download_url,
)
elif source_name == 'YouTube' and download_url:
if source_name == 'YouTube' and download_url:
# For YouTube without remote, download directly using URL from state_data
# This avoids relying on in-memory cache which doesn't persist across instances
logger.info(f"Downloading YouTube audio directly from URL: {download_url}")
@@ -396,11 +423,13 @@ async def _download_and_start_processing(
filename = os.path.basename(result.filepath)
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
# Detect content type from file extension (YouTube downloads are typically .webm/.opus, torrent downloads are .flac)
content_type, _ = mimetypes.guess_type(result.filepath)
with open(result.filepath, 'rb') as f:
storage_service.upload_fileobj(
f,
audio_gcs_path,
content_type='audio/flac' # flacfetch typically returns FLAC
content_type=content_type or 'application/octet-stream'
)
logger.info(f"Uploaded audio to GCS: {audio_gcs_path}")
+101 -39
View File
@@ -1219,67 +1219,129 @@ class TestAudioSearchApiRouteDownload:
assert 'gcs_path' in call_kwargs
assert call_kwargs['gcs_path'] == "uploads/job123/audio/"
def test_local_youtube_download_does_not_pass_gcs_path(self, mock_job_manager, mock_storage_service):
"""Test that YouTube downloads don't use gcs_path (download locally, upload manually)."""
def test_youtube_remote_download_passes_gcs_path(self, mock_job_manager, mock_storage_service):
"""
Test that YouTube downloads via remote flacfetch include GCS path.
This is the same bug pattern as the torrent download bug fixed in 2025-12:
when remote is enabled, we must pass gcs_path so flacfetch uploads to GCS.
Without gcs_path, flacfetch downloads locally on the VM and returns a local
path that Cloud Run can't access.
"""
from backend.api.routes.audio_search import _download_and_start_processing
import asyncio
# Setup mock job with YouTube search result that has source_id
mock_job = Mock()
mock_job.state_data = {
'audio_search_results': [{
'title': 'Demons',
'artist': 'Kenny Chesney',
'provider': 'YouTube',
'quality': 'Opus 128kbps',
'source_id': 'qmTjyGH9ro4', # YouTube video ID
'url': 'https://www.youtube.com/watch?v=qmTjyGH9ro4',
}]
}
mock_job.audio_search_artist = 'Kenny Chesney'
mock_job.audio_search_title = 'Demons'
mock_job_manager.get_job.return_value = mock_job
# Create mock audio search service with remote enabled
mock_audio_service = Mock()
mock_audio_service.is_remote_enabled.return_value = True
# Mock download_by_id to return GCS path (as remote flacfetch should)
mock_download_result = Mock()
mock_download_result.filepath = "gs://bucket/uploads/job456/audio/Kenny Chesney - Demons.webm"
mock_audio_service.download_by_id.return_value = mock_download_result
# Create mock background tasks
mock_bg_tasks = Mock()
# Run the async function
loop = asyncio.new_event_loop()
try:
result = loop.run_until_complete(
_download_and_start_processing(
job_id="job456",
selection_index=0,
audio_search_service=mock_audio_service,
background_tasks=mock_bg_tasks,
)
)
finally:
loop.close()
# CRITICAL: Verify download_by_id was called with gcs_path for remote YouTube source
mock_audio_service.download_by_id.assert_called_once()
call_kwargs = mock_audio_service.download_by_id.call_args.kwargs
# The gcs_path should be set for remote YouTube downloads
assert 'gcs_path' in call_kwargs
assert call_kwargs['gcs_path'] == "uploads/job456/audio/"
# Storage service should NOT be called to upload (flacfetch already uploaded)
mock_storage_service.upload_fileobj.assert_not_called()
def test_youtube_local_download_when_remote_disabled(self, mock_job_manager, mock_storage_service):
"""Test that YouTube downloads locally when remote is disabled."""
from backend.api.routes.audio_search import _download_and_start_processing
import asyncio
import tempfile
# Setup mock job with YouTube search result
mock_job = Mock()
mock_job.state_data = {
'audio_search_results': [{
'title': 'Unwanted',
'artist': 'Avril Lavigne',
'provider': 'YouTube', # NOT a torrent source
'provider': 'YouTube',
'quality': 'Opus 128kbps',
'url': 'https://www.youtube.com/watch?v=abc123',
}]
}
mock_job.audio_search_artist = 'Avril Lavigne'
mock_job.audio_search_title = 'Unwanted'
mock_job_manager.get_job.return_value = mock_job
# Create mock audio search service with remote client (but YouTube doesn't use it)
# Create mock audio search service with remote DISABLED
mock_audio_service = Mock()
mock_audio_service.is_remote_enabled.return_value = True
mock_audio_service.is_remote_enabled.return_value = False
# Create a temp file to simulate downloaded file
temp_file = tempfile.NamedTemporaryFile(suffix='.opus', delete=False)
temp_file.write(b'fake audio data')
temp_file.close()
try:
# Mock download to return local path
mock_download_result = Mock()
mock_download_result.filepath = temp_file.name
mock_audio_service.download.return_value = mock_download_result
# Create mock background tasks
mock_bg_tasks = Mock()
# Run the async function
loop = asyncio.new_event_loop()
try:
result = loop.run_until_complete(
_download_and_start_processing(
job_id="job456",
selection_index=0,
audio_search_service=mock_audio_service,
background_tasks=mock_bg_tasks,
# Mock download_from_url to return temp path
# Note: download_from_url is imported inside the function, so patch at source
with patch('backend.workers.audio_worker.download_from_url') as mock_download_from_url:
mock_download_from_url.return_value = temp_file.name
# Create mock background tasks
mock_bg_tasks = Mock()
# Run the async function
loop = asyncio.new_event_loop()
try:
result = loop.run_until_complete(
_download_and_start_processing(
job_id="job789",
selection_index=0,
audio_search_service=mock_audio_service,
background_tasks=mock_bg_tasks,
)
)
)
finally:
loop.close()
# YouTube should NOT have gcs_path in kwargs
mock_audio_service.download.assert_called_once()
call_kwargs = mock_audio_service.download.call_args.kwargs
# For YouTube, gcs_path should NOT be set (or should be None)
assert call_kwargs.get('gcs_path') is None
# Instead, storage service should be called to upload
mock_storage_service.upload_fileobj.assert_called_once()
finally:
loop.close()
# download_from_url should be called for local YouTube download
mock_download_from_url.assert_called_once()
# Storage service SHOULD be called to upload (local download)
mock_storage_service.upload_fileobj.assert_called_once()
finally:
import os
try:
+2
View File
@@ -42,6 +42,8 @@ When multiple code paths create jobs (file upload, audio search, webhooks), use
### Fix Both Sides of Dual Code Paths
When fixing a bug in a system with multiple code paths (e.g., legacy vs orchestrator, local vs cloud), verify ALL paths are fixed. PR #271 fixed the GCE worker to READ `instrumental_selection` but only checked the legacy path which was already SENDING it. The orchestrator path (production default) wasn't sending it. **Pattern**: If a component receives config from multiple callers, check ALL callers when fixing the receiving side. Write integration tests that cover each path.
**Example:** The `gcs_path` parameter bug for remote flacfetch downloads was fixed for RED/OPS torrent sources in December 2025, but the same bug existed for YouTube sources. The fix only addressed one branch of the conditional, leaving YouTube downloads broken when remote was enabled. Always search for ALL code paths that might need the same fix.
### Defense in Depth
Enforce critical requirements at multiple layers (e.g., reject at creation in JobManager + safety net at processing time).
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.108.8"
version = "0.108.9"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"