Files
karaoke-gen/.claude/plans/flacfetch-retry-logic.plan.md
Andrew Beveridge ☄️andClaude Sonnet 4.5 716936c8a9 Add retry logic for flacfetch download failures and audio search recovery (#382)
## Summary

Adds comprehensive retry logic to handle transient flacfetch network
failures, preventing permanent job failures from temporary service
issues. Includes state persistence for retry capability, enhanced retry
endpoint for audio search jobs, and user-facing error feedback.

## Key Changes

**Retry Logic (backend/services/flacfetch_client.py):**
- Manual exponential backoff: 9 attempts over ~6 minutes (10s → 20s →
40s → 60s capped)
- Selective retry: retries on connection errors and 5xx, immediate fail
on 4xx
- Runtime-configurable via environment variables
(FLACFETCH_RETRY_MAX_ATTEMPTS, etc.)

**State Persistence (backend/api/routes/audio_search.py):**
- Save download parameters (audio_source_type, source_name, source_id,
target_file, download_url) BEFORE attempting download
- Enables retry even if download fails completely
- Added 5 new fields to Job model (backend/models/job.py)

**Enhanced Retry Endpoint (backend/api/routes/jobs.py):**
- Added ~150 lines handling audio search retry cases
- Re-downloads using saved parameters for YouTube, RED, OPS, Spotify
sources
- Clears error state and transitions to DOWNLOADING before triggering
workers

**User Feedback (frontend/components/job/JobActions.tsx):**
- Error toast shows actual API error message on retry failure
- Success toast confirms retry started
- No more silent failures

**Configuration (backend/config.py):**
- Added retry config: max_attempts (9), min_wait (10s), max_wait (60s)

## Testing

- [x] Added 5 comprehensive retry tests in
`backend/tests/test_flacfetch_client.py`
  - Network error retry with exponential backoff
  - No retry on 4xx client errors
  - Retry on 5xx server errors
  - Max retries exhausted
  - Wrapper consistency
- [x] All backend tests pass: 103 passed
- [x] Frontend E2E tests: 142 passed (21 pre-existing failures unrelated
to changes)

## Review

- [x] CodeRabbit CLI review completed locally
- [x] Fixed unused import identified in review
- [x] Documentation updated (README.md Recent Changes,
LESSONS-LEARNED.md retry patterns)
- [x] Implementation plan archived to
docs/archive/2026-02-10-flacfetch-retry-logic.md

## Root Cause

Job 749141f8 failed because flacfetch network call failed during audio
search, and no source info was saved, making retry impossible. This PR
fixes all four related issues identified during investigation.

## Version

Bumped 0.115.5 → 0.116.0 (minor version for new feature)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-10 14:21:14 -05:00

15 KiB

Plan: Add Retry Logic for Flacfetch Download Failures

Created: 2026-02-10 Branch: feat/sess-20260210-1158-investigate-job-749141f8 Status: Draft

Overview

Job 749141f8 failed with error: "Audio download failed: Remote download by ID failed: Download by ID request failed:"

Investigation revealed multiple related issues when flacfetch network calls fail:

  1. No retry logic - A single network hiccup causes immediate job failure
  2. Missing source info - When download fails, source_id/source_name aren't saved to job
  3. Broken retry button - Can't retry because no source info was persisted
  4. Silent UI failures - Frontend doesn't show retry error messages to user

Root Cause

Audio search flow saves selection info to state_data, but doesn't save download parameters (source_id, source_name, etc.) to the job document until AFTER download succeeds. If the flacfetch call fails (network error), the job fails with no way to retry.

Solution

This plan fixes all four issues:

  • Add retry logic to flacfetch client (network resilience)
  • Persist download info before attempting download (enables retry)
  • Update retry endpoint to handle audio search jobs
  • Show error messages in frontend when retry fails

Requirements

Functional Requirements - Flacfetch Retry Logic

  • Retry download_by_id() on httpx.RequestError (network failures)
  • Use exponential backoff spread over ~5 minutes
  • Configurable max retries via environment variable
  • Log each retry attempt with context
  • Do NOT retry on HTTP 4xx errors (client errors - immediate failure)
  • Do retry on HTTP 5xx errors (server errors - transient)

Functional Requirements - Persist Download Info

  • Save source_id, source_name, target_file, download_url to job document
  • Save BEFORE attempting download (not after)
  • Store in top-level fields for easy retry access
  • Preserve existing state_data fields (selected_audio_index, etc.)

Functional Requirements - Retry Endpoint

  • Add case for audio search jobs in /jobs/{id}/retry endpoint
  • Check for saved download params (source_id, source_name)
  • Re-download using saved params
  • Clear error state before retrying
  • Trigger appropriate workers after successful download

Functional Requirements - Frontend Error Handling

  • Show toast notification when retry fails
  • Display actual error message from API
  • Don't just console.error - make it visible to user

Non-Functional Requirements

  • Minimal code changes (use existing patterns)
  • Clear error messages that include retry context
  • Backwards compatible (no breaking changes)
  • Should not significantly increase total timeout duration

Technical Approach

Architecture Decisions

Option 1: Tenacity library (Recommended)

  • ✅ Industry standard, well-tested
  • ✅ Declarative retry configuration
  • ✅ Built-in exponential backoff
  • ✅ Already used in project (check poetry.lock)
  • ❌ New dependency if not already present

Option 2: Manual retry loop

  • ✅ No new dependencies
  • ✅ Full control over logic
  • ❌ More code to maintain
  • ❌ Easy to get wrong (edge cases)

Decision: Use tenacity if already in dependencies, otherwise implement manual retry loop.

Retry Strategy

# Retry on network errors only (RequestError)
# Do NOT retry on:
#   - HTTP 4xx (client errors like 400, 401, 404)
#   - Already completed downloads
# Do retry on:
#   - httpx.RequestError (connection timeout, refused, DNS failure)
#   - HTTP 5xx (server errors - service temporarily down)

Max retries: 8 (total 9 attempts)
Backoff: Exponential with cap, multiplier=2, min=10s, max=60s
  - Attempt 1: immediate
  - Attempt 2: wait 10s
  - Attempt 3: wait 20s
  - Attempt 4: wait 40s
  - Attempt 5: wait 60s (capped)
  - Attempt 6: wait 60s (capped)
  - Attempt 7: wait 60s (capped)
  - Attempt 8: wait 60s (capped)
  - Attempt 9: wait 60s (capped)
Total max time: ~370s (6+ minutes) of backoff + request timeouts

This gives the flacfetch service time to recover from restarts,
network issues, or temporary VM problems.

Configuration

Add to backend/config.py:

# Flacfetch client retry settings
flacfetch_retry_max_attempts: int = int(os.getenv("FLACFETCH_RETRY_MAX_ATTEMPTS", "9"))  # Total attempts (1 initial + 8 retries)
flacfetch_retry_min_wait: float = float(os.getenv("FLACFETCH_RETRY_MIN_WAIT", "10.0"))  # seconds
flacfetch_retry_max_wait: float = float(os.getenv("FLACFETCH_RETRY_MAX_WAIT", "60.0"))  # seconds

Error Message Improvements

Change from:

Download by ID request failed:

To:

Download by ID request failed after 4 attempts: [ConnectTimeout] Connection to http://10.0.0.5:8080 timed out

Implementation Steps

Phase 1: Check Dependencies

  1. Check if tenacity is in pyproject.toml dependencies
  2. If not present, add tenacity = "^8.2.0" to dependencies
  3. Run poetry lock if dependencies changed

Phase 2: Add Configuration

  1. Add retry configuration to backend/config.py
    • flacfetch_retry_max_attempts = 9
    • flacfetch_retry_min_wait = 10.0
    • flacfetch_retry_max_wait = 60.0

Phase 3: Implement Flacfetch Retry Logic

  1. Import tenacity in backend/services/flacfetch_client.py
  2. Create retry decorator with:
    • Retry on httpx.RequestError
    • Retry on httpx.HTTPStatusError with 5xx status
    • Stop on httpx.HTTPStatusError with 4xx status
    • Exponential backoff with cap at 60s
    • Before-sleep logging callback
  3. Apply decorator to download_by_id() method
  4. Improve error messages to include:
    • Number of attempts made
    • Original exception type and message

Phase 4: Persist Download Info Before Download

  1. Update audio_search.py:_download_and_start_processing()
  2. After getting source info from selected result (line 307-311)
  3. Save to job document BEFORE download attempt:
    job_manager.update_job(job_id, {
        'audio_source_type': 'audio_search',
        'source_name': source_name,
        'source_id': source_id,
        'target_file': target_file,
        'download_url': download_url,
    })
    
  4. Then proceed with download (existing code)
  5. Add to Job model if needed (check if fields exist)
  1. Update backend/api/routes/jobs.py:retry_job()
  2. Add new case before "no input audio" error (line ~1088)
  3. Check if job has saved download params:
    elif job.audio_source_type == 'audio_search' and job.source_id:
        # Has audio search download info - retry download
    
  4. Re-call audio search download logic with saved params
  5. Or refactor download into reusable function

Phase 6: Frontend Error Handling

  1. Update frontend/components/job/JobActions.tsx
  2. Add toast notification library (check if exists)
  3. In handleRetry() catch block, show toast:
    catch (error) {
      const message = error.response?.data?.detail || "Failed to retry job"
      toast.error(message)
      console.error("Failed to retry job:", error)
    }
    

Phase 7: Testing - Flacfetch Retry

  1. Add unit tests to backend/tests/test_flacfetch_client.py
  2. Test retry on RequestError (mock fails N times then succeeds)
  3. Test no retry on 4xx errors
  4. Test retry on 5xx errors
  5. Test max retries exhausted
  6. Verify logging output includes attempt numbers

Phase 8: Testing - End-to-End

  1. Test audio search job with flacfetch down
  2. Verify source info is saved even when download fails
  3. Verify retry button works after source info saved
  4. Test retry succeeds when flacfetch comes back up
  5. Verify error toast appears when retry fails

Phase 9: Documentation

  1. Update backend/services/flacfetch_client.py docstring
  2. Add to docs/LESSONS-LEARNED.md:
    • "Always add retries for external service calls"
    • "Save operation parameters before attempting operations"
    • "Persist enough state to enable retry from any failure point"
  3. Update docs/ARCHITECTURE.md if needed
  4. Document the new job fields in API.md

Files to Create/Modify

File Action Description
backend/config.py Modify Add retry configuration variables
backend/services/flacfetch_client.py Modify Add retry logic to download_by_id()
backend/api/routes/audio_search.py Modify Save download info before attempting download
backend/api/routes/jobs.py Modify Add audio search case to retry endpoint
backend/models/job.py Modify (maybe) Add fields: source_name, source_id, target_file, download_url
frontend/components/job/JobActions.tsx Modify Show error toast when retry fails
backend/tests/test_flacfetch_client.py Modify Add retry tests
backend/tests/test_audio_search.py Modify Test download info persistence
backend/tests/test_jobs.py Modify Test retry with audio search jobs
pyproject.toml Modify (maybe) Add tenacity if not present
docs/LESSONS-LEARNED.md Modify Document patterns
docs/API.md Modify Document new job fields

Testing Strategy

Unit Tests (backend/tests/test_flacfetch_client.py)

  1. Test retry on RequestError

    @pytest.mark.asyncio
    async def test_download_by_id_retries_on_network_error():
        """Should retry 3 times on connection errors."""
        client = FlacfetchClient(base_url="http://test", api_key="key")
    
        with patch("httpx.AsyncClient.post") as mock_post:
            # Fail 2 times, then succeed
            mock_post.side_effect = [
                httpx.RequestError("Connection timeout"),
                httpx.RequestError("Connection refused"),
                Mock(json=lambda: {"download_id": "test123"}, status_code=200),
            ]
    
            result = await client.download_by_id("YouTube", "abc123")
    
            assert result == "test123"
            assert mock_post.call_count == 3
    
  2. Test no retry on 4xx errors

    @pytest.mark.asyncio
    async def test_download_by_id_no_retry_on_client_error():
        """Should NOT retry on 400/401/404 errors."""
        client = FlacfetchClient(base_url="http://test", api_key="key")
    
        with patch("httpx.AsyncClient.post") as mock_post:
            response = Mock(status_code=404, text="Not found")
            response.raise_for_status.side_effect = httpx.HTTPStatusError(
                "Not found", request=Mock(), response=response
            )
            mock_post.return_value = response
    
            with pytest.raises(FlacfetchServiceError, match="404"):
                await client.download_by_id("YouTube", "abc123")
    
            assert mock_post.call_count == 1  # No retries
    
  3. Test retry on 5xx errors

    @pytest.mark.asyncio
    async def test_download_by_id_retries_on_server_error():
        """Should retry on 500/503 errors."""
        # Similar to test 1 but with HTTP 503 errors
    
  4. Test max retries exhausted

    @pytest.mark.asyncio
    async def test_download_by_id_fails_after_max_retries():
        """Should fail with clear message after max retries."""
        # All attempts fail, verify error message includes attempt count
    

Integration Tests

  1. Test with actual flacfetch service (if available in test environment)
    • Verify retry doesn't break normal operation
    • Test with deliberate network issues

Manual Testing

  1. Simulate network failure

    # Temporarily stop flacfetch VM
    gcloud compute instances stop flacfetch-service --zone=us-central1-a
    
    # Submit a test job
    # Observe logs showing retry attempts
    
    # Restart VM during retries
    gcloud compute instances start flacfetch-service --zone=us-central1-a
    
    # Job should succeed
    
  2. Test with intentional delays

    • Add sleep in flacfetch API handler
    • Verify retries don't cause issues

Open Questions

  • Is tenacity already in dependencies? (Will check during implementation)
  • What's the appropriate total timeout budget? - 9 attempts over ~6 minutes
  • Should we fix the Retry button UX in this PR? - YES, it's the same root cause
  • Why did job 749141f8 fail before saving audio source info? - Download info saved AFTER download, not before
  • Should we also add retries to wait_for_download()?
    • Probably NO - that already has its own polling/timeout logic
  • Should we add circuit breaker pattern?
    • Probably NO for now - keep it simple
    • Could add in future if we see cascading failures
  • Should retry logic be applied to other flacfetch methods?
    • start_download() - probably YES (same pattern)
    • search() - maybe YES (search can also have network issues)
    • get_download_status() - probably NO (polling already handles this)
    • Decision: Apply to start_download() too, skip search() for now
  • Which toast library should frontend use?
    • Check if already exists (react-hot-toast, sonner, etc.)
    • If not, add lightweight library
  • Should we save download params to state_data or top-level fields?
    • Top-level fields more visible and easier to query
    • Decision: Top-level fields (source_id, source_name, etc.)

Rollback Plan

If this causes issues:

  1. Quick rollback: Set env var FLACFETCH_RETRY_MAX_ATTEMPTS=1 in Cloud Run

    • Disables retries without code change
    • Can deploy in ~2 minutes
  2. Full rollback: Revert the PR

    • Git revert the merge commit
    • Deploy previous version
  3. Monitoring: Watch for:

    • Increased request duration to flacfetch
    • Jobs timing out at higher layer (Cloud Tasks, Cloud Run)
    • Duplicate download attempts causing issues

Success Criteria

  • Jobs with transient flacfetch failures succeed instead of failing
  • Error messages are more informative (show attempts and root cause)
  • No increase in false positives (jobs that should fail, now hang)
  • All tests pass
  • No impact on successful jobs (retry overhead is minimal)

Job 749141f8 - The Incident

What happened:

  • User searched for audio via audio search
  • Selected "Elizabeth Moen - Songbird"
  • Backend tried to download via flacfetch
  • Flacfetch call failed with network error (connection timeout/refused)
  • Job immediately failed
  • No source info saved → Retry button broken
  • User sees "Retry" button but clicking does nothing (silent 400 error)

How this fix helps:

  1. ✅ Flacfetch retry logic → Download would have succeeded after retry
  2. ✅ Persist download info → Even if all retries fail, user can retry later
  3. ✅ Retry endpoint fix → Retry button will actually work
  4. ✅ Frontend toast → User sees helpful error if retry still fails

Note: Job 749141f8 cannot be fixed retroactively (no source info saved). But future jobs with same issue will be fixable via retry button.

Future Work (Not in This PR)

Similar pattern should be applied to other external service calls:

  • AudioShake API (lyrics transcription)
  • Modal API (audio separation)
  • YouTube direct API calls (not via flacfetch)
  • Email sending (SendGrid)

References