## Summary - Adds configurable timeout (default 3 minutes) for agentic AI lyrics correction - When timeout occurs, skips remaining gaps and returns uncorrected transcription - Job proceeds to human review where user can manually correct lyrics - Prevents stuck jobs in production (songs with 74+ gaps could take 30+ minutes) ## Implementation Two-layer timeout approach: 1. **Inner deadline check** in `corrector.py` gap loop - cooperative exit before each gap 2. **Outer `asyncio.wait_for()`** in `lyrics_worker.py` - safety net catches hung LLM calls ## Files Changed - `backend/config.py` - Add `agentic_correction_timeout_seconds` config - `backend/workers/lyrics_worker.py` - Add deadline calculation and outer timeout - `karaoke_gen/lyrics_processor.py` - Pass deadline through to controller - `lyrics_transcriber_temp/lyrics_transcriber/core/controller.py` - Accept deadline parameter - `lyrics_transcriber_temp/lyrics_transcriber/correction/corrector.py` - Add deadline check in gap loop - `pyproject.toml` - Bump version to 0.85.0 ## Test plan - [x] Unit tests pass (812 passed, 29 skipped) - [ ] Deploy to production and monitor agentic correction jobs - [ ] Verify timeout triggers after ~3 minutes on long-running corrections - [ ] Verify job proceeds to `awaiting_review` with uncorrected lyrics 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes **New Features** - Agentic AI lyrics correction now includes a configurable timeout (default 3 minutes). If correction exceeds the timeout, jobs proceed with raw transcription available for human review. **Documentation** - Added implementation notes on timeout behavior and the two-layer timeout pattern architecture. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2.6 KiB
Agentic AI Correction Timeout Implementation
Date: 2025-12-31 PR: #149 Version: 0.85.0
Problem
Agentic AI lyrics correction was causing stuck jobs in production. Songs with many gaps (74+) could take 30+ minutes to process since each gap takes 10-30 seconds for LLM inference. This blocked the entire job pipeline.
Example from production logs:
Processing 22/74 gaps (~15s each) = ~18 minutes elapsed
Estimated total: 74 gaps × 15s = 18+ minutes
Solution
Implemented a configurable 3-minute timeout (default) with two layers:
Layer 1: Inner Deadline Check (Cooperative)
In corrector.py, check deadline before processing each gap:
if deadline and time.time() > deadline:
self.logger.warning("AGENTIC TIMEOUT: Deadline exceeded...")
break # Return uncorrected transcription for human review
Layer 2: Outer asyncio.wait_for (Safety Net)
In lyrics_worker.py, wrap the entire transcription call:
result = await asyncio.wait_for(
asyncio.to_thread(lyrics_processor.transcribe_lyrics, ...),
timeout=timeout_seconds + 60 # Extra buffer
)
Behavior
| Scenario | What Happens |
|---|---|
| Completes in <3 min | Normal flow, corrections applied |
| Gap loop exceeds deadline | Inner check breaks, returns uncorrected |
| LLM hangs indefinitely | Outer timeout fires (rare), job fails |
| Any timeout | Job proceeds to human review with raw transcription |
Files Changed
backend/config.py- Addedagentic_correction_timeout_secondsconfigbackend/workers/lyrics_worker.py- Deadline calculation, outer timeout wrapperkaraoke_gen/lyrics_processor.py- Pass deadline to controllerlyrics_transcriber_temp/lyrics_transcriber/core/controller.py- Accept deadline parameterlyrics_transcriber_temp/lyrics_transcriber/correction/corrector.py- Deadline check in gap loop
Configuration
Environment variable: AGENTIC_CORRECTION_TIMEOUT_SECONDS (default: 180)
Key Design Decisions
-
Break, don't raise: Inner timeout uses
breakto exit the gap loop gracefully, returning whatever we have (uncorrected). This is better than raising an exception because:- No retry logic needed
- Job continues to human review
- User can manually correct any issues
-
Two-layer protection: The inner check handles normal cases (many gaps). The outer timeout catches edge cases (single LLM call hanging). Both are needed for reliability.
-
Configurable timeout: Default 3 minutes balances correction quality vs. user experience. It can be adjusted via environment variable per deployment.