Files
karaoke-gen/docs/archive/2025-12-31-agentic-timeout-implementation.md
Andrew Beveridge ☄️andClaude Opus 4.5 c68c221ba3 feat(agentic): Add 3-minute timeout for lyrics correction (#149)
## Summary

- Adds configurable timeout (default 3 minutes) for agentic AI lyrics
correction
- When timeout occurs, skips remaining gaps and returns uncorrected
transcription
- Job proceeds to human review where user can manually correct lyrics
- Prevents stuck jobs in production (songs with 74+ gaps could take 30+
minutes)

## Implementation

Two-layer timeout approach:
1. **Inner deadline check** in `corrector.py` gap loop - cooperative
exit before each gap
2. **Outer `asyncio.wait_for()`** in `lyrics_worker.py` - safety net
catches hung LLM calls

## Files Changed

- `backend/config.py` - Add `agentic_correction_timeout_seconds` config
- `backend/workers/lyrics_worker.py` - Add deadline calculation and
outer timeout
- `karaoke_gen/lyrics_processor.py` - Pass deadline through to
controller
- `lyrics_transcriber_temp/lyrics_transcriber/core/controller.py` -
Accept deadline parameter
- `lyrics_transcriber_temp/lyrics_transcriber/correction/corrector.py` -
Add deadline check in gap loop
- `pyproject.toml` - Bump version to 0.85.0

## Test plan

- [x] Unit tests pass (812 passed, 29 skipped)
- [ ] Deploy to production and monitor agentic correction jobs
- [ ] Verify timeout triggers after ~3 minutes on long-running
corrections
- [ ] Verify job proceeds to `awaiting_review` with uncorrected lyrics

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Release Notes

**New Features**
- Agentic AI lyrics correction now includes a configurable timeout
(default 3 minutes). If correction exceeds the timeout, jobs proceed
with raw transcription available for human review.

**Documentation**
- Added implementation notes on timeout behavior and the two-layer
timeout pattern architecture.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-31 16:13:46 -05:00

2.6 KiB
Raw Permalink Blame History

Agentic AI Correction Timeout Implementation

Date: 2025-12-31 PR: #149 Version: 0.85.0

Problem

Agentic AI lyrics correction was causing stuck jobs in production. Songs with many gaps (74+) could take 30+ minutes to process since each gap takes 10-30 seconds for LLM inference. This blocked the entire job pipeline.

Example from production logs:

Processing 22/74 gaps (~15s each) = ~18 minutes elapsed
Estimated total: 74 gaps × 15s = 18+ minutes

Solution

Implemented a configurable 3-minute timeout (default) with two layers:

Layer 1: Inner Deadline Check (Cooperative)

In corrector.py, check deadline before processing each gap:

if deadline and time.time() > deadline:
    self.logger.warning("AGENTIC TIMEOUT: Deadline exceeded...")
    break  # Return uncorrected transcription for human review

Layer 2: Outer asyncio.wait_for (Safety Net)

In lyrics_worker.py, wrap the entire transcription call:

result = await asyncio.wait_for(
    asyncio.to_thread(lyrics_processor.transcribe_lyrics, ...),
    timeout=timeout_seconds + 60  # Extra buffer
)

Behavior

Scenario What Happens
Completes in <3 min Normal flow, corrections applied
Gap loop exceeds deadline Inner check breaks, returns uncorrected
LLM hangs indefinitely Outer timeout fires (rare), job fails
Any timeout Job proceeds to human review with raw transcription

Files Changed

  • backend/config.py - Added agentic_correction_timeout_seconds config
  • backend/workers/lyrics_worker.py - Deadline calculation, outer timeout wrapper
  • karaoke_gen/lyrics_processor.py - Pass deadline to controller
  • lyrics_transcriber_temp/lyrics_transcriber/core/controller.py - Accept deadline parameter
  • lyrics_transcriber_temp/lyrics_transcriber/correction/corrector.py - Deadline check in gap loop

Configuration

Environment variable: AGENTIC_CORRECTION_TIMEOUT_SECONDS (default: 180)

Key Design Decisions

  1. Break, don't raise: Inner timeout uses break to exit the gap loop gracefully, returning whatever we have (uncorrected). This is better than raising an exception because:

    • No retry logic needed
    • Job continues to human review
    • User can manually correct any issues
  2. Two-layer protection: The inner check handles normal cases (many gaps). The outer timeout catches edge cases (single LLM call hanging). Both are needed for reliability.

  3. Configurable timeout: Default 3 minutes balances correction quality vs. user experience. It can be adjusted via environment variable per deployment.