## Summary
In prod (2026-08-31 to 10-29), about a third of jobs were last scored
**before stems existed**. People opened the review link 15–40 s after
AWAITING_REVIEW. The audio worker's "second chance" re-score then hit
IN_REVIEW (`wrong_status`), so:
- 134 jobs never got a backing-vocals verdict;
- 7 jobs whose lyrics were in the auto tier went to a human anyway (e.g.
6c4be7d8, ed0b0c33, c4d8a256).
This is the first quick win from the full-auto v2 re-evaluation, which
targets >50% zero-touch jobs.
## Changes
- **`backend/services/auto_approval/review_gate.py` (new).**
- When the text verdict is AUTO and the only blocker is
`audio_incomplete`, the screens worker parks the job in
`GENERATING_SCREENS` with `state_data.auto_review_waiting_for_stems`. No
review link or notification goes out.
- The hold is claimed atomically (`claim_job_state_flag`) by one of:
- the audio worker once stems land (normal path);
- the screens worker itself, if stems landed while it was parking;
- `recover_stuck_jobs`, for holds older than 20 min.
- The claimant re-scores (`trigger="audio_hold"`), then either
auto-completes or opens review exactly like before, with server-side
pre-apply.
- If the post-claim re-score fails, the job falls back to human review.
If that also fails, the hold marker is restored so the sweep retries. A
job is never stranded.
- **Executor:** `hold_for_audio` is recorded in the verdict. It never
applies with `always_review`, `made_for_you`,
`backing_preference=review`, unconfident lyrics, or the kill switch on.
- **Kill switch:** `AUTO_APPROVAL_AUDIO_HOLD_ENABLED` (code default
true, because the executor runs in both the service and the
audio-separation job).
- Docs: LESSONS-LEARNED, README status row.
## Testing
- New: `test_review_gate.py` (11), `test_screens_worker_review_hold.py`
(2), executor hold cases (7), and recover-stuck-jobs hold sweep (3).
- Full backend suite locally: 6962 passed. The only failures are
`test_emulator_integration`, which needs the Firestore emulator and also
fails on pristine main.
- CodeRabbit CLI ran locally. Its one finding (stranded job on
post-claim failure) is fixed; the re-review is clean.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012RYLVVVPsvPbXFhRYY5VSJ
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Karaoke Generation Workers
This directory contains background workers that handle long-running karaoke generation tasks.
Overview
Workers are asynchronous Python functions that process different stages of the karaoke generation workflow. They are triggered via internal HTTP API endpoints and run as FastAPI background tasks.
Architecture
Job Submission
│
├─→ Audio Worker (parallel)
│ ├─→ Stage 1: Clean instrumental (3-5 min)
│ ├─→ Stage 2: Backing vocals (2-3 min)
│ └─→ mark_audio_complete()
│
└─→ Lyrics Worker (parallel)
├─→ Fetch reference lyrics
├─→ Transcribe with AudioShake (1-2 min)
├─→ Auto-correct
└─→ mark_lyrics_complete()
↓ (when both complete)
Screens Worker (auto-triggered)
├─→ Generate title screen
├─→ Generate end screen
└─→ AWAITING_INSTRUMENTAL_SELECTION
↓ (user selects)
Video Worker (user-triggered)
├─→ Remux with instrumental
├─→ Encode multiple formats
├─→ Package (CDG, TXT)
└─→ COMPLETE
Workers
1. Audio Worker (audio_worker.py)
Purpose: Separates audio into stems using GPU-accelerated Modal API
Stages:
-
Stage 1: Clean instrumental separation
- Model:
model_bs_roformer_ep_317_sdr_12.9755.ckpt - Also: 6-stem separation (bass, drums, guitar, piano, other, vocals)
- Time: 3-5 minutes
- Model:
-
Stage 2: Backing vocals separation
- Model:
mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt - Separates lead vocals from backing vocals
- Time: 2-3 minutes
- Model:
-
Post-Processing:
- Combine instrumentals
- Normalize audio levels
- Upload all stems to GCS
Triggers: Automatically on job creation
Next Stage: Coordinates with lyrics worker
State Transitions: PENDING → SEPARATING_STAGE1 → SEPARATING_STAGE2 → AUDIO_COMPLETE
Integration: Uses karaoke_gen.AudioProcessor with remote Modal API
2. Lyrics Worker (lyrics_worker.py)
Purpose: Fetches, transcribes, and corrects lyrics
Stages:
-
Fetch Reference Lyrics:
- Sources: Genius, Spotify, Musixmatch (via RapidAPI)
- Fallback order: Genius → Spotify → Musixmatch
- Time: <1 minute
-
Transcribe Audio:
- Service: AudioShake API
- Output: Word-level timestamps with confidence scores
- Time: 1-2 minutes
-
Automatic Correction:
- Algorithm:
LyricsTranscriber(ExtendAnchorHandler, SyllablesMatchHandler) - Matches transcription to reference lyrics
- Fixes common errors (homophones, etc.)
- Generates corrections JSON
- Time: 30 seconds
- Algorithm:
-
Upload for Review:
- corrections.json → For review interface
- audio.flac → For playback
- reference.txt → For comparison
Triggers: Automatically on job creation
Next Stage: AWAITING_REVIEW (human interaction required)
State Transitions: PENDING → TRANSCRIBING → CORRECTING → LYRICS_COMPLETE → AWAITING_REVIEW
Integration: Uses karaoke_gen.LyricsProcessor and LyricsTranscriber
3. Screens Worker (screens_worker.py)
Purpose: Generates title and end screen videos
Stages:
-
Generate Title Screen:
- Artist and song title
- Style parameters (font, colors, background)
- Output:
.movfile - Time: 10-15 seconds
-
Generate End Screen:
- "Thank you for singing!" message
- Matching style
- Output:
.movfile - Time: 10-15 seconds
-
Upload to GCS:
- Both screens uploaded
- URLs stored in job metadata
-
Countdown Padding (if needed):
- Detects if countdown was added to vocals
- Pads instrumentals to match
- Re-uploads padded versions
Triggers: Automatically when audio + lyrics both complete
Next Stage: AWAITING_INSTRUMENTAL_SELECTION (human interaction required)
State Transitions: AUDIO_COMPLETE + LYRICS_COMPLETE → GENERATING_SCREENS → AWAITING_INSTRUMENTAL_SELECTION
Integration: Uses karaoke_gen.VideoGenerator
4. Video Worker (video_worker.py) - TODO
Purpose: Generates final karaoke videos in multiple formats
Stages:
-
Remux:
- Combine lyrics video with selected instrumental
- Add title and end screens
- Create base karaoke video
-
Encode Multiple Formats:
- Lossless 4K MP4 (PCM audio)
- Lossless 4K MKV (FLAC audio)
- Lossy 4K MP4 (AAC audio)
- Lossy 720p MP4 (AAC audio)
- Time: 15-20 minutes
-
Package:
- CDG+MP3 ZIP (karaoke machines)
- TXT+MP3 ZIP (simple apps)
-
Upload:
- All formats to GCS
- Optional: YouTube upload
- Optional: Dropbox sync
Triggers: User selects instrumental
Next Stage: COMPLETE or UPLOADING
State Transitions: INSTRUMENTAL_SELECTED → GENERATING_VIDEO → ENCODING → PACKAGING → COMPLETE
Integration: Uses karaoke_gen.KaraokeFinalise (Cloud Build recommended for encoding)
Status: Not yet implemented
Worker Pattern
SOLID Principles
Each worker follows SOLID principles:
Single Responsibility: Each worker handles exactly one stage
- Audio: Only audio separation
- Lyrics: Only transcription/correction
- Screens: Only screen generation
- Video: Only final video generation
Open/Closed: Extensible without modification
- New workers can be added easily
- Style parameters allow customization
- No modification of existing workers needed
Liskov Substitution: Interface consistency
- All workers follow same pattern
- Can swap implementations
karaoke_genclasses are abstractions
Interface Segregation: Focused interfaces
- No bloated base class
- Each worker has minimal dependencies
- Clear, focused API
Dependency Inversion: Depends on abstractions
- Workers use
karaoke_genclasses (abstractions) - Not coupled to specific implementations
- Easy to test and mock
Common Pattern
All workers follow this structure:
async def process_stage(job_id: str) -> bool:
"""Main entry point for worker."""
job_manager = JobManager()
storage = StorageService()
settings = get_settings()
# Get job
job = job_manager.get_job(job_id)
# Create temp directory
temp_dir = tempfile.mkdtemp(prefix=f"karaoke_{stage}_{job_id}_")
try:
# Transition to processing state
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.PROCESSING_STATE,
progress=X,
message="Processing..."
)
# Download inputs from GCS
inputs = await download_inputs(...)
# Process with karaoke_gen
outputs = await process_with_karaoke_gen(inputs)
# Upload outputs to GCS
await upload_outputs(outputs)
# Transition to complete state
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.COMPLETE_STATE,
progress=Y,
message="Complete"
)
# Trigger next stage if applicable
job_manager.mark_stage_complete(job_id)
return True
except Exception as e:
# Structured error handling
job_manager.mark_job_failed(
job_id=job_id,
error_message=str(e),
error_details={"stage": "stage_name", "error": str(e)}
)
return False
finally:
# Always cleanup
shutil.rmtree(temp_dir)
Error Handling
All workers implement comprehensive error handling:
- Try/Except/Finally: Standard pattern
- Structured Errors:
error_detailsdict with context - State Updates: Job marked as
FAILEDwith error message - Cleanup: Temp directories always removed
- Logging: All errors logged with stack traces
- Retry: Failed jobs can be retried from last checkpoint
File Management
Workers use temporary directories for isolation:
temp_dir = tempfile.mkdtemp(prefix=f"karaoke_{worker}_{job_id}_")
try:
# Processing...
finally:
shutil.rmtree(temp_dir) # Always cleanup
Benefits:
- Isolation: Each job has separate workspace
- No Leaks: Automatic cleanup
- Concurrency: Multiple jobs don't interfere
- Security: Temp files are private
GCS Integration
Workers upload/download files from Google Cloud Storage:
# Download
local_path = os.path.join(temp_dir, "input.flac")
storage.download_file(gcs_url, local_path)
# Upload
gcs_path = f"jobs/{job_id}/category/file.ext"
url = storage.upload_file(local_path, gcs_path)
job_manager.update_file_url(job_id, 'category', 'file_type', url)
File organization:
gs://bucket/jobs/{job_id}/
├── input.flac # Original audio
├── stems/
│ ├── instrumental_clean.flac
│ ├── instrumental_with_backing.flac
│ ├── vocals.flac
│ ├── backing_vocals.flac
│ └── lead_vocals.flac
├── lyrics/
│ ├── corrections.json
│ ├── audio.flac
│ └── reference.txt
├── screens/
│ ├── title.mov
│ └── end.mov
├── videos/
│ └── with_vocals.mkv
└── finals/
├── lossless_4k_mp4.mp4
├── lossless_4k_mkv.mkv
├── lossy_4k_mp4.mp4
└── lossy_720p_mp4.mp4
Worker Coordination
Parallel Processing
Audio and lyrics workers run in parallel:
# Job submission triggers both
background_tasks.add_task(worker_service.trigger_audio_worker, job_id)
background_tasks.add_task(worker_service.trigger_lyrics_worker, job_id)
Coordination via job state:
# Screen generation is gated on LYRICS ONLY — audio separation is decoupled so
# reviewers can start while it finishes. mark_*_complete only persist a flag; the
# screens dispatch is an AWAITED, idempotent call (advance_to_screens_if_ready).
# A previous fire-and-forget asyncio trigger was cancelled on Cloud Run Job loop
# teardown, orphaning jobs at `downloading` (E2E job-orchestration stall, 2026-09).
# Lyrics worker completes (PRIMARY trigger)
job_manager.mark_lyrics_complete(job_id) # sets lyrics_complete flag
await job_manager.advance_to_screens_if_ready(job_id)
→ if lyrics_complete and status == DOWNLOADING: await trigger_screens_worker()
# Audio worker completes (FALLBACK trigger — rescues a lost lyrics dispatch)
job_manager.mark_audio_complete(job_id) # sets audio_complete flag
await job_manager.advance_to_screens_if_ready(job_id)
→ no-op unless lyrics already done AND still at DOWNLOADING (status guard)
# Defense-in-depth: recover-stuck-jobs (every 5 min) re-triggers any job left at
# DOWNLOADING with lyrics_complete but no screens (a lost/failed dispatch).
Benefits:
- No race conditions: Firestore handles atomicity
- Independent processing: Workers don't block each other
- Automatic progression: No manual coordination needed
- Clear state tracking: Easy to debug
Sequential Processing
Some stages must be sequential:
Screens Worker
↓ (auto-triggered when audio + lyrics complete)
AWAITING_INSTRUMENTAL_SELECTION
↓ (user selects)
Video Worker
↓ (generates all formats)
COMPLETE
Coordination via state transitions:
# Screens worker transitions to AWAITING_INSTRUMENTAL_SELECTION
# User submits selection
# API endpoint triggers video worker
background_tasks.add_task(worker_service.trigger_video_worker, job_id)
Triggering Workers
Workers are triggered via WorkerService (see backend/services/worker_service.py):
from backend.services.worker_service import get_worker_service
worker_service = get_worker_service()
# Trigger specific worker
await worker_service.trigger_audio_worker(job_id)
await worker_service.trigger_lyrics_worker(job_id)
await worker_service.trigger_screens_worker(job_id)
await worker_service.trigger_video_worker(job_id)
Internal API endpoints (see backend/api/routes/internal.py):
POST /api/internal/workers/audio
POST /api/internal/workers/lyrics
POST /api/internal/workers/screens
POST /api/internal/workers/video
Request format:
{
"job_id": "abc123"
}
Response format:
{
"status": "started",
"job_id": "abc123",
"message": "Worker started"
}
Testing Workers
Manual Testing
-
Submit a job:
curl -X POST http://localhost:8080/api/jobs \ -H "Content-Type: application/json" \ -d '{ "url": "https://youtube.com/watch?v=...", "artist": "ABBA", "title": "Waterloo" }' -
Check status:
curl http://localhost:8080/api/jobs/{job_id} -
Monitor logs:
# Watch worker progress tail -f logs/backend.log | grep "Job {job_id}"
Automated Testing
TODO: Create unit tests for each worker
- Test audio separation with mock Modal API
- Test lyrics transcription with mock AudioShake API
- Test screen generation with mock VideoGenerator
- Test error handling
- Test state transitions
Environment Variables
Workers require these environment variables:
Required
AUDIO_SEPARATOR_API_URL- Modal audio separation APIAUDIOSHAKE_API_TOKEN- AudioShake transcription API
Optional
GENIUS_API_TOKEN- Genius lyrics APISPOTIFY_COOKIE_SP_DC- Spotify lyricsRAPIDAPI_KEY- Musixmatch via RapidAPI
GCP
GOOGLE_CLOUD_PROJECT- GCP project IDGCS_UPLOAD_BUCKET- Upload bucketGCS_TEMP_BUCKET- Temp files bucketGCS_OUTPUT_BUCKET- Final outputs bucket
All credentials can be stored in Google Secret Manager (production) or environment variables (development).
Performance
Processing Times
| Worker | Stage | Time | Can Parallelize |
|---|---|---|---|
| Audio | Stage 1 | 3-5 min | Yes (with lyrics) |
| Audio | Stage 2 | 2-3 min | Yes (with lyrics) |
| Lyrics | Fetch | <1 min | Yes (with audio) |
| Lyrics | Transcribe | 1-2 min | Yes (with audio) |
| Lyrics | Correct | 30 sec | Yes (with audio) |
| Screens | Generate | 30 sec | No |
| Video | Encode | 15-20 min | No |
Total Time (ideal): ~25-30 minutes
- Parallel phase: 5-10 minutes (audio + lyrics)
- Human review: 5-15 minutes (variable)
- Screens: 30 seconds
- Instrumental selection: 30 seconds (user)
- Video encoding: 15-20 minutes
Resource Usage
Per job:
- CPU: Low (workers are I/O bound, encoding is Cloud Build)
- Memory: ~500MB (temporary files)
- Disk: ~2GB temporary (cleaned up)
- Network: ~500MB download, ~1GB upload
Scaling
Workers scale horizontally:
- Each Cloud Run instance can handle multiple concurrent workers
- Workers are stateless (state in Firestore)
- Files in GCS (not local disk)
- Can run 10+ jobs concurrently per instance
Future Improvements
- Cloud Tasks: Replace HTTP triggers with Cloud Tasks for better reliability
- Retries: Automatic retry with exponential backoff
- Checkpoints: Resume from last successful stage
- Progress: Real-time progress updates (percentage)
- Notifications: Email/SMS when review needed
- Caching: Cache model files, reference lyrics
- Metrics: Worker duration, success rate, error rate
- Monitoring: Dashboards for worker health
Troubleshooting
Worker Not Starting
Symptom: Job stuck in PENDING
Cause: Worker trigger failed
Solution: Check logs for HTTP errors, verify internal API is accessible
Worker Failing
Symptom: Job transitions to FAILED
Cause: Exception in worker
Solution: Check error_details in job, review logs
Slow Processing
Symptom: Worker takes longer than expected
Cause: API slowness, large file
Solution: Check external API status, optimize file sizes
Files Not Uploading
Symptom: Job completes but no files in GCS
Cause: GCS permissions, network issue
Solution: Verify service account has Storage Admin role
Coordination Issues
Symptom: Screens worker not triggered
Cause: One parallel worker failed
Solution: Check both audio and lyrics worker completed successfully