Files
karaoke-gen/backend/workers
Andrew Beveridge ☄️andClaude Opus 5.5 70bdd0e693 fix(alerts): start-review 409, per-cause crash patterns, one alert per job failure (v0.290.1) (#1243)
Section 1 of the 2026-11-03 #alerts triage handoff: a stale push on a
finished song got a 500, and the error monitor could stay silent on a
new crash.

Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980744448843886
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980746394996879
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566980747905073224
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565697725280948294
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565803187334815775
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1565818418072916110
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566225593354420226
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1566256141074169987
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567207141994135654
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252495535841386
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1567252497620537415

## What changed

**1a. start-review:** a job not in review gets 409
(`jobs.jobNotInReviewState`) with no transition attempted. Losing the
auto-approver race is a 409 too. `correction-data` has the same race,
and its `except Exception` turned it into a 500; it now returns 409
(`except HTTPException: raise` added). Its status gate still returns
400, as before.

**1b. Crash patterns:** `Exception in ASGI application` and
`request_audit_error` become `Unhandled exception: <Type: message>`,
read from the entry's `exception` field. The handoff's suspicion is
confirmed in `upsert_pattern`: an existing `new` pattern returns
`is_new=False`, so a second, different crash stayed silent. Both log
lines of one crash now share one pattern. The path and method go in the
sample only.

I read the line after the last frame rather than the traceback's last
line. pydantic's last line is a docs link shared by every failure, and
group tracebacks end in a separator.

**1c. One alert per failure:**
- YouTube ids are normalized out of signatures, which also fixes the
false "first time" line in `ops_alerts`.
- `Job <id> failed` (both call sites in `job_manager`) logs WARNING,
since the status write sends the job-failed alert.
- `URL download failed:` is ignored in the monitor, not demoted, because
`file_upload.py` belongs to the section 4 PR.
- `[job:<id>] Download failed` logs WARNING for the expected
`DownloadError`/`FlacfetchServiceError` types. This wasn't named in the
handoff, but it is the same duplicate on the audio-search path.
Unexpected exceptions there stay ERROR with a traceback.

**1d. Suppressions and demotions:**
- Zalo's `zaloJSV2 is not defined`: dropped at `/api/client-errors`,
only with a Zalo UA.
- `python -c` experiments on `audio-separation-job`: ignored, using the
handoff's regex.
- Spotify 403 on `/lyrics`: WARNING with `response.text[:200]`. 401 on
`/lyrics` and 401/403 on `/search` stay ERROR.
- Dropbox token probe: WARNING. A rejected refresh used to log nothing,
so it now logs ERROR, as does having no refresh credentials.
- decide-search health aborts: this was a small change. `should_ignore`
takes an optional `request_url` and matches the exact path
`/api/search/health`.

**1e. Cold start:** `anyio._backends._asyncio` is added to
`_LAZY_ROUTE_IMPORTS`. The prod stack from 10-30 confirms the stall was
in that import. Lifespan also sends one `/api/health` request through
the app in a worker thread, bounded by the existing 30s timeout and
best-effort.

## Testing

- TDD: new tests failed first, then passed.
- Affected suites: 826 passed.
- `backend/tests` (excluding `backend/tests/emulator`): 7088 passed, 8
failed, 29 skipped. All 8 are in `test_emulator_integration.py`, and
that file also shows 8 failures at the base commit a8f9618a. I did not
compare the 8 test names one by one.
- `tests/unit` outside the listed files was not run.
`test_audio_fetcher.py` and `test_remote_cli.py` fail to collect in my
environment; I didn't change either file.
- Read-only replay of real prod entries through the new code: the 11-03
crash gave one specific pattern with `POST
/api/jobs/7f89f884/start-review` in the sample, the decide-search health
aborts and the experiment traceback were ignored.
- Local CodeRabbit review completed with 0 findings.

## Not verified

- Nothing is live, so the alert volume and the cold-start stall haven't
been measured. The warm request has only run in tests, not on a real
Cloud Run cold start.
- A request's first-use imports beyond the anyio backend aren't
individually checked.
- My test environment has no `flacfetch`, so its lazy imports only log a
warning here.
- Zalo: the trigger is one report (11-03).

## Known leftovers

- `validate_state_transition` still logs ERROR before raising, so the
auto-approver race (and the existing in_review race) still emits one
`Invalid state transition` ERROR even though the client now gets a clean
response. The stale-push case no longer reaches it. I left this because
other callers rely on that log.
- A crash that also logs its own ERROR first still gives two patterns,
the ERROR and `Unhandled exception`.
- The first bot block after deploy shows "first time" once, because the
normalized signature is new.
- Old `Exception in ASGI application` and `request_audit_error` patterns
stop recurring and will auto-resolve.
- If the section 4 PR rewords `URL download failed:`, the ignore entry
in `known_issues.py` needs the same wording.

## For Andrew (infra and cost, not changed here)

- Cloud Run startup probe and min-instances for `karaoke-backend`, if
cold-start stalls continue after this ships.
- decide-search's min-0 is why its health pings abort. The suppression
hides those, but real search paths will still alert on the same abort.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012RYLVTvqXyL4QyA2EjzQMG

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 23:04:04 +00:00
..
2025-11-30 23:02:28 -05:00

Karaoke Generation Workers

This directory contains background workers that handle long-running karaoke generation tasks.

Overview

Workers are asynchronous Python functions that process different stages of the karaoke generation workflow. They are triggered via internal HTTP API endpoints and run as FastAPI background tasks.

Architecture

Job Submission
    │
    ├─→ Audio Worker (parallel)
    │   ├─→ Stage 1: Clean instrumental (3-5 min)
    │   ├─→ Stage 2: Backing vocals (2-3 min)
    │   └─→ mark_audio_complete()
    │
    └─→ Lyrics Worker (parallel)
        ├─→ Fetch reference lyrics
        ├─→ Transcribe with AudioShake (1-2 min)
        ├─→ Auto-correct
        └─→ mark_lyrics_complete()
    
    ↓ (when both complete)
    
Screens Worker (auto-triggered)
    ├─→ Generate title screen
    ├─→ Generate end screen
    └─→ AWAITING_INSTRUMENTAL_SELECTION
    
    ↓ (user selects)
    
Video Worker (user-triggered)
    ├─→ Remux with instrumental
    ├─→ Encode multiple formats
    ├─→ Package (CDG, TXT)
    └─→ COMPLETE

Workers

1. Audio Worker (audio_worker.py)

Purpose: Separates audio into stems using GPU-accelerated Modal API

Stages:

  1. Stage 1: Clean instrumental separation

    • Model: model_bs_roformer_ep_317_sdr_12.9755.ckpt
    • Also: 6-stem separation (bass, drums, guitar, piano, other, vocals)
    • Time: 3-5 minutes
  2. Stage 2: Backing vocals separation

    • Model: mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt
    • Separates lead vocals from backing vocals
    • Time: 2-3 minutes
  3. Post-Processing:

    • Combine instrumentals
    • Normalize audio levels
    • Upload all stems to GCS

Triggers: Automatically on job creation
Next Stage: Coordinates with lyrics worker
State Transitions: PENDING → SEPARATING_STAGE1 → SEPARATING_STAGE2 → AUDIO_COMPLETE

Integration: Uses karaoke_gen.AudioProcessor with remote Modal API


2. Lyrics Worker (lyrics_worker.py)

Purpose: Fetches, transcribes, and corrects lyrics

Stages:

  1. Fetch Reference Lyrics:

    • Sources: Genius, Spotify, Musixmatch (via RapidAPI)
    • Fallback order: Genius → Spotify → Musixmatch
    • Time: <1 minute
  2. Transcribe Audio:

    • Service: AudioShake API
    • Output: Word-level timestamps with confidence scores
    • Time: 1-2 minutes
  3. Automatic Correction:

    • Algorithm: LyricsTranscriber (ExtendAnchorHandler, SyllablesMatchHandler)
    • Matches transcription to reference lyrics
    • Fixes common errors (homophones, etc.)
    • Generates corrections JSON
    • Time: 30 seconds
  4. Upload for Review:

    • corrections.json → For review interface
    • audio.flac → For playback
    • reference.txt → For comparison

Triggers: Automatically on job creation
Next Stage: AWAITING_REVIEW (human interaction required)
State Transitions: PENDING → TRANSCRIBING → CORRECTING → LYRICS_COMPLETE → AWAITING_REVIEW

Integration: Uses karaoke_gen.LyricsProcessor and LyricsTranscriber


3. Screens Worker (screens_worker.py)

Purpose: Generates title and end screen videos

Stages:

  1. Generate Title Screen:

    • Artist and song title
    • Style parameters (font, colors, background)
    • Output: .mov file
    • Time: 10-15 seconds
  2. Generate End Screen:

    • "Thank you for singing!" message
    • Matching style
    • Output: .mov file
    • Time: 10-15 seconds
  3. Upload to GCS:

    • Both screens uploaded
    • URLs stored in job metadata
  4. Countdown Padding (if needed):

    • Detects if countdown was added to vocals
    • Pads instrumentals to match
    • Re-uploads padded versions

Triggers: Automatically when audio + lyrics both complete
Next Stage: AWAITING_INSTRUMENTAL_SELECTION (human interaction required)
State Transitions: AUDIO_COMPLETE + LYRICS_COMPLETE → GENERATING_SCREENS → AWAITING_INSTRUMENTAL_SELECTION

Integration: Uses karaoke_gen.VideoGenerator


4. Video Worker (video_worker.py) - TODO

Purpose: Generates final karaoke videos in multiple formats

Stages:

  1. Remux:

    • Combine lyrics video with selected instrumental
    • Add title and end screens
    • Create base karaoke video
  2. Encode Multiple Formats:

    • Lossless 4K MP4 (PCM audio)
    • Lossless 4K MKV (FLAC audio)
    • Lossy 4K MP4 (AAC audio)
    • Lossy 720p MP4 (AAC audio)
    • Time: 15-20 minutes
  3. Package:

    • CDG+MP3 ZIP (karaoke machines)
    • TXT+MP3 ZIP (simple apps)
  4. Upload:

    • All formats to GCS
    • Optional: YouTube upload
    • Optional: Dropbox sync

Triggers: User selects instrumental
Next Stage: COMPLETE or UPLOADING
State Transitions: INSTRUMENTAL_SELECTED → GENERATING_VIDEO → ENCODING → PACKAGING → COMPLETE

Integration: Uses karaoke_gen.KaraokeFinalise (Cloud Build recommended for encoding)

Status: Not yet implemented


Worker Pattern

SOLID Principles

Each worker follows SOLID principles:

Single Responsibility: Each worker handles exactly one stage

  • Audio: Only audio separation
  • Lyrics: Only transcription/correction
  • Screens: Only screen generation
  • Video: Only final video generation

Open/Closed: Extensible without modification

  • New workers can be added easily
  • Style parameters allow customization
  • No modification of existing workers needed

Liskov Substitution: Interface consistency

  • All workers follow same pattern
  • Can swap implementations
  • karaoke_gen classes are abstractions

Interface Segregation: Focused interfaces

  • No bloated base class
  • Each worker has minimal dependencies
  • Clear, focused API

Dependency Inversion: Depends on abstractions

  • Workers use karaoke_gen classes (abstractions)
  • Not coupled to specific implementations
  • Easy to test and mock

Common Pattern

All workers follow this structure:

async def process_stage(job_id: str) -> bool:
    """Main entry point for worker."""
    job_manager = JobManager()
    storage = StorageService()
    settings = get_settings()
    
    # Get job
    job = job_manager.get_job(job_id)
    
    # Create temp directory
    temp_dir = tempfile.mkdtemp(prefix=f"karaoke_{stage}_{job_id}_")
    
    try:
        # Transition to processing state
        job_manager.transition_to_state(
            job_id=job_id,
            new_status=JobStatus.PROCESSING_STATE,
            progress=X,
            message="Processing..."
        )
        
        # Download inputs from GCS
        inputs = await download_inputs(...)
        
        # Process with karaoke_gen
        outputs = await process_with_karaoke_gen(inputs)
        
        # Upload outputs to GCS
        await upload_outputs(outputs)
        
        # Transition to complete state
        job_manager.transition_to_state(
            job_id=job_id,
            new_status=JobStatus.COMPLETE_STATE,
            progress=Y,
            message="Complete"
        )
        
        # Trigger next stage if applicable
        job_manager.mark_stage_complete(job_id)
        
        return True
        
    except Exception as e:
        # Structured error handling
        job_manager.mark_job_failed(
            job_id=job_id,
            error_message=str(e),
            error_details={"stage": "stage_name", "error": str(e)}
        )
        return False
        
    finally:
        # Always cleanup
        shutil.rmtree(temp_dir)

Error Handling

All workers implement comprehensive error handling:

  1. Try/Except/Finally: Standard pattern
  2. Structured Errors: error_details dict with context
  3. State Updates: Job marked as FAILED with error message
  4. Cleanup: Temp directories always removed
  5. Logging: All errors logged with stack traces
  6. Retry: Failed jobs can be retried from last checkpoint

File Management

Workers use temporary directories for isolation:

temp_dir = tempfile.mkdtemp(prefix=f"karaoke_{worker}_{job_id}_")
try:
    # Processing...
finally:
    shutil.rmtree(temp_dir)  # Always cleanup

Benefits:

  • Isolation: Each job has separate workspace
  • No Leaks: Automatic cleanup
  • Concurrency: Multiple jobs don't interfere
  • Security: Temp files are private

GCS Integration

Workers upload/download files from Google Cloud Storage:

# Download
local_path = os.path.join(temp_dir, "input.flac")
storage.download_file(gcs_url, local_path)

# Upload
gcs_path = f"jobs/{job_id}/category/file.ext"
url = storage.upload_file(local_path, gcs_path)
job_manager.update_file_url(job_id, 'category', 'file_type', url)

File organization:

gs://bucket/jobs/{job_id}/
├── input.flac                  # Original audio
├── stems/
│   ├── instrumental_clean.flac
│   ├── instrumental_with_backing.flac
│   ├── vocals.flac
│   ├── backing_vocals.flac
│   └── lead_vocals.flac
├── lyrics/
│   ├── corrections.json
│   ├── audio.flac
│   └── reference.txt
├── screens/
│   ├── title.mov
│   └── end.mov
├── videos/
│   └── with_vocals.mkv
└── finals/
    ├── lossless_4k_mp4.mp4
    ├── lossless_4k_mkv.mkv
    ├── lossy_4k_mp4.mp4
    └── lossy_720p_mp4.mp4

Worker Coordination

Parallel Processing

Audio and lyrics workers run in parallel:

# Job submission triggers both
background_tasks.add_task(worker_service.trigger_audio_worker, job_id)
background_tasks.add_task(worker_service.trigger_lyrics_worker, job_id)

Coordination via job state:

# Screen generation is gated on LYRICS ONLY — audio separation is decoupled so
# reviewers can start while it finishes. mark_*_complete only persist a flag; the
# screens dispatch is an AWAITED, idempotent call (advance_to_screens_if_ready).
# A previous fire-and-forget asyncio trigger was cancelled on Cloud Run Job loop
# teardown, orphaning jobs at `downloading` (E2E job-orchestration stall, 2026-09).

# Lyrics worker completes (PRIMARY trigger)
job_manager.mark_lyrics_complete(job_id)          # sets lyrics_complete flag
await job_manager.advance_to_screens_if_ready(job_id)
→ if lyrics_complete and status == DOWNLOADING: await trigger_screens_worker()

# Audio worker completes (FALLBACK trigger — rescues a lost lyrics dispatch)
job_manager.mark_audio_complete(job_id)           # sets audio_complete flag
await job_manager.advance_to_screens_if_ready(job_id)
→ no-op unless lyrics already done AND still at DOWNLOADING (status guard)

# Defense-in-depth: recover-stuck-jobs (every 5 min) re-triggers any job left at
# DOWNLOADING with lyrics_complete but no screens (a lost/failed dispatch).

Benefits:

  • No race conditions: Firestore handles atomicity
  • Independent processing: Workers don't block each other
  • Automatic progression: No manual coordination needed
  • Clear state tracking: Easy to debug

Sequential Processing

Some stages must be sequential:

Screens Worker
    ↓ (auto-triggered when audio + lyrics complete)
AWAITING_INSTRUMENTAL_SELECTION
    ↓ (user selects)
Video Worker
    ↓ (generates all formats)
COMPLETE

Coordination via state transitions:

# Screens worker transitions to AWAITING_INSTRUMENTAL_SELECTION
# User submits selection
# API endpoint triggers video worker
background_tasks.add_task(worker_service.trigger_video_worker, job_id)

Triggering Workers

Workers are triggered via WorkerService (see backend/services/worker_service.py):

from backend.services.worker_service import get_worker_service

worker_service = get_worker_service()

# Trigger specific worker
await worker_service.trigger_audio_worker(job_id)
await worker_service.trigger_lyrics_worker(job_id)
await worker_service.trigger_screens_worker(job_id)
await worker_service.trigger_video_worker(job_id)

Internal API endpoints (see backend/api/routes/internal.py):

POST /api/internal/workers/audio
POST /api/internal/workers/lyrics
POST /api/internal/workers/screens
POST /api/internal/workers/video

Request format:

{
  "job_id": "abc123"
}

Response format:

{
  "status": "started",
  "job_id": "abc123",
  "message": "Worker started"
}

Testing Workers

Manual Testing

  1. Submit a job:

    curl -X POST http://localhost:8080/api/jobs \
      -H "Content-Type: application/json" \
      -d '{
        "url": "https://youtube.com/watch?v=...",
        "artist": "ABBA",
        "title": "Waterloo"
      }'
    
  2. Check status:

    curl http://localhost:8080/api/jobs/{job_id}
    
  3. Monitor logs:

    # Watch worker progress
    tail -f logs/backend.log | grep "Job {job_id}"
    

Automated Testing

TODO: Create unit tests for each worker

  • Test audio separation with mock Modal API
  • Test lyrics transcription with mock AudioShake API
  • Test screen generation with mock VideoGenerator
  • Test error handling
  • Test state transitions

Environment Variables

Workers require these environment variables:

Required

  • AUDIO_SEPARATOR_API_URL - Modal audio separation API
  • AUDIOSHAKE_API_TOKEN - AudioShake transcription API

Optional

  • GENIUS_API_TOKEN - Genius lyrics API
  • SPOTIFY_COOKIE_SP_DC - Spotify lyrics
  • RAPIDAPI_KEY - Musixmatch via RapidAPI

GCP

  • GOOGLE_CLOUD_PROJECT - GCP project ID
  • GCS_UPLOAD_BUCKET - Upload bucket
  • GCS_TEMP_BUCKET - Temp files bucket
  • GCS_OUTPUT_BUCKET - Final outputs bucket

All credentials can be stored in Google Secret Manager (production) or environment variables (development).


Performance

Processing Times

Worker Stage Time Can Parallelize
Audio Stage 1 3-5 min Yes (with lyrics)
Audio Stage 2 2-3 min Yes (with lyrics)
Lyrics Fetch <1 min Yes (with audio)
Lyrics Transcribe 1-2 min Yes (with audio)
Lyrics Correct 30 sec Yes (with audio)
Screens Generate 30 sec No
Video Encode 15-20 min No

Total Time (ideal): ~25-30 minutes

  • Parallel phase: 5-10 minutes (audio + lyrics)
  • Human review: 5-15 minutes (variable)
  • Screens: 30 seconds
  • Instrumental selection: 30 seconds (user)
  • Video encoding: 15-20 minutes

Resource Usage

Per job:

  • CPU: Low (workers are I/O bound, encoding is Cloud Build)
  • Memory: ~500MB (temporary files)
  • Disk: ~2GB temporary (cleaned up)
  • Network: ~500MB download, ~1GB upload

Scaling

Workers scale horizontally:

  • Each Cloud Run instance can handle multiple concurrent workers
  • Workers are stateless (state in Firestore)
  • Files in GCS (not local disk)
  • Can run 10+ jobs concurrently per instance

Future Improvements

  1. Cloud Tasks: Replace HTTP triggers with Cloud Tasks for better reliability
  2. Retries: Automatic retry with exponential backoff
  3. Checkpoints: Resume from last successful stage
  4. Progress: Real-time progress updates (percentage)
  5. Notifications: Email/SMS when review needed
  6. Caching: Cache model files, reference lyrics
  7. Metrics: Worker duration, success rate, error rate
  8. Monitoring: Dashboards for worker health

Troubleshooting

Worker Not Starting

Symptom: Job stuck in PENDING
Cause: Worker trigger failed
Solution: Check logs for HTTP errors, verify internal API is accessible

Worker Failing

Symptom: Job transitions to FAILED
Cause: Exception in worker
Solution: Check error_details in job, review logs

Slow Processing

Symptom: Worker takes longer than expected
Cause: API slowness, large file
Solution: Check external API status, optimize file sizes

Files Not Uploading

Symptom: Job completes but no files in GCS
Cause: GCS permissions, network issue
Solution: Verify service account has Storage Admin role

Coordination Issues

Symptom: Screens worker not triggered
Cause: One parallel worker failed
Solution: Check both audio and lyrics worker completed successfully