Files
karaoke-gen/backend/workers
Andrew Beveridge ☄️andClaude Opus 5.5 c277bc8ee3 feat: local E2E stack (fake workers + test API) + email-code sign-in (v0.275.0) (#1165)
## Summary
- **Local stack:** `scripts/local-stack.sh up --frontend` runs the
**real** backend on the Firestore + GCS emulators with `FAKE_WORKERS=1`.
In-process fake workers move jobs through their real states in seconds,
using tiny real media, so web Playwright and mobile Maestro full-journey
tests can run offline against the same endpoints.
- **Email-code sign-in** (nomad-mobile spec §4.1): `POST
/api/users/auth/send-code` + `verify-code`. The code logic is shared
with kjbox via `LoginCodeStore`, and sends are throttled per email and
per IP.
- **Bug fix found by the new suite:** `confirm-duration` moved jobs to
`separating_stage1`, from which they can never reach review. It now
resumes at `downloading`. No prod job had taken this path yet.

## Changes
- `backend/services/fake_workers/`: dispatch seam in
`WorkerService.trigger_*`, plus fake search, download, checkout and
outbox, scenarios, and per-user config. The real stem upload, transcode,
backing-vocals analysis and preview render are reused. Every check is
strict (`is True`), so a MagicMock settings object never enables the
fakes, and startup **refuses FAKE_WORKERS in production**.
- `/api/test/*` router (mounted only in that mode): `status`,
`scenario`, `fake-config`, `emails`, `reset`, plus the fake Stripe
checkout page. Paying it runs the real webhook handler.
- `frontend/e2e/local-stack/` (9 specs, ~50s):
  - sign-up → make → review → instrumental → download
  - buy and cancel credits
  - duration confirm, failed, complete and review states
  - code sign-in and lockout
- New required CI job `local-stack-e2e`.
- Docs: TESTING.md § Local stack, DEVELOPMENT.md, API.md, README,
LESSONS-LEARNED.
- `MAX_SIGNUPS_PER_IP` can now be set by env. Only the local stack
raises it; production keeps the default of 2.

## Testing
- [x] Backend unit: 6175 passed
- [x] Emulator: 152 passed
- [x] Frontend Jest: 1706 passed
- [x] Local-stack Playwright: 9/9 passed (CI mode)

## Review
- [x] CodeRabbit CLI review completed locally. 1 minor finding declined:
it suggested swallowing infrastructure errors in send-code as fake
"sent" responses, which would hide real outages from users.

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01AGuxtgoscF6RJGtpjHPg3W

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:00:24 -04:00
..
2025-11-30 23:02:28 -05:00

Karaoke Generation Workers

This directory contains background workers that handle long-running karaoke generation tasks.

Overview

Workers are asynchronous Python functions that process different stages of the karaoke generation workflow. They are triggered via internal HTTP API endpoints and run as FastAPI background tasks.

Architecture

Job Submission
    │
    ├─→ Audio Worker (parallel)
    │   ├─→ Stage 1: Clean instrumental (3-5 min)
    │   ├─→ Stage 2: Backing vocals (2-3 min)
    │   └─→ mark_audio_complete()
    │
    └─→ Lyrics Worker (parallel)
        ├─→ Fetch reference lyrics
        ├─→ Transcribe with AudioShake (1-2 min)
        ├─→ Auto-correct
        └─→ mark_lyrics_complete()
    
    ↓ (when both complete)
    
Screens Worker (auto-triggered)
    ├─→ Generate title screen
    ├─→ Generate end screen
    └─→ AWAITING_INSTRUMENTAL_SELECTION
    
    ↓ (user selects)
    
Video Worker (user-triggered)
    ├─→ Remux with instrumental
    ├─→ Encode multiple formats
    ├─→ Package (CDG, TXT)
    └─→ COMPLETE

Workers

1. Audio Worker (audio_worker.py)

Purpose: Separates audio into stems using GPU-accelerated Modal API

Stages:

  1. Stage 1: Clean instrumental separation

    • Model: model_bs_roformer_ep_317_sdr_12.9755.ckpt
    • Also: 6-stem separation (bass, drums, guitar, piano, other, vocals)
    • Time: 3-5 minutes
  2. Stage 2: Backing vocals separation

    • Model: mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt
    • Separates lead vocals from backing vocals
    • Time: 2-3 minutes
  3. Post-Processing:

    • Combine instrumentals
    • Normalize audio levels
    • Upload all stems to GCS

Triggers: Automatically on job creation
Next Stage: Coordinates with lyrics worker
State Transitions: PENDING → SEPARATING_STAGE1 → SEPARATING_STAGE2 → AUDIO_COMPLETE

Integration: Uses karaoke_gen.AudioProcessor with remote Modal API


2. Lyrics Worker (lyrics_worker.py)

Purpose: Fetches, transcribes, and corrects lyrics

Stages:

  1. Fetch Reference Lyrics:

    • Sources: Genius, Spotify, Musixmatch (via RapidAPI)
    • Fallback order: Genius → Spotify → Musixmatch
    • Time: <1 minute
  2. Transcribe Audio:

    • Service: AudioShake API
    • Output: Word-level timestamps with confidence scores
    • Time: 1-2 minutes
  3. Automatic Correction:

    • Algorithm: LyricsTranscriber (ExtendAnchorHandler, SyllablesMatchHandler)
    • Matches transcription to reference lyrics
    • Fixes common errors (homophones, etc.)
    • Generates corrections JSON
    • Time: 30 seconds
  4. Upload for Review:

    • corrections.json → For review interface
    • audio.flac → For playback
    • reference.txt → For comparison

Triggers: Automatically on job creation
Next Stage: AWAITING_REVIEW (human interaction required)
State Transitions: PENDING → TRANSCRIBING → CORRECTING → LYRICS_COMPLETE → AWAITING_REVIEW

Integration: Uses karaoke_gen.LyricsProcessor and LyricsTranscriber


3. Screens Worker (screens_worker.py)

Purpose: Generates title and end screen videos

Stages:

  1. Generate Title Screen:

    • Artist and song title
    • Style parameters (font, colors, background)
    • Output: .mov file
    • Time: 10-15 seconds
  2. Generate End Screen:

    • "Thank you for singing!" message
    • Matching style
    • Output: .mov file
    • Time: 10-15 seconds
  3. Upload to GCS:

    • Both screens uploaded
    • URLs stored in job metadata
  4. Countdown Padding (if needed):

    • Detects if countdown was added to vocals
    • Pads instrumentals to match
    • Re-uploads padded versions

Triggers: Automatically when audio + lyrics both complete
Next Stage: AWAITING_INSTRUMENTAL_SELECTION (human interaction required)
State Transitions: AUDIO_COMPLETE + LYRICS_COMPLETE → GENERATING_SCREENS → AWAITING_INSTRUMENTAL_SELECTION

Integration: Uses karaoke_gen.VideoGenerator


4. Video Worker (video_worker.py) - TODO

Purpose: Generates final karaoke videos in multiple formats

Stages:

  1. Remux:

    • Combine lyrics video with selected instrumental
    • Add title and end screens
    • Create base karaoke video
  2. Encode Multiple Formats:

    • Lossless 4K MP4 (PCM audio)
    • Lossless 4K MKV (FLAC audio)
    • Lossy 4K MP4 (AAC audio)
    • Lossy 720p MP4 (AAC audio)
    • Time: 15-20 minutes
  3. Package:

    • CDG+MP3 ZIP (karaoke machines)
    • TXT+MP3 ZIP (simple apps)
  4. Upload:

    • All formats to GCS
    • Optional: YouTube upload
    • Optional: Dropbox sync

Triggers: User selects instrumental
Next Stage: COMPLETE or UPLOADING
State Transitions: INSTRUMENTAL_SELECTED → GENERATING_VIDEO → ENCODING → PACKAGING → COMPLETE

Integration: Uses karaoke_gen.KaraokeFinalise (Cloud Build recommended for encoding)

Status: Not yet implemented


Worker Pattern

SOLID Principles

Each worker follows SOLID principles:

Single Responsibility: Each worker handles exactly one stage

  • Audio: Only audio separation
  • Lyrics: Only transcription/correction
  • Screens: Only screen generation
  • Video: Only final video generation

Open/Closed: Extensible without modification

  • New workers can be added easily
  • Style parameters allow customization
  • No modification of existing workers needed

Liskov Substitution: Interface consistency

  • All workers follow same pattern
  • Can swap implementations
  • karaoke_gen classes are abstractions

Interface Segregation: Focused interfaces

  • No bloated base class
  • Each worker has minimal dependencies
  • Clear, focused API

Dependency Inversion: Depends on abstractions

  • Workers use karaoke_gen classes (abstractions)
  • Not coupled to specific implementations
  • Easy to test and mock

Common Pattern

All workers follow this structure:

async def process_stage(job_id: str) -> bool:
    """Main entry point for worker."""
    job_manager = JobManager()
    storage = StorageService()
    settings = get_settings()
    
    # Get job
    job = job_manager.get_job(job_id)
    
    # Create temp directory
    temp_dir = tempfile.mkdtemp(prefix=f"karaoke_{stage}_{job_id}_")
    
    try:
        # Transition to processing state
        job_manager.transition_to_state(
            job_id=job_id,
            new_status=JobStatus.PROCESSING_STATE,
            progress=X,
            message="Processing..."
        )
        
        # Download inputs from GCS
        inputs = await download_inputs(...)
        
        # Process with karaoke_gen
        outputs = await process_with_karaoke_gen(inputs)
        
        # Upload outputs to GCS
        await upload_outputs(outputs)
        
        # Transition to complete state
        job_manager.transition_to_state(
            job_id=job_id,
            new_status=JobStatus.COMPLETE_STATE,
            progress=Y,
            message="Complete"
        )
        
        # Trigger next stage if applicable
        job_manager.mark_stage_complete(job_id)
        
        return True
        
    except Exception as e:
        # Structured error handling
        job_manager.mark_job_failed(
            job_id=job_id,
            error_message=str(e),
            error_details={"stage": "stage_name", "error": str(e)}
        )
        return False
        
    finally:
        # Always cleanup
        shutil.rmtree(temp_dir)

Error Handling

All workers implement comprehensive error handling:

  1. Try/Except/Finally: Standard pattern
  2. Structured Errors: error_details dict with context
  3. State Updates: Job marked as FAILED with error message
  4. Cleanup: Temp directories always removed
  5. Logging: All errors logged with stack traces
  6. Retry: Failed jobs can be retried from last checkpoint

File Management

Workers use temporary directories for isolation:

temp_dir = tempfile.mkdtemp(prefix=f"karaoke_{worker}_{job_id}_")
try:
    # Processing...
finally:
    shutil.rmtree(temp_dir)  # Always cleanup

Benefits:

  • Isolation: Each job has separate workspace
  • No Leaks: Automatic cleanup
  • Concurrency: Multiple jobs don't interfere
  • Security: Temp files are private

GCS Integration

Workers upload/download files from Google Cloud Storage:

# Download
local_path = os.path.join(temp_dir, "input.flac")
storage.download_file(gcs_url, local_path)

# Upload
gcs_path = f"jobs/{job_id}/category/file.ext"
url = storage.upload_file(local_path, gcs_path)
job_manager.update_file_url(job_id, 'category', 'file_type', url)

File organization:

gs://bucket/jobs/{job_id}/
├── input.flac                  # Original audio
├── stems/
│   ├── instrumental_clean.flac
│   ├── instrumental_with_backing.flac
│   ├── vocals.flac
│   ├── backing_vocals.flac
│   └── lead_vocals.flac
├── lyrics/
│   ├── corrections.json
│   ├── audio.flac
│   └── reference.txt
├── screens/
│   ├── title.mov
│   └── end.mov
├── videos/
│   └── with_vocals.mkv
└── finals/
    ├── lossless_4k_mp4.mp4
    ├── lossless_4k_mkv.mkv
    ├── lossy_4k_mp4.mp4
    └── lossy_720p_mp4.mp4

Worker Coordination

Parallel Processing

Audio and lyrics workers run in parallel:

# Job submission triggers both
background_tasks.add_task(worker_service.trigger_audio_worker, job_id)
background_tasks.add_task(worker_service.trigger_lyrics_worker, job_id)

Coordination via job state:

# Screen generation is gated on LYRICS ONLY — audio separation is decoupled so
# reviewers can start while it finishes. mark_*_complete only persist a flag; the
# screens dispatch is an AWAITED, idempotent call (advance_to_screens_if_ready).
# A previous fire-and-forget asyncio trigger was cancelled on Cloud Run Job loop
# teardown, orphaning jobs at `downloading` (E2E job-orchestration stall, 2026-09).

# Lyrics worker completes (PRIMARY trigger)
job_manager.mark_lyrics_complete(job_id)          # sets lyrics_complete flag
await job_manager.advance_to_screens_if_ready(job_id)
→ if lyrics_complete and status == DOWNLOADING: await trigger_screens_worker()

# Audio worker completes (FALLBACK trigger — rescues a lost lyrics dispatch)
job_manager.mark_audio_complete(job_id)           # sets audio_complete flag
await job_manager.advance_to_screens_if_ready(job_id)
→ no-op unless lyrics already done AND still at DOWNLOADING (status guard)

# Defense-in-depth: recover-stuck-jobs (every 5 min) re-triggers any job left at
# DOWNLOADING with lyrics_complete but no screens (a lost/failed dispatch).

Benefits:

  • No race conditions: Firestore handles atomicity
  • Independent processing: Workers don't block each other
  • Automatic progression: No manual coordination needed
  • Clear state tracking: Easy to debug

Sequential Processing

Some stages must be sequential:

Screens Worker
    ↓ (auto-triggered when audio + lyrics complete)
AWAITING_INSTRUMENTAL_SELECTION
    ↓ (user selects)
Video Worker
    ↓ (generates all formats)
COMPLETE

Coordination via state transitions:

# Screens worker transitions to AWAITING_INSTRUMENTAL_SELECTION
# User submits selection
# API endpoint triggers video worker
background_tasks.add_task(worker_service.trigger_video_worker, job_id)

Triggering Workers

Workers are triggered via WorkerService (see backend/services/worker_service.py):

from backend.services.worker_service import get_worker_service

worker_service = get_worker_service()

# Trigger specific worker
await worker_service.trigger_audio_worker(job_id)
await worker_service.trigger_lyrics_worker(job_id)
await worker_service.trigger_screens_worker(job_id)
await worker_service.trigger_video_worker(job_id)

Internal API endpoints (see backend/api/routes/internal.py):

POST /api/internal/workers/audio
POST /api/internal/workers/lyrics
POST /api/internal/workers/screens
POST /api/internal/workers/video

Request format:

{
  "job_id": "abc123"
}

Response format:

{
  "status": "started",
  "job_id": "abc123",
  "message": "Worker started"
}

Testing Workers

Manual Testing

  1. Submit a job:

    curl -X POST http://localhost:8080/api/jobs \
      -H "Content-Type: application/json" \
      -d '{
        "url": "https://youtube.com/watch?v=...",
        "artist": "ABBA",
        "title": "Waterloo"
      }'
    
  2. Check status:

    curl http://localhost:8080/api/jobs/{job_id}
    
  3. Monitor logs:

    # Watch worker progress
    tail -f logs/backend.log | grep "Job {job_id}"
    

Automated Testing

TODO: Create unit tests for each worker

  • Test audio separation with mock Modal API
  • Test lyrics transcription with mock AudioShake API
  • Test screen generation with mock VideoGenerator
  • Test error handling
  • Test state transitions

Environment Variables

Workers require these environment variables:

Required

  • AUDIO_SEPARATOR_API_URL - Modal audio separation API
  • AUDIOSHAKE_API_TOKEN - AudioShake transcription API

Optional

  • GENIUS_API_TOKEN - Genius lyrics API
  • SPOTIFY_COOKIE_SP_DC - Spotify lyrics
  • RAPIDAPI_KEY - Musixmatch via RapidAPI

GCP

  • GOOGLE_CLOUD_PROJECT - GCP project ID
  • GCS_UPLOAD_BUCKET - Upload bucket
  • GCS_TEMP_BUCKET - Temp files bucket
  • GCS_OUTPUT_BUCKET - Final outputs bucket

All credentials can be stored in Google Secret Manager (production) or environment variables (development).


Performance

Processing Times

Worker Stage Time Can Parallelize
Audio Stage 1 3-5 min Yes (with lyrics)
Audio Stage 2 2-3 min Yes (with lyrics)
Lyrics Fetch <1 min Yes (with audio)
Lyrics Transcribe 1-2 min Yes (with audio)
Lyrics Correct 30 sec Yes (with audio)
Screens Generate 30 sec No
Video Encode 15-20 min No

Total Time (ideal): ~25-30 minutes

  • Parallel phase: 5-10 minutes (audio + lyrics)
  • Human review: 5-15 minutes (variable)
  • Screens: 30 seconds
  • Instrumental selection: 30 seconds (user)
  • Video encoding: 15-20 minutes

Resource Usage

Per job:

  • CPU: Low (workers are I/O bound, encoding is Cloud Build)
  • Memory: ~500MB (temporary files)
  • Disk: ~2GB temporary (cleaned up)
  • Network: ~500MB download, ~1GB upload

Scaling

Workers scale horizontally:

  • Each Cloud Run instance can handle multiple concurrent workers
  • Workers are stateless (state in Firestore)
  • Files in GCS (not local disk)
  • Can run 10+ jobs concurrently per instance

Future Improvements

  1. Cloud Tasks: Replace HTTP triggers with Cloud Tasks for better reliability
  2. Retries: Automatic retry with exponential backoff
  3. Checkpoints: Resume from last successful stage
  4. Progress: Real-time progress updates (percentage)
  5. Notifications: Email/SMS when review needed
  6. Caching: Cache model files, reference lyrics
  7. Metrics: Worker duration, success rate, error rate
  8. Monitoring: Dashboards for worker health

Troubleshooting

Worker Not Starting

Symptom: Job stuck in PENDING
Cause: Worker trigger failed
Solution: Check logs for HTTP errors, verify internal API is accessible

Worker Failing

Symptom: Job transitions to FAILED
Cause: Exception in worker
Solution: Check error_details in job, review logs

Slow Processing

Symptom: Worker takes longer than expected
Cause: API slowness, large file
Solution: Check external API status, optimize file sizes

Files Not Uploading

Symptom: Job completes but no files in GCS
Cause: GCS permissions, network issue
Solution: Verify service account has Storage Admin role

Coordination Issues

Symptom: Screens worker not triggered
Cause: One parallel worker failed
Solution: Check both audio and lyrics worker completed successfully