🎤 Karaoke Job Inspector

Debug failed jobs in seconds

📚 Understanding the Job Pipeline

▼

Job Processing Flow

When a job is submitted, it goes through these stages:

Job Created → Audio Worker → Lyrics Worker → Screens Worker → Human Review → Render Video → Video Worker

Worker Descriptions

  • Audio Worker: Downloads audio, separates vocals/instrumental using Modal API (5-10 min)
  • Lyrics Worker: Transcribes lyrics via AudioShake, fetches from Genius, auto-corrects (2-5 min)
  • Screens Worker: Generates title and end screen images
  • Render Video Worker: After human review, renders the karaoke video with corrections
  • Video Worker: Final encoding, thumbnail generation, distribution (YouTube/Dropbox/GDrive)

Job Statuses

  • searching_audio Looking for audio source
  • downloading Downloading/preparing input files
  • separating Audio separation in progress (Modal)
  • transcribing Lyrics transcription (AudioShake)
  • awaiting_review Ready for human review
  • rendering Video generation in progress
  • complete Job finished successfully
  • failed Job encountered an error

🔍 How to Debug a Failed Job

▼

Step 1: Check the Logs

Enter the job ID above and click "View Logs". Look for:

  • ERROR severity entries - these show what went wrong
  • Messages like Created Cloud Task for audio worker - confirms task was queued
  • Messages like AUDIO WORKER STARTED - confirms worker began processing
  • The last log entry before failure - often reveals the cause

Step 2: Check Cloud Tasks

Go to Cloud Tasks Queues and look for:

  • Queue depth - if > 0, tasks are waiting to be processed
  • Failed tasks - click into the queue to see failed task details
  • Retry count - tasks retry up to 5 times before failing permanently

Step 3: Check Trace (for timing issues)

Click "View Trace" to see the distributed trace. Look for:

  • Spans with long durations - indicates slow operations
  • Missing spans - worker may have crashed before creating span
  • Error annotations on spans

Step 4: Common Issues

401 on worker endpoints Auth token issue - check ADMIN_TOKENS secret is set
Job stuck at "downloading" Audio download failed - check if source URL is valid
No worker logs appear Cloud Task failed to trigger - check Cloud Tasks queue
Modal API timeout Audio separation took > 30min - try a shorter track
AudioShake error Transcription failed - check API key and credits

💻 Useful gcloud Commands

▼

View logs for a specific job

gcloud logging read 'resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND "YOUR_JOB_ID"' \
  --project=nomadkaraoke --limit=50 --format="table(timestamp,severity,textPayload)"

View recent errors only

gcloud logging read 'resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND severity>=ERROR' \
  --project=nomadkaraoke --limit=20 --freshness=1h --format="table(timestamp,jsonPayload.message,textPayload)"

Check Cloud Tasks queue status

gcloud tasks queues list --location=us-central1 --project=nomadkaraoke

View pending tasks in a queue

gcloud tasks list --queue=audio-worker-queue --location=us-central1 --project=nomadkaraoke

Check Cloud Run service status

gcloud run services describe karaoke-backend --region=us-central1 --project=nomadkaraoke \
  --format="yaml(status.conditions)"

Get job data from Firestore (via API)

curl -H "Authorization: Bearer YOUR_TOKEN" \
  "https://karaoke-backend-723537321261.us-central1.run.app/api/jobs/YOUR_JOB_ID"

Force redeploy Cloud Run

gcloud run services update karaoke-backend --region=us-central1 --project=nomadkaraoke \
  --update-labels=force-redeploy=$(date +%s)

📝 Log Patterns to Look For

▼

Successful Worker Execution

[job:abc123] Created Cloud Task for audio worker: projects/.../tasks/123
[job:abc123] === AUDIO WORKER STARTED ===
[job:abc123] Downloading audio file...
[job:abc123] Audio downloaded: track.flac
[job:abc123] Starting audio separation...
[job:abc123] Audio separation complete!
[job:abc123] Uploaded stems to GCS

Authentication Failure (401)

POST /api/internal/workers/audio HTTP/1.1" 401
Authentication failed: Invalid token. Token provided: True, Token length: 820

Cause: Cloud Tasks OIDC token overwriting admin token (fixed in v0.75.3+)

Worker Timeout

DeadlineExceeded: 504 Deadline Exceeded
The request deadline was exceeded.

Cause: Worker took longer than Cloud Run timeout (default 30min)

External API Failure

Modal API error: Connection refused
AudioShake error: 429 Too Many Requests
Genius API: 401 Unauthorized

Cause: External service issue - check API keys and rate limits

File Not Found

Failed to download audio file
GCS object not found: gs://bucket/path/file.flac

Cause: Input file missing or path incorrect

Tip: Press Enter after typing the job ID to view logs. All guides above are collapsible - click to expand.