Debug failed jobs in seconds
When a job is submitted, it goes through these stages:
Enter the job ID above and click "View Logs". Look for:
Created Cloud Task for audio worker - confirms task was queuedAUDIO WORKER STARTED - confirms worker began processingGo to Cloud Tasks Queues and look for:
Click "View Trace" to see the distributed trace. Look for:
| 401 on worker endpoints | Auth token issue - check ADMIN_TOKENS secret is set |
| Job stuck at "downloading" | Audio download failed - check if source URL is valid |
| No worker logs appear | Cloud Task failed to trigger - check Cloud Tasks queue |
| Modal API timeout | Audio separation took > 30min - try a shorter track |
| AudioShake error | Transcription failed - check API key and credits |
gcloud logging read 'resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND "YOUR_JOB_ID"' \
--project=nomadkaraoke --limit=50 --format="table(timestamp,severity,textPayload)"
gcloud logging read 'resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND severity>=ERROR' \
--project=nomadkaraoke --limit=20 --freshness=1h --format="table(timestamp,jsonPayload.message,textPayload)"
gcloud tasks queues list --location=us-central1 --project=nomadkaraoke
gcloud tasks list --queue=audio-worker-queue --location=us-central1 --project=nomadkaraoke
gcloud run services describe karaoke-backend --region=us-central1 --project=nomadkaraoke \
--format="yaml(status.conditions)"
curl -H "Authorization: Bearer YOUR_TOKEN" \
"https://karaoke-backend-723537321261.us-central1.run.app/api/jobs/YOUR_JOB_ID"
gcloud run services update karaoke-backend --region=us-central1 --project=nomadkaraoke \
--update-labels=force-redeploy=$(date +%s)
[job:abc123] Created Cloud Task for audio worker: projects/.../tasks/123
[job:abc123] === AUDIO WORKER STARTED ===
[job:abc123] Downloading audio file...
[job:abc123] Audio downloaded: track.flac
[job:abc123] Starting audio separation...
[job:abc123] Audio separation complete!
[job:abc123] Uploaded stems to GCS
POST /api/internal/workers/audio HTTP/1.1" 401
Authentication failed: Invalid token. Token provided: True, Token length: 820
Cause: Cloud Tasks OIDC token overwriting admin token (fixed in v0.75.3+)
DeadlineExceeded: 504 Deadline Exceeded
The request deadline was exceeded.
Cause: Worker took longer than Cloud Run timeout (default 30min)
Modal API error: Connection refused
AudioShake error: 429 Too Many Requests
Genius API: 401 Unauthorized
Cause: External service issue - check API keys and rate limits
Failed to download audio file
GCS object not found: gs://bucket/path/file.flac
Cause: Input file missing or path incorrect
Tip: Press Enter after typing the job ID to view logs. All guides above are collapsible - click to expand.