## Summary Job `dd892383` (NOMAD-1800) finished while the business Dropbox was at its 6 TiB quota. Both Dropbox upload passes failed with `insufficient_space`, but the job still went COMPLETE and emailed the customer a delivery email whose link was the literal `[Dropbox URL not available]`. The only #alerts signal was a raw traceback. Alert: https://discord.com/channels/1198691101943541770/1265517528608669707/1562874010939949110 ## Changes - **New `backend/services/delivery_guard.py`** - `completion_email_hold_reason(job)`: if the job's *effective* distribution settings deliver to Dropbox (`dropbox_path` and `brand_prefix`, the same condition the workers use, so private/NonPublished jobs count too) and `state_data.dropbox_link` is missing, the email is **held**. - `hold_completion_email(...)`: records `state_data.completion_email_held = {reason_code, reason, recipient, at}`, logs a WARNING, and sends **one actionable #alerts message per job** (`send_throttled_ops_alert`, keyed per job). The message covers the job, brand code, the masked recipient whose email was held, the reason (`insufficient_space` is classified as "Dropbox is FULL"), the fix steps and the runbook. The Pushbullet push is unchanged. - `check_dropbox_space_usage()`: calls `users_get_space_usage()`, using team allocation for the business account. It posts one #alerts message when usage reaches 95% and records its state in Firestore (`ops_failure_alerts/dropbox-space-usage`), so it stays quiet while usage stays high and re-arms once usage drops below 95%. It runs on the existing hourly `process-stale-reviews` cron, so no infra change is needed. - **`JobManager._schedule_completion_email`**: checks the hold before sending. Made-for-you ownership still transfers to the customer. - **Admin `POST /api/admin/jobs/{id}/send-completion-email`**: returns 409 when the link is missing, unless `allow_missing_dropbox_link: true` is passed. A successful send clears `completion_email_held`. - **`TemplateService.render_job_completion`**: no more `[Dropbox URL not available]` / `[YouTube URL not available]` placeholders. When a link is missing, that whole section is dropped (English default and GCS wording, plus the localized template). Jobs with no Dropbox delivery at all (tenant, no Dropbox configured) still get their email, just without a Dropbox paragraph. The YouTube-queued and private paths are unchanged. - Dropbox upload failure logs (orchestrator and native pass) go from ERROR to WARNING, so one root cause raises one alert (the held-delivery alert) instead of an error-monitor traceback. - Docs: `TROUBLESHOOTING.md` § "Held completion email (Dropbox delivery failed)" with the manual recovery steps, and `API.md` for the new 409 and override. - Version 0.282.0. Not done: item 5 of the handoff (an admin "redistribute Dropbox only" action). The existing admin re-render also re-uploads YouTube, so it isn't a clean fix for public jobs. The manual recovery steps are in TROUBLESHOOTING.md instead. ## Testing - New `backend/tests/test_delivery_guard.py` (31 tests): failure classification, hold/no-hold matrix (no link, link present, no Dropbox, tenant, private defaults, settings error), held marker and alert content, quota threshold, alert-once and re-arm, failed send does not latch, API error never raises. - `test_made_for_you.py`: email held when the link is missing (ownership still transfers, marker recorded, alert sent); email sent when the link is present; email sent for a job with no Dropbox delivery. - `test_admin_email_endpoints.py`: 409 without the link; override sends and clears the marker; fixed link sends and clears the marker; job with no Dropbox delivery sends normally. - `test_template_service.py`: missing links never render placeholders (en, de, unknown GCS template). - Backend unit suite (`backend/tests`, excluding emulator tests): 6637 passed, 7 failed, 25 skipped. All 7 failures are `test_internal_api.py::*requires_auth` / `*requires_admin`, which fail the same way on unmodified `origin/main` locally (they get 200 instead of 401, apparently from the local environment), and this PR doesn't touch them. - The local CodeRabbit CLI failed ("Invalid token"), so this PR is left open for the bot review. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_016F41fLoBotUw2wUjGCuAok Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
69 KiB
Troubleshooting
Operational runbooks for known production issues.
Review pages hang on "Loading lyrics data…" / uptime alert "Service Unavailable"
Symptom: /app/jobs/#/{id}/review spins forever; the browser is stuck fetching
GET /api/review/{id}/correction-data, which eventually returns "server temporarily
unavailable". A GCP uptime alert ("Karaoke Backend - Service Unavailable") may fire.
/api/health and /api/jobs still respond fast.
Root cause (incident 2026-09-17): GCS signed-URL generation stalls. On Cloud
Run we have no private key, so signing goes through the IAM signBlob API over the
network. correction-data signs 2-3 audio/waveform URLs per load; when signBlob
stalls (transient IAM/metadata networking; no GCP incident is necessarily declared),
those calls hang and pile up worker threads until the service looks down. Tell-tale
in logs: "Generated signed … URL" lines stop appearing while everything else
keeps logging, and there are no "Error generating signed" lines (a hang, not a
failure). Confirm it's not job-specific: correction-data hangs for every in-review
job, not just one.
Immediate fix — restart to get fresh instances / connection pools:
gcloud run services update karaoke-backend --region=us-central1 --project=nomadkaraoke \
--update-labels=forcerestart=$(date +%s)
Non-destructive (config unchanged, new revision). Then verify signing resumed
(replace JOB_ID with any in-review job id — the URL is quoted so no placeholder
angle-brackets reach the shell):
# should return 200 fast, not hang
JOB_ID="d93747cd" # any in-review job
curl -s -m 30 -o /dev/null -w "%{http_code} %{time_total}s\n" \
-H "Authorization: Bearer $(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke | cut -d, -f1)" \
"https://api.nomadkaraoke.com/api/review/${JOB_ID}/correction-data"
# and check signing is logging again
gcloud logging read 'resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND jsonPayload.message:"Generated signed"' \
--project=nomadkaraoke --freshness=3m --limit=3 --format="value(timestamp,jsonPayload.message)"
If a restart doesn't clear it, the IAM Credentials API is genuinely degraded in the region — wait it out / open a GCP ticket.
Root-cause fix (shipped v0.229.0 — Option B): correction-data no longer signs any
URL on the review hot path, so it can't stall on IAM signBlob at all. The two instrumental
options are delivered by a same-origin byte proxy (GET /api/review/{job_id}/instrumental-audio/{option_id},
served with HTTP Range so <audio> can seek — no signing, never expires), and the dead
backing_vocals_waveform_url (never consumed; the waveform comes from the JSON
GET /api/review/{job_id}/waveform-data endpoint) was removed. The frontend turns each option's relative
audio_url into an absolute token URL exactly like getVocalsAudioUrl. See
docs/archive/2026-09-17-review-fast-full-load-plan.md.
Safety net (still in place, v0.228.1): signing anywhere else (e.g. preview-video)
still runs on a dedicated bounded thread pool with a wall-clock budget
(SIGNED_URL_TIMEOUT_S, default 12s) and bounded admission (SIGNED_URL_MAX_INFLIGHT) in
storage_service._generate_signed_url_internal — a stall raises SignedUrlTimeout fast
instead of hanging, and saturation is refused immediately rather than queued. Left as
defense-in-depth beneath the signing-free review path.
Users see "Reconnecting" / "servers unavailable" banner (API event loop frozen)
Symptom: Users report the orange banner, or client_events shows banner_unavailable bursts. In the request logs, every request on one instance (including /api/health) finishes at the same moment after 10-45s.
-
Find the blocking line:
gcloud logging read 'resource.labels.service_name="karaoke-backend" AND jsonPayload.message:"EVENT_LOOP_STALL"' --project nomadkaraoke --limit 20 --format='value(timestamp,severity,jsonPayload.message)'The
in progresslines include the loop thread's stack;culprit=is the innermostbackend/orkaraoke_gen/frame. -
Fix: move that sync work off the loop with
await asyncio.to_thread(...), or make the route a plaindef.test_no_blocking_async_routes.pyalready blocks async routes that never await. -
Measure user impact:
python scripts/client_events_report.py --days 7. It excludes admin/internal/test accounts unless you pass--all, and listsserver_loop_stallculprits. -
If there are no
EVENT_LOOP_STALLlines: the cause is elsewhere (cold start, deploy swap, Cloudflare). Checkbanner_wakingevents and revision times. -
Alert "≥2 event-loop stalls of 10s+" fired but no user complained: check
requests_affected=on theEVENT_LOOP_STALL endedlines. Since v0.268.5 a stall is ERROR, the level the alert counts, only if a request was in flight or queued behind it. Before that, idle autoscaled instances that were CPU-throttled during the ~10-minute SpaCy warmup logged 10-40s "stalls" with an emptyculprit=and paged (2026-10-12/13). Lines withrequests_affected=0stay at WARNING.
Background: docs/archive/2026-10-03-backend-loop-freezes-telemetry-plan.md.
Emails not arriving / E2E "Timed out waiting for email" (Postmark HTML 403)
Symptom: magic links or job emails never arrive; E2E Daily Stage 1 fails at
"Sign up via magic link". Backend logs show
Postmark returned status 403: <html>…403 Forbidden… followed by
Failed to send magic link email to ….
Cause: a bare nginx HTML 403 (not Postmark's JSON error format) is an edge/IP-level
block at api.postmarkapp.com, not a token or payload problem. Seen from
karaoke-backend and karaoke-decide (us-central1, shared Cloud Run egress IPs) since
2026-09-30, in blocks lasting hours (all sends failed 2026-10-02 11:25 UTC onward);
us-east4 Cloud Run Jobs were unaffected. IP Allowlisting is OFF in Postmark (checked
2026-10-02), so this is Postmark's network firewall — ticket #11562402, where Postmark
confirmed an IP-reputation block on the shared Google IP that they won't lift. A JSON 403
(ErrorCode=10) is a real bad-token error instead.
Mitigation in code: on an HTML 403, PostmarkEmailProvider immediately re-sends via
Postmark SMTP (smtp.postmarkapp.com:587, STARTTLS, server token as user+password,
X-PM-Message-Stream: outbound) — API retries never got past a block. Also falls back
when every API attempt hits a connect timeout. 429 / 5xx / connect timeouts are still
retried on the API (backoff 1s, 3s). JSON Postmark errors and read timeouts never fall
back (deterministic, or risk of a duplicate send). SMTP sends log
Email sent to … via Postmark SMTP fallback and have no Postmark MessageID; SMTP access
must stay enabled on the Postmark server (Server → Settings).
Diagnosis:
gcloud logging read 'resource.labels.service_name="karaoke-backend" AND (textPayload:"Postmark returned status" OR textPayload:"Failed to send" OR textPayload:"SMTP")' \
--project nomadkaraoke --freshness 2d --limit 50 --format='value(timestamp,textPayload)'
The E2E waitForEmail helper now checks GET /api/admin/users/{email}/emails on
timeout and reports whether the backend sent anything ("backend has NO record…" =
server-side send failure; "backend sent N email(s)…" = delivery/testmail problem).
Alert: "Email - Postmark SMTP fallback failing (emails being lost)" (log-based metric
email/postmark_smtp_fallback_failures, infrastructure/modules/monitoring.py) fires if a
fallback send fails in karaoke-backend or karaoke-decide.
If SMTP fallback also fails / the alert fires: Postmark is likely blocking SMTP from our
IPs too → give the services a static egress IP. Options, measured costs (~$4–7/mo for the
recommended reuse-the-existing-NAT route) and Pulumi steps:
docs/archive/2026-10-02-postmark-ip-block-options.md.
Fast rollback (bad backend deploy)
When to use: a merge-to-main deploy shipped a bad backend revision — the daily E2E failed, the error monitor/Discord is alerting, or you can see failing jobs/5xx in production — and you want to get back to the last known-good state now rather than wait for a forward fix to build and deploy.
What it rolls back:
- the
karaoke-backendCloud Run service (shifts 100% traffic to a previous revision — instant, no rebuild), and - the three Cloud Run jobs that share the backend image
(
video-encoding-job,lyrics-transcription-job,audio-separation-job) by re-pinning them to a previous image tag.lyrics-transcription-jobruns in us-east4 from thekaraoke-backend-cpuimage copy. That copy only has tags from releases after the region move, sorollback.shskips (with a warning) any job whose target tag is missing and re-pins the rest.audio-download-jobandbulk-search-jobtrack:latestand aren't re-pinned. Cloud Run Jobs can't traffic-split, so they are re-pinned; the old image takes effect on each job's next invocation (in-flight runs are unaffected).
Runbook:
# 1. See what known-good targets exist (service revisions + image tags).
scripts/rollback.sh --list
# 2. Preview the exact gcloud commands without running them.
scripts/rollback.sh \
--to-revision <PREVIOUS_GOOD_REVISION> \
--image-version <PREVIOUS_GOOD_VERSION e.g. v0.222.2> \
--dry-run
# 3. Execute. Roll service + jobs together (typical full rollback):
scripts/rollback.sh \
--to-revision karaoke-backend-00XYZ-abc \
--image-version v0.222.2
# Scope to just one surface if that's all that's bad:
scripts/rollback.sh --to-revision karaoke-backend-00XYZ-abc --service-only
scripts/rollback.sh --image-version v0.222.2 --jobs-only
Notes:
- The script requires an explicit target — it never guesses a revision/version.
- Images are addressable by version tag (
vX.Y.Z) or git SHA — both are pushed by CI (.github/workflows/ci.yml~:1809). - After rollback, confirm the serving version:
curl -s https://api.nomadkaraoke.com/api/health/detailed | jq .version. - This is the manual counterpart to CI's automatic service canary: a failed
candidate-revision smoke leaves the service on the prior revision automatically
(the new revision is deployed
--no-traffic), but the jobs have no auto-rollback — if a bad deploy reached the jobs, re-pin them with this script.
CI "Deploy - Publish to PyPI" fails: "Project size too large"
Cause: PyPI caps total project size at 10 GB. Each karaoke-gen release is a
~62 MiB wheel (bundled 33-locale frontend). Until v0.268.9 every merge published
one, which refilled the cap every few weeks. Since then CI publishes only when
CLI-affecting code changed (scripts/pypi_publish_gate.py), about 1 in 6 merges.
Only the public pip install publish
step fails — Cloud Run and the GCE encoding worker (pulls the wheel from GCS) are
unaffected.
Fix: prune old releases per the retention policy. A monthly workflow
(pypi-prune-reminder.yml) opens/updates a tracking issue with a paste-in
deletion snippet, or run python scripts/prune_pypi_releases.py yourself. Full
runbook: docs/PYPI-STORAGE-PRUNE.md. After pruning,
re-run the failed job: the gate diffs against the version on PyPI, so it still
sees the unpublished CLI change. The package-dist build artifact only lives for
one day, so if the failure is older, run a fresh CI workflow on main instead
(Actions → CI → Run workflow, ticking "Publish to PyPI").
A CLI change isn't on PyPI
Expected if the merge only touched paths outside scripts/pypi_publish_paths.txt
(e.g. a review-UI tweak in frontend/). The "Decide PyPI publish" step in the
Deploy - Publish to PyPI job logs why. To publish now: Actions → CI → Run
workflow on main with "Publish to PyPI" ticked, or put [publish-pypi] in
the PR title before squash-merging.
Job stuck at downloading_audio status
Cause: Before v0.130.0, audio downloads ran as FastAPI BackgroundTasks. Cloud Run would terminate "idle" instances mid-download. Since v0.130.0, downloads use a Cloud Run Job (audio-download-job) and a Cloud Scheduler recovery job runs every 10 minutes (5 before 2026-09-26) to fail stuck downloads automatically.
Auto-recovery: The recover-stuck-downloads scheduler (/api/internal/recover-stuck-jobs, every 10 min) detects jobs stuck in downloading_audio for >10 minutes and, since v0.192.3:
- Torrent sources (RED/OPS) → parks the job in
download_pending_retryand keeps re-attempting the download for up to 24 hours (handles rare tracks with few/intermittent seeders and transient tracker outages), then fails permanently with a clear message. No manual action needed. Seedownload_pending_retrybelow. - Other sources (YouTube/Spotify/URL) → fails the job (deterministic); use the admin retry button to re-attempt.
The stuck check needs a heartbeat. A torrent wait can legitimately last 20-60 min
(see "Audio download stalled" below), so while waiting the worker writes
state_data.audio_download_progress (progress, peers, at) every 2 min. That bumps
updated_at, so only a genuinely dead worker (no update for 10 min) gets parked.
Manual recovery:
# 1. Check for stuck downloads
curl -s "https://api.nomadkaraoke.com/api/health/job-consistency" \
-H "X-Admin-Token: $(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke)" \
| python3 -m json.tool | grep -A3 "downloading_audio_stuck"
# 2. Trigger recovery manually (if scheduler hasn't run yet)
curl -X POST "https://api.nomadkaraoke.com/api/internal/recover-stuck-jobs" \
-H "X-Admin-Token: $(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke)"
# 3. Retry the failed job via admin dashboard or API
curl -X POST "https://api.nomadkaraoke.com/api/jobs/YOUR_JOB_ID/retry" \
-H "X-Admin-Token: $(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke)"
Audio download stalled ("Your audio hasn't started downloading after N minutes")
Symptoms: A failed job with error_details.code == "audio_download_stalled". The job
card shows an amber prompt with Keep trying (up to 1 hour) and Choose different
audio instead of the raw error. Discord gets a 🟠 "Audio download stalled" alert.
Cause: A torrent (RED/OPS) made no progress for the whole stall budget, almost
always because its only seeder is offline (flacfetch logs peers: 0 on every re-announce).
Seeders often come back soon; job 64be8a87 (2026-09-29) sat at 0 peers for 10 min, then
finished in 17s on the next attempt. flacfetch ≥ 0.31 reports error_code: "torrent_stalled" for this case.
Behaviour (v0.250.0+):
- First attempt waits 20 min with no progress (
max_stall_secondssent per request to flacfetch; the Cloud Run execution timeout is overridden to fit, seebackend/services/audio_download_limits.py). - On a stall the worker fails the job with the code above and exits 0, so there is no Cloud Run auto-retry of the same torrent. Other download failures still auto-retry.
- Keep trying →
POST /api/jobs/{id}/retry {"keep_trying": true}, which waits up to 60 min (flacfetch's cap). - Choose different audio →
POST /api/jobs/{id}/choose-different-audio, which sets the job back toawaiting_audio_selectionwith its saved results.
Admin: use the same buttons on the job card, or reset to "Audio" in admin.
Job stuck at downloading or audio_edit_complete (no error)
Two known signatures (both fixed in v0.238.5, job 2c1b922e):
- Audio-edit job stuck at
downloading,requires_audio_edit=true, never reachedawaiting_audio_edit. The download worker was OOM-killed while pre-generating the editor waveform (a 24-bit/96kHz FLAC decoded at native resolution peaked at ~1 GB in the 1 GiBaudio-download-job), and the Cloud Run auto-retry sawdownloadingand skipped. Since v0.238.5 the waveform decodes at 16 kHz mono and a retry resumes the unfinished editor prep. Manual fix:POST /api/admin/jobs/{id}/resetwith{"target_state": "awaiting_audio_edit"}. - Audio-edited job stuck at
audio_edit_completewithlyrics_complete=true. Logs showscreens not (re)triggered — status=audio_edit_complete (expected downloading). The lyrics→screens handoff only accepteddownloading; since v0.238.5 it (and therecover-stuck-jobslost-screens sweep) also acceptsaudio_edit_complete. Manual fix — dispatch the screens worker directly (don't useregenerate-screens: it restores the original status afterwards, so the job never reaches review):
curl -X POST "https://api.nomadkaraoke.com/api/internal/workers/screens" \
-H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" -H "Content-Type: application/json" \
-d '{"job_id": "<job_id>"}'
Job at instrumental_selected "Waiting for audio separation to finish"
This is normal when review finishes before stem separation does. The video worker
parks the job (state_data.video_waiting_for_stems=true) and exits cleanly. The
audio worker resumes the final encode as soon as the stems land
(resume_video_if_waiting_for_stems). Backstop: recover_stuck_jobs re-triggers
the video worker once a job has sat in instrumental_selected for 10+ min with its
stem present. If separation never finishes, it fails the job after 60 min
(error_details.stage = "video_waiting_for_stems"), so it never stays stuck silently.
Before 2026-10-19 (v0.278.11) the video worker failed its prerequisites instead
("Missing instrumental: instrumental_clean") and the job stayed at "Starting final
encode" forever. That got much more likely when the GPU image picked up
audio-separator 0.47.0 (karaoke models 3–5× slower on L4). The version is now pinned
in backend/Dockerfile.gpu-base (AUDIO_SEPARATOR_VERSION). Time a real job before
bumping it.
Job stuck at encoding status
Cause: A Cloud Run deployment killed the poller mid-encoding. The GCE worker finished but nobody received the result. The health service will flag this as encoding_stuck after 50 minutes.
Recovery:
# 1. Confirm the job is flagged
curl -s "https://api.nomadkaraoke.com/api/health/job-consistency" \
-H "X-Admin-Token: $(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke)" \
| python3 -m json.tool | grep -A3 "encoding_stuck"
# 2. Check what the GCE worker knows about the job
WORKER_URL=$(gcloud secrets versions access latest --secret=encoding-worker-url --project=nomadkaraoke)
WORKER_KEY=$(gcloud secrets versions access latest --secret=encoding-worker-api-key --project=nomadkaraoke)
curl -s "$WORKER_URL/status/YOUR_JOB_ID" -H "X-API-Key: $WORKER_KEY" | python3 -m json.tool
# 3. Re-trigger the video worker — it will pick up the cached GCE result if encoding
# already finished, or rejoin the poll if it's still running
curl -X POST "https://api.nomadkaraoke.com/api/internal/workers/video" \
-H "X-Admin-Token: $(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke)" \
-H "Content-Type: application/json" \
-d '{"job_id": "YOUR_JOB_ID"}'
Note: SSH-restarting the encoding worker is not needed since v0.119.6 (PR #413). The /encode endpoint is now idempotent — re-triggering the video worker is sufficient.
Prevention: The video worker now runs as a Cloud Run Job (USE_CLOUD_RUN_JOBS_FOR_VIDEO=true), which runs to completion and is immune to Cloud Run Service deployment rollouts. This replaces the BackgroundTask pattern that was vulnerable to instance termination during deployments. Additionally, CI performs a graceful drain before restarting the GCE encoding worker, the encoding client retries for ~90s, and status polling tolerates up to 5 consecutive failures.
Since v0.184.2, in-flight status polls are also pinned to the worker that accepted the job (get_job_status(..., worker_url=)), so a blue-green deploy that swaps the active_url primary pointer mid-render no longer migrates the poll to a worker that never received the job. If you still see lost contact with worker after 5 consecutive poll failures: Encoding job <ID> not found, confirm via step 2 above that the worker the job was submitted to (not necessarily the current primary) still has the job — a not found from the current primary while the job is alive on the old primary was the pre-v0.184.2 failure mode (incident 2026-06-16, job d3af33ae).
Since v0.194.0, a mid-run job loss is also auto-recovered: the encoding worker processes heavy renders/encodes one at a time (heavy_executor, ENCODING_HEAVY_CONCURRENCY=1) so it can no longer OOM from concurrent 4K encodes (the trigger for the 2026-08-15 Arctic Monkeys batch failure — 3 concurrent encodes OOM-killed the 32 GB fallback worker and restarted it, wiping its in-memory jobs). If the worker does restart mid-render (deploy/crash), wait_for_completion now raises EncodingJobLostError and the render/encode worker resubmits the job under a fresh <id>_retry_<hex8> (bounded by ENCODING_RESUBMIT_MAX=2) instead of failing. To confirm serialization is live on a worker: curl -s localhost:8080/health shows queue_length growing while active_jobs stays 1 under load. Note: main.py ships in the wheel but the running uvicorn process only picks up worker-side changes (executor split, queue_position) on a fresh boot / service restart — ensure_latest_wheel alone does not reload the app process.
Since v0.217.0, a worker-side infrastructure/auth failure is also auto-recovered rather than failing the customer's job. When a fallback VM boots and accepts a job but then can't do real work — e.g. it can't reach the GCE metadata server to fetch its service-account token (Failed to retrieve https://metadata.google.internal/.../service-accounts/default/ … Compute Engine Metadata server unavailable … SSLCertVerificationError, job 6452888e, 2026-08-31) — wait_for_completion now classifies it via is_worker_infra_error() and raises EncodingWorkerInfraError (a subclass of EncodingWorkerStartError) instead of a bare RuntimeError. Before raising, it demotes the broken worker (EncodingWorkerManager.demote_active_worker() records a capacity-state cooldown for that fallback's machine_type@zone and clears active_override) and invalidates the URL cache, so the retry prefers a different worker (the cooldown deprioritises that family in ordered_candidates). It is a preference, not a hard exclusion — if no better candidate can start, the same fallback family can still be selected; the bounded retry then handles a repeat, and a VM that fails repeatedly should be rebooted/re-imaged. The render worker then parks the job for auto-retry (its existing except EncodingWorkerStartError path) and the final-encode Cloud Run Job retries — no manual intervention. This is deliberately scoped to fallback VMs: a metadata blip on the stable c4d primary pair is vanishingly rare, and family-demoting the fast primary would push all traffic onto slow fallbacks, so the primary case is left to a bounded retry. If a job is stuck after this, grep logs for Demoted fallback encoding worker to see which VM was skipped; the VM itself may need a reboot/re-image if the failure recurs on it specifically.
Since v0.194.2, the CI deploy handles that restart automatically even during a c4d Spot stockout. The old blue-green only targeted the c4d primary/secondary; when both were down it rolled back and never refreshed the serving n2 fallback (recorded as active_override_vm), so worker-side changes didn't reach prod until a manual restart (primary_version in the config doc went stale). The deploy is now capacity-aware (infrastructure/encoding-worker/deploy_promote.py): it validates the new wheel on a fresh green worker selected from the ranked 6-family pool (select_green_candidates → the shared encoding_worker_preference.ordered_candidates — fastest-first with capacity cooldown; the c4d secondary if it can start, else the fastest available fallback family, never the current override so it keeps serving) — then promotes the green (primary/secondary swap for a c4d green, or sets active_override for a fallback green) and drains+stops the retired worker. A c4d green also clears a stale override so traffic returns to the fresh primary. Zero-downtime, and it works while c4d is exhausted. Last resort if no separate green can start: an in-place restart of the serving override (brief blip, covered by auto-resubmit).
CDG/TXT packages missing from completed job
Cause: Before v0.119.7, CDG/TXT generation failures were silently caught. Jobs would complete with enable_cdg=True but no CDG ZIP in outputs. Fixed in v0.119.7 with fail-fast validation — new jobs will now fail loudly if CDG/TXT generation fails.
Recovery (for jobs that already completed without CDG/TXT):
# Regenerate and distribute CDG/TXT packages for specific jobs
GCS_BUCKET_NAME=karaoke-gen-storage-nomadkaraoke \
GOOGLE_CLOUD_PROJECT=nomadkaraoke \
python -m scripts.regenerate_cdg JOB_ID [JOB_ID ...]
# Example:
GCS_BUCKET_NAME=karaoke-gen-storage-nomadkaraoke \
GOOGLE_CLOUD_PROJECT=nomadkaraoke \
python -m scripts.regenerate_cdg 5b6aba25 5161b069
The script is idempotent — re-running it skips regeneration if the CDG ZIP already exists in GCS and proceeds to any missing distribution steps (GDrive, Dropbox).
GCE encoding worker on wrong wheel version
Cause: Worker picks up a new deploy but doesn't restart automatically.
Target the currently-serving VM — active_override_vm in config/encoding-worker
if set, else primary_vm (with its zone). There is no VM literally named
encoding-worker; the examples below use the primary encoding-worker-a — swap in
the active VM/zone if a fallback is serving.
# Check current wheel version (example: the primary a in us-central1-c)
gcloud compute ssh encoding-worker-a --zone=us-central1-c --project=nomadkaraoke \
--command="curl -s http://localhost:8080/health | python3 -m json.tool"
# Restart to pick up latest wheel
gcloud compute ssh encoding-worker-a --zone=us-central1-c --project=nomadkaraoke \
--command="sudo systemctl restart encoding-worker"
Render fails: No module named '<pkg>' (e.g. tenacity) on encoding worker
Symptoms: A render fails with OutputGenerator not available: No module named 'tenacity'. The karaoke-gen wheel must be installed. Check that ensure_latest_wheel() succeeded. (or another missing package). Retries hit the same VM and fail identically.
Largely fixed in v0.195.0 (dep-bake): the Packer image now bakes the full karaoke-gen dependency tree (with CPU-only torch) into the venv at build time, so a freshly-provisioned VM of any family boots self-sufficient — no first-job dependency install. This failure mode should now only appear on a VM whose boot disk predates the dep-bake image, or a venv left partial by an interrupted repair. The ensure_latest_wheel() runtime backstop below still exists for those cases.
Original cause (pre-v0.195.0): A freshly-provisioned VM came up with only a handful of packages baked in, and boot installed the wheel with --no-deps. The full dependency tree (torch, langchain, tenacity, …) was installed lazily at the first job by ensure_latest_wheel(); a failed/partial install left the venv importable-but-incomplete and the render died on the first missing import. First seen 2026-08-13 (job 233b9536) on a fresh n2-highcpu-32 fallback added in #903.
Fixed in v0.192.7: ensure_latest_wheel() now downloads only the single latest wheel (was: copying the entire ~6 GiB wheels/ dir every job with a 60 s timeout), retries the install (3× at 900 s), and verifies the render/encode import chain before returning — a clean pip exit is no longer trusted. Callers fail the job with a clear retryable error instead of proceeding. Fresh workers pick this up via the wheel-at-boot.
Also fixed in v0.194.1 (concurrency race): ensure_latest_wheel() runs on the light lane (>1 thread) at the start of every job, so on a fresh boot several jobs would call it at once and race on the shared /tmp wheel path (gsutil cp clobbered mid-copy) and the venv (two pip installs at once) — one job then failed with wheel/dependencies not ready ... retry on a healthy worker (2026-08-15, job 374dec26, 3 jobs on a fresh n2f boot). Now it's guarded by a process-wide lock and short-circuits when the installed version already matches the latest GCS wheel (the common case — startup.sh installs it at boot). So only the first job after a new wheel actually installs; the rest wait briefly then take the verified fast path. A single job failing this way can simply be retried once a worker is warm.
Manual repair (if a running VM is stuck on an old wheel): the venv is root-owned, so use sudo:
# Identify the running worker
gcloud compute instances list --project=nomadkaraoke --filter="name~encoding AND status=RUNNING"
# Repair: install the latest versioned wheel WITH deps, then verify
gcloud compute ssh <worker-name> --zone=<zone> --project=nomadkaraoke --tunnel-through-iap --command='
V=$(gsutil ls "gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-*.whl" | grep -v current | sort -V | tail -1)
WHEEL="/tmp/$(basename "$V")" # keep the PEP 427 filename so pip accepts it
gsutil cp "$V" "$WHEEL"
sudo /opt/encoding-worker/venv/bin/python -m pip install --upgrade "$WHEEL"
/opt/encoding-worker/venv/bin/python -c "import tenacity; from karaoke_gen.lyrics_transcriber.output.generator import OutputGenerator; from karaoke_gen.lyrics_transcriber.correction.operations import CorrectionOperations; from backend.services.local_encoding_service import LocalEncodingService; print(\"OK\")"
'
Re-render a job that already failed past review: a failed job is terminal, and admin /jobs/{id}/reset has no review_complete target. Reset to awaiting_review, then re-submit the review (reuses the existing GCS corrections):
ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke | cut -d',' -f1)
curl -s -X POST "https://api.nomadkaraoke.com/api/admin/jobs/<id>/reset" \
-H "X-Admin-Token: $ADMIN_TOKEN" -H "Content-Type: application/json" \
-d '{"target_state": "awaiting_review"}'
# instrumental_selection is "clean" or "with_backing" (read prior value from Firestore state_data before reset)
curl -s -X POST "https://api.nomadkaraoke.com/api/review/<id>/complete?review_token=<token>" \
-H "Content-Type: application/json" -d '{"instrumental_selection": "with_backing"}'
Do not reset to instrumental_selected — that triggers the final video worker, which needs the (missing) lyrics video and fails "prerequisites not met"; and triggering render-video from instrumental_selected gets superseded (invalid instrumental_selected → rendering_video).
GDrive validator reports sequence gap
Symptoms: Email from "Nomad Karaoke GDrive Validator" reporting SEQUENCE GAPS: MP4: missing XXXX.
Key principle: Never add to KNOWN_GAPS. All known gaps are historical (pre-generator, 2024). Every new gap is a real bug.
Full investigation and fix procedure: See docs/GDRIVE-VALIDATOR.md § Sequence Gap Detected.
Quick reference:
- Query Firestore for the missing brand code (
state_data.brand_code == 'NOMAD-XXXX') - If no job found, check Cloud Run logs for
"Allocated brand code: NOMAD-XXXX"to find when/how it was consumed - Determine root cause from logs (job re-trigger, failed distribution, etc.)
- Recycle the brand code number into
brand_code_counters/NOMAD.recycled— the next public job will fill the gap - Report any orphan GDrive files to the user for manual cleanup
Held completion email (Dropbox delivery failed)
Signal: #alerts "🔴 Customer delivery HELD — Dropbox upload failed" (from
backend/services/delivery_guard.py), or "🔴 Dropbox almost FULL" from the hourly quota check
(rides on /api/internal/process-stale-reviews; fires once at ≥95% used and re-arms after usage
drops back below).
What happened: the job was meant to deliver to Dropbox (dropbox_path + brand_prefix on its
effective distribution settings) but finished with no state_data.dropbox_link. Instead of emailing
the customer a delivery email with no files link (NOMAD-1800), the job went COMPLETE, the email
was held, and state_data.completion_email_held = {reason_code, reason, recipient, at} was set.
For made-for-you orders ownership still transfers to the customer. reason_code is
dropbox_full (insufficient_space), dropbox_upload_failed, or dropbox_link_missing.
Recovery (manual):
- If
dropbox_full: free space in the business Dropbox (rclone aboutshows usage) or raise the quota. - Upload the job's outputs (GCS
jobs/{job_id}/finals/,stems/,lyrics/) to its Dropbox folder{dropbox_path}/{brand_code} - {Artist} - {Title}, create a share link, and setstate_data.dropbox_linkon the job. (An admin re-render also re-uploads to Dropbox, but it deletes and re-uploads the YouTube video too, so the YouTube URL changes: only use it for private jobs.) POST /api/admin/jobs/{job_id}/send-completion-emailwith the recipient fromcompletion_email_held.recipient. It returns 409 while the link is still missing (allow_missing_dropbox_link: trueoverrides). A successful send clears the held marker.
Google Drive uploads missing (gdrive_files empty)
Symptoms: Jobs complete successfully but state_data.gdrive_files is empty {}. Cloud Run logs show BrokenPipeError or SSL: UNEXPECTED_EOF_WHILE_READING.
Cause: Stale HTTP connections in the singleton GoogleDriveService. Since v0.119.7, this is handled automatically with retry + connection reset.
Backfill affected jobs:
# Dry run - list affected jobs
GCS_BUCKET_NAME=karaoke-gen-storage-nomadkaraoke \
GOOGLE_CLOUD_PROJECT=nomadkaraoke \
python scripts/backfill_gdrive_uploads.py --all-missing --dry-run
# Run for real
GCS_BUCKET_NAME=karaoke-gen-storage-nomadkaraoke \
GOOGLE_CLOUD_PROJECT=nomadkaraoke \
python scripts/backfill_gdrive_uploads.py --all-missing
# Specific jobs
python scripts/backfill_gdrive_uploads.py --job-ids JOB1,JOB2
Job failed: "CDG generation was enabled but failed"
Cause: The LRC file has no lyrics content (just metadata like [re:MidiCo]). This happens when AudioShake returns 0 segments — typically because the input audio has no vocals (e.g. user uploaded a karaoke track).
Fix (v0.135+): The video orchestrator now skips CDG/TXT gracefully when the LRC has no content. The complete_review endpoint also blocks 0-segment submissions with a user-friendly error, and the frontend shows a warning with guidance.
For older jobs: Retry won't help unless the user provides different audio or pastes lyrics manually via "Replace All".
Review waveform 500 / preview encode fails: "Could not find codec parameters for stream 0 (Audio: flac, 0 channels)"
Symptoms (all at once, for one job): waveform-data returns 500 ("Decoding failed. ffmpeg returned error code: 8"), review OGG transcode falls back to FLAC ("Cannot determine format of input 0:0 after EOF"), preview video fails on both GCE and local fallback ("Invalid data found when processing input"), and "Failed to get audio duration: 'duration'" appears in logs. The pipeline itself completed fine (stems/lyrics exist).
Cause: The uploaded input audio is a valid MP3 with a very large ID3 tag (megabytes of embedded cover art — common in ytmp3-site rips; a majorbrand: dash ID3 tag is a telltale, but it's just copied metadata, not a mislabeled MP4). The backend downloaded job audio to temp files hardcoded as *.flac; ffmpeg's content probe is inconclusive inside the huge tag, so it falls back to the file extension, commits to the flac demuxer, and fails. The file itself is fine — ffprobe succeeds when the local copy keeps its real extension.
Fix (v0.229.4+): All temp downloads of job audio now preserve the source object's extension (backend/utils/audio_filenames.local_audio_filename). No job data repair is needed — once deployed, the same job's review page works as-is.
Diagnosis tip: Download the input object and run ffprobe on it twice — once named .mp3/no extension, once named .flac. If only the .flac-named copy fails, it's this extension-fallback class, not a corrupt upload.
Job failed: "Audio separation failed: expected str, bytes or os.PathLike object, not NoneType"
Cause (historical): Modal API intermittently returned fewer stems than expected from stage 2 (backing vocals) separation. Missing stems caused NoneType crashes in downstream processing. This was resolved by migrating to Cloud Run GPU (see docs/archive/2026-03-22-modal-to-gcp-migration-plan.md).
Fix (v0.135+): Stage 2 now validates that all expected stems are present. Defensive null checks prevent confusing TypeError messages. Full tracebacks are logged with exc_info=True.
Recovery: Retry the job. With the Cloud Run GPU deployment, transient API failures are much less frequent than they were with Modal.
Job failed: "Lyrics transcription failed: 502 Server Error: Bad Gateway for url: https://api.audioshake.ai/tasks"
Cause: AudioShake API returned a transient 5xx error.
Fix (v0.135+): AudioShake API calls now retry up to 5 times with exponential backoff (60s base, 3x multiplier, ~40 minutes total spread). Retries on 5xx, 429, connection errors, and timeouts.
For older jobs: Simply retry the job via admin dashboard.
Encoding worker not starting
Symptoms: Encoding jobs time out waiting for the VM to respond. The /api/internal/encoding-worker/start call returns success but the VM never accepts connections.
Check Firestore config exists:
python3 -c "
import os; os.environ['GOOGLE_CLOUD_PROJECT']='nomadkaraoke'
from google.cloud import firestore
doc = firestore.Client(project='nomadkaraoke').collection('config').document('encoding-worker').get()
print(doc.to_dict() if doc.exists else 'MISSING - run seed script')
"
Check VM status:
gcloud compute instances describe encoding-worker-a \
--zone=us-central1-c --project=nomadkaraoke --format='value(status)'
Check backend service account permissions: The backend Cloud Run SA needs compute.instances.start on the encoding worker VMs. Verify in IAM or check Cloud Run logs for permission denied errors when the start call is made.
Every render/preview fails with CERTIFICATE_VERIFY_FAILED on metadata.google.internal
Symptoms: Jobs park in render_pending_capacity with last_code: worker_infra_failure and empty last_vm/last_zone, and retry every 10 min without recovering. Backend logs show Worker reported an infrastructure/auth failure with Failed to retrieve https://metadata.google.internal/... / SSLCertVerificationError. The worker still answers /health.
Cause (2026-10-06): the VM booted before its network was up. bootstrap.sh couldn't download startup.sh from GCS and used the image's startup-fallback.sh, which (on images built before v0.266.4) writes an env file without ENCODING_API_KEY or GCE_METADATA_MTLS_MODE=none. google-auth then uses mTLS to the metadata server and fails.
Diagnose: worker app logs are not in Cloud Logging; read the serial console (journald goes there):
gcloud compute instances get-serial-port-output encoding-worker-b --zone=us-central1-c --project=nomadkaraoke \
| grep -E "Bootstrap|FALLBACK|Network is unreachable|No API key configured|SSLCertVerificationError" | tail -20
No API key configured - authentication disabled on every request is the tell.
Fix: since v0.266.4 the worker exits on its own (boot check + post-failure credential probe) and systemd re-runs the bootstrap. If that isn't happening, restart it by hand. Network is up by then, so the real startup.sh runs:
gcloud compute ssh encoding-worker-b --zone=us-central1-c --project=nomadkaraoke \
--command='sudo systemctl restart encoding-worker'
(Stale host key error → ssh-keygen -R compute.<instance_id> -f ~/.ssh/google_compute_known_hosts.) Parked jobs pick up on the next scheduler tick, or force it with retry-pending-render-jobs (see render_pending_capacity below).
Render fails "Cannot connect to host :8080" and the VM never answers /health
Symptoms: a render that cold-started the primary VM retries 8 times (~90s) and fails with ClientConnectorError ... Connect call failed. The backend keeps logging encoding-worker-b /health unconfirmed; keeping alive (fail-safe) until the VM idle-stops.
Cause (2026-10-09 f4310aea, 2026-10-12 82f48738): startup.sh's first gcloud call ran before DHCP, so gcloud cached "not on GCE" (/root/.config/gcloud/gce = False). Every later gcloud call in that boot had no credentials: no API key, the self-heal refuses to serve, systemd re-runs startup.sh, and it hits the same cache forever.
Diagnose: the serial log shows repeated WARNING: No API key found in Secret Manager, and on the VM sudo cat /root/.config/gcloud/gce prints False.
Fix: since v0.268.3 startup.sh deletes that cache, waits for the metadata server and retries the secret fetch, so the next restart recovers by itself. By hand: sudo rm -f /root/.config/gcloud/gce && sudo systemctl restart encoding-worker.
Job stuck at rendering_video (orphaned render)
Symptoms: Job frozen at rendering_video (step 7/10) for a long time with no error and no Retry button (it's a processing state, not failed).
Cause: The render worker started (VM up, rendering underway) then died mid-render — Cloud Run instance recycle / SIGKILL / OOM / lost GCE VM — without unregistering. updated_at stops advancing. Before v0.192.3 nothing recovered this: the capacity-retry cron only looks at render_pending_capacity, recover-stuck-jobs only looked at downloading_audio, and the render Cloud Task has finite attempts.
Auto-recovery (v0.192.3+): recover-stuck-jobs (every 10 min) now flags a job in rendering_video with no progress for >45 min (rendering_video_stuck) and re-parks it into render_pending_capacity, so the existing retry-pending-render-jobs cron resets it to review_complete and re-renders (same 24h ceiling). No manual action needed.
Root-cause fix (v0.224.0+): the dominant cause of this state was backend deploys: the render worker ran as a FastAPI BackgroundTask on the Cloud Run service, and every deploy rollout that landed mid-render SIGKILLed it ~10s after SIGTERM (the lifespan "park on shutdown" hook never runs in that scenario — uvicorn waits for background tasks before running lifespan shutdown, so it dies waiting; incident 2026-09-13, job 41e06b90, tenant-E2E run #107). Since v0.224.0 the render worker runs as a Cloud Run Job (video-encoding-job with a render_video_worker args override, flag USE_CLOUD_RUN_JOBS_FOR_RENDER), which runs to completion regardless of service deploys — same fix the video worker got in 2026-03 (USE_CLOUD_RUN_JOBS_FOR_VIDEO). The 45-min sweep remains as the safety net for the remaining causes (job execution crash/OOM/timeout, lost GCE VM). Rollback: remove USE_CLOUD_RUN_JOBS_FOR_RENDER=true from the service env to fall back to the legacy Cloud Tasks path.
Manual unstick (if needed): don't re-trigger the render worker while status is rendering_video (it starts mid-state and fails with "Invalid state transition"). Instead let it fail (or re-park it), then use the admin Retry which resumes cleanly from review_complete. If a ghost render_progress.stage='running' blocks re-trigger, clear it first (update({'state_data.render_progress': {'stage': 'pending'}})).
Job flips to failed right after an admin reset (reset/render race)
Symptoms: An operator hit an admin Reset button (e.g. "Review") on a job that was rendering_video, and moments later the job went to failed with a message like "Video render failed: Invalid state transition for job …: in_review -> instrumental_selected". The reset itself looked fine, but the job died on its own.
Cause (fixed in v0.192.4): The reset moved the job backwards while the render worker was still running on the encoder. When that render finished it attempted its normal terminal transition (rendering_video -> instrumental_selected), which was now illegal, and the worker's generic error handler flipped the job to failed — clobbering the operator's reset. First seen on job 7f457087.
Auto-handling (v0.192.4+): Long-running render/video workers are now fenced against supersession — a reset or a newer trigger causes the in-flight worker to discard its stale result quietly instead of failing the job. Two nets: a status fence (job no longer in the status the worker owns) and a generation fence (state_data.worker_generation, bumped by every reset and every render/video trigger). An InvalidStateTransitionError at the terminal step is treated as supersession, not failure. So resetting a rendering job is now safe at any timing — no manual action needed.
If you still see a failed job from an older occurrence: use the admin Retry endpoint. For a job that has corrections + screens but no committed instrumental_selection, it returns cleanly to awaiting_review so the review (and instrumental choice) can be re-submitted.
Job in download_pending_retry
Symptoms: Job status is download_pending_retry (introduced v0.192.3), step 3/10, message "Still finding a good source for this track — … We'll keep trying automatically for up to 24 hours; no action needed."
Background: A torrent (RED/OPS) download stalled (few/no seeders or a brief tracker outage). Rather than dead-end the job at failed after ~1h, recover-stuck-jobs parks it here and re-triggers the download on later ticks for up to 24h, then fails permanently with a "too rare to source right now" message. This is expected, self-healing behaviour — a seedless release simply may never complete. state_data.download_retry holds {first_seen_at, attempt_count}. To source it a different way, retry with an alternate release/source.
Job stuck in render_pending_capacity
Symptoms: Job status is render_pending_capacity (introduced 2026-05-05). User-facing message says "Encoding capacity is temporarily unavailable. Your job will retry automatically — no action needed." (As of v0.192.3 a mid-render stall can also land here — see "orphaned render" above.)
Background — what this state means: GCE returned ZONE_RESOURCE_POOL_EXHAUSTED or transient 503 SERVICE_UNAVAILABLE from compute.instances.start on every encoding-worker VM. The render worker parks the job in RENDER_PENDING_CAPACITY instead of failing it; Cloud Scheduler retries it every 10 min via /api/internal/retry-pending-render-jobs. Hard timeout is 24 hours (then transitions to failed with a clear permanent-failure message).
Since 2026-09-26 the fleet is 3 VMs: Spot c4d-highcpu-16 primaries (encoding-worker-a/-b) in us-central1-c, plus ONE stopped on-demand fallback encoding-worker-fallback-c2df (c2d-highcpu-16, us-central1-f). Candidates are tried fastest-first with a 15-min capacity cooldown — see backend/services/encoding_worker_preference.py. If both the Spot pair and the c2d fallback return stockout, that's a genuinely severe regional event — wait, or (re-)add a fallback in another family/zone to EncodingWorkerConfig.FALLBACKS and the encoding-worker-fallback-vms secret (the 2026-08-15 pool had 8 fallbacks across 6 families). NOTE: creating a new fallback VM needs a one-time boot allocation, so a deep enough crunch can even block provisioning.
Spot preemption mid-encode is expected occasionally: the backend sees the pinned worker VM is TERMINATED/STOPPING, logs Encoding worker VM stopped mid-job (... ) — likely Spot preemption, and resubmits the encode as <job>_retry_<hex> (up to ENCODING_RESUBMIT_MAX, default 2). Repeated preemptions → the job fails with the lost-job error; check gcloud compute operations list --filter="operationType=compute.instances.preempted".
Related symptom — preview 524 / "NetworkError": the same capacity exhaustion also makes POST /api/review/{id}/preview-video fall back to slow local encoding (~130–160 s), which exceeds Cloudflare's ~100 s edge timeout → the browser shows a 524 or Firefox "NetworkError when attempting to fetch resource" even though the backend returns 200. If a user reports the review preview failing, check for a concurrent stockout; the already-rendered preview mp4 (if any) is fetchable via GET /api/review/{id}/preview-video/{hash} (302 → signed GCS URL) with a Bearer admin token. Completing the review does not require the preview.
Check how many retries have run:
python3 -c "
import os; os.environ['GOOGLE_CLOUD_PROJECT']='nomadkaraoke'
from google.cloud import firestore
d = firestore.Client(project='nomadkaraoke').collection('jobs').document('JOB_ID').get().to_dict()
print((d.get('state_data') or {}).get('render_pending_capacity'))
"
Force an immediate retry (Cloud Scheduler runs every 10 min, but you can poke it manually):
ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke | cut -d',' -f1)
curl -X POST https://api.nomadkaraoke.com/api/internal/retry-pending-render-jobs \
-H "X-Admin-Token: $ADMIN_TOKEN"
Verify GCE has any capacity right now (the real error reason is in the Compute op, not the app's "503"):
# See the actual stockout reason (ZONE_RESOURCE_POOL_EXHAUSTED / stockout) + vmType
gcloud logging read 'protoPayload.methodName:"instances.start" AND "ZONE_RESOURCE_POOL_EXHAUSTED"' \
--project=nomadkaraoke --freshness=30m --limit=5 \
--format='value(timestamp, protoPayload.resourceName)'
# Try starting each worker; if ALL refuse, it's severe — wait.
# Synchronous (no --async) so the start op surfaces stockout inline, and we capture
# gcloud's own exit status rather than piping (a pipe would return sed's status).
for VM_ZONE in \
"encoding-worker-a:us-central1-c" \
"encoding-worker-b:us-central1-c" \
"encoding-worker-fallback-c2df:us-central1-f"; do
VM="${VM_ZONE%%:*}"; ZONE="${VM_ZONE##*:}"
if OUT=$(gcloud compute instances start "$VM" --zone="$ZONE" --project=nomadkaraoke 2>&1); then
echo "$VM: OK (started — remember to stop it)"
else
echo "$VM: FAILED — $(echo "$OUT" | tr '\n' ' ')"
fi
done
Configuration knobs: MAX_PER_TICK = 1 and MAX_WAIT_SECONDS = 24*3600 constants in backend/api/routes/internal.py::retry_pending_render_jobs.
Not an incident: a primary warmup hitting 503 SERVICE_UNAVAILABLE / capacity exhaustion is self-healing — the encoding flow's ensure_any_running() falls back to an alternate-zone worker and the job completes. As of 0.174.12 the /internal/encoding-worker/warmup/{job_id} endpoint logs this at WARNING ("Primary encoding worker warmup failed (...); encoding will fall back..."), specifically so it stays below the error monitor's severity>=ERROR filter and does not page Discord. Only a genuine, unexpected warmup failure (non-EncodingWorkerStartError) logs at ERROR. If you see the WARNING in logs with a job that still completed, no action is needed.
403 on /internal/encoding-worker/warmup or /heartbeat: as of 0.188.9 these endpoints are {job_id}-scoped and gated on require_review_auth (not require_admin). The lyrics-review page calls them on load / while editing so the encoding VM is warm by the time the reviewer previews a render. If you see 403s in the browser console, the caller lacks review access to that job — before 0.188.9 they required admin, so every non-admin customer 403'd and the JIT pre-warm never fired (renders paid the full cold-boot latency).
Job failed: "Video render failed: TimeoutError()"
Background: This was the original signature of the 2026-05-05 capacity bug — empty exception message because f"...{e}" produced empty string for bare TimeoutError(). As of 0.174.x the empty-message version should not happen (we use repr(e) as fallback). If you see this exact form on a job after 2026-05-06, the render worker hit a TimeoutError not preceded by a typed EncodingWorkerStartError — most likely the URL re-resolve isn't engaging or a Cloud Run instance was killed mid-flight.
Check what the render actually saw:
gcloud logging read \
"jsonPayload.message=~\"JOB_ID\" AND timestamp>=\"YYYY-MM-DDTHH:MM:SSZ\"" \
--project=nomadkaraoke --limit=50 --format='value(timestamp,jsonPayload.message)'
Look for these markers:
Submitting render-video job to GCE worker: http://X.X.X.X:8080/render-video— initial URLTrying candidate N/3: VM ... in us-central1-X— multi-zone iteration startingSet active_override to ...— fallback engagedRe-resolved encoding URL after warmup: A -> B— added in 0.174.3, confirms the URL is updating between retriesWORKER_END worker=render-video status=pending_capacity— happy parking pathWORKER_END worker=render-video status=error— failed instead of parked (bug)
Most likely cause if no Re-resolved log appeared: The job was rendered on a pre-0.174.3 revision. Just retry — the new revision is now serving.
Recovery: retry the job via the admin endpoint or /retry-pending-render-jobs. The system is self-healing as long as at least one zone has capacity.
Job stuck in rendering_video for hours (no failure log)
Symptoms: Status frozen at rendering_video (progress 75%) for >30 minutes. No WORKER_END worker=render-video log entry exists. Last log is mid-retry like "GCE worker connection failed (attempt 7/8)".
Cause: The Cloud Run instance handling the render task was killed (autoscaler scaled down, OOM, or revision rollover) before the retry loop completed and could write a failure state.
Mitigation in place since v0.174.4: The FastAPI shutdown hook now waits up to 480s for the render worker to finish, then parks any still-active render jobs in RENDER_PENDING_CAPACITY (last_code: WORKER_SHUTDOWN) before exit. The /api/internal/retry-pending-render-jobs Cloud Scheduler job picks them up within 5 minutes. Jobs should now self-recover. If you see WORKER_END worker=render-video status=shutdown_parked in logs, that's the new graceful-shutdown path.
Recovery (only if a job somehow still gets stuck — e.g. SIGKILL hit before park completed): manually transition to review_complete and re-trigger the render. The instrumental selection in state_data is preserved so no user re-work is needed:
ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke | cut -d',' -f1)
JID=YOUR_JOB_ID
python3 << EOF
import os; os.environ['GOOGLE_CLOUD_PROJECT']='nomadkaraoke'
from google.cloud import firestore
from datetime import datetime, UTC
db = firestore.Client(project='nomadkaraoke')
now = datetime.now(UTC)
db.collection('jobs').document('$JID').update({
'status': 'review_complete', 'progress': 70, 'updated_at': now,
'error_message': None, 'error_details': None,
'state_data.render_progress': {'stage': 'pending'},
})
EOF
curl -X POST https://api.nomadkaraoke.com/api/internal/workers/render-video \
-H "X-Admin-Token: $ADMIN_TOKEN" -H "Content-Type: application/json" \
-d "{\"job_id\":\"$JID\"}"
Deploy stuck in progress (deploy_in_progress: true)
Symptoms: CI deploys fail immediately with "deploy already in progress". The flag in Firestore was never cleared after a previous failed deploy.
Auto-clear: The Cloud Function that manages idle shutdown also clears stale deploy_in_progress flags after 20 minutes. Wait for the next Cloud Scheduler tick (every 5 minutes) and the function will clear it.
Manual clear:
python3 << 'EOF'
import os; os.environ['GOOGLE_CLOUD_PROJECT']='nomadkaraoke'
from google.cloud import firestore
firestore.Client(project='nomadkaraoke').collection('config').document('encoding-worker').update({
'deploy_in_progress': False
})
print("Cleared")
EOF
Both VMs running unexpectedly
Symptoms: GCP billing alert, or you notice both encoding-worker-a and encoding-worker-b are RUNNING.
Check Cloud Function logs:
gcloud logging read 'resource.type="cloud_function" resource.labels.function_name="encoding-worker-idle-shutdown"' \
--project=nomadkaraoke --limit=20 --format="table(timestamp,textPayload)"
Verify scheduler is firing: Go to Cloud Scheduler console and check the encoding-worker-idle-shutdown job's last execution time and status.
Check activity timestamps in Firestore: Look at config/encoding-worker — the activity_a and activity_b fields show when each VM last reported activity. If a timestamp is very recent, that VM is actively being used (or the heartbeat is stuck).
Force idle check:
gcloud scheduler jobs run encoding-worker-idle-shutdown \
--location=us-central1 --project=nomadkaraoke
Encoding requests failing after deploy
Symptoms: Jobs fail at encoding immediately after a blue-green deploy swap. Logs show connection refused or 404 to the old primary IP.
Cause: The backend caches the encoding worker URL for 30 seconds. Requests in-flight during the swap may hit the old (now stopped) primary.
Fix: Wait 30 seconds after the swap for the cache to expire. In-flight jobs can be retried via the admin dashboard — re-triggering the video worker will pick up the new primary URL.
Verify the swap happened:
python3 -c "
import os; os.environ['GOOGLE_CLOUD_PROJECT']='nomadkaraoke'
from google.cloud import firestore
doc = firestore.Client(project='nomadkaraoke').collection('config').document('encoding-worker').get()
print('primary:', doc.to_dict().get('primary'))
"
Verifying GCE encoding worker cold-start fix
Background: The encoding worker VM auto-stops when idle to save cost. When a render request hits a TERMINATED VM, the backend awaits a readiness gate (EncodingWorkerManager.wait_for_worker_ready) instead of relying on the deploy-restart-sized 90s retry window. This was added in v0.172.3 to fix the 2026-04-24 incident (job 2c577535) where a cold VM exceeded the retry budget.
To verify in production after a deploy:
-
Get the primary VM name from Firestore config:
python3 -c " import os; os.environ['GOOGLE_CLOUD_PROJECT']='nomadkaraoke' from google.cloud import firestore db = firestore.Client(project='nomadkaraoke') print(db.collection('config').document('encoding-worker').get().to_dict()['primary_vm']) " -
Stop it:
gcloud compute instances stop <vm-name> --zone=us-central1-c --project=nomadkaraoke -
Submit a test render through the dashboard, or trigger one with a known review-complete job.
-
Watch the backend logs for the readiness-wait sequence:
gcloud logging read 'resource.labels.service_name="karaoke-backend" AND textPayload=~"Waiting for VM|Worker at .* is healthy|Cold-started VM"' \ --project=nomadkaraoke --freshness=10m --order=asc \ --format='value(timestamp,textPayload)'Expected sequence:
Encoding worker unreachable — started VM <name> as fallbackWaiting for VM <name> (status=STAGING)(every 30s during VM boot)VM <name> reached RUNNINGWaiting for worker /health at ...(every 30s during service init)Worker at .../health is healthyCold-started VM <name> is now ready- Job proceeds to encoding without retry exhaustion.
-
If the readiness wait times out (~5 min: 120s waiting for VM RUNNING, then 180s waiting for /health), the log will show
Cold-start readiness wait timed out: .... The retry loop then continues with attempts 1-7 (~90s of backoff: 5+10+15+15+15+15+15), so the worst-case end-to-end before the job fails is ~6.5 minutes. Investigate VM boot:gcloud compute instances get-serial-port-output <vm-name> --zone=us-central1-c --project=nomadkaraoke | tail -100 gcloud compute ssh <vm-name> --zone=us-central1-c --project=nomadkaraoke --command="sudo systemctl status encoding-worker"
Surgical job repair: re-run audio separation without losing lyrics/review
When to use: Jobs where audio separation failed or produced incomplete stems, but the user has already completed lyrics review. The admin /restart endpoint clears ALL state including lyrics — use direct Firestore updates instead.
Approach: Clear only audio-related state, trigger the Cloud Run GPU audio job, then advance the job to the correct next step.
import os
os.environ["GOOGLE_CLOUD_PROJECT"] = "nomadkaraoke"
from google.cloud import firestore, run_v2
from google.cloud.firestore_v1 import DELETE_FIELD, ArrayUnion
from datetime import datetime, timezone
db = firestore.Client(project="nomadkaraoke")
job_id = "YOUR_JOB_ID"
# Step 1: Clear audio state only (preserves lyrics, review, instrumental selection)
job_ref = db.collection("jobs").document(job_id)
job_ref.update({
"status": "downloading", # Required: idempotency checks reject terminal states
"state_data.audio_progress": DELETE_FIELD,
"state_data.audio_complete": False,
"state_data.instrumental_options": DELETE_FIELD,
"state_data.backing_vocals_analysis": DELETE_FIELD,
"file_urls.stems": DELETE_FIELD,
"error_message": DELETE_FIELD,
"error_details": DELETE_FIELD,
"progress": 15,
"timeline": ArrayUnion([{
"status": "downloading",
"timestamp": datetime.now(timezone.utc).isoformat(),
"message": "Admin repair: re-running audio separation",
}]),
})
# Step 2: Trigger Cloud Run GPU audio job directly
# (bypasses internal HTTP endpoint which has idempotency checks)
client = run_v2.JobsClient()
client.run_job(request=run_v2.RunJobRequest(
name="projects/nomadkaraoke/locations/us-east4/jobs/audio-separation-job",
overrides=run_v2.RunJobRequest.Overrides(
container_overrides=[run_v2.RunJobRequest.Overrides.ContainerOverride(
args=["python", "-m", "backend.workers.audio_worker", "--job-id", job_id],
)]
)
))
# Step 3: After audio completes, advance the job based on its previous state:
# - If job was "failed" with lyrics: trigger screens worker → awaiting_review
# - If job was "instrumental_selected": set status back, trigger video worker
Key gotchas:
- Must set
statusto a non-terminal state ("downloading") before triggering workers mark_audio_complete()only sets a flag — it does NOT trigger the next pipeline step- After audio completes, you must manually trigger screens (for review path) or video (for post-review path)
- The admin
/restartendpoint withpreserve_audio_stems=falseis the simpler option when lyrics don't need preserving
Triggering Cloud Run Jobs directly (bypassing internal API)
When to use: The internal worker endpoints (/api/internal/workers/audio, etc.) have idempotency checks that reject jobs in terminal states (failed, complete, etc.). When repairing jobs, trigger Cloud Run Jobs directly via the API.
from google.cloud import run_v2
client = run_v2.JobsClient()
client.run_job(request=run_v2.RunJobRequest(
name="projects/nomadkaraoke/locations/{REGION}/jobs/{JOB_NAME}",
overrides=run_v2.RunJobRequest.Overrides(
container_overrides=[run_v2.RunJobRequest.Overrides.ContainerOverride(
args=["python", "-m", "backend.workers.{MODULE}", "--job-id", "{JOB_ID}"],
)]
)
))
Available Cloud Run Jobs:
| Job Name | Region | Module | GPU |
|---|---|---|---|
audio-separation-job |
us-east4 |
audio_worker |
L4 |
audio-download-job |
us-east4 |
audio_download_worker |
No |
lyrics-transcription-job |
us-east4 |
lyrics_worker |
No |
bulk-search-job |
us-east4 |
bulk_search_worker |
No |
video-encoding-job |
us-east4 |
video_worker / render_video_worker |
No |
The CPU jobs above run in us-east4 (CPU_JOBS_REGION) even though everything else lives in
us-central1. See "Cloud Run Jobs take 2-5 minutes to start" below.
Cloud Run Jobs take 2-5 minutes to start
Symptoms: A job execution sits between "created" and its first Python log line for 2-5 minutes.
gcloud run jobs executions describe <exec> shows Started deployed execution in 3m.... Concurrent
executions often start in the same second, as if a queue drained.
Cause: Google-side scheduling of Cloud Run Jobs in us-central1. It doesn't depend on our
image, CPU or code. Measured 2026-09-28 with Google's tiny sample job image
(us-docker.pkg.dev/cloudrun/container/job) on 1 CPU: us-central1 took 4-5 min, while us-east4
and us-east1 took 8-25s. decide's 203 MB image on 4 CPU was just as slow in us-central1, so
slimming our image or adding CPU doesn't help. Quotas were nowhere near their limits.
Fix in place: the latency-critical CPU jobs (audio-download-job, lyrics-transcription-job,
bulk-search-job, video-encoding-job) run in us-east4. The backend setting is CPU_JOBS_REGION (default us-east4),
and the Pulumi constant CPU_JOBS_REGION must match it (a unit test checks this). CI pushes a copy of
the CPU image to us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-cpu so the
jobs pull from their own region. Inside the jobs, GCP_REGION stays us-central1. A new image digest takes a
one-time ~2.5 min "Container image import" on its first execution in a region, so after each
deploy CI starts one no-op audio-download-job execution (python -c pass) to do that import.
video-encoding-job (video + post-review render) moved in v0.249.1. It reads about 0.3 GB per run
from the us-central1 bucket, which is roughly 156 GB/month cross-region: about $1.60/month after
GCS's 100 GB/month free inter-region tier, or $3/month worst case. Its writes to GCS and its
internet uploads cost the same from either region. The GCE encoding VMs stay in us-central1; they
have public IPs (port 8080, API key), so the job reaches them from us-east4.
Measure:
for e in $(gcloud run jobs executions list --job audio-download-job --region us-east4 \
--project nomadkaraoke --limit 8 --format='value(name)'); do
gcloud run jobs executions describe $e --region us-east4 --project nomadkaraoke --format=json \
| jq -r '[.metadata.creationTimestamp, (.status.conditions[]|select(.type=="Started")|.message)]|@tsv'
done
Rollback to us-central1. The legacy us-central1 copies were removed on 2026-09-30, after the us-east4 jobs were verified in prod. To move back:
- Add the us-central1 jobs in Pulumi alongside the us-east4 ones: in
infrastructure/__main__.py, addcreate_*_job(bucket, backend_service_account)calls with noregion=argument, so they use the default region and thekaraoke-repoimage. Runpulumi up. Don't change theCPU_JOBS_REGIONconstant yet, because that would replace the us-east4 jobs the live backend is still using. - Switch the backend with the env var, not the constants. Add
CPU_JOBS_REGION=us-central1to the service's--set-env-varsinci.yml, and point the CI "Update Cloud Run Jobs" step at us-central1 andkaraoke-repo. Changing the Pulumi constant now would re-point the us-east4 declarations onto the same logical names as the step-1 jobs. Deploy, and confirm new executions land in us-central1. The download/bulk-search/video jobs trigger other jobs, so add the same env var to the us-central1 job definitions from step 1. - Remove the us-east4 job declarations in Pulumi, then set both
CPU_JOBS_REGIONdefaults (backend/config.pyandinfrastructure/modules/cloud_run.py; a unit test enforces that they match) tous-central1. Runpulumi up. Updatescripts/rollback.shto use us-central1 and thekaraoke-repoimage.
A rollback of the backend service to before v0.247.1 also needs step 1 first, because older revisions trigger the jobs in us-central1.
Frontend error alert ("New Error Pattern Detected", service frontend)
The Discord sample lists, in order: message → At: location (only when the browser supplied no stack) → URL / UA / Build → Diag: (page age, DOM/canvas/audio counts, live blob URLs, JS heap on Chromium, reports_this_page) → Trail: (last 10 breadcrumbs: navigation, clicks, upload phases, errors) → trimmed stack.
For the full payload (all 30 breadcrumbs and every diagnostics field), query the ingest log:
gcloud logging read 'jsonPayload.message:"frontend_crash_reported" AND jsonPayload.error_pattern_id="<pattern_id>"' \
--limit=5 --format=json | jq '.[].jsonPayload | {client_url, client_release, client_extra}'
synthetic_error: true means the browser gave no Error object (e.g. Firefox out of memory), so there's no real stack. Use the trail plus page_age_s / js_heap_used_mb / live_blob_urls to see what the tab was doing. A very high page_age_s with no obvious trigger in the trail points at a long-lived tab or browser leak, not a specific action.