Files
karaoke-gen/backend/workers/youtube_queue_processor.py
T
Andrew Beveridge ☄️andClaude Opus 5.5 f2612ac110 fix: close edit→delete refund loop, retry lost YouTube uploads, unblock event loop (v0.268.1) (#1135)
## Summary
Investigation of two 2026-10-08 Discord alerts (plus a third that
arrived mid-session) turned up four issues:

1. **Credit-refund loop.** A brand-new welcome-credit account received a
finished video (job 8a10a22d), clicked **Edit** (status back to
`awaiting_review`), then **deleted** the job and got its credit back.
`delete_job` refunds any non-complete status, and Edit makes a delivered
job look "never completed".
2. **YouTube 410 "Gone" dropped the upload.** The resumable session was
lost 1.3s into the upload. `upload_video` called `execute()` once with
no retries, and the orchestrator only queued quota errors, so any other
failure lost the upload.
3. **EVENT_LOOP_STALL 21s in `upload_json`.** `complete_review` (an
async route) wrote the corrections JSON to GCS directly on the event
loop. Ranking a week of watchdog stacks by route showed the same pattern
in `recover_stuck_jobs` (~260 stalls/week), the YouTube queue processor
(its `_process_single_upload` ran a whole upload on the loop),
`create_job_from_search` (theme prep), `_validate_audio_durations`
(15s), and the sync `run_job` / generation bump inside worker triggers.
4. **False 30s "stall" in asyncio `get_debug`.** Cloud Run CPU
throttling froze the whole instance mid-callback with no request in
flight. The watchdog only discounted freezes where the loop was parked
in `select()`.

## Changes
- **Refund:** `_refund_credit_for_job` skips jobs with `edit_count > 0`
(already delivered). This covers delete, cancel, fail and the 48h review
auto-expiry.
- **YouTube:** on 404/410 the upload restarts with a fresh session (3
attempts, with backoff). `execute(num_retries=3)` lets googleapiclient
resume chunks on 5xx/429 within a session. Any other failure is queued
in `youtube_upload_queue` (`reason="upload_error"`) for the existing
processor, which retries up to 5 times. Admins still get a Discord
warning: "queued for automatic retry".
- **Event loop:**
- New `backend/utils/loop_bridge.py`: `run_in_body_thread` runs sync
route bodies on a dedicated 16-thread pool, so they can't exhaust the
default executor the dispatched coroutines need. `run_on_loop`
dispatches coroutines back to the API loop (`run_coroutine_threadsafe`,
120s timeout).
- `complete_review`: its sync save runs via `to_thread`, and the render
trigger stays on the loop.
- `create_job_from_search`, `recover_stuck_jobs` and
`retry_pending_render_jobs`: bodies run in the route-body pool.
`_retrigger_parked_download` now does its writes in that thread.
- `_validate_audio_durations` and the YouTube queue's
`_process_single_upload` keep their async signatures but run in threads.
The queue's quota and Firestore calls go through `to_thread`.
- `worker_service`: the generation bump and `client.run_job` go through
`to_thread`.
- **Watchdog:** if the watchdog thread itself oversleeps by 2s or more
while the process uses less than 25% CPU, the whole process was frozen.
That time is subtracted from the stall and isn't sampled. Blocking I/O
(the watchdog still wakes on time) and GIL-hogging CPU work (CPU keeps
advancing) are still reported.
- Docs: `API.md` (edit/refund note) and `LESSONS-LEARNED.md`
(mostly-sync routes that await one trigger; throttling freezes).

## Testing
- Backend unit suite: **5863 passed**, 36 skipped (`pytest backend/tests
--ignore=backend/tests/emulator`).
- New tests:
- edited-track refund cases (delete in awaiting_review /
lyrics_complete, cancel);
  - YouTube restart on 410, give up after 3 attempts, no restart on 403;
  - orchestrator queues failed uploads and warns when queueing fails;
  - loop-responsiveness ticker tests for each offloaded path;
  - `run_on_loop` result, exception and timeout-with-cancel;
  - `run_in_body_thread` copies context and uses its own pool;
- watchdog: freeze ignored, I/O stall and CPU hog still reported, freeze
subtracted from a real stall.
- Local CodeRabbit: 3 cycles. Fixed the nested-executor deadlock risk,
the missing `run_on_loop` timeout, the parked-download writes on the
loop, and a coroutine leak. The last cycle had 0 findings.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 16:15:47 -04:00

441 lines
17 KiB
Python

"""
YouTube upload queue processor.
Processes deferred YouTube uploads when quota is available.
Called by Cloud Scheduler via an internal endpoint (hourly).
"""
import asyncio
import logging
import os
import shutil
import tempfile
from typing import Dict, Any, Optional
from backend.config import get_settings
from backend.services.community_publish import notify_community_publish
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.youtube_quota_service import get_youtube_quota_service
from backend.services.youtube_upload_queue_service import get_youtube_upload_queue_service
logger = logging.getLogger(__name__)
# Uploads attempted per run (quota-bound anyway) vs entries fetched: the fetch
# is larger so entries whose claim is refused (job mid admin re-render) can't
# fill the page and starve everything behind them.
MAX_UPLOADS_PER_RUN = 20
QUEUE_FETCH_LIMIT = 100
async def process_youtube_upload_queue() -> Dict[str, Any]:
"""
Process queued YouTube uploads.
Checks quota availability, then processes queued uploads one at a time.
Stops if quota is exhausted during processing.
Returns:
Summary dict with counts of processed, failed, and remaining items
"""
settings = get_settings()
quota_service = get_youtube_quota_service()
queue_service = get_youtube_upload_queue_service()
# Check if any quota is available
# Every quota/queue call below is sync Firestore (+ Cloud Monitoring) I/O —
# each goes through asyncio.to_thread so this coroutine never blocks the loop.
allowed, remaining, message = await asyncio.to_thread(quota_service.check_quota_available)
if not allowed:
logger.info(f"YouTube queue processor: no quota available, skipping. {message}")
return {
"status": "skipped",
"reason": "no_quota",
"message": message,
"processed": 0,
"failed": 0,
"remaining": len(await asyncio.to_thread(queue_service.get_queued_uploads)),
}
# Get queued uploads. Fetch more than we'll upload so entries whose claim is
# refused (job mid admin re-render) can't fill the page and starve others.
queued = await asyncio.to_thread(queue_service.get_queued_uploads, limit=QUEUE_FETCH_LIMIT)
if not queued:
logger.info("YouTube queue processor: no uploads queued")
return {
"status": "empty",
"message": "No uploads queued",
"processed": 0,
"failed": 0,
"remaining": 0,
}
logger.info(f"YouTube queue processor: processing {len(queued)} queued uploads")
# One batched read of the jobs (for the admin re-render notification
# choice) instead of a Firestore round-trip per entry.
jobs = await asyncio.to_thread(_get_jobs_batch, queue_service.db, [e["job_id"] for e in queued])
processed = 0
failed = 0
attempted = 0
for entry in queued:
job_id = entry["job_id"]
if attempted >= MAX_UPLOADS_PER_RUN:
break
# Re-check quota before each upload
allowed, remaining, message = await asyncio.to_thread(quota_service.check_quota_available)
if not allowed:
logger.info(f"YouTube queue processor: quota exhausted after {processed} uploads")
break
# Claim the entry. The claim transaction also reads the job and refuses
# while an admin re-render is active on it (the re-render's claim in turn
# refuses while an upload is processing), so the two never overlap.
if not await asyncio.to_thread(queue_service.mark_processing, job_id):
logger.info(f"YouTube queue processor: could not claim job {job_id}, skipping")
continue
attempted += 1
uploaded_url = None
try:
youtube_url = await _process_single_upload(job_id, entry, quota_service, settings)
if youtube_url:
uploaded_url = youtube_url
# Record completion FIRST: from here on the entry must never go
# back to "queued" (that would upload the video a second time).
await asyncio.to_thread(queue_service.mark_completed, job_id, youtube_url)
await _after_successful_upload(job_id, entry, youtube_url, jobs.get(job_id), queue_service)
processed += 1
else:
await asyncio.to_thread(queue_service.mark_failed, job_id, "Upload returned no URL")
failed += 1
except Exception as e:
error_str = str(e)
logger.exception(f"YouTube queue processor: failed to process job {job_id}: {e}")
if uploaded_url:
# The video IS on YouTube; only recording it failed. Never
# re-queue — flag for attention instead.
await asyncio.to_thread(_flag_post_upload_error, queue_service, job_id, uploaded_url, error_str)
processed += 1
continue
# If quota exceeded, stop processing entirely
if "quotaExceeded" in error_str:
await asyncio.to_thread(queue_service.mark_failed, job_id, f"Quota exceeded: {error_str}")
logger.warning("YouTube queue processor: quota exceeded, stopping")
failed += 1
break
await asyncio.to_thread(queue_service.mark_failed, job_id, error_str)
failed += 1
remaining_count = len(await asyncio.to_thread(queue_service.get_queued_uploads))
logger.info(
f"YouTube queue processor: done. processed={processed} failed={failed} remaining={remaining_count}"
)
return {
"status": "processed",
"processed": processed,
"failed": failed,
"remaining": remaining_count,
}
async def _after_successful_upload(job_id: str, entry: Dict[str, Any], youtube_url: str, job, queue_service) -> None:
"""Post-upload side effects. Each is isolated: a failure is logged and the
entry flagged for attention, never re-queued (the upload already happened)."""
try:
# Update job state_data with the YouTube URL
await asyncio.to_thread(_update_job_youtube_url, job_id, youtube_url)
except Exception as e:
logger.exception(f"YouTube queue processor: failed to record URL for job {job_id}")
_flag_post_upload_error(queue_service, job_id, youtube_url, f"update job: {e}")
# Send follow-up email — unless queued by (or processed during) an admin
# re-render that wasn't meant to notify the customer.
from backend.services.admin_rerender_service import suppress_customer_notifications
if entry.get("notify_user", True) and not (job is not None and suppress_customer_notifications(job)):
await _send_youtube_upload_notification(job_id, entry, youtube_url)
else:
logger.info(f"YouTube queue processor: skipping follow-up email for job {job_id} (notifications suppressed)")
# If this was a requests-board community pick, mark it published and fan out
# "your track is live" emails to everyone who voted (idempotent: voters
# already notified are never re-emailed).
try:
await notify_community_publish(job_id, youtube_url)
except Exception as e:
logger.exception(f"YouTube queue processor: community publish failed for job {job_id}")
_flag_post_upload_error(queue_service, job_id, youtube_url, f"community publish: {e}")
def _flag_post_upload_error(queue_service, job_id: str, youtube_url: str, error: str) -> None:
"""Mark an uploaded entry as needing attention — terminal, never re-queued."""
try:
queue_service.mark_post_upload_error(job_id, youtube_url, error)
except Exception:
logger.exception(f"YouTube queue processor: could not flag post-upload error for job {job_id}")
def _get_jobs_batch(db, job_ids) -> Dict[str, Any]:
"""Read the given jobs in one ``get_all`` call; returns ``{job_id: job view}``.
The view carries only what the notification check needs (status,
review_token, state_data). Best-effort: on error, returns ``{}`` (the
authoritative admin re-render check is in the claim transaction).
"""
from types import SimpleNamespace
unique_ids = list(dict.fromkeys(job_ids))
if not unique_ids:
return {}
try:
collection = db.collection(get_settings().firestore_collection)
refs = [collection.document(job_id) for job_id in unique_ids]
jobs = {}
for snapshot in db.get_all(refs):
if not snapshot.exists:
continue
data = snapshot.to_dict() or {}
jobs[snapshot.id] = SimpleNamespace(
job_id=snapshot.id,
status=data.get("status"),
review_token=data.get("review_token"),
state_data=data.get("state_data") or {},
)
return jobs
except Exception as e:
logger.warning(f"YouTube queue processor: batch job read failed: {e}")
return {}
async def _process_single_upload(
job_id: str,
entry: Dict[str, Any],
quota_service,
settings,
) -> Optional[str]:
"""
Process a single queued YouTube upload.
Downloads the video from GCS, uploads to YouTube, records quota.
Returns:
YouTube URL if successful, None otherwise
"""
# GCS download + the whole YouTube upload are blocking, and this runs inside
# the API process (internal/admin queue routes) — keep it off the event loop.
return await asyncio.to_thread(_process_single_upload_sync, job_id, entry, quota_service, settings)
def _process_single_upload_sync(
job_id: str,
entry: Dict[str, Any],
quota_service,
settings,
) -> Optional[str]:
job_manager = JobManager()
storage = StorageService()
job = job_manager.get_job(job_id)
if not job:
logger.error(f"YouTube queue processor: job {job_id} not found")
return None
# Create temp directory for the download
temp_dir = tempfile.mkdtemp(prefix=f"yt-queue-{job_id[:8]}-")
try:
# Find the video file in GCS (prefer MKV, then lossless MP4, then lossy)
video_path = _download_video_from_gcs(job_id, job, storage, temp_dir)
if not video_path:
logger.error(f"YouTube queue processor: no video file found for job {job_id}")
return None
# Download thumbnail if available
thumbnail_path = _download_thumbnail_from_gcs(job_id, job, storage, temp_dir)
# Build YouTube service with fresh credentials
youtube_service = _create_youtube_service(settings)
if not youtube_service:
logger.error("YouTube queue processor: failed to create YouTube service")
return None
# Build metadata
artist = entry.get("artist", job.artist or "Unknown")
title = entry.get("title", job.title or "Unknown")
# Render title + description + tags via the shared renderer (single source
# of truth, also used by the live upload + bulk-rewrite tool).
from backend.services.youtube_description import (
build_youtube_tags,
build_youtube_title,
render_youtube_description,
translated_language_name,
)
translation_name = translated_language_name(job.state_data)
youtube_title = build_youtube_title(artist, title, translation_name)
if len(youtube_title) > 95:
youtube_title = youtube_title[:92] + " ..."
brand_code = entry.get("brand_code") or job.state_data.get("brand_code")
description = render_youtube_description(
artist=artist,
title=title,
brand_code=brand_code,
template=settings.default_youtube_description or None,
translation_language_name=translation_name,
)
# Upload
video_id, video_url = youtube_service.upload_video(
video_path=video_path,
title=youtube_title,
description=description,
thumbnail_path=thumbnail_path,
tags=build_youtube_tags(artist, title),
replace_existing=True,
)
if video_url:
# Record upload in pending buffer (bridges ~7min GCP monitoring delay)
quota_service.record_upload(job_id)
logger.info(f"YouTube queue processor: uploaded job {job_id} -> {video_url}")
return video_url
return None
finally:
# Cleanup temp directory
if os.path.exists(temp_dir):
shutil.rmtree(temp_dir, ignore_errors=True)
def _download_video_from_gcs(
job_id: str, job, storage: StorageService, temp_dir: str
) -> Optional[str]:
"""Download the best available video file from GCS."""
file_urls = job.file_urls or {}
finals = file_urls.get("finals", {}) if isinstance(file_urls.get("finals"), dict) else {}
# Priority order: MKV (FLAC audio) > lossless MP4 > lossy 4K MP4 > lossy 720p MP4
candidates = [
(finals.get("lossless_4k_mkv"), "lossless_4k_mkv", ".mkv"),
(finals.get("lossless_4k_mp4"), "lossless_4k_mp4", ".mp4"),
(finals.get("lossy_4k_mp4"), "lossy_4k_mp4", ".mp4"),
(finals.get("lossy_720p_mp4"), "lossy_720p_mp4", ".mp4"),
]
for gcs_path, key, ext in candidates:
if gcs_path:
local_path = os.path.join(temp_dir, f"video{ext}")
try:
storage.download_file(gcs_path, local_path)
if os.path.isfile(local_path) and os.path.getsize(local_path) > 0:
logger.info(f"Downloaded {key} from GCS for job {job_id}")
return local_path
except Exception as e:
logger.warning(f"Failed to download {key} for job {job_id}: {e}")
return None
def _download_thumbnail_from_gcs(
job_id: str, job, storage: StorageService, temp_dir: str
) -> Optional[str]:
"""Download the thumbnail from GCS if available."""
file_urls = job.file_urls or {}
screens = file_urls.get("screens", {}) if isinstance(file_urls.get("screens"), dict) else {}
thumbnail_url = screens.get("title_jpg")
if not thumbnail_url:
return None
local_path = os.path.join(temp_dir, "thumbnail.jpg")
try:
storage.download_file(thumbnail_url, local_path)
if os.path.isfile(local_path) and os.path.getsize(local_path) > 0:
return local_path
except Exception as e:
logger.warning(f"Failed to download thumbnail for job {job_id}: {e}")
return None
def _create_youtube_service(settings):
"""Create a YouTube upload service with fresh credentials from Secret Manager."""
try:
from backend.services.youtube_upload_service import YouTubeUploadService
import json
youtube_creds_json = settings.get_secret("youtube-oauth-credentials")
if not youtube_creds_json:
logger.error("YouTube OAuth credentials not found in Secret Manager")
return None
credentials = json.loads(youtube_creds_json)
return YouTubeUploadService(
credentials=credentials,
non_interactive=True,
server_side_mode=True,
logger=logger,
)
except Exception as e:
logger.exception(f"Failed to create YouTube service: {e}")
return None
def _update_job_youtube_url(job_id: str, youtube_url: str) -> None:
"""Update job state_data with the YouTube URL after deferred upload.
Raises on failure (including a missing job) so the caller records the queue
entry as completed-with-``needs_attention`` instead of silently losing the URL.
"""
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
raise RuntimeError(f"job {job_id} not found while recording YouTube URL {youtube_url}")
# Atomic per-field writes: this deferred upload can land while other
# state_data is being written, so rewriting the whole map from a
# snapshot could clobber a sibling key.
job_manager.update_job(job_id, {
"state_data.youtube_url": youtube_url,
"state_data.youtube_upload_queued": False, # No longer queued
})
logger.info(f"Updated job {job_id} state_data with YouTube URL")
async def _send_youtube_upload_notification(
job_id: str, entry: Dict[str, Any], youtube_url: str
) -> None:
"""Send follow-up email notifying user their YouTube upload is complete."""
try:
from backend.services.job_notification_service import get_job_notification_service
notification_service = get_job_notification_service()
await notification_service.send_youtube_upload_complete_email(
job_id=job_id,
user_email=entry.get("user_email", ""),
artist=entry.get("artist", ""),
title=entry.get("title", ""),
youtube_url=youtube_url,
brand_code=entry.get("brand_code"),
)
except Exception as e:
logger.error(f"Failed to send YouTube upload notification for job {job_id}: {e}")
# Don't fail the upload over a notification error - mark it in the queue
try:
from backend.services.youtube_upload_queue_service import get_youtube_upload_queue_service
queue_service = get_youtube_upload_queue_service()
doc_ref = queue_service.db.collection("youtube_upload_queue").document(job_id)
doc_ref.update({"notification_sent": False, "notification_error": str(e)})
except Exception:
pass