Files
Andrew Beveridge ☄️andClaude Opus 5.5 fe89795aec fix: deliver worker-sent pushes/emails before Cloud Run Job exit; Web Push from workers (v0.284.1) (#1221)
Follow-up to #1220 (mobile Phase 6), from its whole-branch review.

## Problem
Cloud Run Job workers run inside `asyncio.run(...)`.
`JobManager._send_push_notification` (and the completion email / idle
reminder) used `loop.create_task(...)` and the worker returned
immediately, so `asyncio.run` cancelled the send. Prod logs: all 62
"Push notifications sent" lines in 14 days came from the API service,
none from worker jobs. Separately, worker jobs never had
`ENABLE_PUSH_NOTIFICATIONS` / VAPID keys, so even a delivered Web Push
would have been skipped.


## Customer impact check (emails)
Checked 2026-09-25 → 2026-10-25 (full log retention): **no customer
emails were lost.** 289 COMPLETE transitions → 11 deliberately
suppressed re-renders, 278 completion emails scheduled and 278 sent
(email_log has 297 completion records, all with Postmark MessageIDs;
Postmark outbound ≥ email_log). The email coroutines happen to do all
their I/O synchronously, so they finished in the task's first step
before `asyncio.run` cancelled; pushes hand off to a thread and were
cancelled. The drain removes that fragility for emails too. **Only
pushes were lost** (0 worker pushes in 14 days).

## Fix
- Notification tasks are tracked; every worker `main()` wraps its job in
`with_notification_drain` (bounded 10 s wait on our coroutines; provider
calls keep their own timeouts). The API server path stays
fire-and-forget.
- `failure_push_sent` is written only after the push was attempted;
in-flight id cleared via done-callback.
- Worker jobs (audio-download, audio-separation, bulk-search,
lyrics-transcription, video-encoding) get
`ENABLE_PUSH_NOTIFICATIONS=true` +
`VAPID_PUBLIC_KEY`/`VAPID_PRIVATE_KEY` (same secrets as the API).
CI-pinned jobs now `ignore_changes` their image (CI owns it), so a
targeted `pulumi up` changes env only.
- Expo messages carry `channelId: "jobs"` (the app's Android channel).

## Deploy
After merge, a targeted `pulumi up` from main on the 5 jobs. Preview
(read-only, from this branch): 5 updates, each only
+`ENABLE_PUSH_NOTIFICATIONS`/+`VAPID_*`, no removals, no image change,
283 unchanged.

**Behaviour change:** Web Push subscribers will start receiving the
worker-triggered pushes (video ready, needs review, failed) they were
always meant to get.

## Tests
6901 backend tests pass (emulator-integration file excluded: local env).
Tests reproduce the cancellation through each worker's real `main()`.
Per-task review + local CodeRabbit (no findings).

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01CZf1cM9vefnNw5djARRhyA

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 18:12:37 -04:00

1053 lines
51 KiB
Python
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""
Lyrics transcription and correction worker.
Handles the lyrics processing track of parallel processing:
1. Fetch reference lyrics from multiple sources (Genius, Spotify, Musixmatch, LRCLib)
2. Transcribe audio with AudioShake API (1-2 min)
3. Run automatic correction using LyricsTranscriber
4. Generate corrections JSON for human review
5. Upload all data to GCS
6. Transition to AWAITING_REVIEW state
Re-uses:
- karaoke_gen.lyrics_processor.LyricsProcessor for lyrics fetching and orchestration
- lyrics_transcriber library (submodule) for transcription and correction
Observability:
- All operations wrapped in tracing spans for Cloud Trace visibility
- Logs include [job:ID] prefix for easy filtering in Cloud Logging
- Worker start/end timing logged with WORKER_START/WORKER_END markers
"""
import asyncio
import logging
import os
import shutil
import tempfile
import time
import json
from typing import Optional, Dict, Any
from pathlib import Path
from backend.models.job import JobStatus
from backend.services.job_manager import JobManager, with_notification_drain
from backend.services.input_persistence import ensure_job_inputs_persisted
from backend.services.storage_service import StorageService
from backend.services.audio_search_service import DownloadError
from backend.services.job_health_service import validate_worker_can_run
from backend.services.lyrics_cache_service import LyricsCacheService
from backend.utils.audio_filenames import local_audio_filename
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
from backend.workers.style_helper import load_style_config
from backend.workers.registry import worker_registry
from backend.services.tracing import job_span, add_span_event, add_span_attribute
from backend.services.metrics import metrics
# Import from karaoke_gen package
from karaoke_gen.lyrics_processor import LyricsProcessor
from backend.config import get_settings
logger = logging.getLogger(__name__)
# Timeout for entire transcription process including agentic AI correction (20 minutes)
# This needs to account for:
# - Cloud Run cold start / worker initialization (1-5 min)
# - AudioShake transcription (1-2 min)
# - spaCy model loading for correction (2-3 min on cold start)
# - Agentic AI correction (1-3 min)
TRANSCRIPTION_TIMEOUT_SECONDS = 1200
# Default agentic correction timeout (3 minutes)
# Configurable via AGENTIC_CORRECTION_TIMEOUT_SECONDS environment variable
# If agentic AI takes longer than this, abort and use uncorrected transcription
# Human review will correct any issues
DEFAULT_AGENTIC_TIMEOUT_SECONDS = 180
def probe_audio_duration_seconds(audio_path: str) -> Optional[float]:
"""Return the audio duration in seconds via ffprobe, or None on failure.
Used to scale the transcription timeout for long inputs. Non-fatal: any
failure returns None and the caller falls back to the floor timeout.
"""
import subprocess
try:
result = subprocess.run(
[
"ffprobe", "-v", "quiet", "-print_format", "json",
"-show_format", audio_path,
],
capture_output=True, text=True, timeout=30,
)
if result.returncode != 0:
return None
fmt = json.loads(result.stdout).get("format", {})
duration = float(fmt.get("duration", 0) or 0)
return duration if duration > 0 else None
except Exception:
return None
def compute_transcription_timeout(audio_duration_seconds: Optional[float], settings=None) -> int:
"""Compute the outer transcription timeout, scaled by audio duration.
Long inputs (e.g. 30+ minute mashups) legitimately take longer to
transcribe, so we scale the safety-net timeout with audio length while
bounding it to [floor, cap]. The cap stays below the Cloud Run Job task
timeout so we raise a clean RuntimeError before the platform hard-kills the
task. When the duration is unknown (probe failed), fall back to the floor.
"""
if settings is None:
settings = get_settings()
floor = settings.transcription_timeout_floor_seconds
cap = settings.transcription_timeout_cap_seconds
if not audio_duration_seconds or audio_duration_seconds <= 0:
return floor
scaled = int(audio_duration_seconds * settings.transcription_timeout_per_audio_second)
return max(floor, min(cap, scaled))
def _configure_agentic_ai():
"""Configure environment variables for agentic AI correction.
The lyrics_transcriber library reads these directly from environment.
This ensures the settings from backend config are available.
"""
settings = get_settings()
# Enable agentic AI if configured
if settings.use_agentic_ai:
os.environ["USE_AGENTIC_AI"] = "1"
os.environ["AGENTIC_AI_MODEL"] = settings.agentic_ai_model
logger.info(f"Agentic AI enabled with model: {settings.agentic_ai_model}")
else:
os.environ["USE_AGENTIC_AI"] = "0"
logger.info("Agentic AI disabled")
# Loggers to capture for lyrics worker (top-level package loggers only)
# Note: We only capture at top-level package loggers to avoid duplicate log entries.
# Logs from sub-modules (e.g., lyrics_transcriber.correction.anchor_sequence) will
# propagate up to their parent logger (lyrics_transcriber) where they get captured.
LYRICS_WORKER_LOGGERS = [
"karaoke_gen.lyrics_processor",
"lyrics_transcriber",
]
def create_lyrics_processor(
style_params_json: Optional[str] = None,
lyrics_file: Optional[str] = None,
subtitle_offset_ms: int = 0
) -> LyricsProcessor:
"""
Create a LyricsProcessor instance configured for Cloud Run processing.
This reuses the karaoke_gen LyricsProcessor with settings optimized for Cloud Run:
- Uses AudioShake API for transcription (via AUDIOSHAKE_API_TOKEN env var)
- Uses Genius/Spotify/Musixmatch APIs for reference lyrics (via env vars)
- Skips interactive review (will be handled by separate React UI)
- Generates corrections JSON for review interface
Args:
style_params_json: Optional path to style parameters JSON file
lyrics_file: Optional path to user-provided lyrics file (overrides API fetch)
subtitle_offset_ms: Offset for subtitle timing in milliseconds
Returns:
Configured LyricsProcessor instance
"""
# Configure logger for LyricsProcessor
lyrics_logger = logging.getLogger("karaoke_gen.lyrics_processor")
lyrics_logger.setLevel(logging.INFO)
return LyricsProcessor(
logger=lyrics_logger,
style_params_json=style_params_json,
lyrics_file=lyrics_file, # Use user-provided lyrics if available
skip_transcription=False, # We want transcription
skip_transcription_review=True, # Skip interactive review (use React UI instead)
render_video=False, # Skip video generation for now (will be done after review)
subtitle_offset_ms=subtitle_offset_ms
)
async def _trigger_proactive_auto_correct(job_id, job_log) -> None:
"""Trigger proactive auto-correct generation on the API service.
The actual multi-model LLM call + caching runs on the API service (which
has the working Anthropic key / compare-models config) — see
backend/workers/auto_correct_worker.py. Here we just fire the HTTP trigger
and await it. Best-effort and fully isolated: it must NEVER raise, fail the
job, or block the pipeline — downstream stages are already triggered before
this runs, and the service gates the work behind AUTO_CORRECT_PROACTIVE_ENABLED.
"""
try:
from backend.services.worker_service import WorkerService
ok = await WorkerService().trigger_auto_correct(job_id)
job_log.info("Proactive auto-correct trigger %s", "ok" if ok else "skipped/failed")
except Exception as e: # noqa: BLE001 — never let auto-correct affect the job
job_log.warning(f"Proactive auto-correct trigger failed (non-fatal): {e}")
logger.warning(f"[job:{job_id}] Proactive auto-correct trigger failed (non-fatal): {e}")
async def process_lyrics_transcription(job_id: str) -> bool:
"""
Process lyrics transcription and correction for a job using karaoke_gen.LyricsProcessor.
This is the main entry point for the lyrics worker.
Called asynchronously from the job submission endpoint.
Runs in parallel with audio_worker, coordinated via job state.
Workflow:
1. Download audio from GCS
2. Create LyricsProcessor instance
3. Run transcription (calls AudioShake API internally)
4. Fetch reference lyrics from Genius/Spotify/Musixmatch/LRCLib
5. Run automatic correction using lyrics_transcriber library
6. Generate corrections JSON for review interface
7. Upload results to GCS
8. Transition to AWAITING_REVIEW state
Args:
job_id: Job ID to process
Returns:
True if successful, False otherwise
"""
start_time = time.time()
job_manager = JobManager()
storage = StorageService()
# Configure agentic AI before any lyrics processing
_configure_agentic_ai()
# Create job logger for remote debugging FIRST
job_log = create_job_logger(job_id, "lyrics")
# Log with structured markers for easy Cloud Logging queries
logger.info(f"[job:{job_id}] WORKER_START worker=lyrics")
job_log.info("=== LYRICS WORKER STARTED ===")
job_log.info(f"Job ID: {job_id}")
# Register with worker registry to prevent premature container shutdown
# This ensures Cloud Run waits for this worker to complete before terminating
await worker_registry.register(job_id, "lyrics")
# Set up log capture for LyricsTranscriber and its dependencies
# This ensures logs from the lyrics_transcriber library are also captured
log_handler = setup_job_logging(job_id, "lyrics", *LYRICS_WORKER_LOGGERS)
job_log.info(f"Log handler attached for {len(LYRICS_WORKER_LOGGERS)} loggers")
# Initialize temp_dir before try block so finally can check it
temp_dir = None
try:
# Wrap entire worker in a tracing span
with job_span("lyrics-worker", job_id) as root_span:
# Use job_logging_context for proper log isolation when multiple jobs run concurrently
# This ensures logs from third-party libraries (lyrics_transcriber) are only captured
# by this job's handler, not handlers from other concurrent jobs
with job_logging_context(job_id):
job = job_manager.get_job(job_id)
if not job:
logger.error(f"[job:{job_id}] Job not found in Firestore")
job_log.error(f"Job {job_id} not found in Firestore!")
return False
# Copy uploads/{job_id}/** into jobs/{job_id}/input/ (uploads/ expires) and repoint the job.
await asyncio.to_thread(ensure_job_inputs_persisted, job, storage, job_manager, "lyrics")
# Validate job status is appropriate for lyrics worker
# This helps catch bugs where the worker is triggered incorrectly
status_error = validate_worker_can_run("lyrics_worker", job)
if status_error:
logger.warning(f"[job:{job_id}] {status_error}")
job_log.warning(status_error)
# Continue anyway - this is a safety net warning, not a hard failure
add_span_attribute("artist", job.artist)
add_span_attribute("title", job.title)
# Create temporary working directory
temp_dir = tempfile.mkdtemp(prefix=f"karaoke_lyrics_{job_id}_")
job_log.info(f"Created temp directory: {temp_dir}")
job_log.info(f"Starting lyrics transcription for {job.artist} - {job.title}")
job_log.info(f"Log capture enabled for: {', '.join(LYRICS_WORKER_LOGGERS)}")
logger.info(f"[job:{job_id}] Starting lyrics transcription for {job.artist} - {job.title}")
# Log environment configuration
with job_span("check-api-config", job_id):
job_log.info("Checking API configuration...")
apis_configured = []
if os.environ.get("GENIUS_API_TOKEN"):
job_log.info(" Genius API: configured")
apis_configured.append("genius")
else:
job_log.warning(" Genius API: NOT configured")
if os.environ.get("SPOTIFY_COOKIE_SP_DC"):
job_log.info(" Spotify: configured")
apis_configured.append("spotify")
else:
job_log.warning(" Spotify: NOT configured")
if os.environ.get("RAPIDAPI_KEY"):
job_log.info(" Musixmatch (RapidAPI): configured")
apis_configured.append("musixmatch")
else:
job_log.warning(" Musixmatch (RapidAPI): NOT configured")
add_span_attribute("lyrics_apis", ",".join(apis_configured))
# Ensure required environment variables are set
if not os.environ.get("AUDIOSHAKE_API_TOKEN"):
job_log.warning("AUDIOSHAKE_API_TOKEN not set - transcription may fail")
logger.warning(f"[job:{job_id}] AUDIOSHAKE_API_TOKEN not set - transcription may fail")
# Download audio file from GCS (waits for audio worker if URL job)
with job_span("download-audio", job_id) as download_span:
job_log.info("Downloading audio file from GCS...")
# download_audio raises DownloadError with a specific reason
# on failure (no silent None → generic-error fallback).
audio_path = await download_audio(job_id, temp_dir, storage, job, job_manager)
job_log.info(f"Audio downloaded: {os.path.basename(audio_path)}")
download_span.set_attribute("audio_file", os.path.basename(audio_path))
# Jobs with the user's own instrumental skip separation, so there's
# no vocals stem for the review waveform. Derive an approximation
# (mix − instrumental) on CPU alongside transcription.
derived_vocals_task = None
if getattr(job, 'existing_instrumental_gcs_path', None):
derived_vocals_task = asyncio.create_task(asyncio.to_thread(
_store_derived_vocals, job_id, job.existing_instrumental_gcs_path,
audio_path, storage, job_manager, job_log,
))
# Set up LyricsTranscriber cache directory and sync from GCS
# This allows reusing cached AudioShake/lyrics API responses across Cloud Run instances
cache_dir = os.path.join(temp_dir, "lyrics-cache")
os.makedirs(cache_dir, exist_ok=True)
os.environ["LYRICS_TRANSCRIBER_CACHE_DIR"] = cache_dir
job_log.info(f"LyricsTranscriber cache dir: {cache_dir}")
# Sync cache from GCS (download any existing cached API responses)
# Note: Cache sync is best-effort - failures should not fail the job
cache_service = LyricsCacheService(storage)
audio_hash = None
lyrics_hash = None
with job_span("sync-cache-from-gcs", job_id):
try:
# Compute cache keys
audio_hash = cache_service.compute_audio_hash(audio_path)
# Use lyrics_artist/lyrics_title for lyrics hash (these are what LyricsTranscriber uses)
lyrics_search_artist = getattr(job, 'lyrics_artist', None) or job.artist
lyrics_search_title = getattr(job, 'lyrics_title', None) or job.title
lyrics_hash = cache_service.compute_lyrics_hash(lyrics_search_artist, lyrics_search_title)
job_log.info(f"Cache keys: audio_hash={audio_hash[:8]}..., lyrics_hash={lyrics_hash[:8]}...")
add_span_attribute("audio_hash", audio_hash[:8])
add_span_attribute("lyrics_hash", lyrics_hash[:8])
# Download relevant cache files from GCS
cache_stats = cache_service.sync_cache_from_gcs(cache_dir, audio_hash, lyrics_hash)
job_log.info(f"Cache sync from GCS: {cache_stats['downloaded']} downloaded, {cache_stats['not_found']} not found")
add_span_attribute("cache_hits", cache_stats["downloaded"])
except Exception as e:
job_log.warning(f"Cache sync from GCS failed (non-fatal): {e}", exc_info=True)
add_span_attribute("cache_sync_error", str(e)[:100])
# Update progress using state_data (don't change status during parallel processing)
# The status is managed at a higher level - workers just track their progress
job_manager.update_state_data(job_id, 'lyrics_progress', {
'stage': 'transcribing',
'progress': 10,
'message': 'Starting lyrics transcription via AudioShake'
})
# Load style configuration (downloads assets from GCS if available)
# This is needed for max_line_length and other style settings in LyricsTranscriber
with job_span("load-style-config", job_id):
job_log.info("Loading style configuration for lyrics processing...")
style_config = await load_style_config(job, storage, temp_dir)
style_params_json_path = style_config.get_style_params_path() if style_config.has_custom_styles() else None
if style_params_json_path:
job_log.info(f"Using custom style params: {style_params_json_path}")
# Log the contents of the style params for debugging
try:
with open(style_params_json_path, 'r') as f:
style_content = json.load(f)
job_log.info(f"Style params sections: {list(style_content.keys())}")
if 'karaoke' in style_content:
karaoke_style = style_content['karaoke']
job_log.info(f" karaoke.background_image: {karaoke_style.get('background_image', 'NOT SET')}")
job_log.info(f" karaoke.font_path: {karaoke_style.get('font_path', 'NOT SET')}")
except Exception as e:
job_log.warning(f"Could not read style params for logging: {e}")
else:
job_log.info("No custom style params found, using defaults")
# Download user-provided lyrics file if available
lyrics_file_path = None
if hasattr(job, 'lyrics_file_gcs_path') and job.lyrics_file_gcs_path:
with job_span("download-user-lyrics", job_id):
lyrics_file_path = os.path.join(temp_dir, "user_lyrics.txt")
job_log.info(f"Downloading user-provided lyrics file: {job.lyrics_file_gcs_path}")
storage.download_file(job.lyrics_file_gcs_path, lyrics_file_path)
job_log.info(f"User lyrics file downloaded to: {lyrics_file_path}")
# Get lyrics configuration from job
subtitle_offset = getattr(job, 'subtitle_offset_ms', 0) or 0
if subtitle_offset != 0:
job_log.info(f"Subtitle offset: {subtitle_offset}ms")
add_span_attribute("subtitle_offset_ms", subtitle_offset)
# Create LyricsProcessor instance (reuses karaoke_gen code)
job_log.info("Creating LyricsProcessor instance...")
job_log.info(f" style_params_json: {style_params_json_path}")
job_log.info(f" lyrics_file: {lyrics_file_path}")
job_log.info(f" subtitle_offset_ms: {subtitle_offset}")
lyrics_processor = create_lyrics_processor(
style_params_json=style_params_json_path,
lyrics_file=lyrics_file_path,
subtitle_offset_ms=subtitle_offset
)
# Use lyrics_artist/lyrics_title overrides if provided, else fall back to job artist/title
lyrics_search_artist = getattr(job, 'lyrics_artist', None) or job.artist
lyrics_search_title = getattr(job, 'lyrics_title', None) or job.title
job_log.info("Starting LyricsTranscriber processing...")
job_log.info(f" Artist: {job.artist}")
job_log.info(f" Title: {job.title}")
if lyrics_search_artist != job.artist:
job_log.info(f" Lyrics search artist override: {lyrics_search_artist}")
if lyrics_search_title != job.title:
job_log.info(f" Lyrics search title override: {lyrics_search_title}")
logger.info(f"[job:{job_id}] Calling lyrics_processor.transcribe_lyrics()")
# Run transcription + correction with timeout
# AudioShake typically takes 1-2 minutes, but can hang.
with job_span("audioshake-transcription", job_id) as trans_span:
trans_start = time.time()
add_span_event("transcription_started")
# Calculate deadline for agentic correction
# If agentic AI is enabled, the correction loop will check this deadline
# and abort early if exceeded, returning uncorrected transcription for human review
settings = get_settings()
agentic_deadline = None
timeout_seconds = settings.agentic_correction_timeout_seconds
if settings.use_agentic_ai:
agentic_deadline = time.time() + timeout_seconds
job_log.info(
f"Agentic AI enabled with {timeout_seconds}s timeout"
)
# Scale the outer (safety-net) timeout by audio duration so long
# inputs (e.g. 30+ minute mashups) aren't killed prematurely. A
# fixed 1200s cap previously flapped long tracks to 'failed' even
# when AudioShake would have finished. Bounded by [floor, cap]
# (cap stays under the Cloud Run Job task timeout).
audio_duration_seconds = probe_audio_duration_seconds(audio_path)
outer_timeout = compute_transcription_timeout(audio_duration_seconds, settings)
job_log.info(
f"Transcription timeout set to {outer_timeout}s "
f"(audio duration: {audio_duration_seconds}s)"
)
# Run transcription in thread pool
# When agentic AI is enabled, wrap with outer timeout as safety net
# Inner deadline check in corrector.py will break out of gap loop gracefully
# Outer timeout catches case where single LLM call hangs for too long
transcription_coro = asyncio.to_thread(
lyrics_processor.transcribe_lyrics,
input_audio_wav=audio_path,
artist=job.artist, # Original artist for file naming
title=job.title, # Original title for file naming
track_output_dir=temp_dir,
lyrics_artist=lyrics_search_artist, # Override for lyrics search
lyrics_title=lyrics_search_title, # Override for lyrics search
agentic_deadline=agentic_deadline, # Deadline for agentic timeout
)
with metrics.time_external_api("audioshake", job_id):
if settings.use_agentic_ai:
# Outer timeout must be generous to allow for AudioShake time
# (can take 2-5+ minutes for long songs) PLUS agentic correction.
# The inner deadline check in corrector.py handles the 3-minute
# correction limit gracefully by returning partial results.
# This outer timeout is a safety net for completely hung LLM calls.
try:
result = await asyncio.wait_for(
transcription_coro,
timeout=outer_timeout
)
except asyncio.TimeoutError:
# This should rarely trigger since inner deadline check breaks first
# But provides hard stop if e.g. single LLM call hangs completely
job_log.error(
f"HARD TIMEOUT: Transcription exceeded {outer_timeout}s. "
"This indicates the inner deadline check failed to trigger."
)
raise RuntimeError(
f"Lyrics transcription timed out after {outer_timeout}s"
) from None
else:
# Non-agentic mode: use the same duration-scaled timeout.
try:
result = await asyncio.wait_for(
transcription_coro,
timeout=outer_timeout
)
except asyncio.TimeoutError:
raise Exception(f"Transcription timed out after {outer_timeout} seconds")
trans_duration = time.time() - trans_start
trans_span.set_attribute("duration_seconds", trans_duration)
add_span_event("transcription_completed", {"duration_seconds": trans_duration})
job_log.info("Transcription processing complete")
logger.info(f"[job:{job_id}] Transcription complete, uploading results")
# Store processing metadata for transcription + correction provenance
_store_lyrics_processing_metadata(
job_manager, job_id, result, trans_duration,
cache_stats if 'cache_stats' in locals() else None, job_log
)
# Upload lyrics results to GCS
with job_span("upload-lyrics-results", job_id):
job_log.info("Uploading lyrics results to GCS...")
await upload_lyrics_results(job_id, temp_dir, result, storage, job_manager, job_log)
job_log.info("All lyrics data uploaded successfully")
logger.info(f"[job:{job_id}] All lyrics data uploaded successfully")
# Sync new cache files to GCS (upload any newly created cache entries)
# This persists cache across Cloud Run instances for future jobs
# Note: Cache upload is best-effort - failures should not fail the job
with job_span("sync-cache-to-gcs", job_id):
try:
if audio_hash and lyrics_hash:
upload_stats = cache_service.sync_cache_to_gcs(cache_dir, audio_hash, lyrics_hash)
job_log.info(f"Cache sync to GCS: {upload_stats['uploaded']} uploaded, {upload_stats['skipped']} already existed")
add_span_attribute("cache_uploads", upload_stats["uploaded"])
else:
job_log.warning("Skipping cache upload: missing audio_hash or lyrics_hash")
add_span_attribute("cache_upload_skipped", "missing_hashes")
except Exception as e:
job_log.warning(f"Cache upload to GCS failed (non-fatal): {e}")
add_span_attribute("cache_upload_error", str(e)[:100])
# Update progress using state_data (don't change status during parallel processing)
job_manager.update_state_data(job_id, 'lyrics_progress', {
'stage': 'lyrics_complete',
'progress': 45,
'message': 'Lyrics transcription complete'
})
# Mark lyrics processing complete, then durably (AWAITED) trigger
# the screens worker. The old fire-and-forget trigger was being
# cancelled on Cloud Run Job event-loop teardown, orphaning jobs at
# `downloading`; awaiting guarantees the screens Cloud Task is
# enqueued before this worker returns.
# Screens pre-computes the vocals peaks, so the derived stem must be
# registered before we advance. Never fails the job.
if derived_vocals_task is not None:
try:
# shield: on timeout stop waiting but let the thread finish
# on its own (threads can't be cancelled; it cleans up after itself)
await asyncio.wait_for(asyncio.shield(derived_vocals_task), DERIVED_VOCALS_TIMEOUT_SECONDS)
except asyncio.TimeoutError:
job_log.warning(
f"Derived vocals still running after {DERIVED_VOCALS_TIMEOUT_SECONDS}s; "
"continuing without waiting (non-fatal)"
)
except Exception as e:
job_log.warning(f"Derived vocals task failed (non-fatal): {e}")
job_log.info("Lyrics worker complete, advancing to screen generation...")
job_manager.mark_lyrics_complete(job_id)
await job_manager.advance_to_screens_if_ready(job_id)
# Proactively pre-generate + cache AI auto-correct suggestions so
# they're ready when the reviewer opens the lyrics UI. The work
# runs on the API service (it has the Anthropic key); we just fire
# the trigger here. Done AFTER mark_lyrics_complete (downstream is
# already triggered, so this never blocks it) and fully isolated —
# it can never fail or delay the job beyond its own timeout.
await _trigger_proactive_auto_correct(job_id, job_log)
duration = time.time() - start_time
# Store worker-level timing
job_manager.update_processing_metadata(job_id, "timing.lyrics_worker_seconds", round(duration, 1))
root_span.set_attribute("duration_seconds", duration)
logger.info(f"[job:{job_id}] WORKER_END worker=lyrics status=success duration={duration:.1f}s")
return True
except DownloadError as e:
duration = time.time() - start_time
job_log.error(f"Audio download failed: {str(e)}", exc_info=True)
logger.error(f"[job:{job_id}] WORKER_END worker=lyrics status=error duration={duration:.1f}s error={e}")
job_manager.mark_job_failed(
job_id=job_id,
error_message=f"Audio download failed: {str(e)}",
error_details={"stage": "download_audio", "error": str(e)}
)
return False
except Exception as e:
duration = time.time() - start_time
job_log.error(f"Lyrics transcription failed: {str(e)}", exc_info=True)
logger.error(f"[job:{job_id}] WORKER_END worker=lyrics status=error duration={duration:.1f}s error={e}")
job_manager.mark_job_failed(
job_id=job_id,
error_message=f"Lyrics transcription failed: {str(e)}",
error_details={"stage": "lyrics_transcription", "error": str(e)}
)
return False
finally:
# Unregister from worker registry to signal completion
await worker_registry.unregister(job_id, "lyrics")
# Remove log handler to avoid duplicate logging on future runs
for logger_name in LYRICS_WORKER_LOGGERS:
try:
logging.getLogger(logger_name).removeHandler(log_handler)
except Exception:
pass
# Cleanup temporary directory (only if it was created)
if temp_dir and os.path.exists(temp_dir):
shutil.rmtree(temp_dir)
logger.debug(f"[job:{job_id}] Cleaned up temp directory: {temp_dir}")
# Waveform-only nicety: never let it hold up the lyrics -> screens handoff for long.
DERIVED_VOCALS_TIMEOUT_SECONDS = 180
def _store_derived_vocals(job_id, instrumental_gcs_path, mix_path, storage, job_manager, job_log) -> None:
"""Derive approximate vocals (mix − user instrumental) for the review waveform
and register them as stems.vocals_derived. Best-effort: logs and returns on failure.
Uses its own temp dir (the worker's temp_dir may be removed on failure paths
while this thread is still running); only reads mix_path.
"""
import shutil
import tempfile
from backend.services.derived_vocals import derive_vocals_file
work_dir = tempfile.mkdtemp(prefix="derived-vocals-")
try:
ext = os.path.splitext(instrumental_gcs_path)[1] or ".audio"
inst_path = os.path.join(work_dir, f"user_instrumental{ext}")
storage.download_file(instrumental_gcs_path, inst_path)
out_path = os.path.join(work_dir, "vocals_derived.flac")
result = derive_vocals_file(mix_path, inst_path, out_path)
summary = (f"offset {result.offset_samples} samples, gain {result.gain:.3f}, "
f"residual {result.residual_ratio:.2f}")
if not result.useful:
job_log.info(f"Instrumental didn't cancel against the mix ({summary}); no vocal waveform")
return
gcs_path = f"jobs/{job_id}/stems/vocals_derived.flac"
storage.upload_file(out_path, gcs_path)
job_manager.update_file_url(job_id, 'stems', 'vocals_derived', gcs_path)
job_log.info(f"Derived vocals for review waveform ({summary})")
except Exception as e:
job_log.warning(f"Could not derive vocals for review waveform (non-fatal): {e}")
finally:
shutil.rmtree(work_dir, ignore_errors=True)
async def download_audio(
job_id: str,
temp_dir: str,
storage: StorageService,
job,
job_manager: JobManager,
max_wait_seconds: int = 300
) -> str:
"""
Download audio file from GCS to local temp directory.
For URL jobs, the audio worker fetches the source via the flacfetch service
first and uploads it to GCS. We wait for the input_media_gcs_path to be set
by the audio worker.
Args:
job_id: Job ID
temp_dir: Temporary directory for download
storage: StorageService instance
job: Job object
job_manager: JobManager instance
max_wait_seconds: Maximum time to wait for audio worker to download
Returns:
Path to the downloaded audio file.
Raises:
DownloadError: with a specific reason if the audio can't be obtained
(fail loudly with the real cause instead of returning None and
surfacing a generic "Failed to download audio file").
"""
import time
try:
# If input_media_gcs_path is already set, download directly
if job.input_media_gcs_path:
local_path = os.path.join(temp_dir, job.filename or local_audio_filename(job.input_media_gcs_path, "input"))
storage.download_file(job.input_media_gcs_path, local_path)
logger.info(f"Job {job_id}: Downloaded audio from {job.input_media_gcs_path} to {local_path}")
return local_path
# For URL jobs, wait for audio worker to download and upload to GCS
if job.url:
logger.info(f"Job {job_id}: Waiting for audio worker to download from URL...")
start_time = time.time()
poll_interval = 5 # seconds
while time.time() - start_time < max_wait_seconds:
# Refresh job from Firestore
updated_job = job_manager.get_job(job_id)
if updated_job and updated_job.input_media_gcs_path:
# Audio worker has uploaded the file
local_path = os.path.join(temp_dir, local_audio_filename(updated_job.input_media_gcs_path, "input"))
storage.download_file(updated_job.input_media_gcs_path, local_path)
logger.info(f"Job {job_id}: Downloaded audio from {updated_job.input_media_gcs_path}")
return local_path
# Check if audio worker failed
if updated_job and updated_job.status == JobStatus.FAILED:
logger.error(f"Job {job_id}: Audio worker failed, cannot proceed with lyrics")
raise DownloadError(
"Audio worker failed before lyrics could run; cannot "
"proceed without the downloaded/separated audio."
)
# Wait before next poll - use asyncio.sleep to not block event loop
await asyncio.sleep(poll_interval)
logger.error(f"Job {job_id}: Timed out waiting for audio download (waited {max_wait_seconds}s)")
raise DownloadError(
f"Timed out after {max_wait_seconds}s waiting for the audio worker "
"to download the input audio."
)
logger.error(f"Job {job_id}: No input_media_gcs_path found and no URL")
raise DownloadError(
"No input audio available (no input_media_gcs_path and no URL)."
)
except DownloadError:
raise
except Exception as e:
logger.error(f"Job {job_id}: Failed to download audio: {e}", exc_info=True)
raise DownloadError(f"Failed to download audio for lyrics: {e}") from e
async def upload_lyrics_results(
job_id: str,
temp_dir: str,
transcription_result: Dict[str, Any],
storage: StorageService,
job_manager: JobManager,
job_log = None
) -> None:
"""
Upload all lyrics transcription results to GCS.
The transcription_result dict from LyricsProcessor.transcribe_lyrics() contains:
- lrc_filepath: Path to LRC file (timed lyrics)
- ass_filepath: Path to ASS file (karaoke subtitles, may be video file)
Additional files in lyrics directory:
- Corrections JSON (for review interface)
- Reference lyrics (from Genius/Spotify/Musixmatch/LRCLib)
- Uncorrected transcription
Args:
job_id: Job ID
temp_dir: Temporary directory with results
transcription_result: Result dict from LyricsProcessor
storage: StorageService instance
job_manager: JobManager instance
job_log: Optional JobLogger for remote debugging
"""
logger.info(f"Job {job_id}: Uploading lyrics results to GCS")
lyrics_dir = os.path.join(temp_dir, "lyrics")
# Upload LRC file (timed lyrics)
if transcription_result.get("lrc_filepath") and os.path.exists(transcription_result["lrc_filepath"]):
gcs_path = f"jobs/{job_id}/lyrics/karaoke.lrc"
url = storage.upload_file(transcription_result["lrc_filepath"], gcs_path)
job_manager.update_file_url(job_id, 'lyrics', 'lrc', url)
logger.info(f"Job {job_id}: Uploaded LRC file")
# Upload corrections JSON (for review interface)
# LyricsProcessor saves it as "{artist} - {title} (Lyrics Corrections).json"
# Scan the directory instead of reconstructing the filename, because the artist/title
# in Firestore may have been updated (e.g. by admin) after LyricsProcessor generated the file.
corrections_file = None
if os.path.isdir(lyrics_dir):
for fname in os.listdir(lyrics_dir):
if fname.endswith("(Lyrics Corrections).json"):
corrections_file = os.path.join(lyrics_dir, fname)
break
# Fallback: check for generic corrections.json
if not corrections_file:
fallback = os.path.join(lyrics_dir, "corrections.json")
if os.path.exists(fallback):
corrections_file = fallback
if corrections_file and os.path.exists(corrections_file):
# Always upload as corrections.json for consistent access
gcs_path = f"jobs/{job_id}/lyrics/corrections.json"
url = storage.upload_file(corrections_file, gcs_path)
job_manager.update_file_url(job_id, 'lyrics', 'corrections', url)
logger.info(f"Job {job_id}: Uploaded corrections JSON from {corrections_file}")
# Load corrections to get metadata
try:
with open(corrections_file, 'r', encoding='utf-8') as f:
corrections_data = json.load(f)
# Store metadata in state_data
# Include countdown padding info for instrumental synchronization
lyrics_metadata = {
'segment_count': len(corrections_data.get('corrected_segments', [])),
'has_corrections': True,
'ready_for_review': True,
}
# Add countdown padding info from transcription result
# This is needed by video_worker to pad instrumentals for sync
if transcription_result.get("countdown_padding_added"):
lyrics_metadata['has_countdown_padding'] = True
lyrics_metadata['countdown_padding_seconds'] = transcription_result.get("countdown_padding_seconds", 3.0)
logger.info(f"Job {job_id}: Countdown padding detected: {lyrics_metadata['countdown_padding_seconds']}s")
else:
lyrics_metadata['has_countdown_padding'] = False
lyrics_metadata['countdown_padding_seconds'] = 0.0
job_manager.update_state_data(job_id, 'lyrics_metadata', lyrics_metadata)
except Exception as e:
logger.warning(f"Job {job_id}: Could not parse corrections JSON: {e}")
else:
# CRITICAL: corrections.json is required for the review UI
# If it's missing, the job cannot proceed to review
error_msg = f"No corrections JSON found in {lyrics_dir}. Transcription may have produced no lyrics."
logger.error(f"Job {job_id}: {error_msg}")
raise Exception(error_msg)
# Upload ALL reference lyrics files (not just first) so they're available for distribution
# Scan directory for reference files instead of reconstructing filenames,
# since artist/title may have changed in Firestore after file generation.
source_keys = {"Genius": "genius", "Spotify": "spotify", "Musixmatch": "musixmatch", "Lrclib": "lrclib"}
found_references = []
if os.path.isdir(lyrics_dir):
for fname in os.listdir(lyrics_dir):
for source_label, source_key in source_keys.items():
if fname.endswith(f"(Lyrics {source_label}).txt"):
ref_path = os.path.join(lyrics_dir, fname)
gcs_path = f"jobs/{job_id}/lyrics/{fname}"
url = storage.upload_file(ref_path, gcs_path)
job_manager.update_file_url(job_id, 'lyrics', f'reference_{source_key}', url)
if job_log:
job_log.info(f"Found reference lyrics from {source_key}")
logger.info(f"Job {job_id}: Uploaded reference lyrics: {fname}")
found_references.append(source_key)
break
if not found_references:
if job_log:
job_log.warning("No reference lyrics found from any source (Genius, Spotify, Musixmatch, LRCLib)")
logger.warning(f"Job {job_id}: No reference lyrics found from any source")
else:
logger.info(f"Job {job_id}: Found {len(found_references)} reference lyrics sources: {found_references}")
# Upload uncorrected transcription if available
# Scan directory instead of reconstructing filename (artist/title may have changed)
if os.path.isdir(lyrics_dir):
for fname in os.listdir(lyrics_dir):
if fname.endswith("(Lyrics Uncorrected).txt"):
uncorrected_file = os.path.join(lyrics_dir, fname)
gcs_path = f"jobs/{job_id}/lyrics/{fname}"
url = storage.upload_file(uncorrected_file, gcs_path)
job_manager.update_file_url(job_id, 'lyrics', 'uncorrected', url)
logger.info(f"Job {job_id}: Uploaded uncorrected transcription: {fname}")
break
logger.info(f"Job {job_id}: All lyrics results uploaded successfully")
def _store_lyrics_processing_metadata(
job_manager: JobManager,
job_id: str,
transcription_result: Dict[str, Any],
transcription_duration: float,
cache_stats: Optional[Dict[str, Any]],
job_log,
) -> None:
"""Store transcription and correction provenance in processing_metadata.
Extracts AudioShake task_id/asset_id, correction stats, reference sources,
and cache hit/miss info from the transcription result and stores them
permanently for future analytics, debugging, and data export.
"""
try:
# Load the corrections.json data from the result to get metadata
# The transcription_result is the dict returned by LyricsProcessor.transcribe_lyrics()
# which contains file paths, not the corrections data directly.
# We need to find and read the corrections JSON from the output directory.
corrections_metadata = {}
transcription_metadata = {}
# Try to find corrections file in the result's output directory
lyrics_dir = transcription_result.get("lyrics_dir") or ""
corrections_file = None
# Check common paths for corrections JSON
if lyrics_dir:
for fname in os.listdir(lyrics_dir) if os.path.isdir(lyrics_dir) else []:
if fname.endswith("(Lyrics Corrections).json") or fname == "corrections.json":
corrections_file = os.path.join(lyrics_dir, fname)
break
if corrections_file and os.path.exists(corrections_file):
with open(corrections_file, 'r', encoding='utf-8') as f:
corrections_data = json.load(f)
metadata = corrections_data.get("metadata", {})
# Extract AudioShake IDs from transcription_metadata passthrough
trans_meta = metadata.get("transcription_metadata", {})
if trans_meta:
transcription_metadata = {
"provider": "audioshake",
"audioshake_task_id": trans_meta.get("task_id"),
"audioshake_asset_id": trans_meta.get("asset_id"),
"language_detected": trans_meta.get("language"),
}
# Extract correction stats
corrections_metadata = {
"handlers_applied": metadata.get("enabled_handlers", []),
"corrections_made": corrections_data.get("corrections_made", 0),
"correction_ratio": metadata.get("correction_ratio"),
"agentic_routing": metadata.get("agentic_routing"),
"total_words": metadata.get("total_words"),
"anchor_sequences_count": metadata.get("anchor_sequences_count"),
"gap_sequences_count": metadata.get("gap_sequences_count"),
}
# Count segments and words from corrected_segments
corrected_segments = corrections_data.get("corrected_segments", [])
transcription_metadata["segment_count"] = len(corrected_segments)
transcription_metadata["word_count"] = sum(
len(seg.get("words", [])) for seg in corrected_segments
)
# Determine which reference sources were found
reference_lyrics = corrections_data.get("reference_lyrics", {})
if reference_lyrics:
corrections_metadata["reference_sources_found"] = list(reference_lyrics.keys())
# Add transcription duration
transcription_metadata["duration_seconds"] = round(transcription_duration, 1)
# Store transcription metadata
if transcription_metadata:
job_manager.update_processing_metadata(job_id, "transcription", transcription_metadata)
# Store correction metadata
if corrections_metadata:
job_manager.update_processing_metadata(job_id, "correction", corrections_metadata)
# Store cache stats
if cache_stats:
job_manager.update_processing_metadata(job_id, "cache", {
"transcription_cache_downloaded": cache_stats.get("downloaded", 0),
"transcription_cache_not_found": cache_stats.get("not_found", 0),
})
job_log.info("Stored processing metadata (transcription + correction provenance)")
except Exception as e:
# Never fail a job because metadata storage failed
job_log.warning(f"Failed to store lyrics processing metadata (non-fatal): {e}")
logger.warning(f"Job {job_id}: Failed to store lyrics processing metadata: {e}")
# ==================== CLI Entry Point for Cloud Run Jobs ====================
# This allows the lyrics worker to be run as a standalone Cloud Run Job.
# Usage: python -m backend.workers.lyrics_worker --job-id <job_id>
def main():
"""
CLI entry point for running lyrics worker as a Cloud Run Job.
Cloud Run Jobs run to completion without HTTP request lifecycle concerns,
avoiding the instance termination issue where Cloud Run would shut down
instances mid-processing when using BackgroundTasks.
Usage:
python -m backend.workers.lyrics_worker --job-id abc123
Environment Variables:
GOOGLE_CLOUD_PROJECT: GCP project ID (required)
GCS_BUCKET_NAME: Storage bucket name (required)
AUDIOSHAKE_API_TOKEN: AudioShake API key (required)
"""
import argparse
import sys
# Set up argument parser
parser = argparse.ArgumentParser(
description="Lyrics transcription worker for karaoke generation"
)
parser.add_argument(
"--job-id",
required=True,
help="Job ID to process"
)
args = parser.parse_args()
job_id = args.job_id
# Configure logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
)
logger.info(f"Starting lyrics worker CLI for job {job_id}")
# Run the async worker function
try:
# Drain: push/email sends scheduled by the job's last state change must
# finish before asyncio.run cancels pending tasks on exit.
success = asyncio.run(with_notification_drain(process_lyrics_transcription(job_id)))
if success:
logger.info(f"Lyrics transcription completed successfully for job {job_id}")
sys.exit(0)
else:
logger.error(f"Lyrics transcription failed for job {job_id}")
sys.exit(1)
except Exception as e:
logger.error(f"Lyrics worker crashed: {e}", exc_info=True)
sys.exit(1)
if __name__ == "__main__":
main()