Compare commits

...
Author SHA1 Message Date
Andrew BeveridgeandClaude Opus 4.5 7c0575440d Add dispatch_deadline to Cloud Tasks to prevent audio worker timeout
The audio worker was being killed by Cloud Tasks before Modal API could
complete audio separation. Cloud Tasks defaults to a 10-minute dispatch
deadline, but Modal separation can take 15-20+ minutes.

This adds explicit dispatch_deadline configuration per worker type:
- Audio: 30 minutes (max) - for Modal audio separation
- Lyrics: 25 minutes - for AudioShake + agentic correction
- Screens: 10 minutes - fast screen generation
- Video/Render: 30 minutes - for encoding

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-31 21:11:08 -05:00
Andrew Beveridge 5c2260660c Merge branch 'main' of github.com:nomadkaraoke/karaoke-gen 2025-12-31 20:43:33 -05:00
Andrew BeveridgeandClaude Opus 4.5 0a731df499 Increase lyrics transcription timeout to 20 minutes
The outer transcription timeout was hitting 600s before agentic AI
correction could complete. The timeout needs to account for:
- Cloud Run cold start / worker initialization (1-5 min)
- AudioShake transcription (1-2 min)
- spaCy model loading for correction (2-3 min on cold start)
- Agentic AI correction (1-3 min)

Changes:
- Increase TRANSCRIPTION_TIMEOUT_SECONDS from 600s to 1200s (20 min)
- Update E2E test to use API calls for human review steps
- Extend E2E test timeouts for longer processing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-31 20:29:33 -05:00
2 changed files with 23 additions and 2 deletions

No files matched your search

+22 -1
View File
@@ -20,6 +20,7 @@ import os
import json
from typing import Optional
import httpx
from google.protobuf import duration_pb2
from backend.config import get_settings
from backend.services.tracing import inject_trace_context
@@ -38,6 +39,18 @@ WORKER_QUEUES = {
"video": "video-worker-queue",
}
# Dispatch deadlines for each worker type (in seconds)
# Cloud Tasks max is 1800s (30 min). These set how long Cloud Tasks waits
# for the HTTP handler to respond before considering it failed.
# Must be >= actual worker timeout + buffer for startup/upload.
WORKER_DISPATCH_DEADLINES = {
"audio": 1800, # 30 min - Modal separation can take 15-20 min
"lyrics": 1500, # 25 min - TRANSCRIPTION_TIMEOUT_SECONDS is 1200s (20 min)
"screens": 600, # 10 min - Screen generation is fast
"render-video": 1800, # 30 min - Video encoding can be slow
"video": 1800, # 30 min - Video encoding can be slow
}
class WorkerService:
"""
@@ -234,6 +247,9 @@ class WorkerService:
# This allows worker spans to link back to the original request trace
headers = inject_trace_context(headers)
# Get dispatch deadline for this worker type (how long Cloud Tasks waits for response)
dispatch_deadline_seconds = WORKER_DISPATCH_DEADLINES.get(worker_type, 600)
# Build task payload
task = {
"http_request": {
@@ -247,6 +263,10 @@ class WorkerService:
"service_account_email": f"karaoke-backend@{project}.iam.gserviceaccount.com",
},
},
# Set dispatch_deadline - how long Cloud Tasks waits for the handler to respond
# Default is 10 min, but audio separation can take 30+ min
# Max is 1800s (30 min) for Cloud Tasks
"dispatch_deadline": duration_pb2.Duration(seconds=dispatch_deadline_seconds),
}
# Create task
@@ -254,7 +274,8 @@ class WorkerService:
# This prevents duplicate task errors on retries
response = self.tasks_client.create_task(parent=parent, task=task)
logger.info(
f"[job:{job_id}] Created Cloud Task for {worker_type} worker: {response.name}"
f"[job:{job_id}] Created Cloud Task for {worker_type} worker: {response.name} "
f"(dispatch_deadline={dispatch_deadline_seconds}s)"
)
return True
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.86.3"
version = "0.86.4"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"