Compare commits

...
Author SHA1 Message Date
Andrew Beveridge 89cc9e9113 fix: Enable Cloud Tasks in deployment for scalable architecture
The scalable architecture (Cloud Tasks queues, IAM, infrastructure) was
deployed via Pulumi but never activated in the Cloud Run deployment.

Changes:
- Enable ENABLE_CLOUD_TASKS=true environment variable
- Set CLOUD_RUN_SERVICE_URL for Cloud Tasks callbacks
- Set GCP_REGION for queue location consistency
- Change concurrency from 4 to 1 for true container isolation
- Increase max-instances from 10 to 50 for higher throughput

This completes Phase 1 of the scalable architecture migration, ensuring
each job gets dedicated container resources with no contention.
2025-12-11 15:45:03 -05:00
Andrew Beveridge ☄️ ec5a816b68 Merge pull request #20 from nomadkaraoke/anchor-sequence-optimization
feat: Optimize AnchorSequenceFinder for cloud compatibility and performance
2025-12-11 14:37:06 -05:00
3 changed files with 11 additions and 6 deletions

No files matched your search

+4 -3
View File
@@ -54,6 +54,7 @@ steps:
- 'us-central1-docker.pkg.dev/$PROJECT_ID/karaoke-repo/karaoke-backend:latest'
# Deploy to Cloud Run with version from pyproject.toml
# Uses Cloud Tasks for worker coordination (scalable architecture)
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: 'bash'
args:
@@ -71,10 +72,10 @@ steps:
--cpu-boost \
--execution-environment gen2 \
--timeout 1800 \
--concurrency 4 \
--max-instances 10 \
--concurrency 1 \
--max-instances 50 \
--min-instances 0 \
--set-env-vars "GOOGLE_CLOUD_PROJECT=$PROJECT_ID,GCS_BUCKET_NAME=karaoke-gen-storage-$PROJECT_ID,FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX" \
--set-env-vars "GOOGLE_CLOUD_PROJECT=$PROJECT_ID,GCS_BUCKET_NAME=karaoke-gen-storage-$PROJECT_ID,FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1" \
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest"
images:
@@ -1,7 +1,7 @@
# Scalable Architecture Plan
**Last Updated:** 2025-12-11
**Status:** Analysis Complete - Ready for Implementation
**Status:** Phase 1 Complete - Cloud Tasks Enabled
This document outlines the plan for evolving karaoke-gen's cloud backend to support **100+ concurrent jobs** with no performance degradation, while also achieving a clean, maintainable codebase with proper separation of concerns.
@@ -756,7 +756,11 @@ wait
## Success Criteria
### Phase 1 (Cloud Tasks)
- [ ] 50 concurrent jobs complete without failures
- [x] Cloud Tasks queues created and configured via Pulumi
- [x] WorkerService updated to support Cloud Tasks mode
- [x] Deployment configured with ENABLE_CLOUD_TASKS=true
- [x] Container concurrency set to 1 for true isolation
- [ ] 50 concurrent jobs complete without failures (to be validated)
- [ ] Processing time per job unchanged (same as 1 job)
- [ ] No resource contention (CPU/memory per job consistent)
- [ ] Automatic retry on transient failures works
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.71.35"
version = "0.71.36"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"