mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 21:17:56 +09:00
Compare commits
34
Commits
No files matched your search
@@ -0,0 +1,218 @@
|
||||
# Plan: Job Email Notifications & Admin Message Tools
|
||||
|
||||
**Created:** 2026-01-06
|
||||
**Branch:** feat/sess-20260106-0116-add-email-button
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Add comprehensive email notification system for karaoke-gen that:
|
||||
1. **Auto-sends emails** when jobs complete or need user input (after 5-minute idle threshold)
|
||||
2. **Uses cloud-stored templates** for the job completion message
|
||||
3. **Provides admin tools** to copy/send completion messages manually (for Fiverr, etc.)
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements
|
||||
- [ ] Auto-send job completion email to user (CC: gen@nomadkaraoke.com)
|
||||
- [ ] Auto-send "action needed" reminder emails when user is idle for 5+ minutes at blocking states (lyrics review, instrumental selection)
|
||||
- [ ] Store email templates in GCS (editable without code deploy)
|
||||
- [ ] Admin button: "Copy Message" - copies templated completion message to clipboard
|
||||
- [ ] Admin button: "Send Email" - prompts for email address, then sends completion message
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [ ] Email sending must not block job completion (async/background)
|
||||
- [ ] Idle detection via Cloud Tasks scheduled task (not polling)
|
||||
- [ ] Template changes should take effect without restart (no long-term caching)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### 1. Template Storage (GCS)
|
||||
Store templates in `gs://karaoke-gen-storage/templates/`:
|
||||
- `job-completion.txt` - Plain text template for job completion
|
||||
- `job-completion.html` - HTML version (optional, for rich emails)
|
||||
- `action-needed-lyrics.txt` - Reminder for lyrics review
|
||||
- `action-needed-instrumental.txt` - Reminder for instrumental selection
|
||||
|
||||
**Template Variables:**
|
||||
- `{name}` - User's display name or "there" if unknown
|
||||
- `{youtube_url}` - YouTube video URL
|
||||
- `{dropbox_url}` - Dropbox folder URL
|
||||
- `{artist}` - Artist name
|
||||
- `{title}` - Song title
|
||||
- `{job_id}` - Job ID
|
||||
- `{review_url}` - Lyrics review URL (for action-needed emails)
|
||||
- `{instrumental_url}` - Instrumental selection URL
|
||||
- `{feedback_url}` - Feedback form URL (for completion emails)
|
||||
|
||||
### 2. Email Service Extensions
|
||||
|
||||
Extend `backend/services/email_service.py`:
|
||||
- Add `send_job_completion()` method with CC support
|
||||
- Add `send_action_reminder()` method
|
||||
- Add `render_template()` helper to fetch & render GCS templates
|
||||
|
||||
### 3. Auto-Email Triggers
|
||||
|
||||
**Job Completion:**
|
||||
- Trigger point: When job enters `COMPLETE` status in `job_manager.transition_to_state()`
|
||||
- Implementation: Call email service async in background task
|
||||
|
||||
**Action Reminders (5-min idle detection):**
|
||||
- When job enters `AWAITING_REVIEW` or `AWAITING_INSTRUMENTAL_SELECTION`:
|
||||
- Record `blocking_state_entered_at` timestamp in `state_data`
|
||||
- Schedule a Cloud Tasks task for 5 minutes later
|
||||
- The scheduled task checks if job is still in blocking state and user hasn't started review
|
||||
- If idle, send reminder email and set `reminder_sent` flag to prevent duplicates
|
||||
- **Only one reminder per job** - if `reminder_sent` is true, skip sending
|
||||
|
||||
### 4. Admin UI (Frontend)
|
||||
|
||||
Add admin-only buttons to completed jobs in `OutputLinks.tsx`:
|
||||
- "Copy Message" button - fetches rendered template, copies to clipboard
|
||||
- "Send Email" button - opens dialog for email input, calls API to send
|
||||
|
||||
### 5. Backend API Endpoints
|
||||
|
||||
New endpoints in `backend/api/routes/admin.py`:
|
||||
- `GET /api/admin/jobs/{job_id}/completion-message` - Returns rendered completion message
|
||||
- `POST /api/admin/jobs/{job_id}/send-completion-email` - Sends email to specified address
|
||||
|
||||
New internal endpoint for scheduled reminder:
|
||||
- `POST /api/internal/jobs/{job_id}/check-idle-reminder` - Called by Cloud Tasks
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Template Infrastructure
|
||||
1. [ ] Upload initial template to GCS from user's existing template file
|
||||
2. [ ] Create `template_service.py` to fetch/render templates from GCS
|
||||
3. [ ] Add unit tests for template rendering
|
||||
|
||||
### Phase 2: Email Extensions
|
||||
4. [ ] Extend `email_service.py` with CC support in `send_email()`
|
||||
5. [ ] Add `send_job_completion()` method
|
||||
6. [ ] Add `send_action_reminder()` method
|
||||
7. [ ] Add unit tests for new email methods
|
||||
|
||||
### Phase 3: Auto-Email on Completion
|
||||
8. [ ] Modify `job_manager.py` to trigger completion email when entering COMPLETE
|
||||
9. [ ] Add integration test for completion email trigger
|
||||
|
||||
### Phase 4: Idle Reminder System
|
||||
10. [ ] Add `blocking_state_entered_at` field tracking to job state transitions
|
||||
11. [ ] Create Cloud Tasks queue for idle reminders (`idle-reminder-queue`)
|
||||
12. [ ] Schedule 5-minute delayed task when entering blocking states
|
||||
13. [ ] Implement `/api/internal/jobs/{job_id}/check-idle-reminder` endpoint
|
||||
14. [ ] Add tests for idle reminder flow
|
||||
|
||||
### Phase 5: Admin API
|
||||
15. [ ] Add `GET /api/admin/jobs/{job_id}/completion-message` endpoint
|
||||
16. [ ] Add `POST /api/admin/jobs/{job_id}/send-completion-email` endpoint
|
||||
17. [ ] Add tests for admin endpoints
|
||||
|
||||
### Phase 6: Admin UI
|
||||
18. [ ] Add `isAdmin` check to `OutputLinks.tsx` component
|
||||
19. [ ] Add "Copy Message" button (admin-only) with loading state
|
||||
20. [ ] Add "Send Email" button (admin-only) with email input dialog
|
||||
21. [ ] Add E2E test for admin buttons
|
||||
|
||||
### Phase 7: Documentation
|
||||
22. [ ] Document template management in docs/DEVELOPMENT.md
|
||||
23. [ ] Update docs/API.md with new endpoints
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/services/template_service.py` | Create | GCS template fetching and rendering |
|
||||
| `backend/services/email_service.py` | Modify | Add CC support, completion/reminder methods |
|
||||
| `backend/services/job_manager.py` | Modify | Trigger emails on state transitions |
|
||||
| `backend/services/worker_service.py` | Modify | Add idle reminder task scheduling |
|
||||
| `backend/api/routes/admin.py` | Modify | Add completion message/email endpoints |
|
||||
| `backend/api/routes/internal.py` | Modify | Add idle reminder check endpoint |
|
||||
| `backend/models/job.py` | Modify | Add `blocking_state_entered_at`, `reminder_sent_at` to state_data docs |
|
||||
| `infrastructure/__main__.py` | Modify | Add idle-reminder-queue Cloud Tasks queue |
|
||||
| `frontend/components/job/OutputLinks.tsx` | Modify | Add admin-only Copy/Send buttons |
|
||||
| `frontend/lib/api.ts` | Modify | Add API methods for admin endpoints |
|
||||
| `tests/unit/test_template_service.py` | Create | Template rendering tests |
|
||||
| `tests/unit/test_email_service_completion.py` | Create | Completion email tests |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
**Unit Tests:**
|
||||
- Template rendering with all variable combinations
|
||||
- Email service CC support
|
||||
- Completion/reminder email generation
|
||||
- Idle detection logic
|
||||
|
||||
**Integration Tests:**
|
||||
- Full flow: job completion → email sent
|
||||
- Idle reminder: job enters blocking state → 5 min → reminder sent
|
||||
- Admin endpoint authentication
|
||||
|
||||
**Manual Testing:**
|
||||
- Verify emails render correctly in Gmail/Outlook
|
||||
- Test admin buttons on completed job
|
||||
- Test clipboard copy works across browsers
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] Where to store templates? → **GCS bucket: `gs://karaoke-gen-storage/templates/`**
|
||||
- [x] Should we rate-limit reminder emails? → **Max 1 reminder per job** (send once after 5 min idle, then stop)
|
||||
- [x] Should completion emails also include a feedback request link? → **Yes, for all users**
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. **Email sending failures** - All email sends are async/fire-and-forget; job completion unaffected
|
||||
2. **Template fetch failures** - Fall back to hardcoded template in code
|
||||
3. **Idle reminder failures** - Jobs continue normally; users can still access review UI manually
|
||||
4. **Feature flag** - Add `ENABLE_AUTO_EMAILS=true/false` env var for quick disable
|
||||
|
||||
## Template Content
|
||||
|
||||
Initial template (from user's file, with feedback link added):
|
||||
|
||||
```
|
||||
Hi {name},
|
||||
|
||||
Thanks for your order!
|
||||
|
||||
Here's the link for the karaoke video published to YouTube:
|
||||
{youtube_url}
|
||||
|
||||
Here's the dropbox folder with all the finished files and source files, including:
|
||||
- "(Final Karaoke Lossless).mkv": combined karaoke video in 4k H264 with lossless FLAC audio
|
||||
- "(Final Karaoke).mp4": combined karaoke video with title/end screen in 4k H264/AAC
|
||||
- "(Final Karaoke 720p).mp4": combined karaoke video in 720p H264/AAC (smaller file for older systems)
|
||||
- "(With Vocals).mp4": sing along video in 4k H264/AAC with original vocals
|
||||
- "(Karaoke).mov": karaoke video output from MidiCo (no title/end screen)
|
||||
- "(Title).mov"/"(End).mov": title card and end screen videos
|
||||
- "(Final Karaoke CDG).zip": CDG+MP3 format for older/commercial karaoke systems
|
||||
- "(Final Karaoke TXT).zip": TXT+MP3 format for Power Karaoke
|
||||
- stems/*.flac: various separated instrumental and vocal audio stems in lossless format
|
||||
- lyrics/*.txt song lyrics from various sources in plain text format
|
||||
|
||||
{dropbox_url}
|
||||
|
||||
Let me know if anything isn't perfect and I'll happily tweak / fix, or if you need it in any other format I can probably convert it for you!
|
||||
|
||||
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
|
||||
{feedback_url}
|
||||
|
||||
Thanks again and have a great day!
|
||||
-Andrew
|
||||
```
|
||||
|
||||
**Note:** The `{feedback_url}` placeholder will be populated with the feedback form URL. If no feedback URL is configured, this section can be omitted from the rendered output.
|
||||
|
||||
## Updating Templates
|
||||
|
||||
To update email templates after deployment:
|
||||
|
||||
```bash
|
||||
# Upload new template
|
||||
gsutil cp my-template.txt gs://karaoke-gen-storage/templates/job-completion.txt
|
||||
|
||||
# Templates are fetched fresh on each use (no caching in code)
|
||||
# Changes take effect immediately
|
||||
```
|
||||
@@ -0,0 +1,115 @@
|
||||
# Build Encoding Worker GCE Image
|
||||
#
|
||||
# Rebuilds the custom Packer image when provisioning changes.
|
||||
# The image includes Python 3.13, FFmpeg, fonts, and dependencies
|
||||
# pre-installed to reduce VM startup time from ~10 min to ~30 sec.
|
||||
#
|
||||
# Triggers:
|
||||
# - Push to main with changes to infrastructure/packer/**
|
||||
# - Manual dispatch (workflow_dispatch)
|
||||
#
|
||||
# After image build, the encoding-worker VM will use the new image
|
||||
# on next recreation (e.g., via pulumi up or manual restart).
|
||||
|
||||
name: Build Encoding Worker Image
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'infrastructure/packer/**'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
python_version:
|
||||
description: 'Python version to install'
|
||||
required: false
|
||||
default: '3.13.1'
|
||||
|
||||
concurrency:
|
||||
group: build-encoding-worker-image
|
||||
cancel-in-progress: false # Don't cancel in-progress builds
|
||||
|
||||
jobs:
|
||||
build-image:
|
||||
name: Build Packer Image
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write # Required for Workload Identity Federation
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Install Packer
|
||||
run: |
|
||||
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
|
||||
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
|
||||
sudo apt-get update && sudo apt-get install -y packer
|
||||
|
||||
- name: Initialize Packer
|
||||
working-directory: infrastructure/packer
|
||||
run: packer init encoding-worker.pkr.hcl
|
||||
|
||||
- name: Validate Packer Template
|
||||
working-directory: infrastructure/packer
|
||||
run: packer validate encoding-worker.pkr.hcl
|
||||
|
||||
- name: Build Image
|
||||
working-directory: infrastructure/packer
|
||||
env:
|
||||
PYTHON_VERSION: ${{ github.event.inputs.python_version || '3.13.1' }}
|
||||
run: |
|
||||
echo "Building encoding worker image with Python ${PYTHON_VERSION}..."
|
||||
packer build \
|
||||
-var "python_version=${PYTHON_VERSION}" \
|
||||
encoding-worker.pkr.hcl
|
||||
|
||||
- name: Get Latest Image Info
|
||||
id: image-info
|
||||
run: |
|
||||
IMAGE_NAME=$(gcloud compute images list \
|
||||
--project=nomadkaraoke \
|
||||
--filter="family:encoding-worker" \
|
||||
--sort-by="~creationTimestamp" \
|
||||
--limit=1 \
|
||||
--format="value(name)")
|
||||
echo "image_name=${IMAGE_NAME}" >> $GITHUB_OUTPUT
|
||||
echo "Built image: ${IMAGE_NAME}"
|
||||
|
||||
- name: Summary
|
||||
run: |
|
||||
echo "## Encoding Worker Image Built" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Image:** \`${{ steps.image-info.outputs.image_name }}\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Family:** \`encoding-worker\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Project:** \`nomadkaraoke\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "The encoding-worker VM will use this image on next recreation." >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "To force VM recreation now:" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`bash" >> $GITHUB_STEP_SUMMARY
|
||||
echo "gcloud compute instances delete encoding-worker --zone=us-central1-a --project=nomadkaraoke" >> $GITHUB_STEP_SUMMARY
|
||||
echo "# Then run pulumi up or wait for next infrastructure deploy" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
notify-on-failure:
|
||||
name: Notify on Failure
|
||||
runs-on: ubuntu-latest
|
||||
needs: build-image
|
||||
if: failure()
|
||||
steps:
|
||||
- name: Log Failure
|
||||
run: |
|
||||
echo "::error::Encoding worker image build failed. Check the build-image job logs."
|
||||
+197
-21
@@ -903,11 +903,14 @@ jobs:
|
||||
|
||||
deploy-backend:
|
||||
name: "Deploy - Backend (Cloud Run)"
|
||||
needs: [package-build]
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
outputs:
|
||||
base_rebuilt: ${{ steps.check-base.outputs.needs_rebuild }}
|
||||
env:
|
||||
REGISTRY: us-central1-docker.pkg.dev
|
||||
PROJECT_ID: nomadkaraoke
|
||||
@@ -965,6 +968,32 @@ jobs:
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
# ============================================
|
||||
# Upload wheel to GCS for GCE encoding worker
|
||||
# The GCE worker installs this wheel at job start for hot code updates
|
||||
# ============================================
|
||||
|
||||
- name: Download wheel artifact
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
name: package-dist
|
||||
path: dist/
|
||||
|
||||
- name: Upload wheel to GCS for GCE worker
|
||||
run: |
|
||||
echo "📦 Uploading wheel to GCS for GCE encoding worker..."
|
||||
WHEEL_FILE=$(ls dist/*.whl | head -1)
|
||||
if [ -z "$WHEEL_FILE" ]; then
|
||||
echo "::error::No wheel file found in dist/"
|
||||
exit 1
|
||||
fi
|
||||
echo "Found wheel: $WHEEL_FILE"
|
||||
|
||||
# Upload to GCS (GCE worker downloads latest at job start)
|
||||
gsutil cp "$WHEEL_FILE" gs://karaoke-gen-storage-${{ env.PROJECT_ID }}/wheels/
|
||||
|
||||
echo "✅ Wheel uploaded - GCE worker will pick up changes on next job"
|
||||
|
||||
- name: Configure Docker for Artifact Registry
|
||||
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
|
||||
|
||||
@@ -990,32 +1019,32 @@ jobs:
|
||||
run: |
|
||||
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
|
||||
|
||||
# Try to get the existing base image's hash label
|
||||
# First check if image exists locally (fast, no network call)
|
||||
# Only pull from registry if not cached locally
|
||||
# Fetch hash label directly from registry (10KB config blob, not full image)
|
||||
# This avoids issues with stale local Docker cache on GitHub runners
|
||||
EXISTING_HASH=""
|
||||
if docker image inspect $BASE_IMAGE >/dev/null 2>&1; then
|
||||
echo "✅ Base image found in local cache (skipping slow registry pull)"
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
|
||||
# Fallback to old label name for backwards compatibility
|
||||
if [ -z "$EXISTING_HASH" ]; then
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "poetry.lock.hash"}}' 2>/dev/null || echo "")
|
||||
fi
|
||||
echo "Existing base image hash: $EXISTING_HASH"
|
||||
elif docker pull $BASE_IMAGE 2>/dev/null; then
|
||||
echo "📥 Base image pulled from registry"
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
|
||||
# Fallback to old label name for backwards compatibility
|
||||
if [ -z "$EXISTING_HASH" ]; then
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "poetry.lock.hash"}}' 2>/dev/null || echo "")
|
||||
fi
|
||||
echo "Existing base image hash: $EXISTING_HASH"
|
||||
|
||||
# Get the manifest (small JSON, ~1KB) - includes config digest
|
||||
MANIFEST=$(docker manifest inspect $BASE_IMAGE 2>/dev/null || echo "")
|
||||
CONFIG_DIGEST=$(echo "$MANIFEST" | jq -r '.config.digest // ""' 2>/dev/null || echo "")
|
||||
|
||||
if [ -n "$CONFIG_DIGEST" ] && [ "$CONFIG_DIGEST" != "null" ] && [ "$CONFIG_DIGEST" != "" ]; then
|
||||
echo "📋 Found manifest, config digest: $CONFIG_DIGEST"
|
||||
|
||||
# Fetch just the config blob (~10KB) to get labels - much faster than pulling full image
|
||||
TOKEN=$(gcloud auth print-access-token)
|
||||
CONFIG_JSON=$(curl -sL -H "Authorization: Bearer $TOKEN" \
|
||||
"https://${{ env.REGISTRY }}/v2/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}/blobs/$CONFIG_DIGEST" 2>/dev/null)
|
||||
|
||||
EXISTING_HASH=$(echo "$CONFIG_JSON" | jq -r '.config.Labels["base.content.hash"] // .config.Labels["poetry.lock.hash"] // ""' 2>/dev/null || echo "")
|
||||
echo "📋 Registry base image hash: $EXISTING_HASH"
|
||||
else
|
||||
echo "⚠️ No existing base image found - will build"
|
||||
echo "⚠️ No existing base image found in registry - will build"
|
||||
fi
|
||||
|
||||
if [ "${{ steps.base-hash.outputs.hash }}" != "$EXISTING_HASH" ]; then
|
||||
echo "🔄 Base image needs rebuild (poetry.lock or Dockerfile.base changed)"
|
||||
echo " Current: ${{ steps.base-hash.outputs.hash }}"
|
||||
echo " Registry: $EXISTING_HASH"
|
||||
echo "needs_rebuild=true" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "✅ Base image is up to date"
|
||||
@@ -1043,6 +1072,31 @@ jobs:
|
||||
# Provenance disabled for smaller image and faster push
|
||||
provenance: false
|
||||
|
||||
- name: Ensure local base image is current (if not rebuilt)
|
||||
if: steps.check-base.outputs.needs_rebuild == 'false'
|
||||
run: |
|
||||
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
|
||||
|
||||
# Get the config digest from registry manifest (identifies the image content)
|
||||
# This is the same value as Docker's Image ID for the same image
|
||||
REGISTRY_CONFIG_DIGEST=$(docker manifest inspect $BASE_IMAGE 2>/dev/null | jq -r '.config.digest' || echo "")
|
||||
|
||||
# Get local image's ID (which is the config digest)
|
||||
LOCAL_IMAGE_ID=$(docker image inspect $BASE_IMAGE --format='{{.Id}}' 2>/dev/null || echo "")
|
||||
|
||||
echo "Registry config digest: $REGISTRY_CONFIG_DIGEST"
|
||||
echo "Local image ID: $LOCAL_IMAGE_ID"
|
||||
|
||||
if [ -z "$LOCAL_IMAGE_ID" ]; then
|
||||
echo "📥 Base image not in local cache - pulling from registry"
|
||||
docker pull $BASE_IMAGE
|
||||
elif [ "$LOCAL_IMAGE_ID" != "$REGISTRY_CONFIG_DIGEST" ]; then
|
||||
echo "🔄 Local cache has different image - pulling latest"
|
||||
docker pull $BASE_IMAGE
|
||||
else
|
||||
echo "✅ Local cache matches registry - skipping pull"
|
||||
fi
|
||||
|
||||
- name: Build and push app image
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
@@ -1085,7 +1139,7 @@ jobs:
|
||||
--concurrency 1 \
|
||||
--max-instances 50 \
|
||||
--min-instances 4 \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_GCE_ENCODING=true" \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true" \
|
||||
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,SENDGRID_API_KEY=sendgrid-api-key:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest"
|
||||
|
||||
- name: Verify deployment and version
|
||||
@@ -1139,3 +1193,125 @@ jobs:
|
||||
echo ""
|
||||
echo "📊 Built on self-hosted GCP runner (200GB SSD, no disk space issues)"
|
||||
echo " Runner: gcp-runner with 4 vCPU, 16GB RAM"
|
||||
|
||||
# ============================================
|
||||
# PROPAGATE BASE IMAGE TO ALL RUNNERS
|
||||
# When base image is rebuilt, pull it to all 20 runners
|
||||
# so subsequent app builds don't wait 8 min for the pull
|
||||
#
|
||||
# NOTE: This is best-effort distribution. GitHub Actions doesn't
|
||||
# support targeting specific runners by name, so we spawn 20 jobs
|
||||
# and rely on the scheduler to distribute them across idle runners.
|
||||
# In practice, with all runners idle after deploy-backend, distribution
|
||||
# is good. Runners that miss this pull will get the image on their
|
||||
# next app build (with the 8 min delay).
|
||||
# ============================================
|
||||
|
||||
propagate-base-image:
|
||||
name: "Propagate Base Image (${{ matrix.runner_index }})"
|
||||
needs: deploy-backend
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push' && needs.deploy-backend.outputs.base_rebuilt == 'true'
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
strategy:
|
||||
matrix:
|
||||
# 20 parallel jobs - GitHub distributes across available runners
|
||||
runner_index: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
|
||||
max-parallel: 20
|
||||
fail-fast: false # Don't cancel other runners if one fails
|
||||
env:
|
||||
REGISTRY: us-central1-docker.pkg.dev
|
||||
PROJECT_ID: nomadkaraoke
|
||||
REPOSITORY: karaoke-repo
|
||||
BASE_IMAGE_NAME: karaoke-backend-base
|
||||
steps:
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Configure Docker for Artifact Registry
|
||||
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
|
||||
|
||||
- name: Pull latest base image
|
||||
run: |
|
||||
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
|
||||
RUNNER_NAME=$(hostname)
|
||||
echo "📥 Pulling latest base image on runner $RUNNER_NAME..."
|
||||
docker pull $BASE_IMAGE
|
||||
echo "✅ Base image cached on $RUNNER_NAME"
|
||||
|
||||
deploy-infrastructure:
|
||||
name: "Deploy - Infrastructure (Pulumi)"
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.13'
|
||||
|
||||
- name: Install Pulumi
|
||||
run: |
|
||||
curl -fsSL https://get.pulumi.com | sh
|
||||
echo "$HOME/.pulumi/bin" >> $GITHUB_PATH
|
||||
|
||||
- name: Install Python dependencies
|
||||
working-directory: infrastructure
|
||||
run: |
|
||||
pip install pulumi pulumi-gcp pulumi-docker
|
||||
|
||||
- name: Configure Pulumi
|
||||
env:
|
||||
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
|
||||
run: |
|
||||
pulumi login
|
||||
|
||||
- name: Deploy Infrastructure
|
||||
working-directory: infrastructure
|
||||
env:
|
||||
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
|
||||
run: |
|
||||
echo "🚀 Deploying infrastructure changes..."
|
||||
# Skip preview to avoid needing broad read permissions on all GCP resources
|
||||
# Pulumi state is authoritative since infrastructure is only managed via Pulumi
|
||||
pulumi up --yes --skip-preview --stack nomadkaraoke/karaoke-gen-infrastructure/prod
|
||||
|
||||
- name: Restart GCE Encoding Worker
|
||||
run: |
|
||||
echo "🔄 Restarting encoding worker to pick up new code..."
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
|
||||
--command="sudo systemctl restart encoding-worker" || true
|
||||
echo "✅ Encoding worker restart triggered"
|
||||
|
||||
- name: Summary
|
||||
if: success()
|
||||
run: |
|
||||
echo "🎉 Infrastructure deployment successful!"
|
||||
echo "Commit: ${{ github.sha }}"
|
||||
echo "Branch: ${{ github.ref_name }}"
|
||||
@@ -48,8 +48,10 @@ cd frontend && npm run test:all 2>&1 | tail -n 200
|
||||
- Skip for docs-only changes
|
||||
|
||||
### Infrastructure
|
||||
- All GCP changes via Pulumi in `infrastructure/`
|
||||
- `gcloud` CLI for reading/debugging only
|
||||
- **All GCP changes via Pulumi PRs** - changes in `infrastructure/` deploy automatically on merge to main
|
||||
- Never modify GCP resources directly via console or `gcloud` CLI
|
||||
- `gcloud` CLI for reading/debugging only (e.g., checking logs, SSH to VMs)
|
||||
- Local `pulumi preview` works but `pulumi up` requires broader permissions than CI
|
||||
- Stop and notify user on auth issues
|
||||
|
||||
## API Authentication (for Agents)
|
||||
|
||||
+156
-1
@@ -579,4 +579,159 @@ async def get_flacfetch_cache_stats(
|
||||
status_code=502,
|
||||
detail=f"Failed to get cache stats: {e}"
|
||||
)
|
||||
# Trigger deployment
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Job Completion Message Endpoints (for admin copy/send functionality)
|
||||
# =============================================================================
|
||||
|
||||
class CompletionMessageResponse(BaseModel):
|
||||
"""Response containing the rendered completion message."""
|
||||
job_id: str
|
||||
message: str
|
||||
subject: str
|
||||
youtube_url: Optional[str] = None
|
||||
dropbox_url: Optional[str] = None
|
||||
|
||||
|
||||
class SendCompletionEmailRequest(BaseModel):
|
||||
"""Request to send a completion email."""
|
||||
to_email: str
|
||||
cc_admin: bool = True
|
||||
|
||||
|
||||
class SendCompletionEmailResponse(BaseModel):
|
||||
"""Response from sending a completion email."""
|
||||
success: bool
|
||||
job_id: str
|
||||
to_email: str
|
||||
message: str
|
||||
|
||||
|
||||
@router.get("/jobs/{job_id}/completion-message", response_model=CompletionMessageResponse)
|
||||
async def get_job_completion_message(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Get the rendered completion message for a job.
|
||||
|
||||
Returns the plain text message that would be sent to the user,
|
||||
rendered using the job completion template with the job's details.
|
||||
|
||||
This is useful for:
|
||||
- Copying the message to clipboard (e.g., for Fiverr)
|
||||
- Previewing the email before sending
|
||||
|
||||
Requires admin authentication.
|
||||
"""
|
||||
from backend.services.job_notification_service import get_job_notification_service
|
||||
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Get youtube, dropbox URLs, and brand_code from state_data (may be None)
|
||||
state_data = job.state_data or {}
|
||||
youtube_url = state_data.get('youtube_url')
|
||||
dropbox_url = state_data.get('dropbox_link')
|
||||
brand_code = state_data.get('brand_code')
|
||||
|
||||
# Render the completion message
|
||||
notification_service = get_job_notification_service()
|
||||
message = notification_service.get_completion_message(
|
||||
job_id=job.job_id,
|
||||
user_name=None, # Use default "there"
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
youtube_url=youtube_url,
|
||||
dropbox_url=dropbox_url,
|
||||
)
|
||||
|
||||
# Build subject: "NOMAD-1178: Artist - Title (Your karaoke video is ready!)"
|
||||
if brand_code and job.artist and job.title:
|
||||
subject = f"{brand_code}: {job.artist} - {job.title} (Your karaoke video is ready!)"
|
||||
elif job.artist and job.title:
|
||||
subject = f"{job.artist} - {job.title} (Your karaoke video is ready!)"
|
||||
else:
|
||||
subject = "Your karaoke video is ready!"
|
||||
|
||||
return CompletionMessageResponse(
|
||||
job_id=job_id,
|
||||
message=message,
|
||||
subject=subject,
|
||||
youtube_url=youtube_url,
|
||||
dropbox_url=dropbox_url,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/send-completion-email", response_model=SendCompletionEmailResponse)
|
||||
async def send_job_completion_email(
|
||||
job_id: str,
|
||||
request: SendCompletionEmailRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Send a completion email for a job to a specified email address.
|
||||
|
||||
This allows admins to manually send (or re-send) completion emails,
|
||||
useful for:
|
||||
- Sending to customers who didn't have an email on file
|
||||
- Re-sending if the original email was lost
|
||||
- Sending to alternate email addresses
|
||||
|
||||
Requires admin authentication.
|
||||
"""
|
||||
from backend.services.job_notification_service import get_job_notification_service
|
||||
from backend.services.email_service import get_email_service
|
||||
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Get youtube, dropbox URLs, and brand_code from state_data (may be None)
|
||||
state_data = job.state_data or {}
|
||||
youtube_url = state_data.get('youtube_url')
|
||||
dropbox_url = state_data.get('dropbox_link')
|
||||
brand_code = state_data.get('brand_code')
|
||||
|
||||
# Render the completion message
|
||||
notification_service = get_job_notification_service()
|
||||
message = notification_service.get_completion_message(
|
||||
job_id=job.job_id,
|
||||
user_name=None, # Use default "there"
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
youtube_url=youtube_url,
|
||||
dropbox_url=dropbox_url,
|
||||
)
|
||||
|
||||
# Send the email
|
||||
email_service = get_email_service()
|
||||
success = email_service.send_job_completion(
|
||||
to_email=request.to_email,
|
||||
message_content=message,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
brand_code=brand_code,
|
||||
cc_admin=request.cc_admin,
|
||||
)
|
||||
|
||||
if success:
|
||||
logger.info(f"Admin sent completion email for job {job_id} to {request.to_email}")
|
||||
return SendCompletionEmailResponse(
|
||||
success=True,
|
||||
job_id=job_id,
|
||||
to_email=request.to_email,
|
||||
message=f"Completion email sent to {request.to_email}",
|
||||
)
|
||||
else:
|
||||
logger.error(f"Failed to send completion email for job {job_id}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to send email. Check email service configuration."
|
||||
)
|
||||
@@ -95,7 +95,7 @@ class AudioSearchRequest(BaseModel):
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
|
||||
@@ -107,6 +107,10 @@ class AudioSearchRequest(BaseModel):
|
||||
lyrics_artist: Optional[str] = Field(None, description="Override artist name for lyrics search")
|
||||
lyrics_title: Optional[str] = Field(None, description="Override title for lyrics search")
|
||||
subtitle_offset_ms: int = Field(0, description="Subtitle timing offset in milliseconds")
|
||||
|
||||
# Display overrides (optional - if empty, search values are used for display)
|
||||
display_artist: Optional[str] = Field(None, description="Artist name for title screens/filenames. If empty, uses search artist.")
|
||||
display_title: Optional[str] = Field(None, description="Title for title screens/filenames. If empty, uses search title.")
|
||||
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model: Optional[str] = Field(None, description="Model for clean instrumental separation")
|
||||
@@ -485,7 +489,8 @@ async def search_audio(
|
||||
effective_discord_webhook_url = body.discord_webhook_url or settings.default_discord_webhook_url
|
||||
|
||||
# Apply defaults for YouTube/Dropbox distribution (for web service)
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload or settings.default_enable_youtube_upload
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
effective_brand_prefix = body.brand_prefix or settings.default_brand_prefix
|
||||
effective_youtube_description = body.youtube_description or settings.default_youtube_description
|
||||
|
||||
@@ -529,10 +534,16 @@ async def search_audio(
|
||||
# Use authenticated user's email
|
||||
effective_user_email = auth_result.user_email
|
||||
|
||||
# Determine display values - use display_* if provided, otherwise fall back to search values
|
||||
# Display values are used for title screens, filenames, YouTube, etc.
|
||||
# Search values (body.artist, body.title) are used for audio search
|
||||
effective_display_artist = body.display_artist.strip() if body.display_artist else body.artist
|
||||
effective_display_title = body.display_title.strip() if body.display_title else body.title
|
||||
|
||||
# Create job
|
||||
job_create = JobCreate(
|
||||
artist=body.artist,
|
||||
title=body.title,
|
||||
artist=effective_display_artist, # Display value for title screens, filenames
|
||||
title=effective_display_title, # Display value for title screens, filenames
|
||||
theme_id=body.theme_id,
|
||||
color_overrides=body.color_overrides or {},
|
||||
enable_cdg=resolved_cdg,
|
||||
|
||||
@@ -69,7 +69,7 @@ class CreateJobFromUrlRequest(BaseModel):
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
webhook_url: Optional[str] = Field(None, description="Generic webhook URL")
|
||||
@@ -123,7 +123,7 @@ class CreateJobWithUploadUrlsRequest(BaseModel):
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
webhook_url: Optional[str] = Field(None, description="Generic webhook URL")
|
||||
@@ -406,7 +406,7 @@ async def upload_and_create_job(
|
||||
enable_txt: Optional[bool] = Form(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise"),
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Form(None, description="Brand code prefix (e.g., NOMAD)"),
|
||||
enable_youtube_upload: bool = Form(False, description="Upload to YouTube"),
|
||||
enable_youtube_upload: Optional[bool] = Form(None, description="Upload to YouTube. None = use server default"),
|
||||
youtube_description: Optional[str] = Form(None, description="YouTube video description text"),
|
||||
discord_webhook_url: Optional[str] = Form(None, description="Discord webhook URL for notifications"),
|
||||
webhook_url: Optional[str] = Form(None, description="Generic webhook URL"),
|
||||
@@ -510,12 +510,19 @@ async def upload_and_create_job(
|
||||
if dist.brand_prefix and not brand_prefix:
|
||||
logger.info(f"Using default brand_prefix: {dist.brand_prefix}")
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_enable_youtube_upload = enable_youtube_upload if enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
if effective_enable_youtube_upload and enable_youtube_upload is None:
|
||||
logger.info("Using default enable_youtube_upload: True (from env)")
|
||||
|
||||
# Validate credentials for requested distribution services (including defaults)
|
||||
# This prevents accepting jobs that will fail later due to missing credentials
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if enable_youtube_upload:
|
||||
if effective_enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
@@ -594,7 +601,7 @@ async def upload_and_create_job(
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=dist.brand_prefix,
|
||||
enable_youtube_upload=enable_youtube_upload,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=youtube_description,
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
webhook_url=webhook_url,
|
||||
@@ -815,11 +822,12 @@ async def upload_and_create_job(
|
||||
"using_default": gdrive_folder_id is None,
|
||||
}
|
||||
|
||||
if enable_youtube_upload:
|
||||
if effective_enable_youtube_upload:
|
||||
youtube_result = credential_manager.check_youtube_credentials()
|
||||
distribution_services["youtube"] = {
|
||||
"enabled": True,
|
||||
"credentials_valid": youtube_result.status == CredentialStatus.VALID,
|
||||
"using_default": enable_youtube_upload is None,
|
||||
}
|
||||
|
||||
if dist.discord_webhook_url:
|
||||
@@ -908,7 +916,7 @@ class CreateFinaliseOnlyJobRequest(BaseModel):
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
keep_brand_code: Optional[str] = Field(None, description="Preserve existing brand code from folder name")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
|
||||
@@ -1048,11 +1056,16 @@ async def create_job_with_upload_urls(
|
||||
brand_prefix=body.brand_prefix,
|
||||
)
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
|
||||
# Validate credentials for requested distribution services
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if body.enable_youtube_upload:
|
||||
if effective_enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
@@ -1105,7 +1118,7 @@ async def create_job_with_upload_urls(
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=dist.brand_prefix,
|
||||
enable_youtube_upload=body.enable_youtube_upload,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=body.youtube_description,
|
||||
youtube_description_template=body.youtube_description, # video_worker reads this field
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
@@ -1466,11 +1479,16 @@ async def create_job_from_url(
|
||||
brand_prefix=body.brand_prefix,
|
||||
)
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
|
||||
# Validate credentials for requested distribution services
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if body.enable_youtube_upload:
|
||||
if effective_enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
@@ -1522,7 +1540,7 @@ async def create_job_from_url(
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=dist.brand_prefix,
|
||||
enable_youtube_upload=body.enable_youtube_upload,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=body.youtube_description,
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
webhook_url=body.webhook_url,
|
||||
@@ -1712,11 +1730,16 @@ async def create_finalise_only_job(
|
||||
brand_prefix=body.brand_prefix,
|
||||
)
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
|
||||
# Validate distribution credentials if services are requested
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if body.enable_youtube_upload:
|
||||
if effective_enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
@@ -1766,7 +1789,7 @@ async def create_finalise_only_job(
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=dist.brand_prefix,
|
||||
enable_youtube_upload=body.enable_youtube_upload,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=body.youtube_description,
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
dropbox_path=dist.dropbox_path,
|
||||
|
||||
@@ -227,6 +227,7 @@ async def check_encoding_worker_status() -> Dict[str, Any]:
|
||||
"active_jobs": health.get("active_jobs", 0),
|
||||
"queue_length": health.get("queue_length", 0),
|
||||
"ffmpeg_version": health.get("ffmpeg_version"),
|
||||
"wheel_version": health.get("wheel_version"),
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
@@ -237,6 +238,39 @@ async def check_encoding_worker_status() -> Dict[str, Any]:
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/encoding-worker")
|
||||
async def encoding_worker_health() -> Dict[str, Any]:
|
||||
"""
|
||||
Lightweight endpoint to check encoding worker status.
|
||||
|
||||
Returns minimal info for frontend footer display.
|
||||
No authentication required.
|
||||
"""
|
||||
status = await check_encoding_worker_status()
|
||||
|
||||
# Return simplified response for frontend
|
||||
if not status.get("configured"):
|
||||
return {
|
||||
"available": False,
|
||||
"status": "not_configured",
|
||||
}
|
||||
|
||||
if not status.get("available"):
|
||||
return {
|
||||
"available": False,
|
||||
"status": "offline",
|
||||
"error": status.get("error"),
|
||||
}
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"status": "ok",
|
||||
"version": status.get("wheel_version"),
|
||||
"active_jobs": status.get("active_jobs", 0),
|
||||
"queue_length": status.get("queue_length", 0),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/detailed")
|
||||
async def detailed_health_check() -> Dict[str, Any]:
|
||||
"""
|
||||
|
||||
@@ -316,13 +316,118 @@ async def trigger_render_video_worker(
|
||||
)
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/check-idle-reminder")
|
||||
async def check_idle_reminder(
|
||||
job_id: str,
|
||||
http_request: Request,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Check if a job needs an idle reminder email.
|
||||
|
||||
This endpoint is called by a Cloud Tasks scheduled task 5 minutes after
|
||||
a job enters a blocking state (AWAITING_REVIEW or AWAITING_INSTRUMENTAL_SELECTION).
|
||||
|
||||
If the job is still in the blocking state and no reminder has been sent yet,
|
||||
sends a reminder email to the user.
|
||||
|
||||
Idempotency: Only one reminder per job (tracked via reminder_sent flag).
|
||||
"""
|
||||
from backend.models.job import JobStatus
|
||||
from backend.services.job_notification_service import get_job_notification_service
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] IDLE_REMINDER_CHECK starting")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("operation", "idle_reminder_check")
|
||||
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
logger.warning(f"[job:{job_id}] Job not found for idle reminder check")
|
||||
add_span_event("job_not_found")
|
||||
return {"status": "not_found", "job_id": job_id, "message": "Job not found"}
|
||||
|
||||
# Check if job is still in a blocking state
|
||||
blocking_states = [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION]
|
||||
if job.status not in [s.value for s in blocking_states]:
|
||||
logger.info(f"[job:{job_id}] Job no longer in blocking state ({job.status}), skipping reminder")
|
||||
add_span_event("not_blocking", {"current_status": job.status})
|
||||
return {
|
||||
"status": "skipped",
|
||||
"job_id": job_id,
|
||||
"message": f"Job not in blocking state (current: {job.status})"
|
||||
}
|
||||
|
||||
# Normalize state_data to prevent None errors
|
||||
state_data = job.state_data or {}
|
||||
|
||||
# Check if reminder was already sent (idempotency)
|
||||
if state_data.get('reminder_sent'):
|
||||
logger.info(f"[job:{job_id}] Reminder already sent, skipping")
|
||||
add_span_event("already_sent")
|
||||
return {"status": "already_sent", "job_id": job_id, "message": "Reminder already sent"}
|
||||
|
||||
# Check if user has an email
|
||||
if not job.user_email:
|
||||
logger.warning(f"[job:{job_id}] No user email, cannot send reminder")
|
||||
add_span_event("no_email")
|
||||
return {"status": "no_email", "job_id": job_id, "message": "No user email configured"}
|
||||
|
||||
# Determine action type
|
||||
action_type = state_data.get('blocking_action_type')
|
||||
if not action_type:
|
||||
action_type = "lyrics" if job.status == JobStatus.AWAITING_REVIEW.value else "instrumental"
|
||||
|
||||
# Send the reminder email
|
||||
try:
|
||||
notification_service = get_job_notification_service()
|
||||
|
||||
success = await notification_service.send_action_reminder_email(
|
||||
job_id=job.job_id,
|
||||
user_email=job.user_email,
|
||||
action_type=action_type,
|
||||
user_name=None, # Could fetch from user service if needed
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
audio_hash=job.audio_hash,
|
||||
review_token=job.review_token,
|
||||
instrumental_token=job.instrumental_token,
|
||||
)
|
||||
|
||||
if success:
|
||||
# Mark reminder as sent (prevents duplicate sends)
|
||||
job_manager.firestore.update_job(job_id, {
|
||||
'state_data': {**state_data, 'reminder_sent': True}
|
||||
})
|
||||
logger.info(f"[job:{job_id}] Sent {action_type} reminder email to {job.user_email}")
|
||||
add_span_event("reminder_sent", {"action_type": action_type})
|
||||
return {
|
||||
"status": "sent",
|
||||
"job_id": job_id,
|
||||
"message": f"Sent {action_type} reminder to {job.user_email}"
|
||||
}
|
||||
else:
|
||||
logger.error(f"[job:{job_id}] Failed to send reminder email")
|
||||
add_span_event("send_failed")
|
||||
return {"status": "failed", "job_id": job_id, "message": "Failed to send reminder"}
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"[job:{job_id}] Error sending reminder: {e}")
|
||||
add_span_event("error", {"error": str(e)})
|
||||
return {"status": "error", "job_id": job_id, "message": str(e)}
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def internal_health(
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Internal health check endpoint.
|
||||
|
||||
|
||||
Used to verify the internal API is responsive.
|
||||
Requires admin authentication.
|
||||
"""
|
||||
|
||||
@@ -87,6 +87,11 @@ async def create_job(
|
||||
user_email = request.user_email
|
||||
logger.info(f"Admin {auth_result.user_email} creating job on behalf of {user_email}")
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_enable_youtube_upload = request.enable_youtube_upload if request.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
|
||||
# Create job with all preferences
|
||||
job_create = JobCreate(
|
||||
url=str(request.url),
|
||||
@@ -94,7 +99,7 @@ async def create_job(
|
||||
title=request.title,
|
||||
enable_cdg=request.enable_cdg,
|
||||
enable_txt=request.enable_txt,
|
||||
enable_youtube_upload=request.enable_youtube_upload,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=request.youtube_description,
|
||||
webhook_url=request.webhook_url,
|
||||
user_email=user_email
|
||||
|
||||
+122
-40
@@ -27,8 +27,10 @@ from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.job_logging import job_log_context, JobLogger
|
||||
from backend.services.tracing import create_span, add_span_attribute, add_span_event
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
from backend.api.dependencies import require_auth, require_review_auth
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.config import get_settings
|
||||
|
||||
# LyricsTranscriber imports for preview generation
|
||||
from lyrics_transcriber.types import CorrectionResult
|
||||
@@ -385,28 +387,35 @@ async def generate_preview_video(
|
||||
):
|
||||
"""
|
||||
Generate a preview video with the current corrections.
|
||||
|
||||
|
||||
Uses the LyricsTranscriber's CorrectionOperations to generate a 360p preview
|
||||
video with the user's current corrections applied.
|
||||
|
||||
When USE_GCE_PREVIEW_ENCODING is enabled, video encoding is offloaded to
|
||||
the high-performance GCE worker for faster generation (15-20s vs 60+s).
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
settings = get_settings()
|
||||
encoding_service = get_encoding_service()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Job must be in review state to generate preview
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job not in review state (current status: {job.status})"
|
||||
)
|
||||
|
||||
logger.info(f"Job {job_id}: Generating preview video")
|
||||
|
||||
|
||||
# Check if GCE preview encoding is enabled
|
||||
use_gce_preview = encoding_service.is_preview_enabled
|
||||
logger.info(f"Job {job_id}: Generating preview video (GCE preview: {use_gce_preview})")
|
||||
|
||||
# Use tracing and job_log_context for full observability
|
||||
with create_span("generate-preview-video", {"job_id": job_id}) as span:
|
||||
with create_span("generate-preview-video", {"job_id": job_id, "use_gce": use_gce_preview}) as span:
|
||||
with job_log_context(job_id, worker="preview"):
|
||||
try:
|
||||
# Create temp directory for this preview operation
|
||||
@@ -416,19 +425,19 @@ async def generate_preview_video(
|
||||
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
corrections_path = os.path.join(temp_dir, "corrections.json")
|
||||
storage.download_file(corrections_gcs, corrections_path)
|
||||
|
||||
|
||||
with open(corrections_path, 'r', encoding='utf-8') as f:
|
||||
original_data = json.load(f)
|
||||
|
||||
|
||||
# 2. Download input audio
|
||||
audio_path = os.path.join(temp_dir, "audio.flac")
|
||||
storage.download_file(job.input_media_gcs_path, audio_path)
|
||||
download_span.set_attribute("audio_gcs_path", job.input_media_gcs_path)
|
||||
|
||||
|
||||
# 3. Load original as CorrectionResult
|
||||
correction_result = CorrectionResult.from_dict(original_data)
|
||||
add_span_event("corrections_loaded")
|
||||
|
||||
|
||||
# 4. Get or create styles file for preview using unified style loader
|
||||
with create_span("load-styles") as styles_span:
|
||||
styles_path, _ = load_styles_from_gcs(
|
||||
@@ -439,54 +448,127 @@ async def generate_preview_video(
|
||||
logger=logger,
|
||||
)
|
||||
styles_span.set_attribute("styles_path", styles_path)
|
||||
|
||||
|
||||
# 5. Set up output config for preview
|
||||
output_dir = os.path.join(temp_dir, "output")
|
||||
cache_dir = os.path.join(temp_dir, "cache")
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
|
||||
|
||||
output_config = OutputConfig(
|
||||
output_styles_json=styles_path,
|
||||
output_dir=output_dir,
|
||||
cache_dir=cache_dir,
|
||||
video_resolution="360p",
|
||||
)
|
||||
|
||||
# 6. Generate preview video using CorrectionOperations (heavy operation)
|
||||
with create_span("render-preview-video") as render_span:
|
||||
render_span.set_attribute("resolution", "360p")
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
correction_result=correction_result,
|
||||
updated_data=updated_data,
|
||||
output_config=output_config,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
logger=logger
|
||||
)
|
||||
add_span_event("render_complete")
|
||||
|
||||
# 7. Upload preview video to GCS for serving
|
||||
preview_hash = result["preview_hash"]
|
||||
video_path = result["video_path"]
|
||||
|
||||
with create_span("upload-preview-video") as upload_span:
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
storage.upload_file(video_path, preview_gcs_path)
|
||||
upload_span.set_attribute("gcs_path", preview_gcs_path)
|
||||
|
||||
|
||||
# 6. Generate preview (ASS-only if using GCE, or full video if local)
|
||||
preview_gcs_path = None
|
||||
|
||||
if use_gce_preview:
|
||||
# GCE path: Generate ASS only, then offload encoding to GCE
|
||||
try:
|
||||
with create_span("generate-ass-subtitles") as ass_span:
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
correction_result=correction_result,
|
||||
updated_data=updated_data,
|
||||
output_config=output_config,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
logger=logger,
|
||||
ass_only=True, # Only generate ASS, skip video encoding
|
||||
)
|
||||
preview_hash = result["preview_hash"]
|
||||
ass_path = result["ass_path"]
|
||||
ass_span.set_attribute("ass_path", ass_path)
|
||||
add_span_event("ass_generated")
|
||||
|
||||
# Upload ASS to GCS
|
||||
with create_span("upload-ass-to-gcs") as upload_ass_span:
|
||||
ass_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.ass"
|
||||
storage.upload_file(ass_path, ass_gcs_path)
|
||||
upload_ass_span.set_attribute("ass_gcs_path", ass_gcs_path)
|
||||
|
||||
# Call GCE encoding service
|
||||
with create_span("gce-preview-encoding") as gce_span:
|
||||
bucket_name = settings.gcs_bucket_name
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
|
||||
# Get background image and font from style assets if available
|
||||
style_assets = job.style_assets or {}
|
||||
|
||||
background_image_gcs_path = None
|
||||
for key in ["karaoke_background", "style_karaoke_background"]:
|
||||
if key in style_assets:
|
||||
background_image_gcs_path = f"gs://{bucket_name}/{style_assets[key]}"
|
||||
gce_span.set_attribute("background_image", background_image_gcs_path)
|
||||
break
|
||||
|
||||
font_gcs_path = None
|
||||
for key in ["font", "style_font"]:
|
||||
if key in style_assets:
|
||||
font_gcs_path = f"gs://{bucket_name}/{style_assets[key]}"
|
||||
gce_span.set_attribute("font", font_gcs_path)
|
||||
break
|
||||
|
||||
gce_result = await encoding_service.encode_preview_video(
|
||||
job_id=f"preview_{job_id}_{preview_hash}",
|
||||
ass_gcs_path=f"gs://{bucket_name}/{ass_gcs_path}",
|
||||
audio_gcs_path=f"gs://{bucket_name}/{job.input_media_gcs_path}",
|
||||
output_gcs_path=f"gs://{bucket_name}/{preview_gcs_path}",
|
||||
background_color="black",
|
||||
background_image_gcs_path=background_image_gcs_path,
|
||||
font_gcs_path=font_gcs_path,
|
||||
)
|
||||
gce_span.set_attribute("gce_status", gce_result.get("status"))
|
||||
add_span_event("gce_encoding_complete")
|
||||
|
||||
logger.info(f"Job {job_id}: Preview generated via GCE: {preview_hash}")
|
||||
|
||||
except Exception as gce_error:
|
||||
# Fall back to local encoding if GCE fails
|
||||
logger.warning(
|
||||
f"Job {job_id}: GCE preview encoding failed, falling back to local: {gce_error}"
|
||||
)
|
||||
span.set_attribute("gce_fallback", True)
|
||||
use_gce_preview = False # Fall through to local encoding below
|
||||
|
||||
if not use_gce_preview:
|
||||
# Local path: Generate full preview video locally
|
||||
with create_span("render-preview-video-local") as render_span:
|
||||
render_span.set_attribute("resolution", "360p")
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
correction_result=correction_result,
|
||||
updated_data=updated_data,
|
||||
output_config=output_config,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
logger=logger,
|
||||
ass_only=False, # Generate full video locally
|
||||
)
|
||||
preview_hash = result["preview_hash"]
|
||||
video_path = result["video_path"]
|
||||
add_span_event("render_complete")
|
||||
|
||||
# Upload preview video to GCS
|
||||
with create_span("upload-preview-video") as upload_span:
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
storage.upload_file(video_path, preview_gcs_path)
|
||||
upload_span.set_attribute("gcs_path", preview_gcs_path)
|
||||
|
||||
# Store the GCS path for serving
|
||||
if job_id not in _preview_videos:
|
||||
_preview_videos[job_id] = {}
|
||||
_preview_videos[job_id][preview_hash] = preview_gcs_path
|
||||
|
||||
|
||||
logger.info(f"Job {job_id}: Preview video generated: {preview_hash}")
|
||||
span.set_attribute("preview_hash", preview_hash)
|
||||
span.set_attribute("success", True)
|
||||
|
||||
|
||||
return {"status": "success", "preview_hash": preview_hash}
|
||||
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Failed to generate preview video: {e}", exc_info=True)
|
||||
span.set_attribute("error", str(e))
|
||||
|
||||
@@ -68,6 +68,12 @@ class Settings(BaseSettings):
|
||||
use_gce_encoding: bool = os.getenv("USE_GCE_ENCODING", "false").lower() in ("true", "1", "yes")
|
||||
encoding_worker_url: Optional[str] = os.getenv("ENCODING_WORKER_URL") # e.g., http://136.119.50.148:8080
|
||||
encoding_worker_api_key: Optional[str] = os.getenv("ENCODING_WORKER_API_KEY")
|
||||
|
||||
# GCE Preview Encoding (for faster preview video generation)
|
||||
# When enabled, preview video encoding during lyrics review is offloaded to the GCE worker.
|
||||
# This reduces preview generation time from 60+ seconds to ~15-20 seconds.
|
||||
# Requires use_gce_encoding to be enabled and the GCE worker to support /encode-preview endpoint.
|
||||
use_gce_preview_encoding: bool = os.getenv("USE_GCE_PREVIEW_ENCODING", "false").lower() in ("true", "1", "yes")
|
||||
|
||||
# Storage paths
|
||||
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
|
||||
|
||||
@@ -14,7 +14,7 @@ class URLSubmissionRequest(BaseModel):
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: bool = False
|
||||
enable_youtube_upload: Optional[bool] = None # None = use server default
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
@@ -28,7 +28,7 @@ class UploadSubmissionRequest(BaseModel):
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: bool = False
|
||||
enable_youtube_upload: Optional[bool] = None # None = use server default
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
|
||||
@@ -14,7 +14,7 @@ import html
|
||||
import logging
|
||||
import os
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Optional
|
||||
from typing import Optional, List
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
@@ -31,7 +31,8 @@ class EmailProvider(ABC):
|
||||
to_email: str,
|
||||
subject: str,
|
||||
html_content: str,
|
||||
text_content: Optional[str] = None
|
||||
text_content: Optional[str] = None,
|
||||
cc_emails: Optional[List[str]] = None,
|
||||
) -> bool:
|
||||
"""Send an email. Returns True if successful."""
|
||||
pass
|
||||
@@ -49,10 +50,13 @@ class ConsoleEmailProvider(EmailProvider):
|
||||
to_email: str,
|
||||
subject: str,
|
||||
html_content: str,
|
||||
text_content: Optional[str] = None
|
||||
text_content: Optional[str] = None,
|
||||
cc_emails: Optional[List[str]] = None,
|
||||
) -> bool:
|
||||
logger.info("=" * 60)
|
||||
logger.info(f"EMAIL TO: {to_email}")
|
||||
if cc_emails:
|
||||
logger.info(f"CC: {', '.join(cc_emails)}")
|
||||
logger.info(f"SUBJECT: {subject}")
|
||||
logger.info("-" * 60)
|
||||
logger.info(text_content or html_content)
|
||||
@@ -77,12 +81,13 @@ class SendGridEmailProvider(EmailProvider):
|
||||
to_email: str,
|
||||
subject: str,
|
||||
html_content: str,
|
||||
text_content: Optional[str] = None
|
||||
text_content: Optional[str] = None,
|
||||
cc_emails: Optional[List[str]] = None,
|
||||
) -> bool:
|
||||
try:
|
||||
# Import here to avoid requiring sendgrid in all environments
|
||||
from sendgrid import SendGridAPIClient
|
||||
from sendgrid.helpers.mail import Mail, Email, To, Content
|
||||
from sendgrid.helpers.mail import Mail, Email, To, Content, Cc
|
||||
|
||||
sg = SendGridAPIClient(api_key=self.api_key)
|
||||
|
||||
@@ -93,13 +98,28 @@ class SendGridEmailProvider(EmailProvider):
|
||||
html_content=Content("text/html", html_content)
|
||||
)
|
||||
|
||||
# Add CC recipients if provided (deduplicated and normalized)
|
||||
if cc_emails:
|
||||
# Normalize emails (lowercase, strip whitespace) and deduplicate
|
||||
seen = set()
|
||||
unique_cc_emails = []
|
||||
for cc_email in cc_emails:
|
||||
normalized = cc_email.strip().lower()
|
||||
if normalized and normalized not in seen:
|
||||
seen.add(normalized)
|
||||
unique_cc_emails.append(cc_email.strip())
|
||||
|
||||
for cc_email in unique_cc_emails:
|
||||
message.add_cc(Cc(cc_email))
|
||||
|
||||
if text_content:
|
||||
message.add_content(Content("text/plain", text_content))
|
||||
|
||||
response = sg.send(message)
|
||||
|
||||
if response.status_code >= 200 and response.status_code < 300:
|
||||
logger.info(f"Email sent to {to_email} via SendGrid")
|
||||
cc_info = f" (CC: {', '.join(cc_emails)})" if cc_emails else ""
|
||||
logger.info(f"Email sent to {to_email}{cc_info} via SendGrid")
|
||||
return True
|
||||
else:
|
||||
logger.error(f"SendGrid returned status {response.status_code}")
|
||||
@@ -816,6 +836,398 @@ Thanks for being part of making Nomad Karaoke better!
|
||||
from datetime import datetime
|
||||
return datetime.now().year
|
||||
|
||||
def _get_email_signature(self) -> str:
|
||||
"""Get the HTML email signature for Nomad Karaoke."""
|
||||
return """
|
||||
<table cellpadding="0" cellspacing="0" border="0" style="font-size: medium; font-family: Trebuchet MS;">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>
|
||||
<table cellpadding="0" cellspacing="0" border="0" style="font-size: medium; font-family: Trebuchet MS;">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style="vertical-align: top;">
|
||||
<table cellpadding="0" cellspacing="0" border="0"
|
||||
style="font-size: medium; font-family: Trebuchet MS;">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style="text-align: center;"><a
|
||||
href="https://www.linkedin.com/in/andrewbeveridge"
|
||||
target="_blank"><img
|
||||
src="https://beveradb.github.io/public-images/andrew-buildspace-circle-150px.png"
|
||||
role="presentation" style="display: block; max-width: 128px;"
|
||||
width="130"></a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td height="5"></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align: center;"><a href="https://nomadkaraoke.com"
|
||||
target="_blank"><img role="presentation" width="130"
|
||||
style="display: block; max-width: 130px; border-radius: 7px;"
|
||||
src="https://beveradb.github.io/public-images/Nomad-Karaoke-Logo-small-indexed-websafe-rectangle.gif"></a>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</td>
|
||||
<td width="10">
|
||||
<div></div>
|
||||
</td>
|
||||
<td style="padding: 0px; vertical-align: middle;">
|
||||
<h2 color="#000000"
|
||||
style="margin: 0px; font-size: 18px; color: rgb(0, 0, 0); font-weight: 600;">
|
||||
<span>Andrew</span><span> </span><span>Beveridge</span>
|
||||
</h2>
|
||||
<p color="#000000" font-size="medium"
|
||||
style="margin: 0px; color: rgb(0, 0, 0); font-size: 14px; line-height: 22px;">
|
||||
<span>Founder</span>
|
||||
</p>
|
||||
<p color="#000000" font-size="medium"
|
||||
style="margin: 0px; font-weight: 500; color: rgb(0, 0, 0); font-size: 14px; line-height: 22px;">
|
||||
<span>Nomad Karaoke
|
||||
LLC</span>
|
||||
</p>
|
||||
<table cellpadding="0" cellspacing="0" border="0"
|
||||
style="width: 100%; font-size: medium; font-family: Trebuchet MS; margin-top: 3px; margin-bottom: 3px;">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td color="#ff7acc" direction="horizontal" width="auto" height="15"
|
||||
style="width: 100%; display: block; line-height:0; font-size:0;">
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td color="#ff7acc" direction="horizontal" width="auto" height="1"
|
||||
style="width: 100%; border-bottom: 1px solid rgb(255, 122, 204); border-left: medium; display: block; line-height:0; font-size:0;">
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<table cellpadding="0" cellspacing="0" border="0"
|
||||
style="width: 100%; font-size: medium; font-family: Trebuchet MS; margin-top: 3px; margin-bottom: 3px;">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style="text-align: center; line-height:0; font-size:0;">
|
||||
<table cellpadding="0" cellspacing="0" border="0"
|
||||
style="display: inline-block; font-size: medium; font-family: Trebuchet MS;">
|
||||
<tbody>
|
||||
<tr style="text-align: center;">
|
||||
<td><a href="https://www.youtube.com/@nomadkaraoke"
|
||||
color="#230a89"
|
||||
style="display: inline-block; padding: 0px; background-color: rgb(35, 10, 137);"
|
||||
target="_blank"><img
|
||||
src="https://beveradb.github.io/public-images/youtube-icon-2x.png"
|
||||
alt="youtube" color="#230a89" width="24"
|
||||
style="background-color: rgb(35, 10, 137); max-width: 135px; display: block;"></a>
|
||||
</td>
|
||||
<td width="5">
|
||||
<div></div>
|
||||
</td>
|
||||
<td><a href="https://github.com/nomadkaraoke"
|
||||
color="#230a89"
|
||||
style="display: inline-block; padding: 0px; background-color: rgb(35, 10, 137);"
|
||||
target="_blank"><img
|
||||
src="https://beveradb.github.io/public-images/github-icon-2x.png"
|
||||
alt="github" color="#230a89" width="24"
|
||||
style="background-color: rgb(35, 10, 137); max-width: 135px; display: block;"></a>
|
||||
</td>
|
||||
<td width="5">
|
||||
<div></div>
|
||||
</td>
|
||||
<td><a href="https://www.linkedin.com/in/andrewbeveridge"
|
||||
color="#230a89"
|
||||
style="display: inline-block; padding: 0px; background-color: rgb(35, 10, 137);"
|
||||
target="_blank"><img
|
||||
src="https://beveradb.github.io/public-images/linkedin-icon-2x.png"
|
||||
alt="linkedin" color="#230a89" width="24"
|
||||
style="background-color: rgb(35, 10, 137); max-width: 135px; display: block;"></a>
|
||||
</td>
|
||||
<td width="5">
|
||||
<div></div>
|
||||
</td>
|
||||
<td><a href="https://twitter.com/beveradb" color="#230a89"
|
||||
style="display: inline-block; padding: 0px; background-color: rgb(35, 10, 137);"><img
|
||||
src="https://beveradb.github.io/public-images/twitter-icon-2x.png"
|
||||
alt="twitter" color="#230a89" width="24"
|
||||
style="background-color: rgb(35, 10, 137); max-width: 135px; display: block;"></a>
|
||||
</td>
|
||||
<td width="5">
|
||||
<div></div>
|
||||
</td>
|
||||
<td><a href="https://www.instagram.com/beveradb/"
|
||||
color="#230a89"
|
||||
style="display: inline-block; padding: 0px; background-color: rgb(35, 10, 137);"><img
|
||||
src="https://beveradb.github.io/public-images/instagram-icon-2x.png"
|
||||
alt="instagram" color="#230a89" width="24"
|
||||
style="background-color: rgb(35, 10, 137); max-width: 135px; display: block;"></a>
|
||||
</td>
|
||||
<td width="5">
|
||||
<div></div>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<table cellpadding="0" cellspacing="0" border="0"
|
||||
style="width: 100%; font-size: medium; font-family: Trebuchet MS; margin-top: 3px; margin-bottom: 3px;">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td color="#ff7acc" direction="horizontal" width="auto" height="1"
|
||||
style="width: 100%; border-bottom: 1px solid rgb(255, 122, 204); border-left: medium; display: block; line-height:0; font-size:0;">
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td color="#ff7acc" direction="horizontal" width="auto" height="15"
|
||||
style="width: 100%; display: block;">
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<table cellpadding="0" cellspacing="0" border="0"
|
||||
style="font-size: medium; font-family: Trebuchet MS;">
|
||||
<tbody>
|
||||
<tr style="vertical-align: middle;" height="20">
|
||||
<td width="30" style="vertical-align: middle;">
|
||||
<table cellpadding="0" cellspacing="0" border="0"
|
||||
style="font-size: medium; font-family: Trebuchet MS;">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style="vertical-align: bottom;"><span color="#ff7acc"
|
||||
width="11"
|
||||
style="display: inline-block; background-color: rgb(255, 122, 204);"><img
|
||||
src="https://beveradb.github.io/public-images/phone-icon-2x.png"
|
||||
color="#ff7acc" alt="mobilePhone" width="13"
|
||||
style="display: block; background-color: rgb(255, 122, 204);"></span>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</td>
|
||||
<td style="padding: 0px; color: rgb(0, 0, 0);"><a href="tel:8036363267"
|
||||
color="#000000"
|
||||
style="text-decoration: none; color: rgb(0, 0, 0); font-size: 12px;"><span>+1 (803) 636-3267</span></a> | <a href="tel:07835171222"
|
||||
color="#000000"
|
||||
style="text-decoration: none; color: rgb(0, 0, 0); font-size: 12px;"><span>+44 07835171222</span></a></td>
|
||||
</tr>
|
||||
<tr style="vertical-align: middle;" height="20">
|
||||
<td width="30" style="vertical-align: middle;">
|
||||
<table cellpadding="0" cellspacing="0" border="0"
|
||||
style="font-size: medium; font-family: Trebuchet MS;">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style="vertical-align: bottom;"><span color="#ff7acc"
|
||||
width="11"
|
||||
style="display: inline-block; background-color: rgb(255, 122, 204);"><img
|
||||
src="https://beveradb.github.io/public-images/email-icon-2x.png"
|
||||
color="#ff7acc" alt="emailAddress" width="13"
|
||||
style="display: block; background-color: rgb(255, 122, 204);"></span>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</td>
|
||||
<td style="padding: 0px;"><a href="mailto:andrew@nomadkaraoke.com"
|
||||
color="#000000"
|
||||
style="text-decoration: none; color: rgb(0, 0, 0); font-size: 12px;"><span>andrew@nomadkaraoke.com</span></a>
|
||||
</td>
|
||||
</tr>
|
||||
<tr style="vertical-align: middle;" height="20">
|
||||
<td width="30" style="vertical-align: middle;">
|
||||
<table cellpadding="0" cellspacing="0" border="0"
|
||||
style="font-size: medium; font-family: Trebuchet MS;">
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style="vertical-align: bottom;"><span color="#ff7acc"
|
||||
width="11"
|
||||
style="display: inline-block; background-color: rgb(255, 122, 204);"><img
|
||||
src="https://beveradb.github.io/public-images/link-icon-2x.png"
|
||||
color="#ff7acc" alt="website" width="13"
|
||||
style="display: block; background-color: rgb(255, 122, 204);"></span>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</td>
|
||||
<td style="padding: 0px;"><a href="https://nomadkaraoke.com" target="_blank"
|
||||
color="#000000"
|
||||
style="text-decoration: none; color: rgb(0, 0, 0); font-size: 12px;"><span>nomadkaraoke.com</span></a>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
"""
|
||||
|
||||
def send_job_completion(
|
||||
self,
|
||||
to_email: str,
|
||||
message_content: str,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
brand_code: Optional[str] = None,
|
||||
cc_admin: bool = True,
|
||||
) -> bool:
|
||||
"""
|
||||
Send job completion email with the rendered template content.
|
||||
|
||||
Args:
|
||||
to_email: User's email address
|
||||
message_content: Pre-rendered message content (plain text)
|
||||
artist: Artist name for subject line
|
||||
title: Song title for subject line
|
||||
brand_code: Release ID (e.g., "NOMAD-1178") for subject line
|
||||
cc_admin: Whether to CC gen@nomadkaraoke.com
|
||||
|
||||
Returns:
|
||||
True if email was sent successfully
|
||||
"""
|
||||
# Build subject: "NOMAD-1178: Artist - Title (Your karaoke video is ready!)"
|
||||
if brand_code and artist and title:
|
||||
subject = f"{brand_code}: {artist} - {title} (Your karaoke video is ready!)"
|
||||
elif artist and title:
|
||||
subject = f"{artist} - {title} (Your karaoke video is ready!)"
|
||||
else:
|
||||
subject = "Your karaoke video is ready!"
|
||||
|
||||
# Convert plain text to simple HTML (preserve line breaks)
|
||||
html_content = f"""
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<style>
|
||||
body {{
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
line-height: 1.6;
|
||||
color: #333;
|
||||
max-width: 600px;
|
||||
margin: 0 auto;
|
||||
padding: 20px;
|
||||
white-space: pre-wrap;
|
||||
}}
|
||||
.content {{
|
||||
white-space: pre-wrap;
|
||||
font-size: 14px;
|
||||
}}
|
||||
.signature {{
|
||||
margin-top: 30px;
|
||||
padding-top: 20px;
|
||||
}}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="content">{html.escape(message_content)}</div>
|
||||
|
||||
<div class="signature">
|
||||
{self._get_email_signature()}
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
cc_emails = ["gen@nomadkaraoke.com"] if cc_admin else None
|
||||
|
||||
return self.provider.send_email(
|
||||
to_email=to_email,
|
||||
subject=subject,
|
||||
html_content=html_content,
|
||||
text_content=message_content,
|
||||
cc_emails=cc_emails,
|
||||
)
|
||||
|
||||
def send_action_reminder(
|
||||
self,
|
||||
to_email: str,
|
||||
message_content: str,
|
||||
action_type: str,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Send action-needed reminder email.
|
||||
|
||||
Args:
|
||||
to_email: User's email address
|
||||
message_content: Pre-rendered message content (plain text)
|
||||
action_type: Type of action needed ("lyrics" or "instrumental")
|
||||
artist: Artist name for subject line
|
||||
title: Song title for subject line
|
||||
|
||||
Returns:
|
||||
True if email was sent successfully
|
||||
"""
|
||||
# Build subject based on action type
|
||||
song_info = f" for {artist} - {title}" if artist and title else ""
|
||||
if action_type == "lyrics":
|
||||
subject = f"Action needed: Review lyrics{song_info}"
|
||||
elif action_type == "instrumental":
|
||||
subject = f"Action needed: Select instrumental{song_info}"
|
||||
else:
|
||||
subject = f"Action needed{song_info}"
|
||||
|
||||
# Convert plain text to simple HTML
|
||||
html_content = f"""
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<style>
|
||||
body {{
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
line-height: 1.6;
|
||||
color: #333;
|
||||
max-width: 600px;
|
||||
margin: 0 auto;
|
||||
padding: 20px;
|
||||
}}
|
||||
.alert {{
|
||||
background-color: #fef3c7;
|
||||
border: 1px solid #fcd34d;
|
||||
border-radius: 8px;
|
||||
padding: 16px;
|
||||
margin: 20px 0;
|
||||
text-align: center;
|
||||
}}
|
||||
.content {{
|
||||
white-space: pre-wrap;
|
||||
font-size: 14px;
|
||||
}}
|
||||
.signature {{
|
||||
margin-top: 30px;
|
||||
padding-top: 20px;
|
||||
}}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="alert">
|
||||
⏰ Your karaoke video is waiting for you!
|
||||
</div>
|
||||
|
||||
<div class="content">{html.escape(message_content)}</div>
|
||||
|
||||
<div class="signature">
|
||||
{self._get_email_signature()}
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
return self.provider.send_email(
|
||||
to_email=to_email,
|
||||
subject=subject,
|
||||
html_content=html_content,
|
||||
text_content=message_content,
|
||||
)
|
||||
|
||||
|
||||
# Global instance
|
||||
_email_service = None
|
||||
|
||||
@@ -351,8 +351,8 @@ class GCEEncodingBackend(EncodingBackend):
|
||||
result = {}
|
||||
raw_output_files = result.get("output_files", {})
|
||||
|
||||
# Convert output_files from list of paths to dict if needed
|
||||
# GCE worker returns list like: ["path/output_4k_lossless.mp4", "path/output_720p.mp4"]
|
||||
# Convert output_files from list of paths to dict
|
||||
# GCE worker returns list like: ["path/Artist - Title (Final Karaoke Lossless 4k).mp4", ...]
|
||||
# We need dict like: {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
|
||||
if isinstance(raw_output_files, list):
|
||||
output_files = {}
|
||||
@@ -360,15 +360,18 @@ class GCEEncodingBackend(EncodingBackend):
|
||||
if not isinstance(path, str):
|
||||
continue
|
||||
filename = path.split("/")[-1] if "/" in path else path
|
||||
filename_lower = filename.lower()
|
||||
# Map filename patterns to output format keys
|
||||
if "4k_lossless" in filename:
|
||||
output_files["mp4_4k_lossless"] = path
|
||||
elif "4k_lossy" in filename:
|
||||
# Files are named like "Artist - Title (Final Karaoke Lossless 4k).mp4"
|
||||
if "lossless 4k" in filename_lower:
|
||||
if filename.endswith(".mkv"):
|
||||
output_files["mkv_4k"] = path
|
||||
else:
|
||||
output_files["mp4_4k_lossless"] = path
|
||||
elif "lossy 4k" in filename_lower:
|
||||
output_files["mp4_4k_lossy"] = path
|
||||
elif "720p" in filename:
|
||||
elif "720p" in filename_lower:
|
||||
output_files["mp4_720p"] = path
|
||||
elif filename.endswith(".mkv"):
|
||||
output_files["mkv_4k"] = path
|
||||
self.logger.info(f"Converted output_files list to dict: {output_files}")
|
||||
else:
|
||||
output_files = raw_output_files if isinstance(raw_output_files, dict) else {}
|
||||
|
||||
@@ -63,6 +63,11 @@ class EncodingService:
|
||||
"""Check if GCE encoding is enabled and configured."""
|
||||
return self.settings.use_gce_encoding and self.is_configured
|
||||
|
||||
@property
|
||||
def is_preview_enabled(self) -> bool:
|
||||
"""Check if GCE preview encoding is enabled and configured."""
|
||||
return self.settings.use_gce_preview_encoding and self.is_configured
|
||||
|
||||
async def submit_encoding_job(
|
||||
self,
|
||||
job_id: str,
|
||||
@@ -242,6 +247,127 @@ class EncodingService:
|
||||
job_id, progress_callback=progress_callback
|
||||
)
|
||||
|
||||
async def submit_preview_encoding_job(
|
||||
self,
|
||||
job_id: str,
|
||||
ass_gcs_path: str,
|
||||
audio_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
background_color: str = "black",
|
||||
background_image_gcs_path: Optional[str] = None,
|
||||
font_gcs_path: Optional[str] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit a preview video encoding job to the GCE worker.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
ass_gcs_path: GCS path to ASS subtitles file (gs://bucket/path/file.ass)
|
||||
audio_gcs_path: GCS path to audio file
|
||||
output_gcs_path: GCS path for output video
|
||||
background_color: Background color (default: black)
|
||||
background_image_gcs_path: Optional GCS path to background image
|
||||
font_gcs_path: Optional GCS path to custom font file
|
||||
|
||||
Returns:
|
||||
Response from the encoding worker
|
||||
|
||||
Raises:
|
||||
Exception: If submission fails
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._url}/encode-preview"
|
||||
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
|
||||
payload = {
|
||||
"job_id": job_id,
|
||||
"ass_gcs_path": ass_gcs_path,
|
||||
"audio_gcs_path": audio_gcs_path,
|
||||
"output_gcs_path": output_gcs_path,
|
||||
"background_color": background_color,
|
||||
}
|
||||
if background_image_gcs_path:
|
||||
payload["background_image_gcs_path"] = background_image_gcs_path
|
||||
if font_gcs_path:
|
||||
payload["font_gcs_path"] = font_gcs_path
|
||||
|
||||
logger.info(f"[job:{job_id}] Submitting preview encoding job to GCE worker: {url}")
|
||||
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.post(url, json=payload, headers=headers, timeout=30) as resp:
|
||||
if resp.status == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp.status == 409:
|
||||
raise RuntimeError(f"Preview encoding job {job_id} already exists")
|
||||
if resp.status != 200:
|
||||
text = await resp.text()
|
||||
raise RuntimeError(f"Failed to submit preview encoding job: {resp.status} - {text}")
|
||||
|
||||
return await resp.json()
|
||||
|
||||
async def encode_preview_video(
|
||||
self,
|
||||
job_id: str,
|
||||
ass_gcs_path: str,
|
||||
audio_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
background_color: str = "black",
|
||||
background_image_gcs_path: Optional[str] = None,
|
||||
font_gcs_path: Optional[str] = None,
|
||||
timeout: float = 90.0,
|
||||
poll_interval: float = 2.0,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit preview encoding job and wait for completion.
|
||||
|
||||
This is a convenience method that combines submit + wait with shorter
|
||||
timeouts suitable for preview videos.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
ass_gcs_path: GCS path to ASS subtitles file
|
||||
audio_gcs_path: GCS path to audio file
|
||||
output_gcs_path: GCS path for output video
|
||||
background_color: Background color (default: black)
|
||||
background_image_gcs_path: Optional GCS path to background image
|
||||
font_gcs_path: Optional GCS path to custom font file
|
||||
timeout: Maximum time to wait (default 90s for preview)
|
||||
poll_interval: Seconds between status checks (default 2s)
|
||||
|
||||
Returns:
|
||||
Final job status with output files
|
||||
"""
|
||||
# Submit the job
|
||||
submit_result = await self.submit_preview_encoding_job(
|
||||
job_id=job_id,
|
||||
ass_gcs_path=ass_gcs_path,
|
||||
audio_gcs_path=audio_gcs_path,
|
||||
output_gcs_path=output_gcs_path,
|
||||
background_color=background_color,
|
||||
background_image_gcs_path=background_image_gcs_path,
|
||||
font_gcs_path=font_gcs_path,
|
||||
)
|
||||
|
||||
# If cached, return immediately - video already exists in GCS
|
||||
submit_status = submit_result.get("status")
|
||||
if submit_status == "cached":
|
||||
logger.info(f"[job:{job_id}] Preview already cached, returning immediately")
|
||||
return {"status": "complete", "output_path": submit_result.get("output_path")}
|
||||
|
||||
# If in_progress, another request is encoding it - just wait for that
|
||||
if submit_status == "in_progress":
|
||||
logger.info(f"[job:{job_id}] Preview encoding already in progress, waiting")
|
||||
|
||||
# Wait for completion with shorter timeout
|
||||
return await self.wait_for_completion(
|
||||
job_id=job_id,
|
||||
poll_interval=poll_interval,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
async def health_check(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Check the health of the encoding worker.
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
# GCE Encoding Worker
|
||||
|
||||
HTTP API service that runs on the GCE encoding worker VM for video encoding jobs.
|
||||
|
||||
## Overview
|
||||
|
||||
This service provides HTTP endpoints for submitting and monitoring video encoding jobs. It runs as a systemd service on the `encoding-worker` GCE VM instance.
|
||||
|
||||
## Endpoints
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/encode` | POST | Submit a full encoding job |
|
||||
| `/encode-preview` | POST | Submit a preview encoding job (480x270, fast) |
|
||||
| `/status/{job_id}` | GET | Get job status and progress |
|
||||
| `/health` | GET | Health check |
|
||||
|
||||
## Authentication
|
||||
|
||||
All endpoints (except `/health`) require an API key via the `X-API-Key` header. The key is stored in Secret Manager (`encoding-worker-api-key`).
|
||||
|
||||
## Encoding Process
|
||||
|
||||
### Full Encoding (`/encode`)
|
||||
1. Downloads input files from GCS (title screen, karaoke video, end screen, instrumental audio)
|
||||
2. Uses `LocalEncodingService` from the karaoke-gen wheel to produce:
|
||||
- Lossless 4K MP4
|
||||
- Lossy 4K MP4
|
||||
- Lossless MKV
|
||||
- 720p MP4
|
||||
3. Uploads results to GCS
|
||||
4. Reports progress via status endpoint
|
||||
|
||||
### Preview Encoding (`/encode-preview`)
|
||||
1. Downloads ASS subtitle and audio files from GCS
|
||||
2. Runs FFmpeg to create a quick 480x270 preview video
|
||||
3. Uploads result to GCS
|
||||
4. Used for real-time preview in the web UI
|
||||
|
||||
## Dependencies
|
||||
|
||||
- **FFmpeg**: Static build from John Van Sickle
|
||||
- **Python 3.13**: Built from source (required for karaoke-gen)
|
||||
- **karaoke-gen wheel**: Downloaded from GCS at job start (enables hot updates)
|
||||
|
||||
## Deployment
|
||||
|
||||
This service is deployed via the VM's startup script. The script:
|
||||
1. Installs system dependencies (FFmpeg, fonts, Python build deps)
|
||||
2. Builds Python 3.13 from source
|
||||
3. Creates venv and installs dependencies
|
||||
4. Writes `main.py` to `/opt/encoding-worker/`
|
||||
5. Creates systemd service
|
||||
6. Starts the service on port 8080
|
||||
|
||||
## Local Development
|
||||
|
||||
This code is extracted from the infrastructure for maintainability. To test locally:
|
||||
|
||||
```bash
|
||||
cd backend/services/gce_encoding
|
||||
pip install -r requirements.txt
|
||||
pip install ../../../ # Install karaoke-gen wheel
|
||||
ENCODING_API_KEY=test python main.py
|
||||
```
|
||||
|
||||
## Architecture Notes
|
||||
|
||||
- **Hot code updates**: The karaoke-gen wheel is re-downloaded at the start of each job, allowing code updates without VM restart
|
||||
- **In-memory job tracking**: Job state is stored in memory. Restart clears queue.
|
||||
- **Parallel encoding**: ThreadPoolExecutor with 4 workers
|
||||
- **GCS integration**: Direct download/upload via google-cloud-storage client
|
||||
@@ -0,0 +1,22 @@
|
||||
"""
|
||||
GCE Encoding Worker Service.
|
||||
|
||||
This package contains the FastAPI application that runs on the GCE encoding worker VM.
|
||||
It provides HTTP endpoints for video encoding jobs.
|
||||
|
||||
The service:
|
||||
- Downloads input files from GCS
|
||||
- Runs FFmpeg encoding via LocalEncodingService
|
||||
- Uploads results back to GCS
|
||||
- Provides job status tracking
|
||||
|
||||
Endpoints:
|
||||
- POST /encode - Submit a full encoding job
|
||||
- POST /encode-preview - Submit a preview encoding job (fast, low-res)
|
||||
- GET /status/{job_id} - Get job status
|
||||
- GET /health - Health check
|
||||
"""
|
||||
|
||||
from .main import app
|
||||
|
||||
__all__ = ["app"]
|
||||
@@ -0,0 +1,589 @@
|
||||
import asyncio
|
||||
import glob as glob_module
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import uuid
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import FastAPI, BackgroundTasks, HTTPException, Header, Depends
|
||||
from google.cloud import storage
|
||||
from pydantic import BaseModel
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
app = FastAPI(title="Encoding Worker", version="1.0.0")
|
||||
|
||||
# API key authentication
|
||||
API_KEY = os.environ.get("ENCODING_API_KEY", "")
|
||||
|
||||
|
||||
async def verify_api_key(x_api_key: str = Header(None)):
|
||||
# Verify API key for authentication
|
||||
if not API_KEY:
|
||||
logger.warning("No API key configured - authentication disabled")
|
||||
return True
|
||||
if x_api_key != API_KEY:
|
||||
raise HTTPException(status_code=401, detail="Invalid API key")
|
||||
return True
|
||||
|
||||
# Job tracking
|
||||
jobs: dict[str, dict] = {}
|
||||
executor = ThreadPoolExecutor(max_workers=4) # 4 parallel encoding jobs
|
||||
|
||||
# GCS client
|
||||
storage_client = storage.Client()
|
||||
|
||||
class EncodeRequest(BaseModel):
|
||||
job_id: str
|
||||
input_gcs_path: str # gs://bucket/path/to/inputs/
|
||||
output_gcs_path: str # gs://bucket/path/to/outputs/
|
||||
encoding_config: dict # Video formats to generate
|
||||
|
||||
|
||||
class EncodePreviewRequest(BaseModel):
|
||||
job_id: str
|
||||
ass_gcs_path: str # gs://bucket/path/to/subtitles.ass
|
||||
audio_gcs_path: str # gs://bucket/path/to/audio.flac
|
||||
output_gcs_path: str # gs://bucket/path/to/output.mp4
|
||||
background_color: str = "black"
|
||||
background_image_gcs_path: Optional[str] = None
|
||||
font_gcs_path: Optional[str] = None # gs://bucket/path/to/custom-font.ttf
|
||||
|
||||
|
||||
class JobStatus(BaseModel):
|
||||
job_id: str
|
||||
status: str # pending, running, complete, failed
|
||||
progress: int # 0-100
|
||||
error: Optional[str] = None
|
||||
output_files: Optional[list[str]] = None
|
||||
|
||||
|
||||
def download_from_gcs(gcs_uri: str, local_path: Path):
|
||||
# Download a file or folder from GCS
|
||||
# Parse gs://bucket/path
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
prefix = parts[1] if len(parts) > 1 else ""
|
||||
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
blobs = list(bucket.list_blobs(prefix=prefix))
|
||||
|
||||
for blob in blobs:
|
||||
# Get relative path from prefix
|
||||
rel_path = blob.name[len(prefix):].lstrip("/")
|
||||
if not rel_path:
|
||||
continue
|
||||
dest = local_path / rel_path
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
blob.download_to_filename(str(dest))
|
||||
logger.info(f"Downloaded: {blob.name} -> {dest}")
|
||||
|
||||
|
||||
def upload_to_gcs(local_path: Path, gcs_uri: str):
|
||||
# Upload a file or folder to GCS
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
|
||||
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
|
||||
if local_path.is_file():
|
||||
blob_name = f"{prefix}/{local_path.name}" if prefix else local_path.name
|
||||
blob = bucket.blob(blob_name)
|
||||
blob.upload_from_filename(str(local_path))
|
||||
logger.info(f"Uploaded: {local_path} -> gs://{bucket_name}/{blob_name}")
|
||||
else:
|
||||
for file in local_path.rglob("*"):
|
||||
if file.is_file():
|
||||
rel_path = file.relative_to(local_path)
|
||||
blob_name = f"{prefix}/{rel_path}" if prefix else str(rel_path)
|
||||
blob = bucket.blob(blob_name)
|
||||
blob.upload_from_filename(str(file))
|
||||
logger.info(f"Uploaded: {file} -> gs://{bucket_name}/{blob_name}")
|
||||
|
||||
|
||||
def download_single_file_from_gcs(gcs_uri: str, local_path: Path):
|
||||
# Download a single file from GCS
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
blob_name = parts[1] if len(parts) > 1 else ""
|
||||
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
blob = bucket.blob(blob_name)
|
||||
local_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
blob.download_to_filename(str(local_path))
|
||||
logger.info(f"Downloaded: {gcs_uri} -> {local_path}")
|
||||
|
||||
|
||||
def upload_single_file_to_gcs(local_path: Path, gcs_uri: str):
|
||||
# Upload a single file to a specific GCS path
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
blob_name = parts[1] if len(parts) > 1 else ""
|
||||
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
blob = bucket.blob(blob_name)
|
||||
blob.upload_from_filename(str(local_path))
|
||||
logger.info(f"Uploaded: {local_path} -> {gcs_uri}")
|
||||
|
||||
|
||||
def run_preview_encoding(job_id: str, work_dir: Path, request: "EncodePreviewRequest"):
|
||||
# Run FFmpeg encoding for preview video (480x270, fast settings)
|
||||
jobs[job_id]["status"] = "running"
|
||||
jobs[job_id]["progress"] = 10
|
||||
|
||||
try:
|
||||
# Download input files
|
||||
ass_path = work_dir / "subtitles.ass"
|
||||
audio_path = work_dir / "audio.flac"
|
||||
|
||||
download_single_file_from_gcs(request.ass_gcs_path, ass_path)
|
||||
jobs[job_id]["progress"] = 20
|
||||
|
||||
download_single_file_from_gcs(request.audio_gcs_path, audio_path)
|
||||
jobs[job_id]["progress"] = 30
|
||||
|
||||
# Download background image if provided
|
||||
bg_image_path = None
|
||||
if request.background_image_gcs_path:
|
||||
bg_image_path = work_dir / "background.png"
|
||||
download_single_file_from_gcs(request.background_image_gcs_path, bg_image_path)
|
||||
|
||||
# Download custom font if provided and register with fontconfig
|
||||
if request.font_gcs_path:
|
||||
# Use standard fontconfig location that's already in the search path
|
||||
fonts_dir = Path("/usr/local/share/fonts/custom")
|
||||
fonts_dir.mkdir(parents=True, exist_ok=True)
|
||||
font_filename = request.font_gcs_path.split("/")[-1]
|
||||
font_path = fonts_dir / font_filename
|
||||
download_single_file_from_gcs(request.font_gcs_path, font_path)
|
||||
logger.info(f"Downloaded custom font: {font_path}")
|
||||
# Update fontconfig cache so libass can find the font
|
||||
subprocess.run(["fc-cache", "-fv"], capture_output=True)
|
||||
logger.info(f"Updated fontconfig cache with custom font: {font_filename}")
|
||||
|
||||
# Build FFmpeg command
|
||||
output_path = work_dir / "preview.mp4"
|
||||
|
||||
# Escape special characters in path for FFmpeg filter syntax
|
||||
# FFmpeg filter parsing requires escaping: \ : , [ ] ;
|
||||
def escape_ffmpeg_filter_path(path: str) -> str:
|
||||
# Note: Extra escaping needed since this is inside a triple-quoted string in Pulumi
|
||||
return path.replace("\\", "\\\\").replace(":", "\\:").replace(",", "\\,").replace("[", "\\[").replace("]", "\\]").replace(";", "\\;")
|
||||
|
||||
escaped_ass_path = escape_ffmpeg_filter_path(str(ass_path))
|
||||
|
||||
# Base command with frame rate
|
||||
cmd = ["ffmpeg", "-y", "-r", "24"]
|
||||
|
||||
# Video input: background image or solid color
|
||||
if bg_image_path and bg_image_path.exists():
|
||||
cmd.extend(["-loop", "1", "-i", str(bg_image_path)])
|
||||
# Scale and pad background to 480x270
|
||||
vf = f"scale=480:270:force_original_aspect_ratio=decrease,pad=480:270:(ow-iw)/2:(oh-ih)/2,ass={escaped_ass_path}"
|
||||
else:
|
||||
# Solid color background
|
||||
color = request.background_color or "black"
|
||||
cmd.extend(["-f", "lavfi", "-i", f"color=c={color}:s=480x270:r=24"])
|
||||
vf = f"ass={escaped_ass_path}"
|
||||
|
||||
# Audio input
|
||||
cmd.extend(["-i", str(audio_path)])
|
||||
|
||||
# Video filter and encoding settings
|
||||
cmd.extend([
|
||||
"-vf", vf,
|
||||
"-c:a", "aac", "-b:a", "96k",
|
||||
"-c:v", "libx264",
|
||||
"-preset", "superfast",
|
||||
"-crf", "28",
|
||||
"-pix_fmt", "yuv420p",
|
||||
"-movflags", "+faststart",
|
||||
"-threads", "8",
|
||||
"-shortest",
|
||||
str(output_path)
|
||||
])
|
||||
|
||||
jobs[job_id]["progress"] = 40
|
||||
logger.info(f"Running preview encoding: {' '.join(cmd)}")
|
||||
|
||||
result = subprocess.run(cmd, capture_output=True, text=True)
|
||||
|
||||
if result.returncode != 0:
|
||||
logger.error(f"FFmpeg failed: {result.stderr}")
|
||||
raise RuntimeError(f"FFmpeg preview encoding failed: {result.stderr[-500:]}")
|
||||
|
||||
jobs[job_id]["progress"] = 80
|
||||
logger.info(f"Preview encoded: {output_path}")
|
||||
|
||||
# Upload output to GCS
|
||||
upload_single_file_to_gcs(output_path, request.output_gcs_path)
|
||||
jobs[job_id]["progress"] = 95
|
||||
|
||||
jobs[job_id]["output_files"] = [request.output_gcs_path]
|
||||
return output_path
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Preview encoding failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
raise
|
||||
|
||||
|
||||
def ensure_latest_wheel():
|
||||
'''Download and install latest karaoke-gen wheel from GCS.
|
||||
|
||||
Called at the start of each job to enable hot code updates without restart.
|
||||
In-progress jobs continue with their version, new jobs get latest code.
|
||||
'''
|
||||
try:
|
||||
logger.info("Checking for latest karaoke-gen wheel in GCS...")
|
||||
|
||||
# Download latest wheel
|
||||
result = subprocess.run(
|
||||
["gsutil", "cp", "gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-*.whl", "/tmp/"],
|
||||
capture_output=True, text=True, timeout=60
|
||||
)
|
||||
|
||||
# Find the downloaded wheel (get the latest by version sorting)
|
||||
wheels = glob_module.glob("/tmp/karaoke_gen-*.whl")
|
||||
if not wheels:
|
||||
logger.warning("No wheel found in GCS, using fallback encoding logic")
|
||||
return False
|
||||
|
||||
# Sort to get latest version
|
||||
wheel_path = sorted(wheels)[-1]
|
||||
logger.info(f"Installing wheel: {wheel_path}")
|
||||
|
||||
# Install (or upgrade) the wheel
|
||||
# Use 5-minute timeout - first install at job start may need to resolve dependencies
|
||||
# Subsequent installs are faster since dependencies are cached
|
||||
install_result = subprocess.run(
|
||||
[sys.executable, "-m", "pip", "install", "--upgrade", "--quiet", wheel_path],
|
||||
capture_output=True, text=True, timeout=300
|
||||
)
|
||||
|
||||
if install_result.returncode != 0:
|
||||
logger.warning(f"Wheel installation failed: {install_result.stderr}")
|
||||
return False
|
||||
|
||||
logger.info(f"Successfully installed wheel: {wheel_path}")
|
||||
return True
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning("Wheel download/install timed out, using fallback")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to ensure latest wheel: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def find_file(work_dir: Path, *patterns):
|
||||
'''Find a file matching any of the given glob patterns.'''
|
||||
for pattern in patterns:
|
||||
matches = list(work_dir.glob(f"**/{pattern}"))
|
||||
if matches:
|
||||
return matches[0]
|
||||
return None
|
||||
|
||||
|
||||
def run_encoding(job_id: str, work_dir: Path, config: dict):
|
||||
'''Run encoding using LocalEncodingService (single source of truth).
|
||||
|
||||
Uses LocalEncodingService from the installed karaoke-gen wheel to ensure
|
||||
output files match local CLI exactly:
|
||||
- Proper names like "Artist - Title (Final Karaoke Lossless 4k).mp4"
|
||||
- Concatenated title + karaoke + end screens
|
||||
- All formats: lossless 4K MP4, lossy 4K MP4, lossless MKV, 720p MP4
|
||||
|
||||
Requires the karaoke-gen wheel to be installed (done by ensure_latest_wheel).
|
||||
'''
|
||||
jobs[job_id]["status"] = "running"
|
||||
jobs[job_id]["progress"] = 10
|
||||
|
||||
try:
|
||||
# Import LocalEncodingService from installed wheel (required, no fallback)
|
||||
from backend.services.local_encoding_service import LocalEncodingService, EncodingConfig
|
||||
logger.info("Using LocalEncodingService from installed wheel")
|
||||
|
||||
# Get artist/title from config for proper naming
|
||||
artist = config.get("artist", "Unknown Artist")
|
||||
title = config.get("title", "Unknown Title")
|
||||
base_name = f"{artist} - {title}"
|
||||
logger.info(f"Encoding for: {base_name}")
|
||||
|
||||
# Find input files in work_dir
|
||||
# Title/end screens are in screens/ subdirectory
|
||||
title_video = find_file(work_dir, "screens/title.mov", "*Title*.mov", "*title*.mov")
|
||||
end_video = find_file(work_dir, "screens/end.mov", "*End*.mov", "*end*.mov")
|
||||
|
||||
# Karaoke video - search for With Vocals or main karaoke video
|
||||
karaoke_video = find_file(
|
||||
work_dir,
|
||||
"*With Vocals*.mov", "*With Vocals*.mkv",
|
||||
"*Vocals*.mov", "*Vocals*.mkv",
|
||||
"*.mkv", "*.mov"
|
||||
)
|
||||
# Exclude title/end/output videos
|
||||
if karaoke_video:
|
||||
name_lower = karaoke_video.name.lower()
|
||||
if "title" in name_lower or "end" in name_lower or "outputs" in str(karaoke_video):
|
||||
# Search more specifically for karaoke video
|
||||
karaoke_video = find_file(work_dir, "*Karaoke*.mkv", "*Karaoke*.mov", "*vocals*.mkv")
|
||||
|
||||
# Instrumental audio
|
||||
instrumental = find_file(
|
||||
work_dir,
|
||||
"*instrumental_clean*.flac", "*Instrumental Clean*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
|
||||
logger.info(f"Found files:")
|
||||
logger.info(f" Title video: {title_video}")
|
||||
logger.info(f" Karaoke video: {karaoke_video}")
|
||||
logger.info(f" End video: {end_video}")
|
||||
logger.info(f" Instrumental: {instrumental}")
|
||||
|
||||
# Validate required files
|
||||
if not title_video:
|
||||
raise ValueError(f"No title video found in {work_dir}. Check screens/ subdirectory.")
|
||||
if not karaoke_video:
|
||||
raise ValueError(f"No karaoke video found in {work_dir}")
|
||||
if not instrumental:
|
||||
raise ValueError(f"No instrumental audio found in {work_dir}")
|
||||
|
||||
output_dir = work_dir / "outputs"
|
||||
output_dir.mkdir(exist_ok=True)
|
||||
|
||||
jobs[job_id]["progress"] = 20
|
||||
|
||||
# Build encoding config with proper file names
|
||||
encoding_config = EncodingConfig(
|
||||
title_video=str(title_video),
|
||||
karaoke_video=str(karaoke_video),
|
||||
instrumental_audio=str(instrumental),
|
||||
end_video=str(end_video) if end_video else None,
|
||||
output_karaoke_mp4=str(output_dir / f"{base_name} (Karaoke).mp4"),
|
||||
output_with_vocals_mp4=str(output_dir / f"{base_name} (With Vocals).mp4"),
|
||||
output_lossless_4k_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossless 4k).mp4"),
|
||||
output_lossy_4k_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossy 4k).mp4"),
|
||||
output_lossless_mkv=str(output_dir / f"{base_name} (Final Karaoke Lossless 4k).mkv"),
|
||||
output_720p_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossy 720p).mp4"),
|
||||
)
|
||||
|
||||
# Create service and run encoding
|
||||
service = LocalEncodingService(logger=logger)
|
||||
|
||||
jobs[job_id]["progress"] = 30
|
||||
logger.info("Starting LocalEncodingService.encode_all_formats()")
|
||||
|
||||
result = service.encode_all_formats(encoding_config)
|
||||
|
||||
if not result.success:
|
||||
raise RuntimeError(f"Encoding failed: {result.error}")
|
||||
|
||||
jobs[job_id]["progress"] = 90
|
||||
|
||||
# Collect output files
|
||||
output_files = [str(f) for f in output_dir.glob("*") if f.is_file()]
|
||||
jobs[job_id]["output_files"] = output_files
|
||||
|
||||
logger.info(f"Encoding complete. Output files: {output_files}")
|
||||
return output_dir
|
||||
|
||||
except ImportError as e:
|
||||
# No fallback - wheel must be installed
|
||||
error_msg = (
|
||||
f"LocalEncodingService not available: {e}. "
|
||||
"The karaoke-gen wheel must be installed. "
|
||||
"Check that ensure_latest_wheel() succeeded and wheel exists in GCS."
|
||||
)
|
||||
logger.error(error_msg)
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = error_msg
|
||||
raise RuntimeError(error_msg) from e
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Encoding failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
raise
|
||||
|
||||
|
||||
async def process_job(job_id: str, request: EncodeRequest):
|
||||
# Process an encoding job asynchronously
|
||||
try:
|
||||
# Download and install latest wheel at job start (allows hot updates without restart)
|
||||
# This means in-progress jobs continue with their version, new jobs get latest code
|
||||
ensure_latest_wheel()
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
work_dir = Path(temp_dir) / "work"
|
||||
work_dir.mkdir()
|
||||
|
||||
# Download input files
|
||||
jobs[job_id]["progress"] = 5
|
||||
logger.info(f"Downloading from {request.input_gcs_path}")
|
||||
download_from_gcs(request.input_gcs_path, work_dir)
|
||||
|
||||
# Run encoding in thread pool (CPU-bound)
|
||||
loop = asyncio.get_event_loop()
|
||||
output_dir = await loop.run_in_executor(
|
||||
executor,
|
||||
run_encoding,
|
||||
job_id,
|
||||
work_dir,
|
||||
request.encoding_config
|
||||
)
|
||||
|
||||
# Upload outputs
|
||||
jobs[job_id]["progress"] = 95
|
||||
logger.info(f"Uploading to {request.output_gcs_path}")
|
||||
upload_to_gcs(output_dir, request.output_gcs_path)
|
||||
|
||||
# Convert local paths to blob paths (backend expects blob paths, not full gs:// URIs)
|
||||
# output_gcs_path is like "gs://bucket/jobs/id/encoded/"
|
||||
# We need paths like "jobs/id/encoded/Artist - Title (Final Karaoke Lossless 4k).mp4"
|
||||
gcs_path = request.output_gcs_path.replace("gs://", "")
|
||||
parts = gcs_path.split("/", 1)
|
||||
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
|
||||
local_output_files = jobs[job_id].get("output_files", [])
|
||||
blob_paths = []
|
||||
for local_path in local_output_files:
|
||||
filename = Path(local_path).name
|
||||
blob_path = f"{prefix}/{filename}" if prefix else filename
|
||||
blob_paths.append(blob_path)
|
||||
jobs[job_id]["output_files"] = blob_paths
|
||||
logger.info(f"Output files (blob paths): {blob_paths}")
|
||||
|
||||
jobs[job_id]["status"] = "complete"
|
||||
jobs[job_id]["progress"] = 100
|
||||
logger.info(f"Job {job_id} complete")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id} failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
|
||||
|
||||
async def process_preview_job(job_id: str, request: EncodePreviewRequest):
|
||||
# Process a preview encoding job asynchronously
|
||||
try:
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
work_dir = Path(temp_dir) / "work"
|
||||
work_dir.mkdir()
|
||||
|
||||
# Run preview encoding in thread pool (CPU-bound)
|
||||
# Note: run_preview_encoding handles download/upload internally
|
||||
loop = asyncio.get_event_loop()
|
||||
await loop.run_in_executor(
|
||||
executor,
|
||||
run_preview_encoding,
|
||||
job_id,
|
||||
work_dir,
|
||||
request
|
||||
)
|
||||
|
||||
jobs[job_id]["status"] = "complete"
|
||||
jobs[job_id]["progress"] = 100
|
||||
logger.info(f"Preview job {job_id} complete")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Preview job {job_id} failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
|
||||
|
||||
@app.post("/encode-preview")
|
||||
async def submit_preview_encode_job(request: EncodePreviewRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
|
||||
# Submit a preview encoding job
|
||||
job_id = request.job_id
|
||||
|
||||
# If job already exists, return cached result or current status
|
||||
if job_id in jobs:
|
||||
existing_job = jobs[job_id]
|
||||
if existing_job["status"] == "complete":
|
||||
# Return cached result - preview already encoded
|
||||
return {"status": "cached", "job_id": job_id, "output_path": existing_job.get("output_path")}
|
||||
elif existing_job["status"] == "failed":
|
||||
# Previous attempt failed, allow retry by replacing the job
|
||||
pass
|
||||
else:
|
||||
# Job is still in progress
|
||||
return {"status": "in_progress", "job_id": job_id}
|
||||
|
||||
jobs[job_id] = {
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"progress": 0,
|
||||
"error": None,
|
||||
"output_files": None,
|
||||
}
|
||||
|
||||
background_tasks.add_task(process_preview_job, job_id, request)
|
||||
|
||||
return {"status": "accepted", "job_id": job_id}
|
||||
|
||||
|
||||
@app.post("/encode")
|
||||
async def submit_encode_job(request: EncodeRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
|
||||
# Submit an encoding job
|
||||
job_id = request.job_id
|
||||
|
||||
if job_id in jobs:
|
||||
raise HTTPException(status_code=409, detail=f"Job {job_id} already exists")
|
||||
|
||||
jobs[job_id] = {
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"progress": 0,
|
||||
"error": None,
|
||||
"output_files": None,
|
||||
}
|
||||
|
||||
background_tasks.add_task(process_job, job_id, request)
|
||||
|
||||
return {"status": "accepted", "job_id": job_id}
|
||||
|
||||
|
||||
@app.get("/status/{job_id}")
|
||||
async def get_job_status(job_id: str, _auth: bool = Depends(verify_api_key)) -> JobStatus:
|
||||
# Get the status of an encoding job
|
||||
if job_id not in jobs:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
return JobStatus(**jobs[job_id])
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
async def health_check():
|
||||
# Health check endpoint
|
||||
active_jobs = sum(1 for j in jobs.values() if j["status"] == "running")
|
||||
|
||||
# Get karaoke-gen wheel version if installed
|
||||
wheel_version = None
|
||||
try:
|
||||
from importlib.metadata import version as get_version
|
||||
wheel_version = get_version("karaoke-gen")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return {
|
||||
"status": "ok",
|
||||
"active_jobs": active_jobs,
|
||||
"queue_length": sum(1 for j in jobs.values() if j["status"] == "pending"),
|
||||
"ffmpeg_version": subprocess.run(["ffmpeg", "-version"], capture_output=True, text=True).stdout.split("\n")[0],
|
||||
"wheel_version": wheel_version,
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=8080)
|
||||
@@ -0,0 +1,16 @@
|
||||
# GCE Encoding Worker dependencies
|
||||
# These are installed in the VM's Python virtual environment
|
||||
|
||||
# Web framework
|
||||
fastapi>=0.109.0
|
||||
uvicorn>=0.27.0
|
||||
|
||||
# GCS client
|
||||
google-cloud-storage>=2.14.0
|
||||
|
||||
# Async utilities (optional, for future improvements)
|
||||
aiofiles>=23.2.0
|
||||
aiohttp>=3.9.0
|
||||
|
||||
# Note: The karaoke-gen wheel is installed separately from GCS.
|
||||
# It provides LocalEncodingService which does the actual encoding.
|
||||
@@ -359,9 +359,147 @@ class JobManager:
|
||||
'instrumental_token': updates['instrumental_token'],
|
||||
'instrumental_token_expires_at': updates['instrumental_token_expires_at']
|
||||
})
|
||||
|
||||
|
||||
logger.info(f"Job {job_id} transitioned to {new_status}")
|
||||
|
||||
# Trigger notifications asynchronously (fire-and-forget)
|
||||
self._trigger_state_notifications(job_id, new_status)
|
||||
|
||||
return True
|
||||
|
||||
def _trigger_state_notifications(self, job_id: str, new_status: JobStatus) -> None:
|
||||
"""
|
||||
Trigger email notifications based on state transitions.
|
||||
|
||||
This is fire-and-forget - notification failures don't affect job processing.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
new_status: New job status
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
try:
|
||||
# Get the job to access user info
|
||||
job = self.get_job(job_id)
|
||||
if not job or not job.user_email:
|
||||
logger.debug(f"No user email for job {job_id}, skipping notifications")
|
||||
return
|
||||
|
||||
# Job completion notification
|
||||
if new_status == JobStatus.COMPLETE:
|
||||
self._schedule_completion_email(job)
|
||||
|
||||
# Idle reminder scheduling for blocking states
|
||||
elif new_status in [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION]:
|
||||
self._schedule_idle_reminder(job, new_status)
|
||||
|
||||
except Exception as e:
|
||||
# Never let notification failures affect job processing
|
||||
logger.error(f"Error triggering notifications for job {job_id}: {e}")
|
||||
|
||||
def _schedule_completion_email(self, job: Job) -> None:
|
||||
"""
|
||||
Schedule sending a job completion email.
|
||||
|
||||
Uses asyncio to fire-and-forget the email sending.
|
||||
"""
|
||||
import asyncio
|
||||
import threading
|
||||
|
||||
try:
|
||||
from backend.services.job_notification_service import get_job_notification_service
|
||||
|
||||
notification_service = get_job_notification_service()
|
||||
|
||||
# Get youtube, dropbox URLs, and brand_code from state_data (may be None)
|
||||
state_data = job.state_data or {}
|
||||
youtube_url = state_data.get('youtube_url')
|
||||
dropbox_url = state_data.get('dropbox_link')
|
||||
brand_code = state_data.get('brand_code')
|
||||
|
||||
# Create async task (fire-and-forget)
|
||||
async def send_email():
|
||||
await notification_service.send_job_completion_email(
|
||||
job_id=job.job_id,
|
||||
user_email=job.user_email,
|
||||
user_name=None, # Could fetch from user service if needed
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
youtube_url=youtube_url,
|
||||
dropbox_url=dropbox_url,
|
||||
brand_code=brand_code,
|
||||
)
|
||||
|
||||
# Try to get existing event loop, create new one if none exists
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
# If we're in an async context, create a task
|
||||
loop.create_task(send_email())
|
||||
except RuntimeError:
|
||||
# No event loop - we're likely in a sync context
|
||||
# Use daemon thread to avoid blocking job completion
|
||||
def run_in_thread():
|
||||
asyncio.run(send_email())
|
||||
thread = threading.Thread(target=run_in_thread, daemon=True)
|
||||
thread.start()
|
||||
|
||||
logger.info(f"Scheduled completion email for job {job.job_id}")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to schedule completion email for job {job.job_id}: {e}")
|
||||
|
||||
def _schedule_idle_reminder(self, job: Job, new_status: JobStatus) -> None:
|
||||
"""
|
||||
Schedule an idle reminder for a blocking state.
|
||||
|
||||
Records the timestamp when the blocking state was entered and
|
||||
schedules a Cloud Tasks task for 5 minutes later.
|
||||
"""
|
||||
import asyncio
|
||||
import threading
|
||||
|
||||
try:
|
||||
# Record when we entered blocking state (for idle detection)
|
||||
blocking_entered_at = datetime.utcnow().isoformat()
|
||||
|
||||
action_type = "lyrics" if new_status == JobStatus.AWAITING_REVIEW else "instrumental"
|
||||
|
||||
# Update state_data with blocking state info (handle None state_data)
|
||||
existing_state_data = job.state_data or {}
|
||||
state_data_update = {
|
||||
'blocking_state_entered_at': blocking_entered_at,
|
||||
'blocking_action_type': action_type,
|
||||
'reminder_sent': False, # Will be set to True after reminder is sent
|
||||
}
|
||||
|
||||
self.firestore.update_job(job.job_id, {
|
||||
'state_data': {**existing_state_data, **state_data_update}
|
||||
})
|
||||
|
||||
# Schedule the idle reminder check via worker service (5 min delay)
|
||||
from backend.services.worker_service import get_worker_service
|
||||
|
||||
async def schedule_reminder():
|
||||
worker_service = get_worker_service()
|
||||
await worker_service.schedule_idle_reminder(job.job_id)
|
||||
|
||||
# Try to get existing event loop, create new one if none exists
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
loop.create_task(schedule_reminder())
|
||||
except RuntimeError:
|
||||
# No event loop - we're in a sync context
|
||||
# Use daemon thread to avoid blocking job processing
|
||||
def run_in_thread():
|
||||
asyncio.run(schedule_reminder())
|
||||
thread = threading.Thread(target=run_in_thread, daemon=True)
|
||||
thread.start()
|
||||
|
||||
logger.info(f"Scheduled idle reminder for job {job.job_id} ({action_type})")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to schedule idle reminder for job {job.job_id}: {e}")
|
||||
|
||||
def update_state_data(self, job_id: str, key: str, value: Any) -> None:
|
||||
"""
|
||||
|
||||
@@ -0,0 +1,271 @@
|
||||
"""
|
||||
Job notification service for sending emails on job state changes.
|
||||
|
||||
Handles:
|
||||
- Job completion emails (when job enters COMPLETE status)
|
||||
- Action reminder emails (when user is idle at blocking states)
|
||||
|
||||
This service orchestrates between the template service and email service,
|
||||
building the complete notification flow for jobs.
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
from backend.services.email_service import get_email_service
|
||||
from backend.services.template_service import get_template_service
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Environment variable to enable/disable auto emails
|
||||
ENABLE_AUTO_EMAILS = os.getenv("ENABLE_AUTO_EMAILS", "true").lower() == "true"
|
||||
|
||||
# Feedback form URL (configured per environment, empty by default to avoid placeholder in emails)
|
||||
FEEDBACK_FORM_URL = os.getenv("FEEDBACK_FORM_URL", "")
|
||||
|
||||
|
||||
def _mask_email(email: str) -> str:
|
||||
"""Mask email for logging to protect PII. Shows first char + domain."""
|
||||
if not email or "@" not in email:
|
||||
return "***"
|
||||
local, domain = email.split("@", 1)
|
||||
if len(local) <= 1:
|
||||
return f"*@{domain}"
|
||||
return f"{local[0]}***@{domain}"
|
||||
|
||||
|
||||
class JobNotificationService:
|
||||
"""
|
||||
Service for sending job-related email notifications.
|
||||
|
||||
Coordinates between template service (for message rendering) and
|
||||
email service (for sending).
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize notification service."""
|
||||
self.email_service = get_email_service()
|
||||
self.template_service = get_template_service()
|
||||
self.frontend_url = os.getenv("FRONTEND_URL", "https://gen.nomadkaraoke.com")
|
||||
self.backend_url = os.getenv("BACKEND_URL", "https://api.nomadkaraoke.com")
|
||||
|
||||
def _build_review_url(self, job_id: str, audio_hash: Optional[str] = None, review_token: Optional[str] = None) -> str:
|
||||
"""Build the lyrics review URL for a job."""
|
||||
import urllib.parse
|
||||
|
||||
review_ui_url = os.getenv("REVIEW_UI_URL", f"{self.frontend_url}/lyrics/")
|
||||
base_api_url = f"{self.backend_url}/api/review/{job_id}"
|
||||
encoded_api_url = urllib.parse.quote(base_api_url, safe='')
|
||||
|
||||
url = f"{review_ui_url}?baseApiUrl={encoded_api_url}"
|
||||
if audio_hash:
|
||||
url += f"&audioHash={urllib.parse.quote(audio_hash, safe='')}"
|
||||
if review_token:
|
||||
url += f"&reviewToken={urllib.parse.quote(review_token, safe='')}"
|
||||
return url
|
||||
|
||||
def _build_instrumental_url(self, job_id: str, instrumental_token: Optional[str] = None) -> str:
|
||||
"""Build the instrumental selection URL for a job."""
|
||||
base_api_url = f"{self.backend_url}/api/jobs/{job_id}"
|
||||
|
||||
import urllib.parse
|
||||
encoded_api_url = urllib.parse.quote(base_api_url, safe='')
|
||||
|
||||
url = f"{self.frontend_url}/instrumental/?baseApiUrl={encoded_api_url}"
|
||||
if instrumental_token:
|
||||
url += f"&instrumentalToken={urllib.parse.quote(instrumental_token, safe='')}"
|
||||
return url
|
||||
|
||||
async def send_job_completion_email(
|
||||
self,
|
||||
job_id: str,
|
||||
user_email: str,
|
||||
user_name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
youtube_url: Optional[str] = None,
|
||||
dropbox_url: Optional[str] = None,
|
||||
brand_code: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Send job completion email to user.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
user_email: User's email address
|
||||
user_name: User's display name (optional)
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
youtube_url: YouTube video URL
|
||||
dropbox_url: Dropbox folder URL
|
||||
brand_code: Release ID (e.g., "NOMAD-1178")
|
||||
|
||||
Returns:
|
||||
True if email was sent successfully
|
||||
"""
|
||||
if not ENABLE_AUTO_EMAILS:
|
||||
logger.info(f"Auto emails disabled, skipping completion email for job {job_id}")
|
||||
return False
|
||||
|
||||
if not user_email:
|
||||
logger.warning(f"No user email for job {job_id}, skipping completion email")
|
||||
return False
|
||||
|
||||
try:
|
||||
# Render the completion message using template service
|
||||
message_content = self.template_service.render_job_completion(
|
||||
name=user_name,
|
||||
youtube_url=youtube_url,
|
||||
dropbox_url=dropbox_url,
|
||||
artist=artist,
|
||||
title=title,
|
||||
job_id=job_id,
|
||||
feedback_url=FEEDBACK_FORM_URL,
|
||||
)
|
||||
|
||||
# Send the email with CC to admin
|
||||
success = self.email_service.send_job_completion(
|
||||
to_email=user_email,
|
||||
message_content=message_content,
|
||||
artist=artist,
|
||||
title=title,
|
||||
brand_code=brand_code,
|
||||
cc_admin=True,
|
||||
)
|
||||
|
||||
if success:
|
||||
logger.info(f"Sent completion email for job {job_id} to {_mask_email(user_email)}")
|
||||
else:
|
||||
logger.error(f"Failed to send completion email for job {job_id}")
|
||||
|
||||
return success
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"Error sending completion email for job {job_id}: {e}")
|
||||
return False
|
||||
|
||||
async def send_action_reminder_email(
|
||||
self,
|
||||
job_id: str,
|
||||
user_email: str,
|
||||
action_type: str,
|
||||
user_name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
audio_hash: Optional[str] = None,
|
||||
review_token: Optional[str] = None,
|
||||
instrumental_token: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Send action-needed reminder email to user.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
user_email: User's email address
|
||||
action_type: Type of action needed ("lyrics" or "instrumental")
|
||||
user_name: User's display name
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
audio_hash: Audio hash for review URL
|
||||
review_token: Review token for URL
|
||||
instrumental_token: Instrumental token for URL
|
||||
|
||||
Returns:
|
||||
True if email was sent successfully
|
||||
"""
|
||||
if not ENABLE_AUTO_EMAILS:
|
||||
logger.info(f"Auto emails disabled, skipping reminder for job {job_id}")
|
||||
return False
|
||||
|
||||
if not user_email:
|
||||
logger.warning(f"No user email for job {job_id}, skipping reminder")
|
||||
return False
|
||||
|
||||
try:
|
||||
# Render the appropriate template
|
||||
if action_type == "lyrics":
|
||||
review_url = self._build_review_url(job_id, audio_hash, review_token)
|
||||
message_content = self.template_service.render_action_needed_lyrics(
|
||||
name=user_name,
|
||||
artist=artist,
|
||||
title=title,
|
||||
review_url=review_url,
|
||||
)
|
||||
elif action_type == "instrumental":
|
||||
instrumental_url = self._build_instrumental_url(job_id, instrumental_token)
|
||||
message_content = self.template_service.render_action_needed_instrumental(
|
||||
name=user_name,
|
||||
artist=artist,
|
||||
title=title,
|
||||
instrumental_url=instrumental_url,
|
||||
)
|
||||
else:
|
||||
logger.error(f"Unknown action type: {action_type}")
|
||||
return False
|
||||
|
||||
# Send the reminder email (no CC)
|
||||
success = self.email_service.send_action_reminder(
|
||||
to_email=user_email,
|
||||
message_content=message_content,
|
||||
action_type=action_type,
|
||||
artist=artist,
|
||||
title=title,
|
||||
)
|
||||
|
||||
if success:
|
||||
logger.info(f"Sent {action_type} reminder for job {job_id} to {_mask_email(user_email)}")
|
||||
else:
|
||||
logger.error(f"Failed to send {action_type} reminder for job {job_id}")
|
||||
|
||||
return success
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"Error sending {action_type} reminder for job {job_id}: {e}")
|
||||
return False
|
||||
|
||||
def get_completion_message(
|
||||
self,
|
||||
job_id: str,
|
||||
user_name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
youtube_url: Optional[str] = None,
|
||||
dropbox_url: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Get the rendered completion message for a job (for admin copy functionality).
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
user_name: User's display name
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
youtube_url: YouTube video URL
|
||||
dropbox_url: Dropbox folder URL
|
||||
|
||||
Returns:
|
||||
Rendered message content as plain text
|
||||
"""
|
||||
return self.template_service.render_job_completion(
|
||||
name=user_name,
|
||||
youtube_url=youtube_url,
|
||||
dropbox_url=dropbox_url,
|
||||
artist=artist,
|
||||
title=title,
|
||||
job_id=job_id,
|
||||
feedback_url=FEEDBACK_FORM_URL,
|
||||
)
|
||||
|
||||
|
||||
# Global instance
|
||||
_job_notification_service: Optional[JobNotificationService] = None
|
||||
|
||||
|
||||
def get_job_notification_service() -> JobNotificationService:
|
||||
"""Get the global job notification service instance."""
|
||||
global _job_notification_service
|
||||
if _job_notification_service is None:
|
||||
_job_notification_service = JobNotificationService()
|
||||
return _job_notification_service
|
||||
@@ -0,0 +1,407 @@
|
||||
"""
|
||||
Local Preview Encoding Service.
|
||||
|
||||
Provides local FFmpeg-based preview video encoding functionality, extracted from
|
||||
VideoGenerator for use by the GCE worker. This ensures the same encoding logic
|
||||
is used across local CLI, Cloud Run, and GCE worker environments.
|
||||
|
||||
This service handles:
|
||||
- Generating preview videos with ASS subtitle overlay
|
||||
- Hardware acceleration detection (NVENC) with fallback to software encoding
|
||||
- Background image/color support
|
||||
- Custom font support
|
||||
- Optimized settings for fast preview generation
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Optional, List
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class PreviewEncodingConfig:
|
||||
"""Configuration for preview video encoding."""
|
||||
ass_path: str # Path to ASS subtitles file
|
||||
audio_path: str # Path to audio file
|
||||
output_path: str # Path for output video file
|
||||
|
||||
# Optional background settings
|
||||
background_image_path: Optional[str] = None # Path to background image
|
||||
background_color: str = "black" # Fallback background color
|
||||
|
||||
# Optional font settings
|
||||
font_path: Optional[str] = None # Path to custom font file
|
||||
|
||||
|
||||
@dataclass
|
||||
class PreviewEncodingResult:
|
||||
"""Result of preview encoding operation."""
|
||||
success: bool
|
||||
output_path: Optional[str] = None
|
||||
error: Optional[str] = None
|
||||
|
||||
|
||||
class LocalPreviewEncodingService:
|
||||
"""
|
||||
Service for local FFmpeg-based preview video encoding.
|
||||
|
||||
This is the single source of truth for preview encoding logic, used by:
|
||||
- Local CLI (via VideoGenerator which delegates here)
|
||||
- Cloud Run (when GCE is unavailable)
|
||||
- GCE worker (via installed wheel)
|
||||
|
||||
Supports hardware acceleration (NVENC) with automatic fallback
|
||||
to software encoding (libx264) when unavailable.
|
||||
"""
|
||||
|
||||
# Preview video settings - these are the canonical values
|
||||
PREVIEW_WIDTH = 480
|
||||
PREVIEW_HEIGHT = 270
|
||||
PREVIEW_FPS = 24
|
||||
PREVIEW_AUDIO_BITRATE = "96k"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the local preview encoding service.
|
||||
|
||||
Args:
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
# Hardware acceleration settings (detected on first use)
|
||||
self._nvenc_available: Optional[bool] = None
|
||||
self._video_encoder: Optional[str] = None
|
||||
self._hwaccel_flags: Optional[List[str]] = None
|
||||
|
||||
def _detect_nvenc_support(self) -> bool:
|
||||
"""
|
||||
Detect if NVENC hardware encoding is available.
|
||||
|
||||
Returns:
|
||||
True if NVENC is available, False otherwise
|
||||
"""
|
||||
try:
|
||||
self.logger.info("Detecting NVENC hardware acceleration...")
|
||||
|
||||
# Test h264_nvenc encoder directly
|
||||
test_cmd = [
|
||||
"ffmpeg", "-hide_banner", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", "testsrc=duration=1:size=320x240:rate=1",
|
||||
"-c:v", "h264_nvenc", "-f", "null", "-"
|
||||
]
|
||||
|
||||
result = subprocess.run(
|
||||
test_cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
self.logger.info("NVENC hardware acceleration available")
|
||||
return True
|
||||
|
||||
# Try alternative test with different source
|
||||
alt_test_cmd = [
|
||||
"ffmpeg", "-hide_banner", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", "color=red:size=320x240:duration=0.1",
|
||||
"-c:v", "h264_nvenc", "-preset", "fast", "-f", "null", "-"
|
||||
]
|
||||
|
||||
alt_result = subprocess.run(
|
||||
alt_test_cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30
|
||||
)
|
||||
|
||||
if alt_result.returncode == 0:
|
||||
self.logger.info("NVENC hardware acceleration available")
|
||||
return True
|
||||
|
||||
self.logger.info("NVENC not available, using software encoding")
|
||||
return False
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
self.logger.debug("NVENC detection timed out")
|
||||
return False
|
||||
except Exception as e:
|
||||
self.logger.debug(f"NVENC detection failed: {e}")
|
||||
return False
|
||||
|
||||
def _configure_hardware_acceleration(self) -> None:
|
||||
"""Configure hardware acceleration settings based on detected capabilities."""
|
||||
if self._nvenc_available is None:
|
||||
self._nvenc_available = self._detect_nvenc_support()
|
||||
|
||||
if self._nvenc_available:
|
||||
self._video_encoder = "h264_nvenc"
|
||||
self._hwaccel_flags = ["-hwaccel", "cuda", "-hwaccel_output_format", "cuda"]
|
||||
self.logger.info("Using NVENC hardware acceleration for preview encoding")
|
||||
else:
|
||||
self._video_encoder = "libx264"
|
||||
self._hwaccel_flags = []
|
||||
self.logger.info("Using software encoding (libx264) for preview")
|
||||
|
||||
@property
|
||||
def nvenc_available(self) -> bool:
|
||||
"""Check if NVENC hardware acceleration is available."""
|
||||
if self._nvenc_available is None:
|
||||
self._configure_hardware_acceleration()
|
||||
return self._nvenc_available
|
||||
|
||||
@property
|
||||
def video_encoder(self) -> str:
|
||||
"""Get the video encoder to use."""
|
||||
if self._video_encoder is None:
|
||||
self._configure_hardware_acceleration()
|
||||
return self._video_encoder
|
||||
|
||||
@property
|
||||
def hwaccel_flags(self) -> List[str]:
|
||||
"""Get hardware acceleration flags."""
|
||||
if self._hwaccel_flags is None:
|
||||
self._configure_hardware_acceleration()
|
||||
return self._hwaccel_flags
|
||||
|
||||
def _escape_ffmpeg_filter_path(self, path: str) -> str:
|
||||
"""
|
||||
Escape a path for FFmpeg filter expressions (for subprocess without shell).
|
||||
|
||||
When using subprocess with a command list (no shell), FFmpeg receives the
|
||||
filter string directly. FFmpeg's filter parser requires escaping:
|
||||
- Backslashes: double them (\\ -> \\\\)
|
||||
- Single quotes/apostrophes: escape with three backslashes (' -> \\\\')
|
||||
- Spaces: escape with backslash ( -> \\ )
|
||||
- Special characters: :,[];
|
||||
|
||||
Example: "I'm With You" becomes "I\\\\'m\\ With\\ You"
|
||||
"""
|
||||
# First escape existing backslashes
|
||||
escaped = path.replace("\\", "\\\\")
|
||||
# Escape single quotes
|
||||
escaped = escaped.replace("'", "\\\\\\'")
|
||||
# Escape spaces
|
||||
escaped = escaped.replace(" ", "\\ ")
|
||||
# Escape FFmpeg filter special characters
|
||||
escaped = escaped.replace(":", "\\:")
|
||||
escaped = escaped.replace(",", "\\,")
|
||||
escaped = escaped.replace("[", "\\[")
|
||||
escaped = escaped.replace("]", "\\]")
|
||||
escaped = escaped.replace(";", "\\;")
|
||||
return escaped
|
||||
|
||||
def _build_ass_filter(self, ass_path: str, font_path: Optional[str] = None) -> str:
|
||||
"""
|
||||
Build ASS filter with optional font directory support.
|
||||
|
||||
Args:
|
||||
ass_path: Path to ASS subtitles file
|
||||
font_path: Optional path to custom font file
|
||||
|
||||
Returns:
|
||||
FFmpeg ASS filter string
|
||||
"""
|
||||
escaped_ass_path = self._escape_ffmpeg_filter_path(ass_path)
|
||||
ass_filter = f"ass={escaped_ass_path}"
|
||||
|
||||
if font_path and os.path.isfile(font_path):
|
||||
font_dir = os.path.dirname(font_path)
|
||||
escaped_font_dir = self._escape_ffmpeg_filter_path(font_dir)
|
||||
ass_filter += f":fontsdir={escaped_font_dir}"
|
||||
self.logger.debug(f"Using font directory: {font_dir}")
|
||||
|
||||
return ass_filter
|
||||
|
||||
def _build_preview_ffmpeg_command(self, config: PreviewEncodingConfig) -> List[str]:
|
||||
"""
|
||||
Build FFmpeg command for preview video generation.
|
||||
|
||||
This is the canonical preview encoding command, ensuring consistency
|
||||
across all environments (local CLI, Cloud Run, GCE worker).
|
||||
|
||||
Args:
|
||||
config: Preview encoding configuration
|
||||
|
||||
Returns:
|
||||
FFmpeg command as a list of arguments
|
||||
"""
|
||||
width, height = self.PREVIEW_WIDTH, self.PREVIEW_HEIGHT
|
||||
|
||||
cmd = [
|
||||
"ffmpeg",
|
||||
"-hide_banner",
|
||||
"-loglevel", "error",
|
||||
"-r", str(self.PREVIEW_FPS),
|
||||
]
|
||||
|
||||
# Add hardware acceleration flags if available
|
||||
cmd.extend(self.hwaccel_flags)
|
||||
|
||||
# Input source (background image or solid color)
|
||||
if config.background_image_path and os.path.isfile(config.background_image_path):
|
||||
self.logger.debug(f"Using background image: {config.background_image_path}")
|
||||
cmd.extend([
|
||||
"-loop", "1",
|
||||
"-i", config.background_image_path,
|
||||
])
|
||||
# Build video filter with scaling and ASS subtitles
|
||||
video_filter = (
|
||||
f"scale={width}:{height}:force_original_aspect_ratio=decrease,"
|
||||
f"pad={width}:{height}:(ow-iw)/2:(oh-ih)/2,"
|
||||
f"{self._build_ass_filter(config.ass_path, config.font_path)}"
|
||||
)
|
||||
else:
|
||||
self.logger.debug(f"Using solid {config.background_color} background")
|
||||
cmd.extend([
|
||||
"-f", "lavfi",
|
||||
"-i", f"color=c={config.background_color}:s={width}x{height}:r={self.PREVIEW_FPS}",
|
||||
])
|
||||
# Just ASS subtitles, no scaling needed
|
||||
video_filter = self._build_ass_filter(config.ass_path, config.font_path)
|
||||
|
||||
cmd.extend([
|
||||
"-i", config.audio_path,
|
||||
"-vf", video_filter,
|
||||
"-c:a", "aac",
|
||||
"-b:a", self.PREVIEW_AUDIO_BITRATE,
|
||||
"-c:v", self.video_encoder,
|
||||
])
|
||||
|
||||
# Add encoder-specific settings optimized for speed
|
||||
if self.nvenc_available:
|
||||
cmd.extend([
|
||||
"-preset", "p1", # Fastest NVENC preset
|
||||
"-tune", "ll", # Low latency
|
||||
"-rc", "cbr", # Constant bitrate for speed
|
||||
"-b:v", "800k", # Lower bitrate for speed
|
||||
"-profile:v", "baseline", # Most compatible profile
|
||||
"-level", "3.1", # Lower level for speed
|
||||
])
|
||||
self.logger.debug("Using NVENC with maximum speed settings")
|
||||
else:
|
||||
cmd.extend([
|
||||
"-profile:v", "baseline",
|
||||
"-level", "3.0",
|
||||
"-preset", "superfast",
|
||||
"-tune", "fastdecode",
|
||||
"-b:v", "600k",
|
||||
"-maxrate", "800k",
|
||||
"-bufsize", "1200k",
|
||||
"-crf", "28",
|
||||
])
|
||||
self.logger.debug("Using software encoding with maximum speed settings")
|
||||
|
||||
cmd.extend([
|
||||
"-pix_fmt", "yuv420p", # Required for browser compatibility
|
||||
"-movflags", "+faststart+frag_keyframe+empty_moov+dash",
|
||||
"-g", "48", # Keyframe every 48 frames (2 seconds at 24fps)
|
||||
"-keyint_min", "48",
|
||||
"-sc_threshold", "0", # Disable scene change detection for speed
|
||||
"-threads", "0", # Use all available CPU threads
|
||||
"-shortest",
|
||||
"-y",
|
||||
config.output_path,
|
||||
])
|
||||
|
||||
return cmd
|
||||
|
||||
def encode_preview(self, config: PreviewEncodingConfig) -> PreviewEncodingResult:
|
||||
"""
|
||||
Encode a preview video.
|
||||
|
||||
Args:
|
||||
config: Preview encoding configuration
|
||||
|
||||
Returns:
|
||||
PreviewEncodingResult with success status and output path
|
||||
"""
|
||||
self.logger.info(f"Encoding preview video: {config.output_path}")
|
||||
|
||||
# Validate input files
|
||||
if not os.path.isfile(config.ass_path):
|
||||
return PreviewEncodingResult(
|
||||
success=False,
|
||||
error=f"ASS subtitles file not found: {config.ass_path}"
|
||||
)
|
||||
|
||||
if not os.path.isfile(config.audio_path):
|
||||
return PreviewEncodingResult(
|
||||
success=False,
|
||||
error=f"Audio file not found: {config.audio_path}"
|
||||
)
|
||||
|
||||
# Ensure output directory exists
|
||||
output_dir = os.path.dirname(config.output_path)
|
||||
if output_dir:
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
|
||||
try:
|
||||
# Build and execute FFmpeg command
|
||||
cmd = self._build_preview_ffmpeg_command(config)
|
||||
self.logger.debug(f"FFmpeg command: {' '.join(cmd)}")
|
||||
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=300 # 5 minute timeout for preview encoding
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
error_msg = result.stderr[-500:] if result.stderr else "Unknown error"
|
||||
self.logger.error(f"FFmpeg failed: {error_msg}")
|
||||
return PreviewEncodingResult(
|
||||
success=False,
|
||||
error=f"FFmpeg preview encoding failed: {error_msg}"
|
||||
)
|
||||
|
||||
self.logger.info(f"Preview encoded successfully: {config.output_path}")
|
||||
return PreviewEncodingResult(
|
||||
success=True,
|
||||
output_path=config.output_path
|
||||
)
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
self.logger.error("Preview encoding timed out")
|
||||
return PreviewEncodingResult(
|
||||
success=False,
|
||||
error="Preview encoding timed out after 5 minutes"
|
||||
)
|
||||
except Exception as e:
|
||||
self.logger.error(f"Preview encoding failed: {e}")
|
||||
return PreviewEncodingResult(
|
||||
success=False,
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
|
||||
# Singleton instance and factory function
|
||||
_local_preview_encoding_service: Optional[LocalPreviewEncodingService] = None
|
||||
|
||||
|
||||
def get_local_preview_encoding_service(**kwargs) -> LocalPreviewEncodingService:
|
||||
"""
|
||||
Get a local preview encoding service instance.
|
||||
|
||||
Args:
|
||||
**kwargs: Arguments passed to LocalPreviewEncodingService
|
||||
|
||||
Returns:
|
||||
LocalPreviewEncodingService instance
|
||||
"""
|
||||
global _local_preview_encoding_service
|
||||
|
||||
if _local_preview_encoding_service is None:
|
||||
_local_preview_encoding_service = LocalPreviewEncodingService(**kwargs)
|
||||
|
||||
return _local_preview_encoding_service
|
||||
@@ -0,0 +1,330 @@
|
||||
"""
|
||||
Template service for managing email templates stored in GCS.
|
||||
|
||||
Templates are stored in GCS bucket and rendered with job-specific variables.
|
||||
This allows updating email content without code deployment.
|
||||
|
||||
Template locations:
|
||||
- gs://{bucket}/templates/job-completion.txt - Job completion email (plain text)
|
||||
- gs://{bucket}/templates/action-needed-lyrics.txt - Lyrics review reminder
|
||||
- gs://{bucket}/templates/action-needed-instrumental.txt - Instrumental selection reminder
|
||||
"""
|
||||
import logging
|
||||
import re
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Default templates (fallback if GCS fetch fails)
|
||||
DEFAULT_JOB_COMPLETION_TEMPLATE = """Hi {name},
|
||||
|
||||
Thanks for your order!
|
||||
|
||||
Here's the link for the karaoke video published to YouTube:
|
||||
{youtube_url}
|
||||
|
||||
Here's the dropbox folder with all the finished files and source files, including:
|
||||
- "(Final Karaoke Lossless).mkv": combined karaoke video in 4k H264 with lossless FLAC audio
|
||||
- "(Final Karaoke).mp4": combined karaoke video with title/end screen in 4k H264/AAC
|
||||
- "(Final Karaoke 720p).mp4": combined karaoke video in 720p H264/AAC (smaller file for older systems)
|
||||
- "(With Vocals).mp4": sing along video in 4k H264/AAC with original vocals
|
||||
- "(Karaoke).mov": karaoke video output from MidiCo (no title/end screen)
|
||||
- "(Title).mov"/"(End).mov": title card and end screen videos
|
||||
- "(Final Karaoke CDG).zip": CDG+MP3 format for older/commercial karaoke systems
|
||||
- "(Final Karaoke TXT).zip": TXT+MP3 format for Power Karaoke
|
||||
- stems/*.flac: various separated instrumental and vocal audio stems in lossless format
|
||||
- lyrics/*.txt song lyrics from various sources in plain text format
|
||||
|
||||
{dropbox_url}
|
||||
|
||||
Let me know if anything isn't perfect and I'll happily tweak / fix, or if you need it in any other format I can probably convert it for you!
|
||||
|
||||
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
|
||||
{feedback_url}
|
||||
|
||||
Thanks again and have a great day!
|
||||
-Andrew
|
||||
"""
|
||||
|
||||
DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE = """Hi {name},
|
||||
|
||||
Your karaoke video for "{artist} - {title}" is ready for lyrics review!
|
||||
|
||||
The AI has transcribed and synchronized the lyrics, but they may need some corrections. Please review and approve them so we can finish generating your video.
|
||||
|
||||
Review your lyrics here:
|
||||
{review_url}
|
||||
|
||||
This usually takes just a few minutes.
|
||||
|
||||
Thanks!
|
||||
-Nomad Karaoke
|
||||
"""
|
||||
|
||||
DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE = """Hi {name},
|
||||
|
||||
Your karaoke video for "{artist} - {title}" is almost done!
|
||||
|
||||
We've separated the audio into different versions. Please select which instrumental track you'd like to use for the final video.
|
||||
|
||||
Select your instrumental here:
|
||||
{instrumental_url}
|
||||
|
||||
Thanks!
|
||||
-Nomad Karaoke
|
||||
"""
|
||||
|
||||
|
||||
class TemplateService:
|
||||
"""
|
||||
Service for fetching and rendering email templates from GCS.
|
||||
|
||||
Templates support the following variables:
|
||||
- {name} - User's display name or "there" if unknown
|
||||
- {youtube_url} - YouTube video URL
|
||||
- {dropbox_url} - Dropbox folder URL
|
||||
- {artist} - Artist name
|
||||
- {title} - Song title
|
||||
- {job_id} - Job ID
|
||||
- {review_url} - Lyrics review URL
|
||||
- {instrumental_url} - Instrumental selection URL
|
||||
- {feedback_url} - Feedback form URL
|
||||
"""
|
||||
|
||||
TEMPLATE_PREFIX = "templates/"
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize template service."""
|
||||
self.settings = get_settings()
|
||||
self._storage_client = None
|
||||
self._bucket = None
|
||||
|
||||
@property
|
||||
def storage_client(self):
|
||||
"""Lazy-initialize storage client."""
|
||||
if self._storage_client is None:
|
||||
from google.cloud import storage
|
||||
self._storage_client = storage.Client(project=self.settings.google_cloud_project)
|
||||
return self._storage_client
|
||||
|
||||
@property
|
||||
def bucket(self):
|
||||
"""Get the GCS bucket for templates."""
|
||||
if self._bucket is None:
|
||||
self._bucket = self.storage_client.bucket(self.settings.gcs_bucket_name)
|
||||
return self._bucket
|
||||
|
||||
def _fetch_template_from_gcs(self, template_name: str) -> Optional[str]:
|
||||
"""
|
||||
Fetch a template from GCS.
|
||||
|
||||
Args:
|
||||
template_name: Name of template file (e.g., "job-completion.txt")
|
||||
|
||||
Returns:
|
||||
Template content as string, or None if not found
|
||||
"""
|
||||
blob_path = f"{self.TEMPLATE_PREFIX}{template_name}"
|
||||
try:
|
||||
blob = self.bucket.blob(blob_path)
|
||||
if blob.exists():
|
||||
content = blob.download_as_text()
|
||||
logger.debug(f"Fetched template from GCS: {blob_path}")
|
||||
return content
|
||||
else:
|
||||
logger.warning(f"Template not found in GCS: {blob_path}")
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to fetch template {blob_path}: {e}")
|
||||
return None
|
||||
|
||||
def get_job_completion_template(self) -> str:
|
||||
"""Get the job completion email template."""
|
||||
template = self._fetch_template_from_gcs("job-completion.txt")
|
||||
if template is None:
|
||||
logger.info("Using default job completion template")
|
||||
return DEFAULT_JOB_COMPLETION_TEMPLATE
|
||||
return template
|
||||
|
||||
def get_action_needed_lyrics_template(self) -> str:
|
||||
"""Get the lyrics review reminder template."""
|
||||
template = self._fetch_template_from_gcs("action-needed-lyrics.txt")
|
||||
if template is None:
|
||||
logger.info("Using default lyrics reminder template")
|
||||
return DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE
|
||||
return template
|
||||
|
||||
def get_action_needed_instrumental_template(self) -> str:
|
||||
"""Get the instrumental selection reminder template."""
|
||||
template = self._fetch_template_from_gcs("action-needed-instrumental.txt")
|
||||
if template is None:
|
||||
logger.info("Using default instrumental reminder template")
|
||||
return DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE
|
||||
return template
|
||||
|
||||
def render_template(self, template: str, variables: Dict[str, Any]) -> str:
|
||||
"""
|
||||
Render a template with the given variables.
|
||||
|
||||
Missing variables are replaced with empty strings.
|
||||
Handles conditional sections like feedback URL.
|
||||
|
||||
Args:
|
||||
template: Template string with {variable} placeholders
|
||||
variables: Dictionary of variable values
|
||||
|
||||
Returns:
|
||||
Rendered template string
|
||||
"""
|
||||
result = template
|
||||
|
||||
# Handle feedback URL section - remove if not provided
|
||||
if not variables.get("feedback_url"):
|
||||
# Remove feedback section (lines containing feedback_url placeholder and surrounding text)
|
||||
result = re.sub(
|
||||
r'\n*If you have a moment.*?\{feedback_url\}\n*',
|
||||
'\n',
|
||||
result,
|
||||
flags=re.DOTALL
|
||||
)
|
||||
variables["feedback_url"] = ""
|
||||
|
||||
# Replace all variables
|
||||
for key, value in variables.items():
|
||||
placeholder = "{" + key + "}"
|
||||
result = result.replace(placeholder, str(value) if value else "")
|
||||
|
||||
# Clean up any remaining unreplaced placeholders
|
||||
result = re.sub(r'\{[a-z_]+\}', '', result)
|
||||
|
||||
return result.strip()
|
||||
|
||||
def render_job_completion(
|
||||
self,
|
||||
name: Optional[str] = None,
|
||||
youtube_url: Optional[str] = None,
|
||||
dropbox_url: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
job_id: Optional[str] = None,
|
||||
feedback_url: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Render the job completion email template.
|
||||
|
||||
Args:
|
||||
name: User's display name (defaults to "there")
|
||||
youtube_url: YouTube video URL
|
||||
dropbox_url: Dropbox folder URL
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
job_id: Job ID
|
||||
feedback_url: Feedback form URL (optional)
|
||||
|
||||
Returns:
|
||||
Rendered email content
|
||||
"""
|
||||
template = self.get_job_completion_template()
|
||||
variables = {
|
||||
"name": name or "there",
|
||||
"youtube_url": youtube_url or "[YouTube URL not available]",
|
||||
"dropbox_url": dropbox_url or "[Dropbox URL not available]",
|
||||
"artist": artist or "Unknown Artist",
|
||||
"title": title or "Unknown Title",
|
||||
"job_id": job_id or "",
|
||||
"feedback_url": feedback_url,
|
||||
}
|
||||
return self.render_template(template, variables)
|
||||
|
||||
def render_action_needed_lyrics(
|
||||
self,
|
||||
name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
review_url: str = "",
|
||||
) -> str:
|
||||
"""
|
||||
Render the lyrics review reminder template.
|
||||
|
||||
Args:
|
||||
name: User's display name
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
review_url: Lyrics review URL
|
||||
|
||||
Returns:
|
||||
Rendered email content
|
||||
"""
|
||||
template = self.get_action_needed_lyrics_template()
|
||||
variables = {
|
||||
"name": name or "there",
|
||||
"artist": artist or "Unknown Artist",
|
||||
"title": title or "Unknown Title",
|
||||
"review_url": review_url,
|
||||
}
|
||||
return self.render_template(template, variables)
|
||||
|
||||
def render_action_needed_instrumental(
|
||||
self,
|
||||
name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
instrumental_url: str = "",
|
||||
) -> str:
|
||||
"""
|
||||
Render the instrumental selection reminder template.
|
||||
|
||||
Args:
|
||||
name: User's display name
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
instrumental_url: Instrumental selection URL
|
||||
|
||||
Returns:
|
||||
Rendered email content
|
||||
"""
|
||||
template = self.get_action_needed_instrumental_template()
|
||||
variables = {
|
||||
"name": name or "there",
|
||||
"artist": artist or "Unknown Artist",
|
||||
"title": title or "Unknown Title",
|
||||
"instrumental_url": instrumental_url,
|
||||
}
|
||||
return self.render_template(template, variables)
|
||||
|
||||
def upload_template(self, template_name: str, content: str) -> bool:
|
||||
"""
|
||||
Upload a template to GCS.
|
||||
|
||||
Args:
|
||||
template_name: Name of template file (e.g., "job-completion.txt")
|
||||
content: Template content
|
||||
|
||||
Returns:
|
||||
True if upload successful
|
||||
"""
|
||||
blob_path = f"{self.TEMPLATE_PREFIX}{template_name}"
|
||||
try:
|
||||
blob = self.bucket.blob(blob_path)
|
||||
blob.upload_from_string(content, content_type="text/plain")
|
||||
logger.info(f"Uploaded template to GCS: {blob_path}")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to upload template {blob_path}: {e}")
|
||||
return False
|
||||
|
||||
|
||||
# Global instance
|
||||
_template_service: Optional[TemplateService] = None
|
||||
|
||||
|
||||
def get_template_service() -> TemplateService:
|
||||
"""Get the global template service instance."""
|
||||
global _template_service
|
||||
if _template_service is None:
|
||||
_template_service = TemplateService()
|
||||
return _template_service
|
||||
@@ -37,6 +37,7 @@ WORKER_QUEUES = {
|
||||
"screens": "screens-worker-queue",
|
||||
"render-video": "render-worker-queue",
|
||||
"video": "video-worker-queue",
|
||||
"idle-reminder": "idle-reminder-queue", # For delayed idle reminder checks
|
||||
}
|
||||
|
||||
# Dispatch deadlines for each worker type (in seconds)
|
||||
@@ -49,8 +50,12 @@ WORKER_DISPATCH_DEADLINES = {
|
||||
"screens": 600, # 10 min - Screen generation is fast
|
||||
"render-video": 1800, # 30 min - Video encoding can be slow
|
||||
"video": 1800, # 30 min - Video encoding can be slow
|
||||
"idle-reminder": 60, # 1 min - Quick check and potential email send
|
||||
}
|
||||
|
||||
# Default delay for idle reminders (seconds)
|
||||
IDLE_REMINDER_DELAY_SECONDS = 5 * 60 # 5 minutes
|
||||
|
||||
|
||||
class WorkerService:
|
||||
"""
|
||||
@@ -430,6 +435,99 @@ class WorkerService:
|
||||
"""Trigger render video worker (post-review)."""
|
||||
return await self.trigger_worker("render-video", job_id)
|
||||
|
||||
async def schedule_idle_reminder(
|
||||
self,
|
||||
job_id: str,
|
||||
delay_seconds: int = IDLE_REMINDER_DELAY_SECONDS
|
||||
) -> bool:
|
||||
"""
|
||||
Schedule an idle reminder check for a job.
|
||||
|
||||
The reminder task will be delivered after the specified delay.
|
||||
When the task executes, it checks if the job is still in a blocking
|
||||
state and sends a reminder email if the user hasn't taken action.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to check
|
||||
delay_seconds: Delay before executing the check (default: 5 minutes)
|
||||
|
||||
Returns:
|
||||
True if task was scheduled successfully, False otherwise
|
||||
"""
|
||||
if not self._use_cloud_tasks:
|
||||
# In development mode, log and skip (no delayed execution support)
|
||||
logger.info(
|
||||
f"[job:{job_id}] Idle reminder not scheduled (Cloud Tasks disabled). "
|
||||
f"Would have fired in {delay_seconds}s."
|
||||
)
|
||||
return True
|
||||
|
||||
try:
|
||||
from google.cloud import tasks_v2
|
||||
from google.protobuf import timestamp_pb2
|
||||
import time
|
||||
|
||||
queue_name = WORKER_QUEUES.get("idle-reminder")
|
||||
if not queue_name:
|
||||
logger.error("Idle reminder queue not configured")
|
||||
return False
|
||||
|
||||
project = self.settings.google_cloud_project
|
||||
if not project:
|
||||
logger.error("GOOGLE_CLOUD_PROJECT not set")
|
||||
return False
|
||||
|
||||
location = self.settings.gcp_region
|
||||
|
||||
# Build queue path
|
||||
parent = self.tasks_client.queue_path(project, location, queue_name)
|
||||
|
||||
# Build headers
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
if self._admin_token:
|
||||
headers["X-Admin-Token"] = self._admin_token
|
||||
|
||||
# Inject trace context
|
||||
headers = inject_trace_context(headers)
|
||||
|
||||
# Calculate schedule time
|
||||
schedule_time = timestamp_pb2.Timestamp()
|
||||
schedule_time.FromSeconds(int(time.time()) + delay_seconds)
|
||||
|
||||
dispatch_deadline_seconds = WORKER_DISPATCH_DEADLINES.get("idle-reminder", 60)
|
||||
|
||||
# Build task payload
|
||||
task = {
|
||||
"http_request": {
|
||||
"http_method": tasks_v2.HttpMethod.POST,
|
||||
"url": f"{self._base_url}/api/internal/jobs/{job_id}/check-idle-reminder",
|
||||
"headers": headers,
|
||||
"body": json.dumps({"job_id": job_id}).encode(),
|
||||
"oidc_token": {
|
||||
"service_account_email": f"karaoke-backend@{project}.iam.gserviceaccount.com",
|
||||
},
|
||||
},
|
||||
"schedule_time": schedule_time,
|
||||
"dispatch_deadline": duration_pb2.Duration(seconds=dispatch_deadline_seconds),
|
||||
}
|
||||
|
||||
# Create task
|
||||
response = self.tasks_client.create_task(parent=parent, task=task)
|
||||
logger.info(
|
||||
f"[job:{job_id}] Scheduled idle reminder check in {delay_seconds}s: {response.name}"
|
||||
)
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"[job:{job_id}] Failed to schedule idle reminder: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
return False
|
||||
|
||||
|
||||
# Global worker service instance
|
||||
_worker_service: Optional[WorkerService] = None
|
||||
|
||||
@@ -0,0 +1,411 @@
|
||||
"""
|
||||
Unit tests for admin email endpoints.
|
||||
|
||||
Tests the completion message and send email API endpoints.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, MagicMock, AsyncMock
|
||||
from fastapi.testclient import TestClient
|
||||
from fastapi import FastAPI
|
||||
|
||||
from backend.api.routes.admin import router
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
# Create a test app with the admin router
|
||||
# The router already has prefix="/admin", so we add /api prefix
|
||||
app = FastAPI()
|
||||
app.include_router(router, prefix="/api")
|
||||
|
||||
|
||||
def get_mock_admin():
|
||||
"""Override for require_admin dependency."""
|
||||
return ("admin@example.com", "admin", 1)
|
||||
|
||||
|
||||
# Override the require_admin dependency
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
"""Create a test client."""
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job():
|
||||
"""Create a mock job."""
|
||||
job = Mock(spec=Job)
|
||||
job.job_id = "test-job-123"
|
||||
job.user_email = "user@example.com"
|
||||
job.artist = "Test Artist"
|
||||
job.title = "Test Song"
|
||||
job.status = JobStatus.COMPLETE
|
||||
job.audio_hash = "hash123"
|
||||
job.review_token = "review123"
|
||||
job.instrumental_token = "inst123"
|
||||
job.state_data = {
|
||||
"youtube_url": "https://youtube.com/watch?v=test123",
|
||||
"dropbox_link": "https://dropbox.com/folder/test",
|
||||
"brand_code": "NOMAD-1234",
|
||||
}
|
||||
return job
|
||||
|
||||
|
||||
class TestGetCompletionMessage:
|
||||
"""Tests for GET /api/admin/jobs/{job_id}/completion-message endpoint."""
|
||||
|
||||
def test_returns_completion_message(self, client, mock_job):
|
||||
"""Test successful completion message retrieval."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
|
||||
|
||||
# Setup mocks
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Your video is ready!"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
response = client.get(
|
||||
"/api/admin/jobs/test-job-123/completion-message",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["job_id"] == "test-job-123"
|
||||
assert data["message"] == "Your video is ready!"
|
||||
# Subject format: "NOMAD-1234: Test Artist - Test Song (Your karaoke video is ready!)"
|
||||
assert data["subject"] == "NOMAD-1234: Test Artist - Test Song (Your karaoke video is ready!)"
|
||||
assert data["youtube_url"] == "https://youtube.com/watch?v=test123"
|
||||
assert data["dropbox_url"] == "https://dropbox.com/folder/test"
|
||||
|
||||
def test_returns_404_when_job_not_found(self, client):
|
||||
"""Test 404 when job doesn't exist."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = None
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.get(
|
||||
"/api/admin/jobs/nonexistent-job/completion-message",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
def test_handles_none_state_data(self, client, mock_job):
|
||||
"""Test handling of job with None state_data."""
|
||||
mock_job.state_data = None
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Your video is ready!"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
response = client.get(
|
||||
"/api/admin/jobs/test-job-123/completion-message",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["youtube_url"] is None
|
||||
assert data["dropbox_url"] is None
|
||||
|
||||
def test_default_subject_without_song_info(self, client, mock_job):
|
||||
"""Test default subject when no artist/title."""
|
||||
mock_job.artist = None
|
||||
mock_job.title = None
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Message"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
response = client.get(
|
||||
"/api/admin/jobs/test-job-123/completion-message",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["subject"] == "Your karaoke video is ready!"
|
||||
|
||||
|
||||
class TestSendCompletionEmail:
|
||||
"""Tests for POST /api/admin/jobs/{job_id}/send-completion-email endpoint."""
|
||||
|
||||
def test_sends_email_successfully(self, client, mock_job):
|
||||
"""Test successful email sending."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
|
||||
patch('backend.services.email_service.get_email_service') as mock_get_es:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Your video is ready!"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
mock_es = Mock()
|
||||
mock_es.send_job_completion.return_value = True
|
||||
mock_get_es.return_value = mock_es
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/send-completion-email",
|
||||
json={"to_email": "customer@example.com", "cc_admin": True},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["success"] is True
|
||||
assert data["job_id"] == "test-job-123"
|
||||
assert data["to_email"] == "customer@example.com"
|
||||
|
||||
# Verify email was sent with correct params
|
||||
mock_es.send_job_completion.assert_called_once_with(
|
||||
to_email="customer@example.com",
|
||||
message_content="Your video is ready!",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
brand_code="NOMAD-1234",
|
||||
cc_admin=True,
|
||||
)
|
||||
|
||||
def test_sends_email_without_cc(self, client, mock_job):
|
||||
"""Test email sending without CC."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
|
||||
patch('backend.services.email_service.get_email_service') as mock_get_es:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Message"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
mock_es = Mock()
|
||||
mock_es.send_job_completion.return_value = True
|
||||
mock_get_es.return_value = mock_es
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/send-completion-email",
|
||||
json={"to_email": "customer@example.com", "cc_admin": False},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
# Verify CC was not included
|
||||
call_kwargs = mock_es.send_job_completion.call_args.kwargs
|
||||
assert call_kwargs["cc_admin"] is False
|
||||
|
||||
def test_returns_404_when_job_not_found(self, client):
|
||||
"""Test 404 when job doesn't exist."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = None
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/nonexistent-job/send-completion-email",
|
||||
json={"to_email": "customer@example.com"},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
def test_returns_500_when_email_fails(self, client, mock_job):
|
||||
"""Test 500 when email sending fails."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
|
||||
patch('backend.services.email_service.get_email_service') as mock_get_es:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Message"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
mock_es = Mock()
|
||||
mock_es.send_job_completion.return_value = False # Email failed
|
||||
mock_get_es.return_value = mock_es
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/send-completion-email",
|
||||
json={"to_email": "customer@example.com"},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 500
|
||||
assert "Failed to send email" in response.json()["detail"]
|
||||
|
||||
def test_handles_none_state_data(self, client, mock_job):
|
||||
"""Test handling of job with None state_data."""
|
||||
mock_job.state_data = None
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
|
||||
patch('backend.services.email_service.get_email_service') as mock_get_es:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Message"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
mock_es = Mock()
|
||||
mock_es.send_job_completion.return_value = True
|
||||
mock_get_es.return_value = mock_es
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/send-completion-email",
|
||||
json={"to_email": "customer@example.com"},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_accepts_any_email_string(self, client, mock_job):
|
||||
"""Test that any string is currently accepted as email (no validation).
|
||||
|
||||
Documents current behavior: FastAPI/Pydantic doesn't validate email
|
||||
format by default. If validation is added later, this test should
|
||||
be updated to expect 422 for invalid emails.
|
||||
"""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
|
||||
patch('backend.services.email_service.get_email_service') as mock_get_es:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Message"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
mock_es = Mock()
|
||||
mock_es.send_job_completion.return_value = True
|
||||
mock_get_es.return_value = mock_es
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/send-completion-email",
|
||||
json={"to_email": "invalid-email"}, # Not a valid email format
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
# Currently accepts any string - endpoint succeeds
|
||||
assert response.status_code == 200
|
||||
# Verify the invalid email was passed through
|
||||
mock_es.send_job_completion.assert_called_once()
|
||||
call_kwargs = mock_es.send_job_completion.call_args.kwargs
|
||||
assert call_kwargs["to_email"] == "invalid-email"
|
||||
|
||||
|
||||
class TestIdleReminderEndpoint:
|
||||
"""Tests for the internal idle reminder endpoint."""
|
||||
|
||||
def test_sends_reminder_when_still_idle(self, mock_job):
|
||||
"""Test that reminder is sent when user is still idle."""
|
||||
from backend.api.routes.internal import router as internal_router
|
||||
from backend.api.dependencies import require_admin
|
||||
|
||||
# Create test app for internal router
|
||||
# The router already has prefix="/internal", so we add /api prefix
|
||||
test_app = FastAPI()
|
||||
test_app.include_router(internal_router, prefix="/api")
|
||||
test_app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
mock_job.status = JobStatus.AWAITING_REVIEW.value # Use .value for string comparison
|
||||
mock_job.state_data = {
|
||||
"blocking_state_entered_at": "2024-01-01T00:00:00",
|
||||
"blocking_action_type": "lyrics",
|
||||
"reminder_sent": False,
|
||||
}
|
||||
|
||||
with patch('backend.api.routes.internal.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.firestore = Mock() # Mock firestore for update_job call
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.send_action_reminder_email = AsyncMock(return_value=True)
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
test_client = TestClient(test_app)
|
||||
response = test_client.post(
|
||||
"/api/internal/jobs/test-job-123/check-idle-reminder",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
# Should succeed and return "sent" status
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "sent"
|
||||
assert data["job_id"] == "test-job-123"
|
||||
|
||||
def test_skips_reminder_when_already_sent(self, mock_job):
|
||||
"""Test that reminder is skipped if already sent."""
|
||||
from backend.api.routes.internal import router as internal_router
|
||||
from backend.api.dependencies import require_admin
|
||||
|
||||
# Create test app for internal router
|
||||
# The router already has prefix="/internal", so we add /api prefix
|
||||
test_app = FastAPI()
|
||||
test_app.include_router(internal_router, prefix="/api")
|
||||
test_app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
mock_job.status = JobStatus.AWAITING_REVIEW.value # Use .value for string comparison
|
||||
mock_job.state_data = {
|
||||
"blocking_state_entered_at": "2024-01-01T00:00:00",
|
||||
"blocking_action_type": "lyrics",
|
||||
"reminder_sent": True, # Already sent
|
||||
}
|
||||
|
||||
with patch('backend.api.routes.internal.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
test_client = TestClient(test_app)
|
||||
response = test_client.post(
|
||||
"/api/internal/jobs/test-job-123/check-idle-reminder",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
# Should succeed with "already_sent" status
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "already_sent"
|
||||
assert data["job_id"] == "test-job-123"
|
||||
@@ -0,0 +1,492 @@
|
||||
"""
|
||||
Unit tests for email service.
|
||||
|
||||
Tests the new job completion and action reminder email methods,
|
||||
as well as CC support.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, MagicMock
|
||||
|
||||
from backend.services.email_service import (
|
||||
EmailService,
|
||||
EmailProvider,
|
||||
ConsoleEmailProvider,
|
||||
SendGridEmailProvider,
|
||||
get_email_service,
|
||||
)
|
||||
|
||||
|
||||
class TestConsoleEmailProvider:
|
||||
"""Tests for console email provider."""
|
||||
|
||||
def test_send_email_basic(self):
|
||||
"""Test basic email logging."""
|
||||
provider = ConsoleEmailProvider()
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test content</p>",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
|
||||
def test_send_email_with_text_content(self):
|
||||
"""Test email with text content."""
|
||||
provider = ConsoleEmailProvider()
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
text_content="Test plain text",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
|
||||
def test_send_email_with_cc(self):
|
||||
"""Test email with CC recipients."""
|
||||
provider = ConsoleEmailProvider()
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
cc_emails=["cc1@example.com", "cc2@example.com"],
|
||||
)
|
||||
|
||||
assert result is True
|
||||
|
||||
|
||||
class TestSendGridEmailProvider:
|
||||
"""Tests for SendGrid email provider."""
|
||||
|
||||
def test_send_email_success(self):
|
||||
"""Test successful email sending via SendGrid."""
|
||||
provider = SendGridEmailProvider(
|
||||
api_key="test-api-key",
|
||||
from_email="from@example.com",
|
||||
from_name="Test Sender",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 202
|
||||
|
||||
# Patch at the sendgrid module level since it's imported inline
|
||||
with patch('sendgrid.SendGridAPIClient') as mock_sg:
|
||||
mock_client = Mock()
|
||||
mock_client.send.return_value = mock_response
|
||||
mock_sg.return_value = mock_client
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
mock_client.send.assert_called_once()
|
||||
|
||||
def test_send_email_with_cc(self):
|
||||
"""Test email sending with CC via SendGrid."""
|
||||
provider = SendGridEmailProvider(
|
||||
api_key="test-api-key",
|
||||
from_email="from@example.com",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 202
|
||||
|
||||
with patch('sendgrid.SendGridAPIClient') as mock_sg:
|
||||
mock_client = Mock()
|
||||
mock_client.send.return_value = mock_response
|
||||
mock_sg.return_value = mock_client
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
cc_emails=["cc@example.com"],
|
||||
)
|
||||
|
||||
assert result is True
|
||||
|
||||
def test_send_email_failure_status(self):
|
||||
"""Test email sending failure due to bad status."""
|
||||
provider = SendGridEmailProvider(
|
||||
api_key="test-api-key",
|
||||
from_email="from@example.com",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 400
|
||||
|
||||
with patch('sendgrid.SendGridAPIClient') as mock_sg:
|
||||
mock_client = Mock()
|
||||
mock_client.send.return_value = mock_response
|
||||
mock_sg.return_value = mock_client
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_send_email_exception(self):
|
||||
"""Test email sending exception handling."""
|
||||
provider = SendGridEmailProvider(
|
||||
api_key="test-api-key",
|
||||
from_email="from@example.com",
|
||||
)
|
||||
|
||||
with patch('sendgrid.SendGridAPIClient') as mock_sg:
|
||||
mock_sg.return_value.send.side_effect = Exception("API error")
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
|
||||
class TestEmailServiceJobCompletion:
|
||||
"""Tests for job completion email method."""
|
||||
|
||||
def test_send_job_completion_basic(self):
|
||||
"""Test basic job completion email."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
result = service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
service.provider.send_email.assert_called_once()
|
||||
|
||||
def test_send_job_completion_with_song_info(self):
|
||||
"""Test job completion email includes song in subject."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
)
|
||||
|
||||
call_args = service.provider.send_email.call_args
|
||||
subject = call_args.kwargs.get('subject') or call_args[0][1]
|
||||
assert "Test Artist" in subject
|
||||
assert "Test Song" in subject
|
||||
|
||||
def test_send_job_completion_with_brand_code(self):
|
||||
"""Test job completion email includes brand code in subject."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
artist="Seether",
|
||||
title="Tonight",
|
||||
brand_code="NOMAD-1178",
|
||||
)
|
||||
|
||||
call_args = service.provider.send_email.call_args
|
||||
subject = call_args.kwargs.get('subject') or call_args[0][1]
|
||||
# Subject format: "NOMAD-1178: Seether - Tonight (Your karaoke video is ready!)"
|
||||
assert subject == "NOMAD-1178: Seether - Tonight (Your karaoke video is ready!)"
|
||||
|
||||
def test_send_job_completion_default_subject(self):
|
||||
"""Test job completion email default subject without song info."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
)
|
||||
|
||||
call_args = service.provider.send_email.call_args
|
||||
subject = call_args.kwargs.get('subject') or call_args[0][1]
|
||||
assert "karaoke video is ready" in subject.lower()
|
||||
|
||||
def test_send_job_completion_with_cc(self):
|
||||
"""Test job completion email with CC to admin."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
cc_admin=True,
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
assert "gen@nomadkaraoke.com" in call_kwargs.get('cc_emails', [])
|
||||
|
||||
def test_send_job_completion_without_cc(self):
|
||||
"""Test job completion email without CC."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
cc_admin=False,
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
assert call_kwargs.get('cc_emails') is None
|
||||
|
||||
def test_send_job_completion_escapes_html(self):
|
||||
"""Test that message content is HTML-escaped."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Test <script>alert('xss')</script>",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
html_content = call_kwargs.get('html_content')
|
||||
assert "<script>" not in html_content
|
||||
assert "<script>" in html_content
|
||||
|
||||
def test_send_job_completion_includes_plain_text(self):
|
||||
"""Test that plain text content is included."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
message = "Your video is ready!"
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content=message,
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
assert call_kwargs.get('text_content') == message
|
||||
|
||||
|
||||
class TestEmailServiceActionReminder:
|
||||
"""Tests for action reminder email method."""
|
||||
|
||||
def test_send_action_reminder_lyrics(self):
|
||||
"""Test lyrics action reminder email."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
result = service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Please review your lyrics",
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
subject = call_kwargs.get('subject')
|
||||
assert "lyrics" in subject.lower()
|
||||
|
||||
def test_send_action_reminder_instrumental(self):
|
||||
"""Test instrumental action reminder email."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
result = service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Please select instrumental",
|
||||
action_type="instrumental",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
subject = call_kwargs.get('subject')
|
||||
assert "instrumental" in subject.lower()
|
||||
|
||||
def test_send_action_reminder_with_song_info(self):
|
||||
"""Test action reminder includes song in subject."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Please review",
|
||||
action_type="lyrics",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
subject = call_kwargs.get('subject')
|
||||
assert "Test Artist" in subject
|
||||
assert "Test Song" in subject
|
||||
|
||||
def test_send_action_reminder_unknown_type(self):
|
||||
"""Test action reminder with unknown type still sends."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
result = service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Please take action",
|
||||
action_type="unknown",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
subject = call_kwargs.get('subject')
|
||||
assert "Action needed" in subject
|
||||
|
||||
def test_send_action_reminder_escapes_html(self):
|
||||
"""Test that message content is HTML-escaped."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Test <script>alert('xss')</script>",
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
html_content = call_kwargs.get('html_content')
|
||||
assert "<script>" not in html_content
|
||||
assert "<script>" in html_content
|
||||
|
||||
def test_send_action_reminder_no_cc(self):
|
||||
"""Test that action reminders don't have CC."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Please review",
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
# Action reminders should not have CC
|
||||
assert call_kwargs.get('cc_emails') is None
|
||||
|
||||
|
||||
class TestEmailServiceConfiguration:
|
||||
"""Tests for email service configuration."""
|
||||
|
||||
def test_uses_sendgrid_when_configured(self):
|
||||
"""Test that SendGrid is used when API key is set."""
|
||||
with patch.dict('os.environ', {'SENDGRID_API_KEY': 'test-key'}):
|
||||
service = EmailService()
|
||||
assert isinstance(service.provider, SendGridEmailProvider)
|
||||
|
||||
def test_uses_console_when_not_configured(self):
|
||||
"""Test that console is used when no API key."""
|
||||
with patch.dict('os.environ', {}, clear=True):
|
||||
# Remove SENDGRID_API_KEY if it exists
|
||||
import os
|
||||
original = os.environ.pop('SENDGRID_API_KEY', None)
|
||||
try:
|
||||
service = EmailService()
|
||||
assert isinstance(service.provider, ConsoleEmailProvider)
|
||||
finally:
|
||||
if original:
|
||||
os.environ['SENDGRID_API_KEY'] = original
|
||||
|
||||
def test_is_configured_true_for_sendgrid(self):
|
||||
"""Test is_configured returns True for SendGrid."""
|
||||
service = EmailService()
|
||||
service.provider = SendGridEmailProvider("key", "from@example.com")
|
||||
assert service.is_configured() is True
|
||||
|
||||
def test_is_configured_false_for_console(self):
|
||||
"""Test is_configured returns False for console."""
|
||||
service = EmailService()
|
||||
service.provider = ConsoleEmailProvider()
|
||||
assert service.is_configured() is False
|
||||
|
||||
|
||||
class TestGlobalInstance:
|
||||
"""Tests for global instance management."""
|
||||
|
||||
def test_get_email_service_returns_same_instance(self):
|
||||
"""Test that get_email_service returns singleton."""
|
||||
# Reset global
|
||||
import backend.services.email_service as es
|
||||
es._email_service = None
|
||||
|
||||
service1 = get_email_service()
|
||||
service2 = get_email_service()
|
||||
|
||||
assert service1 is service2
|
||||
|
||||
|
||||
class TestCCFunctionality:
|
||||
"""Integration tests for CC functionality across providers."""
|
||||
|
||||
def test_console_provider_logs_cc(self, caplog):
|
||||
"""Test that console provider logs CC recipients."""
|
||||
import logging
|
||||
caplog.set_level(logging.INFO)
|
||||
|
||||
provider = ConsoleEmailProvider()
|
||||
provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test",
|
||||
html_content="<p>Test</p>",
|
||||
cc_emails=["cc1@example.com", "cc2@example.com"],
|
||||
)
|
||||
|
||||
# Check that CC was logged
|
||||
assert "cc1@example.com" in caplog.text or "CC:" in caplog.text
|
||||
|
||||
def test_sendgrid_provider_adds_cc_recipients(self):
|
||||
"""Test that SendGrid provider adds CC recipients to message."""
|
||||
provider = SendGridEmailProvider(
|
||||
api_key="test-api-key",
|
||||
from_email="from@example.com",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 202
|
||||
|
||||
# Patch at sendgrid module level since imports are inline
|
||||
with patch('sendgrid.SendGridAPIClient') as mock_sg:
|
||||
mock_client = Mock()
|
||||
mock_client.send.return_value = mock_response
|
||||
mock_sg.return_value = mock_client
|
||||
|
||||
# We need to capture the Mail object
|
||||
with patch('sendgrid.helpers.mail.Mail') as mock_mail_class:
|
||||
mock_mail = MagicMock()
|
||||
mock_mail_class.return_value = mock_mail
|
||||
|
||||
provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test",
|
||||
html_content="<p>Test</p>",
|
||||
cc_emails=["cc@example.com"],
|
||||
)
|
||||
|
||||
# Verify add_cc was called
|
||||
mock_mail.add_cc.assert_called()
|
||||
@@ -1128,7 +1128,8 @@ class TestFinaliseOnlyModels:
|
||||
assert request.enable_txt is None
|
||||
assert request.brand_prefix is None
|
||||
assert request.keep_brand_code is None
|
||||
assert request.enable_youtube_upload is False
|
||||
# YouTube upload default is None (server applies default_enable_youtube_upload)
|
||||
assert request.enable_youtube_upload is None
|
||||
assert request.dropbox_path is None
|
||||
assert request.gdrive_folder_id is None
|
||||
|
||||
|
||||
@@ -0,0 +1,329 @@
|
||||
"""
|
||||
Unit tests for job manager notification methods.
|
||||
|
||||
Tests the state transition triggers for email notifications.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, MagicMock, AsyncMock
|
||||
from datetime import datetime
|
||||
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
class TestTriggerStateNotifications:
|
||||
"""Tests for _trigger_state_notifications method."""
|
||||
|
||||
def test_triggers_completion_email_on_complete(self):
|
||||
"""Test that completion email is triggered when job completes."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {"youtube_url": "https://youtube.com/123"}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_schedule_completion_email') as mock_schedule:
|
||||
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
|
||||
|
||||
mock_schedule.assert_called_once_with(mock_job)
|
||||
|
||||
def test_triggers_idle_reminder_on_awaiting_review(self):
|
||||
"""Test that idle reminder is triggered when job enters awaiting review."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_schedule_idle_reminder') as mock_schedule:
|
||||
manager._trigger_state_notifications("job-123", JobStatus.AWAITING_REVIEW)
|
||||
|
||||
mock_schedule.assert_called_once_with(mock_job, JobStatus.AWAITING_REVIEW)
|
||||
|
||||
def test_triggers_idle_reminder_on_awaiting_instrumental(self):
|
||||
"""Test that idle reminder is triggered when job enters awaiting instrumental."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_schedule_idle_reminder') as mock_schedule:
|
||||
manager._trigger_state_notifications(
|
||||
"job-123",
|
||||
JobStatus.AWAITING_INSTRUMENTAL_SELECTION
|
||||
)
|
||||
|
||||
mock_schedule.assert_called_once_with(
|
||||
mock_job,
|
||||
JobStatus.AWAITING_INSTRUMENTAL_SELECTION
|
||||
)
|
||||
|
||||
def test_skips_notification_when_no_job(self):
|
||||
"""Test that notification is skipped when job not found."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
manager.get_job = Mock(return_value=None)
|
||||
|
||||
with patch.object(manager, '_schedule_completion_email') as mock_schedule:
|
||||
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
|
||||
|
||||
mock_schedule.assert_not_called()
|
||||
|
||||
def test_skips_notification_when_no_user_email(self):
|
||||
"""Test that notification is skipped when job has no user email."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = None
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_schedule_completion_email') as mock_schedule:
|
||||
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
|
||||
|
||||
mock_schedule.assert_not_called()
|
||||
|
||||
def test_does_not_trigger_on_other_states(self):
|
||||
"""Test that no notification is triggered for non-notification states."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_schedule_completion_email') as mock_complete, \
|
||||
patch.object(manager, '_schedule_idle_reminder') as mock_idle:
|
||||
|
||||
# Test various non-notification states
|
||||
for status in [JobStatus.PENDING, JobStatus.DOWNLOADING, JobStatus.TRANSCRIBING,
|
||||
JobStatus.RENDERING_VIDEO, JobStatus.FAILED]:
|
||||
manager._trigger_state_notifications("job-123", status)
|
||||
|
||||
mock_complete.assert_not_called()
|
||||
mock_idle.assert_not_called()
|
||||
|
||||
def test_handles_exception_gracefully(self):
|
||||
"""Test that exceptions don't propagate from notifications."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
manager.get_job = Mock(side_effect=Exception("Database error"))
|
||||
|
||||
# Should not raise
|
||||
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
|
||||
|
||||
|
||||
class TestScheduleCompletionEmail:
|
||||
"""Tests for _schedule_completion_email method."""
|
||||
|
||||
def test_extracts_urls_from_state_data(self):
|
||||
"""Test that YouTube and Dropbox URLs are extracted from state_data."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.artist = "Test Artist"
|
||||
mock_job.title = "Test Song"
|
||||
mock_job.state_data = {
|
||||
"youtube_url": "https://youtube.com/watch?v=123",
|
||||
"dropbox_link": "https://dropbox.com/folder/abc",
|
||||
}
|
||||
|
||||
with patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.send_job_completion_email = AsyncMock(return_value=True)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
# Run in sync context - this will start a daemon thread
|
||||
manager._schedule_completion_email(mock_job)
|
||||
|
||||
# Give the thread a moment to execute
|
||||
import time
|
||||
time.sleep(0.1)
|
||||
|
||||
def test_handles_none_state_data(self):
|
||||
"""Test that None state_data is handled gracefully."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.artist = None
|
||||
mock_job.title = None
|
||||
mock_job.state_data = None # None state_data
|
||||
|
||||
with patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.send_job_completion_email = AsyncMock(return_value=True)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
# Should not raise
|
||||
manager._schedule_completion_email(mock_job)
|
||||
|
||||
def test_handles_exception_gracefully(self):
|
||||
"""Test that exceptions don't propagate."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
|
||||
with patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_service:
|
||||
mock_get_service.side_effect = Exception("Service error")
|
||||
|
||||
# Should not raise
|
||||
manager._schedule_completion_email(mock_job)
|
||||
|
||||
|
||||
class TestScheduleIdleReminder:
|
||||
"""Tests for _schedule_idle_reminder method."""
|
||||
|
||||
def test_updates_state_data_for_lyrics_review(self):
|
||||
"""Test that state_data is updated with blocking state info for lyrics."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {"existing_key": "value"}
|
||||
|
||||
with patch('backend.services.worker_service.get_worker_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.schedule_idle_reminder = AsyncMock(return_value=True)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
manager._schedule_idle_reminder(mock_job, JobStatus.AWAITING_REVIEW)
|
||||
|
||||
# Verify state_data was updated
|
||||
update_call = manager.firestore.update_job.call_args
|
||||
assert update_call is not None
|
||||
updated_data = update_call[0][1] # Second positional arg
|
||||
state_data = updated_data.get('state_data', {})
|
||||
|
||||
assert state_data.get('blocking_action_type') == 'lyrics'
|
||||
assert state_data.get('reminder_sent') is False
|
||||
assert 'blocking_state_entered_at' in state_data
|
||||
assert state_data.get('existing_key') == 'value' # Preserved
|
||||
|
||||
def test_updates_state_data_for_instrumental_selection(self):
|
||||
"""Test that state_data is updated with blocking state info for instrumental."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
|
||||
with patch('backend.services.worker_service.get_worker_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.schedule_idle_reminder = AsyncMock(return_value=True)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
manager._schedule_idle_reminder(
|
||||
mock_job,
|
||||
JobStatus.AWAITING_INSTRUMENTAL_SELECTION
|
||||
)
|
||||
|
||||
# Verify state_data was updated
|
||||
update_call = manager.firestore.update_job.call_args
|
||||
updated_data = update_call[0][1]
|
||||
state_data = updated_data.get('state_data', {})
|
||||
|
||||
assert state_data.get('blocking_action_type') == 'instrumental'
|
||||
|
||||
def test_handles_none_state_data(self):
|
||||
"""Test that None state_data is handled gracefully."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = None
|
||||
|
||||
with patch('backend.services.worker_service.get_worker_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.schedule_idle_reminder = AsyncMock(return_value=True)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
# Should not raise
|
||||
manager._schedule_idle_reminder(mock_job, JobStatus.AWAITING_REVIEW)
|
||||
|
||||
# Verify update was still called
|
||||
manager.firestore.update_job.assert_called_once()
|
||||
|
||||
def test_handles_exception_gracefully(self):
|
||||
"""Test that exceptions don't propagate."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
|
||||
# Make firestore raise an exception
|
||||
manager.firestore.update_job.side_effect = Exception("Database error")
|
||||
|
||||
# Should not raise
|
||||
manager._schedule_idle_reminder(mock_job, JobStatus.AWAITING_REVIEW)
|
||||
|
||||
|
||||
class TestTransitionTriggersNotifications:
|
||||
"""Integration tests for transition_to_state triggering notifications."""
|
||||
|
||||
def test_transition_to_complete_triggers_notification(self):
|
||||
"""Test that transitioning to COMPLETE triggers notification."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
# Use ENCODING as the starting state since ENCODING -> COMPLETE is valid
|
||||
mock_job.status = JobStatus.ENCODING
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_trigger_state_notifications') as mock_trigger:
|
||||
manager.transition_to_state("job-123", JobStatus.COMPLETE)
|
||||
|
||||
mock_trigger.assert_called_once_with("job-123", JobStatus.COMPLETE)
|
||||
|
||||
def test_transition_to_awaiting_review_triggers_notification(self):
|
||||
"""Test that transitioning to AWAITING_REVIEW triggers notification."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
# Use APPLYING_PADDING as the starting state since APPLYING_PADDING -> AWAITING_REVIEW is valid
|
||||
mock_job.status = JobStatus.APPLYING_PADDING
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_trigger_state_notifications') as mock_trigger:
|
||||
manager.transition_to_state("job-123", JobStatus.AWAITING_REVIEW)
|
||||
|
||||
mock_trigger.assert_called_once_with("job-123", JobStatus.AWAITING_REVIEW)
|
||||
@@ -0,0 +1,443 @@
|
||||
"""
|
||||
Unit tests for job notification service.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, AsyncMock
|
||||
import urllib.parse
|
||||
|
||||
from backend.services.job_notification_service import (
|
||||
JobNotificationService,
|
||||
get_job_notification_service,
|
||||
ENABLE_AUTO_EMAILS,
|
||||
FEEDBACK_FORM_URL,
|
||||
)
|
||||
|
||||
|
||||
class TestURLBuilding:
|
||||
"""Tests for URL building methods."""
|
||||
|
||||
def test_build_review_url_basic(self):
|
||||
"""Test basic review URL building."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_review_url("job-123")
|
||||
|
||||
assert "gen.nomadkaraoke.com/lyrics/" in url
|
||||
assert "baseApiUrl=" in url
|
||||
# The API URL should be URL-encoded
|
||||
assert urllib.parse.quote("https://api.nomadkaraoke.com/api/review/job-123", safe='') in url
|
||||
|
||||
def test_build_review_url_with_audio_hash(self):
|
||||
"""Test review URL with audio hash parameter."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_review_url("job-123", audio_hash="abc123")
|
||||
|
||||
assert "audioHash=abc123" in url
|
||||
|
||||
def test_build_review_url_with_review_token(self):
|
||||
"""Test review URL with review token parameter."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_review_url("job-123", review_token="token456")
|
||||
|
||||
assert "reviewToken=token456" in url
|
||||
|
||||
def test_build_review_url_with_all_params(self):
|
||||
"""Test review URL with all parameters."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_review_url(
|
||||
"job-123",
|
||||
audio_hash="hash789",
|
||||
review_token="token456"
|
||||
)
|
||||
|
||||
assert "baseApiUrl=" in url
|
||||
assert "audioHash=hash789" in url
|
||||
assert "reviewToken=token456" in url
|
||||
|
||||
def test_build_review_url_encodes_special_chars(self):
|
||||
"""Test that special characters in job ID are encoded."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_review_url("job/with/slashes")
|
||||
|
||||
# The baseApiUrl parameter should have encoded slashes
|
||||
assert "%2F" in url
|
||||
|
||||
def test_build_instrumental_url_basic(self):
|
||||
"""Test basic instrumental URL building."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_instrumental_url("job-123")
|
||||
|
||||
assert "gen.nomadkaraoke.com/instrumental/" in url
|
||||
assert "baseApiUrl=" in url
|
||||
|
||||
def test_build_instrumental_url_with_token(self):
|
||||
"""Test instrumental URL with token parameter."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_instrumental_url("job-123", instrumental_token="inst-token")
|
||||
|
||||
assert "instrumentalToken=inst-token" in url
|
||||
|
||||
|
||||
class TestCompletionEmail:
|
||||
"""Tests for job completion email sending."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_success(self):
|
||||
"""Test successful completion email sending."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_job_completion.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Test message"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
user_name="Test User",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
youtube_url="https://youtube.com/watch?v=123",
|
||||
dropbox_url="https://dropbox.com/folder/abc",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
service.template_service.render_job_completion.assert_called_once()
|
||||
service.email_service.send_job_completion.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_disabled(self):
|
||||
"""Test that completion email is skipped when auto emails are disabled."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.template_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', False):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
service.email_service.send_job_completion.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_no_email(self):
|
||||
"""Test that completion email is skipped when no user email."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.template_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email=None,
|
||||
)
|
||||
|
||||
assert result is False
|
||||
service.email_service.send_job_completion.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_empty_email(self):
|
||||
"""Test that completion email is skipped when user email is empty."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.template_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
service.email_service.send_job_completion.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_send_failure(self):
|
||||
"""Test handling of email send failure."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_job_completion.return_value = False
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Test message"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_exception(self):
|
||||
"""Test handling of exceptions during email sending."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_job_completion.side_effect = Exception("Send error")
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Test message"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_passes_cc_admin(self):
|
||||
"""Test that completion email is sent with CC to admin."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_job_completion.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Test message"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
# Verify cc_admin=True was passed
|
||||
call_kwargs = service.email_service.send_job_completion.call_args.kwargs
|
||||
assert call_kwargs.get('cc_admin') is True
|
||||
|
||||
|
||||
class TestActionReminderEmail:
|
||||
"""Tests for action reminder email sending."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_lyrics_reminder_success(self):
|
||||
"""Test successful lyrics reminder email sending."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_action_reminder.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_action_needed_lyrics.return_value = "Review your lyrics"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="lyrics",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
service.template_service.render_action_needed_lyrics.assert_called_once()
|
||||
service.email_service.send_action_reminder.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_instrumental_reminder_success(self):
|
||||
"""Test successful instrumental reminder email sending."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_action_reminder.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_action_needed_instrumental.return_value = "Select instrumental"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="instrumental",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
service.template_service.render_action_needed_instrumental.assert_called_once()
|
||||
service.email_service.send_action_reminder.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_reminder_disabled(self):
|
||||
"""Test that reminder email is skipped when auto emails are disabled."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', False):
|
||||
result = await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
service.email_service.send_action_reminder.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_reminder_no_email(self):
|
||||
"""Test that reminder email is skipped when no user email."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email=None,
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_reminder_unknown_action_type(self):
|
||||
"""Test handling of unknown action type."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.template_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="unknown",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
service.email_service.send_action_reminder.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_lyrics_reminder_includes_review_url(self):
|
||||
"""Test that lyrics reminder includes correct review URL."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_action_reminder.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_action_needed_lyrics.return_value = "Review"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="lyrics",
|
||||
audio_hash="hash123",
|
||||
review_token="token456",
|
||||
)
|
||||
|
||||
# Verify the review URL was passed to template
|
||||
call_kwargs = service.template_service.render_action_needed_lyrics.call_args.kwargs
|
||||
review_url = call_kwargs.get('review_url')
|
||||
assert "audioHash=hash123" in review_url
|
||||
assert "reviewToken=token456" in review_url
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_instrumental_reminder_includes_url(self):
|
||||
"""Test that instrumental reminder includes correct URL."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_action_reminder.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_action_needed_instrumental.return_value = "Select"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="instrumental",
|
||||
instrumental_token="inst-token",
|
||||
)
|
||||
|
||||
# Verify the instrumental URL was passed to template
|
||||
call_kwargs = service.template_service.render_action_needed_instrumental.call_args.kwargs
|
||||
instrumental_url = call_kwargs.get('instrumental_url')
|
||||
assert "instrumentalToken=inst-token" in instrumental_url
|
||||
|
||||
|
||||
class TestGetCompletionMessage:
|
||||
"""Tests for get_completion_message method."""
|
||||
|
||||
def test_get_completion_message_basic(self):
|
||||
"""Test basic completion message retrieval."""
|
||||
service = JobNotificationService()
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Your video is ready!"
|
||||
|
||||
result = service.get_completion_message(
|
||||
job_id="job-123",
|
||||
user_name="Test User",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
youtube_url="https://youtube.com/watch?v=123",
|
||||
dropbox_url="https://dropbox.com/folder/abc",
|
||||
)
|
||||
|
||||
assert result == "Your video is ready!"
|
||||
service.template_service.render_job_completion.assert_called_once()
|
||||
|
||||
def test_get_completion_message_includes_feedback_url(self):
|
||||
"""Test that completion message includes feedback URL."""
|
||||
service = JobNotificationService()
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Message"
|
||||
|
||||
service.get_completion_message(job_id="job-123")
|
||||
|
||||
# Verify feedback URL was passed
|
||||
call_kwargs = service.template_service.render_job_completion.call_args.kwargs
|
||||
assert 'feedback_url' in call_kwargs
|
||||
|
||||
def test_get_completion_message_with_defaults(self):
|
||||
"""Test completion message with default values."""
|
||||
service = JobNotificationService()
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Message"
|
||||
|
||||
service.get_completion_message(job_id="job-123")
|
||||
|
||||
call_kwargs = service.template_service.render_job_completion.call_args.kwargs
|
||||
assert call_kwargs.get('job_id') == "job-123"
|
||||
# Other params should be None/default
|
||||
assert call_kwargs.get('name') is None
|
||||
assert call_kwargs.get('artist') is None
|
||||
|
||||
|
||||
class TestGlobalInstance:
|
||||
"""Tests for global instance management."""
|
||||
|
||||
def test_get_job_notification_service_returns_same_instance(self):
|
||||
"""Test that get_job_notification_service returns singleton."""
|
||||
# Reset global
|
||||
import backend.services.job_notification_service as jns
|
||||
jns._job_notification_service = None
|
||||
|
||||
service1 = get_job_notification_service()
|
||||
service2 = get_job_notification_service()
|
||||
|
||||
assert service1 is service2
|
||||
|
||||
def test_service_initializes_with_dependencies(self):
|
||||
"""Test that service initializes with email and template services."""
|
||||
service = JobNotificationService()
|
||||
|
||||
assert service.email_service is not None
|
||||
assert service.template_service is not None
|
||||
assert service.frontend_url is not None
|
||||
assert service.backend_url is not None
|
||||
@@ -0,0 +1,567 @@
|
||||
"""
|
||||
Tests for LocalPreviewEncodingService.
|
||||
|
||||
Tests cover:
|
||||
- Service initialization
|
||||
- Hardware acceleration detection (NVENC)
|
||||
- FFmpeg filter path escaping
|
||||
- ASS filter building
|
||||
- Preview FFmpeg command construction
|
||||
- Preview encoding execution
|
||||
- Error handling
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from backend.services.local_preview_encoding_service import (
|
||||
LocalPreviewEncodingService,
|
||||
PreviewEncodingConfig,
|
||||
PreviewEncodingResult,
|
||||
get_local_preview_encoding_service,
|
||||
)
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServiceInit:
|
||||
"""Test service initialization."""
|
||||
|
||||
def test_init_default_values(self):
|
||||
"""Test default initialization."""
|
||||
service = LocalPreviewEncodingService()
|
||||
assert service._nvenc_available is None # Lazy detection
|
||||
assert service._video_encoder is None # Lazy detection
|
||||
assert service._hwaccel_flags is None # Lazy detection
|
||||
|
||||
def test_init_with_custom_logger(self):
|
||||
"""Test initialization with custom logger."""
|
||||
import logging
|
||||
custom_logger = logging.getLogger("test")
|
||||
service = LocalPreviewEncodingService(logger=custom_logger)
|
||||
assert service.logger is custom_logger
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServiceHWAccel:
|
||||
"""Test hardware acceleration detection."""
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_detect_nvenc_available(self, mock_run):
|
||||
"""Test NVENC detection when available."""
|
||||
mock_run.return_value = MagicMock(returncode=0)
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._detect_nvenc_support()
|
||||
|
||||
assert result is True
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_detect_nvenc_not_available(self, mock_run):
|
||||
"""Test fallback when NVENC not available."""
|
||||
import subprocess
|
||||
mock_run.side_effect = subprocess.CalledProcessError(1, "test")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._detect_nvenc_support()
|
||||
|
||||
assert result is False
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_detect_nvenc_timeout(self, mock_run):
|
||||
"""Test NVENC detection timeout handling."""
|
||||
import subprocess
|
||||
mock_run.side_effect = subprocess.TimeoutExpired("test", 30)
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._detect_nvenc_support()
|
||||
|
||||
assert result is False
|
||||
|
||||
@patch.object(LocalPreviewEncodingService, "_detect_nvenc_support")
|
||||
def test_configure_nvenc_available(self, mock_detect):
|
||||
"""Test configuration when NVENC is available."""
|
||||
mock_detect.return_value = True
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
service._configure_hardware_acceleration()
|
||||
|
||||
assert service._nvenc_available is True
|
||||
assert service._video_encoder == "h264_nvenc"
|
||||
assert "-hwaccel" in service._hwaccel_flags
|
||||
|
||||
@patch.object(LocalPreviewEncodingService, "_detect_nvenc_support")
|
||||
def test_configure_no_nvenc(self, mock_detect):
|
||||
"""Test configuration when NVENC not available."""
|
||||
mock_detect.return_value = False
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
service._configure_hardware_acceleration()
|
||||
|
||||
assert service._nvenc_available is False
|
||||
assert service._video_encoder == "libx264"
|
||||
assert service._hwaccel_flags == []
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServiceProperties:
|
||||
"""Test property accessors that trigger lazy detection."""
|
||||
|
||||
@patch.object(LocalPreviewEncodingService, "_configure_hardware_acceleration")
|
||||
def test_nvenc_available_triggers_detection(self, mock_config):
|
||||
"""Test that nvenc_available property triggers detection."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = None
|
||||
|
||||
# Access property
|
||||
_ = service.nvenc_available
|
||||
|
||||
mock_config.assert_called_once()
|
||||
|
||||
@patch.object(LocalPreviewEncodingService, "_configure_hardware_acceleration")
|
||||
def test_video_encoder_triggers_detection(self, mock_config):
|
||||
"""Test that video_encoder property triggers detection."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._video_encoder = None
|
||||
|
||||
# Access property
|
||||
_ = service.video_encoder
|
||||
|
||||
mock_config.assert_called_once()
|
||||
|
||||
@patch.object(LocalPreviewEncodingService, "_configure_hardware_acceleration")
|
||||
def test_hwaccel_flags_triggers_detection(self, mock_config):
|
||||
"""Test that hwaccel_flags property triggers detection."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._hwaccel_flags = None
|
||||
|
||||
# Access property
|
||||
_ = service.hwaccel_flags
|
||||
|
||||
mock_config.assert_called_once()
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServicePathEscaping:
|
||||
"""Test FFmpeg filter path escaping."""
|
||||
|
||||
def test_escape_simple_path(self):
|
||||
"""Test escaping a simple path."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._escape_ffmpeg_filter_path("/simple/path.ass")
|
||||
assert result == "/simple/path.ass"
|
||||
|
||||
def test_escape_path_with_spaces(self):
|
||||
"""Test escaping path with spaces."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._escape_ffmpeg_filter_path("/path/with spaces/file.ass")
|
||||
assert "\\ " in result # Spaces should be escaped
|
||||
|
||||
def test_escape_path_with_apostrophe(self):
|
||||
"""Test escaping path with apostrophe (common in song titles)."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._escape_ffmpeg_filter_path("/path/I'm With You/file.ass")
|
||||
assert "\\\\\\'" in result # Apostrophe should be triple-backslash escaped
|
||||
|
||||
def test_escape_path_with_special_chars(self):
|
||||
"""Test escaping path with FFmpeg special characters."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._escape_ffmpeg_filter_path("/path:with[special];chars,here.ass")
|
||||
# Should escape :,[];
|
||||
assert "\\:" in result
|
||||
assert "\\[" in result
|
||||
assert "\\]" in result
|
||||
assert "\\;" in result
|
||||
assert "\\," in result
|
||||
|
||||
def test_escape_path_with_backslashes(self):
|
||||
"""Test escaping path with existing backslashes."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._escape_ffmpeg_filter_path("/path\\with\\backslashes.ass")
|
||||
assert "\\\\" in result
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServiceASSFilter:
|
||||
"""Test ASS filter building."""
|
||||
|
||||
def test_build_ass_filter_simple(self):
|
||||
"""Test building ASS filter without font."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._build_ass_filter("/path/to/file.ass")
|
||||
assert result.startswith("ass=")
|
||||
assert "fontsdir" not in result
|
||||
|
||||
def test_build_ass_filter_with_font(self):
|
||||
"""Test building ASS filter with custom font."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
font_path = os.path.join(tmpdir, "custom.ttf")
|
||||
with open(font_path, "w") as f:
|
||||
f.write("fake font")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._build_ass_filter("/path/to/file.ass", font_path)
|
||||
|
||||
assert "fontsdir=" in result
|
||||
|
||||
def test_build_ass_filter_nonexistent_font(self):
|
||||
"""Test building ASS filter with nonexistent font file."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._build_ass_filter("/path/to/file.ass", "/nonexistent/font.ttf")
|
||||
# Should not include fontsdir for nonexistent font
|
||||
assert "fontsdir" not in result
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServiceFFmpegCommand:
|
||||
"""Test FFmpeg command building."""
|
||||
|
||||
def test_build_command_solid_background_libx264(self):
|
||||
"""Test command building with solid background and libx264."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4",
|
||||
background_color="black"
|
||||
)
|
||||
|
||||
cmd = service._build_preview_ffmpeg_command(config)
|
||||
|
||||
assert "ffmpeg" in cmd
|
||||
assert "-r" in cmd and "24" in cmd # Frame rate
|
||||
assert "-c:v" in cmd
|
||||
assert "libx264" in cmd
|
||||
assert "-preset" in cmd and "superfast" in cmd
|
||||
assert "-crf" in cmd and "28" in cmd
|
||||
assert "color=c=black" in " ".join(cmd)
|
||||
|
||||
def test_build_command_with_nvenc(self):
|
||||
"""Test command building with NVENC hardware acceleration."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = True
|
||||
service._video_encoder = "h264_nvenc"
|
||||
service._hwaccel_flags = ["-hwaccel", "cuda", "-hwaccel_output_format", "cuda"]
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4"
|
||||
)
|
||||
|
||||
cmd = service._build_preview_ffmpeg_command(config)
|
||||
|
||||
assert "h264_nvenc" in cmd
|
||||
assert "-hwaccel" in cmd
|
||||
assert "-preset" in cmd and "p1" in cmd # Fastest NVENC preset
|
||||
|
||||
def test_build_command_with_background_image(self):
|
||||
"""Test command building with background image."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
bg_path = os.path.join(tmpdir, "background.png")
|
||||
with open(bg_path, "wb") as f:
|
||||
f.write(b"fake png")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4",
|
||||
background_image_path=bg_path
|
||||
)
|
||||
|
||||
cmd = service._build_preview_ffmpeg_command(config)
|
||||
|
||||
assert "-loop" in cmd
|
||||
assert "1" in cmd
|
||||
assert bg_path in cmd
|
||||
# Should have scale filter for background image
|
||||
vf_index = cmd.index("-vf")
|
||||
vf_value = cmd[vf_index + 1]
|
||||
assert "scale=" in vf_value
|
||||
assert "pad=" in vf_value
|
||||
|
||||
def test_build_command_resolution(self):
|
||||
"""Test that command uses correct preview resolution."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4"
|
||||
)
|
||||
|
||||
cmd = service._build_preview_ffmpeg_command(config)
|
||||
cmd_str = " ".join(cmd)
|
||||
|
||||
# Resolution should be 480x270
|
||||
assert "480" in cmd_str
|
||||
assert "270" in cmd_str
|
||||
|
||||
def test_build_command_audio_settings(self):
|
||||
"""Test that command uses correct audio settings."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4"
|
||||
)
|
||||
|
||||
cmd = service._build_preview_ffmpeg_command(config)
|
||||
|
||||
assert "-c:a" in cmd
|
||||
assert "aac" in cmd
|
||||
assert "-b:a" in cmd
|
||||
assert "96k" in cmd
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServiceEncode:
|
||||
"""Test preview encoding execution."""
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_encode_preview_success(self, mock_run):
|
||||
"""Test successful preview encoding."""
|
||||
mock_run.return_value = MagicMock(returncode=0, stderr="")
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
ass_path = os.path.join(tmpdir, "subs.ass")
|
||||
audio_path = os.path.join(tmpdir, "audio.flac")
|
||||
output_path = os.path.join(tmpdir, "output.mp4")
|
||||
|
||||
# Create input files
|
||||
with open(ass_path, "w") as f:
|
||||
f.write("[Script Info]\n")
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(b"fake audio")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path=ass_path,
|
||||
audio_path=audio_path,
|
||||
output_path=output_path
|
||||
)
|
||||
|
||||
result = service.encode_preview(config)
|
||||
|
||||
assert result.success is True
|
||||
assert result.output_path == output_path
|
||||
assert result.error is None
|
||||
mock_run.assert_called_once()
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_encode_preview_ffmpeg_failure(self, mock_run):
|
||||
"""Test preview encoding with FFmpeg failure."""
|
||||
mock_run.return_value = MagicMock(returncode=1, stderr="FFmpeg error message")
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
ass_path = os.path.join(tmpdir, "subs.ass")
|
||||
audio_path = os.path.join(tmpdir, "audio.flac")
|
||||
output_path = os.path.join(tmpdir, "output.mp4")
|
||||
|
||||
with open(ass_path, "w") as f:
|
||||
f.write("[Script Info]\n")
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(b"fake audio")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path=ass_path,
|
||||
audio_path=audio_path,
|
||||
output_path=output_path
|
||||
)
|
||||
|
||||
result = service.encode_preview(config)
|
||||
|
||||
assert result.success is False
|
||||
assert "FFmpeg preview encoding failed" in result.error
|
||||
|
||||
def test_encode_preview_missing_ass_file(self):
|
||||
"""Test preview encoding with missing ASS file."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
audio_path = os.path.join(tmpdir, "audio.flac")
|
||||
output_path = os.path.join(tmpdir, "output.mp4")
|
||||
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(b"fake audio")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/nonexistent/subs.ass",
|
||||
audio_path=audio_path,
|
||||
output_path=output_path
|
||||
)
|
||||
|
||||
result = service.encode_preview(config)
|
||||
|
||||
assert result.success is False
|
||||
assert "not found" in result.error
|
||||
|
||||
def test_encode_preview_missing_audio_file(self):
|
||||
"""Test preview encoding with missing audio file."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
ass_path = os.path.join(tmpdir, "subs.ass")
|
||||
output_path = os.path.join(tmpdir, "output.mp4")
|
||||
|
||||
with open(ass_path, "w") as f:
|
||||
f.write("[Script Info]\n")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path=ass_path,
|
||||
audio_path="/nonexistent/audio.flac",
|
||||
output_path=output_path
|
||||
)
|
||||
|
||||
result = service.encode_preview(config)
|
||||
|
||||
assert result.success is False
|
||||
assert "not found" in result.error
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_encode_preview_timeout(self, mock_run):
|
||||
"""Test preview encoding timeout."""
|
||||
import subprocess
|
||||
mock_run.side_effect = subprocess.TimeoutExpired("ffmpeg", 300)
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
ass_path = os.path.join(tmpdir, "subs.ass")
|
||||
audio_path = os.path.join(tmpdir, "audio.flac")
|
||||
output_path = os.path.join(tmpdir, "output.mp4")
|
||||
|
||||
with open(ass_path, "w") as f:
|
||||
f.write("[Script Info]\n")
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(b"fake audio")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path=ass_path,
|
||||
audio_path=audio_path,
|
||||
output_path=output_path
|
||||
)
|
||||
|
||||
result = service.encode_preview(config)
|
||||
|
||||
assert result.success is False
|
||||
assert "timed out" in result.error
|
||||
|
||||
|
||||
class TestPreviewEncodingConfig:
|
||||
"""Test PreviewEncodingConfig dataclass."""
|
||||
|
||||
def test_config_required_fields(self):
|
||||
"""Test that required fields must be provided."""
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4"
|
||||
)
|
||||
assert config.ass_path == "/path/to/subs.ass"
|
||||
assert config.background_color == "black" # Default
|
||||
assert config.background_image_path is None # Optional
|
||||
|
||||
def test_config_all_fields(self):
|
||||
"""Test config with all fields."""
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4",
|
||||
background_image_path="/path/to/bg.png",
|
||||
background_color="red",
|
||||
font_path="/path/to/font.ttf"
|
||||
)
|
||||
assert config.background_image_path == "/path/to/bg.png"
|
||||
assert config.background_color == "red"
|
||||
assert config.font_path == "/path/to/font.ttf"
|
||||
|
||||
|
||||
class TestPreviewEncodingResult:
|
||||
"""Test PreviewEncodingResult dataclass."""
|
||||
|
||||
def test_result_success(self):
|
||||
"""Test successful result."""
|
||||
result = PreviewEncodingResult(
|
||||
success=True,
|
||||
output_path="/path/to/output.mp4"
|
||||
)
|
||||
assert result.success is True
|
||||
assert result.error is None
|
||||
assert result.output_path == "/path/to/output.mp4"
|
||||
|
||||
def test_result_failure(self):
|
||||
"""Test failure result."""
|
||||
result = PreviewEncodingResult(
|
||||
success=False,
|
||||
error="Something went wrong"
|
||||
)
|
||||
assert result.success is False
|
||||
assert result.error == "Something went wrong"
|
||||
assert result.output_path is None
|
||||
|
||||
|
||||
class TestGetLocalPreviewEncodingService:
|
||||
"""Test factory function."""
|
||||
|
||||
def test_get_service_creates_instance(self):
|
||||
"""Test that factory function creates a new instance."""
|
||||
import backend.services.local_preview_encoding_service as module
|
||||
module._local_preview_encoding_service = None
|
||||
|
||||
service = get_local_preview_encoding_service()
|
||||
|
||||
assert service is not None
|
||||
assert isinstance(service, LocalPreviewEncodingService)
|
||||
|
||||
def test_get_service_returns_singleton(self):
|
||||
"""Test that factory function returns the same instance."""
|
||||
import backend.services.local_preview_encoding_service as module
|
||||
module._local_preview_encoding_service = None
|
||||
|
||||
service1 = get_local_preview_encoding_service()
|
||||
service2 = get_local_preview_encoding_service()
|
||||
|
||||
assert service1 is service2
|
||||
|
||||
|
||||
class TestPreviewEncodingConstants:
|
||||
"""Test that encoding constants match expected values."""
|
||||
|
||||
def test_preview_resolution(self):
|
||||
"""Test preview resolution constants."""
|
||||
service = LocalPreviewEncodingService()
|
||||
assert service.PREVIEW_WIDTH == 480
|
||||
assert service.PREVIEW_HEIGHT == 270
|
||||
|
||||
def test_preview_fps(self):
|
||||
"""Test preview frame rate constant."""
|
||||
service = LocalPreviewEncodingService()
|
||||
assert service.PREVIEW_FPS == 24
|
||||
|
||||
def test_preview_audio_bitrate(self):
|
||||
"""Test preview audio bitrate constant."""
|
||||
service = LocalPreviewEncodingService()
|
||||
assert service.PREVIEW_AUDIO_BITRATE == "96k"
|
||||
@@ -149,7 +149,8 @@ class TestUploadSubmissionRequest:
|
||||
# CDG/TXT disabled by default (requires style config)
|
||||
assert request.enable_cdg is False
|
||||
assert request.enable_txt is False
|
||||
assert request.enable_youtube_upload is False
|
||||
# YouTube upload default is None (use server default)
|
||||
assert request.enable_youtube_upload is None
|
||||
|
||||
|
||||
class TestStartReviewRequest:
|
||||
|
||||
@@ -0,0 +1,295 @@
|
||||
"""
|
||||
Unit tests for template service.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, MagicMock
|
||||
|
||||
from backend.services.template_service import (
|
||||
TemplateService,
|
||||
get_template_service,
|
||||
DEFAULT_JOB_COMPLETION_TEMPLATE,
|
||||
DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE,
|
||||
DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE,
|
||||
)
|
||||
|
||||
|
||||
class TestTemplateRendering:
|
||||
"""Tests for template rendering functionality."""
|
||||
|
||||
def test_render_template_basic(self):
|
||||
"""Test basic template variable replacement."""
|
||||
service = TemplateService()
|
||||
template = "Hello {name}, your job {job_id} is ready!"
|
||||
variables = {"name": "Alice", "job_id": "123"}
|
||||
|
||||
result = service.render_template(template, variables)
|
||||
|
||||
assert result == "Hello Alice, your job 123 is ready!"
|
||||
|
||||
def test_render_template_missing_variables(self):
|
||||
"""Test that missing variables are replaced with empty strings."""
|
||||
service = TemplateService()
|
||||
template = "Hello {name}, your {thing} is {status}!"
|
||||
variables = {"name": "Bob"}
|
||||
|
||||
result = service.render_template(template, variables)
|
||||
|
||||
assert result == "Hello Bob, your is !"
|
||||
|
||||
def test_render_template_none_values(self):
|
||||
"""Test that None values are handled correctly."""
|
||||
service = TemplateService()
|
||||
template = "Hello {name}!"
|
||||
variables = {"name": None}
|
||||
|
||||
result = service.render_template(template, variables)
|
||||
|
||||
assert result == "Hello !"
|
||||
|
||||
def test_render_template_removes_feedback_section_when_no_url(self):
|
||||
"""Test that feedback section is removed when feedback_url is not provided."""
|
||||
service = TemplateService()
|
||||
template = """Thanks for your order!
|
||||
|
||||
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
|
||||
{feedback_url}
|
||||
|
||||
Have a great day!"""
|
||||
variables = {"feedback_url": None}
|
||||
|
||||
result = service.render_template(template, variables)
|
||||
|
||||
assert "feedback" not in result.lower()
|
||||
assert "Thanks for your order!" in result
|
||||
assert "Have a great day!" in result
|
||||
|
||||
def test_render_template_keeps_feedback_section_when_url_provided(self):
|
||||
"""Test that feedback section is kept when feedback_url is provided."""
|
||||
service = TemplateService()
|
||||
template = """Thanks!
|
||||
|
||||
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
|
||||
{feedback_url}
|
||||
|
||||
Bye!"""
|
||||
variables = {"feedback_url": "https://example.com/feedback"}
|
||||
|
||||
result = service.render_template(template, variables)
|
||||
|
||||
assert "https://example.com/feedback" in result
|
||||
assert "feedback" in result.lower()
|
||||
|
||||
|
||||
class TestJobCompletionTemplate:
|
||||
"""Tests for job completion template rendering."""
|
||||
|
||||
def test_render_job_completion_all_fields(self):
|
||||
"""Test rendering job completion with all fields."""
|
||||
service = TemplateService()
|
||||
|
||||
# Mock _fetch_template_from_gcs to return None (use default)
|
||||
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
|
||||
result = service.render_job_completion(
|
||||
name="Alice",
|
||||
youtube_url="https://youtube.com/watch?v=123",
|
||||
dropbox_url="https://dropbox.com/folder/abc",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="job-123",
|
||||
feedback_url="https://example.com/feedback",
|
||||
)
|
||||
|
||||
assert "Alice" in result
|
||||
assert "https://youtube.com/watch?v=123" in result
|
||||
assert "https://dropbox.com/folder/abc" in result
|
||||
assert "https://example.com/feedback" in result
|
||||
|
||||
def test_render_job_completion_defaults(self):
|
||||
"""Test rendering job completion with default values."""
|
||||
service = TemplateService()
|
||||
|
||||
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
|
||||
result = service.render_job_completion()
|
||||
|
||||
assert "there" in result # Default name
|
||||
assert "[YouTube URL not available]" in result
|
||||
assert "[Dropbox URL not available]" in result
|
||||
|
||||
def test_render_job_completion_no_feedback_url(self):
|
||||
"""Test rendering job completion without feedback URL removes section."""
|
||||
service = TemplateService()
|
||||
|
||||
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
|
||||
result = service.render_job_completion(
|
||||
name="Bob",
|
||||
youtube_url="https://youtube.com/123",
|
||||
dropbox_url="https://dropbox.com/abc",
|
||||
)
|
||||
|
||||
assert "Bob" in result
|
||||
# Feedback section should be removed
|
||||
assert "really appreciate your feedback" not in result
|
||||
|
||||
|
||||
class TestActionNeededTemplates:
|
||||
"""Tests for action-needed template rendering."""
|
||||
|
||||
def test_render_action_needed_lyrics(self):
|
||||
"""Test rendering lyrics review reminder."""
|
||||
service = TemplateService()
|
||||
|
||||
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
|
||||
result = service.render_action_needed_lyrics(
|
||||
name="Charlie",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
review_url="https://example.com/review",
|
||||
)
|
||||
|
||||
assert "Charlie" in result
|
||||
assert "Test Artist" in result
|
||||
assert "Test Song" in result
|
||||
assert "https://example.com/review" in result
|
||||
|
||||
def test_render_action_needed_instrumental(self):
|
||||
"""Test rendering instrumental selection reminder."""
|
||||
service = TemplateService()
|
||||
|
||||
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
|
||||
result = service.render_action_needed_instrumental(
|
||||
name="Diana",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
instrumental_url="https://example.com/instrumental",
|
||||
)
|
||||
|
||||
assert "Diana" in result
|
||||
assert "Test Artist" in result
|
||||
assert "Test Song" in result
|
||||
assert "https://example.com/instrumental" in result
|
||||
|
||||
|
||||
class TestGCSFetching:
|
||||
"""Tests for GCS template fetching."""
|
||||
|
||||
def test_fetch_template_from_gcs_success(self):
|
||||
"""Test successful template fetch from GCS."""
|
||||
service = TemplateService()
|
||||
|
||||
# Mock the bucket and blob
|
||||
mock_blob = Mock()
|
||||
mock_blob.exists.return_value = True
|
||||
mock_blob.download_as_text.return_value = "Custom template {name}"
|
||||
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
|
||||
with patch.object(service, "_bucket", mock_bucket):
|
||||
result = service._fetch_template_from_gcs("test.txt")
|
||||
|
||||
assert result == "Custom template {name}"
|
||||
mock_bucket.blob.assert_called_once_with("templates/test.txt")
|
||||
|
||||
def test_fetch_template_from_gcs_not_found(self):
|
||||
"""Test template fetch when file doesn't exist in GCS."""
|
||||
service = TemplateService()
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_blob.exists.return_value = False
|
||||
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
|
||||
with patch.object(service, "_bucket", mock_bucket):
|
||||
result = service._fetch_template_from_gcs("nonexistent.txt")
|
||||
|
||||
assert result is None
|
||||
|
||||
def test_fetch_template_from_gcs_error(self):
|
||||
"""Test template fetch handles errors gracefully."""
|
||||
service = TemplateService()
|
||||
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.side_effect = Exception("GCS error")
|
||||
|
||||
with patch.object(service, "_bucket", mock_bucket):
|
||||
result = service._fetch_template_from_gcs("test.txt")
|
||||
|
||||
assert result is None
|
||||
|
||||
def test_get_job_completion_template_fallback(self):
|
||||
"""Test that default template is used when GCS fetch fails."""
|
||||
service = TemplateService()
|
||||
|
||||
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
|
||||
result = service.get_job_completion_template()
|
||||
|
||||
assert result == DEFAULT_JOB_COMPLETION_TEMPLATE
|
||||
|
||||
def test_get_action_needed_lyrics_template_fallback(self):
|
||||
"""Test that default lyrics template is used when GCS fetch fails."""
|
||||
service = TemplateService()
|
||||
|
||||
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
|
||||
result = service.get_action_needed_lyrics_template()
|
||||
|
||||
assert result == DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE
|
||||
|
||||
def test_get_action_needed_instrumental_template_fallback(self):
|
||||
"""Test that default instrumental template is used when GCS fetch fails."""
|
||||
service = TemplateService()
|
||||
|
||||
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
|
||||
result = service.get_action_needed_instrumental_template()
|
||||
|
||||
assert result == DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE
|
||||
|
||||
|
||||
class TestTemplateUpload:
|
||||
"""Tests for template upload functionality."""
|
||||
|
||||
def test_upload_template_success(self):
|
||||
"""Test successful template upload."""
|
||||
service = TemplateService()
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
|
||||
with patch.object(service, "_bucket", mock_bucket):
|
||||
result = service.upload_template("test.txt", "Test content")
|
||||
|
||||
assert result is True
|
||||
mock_bucket.blob.assert_called_once_with("templates/test.txt")
|
||||
mock_blob.upload_from_string.assert_called_once_with(
|
||||
"Test content", content_type="text/plain"
|
||||
)
|
||||
|
||||
def test_upload_template_error(self):
|
||||
"""Test template upload handles errors."""
|
||||
service = TemplateService()
|
||||
|
||||
mock_blob = Mock()
|
||||
mock_blob.upload_from_string.side_effect = Exception("Upload failed")
|
||||
mock_bucket = Mock()
|
||||
mock_bucket.blob.return_value = mock_blob
|
||||
|
||||
with patch.object(service, "_bucket", mock_bucket):
|
||||
result = service.upload_template("test.txt", "Test content")
|
||||
|
||||
assert result is False
|
||||
|
||||
|
||||
class TestGlobalInstance:
|
||||
"""Tests for global instance management."""
|
||||
|
||||
def test_get_template_service_returns_same_instance(self):
|
||||
"""Test that get_template_service returns singleton."""
|
||||
# Reset global
|
||||
import backend.services.template_service as ts
|
||||
ts._template_service = None
|
||||
|
||||
service1 = get_template_service()
|
||||
service2 = get_template_service()
|
||||
|
||||
assert service1 is service2
|
||||
@@ -425,6 +425,111 @@ class TestVideoWorkerOrchestratorEncoding:
|
||||
|
||||
assert "Encoding failed" in str(exc_info.value)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_encoding_gce_downloads_files(self):
|
||||
"""Test that GCE encoding downloads files from GCS to local directory.
|
||||
|
||||
This test verifies the fix for YouTube upload failure when using GCE encoding.
|
||||
GCE encoding returns GCS blob paths, which need to be downloaded locally
|
||||
before YouTube upload can access them.
|
||||
"""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path=os.path.join(temp_dir, "title.mov"),
|
||||
karaoke_video_path=os.path.join(temp_dir, "karaoke.mov"),
|
||||
instrumental_audio_path=os.path.join(temp_dir, "audio.flac"),
|
||||
output_dir=temp_dir,
|
||||
)
|
||||
|
||||
# Mock storage service
|
||||
mock_storage = MagicMock()
|
||||
mock_storage.download_file = MagicMock()
|
||||
|
||||
orchestrator = VideoWorkerOrchestrator(
|
||||
config,
|
||||
storage=mock_storage,
|
||||
)
|
||||
|
||||
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
|
||||
from backend.services.encoding_interface import EncodingOutput
|
||||
|
||||
# GCE backend returns GCS blob paths (not local paths)
|
||||
mock_backend = MagicMock()
|
||||
mock_backend.name = "gce" # Important: must be "gce" to trigger download
|
||||
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
|
||||
success=True,
|
||||
lossless_4k_mp4_path="jobs/test-job/finals/output_4k_lossless.mp4",
|
||||
lossy_4k_mp4_path="jobs/test-job/finals/output_4k_lossy.mp4",
|
||||
lossy_720p_mp4_path="jobs/test-job/finals/output_720p.mp4",
|
||||
lossless_mkv_path="jobs/test-job/finals/output_4k.mkv",
|
||||
encoding_time_seconds=60.0,
|
||||
encoding_backend="gce",
|
||||
))
|
||||
mock_get.return_value = mock_backend
|
||||
|
||||
await orchestrator._run_encoding()
|
||||
|
||||
# Verify download_file was called for each output file
|
||||
assert mock_storage.download_file.call_count == 4
|
||||
|
||||
# Verify the result paths were updated to local paths
|
||||
assert orchestrator.result.final_video.startswith(temp_dir)
|
||||
assert orchestrator.result.final_video_mkv.startswith(temp_dir)
|
||||
assert orchestrator.result.final_video_lossy.startswith(temp_dir)
|
||||
assert orchestrator.result.final_video_720p.startswith(temp_dir)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_encoding_local_does_not_download(self):
|
||||
"""Test that local encoding does NOT trigger GCS download.
|
||||
|
||||
Local encoding produces files directly in the output directory,
|
||||
so no download is needed.
|
||||
"""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path=os.path.join(temp_dir, "title.mov"),
|
||||
karaoke_video_path=os.path.join(temp_dir, "karaoke.mov"),
|
||||
instrumental_audio_path=os.path.join(temp_dir, "audio.flac"),
|
||||
output_dir=temp_dir,
|
||||
)
|
||||
|
||||
# Mock storage service
|
||||
mock_storage = MagicMock()
|
||||
mock_storage.download_file = MagicMock()
|
||||
|
||||
orchestrator = VideoWorkerOrchestrator(
|
||||
config,
|
||||
storage=mock_storage,
|
||||
)
|
||||
|
||||
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
|
||||
from backend.services.encoding_interface import EncodingOutput
|
||||
|
||||
# Local backend returns local paths directly
|
||||
mock_backend = MagicMock()
|
||||
mock_backend.name = "local" # Local backend
|
||||
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
|
||||
success=True,
|
||||
lossless_4k_mp4_path=os.path.join(temp_dir, "lossless.mp4"),
|
||||
lossy_4k_mp4_path=os.path.join(temp_dir, "lossy.mp4"),
|
||||
lossy_720p_mp4_path=os.path.join(temp_dir, "720p.mp4"),
|
||||
lossless_mkv_path=os.path.join(temp_dir, "lossless.mkv"),
|
||||
encoding_time_seconds=120.0,
|
||||
encoding_backend="local",
|
||||
))
|
||||
mock_get.return_value = mock_backend
|
||||
|
||||
await orchestrator._run_encoding()
|
||||
|
||||
# Verify download_file was NOT called for local encoding
|
||||
mock_storage.download_file.assert_not_called()
|
||||
|
||||
|
||||
class TestVideoWorkerOrchestratorOrganization:
|
||||
"""Test organization stage."""
|
||||
|
||||
@@ -639,40 +639,45 @@ async def upload_lyrics_results(
|
||||
logger.error(f"Job {job_id}: {error_msg}")
|
||||
raise Exception(error_msg)
|
||||
|
||||
# Upload reference lyrics if available
|
||||
# Upload ALL reference lyrics files (not just first) so they're available for distribution
|
||||
# Note: Source names use .title() so "lrclib" -> "Lrclib", "genius" -> "Genius"
|
||||
# Use sanitized artist/title to match LyricsProcessor's file naming
|
||||
reference_files = [
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Genius).txt", "Genius"),
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Spotify).txt", "Spotify"),
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Musixmatch).txt", "Musixmatch"),
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Lrclib).txt", "LRCLib"), # Note: source is 'lrclib', .title() = 'Lrclib'
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Genius).txt", "genius"),
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Spotify).txt", "spotify"),
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Musixmatch).txt", "musixmatch"),
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Lrclib).txt", "lrclib"),
|
||||
]
|
||||
|
||||
found_reference = False
|
||||
for ref_filename, source_name in reference_files:
|
||||
|
||||
found_references = []
|
||||
for ref_filename, source_key in reference_files:
|
||||
ref_path = os.path.join(lyrics_dir, ref_filename)
|
||||
if os.path.exists(ref_path):
|
||||
# Upload with original filename to preserve proper naming
|
||||
gcs_path = f"jobs/{job_id}/lyrics/{ref_filename}"
|
||||
url = storage.upload_file(ref_path, gcs_path)
|
||||
# Track in job.files so video_worker can download for distribution
|
||||
job_manager.update_file_url(job_id, 'lyrics', f'reference_{source_key}', url)
|
||||
if job_log:
|
||||
job_log.info(f"Found reference lyrics from {source_name}")
|
||||
job_log.info(f"Found reference lyrics from {source_key}")
|
||||
logger.info(f"Job {job_id}: Uploaded reference lyrics: {ref_filename}")
|
||||
found_reference = True
|
||||
break # Only upload first available reference
|
||||
|
||||
if not found_reference:
|
||||
found_references.append(source_key)
|
||||
|
||||
if not found_references:
|
||||
if job_log:
|
||||
job_log.warning("No reference lyrics found from any source (Genius, Spotify, Musixmatch, LRCLib)")
|
||||
logger.warning(f"Job {job_id}: No reference lyrics found from any source")
|
||||
else:
|
||||
logger.info(f"Job {job_id}: Found {len(found_references)} reference lyrics sources: {found_references}")
|
||||
|
||||
# Upload uncorrected transcription if available (use sanitized names)
|
||||
uncorrected_file = os.path.join(lyrics_dir, f"{safe_artist} - {safe_title} (Lyrics Uncorrected).txt")
|
||||
# Upload uncorrected transcription if available (preserve original filename for distribution)
|
||||
uncorrected_filename = f"{safe_artist} - {safe_title} (Lyrics Uncorrected).txt"
|
||||
uncorrected_file = os.path.join(lyrics_dir, uncorrected_filename)
|
||||
if os.path.exists(uncorrected_file):
|
||||
gcs_path = f"jobs/{job_id}/lyrics/uncorrected.txt"
|
||||
gcs_path = f"jobs/{job_id}/lyrics/{uncorrected_filename}"
|
||||
url = storage.upload_file(uncorrected_file, gcs_path)
|
||||
job_manager.update_file_url(job_id, 'lyrics', 'uncorrected', url)
|
||||
logger.info(f"Job {job_id}: Uploaded uncorrected transcription")
|
||||
logger.info(f"Job {job_id}: Uploaded uncorrected transcription: {uncorrected_filename}")
|
||||
|
||||
logger.info(f"Job {job_id}: All lyrics results uploaded successfully")
|
||||
|
||||
@@ -160,14 +160,17 @@ async def _encode_via_gce(
|
||||
storage.download_file(gcs_path, local_path)
|
||||
|
||||
# Map to result keys expected by _upload_results
|
||||
if "4K Lossless" in filename or "4k_lossless" in filename.lower():
|
||||
local_files["final_video"] = local_path
|
||||
elif "4K Lossy" in filename or "4k_lossy" in filename.lower():
|
||||
# Files are named like "Artist - Title (Final Karaoke Lossless 4k).mp4"
|
||||
filename_lower = filename.lower()
|
||||
if "lossless 4k" in filename_lower:
|
||||
if filename.endswith(".mkv"):
|
||||
local_files["final_video_mkv"] = local_path
|
||||
else:
|
||||
local_files["final_video"] = local_path
|
||||
elif "lossy 4k" in filename_lower:
|
||||
local_files["final_video_lossy"] = local_path
|
||||
elif "720p" in filename:
|
||||
elif "720p" in filename_lower:
|
||||
local_files["final_video_720p"] = local_path
|
||||
elif filename.endswith(".mkv"):
|
||||
local_files["final_video_mkv"] = local_path
|
||||
|
||||
job_log.info(f"Downloaded {len(local_files)} encoded files")
|
||||
|
||||
@@ -332,16 +335,9 @@ async def generate_video_orchestrated(job_id: str) -> bool:
|
||||
'final_karaoke_txt_zip': result.final_karaoke_txt_zip,
|
||||
})
|
||||
|
||||
# Mark job as complete
|
||||
# Store result metadata in job BEFORE transitioning to COMPLETE
|
||||
# This ensures youtube_url is available when completion email is sent
|
||||
logger.info(f"[job:{job_id}] Video generation complete")
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.COMPLETE,
|
||||
progress=100,
|
||||
message="Karaoke generation complete!"
|
||||
)
|
||||
|
||||
# Store result metadata in job
|
||||
job_manager.update_job(job_id, {
|
||||
'state_data': {
|
||||
**job.state_data,
|
||||
@@ -352,6 +348,14 @@ async def generate_video_orchestrated(job_id: str) -> bool:
|
||||
}
|
||||
})
|
||||
|
||||
# Mark job as complete (triggers completion email with youtube_url now available)
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.COMPLETE,
|
||||
progress=100,
|
||||
message="Karaoke generation complete!"
|
||||
)
|
||||
|
||||
duration = time.time() - start_time
|
||||
root_span.set_attribute("duration_seconds", duration)
|
||||
root_span.set_attribute("brand_code", result.brand_code or '')
|
||||
@@ -662,19 +666,12 @@ async def generate_video_legacy(job_id: str) -> bool:
|
||||
with job_span("upload-results", job_id):
|
||||
await _upload_results(job_id, job_manager, storage, temp_dir, result)
|
||||
|
||||
# Mark job as complete
|
||||
logger.info(f"[job:{job_id}] Video generation complete")
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.COMPLETE,
|
||||
progress=100,
|
||||
message="Karaoke generation complete!"
|
||||
)
|
||||
|
||||
# Store result metadata in job
|
||||
# Store result metadata in job BEFORE transitioning to COMPLETE
|
||||
# This ensures youtube_url is available when completion email is sent
|
||||
# NOTE: Must include dropbox_link and gdrive_files from result, since
|
||||
# _handle_native_distribution already saved them but job.state_data
|
||||
# is stale (fetched before distribution ran)
|
||||
logger.info(f"[job:{job_id}] Video generation complete")
|
||||
job_manager.update_job(job_id, {
|
||||
'state_data': {
|
||||
**job.state_data,
|
||||
@@ -684,7 +681,15 @@ async def generate_video_legacy(job_id: str) -> bool:
|
||||
'gdrive_files': result.get('gdrive_files'),
|
||||
}
|
||||
})
|
||||
|
||||
|
||||
# Mark job as complete (triggers completion email with youtube_url now available)
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.COMPLETE,
|
||||
progress=100,
|
||||
message="Karaoke generation complete!"
|
||||
)
|
||||
|
||||
duration = time.time() - start_time
|
||||
root_span.set_attribute("duration_seconds", duration)
|
||||
root_span.set_attribute("brand_code", result.get('brand_code', ''))
|
||||
@@ -1108,29 +1113,63 @@ async def _prepare_distribution_directory(
|
||||
log_progress(f"Downloaded {stem_key} to stems/")
|
||||
|
||||
# --- Download lyrics files to lyrics/ subfolder ---
|
||||
# Use proper filenames matching local CLI output structure
|
||||
|
||||
# Common lyrics file types
|
||||
lyrics_file_keys = ['lrc', 'ass', 'srt', 'txt', 'json', 'original_json', 'corrected_json']
|
||||
for lyrics_key in lyrics_file_keys:
|
||||
# Map job.files.lyrics keys to proper local filenames
|
||||
# Format: (job_files_key, local_filename_pattern)
|
||||
lyrics_file_mappings = [
|
||||
# Karaoke output files
|
||||
('lrc', f"{base_name} (Karaoke).lrc"),
|
||||
('ass', f"{base_name} (Karaoke).ass"),
|
||||
# Corrections and transcription files
|
||||
('corrections', f"{base_name} (Lyrics Corrections).json"),
|
||||
('corrected_txt', f"{base_name} (Lyrics Corrected).txt"),
|
||||
('uncorrected', f"{base_name} (Lyrics Uncorrected).txt"),
|
||||
# Reference lyrics from various sources
|
||||
('reference_genius', f"{base_name} (Lyrics Genius).txt"),
|
||||
('reference_spotify', f"{base_name} (Lyrics Spotify).txt"),
|
||||
('reference_musixmatch', f"{base_name} (Lyrics Musixmatch).txt"),
|
||||
('reference_lrclib', f"{base_name} (Lyrics Lrclib).txt"),
|
||||
]
|
||||
|
||||
for lyrics_key, local_filename in lyrics_file_mappings:
|
||||
if lyrics.get(lyrics_key):
|
||||
# Determine proper extension
|
||||
ext_map = {
|
||||
'lrc': '.lrc',
|
||||
'ass': '.ass',
|
||||
'srt': '.srt',
|
||||
'txt': '.txt',
|
||||
'json': '.json',
|
||||
'original_json': '_original.json',
|
||||
'corrected_json': '_corrected.json',
|
||||
}
|
||||
ext = ext_map.get(lyrics_key, f'.{lyrics_key}')
|
||||
dest_path = os.path.join(lyrics_dir, f"{base_name} (Karaoke){ext}")
|
||||
dest_path = os.path.join(lyrics_dir, local_filename)
|
||||
try:
|
||||
storage.download_file(lyrics[lyrics_key], dest_path)
|
||||
log_progress(f"Downloaded {lyrics_key} to lyrics/")
|
||||
log_progress(f"Downloaded {lyrics_key} to lyrics/{local_filename}")
|
||||
except Exception as e:
|
||||
log_progress(f"Could not download {lyrics_key}: {e}")
|
||||
|
||||
# --- Download (With Vocals).mkv to lyrics/ subfolder ---
|
||||
# Local CLI places the karaoke video with vocals in the lyrics folder
|
||||
videos = job.file_urls.get('videos', {})
|
||||
if videos.get('with_vocals'):
|
||||
with_vocals_dest = os.path.join(lyrics_dir, f"{base_name} (With Vocals).mkv")
|
||||
try:
|
||||
storage.download_file(videos['with_vocals'], with_vocals_dest)
|
||||
log_progress(f"Downloaded with_vocals to lyrics/{base_name} (With Vocals).mkv")
|
||||
except Exception as e:
|
||||
log_progress(f"Could not download with_vocals to lyrics/: {e}")
|
||||
|
||||
# --- Download preview ASS files to lyrics/previews/ subfolder ---
|
||||
# Preview files are stored in jobs/{job_id}/previews/ in GCS
|
||||
previews_dir = os.path.join(lyrics_dir, "previews")
|
||||
os.makedirs(previews_dir, exist_ok=True)
|
||||
|
||||
# List all preview files from GCS and download them
|
||||
try:
|
||||
preview_prefix = f"jobs/{job_id}/previews/"
|
||||
preview_files = storage.list_files(preview_prefix)
|
||||
for blob_name in preview_files:
|
||||
if blob_name.endswith('.ass'):
|
||||
filename = os.path.basename(blob_name)
|
||||
dest_path = os.path.join(previews_dir, filename)
|
||||
storage.download_file(blob_name, dest_path)
|
||||
log_progress(f"Downloaded preview {filename} to lyrics/previews/")
|
||||
except Exception as e:
|
||||
log_progress(f"Could not download preview files: {e}")
|
||||
|
||||
log_progress("Distribution directory prepared with stems/ and lyrics/ subfolders")
|
||||
|
||||
|
||||
|
||||
@@ -367,15 +367,70 @@ class VideoWorkerOrchestrator:
|
||||
if not output.success:
|
||||
raise Exception(f"Encoding failed: {output.error_message}")
|
||||
|
||||
# Store results
|
||||
# Store results - for GCE backend, these are GCS blob paths that need to be downloaded
|
||||
self.result.final_video = output.lossless_4k_mp4_path
|
||||
self.result.final_video_mkv = output.lossless_mkv_path
|
||||
self.result.final_video_lossy = output.lossy_4k_mp4_path
|
||||
self.result.final_video_720p = output.lossy_720p_mp4_path
|
||||
self.result.encoding_time_seconds = output.encoding_time_seconds
|
||||
|
||||
# For GCE encoding, download the encoded files from GCS to local directory
|
||||
# This is required for YouTube upload and other local file operations
|
||||
if encoding_backend.name == "gce" and self.storage:
|
||||
await self._download_gce_encoded_files(output)
|
||||
|
||||
self.job_log.info(f"Encoding complete ({encoding_backend.name}) in {output.encoding_time_seconds:.1f}s")
|
||||
|
||||
async def _download_gce_encoded_files(self, output):
|
||||
"""
|
||||
Download GCE-encoded files from GCS to the local output directory.
|
||||
|
||||
GCE encoding stores files in GCS and returns blob paths like:
|
||||
'jobs/{job_id}/finals/Artist - Title (Final Karaoke Lossless 4k).mp4'
|
||||
|
||||
This method downloads those files locally so that subsequent stages
|
||||
(YouTube upload, etc.) can access them as local files.
|
||||
|
||||
Args:
|
||||
output: EncodingOutput from the GCE backend with GCS blob paths
|
||||
"""
|
||||
self.job_log.info("Downloading GCE-encoded files from GCS")
|
||||
|
||||
# Map of result attributes to download
|
||||
file_mappings = [
|
||||
('lossless_4k_mp4_path', 'final_video'),
|
||||
('lossless_mkv_path', 'final_video_mkv'),
|
||||
('lossy_4k_mp4_path', 'final_video_lossy'),
|
||||
('lossy_720p_mp4_path', 'final_video_720p'),
|
||||
]
|
||||
|
||||
downloaded_count = 0
|
||||
for output_attr, result_attr in file_mappings:
|
||||
gcs_path = getattr(output, output_attr, None)
|
||||
if not gcs_path:
|
||||
continue
|
||||
|
||||
# Extract filename from GCS path
|
||||
filename = os.path.basename(gcs_path)
|
||||
local_path = os.path.join(self.config.output_dir, filename)
|
||||
|
||||
try:
|
||||
self.job_log.info(f"Downloading {filename} from GCS")
|
||||
self.storage.download_file(gcs_path, local_path)
|
||||
|
||||
# Update the result to point to local file
|
||||
setattr(self.result, result_attr, local_path)
|
||||
downloaded_count += 1
|
||||
self.job_log.info(f"Downloaded {filename} to {local_path}")
|
||||
|
||||
except Exception as e:
|
||||
self.job_log.error(f"Failed to download {filename}: {e}")
|
||||
# Clear the result attribute so downstream doesn't try to use invalid GCS path
|
||||
setattr(self.result, result_attr, None)
|
||||
# Don't fail - some formats might not be generated
|
||||
|
||||
self.job_log.info(f"Downloaded {downloaded_count} encoded files from GCS")
|
||||
|
||||
async def _run_organization(self):
|
||||
"""Run the organization stage (brand code generation)."""
|
||||
self.job_log.info("Starting organization stage")
|
||||
|
||||
+127
-2
@@ -20,6 +20,30 @@ GET /health
|
||||
|
||||
No auth required. Returns service status.
|
||||
|
||||
#### Encoding Worker Health
|
||||
|
||||
```http
|
||||
GET /health/encoding-worker
|
||||
```
|
||||
|
||||
No auth required. Returns GCE encoding worker status for frontend display.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"available": true,
|
||||
"status": "ok",
|
||||
"version": "0.95.1",
|
||||
"active_jobs": 0,
|
||||
"queue_length": 0
|
||||
}
|
||||
```
|
||||
|
||||
Status values:
|
||||
- `ok` - Worker is healthy and available
|
||||
- `offline` - Worker is unavailable (includes `error` field)
|
||||
- `not_configured` - Encoding worker not configured on backend
|
||||
|
||||
### Jobs
|
||||
|
||||
#### Create Job with Upload
|
||||
@@ -114,11 +138,52 @@ Content-Type: application/json
|
||||
|
||||
### Audio Search
|
||||
|
||||
#### Search for Audio
|
||||
|
||||
```http
|
||||
GET /api/audio-search?q=song+name
|
||||
POST /api/audio-search/search
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"artist": "Artist Name",
|
||||
"title": "Song Title",
|
||||
"auto_download": false,
|
||||
"theme_id": "nomad",
|
||||
"display_artist": "Display Artist (optional)",
|
||||
"display_title": "Display Title (optional)"
|
||||
}
|
||||
```
|
||||
|
||||
Search for songs (flacfetch integration).
|
||||
Searches for audio sources (flacfetch integration). Creates a job and returns search results.
|
||||
|
||||
**Key fields:**
|
||||
- `artist`, `title` - Used to search for audio on trackers
|
||||
- `display_artist`, `display_title` - Optional overrides for how artist/title appear on title screens and filenames. If omitted, search values are used.
|
||||
- `auto_download` - If true, automatically selects best result and starts processing
|
||||
- `theme_id` - Video theme to use
|
||||
|
||||
**Use case:** Search for "Jeremy Kushnier" but display "Footloose (Broadway Cast)" on the karaoke video.
|
||||
|
||||
#### Get Search Results
|
||||
|
||||
```http
|
||||
GET /api/audio-search/{job_id}/results
|
||||
```
|
||||
|
||||
Returns cached search results for a job awaiting audio selection.
|
||||
|
||||
#### Select Audio Source
|
||||
|
||||
```http
|
||||
POST /api/audio-search/{job_id}/select
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"selection_index": 0
|
||||
}
|
||||
```
|
||||
|
||||
Selects an audio source from the search results and starts processing.
|
||||
|
||||
### Themes
|
||||
|
||||
@@ -487,6 +552,66 @@ Response:
|
||||
}
|
||||
```
|
||||
|
||||
### Email Notifications (Admin)
|
||||
|
||||
#### Get Completion Message
|
||||
|
||||
```http
|
||||
GET /api/admin/jobs/{job_id}/completion-message
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Returns the rendered completion message for a job (for copying to clipboard).
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"job_id": "abc123",
|
||||
"message": "Hi there! Your karaoke video is ready...",
|
||||
"subject": "Your Karaoke Video is Ready! 🎤",
|
||||
"youtube_url": "https://youtube.com/watch?v=...",
|
||||
"dropbox_url": "https://dropbox.com/..."
|
||||
}
|
||||
```
|
||||
|
||||
#### Send Completion Email
|
||||
|
||||
```http
|
||||
POST /api/admin/jobs/{job_id}/send-completion-email
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"to_email": "customer@example.com",
|
||||
"cc_admin": true
|
||||
}
|
||||
```
|
||||
|
||||
Sends the job completion email to the specified address. When `cc_admin` is true (default), CCs gen@nomadkaraoke.com.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"job_id": "abc123",
|
||||
"to_email": "customer@example.com",
|
||||
"message": "Completion email sent successfully"
|
||||
}
|
||||
```
|
||||
|
||||
### Internal Email Endpoints
|
||||
|
||||
These endpoints are used by Cloud Tasks for automated notifications.
|
||||
|
||||
#### Check Idle Reminder
|
||||
|
||||
```http
|
||||
POST /api/internal/jobs/{job_id}/check-idle-reminder
|
||||
Authorization: Bearer ADMIN_TOKEN
|
||||
```
|
||||
|
||||
Called by Cloud Tasks 5 minutes after a job enters a blocking state (awaiting_review or awaiting_instrumental_selection). Sends a reminder email if the user is still idle and no reminder has been sent yet.
|
||||
|
||||
## Rate Limits
|
||||
|
||||
No rate limits currently implemented.
|
||||
|
||||
+12
-3
@@ -31,7 +31,8 @@
|
||||
└──────────┘ └──────────┘ └──────────────┘ └──────────────┘
|
||||
|
||||
(*) GCE Encoding Worker: C4-standard-8 VM with Intel Granite Rapids for
|
||||
high-performance FFmpeg encoding. Optional - falls back to Cloud Run.
|
||||
high-performance FFmpeg encoding. Used for both final video encoding and
|
||||
preview video generation. Uses LocalEncodingService via GCS wheel deployment.
|
||||
```
|
||||
|
||||
## Processing Pipeline
|
||||
@@ -57,7 +58,7 @@
|
||||
▼
|
||||
4. HUMAN REVIEW (AWAITING_REVIEW)
|
||||
• React UI for lyrics correction
|
||||
• Preview video generation
|
||||
• Preview video generation (optionally via GCE for speed)
|
||||
• Submit corrections
|
||||
▼
|
||||
5. VIDEO RENDERING
|
||||
@@ -154,6 +155,8 @@ LyricsTranscriber LyricsTranscriber
|
||||
| Vertex AI | Agentic AI correction (Gemini 3 Flash) | Default on |
|
||||
| Genius | Reference lyrics | Yes |
|
||||
| YouTube API | Video upload | Optional |
|
||||
| SendGrid | Email notifications | Optional |
|
||||
| Cloud Tasks | Delayed task scheduling (idle reminders) | Optional |
|
||||
|
||||
## Firestore Collections
|
||||
|
||||
@@ -190,7 +193,8 @@ The Video Worker uses an orchestrator pattern to ensure all features work regard
|
||||
│ └─ YouTube upload │
|
||||
│ │
|
||||
│ Stage 5: NOTIFICATIONS │
|
||||
│ └─ Discord webhooks │
|
||||
│ ├─ Discord webhooks │
|
||||
│ └─ Email notifications (completion, reminders) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
@@ -208,6 +212,9 @@ The Video Worker uses an orchestrator pattern to ensure all features work regard
|
||||
| `local_encoding_service.py` | FFmpeg encoding (Cloud Run) |
|
||||
| `encoding_interface.py` | Abstract interface + GCE backend |
|
||||
| `video_worker_orchestrator.py` | Stage coordination |
|
||||
| `email_service.py` | SendGrid email delivery with CC support |
|
||||
| `template_service.py` | GCS-backed email templates |
|
||||
| `job_notification_service.py` | Email orchestration (completion, reminders) |
|
||||
|
||||
## Tech Stack
|
||||
|
||||
@@ -216,5 +223,7 @@ The Video Worker uses an orchestrator pattern to ensure all features work regard
|
||||
- **Database**: Firestore
|
||||
- **Storage**: Google Cloud Storage
|
||||
- **Secrets**: Google Secret Manager
|
||||
- **Task Queues**: Cloud Tasks (idle reminders)
|
||||
- **Email**: SendGrid
|
||||
- **IaC**: Pulumi
|
||||
- **CI/CD**: GitHub Actions
|
||||
+29
-5
@@ -110,17 +110,41 @@ Push to `main` triggers GitHub Actions:
|
||||
3. Deploy to Cloud Run
|
||||
4. Deploy frontend to Cloudflare Pages
|
||||
|
||||
### Manual Backend Deploy
|
||||
### Infrastructure Changes (Pulumi)
|
||||
|
||||
**All infrastructure changes must go through PRs.** The CI pipeline automatically deploys Pulumi changes when PRs merge to main.
|
||||
|
||||
```bash
|
||||
# Preview infrastructure changes
|
||||
# Preview changes locally (requires GCP auth)
|
||||
cd infrastructure
|
||||
pulumi preview
|
||||
pulumi preview --stack nomadkaraoke/karaoke-gen-infrastructure/prod
|
||||
|
||||
# Apply changes
|
||||
pulumi up
|
||||
# DO NOT run `pulumi up` locally - let CI handle deployment
|
||||
# CI uses --skip-preview due to limited service account permissions
|
||||
```
|
||||
|
||||
**What's managed by Pulumi:**
|
||||
- Firestore database and indexes
|
||||
- GCS buckets
|
||||
- Cloud Run service configuration
|
||||
- Cloud Tasks queues
|
||||
- Secret Manager secrets
|
||||
- Service accounts and IAM bindings
|
||||
- Workload Identity Federation
|
||||
- GCE instances (GitHub runners, encoding worker)
|
||||
|
||||
**Workflow:**
|
||||
1. Make changes in the appropriate module under `infrastructure/`
|
||||
- `modules/` - Core GCP resources (database, storage, secrets, IAM, etc.)
|
||||
- `compute/` - VM definitions and startup scripts
|
||||
- `config.py` - Shared configuration constants
|
||||
- `__main__.py` - Entry point that wires modules together
|
||||
2. Run `pulumi preview` locally to verify
|
||||
3. Create PR and merge
|
||||
4. CI deploys automatically
|
||||
|
||||
See `docs/LESSONS-LEARNED.md` for Pulumi CI gotchas.
|
||||
|
||||
### Manual Frontend Deploy
|
||||
|
||||
```bash
|
||||
|
||||
@@ -222,6 +222,52 @@ escaped = filename.replace("'", "\\'")
|
||||
query = f"name='{escaped}' and '{parent_id}' in parents"
|
||||
```
|
||||
|
||||
### FFmpeg Filter Path Escaping (subprocess without shell)
|
||||
|
||||
**Problem**: Song titles with apostrophes (e.g., "I'm With You") cause FFmpeg's ASS subtitle filter to fail because paths aren't escaped correctly.
|
||||
|
||||
```python
|
||||
# BAD - FFmpeg strips the apostrophe, looks for wrong path
|
||||
ass_filter = f"ass=./Avril Lavigne - I'm With You/cache/temp.ass"
|
||||
# FFmpeg error: Could not create a libass track when reading file './Avril Lavigne - Im With You/cache/temp.ass'
|
||||
```
|
||||
|
||||
**Wrong approach #1**: Simple backslash escaping like `\'` doesn't work because FFmpeg's filter parser requires more escaping.
|
||||
|
||||
**Wrong approach #2**: Shell-style escaping `'\''` (end quote, escaped quote, start quote) only works when passing through a shell. When using Python `subprocess` with a command list, there's no shell - arguments go directly to FFmpeg, so shell escaping patterns don't work.
|
||||
|
||||
**Solution**: Use FFmpeg filter escaping (for subprocess without shell):
|
||||
```python
|
||||
def _escape_ffmpeg_filter_path(path: str) -> str:
|
||||
"""Escape a path for FFmpeg filter expressions (for subprocess without shell).
|
||||
|
||||
FFmpeg filter escaping rules:
|
||||
- Backslashes: double them (\ -> \\)
|
||||
- Single quotes/apostrophes: escape with three backslashes (' -> \\\')
|
||||
- Spaces: escape with backslash ( -> \ )
|
||||
"""
|
||||
# First escape existing backslashes (\ -> \\)
|
||||
escaped = path.replace("\\", "\\\\")
|
||||
# Escape single quotes (' -> \\\')
|
||||
# In Python source: 6 backslashes + apostrophe = 3 backslashes + apostrophe in string
|
||||
escaped = escaped.replace("'", "\\\\\\'")
|
||||
# Escape spaces
|
||||
escaped = escaped.replace(" ", "\\ ")
|
||||
return escaped
|
||||
|
||||
# Result: "I'm With You" becomes "I\\\'m\ With\ You"
|
||||
# FFmpeg correctly interprets this as: I'm With You
|
||||
```
|
||||
|
||||
**Symptoms**:
|
||||
- Video generation fails with "Could not create a libass track"
|
||||
- Error path is subtly different from the path in your code (missing apostrophe)
|
||||
- Works fine for songs without special characters
|
||||
|
||||
**Key insight**: FFmpeg filter escaping is different from shell escaping. When using `subprocess.run(cmd_list)` (no `shell=True`), there's no shell to interpret the arguments. FFmpeg receives the strings directly and applies its own filter expression parsing rules. The `'\''` shell idiom does NOT work in this context.
|
||||
|
||||
**Lesson**: When debugging FFmpeg filter path issues, check whether you're using shell or subprocess. For subprocess without shell, escape spaces with `\ ` and apostrophes with `\\\'` (3 backslashes + quote in the actual string).
|
||||
|
||||
### Duplicate Code Leads to Inconsistent Behavior
|
||||
|
||||
**Problem**: Multiple endpoints computing the same defaults independently. `audio_search.py` had `effective_brand_prefix = body.brand_prefix or settings.default_brand_prefix` but `file_upload.py` was missing it for some endpoints.
|
||||
@@ -247,6 +293,43 @@ def _get_effective_distribution_settings(...) -> EffectiveDistributionSettings:
|
||||
|
||||
**Lesson**: When you see the same logic repeated in multiple places, refactor into a helper function immediately. Otherwise bugs will be fixed in one place but not others.
|
||||
|
||||
### Python `or` Operator Can't Override True Defaults with False
|
||||
|
||||
**Problem**: Using Python's `or` operator for default logic prevents explicit `false` from overriding a `true` server default.
|
||||
|
||||
```python
|
||||
# BAD - can't explicitly disable if server default is true
|
||||
effective_value = body.enable_feature or settings.default_enable_feature
|
||||
# When body.enable_feature=False and settings.default=True:
|
||||
# False or True = True (ignores explicit False!)
|
||||
```
|
||||
|
||||
**Symptoms**:
|
||||
- Feature can be enabled but never disabled via API
|
||||
- Works fine when server default is `false` but breaks when default is `true`
|
||||
- No errors - just ignores the explicit `false` value
|
||||
|
||||
**Solution**: Use `Optional[bool] = None` with explicit `is not None` check:
|
||||
|
||||
```python
|
||||
# Request model
|
||||
enable_feature: Optional[bool] = None # None = use server default
|
||||
|
||||
# Default logic
|
||||
effective_value = (
|
||||
body.enable_feature
|
||||
if body.enable_feature is not None
|
||||
else settings.default_enable_feature
|
||||
)
|
||||
```
|
||||
|
||||
This allows three states:
|
||||
- `None` (not specified) → use server default
|
||||
- `True` → explicitly enable
|
||||
- `False` → explicitly disable (overrides server default)
|
||||
|
||||
**Lesson**: When a boolean field needs to support "use server default" behavior, use `Optional[bool] = None` instead of `bool = False`. The `or` operator conflates "not specified" with "explicitly false".
|
||||
|
||||
### Field Name Mismatches Between Endpoints
|
||||
|
||||
**Problem**: Different endpoints setting different field names for the same logical value, where consumers check only one field name.
|
||||
@@ -588,6 +671,43 @@ GitHub-hosted runners have ~14GB disk, which caused failures during Docker build
|
||||
|
||||
See `docs/archive/2025-12-28-self-hosted-github-runner.md` for full details.
|
||||
|
||||
### Pulumi CI Deployments Need Skip-Preview for Limited Service Accounts
|
||||
|
||||
**Problem**: Running `pulumi up --refresh` in CI requires the service account to have read permissions on ALL managed GCP resources. Pulumi refreshes state by querying each resource, and permission errors cause the deployment to fail before any changes are applied.
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
Error when reading or editing Resource "project" with IAM Member: Role "roles/...":
|
||||
Error retrieving IAM policy for project "...": googleapi: Error 403: The caller does not have permission
|
||||
```
|
||||
|
||||
**Root cause**: `--refresh` (default behavior) makes read API calls to verify current state matches Pulumi's state. This requires permissions like:
|
||||
- `resourcemanager.projects.getIamPolicy` (for IAM bindings)
|
||||
- `secretmanager.secrets.get` (for secrets)
|
||||
- `cloudtasks.queues.get` (for Cloud Tasks)
|
||||
- `iam.workloadIdentityPools.get` (for Workload Identity)
|
||||
- And many more for each resource type managed
|
||||
|
||||
**Solution**: Use `--skip-preview` for CI deployments where Pulumi state is authoritative:
|
||||
|
||||
```yaml
|
||||
# In CI workflow
|
||||
pulumi up --yes --skip-preview --stack org/project/stack
|
||||
```
|
||||
|
||||
**Why this is safe**:
|
||||
- Infrastructure is only modified through Pulumi (single source of truth)
|
||||
- Preview/refresh is still useful during local development with broader permissions
|
||||
- Reduces permissions surface for CI service account (principle of least privilege)
|
||||
|
||||
**Alternative** (if refresh is required): Grant the service account `roles/editor` or individual read roles for each resource type. This is less secure but allows drift detection.
|
||||
|
||||
**Lesson**: When setting up Pulumi in CI, either:
|
||||
1. Grant broad read permissions and keep `--refresh` (detects drift)
|
||||
2. Use minimal permissions and `--skip-preview` (trusts Pulumi state)
|
||||
|
||||
Choose based on your threat model. For most projects, trusting Pulumi state is acceptable since manual console changes are rare.
|
||||
|
||||
### Docker Disk Space Management on Self-Hosted Runners
|
||||
|
||||
**Problem**: Even with 200GB SSDs, self-hosted runners can fill up with Docker images. Each CI build creates new dangling images (~15GB each), and without aggressive cleanup, disk fills in days.
|
||||
@@ -667,6 +787,61 @@ RUN apt-get update && apt-get install -y \
|
||||
|
||||
**Lesson**: When building Docker images for video/subtitle rendering, install fonts explicitly. The default slim images have nothing, and font fallback behavior differs from local development machines.
|
||||
|
||||
### Reuse LLM Model Instances Across Operations
|
||||
|
||||
**Problem**: Creating a new LLM model instance for each operation in a loop causes massive overhead. Each instance creation triggers model initialization, connection establishment, and optional "warm-up" calls - adding 2+ seconds per operation.
|
||||
|
||||
**Example**: Agentic lyrics correction was creating a new `AgenticCorrector` for each gap (lyric segment needing correction):
|
||||
|
||||
```python
|
||||
# BAD - 31 gaps × 2s initialization = 62s overhead
|
||||
for gap in gaps_to_correct:
|
||||
agent = AgenticCorrector.from_model(model_id, ...) # New instance every time!
|
||||
agent.correct_gap(gap)
|
||||
```
|
||||
|
||||
**Symptoms**:
|
||||
- Operations that should take ~1 minute take 5+ minutes
|
||||
- Logs show "Initializing model..." or "Warming up..." repeatedly
|
||||
- Each loop iteration has a consistent ~2s overhead before actual work
|
||||
- Total time scales linearly with loop count, not with actual work
|
||||
|
||||
**Solution**: Create the model instance ONCE before the loop, then reuse:
|
||||
|
||||
```python
|
||||
# GOOD - 1 initialization, reused for all gaps
|
||||
agent = AgenticCorrector.from_model(model_id, ...) # Create once
|
||||
for gap in gaps_to_correct:
|
||||
agent.correct_gap(gap) # Reuse existing instance
|
||||
```
|
||||
|
||||
**Additional optimization**: If operations are independent, process them in parallel:
|
||||
|
||||
```python
|
||||
# BETTER - parallel processing with shared model
|
||||
agent = AgenticCorrector.from_model(model_id, ...)
|
||||
with ThreadPoolExecutor(max_workers=5) as executor:
|
||||
futures = {executor.submit(agent.correct_gap, gap): gap for gap in gaps}
|
||||
for future in as_completed(futures):
|
||||
result = future.result()
|
||||
```
|
||||
|
||||
**Results**:
|
||||
- Before: 31 gaps → ~5 minutes (2s init + 3s work per gap)
|
||||
- After: 20 gaps → ~55 seconds (1 init + parallel processing)
|
||||
- Speedup: ~5-6x
|
||||
|
||||
**Why warm-up is often unnecessary**: The warm-up pattern (sending a trivial request to "prime" the model) made sense for cold-start scenarios in older systems. Modern LLM APIs:
|
||||
- Don't have cold-start penalties per request
|
||||
- Handle connection pooling internally
|
||||
- May even rate-limit warm-up calls
|
||||
|
||||
If you're already making real requests, warm-up adds latency without benefit.
|
||||
|
||||
**Key insight**: LLM client instances are designed for reuse. They maintain connection pools, authentication state, and caching. Creating a new instance per request throws away these benefits.
|
||||
|
||||
**Configuration**: Added `AGENTIC_MAX_PARALLEL_GAPS` env var (default: 5) to control parallelism. Higher values increase throughput but may hit rate limits.
|
||||
|
||||
## Performance Observations
|
||||
|
||||
| Operation | Duration |
|
||||
@@ -755,6 +930,48 @@ export function getToken() {
|
||||
|
||||
**Lesson**: For values that can change (auth tokens, user preferences), read fresh from storage on each access rather than caching in module scope.
|
||||
|
||||
### Zustand Module-Level Initialization Doesn't Re-run on Client Navigation
|
||||
|
||||
**Problem**: Module-level initialization code in zustand stores only runs once when the module is first imported. Client-side navigation (e.g., `router.push()`) doesn't reload modules, so initialization code doesn't re-run.
|
||||
|
||||
**Example bug**: Auth module had initialization that fetched user data if a token existed:
|
||||
```typescript
|
||||
// auth.ts - module level (runs once at import)
|
||||
if (typeof window !== 'undefined') {
|
||||
const token = getAccessToken()
|
||||
if (token) {
|
||||
useAuth.getState().fetchUser() // Only runs on initial page load!
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
After beta enrollment:
|
||||
1. User lands on `/` (landing page) - auth.ts imported, no token yet, `fetchUser()` not called
|
||||
2. User completes enrollment, token stored via `setAccessToken()`
|
||||
3. `router.push('/app')` - client-side navigation, modules already loaded
|
||||
4. Auth module initialization doesn't re-run - `user` stays `null`
|
||||
5. UI shows "Login" button instead of user credits
|
||||
|
||||
**Symptoms**:
|
||||
- Feature works after page reload but not after client-side navigation
|
||||
- Zustand store has correct persisted data (token) but not computed data (user profile)
|
||||
- E2E tests that use reload pass; tests that use `router.push` fail
|
||||
|
||||
**Solution**: Components that need fresh data should trigger fetches in useEffect:
|
||||
```typescript
|
||||
// In /app page component
|
||||
const { user, fetchUser } = useAuth()
|
||||
|
||||
useEffect(() => {
|
||||
const token = getAccessToken()
|
||||
if (token && !user) {
|
||||
fetchUser() // Ensure user data is loaded regardless of how we got here
|
||||
}
|
||||
}, [user, fetchUser])
|
||||
```
|
||||
|
||||
**Lesson**: Don't rely on module-level initialization for data that might need to be fetched after client-side navigation. Use useEffect in components to ensure data is loaded when needed.
|
||||
|
||||
### GCP Organization Policies Block Service Account Keys
|
||||
|
||||
**Problem**: GCP organization policies can block service account key creation even when you have Owner access to the project.
|
||||
@@ -1079,6 +1296,45 @@ logs_ref.document(log_id).set({
|
||||
|
||||
**Lesson**: Never use embedded arrays for unbounded data. If an array can grow indefinitely based on operation duration or user input, use a subcollection instead. The 1MB limit applies to the entire document, including all fields.
|
||||
|
||||
### Splitting Fast Generation from Slow Encoding
|
||||
|
||||
**Problem**: Preview video generation took 60+ seconds on Cloud Run, causing poor UX during lyrics review. The bottleneck was FFmpeg encoding (CPU-bound), not the subtitle generation (fast text processing).
|
||||
|
||||
**Solution**: Split the operation into two phases with optional offloading:
|
||||
|
||||
1. **ASS Generation (Cloud Run)**: Generate ASS subtitle file locally (~2s). This is pure text processing with no CPU-intensive encoding.
|
||||
|
||||
2. **Video Encoding (GCE)**: Upload ASS to GCS and offload FFmpeg encoding to the high-performance GCE worker (~15-20s on C4-standard-8).
|
||||
|
||||
```python
|
||||
# Generate ASS only (fast, local)
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
...,
|
||||
ass_only=True, # Skip video encoding
|
||||
)
|
||||
|
||||
# Upload ASS to GCS
|
||||
storage.upload_file(result["ass_path"], f"jobs/{job_id}/previews/{preview_hash}.ass")
|
||||
|
||||
# Offload encoding to GCE
|
||||
gce_result = await encoding_service.encode_preview_video(
|
||||
job_id=preview_hash,
|
||||
ass_gcs_path=f"gs://bucket/jobs/{job_id}/previews/{preview_hash}.ass",
|
||||
audio_gcs_path=audio_gcs_path,
|
||||
output_gcs_path=f"gs://bucket/jobs/{job_id}/previews/{preview_hash}.mp4",
|
||||
)
|
||||
```
|
||||
|
||||
**Key insight**: When a pipeline has both fast (text generation) and slow (media encoding) steps, design for separability:
|
||||
- Add `_only` parameters to skip slow steps
|
||||
- Return intermediate outputs (paths to generated files)
|
||||
- Allow external services to handle the slow part
|
||||
- Keep fallback to local processing for reliability
|
||||
|
||||
**Configuration**: Controlled by `USE_GCE_PREVIEW_ENCODING` env var. Falls back to local encoding if GCE fails or is disabled.
|
||||
|
||||
**Result**: Preview generation dropped from 60+ seconds to ~15-20 seconds.
|
||||
|
||||
### External Service Response Format Mismatches
|
||||
|
||||
**Problem**: The GCE encoding worker returned responses in unexpected formats, causing `'list' object has no attribute 'get'` errors that were hard to debug through the async pipeline.
|
||||
@@ -1127,6 +1383,214 @@ if isinstance(output_files, list):
|
||||
|
||||
**Debugging tip**: When hitting format errors in async pipelines, check the raw response from the external service directly (curl with API key) before assuming the bug is in your code.
|
||||
|
||||
### GCE Encoding Returns GCS Paths, Not Local Files
|
||||
|
||||
**Problem**: When using GCE encoding (remote high-performance encoder), the orchestrator received GCS blob paths like `jobs/{id}/finals/output_4k_lossless.mp4` instead of local file paths. Subsequent operations like YouTube upload checked `os.path.isfile()` which returned `False`, causing silent failures.
|
||||
|
||||
**Symptoms**:
|
||||
- YouTube upload logged "No video file available for YouTube upload"
|
||||
- Other file-based operations silently skipped
|
||||
- Dropbox/GDrive uploads succeeded (they use different code paths)
|
||||
- No errors in logs - just warnings that were easy to miss
|
||||
|
||||
**Root cause**: The `VideoWorkerOrchestrator._run_encoding()` method stored the paths returned by `GCEEncodingBackend` directly without downloading the files. For local encoding, the paths are already local. For GCE encoding, the paths are GCS blob paths that need to be downloaded first.
|
||||
|
||||
**Solution**: Added `_download_gce_encoded_files()` method that:
|
||||
1. Detects when GCE backend was used (`encoding_backend.name == "gce"`)
|
||||
2. Downloads each encoded file from GCS to the local temp directory
|
||||
3. Updates result paths to point to local files
|
||||
|
||||
```python
|
||||
# After encoding completes
|
||||
if encoding_backend.name == "gce" and self.storage:
|
||||
await self._download_gce_encoded_files(output)
|
||||
```
|
||||
|
||||
**Key insight**: When an operation can run on multiple backends (local vs remote), ensure the outputs are normalized to a common format before subsequent stages consume them. The orchestrator pattern should abstract away backend differences, not leak them to downstream consumers.
|
||||
|
||||
**Testing tip**: Test the full pipeline with each backend type, not just the encoding step in isolation. The bug only manifested when YouTube upload ran after GCE encoding - unit tests of each stage passed.
|
||||
|
||||
### Unify Encoding Logic with GCS Wheel Deployment
|
||||
|
||||
**Problem**: The GCE encoding worker had simplified, duplicated encoding logic that:
|
||||
- Didn't concatenate title + karaoke + end screens
|
||||
- Used generic filenames like `output_4k_lossless.mp4` instead of proper names
|
||||
- Lacked feature parity with the local CLI's `LocalEncodingService`
|
||||
|
||||
This caused cloud-generated videos to be incomplete (no title/end screens) and have unhelpful filenames that didn't match the local CLI output.
|
||||
|
||||
**Solution**: Deploy the karaoke-gen wheel to GCS, have the GCE worker install and use `LocalEncodingService`:
|
||||
|
||||
1. **CI uploads wheel to GCS**: `deploy-backend` job uploads wheel to `gs://bucket/wheels/`
|
||||
|
||||
2. **Dynamic wheel loading**: GCE worker downloads and installs latest wheel at job start:
|
||||
```python
|
||||
def ensure_latest_wheel():
|
||||
"""Download and install latest karaoke-gen wheel from GCS."""
|
||||
subprocess.run(["gsutil", "cp", "gs://bucket/wheels/karaoke_gen-*.whl", "/tmp/"])
|
||||
wheel = sorted(glob.glob("/tmp/karaoke_gen-*.whl"))[-1]
|
||||
subprocess.run([sys.executable, "-m", "pip", "install", "--upgrade", wheel])
|
||||
|
||||
async def process_job(job_id, request):
|
||||
ensure_latest_wheel() # Get latest code at job start
|
||||
# ... rest of processing
|
||||
```
|
||||
|
||||
3. **Single encoding path**: GCE worker imports and uses `LocalEncodingService`:
|
||||
```python
|
||||
from backend.services.local_encoding_service import LocalEncodingService, EncodingConfig
|
||||
|
||||
service = LocalEncodingService(logger=logger)
|
||||
config = EncodingConfig(
|
||||
title_video=str(title_video),
|
||||
karaoke_video=str(karaoke_video),
|
||||
output_lossless_4k_mp4=f"{base_name} (Final Karaoke Lossless 4k).mp4",
|
||||
# ... proper naming for all formats
|
||||
)
|
||||
result = service.encode_all_formats(config)
|
||||
```
|
||||
|
||||
**Key benefits**:
|
||||
1. **Single source of truth**: Same encoding logic used by CLI, Cloud Run, and GCE
|
||||
2. **Hot updates without restart**: New code deployed on next job, not VM restart
|
||||
3. **In-progress jobs unaffected**: Jobs use the code version they started with
|
||||
4. **Proper output**: Titles/end screens concatenated, files named correctly
|
||||
|
||||
**No fallback by design**: We removed all fallback logic to ensure there's only ONE code path. If the wheel fails to load, the job fails clearly rather than producing inconsistent output.
|
||||
|
||||
**Lesson**: When multiple deployment targets (CLI, Cloud Run, GCE) need the same complex logic, package it as a library and deploy via a shared mechanism (GCS wheel). Duplicating logic leads to feature drift and inconsistent behavior.
|
||||
|
||||
### Unify Preview Encoding with LocalPreviewEncodingService
|
||||
|
||||
**Problem**: Following the pattern of final encoding, the GCE preview encoding also had divergent code. The `run_preview_encoding` function in `infrastructure/__main__.py` was a 100-line "simplified" reimplementation that:
|
||||
- Hardcoded resolution (480x270) instead of using constants
|
||||
- Had its own FFmpeg filter path escaping function with different logic
|
||||
- Didn't support hardware acceleration (NVENC)
|
||||
- Lacked font directory support for custom fonts
|
||||
- Had different movflags and encoding settings
|
||||
|
||||
While the final encoding was unified earlier, preview encoding was overlooked, creating a potential source of inconsistent preview videos between local development and cloud jobs.
|
||||
|
||||
**Solution**: Created `LocalPreviewEncodingService` in `backend/services/local_preview_encoding_service.py`:
|
||||
|
||||
1. **Extracted preview logic from VideoGenerator**: The service encapsulates the same FFmpeg command building from `VideoGenerator._build_preview_ffmpeg_command`
|
||||
|
||||
2. **Single source of truth for preview settings**:
|
||||
```python
|
||||
class LocalPreviewEncodingService:
|
||||
PREVIEW_WIDTH = 480
|
||||
PREVIEW_HEIGHT = 270
|
||||
PREVIEW_FPS = 24
|
||||
PREVIEW_AUDIO_BITRATE = "96k"
|
||||
```
|
||||
|
||||
3. **GCE worker imports from installed wheel**:
|
||||
```python
|
||||
from backend.services.local_preview_encoding_service import (
|
||||
LocalPreviewEncodingService,
|
||||
PreviewEncodingConfig,
|
||||
)
|
||||
|
||||
service = LocalPreviewEncodingService(logger=logger)
|
||||
result = service.encode_preview(config)
|
||||
```
|
||||
|
||||
4. **Process preview jobs call ensure_latest_wheel()**:
|
||||
```python
|
||||
async def process_preview_job(job_id, request):
|
||||
ensure_latest_wheel() # Get latest code at job start
|
||||
# ... rest of processing
|
||||
```
|
||||
|
||||
**Benefits**:
|
||||
- Consistent preview encoding across CLI, Cloud Run, and GCE
|
||||
- Hardware acceleration (NVENC) when available
|
||||
- Proper FFmpeg filter escaping for special characters in paths
|
||||
- Custom font support via fontsdir filter
|
||||
- Single place to modify preview settings
|
||||
|
||||
**Lesson**: After unifying a major component (final encoding), audit related features (preview encoding) that may have similar duplication. It's easy to overlook "simpler" variations when focusing on the main feature.
|
||||
|
||||
### Cloud Distribution Must Match Local CLI Output Structure
|
||||
|
||||
**Problem**: Cloud-generated karaoke outputs had different file organization than local CLI, causing confusion and missing files in Dropbox uploads:
|
||||
|
||||
1. **lyrics/ folder filename differences**:
|
||||
- Cloud used generic `(Karaoke).json` instead of `(Lyrics Corrections).json`
|
||||
- Cloud used `(Karaoke).txt` instead of `(Lyrics Corrected).txt`
|
||||
- Reference lyrics files (`(Lyrics Genius).txt`, etc.) weren't downloaded
|
||||
|
||||
2. **Missing files in lyrics/ folder**:
|
||||
- `(With Vocals).mkv` wasn't copied to lyrics/ subfolder
|
||||
- `previews/` subfolder with preview ASS files wasn't created
|
||||
- Reference lyrics from multiple sources weren't included
|
||||
|
||||
**Root causes**:
|
||||
- Lyrics worker uploaded reference files to GCS but didn't track them in `job.files`
|
||||
- Video worker used generic naming pattern for all lyrics files
|
||||
- Video worker didn't copy `(With Vocals).mkv` or previews to lyrics folder
|
||||
|
||||
**Solution**:
|
||||
|
||||
1. **Track all files in job.files** (lyrics_worker.py):
|
||||
```python
|
||||
# Upload ALL reference lyrics and track them for distribution
|
||||
for ref_filename, source_key in reference_files:
|
||||
if os.path.exists(ref_path):
|
||||
url = storage.upload_file(ref_path, gcs_path)
|
||||
# Track in job.files so video_worker can download
|
||||
job_manager.update_file_url(job_id, 'lyrics', f'reference_{source_key}', url)
|
||||
```
|
||||
|
||||
2. **Use proper filenames in distribution** (video_worker.py):
|
||||
```python
|
||||
lyrics_file_mappings = [
|
||||
('lrc', f"{base_name} (Karaoke).lrc"),
|
||||
('corrections', f"{base_name} (Lyrics Corrections).json"),
|
||||
('corrected_txt', f"{base_name} (Lyrics Corrected).txt"),
|
||||
('uncorrected', f"{base_name} (Lyrics Uncorrected).txt"),
|
||||
('reference_genius', f"{base_name} (Lyrics Genius).txt"),
|
||||
# ... etc
|
||||
]
|
||||
```
|
||||
|
||||
3. **Copy (With Vocals).mkv to lyrics folder**:
|
||||
```python
|
||||
if videos.get('with_vocals'):
|
||||
dest = os.path.join(lyrics_dir, f"{base_name} (With Vocals).mkv")
|
||||
storage.download_file(videos['with_vocals'], dest)
|
||||
```
|
||||
|
||||
4. **Create previews/ subfolder**:
|
||||
```python
|
||||
previews_dir = os.path.join(lyrics_dir, "previews")
|
||||
os.makedirs(previews_dir, exist_ok=True)
|
||||
preview_files = storage.list_files(f"jobs/{job_id}/previews/")
|
||||
for blob_name in preview_files:
|
||||
if blob_name.endswith('.ass'):
|
||||
storage.download_file(blob_name, dest_path)
|
||||
```
|
||||
|
||||
**Expected output structure** (matches local CLI):
|
||||
```
|
||||
NOMAD-XXXX - Artist - Title/
|
||||
├── stems/
|
||||
│ ├── Artist - Title (Vocals model_bs_roformer_...).flac
|
||||
│ └── ...
|
||||
└── lyrics/
|
||||
├── Artist - Title (Karaoke).lrc
|
||||
├── Artist - Title (Karaoke).ass
|
||||
├── Artist - Title (Lyrics Corrections).json
|
||||
├── Artist - Title (Lyrics Corrected).txt
|
||||
├── Artist - Title (Lyrics Uncorrected).txt
|
||||
├── Artist - Title (Lyrics Genius).txt
|
||||
├── Artist - Title (With Vocals).mkv
|
||||
└── previews/
|
||||
└── preview_abc123 (Karaoke).ass
|
||||
```
|
||||
|
||||
**Lesson**: When building distribution packages from cloud storage, the organization code must replicate the exact structure users expect from local execution. Track ALL intermediate files in `job.files` even if they're "optional" - users may rely on them.
|
||||
|
||||
## What We'd Do Differently
|
||||
|
||||
1. **Add Pydantic model fields test first** - Would have caught the silent field issue immediately
|
||||
|
||||
@@ -36,6 +36,22 @@
|
||||
|
||||
## Recent Changes
|
||||
|
||||
- **Audio Search Display Override** (2026-01-06): Added optional "Display As" fields to audio search, allowing users to search for audio by one artist/title (e.g., "Jeremy Kushnier") but display a different artist/title on title screens and filenames (e.g., "Footloose (Broadway Cast)"). Useful for Broadway casts, covers, remixes where tracker metadata differs from desired display. See [API.md](API.md#audio-search).
|
||||
|
||||
- **Infrastructure Modularization** (2026-01-06): Refactored `infrastructure/__main__.py` from 2,602 lines to 339 lines (87% reduction). Split into organized modules: `modules/` for core GCP resources, `compute/` for VMs and startup scripts, `config.py` for shared constants. Extracted embedded encoding worker Python app to `backend/services/gce_encoding/`. See `infrastructure/README.md` for new structure and `infrastructure/docs/PHASE5-PACKER-IMAGE-PLAN.md` for planned Packer image optimization.
|
||||
|
||||
- **Email Notification System** (2026-01-06): Added automated email notifications for job completion and user action reminders. Features: GCS-backed HTML email templates with fallback defaults, SendGrid with CC support, auto-completion emails on job finish, idle reminder emails via Cloud Tasks (5-min delay for blocking states), admin UI buttons to copy message or send email manually. Endpoints: `GET /api/admin/jobs/{id}/completion-message`, `POST /api/admin/jobs/{id}/send-completion-email`. Feature flag `ENABLE_AUTO_EMAILS` (default: false). See [API.md](API.md#email-notifications-admin) and [ARCHITECTURE.md](ARCHITECTURE.md#video-worker-orchestrator).
|
||||
|
||||
- **GCE Preview Encoding Unified with LocalPreviewEncodingService** (2026-01-06): GCE preview encoding now uses the same `LocalPreviewEncodingService` from the installed wheel, eliminating 100 lines of duplicated FFmpeg logic. Preview videos are now identical across local CLI, Cloud Run, and GCE worker environments. Benefits include: consistent encoding settings (480x270, 24fps, crf 28), proper FFmpeg filter path escaping for special characters, hardware acceleration (NVENC) when available, and custom font support. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#unify-preview-encoding-with-localpreviewencodingservice).
|
||||
|
||||
- **GCE Encoding Worker Python 3.13** (2026-01-06): Upgraded GCE encoding worker from Debian's Python 3.11 to Python 3.13 built from source. Uses dedicated virtual environment at `/opt/encoding-worker/venv`, removing the need for `--break-system-packages`. Aligns the encoding worker with CI/Cloud Run Python version.
|
||||
|
||||
- **GCE Encoding Unified with LocalEncodingService** (2026-01-06): GCE encoding worker now uses the same `LocalEncodingService` as the local CLI, eliminating duplicated encoding logic. Output files now have proper names (`Artist - Title (Final Karaoke Lossless 4k).mp4` instead of `output_4k_lossless.mp4`) and include title/end screen concatenation. Wheel deployed to GCS; worker installs at job start for hot updates without VM restart. Removed all fallback logic - single code path for consistent output across CLI, Cloud Run, and GCE. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#unify-encoding-logic-with-gcs-wheel-deployment).
|
||||
|
||||
- **Cloud Distribution Matches Local CLI** (2026-01-06): Fixed cloud distribution (Dropbox uploads) to match local CLI output structure. The `lyrics/` folder now includes: proper filenames (`(Lyrics Corrections).json` instead of `(Karaoke).json`), all reference lyrics files (`(Lyrics Genius).txt`, `(Lyrics Spotify).txt`, etc.), `(With Vocals).mkv`, and `previews/` subfolder with preview ASS files. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#cloud-distribution-must-match-local-cli-output-structure).
|
||||
|
||||
- **Agentic Correction Performance** (2026-01-05): Optimized agentic AI correction from ~5 minutes to ~55 seconds for 20 gaps (~5-6x speedup). Fixed anti-pattern of creating new model instance per gap (caused repeated 2s warm-up overhead). Now creates model once and processes gaps in parallel using ThreadPoolExecutor (default 5 workers). Configure via `AGENTIC_MAX_PARALLEL_GAPS` env var. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#reuse-llm-model-instances-across-operations).
|
||||
|
||||
- **Worker Logs Subcollection** (2026-01-04): Moved `worker_logs` from embedded array in job documents to Firestore subcollection (`jobs/{job_id}/logs`). Fixes job failures when logs exceed 1MB (job 501258e1 had 1.26MB of logs). New logs stored with 30-day TTL via Firestore TTL policy. Feature flag `USE_LOG_SUBCOLLECTION=true` (default). See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#firestore-document-1mb-limit-with-embedded-arrays).
|
||||
|
||||
- **GCE Encoding Response Fixes** (2026-01-04): Fixed multiple response format mismatches with GCE encoding worker. The worker returns `output_files` as a list of paths, not a dict with format keys - added conversion logic. Also added defensive type checking for status responses that could be lists. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#external-service-response-format-mismatches).
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
import { useState, useEffect, useCallback } from "react"
|
||||
import { useRouter } from "next/navigation"
|
||||
import { api, Job, getAccessToken } from "@/lib/api"
|
||||
import { useAuth } from "@/lib/auth"
|
||||
import { useJobNotifications, useVisibilityRefresh } from "@/hooks/use-notifications"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
|
||||
@@ -28,6 +29,7 @@ export default function AppPage() {
|
||||
const [isInitialLoad, setIsInitialLoad] = useState(true)
|
||||
const [isAuthenticated, setIsAuthenticated] = useState<boolean | null>(null)
|
||||
const { isDarkMode, toggleTheme, mounted } = useTheme()
|
||||
const { user, fetchUser } = useAuth()
|
||||
|
||||
// Memoize loadJobs for use with visibility refresh
|
||||
const loadJobs = useCallback(async () => {
|
||||
@@ -67,6 +69,16 @@ export default function AppPage() {
|
||||
setIsAuthenticated(true)
|
||||
}, [router])
|
||||
|
||||
// Ensure user data is loaded after authentication
|
||||
// This handles the case where user navigates via client-side navigation
|
||||
// (e.g., after beta enrollment) and the module-level fetchUser() didn't run
|
||||
useEffect(() => {
|
||||
const token = getAccessToken()
|
||||
if (token && !user) {
|
||||
fetchUser()
|
||||
}
|
||||
}, [user, fetchUser])
|
||||
|
||||
// Load jobs on mount (only if authenticated)
|
||||
useEffect(() => {
|
||||
if (isAuthenticated !== true) {
|
||||
|
||||
@@ -34,6 +34,9 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
|
||||
// Audio search form
|
||||
const [searchArtist, setSearchArtist] = useState("")
|
||||
const [searchTitle, setSearchTitle] = useState("")
|
||||
// Display As overrides (optional - if empty, search values used for display)
|
||||
const [displayArtist, setDisplayArtist] = useState("")
|
||||
const [displayTitle, setDisplayTitle] = useState("")
|
||||
|
||||
// Theme selection (shared across all tabs)
|
||||
const [selectedTheme, setSelectedTheme] = useState<string | undefined>()
|
||||
@@ -128,9 +131,14 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
|
||||
theme_id: selectedTheme,
|
||||
color_overrides: cleanColorOverrides(colorOverrides),
|
||||
non_interactive: nonInteractive,
|
||||
// Display overrides (empty string means use search values)
|
||||
display_artist: displayArtist.trim() || undefined,
|
||||
display_title: displayTitle.trim() || undefined,
|
||||
})
|
||||
setSearchArtist("")
|
||||
setSearchTitle("")
|
||||
setDisplayArtist("")
|
||||
setDisplayTitle("")
|
||||
onJobCreated()
|
||||
} catch (err) {
|
||||
if (err instanceof ApiError) {
|
||||
@@ -407,33 +415,69 @@ export function JobSubmission({ onJobCreated }: JobSubmissionProps) {
|
||||
|
||||
<TabsContent value="search" className="mt-4">
|
||||
<form onSubmit={handleSearchSubmit} className="space-y-4">
|
||||
<div className="grid grid-cols-1 sm:grid-cols-2 gap-3 sm:gap-4">
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="search-artist" style={{ color: 'var(--text)' }}>Artist</Label>
|
||||
<Input
|
||||
id="search-artist"
|
||||
placeholder="Artist name"
|
||||
value={searchArtist}
|
||||
onChange={(e) => setSearchArtist(e.target.value)}
|
||||
disabled={isSubmitting}
|
||||
style={{ backgroundColor: 'var(--secondary)', borderColor: 'var(--card-border)', color: 'var(--text)' }}
|
||||
/>
|
||||
{/* Search For section */}
|
||||
<div className="space-y-2">
|
||||
<Label style={{ color: 'var(--text)' }}>Search For</Label>
|
||||
<div className="grid grid-cols-1 sm:grid-cols-2 gap-3 sm:gap-4">
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="search-artist" className="text-xs" style={{ color: 'var(--text-muted)' }}>Artist</Label>
|
||||
<Input
|
||||
id="search-artist"
|
||||
placeholder="Artist name on trackers"
|
||||
value={searchArtist}
|
||||
onChange={(e) => setSearchArtist(e.target.value)}
|
||||
disabled={isSubmitting}
|
||||
style={{ backgroundColor: 'var(--secondary)', borderColor: 'var(--card-border)', color: 'var(--text)' }}
|
||||
/>
|
||||
</div>
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="search-title" className="text-xs" style={{ color: 'var(--text-muted)' }}>Title</Label>
|
||||
<Input
|
||||
id="search-title"
|
||||
placeholder="Song title on trackers"
|
||||
value={searchTitle}
|
||||
onChange={(e) => setSearchTitle(e.target.value)}
|
||||
disabled={isSubmitting}
|
||||
style={{ backgroundColor: 'var(--secondary)', borderColor: 'var(--card-border)', color: 'var(--text)' }}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="search-title" style={{ color: 'var(--text)' }}>Title</Label>
|
||||
<Input
|
||||
id="search-title"
|
||||
placeholder="Song title"
|
||||
value={searchTitle}
|
||||
onChange={(e) => setSearchTitle(e.target.value)}
|
||||
disabled={isSubmitting}
|
||||
style={{ backgroundColor: 'var(--secondary)', borderColor: 'var(--card-border)', color: 'var(--text)' }}
|
||||
/>
|
||||
</div>
|
||||
|
||||
{/* Display As section (optional) */}
|
||||
<div className="space-y-2">
|
||||
<div className="flex items-center gap-2">
|
||||
<Label style={{ color: 'var(--text)' }}>Display As</Label>
|
||||
<span className="text-xs px-1.5 py-0.5 rounded" style={{ color: 'var(--text-muted)', backgroundColor: 'var(--secondary)' }}>optional</span>
|
||||
</div>
|
||||
<p className="text-xs" style={{ color: 'var(--text-muted)' }}>
|
||||
Override how artist/title appear on the title card and filename. Leave empty to use search values.
|
||||
</p>
|
||||
<div className="grid grid-cols-1 sm:grid-cols-2 gap-3 sm:gap-4">
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="display-artist" className="text-xs" style={{ color: 'var(--text-muted)' }}>Display Artist</Label>
|
||||
<Input
|
||||
id="display-artist"
|
||||
placeholder="e.g., Footloose (Broadway Cast)"
|
||||
value={displayArtist}
|
||||
onChange={(e) => setDisplayArtist(e.target.value)}
|
||||
disabled={isSubmitting}
|
||||
style={{ backgroundColor: 'var(--secondary)', borderColor: 'var(--card-border)', color: 'var(--text)' }}
|
||||
/>
|
||||
</div>
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="display-title" className="text-xs" style={{ color: 'var(--text-muted)' }}>Display Title</Label>
|
||||
<Input
|
||||
id="display-title"
|
||||
placeholder="e.g., I Can't Stand Still"
|
||||
value={displayTitle}
|
||||
onChange={(e) => setDisplayTitle(e.target.value)}
|
||||
disabled={isSubmitting}
|
||||
style={{ backgroundColor: 'var(--secondary)', borderColor: 'var(--card-border)', color: 'var(--text)' }}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p className="text-xs" style={{ color: 'var(--text-muted)' }}>
|
||||
<span className="text-amber-500 font-medium">Note:</span> Format these exactly as you want them on the title card and video filename.
|
||||
</p>
|
||||
|
||||
{/* Theme Selection */}
|
||||
<div className="space-y-4 pt-4 border-t" style={{ borderColor: 'var(--card-border)' }}>
|
||||
|
||||
@@ -1,9 +1,20 @@
|
||||
"use client"
|
||||
|
||||
import { useEffect, useState, useRef, useCallback } from "react"
|
||||
import { api } from "@/lib/api"
|
||||
import { api, adminApi } from "@/lib/api"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Download, Loader2, ExternalLink, FolderOpen, Copy } from "lucide-react"
|
||||
import { Download, Loader2, ExternalLink, FolderOpen, Copy, Mail, MessageSquare } from "lucide-react"
|
||||
import { useAuth } from "@/lib/auth"
|
||||
import {
|
||||
Dialog,
|
||||
DialogContent,
|
||||
DialogDescription,
|
||||
DialogFooter,
|
||||
DialogHeader,
|
||||
DialogTitle,
|
||||
} from "@/components/ui/dialog"
|
||||
import { Input } from "@/components/ui/input"
|
||||
import { Label } from "@/components/ui/label"
|
||||
|
||||
interface OutputLinksProps {
|
||||
jobId: string
|
||||
@@ -15,18 +26,35 @@ export function OutputLinks({ jobId }: OutputLinksProps) {
|
||||
const [dropboxUrl, setDropboxUrl] = useState<string | null>(null)
|
||||
const [isLoading, setIsLoading] = useState(false)
|
||||
const [copiedUrl, setCopiedUrl] = useState<string | null>(null)
|
||||
const [copiedMessage, setCopiedMessage] = useState(false)
|
||||
const [isLoadingMessage, setIsLoadingMessage] = useState(false)
|
||||
const [showEmailDialog, setShowEmailDialog] = useState(false)
|
||||
const [emailInput, setEmailInput] = useState("")
|
||||
const [isSendingEmail, setIsSendingEmail] = useState(false)
|
||||
const [emailSent, setEmailSent] = useState(false)
|
||||
const copyTimeoutRef = useRef<NodeJS.Timeout | null>(null)
|
||||
const messageTimeoutRef = useRef<NodeJS.Timeout | null>(null)
|
||||
const emailTimeoutRef = useRef<NodeJS.Timeout | null>(null)
|
||||
|
||||
const { user } = useAuth()
|
||||
const isAdmin = user?.role === 'admin'
|
||||
|
||||
useEffect(() => {
|
||||
loadOutputLinks()
|
||||
}, [jobId])
|
||||
|
||||
// Cleanup timeout on unmount
|
||||
// Cleanup timeouts on unmount
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
if (copyTimeoutRef.current) {
|
||||
clearTimeout(copyTimeoutRef.current)
|
||||
}
|
||||
if (messageTimeoutRef.current) {
|
||||
clearTimeout(messageTimeoutRef.current)
|
||||
}
|
||||
if (emailTimeoutRef.current) {
|
||||
clearTimeout(emailTimeoutRef.current)
|
||||
}
|
||||
}
|
||||
}, [])
|
||||
|
||||
@@ -76,6 +104,60 @@ export function OutputLinks({ jobId }: OutputLinksProps) {
|
||||
}
|
||||
}, [])
|
||||
|
||||
const copyCompletionMessage = useCallback(async () => {
|
||||
if (!navigator?.clipboard?.writeText) {
|
||||
console.error("Clipboard API not available")
|
||||
return
|
||||
}
|
||||
|
||||
setIsLoadingMessage(true)
|
||||
try {
|
||||
const response = await adminApi.getCompletionMessage(jobId)
|
||||
await navigator.clipboard.writeText(response.message)
|
||||
setCopiedMessage(true)
|
||||
|
||||
// Clear any existing timeout
|
||||
if (messageTimeoutRef.current) {
|
||||
clearTimeout(messageTimeoutRef.current)
|
||||
}
|
||||
|
||||
// Set new timeout
|
||||
messageTimeoutRef.current = setTimeout(() => {
|
||||
setCopiedMessage(false)
|
||||
}, 2000)
|
||||
} catch (err) {
|
||||
console.error("Failed to copy completion message:", err)
|
||||
} finally {
|
||||
setIsLoadingMessage(false)
|
||||
}
|
||||
}, [jobId])
|
||||
|
||||
const handleSendEmail = useCallback(async () => {
|
||||
if (!emailInput.trim()) return
|
||||
|
||||
setIsSendingEmail(true)
|
||||
try {
|
||||
await adminApi.sendCompletionEmail(jobId, emailInput.trim(), true)
|
||||
setEmailSent(true)
|
||||
|
||||
// Clear any existing timeout
|
||||
if (emailTimeoutRef.current) {
|
||||
clearTimeout(emailTimeoutRef.current)
|
||||
}
|
||||
// Set new timeout to close dialog after success message
|
||||
emailTimeoutRef.current = setTimeout(() => {
|
||||
setShowEmailDialog(false)
|
||||
setEmailSent(false)
|
||||
setEmailInput("")
|
||||
}, 1500)
|
||||
} catch (err) {
|
||||
console.error("Failed to send email:", err)
|
||||
alert("Failed to send email. Please check the email address and try again.")
|
||||
} finally {
|
||||
setIsSendingEmail(false)
|
||||
}
|
||||
}, [jobId, emailInput])
|
||||
|
||||
if (isLoading) {
|
||||
return (
|
||||
<div className="flex items-center gap-2 text-xs" style={{ color: 'var(--text-muted)' }}>
|
||||
@@ -193,12 +275,99 @@ export function OutputLinks({ jobId }: OutputLinksProps) {
|
||||
</a>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* Admin-only buttons */}
|
||||
{isAdmin && (
|
||||
<div className="flex flex-wrap gap-2 pt-1 border-t border-gray-700/50 mt-2">
|
||||
<button
|
||||
type="button"
|
||||
onClick={(e) => { e.stopPropagation(); copyCompletionMessage() }}
|
||||
disabled={isLoadingMessage}
|
||||
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-amber-600 hover:bg-amber-500 text-white disabled:opacity-50"
|
||||
title="Copy completion message to clipboard"
|
||||
>
|
||||
{isLoadingMessage ? (
|
||||
<Loader2 className="w-3 h-3 animate-spin" />
|
||||
) : (
|
||||
<MessageSquare className="w-3 h-3" />
|
||||
)}
|
||||
{copiedMessage ? "Copied!" : "Copy Message"}
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
onClick={(e) => { e.stopPropagation(); setShowEmailDialog(true) }}
|
||||
className="inline-flex items-center gap-1 text-xs px-3 py-1.5 rounded bg-teal-600 hover:bg-teal-500 text-white"
|
||||
title="Send completion email"
|
||||
>
|
||||
<Mail className="w-3 h-3" />
|
||||
Send Email
|
||||
</button>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{!hasOutputs && (
|
||||
<p className="text-xs" style={{ color: 'var(--text-muted)' }}>No outputs available yet</p>
|
||||
)}
|
||||
|
||||
{/* Send Email Dialog */}
|
||||
<Dialog open={showEmailDialog} onOpenChange={setShowEmailDialog}>
|
||||
<DialogContent className="sm:max-w-[425px]" onClick={(e) => e.stopPropagation()}>
|
||||
<DialogHeader>
|
||||
<DialogTitle>Send Completion Email</DialogTitle>
|
||||
<DialogDescription>
|
||||
Send the job completion message to an email address. The email will be CC'd to gen@nomadkaraoke.com.
|
||||
</DialogDescription>
|
||||
</DialogHeader>
|
||||
<div className="grid gap-4 py-4">
|
||||
<div className="grid grid-cols-4 items-center gap-4">
|
||||
<Label htmlFor="email" className="text-right">
|
||||
Email
|
||||
</Label>
|
||||
<Input
|
||||
id="email"
|
||||
type="email"
|
||||
value={emailInput}
|
||||
onChange={(e) => setEmailInput(e.target.value)}
|
||||
placeholder="customer@example.com"
|
||||
className="col-span-3"
|
||||
onKeyDown={(e) => {
|
||||
if (e.key === 'Enter') {
|
||||
handleSendEmail()
|
||||
}
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
<DialogFooter>
|
||||
<Button
|
||||
type="button"
|
||||
variant="outline"
|
||||
onClick={() => setShowEmailDialog(false)}
|
||||
disabled={isSendingEmail}
|
||||
>
|
||||
Cancel
|
||||
</Button>
|
||||
<Button
|
||||
type="button"
|
||||
onClick={handleSendEmail}
|
||||
disabled={isSendingEmail || !emailInput.trim() || emailSent}
|
||||
>
|
||||
{isSendingEmail ? (
|
||||
<>
|
||||
<Loader2 className="w-4 h-4 mr-2 animate-spin" />
|
||||
Sending...
|
||||
</>
|
||||
) : emailSent ? (
|
||||
"Sent!"
|
||||
) : (
|
||||
"Send Email"
|
||||
)}
|
||||
</Button>
|
||||
</DialogFooter>
|
||||
</DialogContent>
|
||||
</Dialog>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -1,10 +1,11 @@
|
||||
"use client";
|
||||
|
||||
import { useEffect, useState } from "react";
|
||||
import { api } from "@/lib/api";
|
||||
import { api, EncodingWorkerHealth } from "@/lib/api";
|
||||
|
||||
export function VersionFooter() {
|
||||
const [backendVersion, setBackendVersion] = useState<string | null>(null);
|
||||
const [encodingWorker, setEncodingWorker] = useState<EncodingWorkerHealth | null>(null);
|
||||
// Frontend version is injected at build time from pyproject.toml
|
||||
const frontendVersion = process.env.NEXT_PUBLIC_APP_VERSION || "dev";
|
||||
|
||||
@@ -19,9 +20,56 @@ export function VersionFooter() {
|
||||
}
|
||||
};
|
||||
|
||||
const fetchEncodingWorkerHealth = async () => {
|
||||
try {
|
||||
const health = await api.getEncodingWorkerHealth();
|
||||
setEncodingWorker(health);
|
||||
} catch (error) {
|
||||
console.error("Failed to fetch encoding worker health:", error);
|
||||
setEncodingWorker(null);
|
||||
}
|
||||
};
|
||||
|
||||
fetchBackendVersion();
|
||||
fetchEncodingWorkerHealth();
|
||||
}, []);
|
||||
|
||||
// Render encoding worker status indicator
|
||||
const renderEncodingWorkerStatus = () => {
|
||||
if (!encodingWorker) {
|
||||
return null;
|
||||
}
|
||||
|
||||
if (encodingWorker.status === "not_configured") {
|
||||
return null; // Don't show if not configured
|
||||
}
|
||||
|
||||
const isHealthy = encodingWorker.available && encodingWorker.status === "ok";
|
||||
const statusColor = isHealthy ? "text-green-500" : "text-red-500";
|
||||
const statusDot = isHealthy ? "●" : "○";
|
||||
|
||||
// Build status text
|
||||
let statusText = isHealthy ? "Encoder" : "Encoder offline";
|
||||
if (isHealthy && encodingWorker.version) {
|
||||
statusText = `Encoder: v${encodingWorker.version}`;
|
||||
}
|
||||
|
||||
// Add active jobs indicator if there are any
|
||||
if (isHealthy && encodingWorker.active_jobs && encodingWorker.active_jobs > 0) {
|
||||
statusText += ` (${encodingWorker.active_jobs} active)`;
|
||||
}
|
||||
|
||||
return (
|
||||
<>
|
||||
{" | "}
|
||||
<span className={statusColor} title={isHealthy ? "Encoding worker healthy" : encodingWorker.error || "Encoding worker offline"}>
|
||||
{statusDot}
|
||||
</span>{" "}
|
||||
{statusText}
|
||||
</>
|
||||
);
|
||||
};
|
||||
|
||||
return (
|
||||
<footer className="border-t mt-12 py-6" style={{ borderColor: 'var(--card-border)' }}>
|
||||
<div className="px-4 text-center text-sm" style={{ color: 'var(--text-muted)' }}>
|
||||
@@ -39,6 +87,7 @@ export function VersionFooter() {
|
||||
Backend: v{backendVersion}
|
||||
</>
|
||||
)}
|
||||
{renderEncodingWorkerStatus()}
|
||||
</p>
|
||||
</div>
|
||||
</footer>
|
||||
|
||||
@@ -43,8 +43,10 @@ export interface EmailHelper {
|
||||
isAvailable: boolean;
|
||||
createInbox: () => Promise<InboxDto>;
|
||||
waitForEmail: (inboxId: string, timeout?: number) => Promise<Email>;
|
||||
waitForCompletionEmail: (inboxId: string, artist?: string, title?: string, timeout?: number) => Promise<Email>;
|
||||
extractMagicLink: (email: Email) => string | null;
|
||||
extractVerificationCode: (email: Email) => string | null;
|
||||
isCompletionEmail: (email: Email) => boolean;
|
||||
deleteInbox: (inboxId: string) => Promise<void>;
|
||||
}
|
||||
|
||||
@@ -78,6 +80,37 @@ function generateUniqueTag(): string {
|
||||
return `test-${timestamp}-${random}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if an email is a job completion email.
|
||||
* Extracted to standalone function for TypeScript compatibility.
|
||||
*/
|
||||
function checkIsCompletionEmail(email: Email): boolean {
|
||||
const subject = (email.subject || '').toLowerCase();
|
||||
const body = (email.body || '').toLowerCase();
|
||||
|
||||
// Check subject for completion indicators
|
||||
const subjectIndicators = [
|
||||
'karaoke video is ready',
|
||||
'your video is ready',
|
||||
'video ready',
|
||||
'complete',
|
||||
];
|
||||
|
||||
// Check body for completion indicators
|
||||
const bodyIndicators = [
|
||||
'your karaoke video is ready',
|
||||
'video has been created',
|
||||
'youtube',
|
||||
'dropbox',
|
||||
'download your video',
|
||||
];
|
||||
|
||||
const hasSubjectMatch = subjectIndicators.some(indicator => subject.includes(indicator));
|
||||
const hasBodyMatch = bodyIndicators.some(indicator => body.includes(indicator));
|
||||
|
||||
return hasSubjectMatch || hasBodyMatch;
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates an email testing helper.
|
||||
* Returns a helper with isAvailable=false if testmail.app env vars are not set.
|
||||
@@ -234,6 +267,110 @@ export async function createEmailHelper(): Promise<EmailHelper> {
|
||||
return null;
|
||||
},
|
||||
|
||||
/**
|
||||
* Check if an email is a job completion email
|
||||
*/
|
||||
isCompletionEmail(email: Email): boolean {
|
||||
return checkIsCompletionEmail(email);
|
||||
},
|
||||
|
||||
/**
|
||||
* Wait for a job completion email to arrive
|
||||
* @param inboxId - The tag to check (returned from createInbox as id)
|
||||
* @param artist - Optional artist name to verify in email
|
||||
* @param title - Optional song title to verify in email
|
||||
* @param timeout - Timeout in ms (default: 120000 = 2 minutes)
|
||||
*/
|
||||
async waitForCompletionEmail(
|
||||
inboxId: string,
|
||||
artist?: string,
|
||||
title?: string,
|
||||
timeout = 120000
|
||||
): Promise<Email> {
|
||||
console.log(`Waiting for completion email with tag ${inboxId}...`);
|
||||
|
||||
const pollInterval = Math.min(timeout, 30000);
|
||||
const startTime = Date.now();
|
||||
|
||||
// Track emails we've already seen (by timestamp)
|
||||
const seenTimestamps = new Set<number>();
|
||||
|
||||
while (Date.now() - startTime < timeout) {
|
||||
const remainingTime = timeout - (Date.now() - startTime);
|
||||
const waitTime = Math.min(pollInterval, remainingTime);
|
||||
const waitSeconds = Math.ceil(waitTime / 1000);
|
||||
|
||||
const url = new URL(TESTMAIL_API_URL);
|
||||
url.searchParams.set('apikey', TESTMAIL_API_KEY!);
|
||||
url.searchParams.set('namespace', TESTMAIL_NAMESPACE!);
|
||||
url.searchParams.set('tag', inboxId);
|
||||
url.searchParams.set('livequery', 'true');
|
||||
url.searchParams.set('wait', waitSeconds.toString());
|
||||
|
||||
try {
|
||||
const response = await fetch(url.toString());
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`testmail.app API error: ${response.status} ${response.statusText}`);
|
||||
}
|
||||
|
||||
const data: TestmailResponse = await response.json();
|
||||
|
||||
if (data.result !== 'success') {
|
||||
throw new Error(`testmail.app API error: ${data.message || 'Unknown error'}`);
|
||||
}
|
||||
|
||||
if (data.emails && data.emails.length > 0) {
|
||||
// Look for completion emails we haven't seen yet
|
||||
for (const testmailEmail of data.emails.sort((a, b) => b.timestamp - a.timestamp)) {
|
||||
// Skip if we've already processed this email
|
||||
if (seenTimestamps.has(testmailEmail.timestamp)) {
|
||||
continue;
|
||||
}
|
||||
seenTimestamps.add(testmailEmail.timestamp);
|
||||
|
||||
const email: Email = {
|
||||
subject: testmailEmail.subject,
|
||||
body: testmailEmail.html || testmailEmail.text,
|
||||
from: testmailEmail.from,
|
||||
timestamp: testmailEmail.timestamp,
|
||||
};
|
||||
|
||||
// Check if this is a completion email
|
||||
if (checkIsCompletionEmail(email)) {
|
||||
console.log(`Found completion email: ${email.subject}`);
|
||||
|
||||
// Optionally verify artist/title in subject or body
|
||||
if (artist && title) {
|
||||
const content = `${email.subject} ${email.body}`.toLowerCase();
|
||||
const hasArtist = content.includes(artist.toLowerCase());
|
||||
const hasTitle = content.includes(title.toLowerCase());
|
||||
|
||||
if (hasArtist && hasTitle) {
|
||||
console.log(` ✓ Verified artist "${artist}" and title "${title}" in email`);
|
||||
} else {
|
||||
console.log(` ⚠ Artist/title verification: artist=${hasArtist}, title=${hasTitle}`);
|
||||
}
|
||||
}
|
||||
|
||||
return email;
|
||||
} else {
|
||||
console.log(` Skipping non-completion email: ${testmailEmail.subject}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
if (Date.now() - startTime < timeout) {
|
||||
console.log(`Polling for completion email... (${Math.round((Date.now() - startTime) / 1000)}s elapsed)`);
|
||||
continue;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
throw new Error(`Timed out waiting for completion email after ${timeout}ms`);
|
||||
},
|
||||
|
||||
/**
|
||||
* Delete an inbox after testing.
|
||||
* In testmail.app, emails auto-expire - this is a no-op.
|
||||
@@ -257,8 +394,10 @@ function createDisabledHelper(): EmailHelper {
|
||||
isAvailable: false,
|
||||
createInbox: notAvailable,
|
||||
waitForEmail: notAvailable,
|
||||
waitForCompletionEmail: notAvailable,
|
||||
extractMagicLink: () => null,
|
||||
extractVerificationCode: () => null,
|
||||
isCompletionEmail: () => false,
|
||||
deleteInbox: notAvailable,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -154,6 +154,23 @@ test.describe('E2E Happy Path - Real User with Full UI Interactions', () => {
|
||||
// Only create email helper if we're doing the full signup flow
|
||||
const emailHelper = usePreConfiguredToken ? null : await createEmailHelper();
|
||||
|
||||
// Intercept API requests to disable YouTube upload for E2E tests
|
||||
// This prevents test jobs from uploading to YouTube when server default is true
|
||||
await page.route('**/api/audio-search/search', async (route) => {
|
||||
const request = route.request();
|
||||
if (request.method() === 'POST') {
|
||||
const postData = request.postDataJSON();
|
||||
// Explicitly disable YouTube upload for E2E test jobs
|
||||
postData.enable_youtube_upload = false;
|
||||
console.log(' [Request Intercept] Disabled YouTube upload for audio search');
|
||||
await route.continue({
|
||||
postData: JSON.stringify(postData),
|
||||
});
|
||||
} else {
|
||||
await route.continue();
|
||||
}
|
||||
});
|
||||
|
||||
try {
|
||||
// =========================================================================
|
||||
// STEP 1: Landing Page
|
||||
@@ -815,6 +832,62 @@ test.describe('E2E Happy Path - Real User with Full UI Interactions', () => {
|
||||
await page.screenshot({ path: 'test-results/09-job-complete.png' });
|
||||
console.log('STEP 9 COMPLETE: Job finished');
|
||||
|
||||
// =========================================================================
|
||||
// STEP 9.5: Verify Completion Email (if email testing available)
|
||||
// =========================================================================
|
||||
if (emailHelper && inboxId) {
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 9.5: Verify Completion Email');
|
||||
console.log('========================================');
|
||||
|
||||
try {
|
||||
console.log(' Waiting for completion email to arrive...');
|
||||
|
||||
// Wait up to 3 minutes for the completion email
|
||||
// Email is sent asynchronously after job completion, may take a moment
|
||||
const completionEmail = await emailHelper.waitForCompletionEmail(
|
||||
inboxId,
|
||||
TEST_SONG.artist,
|
||||
TEST_SONG.title,
|
||||
180_000 // 3 minutes
|
||||
);
|
||||
|
||||
console.log(` ✓ Completion email received: "${completionEmail.subject}"`);
|
||||
|
||||
// Verify email content
|
||||
const emailBody = completionEmail.body || '';
|
||||
const emailSubject = completionEmail.subject || '';
|
||||
|
||||
// Check for expected content indicators
|
||||
const hasVideoReadyText = emailBody.toLowerCase().includes('ready') ||
|
||||
emailSubject.toLowerCase().includes('ready');
|
||||
const hasSongInfo = emailBody.toLowerCase().includes(TEST_SONG.artist.toLowerCase()) ||
|
||||
emailSubject.toLowerCase().includes(TEST_SONG.artist.toLowerCase());
|
||||
|
||||
console.log(` Email verification:`);
|
||||
console.log(` - "ready" indicator: ${hasVideoReadyText ? '✓' : '✗'}`);
|
||||
console.log(` - Artist name: ${hasSongInfo ? '✓' : '✗'}`);
|
||||
|
||||
// Log email details for debugging
|
||||
console.log(` Email from: ${completionEmail.from}`);
|
||||
console.log(` Subject: ${emailSubject}`);
|
||||
console.log(` Body preview: ${emailBody.substring(0, 200)}...`);
|
||||
|
||||
await page.screenshot({ path: 'test-results/09.5-completion-email-received.png' });
|
||||
console.log('STEP 9.5 COMPLETE: Completion email verified');
|
||||
} catch (e) {
|
||||
// Don't fail the test if email verification times out - it's an enhancement
|
||||
console.log(` ⚠ WARNING: Completion email verification failed: ${e}`);
|
||||
console.log(' This may indicate email sending is disabled or delayed.');
|
||||
console.log(' Continuing with remaining steps...');
|
||||
}
|
||||
} else {
|
||||
console.log('\n========================================');
|
||||
console.log('STEP 9.5: Skip Completion Email Verification');
|
||||
console.log('========================================');
|
||||
console.log(' Skipping - using pre-configured token (no testmail inbox)');
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// STEP 10: Verify Downloads
|
||||
// =========================================================================
|
||||
|
||||
@@ -171,6 +171,71 @@ test.describe('Authentication - Beta Enrollment', () => {
|
||||
await page.waitForURL(/\/app/, { timeout: 10000 });
|
||||
expect(page.url()).toContain('/app');
|
||||
});
|
||||
|
||||
test('beta enrollment shows credits after redirect to app', async ({ page }) => {
|
||||
// This test verifies that after beta enrollment, the /app page
|
||||
// correctly fetches and displays user data including credits.
|
||||
// This catches the bug where client-side navigation didn't trigger fetchUser().
|
||||
await setupApiFixtures(page, {
|
||||
mocks: [
|
||||
{
|
||||
method: 'POST',
|
||||
path: '/api/users/beta/enroll',
|
||||
response: {
|
||||
status: 200,
|
||||
body: {
|
||||
status: 'success',
|
||||
message: 'Enrolled in beta',
|
||||
credits_granted: 1,
|
||||
session_token: 'mock-session-token-456',
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
method: 'GET',
|
||||
path: '/api/users/me',
|
||||
response: {
|
||||
status: 200,
|
||||
body: {
|
||||
user: {
|
||||
email: 'beta-tester@example.com',
|
||||
credits: 1,
|
||||
role: 'user',
|
||||
display_name: null,
|
||||
total_jobs_created: 0,
|
||||
total_jobs_completed: 0,
|
||||
},
|
||||
has_session: true,
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
method: 'GET',
|
||||
path: '/api/jobs',
|
||||
response: { body: [] },
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
await page.goto('/');
|
||||
await page.waitForLoadState('networkidle');
|
||||
|
||||
// Complete beta enrollment
|
||||
await page.getByRole('button', { name: /join beta program/i }).click();
|
||||
await page.locator('#beta-email').fill('beta-tester@example.com');
|
||||
await page.locator('textarea').fill(
|
||||
'I want to create a karaoke video for my favorite song!'
|
||||
);
|
||||
await page.locator('input[type="checkbox"]').check();
|
||||
await page.getByRole('button', { name: /get my free credit/i }).click();
|
||||
|
||||
// Wait for redirect to /app
|
||||
await page.waitForURL(/\/app/, { timeout: 10000 });
|
||||
|
||||
// Verify credits are displayed in the UI
|
||||
// The AuthStatus component should show "1 credit" after fetchUser() completes
|
||||
await expect(page.getByText(/1 credit/i)).toBeVisible({ timeout: 10000 });
|
||||
});
|
||||
});
|
||||
|
||||
test.describe('Authentication - Token Persistence', () => {
|
||||
|
||||
+69
-1
@@ -138,6 +138,15 @@ export interface JobLog {
|
||||
details?: Record<string, any>;
|
||||
}
|
||||
|
||||
export interface EncodingWorkerHealth {
|
||||
available: boolean;
|
||||
status: string; // 'ok' | 'offline' | 'not_configured'
|
||||
version?: string | null;
|
||||
active_jobs?: number;
|
||||
queue_length?: number;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
class ApiError extends Error {
|
||||
status: number;
|
||||
data?: any;
|
||||
@@ -418,6 +427,9 @@ export const api = {
|
||||
theme_id?: string;
|
||||
color_overrides?: ColorOverrides;
|
||||
non_interactive?: boolean;
|
||||
// Display overrides - if provided, these appear on title screens/filenames instead of search values
|
||||
display_artist?: string;
|
||||
display_title?: string;
|
||||
}
|
||||
): Promise<AudioSearchResponse> {
|
||||
const body: Record<string, any> = { artist, title, auto_download: autoDownload };
|
||||
@@ -428,6 +440,9 @@ export const api = {
|
||||
if (options?.theme_id) body.theme_id = options.theme_id;
|
||||
if (options?.color_overrides) body.color_overrides = options.color_overrides;
|
||||
if (options?.non_interactive !== undefined) body.non_interactive = options.non_interactive;
|
||||
// Display overrides
|
||||
if (options?.display_artist) body.display_artist = options.display_artist;
|
||||
if (options?.display_title) body.display_title = options.display_title;
|
||||
|
||||
const response = await fetch(`${API_BASE_URL}/api/audio-search/search`, {
|
||||
method: 'POST',
|
||||
@@ -583,7 +598,7 @@ export const api = {
|
||||
},
|
||||
|
||||
// ==========================================================================
|
||||
// Version API endpoint
|
||||
// Version & Health API endpoints
|
||||
// ==========================================================================
|
||||
|
||||
/**
|
||||
@@ -596,6 +611,14 @@ export const api = {
|
||||
return handleResponse(response);
|
||||
},
|
||||
|
||||
/**
|
||||
* Get encoding worker health status
|
||||
*/
|
||||
async getEncodingWorkerHealth(): Promise<EncodingWorkerHealth> {
|
||||
const response = await fetch(`${API_BASE_URL}/health/encoding-worker`);
|
||||
return handleResponse(response);
|
||||
},
|
||||
|
||||
// ==========================================================================
|
||||
// Credits/Payment API endpoints
|
||||
// ==========================================================================
|
||||
@@ -1038,7 +1061,52 @@ export const adminApi = {
|
||||
);
|
||||
return handleResponse(response);
|
||||
},
|
||||
|
||||
/**
|
||||
* Get rendered completion message for a job (for copy to clipboard)
|
||||
*/
|
||||
async getCompletionMessage(jobId: string): Promise<CompletionMessageResponse> {
|
||||
const response = await fetch(
|
||||
`${API_BASE_URL}/api/admin/jobs/${jobId}/completion-message`,
|
||||
{ headers: getAuthHeaders() }
|
||||
);
|
||||
return handleResponse(response);
|
||||
},
|
||||
|
||||
/**
|
||||
* Send completion email for a job
|
||||
*/
|
||||
async sendCompletionEmail(jobId: string, toEmail: string, ccAdmin: boolean = true): Promise<SendCompletionEmailResponse> {
|
||||
const response = await fetch(
|
||||
`${API_BASE_URL}/api/admin/jobs/${jobId}/send-completion-email`,
|
||||
{
|
||||
method: 'POST',
|
||||
headers: {
|
||||
...getAuthHeaders(),
|
||||
'Content-Type': 'application/json',
|
||||
},
|
||||
body: JSON.stringify({ to_email: toEmail, cc_admin: ccAdmin }),
|
||||
}
|
||||
);
|
||||
return handleResponse(response);
|
||||
},
|
||||
};
|
||||
|
||||
// Types for admin completion message API
|
||||
export interface CompletionMessageResponse {
|
||||
job_id: string;
|
||||
message: string;
|
||||
subject: string;
|
||||
youtube_url?: string;
|
||||
dropbox_url?: string;
|
||||
}
|
||||
|
||||
export interface SendCompletionEmailResponse {
|
||||
success: boolean;
|
||||
job_id: string;
|
||||
to_email: string;
|
||||
message: string;
|
||||
}
|
||||
|
||||
export { ApiError };
|
||||
|
||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "karaoke-gen-frontend",
|
||||
"version": "0.79.0",
|
||||
"version": "0.80.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "karaoke-gen-frontend",
|
||||
"version": "0.79.0",
|
||||
"version": "0.80.0",
|
||||
"dependencies": {
|
||||
"@hookform/resolvers": "^3.10.0",
|
||||
"@radix-ui/react-accordion": "1.2.2",
|
||||
|
||||
+162
-269
@@ -2,340 +2,233 @@
|
||||
|
||||
This directory contains Pulumi infrastructure code that manages all Google Cloud Platform resources for the karaoke generator backend.
|
||||
|
||||
## What This Replaces
|
||||
## Directory Structure
|
||||
|
||||
This replaces the manual setup in:
|
||||
- `backend/setup-gcp.sh` (shell script)
|
||||
- `docs/GCP-SETUP.md` (manual instructions)
|
||||
```
|
||||
infrastructure/
|
||||
├── __main__.py # Entry point - wires modules together (~340 lines)
|
||||
├── config.py # Centralized configuration constants
|
||||
├── Pulumi.yaml # Pulumi project config
|
||||
├── Pulumi.prod.yaml # Production stack config
|
||||
├── requirements.txt # Python dependencies
|
||||
│
|
||||
├── modules/ # Core GCP resources
|
||||
│ ├── database.py # Firestore database + indexes
|
||||
│ ├── storage.py # GCS bucket + lifecycle rules
|
||||
│ ├── artifact_registry.py # Docker repository
|
||||
│ ├── secrets.py # Secret Manager secrets
|
||||
│ ├── cloud_tasks.py # Task queues
|
||||
│ ├── cloud_run.py # Domain mapping, Cloud Run Jobs
|
||||
│ ├── monitoring.py # Alert policies
|
||||
│ └── iam/ # IAM by purpose
|
||||
│ ├── backend_sa.py # karaoke-backend service account
|
||||
│ ├── github_actions_sa.py # WIF + deployer permissions
|
||||
│ ├── claude_automation_sa.py # Claude Code automation
|
||||
│ └── worker_sas.py # VM service accounts
|
||||
│
|
||||
├── compute/ # VM-based resources
|
||||
│ ├── flacfetch_vm.py # Audio download service
|
||||
│ ├── encoding_worker_vm.py # Video encoding VM
|
||||
│ ├── github_runners.py # Self-hosted CI runners
|
||||
│ └── startup_scripts/ # VM startup scripts (.sh)
|
||||
│ ├── flacfetch.sh
|
||||
│ ├── encoding_worker.sh
|
||||
│ └── github_runner.sh
|
||||
│
|
||||
├── functions/ # Cloud Functions
|
||||
│ └── gdrive_validator/
|
||||
│ ├── main.py
|
||||
│ ├── requirements.txt
|
||||
│ └── deploy.sh # Source upload script
|
||||
│
|
||||
└── monitoring/ # Monitoring configs
|
||||
├── dashboard.json
|
||||
└── job-inspector.html
|
||||
```
|
||||
|
||||
## Resources Managed
|
||||
|
||||
This Pulumi program manages:
|
||||
- **Firestore Database**: Native mode database for job state
|
||||
- **Cloud Storage Bucket**: For uploads, outputs, and temp files
|
||||
- **Artifact Registry**: Docker repository for backend images
|
||||
- **Service Account**: For Cloud Run with appropriate IAM roles
|
||||
- **Secrets**: Secret Manager secrets (you add the values)
|
||||
- **IAM Bindings**: All necessary permissions
|
||||
- **Cloud Function**: GDrive Validator for checking public share folder
|
||||
- **Cloud Scheduler**: Daily trigger for GDrive validation
|
||||
| Category | Resources |
|
||||
|----------|-----------|
|
||||
| **Database** | Firestore (Native mode), 10 indexes, TTL policy |
|
||||
| **Storage** | GCS bucket with lifecycle rules |
|
||||
| **Registry** | Artifact Registry for Docker images |
|
||||
| **Secrets** | 18 Secret Manager secrets |
|
||||
| **IAM** | 7 service accounts, WIF, 50+ IAM bindings |
|
||||
| **Compute** | Encoding worker, Flacfetch, 10 GitHub runners |
|
||||
| **Tasks** | 5 Cloud Tasks queues |
|
||||
| **Monitoring** | 4 alert policies |
|
||||
| **Functions** | GDrive validator + scheduler |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **Pulumi CLI**: Install with `brew install pulumi`
|
||||
2. **Pulumi Account**: Sign up at https://app.pulumi.com
|
||||
3. **Google Cloud SDK**: Already installed
|
||||
4. **Python 3.10+**: For running Pulumi Python
|
||||
|
||||
## Setup
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
cd infrastructure
|
||||
|
||||
# Login to Pulumi (interactive)
|
||||
pulumi login
|
||||
# Preview changes
|
||||
pulumi preview --diff
|
||||
|
||||
# Create a new stack (e.g., "dev" or "prod")
|
||||
pulumi stack init dev
|
||||
|
||||
# Set your GCP project
|
||||
pulumi config set gcp:project nomadkaraoke
|
||||
pulumi config set gcp:region us-central1
|
||||
|
||||
# Install Python dependencies
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Preview Changes
|
||||
|
||||
```bash
|
||||
pulumi preview
|
||||
```
|
||||
|
||||
This shows you what will be created/updated/deleted without making changes.
|
||||
|
||||
### Apply Changes
|
||||
|
||||
```bash
|
||||
# Apply changes
|
||||
pulumi up
|
||||
```
|
||||
|
||||
This prompts you to confirm, then applies the changes.
|
||||
|
||||
### View Current State
|
||||
|
||||
```bash
|
||||
# View outputs
|
||||
pulumi stack output
|
||||
```
|
||||
|
||||
Shows all exported values (bucket name, service account email, etc.)
|
||||
## Prerequisites
|
||||
|
||||
### Destroy Everything
|
||||
1. **Pulumi CLI**: `brew install pulumi`
|
||||
2. **Pulumi Account**: Sign up at https://app.pulumi.com
|
||||
3. **Google Cloud SDK**: With authenticated account
|
||||
4. **Python 3.10+**: For running Pulumi
|
||||
|
||||
```bash
|
||||
pulumi destroy
|
||||
```
|
||||
## Module Guide
|
||||
|
||||
**Warning**: This deletes all infrastructure!
|
||||
### config.py
|
||||
Centralized configuration for the entire infrastructure:
|
||||
- Project ID, region, zone
|
||||
- Machine types and disk sizes
|
||||
- Google Drive folder ID
|
||||
- Number of GitHub runners
|
||||
|
||||
### modules/database.py
|
||||
Firestore database and composite indexes. Indexes are required for queries that filter/order by multiple fields.
|
||||
|
||||
### modules/storage.py
|
||||
GCS bucket with lifecycle rules:
|
||||
- 7-day cleanup of staging data
|
||||
- 24-hour cleanup of raw audio files
|
||||
|
||||
### modules/iam/
|
||||
Service accounts organized by purpose:
|
||||
- `backend_sa.py`: Cloud Run backend permissions
|
||||
- `github_actions_sa.py`: CI/CD via Workload Identity Federation
|
||||
- `worker_sas.py`: VM service accounts (flacfetch, encoding, runners)
|
||||
|
||||
### compute/
|
||||
VM definitions with startup scripts as separate files. Each module creates:
|
||||
- Static IP (where needed)
|
||||
- VM instance
|
||||
- Firewall rules
|
||||
|
||||
## Adding Secret Values
|
||||
|
||||
After running `pulumi up`, you need to add the actual secret values:
|
||||
After `pulumi up`, add actual secret values:
|
||||
|
||||
```bash
|
||||
# Add AudioShake API key
|
||||
echo -n "your-audioshake-key" | \
|
||||
gcloud secrets versions add audioshake-api-key --data-file=-
|
||||
# API keys
|
||||
echo -n "key" | gcloud secrets versions add audioshake-api-key --data-file=-
|
||||
echo -n "key" | gcloud secrets versions add genius-api-key --data-file=-
|
||||
|
||||
# Add Genius API key
|
||||
echo -n "your-genius-key" | \
|
||||
gcloud secrets versions add genius-api-key --data-file=-
|
||||
# Private tracker credentials
|
||||
echo -n "key" | gcloud secrets versions add red-api-key --data-file=-
|
||||
echo -n "url" | gcloud secrets versions add red-api-url --data-file=-
|
||||
|
||||
# Add Audio Separator API URL
|
||||
echo -n "https://your-modal-url" | \
|
||||
gcloud secrets versions add audio-separator-api-url --data-file=-
|
||||
|
||||
# Add RED API key (for flacfetch audio search - private tracker)
|
||||
echo -n "your-red-api-key" | \
|
||||
gcloud secrets versions add red-api-key --data-file=-
|
||||
|
||||
# Add RED API URL (for flacfetch audio search - private tracker)
|
||||
echo -n "https://your.red.url" | \
|
||||
gcloud secrets versions add red-api-url --data-file=-
|
||||
|
||||
# Add OPS API key (for flacfetch audio search - private tracker)
|
||||
echo -n "your-ops-api-key" | \
|
||||
gcloud secrets versions add ops-api-key --data-file=-
|
||||
|
||||
# Add Pushbullet API key (for GDrive validator notifications)
|
||||
echo -n "your-pushbullet-api-key" | \
|
||||
gcloud secrets versions add pushbullet-api-key --data-file=-
|
||||
# GitHub runner PAT (repo scope)
|
||||
echo -n "ghp_xxx" | gcloud secrets versions add github-runner-pat --data-file=-
|
||||
```
|
||||
|
||||
Or use Pulumi config secrets:
|
||||
## GDrive Validator
|
||||
|
||||
The Cloud Function validates your public Google Drive folder daily:
|
||||
- Checks for duplicate sequence numbers
|
||||
- Validates filename formats
|
||||
- Detects sequence gaps
|
||||
- Sends Pushbullet notifications
|
||||
|
||||
**Deploy:**
|
||||
```bash
|
||||
pulumi config set --secret audioshake-api-key "your-key"
|
||||
pulumi config set --secret genius-api-key "your-key"
|
||||
pulumi config set audio-separator-api-url "https://your-modal-url"
|
||||
cd functions/gdrive_validator
|
||||
./deploy.sh
|
||||
```
|
||||
|
||||
Then update `__main__.py` to create secret versions with these values.
|
||||
Note: `deploy.sh` uploads the source ZIP to GCS. Pulumi creates the function resources but doesn't handle source upload.
|
||||
|
||||
## Deploying Cloud Run
|
||||
## Self-Hosted GitHub Runners
|
||||
|
||||
After infrastructure is set up, deploy Cloud Run separately:
|
||||
20 GCP VMs serve as self-hosted runners with:
|
||||
- 200GB SSD (vs 14GB on GitHub-hosted)
|
||||
- Pre-installed: Docker, Python 3.13, Node 20, Java 21, FFmpeg
|
||||
- Labels: `self-hosted`, `linux`, `x64`, `gcp`, `large-disk`
|
||||
|
||||
**Usage in workflows:**
|
||||
```yaml
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
```
|
||||
|
||||
**Troubleshooting:**
|
||||
```bash
|
||||
# Build image
|
||||
gcloud builds submit --config=../cloudbuild.yaml --timeout=20m
|
||||
# View startup logs
|
||||
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
|
||||
"sudo cat /var/log/github-runner-startup.log"
|
||||
|
||||
# Deploy (using outputs from Pulumi)
|
||||
SA_EMAIL=$(pulumi stack output service_account_email)
|
||||
BUCKET_NAME=$(pulumi stack output bucket_name)
|
||||
|
||||
gcloud run deploy karaoke-backend \
|
||||
--image us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend:latest \
|
||||
--platform managed \
|
||||
--region us-central1 \
|
||||
--allow-unauthenticated \
|
||||
--service-account $SA_EMAIL \
|
||||
--memory 2Gi \
|
||||
--cpu 2 \
|
||||
--timeout 600 \
|
||||
--max-instances 10 \
|
||||
--min-instances 1 \
|
||||
--set-env-vars="GOOGLE_CLOUD_PROJECT=nomadkaraoke,GCS_BUCKET_NAME=$BUCKET_NAME,FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,LOG_LEVEL=INFO" \
|
||||
--set-secrets="AUDIOSHAKE_API_KEY=audioshake-api-key:latest,GENIUS_API_KEY=genius-api-key:latest,AUDIO_SEPARATOR_API_URL=audio-separator-api-url:latest"
|
||||
# Check runner service
|
||||
gcloud compute ssh github-runner-1 --zone=us-central1-a -- \
|
||||
"sudo systemctl status actions.runner.*"
|
||||
```
|
||||
|
||||
## Benefits
|
||||
## Encoding Worker
|
||||
|
||||
✅ **Reproducible**: Run `pulumi up` to create identical infrastructure
|
||||
✅ **Tracked**: All resources are tracked in Pulumi state
|
||||
✅ **Versioned**: Infrastructure changes are in git
|
||||
✅ **Safe**: Preview changes before applying
|
||||
✅ **Documented**: Code is self-documenting
|
||||
✅ **Collaborative**: Team can see and modify infrastructure
|
||||
High-performance VM for video encoding:
|
||||
- Machine: c4-standard-8 (8 vCPU, 32GB RAM)
|
||||
- Disk: 100GB hyperdisk-balanced
|
||||
- Pre-installed: FFmpeg (static build), Python 3.13, Noto fonts
|
||||
|
||||
**API Endpoints:**
|
||||
- `POST /encode` - Full encoding job
|
||||
- `POST /encode-preview` - Quick 480p preview
|
||||
- `GET /status/{job_id}` - Job status
|
||||
- `GET /health` - Health check
|
||||
|
||||
**Troubleshooting:**
|
||||
```bash
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-a -- \
|
||||
"sudo systemctl status encoding-worker"
|
||||
```
|
||||
|
||||
## State Management
|
||||
|
||||
Pulumi state is stored in Pulumi Cloud by default. This includes:
|
||||
- Current infrastructure state
|
||||
- Resource IDs and properties
|
||||
- Stack outputs
|
||||
- Configuration values
|
||||
Pulumi state is stored in Pulumi Cloud. To sync with actual cloud resources:
|
||||
|
||||
```bash
|
||||
pulumi refresh
|
||||
```
|
||||
|
||||
## Multi-Environment
|
||||
|
||||
To create separate environments:
|
||||
|
||||
```bash
|
||||
# Create dev stack
|
||||
pulumi stack init dev
|
||||
pulumi config set gcp:project nomadkaraoke-dev
|
||||
pulumi up
|
||||
|
||||
# Create prod stack
|
||||
pulumi stack init prod
|
||||
pulumi config set gcp:project nomadkaraoke
|
||||
pulumi up
|
||||
|
||||
# Switch between stacks
|
||||
pulumi stack select dev
|
||||
pulumi stack select prod
|
||||
pulumi stack select prod # Switch stacks
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Resource already exists"
|
||||
|
||||
If resources were created manually, you can import them:
|
||||
|
||||
Import manually-created resources:
|
||||
```bash
|
||||
pulumi import gcp:storage/bucket:Bucket karaoke-storage karaoke-gen-storage-nomadkaraoke
|
||||
```
|
||||
|
||||
### Permissions errors
|
||||
|
||||
Ensure your gcloud account has:
|
||||
- Owner or Editor role on the project
|
||||
- Or specific roles: Cloud Build Editor, Storage Admin, etc.
|
||||
|
||||
### State conflicts
|
||||
|
||||
If multiple people are working:
|
||||
```bash
|
||||
pulumi refresh # Sync state with actual cloud resources
|
||||
pulumi refresh # Sync state with cloud
|
||||
```
|
||||
|
||||
## Deploying GDrive Validator
|
||||
### Permissions errors
|
||||
Ensure your gcloud account has Owner or Editor role, or specific roles like Cloud Build Editor, Storage Admin.
|
||||
|
||||
The GDrive Validator is a Cloud Function that runs daily to check for duplicate
|
||||
sequence numbers, invalid filenames, and gaps in your public Google Drive folder.
|
||||
## Future Improvements
|
||||
|
||||
### Initial Setup
|
||||
### Phase 5: Custom Machine Images (Planned)
|
||||
|
||||
1. Add your Pushbullet API key to Secret Manager:
|
||||
```bash
|
||||
echo -n "your-pushbullet-api-key" | \
|
||||
gcloud secrets versions add pushbullet-api-key --data-file=-
|
||||
```
|
||||
The encoding worker currently takes ~10 minutes to start because it builds Python 3.13 from source on every boot. A Packer-built custom image would reduce this to ~30 seconds.
|
||||
|
||||
2. Deploy the function:
|
||||
```bash
|
||||
cd functions/gdrive_validator
|
||||
./deploy.sh
|
||||
```
|
||||
|
||||
3. Test the function manually:
|
||||
```bash
|
||||
gcloud functions call gdrive-validator --region=us-central1
|
||||
```
|
||||
|
||||
### How It Works
|
||||
|
||||
- Runs daily at 9 PM EST (6 PM Pacific) via Cloud Scheduler
|
||||
- Lists all files in your Google Drive folder (CDG, MP4, MP4-720p subfolders)
|
||||
- Checks for:
|
||||
- Duplicate sequence numbers (e.g., two files with NOMAD-1107)
|
||||
- Invalid filename formats (should be `NOMAD-XXXX - Artist - Title.ext`)
|
||||
- Sequence gaps (with configurable known gaps)
|
||||
- Sends Pushbullet notification if issues are found
|
||||
- Returns JSON with validation results
|
||||
|
||||
### Configuration
|
||||
|
||||
The function uses environment variables:
|
||||
- `GDRIVE_FOLDER_ID`: The Google Drive folder ID to validate (default: your public share)
|
||||
- `PUSHBULLET_API_KEY`: From Secret Manager, for sending notifications
|
||||
|
||||
Known gaps are configured in `main.py` and should match your local `validate_and_sync.py`.
|
||||
|
||||
## Self-Hosted GitHub Actions Runners
|
||||
|
||||
Multiple GCP VMs run self-hosted GitHub Actions runners to:
|
||||
- Solve disk space issues on GitHub-hosted runners (14GB vs 200GB)
|
||||
- Enable parallel job execution across multiple runners
|
||||
|
||||
### Specs
|
||||
|
||||
- **Instances**: 20x e2-standard-4 (4 vCPU, 16GB RAM each)
|
||||
- **Disk**: 200GB SSD per runner
|
||||
- **Pre-installed**: Docker, Python 3.13, Node.js 20, Java 21, Poetry, FFmpeg, gcloud CLI
|
||||
- **Labels**: `self-hosted`, `linux`, `x64`, `gcp`, `large-disk`
|
||||
|
||||
### Setup
|
||||
|
||||
1. Create a GitHub Personal Access Token (PAT) with `repo` scope:
|
||||
- Go to https://github.com/settings/tokens
|
||||
- Generate new token (classic)
|
||||
- Select `repo` scope (full control of private repositories)
|
||||
- Copy the token
|
||||
|
||||
2. Add the PAT to Secret Manager:
|
||||
```bash
|
||||
echo -n "ghp_your_token_here" | \
|
||||
gcloud secrets versions add github-runner-pat --data-file=-
|
||||
```
|
||||
|
||||
3. Run `pulumi up` to create the runner VMs
|
||||
|
||||
4. Verify the runners registered:
|
||||
- Go to https://github.com/nomadkaraoke/karaoke-gen/settings/actions/runners
|
||||
- You should see `gcp-runner-github-runner-1` through `gcp-runner-github-runner-20`
|
||||
|
||||
### Usage in Workflows
|
||||
|
||||
Jobs that need more disk space use the self-hosted runners:
|
||||
|
||||
```yaml
|
||||
jobs:
|
||||
build:
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
# ... rest of job
|
||||
```
|
||||
|
||||
GitHub automatically distributes jobs across available runners.
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
**Runner not appearing in GitHub:**
|
||||
```bash
|
||||
# Check startup logs (replace N with runner number 1-10)
|
||||
gcloud compute ssh github-runner-N --zone=us-central1-a -- \
|
||||
"sudo cat /var/log/github-runner-startup.log"
|
||||
```
|
||||
|
||||
**Re-register runner:**
|
||||
```bash
|
||||
# Restart the VM to re-run startup script
|
||||
gcloud compute instances reset github-runner-N --zone=us-central1-a
|
||||
```
|
||||
|
||||
**Check runner service status:**
|
||||
```bash
|
||||
gcloud compute ssh github-runner-N --zone=us-central1-a -- \
|
||||
"sudo systemctl status actions.runner.*"
|
||||
```
|
||||
|
||||
**List all runners:**
|
||||
```bash
|
||||
for i in {1..20}; do
|
||||
echo "=== github-runner-$i ==="
|
||||
gcloud compute instances describe github-runner-$i --zone=us-central1-a --format='get(status)'
|
||||
done
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Run `pulumi up` to create infrastructure
|
||||
2. Add secret values (including GitHub runner PAT)
|
||||
3. Build and deploy Cloud Run service
|
||||
4. Deploy GDrive validator function
|
||||
5. Update documentation to reference Pulumi workflow
|
||||
**See:** [`docs/PHASE5-PACKER-IMAGE-PLAN.md`](docs/PHASE5-PACKER-IMAGE-PLAN.md) for the implementation plan.
|
||||
|
||||
Benefits:
|
||||
- Startup time: 10 min → 30 sec
|
||||
- Pre-baked: Python 3.13, FFmpeg, fonts
|
||||
- Hot updates: karaoke-gen wheel still downloaded at runtime
|
||||
+210
-2144
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,26 @@
|
||||
"""
|
||||
Compute resources module.
|
||||
|
||||
Contains VM definitions for:
|
||||
- Flacfetch service (audio download)
|
||||
- Encoding worker (video encoding)
|
||||
- GitHub self-hosted runners
|
||||
"""
|
||||
|
||||
from .flacfetch_vm import create_flacfetch_vm, create_flacfetch_ip, create_flacfetch_firewall
|
||||
from .encoding_worker_vm import (
|
||||
create_encoding_worker_vm,
|
||||
create_encoding_worker_ip,
|
||||
create_encoding_worker_firewall,
|
||||
)
|
||||
from .github_runners import create_github_runners
|
||||
|
||||
__all__ = [
|
||||
"create_flacfetch_vm",
|
||||
"create_flacfetch_ip",
|
||||
"create_flacfetch_firewall",
|
||||
"create_encoding_worker_vm",
|
||||
"create_encoding_worker_ip",
|
||||
"create_encoding_worker_firewall",
|
||||
"create_github_runners",
|
||||
]
|
||||
@@ -0,0 +1,109 @@
|
||||
"""
|
||||
Encoding Worker VM resources.
|
||||
|
||||
Manages the high-performance VM instance for video encoding,
|
||||
including static IP and firewall rules.
|
||||
"""
|
||||
|
||||
import pulumi
|
||||
from pulumi_gcp import compute, serviceaccount
|
||||
|
||||
from config import REGION, ZONE, PROJECT_ID, MachineTypes, DiskSizes
|
||||
from .startup_scripts import read_script
|
||||
|
||||
|
||||
def create_encoding_worker_ip() -> compute.Address:
|
||||
"""
|
||||
Create static IP for the encoding worker.
|
||||
|
||||
Returns:
|
||||
compute.Address: The static IP resource.
|
||||
"""
|
||||
return compute.Address(
|
||||
"encoding-worker-ip",
|
||||
name="encoding-worker-static-ip",
|
||||
region=REGION,
|
||||
address_type="EXTERNAL",
|
||||
description="Static external IP for encoding worker service",
|
||||
)
|
||||
|
||||
|
||||
def create_encoding_worker_vm(
|
||||
ip: compute.Address,
|
||||
service_account: serviceaccount.Account,
|
||||
) -> compute.Instance:
|
||||
"""
|
||||
Create the encoding worker VM instance.
|
||||
|
||||
This VM runs video encoding jobs with high CPU/memory resources.
|
||||
Uses hyperdisk-balanced for fast I/O during encoding.
|
||||
|
||||
Uses a custom Packer-built image with Python 3.13, FFmpeg, and fonts
|
||||
pre-installed to reduce startup time from ~10 minutes to ~30 seconds.
|
||||
See infrastructure/packer/README.md for image build instructions.
|
||||
|
||||
Args:
|
||||
ip: The static IP address for the VM.
|
||||
service_account: The encoding worker service account.
|
||||
|
||||
Returns:
|
||||
compute.Instance: The VM instance resource.
|
||||
"""
|
||||
startup_script = read_script("encoding_worker.sh")
|
||||
|
||||
# Use custom Packer image with dependencies pre-installed
|
||||
# Image family returns latest image in the family
|
||||
# Fallback: "debian-cloud/debian-12" if image doesn't exist yet
|
||||
custom_image = f"projects/{PROJECT_ID}/global/images/family/encoding-worker"
|
||||
|
||||
return compute.Instance(
|
||||
"encoding-worker",
|
||||
name="encoding-worker",
|
||||
machine_type=MachineTypes.ENCODING_WORKER,
|
||||
zone=ZONE,
|
||||
boot_disk=compute.InstanceBootDiskArgs(
|
||||
initialize_params=compute.InstanceBootDiskInitializeParamsArgs(
|
||||
image=custom_image,
|
||||
size=DiskSizes.ENCODING_WORKER,
|
||||
type="hyperdisk-balanced",
|
||||
),
|
||||
),
|
||||
network_interfaces=[compute.InstanceNetworkInterfaceArgs(
|
||||
network="default",
|
||||
access_configs=[compute.InstanceNetworkInterfaceAccessConfigArgs(
|
||||
nat_ip=ip.address,
|
||||
)],
|
||||
)],
|
||||
service_account=compute.InstanceServiceAccountArgs(
|
||||
email=service_account.email,
|
||||
scopes=["cloud-platform"],
|
||||
),
|
||||
metadata_startup_script=startup_script,
|
||||
tags=["encoding-worker"],
|
||||
allow_stopping_for_update=True,
|
||||
advanced_machine_features=compute.InstanceAdvancedMachineFeaturesArgs(
|
||||
threads_per_core=2,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def create_encoding_worker_firewall() -> compute.Firewall:
|
||||
"""
|
||||
Create firewall rules for the encoding worker.
|
||||
|
||||
Opens port 8080 for the HTTP API (authentication required).
|
||||
|
||||
Returns:
|
||||
compute.Firewall: The firewall resource.
|
||||
"""
|
||||
return compute.Firewall(
|
||||
"encoding-worker-firewall",
|
||||
name="encoding-worker-allow-http",
|
||||
network="default",
|
||||
allows=[
|
||||
compute.FirewallAllowArgs(protocol="tcp", ports=["8080"]),
|
||||
],
|
||||
source_ranges=["0.0.0.0/0"],
|
||||
target_tags=["encoding-worker"],
|
||||
description="Allow HTTP access to encoding worker (auth required)",
|
||||
)
|
||||
@@ -0,0 +1,105 @@
|
||||
"""
|
||||
Flacfetch VM resources.
|
||||
|
||||
Manages the VM instance for the flacfetch audio download service,
|
||||
including static IP and firewall rules.
|
||||
"""
|
||||
|
||||
import pulumi
|
||||
from pulumi_gcp import compute, storage, serviceaccount
|
||||
|
||||
from config import REGION, ZONE, MachineTypes, DiskSizes
|
||||
from .startup_scripts import read_script
|
||||
|
||||
|
||||
def create_flacfetch_ip() -> compute.Address:
|
||||
"""
|
||||
Create static IP for flacfetch.
|
||||
|
||||
This IP is whitelisted on private trackers, so it must remain stable.
|
||||
|
||||
Returns:
|
||||
compute.Address: The static IP resource.
|
||||
"""
|
||||
return compute.Address(
|
||||
"flacfetch-ip",
|
||||
name="flacfetch-static-ip",
|
||||
region=REGION,
|
||||
address_type="EXTERNAL",
|
||||
description="Static IP for flacfetch service (torrent downloads)",
|
||||
)
|
||||
|
||||
|
||||
def create_flacfetch_vm(
|
||||
ip: compute.Address,
|
||||
service_account: serviceaccount.Account,
|
||||
bucket: storage.Bucket,
|
||||
) -> compute.Instance:
|
||||
"""
|
||||
Create the flacfetch VM instance.
|
||||
|
||||
Args:
|
||||
ip: The static IP address for the VM.
|
||||
service_account: The flacfetch service account.
|
||||
bucket: The GCS bucket for storing downloaded files.
|
||||
|
||||
Returns:
|
||||
compute.Instance: The VM instance resource.
|
||||
"""
|
||||
startup_script = read_script("flacfetch.sh")
|
||||
|
||||
return compute.Instance(
|
||||
"flacfetch-service",
|
||||
name="flacfetch-service",
|
||||
machine_type=MachineTypes.FLACFETCH,
|
||||
zone=ZONE,
|
||||
boot_disk=compute.InstanceBootDiskArgs(
|
||||
initialize_params=compute.InstanceBootDiskInitializeParamsArgs(
|
||||
image="debian-cloud/debian-12",
|
||||
size=DiskSizes.FLACFETCH,
|
||||
type="pd-ssd",
|
||||
),
|
||||
),
|
||||
network_interfaces=[compute.InstanceNetworkInterfaceArgs(
|
||||
network="default",
|
||||
access_configs=[compute.InstanceNetworkInterfaceAccessConfigArgs(
|
||||
nat_ip=ip.address,
|
||||
)],
|
||||
)],
|
||||
service_account=compute.InstanceServiceAccountArgs(
|
||||
email=service_account.email,
|
||||
scopes=["cloud-platform"],
|
||||
),
|
||||
metadata_startup_script=startup_script,
|
||||
metadata={
|
||||
"gcs-bucket": bucket.name,
|
||||
},
|
||||
tags=["flacfetch-service"],
|
||||
allow_stopping_for_update=True,
|
||||
)
|
||||
|
||||
|
||||
def create_flacfetch_firewall() -> compute.Firewall:
|
||||
"""
|
||||
Create firewall rules for flacfetch.
|
||||
|
||||
Opens ports for:
|
||||
- 51413 TCP/UDP: BitTorrent peer connections
|
||||
- 8080 TCP: HTTP API
|
||||
|
||||
Returns:
|
||||
compute.Firewall: The firewall resource.
|
||||
"""
|
||||
return compute.Firewall(
|
||||
"flacfetch-firewall",
|
||||
name="flacfetch-firewall",
|
||||
network="default",
|
||||
allows=[
|
||||
compute.FirewallAllowArgs(protocol="tcp", ports=["51413"]),
|
||||
compute.FirewallAllowArgs(protocol="udp", ports=["51413"]),
|
||||
compute.FirewallAllowArgs(protocol="tcp", ports=["8080"]),
|
||||
],
|
||||
source_ranges=["0.0.0.0/0"],
|
||||
target_tags=["flacfetch-service"],
|
||||
description="Firewall rules for flacfetch torrent/API service",
|
||||
)
|
||||
@@ -0,0 +1,65 @@
|
||||
"""
|
||||
GitHub Actions self-hosted runner VMs.
|
||||
|
||||
Manages multiple VM instances that serve as self-hosted runners
|
||||
for GitHub Actions CI/CD workflows.
|
||||
"""
|
||||
|
||||
import pulumi
|
||||
from pulumi_gcp import compute, serviceaccount, secretmanager
|
||||
|
||||
from config import ZONE, MachineTypes, DiskSizes, NUM_GITHUB_RUNNERS
|
||||
from .startup_scripts import read_script
|
||||
|
||||
|
||||
def create_github_runners(
|
||||
service_account: serviceaccount.Account,
|
||||
runner_pat_secret: secretmanager.Secret,
|
||||
) -> list[compute.Instance]:
|
||||
"""
|
||||
Create GitHub Actions self-hosted runner VM instances.
|
||||
|
||||
Creates NUM_GITHUB_RUNNERS VMs, each configured as a GitHub Actions
|
||||
self-hosted runner for the karaoke-gen repository.
|
||||
|
||||
Args:
|
||||
service_account: The GitHub runner service account.
|
||||
runner_pat_secret: The Secret Manager secret containing the GitHub PAT.
|
||||
|
||||
Returns:
|
||||
list[compute.Instance]: List of runner VM instances.
|
||||
"""
|
||||
startup_script = read_script("github_runner.sh")
|
||||
|
||||
runner_vms = []
|
||||
for i in range(1, NUM_GITHUB_RUNNERS + 1):
|
||||
vm = compute.Instance(
|
||||
f"github-runner-{i}",
|
||||
name=f"github-runner-{i}",
|
||||
machine_type=MachineTypes.GITHUB_RUNNER,
|
||||
zone=ZONE,
|
||||
boot_disk=compute.InstanceBootDiskArgs(
|
||||
initialize_params=compute.InstanceBootDiskInitializeParamsArgs(
|
||||
image="debian-cloud/debian-12",
|
||||
size=DiskSizes.GITHUB_RUNNER,
|
||||
type="pd-ssd",
|
||||
),
|
||||
),
|
||||
network_interfaces=[compute.InstanceNetworkInterfaceArgs(
|
||||
network="default",
|
||||
access_configs=[compute.InstanceNetworkInterfaceAccessConfigArgs()],
|
||||
)],
|
||||
service_account=compute.InstanceServiceAccountArgs(
|
||||
email=service_account.email,
|
||||
scopes=["cloud-platform"],
|
||||
),
|
||||
metadata_startup_script=startup_script,
|
||||
tags=["github-runner"],
|
||||
allow_stopping_for_update=True,
|
||||
opts=pulumi.ResourceOptions(
|
||||
depends_on=[runner_pat_secret],
|
||||
),
|
||||
)
|
||||
runner_vms.append(vm)
|
||||
|
||||
return runner_vms
|
||||
@@ -0,0 +1,38 @@
|
||||
"""
|
||||
Startup scripts for VM instances.
|
||||
|
||||
Provides helper functions to load startup scripts from external files,
|
||||
making them easier to edit, test, and maintain.
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def read_script(name: str, **substitutions) -> str:
|
||||
"""
|
||||
Read a startup script file and apply substitutions.
|
||||
|
||||
Args:
|
||||
name: Name of the script file (e.g., "flacfetch.sh")
|
||||
**substitutions: Key-value pairs for template substitution.
|
||||
Keys should match ${KEY} patterns in the script.
|
||||
|
||||
Returns:
|
||||
str: The script content with substitutions applied.
|
||||
|
||||
Example:
|
||||
script = read_script("github_runner.sh",
|
||||
RUNNER_VERSION="2.321.0",
|
||||
RUNNER_LABELS="self-hosted,linux,x64"
|
||||
)
|
||||
"""
|
||||
script_path = Path(__file__).parent / name
|
||||
content = script_path.read_text()
|
||||
|
||||
for key, value in substitutions.items():
|
||||
content = content.replace(f"${{{key}}}", str(value))
|
||||
|
||||
return content
|
||||
|
||||
|
||||
__all__ = ["read_script"]
|
||||
@@ -0,0 +1,31 @@
|
||||
#!/bin/bash
|
||||
# Encoding worker startup script (for Packer-built custom image)
|
||||
#
|
||||
# Most dependencies are pre-installed in the custom GCE image:
|
||||
# - Python 3.13 at /opt/python313
|
||||
# - FFmpeg at /usr/local/bin/ffmpeg
|
||||
# - Virtual environment at /opt/encoding-worker/venv
|
||||
# - Systemd service: encoding-worker.service
|
||||
#
|
||||
# This startup script just starts the service.
|
||||
# The service's ExecStartPre (/opt/encoding-worker/startup.sh) handles:
|
||||
# - Fetching API key from Secret Manager
|
||||
# - Installing latest karaoke-gen wheel from GCS
|
||||
#
|
||||
# See infrastructure/packer/README.md for image build instructions.
|
||||
|
||||
set -e
|
||||
exec > >(tee /var/log/encoding-worker-startup.log) 2>&1
|
||||
|
||||
echo "=== Encoding worker VM startup at $(date) ==="
|
||||
|
||||
# Start Docker (in case it's not running)
|
||||
systemctl start docker || echo "Docker already running or not available"
|
||||
|
||||
# Start the encoding worker service
|
||||
# This triggers ExecStartPre which fetches API key and installs wheel
|
||||
echo "Starting encoding-worker service..."
|
||||
systemctl start encoding-worker
|
||||
|
||||
echo "=== Encoding worker started at $(date) ==="
|
||||
echo "Service listening on port 8080"
|
||||
@@ -0,0 +1,119 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
# Log to a file for debugging
|
||||
exec > >(tee /var/log/flacfetch-startup.log) 2>&1
|
||||
echo "Starting flacfetch setup at $(date)"
|
||||
|
||||
# Install dependencies
|
||||
apt-get update
|
||||
apt-get install -y python3-pip python3-venv transmission-daemon ffmpeg git curl
|
||||
|
||||
# Configure Transmission
|
||||
echo "Configuring Transmission daemon..."
|
||||
systemctl stop transmission-daemon || true
|
||||
|
||||
# Create transmission directories
|
||||
mkdir -p /var/lib/transmission-daemon/downloads
|
||||
mkdir -p /var/lib/transmission-daemon/.config/transmission-daemon
|
||||
chown -R debian-transmission:debian-transmission /var/lib/transmission-daemon
|
||||
|
||||
# Configure transmission settings
|
||||
cat > /var/lib/transmission-daemon/.config/transmission-daemon/settings.json << 'SETTINGS'
|
||||
{
|
||||
"download-dir": "/var/lib/transmission-daemon/downloads",
|
||||
"incomplete-dir": "/var/lib/transmission-daemon/.incomplete",
|
||||
"incomplete-dir-enabled": true,
|
||||
"rpc-authentication-required": false,
|
||||
"rpc-bind-address": "127.0.0.1",
|
||||
"rpc-enabled": true,
|
||||
"rpc-port": 9091,
|
||||
"rpc-whitelist-enabled": false,
|
||||
"peer-port": 51413,
|
||||
"port-forwarding-enabled": false,
|
||||
"speed-limit-down": 0,
|
||||
"speed-limit-down-enabled": false,
|
||||
"speed-limit-up": 0,
|
||||
"speed-limit-up-enabled": false,
|
||||
"ratio-limit-enabled": false,
|
||||
"umask": 2,
|
||||
"encryption": 1
|
||||
}
|
||||
SETTINGS
|
||||
|
||||
chown debian-transmission:debian-transmission /var/lib/transmission-daemon/.config/transmission-daemon/settings.json
|
||||
systemctl start transmission-daemon
|
||||
echo "Transmission daemon started"
|
||||
|
||||
# Install flacfetch
|
||||
echo "Installing flacfetch..."
|
||||
cd /opt
|
||||
|
||||
# Clone flacfetch (we'll use the repo from GitHub)
|
||||
if [ ! -d "flacfetch" ]; then
|
||||
git clone https://github.com/nomadkaraoke/flacfetch.git
|
||||
fi
|
||||
cd flacfetch
|
||||
|
||||
# Create and activate virtual environment
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
|
||||
# Install flacfetch with API dependencies
|
||||
pip install -e ".[api]"
|
||||
|
||||
# Get secrets from Secret Manager
|
||||
echo "Fetching secrets from Secret Manager..."
|
||||
FLACFETCH_API_KEY=$(gcloud secrets versions access latest --secret=flacfetch-api-key 2>/dev/null || echo "")
|
||||
RED_API_KEY=$(gcloud secrets versions access latest --secret=red-api-key 2>/dev/null || echo "")
|
||||
RED_API_URL=$(gcloud secrets versions access latest --secret=red-api-url 2>/dev/null || echo "")
|
||||
OPS_API_KEY=$(gcloud secrets versions access latest --secret=ops-api-key 2>/dev/null || echo "")
|
||||
OPS_API_URL=$(gcloud secrets versions access latest --secret=ops-api-url 2>/dev/null || echo "")
|
||||
|
||||
# Get bucket name from project metadata
|
||||
GCS_BUCKET=$(gcloud compute project-info describe --format='value(commonInstanceMetadata.items.gcs-bucket)' 2>/dev/null || echo "")
|
||||
if [ -z "$GCS_BUCKET" ]; then
|
||||
# Fallback: construct from project ID
|
||||
PROJECT_ID=$(curl -s "http://metadata.google.internal/computeMetadata/v1/project/project-id" -H "Metadata-Flavor: Google")
|
||||
GCS_BUCKET="karaoke-gen-storage-${PROJECT_ID}"
|
||||
fi
|
||||
|
||||
echo "Using GCS bucket: $GCS_BUCKET"
|
||||
|
||||
# Create systemd service for flacfetch
|
||||
echo "Creating systemd service..."
|
||||
cat > /etc/systemd/system/flacfetch.service << SYSTEMD
|
||||
[Unit]
|
||||
Description=Flacfetch HTTP API Service
|
||||
After=network.target transmission-daemon.service
|
||||
Requires=transmission-daemon.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
WorkingDirectory=/opt/flacfetch
|
||||
Environment="FLACFETCH_API_KEY=${FLACFETCH_API_KEY}"
|
||||
Environment="RED_API_KEY=${RED_API_KEY}"
|
||||
Environment="RED_API_URL=${RED_API_URL}"
|
||||
Environment="OPS_API_KEY=${OPS_API_KEY}"
|
||||
Environment="OPS_API_URL=${OPS_API_URL}"
|
||||
Environment="GCS_BUCKET=${GCS_BUCKET}"
|
||||
Environment="FLACFETCH_KEEP_SEEDING=true"
|
||||
Environment="FLACFETCH_MIN_FREE_GB=5"
|
||||
Environment="FLACFETCH_DOWNLOAD_DIR=/var/lib/transmission-daemon/downloads"
|
||||
Environment="TRANSMISSION_HOST=localhost"
|
||||
Environment="TRANSMISSION_PORT=9091"
|
||||
ExecStart=/opt/flacfetch/venv/bin/flacfetch serve --host 0.0.0.0 --port 8080
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
SYSTEMD
|
||||
|
||||
systemctl daemon-reload
|
||||
systemctl enable flacfetch
|
||||
systemctl start flacfetch
|
||||
|
||||
echo "Flacfetch service started at $(date)"
|
||||
echo "Setup complete!"
|
||||
@@ -0,0 +1,261 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
# Log to a file for debugging
|
||||
exec > >(tee /var/log/github-runner-startup.log) 2>&1
|
||||
echo "Starting GitHub Actions runner setup at $(date)"
|
||||
|
||||
# System updates and core dependencies
|
||||
apt-get update
|
||||
apt-get install -y \
|
||||
curl \
|
||||
git \
|
||||
jq \
|
||||
unzip \
|
||||
build-essential \
|
||||
libssl-dev \
|
||||
libffi-dev \
|
||||
software-properties-common \
|
||||
apt-transport-https \
|
||||
ca-certificates \
|
||||
gnupg \
|
||||
lsb-release
|
||||
|
||||
# ==================== Docker ====================
|
||||
echo "Installing Docker..."
|
||||
curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --batch --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
|
||||
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable" > /etc/apt/sources.list.d/docker.list
|
||||
apt-get update
|
||||
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
|
||||
|
||||
# Configure Docker for the runner user
|
||||
systemctl enable docker
|
||||
systemctl start docker
|
||||
|
||||
# ==================== Python 3.13 via pyenv ====================
|
||||
# setup-python action on self-hosted runners needs Python in the tool cache
|
||||
echo "Installing Python 3.13 via pyenv..."
|
||||
|
||||
# Install pyenv dependencies
|
||||
apt-get install -y make build-essential libssl-dev zlib1g-dev \
|
||||
libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm \
|
||||
libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev
|
||||
|
||||
# Install pyenv system-wide
|
||||
export PYENV_ROOT=/opt/pyenv
|
||||
curl https://pyenv.run | bash
|
||||
|
||||
# Add to system profile for future sessions
|
||||
cat > /etc/profile.d/pyenv.sh << 'PYENV_PROFILE'
|
||||
export PYENV_ROOT=/opt/pyenv
|
||||
export PATH="$PYENV_ROOT/bin:$PATH"
|
||||
eval "$(pyenv init -)"
|
||||
PYENV_PROFILE
|
||||
|
||||
# Source pyenv for current session
|
||||
export PATH="/opt/pyenv/bin:$PATH"
|
||||
eval "$(/opt/pyenv/bin/pyenv init -)"
|
||||
|
||||
# Install Python 3.13
|
||||
/opt/pyenv/bin/pyenv install 3.13.0
|
||||
/opt/pyenv/bin/pyenv global 3.13.0
|
||||
|
||||
# Create symlinks for system-wide access
|
||||
ln -sf /opt/pyenv/shims/python3.13 /usr/local/bin/python3.13
|
||||
ln -sf /opt/pyenv/shims/python3 /usr/local/bin/python3
|
||||
ln -sf /opt/pyenv/shims/pip3 /usr/local/bin/pip3
|
||||
|
||||
# ==================== Node.js 20 ====================
|
||||
echo "Installing Node.js 20..."
|
||||
curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
|
||||
apt-get install -y nodejs
|
||||
|
||||
# ==================== Java 21 (for Firestore emulator) ====================
|
||||
echo "Installing Java 21 via Temurin..."
|
||||
# Use Temurin (Adoptium) - more reliable than Debian repos
|
||||
curl -fsSL https://packages.adoptium.net/artifactory/api/gpg/key/public | gpg --batch --dearmor -o /usr/share/keyrings/adoptium.gpg
|
||||
echo "deb [signed-by=/usr/share/keyrings/adoptium.gpg] https://packages.adoptium.net/artifactory/deb $(lsb_release -cs) main" > /etc/apt/sources.list.d/adoptium.list
|
||||
apt-get update
|
||||
apt-get install -y temurin-21-jdk
|
||||
|
||||
# ==================== FFmpeg ====================
|
||||
echo "Installing FFmpeg..."
|
||||
apt-get install -y ffmpeg
|
||||
|
||||
# ==================== Poetry ====================
|
||||
echo "Installing Poetry..."
|
||||
curl -sSL https://install.python-poetry.org | python3 -
|
||||
ln -sf /root/.local/bin/poetry /usr/local/bin/poetry
|
||||
|
||||
# ==================== Google Cloud SDK ====================
|
||||
echo "Installing Google Cloud SDK..."
|
||||
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --batch --dearmor -o /usr/share/keyrings/cloud.google.gpg
|
||||
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" > /etc/apt/sources.list.d/google-cloud-sdk.list
|
||||
apt-get update
|
||||
apt-get install -y google-cloud-cli google-cloud-cli-firestore-emulator
|
||||
|
||||
# ==================== Create runner user ====================
|
||||
echo "Creating runner user..."
|
||||
useradd -m -s /bin/bash runner || true
|
||||
usermod -aG docker runner
|
||||
|
||||
# ==================== GitHub Actions Runner ====================
|
||||
echo "Setting up GitHub Actions Runner..."
|
||||
RUNNER_VERSION="2.321.0"
|
||||
RUNNER_DIR="/home/runner/actions-runner"
|
||||
|
||||
mkdir -p $RUNNER_DIR
|
||||
cd $RUNNER_DIR
|
||||
|
||||
# Download runner
|
||||
curl -o actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz -L \
|
||||
https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz
|
||||
tar xzf actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz
|
||||
rm actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz
|
||||
|
||||
chown -R runner:runner $RUNNER_DIR
|
||||
|
||||
# Get GitHub PAT from Secret Manager
|
||||
echo "Fetching GitHub PAT from Secret Manager..."
|
||||
GITHUB_PAT=$(gcloud secrets versions access latest --secret=github-runner-pat)
|
||||
|
||||
if [ -z "$GITHUB_PAT" ]; then
|
||||
echo "ERROR: GitHub PAT not found in Secret Manager"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Get runner registration token using PAT
|
||||
echo "Getting runner registration token..."
|
||||
REGISTRATION_TOKEN=$(curl -s -X POST \
|
||||
-H "Authorization: token $GITHUB_PAT" \
|
||||
-H "Accept: application/vnd.github.v3+json" \
|
||||
https://api.github.com/repos/nomadkaraoke/karaoke-gen/actions/runners/registration-token | jq -r .token)
|
||||
|
||||
if [ -z "$REGISTRATION_TOKEN" ] || [ "$REGISTRATION_TOKEN" = "null" ]; then
|
||||
echo "ERROR: Failed to get registration token"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Configure runner (non-interactive)
|
||||
echo "Configuring runner..."
|
||||
cd $RUNNER_DIR
|
||||
sudo -u runner ./config.sh \
|
||||
--url https://github.com/nomadkaraoke/karaoke-gen \
|
||||
--token "$REGISTRATION_TOKEN" \
|
||||
--name "gcp-runner-$(hostname)" \
|
||||
--labels "self-hosted,linux,x64,gcp,large-disk" \
|
||||
--work "_work" \
|
||||
--unattended \
|
||||
--replace
|
||||
|
||||
# Install and start runner as a service
|
||||
echo "Installing runner service..."
|
||||
./svc.sh install runner
|
||||
./svc.sh start
|
||||
|
||||
echo "GitHub Actions runner setup complete at $(date)"
|
||||
echo "Runner registered with labels: self-hosted,linux,x64,gcp,large-disk"
|
||||
|
||||
# ==================== Setup Python in tool cache ====================
|
||||
# setup-python action looks for Python in RUNNER_TOOL_CACHE/_tool/Python
|
||||
echo "Setting up Python 3.13 in tool cache for setup-python action..."
|
||||
TOOL_CACHE="/home/runner/actions-runner/_work/_tool"
|
||||
mkdir -p $TOOL_CACHE/Python/3.13.0/x64
|
||||
|
||||
# Copy pyenv Python to tool cache
|
||||
cp -r /opt/pyenv/versions/3.13.0/* $TOOL_CACHE/Python/3.13.0/x64/
|
||||
|
||||
# Create marker file that setup-python looks for
|
||||
touch $TOOL_CACHE/Python/3.13.0/x64.complete
|
||||
|
||||
# Fix permissions - must own entire _work directory, not just _tool
|
||||
chown -R runner:runner /home/runner/actions-runner/_work
|
||||
|
||||
echo "Python 3.13 added to tool cache"
|
||||
|
||||
# ==================== Docker cleanup cron ====================
|
||||
# Prevent disk from filling up with old Docker images
|
||||
# Run hourly instead of daily, with threshold-based cleanup
|
||||
echo "Setting up Docker cleanup cron..."
|
||||
cat > /etc/cron.hourly/docker-cleanup << 'CRON'
|
||||
#!/bin/bash
|
||||
# Docker disk space cleanup for GitHub Actions runners
|
||||
# Runs hourly, with aggressive cleanup when disk usage exceeds threshold
|
||||
|
||||
# Log to syslog
|
||||
exec 1> >(logger -t docker-cleanup) 2>&1
|
||||
|
||||
THRESHOLD=70 # Cleanup when disk usage exceeds this percentage
|
||||
|
||||
# Get current disk usage percentage (remove % sign)
|
||||
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
|
||||
echo "Current disk usage: ${USAGE}%"
|
||||
|
||||
if [ "$USAGE" -gt "$THRESHOLD" ]; then
|
||||
echo "Disk usage ${USAGE}% exceeds threshold ${THRESHOLD}%, running cleanup..."
|
||||
|
||||
# Remove ALL unused images (not just old ones) - this is the key fix
|
||||
# The previous 168h filter let recent images accumulate
|
||||
docker system prune -af
|
||||
|
||||
# Also clean builder cache which can grow large
|
||||
docker builder prune -af
|
||||
|
||||
# Report new usage
|
||||
NEW_USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
|
||||
echo "Cleanup complete. Disk usage: ${NEW_USAGE}%"
|
||||
else
|
||||
echo "Disk usage ${USAGE}% is below threshold ${THRESHOLD}%, skipping cleanup"
|
||||
fi
|
||||
CRON
|
||||
chmod +x /etc/cron.hourly/docker-cleanup
|
||||
|
||||
# Remove old daily cleanup if it exists (migrating to hourly)
|
||||
rm -f /etc/cron.daily/docker-cleanup
|
||||
|
||||
# ==================== Docker pre-warm cron ====================
|
||||
# Keep base images fresh by pulling hourly
|
||||
echo "Setting up Docker pre-warm cron..."
|
||||
cat > /etc/cron.hourly/docker-prewarm << 'CRON'
|
||||
#!/bin/bash
|
||||
# Pre-warm Docker images used by CI/CD
|
||||
# This ensures runners always have the latest base images cached
|
||||
|
||||
# Log to syslog
|
||||
exec 1> >(logger -t docker-prewarm) 2>&1
|
||||
|
||||
echo "Starting Docker image pre-warm..."
|
||||
|
||||
# Configure Docker auth (uses instance service account)
|
||||
gcloud auth configure-docker us-central1-docker.pkg.dev --quiet 2>/dev/null
|
||||
|
||||
# Pull latest images (|| true to not fail if network issues)
|
||||
docker pull us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest || true
|
||||
docker pull us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend:latest || true
|
||||
|
||||
echo "Docker image pre-warm complete"
|
||||
CRON
|
||||
chmod +x /etc/cron.hourly/docker-prewarm
|
||||
|
||||
# ==================== Pre-warm Docker images ====================
|
||||
# Pre-fetch commonly used Docker images to speed up first CI runs
|
||||
echo "Pre-warming Docker images for faster CI builds..."
|
||||
|
||||
# Configure Docker to authenticate with Artifact Registry
|
||||
gcloud auth configure-docker us-central1-docker.pkg.dev --quiet
|
||||
|
||||
# Pull base images used by deploy-backend job (largest layers, most benefit from caching)
|
||||
echo "Pulling karaoke-backend-base image..."
|
||||
docker pull us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest || true
|
||||
|
||||
echo "Pulling karaoke-backend image..."
|
||||
docker pull us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend:latest || true
|
||||
|
||||
# Pull GCS emulator image used by backend-emulator-tests job
|
||||
echo "Pulling fake-gcs-server image..."
|
||||
docker pull fsouza/fake-gcs-server:latest || true
|
||||
|
||||
echo "Docker images pre-warmed"
|
||||
|
||||
echo "Setup complete!"
|
||||
@@ -0,0 +1,93 @@
|
||||
"""
|
||||
Centralized configuration for Pulumi infrastructure.
|
||||
|
||||
This module provides shared constants and configuration used across all infrastructure modules.
|
||||
"""
|
||||
|
||||
import pulumi
|
||||
import pulumi_gcp as gcp
|
||||
|
||||
# GCP project info - lazily loaded to work with Pulumi runtime
|
||||
_project = None
|
||||
|
||||
|
||||
def _get_project():
|
||||
"""Get GCP project info (lazy-loaded for Pulumi runtime compatibility)."""
|
||||
global _project
|
||||
if _project is None:
|
||||
_project = gcp.organizations.get_project()
|
||||
return _project
|
||||
|
||||
|
||||
# Project ID and number are accessed as properties
|
||||
# These will resolve correctly during Pulumi execution
|
||||
PROJECT_ID = "nomadkaraoke" # Hardcoded for now - matches existing behavior
|
||||
PROJECT_NUMBER = None # Set lazily when needed
|
||||
|
||||
|
||||
def get_project_number() -> str:
|
||||
"""Get the GCP project number."""
|
||||
return _get_project().number
|
||||
|
||||
# Region configuration
|
||||
REGION = "us-central1"
|
||||
ZONE = f"{REGION}-a"
|
||||
|
||||
# Pulumi config accessor
|
||||
config = pulumi.Config()
|
||||
|
||||
|
||||
class MachineTypes:
|
||||
"""Machine type configurations for GCE instances."""
|
||||
|
||||
GITHUB_RUNNER = "e2-standard-4" # 4 vCPU, 16GB RAM
|
||||
ENCODING_WORKER = "c4-standard-8" # 8 vCPU, Intel Granite Rapids 3.9 GHz
|
||||
FLACFETCH = "e2-small" # 0.5 vCPU, 2GB RAM
|
||||
|
||||
|
||||
class DiskSizes:
|
||||
"""Disk size configurations in GB."""
|
||||
|
||||
GITHUB_RUNNER = 200 # Large for Docker builds/caches
|
||||
ENCODING_WORKER = 100 # For temp video files during encoding
|
||||
FLACFETCH = 30 # For torrent storage
|
||||
|
||||
|
||||
class QueueConfigs:
|
||||
"""Cloud Tasks queue rate limiting configurations."""
|
||||
|
||||
class Audio:
|
||||
MAX_DISPATCHES_PER_SECOND = 10
|
||||
MAX_CONCURRENT_DISPATCHES = 50
|
||||
MAX_RETRY_DURATION = "1800s" # 30 min
|
||||
|
||||
class Lyrics:
|
||||
MAX_DISPATCHES_PER_SECOND = 10
|
||||
MAX_CONCURRENT_DISPATCHES = 50
|
||||
MAX_RETRY_DURATION = "1200s" # 20 min
|
||||
|
||||
class Screens:
|
||||
MAX_DISPATCHES_PER_SECOND = 50
|
||||
MAX_CONCURRENT_DISPATCHES = 100
|
||||
MAX_RETRY_DURATION = "300s" # 5 min
|
||||
|
||||
class Render:
|
||||
MAX_DISPATCHES_PER_SECOND = 5
|
||||
MAX_CONCURRENT_DISPATCHES = 20
|
||||
MAX_RETRY_DURATION = "3600s" # 60 min
|
||||
|
||||
class Video:
|
||||
MAX_DISPATCHES_PER_SECOND = 3
|
||||
MAX_CONCURRENT_DISPATCHES = 10
|
||||
MAX_RETRY_DURATION = "7200s" # 2 hours
|
||||
|
||||
|
||||
# Number of self-hosted GitHub Action runners
|
||||
NUM_GITHUB_RUNNERS = 20
|
||||
|
||||
# Google Drive folder ID for validator (public share folder)
|
||||
GDRIVE_FOLDER_ID = "1laRKAyxo0v817SstfM5XkpbWiNKNAMSX"
|
||||
|
||||
# GitHub repository for runner registration
|
||||
GITHUB_REPO_OWNER = "nomadkaraoke"
|
||||
GITHUB_REPO_NAME = "karaoke-gen"
|
||||
@@ -0,0 +1,370 @@
|
||||
# Phase 5: Encoding Worker Custom Machine Image
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The encoding worker VM currently takes **~10 minutes** to become ready after startup because the startup script:
|
||||
|
||||
1. Builds Python 3.13 from source (`./configure && make -j$(nproc)`) - ~7 minutes
|
||||
2. Downloads and installs FFmpeg static build - ~1 minute
|
||||
3. Installs system packages and fonts - ~1 minute
|
||||
4. Creates Python venv and installs dependencies - ~1 minute
|
||||
|
||||
This slow startup affects:
|
||||
- **VM restarts**: Any maintenance event or restart requires waiting 10 minutes
|
||||
- **Scaling**: Can't quickly add capacity if needed
|
||||
- **Development**: Testing infrastructure changes is slow
|
||||
|
||||
## Solution: Pre-baked Machine Image with Packer
|
||||
|
||||
Create a custom GCE machine image using [Packer](https://www.packer.io/) that pre-installs all dependencies. The startup script then only needs to:
|
||||
|
||||
1. Fetch API key from Secret Manager (~5 seconds)
|
||||
2. Download latest karaoke-gen wheel from GCS (~10 seconds)
|
||||
3. Start the systemd service (~5 seconds)
|
||||
|
||||
**Expected startup time: ~30 seconds** (vs current ~10 minutes)
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### 1. Create Packer Directory Structure
|
||||
|
||||
```
|
||||
infrastructure/packer/
|
||||
├── encoding-worker.pkr.hcl # Packer template
|
||||
├── scripts/
|
||||
│ └── provision.sh # Provisioning script
|
||||
└── README.md # Usage documentation
|
||||
```
|
||||
|
||||
### 2. Packer Template (`encoding-worker.pkr.hcl`)
|
||||
|
||||
```hcl
|
||||
packer {
|
||||
required_plugins {
|
||||
googlecompute = {
|
||||
source = "github.com/hashicorp/googlecompute"
|
||||
version = "~> 1.1"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
variable "project_id" {
|
||||
type = string
|
||||
default = "nomadkaraoke"
|
||||
}
|
||||
|
||||
variable "zone" {
|
||||
type = string
|
||||
default = "us-central1-a"
|
||||
}
|
||||
|
||||
variable "python_version" {
|
||||
type = string
|
||||
default = "3.13.1"
|
||||
}
|
||||
|
||||
source "googlecompute" "encoding-worker" {
|
||||
project_id = var.project_id
|
||||
zone = var.zone
|
||||
source_image_family = "debian-12"
|
||||
source_image_project_id = "debian-cloud"
|
||||
|
||||
# Build on same machine type as production for consistency
|
||||
machine_type = "c4-standard-8"
|
||||
disk_size = 100
|
||||
disk_type = "pd-ssd" # Use SSD for faster builds
|
||||
|
||||
image_name = "encoding-worker-{{timestamp}}"
|
||||
image_family = "encoding-worker"
|
||||
image_description = "Encoding worker with Python ${var.python_version}, FFmpeg, and fonts pre-installed"
|
||||
|
||||
ssh_username = "packer"
|
||||
|
||||
# Tags for firewall rules during build
|
||||
tags = ["packer-build"]
|
||||
}
|
||||
|
||||
build {
|
||||
sources = ["source.googlecompute.encoding-worker"]
|
||||
|
||||
# Upload provisioning script
|
||||
provisioner "file" {
|
||||
source = "scripts/provision.sh"
|
||||
destination = "/tmp/provision.sh"
|
||||
}
|
||||
|
||||
# Run provisioning
|
||||
provisioner "shell" {
|
||||
inline = [
|
||||
"chmod +x /tmp/provision.sh",
|
||||
"sudo PYTHON_VERSION=${var.python_version} /tmp/provision.sh"
|
||||
]
|
||||
}
|
||||
|
||||
# Clean up for smaller image
|
||||
provisioner "shell" {
|
||||
inline = [
|
||||
"sudo apt-get clean",
|
||||
"sudo rm -rf /var/lib/apt/lists/*",
|
||||
"sudo rm -rf /tmp/*",
|
||||
"sudo rm -rf /var/tmp/*"
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Provisioning Script (`scripts/provision.sh`)
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Provision encoding worker image with all dependencies pre-installed
|
||||
set -e
|
||||
|
||||
echo "=== Starting encoding worker image provisioning ==="
|
||||
|
||||
# Install system packages
|
||||
apt-get update
|
||||
apt-get install -y \
|
||||
docker.io \
|
||||
curl \
|
||||
git \
|
||||
xz-utils \
|
||||
build-essential \
|
||||
libssl-dev \
|
||||
zlib1g-dev \
|
||||
libbz2-dev \
|
||||
libreadline-dev \
|
||||
libsqlite3-dev \
|
||||
libncursesw5-dev \
|
||||
tk-dev \
|
||||
libxml2-dev \
|
||||
libxmlsec1-dev \
|
||||
libffi-dev \
|
||||
liblzma-dev
|
||||
|
||||
# Install fonts for ASS subtitle rendering
|
||||
apt-get install -y fonts-noto fonts-noto-cjk fontconfig
|
||||
fc-cache -fv
|
||||
|
||||
# Enable Docker
|
||||
systemctl enable docker
|
||||
|
||||
# Install optimized static FFmpeg build (John Van Sickle)
|
||||
echo "Installing FFmpeg..."
|
||||
curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz -o /tmp/ffmpeg.tar.xz
|
||||
tar -xf /tmp/ffmpeg.tar.xz -C /tmp
|
||||
cp /tmp/ffmpeg-*-amd64-static/ffmpeg /usr/local/bin/
|
||||
cp /tmp/ffmpeg-*-amd64-static/ffprobe /usr/local/bin/
|
||||
chmod +x /usr/local/bin/ffmpeg /usr/local/bin/ffprobe
|
||||
rm -rf /tmp/ffmpeg*
|
||||
ffmpeg -version
|
||||
|
||||
# Build and install Python from source
|
||||
echo "Building Python ${PYTHON_VERSION}..."
|
||||
cd /tmp
|
||||
curl -L "https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tar.xz" -o python.tar.xz
|
||||
tar -xf python.tar.xz
|
||||
cd "Python-${PYTHON_VERSION}"
|
||||
./configure --prefix=/opt/python313 --enable-optimizations --with-lto
|
||||
make -j$(nproc)
|
||||
make install
|
||||
rm -rf /tmp/python.tar.xz /tmp/Python-${PYTHON_VERSION}
|
||||
/opt/python313/bin/python3.13 --version
|
||||
echo "Python ${PYTHON_VERSION} installed at /opt/python313"
|
||||
|
||||
# Create working directory and venv
|
||||
mkdir -p /opt/encoding-worker
|
||||
cd /opt/encoding-worker
|
||||
/opt/python313/bin/python3.13 -m venv venv
|
||||
|
||||
# Install base Python dependencies (wheel will add more at runtime)
|
||||
source venv/bin/activate
|
||||
pip install --upgrade pip
|
||||
pip install fastapi uvicorn google-cloud-storage aiofiles aiohttp
|
||||
|
||||
# Create systemd service template (API key filled at runtime)
|
||||
cat > /etc/systemd/system/encoding-worker.service << 'SYSTEMD'
|
||||
[Unit]
|
||||
Description=Encoding Worker HTTP API Service
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
WorkingDirectory=/opt/encoding-worker
|
||||
ExecStartPre=/opt/encoding-worker/startup.sh
|
||||
ExecStart=/opt/encoding-worker/venv/bin/uvicorn backend.services.gce_encoding.main:app --host 0.0.0.0 --port 8080
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
Environment="GOOGLE_CLOUD_PROJECT=nomadkaraoke"
|
||||
EnvironmentFile=/opt/encoding-worker/env
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
SYSTEMD
|
||||
|
||||
# Create startup helper script (runs before service starts)
|
||||
cat > /opt/encoding-worker/startup.sh << 'STARTUP'
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
# Fetch API key from Secret Manager
|
||||
ENCODING_API_KEY=$(gcloud secrets versions access latest --secret=encoding-worker-api-key 2>/dev/null || echo "")
|
||||
echo "ENCODING_API_KEY=${ENCODING_API_KEY}" > /opt/encoding-worker/env
|
||||
|
||||
# Download and install latest karaoke-gen wheel
|
||||
gsutil cp "gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-*.whl" /tmp/ 2>/dev/null || true
|
||||
WHEEL_FILE=$(ls -t /tmp/karaoke_gen-*.whl 2>/dev/null | head -1)
|
||||
if [ -n "$WHEEL_FILE" ]; then
|
||||
/opt/encoding-worker/venv/bin/pip install --upgrade --quiet "$WHEEL_FILE"
|
||||
fi
|
||||
STARTUP
|
||||
|
||||
chmod +x /opt/encoding-worker/startup.sh
|
||||
|
||||
# Enable service (will start on boot)
|
||||
systemctl daemon-reload
|
||||
systemctl enable encoding-worker
|
||||
|
||||
echo "=== Provisioning complete ==="
|
||||
```
|
||||
|
||||
### 4. Update Pulumi to Use Custom Image
|
||||
|
||||
In `infrastructure/compute/encoding_worker_vm.py`, change:
|
||||
|
||||
```python
|
||||
# Before (current)
|
||||
boot_disk=compute.InstanceBootDiskArgs(
|
||||
initialize_params=compute.InstanceBootDiskInitializeParamsArgs(
|
||||
image="debian-cloud/debian-12",
|
||||
size=DiskSizes.ENCODING_WORKER,
|
||||
type="hyperdisk-balanced",
|
||||
),
|
||||
),
|
||||
|
||||
# After (with custom image)
|
||||
boot_disk=compute.InstanceBootDiskArgs(
|
||||
initialize_params=compute.InstanceBootDiskInitializeParamsArgs(
|
||||
image=f"projects/{PROJECT_ID}/global/images/family/encoding-worker",
|
||||
size=DiskSizes.ENCODING_WORKER,
|
||||
type="hyperdisk-balanced",
|
||||
),
|
||||
),
|
||||
```
|
||||
|
||||
### 5. Simplified Startup Script
|
||||
|
||||
Replace the 700-line `encoding_worker.sh` with a minimal script:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Encoding worker startup script (for custom image)
|
||||
# Most dependencies are pre-installed in the image.
|
||||
# This script just starts the service.
|
||||
|
||||
set -e
|
||||
exec > >(tee /var/log/encoding-worker-startup.log) 2>&1
|
||||
echo "Starting encoding worker at $(date)"
|
||||
|
||||
# Start Docker (in case it's not running)
|
||||
systemctl start docker
|
||||
|
||||
# Start the encoding worker service
|
||||
# The service's ExecStartPre handles:
|
||||
# - Fetching API key from Secret Manager
|
||||
# - Installing latest karaoke-gen wheel
|
||||
systemctl start encoding-worker
|
||||
|
||||
echo "Encoding worker started at $(date)"
|
||||
```
|
||||
|
||||
### 6. CI Integration (Optional)
|
||||
|
||||
Add a GitHub Actions workflow to rebuild the image when Packer files change:
|
||||
|
||||
```yaml
|
||||
# .github/workflows/build-encoding-worker-image.yml
|
||||
name: Build Encoding Worker Image
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'infrastructure/packer/**'
|
||||
workflow_dispatch: # Manual trigger
|
||||
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Authenticate to Google Cloud
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: ${{ secrets.WIF_PROVIDER }}
|
||||
service_account: github-actions-deployer@nomadkaraoke.iam.gserviceaccount.com
|
||||
|
||||
- name: Setup Packer
|
||||
uses: hashicorp/setup-packer@main
|
||||
with:
|
||||
version: latest
|
||||
|
||||
- name: Build Image
|
||||
working-directory: infrastructure/packer
|
||||
run: |
|
||||
packer init encoding-worker.pkr.hcl
|
||||
packer build encoding-worker.pkr.hcl
|
||||
```
|
||||
|
||||
## Execution Steps
|
||||
|
||||
1. **Create the Packer files** in `infrastructure/packer/`
|
||||
|
||||
2. **Build the image locally** (one-time, or via CI):
|
||||
```bash
|
||||
cd infrastructure/packer
|
||||
packer init encoding-worker.pkr.hcl
|
||||
packer build encoding-worker.pkr.hcl
|
||||
```
|
||||
|
||||
3. **Update Pulumi** to use the new image family
|
||||
|
||||
4. **Simplify the startup script** to just start the service
|
||||
|
||||
5. **Run `pulumi up`** to deploy with the new image
|
||||
|
||||
6. **Verify** the VM starts in ~30 seconds instead of ~10 minutes
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues occur:
|
||||
1. Change the image back to `debian-cloud/debian-12` in Pulumi
|
||||
2. Restore the full startup script
|
||||
3. Run `pulumi up`
|
||||
|
||||
The old startup script is preserved in git history and can be restored.
|
||||
|
||||
## Benefits
|
||||
|
||||
| Metric | Before | After |
|
||||
|--------|--------|-------|
|
||||
| Startup time | ~10 min | ~30 sec |
|
||||
| Python build | Every start | Never (pre-baked) |
|
||||
| FFmpeg install | Every start | Never (pre-baked) |
|
||||
| Image rebuild needed | N/A | Only when deps change |
|
||||
|
||||
## Maintenance
|
||||
|
||||
Rebuild the image when:
|
||||
- Python version needs updating
|
||||
- FFmpeg version needs updating
|
||||
- New system packages required
|
||||
- Font packages change
|
||||
|
||||
The karaoke-gen wheel is still downloaded at runtime, so application code updates don't require image rebuilds.
|
||||
@@ -0,0 +1,25 @@
|
||||
"""
|
||||
Infrastructure modules for Pulumi resources.
|
||||
|
||||
Each module is responsible for a specific category of cloud resources.
|
||||
"""
|
||||
|
||||
from . import database
|
||||
from . import storage
|
||||
from . import artifact_registry
|
||||
from . import secrets
|
||||
from . import cloud_tasks
|
||||
from . import cloud_run
|
||||
from . import monitoring
|
||||
from . import iam
|
||||
|
||||
__all__ = [
|
||||
"database",
|
||||
"storage",
|
||||
"artifact_registry",
|
||||
"secrets",
|
||||
"cloud_tasks",
|
||||
"cloud_run",
|
||||
"monitoring",
|
||||
"iam",
|
||||
]
|
||||
@@ -0,0 +1,42 @@
|
||||
"""
|
||||
Artifact Registry resources.
|
||||
|
||||
Manages the Docker repository for backend container images.
|
||||
"""
|
||||
|
||||
import pulumi
|
||||
import pulumi_gcp as gcp
|
||||
from pulumi_gcp import artifactregistry
|
||||
|
||||
from config import PROJECT_ID, REGION
|
||||
|
||||
|
||||
def create_repository() -> artifactregistry.Repository:
|
||||
"""
|
||||
Create the Artifact Registry repository for Docker images.
|
||||
|
||||
Returns:
|
||||
artifactregistry.Repository: The created repository resource.
|
||||
"""
|
||||
artifact_repo = artifactregistry.Repository(
|
||||
"karaoke-artifact-repo",
|
||||
repository_id="karaoke-repo",
|
||||
location=REGION,
|
||||
format="DOCKER",
|
||||
description="Docker repository for karaoke backend images",
|
||||
)
|
||||
|
||||
return artifact_repo
|
||||
|
||||
|
||||
def get_repo_url(repo: artifactregistry.Repository) -> pulumi.Output[str]:
|
||||
"""
|
||||
Get the full URL for the repository.
|
||||
|
||||
Args:
|
||||
repo: The artifact registry repository.
|
||||
|
||||
Returns:
|
||||
pulumi.Output[str]: The repository URL (e.g., us-central1-docker.pkg.dev/project/repo)
|
||||
"""
|
||||
return repo.name.apply(lambda name: f"{REGION}-docker.pkg.dev/{PROJECT_ID}/karaoke-repo")
|
||||
@@ -0,0 +1,88 @@
|
||||
"""
|
||||
Cloud Run resources.
|
||||
|
||||
Manages Cloud Run domain mapping and Cloud Run Jobs.
|
||||
"""
|
||||
|
||||
import pulumi
|
||||
import pulumi_gcp as gcp
|
||||
from pulumi_gcp import cloudrun, cloudrunv2
|
||||
|
||||
from config import PROJECT_ID, REGION
|
||||
|
||||
|
||||
def create_domain_mapping() -> cloudrun.DomainMapping:
|
||||
"""
|
||||
Create the Cloud Run domain mapping for api.nomadkaraoke.com.
|
||||
|
||||
Returns:
|
||||
cloudrun.DomainMapping: The domain mapping resource.
|
||||
"""
|
||||
domain_mapping = cloudrun.DomainMapping(
|
||||
"karaoke-backend-domain",
|
||||
location=REGION,
|
||||
name="api.nomadkaraoke.com",
|
||||
metadata=cloudrun.DomainMappingMetadataArgs(
|
||||
namespace=PROJECT_ID,
|
||||
),
|
||||
spec=cloudrun.DomainMappingSpecArgs(
|
||||
route_name="karaoke-backend",
|
||||
),
|
||||
)
|
||||
|
||||
return domain_mapping
|
||||
|
||||
|
||||
def create_video_encoding_job(
|
||||
bucket: gcp.storage.Bucket,
|
||||
service_account: gcp.serviceaccount.Account,
|
||||
) -> cloudrunv2.Job:
|
||||
"""
|
||||
Create the Cloud Run Job for video encoding.
|
||||
|
||||
Cloud Run Jobs provide up to 24-hour execution time for long-running video encoding.
|
||||
This is used when video encoding might exceed the 30-minute Cloud Tasks timeout.
|
||||
|
||||
Args:
|
||||
bucket: The GCS bucket for job artifacts.
|
||||
service_account: The service account to run the job.
|
||||
|
||||
Returns:
|
||||
cloudrunv2.Job: The Cloud Run Job resource.
|
||||
"""
|
||||
video_encoding_job = cloudrunv2.Job(
|
||||
"video-encoding-job",
|
||||
name="video-encoding-job",
|
||||
location=REGION,
|
||||
template=cloudrunv2.JobTemplateArgs(
|
||||
template=cloudrunv2.JobTemplateTemplateArgs(
|
||||
containers=[
|
||||
cloudrunv2.JobTemplateTemplateContainerArgs(
|
||||
image=f"{REGION}-docker.pkg.dev/{PROJECT_ID}/karaoke-repo/karaoke-backend:latest",
|
||||
args=["python", "-m", "backend.workers.video_worker"],
|
||||
resources=cloudrunv2.JobTemplateTemplateContainerResourcesArgs(
|
||||
limits={
|
||||
"cpu": "4",
|
||||
"memory": "8Gi",
|
||||
},
|
||||
),
|
||||
envs=[
|
||||
cloudrunv2.JobTemplateTemplateContainerEnvArgs(
|
||||
name="GOOGLE_CLOUD_PROJECT",
|
||||
value=PROJECT_ID,
|
||||
),
|
||||
cloudrunv2.JobTemplateTemplateContainerEnvArgs(
|
||||
name="GCS_BUCKET_NAME",
|
||||
value=bucket.name,
|
||||
),
|
||||
],
|
||||
)
|
||||
],
|
||||
service_account=service_account.email,
|
||||
timeout="3600s", # 1 hour max per task
|
||||
max_retries=2,
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
return video_encoding_job
|
||||
@@ -0,0 +1,133 @@
|
||||
"""
|
||||
Cloud Tasks resources.
|
||||
|
||||
Manages task queues for asynchronous worker processing.
|
||||
"""
|
||||
|
||||
import pulumi
|
||||
import pulumi_gcp as gcp
|
||||
from pulumi_gcp import cloudtasks
|
||||
|
||||
from config import REGION, QueueConfigs
|
||||
|
||||
|
||||
def create_queues() -> dict[str, cloudtasks.Queue]:
|
||||
"""
|
||||
Create all Cloud Tasks queues.
|
||||
|
||||
These queues provide guaranteed delivery and horizontal scaling for worker tasks.
|
||||
Workers are triggered via Cloud Tasks instead of BackgroundTasks, ensuring:
|
||||
- Tasks survive container restarts
|
||||
- Each task gets dedicated resources
|
||||
- Automatic retries on failure
|
||||
- Rate limiting to protect external APIs
|
||||
|
||||
Returns:
|
||||
dict: Dictionary mapping queue names to Queue resources.
|
||||
"""
|
||||
queues = {}
|
||||
|
||||
# Audio worker queue - calls Modal API for audio separation
|
||||
queues["audio-worker-queue"] = cloudtasks.Queue(
|
||||
"audio-worker-queue",
|
||||
name="audio-worker-queue",
|
||||
location=REGION,
|
||||
rate_limits=cloudtasks.QueueRateLimitsArgs(
|
||||
max_dispatches_per_second=QueueConfigs.Audio.MAX_DISPATCHES_PER_SECOND,
|
||||
max_concurrent_dispatches=QueueConfigs.Audio.MAX_CONCURRENT_DISPATCHES,
|
||||
),
|
||||
retry_config=cloudtasks.QueueRetryConfigArgs(
|
||||
max_attempts=3,
|
||||
min_backoff="10s",
|
||||
max_backoff="300s",
|
||||
max_retry_duration=QueueConfigs.Audio.MAX_RETRY_DURATION,
|
||||
),
|
||||
)
|
||||
|
||||
# Lyrics worker queue - calls AudioShake API for transcription
|
||||
queues["lyrics-worker-queue"] = cloudtasks.Queue(
|
||||
"lyrics-worker-queue",
|
||||
name="lyrics-worker-queue",
|
||||
location=REGION,
|
||||
rate_limits=cloudtasks.QueueRateLimitsArgs(
|
||||
max_dispatches_per_second=QueueConfigs.Lyrics.MAX_DISPATCHES_PER_SECOND,
|
||||
max_concurrent_dispatches=QueueConfigs.Lyrics.MAX_CONCURRENT_DISPATCHES,
|
||||
),
|
||||
retry_config=cloudtasks.QueueRetryConfigArgs(
|
||||
max_attempts=3,
|
||||
min_backoff="10s",
|
||||
max_backoff="300s",
|
||||
max_retry_duration=QueueConfigs.Lyrics.MAX_RETRY_DURATION,
|
||||
),
|
||||
)
|
||||
|
||||
# Screens worker queue - generates title/end screens (fast, CPU-light)
|
||||
queues["screens-worker-queue"] = cloudtasks.Queue(
|
||||
"screens-worker-queue",
|
||||
name="screens-worker-queue",
|
||||
location=REGION,
|
||||
rate_limits=cloudtasks.QueueRateLimitsArgs(
|
||||
max_dispatches_per_second=QueueConfigs.Screens.MAX_DISPATCHES_PER_SECOND,
|
||||
max_concurrent_dispatches=QueueConfigs.Screens.MAX_CONCURRENT_DISPATCHES,
|
||||
),
|
||||
retry_config=cloudtasks.QueueRetryConfigArgs(
|
||||
max_attempts=3,
|
||||
min_backoff="5s",
|
||||
max_backoff="60s",
|
||||
max_retry_duration=QueueConfigs.Screens.MAX_RETRY_DURATION,
|
||||
),
|
||||
)
|
||||
|
||||
# Render worker queue - LyricsTranscriber + FFmpeg (CPU-intensive)
|
||||
queues["render-worker-queue"] = cloudtasks.Queue(
|
||||
"render-worker-queue",
|
||||
name="render-worker-queue",
|
||||
location=REGION,
|
||||
rate_limits=cloudtasks.QueueRateLimitsArgs(
|
||||
max_dispatches_per_second=QueueConfigs.Render.MAX_DISPATCHES_PER_SECOND,
|
||||
max_concurrent_dispatches=QueueConfigs.Render.MAX_CONCURRENT_DISPATCHES,
|
||||
),
|
||||
retry_config=cloudtasks.QueueRetryConfigArgs(
|
||||
max_attempts=2,
|
||||
min_backoff="30s",
|
||||
max_backoff="300s",
|
||||
max_retry_duration=QueueConfigs.Render.MAX_RETRY_DURATION,
|
||||
),
|
||||
)
|
||||
|
||||
# Video worker queue - final encoding (very CPU-intensive, longest running)
|
||||
queues["video-worker-queue"] = cloudtasks.Queue(
|
||||
"video-worker-queue",
|
||||
name="video-worker-queue",
|
||||
location=REGION,
|
||||
rate_limits=cloudtasks.QueueRateLimitsArgs(
|
||||
max_dispatches_per_second=QueueConfigs.Video.MAX_DISPATCHES_PER_SECOND,
|
||||
max_concurrent_dispatches=QueueConfigs.Video.MAX_CONCURRENT_DISPATCHES,
|
||||
),
|
||||
retry_config=cloudtasks.QueueRetryConfigArgs(
|
||||
max_attempts=2,
|
||||
min_backoff="60s",
|
||||
max_backoff="600s",
|
||||
max_retry_duration=QueueConfigs.Video.MAX_RETRY_DURATION,
|
||||
),
|
||||
)
|
||||
|
||||
# Idle reminder queue - delayed tasks for sending reminder emails
|
||||
# Tasks are scheduled with a 5-minute delay when jobs enter blocking states
|
||||
queues["idle-reminder-queue"] = cloudtasks.Queue(
|
||||
"idle-reminder-queue",
|
||||
name="idle-reminder-queue",
|
||||
location=REGION,
|
||||
rate_limits=cloudtasks.QueueRateLimitsArgs(
|
||||
max_dispatches_per_second=10, # Email sending is fast
|
||||
max_concurrent_dispatches=50,
|
||||
),
|
||||
retry_config=cloudtasks.QueueRetryConfigArgs(
|
||||
max_attempts=3,
|
||||
min_backoff="10s",
|
||||
max_backoff="60s",
|
||||
max_retry_duration="600s", # 10 min total
|
||||
),
|
||||
)
|
||||
|
||||
return queues
|
||||
@@ -0,0 +1,173 @@
|
||||
"""
|
||||
Firestore database resources.
|
||||
|
||||
Manages the Firestore database, composite indexes, and TTL configuration.
|
||||
"""
|
||||
|
||||
import pulumi
|
||||
import pulumi_gcp as gcp
|
||||
from pulumi_gcp import firestore
|
||||
|
||||
from config import PROJECT_ID
|
||||
|
||||
|
||||
def create_database() -> dict:
|
||||
"""
|
||||
Create Firestore database and all composite indexes.
|
||||
|
||||
Returns:
|
||||
dict: Dictionary containing all created resources for export.
|
||||
"""
|
||||
resources = {}
|
||||
|
||||
# ==================== Firestore Database ====================
|
||||
firestore_db = firestore.Database(
|
||||
"karaoke-firestore",
|
||||
project=PROJECT_ID,
|
||||
name="(default)",
|
||||
location_id="us-central1",
|
||||
type="FIRESTORE_NATIVE",
|
||||
concurrency_mode="PESSIMISTIC",
|
||||
app_engine_integration_mode="DISABLED",
|
||||
)
|
||||
resources["firestore_db"] = firestore_db
|
||||
|
||||
# ==================== Firestore Composite Indexes ====================
|
||||
# These are required for queries that filter/sort on multiple fields.
|
||||
|
||||
# Jobs: query by user_email, order by created_at (for user job lists)
|
||||
resources["firestore_index_jobs_user_email"] = firestore.Index(
|
||||
"firestore-index-jobs-user-email",
|
||||
project=PROJECT_ID,
|
||||
database=firestore_db.name,
|
||||
collection="jobs",
|
||||
fields=[
|
||||
firestore.IndexFieldArgs(field_path="user_email", order="ASCENDING"),
|
||||
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
|
||||
],
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
# Jobs: query by status, order by created_at (for admin/system job lists)
|
||||
resources["firestore_index_jobs_status"] = firestore.Index(
|
||||
"firestore-index-jobs-status",
|
||||
project=PROJECT_ID,
|
||||
database=firestore_db.name,
|
||||
collection="jobs",
|
||||
fields=[
|
||||
firestore.IndexFieldArgs(field_path="status", order="ASCENDING"),
|
||||
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
|
||||
],
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
# Sessions: query by user_email + is_active, order by created_at
|
||||
resources["firestore_index_sessions_active"] = firestore.Index(
|
||||
"firestore-index-sessions-active",
|
||||
project=PROJECT_ID,
|
||||
database=firestore_db.name,
|
||||
collection="sessions",
|
||||
fields=[
|
||||
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
|
||||
firestore.IndexFieldArgs(field_path="user_email", order="ASCENDING"),
|
||||
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
|
||||
],
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
# Magic links: query by email, order by created_at
|
||||
resources["firestore_index_magic_links"] = firestore.Index(
|
||||
"firestore-index-magic-links",
|
||||
project=PROJECT_ID,
|
||||
database=firestore_db.name,
|
||||
collection="magic_links",
|
||||
fields=[
|
||||
firestore.IndexFieldArgs(field_path="email", order="ASCENDING"),
|
||||
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
|
||||
],
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
# Gen Users: query by is_active, order by created_at
|
||||
resources["firestore_index_gen_users_active_created"] = firestore.Index(
|
||||
"firestore-index-gen-users-active-created",
|
||||
project=PROJECT_ID,
|
||||
database=firestore_db.name,
|
||||
collection="gen_users",
|
||||
fields=[
|
||||
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
|
||||
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
|
||||
],
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
# Gen Users: query by is_active, order by last_login_at
|
||||
resources["firestore_index_gen_users_active_login"] = firestore.Index(
|
||||
"firestore-index-gen-users-active-login",
|
||||
project=PROJECT_ID,
|
||||
database=firestore_db.name,
|
||||
collection="gen_users",
|
||||
fields=[
|
||||
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
|
||||
firestore.IndexFieldArgs(field_path="last_login_at", order="DESCENDING"),
|
||||
],
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
# Gen Users: query by is_active, order by credits
|
||||
resources["firestore_index_gen_users_active_credits"] = firestore.Index(
|
||||
"firestore-index-gen-users-active-credits",
|
||||
project=PROJECT_ID,
|
||||
database=firestore_db.name,
|
||||
collection="gen_users",
|
||||
fields=[
|
||||
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
|
||||
firestore.IndexFieldArgs(field_path="credits", order="DESCENDING"),
|
||||
],
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
# Gen Users: query by is_active, order by email
|
||||
resources["firestore_index_gen_users_active_email"] = firestore.Index(
|
||||
"firestore-index-gen-users-active-email",
|
||||
project=PROJECT_ID,
|
||||
database=firestore_db.name,
|
||||
collection="gen_users",
|
||||
fields=[
|
||||
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
|
||||
firestore.IndexFieldArgs(field_path="email", order="ASCENDING"),
|
||||
],
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
# ==================== Firestore TTL Field Configuration ====================
|
||||
# TTL policies automatically delete documents after expiration.
|
||||
|
||||
# Worker logs TTL: logs subcollection documents expire after 30 days
|
||||
resources["firestore_field_logs_ttl"] = firestore.Field(
|
||||
"firestore-field-logs-ttl",
|
||||
project=PROJECT_ID,
|
||||
database=firestore_db.name,
|
||||
collection="logs", # Subcollection name (applies to all jobs/{job_id}/logs)
|
||||
field="ttl_expiry",
|
||||
ttl_config={}, # Empty block enables TTL based on ttl_expiry field
|
||||
index_config={}, # Disable indexing on TTL field
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
# ==================== Firestore Indexes for Logs Subcollection ====================
|
||||
|
||||
# Logs: query by worker, order by timestamp
|
||||
resources["firestore_index_logs_worker_timestamp"] = firestore.Index(
|
||||
"firestore-index-logs-worker-timestamp",
|
||||
project=PROJECT_ID,
|
||||
database=firestore_db.name,
|
||||
collection="logs", # Subcollection name
|
||||
fields=[
|
||||
firestore.IndexFieldArgs(field_path="worker", order="ASCENDING"),
|
||||
firestore.IndexFieldArgs(field_path="timestamp", order="ASCENDING"),
|
||||
],
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
return resources
|
||||
@@ -0,0 +1,21 @@
|
||||
"""
|
||||
IAM modules for service accounts and role bindings.
|
||||
|
||||
Organized by purpose:
|
||||
- backend_sa: Main karaoke-backend service account
|
||||
- github_actions_sa: GitHub Actions deployer with Workload Identity Federation
|
||||
- claude_automation_sa: Claude Code automation service account
|
||||
- worker_sas: Service accounts for VMs (flacfetch, encoding, gdrive, runners)
|
||||
"""
|
||||
|
||||
from . import backend_sa
|
||||
from . import github_actions_sa
|
||||
from . import claude_automation_sa
|
||||
from . import worker_sas
|
||||
|
||||
__all__ = [
|
||||
"backend_sa",
|
||||
"github_actions_sa",
|
||||
"claude_automation_sa",
|
||||
"worker_sas",
|
||||
]
|
||||
@@ -0,0 +1,132 @@
|
||||
"""
|
||||
Backend service account and IAM bindings.
|
||||
|
||||
Manages the karaoke-backend service account used by Cloud Run and its associated
|
||||
IAM roles for accessing Firestore, GCS, Secret Manager, and other services.
|
||||
"""
|
||||
|
||||
import pulumi
|
||||
import pulumi_gcp as gcp
|
||||
from pulumi_gcp import serviceaccount
|
||||
|
||||
from config import PROJECT_ID, get_project_number
|
||||
|
||||
|
||||
def create_backend_service_account() -> serviceaccount.Account:
|
||||
"""
|
||||
Create the main backend service account.
|
||||
|
||||
Returns:
|
||||
serviceaccount.Account: The service account resource.
|
||||
"""
|
||||
return serviceaccount.Account(
|
||||
"karaoke-backend-sa",
|
||||
account_id="karaoke-backend",
|
||||
display_name="Karaoke Backend Service Account",
|
||||
)
|
||||
|
||||
|
||||
def grant_backend_permissions(
|
||||
service_account: serviceaccount.Account,
|
||||
) -> dict:
|
||||
"""
|
||||
Grant all necessary permissions to the backend service account.
|
||||
|
||||
Args:
|
||||
service_account: The backend service account.
|
||||
|
||||
Returns:
|
||||
dict: Dictionary of IAM binding resources.
|
||||
"""
|
||||
bindings = {}
|
||||
|
||||
# Grant Firestore permissions
|
||||
bindings["firestore_iam"] = gcp.projects.IAMMember(
|
||||
"karaoke-backend-firestore-access",
|
||||
project=PROJECT_ID,
|
||||
role="roles/datastore.user",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Storage permissions
|
||||
bindings["storage_iam"] = gcp.projects.IAMMember(
|
||||
"karaoke-backend-storage-access",
|
||||
project=PROJECT_ID,
|
||||
role="roles/storage.objectAdmin",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Secret Manager read permissions
|
||||
bindings["secrets_read_iam"] = gcp.projects.IAMMember(
|
||||
"karaoke-backend-secrets-access",
|
||||
project=PROJECT_ID,
|
||||
role="roles/secretmanager.secretAccessor",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Secret Manager write permissions (for OAuth token refresh/device auth)
|
||||
bindings["secrets_write_iam"] = gcp.projects.IAMMember(
|
||||
"karaoke-backend-secrets-write",
|
||||
project=PROJECT_ID,
|
||||
role="roles/secretmanager.secretVersionManager",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Cloud Trace write permissions (for OpenTelemetry tracing)
|
||||
bindings["cloud_trace_iam"] = gcp.projects.IAMMember(
|
||||
"karaoke-backend-cloudtrace-agent",
|
||||
project=PROJECT_ID,
|
||||
role="roles/cloudtrace.agent",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Vertex AI User permissions (for Gemini agentic AI correction)
|
||||
bindings["vertex_ai_iam"] = gcp.projects.IAMMember(
|
||||
"karaoke-backend-vertexai-user",
|
||||
project=PROJECT_ID,
|
||||
role="roles/aiplatform.user",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant backend service account permission to enqueue Cloud Tasks
|
||||
bindings["cloud_tasks_enqueuer"] = gcp.projects.IAMMember(
|
||||
"karaoke-backend-cloudtasks-enqueuer",
|
||||
project=PROJECT_ID,
|
||||
role="roles/cloudtasks.enqueuer",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Service Account User - to act as itself for Cloud Tasks OIDC
|
||||
bindings["act_as_self"] = serviceaccount.IAMBinding(
|
||||
"karaoke-backend-act-as-self",
|
||||
service_account_id=service_account.name,
|
||||
role="roles/iam.serviceAccountUser",
|
||||
members=[service_account.email.apply(lambda email: f"serviceAccount:{email}")],
|
||||
)
|
||||
|
||||
# Cloud Run Jobs Developer - run encoding jobs
|
||||
bindings["run_jobs_developer"] = gcp.projects.IAMMember(
|
||||
"karaoke-backend-run-jobs-developer",
|
||||
project=PROJECT_ID,
|
||||
role="roles/run.developer",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
return bindings
|
||||
|
||||
|
||||
def grant_cloud_tasks_invoker_permission() -> gcp.projects.IAMMember:
|
||||
"""
|
||||
Grant Cloud Tasks service agent permission to invoke Cloud Run.
|
||||
|
||||
This allows Cloud Tasks to authenticate when calling Cloud Run endpoints.
|
||||
|
||||
Returns:
|
||||
gcp.projects.IAMMember: The IAM binding resource.
|
||||
"""
|
||||
return gcp.projects.IAMMember(
|
||||
"cloud-tasks-invoker",
|
||||
project=PROJECT_ID,
|
||||
role="roles/run.invoker",
|
||||
member=f"serviceAccount:service-{get_project_number()}@gcp-sa-cloudtasks.iam.gserviceaccount.com",
|
||||
)
|
||||
@@ -0,0 +1,131 @@
|
||||
"""
|
||||
Claude Code automation service account.
|
||||
|
||||
Manages the service account used by Claude Code for read-only access
|
||||
to production resources for debugging and monitoring.
|
||||
"""
|
||||
|
||||
import pulumi
|
||||
import pulumi_gcp as gcp
|
||||
from pulumi_gcp import serviceaccount
|
||||
|
||||
from config import PROJECT_ID
|
||||
|
||||
|
||||
def create_claude_automation_service_account() -> serviceaccount.Account:
|
||||
"""
|
||||
Create the Claude Code automation service account.
|
||||
|
||||
This service account provides read-only access to production resources
|
||||
for debugging and monitoring purposes.
|
||||
|
||||
Returns:
|
||||
serviceaccount.Account: The service account resource.
|
||||
"""
|
||||
return serviceaccount.Account(
|
||||
"claude-automation-sa",
|
||||
account_id="claude-automation",
|
||||
display_name="Claude Code Automation",
|
||||
description="Service account for Claude Code CLI with read-only GCP access",
|
||||
)
|
||||
|
||||
|
||||
def grant_claude_automation_permissions(
|
||||
service_account: serviceaccount.Account,
|
||||
) -> dict:
|
||||
"""
|
||||
Grant read-only permissions to the Claude automation service account.
|
||||
|
||||
All permissions are read-only/viewer level to prevent accidental modifications.
|
||||
|
||||
Args:
|
||||
service_account: The Claude automation service account.
|
||||
|
||||
Returns:
|
||||
dict: Dictionary of IAM binding resources.
|
||||
"""
|
||||
bindings = {}
|
||||
|
||||
# Grant Cloud Run viewer permissions (view services, revisions, status)
|
||||
bindings["run_viewer"] = gcp.projects.IAMMember(
|
||||
"claude-automation-run-viewer",
|
||||
project=PROJECT_ID,
|
||||
role="roles/run.viewer",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Logging viewer permissions (read logs)
|
||||
bindings["logging_viewer"] = gcp.projects.IAMMember(
|
||||
"claude-automation-logging-viewer",
|
||||
project=PROJECT_ID,
|
||||
role="roles/logging.viewer",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Monitoring viewer permissions (read metrics, dashboards)
|
||||
bindings["monitoring_viewer"] = gcp.projects.IAMMember(
|
||||
"claude-automation-monitoring-viewer",
|
||||
project=PROJECT_ID,
|
||||
role="roles/monitoring.viewer",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Cloud Build viewer permissions (view build status, logs)
|
||||
bindings["cloudbuild_viewer"] = gcp.projects.IAMMember(
|
||||
"claude-automation-cloudbuild-viewer",
|
||||
project=PROJECT_ID,
|
||||
role="roles/cloudbuild.builds.viewer",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Cloud Trace viewer permissions (view distributed traces)
|
||||
bindings["cloudtrace_viewer"] = gcp.projects.IAMMember(
|
||||
"claude-automation-cloudtrace-viewer",
|
||||
project=PROJECT_ID,
|
||||
role="roles/cloudtrace.user",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Cloud Tasks viewer permissions (view queue status)
|
||||
bindings["cloudtasks_viewer"] = gcp.projects.IAMMember(
|
||||
"claude-automation-cloudtasks-viewer",
|
||||
project=PROJECT_ID,
|
||||
role="roles/cloudtasks.viewer",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Storage viewer permissions (list buckets, view metadata - not object content)
|
||||
bindings["storage_viewer"] = gcp.projects.IAMMember(
|
||||
"claude-automation-storage-viewer",
|
||||
project=PROJECT_ID,
|
||||
role="roles/storage.objectViewer",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
return bindings
|
||||
|
||||
|
||||
def grant_impersonation_permission(
|
||||
service_account: serviceaccount.Account,
|
||||
) -> serviceaccount.IAMBinding:
|
||||
"""
|
||||
Allow users to impersonate the Claude automation service account.
|
||||
|
||||
This enables using `gcloud auth application-default login --impersonate-service-account`
|
||||
for long-lived credentials without downloadable keys.
|
||||
|
||||
Args:
|
||||
service_account: The Claude automation service account.
|
||||
|
||||
Returns:
|
||||
serviceaccount.IAMBinding: The IAM binding resource.
|
||||
"""
|
||||
return serviceaccount.IAMBinding(
|
||||
"claude-automation-impersonation",
|
||||
service_account_id=service_account.name,
|
||||
role="roles/iam.serviceAccountTokenCreator",
|
||||
members=[
|
||||
"user:andrew@beveridge.uk",
|
||||
"user:admin@nomadkaraoke.com",
|
||||
],
|
||||
)
|
||||
@@ -0,0 +1,223 @@
|
||||
"""
|
||||
GitHub Actions service account and Workload Identity Federation.
|
||||
|
||||
Manages the service account used by GitHub Actions for CI/CD deployments,
|
||||
including keyless authentication via Workload Identity Federation.
|
||||
"""
|
||||
|
||||
import pulumi
|
||||
import pulumi_gcp as gcp
|
||||
from pulumi_gcp import serviceaccount, iam
|
||||
|
||||
from config import PROJECT_ID
|
||||
|
||||
|
||||
def create_github_actions_service_account() -> serviceaccount.Account:
|
||||
"""
|
||||
Create the GitHub Actions service account for CI/CD.
|
||||
|
||||
Returns:
|
||||
serviceaccount.Account: The service account resource.
|
||||
"""
|
||||
return serviceaccount.Account(
|
||||
"github-actions-deployer-sa",
|
||||
account_id="github-actions-deployer",
|
||||
display_name="GitHub Actions Deployer",
|
||||
description="Service account for GitHub Actions CD via Workload Identity",
|
||||
)
|
||||
|
||||
|
||||
def create_workload_identity_pool() -> iam.WorkloadIdentityPool:
|
||||
"""
|
||||
Create the Workload Identity Pool for GitHub OIDC.
|
||||
|
||||
Returns:
|
||||
iam.WorkloadIdentityPool: The pool resource.
|
||||
"""
|
||||
return iam.WorkloadIdentityPool(
|
||||
"github-actions-pool",
|
||||
workload_identity_pool_id="github-actions-pool",
|
||||
display_name="GitHub Actions Pool",
|
||||
description="Workload Identity Pool for GitHub Actions",
|
||||
disabled=False,
|
||||
)
|
||||
|
||||
|
||||
def create_workload_identity_provider(
|
||||
pool: iam.WorkloadIdentityPool,
|
||||
) -> iam.WorkloadIdentityPoolProvider:
|
||||
"""
|
||||
Create the Workload Identity Provider for GitHub OIDC.
|
||||
|
||||
Args:
|
||||
pool: The Workload Identity Pool.
|
||||
|
||||
Returns:
|
||||
iam.WorkloadIdentityPoolProvider: The provider resource.
|
||||
"""
|
||||
return iam.WorkloadIdentityPoolProvider(
|
||||
"github-actions-provider",
|
||||
workload_identity_pool_id=pool.workload_identity_pool_id,
|
||||
workload_identity_pool_provider_id="github-actions-provider",
|
||||
display_name="GitHub Actions Provider",
|
||||
attribute_mapping={
|
||||
"google.subject": "assertion.sub",
|
||||
"attribute.actor": "assertion.actor",
|
||||
"attribute.repository": "assertion.repository",
|
||||
"attribute.repository_owner": "assertion.repository_owner",
|
||||
},
|
||||
attribute_condition="assertion.repository_owner == 'nomadkaraoke'",
|
||||
oidc=iam.WorkloadIdentityPoolProviderOidcArgs(
|
||||
issuer_uri="https://token.actions.githubusercontent.com",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def grant_github_actions_permissions(
|
||||
service_account: serviceaccount.Account,
|
||||
pool: iam.WorkloadIdentityPool,
|
||||
) -> dict:
|
||||
"""
|
||||
Grant all necessary permissions to the GitHub Actions service account.
|
||||
|
||||
Args:
|
||||
service_account: The GitHub Actions service account.
|
||||
pool: The Workload Identity Pool.
|
||||
|
||||
Returns:
|
||||
dict: Dictionary of IAM binding resources.
|
||||
"""
|
||||
bindings = {}
|
||||
|
||||
# Grant Cloud Run admin permissions
|
||||
bindings["run_admin"] = gcp.projects.IAMMember(
|
||||
"github-actions-run-admin",
|
||||
project=PROJECT_ID,
|
||||
role="roles/run.admin",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant service account user permissions (to deploy Cloud Run with other SA)
|
||||
bindings["sa_user"] = gcp.projects.IAMMember(
|
||||
"github-actions-sa-user",
|
||||
project=PROJECT_ID,
|
||||
role="roles/iam.serviceAccountUser",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Artifact Registry writer permissions
|
||||
bindings["artifact_writer"] = gcp.projects.IAMMember(
|
||||
"github-actions-artifact-writer",
|
||||
project=PROJECT_ID,
|
||||
role="roles/artifactregistry.writer",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Cloud Build permissions - REQUIRED for gcloud builds submit
|
||||
bindings["cloudbuild"] = gcp.projects.IAMMember(
|
||||
"github-actions-cloudbuild",
|
||||
project=PROJECT_ID,
|
||||
role="roles/cloudbuild.builds.editor",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Storage Admin for Cloud Build bucket uploads
|
||||
bindings["storage_admin"] = gcp.projects.IAMMember(
|
||||
"github-actions-storage-admin",
|
||||
project=PROJECT_ID,
|
||||
role="roles/storage.admin",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Logging write permissions for Cloud Build logs
|
||||
bindings["logging"] = gcp.projects.IAMMember(
|
||||
"github-actions-logging",
|
||||
project=PROJECT_ID,
|
||||
role="roles/logging.logWriter",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Cloud Build logs viewer - needed for `gcloud builds log` to read build output
|
||||
bindings["logging_viewer"] = gcp.projects.IAMMember(
|
||||
"github-actions-logging-viewer",
|
||||
project=PROJECT_ID,
|
||||
role="roles/logging.viewer",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Service Usage Consumer - required for serviceusage.services.use permission
|
||||
bindings["service_usage"] = gcp.projects.IAMMember(
|
||||
"github-actions-service-usage",
|
||||
project=PROJECT_ID,
|
||||
role="roles/serviceusage.serviceUsageConsumer",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Project IAM Admin - required for Pulumi to refresh IAM binding state
|
||||
bindings["iam_admin"] = gcp.projects.IAMMember(
|
||||
"github-actions-iam-admin",
|
||||
project=PROJECT_ID,
|
||||
role="roles/resourcemanager.projectIamAdmin",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Secret Manager Admin - required for Pulumi to refresh secret state
|
||||
bindings["secretmanager"] = gcp.projects.IAMMember(
|
||||
"github-actions-secretmanager",
|
||||
project=PROJECT_ID,
|
||||
role="roles/secretmanager.admin",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Allow GitHub Actions to impersonate the service account
|
||||
# All nomadkaraoke repos that deploy to GCP need access
|
||||
bindings["wif_binding"] = serviceaccount.IAMBinding(
|
||||
"github-actions-wif-binding",
|
||||
service_account_id=service_account.name,
|
||||
role="roles/iam.workloadIdentityUser",
|
||||
members=[
|
||||
pool.name.apply(
|
||||
lambda pool_name: f"principalSet://iam.googleapis.com/{pool_name}/attribute.repository/nomadkaraoke/karaoke-gen"
|
||||
),
|
||||
pool.name.apply(
|
||||
lambda pool_name: f"principalSet://iam.googleapis.com/{pool_name}/attribute.repository/nomadkaraoke/flacfetch"
|
||||
),
|
||||
pool.name.apply(
|
||||
lambda pool_name: f"principalSet://iam.googleapis.com/{pool_name}/attribute.repository/nomadkaraoke/karaoke-decide"
|
||||
),
|
||||
],
|
||||
)
|
||||
|
||||
# Grant Compute Engine permissions for flacfetch VM deployment
|
||||
bindings["compute_admin"] = gcp.projects.IAMMember(
|
||||
"github-actions-compute-admin",
|
||||
project=PROJECT_ID,
|
||||
role="roles/compute.instanceAdmin.v1",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant OS Login permissions for SSH access
|
||||
bindings["os_login"] = gcp.projects.IAMMember(
|
||||
"github-actions-os-login",
|
||||
project=PROJECT_ID,
|
||||
role="roles/compute.osLogin",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant IAP tunnel user for secure SSH (used by gcloud compute ssh)
|
||||
bindings["iap_tunnel"] = gcp.projects.IAMMember(
|
||||
"github-actions-iap-tunnel",
|
||||
project=PROJECT_ID,
|
||||
role="roles/iap.tunnelResourceAccessor",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant broad Editor access to avoid permission issues when adding new resources
|
||||
bindings["editor"] = gcp.projects.IAMMember(
|
||||
"github-actions-editor",
|
||||
project=PROJECT_ID,
|
||||
role="roles/editor",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
return bindings
|
||||
@@ -0,0 +1,197 @@
|
||||
"""
|
||||
Worker service accounts for VM-based services.
|
||||
|
||||
Manages service accounts for:
|
||||
- Flacfetch torrent/download VM
|
||||
- Encoding worker VM
|
||||
- GDrive validator Cloud Function
|
||||
- GitHub Actions self-hosted runners
|
||||
"""
|
||||
|
||||
import pulumi
|
||||
import pulumi_gcp as gcp
|
||||
from pulumi_gcp import serviceaccount, storage
|
||||
|
||||
from config import PROJECT_ID
|
||||
|
||||
|
||||
# ==================== Flacfetch Service Account ====================
|
||||
|
||||
def create_flacfetch_service_account() -> serviceaccount.Account:
|
||||
"""Create the flacfetch service account for torrent downloads."""
|
||||
return serviceaccount.Account(
|
||||
"flacfetch-sa",
|
||||
account_id="flacfetch-service",
|
||||
display_name="Flacfetch Service Account",
|
||||
description="Service account for flacfetch torrent/audio download VM",
|
||||
)
|
||||
|
||||
|
||||
def grant_flacfetch_permissions(
|
||||
service_account: serviceaccount.Account,
|
||||
bucket: storage.Bucket,
|
||||
) -> dict:
|
||||
"""Grant permissions to the flacfetch service account."""
|
||||
bindings = {}
|
||||
|
||||
# Storage Object Creator - write downloaded files to GCS
|
||||
bindings["storage_writer"] = storage.BucketIAMMember(
|
||||
"flacfetch-storage-writer",
|
||||
bucket=bucket.name,
|
||||
role="roles/storage.objectCreator",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Storage Object Viewer - read uploads folder
|
||||
bindings["storage_reader"] = storage.BucketIAMMember(
|
||||
"flacfetch-storage-reader",
|
||||
bucket=bucket.name,
|
||||
role="roles/storage.objectViewer",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Secret Manager Accessor - read API keys
|
||||
bindings["secrets_access"] = gcp.projects.IAMMember(
|
||||
"flacfetch-secrets-access",
|
||||
project=PROJECT_ID,
|
||||
role="roles/secretmanager.secretAccessor",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
return bindings
|
||||
|
||||
|
||||
# ==================== Encoding Worker Service Account ====================
|
||||
|
||||
def create_encoding_worker_service_account() -> serviceaccount.Account:
|
||||
"""Create the encoding worker service account for video encoding."""
|
||||
return serviceaccount.Account(
|
||||
"encoding-worker-sa",
|
||||
account_id="encoding-worker",
|
||||
display_name="Encoding Worker Service Account",
|
||||
description="Service account for high-performance video encoding VM",
|
||||
)
|
||||
|
||||
|
||||
def grant_encoding_worker_permissions(
|
||||
service_account: serviceaccount.Account,
|
||||
bucket: storage.Bucket,
|
||||
) -> dict:
|
||||
"""Grant permissions to the encoding worker service account."""
|
||||
bindings = {}
|
||||
|
||||
# Storage Object Admin - read/write video files
|
||||
bindings["storage_admin"] = storage.BucketIAMMember(
|
||||
"encoding-worker-storage-admin",
|
||||
bucket=bucket.name,
|
||||
role="roles/storage.objectAdmin",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Secret Manager Accessor - read API key
|
||||
bindings["secrets_access"] = gcp.projects.IAMMember(
|
||||
"encoding-worker-secrets-access",
|
||||
project=PROJECT_ID,
|
||||
role="roles/secretmanager.secretAccessor",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Logging Writer - write structured logs
|
||||
bindings["logging_access"] = gcp.projects.IAMMember(
|
||||
"encoding-worker-logging-access",
|
||||
project=PROJECT_ID,
|
||||
role="roles/logging.logWriter",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Monitoring Metric Writer - write metrics for Ops Agent
|
||||
bindings["monitoring_access"] = gcp.projects.IAMMember(
|
||||
"encoding-worker-monitoring-access",
|
||||
project=PROJECT_ID,
|
||||
role="roles/monitoring.metricWriter",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
return bindings
|
||||
|
||||
|
||||
# ==================== GDrive Validator Service Account ====================
|
||||
|
||||
def create_gdrive_validator_service_account() -> serviceaccount.Account:
|
||||
"""Create the GDrive validator service account for Cloud Function."""
|
||||
return serviceaccount.Account(
|
||||
"gdrive-validator-sa",
|
||||
account_id="gdrive-validator",
|
||||
display_name="Google Drive Validator Function",
|
||||
description="Service account for the GDrive validation Cloud Function",
|
||||
)
|
||||
|
||||
|
||||
def grant_gdrive_validator_permissions(
|
||||
service_account: serviceaccount.Account,
|
||||
) -> dict:
|
||||
"""Grant permissions to the GDrive validator service account."""
|
||||
bindings = {}
|
||||
|
||||
# Secret Manager Accessor - read Pushbullet API key
|
||||
bindings["secrets_access"] = gcp.projects.IAMMember(
|
||||
"gdrive-validator-secrets-access",
|
||||
project=PROJECT_ID,
|
||||
role="roles/secretmanager.secretAccessor",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
return bindings
|
||||
|
||||
|
||||
# ==================== GitHub Runner Service Account ====================
|
||||
|
||||
def create_github_runner_service_account() -> serviceaccount.Account:
|
||||
"""Create the GitHub runner service account for self-hosted runners."""
|
||||
return serviceaccount.Account(
|
||||
"github-runner-sa",
|
||||
account_id="github-runner",
|
||||
display_name="GitHub Actions Runner",
|
||||
description="Service account for self-hosted GitHub Actions runner VM",
|
||||
)
|
||||
|
||||
|
||||
def grant_github_runner_permissions(
|
||||
service_account: serviceaccount.Account,
|
||||
) -> dict:
|
||||
"""Grant permissions to the GitHub runner service account."""
|
||||
bindings = {}
|
||||
|
||||
# Artifact Registry Reader - pull base images
|
||||
bindings["artifact_reader"] = gcp.projects.IAMMember(
|
||||
"github-runner-artifact-reader",
|
||||
project=PROJECT_ID,
|
||||
role="roles/artifactregistry.reader",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Secret Manager Accessor - read GitHub PAT
|
||||
bindings["secrets_access"] = gcp.projects.IAMMember(
|
||||
"github-runner-secrets-access",
|
||||
project=PROJECT_ID,
|
||||
role="roles/secretmanager.secretAccessor",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Logging Writer - write logs
|
||||
bindings["logging_access"] = gcp.projects.IAMMember(
|
||||
"github-runner-logging-access",
|
||||
project=PROJECT_ID,
|
||||
role="roles/logging.logWriter",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Storage Object Admin - upload/overwrite wheels in GCS
|
||||
bindings["storage_access"] = gcp.projects.IAMMember(
|
||||
"github-runner-storage-write",
|
||||
project=PROJECT_ID,
|
||||
role="roles/storage.objectAdmin",
|
||||
member=service_account.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
return bindings
|
||||
@@ -0,0 +1,159 @@
|
||||
"""
|
||||
Cloud Monitoring resources.
|
||||
|
||||
Manages alert policies for proactive monitoring and incident response.
|
||||
"""
|
||||
|
||||
import pulumi
|
||||
import pulumi_gcp as gcp
|
||||
|
||||
|
||||
def create_alert_policies() -> dict[str, gcp.monitoring.AlertPolicy]:
|
||||
"""
|
||||
Create all Cloud Monitoring alert policies.
|
||||
|
||||
Returns:
|
||||
dict: Dictionary mapping alert names to AlertPolicy resources.
|
||||
"""
|
||||
alerts = {}
|
||||
|
||||
# Alert: High Error Rate (>10% of requests returning errors)
|
||||
alerts["error_rate"] = gcp.monitoring.AlertPolicy(
|
||||
"high-error-rate-alert",
|
||||
display_name="Karaoke Backend - High Error Rate",
|
||||
combiner="OR",
|
||||
conditions=[
|
||||
gcp.monitoring.AlertPolicyConditionArgs(
|
||||
display_name="Error rate > 10%",
|
||||
condition_threshold=gcp.monitoring.AlertPolicyConditionConditionThresholdArgs(
|
||||
filter='resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND metric.type="run.googleapis.com/request_count" AND metric.labels.response_code_class!="2xx"',
|
||||
comparison="COMPARISON_GT",
|
||||
threshold_value=0.1,
|
||||
duration="300s",
|
||||
aggregations=[
|
||||
gcp.monitoring.AlertPolicyConditionConditionThresholdAggregationArgs(
|
||||
alignment_period="60s",
|
||||
per_series_aligner="ALIGN_RATE",
|
||||
cross_series_reducer="REDUCE_SUM",
|
||||
),
|
||||
],
|
||||
trigger=gcp.monitoring.AlertPolicyConditionConditionThresholdTriggerArgs(
|
||||
count=1,
|
||||
),
|
||||
),
|
||||
),
|
||||
],
|
||||
alert_strategy=gcp.monitoring.AlertPolicyAlertStrategyArgs(
|
||||
auto_close="3600s",
|
||||
),
|
||||
documentation=gcp.monitoring.AlertPolicyDocumentationArgs(
|
||||
content="High error rate detected on karaoke-backend. Check Cloud Run logs for details.\n\nDashboard: https://console.cloud.google.com/run/detail/us-central1/karaoke-backend/logs?project=nomadkaraoke",
|
||||
mime_type="text/markdown",
|
||||
),
|
||||
enabled=True,
|
||||
)
|
||||
|
||||
# Alert: Cloud Tasks Queue Backlog (tasks piling up)
|
||||
alerts["queue_backlog"] = gcp.monitoring.AlertPolicy(
|
||||
"queue-backlog-alert",
|
||||
display_name="Karaoke Backend - Queue Backlog",
|
||||
combiner="OR",
|
||||
conditions=[
|
||||
gcp.monitoring.AlertPolicyConditionArgs(
|
||||
display_name="Queue depth > 50 tasks",
|
||||
condition_threshold=gcp.monitoring.AlertPolicyConditionConditionThresholdArgs(
|
||||
filter='resource.type="cloud_tasks_queue" AND metric.type="cloudtasks.googleapis.com/queue/depth"',
|
||||
comparison="COMPARISON_GT",
|
||||
threshold_value=50,
|
||||
duration="600s",
|
||||
aggregations=[
|
||||
gcp.monitoring.AlertPolicyConditionConditionThresholdAggregationArgs(
|
||||
alignment_period="60s",
|
||||
per_series_aligner="ALIGN_MEAN",
|
||||
cross_series_reducer="REDUCE_MAX",
|
||||
),
|
||||
],
|
||||
trigger=gcp.monitoring.AlertPolicyConditionConditionThresholdTriggerArgs(
|
||||
count=1,
|
||||
),
|
||||
),
|
||||
),
|
||||
],
|
||||
alert_strategy=gcp.monitoring.AlertPolicyAlertStrategyArgs(
|
||||
auto_close="1800s",
|
||||
),
|
||||
documentation=gcp.monitoring.AlertPolicyDocumentationArgs(
|
||||
content="Cloud Tasks queue backlog detected. Tasks are piling up faster than workers can process.\n\nPossible causes:\n- Worker errors (check logs)\n- External API issues (Modal, AudioShake)\n- Cloud Run scaling limits\n\nDashboard: https://console.cloud.google.com/cloudtasks?project=nomadkaraoke",
|
||||
mime_type="text/markdown",
|
||||
),
|
||||
enabled=True,
|
||||
)
|
||||
|
||||
# Alert: High Memory Utilization (workers may be crashing)
|
||||
alerts["memory"] = gcp.monitoring.AlertPolicy(
|
||||
"high-memory-alert",
|
||||
display_name="Karaoke Backend - High Memory Usage",
|
||||
combiner="OR",
|
||||
conditions=[
|
||||
gcp.monitoring.AlertPolicyConditionArgs(
|
||||
display_name="Memory > 85%",
|
||||
condition_threshold=gcp.monitoring.AlertPolicyConditionConditionThresholdArgs(
|
||||
filter='resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND metric.type="run.googleapis.com/container/memory/utilizations"',
|
||||
comparison="COMPARISON_GT",
|
||||
threshold_value=0.85,
|
||||
duration="300s",
|
||||
aggregations=[
|
||||
gcp.monitoring.AlertPolicyConditionConditionThresholdAggregationArgs(
|
||||
alignment_period="60s",
|
||||
per_series_aligner="ALIGN_PERCENTILE_95",
|
||||
cross_series_reducer="REDUCE_MAX",
|
||||
),
|
||||
],
|
||||
trigger=gcp.monitoring.AlertPolicyConditionConditionThresholdTriggerArgs(
|
||||
count=1,
|
||||
),
|
||||
),
|
||||
),
|
||||
],
|
||||
alert_strategy=gcp.monitoring.AlertPolicyAlertStrategyArgs(
|
||||
auto_close="1800s",
|
||||
),
|
||||
documentation=gcp.monitoring.AlertPolicyDocumentationArgs(
|
||||
content="High memory utilization on karaoke-backend. Workers may be running out of memory.\n\nActions:\n- Check for memory leaks in worker code\n- Consider increasing Cloud Run memory limits\n- Review temp file cleanup",
|
||||
mime_type="text/markdown",
|
||||
),
|
||||
enabled=True,
|
||||
)
|
||||
|
||||
# Alert: Cloud Run Service Unavailable
|
||||
alerts["service_unavailable"] = gcp.monitoring.AlertPolicy(
|
||||
"service-unavailable-alert",
|
||||
display_name="Karaoke Backend - Service Unavailable",
|
||||
combiner="OR",
|
||||
conditions=[
|
||||
gcp.monitoring.AlertPolicyConditionArgs(
|
||||
display_name="No healthy instances",
|
||||
condition_absent=gcp.monitoring.AlertPolicyConditionConditionAbsentArgs(
|
||||
filter='resource.type="cloud_run_revision" AND resource.labels.service_name="karaoke-backend" AND metric.type="run.googleapis.com/container/instance_count"',
|
||||
duration="300s",
|
||||
aggregations=[
|
||||
gcp.monitoring.AlertPolicyConditionConditionAbsentAggregationArgs(
|
||||
alignment_period="60s",
|
||||
per_series_aligner="ALIGN_MEAN",
|
||||
cross_series_reducer="REDUCE_SUM",
|
||||
),
|
||||
],
|
||||
),
|
||||
),
|
||||
],
|
||||
alert_strategy=gcp.monitoring.AlertPolicyAlertStrategyArgs(
|
||||
auto_close="1800s",
|
||||
),
|
||||
documentation=gcp.monitoring.AlertPolicyDocumentationArgs(
|
||||
content="Karaoke backend service appears to be down - no healthy Cloud Run instances detected.\n\nImmediate actions:\n1. Check Cloud Run logs for crash reasons\n2. Verify recent deployments\n3. Check external dependencies (Modal, AudioShake)\n\nService URL: https://api.nomadkaraoke.com/api/health",
|
||||
mime_type="text/markdown",
|
||||
),
|
||||
enabled=True,
|
||||
)
|
||||
|
||||
return alerts
|
||||
@@ -0,0 +1,65 @@
|
||||
"""
|
||||
Secret Manager resources.
|
||||
|
||||
Manages all secrets for the application. Secret values are added manually
|
||||
via gcloud or the Cloud Console after resources are created.
|
||||
"""
|
||||
|
||||
import pulumi
|
||||
from pulumi_gcp import secretmanager
|
||||
|
||||
|
||||
def create_secrets() -> dict[str, secretmanager.Secret]:
|
||||
"""
|
||||
Create all Secret Manager secrets.
|
||||
|
||||
Returns:
|
||||
dict: Dictionary mapping secret names to Secret resources.
|
||||
"""
|
||||
# Define all secret names (matching original __main__.py exactly)
|
||||
secret_names = [
|
||||
# Audio processing
|
||||
"audioshake-api-key",
|
||||
"genius-api-key",
|
||||
"audio-separator-api-url",
|
||||
|
||||
# Private tracker APIs (for flacfetch audio search)
|
||||
"red-api-key",
|
||||
"red-api-url",
|
||||
"ops-api-key",
|
||||
"ops-api-url",
|
||||
|
||||
# Payment and email
|
||||
"stripe-secret-key",
|
||||
"stripe-webhook-secret",
|
||||
"sendgrid-api-key",
|
||||
|
||||
# Observability (Langfuse for LLM tracing)
|
||||
"langfuse-public-key",
|
||||
"langfuse-secret-key",
|
||||
"langfuse-host",
|
||||
|
||||
# Service authentication
|
||||
"flacfetch-api-key",
|
||||
"flacfetch-api-url",
|
||||
"encoding-worker-api-key",
|
||||
|
||||
# Notifications
|
||||
"pushbullet-api-key",
|
||||
|
||||
# GitHub
|
||||
"github-runner-pat",
|
||||
]
|
||||
|
||||
secrets = {}
|
||||
|
||||
for secret_id in secret_names:
|
||||
secrets[secret_id] = secretmanager.Secret(
|
||||
secret_id,
|
||||
secret_id=secret_id,
|
||||
replication=secretmanager.SecretReplicationArgs(
|
||||
auto=secretmanager.SecretReplicationAutoArgs(),
|
||||
),
|
||||
)
|
||||
|
||||
return secrets
|
||||
@@ -0,0 +1,38 @@
|
||||
"""
|
||||
Cloud Storage resources.
|
||||
|
||||
Manages the GCS bucket for uploads, outputs, and temporary files.
|
||||
"""
|
||||
|
||||
import pulumi
|
||||
import pulumi_gcp as gcp
|
||||
from pulumi_gcp import storage
|
||||
|
||||
from config import PROJECT_ID
|
||||
|
||||
|
||||
def create_bucket() -> storage.Bucket:
|
||||
"""
|
||||
Create the main GCS bucket with lifecycle rules.
|
||||
|
||||
Returns:
|
||||
storage.Bucket: The created bucket resource.
|
||||
"""
|
||||
bucket = storage.Bucket(
|
||||
"karaoke-storage",
|
||||
name=f"karaoke-gen-storage-{PROJECT_ID}",
|
||||
location="US-CENTRAL1",
|
||||
force_destroy=False, # Prevent accidental deletion
|
||||
uniform_bucket_level_access=True,
|
||||
lifecycle_rules=[
|
||||
storage.BucketLifecycleRuleArgs(
|
||||
action=storage.BucketLifecycleRuleActionArgs(type="Delete"),
|
||||
condition=storage.BucketLifecycleRuleConditionArgs(
|
||||
age=7,
|
||||
matches_prefixes=["temp/", "uploads/"]
|
||||
),
|
||||
),
|
||||
],
|
||||
)
|
||||
|
||||
return bucket
|
||||
@@ -0,0 +1,122 @@
|
||||
# Encoding Worker Packer Image
|
||||
|
||||
Custom GCE machine image for the encoding worker VM with all dependencies pre-installed.
|
||||
|
||||
## Why?
|
||||
|
||||
The encoding worker previously took **~10 minutes** to start because it built Python 3.13 from source on every boot. With this pre-baked image, startup time is reduced to **~30 seconds**.
|
||||
|
||||
| Component | Before (startup script) | After (Packer image) |
|
||||
|-----------|------------------------|---------------------|
|
||||
| Python 3.13 build | ~7 min (every boot) | Pre-installed |
|
||||
| FFmpeg install | ~1 min (every boot) | Pre-installed |
|
||||
| System packages | ~1 min (every boot) | Pre-installed |
|
||||
| Font packages | ~30 sec (every boot) | Pre-installed |
|
||||
| API key + wheel | ~30 sec | ~30 sec |
|
||||
| **Total** | **~10 min** | **~30 sec** |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **Packer installed**: `brew install packer` or [download from HashiCorp](https://www.packer.io/downloads)
|
||||
2. **GCP authentication**: `gcloud auth application-default login`
|
||||
3. **Required permissions**: `roles/compute.instanceAdmin.v1` and `roles/iam.serviceAccountUser`
|
||||
|
||||
## Building the Image
|
||||
|
||||
```bash
|
||||
cd infrastructure/packer
|
||||
|
||||
# Initialize Packer plugins (first time only)
|
||||
packer init encoding-worker.pkr.hcl
|
||||
|
||||
# Build the image
|
||||
packer build encoding-worker.pkr.hcl
|
||||
```
|
||||
|
||||
Build takes approximately **15-20 minutes** (mostly Python compilation with optimizations).
|
||||
|
||||
### Build Variables
|
||||
|
||||
Override defaults with `-var`:
|
||||
|
||||
```bash
|
||||
packer build \
|
||||
-var 'python_version=3.13.2' \
|
||||
-var 'zone=us-west1-a' \
|
||||
encoding-worker.pkr.hcl
|
||||
```
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `project_id` | `nomadkaraoke` | GCP project ID |
|
||||
| `zone` | `us-central1-a` | Build zone |
|
||||
| `python_version` | `3.13.1` | Python version to install |
|
||||
|
||||
## What's Included
|
||||
|
||||
The image contains:
|
||||
|
||||
- **Python 3.13** at `/opt/python313` (built with optimizations)
|
||||
- **FFmpeg 7.x** at `/usr/local/bin/ffmpeg` (John Van Sickle static build)
|
||||
- **Noto fonts** including CJK support for subtitle rendering
|
||||
- **Virtual environment** at `/opt/encoding-worker/venv`
|
||||
- **Systemd service** `encoding-worker.service` (enabled, not started)
|
||||
- **Startup script** `/opt/encoding-worker/startup.sh`
|
||||
|
||||
## Runtime Behavior
|
||||
|
||||
When the VM boots with this image:
|
||||
|
||||
1. **Startup script** (`/opt/encoding-worker/startup.sh`) runs via `ExecStartPre`:
|
||||
- Fetches API key from Secret Manager
|
||||
- Downloads latest `karaoke-gen` wheel from GCS
|
||||
- Installs wheel into venv
|
||||
|
||||
2. **Encoding worker service** starts via `ExecStart`:
|
||||
- Runs `uvicorn backend.services.gce_encoding.main:app`
|
||||
- Listens on port 8080
|
||||
|
||||
## When to Rebuild
|
||||
|
||||
Rebuild the image when:
|
||||
|
||||
- Python version needs updating
|
||||
- FFmpeg version needs updating
|
||||
- New system packages required
|
||||
- Font packages change
|
||||
|
||||
**No rebuild needed** for application code changes - the wheel is downloaded at runtime.
|
||||
|
||||
## Image Family
|
||||
|
||||
Images are created with family `encoding-worker`. Pulumi uses the family to always get the latest image:
|
||||
|
||||
```python
|
||||
image=f"projects/nomadkaraoke/global/images/family/encoding-worker"
|
||||
```
|
||||
|
||||
Old images are automatically replaced in the family, but retained for rollback.
|
||||
|
||||
## Rollback
|
||||
|
||||
If the new image has issues:
|
||||
|
||||
1. Find previous image: `gcloud compute images list --filter="family:encoding-worker"`
|
||||
2. Update Pulumi to use specific image instead of family
|
||||
3. Or rebuild from a previous git commit
|
||||
|
||||
## CI Integration
|
||||
|
||||
For automated rebuilds, add a GitHub Actions workflow that triggers on changes to `infrastructure/packer/`:
|
||||
|
||||
```yaml
|
||||
name: Build Encoding Worker Image
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'infrastructure/packer/**'
|
||||
workflow_dispatch:
|
||||
```
|
||||
|
||||
See `PHASE5-PACKER-IMAGE-PLAN.md` for full workflow example.
|
||||
@@ -0,0 +1,94 @@
|
||||
# Packer template for encoding worker GCE image
|
||||
#
|
||||
# This image pre-installs all dependencies to reduce VM startup time from
|
||||
# ~10 minutes (building Python from source) to ~30 seconds (just start service).
|
||||
#
|
||||
# Usage:
|
||||
# cd infrastructure/packer
|
||||
# packer init encoding-worker.pkr.hcl
|
||||
# packer build encoding-worker.pkr.hcl
|
||||
|
||||
packer {
|
||||
required_plugins {
|
||||
googlecompute = {
|
||||
source = "github.com/hashicorp/googlecompute"
|
||||
version = "~> 1.1"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
variable "project_id" {
|
||||
type = string
|
||||
default = "nomadkaraoke"
|
||||
description = "GCP project ID"
|
||||
}
|
||||
|
||||
variable "zone" {
|
||||
type = string
|
||||
default = "us-central1-a"
|
||||
description = "GCE zone for building the image"
|
||||
}
|
||||
|
||||
variable "python_version" {
|
||||
type = string
|
||||
default = "3.13.1"
|
||||
description = "Python version to install from source"
|
||||
}
|
||||
|
||||
source "googlecompute" "encoding-worker" {
|
||||
project_id = var.project_id
|
||||
zone = var.zone
|
||||
source_image_family = "debian-12"
|
||||
source_image_project_id = ["debian-cloud"]
|
||||
|
||||
# Use n2 machine type for building (c4 requires hyperdisk which isn't supported by Packer)
|
||||
# The resulting image will work on c4-standard-8 in production
|
||||
machine_type = "n2-standard-8"
|
||||
disk_size = 100
|
||||
disk_type = "pd-ssd" # Use SSD for faster builds
|
||||
|
||||
# Image naming - uses timestamp for versioning, family for latest
|
||||
image_name = "encoding-worker-{{timestamp}}"
|
||||
image_family = "encoding-worker"
|
||||
image_description = "Encoding worker with Python ${var.python_version}, FFmpeg 7.x, and fonts pre-installed"
|
||||
|
||||
ssh_username = "packer"
|
||||
|
||||
# Tags for firewall rules during build (if needed)
|
||||
tags = ["packer-build"]
|
||||
|
||||
# Metadata for build tracking
|
||||
image_labels = {
|
||||
"managed-by" = "packer"
|
||||
"python" = replace(var.python_version, ".", "-")
|
||||
}
|
||||
}
|
||||
|
||||
build {
|
||||
sources = ["source.googlecompute.encoding-worker"]
|
||||
|
||||
# Upload provisioning script
|
||||
provisioner "file" {
|
||||
source = "scripts/provision.sh"
|
||||
destination = "/tmp/provision.sh"
|
||||
}
|
||||
|
||||
# Run provisioning
|
||||
provisioner "shell" {
|
||||
inline = [
|
||||
"chmod +x /tmp/provision.sh",
|
||||
"sudo PYTHON_VERSION=${var.python_version} /tmp/provision.sh"
|
||||
]
|
||||
}
|
||||
|
||||
# Clean up for smaller image
|
||||
provisioner "shell" {
|
||||
inline = [
|
||||
"sudo apt-get clean",
|
||||
"sudo rm -rf /var/lib/apt/lists/*",
|
||||
"sudo rm -rf /tmp/*",
|
||||
"sudo rm -rf /var/tmp/*",
|
||||
"sudo journalctl --vacuum-time=1d"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,177 @@
|
||||
#!/bin/bash
|
||||
# Provision encoding worker image with all dependencies pre-installed
|
||||
#
|
||||
# This script runs during Packer image build. After it completes:
|
||||
# - Python 3.13 is installed at /opt/python313
|
||||
# - FFmpeg 7.x is installed at /usr/local/bin/ffmpeg
|
||||
# - All fonts (including CJK) are installed
|
||||
# - Virtual environment exists at /opt/encoding-worker/venv
|
||||
# - Systemd service is configured (but not started - no API key yet)
|
||||
#
|
||||
# At runtime, the startup script only needs to:
|
||||
# 1. Fetch API key from Secret Manager
|
||||
# 2. Download latest karaoke-gen wheel
|
||||
# 3. Start the systemd service
|
||||
|
||||
set -e
|
||||
|
||||
echo "=== Starting encoding worker image provisioning ==="
|
||||
echo "Python version: ${PYTHON_VERSION}"
|
||||
|
||||
# Install system packages
|
||||
echo "Installing system packages..."
|
||||
apt-get update
|
||||
apt-get install -y \
|
||||
docker.io \
|
||||
curl \
|
||||
git \
|
||||
xz-utils \
|
||||
build-essential \
|
||||
libssl-dev \
|
||||
zlib1g-dev \
|
||||
libbz2-dev \
|
||||
libreadline-dev \
|
||||
libsqlite3-dev \
|
||||
libncursesw5-dev \
|
||||
tk-dev \
|
||||
libxml2-dev \
|
||||
libxmlsec1-dev \
|
||||
libffi-dev \
|
||||
liblzma-dev
|
||||
|
||||
# Install fonts for ASS subtitle rendering (libass uses fontconfig)
|
||||
# - fonts-noto: Noto Sans (default karaoke font) + musical symbols
|
||||
# - fonts-noto-cjk: Chinese, Japanese, Korean character support
|
||||
# - fontconfig: Font configuration system
|
||||
echo "Installing fonts..."
|
||||
apt-get install -y fonts-noto fonts-noto-cjk fontconfig
|
||||
fc-cache -fv
|
||||
|
||||
# Enable Docker (will start on boot)
|
||||
systemctl enable docker
|
||||
|
||||
# Install optimized static FFmpeg build (John Van Sickle)
|
||||
# This is faster than Debian's package and has more codecs enabled
|
||||
echo "Installing FFmpeg..."
|
||||
curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz -o /tmp/ffmpeg.tar.xz
|
||||
tar -xf /tmp/ffmpeg.tar.xz -C /tmp
|
||||
cp /tmp/ffmpeg-*-amd64-static/ffmpeg /usr/local/bin/
|
||||
cp /tmp/ffmpeg-*-amd64-static/ffprobe /usr/local/bin/
|
||||
chmod +x /usr/local/bin/ffmpeg /usr/local/bin/ffprobe
|
||||
rm -rf /tmp/ffmpeg*
|
||||
echo "FFmpeg version:"
|
||||
ffmpeg -version | head -1
|
||||
|
||||
# Build and install Python from source
|
||||
# This is the slow part (~7 min) - pre-baking it saves startup time
|
||||
echo "Building Python ${PYTHON_VERSION} from source..."
|
||||
cd /tmp
|
||||
curl -L "https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tar.xz" -o python.tar.xz
|
||||
tar -xf python.tar.xz
|
||||
cd "Python-${PYTHON_VERSION}"
|
||||
|
||||
# Configure with optimizations
|
||||
# --enable-optimizations: Profile-guided optimization (slower build, faster runtime)
|
||||
# --with-lto: Link-time optimization
|
||||
./configure --prefix=/opt/python313 --enable-optimizations --with-lto
|
||||
make -j$(nproc)
|
||||
make install
|
||||
|
||||
# Clean up Python build artifacts
|
||||
rm -rf /tmp/python.tar.xz /tmp/Python-${PYTHON_VERSION}
|
||||
|
||||
echo "Python ${PYTHON_VERSION} installed:"
|
||||
/opt/python313/bin/python3.13 --version
|
||||
|
||||
# Create working directory and venv
|
||||
mkdir -p /opt/encoding-worker
|
||||
cd /opt/encoding-worker
|
||||
|
||||
echo "Creating Python virtual environment..."
|
||||
/opt/python313/bin/python3.13 -m venv venv
|
||||
|
||||
# Install base Python dependencies (wheel will add more at runtime)
|
||||
source venv/bin/activate
|
||||
pip install --upgrade pip
|
||||
pip install fastapi uvicorn google-cloud-storage aiofiles aiohttp
|
||||
|
||||
# Create startup helper script (runs via ExecStartPre before service starts)
|
||||
# This fetches API key and installs latest wheel at boot time
|
||||
cat > /opt/encoding-worker/startup.sh << 'STARTUP'
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
echo "=== Encoding worker startup at $(date) ==="
|
||||
|
||||
# Fetch API key from Secret Manager
|
||||
echo "Fetching API key from Secret Manager..."
|
||||
ENCODING_API_KEY=$(gcloud secrets versions access latest --secret=encoding-worker-api-key 2>/dev/null || echo "")
|
||||
|
||||
if [ -z "$ENCODING_API_KEY" ]; then
|
||||
echo "WARNING: No API key found in Secret Manager"
|
||||
fi
|
||||
|
||||
# Write environment file for systemd
|
||||
echo "ENCODING_API_KEY=${ENCODING_API_KEY}" > /opt/encoding-worker/env
|
||||
|
||||
# Download and install latest karaoke-gen wheel from GCS
|
||||
# This enables hot code updates without rebuilding the image
|
||||
echo "Installing latest karaoke-gen wheel from GCS..."
|
||||
gsutil cp "gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-*.whl" /tmp/ 2>/dev/null || true
|
||||
WHEEL_FILE=$(ls -t /tmp/karaoke_gen-*.whl 2>/dev/null | head -1)
|
||||
if [ -n "$WHEEL_FILE" ]; then
|
||||
echo "Installing wheel: $WHEEL_FILE"
|
||||
/opt/encoding-worker/venv/bin/pip install --upgrade --quiet "$WHEEL_FILE" || echo "WARNING: Failed to install wheel"
|
||||
else
|
||||
echo "No wheel found in GCS"
|
||||
fi
|
||||
|
||||
echo "=== Startup script complete ==="
|
||||
STARTUP
|
||||
|
||||
chmod +x /opt/encoding-worker/startup.sh
|
||||
|
||||
# Create systemd service
|
||||
# ExecStartPre runs the startup script to fetch API key and install wheel
|
||||
# ExecStart runs uvicorn with the encoding worker from the installed wheel
|
||||
cat > /etc/systemd/system/encoding-worker.service << 'SYSTEMD'
|
||||
[Unit]
|
||||
Description=Encoding Worker HTTP API Service
|
||||
After=network.target docker.service
|
||||
Wants=docker.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
WorkingDirectory=/opt/encoding-worker
|
||||
ExecStartPre=/opt/encoding-worker/startup.sh
|
||||
ExecStart=/opt/encoding-worker/venv/bin/uvicorn backend.services.gce_encoding.main:app --host 0.0.0.0 --port 8080
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
TimeoutStartSec=300
|
||||
Environment="GOOGLE_CLOUD_PROJECT=nomadkaraoke"
|
||||
EnvironmentFile=/opt/encoding-worker/env
|
||||
|
||||
# Logging
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
SyslogIdentifier=encoding-worker
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
SYSTEMD
|
||||
|
||||
# Create empty env file (will be populated at runtime)
|
||||
touch /opt/encoding-worker/env
|
||||
|
||||
# Reload systemd and enable service (will start on boot)
|
||||
systemctl daemon-reload
|
||||
systemctl enable encoding-worker
|
||||
|
||||
echo "=== Provisioning complete ==="
|
||||
echo "Image includes:"
|
||||
echo " - Python ${PYTHON_VERSION} at /opt/python313"
|
||||
echo " - FFmpeg at /usr/local/bin/ffmpeg"
|
||||
echo " - Noto fonts (including CJK)"
|
||||
echo " - Virtual environment at /opt/encoding-worker/venv"
|
||||
echo " - Systemd service: encoding-worker.service"
|
||||
@@ -156,15 +156,20 @@ class VideoBackgroundProcessor:
|
||||
"""
|
||||
Escape a file path for use in ffmpeg filter expressions.
|
||||
|
||||
FFmpeg filter syntax uses single quotes to protect special characters
|
||||
like spaces, colons, and semicolons. Single quotes within the path
|
||||
are escaped using the '\\'' pattern (end quote, literal quote, start quote).
|
||||
|
||||
Args:
|
||||
path: File path to escape
|
||||
|
||||
Returns:
|
||||
str: Escaped path
|
||||
str: Escaped path (wrapped in single quotes)
|
||||
"""
|
||||
# Escape backslashes and colons for ffmpeg filter syntax
|
||||
escaped = path.replace("\\", "\\\\").replace(":", "\\:")
|
||||
return escaped
|
||||
# Escape single quotes: ' becomes '\'' (end quote, \', start quote)
|
||||
escaped = path.replace("'", "'\\''")
|
||||
# Wrap entire path in single quotes
|
||||
return f"'{escaped}'"
|
||||
|
||||
def build_video_filter(self, ass_subtitles_path, darkness_percent, fonts_dir=None):
|
||||
"""
|
||||
|
||||
@@ -35,10 +35,14 @@ class ProviderConfig:
|
||||
circuit_breaker_failure_threshold: int = 3
|
||||
circuit_breaker_open_seconds: int = 60
|
||||
|
||||
# Initialization timeouts - fail fast instead of hanging forever
|
||||
# These are separate from request_timeout to catch connection establishment issues
|
||||
initialization_timeout_seconds: float = 30.0 # Model creation + warm-up
|
||||
warmup_timeout_seconds: float = 15.0 # Just the warm-up call
|
||||
# Initialization timeout - fail fast instead of hanging forever
|
||||
# This is separate from request_timeout to catch connection establishment issues
|
||||
initialization_timeout_seconds: float = 30.0 # Model creation
|
||||
|
||||
# Parallel processing settings
|
||||
# Process multiple gaps concurrently to reduce total correction time
|
||||
# Set to 1 to disable parallelism, higher values increase throughput but may hit rate limits
|
||||
max_parallel_gaps: int = 5
|
||||
|
||||
@staticmethod
|
||||
def from_env(cache_dir: Optional[str] = None) -> "ProviderConfig":
|
||||
@@ -70,7 +74,7 @@ class ProviderConfig:
|
||||
circuit_breaker_failure_threshold=int(os.getenv("AGENTIC_CIRCUIT_THRESHOLD", "3")),
|
||||
circuit_breaker_open_seconds=int(os.getenv("AGENTIC_CIRCUIT_OPEN_SECONDS", "60")),
|
||||
initialization_timeout_seconds=float(os.getenv("AGENTIC_INIT_TIMEOUT_SECONDS", "30.0")),
|
||||
warmup_timeout_seconds=float(os.getenv("AGENTIC_WARMUP_TIMEOUT_SECONDS", "15.0")),
|
||||
max_parallel_gaps=int(os.getenv("AGENTIC_MAX_PARALLEL_GAPS", "5")),
|
||||
)
|
||||
|
||||
def validate_environment(self, logger: Optional[object] = None) -> None:
|
||||
|
||||
+1
-51
@@ -97,7 +97,6 @@ class LangChainBridge(BaseAIProvider):
|
||||
|
||||
# Lazy-initialized chat model
|
||||
self._chat_model: Optional[Any] = None
|
||||
self._warmed_up: bool = False
|
||||
|
||||
def name(self) -> str:
|
||||
"""Return provider name for logging."""
|
||||
@@ -164,13 +163,7 @@ class LangChainBridge(BaseAIProvider):
|
||||
) from None
|
||||
|
||||
init_elapsed = time.time() - init_start
|
||||
logger.info(f"🤖 Model created in {init_elapsed:.2f}s, starting warm-up...")
|
||||
|
||||
# Warm up the model to establish connection before real work
|
||||
self._warm_up_model()
|
||||
|
||||
total_elapsed = time.time() - init_start
|
||||
logger.info(f"🤖 Model initialization complete in {total_elapsed:.2f}s")
|
||||
logger.info(f"🤖 Model initialized in {init_elapsed:.2f}s")
|
||||
|
||||
except InitializationTimeoutError as e:
|
||||
self._circuit_breaker.record_failure(self._model)
|
||||
@@ -271,47 +264,4 @@ class LangChainBridge(BaseAIProvider):
|
||||
|
||||
return content
|
||||
|
||||
def _warm_up_model(self) -> None:
|
||||
"""Send a lightweight request to warm up the model connection.
|
||||
|
||||
This helps establish the REST connection and potentially warm up any
|
||||
server-side resources before processing real correction requests.
|
||||
The warm-up uses a timeout to fail fast if the service is unresponsive.
|
||||
"""
|
||||
if self._warmed_up:
|
||||
return
|
||||
|
||||
timeout = self._config.warmup_timeout_seconds
|
||||
# Use print with flush=True for visibility when output is redirected
|
||||
print(f"🔥 Warming up {self._model} connection (timeout: {timeout}s)...", flush=True)
|
||||
logger.info(f"🔥 Warming up {self._model} connection (timeout: {timeout}s)...")
|
||||
|
||||
warmup_start = time.time()
|
||||
try:
|
||||
from langchain_core.messages import HumanMessage
|
||||
|
||||
# Minimal prompt that requires almost no processing
|
||||
warm_up_prompt = 'Respond with exactly: {"status":"ready"}'
|
||||
|
||||
# Use ThreadPoolExecutor for timeout on warm-up call
|
||||
with ThreadPoolExecutor(max_workers=1) as executor:
|
||||
future = executor.submit(
|
||||
self._chat_model.invoke,
|
||||
[HumanMessage(content=warm_up_prompt)]
|
||||
)
|
||||
try:
|
||||
future.result(timeout=timeout)
|
||||
except FuturesTimeoutError:
|
||||
raise TimeoutError(f"Warm-up timed out after {timeout}s") from None
|
||||
|
||||
elapsed = time.time() - warmup_start
|
||||
self._warmed_up = True
|
||||
print(f"🔥 Warm-up complete for {self._model} in {elapsed:.2f}s", flush=True)
|
||||
logger.info(f"🔥 Warm-up complete for {self._model} in {elapsed:.2f}s")
|
||||
except Exception as e:
|
||||
elapsed = time.time() - warmup_start
|
||||
# Don't fail the actual request if warm-up fails
|
||||
# Just log and continue - the real request might still work
|
||||
print(f"🔥 Warm-up failed for {self._model} after {elapsed:.2f}s: {e} (continuing anyway)", flush=True)
|
||||
logger.warning(f"🔥 Warm-up failed for {self._model} after {elapsed:.2f}s: {e} (continuing anyway)")
|
||||
|
||||
@@ -268,12 +268,26 @@ class LyricsCorrector:
|
||||
_adapt = None
|
||||
_ModelRouter = None
|
||||
|
||||
# Pre-initialized agentic corrector (created once, reused for all gaps)
|
||||
_agentic_agent = None
|
||||
|
||||
if use_agentic_env:
|
||||
try:
|
||||
from lyrics_transcriber.correction.agentic.agent import AgenticCorrector as _AgenticCorrector
|
||||
from lyrics_transcriber.correction.agentic.adapter import adapt_proposals_to_word_corrections as _adapt
|
||||
from lyrics_transcriber.correction.agentic.router import ModelRouter as _ModelRouter
|
||||
self.logger.info("🤖 Agentic modules imported successfully - running in AGENTIC-ONLY mode")
|
||||
|
||||
# Create agent ONCE and reuse for all gaps (avoids repeated model initialization)
|
||||
_router = _ModelRouter()
|
||||
model_id = _router.choose_model("gap", uncertainty=0.5) # Use default uncertainty
|
||||
self.logger.info(f"🤖 Creating single AgenticCorrector with model: {model_id}")
|
||||
_agentic_agent = _AgenticCorrector.from_model(
|
||||
model=model_id,
|
||||
session_id=session_id,
|
||||
cache_dir=str(self._cache_dir)
|
||||
)
|
||||
self.logger.info("🤖 AgenticCorrector initialized and ready for all gaps")
|
||||
except Exception as e:
|
||||
self.logger.error(f"🤖 Failed to import agentic modules but USE_AGENTIC_AI=1: {e}")
|
||||
raise RuntimeError(f"Agentic AI correction is enabled but required modules could not be imported: {e}") from e
|
||||
@@ -443,16 +457,186 @@ class LyricsCorrector:
|
||||
sys.exit(0)
|
||||
# === END TEMPORARY CODE ===
|
||||
|
||||
# AGENTIC-ONLY MODE: Process all gaps in parallel for better performance
|
||||
if use_agentic_env:
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from lyrics_transcriber.correction.agentic.providers.config import ProviderConfig
|
||||
|
||||
# Get parallel processing config
|
||||
_config = ProviderConfig.from_env()
|
||||
max_workers = _config.max_parallel_gaps
|
||||
self.logger.info(f"🤖 Processing {len(gap_sequences)} gaps in parallel (max_workers={max_workers})")
|
||||
|
||||
# Pre-compute shared data structures once (not per-gap)
|
||||
all_transcribed_words = []
|
||||
for seg in segments:
|
||||
all_transcribed_words.extend(seg.words)
|
||||
word_position = {w.id: idx for idx, w in enumerate(all_transcribed_words)}
|
||||
|
||||
# Build reference contexts once (same for all gaps)
|
||||
reference_contexts = {}
|
||||
for source, lyrics_data in self.reference_lyrics.items():
|
||||
if lyrics_data and lyrics_data.segments:
|
||||
ref_words = []
|
||||
for seg in lyrics_data.segments:
|
||||
ref_words.extend([w.text for w in seg.words])
|
||||
reference_contexts[source] = " ".join(ref_words)
|
||||
|
||||
# Get artist and title once
|
||||
artist = metadata.get("artist") if metadata else None
|
||||
title = metadata.get("title") if metadata else None
|
||||
|
||||
# Prepare all gap inputs upfront
|
||||
gap_inputs = []
|
||||
for i, gap in enumerate(gap_sequences, 1):
|
||||
# Prepare gap words data
|
||||
gap_words_data = []
|
||||
for word_id in gap.transcribed_word_ids:
|
||||
if word_id in word_map:
|
||||
word = word_map[word_id]
|
||||
gap_words_data.append({
|
||||
"id": word_id,
|
||||
"text": word.text,
|
||||
"start_time": getattr(word, 'start_time', 0),
|
||||
"end_time": getattr(word, 'end_time', 0)
|
||||
})
|
||||
|
||||
# Compute context words
|
||||
gap_positions = [word_position[wid] for wid in gap.transcribed_word_ids if wid in word_position]
|
||||
preceding_words = ""
|
||||
following_words = ""
|
||||
|
||||
if gap_positions:
|
||||
first_gap_pos = min(gap_positions)
|
||||
last_gap_pos = max(gap_positions)
|
||||
|
||||
# Get 10 words before
|
||||
start_pos = max(0, first_gap_pos - 10)
|
||||
preceding_list = [all_transcribed_words[idx].text for idx in range(start_pos, first_gap_pos) if idx < len(all_transcribed_words)]
|
||||
preceding_words = " ".join(preceding_list)
|
||||
|
||||
# Get 10 words after
|
||||
end_pos = min(len(all_transcribed_words), last_gap_pos + 11)
|
||||
following_list = [all_transcribed_words[idx].text for idx in range(last_gap_pos + 1, end_pos) if idx < len(all_transcribed_words)]
|
||||
following_words = " ".join(following_list)
|
||||
|
||||
gap_inputs.append({
|
||||
'index': i,
|
||||
'gap': gap,
|
||||
'gap_id': f"gap_{i}",
|
||||
'gap_words': gap_words_data,
|
||||
'preceding_words': preceding_words,
|
||||
'following_words': following_words,
|
||||
'reference_contexts': reference_contexts,
|
||||
'artist': artist,
|
||||
'title': title
|
||||
})
|
||||
|
||||
# Function to process a single gap (runs in thread pool)
|
||||
def process_single_gap(gap_input):
|
||||
"""Process a single gap and return proposals. Thread-safe."""
|
||||
idx = gap_input['index']
|
||||
try:
|
||||
proposals = _agentic_agent.propose_for_gap(
|
||||
gap_id=gap_input['gap_id'],
|
||||
gap_words=gap_input['gap_words'],
|
||||
preceding_words=gap_input['preceding_words'],
|
||||
following_words=gap_input['following_words'],
|
||||
reference_contexts=gap_input['reference_contexts'],
|
||||
artist=gap_input['artist'],
|
||||
title=gap_input['title']
|
||||
)
|
||||
return {'index': idx, 'gap': gap_input['gap'], 'proposals': proposals, 'error': None}
|
||||
except Exception as e:
|
||||
return {'index': idx, 'gap': gap_input['gap'], 'proposals': None, 'error': str(e)}
|
||||
|
||||
# Process gaps in parallel
|
||||
results = [None] * len(gap_inputs)
|
||||
completed_count = 0
|
||||
errors = []
|
||||
|
||||
with ThreadPoolExecutor(max_workers=max_workers) as executor:
|
||||
# Submit all tasks
|
||||
future_to_input = {executor.submit(process_single_gap, g): g for g in gap_inputs}
|
||||
|
||||
# Collect results as they complete
|
||||
for future in as_completed(future_to_input):
|
||||
# Check deadline
|
||||
if deadline and time.time() > deadline:
|
||||
self.logger.warning(
|
||||
f"⏰ AGENTIC TIMEOUT: Deadline exceeded after processing {completed_count}/{len(gap_sequences)} gaps. "
|
||||
"Cancelling remaining gaps - human review will correct any issues."
|
||||
)
|
||||
# Cancel remaining futures (use list() to avoid mutating dict during iteration)
|
||||
for f in list(future_to_input.keys()):
|
||||
f.cancel()
|
||||
break
|
||||
|
||||
result = future.result()
|
||||
idx = result['index'] - 1 # Convert 1-based to 0-based
|
||||
results[idx] = result
|
||||
completed_count += 1
|
||||
|
||||
if result['error']:
|
||||
errors.append(f"Gap {result['index']}: {result['error']}")
|
||||
self.logger.error(f"🤖 Gap {result['index']} failed: {result['error']}")
|
||||
else:
|
||||
proposal_count = len(result['proposals']) if result['proposals'] else 0
|
||||
self.logger.info(f"🤖 Gap {result['index']}/{len(gap_sequences)} completed ({proposal_count} proposals)")
|
||||
|
||||
self.logger.info(f"🤖 Parallel processing complete: {completed_count}/{len(gap_sequences)} gaps processed")
|
||||
|
||||
# If any errors occurred, fail fast
|
||||
if errors:
|
||||
raise RuntimeError(f"Agentic AI correction failed for {len(errors)} gaps: {'; '.join(errors)}")
|
||||
|
||||
# Apply corrections sequentially (must be in order due to segment modifications)
|
||||
for result in results:
|
||||
if result is None:
|
||||
continue # Skipped due to deadline
|
||||
|
||||
i = result['index']
|
||||
gap = result['gap']
|
||||
_proposals = result['proposals']
|
||||
|
||||
_agentic_corrections = _adapt(_proposals, word_map, linear_position_map) if _proposals else []
|
||||
|
||||
if _agentic_corrections:
|
||||
self.logger.info(f"🤖 Applying {len(_agentic_corrections)} agentic corrections for gap {i}")
|
||||
affected_word_ids = [w.id for w in self._get_affected_words(gap, segments)]
|
||||
affected_segment_ids = [s.id for s in self._get_affected_segments(gap, segments)]
|
||||
updated_segments = self._apply_corrections_to_segments(self._get_affected_segments(gap, segments), _agentic_corrections)
|
||||
for correction in _agentic_corrections:
|
||||
if correction.word_id and correction.corrected_word_id:
|
||||
word_id_map[correction.word_id] = correction.corrected_word_id
|
||||
for old_seg, new_seg in zip(self._get_affected_segments(gap, segments), updated_segments):
|
||||
segment_id_map[old_seg.id] = new_seg.id
|
||||
step = CorrectionStep(
|
||||
handler_name="AgenticCorrector",
|
||||
affected_word_ids=affected_word_ids,
|
||||
affected_segment_ids=affected_segment_ids,
|
||||
corrections=_agentic_corrections,
|
||||
segments_before=self._get_affected_segments(gap, segments),
|
||||
segments_after=updated_segments,
|
||||
created_word_ids=[w.id for w in self._get_new_words(updated_segments, affected_word_ids)],
|
||||
deleted_word_ids=[id for id in affected_word_ids if not self._word_exists(id, updated_segments)],
|
||||
)
|
||||
correction_steps.append(step)
|
||||
all_corrections.extend(_agentic_corrections)
|
||||
# Log corrections made
|
||||
for correction in _agentic_corrections:
|
||||
self.logger.info(
|
||||
f"Made correction: '{correction.original_word}' -> '{correction.corrected_word}' "
|
||||
f"(confidence: {correction.confidence:.2f}, reason: {correction.reason})"
|
||||
)
|
||||
else:
|
||||
self.logger.debug(f"🤖 No agentic corrections needed for gap {i}")
|
||||
|
||||
# RULE-BASED MODE: Process gaps sequentially
|
||||
for i, gap in enumerate(gap_sequences, 1):
|
||||
# Check deadline before processing each gap (agentic mode only)
|
||||
# This allows us to abort early and return uncorrected results for human review
|
||||
if deadline and use_agentic_env and time.time() > deadline:
|
||||
self.logger.warning(
|
||||
f"⏰ AGENTIC TIMEOUT: Deadline exceeded after processing {i-1}/{len(gap_sequences)} gaps. "
|
||||
"Skipping remaining gaps - human review will correct any issues."
|
||||
)
|
||||
# Break out of loop - continue with whatever corrections we have (likely none)
|
||||
break
|
||||
# Skip if we already processed in agentic mode
|
||||
if use_agentic_env:
|
||||
continue
|
||||
|
||||
self.logger.info(f"Processing gap {i}/{len(gap_sequences)} at position {gap.transcription_position}")
|
||||
|
||||
@@ -460,128 +644,6 @@ class LyricsCorrector:
|
||||
gap_words = [word_map[word_id] for word_id in gap.transcribed_word_ids]
|
||||
self.logger.debug(f"Gap text: '{' '.join(w.text for w in gap_words)}'")
|
||||
|
||||
# AGENTIC-ONLY MODE: Use agentic correction exclusively
|
||||
if use_agentic_env:
|
||||
self.logger.info(f"🤖 Attempting agentic correction for gap {i}/{len(gap_sequences)}")
|
||||
try:
|
||||
# Prepare gap data for classification-first workflow
|
||||
gap_words_data = []
|
||||
for word_id in gap.transcribed_word_ids:
|
||||
if word_id in word_map:
|
||||
word = word_map[word_id]
|
||||
gap_words_data.append({
|
||||
"id": word_id,
|
||||
"text": word.text,
|
||||
"start_time": getattr(word, 'start_time', 0),
|
||||
"end_time": getattr(word, 'end_time', 0)
|
||||
})
|
||||
|
||||
# Get context words
|
||||
all_transcribed_words = []
|
||||
for seg in segments:
|
||||
all_transcribed_words.extend(seg.words)
|
||||
word_position = {w.id: idx for idx, w in enumerate(all_transcribed_words)}
|
||||
|
||||
gap_positions = [word_position[wid] for wid in gap.transcribed_word_ids if wid in word_position]
|
||||
preceding_words = ""
|
||||
following_words = ""
|
||||
|
||||
if gap_positions:
|
||||
first_gap_pos = min(gap_positions)
|
||||
last_gap_pos = max(gap_positions)
|
||||
|
||||
# Get 10 words before
|
||||
start_pos = max(0, first_gap_pos - 10)
|
||||
preceding_list = [all_transcribed_words[idx].text for idx in range(start_pos, first_gap_pos) if idx < len(all_transcribed_words)]
|
||||
preceding_words = " ".join(preceding_list)
|
||||
|
||||
# Get 10 words after
|
||||
end_pos = min(len(all_transcribed_words), last_gap_pos + 11)
|
||||
following_list = [all_transcribed_words[idx].text for idx in range(last_gap_pos + 1, end_pos) if idx < len(all_transcribed_words)]
|
||||
following_words = " ".join(following_list)
|
||||
|
||||
# Get reference contexts from all sources
|
||||
reference_contexts = {}
|
||||
for source, lyrics_data in self.reference_lyrics.items():
|
||||
if lyrics_data and lyrics_data.segments:
|
||||
ref_words = []
|
||||
for seg in lyrics_data.segments:
|
||||
ref_words.extend([w.text for w in seg.words])
|
||||
# For now, use full text (handlers will extract relevant portions)
|
||||
reference_contexts[source] = " ".join(ref_words)
|
||||
|
||||
# Get artist and title from metadata
|
||||
artist = metadata.get("artist") if metadata else None
|
||||
title = metadata.get("title") if metadata else None
|
||||
|
||||
# Choose model via router
|
||||
_router = _ModelRouter()
|
||||
uncertainty = 0.3 if len(gap_words_data) <= 2 else 0.7
|
||||
model_id = _router.choose_model("gap", uncertainty)
|
||||
self.logger.debug(f"🤖 Router selected model: {model_id}")
|
||||
|
||||
# Create agent and use new classification-first workflow
|
||||
self.logger.debug(f"🤖 Creating AgenticCorrector with model: {model_id}")
|
||||
_agent = _AgenticCorrector.from_model(
|
||||
model=model_id,
|
||||
session_id=session_id,
|
||||
cache_dir=str(self._cache_dir)
|
||||
)
|
||||
|
||||
# Use new propose_for_gap method
|
||||
self.logger.debug(f"🤖 Calling agent.propose_for_gap() for gap {i}")
|
||||
_proposals = _agent.propose_for_gap(
|
||||
gap_id=f"gap_{i}",
|
||||
gap_words=gap_words_data,
|
||||
preceding_words=preceding_words,
|
||||
following_words=following_words,
|
||||
reference_contexts=reference_contexts,
|
||||
artist=artist,
|
||||
title=title
|
||||
)
|
||||
self.logger.debug(f"🤖 Agent returned {len(_proposals) if _proposals else 0} proposals")
|
||||
_agentic_corrections = _adapt(_proposals, word_map, linear_position_map) if _proposals else []
|
||||
self.logger.debug(f"🤖 Adapter returned {len(_agentic_corrections)} corrections")
|
||||
|
||||
if _agentic_corrections:
|
||||
self.logger.info(f"🤖 Applying {len(_agentic_corrections)} agentic corrections for gap {i}")
|
||||
affected_word_ids = [w.id for w in self._get_affected_words(gap, segments)]
|
||||
affected_segment_ids = [s.id for s in self._get_affected_segments(gap, segments)]
|
||||
updated_segments = self._apply_corrections_to_segments(self._get_affected_segments(gap, segments), _agentic_corrections)
|
||||
for correction in _agentic_corrections:
|
||||
if correction.word_id and correction.corrected_word_id:
|
||||
word_id_map[correction.word_id] = correction.corrected_word_id
|
||||
for old_seg, new_seg in zip(self._get_affected_segments(gap, segments), updated_segments):
|
||||
segment_id_map[old_seg.id] = new_seg.id
|
||||
step = CorrectionStep(
|
||||
handler_name="AgenticCorrector",
|
||||
affected_word_ids=affected_word_ids,
|
||||
affected_segment_ids=affected_segment_ids,
|
||||
corrections=_agentic_corrections,
|
||||
segments_before=self._get_affected_segments(gap, segments),
|
||||
segments_after=updated_segments,
|
||||
created_word_ids=[w.id for w in self._get_new_words(updated_segments, affected_word_ids)],
|
||||
deleted_word_ids=[id for id in affected_word_ids if not self._word_exists(id, updated_segments)],
|
||||
)
|
||||
correction_steps.append(step)
|
||||
all_corrections.extend(_agentic_corrections)
|
||||
# Log corrections made
|
||||
for correction in _agentic_corrections:
|
||||
self.logger.info(
|
||||
f"Made correction: '{correction.original_word}' -> '{correction.corrected_word}' "
|
||||
f"(confidence: {correction.confidence:.2f}, reason: {correction.reason})"
|
||||
)
|
||||
else:
|
||||
self.logger.info(f"🤖 No agentic corrections needed for gap {i}")
|
||||
|
||||
except Exception as e:
|
||||
# In agentic-only mode, fail fast instead of falling back
|
||||
self.logger.error(f"🤖 Agentic correction failed for gap {i}: {e}", exc_info=True)
|
||||
raise RuntimeError(f"Agentic AI correction failed for gap {i}: {e}") from e
|
||||
|
||||
# Skip rule-based handlers completely in agentic mode
|
||||
continue
|
||||
|
||||
# RULE-BASED MODE: Try each handler in order
|
||||
for handler in self.handlers:
|
||||
handler_name = handler.__class__.__name__
|
||||
|
||||
@@ -274,11 +274,12 @@ class CorrectionOperations:
|
||||
audio_filepath: str,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
logger: Optional[logging.Logger] = None
|
||||
logger: Optional[logging.Logger] = None,
|
||||
ass_only: bool = False,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Generate a preview video with current corrections.
|
||||
|
||||
|
||||
Args:
|
||||
correction_result: Current correction result
|
||||
updated_data: Updated correction data for preview
|
||||
@@ -287,10 +288,12 @@ class CorrectionOperations:
|
||||
artist: Optional artist name
|
||||
title: Optional title
|
||||
logger: Optional logger instance
|
||||
|
||||
ass_only: If True, generate only ASS subtitles without video encoding.
|
||||
Useful when video encoding is offloaded to external service.
|
||||
|
||||
Returns:
|
||||
Dict with status, preview_hash, and video_path
|
||||
|
||||
Dict with status, preview_hash, and video_path (or ass_path if ass_only)
|
||||
|
||||
Raises:
|
||||
ValueError: If preview video generation fails
|
||||
"""
|
||||
@@ -338,15 +341,27 @@ class CorrectionOperations:
|
||||
audio_filepath=audio_filepath,
|
||||
artist=artist,
|
||||
title=title,
|
||||
ass_only=ass_only,
|
||||
)
|
||||
|
||||
|
||||
# When ass_only, we only need the ASS file (video encoding done externally)
|
||||
if ass_only:
|
||||
if not preview_outputs.ass:
|
||||
raise ValueError("Preview ASS generation failed")
|
||||
logger.info(f"Generated preview ASS: {preview_outputs.ass}")
|
||||
return {
|
||||
"status": "success",
|
||||
"preview_hash": preview_hash,
|
||||
"ass_path": preview_outputs.ass,
|
||||
}
|
||||
|
||||
if not preview_outputs.video:
|
||||
raise ValueError("Preview video generation failed")
|
||||
|
||||
|
||||
logger.info(f"Generated preview video: {preview_outputs.video}")
|
||||
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"preview_hash": preview_hash,
|
||||
"video_path": preview_outputs.video
|
||||
"video_path": preview_outputs.video,
|
||||
}
|
||||
@@ -2,7 +2,7 @@
|
||||
"name": "lyrics-transcriber-frontend",
|
||||
"private": true,
|
||||
"homepage": "https://nomadkaraoke.github.io/lyrics-transcriber-frontend",
|
||||
"version": "0.83.0",
|
||||
"version": "0.84.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
|
||||
@@ -1 +1 @@
|
||||
{"root":["./src/app.tsx","./src/api.ts","./src/main.tsx","./src/theme.ts","./src/types.ts","./src/validation.ts","./src/vite-env.d.ts","./src/components/aifeedbackmodal.tsx","./src/components/addlyricsmodal.tsx","./src/components/agenticcorrectionmetrics.tsx","./src/components/audioplayer.tsx","./src/components/correctedwordwithactions.tsx","./src/components/correctionannotationmodal.tsx","./src/components/correctiondetailcard.tsx","./src/components/correctionmetrics.tsx","./src/components/durationtimelineview.tsx","./src/components/editactionbar.tsx","./src/components/editmodal.tsx","./src/components/edittimelinesection.tsx","./src/components/editwordlist.tsx","./src/components/fileupload.tsx","./src/components/findreplacemodal.tsx","./src/components/header.tsx","./src/components/lyricsanalyzer.tsx","./src/components/metricsdashboard.tsx","./src/components/modeselectionmodal.tsx","./src/components/modeselector.tsx","./src/components/modelselector.tsx","./src/components/previewvideosection.tsx","./src/components/referenceview.tsx","./src/components/replacealllyricsmodal.tsx","./src/components/reviewchangesmodal.tsx","./src/components/segmentdetailsmodal.tsx","./src/components/timelineeditor.tsx","./src/components/timingoffsetmodal.tsx","./src/components/transcriptionview.tsx","./src/components/worddivider.tsx","./src/components/lyricssynchronizer/synccontrols.tsx","./src/components/lyricssynchronizer/timelinecanvas.tsx","./src/components/lyricssynchronizer/upcomingwordsbar.tsx","./src/components/lyricssynchronizer/index.tsx","./src/components/shared/constants.ts","./src/components/shared/styles.ts","./src/components/shared/types.ts","./src/components/shared/components/highlightedtext.tsx","./src/components/shared/components/sourceselector.tsx","./src/components/shared/components/word.tsx","./src/components/shared/hooks/usewordclick.ts","./src/components/shared/utils/keyboardhandlers.ts","./src/components/shared/utils/localstorage.ts","./src/components/shared/utils/referencelinecalculator.ts","./src/components/shared/utils/segmentoperations.ts","./src/components/shared/utils/timingutils.ts","./src/components/shared/utils/wordutils.ts","./src/hooks/usemanualsync.ts","./src/types/global.d.ts"],"version":"5.6.3"}
|
||||
{"root":["./src/app.tsx","./src/api.ts","./src/main.tsx","./src/theme.ts","./src/types.ts","./src/validation.ts","./src/vite-env.d.ts","./src/components/aifeedbackmodal.tsx","./src/components/addlyricsmodal.tsx","./src/components/agenticcorrectionmetrics.tsx","./src/components/appheader.tsx","./src/components/audioplayer.tsx","./src/components/correctedwordwithactions.tsx","./src/components/correctionannotationmodal.tsx","./src/components/correctiondetailcard.tsx","./src/components/correctionmetrics.tsx","./src/components/durationtimelineview.tsx","./src/components/editactionbar.tsx","./src/components/editmodal.tsx","./src/components/edittimelinesection.tsx","./src/components/editwordlist.tsx","./src/components/fileupload.tsx","./src/components/findreplacemodal.tsx","./src/components/header.tsx","./src/components/lyricsanalyzer.tsx","./src/components/metricsdashboard.tsx","./src/components/modeselectionmodal.tsx","./src/components/modeselector.tsx","./src/components/modelselector.tsx","./src/components/previewvideosection.tsx","./src/components/referenceview.tsx","./src/components/replacealllyricsmodal.tsx","./src/components/reviewchangesmodal.tsx","./src/components/segmentdetailsmodal.tsx","./src/components/timelineeditor.tsx","./src/components/timingoffsetmodal.tsx","./src/components/transcriptionview.tsx","./src/components/worddivider.tsx","./src/components/lyricssynchronizer/synccontrols.tsx","./src/components/lyricssynchronizer/timelinecanvas.tsx","./src/components/lyricssynchronizer/upcomingwordsbar.tsx","./src/components/lyricssynchronizer/index.tsx","./src/components/shared/constants.ts","./src/components/shared/styles.ts","./src/components/shared/types.ts","./src/components/shared/components/highlightedtext.tsx","./src/components/shared/components/sourceselector.tsx","./src/components/shared/components/word.tsx","./src/components/shared/hooks/usewordclick.ts","./src/components/shared/utils/keyboardhandlers.ts","./src/components/shared/utils/localstorage.ts","./src/components/shared/utils/referencelinecalculator.ts","./src/components/shared/utils/segmentoperations.ts","./src/components/shared/utils/timingutils.ts","./src/components/shared/utils/wordutils.ts","./src/hooks/usemanualsync.ts","./src/types/global.d.ts"],"version":"5.6.3"}
|
||||
@@ -3,6 +3,11 @@ import react from '@vitejs/plugin-react';
|
||||
// https://vite.dev/config/
|
||||
export default defineConfig({
|
||||
plugins: [react()],
|
||||
server: {
|
||||
port: 5173,
|
||||
// Enable CORS for local development
|
||||
cors: true,
|
||||
},
|
||||
build: {
|
||||
minify: false,
|
||||
sourcemap: true,
|
||||
|
||||
-1
@@ -1 +0,0 @@
|
||||
{"version":3,"file":"index-BECn1o8Q.js","sources":["../../node_modules/react/cjs/react.production.min.js","../../node_modules/react/index.js","../../node_modules/react/cjs/react-jsx-runtime.production.min.js","../../node_modules/react/jsx-runtime.js","../../node_modules/scheduler/cjs/scheduler.production.min.js","../../node_modules/scheduler/index.js","../../node_modules/react-dom/cjs/react-dom.production.min.js","../../node_modules/react-dom/index.js","../../node_modules/react-dom/client.js","../../node_modules/@mui/utils/esm/formatMuiErrorMessage/formatMuiErrorMessage.js","../../node_modules/@mui/material/styles/identifier.js","../../node_modules/@babel/runtime/helpers/esm/extends.js","../../node_modules/@emotion/sheet/dist/emotion-sheet.esm.js","../../node_modules/stylis/src/Enum.js","../../node_modules/stylis/src/Utility.js","../../node_modules/stylis/src/Tokenizer.js","../../node_modules/stylis/src/Parser.js","../../node_modules/stylis/src/Serializer.js","../../node_modules/stylis/src/Middleware.js","../../node_modules/@emotion/memoize/dist/emotion-memoize.esm.js","../../node_modules/@emotion/cache/dist/emotion-cache.browser.esm.js","../../node_modules/react-is/cjs/react-is.production.min.js","../../node_modules/react-is/index.js","../../node_modules/hoist-non-react-statics/dist/hoist-non-react-statics.cjs.js","../../node_modules/@emotion/utils/dist/emotion-utils.browser.esm.js","../../node_modules/@emotion/hash/dist/emotion-hash.esm.js","../../node_modules/@emotion/unitless/dist/emotion-unitless.esm.js","../../node_modules/@emotion/serialize/dist/emotion-serialize.esm.js","../../node_modules/@emotion/use-insertion-effect-with-fallbacks/dist/emotion-use-insertion-effect-with-fallbacks.browser.esm.js","../../node_modules/@emotion/react/dist/emotion-element-f0de968e.browser.esm.js","../../node_modules/@emotion/react/dist/emotion-react.browser.esm.js","../../node_modules/@emotion/is-prop-valid/dist/emotion-is-prop-valid.esm.js","../../node_modules/@emotion/styled/base/dist/emotion-styled-base.browser.esm.js","../../node_modules/@emotion/styled/dist/emotion-styled.browser.esm.js","../../node_modules/@mui/styled-engine/GlobalStyles/GlobalStyles.js","../../node_modules/@mui/styled-engine/index.js","../../node_modules/@mui/utils/node_modules/react-is/cjs/react-is.production.js","../../node_modules/@mui/utils/node_modules/react-is/index.js","../../node_modules/@mui/utils/esm/deepmerge/deepmerge.js","../../node_modules/@mui/system/esm/createBreakpoints/createBreakpoints.js","../../node_modules/@mui/system/esm/cssContainerQueries/cssContainerQueries.js","../../node_modules/@mui/system/esm/createTheme/shape.js","../../node_modules/@mui/system/esm/merge/merge.js","../../node_modules/@mui/system/esm/breakpoints/breakpoints.js","../../node_modules/@mui/utils/esm/capitalize/capitalize.js","../../node_modules/@mui/system/esm/style/style.js","../../node_modules/@mui/system/esm/memoize/memoize.js","../../node_modules/@mui/system/esm/spacing/spacing.js","../../node_modules/@mui/system/esm/createTheme/createSpacing.js","../../node_modules/@mui/system/esm/compose/compose.js","../../node_modules/@mui/system/esm/borders/borders.js","../../node_modules/@mui/system/esm/cssGrid/cssGrid.js","../../node_modules/@mui/system/esm/palette/palette.js","../../node_modules/@mui/system/esm/sizing/sizing.js","../../node_modules/@mui/system/esm/styleFunctionSx/defaultSxConfig.js","../../node_modules/@mui/system/esm/styleFunctionSx/styleFunctionSx.js","../../node_modules/@mui/system/esm/createTheme/applyStyles.js","../../node_modules/@mui/system/esm/createTheme/createTheme.js","../../node_modules/@mui/system/esm/useThemeWithoutDefault/useThemeWithoutDefault.js","../../node_modules/@mui/system/esm/useTheme/useTheme.js","../../node_modules/@mui/system/esm/GlobalStyles/GlobalStyles.js","../../node_modules/@mui/system/esm/styleFunctionSx/extendSxProp.js","../../node_modules/@mui/utils/esm/ClassNameGenerator/ClassNameGenerator.js","../../node_modules/clsx/dist/clsx.mjs","../../node_modules/@mui/system/esm/createBox/createBox.js","../../node_modules/@mui/utils/esm/generateUtilityClass/generateUtilityClass.js","../../node_modules/@mui/utils/esm/generateUtilityClasses/generateUtilityClasses.js","../../node_modules/@mui/system/esm/preprocessStyles.js","../../node_modules/@mui/system/esm/createStyled/createStyled.js","../../node_modules/@mui/system/esm/styled/styled.js","../../node_modules/@mui/utils/esm/resolveProps/resolveProps.js","../../node_modules/@mui/system/esm/useThemeProps/getThemeProps.js","../../node_modules/@mui/system/esm/useThemeProps/useThemeProps.js","../../node_modules/@mui/utils/esm/useEnhancedEffect/useEnhancedEffect.js","../../node_modules/@mui/system/esm/useMediaQuery/useMediaQuery.js","../../node_modules/@mui/utils/esm/clamp/clamp.js","../../node_modules/@mui/system/esm/colorManipulator/colorManipulator.js","../../node_modules/@mui/utils/esm/createChainedFunction/createChainedFunction.js","../../node_modules/@mui/utils/esm/debLine truncated
|
||||
+5248
-4480
File diff suppressed because it is too large.
Load diff
+1
@@ -0,0 +1 @@
|
||||
{"version":3,"file":"index-V91xgc4E.js","sources":["../../node_modules/react/cjs/react.production.min.js","../../node_modules/react/index.js","../../node_modules/react/cjs/react-jsx-runtime.production.min.js","../../node_modules/react/jsx-runtime.js","../../node_modules/scheduler/cjs/scheduler.production.min.js","../../node_modules/scheduler/index.js","../../node_modules/react-dom/cjs/react-dom.production.min.js","../../node_modules/react-dom/index.js","../../node_modules/react-dom/client.js","../../node_modules/@mui/utils/node_modules/react-is/cjs/react-is.production.js","../../node_modules/@mui/utils/node_modules/react-is/index.js","../../node_modules/@mui/utils/esm/deepmerge/deepmerge.js","../../node_modules/@mui/utils/esm/formatMuiErrorMessage/formatMuiErrorMessage.js","../../node_modules/@mui/utils/esm/capitalize/capitalize.js","../../node_modules/@mui/utils/esm/createChainedFunction/createChainedFunction.js","../../node_modules/@mui/utils/esm/debounce/debounce.js","../../node_modules/@mui/utils/esm/isMuiElement/isMuiElement.js","../../node_modules/@mui/utils/esm/ownerDocument/ownerDocument.js","../../node_modules/@mui/utils/esm/ownerWindow/ownerWindow.js","../../node_modules/@mui/utils/esm/setRef/setRef.js","../../node_modules/@mui/utils/esm/useEnhancedEffect/useEnhancedEffect.js","../../node_modules/@mui/utils/esm/useId/useId.js","../../node_modules/@mui/utils/esm/useControlled/useControlled.js","../../node_modules/@mui/utils/esm/useEventCallback/useEventCallback.js","../../node_modules/@mui/utils/esm/useForkRef/useForkRef.js","../../node_modules/@mui/utils/esm/useLazyRef/useLazyRef.js","../../node_modules/@mui/utils/esm/useOnMount/useOnMount.js","../../node_modules/@mui/utils/esm/useTimeout/useTimeout.js","../../node_modules/@mui/utils/esm/isFocusVisible/isFocusVisible.js","../../node_modules/@mui/utils/esm/getScrollbarSize/getScrollbarSize.js","../../node_modules/@mui/utils/esm/getValidReactChildren/getValidReactChildren.js","../../node_modules/@mui/utils/esm/visuallyHidden/visuallyHidden.js","../../node_modules/@mui/utils/esm/resolveProps/resolveProps.js","../../node_modules/@mui/utils/esm/composeClasses/composeClasses.js","../../node_modules/@mui/utils/esm/ClassNameGenerator/ClassNameGenerator.js","../../node_modules/@mui/utils/esm/generateUtilityClass/generateUtilityClass.js","../../node_modules/@mui/utils/esm/generateUtilityClasses/generateUtilityClasses.js","../../node_modules/@mui/utils/esm/clamp/clamp.js","../../node_modules/@mui/utils/esm/isHostComponent/isHostComponent.js","../../node_modules/@mui/utils/esm/appendOwnerState/appendOwnerState.js","../../node_modules/clsx/dist/clsx.mjs","../../node_modules/@mui/utils/esm/extractEventHandlers/extractEventHandlers.js","../../node_modules/@mui/utils/esm/omitEventHandlers/omitEventHandlers.js","../../node_modules/@mui/utils/esm/mergeSlotProps/mergeSlotProps.js","../../node_modules/@mui/utils/esm/resolveComponentProps/resolveComponentProps.js","../../node_modules/@mui/utils/esm/useSlotProps/useSlotProps.js","../../node_modules/@mui/utils/esm/getReactElementRef/getReactElementRef.js","../../node_modules/@mui/system/esm/merge/merge.js","../../node_modules/@mui/system/esm/cssContainerQueries/cssContainerQueries.js","../../node_modules/@mui/system/esm/breakpoints/breakpoints.js","../../node_modules/@mui/system/esm/style/style.js","../../node_modules/@mui/system/esm/memoize/memoize.js","../../node_modules/@mui/system/esm/spacing/spacing.js","../../node_modules/@mui/system/esm/compose/compose.js","../../node_modules/@mui/system/esm/borders/borders.js","../../node_modules/@mui/system/esm/cssGrid/cssGrid.js","../../node_modules/@mui/system/esm/palette/palette.js","../../node_modules/@mui/system/esm/sizing/sizing.js","../../node_modules/@mui/system/esm/styleFunctionSx/defaultSxConfig.js","../../node_modules/@mui/system/esm/styleFunctionSx/styleFunctionSx.js","../../node_modules/@mui/system/esm/styleFunctionSx/extendSxProp.js","../../node_modules/@babel/runtime/helpers/esm/extends.js","../../node_modules/@emotion/sheet/dist/emotion-sheet.esm.js","../../node_modules/stylis/src/Enum.js","../../node_modules/stylis/src/Utility.js","../../node_modules/stylis/src/Tokenizer.js","../../node_modules/stylis/src/Parser.js","../../node_modules/stylis/src/Serializer.js","../../node_modules/stylis/src/Middleware.js","../../node_modules/@emotion/memoize/dist/emotion-memoize.esm.js","../../node_modules/@emotion/cache/dist/emotion-cache.browser.esm.js","../../node_modules/react-is/cjs/react-is.production.min.js","../../node_modules/react-is/index.js","../../node_modules/hoist-non-react-statics/dist/hoist-non-react-statics.cjs.js","../../node_modules/@emotion/utils/dist/emotion-utils.browser.esm.js","../../node_modules/@emotion/hash/dist/emotion-hash.esm.js","../../node_modules/@emotion/unitless/dist/emotion-unitless.esm.js","../../node_modules/@emotion/serialize/dist/emotion-serialize.esm.js","../../node_modules/@emotion/use-insertion-effect-with-fallbacks/dist/emotion-useLine truncated
|
||||
@@ -2,6 +2,10 @@
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<!-- Load Inter font to match karaoke-gen styling -->
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com">
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
||||
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700&display=swap" rel="stylesheet">
|
||||
<link rel="icon" type="image/x-icon" href="/favicon.ico" />
|
||||
<link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png" />
|
||||
<link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png" />
|
||||
@@ -9,8 +13,8 @@
|
||||
<link rel="icon" type="image/png" sizes="192x192" href="/android-chrome-192x192.png" />
|
||||
<link rel="icon" type="image/png" sizes="512x512" href="/android-chrome-512x512.png" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>Nomad Karaoke: Lyrics Review</title>
|
||||
<script type="module" crossorigin src="/assets/index-BECn1o8Q.js"></script>
|
||||
<title>Nomad Karaoke: Lyrics Transcription Review</title>
|
||||
<script type="module" crossorigin src="/assets/index-V91xgc4E.js"></script>
|
||||
</head>
|
||||
<body>
|
||||
<div id="root"></div>
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" version="1.1" viewBox="45 60 255 135">
|
||||
<g data-v-70b83f88="" fill="#ff7acc" class="basesvg" transform="translate(48.368003845214844,62.62239074707031)"><g fill-rule="" class="tp-name" transform="translate(0,0)"><g transform="scale(1.4000000000000004)"><g stroke="#ff7acc" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" paint-order="stroke" data-gra-attr="stroke" fill-opacity="0"><path d="M1.4-33.44L1.4 0C31.48 0 3.96 0 34.04 0L34.04-32.64 17.81-32.64 17.81-20.61 1.4-33.44ZM52.21-33.76C42.6-33.76 34.81-25.97 34.81-16.37 34.81-6.76 42.6 1.03 52.21 1.03 61.81 1.03 69.6-6.76 69.6-16.37 69.6-25.97 61.81-33.76 52.21-33.76ZM70.7 0L103.34 0 103.25-33.9 86.93-21.54 70.61-33.9 70.7 0ZM102.34 0C138.2 0 107.52 0 143.56 0L122.95-34.93 102.34 0ZM158.84 0.05C167.84 0.05 175.16-7.27 175.16-16.27 175.16-25.27 167.84-32.64 158.84-32.64L142.52-32.64C142.52 0.75 142.52-33.34 142.52 0.05L158.84 0.05Z" transform="translate(-1.399999976158142, 34.93000030517578)"></path></g><g transform="translate(0,38.959999084472656)"><g stroke="#ff7acc" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" paint-order="stroke" data-gra-attr="stroke" fill="#ff7acc" fill-opacity="0" transform="scale(1.02719)"><path d="M24.43 0L17.67 0 7.98-11.87 7.98 0 2.63 0 2.63-26.64 7.98-26.64 7.98-14.69 17.67-26.64 24.12-26.64 13.13-13.44 24.43 0ZM45.03 0L43.27-5.08 32.66-5.08 30.91 0 25.3 0 34.88-26.68 41.1-26.68 50.68 0 45.03 0ZM34.11-9.35L41.82-9.35 37.97-20.5 34.11-9.35ZM73.19 0L67.01 0 61.13-10.38 58.61-10.38 58.61 0 53.27 0 53.27-26.64 63.27-26.64Q66.36-26.64 68.53-25.55 70.71-24.47 71.8-22.61 72.89-20.76 72.89-18.47L72.89-18.47Q72.89-15.84 71.36-13.72 69.83-11.6 66.82-10.8L66.82-10.8 73.19 0ZM58.61-22.21L58.61-14.39 63.08-14.39Q65.25-14.39 66.32-15.44 67.39-16.49 67.39-18.36L67.39-18.36Q67.39-20.19 66.32-21.2 65.25-22.21 63.08-22.21L63.08-22.21 58.61-22.21ZM94.82 0L93.07-5.08 82.46-5.08 80.7 0 75.09 0 84.67-26.68 90.89-26.68 100.47 0 94.82 0ZM83.91-9.35L91.62-9.35 87.76-20.5 83.91-9.35ZM115.43 0.27Q111.69 0.27 108.56-1.49 105.43-3.24 103.6-6.35 101.76-9.47 101.76-13.4L101.76-13.4Q101.76-17.29 103.6-20.4 105.43-23.51 108.56-25.27 111.69-27.02 115.43-27.02L115.43-27.02Q119.21-27.02 122.32-25.27 125.43-23.51 127.24-20.4 129.05-17.29 129.05-13.4L129.05-13.4Q129.05-9.47 127.24-6.35 125.43-3.24 122.3-1.49 119.17 0.27 115.43 0.27L115.43 0.27ZM115.43-4.5Q117.83-4.5 119.66-5.59 121.5-6.68 122.53-8.7 123.56-10.73 123.56-13.4L123.56-13.4Q123.56-16.07 122.53-18.07 121.5-20.08 119.66-21.15 117.83-22.21 115.43-22.21L115.43-22.21Q113.02-22.21 111.17-21.15 109.32-20.08 108.29-18.07 107.26-16.07 107.26-13.4L107.26-13.4Q107.26-10.73 108.29-8.7 109.32-6.68 111.17-5.59 113.02-4.5 115.43-4.5L115.43-4.5ZM153.82 0L147.06 0 137.37-11.87 137.37 0 132.02 0 132.02-26.64 137.37-26.64 137.37-14.69 147.06-26.64 153.51-26.64 142.52-13.44 153.82 0ZM171.79-22.33L161.67-22.33 161.67-15.65 170.64-15.65 170.64-11.41 161.67-11.41 161.67-4.35 171.79-4.35 171.79 0 156.33 0 156.33-26.68 171.79-26.68 171.79-22.33Z" transform="translate(-2.630000114440918, 27.020000457763672)"></path></g></g></g></g> <g data-gra="path-slogan" fill-rule="" class="tp-slogan" fill="#ffdf6b" transform="translate(5,111.788818359375)"><rect x="0" height="1" y="5.9832000732421875" width="14.278396606445312"></rect> <rect height="1" y="5.9832000732421875" width="14.278396606445312" x="218.985595703125"></rect> <g transform="translate(17.278396606445312,0)"><g transform="scale(1.2800000000000002)"><path d="M10.21-8.38L12.02-8.38L9.68 0L7.70 0L6.13-5.96L4.49 0L2.52 0.01L0.26-8.38L2.06-8.38L3.54-1.87L5.24-8.38L7.12-8.38L8.72-1.91L10.21-8.38ZM18.38-8.38L20.06-8.38L20.06 0L18.38 0L18.38-3.56L14.80-3.56L14.80 0L13.12 0L13.12-8.38L14.80-8.38L14.80-4.93L18.38-4.93L18.38-8.38ZM26.58-7.02L23.40-7.02L23.40-4.92L26.22-4.92L26.22-3.59L23.40-3.59L23.40-1.37L26.58-1.37L26.58 0L21.72 0L21.72-8.39L26.58-8.39L26.58-7.02ZM34.37 0L32.42 0L30.58-3.26L29.78-3.26L29.78 0L28.10 0L28.10-8.38L31.25-8.38Q32.22-8.38 32.90-8.03Q33.59-7.69 33.93-7.11Q34.27-6.53 34.27-5.81L34.27-5.81Q34.27-4.98 33.79-4.31Q33.31-3.65 32.36-3.40L32.36-3.40L34.37 0ZM29.78-6.98L29.78-4.52L31.19-4.52Q31.87-4.52 32.21-4.85Q32.54-5.18 32.54-5.77L32.54-5.77Q32.54-6.35 32.21-6.67Q31.87-6.98 31.19-6.98L31.19-6.98L29.78-6.98ZM40.66-7.02L37.48-7.02L37.48-4.92L40.30-4.92L40.30-3.59L37.48-3.59L37.48-1.37L40.66-1.37L40.66 0L35.80 0L35.80-8.39L40.66-8.39L40.66-7.02ZM47.92-8.38L49.70-8.38L46.63 0L44.59 0L41.52-8.38L43.32-8.38L45.62-1.72L47.92-8.38ZM55.57-7.02L52.39-7.02L52.39-4.92L55.21-4.92L55.21-3.59L52.39-3.59L52.39-1.37L55.57-1.37L55.57 0L50.71 0L50.71-8.39L55.57-8.39L55.57-7.02ZM63.36 0L61.42 0L59.57-3.26L58.78-3.26L58.78 0L57.10 0L57.10-8.38L60.24-8.38Q61.21-8.38 61.90-8.03Q62.58-7.69 62.92-7.11Q63.26-6.53 63.26-5.81L63.26-5.81Q63.26-4.98 62.78-4.31Q62.30-3.65 61.36-3.40L61.36-3.40L63.36 0ZM58.78-6.98L58.78-4.52L60.18-4.52Q60.86-4.52 61.20-4.85Q61.54-5.18 61.54-5.77L61.54-5.77Q61.54-6.35 61.20-6.67Q60.86-6.98 60.18-6.98L60.18-6.98L58.78-6.98ZM72.43-8.38L74.30-8.38L71.47-2.92L71.47 0L69.Line truncated
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 9.0 KiB |
@@ -148,8 +148,21 @@ class OutputGenerator:
|
||||
audio_filepath: str,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
ass_only: bool = False,
|
||||
) -> OutputPaths:
|
||||
"""Generate all requested output formats."""
|
||||
"""Generate all requested output formats.
|
||||
|
||||
Args:
|
||||
transcription_corrected: Corrected transcription data
|
||||
lyrics_results: Lyrics data from various providers
|
||||
output_prefix: Prefix for output filenames
|
||||
audio_filepath: Path to audio file
|
||||
artist: Optional artist name
|
||||
title: Optional title
|
||||
ass_only: If True (only in preview_mode), generate only ASS subtitles
|
||||
without video encoding. Useful when video encoding is offloaded
|
||||
to an external service.
|
||||
"""
|
||||
outputs = OutputPaths()
|
||||
|
||||
try:
|
||||
@@ -165,8 +178,9 @@ class OutputGenerator:
|
||||
# Generate ASS subtitles for preview
|
||||
outputs.ass = self.subtitle.generate_ass(transcription_corrected.resized_segments, output_prefix, audio_filepath)
|
||||
|
||||
# Generate preview video
|
||||
outputs.video = self.video.generate_preview_video(outputs.ass, audio_filepath, output_prefix)
|
||||
# Generate preview video (unless ass_only mode for GCE encoding)
|
||||
if not ass_only:
|
||||
outputs.video = self.video.generate_preview_video(outputs.ass, audio_filepath, output_prefix)
|
||||
|
||||
return outputs
|
||||
|
||||
|
||||
@@ -249,6 +249,9 @@ class VideoGenerator:
|
||||
def generate_preview_video(self, ass_path: str, audio_path: str, output_prefix: str) -> str:
|
||||
"""Generate lower resolution MP4 preview video with lyrics overlay.
|
||||
|
||||
Delegates to LocalPreviewEncodingService for the actual encoding to ensure
|
||||
consistent preview generation across local CLI, Cloud Run, and GCE worker.
|
||||
|
||||
Args:
|
||||
ass_path: Path to ASS subtitles file
|
||||
audio_path: Path to audio file
|
||||
@@ -257,6 +260,11 @@ class VideoGenerator:
|
||||
Returns:
|
||||
Path to generated preview video file
|
||||
"""
|
||||
from backend.services.local_preview_encoding_service import (
|
||||
LocalPreviewEncodingService,
|
||||
PreviewEncodingConfig,
|
||||
)
|
||||
|
||||
self.logger.info("Generating preview video with lyrics overlay")
|
||||
output_path = os.path.join(self.cache_dir, f"{output_prefix}_preview.mp4")
|
||||
|
||||
@@ -269,17 +277,36 @@ class VideoGenerator:
|
||||
try:
|
||||
# Create a temporary copy of the ASS file with a unique filename
|
||||
import time
|
||||
import shutil
|
||||
|
||||
safe_prefix = "".join(c if c.isalnum() else "_" for c in output_prefix)
|
||||
timestamp = int(time.time() * 1000)
|
||||
temp_ass_path = os.path.join(self.cache_dir, f"temp_preview_subtitles_{safe_prefix}_{timestamp}.ass")
|
||||
import shutil
|
||||
|
||||
shutil.copy2(ass_path, temp_ass_path)
|
||||
self.logger.debug(f"Created temporary ASS file: {temp_ass_path}")
|
||||
|
||||
cmd = self._build_preview_ffmpeg_command(temp_ass_path, audio_path, output_path)
|
||||
self._run_ffmpeg_command(cmd)
|
||||
# Get font path from styles configuration
|
||||
karaoke_styles = self.styles.get("karaoke", {})
|
||||
font_path = karaoke_styles.get("font_path")
|
||||
|
||||
# Build config for LocalPreviewEncodingService
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path=temp_ass_path,
|
||||
audio_path=audio_path,
|
||||
output_path=output_path,
|
||||
background_image_path=self.background_image,
|
||||
background_color=self.background_color,
|
||||
font_path=font_path if font_path and os.path.isfile(font_path) else None,
|
||||
)
|
||||
|
||||
# Use LocalPreviewEncodingService for consistent encoding
|
||||
service = LocalPreviewEncodingService(logger=self.logger)
|
||||
result = service.encode_preview(config)
|
||||
|
||||
if not result.success:
|
||||
raise RuntimeError(f"Preview encoding failed: {result.error}")
|
||||
|
||||
self.logger.info(f"Preview video generated: {output_path}")
|
||||
|
||||
# Clean up temporary file
|
||||
@@ -352,19 +379,44 @@ class VideoGenerator:
|
||||
self.logger.error(f"Failed to resize background image: {e.output}")
|
||||
raise
|
||||
|
||||
def _escape_ffmpeg_filter_path(self, path: str) -> str:
|
||||
"""Escape a path for FFmpeg filter expressions (for subprocess without shell).
|
||||
|
||||
When using subprocess with a command list (no shell), FFmpeg receives the
|
||||
filter string directly. FFmpeg's filter parser requires escaping:
|
||||
- Backslashes: double them (\ -> \\)
|
||||
- Single quotes/apostrophes: escape with three backslashes (' -> \\')
|
||||
- Spaces: escape with backslash ( -> \ )
|
||||
|
||||
Note: This is different from shell escaping. The '\\'\\''' pattern used for
|
||||
shell escaping does NOT work when subprocess passes args directly to FFmpeg.
|
||||
|
||||
Example: "I'm With You" becomes "I\\\\'m\\ With\\ You"
|
||||
"""
|
||||
# First escape existing backslashes (\ -> \\)
|
||||
escaped = path.replace("\\", "\\\\")
|
||||
# Escape single quotes (' -> \\')
|
||||
# In the actual string we need 3 backslashes before the quote
|
||||
escaped = escaped.replace("'", "\\\\\\'")
|
||||
# Escape spaces
|
||||
escaped = escaped.replace(" ", "\\ ")
|
||||
return escaped
|
||||
|
||||
def _build_ass_filter(self, ass_path: str) -> str:
|
||||
"""Build ASS filter with font directory support."""
|
||||
ass_filter = f"ass={ass_path}"
|
||||
|
||||
escaped_ass_path = self._escape_ffmpeg_filter_path(ass_path)
|
||||
ass_filter = f"ass={escaped_ass_path}"
|
||||
|
||||
# Get font path from styles configuration
|
||||
karaoke_styles = self.styles.get("karaoke", {})
|
||||
font_path = karaoke_styles.get("font_path")
|
||||
|
||||
|
||||
if font_path and os.path.isfile(font_path):
|
||||
font_dir = os.path.dirname(font_path)
|
||||
ass_filter += f":fontsdir={font_dir}"
|
||||
escaped_font_dir = self._escape_ffmpeg_filter_path(font_dir)
|
||||
ass_filter += f":fontsdir={escaped_font_dir}"
|
||||
self.logger.info(f"Returning ASS filter with fonts dir: {ass_filter}")
|
||||
|
||||
|
||||
return ass_filter
|
||||
|
||||
def _build_ffmpeg_command(self, ass_path: str, audio_path: str, output_path: str) -> List[str]:
|
||||
@@ -439,93 +491,6 @@ class VideoGenerator:
|
||||
|
||||
return cmd
|
||||
|
||||
def _build_preview_ffmpeg_command(self, ass_path: str, audio_path: str, output_path: str) -> List[str]:
|
||||
"""Build FFmpeg command for preview video generation with hardware acceleration when available."""
|
||||
# Use even lower resolution for preview (480x270 instead of 640x360 for faster encoding)
|
||||
width, height = 480, 270
|
||||
|
||||
cmd = [
|
||||
"ffmpeg",
|
||||
"-hide_banner",
|
||||
"-loglevel", "error",
|
||||
"-r", "24", # Reduced frame rate to 24 fps for faster encoding
|
||||
]
|
||||
|
||||
# Add hardware acceleration flags if available
|
||||
cmd.extend(self.hwaccel_flags)
|
||||
|
||||
# Input source (background) - simplified for preview
|
||||
if self.background_image:
|
||||
# For preview, use the original image without resizing to save time
|
||||
self.logger.debug(f"Using original background image for preview: {self.background_image}")
|
||||
cmd.extend([
|
||||
"-loop", "1", # Loop the image
|
||||
"-i", self.background_image,
|
||||
])
|
||||
# Build video filter with scaling and ASS subtitles
|
||||
video_filter = f"scale={width}:{height}:force_original_aspect_ratio=decrease,pad={width}:{height}:(ow-iw)/2:(oh-ih)/2,{self._build_ass_filter(ass_path)}"
|
||||
else:
|
||||
self.logger.debug(
|
||||
f"Using solid {self.background_color} background "
|
||||
f"with resolution: {width}x{height}"
|
||||
)
|
||||
cmd.extend([
|
||||
"-f", "lavfi",
|
||||
"-i", f"color=c={self.background_color}:s={width}x{height}:r=24",
|
||||
])
|
||||
# Build video filter with just ASS subtitles (no scaling needed)
|
||||
video_filter = self._build_ass_filter(ass_path)
|
||||
|
||||
cmd.extend([
|
||||
"-i", audio_path,
|
||||
"-vf", video_filter, # Apply the video filter
|
||||
"-c:a", "aac", # Use AAC for audio compatibility
|
||||
"-b:a", "96k", # Reduced audio bitrate for faster encoding
|
||||
"-c:v", self.video_encoder,
|
||||
])
|
||||
|
||||
# Add encoder-specific settings for preview with maximum speed priority
|
||||
if self.nvenc_available:
|
||||
# NVENC settings optimized for maximum speed
|
||||
cmd.extend([
|
||||
"-preset", "p1", # Fastest NVENC preset
|
||||
"-tune", "ll", # Low latency
|
||||
"-rc", "cbr", # Constant bitrate for speed
|
||||
"-b:v", "800k", # Lower bitrate for speed
|
||||
"-profile:v", "baseline", # Most compatible profile
|
||||
"-level", "3.1", # Lower level for speed
|
||||
])
|
||||
self.logger.debug("Using NVENC encoding with maximum speed settings for preview video generation")
|
||||
else:
|
||||
# Software encoding with maximum speed priority
|
||||
cmd.extend([
|
||||
"-profile:v", "baseline", # Most compatible H.264 profile
|
||||
"-level", "3.0", # Compatibility level
|
||||
"-preset", "superfast", # Even faster than ultrafast for preview
|
||||
"-tune", "fastdecode", # Optimize for fast decoding
|
||||
"-b:v", "600k", # Lower base bitrate for speed
|
||||
"-maxrate", "800k", # Lower max bitrate
|
||||
"-bufsize", "1200k", # Smaller buffer size
|
||||
"-crf", "28", # Higher CRF for faster encoding (lower quality but faster)
|
||||
])
|
||||
self.logger.debug("Using software encoding with maximum speed settings for preview video generation")
|
||||
|
||||
cmd.extend([
|
||||
"-pix_fmt", "yuv420p", # Required for browser compatibility
|
||||
"-movflags", "+faststart+frag_keyframe+empty_moov+dash", # Enhanced streaming with dash for faster start
|
||||
"-g", "48", # Keyframe every 48 frames (2 seconds at 24fps) - fewer keyframes for speed
|
||||
"-keyint_min", "48", # Minimum keyframe interval
|
||||
"-sc_threshold", "0", # Disable scene change detection for speed
|
||||
"-threads", "0", # Use all available CPU threads
|
||||
"-shortest", # End encoding after shortest stream
|
||||
"-y" # Overwrite output without asking
|
||||
])
|
||||
|
||||
# Add output path
|
||||
cmd.append(output_path)
|
||||
|
||||
return cmd
|
||||
|
||||
def _get_video_codec(self) -> str:
|
||||
"""Determine the best available video codec (legacy method - use video_encoder instead)."""
|
||||
# This method is kept for backwards compatibility but is deprecated
|
||||
|
||||
@@ -210,6 +210,7 @@ def test_build_ass_filter_no_font_path(video_generator):
|
||||
"""Test _build_ass_filter with no font path configured."""
|
||||
ass_path = "test.ass"
|
||||
result = video_generator._build_ass_filter(ass_path)
|
||||
# No special chars = no escaping needed
|
||||
assert result == "ass=test.ass"
|
||||
|
||||
|
||||
@@ -220,13 +221,16 @@ def test_build_ass_filter_with_valid_font_path(video_generator, tmp_path):
|
||||
font_dir.mkdir()
|
||||
font_file = font_dir / "test.ttf"
|
||||
font_file.touch()
|
||||
|
||||
|
||||
# Update styles to include font path
|
||||
video_generator.styles["karaoke"]["font_path"] = str(font_file)
|
||||
|
||||
|
||||
ass_path = "test.ass"
|
||||
result = video_generator._build_ass_filter(ass_path)
|
||||
expected = f"ass=test.ass:fontsdir={str(font_dir)}"
|
||||
# Paths with no special chars don't need escaping
|
||||
# But font_dir path from tmp_path may have spaces on some systems
|
||||
escaped_font_dir = str(font_dir).replace("\\", "\\\\").replace("'", "\\\\'").replace(" ", "\\ ")
|
||||
expected = f"ass=test.ass:fontsdir={escaped_font_dir}"
|
||||
assert result == expected
|
||||
|
||||
|
||||
@@ -234,9 +238,10 @@ def test_build_ass_filter_with_invalid_font_path(video_generator):
|
||||
"""Test _build_ass_filter with invalid font path (file doesn't exist)."""
|
||||
# Set non-existent font path
|
||||
video_generator.styles["karaoke"]["font_path"] = "/nonexistent/font.ttf"
|
||||
|
||||
|
||||
ass_path = "test.ass"
|
||||
result = video_generator._build_ass_filter(ass_path)
|
||||
# No special chars = no escaping
|
||||
assert result == "ass=test.ass"
|
||||
|
||||
|
||||
@@ -244,9 +249,10 @@ def test_build_ass_filter_with_empty_font_path(video_generator):
|
||||
"""Test _build_ass_filter with empty font path."""
|
||||
# Set empty font path
|
||||
video_generator.styles["karaoke"]["font_path"] = ""
|
||||
|
||||
|
||||
ass_path = "test.ass"
|
||||
result = video_generator._build_ass_filter(ass_path)
|
||||
# No special chars = no escaping
|
||||
assert result == "ass=test.ass"
|
||||
|
||||
|
||||
@@ -254,9 +260,10 @@ def test_build_ass_filter_with_none_font_path(video_generator):
|
||||
"""Test _build_ass_filter with None font path."""
|
||||
# Set None font path
|
||||
video_generator.styles["karaoke"]["font_path"] = None
|
||||
|
||||
|
||||
ass_path = "test.ass"
|
||||
result = video_generator._build_ass_filter(ass_path)
|
||||
# No special chars = no escaping
|
||||
assert result == "ass=test.ass"
|
||||
|
||||
|
||||
@@ -269,9 +276,10 @@ def test_build_ass_filter_no_karaoke_section(tmp_path):
|
||||
video_resolution=(1920, 1080),
|
||||
styles=styles,
|
||||
)
|
||||
|
||||
|
||||
ass_path = "test.ass"
|
||||
result = generator._build_ass_filter(ass_path)
|
||||
# No special chars = no escaping
|
||||
assert result == "ass=test.ass"
|
||||
|
||||
|
||||
@@ -279,19 +287,73 @@ def test_build_ass_filter_logging(video_generator, tmp_path, caplog):
|
||||
"""Test _build_ass_filter logging behavior."""
|
||||
# Set log level to INFO to capture the logs
|
||||
caplog.set_level(logging.INFO)
|
||||
|
||||
|
||||
# Create a test font file
|
||||
font_dir = tmp_path / "fonts"
|
||||
font_dir.mkdir()
|
||||
font_file = font_dir / "test.ttf"
|
||||
font_file.touch()
|
||||
|
||||
|
||||
# Update styles to include font path
|
||||
video_generator.styles["karaoke"]["font_path"] = str(font_file)
|
||||
|
||||
|
||||
ass_path = "test.ass"
|
||||
result = video_generator._build_ass_filter(ass_path)
|
||||
|
||||
|
||||
# Check that info log was generated
|
||||
assert "Returning ASS filter with fonts dir:" in caplog.text
|
||||
assert str(font_dir) in caplog.text
|
||||
|
||||
|
||||
def test_escape_ffmpeg_filter_path(video_generator):
|
||||
"""Test _escape_ffmpeg_filter_path uses FFmpeg filter escaping (not shell escaping).
|
||||
|
||||
FFmpeg filter escaping rules (for subprocess without shell):
|
||||
- Backslashes: double them (\ -> \\)
|
||||
- Single quotes/apostrophes: escape with three backslashes (' -> \\')
|
||||
- Spaces: escape with backslash ( -> \ )
|
||||
"""
|
||||
# Test simple path without special chars (no escaping needed)
|
||||
assert video_generator._escape_ffmpeg_filter_path("/path/to/file.ass") == "/path/to/file.ass"
|
||||
|
||||
# Test apostrophe escaping (fixes paths like "I'm With You")
|
||||
# Apostrophe becomes \\\' (three backslashes + quote) in the actual string
|
||||
result = video_generator._escape_ffmpeg_filter_path("./Avril Lavigne - I'm With You/file.ass")
|
||||
assert result == "./Avril\\ Lavigne\\ -\\ I\\\\\\'m\\ With\\ You/file.ass"
|
||||
|
||||
# Test path with spaces (escaped with backslash)
|
||||
assert video_generator._escape_ffmpeg_filter_path("/path/with spaces/file.ass") == "/path/with\\ spaces/file.ass"
|
||||
|
||||
# Test path with colons (no escaping needed - colon is only special in filter options)
|
||||
assert video_generator._escape_ffmpeg_filter_path("C:/path/file.ass") == "C:/path/file.ass"
|
||||
|
||||
# Test multiple apostrophes
|
||||
result = video_generator._escape_ffmpeg_filter_path("It's Bob's file.ass")
|
||||
assert result == "It\\\\\\'s\\ Bob\\\\\\'s\\ file.ass"
|
||||
|
||||
|
||||
def test_build_ass_filter_escapes_path_with_apostrophe(video_generator):
|
||||
"""Test _build_ass_filter properly escapes paths containing apostrophes."""
|
||||
ass_path = "./Avril Lavigne - I'm With You/cache/temp.ass"
|
||||
result = video_generator._build_ass_filter(ass_path)
|
||||
# Spaces escaped with backslash, apostrophe escaped with \\\'
|
||||
assert result == "ass=./Avril\\ Lavigne\\ -\\ I\\\\\\'m\\ With\\ You/cache/temp.ass"
|
||||
|
||||
|
||||
def test_build_ass_filter_escapes_font_dir_with_special_chars(video_generator, tmp_path):
|
||||
"""Test _build_ass_filter properly escapes font directory with special characters."""
|
||||
# Create a font directory with special characters
|
||||
font_dir = tmp_path / "It's a Font Dir"
|
||||
font_dir.mkdir()
|
||||
font_file = font_dir / "test.ttf"
|
||||
font_file.touch()
|
||||
|
||||
# Update styles to include font path
|
||||
video_generator.styles["karaoke"]["font_path"] = str(font_file)
|
||||
|
||||
ass_path = "test.ass"
|
||||
result = video_generator._build_ass_filter(ass_path)
|
||||
# Should have fontsdir with escaped path
|
||||
assert "fontsdir=" in result
|
||||
# Apostrophe should be escaped using \\\' pattern (3 backslashes + quote)
|
||||
assert "\\\\\\'s" in result
|
||||
Generated
+467
-454
File diff suppressed because it is too large.
Load diff
+3
-2
@@ -1,13 +1,14 @@
|
||||
[tool.poetry]
|
||||
name = "karaoke-gen"
|
||||
version = "0.90.0"
|
||||
version = "0.95.2"
|
||||
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
|
||||
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
|
||||
license = "MIT"
|
||||
readme = "README.md"
|
||||
packages = [
|
||||
{ include = "karaoke_gen" },
|
||||
{ include = "lyrics_transcriber", from = "lyrics_transcriber_temp" }
|
||||
{ include = "lyrics_transcriber", from = "lyrics_transcriber_temp" },
|
||||
{ include = "backend" }
|
||||
]
|
||||
include = [
|
||||
"karaoke_gen/instrumental_review/static/*",
|
||||
|
||||
Executable
+41
@@ -0,0 +1,41 @@
|
||||
#!/bin/bash
|
||||
# Admin API helper script
|
||||
# Usage: ./scripts/admin_api.sh <endpoint> [method] [data]
|
||||
#
|
||||
# Examples:
|
||||
# ./scripts/admin_api.sh /api/jobs # List all jobs (admin sees all)
|
||||
# ./scripts/admin_api.sh /api/jobs?limit=10 # List 10 jobs
|
||||
# ./scripts/admin_api.sh /api/jobs/abc123 # Get specific job
|
||||
# ./scripts/admin_api.sh /api/jobs/abc123/retry POST # Retry a job
|
||||
# ./scripts/admin_api.sh /api/jobs/abc123/select-instrumental POST '{"selection":"clean"}'
|
||||
|
||||
set -e
|
||||
|
||||
API_URL="${KARAOKE_API_URL:-https://api.nomadkaraoke.com}"
|
||||
|
||||
# Get admin token from Secret Manager if not set
|
||||
if [ -z "$KARAOKE_ADMIN_TOKEN" ]; then
|
||||
KARAOKE_ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke 2>/dev/null)
|
||||
fi
|
||||
|
||||
if [ -z "$KARAOKE_ADMIN_TOKEN" ]; then
|
||||
echo "Error: Could not get admin token. Run 'gcloud auth login' first." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ENDPOINT="${1:-/api/admin/jobs}"
|
||||
METHOD="${2:-GET}"
|
||||
DATA="$3"
|
||||
|
||||
if [ "$METHOD" = "GET" ]; then
|
||||
curl -s "${API_URL}${ENDPOINT}" \
|
||||
-H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" | jq .
|
||||
elif [ -n "$DATA" ]; then
|
||||
curl -s -X "$METHOD" "${API_URL}${ENDPOINT}" \
|
||||
-H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "$DATA" | jq .
|
||||
else
|
||||
curl -s -X "$METHOD" "${API_URL}${ENDPOINT}" \
|
||||
-H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" | jq .
|
||||
fi
|
||||
@@ -121,16 +121,25 @@ class TestVideoBackgroundProcessor:
|
||||
processor.get_audio_duration("/path/to/audio.wav")
|
||||
|
||||
def test_escape_filter_path(self, processor):
|
||||
"""Test path escaping for ffmpeg filters."""
|
||||
# Test backslash escaping
|
||||
assert processor.escape_filter_path("C:\\Users\\test\\file.ass") == "C\\:\\\\Users\\\\test\\\\file.ass"
|
||||
|
||||
# Test colon escaping
|
||||
assert processor.escape_filter_path("/path/to/file.ass") == "/path/to/file.ass"
|
||||
assert processor.escape_filter_path("C:/path/file.ass") == "C\\:/path/file.ass"
|
||||
|
||||
# Test combined escaping
|
||||
assert processor.escape_filter_path("C:\\path:with:colons\\file.ass") == "C\\:\\\\path\\:with\\:colons\\\\file.ass"
|
||||
"""Test path escaping for ffmpeg filters using single-quote wrapping."""
|
||||
# Test simple path (wrapped in single quotes)
|
||||
assert processor.escape_filter_path("/path/to/file.ass") == "'/path/to/file.ass'"
|
||||
|
||||
# Test path with spaces (protected by single quotes)
|
||||
assert processor.escape_filter_path("/path/with spaces/file.ass") == "'/path/with spaces/file.ass'"
|
||||
|
||||
# Test apostrophe escaping (fixes paths like "I'm With You")
|
||||
# Single quote becomes '\'' (end quote, escaped quote, start quote)
|
||||
assert processor.escape_filter_path("./Avril Lavigne - I'm With You/file.ass") == "'./Avril Lavigne - I'\\''m With You/file.ass'"
|
||||
|
||||
# Test path with colons (protected by single quotes)
|
||||
assert processor.escape_filter_path("C:/path/file.ass") == "'C:/path/file.ass'"
|
||||
|
||||
# Test Windows-style path (backslashes preserved, wrapped in quotes)
|
||||
assert processor.escape_filter_path("C:\\Users\\test\\file.ass") == "'C:\\Users\\test\\file.ass'"
|
||||
|
||||
# Test multiple apostrophes
|
||||
assert processor.escape_filter_path("It's Bob's file.ass") == "'It'\\''s Bob'\\''s file.ass'"
|
||||
|
||||
def test_build_video_filter_no_darkness(self, processor):
|
||||
"""Test video filter building without darkness overlay."""
|
||||
@@ -139,13 +148,13 @@ class TestVideoBackgroundProcessor:
|
||||
darkness_percent=0,
|
||||
fonts_dir=None
|
||||
)
|
||||
|
||||
|
||||
# Should contain scale and crop
|
||||
assert "scale=w=3840:h=2160:force_original_aspect_ratio=increase,crop=3840:2160" in vf_filter
|
||||
|
||||
# Should contain ASS filter
|
||||
assert "ass=/path/to/subs.ass" in vf_filter
|
||||
|
||||
|
||||
# Should contain ASS filter with single-quoted path
|
||||
assert "ass='/path/to/subs.ass'" in vf_filter
|
||||
|
||||
# Should NOT contain darkening
|
||||
assert "drawbox" not in vf_filter
|
||||
|
||||
@@ -156,10 +165,10 @@ class TestVideoBackgroundProcessor:
|
||||
darkness_percent=50,
|
||||
fonts_dir=None
|
||||
)
|
||||
|
||||
|
||||
# Should contain darkening with correct alpha
|
||||
assert "drawbox=x=0:y=0:w=iw:h=ih:color=black@0.50:t=fill" in vf_filter
|
||||
|
||||
|
||||
# Should contain ASS filter after darkening
|
||||
assert vf_filter.index("drawbox") < vf_filter.index("ass=")
|
||||
|
||||
@@ -171,20 +180,21 @@ class TestVideoBackgroundProcessor:
|
||||
darkness_percent=0,
|
||||
fonts_dir="/path/to/fonts"
|
||||
)
|
||||
|
||||
# Should contain fonts directory in ASS filter
|
||||
assert "fontsdir=/path/to/fonts" in vf_filter
|
||||
|
||||
# Should contain fonts directory in ASS filter (single-quoted)
|
||||
assert "fontsdir='/path/to/fonts'" in vf_filter
|
||||
|
||||
def test_build_video_filter_path_escaping(self, processor):
|
||||
"""Test video filter building with special characters in paths."""
|
||||
# Test with apostrophe in path
|
||||
vf_filter = processor.build_video_filter(
|
||||
ass_subtitles_path="C:\\Users\\test\\subs.ass",
|
||||
ass_subtitles_path="./Avril Lavigne - I'm With You/subs.ass",
|
||||
darkness_percent=0,
|
||||
fonts_dir=None
|
||||
)
|
||||
|
||||
# Paths should be escaped
|
||||
assert "C\\:\\\\Users\\\\test\\\\subs.ass" in vf_filter
|
||||
|
||||
# Paths should be single-quoted with apostrophes escaped as '\''
|
||||
assert "'./Avril Lavigne - I'\\''m With You/subs.ass'" in vf_filter
|
||||
|
||||
@patch('subprocess.run')
|
||||
def test_execute_command_with_fallback_gpu_success(self, mock_run, processor_with_nvenc, mock_logger):
|
||||
|
||||
Reference in new issue
Block a user