mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-21 05:18:08 +09:00
Compare commits
27
Commits
No files matched your search
@@ -0,0 +1,222 @@
|
||||
# Plan: Worker Logs Rearchitecture
|
||||
|
||||
**Created:** 2026-01-04
|
||||
**Branch:** `feat/sess-20260104-1931-logs-research`
|
||||
**Status:** Implementation Complete - Pending PR Merge & Deployment
|
||||
|
||||
## Overview
|
||||
|
||||
Move `worker_logs` from embedded array in job documents to a Firestore subcollection to avoid the 1MB document size limit. The current approach caused job 501258e1 to fail when logs reached 1.26 MB (98.6% of document size).
|
||||
|
||||
## Problem Statement
|
||||
|
||||
- Firestore documents have a 1 MB size limit
|
||||
- `worker_logs` is stored as an array field within job documents
|
||||
- Long-running jobs (especially video encoding) can generate thousands of log entries
|
||||
- When the limit is exceeded, job updates fail with: `Document cannot be written because its size exceeds the maximum allowed size`
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Logs must remain associated with their job
|
||||
- [x] Logs should be queryable by job_id, worker type, timestamp
|
||||
- [x] Old logs should auto-cleanup (TTL: 30 days)
|
||||
- [x] Existing API endpoint (`GET /api/jobs/{id}/logs`) must continue to work
|
||||
- [x] Frontend must continue to display logs without changes
|
||||
- [x] Migration path for existing jobs (optional - can just start fresh)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Option A: Subcollection (Recommended)
|
||||
Store logs in `jobs/{job_id}/logs/{log_id}` subcollection.
|
||||
|
||||
**Pros:**
|
||||
- Natural association with job (delete job = delete logs with cascading)
|
||||
- Each log entry is its own document (no size limit concerns)
|
||||
- Can add TTL via Firestore TTL policies
|
||||
- Efficient queries with composite indexes
|
||||
|
||||
**Cons:**
|
||||
- More Firestore read operations for paginated log fetching
|
||||
- Slightly more complex write logic
|
||||
|
||||
### Option B: Separate Collection
|
||||
Store logs in `job_logs` collection with `job_id` field.
|
||||
|
||||
**Pros:**
|
||||
- Can query across all jobs
|
||||
- Simpler TTL management
|
||||
|
||||
**Cons:**
|
||||
- Orphaned logs if job deleted (need cleanup logic)
|
||||
- Extra index needed for job_id queries
|
||||
|
||||
### Decision: Option A (Subcollection)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Backend Changes
|
||||
|
||||
1. [x] **Create log entry model** - `backend/models/worker_log.py`
|
||||
- `WorkerLogEntry` dataclass with: id, job_id, timestamp, level, worker, message, metadata, ttl_expiry
|
||||
- Factory method `create()` for easy instantiation
|
||||
- `to_dict()` for Firestore storage
|
||||
- `to_legacy_dict()` for API compatibility
|
||||
- `from_dict()` for Firestore retrieval
|
||||
|
||||
2. [x] **Update FirestoreService** - `backend/services/firestore_service.py`
|
||||
- Added `append_log_to_subcollection(job_id, log_entry)` method
|
||||
- Added `get_logs_from_subcollection(job_id, limit, since_timestamp, worker, offset)` method
|
||||
- Added `get_logs_count_from_subcollection(job_id)` method
|
||||
- Added `delete_logs_subcollection(job_id, batch_size)` method
|
||||
- Old `append_worker_log` marked as deprecated
|
||||
|
||||
3. [x] **Update JobManager** - `backend/services/job_manager.py`
|
||||
- Modified `append_worker_log()` to use subcollection when `USE_LOG_SUBCOLLECTION=true`
|
||||
- Modified `get_worker_logs()` to read from subcollection with fallback to embedded array
|
||||
- Added `get_worker_logs_count()` for total count
|
||||
- Updated `delete_job()` to clean up logs subcollection
|
||||
|
||||
4. [x] **Update jobs API** - `backend/api/routes/jobs.py`
|
||||
- `GET /api/jobs/{id}/logs` - Updated to use new methods
|
||||
- Response format stays the same (backward compatible)
|
||||
- Fixed `next_index` calculation
|
||||
|
||||
5. [x] **Add feature flag** - `backend/config.py`
|
||||
- Added `USE_LOG_SUBCOLLECTION` environment variable (default: true)
|
||||
|
||||
6. [x] **Add TTL configuration** - `infrastructure/__main__.py`
|
||||
- Added Firestore TTL policy for logs subcollection (30 days)
|
||||
- Added composite index for worker + timestamp queries
|
||||
|
||||
### Phase 2: Testing
|
||||
|
||||
7. [x] **Unit tests** - `backend/tests/test_worker_log_subcollection.py`
|
||||
- 24 tests covering:
|
||||
- WorkerLogEntry model creation and conversion
|
||||
- FirestoreService subcollection methods
|
||||
- JobManager logging with feature flag
|
||||
- Edge cases (unicode, long messages, empty messages)
|
||||
|
||||
8. [x] **Integration tests** - `backend/tests/emulator/test_worker_logs_subcollection.py`
|
||||
- 11 tests using real Firestore emulator:
|
||||
- Single and sequential writes
|
||||
- TTL field verification
|
||||
- Concurrent writes (no race conditions)
|
||||
- Large volume (500 logs)
|
||||
- Worker filtering
|
||||
- Timestamp ordering
|
||||
- Subcollection deletion
|
||||
- Concurrent workers interleaved
|
||||
- Job deletion cleanup
|
||||
- Comparison with embedded array
|
||||
|
||||
### Phase 3: Deployment & Migration
|
||||
|
||||
9. [ ] **Deploy with feature flag**
|
||||
- Add `USE_LOG_SUBCOLLECTION=true` env var (default is true)
|
||||
- Deploy to production
|
||||
|
||||
10. [ ] **Monitor**
|
||||
- Watch for errors in Cloud Logging
|
||||
- Verify logs appear in frontend
|
||||
|
||||
11. [x] **Backward compatibility**
|
||||
- Old jobs with embedded arrays still work (fallback in get_worker_logs)
|
||||
- No migration needed - new jobs use subcollection, old jobs use array
|
||||
|
||||
### Phase 4: Cleanup (Future)
|
||||
|
||||
12. [ ] **Remove old code path** (after confidence period)
|
||||
- Remove `worker_logs` field from job model
|
||||
- Remove old `append_worker_log` method
|
||||
- Remove feature flag
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/worker_log.py` | Created | WorkerLogEntry dataclass |
|
||||
| `backend/services/firestore_service.py` | Modified | Added subcollection methods |
|
||||
| `backend/services/job_manager.py` | Modified | Switch to subcollection with fallback |
|
||||
| `backend/api/routes/jobs.py` | Modified | Updated logs endpoint |
|
||||
| `backend/config.py` | Modified | Added USE_LOG_SUBCOLLECTION setting |
|
||||
| `backend/tests/test_worker_log_subcollection.py` | Created | Unit tests (24 tests) |
|
||||
| `backend/tests/emulator/test_worker_logs_subcollection.py` | Created | Integration tests (11 tests) |
|
||||
| `infrastructure/__main__.py` | Modified | Added TTL policy and index |
|
||||
|
||||
## Log Entry Schema
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class WorkerLogEntry:
|
||||
timestamp: datetime
|
||||
level: str # "DEBUG", "INFO", "WARNING", "ERROR"
|
||||
worker: str # "audio", "lyrics", "video", "render", "screens", "distribution"
|
||||
message: str
|
||||
id: str # Auto-generated UUID
|
||||
job_id: str
|
||||
ttl_expiry: datetime # timestamp + 30 days
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
```
|
||||
|
||||
## Firestore Structure
|
||||
|
||||
```
|
||||
jobs/
|
||||
{job_id}/
|
||||
... (job fields, worker_logs still present for old jobs)
|
||||
logs/ # subcollection for new jobs
|
||||
{log_id}/
|
||||
id: string
|
||||
job_id: string
|
||||
timestamp: datetime
|
||||
level: string
|
||||
worker: string
|
||||
message: string
|
||||
metadata: map (optional)
|
||||
ttl_expiry: datetime
|
||||
```
|
||||
|
||||
## API Response (Unchanged)
|
||||
|
||||
```json
|
||||
{
|
||||
"logs": [
|
||||
{
|
||||
"timestamp": "2026-01-04T12:00:00Z",
|
||||
"level": "INFO",
|
||||
"worker": "video",
|
||||
"message": "Starting video generation"
|
||||
}
|
||||
],
|
||||
"next_index": 42,
|
||||
"total_logs": 150
|
||||
}
|
||||
```
|
||||
|
||||
## Testing Results
|
||||
|
||||
All tests pass:
|
||||
- 24 unit tests in `test_worker_log_subcollection.py`
|
||||
- 11 emulator integration tests in `test_worker_logs_subcollection.py`
|
||||
- 68 total tests in emulator suite pass
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. Set `USE_LOG_SUBCOLLECTION=false` env var
|
||||
2. Redeploy backend
|
||||
3. New logs will go back to embedded array
|
||||
4. Subcollection logs will remain but won't be read
|
||||
|
||||
## Answers to Open Questions
|
||||
|
||||
- [x] **Should we paginate logs in the API?** - Yes, added `offset` parameter support
|
||||
- [x] **What's the optimal TTL period?** - 30 days (configurable via DEFAULT_LOG_TTL_DAYS)
|
||||
- [x] **Should we index by timestamp for efficient range queries?** - Yes, added composite index (worker + timestamp)
|
||||
|
||||
## References
|
||||
|
||||
- [Firestore TTL](https://cloud.google.com/firestore/docs/ttl)
|
||||
- [Firestore Subcollections](https://firebase.google.com/docs/firestore/data-model#subcollections)
|
||||
- [Firestore Limits](https://firebase.google.com/docs/firestore/quotas#limits)
|
||||
- [Pulumi Firestore Field](https://www.pulumi.com/registry/packages/gcp/api-docs/firestore/field/)
|
||||
+197
-21
@@ -903,11 +903,14 @@ jobs:
|
||||
|
||||
deploy-backend:
|
||||
name: "Deploy - Backend (Cloud Run)"
|
||||
needs: [package-build]
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
outputs:
|
||||
base_rebuilt: ${{ steps.check-base.outputs.needs_rebuild }}
|
||||
env:
|
||||
REGISTRY: us-central1-docker.pkg.dev
|
||||
PROJECT_ID: nomadkaraoke
|
||||
@@ -965,6 +968,32 @@ jobs:
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
# ============================================
|
||||
# Upload wheel to GCS for GCE encoding worker
|
||||
# The GCE worker installs this wheel at job start for hot code updates
|
||||
# ============================================
|
||||
|
||||
- name: Download wheel artifact
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
name: package-dist
|
||||
path: dist/
|
||||
|
||||
- name: Upload wheel to GCS for GCE worker
|
||||
run: |
|
||||
echo "📦 Uploading wheel to GCS for GCE encoding worker..."
|
||||
WHEEL_FILE=$(ls dist/*.whl | head -1)
|
||||
if [ -z "$WHEEL_FILE" ]; then
|
||||
echo "::error::No wheel file found in dist/"
|
||||
exit 1
|
||||
fi
|
||||
echo "Found wheel: $WHEEL_FILE"
|
||||
|
||||
# Upload to GCS (GCE worker downloads latest at job start)
|
||||
gsutil cp "$WHEEL_FILE" gs://karaoke-gen-storage-${{ env.PROJECT_ID }}/wheels/
|
||||
|
||||
echo "✅ Wheel uploaded - GCE worker will pick up changes on next job"
|
||||
|
||||
- name: Configure Docker for Artifact Registry
|
||||
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
|
||||
|
||||
@@ -990,32 +1019,32 @@ jobs:
|
||||
run: |
|
||||
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
|
||||
|
||||
# Try to get the existing base image's hash label
|
||||
# First check if image exists locally (fast, no network call)
|
||||
# Only pull from registry if not cached locally
|
||||
# Fetch hash label directly from registry (10KB config blob, not full image)
|
||||
# This avoids issues with stale local Docker cache on GitHub runners
|
||||
EXISTING_HASH=""
|
||||
if docker image inspect $BASE_IMAGE >/dev/null 2>&1; then
|
||||
echo "✅ Base image found in local cache (skipping slow registry pull)"
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
|
||||
# Fallback to old label name for backwards compatibility
|
||||
if [ -z "$EXISTING_HASH" ]; then
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "poetry.lock.hash"}}' 2>/dev/null || echo "")
|
||||
fi
|
||||
echo "Existing base image hash: $EXISTING_HASH"
|
||||
elif docker pull $BASE_IMAGE 2>/dev/null; then
|
||||
echo "📥 Base image pulled from registry"
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
|
||||
# Fallback to old label name for backwards compatibility
|
||||
if [ -z "$EXISTING_HASH" ]; then
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "poetry.lock.hash"}}' 2>/dev/null || echo "")
|
||||
fi
|
||||
echo "Existing base image hash: $EXISTING_HASH"
|
||||
|
||||
# Get the manifest (small JSON, ~1KB) - includes config digest
|
||||
MANIFEST=$(docker manifest inspect $BASE_IMAGE 2>/dev/null || echo "")
|
||||
CONFIG_DIGEST=$(echo "$MANIFEST" | jq -r '.config.digest // ""' 2>/dev/null || echo "")
|
||||
|
||||
if [ -n "$CONFIG_DIGEST" ] && [ "$CONFIG_DIGEST" != "null" ] && [ "$CONFIG_DIGEST" != "" ]; then
|
||||
echo "📋 Found manifest, config digest: $CONFIG_DIGEST"
|
||||
|
||||
# Fetch just the config blob (~10KB) to get labels - much faster than pulling full image
|
||||
TOKEN=$(gcloud auth print-access-token)
|
||||
CONFIG_JSON=$(curl -sL -H "Authorization: Bearer $TOKEN" \
|
||||
"https://${{ env.REGISTRY }}/v2/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}/blobs/$CONFIG_DIGEST" 2>/dev/null)
|
||||
|
||||
EXISTING_HASH=$(echo "$CONFIG_JSON" | jq -r '.config.Labels["base.content.hash"] // .config.Labels["poetry.lock.hash"] // ""' 2>/dev/null || echo "")
|
||||
echo "📋 Registry base image hash: $EXISTING_HASH"
|
||||
else
|
||||
echo "⚠️ No existing base image found - will build"
|
||||
echo "⚠️ No existing base image found in registry - will build"
|
||||
fi
|
||||
|
||||
if [ "${{ steps.base-hash.outputs.hash }}" != "$EXISTING_HASH" ]; then
|
||||
echo "🔄 Base image needs rebuild (poetry.lock or Dockerfile.base changed)"
|
||||
echo " Current: ${{ steps.base-hash.outputs.hash }}"
|
||||
echo " Registry: $EXISTING_HASH"
|
||||
echo "needs_rebuild=true" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "✅ Base image is up to date"
|
||||
@@ -1043,6 +1072,31 @@ jobs:
|
||||
# Provenance disabled for smaller image and faster push
|
||||
provenance: false
|
||||
|
||||
- name: Ensure local base image is current (if not rebuilt)
|
||||
if: steps.check-base.outputs.needs_rebuild == 'false'
|
||||
run: |
|
||||
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
|
||||
|
||||
# Get the config digest from registry manifest (identifies the image content)
|
||||
# This is the same value as Docker's Image ID for the same image
|
||||
REGISTRY_CONFIG_DIGEST=$(docker manifest inspect $BASE_IMAGE 2>/dev/null | jq -r '.config.digest' || echo "")
|
||||
|
||||
# Get local image's ID (which is the config digest)
|
||||
LOCAL_IMAGE_ID=$(docker image inspect $BASE_IMAGE --format='{{.Id}}' 2>/dev/null || echo "")
|
||||
|
||||
echo "Registry config digest: $REGISTRY_CONFIG_DIGEST"
|
||||
echo "Local image ID: $LOCAL_IMAGE_ID"
|
||||
|
||||
if [ -z "$LOCAL_IMAGE_ID" ]; then
|
||||
echo "📥 Base image not in local cache - pulling from registry"
|
||||
docker pull $BASE_IMAGE
|
||||
elif [ "$LOCAL_IMAGE_ID" != "$REGISTRY_CONFIG_DIGEST" ]; then
|
||||
echo "🔄 Local cache has different image - pulling latest"
|
||||
docker pull $BASE_IMAGE
|
||||
else
|
||||
echo "✅ Local cache matches registry - skipping pull"
|
||||
fi
|
||||
|
||||
- name: Build and push app image
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
@@ -1085,7 +1139,7 @@ jobs:
|
||||
--concurrency 1 \
|
||||
--max-instances 50 \
|
||||
--min-instances 4 \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_GCE_ENCODING=true" \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true" \
|
||||
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,SENDGRID_API_KEY=sendgrid-api-key:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest"
|
||||
|
||||
- name: Verify deployment and version
|
||||
@@ -1139,3 +1193,125 @@ jobs:
|
||||
echo ""
|
||||
echo "📊 Built on self-hosted GCP runner (200GB SSD, no disk space issues)"
|
||||
echo " Runner: gcp-runner with 4 vCPU, 16GB RAM"
|
||||
|
||||
# ============================================
|
||||
# PROPAGATE BASE IMAGE TO ALL RUNNERS
|
||||
# When base image is rebuilt, pull it to all 20 runners
|
||||
# so subsequent app builds don't wait 8 min for the pull
|
||||
#
|
||||
# NOTE: This is best-effort distribution. GitHub Actions doesn't
|
||||
# support targeting specific runners by name, so we spawn 20 jobs
|
||||
# and rely on the scheduler to distribute them across idle runners.
|
||||
# In practice, with all runners idle after deploy-backend, distribution
|
||||
# is good. Runners that miss this pull will get the image on their
|
||||
# next app build (with the 8 min delay).
|
||||
# ============================================
|
||||
|
||||
propagate-base-image:
|
||||
name: "Propagate Base Image (${{ matrix.runner_index }})"
|
||||
needs: deploy-backend
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push' && needs.deploy-backend.outputs.base_rebuilt == 'true'
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
strategy:
|
||||
matrix:
|
||||
# 20 parallel jobs - GitHub distributes across available runners
|
||||
runner_index: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
|
||||
max-parallel: 20
|
||||
fail-fast: false # Don't cancel other runners if one fails
|
||||
env:
|
||||
REGISTRY: us-central1-docker.pkg.dev
|
||||
PROJECT_ID: nomadkaraoke
|
||||
REPOSITORY: karaoke-repo
|
||||
BASE_IMAGE_NAME: karaoke-backend-base
|
||||
steps:
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Configure Docker for Artifact Registry
|
||||
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
|
||||
|
||||
- name: Pull latest base image
|
||||
run: |
|
||||
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
|
||||
RUNNER_NAME=$(hostname)
|
||||
echo "📥 Pulling latest base image on runner $RUNNER_NAME..."
|
||||
docker pull $BASE_IMAGE
|
||||
echo "✅ Base image cached on $RUNNER_NAME"
|
||||
|
||||
deploy-infrastructure:
|
||||
name: "Deploy - Infrastructure (Pulumi)"
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.13'
|
||||
|
||||
- name: Install Pulumi
|
||||
run: |
|
||||
curl -fsSL https://get.pulumi.com | sh
|
||||
echo "$HOME/.pulumi/bin" >> $GITHUB_PATH
|
||||
|
||||
- name: Install Python dependencies
|
||||
working-directory: infrastructure
|
||||
run: |
|
||||
pip install pulumi pulumi-gcp pulumi-docker
|
||||
|
||||
- name: Configure Pulumi
|
||||
env:
|
||||
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
|
||||
run: |
|
||||
pulumi login
|
||||
|
||||
- name: Deploy Infrastructure
|
||||
working-directory: infrastructure
|
||||
env:
|
||||
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
|
||||
run: |
|
||||
echo "🚀 Deploying infrastructure changes..."
|
||||
# Skip preview to avoid needing broad read permissions on all GCP resources
|
||||
# Pulumi state is authoritative since infrastructure is only managed via Pulumi
|
||||
pulumi up --yes --skip-preview --stack nomadkaraoke/karaoke-gen-infrastructure/prod
|
||||
|
||||
- name: Restart GCE Encoding Worker
|
||||
run: |
|
||||
echo "🔄 Restarting encoding worker to pick up new code..."
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
|
||||
--command="sudo systemctl restart encoding-worker" || true
|
||||
echo "✅ Encoding worker restart triggered"
|
||||
|
||||
- name: Summary
|
||||
if: success()
|
||||
run: |
|
||||
echo "🎉 Infrastructure deployment successful!"
|
||||
echo "Commit: ${{ github.sha }}"
|
||||
echo "Branch: ${{ github.ref_name }}"
|
||||
@@ -48,10 +48,58 @@ cd frontend && npm run test:all 2>&1 | tail -n 200
|
||||
- Skip for docs-only changes
|
||||
|
||||
### Infrastructure
|
||||
- All GCP changes via Pulumi in `infrastructure/`
|
||||
- `gcloud` CLI for reading/debugging only
|
||||
- **All GCP changes via Pulumi PRs** - changes in `infrastructure/` deploy automatically on merge to main
|
||||
- Never modify GCP resources directly via console or `gcloud` CLI
|
||||
- `gcloud` CLI for reading/debugging only (e.g., checking logs, SSH to VMs)
|
||||
- Local `pulumi preview` works but `pulumi up` requires broader permissions than CI
|
||||
- Stop and notify user on auth issues
|
||||
|
||||
## API Authentication (for Agents)
|
||||
|
||||
When you need to call backend APIs programmatically:
|
||||
|
||||
### Internal Worker Endpoints
|
||||
Use the `X-Admin-Token` header with the token from Secret Manager:
|
||||
```bash
|
||||
# Trigger video worker
|
||||
curl -X POST "https://api.nomadkaraoke.com/api/internal/workers/video" \
|
||||
-H "X-Admin-Token: $(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke)" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"job_id": "YOUR_JOB_ID"}'
|
||||
|
||||
# Other internal endpoints follow the same pattern
|
||||
```
|
||||
|
||||
### Firestore Direct Access
|
||||
```bash
|
||||
# Set the correct project
|
||||
export GOOGLE_CLOUD_PROJECT=nomadkaraoke
|
||||
|
||||
# Use Python with google-cloud-firestore
|
||||
python3 << 'EOF'
|
||||
import os
|
||||
os.environ['GOOGLE_CLOUD_PROJECT'] = 'nomadkaraoke'
|
||||
from google.cloud import firestore
|
||||
db = firestore.Client(project='nomadkaraoke')
|
||||
# Query/update documents...
|
||||
EOF
|
||||
```
|
||||
|
||||
### GCE Encoding Worker
|
||||
```bash
|
||||
# SSH to restart service (clears in-memory job queue)
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
|
||||
--command="sudo systemctl restart encoding-worker"
|
||||
|
||||
# Check health
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
|
||||
--command="curl -s http://localhost:8080/health"
|
||||
```
|
||||
|
||||
### What Doesn't Work
|
||||
- `gcloud auth print-identity-token` - Wrong token type for internal endpoints
|
||||
- Cloud Tasks without proper service account - Gets 401 from Cloud Run
|
||||
|
||||
## Documentation Maintenance
|
||||
|
||||
### Structure
|
||||
|
||||
@@ -95,7 +95,7 @@ class AudioSearchRequest(BaseModel):
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
|
||||
@@ -485,7 +485,8 @@ async def search_audio(
|
||||
effective_discord_webhook_url = body.discord_webhook_url or settings.default_discord_webhook_url
|
||||
|
||||
# Apply defaults for YouTube/Dropbox distribution (for web service)
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload or settings.default_enable_youtube_upload
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
effective_brand_prefix = body.brand_prefix or settings.default_brand_prefix
|
||||
effective_youtube_description = body.youtube_description or settings.default_youtube_description
|
||||
|
||||
|
||||
@@ -69,7 +69,7 @@ class CreateJobFromUrlRequest(BaseModel):
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
webhook_url: Optional[str] = Field(None, description="Generic webhook URL")
|
||||
@@ -123,7 +123,7 @@ class CreateJobWithUploadUrlsRequest(BaseModel):
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
webhook_url: Optional[str] = Field(None, description="Generic webhook URL")
|
||||
@@ -406,7 +406,7 @@ async def upload_and_create_job(
|
||||
enable_txt: Optional[bool] = Form(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise"),
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Form(None, description="Brand code prefix (e.g., NOMAD)"),
|
||||
enable_youtube_upload: bool = Form(False, description="Upload to YouTube"),
|
||||
enable_youtube_upload: Optional[bool] = Form(None, description="Upload to YouTube. None = use server default"),
|
||||
youtube_description: Optional[str] = Form(None, description="YouTube video description text"),
|
||||
discord_webhook_url: Optional[str] = Form(None, description="Discord webhook URL for notifications"),
|
||||
webhook_url: Optional[str] = Form(None, description="Generic webhook URL"),
|
||||
@@ -510,12 +510,19 @@ async def upload_and_create_job(
|
||||
if dist.brand_prefix and not brand_prefix:
|
||||
logger.info(f"Using default brand_prefix: {dist.brand_prefix}")
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_enable_youtube_upload = enable_youtube_upload if enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
if effective_enable_youtube_upload and enable_youtube_upload is None:
|
||||
logger.info("Using default enable_youtube_upload: True (from env)")
|
||||
|
||||
# Validate credentials for requested distribution services (including defaults)
|
||||
# This prevents accepting jobs that will fail later due to missing credentials
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if enable_youtube_upload:
|
||||
if effective_enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
@@ -594,7 +601,7 @@ async def upload_and_create_job(
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=dist.brand_prefix,
|
||||
enable_youtube_upload=enable_youtube_upload,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=youtube_description,
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
webhook_url=webhook_url,
|
||||
@@ -815,11 +822,12 @@ async def upload_and_create_job(
|
||||
"using_default": gdrive_folder_id is None,
|
||||
}
|
||||
|
||||
if enable_youtube_upload:
|
||||
if effective_enable_youtube_upload:
|
||||
youtube_result = credential_manager.check_youtube_credentials()
|
||||
distribution_services["youtube"] = {
|
||||
"enabled": True,
|
||||
"credentials_valid": youtube_result.status == CredentialStatus.VALID,
|
||||
"using_default": enable_youtube_upload is None,
|
||||
}
|
||||
|
||||
if dist.discord_webhook_url:
|
||||
@@ -908,7 +916,7 @@ class CreateFinaliseOnlyJobRequest(BaseModel):
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
keep_brand_code: Optional[str] = Field(None, description="Preserve existing brand code from folder name")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
|
||||
@@ -1048,11 +1056,16 @@ async def create_job_with_upload_urls(
|
||||
brand_prefix=body.brand_prefix,
|
||||
)
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
|
||||
# Validate credentials for requested distribution services
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if body.enable_youtube_upload:
|
||||
if effective_enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
@@ -1105,7 +1118,7 @@ async def create_job_with_upload_urls(
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=dist.brand_prefix,
|
||||
enable_youtube_upload=body.enable_youtube_upload,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=body.youtube_description,
|
||||
youtube_description_template=body.youtube_description, # video_worker reads this field
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
@@ -1466,11 +1479,16 @@ async def create_job_from_url(
|
||||
brand_prefix=body.brand_prefix,
|
||||
)
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
|
||||
# Validate credentials for requested distribution services
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if body.enable_youtube_upload:
|
||||
if effective_enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
@@ -1522,7 +1540,7 @@ async def create_job_from_url(
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=dist.brand_prefix,
|
||||
enable_youtube_upload=body.enable_youtube_upload,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=body.youtube_description,
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
webhook_url=body.webhook_url,
|
||||
@@ -1712,11 +1730,16 @@ async def create_finalise_only_job(
|
||||
brand_prefix=body.brand_prefix,
|
||||
)
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
|
||||
# Validate distribution credentials if services are requested
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if body.enable_youtube_upload:
|
||||
if effective_enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
@@ -1766,7 +1789,7 @@ async def create_finalise_only_job(
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=dist.brand_prefix,
|
||||
enable_youtube_upload=body.enable_youtube_upload,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=body.youtube_description,
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
dropbox_path=dist.dropbox_path,
|
||||
|
||||
@@ -87,6 +87,11 @@ async def create_job(
|
||||
user_email = request.user_email
|
||||
logger.info(f"Admin {auth_result.user_email} creating job on behalf of {user_email}")
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_enable_youtube_upload = request.enable_youtube_upload if request.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
|
||||
# Create job with all preferences
|
||||
job_create = JobCreate(
|
||||
url=str(request.url),
|
||||
@@ -94,7 +99,7 @@ async def create_job(
|
||||
title=request.title,
|
||||
enable_cdg=request.enable_cdg,
|
||||
enable_txt=request.enable_txt,
|
||||
enable_youtube_upload=request.enable_youtube_upload,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=request.youtube_description,
|
||||
webhook_url=request.webhook_url,
|
||||
user_email=user_email
|
||||
@@ -1452,10 +1457,14 @@ async def get_worker_logs(
|
||||
This endpoint returns worker logs stored in Firestore.
|
||||
Use `since_index` for efficient polling (returns only new logs).
|
||||
|
||||
Logs are stored in a subcollection (jobs/{job_id}/logs) to avoid
|
||||
the 1MB document size limit. Older jobs may have logs in an embedded
|
||||
array (worker_logs field) - this endpoint handles both transparently.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
since_index: Return only logs after this index (for pagination/polling)
|
||||
worker: Filter by worker name (audio, lyrics, screens, video, render)
|
||||
worker: Filter by worker name (audio, lyrics, screens, video, render, distribution)
|
||||
|
||||
Returns:
|
||||
{
|
||||
@@ -1473,11 +1482,11 @@ async def get_worker_logs(
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to access logs for this job")
|
||||
|
||||
logs = job_manager.get_worker_logs(job_id, since_index=since_index, worker=worker)
|
||||
total = len(job.worker_logs) if job.worker_logs else 0
|
||||
total = job_manager.get_worker_logs_count(job_id)
|
||||
|
||||
return {
|
||||
"logs": logs,
|
||||
"next_index": total,
|
||||
"next_index": since_index + len(logs),
|
||||
"total_logs": total
|
||||
}
|
||||
|
||||
|
||||
+122
-40
@@ -27,8 +27,10 @@ from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.job_logging import job_log_context, JobLogger
|
||||
from backend.services.tracing import create_span, add_span_attribute, add_span_event
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
from backend.api.dependencies import require_auth, require_review_auth
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.config import get_settings
|
||||
|
||||
# LyricsTranscriber imports for preview generation
|
||||
from lyrics_transcriber.types import CorrectionResult
|
||||
@@ -385,28 +387,35 @@ async def generate_preview_video(
|
||||
):
|
||||
"""
|
||||
Generate a preview video with the current corrections.
|
||||
|
||||
|
||||
Uses the LyricsTranscriber's CorrectionOperations to generate a 360p preview
|
||||
video with the user's current corrections applied.
|
||||
|
||||
When USE_GCE_PREVIEW_ENCODING is enabled, video encoding is offloaded to
|
||||
the high-performance GCE worker for faster generation (15-20s vs 60+s).
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
settings = get_settings()
|
||||
encoding_service = get_encoding_service()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Job must be in review state to generate preview
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job not in review state (current status: {job.status})"
|
||||
)
|
||||
|
||||
logger.info(f"Job {job_id}: Generating preview video")
|
||||
|
||||
|
||||
# Check if GCE preview encoding is enabled
|
||||
use_gce_preview = encoding_service.is_preview_enabled
|
||||
logger.info(f"Job {job_id}: Generating preview video (GCE preview: {use_gce_preview})")
|
||||
|
||||
# Use tracing and job_log_context for full observability
|
||||
with create_span("generate-preview-video", {"job_id": job_id}) as span:
|
||||
with create_span("generate-preview-video", {"job_id": job_id, "use_gce": use_gce_preview}) as span:
|
||||
with job_log_context(job_id, worker="preview"):
|
||||
try:
|
||||
# Create temp directory for this preview operation
|
||||
@@ -416,19 +425,19 @@ async def generate_preview_video(
|
||||
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
corrections_path = os.path.join(temp_dir, "corrections.json")
|
||||
storage.download_file(corrections_gcs, corrections_path)
|
||||
|
||||
|
||||
with open(corrections_path, 'r', encoding='utf-8') as f:
|
||||
original_data = json.load(f)
|
||||
|
||||
|
||||
# 2. Download input audio
|
||||
audio_path = os.path.join(temp_dir, "audio.flac")
|
||||
storage.download_file(job.input_media_gcs_path, audio_path)
|
||||
download_span.set_attribute("audio_gcs_path", job.input_media_gcs_path)
|
||||
|
||||
|
||||
# 3. Load original as CorrectionResult
|
||||
correction_result = CorrectionResult.from_dict(original_data)
|
||||
add_span_event("corrections_loaded")
|
||||
|
||||
|
||||
# 4. Get or create styles file for preview using unified style loader
|
||||
with create_span("load-styles") as styles_span:
|
||||
styles_path, _ = load_styles_from_gcs(
|
||||
@@ -439,54 +448,127 @@ async def generate_preview_video(
|
||||
logger=logger,
|
||||
)
|
||||
styles_span.set_attribute("styles_path", styles_path)
|
||||
|
||||
|
||||
# 5. Set up output config for preview
|
||||
output_dir = os.path.join(temp_dir, "output")
|
||||
cache_dir = os.path.join(temp_dir, "cache")
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
|
||||
|
||||
output_config = OutputConfig(
|
||||
output_styles_json=styles_path,
|
||||
output_dir=output_dir,
|
||||
cache_dir=cache_dir,
|
||||
video_resolution="360p",
|
||||
)
|
||||
|
||||
# 6. Generate preview video using CorrectionOperations (heavy operation)
|
||||
with create_span("render-preview-video") as render_span:
|
||||
render_span.set_attribute("resolution", "360p")
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
correction_result=correction_result,
|
||||
updated_data=updated_data,
|
||||
output_config=output_config,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
logger=logger
|
||||
)
|
||||
add_span_event("render_complete")
|
||||
|
||||
# 7. Upload preview video to GCS for serving
|
||||
preview_hash = result["preview_hash"]
|
||||
video_path = result["video_path"]
|
||||
|
||||
with create_span("upload-preview-video") as upload_span:
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
storage.upload_file(video_path, preview_gcs_path)
|
||||
upload_span.set_attribute("gcs_path", preview_gcs_path)
|
||||
|
||||
|
||||
# 6. Generate preview (ASS-only if using GCE, or full video if local)
|
||||
preview_gcs_path = None
|
||||
|
||||
if use_gce_preview:
|
||||
# GCE path: Generate ASS only, then offload encoding to GCE
|
||||
try:
|
||||
with create_span("generate-ass-subtitles") as ass_span:
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
correction_result=correction_result,
|
||||
updated_data=updated_data,
|
||||
output_config=output_config,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
logger=logger,
|
||||
ass_only=True, # Only generate ASS, skip video encoding
|
||||
)
|
||||
preview_hash = result["preview_hash"]
|
||||
ass_path = result["ass_path"]
|
||||
ass_span.set_attribute("ass_path", ass_path)
|
||||
add_span_event("ass_generated")
|
||||
|
||||
# Upload ASS to GCS
|
||||
with create_span("upload-ass-to-gcs") as upload_ass_span:
|
||||
ass_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.ass"
|
||||
storage.upload_file(ass_path, ass_gcs_path)
|
||||
upload_ass_span.set_attribute("ass_gcs_path", ass_gcs_path)
|
||||
|
||||
# Call GCE encoding service
|
||||
with create_span("gce-preview-encoding") as gce_span:
|
||||
bucket_name = settings.gcs_bucket_name
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
|
||||
# Get background image and font from style assets if available
|
||||
style_assets = job.style_assets or {}
|
||||
|
||||
background_image_gcs_path = None
|
||||
for key in ["karaoke_background", "style_karaoke_background"]:
|
||||
if key in style_assets:
|
||||
background_image_gcs_path = f"gs://{bucket_name}/{style_assets[key]}"
|
||||
gce_span.set_attribute("background_image", background_image_gcs_path)
|
||||
break
|
||||
|
||||
font_gcs_path = None
|
||||
for key in ["font", "style_font"]:
|
||||
if key in style_assets:
|
||||
font_gcs_path = f"gs://{bucket_name}/{style_assets[key]}"
|
||||
gce_span.set_attribute("font", font_gcs_path)
|
||||
break
|
||||
|
||||
gce_result = await encoding_service.encode_preview_video(
|
||||
job_id=f"preview_{job_id}_{preview_hash}",
|
||||
ass_gcs_path=f"gs://{bucket_name}/{ass_gcs_path}",
|
||||
audio_gcs_path=f"gs://{bucket_name}/{job.input_media_gcs_path}",
|
||||
output_gcs_path=f"gs://{bucket_name}/{preview_gcs_path}",
|
||||
background_color="black",
|
||||
background_image_gcs_path=background_image_gcs_path,
|
||||
font_gcs_path=font_gcs_path,
|
||||
)
|
||||
gce_span.set_attribute("gce_status", gce_result.get("status"))
|
||||
add_span_event("gce_encoding_complete")
|
||||
|
||||
logger.info(f"Job {job_id}: Preview generated via GCE: {preview_hash}")
|
||||
|
||||
except Exception as gce_error:
|
||||
# Fall back to local encoding if GCE fails
|
||||
logger.warning(
|
||||
f"Job {job_id}: GCE preview encoding failed, falling back to local: {gce_error}"
|
||||
)
|
||||
span.set_attribute("gce_fallback", True)
|
||||
use_gce_preview = False # Fall through to local encoding below
|
||||
|
||||
if not use_gce_preview:
|
||||
# Local path: Generate full preview video locally
|
||||
with create_span("render-preview-video-local") as render_span:
|
||||
render_span.set_attribute("resolution", "360p")
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
correction_result=correction_result,
|
||||
updated_data=updated_data,
|
||||
output_config=output_config,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
logger=logger,
|
||||
ass_only=False, # Generate full video locally
|
||||
)
|
||||
preview_hash = result["preview_hash"]
|
||||
video_path = result["video_path"]
|
||||
add_span_event("render_complete")
|
||||
|
||||
# Upload preview video to GCS
|
||||
with create_span("upload-preview-video") as upload_span:
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
storage.upload_file(video_path, preview_gcs_path)
|
||||
upload_span.set_attribute("gcs_path", preview_gcs_path)
|
||||
|
||||
# Store the GCS path for serving
|
||||
if job_id not in _preview_videos:
|
||||
_preview_videos[job_id] = {}
|
||||
_preview_videos[job_id][preview_hash] = preview_gcs_path
|
||||
|
||||
|
||||
logger.info(f"Job {job_id}: Preview video generated: {preview_hash}")
|
||||
span.set_attribute("preview_hash", preview_hash)
|
||||
span.set_attribute("success", True)
|
||||
|
||||
|
||||
return {"status": "success", "preview_hash": preview_hash}
|
||||
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Failed to generate preview video: {e}", exc_info=True)
|
||||
span.set_attribute("error", str(e))
|
||||
|
||||
@@ -68,9 +68,21 @@ class Settings(BaseSettings):
|
||||
use_gce_encoding: bool = os.getenv("USE_GCE_ENCODING", "false").lower() in ("true", "1", "yes")
|
||||
encoding_worker_url: Optional[str] = os.getenv("ENCODING_WORKER_URL") # e.g., http://136.119.50.148:8080
|
||||
encoding_worker_api_key: Optional[str] = os.getenv("ENCODING_WORKER_API_KEY")
|
||||
|
||||
# GCE Preview Encoding (for faster preview video generation)
|
||||
# When enabled, preview video encoding during lyrics review is offloaded to the GCE worker.
|
||||
# This reduces preview generation time from 60+ seconds to ~15-20 seconds.
|
||||
# Requires use_gce_encoding to be enabled and the GCE worker to support /encode-preview endpoint.
|
||||
use_gce_preview_encoding: bool = os.getenv("USE_GCE_PREVIEW_ENCODING", "false").lower() in ("true", "1", "yes")
|
||||
|
||||
# Storage paths
|
||||
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
|
||||
|
||||
# Worker logs storage mode
|
||||
# When enabled, worker logs are stored in a Firestore subcollection (jobs/{job_id}/logs)
|
||||
# instead of an embedded array. This avoids the 1MB document size limit.
|
||||
# Default is true for new deployments.
|
||||
use_log_subcollection: bool = os.getenv("USE_LOG_SUBCOLLECTION", "true").lower() in ("true", "1", "yes")
|
||||
|
||||
# Flacfetch remote service (for torrent downloads)
|
||||
# When configured, audio search uses the remote flacfetch HTTP API instead of local flacfetch.
|
||||
|
||||
@@ -14,7 +14,7 @@ class URLSubmissionRequest(BaseModel):
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: bool = False
|
||||
enable_youtube_upload: Optional[bool] = None # None = use server default
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
@@ -28,7 +28,7 @@ class UploadSubmissionRequest(BaseModel):
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: bool = False
|
||||
enable_youtube_upload: Optional[bool] = None # None = use server default
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
|
||||
@@ -0,0 +1,164 @@
|
||||
"""
|
||||
Worker log entry model for subcollection storage.
|
||||
|
||||
This module defines the LogEntry model for storing worker logs in a Firestore
|
||||
subcollection instead of an embedded array. This avoids the 1MB document size
|
||||
limit that caused job 501258e1 to fail when logs reached 1.26 MB.
|
||||
"""
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from typing import Optional, Dict, Any
|
||||
import uuid
|
||||
|
||||
|
||||
# Default TTL for log entries (30 days)
|
||||
DEFAULT_LOG_TTL_DAYS = 30
|
||||
|
||||
|
||||
@dataclass
|
||||
class WorkerLogEntry:
|
||||
"""
|
||||
Worker log entry for Firestore subcollection storage.
|
||||
|
||||
Stored at: jobs/{job_id}/logs/{log_id}
|
||||
|
||||
This model is separate from the legacy LogEntry (in job.py) which is
|
||||
stored as an embedded array in the job document. This new model supports
|
||||
the subcollection approach with TTL and richer metadata.
|
||||
"""
|
||||
# Core fields (required)
|
||||
timestamp: datetime
|
||||
level: str # "DEBUG", "INFO", "WARNING", "ERROR"
|
||||
worker: str # "audio", "lyrics", "screens", "video", "render", "distribution"
|
||||
message: str
|
||||
|
||||
# Identifiers
|
||||
id: str = field(default_factory=lambda: str(uuid.uuid4()))
|
||||
job_id: str = "" # Set when writing to Firestore
|
||||
|
||||
# TTL for automatic cleanup
|
||||
ttl_expiry: datetime = field(
|
||||
default_factory=lambda: datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
|
||||
)
|
||||
|
||||
# Optional metadata for debugging
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
|
||||
@classmethod
|
||||
def create(
|
||||
cls,
|
||||
job_id: str,
|
||||
worker: str,
|
||||
level: str,
|
||||
message: str,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
ttl_days: int = DEFAULT_LOG_TTL_DAYS
|
||||
) -> "WorkerLogEntry":
|
||||
"""
|
||||
Factory method to create a new log entry.
|
||||
|
||||
Args:
|
||||
job_id: Job ID this log belongs to
|
||||
worker: Worker name (audio, lyrics, screens, video, render, distribution)
|
||||
level: Log level (DEBUG, INFO, WARNING, ERROR)
|
||||
message: Log message (truncated to 1000 chars)
|
||||
metadata: Optional additional metadata
|
||||
ttl_days: Days until log expires (default 30)
|
||||
|
||||
Returns:
|
||||
New WorkerLogEntry instance
|
||||
"""
|
||||
now = datetime.now(timezone.utc)
|
||||
return cls(
|
||||
id=str(uuid.uuid4()),
|
||||
job_id=job_id,
|
||||
timestamp=now,
|
||||
level=level.upper(),
|
||||
worker=worker,
|
||||
message=message[:1000], # Truncate long messages
|
||||
metadata=metadata,
|
||||
ttl_expiry=now + timedelta(days=ttl_days)
|
||||
)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert to dictionary for Firestore storage.
|
||||
|
||||
Returns:
|
||||
Dictionary representation for Firestore
|
||||
"""
|
||||
result = {
|
||||
"id": self.id,
|
||||
"job_id": self.job_id,
|
||||
"timestamp": self.timestamp,
|
||||
"level": self.level,
|
||||
"worker": self.worker,
|
||||
"message": self.message,
|
||||
"ttl_expiry": self.ttl_expiry
|
||||
}
|
||||
if self.metadata:
|
||||
result["metadata"] = self.metadata
|
||||
return result
|
||||
|
||||
def to_legacy_dict(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert to legacy format for backward compatibility with API responses.
|
||||
|
||||
The existing API returns logs in this format:
|
||||
{"timestamp": "...", "level": "INFO", "worker": "audio", "message": "..."}
|
||||
|
||||
Returns:
|
||||
Dictionary in legacy format
|
||||
"""
|
||||
# Format timestamp: use "Z" suffix for UTC, isoformat() for others
|
||||
if self.timestamp.tzinfo is None:
|
||||
# Naive datetime - use as-is
|
||||
timestamp_str = self.timestamp.isoformat()
|
||||
elif self.timestamp.utcoffset() == timedelta(0):
|
||||
# UTC datetime - replace +00:00 with Z for cleaner format
|
||||
timestamp_str = self.timestamp.replace(tzinfo=None).isoformat() + "Z"
|
||||
else:
|
||||
# Non-UTC timezone-aware - use full isoformat with offset
|
||||
timestamp_str = self.timestamp.isoformat()
|
||||
|
||||
return {
|
||||
"timestamp": timestamp_str,
|
||||
"level": self.level,
|
||||
"worker": self.worker,
|
||||
"message": self.message
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, Any]) -> "WorkerLogEntry":
|
||||
"""
|
||||
Create from Firestore document data.
|
||||
|
||||
Args:
|
||||
data: Dictionary from Firestore document
|
||||
|
||||
Returns:
|
||||
WorkerLogEntry instance
|
||||
"""
|
||||
# Handle timestamp conversion (Firestore returns datetime objects)
|
||||
timestamp = data.get("timestamp")
|
||||
if isinstance(timestamp, str):
|
||||
# Parse ISO format string
|
||||
timestamp = datetime.fromisoformat(timestamp.rstrip("Z")).replace(tzinfo=timezone.utc)
|
||||
|
||||
ttl_expiry = data.get("ttl_expiry")
|
||||
if isinstance(ttl_expiry, str):
|
||||
ttl_expiry = datetime.fromisoformat(ttl_expiry.rstrip("Z")).replace(tzinfo=timezone.utc)
|
||||
elif ttl_expiry is None:
|
||||
# Default TTL if not set
|
||||
ttl_expiry = datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
|
||||
|
||||
return cls(
|
||||
id=data.get("id", str(uuid.uuid4())),
|
||||
job_id=data.get("job_id", ""),
|
||||
timestamp=timestamp or datetime.now(timezone.utc),
|
||||
level=data.get("level", "INFO"),
|
||||
worker=data.get("worker", "unknown"),
|
||||
message=data.get("message", ""),
|
||||
metadata=data.get("metadata"),
|
||||
ttl_expiry=ttl_expiry
|
||||
)
|
||||
@@ -349,7 +349,32 @@ class GCEEncodingBackend(EncodingBackend):
|
||||
if not isinstance(result, dict):
|
||||
self.logger.error(f"Unexpected GCE result type: {type(result)}")
|
||||
result = {}
|
||||
output_files = result.get("output_files", {})
|
||||
raw_output_files = result.get("output_files", {})
|
||||
|
||||
# Convert output_files from list of paths to dict
|
||||
# GCE worker returns list like: ["path/Artist - Title (Final Karaoke Lossless 4k).mp4", ...]
|
||||
# We need dict like: {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
|
||||
if isinstance(raw_output_files, list):
|
||||
output_files = {}
|
||||
for path in raw_output_files:
|
||||
if not isinstance(path, str):
|
||||
continue
|
||||
filename = path.split("/")[-1] if "/" in path else path
|
||||
filename_lower = filename.lower()
|
||||
# Map filename patterns to output format keys
|
||||
# Files are named like "Artist - Title (Final Karaoke Lossless 4k).mp4"
|
||||
if "lossless 4k" in filename_lower:
|
||||
if filename.endswith(".mkv"):
|
||||
output_files["mkv_4k"] = path
|
||||
else:
|
||||
output_files["mp4_4k_lossless"] = path
|
||||
elif "lossy 4k" in filename_lower:
|
||||
output_files["mp4_4k_lossy"] = path
|
||||
elif "720p" in filename_lower:
|
||||
output_files["mp4_720p"] = path
|
||||
self.logger.info(f"Converted output_files list to dict: {output_files}")
|
||||
else:
|
||||
output_files = raw_output_files if isinstance(raw_output_files, dict) else {}
|
||||
|
||||
return EncodingOutput(
|
||||
success=True,
|
||||
|
||||
@@ -63,6 +63,11 @@ class EncodingService:
|
||||
"""Check if GCE encoding is enabled and configured."""
|
||||
return self.settings.use_gce_encoding and self.is_configured
|
||||
|
||||
@property
|
||||
def is_preview_enabled(self) -> bool:
|
||||
"""Check if GCE preview encoding is enabled and configured."""
|
||||
return self.settings.use_gce_preview_encoding and self.is_configured
|
||||
|
||||
async def submit_encoding_job(
|
||||
self,
|
||||
job_id: str,
|
||||
@@ -177,6 +182,15 @@ class EncodingService:
|
||||
raise TimeoutError(f"Encoding job {job_id} timed out after {timeout}s")
|
||||
|
||||
status = await self.get_job_status(job_id)
|
||||
|
||||
# Handle case where GCE worker returns a list instead of dict
|
||||
if isinstance(status, list):
|
||||
logger.warning(f"[job:{job_id}] GCE returned list instead of dict: {status}")
|
||||
status = status[0] if status and isinstance(status[0], dict) else {}
|
||||
if not isinstance(status, dict):
|
||||
logger.error(f"[job:{job_id}] Unexpected status type: {type(status)}")
|
||||
status = {}
|
||||
|
||||
job_status = status.get("status", "unknown")
|
||||
progress = status.get("progress", 0)
|
||||
|
||||
@@ -233,6 +247,127 @@ class EncodingService:
|
||||
job_id, progress_callback=progress_callback
|
||||
)
|
||||
|
||||
async def submit_preview_encoding_job(
|
||||
self,
|
||||
job_id: str,
|
||||
ass_gcs_path: str,
|
||||
audio_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
background_color: str = "black",
|
||||
background_image_gcs_path: Optional[str] = None,
|
||||
font_gcs_path: Optional[str] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit a preview video encoding job to the GCE worker.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
ass_gcs_path: GCS path to ASS subtitles file (gs://bucket/path/file.ass)
|
||||
audio_gcs_path: GCS path to audio file
|
||||
output_gcs_path: GCS path for output video
|
||||
background_color: Background color (default: black)
|
||||
background_image_gcs_path: Optional GCS path to background image
|
||||
font_gcs_path: Optional GCS path to custom font file
|
||||
|
||||
Returns:
|
||||
Response from the encoding worker
|
||||
|
||||
Raises:
|
||||
Exception: If submission fails
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._url}/encode-preview"
|
||||
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
|
||||
payload = {
|
||||
"job_id": job_id,
|
||||
"ass_gcs_path": ass_gcs_path,
|
||||
"audio_gcs_path": audio_gcs_path,
|
||||
"output_gcs_path": output_gcs_path,
|
||||
"background_color": background_color,
|
||||
}
|
||||
if background_image_gcs_path:
|
||||
payload["background_image_gcs_path"] = background_image_gcs_path
|
||||
if font_gcs_path:
|
||||
payload["font_gcs_path"] = font_gcs_path
|
||||
|
||||
logger.info(f"[job:{job_id}] Submitting preview encoding job to GCE worker: {url}")
|
||||
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.post(url, json=payload, headers=headers, timeout=30) as resp:
|
||||
if resp.status == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp.status == 409:
|
||||
raise RuntimeError(f"Preview encoding job {job_id} already exists")
|
||||
if resp.status != 200:
|
||||
text = await resp.text()
|
||||
raise RuntimeError(f"Failed to submit preview encoding job: {resp.status} - {text}")
|
||||
|
||||
return await resp.json()
|
||||
|
||||
async def encode_preview_video(
|
||||
self,
|
||||
job_id: str,
|
||||
ass_gcs_path: str,
|
||||
audio_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
background_color: str = "black",
|
||||
background_image_gcs_path: Optional[str] = None,
|
||||
font_gcs_path: Optional[str] = None,
|
||||
timeout: float = 90.0,
|
||||
poll_interval: float = 2.0,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit preview encoding job and wait for completion.
|
||||
|
||||
This is a convenience method that combines submit + wait with shorter
|
||||
timeouts suitable for preview videos.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
ass_gcs_path: GCS path to ASS subtitles file
|
||||
audio_gcs_path: GCS path to audio file
|
||||
output_gcs_path: GCS path for output video
|
||||
background_color: Background color (default: black)
|
||||
background_image_gcs_path: Optional GCS path to background image
|
||||
font_gcs_path: Optional GCS path to custom font file
|
||||
timeout: Maximum time to wait (default 90s for preview)
|
||||
poll_interval: Seconds between status checks (default 2s)
|
||||
|
||||
Returns:
|
||||
Final job status with output files
|
||||
"""
|
||||
# Submit the job
|
||||
submit_result = await self.submit_preview_encoding_job(
|
||||
job_id=job_id,
|
||||
ass_gcs_path=ass_gcs_path,
|
||||
audio_gcs_path=audio_gcs_path,
|
||||
output_gcs_path=output_gcs_path,
|
||||
background_color=background_color,
|
||||
background_image_gcs_path=background_image_gcs_path,
|
||||
font_gcs_path=font_gcs_path,
|
||||
)
|
||||
|
||||
# If cached, return immediately - video already exists in GCS
|
||||
submit_status = submit_result.get("status")
|
||||
if submit_status == "cached":
|
||||
logger.info(f"[job:{job_id}] Preview already cached, returning immediately")
|
||||
return {"status": "complete", "output_path": submit_result.get("output_path")}
|
||||
|
||||
# If in_progress, another request is encoding it - just wait for that
|
||||
if submit_status == "in_progress":
|
||||
logger.info(f"[job:{job_id}] Preview encoding already in progress, waiting")
|
||||
|
||||
# Wait for completion with shorter timeout
|
||||
return await self.wait_for_completion(
|
||||
job_id=job_id,
|
||||
poll_interval=poll_interval,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
async def health_check(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Check the health of the encoding worker.
|
||||
|
||||
@@ -9,6 +9,7 @@ from google.cloud.firestore_v1 import FieldFilter
|
||||
|
||||
from backend.config import settings
|
||||
from backend.models.job import Job, JobStatus, TimelineEvent
|
||||
from backend.models.worker_log import WorkerLogEntry
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -245,10 +246,13 @@ class FirestoreService:
|
||||
def append_worker_log(self, job_id: str, log_entry: Dict[str, Any]) -> None:
|
||||
"""
|
||||
Atomically append a log entry to worker_logs using ArrayUnion.
|
||||
|
||||
|
||||
This avoids the race condition of read-modify-write when multiple
|
||||
workers are logging concurrently.
|
||||
|
||||
|
||||
DEPRECATED: Use append_log_to_subcollection() instead to avoid
|
||||
the 1MB document size limit.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
log_entry: Log entry dict with timestamp, level, worker, message
|
||||
@@ -263,6 +267,165 @@ class FirestoreService:
|
||||
except Exception as e:
|
||||
# Log but don't raise - logging shouldn't break workers
|
||||
logger.debug(f"Error appending worker log for job {job_id}: {e}")
|
||||
|
||||
# ============================================
|
||||
# Worker Log Subcollection Methods
|
||||
# ============================================
|
||||
# These methods store logs in a subcollection (jobs/{job_id}/logs)
|
||||
# instead of an embedded array to avoid the 1MB document size limit.
|
||||
|
||||
def append_log_to_subcollection(self, job_id: str, log_entry: WorkerLogEntry) -> None:
|
||||
"""
|
||||
Append a log entry to the logs subcollection.
|
||||
|
||||
Stores logs at: jobs/{job_id}/logs/{log_id}
|
||||
|
||||
This approach avoids the 1MB document size limit by storing each
|
||||
log entry as a separate document in a subcollection.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
log_entry: WorkerLogEntry instance
|
||||
"""
|
||||
try:
|
||||
# Ensure job_id is set on the log entry
|
||||
log_entry.job_id = job_id
|
||||
|
||||
# Get subcollection reference
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
|
||||
# Add document with auto-generated ID or use log_entry.id
|
||||
doc_ref = logs_ref.document(log_entry.id)
|
||||
doc_ref.set(log_entry.to_dict())
|
||||
|
||||
# Don't log every append - too spammy
|
||||
except Exception as e:
|
||||
# Log but don't raise - logging shouldn't break workers
|
||||
logger.debug(f"Error appending log to subcollection for job {job_id}: {e}")
|
||||
|
||||
def get_logs_from_subcollection(
|
||||
self,
|
||||
job_id: str,
|
||||
limit: int = 500,
|
||||
since_timestamp: Optional[datetime] = None,
|
||||
worker: Optional[str] = None,
|
||||
offset: int = 0
|
||||
) -> List[WorkerLogEntry]:
|
||||
"""
|
||||
Get log entries from the logs subcollection.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
limit: Maximum number of logs to return (default 500)
|
||||
since_timestamp: Return only logs after this timestamp
|
||||
worker: Filter by worker name (optional)
|
||||
offset: Number of logs to skip (for pagination)
|
||||
|
||||
Returns:
|
||||
List of WorkerLogEntry instances, ordered by timestamp ascending
|
||||
"""
|
||||
try:
|
||||
# Get subcollection reference
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
|
||||
# Build query
|
||||
query = logs_ref.order_by("timestamp", direction=firestore.Query.ASCENDING)
|
||||
|
||||
if since_timestamp:
|
||||
query = query.where(filter=FieldFilter("timestamp", ">", since_timestamp))
|
||||
|
||||
if worker:
|
||||
query = query.where(filter=FieldFilter("worker", "==", worker))
|
||||
|
||||
# Apply offset and limit
|
||||
if offset > 0:
|
||||
query = query.offset(offset)
|
||||
|
||||
query = query.limit(limit)
|
||||
|
||||
# Execute query
|
||||
docs = query.stream()
|
||||
logs = [WorkerLogEntry.from_dict(doc.to_dict()) for doc in docs]
|
||||
|
||||
return logs
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting logs from subcollection for job {job_id}: {e}")
|
||||
return []
|
||||
|
||||
def get_logs_count_from_subcollection(self, job_id: str) -> int:
|
||||
"""
|
||||
Get the total count of log entries in the subcollection.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
|
||||
Returns:
|
||||
Total count of log entries
|
||||
"""
|
||||
try:
|
||||
# Get subcollection reference
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
|
||||
# Use aggregation query for efficient counting
|
||||
count_query = logs_ref.count()
|
||||
result = count_query.get()
|
||||
|
||||
# Result is a list of AggregationResult, we want the first one's count
|
||||
if result and len(result) > 0:
|
||||
return result[0][0].value
|
||||
return 0
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error counting logs for job {job_id}: {e}")
|
||||
return 0
|
||||
|
||||
def delete_logs_subcollection(self, job_id: str, batch_size: int = 500) -> int:
|
||||
"""
|
||||
Delete all log entries in the logs subcollection.
|
||||
|
||||
This is used when deleting a job to clean up its logs.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
batch_size: Number of documents to delete per batch
|
||||
|
||||
Returns:
|
||||
Number of logs deleted
|
||||
"""
|
||||
try:
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
deleted_count = 0
|
||||
|
||||
while True:
|
||||
# Get a batch of documents
|
||||
docs = logs_ref.limit(batch_size).stream()
|
||||
deleted_in_batch = 0
|
||||
|
||||
# Delete in a batch
|
||||
batch = self.db.batch()
|
||||
for doc in docs:
|
||||
batch.delete(doc.reference)
|
||||
deleted_in_batch += 1
|
||||
|
||||
if deleted_in_batch == 0:
|
||||
break
|
||||
|
||||
batch.commit()
|
||||
deleted_count += deleted_in_batch
|
||||
|
||||
# If we deleted less than batch_size, we're done
|
||||
if deleted_in_batch < batch_size:
|
||||
break
|
||||
|
||||
if deleted_count > 0:
|
||||
logger.info(f"Deleted {deleted_count} logs for job {job_id}")
|
||||
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting logs subcollection for job {job_id}: {e}")
|
||||
return 0
|
||||
|
||||
# ============================================
|
||||
# Token Management Methods
|
||||
|
||||
@@ -13,7 +13,9 @@ import uuid
|
||||
from datetime import datetime
|
||||
from typing import Optional, Dict, Any, List
|
||||
|
||||
from backend.config import settings
|
||||
from backend.models.job import Job, JobStatus, JobCreate, STATE_TRANSITIONS
|
||||
from backend.models.worker_log import WorkerLogEntry
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
@@ -224,7 +226,7 @@ class JobManager:
|
||||
return urls
|
||||
|
||||
def delete_job(self, job_id: str, delete_files: bool = True) -> None:
|
||||
"""Delete a job and optionally its files."""
|
||||
"""Delete a job, its files, and its logs subcollection."""
|
||||
if delete_files:
|
||||
job = self.get_job(job_id)
|
||||
if job and job.output_files:
|
||||
@@ -233,7 +235,15 @@ class JobManager:
|
||||
self.storage.delete_file(gcs_path)
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting file {gcs_path}: {e}")
|
||||
|
||||
|
||||
# Delete logs subcollection first (must be done before deleting parent doc)
|
||||
try:
|
||||
deleted_logs = self.firestore.delete_logs_subcollection(job_id)
|
||||
if deleted_logs > 0:
|
||||
logger.info(f"Deleted {deleted_logs} log entries for job {job_id}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Error deleting logs subcollection for job {job_id}: {e}")
|
||||
|
||||
self.firestore.delete_job(job_id)
|
||||
logger.info(f"Deleted job {job_id}")
|
||||
|
||||
@@ -579,33 +589,45 @@ class JobManager:
|
||||
worker: str,
|
||||
level: str,
|
||||
message: str,
|
||||
max_logs: int = 500
|
||||
max_logs: int = 500,
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
) -> None:
|
||||
"""
|
||||
Append a log entry to the job's worker_logs atomically.
|
||||
|
||||
Uses Firestore ArrayUnion for atomic append to avoid race conditions
|
||||
when multiple workers (audio + lyrics) are logging concurrently.
|
||||
|
||||
Note: max_logs is not enforced per-append (would require transactions).
|
||||
Logs may grow beyond max_logs; trim periodically if needed.
|
||||
|
||||
Append a log entry to the job's logs.
|
||||
|
||||
By default (USE_LOG_SUBCOLLECTION=true), logs are stored in a Firestore
|
||||
subcollection (jobs/{job_id}/logs) to avoid the 1MB document size limit.
|
||||
|
||||
For backward compatibility (USE_LOG_SUBCOLLECTION=false), logs are stored
|
||||
in the embedded worker_logs array using atomic ArrayUnion.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
worker: Worker name (audio, lyrics, screens, video, render)
|
||||
worker: Worker name (audio, lyrics, screens, video, render, distribution)
|
||||
level: Log level (DEBUG, INFO, WARNING, ERROR)
|
||||
message: Log message
|
||||
max_logs: Not used (kept for API compatibility)
|
||||
metadata: Optional additional metadata dict
|
||||
"""
|
||||
log_entry = {
|
||||
'timestamp': datetime.utcnow().isoformat() + 'Z',
|
||||
'level': level,
|
||||
'worker': worker,
|
||||
'message': message[:1000] # Truncate long messages
|
||||
}
|
||||
|
||||
# Use atomic ArrayUnion to avoid race conditions
|
||||
self.firestore.append_worker_log(job_id, log_entry)
|
||||
if settings.use_log_subcollection:
|
||||
# New subcollection approach - avoids 1MB limit
|
||||
log_entry = WorkerLogEntry.create(
|
||||
job_id=job_id,
|
||||
worker=worker,
|
||||
level=level,
|
||||
message=message,
|
||||
metadata=metadata
|
||||
)
|
||||
self.firestore.append_log_to_subcollection(job_id, log_entry)
|
||||
else:
|
||||
# Legacy embedded array approach
|
||||
log_entry = {
|
||||
'timestamp': datetime.utcnow().isoformat() + 'Z',
|
||||
'level': level,
|
||||
'worker': worker,
|
||||
'message': message[:1000] # Truncate long messages
|
||||
}
|
||||
self.firestore.append_worker_log(job_id, log_entry)
|
||||
|
||||
def get_worker_logs(
|
||||
self,
|
||||
@@ -615,28 +637,68 @@ class JobManager:
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Get worker logs for a job, optionally filtered by worker and index.
|
||||
|
||||
|
||||
By default (USE_LOG_SUBCOLLECTION=true), logs are read from the
|
||||
subcollection. Falls back to embedded array for older jobs.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
since_index: Return only logs after this index (for pagination)
|
||||
worker: Filter by worker name (optional)
|
||||
|
||||
|
||||
Returns:
|
||||
List of log entries as dicts
|
||||
List of log entries as dicts (in legacy format for API compatibility)
|
||||
"""
|
||||
if settings.use_log_subcollection:
|
||||
# Try subcollection first
|
||||
subcollection_logs = self.firestore.get_logs_from_subcollection(
|
||||
job_id=job_id,
|
||||
offset=since_index,
|
||||
worker=worker,
|
||||
limit=500
|
||||
)
|
||||
if subcollection_logs:
|
||||
# Convert to legacy format for API compatibility
|
||||
return [log.to_legacy_dict() for log in subcollection_logs]
|
||||
# Fall through to check embedded array for older jobs
|
||||
|
||||
# Embedded array approach (legacy jobs or fallback)
|
||||
job = self.get_job(job_id)
|
||||
if not job or not job.worker_logs:
|
||||
return []
|
||||
|
||||
|
||||
logs = [log.dict() if hasattr(log, 'dict') else log for log in job.worker_logs]
|
||||
|
||||
|
||||
# Filter by index
|
||||
if since_index > 0:
|
||||
logs = logs[since_index:]
|
||||
|
||||
|
||||
# Filter by worker
|
||||
if worker:
|
||||
logs = [log for log in logs if log.get('worker') == worker]
|
||||
|
||||
|
||||
return logs
|
||||
|
||||
def get_worker_logs_count(self, job_id: str) -> int:
|
||||
"""
|
||||
Get the total count of worker logs for a job.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
|
||||
Returns:
|
||||
Total count of logs
|
||||
"""
|
||||
if settings.use_log_subcollection:
|
||||
# Try subcollection first
|
||||
count = self.firestore.get_logs_count_from_subcollection(job_id)
|
||||
if count > 0:
|
||||
return count
|
||||
# Fall through to check embedded array
|
||||
|
||||
# Embedded array (legacy jobs or fallback)
|
||||
job = self.get_job(job_id)
|
||||
if not job or not job.worker_logs:
|
||||
return 0
|
||||
return len(job.worker_logs)
|
||||
|
||||
@@ -0,0 +1,443 @@
|
||||
"""
|
||||
Integration tests for worker log subcollection storage.
|
||||
|
||||
Tests the subcollection approach using real Firestore emulator to verify:
|
||||
- Logs are stored in subcollection (jobs/{job_id}/logs)
|
||||
- TTL expiry field is set correctly
|
||||
- Large log volumes work without 1MB limit issues
|
||||
- Concurrent writes are handled correctly
|
||||
- Logs can be queried efficiently
|
||||
|
||||
Run with: ./scripts/run-emulator-tests.sh
|
||||
"""
|
||||
import pytest
|
||||
import time
|
||||
import requests
|
||||
import os
|
||||
import threading
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from datetime import datetime, timezone, timedelta
|
||||
|
||||
|
||||
def emulators_running() -> bool:
|
||||
"""Check if GCP emulators are running."""
|
||||
try:
|
||||
requests.get("http://127.0.0.1:8080", timeout=1)
|
||||
return True
|
||||
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
|
||||
return False
|
||||
|
||||
|
||||
# Skip all tests in this module if emulators aren't running
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not emulators_running(),
|
||||
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
|
||||
)
|
||||
|
||||
# Set up environment for emulator
|
||||
os.environ["FIRESTORE_EMULATOR_HOST"] = "127.0.0.1:8080"
|
||||
os.environ["GOOGLE_CLOUD_PROJECT"] = "test-project"
|
||||
|
||||
|
||||
class TestWorkerLogsSubcollectionDirect:
|
||||
"""Direct Firestore tests for worker logs subcollection."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup_firestore(self):
|
||||
"""Set up Firestore client for each test."""
|
||||
from google.cloud import firestore
|
||||
self.db = firestore.Client(project="test-project")
|
||||
self.collection = "test-subcollection-jobs"
|
||||
yield
|
||||
|
||||
def _create_test_job(self):
|
||||
"""Create a test job document."""
|
||||
job_id = f"test-sub-{int(time.time() * 1000)}"
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.set({
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"created_at": datetime.now(timezone.utc),
|
||||
"worker_logs": [] # Legacy field - kept empty for new jobs
|
||||
})
|
||||
return job_id
|
||||
|
||||
def _append_log_to_subcollection(self, job_id: str, worker: str, message: str, ttl_days: int = 30):
|
||||
"""Add log to subcollection at jobs/{job_id}/logs."""
|
||||
import uuid
|
||||
log_id = str(uuid.uuid4())
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
logs_ref.document(log_id).set({
|
||||
"id": log_id,
|
||||
"job_id": job_id,
|
||||
"timestamp": now,
|
||||
"level": "INFO",
|
||||
"worker": worker,
|
||||
"message": message,
|
||||
"ttl_expiry": now + timedelta(days=ttl_days)
|
||||
})
|
||||
return log_id
|
||||
|
||||
def _get_logs_from_subcollection(self, job_id: str, worker: str = None, limit: int = 500):
|
||||
"""Get logs from subcollection."""
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
query = logs_ref.order_by("timestamp")
|
||||
|
||||
if worker:
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
query = query.where(filter=FieldFilter("worker", "==", worker))
|
||||
|
||||
query = query.limit(limit)
|
||||
|
||||
return [doc.to_dict() for doc in query.stream()]
|
||||
|
||||
def _count_logs_in_subcollection(self, job_id: str) -> int:
|
||||
"""Count logs in subcollection."""
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
count_query = logs_ref.count()
|
||||
result = count_query.get()
|
||||
if result and len(result) > 0:
|
||||
return result[0][0].value
|
||||
return 0
|
||||
|
||||
def _delete_logs_subcollection(self, job_id: str, batch_size: int = 100) -> int:
|
||||
"""Delete all logs in subcollection."""
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
deleted_count = 0
|
||||
|
||||
while True:
|
||||
docs = list(logs_ref.limit(batch_size).stream())
|
||||
if not docs:
|
||||
break
|
||||
|
||||
batch = self.db.batch()
|
||||
for doc in docs:
|
||||
batch.delete(doc.reference)
|
||||
deleted_count += 1
|
||||
|
||||
batch.commit()
|
||||
|
||||
return deleted_count
|
||||
|
||||
def test_subcollection_single_write(self):
|
||||
"""Test writing single log to subcollection."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
log_id = self._append_log_to_subcollection(job_id, "test", "Single log message")
|
||||
|
||||
time.sleep(0.1)
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(logs) == 1
|
||||
assert logs[0]["message"] == "Single log message"
|
||||
assert logs[0]["worker"] == "test"
|
||||
assert logs[0]["id"] == log_id
|
||||
print("Single write to subcollection works")
|
||||
|
||||
def test_subcollection_ttl_field_is_set(self):
|
||||
"""Test TTL expiry field is set correctly."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
self._append_log_to_subcollection(job_id, "test", "Log with TTL", ttl_days=7)
|
||||
|
||||
time.sleep(0.1)
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(logs) == 1
|
||||
ttl_expiry = logs[0]["ttl_expiry"]
|
||||
assert ttl_expiry is not None
|
||||
|
||||
# TTL should be ~7 days from now
|
||||
expected_ttl = datetime.now(timezone.utc) + timedelta(days=7)
|
||||
if hasattr(ttl_expiry, 'timestamp'):
|
||||
# Firestore datetime object
|
||||
diff = abs((ttl_expiry.replace(tzinfo=timezone.utc) - expected_ttl).total_seconds())
|
||||
else:
|
||||
diff = 0 # If already datetime
|
||||
assert diff < 60, f"TTL expiry should be ~7 days from now, diff was {diff}s"
|
||||
print("TTL field is set correctly")
|
||||
|
||||
def test_subcollection_sequential_writes(self):
|
||||
"""Test multiple sequential writes to subcollection."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
for i in range(20):
|
||||
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
|
||||
|
||||
time.sleep(0.2)
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(logs) == 20, f"Expected 20 logs, got {len(logs)}"
|
||||
print(f"Sequential writes: {len(logs)} logs created")
|
||||
|
||||
def test_subcollection_concurrent_writes(self):
|
||||
"""Test concurrent writes to subcollection - no race conditions."""
|
||||
job_id = self._create_test_job()
|
||||
num_writes = 50
|
||||
|
||||
def write_log(index):
|
||||
self._append_log_to_subcollection(job_id, "worker", f"Concurrent Log {index}")
|
||||
|
||||
with ThreadPoolExecutor(max_workers=10) as executor:
|
||||
futures = [executor.submit(write_log, i) for i in range(num_writes)]
|
||||
for future in as_completed(futures):
|
||||
future.result()
|
||||
|
||||
time.sleep(0.5)
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(logs) == num_writes, f"Expected {num_writes} logs, got {len(logs)}"
|
||||
print(f"Concurrent writes: All {num_writes} logs preserved")
|
||||
|
||||
def test_subcollection_large_volume(self):
|
||||
"""Test large volume of logs - demonstrates no 1MB limit."""
|
||||
job_id = self._create_test_job()
|
||||
num_logs = 500 # Would exceed 1MB in embedded array
|
||||
|
||||
# Write logs in batches
|
||||
batch_size = 50
|
||||
for batch_start in range(0, num_logs, batch_size):
|
||||
with ThreadPoolExecutor(max_workers=10) as executor:
|
||||
futures = []
|
||||
for i in range(batch_start, min(batch_start + batch_size, num_logs)):
|
||||
futures.append(executor.submit(
|
||||
self._append_log_to_subcollection,
|
||||
job_id,
|
||||
"stress",
|
||||
f"Large volume test log {i} - " + "x" * 500 # ~500 bytes per log
|
||||
))
|
||||
for future in as_completed(futures):
|
||||
future.result()
|
||||
|
||||
time.sleep(1)
|
||||
count = self._count_logs_in_subcollection(job_id)
|
||||
|
||||
assert count == num_logs, f"Expected {num_logs} logs, got {count}"
|
||||
print(f"Large volume: {count} logs created (~{count * 500 / 1024:.1f}KB would exceed 1MB if embedded)")
|
||||
|
||||
def test_subcollection_filter_by_worker(self):
|
||||
"""Test filtering logs by worker type."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
# Add logs from different workers
|
||||
for i in range(10):
|
||||
self._append_log_to_subcollection(job_id, "audio", f"Audio log {i}")
|
||||
self._append_log_to_subcollection(job_id, "lyrics", f"Lyrics log {i}")
|
||||
self._append_log_to_subcollection(job_id, "video", f"Video log {i}")
|
||||
|
||||
time.sleep(0.3)
|
||||
|
||||
# Query by worker
|
||||
audio_logs = self._get_logs_from_subcollection(job_id, worker="audio")
|
||||
lyrics_logs = self._get_logs_from_subcollection(job_id, worker="lyrics")
|
||||
all_logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(audio_logs) == 10, f"Expected 10 audio logs, got {len(audio_logs)}"
|
||||
assert len(lyrics_logs) == 10, f"Expected 10 lyrics logs, got {len(lyrics_logs)}"
|
||||
assert len(all_logs) == 30, f"Expected 30 total logs, got {len(all_logs)}"
|
||||
print("Worker filtering works correctly")
|
||||
|
||||
def test_subcollection_ordered_by_timestamp(self):
|
||||
"""Test logs are returned in timestamp order."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
# Add logs with slight delays to ensure different timestamps
|
||||
for i in range(5):
|
||||
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
|
||||
time.sleep(0.02)
|
||||
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
# Verify order (timestamps should be ascending)
|
||||
for i in range(len(logs) - 1):
|
||||
assert logs[i]["timestamp"] <= logs[i + 1]["timestamp"], \
|
||||
f"Logs not in order at index {i}"
|
||||
print("Logs are ordered by timestamp")
|
||||
|
||||
def test_subcollection_delete_all_logs(self):
|
||||
"""Test deleting all logs in subcollection."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
# Add logs
|
||||
for i in range(25):
|
||||
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
|
||||
|
||||
time.sleep(0.2)
|
||||
count_before = self._count_logs_in_subcollection(job_id)
|
||||
assert count_before == 25
|
||||
|
||||
# Delete all logs
|
||||
deleted = self._delete_logs_subcollection(job_id)
|
||||
|
||||
time.sleep(0.2)
|
||||
count_after = self._count_logs_in_subcollection(job_id)
|
||||
|
||||
assert deleted == 25, f"Expected to delete 25 logs, deleted {deleted}"
|
||||
assert count_after == 0, f"Expected 0 logs after delete, got {count_after}"
|
||||
print(f"Deleted {deleted} logs from subcollection")
|
||||
|
||||
def test_subcollection_concurrent_workers_interleaved(self):
|
||||
"""Test simulating audio and lyrics workers writing concurrently."""
|
||||
job_id = self._create_test_job()
|
||||
logs_per_worker = 20
|
||||
|
||||
def audio_worker():
|
||||
for i in range(logs_per_worker):
|
||||
self._append_log_to_subcollection(job_id, "audio", f"Audio processing step {i}")
|
||||
time.sleep(0.005)
|
||||
|
||||
def lyrics_worker():
|
||||
for i in range(logs_per_worker):
|
||||
self._append_log_to_subcollection(job_id, "lyrics", f"Lyrics processing step {i}")
|
||||
time.sleep(0.005)
|
||||
|
||||
def video_worker():
|
||||
for i in range(logs_per_worker):
|
||||
self._append_log_to_subcollection(job_id, "video", f"Video encoding step {i}")
|
||||
time.sleep(0.005)
|
||||
|
||||
# Start all workers
|
||||
threads = [
|
||||
threading.Thread(target=audio_worker),
|
||||
threading.Thread(target=lyrics_worker),
|
||||
threading.Thread(target=video_worker)
|
||||
]
|
||||
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
|
||||
time.sleep(0.5)
|
||||
|
||||
audio_logs = self._get_logs_from_subcollection(job_id, worker="audio")
|
||||
lyrics_logs = self._get_logs_from_subcollection(job_id, worker="lyrics")
|
||||
video_logs = self._get_logs_from_subcollection(job_id, worker="video")
|
||||
total = self._count_logs_in_subcollection(job_id)
|
||||
|
||||
assert len(audio_logs) == logs_per_worker
|
||||
assert len(lyrics_logs) == logs_per_worker
|
||||
assert len(video_logs) == logs_per_worker
|
||||
assert total == logs_per_worker * 3
|
||||
|
||||
print(f"Concurrent workers: {len(audio_logs)} audio + {len(lyrics_logs)} lyrics + {len(video_logs)} video = {total}")
|
||||
|
||||
def test_subcollection_job_deletion_cleans_up_logs(self):
|
||||
"""Test that deleting job document doesn't orphan logs."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
# Add logs
|
||||
for i in range(10):
|
||||
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
|
||||
|
||||
time.sleep(0.2)
|
||||
assert self._count_logs_in_subcollection(job_id) == 10
|
||||
|
||||
# Delete logs first (must happen before parent doc deletion in real code)
|
||||
deleted = self._delete_logs_subcollection(job_id)
|
||||
assert deleted == 10
|
||||
|
||||
# Now delete job document
|
||||
self.db.collection(self.collection).document(job_id).delete()
|
||||
|
||||
# Verify subcollection is empty (Firestore doesn't cascade delete subcollections)
|
||||
time.sleep(0.2)
|
||||
assert self._count_logs_in_subcollection(job_id) == 0
|
||||
print("Job deletion with log cleanup works correctly")
|
||||
|
||||
|
||||
class TestSubcollectionVsEmbeddedArray:
|
||||
"""Compare subcollection approach with embedded array."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup_firestore(self):
|
||||
"""Set up Firestore client for each test."""
|
||||
from google.cloud import firestore
|
||||
self.db = firestore.Client(project="test-project")
|
||||
self.collection = "test-comparison-jobs"
|
||||
yield
|
||||
|
||||
def _create_test_job(self, use_array: bool):
|
||||
"""Create test job."""
|
||||
job_id = f"test-cmp-{int(time.time() * 1000)}-{'arr' if use_array else 'sub'}"
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.set({
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"created_at": datetime.now(timezone.utc),
|
||||
"worker_logs": []
|
||||
})
|
||||
return job_id
|
||||
|
||||
def _append_log_array(self, job_id: str, message: str):
|
||||
"""Add log to embedded array."""
|
||||
from google.cloud import firestore
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.update({
|
||||
"worker_logs": firestore.ArrayUnion([{
|
||||
"timestamp": datetime.now(timezone.utc).isoformat() + "Z",
|
||||
"level": "INFO",
|
||||
"worker": "test",
|
||||
"message": message
|
||||
}])
|
||||
})
|
||||
|
||||
def _append_log_subcollection(self, job_id: str, message: str):
|
||||
"""Add log to subcollection."""
|
||||
import uuid
|
||||
now = datetime.now(timezone.utc)
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
logs_ref.document(str(uuid.uuid4())).set({
|
||||
"timestamp": now,
|
||||
"level": "INFO",
|
||||
"worker": "test",
|
||||
"message": message,
|
||||
"ttl_expiry": now + timedelta(days=30)
|
||||
})
|
||||
|
||||
def test_document_size_comparison(self):
|
||||
"""Compare document sizes between approaches."""
|
||||
array_job_id = self._create_test_job(use_array=True)
|
||||
sub_job_id = self._create_test_job(use_array=False)
|
||||
|
||||
num_logs = 100
|
||||
large_message = "x" * 5000 # 5KB per log
|
||||
|
||||
# Add logs to both
|
||||
print("\nAdding logs to both approaches...")
|
||||
for i in range(num_logs):
|
||||
self._append_log_array(array_job_id, f"Array log {i}: {large_message}")
|
||||
self._append_log_subcollection(sub_job_id, f"Sub log {i}: {large_message}")
|
||||
|
||||
time.sleep(0.5)
|
||||
|
||||
# Get document sizes (approximate via raw data)
|
||||
array_doc = self.db.collection(self.collection).document(array_job_id).get()
|
||||
sub_doc = self.db.collection(self.collection).document(sub_job_id).get()
|
||||
|
||||
array_data = array_doc.to_dict()
|
||||
sub_data = sub_doc.to_dict()
|
||||
|
||||
array_logs = array_data.get("worker_logs", [])
|
||||
sub_logs = sub_data.get("worker_logs", [])
|
||||
|
||||
print(f"\nEmbedded array: {len(array_logs)} logs in document")
|
||||
print(f"Subcollection: {len(sub_logs)} logs in document (0 expected)")
|
||||
|
||||
# Embedded array should have all logs in document
|
||||
assert len(array_logs) == num_logs
|
||||
|
||||
# Subcollection should have empty array in main document
|
||||
assert len(sub_logs) == 0
|
||||
|
||||
# Subcollection logs are in subcollection
|
||||
sub_count = self.db.collection(self.collection).document(sub_job_id).collection("logs").count().get()[0][0].value
|
||||
assert sub_count == num_logs
|
||||
|
||||
print(f"Subcollection logs stored separately: {sub_count}")
|
||||
print("Subcollection approach keeps main document small")
|
||||
|
||||
|
||||
print("Worker logs subcollection integration tests ready")
|
||||
@@ -1128,7 +1128,8 @@ class TestFinaliseOnlyModels:
|
||||
assert request.enable_txt is None
|
||||
assert request.brand_prefix is None
|
||||
assert request.keep_brand_code is None
|
||||
assert request.enable_youtube_upload is False
|
||||
# YouTube upload default is None (server applies default_enable_youtube_upload)
|
||||
assert request.enable_youtube_upload is None
|
||||
assert request.dropbox_path is None
|
||||
assert request.gdrive_folder_id is None
|
||||
|
||||
|
||||
@@ -149,7 +149,8 @@ class TestUploadSubmissionRequest:
|
||||
# CDG/TXT disabled by default (requires style config)
|
||||
assert request.enable_cdg is False
|
||||
assert request.enable_txt is False
|
||||
assert request.enable_youtube_upload is False
|
||||
# YouTube upload default is None (use server default)
|
||||
assert request.enable_youtube_upload is None
|
||||
|
||||
|
||||
class TestStartReviewRequest:
|
||||
|
||||
@@ -425,6 +425,111 @@ class TestVideoWorkerOrchestratorEncoding:
|
||||
|
||||
assert "Encoding failed" in str(exc_info.value)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_encoding_gce_downloads_files(self):
|
||||
"""Test that GCE encoding downloads files from GCS to local directory.
|
||||
|
||||
This test verifies the fix for YouTube upload failure when using GCE encoding.
|
||||
GCE encoding returns GCS blob paths, which need to be downloaded locally
|
||||
before YouTube upload can access them.
|
||||
"""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path=os.path.join(temp_dir, "title.mov"),
|
||||
karaoke_video_path=os.path.join(temp_dir, "karaoke.mov"),
|
||||
instrumental_audio_path=os.path.join(temp_dir, "audio.flac"),
|
||||
output_dir=temp_dir,
|
||||
)
|
||||
|
||||
# Mock storage service
|
||||
mock_storage = MagicMock()
|
||||
mock_storage.download_file = MagicMock()
|
||||
|
||||
orchestrator = VideoWorkerOrchestrator(
|
||||
config,
|
||||
storage=mock_storage,
|
||||
)
|
||||
|
||||
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
|
||||
from backend.services.encoding_interface import EncodingOutput
|
||||
|
||||
# GCE backend returns GCS blob paths (not local paths)
|
||||
mock_backend = MagicMock()
|
||||
mock_backend.name = "gce" # Important: must be "gce" to trigger download
|
||||
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
|
||||
success=True,
|
||||
lossless_4k_mp4_path="jobs/test-job/finals/output_4k_lossless.mp4",
|
||||
lossy_4k_mp4_path="jobs/test-job/finals/output_4k_lossy.mp4",
|
||||
lossy_720p_mp4_path="jobs/test-job/finals/output_720p.mp4",
|
||||
lossless_mkv_path="jobs/test-job/finals/output_4k.mkv",
|
||||
encoding_time_seconds=60.0,
|
||||
encoding_backend="gce",
|
||||
))
|
||||
mock_get.return_value = mock_backend
|
||||
|
||||
await orchestrator._run_encoding()
|
||||
|
||||
# Verify download_file was called for each output file
|
||||
assert mock_storage.download_file.call_count == 4
|
||||
|
||||
# Verify the result paths were updated to local paths
|
||||
assert orchestrator.result.final_video.startswith(temp_dir)
|
||||
assert orchestrator.result.final_video_mkv.startswith(temp_dir)
|
||||
assert orchestrator.result.final_video_lossy.startswith(temp_dir)
|
||||
assert orchestrator.result.final_video_720p.startswith(temp_dir)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_encoding_local_does_not_download(self):
|
||||
"""Test that local encoding does NOT trigger GCS download.
|
||||
|
||||
Local encoding produces files directly in the output directory,
|
||||
so no download is needed.
|
||||
"""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
config = OrchestratorConfig(
|
||||
job_id="test-job",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
title_video_path=os.path.join(temp_dir, "title.mov"),
|
||||
karaoke_video_path=os.path.join(temp_dir, "karaoke.mov"),
|
||||
instrumental_audio_path=os.path.join(temp_dir, "audio.flac"),
|
||||
output_dir=temp_dir,
|
||||
)
|
||||
|
||||
# Mock storage service
|
||||
mock_storage = MagicMock()
|
||||
mock_storage.download_file = MagicMock()
|
||||
|
||||
orchestrator = VideoWorkerOrchestrator(
|
||||
config,
|
||||
storage=mock_storage,
|
||||
)
|
||||
|
||||
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
|
||||
from backend.services.encoding_interface import EncodingOutput
|
||||
|
||||
# Local backend returns local paths directly
|
||||
mock_backend = MagicMock()
|
||||
mock_backend.name = "local" # Local backend
|
||||
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
|
||||
success=True,
|
||||
lossless_4k_mp4_path=os.path.join(temp_dir, "lossless.mp4"),
|
||||
lossy_4k_mp4_path=os.path.join(temp_dir, "lossy.mp4"),
|
||||
lossy_720p_mp4_path=os.path.join(temp_dir, "720p.mp4"),
|
||||
lossless_mkv_path=os.path.join(temp_dir, "lossless.mkv"),
|
||||
encoding_time_seconds=120.0,
|
||||
encoding_backend="local",
|
||||
))
|
||||
mock_get.return_value = mock_backend
|
||||
|
||||
await orchestrator._run_encoding()
|
||||
|
||||
# Verify download_file was NOT called for local encoding
|
||||
mock_storage.download_file.assert_not_called()
|
||||
|
||||
|
||||
class TestVideoWorkerOrchestratorOrganization:
|
||||
"""Test organization stage."""
|
||||
|
||||
@@ -0,0 +1,509 @@
|
||||
"""
|
||||
Unit tests for worker log subcollection functionality.
|
||||
|
||||
Tests the WorkerLogEntry model, FirestoreService subcollection methods,
|
||||
and JobManager log operations with the feature flag.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, patch
|
||||
from datetime import datetime, timezone, timedelta
|
||||
import uuid
|
||||
|
||||
# Mock Firestore before imports
|
||||
import sys
|
||||
sys.modules['google.cloud.firestore'] = MagicMock()
|
||||
sys.modules['google.cloud.firestore_v1'] = MagicMock()
|
||||
|
||||
from backend.models.worker_log import WorkerLogEntry, DEFAULT_LOG_TTL_DAYS
|
||||
|
||||
|
||||
class TestWorkerLogEntry:
|
||||
"""Tests for WorkerLogEntry dataclass."""
|
||||
|
||||
def test_create_log_entry(self):
|
||||
"""Test creating a log entry with factory method."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test message"
|
||||
)
|
||||
|
||||
assert entry.job_id == "job123"
|
||||
assert entry.worker == "audio"
|
||||
assert entry.level == "INFO"
|
||||
assert entry.message == "Test message"
|
||||
assert entry.id is not None
|
||||
assert entry.timestamp is not None
|
||||
assert entry.ttl_expiry is not None
|
||||
# TTL should be ~30 days from now
|
||||
expected_ttl = datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
|
||||
assert abs((entry.ttl_expiry - expected_ttl).total_seconds()) < 5
|
||||
|
||||
def test_create_log_entry_truncates_long_message(self):
|
||||
"""Test that messages longer than 1000 chars are truncated."""
|
||||
long_message = "x" * 2000
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message=long_message
|
||||
)
|
||||
|
||||
assert len(entry.message) == 1000
|
||||
|
||||
def test_create_log_entry_normalizes_level(self):
|
||||
"""Test that log level is normalized to uppercase."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="info",
|
||||
message="Test"
|
||||
)
|
||||
|
||||
assert entry.level == "INFO"
|
||||
|
||||
def test_create_log_entry_custom_ttl(self):
|
||||
"""Test creating entry with custom TTL."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test",
|
||||
ttl_days=7
|
||||
)
|
||||
|
||||
expected_ttl = datetime.now(timezone.utc) + timedelta(days=7)
|
||||
assert abs((entry.ttl_expiry - expected_ttl).total_seconds()) < 5
|
||||
|
||||
def test_create_log_entry_with_metadata(self):
|
||||
"""Test creating entry with metadata."""
|
||||
metadata = {"file_size": 1024, "duration": 300}
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test",
|
||||
metadata=metadata
|
||||
)
|
||||
|
||||
assert entry.metadata == metadata
|
||||
|
||||
def test_to_dict(self):
|
||||
"""Test converting entry to dict for Firestore."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test message",
|
||||
metadata={"key": "value"}
|
||||
)
|
||||
|
||||
d = entry.to_dict()
|
||||
|
||||
assert d["job_id"] == "job123"
|
||||
assert d["worker"] == "audio"
|
||||
assert d["level"] == "INFO"
|
||||
assert d["message"] == "Test message"
|
||||
assert d["metadata"] == {"key": "value"}
|
||||
assert "id" in d
|
||||
assert "timestamp" in d
|
||||
assert "ttl_expiry" in d
|
||||
|
||||
def test_to_dict_without_metadata(self):
|
||||
"""Test to_dict doesn't include metadata when None."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test"
|
||||
)
|
||||
|
||||
d = entry.to_dict()
|
||||
assert "metadata" not in d
|
||||
|
||||
def test_to_legacy_dict(self):
|
||||
"""Test converting to legacy format for API compatibility."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test message"
|
||||
)
|
||||
|
||||
d = entry.to_legacy_dict()
|
||||
|
||||
assert "timestamp" in d
|
||||
assert d["level"] == "INFO"
|
||||
assert d["worker"] == "audio"
|
||||
assert d["message"] == "Test message"
|
||||
# Should not include id, job_id, ttl_expiry, metadata
|
||||
assert "id" not in d
|
||||
assert "job_id" not in d
|
||||
assert "ttl_expiry" not in d
|
||||
|
||||
def test_from_dict(self):
|
||||
"""Test creating entry from Firestore document."""
|
||||
timestamp = datetime.now(timezone.utc)
|
||||
ttl_expiry = timestamp + timedelta(days=30)
|
||||
|
||||
data = {
|
||||
"id": "log123",
|
||||
"job_id": "job123",
|
||||
"timestamp": timestamp,
|
||||
"level": "WARNING",
|
||||
"worker": "video",
|
||||
"message": "Warning message",
|
||||
"metadata": {"error_code": 500},
|
||||
"ttl_expiry": ttl_expiry
|
||||
}
|
||||
|
||||
entry = WorkerLogEntry.from_dict(data)
|
||||
|
||||
assert entry.id == "log123"
|
||||
assert entry.job_id == "job123"
|
||||
assert entry.level == "WARNING"
|
||||
assert entry.worker == "video"
|
||||
assert entry.message == "Warning message"
|
||||
assert entry.metadata == {"error_code": 500}
|
||||
|
||||
def test_from_dict_with_iso_strings(self):
|
||||
"""Test creating entry from dict with ISO format strings."""
|
||||
data = {
|
||||
"timestamp": "2026-01-04T12:00:00Z",
|
||||
"level": "INFO",
|
||||
"worker": "audio",
|
||||
"message": "Test"
|
||||
}
|
||||
|
||||
entry = WorkerLogEntry.from_dict(data)
|
||||
|
||||
assert entry.timestamp.tzinfo == timezone.utc
|
||||
assert entry.message == "Test"
|
||||
|
||||
def test_from_dict_missing_fields(self):
|
||||
"""Test creating entry with missing fields uses defaults."""
|
||||
data = {
|
||||
"message": "Minimal log"
|
||||
}
|
||||
|
||||
entry = WorkerLogEntry.from_dict(data)
|
||||
|
||||
assert entry.message == "Minimal log"
|
||||
assert entry.level == "INFO" # Default
|
||||
assert entry.worker == "unknown" # Default
|
||||
assert entry.id is not None # Generated
|
||||
|
||||
|
||||
class TestFirestoreServiceSubcollection:
|
||||
"""Tests for FirestoreService log subcollection methods."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_db(self):
|
||||
"""Create mock Firestore database."""
|
||||
return MagicMock()
|
||||
|
||||
@pytest.fixture
|
||||
def firestore_service(self, mock_db):
|
||||
"""Create FirestoreService with mocked DB."""
|
||||
with patch('backend.services.firestore_service.firestore') as mock_firestore:
|
||||
mock_firestore.Client.return_value = mock_db
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
service = FirestoreService()
|
||||
service.db = mock_db
|
||||
return service
|
||||
|
||||
def test_append_log_to_subcollection(self, firestore_service, mock_db):
|
||||
"""Test appending log to subcollection."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test message"
|
||||
)
|
||||
|
||||
# Setup mock chain
|
||||
mock_collection = MagicMock()
|
||||
mock_doc = MagicMock()
|
||||
mock_logs_ref = MagicMock()
|
||||
mock_log_doc = MagicMock()
|
||||
|
||||
mock_db.collection.return_value = mock_collection
|
||||
mock_collection.document.return_value = mock_doc
|
||||
mock_doc.collection.return_value = mock_logs_ref
|
||||
mock_logs_ref.document.return_value = mock_log_doc
|
||||
|
||||
firestore_service.append_log_to_subcollection("job123", entry)
|
||||
|
||||
# Verify correct path: jobs/{job_id}/logs/{log_id}
|
||||
mock_db.collection.assert_called_with("jobs")
|
||||
mock_collection.document.assert_called_with("job123")
|
||||
mock_doc.collection.assert_called_with("logs")
|
||||
mock_logs_ref.document.assert_called_with(entry.id)
|
||||
mock_log_doc.set.assert_called_once()
|
||||
|
||||
def test_get_logs_from_subcollection(self, firestore_service, mock_db):
|
||||
"""Test getting logs from subcollection."""
|
||||
# Setup mock chain
|
||||
mock_collection = MagicMock()
|
||||
mock_doc = MagicMock()
|
||||
mock_logs_ref = MagicMock()
|
||||
mock_query = MagicMock()
|
||||
|
||||
mock_db.collection.return_value = mock_collection
|
||||
mock_collection.document.return_value = mock_doc
|
||||
mock_doc.collection.return_value = mock_logs_ref
|
||||
mock_logs_ref.order_by.return_value = mock_query
|
||||
mock_query.limit.return_value = mock_query
|
||||
|
||||
# Mock document data
|
||||
mock_doc1 = MagicMock()
|
||||
mock_doc1.to_dict.return_value = {
|
||||
"timestamp": datetime.now(timezone.utc),
|
||||
"level": "INFO",
|
||||
"worker": "audio",
|
||||
"message": "Log 1"
|
||||
}
|
||||
|
||||
mock_query.stream.return_value = iter([mock_doc1])
|
||||
|
||||
logs = firestore_service.get_logs_from_subcollection("job123")
|
||||
|
||||
assert len(logs) == 1
|
||||
assert logs[0].message == "Log 1"
|
||||
mock_logs_ref.order_by.assert_called_once()
|
||||
|
||||
def test_get_logs_from_subcollection_with_worker_filter(self, firestore_service, mock_db):
|
||||
"""Test filtering logs by worker."""
|
||||
mock_collection = MagicMock()
|
||||
mock_doc = MagicMock()
|
||||
mock_logs_ref = MagicMock()
|
||||
mock_query = MagicMock()
|
||||
|
||||
mock_db.collection.return_value = mock_collection
|
||||
mock_collection.document.return_value = mock_doc
|
||||
mock_doc.collection.return_value = mock_logs_ref
|
||||
mock_logs_ref.order_by.return_value = mock_query
|
||||
mock_query.where.return_value = mock_query
|
||||
mock_query.limit.return_value = mock_query
|
||||
mock_query.stream.return_value = iter([])
|
||||
|
||||
firestore_service.get_logs_from_subcollection("job123", worker="audio")
|
||||
|
||||
# Verify where clause was added
|
||||
mock_query.where.assert_called()
|
||||
|
||||
def test_delete_logs_subcollection(self, firestore_service, mock_db):
|
||||
"""Test deleting all logs in subcollection."""
|
||||
mock_collection = MagicMock()
|
||||
mock_doc = MagicMock()
|
||||
mock_logs_ref = MagicMock()
|
||||
mock_batch = MagicMock()
|
||||
|
||||
mock_db.collection.return_value = mock_collection
|
||||
mock_collection.document.return_value = mock_doc
|
||||
mock_doc.collection.return_value = mock_logs_ref
|
||||
mock_db.batch.return_value = mock_batch
|
||||
|
||||
# First call returns 2 docs, second call returns empty
|
||||
mock_log_doc1 = MagicMock()
|
||||
mock_log_doc2 = MagicMock()
|
||||
|
||||
call_count = [0]
|
||||
def stream_side_effect(*args, **kwargs):
|
||||
call_count[0] += 1
|
||||
if call_count[0] == 1:
|
||||
return iter([mock_log_doc1, mock_log_doc2])
|
||||
return iter([])
|
||||
|
||||
mock_logs_ref.limit.return_value = mock_logs_ref
|
||||
mock_logs_ref.stream.side_effect = stream_side_effect
|
||||
|
||||
deleted = firestore_service.delete_logs_subcollection("job123")
|
||||
|
||||
assert deleted == 2
|
||||
mock_batch.commit.assert_called()
|
||||
|
||||
|
||||
class TestJobManagerLogging:
|
||||
"""Tests for JobManager log methods with feature flag."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_firestore_service(self):
|
||||
"""Mock FirestoreService."""
|
||||
with patch('backend.services.job_manager.FirestoreService') as mock:
|
||||
service = MagicMock()
|
||||
mock.return_value = service
|
||||
yield service
|
||||
|
||||
@pytest.fixture
|
||||
def mock_storage_service(self):
|
||||
"""Mock StorageService."""
|
||||
with patch('backend.services.job_manager.StorageService') as mock:
|
||||
service = MagicMock()
|
||||
mock.return_value = service
|
||||
yield service
|
||||
|
||||
def test_append_worker_log_uses_subcollection_when_enabled(
|
||||
self, mock_firestore_service, mock_storage_service
|
||||
):
|
||||
"""Test that logs go to subcollection when feature is enabled."""
|
||||
with patch('backend.services.job_manager.settings') as mock_settings:
|
||||
mock_settings.use_log_subcollection = True
|
||||
|
||||
from backend.services.job_manager import JobManager
|
||||
manager = JobManager()
|
||||
|
||||
manager.append_worker_log(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test message"
|
||||
)
|
||||
|
||||
# Should call subcollection method
|
||||
mock_firestore_service.append_log_to_subcollection.assert_called_once()
|
||||
# Should NOT call legacy method
|
||||
mock_firestore_service.append_worker_log.assert_not_called()
|
||||
|
||||
def test_append_worker_log_uses_array_when_disabled(
|
||||
self, mock_firestore_service, mock_storage_service
|
||||
):
|
||||
"""Test that logs go to embedded array when feature is disabled."""
|
||||
with patch('backend.services.job_manager.settings') as mock_settings:
|
||||
mock_settings.use_log_subcollection = False
|
||||
|
||||
from backend.services.job_manager import JobManager
|
||||
manager = JobManager()
|
||||
|
||||
manager.append_worker_log(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test message"
|
||||
)
|
||||
|
||||
# Should call legacy method
|
||||
mock_firestore_service.append_worker_log.assert_called_once()
|
||||
# Should NOT call subcollection method
|
||||
mock_firestore_service.append_log_to_subcollection.assert_not_called()
|
||||
|
||||
def test_get_worker_logs_from_subcollection(
|
||||
self, mock_firestore_service, mock_storage_service
|
||||
):
|
||||
"""Test getting logs from subcollection."""
|
||||
with patch('backend.services.job_manager.settings') as mock_settings:
|
||||
mock_settings.use_log_subcollection = True
|
||||
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Test"
|
||||
)
|
||||
mock_firestore_service.get_logs_from_subcollection.return_value = [entry]
|
||||
|
||||
from backend.services.job_manager import JobManager
|
||||
manager = JobManager()
|
||||
|
||||
logs = manager.get_worker_logs("job123")
|
||||
|
||||
assert len(logs) == 1
|
||||
assert logs[0]["message"] == "Test"
|
||||
mock_firestore_service.get_logs_from_subcollection.assert_called_once()
|
||||
|
||||
def test_get_worker_logs_falls_back_to_array(
|
||||
self, mock_firestore_service, mock_storage_service
|
||||
):
|
||||
"""Test fallback to embedded array for legacy jobs."""
|
||||
with patch('backend.services.job_manager.settings') as mock_settings:
|
||||
mock_settings.use_log_subcollection = True
|
||||
|
||||
# Subcollection returns empty
|
||||
mock_firestore_service.get_logs_from_subcollection.return_value = []
|
||||
|
||||
# Mock job with legacy logs
|
||||
from backend.models.job import Job, JobStatus
|
||||
mock_job = Job(
|
||||
job_id="job123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
worker_logs=[
|
||||
{"timestamp": "2026-01-04T12:00:00Z", "level": "INFO", "worker": "audio", "message": "Legacy log"}
|
||||
]
|
||||
)
|
||||
mock_firestore_service.get_job.return_value = mock_job
|
||||
|
||||
from backend.services.job_manager import JobManager
|
||||
manager = JobManager()
|
||||
|
||||
logs = manager.get_worker_logs("job123")
|
||||
|
||||
assert len(logs) == 1
|
||||
assert logs[0]["message"] == "Legacy log"
|
||||
|
||||
def test_delete_job_deletes_logs_subcollection(
|
||||
self, mock_firestore_service, mock_storage_service
|
||||
):
|
||||
"""Test that deleting a job also deletes its logs subcollection."""
|
||||
mock_firestore_service.get_job.return_value = None
|
||||
mock_firestore_service.delete_logs_subcollection.return_value = 5
|
||||
|
||||
from backend.services.job_manager import JobManager
|
||||
manager = JobManager()
|
||||
|
||||
manager.delete_job("job123", delete_files=False)
|
||||
|
||||
mock_firestore_service.delete_logs_subcollection.assert_called_once_with("job123")
|
||||
mock_firestore_service.delete_job.assert_called_once_with("job123")
|
||||
|
||||
|
||||
class TestWorkerLogEntryEdgeCases:
|
||||
"""Edge case tests for WorkerLogEntry."""
|
||||
|
||||
def test_create_with_empty_message(self):
|
||||
"""Test creating entry with empty message."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message=""
|
||||
)
|
||||
assert entry.message == ""
|
||||
|
||||
def test_create_with_unicode_message(self):
|
||||
"""Test creating entry with Unicode characters."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="INFO",
|
||||
message="Processing song: 日本語の曲 - アーティスト"
|
||||
)
|
||||
assert "日本語" in entry.message
|
||||
|
||||
def test_create_with_newlines_in_message(self):
|
||||
"""Test creating entry with newlines."""
|
||||
entry = WorkerLogEntry.create(
|
||||
job_id="job123",
|
||||
worker="audio",
|
||||
level="ERROR",
|
||||
message="Error occurred:\nLine 1\nLine 2"
|
||||
)
|
||||
assert "\n" in entry.message
|
||||
|
||||
def test_from_dict_handles_missing_ttl_expiry(self):
|
||||
"""Test from_dict creates default TTL when missing."""
|
||||
data = {
|
||||
"timestamp": datetime.now(timezone.utc),
|
||||
"level": "INFO",
|
||||
"worker": "audio",
|
||||
"message": "Test"
|
||||
}
|
||||
|
||||
entry = WorkerLogEntry.from_dict(data)
|
||||
|
||||
assert entry.ttl_expiry is not None
|
||||
# Should be ~30 days from now
|
||||
expected = datetime.now(timezone.utc) + timedelta(days=30)
|
||||
assert abs((entry.ttl_expiry - expected).total_seconds()) < 10
|
||||
@@ -639,40 +639,45 @@ async def upload_lyrics_results(
|
||||
logger.error(f"Job {job_id}: {error_msg}")
|
||||
raise Exception(error_msg)
|
||||
|
||||
# Upload reference lyrics if available
|
||||
# Upload ALL reference lyrics files (not just first) so they're available for distribution
|
||||
# Note: Source names use .title() so "lrclib" -> "Lrclib", "genius" -> "Genius"
|
||||
# Use sanitized artist/title to match LyricsProcessor's file naming
|
||||
reference_files = [
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Genius).txt", "Genius"),
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Spotify).txt", "Spotify"),
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Musixmatch).txt", "Musixmatch"),
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Lrclib).txt", "LRCLib"), # Note: source is 'lrclib', .title() = 'Lrclib'
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Genius).txt", "genius"),
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Spotify).txt", "spotify"),
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Musixmatch).txt", "musixmatch"),
|
||||
(f"{safe_artist} - {safe_title} (Lyrics Lrclib).txt", "lrclib"),
|
||||
]
|
||||
|
||||
found_reference = False
|
||||
for ref_filename, source_name in reference_files:
|
||||
|
||||
found_references = []
|
||||
for ref_filename, source_key in reference_files:
|
||||
ref_path = os.path.join(lyrics_dir, ref_filename)
|
||||
if os.path.exists(ref_path):
|
||||
# Upload with original filename to preserve proper naming
|
||||
gcs_path = f"jobs/{job_id}/lyrics/{ref_filename}"
|
||||
url = storage.upload_file(ref_path, gcs_path)
|
||||
# Track in job.files so video_worker can download for distribution
|
||||
job_manager.update_file_url(job_id, 'lyrics', f'reference_{source_key}', url)
|
||||
if job_log:
|
||||
job_log.info(f"Found reference lyrics from {source_name}")
|
||||
job_log.info(f"Found reference lyrics from {source_key}")
|
||||
logger.info(f"Job {job_id}: Uploaded reference lyrics: {ref_filename}")
|
||||
found_reference = True
|
||||
break # Only upload first available reference
|
||||
|
||||
if not found_reference:
|
||||
found_references.append(source_key)
|
||||
|
||||
if not found_references:
|
||||
if job_log:
|
||||
job_log.warning("No reference lyrics found from any source (Genius, Spotify, Musixmatch, LRCLib)")
|
||||
logger.warning(f"Job {job_id}: No reference lyrics found from any source")
|
||||
else:
|
||||
logger.info(f"Job {job_id}: Found {len(found_references)} reference lyrics sources: {found_references}")
|
||||
|
||||
# Upload uncorrected transcription if available (use sanitized names)
|
||||
uncorrected_file = os.path.join(lyrics_dir, f"{safe_artist} - {safe_title} (Lyrics Uncorrected).txt")
|
||||
# Upload uncorrected transcription if available (preserve original filename for distribution)
|
||||
uncorrected_filename = f"{safe_artist} - {safe_title} (Lyrics Uncorrected).txt"
|
||||
uncorrected_file = os.path.join(lyrics_dir, uncorrected_filename)
|
||||
if os.path.exists(uncorrected_file):
|
||||
gcs_path = f"jobs/{job_id}/lyrics/uncorrected.txt"
|
||||
gcs_path = f"jobs/{job_id}/lyrics/{uncorrected_filename}"
|
||||
url = storage.upload_file(uncorrected_file, gcs_path)
|
||||
job_manager.update_file_url(job_id, 'lyrics', 'uncorrected', url)
|
||||
logger.info(f"Job {job_id}: Uploaded uncorrected transcription")
|
||||
logger.info(f"Job {job_id}: Uploaded uncorrected transcription: {uncorrected_filename}")
|
||||
|
||||
logger.info(f"Job {job_id}: All lyrics results uploaded successfully")
|
||||
|
||||
@@ -160,14 +160,17 @@ async def _encode_via_gce(
|
||||
storage.download_file(gcs_path, local_path)
|
||||
|
||||
# Map to result keys expected by _upload_results
|
||||
if "4K Lossless" in filename or "4k_lossless" in filename.lower():
|
||||
local_files["final_video"] = local_path
|
||||
elif "4K Lossy" in filename or "4k_lossy" in filename.lower():
|
||||
# Files are named like "Artist - Title (Final Karaoke Lossless 4k).mp4"
|
||||
filename_lower = filename.lower()
|
||||
if "lossless 4k" in filename_lower:
|
||||
if filename.endswith(".mkv"):
|
||||
local_files["final_video_mkv"] = local_path
|
||||
else:
|
||||
local_files["final_video"] = local_path
|
||||
elif "lossy 4k" in filename_lower:
|
||||
local_files["final_video_lossy"] = local_path
|
||||
elif "720p" in filename:
|
||||
elif "720p" in filename_lower:
|
||||
local_files["final_video_720p"] = local_path
|
||||
elif filename.endswith(".mkv"):
|
||||
local_files["final_video_mkv"] = local_path
|
||||
|
||||
job_log.info(f"Downloaded {len(local_files)} encoded files")
|
||||
|
||||
@@ -1108,29 +1111,63 @@ async def _prepare_distribution_directory(
|
||||
log_progress(f"Downloaded {stem_key} to stems/")
|
||||
|
||||
# --- Download lyrics files to lyrics/ subfolder ---
|
||||
# Use proper filenames matching local CLI output structure
|
||||
|
||||
# Common lyrics file types
|
||||
lyrics_file_keys = ['lrc', 'ass', 'srt', 'txt', 'json', 'original_json', 'corrected_json']
|
||||
for lyrics_key in lyrics_file_keys:
|
||||
# Map job.files.lyrics keys to proper local filenames
|
||||
# Format: (job_files_key, local_filename_pattern)
|
||||
lyrics_file_mappings = [
|
||||
# Karaoke output files
|
||||
('lrc', f"{base_name} (Karaoke).lrc"),
|
||||
('ass', f"{base_name} (Karaoke).ass"),
|
||||
# Corrections and transcription files
|
||||
('corrections', f"{base_name} (Lyrics Corrections).json"),
|
||||
('corrected_txt', f"{base_name} (Lyrics Corrected).txt"),
|
||||
('uncorrected', f"{base_name} (Lyrics Uncorrected).txt"),
|
||||
# Reference lyrics from various sources
|
||||
('reference_genius', f"{base_name} (Lyrics Genius).txt"),
|
||||
('reference_spotify', f"{base_name} (Lyrics Spotify).txt"),
|
||||
('reference_musixmatch', f"{base_name} (Lyrics Musixmatch).txt"),
|
||||
('reference_lrclib', f"{base_name} (Lyrics Lrclib).txt"),
|
||||
]
|
||||
|
||||
for lyrics_key, local_filename in lyrics_file_mappings:
|
||||
if lyrics.get(lyrics_key):
|
||||
# Determine proper extension
|
||||
ext_map = {
|
||||
'lrc': '.lrc',
|
||||
'ass': '.ass',
|
||||
'srt': '.srt',
|
||||
'txt': '.txt',
|
||||
'json': '.json',
|
||||
'original_json': '_original.json',
|
||||
'corrected_json': '_corrected.json',
|
||||
}
|
||||
ext = ext_map.get(lyrics_key, f'.{lyrics_key}')
|
||||
dest_path = os.path.join(lyrics_dir, f"{base_name} (Karaoke){ext}")
|
||||
dest_path = os.path.join(lyrics_dir, local_filename)
|
||||
try:
|
||||
storage.download_file(lyrics[lyrics_key], dest_path)
|
||||
log_progress(f"Downloaded {lyrics_key} to lyrics/")
|
||||
log_progress(f"Downloaded {lyrics_key} to lyrics/{local_filename}")
|
||||
except Exception as e:
|
||||
log_progress(f"Could not download {lyrics_key}: {e}")
|
||||
|
||||
# --- Download (With Vocals).mkv to lyrics/ subfolder ---
|
||||
# Local CLI places the karaoke video with vocals in the lyrics folder
|
||||
videos = job.file_urls.get('videos', {})
|
||||
if videos.get('with_vocals'):
|
||||
with_vocals_dest = os.path.join(lyrics_dir, f"{base_name} (With Vocals).mkv")
|
||||
try:
|
||||
storage.download_file(videos['with_vocals'], with_vocals_dest)
|
||||
log_progress(f"Downloaded with_vocals to lyrics/{base_name} (With Vocals).mkv")
|
||||
except Exception as e:
|
||||
log_progress(f"Could not download with_vocals to lyrics/: {e}")
|
||||
|
||||
# --- Download preview ASS files to lyrics/previews/ subfolder ---
|
||||
# Preview files are stored in jobs/{job_id}/previews/ in GCS
|
||||
previews_dir = os.path.join(lyrics_dir, "previews")
|
||||
os.makedirs(previews_dir, exist_ok=True)
|
||||
|
||||
# List all preview files from GCS and download them
|
||||
try:
|
||||
preview_prefix = f"jobs/{job_id}/previews/"
|
||||
preview_files = storage.list_files(preview_prefix)
|
||||
for blob_name in preview_files:
|
||||
if blob_name.endswith('.ass'):
|
||||
filename = os.path.basename(blob_name)
|
||||
dest_path = os.path.join(previews_dir, filename)
|
||||
storage.download_file(blob_name, dest_path)
|
||||
log_progress(f"Downloaded preview {filename} to lyrics/previews/")
|
||||
except Exception as e:
|
||||
log_progress(f"Could not download preview files: {e}")
|
||||
|
||||
log_progress("Distribution directory prepared with stems/ and lyrics/ subfolders")
|
||||
|
||||
|
||||
|
||||
@@ -347,6 +347,7 @@ class VideoWorkerOrchestrator:
|
||||
brand_code=self.config.keep_brand_code,
|
||||
output_dir=self.config.output_dir,
|
||||
options={
|
||||
"job_id": self.config.job_id,
|
||||
"input_gcs_path": input_gcs_path,
|
||||
"output_gcs_path": output_gcs_path,
|
||||
},
|
||||
@@ -366,15 +367,70 @@ class VideoWorkerOrchestrator:
|
||||
if not output.success:
|
||||
raise Exception(f"Encoding failed: {output.error_message}")
|
||||
|
||||
# Store results
|
||||
# Store results - for GCE backend, these are GCS blob paths that need to be downloaded
|
||||
self.result.final_video = output.lossless_4k_mp4_path
|
||||
self.result.final_video_mkv = output.lossless_mkv_path
|
||||
self.result.final_video_lossy = output.lossy_4k_mp4_path
|
||||
self.result.final_video_720p = output.lossy_720p_mp4_path
|
||||
self.result.encoding_time_seconds = output.encoding_time_seconds
|
||||
|
||||
# For GCE encoding, download the encoded files from GCS to local directory
|
||||
# This is required for YouTube upload and other local file operations
|
||||
if encoding_backend.name == "gce" and self.storage:
|
||||
await self._download_gce_encoded_files(output)
|
||||
|
||||
self.job_log.info(f"Encoding complete ({encoding_backend.name}) in {output.encoding_time_seconds:.1f}s")
|
||||
|
||||
async def _download_gce_encoded_files(self, output):
|
||||
"""
|
||||
Download GCE-encoded files from GCS to the local output directory.
|
||||
|
||||
GCE encoding stores files in GCS and returns blob paths like:
|
||||
'jobs/{job_id}/finals/Artist - Title (Final Karaoke Lossless 4k).mp4'
|
||||
|
||||
This method downloads those files locally so that subsequent stages
|
||||
(YouTube upload, etc.) can access them as local files.
|
||||
|
||||
Args:
|
||||
output: EncodingOutput from the GCE backend with GCS blob paths
|
||||
"""
|
||||
self.job_log.info("Downloading GCE-encoded files from GCS")
|
||||
|
||||
# Map of result attributes to download
|
||||
file_mappings = [
|
||||
('lossless_4k_mp4_path', 'final_video'),
|
||||
('lossless_mkv_path', 'final_video_mkv'),
|
||||
('lossy_4k_mp4_path', 'final_video_lossy'),
|
||||
('lossy_720p_mp4_path', 'final_video_720p'),
|
||||
]
|
||||
|
||||
downloaded_count = 0
|
||||
for output_attr, result_attr in file_mappings:
|
||||
gcs_path = getattr(output, output_attr, None)
|
||||
if not gcs_path:
|
||||
continue
|
||||
|
||||
# Extract filename from GCS path
|
||||
filename = os.path.basename(gcs_path)
|
||||
local_path = os.path.join(self.config.output_dir, filename)
|
||||
|
||||
try:
|
||||
self.job_log.info(f"Downloading {filename} from GCS")
|
||||
self.storage.download_file(gcs_path, local_path)
|
||||
|
||||
# Update the result to point to local file
|
||||
setattr(self.result, result_attr, local_path)
|
||||
downloaded_count += 1
|
||||
self.job_log.info(f"Downloaded {filename} to {local_path}")
|
||||
|
||||
except Exception as e:
|
||||
self.job_log.error(f"Failed to download {filename}: {e}")
|
||||
# Clear the result attribute so downstream doesn't try to use invalid GCS path
|
||||
setattr(self.result, result_attr, None)
|
||||
# Don't fail - some formats might not be generated
|
||||
|
||||
self.job_log.info(f"Downloaded {downloaded_count} encoded files from GCS")
|
||||
|
||||
async def _run_organization(self):
|
||||
"""Run the organization stage (brand code generation)."""
|
||||
self.job_log.info("Starting organization stage")
|
||||
|
||||
@@ -31,7 +31,8 @@
|
||||
└──────────┘ └──────────┘ └──────────────┘ └──────────────┘
|
||||
|
||||
(*) GCE Encoding Worker: C4-standard-8 VM with Intel Granite Rapids for
|
||||
high-performance FFmpeg encoding. Optional - falls back to Cloud Run.
|
||||
high-performance FFmpeg encoding. Used for both final video encoding and
|
||||
preview video generation. Uses LocalEncodingService via GCS wheel deployment.
|
||||
```
|
||||
|
||||
## Processing Pipeline
|
||||
@@ -57,7 +58,7 @@
|
||||
▼
|
||||
4. HUMAN REVIEW (AWAITING_REVIEW)
|
||||
• React UI for lyrics correction
|
||||
• Preview video generation
|
||||
• Preview video generation (optionally via GCE for speed)
|
||||
• Submit corrections
|
||||
▼
|
||||
5. VIDEO RENDERING
|
||||
@@ -163,12 +164,15 @@ karaoke-gen shares a GCP project (`nomadkaraoke`) with karaoke-decide, but uses
|
||||
|------------|---------|------------|
|
||||
| `gen_users` | karaoke-gen user accounts | email, credits, role, is_active |
|
||||
| `jobs` | Karaoke generation jobs | job_id, user_email, status, state_data |
|
||||
| `jobs/{job_id}/logs` | Worker log entries (subcollection) | timestamp, level, worker, message, ttl_expiry |
|
||||
| `sessions` | Magic link auth sessions | user_email, token, expires_at |
|
||||
| `magic_links` | Passwordless auth tokens | email, token, expires_at, used |
|
||||
| `beta_feedback` | Beta program feedback | user_email, ratings, comments |
|
||||
|
||||
**Note**: The `users` collection in the same Firestore instance belongs to karaoke-decide (different schema: user_id, is_guest, quiz_* fields). Don't use it for karaoke-gen.
|
||||
|
||||
**Worker Logs**: Stored in subcollection (`jobs/{job_id}/logs`) instead of embedded array to avoid Firestore's 1MB document size limit. TTL policy auto-deletes logs after 30 days. Feature flag: `USE_LOG_SUBCOLLECTION` (default: true). See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#firestore-document-1mb-limit-with-embedded-arrays).
|
||||
|
||||
## Video Worker Orchestrator
|
||||
|
||||
The Video Worker uses an orchestrator pattern to ensure all features work regardless of encoding backend (local Cloud Run or GCE VM).
|
||||
|
||||
+25
-5
@@ -110,17 +110,37 @@ Push to `main` triggers GitHub Actions:
|
||||
3. Deploy to Cloud Run
|
||||
4. Deploy frontend to Cloudflare Pages
|
||||
|
||||
### Manual Backend Deploy
|
||||
### Infrastructure Changes (Pulumi)
|
||||
|
||||
**All infrastructure changes must go through PRs.** The CI pipeline automatically deploys Pulumi changes when PRs merge to main.
|
||||
|
||||
```bash
|
||||
# Preview infrastructure changes
|
||||
# Preview changes locally (requires GCP auth)
|
||||
cd infrastructure
|
||||
pulumi preview
|
||||
pulumi preview --stack nomadkaraoke/karaoke-gen-infrastructure/prod
|
||||
|
||||
# Apply changes
|
||||
pulumi up
|
||||
# DO NOT run `pulumi up` locally - let CI handle deployment
|
||||
# CI uses --skip-preview due to limited service account permissions
|
||||
```
|
||||
|
||||
**What's managed by Pulumi:**
|
||||
- Firestore database and indexes
|
||||
- GCS buckets
|
||||
- Cloud Run service configuration
|
||||
- Cloud Tasks queues
|
||||
- Secret Manager secrets
|
||||
- Service accounts and IAM bindings
|
||||
- Workload Identity Federation
|
||||
- GCE instances (GitHub runners, encoding worker)
|
||||
|
||||
**Workflow:**
|
||||
1. Make changes in `infrastructure/__main__.py`
|
||||
2. Run `pulumi preview` locally to verify
|
||||
3. Create PR and merge
|
||||
4. CI deploys automatically
|
||||
|
||||
See `docs/LESSONS-LEARNED.md` for Pulumi CI gotchas.
|
||||
|
||||
### Manual Frontend Deploy
|
||||
|
||||
```bash
|
||||
|
||||
@@ -222,6 +222,52 @@ escaped = filename.replace("'", "\\'")
|
||||
query = f"name='{escaped}' and '{parent_id}' in parents"
|
||||
```
|
||||
|
||||
### FFmpeg Filter Path Escaping (subprocess without shell)
|
||||
|
||||
**Problem**: Song titles with apostrophes (e.g., "I'm With You") cause FFmpeg's ASS subtitle filter to fail because paths aren't escaped correctly.
|
||||
|
||||
```python
|
||||
# BAD - FFmpeg strips the apostrophe, looks for wrong path
|
||||
ass_filter = f"ass=./Avril Lavigne - I'm With You/cache/temp.ass"
|
||||
# FFmpeg error: Could not create a libass track when reading file './Avril Lavigne - Im With You/cache/temp.ass'
|
||||
```
|
||||
|
||||
**Wrong approach #1**: Simple backslash escaping like `\'` doesn't work because FFmpeg's filter parser requires more escaping.
|
||||
|
||||
**Wrong approach #2**: Shell-style escaping `'\''` (end quote, escaped quote, start quote) only works when passing through a shell. When using Python `subprocess` with a command list, there's no shell - arguments go directly to FFmpeg, so shell escaping patterns don't work.
|
||||
|
||||
**Solution**: Use FFmpeg filter escaping (for subprocess without shell):
|
||||
```python
|
||||
def _escape_ffmpeg_filter_path(path: str) -> str:
|
||||
"""Escape a path for FFmpeg filter expressions (for subprocess without shell).
|
||||
|
||||
FFmpeg filter escaping rules:
|
||||
- Backslashes: double them (\ -> \\)
|
||||
- Single quotes/apostrophes: escape with three backslashes (' -> \\\')
|
||||
- Spaces: escape with backslash ( -> \ )
|
||||
"""
|
||||
# First escape existing backslashes (\ -> \\)
|
||||
escaped = path.replace("\\", "\\\\")
|
||||
# Escape single quotes (' -> \\\')
|
||||
# In Python source: 6 backslashes + apostrophe = 3 backslashes + apostrophe in string
|
||||
escaped = escaped.replace("'", "\\\\\\'")
|
||||
# Escape spaces
|
||||
escaped = escaped.replace(" ", "\\ ")
|
||||
return escaped
|
||||
|
||||
# Result: "I'm With You" becomes "I\\\'m\ With\ You"
|
||||
# FFmpeg correctly interprets this as: I'm With You
|
||||
```
|
||||
|
||||
**Symptoms**:
|
||||
- Video generation fails with "Could not create a libass track"
|
||||
- Error path is subtly different from the path in your code (missing apostrophe)
|
||||
- Works fine for songs without special characters
|
||||
|
||||
**Key insight**: FFmpeg filter escaping is different from shell escaping. When using `subprocess.run(cmd_list)` (no `shell=True`), there's no shell to interpret the arguments. FFmpeg receives the strings directly and applies its own filter expression parsing rules. The `'\''` shell idiom does NOT work in this context.
|
||||
|
||||
**Lesson**: When debugging FFmpeg filter path issues, check whether you're using shell or subprocess. For subprocess without shell, escape spaces with `\ ` and apostrophes with `\\\'` (3 backslashes + quote in the actual string).
|
||||
|
||||
### Duplicate Code Leads to Inconsistent Behavior
|
||||
|
||||
**Problem**: Multiple endpoints computing the same defaults independently. `audio_search.py` had `effective_brand_prefix = body.brand_prefix or settings.default_brand_prefix` but `file_upload.py` was missing it for some endpoints.
|
||||
@@ -247,6 +293,43 @@ def _get_effective_distribution_settings(...) -> EffectiveDistributionSettings:
|
||||
|
||||
**Lesson**: When you see the same logic repeated in multiple places, refactor into a helper function immediately. Otherwise bugs will be fixed in one place but not others.
|
||||
|
||||
### Python `or` Operator Can't Override True Defaults with False
|
||||
|
||||
**Problem**: Using Python's `or` operator for default logic prevents explicit `false` from overriding a `true` server default.
|
||||
|
||||
```python
|
||||
# BAD - can't explicitly disable if server default is true
|
||||
effective_value = body.enable_feature or settings.default_enable_feature
|
||||
# When body.enable_feature=False and settings.default=True:
|
||||
# False or True = True (ignores explicit False!)
|
||||
```
|
||||
|
||||
**Symptoms**:
|
||||
- Feature can be enabled but never disabled via API
|
||||
- Works fine when server default is `false` but breaks when default is `true`
|
||||
- No errors - just ignores the explicit `false` value
|
||||
|
||||
**Solution**: Use `Optional[bool] = None` with explicit `is not None` check:
|
||||
|
||||
```python
|
||||
# Request model
|
||||
enable_feature: Optional[bool] = None # None = use server default
|
||||
|
||||
# Default logic
|
||||
effective_value = (
|
||||
body.enable_feature
|
||||
if body.enable_feature is not None
|
||||
else settings.default_enable_feature
|
||||
)
|
||||
```
|
||||
|
||||
This allows three states:
|
||||
- `None` (not specified) → use server default
|
||||
- `True` → explicitly enable
|
||||
- `False` → explicitly disable (overrides server default)
|
||||
|
||||
**Lesson**: When a boolean field needs to support "use server default" behavior, use `Optional[bool] = None` instead of `bool = False`. The `or` operator conflates "not specified" with "explicitly false".
|
||||
|
||||
### Field Name Mismatches Between Endpoints
|
||||
|
||||
**Problem**: Different endpoints setting different field names for the same logical value, where consumers check only one field name.
|
||||
@@ -588,6 +671,43 @@ GitHub-hosted runners have ~14GB disk, which caused failures during Docker build
|
||||
|
||||
See `docs/archive/2025-12-28-self-hosted-github-runner.md` for full details.
|
||||
|
||||
### Pulumi CI Deployments Need Skip-Preview for Limited Service Accounts
|
||||
|
||||
**Problem**: Running `pulumi up --refresh` in CI requires the service account to have read permissions on ALL managed GCP resources. Pulumi refreshes state by querying each resource, and permission errors cause the deployment to fail before any changes are applied.
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
Error when reading or editing Resource "project" with IAM Member: Role "roles/...":
|
||||
Error retrieving IAM policy for project "...": googleapi: Error 403: The caller does not have permission
|
||||
```
|
||||
|
||||
**Root cause**: `--refresh` (default behavior) makes read API calls to verify current state matches Pulumi's state. This requires permissions like:
|
||||
- `resourcemanager.projects.getIamPolicy` (for IAM bindings)
|
||||
- `secretmanager.secrets.get` (for secrets)
|
||||
- `cloudtasks.queues.get` (for Cloud Tasks)
|
||||
- `iam.workloadIdentityPools.get` (for Workload Identity)
|
||||
- And many more for each resource type managed
|
||||
|
||||
**Solution**: Use `--skip-preview` for CI deployments where Pulumi state is authoritative:
|
||||
|
||||
```yaml
|
||||
# In CI workflow
|
||||
pulumi up --yes --skip-preview --stack org/project/stack
|
||||
```
|
||||
|
||||
**Why this is safe**:
|
||||
- Infrastructure is only modified through Pulumi (single source of truth)
|
||||
- Preview/refresh is still useful during local development with broader permissions
|
||||
- Reduces permissions surface for CI service account (principle of least privilege)
|
||||
|
||||
**Alternative** (if refresh is required): Grant the service account `roles/editor` or individual read roles for each resource type. This is less secure but allows drift detection.
|
||||
|
||||
**Lesson**: When setting up Pulumi in CI, either:
|
||||
1. Grant broad read permissions and keep `--refresh` (detects drift)
|
||||
2. Use minimal permissions and `--skip-preview` (trusts Pulumi state)
|
||||
|
||||
Choose based on your threat model. For most projects, trusting Pulumi state is acceptable since manual console changes are rare.
|
||||
|
||||
### Docker Disk Space Management on Self-Hosted Runners
|
||||
|
||||
**Problem**: Even with 200GB SSDs, self-hosted runners can fill up with Docker images. Each CI build creates new dangling images (~15GB each), and without aggressive cleanup, disk fills in days.
|
||||
@@ -667,6 +787,61 @@ RUN apt-get update && apt-get install -y \
|
||||
|
||||
**Lesson**: When building Docker images for video/subtitle rendering, install fonts explicitly. The default slim images have nothing, and font fallback behavior differs from local development machines.
|
||||
|
||||
### Reuse LLM Model Instances Across Operations
|
||||
|
||||
**Problem**: Creating a new LLM model instance for each operation in a loop causes massive overhead. Each instance creation triggers model initialization, connection establishment, and optional "warm-up" calls - adding 2+ seconds per operation.
|
||||
|
||||
**Example**: Agentic lyrics correction was creating a new `AgenticCorrector` for each gap (lyric segment needing correction):
|
||||
|
||||
```python
|
||||
# BAD - 31 gaps × 2s initialization = 62s overhead
|
||||
for gap in gaps_to_correct:
|
||||
agent = AgenticCorrector.from_model(model_id, ...) # New instance every time!
|
||||
agent.correct_gap(gap)
|
||||
```
|
||||
|
||||
**Symptoms**:
|
||||
- Operations that should take ~1 minute take 5+ minutes
|
||||
- Logs show "Initializing model..." or "Warming up..." repeatedly
|
||||
- Each loop iteration has a consistent ~2s overhead before actual work
|
||||
- Total time scales linearly with loop count, not with actual work
|
||||
|
||||
**Solution**: Create the model instance ONCE before the loop, then reuse:
|
||||
|
||||
```python
|
||||
# GOOD - 1 initialization, reused for all gaps
|
||||
agent = AgenticCorrector.from_model(model_id, ...) # Create once
|
||||
for gap in gaps_to_correct:
|
||||
agent.correct_gap(gap) # Reuse existing instance
|
||||
```
|
||||
|
||||
**Additional optimization**: If operations are independent, process them in parallel:
|
||||
|
||||
```python
|
||||
# BETTER - parallel processing with shared model
|
||||
agent = AgenticCorrector.from_model(model_id, ...)
|
||||
with ThreadPoolExecutor(max_workers=5) as executor:
|
||||
futures = {executor.submit(agent.correct_gap, gap): gap for gap in gaps}
|
||||
for future in as_completed(futures):
|
||||
result = future.result()
|
||||
```
|
||||
|
||||
**Results**:
|
||||
- Before: 31 gaps → ~5 minutes (2s init + 3s work per gap)
|
||||
- After: 20 gaps → ~55 seconds (1 init + parallel processing)
|
||||
- Speedup: ~5-6x
|
||||
|
||||
**Why warm-up is often unnecessary**: The warm-up pattern (sending a trivial request to "prime" the model) made sense for cold-start scenarios in older systems. Modern LLM APIs:
|
||||
- Don't have cold-start penalties per request
|
||||
- Handle connection pooling internally
|
||||
- May even rate-limit warm-up calls
|
||||
|
||||
If you're already making real requests, warm-up adds latency without benefit.
|
||||
|
||||
**Key insight**: LLM client instances are designed for reuse. They maintain connection pools, authentication state, and caching. Creating a new instance per request throws away these benefits.
|
||||
|
||||
**Configuration**: Added `AGENTIC_MAX_PARALLEL_GAPS` env var (default: 5) to control parallelism. Higher values increase throughput but may hit rate limits.
|
||||
|
||||
## Performance Observations
|
||||
|
||||
| Operation | Duration |
|
||||
@@ -755,6 +930,48 @@ export function getToken() {
|
||||
|
||||
**Lesson**: For values that can change (auth tokens, user preferences), read fresh from storage on each access rather than caching in module scope.
|
||||
|
||||
### Zustand Module-Level Initialization Doesn't Re-run on Client Navigation
|
||||
|
||||
**Problem**: Module-level initialization code in zustand stores only runs once when the module is first imported. Client-side navigation (e.g., `router.push()`) doesn't reload modules, so initialization code doesn't re-run.
|
||||
|
||||
**Example bug**: Auth module had initialization that fetched user data if a token existed:
|
||||
```typescript
|
||||
// auth.ts - module level (runs once at import)
|
||||
if (typeof window !== 'undefined') {
|
||||
const token = getAccessToken()
|
||||
if (token) {
|
||||
useAuth.getState().fetchUser() // Only runs on initial page load!
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
After beta enrollment:
|
||||
1. User lands on `/` (landing page) - auth.ts imported, no token yet, `fetchUser()` not called
|
||||
2. User completes enrollment, token stored via `setAccessToken()`
|
||||
3. `router.push('/app')` - client-side navigation, modules already loaded
|
||||
4. Auth module initialization doesn't re-run - `user` stays `null`
|
||||
5. UI shows "Login" button instead of user credits
|
||||
|
||||
**Symptoms**:
|
||||
- Feature works after page reload but not after client-side navigation
|
||||
- Zustand store has correct persisted data (token) but not computed data (user profile)
|
||||
- E2E tests that use reload pass; tests that use `router.push` fail
|
||||
|
||||
**Solution**: Components that need fresh data should trigger fetches in useEffect:
|
||||
```typescript
|
||||
// In /app page component
|
||||
const { user, fetchUser } = useAuth()
|
||||
|
||||
useEffect(() => {
|
||||
const token = getAccessToken()
|
||||
if (token && !user) {
|
||||
fetchUser() // Ensure user data is loaded regardless of how we got here
|
||||
}
|
||||
}, [user, fetchUser])
|
||||
```
|
||||
|
||||
**Lesson**: Don't rely on module-level initialization for data that might need to be fetched after client-side navigation. Use useEffect in components to ensure data is loaded when needed.
|
||||
|
||||
### GCP Organization Policies Block Service Account Keys
|
||||
|
||||
**Problem**: GCP organization policies can block service account key creation even when you have Owner access to the project.
|
||||
@@ -1038,6 +1255,291 @@ result = await orchestrator.run()
|
||||
|
||||
See `docs/archive/2026-01-04-video-worker-orchestrator-refactor.md` for full details.
|
||||
|
||||
### Firestore Document 1MB Limit with Embedded Arrays
|
||||
|
||||
**Problem**: Firestore documents have a 1MB size limit. Storing logs as an embedded array (`worker_logs: [{...}, {...}, ...]`) within job documents caused job failures when logs exceeded the limit during long-running operations like video encoding.
|
||||
|
||||
**Example**: Job `501258e1` failed when logs reached 1.26MB (2,547 log entries), representing 98.6% of the maximum document size. The job update failed with:
|
||||
```
|
||||
Document cannot be written because its size exceeds the maximum allowed size
|
||||
```
|
||||
|
||||
**Root cause**: Video encoding jobs generate thousands of log entries (FFmpeg output, progress updates). Each log entry is ~500 bytes. 2,000+ entries easily exceed 1MB.
|
||||
|
||||
**Solution**: Store logs in a Firestore subcollection instead of embedded array:
|
||||
|
||||
```python
|
||||
# BAD - embedded array hits 1MB limit
|
||||
doc_ref.update({
|
||||
"worker_logs": firestore.ArrayUnion([{"message": "..."}])
|
||||
})
|
||||
|
||||
# GOOD - subcollection has no size limit per parent doc
|
||||
logs_ref = db.collection("jobs").document(job_id).collection("logs")
|
||||
logs_ref.document(log_id).set({
|
||||
"timestamp": now,
|
||||
"level": "INFO",
|
||||
"worker": "video",
|
||||
"message": "...",
|
||||
"ttl_expiry": now + timedelta(days=30)
|
||||
})
|
||||
```
|
||||
|
||||
**Key benefits of subcollection approach**:
|
||||
1. Each log entry is its own document (no size limit concerns)
|
||||
2. Natural association with parent job (path: `jobs/{job_id}/logs/{log_id}`)
|
||||
3. Can add TTL via Firestore TTL policies for automatic cleanup
|
||||
4. Efficient queries with composite indexes (worker + timestamp)
|
||||
5. Concurrent workers can write without race conditions (unlike ArrayUnion which can conflict)
|
||||
|
||||
**Migration strategy**: Used feature flag (`USE_LOG_SUBCOLLECTION=true`) for rollback capability. New jobs use subcollection; old jobs with embedded arrays continue to work via fallback read logic.
|
||||
|
||||
**Lesson**: Never use embedded arrays for unbounded data. If an array can grow indefinitely based on operation duration or user input, use a subcollection instead. The 1MB limit applies to the entire document, including all fields.
|
||||
|
||||
### Splitting Fast Generation from Slow Encoding
|
||||
|
||||
**Problem**: Preview video generation took 60+ seconds on Cloud Run, causing poor UX during lyrics review. The bottleneck was FFmpeg encoding (CPU-bound), not the subtitle generation (fast text processing).
|
||||
|
||||
**Solution**: Split the operation into two phases with optional offloading:
|
||||
|
||||
1. **ASS Generation (Cloud Run)**: Generate ASS subtitle file locally (~2s). This is pure text processing with no CPU-intensive encoding.
|
||||
|
||||
2. **Video Encoding (GCE)**: Upload ASS to GCS and offload FFmpeg encoding to the high-performance GCE worker (~15-20s on C4-standard-8).
|
||||
|
||||
```python
|
||||
# Generate ASS only (fast, local)
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
...,
|
||||
ass_only=True, # Skip video encoding
|
||||
)
|
||||
|
||||
# Upload ASS to GCS
|
||||
storage.upload_file(result["ass_path"], f"jobs/{job_id}/previews/{preview_hash}.ass")
|
||||
|
||||
# Offload encoding to GCE
|
||||
gce_result = await encoding_service.encode_preview_video(
|
||||
job_id=preview_hash,
|
||||
ass_gcs_path=f"gs://bucket/jobs/{job_id}/previews/{preview_hash}.ass",
|
||||
audio_gcs_path=audio_gcs_path,
|
||||
output_gcs_path=f"gs://bucket/jobs/{job_id}/previews/{preview_hash}.mp4",
|
||||
)
|
||||
```
|
||||
|
||||
**Key insight**: When a pipeline has both fast (text generation) and slow (media encoding) steps, design for separability:
|
||||
- Add `_only` parameters to skip slow steps
|
||||
- Return intermediate outputs (paths to generated files)
|
||||
- Allow external services to handle the slow part
|
||||
- Keep fallback to local processing for reliability
|
||||
|
||||
**Configuration**: Controlled by `USE_GCE_PREVIEW_ENCODING` env var. Falls back to local encoding if GCE fails or is disabled.
|
||||
|
||||
**Result**: Preview generation dropped from 60+ seconds to ~15-20 seconds.
|
||||
|
||||
### External Service Response Format Mismatches
|
||||
|
||||
**Problem**: The GCE encoding worker returned responses in unexpected formats, causing `'list' object has no attribute 'get'` errors that were hard to debug through the async pipeline.
|
||||
|
||||
**Issues encountered**:
|
||||
|
||||
1. **Status endpoint returning list instead of dict**: The `/status/{job_id}` endpoint sometimes returned a list `[{...}]` instead of a dict `{...}`. Calling `.get()` on the list caused failures.
|
||||
|
||||
2. **output_files as list of paths instead of dict**: The worker returned:
|
||||
```json
|
||||
"output_files": ["jobs/id/finals/output_4k_lossless.mp4", "jobs/id/finals/output_720p.mp4"]
|
||||
```
|
||||
But the backend expected:
|
||||
```json
|
||||
"output_files": {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
|
||||
```
|
||||
|
||||
**Symptoms**:
|
||||
- Job fails late in pipeline (during encoding completion handling)
|
||||
- Error message is cryptic (`'list' object has no attribute 'get'`)
|
||||
- Hard to reproduce locally (depends on GCE worker state)
|
||||
- Multiple similar errors from different root causes
|
||||
|
||||
**Solution**: Add defensive type checking for all external service responses:
|
||||
|
||||
```python
|
||||
# Always check response type before accessing
|
||||
if isinstance(status, list):
|
||||
logger.warning(f"GCE returned list instead of dict: {status}")
|
||||
status = status[0] if status and isinstance(status[0], dict) else {}
|
||||
if not isinstance(status, dict):
|
||||
status = {}
|
||||
|
||||
# Convert list of paths to expected dict format
|
||||
if isinstance(output_files, list):
|
||||
output_dict = {}
|
||||
for path in output_files:
|
||||
filename = path.split("/")[-1]
|
||||
if "4k_lossless" in filename:
|
||||
output_dict["mp4_4k_lossless"] = path
|
||||
# ... etc
|
||||
output_files = output_dict
|
||||
```
|
||||
|
||||
**Key insight**: When integrating with external services, never assume the response format. Add explicit type checks and conversions, with logging when unexpected formats are encountered. This is especially important for services you don't control (internal microservices, third-party APIs).
|
||||
|
||||
**Debugging tip**: When hitting format errors in async pipelines, check the raw response from the external service directly (curl with API key) before assuming the bug is in your code.
|
||||
|
||||
### GCE Encoding Returns GCS Paths, Not Local Files
|
||||
|
||||
**Problem**: When using GCE encoding (remote high-performance encoder), the orchestrator received GCS blob paths like `jobs/{id}/finals/output_4k_lossless.mp4` instead of local file paths. Subsequent operations like YouTube upload checked `os.path.isfile()` which returned `False`, causing silent failures.
|
||||
|
||||
**Symptoms**:
|
||||
- YouTube upload logged "No video file available for YouTube upload"
|
||||
- Other file-based operations silently skipped
|
||||
- Dropbox/GDrive uploads succeeded (they use different code paths)
|
||||
- No errors in logs - just warnings that were easy to miss
|
||||
|
||||
**Root cause**: The `VideoWorkerOrchestrator._run_encoding()` method stored the paths returned by `GCEEncodingBackend` directly without downloading the files. For local encoding, the paths are already local. For GCE encoding, the paths are GCS blob paths that need to be downloaded first.
|
||||
|
||||
**Solution**: Added `_download_gce_encoded_files()` method that:
|
||||
1. Detects when GCE backend was used (`encoding_backend.name == "gce"`)
|
||||
2. Downloads each encoded file from GCS to the local temp directory
|
||||
3. Updates result paths to point to local files
|
||||
|
||||
```python
|
||||
# After encoding completes
|
||||
if encoding_backend.name == "gce" and self.storage:
|
||||
await self._download_gce_encoded_files(output)
|
||||
```
|
||||
|
||||
**Key insight**: When an operation can run on multiple backends (local vs remote), ensure the outputs are normalized to a common format before subsequent stages consume them. The orchestrator pattern should abstract away backend differences, not leak them to downstream consumers.
|
||||
|
||||
**Testing tip**: Test the full pipeline with each backend type, not just the encoding step in isolation. The bug only manifested when YouTube upload ran after GCE encoding - unit tests of each stage passed.
|
||||
|
||||
### Unify Encoding Logic with GCS Wheel Deployment
|
||||
|
||||
**Problem**: The GCE encoding worker had simplified, duplicated encoding logic that:
|
||||
- Didn't concatenate title + karaoke + end screens
|
||||
- Used generic filenames like `output_4k_lossless.mp4` instead of proper names
|
||||
- Lacked feature parity with the local CLI's `LocalEncodingService`
|
||||
|
||||
This caused cloud-generated videos to be incomplete (no title/end screens) and have unhelpful filenames that didn't match the local CLI output.
|
||||
|
||||
**Solution**: Deploy the karaoke-gen wheel to GCS, have the GCE worker install and use `LocalEncodingService`:
|
||||
|
||||
1. **CI uploads wheel to GCS**: `deploy-backend` job uploads wheel to `gs://bucket/wheels/`
|
||||
|
||||
2. **Dynamic wheel loading**: GCE worker downloads and installs latest wheel at job start:
|
||||
```python
|
||||
def ensure_latest_wheel():
|
||||
"""Download and install latest karaoke-gen wheel from GCS."""
|
||||
subprocess.run(["gsutil", "cp", "gs://bucket/wheels/karaoke_gen-*.whl", "/tmp/"])
|
||||
wheel = sorted(glob.glob("/tmp/karaoke_gen-*.whl"))[-1]
|
||||
subprocess.run([sys.executable, "-m", "pip", "install", "--upgrade", wheel])
|
||||
|
||||
async def process_job(job_id, request):
|
||||
ensure_latest_wheel() # Get latest code at job start
|
||||
# ... rest of processing
|
||||
```
|
||||
|
||||
3. **Single encoding path**: GCE worker imports and uses `LocalEncodingService`:
|
||||
```python
|
||||
from backend.services.local_encoding_service import LocalEncodingService, EncodingConfig
|
||||
|
||||
service = LocalEncodingService(logger=logger)
|
||||
config = EncodingConfig(
|
||||
title_video=str(title_video),
|
||||
karaoke_video=str(karaoke_video),
|
||||
output_lossless_4k_mp4=f"{base_name} (Final Karaoke Lossless 4k).mp4",
|
||||
# ... proper naming for all formats
|
||||
)
|
||||
result = service.encode_all_formats(config)
|
||||
```
|
||||
|
||||
**Key benefits**:
|
||||
1. **Single source of truth**: Same encoding logic used by CLI, Cloud Run, and GCE
|
||||
2. **Hot updates without restart**: New code deployed on next job, not VM restart
|
||||
3. **In-progress jobs unaffected**: Jobs use the code version they started with
|
||||
4. **Proper output**: Titles/end screens concatenated, files named correctly
|
||||
|
||||
**No fallback by design**: We removed all fallback logic to ensure there's only ONE code path. If the wheel fails to load, the job fails clearly rather than producing inconsistent output.
|
||||
|
||||
**Lesson**: When multiple deployment targets (CLI, Cloud Run, GCE) need the same complex logic, package it as a library and deploy via a shared mechanism (GCS wheel). Duplicating logic leads to feature drift and inconsistent behavior.
|
||||
|
||||
### Cloud Distribution Must Match Local CLI Output Structure
|
||||
|
||||
**Problem**: Cloud-generated karaoke outputs had different file organization than local CLI, causing confusion and missing files in Dropbox uploads:
|
||||
|
||||
1. **lyrics/ folder filename differences**:
|
||||
- Cloud used generic `(Karaoke).json` instead of `(Lyrics Corrections).json`
|
||||
- Cloud used `(Karaoke).txt` instead of `(Lyrics Corrected).txt`
|
||||
- Reference lyrics files (`(Lyrics Genius).txt`, etc.) weren't downloaded
|
||||
|
||||
2. **Missing files in lyrics/ folder**:
|
||||
- `(With Vocals).mkv` wasn't copied to lyrics/ subfolder
|
||||
- `previews/` subfolder with preview ASS files wasn't created
|
||||
- Reference lyrics from multiple sources weren't included
|
||||
|
||||
**Root causes**:
|
||||
- Lyrics worker uploaded reference files to GCS but didn't track them in `job.files`
|
||||
- Video worker used generic naming pattern for all lyrics files
|
||||
- Video worker didn't copy `(With Vocals).mkv` or previews to lyrics folder
|
||||
|
||||
**Solution**:
|
||||
|
||||
1. **Track all files in job.files** (lyrics_worker.py):
|
||||
```python
|
||||
# Upload ALL reference lyrics and track them for distribution
|
||||
for ref_filename, source_key in reference_files:
|
||||
if os.path.exists(ref_path):
|
||||
url = storage.upload_file(ref_path, gcs_path)
|
||||
# Track in job.files so video_worker can download
|
||||
job_manager.update_file_url(job_id, 'lyrics', f'reference_{source_key}', url)
|
||||
```
|
||||
|
||||
2. **Use proper filenames in distribution** (video_worker.py):
|
||||
```python
|
||||
lyrics_file_mappings = [
|
||||
('lrc', f"{base_name} (Karaoke).lrc"),
|
||||
('corrections', f"{base_name} (Lyrics Corrections).json"),
|
||||
('corrected_txt', f"{base_name} (Lyrics Corrected).txt"),
|
||||
('uncorrected', f"{base_name} (Lyrics Uncorrected).txt"),
|
||||
('reference_genius', f"{base_name} (Lyrics Genius).txt"),
|
||||
# ... etc
|
||||
]
|
||||
```
|
||||
|
||||
3. **Copy (With Vocals).mkv to lyrics folder**:
|
||||
```python
|
||||
if videos.get('with_vocals'):
|
||||
dest = os.path.join(lyrics_dir, f"{base_name} (With Vocals).mkv")
|
||||
storage.download_file(videos['with_vocals'], dest)
|
||||
```
|
||||
|
||||
4. **Create previews/ subfolder**:
|
||||
```python
|
||||
previews_dir = os.path.join(lyrics_dir, "previews")
|
||||
os.makedirs(previews_dir, exist_ok=True)
|
||||
preview_files = storage.list_files(f"jobs/{job_id}/previews/")
|
||||
for blob_name in preview_files:
|
||||
if blob_name.endswith('.ass'):
|
||||
storage.download_file(blob_name, dest_path)
|
||||
```
|
||||
|
||||
**Expected output structure** (matches local CLI):
|
||||
```
|
||||
NOMAD-XXXX - Artist - Title/
|
||||
├── stems/
|
||||
│ ├── Artist - Title (Vocals model_bs_roformer_...).flac
|
||||
│ └── ...
|
||||
└── lyrics/
|
||||
├── Artist - Title (Karaoke).lrc
|
||||
├── Artist - Title (Karaoke).ass
|
||||
├── Artist - Title (Lyrics Corrections).json
|
||||
├── Artist - Title (Lyrics Corrected).txt
|
||||
├── Artist - Title (Lyrics Uncorrected).txt
|
||||
├── Artist - Title (Lyrics Genius).txt
|
||||
├── Artist - Title (With Vocals).mkv
|
||||
└── previews/
|
||||
└── preview_abc123 (Karaoke).ass
|
||||
```
|
||||
|
||||
**Lesson**: When building distribution packages from cloud storage, the organization code must replicate the exact structure users expect from local execution. Track ALL intermediate files in `job.files` even if they're "optional" - users may rely on them.
|
||||
|
||||
## What We'd Do Differently
|
||||
|
||||
1. **Add Pydantic model fields test first** - Would have caught the silent field issue immediately
|
||||
|
||||
@@ -36,6 +36,16 @@
|
||||
|
||||
## Recent Changes
|
||||
|
||||
- **GCE Encoding Unified with LocalEncodingService** (2026-01-06): GCE encoding worker now uses the same `LocalEncodingService` as the local CLI, eliminating duplicated encoding logic. Output files now have proper names (`Artist - Title (Final Karaoke Lossless 4k).mp4` instead of `output_4k_lossless.mp4`) and include title/end screen concatenation. Wheel deployed to GCS; worker installs at job start for hot updates without VM restart. Removed all fallback logic - single code path for consistent output across CLI, Cloud Run, and GCE. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#unify-encoding-logic-with-gcs-wheel-deployment).
|
||||
|
||||
- **Cloud Distribution Matches Local CLI** (2026-01-06): Fixed cloud distribution (Dropbox uploads) to match local CLI output structure. The `lyrics/` folder now includes: proper filenames (`(Lyrics Corrections).json` instead of `(Karaoke).json`), all reference lyrics files (`(Lyrics Genius).txt`, `(Lyrics Spotify).txt`, etc.), `(With Vocals).mkv`, and `previews/` subfolder with preview ASS files. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#cloud-distribution-must-match-local-cli-output-structure).
|
||||
|
||||
- **Agentic Correction Performance** (2026-01-05): Optimized agentic AI correction from ~5 minutes to ~55 seconds for 20 gaps (~5-6x speedup). Fixed anti-pattern of creating new model instance per gap (caused repeated 2s warm-up overhead). Now creates model once and processes gaps in parallel using ThreadPoolExecutor (default 5 workers). Configure via `AGENTIC_MAX_PARALLEL_GAPS` env var. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#reuse-llm-model-instances-across-operations).
|
||||
|
||||
- **Worker Logs Subcollection** (2026-01-04): Moved `worker_logs` from embedded array in job documents to Firestore subcollection (`jobs/{job_id}/logs`). Fixes job failures when logs exceed 1MB (job 501258e1 had 1.26MB of logs). New logs stored with 30-day TTL via Firestore TTL policy. Feature flag `USE_LOG_SUBCOLLECTION=true` (default). See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#firestore-document-1mb-limit-with-embedded-arrays).
|
||||
|
||||
- **GCE Encoding Response Fixes** (2026-01-04): Fixed multiple response format mismatches with GCE encoding worker. The worker returns `output_files` as a list of paths, not a dict with format keys - added conversion logic. Also added defensive type checking for status responses that could be lists. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#external-service-response-format-mismatches).
|
||||
|
||||
- **Video Worker Orchestrator** (2026-01-04): Major refactor to unify video generation pipeline. Created VideoWorkerOrchestrator that coordinates all stages (packaging, encoding, distribution, notifications) regardless of encoding backend (GCE or local). Fixes issue where GCE encoding path bypassed YouTube upload, Discord notifications, and CDG/TXT packaging. Feature flag `USE_NEW_ORCHESTRATOR` (default: true) enables rollback. 139 new tests across 6 new service modules. See [ARCHITECTURE.md](ARCHITECTURE.md#video-worker-orchestrator) and [archive/2026-01-04-video-worker-orchestrator-refactor.md](archive/2026-01-04-video-worker-orchestrator-refactor.md).
|
||||
|
||||
- **Full Unicode Font Support** (2026-01-03): Fixed rendering of musical symbols (♪) and added comprehensive international font support to Docker base image. Installed Noto fonts covering Latin, CJK (Chinese/Japanese/Korean), Arabic, Hebrew, Thai, and other scripts. Changed default karaoke font from Arial to Noto Sans. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#fonts-in-docker-for-video-rendering).
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
import { useState, useEffect, useCallback } from "react"
|
||||
import { useRouter } from "next/navigation"
|
||||
import { api, Job, getAccessToken } from "@/lib/api"
|
||||
import { useAuth } from "@/lib/auth"
|
||||
import { useJobNotifications, useVisibilityRefresh } from "@/hooks/use-notifications"
|
||||
import { Button } from "@/components/ui/button"
|
||||
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
|
||||
@@ -28,6 +29,7 @@ export default function AppPage() {
|
||||
const [isInitialLoad, setIsInitialLoad] = useState(true)
|
||||
const [isAuthenticated, setIsAuthenticated] = useState<boolean | null>(null)
|
||||
const { isDarkMode, toggleTheme, mounted } = useTheme()
|
||||
const { user, fetchUser } = useAuth()
|
||||
|
||||
// Memoize loadJobs for use with visibility refresh
|
||||
const loadJobs = useCallback(async () => {
|
||||
@@ -67,6 +69,16 @@ export default function AppPage() {
|
||||
setIsAuthenticated(true)
|
||||
}, [router])
|
||||
|
||||
// Ensure user data is loaded after authentication
|
||||
// This handles the case where user navigates via client-side navigation
|
||||
// (e.g., after beta enrollment) and the module-level fetchUser() didn't run
|
||||
useEffect(() => {
|
||||
const token = getAccessToken()
|
||||
if (token && !user) {
|
||||
fetchUser()
|
||||
}
|
||||
}, [user, fetchUser])
|
||||
|
||||
// Load jobs on mount (only if authenticated)
|
||||
useEffect(() => {
|
||||
if (isAuthenticated !== true) {
|
||||
|
||||
@@ -154,6 +154,23 @@ test.describe('E2E Happy Path - Real User with Full UI Interactions', () => {
|
||||
// Only create email helper if we're doing the full signup flow
|
||||
const emailHelper = usePreConfiguredToken ? null : await createEmailHelper();
|
||||
|
||||
// Intercept API requests to disable YouTube upload for E2E tests
|
||||
// This prevents test jobs from uploading to YouTube when server default is true
|
||||
await page.route('**/api/audio-search/search', async (route) => {
|
||||
const request = route.request();
|
||||
if (request.method() === 'POST') {
|
||||
const postData = request.postDataJSON();
|
||||
// Explicitly disable YouTube upload for E2E test jobs
|
||||
postData.enable_youtube_upload = false;
|
||||
console.log(' [Request Intercept] Disabled YouTube upload for audio search');
|
||||
await route.continue({
|
||||
postData: JSON.stringify(postData),
|
||||
});
|
||||
} else {
|
||||
await route.continue();
|
||||
}
|
||||
});
|
||||
|
||||
try {
|
||||
// =========================================================================
|
||||
// STEP 1: Landing Page
|
||||
|
||||
@@ -171,6 +171,71 @@ test.describe('Authentication - Beta Enrollment', () => {
|
||||
await page.waitForURL(/\/app/, { timeout: 10000 });
|
||||
expect(page.url()).toContain('/app');
|
||||
});
|
||||
|
||||
test('beta enrollment shows credits after redirect to app', async ({ page }) => {
|
||||
// This test verifies that after beta enrollment, the /app page
|
||||
// correctly fetches and displays user data including credits.
|
||||
// This catches the bug where client-side navigation didn't trigger fetchUser().
|
||||
await setupApiFixtures(page, {
|
||||
mocks: [
|
||||
{
|
||||
method: 'POST',
|
||||
path: '/api/users/beta/enroll',
|
||||
response: {
|
||||
status: 200,
|
||||
body: {
|
||||
status: 'success',
|
||||
message: 'Enrolled in beta',
|
||||
credits_granted: 1,
|
||||
session_token: 'mock-session-token-456',
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
method: 'GET',
|
||||
path: '/api/users/me',
|
||||
response: {
|
||||
status: 200,
|
||||
body: {
|
||||
user: {
|
||||
email: 'beta-tester@example.com',
|
||||
credits: 1,
|
||||
role: 'user',
|
||||
display_name: null,
|
||||
total_jobs_created: 0,
|
||||
total_jobs_completed: 0,
|
||||
},
|
||||
has_session: true,
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
method: 'GET',
|
||||
path: '/api/jobs',
|
||||
response: { body: [] },
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
await page.goto('/');
|
||||
await page.waitForLoadState('networkidle');
|
||||
|
||||
// Complete beta enrollment
|
||||
await page.getByRole('button', { name: /join beta program/i }).click();
|
||||
await page.locator('#beta-email').fill('beta-tester@example.com');
|
||||
await page.locator('textarea').fill(
|
||||
'I want to create a karaoke video for my favorite song!'
|
||||
);
|
||||
await page.locator('input[type="checkbox"]').check();
|
||||
await page.getByRole('button', { name: /get my free credit/i }).click();
|
||||
|
||||
// Wait for redirect to /app
|
||||
await page.waitForURL(/\/app/, { timeout: 10000 });
|
||||
|
||||
// Verify credits are displayed in the UI
|
||||
// The AuthStatus component should show "1 credit" after fetchUser() completes
|
||||
await expect(page.getByText(/1 credit/i)).toBeVisible({ timeout: 10000 });
|
||||
});
|
||||
});
|
||||
|
||||
test.describe('Authentication - Token Persistence', () => {
|
||||
|
||||
+426
-97
@@ -144,6 +144,42 @@ firestore_index_gen_users_active_email = firestore.Index(
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
# =============================================================================
|
||||
# Firestore TTL Field Configuration
|
||||
# TTL (Time-to-Live) policies automatically delete documents after expiration.
|
||||
# =============================================================================
|
||||
|
||||
# Worker logs TTL: logs subcollection documents expire after 30 days
|
||||
# Collection path for subcollections: "jobs/{job_id}/logs"
|
||||
# When using subcollection groups, use just the subcollection name "logs"
|
||||
firestore_field_logs_ttl = firestore.Field(
|
||||
"firestore-field-logs-ttl",
|
||||
project=project_id,
|
||||
database=firestore_db.name,
|
||||
collection="logs", # Subcollection name (applies to all jobs/{job_id}/logs)
|
||||
field="ttl_expiry",
|
||||
ttl_config={}, # Empty block enables TTL based on ttl_expiry field
|
||||
index_config={}, # Disable indexing on TTL field
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
# =============================================================================
|
||||
# Firestore Composite Indexes for Logs Subcollection
|
||||
# =============================================================================
|
||||
|
||||
# Logs: query by worker, order by timestamp (for filtering logs by worker type)
|
||||
firestore_index_logs_worker_timestamp = firestore.Index(
|
||||
"firestore-index-logs-worker-timestamp",
|
||||
project=project_id,
|
||||
database=firestore_db.name,
|
||||
collection="logs", # Subcollection name
|
||||
fields=[
|
||||
firestore.IndexFieldArgs(field_path="worker", order="ASCENDING"),
|
||||
firestore.IndexFieldArgs(field_path="timestamp", order="ASCENDING"),
|
||||
],
|
||||
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
||||
)
|
||||
|
||||
# Create Cloud Storage Bucket
|
||||
bucket = storage.Bucket(
|
||||
"karaoke-storage",
|
||||
@@ -337,6 +373,22 @@ github_actions_service_usage = gcp.projects.IAMMember(
|
||||
member=github_actions_sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Project IAM Admin - required for Pulumi to refresh IAM binding state
|
||||
github_actions_iam_admin = gcp.projects.IAMMember(
|
||||
"github-actions-iam-admin",
|
||||
project=project_id,
|
||||
role="roles/resourcemanager.projectIamAdmin",
|
||||
member=github_actions_sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant Secret Manager Admin - required for Pulumi to refresh secret state
|
||||
github_actions_secretmanager = gcp.projects.IAMMember(
|
||||
"github-actions-secretmanager",
|
||||
project=project_id,
|
||||
role="roles/secretmanager.admin",
|
||||
member=github_actions_sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# ==================== Workload Identity Federation ====================
|
||||
# Allows GitHub Actions to authenticate without service account keys
|
||||
|
||||
@@ -811,6 +863,8 @@ pulumi.export("firestore_index_gen_users_active_created", firestore_index_gen_us
|
||||
pulumi.export("firestore_index_gen_users_active_login", firestore_index_gen_users_active_login.name)
|
||||
pulumi.export("firestore_index_gen_users_active_credits", firestore_index_gen_users_active_credits.name)
|
||||
pulumi.export("firestore_index_gen_users_active_email", firestore_index_gen_users_active_email.name)
|
||||
pulumi.export("firestore_field_logs_ttl", firestore_field_logs_ttl.name)
|
||||
pulumi.export("firestore_index_logs_worker_timestamp", firestore_index_logs_worker_timestamp.name)
|
||||
pulumi.export("service_account_email", service_account.email)
|
||||
pulumi.export("artifact_repo_url", artifact_repo.name.apply(
|
||||
lambda name: f"us-central1-docker.pkg.dev/{project_id}/karaoke-repo"
|
||||
@@ -1130,6 +1184,14 @@ encoding_worker_logging_iam = gcp.projects.IAMMember(
|
||||
member=encoding_worker_sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Grant encoding worker monitoring permissions (for Ops Agent metrics)
|
||||
encoding_worker_monitoring_iam = gcp.projects.IAMMember(
|
||||
"encoding-worker-monitoring-access",
|
||||
project=project_id,
|
||||
role="roles/monitoring.metricWriter",
|
||||
member=encoding_worker_sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
||||
)
|
||||
|
||||
# Static external IP for encoding worker (Cloud Run doesn't have VPC access by default)
|
||||
encoding_worker_ip = gcp.compute.Address(
|
||||
"encoding-worker-ip",
|
||||
@@ -1151,6 +1213,13 @@ echo "Starting encoding worker setup at $(date)"
|
||||
apt-get update
|
||||
apt-get install -y python3-pip python3-venv docker.io curl git xz-utils
|
||||
|
||||
# Install fonts for ASS subtitle rendering (libass uses fontconfig)
|
||||
# - fonts-noto: Noto Sans (default karaoke font)
|
||||
# - fonts-noto-cjk: CJK character support
|
||||
# - fontconfig: Font configuration system
|
||||
apt-get install -y fonts-noto fonts-noto-cjk fontconfig
|
||||
fc-cache -fv # Rebuild font cache
|
||||
|
||||
# Enable and start Docker
|
||||
systemctl enable docker
|
||||
systemctl start docker
|
||||
@@ -1190,10 +1259,12 @@ fi
|
||||
# Create the encoding worker service
|
||||
cat > /opt/encoding-worker/main.py << 'PYTHON_CODE'
|
||||
import asyncio
|
||||
import glob as glob_module
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import uuid
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
@@ -1235,6 +1306,17 @@ class EncodeRequest(BaseModel):
|
||||
output_gcs_path: str # gs://bucket/path/to/outputs/
|
||||
encoding_config: dict # Video formats to generate
|
||||
|
||||
|
||||
class EncodePreviewRequest(BaseModel):
|
||||
job_id: str
|
||||
ass_gcs_path: str # gs://bucket/path/to/subtitles.ass
|
||||
audio_gcs_path: str # gs://bucket/path/to/audio.flac
|
||||
output_gcs_path: str # gs://bucket/path/to/output.mp4
|
||||
background_color: str = "black"
|
||||
background_image_gcs_path: Optional[str] = None
|
||||
font_gcs_path: Optional[str] = None # gs://bucket/path/to/custom-font.ttf
|
||||
|
||||
|
||||
class JobStatus(BaseModel):
|
||||
job_id: str
|
||||
status: str # pending, running, complete, failed
|
||||
@@ -1287,122 +1369,306 @@ def upload_to_gcs(local_path: Path, gcs_uri: str):
|
||||
logger.info(f"Uploaded: {file} -> gs://{bucket_name}/{blob_name}")
|
||||
|
||||
|
||||
def run_encoding(job_id: str, work_dir: Path, config: dict):
|
||||
# Run FFmpeg encoding for all video formats
|
||||
def download_single_file_from_gcs(gcs_uri: str, local_path: Path):
|
||||
# Download a single file from GCS
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
blob_name = parts[1] if len(parts) > 1 else ""
|
||||
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
blob = bucket.blob(blob_name)
|
||||
local_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
blob.download_to_filename(str(local_path))
|
||||
logger.info(f"Downloaded: {gcs_uri} -> {local_path}")
|
||||
|
||||
|
||||
def upload_single_file_to_gcs(local_path: Path, gcs_uri: str):
|
||||
# Upload a single file to a specific GCS path
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
blob_name = parts[1] if len(parts) > 1 else ""
|
||||
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
blob = bucket.blob(blob_name)
|
||||
blob.upload_from_filename(str(local_path))
|
||||
logger.info(f"Uploaded: {local_path} -> {gcs_uri}")
|
||||
|
||||
|
||||
def run_preview_encoding(job_id: str, work_dir: Path, request: "EncodePreviewRequest"):
|
||||
# Run FFmpeg encoding for preview video (480x270, fast settings)
|
||||
jobs[job_id]["status"] = "running"
|
||||
jobs[job_id]["progress"] = 10
|
||||
|
||||
try:
|
||||
# Find input files (search recursively since GCS downloads preserve directory structure)
|
||||
input_files = list(work_dir.glob("**/*.mkv")) + list(work_dir.glob("**/*.mp4")) + list(work_dir.glob("**/*.mov"))
|
||||
# Exclude files in the outputs directory to avoid re-encoding previous outputs
|
||||
input_files = [f for f in input_files if "outputs" not in str(f)]
|
||||
if not input_files:
|
||||
raise ValueError(f"No video files found in {work_dir}")
|
||||
# Download input files
|
||||
ass_path = work_dir / "subtitles.ass"
|
||||
audio_path = work_dir / "audio.flac"
|
||||
|
||||
# Use first video file as input (With Vocals or Karaoke)
|
||||
input_video = input_files[0]
|
||||
logger.info(f"Using input video: {input_video}")
|
||||
download_single_file_from_gcs(request.ass_gcs_path, ass_path)
|
||||
jobs[job_id]["progress"] = 20
|
||||
|
||||
# Find instrumental audio (search recursively, case-insensitive patterns)
|
||||
# GCS stores as: instrumental_clean.flac, instrumental_with_backing.flac
|
||||
instrumental_files = (
|
||||
list(work_dir.glob("**/*instrumental*.flac")) +
|
||||
list(work_dir.glob("**/*instrumental*.wav")) +
|
||||
list(work_dir.glob("**/*Instrumental*.flac")) +
|
||||
list(work_dir.glob("**/*Instrumental*.wav"))
|
||||
download_single_file_from_gcs(request.audio_gcs_path, audio_path)
|
||||
jobs[job_id]["progress"] = 30
|
||||
|
||||
# Download background image if provided
|
||||
bg_image_path = None
|
||||
if request.background_image_gcs_path:
|
||||
bg_image_path = work_dir / "background.png"
|
||||
download_single_file_from_gcs(request.background_image_gcs_path, bg_image_path)
|
||||
|
||||
# Download custom font if provided and register with fontconfig
|
||||
if request.font_gcs_path:
|
||||
# Use standard fontconfig location that's already in the search path
|
||||
fonts_dir = Path("/usr/local/share/fonts/custom")
|
||||
fonts_dir.mkdir(parents=True, exist_ok=True)
|
||||
font_filename = request.font_gcs_path.split("/")[-1]
|
||||
font_path = fonts_dir / font_filename
|
||||
download_single_file_from_gcs(request.font_gcs_path, font_path)
|
||||
logger.info(f"Downloaded custom font: {font_path}")
|
||||
# Update fontconfig cache so libass can find the font
|
||||
subprocess.run(["fc-cache", "-fv"], capture_output=True)
|
||||
logger.info(f"Updated fontconfig cache with custom font: {font_filename}")
|
||||
|
||||
# Build FFmpeg command
|
||||
output_path = work_dir / "preview.mp4"
|
||||
|
||||
# Escape special characters in path for FFmpeg filter syntax
|
||||
# FFmpeg filter parsing requires escaping: \\ : , [ ] ;
|
||||
def escape_ffmpeg_filter_path(path: str) -> str:
|
||||
# Note: Extra escaping needed since this is inside a triple-quoted string in Pulumi
|
||||
return path.replace("\\\\", "\\\\\\\\").replace(":", "\\\\:").replace(",", "\\\\,").replace("[", "\\\\[").replace("]", "\\\\]").replace(";", "\\\\;")
|
||||
|
||||
escaped_ass_path = escape_ffmpeg_filter_path(str(ass_path))
|
||||
|
||||
# Base command with frame rate
|
||||
cmd = ["ffmpeg", "-y", "-r", "24"]
|
||||
|
||||
# Video input: background image or solid color
|
||||
if bg_image_path and bg_image_path.exists():
|
||||
cmd.extend(["-loop", "1", "-i", str(bg_image_path)])
|
||||
# Scale and pad background to 480x270
|
||||
vf = f"scale=480:270:force_original_aspect_ratio=decrease,pad=480:270:(ow-iw)/2:(oh-ih)/2,ass={escaped_ass_path}"
|
||||
else:
|
||||
# Solid color background
|
||||
color = request.background_color or "black"
|
||||
cmd.extend(["-f", "lavfi", "-i", f"color=c={color}:s=480x270:r=24"])
|
||||
vf = f"ass={escaped_ass_path}"
|
||||
|
||||
# Audio input
|
||||
cmd.extend(["-i", str(audio_path)])
|
||||
|
||||
# Video filter and encoding settings
|
||||
cmd.extend([
|
||||
"-vf", vf,
|
||||
"-c:a", "aac", "-b:a", "96k",
|
||||
"-c:v", "libx264",
|
||||
"-preset", "superfast",
|
||||
"-crf", "28",
|
||||
"-pix_fmt", "yuv420p",
|
||||
"-movflags", "+faststart",
|
||||
"-threads", "8",
|
||||
"-shortest",
|
||||
str(output_path)
|
||||
])
|
||||
|
||||
jobs[job_id]["progress"] = 40
|
||||
logger.info(f"Running preview encoding: {' '.join(cmd)}")
|
||||
|
||||
result = subprocess.run(cmd, capture_output=True, text=True)
|
||||
|
||||
if result.returncode != 0:
|
||||
logger.error(f"FFmpeg failed: {result.stderr}")
|
||||
raise RuntimeError(f"FFmpeg preview encoding failed: {result.stderr[-500:]}")
|
||||
|
||||
jobs[job_id]["progress"] = 80
|
||||
logger.info(f"Preview encoded: {output_path}")
|
||||
|
||||
# Upload output to GCS
|
||||
upload_single_file_to_gcs(output_path, request.output_gcs_path)
|
||||
jobs[job_id]["progress"] = 95
|
||||
|
||||
jobs[job_id]["output_files"] = [request.output_gcs_path]
|
||||
return output_path
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Preview encoding failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
raise
|
||||
|
||||
|
||||
def ensure_latest_wheel():
|
||||
"""Download and install latest karaoke-gen wheel from GCS.
|
||||
|
||||
Called at the start of each job to enable hot code updates without restart.
|
||||
In-progress jobs continue with their version, new jobs get latest code.
|
||||
"""
|
||||
try:
|
||||
logger.info("Checking for latest karaoke-gen wheel in GCS...")
|
||||
|
||||
# Download latest wheel
|
||||
result = subprocess.run(
|
||||
["gsutil", "cp", "gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-*.whl", "/tmp/"],
|
||||
capture_output=True, text=True, timeout=60
|
||||
)
|
||||
instrumental = instrumental_files[0] if instrumental_files else None
|
||||
|
||||
# Find the downloaded wheel (get the latest by version sorting)
|
||||
wheels = glob_module.glob("/tmp/karaoke_gen-*.whl")
|
||||
if not wheels:
|
||||
logger.warning("No wheel found in GCS, using fallback encoding logic")
|
||||
return False
|
||||
|
||||
# Sort to get latest version
|
||||
wheel_path = sorted(wheels)[-1]
|
||||
logger.info(f"Installing wheel: {wheel_path}")
|
||||
|
||||
# Install (or upgrade) the wheel
|
||||
install_result = subprocess.run(
|
||||
[sys.executable, "-m", "pip", "install", "--upgrade", "--quiet", wheel_path],
|
||||
capture_output=True, text=True, timeout=120
|
||||
)
|
||||
|
||||
if install_result.returncode != 0:
|
||||
logger.warning(f"Wheel installation failed: {install_result.stderr}")
|
||||
return False
|
||||
|
||||
logger.info(f"Successfully installed wheel: {wheel_path}")
|
||||
return True
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning("Wheel download/install timed out, using fallback")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to ensure latest wheel: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def find_file(work_dir: Path, *patterns):
|
||||
"""Find a file matching any of the given glob patterns."""
|
||||
for pattern in patterns:
|
||||
matches = list(work_dir.glob(f"**/{pattern}"))
|
||||
if matches:
|
||||
return matches[0]
|
||||
return None
|
||||
|
||||
|
||||
def run_encoding(job_id: str, work_dir: Path, config: dict):
|
||||
"""Run encoding using LocalEncodingService (single source of truth).
|
||||
|
||||
Uses LocalEncodingService from the installed karaoke-gen wheel to ensure
|
||||
output files match local CLI exactly:
|
||||
- Proper names like "Artist - Title (Final Karaoke Lossless 4k).mp4"
|
||||
- Concatenated title + karaoke + end screens
|
||||
- All formats: lossless 4K MP4, lossy 4K MP4, lossless MKV, 720p MP4
|
||||
|
||||
Requires the karaoke-gen wheel to be installed (done by ensure_latest_wheel).
|
||||
"""
|
||||
jobs[job_id]["status"] = "running"
|
||||
jobs[job_id]["progress"] = 10
|
||||
|
||||
try:
|
||||
# Import LocalEncodingService from installed wheel (required, no fallback)
|
||||
from backend.services.local_encoding_service import LocalEncodingService, EncodingConfig
|
||||
logger.info("Using LocalEncodingService from installed wheel")
|
||||
|
||||
# Get artist/title from config for proper naming
|
||||
artist = config.get("artist", "Unknown Artist")
|
||||
title = config.get("title", "Unknown Title")
|
||||
base_name = f"{artist} - {title}"
|
||||
logger.info(f"Encoding for: {base_name}")
|
||||
|
||||
# Find input files in work_dir
|
||||
# Title/end screens are in screens/ subdirectory
|
||||
title_video = find_file(work_dir, "screens/title.mov", "*Title*.mov", "*title*.mov")
|
||||
end_video = find_file(work_dir, "screens/end.mov", "*End*.mov", "*end*.mov")
|
||||
|
||||
# Karaoke video - search for With Vocals or main karaoke video
|
||||
karaoke_video = find_file(
|
||||
work_dir,
|
||||
"*With Vocals*.mov", "*With Vocals*.mkv",
|
||||
"*Vocals*.mov", "*Vocals*.mkv",
|
||||
"*.mkv", "*.mov"
|
||||
)
|
||||
# Exclude title/end/output videos
|
||||
if karaoke_video:
|
||||
name_lower = karaoke_video.name.lower()
|
||||
if "title" in name_lower or "end" in name_lower or "outputs" in str(karaoke_video):
|
||||
# Search more specifically for karaoke video
|
||||
karaoke_video = find_file(work_dir, "*Karaoke*.mkv", "*Karaoke*.mov", "*vocals*.mkv")
|
||||
|
||||
# Instrumental audio
|
||||
instrumental = find_file(
|
||||
work_dir,
|
||||
"*instrumental_clean*.flac", "*Instrumental Clean*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
|
||||
logger.info(f"Found files:")
|
||||
logger.info(f" Title video: {title_video}")
|
||||
logger.info(f" Karaoke video: {karaoke_video}")
|
||||
logger.info(f" End video: {end_video}")
|
||||
logger.info(f" Instrumental: {instrumental}")
|
||||
|
||||
# Validate required files
|
||||
if not title_video:
|
||||
raise ValueError(f"No title video found in {work_dir}. Check screens/ subdirectory.")
|
||||
if not karaoke_video:
|
||||
raise ValueError(f"No karaoke video found in {work_dir}")
|
||||
if not instrumental:
|
||||
raise ValueError(f"No instrumental audio found in {work_dir}")
|
||||
|
||||
output_dir = work_dir / "outputs"
|
||||
output_dir.mkdir(exist_ok=True)
|
||||
|
||||
output_files = []
|
||||
formats = config.get("formats", ["mp4_4k", "mp4_720p"])
|
||||
total_formats = len(formats)
|
||||
jobs[job_id]["progress"] = 20
|
||||
|
||||
for i, fmt in enumerate(formats):
|
||||
progress = 10 + int((i / total_formats) * 80)
|
||||
jobs[job_id]["progress"] = progress
|
||||
# Build encoding config with proper file names
|
||||
encoding_config = EncodingConfig(
|
||||
title_video=str(title_video),
|
||||
karaoke_video=str(karaoke_video),
|
||||
instrumental_audio=str(instrumental),
|
||||
end_video=str(end_video) if end_video else None,
|
||||
output_karaoke_mp4=str(output_dir / f"{base_name} (Karaoke).mp4"),
|
||||
output_with_vocals_mp4=str(output_dir / f"{base_name} (With Vocals).mp4"),
|
||||
output_lossless_4k_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossless 4k).mp4"),
|
||||
output_lossy_4k_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossy 4k).mp4"),
|
||||
output_lossless_mkv=str(output_dir / f"{base_name} (Final Karaoke Lossless 4k).mkv"),
|
||||
output_720p_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossy 720p).mp4"),
|
||||
)
|
||||
|
||||
if fmt == "mp4_4k_lossless" or fmt == "mp4_4k":
|
||||
# High quality 4K - CRF 18 (visually lossless)
|
||||
output_file = output_dir / "output_4k_lossless.mp4"
|
||||
cmd = [
|
||||
"ffmpeg", "-y",
|
||||
"-i", str(input_video),
|
||||
]
|
||||
if instrumental:
|
||||
cmd.extend(["-i", str(instrumental)])
|
||||
cmd.extend([
|
||||
"-map", "0:v",
|
||||
"-map", "1:a" if instrumental else "0:a",
|
||||
"-c:v", "libx264",
|
||||
"-preset", "medium",
|
||||
"-crf", "18",
|
||||
"-c:a", "aac",
|
||||
"-b:a", "256k",
|
||||
"-threads", "24",
|
||||
str(output_file)
|
||||
])
|
||||
elif fmt == "mp4_4k_lossy":
|
||||
# Smaller 4K file - CRF 23 (good quality, smaller size)
|
||||
output_file = output_dir / "output_4k_lossy.mp4"
|
||||
cmd = [
|
||||
"ffmpeg", "-y",
|
||||
"-i", str(input_video),
|
||||
]
|
||||
if instrumental:
|
||||
cmd.extend(["-i", str(instrumental)])
|
||||
cmd.extend([
|
||||
"-map", "0:v",
|
||||
"-map", "1:a" if instrumental else "0:a",
|
||||
"-c:v", "libx264",
|
||||
"-preset", "medium",
|
||||
"-crf", "23",
|
||||
"-c:a", "aac",
|
||||
"-b:a", "192k",
|
||||
"-threads", "24",
|
||||
str(output_file)
|
||||
])
|
||||
elif fmt == "mp4_720p":
|
||||
output_file = output_dir / "output_720p.mp4"
|
||||
cmd = [
|
||||
"ffmpeg", "-y",
|
||||
"-i", str(input_video),
|
||||
]
|
||||
if instrumental:
|
||||
cmd.extend(["-i", str(instrumental)])
|
||||
cmd.extend([
|
||||
"-map", "0:v",
|
||||
"-map", "1:a" if instrumental else "0:a",
|
||||
"-vf", "scale=-1:720",
|
||||
"-c:v", "libx264",
|
||||
"-preset", "medium",
|
||||
"-crf", "20",
|
||||
"-c:a", "aac",
|
||||
"-b:a", "192k",
|
||||
"-threads", "24",
|
||||
str(output_file)
|
||||
])
|
||||
else:
|
||||
logger.warning(f"Unknown format: {fmt}")
|
||||
continue
|
||||
# Create service and run encoding
|
||||
service = LocalEncodingService(logger=logger)
|
||||
|
||||
logger.info(f"Running: {' '.join(cmd)}")
|
||||
result = subprocess.run(cmd, capture_output=True, text=True)
|
||||
jobs[job_id]["progress"] = 30
|
||||
logger.info("Starting LocalEncodingService.encode_all_formats()")
|
||||
|
||||
if result.returncode != 0:
|
||||
logger.error(f"FFmpeg failed: {result.stderr}")
|
||||
raise RuntimeError(f"FFmpeg encoding failed: {result.stderr[-500:]}")
|
||||
result = service.encode_all_formats(encoding_config)
|
||||
|
||||
output_files.append(str(output_file))
|
||||
logger.info(f"Encoded: {output_file}")
|
||||
if not result.success:
|
||||
raise RuntimeError(f"Encoding failed: {result.error}")
|
||||
|
||||
jobs[job_id]["output_files"] = output_files
|
||||
jobs[job_id]["progress"] = 90
|
||||
|
||||
# Collect output files
|
||||
output_files = [str(f) for f in output_dir.glob("*") if f.is_file()]
|
||||
jobs[job_id]["output_files"] = output_files
|
||||
|
||||
logger.info(f"Encoding complete. Output files: {output_files}")
|
||||
return output_dir
|
||||
|
||||
except ImportError as e:
|
||||
# No fallback - wheel must be installed
|
||||
error_msg = (
|
||||
f"LocalEncodingService not available: {e}. "
|
||||
"The karaoke-gen wheel must be installed. "
|
||||
"Check that ensure_latest_wheel() succeeded and wheel exists in GCS."
|
||||
)
|
||||
logger.error(error_msg)
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = error_msg
|
||||
raise RuntimeError(error_msg) from e
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Encoding failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
@@ -1413,6 +1679,10 @@ def run_encoding(job_id: str, work_dir: Path, config: dict):
|
||||
async def process_job(job_id: str, request: EncodeRequest):
|
||||
# Process an encoding job asynchronously
|
||||
try:
|
||||
# Download and install latest wheel at job start (allows hot updates without restart)
|
||||
# This means in-progress jobs continue with their version, new jobs get latest code
|
||||
ensure_latest_wheel()
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
work_dir = Path(temp_dir) / "work"
|
||||
work_dir.mkdir()
|
||||
@@ -1439,7 +1709,7 @@ async def process_job(job_id: str, request: EncodeRequest):
|
||||
|
||||
# Convert local paths to blob paths (backend expects blob paths, not full gs:// URIs)
|
||||
# output_gcs_path is like "gs://bucket/jobs/id/encoded/"
|
||||
# We need paths like "jobs/id/encoded/output_4k_lossless.mp4"
|
||||
# We need paths like "jobs/id/encoded/Artist - Title (Final Karaoke Lossless 4k).mp4"
|
||||
gcs_path = request.output_gcs_path.replace("gs://", "")
|
||||
parts = gcs_path.split("/", 1)
|
||||
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
|
||||
@@ -1462,6 +1732,65 @@ async def process_job(job_id: str, request: EncodeRequest):
|
||||
jobs[job_id]["error"] = str(e)
|
||||
|
||||
|
||||
async def process_preview_job(job_id: str, request: EncodePreviewRequest):
|
||||
# Process a preview encoding job asynchronously
|
||||
try:
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
work_dir = Path(temp_dir) / "work"
|
||||
work_dir.mkdir()
|
||||
|
||||
# Run preview encoding in thread pool (CPU-bound)
|
||||
# Note: run_preview_encoding handles download/upload internally
|
||||
loop = asyncio.get_event_loop()
|
||||
await loop.run_in_executor(
|
||||
executor,
|
||||
run_preview_encoding,
|
||||
job_id,
|
||||
work_dir,
|
||||
request
|
||||
)
|
||||
|
||||
jobs[job_id]["status"] = "complete"
|
||||
jobs[job_id]["progress"] = 100
|
||||
logger.info(f"Preview job {job_id} complete")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Preview job {job_id} failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
|
||||
|
||||
@app.post("/encode-preview")
|
||||
async def submit_preview_encode_job(request: EncodePreviewRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
|
||||
# Submit a preview encoding job
|
||||
job_id = request.job_id
|
||||
|
||||
# If job already exists, return cached result or current status
|
||||
if job_id in jobs:
|
||||
existing_job = jobs[job_id]
|
||||
if existing_job["status"] == "complete":
|
||||
# Return cached result - preview already encoded
|
||||
return {"status": "cached", "job_id": job_id, "output_path": existing_job.get("output_path")}
|
||||
elif existing_job["status"] == "failed":
|
||||
# Previous attempt failed, allow retry by replacing the job
|
||||
pass
|
||||
else:
|
||||
# Job is still in progress
|
||||
return {"status": "in_progress", "job_id": job_id}
|
||||
|
||||
jobs[job_id] = {
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"progress": 0,
|
||||
"error": None,
|
||||
"output_files": None,
|
||||
}
|
||||
|
||||
background_tasks.add_task(process_preview_job, job_id, request)
|
||||
|
||||
return {"status": "accepted", "job_id": job_id}
|
||||
|
||||
|
||||
@app.post("/encode")
|
||||
async def submit_encode_job(request: EncodeRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
|
||||
# Submit an encoding job
|
||||
|
||||
@@ -156,15 +156,20 @@ class VideoBackgroundProcessor:
|
||||
"""
|
||||
Escape a file path for use in ffmpeg filter expressions.
|
||||
|
||||
FFmpeg filter syntax uses single quotes to protect special characters
|
||||
like spaces, colons, and semicolons. Single quotes within the path
|
||||
are escaped using the '\\'' pattern (end quote, literal quote, start quote).
|
||||
|
||||
Args:
|
||||
path: File path to escape
|
||||
|
||||
Returns:
|
||||
str: Escaped path
|
||||
str: Escaped path (wrapped in single quotes)
|
||||
"""
|
||||
# Escape backslashes and colons for ffmpeg filter syntax
|
||||
escaped = path.replace("\\", "\\\\").replace(":", "\\:")
|
||||
return escaped
|
||||
# Escape single quotes: ' becomes '\'' (end quote, \', start quote)
|
||||
escaped = path.replace("'", "'\\''")
|
||||
# Wrap entire path in single quotes
|
||||
return f"'{escaped}'"
|
||||
|
||||
def build_video_filter(self, ass_subtitles_path, darkness_percent, fonts_dir=None):
|
||||
"""
|
||||
|
||||
@@ -35,10 +35,14 @@ class ProviderConfig:
|
||||
circuit_breaker_failure_threshold: int = 3
|
||||
circuit_breaker_open_seconds: int = 60
|
||||
|
||||
# Initialization timeouts - fail fast instead of hanging forever
|
||||
# These are separate from request_timeout to catch connection establishment issues
|
||||
initialization_timeout_seconds: float = 30.0 # Model creation + warm-up
|
||||
warmup_timeout_seconds: float = 15.0 # Just the warm-up call
|
||||
# Initialization timeout - fail fast instead of hanging forever
|
||||
# This is separate from request_timeout to catch connection establishment issues
|
||||
initialization_timeout_seconds: float = 30.0 # Model creation
|
||||
|
||||
# Parallel processing settings
|
||||
# Process multiple gaps concurrently to reduce total correction time
|
||||
# Set to 1 to disable parallelism, higher values increase throughput but may hit rate limits
|
||||
max_parallel_gaps: int = 5
|
||||
|
||||
@staticmethod
|
||||
def from_env(cache_dir: Optional[str] = None) -> "ProviderConfig":
|
||||
@@ -70,7 +74,7 @@ class ProviderConfig:
|
||||
circuit_breaker_failure_threshold=int(os.getenv("AGENTIC_CIRCUIT_THRESHOLD", "3")),
|
||||
circuit_breaker_open_seconds=int(os.getenv("AGENTIC_CIRCUIT_OPEN_SECONDS", "60")),
|
||||
initialization_timeout_seconds=float(os.getenv("AGENTIC_INIT_TIMEOUT_SECONDS", "30.0")),
|
||||
warmup_timeout_seconds=float(os.getenv("AGENTIC_WARMUP_TIMEOUT_SECONDS", "15.0")),
|
||||
max_parallel_gaps=int(os.getenv("AGENTIC_MAX_PARALLEL_GAPS", "5")),
|
||||
)
|
||||
|
||||
def validate_environment(self, logger: Optional[object] = None) -> None:
|
||||
|
||||
+1
-51
@@ -97,7 +97,6 @@ class LangChainBridge(BaseAIProvider):
|
||||
|
||||
# Lazy-initialized chat model
|
||||
self._chat_model: Optional[Any] = None
|
||||
self._warmed_up: bool = False
|
||||
|
||||
def name(self) -> str:
|
||||
"""Return provider name for logging."""
|
||||
@@ -164,13 +163,7 @@ class LangChainBridge(BaseAIProvider):
|
||||
) from None
|
||||
|
||||
init_elapsed = time.time() - init_start
|
||||
logger.info(f"🤖 Model created in {init_elapsed:.2f}s, starting warm-up...")
|
||||
|
||||
# Warm up the model to establish connection before real work
|
||||
self._warm_up_model()
|
||||
|
||||
total_elapsed = time.time() - init_start
|
||||
logger.info(f"🤖 Model initialization complete in {total_elapsed:.2f}s")
|
||||
logger.info(f"🤖 Model initialized in {init_elapsed:.2f}s")
|
||||
|
||||
except InitializationTimeoutError as e:
|
||||
self._circuit_breaker.record_failure(self._model)
|
||||
@@ -271,47 +264,4 @@ class LangChainBridge(BaseAIProvider):
|
||||
|
||||
return content
|
||||
|
||||
def _warm_up_model(self) -> None:
|
||||
"""Send a lightweight request to warm up the model connection.
|
||||
|
||||
This helps establish the REST connection and potentially warm up any
|
||||
server-side resources before processing real correction requests.
|
||||
The warm-up uses a timeout to fail fast if the service is unresponsive.
|
||||
"""
|
||||
if self._warmed_up:
|
||||
return
|
||||
|
||||
timeout = self._config.warmup_timeout_seconds
|
||||
# Use print with flush=True for visibility when output is redirected
|
||||
print(f"🔥 Warming up {self._model} connection (timeout: {timeout}s)...", flush=True)
|
||||
logger.info(f"🔥 Warming up {self._model} connection (timeout: {timeout}s)...")
|
||||
|
||||
warmup_start = time.time()
|
||||
try:
|
||||
from langchain_core.messages import HumanMessage
|
||||
|
||||
# Minimal prompt that requires almost no processing
|
||||
warm_up_prompt = 'Respond with exactly: {"status":"ready"}'
|
||||
|
||||
# Use ThreadPoolExecutor for timeout on warm-up call
|
||||
with ThreadPoolExecutor(max_workers=1) as executor:
|
||||
future = executor.submit(
|
||||
self._chat_model.invoke,
|
||||
[HumanMessage(content=warm_up_prompt)]
|
||||
)
|
||||
try:
|
||||
future.result(timeout=timeout)
|
||||
except FuturesTimeoutError:
|
||||
raise TimeoutError(f"Warm-up timed out after {timeout}s") from None
|
||||
|
||||
elapsed = time.time() - warmup_start
|
||||
self._warmed_up = True
|
||||
print(f"🔥 Warm-up complete for {self._model} in {elapsed:.2f}s", flush=True)
|
||||
logger.info(f"🔥 Warm-up complete for {self._model} in {elapsed:.2f}s")
|
||||
except Exception as e:
|
||||
elapsed = time.time() - warmup_start
|
||||
# Don't fail the actual request if warm-up fails
|
||||
# Just log and continue - the real request might still work
|
||||
print(f"🔥 Warm-up failed for {self._model} after {elapsed:.2f}s: {e} (continuing anyway)", flush=True)
|
||||
logger.warning(f"🔥 Warm-up failed for {self._model} after {elapsed:.2f}s: {e} (continuing anyway)")
|
||||
|
||||
@@ -268,12 +268,26 @@ class LyricsCorrector:
|
||||
_adapt = None
|
||||
_ModelRouter = None
|
||||
|
||||
# Pre-initialized agentic corrector (created once, reused for all gaps)
|
||||
_agentic_agent = None
|
||||
|
||||
if use_agentic_env:
|
||||
try:
|
||||
from lyrics_transcriber.correction.agentic.agent import AgenticCorrector as _AgenticCorrector
|
||||
from lyrics_transcriber.correction.agentic.adapter import adapt_proposals_to_word_corrections as _adapt
|
||||
from lyrics_transcriber.correction.agentic.router import ModelRouter as _ModelRouter
|
||||
self.logger.info("🤖 Agentic modules imported successfully - running in AGENTIC-ONLY mode")
|
||||
|
||||
# Create agent ONCE and reuse for all gaps (avoids repeated model initialization)
|
||||
_router = _ModelRouter()
|
||||
model_id = _router.choose_model("gap", uncertainty=0.5) # Use default uncertainty
|
||||
self.logger.info(f"🤖 Creating single AgenticCorrector with model: {model_id}")
|
||||
_agentic_agent = _AgenticCorrector.from_model(
|
||||
model=model_id,
|
||||
session_id=session_id,
|
||||
cache_dir=str(self._cache_dir)
|
||||
)
|
||||
self.logger.info("🤖 AgenticCorrector initialized and ready for all gaps")
|
||||
except Exception as e:
|
||||
self.logger.error(f"🤖 Failed to import agentic modules but USE_AGENTIC_AI=1: {e}")
|
||||
raise RuntimeError(f"Agentic AI correction is enabled but required modules could not be imported: {e}") from e
|
||||
@@ -443,16 +457,186 @@ class LyricsCorrector:
|
||||
sys.exit(0)
|
||||
# === END TEMPORARY CODE ===
|
||||
|
||||
# AGENTIC-ONLY MODE: Process all gaps in parallel for better performance
|
||||
if use_agentic_env:
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from lyrics_transcriber.correction.agentic.providers.config import ProviderConfig
|
||||
|
||||
# Get parallel processing config
|
||||
_config = ProviderConfig.from_env()
|
||||
max_workers = _config.max_parallel_gaps
|
||||
self.logger.info(f"🤖 Processing {len(gap_sequences)} gaps in parallel (max_workers={max_workers})")
|
||||
|
||||
# Pre-compute shared data structures once (not per-gap)
|
||||
all_transcribed_words = []
|
||||
for seg in segments:
|
||||
all_transcribed_words.extend(seg.words)
|
||||
word_position = {w.id: idx for idx, w in enumerate(all_transcribed_words)}
|
||||
|
||||
# Build reference contexts once (same for all gaps)
|
||||
reference_contexts = {}
|
||||
for source, lyrics_data in self.reference_lyrics.items():
|
||||
if lyrics_data and lyrics_data.segments:
|
||||
ref_words = []
|
||||
for seg in lyrics_data.segments:
|
||||
ref_words.extend([w.text for w in seg.words])
|
||||
reference_contexts[source] = " ".join(ref_words)
|
||||
|
||||
# Get artist and title once
|
||||
artist = metadata.get("artist") if metadata else None
|
||||
title = metadata.get("title") if metadata else None
|
||||
|
||||
# Prepare all gap inputs upfront
|
||||
gap_inputs = []
|
||||
for i, gap in enumerate(gap_sequences, 1):
|
||||
# Prepare gap words data
|
||||
gap_words_data = []
|
||||
for word_id in gap.transcribed_word_ids:
|
||||
if word_id in word_map:
|
||||
word = word_map[word_id]
|
||||
gap_words_data.append({
|
||||
"id": word_id,
|
||||
"text": word.text,
|
||||
"start_time": getattr(word, 'start_time', 0),
|
||||
"end_time": getattr(word, 'end_time', 0)
|
||||
})
|
||||
|
||||
# Compute context words
|
||||
gap_positions = [word_position[wid] for wid in gap.transcribed_word_ids if wid in word_position]
|
||||
preceding_words = ""
|
||||
following_words = ""
|
||||
|
||||
if gap_positions:
|
||||
first_gap_pos = min(gap_positions)
|
||||
last_gap_pos = max(gap_positions)
|
||||
|
||||
# Get 10 words before
|
||||
start_pos = max(0, first_gap_pos - 10)
|
||||
preceding_list = [all_transcribed_words[idx].text for idx in range(start_pos, first_gap_pos) if idx < len(all_transcribed_words)]
|
||||
preceding_words = " ".join(preceding_list)
|
||||
|
||||
# Get 10 words after
|
||||
end_pos = min(len(all_transcribed_words), last_gap_pos + 11)
|
||||
following_list = [all_transcribed_words[idx].text for idx in range(last_gap_pos + 1, end_pos) if idx < len(all_transcribed_words)]
|
||||
following_words = " ".join(following_list)
|
||||
|
||||
gap_inputs.append({
|
||||
'index': i,
|
||||
'gap': gap,
|
||||
'gap_id': f"gap_{i}",
|
||||
'gap_words': gap_words_data,
|
||||
'preceding_words': preceding_words,
|
||||
'following_words': following_words,
|
||||
'reference_contexts': reference_contexts,
|
||||
'artist': artist,
|
||||
'title': title
|
||||
})
|
||||
|
||||
# Function to process a single gap (runs in thread pool)
|
||||
def process_single_gap(gap_input):
|
||||
"""Process a single gap and return proposals. Thread-safe."""
|
||||
idx = gap_input['index']
|
||||
try:
|
||||
proposals = _agentic_agent.propose_for_gap(
|
||||
gap_id=gap_input['gap_id'],
|
||||
gap_words=gap_input['gap_words'],
|
||||
preceding_words=gap_input['preceding_words'],
|
||||
following_words=gap_input['following_words'],
|
||||
reference_contexts=gap_input['reference_contexts'],
|
||||
artist=gap_input['artist'],
|
||||
title=gap_input['title']
|
||||
)
|
||||
return {'index': idx, 'gap': gap_input['gap'], 'proposals': proposals, 'error': None}
|
||||
except Exception as e:
|
||||
return {'index': idx, 'gap': gap_input['gap'], 'proposals': None, 'error': str(e)}
|
||||
|
||||
# Process gaps in parallel
|
||||
results = [None] * len(gap_inputs)
|
||||
completed_count = 0
|
||||
errors = []
|
||||
|
||||
with ThreadPoolExecutor(max_workers=max_workers) as executor:
|
||||
# Submit all tasks
|
||||
future_to_input = {executor.submit(process_single_gap, g): g for g in gap_inputs}
|
||||
|
||||
# Collect results as they complete
|
||||
for future in as_completed(future_to_input):
|
||||
# Check deadline
|
||||
if deadline and time.time() > deadline:
|
||||
self.logger.warning(
|
||||
f"⏰ AGENTIC TIMEOUT: Deadline exceeded after processing {completed_count}/{len(gap_sequences)} gaps. "
|
||||
"Cancelling remaining gaps - human review will correct any issues."
|
||||
)
|
||||
# Cancel remaining futures (use list() to avoid mutating dict during iteration)
|
||||
for f in list(future_to_input.keys()):
|
||||
f.cancel()
|
||||
break
|
||||
|
||||
result = future.result()
|
||||
idx = result['index'] - 1 # Convert 1-based to 0-based
|
||||
results[idx] = result
|
||||
completed_count += 1
|
||||
|
||||
if result['error']:
|
||||
errors.append(f"Gap {result['index']}: {result['error']}")
|
||||
self.logger.error(f"🤖 Gap {result['index']} failed: {result['error']}")
|
||||
else:
|
||||
proposal_count = len(result['proposals']) if result['proposals'] else 0
|
||||
self.logger.info(f"🤖 Gap {result['index']}/{len(gap_sequences)} completed ({proposal_count} proposals)")
|
||||
|
||||
self.logger.info(f"🤖 Parallel processing complete: {completed_count}/{len(gap_sequences)} gaps processed")
|
||||
|
||||
# If any errors occurred, fail fast
|
||||
if errors:
|
||||
raise RuntimeError(f"Agentic AI correction failed for {len(errors)} gaps: {'; '.join(errors)}")
|
||||
|
||||
# Apply corrections sequentially (must be in order due to segment modifications)
|
||||
for result in results:
|
||||
if result is None:
|
||||
continue # Skipped due to deadline
|
||||
|
||||
i = result['index']
|
||||
gap = result['gap']
|
||||
_proposals = result['proposals']
|
||||
|
||||
_agentic_corrections = _adapt(_proposals, word_map, linear_position_map) if _proposals else []
|
||||
|
||||
if _agentic_corrections:
|
||||
self.logger.info(f"🤖 Applying {len(_agentic_corrections)} agentic corrections for gap {i}")
|
||||
affected_word_ids = [w.id for w in self._get_affected_words(gap, segments)]
|
||||
affected_segment_ids = [s.id for s in self._get_affected_segments(gap, segments)]
|
||||
updated_segments = self._apply_corrections_to_segments(self._get_affected_segments(gap, segments), _agentic_corrections)
|
||||
for correction in _agentic_corrections:
|
||||
if correction.word_id and correction.corrected_word_id:
|
||||
word_id_map[correction.word_id] = correction.corrected_word_id
|
||||
for old_seg, new_seg in zip(self._get_affected_segments(gap, segments), updated_segments):
|
||||
segment_id_map[old_seg.id] = new_seg.id
|
||||
step = CorrectionStep(
|
||||
handler_name="AgenticCorrector",
|
||||
affected_word_ids=affected_word_ids,
|
||||
affected_segment_ids=affected_segment_ids,
|
||||
corrections=_agentic_corrections,
|
||||
segments_before=self._get_affected_segments(gap, segments),
|
||||
segments_after=updated_segments,
|
||||
created_word_ids=[w.id for w in self._get_new_words(updated_segments, affected_word_ids)],
|
||||
deleted_word_ids=[id for id in affected_word_ids if not self._word_exists(id, updated_segments)],
|
||||
)
|
||||
correction_steps.append(step)
|
||||
all_corrections.extend(_agentic_corrections)
|
||||
# Log corrections made
|
||||
for correction in _agentic_corrections:
|
||||
self.logger.info(
|
||||
f"Made correction: '{correction.original_word}' -> '{correction.corrected_word}' "
|
||||
f"(confidence: {correction.confidence:.2f}, reason: {correction.reason})"
|
||||
)
|
||||
else:
|
||||
self.logger.debug(f"🤖 No agentic corrections needed for gap {i}")
|
||||
|
||||
# RULE-BASED MODE: Process gaps sequentially
|
||||
for i, gap in enumerate(gap_sequences, 1):
|
||||
# Check deadline before processing each gap (agentic mode only)
|
||||
# This allows us to abort early and return uncorrected results for human review
|
||||
if deadline and use_agentic_env and time.time() > deadline:
|
||||
self.logger.warning(
|
||||
f"⏰ AGENTIC TIMEOUT: Deadline exceeded after processing {i-1}/{len(gap_sequences)} gaps. "
|
||||
"Skipping remaining gaps - human review will correct any issues."
|
||||
)
|
||||
# Break out of loop - continue with whatever corrections we have (likely none)
|
||||
break
|
||||
# Skip if we already processed in agentic mode
|
||||
if use_agentic_env:
|
||||
continue
|
||||
|
||||
self.logger.info(f"Processing gap {i}/{len(gap_sequences)} at position {gap.transcription_position}")
|
||||
|
||||
@@ -460,128 +644,6 @@ class LyricsCorrector:
|
||||
gap_words = [word_map[word_id] for word_id in gap.transcribed_word_ids]
|
||||
self.logger.debug(f"Gap text: '{' '.join(w.text for w in gap_words)}'")
|
||||
|
||||
# AGENTIC-ONLY MODE: Use agentic correction exclusively
|
||||
if use_agentic_env:
|
||||
self.logger.info(f"🤖 Attempting agentic correction for gap {i}/{len(gap_sequences)}")
|
||||
try:
|
||||
# Prepare gap data for classification-first workflow
|
||||
gap_words_data = []
|
||||
for word_id in gap.transcribed_word_ids:
|
||||
if word_id in word_map:
|
||||
word = word_map[word_id]
|
||||
gap_words_data.append({
|
||||
"id": word_id,
|
||||
"text": word.text,
|
||||
"start_time": getattr(word, 'start_time', 0),
|
||||
"end_time": getattr(word, 'end_time', 0)
|
||||
})
|
||||
|
||||
# Get context words
|
||||
all_transcribed_words = []
|
||||
for seg in segments:
|
||||
all_transcribed_words.extend(seg.words)
|
||||
word_position = {w.id: idx for idx, w in enumerate(all_transcribed_words)}
|
||||
|
||||
gap_positions = [word_position[wid] for wid in gap.transcribed_word_ids if wid in word_position]
|
||||
preceding_words = ""
|
||||
following_words = ""
|
||||
|
||||
if gap_positions:
|
||||
first_gap_pos = min(gap_positions)
|
||||
last_gap_pos = max(gap_positions)
|
||||
|
||||
# Get 10 words before
|
||||
start_pos = max(0, first_gap_pos - 10)
|
||||
preceding_list = [all_transcribed_words[idx].text for idx in range(start_pos, first_gap_pos) if idx < len(all_transcribed_words)]
|
||||
preceding_words = " ".join(preceding_list)
|
||||
|
||||
# Get 10 words after
|
||||
end_pos = min(len(all_transcribed_words), last_gap_pos + 11)
|
||||
following_list = [all_transcribed_words[idx].text for idx in range(last_gap_pos + 1, end_pos) if idx < len(all_transcribed_words)]
|
||||
following_words = " ".join(following_list)
|
||||
|
||||
# Get reference contexts from all sources
|
||||
reference_contexts = {}
|
||||
for source, lyrics_data in self.reference_lyrics.items():
|
||||
if lyrics_data and lyrics_data.segments:
|
||||
ref_words = []
|
||||
for seg in lyrics_data.segments:
|
||||
ref_words.extend([w.text for w in seg.words])
|
||||
# For now, use full text (handlers will extract relevant portions)
|
||||
reference_contexts[source] = " ".join(ref_words)
|
||||
|
||||
# Get artist and title from metadata
|
||||
artist = metadata.get("artist") if metadata else None
|
||||
title = metadata.get("title") if metadata else None
|
||||
|
||||
# Choose model via router
|
||||
_router = _ModelRouter()
|
||||
uncertainty = 0.3 if len(gap_words_data) <= 2 else 0.7
|
||||
model_id = _router.choose_model("gap", uncertainty)
|
||||
self.logger.debug(f"🤖 Router selected model: {model_id}")
|
||||
|
||||
# Create agent and use new classification-first workflow
|
||||
self.logger.debug(f"🤖 Creating AgenticCorrector with model: {model_id}")
|
||||
_agent = _AgenticCorrector.from_model(
|
||||
model=model_id,
|
||||
session_id=session_id,
|
||||
cache_dir=str(self._cache_dir)
|
||||
)
|
||||
|
||||
# Use new propose_for_gap method
|
||||
self.logger.debug(f"🤖 Calling agent.propose_for_gap() for gap {i}")
|
||||
_proposals = _agent.propose_for_gap(
|
||||
gap_id=f"gap_{i}",
|
||||
gap_words=gap_words_data,
|
||||
preceding_words=preceding_words,
|
||||
following_words=following_words,
|
||||
reference_contexts=reference_contexts,
|
||||
artist=artist,
|
||||
title=title
|
||||
)
|
||||
self.logger.debug(f"🤖 Agent returned {len(_proposals) if _proposals else 0} proposals")
|
||||
_agentic_corrections = _adapt(_proposals, word_map, linear_position_map) if _proposals else []
|
||||
self.logger.debug(f"🤖 Adapter returned {len(_agentic_corrections)} corrections")
|
||||
|
||||
if _agentic_corrections:
|
||||
self.logger.info(f"🤖 Applying {len(_agentic_corrections)} agentic corrections for gap {i}")
|
||||
affected_word_ids = [w.id for w in self._get_affected_words(gap, segments)]
|
||||
affected_segment_ids = [s.id for s in self._get_affected_segments(gap, segments)]
|
||||
updated_segments = self._apply_corrections_to_segments(self._get_affected_segments(gap, segments), _agentic_corrections)
|
||||
for correction in _agentic_corrections:
|
||||
if correction.word_id and correction.corrected_word_id:
|
||||
word_id_map[correction.word_id] = correction.corrected_word_id
|
||||
for old_seg, new_seg in zip(self._get_affected_segments(gap, segments), updated_segments):
|
||||
segment_id_map[old_seg.id] = new_seg.id
|
||||
step = CorrectionStep(
|
||||
handler_name="AgenticCorrector",
|
||||
affected_word_ids=affected_word_ids,
|
||||
affected_segment_ids=affected_segment_ids,
|
||||
corrections=_agentic_corrections,
|
||||
segments_before=self._get_affected_segments(gap, segments),
|
||||
segments_after=updated_segments,
|
||||
created_word_ids=[w.id for w in self._get_new_words(updated_segments, affected_word_ids)],
|
||||
deleted_word_ids=[id for id in affected_word_ids if not self._word_exists(id, updated_segments)],
|
||||
)
|
||||
correction_steps.append(step)
|
||||
all_corrections.extend(_agentic_corrections)
|
||||
# Log corrections made
|
||||
for correction in _agentic_corrections:
|
||||
self.logger.info(
|
||||
f"Made correction: '{correction.original_word}' -> '{correction.corrected_word}' "
|
||||
f"(confidence: {correction.confidence:.2f}, reason: {correction.reason})"
|
||||
)
|
||||
else:
|
||||
self.logger.info(f"🤖 No agentic corrections needed for gap {i}")
|
||||
|
||||
except Exception as e:
|
||||
# In agentic-only mode, fail fast instead of falling back
|
||||
self.logger.error(f"🤖 Agentic correction failed for gap {i}: {e}", exc_info=True)
|
||||
raise RuntimeError(f"Agentic AI correction failed for gap {i}: {e}") from e
|
||||
|
||||
# Skip rule-based handlers completely in agentic mode
|
||||
continue
|
||||
|
||||
# RULE-BASED MODE: Try each handler in order
|
||||
for handler in self.handlers:
|
||||
handler_name = handler.__class__.__name__
|
||||
|
||||
@@ -274,11 +274,12 @@ class CorrectionOperations:
|
||||
audio_filepath: str,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
logger: Optional[logging.Logger] = None
|
||||
logger: Optional[logging.Logger] = None,
|
||||
ass_only: bool = False,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Generate a preview video with current corrections.
|
||||
|
||||
|
||||
Args:
|
||||
correction_result: Current correction result
|
||||
updated_data: Updated correction data for preview
|
||||
@@ -287,10 +288,12 @@ class CorrectionOperations:
|
||||
artist: Optional artist name
|
||||
title: Optional title
|
||||
logger: Optional logger instance
|
||||
|
||||
ass_only: If True, generate only ASS subtitles without video encoding.
|
||||
Useful when video encoding is offloaded to external service.
|
||||
|
||||
Returns:
|
||||
Dict with status, preview_hash, and video_path
|
||||
|
||||
Dict with status, preview_hash, and video_path (or ass_path if ass_only)
|
||||
|
||||
Raises:
|
||||
ValueError: If preview video generation fails
|
||||
"""
|
||||
@@ -338,15 +341,27 @@ class CorrectionOperations:
|
||||
audio_filepath=audio_filepath,
|
||||
artist=artist,
|
||||
title=title,
|
||||
ass_only=ass_only,
|
||||
)
|
||||
|
||||
|
||||
# When ass_only, we only need the ASS file (video encoding done externally)
|
||||
if ass_only:
|
||||
if not preview_outputs.ass:
|
||||
raise ValueError("Preview ASS generation failed")
|
||||
logger.info(f"Generated preview ASS: {preview_outputs.ass}")
|
||||
return {
|
||||
"status": "success",
|
||||
"preview_hash": preview_hash,
|
||||
"ass_path": preview_outputs.ass,
|
||||
}
|
||||
|
||||
if not preview_outputs.video:
|
||||
raise ValueError("Preview video generation failed")
|
||||
|
||||
|
||||
logger.info(f"Generated preview video: {preview_outputs.video}")
|
||||
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"preview_hash": preview_hash,
|
||||
"video_path": preview_outputs.video
|
||||
"video_path": preview_outputs.video,
|
||||
}
|
||||
@@ -2,7 +2,7 @@
|
||||
"name": "lyrics-transcriber-frontend",
|
||||
"private": true,
|
||||
"homepage": "https://nomadkaraoke.github.io/lyrics-transcriber-frontend",
|
||||
"version": "0.83.0",
|
||||
"version": "0.84.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
|
||||
@@ -1 +1 @@
|
||||
{"root":["./src/app.tsx","./src/api.ts","./src/main.tsx","./src/theme.ts","./src/types.ts","./src/validation.ts","./src/vite-env.d.ts","./src/components/aifeedbackmodal.tsx","./src/components/addlyricsmodal.tsx","./src/components/agenticcorrectionmetrics.tsx","./src/components/audioplayer.tsx","./src/components/correctedwordwithactions.tsx","./src/components/correctionannotationmodal.tsx","./src/components/correctiondetailcard.tsx","./src/components/correctionmetrics.tsx","./src/components/durationtimelineview.tsx","./src/components/editactionbar.tsx","./src/components/editmodal.tsx","./src/components/edittimelinesection.tsx","./src/components/editwordlist.tsx","./src/components/fileupload.tsx","./src/components/findreplacemodal.tsx","./src/components/header.tsx","./src/components/lyricsanalyzer.tsx","./src/components/metricsdashboard.tsx","./src/components/modeselectionmodal.tsx","./src/components/modeselector.tsx","./src/components/modelselector.tsx","./src/components/previewvideosection.tsx","./src/components/referenceview.tsx","./src/components/replacealllyricsmodal.tsx","./src/components/reviewchangesmodal.tsx","./src/components/segmentdetailsmodal.tsx","./src/components/timelineeditor.tsx","./src/components/timingoffsetmodal.tsx","./src/components/transcriptionview.tsx","./src/components/worddivider.tsx","./src/components/lyricssynchronizer/synccontrols.tsx","./src/components/lyricssynchronizer/timelinecanvas.tsx","./src/components/lyricssynchronizer/upcomingwordsbar.tsx","./src/components/lyricssynchronizer/index.tsx","./src/components/shared/constants.ts","./src/components/shared/styles.ts","./src/components/shared/types.ts","./src/components/shared/components/highlightedtext.tsx","./src/components/shared/components/sourceselector.tsx","./src/components/shared/components/word.tsx","./src/components/shared/hooks/usewordclick.ts","./src/components/shared/utils/keyboardhandlers.ts","./src/components/shared/utils/localstorage.ts","./src/components/shared/utils/referencelinecalculator.ts","./src/components/shared/utils/segmentoperations.ts","./src/components/shared/utils/timingutils.ts","./src/components/shared/utils/wordutils.ts","./src/hooks/usemanualsync.ts","./src/types/global.d.ts"],"version":"5.6.3"}
|
||||
{"root":["./src/app.tsx","./src/api.ts","./src/main.tsx","./src/theme.ts","./src/types.ts","./src/validation.ts","./src/vite-env.d.ts","./src/components/aifeedbackmodal.tsx","./src/components/addlyricsmodal.tsx","./src/components/agenticcorrectionmetrics.tsx","./src/components/appheader.tsx","./src/components/audioplayer.tsx","./src/components/correctedwordwithactions.tsx","./src/components/correctionannotationmodal.tsx","./src/components/correctiondetailcard.tsx","./src/components/correctionmetrics.tsx","./src/components/durationtimelineview.tsx","./src/components/editactionbar.tsx","./src/components/editmodal.tsx","./src/components/edittimelinesection.tsx","./src/components/editwordlist.tsx","./src/components/fileupload.tsx","./src/components/findreplacemodal.tsx","./src/components/header.tsx","./src/components/lyricsanalyzer.tsx","./src/components/metricsdashboard.tsx","./src/components/modeselectionmodal.tsx","./src/components/modeselector.tsx","./src/components/modelselector.tsx","./src/components/previewvideosection.tsx","./src/components/referenceview.tsx","./src/components/replacealllyricsmodal.tsx","./src/components/reviewchangesmodal.tsx","./src/components/segmentdetailsmodal.tsx","./src/components/timelineeditor.tsx","./src/components/timingoffsetmodal.tsx","./src/components/transcriptionview.tsx","./src/components/worddivider.tsx","./src/components/lyricssynchronizer/synccontrols.tsx","./src/components/lyricssynchronizer/timelinecanvas.tsx","./src/components/lyricssynchronizer/upcomingwordsbar.tsx","./src/components/lyricssynchronizer/index.tsx","./src/components/shared/constants.ts","./src/components/shared/styles.ts","./src/components/shared/types.ts","./src/components/shared/components/highlightedtext.tsx","./src/components/shared/components/sourceselector.tsx","./src/components/shared/components/word.tsx","./src/components/shared/hooks/usewordclick.ts","./src/components/shared/utils/keyboardhandlers.ts","./src/components/shared/utils/localstorage.ts","./src/components/shared/utils/referencelinecalculator.ts","./src/components/shared/utils/segmentoperations.ts","./src/components/shared/utils/timingutils.ts","./src/components/shared/utils/wordutils.ts","./src/hooks/usemanualsync.ts","./src/types/global.d.ts"],"version":"5.6.3"}
|
||||
@@ -3,6 +3,11 @@ import react from '@vitejs/plugin-react';
|
||||
// https://vite.dev/config/
|
||||
export default defineConfig({
|
||||
plugins: [react()],
|
||||
server: {
|
||||
port: 5173,
|
||||
// Enable CORS for local development
|
||||
cors: true,
|
||||
},
|
||||
build: {
|
||||
minify: false,
|
||||
sourcemap: true,
|
||||
|
||||
-1
@@ -1 +0,0 @@
|
||||
{"version":3,"file":"index-BECn1o8Q.js","sources":["../../node_modules/react/cjs/react.production.min.js","../../node_modules/react/index.js","../../node_modules/react/cjs/react-jsx-runtime.production.min.js","../../node_modules/react/jsx-runtime.js","../../node_modules/scheduler/cjs/scheduler.production.min.js","../../node_modules/scheduler/index.js","../../node_modules/react-dom/cjs/react-dom.production.min.js","../../node_modules/react-dom/index.js","../../node_modules/react-dom/client.js","../../node_modules/@mui/utils/esm/formatMuiErrorMessage/formatMuiErrorMessage.js","../../node_modules/@mui/material/styles/identifier.js","../../node_modules/@babel/runtime/helpers/esm/extends.js","../../node_modules/@emotion/sheet/dist/emotion-sheet.esm.js","../../node_modules/stylis/src/Enum.js","../../node_modules/stylis/src/Utility.js","../../node_modules/stylis/src/Tokenizer.js","../../node_modules/stylis/src/Parser.js","../../node_modules/stylis/src/Serializer.js","../../node_modules/stylis/src/Middleware.js","../../node_modules/@emotion/memoize/dist/emotion-memoize.esm.js","../../node_modules/@emotion/cache/dist/emotion-cache.browser.esm.js","../../node_modules/react-is/cjs/react-is.production.min.js","../../node_modules/react-is/index.js","../../node_modules/hoist-non-react-statics/dist/hoist-non-react-statics.cjs.js","../../node_modules/@emotion/utils/dist/emotion-utils.browser.esm.js","../../node_modules/@emotion/hash/dist/emotion-hash.esm.js","../../node_modules/@emotion/unitless/dist/emotion-unitless.esm.js","../../node_modules/@emotion/serialize/dist/emotion-serialize.esm.js","../../node_modules/@emotion/use-insertion-effect-with-fallbacks/dist/emotion-use-insertion-effect-with-fallbacks.browser.esm.js","../../node_modules/@emotion/react/dist/emotion-element-f0de968e.browser.esm.js","../../node_modules/@emotion/react/dist/emotion-react.browser.esm.js","../../node_modules/@emotion/is-prop-valid/dist/emotion-is-prop-valid.esm.js","../../node_modules/@emotion/styled/base/dist/emotion-styled-base.browser.esm.js","../../node_modules/@emotion/styled/dist/emotion-styled.browser.esm.js","../../node_modules/@mui/styled-engine/GlobalStyles/GlobalStyles.js","../../node_modules/@mui/styled-engine/index.js","../../node_modules/@mui/utils/node_modules/react-is/cjs/react-is.production.js","../../node_modules/@mui/utils/node_modules/react-is/index.js","../../node_modules/@mui/utils/esm/deepmerge/deepmerge.js","../../node_modules/@mui/system/esm/createBreakpoints/createBreakpoints.js","../../node_modules/@mui/system/esm/cssContainerQueries/cssContainerQueries.js","../../node_modules/@mui/system/esm/createTheme/shape.js","../../node_modules/@mui/system/esm/merge/merge.js","../../node_modules/@mui/system/esm/breakpoints/breakpoints.js","../../node_modules/@mui/utils/esm/capitalize/capitalize.js","../../node_modules/@mui/system/esm/style/style.js","../../node_modules/@mui/system/esm/memoize/memoize.js","../../node_modules/@mui/system/esm/spacing/spacing.js","../../node_modules/@mui/system/esm/createTheme/createSpacing.js","../../node_modules/@mui/system/esm/compose/compose.js","../../node_modules/@mui/system/esm/borders/borders.js","../../node_modules/@mui/system/esm/cssGrid/cssGrid.js","../../node_modules/@mui/system/esm/palette/palette.js","../../node_modules/@mui/system/esm/sizing/sizing.js","../../node_modules/@mui/system/esm/styleFunctionSx/defaultSxConfig.js","../../node_modules/@mui/system/esm/styleFunctionSx/styleFunctionSx.js","../../node_modules/@mui/system/esm/createTheme/applyStyles.js","../../node_modules/@mui/system/esm/createTheme/createTheme.js","../../node_modules/@mui/system/esm/useThemeWithoutDefault/useThemeWithoutDefault.js","../../node_modules/@mui/system/esm/useTheme/useTheme.js","../../node_modules/@mui/system/esm/GlobalStyles/GlobalStyles.js","../../node_modules/@mui/system/esm/styleFunctionSx/extendSxProp.js","../../node_modules/@mui/utils/esm/ClassNameGenerator/ClassNameGenerator.js","../../node_modules/clsx/dist/clsx.mjs","../../node_modules/@mui/system/esm/createBox/createBox.js","../../node_modules/@mui/utils/esm/generateUtilityClass/generateUtilityClass.js","../../node_modules/@mui/utils/esm/generateUtilityClasses/generateUtilityClasses.js","../../node_modules/@mui/system/esm/preprocessStyles.js","../../node_modules/@mui/system/esm/createStyled/createStyled.js","../../node_modules/@mui/system/esm/styled/styled.js","../../node_modules/@mui/utils/esm/resolveProps/resolveProps.js","../../node_modules/@mui/system/esm/useThemeProps/getThemeProps.js","../../node_modules/@mui/system/esm/useThemeProps/useThemeProps.js","../../node_modules/@mui/utils/esm/useEnhancedEffect/useEnhancedEffect.js","../../node_modules/@mui/system/esm/useMediaQuery/useMediaQuery.js","../../node_modules/@mui/utils/esm/clamp/clamp.js","../../node_modules/@mui/system/esm/colorManipulator/colorManipulator.js","../../node_modules/@mui/utils/esm/createChainedFunction/createChainedFunction.js","../../node_modules/@mui/utils/esm/debLine truncated
|
||||
+5248
-4480
File diff suppressed because it is too large.
Load diff
+1
@@ -0,0 +1 @@
|
||||
{"version":3,"file":"index-V91xgc4E.js","sources":["../../node_modules/react/cjs/react.production.min.js","../../node_modules/react/index.js","../../node_modules/react/cjs/react-jsx-runtime.production.min.js","../../node_modules/react/jsx-runtime.js","../../node_modules/scheduler/cjs/scheduler.production.min.js","../../node_modules/scheduler/index.js","../../node_modules/react-dom/cjs/react-dom.production.min.js","../../node_modules/react-dom/index.js","../../node_modules/react-dom/client.js","../../node_modules/@mui/utils/node_modules/react-is/cjs/react-is.production.js","../../node_modules/@mui/utils/node_modules/react-is/index.js","../../node_modules/@mui/utils/esm/deepmerge/deepmerge.js","../../node_modules/@mui/utils/esm/formatMuiErrorMessage/formatMuiErrorMessage.js","../../node_modules/@mui/utils/esm/capitalize/capitalize.js","../../node_modules/@mui/utils/esm/createChainedFunction/createChainedFunction.js","../../node_modules/@mui/utils/esm/debounce/debounce.js","../../node_modules/@mui/utils/esm/isMuiElement/isMuiElement.js","../../node_modules/@mui/utils/esm/ownerDocument/ownerDocument.js","../../node_modules/@mui/utils/esm/ownerWindow/ownerWindow.js","../../node_modules/@mui/utils/esm/setRef/setRef.js","../../node_modules/@mui/utils/esm/useEnhancedEffect/useEnhancedEffect.js","../../node_modules/@mui/utils/esm/useId/useId.js","../../node_modules/@mui/utils/esm/useControlled/useControlled.js","../../node_modules/@mui/utils/esm/useEventCallback/useEventCallback.js","../../node_modules/@mui/utils/esm/useForkRef/useForkRef.js","../../node_modules/@mui/utils/esm/useLazyRef/useLazyRef.js","../../node_modules/@mui/utils/esm/useOnMount/useOnMount.js","../../node_modules/@mui/utils/esm/useTimeout/useTimeout.js","../../node_modules/@mui/utils/esm/isFocusVisible/isFocusVisible.js","../../node_modules/@mui/utils/esm/getScrollbarSize/getScrollbarSize.js","../../node_modules/@mui/utils/esm/getValidReactChildren/getValidReactChildren.js","../../node_modules/@mui/utils/esm/visuallyHidden/visuallyHidden.js","../../node_modules/@mui/utils/esm/resolveProps/resolveProps.js","../../node_modules/@mui/utils/esm/composeClasses/composeClasses.js","../../node_modules/@mui/utils/esm/ClassNameGenerator/ClassNameGenerator.js","../../node_modules/@mui/utils/esm/generateUtilityClass/generateUtilityClass.js","../../node_modules/@mui/utils/esm/generateUtilityClasses/generateUtilityClasses.js","../../node_modules/@mui/utils/esm/clamp/clamp.js","../../node_modules/@mui/utils/esm/isHostComponent/isHostComponent.js","../../node_modules/@mui/utils/esm/appendOwnerState/appendOwnerState.js","../../node_modules/clsx/dist/clsx.mjs","../../node_modules/@mui/utils/esm/extractEventHandlers/extractEventHandlers.js","../../node_modules/@mui/utils/esm/omitEventHandlers/omitEventHandlers.js","../../node_modules/@mui/utils/esm/mergeSlotProps/mergeSlotProps.js","../../node_modules/@mui/utils/esm/resolveComponentProps/resolveComponentProps.js","../../node_modules/@mui/utils/esm/useSlotProps/useSlotProps.js","../../node_modules/@mui/utils/esm/getReactElementRef/getReactElementRef.js","../../node_modules/@mui/system/esm/merge/merge.js","../../node_modules/@mui/system/esm/cssContainerQueries/cssContainerQueries.js","../../node_modules/@mui/system/esm/breakpoints/breakpoints.js","../../node_modules/@mui/system/esm/style/style.js","../../node_modules/@mui/system/esm/memoize/memoize.js","../../node_modules/@mui/system/esm/spacing/spacing.js","../../node_modules/@mui/system/esm/compose/compose.js","../../node_modules/@mui/system/esm/borders/borders.js","../../node_modules/@mui/system/esm/cssGrid/cssGrid.js","../../node_modules/@mui/system/esm/palette/palette.js","../../node_modules/@mui/system/esm/sizing/sizing.js","../../node_modules/@mui/system/esm/styleFunctionSx/defaultSxConfig.js","../../node_modules/@mui/system/esm/styleFunctionSx/styleFunctionSx.js","../../node_modules/@mui/system/esm/styleFunctionSx/extendSxProp.js","../../node_modules/@babel/runtime/helpers/esm/extends.js","../../node_modules/@emotion/sheet/dist/emotion-sheet.esm.js","../../node_modules/stylis/src/Enum.js","../../node_modules/stylis/src/Utility.js","../../node_modules/stylis/src/Tokenizer.js","../../node_modules/stylis/src/Parser.js","../../node_modules/stylis/src/Serializer.js","../../node_modules/stylis/src/Middleware.js","../../node_modules/@emotion/memoize/dist/emotion-memoize.esm.js","../../node_modules/@emotion/cache/dist/emotion-cache.browser.esm.js","../../node_modules/react-is/cjs/react-is.production.min.js","../../node_modules/react-is/index.js","../../node_modules/hoist-non-react-statics/dist/hoist-non-react-statics.cjs.js","../../node_modules/@emotion/utils/dist/emotion-utils.browser.esm.js","../../node_modules/@emotion/hash/dist/emotion-hash.esm.js","../../node_modules/@emotion/unitless/dist/emotion-unitless.esm.js","../../node_modules/@emotion/serialize/dist/emotion-serialize.esm.js","../../node_modules/@emotion/use-insertion-effect-with-fallbacks/dist/emotion-useLine truncated
|
||||
@@ -2,6 +2,10 @@
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<!-- Load Inter font to match karaoke-gen styling -->
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com">
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
||||
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700&display=swap" rel="stylesheet">
|
||||
<link rel="icon" type="image/x-icon" href="/favicon.ico" />
|
||||
<link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png" />
|
||||
<link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png" />
|
||||
@@ -9,8 +13,8 @@
|
||||
<link rel="icon" type="image/png" sizes="192x192" href="/android-chrome-192x192.png" />
|
||||
<link rel="icon" type="image/png" sizes="512x512" href="/android-chrome-512x512.png" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>Nomad Karaoke: Lyrics Review</title>
|
||||
<script type="module" crossorigin src="/assets/index-BECn1o8Q.js"></script>
|
||||
<title>Nomad Karaoke: Lyrics Transcription Review</title>
|
||||
<script type="module" crossorigin src="/assets/index-V91xgc4E.js"></script>
|
||||
</head>
|
||||
<body>
|
||||
<div id="root"></div>
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" version="1.1" viewBox="45 60 255 135">
|
||||
<g data-v-70b83f88="" fill="#ff7acc" class="basesvg" transform="translate(48.368003845214844,62.62239074707031)"><g fill-rule="" class="tp-name" transform="translate(0,0)"><g transform="scale(1.4000000000000004)"><g stroke="#ff7acc" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" paint-order="stroke" data-gra-attr="stroke" fill-opacity="0"><path d="M1.4-33.44L1.4 0C31.48 0 3.96 0 34.04 0L34.04-32.64 17.81-32.64 17.81-20.61 1.4-33.44ZM52.21-33.76C42.6-33.76 34.81-25.97 34.81-16.37 34.81-6.76 42.6 1.03 52.21 1.03 61.81 1.03 69.6-6.76 69.6-16.37 69.6-25.97 61.81-33.76 52.21-33.76ZM70.7 0L103.34 0 103.25-33.9 86.93-21.54 70.61-33.9 70.7 0ZM102.34 0C138.2 0 107.52 0 143.56 0L122.95-34.93 102.34 0ZM158.84 0.05C167.84 0.05 175.16-7.27 175.16-16.27 175.16-25.27 167.84-32.64 158.84-32.64L142.52-32.64C142.52 0.75 142.52-33.34 142.52 0.05L158.84 0.05Z" transform="translate(-1.399999976158142, 34.93000030517578)"></path></g><g transform="translate(0,38.959999084472656)"><g stroke="#ff7acc" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" paint-order="stroke" data-gra-attr="stroke" fill="#ff7acc" fill-opacity="0" transform="scale(1.02719)"><path d="M24.43 0L17.67 0 7.98-11.87 7.98 0 2.63 0 2.63-26.64 7.98-26.64 7.98-14.69 17.67-26.64 24.12-26.64 13.13-13.44 24.43 0ZM45.03 0L43.27-5.08 32.66-5.08 30.91 0 25.3 0 34.88-26.68 41.1-26.68 50.68 0 45.03 0ZM34.11-9.35L41.82-9.35 37.97-20.5 34.11-9.35ZM73.19 0L67.01 0 61.13-10.38 58.61-10.38 58.61 0 53.27 0 53.27-26.64 63.27-26.64Q66.36-26.64 68.53-25.55 70.71-24.47 71.8-22.61 72.89-20.76 72.89-18.47L72.89-18.47Q72.89-15.84 71.36-13.72 69.83-11.6 66.82-10.8L66.82-10.8 73.19 0ZM58.61-22.21L58.61-14.39 63.08-14.39Q65.25-14.39 66.32-15.44 67.39-16.49 67.39-18.36L67.39-18.36Q67.39-20.19 66.32-21.2 65.25-22.21 63.08-22.21L63.08-22.21 58.61-22.21ZM94.82 0L93.07-5.08 82.46-5.08 80.7 0 75.09 0 84.67-26.68 90.89-26.68 100.47 0 94.82 0ZM83.91-9.35L91.62-9.35 87.76-20.5 83.91-9.35ZM115.43 0.27Q111.69 0.27 108.56-1.49 105.43-3.24 103.6-6.35 101.76-9.47 101.76-13.4L101.76-13.4Q101.76-17.29 103.6-20.4 105.43-23.51 108.56-25.27 111.69-27.02 115.43-27.02L115.43-27.02Q119.21-27.02 122.32-25.27 125.43-23.51 127.24-20.4 129.05-17.29 129.05-13.4L129.05-13.4Q129.05-9.47 127.24-6.35 125.43-3.24 122.3-1.49 119.17 0.27 115.43 0.27L115.43 0.27ZM115.43-4.5Q117.83-4.5 119.66-5.59 121.5-6.68 122.53-8.7 123.56-10.73 123.56-13.4L123.56-13.4Q123.56-16.07 122.53-18.07 121.5-20.08 119.66-21.15 117.83-22.21 115.43-22.21L115.43-22.21Q113.02-22.21 111.17-21.15 109.32-20.08 108.29-18.07 107.26-16.07 107.26-13.4L107.26-13.4Q107.26-10.73 108.29-8.7 109.32-6.68 111.17-5.59 113.02-4.5 115.43-4.5L115.43-4.5ZM153.82 0L147.06 0 137.37-11.87 137.37 0 132.02 0 132.02-26.64 137.37-26.64 137.37-14.69 147.06-26.64 153.51-26.64 142.52-13.44 153.82 0ZM171.79-22.33L161.67-22.33 161.67-15.65 170.64-15.65 170.64-11.41 161.67-11.41 161.67-4.35 171.79-4.35 171.79 0 156.33 0 156.33-26.68 171.79-26.68 171.79-22.33Z" transform="translate(-2.630000114440918, 27.020000457763672)"></path></g></g></g></g> <g data-gra="path-slogan" fill-rule="" class="tp-slogan" fill="#ffdf6b" transform="translate(5,111.788818359375)"><rect x="0" height="1" y="5.9832000732421875" width="14.278396606445312"></rect> <rect height="1" y="5.9832000732421875" width="14.278396606445312" x="218.985595703125"></rect> <g transform="translate(17.278396606445312,0)"><g transform="scale(1.2800000000000002)"><path d="M10.21-8.38L12.02-8.38L9.68 0L7.70 0L6.13-5.96L4.49 0L2.52 0.01L0.26-8.38L2.06-8.38L3.54-1.87L5.24-8.38L7.12-8.38L8.72-1.91L10.21-8.38ZM18.38-8.38L20.06-8.38L20.06 0L18.38 0L18.38-3.56L14.80-3.56L14.80 0L13.12 0L13.12-8.38L14.80-8.38L14.80-4.93L18.38-4.93L18.38-8.38ZM26.58-7.02L23.40-7.02L23.40-4.92L26.22-4.92L26.22-3.59L23.40-3.59L23.40-1.37L26.58-1.37L26.58 0L21.72 0L21.72-8.39L26.58-8.39L26.58-7.02ZM34.37 0L32.42 0L30.58-3.26L29.78-3.26L29.78 0L28.10 0L28.10-8.38L31.25-8.38Q32.22-8.38 32.90-8.03Q33.59-7.69 33.93-7.11Q34.27-6.53 34.27-5.81L34.27-5.81Q34.27-4.98 33.79-4.31Q33.31-3.65 32.36-3.40L32.36-3.40L34.37 0ZM29.78-6.98L29.78-4.52L31.19-4.52Q31.87-4.52 32.21-4.85Q32.54-5.18 32.54-5.77L32.54-5.77Q32.54-6.35 32.21-6.67Q31.87-6.98 31.19-6.98L31.19-6.98L29.78-6.98ZM40.66-7.02L37.48-7.02L37.48-4.92L40.30-4.92L40.30-3.59L37.48-3.59L37.48-1.37L40.66-1.37L40.66 0L35.80 0L35.80-8.39L40.66-8.39L40.66-7.02ZM47.92-8.38L49.70-8.38L46.63 0L44.59 0L41.52-8.38L43.32-8.38L45.62-1.72L47.92-8.38ZM55.57-7.02L52.39-7.02L52.39-4.92L55.21-4.92L55.21-3.59L52.39-3.59L52.39-1.37L55.57-1.37L55.57 0L50.71 0L50.71-8.39L55.57-8.39L55.57-7.02ZM63.36 0L61.42 0L59.57-3.26L58.78-3.26L58.78 0L57.10 0L57.10-8.38L60.24-8.38Q61.21-8.38 61.90-8.03Q62.58-7.69 62.92-7.11Q63.26-6.53 63.26-5.81L63.26-5.81Q63.26-4.98 62.78-4.31Q62.30-3.65 61.36-3.40L61.36-3.40L63.36 0ZM58.78-6.98L58.78-4.52L60.18-4.52Q60.86-4.52 61.20-4.85Q61.54-5.18 61.54-5.77L61.54-5.77Q61.54-6.35 61.20-6.67Q60.86-6.98 60.18-6.98L60.18-6.98L58.78-6.98ZM72.43-8.38L74.30-8.38L71.47-2.92L71.47 0L69.Line truncated
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 9.0 KiB |
@@ -148,8 +148,21 @@ class OutputGenerator:
|
||||
audio_filepath: str,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
ass_only: bool = False,
|
||||
) -> OutputPaths:
|
||||
"""Generate all requested output formats."""
|
||||
"""Generate all requested output formats.
|
||||
|
||||
Args:
|
||||
transcription_corrected: Corrected transcription data
|
||||
lyrics_results: Lyrics data from various providers
|
||||
output_prefix: Prefix for output filenames
|
||||
audio_filepath: Path to audio file
|
||||
artist: Optional artist name
|
||||
title: Optional title
|
||||
ass_only: If True (only in preview_mode), generate only ASS subtitles
|
||||
without video encoding. Useful when video encoding is offloaded
|
||||
to an external service.
|
||||
"""
|
||||
outputs = OutputPaths()
|
||||
|
||||
try:
|
||||
@@ -165,8 +178,9 @@ class OutputGenerator:
|
||||
# Generate ASS subtitles for preview
|
||||
outputs.ass = self.subtitle.generate_ass(transcription_corrected.resized_segments, output_prefix, audio_filepath)
|
||||
|
||||
# Generate preview video
|
||||
outputs.video = self.video.generate_preview_video(outputs.ass, audio_filepath, output_prefix)
|
||||
# Generate preview video (unless ass_only mode for GCE encoding)
|
||||
if not ass_only:
|
||||
outputs.video = self.video.generate_preview_video(outputs.ass, audio_filepath, output_prefix)
|
||||
|
||||
return outputs
|
||||
|
||||
|
||||
@@ -352,19 +352,44 @@ class VideoGenerator:
|
||||
self.logger.error(f"Failed to resize background image: {e.output}")
|
||||
raise
|
||||
|
||||
def _escape_ffmpeg_filter_path(self, path: str) -> str:
|
||||
"""Escape a path for FFmpeg filter expressions (for subprocess without shell).
|
||||
|
||||
When using subprocess with a command list (no shell), FFmpeg receives the
|
||||
filter string directly. FFmpeg's filter parser requires escaping:
|
||||
- Backslashes: double them (\ -> \\)
|
||||
- Single quotes/apostrophes: escape with three backslashes (' -> \\')
|
||||
- Spaces: escape with backslash ( -> \ )
|
||||
|
||||
Note: This is different from shell escaping. The '\\'\\''' pattern used for
|
||||
shell escaping does NOT work when subprocess passes args directly to FFmpeg.
|
||||
|
||||
Example: "I'm With You" becomes "I\\\\'m\\ With\\ You"
|
||||
"""
|
||||
# First escape existing backslashes (\ -> \\)
|
||||
escaped = path.replace("\\", "\\\\")
|
||||
# Escape single quotes (' -> \\')
|
||||
# In the actual string we need 3 backslashes before the quote
|
||||
escaped = escaped.replace("'", "\\\\\\'")
|
||||
# Escape spaces
|
||||
escaped = escaped.replace(" ", "\\ ")
|
||||
return escaped
|
||||
|
||||
def _build_ass_filter(self, ass_path: str) -> str:
|
||||
"""Build ASS filter with font directory support."""
|
||||
ass_filter = f"ass={ass_path}"
|
||||
|
||||
escaped_ass_path = self._escape_ffmpeg_filter_path(ass_path)
|
||||
ass_filter = f"ass={escaped_ass_path}"
|
||||
|
||||
# Get font path from styles configuration
|
||||
karaoke_styles = self.styles.get("karaoke", {})
|
||||
font_path = karaoke_styles.get("font_path")
|
||||
|
||||
|
||||
if font_path and os.path.isfile(font_path):
|
||||
font_dir = os.path.dirname(font_path)
|
||||
ass_filter += f":fontsdir={font_dir}"
|
||||
escaped_font_dir = self._escape_ffmpeg_filter_path(font_dir)
|
||||
ass_filter += f":fontsdir={escaped_font_dir}"
|
||||
self.logger.info(f"Returning ASS filter with fonts dir: {ass_filter}")
|
||||
|
||||
|
||||
return ass_filter
|
||||
|
||||
def _build_ffmpeg_command(self, ass_path: str, audio_path: str, output_path: str) -> List[str]:
|
||||
|
||||
@@ -210,6 +210,7 @@ def test_build_ass_filter_no_font_path(video_generator):
|
||||
"""Test _build_ass_filter with no font path configured."""
|
||||
ass_path = "test.ass"
|
||||
result = video_generator._build_ass_filter(ass_path)
|
||||
# No special chars = no escaping needed
|
||||
assert result == "ass=test.ass"
|
||||
|
||||
|
||||
@@ -220,13 +221,16 @@ def test_build_ass_filter_with_valid_font_path(video_generator, tmp_path):
|
||||
font_dir.mkdir()
|
||||
font_file = font_dir / "test.ttf"
|
||||
font_file.touch()
|
||||
|
||||
|
||||
# Update styles to include font path
|
||||
video_generator.styles["karaoke"]["font_path"] = str(font_file)
|
||||
|
||||
|
||||
ass_path = "test.ass"
|
||||
result = video_generator._build_ass_filter(ass_path)
|
||||
expected = f"ass=test.ass:fontsdir={str(font_dir)}"
|
||||
# Paths with no special chars don't need escaping
|
||||
# But font_dir path from tmp_path may have spaces on some systems
|
||||
escaped_font_dir = str(font_dir).replace("\\", "\\\\").replace("'", "\\\\'").replace(" ", "\\ ")
|
||||
expected = f"ass=test.ass:fontsdir={escaped_font_dir}"
|
||||
assert result == expected
|
||||
|
||||
|
||||
@@ -234,9 +238,10 @@ def test_build_ass_filter_with_invalid_font_path(video_generator):
|
||||
"""Test _build_ass_filter with invalid font path (file doesn't exist)."""
|
||||
# Set non-existent font path
|
||||
video_generator.styles["karaoke"]["font_path"] = "/nonexistent/font.ttf"
|
||||
|
||||
|
||||
ass_path = "test.ass"
|
||||
result = video_generator._build_ass_filter(ass_path)
|
||||
# No special chars = no escaping
|
||||
assert result == "ass=test.ass"
|
||||
|
||||
|
||||
@@ -244,9 +249,10 @@ def test_build_ass_filter_with_empty_font_path(video_generator):
|
||||
"""Test _build_ass_filter with empty font path."""
|
||||
# Set empty font path
|
||||
video_generator.styles["karaoke"]["font_path"] = ""
|
||||
|
||||
|
||||
ass_path = "test.ass"
|
||||
result = video_generator._build_ass_filter(ass_path)
|
||||
# No special chars = no escaping
|
||||
assert result == "ass=test.ass"
|
||||
|
||||
|
||||
@@ -254,9 +260,10 @@ def test_build_ass_filter_with_none_font_path(video_generator):
|
||||
"""Test _build_ass_filter with None font path."""
|
||||
# Set None font path
|
||||
video_generator.styles["karaoke"]["font_path"] = None
|
||||
|
||||
|
||||
ass_path = "test.ass"
|
||||
result = video_generator._build_ass_filter(ass_path)
|
||||
# No special chars = no escaping
|
||||
assert result == "ass=test.ass"
|
||||
|
||||
|
||||
@@ -269,9 +276,10 @@ def test_build_ass_filter_no_karaoke_section(tmp_path):
|
||||
video_resolution=(1920, 1080),
|
||||
styles=styles,
|
||||
)
|
||||
|
||||
|
||||
ass_path = "test.ass"
|
||||
result = generator._build_ass_filter(ass_path)
|
||||
# No special chars = no escaping
|
||||
assert result == "ass=test.ass"
|
||||
|
||||
|
||||
@@ -279,19 +287,73 @@ def test_build_ass_filter_logging(video_generator, tmp_path, caplog):
|
||||
"""Test _build_ass_filter logging behavior."""
|
||||
# Set log level to INFO to capture the logs
|
||||
caplog.set_level(logging.INFO)
|
||||
|
||||
|
||||
# Create a test font file
|
||||
font_dir = tmp_path / "fonts"
|
||||
font_dir.mkdir()
|
||||
font_file = font_dir / "test.ttf"
|
||||
font_file.touch()
|
||||
|
||||
|
||||
# Update styles to include font path
|
||||
video_generator.styles["karaoke"]["font_path"] = str(font_file)
|
||||
|
||||
|
||||
ass_path = "test.ass"
|
||||
result = video_generator._build_ass_filter(ass_path)
|
||||
|
||||
|
||||
# Check that info log was generated
|
||||
assert "Returning ASS filter with fonts dir:" in caplog.text
|
||||
assert str(font_dir) in caplog.text
|
||||
|
||||
|
||||
def test_escape_ffmpeg_filter_path(video_generator):
|
||||
"""Test _escape_ffmpeg_filter_path uses FFmpeg filter escaping (not shell escaping).
|
||||
|
||||
FFmpeg filter escaping rules (for subprocess without shell):
|
||||
- Backslashes: double them (\ -> \\)
|
||||
- Single quotes/apostrophes: escape with three backslashes (' -> \\')
|
||||
- Spaces: escape with backslash ( -> \ )
|
||||
"""
|
||||
# Test simple path without special chars (no escaping needed)
|
||||
assert video_generator._escape_ffmpeg_filter_path("/path/to/file.ass") == "/path/to/file.ass"
|
||||
|
||||
# Test apostrophe escaping (fixes paths like "I'm With You")
|
||||
# Apostrophe becomes \\\' (three backslashes + quote) in the actual string
|
||||
result = video_generator._escape_ffmpeg_filter_path("./Avril Lavigne - I'm With You/file.ass")
|
||||
assert result == "./Avril\\ Lavigne\\ -\\ I\\\\\\'m\\ With\\ You/file.ass"
|
||||
|
||||
# Test path with spaces (escaped with backslash)
|
||||
assert video_generator._escape_ffmpeg_filter_path("/path/with spaces/file.ass") == "/path/with\\ spaces/file.ass"
|
||||
|
||||
# Test path with colons (no escaping needed - colon is only special in filter options)
|
||||
assert video_generator._escape_ffmpeg_filter_path("C:/path/file.ass") == "C:/path/file.ass"
|
||||
|
||||
# Test multiple apostrophes
|
||||
result = video_generator._escape_ffmpeg_filter_path("It's Bob's file.ass")
|
||||
assert result == "It\\\\\\'s\\ Bob\\\\\\'s\\ file.ass"
|
||||
|
||||
|
||||
def test_build_ass_filter_escapes_path_with_apostrophe(video_generator):
|
||||
"""Test _build_ass_filter properly escapes paths containing apostrophes."""
|
||||
ass_path = "./Avril Lavigne - I'm With You/cache/temp.ass"
|
||||
result = video_generator._build_ass_filter(ass_path)
|
||||
# Spaces escaped with backslash, apostrophe escaped with \\\'
|
||||
assert result == "ass=./Avril\\ Lavigne\\ -\\ I\\\\\\'m\\ With\\ You/cache/temp.ass"
|
||||
|
||||
|
||||
def test_build_ass_filter_escapes_font_dir_with_special_chars(video_generator, tmp_path):
|
||||
"""Test _build_ass_filter properly escapes font directory with special characters."""
|
||||
# Create a font directory with special characters
|
||||
font_dir = tmp_path / "It's a Font Dir"
|
||||
font_dir.mkdir()
|
||||
font_file = font_dir / "test.ttf"
|
||||
font_file.touch()
|
||||
|
||||
# Update styles to include font path
|
||||
video_generator.styles["karaoke"]["font_path"] = str(font_file)
|
||||
|
||||
ass_path = "test.ass"
|
||||
result = video_generator._build_ass_filter(ass_path)
|
||||
# Should have fontsdir with escaped path
|
||||
assert "fontsdir=" in result
|
||||
# Apostrophe should be escaped using \\\' pattern (3 backslashes + quote)
|
||||
assert "\\\\\\'s" in result
|
||||
Generated
+467
-454
File diff suppressed because it is too large.
Load diff
+1
-1
@@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "karaoke-gen"
|
||||
version = "0.89.3"
|
||||
version = "0.92.0"
|
||||
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
|
||||
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
|
||||
license = "MIT"
|
||||
|
||||
Executable
+41
@@ -0,0 +1,41 @@
|
||||
#!/bin/bash
|
||||
# Admin API helper script
|
||||
# Usage: ./scripts/admin_api.sh <endpoint> [method] [data]
|
||||
#
|
||||
# Examples:
|
||||
# ./scripts/admin_api.sh /api/jobs # List all jobs (admin sees all)
|
||||
# ./scripts/admin_api.sh /api/jobs?limit=10 # List 10 jobs
|
||||
# ./scripts/admin_api.sh /api/jobs/abc123 # Get specific job
|
||||
# ./scripts/admin_api.sh /api/jobs/abc123/retry POST # Retry a job
|
||||
# ./scripts/admin_api.sh /api/jobs/abc123/select-instrumental POST '{"selection":"clean"}'
|
||||
|
||||
set -e
|
||||
|
||||
API_URL="${KARAOKE_API_URL:-https://api.nomadkaraoke.com}"
|
||||
|
||||
# Get admin token from Secret Manager if not set
|
||||
if [ -z "$KARAOKE_ADMIN_TOKEN" ]; then
|
||||
KARAOKE_ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke 2>/dev/null)
|
||||
fi
|
||||
|
||||
if [ -z "$KARAOKE_ADMIN_TOKEN" ]; then
|
||||
echo "Error: Could not get admin token. Run 'gcloud auth login' first." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ENDPOINT="${1:-/api/admin/jobs}"
|
||||
METHOD="${2:-GET}"
|
||||
DATA="$3"
|
||||
|
||||
if [ "$METHOD" = "GET" ]; then
|
||||
curl -s "${API_URL}${ENDPOINT}" \
|
||||
-H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" | jq .
|
||||
elif [ -n "$DATA" ]; then
|
||||
curl -s -X "$METHOD" "${API_URL}${ENDPOINT}" \
|
||||
-H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "$DATA" | jq .
|
||||
else
|
||||
curl -s -X "$METHOD" "${API_URL}${ENDPOINT}" \
|
||||
-H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" | jq .
|
||||
fi
|
||||
@@ -121,16 +121,25 @@ class TestVideoBackgroundProcessor:
|
||||
processor.get_audio_duration("/path/to/audio.wav")
|
||||
|
||||
def test_escape_filter_path(self, processor):
|
||||
"""Test path escaping for ffmpeg filters."""
|
||||
# Test backslash escaping
|
||||
assert processor.escape_filter_path("C:\\Users\\test\\file.ass") == "C\\:\\\\Users\\\\test\\\\file.ass"
|
||||
|
||||
# Test colon escaping
|
||||
assert processor.escape_filter_path("/path/to/file.ass") == "/path/to/file.ass"
|
||||
assert processor.escape_filter_path("C:/path/file.ass") == "C\\:/path/file.ass"
|
||||
|
||||
# Test combined escaping
|
||||
assert processor.escape_filter_path("C:\\path:with:colons\\file.ass") == "C\\:\\\\path\\:with\\:colons\\\\file.ass"
|
||||
"""Test path escaping for ffmpeg filters using single-quote wrapping."""
|
||||
# Test simple path (wrapped in single quotes)
|
||||
assert processor.escape_filter_path("/path/to/file.ass") == "'/path/to/file.ass'"
|
||||
|
||||
# Test path with spaces (protected by single quotes)
|
||||
assert processor.escape_filter_path("/path/with spaces/file.ass") == "'/path/with spaces/file.ass'"
|
||||
|
||||
# Test apostrophe escaping (fixes paths like "I'm With You")
|
||||
# Single quote becomes '\'' (end quote, escaped quote, start quote)
|
||||
assert processor.escape_filter_path("./Avril Lavigne - I'm With You/file.ass") == "'./Avril Lavigne - I'\\''m With You/file.ass'"
|
||||
|
||||
# Test path with colons (protected by single quotes)
|
||||
assert processor.escape_filter_path("C:/path/file.ass") == "'C:/path/file.ass'"
|
||||
|
||||
# Test Windows-style path (backslashes preserved, wrapped in quotes)
|
||||
assert processor.escape_filter_path("C:\\Users\\test\\file.ass") == "'C:\\Users\\test\\file.ass'"
|
||||
|
||||
# Test multiple apostrophes
|
||||
assert processor.escape_filter_path("It's Bob's file.ass") == "'It'\\''s Bob'\\''s file.ass'"
|
||||
|
||||
def test_build_video_filter_no_darkness(self, processor):
|
||||
"""Test video filter building without darkness overlay."""
|
||||
@@ -139,13 +148,13 @@ class TestVideoBackgroundProcessor:
|
||||
darkness_percent=0,
|
||||
fonts_dir=None
|
||||
)
|
||||
|
||||
|
||||
# Should contain scale and crop
|
||||
assert "scale=w=3840:h=2160:force_original_aspect_ratio=increase,crop=3840:2160" in vf_filter
|
||||
|
||||
# Should contain ASS filter
|
||||
assert "ass=/path/to/subs.ass" in vf_filter
|
||||
|
||||
|
||||
# Should contain ASS filter with single-quoted path
|
||||
assert "ass='/path/to/subs.ass'" in vf_filter
|
||||
|
||||
# Should NOT contain darkening
|
||||
assert "drawbox" not in vf_filter
|
||||
|
||||
@@ -156,10 +165,10 @@ class TestVideoBackgroundProcessor:
|
||||
darkness_percent=50,
|
||||
fonts_dir=None
|
||||
)
|
||||
|
||||
|
||||
# Should contain darkening with correct alpha
|
||||
assert "drawbox=x=0:y=0:w=iw:h=ih:color=black@0.50:t=fill" in vf_filter
|
||||
|
||||
|
||||
# Should contain ASS filter after darkening
|
||||
assert vf_filter.index("drawbox") < vf_filter.index("ass=")
|
||||
|
||||
@@ -171,20 +180,21 @@ class TestVideoBackgroundProcessor:
|
||||
darkness_percent=0,
|
||||
fonts_dir="/path/to/fonts"
|
||||
)
|
||||
|
||||
# Should contain fonts directory in ASS filter
|
||||
assert "fontsdir=/path/to/fonts" in vf_filter
|
||||
|
||||
# Should contain fonts directory in ASS filter (single-quoted)
|
||||
assert "fontsdir='/path/to/fonts'" in vf_filter
|
||||
|
||||
def test_build_video_filter_path_escaping(self, processor):
|
||||
"""Test video filter building with special characters in paths."""
|
||||
# Test with apostrophe in path
|
||||
vf_filter = processor.build_video_filter(
|
||||
ass_subtitles_path="C:\\Users\\test\\subs.ass",
|
||||
ass_subtitles_path="./Avril Lavigne - I'm With You/subs.ass",
|
||||
darkness_percent=0,
|
||||
fonts_dir=None
|
||||
)
|
||||
|
||||
# Paths should be escaped
|
||||
assert "C\\:\\\\Users\\\\test\\\\subs.ass" in vf_filter
|
||||
|
||||
# Paths should be single-quoted with apostrophes escaped as '\''
|
||||
assert "'./Avril Lavigne - I'\\''m With You/subs.ass'" in vf_filter
|
||||
|
||||
@patch('subprocess.run')
|
||||
def test_execute_command_with_fallback_gpu_success(self, mock_run, processor_with_nvenc, mock_logger):
|
||||
|
||||
Reference in new issue
Block a user