Compare commits

...
Author SHA1 Message Date
Andrew Beveridge ☄️ 4fcd899f1b Update pyproject.toml 2026-01-06 01:26:16 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 b345135be1 feat: Unify GCE encoding with LocalEncodingService (single code path) (#214)
## Summary
- Unifies GCE encoding worker with LocalEncodingService (same code as
local CLI)
- Removes all fallback/simplified encoding logic
- Produces proper filenames: `Artist - Title (Final Karaoke Lossless
4k).mp4`
- Includes concatenated title + karaoke + end screens
- CI uploads wheel to GCS for hot code updates without restart

## Problem
Cloud/remote karaoke generation was using simplified, duplicated
encoding logic that:
- Used generic filenames like `output_4k_lossless.mp4`
- Did NOT concatenate title + karaoke + end screens
- Was missing MKV format for YouTube uploads

## Solution
GCE worker now:
1. Downloads latest karaoke-gen wheel from GCS at job start
2. Imports and uses `LocalEncodingService` (same as local CLI)
3. Produces identical output to local karaoke-gen

## Test plan
- [ ] Run a cloud generation job after merging
- [ ] Verify output files have proper names
- [ ] Verify videos include title/end screens
- [ ] Verify Dropbox/GDrive uploads work correctly

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-06 01:25:45 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 1114bd5554 fix: Use FFmpeg filter escaping instead of shell escaping for paths (#213)
## Summary
- Fixes video generation for songs with apostrophes in titles (e.g.,
"I'm With You")
- PR #212 introduced shell-style escaping (`'\''`) which doesn't work
with subprocess command lists
- Changed to FFmpeg's native filter escaping rules

## Test plan
- [x] All 22 video.py unit tests pass
- [x] All 68 backend tests pass  
- [x] End-to-end FFmpeg test with apostrophe in path succeeds
- [ ] Manual test with "Avril Lavigne - I'm With You"

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 23:45:27 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 60af90de31 fix: Escape FFmpeg filter paths using single-quote wrapping (#212)
## Summary
- Fix FFmpeg failing on paths with apostrophes (e.g., "I'm With You")
- Use single-quote wrapping with `'\''` pattern for embedded quotes
- Update both `video.py` and `video_background_processor.py`

## Test plan
- [x] Unit tests updated and passing (1006 tests)
- [x] E2E tests passing (68 tests)
- [x] Manually tested with "Avril Lavigne - I'm With You" - video
generates successfully

## Technical details
FFmpeg filter syntax interprets apostrophes as special characters.
Simple backslash escaping (`\'`) doesn't work because FFmpeg's filter
parser consumes the backslash. The solution is to wrap paths in single
quotes and escape embedded quotes using the `'\''` pattern (end quote,
literal quote, start quote).

Example: `I'm With You` becomes `'I'\''m With You'`

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-06 03:50:24 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 2110d0f84b perf: Optimize agentic correction with parallel gap processing (~5-6x speedup) (#211)
## Summary

- Fix anti-pattern of creating new LLM model instance per gap (caused
repeated 2s warm-up overhead)
- Create agentic corrector once and reuse for all gaps
- Process gaps in parallel using ThreadPoolExecutor (default 5 workers)
- Remove unnecessary warm-up code from langchain_bridge.py
- Add `AGENTIC_MAX_PARALLEL_GAPS` env var to configure parallelism

**Performance improvement**: ~5-6x speedup (20 gaps: ~55s vs ~5+ minutes
previously)

## Test plan

- [x] Ran local test with 20 gaps - completed in ~55 seconds
- [x] Verified parallel processing logs show concurrent gap processing
- [x] Verified model initialization only happens once at start
- [x] CodeRabbit review passed locally

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 22:43:13 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 4b0dd6169f ci: Propagate base image to all runners after rebuild (#210)
## Summary
- Adds new `propagate-base-image` job that runs after base image rebuild
- Spawns 20 parallel jobs to pull the new base image to all self-hosted
runners
- Eliminates ~8 minute pull delay on subsequent app builds

## Problem
When the base image is rebuilt (due to `poetry.lock` or
`Dockerfile.base` changes), other runners still have the old image
cached. The first app build on each runner must pull the new ~5GB image,
adding ~8 minutes to deployment time.

Example:
https://github.com/nomadkaraoke/karaoke-gen/actions/runs/20735879195/job/59532914508

## Solution
After base image is pushed, trigger 20 parallel jobs that pull the new
image to all runners immediately. Future app builds will already have
the latest base image cached.

## Implementation
- Added `base_rebuilt` output to `deploy-backend` job
- New `propagate-base-image` job with matrix strategy (20 parallel jobs)
- `fail-fast: false` so one offline runner doesn't block others
- Uses same Workload Identity auth as other deployment jobs

## Test plan
- [ ] Verify workflow syntax is valid (YAML parses correctly)
- [ ] Merge and observe next base image rebuild triggers propagation
- [ ] Confirm subsequent app builds don't need to pull base image

<!-- @coderabbitai ignore -->

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 21:54:55 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 8e3e3e4713 fix: Check base image hash from registry instead of stale local cache (#209)
## Summary
- Fix GitHub Actions base image cache check to query registry directly
instead of trusting potentially stale local Docker cache
- Prevent unnecessary base image rebuilds caused by runners with
outdated cached images
- Ensure app image is always built on the correct base image

## Changes
- Fetch image hash label via lightweight registry API (~10KB config
blob) instead of pulling full image
- Add step to verify local cache matches registry before building app
image
- Compare config digests consistently for reliable image matching

## Problem
GitHub Actions runners can retain Docker images between runs. When the
runner's local cache has an old base image, the previous logic would:
1. Check local cache first → find old image with wrong hash
2. Conclude base image needs rebuild → unnecessary ~10 min rebuild
3. Or worse: build app image on stale base if hashes happened to match
older state

## Testing
- [x] Verified registry API returns correct hash label
- [x] CodeRabbit CLI review completed locally

## Review
- [x] Local CodeRabbit review completed
- [x] Review feedback addressed (fixed digest comparison types)

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 21:27:02 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 fef9a45739 docs: Add Pulumi CI skip-preview lesson learned (#208)
## Summary
- Add Pulumi CI skip-preview lesson to `LESSONS-LEARNED.md`
- Update `CLAUDE.md` to emphasize infrastructure changes require PRs
- Add detailed Infrastructure section to `DEVELOPMENT.md` documenting:
  - What's managed by Pulumi
  - The PR-based workflow
  - Why not to run `pulumi up` locally

## Changes
| File | Change |
|------|--------|
| `docs/LESSONS-LEARNED.md` | Added "Pulumi CI Deployments Need
Skip-Preview" section |
| `CLAUDE.md` | Strengthened Infrastructure section with PR workflow |
| `docs/DEVELOPMENT.md` | Added comprehensive Infrastructure Changes
section |

## Test plan
- [x] Documentation only - no code changes

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 21:09:45 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 69b87c232a fix: Add IAM permissions for Pulumi to refresh state in CI (#207)
## Summary
- Add `roles/resourcemanager.projectIamAdmin` for IAM binding state
refresh
- Add `roles/secretmanager.admin` for secret state refresh
- Use `--skip-preview` instead of `--refresh` to avoid needing read
permissions on all GCP resource types

The service account would need many additional roles (Cloud Tasks,
Firestore, Monitoring, etc.) for Pulumi to refresh all resources. Using
`--skip-preview` is the standard approach for CI/CD where Pulumi state
is authoritative.

## Test plan
- [ ] CI passes on this PR
- [ ] After merge, Pulumi deployment succeeds

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 18:57:10 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 0530e5f226 fix: Use correct Pulumi stack name in CI infrastructure deploy (#206)
## Summary
- Fixed Pulumi stack reference from `nomadkaraoke/karaoke-gen/prod` to
`nomadkaraoke/karaoke-gen-infrastructure/prod` (matches project name in
`Pulumi.yaml`)
- Added GCP Workload Identity authentication for Pulumi deployment step

## Test plan
- [ ] CI passes on this PR
- [ ] After merge, infrastructure deployment job succeeds

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 15:34:00 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 d8710bc3d4 ci: Add infrastructure deployment job for GCE encoding worker (#205)
## Summary
Adds a Pulumi deployment job that runs on main branch pushes to
auto-deploy infrastructure changes.

## Problem
Infrastructure changes (like GCE encoding worker code) weren't being
deployed automatically. This caused issues where the backend was
deployed with new code but the GCE worker still had old code.

## Solution
New `deploy-infrastructure` job that:
- Runs Pulumi to deploy infrastructure changes
- Restarts the encoding worker service to pick up new code

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 15:07:41 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 c83ac4376c fix: Handle cached preview responses from GCE worker (#204)
## Summary
- Fixes issue where subsequent preview requests hang indefinitely
- GCE worker now returns "cached" status for already-encoded previews
instead of 409 Conflict
- Backend handles cached response by returning immediately without
polling

## Problem
When users clicked preview multiple times without changing lyrics:
1. First preview worked correctly
2. Subsequent previews hung forever because GCE returned 409 Conflict
3. Backend raised an error instead of handling the cached case

## Solution
- GCE `/encode-preview` endpoint now returns `{"status": "cached", ...}`
for completed jobs
- Backend `encode_preview_video()` checks submit response and returns
immediately for cached
- Also handles "in_progress" status when another request is already
encoding

## Test plan
- [x] First preview generates correctly
- [ ] Subsequent previews (unchanged lyrics) return cached video
instantly
- [ ] Changed lyrics generate new preview video

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 19:47:35 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 e70ce2b558 chore: Enable USE_GCE_PREVIEW_ENCODING in production CI (#203)
## Summary
Enable the GCE preview encoding feature in production CI deployments.

## Changes
- Add `USE_GCE_PREVIEW_ENCODING=true` to Cloud Run environment variables
in CI deploy

## Note
To enable immediately (before next deploy), run:
```bash
gcloud run services update karaoke-backend \
  --region us-central1 \
  --project nomadkaraoke \
  --update-env-vars "USE_GCE_PREVIEW_ENCODING=true"
```

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 18:17:40 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 506db844c2 fix: Double-escape backslashes in GCE embedded Python code (#202)
## Summary
Fix syntax error in GCE encoding worker startup script caused by
incorrect backslash escaping.

## Problem
The `escape_ffmpeg_filter_path` function in the GCE worker startup
script had a syntax error:
```
SyntaxError: unexpected character after line continuation character
```

The issue was that backslashes in the embedded Python code weren't
properly escaped for the triple-quoted string context in Pulumi.

## Solution
Double the backslashes in the Pulumi source:
- 4 backslashes in source → 2 in string value → 2 on VM (valid Python)

## Testing
- GCE encoding worker now starts successfully
- Health endpoint responds correctly
- Already deployed via `pulumi up` and verified working

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 02:03:33 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 be04d3efaf feat: Offload preview video encoding to GCE for faster generation (#200)
## Summary
- Move preview video encoding from Cloud Run to GCE encoding worker
- Reduce preview generation time from 60+ seconds to ~15-20 seconds
- Improve UX during lyrics review by dramatically speeding up preview
generation

## Changes
- **GCE Worker** (`infrastructure/__main__.py`): Add `/encode-preview`
endpoint with `EncodePreviewRequest` model, FFmpeg command for 480x270
preview encoding, and FFmpeg filter path escaping
- **EncodingService** (`backend/services/encoding_service.py`): Add
`is_preview_enabled` property, `submit_preview_encoding_job()`, and
`encode_preview_video()` methods
- **Config** (`backend/config.py`): Add `USE_GCE_PREVIEW_ENCODING`
setting (default: false)
- **Review Route** (`backend/api/routes/review.py`): Modify
`generate_preview_video()` to use GCE with fallback to local encoding
- **LyricsTranscriber** (`operations.py`, `generator.py`): Add
`ass_only` parameter to generate ASS subtitles without video encoding
- **Documentation**: Update ARCHITECTURE.md and LESSONS-LEARNED.md

## Technical Approach
The optimization splits preview generation into two phases:
1. **ASS Generation (Cloud Run)** - Fast text processing (~2s)
2. **Video Encoding (GCE)** - Offloaded to C4-standard-8 VM (~15-20s)

If GCE encoding fails or is disabled, falls back to local Cloud Run
encoding.

## Testing
- [x] Unit tests pass (1004 tests)
- [x] Emulator tests pass (68 tests)
- [x] CodeRabbit CLI review completed locally

## Deployment
1. Deploy infrastructure changes (`pulumi up` or restart GCE worker)
2. Deploy backend to Cloud Run
3. Enable via `USE_GCE_PREVIEW_ENCODING=true` in Cloud Run env

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 01:04:35 -05:00
Andrew Beveridge ☄️andClaude Opus 4.5 f2639754a9 fix: Download GCE-encoded files locally for YouTube upload (#201)
## Summary
- Fix YouTube upload failing when using GCE encoding backend
- GCE encoding returns GCS blob paths, but YouTube upload expected local
files
- Added `_download_gce_encoded_files()` to download encoded videos from
GCS before distribution

## Root Cause
Job `62a95e01` completed but didn't upload to YouTube. Investigation
showed:
1. GCE encoding stores files in GCS and returns blob paths like
`jobs/{id}/finals/output_4k_lossless.mp4`
2. The orchestrator stored these paths directly without downloading
3. YouTube upload checks `os.path.isfile()` which returns `False` for
GCS paths
4. Result: "No video file available for YouTube upload" warning

## Changes
| File | Change |
|------|--------|
| `backend/workers/video_worker_orchestrator.py` | Added
`_download_gce_encoded_files()` method called after GCE encoding |
| `backend/tests/test_video_worker_orchestrator.py` | Added 2 tests for
GCE download behavior |
| `docs/LESSONS-LEARNED.md` | Documented the issue and fix |

## Test plan
- [x] All 30 orchestrator tests pass
- [x] New test `test_run_encoding_gce_downloads_files` verifies GCE
paths are downloaded
- [x] New test `test_run_encoding_local_does_not_download` verifies
local encoding skips download
- [ ] Manual test: Create job via web UI and verify YouTube upload works

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 05:51:52 +00:00
Andrew Beveridge ☄️ 94182277fa Update pyproject.toml (#199) 2026-01-05 04:14:37 +00:00
Andrew Beveridge ☄️andClaude Opus 4.5 e16da0fa5f fix: Allow explicit override of YouTube upload server default (#198)
## Summary
- Changed `enable_youtube_upload` from `bool=False` to
`Optional[bool]=None` in request models
- Allows explicit `false` to override server default when
`DEFAULT_ENABLE_YOUTUBE_UPLOAD=true`
- E2E tests now intercept requests to disable YouTube upload for test
jobs

## Changes
- **Request models**: Changed to `Optional[bool] = None` to support
three states (None/True/False)
- **API routes**: All job creation endpoints now compute effective value
using `if is not None else` pattern
- **E2E test**: Added Playwright request interception to set
`enable_youtube_upload=false` for test jobs
- **Tests**: Updated to expect `None` instead of `False` for request
model defaults
- **Docs**: Added lesson learned about Python `or` operator gotcha with
boolean defaults

## Testing
- [x] Backend tests pass (`make test`)
- [x] No TypeScript errors in frontend changes

## Review
- [x] CodeRabbit CLI review completed locally
- [x] No issues found

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 23:04:06 -05:00
Andrew Beveridge ☄️andClaude b8e15b2af5 fix: Ensure user data is fetched after client-side navigation to /app (#197)
## Summary

- Fix E2E happy path test failure at Step 3 (Verify Credits)
- After beta enrollment, client-side navigation to `/app` didn't fetch
user data
- Credits weren't displayed because zustand auth store had `user: null`

## Changes

- **frontend/app/app/page.tsx**: Added `useEffect` that calls
`fetchUser()` when there's a token but no user data loaded. This ensures
user data is fetched regardless of how the user navigated to the page
(page reload vs client-side navigation).

- **frontend/e2e/regression/authentication.spec.ts**: Added new test
`'beta enrollment shows credits after redirect to app'` that
specifically verifies credits are displayed after the beta enrollment
flow.

- **docs/LESSONS-LEARNED.md**: Documented the root cause (zustand
module-level initialization only runs once, not on client-side
navigation) for future reference.

## Root Cause

The auth module's initialization code (`fetchUser()` call) only runs
once when the module is first imported. After beta enrollment,
`router.push('/app')` performs client-side navigation which doesn't
reload modules. The user data was never fetched, leaving `user: null` in
the store.

## Testing

- [x] New regression test passes locally (9/9 auth tests pass)
- [x] Build succeeds
- [x] CodeRabbit CLI review completed locally

@coderabbitai ignore

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-04 21:33:01 -05:00
Andrew Beveridge ☄️ 2ee3cabc92 Added admin API script (#196) 2026-01-05 01:33:17 +00:00
44 changed files with 7854 additions and 5418 deletions

No files matched your search

+197 -21
View File
@@ -903,11 +903,14 @@ jobs:
deploy-backend:
name: "Deploy - Backend (Cloud Run)"
needs: [package-build]
runs-on: [self-hosted, linux, gcp]
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
permissions:
contents: read
id-token: write
outputs:
base_rebuilt: ${{ steps.check-base.outputs.needs_rebuild }}
env:
REGISTRY: us-central1-docker.pkg.dev
PROJECT_ID: nomadkaraoke
@@ -965,6 +968,32 @@ jobs:
with:
version: "latest"
# ============================================
# Upload wheel to GCS for GCE encoding worker
# The GCE worker installs this wheel at job start for hot code updates
# ============================================
- name: Download wheel artifact
uses: actions/download-artifact@v4
with:
name: package-dist
path: dist/
- name: Upload wheel to GCS for GCE worker
run: |
echo "📦 Uploading wheel to GCS for GCE encoding worker..."
WHEEL_FILE=$(ls dist/*.whl | head -1)
if [ -z "$WHEEL_FILE" ]; then
echo "::error::No wheel file found in dist/"
exit 1
fi
echo "Found wheel: $WHEEL_FILE"
# Upload to GCS (GCE worker downloads latest at job start)
gsutil cp "$WHEEL_FILE" gs://karaoke-gen-storage-${{ env.PROJECT_ID }}/wheels/
echo "✅ Wheel uploaded - GCE worker will pick up changes on next job"
- name: Configure Docker for Artifact Registry
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
@@ -990,32 +1019,32 @@ jobs:
run: |
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
# Try to get the existing base image's hash label
# First check if image exists locally (fast, no network call)
# Only pull from registry if not cached locally
# Fetch hash label directly from registry (10KB config blob, not full image)
# This avoids issues with stale local Docker cache on GitHub runners
EXISTING_HASH=""
if docker image inspect $BASE_IMAGE >/dev/null 2>&1; then
echo "✅ Base image found in local cache (skipping slow registry pull)"
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
# Fallback to old label name for backwards compatibility
if [ -z "$EXISTING_HASH" ]; then
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "poetry.lock.hash"}}' 2>/dev/null || echo "")
fi
echo "Existing base image hash: $EXISTING_HASH"
elif docker pull $BASE_IMAGE 2>/dev/null; then
echo "📥 Base image pulled from registry"
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
# Fallback to old label name for backwards compatibility
if [ -z "$EXISTING_HASH" ]; then
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "poetry.lock.hash"}}' 2>/dev/null || echo "")
fi
echo "Existing base image hash: $EXISTING_HASH"
# Get the manifest (small JSON, ~1KB) - includes config digest
MANIFEST=$(docker manifest inspect $BASE_IMAGE 2>/dev/null || echo "")
CONFIG_DIGEST=$(echo "$MANIFEST" | jq -r '.config.digest // ""' 2>/dev/null || echo "")
if [ -n "$CONFIG_DIGEST" ] && [ "$CONFIG_DIGEST" != "null" ] && [ "$CONFIG_DIGEST" != "" ]; then
echo "📋 Found manifest, config digest: $CONFIG_DIGEST"
# Fetch just the config blob (~10KB) to get labels - much faster than pulling full image
TOKEN=$(gcloud auth print-access-token)
CONFIG_JSON=$(curl -sL -H "Authorization: Bearer $TOKEN" \
"https://${{ env.REGISTRY }}/v2/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}/blobs/$CONFIG_DIGEST" 2>/dev/null)
EXISTING_HASH=$(echo "$CONFIG_JSON" | jq -r '.config.Labels["base.content.hash"] // .config.Labels["poetry.lock.hash"] // ""' 2>/dev/null || echo "")
echo "📋 Registry base image hash: $EXISTING_HASH"
else
echo "⚠️ No existing base image found - will build"
echo "⚠️ No existing base image found in registry - will build"
fi
if [ "${{ steps.base-hash.outputs.hash }}" != "$EXISTING_HASH" ]; then
echo "🔄 Base image needs rebuild (poetry.lock or Dockerfile.base changed)"
echo " Current: ${{ steps.base-hash.outputs.hash }}"
echo " Registry: $EXISTING_HASH"
echo "needs_rebuild=true" >> $GITHUB_OUTPUT
else
echo "✅ Base image is up to date"
@@ -1043,6 +1072,31 @@ jobs:
# Provenance disabled for smaller image and faster push
provenance: false
- name: Ensure local base image is current (if not rebuilt)
if: steps.check-base.outputs.needs_rebuild == 'false'
run: |
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
# Get the config digest from registry manifest (identifies the image content)
# This is the same value as Docker's Image ID for the same image
REGISTRY_CONFIG_DIGEST=$(docker manifest inspect $BASE_IMAGE 2>/dev/null | jq -r '.config.digest' || echo "")
# Get local image's ID (which is the config digest)
LOCAL_IMAGE_ID=$(docker image inspect $BASE_IMAGE --format='{{.Id}}' 2>/dev/null || echo "")
echo "Registry config digest: $REGISTRY_CONFIG_DIGEST"
echo "Local image ID: $LOCAL_IMAGE_ID"
if [ -z "$LOCAL_IMAGE_ID" ]; then
echo "📥 Base image not in local cache - pulling from registry"
docker pull $BASE_IMAGE
elif [ "$LOCAL_IMAGE_ID" != "$REGISTRY_CONFIG_DIGEST" ]; then
echo "🔄 Local cache has different image - pulling latest"
docker pull $BASE_IMAGE
else
echo "✅ Local cache matches registry - skipping pull"
fi
- name: Build and push app image
uses: docker/build-push-action@v6
with:
@@ -1085,7 +1139,7 @@ jobs:
--concurrency 1 \
--max-instances 50 \
--min-instances 4 \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_GCE_ENCODING=true" \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true" \
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,SENDGRID_API_KEY=sendgrid-api-key:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest"
- name: Verify deployment and version
@@ -1139,3 +1193,125 @@ jobs:
echo ""
echo "📊 Built on self-hosted GCP runner (200GB SSD, no disk space issues)"
echo " Runner: gcp-runner with 4 vCPU, 16GB RAM"
# ============================================
# PROPAGATE BASE IMAGE TO ALL RUNNERS
# When base image is rebuilt, pull it to all 20 runners
# so subsequent app builds don't wait 8 min for the pull
#
# NOTE: This is best-effort distribution. GitHub Actions doesn't
# support targeting specific runners by name, so we spawn 20 jobs
# and rely on the scheduler to distribute them across idle runners.
# In practice, with all runners idle after deploy-backend, distribution
# is good. Runners that miss this pull will get the image on their
# next app build (with the 8 min delay).
# ============================================
propagate-base-image:
name: "Propagate Base Image (${{ matrix.runner_index }})"
needs: deploy-backend
if: github.ref == 'refs/heads/main' && github.event_name == 'push' && needs.deploy-backend.outputs.base_rebuilt == 'true'
runs-on: [self-hosted, linux, gcp]
permissions:
contents: read
id-token: write
strategy:
matrix:
# 20 parallel jobs - GitHub distributes across available runners
runner_index: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
max-parallel: 20
fail-fast: false # Don't cancel other runners if one fails
env:
REGISTRY: us-central1-docker.pkg.dev
PROJECT_ID: nomadkaraoke
REPOSITORY: karaoke-repo
BASE_IMAGE_NAME: karaoke-backend-base
steps:
- name: Authenticate to Google Cloud (Workload Identity)
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
with:
version: "latest"
- name: Configure Docker for Artifact Registry
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
- name: Pull latest base image
run: |
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
RUNNER_NAME=$(hostname)
echo "📥 Pulling latest base image on runner $RUNNER_NAME..."
docker pull $BASE_IMAGE
echo "✅ Base image cached on $RUNNER_NAME"
deploy-infrastructure:
name: "Deploy - Infrastructure (Pulumi)"
runs-on: [self-hosted, linux, gcp]
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
permissions:
contents: read
id-token: write
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Authenticate to Google Cloud (Workload Identity)
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
with:
version: "latest"
- name: Set up Python 3.13
uses: actions/setup-python@v5
with:
python-version: '3.13'
- name: Install Pulumi
run: |
curl -fsSL https://get.pulumi.com | sh
echo "$HOME/.pulumi/bin" >> $GITHUB_PATH
- name: Install Python dependencies
working-directory: infrastructure
run: |
pip install pulumi pulumi-gcp pulumi-docker
- name: Configure Pulumi
env:
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
run: |
pulumi login
- name: Deploy Infrastructure
working-directory: infrastructure
env:
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
run: |
echo "🚀 Deploying infrastructure changes..."
# Skip preview to avoid needing broad read permissions on all GCP resources
# Pulumi state is authoritative since infrastructure is only managed via Pulumi
pulumi up --yes --skip-preview --stack nomadkaraoke/karaoke-gen-infrastructure/prod
- name: Restart GCE Encoding Worker
run: |
echo "🔄 Restarting encoding worker to pick up new code..."
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
--command="sudo systemctl restart encoding-worker" || true
echo "✅ Encoding worker restart triggered"
- name: Summary
if: success()
run: |
echo "🎉 Infrastructure deployment successful!"
echo "Commit: ${{ github.sha }}"
echo "Branch: ${{ github.ref_name }}"
+4 -2
View File
@@ -48,8 +48,10 @@ cd frontend && npm run test:all 2>&1 | tail -n 200
- Skip for docs-only changes
### Infrastructure
- All GCP changes via Pulumi in `infrastructure/`
- `gcloud` CLI for reading/debugging only
- **All GCP changes via Pulumi PRs** - changes in `infrastructure/` deploy automatically on merge to main
- Never modify GCP resources directly via console or `gcloud` CLI
- `gcloud` CLI for reading/debugging only (e.g., checking logs, SSH to VMs)
- Local `pulumi preview` works but `pulumi up` requires broader permissions than CI
- Stop and notify user on auth issues
## API Authentication (for Agents)
+3 -2
View File
@@ -95,7 +95,7 @@ class AudioSearchRequest(BaseModel):
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
@@ -485,7 +485,8 @@ async def search_audio(
effective_discord_webhook_url = body.discord_webhook_url or settings.default_discord_webhook_url
# Apply defaults for YouTube/Dropbox distribution (for web service)
effective_enable_youtube_upload = body.enable_youtube_upload or settings.default_enable_youtube_upload
# Use explicit value if provided, otherwise fall back to server default
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
effective_brand_prefix = body.brand_prefix or settings.default_brand_prefix
effective_youtube_description = body.youtube_description or settings.default_youtube_description
+36 -13
View File
@@ -69,7 +69,7 @@ class CreateJobFromUrlRequest(BaseModel):
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
webhook_url: Optional[str] = Field(None, description="Generic webhook URL")
@@ -123,7 +123,7 @@ class CreateJobWithUploadUrlsRequest(BaseModel):
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
webhook_url: Optional[str] = Field(None, description="Generic webhook URL")
@@ -406,7 +406,7 @@ async def upload_and_create_job(
enable_txt: Optional[bool] = Form(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise"),
# Finalisation options
brand_prefix: Optional[str] = Form(None, description="Brand code prefix (e.g., NOMAD)"),
enable_youtube_upload: bool = Form(False, description="Upload to YouTube"),
enable_youtube_upload: Optional[bool] = Form(None, description="Upload to YouTube. None = use server default"),
youtube_description: Optional[str] = Form(None, description="YouTube video description text"),
discord_webhook_url: Optional[str] = Form(None, description="Discord webhook URL for notifications"),
webhook_url: Optional[str] = Form(None, description="Generic webhook URL"),
@@ -510,12 +510,19 @@ async def upload_and_create_job(
if dist.brand_prefix and not brand_prefix:
logger.info(f"Using default brand_prefix: {dist.brand_prefix}")
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = enable_youtube_upload if enable_youtube_upload is not None else settings.default_enable_youtube_upload
if effective_enable_youtube_upload and enable_youtube_upload is None:
logger.info("Using default enable_youtube_upload: True (from env)")
# Validate credentials for requested distribution services (including defaults)
# This prevents accepting jobs that will fail later due to missing credentials
invalid_services = []
credential_manager = get_credential_manager()
if enable_youtube_upload:
if effective_enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -594,7 +601,7 @@ async def upload_and_create_job(
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=dist.brand_prefix,
enable_youtube_upload=enable_youtube_upload,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=youtube_description,
discord_webhook_url=dist.discord_webhook_url,
webhook_url=webhook_url,
@@ -815,11 +822,12 @@ async def upload_and_create_job(
"using_default": gdrive_folder_id is None,
}
if enable_youtube_upload:
if effective_enable_youtube_upload:
youtube_result = credential_manager.check_youtube_credentials()
distribution_services["youtube"] = {
"enabled": True,
"credentials_valid": youtube_result.status == CredentialStatus.VALID,
"using_default": enable_youtube_upload is None,
}
if dist.discord_webhook_url:
@@ -908,7 +916,7 @@ class CreateFinaliseOnlyJobRequest(BaseModel):
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
keep_brand_code: Optional[str] = Field(None, description="Preserve existing brand code from folder name")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
@@ -1048,11 +1056,16 @@ async def create_job_with_upload_urls(
brand_prefix=body.brand_prefix,
)
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Validate credentials for requested distribution services
invalid_services = []
credential_manager = get_credential_manager()
if body.enable_youtube_upload:
if effective_enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -1105,7 +1118,7 @@ async def create_job_with_upload_urls(
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=dist.brand_prefix,
enable_youtube_upload=body.enable_youtube_upload,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=body.youtube_description,
youtube_description_template=body.youtube_description, # video_worker reads this field
discord_webhook_url=dist.discord_webhook_url,
@@ -1466,11 +1479,16 @@ async def create_job_from_url(
brand_prefix=body.brand_prefix,
)
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Validate credentials for requested distribution services
invalid_services = []
credential_manager = get_credential_manager()
if body.enable_youtube_upload:
if effective_enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -1522,7 +1540,7 @@ async def create_job_from_url(
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=dist.brand_prefix,
enable_youtube_upload=body.enable_youtube_upload,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=body.youtube_description,
discord_webhook_url=dist.discord_webhook_url,
webhook_url=body.webhook_url,
@@ -1712,11 +1730,16 @@ async def create_finalise_only_job(
brand_prefix=body.brand_prefix,
)
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Validate distribution credentials if services are requested
invalid_services = []
credential_manager = get_credential_manager()
if body.enable_youtube_upload:
if effective_enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -1766,7 +1789,7 @@ async def create_finalise_only_job(
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=dist.brand_prefix,
enable_youtube_upload=body.enable_youtube_upload,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=body.youtube_description,
discord_webhook_url=dist.discord_webhook_url,
dropbox_path=dist.dropbox_path,
+6 -1
View File
@@ -87,6 +87,11 @@ async def create_job(
user_email = request.user_email
logger.info(f"Admin {auth_result.user_email} creating job on behalf of {user_email}")
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = request.enable_youtube_upload if request.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Create job with all preferences
job_create = JobCreate(
url=str(request.url),
@@ -94,7 +99,7 @@ async def create_job(
title=request.title,
enable_cdg=request.enable_cdg,
enable_txt=request.enable_txt,
enable_youtube_upload=request.enable_youtube_upload,
enable_youtube_upload=effective_enable_youtube_upload,
youtube_description=request.youtube_description,
webhook_url=request.webhook_url,
user_email=user_email
+122 -40
View File
@@ -27,8 +27,10 @@ from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.job_logging import job_log_context, JobLogger
from backend.services.tracing import create_span, add_span_attribute, add_span_event
from backend.services.encoding_service import get_encoding_service
from backend.api.dependencies import require_auth, require_review_auth
from backend.services.auth_service import UserType
from backend.config import get_settings
# LyricsTranscriber imports for preview generation
from lyrics_transcriber.types import CorrectionResult
@@ -385,28 +387,35 @@ async def generate_preview_video(
):
"""
Generate a preview video with the current corrections.
Uses the LyricsTranscriber's CorrectionOperations to generate a 360p preview
video with the user's current corrections applied.
When USE_GCE_PREVIEW_ENCODING is enabled, video encoding is offloaded to
the high-performance GCE worker for faster generation (15-20s vs 60+s).
"""
job_manager = JobManager()
storage = StorageService()
settings = get_settings()
encoding_service = get_encoding_service()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
# Job must be in review state to generate preview
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
raise HTTPException(
status_code=400,
detail=f"Job not in review state (current status: {job.status})"
)
logger.info(f"Job {job_id}: Generating preview video")
# Check if GCE preview encoding is enabled
use_gce_preview = encoding_service.is_preview_enabled
logger.info(f"Job {job_id}: Generating preview video (GCE preview: {use_gce_preview})")
# Use tracing and job_log_context for full observability
with create_span("generate-preview-video", {"job_id": job_id}) as span:
with create_span("generate-preview-video", {"job_id": job_id, "use_gce": use_gce_preview}) as span:
with job_log_context(job_id, worker="preview"):
try:
# Create temp directory for this preview operation
@@ -416,19 +425,19 @@ async def generate_preview_video(
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
corrections_path = os.path.join(temp_dir, "corrections.json")
storage.download_file(corrections_gcs, corrections_path)
with open(corrections_path, 'r', encoding='utf-8') as f:
original_data = json.load(f)
# 2. Download input audio
audio_path = os.path.join(temp_dir, "audio.flac")
storage.download_file(job.input_media_gcs_path, audio_path)
download_span.set_attribute("audio_gcs_path", job.input_media_gcs_path)
# 3. Load original as CorrectionResult
correction_result = CorrectionResult.from_dict(original_data)
add_span_event("corrections_loaded")
# 4. Get or create styles file for preview using unified style loader
with create_span("load-styles") as styles_span:
styles_path, _ = load_styles_from_gcs(
@@ -439,54 +448,127 @@ async def generate_preview_video(
logger=logger,
)
styles_span.set_attribute("styles_path", styles_path)
# 5. Set up output config for preview
output_dir = os.path.join(temp_dir, "output")
cache_dir = os.path.join(temp_dir, "cache")
os.makedirs(output_dir, exist_ok=True)
os.makedirs(cache_dir, exist_ok=True)
output_config = OutputConfig(
output_styles_json=styles_path,
output_dir=output_dir,
cache_dir=cache_dir,
video_resolution="360p",
)
# 6. Generate preview video using CorrectionOperations (heavy operation)
with create_span("render-preview-video") as render_span:
render_span.set_attribute("resolution", "360p")
result = CorrectionOperations.generate_preview_video(
correction_result=correction_result,
updated_data=updated_data,
output_config=output_config,
audio_filepath=audio_path,
artist=job.artist,
title=job.title,
logger=logger
)
add_span_event("render_complete")
# 7. Upload preview video to GCS for serving
preview_hash = result["preview_hash"]
video_path = result["video_path"]
with create_span("upload-preview-video") as upload_span:
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
storage.upload_file(video_path, preview_gcs_path)
upload_span.set_attribute("gcs_path", preview_gcs_path)
# 6. Generate preview (ASS-only if using GCE, or full video if local)
preview_gcs_path = None
if use_gce_preview:
# GCE path: Generate ASS only, then offload encoding to GCE
try:
with create_span("generate-ass-subtitles") as ass_span:
result = CorrectionOperations.generate_preview_video(
correction_result=correction_result,
updated_data=updated_data,
output_config=output_config,
audio_filepath=audio_path,
artist=job.artist,
title=job.title,
logger=logger,
ass_only=True, # Only generate ASS, skip video encoding
)
preview_hash = result["preview_hash"]
ass_path = result["ass_path"]
ass_span.set_attribute("ass_path", ass_path)
add_span_event("ass_generated")
# Upload ASS to GCS
with create_span("upload-ass-to-gcs") as upload_ass_span:
ass_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.ass"
storage.upload_file(ass_path, ass_gcs_path)
upload_ass_span.set_attribute("ass_gcs_path", ass_gcs_path)
# Call GCE encoding service
with create_span("gce-preview-encoding") as gce_span:
bucket_name = settings.gcs_bucket_name
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
# Get background image and font from style assets if available
style_assets = job.style_assets or {}
background_image_gcs_path = None
for key in ["karaoke_background", "style_karaoke_background"]:
if key in style_assets:
background_image_gcs_path = f"gs://{bucket_name}/{style_assets[key]}"
gce_span.set_attribute("background_image", background_image_gcs_path)
break
font_gcs_path = None
for key in ["font", "style_font"]:
if key in style_assets:
font_gcs_path = f"gs://{bucket_name}/{style_assets[key]}"
gce_span.set_attribute("font", font_gcs_path)
break
gce_result = await encoding_service.encode_preview_video(
job_id=f"preview_{job_id}_{preview_hash}",
ass_gcs_path=f"gs://{bucket_name}/{ass_gcs_path}",
audio_gcs_path=f"gs://{bucket_name}/{job.input_media_gcs_path}",
output_gcs_path=f"gs://{bucket_name}/{preview_gcs_path}",
background_color="black",
background_image_gcs_path=background_image_gcs_path,
font_gcs_path=font_gcs_path,
)
gce_span.set_attribute("gce_status", gce_result.get("status"))
add_span_event("gce_encoding_complete")
logger.info(f"Job {job_id}: Preview generated via GCE: {preview_hash}")
except Exception as gce_error:
# Fall back to local encoding if GCE fails
logger.warning(
f"Job {job_id}: GCE preview encoding failed, falling back to local: {gce_error}"
)
span.set_attribute("gce_fallback", True)
use_gce_preview = False # Fall through to local encoding below
if not use_gce_preview:
# Local path: Generate full preview video locally
with create_span("render-preview-video-local") as render_span:
render_span.set_attribute("resolution", "360p")
result = CorrectionOperations.generate_preview_video(
correction_result=correction_result,
updated_data=updated_data,
output_config=output_config,
audio_filepath=audio_path,
artist=job.artist,
title=job.title,
logger=logger,
ass_only=False, # Generate full video locally
)
preview_hash = result["preview_hash"]
video_path = result["video_path"]
add_span_event("render_complete")
# Upload preview video to GCS
with create_span("upload-preview-video") as upload_span:
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
storage.upload_file(video_path, preview_gcs_path)
upload_span.set_attribute("gcs_path", preview_gcs_path)
# Store the GCS path for serving
if job_id not in _preview_videos:
_preview_videos[job_id] = {}
_preview_videos[job_id][preview_hash] = preview_gcs_path
logger.info(f"Job {job_id}: Preview video generated: {preview_hash}")
span.set_attribute("preview_hash", preview_hash)
span.set_attribute("success", True)
return {"status": "success", "preview_hash": preview_hash}
except Exception as e:
logger.error(f"Job {job_id}: Failed to generate preview video: {e}", exc_info=True)
span.set_attribute("error", str(e))
+6
View File
@@ -68,6 +68,12 @@ class Settings(BaseSettings):
use_gce_encoding: bool = os.getenv("USE_GCE_ENCODING", "false").lower() in ("true", "1", "yes")
encoding_worker_url: Optional[str] = os.getenv("ENCODING_WORKER_URL") # e.g., http://136.119.50.148:8080
encoding_worker_api_key: Optional[str] = os.getenv("ENCODING_WORKER_API_KEY")
# GCE Preview Encoding (for faster preview video generation)
# When enabled, preview video encoding during lyrics review is offloaded to the GCE worker.
# This reduces preview generation time from 60+ seconds to ~15-20 seconds.
# Requires use_gce_encoding to be enabled and the GCE worker to support /encode-preview endpoint.
use_gce_preview_encoding: bool = os.getenv("USE_GCE_PREVIEW_ENCODING", "false").lower() in ("true", "1", "yes")
# Storage paths
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
+2 -2
View File
@@ -14,7 +14,7 @@ class URLSubmissionRequest(BaseModel):
# Optional preferences
enable_cdg: bool = False # Requires style config
enable_txt: bool = False # Requires style config
enable_youtube_upload: bool = False
enable_youtube_upload: Optional[bool] = None # None = use server default
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
@@ -28,7 +28,7 @@ class UploadSubmissionRequest(BaseModel):
# Optional preferences
enable_cdg: bool = False # Requires style config
enable_txt: bool = False # Requires style config
enable_youtube_upload: bool = False
enable_youtube_upload: Optional[bool] = None # None = use server default
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
+11 -8
View File
@@ -351,8 +351,8 @@ class GCEEncodingBackend(EncodingBackend):
result = {}
raw_output_files = result.get("output_files", {})
# Convert output_files from list of paths to dict if needed
# GCE worker returns list like: ["path/output_4k_lossless.mp4", "path/output_720p.mp4"]
# Convert output_files from list of paths to dict
# GCE worker returns list like: ["path/Artist - Title (Final Karaoke Lossless 4k).mp4", ...]
# We need dict like: {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
if isinstance(raw_output_files, list):
output_files = {}
@@ -360,15 +360,18 @@ class GCEEncodingBackend(EncodingBackend):
if not isinstance(path, str):
continue
filename = path.split("/")[-1] if "/" in path else path
filename_lower = filename.lower()
# Map filename patterns to output format keys
if "4k_lossless" in filename:
output_files["mp4_4k_lossless"] = path
elif "4k_lossy" in filename:
# Files are named like "Artist - Title (Final Karaoke Lossless 4k).mp4"
if "lossless 4k" in filename_lower:
if filename.endswith(".mkv"):
output_files["mkv_4k"] = path
else:
output_files["mp4_4k_lossless"] = path
elif "lossy 4k" in filename_lower:
output_files["mp4_4k_lossy"] = path
elif "720p" in filename:
elif "720p" in filename_lower:
output_files["mp4_720p"] = path
elif filename.endswith(".mkv"):
output_files["mkv_4k"] = path
self.logger.info(f"Converted output_files list to dict: {output_files}")
else:
output_files = raw_output_files if isinstance(raw_output_files, dict) else {}
+126
View File
@@ -63,6 +63,11 @@ class EncodingService:
"""Check if GCE encoding is enabled and configured."""
return self.settings.use_gce_encoding and self.is_configured
@property
def is_preview_enabled(self) -> bool:
"""Check if GCE preview encoding is enabled and configured."""
return self.settings.use_gce_preview_encoding and self.is_configured
async def submit_encoding_job(
self,
job_id: str,
@@ -242,6 +247,127 @@ class EncodingService:
job_id, progress_callback=progress_callback
)
async def submit_preview_encoding_job(
self,
job_id: str,
ass_gcs_path: str,
audio_gcs_path: str,
output_gcs_path: str,
background_color: str = "black",
background_image_gcs_path: Optional[str] = None,
font_gcs_path: Optional[str] = None,
) -> Dict[str, Any]:
"""
Submit a preview video encoding job to the GCE worker.
Args:
job_id: Unique job identifier
ass_gcs_path: GCS path to ASS subtitles file (gs://bucket/path/file.ass)
audio_gcs_path: GCS path to audio file
output_gcs_path: GCS path for output video
background_color: Background color (default: black)
background_image_gcs_path: Optional GCS path to background image
font_gcs_path: Optional GCS path to custom font file
Returns:
Response from the encoding worker
Raises:
Exception: If submission fails
"""
self._load_credentials()
if not self.is_configured:
raise RuntimeError("Encoding service not configured")
url = f"{self._url}/encode-preview"
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
payload = {
"job_id": job_id,
"ass_gcs_path": ass_gcs_path,
"audio_gcs_path": audio_gcs_path,
"output_gcs_path": output_gcs_path,
"background_color": background_color,
}
if background_image_gcs_path:
payload["background_image_gcs_path"] = background_image_gcs_path
if font_gcs_path:
payload["font_gcs_path"] = font_gcs_path
logger.info(f"[job:{job_id}] Submitting preview encoding job to GCE worker: {url}")
async with aiohttp.ClientSession() as session:
async with session.post(url, json=payload, headers=headers, timeout=30) as resp:
if resp.status == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp.status == 409:
raise RuntimeError(f"Preview encoding job {job_id} already exists")
if resp.status != 200:
text = await resp.text()
raise RuntimeError(f"Failed to submit preview encoding job: {resp.status} - {text}")
return await resp.json()
async def encode_preview_video(
self,
job_id: str,
ass_gcs_path: str,
audio_gcs_path: str,
output_gcs_path: str,
background_color: str = "black",
background_image_gcs_path: Optional[str] = None,
font_gcs_path: Optional[str] = None,
timeout: float = 90.0,
poll_interval: float = 2.0,
) -> Dict[str, Any]:
"""
Submit preview encoding job and wait for completion.
This is a convenience method that combines submit + wait with shorter
timeouts suitable for preview videos.
Args:
job_id: Unique job identifier
ass_gcs_path: GCS path to ASS subtitles file
audio_gcs_path: GCS path to audio file
output_gcs_path: GCS path for output video
background_color: Background color (default: black)
background_image_gcs_path: Optional GCS path to background image
font_gcs_path: Optional GCS path to custom font file
timeout: Maximum time to wait (default 90s for preview)
poll_interval: Seconds between status checks (default 2s)
Returns:
Final job status with output files
"""
# Submit the job
submit_result = await self.submit_preview_encoding_job(
job_id=job_id,
ass_gcs_path=ass_gcs_path,
audio_gcs_path=audio_gcs_path,
output_gcs_path=output_gcs_path,
background_color=background_color,
background_image_gcs_path=background_image_gcs_path,
font_gcs_path=font_gcs_path,
)
# If cached, return immediately - video already exists in GCS
submit_status = submit_result.get("status")
if submit_status == "cached":
logger.info(f"[job:{job_id}] Preview already cached, returning immediately")
return {"status": "complete", "output_path": submit_result.get("output_path")}
# If in_progress, another request is encoding it - just wait for that
if submit_status == "in_progress":
logger.info(f"[job:{job_id}] Preview encoding already in progress, waiting")
# Wait for completion with shorter timeout
return await self.wait_for_completion(
job_id=job_id,
poll_interval=poll_interval,
timeout=timeout,
)
async def health_check(self) -> Dict[str, Any]:
"""
Check the health of the encoding worker.
+2 -1
View File
@@ -1128,7 +1128,8 @@ class TestFinaliseOnlyModels:
assert request.enable_txt is None
assert request.brand_prefix is None
assert request.keep_brand_code is None
assert request.enable_youtube_upload is False
# YouTube upload default is None (server applies default_enable_youtube_upload)
assert request.enable_youtube_upload is None
assert request.dropbox_path is None
assert request.gdrive_folder_id is None
+2 -1
View File
@@ -149,7 +149,8 @@ class TestUploadSubmissionRequest:
# CDG/TXT disabled by default (requires style config)
assert request.enable_cdg is False
assert request.enable_txt is False
assert request.enable_youtube_upload is False
# YouTube upload default is None (use server default)
assert request.enable_youtube_upload is None
class TestStartReviewRequest:
@@ -425,6 +425,111 @@ class TestVideoWorkerOrchestratorEncoding:
assert "Encoding failed" in str(exc_info.value)
@pytest.mark.asyncio
async def test_run_encoding_gce_downloads_files(self):
"""Test that GCE encoding downloads files from GCS to local directory.
This test verifies the fix for YouTube upload failure when using GCE encoding.
GCE encoding returns GCS blob paths, which need to be downloaded locally
before YouTube upload can access them.
"""
with tempfile.TemporaryDirectory() as temp_dir:
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path=os.path.join(temp_dir, "title.mov"),
karaoke_video_path=os.path.join(temp_dir, "karaoke.mov"),
instrumental_audio_path=os.path.join(temp_dir, "audio.flac"),
output_dir=temp_dir,
)
# Mock storage service
mock_storage = MagicMock()
mock_storage.download_file = MagicMock()
orchestrator = VideoWorkerOrchestrator(
config,
storage=mock_storage,
)
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
from backend.services.encoding_interface import EncodingOutput
# GCE backend returns GCS blob paths (not local paths)
mock_backend = MagicMock()
mock_backend.name = "gce" # Important: must be "gce" to trigger download
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
success=True,
lossless_4k_mp4_path="jobs/test-job/finals/output_4k_lossless.mp4",
lossy_4k_mp4_path="jobs/test-job/finals/output_4k_lossy.mp4",
lossy_720p_mp4_path="jobs/test-job/finals/output_720p.mp4",
lossless_mkv_path="jobs/test-job/finals/output_4k.mkv",
encoding_time_seconds=60.0,
encoding_backend="gce",
))
mock_get.return_value = mock_backend
await orchestrator._run_encoding()
# Verify download_file was called for each output file
assert mock_storage.download_file.call_count == 4
# Verify the result paths were updated to local paths
assert orchestrator.result.final_video.startswith(temp_dir)
assert orchestrator.result.final_video_mkv.startswith(temp_dir)
assert orchestrator.result.final_video_lossy.startswith(temp_dir)
assert orchestrator.result.final_video_720p.startswith(temp_dir)
@pytest.mark.asyncio
async def test_run_encoding_local_does_not_download(self):
"""Test that local encoding does NOT trigger GCS download.
Local encoding produces files directly in the output directory,
so no download is needed.
"""
with tempfile.TemporaryDirectory() as temp_dir:
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path=os.path.join(temp_dir, "title.mov"),
karaoke_video_path=os.path.join(temp_dir, "karaoke.mov"),
instrumental_audio_path=os.path.join(temp_dir, "audio.flac"),
output_dir=temp_dir,
)
# Mock storage service
mock_storage = MagicMock()
mock_storage.download_file = MagicMock()
orchestrator = VideoWorkerOrchestrator(
config,
storage=mock_storage,
)
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
from backend.services.encoding_interface import EncodingOutput
# Local backend returns local paths directly
mock_backend = MagicMock()
mock_backend.name = "local" # Local backend
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
success=True,
lossless_4k_mp4_path=os.path.join(temp_dir, "lossless.mp4"),
lossy_4k_mp4_path=os.path.join(temp_dir, "lossy.mp4"),
lossy_720p_mp4_path=os.path.join(temp_dir, "720p.mp4"),
lossless_mkv_path=os.path.join(temp_dir, "lossless.mkv"),
encoding_time_seconds=120.0,
encoding_backend="local",
))
mock_get.return_value = mock_backend
await orchestrator._run_encoding()
# Verify download_file was NOT called for local encoding
mock_storage.download_file.assert_not_called()
class TestVideoWorkerOrchestratorOrganization:
"""Test organization stage."""
+22 -17
View File
@@ -639,40 +639,45 @@ async def upload_lyrics_results(
logger.error(f"Job {job_id}: {error_msg}")
raise Exception(error_msg)
# Upload reference lyrics if available
# Upload ALL reference lyrics files (not just first) so they're available for distribution
# Note: Source names use .title() so "lrclib" -> "Lrclib", "genius" -> "Genius"
# Use sanitized artist/title to match LyricsProcessor's file naming
reference_files = [
(f"{safe_artist} - {safe_title} (Lyrics Genius).txt", "Genius"),
(f"{safe_artist} - {safe_title} (Lyrics Spotify).txt", "Spotify"),
(f"{safe_artist} - {safe_title} (Lyrics Musixmatch).txt", "Musixmatch"),
(f"{safe_artist} - {safe_title} (Lyrics Lrclib).txt", "LRCLib"), # Note: source is 'lrclib', .title() = 'Lrclib'
(f"{safe_artist} - {safe_title} (Lyrics Genius).txt", "genius"),
(f"{safe_artist} - {safe_title} (Lyrics Spotify).txt", "spotify"),
(f"{safe_artist} - {safe_title} (Lyrics Musixmatch).txt", "musixmatch"),
(f"{safe_artist} - {safe_title} (Lyrics Lrclib).txt", "lrclib"),
]
found_reference = False
for ref_filename, source_name in reference_files:
found_references = []
for ref_filename, source_key in reference_files:
ref_path = os.path.join(lyrics_dir, ref_filename)
if os.path.exists(ref_path):
# Upload with original filename to preserve proper naming
gcs_path = f"jobs/{job_id}/lyrics/{ref_filename}"
url = storage.upload_file(ref_path, gcs_path)
# Track in job.files so video_worker can download for distribution
job_manager.update_file_url(job_id, 'lyrics', f'reference_{source_key}', url)
if job_log:
job_log.info(f"Found reference lyrics from {source_name}")
job_log.info(f"Found reference lyrics from {source_key}")
logger.info(f"Job {job_id}: Uploaded reference lyrics: {ref_filename}")
found_reference = True
break # Only upload first available reference
if not found_reference:
found_references.append(source_key)
if not found_references:
if job_log:
job_log.warning("No reference lyrics found from any source (Genius, Spotify, Musixmatch, LRCLib)")
logger.warning(f"Job {job_id}: No reference lyrics found from any source")
else:
logger.info(f"Job {job_id}: Found {len(found_references)} reference lyrics sources: {found_references}")
# Upload uncorrected transcription if available (use sanitized names)
uncorrected_file = os.path.join(lyrics_dir, f"{safe_artist} - {safe_title} (Lyrics Uncorrected).txt")
# Upload uncorrected transcription if available (preserve original filename for distribution)
uncorrected_filename = f"{safe_artist} - {safe_title} (Lyrics Uncorrected).txt"
uncorrected_file = os.path.join(lyrics_dir, uncorrected_filename)
if os.path.exists(uncorrected_file):
gcs_path = f"jobs/{job_id}/lyrics/uncorrected.txt"
gcs_path = f"jobs/{job_id}/lyrics/{uncorrected_filename}"
url = storage.upload_file(uncorrected_file, gcs_path)
job_manager.update_file_url(job_id, 'lyrics', 'uncorrected', url)
logger.info(f"Job {job_id}: Uploaded uncorrected transcription")
logger.info(f"Job {job_id}: Uploaded uncorrected transcription: {uncorrected_filename}")
logger.info(f"Job {job_id}: All lyrics results uploaded successfully")
+59 -22
View File
@@ -160,14 +160,17 @@ async def _encode_via_gce(
storage.download_file(gcs_path, local_path)
# Map to result keys expected by _upload_results
if "4K Lossless" in filename or "4k_lossless" in filename.lower():
local_files["final_video"] = local_path
elif "4K Lossy" in filename or "4k_lossy" in filename.lower():
# Files are named like "Artist - Title (Final Karaoke Lossless 4k).mp4"
filename_lower = filename.lower()
if "lossless 4k" in filename_lower:
if filename.endswith(".mkv"):
local_files["final_video_mkv"] = local_path
else:
local_files["final_video"] = local_path
elif "lossy 4k" in filename_lower:
local_files["final_video_lossy"] = local_path
elif "720p" in filename:
elif "720p" in filename_lower:
local_files["final_video_720p"] = local_path
elif filename.endswith(".mkv"):
local_files["final_video_mkv"] = local_path
job_log.info(f"Downloaded {len(local_files)} encoded files")
@@ -1108,29 +1111,63 @@ async def _prepare_distribution_directory(
log_progress(f"Downloaded {stem_key} to stems/")
# --- Download lyrics files to lyrics/ subfolder ---
# Use proper filenames matching local CLI output structure
# Common lyrics file types
lyrics_file_keys = ['lrc', 'ass', 'srt', 'txt', 'json', 'original_json', 'corrected_json']
for lyrics_key in lyrics_file_keys:
# Map job.files.lyrics keys to proper local filenames
# Format: (job_files_key, local_filename_pattern)
lyrics_file_mappings = [
# Karaoke output files
('lrc', f"{base_name} (Karaoke).lrc"),
('ass', f"{base_name} (Karaoke).ass"),
# Corrections and transcription files
('corrections', f"{base_name} (Lyrics Corrections).json"),
('corrected_txt', f"{base_name} (Lyrics Corrected).txt"),
('uncorrected', f"{base_name} (Lyrics Uncorrected).txt"),
# Reference lyrics from various sources
('reference_genius', f"{base_name} (Lyrics Genius).txt"),
('reference_spotify', f"{base_name} (Lyrics Spotify).txt"),
('reference_musixmatch', f"{base_name} (Lyrics Musixmatch).txt"),
('reference_lrclib', f"{base_name} (Lyrics Lrclib).txt"),
]
for lyrics_key, local_filename in lyrics_file_mappings:
if lyrics.get(lyrics_key):
# Determine proper extension
ext_map = {
'lrc': '.lrc',
'ass': '.ass',
'srt': '.srt',
'txt': '.txt',
'json': '.json',
'original_json': '_original.json',
'corrected_json': '_corrected.json',
}
ext = ext_map.get(lyrics_key, f'.{lyrics_key}')
dest_path = os.path.join(lyrics_dir, f"{base_name} (Karaoke){ext}")
dest_path = os.path.join(lyrics_dir, local_filename)
try:
storage.download_file(lyrics[lyrics_key], dest_path)
log_progress(f"Downloaded {lyrics_key} to lyrics/")
log_progress(f"Downloaded {lyrics_key} to lyrics/{local_filename}")
except Exception as e:
log_progress(f"Could not download {lyrics_key}: {e}")
# --- Download (With Vocals).mkv to lyrics/ subfolder ---
# Local CLI places the karaoke video with vocals in the lyrics folder
videos = job.file_urls.get('videos', {})
if videos.get('with_vocals'):
with_vocals_dest = os.path.join(lyrics_dir, f"{base_name} (With Vocals).mkv")
try:
storage.download_file(videos['with_vocals'], with_vocals_dest)
log_progress(f"Downloaded with_vocals to lyrics/{base_name} (With Vocals).mkv")
except Exception as e:
log_progress(f"Could not download with_vocals to lyrics/: {e}")
# --- Download preview ASS files to lyrics/previews/ subfolder ---
# Preview files are stored in jobs/{job_id}/previews/ in GCS
previews_dir = os.path.join(lyrics_dir, "previews")
os.makedirs(previews_dir, exist_ok=True)
# List all preview files from GCS and download them
try:
preview_prefix = f"jobs/{job_id}/previews/"
preview_files = storage.list_files(preview_prefix)
for blob_name in preview_files:
if blob_name.endswith('.ass'):
filename = os.path.basename(blob_name)
dest_path = os.path.join(previews_dir, filename)
storage.download_file(blob_name, dest_path)
log_progress(f"Downloaded preview {filename} to lyrics/previews/")
except Exception as e:
log_progress(f"Could not download preview files: {e}")
log_progress("Distribution directory prepared with stems/ and lyrics/ subfolders")
+56 -1
View File
@@ -367,15 +367,70 @@ class VideoWorkerOrchestrator:
if not output.success:
raise Exception(f"Encoding failed: {output.error_message}")
# Store results
# Store results - for GCE backend, these are GCS blob paths that need to be downloaded
self.result.final_video = output.lossless_4k_mp4_path
self.result.final_video_mkv = output.lossless_mkv_path
self.result.final_video_lossy = output.lossy_4k_mp4_path
self.result.final_video_720p = output.lossy_720p_mp4_path
self.result.encoding_time_seconds = output.encoding_time_seconds
# For GCE encoding, download the encoded files from GCS to local directory
# This is required for YouTube upload and other local file operations
if encoding_backend.name == "gce" and self.storage:
await self._download_gce_encoded_files(output)
self.job_log.info(f"Encoding complete ({encoding_backend.name}) in {output.encoding_time_seconds:.1f}s")
async def _download_gce_encoded_files(self, output):
"""
Download GCE-encoded files from GCS to the local output directory.
GCE encoding stores files in GCS and returns blob paths like:
'jobs/{job_id}/finals/Artist - Title (Final Karaoke Lossless 4k).mp4'
This method downloads those files locally so that subsequent stages
(YouTube upload, etc.) can access them as local files.
Args:
output: EncodingOutput from the GCE backend with GCS blob paths
"""
self.job_log.info("Downloading GCE-encoded files from GCS")
# Map of result attributes to download
file_mappings = [
('lossless_4k_mp4_path', 'final_video'),
('lossless_mkv_path', 'final_video_mkv'),
('lossy_4k_mp4_path', 'final_video_lossy'),
('lossy_720p_mp4_path', 'final_video_720p'),
]
downloaded_count = 0
for output_attr, result_attr in file_mappings:
gcs_path = getattr(output, output_attr, None)
if not gcs_path:
continue
# Extract filename from GCS path
filename = os.path.basename(gcs_path)
local_path = os.path.join(self.config.output_dir, filename)
try:
self.job_log.info(f"Downloading {filename} from GCS")
self.storage.download_file(gcs_path, local_path)
# Update the result to point to local file
setattr(self.result, result_attr, local_path)
downloaded_count += 1
self.job_log.info(f"Downloaded {filename} to {local_path}")
except Exception as e:
self.job_log.error(f"Failed to download {filename}: {e}")
# Clear the result attribute so downstream doesn't try to use invalid GCS path
setattr(self.result, result_attr, None)
# Don't fail - some formats might not be generated
self.job_log.info(f"Downloaded {downloaded_count} encoded files from GCS")
async def _run_organization(self):
"""Run the organization stage (brand code generation)."""
self.job_log.info("Starting organization stage")
+3 -2
View File
@@ -31,7 +31,8 @@
└──────────┘ └──────────┘ └──────────────┘ └──────────────┘
(*) GCE Encoding Worker: C4-standard-8 VM with Intel Granite Rapids for
high-performance FFmpeg encoding. Optional - falls back to Cloud Run.
high-performance FFmpeg encoding. Used for both final video encoding and
preview video generation. Uses LocalEncodingService via GCS wheel deployment.
```
## Processing Pipeline
@@ -57,7 +58,7 @@
▼
4. HUMAN REVIEW (AWAITING_REVIEW)
• React UI for lyrics correction
• Preview video generation
• Preview video generation (optionally via GCE for speed)
• Submit corrections
▼
5. VIDEO RENDERING
+25 -5
View File
@@ -110,17 +110,37 @@ Push to `main` triggers GitHub Actions:
3. Deploy to Cloud Run
4. Deploy frontend to Cloudflare Pages
### Manual Backend Deploy
### Infrastructure Changes (Pulumi)
**All infrastructure changes must go through PRs.** The CI pipeline automatically deploys Pulumi changes when PRs merge to main.
```bash
# Preview infrastructure changes
# Preview changes locally (requires GCP auth)
cd infrastructure
pulumi preview
pulumi preview --stack nomadkaraoke/karaoke-gen-infrastructure/prod
# Apply changes
pulumi up
# DO NOT run `pulumi up` locally - let CI handle deployment
# CI uses --skip-preview due to limited service account permissions
```
**What's managed by Pulumi:**
- Firestore database and indexes
- GCS buckets
- Cloud Run service configuration
- Cloud Tasks queues
- Secret Manager secrets
- Service accounts and IAM bindings
- Workload Identity Federation
- GCE instances (GitHub runners, encoding worker)
**Workflow:**
1. Make changes in `infrastructure/__main__.py`
2. Run `pulumi preview` locally to verify
3. Create PR and merge
4. CI deploys automatically
See `docs/LESSONS-LEARNED.md` for Pulumi CI gotchas.
### Manual Frontend Deploy
```bash
+413
View File
@@ -222,6 +222,52 @@ escaped = filename.replace("'", "\\'")
query = f"name='{escaped}' and '{parent_id}' in parents"
```
### FFmpeg Filter Path Escaping (subprocess without shell)
**Problem**: Song titles with apostrophes (e.g., "I'm With You") cause FFmpeg's ASS subtitle filter to fail because paths aren't escaped correctly.
```python
# BAD - FFmpeg strips the apostrophe, looks for wrong path
ass_filter = f"ass=./Avril Lavigne - I'm With You/cache/temp.ass"
# FFmpeg error: Could not create a libass track when reading file './Avril Lavigne - Im With You/cache/temp.ass'
```
**Wrong approach #1**: Simple backslash escaping like `\'` doesn't work because FFmpeg's filter parser requires more escaping.
**Wrong approach #2**: Shell-style escaping `'\''` (end quote, escaped quote, start quote) only works when passing through a shell. When using Python `subprocess` with a command list, there's no shell - arguments go directly to FFmpeg, so shell escaping patterns don't work.
**Solution**: Use FFmpeg filter escaping (for subprocess without shell):
```python
def _escape_ffmpeg_filter_path(path: str) -> str:
"""Escape a path for FFmpeg filter expressions (for subprocess without shell).
FFmpeg filter escaping rules:
- Backslashes: double them (\ -> \\)
- Single quotes/apostrophes: escape with three backslashes (' -> \\\')
- Spaces: escape with backslash ( -> \ )
"""
# First escape existing backslashes (\ -> \\)
escaped = path.replace("\\", "\\\\")
# Escape single quotes (' -> \\\')
# In Python source: 6 backslashes + apostrophe = 3 backslashes + apostrophe in string
escaped = escaped.replace("'", "\\\\\\'")
# Escape spaces
escaped = escaped.replace(" ", "\\ ")
return escaped
# Result: "I'm With You" becomes "I\\\'m\ With\ You"
# FFmpeg correctly interprets this as: I'm With You
```
**Symptoms**:
- Video generation fails with "Could not create a libass track"
- Error path is subtly different from the path in your code (missing apostrophe)
- Works fine for songs without special characters
**Key insight**: FFmpeg filter escaping is different from shell escaping. When using `subprocess.run(cmd_list)` (no `shell=True`), there's no shell to interpret the arguments. FFmpeg receives the strings directly and applies its own filter expression parsing rules. The `'\''` shell idiom does NOT work in this context.
**Lesson**: When debugging FFmpeg filter path issues, check whether you're using shell or subprocess. For subprocess without shell, escape spaces with `\ ` and apostrophes with `\\\'` (3 backslashes + quote in the actual string).
### Duplicate Code Leads to Inconsistent Behavior
**Problem**: Multiple endpoints computing the same defaults independently. `audio_search.py` had `effective_brand_prefix = body.brand_prefix or settings.default_brand_prefix` but `file_upload.py` was missing it for some endpoints.
@@ -247,6 +293,43 @@ def _get_effective_distribution_settings(...) -> EffectiveDistributionSettings:
**Lesson**: When you see the same logic repeated in multiple places, refactor into a helper function immediately. Otherwise bugs will be fixed in one place but not others.
### Python `or` Operator Can't Override True Defaults with False
**Problem**: Using Python's `or` operator for default logic prevents explicit `false` from overriding a `true` server default.
```python
# BAD - can't explicitly disable if server default is true
effective_value = body.enable_feature or settings.default_enable_feature
# When body.enable_feature=False and settings.default=True:
# False or True = True (ignores explicit False!)
```
**Symptoms**:
- Feature can be enabled but never disabled via API
- Works fine when server default is `false` but breaks when default is `true`
- No errors - just ignores the explicit `false` value
**Solution**: Use `Optional[bool] = None` with explicit `is not None` check:
```python
# Request model
enable_feature: Optional[bool] = None # None = use server default
# Default logic
effective_value = (
body.enable_feature
if body.enable_feature is not None
else settings.default_enable_feature
)
```
This allows three states:
- `None` (not specified) → use server default
- `True` → explicitly enable
- `False` → explicitly disable (overrides server default)
**Lesson**: When a boolean field needs to support "use server default" behavior, use `Optional[bool] = None` instead of `bool = False`. The `or` operator conflates "not specified" with "explicitly false".
### Field Name Mismatches Between Endpoints
**Problem**: Different endpoints setting different field names for the same logical value, where consumers check only one field name.
@@ -588,6 +671,43 @@ GitHub-hosted runners have ~14GB disk, which caused failures during Docker build
See `docs/archive/2025-12-28-self-hosted-github-runner.md` for full details.
### Pulumi CI Deployments Need Skip-Preview for Limited Service Accounts
**Problem**: Running `pulumi up --refresh` in CI requires the service account to have read permissions on ALL managed GCP resources. Pulumi refreshes state by querying each resource, and permission errors cause the deployment to fail before any changes are applied.
**Symptoms**:
```
Error when reading or editing Resource "project" with IAM Member: Role "roles/...":
Error retrieving IAM policy for project "...": googleapi: Error 403: The caller does not have permission
```
**Root cause**: `--refresh` (default behavior) makes read API calls to verify current state matches Pulumi's state. This requires permissions like:
- `resourcemanager.projects.getIamPolicy` (for IAM bindings)
- `secretmanager.secrets.get` (for secrets)
- `cloudtasks.queues.get` (for Cloud Tasks)
- `iam.workloadIdentityPools.get` (for Workload Identity)
- And many more for each resource type managed
**Solution**: Use `--skip-preview` for CI deployments where Pulumi state is authoritative:
```yaml
# In CI workflow
pulumi up --yes --skip-preview --stack org/project/stack
```
**Why this is safe**:
- Infrastructure is only modified through Pulumi (single source of truth)
- Preview/refresh is still useful during local development with broader permissions
- Reduces permissions surface for CI service account (principle of least privilege)
**Alternative** (if refresh is required): Grant the service account `roles/editor` or individual read roles for each resource type. This is less secure but allows drift detection.
**Lesson**: When setting up Pulumi in CI, either:
1. Grant broad read permissions and keep `--refresh` (detects drift)
2. Use minimal permissions and `--skip-preview` (trusts Pulumi state)
Choose based on your threat model. For most projects, trusting Pulumi state is acceptable since manual console changes are rare.
### Docker Disk Space Management on Self-Hosted Runners
**Problem**: Even with 200GB SSDs, self-hosted runners can fill up with Docker images. Each CI build creates new dangling images (~15GB each), and without aggressive cleanup, disk fills in days.
@@ -667,6 +787,61 @@ RUN apt-get update && apt-get install -y \
**Lesson**: When building Docker images for video/subtitle rendering, install fonts explicitly. The default slim images have nothing, and font fallback behavior differs from local development machines.
### Reuse LLM Model Instances Across Operations
**Problem**: Creating a new LLM model instance for each operation in a loop causes massive overhead. Each instance creation triggers model initialization, connection establishment, and optional "warm-up" calls - adding 2+ seconds per operation.
**Example**: Agentic lyrics correction was creating a new `AgenticCorrector` for each gap (lyric segment needing correction):
```python
# BAD - 31 gaps × 2s initialization = 62s overhead
for gap in gaps_to_correct:
agent = AgenticCorrector.from_model(model_id, ...) # New instance every time!
agent.correct_gap(gap)
```
**Symptoms**:
- Operations that should take ~1 minute take 5+ minutes
- Logs show "Initializing model..." or "Warming up..." repeatedly
- Each loop iteration has a consistent ~2s overhead before actual work
- Total time scales linearly with loop count, not with actual work
**Solution**: Create the model instance ONCE before the loop, then reuse:
```python
# GOOD - 1 initialization, reused for all gaps
agent = AgenticCorrector.from_model(model_id, ...) # Create once
for gap in gaps_to_correct:
agent.correct_gap(gap) # Reuse existing instance
```
**Additional optimization**: If operations are independent, process them in parallel:
```python
# BETTER - parallel processing with shared model
agent = AgenticCorrector.from_model(model_id, ...)
with ThreadPoolExecutor(max_workers=5) as executor:
futures = {executor.submit(agent.correct_gap, gap): gap for gap in gaps}
for future in as_completed(futures):
result = future.result()
```
**Results**:
- Before: 31 gaps → ~5 minutes (2s init + 3s work per gap)
- After: 20 gaps → ~55 seconds (1 init + parallel processing)
- Speedup: ~5-6x
**Why warm-up is often unnecessary**: The warm-up pattern (sending a trivial request to "prime" the model) made sense for cold-start scenarios in older systems. Modern LLM APIs:
- Don't have cold-start penalties per request
- Handle connection pooling internally
- May even rate-limit warm-up calls
If you're already making real requests, warm-up adds latency without benefit.
**Key insight**: LLM client instances are designed for reuse. They maintain connection pools, authentication state, and caching. Creating a new instance per request throws away these benefits.
**Configuration**: Added `AGENTIC_MAX_PARALLEL_GAPS` env var (default: 5) to control parallelism. Higher values increase throughput but may hit rate limits.
## Performance Observations
| Operation | Duration |
@@ -755,6 +930,48 @@ export function getToken() {
**Lesson**: For values that can change (auth tokens, user preferences), read fresh from storage on each access rather than caching in module scope.
### Zustand Module-Level Initialization Doesn't Re-run on Client Navigation
**Problem**: Module-level initialization code in zustand stores only runs once when the module is first imported. Client-side navigation (e.g., `router.push()`) doesn't reload modules, so initialization code doesn't re-run.
**Example bug**: Auth module had initialization that fetched user data if a token existed:
```typescript
// auth.ts - module level (runs once at import)
if (typeof window !== 'undefined') {
const token = getAccessToken()
if (token) {
useAuth.getState().fetchUser() // Only runs on initial page load!
}
}
```
After beta enrollment:
1. User lands on `/` (landing page) - auth.ts imported, no token yet, `fetchUser()` not called
2. User completes enrollment, token stored via `setAccessToken()`
3. `router.push('/app')` - client-side navigation, modules already loaded
4. Auth module initialization doesn't re-run - `user` stays `null`
5. UI shows "Login" button instead of user credits
**Symptoms**:
- Feature works after page reload but not after client-side navigation
- Zustand store has correct persisted data (token) but not computed data (user profile)
- E2E tests that use reload pass; tests that use `router.push` fail
**Solution**: Components that need fresh data should trigger fetches in useEffect:
```typescript
// In /app page component
const { user, fetchUser } = useAuth()
useEffect(() => {
const token = getAccessToken()
if (token && !user) {
fetchUser() // Ensure user data is loaded regardless of how we got here
}
}, [user, fetchUser])
```
**Lesson**: Don't rely on module-level initialization for data that might need to be fetched after client-side navigation. Use useEffect in components to ensure data is loaded when needed.
### GCP Organization Policies Block Service Account Keys
**Problem**: GCP organization policies can block service account key creation even when you have Owner access to the project.
@@ -1079,6 +1296,45 @@ logs_ref.document(log_id).set({
**Lesson**: Never use embedded arrays for unbounded data. If an array can grow indefinitely based on operation duration or user input, use a subcollection instead. The 1MB limit applies to the entire document, including all fields.
### Splitting Fast Generation from Slow Encoding
**Problem**: Preview video generation took 60+ seconds on Cloud Run, causing poor UX during lyrics review. The bottleneck was FFmpeg encoding (CPU-bound), not the subtitle generation (fast text processing).
**Solution**: Split the operation into two phases with optional offloading:
1. **ASS Generation (Cloud Run)**: Generate ASS subtitle file locally (~2s). This is pure text processing with no CPU-intensive encoding.
2. **Video Encoding (GCE)**: Upload ASS to GCS and offload FFmpeg encoding to the high-performance GCE worker (~15-20s on C4-standard-8).
```python
# Generate ASS only (fast, local)
result = CorrectionOperations.generate_preview_video(
...,
ass_only=True, # Skip video encoding
)
# Upload ASS to GCS
storage.upload_file(result["ass_path"], f"jobs/{job_id}/previews/{preview_hash}.ass")
# Offload encoding to GCE
gce_result = await encoding_service.encode_preview_video(
job_id=preview_hash,
ass_gcs_path=f"gs://bucket/jobs/{job_id}/previews/{preview_hash}.ass",
audio_gcs_path=audio_gcs_path,
output_gcs_path=f"gs://bucket/jobs/{job_id}/previews/{preview_hash}.mp4",
)
```
**Key insight**: When a pipeline has both fast (text generation) and slow (media encoding) steps, design for separability:
- Add `_only` parameters to skip slow steps
- Return intermediate outputs (paths to generated files)
- Allow external services to handle the slow part
- Keep fallback to local processing for reliability
**Configuration**: Controlled by `USE_GCE_PREVIEW_ENCODING` env var. Falls back to local encoding if GCE fails or is disabled.
**Result**: Preview generation dropped from 60+ seconds to ~15-20 seconds.
### External Service Response Format Mismatches
**Problem**: The GCE encoding worker returned responses in unexpected formats, causing `'list' object has no attribute 'get'` errors that were hard to debug through the async pipeline.
@@ -1127,6 +1383,163 @@ if isinstance(output_files, list):
**Debugging tip**: When hitting format errors in async pipelines, check the raw response from the external service directly (curl with API key) before assuming the bug is in your code.
### GCE Encoding Returns GCS Paths, Not Local Files
**Problem**: When using GCE encoding (remote high-performance encoder), the orchestrator received GCS blob paths like `jobs/{id}/finals/output_4k_lossless.mp4` instead of local file paths. Subsequent operations like YouTube upload checked `os.path.isfile()` which returned `False`, causing silent failures.
**Symptoms**:
- YouTube upload logged "No video file available for YouTube upload"
- Other file-based operations silently skipped
- Dropbox/GDrive uploads succeeded (they use different code paths)
- No errors in logs - just warnings that were easy to miss
**Root cause**: The `VideoWorkerOrchestrator._run_encoding()` method stored the paths returned by `GCEEncodingBackend` directly without downloading the files. For local encoding, the paths are already local. For GCE encoding, the paths are GCS blob paths that need to be downloaded first.
**Solution**: Added `_download_gce_encoded_files()` method that:
1. Detects when GCE backend was used (`encoding_backend.name == "gce"`)
2. Downloads each encoded file from GCS to the local temp directory
3. Updates result paths to point to local files
```python
# After encoding completes
if encoding_backend.name == "gce" and self.storage:
await self._download_gce_encoded_files(output)
```
**Key insight**: When an operation can run on multiple backends (local vs remote), ensure the outputs are normalized to a common format before subsequent stages consume them. The orchestrator pattern should abstract away backend differences, not leak them to downstream consumers.
**Testing tip**: Test the full pipeline with each backend type, not just the encoding step in isolation. The bug only manifested when YouTube upload ran after GCE encoding - unit tests of each stage passed.
### Unify Encoding Logic with GCS Wheel Deployment
**Problem**: The GCE encoding worker had simplified, duplicated encoding logic that:
- Didn't concatenate title + karaoke + end screens
- Used generic filenames like `output_4k_lossless.mp4` instead of proper names
- Lacked feature parity with the local CLI's `LocalEncodingService`
This caused cloud-generated videos to be incomplete (no title/end screens) and have unhelpful filenames that didn't match the local CLI output.
**Solution**: Deploy the karaoke-gen wheel to GCS, have the GCE worker install and use `LocalEncodingService`:
1. **CI uploads wheel to GCS**: `deploy-backend` job uploads wheel to `gs://bucket/wheels/`
2. **Dynamic wheel loading**: GCE worker downloads and installs latest wheel at job start:
```python
def ensure_latest_wheel():
"""Download and install latest karaoke-gen wheel from GCS."""
subprocess.run(["gsutil", "cp", "gs://bucket/wheels/karaoke_gen-*.whl", "/tmp/"])
wheel = sorted(glob.glob("/tmp/karaoke_gen-*.whl"))[-1]
subprocess.run([sys.executable, "-m", "pip", "install", "--upgrade", wheel])
async def process_job(job_id, request):
ensure_latest_wheel() # Get latest code at job start
# ... rest of processing
```
3. **Single encoding path**: GCE worker imports and uses `LocalEncodingService`:
```python
from backend.services.local_encoding_service import LocalEncodingService, EncodingConfig
service = LocalEncodingService(logger=logger)
config = EncodingConfig(
title_video=str(title_video),
karaoke_video=str(karaoke_video),
output_lossless_4k_mp4=f"{base_name} (Final Karaoke Lossless 4k).mp4",
# ... proper naming for all formats
)
result = service.encode_all_formats(config)
```
**Key benefits**:
1. **Single source of truth**: Same encoding logic used by CLI, Cloud Run, and GCE
2. **Hot updates without restart**: New code deployed on next job, not VM restart
3. **In-progress jobs unaffected**: Jobs use the code version they started with
4. **Proper output**: Titles/end screens concatenated, files named correctly
**No fallback by design**: We removed all fallback logic to ensure there's only ONE code path. If the wheel fails to load, the job fails clearly rather than producing inconsistent output.
**Lesson**: When multiple deployment targets (CLI, Cloud Run, GCE) need the same complex logic, package it as a library and deploy via a shared mechanism (GCS wheel). Duplicating logic leads to feature drift and inconsistent behavior.
### Cloud Distribution Must Match Local CLI Output Structure
**Problem**: Cloud-generated karaoke outputs had different file organization than local CLI, causing confusion and missing files in Dropbox uploads:
1. **lyrics/ folder filename differences**:
- Cloud used generic `(Karaoke).json` instead of `(Lyrics Corrections).json`
- Cloud used `(Karaoke).txt` instead of `(Lyrics Corrected).txt`
- Reference lyrics files (`(Lyrics Genius).txt`, etc.) weren't downloaded
2. **Missing files in lyrics/ folder**:
- `(With Vocals).mkv` wasn't copied to lyrics/ subfolder
- `previews/` subfolder with preview ASS files wasn't created
- Reference lyrics from multiple sources weren't included
**Root causes**:
- Lyrics worker uploaded reference files to GCS but didn't track them in `job.files`
- Video worker used generic naming pattern for all lyrics files
- Video worker didn't copy `(With Vocals).mkv` or previews to lyrics folder
**Solution**:
1. **Track all files in job.files** (lyrics_worker.py):
```python
# Upload ALL reference lyrics and track them for distribution
for ref_filename, source_key in reference_files:
if os.path.exists(ref_path):
url = storage.upload_file(ref_path, gcs_path)
# Track in job.files so video_worker can download
job_manager.update_file_url(job_id, 'lyrics', f'reference_{source_key}', url)
```
2. **Use proper filenames in distribution** (video_worker.py):
```python
lyrics_file_mappings = [
('lrc', f"{base_name} (Karaoke).lrc"),
('corrections', f"{base_name} (Lyrics Corrections).json"),
('corrected_txt', f"{base_name} (Lyrics Corrected).txt"),
('uncorrected', f"{base_name} (Lyrics Uncorrected).txt"),
('reference_genius', f"{base_name} (Lyrics Genius).txt"),
# ... etc
]
```
3. **Copy (With Vocals).mkv to lyrics folder**:
```python
if videos.get('with_vocals'):
dest = os.path.join(lyrics_dir, f"{base_name} (With Vocals).mkv")
storage.download_file(videos['with_vocals'], dest)
```
4. **Create previews/ subfolder**:
```python
previews_dir = os.path.join(lyrics_dir, "previews")
os.makedirs(previews_dir, exist_ok=True)
preview_files = storage.list_files(f"jobs/{job_id}/previews/")
for blob_name in preview_files:
if blob_name.endswith('.ass'):
storage.download_file(blob_name, dest_path)
```
**Expected output structure** (matches local CLI):
```
NOMAD-XXXX - Artist - Title/
├── stems/
│ ├── Artist - Title (Vocals model_bs_roformer_...).flac
│ └── ...
└── lyrics/
├── Artist - Title (Karaoke).lrc
├── Artist - Title (Karaoke).ass
├── Artist - Title (Lyrics Corrections).json
├── Artist - Title (Lyrics Corrected).txt
├── Artist - Title (Lyrics Uncorrected).txt
├── Artist - Title (Lyrics Genius).txt
├── Artist - Title (With Vocals).mkv
└── previews/
└── preview_abc123 (Karaoke).ass
```
**Lesson**: When building distribution packages from cloud storage, the organization code must replicate the exact structure users expect from local execution. Track ALL intermediate files in `job.files` even if they're "optional" - users may rely on them.
## What We'd Do Differently
1. **Add Pydantic model fields test first** - Would have caught the silent field issue immediately
+6
View File
@@ -36,6 +36,12 @@
## Recent Changes
- **GCE Encoding Unified with LocalEncodingService** (2026-01-06): GCE encoding worker now uses the same `LocalEncodingService` as the local CLI, eliminating duplicated encoding logic. Output files now have proper names (`Artist - Title (Final Karaoke Lossless 4k).mp4` instead of `output_4k_lossless.mp4`) and include title/end screen concatenation. Wheel deployed to GCS; worker installs at job start for hot updates without VM restart. Removed all fallback logic - single code path for consistent output across CLI, Cloud Run, and GCE. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#unify-encoding-logic-with-gcs-wheel-deployment).
- **Cloud Distribution Matches Local CLI** (2026-01-06): Fixed cloud distribution (Dropbox uploads) to match local CLI output structure. The `lyrics/` folder now includes: proper filenames (`(Lyrics Corrections).json` instead of `(Karaoke).json`), all reference lyrics files (`(Lyrics Genius).txt`, `(Lyrics Spotify).txt`, etc.), `(With Vocals).mkv`, and `previews/` subfolder with preview ASS files. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#cloud-distribution-must-match-local-cli-output-structure).
- **Agentic Correction Performance** (2026-01-05): Optimized agentic AI correction from ~5 minutes to ~55 seconds for 20 gaps (~5-6x speedup). Fixed anti-pattern of creating new model instance per gap (caused repeated 2s warm-up overhead). Now creates model once and processes gaps in parallel using ThreadPoolExecutor (default 5 workers). Configure via `AGENTIC_MAX_PARALLEL_GAPS` env var. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#reuse-llm-model-instances-across-operations).
- **Worker Logs Subcollection** (2026-01-04): Moved `worker_logs` from embedded array in job documents to Firestore subcollection (`jobs/{job_id}/logs`). Fixes job failures when logs exceed 1MB (job 501258e1 had 1.26MB of logs). New logs stored with 30-day TTL via Firestore TTL policy. Feature flag `USE_LOG_SUBCOLLECTION=true` (default). See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#firestore-document-1mb-limit-with-embedded-arrays).
- **GCE Encoding Response Fixes** (2026-01-04): Fixed multiple response format mismatches with GCE encoding worker. The worker returns `output_files` as a list of paths, not a dict with format keys - added conversion logic. Also added defensive type checking for status responses that could be lists. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md#external-service-response-format-mismatches).
+12
View File
@@ -3,6 +3,7 @@
import { useState, useEffect, useCallback } from "react"
import { useRouter } from "next/navigation"
import { api, Job, getAccessToken } from "@/lib/api"
import { useAuth } from "@/lib/auth"
import { useJobNotifications, useVisibilityRefresh } from "@/hooks/use-notifications"
import { Button } from "@/components/ui/button"
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
@@ -28,6 +29,7 @@ export default function AppPage() {
const [isInitialLoad, setIsInitialLoad] = useState(true)
const [isAuthenticated, setIsAuthenticated] = useState<boolean | null>(null)
const { isDarkMode, toggleTheme, mounted } = useTheme()
const { user, fetchUser } = useAuth()
// Memoize loadJobs for use with visibility refresh
const loadJobs = useCallback(async () => {
@@ -67,6 +69,16 @@ export default function AppPage() {
setIsAuthenticated(true)
}, [router])
// Ensure user data is loaded after authentication
// This handles the case where user navigates via client-side navigation
// (e.g., after beta enrollment) and the module-level fetchUser() didn't run
useEffect(() => {
const token = getAccessToken()
if (token && !user) {
fetchUser()
}
}, [user, fetchUser])
// Load jobs on mount (only if authenticated)
useEffect(() => {
if (isAuthenticated !== true) {
@@ -154,6 +154,23 @@ test.describe('E2E Happy Path - Real User with Full UI Interactions', () => {
// Only create email helper if we're doing the full signup flow
const emailHelper = usePreConfiguredToken ? null : await createEmailHelper();
// Intercept API requests to disable YouTube upload for E2E tests
// This prevents test jobs from uploading to YouTube when server default is true
await page.route('**/api/audio-search/search', async (route) => {
const request = route.request();
if (request.method() === 'POST') {
const postData = request.postDataJSON();
// Explicitly disable YouTube upload for E2E test jobs
postData.enable_youtube_upload = false;
console.log(' [Request Intercept] Disabled YouTube upload for audio search');
await route.continue({
postData: JSON.stringify(postData),
});
} else {
await route.continue();
}
});
try {
// =========================================================================
// STEP 1: Landing Page
@@ -171,6 +171,71 @@ test.describe('Authentication - Beta Enrollment', () => {
await page.waitForURL(/\/app/, { timeout: 10000 });
expect(page.url()).toContain('/app');
});
test('beta enrollment shows credits after redirect to app', async ({ page }) => {
// This test verifies that after beta enrollment, the /app page
// correctly fetches and displays user data including credits.
// This catches the bug where client-side navigation didn't trigger fetchUser().
await setupApiFixtures(page, {
mocks: [
{
method: 'POST',
path: '/api/users/beta/enroll',
response: {
status: 200,
body: {
status: 'success',
message: 'Enrolled in beta',
credits_granted: 1,
session_token: 'mock-session-token-456',
},
},
},
{
method: 'GET',
path: '/api/users/me',
response: {
status: 200,
body: {
user: {
email: 'beta-tester@example.com',
credits: 1,
role: 'user',
display_name: null,
total_jobs_created: 0,
total_jobs_completed: 0,
},
has_session: true,
},
},
},
{
method: 'GET',
path: '/api/jobs',
response: { body: [] },
},
],
});
await page.goto('/');
await page.waitForLoadState('networkidle');
// Complete beta enrollment
await page.getByRole('button', { name: /join beta program/i }).click();
await page.locator('#beta-email').fill('beta-tester@example.com');
await page.locator('textarea').fill(
'I want to create a karaoke video for my favorite song!'
);
await page.locator('input[type="checkbox"]').check();
await page.getByRole('button', { name: /get my free credit/i }).click();
// Wait for redirect to /app
await page.waitForURL(/\/app/, { timeout: 10000 });
// Verify credits are displayed in the UI
// The AuthStatus component should show "1 credit" after fetchUser() completes
await expect(page.getByText(/1 credit/i)).toBeVisible({ timeout: 10000 });
});
});
test.describe('Authentication - Token Persistence', () => {
+388 -97
View File
@@ -373,6 +373,22 @@ github_actions_service_usage = gcp.projects.IAMMember(
member=github_actions_sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Project IAM Admin - required for Pulumi to refresh IAM binding state
github_actions_iam_admin = gcp.projects.IAMMember(
"github-actions-iam-admin",
project=project_id,
role="roles/resourcemanager.projectIamAdmin",
member=github_actions_sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant Secret Manager Admin - required for Pulumi to refresh secret state
github_actions_secretmanager = gcp.projects.IAMMember(
"github-actions-secretmanager",
project=project_id,
role="roles/secretmanager.admin",
member=github_actions_sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# ==================== Workload Identity Federation ====================
# Allows GitHub Actions to authenticate without service account keys
@@ -1168,6 +1184,14 @@ encoding_worker_logging_iam = gcp.projects.IAMMember(
member=encoding_worker_sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Grant encoding worker monitoring permissions (for Ops Agent metrics)
encoding_worker_monitoring_iam = gcp.projects.IAMMember(
"encoding-worker-monitoring-access",
project=project_id,
role="roles/monitoring.metricWriter",
member=encoding_worker_sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Static external IP for encoding worker (Cloud Run doesn't have VPC access by default)
encoding_worker_ip = gcp.compute.Address(
"encoding-worker-ip",
@@ -1189,6 +1213,13 @@ echo "Starting encoding worker setup at $(date)"
apt-get update
apt-get install -y python3-pip python3-venv docker.io curl git xz-utils
# Install fonts for ASS subtitle rendering (libass uses fontconfig)
# - fonts-noto: Noto Sans (default karaoke font)
# - fonts-noto-cjk: CJK character support
# - fontconfig: Font configuration system
apt-get install -y fonts-noto fonts-noto-cjk fontconfig
fc-cache -fv # Rebuild font cache
# Enable and start Docker
systemctl enable docker
systemctl start docker
@@ -1228,10 +1259,12 @@ fi
# Create the encoding worker service
cat > /opt/encoding-worker/main.py << 'PYTHON_CODE'
import asyncio
import glob as glob_module
import json
import logging
import os
import subprocess
import sys
import tempfile
import uuid
from concurrent.futures import ThreadPoolExecutor
@@ -1273,6 +1306,17 @@ class EncodeRequest(BaseModel):
output_gcs_path: str # gs://bucket/path/to/outputs/
encoding_config: dict # Video formats to generate
class EncodePreviewRequest(BaseModel):
job_id: str
ass_gcs_path: str # gs://bucket/path/to/subtitles.ass
audio_gcs_path: str # gs://bucket/path/to/audio.flac
output_gcs_path: str # gs://bucket/path/to/output.mp4
background_color: str = "black"
background_image_gcs_path: Optional[str] = None
font_gcs_path: Optional[str] = None # gs://bucket/path/to/custom-font.ttf
class JobStatus(BaseModel):
job_id: str
status: str # pending, running, complete, failed
@@ -1325,122 +1369,306 @@ def upload_to_gcs(local_path: Path, gcs_uri: str):
logger.info(f"Uploaded: {file} -> gs://{bucket_name}/{blob_name}")
def run_encoding(job_id: str, work_dir: Path, config: dict):
# Run FFmpeg encoding for all video formats
def download_single_file_from_gcs(gcs_uri: str, local_path: Path):
# Download a single file from GCS
parts = gcs_uri.replace("gs://", "").split("/", 1)
bucket_name = parts[0]
blob_name = parts[1] if len(parts) > 1 else ""
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(blob_name)
local_path.parent.mkdir(parents=True, exist_ok=True)
blob.download_to_filename(str(local_path))
logger.info(f"Downloaded: {gcs_uri} -> {local_path}")
def upload_single_file_to_gcs(local_path: Path, gcs_uri: str):
# Upload a single file to a specific GCS path
parts = gcs_uri.replace("gs://", "").split("/", 1)
bucket_name = parts[0]
blob_name = parts[1] if len(parts) > 1 else ""
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(blob_name)
blob.upload_from_filename(str(local_path))
logger.info(f"Uploaded: {local_path} -> {gcs_uri}")
def run_preview_encoding(job_id: str, work_dir: Path, request: "EncodePreviewRequest"):
# Run FFmpeg encoding for preview video (480x270, fast settings)
jobs[job_id]["status"] = "running"
jobs[job_id]["progress"] = 10
try:
# Find input files (search recursively since GCS downloads preserve directory structure)
input_files = list(work_dir.glob("**/*.mkv")) + list(work_dir.glob("**/*.mp4")) + list(work_dir.glob("**/*.mov"))
# Exclude files in the outputs directory to avoid re-encoding previous outputs
input_files = [f for f in input_files if "outputs" not in str(f)]
if not input_files:
raise ValueError(f"No video files found in {work_dir}")
# Download input files
ass_path = work_dir / "subtitles.ass"
audio_path = work_dir / "audio.flac"
# Use first video file as input (With Vocals or Karaoke)
input_video = input_files[0]
logger.info(f"Using input video: {input_video}")
download_single_file_from_gcs(request.ass_gcs_path, ass_path)
jobs[job_id]["progress"] = 20
# Find instrumental audio (search recursively, case-insensitive patterns)
# GCS stores as: instrumental_clean.flac, instrumental_with_backing.flac
instrumental_files = (
list(work_dir.glob("**/*instrumental*.flac")) +
list(work_dir.glob("**/*instrumental*.wav")) +
list(work_dir.glob("**/*Instrumental*.flac")) +
list(work_dir.glob("**/*Instrumental*.wav"))
download_single_file_from_gcs(request.audio_gcs_path, audio_path)
jobs[job_id]["progress"] = 30
# Download background image if provided
bg_image_path = None
if request.background_image_gcs_path:
bg_image_path = work_dir / "background.png"
download_single_file_from_gcs(request.background_image_gcs_path, bg_image_path)
# Download custom font if provided and register with fontconfig
if request.font_gcs_path:
# Use standard fontconfig location that's already in the search path
fonts_dir = Path("/usr/local/share/fonts/custom")
fonts_dir.mkdir(parents=True, exist_ok=True)
font_filename = request.font_gcs_path.split("/")[-1]
font_path = fonts_dir / font_filename
download_single_file_from_gcs(request.font_gcs_path, font_path)
logger.info(f"Downloaded custom font: {font_path}")
# Update fontconfig cache so libass can find the font
subprocess.run(["fc-cache", "-fv"], capture_output=True)
logger.info(f"Updated fontconfig cache with custom font: {font_filename}")
# Build FFmpeg command
output_path = work_dir / "preview.mp4"
# Escape special characters in path for FFmpeg filter syntax
# FFmpeg filter parsing requires escaping: \\ : , [ ] ;
def escape_ffmpeg_filter_path(path: str) -> str:
# Note: Extra escaping needed since this is inside a triple-quoted string in Pulumi
return path.replace("\\\\", "\\\\\\\\").replace(":", "\\\\:").replace(",", "\\\\,").replace("[", "\\\\[").replace("]", "\\\\]").replace(";", "\\\\;")
escaped_ass_path = escape_ffmpeg_filter_path(str(ass_path))
# Base command with frame rate
cmd = ["ffmpeg", "-y", "-r", "24"]
# Video input: background image or solid color
if bg_image_path and bg_image_path.exists():
cmd.extend(["-loop", "1", "-i", str(bg_image_path)])
# Scale and pad background to 480x270
vf = f"scale=480:270:force_original_aspect_ratio=decrease,pad=480:270:(ow-iw)/2:(oh-ih)/2,ass={escaped_ass_path}"
else:
# Solid color background
color = request.background_color or "black"
cmd.extend(["-f", "lavfi", "-i", f"color=c={color}:s=480x270:r=24"])
vf = f"ass={escaped_ass_path}"
# Audio input
cmd.extend(["-i", str(audio_path)])
# Video filter and encoding settings
cmd.extend([
"-vf", vf,
"-c:a", "aac", "-b:a", "96k",
"-c:v", "libx264",
"-preset", "superfast",
"-crf", "28",
"-pix_fmt", "yuv420p",
"-movflags", "+faststart",
"-threads", "8",
"-shortest",
str(output_path)
])
jobs[job_id]["progress"] = 40
logger.info(f"Running preview encoding: {' '.join(cmd)}")
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
logger.error(f"FFmpeg failed: {result.stderr}")
raise RuntimeError(f"FFmpeg preview encoding failed: {result.stderr[-500:]}")
jobs[job_id]["progress"] = 80
logger.info(f"Preview encoded: {output_path}")
# Upload output to GCS
upload_single_file_to_gcs(output_path, request.output_gcs_path)
jobs[job_id]["progress"] = 95
jobs[job_id]["output_files"] = [request.output_gcs_path]
return output_path
except Exception as e:
logger.error(f"Preview encoding failed: {e}")
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = str(e)
raise
def ensure_latest_wheel():
"""Download and install latest karaoke-gen wheel from GCS.
Called at the start of each job to enable hot code updates without restart.
In-progress jobs continue with their version, new jobs get latest code.
"""
try:
logger.info("Checking for latest karaoke-gen wheel in GCS...")
# Download latest wheel
result = subprocess.run(
["gsutil", "cp", "gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-*.whl", "/tmp/"],
capture_output=True, text=True, timeout=60
)
instrumental = instrumental_files[0] if instrumental_files else None
# Find the downloaded wheel (get the latest by version sorting)
wheels = glob_module.glob("/tmp/karaoke_gen-*.whl")
if not wheels:
logger.warning("No wheel found in GCS, using fallback encoding logic")
return False
# Sort to get latest version
wheel_path = sorted(wheels)[-1]
logger.info(f"Installing wheel: {wheel_path}")
# Install (or upgrade) the wheel
install_result = subprocess.run(
[sys.executable, "-m", "pip", "install", "--upgrade", "--quiet", wheel_path],
capture_output=True, text=True, timeout=120
)
if install_result.returncode != 0:
logger.warning(f"Wheel installation failed: {install_result.stderr}")
return False
logger.info(f"Successfully installed wheel: {wheel_path}")
return True
except subprocess.TimeoutExpired:
logger.warning("Wheel download/install timed out, using fallback")
return False
except Exception as e:
logger.warning(f"Failed to ensure latest wheel: {e}")
return False
def find_file(work_dir: Path, *patterns):
"""Find a file matching any of the given glob patterns."""
for pattern in patterns:
matches = list(work_dir.glob(f"**/{pattern}"))
if matches:
return matches[0]
return None
def run_encoding(job_id: str, work_dir: Path, config: dict):
"""Run encoding using LocalEncodingService (single source of truth).
Uses LocalEncodingService from the installed karaoke-gen wheel to ensure
output files match local CLI exactly:
- Proper names like "Artist - Title (Final Karaoke Lossless 4k).mp4"
- Concatenated title + karaoke + end screens
- All formats: lossless 4K MP4, lossy 4K MP4, lossless MKV, 720p MP4
Requires the karaoke-gen wheel to be installed (done by ensure_latest_wheel).
"""
jobs[job_id]["status"] = "running"
jobs[job_id]["progress"] = 10
try:
# Import LocalEncodingService from installed wheel (required, no fallback)
from backend.services.local_encoding_service import LocalEncodingService, EncodingConfig
logger.info("Using LocalEncodingService from installed wheel")
# Get artist/title from config for proper naming
artist = config.get("artist", "Unknown Artist")
title = config.get("title", "Unknown Title")
base_name = f"{artist} - {title}"
logger.info(f"Encoding for: {base_name}")
# Find input files in work_dir
# Title/end screens are in screens/ subdirectory
title_video = find_file(work_dir, "screens/title.mov", "*Title*.mov", "*title*.mov")
end_video = find_file(work_dir, "screens/end.mov", "*End*.mov", "*end*.mov")
# Karaoke video - search for With Vocals or main karaoke video
karaoke_video = find_file(
work_dir,
"*With Vocals*.mov", "*With Vocals*.mkv",
"*Vocals*.mov", "*Vocals*.mkv",
"*.mkv", "*.mov"
)
# Exclude title/end/output videos
if karaoke_video:
name_lower = karaoke_video.name.lower()
if "title" in name_lower or "end" in name_lower or "outputs" in str(karaoke_video):
# Search more specifically for karaoke video
karaoke_video = find_file(work_dir, "*Karaoke*.mkv", "*Karaoke*.mov", "*vocals*.mkv")
# Instrumental audio
instrumental = find_file(
work_dir,
"*instrumental_clean*.flac", "*Instrumental Clean*.flac",
"*instrumental*.flac", "*Instrumental*.flac",
"*instrumental*.wav"
)
logger.info(f"Found files:")
logger.info(f" Title video: {title_video}")
logger.info(f" Karaoke video: {karaoke_video}")
logger.info(f" End video: {end_video}")
logger.info(f" Instrumental: {instrumental}")
# Validate required files
if not title_video:
raise ValueError(f"No title video found in {work_dir}. Check screens/ subdirectory.")
if not karaoke_video:
raise ValueError(f"No karaoke video found in {work_dir}")
if not instrumental:
raise ValueError(f"No instrumental audio found in {work_dir}")
output_dir = work_dir / "outputs"
output_dir.mkdir(exist_ok=True)
output_files = []
formats = config.get("formats", ["mp4_4k", "mp4_720p"])
total_formats = len(formats)
jobs[job_id]["progress"] = 20
for i, fmt in enumerate(formats):
progress = 10 + int((i / total_formats) * 80)
jobs[job_id]["progress"] = progress
# Build encoding config with proper file names
encoding_config = EncodingConfig(
title_video=str(title_video),
karaoke_video=str(karaoke_video),
instrumental_audio=str(instrumental),
end_video=str(end_video) if end_video else None,
output_karaoke_mp4=str(output_dir / f"{base_name} (Karaoke).mp4"),
output_with_vocals_mp4=str(output_dir / f"{base_name} (With Vocals).mp4"),
output_lossless_4k_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossless 4k).mp4"),
output_lossy_4k_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossy 4k).mp4"),
output_lossless_mkv=str(output_dir / f"{base_name} (Final Karaoke Lossless 4k).mkv"),
output_720p_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossy 720p).mp4"),
)
if fmt == "mp4_4k_lossless" or fmt == "mp4_4k":
# High quality 4K - CRF 18 (visually lossless)
output_file = output_dir / "output_4k_lossless.mp4"
cmd = [
"ffmpeg", "-y",
"-i", str(input_video),
]
if instrumental:
cmd.extend(["-i", str(instrumental)])
cmd.extend([
"-map", "0:v",
"-map", "1:a" if instrumental else "0:a",
"-c:v", "libx264",
"-preset", "medium",
"-crf", "18",
"-c:a", "aac",
"-b:a", "256k",
"-threads", "24",
str(output_file)
])
elif fmt == "mp4_4k_lossy":
# Smaller 4K file - CRF 23 (good quality, smaller size)
output_file = output_dir / "output_4k_lossy.mp4"
cmd = [
"ffmpeg", "-y",
"-i", str(input_video),
]
if instrumental:
cmd.extend(["-i", str(instrumental)])
cmd.extend([
"-map", "0:v",
"-map", "1:a" if instrumental else "0:a",
"-c:v", "libx264",
"-preset", "medium",
"-crf", "23",
"-c:a", "aac",
"-b:a", "192k",
"-threads", "24",
str(output_file)
])
elif fmt == "mp4_720p":
output_file = output_dir / "output_720p.mp4"
cmd = [
"ffmpeg", "-y",
"-i", str(input_video),
]
if instrumental:
cmd.extend(["-i", str(instrumental)])
cmd.extend([
"-map", "0:v",
"-map", "1:a" if instrumental else "0:a",
"-vf", "scale=-1:720",
"-c:v", "libx264",
"-preset", "medium",
"-crf", "20",
"-c:a", "aac",
"-b:a", "192k",
"-threads", "24",
str(output_file)
])
else:
logger.warning(f"Unknown format: {fmt}")
continue
# Create service and run encoding
service = LocalEncodingService(logger=logger)
logger.info(f"Running: {' '.join(cmd)}")
result = subprocess.run(cmd, capture_output=True, text=True)
jobs[job_id]["progress"] = 30
logger.info("Starting LocalEncodingService.encode_all_formats()")
if result.returncode != 0:
logger.error(f"FFmpeg failed: {result.stderr}")
raise RuntimeError(f"FFmpeg encoding failed: {result.stderr[-500:]}")
result = service.encode_all_formats(encoding_config)
output_files.append(str(output_file))
logger.info(f"Encoded: {output_file}")
if not result.success:
raise RuntimeError(f"Encoding failed: {result.error}")
jobs[job_id]["output_files"] = output_files
jobs[job_id]["progress"] = 90
# Collect output files
output_files = [str(f) for f in output_dir.glob("*") if f.is_file()]
jobs[job_id]["output_files"] = output_files
logger.info(f"Encoding complete. Output files: {output_files}")
return output_dir
except ImportError as e:
# No fallback - wheel must be installed
error_msg = (
f"LocalEncodingService not available: {e}. "
"The karaoke-gen wheel must be installed. "
"Check that ensure_latest_wheel() succeeded and wheel exists in GCS."
)
logger.error(error_msg)
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = error_msg
raise RuntimeError(error_msg) from e
except Exception as e:
logger.error(f"Encoding failed: {e}")
jobs[job_id]["status"] = "failed"
@@ -1451,6 +1679,10 @@ def run_encoding(job_id: str, work_dir: Path, config: dict):
async def process_job(job_id: str, request: EncodeRequest):
# Process an encoding job asynchronously
try:
# Download and install latest wheel at job start (allows hot updates without restart)
# This means in-progress jobs continue with their version, new jobs get latest code
ensure_latest_wheel()
with tempfile.TemporaryDirectory() as temp_dir:
work_dir = Path(temp_dir) / "work"
work_dir.mkdir()
@@ -1477,7 +1709,7 @@ async def process_job(job_id: str, request: EncodeRequest):
# Convert local paths to blob paths (backend expects blob paths, not full gs:// URIs)
# output_gcs_path is like "gs://bucket/jobs/id/encoded/"
# We need paths like "jobs/id/encoded/output_4k_lossless.mp4"
# We need paths like "jobs/id/encoded/Artist - Title (Final Karaoke Lossless 4k).mp4"
gcs_path = request.output_gcs_path.replace("gs://", "")
parts = gcs_path.split("/", 1)
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
@@ -1500,6 +1732,65 @@ async def process_job(job_id: str, request: EncodeRequest):
jobs[job_id]["error"] = str(e)
async def process_preview_job(job_id: str, request: EncodePreviewRequest):
# Process a preview encoding job asynchronously
try:
with tempfile.TemporaryDirectory() as temp_dir:
work_dir = Path(temp_dir) / "work"
work_dir.mkdir()
# Run preview encoding in thread pool (CPU-bound)
# Note: run_preview_encoding handles download/upload internally
loop = asyncio.get_event_loop()
await loop.run_in_executor(
executor,
run_preview_encoding,
job_id,
work_dir,
request
)
jobs[job_id]["status"] = "complete"
jobs[job_id]["progress"] = 100
logger.info(f"Preview job {job_id} complete")
except Exception as e:
logger.error(f"Preview job {job_id} failed: {e}")
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = str(e)
@app.post("/encode-preview")
async def submit_preview_encode_job(request: EncodePreviewRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
# Submit a preview encoding job
job_id = request.job_id
# If job already exists, return cached result or current status
if job_id in jobs:
existing_job = jobs[job_id]
if existing_job["status"] == "complete":
# Return cached result - preview already encoded
return {"status": "cached", "job_id": job_id, "output_path": existing_job.get("output_path")}
elif existing_job["status"] == "failed":
# Previous attempt failed, allow retry by replacing the job
pass
else:
# Job is still in progress
return {"status": "in_progress", "job_id": job_id}
jobs[job_id] = {
"job_id": job_id,
"status": "pending",
"progress": 0,
"error": None,
"output_files": None,
}
background_tasks.add_task(process_preview_job, job_id, request)
return {"status": "accepted", "job_id": job_id}
@app.post("/encode")
async def submit_encode_job(request: EncodeRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
# Submit an encoding job
+9 -4
View File
@@ -156,15 +156,20 @@ class VideoBackgroundProcessor:
"""
Escape a file path for use in ffmpeg filter expressions.
FFmpeg filter syntax uses single quotes to protect special characters
like spaces, colons, and semicolons. Single quotes within the path
are escaped using the '\\'' pattern (end quote, literal quote, start quote).
Args:
path: File path to escape
Returns:
str: Escaped path
str: Escaped path (wrapped in single quotes)
"""
# Escape backslashes and colons for ffmpeg filter syntax
escaped = path.replace("\\", "\\\\").replace(":", "\\:")
return escaped
# Escape single quotes: ' becomes '\'' (end quote, \', start quote)
escaped = path.replace("'", "'\\''")
# Wrap entire path in single quotes
return f"'{escaped}'"
def build_video_filter(self, ass_subtitles_path, darkness_percent, fonts_dir=None):
"""
@@ -35,10 +35,14 @@ class ProviderConfig:
circuit_breaker_failure_threshold: int = 3
circuit_breaker_open_seconds: int = 60
# Initialization timeouts - fail fast instead of hanging forever
# These are separate from request_timeout to catch connection establishment issues
initialization_timeout_seconds: float = 30.0 # Model creation + warm-up
warmup_timeout_seconds: float = 15.0 # Just the warm-up call
# Initialization timeout - fail fast instead of hanging forever
# This is separate from request_timeout to catch connection establishment issues
initialization_timeout_seconds: float = 30.0 # Model creation
# Parallel processing settings
# Process multiple gaps concurrently to reduce total correction time
# Set to 1 to disable parallelism, higher values increase throughput but may hit rate limits
max_parallel_gaps: int = 5
@staticmethod
def from_env(cache_dir: Optional[str] = None) -> "ProviderConfig":
@@ -70,7 +74,7 @@ class ProviderConfig:
circuit_breaker_failure_threshold=int(os.getenv("AGENTIC_CIRCUIT_THRESHOLD", "3")),
circuit_breaker_open_seconds=int(os.getenv("AGENTIC_CIRCUIT_OPEN_SECONDS", "60")),
initialization_timeout_seconds=float(os.getenv("AGENTIC_INIT_TIMEOUT_SECONDS", "30.0")),
warmup_timeout_seconds=float(os.getenv("AGENTIC_WARMUP_TIMEOUT_SECONDS", "15.0")),
max_parallel_gaps=int(os.getenv("AGENTIC_MAX_PARALLEL_GAPS", "5")),
)
def validate_environment(self, logger: Optional[object] = None) -> None:
@@ -97,7 +97,6 @@ class LangChainBridge(BaseAIProvider):
# Lazy-initialized chat model
self._chat_model: Optional[Any] = None
self._warmed_up: bool = False
def name(self) -> str:
"""Return provider name for logging."""
@@ -164,13 +163,7 @@ class LangChainBridge(BaseAIProvider):
) from None
init_elapsed = time.time() - init_start
logger.info(f"🤖 Model created in {init_elapsed:.2f}s, starting warm-up...")
# Warm up the model to establish connection before real work
self._warm_up_model()
total_elapsed = time.time() - init_start
logger.info(f"🤖 Model initialization complete in {total_elapsed:.2f}s")
logger.info(f"🤖 Model initialized in {init_elapsed:.2f}s")
except InitializationTimeoutError as e:
self._circuit_breaker.record_failure(self._model)
@@ -271,47 +264,4 @@ class LangChainBridge(BaseAIProvider):
return content
def _warm_up_model(self) -> None:
"""Send a lightweight request to warm up the model connection.
This helps establish the REST connection and potentially warm up any
server-side resources before processing real correction requests.
The warm-up uses a timeout to fail fast if the service is unresponsive.
"""
if self._warmed_up:
return
timeout = self._config.warmup_timeout_seconds
# Use print with flush=True for visibility when output is redirected
print(f"🔥 Warming up {self._model} connection (timeout: {timeout}s)...", flush=True)
logger.info(f"🔥 Warming up {self._model} connection (timeout: {timeout}s)...")
warmup_start = time.time()
try:
from langchain_core.messages import HumanMessage
# Minimal prompt that requires almost no processing
warm_up_prompt = 'Respond with exactly: {"status":"ready"}'
# Use ThreadPoolExecutor for timeout on warm-up call
with ThreadPoolExecutor(max_workers=1) as executor:
future = executor.submit(
self._chat_model.invoke,
[HumanMessage(content=warm_up_prompt)]
)
try:
future.result(timeout=timeout)
except FuturesTimeoutError:
raise TimeoutError(f"Warm-up timed out after {timeout}s") from None
elapsed = time.time() - warmup_start
self._warmed_up = True
print(f"🔥 Warm-up complete for {self._model} in {elapsed:.2f}s", flush=True)
logger.info(f"🔥 Warm-up complete for {self._model} in {elapsed:.2f}s")
except Exception as e:
elapsed = time.time() - warmup_start
# Don't fail the actual request if warm-up fails
# Just log and continue - the real request might still work
print(f"🔥 Warm-up failed for {self._model} after {elapsed:.2f}s: {e} (continuing anyway)", flush=True)
logger.warning(f"🔥 Warm-up failed for {self._model} after {elapsed:.2f}s: {e} (continuing anyway)")
@@ -268,12 +268,26 @@ class LyricsCorrector:
_adapt = None
_ModelRouter = None
# Pre-initialized agentic corrector (created once, reused for all gaps)
_agentic_agent = None
if use_agentic_env:
try:
from lyrics_transcriber.correction.agentic.agent import AgenticCorrector as _AgenticCorrector
from lyrics_transcriber.correction.agentic.adapter import adapt_proposals_to_word_corrections as _adapt
from lyrics_transcriber.correction.agentic.router import ModelRouter as _ModelRouter
self.logger.info("🤖 Agentic modules imported successfully - running in AGENTIC-ONLY mode")
# Create agent ONCE and reuse for all gaps (avoids repeated model initialization)
_router = _ModelRouter()
model_id = _router.choose_model("gap", uncertainty=0.5) # Use default uncertainty
self.logger.info(f"🤖 Creating single AgenticCorrector with model: {model_id}")
_agentic_agent = _AgenticCorrector.from_model(
model=model_id,
session_id=session_id,
cache_dir=str(self._cache_dir)
)
self.logger.info("🤖 AgenticCorrector initialized and ready for all gaps")
except Exception as e:
self.logger.error(f"🤖 Failed to import agentic modules but USE_AGENTIC_AI=1: {e}")
raise RuntimeError(f"Agentic AI correction is enabled but required modules could not be imported: {e}") from e
@@ -443,16 +457,186 @@ class LyricsCorrector:
sys.exit(0)
# === END TEMPORARY CODE ===
# AGENTIC-ONLY MODE: Process all gaps in parallel for better performance
if use_agentic_env:
from concurrent.futures import ThreadPoolExecutor, as_completed
from lyrics_transcriber.correction.agentic.providers.config import ProviderConfig
# Get parallel processing config
_config = ProviderConfig.from_env()
max_workers = _config.max_parallel_gaps
self.logger.info(f"🤖 Processing {len(gap_sequences)} gaps in parallel (max_workers={max_workers})")
# Pre-compute shared data structures once (not per-gap)
all_transcribed_words = []
for seg in segments:
all_transcribed_words.extend(seg.words)
word_position = {w.id: idx for idx, w in enumerate(all_transcribed_words)}
# Build reference contexts once (same for all gaps)
reference_contexts = {}
for source, lyrics_data in self.reference_lyrics.items():
if lyrics_data and lyrics_data.segments:
ref_words = []
for seg in lyrics_data.segments:
ref_words.extend([w.text for w in seg.words])
reference_contexts[source] = " ".join(ref_words)
# Get artist and title once
artist = metadata.get("artist") if metadata else None
title = metadata.get("title") if metadata else None
# Prepare all gap inputs upfront
gap_inputs = []
for i, gap in enumerate(gap_sequences, 1):
# Prepare gap words data
gap_words_data = []
for word_id in gap.transcribed_word_ids:
if word_id in word_map:
word = word_map[word_id]
gap_words_data.append({
"id": word_id,
"text": word.text,
"start_time": getattr(word, 'start_time', 0),
"end_time": getattr(word, 'end_time', 0)
})
# Compute context words
gap_positions = [word_position[wid] for wid in gap.transcribed_word_ids if wid in word_position]
preceding_words = ""
following_words = ""
if gap_positions:
first_gap_pos = min(gap_positions)
last_gap_pos = max(gap_positions)
# Get 10 words before
start_pos = max(0, first_gap_pos - 10)
preceding_list = [all_transcribed_words[idx].text for idx in range(start_pos, first_gap_pos) if idx < len(all_transcribed_words)]
preceding_words = " ".join(preceding_list)
# Get 10 words after
end_pos = min(len(all_transcribed_words), last_gap_pos + 11)
following_list = [all_transcribed_words[idx].text for idx in range(last_gap_pos + 1, end_pos) if idx < len(all_transcribed_words)]
following_words = " ".join(following_list)
gap_inputs.append({
'index': i,
'gap': gap,
'gap_id': f"gap_{i}",
'gap_words': gap_words_data,
'preceding_words': preceding_words,
'following_words': following_words,
'reference_contexts': reference_contexts,
'artist': artist,
'title': title
})
# Function to process a single gap (runs in thread pool)
def process_single_gap(gap_input):
"""Process a single gap and return proposals. Thread-safe."""
idx = gap_input['index']
try:
proposals = _agentic_agent.propose_for_gap(
gap_id=gap_input['gap_id'],
gap_words=gap_input['gap_words'],
preceding_words=gap_input['preceding_words'],
following_words=gap_input['following_words'],
reference_contexts=gap_input['reference_contexts'],
artist=gap_input['artist'],
title=gap_input['title']
)
return {'index': idx, 'gap': gap_input['gap'], 'proposals': proposals, 'error': None}
except Exception as e:
return {'index': idx, 'gap': gap_input['gap'], 'proposals': None, 'error': str(e)}
# Process gaps in parallel
results = [None] * len(gap_inputs)
completed_count = 0
errors = []
with ThreadPoolExecutor(max_workers=max_workers) as executor:
# Submit all tasks
future_to_input = {executor.submit(process_single_gap, g): g for g in gap_inputs}
# Collect results as they complete
for future in as_completed(future_to_input):
# Check deadline
if deadline and time.time() > deadline:
self.logger.warning(
f"⏰ AGENTIC TIMEOUT: Deadline exceeded after processing {completed_count}/{len(gap_sequences)} gaps. "
"Cancelling remaining gaps - human review will correct any issues."
)
# Cancel remaining futures (use list() to avoid mutating dict during iteration)
for f in list(future_to_input.keys()):
f.cancel()
break
result = future.result()
idx = result['index'] - 1 # Convert 1-based to 0-based
results[idx] = result
completed_count += 1
if result['error']:
errors.append(f"Gap {result['index']}: {result['error']}")
self.logger.error(f"🤖 Gap {result['index']} failed: {result['error']}")
else:
proposal_count = len(result['proposals']) if result['proposals'] else 0
self.logger.info(f"🤖 Gap {result['index']}/{len(gap_sequences)} completed ({proposal_count} proposals)")
self.logger.info(f"🤖 Parallel processing complete: {completed_count}/{len(gap_sequences)} gaps processed")
# If any errors occurred, fail fast
if errors:
raise RuntimeError(f"Agentic AI correction failed for {len(errors)} gaps: {'; '.join(errors)}")
# Apply corrections sequentially (must be in order due to segment modifications)
for result in results:
if result is None:
continue # Skipped due to deadline
i = result['index']
gap = result['gap']
_proposals = result['proposals']
_agentic_corrections = _adapt(_proposals, word_map, linear_position_map) if _proposals else []
if _agentic_corrections:
self.logger.info(f"🤖 Applying {len(_agentic_corrections)} agentic corrections for gap {i}")
affected_word_ids = [w.id for w in self._get_affected_words(gap, segments)]
affected_segment_ids = [s.id for s in self._get_affected_segments(gap, segments)]
updated_segments = self._apply_corrections_to_segments(self._get_affected_segments(gap, segments), _agentic_corrections)
for correction in _agentic_corrections:
if correction.word_id and correction.corrected_word_id:
word_id_map[correction.word_id] = correction.corrected_word_id
for old_seg, new_seg in zip(self._get_affected_segments(gap, segments), updated_segments):
segment_id_map[old_seg.id] = new_seg.id
step = CorrectionStep(
handler_name="AgenticCorrector",
affected_word_ids=affected_word_ids,
affected_segment_ids=affected_segment_ids,
corrections=_agentic_corrections,
segments_before=self._get_affected_segments(gap, segments),
segments_after=updated_segments,
created_word_ids=[w.id for w in self._get_new_words(updated_segments, affected_word_ids)],
deleted_word_ids=[id for id in affected_word_ids if not self._word_exists(id, updated_segments)],
)
correction_steps.append(step)
all_corrections.extend(_agentic_corrections)
# Log corrections made
for correction in _agentic_corrections:
self.logger.info(
f"Made correction: '{correction.original_word}' -> '{correction.corrected_word}' "
f"(confidence: {correction.confidence:.2f}, reason: {correction.reason})"
)
else:
self.logger.debug(f"🤖 No agentic corrections needed for gap {i}")
# RULE-BASED MODE: Process gaps sequentially
for i, gap in enumerate(gap_sequences, 1):
# Check deadline before processing each gap (agentic mode only)
# This allows us to abort early and return uncorrected results for human review
if deadline and use_agentic_env and time.time() > deadline:
self.logger.warning(
f"⏰ AGENTIC TIMEOUT: Deadline exceeded after processing {i-1}/{len(gap_sequences)} gaps. "
"Skipping remaining gaps - human review will correct any issues."
)
# Break out of loop - continue with whatever corrections we have (likely none)
break
# Skip if we already processed in agentic mode
if use_agentic_env:
continue
self.logger.info(f"Processing gap {i}/{len(gap_sequences)} at position {gap.transcription_position}")
@@ -460,128 +644,6 @@ class LyricsCorrector:
gap_words = [word_map[word_id] for word_id in gap.transcribed_word_ids]
self.logger.debug(f"Gap text: '{' '.join(w.text for w in gap_words)}'")
# AGENTIC-ONLY MODE: Use agentic correction exclusively
if use_agentic_env:
self.logger.info(f"🤖 Attempting agentic correction for gap {i}/{len(gap_sequences)}")
try:
# Prepare gap data for classification-first workflow
gap_words_data = []
for word_id in gap.transcribed_word_ids:
if word_id in word_map:
word = word_map[word_id]
gap_words_data.append({
"id": word_id,
"text": word.text,
"start_time": getattr(word, 'start_time', 0),
"end_time": getattr(word, 'end_time', 0)
})
# Get context words
all_transcribed_words = []
for seg in segments:
all_transcribed_words.extend(seg.words)
word_position = {w.id: idx for idx, w in enumerate(all_transcribed_words)}
gap_positions = [word_position[wid] for wid in gap.transcribed_word_ids if wid in word_position]
preceding_words = ""
following_words = ""
if gap_positions:
first_gap_pos = min(gap_positions)
last_gap_pos = max(gap_positions)
# Get 10 words before
start_pos = max(0, first_gap_pos - 10)
preceding_list = [all_transcribed_words[idx].text for idx in range(start_pos, first_gap_pos) if idx < len(all_transcribed_words)]
preceding_words = " ".join(preceding_list)
# Get 10 words after
end_pos = min(len(all_transcribed_words), last_gap_pos + 11)
following_list = [all_transcribed_words[idx].text for idx in range(last_gap_pos + 1, end_pos) if idx < len(all_transcribed_words)]
following_words = " ".join(following_list)
# Get reference contexts from all sources
reference_contexts = {}
for source, lyrics_data in self.reference_lyrics.items():
if lyrics_data and lyrics_data.segments:
ref_words = []
for seg in lyrics_data.segments:
ref_words.extend([w.text for w in seg.words])
# For now, use full text (handlers will extract relevant portions)
reference_contexts[source] = " ".join(ref_words)
# Get artist and title from metadata
artist = metadata.get("artist") if metadata else None
title = metadata.get("title") if metadata else None
# Choose model via router
_router = _ModelRouter()
uncertainty = 0.3 if len(gap_words_data) <= 2 else 0.7
model_id = _router.choose_model("gap", uncertainty)
self.logger.debug(f"🤖 Router selected model: {model_id}")
# Create agent and use new classification-first workflow
self.logger.debug(f"🤖 Creating AgenticCorrector with model: {model_id}")
_agent = _AgenticCorrector.from_model(
model=model_id,
session_id=session_id,
cache_dir=str(self._cache_dir)
)
# Use new propose_for_gap method
self.logger.debug(f"🤖 Calling agent.propose_for_gap() for gap {i}")
_proposals = _agent.propose_for_gap(
gap_id=f"gap_{i}",
gap_words=gap_words_data,
preceding_words=preceding_words,
following_words=following_words,
reference_contexts=reference_contexts,
artist=artist,
title=title
)
self.logger.debug(f"🤖 Agent returned {len(_proposals) if _proposals else 0} proposals")
_agentic_corrections = _adapt(_proposals, word_map, linear_position_map) if _proposals else []
self.logger.debug(f"🤖 Adapter returned {len(_agentic_corrections)} corrections")
if _agentic_corrections:
self.logger.info(f"🤖 Applying {len(_agentic_corrections)} agentic corrections for gap {i}")
affected_word_ids = [w.id for w in self._get_affected_words(gap, segments)]
affected_segment_ids = [s.id for s in self._get_affected_segments(gap, segments)]
updated_segments = self._apply_corrections_to_segments(self._get_affected_segments(gap, segments), _agentic_corrections)
for correction in _agentic_corrections:
if correction.word_id and correction.corrected_word_id:
word_id_map[correction.word_id] = correction.corrected_word_id
for old_seg, new_seg in zip(self._get_affected_segments(gap, segments), updated_segments):
segment_id_map[old_seg.id] = new_seg.id
step = CorrectionStep(
handler_name="AgenticCorrector",
affected_word_ids=affected_word_ids,
affected_segment_ids=affected_segment_ids,
corrections=_agentic_corrections,
segments_before=self._get_affected_segments(gap, segments),
segments_after=updated_segments,
created_word_ids=[w.id for w in self._get_new_words(updated_segments, affected_word_ids)],
deleted_word_ids=[id for id in affected_word_ids if not self._word_exists(id, updated_segments)],
)
correction_steps.append(step)
all_corrections.extend(_agentic_corrections)
# Log corrections made
for correction in _agentic_corrections:
self.logger.info(
f"Made correction: '{correction.original_word}' -> '{correction.corrected_word}' "
f"(confidence: {correction.confidence:.2f}, reason: {correction.reason})"
)
else:
self.logger.info(f"🤖 No agentic corrections needed for gap {i}")
except Exception as e:
# In agentic-only mode, fail fast instead of falling back
self.logger.error(f"🤖 Agentic correction failed for gap {i}: {e}", exc_info=True)
raise RuntimeError(f"Agentic AI correction failed for gap {i}: {e}") from e
# Skip rule-based handlers completely in agentic mode
continue
# RULE-BASED MODE: Try each handler in order
for handler in self.handlers:
handler_name = handler.__class__.__name__
@@ -274,11 +274,12 @@ class CorrectionOperations:
audio_filepath: str,
artist: Optional[str] = None,
title: Optional[str] = None,
logger: Optional[logging.Logger] = None
logger: Optional[logging.Logger] = None,
ass_only: bool = False,
) -> Dict[str, Any]:
"""
Generate a preview video with current corrections.
Args:
correction_result: Current correction result
updated_data: Updated correction data for preview
@@ -287,10 +288,12 @@ class CorrectionOperations:
artist: Optional artist name
title: Optional title
logger: Optional logger instance
ass_only: If True, generate only ASS subtitles without video encoding.
Useful when video encoding is offloaded to external service.
Returns:
Dict with status, preview_hash, and video_path
Dict with status, preview_hash, and video_path (or ass_path if ass_only)
Raises:
ValueError: If preview video generation fails
"""
@@ -338,15 +341,27 @@ class CorrectionOperations:
audio_filepath=audio_filepath,
artist=artist,
title=title,
ass_only=ass_only,
)
# When ass_only, we only need the ASS file (video encoding done externally)
if ass_only:
if not preview_outputs.ass:
raise ValueError("Preview ASS generation failed")
logger.info(f"Generated preview ASS: {preview_outputs.ass}")
return {
"status": "success",
"preview_hash": preview_hash,
"ass_path": preview_outputs.ass,
}
if not preview_outputs.video:
raise ValueError("Preview video generation failed")
logger.info(f"Generated preview video: {preview_outputs.video}")
return {
"status": "success",
"preview_hash": preview_hash,
"video_path": preview_outputs.video
"video_path": preview_outputs.video,
}
@@ -2,7 +2,7 @@
"name": "lyrics-transcriber-frontend",
"private": true,
"homepage": "https://nomadkaraoke.github.io/lyrics-transcriber-frontend",
"version": "0.83.0",
"version": "0.84.0",
"type": "module",
"scripts": {
"dev": "vite",
@@ -1 +1 @@
{"root":["./src/app.tsx","./src/api.ts","./src/main.tsx","./src/theme.ts","./src/types.ts","./src/validation.ts","./src/vite-env.d.ts","./src/components/aifeedbackmodal.tsx","./src/components/addlyricsmodal.tsx","./src/components/agenticcorrectionmetrics.tsx","./src/components/audioplayer.tsx","./src/components/correctedwordwithactions.tsx","./src/components/correctionannotationmodal.tsx","./src/components/correctiondetailcard.tsx","./src/components/correctionmetrics.tsx","./src/components/durationtimelineview.tsx","./src/components/editactionbar.tsx","./src/components/editmodal.tsx","./src/components/edittimelinesection.tsx","./src/components/editwordlist.tsx","./src/components/fileupload.tsx","./src/components/findreplacemodal.tsx","./src/components/header.tsx","./src/components/lyricsanalyzer.tsx","./src/components/metricsdashboard.tsx","./src/components/modeselectionmodal.tsx","./src/components/modeselector.tsx","./src/components/modelselector.tsx","./src/components/previewvideosection.tsx","./src/components/referenceview.tsx","./src/components/replacealllyricsmodal.tsx","./src/components/reviewchangesmodal.tsx","./src/components/segmentdetailsmodal.tsx","./src/components/timelineeditor.tsx","./src/components/timingoffsetmodal.tsx","./src/components/transcriptionview.tsx","./src/components/worddivider.tsx","./src/components/lyricssynchronizer/synccontrols.tsx","./src/components/lyricssynchronizer/timelinecanvas.tsx","./src/components/lyricssynchronizer/upcomingwordsbar.tsx","./src/components/lyricssynchronizer/index.tsx","./src/components/shared/constants.ts","./src/components/shared/styles.ts","./src/components/shared/types.ts","./src/components/shared/components/highlightedtext.tsx","./src/components/shared/components/sourceselector.tsx","./src/components/shared/components/word.tsx","./src/components/shared/hooks/usewordclick.ts","./src/components/shared/utils/keyboardhandlers.ts","./src/components/shared/utils/localstorage.ts","./src/components/shared/utils/referencelinecalculator.ts","./src/components/shared/utils/segmentoperations.ts","./src/components/shared/utils/timingutils.ts","./src/components/shared/utils/wordutils.ts","./src/hooks/usemanualsync.ts","./src/types/global.d.ts"],"version":"5.6.3"}
{"root":["./src/app.tsx","./src/api.ts","./src/main.tsx","./src/theme.ts","./src/types.ts","./src/validation.ts","./src/vite-env.d.ts","./src/components/aifeedbackmodal.tsx","./src/components/addlyricsmodal.tsx","./src/components/agenticcorrectionmetrics.tsx","./src/components/appheader.tsx","./src/components/audioplayer.tsx","./src/components/correctedwordwithactions.tsx","./src/components/correctionannotationmodal.tsx","./src/components/correctiondetailcard.tsx","./src/components/correctionmetrics.tsx","./src/components/durationtimelineview.tsx","./src/components/editactionbar.tsx","./src/components/editmodal.tsx","./src/components/edittimelinesection.tsx","./src/components/editwordlist.tsx","./src/components/fileupload.tsx","./src/components/findreplacemodal.tsx","./src/components/header.tsx","./src/components/lyricsanalyzer.tsx","./src/components/metricsdashboard.tsx","./src/components/modeselectionmodal.tsx","./src/components/modeselector.tsx","./src/components/modelselector.tsx","./src/components/previewvideosection.tsx","./src/components/referenceview.tsx","./src/components/replacealllyricsmodal.tsx","./src/components/reviewchangesmodal.tsx","./src/components/segmentdetailsmodal.tsx","./src/components/timelineeditor.tsx","./src/components/timingoffsetmodal.tsx","./src/components/transcriptionview.tsx","./src/components/worddivider.tsx","./src/components/lyricssynchronizer/synccontrols.tsx","./src/components/lyricssynchronizer/timelinecanvas.tsx","./src/components/lyricssynchronizer/upcomingwordsbar.tsx","./src/components/lyricssynchronizer/index.tsx","./src/components/shared/constants.ts","./src/components/shared/styles.ts","./src/components/shared/types.ts","./src/components/shared/components/highlightedtext.tsx","./src/components/shared/components/sourceselector.tsx","./src/components/shared/components/word.tsx","./src/components/shared/hooks/usewordclick.ts","./src/components/shared/utils/keyboardhandlers.ts","./src/components/shared/utils/localstorage.ts","./src/components/shared/utils/referencelinecalculator.ts","./src/components/shared/utils/segmentoperations.ts","./src/components/shared/utils/timingutils.ts","./src/components/shared/utils/wordutils.ts","./src/hooks/usemanualsync.ts","./src/types/global.d.ts"],"version":"5.6.3"}
@@ -3,6 +3,11 @@ import react from '@vitejs/plugin-react';
// https://vite.dev/config/
export default defineConfig({
plugins: [react()],
server: {
port: 5173,
// Enable CORS for local development
cors: true,
},
build: {
minify: false,
sourcemap: true,
@@ -1 +0,0 @@
{"version":3,"file":"index-BECn1o8Q.js","sources":["../../node_modules/react/cjs/react.production.min.js","../../node_modules/react/index.js","../../node_modules/react/cjs/react-jsx-runtime.production.min.js","../../node_modules/react/jsx-runtime.js","../../node_modules/scheduler/cjs/scheduler.production.min.js","../../node_modules/scheduler/index.js","../../node_modules/react-dom/cjs/react-dom.production.min.js","../../node_modules/react-dom/index.js","../../node_modules/react-dom/client.js","../../node_modules/@mui/utils/esm/formatMuiErrorMessage/formatMuiErrorMessage.js","../../node_modules/@mui/material/styles/identifier.js","../../node_modules/@babel/runtime/helpers/esm/extends.js","../../node_modules/@emotion/sheet/dist/emotion-sheet.esm.js","../../node_modules/stylis/src/Enum.js","../../node_modules/stylis/src/Utility.js","../../node_modules/stylis/src/Tokenizer.js","../../node_modules/stylis/src/Parser.js","../../node_modules/stylis/src/Serializer.js","../../node_modules/stylis/src/Middleware.js","../../node_modules/@emotion/memoize/dist/emotion-memoize.esm.js","../../node_modules/@emotion/cache/dist/emotion-cache.browser.esm.js","../../node_modules/react-is/cjs/react-is.production.min.js","../../node_modules/react-is/index.js","../../node_modules/hoist-non-react-statics/dist/hoist-non-react-statics.cjs.js","../../node_modules/@emotion/utils/dist/emotion-utils.browser.esm.js","../../node_modules/@emotion/hash/dist/emotion-hash.esm.js","../../node_modules/@emotion/unitless/dist/emotion-unitless.esm.js","../../node_modules/@emotion/serialize/dist/emotion-serialize.esm.js","../../node_modules/@emotion/use-insertion-effect-with-fallbacks/dist/emotion-use-insertion-effect-with-fallbacks.browser.esm.js","../../node_modules/@emotion/react/dist/emotion-element-f0de968e.browser.esm.js","../../node_modules/@emotion/react/dist/emotion-react.browser.esm.js","../../node_modules/@emotion/is-prop-valid/dist/emotion-is-prop-valid.esm.js","../../node_modules/@emotion/styled/base/dist/emotion-styled-base.browser.esm.js","../../node_modules/@emotion/styled/dist/emotion-styled.browser.esm.js","../../node_modules/@mui/styled-engine/GlobalStyles/GlobalStyles.js","../../node_modules/@mui/styled-engine/index.js","../../node_modules/@mui/utils/node_modules/react-is/cjs/react-is.production.js","../../node_modules/@mui/utils/node_modules/react-is/index.js","../../node_modules/@mui/utils/esm/deepmerge/deepmerge.js","../../node_modules/@mui/system/esm/createBreakpoints/createBreakpoints.js","../../node_modules/@mui/system/esm/cssContainerQueries/cssContainerQueries.js","../../node_modules/@mui/system/esm/createTheme/shape.js","../../node_modules/@mui/system/esm/merge/merge.js","../../node_modules/@mui/system/esm/breakpoints/breakpoints.js","../../node_modules/@mui/utils/esm/capitalize/capitalize.js","../../node_modules/@mui/system/esm/style/style.js","../../node_modules/@mui/system/esm/memoize/memoize.js","../../node_modules/@mui/system/esm/spacing/spacing.js","../../node_modules/@mui/system/esm/createTheme/createSpacing.js","../../node_modules/@mui/system/esm/compose/compose.js","../../node_modules/@mui/system/esm/borders/borders.js","../../node_modules/@mui/system/esm/cssGrid/cssGrid.js","../../node_modules/@mui/system/esm/palette/palette.js","../../node_modules/@mui/system/esm/sizing/sizing.js","../../node_modules/@mui/system/esm/styleFunctionSx/defaultSxConfig.js","../../node_modules/@mui/system/esm/styleFunctionSx/styleFunctionSx.js","../../node_modules/@mui/system/esm/createTheme/applyStyles.js","../../node_modules/@mui/system/esm/createTheme/createTheme.js","../../node_modules/@mui/system/esm/useThemeWithoutDefault/useThemeWithoutDefault.js","../../node_modules/@mui/system/esm/useTheme/useTheme.js","../../node_modules/@mui/system/esm/GlobalStyles/GlobalStyles.js","../../node_modules/@mui/system/esm/styleFunctionSx/extendSxProp.js","../../node_modules/@mui/utils/esm/ClassNameGenerator/ClassNameGenerator.js","../../node_modules/clsx/dist/clsx.mjs","../../node_modules/@mui/system/esm/createBox/createBox.js","../../node_modules/@mui/utils/esm/generateUtilityClass/generateUtilityClass.js","../../node_modules/@mui/utils/esm/generateUtilityClasses/generateUtilityClasses.js","../../node_modules/@mui/system/esm/preprocessStyles.js","../../node_modules/@mui/system/esm/createStyled/createStyled.js","../../node_modules/@mui/system/esm/styled/styled.js","../../node_modules/@mui/utils/esm/resolveProps/resolveProps.js","../../node_modules/@mui/system/esm/useThemeProps/getThemeProps.js","../../node_modules/@mui/system/esm/useThemeProps/useThemeProps.js","../../node_modules/@mui/utils/esm/useEnhancedEffect/useEnhancedEffect.js","../../node_modules/@mui/system/esm/useMediaQuery/useMediaQuery.js","../../node_modules/@mui/utils/esm/clamp/clamp.js","../../node_modules/@mui/system/esm/colorManipulator/colorManipulator.js","../../node_modules/@mui/utils/esm/createChainedFunction/createChainedFunction.js","../../node_modules/@mui/utils/esm/debLine truncated
@@ -0,0 +1 @@
{"version":3,"file":"index-V91xgc4E.js","sources":["../../node_modules/react/cjs/react.production.min.js","../../node_modules/react/index.js","../../node_modules/react/cjs/react-jsx-runtime.production.min.js","../../node_modules/react/jsx-runtime.js","../../node_modules/scheduler/cjs/scheduler.production.min.js","../../node_modules/scheduler/index.js","../../node_modules/react-dom/cjs/react-dom.production.min.js","../../node_modules/react-dom/index.js","../../node_modules/react-dom/client.js","../../node_modules/@mui/utils/node_modules/react-is/cjs/react-is.production.js","../../node_modules/@mui/utils/node_modules/react-is/index.js","../../node_modules/@mui/utils/esm/deepmerge/deepmerge.js","../../node_modules/@mui/utils/esm/formatMuiErrorMessage/formatMuiErrorMessage.js","../../node_modules/@mui/utils/esm/capitalize/capitalize.js","../../node_modules/@mui/utils/esm/createChainedFunction/createChainedFunction.js","../../node_modules/@mui/utils/esm/debounce/debounce.js","../../node_modules/@mui/utils/esm/isMuiElement/isMuiElement.js","../../node_modules/@mui/utils/esm/ownerDocument/ownerDocument.js","../../node_modules/@mui/utils/esm/ownerWindow/ownerWindow.js","../../node_modules/@mui/utils/esm/setRef/setRef.js","../../node_modules/@mui/utils/esm/useEnhancedEffect/useEnhancedEffect.js","../../node_modules/@mui/utils/esm/useId/useId.js","../../node_modules/@mui/utils/esm/useControlled/useControlled.js","../../node_modules/@mui/utils/esm/useEventCallback/useEventCallback.js","../../node_modules/@mui/utils/esm/useForkRef/useForkRef.js","../../node_modules/@mui/utils/esm/useLazyRef/useLazyRef.js","../../node_modules/@mui/utils/esm/useOnMount/useOnMount.js","../../node_modules/@mui/utils/esm/useTimeout/useTimeout.js","../../node_modules/@mui/utils/esm/isFocusVisible/isFocusVisible.js","../../node_modules/@mui/utils/esm/getScrollbarSize/getScrollbarSize.js","../../node_modules/@mui/utils/esm/getValidReactChildren/getValidReactChildren.js","../../node_modules/@mui/utils/esm/visuallyHidden/visuallyHidden.js","../../node_modules/@mui/utils/esm/resolveProps/resolveProps.js","../../node_modules/@mui/utils/esm/composeClasses/composeClasses.js","../../node_modules/@mui/utils/esm/ClassNameGenerator/ClassNameGenerator.js","../../node_modules/@mui/utils/esm/generateUtilityClass/generateUtilityClass.js","../../node_modules/@mui/utils/esm/generateUtilityClasses/generateUtilityClasses.js","../../node_modules/@mui/utils/esm/clamp/clamp.js","../../node_modules/@mui/utils/esm/isHostComponent/isHostComponent.js","../../node_modules/@mui/utils/esm/appendOwnerState/appendOwnerState.js","../../node_modules/clsx/dist/clsx.mjs","../../node_modules/@mui/utils/esm/extractEventHandlers/extractEventHandlers.js","../../node_modules/@mui/utils/esm/omitEventHandlers/omitEventHandlers.js","../../node_modules/@mui/utils/esm/mergeSlotProps/mergeSlotProps.js","../../node_modules/@mui/utils/esm/resolveComponentProps/resolveComponentProps.js","../../node_modules/@mui/utils/esm/useSlotProps/useSlotProps.js","../../node_modules/@mui/utils/esm/getReactElementRef/getReactElementRef.js","../../node_modules/@mui/system/esm/merge/merge.js","../../node_modules/@mui/system/esm/cssContainerQueries/cssContainerQueries.js","../../node_modules/@mui/system/esm/breakpoints/breakpoints.js","../../node_modules/@mui/system/esm/style/style.js","../../node_modules/@mui/system/esm/memoize/memoize.js","../../node_modules/@mui/system/esm/spacing/spacing.js","../../node_modules/@mui/system/esm/compose/compose.js","../../node_modules/@mui/system/esm/borders/borders.js","../../node_modules/@mui/system/esm/cssGrid/cssGrid.js","../../node_modules/@mui/system/esm/palette/palette.js","../../node_modules/@mui/system/esm/sizing/sizing.js","../../node_modules/@mui/system/esm/styleFunctionSx/defaultSxConfig.js","../../node_modules/@mui/system/esm/styleFunctionSx/styleFunctionSx.js","../../node_modules/@mui/system/esm/styleFunctionSx/extendSxProp.js","../../node_modules/@babel/runtime/helpers/esm/extends.js","../../node_modules/@emotion/sheet/dist/emotion-sheet.esm.js","../../node_modules/stylis/src/Enum.js","../../node_modules/stylis/src/Utility.js","../../node_modules/stylis/src/Tokenizer.js","../../node_modules/stylis/src/Parser.js","../../node_modules/stylis/src/Serializer.js","../../node_modules/stylis/src/Middleware.js","../../node_modules/@emotion/memoize/dist/emotion-memoize.esm.js","../../node_modules/@emotion/cache/dist/emotion-cache.browser.esm.js","../../node_modules/react-is/cjs/react-is.production.min.js","../../node_modules/react-is/index.js","../../node_modules/hoist-non-react-statics/dist/hoist-non-react-statics.cjs.js","../../node_modules/@emotion/utils/dist/emotion-utils.browser.esm.js","../../node_modules/@emotion/hash/dist/emotion-hash.esm.js","../../node_modules/@emotion/unitless/dist/emotion-unitless.esm.js","../../node_modules/@emotion/serialize/dist/emotion-serialize.esm.js","../../node_modules/@emotion/use-insertion-effect-with-fallbacks/dist/emotion-useLine truncated
@@ -2,6 +2,10 @@
<html lang="en">
<head>
<meta charset="UTF-8" />
<!-- Load Inter font to match karaoke-gen styling -->
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700&display=swap" rel="stylesheet">
<link rel="icon" type="image/x-icon" href="/favicon.ico" />
<link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png" />
<link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png" />
@@ -9,8 +13,8 @@
<link rel="icon" type="image/png" sizes="192x192" href="/android-chrome-192x192.png" />
<link rel="icon" type="image/png" sizes="512x512" href="/android-chrome-512x512.png" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Nomad Karaoke: Lyrics Review</title>
<script type="module" crossorigin src="/assets/index-BECn1o8Q.js"></script>
<title>Nomad Karaoke: Lyrics Transcription Review</title>
<script type="module" crossorigin src="/assets/index-V91xgc4E.js"></script>
</head>
<body>
<div id="root"></div>
@@ -0,0 +1,5 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns="http://www.w3.org/2000/svg" version="1.1" viewBox="45 60 255 135">
<g data-v-70b83f88="" fill="#ff7acc" class="basesvg" transform="translate(48.368003845214844,62.62239074707031)"><g fill-rule="" class="tp-name" transform="translate(0,0)"><g transform="scale(1.4000000000000004)"><g stroke="#ff7acc" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" paint-order="stroke" data-gra-attr="stroke" fill-opacity="0"><path d="M1.4-33.44L1.4 0C31.48 0 3.96 0 34.04 0L34.04-32.64 17.81-32.64 17.81-20.61 1.4-33.44ZM52.21-33.76C42.6-33.76 34.81-25.97 34.81-16.37 34.81-6.76 42.6 1.03 52.21 1.03 61.81 1.03 69.6-6.76 69.6-16.37 69.6-25.97 61.81-33.76 52.21-33.76ZM70.7 0L103.34 0 103.25-33.9 86.93-21.54 70.61-33.9 70.7 0ZM102.34 0C138.2 0 107.52 0 143.56 0L122.95-34.93 102.34 0ZM158.84 0.05C167.84 0.05 175.16-7.27 175.16-16.27 175.16-25.27 167.84-32.64 158.84-32.64L142.52-32.64C142.52 0.75 142.52-33.34 142.52 0.05L158.84 0.05Z" transform="translate(-1.399999976158142, 34.93000030517578)"></path></g><g transform="translate(0,38.959999084472656)"><g stroke="#ff7acc" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" paint-order="stroke" data-gra-attr="stroke" fill="#ff7acc" fill-opacity="0" transform="scale(1.02719)"><path d="M24.43 0L17.67 0 7.98-11.87 7.98 0 2.63 0 2.63-26.64 7.98-26.64 7.98-14.69 17.67-26.64 24.12-26.64 13.13-13.44 24.43 0ZM45.03 0L43.27-5.08 32.66-5.08 30.91 0 25.3 0 34.88-26.68 41.1-26.68 50.68 0 45.03 0ZM34.11-9.35L41.82-9.35 37.97-20.5 34.11-9.35ZM73.19 0L67.01 0 61.13-10.38 58.61-10.38 58.61 0 53.27 0 53.27-26.64 63.27-26.64Q66.36-26.64 68.53-25.55 70.71-24.47 71.8-22.61 72.89-20.76 72.89-18.47L72.89-18.47Q72.89-15.84 71.36-13.72 69.83-11.6 66.82-10.8L66.82-10.8 73.19 0ZM58.61-22.21L58.61-14.39 63.08-14.39Q65.25-14.39 66.32-15.44 67.39-16.49 67.39-18.36L67.39-18.36Q67.39-20.19 66.32-21.2 65.25-22.21 63.08-22.21L63.08-22.21 58.61-22.21ZM94.82 0L93.07-5.08 82.46-5.08 80.7 0 75.09 0 84.67-26.68 90.89-26.68 100.47 0 94.82 0ZM83.91-9.35L91.62-9.35 87.76-20.5 83.91-9.35ZM115.43 0.27Q111.69 0.27 108.56-1.49 105.43-3.24 103.6-6.35 101.76-9.47 101.76-13.4L101.76-13.4Q101.76-17.29 103.6-20.4 105.43-23.51 108.56-25.27 111.69-27.02 115.43-27.02L115.43-27.02Q119.21-27.02 122.32-25.27 125.43-23.51 127.24-20.4 129.05-17.29 129.05-13.4L129.05-13.4Q129.05-9.47 127.24-6.35 125.43-3.24 122.3-1.49 119.17 0.27 115.43 0.27L115.43 0.27ZM115.43-4.5Q117.83-4.5 119.66-5.59 121.5-6.68 122.53-8.7 123.56-10.73 123.56-13.4L123.56-13.4Q123.56-16.07 122.53-18.07 121.5-20.08 119.66-21.15 117.83-22.21 115.43-22.21L115.43-22.21Q113.02-22.21 111.17-21.15 109.32-20.08 108.29-18.07 107.26-16.07 107.26-13.4L107.26-13.4Q107.26-10.73 108.29-8.7 109.32-6.68 111.17-5.59 113.02-4.5 115.43-4.5L115.43-4.5ZM153.82 0L147.06 0 137.37-11.87 137.37 0 132.02 0 132.02-26.64 137.37-26.64 137.37-14.69 147.06-26.64 153.51-26.64 142.52-13.44 153.82 0ZM171.79-22.33L161.67-22.33 161.67-15.65 170.64-15.65 170.64-11.41 161.67-11.41 161.67-4.35 171.79-4.35 171.79 0 156.33 0 156.33-26.68 171.79-26.68 171.79-22.33Z" transform="translate(-2.630000114440918, 27.020000457763672)"></path></g></g></g></g> <g data-gra="path-slogan" fill-rule="" class="tp-slogan" fill="#ffdf6b" transform="translate(5,111.788818359375)"><rect x="0" height="1" y="5.9832000732421875" width="14.278396606445312"></rect> <rect height="1" y="5.9832000732421875" width="14.278396606445312" x="218.985595703125"></rect> <g transform="translate(17.278396606445312,0)"><g transform="scale(1.2800000000000002)"><path d="M10.21-8.38L12.02-8.38L9.68 0L7.70 0L6.13-5.96L4.49 0L2.52 0.01L0.26-8.38L2.06-8.38L3.54-1.87L5.24-8.38L7.12-8.38L8.72-1.91L10.21-8.38ZM18.38-8.38L20.06-8.38L20.06 0L18.38 0L18.38-3.56L14.80-3.56L14.80 0L13.12 0L13.12-8.38L14.80-8.38L14.80-4.93L18.38-4.93L18.38-8.38ZM26.58-7.02L23.40-7.02L23.40-4.92L26.22-4.92L26.22-3.59L23.40-3.59L23.40-1.37L26.58-1.37L26.58 0L21.72 0L21.72-8.39L26.58-8.39L26.58-7.02ZM34.37 0L32.42 0L30.58-3.26L29.78-3.26L29.78 0L28.10 0L28.10-8.38L31.25-8.38Q32.22-8.38 32.90-8.03Q33.59-7.69 33.93-7.11Q34.27-6.53 34.27-5.81L34.27-5.81Q34.27-4.98 33.79-4.31Q33.31-3.65 32.36-3.40L32.36-3.40L34.37 0ZM29.78-6.98L29.78-4.52L31.19-4.52Q31.87-4.52 32.21-4.85Q32.54-5.18 32.54-5.77L32.54-5.77Q32.54-6.35 32.21-6.67Q31.87-6.98 31.19-6.98L31.19-6.98L29.78-6.98ZM40.66-7.02L37.48-7.02L37.48-4.92L40.30-4.92L40.30-3.59L37.48-3.59L37.48-1.37L40.66-1.37L40.66 0L35.80 0L35.80-8.39L40.66-8.39L40.66-7.02ZM47.92-8.38L49.70-8.38L46.63 0L44.59 0L41.52-8.38L43.32-8.38L45.62-1.72L47.92-8.38ZM55.57-7.02L52.39-7.02L52.39-4.92L55.21-4.92L55.21-3.59L52.39-3.59L52.39-1.37L55.57-1.37L55.57 0L50.71 0L50.71-8.39L55.57-8.39L55.57-7.02ZM63.36 0L61.42 0L59.57-3.26L58.78-3.26L58.78 0L57.10 0L57.10-8.38L60.24-8.38Q61.21-8.38 61.90-8.03Q62.58-7.69 62.92-7.11Q63.26-6.53 63.26-5.81L63.26-5.81Q63.26-4.98 62.78-4.31Q62.30-3.65 61.36-3.40L61.36-3.40L63.36 0ZM58.78-6.98L58.78-4.52L60.18-4.52Q60.86-4.52 61.20-4.85Q61.54-5.18 61.54-5.77L61.54-5.77Q61.54-6.35 61.20-6.67Q60.86-6.98 60.18-6.98L60.18-6.98L58.78-6.98ZM72.43-8.38L74.30-8.38L71.47-2.92L71.47 0L69.Line truncated
</svg>

After

Width:  |  Height:  |  Size: 9.0 KiB

@@ -148,8 +148,21 @@ class OutputGenerator:
audio_filepath: str,
artist: Optional[str] = None,
title: Optional[str] = None,
ass_only: bool = False,
) -> OutputPaths:
"""Generate all requested output formats."""
"""Generate all requested output formats.
Args:
transcription_corrected: Corrected transcription data
lyrics_results: Lyrics data from various providers
output_prefix: Prefix for output filenames
audio_filepath: Path to audio file
artist: Optional artist name
title: Optional title
ass_only: If True (only in preview_mode), generate only ASS subtitles
without video encoding. Useful when video encoding is offloaded
to an external service.
"""
outputs = OutputPaths()
try:
@@ -165,8 +178,9 @@ class OutputGenerator:
# Generate ASS subtitles for preview
outputs.ass = self.subtitle.generate_ass(transcription_corrected.resized_segments, output_prefix, audio_filepath)
# Generate preview video
outputs.video = self.video.generate_preview_video(outputs.ass, audio_filepath, output_prefix)
# Generate preview video (unless ass_only mode for GCE encoding)
if not ass_only:
outputs.video = self.video.generate_preview_video(outputs.ass, audio_filepath, output_prefix)
return outputs
@@ -352,19 +352,44 @@ class VideoGenerator:
self.logger.error(f"Failed to resize background image: {e.output}")
raise
def _escape_ffmpeg_filter_path(self, path: str) -> str:
"""Escape a path for FFmpeg filter expressions (for subprocess without shell).
When using subprocess with a command list (no shell), FFmpeg receives the
filter string directly. FFmpeg's filter parser requires escaping:
- Backslashes: double them (\ -> \\)
- Single quotes/apostrophes: escape with three backslashes (' -> \\')
- Spaces: escape with backslash ( -> \ )
Note: This is different from shell escaping. The '\\'\\''' pattern used for
shell escaping does NOT work when subprocess passes args directly to FFmpeg.
Example: "I'm With You" becomes "I\\\\'m\\ With\\ You"
"""
# First escape existing backslashes (\ -> \\)
escaped = path.replace("\\", "\\\\")
# Escape single quotes (' -> \\')
# In the actual string we need 3 backslashes before the quote
escaped = escaped.replace("'", "\\\\\\'")
# Escape spaces
escaped = escaped.replace(" ", "\\ ")
return escaped
def _build_ass_filter(self, ass_path: str) -> str:
"""Build ASS filter with font directory support."""
ass_filter = f"ass={ass_path}"
escaped_ass_path = self._escape_ffmpeg_filter_path(ass_path)
ass_filter = f"ass={escaped_ass_path}"
# Get font path from styles configuration
karaoke_styles = self.styles.get("karaoke", {})
font_path = karaoke_styles.get("font_path")
if font_path and os.path.isfile(font_path):
font_dir = os.path.dirname(font_path)
ass_filter += f":fontsdir={font_dir}"
escaped_font_dir = self._escape_ffmpeg_filter_path(font_dir)
ass_filter += f":fontsdir={escaped_font_dir}"
self.logger.info(f"Returning ASS filter with fonts dir: {ass_filter}")
return ass_filter
def _build_ffmpeg_command(self, ass_path: str, audio_path: str, output_path: str) -> List[str]:
@@ -210,6 +210,7 @@ def test_build_ass_filter_no_font_path(video_generator):
"""Test _build_ass_filter with no font path configured."""
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
# No special chars = no escaping needed
assert result == "ass=test.ass"
@@ -220,13 +221,16 @@ def test_build_ass_filter_with_valid_font_path(video_generator, tmp_path):
font_dir.mkdir()
font_file = font_dir / "test.ttf"
font_file.touch()
# Update styles to include font path
video_generator.styles["karaoke"]["font_path"] = str(font_file)
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
expected = f"ass=test.ass:fontsdir={str(font_dir)}"
# Paths with no special chars don't need escaping
# But font_dir path from tmp_path may have spaces on some systems
escaped_font_dir = str(font_dir).replace("\\", "\\\\").replace("'", "\\\\'").replace(" ", "\\ ")
expected = f"ass=test.ass:fontsdir={escaped_font_dir}"
assert result == expected
@@ -234,9 +238,10 @@ def test_build_ass_filter_with_invalid_font_path(video_generator):
"""Test _build_ass_filter with invalid font path (file doesn't exist)."""
# Set non-existent font path
video_generator.styles["karaoke"]["font_path"] = "/nonexistent/font.ttf"
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
# No special chars = no escaping
assert result == "ass=test.ass"
@@ -244,9 +249,10 @@ def test_build_ass_filter_with_empty_font_path(video_generator):
"""Test _build_ass_filter with empty font path."""
# Set empty font path
video_generator.styles["karaoke"]["font_path"] = ""
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
# No special chars = no escaping
assert result == "ass=test.ass"
@@ -254,9 +260,10 @@ def test_build_ass_filter_with_none_font_path(video_generator):
"""Test _build_ass_filter with None font path."""
# Set None font path
video_generator.styles["karaoke"]["font_path"] = None
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
# No special chars = no escaping
assert result == "ass=test.ass"
@@ -269,9 +276,10 @@ def test_build_ass_filter_no_karaoke_section(tmp_path):
video_resolution=(1920, 1080),
styles=styles,
)
ass_path = "test.ass"
result = generator._build_ass_filter(ass_path)
# No special chars = no escaping
assert result == "ass=test.ass"
@@ -279,19 +287,73 @@ def test_build_ass_filter_logging(video_generator, tmp_path, caplog):
"""Test _build_ass_filter logging behavior."""
# Set log level to INFO to capture the logs
caplog.set_level(logging.INFO)
# Create a test font file
font_dir = tmp_path / "fonts"
font_dir.mkdir()
font_file = font_dir / "test.ttf"
font_file.touch()
# Update styles to include font path
video_generator.styles["karaoke"]["font_path"] = str(font_file)
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
# Check that info log was generated
assert "Returning ASS filter with fonts dir:" in caplog.text
assert str(font_dir) in caplog.text
def test_escape_ffmpeg_filter_path(video_generator):
"""Test _escape_ffmpeg_filter_path uses FFmpeg filter escaping (not shell escaping).
FFmpeg filter escaping rules (for subprocess without shell):
- Backslashes: double them (\ -> \\)
- Single quotes/apostrophes: escape with three backslashes (' -> \\')
- Spaces: escape with backslash ( -> \ )
"""
# Test simple path without special chars (no escaping needed)
assert video_generator._escape_ffmpeg_filter_path("/path/to/file.ass") == "/path/to/file.ass"
# Test apostrophe escaping (fixes paths like "I'm With You")
# Apostrophe becomes \\\' (three backslashes + quote) in the actual string
result = video_generator._escape_ffmpeg_filter_path("./Avril Lavigne - I'm With You/file.ass")
assert result == "./Avril\\ Lavigne\\ -\\ I\\\\\\'m\\ With\\ You/file.ass"
# Test path with spaces (escaped with backslash)
assert video_generator._escape_ffmpeg_filter_path("/path/with spaces/file.ass") == "/path/with\\ spaces/file.ass"
# Test path with colons (no escaping needed - colon is only special in filter options)
assert video_generator._escape_ffmpeg_filter_path("C:/path/file.ass") == "C:/path/file.ass"
# Test multiple apostrophes
result = video_generator._escape_ffmpeg_filter_path("It's Bob's file.ass")
assert result == "It\\\\\\'s\\ Bob\\\\\\'s\\ file.ass"
def test_build_ass_filter_escapes_path_with_apostrophe(video_generator):
"""Test _build_ass_filter properly escapes paths containing apostrophes."""
ass_path = "./Avril Lavigne - I'm With You/cache/temp.ass"
result = video_generator._build_ass_filter(ass_path)
# Spaces escaped with backslash, apostrophe escaped with \\\'
assert result == "ass=./Avril\\ Lavigne\\ -\\ I\\\\\\'m\\ With\\ You/cache/temp.ass"
def test_build_ass_filter_escapes_font_dir_with_special_chars(video_generator, tmp_path):
"""Test _build_ass_filter properly escapes font directory with special characters."""
# Create a font directory with special characters
font_dir = tmp_path / "It's a Font Dir"
font_dir.mkdir()
font_file = font_dir / "test.ttf"
font_file.touch()
# Update styles to include font path
video_generator.styles["karaoke"]["font_path"] = str(font_file)
ass_path = "test.ass"
result = video_generator._build_ass_filter(ass_path)
# Should have fontsdir with escaped path
assert "fontsdir=" in result
# Apostrophe should be escaped using \\\' pattern (3 backslashes + quote)
assert "\\\\\\'s" in result
Generated
+467 -454
View File
File diff suppressed because it is too large. Load diff
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.90.0"
version = "0.92.0"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"
+41
View File
@@ -0,0 +1,41 @@
#!/bin/bash
# Admin API helper script
# Usage: ./scripts/admin_api.sh <endpoint> [method] [data]
#
# Examples:
# ./scripts/admin_api.sh /api/jobs # List all jobs (admin sees all)
# ./scripts/admin_api.sh /api/jobs?limit=10 # List 10 jobs
# ./scripts/admin_api.sh /api/jobs/abc123 # Get specific job
# ./scripts/admin_api.sh /api/jobs/abc123/retry POST # Retry a job
# ./scripts/admin_api.sh /api/jobs/abc123/select-instrumental POST '{"selection":"clean"}'
set -e
API_URL="${KARAOKE_API_URL:-https://api.nomadkaraoke.com}"
# Get admin token from Secret Manager if not set
if [ -z "$KARAOKE_ADMIN_TOKEN" ]; then
KARAOKE_ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke 2>/dev/null)
fi
if [ -z "$KARAOKE_ADMIN_TOKEN" ]; then
echo "Error: Could not get admin token. Run 'gcloud auth login' first." >&2
exit 1
fi
ENDPOINT="${1:-/api/admin/jobs}"
METHOD="${2:-GET}"
DATA="$3"
if [ "$METHOD" = "GET" ]; then
curl -s "${API_URL}${ENDPOINT}" \
-H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" | jq .
elif [ -n "$DATA" ]; then
curl -s -X "$METHOD" "${API_URL}${ENDPOINT}" \
-H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d "$DATA" | jq .
else
curl -s -X "$METHOD" "${API_URL}${ENDPOINT}" \
-H "Authorization: Bearer $KARAOKE_ADMIN_TOKEN" | jq .
fi
+34 -24
View File
@@ -121,16 +121,25 @@ class TestVideoBackgroundProcessor:
processor.get_audio_duration("/path/to/audio.wav")
def test_escape_filter_path(self, processor):
"""Test path escaping for ffmpeg filters."""
# Test backslash escaping
assert processor.escape_filter_path("C:\\Users\\test\\file.ass") == "C\\:\\\\Users\\\\test\\\\file.ass"
# Test colon escaping
assert processor.escape_filter_path("/path/to/file.ass") == "/path/to/file.ass"
assert processor.escape_filter_path("C:/path/file.ass") == "C\\:/path/file.ass"
# Test combined escaping
assert processor.escape_filter_path("C:\\path:with:colons\\file.ass") == "C\\:\\\\path\\:with\\:colons\\\\file.ass"
"""Test path escaping for ffmpeg filters using single-quote wrapping."""
# Test simple path (wrapped in single quotes)
assert processor.escape_filter_path("/path/to/file.ass") == "'/path/to/file.ass'"
# Test path with spaces (protected by single quotes)
assert processor.escape_filter_path("/path/with spaces/file.ass") == "'/path/with spaces/file.ass'"
# Test apostrophe escaping (fixes paths like "I'm With You")
# Single quote becomes '\'' (end quote, escaped quote, start quote)
assert processor.escape_filter_path("./Avril Lavigne - I'm With You/file.ass") == "'./Avril Lavigne - I'\\''m With You/file.ass'"
# Test path with colons (protected by single quotes)
assert processor.escape_filter_path("C:/path/file.ass") == "'C:/path/file.ass'"
# Test Windows-style path (backslashes preserved, wrapped in quotes)
assert processor.escape_filter_path("C:\\Users\\test\\file.ass") == "'C:\\Users\\test\\file.ass'"
# Test multiple apostrophes
assert processor.escape_filter_path("It's Bob's file.ass") == "'It'\\''s Bob'\\''s file.ass'"
def test_build_video_filter_no_darkness(self, processor):
"""Test video filter building without darkness overlay."""
@@ -139,13 +148,13 @@ class TestVideoBackgroundProcessor:
darkness_percent=0,
fonts_dir=None
)
# Should contain scale and crop
assert "scale=w=3840:h=2160:force_original_aspect_ratio=increase,crop=3840:2160" in vf_filter
# Should contain ASS filter
assert "ass=/path/to/subs.ass" in vf_filter
# Should contain ASS filter with single-quoted path
assert "ass='/path/to/subs.ass'" in vf_filter
# Should NOT contain darkening
assert "drawbox" not in vf_filter
@@ -156,10 +165,10 @@ class TestVideoBackgroundProcessor:
darkness_percent=50,
fonts_dir=None
)
# Should contain darkening with correct alpha
assert "drawbox=x=0:y=0:w=iw:h=ih:color=black@0.50:t=fill" in vf_filter
# Should contain ASS filter after darkening
assert vf_filter.index("drawbox") < vf_filter.index("ass=")
@@ -171,20 +180,21 @@ class TestVideoBackgroundProcessor:
darkness_percent=0,
fonts_dir="/path/to/fonts"
)
# Should contain fonts directory in ASS filter
assert "fontsdir=/path/to/fonts" in vf_filter
# Should contain fonts directory in ASS filter (single-quoted)
assert "fontsdir='/path/to/fonts'" in vf_filter
def test_build_video_filter_path_escaping(self, processor):
"""Test video filter building with special characters in paths."""
# Test with apostrophe in path
vf_filter = processor.build_video_filter(
ass_subtitles_path="C:\\Users\\test\\subs.ass",
ass_subtitles_path="./Avril Lavigne - I'm With You/subs.ass",
darkness_percent=0,
fonts_dir=None
)
# Paths should be escaped
assert "C\\:\\\\Users\\\\test\\\\subs.ass" in vf_filter
# Paths should be single-quoted with apostrophes escaped as '\''
assert "'./Avril Lavigne - I'\\''m With You/subs.ass'" in vf_filter
@patch('subprocess.run')
def test_execute_command_with_fallback_gpu_success(self, mock_run, processor_with_nvenc, mock_logger):